2024-04-29T13:46:17.271 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-29T13:46:17.271 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/radosbench} --owner scheduled_pdiazbou@teuthology --verbose --name pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi --archive /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421 -- /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/orig.config.yaml 2024-04-29T13:46:17.316 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-29T13:46:17.411 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421 branch: wip-pereman2-buffer-space description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/radosbench} email: pdiazbou@redhat.com first_in_suite: false job_id: '7678421' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-pereman2-buffer-space ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum mon_bind_msgr2: false sha1: 7fe685c922bae3b52755209e35feddd972749cff ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7fe685c922bae3b52755209e35feddd972749cff selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-pereman2-buffer-space sha1: 7fe685c922bae3b52755209e35feddd972749cff owner: scheduled_pdiazbou@teuthology priority: 200 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 4693 sha1: 7fe685c922bae3b52755209e35feddd972749cff sleep_before_teardown: 0 subset: 512/20000 suite: rados suite_branch: wip-pereman2-buffer-space suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7fe685c922bae3b52755209e35feddd972749cff/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7fe685c922bae3b52755209e35feddd972749cff targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJmHQQkD1bREDUR//8JHRmFsHHxnXXjYGdMeG8lxVxeTmtyrZm8NJXJWTRvSgb+4/L98vO+TzhzZw6jRJBnsA2o= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA1VeLUMJ6vUm3FCY9hiAxRSrogfkKei1PjjD4Cu3FvinZMnK9/482YsgEEZ4c2eFN8ftqJ6BPgE70NXO5CNFas= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjeWGqWUSXhzb7sXQf/Q0dcM/QY+48mbtd09GedxFcMBZ9JffGaI6hrOGJZYVNQxT2ja7ImS64Bxf+a74Q0g24= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-29_10:44:48 tube: smithi user: pdiazbou verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-29T13:46:17.411 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7fe685c922bae3b52755209e35feddd972749cff/qa; will attempt to use it 2024-04-29T13:46:17.412 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7fe685c922bae3b52755209e35feddd972749cff/qa/tasks 2024-04-29T13:46:17.412 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-29T13:46:17.413 INFO:teuthology.task.internal:Saving configuration 2024-04-29T13:46:17.436 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-29T13:46:17.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-29T13:46:17.464 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 13:41:08.899798', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJmHQQkD1bREDUR//8JHRmFsHHxnXXjYGdMeG8lxVxeTmtyrZm8NJXJWTRvSgb+4/L98vO+TzhzZw6jRJBnsA2o='} 2024-04-29T13:46:17.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 13:41:08.903325', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA1VeLUMJ6vUm3FCY9hiAxRSrogfkKei1PjjD4Cu3FvinZMnK9/482YsgEEZ4c2eFN8ftqJ6BPgE70NXO5CNFas='} 2024-04-29T13:46:17.505 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 13:41:08.901804', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjeWGqWUSXhzb7sXQf/Q0dcM/QY+48mbtd09GedxFcMBZ9JffGaI6hrOGJZYVNQxT2ja7ImS64Bxf+a74Q0g24='} 2024-04-29T13:46:17.505 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-29T13:46:17.512 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-29T13:46:17.512 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-29T13:46:17.512 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-29T13:46:17.512 INFO:teuthology.run_tasks:Running task console_log... 2024-04-29T13:46:17.630 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f899d863160>, signals=[15]) 2024-04-29T13:46:17.631 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-29T13:46:17.637 INFO:teuthology.task.internal:Opening connections... 2024-04-29T13:46:17.637 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-29T13:46:17.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:46:17.722 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-04-29T13:46:17.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:46:17.800 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-04-29T13:46:17.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:46:17.872 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-29T13:46:17.879 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-29T13:46:17.903 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-29T13:46:17.903 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-29T13:46:17.965 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-29T13:46:17.965 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-29T13:46:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-29T13:46:17.967 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-29T13:46:17.999 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-04-29T13:46:18.021 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-04-29T13:46:18.021 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-29T13:46:18.081 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-04-29T13:46:18.082 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-29T13:46:18.082 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-29T13:46:18.082 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-29T13:46:18.082 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-04-29T13:46:18.099 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-04-29T13:46:18.121 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-04-29T13:46:18.122 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8" 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8" 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8" 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-29T13:46:18.182 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-04-29T13:46:18.183 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-29T13:46:18.183 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-04-29T13:46:18.183 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-29T13:46:18.183 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-29T13:46:18.183 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-29T13:46:18.183 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-04-29T13:46:18.207 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-29T13:46:18.218 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-29T13:46:18.253 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-29T13:46:18.253 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-29T13:46:18.256 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-04-29T13:46:18.258 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-04-29T13:46:18.277 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-29T13:46:18.284 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-29T13:46:18.284 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-29T13:46:18.317 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-04-29T13:46:18.319 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-04-29T13:46:18.347 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-29T13:46:18.424 INFO:teuthology.run_tasks:Running task kernel... 2024-04-29T13:46:18.441 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-29T13:46:18.441 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-29T13:46:18.442 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-29T13:46:18.442 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-29T13:46:18.443 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-29T13:46:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T13:46:18.462 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-29T13:46:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T13:46:18.463 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-04-29T13:46:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T13:46:18.464 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-29T13:46:18.522 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-489.el8.x86_64 2024-04-29T13:46:18.522 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.18.0-489.el8.x86_64 2024-04-29T13:46:18.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-04-29T13:46:18.523 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-489.el8.x86_64 2024-04-29T13:46:18.524 INFO:teuthology.task.kernel:Running kernel on smithi194: 4.18.0-489.el8.x86_64 2024-04-29T13:46:18.524 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-04-29T13:46:18.527 INFO:teuthology.orchestra.run.smithi040.stdout:4.18.0-489.el8.x86_64 2024-04-29T13:46:18.527 INFO:teuthology.task.kernel:Running kernel on smithi040: 4.18.0-489.el8.x86_64 2024-04-29T13:46:18.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-04-29T13:46:22.636 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - AppStream 17 MB/s | 28 MB 00:01 2024-04-29T13:46:22.821 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 16 MB/s | 28 MB 00:01 2024-04-29T13:46:23.221 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-04-29T13:46:28.838 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-29T13:46:28.978 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:00 2024-04-29T13:46:29.205 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-29T13:46:31.039 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-04-29T13:46:31.337 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras 28 kB/s | 18 kB 00:00 2024-04-29T13:46:31.372 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 34 kB/s | 7.7 kB 00:00 2024-04-29T13:46:31.480 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 27 kB/s | 18 kB 00:00 2024-04-29T13:46:31.682 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-04-29T13:46:31.712 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras common packages 30 kB/s | 7.7 kB 00:00 2024-04-29T13:46:31.972 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 7.7 kB 00:00 2024-04-29T13:46:31.989 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-04-29T13:46:32.224 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-04-29T13:46:33.435 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 16 MB 00:01 2024-04-29T13:46:33.651 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-04-29T13:46:37.191 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 355 kB/s | 24 kB 00:00 2024-04-29T13:46:37.391 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 447 kB/s | 24 kB 00:00 2024-04-29T13:46:38.064 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 2.7 MB/s | 16 MB 00:05 2024-04-29T13:46:40.296 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-29T13:46:40.296 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-29T13:46:40.413 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-29T13:46:40.413 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-29T13:46:40.422 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:46:40.423 INFO:teuthology.orchestra.run.smithi121.stdout:Install 3 Packages 2024-04-29T13:46:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:46:40.424 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 90 M 2024-04-29T13:46:40.424 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 96 M 2024-04-29T13:46:40.424 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:46:40.528 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:46:40.529 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:46:40.529 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:46:40.529 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:46:40.529 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:46:40.529 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-29T13:46:40.529 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:46:40.529 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-29T13:46:40.530 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-29T13:46:40.530 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:46:40.530 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:46:40.530 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:46:40.530 INFO:teuthology.orchestra.run.smithi194.stdout:Install 3 Packages 2024-04-29T13:46:40.530 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:46:40.531 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 90 M 2024-04-29T13:46:40.531 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 96 M 2024-04-29T13:46:40.531 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:46:41.234 INFO:teuthology.orchestra.run.smithi121.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-04-29T13:46:41.534 INFO:teuthology.orchestra.run.smithi194.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2024-04-29T13:46:41.992 INFO:teuthology.orchestra.run.smithi121.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 24 MB/s | 36 MB 00:01 2024-04-29T13:46:42.039 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 512 kB/s | 24 kB 00:00 2024-04-29T13:46:42.284 INFO:teuthology.orchestra.run.smithi194.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 24 MB/s | 36 MB 00:01 2024-04-29T13:46:42.442 INFO:teuthology.orchestra.run.smithi121.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 22 MB/s | 44 MB 00:01 2024-04-29T13:46:42.443 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:46:42.443 INFO:teuthology.orchestra.run.smithi121.stdout:Total 45 MB/s | 90 MB 00:02 2024-04-29T13:46:42.676 INFO:teuthology.orchestra.run.smithi194.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 23 MB/s | 44 MB 00:01 2024-04-29T13:46:42.676 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:46:42.676 INFO:teuthology.orchestra.run.smithi194.stdout:Total 42 MB/s | 90 MB 00:02 2024-04-29T13:46:43.122 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:46:43.362 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:46:43.540 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:46:43.540 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:46:43.776 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:46:43.776 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:46:44.783 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:46:44.802 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:46:45.037 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:46:45.050 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:46:45.236 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-29T13:46:45.236 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-29T13:46:45.357 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:46:45.358 INFO:teuthology.orchestra.run.smithi040.stdout:Install 3 Packages 2024-04-29T13:46:45.359 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:46:45.359 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 90 M 2024-04-29T13:46:45.359 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 96 M 2024-04-29T13:46:45.359 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:46:46.174 INFO:teuthology.orchestra.run.smithi040.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-04-29T13:46:46.907 INFO:teuthology.orchestra.run.smithi040.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 24 MB/s | 36 MB 00:01 2024-04-29T13:46:47.445 INFO:teuthology.orchestra.run.smithi040.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 21 MB/s | 44 MB 00:02 2024-04-29T13:46:47.445 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:46:47.445 INFO:teuthology.orchestra.run.smithi040.stdout:Total 43 MB/s | 90 MB 00:02 2024-04-29T13:46:47.610 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:46:47.844 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:46:48.087 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:46:48.166 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:46:48.387 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:46:48.615 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:46:48.615 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:46:49.940 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:46:49.967 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:46:51.649 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:46:51.963 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:46:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:46:52.370 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:46:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:46:53.402 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:46:55.459 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:46:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:46:55.868 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:46:56.165 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:46:57.159 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:46:57.551 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:47:01.234 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:47:01.562 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:17.669 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:17.692 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:18.049 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:18.049 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:47:18.050 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:47:18.271 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:18.294 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:47:18.552 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:47:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-29T13:47:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-29T13:47:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:47:18.552 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:47:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:47:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:47:18.697 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T13:47:18.912 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-29T13:47:18.912 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-29T13:47:18.912 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-29T13:47:18.912 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-29T13:47:18.912 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-29T13:47:18.913 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-29T13:47:18.913 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-04-29T13:47:18.913 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T13:47:19.124 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-29T13:47:19.124 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-29T13:47:19.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-04-29T13:47:19.153 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:19.153 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:47:19.153 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:47:19.153 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-29T13:47:19.153 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-29T13:47:19.153 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:47:19.153 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:47:19.157 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.02 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 129.el8 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Group : System Environment/Base 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 9265586 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-29T13:47:19.158 INFO:teuthology.orchestra.run.smithi121.stdout:Relocations : (not relocatable) 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : CentOS BuildSystem 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:47:19.159 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-04-29T13:47:19.160 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-29T13:47:19.160 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 4.18.0-552.el8.x86_64 2024-04-29T13:47:19.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-29T13:47:19.307 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T13:47:19.542 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-29T13:47:19.542 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-29T13:47:19.543 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-29T13:47:19.543 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-29T13:47:19.543 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-29T13:47:19.543 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-29T13:47:19.543 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-04-29T13:47:19.543 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T13:47:19.755 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-29T13:47:19.755 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-29T13:47:19.755 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-04-29T13:47:19.788 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.02 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 129.el8 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Group : System Environment/Base 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 9265586 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-29T13:47:19.789 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:Relocations : (not relocatable) 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : CentOS BuildSystem 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:47:19.790 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-04-29T13:47:19.791 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-29T13:47:19.791 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 4.18.0-552.el8.x86_64 2024-04-29T13:47:19.792 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-29T13:47:20.658 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-04-29T13:47:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-04-29T13:47:22.488 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-04-29T13:47:22.490 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-29T13:47:22.528 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.QKpD6JuS8B 2024-04-29T13:47:22.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QKpD6JuS8B 2024-04-29T13:47:22.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.QKpD6JuS8B 2024-04-29T13:47:22.754 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.QKpD6JuS8B is 6KB 2024-04-29T13:47:22.803 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.QKpD6JuS8B 2024-04-29T13:47:22.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-04-29T13:47:22.885 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-29T13:47:22.885 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-29T13:47:22.885 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-29T13:47:22.885 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-29T13:47:22.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-29T13:47:23.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-04-29T13:47:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-04-29T13:47:23.291 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-04-29T13:47:23.325 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.G3GKcJ67nH 2024-04-29T13:47:23.325 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.G3GKcJ67nH 2024-04-29T13:47:23.399 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.G3GKcJ67nH 2024-04-29T13:47:23.543 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.G3GKcJ67nH is 6KB 2024-04-29T13:47:23.591 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.G3GKcJ67nH 2024-04-29T13:47:23.609 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-04-29T13:47:23.675 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-29T13:47:23.675 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-29T13:47:23.675 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-29T13:47:23.675 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-29T13:47:23.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-29T13:47:23.840 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-04-29T13:47:25.043 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:25.068 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:25.428 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:25.428 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-29T13:47:25.428 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-29T13:47:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-29T13:47:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:47:26.058 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:47:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-29T13:47:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-29T13:47:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:47:26.059 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:47:26.201 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T13:47:26.470 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-29T13:47:26.470 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-29T13:47:26.470 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-29T13:47:26.470 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-29T13:47:26.470 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-29T13:47:26.470 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-29T13:47:26.470 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2024-04-29T13:47:26.471 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-04-29T13:47:26.692 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-29T13:47:26.693 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-29T13:47:26.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-04-29T13:47:26.727 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.02 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 129.el8 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Group : System Environment/Base 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 9265586 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-29T13:47:26.728 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:Relocations : (not relocatable) 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : CentOS BuildSystem 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:47:26.729 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-04-29T13:47:26.730 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-29T13:47:26.730 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 4.18.0-552.el8.x86_64 2024-04-29T13:47:26.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-29T13:47:28.397 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-04-29T13:47:30.097 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-04-29T13:47:30.098 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-04-29T13:47:30.126 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.aOL9VZj2DI 2024-04-29T13:47:30.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aOL9VZj2DI 2024-04-29T13:47:30.210 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.aOL9VZj2DI 2024-04-29T13:47:30.352 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.aOL9VZj2DI is 6KB 2024-04-29T13:47:30.400 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.aOL9VZj2DI 2024-04-29T13:47:30.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-04-29T13:47:30.493 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-29T13:47:30.494 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-29T13:47:30.494 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-29T13:47:30.494 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-29T13:47:30.495 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-29T13:47:30.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-04-29T13:47:53.050 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-29T13:47:53.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-29T13:47:53.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:47:53.843 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-29T13:47:53.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-29T13:47:53.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:48:00.677 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-29T13:48:00.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-29T13:48:00.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:48:11.423 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-29T13:48:12.194 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-04-29T13:48:19.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-04-29T13:48:20.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-29T13:48:20.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:48:21.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-29T13:48:21.196 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:48:23.490 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-29T13:48:24.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-04-29T13:48:28.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-29T13:48:28.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:48:35.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-29T13:48:35.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:48:36.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-29T13:48:36.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:48:53.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-29T13:48:54.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-04-29T13:49:08.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-29T13:49:08.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:49:09.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-29T13:49:09.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:49:10.012 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-04-29T13:49:10.232 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-29T13:49:10.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-29T13:49:10.470 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-29T13:49:10.470 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-04-29T13:49:10.487 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-552.el8.x86_64 2024-04-29T13:49:10.487 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-29T13:49:10.487 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-29T13:49:10.487 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-29T13:49:10.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-29T13:49:10.797 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-29T13:49:10.797 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-29T13:49:10.815 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-552.el8.x86_64 2024-04-29T13:49:10.815 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-29T13:49:10.815 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-29T13:49:10.815 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-29T13:49:11.488 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-29T13:49:11.488 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-29T13:49:11.816 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-29T13:49:11.817 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-29T13:49:11.912 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-04-29T13:49:11.926 DEBUG:teuthology.parallel:result is None 2024-04-29T13:49:12.163 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-04-29T13:49:12.200 DEBUG:teuthology.parallel:result is None 2024-04-29T13:49:28.109 DEBUG:teuthology.orchestra.remote:timed out 2024-04-29T13:49:40.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-29T13:49:40.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:49:40.376 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-29T13:49:40.668 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-29T13:49:40.668 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-29T13:49:40.668 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-29T13:49:40.688 INFO:teuthology.orchestra.run.smithi040.stdout:4.18.0-552.el8.x86_64 2024-04-29T13:49:40.688 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-29T13:49:40.688 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-29T13:49:40.688 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-29T13:49:41.689 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-29T13:49:41.689 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-29T13:49:41.779 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-04-29T13:49:41.791 DEBUG:teuthology.parallel:result is None 2024-04-29T13:49:41.792 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-29T13:49:41.799 INFO:teuthology.task.internal:Creating test directory... 2024-04-29T13:49:41.799 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-29T13:49:41.802 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-29T13:49:41.804 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-29T13:49:41.828 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-29T13:49:41.835 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-29T13:49:41.860 INFO:teuthology.task.internal:Creating archive directory... 2024-04-29T13:49:41.860 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-29T13:49:41.865 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-29T13:49:41.867 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-29T13:49:41.904 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-29T13:49:41.912 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-29T13:49:41.913 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-29T13:49:41.938 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-29T13:49:41.941 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-29T13:49:41.980 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T13:49:41.980 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T13:49:41.984 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T13:49:41.991 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T13:49:41.991 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T13:49:41.995 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-29T13:49:41.997 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-29T13:49:42.004 INFO:teuthology.task.internal:Configuring sudo... 2024-04-29T13:49:42.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-29T13:49:42.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-29T13:49:42.035 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-29T13:49:42.064 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-29T13:49:42.074 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-29T13:49:42.074 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-29T13:49:42.077 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-29T13:49:42.105 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-29T13:49:42.124 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T13:49:42.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T13:49:42.236 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T13:49:42.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T13:49:42.366 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T13:49:42.366 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-29T13:49:42.434 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T13:49:42.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T13:49:42.539 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T13:49:42.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T13:49:42.670 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T13:49:42.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-29T13:49:42.737 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T13:49:42.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-29T13:49:42.846 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T13:49:42.894 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-29T13:49:42.937 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T13:49:42.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-29T13:49:43.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-29T13:49:43.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-04-29T13:49:43.011 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-04-29T13:49:43.064 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T13:49:43.073 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T13:49:43.114 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T13:49:43.394 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-29T13:49:43.402 INFO:teuthology.task.internal:Starting timer... 2024-04-29T13:49:43.402 INFO:teuthology.run_tasks:Running task pcp... 2024-04-29T13:49:43.430 INFO:teuthology.run_tasks:Running task selinux... 2024-04-29T13:49:43.445 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-29T13:49:43.446 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-29T13:49:43.560 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-29T13:49:43.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-04-29T13:49:43.629 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-04-29T13:49:43.631 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-04-29T13:49:43.721 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-04-29T13:49:43.723 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-29T13:49:43.723 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-29T13:49:43.761 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-29T13:49:43.761 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-04-29T13:49:43.795 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-04-29T13:49:43.795 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-04-29T13:49:43.859 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-04-29T13:49:43.859 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-04-29T13:49:43.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-29T13:49:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T13:49:43.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-29T13:49:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T13:49:43.918 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-29T13:49:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T13:49:43.945 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-29T13:49:43.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-29T13:49:43.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-04-29T13:49:44.002 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-04-29T13:49:44.030 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-29T13:49:44.039 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-29T13:49:44.054 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-29T13:49:44.055 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-29T13:55:41.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-04-29T13:55:41.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-29T13:55:41.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:55:41.922 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-29T13:55:41.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-29T13:55:41.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-29T13:55:41.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:55:42.066 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-29T13:55:42.132 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-29T13:55:42.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-29T13:55:42.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-29T13:55:42.203 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-04-29T13:55:42.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-29T13:55:42.274 INFO:teuthology.run_tasks:Running task clock... 2024-04-29T13:55:42.286 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-29T13:55:42.286 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-29T13:55:42.287 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T13:55:42.289 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-29T13:55:42.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T13:55:42.291 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-29T13:55:42.292 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T13:55:42.321 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-29T13:55:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-29T13:55:42.335 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-29T13:55:42.338 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-29T13:55:42.348 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-29T13:55:42.362 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-29T13:55:42.404 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-04-29T13:55:42.409 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-04-29T13:55:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-04-29T13:55:42.421 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-04-29T13:55:42.429 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-29T13:55:42.436 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-29T13:55:42.442 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-29T13:55:42.447 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-04-29T13:55:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-29T13:55:42.458 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-04-29T13:55:42.472 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-29T13:55:42.485 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-29T13:55:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2024-04-29T13:55:42.512 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T13:55:42.513 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-29T13:55:42.513 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.513 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.513 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.513 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.513 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2024-04-29T13:55:42.516 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T13:55:42.516 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-29T13:55:42.516 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.516 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.516 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.516 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.554 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-04-29T13:55:42.557 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T13:55:42.557 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-04-29T13:55:42.557 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.557 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.557 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.557 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T13:55:42.558 INFO:teuthology.run_tasks:Running task pexec... 2024-04-29T13:55:42.567 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-29T13:55:42.567 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi040.front.sepia.ceph.com 2024-04-29T13:55:42.568 DEBUG:teuthology.orchestra.run.smithi040:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-29T13:55:42.568 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi121.front.sepia.ceph.com 2024-04-29T13:55:42.568 DEBUG:teuthology.orchestra.run.smithi121:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-29T13:55:42.568 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi194.front.sepia.ceph.com 2024-04-29T13:55:42.568 DEBUG:teuthology.orchestra.run.smithi194:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-29T13:55:43.332 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:30 ago on Mon 29 Apr 2024 01:54:13 PM UTC. 2024-04-29T13:55:43.361 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:22 ago on Mon 29 Apr 2024 01:54:21 PM UTC. 2024-04-29T13:55:43.361 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:31 ago on Mon 29 Apr 2024 01:54:12 PM UTC. 2024-04-29T13:55:45.474 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout:Resetting modules: 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:55:45.475 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:55:45.513 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:55:45.514 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:55:45.514 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:55:45.514 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:55:45.514 INFO:teuthology.orchestra.run.smithi194.stdout:Resetting modules: 2024-04-29T13:55:45.515 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools 2024-04-29T13:55:45.515 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:55:45.515 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:55:45.515 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:55:45.515 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:55:45.674 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:55:45.675 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:55:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:55:45.675 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:55:45.676 INFO:teuthology.orchestra.run.smithi040.stdout:Resetting modules: 2024-04-29T13:55:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools 2024-04-29T13:55:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:55:45.676 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:55:45.676 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:55:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:55:45.729 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:55:45.776 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:55:45.934 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:55:46.524 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:33 ago on Mon 29 Apr 2024 01:54:13 PM UTC. 2024-04-29T13:55:46.587 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:34 ago on Mon 29 Apr 2024 01:54:12 PM UTC. 2024-04-29T13:55:46.783 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:25 ago on Mon 29 Apr 2024 01:54:21 PM UTC. 2024-04-29T13:55:48.677 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:55:48.714 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:55:48.714 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-29T13:55:48.714 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:55:48.714 INFO:teuthology.orchestra.run.smithi121.stdout:Installing group/module packages: 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-04-29T13:55:48.715 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout:Installing module profiles: 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools/common 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools rhel8 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-04-29T13:55:48.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:55:48.717 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 27 M 2024-04-29T13:55:48.717 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 91 M 2024-04-29T13:55:48.717 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:55:48.772 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:55:48.810 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:55:48.810 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-29T13:55:48.810 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout:Installing group/module packages: 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-29T13:55:48.811 INFO:teuthology.orchestra.run.smithi194.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout:Installing module profiles: 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools/common 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling module streams: 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools rhel8 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-04-29T13:55:48.812 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:55:48.813 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 27 M 2024-04-29T13:55:48.813 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 91 M 2024-04-29T13:55:48.813 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:55:49.084 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:55:49.110 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.8 MB/s | 689 kB 00:00 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout:Installing group/module packages: 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-29T13:55:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout:Installing module profiles: 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools/common 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling module streams: 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools rhel8 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:55:49.127 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:55:49.128 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:55:49.128 INFO:teuthology.orchestra.run.smithi040.stdout:Install 13 Packages 2024-04-29T13:55:49.128 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:55:49.128 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 27 M 2024-04-29T13:55:49.128 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 91 M 2024-04-29T13:55:49.128 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:55:49.144 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.6 MB/s | 1.0 MB 00:00 2024-04-29T13:55:49.161 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 753 kB/s | 38 kB 00:00 2024-04-29T13:55:49.194 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 5.0 MB/s | 258 kB 00:00 2024-04-29T13:55:49.218 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.2 MB/s | 1.0 MB 00:00 2024-04-29T13:55:49.251 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): cockpit-podman-84.1-1.module_el8+977+8a 2.0 MB/s | 689 kB 00:00 2024-04-29T13:55:49.277 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 648 kB/s | 38 kB 00:00 2024-04-29T13:55:49.295 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): buildah-1.33.6-2.module_el8+976+1136073 23 MB/s | 9.6 MB 00:00 2024-04-29T13:55:49.319 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 3.8 MB/s | 258 kB 00:00 2024-04-29T13:55:49.320 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-podman-4.9.0-1.module_el8+847+7 1.4 MB/s | 180 kB 00:00 2024-04-29T13:55:49.335 INFO:teuthology.orchestra.run.smithi040.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 4.4 MB/s | 689 kB 00:00 2024-04-29T13:55:49.369 INFO:teuthology.orchestra.run.smithi040.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 5.4 MB/s | 1.0 MB 00:00 2024-04-29T13:55:49.370 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-04-29T13:55:49.377 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 3.0 MB/s | 180 kB 00:00 2024-04-29T13:55:49.394 INFO:teuthology.orchestra.run.smithi040.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 646 kB/s | 38 kB 00:00 2024-04-29T13:55:49.395 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2024-04-29T13:55:49.419 INFO:teuthology.orchestra.run.smithi040.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 5.0 MB/s | 258 kB 00:00 2024-04-29T13:55:49.478 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): buildah-1.33.6-2.module_el8+976+1136073 17 MB/s | 9.6 MB 00:00 2024-04-29T13:55:49.503 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 198 kB/s | 25 kB 00:00 2024-04-29T13:55:49.663 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 8.2 MB/s | 4.1 MB 00:00 2024-04-29T13:55:49.678 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 10 MB/s | 4.1 MB 00:00 2024-04-29T13:55:49.703 INFO:teuthology.orchestra.run.smithi040.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 636 kB/s | 180 kB 00:00 2024-04-29T13:55:49.795 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 22 MB/s | 2.5 MB 00:00 2024-04-29T13:55:49.796 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): skopeo-1.14.3-0.1.module_el8+885+7da14 21 MB/s | 8.8 MB 00:00 2024-04-29T13:55:49.811 INFO:teuthology.orchestra.run.smithi040.stdout:(6/13): buildah-1.33.6-2.module_el8+976+1136073 15 MB/s | 9.6 MB 00:00 2024-04-29T13:55:49.822 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 332 kB/s | 53 kB 00:00 2024-04-29T13:55:49.822 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 2.1 MB/s | 53 kB 00:00 2024-04-29T13:55:49.837 INFO:teuthology.orchestra.run.smithi040.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 185 kB/s | 25 kB 00:00 2024-04-29T13:55:49.846 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 1.6 MB/s | 41 kB 00:00 2024-04-29T13:55:49.846 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 817 kB/s | 41 kB 00:00 2024-04-29T13:55:49.862 INFO:teuthology.orchestra.run.smithi040.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-04-29T13:55:49.871 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-pyxdg-0.25-16.el8.noarch.rpm 239 kB/s | 94 kB 00:00 2024-04-29T13:55:49.897 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 5.0 MB/s | 2.5 MB 00:00 2024-04-29T13:55:49.897 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:55:49.897 INFO:teuthology.orchestra.run.smithi194.stdout:Total 25 MB/s | 27 MB 00:01 2024-04-29T13:55:49.929 INFO:teuthology.orchestra.run.smithi040.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 7.7 MB/s | 4.1 MB 00:00 2024-04-29T13:55:49.988 INFO:teuthology.orchestra.run.smithi040.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 20 MB/s | 2.5 MB 00:00 2024-04-29T13:55:50.013 INFO:teuthology.orchestra.run.smithi040.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 630 kB/s | 53 kB 00:00 2024-04-29T13:55:50.055 INFO:teuthology.orchestra.run.smithi040.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 615 kB/s | 41 kB 00:00 2024-04-29T13:55:50.080 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 15 MB/s | 8.8 MB 00:00 2024-04-29T13:55:50.080 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:55:50.080 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 27 MB 00:01 2024-04-29T13:55:50.164 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:55:50.186 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:55:50.186 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:55:50.264 INFO:teuthology.orchestra.run.smithi040.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 21 MB/s | 8.8 MB 00:00 2024-04-29T13:55:50.264 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:55:50.264 INFO:teuthology.orchestra.run.smithi040.stdout:Total 24 MB/s | 27 MB 00:01 2024-04-29T13:55:50.348 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:55:50.369 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:55:50.369 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:55:50.450 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:55:50.453 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:55:50.543 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:55:50.566 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:55:50.566 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:55:50.623 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:55:50.626 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:55:50.833 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:55:50.836 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:55:51.062 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:55:51.218 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:55:51.473 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:55:51.978 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-29T13:55:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-29T13:55:52.172 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-29T13:55:52.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-29T13:55:52.350 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-29T13:55:52.461 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-29T13:55:52.496 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-29T13:55:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-29T13:55:52.641 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-29T13:55:52.690 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-29T13:55:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-29T13:55:52.794 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-29T13:55:52.955 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-29T13:55:52.978 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-29T13:55:53.244 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-29T13:55:53.295 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-29T13:55:53.488 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-29T13:55:53.806 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-29T13:55:54.278 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-29T13:55:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-29T13:55:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-29T13:55:54.734 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-29T13:55:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-29T13:55:54.832 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-29T13:55:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-29T13:55:55.074 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-29T13:55:55.159 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-29T13:55:55.290 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-29T13:55:55.363 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-29T13:55:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-29T13:55:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-29T13:55:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-29T13:55:55.717 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-29T13:55:55.766 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-29T13:55:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-29T13:55:55.906 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-29T13:55:56.002 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-29T13:55:56.095 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-29T13:55:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-29T13:55:56.193 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-29T13:55:56.193 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-29T13:55:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-29T13:55:56.418 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-29T13:55:56.419 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-29T13:55:56.730 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:55:56.731 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:55:56.834 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-29T13:55:56.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-29T13:55:56.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-29T13:55:56.904 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-29T13:55:56.905 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-29T13:55:56.906 INFO:teuthology.orchestra.run.smithi121.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-29T13:55:56.906 INFO:teuthology.orchestra.run.smithi121.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-29T13:55:56.906 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:55:56.906 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:55:57.023 DEBUG:teuthology.parallel:result is None 2024-04-29T13:55:57.169 DEBUG:teuthology.parallel:result is None 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-29T13:55:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-29T13:55:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-29T13:55:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-29T13:55:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-29T13:55:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:55:57.369 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:55:57.675 DEBUG:teuthology.parallel:result is None 2024-04-29T13:55:57.675 INFO:teuthology.run_tasks:Running task install... 2024-04-29T13:55:57.686 DEBUG:teuthology.task.install:project ceph 2024-04-29T13:55:57.686 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7fe685c922bae3b52755209e35feddd972749cff'}} 2024-04-29T13:55:57.686 DEBUG:teuthology.task.install:config {'branch': 'nautilus', 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'flavor': 'default', 'sha1': '7fe685c922bae3b52755209e35feddd972749cff'} 2024-04-29T13:55:57.686 INFO:teuthology.task.install:Using flavor: default 2024-04-29T13:55:57.692 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-29T13:55:57.692 INFO:teuthology.task.install:extra packages: ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'] 2024-04-29T13:55:57.692 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7fe685c922bae3b52755209e35feddd972749cff', 'tag': None, 'wait_for_package': False} 2024-04-29T13:55:57.693 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:55:57.693 INFO:teuthology.packaging:ref: None 2024-04-29T13:55:57.693 INFO:teuthology.packaging:tag: None 2024-04-29T13:55:57.693 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:55:57.693 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:55:57.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-29T13:55:57.695 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7fe685c922bae3b52755209e35feddd972749cff', 'tag': None, 'wait_for_package': False} 2024-04-29T13:55:57.696 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:55:57.696 INFO:teuthology.packaging:ref: None 2024-04-29T13:55:57.696 INFO:teuthology.packaging:tag: None 2024-04-29T13:55:57.696 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:55:57.696 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:55:57.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-29T13:55:57.697 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7fe685c922bae3b52755209e35feddd972749cff', 'tag': None, 'wait_for_package': False} 2024-04-29T13:55:57.697 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:55:57.697 INFO:teuthology.packaging:ref: None 2024-04-29T13:55:57.697 INFO:teuthology.packaging:tag: None 2024-04-29T13:55:57.697 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:55:57.697 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:55:57.697 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-29T13:55:57.875 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-29T13:55:57.876 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-29T13:55:57.910 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-29T13:55:57.911 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T13:55:57.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-29T13:55:57.912 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-29T13:55:57.912 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-29T13:55:57.940 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-29T13:55:57.940 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T13:55:57.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-29T13:55:57.944 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-29T13:55:57.944 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:55:57.944 INFO:teuthology.packaging:ref: None 2024-04-29T13:55:57.944 INFO:teuthology.packaging:tag: None 2024-04-29T13:55:57.944 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:55:57.944 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:55:57.944 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-29T13:55:57.968 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-29T13:55:57.968 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:55:57.968 INFO:teuthology.packaging:ref: None 2024-04-29T13:55:57.968 INFO:teuthology.packaging:tag: None 2024-04-29T13:55:57.968 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:55:57.968 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:55:57.969 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-29T13:55:57.994 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-29T13:55:57.995 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-29T13:55:58.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-29T13:55:58.023 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-29T13:55:58.023 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T13:55:58.024 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-29T13:55:58.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-29T13:55:58.057 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-29T13:55:58.057 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:55:58.057 INFO:teuthology.packaging:ref: None 2024-04-29T13:55:58.057 INFO:teuthology.packaging:tag: None 2024-04-29T13:55:58.057 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:55:58.057 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:55:58.057 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-29T13:55:58.091 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-29T13:55:58.113 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-29T13:55:58.130 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-29T13:55:58.159 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-04-29T13:55:58.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-04-29T13:55:58.179 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-04-29T13:55:58.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-04-29T13:55:58.209 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-29T13:55:58.278 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-04-29T13:55:58.281 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-04-29T13:55:58.442 INFO:teuthology.orchestra.run.smithi040.stdout:78 files removed 2024-04-29T13:55:58.445 INFO:teuthology.orchestra.run.smithi121.stdout:78 files removed 2024-04-29T13:55:58.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw 2024-04-29T13:55:58.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-04-29T13:55:58.546 INFO:teuthology.orchestra.run.smithi194.stdout:78 files removed 2024-04-29T13:55:58.574 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-04-29T13:55:59.014 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 578 kB/s | 65 kB 00:00 2024-04-29T13:55:59.037 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 1.1 MB/s | 65 kB 00:00 2024-04-29T13:55:59.173 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 96 kB/s | 11 kB 00:00 2024-04-29T13:55:59.188 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 101 kB/s | 11 kB 00:00 2024-04-29T13:55:59.315 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 15 kB/s | 1.7 kB 00:00 2024-04-29T13:55:59.319 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 147 kB/s | 65 kB 00:00 2024-04-29T13:55:59.322 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 16 kB/s | 1.7 kB 00:00 2024-04-29T13:55:59.421 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 180 kB/s | 11 kB 00:00 2024-04-29T13:55:59.513 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 27 kB/s | 1.7 kB 00:00 2024-04-29T13:56:00.539 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 23 MB/s | 28 MB 00:01 2024-04-29T13:56:00.699 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-04-29T13:56:00.888 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-04-29T13:56:06.340 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-04-29T13:56:06.805 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-29T13:56:07.192 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - BaseOS 8.9 MB/s | 10 MB 00:01 2024-04-29T13:56:08.307 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2024-04-29T13:56:08.566 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 51 kB/s | 7.7 kB 00:00 2024-04-29T13:56:08.989 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2024-04-29T13:56:09.256 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 7.7 kB 00:00 2024-04-29T13:56:09.276 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2024-04-29T13:56:09.702 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 7.7 kB 00:00 2024-04-29T13:56:09.909 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - PowerTools 4.0 MB/s | 5.0 MB 00:01 2024-04-29T13:56:10.582 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - PowerTools 4.0 MB/s | 5.0 MB 00:01 2024-04-29T13:56:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - PowerTools 4.1 MB/s | 5.0 MB 00:01 2024-04-29T13:56:11.044 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-04-29T13:56:11.959 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 7.8 kB/s | 3.5 kB 00:00 2024-04-29T13:56:12.188 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-04-29T13:56:12.461 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-04-29T13:56:13.516 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-29T13:56:16.336 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 453 kB/s | 24 kB 00:00 2024-04-29T13:56:17.489 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 618 kB/s | 24 kB 00:00 2024-04-29T13:56:20.018 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:56:20.020 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:56:20.020 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-29T13:56:20.020 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-29T13:56:20.021 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-29T13:56:20.022 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout:Install 20 Packages 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 45 M 2024-04-29T13:56:20.023 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:56:20.612 INFO:teuthology.orchestra.run.smithi194.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 30 MB/s | 5.4 MB 00:00 2024-04-29T13:56:20.796 INFO:teuthology.orchestra.run.smithi194.stdout:(2/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-04-29T13:56:20.821 INFO:teuthology.orchestra.run.smithi194.stdout:(3/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 119 kB/s | 25 kB 00:00 2024-04-29T13:56:20.913 INFO:teuthology.orchestra.run.smithi194.stdout:(4/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 4.4 MB/s | 520 kB 00:00 2024-04-29T13:56:20.938 INFO:teuthology.orchestra.run.smithi194.stdout:(5/22): libradosstriper1-14.2.22-0.el8.x86_64.r 3.1 MB/s | 376 kB 00:00 2024-04-29T13:56:21.055 INFO:teuthology.orchestra.run.smithi194.stdout:(6/22): librgw2-14.2.22-0.el8.x86_64.rpm 37 MB/s | 5.3 MB 00:00 2024-04-29T13:56:21.080 INFO:teuthology.orchestra.run.smithi194.stdout:(7/22): python3-ceph-argparse-14.2.22-0.el8.x86 277 kB/s | 39 kB 00:00 2024-04-29T13:56:21.106 INFO:teuthology.orchestra.run.smithi194.stdout:(8/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-29T13:56:21.165 INFO:teuthology.orchestra.run.smithi194.stdout:(9/22): python3-rados-14.2.22-0.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-04-29T13:56:21.190 INFO:teuthology.orchestra.run.smithi194.stdout:(10/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 3.4 MB/s | 291 kB 00:00 2024-04-29T13:56:21.215 INFO:teuthology.orchestra.run.smithi194.stdout:(11/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 2.2 MB/s | 113 kB 00:00 2024-04-29T13:56:21.371 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:56:21.374 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-29T13:56:21.375 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-29T13:56:21.376 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout:Install 20 Packages 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 45 M 2024-04-29T13:56:21.377 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:56:21.424 INFO:teuthology.orchestra.run.smithi194.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 20 MB/s | 19 MB 00:00 2024-04-29T13:56:21.449 INFO:teuthology.orchestra.run.smithi194.stdout:(13/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2024-04-29T13:56:21.557 INFO:teuthology.orchestra.run.smithi194.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 329 kB/s | 44 kB 00:00 2024-04-29T13:56:21.591 INFO:teuthology.orchestra.run.smithi194.stdout:(15/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-04-29T13:56:21.675 INFO:teuthology.orchestra.run.smithi194.stdout:(16/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 209 kB/s | 47 kB 00:00 2024-04-29T13:56:21.767 INFO:teuthology.orchestra.run.smithi194.stdout:(17/22): leveldb-1.22-1.el8.x86_64.rpm 1.0 MB/s | 181 kB 00:00 2024-04-29T13:56:21.809 INFO:teuthology.orchestra.run.smithi194.stdout:(18/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-04-29T13:56:21.834 INFO:teuthology.orchestra.run.smithi194.stdout:(19/22): liboath-2.6.2-3.el8.x86_64.rpm 370 kB/s | 59 kB 00:00 2024-04-29T13:56:21.859 INFO:teuthology.orchestra.run.smithi194.stdout:(20/22): libunwind-1.3.1-3.el8.x86_64.rpm 814 kB/s | 75 kB 00:00 2024-04-29T13:56:21.890 INFO:teuthology.orchestra.run.smithi040.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 38 MB/s | 5.4 MB 00:00 2024-04-29T13:56:21.960 INFO:teuthology.orchestra.run.smithi194.stdout:(21/22): librados2-14.2.22-0.el8.x86_64.rpm 23 MB/s | 3.5 MB 00:00 2024-04-29T13:56:21.965 INFO:teuthology.orchestra.run.smithi040.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 332 kB/s | 25 kB 00:00 2024-04-29T13:56:22.001 INFO:teuthology.orchestra.run.smithi194.stdout:(22/22): librbd1-14.2.22-0.el8.x86_64.rpm 10 MB/s | 1.7 MB 00:00 2024-04-29T13:56:22.006 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:56:22.007 INFO:teuthology.orchestra.run.smithi194.stdout:Total 23 MB/s | 45 MB 00:01 2024-04-29T13:56:22.049 INFO:teuthology.orchestra.run.smithi040.stdout:(3/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 20 MB/s | 6.0 MB 00:00 2024-04-29T13:56:22.050 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:56:22.074 INFO:teuthology.orchestra.run.smithi040.stdout:(4/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 4.7 MB/s | 520 kB 00:00 2024-04-29T13:56:22.097 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:56:22.097 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:56:22.099 INFO:teuthology.orchestra.run.smithi040.stdout:(5/22): libradosstriper1-14.2.22-0.el8.x86_64.r 7.3 MB/s | 376 kB 00:00 2024-04-29T13:56:22.125 INFO:teuthology.orchestra.run.smithi040.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 1.5 MB/s | 39 kB 00:00 2024-04-29T13:56:22.150 INFO:teuthology.orchestra.run.smithi040.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 6.2 MB/s | 159 kB 00:00 2024-04-29T13:56:22.184 INFO:teuthology.orchestra.run.smithi040.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2024-04-29T13:56:22.209 INFO:teuthology.orchestra.run.smithi040.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 11 MB/s | 291 kB 00:00 2024-04-29T13:56:22.234 INFO:teuthology.orchestra.run.smithi040.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.4 MB/s | 113 kB 00:00 2024-04-29T13:56:22.335 INFO:teuthology.orchestra.run.smithi040.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 20 MB/s | 5.3 MB 00:00 2024-04-29T13:56:22.381 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:56:22.384 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:56:22.758 INFO:teuthology.orchestra.run.smithi040.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 19 MB/s | 19 MB 00:01 2024-04-29T13:56:22.786 INFO:teuthology.orchestra.run.smithi040.stdout:(13/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-04-29T13:56:22.834 INFO:teuthology.orchestra.run.smithi040.stdout:(14/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 520 kB/s | 259 kB 00:00 2024-04-29T13:56:22.853 INFO:teuthology.orchestra.run.smithi040.stdout:(15/22): python3-prettytable-0.7.2-14.el8.noarc 452 kB/s | 44 kB 00:00 2024-04-29T13:56:23.054 INFO:teuthology.orchestra.run.smithi040.stdout:(16/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 176 kB/s | 47 kB 00:00 2024-04-29T13:56:23.278 INFO:teuthology.orchestra.run.smithi040.stdout:(17/22): leveldb-1.22-1.el8.x86_64.rpm 425 kB/s | 181 kB 00:00 2024-04-29T13:56:23.309 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:56:23.387 INFO:teuthology.orchestra.run.smithi040.stdout:(18/22): gperftools-libs-2.7-9.el8.x86_64.rpm 554 kB/s | 306 kB 00:00 2024-04-29T13:56:23.413 INFO:teuthology.orchestra.run.smithi040.stdout:(19/22): liboath-2.6.2-3.el8.x86_64.rpm 164 kB/s | 59 kB 00:00 2024-04-29T13:56:23.471 INFO:teuthology.orchestra.run.smithi040.stdout:(20/22): librbd1-14.2.22-0.el8.x86_64.rpm 30 MB/s | 1.7 MB 00:00 2024-04-29T13:56:23.509 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-29T13:56:23.530 INFO:teuthology.orchestra.run.smithi040.stdout:(21/22): librados2-14.2.22-0.el8.x86_64.rpm 25 MB/s | 3.5 MB 00:00 2024-04-29T13:56:23.638 INFO:teuthology.orchestra.run.smithi040.stdout:(22/22): libunwind-1.3.1-3.el8.x86_64.rpm 209 kB/s | 75 kB 00:00 2024-04-29T13:56:23.641 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:56:23.642 INFO:teuthology.orchestra.run.smithi040.stdout:Total 20 MB/s | 45 MB 00:02 2024-04-29T13:56:23.686 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:56:23.695 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-29T13:56:23.739 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:56:23.739 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:56:24.049 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:56:24.053 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:56:24.270 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-29T13:56:24.695 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:24.979 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:56:24.986 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-29T13:56:25.181 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-29T13:56:25.190 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-29T13:56:25.405 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:25.603 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:25.673 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:25.799 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-29T13:56:25.997 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:26.305 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-29T13:56:26.316 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-29T13:56:26.472 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:26.494 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:26.563 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:26.694 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:26.749 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-29T13:56:26.880 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:26.904 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-29T13:56:26.961 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:27.098 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-29T13:56:27.314 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-29T13:56:27.517 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-29T13:56:27.583 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-29T13:56:27.593 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:15 2024-04-29T13:56:27.661 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-29T13:56:27.733 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-29T13:56:27.755 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:27.817 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-29T13:56:28.145 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-29T13:56:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-29T13:56:28.352 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-29T13:56:28.470 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-29T13:56:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-29T13:56:28.959 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-29T13:56:29.177 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-29T13:56:29.381 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-29T13:56:29.513 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-29T13:56:29.573 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-29T13:56:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-29T13:56:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-29T13:56:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-29T13:56:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-29T13:56:30.574 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:30.769 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:31.366 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 391 kB/s | 24 kB 00:00 2024-04-29T13:56:31.403 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:31.662 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-29T13:56:31.782 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-29T13:56:31.935 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:56:32.619 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:32.814 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:33.471 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:33.797 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-29T13:56:33.916 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-29T13:56:34.046 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:56:35.041 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:56:35.043 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:56:35.043 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-29T13:56:35.044 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-29T13:56:35.045 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:56:35.046 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:56:35.046 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:56:35.046 INFO:teuthology.orchestra.run.smithi121.stdout:Install 20 Packages 2024-04-29T13:56:35.046 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-04-29T13:56:35.046 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:56:35.046 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 45 M 2024-04-29T13:56:35.046 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:56:35.299 INFO:teuthology.orchestra.run.smithi121.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 47 MB/s | 5.4 MB 00:00 2024-04-29T13:56:35.325 INFO:teuthology.orchestra.run.smithi121.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-04-29T13:56:35.367 INFO:teuthology.orchestra.run.smithi121.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 12 MB/s | 520 kB 00:00 2024-04-29T13:56:35.401 INFO:teuthology.orchestra.run.smithi121.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 11 MB/s | 376 kB 00:00 2024-04-29T13:56:35.493 INFO:teuthology.orchestra.run.smithi121.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 20 MB/s | 6.0 MB 00:00 2024-04-29T13:56:35.509 INFO:teuthology.orchestra.run.smithi121.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 2.3 MB/s | 39 kB 00:00 2024-04-29T13:56:35.543 INFO:teuthology.orchestra.run.smithi121.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-04-29T13:56:35.568 INFO:teuthology.orchestra.run.smithi121.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2024-04-29T13:56:35.602 INFO:teuthology.orchestra.run.smithi121.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 8.5 MB/s | 291 kB 00:00 2024-04-29T13:56:35.627 INFO:teuthology.orchestra.run.smithi121.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.4 MB/s | 113 kB 00:00 2024-04-29T13:56:35.719 INFO:teuthology.orchestra.run.smithi121.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 17 MB/s | 5.3 MB 00:00 2024-04-29T13:56:35.936 INFO:teuthology.orchestra.run.smithi121.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 26 MB/s | 19 MB 00:00 2024-04-29T13:56:35.961 INFO:teuthology.orchestra.run.smithi121.stdout:(13/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-04-29T13:56:36.012 INFO:teuthology.orchestra.run.smithi121.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 585 kB/s | 44 kB 00:00 2024-04-29T13:56:36.045 INFO:teuthology.orchestra.run.smithi121.stdout:(15/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-04-29T13:56:36.337 INFO:teuthology.orchestra.run.smithi121.stdout:(16/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 125 kB/s | 47 kB 00:00 2024-04-29T13:56:36.379 INFO:teuthology.orchestra.run.smithi121.stdout:(17/22): leveldb-1.22-1.el8.x86_64.rpm 542 kB/s | 181 kB 00:00 2024-04-29T13:56:36.480 INFO:teuthology.orchestra.run.smithi121.stdout:(18/22): gperftools-libs-2.7-9.el8.x86_64.rpm 655 kB/s | 306 kB 00:00 2024-04-29T13:56:36.496 INFO:teuthology.orchestra.run.smithi121.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 642 kB/s | 75 kB 00:00 2024-04-29T13:56:36.539 INFO:teuthology.orchestra.run.smithi121.stdout:(20/22): liboath-2.6.2-3.el8.x86_64.rpm 293 kB/s | 59 kB 00:00 2024-04-29T13:56:36.580 INFO:teuthology.orchestra.run.smithi121.stdout:(21/22): librbd1-14.2.22-0.el8.x86_64.rpm 21 MB/s | 1.7 MB 00:00 2024-04-29T13:56:36.639 INFO:teuthology.orchestra.run.smithi121.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 22 MB/s | 3.5 MB 00:00 2024-04-29T13:56:36.641 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:56:36.641 INFO:teuthology.orchestra.run.smithi121.stdout:Total 28 MB/s | 45 MB 00:01 2024-04-29T13:56:36.683 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:56:36.731 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:56:36.731 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:56:37.027 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:56:37.030 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:56:37.927 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:56:38.169 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-29T13:56:38.361 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-29T13:56:38.725 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-29T13:56:38.969 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:39.057 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-29T13:56:39.435 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:39.642 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-29T13:56:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-29T13:56:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-29T13:56:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-29T13:56:41.579 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-29T13:56:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-29T13:56:42.051 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-29T13:56:42.121 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-29T13:56:42.354 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-29T13:56:42.568 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-29T13:56:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-29T13:56:42.960 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-29T13:56:45.027 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:45.243 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:45.877 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-29T13:56:46.142 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-29T13:56:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-29T13:56:46.409 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:56:50.900 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:56:50.900 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-04-29T13:56:50.900 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-04-29T13:56:50.900 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-04-29T13:56:50.900 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /run 2024-04-29T13:56:50.901 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-04-29T13:56:50.901 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-04-29T13:56:50.901 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-04-29T13:56:50.901 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-04-29T13:56:50.901 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-04-29T13:56:50.901 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:56:51.600 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-29T13:56:51.620 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-29T13:56:51.620 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-29T13:56:51.849 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-29T13:56:51.850 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:56:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:56:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-29T13:56:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-29T13:56:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.498 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb-1.22-1.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.499 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-29T13:56:53.500 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.500 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.500 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:53.500 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:56:53.500 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /run 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2024-04-29T13:56:53.615 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:56:53.831 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-04-29T13:56:54.327 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-29T13:56:54.348 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-29T13:56:54.348 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-29T13:56:54.545 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-29T13:56:54.545 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:56:54.562 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:38 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:56:55.262 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:56:55.262 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-29T13:56:55.262 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-29T13:56:55.262 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-29T13:56:55.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-29T13:56:55.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-29T13:56:55.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-29T13:56:55.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-29T13:56:55.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-29T13:56:55.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:56:55.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-29T13:56:55.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-29T13:56:56.190 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-29T13:56:56.191 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:56:56.192 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:56:56.192 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:56:56.192 INFO:teuthology.orchestra.run.smithi194.stdout:Install 5 Packages 2024-04-29T13:56:56.192 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:56:56.192 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 36 M 2024-04-29T13:56:56.192 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 127 M 2024-04-29T13:56:56.193 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.206 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: leveldb-1.22-1.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-29T13:56:56.208 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:56:56.208 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:56:56.421 INFO:teuthology.orchestra.run.smithi194.stdout:(1/5): jq-1.6-8.el8.x86_64.rpm 2.2 MB/s | 203 kB 00:00 2024-04-29T13:56:56.446 INFO:teuthology.orchestra.run.smithi194.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-29T13:56:56.488 INFO:teuthology.orchestra.run.smithi194.stdout:(3/5): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2024-04-29T13:56:56.513 INFO:teuthology.orchestra.run.smithi194.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-04-29T13:56:56.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test 2024-04-29T13:56:57.319 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:40 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:56:57.389 INFO:teuthology.orchestra.run.smithi194.stdout:(5/5): ceph-test-14.2.22-0.el8.x86_64.rpm 33 MB/s | 35 MB 00:01 2024-04-29T13:56:57.389 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:56:57.390 INFO:teuthology.orchestra.run.smithi194.stdout:Total 30 MB/s | 36 MB 00:01 2024-04-29T13:56:57.431 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:56:57.451 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:56:57.451 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:56:57.672 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:56:57.675 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:56:58.058 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:56:58.265 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-29T13:56:58.467 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-29T13:56:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:56:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:56:59.096 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:56:59.096 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:56:59.096 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout:Install 5 Packages 2024-04-29T13:56:59.097 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:56:59.098 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 36 M 2024-04-29T13:56:59.098 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 127 M 2024-04-29T13:56:59.098 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:56:59.286 INFO:teuthology.orchestra.run.smithi040.stdout:(1/5): oniguruma-6.8.2-3.el8.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-29T13:56:59.311 INFO:teuthology.orchestra.run.smithi040.stdout:(2/5): jq-1.6-8.el8.x86_64.rpm 1.7 MB/s | 203 kB 00:00 2024-04-29T13:56:59.345 INFO:teuthology.orchestra.run.smithi040.stdout:(3/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 71 kB 00:00 2024-04-29T13:56:59.370 INFO:teuthology.orchestra.run.smithi040.stdout:(4/5): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-04-29T13:56:59.787 INFO:teuthology.orchestra.run.smithi040.stdout:(5/5): ceph-test-14.2.22-0.el8.x86_64.rpm 58 MB/s | 35 MB 00:00 2024-04-29T13:56:59.788 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:56:59.789 INFO:teuthology.orchestra.run.smithi040.stdout:Total 51 MB/s | 36 MB 00:00 2024-04-29T13:56:59.832 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:56:59.855 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:56:59.855 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:57:00.090 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:57:00.093 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:57:00.488 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:57:00.689 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-29T13:57:00.880 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-29T13:57:01.009 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:57:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:57:01.878 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-29T13:57:02.129 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-29T13:57:02.410 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-29T13:57:02.410 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-29T13:57:02.410 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-29T13:57:02.410 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:57:02.410 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:02.884 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:03.139 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-04-29T13:57:03.877 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:47 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:04.490 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-29T13:57:04.699 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-29T13:57:04.988 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-29T13:57:04.988 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-29T13:57:04.988 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-29T13:57:04.988 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:57:04.988 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-29T13:57:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:57:05.087 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-04-29T13:57:05.087 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2024-04-29T13:57:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:05.490 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-29T13:57:05.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:05.491 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:57:05.491 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-29T13:57:05.491 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-29T13:57:05.491 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-29T13:57:05.491 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:05.491 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:57:05.520 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-29T13:57:05.523 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-29T13:57:05.524 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-29T13:57:05.525 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout:Install 31 Packages 2024-04-29T13:57:05.526 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:05.527 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 28 M 2024-04-29T13:57:05.527 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 93 M 2024-04-29T13:57:05.527 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:57:05.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph 2024-04-29T13:57:05.787 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-29T13:57:05.807 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-29T13:57:05.807 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-29T13:57:06.014 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-29T13:57:06.014 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:57:06.164 INFO:teuthology.orchestra.run.smithi194.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 86 kB/s | 6.3 kB 00:00 2024-04-29T13:57:06.214 INFO:teuthology.orchestra.run.smithi194.stdout:(2/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-04-29T13:57:06.472 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:49 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:57:06.656 INFO:teuthology.orchestra.run.smithi194.stdout:(3/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 9.1 MB/s | 4.5 MB 00:00 2024-04-29T13:57:06.698 INFO:teuthology.orchestra.run.smithi194.stdout:(4/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-04-29T13:57:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-29T13:57:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-29T13:57:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-29T13:57:06.906 INFO:teuthology.orchestra.run.smithi194.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 24 MB/s | 17 MB 00:00 2024-04-29T13:57:06.932 INFO:teuthology.orchestra.run.smithi194.stdout:(6/31): python3-markupsafe-0.23-19.el8.x86_64.r 166 kB/s | 39 kB 00:00 2024-04-29T13:57:06.957 INFO:teuthology.orchestra.run.smithi194.stdout:(7/31): python3-mako-1.0.6-14.el8.noarch.rpm 522 kB/s | 157 kB 00:00 2024-04-29T13:57:07.066 INFO:teuthology.orchestra.run.smithi194.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 425 kB/s | 57 kB 00:00 2024-04-29T13:57:07.116 INFO:teuthology.orchestra.run.smithi194.stdout:(9/31): python3-werkzeug-0.12.2-4.el8.noarch.rp 2.8 MB/s | 457 kB 00:00 2024-04-29T13:57:07.158 INFO:teuthology.orchestra.run.smithi194.stdout:(10/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 412 kB/s | 103 kB 00:00 2024-04-29T13:57:07.431 INFO:teuthology.orchestra.run.smithi194.stdout:(11/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 682 kB/s | 249 kB 00:00 2024-04-29T13:57:07.458 INFO:teuthology.orchestra.run.smithi194.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 363 kB/s | 109 kB 00:00 2024-04-29T13:57:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-29T13:57:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: leveldb-1.22-1.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:07.506 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:57:07.567 INFO:teuthology.orchestra.run.smithi194.stdout:(13/31): python3-cryptography-3.2.1-7.el8.x86_6 1.4 MB/s | 641 kB 00:00 2024-04-29T13:57:07.609 INFO:teuthology.orchestra.run.smithi194.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 245 kB/s | 44 kB 00:00 2024-04-29T13:57:07.643 INFO:teuthology.orchestra.run.smithi194.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2024-04-29T13:57:07.702 INFO:teuthology.orchestra.run.smithi194.stdout:(16/31): python3-jaraco-6.2-6.el8.noarch.rpm 191 kB/s | 11 kB 00:00 2024-04-29T13:57:07.727 INFO:teuthology.orchestra.run.smithi194.stdout:(17/31): python3-cherrypy-18.4.0-1.el8.noarch.r 3.2 MB/s | 384 kB 00:00 2024-04-29T13:57:07.752 INFO:teuthology.orchestra.run.smithi194.stdout:(18/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 936 kB/s | 173 kB 00:00 2024-04-29T13:57:07.777 INFO:teuthology.orchestra.run.smithi194.stdout:(19/31): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2024-04-29T13:57:07.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-04-29T13:57:07.802 INFO:teuthology.orchestra.run.smithi194.stdout:(20/31): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2024-04-29T13:57:07.828 INFO:teuthology.orchestra.run.smithi194.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2024-04-29T13:57:07.870 INFO:teuthology.orchestra.run.smithi194.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 234 kB/s | 16 kB 00:00 2024-04-29T13:57:07.895 INFO:teuthology.orchestra.run.smithi194.stdout:(23/31): python3-simplegeneric-0.8.1-17.el8.noa 280 kB/s | 19 kB 00:00 2024-04-29T13:57:07.920 INFO:teuthology.orchestra.run.smithi194.stdout:(24/31): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2024-04-29T13:57:07.945 INFO:teuthology.orchestra.run.smithi194.stdout:(25/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2024-04-29T13:57:07.971 INFO:teuthology.orchestra.run.smithi194.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-04-29T13:57:07.996 INFO:teuthology.orchestra.run.smithi194.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-04-29T13:57:08.021 INFO:teuthology.orchestra.run.smithi194.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-04-29T13:57:08.047 INFO:teuthology.orchestra.run.smithi194.stdout:(29/31): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-04-29T13:57:08.072 INFO:teuthology.orchestra.run.smithi194.stdout:(30/31): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2024-04-29T13:57:08.097 INFO:teuthology.orchestra.run.smithi194.stdout:(31/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.9 MB/s | 251 kB 00:00 2024-04-29T13:57:08.098 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:08.099 INFO:teuthology.orchestra.run.smithi194.stdout:Total 11 MB/s | 28 MB 00:02 2024-04-29T13:57:08.181 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:57:08.221 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:57:08.221 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:57:08.243 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:57:08.246 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:08.246 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-29T13:57:08.246 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-29T13:57:08.247 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-29T13:57:08.248 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout:Install 31 Packages 2024-04-29T13:57:08.249 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:08.250 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 28 M 2024-04-29T13:57:08.250 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 93 M 2024-04-29T13:57:08.250 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:57:08.510 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:37 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:57:08.526 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:57:08.529 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:57:08.647 INFO:teuthology.orchestra.run.smithi040.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 127 kB/s | 6.3 kB 00:00 2024-04-29T13:57:08.697 INFO:teuthology.orchestra.run.smithi040.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-29T13:57:08.739 INFO:teuthology.orchestra.run.smithi040.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-04-29T13:57:08.831 INFO:teuthology.orchestra.run.smithi040.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 24 MB/s | 4.5 MB 00:00 2024-04-29T13:57:09.207 INFO:teuthology.orchestra.run.smithi040.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 33 MB/s | 17 MB 00:00 2024-04-29T13:57:09.231 INFO:teuthology.orchestra.run.smithi040.stdout:(6/31): python3-markupsafe-0.23-19.el8.x86_64.r 97 kB/s | 39 kB 00:00 2024-04-29T13:57:09.257 INFO:teuthology.orchestra.run.smithi040.stdout:(7/31): python3-mako-1.0.6-14.el8.noarch.rpm 303 kB/s | 157 kB 00:00 2024-04-29T13:57:09.282 INFO:teuthology.orchestra.run.smithi040.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-04-29T13:57:09.307 INFO:teuthology.orchestra.run.smithi040.stdout:(9/31): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2024-04-29T13:57:09.333 INFO:teuthology.orchestra.run.smithi040.stdout:(10/31): python3-werkzeug-0.12.2-4.el8.noarch.r 5.9 MB/s | 457 kB 00:00 2024-04-29T13:57:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:57:09.617 INFO:teuthology.orchestra.run.smithi040.stdout:(11/31): python3-pycparser-2.14-14.el8.noarch.r 385 kB/s | 109 kB 00:00 2024-04-29T13:57:09.642 INFO:teuthology.orchestra.run.smithi040.stdout:(12/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 692 kB/s | 249 kB 00:00 2024-04-29T13:57:09.726 INFO:teuthology.orchestra.run.smithi040.stdout:(13/31): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-04-29T13:57:09.809 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-29T13:57:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 153 kB/s | 44 kB 00:00 2024-04-29T13:57:10.009 INFO:teuthology.orchestra.run.smithi040.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 504 kB/s | 185 kB 00:00 2024-04-29T13:57:10.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-29T13:57:10.076 INFO:teuthology.orchestra.run.smithi040.stdout:(16/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 492 kB/s | 173 kB 00:00 2024-04-29T13:57:10.093 INFO:teuthology.orchestra.run.smithi040.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-04-29T13:57:10.144 INFO:teuthology.orchestra.run.smithi040.stdout:(18/31): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2024-04-29T13:57:10.162 INFO:teuthology.orchestra.run.smithi040.stdout:(19/31): python3-jaraco-functools-2.0-4.el8.noa 209 kB/s | 18 kB 00:00 2024-04-29T13:57:10.168 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-29T13:57:10.169 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-29T13:57:10.170 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-29T13:57:10.170 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-29T13:57:10.170 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:10.170 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:57:10.170 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:10.170 INFO:teuthology.orchestra.run.smithi121.stdout:Install 5 Packages 2024-04-29T13:57:10.170 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:10.171 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 36 M 2024-04-29T13:57:10.171 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 127 M 2024-04-29T13:57:10.171 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:57:10.178 INFO:teuthology.orchestra.run.smithi040.stdout:(20/31): python3-logutils-0.3.5-11.el8.noarch.r 578 kB/s | 49 kB 00:00 2024-04-29T13:57:10.220 INFO:teuthology.orchestra.run.smithi040.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 778 kB/s | 59 kB 00:00 2024-04-29T13:57:10.245 INFO:teuthology.orchestra.run.smithi040.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-04-29T13:57:10.287 INFO:teuthology.orchestra.run.smithi040.stdout:(23/31): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2024-04-29T13:57:10.304 INFO:teuthology.orchestra.run.smithi040.stdout:(24/31): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2024-04-29T13:57:10.321 INFO:teuthology.orchestra.run.smithi040.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 316 kB/s | 24 kB 00:00 2024-04-29T13:57:10.321 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-29T13:57:10.325 INFO:teuthology.orchestra.run.smithi121.stdout:(1/5): oniguruma-6.8.2-3.el8.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-29T13:57:10.350 INFO:teuthology.orchestra.run.smithi121.stdout:(2/5): jq-1.6-8.el8.x86_64.rpm 1.6 MB/s | 203 kB 00:00 2024-04-29T13:57:10.379 INFO:teuthology.orchestra.run.smithi040.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 317 kB/s | 29 kB 00:00 2024-04-29T13:57:10.384 INFO:teuthology.orchestra.run.smithi121.stdout:(3/5): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-04-29T13:57:10.396 INFO:teuthology.orchestra.run.smithi040.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2024-04-29T13:57:10.409 INFO:teuthology.orchestra.run.smithi121.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-04-29T13:57:10.455 INFO:teuthology.orchestra.run.smithi040.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2024-04-29T13:57:10.481 INFO:teuthology.orchestra.run.smithi040.stdout:(29/31): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-04-29T13:57:10.514 INFO:teuthology.orchestra.run.smithi040.stdout:(30/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-04-29T13:57:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-29T13:57:10.547 INFO:teuthology.orchestra.run.smithi040.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 249 kB/s | 23 kB 00:00 2024-04-29T13:57:10.552 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:10.552 INFO:teuthology.orchestra.run.smithi040.stdout:Total 12 MB/s | 28 MB 00:02 2024-04-29T13:57:10.638 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:57:10.681 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:57:10.681 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:57:10.741 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-29T13:57:10.785 INFO:teuthology.orchestra.run.smithi121.stdout:(5/5): ceph-test-14.2.22-0.el8.x86_64.rpm 62 MB/s | 35 MB 00:00 2024-04-29T13:57:10.786 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:10.786 INFO:teuthology.orchestra.run.smithi121.stdout:Total 58 MB/s | 36 MB 00:00 2024-04-29T13:57:10.826 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:57:10.847 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:57:10.847 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:57:10.889 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-29T13:57:11.008 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:57:11.011 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:57:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-29T13:57:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:57:11.073 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:57:11.226 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-29T13:57:11.455 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-29T13:57:11.467 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:57:11.670 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-29T13:57:11.682 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-29T13:57:11.889 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-29T13:57:11.917 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-29T13:57:12.028 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:57:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:57:12.105 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:57:12.175 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-29T13:57:12.300 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-29T13:57:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-29T13:57:12.573 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-29T13:57:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-29T13:57:12.822 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-29T13:57:13.016 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-29T13:57:13.017 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-29T13:57:13.245 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-29T13:57:13.280 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-29T13:57:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-29T13:57:13.513 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-29T13:57:13.594 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-29T13:57:13.730 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-29T13:57:13.816 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-29T13:57:13.934 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-29T13:57:14.051 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-29T13:57:14.179 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-29T13:57:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-29T13:57:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-29T13:57:14.408 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-29T13:57:14.564 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-29T13:57:14.648 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-29T13:57:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-29T13:57:14.974 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-29T13:57:15.038 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-29T13:57:15.279 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-29T13:57:15.327 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-29T13:57:15.419 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-29T13:57:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-29T13:57:15.595 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-29T13:57:15.721 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-29T13:57:15.837 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-29T13:57:15.860 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-29T13:57:15.860 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-29T13:57:15.860 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-29T13:57:15.861 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-29T13:57:15.861 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-29T13:57:15.962 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-29T13:57:16.026 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-29T13:57:16.291 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-29T13:57:16.291 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:16.291 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:57:16.292 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-29T13:57:16.292 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-29T13:57:16.292 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-29T13:57:16.292 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:16.292 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:57:16.360 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-29T13:57:16.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-04-29T13:57:16.690 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-29T13:57:16.853 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-29T13:57:17.013 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-29T13:57:17.272 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:46 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:57:17.290 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-29T13:57:17.329 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-29T13:57:17.478 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-29T13:57:17.503 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-29T13:57:17.682 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-29T13:57:17.894 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-29T13:57:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-29T13:57:18.094 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-29T13:57:18.279 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-29T13:57:18.444 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-29T13:57:18.445 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-29T13:57:18.462 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-29T13:57:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-29T13:57:18.673 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-29T13:57:18.922 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-29T13:57:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-29T13:57:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-29T13:57:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:18.929 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:57:18.929 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:18.929 INFO:teuthology.orchestra.run.smithi121.stdout:Install 31 Packages 2024-04-29T13:57:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:18.929 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 28 M 2024-04-29T13:57:18.929 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 93 M 2024-04-29T13:57:18.929 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:57:19.026 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-29T13:57:19.026 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-29T13:57:19.026 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-29T13:57:19.026 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-29T13:57:19.026 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-29T13:57:19.027 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-29T13:57:19.027 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-29T13:57:19.027 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-29T13:57:19.027 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-29T13:57:19.027 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-29T13:57:19.028 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-29T13:57:19.028 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-29T13:57:19.028 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-29T13:57:19.028 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-29T13:57:19.028 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-29T13:57:19.028 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-29T13:57:19.029 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-29T13:57:19.029 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-29T13:57:19.029 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-29T13:57:19.029 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-29T13:57:19.029 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-29T13:57:19.030 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-29T13:57:19.030 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-29T13:57:19.030 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-29T13:57:19.030 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-29T13:57:19.030 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-29T13:57:19.031 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-29T13:57:19.031 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-29T13:57:19.031 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-29T13:57:19.031 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-29T13:57:19.031 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-29T13:57:19.435 INFO:teuthology.orchestra.run.smithi121.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 140 kB/s | 6.3 kB 00:00 2024-04-29T13:57:19.477 INFO:teuthology.orchestra.run.smithi121.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 18 MB/s | 1.6 MB 00:00 2024-04-29T13:57:19.538 INFO:teuthology.orchestra.run.smithi121.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-04-29T13:57:19.653 INFO:teuthology.orchestra.run.smithi121.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 21 MB/s | 4.5 MB 00:00 2024-04-29T13:57:19.678 INFO:teuthology.orchestra.run.smithi121.stdout:(5/31): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-04-29T13:57:19.720 INFO:teuthology.orchestra.run.smithi121.stdout:(6/31): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-04-29T13:57:19.745 INFO:teuthology.orchestra.run.smithi121.stdout:(7/31): python3-markupsafe-0.23-19.el8.x86_64.r 420 kB/s | 39 kB 00:00 2024-04-29T13:57:19.771 INFO:teuthology.orchestra.run.smithi121.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-04-29T13:57:19.938 INFO:teuthology.orchestra.run.smithi121.stdout:(9/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 36 MB/s | 17 MB 00:00 2024-04-29T13:57:19.953 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-29T13:57:19.953 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-29T13:57:19.954 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-2.6-1.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-29T13:57:19.955 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-29T13:57:19.956 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-29T13:57:19.956 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-29T13:57:19.956 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-29T13:57:19.956 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-29T13:57:19.956 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:19.956 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:19.988 INFO:teuthology.orchestra.run.smithi121.stdout:(10/31): python3-werkzeug-0.12.2-4.el8.noarch.r 1.8 MB/s | 457 kB 00:00 2024-04-29T13:57:20.132 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-29T13:57:20.211 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-04-29T13:57:20.230 INFO:teuthology.orchestra.run.smithi121.stdout:(11/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 542 kB/s | 249 kB 00:00 2024-04-29T13:57:20.264 INFO:teuthology.orchestra.run.smithi121.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 396 kB/s | 109 kB 00:00 2024-04-29T13:57:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-29T13:57:20.339 INFO:teuthology.orchestra.run.smithi121.stdout:(13/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 401 kB/s | 44 kB 00:00 2024-04-29T13:57:20.364 INFO:teuthology.orchestra.run.smithi121.stdout:(14/31): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-04-29T13:57:20.398 INFO:teuthology.orchestra.run.smithi121.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2024-04-29T13:57:20.423 INFO:teuthology.orchestra.run.smithi121.stdout:(16/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2024-04-29T13:57:20.440 INFO:teuthology.orchestra.run.smithi121.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2024-04-29T13:57:20.465 INFO:teuthology.orchestra.run.smithi121.stdout:(18/31): python3-jaraco-functools-2.0-4.el8.noa 421 kB/s | 18 kB 00:00 2024-04-29T13:57:20.491 INFO:teuthology.orchestra.run.smithi121.stdout:(19/31): python3-logutils-0.3.5-11.el8.noarch.r 968 kB/s | 49 kB 00:00 2024-04-29T13:57:20.516 INFO:teuthology.orchestra.run.smithi121.stdout:(20/31): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-04-29T13:57:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:(21/31): python3-cherrypy-18.4.0-1.el8.noarch.r 2.0 MB/s | 384 kB 00:00 2024-04-29T13:57:20.575 INFO:teuthology.orchestra.run.smithi121.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 265 kB/s | 16 kB 00:00 2024-04-29T13:57:20.600 INFO:teuthology.orchestra.run.smithi121.stdout:(23/31): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2024-04-29T13:57:20.626 INFO:teuthology.orchestra.run.smithi121.stdout:(24/31): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2024-04-29T13:57:20.651 INFO:teuthology.orchestra.run.smithi121.stdout:(25/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2024-04-29T13:57:20.676 INFO:teuthology.orchestra.run.smithi121.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 386 kB/s | 29 kB 00:00 2024-04-29T13:57:20.702 INFO:teuthology.orchestra.run.smithi121.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-04-29T13:57:20.720 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-29T13:57:20.752 INFO:teuthology.orchestra.run.smithi121.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-04-29T13:57:20.777 INFO:teuthology.orchestra.run.smithi121.stdout:(29/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2024-04-29T13:57:20.802 INFO:teuthology.orchestra.run.smithi121.stdout:(30/31): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2024-04-29T13:57:20.827 INFO:teuthology.orchestra.run.smithi121.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 304 kB/s | 23 kB 00:00 2024-04-29T13:57:20.828 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:20.829 INFO:teuthology.orchestra.run.smithi121.stdout:Total 15 MB/s | 28 MB 00:01 2024-04-29T13:57:20.873 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-29T13:57:20.905 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:57:20.943 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:57:20.943 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:57:20.943 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:04 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:21.068 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-29T13:57:21.237 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:57:21.240 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:57:21.290 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-29T13:57:21.308 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-29T13:57:21.440 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-29T13:57:21.876 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-29T13:57:21.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-29T13:57:21.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-29T13:57:21.879 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-29T13:57:21.879 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-29T13:57:21.879 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-29T13:57:22.272 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:57:22.481 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-29T13:57:22.529 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:57:22.579 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:22.580 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:57:22.580 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:22.688 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-04-29T13:57:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-29T13:57:22.727 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-29T13:57:22.727 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:22.727 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-29T13:57:22.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-2.6-1.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-29T13:57:22.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-29T13:57:22.730 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-29T13:57:22.730 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:22.730 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:57:22.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base 2024-04-29T13:57:22.983 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-29T13:57:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-29T13:57:23.413 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-29T13:57:23.423 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:07 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:23.593 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-29T13:57:23.712 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:06 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:57:23.786 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-29T13:57:23.939 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-29T13:57:24.146 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-29T13:57:24.383 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-29T13:57:24.612 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-29T13:57:24.818 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-29T13:57:25.003 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:57:25.053 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:25.054 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:57:25.054 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:25.182 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-04-29T13:57:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-29T13:57:25.436 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:57:25.453 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-29T13:57:25.489 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:57:25.490 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:57:25.490 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:57:25.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr 2024-04-29T13:57:25.640 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-29T13:57:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-29T13:57:25.909 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:09 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:26.126 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-29T13:57:26.359 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:09 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:57:26.454 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-29T13:57:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-29T13:57:26.827 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-29T13:57:27.013 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-29T13:57:27.255 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-29T13:57:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-29T13:57:27.561 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-29T13:57:27.562 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-29T13:57:27.563 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-29T13:57:27.563 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:27.563 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:57:27.563 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:27.563 INFO:teuthology.orchestra.run.smithi194.stdout:Install 5 Packages 2024-04-29T13:57:27.563 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:27.563 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 4.0 M 2024-04-29T13:57:27.564 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 17 M 2024-04-29T13:57:27.564 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:57:27.669 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-29T13:57:27.876 INFO:teuthology.orchestra.run.smithi194.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 409 kB/s | 24 kB 00:00 2024-04-29T13:57:27.968 INFO:teuthology.orchestra.run.smithi194.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 25 MB/s | 3.7 MB 00:00 2024-04-29T13:57:28.017 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-29T13:57:28.043 INFO:teuthology.orchestra.run.smithi194.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2024-04-29T13:57:28.084 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:57:28.136 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:57:28.138 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:57:28.138 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:57:28.193 INFO:teuthology.orchestra.run.smithi194.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 106 kB/s | 34 kB 00:00 2024-04-29T13:57:28.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-dashboard 2024-04-29T13:57:28.343 INFO:teuthology.orchestra.run.smithi194.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 530 kB/s | 196 kB 00:00 2024-04-29T13:57:28.346 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:28.346 INFO:teuthology.orchestra.run.smithi194.stdout:Total 5.1 MB/s | 4.0 MB 00:00 2024-04-29T13:57:28.356 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:57:28.365 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:57:28.365 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:57:28.463 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-29T13:57:28.538 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:57:28.541 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:57:28.707 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-29T13:57:28.802 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:57:28.989 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-29T13:57:29.008 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:12 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:57:29.217 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-29T13:57:29.424 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-29T13:57:30.075 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-29T13:57:30.282 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-29T13:57:30.304 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-29T13:57:30.585 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-29T13:57:30.669 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-29T13:57:30.763 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-29T13:57:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout:Install 5 Packages 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 4.0 M 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 17 M 2024-04-29T13:57:30.765 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:57:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-29T13:57:30.875 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-29T13:57:30.875 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-29T13:57:30.876 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-29T13:57:30.876 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-29T13:57:30.876 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-29T13:57:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-29T13:57:31.080 INFO:teuthology.orchestra.run.smithi040.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 434 kB/s | 24 kB 00:00 2024-04-29T13:57:31.164 INFO:teuthology.orchestra.run.smithi040.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 27 MB/s | 3.7 MB 00:00 2024-04-29T13:57:31.241 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-29T13:57:31.247 INFO:teuthology.orchestra.run.smithi040.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 191 kB/s | 43 kB 00:00 2024-04-29T13:57:31.259 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-29T13:57:31.271 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:31.272 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:31.395 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-29T13:57:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-29T13:57:31.522 INFO:teuthology.orchestra.run.smithi040.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 76 kB/s | 34 kB 00:00 2024-04-29T13:57:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-29T13:57:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-29T13:57:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-29T13:57:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-29T13:57:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-29T13:57:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-29T13:57:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-29T13:57:31.820 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-29T13:57:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-29T13:57:31.847 INFO:teuthology.orchestra.run.smithi040.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 289 kB/s | 196 kB 00:00 2024-04-29T13:57:31.850 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:31.850 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.7 MB/s | 4.0 MB 00:01 2024-04-29T13:57:31.861 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:57:31.871 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:57:31.872 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:57:32.038 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:57:32.042 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:57:32.132 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:16 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:32.301 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:57:32.479 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-29T13:57:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-29T13:57:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:32.603 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:57:32.723 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-29T13:57:32.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-04-29T13:57:32.912 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-29T13:57:33.619 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:02 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:57:33.769 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:33.770 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-04-29T13:57:33.770 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-29T13:57:33.770 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-04-29T13:57:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:Install 8 Packages 2024-04-29T13:57:33.772 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:33.772 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 30 M 2024-04-29T13:57:33.772 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 172 M 2024-04-29T13:57:33.772 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:57:33.872 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-29T13:57:34.120 INFO:teuthology.orchestra.run.smithi194.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 11 MB/s | 1.1 MB 00:00 2024-04-29T13:57:34.388 INFO:teuthology.orchestra.run.smithi194.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-04-29T13:57:34.480 INFO:teuthology.orchestra.run.smithi194.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-29T13:57:34.547 INFO:teuthology.orchestra.run.smithi194.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 8.9 MB/s | 4.7 MB 00:00 2024-04-29T13:57:34.605 INFO:teuthology.orchestra.run.smithi194.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 7.7 MB/s | 3.7 MB 00:00 2024-04-29T13:57:34.906 INFO:teuthology.orchestra.run.smithi194.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 33 MB/s | 14 MB 00:00 2024-04-29T13:57:34.921 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-29T13:57:34.931 INFO:teuthology.orchestra.run.smithi194.stdout:(7/8): libquadmath-8.5.0-22.el8.x86_64.rpm 531 kB/s | 172 kB 00:00 2024-04-29T13:57:35.073 INFO:teuthology.orchestra.run.smithi194.stdout:(8/8): libgfortran-8.5.0-22.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-04-29T13:57:35.074 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:35.074 INFO:teuthology.orchestra.run.smithi194.stdout:Total 23 MB/s | 30 MB 00:01 2024-04-29T13:57:35.198 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:57:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-29T13:57:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-29T13:57:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-29T13:57:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-29T13:57:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-29T13:57:35.248 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:57:35.249 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:57:35.250 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:57:35.307 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:57:35.325 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:57:35.325 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:57:35.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-04-29T13:57:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-29T13:57:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:57:35.637 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:57:35.640 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:57:35.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-29T13:57:36.066 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:05 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:57:36.161 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:57:36.299 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-04-29T13:57:36.435 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-04-29T13:57:36.541 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:19 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:57:36.574 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-04-29T13:57:37.171 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-04-29T13:57:37.324 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:37.674 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:57:37.723 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:57:37.724 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:57:37.724 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:57:37.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-04-29T13:57:37.986 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:38.141 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-29T13:57:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:57:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-04-29T13:57:38.313 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-29T13:57:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-04-29T13:57:38.314 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:57:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-29T13:57:38.314 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:57:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-04-29T13:57:38.314 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-04-29T13:57:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-29T13:57:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-29T13:57:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-29T13:57:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-29T13:57:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-29T13:57:38.315 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:38.315 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:57:38.316 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-04-29T13:57:38.316 INFO:teuthology.orchestra.run.smithi040.stdout:Install 8 Packages 2024-04-29T13:57:38.316 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:38.317 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 30 M 2024-04-29T13:57:38.317 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 172 M 2024-04-29T13:57:38.317 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:57:38.635 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:07 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:57:38.768 INFO:teuthology.orchestra.run.smithi040.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 18 MB/s | 1.1 MB 00:00 2024-04-29T13:57:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-29T13:57:39.118 INFO:teuthology.orchestra.run.smithi040.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-04-29T13:57:39.160 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-29T13:57:39.185 INFO:teuthology.orchestra.run.smithi040.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 9.0 MB/s | 3.7 MB 00:00 2024-04-29T13:57:39.210 INFO:teuthology.orchestra.run.smithi040.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-29T13:57:39.286 INFO:teuthology.orchestra.run.smithi040.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.4 MB/s | 4.9 MB 00:00 2024-04-29T13:57:39.415 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-29T13:57:39.636 INFO:teuthology.orchestra.run.smithi040.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 31 MB/s | 14 MB 00:00 2024-04-29T13:57:39.661 INFO:teuthology.orchestra.run.smithi040.stdout:(7/8): libquadmath-8.5.0-22.el8.x86_64.rpm 460 kB/s | 172 kB 00:00 2024-04-29T13:57:39.787 INFO:teuthology.orchestra.run.smithi040.stdout:(8/8): libgfortran-8.5.0-22.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2024-04-29T13:57:39.787 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:39.787 INFO:teuthology.orchestra.run.smithi040.stdout:Total 20 MB/s | 30 MB 00:01 2024-04-29T13:57:40.024 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:57:40.044 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:57:40.044 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:57:40.281 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:57:40.285 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-29T13:57:40.285 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:40.286 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:57:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-29T13:57:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:57:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-29T13:57:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-29T13:57:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-29T13:57:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-29T13:57:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:57:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:Install 5 Packages 2024-04-29T13:57:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 4.0 M 2024-04-29T13:57:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 17 M 2024-04-29T13:57:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:57:40.367 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:57:40.370 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:57:40.726 INFO:teuthology.orchestra.run.smithi121.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 471 kB/s | 24 kB 00:00 2024-04-29T13:57:40.794 INFO:teuthology.orchestra.run.smithi121.stdout:(2/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 32 MB/s | 3.7 MB 00:00 2024-04-29T13:57:40.894 INFO:teuthology.orchestra.run.smithi121.stdout:(3/5): python3-jwt-1.6.1-2.el8.noarch.rpm 195 kB/s | 43 kB 00:00 2024-04-29T13:57:40.914 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:57:40.927 INFO:teuthology.orchestra.run.smithi121.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 168 kB/s | 34 kB 00:00 2024-04-29T13:57:40.969 INFO:teuthology.orchestra.run.smithi121.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-04-29T13:57:40.971 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:40.972 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.8 MB/s | 4.0 MB 00:00 2024-04-29T13:57:40.981 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:57:40.991 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:57:40.991 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:57:41.062 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-04-29T13:57:41.161 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-29T13:57:41.163 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:57:41.166 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:57:41.185 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-04-29T13:57:41.303 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-04-29T13:57:41.418 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:57:41.611 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-29T13:57:41.744 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-29T13:57:41.856 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-29T13:57:41.921 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-04-29T13:57:41.965 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-29T13:57:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:42.087 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-29T13:57:42.284 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-29T13:57:42.284 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-29T13:57:42.284 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-29T13:57:42.284 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:42.284 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-29T13:57:42.285 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-29T13:57:42.285 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-29T13:57:42.285 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-04-29T13:57:42.702 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-29T13:57:42.724 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:42.725 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:42.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-04-29T13:57:42.853 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-29T13:57:42.964 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-29T13:57:43.229 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-29T13:57:43.499 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-29T13:57:43.499 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-29T13:57:43.499 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-29T13:57:43.499 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-29T13:57:43.499 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-29T13:57:43.567 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:27 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:43.585 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-29T13:57:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-29T13:57:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:43.889 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:57:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-29T13:57:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-29T13:57:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-29T13:57:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-29T13:57:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-29T13:57:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:43.890 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:57:43.927 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-29T13:57:44.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-29T13:57:44.161 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-29T13:57:44.751 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:13 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:57:45.241 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:45.242 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:45.242 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-29T13:57:45.242 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:45.242 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-29T13:57:45.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:45.244 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:57:45.244 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:45.244 INFO:teuthology.orchestra.run.smithi194.stdout:Install 11 Packages 2024-04-29T13:57:45.244 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:45.244 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.9 M 2024-04-29T13:57:45.244 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 24 M 2024-04-29T13:57:45.244 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:57:45.486 INFO:teuthology.orchestra.run.smithi194.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 184 kB/s | 15 kB 00:00 2024-04-29T13:57:45.512 INFO:teuthology.orchestra.run.smithi194.stdout:(2/11): python3-pyasn1-modules-0.3.7-6.el8.noar 1.0 MB/s | 110 kB 00:00 2024-04-29T13:57:45.538 INFO:teuthology.orchestra.run.smithi194.stdout:(3/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 945 kB/s | 126 kB 00:00 2024-04-29T13:57:45.688 INFO:teuthology.orchestra.run.smithi194.stdout:(4/11): python3-cachetools-3.1.1-4.el8.noarch.r 220 kB/s | 33 kB 00:00 2024-04-29T13:57:45.730 INFO:teuthology.orchestra.run.smithi194.stdout:(5/11): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2024-04-29T13:57:45.755 INFO:teuthology.orchestra.run.smithi194.stdout:(6/11): python3-certifi-2018.10.15-7.el8.noarch 243 kB/s | 16 kB 00:00 2024-04-29T13:57:45.847 INFO:teuthology.orchestra.run.smithi194.stdout:(7/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 430 kB/s | 155 kB 00:00 2024-04-29T13:57:45.922 INFO:teuthology.orchestra.run.smithi194.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 423 kB/s | 81 kB 00:00 2024-04-29T13:57:46.005 INFO:teuthology.orchestra.run.smithi194.stdout:(9/11): python3-websocket-client-0.56.0-5.el8.n 734 kB/s | 61 kB 00:00 2024-04-29T13:57:46.031 INFO:teuthology.orchestra.run.smithi194.stdout:(10/11): python3-rsa-4.9-2.el8.noarch.rpm 349 kB/s | 64 kB 00:00 2024-04-29T13:57:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-29T13:57:46.064 INFO:teuthology.orchestra.run.smithi194.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 3.8 MB/s | 1.2 MB 00:00 2024-04-29T13:57:46.065 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:46.065 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.3 MB/s | 1.9 MB 00:00 2024-04-29T13:57:46.106 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:57:46.121 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:57:46.121 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:57:46.198 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:57:46.200 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:57:46.396 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:57:46.397 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-04-29T13:57:46.397 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-29T13:57:46.397 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-04-29T13:57:46.397 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:57:46.397 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-29T13:57:46.397 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.5.0-22.el8 baseos 645 k 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.5.0-22.el8 baseos 172 k 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout:Install 8 Packages 2024-04-29T13:57:46.398 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:46.399 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 30 M 2024-04-29T13:57:46.399 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 172 M 2024-04-29T13:57:46.399 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:57:46.603 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-29T13:57:46.675 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:57:46.697 INFO:teuthology.orchestra.run.smithi121.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 23 MB/s | 1.1 MB 00:00 2024-04-29T13:57:46.777 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-29T13:57:46.900 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-29T13:57:46.906 INFO:teuthology.orchestra.run.smithi121.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-04-29T13:57:46.973 INFO:teuthology.orchestra.run.smithi121.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-04-29T13:57:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-29T13:57:47.072 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-29T13:57:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-29T13:57:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-29T13:57:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-29T13:57:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-29T13:57:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-29T13:57:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-29T13:57:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-04-29T13:57:47.325 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-29T13:57:47.409 INFO:teuthology.orchestra.run.smithi121.stdout:(5/8): libgfortran-8.5.0-22.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2024-04-29T13:57:47.529 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-29T13:57:47.530 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-29T13:57:47.531 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:47.531 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:57:47.537 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-29T13:57:47.573 INFO:teuthology.orchestra.run.smithi121.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 24 MB/s | 14 MB 00:00 2024-04-29T13:57:47.598 INFO:teuthology.orchestra.run.smithi121.stdout:(7/8): libquadmath-8.5.0-22.el8.x86_64.rpm 900 kB/s | 172 kB 00:00 2024-04-29T13:57:47.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-rook 2024-04-29T13:57:47.761 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-29T13:57:47.774 INFO:teuthology.orchestra.run.smithi121.stdout:(8/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.5 MB/s | 3.7 MB 00:01 2024-04-29T13:57:47.774 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:47.774 INFO:teuthology.orchestra.run.smithi121.stdout:Total 22 MB/s | 30 MB 00:01 2024-04-29T13:57:47.968 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-29T13:57:48.005 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:57:48.023 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:57:48.023 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:57:48.243 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-29T13:57:48.326 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:57:48.329 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:57:48.418 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:31 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:57:48.862 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:57:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.5.0-22.el8.x86_64 1/8 2024-04-29T13:57:49.167 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.5.0-22.el8.x86_64 1/8 2024-04-29T13:57:49.282 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-29T13:57:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.5.0-22.el8.x86_64 2/8 2024-04-29T13:57:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.5.0-22.el8.x86_64 2/8 2024-04-29T13:57:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:50.185 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:57:50.186 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:50.186 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-29T13:57:50.186 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:50.186 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:57:50.186 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:50.187 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:57:50.188 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:57:50.188 INFO:teuthology.orchestra.run.smithi040.stdout:Install 11 Packages 2024-04-29T13:57:50.188 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:50.188 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 1.9 M 2024-04-29T13:57:50.188 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 24 M 2024-04-29T13:57:50.188 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:57:50.320 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-29T13:57:50.583 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-29T13:57:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:50.694 INFO:teuthology.orchestra.run.smithi040.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 290 kB/s | 15 kB 00:00 2024-04-29T13:57:50.719 INFO:teuthology.orchestra.run.smithi040.stdout:(2/11): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2024-04-29T13:57:50.738 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-29T13:57:50.744 INFO:teuthology.orchestra.run.smithi040.stdout:(3/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-04-29T13:57:50.839 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-29T13:57:50.928 INFO:teuthology.orchestra.run.smithi040.stdout:(4/11): python3-requests-oauthlib-1.0.0-1.el8.n 204 kB/s | 43 kB 00:00 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-29T13:57:51.038 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-29T13:57:51.039 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-29T13:57:51.039 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-29T13:57:51.039 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-29T13:57:51.305 INFO:teuthology.orchestra.run.smithi040.stdout:(5/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 254 kB/s | 155 kB 00:00 2024-04-29T13:57:51.488 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-29T13:57:51.488 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:51.488 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:57:51.488 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-29T13:57:51.488 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:51.489 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:51.522 INFO:teuthology.orchestra.run.smithi040.stdout:(6/11): python3-cachetools-3.1.1-4.el8.noarch.r 42 kB/s | 33 kB 00:00 2024-04-29T13:57:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-29T13:57:51.604 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-04-29T13:57:51.756 INFO:teuthology.orchestra.run.smithi040.stdout:(7/11): python3-certifi-2018.10.15-7.el8.noarch 20 kB/s | 16 kB 00:00 2024-04-29T13:57:51.782 INFO:teuthology.orchestra.run.smithi040.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 173 kB/s | 81 kB 00:00 2024-04-29T13:57:51.831 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-29T13:57:51.882 INFO:teuthology.orchestra.run.smithi040.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 513 kB/s | 64 kB 00:00 2024-04-29T13:57:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-29T13:57:52.328 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:36 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:52.765 INFO:teuthology.orchestra.run.smithi040.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 62 kB/s | 61 kB 00:00 2024-04-29T13:57:53.707 INFO:teuthology.orchestra.run.smithi040.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 548 kB/s | 1.2 MB 00:02 2024-04-29T13:57:53.710 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:53.711 INFO:teuthology.orchestra.run.smithi040.stdout:Total 539 kB/s | 1.9 MB 00:03 2024-04-29T13:57:53.754 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:57:53.771 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:57:53.771 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:57:53.844 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-29T13:57:53.852 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:57:53.855 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:57:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:53.951 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:57:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-04-29T13:57:53.952 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 673 k 2024-04-29T13:57:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.9 M 2024-04-29T13:57:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:57:54.062 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 10 MB/s | 590 kB 00:00 2024-04-29T13:57:54.271 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 314 kB/s | 83 kB 00:00 2024-04-29T13:57:54.271 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:54.272 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.1 MB/s | 673 kB 00:00 2024-04-29T13:57:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:57:54.303 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:57:54.303 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:57:54.355 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:57:54.389 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:57:54.395 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:57:54.412 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-29T13:57:54.598 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-29T13:57:54.601 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-29T13:57:54.610 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:57:54.769 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-29T13:57:54.869 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-29T13:57:54.924 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-29T13:57:54.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-29T13:57:54.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-29T13:57:54.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-29T13:57:54.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-29T13:57:54.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-29T13:57:54.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-29T13:57:54.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.5.0-22.el8.x86_64 7/8 2024-04-29T13:57:54.988 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-29T13:57:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.5.0-22.el8.x86_64 8/8 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.5.0-22.el8.x86_64 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.5.0-22.el8.x86_64 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-29T13:57:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-29T13:57:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-29T13:57:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:55.333 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:57:55.376 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-29T13:57:55.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-04-29T13:57:55.490 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-29T13:57:55.490 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-29T13:57:55.592 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-29T13:57:55.827 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-29T13:57:55.901 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-29T13:57:55.901 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:57:55.902 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-29T13:57:55.902 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:56.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-04-29T13:57:56.122 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-29T13:57:56.192 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:25 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:57:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-29T13:57:56.860 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:40 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:57:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-29T13:57:57.820 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-29T13:57:57.842 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-29T13:57:57.844 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout:Install 11 Packages 2024-04-29T13:57:57.845 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:57:57.846 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.9 M 2024-04-29T13:57:57.846 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 24 M 2024-04-29T13:57:57.846 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:57:57.959 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-29T13:57:58.215 INFO:teuthology.orchestra.run.smithi121.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 337 kB/s | 15 kB 00:00 2024-04-29T13:57:58.232 INFO:teuthology.orchestra.run.smithi121.stdout:(2/11): python3-pyasn1-modules-0.3.7-6.el8.noar 1.7 MB/s | 110 kB 00:00 2024-04-29T13:57:58.249 INFO:teuthology.orchestra.run.smithi121.stdout:(3/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2024-04-29T13:57:58.268 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-29T13:57:58.268 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-29T13:57:58.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-29T13:57:58.449 INFO:teuthology.orchestra.run.smithi121.stdout:(4/11): python3-requests-oauthlib-1.0.0-1.el8.n 197 kB/s | 43 kB 00:00 2024-04-29T13:57:58.512 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:57:58.512 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:58.512 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-29T13:57:58.513 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:58.514 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 90 k 2024-04-29T13:57:58.514 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 310 k 2024-04-29T13:57:58.514 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:57:58.524 INFO:teuthology.orchestra.run.smithi121.stdout:(5/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 501 kB/s | 155 kB 00:00 2024-04-29T13:57:58.599 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 1.0 MB/s | 90 kB 00:00 2024-04-29T13:57:58.600 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:58.600 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 90 kB 00:00 2024-04-29T13:57:58.600 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:57:58.611 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:57:58.611 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:57:58.661 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:57:58.664 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:57:58.741 INFO:teuthology.orchestra.run.smithi121.stdout:(6/11): python3-cachetools-3.1.1-4.el8.noarch.r 67 kB/s | 33 kB 00:00 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-29T13:57:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-29T13:57:58.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-29T13:57:58.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-29T13:57:58.760 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-29T13:57:58.760 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:57:58.760 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:57:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:57:58.842 INFO:teuthology.orchestra.run.smithi121.stdout:(7/11): python3-certifi-2018.10.15-7.el8.noarch 42 kB/s | 16 kB 00:00 2024-04-29T13:57:58.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse 2024-04-29T13:57:58.986 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:57:59.084 INFO:teuthology.orchestra.run.smithi121.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 147 kB/s | 81 kB 00:00 2024-04-29T13:57:59.110 INFO:teuthology.orchestra.run.smithi121.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 240 kB/s | 64 kB 00:00 2024-04-29T13:57:59.263 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:57:59.292 INFO:teuthology.orchestra.run.smithi121.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 294 kB/s | 61 kB 00:00 2024-04-29T13:57:59.334 INFO:teuthology.orchestra.run.smithi121.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 2.0 MB/s | 1.2 MB 00:00 2024-04-29T13:57:59.338 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:57:59.338 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2024-04-29T13:57:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:57:59.396 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:57:59.396 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:57:59.471 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:57:59.475 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:57:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:57:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:59.624 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:57:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-29T13:57:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:57:59.625 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:57:59.629 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:42 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:57:59.843 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-04-29T13:57:59.944 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:58:00.177 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-29T13:58:00.367 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-29T13:58:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-29T13:58:00.591 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:44 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:00.799 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-29T13:58:01.032 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-29T13:58:01.288 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-29T13:58:01.399 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:01.399 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:01.399 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:01.399 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:01.399 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2024-04-29T13:58:01.400 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:01.401 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 673 k 2024-04-29T13:58:01.401 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.9 M 2024-04-29T13:58:01.401 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:58:01.554 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-29T13:58:01.670 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 12 MB/s | 590 kB 00:00 2024-04-29T13:58:01.946 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 258 kB/s | 83 kB 00:00 2024-04-29T13:58:01.946 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:01.946 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.2 MB/s | 673 kB 00:00 2024-04-29T13:58:01.957 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:58:01.981 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:58:01.981 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:58:02.074 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:58:02.077 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:58:02.206 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:02.254 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:02.256 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:58:02.256 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:02.259 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:58:02.363 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-04-29T13:58:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-29T13:58:02.660 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-29T13:58:02.867 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-29T13:58:03.104 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:47 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:03.282 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-29T13:58:03.282 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-29T13:58:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-29T13:58:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-29T13:58:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:03.682 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:58:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-29T13:58:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:03.683 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:03.904 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel 2024-04-29T13:58:03.919 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-29T13:58:04.083 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-29T13:58:04.357 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-29T13:58:04.357 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-29T13:58:04.357 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-29T13:58:04.357 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-29T13:58:04.357 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-29T13:58:04.357 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-29T13:58:04.358 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-29T13:58:04.358 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-29T13:58:04.358 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-29T13:58:04.358 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-29T13:58:04.358 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-29T13:58:04.667 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:47 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:04.766 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:04.766 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:04.766 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-29T13:58:04.767 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:04.768 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 22 k 2024-04-29T13:58:04.768 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 77 k 2024-04-29T13:58:04.768 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:58:04.796 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-29T13:58:04.796 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:04.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-04-29T13:58:04.920 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-04-29T13:58:04.921 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:04.921 INFO:teuthology.orchestra.run.smithi194.stdout:Total 140 kB/s | 22 kB 00:00 2024-04-29T13:58:04.921 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:58:04.925 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:58:04.925 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:58:04.927 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:58:04.930 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:58:04.978 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:58:05.151 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:05.418 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:05.644 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:34 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:05.766 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:05.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:05.766 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:58:05.766 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:05.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:05.766 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:05.878 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-04-29T13:58:06.441 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:06.441 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:06.441 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:06.441 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:58:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-29T13:58:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:06.442 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:58:06.442 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:06.442 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-29T13:58:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:06.442 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 90 k 2024-04-29T13:58:06.442 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 310 k 2024-04-29T13:58:06.443 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:58:06.514 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 1.2 MB/s | 90 kB 00:00 2024-04-29T13:58:06.515 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:06.515 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.2 MB/s | 90 kB 00:00 2024-04-29T13:58:06.515 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:58:06.526 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:58:06.526 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:58:06.579 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:58:06.582 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:58:06.611 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:50 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:06.693 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:58:06.882 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:07.181 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:07.306 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:07.307 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:07.307 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:07.307 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:07.307 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:58:07.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-29T13:58:07.307 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-29T13:58:07.307 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-29T13:58:07.308 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:07.308 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:58:07.308 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:07.308 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-04-29T13:58:07.308 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:07.308 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 673 k 2024-04-29T13:58:07.309 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.9 M 2024-04-29T13:58:07.309 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:58:07.566 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 9.4 MB/s | 590 kB 00:00 2024-04-29T13:58:07.794 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:07.794 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:07.795 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:58:07.795 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:07.795 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:07.795 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:07.801 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 281 kB/s | 83 kB 00:00 2024-04-29T13:58:07.804 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:07.804 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.3 MB/s | 673 kB 00:00 2024-04-29T13:58:07.815 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:58:07.837 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:58:07.837 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:58:07.923 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:58:07.926 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:58:08.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2 2024-04-29T13:58:08.206 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:08.256 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:08.257 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:58:08.257 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:08.314 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:58:08.365 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-04-29T13:58:08.555 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-29T13:58:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-29T13:58:08.804 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:51 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:09.097 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:53 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-29T13:58:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-29T13:58:09.551 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-29T13:58:09.551 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:09.551 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:58:09.551 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-29T13:58:09.552 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:09.552 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:09.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-04-29T13:58:10.512 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:10.515 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:39 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:10.564 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:10.566 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:58:10.566 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:10.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs-devel 2024-04-29T13:58:10.706 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:10.867 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:10.868 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:58:10.868 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:11.437 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:54 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:11.499 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-04-29T13:58:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 90 k 2024-04-29T13:58:12.172 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 310 k 2024-04-29T13:58:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:58:12.212 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 2.2 MB/s | 90 kB 00:00 2024-04-29T13:58:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.1 MB/s | 90 kB 00:00 2024-04-29T13:58:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:58:12.224 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:58:12.224 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:58:12.231 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:56 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:12.273 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:58:12.277 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:58:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:58:12.596 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:13.208 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:58:13.209 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:13.209 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-29T13:58:13.209 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:13.209 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 22 k 2024-04-29T13:58:13.209 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 77 k 2024-04-29T13:58:13.209 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:58:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:58:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:13.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-04-29T13:58:13.462 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 85 kB/s | 22 kB 00:00 2024-04-29T13:58:13.463 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:13.463 INFO:teuthology.orchestra.run.smithi040.stdout:Total 85 kB/s | 22 kB 00:00 2024-04-29T13:58:13.464 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:58:13.467 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:58:13.467 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:58:13.470 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:58:13.473 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:58:13.873 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:13.873 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-29T13:58:13.874 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:13.875 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 73 k 2024-04-29T13:58:13.875 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 188 k 2024-04-29T13:58:13.875 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:58:13.942 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.1 MB/s | 73 kB 00:00 2024-04-29T13:58:13.943 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:13.943 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 73 kB 00:00 2024-04-29T13:58:13.943 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:58:13.953 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:58:13.953 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:58:14.003 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:58:14.006 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:58:14.030 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:58:14.112 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:58:14.176 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:43 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:14.198 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:14.316 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:14.482 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:14.605 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:14.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:14.877 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:14.878 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:58:14.878 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:14.878 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:14.878 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:14.995 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:14.996 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:14.996 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:58:14.996 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:14.996 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:14.996 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:15.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2 2024-04-29T13:58:15.208 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-04-29T13:58:15.769 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:15.776 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:58 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:15.817 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:15.818 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:58:15.818 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:15.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-04-29T13:58:15.945 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:59 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:16.651 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:45 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:17.490 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:17.542 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:17.543 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:58:17.544 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:17.598 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:17.599 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.3 M 2024-04-29T13:58:17.600 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 8.4 M 2024-04-29T13:58:17.600 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:58:17.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1 2024-04-29T13:58:17.686 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 27 MB/s | 2.3 MB 00:00 2024-04-29T13:58:17.686 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:17.687 INFO:teuthology.orchestra.run.smithi194.stdout:Total 27 MB/s | 2.3 MB 00:00 2024-04-29T13:58:17.687 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:58:17.699 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:58:17.699 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:58:17.757 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:58:17.760 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:58:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:58:18.225 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:18.302 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:18.302 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-29T13:58:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 22 k 2024-04-29T13:58:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 77 k 2024-04-29T13:58:18.304 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:58:18.350 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 466 kB/s | 22 kB 00:00 2024-04-29T13:58:18.351 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:18.351 INFO:teuthology.orchestra.run.smithi121.stdout:Total 453 kB/s | 22 kB 00:00 2024-04-29T13:58:18.351 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:58:18.355 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:58:18.355 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:58:18.357 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:58:18.360 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:58:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:58:18.413 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:01 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:19.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:19.004 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:19.005 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:58:19.005 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:19.005 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:19.005 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:19.227 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-04-29T13:58:19.227 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:19.227 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:19.227 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:58:19.228 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:19.228 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:19.228 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:19.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-04-29T13:58:19.970 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:03 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:20.078 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:49 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:20.130 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:20.182 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:20.183 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:58:20.183 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:20.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse 2024-04-29T13:58:21.053 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:04 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:21.626 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:21.627 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:21.627 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:21.627 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:21.627 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-29T13:58:21.627 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-29T13:58:21.627 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:21.627 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-29T13:58:21.628 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-29T13:58:21.628 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-29T13:58:21.628 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:21.628 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 142 k 2024-04-29T13:58:21.628 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 422 k 2024-04-29T13:58:21.628 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-29T13:58:21.673 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:21.697 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 2.0 MB/s | 142 kB 00:00 2024-04-29T13:58:21.697 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:21.697 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.0 MB/s | 142 kB 00:00 2024-04-29T13:58:21.698 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-29T13:58:21.708 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-29T13:58:21.709 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-29T13:58:21.723 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:21.724 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:58:21.724 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:21.769 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-29T13:58:21.772 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-29T13:58:21.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-04-29T13:58:21.884 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-29T13:58:22.048 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:22.509 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:22.548 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:51 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:22.837 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:58:22.838 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:22.839 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-29T13:58:22.839 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:22.839 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 73 k 2024-04-29T13:58:22.839 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 188 k 2024-04-29T13:58:22.839 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:58:22.895 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.3 MB/s | 73 kB 00:00 2024-04-29T13:58:22.895 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:22.895 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.3 MB/s | 73 kB 00:00 2024-04-29T13:58:22.896 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:58:22.907 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:58:22.908 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:58:22.931 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:22.932 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:22.932 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-29T13:58:22.932 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:22.932 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T13:58:22.932 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:22.960 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:58:22.963 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:58:23.070 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:58:23.171 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python-rados 2024-04-29T13:58:23.329 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:23.607 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:23.912 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:07 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:24.016 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:24.016 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:24.016 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:58:24.016 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:24.016 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:24.017 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:24.138 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:24.186 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:24.187 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:58:24.188 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:24.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror 2024-04-29T13:58:24.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-04-29T13:58:25.021 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:08 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:25.096 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:54 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:25.517 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:25.567 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:25.568 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:58:25.568 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:25.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python-rgw 2024-04-29T13:58:26.405 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:10 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:26.757 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:26.757 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:26.757 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 73 k 2024-04-29T13:58:26.758 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 188 k 2024-04-29T13:58:26.759 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:58:26.793 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:26.794 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-29T13:58:26.795 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:26.796 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.3 M 2024-04-29T13:58:26.796 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 8.4 M 2024-04-29T13:58:26.796 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:58:26.797 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.8 MB/s | 73 kB 00:00 2024-04-29T13:58:26.798 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:26.798 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.7 MB/s | 73 kB 00:00 2024-04-29T13:58:26.798 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:58:26.809 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:58:26.809 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:58:26.858 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:58:26.861 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:58:26.865 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 33 MB/s | 2.3 MB 00:00 2024-04-29T13:58:26.866 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:26.866 INFO:teuthology.orchestra.run.smithi040.stdout:Total 33 MB/s | 2.3 MB 00:00 2024-04-29T13:58:26.866 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:58:26.879 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:58:26.879 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:58:26.940 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:58:26.943 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:58:26.961 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:58:27.177 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:27.269 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:58:27.455 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:58:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:27.886 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:28.015 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:28.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-04-29T13:58:28.066 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:28.066 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:58:28.066 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:28.171 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python-rbd 2024-04-29T13:58:28.269 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:28.269 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:28.269 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:58:28.269 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:28.270 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:28.270 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:28.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd 2024-04-29T13:58:28.807 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:57 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:28.899 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:12 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:29.257 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:12 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:30.452 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:30.452 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-29T13:58:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:30.454 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.3 M 2024-04-29T13:58:30.454 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 8.4 M 2024-04-29T13:58:30.454 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:58:30.492 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 33 MB/s | 2.3 MB 00:00 2024-04-29T13:58:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:Total 32 MB/s | 2.3 MB 00:00 2024-04-29T13:58:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:58:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:58:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:58:30.542 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:30.544 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:58:30.544 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:58:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:58:30.641 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python-cephfs 2024-04-29T13:58:30.906 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:58:31.042 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-29T13:58:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:31.044 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 142 k 2024-04-29T13:58:31.044 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 422 k 2024-04-29T13:58:31.044 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-29T13:58:31.087 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 3.2 MB/s | 142 kB 00:00 2024-04-29T13:58:31.088 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:31.088 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.1 MB/s | 142 kB 00:00 2024-04-29T13:58:31.088 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-29T13:58:31.100 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-29T13:58:31.101 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-29T13:58:31.119 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:31.163 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-29T13:58:31.167 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-29T13:58:31.284 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-29T13:58:31.373 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:15 ago on Mon 29 Apr 2024 01:56:16 PM UTC. 2024-04-29T13:58:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:31.786 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:31.896 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:58:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:31.896 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:32.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-04-29T13:58:32.164 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:32.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:32.165 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-29T13:58:32.165 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:32.165 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:58:32.165 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:32.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python-rados 2024-04-29T13:58:32.845 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:01 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:32.973 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:33.023 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-29T13:58:33.024 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-29T13:58:33.024 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-29T13:58:33.129 DEBUG:teuthology.parallel:result is None 2024-04-29T13:58:33.132 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:16 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:34.465 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-29T13:58:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:34.467 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 142 k 2024-04-29T13:58:34.467 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 422 k 2024-04-29T13:58:34.467 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-29T13:58:34.512 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 3.1 MB/s | 142 kB 00:00 2024-04-29T13:58:34.513 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:58:34.513 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.0 MB/s | 142 kB 00:00 2024-04-29T13:58:34.513 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-29T13:58:34.524 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-29T13:58:34.524 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-29T13:58:34.583 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-29T13:58:34.586 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-29T13:58:34.710 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-29T13:58:34.851 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:34.904 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:34.905 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:58:34.906 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:34.908 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:35.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python-rgw 2024-04-29T13:58:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:35.563 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-29T13:58:35.563 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:35.563 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-29T13:58:35.563 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-29T13:58:35.563 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-29T13:58:35.563 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:35.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python-rados 2024-04-29T13:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:18 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:36.472 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:05 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:37.488 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:37.541 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:37.542 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:58:37.542 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:37.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python-rbd 2024-04-29T13:58:38.084 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:38.134 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:38.135 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:58:38.135 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:38.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python-rgw 2024-04-29T13:58:38.404 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:21 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:38.961 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:07 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:40.124 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:40.178 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:40.179 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:58:40.179 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:40.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python-cephfs 2024-04-29T13:58:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:58:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:40.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python-rbd 2024-04-29T13:58:41.060 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:24 ago on Mon 29 Apr 2024 01:56:17 PM UTC. 2024-04-29T13:58:41.429 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:10 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:42.775 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:42.829 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-29T13:58:42.830 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-29T13:58:42.830 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-29T13:58:42.937 DEBUG:teuthology.parallel:result is None 2024-04-29T13:58:43.046 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:43.096 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:43.097 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:58:43.097 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:43.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python-cephfs 2024-04-29T13:58:43.918 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:12 ago on Mon 29 Apr 2024 01:56:31 PM UTC. 2024-04-29T13:58:45.525 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-29T13:58:45.576 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-29T13:58:45.578 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-29T13:58:45.578 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-29T13:58:45.680 DEBUG:teuthology.parallel:result is None 2024-04-29T13:58:45.680 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:58:45.681 INFO:teuthology.packaging:ref: None 2024-04-29T13:58:45.681 INFO:teuthology.packaging:tag: None 2024-04-29T13:58:45.681 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:58:45.681 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:58:45.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-29T13:58:45.869 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-29T13:58:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:14.2.22-0.el8 2024-04-29T13:58:45.897 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-29T13:58:45.897 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-29T13:58:45.899 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:58:45.899 INFO:teuthology.packaging:ref: None 2024-04-29T13:58:45.899 INFO:teuthology.packaging:tag: None 2024-04-29T13:58:45.899 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:58:45.899 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:58:45.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-29T13:58:46.165 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-29T13:58:46.188 INFO:teuthology.orchestra.run.smithi121.stdout:14.2.22-0.el8 2024-04-29T13:58:46.188 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-29T13:58:46.188 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-29T13:58:46.190 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-29T13:58:46.190 INFO:teuthology.packaging:ref: None 2024-04-29T13:58:46.190 INFO:teuthology.packaging:tag: None 2024-04-29T13:58:46.190 INFO:teuthology.packaging:branch: nautilus 2024-04-29T13:58:46.190 INFO:teuthology.packaging:sha1: 7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:58:46.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-29T13:58:46.456 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-29T13:58:46.483 INFO:teuthology.orchestra.run.smithi194.stdout:14.2.22-0.el8 2024-04-29T13:58:46.483 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-29T13:58:46.484 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-29T13:58:46.485 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-29T13:58:46.486 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T13:58:46.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-29T13:58:46.519 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T13:58:46.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-29T13:58:46.547 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T13:58:46.547 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-29T13:58:46.579 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-29T13:58:46.579 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T13:58:46.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-04-29T13:58:46.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-29T13:58:46.686 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T13:58:46.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-04-29T13:58:46.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-29T13:58:46.780 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T13:58:46.780 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-04-29T13:58:46.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-29T13:58:46.881 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-29T13:58:46.882 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T13:58:46.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-29T13:58:46.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-29T13:58:46.987 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T13:58:46.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-29T13:58:47.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-29T13:58:47.080 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T13:58:47.081 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-29T13:58:47.111 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-29T13:58:47.182 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-29T13:58:47.182 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T13:58:47.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-04-29T13:58:47.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-29T13:58:47.287 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T13:58:47.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-04-29T13:58:47.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-29T13:58:47.382 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T13:58:47.382 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-04-29T13:58:47.412 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-29T13:58:47.482 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-29T13:58:47.626 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '7fe685c922bae3b52755209e35feddd972749cff'} 2024-04-29T13:58:47.626 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:58:47.627 INFO:tasks.cephadm:Cluster fsid is 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T13:58:47.627 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-29T13:58:47.627 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.40:6789]', 'mon.b': '[v1:172.21.15.121:6789]', 'mon.c': '[v1:172.21.15.194:6789]'} 2024-04-29T13:58:47.627 INFO:tasks.cephadm:First mon is mon.a on smithi040 2024-04-29T13:58:47.627 INFO:tasks.cephadm:First mgr is y 2024-04-29T13:58:47.627 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-29T13:58:47.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-04-29T13:58:47.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-04-29T13:58:47.689 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-04-29T13:58:47.718 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-29T13:58:47.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:58:47.939 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '7fe685c922bae3b52755209e35feddd972749cff', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78917/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3300-g7fe685c9', 'node_name': '172.21.2.4+braggi04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3300.g7fe685c9'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/7fe685c922bae3b52755209e35feddd972749cff/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 11:18:52.741629', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pereman2-buffer-space', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pereman2-buffer-space/7fe685c922bae3b52755209e35feddd972749cff/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-29T13:58:48.083 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pereman2-buffer-space, sha1 7fe685c922bae3b52755209e35feddd972749cff from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=7fe685c922bae3b52755209e35feddd972749cff 2024-04-29T13:58:48.084 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pereman2-buffer-space/7fe685c922bae3b52755209e35feddd972749cff/centos/8/x86_64/flavors/default/cephadm 2024-04-29T13:58:48.085 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pereman2-buffer-space/7fe685c922bae3b52755209e35feddd972749cff/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:49.033 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 29 13:58 /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:49.033 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pereman2-buffer-space/7fe685c922bae3b52755209e35feddd972749cff/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:49.609 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 29 13:58 /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:49.609 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pereman2-buffer-space/7fe685c922bae3b52755209e35feddd972749cff/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:50.074 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 29 13:58 /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:50.074 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:50.098 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:50.120 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-29T13:58:50.155 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff on all hosts... 2024-04-29T13:58:50.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff pull 2024-04-29T13:58:50.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff pull 2024-04-29T13:58:50.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff pull 2024-04-29T13:58:51.265 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff... 2024-04-29T13:58:51.284 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff... 2024-04-29T13:58:51.388 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff... 2024-04-29T13:59:25.494 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T13:59:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-3300-g7fe685c9 (7fe685c922bae3b52755209e35feddd972749cff) squid (dev)", 2024-04-29T13:59:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "939d983ef3873f3c7503aae562cf168464889cb9b1a3b51873cd37cce30d8aae", 2024-04-29T13:59:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-04-29T13:59:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cda992d224e2e50c726e715240eee84b5e97f0c57be048f859f062f583c4d452" 2024-04-29T13:59:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T13:59:25.495 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T13:59:44.786 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T13:59:44.786 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 19.0.0-3300-g7fe685c9 (7fe685c922bae3b52755209e35feddd972749cff) squid (dev)", 2024-04-29T13:59:44.786 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "939d983ef3873f3c7503aae562cf168464889cb9b1a3b51873cd37cce30d8aae", 2024-04-29T13:59:44.787 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-04-29T13:59:44.787 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cda992d224e2e50c726e715240eee84b5e97f0c57be048f859f062f583c4d452" 2024-04-29T13:59:44.787 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T13:59:44.787 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T13:59:45.887 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T13:59:45.887 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-3300-g7fe685c9 (7fe685c922bae3b52755209e35feddd972749cff) squid (dev)", 2024-04-29T13:59:45.887 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "939d983ef3873f3c7503aae562cf168464889cb9b1a3b51873cd37cce30d8aae", 2024-04-29T13:59:45.887 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2024-04-29T13:59:45.887 INFO:teuthology.orchestra.run.smithi121.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cda992d224e2e50c726e715240eee84b5e97f0c57be048f859f062f583c4d452" 2024-04-29T13:59:45.887 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T13:59:45.887 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T13:59:45.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-04-29T13:59:45.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-04-29T13:59:45.981 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-04-29T13:59:46.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-04-29T13:59:46.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-04-29T13:59:46.077 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-04-29T13:59:46.113 INFO:tasks.cephadm:Writing seed config... 2024-04-29T13:59:46.129 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-04-29T13:59:46.129 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-29T13:59:46.129 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-29T13:59:46.129 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [global] ms tcp read timeout = 5 2024-04-29T13:59:46.130 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 2 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd backoff on degraded = True 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd map cache size = 1 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd max backfills = 6 2024-04-29T13:59:46.131 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd scrub during recovery = False 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-29T13:59:46.132 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-29T13:59:46.133 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T13:59:46.133 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-29T13:59:46.155 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 998f2e22-0630-11ef-bc93-c7b262605968 mon client directed command retry = 5 mon election default strategy = 1 ms bind msgr2 = False ms inject socket failures = 2500 ms tcp read timeout = 5 ms type = async osd_pool_default_min_size = 2 osd_pool_default_size = 2 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on degraded = True osd backoff on peering = True osd blocked scrub grace period = 3600 osd map cache size = 1 osd max backfills = 6 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub during recovery = False osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-29T13:59:46.156 DEBUG:teuthology.orchestra.run.smithi040:mon.a> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a.service 2024-04-29T13:59:46.199 DEBUG:teuthology.orchestra.run.smithi040:mgr.y> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.y.service 2024-04-29T13:59:46.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T13:59:46.234 INFO:tasks.cephadm:Bootstrapping... 2024-04-29T13:59:46.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff -v bootstrap --fsid 998f2e22-0630-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.40:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-29T13:59:46.263 INFO:journalctl@ceph.mgr.y.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T13:59:46.455 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-29T13:59:46.456 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff', '-v', 'bootstrap', '--fsid', '998f2e22-0630-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.40:6789]', '--skip-admin-label'] 2024-04-29T13:59:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-29T13:59:46.456 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying podman|docker is present... 2024-04-29T13:59:46.499 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-29T13:59:46.499 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying lvm2 is present... 2024-04-29T13:59:46.499 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying time synchronization is in place... 2024-04-29T13:59:46.506 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-29T13:59:46.506 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-29T13:59:46.512 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-29T13:59:46.512 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-29T13:59:46.518 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-04-29T13:59:46.524 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-04-29T13:59:46.524 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-04-29T13:59:46.524 INFO:teuthology.orchestra.run.smithi040.stdout:Repeating the final host check... 2024-04-29T13:59:46.563 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-29T13:59:46.563 INFO:teuthology.orchestra.run.smithi040.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-29T13:59:46.563 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl is present 2024-04-29T13:59:46.563 INFO:teuthology.orchestra.run.smithi040.stdout:lvcreate is present 2024-04-29T13:59:46.569 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-29T13:59:46.569 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-29T13:59:46.574 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-29T13:59:46.575 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-29T13:59:46.580 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-04-29T13:59:46.587 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-04-29T13:59:46.587 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-04-29T13:59:46.587 INFO:teuthology.orchestra.run.smithi040.stdout:Host looks OK 2024-04-29T13:59:46.587 INFO:teuthology.orchestra.run.smithi040.stdout:Cluster fsid: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T13:59:46.587 INFO:teuthology.orchestra.run.smithi040.stdout:Acquiring lock 140184123199104 on /run/cephadm/998f2e22-0630-11ef-bc93-c7b262605968.lock 2024-04-29T13:59:46.587 INFO:teuthology.orchestra.run.smithi040.stdout:Lock 140184123199104 acquired on /run/cephadm/998f2e22-0630-11ef-bc93-c7b262605968.lock 2024-04-29T13:59:46.588 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 6789 ... 2024-04-29T13:59:46.588 INFO:teuthology.orchestra.run.smithi040.stdout:Base mon IP(s) is [172.21.15.40:6789], mon addrv is [v1:172.21.15.40:6789] 2024-04-29T13:59:46.592 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.40 metric 100 2024-04-29T13:59:46.592 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.40 metric 100 2024-04-29T13:59:46.595 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-29T13:59:46.595 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-29T13:59:46.595 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-04-29T13:59:46.598 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-29T13:59:46.598 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-29T13:59:46.598 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-29T13:59:46.598 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-29T13:59:46.598 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-04-29T13:59:46.598 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-29T13:59:46.599 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2024-04-29T13:59:46.599 INFO:teuthology.orchestra.run.smithi040.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-04-29T13:59:46.599 INFO:teuthology.orchestra.run.smithi040.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-29T13:59:46.600 INFO:teuthology.orchestra.run.smithi040.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff... 2024-04-29T13:59:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 939d983ef3873f3c7503aae562cf168464889cb9b1a3b51873cd37cce30d8aae 2024-04-29T13:59:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff... 2024-04-29T13:59:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Getting image source signatures 2024-04-29T13:59:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:60dffbfa91ecba7783805138bca055a3cb9fe95c01757b9d9e14fee89b50a55f 2024-04-29T13:59:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-29T13:59:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying config sha256:939d983ef3873f3c7503aae562cf168464889cb9b1a3b51873cd37cce30d8aae 2024-04-29T13:59:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-29T13:59:53.718 INFO:teuthology.orchestra.run.smithi040.stdout:ceph: stdout ceph version 19.0.0-3300-g7fe685c9 (7fe685c922bae3b52755209e35feddd972749cff) squid (dev) 2024-04-29T13:59:53.718 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph version: ceph version 19.0.0-3300-g7fe685c9 (7fe685c922bae3b52755209e35feddd972749cff) squid (dev) 2024-04-29T13:59:53.719 INFO:teuthology.orchestra.run.smithi040.stdout:Extracting ceph user uid/gid from container image... 2024-04-29T13:59:54.898 INFO:teuthology.orchestra.run.smithi040.stdout:stat: stdout 167 167 2024-04-29T13:59:54.898 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial keys... 2024-04-29T13:59:55.981 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQDbpy9m0/a6EBAAVwRKFEUkGk2UuLQIiLb35Q== 2024-04-29T13:59:57.071 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQDcpy9mYOOEGhAAe1V+JE3Y0G3D8btw9ENcNw== 2024-04-29T13:59:58.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQDdpy9md2h0IRAAXeY+thBM20PCBkOL6iKAyQ== 2024-04-29T13:59:58.150 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial monmap... 2024-04-29T13:59:59.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-29T13:59:59.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-29T13:59:59.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T13:59:59.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-29T13:59:59.297 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool for a [v1:172.21.15.40:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-29T13:59:59.297 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-04-29T13:59:59.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: set fsid to 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T13:59:59.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-29T13:59:59.297 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T13:59:59.297 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mon... 2024-04-29T14:00:00.362 INFO:teuthology.orchestra.run.smithi040.stdout:create mon.a on 2024-04-29T14:00:00.500 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-29T14:00:00.596 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-29T14:00:00.731 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-998f2e22-0630-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-998f2e22-0630-11ef-bc93-c7b262605968.target. 2024-04-29T14:00:00.732 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-998f2e22-0630-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-998f2e22-0630-11ef-bc93-c7b262605968.target. 2024-04-29T14:00:00.930 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a 2024-04-29T14:00:00.930 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-29T14:00:01.025 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-998f2e22-0630-11ef-bc93-c7b262605968.target.wants/ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-998f2e22-0630-11ef-bc93-c7b262605968@.service. 2024-04-29T14:00:01.605 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-29T14:00:01.605 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-29T14:00:01.613 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-29T14:00:01.613 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-29T14:00:01.614 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-29T14:00:01.614 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-04-29T14:00:01.614 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon to start... 2024-04-29T14:00:01.614 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon... 2024-04-29T14:00:01.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:01 smithi040 bash[39054]: 5c34e536a25d360c791c514081e5368a9ce382453aa3bbd25cbc42469300ada0 2024-04-29T14:00:01.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:01 smithi040 systemd[1]: Started Ceph mon.a for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:00:02.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout cluster: 2024-04-29T14:00:02.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout id: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:02.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-29T14:00:02.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout services: 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.616013s) 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout data: 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pgs: 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:mon is available 2024-04-29T14:00:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:Assimilating anything we can from ceph.conf... 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: monmap epoch 1 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: last_changed 2024-04-29T13:59:58.602057+0000 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: min_mon_release 19 (squid) 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: election_strategy: 1 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: fsmap 2024-04-29T14:00:02.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: osdmap e1: 0 total, 0 up, 0 in 2024-04-29T14:00:02.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: mgrmap e1: no daemons active 2024-04-29T14:00:02.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:02 smithi040 ceph-mon[39092]: from='client.? v1:172.21.15.40:0/3875165843' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-29T14:00:03.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:03 smithi040 ceph-mon[39092]: from='client.? v1:172.21.15.40:0/1983338015' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-29T14:00:03.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:03 smithi040 ceph-mon[39092]: from='client.? v1:172.21.15.40:0/1983338015' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-29T14:00:03.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:04.045 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-04-29T14:00:04.045 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:04.045 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.40:6789] 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-04-29T14:00:04.046 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-29T14:00:04.049 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-29T14:00:04.049 INFO:teuthology.orchestra.run.smithi040.stdout:Generating new minimal ceph.conf... 2024-04-29T14:00:04.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:04 smithi040 ceph-mon[39092]: from='client.? v1:172.21.15.40:0/3188079048' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:00:05.134 INFO:teuthology.orchestra.run.smithi040.stdout:Restarting the monitor... 2024-04-29T14:00:05.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:05 smithi040 systemd[1]: Stopping Ceph mon.a for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:00:05.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:05 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39069]: 2024-04-29T14:00:05.364+0000 7fe0762eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T14:00:05.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:05 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39069]: 2024-04-29T14:00:05.364+0000 7fe0762eb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-29T14:00:06.080 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:05 smithi040 bash[39419]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a 2024-04-29T14:00:06.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 systemd[1]: ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a.service: Succeeded. 2024-04-29T14:00:06.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 systemd[1]: Stopped Ceph mon.a for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:00:06.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 systemd[1]: Starting Ceph mon.a for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:00:06.808 INFO:teuthology.orchestra.run.smithi040.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-29T14:00:07.015 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: set uid:gid to 167:167 (ceph:ceph) 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: ceph version 19.0.0-3300-g7fe685c9 (7fe685c922bae3b52755209e35feddd972749cff) squid (dev), process ceph-mon, pid 7 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: pidfile_write: ignore empty --pid-file 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: load: jerasure load: lrc 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: RocksDB version: 7.9.2 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Git sha 0 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Compile date 2024-04-26 09:58:26 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: DB SUMMARY 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: DB Session ID: 3KI51JRL3PK4YVNI0G6H 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: CURRENT file: CURRENT 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: IDENTITY file: IDENTITY 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-29T14:00:07.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98195 ; 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.error_if_exists: 0 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.create_if_missing: 0 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.paranoid_checks: 1 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.env: 0x555ed65efcc0 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.fs: PosixFileSystem 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.info_log: 0x555ed8e55dc0 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_file_opening_threads: 16 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.statistics: (nil) 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.use_fsync: 0 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_log_file_size: 0 2024-04-29T14:00:07.017 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.keep_log_file_num: 1000 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.recycle_log_file_num: 0 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.allow_fallocate: 1 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.allow_mmap_reads: 0 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.allow_mmap_writes: 0 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.use_direct_reads: 0 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.create_missing_column_families: 0 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.db_log_dir: 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.wal_dir: 2024-04-29T14:00:07.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.advise_random_on_open: 1 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.db_write_buffer_size: 0 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.write_buffer_manager: 0x555ed8e270e0 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-29T14:00:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.rate_limiter: (nil) 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.wal_recovery_mode: 2 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.enable_thread_tracking: 0 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.enable_pipelined_write: 0 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.unordered_write: 0 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.row_cache: None 2024-04-29T14:00:07.022 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.wal_filter: None 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.allow_ingest_behind: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.two_write_queues: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.manual_wal_flush: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.wal_compression: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.atomic_flush: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.log_readahead_size: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.best_efforts_recovery: 0 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-29T14:00:07.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.allow_data_in_errors: 0 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.db_host_id: __hostname__ 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_background_jobs: 2 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_background_compactions: -1 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_subcompactions: 1 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_total_wal_size: 0 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-29T14:00:07.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_open_files: -1 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bytes_per_sync: 0 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_readahead_size: 0 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_background_flushes: -1 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Compression algorithms supported: 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kZSTD supported: 0 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kXpressCompression supported: 0 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kLZ4HCCompression supported: 1 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kZlibCompression supported: 1 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kSnappyCompression supported: 1 2024-04-29T14:00:07.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kLZ4Compression supported: 1 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: kBZip2Compression supported: 0 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.merge_operator: 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_filter: None 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_filter_factory: None 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.sst_partitioner_factory: None 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555ed8e55f80) 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks: 1 2024-04-29T14:00:07.026 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_top_level_index_and_filter: 1 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_type: 0 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_index_type: 0 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_shortening: 1 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: checksum: 4 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: no_block_cache: 0 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache: 0x555ed8e591f0 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_name: BinnedLRUCache 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_options: 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: capacity : 536870912 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_shard_bits : 4 2024-04-29T14:00:07.027 INFO:journalctl@ceph.mon.a.smithi040.stdout: strict_capacity_limit : 0 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: high_pri_pool_ratio: 0.000 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_compressed: (nil) 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: persistent_cache: (nil) 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size: 4096 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size_deviation: 10 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_restart_interval: 16 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_block_restart_interval: 1 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: metadata_block_size: 4096 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: partition_filters: 0 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: use_delta_encoding: 1 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: filter_policy: bloomfilter 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: whole_key_filtering: 1 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: verify_compression: 0 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: read_amp_bytes_per_bit: 0 2024-04-29T14:00:07.028 INFO:journalctl@ceph.mon.a.smithi040.stdout: format_version: 5 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_index_compression: 1 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_align: 0 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout: max_auto_readahead_size: 262144 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout: prepopulate_block_cache: 0 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout: initial_auto_readahead_size: 8192 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_file_reads_for_auto_readahead: 2 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.write_buffer_size: 33554432 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_write_buffer_number: 2 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression: NoCompression 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression: Disabled 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.prefix_extractor: nullptr 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.num_levels: 7 2024-04-29T14:00:07.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-29T14:00:07.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-29T14:00:07.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-29T14:00:07.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-29T14:00:07.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-29T14:00:07.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-29T14:00:07.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-29T14:00:07.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-29T14:00:07.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-29T14:00:07.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-29T14:00:07.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-29T14:00:07.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-29T14:00:07.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.level: 32767 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.strategy: 0 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.enabled: false 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.target_file_size_base: 67108864 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-29T14:00:07.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-29T14:00:07.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.arena_block_size: 1048576 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.disable_auto_compactions: 0 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-29T14:00:07.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.inplace_update_support: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.bloom_locality: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.max_successive_merges: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.paranoid_file_checks: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.force_consistency_checks: 1 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.report_bg_io_stats: 0 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.ttl: 2592000 2024-04-29T14:00:07.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.enable_blob_files: false 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.min_blob_size: 0 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.blob_file_size: 268435456 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.blob_file_starting_level: 0 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-29T14:00:07.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 87e2817f-dfa1-4629-9e72-ef7332f4882c 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399206751641, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399206753295, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93559, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 306, "table_properties": {"data_size": 91560, "index_size": 255, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13199, "raw_average_key_size": 49, "raw_value_size": 84414, "raw_average_value_size": 318, "num_data_blocks": 11, "num_entries": 265, "num_filter_entries": 265, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714399206, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "87e2817f-dfa1-4629-9e72-ef7332f4882c", "db_session_id": "3KI51JRL3PK4YVNI0G6H", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399206753405, "job": 1, "event": "recovery_finished"} 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555ed8f76000 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: DB pointer 0x555ed8f62000 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** DB Stats ** 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-29T14:00:07.037 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: L0 2/0 93.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Sum 2/0 93.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-29T14:00:07.038 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 87.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative compaction: 0.00 GB write, 19.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-29T14:00:07.039 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval compaction: 0.00 GB write, 19.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache BinnedLRUCache@0x555ed8e591f0#7 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: starting mon.a rank 0 at public addrs v1:172.21.15.40:6789/0 at bind addrs v1:172.21.15.40:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???) e1 preinit fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???).mds e1 new map 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???).mds e1 print_map 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout: e1 2024-04-29T14:00:07.040 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-29T14:00:07.041 INFO:journalctl@ceph.mon.a.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-29T14:00:07.041 INFO:journalctl@ceph.mon.a.smithi040.stdout: legacy client fscid: -1 2024-04-29T14:00:07.041 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout: No filesystems configured 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 bash[39543]: 2b5c666c20f4ed01cbf58a0865781b695fcc84fbcad4832c0673681b715f1cf9 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 systemd[1]: Started Ceph mon.a for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: monmap epoch 1 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: last_changed 2024-04-29T13:59:58.602057+0000 2024-04-29T14:00:07.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:00:07.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: min_mon_release 19 (squid) 2024-04-29T14:00:07.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: election_strategy: 1 2024-04-29T14:00:07.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:00:07.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: fsmap 2024-04-29T14:00:07.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: osdmap e1: 0 total, 0 up, 0 in 2024-04-29T14:00:07.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:06 smithi040 ceph-mon[39581]: mgrmap e1: no daemons active 2024-04-29T14:00:07.988 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-29T14:00:08.010 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-29T14:00:08.010 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mgr... 2024-04-29T14:00:08.011 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-29T14:00:08.011 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-29T14:00:08.219 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.y 2024-04-29T14:00:08.219 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-29T14:00:08.315 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-998f2e22-0630-11ef-bc93-c7b262605968.target.wants/ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-998f2e22-0630-11ef-bc93-c7b262605968@.service. 2024-04-29T14:00:08.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/613785978' entity='client.admin' 2024-04-29T14:00:08.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:08 smithi040 systemd[1]: Starting Ceph mgr.y for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:00:08.938 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-29T14:00:08.938 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-29T14:00:08.946 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-29T14:00:08.947 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-29T14:00:08.947 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-29T14:00:08.947 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-04-29T14:00:08.955 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-29T14:00:08.955 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-29T14:00:08.964 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-29T14:00:08.964 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-29T14:00:08.964 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-29T14:00:08.964 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-29T14:00:08.964 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr to start... 2024-04-29T14:00:08.964 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr... 2024-04-29T14:00:09.194 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:08 smithi040 bash[39823]: dd20515bf949d39f74e3ef64c3422c6b2d767cd61fee6e52ffa74638de348962 2024-04-29T14:00:09.194 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:08 smithi040 systemd[1]: Started Ceph mgr.y for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:00:09.194 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:09 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:09.028+0000 7fa62631e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-29T14:00:09.548 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:09 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:09.213+0000 7fa62631e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-29T14:00:09.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/155208147' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-29T14:00:09.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:09 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:09.561+0000 7fa62631e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-29T14:00:10.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:10.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:00:10.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:00:10.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-29T14:00:10.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-29T14:00:10.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-29T14:00:10.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-29T14:00:10.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:10.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-29T14:00:10.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-29T14:00:10.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-29T14:00:10.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-29T14:00:10.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-29T14:00:10.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-29T14:00:10.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-29T14:00:10.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-29T14:00:10.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-29T14:00:10.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:10.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-29T14:00:10.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:10.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-29T14:00:10.203 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:10 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:10.096+0000 7fa62631e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-29T14:00:10.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:10.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-29T14:00:01.569414+0000", 2024-04-29T14:00:10.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:10.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:10.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-29T14:00:10.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:00:10.204 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (1/15)... 2024-04-29T14:00:10.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:10 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:10.350+0000 7fa62631e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-29T14:00:11.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:10 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:10.803+0000 7fa62631e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-29T14:00:11.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:11 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:11.555+0000 7fa62631e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-29T14:00:11.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:11 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:11.644+0000 7fa62631e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-29T14:00:12.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:12.271+0000 7fa62631e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-29T14:00:12.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:12.359+0000 7fa62631e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-29T14:00:12.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:12.451+0000 7fa62631e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-29T14:00:12.962 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:12.690+0000 7fa62631e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-29T14:00:12.962 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:12.788+0000 7fa62631e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-29T14:00:12.962 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:12.899+0000 7fa62631e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-29T14:00:13.241 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:13 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:13.143+0000 7fa62631e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-29T14:00:13.242 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3619252975' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-29T14:00:13.540 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:13.543 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:00:13.543 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:00:13.543 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-29T14:00:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:13.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-29T14:00:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-29T14:00:13.547 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:13 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:13.245+0000 7fa62631e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-29T14:00:13.547 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:13 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:13.438+0000 7fa62631e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-29T14:00:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-29T14:00:01.569414+0000", 2024-04-29T14:00:13.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:13.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:13.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-29T14:00:13.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:00:13.552 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (2/15)... 2024-04-29T14:00:13.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:13 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:13.699+0000 7fa62631e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-29T14:00:14.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:14 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:14.316+0000 7fa62631e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-29T14:00:14.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:14 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:14.537+0000 7fa62631e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-29T14:00:15.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:14 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:14.741+0000 7fa62631e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:00:15.731 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:15 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:15.432+0000 7fa62631e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-29T14:00:15.731 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:15 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:15.644+0000 7fa62631e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:00:16.017 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:15 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:15.844+0000 7fa62631e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-29T14:00:16.018 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:15 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:15.965+0000 7fa62631e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-29T14:00:16.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: Activating manager daemon y 2024-04-29T14:00:16.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: mgrmap e2: y(active, starting, since 0.00313589s) 2024-04-29T14:00:16.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-29T14:00:16.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: Manager daemon y is now available 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' 2024-04-29T14:00:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-29T14:00:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' 2024-04-29T14:00:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:16 smithi040 ceph-mon[39581]: from='mgr.14100 v1:172.21.15.40:0/4241310367' entity='mgr.y' 2024-04-29T14:00:16.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:16.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:00:16.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:00:16.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-29T14:00:16.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-29T14:00:16.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-29T14:00:16.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-29T14:00:16.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:16.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-29T14:00:16.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-29T14:00:16.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-29T14:00:16.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-29T14:00:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-29T14:00:01.569414+0000", 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:00:16.781 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (3/15)... 2024-04-29T14:00:17.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4260465474' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-29T14:00:17.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:17 smithi040 ceph-mon[39581]: mgrmap e3: y(active, since 1.00915s) 2024-04-29T14:00:19.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:19 smithi040 ceph-mon[39581]: mgrmap e4: y(active, since 2s) 2024-04-29T14:00:20.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:20.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:00:20.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:00:20.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-29T14:00:20.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:20.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-29T14:00:20.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-29T14:00:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-29T14:00:01.569414+0000", 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-29T14:00:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-29T14:00:20.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-29T14:00:20.140 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:00:20.140 INFO:teuthology.orchestra.run.smithi040.stdout:mgr is available 2024-04-29T14:00:20.345 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/611148991' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-29T14:00:21.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4169992128' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-29T14:00:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-29T14:00:21.430 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-04-29T14:00:21.430 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-29T14:00:21.430 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-29T14:00:21.430 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling cephadm module... 2024-04-29T14:00:22.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2207044250' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-29T14:00:22.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:22 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: ignoring --setuser ceph since I am not root 2024-04-29T14:00:22.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:22 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: ignoring --setgroup ceph since I am not root 2024-04-29T14:00:22.616 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:22 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:22.457+0000 7f9f9bb29200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-29T14:00:22.894 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:22 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:22.643+0000 7f9f9bb29200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-29T14:00:23.146 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:22 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:22.986+0000 7f9f9bb29200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-29T14:00:23.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:23 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:23.517+0000 7f9f9bb29200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-29T14:00:23.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2207044250' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-29T14:00:23.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:23 smithi040 ceph-mon[39581]: mgrmap e5: y(active, since 6s) 2024-04-29T14:00:24.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:23 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:23.774+0000 7f9f9bb29200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-29T14:00:24.605 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2993187233' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-29T14:00:24.605 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:24 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:24.235+0000 7f9f9bb29200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-04-29T14:00:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 5... 2024-04-29T14:00:25.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:25 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:25.000+0000 7f9f9bb29200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-29T14:00:25.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:25 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:25.091+0000 7f9f9bb29200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-29T14:00:26.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:25 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:25.730+0000 7f9f9bb29200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-29T14:00:26.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:25 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:25.818+0000 7f9f9bb29200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-29T14:00:26.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:25 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:25.910+0000 7f9f9bb29200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-29T14:00:26.579 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:26 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:26.146+0000 7f9f9bb29200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-29T14:00:26.579 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:26 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:26.242+0000 7f9f9bb29200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-29T14:00:26.579 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:26 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:26.340+0000 7f9f9bb29200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-29T14:00:26.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:26 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:26.582+0000 7f9f9bb29200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-29T14:00:26.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:26 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:26.682+0000 7f9f9bb29200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-29T14:00:27.135 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:26 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:26.877+0000 7f9f9bb29200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-29T14:00:27.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:27 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:27.138+0000 7f9f9bb29200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-29T14:00:28.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:27 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:27.754+0000 7f9f9bb29200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-29T14:00:28.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:27 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:27.975+0000 7f9f9bb29200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-29T14:00:28.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:28 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:28.180+0000 7f9f9bb29200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:00:29.271 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:28 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:28.876+0000 7f9f9bb29200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-29T14:00:29.271 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:29.084+0000 7f9f9bb29200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:00:29.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:29.274+0000 7f9f9bb29200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-29T14:00:29.615 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:29.395+0000 7f9f9bb29200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: Active manager daemon y restarted 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: Activating manager daemon y 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: osdmap e2: 0 total, 0 up, 0 in 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: mgrmap e6: y(active, starting, since 0.00320739s) 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: Manager daemon y is now available 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:29 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:00:30.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:30 smithi040 ceph-mon[39581]: Found migration_current of "None". Setting to last migration. 2024-04-29T14:00:30.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:30 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:00:30.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:30 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-29T14:00:30.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:30 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-29T14:00:30.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:30 smithi040 ceph-mon[39581]: mgrmap e7: y(active, since 1.00562s) 2024-04-29T14:00:31.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:31 smithi040 ceph-mon[39581]: from='client.14124 v1:172.21.15.40:0/548451513' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-29T14:00:31.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:31 smithi040 ceph-mon[39581]: from='client.14124 v1:172.21.15.40:0/548451513' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-29T14:00:31.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:31 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:31.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:31 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:31.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:00:31.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-29T14:00:31.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-29T14:00:31.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:00:31.990 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 5 is available 2024-04-29T14:00:31.990 INFO:teuthology.orchestra.run.smithi040.stdout:Setting orchestrator backend to cephadm... 2024-04-29T14:00:32.797 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:32 smithi040 ceph-mon[39581]: from='client.14124 v1:172.21.15.40:0/548451513' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-29T14:00:32.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:32 smithi040 ceph-mon[39581]: mgrmap e8: y(active, since 3s) 2024-04-29T14:00:33.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:33 smithi040 ceph-mon[39581]: from='client.14132 v1:172.21.15.40:0/2259449683' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:00:33.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:33 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:33.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:33 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:00:33.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:33 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:00:34.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-29T14:00:34.645 INFO:teuthology.orchestra.run.smithi040.stdout:Generating ssh key... 2024-04-29T14:00:34.911 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:34 smithi040 ceph-mon[39581]: [29/Apr/2024:14:00:33] ENGINE Bus STARTING 2024-04-29T14:00:34.911 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:34 smithi040 ceph-mon[39581]: [29/Apr/2024:14:00:33] ENGINE Serving on http://172.21.15.40:8765 2024-04-29T14:00:34.911 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:34 smithi040 ceph-mon[39581]: [29/Apr/2024:14:00:33] ENGINE Serving on https://172.21.15.40:7150 2024-04-29T14:00:34.912 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:34 smithi040 ceph-mon[39581]: [29/Apr/2024:14:00:33] ENGINE Bus STARTED 2024-04-29T14:00:34.912 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:34 smithi040 ceph-mon[39581]: from='client.14134 v1:172.21.15.40:0/644799590' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: Generating public/private rsa key pair. 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: Your identification has been saved in /tmp/tmpkhh01sp_/key. 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: Your public key has been saved in /tmp/tmpkhh01sp_/key.pub. 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: The key fingerprint is: 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: SHA256:/Jc0zbKXk7wUu0tTiectDJOZSx+8RswsEtaD7iW2xqQ ceph-998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: The key's randomart image is: 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: +---[RSA 3072]----+ 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | | 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | o | 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | + o | 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | . o . %. .| 2024-04-29T14:00:36.307 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | S * &.@o.| 2024-04-29T14:00:36.308 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | B B #oO.| 2024-04-29T14:00:36.308 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | E * = ^..| 2024-04-29T14:00:36.308 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | . . = * | 2024-04-29T14:00:36.308 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: | +. | 2024-04-29T14:00:36.308 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:36 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: +----[SHA256]-----+ 2024-04-29T14:00:37.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:37 smithi040 ceph-mon[39581]: from='client.14136 v1:172.21.15.40:0/3959007641' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:00:37.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:37 smithi040 ceph-mon[39581]: Generating ssh key... 2024-04-29T14:00:37.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:37 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:37.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:37 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:40.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5JrM3OIVkTz3pORhtZtdiBSbdYGczyqGqVEUk1Cb1XTthaEPIcgrKq0tckFjDhxPhPbx4NqZbRSmfek1EnNbydxbKA3JQXGzE756hIQuD83ADaih6/LIkPRP8CTJbiLnPisSx8ZfzlCiu2AvSpYaM9dz2QJSCmUn0VPFkeepScyUre23ZTn5CohHoBUpHoFMCx7UaHYd8rFsNZrPjZavT8Blnq66sa8LqZUYz2OpmAIurxr6y6Nhlf1c7Qf+IMelwxnhFD9Z/xAnvz10tNR07PTjH+7a/CM4GkMVKeYmkYiCisx111yedt3rcOxUE7PsS0lTit/3rBKwtQfakVaj1IyI+jTVxHiM1LJ7pKGJIL6eYc+0bEdDm3eiqRgC6pyVcbFzKc9oT7TbK47MNvGIw7bcg6Dc+95XaIugOysXLBcQx51Oi1L8b0Elv6Pq8mb+b1djucZp+16HCM8ebl/kA6zKvKPqYgLYYw2pr74/ThFbuOVVn7lycjmchLribZs= ceph-998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:00:40.984 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-29T14:00:40.985 INFO:teuthology.orchestra.run.smithi040.stdout:Adding key to root@localhost authorized_keys... 2024-04-29T14:00:40.985 INFO:teuthology.orchestra.run.smithi040.stdout:Adding host smithi040... 2024-04-29T14:00:41.752 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:41 smithi040 ceph-mon[39581]: from='client.14139 v1:172.21.15.40:0/2198577567' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:00:42.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:42 smithi040 ceph-mon[39581]: from='client.14141 v1:172.21.15.40:0/15287947' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:00:42.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:42 smithi040 ceph-mon[39581]: Deploying cephadm binary to smithi040 2024-04-29T14:00:46.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Added host 'smithi040' with addr '172.21.15.40' 2024-04-29T14:00:46.209 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mon service... 2024-04-29T14:00:46.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:46 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:46.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:46 smithi040 ceph-mon[39581]: Added host smithi040 2024-04-29T14:00:46.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:46 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:00:48.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-29T14:00:48.139 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mgr service... 2024-04-29T14:00:48.338 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:47 smithi040 ceph-mon[39581]: from='client.14143 v1:172.21.15.40:0/4023344740' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:00:48.339 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:47 smithi040 ceph-mon[39581]: Saving service mon spec with placement count:5 2024-04-29T14:00:48.339 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:47 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:49.932 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-29T14:00:49.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:49 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:49.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:49 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:49.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:49 smithi040 ceph-mon[39581]: from='client.14145 v1:172.21.15.40:0/4265640626' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:00:49.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:49 smithi040 ceph-mon[39581]: Saving service mgr spec with placement count:2 2024-04-29T14:00:49.935 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:49 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:50 smithi040 ceph-mon[39581]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:00:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3240583894' entity='client.admin' 2024-04-29T14:00:53.015 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling the dashboard module... 2024-04-29T14:00:53.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:53 smithi040 ceph-mon[39581]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:00:53.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:53 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:53.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1927778104' entity='client.admin' 2024-04-29T14:00:54.319 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:54 smithi040 ceph-mon[39581]: from='mgr.14120 v1:172.21.15.40:0/1355357312' entity='mgr.y' 2024-04-29T14:00:54.320 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/495211331' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-29T14:00:54.320 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:54 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: ignoring --setuser ceph since I am not root 2024-04-29T14:00:54.320 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:54 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: ignoring --setgroup ceph since I am not root 2024-04-29T14:00:54.607 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:54 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:54.322+0000 7f4cea8ea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-29T14:00:54.608 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:54 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:54.506+0000 7f4cea8ea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-29T14:00:54.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:54 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:54.850+0000 7f4cea8ea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-29T14:00:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/495211331' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-29T14:00:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:55 smithi040 ceph-mon[39581]: mgrmap e9: y(active, since 24s) 2024-04-29T14:00:55.636 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:55 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:55.384+0000 7f4cea8ea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-29T14:00:56.098 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:55 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:55.639+0000 7f4cea8ea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-29T14:00:56.365 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:56.101+0000 7f4cea8ea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-29T14:00:56.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:00:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2172361080' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-29T14:00:57.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:56.865+0000 7f4cea8ea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-29T14:00:57.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:56.956+0000 7f4cea8ea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-29T14:00:57.171 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:00:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-29T14:00:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-29T14:00:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-29T14:00:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-29T14:00:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:00:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-04-29T14:00:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 9... 2024-04-29T14:00:57.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:57 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:57.597+0000 7f4cea8ea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-29T14:00:57.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:57 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:57.689+0000 7f4cea8ea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-29T14:00:57.865 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:57 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:57.784+0000 7f4cea8ea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-29T14:00:58.365 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:58 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:58.019+0000 7f4cea8ea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-29T14:00:58.365 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:58 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:58.116+0000 7f4cea8ea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-29T14:00:58.365 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:58 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:58.215+0000 7f4cea8ea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-29T14:00:58.751 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:58 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:58.458+0000 7f4cea8ea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-29T14:00:58.751 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:58 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:58.559+0000 7f4cea8ea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-29T14:00:59.012 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:58 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:58.755+0000 7f4cea8ea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-29T14:00:59.365 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:59 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:59.015+0000 7f4cea8ea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-29T14:01:00.063 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:59 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:59.634+0000 7f4cea8ea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-29T14:01:00.063 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:00:59 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:00:59.858+0000 7f4cea8ea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-29T14:01:00.365 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:01:00 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:01:00.066+0000 7f4cea8ea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:01:01.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:01:00 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:01:00.763+0000 7f4cea8ea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-29T14:01:01.115 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:01:00 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:01:00.973+0000 7f4cea8ea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:01:01.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: Active manager daemon y restarted 2024-04-29T14:01:01.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: Activating manager daemon y 2024-04-29T14:01:01.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: osdmap e3: 0 total, 0 up, 0 in 2024-04-29T14:01:01.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: mgrmap e10: y(active, starting, since 0.00325585s) 2024-04-29T14:01:01.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: Manager daemon y is now available 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:01:01.167+0000 7f4cea8ea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-29T14:01:01.422 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:01:01 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:01:01.289+0000 7f4cea8ea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-29T14:01:02.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-29T14:01:02.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-29T14:01:02.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:02 smithi040 ceph-mon[39581]: mgrmap e11: y(active, since 1.0057s) 2024-04-29T14:01:03.048 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-29T14:01:03.048 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-29T14:01:03.048 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-29T14:01:03.048 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-29T14:01:03.048 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 9 is available 2024-04-29T14:01:03.049 INFO:teuthology.orchestra.run.smithi040.stdout:Generating a dashboard self-signed certificate... 2024-04-29T14:01:04.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:04 smithi040 ceph-mon[39581]: from='client.14157 v1:172.21.15.40:0/2788230608' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-29T14:01:04.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:04 smithi040 ceph-mon[39581]: from='client.14157 v1:172.21.15.40:0/2788230608' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-29T14:01:04.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:04.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:04.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:04.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:04.529 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-29T14:01:04.529 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial admin user... 2024-04-29T14:01:05.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:05 smithi040 ceph-mon[39581]: [29/Apr/2024:14:01:03] ENGINE Bus STARTING 2024-04-29T14:01:05.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:05 smithi040 ceph-mon[39581]: [29/Apr/2024:14:01:03] ENGINE Serving on http://172.21.15.40:8765 2024-04-29T14:01:05.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:05 smithi040 ceph-mon[39581]: [29/Apr/2024:14:01:03] ENGINE Serving on https://172.21.15.40:7150 2024-04-29T14:01:05.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:05 smithi040 ceph-mon[39581]: [29/Apr/2024:14:01:03] ENGINE Bus STARTED 2024-04-29T14:01:05.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:05 smithi040 ceph-mon[39581]: from='client.14165 v1:172.21.15.40:0/2644986661' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:05.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:05 smithi040 ceph-mon[39581]: mgrmap e12: y(active, since 2s) 2024-04-29T14:01:06.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$mhMTP2BJiB0iwGRyuAVrQ.C4m8ICnFT5YqW3eE9KZrvU7i8husYcC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714399265, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-29T14:01:06.683 INFO:teuthology.orchestra.run.smithi040.stdout:Fetching dashboard port number... 2024-04-29T14:01:06.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:06 smithi040 ceph-mon[39581]: from='client.14167 v1:172.21.15.40:0/3761490370' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:06.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:08.392 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 8443 2024-04-29T14:01:08.403 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-29T14:01:08.403 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-29T14:01:08.413 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-29T14:01:08.413 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-29T14:01:08.413 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-29T14:01:08.413 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-29T14:01:08.415 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph Dashboard is now available at: 2024-04-29T14:01:08.415 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:08.415 INFO:teuthology.orchestra.run.smithi040.stdout: URL: https://smithi040.front.sepia.ceph.com:8443/ 2024-04-29T14:01:08.415 INFO:teuthology.orchestra.run.smithi040.stdout: User: admin 2024-04-29T14:01:08.415 INFO:teuthology.orchestra.run.smithi040.stdout: Password: kb9mb0xzfc 2024-04-29T14:01:08.415 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:08.415 INFO:teuthology.orchestra.run.smithi040.stdout:Saving cluster configuration to /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config directory 2024-04-29T14:01:08.416 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling autotune for osd_memory_target 2024-04-29T14:01:08.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:08 smithi040 ceph-mon[39581]: mgrmap e13: y(active, since 6s) 2024-04-29T14:01:08.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1700207165' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-29T14:01:11.163 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-29T14:01:11.163 INFO:teuthology.orchestra.run.smithi040.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-29T14:01:11.163 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout:Or, if you are only running a single cluster on this host: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: ceph telemetry on 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout:For more information see: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:11.164 INFO:teuthology.orchestra.run.smithi040.stdout:Bootstrap complete. 2024-04-29T14:01:11.214 INFO:tasks.cephadm:Fetching config... 2024-04-29T14:01:11.215 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:01:11.215 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-29T14:01:11.235 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-29T14:01:11.235 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:01:11.236 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-29T14:01:11.294 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-29T14:01:11.294 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:01:11.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-29T14:01:11.362 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-29T14:01:11.362 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:01:11.362 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-29T14:01:11.421 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-29T14:01:11.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5JrM3OIVkTz3pORhtZtdiBSbdYGczyqGqVEUk1Cb1XTthaEPIcgrKq0tckFjDhxPhPbx4NqZbRSmfek1EnNbydxbKA3JQXGzE756hIQuD83ADaih6/LIkPRP8CTJbiLnPisSx8ZfzlCiu2AvSpYaM9dz2QJSCmUn0VPFkeepScyUre23ZTn5CohHoBUpHoFMCx7UaHYd8rFsNZrPjZavT8Blnq66sa8LqZUYz2OpmAIurxr6y6Nhlf1c7Qf+IMelwxnhFD9Z/xAnvz10tNR07PTjH+7a/CM4GkMVKeYmkYiCisx111yedt3rcOxUE7PsS0lTit/3rBKwtQfakVaj1IyI+jTVxHiM1LJ7pKGJIL6eYc+0bEdDm3eiqRgC6pyVcbFzKc9oT7TbK47MNvGIw7bcg6Dc+95XaIugOysXLBcQx51Oi1L8b0Elv6Pq8mb+b1djucZp+16HCM8ebl/kA6zKvKPqYgLYYw2pr74/ThFbuOVVn7lycjmchLribZs= ceph-998f2e22-0630-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-29T14:01:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5JrM3OIVkTz3pORhtZtdiBSbdYGczyqGqVEUk1Cb1XTthaEPIcgrKq0tckFjDhxPhPbx4NqZbRSmfek1EnNbydxbKA3JQXGzE756hIQuD83ADaih6/LIkPRP8CTJbiLnPisSx8ZfzlCiu2AvSpYaM9dz2QJSCmUn0VPFkeepScyUre23ZTn5CohHoBUpHoFMCx7UaHYd8rFsNZrPjZavT8Blnq66sa8LqZUYz2OpmAIurxr6y6Nhlf1c7Qf+IMelwxnhFD9Z/xAnvz10tNR07PTjH+7a/CM4GkMVKeYmkYiCisx111yedt3rcOxUE7PsS0lTit/3rBKwtQfakVaj1IyI+jTVxHiM1LJ7pKGJIL6eYc+0bEdDm3eiqRgC6pyVcbFzKc9oT7TbK47MNvGIw7bcg6Dc+95XaIugOysXLBcQx51Oi1L8b0Elv6Pq8mb+b1djucZp+16HCM8ebl/kA6zKvKPqYgLYYw2pr74/ThFbuOVVn7lycjmchLribZs= ceph-998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:01:11.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5JrM3OIVkTz3pORhtZtdiBSbdYGczyqGqVEUk1Cb1XTthaEPIcgrKq0tckFjDhxPhPbx4NqZbRSmfek1EnNbydxbKA3JQXGzE756hIQuD83ADaih6/LIkPRP8CTJbiLnPisSx8ZfzlCiu2AvSpYaM9dz2QJSCmUn0VPFkeepScyUre23ZTn5CohHoBUpHoFMCx7UaHYd8rFsNZrPjZavT8Blnq66sa8LqZUYz2OpmAIurxr6y6Nhlf1c7Qf+IMelwxnhFD9Z/xAnvz10tNR07PTjH+7a/CM4GkMVKeYmkYiCisx111yedt3rcOxUE7PsS0lTit/3rBKwtQfakVaj1IyI+jTVxHiM1LJ7pKGJIL6eYc+0bEdDm3eiqRgC6pyVcbFzKc9oT7TbK47MNvGIw7bcg6Dc+95XaIugOysXLBcQx51Oi1L8b0Elv6Pq8mb+b1djucZp+16HCM8ebl/kA6zKvKPqYgLYYw2pr74/ThFbuOVVn7lycjmchLribZs= ceph-998f2e22-0630-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-29T14:01:11.561 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5JrM3OIVkTz3pORhtZtdiBSbdYGczyqGqVEUk1Cb1XTthaEPIcgrKq0tckFjDhxPhPbx4NqZbRSmfek1EnNbydxbKA3JQXGzE756hIQuD83ADaih6/LIkPRP8CTJbiLnPisSx8ZfzlCiu2AvSpYaM9dz2QJSCmUn0VPFkeepScyUre23ZTn5CohHoBUpHoFMCx7UaHYd8rFsNZrPjZavT8Blnq66sa8LqZUYz2OpmAIurxr6y6Nhlf1c7Qf+IMelwxnhFD9Z/xAnvz10tNR07PTjH+7a/CM4GkMVKeYmkYiCisx111yedt3rcOxUE7PsS0lTit/3rBKwtQfakVaj1IyI+jTVxHiM1LJ7pKGJIL6eYc+0bEdDm3eiqRgC6pyVcbFzKc9oT7TbK47MNvGIw7bcg6Dc+95XaIugOysXLBcQx51Oi1L8b0Elv6Pq8mb+b1djucZp+16HCM8ebl/kA6zKvKPqYgLYYw2pr74/ThFbuOVVn7lycjmchLribZs= ceph-998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:01:11.572 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5JrM3OIVkTz3pORhtZtdiBSbdYGczyqGqVEUk1Cb1XTthaEPIcgrKq0tckFjDhxPhPbx4NqZbRSmfek1EnNbydxbKA3JQXGzE756hIQuD83ADaih6/LIkPRP8CTJbiLnPisSx8ZfzlCiu2AvSpYaM9dz2QJSCmUn0VPFkeepScyUre23ZTn5CohHoBUpHoFMCx7UaHYd8rFsNZrPjZavT8Blnq66sa8LqZUYz2OpmAIurxr6y6Nhlf1c7Qf+IMelwxnhFD9Z/xAnvz10tNR07PTjH+7a/CM4GkMVKeYmkYiCisx111yedt3rcOxUE7PsS0lTit/3rBKwtQfakVaj1IyI+jTVxHiM1LJ7pKGJIL6eYc+0bEdDm3eiqRgC6pyVcbFzKc9oT7TbK47MNvGIw7bcg6Dc+95XaIugOysXLBcQx51Oi1L8b0Elv6Pq8mb+b1djucZp+16HCM8ebl/kA6zKvKPqYgLYYw2pr74/ThFbuOVVn7lycjmchLribZs= ceph-998f2e22-0630-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-29T14:01:11.619 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz5JrM3OIVkTz3pORhtZtdiBSbdYGczyqGqVEUk1Cb1XTthaEPIcgrKq0tckFjDhxPhPbx4NqZbRSmfek1EnNbydxbKA3JQXGzE756hIQuD83ADaih6/LIkPRP8CTJbiLnPisSx8ZfzlCiu2AvSpYaM9dz2QJSCmUn0VPFkeepScyUre23ZTn5CohHoBUpHoFMCx7UaHYd8rFsNZrPjZavT8Blnq66sa8LqZUYz2OpmAIurxr6y6Nhlf1c7Qf+IMelwxnhFD9Z/xAnvz10tNR07PTjH+7a/CM4GkMVKeYmkYiCisx111yedt3rcOxUE7PsS0lTit/3rBKwtQfakVaj1IyI+jTVxHiM1LJ7pKGJIL6eYc+0bEdDm3eiqRgC6pyVcbFzKc9oT7TbK47MNvGIw7bcg6Dc+95XaIugOysXLBcQx51Oi1L8b0Elv6Pq8mb+b1djucZp+16HCM8ebl/kA6zKvKPqYgLYYw2pr74/ThFbuOVVn7lycjmchLribZs= ceph-998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:01:11.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-29T14:01:11.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3083866108' entity='client.admin' 2024-04-29T14:01:11.905 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:01:14.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:14.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:14.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:01:14.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:14.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:14.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:15.367 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-29T14:01:15.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-29T14:01:15.921 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2259790525' entity='client.admin' 2024-04-29T14:01:15.922 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:15.922 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:15.971 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:01:18.520 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi121 2024-04-29T14:01:18.520 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T14:01:18.520 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.conf 2024-04-29T14:01:18.542 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T14:01:18.542 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:18.602 INFO:tasks.cephadm:Adding host smithi121 to orchestrator... 2024-04-29T14:01:18.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch host add smithi121 2024-04-29T14:01:18.898 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:01:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:18 smithi040 ceph-mon[39581]: from='client.14177 v1:172.21.15.40:0/3153696479' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:19.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:19.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:01:19.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:18 smithi040 ceph-mon[39581]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-29T14:01:19.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:18 smithi040 ceph-mon[39581]: Updating smithi040:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:01:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:19 smithi040 ceph-mon[39581]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:19 smithi040 ceph-mon[39581]: Updating smithi040:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-29T14:01:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:20.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:20.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:21.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:21 smithi040 ceph-mon[39581]: from='client.14179 v1:172.21.15.40:0/1189777570' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:22.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:22 smithi040 ceph-mon[39581]: Deploying cephadm binary to smithi121 2024-04-29T14:01:22.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:22 smithi040 ceph-mon[39581]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:24.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:24 smithi040 ceph-mon[39581]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:24.653 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi121' with addr '172.21.15.121' 2024-04-29T14:01:25.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-29T14:01:25.582 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:01:26.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:26.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:25 smithi040 ceph-mon[39581]: Added host smithi121 2024-04-29T14:01:26.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:26.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:26.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:27.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:27 smithi040 ceph-mon[39581]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:27.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:27.702 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}] 2024-04-29T14:01:28.363 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-04-29T14:01:28.363 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T14:01:28.363 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-04-29T14:01:28.385 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T14:01:28.385 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:28.445 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-04-29T14:01:28.446 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch host add smithi194 2024-04-29T14:01:28.723 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:01:29.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:29 smithi040 ceph-mon[39581]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:29.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:29 smithi040 ceph-mon[39581]: from='client.14181 v1:172.21.15.40:0/1791375310' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:01:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: Updating smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:31.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:31 smithi040 ceph-mon[39581]: Updating smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-29T14:01:31.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:31 smithi040 ceph-mon[39581]: from='client.14183 v1:172.21.15.40:0/3383400788' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:32.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:32 smithi040 ceph-mon[39581]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:32.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:32 smithi040 ceph-mon[39581]: Deploying cephadm binary to smithi194 2024-04-29T14:01:34.713 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-04-29T14:01:34.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:34 smithi040 ceph-mon[39581]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:35.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-29T14:01:35.701 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:01:36.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:36.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:35 smithi040 ceph-mon[39581]: Added host smithi194 2024-04-29T14:01:36.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:36.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:36.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:37.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:37 smithi040 ceph-mon[39581]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:37.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:37.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:01:37.630 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-04-29T14:01:38.278 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-29T14:01:38.279 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-29T14:01:38.550 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:01:39.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:39 smithi040 ceph-mon[39581]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:39.426 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:39 smithi040 ceph-mon[39581]: from='client.14185 v1:172.21.15.40:0/3405814543' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:01:40.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:40.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:40.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:40.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:40.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: Updating smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: Updating smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:40.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:41.354 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-04-29T14:01:41.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2066581468' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-29T14:01:41.971 INFO:tasks.cephadm:Adding mon.a on smithi040 2024-04-29T14:01:41.971 INFO:tasks.cephadm:Adding mon.b on smithi121 2024-04-29T14:01:41.971 INFO:tasks.cephadm:Adding mon.c on smithi194 2024-04-29T14:01:41.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi040:[v1:172.21.15.40:6789]=a;smithi121:[v1:172.21.15.121:6789]=b;smithi194:[v1:172.21.15.194:6789]=c' 2024-04-29T14:01:42.275 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:01:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:42 smithi040 ceph-mon[39581]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2066581468' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-29T14:01:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:42 smithi040 ceph-mon[39581]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:01:44.339 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-04-29T14:01:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:44 smithi040 ceph-mon[39581]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:44.967 DEBUG:teuthology.orchestra.run.smithi121:mon.b> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.b.service 2024-04-29T14:01:44.969 DEBUG:teuthology.orchestra.run.smithi194:mon.c> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.c.service 2024-04-29T14:01:44.971 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-29T14:01:44.971 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-29T14:01:45.001 INFO:journalctl@ceph.mon.c.smithi194.stdout:-- Logs begin at Mon 2024-04-29 13:48:49 UTC. -- 2024-04-29T14:01:45.005 INFO:journalctl@ceph.mon.b.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:01:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: from='client.14189 v1:172.21.15.194:0/2012214912' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:[v1:172.21.15.40:6789]=a;smithi121:[v1:172.21.15.121:6789]=b;smithi194:[v1:172.21.15.194:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: Saving service mon spec with placement smithi040:[v1:172.21.15.40:6789]=a;smithi121:[v1:172.21.15.121:6789]=b;smithi194:[v1:172.21.15.194:6789]=c;count:3 2024-04-29T14:01:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:01:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:45.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:01:45.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:45.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:45 smithi040 ceph-mon[39581]: Deploying daemon mon.c on smithi194 2024-04-29T14:01:45.763 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:01:46.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:46 smithi040 ceph-mon[39581]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:48.621 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T14:01:48.621 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","modified":"2024-04-29T13:59:58.602057Z","created":"2024-04-29T13:59:58.602057Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-29T14:01:48.621 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-04-29T14:01:49.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:48 smithi040 ceph-mon[39581]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:49.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/2703238630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:01:49.274 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 systemd[1]: Starting Ceph mon.c for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:01:50.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: set uid:gid to 167:167 (ceph:ceph) 2024-04-29T14:01:50.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: ceph version 19.0.0-3300-g7fe685c9 (7fe685c922bae3b52755209e35feddd972749cff) squid (dev), process ceph-mon, pid 7 2024-04-29T14:01:50.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pidfile_write: ignore empty --pid-file 2024-04-29T14:01:50.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: load: jerasure load: lrc 2024-04-29T14:01:50.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: RocksDB version: 7.9.2 2024-04-29T14:01:50.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Git sha 0 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Compile date 2024-04-26 09:58:26 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: DB SUMMARY 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: DB Session ID: 7UA2M5A4IOPYCANK7EIW 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: CURRENT file: CURRENT 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: IDENTITY file: IDENTITY 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 476 ; 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.error_if_exists: 0 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.create_if_missing: 0 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.paranoid_checks: 1 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-29T14:01:50.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.env: 0x55f5e2a60cc0 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.fs: PosixFileSystem 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.info_log: 0x55f5e348fde0 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_file_opening_threads: 16 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.statistics: (nil) 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.use_fsync: 0 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_log_file_size: 0 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.keep_log_file_num: 1000 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.recycle_log_file_num: 0 2024-04-29T14:01:50.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.allow_fallocate: 1 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.allow_mmap_reads: 0 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.allow_mmap_writes: 0 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.use_direct_reads: 0 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.create_missing_column_families: 0 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.db_log_dir: 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.wal_dir: 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-29T14:01:50.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.advise_random_on_open: 1 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.db_write_buffer_size: 0 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.write_buffer_manager: 0x55f5e34610e0 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.rate_limiter: (nil) 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.wal_recovery_mode: 2 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.enable_thread_tracking: 0 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.enable_pipelined_write: 0 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.unordered_write: 0 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-29T14:01:50.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.row_cache: None 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.wal_filter: None 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.allow_ingest_behind: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.two_write_queues: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.manual_wal_flush: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.wal_compression: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.atomic_flush: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.log_readahead_size: 0 2024-04-29T14:01:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.best_efforts_recovery: 0 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.allow_data_in_errors: 0 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.db_host_id: __hostname__ 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_background_jobs: 2 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_background_compactions: -1 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_subcompactions: 1 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-29T14:01:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_total_wal_size: 0 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_open_files: -1 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bytes_per_sync: 0 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_readahead_size: 0 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_background_flushes: -1 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Compression algorithms supported: 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kZSTD supported: 0 2024-04-29T14:01:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kXpressCompression supported: 0 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kLZ4HCCompression supported: 1 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kZlibCompression supported: 1 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kSnappyCompression supported: 1 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kLZ4Compression supported: 1 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: kBZip2Compression supported: 0 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.merge_operator: 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_filter: None 2024-04-29T14:01:50.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_filter_factory: None 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.sst_partitioner_factory: None 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f5e348ffa0) 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: cache_index_and_filter_blocks: 1 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: pin_top_level_index_and_filter: 1 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: index_type: 0 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: data_block_index_type: 0 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: index_shortening: 1 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: checksum: 4 2024-04-29T14:01:50.077 INFO:journalctl@ceph.mon.c.smithi194.stdout: no_block_cache: 0 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache: 0x55f5e34931f0 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache_name: BinnedLRUCache 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache_options: 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: capacity : 536870912 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: num_shard_bits : 4 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: strict_capacity_limit : 0 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: high_pri_pool_ratio: 0.000 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_cache_compressed: (nil) 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: persistent_cache: (nil) 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_size: 4096 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_size_deviation: 10 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_restart_interval: 16 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: index_block_restart_interval: 1 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: metadata_block_size: 4096 2024-04-29T14:01:50.078 INFO:journalctl@ceph.mon.c.smithi194.stdout: partition_filters: 0 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: use_delta_encoding: 1 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: filter_policy: bloomfilter 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: whole_key_filtering: 1 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: verify_compression: 0 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: read_amp_bytes_per_bit: 0 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: format_version: 5 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: enable_index_compression: 1 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: block_align: 0 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: max_auto_readahead_size: 262144 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: prepopulate_block_cache: 0 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: initial_auto_readahead_size: 8192 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout: num_file_reads_for_auto_readahead: 2 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.write_buffer_size: 33554432 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_write_buffer_number: 2 2024-04-29T14:01:50.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression: NoCompression 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression: Disabled 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.prefix_extractor: nullptr 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.num_levels: 7 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-29T14:01:50.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.level: 32767 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.strategy: 0 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.enabled: false 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-29T14:01:50.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.target_file_size_base: 67108864 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-29T14:01:50.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.arena_block_size: 1048576 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.disable_auto_compactions: 0 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-29T14:01:50.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.inplace_update_support: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.bloom_locality: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.max_successive_merges: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.paranoid_file_checks: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.force_consistency_checks: 1 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.report_bg_io_stats: 0 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.ttl: 2592000 2024-04-29T14:01:50.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.enable_blob_files: false 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.min_blob_size: 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.blob_file_size: 268435456 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.blob_file_starting_level: 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-29T14:01:50.085 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: eaea379c-7005-4382-912a-30c2ff873c24 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399309785953, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399309786345, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714399309, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "eaea379c-7005-4382-912a-30c2ff873c24", "db_session_id": "7UA2M5A4IOPYCANK7EIW", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399309786421, "job": 1, "event": "recovery_finished"} 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f5e35b0000 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: DB pointer 0x55f5e359c000 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** DB Stats ** 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-29T14:01:50.086 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** Compaction Stats [default] ** 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** Compaction Stats [default] ** 2024-04-29T14:01:50.087 INFO:journalctl@ceph.mon.c.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-29T14:01:50.088 INFO:journalctl@ceph.mon.c.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: Block cache BinnedLRUCache@0x55f5e34931f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: using public_addrv v1:172.21.15.194:6789/0 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: starting mon.c rank -1 at public addrs v1:172.21.15.194:6789/0 at bind addrs v1:172.21.15.194:6789/0 mon_data /var/lib/ceph/mon/ceph-c fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(???) e0 preinit fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).mds e1 new map 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: e1 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-29T14:01:50.089 INFO:journalctl@ceph.mon.c.smithi194.stdout: legacy client fscid: -1 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout: No filesystems configured 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: (Original Log Time 2024/04/29-14:01:49.805681) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399309805809, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1124, "num_deletes": 25, "total_data_size": 2274093, "memory_usage": 2297424, "flush_reason": "Manual Compaction"} 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-29T14:01:50.090 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:01:50.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='client.14183 v1:172.21.15.40:0/3383400788' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Deploying cephadm binary to smithi194 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Added host smithi194 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:50.092 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='client.14185 v1:172.21.15.40:0/3405814543' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-29T14:01:50.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Updating smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2066581468' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2066581468' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='client.14189 v1:172.21.15.194:0/2012214912' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:[v1:172.21.15.40:6789]=a;smithi121:[v1:172.21.15.121:6789]=b;smithi194:[v1:172.21.15.194:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:01:50.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Saving service mon spec with placement smithi040:[v1:172.21.15.40:6789]=a;smithi121:[v1:172.21.15.121:6789]=b;smithi194:[v1:172.21.15.194:6789]=c;count:3 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: Deploying daemon mon.c on smithi194 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/2703238630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399309809631, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 1997100, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1129, "table_properties": {"data_size": 1991448, "index_size": 2472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2117, "raw_key_size": 26528, "raw_average_key_size": 31, "raw_value_size": 1973668, "raw_average_value_size": 2341, "num_data_blocks": 114, "num_entries": 843, "num_filter_entries": 843, "num_deletions": 25, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714399309, "oldest_key_time": 1714399309, "file_creation_time": 1714399309, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "eaea379c-7005-4382-912a-30c2ff873c24", "db_session_id": "7UA2M5A4IOPYCANK7EIW", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 3975 microseconds, and 2632 cpu microseconds. 2024-04-29T14:01:50.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: (Original Log Time 2024/04/29-14:01:49.809656) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 1997100 bytes OK 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: (Original Log Time 2024/04/29-14:01:49.809670) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: (Original Log Time 2024/04/29-14:01:49.809775) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: (Original Log Time 2024/04/29-14:01:49.809786) EVENT_LOG_v1 {"time_micros": 1714399309809782, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: (Original Log Time 2024/04/29-14:01:49.809795) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2266260, prev total WAL file size 2266401, number of live WAL files 2. 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: (Original Log Time 2024/04/29-14:01:49.810104) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:22 .. '6D677273746174003132' seq:0, type:0; will stop at (end) 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(1950KB) 8(1613B)] 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 ceph-mon[44089]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714399309810164, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 1998713, "oldest_snapshot_seqno": -1} 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 bash[44052]: eb221a82477f7dd51846e5610e37bd10941be45920fb0f86863ef70150472755 2024-04-29T14:01:50.096 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:49 smithi194 systemd[1]: Started Ceph mon.c for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:01:50.322 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-29T14:01:50.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-29T14:01:50.582 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:01:55.158 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T14:01:55.159 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","modified":"2024-04-29T14:01:49.815258Z","created":"2024-04-29T13:59:58.602057Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-29T14:01:55.161 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-04-29T14:01:55.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:55.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:01:55.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: mon.a calling monitor election 2024-04-29T14:01:55.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.181 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: mon.c calling monitor election 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: monmap epoch 2 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: last_changed 2024-04-29T14:01:49.815258+0000 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: min_mon_release 19 (squid) 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: election_strategy: 1 2024-04-29T14:01:55.182 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:01:55.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:01:55.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: fsmap 2024-04-29T14:01:55.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:01:55.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: mgrmap e13: y(active, since 53s) 2024-04-29T14:01:55.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: overall HEALTH_OK 2024-04-29T14:01:55.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:55.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:01:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: mon.a calling monitor election 2024-04-29T14:01:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:55.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: mon.c calling monitor election 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: monmap epoch 2 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: last_changed 2024-04-29T14:01:49.815258+0000 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: min_mon_release 19 (squid) 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: election_strategy: 1 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:01:55.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:01:55.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: fsmap 2024-04-29T14:01:55.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:01:55.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: mgrmap e13: y(active, since 53s) 2024-04-29T14:01:55.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: overall HEALTH_OK 2024-04-29T14:01:55.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:55.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:56.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:56.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:01:56.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:56.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:55 smithi194 ceph-mon[44089]: Deploying daemon mon.b on smithi121 2024-04-29T14:01:56.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/3576790625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:01:56.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:56.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:01:56.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:01:56.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:01:56.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:55 smithi040 ceph-mon[39581]: Deploying daemon mon.b on smithi121 2024-04-29T14:01:56.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/3576790625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:01:56.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:01:56.830 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-29T14:01:56.830 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-29T14:01:57.098 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:01:57.308 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:01:56 smithi194 ceph-mon[44089]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:57.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:01:56 smithi040 ceph-mon[39581]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:01:59.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:01:58 smithi121 bash[44495]: 34259f0af653c1eba11ab124c24330bb34289fd38116b925d27c2085f0b803f9 2024-04-29T14:01:59.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:01:58 smithi121 systemd[1]: Started Ceph mon.b for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:02:03.774 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-29T14:02:03.774 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","modified":"2024-04-29T14:01:58.747212Z","created":"2024-04-29T13:59:58.602057Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-29T14:02:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-04-29T14:02:04.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:04.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: mon.a calling monitor election 2024-04-29T14:02:04.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: mon.c calling monitor election 2024-04-29T14:02:04.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/1219897412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: monmap epoch 3 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: last_changed 2024-04-29T14:01:58.747212+0000 2024-04-29T14:02:04.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: min_mon_release 19 (squid) 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: election_strategy: 1 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: 2: v1:172.21.15.121:6789/0 mon.b 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: fsmap 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: mgrmap e13: y(active, since 62s) 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:04.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:04.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-29T14:02:04.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-29T14:02:04.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: mon.b (rank 2) addr v1:172.21.15.121:6789/0 is down (out of quorum) 2024-04-29T14:02:04.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:03 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:04.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: mon.a calling monitor election 2024-04-29T14:02:04.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: mon.c calling monitor election 2024-04-29T14:02:04.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/1219897412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: monmap epoch 3 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: last_changed 2024-04-29T14:01:58.747212+0000 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:02:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: min_mon_release 19 (squid) 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: election_strategy: 1 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: 2: v1:172.21.15.121:6789/0 mon.b 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: fsmap 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: mgrmap e13: y(active, since 62s) 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-29T14:02:04.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-29T14:02:04.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: mon.b (rank 2) addr v1:172.21.15.121:6789/0 is down (out of quorum) 2024-04-29T14:02:04.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:03 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:04.408 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-29T14:02:04.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-29T14:02:04.677 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:02:05.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:05.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:05.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:05.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:06.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: mon.a calling monitor election 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: mon.c calling monitor election 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/1219897412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: monmap epoch 3 2024-04-29T14:02:06.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: last_changed 2024-04-29T14:01:58.747212+0000 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: min_mon_release 19 (squid) 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: election_strategy: 1 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: 2: v1:172.21.15.121:6789/0 mon.b 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: fsmap 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: mgrmap e13: y(active, since 62s) 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:06.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: mon.b (rank 2) addr v1:172.21.15.121:6789/0 is down (out of quorum) 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: mon.b calling monitor election 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: mon.b calling monitor election 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: mon.c calling monitor election 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: mon.a calling monitor election 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: monmap epoch 3 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: last_changed 2024-04-29T14:01:58.747212+0000 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:02:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: min_mon_release 19 (squid) 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: election_strategy: 1 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: 2: v1:172.21.15.121:6789/0 mon.b 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: fsmap 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: mgrmap e13: y(active, since 64s) 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: overall HEALTH_OK 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: mon.b calling monitor election 2024-04-29T14:02:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: mon.b calling monitor election 2024-04-29T14:02:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: mon.c calling monitor election 2024-04-29T14:02:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: mon.a calling monitor election 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: monmap epoch 3 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: last_changed 2024-04-29T14:01:58.747212+0000 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: min_mon_release 19 (squid) 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: election_strategy: 1 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: 2: v1:172.21.15.121:6789/0 mon.b 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: fsmap 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: mgrmap e13: y(active, since 64s) 2024-04-29T14:02:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: overall HEALTH_OK 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: mon.b calling monitor election 2024-04-29T14:02:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: mon.b calling monitor election 2024-04-29T14:02:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: mon.c calling monitor election 2024-04-29T14:02:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: mon.a calling monitor election 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: monmap epoch 3 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: last_changed 2024-04-29T14:01:58.747212+0000 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: created 2024-04-29T13:59:58.602057+0000 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: min_mon_release 19 (squid) 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: election_strategy: 1 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: 0: v1:172.21.15.40:6789/0 mon.a 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: 1: v1:172.21.15.194:6789/0 mon.c 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: 2: v1:172.21.15.121:6789/0 mon.b 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: fsmap 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: osdmap e4: 0 total, 0 up, 0 in 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: mgrmap e13: y(active, since 64s) 2024-04-29T14:02:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: overall HEALTH_OK 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:06.647 INFO:teuthology.orchestra.run.smithi040.stdout:# minimal ceph.conf for 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:06.648 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-04-29T14:02:06.648 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:02:06.648 INFO:teuthology.orchestra.run.smithi040.stdout: mon_host = 172.21.15.40:6789/0 172.21.15.121:6789/0 172.21.15.194:6789/0 2024-04-29T14:02:07.600 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:07 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:07.600 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/421258877' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:07.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:07 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:07.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/421258877' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:07.630 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-29T14:02:07.631 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:02:07.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-04-29T14:02:07.664 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:02:07.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:02:07.735 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T14:02:07.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2024-04-29T14:02:07.773 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T14:02:07.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:02:07.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:07 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:07.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/421258877' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:07.843 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T14:02:07.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-04-29T14:02:07.880 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T14:02:07.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-29T14:02:07.952 INFO:tasks.cephadm:Adding mgr.y on smithi040 2024-04-29T14:02:07.952 INFO:tasks.cephadm:Adding mgr.x on smithi121 2024-04-29T14:02:07.952 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi040=y;smithi121=x' 2024-04-29T14:02:08.250 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-29T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-29T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-29T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Updating smithi040:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Updating smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Updating smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Reconfiguring mon.a (unknown last config time)... 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: Reconfiguring daemon mon.a on smithi040 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:08.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-29T14:02:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-29T14:02:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-29T14:02:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Updating smithi040:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Updating smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Updating smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Reconfiguring mon.a (unknown last config time)... 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: Reconfiguring daemon mon.a on smithi040 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:08.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-29T14:02:08.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-29T14:02:08.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Updating smithi040:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Updating smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Updating smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Reconfiguring mon.a (unknown last config time)... 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: Reconfiguring daemon mon.a on smithi040 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:08.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:09.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:09 smithi121 ceph-mon[44533]: Reconfiguring mon.b (monmap changed)... 2024-04-29T14:02:09.475 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:09 smithi121 ceph-mon[44533]: Reconfiguring daemon mon.b on smithi121 2024-04-29T14:02:09.650 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:09 smithi194 ceph-mon[44089]: Reconfiguring mon.b (monmap changed)... 2024-04-29T14:02:09.651 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:09 smithi194 ceph-mon[44089]: Reconfiguring daemon mon.b on smithi121 2024-04-29T14:02:09.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:09 smithi040 ceph-mon[39581]: Reconfiguring mon.b (monmap changed)... 2024-04-29T14:02:09.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:09 smithi040 ceph-mon[39581]: Reconfiguring daemon mon.b on smithi121 2024-04-29T14:02:10.178 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-04-29T14:02:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: Reconfiguring mon.c (monmap changed)... 2024-04-29T14:02:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: Reconfiguring daemon mon.c on smithi194 2024-04-29T14:02:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:10 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: Reconfiguring mon.c (monmap changed)... 2024-04-29T14:02:10.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:10.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:10.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:10.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: Reconfiguring daemon mon.c on smithi194 2024-04-29T14:02:10.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:10 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: Reconfiguring mon.c (monmap changed)... 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: Reconfiguring daemon mon.c on smithi194 2024-04-29T14:02:10.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:10 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='client.24106 v1:172.21.15.194:0/2587926610' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi040=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: Saving service mgr spec with placement smithi040=y;smithi121=x;count:2 2024-04-29T14:02:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-29T14:02:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:11.675 DEBUG:teuthology.orchestra.run.smithi121:mgr.x> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.x.service 2024-04-29T14:02:11.678 INFO:tasks.cephadm:Deploying OSDs... 2024-04-29T14:02:11.678 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:02:11.678 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-04-29T14:02:11.701 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-29T14:02:11.701 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-04-29T14:02:11.708 INFO:journalctl@ceph.mgr.x.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:02:11.762 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-29T14:02:11.762 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:11.762 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 164727 Links: 1 2024-04-29T14:02:11.762 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:11.763 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:11.763 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-29 14:01:14.738883123 +0000 2024-04-29T14:02:11.763 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-29 13:55:10.733581950 +0000 2024-04-29T14:02:11.763 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-29 13:55:10.733581950 +0000 2024-04-29T14:02:11.763 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-04-29T14:02:11.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='client.24106 v1:172.21.15.194:0/2587926610' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi040=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: Saving service mgr spec with placement smithi040=y;smithi121=x;count:2 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:11.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:11.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-29T14:02:11.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-29T14:02:11.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:11.834 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-29T14:02:11.834 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-29T14:02:11.834 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000346577 s, 1.5 MB/s 2024-04-29T14:02:11.836 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-29T14:02:11.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='client.24106 v1:172.21.15.194:0/2587926610' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi040=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:11.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: Saving service mgr spec with placement smithi040=y;smithi121=x;count:2 2024-04-29T14:02:11.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-29T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:11.897 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 167661 Links: 1 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-29 14:01:14.798882383 +0000 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-29 13:55:11.090577592 +0000 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-29 13:55:11.090577592 +0000 2024-04-29T14:02:11.958 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-04-29T14:02:11.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-29T14:02:12.026 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-29T14:02:12.027 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-29T14:02:12.027 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000207639 s, 2.5 MB/s 2024-04-29T14:02:12.028 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-29T14:02:12.089 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-04-29T14:02:12.148 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 170205 Links: 1 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-29 14:01:14.842881840 +0000 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-29 13:55:11.435573381 +0000 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-29 13:55:11.435573381 +0000 2024-04-29T14:02:12.149 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-04-29T14:02:12.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-29T14:02:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-29T14:02:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-29T14:02:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000205274 s, 2.5 MB/s 2024-04-29T14:02:12.218 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-29T14:02:12.278 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-04-29T14:02:12.337 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-29T14:02:12.337 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:12.337 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 168540 Links: 1 2024-04-29T14:02:12.337 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:12.337 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:12.338 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-29 14:01:14.886881297 +0000 2024-04-29T14:02:12.338 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-29 13:55:11.772569268 +0000 2024-04-29T14:02:12.338 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-29 13:55:11.772569268 +0000 2024-04-29T14:02:12.338 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-04-29T14:02:12.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-29T14:02:12.405 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-29T14:02:12.405 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-29T14:02:12.405 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000228991 s, 2.2 MB/s 2024-04-29T14:02:12.406 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-29T14:02:12.468 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T14:02:12.468 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-04-29T14:02:12.487 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-29T14:02:12.487 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 171427 Links: 1 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-29 14:01:58.125505431 +0000 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-29 13:55:02.256254591 +0000 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-29 13:55:02.256254591 +0000 2024-04-29T14:02:12.547 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-29T14:02:12.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-29T14:02:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-29T14:02:12.614 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-29T14:02:12.615 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000213003 s, 2.4 MB/s 2024-04-29T14:02:12.615 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-29T14:02:12.674 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-04-29T14:02:12.732 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 170099 Links: 1 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-29 14:01:58.125505431 +0000 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-29 13:55:02.577250481 +0000 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-29 13:55:02.577250481 +0000 2024-04-29T14:02:12.733 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-29T14:02:12.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-29T14:02:12.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:12 smithi121 ceph-mon[44533]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:12.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:12 smithi121 ceph-mon[44533]: Deploying daemon mgr.x on smithi121 2024-04-29T14:02:12.799 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-29T14:02:12.799 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-29T14:02:12.799 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00020161 s, 2.5 MB/s 2024-04-29T14:02:12.800 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-29T14:02:12.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:12 smithi194 ceph-mon[44089]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:12.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:12 smithi194 ceph-mon[44089]: Deploying daemon mgr.x on smithi121 2024-04-29T14:02:12.858 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-04-29T14:02:12.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:12 smithi040 ceph-mon[39581]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:12.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:12 smithi040 ceph-mon[39581]: Deploying daemon mgr.x on smithi121 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 170151 Links: 1 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-29 14:01:58.125505431 +0000 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-29 13:55:02.909246231 +0000 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-29 13:55:02.909246231 +0000 2024-04-29T14:02:12.926 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-29T14:02:12.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-29T14:02:12.995 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-29T14:02:12.995 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-29T14:02:12.995 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000188138 s, 2.7 MB/s 2024-04-29T14:02:12.996 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-29T14:02:13.058 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-04-29T14:02:13.116 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 169545 Links: 1 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-29 14:02:13.077312331 +0000 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-29 13:55:03.247241904 +0000 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-29 13:55:03.247241904 +0000 2024-04-29T14:02:13.117 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-29T14:02:13.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-29T14:02:13.188 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-29T14:02:13.188 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-29T14:02:13.188 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000217429 s, 2.4 MB/s 2024-04-29T14:02:13.189 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-29T14:02:13.248 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T14:02:13.248 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-04-29T14:02:13.270 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-29T14:02:13.271 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-04-29T14:02:13.332 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-29T14:02:13.333 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 165639 Links: 1 2024-04-29T14:02:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-29 14:01:49.105048991 +0000 2024-04-29T14:02:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-29 13:55:02.268123042 +0000 2024-04-29T14:02:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-29 13:55:02.268123042 +0000 2024-04-29T14:02:13.334 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-29T14:02:13.334 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-29T14:02:13.377 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:13 smithi121 systemd[1]: Starting Ceph mgr.x for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:02:13.405 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-29T14:02:13.406 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-29T14:02:13.406 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000254314 s, 2.0 MB/s 2024-04-29T14:02:13.407 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-29T14:02:13.467 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-04-29T14:02:13.527 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-29T14:02:13.528 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:13.528 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 165702 Links: 1 2024-04-29T14:02:13.528 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:13.528 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:13.528 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-29 14:01:49.105048991 +0000 2024-04-29T14:02:13.528 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-29 13:55:02.608116289 +0000 2024-04-29T14:02:13.528 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-29 13:55:02.608116289 +0000 2024-04-29T14:02:13.529 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-29T14:02:13.529 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-29T14:02:13.596 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-29T14:02:13.596 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-29T14:02:13.596 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000244756 s, 2.1 MB/s 2024-04-29T14:02:13.597 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-29T14:02:13.657 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 169315 Links: 1 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-29 14:01:49.105048991 +0000 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-29 13:55:02.946109575 +0000 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-29 13:55:02.946109575 +0000 2024-04-29T14:02:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-29T14:02:13.717 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-29T14:02:13.782 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-29T14:02:13.782 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-29T14:02:13.782 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00018794 s, 2.7 MB/s 2024-04-29T14:02:13.783 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-29T14:02:13.843 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 172065 Links: 1 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-29 14:01:49.105048991 +0000 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-29 13:55:03.282102901 +0000 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-29 13:55:03.282102901 +0000 2024-04-29T14:02:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-29T14:02:13.903 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-29T14:02:13.968 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-29T14:02:13.968 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-29T14:02:13.968 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000229585 s, 2.2 MB/s 2024-04-29T14:02:13.969 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-29T14:02:14.027 INFO:tasks.cephadm:Deploying osd.0 on smithi040 with /dev/vg_nvme/lv_4... 2024-04-29T14:02:14.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-29T14:02:14.127 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:13 smithi121 bash[46106]: aff8b0c83de1d3bcc3fac1e348e6187152f05439b035a03489263f028e7c20f6 2024-04-29T14:02:14.127 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:13 smithi121 systemd[1]: Started Ceph mgr.x for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:02:14.127 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:13 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:13.761+0000 7f32d7782200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-29T14:02:14.127 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:13 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:13.930+0000 7f32d7782200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-29T14:02:14.314 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:02:14.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:14.270+0000 7f32d7782200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-29T14:02:14.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:14 smithi040 ceph-mon[39581]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:14.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:14 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:14.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:14 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:14.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:14 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:14.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:14 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:14.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:15.053 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:14.813+0000 7f32d7782200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-29T14:02:15.053 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-mon[44533]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:15.053 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.053 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.053 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.053 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.054 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:15.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:14 smithi194 ceph-mon[44089]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:15.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:14 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:14 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:14 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:14 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:15.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:15.377 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:15 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:15.057+0000 7f32d7782200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-29T14:02:15.877 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:15 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:15.505+0000 7f32d7782200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-29T14:02:16.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:16.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: Reconfiguring mgr.y (unknown last config time)... 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:16 smithi194 ceph-mon[44089]: Reconfiguring daemon mgr.y on smithi040 2024-04-29T14:02:16.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: Reconfiguring mgr.y (unknown last config time)... 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:16.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:16 smithi040 ceph-mon[39581]: Reconfiguring daemon mgr.y on smithi040 2024-04-29T14:02:16.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:16.218+0000 7f32d7782200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-29T14:02:16.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:16.303+0000 7f32d7782200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-29T14:02:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: Reconfiguring mgr.y (unknown last config time)... 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-mon[44533]: Reconfiguring daemon mgr.y on smithi040 2024-04-29T14:02:16.660 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-29T14:02:16.660 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-29T14:02:16.660 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-29T14:02:16.660 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-29T14:02:16.661 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108319 s, 968 MB/s 2024-04-29T14:02:16.661 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-29T14:02:17.291 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:16.899+0000 7f32d7782200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-29T14:02:17.291 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:16 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:16.983+0000 7f32d7782200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-29T14:02:17.291 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:17.069+0000 7f32d7782200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-29T14:02:17.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:17 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:17.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:17.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:17.295+0000 7f32d7782200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-29T14:02:17.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:17.387+0000 7f32d7782200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-29T14:02:17.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:17.480+0000 7f32d7782200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-29T14:02:17.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2024-04-29T14:02:17.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:17 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:17.988 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:17.711+0000 7f32d7782200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-29T14:02:17.988 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:17.806+0000 7f32d7782200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-29T14:02:17.988 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:17 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:17.992+0000 7f32d7782200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-29T14:02:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:02:18.377 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:18.239+0000 7f32d7782200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-29T14:02:18.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:18 smithi040 ceph-mon[39581]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:18.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:18 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:18 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:18.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:18.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:18.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:18 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:18 smithi194 ceph-mon[44089]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:18.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:18 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:18 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:18 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-mon[44533]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:18.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:18.818 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:19.127 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:18 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:18.821+0000 7f32d7782200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-29T14:02:19.127 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:19 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:19.032+0000 7f32d7782200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-29T14:02:19.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:19 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:19.228+0000 7f32d7782200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:02:20.257 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:19 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:19.883+0000 7f32d7782200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-29T14:02:20.257 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:20.080+0000 7f32d7782200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-29T14:02:20.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:20.261+0000 7f32d7782200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-29T14:02:20.627 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x[46146]: 2024-04-29T14:02:20.374+0000 7f32d7782200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: Standby manager daemon x started 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-29T14:02:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:20 smithi121 ceph-mon[44533]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-29T14:02:20.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:20.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:02:20.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:02:20.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:20.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: Standby manager daemon x started 2024-04-29T14:02:20.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-29T14:02:20.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-29T14:02:20.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-29T14:02:20.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:20 smithi040 ceph-mon[39581]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-29T14:02:20.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:20.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:02:20.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:02:20.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:20.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: Standby manager daemon x started 2024-04-29T14:02:20.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-29T14:02:20.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-29T14:02:20.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-29T14:02:20.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:20 smithi194 ceph-mon[44089]: from='mgr.? v1:172.21.15.121:0/1939458984' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-29T14:02:21.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:21 smithi194 ceph-mon[44089]: from='client.14220 v1:172.21.15.40:0/2558258318' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:21.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-29T14:02:21.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:21 smithi194 ceph-mon[44089]: mgrmap e14: y(active, since 79s), standbys: x 2024-04-29T14:02:21.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:21 smithi040 ceph-mon[39581]: from='client.14220 v1:172.21.15.40:0/2558258318' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:21.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-29T14:02:21.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:21 smithi040 ceph-mon[39581]: mgrmap e14: y(active, since 79s), standbys: x 2024-04-29T14:02:21.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:21 smithi121 ceph-mon[44533]: from='client.14220 v1:172.21.15.40:0/2558258318' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:21.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-29T14:02:21.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:21 smithi121 ceph-mon[44533]: mgrmap e14: y(active, since 79s), standbys: x 2024-04-29T14:02:22.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:22 smithi194 ceph-mon[44089]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:22.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:22 smithi040 ceph-mon[39581]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:22.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:22 smithi121 ceph-mon[44533]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:23.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/965539837' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462"}]: dispatch 2024-04-29T14:02:23.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/965539837' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462"}]': finished 2024-04-29T14:02:23.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:23.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:23 smithi194 ceph-mon[44089]: osdmap e5: 1 total, 0 up, 1 in 2024-04-29T14:02:23.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1928788906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:02:23.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/965539837' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462"}]: dispatch 2024-04-29T14:02:23.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/965539837' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462"}]': finished 2024-04-29T14:02:23.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:23.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:23 smithi040 ceph-mon[39581]: osdmap e5: 1 total, 0 up, 1 in 2024-04-29T14:02:23.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1928788906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:02:23.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/965539837' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462"}]: dispatch 2024-04-29T14:02:23.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/965539837' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462"}]': finished 2024-04-29T14:02:23.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:23.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:23 smithi121 ceph-mon[44533]: osdmap e5: 1 total, 0 up, 1 in 2024-04-29T14:02:23.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1928788906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:02:24.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:24 smithi194 ceph-mon[44089]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:24.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:24 smithi040 ceph-mon[39581]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:24.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:24 smithi121 ceph-mon[44533]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:26.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:26 smithi040 ceph-mon[39581]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:26.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:26 smithi194 ceph-mon[44089]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:26.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:26 smithi121 ceph-mon[44533]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:28.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:28 smithi040 ceph-mon[39581]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:28.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:28 smithi194 ceph-mon[44089]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:28.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:28 smithi121 ceph-mon[44533]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:29.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-29T14:02:29.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:29.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:29 smithi194 ceph-mon[44089]: Deploying daemon osd.0 on smithi040 2024-04-29T14:02:29.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-29T14:02:29.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:29.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:29 smithi040 ceph-mon[39581]: Deploying daemon osd.0 on smithi040 2024-04-29T14:02:29.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-29T14:02:29.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:29.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:29 smithi121 ceph-mon[44533]: Deploying daemon osd.0 on smithi040 2024-04-29T14:02:30.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:30 smithi040 ceph-mon[39581]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:30.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:30 smithi194 ceph-mon[44089]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:30.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:30 smithi121 ceph-mon[44533]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:32.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:32 smithi194 ceph-mon[44089]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:32.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:32 smithi040 ceph-mon[39581]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:32.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:32 smithi121 ceph-mon[44533]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:34.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:34.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:34.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:34.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:34.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:34.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:34.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:34.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:34.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:34.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:34.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:34.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:34.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:34.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:34.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:34.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:34.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:36.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:36 smithi040 ceph-mon[39581]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:36.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:36 smithi194 ceph-mon[44089]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:36.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:36 smithi121 ceph-mon[44533]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:37.925 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 0 on host 'smithi040' 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:38.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:38.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:38.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:38.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:38.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:38.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:02:38.606 DEBUG:teuthology.orchestra.run.smithi040:osd.0> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.0.service 2024-04-29T14:02:38.609 INFO:tasks.cephadm:Deploying osd.1 on smithi040 with /dev/vg_nvme/lv_3... 2024-04-29T14:02:38.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-29T14:02:38.640 INFO:journalctl@ceph.osd.0.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T14:02:39.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:38 smithi194 ceph-mon[44089]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:39.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:38 smithi194 ceph-mon[44089]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-29T14:02:39.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:38 smithi194 ceph-mon[44089]: osdmap e6: 1 total, 0 up, 1 in 2024-04-29T14:02:39.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:38 smithi194 ceph-mon[44089]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:02:39.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:38 smithi040 ceph-mon[39581]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:38 smithi040 ceph-mon[39581]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-29T14:02:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:38 smithi040 ceph-mon[39581]: osdmap e6: 1 total, 0 up, 1 in 2024-04-29T14:02:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:38 smithi040 ceph-mon[39581]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:02:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:39.366 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:02:38 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-0[58500]: 2024-04-29T14:02:38.980+0000 7fcdba751700 -1 osd.0 0 waiting for initial osdmap 2024-04-29T14:02:39.366 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:02:38 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-0[58500]: 2024-04-29T14:02:38.983+0000 7fcdb6544700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:02:39.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:38 smithi121 ceph-mon[44533]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:39.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:38 smithi121 ceph-mon[44533]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-29T14:02:39.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:38 smithi121 ceph-mon[44533]: osdmap e6: 1 total, 0 up, 1 in 2024-04-29T14:02:39.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:38 smithi121 ceph-mon[44533]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:02:39.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:02:40.096 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:39 smithi040 ceph-mon[39581]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:02:40.096 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:39 smithi040 ceph-mon[39581]: osdmap e7: 1 total, 0 up, 1 in 2024-04-29T14:02:40.096 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:40.096 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:40.096 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:39 smithi040 ceph-mon[39581]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:40.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:39 smithi194 ceph-mon[44089]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:02:40.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:39 smithi194 ceph-mon[44089]: osdmap e7: 1 total, 0 up, 1 in 2024-04-29T14:02:40.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:40.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:40.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:39 smithi194 ceph-mon[44089]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:40.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:39 smithi121 ceph-mon[44533]: from='osd.0 v1:172.21.15.40:6801/4255137508' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:02:40.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:39 smithi121 ceph-mon[44533]: osdmap e7: 1 total, 0 up, 1 in 2024-04-29T14:02:40.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:40.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:40.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:39 smithi121 ceph-mon[44533]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-29T14:02:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:02:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:02:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:02:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:40 smithi194 ceph-mon[44089]: osd.0 v1:172.21.15.40:6801/4255137508 boot 2024-04-29T14:02:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:40 smithi194 ceph-mon[44089]: osdmap e8: 1 total, 1 up, 1 in 2024-04-29T14:02:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:40 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:41.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:02:41.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:02:41.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:02:41.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:40 smithi040 ceph-mon[39581]: osd.0 v1:172.21.15.40:6801/4255137508 boot 2024-04-29T14:02:41.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:40 smithi040 ceph-mon[39581]: osdmap e8: 1 total, 1 up, 1 in 2024-04-29T14:02:41.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:41.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:02:41.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:02:41.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:02:41.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:40 smithi121 ceph-mon[44533]: osd.0 v1:172.21.15.40:6801/4255137508 boot 2024-04-29T14:02:41.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:40 smithi121 ceph-mon[44533]: osdmap e8: 1 total, 1 up, 1 in 2024-04-29T14:02:41.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:40 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-29T14:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-29T14:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-29T14:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-29T14:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-29T14:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0146729 s, 715 MB/s 2024-04-29T14:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-29T14:02:42.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:41 smithi040 ceph-mon[39581]: osdmap e9: 1 total, 1 up, 1 in 2024-04-29T14:02:42.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:41 smithi040 ceph-mon[39581]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:42.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:41 smithi194 ceph-mon[44089]: osdmap e9: 1 total, 1 up, 1 in 2024-04-29T14:02:42.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:41 smithi194 ceph-mon[44089]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:42.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:41 smithi121 ceph-mon[44533]: osdmap e9: 1 total, 1 up, 1 in 2024-04-29T14:02:42.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:41 smithi121 ceph-mon[44533]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:42.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2024-04-29T14:02:43.423 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:02:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi040 2024-04-29T14:02:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:02:44.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi040 to 17126M 2024-04-29T14:02:44.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:44.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:44.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:44.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi040 2024-04-29T14:02:44.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:02:44.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi040 to 17126M 2024-04-29T14:02:44.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:44.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:44.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi040 2024-04-29T14:02:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:02:44.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi040 to 17126M 2024-04-29T14:02:44.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:44.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:44.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:02:44.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:02:45.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:45 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:02:45.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:45 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:02:45.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:45 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:45.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:02:45.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:02:45.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:45.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:45 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:02:45.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:45 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:02:45.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:45 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:46.766 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:46 smithi040 ceph-mon[39581]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:46.766 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:46 smithi040 ceph-mon[39581]: from='client.14247 v1:172.21.15.40:0/273872371' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:46.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:46 smithi194 ceph-mon[44089]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:46.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:46 smithi194 ceph-mon[44089]: from='client.14247 v1:172.21.15.40:0/273872371' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:46.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:46 smithi121 ceph-mon[44533]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:46.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:46 smithi121 ceph-mon[44533]: from='client.14247 v1:172.21.15.40:0/273872371' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:02:48.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:48 smithi194 ceph-mon[44089]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:48.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/677964345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f52729c-855b-465c-b134-8326f99f1422"}]: dispatch 2024-04-29T14:02:48.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/677964345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f52729c-855b-465c-b134-8326f99f1422"}]': finished 2024-04-29T14:02:48.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:48 smithi194 ceph-mon[44089]: osdmap e10: 2 total, 1 up, 2 in 2024-04-29T14:02:48.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:02:48.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:48 smithi040 ceph-mon[39581]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:48.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/677964345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f52729c-855b-465c-b134-8326f99f1422"}]: dispatch 2024-04-29T14:02:48.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/677964345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f52729c-855b-465c-b134-8326f99f1422"}]': finished 2024-04-29T14:02:48.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:48 smithi040 ceph-mon[39581]: osdmap e10: 2 total, 1 up, 2 in 2024-04-29T14:02:48.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:02:48.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:48 smithi121 ceph-mon[44533]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:48.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/677964345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f52729c-855b-465c-b134-8326f99f1422"}]: dispatch 2024-04-29T14:02:48.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/677964345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f52729c-855b-465c-b134-8326f99f1422"}]': finished 2024-04-29T14:02:48.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:48 smithi121 ceph-mon[44533]: osdmap e10: 2 total, 1 up, 2 in 2024-04-29T14:02:48.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:02:49.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1395812636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:02:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1395812636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:02:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1395812636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:02:50.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:50 smithi194 ceph-mon[44089]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:50.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:50 smithi040 ceph-mon[39581]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:50.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:50 smithi121 ceph-mon[44533]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:52.806 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:52 smithi040 ceph-mon[39581]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:52.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:52 smithi194 ceph-mon[44089]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:52.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:52 smithi121 ceph-mon[44533]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:54.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:54 smithi040 ceph-mon[39581]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:54.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-29T14:02:54.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:54 smithi194 ceph-mon[44089]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-29T14:02:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:54 smithi121 ceph-mon[44533]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-29T14:02:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:02:55.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:55 smithi040 ceph-mon[39581]: Deploying daemon osd.1 on smithi040 2024-04-29T14:02:55.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:55 smithi194 ceph-mon[44089]: Deploying daemon osd.1 on smithi040 2024-04-29T14:02:55.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:55 smithi121 ceph-mon[44533]: Deploying daemon osd.1 on smithi040 2024-04-29T14:02:56.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:56 smithi040 ceph-mon[39581]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:56.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:56 smithi194 ceph-mon[44089]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:56.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:56 smithi121 ceph-mon[44533]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:59.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:02:58 smithi194 ceph-mon[44089]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:59.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:02:58 smithi040 ceph-mon[39581]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:02:59.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:02:58 smithi121 ceph-mon[44533]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:00.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:00 smithi040 ceph-mon[39581]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:00.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:00.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:00.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:00.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:00 smithi121 ceph-mon[44533]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:00.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:00.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:00.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:00.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:00 smithi194 ceph-mon[44089]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:00.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:00.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:00.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:02.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:02.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:02.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:02 smithi040 ceph-mon[39581]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:02 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:02 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:02 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:02 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:02 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:02 smithi194 ceph-mon[44089]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:02 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:02 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:02 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:02 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:02 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:02 smithi121 ceph-mon[44533]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:02.779 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 1 on host 'smithi040' 2024-04-29T14:03:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:03 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-29T14:03:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:03.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:03 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-29T14:03:03.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:03.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:03.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:03 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-29T14:03:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:03.639 DEBUG:teuthology.orchestra.run.smithi040:osd.1> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.1.service 2024-04-29T14:03:03.641 INFO:tasks.cephadm:Deploying osd.2 on smithi040 with /dev/vg_nvme/lv_2... 2024-04-29T14:03:03.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-29T14:03:03.674 INFO:journalctl@ceph.osd.1.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T14:03:04.334 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:03:04.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:04 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-29T14:03:04.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:04 smithi040 ceph-mon[39581]: osdmap e11: 2 total, 1 up, 2 in 2024-04-29T14:03:04.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:04 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:04.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:04.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:04 smithi040 ceph-mon[39581]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:04.365 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:04 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[66356]: 2024-04-29T14:03:04.325+0000 7fce47a86700 -1 osd.1 0 waiting for initial osdmap 2024-04-29T14:03:04.365 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:04 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[66356]: 2024-04-29T14:03:04.331+0000 7fce43879700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:03:04.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:04 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-29T14:03:04.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:04 smithi121 ceph-mon[44533]: osdmap e11: 2 total, 1 up, 2 in 2024-04-29T14:03:04.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:04 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:04.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:04.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:04 smithi121 ceph-mon[44533]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:04.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:04 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-29T14:03:04.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:04 smithi194 ceph-mon[44089]: osdmap e11: 2 total, 1 up, 2 in 2024-04-29T14:03:04.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:04 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:04.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:04.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:04 smithi194 ceph-mon[44089]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:05.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:05 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:05.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:05 smithi040 ceph-mon[39581]: osdmap e12: 2 total, 1 up, 2 in 2024-04-29T14:03:05.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:05.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:05.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:05 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:05.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:05 smithi121 ceph-mon[44533]: osdmap e12: 2 total, 1 up, 2 in 2024-04-29T14:03:05.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:05.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:05.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:05 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/4178456629' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:05.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:05 smithi194 ceph-mon[44089]: osdmap e12: 2 total, 1 up, 2 in 2024-04-29T14:03:05.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:05.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: osd.1 v1:172.21.15.40:6805/4178456629 boot 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: osdmap e13: 2 total, 2 up, 2 in 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: osd.1 v1:172.21.15.40:6805/4178456629 boot 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: osdmap e13: 2 total, 2 up, 2 in 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: osd.1 v1:172.21.15.40:6805/4178456629 boot 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: osdmap e13: 2 total, 2 up, 2 in 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T14:03:06.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: osdmap e14: 2 total, 2 up, 2 in 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:07.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71081]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71081]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71081]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71081]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71091]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71091]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71091]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.506 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71091]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.507 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71094]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-29T14:03:07.507 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71094]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.507 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71094]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.507 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71094]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.507 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71097]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-29T14:03:07.507 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71097]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.586 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: osdmap e14: 2 total, 2 up, 2 in 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46442]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46442]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46442]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.587 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46442]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.588 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46445]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: osdmap e14: 2 total, 2 up, 2 in 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:07.623 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:07.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:07.865 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71097]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.865 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71097]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71100]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-29T14:03:07.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71100]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71100]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71100]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71103]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-29T14:03:07.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71103]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:07 smithi040 sudo[71103]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46895]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46895]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46895]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46895]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46898]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46898]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46898]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:07.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:07 smithi121 sudo[46898]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:07.918 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-29T14:03:07.918 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-29T14:03:07.918 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-29T14:03:07.919 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-29T14:03:07.919 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108464 s, 967 MB/s 2024-04-29T14:03:07.919 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-29T14:03:08.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46445]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-29T14:03:08.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46445]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-29T14:03:08.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:07 smithi194 sudo[46445]: pam_unix(sudo:session): session closed for user root 2024-04-29T14:03:08.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi040 2024-04-29T14:03:08.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi040 to 8563M 2024-04-29T14:03:08.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:08.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: osdmap e15: 2 total, 2 up, 2 in 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:08 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:03:08.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2024-04-29T14:03:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi040 2024-04-29T14:03:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi040 to 8563M 2024-04-29T14:03:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:08.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: osdmap e15: 2 total, 2 up, 2 in 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:08 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:03:08.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi040 2024-04-29T14:03:08.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi040 to 8563M 2024-04-29T14:03:08.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:08.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:03:08.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: osdmap e15: 2 total, 2 up, 2 in 2024-04-29T14:03:08.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-29T14:03:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:08 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:03:08.864 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:03:09.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:09 smithi040 ceph-mon[39581]: osdmap e16: 2 total, 2 up, 2 in 2024-04-29T14:03:09.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:09 smithi040 ceph-mon[39581]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-29T14:03:09.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:09 smithi121 ceph-mon[44533]: osdmap e16: 2 total, 2 up, 2 in 2024-04-29T14:03:09.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:09 smithi121 ceph-mon[44533]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-29T14:03:09.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:09 smithi194 ceph-mon[44089]: osdmap e16: 2 total, 2 up, 2 in 2024-04-29T14:03:09.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:09 smithi194 ceph-mon[44089]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-29T14:03:10.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:10 smithi040 ceph-mon[39581]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:10.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:10 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:03:10.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:10 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:03:10.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:10 smithi121 ceph-mon[44533]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:10.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:10 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:03:10.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:10 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:03:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:10 smithi194 ceph-mon[44089]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:10 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:03:10.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:10 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:03:11.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:11 smithi040 ceph-mon[39581]: from='client.14283 v1:172.21.15.40:0/2597560904' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:03:11.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:03:11.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:03:11.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:11 smithi121 ceph-mon[44533]: from='client.14283 v1:172.21.15.40:0/2597560904' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:03:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:03:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:03:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:11.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:11 smithi194 ceph-mon[44089]: from='client.14283 v1:172.21.15.40:0/2597560904' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:03:11.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:03:11.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:03:11.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:12 smithi040 ceph-mon[39581]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:12.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:12 smithi121 ceph-mon[44533]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:12.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:12 smithi194 ceph-mon[44089]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:13.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/25313853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3"}]: dispatch 2024-04-29T14:03:13.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/25313853' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3"}]': finished 2024-04-29T14:03:13.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:13 smithi040 ceph-mon[39581]: osdmap e17: 3 total, 2 up, 3 in 2024-04-29T14:03:13.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:13.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/25313853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3"}]: dispatch 2024-04-29T14:03:13.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/25313853' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3"}]': finished 2024-04-29T14:03:13.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:13 smithi121 ceph-mon[44533]: osdmap e17: 3 total, 2 up, 3 in 2024-04-29T14:03:13.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:13.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/25313853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3"}]: dispatch 2024-04-29T14:03:13.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/25313853' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3"}]': finished 2024-04-29T14:03:13.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:13 smithi194 ceph-mon[44089]: osdmap e17: 3 total, 2 up, 3 in 2024-04-29T14:03:13.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:14.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:14 smithi040 ceph-mon[39581]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:14.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2654247553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:03:14.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:14 smithi121 ceph-mon[44533]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:14.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2654247553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:03:14.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:14 smithi194 ceph-mon[44089]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:14.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2654247553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:03:16.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:16 smithi040 ceph-mon[39581]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:16 smithi121 ceph-mon[44533]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:16.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:16 smithi194 ceph-mon[44089]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:18.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:18 smithi040 ceph-mon[39581]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:19.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:18 smithi194 ceph-mon[44089]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:19.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:18 smithi121 ceph-mon[44533]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:20.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-29T14:03:20.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:20.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-29T14:03:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:20.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-29T14:03:20.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:21.062 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:20 smithi040 ceph-mon[39581]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:21.062 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:20 smithi040 ceph-mon[39581]: Deploying daemon osd.2 on smithi040 2024-04-29T14:03:21.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:20 smithi194 ceph-mon[44089]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:21.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:20 smithi194 ceph-mon[44089]: Deploying daemon osd.2 on smithi040 2024-04-29T14:03:21.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:20 smithi121 ceph-mon[44533]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:21.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:20 smithi121 ceph-mon[44533]: Deploying daemon osd.2 on smithi040 2024-04-29T14:03:23.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:22 smithi194 ceph-mon[44089]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:23.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:22 smithi040 ceph-mon[39581]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:23.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:22 smithi121 ceph-mon[44533]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:25.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:24 smithi194 ceph-mon[44089]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:25.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:24 smithi040 ceph-mon[39581]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:25.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:24 smithi121 ceph-mon[44533]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:26.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:26.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:26.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:26.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:26.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:26.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:26.989 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:26 smithi040 ceph-mon[39581]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:27.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:26 smithi194 ceph-mon[44089]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:27.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:26 smithi121 ceph-mon[44533]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:29.018 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 2 on host 'smithi040' 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: from='osd.2 v1:172.21.15.40:6809/3368423938' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: from='osd.2 v1:172.21.15.40:6809/3368423938' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:03:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:03:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: from='osd.2 v1:172.21.15.40:6809/3368423938' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:29.689 DEBUG:teuthology.orchestra.run.smithi040:osd.2> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2.service 2024-04-29T14:03:29.691 INFO:tasks.cephadm:Deploying osd.3 on smithi040 with /dev/vg_nvme/lv_1... 2024-04-29T14:03:29.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-29T14:03:29.723 INFO:journalctl@ceph.osd.2.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='osd.2 v1:172.21.15.40:6809/3368423938' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: osdmap e18: 3 total, 2 up, 3 in 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:30.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:30.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-29T14:03:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='osd.2 v1:172.21.15.40:6809/3368423938' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: osdmap e18: 3 total, 2 up, 3 in 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:30.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='osd.2 v1:172.21.15.40:6809/3368423938' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: osdmap e18: 3 total, 2 up, 3 in 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:30.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:30.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:30.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:30.865 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[73654]: 2024-04-29T14:03:30.416+0000 7f8fd22fb700 -1 osd.2 0 waiting for initial osdmap 2024-04-29T14:03:30.865 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[73654]: 2024-04-29T14:03:30.420+0000 7f8fca0e6700 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:03:30.907 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:03:31.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-mon[39581]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-mon[39581]: osdmap e19: 3 total, 2 up, 3 in 2024-04-29T14:03:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:30 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' 2024-04-29T14:03:31.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:30 smithi194 ceph-mon[44089]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:31.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:30 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:31.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:30 smithi194 ceph-mon[44089]: osdmap e19: 3 total, 2 up, 3 in 2024-04-29T14:03:31.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:31.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:31.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:30 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' 2024-04-29T14:03:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:30 smithi121 ceph-mon[44533]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:30 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:30 smithi121 ceph-mon[44533]: osdmap e19: 3 total, 2 up, 3 in 2024-04-29T14:03:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:30 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' 2024-04-29T14:03:32.082 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:03:32.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:03:32.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:32.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:32.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:32.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: osd.2 v1:172.21.15.40:6809/3368423938 boot 2024-04-29T14:03:32.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: osdmap e20: 3 total, 3 up, 3 in 2024-04-29T14:03:32.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: osd.2 v1:172.21.15.40:6809/3368423938 boot 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: osdmap e20: 3 total, 3 up, 3 in 2024-04-29T14:03:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: osd.2 v1:172.21.15.40:6809/3368423938 boot 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: osdmap e20: 3 total, 3 up, 3 in 2024-04-29T14:03:32.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:03:33.100 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-29T14:03:33.100 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-29T14:03:33.100 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-29T14:03:33.101 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-29T14:03:33.101 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126548 s, 829 MB/s 2024-04-29T14:03:33.101 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-29T14:03:33.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:32 smithi194 ceph-mon[44089]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:32 smithi040 ceph-mon[39581]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:32 smithi121 ceph-mon[44533]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-04-29T14:03:34.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:33 smithi040 ceph-mon[39581]: osdmap e21: 3 total, 3 up, 3 in 2024-04-29T14:03:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:33 smithi194 ceph-mon[44089]: osdmap e21: 3 total, 3 up, 3 in 2024-04-29T14:03:34.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2024-04-29T14:03:34.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:33 smithi121 ceph-mon[44533]: osdmap e21: 3 total, 3 up, 3 in 2024-04-29T14:03:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi040 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi040 to 5708M 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi040 2024-04-29T14:03:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi040 to 5708M 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi040 2024-04-29T14:03:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:35.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi040 to 5708M 2024-04-29T14:03:35.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:35.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:35.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:35.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:36.227 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:35 smithi040 ceph-mon[39581]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:36.227 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:36.227 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:36.227 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:36.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:35 smithi194 ceph-mon[44089]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:36.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:36.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:36.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:35 smithi121 ceph-mon[44533]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:37.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:36 smithi040 ceph-mon[39581]: from='client.14310 v1:172.21.15.40:0/2233693610' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:03:37.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:03:37.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:03:37.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:36 smithi194 ceph-mon[44089]: from='client.14310 v1:172.21.15.40:0/2233693610' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:03:37.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:03:37.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:03:37.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:37.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:36 smithi121 ceph-mon[44533]: from='client.14310 v1:172.21.15.40:0/2233693610' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:03:37.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:03:37.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:03:37.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:38.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:37 smithi194 ceph-mon[44089]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:38.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:37 smithi040 ceph-mon[39581]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:38.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:37 smithi121 ceph-mon[44533]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3961030537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765"}]: dispatch 2024-04-29T14:03:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3961030537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765"}]': finished 2024-04-29T14:03:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:39 smithi121 ceph-mon[44533]: osdmap e22: 4 total, 3 up, 4 in 2024-04-29T14:03:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:39.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3961030537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765"}]: dispatch 2024-04-29T14:03:39.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3961030537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765"}]': finished 2024-04-29T14:03:39.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:39 smithi194 ceph-mon[44089]: osdmap e22: 4 total, 3 up, 4 in 2024-04-29T14:03:39.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:39.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3961030537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765"}]: dispatch 2024-04-29T14:03:39.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3961030537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765"}]': finished 2024-04-29T14:03:39.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:39 smithi040 ceph-mon[39581]: osdmap e22: 4 total, 3 up, 4 in 2024-04-29T14:03:39.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:40.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:40 smithi194 ceph-mon[44089]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:40.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4227101006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:03:40.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:40 smithi040 ceph-mon[39581]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:40.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4227101006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:03:40.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:40 smithi121 ceph-mon[44533]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:40.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4227101006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:03:42.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:42 smithi040 ceph-mon[39581]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:42 smithi194 ceph-mon[44089]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:42.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:42 smithi121 ceph-mon[44533]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:44.685 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:44 smithi040 ceph-mon[39581]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:44.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:44 smithi194 ceph-mon[44089]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:44 smithi121 ceph-mon[44533]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:46.677 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:46 smithi040 ceph-mon[39581]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:46.677 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-29T14:03:46.677 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:46.677 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:46 smithi040 ceph-mon[39581]: Deploying daemon osd.3 on smithi040 2024-04-29T14:03:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:46 smithi194 ceph-mon[44089]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:46.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-29T14:03:46.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:46.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:46 smithi194 ceph-mon[44089]: Deploying daemon osd.3 on smithi040 2024-04-29T14:03:46.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:46 smithi121 ceph-mon[44533]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:46.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-29T14:03:46.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:46.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:46 smithi121 ceph-mon[44533]: Deploying daemon osd.3 on smithi040 2024-04-29T14:03:48.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:48 smithi040 ceph-mon[39581]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:48.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:48 smithi194 ceph-mon[44089]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:48.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:48 smithi121 ceph-mon[44533]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:50.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:50 smithi194 ceph-mon[44089]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:50.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:50 smithi040 ceph-mon[39581]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:50.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:50 smithi121 ceph-mon[44533]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:52.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:52.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:52 smithi040 ceph-mon[39581]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:52.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:52.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:52.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:52.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:52 smithi121 ceph-mon[44533]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:52.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:52.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:52.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:52.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:52 smithi194 ceph-mon[44089]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:52.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:52.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:54.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:54 smithi040 ceph-mon[39581]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:54.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:54.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:54.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.741 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 3 on host 'smithi040' 2024-04-29T14:03:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:54 smithi194 ceph-mon[44089]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:54.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:54 smithi121 ceph-mon[44533]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:55.473 DEBUG:teuthology.orchestra.run.smithi040:osd.3> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.3.service 2024-04-29T14:03:55.476 INFO:tasks.cephadm:Deploying osd.4 on smithi121 with /dev/vg_nvme/lv_4... 2024-04-29T14:03:55.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-29T14:03:55.509 INFO:journalctl@ceph.osd.3.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T14:03:55.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:55 smithi040 ceph-mon[39581]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-29T14:03:55.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:55.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:55.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:55.746 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:03:55.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:55 smithi194 ceph-mon[44089]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-29T14:03:55.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:55.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:55.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:55.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:55 smithi121 ceph-mon[44533]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-29T14:03:55.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:55 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:55.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:55 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:55.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:55 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:56 smithi040 ceph-mon[39581]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:56 smithi040 ceph-mon[39581]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-29T14:03:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:56 smithi040 ceph-mon[39581]: osdmap e23: 4 total, 3 up, 4 in 2024-04-29T14:03:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:56 smithi040 ceph-mon[39581]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:56.615 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 29 14:03:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-3[80949]: 2024-04-29T14:03:56.398+0000 7f4508f79700 -1 osd.3 0 waiting for initial osdmap 2024-04-29T14:03:56.615 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 29 14:03:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-3[80949]: 2024-04-29T14:03:56.405+0000 7f4504d6c700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:03:56.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:56 smithi194 ceph-mon[44089]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:56.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:56 smithi194 ceph-mon[44089]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-29T14:03:56.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:56 smithi194 ceph-mon[44089]: osdmap e23: 4 total, 3 up, 4 in 2024-04-29T14:03:56.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:56 smithi194 ceph-mon[44089]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:56.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:56.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:56 smithi121 ceph-mon[44533]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:56.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:56 smithi121 ceph-mon[44533]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-29T14:03:56.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:56 smithi121 ceph-mon[44533]: osdmap e23: 4 total, 3 up, 4 in 2024-04-29T14:03:56.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:56 smithi121 ceph-mon[44533]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:03:56.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:57 smithi121 ceph-mon[44533]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:57 smithi121 ceph-mon[44533]: osdmap e24: 4 total, 3 up, 4 in 2024-04-29T14:03:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:57.732 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-29T14:03:57.732 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-29T14:03:57.732 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-04-29T14:03:57.732 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-04-29T14:03:57.732 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112329 s, 933 MB/s 2024-04-29T14:03:57.732 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-04-29T14:03:57.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:57 smithi194 ceph-mon[44089]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:57.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:57 smithi194 ceph-mon[44089]: osdmap e24: 4 total, 3 up, 4 in 2024-04-29T14:03:57.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:57.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:57.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:57 smithi040 ceph-mon[39581]: from='osd.3 v1:172.21.15.40:6813/4012292460' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-29T14:03:57.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:57 smithi040 ceph-mon[39581]: osdmap e24: 4 total, 3 up, 4 in 2024-04-29T14:03:57.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:57.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:58.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: osd.3 v1:172.21.15.40:6813/4012292460 boot 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: osdmap e25: 4 total, 4 up, 4 in 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:58.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:58.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:58.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:03:58.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: osd.3 v1:172.21.15.40:6813/4012292460 boot 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: osdmap e25: 4 total, 4 up, 4 in 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: osd.3 v1:172.21.15.40:6813/4012292460 boot 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: osdmap e25: 4 total, 4 up, 4 in 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:03:58.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:03:59.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi040 2024-04-29T14:03:59.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi040 to 4281M 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.821 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:03:59 smithi121 ceph-mon[44533]: osdmap e26: 4 total, 4 up, 4 in 2024-04-29T14:03:59.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi040 2024-04-29T14:03:59.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi040 to 4281M 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:03:59.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:03:59 smithi040 ceph-mon[39581]: osdmap e26: 4 total, 4 up, 4 in 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi040 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi040 to 4281M 2024-04-29T14:04:00.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:00.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:03:59 smithi194 ceph-mon[44089]: osdmap e26: 4 total, 4 up, 4 in 2024-04-29T14:04:01.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:00 smithi194 ceph-mon[44089]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:01.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:01.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:01.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:01.070 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:00 smithi121 ceph-mon[44533]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:01.070 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:01.070 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:01.070 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:00 smithi040 ceph-mon[39581]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:02.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:01 smithi121 ceph-mon[44533]: from='client.34229 v1:172.21.15.121:0/3592820951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:02.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:01 smithi040 ceph-mon[39581]: from='client.34229 v1:172.21.15.121:0/3592820951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:02.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:01 smithi194 ceph-mon[44089]: from='client.34229 v1:172.21.15.121:0/3592820951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:03.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:02 smithi040 ceph-mon[39581]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:02 smithi121 ceph-mon[44533]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:03.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:02 smithi194 ceph-mon[44089]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:04.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/2709270913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]: dispatch 2024-04-29T14:04:04.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:03 smithi040 ceph-mon[39581]: from='client.34235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]: dispatch 2024-04-29T14:04:04.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:03 smithi040 ceph-mon[39581]: from='client.34235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]': finished 2024-04-29T14:04:04.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:03 smithi040 ceph-mon[39581]: osdmap e27: 5 total, 4 up, 5 in 2024-04-29T14:04:04.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:04.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/4189692904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:04.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/2709270913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]: dispatch 2024-04-29T14:04:04.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:03 smithi121 ceph-mon[44533]: from='client.34235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]: dispatch 2024-04-29T14:04:04.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:03 smithi121 ceph-mon[44533]: from='client.34235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]': finished 2024-04-29T14:04:04.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:03 smithi121 ceph-mon[44533]: osdmap e27: 5 total, 4 up, 5 in 2024-04-29T14:04:04.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:04.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/4189692904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:04.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/2709270913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]: dispatch 2024-04-29T14:04:04.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:03 smithi194 ceph-mon[44089]: from='client.34235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]: dispatch 2024-04-29T14:04:04.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:03 smithi194 ceph-mon[44089]: from='client.34235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6aa13ecc-0959-4915-a788-03c4739da269"}]': finished 2024-04-29T14:04:04.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:03 smithi194 ceph-mon[44089]: osdmap e27: 5 total, 4 up, 5 in 2024-04-29T14:04:04.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:04.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/4189692904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:05.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:04 smithi040 ceph-mon[39581]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:05.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:04:05.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:04:05.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:04 smithi040 ceph-mon[39581]: osdmap e28: 5 total, 4 up, 5 in 2024-04-29T14:04:05.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:05.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:04 smithi121 ceph-mon[44533]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:05.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:04:05.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:04:05.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:04 smithi121 ceph-mon[44533]: osdmap e28: 5 total, 4 up, 5 in 2024-04-29T14:04:05.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:04 smithi194 ceph-mon[44089]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:04:05.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:04:05.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:04 smithi194 ceph-mon[44089]: osdmap e28: 5 total, 4 up, 5 in 2024-04-29T14:04:05.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:06 smithi040 ceph-mon[39581]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:07.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:06 smithi121 ceph-mon[44533]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:07.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:06 smithi194 ceph-mon[44089]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:09.008 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:08 smithi121 ceph-mon[44533]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:09.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:08 smithi040 ceph-mon[39581]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:08 smithi194 ceph-mon[44089]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:10.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-29T14:04:10.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:10.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-29T14:04:10.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:10.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-29T14:04:10.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:11.107 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:10 smithi121 ceph-mon[44533]: Deploying daemon osd.4 on smithi121 2024-04-29T14:04:11.109 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:10 smithi121 ceph-mon[44533]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:11.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:10 smithi040 ceph-mon[39581]: Deploying daemon osd.4 on smithi121 2024-04-29T14:04:11.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:10 smithi040 ceph-mon[39581]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:11.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:10 smithi194 ceph-mon[44089]: Deploying daemon osd.4 on smithi121 2024-04-29T14:04:11.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:10 smithi194 ceph-mon[44089]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:12 smithi040 ceph-mon[39581]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:13.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:12 smithi121 ceph-mon[44533]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:12 smithi194 ceph-mon[44089]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:15.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:14 smithi121 ceph-mon[44533]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:15.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:15.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:15.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:15.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:14 smithi040 ceph-mon[39581]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:15.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:15.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:15.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:15.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:14 smithi194 ceph-mon[44089]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:15.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:15.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:15.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:16 smithi040 ceph-mon[39581]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:17.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:17.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:17.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:16 smithi121 ceph-mon[44533]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:17.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:17.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:17.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:16 smithi194 ceph-mon[44089]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:17.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:17.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:17.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:17.550 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 4 on host 'smithi121' 2024-04-29T14:04:18.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='osd.4 v1:172.21.15.121:6800/3987960551' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='osd.4 v1:172.21.15.121:6800/3987960551' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:18.214 DEBUG:teuthology.orchestra.run.smithi121:osd.4> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.4.service 2024-04-29T14:04:18.216 INFO:tasks.cephadm:Deploying osd.5 on smithi121 with /dev/vg_nvme/lv_3... 2024-04-29T14:04:18.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-29T14:04:18.248 INFO:journalctl@ceph.osd.4.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='osd.4 v1:172.21.15.121:6800/3987960551' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:18.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:04:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:18 smithi040 ceph-mon[39581]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-29T14:04:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:18 smithi040 ceph-mon[39581]: from='osd.4 v1:172.21.15.121:6800/3987960551' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:18 smithi040 ceph-mon[39581]: osdmap e29: 5 total, 4 up, 5 in 2024-04-29T14:04:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:18 smithi040 ceph-mon[39581]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:18 smithi040 ceph-mon[39581]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:04:19.127 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4[49952]: 2024-04-29T14:04:18.857+0000 7f07a4ffd700 -1 osd.4 0 waiting for initial osdmap 2024-04-29T14:04:19.127 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4[49952]: 2024-04-29T14:04:18.862+0000 7f07a0df0700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:04:19.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-mon[44533]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-29T14:04:19.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-mon[44533]: from='osd.4 v1:172.21.15.121:6800/3987960551' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:19.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-mon[44533]: osdmap e29: 5 total, 4 up, 5 in 2024-04-29T14:04:19.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:19.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-mon[44533]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:19.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:18 smithi121 ceph-mon[44533]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:04:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:18 smithi194 ceph-mon[44089]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-29T14:04:19.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:18 smithi194 ceph-mon[44089]: from='osd.4 v1:172.21.15.121:6800/3987960551' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:19.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:18 smithi194 ceph-mon[44089]: osdmap e29: 5 total, 4 up, 5 in 2024-04-29T14:04:19.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:19.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:18 smithi194 ceph-mon[44089]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:19.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:18 smithi194 ceph-mon[44089]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:04:20.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:19 smithi040 ceph-mon[39581]: osdmap e30: 5 total, 4 up, 5 in 2024-04-29T14:04:20.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:20.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:04:20.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:04:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:20.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:19 smithi121 ceph-mon[44533]: osdmap e30: 5 total, 4 up, 5 in 2024-04-29T14:04:20.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:20.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:04:20.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:04:20.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:20.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:19 smithi194 ceph-mon[44089]: osdmap e30: 5 total, 4 up, 5 in 2024-04-29T14:04:20.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:20.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:04:20.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:04:20.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:21.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:04:21.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:04:21.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:04:21.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:20 smithi121 ceph-mon[44533]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:21.085 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:20 smithi121 ceph-mon[44533]: osd.4 v1:172.21.15.121:6800/3987960551 boot 2024-04-29T14:04:21.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:20 smithi121 ceph-mon[44533]: osdmap e31: 5 total, 5 up, 5 in 2024-04-29T14:04:21.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:21.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:04:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:04:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:04:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:20 smithi194 ceph-mon[44089]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:20 smithi194 ceph-mon[44089]: osd.4 v1:172.21.15.121:6800/3987960551 boot 2024-04-29T14:04:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:20 smithi194 ceph-mon[44089]: osdmap e31: 5 total, 5 up, 5 in 2024-04-29T14:04:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:21.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:04:21.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:04:21.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:04:21.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:20 smithi040 ceph-mon[39581]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-29T14:04:21.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:20 smithi040 ceph-mon[39581]: osd.4 v1:172.21.15.121:6800/3987960551 boot 2024-04-29T14:04:21.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:20 smithi040 ceph-mon[39581]: osdmap e31: 5 total, 5 up, 5 in 2024-04-29T14:04:21.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-29T14:04:21.698 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-29T14:04:21.699 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-29T14:04:21.699 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-04-29T14:04:21.699 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-04-29T14:04:21.699 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110265 s, 951 MB/s 2024-04-29T14:04:21.699 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-04-29T14:04:21.967 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:21 smithi121 ceph-mon[44533]: osdmap e32: 5 total, 5 up, 5 in 2024-04-29T14:04:22.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:21 smithi194 ceph-mon[44089]: osdmap e32: 5 total, 5 up, 5 in 2024-04-29T14:04:22.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:21 smithi040 ceph-mon[39581]: osdmap e32: 5 total, 5 up, 5 in 2024-04-29T14:04:22.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2024-04-29T14:04:22.923 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:04:23.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:23.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:23.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:23.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:23.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:23.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:23.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:23.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:23.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:23.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:24.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:23 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi121 2024-04-29T14:04:24.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:23 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi121 to 17122M 2024-04-29T14:04:24.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:23 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi121 2024-04-29T14:04:24.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:23 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi121 to 17122M 2024-04-29T14:04:24.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:23 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi121 2024-04-29T14:04:24.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:23 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi121 to 17122M 2024-04-29T14:04:25.208 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:24 smithi121 ceph-mon[44533]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:25.208 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:25.209 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:25.209 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:25.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:24 smithi194 ceph-mon[44089]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:25.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:25.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:25.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:25.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:24 smithi040 ceph-mon[39581]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:25.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:25.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:25.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:26.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:25 smithi121 ceph-mon[44533]: from='client.34256 v1:172.21.15.121:0/3718963327' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:25 smithi194 ceph-mon[44089]: from='client.34256 v1:172.21.15.121:0/3718963327' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:26.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:25 smithi040 ceph-mon[39581]: from='client.34256 v1:172.21.15.121:0/3718963327' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:27.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:26 smithi194 ceph-mon[44089]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:27.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:26 smithi040 ceph-mon[39581]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:27.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:26 smithi121 ceph-mon[44533]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:28.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:04:28.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:04:28.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/3761505437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]: dispatch 2024-04-29T14:04:28.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: from='client.34262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]: dispatch 2024-04-29T14:04:28.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: from='client.34262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]': finished 2024-04-29T14:04:28.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: osdmap e33: 6 total, 5 up, 6 in 2024-04-29T14:04:28.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:28.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/2200694269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:04:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:04:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/3761505437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]: dispatch 2024-04-29T14:04:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: from='client.34262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]: dispatch 2024-04-29T14:04:28.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: from='client.34262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]': finished 2024-04-29T14:04:28.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: osdmap e33: 6 total, 5 up, 6 in 2024-04-29T14:04:28.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:28.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/2200694269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/3761505437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]: dispatch 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: from='client.34262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]: dispatch 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: from='client.34262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6"}]': finished 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: osdmap e33: 6 total, 5 up, 6 in 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:28.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/2200694269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:28 smithi194 ceph-mon[44089]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:28 smithi040 ceph-mon[39581]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:28 smithi121 ceph-mon[44533]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:31.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:30 smithi121 ceph-mon[44533]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:31.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:30 smithi194 ceph-mon[44089]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:31.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:30 smithi040 ceph-mon[39581]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:32 smithi121 ceph-mon[44533]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:33.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:32 smithi194 ceph-mon[44089]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:32 smithi040 ceph-mon[39581]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-29T14:04:34.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-29T14:04:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-29T14:04:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:34 smithi194 ceph-mon[44089]: Deploying daemon osd.5 on smithi121 2024-04-29T14:04:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:34 smithi194 ceph-mon[44089]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:34 smithi040 ceph-mon[39581]: Deploying daemon osd.5 on smithi121 2024-04-29T14:04:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:34 smithi040 ceph-mon[39581]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:34 smithi121 ceph-mon[44533]: Deploying daemon osd.5 on smithi121 2024-04-29T14:04:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:34 smithi121 ceph-mon[44533]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:37.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:36 smithi121 ceph-mon[44533]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:36 smithi194 ceph-mon[44089]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:37.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:36 smithi040 ceph-mon[39581]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:39.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:38 smithi121 ceph-mon[44533]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:39.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:39.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:39.178 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:39.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:38 smithi194 ceph-mon[44089]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:39.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:39.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:39.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:38 smithi040 ceph-mon[39581]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:41.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:40 smithi121 ceph-mon[44533]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:40 smithi194 ceph-mon[44089]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:41.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:40 smithi040 ceph-mon[39581]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:42.331 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 5 on host 'smithi121' 2024-04-29T14:04:42.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='osd.5 v1:172.21.15.121:6804/3807513901' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:42.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='osd.5 v1:172.21.15.121:6804/3807513901' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:42.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='osd.5 v1:172.21.15.121:6804/3807513901' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:43.009 DEBUG:teuthology.orchestra.run.smithi121:osd.5> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5.service 2024-04-29T14:04:43.011 INFO:tasks.cephadm:Deploying osd.6 on smithi121 with /dev/vg_nvme/lv_2... 2024-04-29T14:04:43.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-29T14:04:43.042 INFO:journalctl@ceph.osd.5.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:04:44.049 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:43 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:04:44.049 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:43 smithi121 ceph-mon[44533]: from='osd.5 v1:172.21.15.121:6804/3807513901' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:44.049 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:43 smithi121 ceph-mon[44533]: osdmap e34: 6 total, 5 up, 6 in 2024-04-29T14:04:44.049 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:44.049 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:43 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:44.050 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:04:43 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[57024]: 2024-04-29T14:04:43.769+0000 7f7e335f1700 -1 osd.5 0 waiting for initial osdmap 2024-04-29T14:04:44.050 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:04:43 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[57024]: 2024-04-29T14:04:43.774+0000 7f7e2f3e4700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:04:44.050 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:04:44.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:43 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:04:44.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:43 smithi194 ceph-mon[44089]: from='osd.5 v1:172.21.15.121:6804/3807513901' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:44.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:43 smithi194 ceph-mon[44089]: osdmap e34: 6 total, 5 up, 6 in 2024-04-29T14:04:44.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:44.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:43 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:43 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:04:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:43 smithi040 ceph-mon[39581]: from='osd.5 v1:172.21.15.121:6804/3807513901' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:43 smithi040 ceph-mon[39581]: osdmap e34: 6 total, 5 up, 6 in 2024-04-29T14:04:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:43 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:04:44.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: osdmap e35: 6 total, 5 up, 6 in 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: osd.5 v1:172.21.15.121:6804/3807513901 boot 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: osdmap e36: 6 total, 6 up, 6 in 2024-04-29T14:04:44.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:45.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:04:45.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:04:45.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:45.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:04:45.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: osdmap e35: 6 total, 5 up, 6 in 2024-04-29T14:04:45.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:04:45.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:45.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:45.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: osd.5 v1:172.21.15.121:6804/3807513901 boot 2024-04-29T14:04:45.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: osdmap e36: 6 total, 6 up, 6 in 2024-04-29T14:04:45.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:45.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:04:45.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:04:45.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-04-29T14:04:45.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:04:45.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: osdmap e35: 6 total, 5 up, 6 in 2024-04-29T14:04:45.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:04:45.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:45.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:45.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: osd.5 v1:172.21.15.121:6804/3807513901 boot 2024-04-29T14:04:45.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: osdmap e36: 6 total, 6 up, 6 in 2024-04-29T14:04:45.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:04:47.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:47.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: osdmap e37: 6 total, 6 up, 6 in 2024-04-29T14:04:47.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:47.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:47.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:47.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:47.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-29T14:04:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-29T14:04:47.088 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-04-29T14:04:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-04-29T14:04:47.088 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110895 s, 946 MB/s 2024-04-29T14:04:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-04-29T14:04:47.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:47.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: osdmap e37: 6 total, 6 up, 6 in 2024-04-29T14:04:47.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:47.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:47.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: osdmap e37: 6 total, 6 up, 6 in 2024-04-29T14:04:47.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:47.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:04:47.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:47.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:47.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:47.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2024-04-29T14:04:48.012 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:04:48.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:47 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi121 2024-04-29T14:04:48.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:47 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi121 to 8561M 2024-04-29T14:04:48.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:47 smithi194 ceph-mon[44089]: osdmap e38: 6 total, 6 up, 6 in 2024-04-29T14:04:48.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:47 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi121 2024-04-29T14:04:48.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:47 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi121 to 8561M 2024-04-29T14:04:48.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:47 smithi040 ceph-mon[39581]: osdmap e38: 6 total, 6 up, 6 in 2024-04-29T14:04:48.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:47 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi121 2024-04-29T14:04:48.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:47 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi121 to 8561M 2024-04-29T14:04:48.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:47 smithi121 ceph-mon[44533]: osdmap e38: 6 total, 6 up, 6 in 2024-04-29T14:04:49.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:48 smithi194 ceph-mon[44089]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:49.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:48 smithi194 ceph-mon[44089]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-29T14:04:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:48 smithi040 ceph-mon[39581]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:48 smithi040 ceph-mon[39581]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-29T14:04:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:48 smithi121 ceph-mon[44533]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:48 smithi121 ceph-mon[44533]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-29T14:04:50.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: osdmap e39: 6 total, 6 up, 6 in 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:50.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:49 smithi194 ceph-mon[44089]: osdmap e40: 6 total, 6 up, 6 in 2024-04-29T14:04:50.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: osdmap e39: 6 total, 6 up, 6 in 2024-04-29T14:04:50.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:04:50.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:04:50.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:04:50.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:50.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:50.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:50.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:50.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:49 smithi040 ceph-mon[39581]: osdmap e40: 6 total, 6 up, 6 in 2024-04-29T14:04:50.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: osdmap e39: 6 total, 6 up, 6 in 2024-04-29T14:04:50.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:04:50.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:04:50.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:04:50.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:04:50.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:50.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:04:50.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:04:50.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:49 smithi121 ceph-mon[44533]: osdmap e40: 6 total, 6 up, 6 in 2024-04-29T14:04:51.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:50 smithi040 ceph-mon[39581]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:50 smithi040 ceph-mon[39581]: from='client.34283 v1:172.21.15.121:0/1035808075' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:51.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:50 smithi121 ceph-mon[44533]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:51.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:50 smithi121 ceph-mon[44533]: from='client.34283 v1:172.21.15.121:0/1035808075' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:51.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:51.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:51.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:50 smithi194 ceph-mon[44089]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:51.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:50 smithi194 ceph-mon[44089]: from='client.34283 v1:172.21.15.121:0/1035808075' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:04:51.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:04:51.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:04:51.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:52.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:51 smithi194 ceph-mon[44089]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-29T14:04:52.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:51 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:04:52.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:51 smithi040 ceph-mon[39581]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-29T14:04:52.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:51 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:04:52.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:51 smithi121 ceph-mon[44533]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-29T14:04:52.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:51 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:04:53.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:52 smithi194 ceph-mon[44089]: pgmap v145: 1 pgs: 1 activating; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:53.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/4107378806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]: dispatch 2024-04-29T14:04:53.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:52 smithi194 ceph-mon[44089]: from='client.34289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]: dispatch 2024-04-29T14:04:53.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:52 smithi194 ceph-mon[44089]: from='client.34289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]': finished 2024-04-29T14:04:53.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:52 smithi194 ceph-mon[44089]: osdmap e41: 7 total, 6 up, 7 in 2024-04-29T14:04:53.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:04:53.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:52 smithi040 ceph-mon[39581]: pgmap v145: 1 pgs: 1 activating; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:53.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/4107378806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]: dispatch 2024-04-29T14:04:53.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:52 smithi040 ceph-mon[39581]: from='client.34289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]: dispatch 2024-04-29T14:04:53.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:52 smithi040 ceph-mon[39581]: from='client.34289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]': finished 2024-04-29T14:04:53.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:52 smithi040 ceph-mon[39581]: osdmap e41: 7 total, 6 up, 7 in 2024-04-29T14:04:53.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:04:53.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:52 smithi121 ceph-mon[44533]: pgmap v145: 1 pgs: 1 activating; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:53.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/4107378806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]: dispatch 2024-04-29T14:04:53.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:52 smithi121 ceph-mon[44533]: from='client.34289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]: dispatch 2024-04-29T14:04:53.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:52 smithi121 ceph-mon[44533]: from='client.34289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf"}]': finished 2024-04-29T14:04:53.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:52 smithi121 ceph-mon[44533]: osdmap e41: 7 total, 6 up, 7 in 2024-04-29T14:04:53.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:04:54.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/2042021963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/2042021963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:54.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/2042021963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:04:55.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:54 smithi194 ceph-mon[44089]: pgmap v147: 1 pgs: 1 activating; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:55.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:54 smithi040 ceph-mon[39581]: pgmap v147: 1 pgs: 1 activating; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:55.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:54 smithi121 ceph-mon[44533]: pgmap v147: 1 pgs: 1 activating; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:57.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:56 smithi121 ceph-mon[44533]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:57.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:56 smithi194 ceph-mon[44089]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:57.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:56 smithi040 ceph-mon[39581]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:59.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:58 smithi121 ceph-mon[44533]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:59.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-29T14:04:59.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:59.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:58 smithi194 ceph-mon[44089]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:59.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-29T14:04:59.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:04:59.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:58 smithi040 ceph-mon[39581]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:04:59.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-29T14:04:59.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:00.230 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:00.230 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:00.230 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:59 smithi121 ceph-mon[44533]: Deploying daemon osd.6 on smithi121 2024-04-29T14:05:00.230 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:59 smithi121 ceph-mon[44533]: osdmap e42: 7 total, 6 up, 7 in 2024-04-29T14:05:00.231 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:04:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:00.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:00.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:00.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:59 smithi194 ceph-mon[44089]: Deploying daemon osd.6 on smithi121 2024-04-29T14:05:00.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:59 smithi194 ceph-mon[44089]: osdmap e42: 7 total, 6 up, 7 in 2024-04-29T14:05:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:04:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:59 smithi040 ceph-mon[39581]: Deploying daemon osd.6 on smithi121 2024-04-29T14:05:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:59 smithi040 ceph-mon[39581]: osdmap e42: 7 total, 6 up, 7 in 2024-04-29T14:05:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:04:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:01.273 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:00 smithi121 ceph-mon[44533]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:00 smithi194 ceph-mon[44089]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:01.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:00 smithi040 ceph-mon[39581]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:03.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:02 smithi121 ceph-mon[44533]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:03.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:02 smithi194 ceph-mon[44089]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:03.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:02 smithi040 ceph-mon[39581]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:04 smithi194 ceph-mon[44089]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:05.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:05:05.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:05:05.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:05:05.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:04 smithi040 ceph-mon[39581]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:05.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:05.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:05.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:05.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:05:05.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:05:05.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:05:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:04 smithi121 ceph-mon[44533]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:05:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:05:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:05:07.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:06 smithi121 ceph-mon[44533]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:07.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:07.248 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:07.248 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:06 smithi194 ceph-mon[44089]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:07.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:06 smithi040 ceph-mon[39581]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:07.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:07.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:07.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:07.635 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 6 on host 'smithi121' 2024-04-29T14:05:08.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:08.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:08.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:08.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='osd.6 v1:172.21.15.121:6808/1755659098' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-29T14:05:08.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-29T14:05:08.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='osd.6 v1:172.21.15.121:6808/1755659098' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-29T14:05:08.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-29T14:05:08.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:08.328 DEBUG:teuthology.orchestra.run.smithi121:osd.6> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.6.service 2024-04-29T14:05:08.330 INFO:tasks.cephadm:Deploying osd.7 on smithi121 with /dev/vg_nvme/lv_1... 2024-04-29T14:05:08.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-29T14:05:08.362 INFO:journalctl@ceph.osd.6.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='osd.6 v1:172.21.15.121:6808/1755659098' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:08 smithi194 ceph-mon[44089]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:08 smithi194 ceph-mon[44089]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-29T14:05:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:08 smithi194 ceph-mon[44089]: from='osd.6 v1:172.21.15.121:6808/1755659098' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:08 smithi194 ceph-mon[44089]: osdmap e43: 7 total, 6 up, 7 in 2024-04-29T14:05:09.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:09.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:08 smithi194 ceph-mon[44089]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:08 smithi040 ceph-mon[39581]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:08 smithi040 ceph-mon[39581]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-29T14:05:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:08 smithi040 ceph-mon[39581]: from='osd.6 v1:172.21.15.121:6808/1755659098' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:08 smithi040 ceph-mon[39581]: osdmap e43: 7 total, 6 up, 7 in 2024-04-29T14:05:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:08 smithi040 ceph-mon[39581]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:09.365 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 14:05:09 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T14:05:09.060+0000 7fcb4af8b700 -1 osd.6 0 waiting for initial osdmap 2024-04-29T14:05:09.365 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 14:05:09 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T14:05:09.067+0000 7fcb46d7e700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:05:09.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:08 smithi121 ceph-mon[44533]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:09.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:08 smithi121 ceph-mon[44533]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-29T14:05:09.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:08 smithi121 ceph-mon[44533]: from='osd.6 v1:172.21.15.121:6808/1755659098' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:09.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:08 smithi121 ceph-mon[44533]: osdmap e43: 7 total, 6 up, 7 in 2024-04-29T14:05:09.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:09.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:08 smithi121 ceph-mon[44533]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:05:10.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:10.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:10.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:10 smithi121 ceph-mon[44533]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:05:10.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:10 smithi121 ceph-mon[44533]: osdmap e44: 7 total, 6 up, 7 in 2024-04-29T14:05:10.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:10.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:10.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:10 smithi121 ceph-mon[44533]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:10.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:10.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:10.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:10 smithi194 ceph-mon[44089]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:05:10.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:10 smithi194 ceph-mon[44089]: osdmap e44: 7 total, 6 up, 7 in 2024-04-29T14:05:10.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:10.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:10.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:10 smithi194 ceph-mon[44089]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:10.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:10.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:10.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:10 smithi040 ceph-mon[39581]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:05:10.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:10 smithi040 ceph-mon[39581]: osdmap e44: 7 total, 6 up, 7 in 2024-04-29T14:05:10.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:10.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:10.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:10 smithi040 ceph-mon[39581]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-04-29T14:05:11.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:11 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:11.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:11 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:11.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:11.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:11 smithi121 ceph-mon[44533]: osd.6 v1:172.21.15.121:6808/1755659098 boot 2024-04-29T14:05:11.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:11 smithi121 ceph-mon[44533]: osdmap e45: 7 total, 7 up, 7 in 2024-04-29T14:05:11.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:11.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:11 smithi121 ceph-mon[44533]: osdmap e46: 7 total, 7 up, 7 in 2024-04-29T14:05:11.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:11 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:11.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:11 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:11.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:11.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:11 smithi194 ceph-mon[44089]: osd.6 v1:172.21.15.121:6808/1755659098 boot 2024-04-29T14:05:11.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:11 smithi194 ceph-mon[44089]: osdmap e45: 7 total, 7 up, 7 in 2024-04-29T14:05:11.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:11.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:11 smithi194 ceph-mon[44089]: osdmap e46: 7 total, 7 up, 7 in 2024-04-29T14:05:11.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:11 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:11.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:11 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:11.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:11.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:11 smithi040 ceph-mon[39581]: osd.6 v1:172.21.15.121:6808/1755659098 boot 2024-04-29T14:05:11.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:11 smithi040 ceph-mon[39581]: osdmap e45: 7 total, 7 up, 7 in 2024-04-29T14:05:11.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-29T14:05:11.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:11 smithi040 ceph-mon[39581]: osdmap e46: 7 total, 7 up, 7 in 2024-04-29T14:05:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-29T14:05:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-29T14:05:12.272 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-04-29T14:05:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-04-29T14:05:12.272 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115735 s, 906 MB/s 2024-04-29T14:05:12.272 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-04-29T14:05:12.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: osdmap e47: 7 total, 7 up, 7 in 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:12.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:12.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2024-04-29T14:05:13.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:13.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: osdmap e47: 7 total, 7 up, 7 in 2024-04-29T14:05:13.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:13.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:13.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:13.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:13.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:13.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: osdmap e47: 7 total, 7 up, 7 in 2024-04-29T14:05:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:13.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:13.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:13.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:13.204 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:05:14.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:13 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi121 2024-04-29T14:05:14.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:13 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi121 to 5707M 2024-04-29T14:05:14.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:13 smithi194 ceph-mon[44089]: osdmap e48: 7 total, 7 up, 7 in 2024-04-29T14:05:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:13 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi121 2024-04-29T14:05:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:13 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi121 to 5707M 2024-04-29T14:05:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:13 smithi040 ceph-mon[39581]: osdmap e48: 7 total, 7 up, 7 in 2024-04-29T14:05:14.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:13 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi121 2024-04-29T14:05:14.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:13 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi121 to 5707M 2024-04-29T14:05:14.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:13 smithi121 ceph-mon[44533]: osdmap e48: 7 total, 7 up, 7 in 2024-04-29T14:05:14.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:14 smithi121 ceph-mon[44533]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:14.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:14 smithi121 ceph-mon[44533]: osdmap e49: 7 total, 7 up, 7 in 2024-04-29T14:05:15.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:14 smithi194 ceph-mon[44089]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:15.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:14 smithi194 ceph-mon[44089]: osdmap e49: 7 total, 7 up, 7 in 2024-04-29T14:05:15.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:14 smithi040 ceph-mon[39581]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:15.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:14 smithi040 ceph-mon[39581]: osdmap e49: 7 total, 7 up, 7 in 2024-04-29T14:05:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:05:15.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:05:15.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:16.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:05:16.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:05:16.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:16.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:05:16.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:05:16.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:16 smithi040 ceph-mon[39581]: from='client.34310 v1:172.21.15.121:0/2862370867' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:05:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:16 smithi040 ceph-mon[39581]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:17.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:16 smithi121 ceph-mon[44533]: from='client.34310 v1:172.21.15.121:0/2862370867' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:05:17.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:16 smithi121 ceph-mon[44533]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:16 smithi194 ceph-mon[44089]: from='client.34310 v1:172.21.15.121:0/2862370867' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:05:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:16 smithi194 ceph-mon[44089]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:18.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/3158838758' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]: dispatch 2024-04-29T14:05:18.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:17 smithi040 ceph-mon[39581]: from='client.34316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]: dispatch 2024-04-29T14:05:18.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:17 smithi040 ceph-mon[39581]: from='client.34316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]': finished 2024-04-29T14:05:18.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:17 smithi040 ceph-mon[39581]: osdmap e50: 8 total, 7 up, 8 in 2024-04-29T14:05:18.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/3158838758' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]: dispatch 2024-04-29T14:05:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:17 smithi121 ceph-mon[44533]: from='client.34316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]: dispatch 2024-04-29T14:05:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:17 smithi121 ceph-mon[44533]: from='client.34316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]': finished 2024-04-29T14:05:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:17 smithi121 ceph-mon[44533]: osdmap e50: 8 total, 7 up, 8 in 2024-04-29T14:05:18.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:18.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/3158838758' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]: dispatch 2024-04-29T14:05:18.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:17 smithi194 ceph-mon[44089]: from='client.34316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]: dispatch 2024-04-29T14:05:18.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:17 smithi194 ceph-mon[44089]: from='client.34316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27"}]': finished 2024-04-29T14:05:18.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:17 smithi194 ceph-mon[44089]: osdmap e50: 8 total, 7 up, 8 in 2024-04-29T14:05:18.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:18 smithi040 ceph-mon[39581]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/1389913124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:05:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:18 smithi121 ceph-mon[44533]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/1389913124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:05:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:18 smithi194 ceph-mon[44089]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/1389913124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:05:21.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:20 smithi040 ceph-mon[39581]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:21.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:20 smithi121 ceph-mon[44533]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:21.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:20 smithi194 ceph-mon[44089]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:23.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:22 smithi040 ceph-mon[39581]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:23.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:22 smithi121 ceph-mon[44533]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:23.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:22 smithi194 ceph-mon[44089]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:25.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:24 smithi121 ceph-mon[44533]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:25.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-29T14:05:25.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:25.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:24 smithi121 ceph-mon[44533]: Deploying daemon osd.7 on smithi121 2024-04-29T14:05:25.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:25.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:25.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:24 smithi040 ceph-mon[39581]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:25.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-29T14:05:25.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:25.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:24 smithi040 ceph-mon[39581]: Deploying daemon osd.7 on smithi121 2024-04-29T14:05:25.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:25.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:25.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:24 smithi194 ceph-mon[44089]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:25.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-29T14:05:25.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:25.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:24 smithi194 ceph-mon[44089]: Deploying daemon osd.7 on smithi121 2024-04-29T14:05:25.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:25.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:26.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:25 smithi040 ceph-mon[39581]: osdmap e51: 8 total, 7 up, 8 in 2024-04-29T14:05:26.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:25 smithi121 ceph-mon[44533]: osdmap e51: 8 total, 7 up, 8 in 2024-04-29T14:05:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:25 smithi194 ceph-mon[44089]: osdmap e51: 8 total, 7 up, 8 in 2024-04-29T14:05:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:26.999 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:26 smithi121 ceph-mon[44533]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:27.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:26 smithi040 ceph-mon[39581]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:27.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:26 smithi194 ceph-mon[44089]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:28 smithi040 ceph-mon[39581]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:28 smithi121 ceph-mon[44533]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:28 smithi194 ceph-mon[44089]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:30 smithi194 ceph-mon[44089]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:30.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:30.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:30.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:30.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:30 smithi040 ceph-mon[39581]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:30 smithi121 ceph-mon[44533]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:32.821 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 7 on host 'smithi121' 2024-04-29T14:05:33.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='osd.7 v1:172.21.15.121:6812/82319362' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:33.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:33.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:33.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:33.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='osd.7 v1:172.21.15.121:6812/82319362' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-29T14:05:33.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-29T14:05:33.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='osd.7 v1:172.21.15.121:6812/82319362' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-29T14:05:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-29T14:05:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:33.560 DEBUG:teuthology.orchestra.run.smithi121:osd.7> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.7.service 2024-04-29T14:05:33.562 INFO:tasks.cephadm:Deploying osd.8 on smithi194 with /dev/vg_nvme/lv_4... 2024-04-29T14:05:33.563 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-29T14:05:33.591 INFO:journalctl@ceph.osd.7.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:05:33.832 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:05:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-mon[44533]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-29T14:05:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-mon[44533]: osdmap e52: 8 total, 7 up, 8 in 2024-04-29T14:05:34.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-mon[44533]: from='osd.7 v1:172.21.15.121:6812/82319362' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:34.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:34.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-mon[44533]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:34.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-mon[44533]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:34 smithi194 ceph-mon[44089]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-29T14:05:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:34 smithi194 ceph-mon[44089]: osdmap e52: 8 total, 7 up, 8 in 2024-04-29T14:05:34.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:34 smithi194 ceph-mon[44089]: from='osd.7 v1:172.21.15.121:6812/82319362' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:34.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:34.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:34 smithi194 ceph-mon[44089]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:34.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:34 smithi194 ceph-mon[44089]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:34 smithi040 ceph-mon[39581]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-29T14:05:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:34 smithi040 ceph-mon[39581]: osdmap e52: 8 total, 7 up, 8 in 2024-04-29T14:05:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:34 smithi040 ceph-mon[39581]: from='osd.7 v1:172.21.15.121:6812/82319362' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:34 smithi040 ceph-mon[39581]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:05:34.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:34 smithi040 ceph-mon[39581]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-29T14:05:34.750 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T14:05:34.457+0000 7f9e0189f700 -1 osd.7 0 waiting for initial osdmap 2024-04-29T14:05:34.751 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 14:05:34 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T14:05:34.465+0000 7f9df968a700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:05:35.273 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:35 smithi194 ceph-mon[44089]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:05:35.274 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:35 smithi194 ceph-mon[44089]: osdmap e53: 8 total, 7 up, 8 in 2024-04-29T14:05:35.274 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:35.274 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:35.274 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:35 smithi194 ceph-mon[44089]: from='osd.7 ' entity='osd.7' 2024-04-29T14:05:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:35 smithi040 ceph-mon[39581]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:05:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:35 smithi040 ceph-mon[39581]: osdmap e53: 8 total, 7 up, 8 in 2024-04-29T14:05:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:35.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:35 smithi040 ceph-mon[39581]: from='osd.7 ' entity='osd.7' 2024-04-29T14:05:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:35 smithi121 ceph-mon[44533]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-29T14:05:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:35 smithi121 ceph-mon[44533]: osdmap e53: 8 total, 7 up, 8 in 2024-04-29T14:05:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:35 smithi121 ceph-mon[44533]: from='osd.7 ' entity='osd.7' 2024-04-29T14:05:35.870 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-29T14:05:35.870 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-29T14:05:35.870 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-29T14:05:35.870 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-29T14:05:35.870 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110917 s, 945 MB/s 2024-04-29T14:05:35.870 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-29T14:05:36.183 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: osd.7 v1:172.21.15.121:6812/82319362 boot 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: osdmap e54: 8 total, 8 up, 8 in 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:36.184 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:36 smithi194 ceph-mon[44089]: osdmap e55: 8 total, 8 up, 8 in 2024-04-29T14:05:36.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:36.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:36.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:36.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:36.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: osd.7 v1:172.21.15.121:6812/82319362 boot 2024-04-29T14:05:36.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: osdmap e54: 8 total, 8 up, 8 in 2024-04-29T14:05:36.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:36.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:36.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:36 smithi040 ceph-mon[39581]: osdmap e55: 8 total, 8 up, 8 in 2024-04-29T14:05:36.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:36.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:36.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: osd.7 v1:172.21.15.121:6812/82319362 boot 2024-04-29T14:05:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: osdmap e54: 8 total, 8 up, 8 in 2024-04-29T14:05:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-29T14:05:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:36.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:36 smithi121 ceph-mon[44533]: osdmap e55: 8 total, 8 up, 8 in 2024-04-29T14:05:36.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-04-29T14:05:36.714 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi121 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi121 to 4280M 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:37.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:37.989 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:37.989 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:37 smithi194 ceph-mon[44089]: osdmap e56: 8 total, 8 up, 8 in 2024-04-29T14:05:38.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi121 2024-04-29T14:05:38.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi121 to 4280M 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:37 smithi040 ceph-mon[39581]: osdmap e56: 8 total, 8 up, 8 in 2024-04-29T14:05:38.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi121 2024-04-29T14:05:38.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi121 to 4280M 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:38.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:37 smithi121 ceph-mon[44533]: osdmap e56: 8 total, 8 up, 8 in 2024-04-29T14:05:39.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:38 smithi194 ceph-mon[44089]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:39.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:38 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:05:39.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:38 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:05:39.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:38 smithi194 ceph-mon[44089]: osdmap e57: 8 total, 8 up, 8 in 2024-04-29T14:05:39.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:38 smithi040 ceph-mon[39581]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:39.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:38 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:05:39.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:38 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:05:39.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:38 smithi040 ceph-mon[39581]: osdmap e57: 8 total, 8 up, 8 in 2024-04-29T14:05:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:38 smithi121 ceph-mon[44533]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:38 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:05:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:38 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:05:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:38 smithi121 ceph-mon[44533]: osdmap e57: 8 total, 8 up, 8 in 2024-04-29T14:05:39.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:39 smithi194 ceph-mon[44089]: from='client.14445 v1:172.21.15.194:0/4169992003' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:05:39.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:05:39.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:05:39.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:39.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:39 smithi194 ceph-mon[44089]: osdmap e58: 8 total, 8 up, 8 in 2024-04-29T14:05:40.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:39 smithi040 ceph-mon[39581]: from='client.14445 v1:172.21.15.194:0/4169992003' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:05:40.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:05:40.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:05:40.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:40.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:39 smithi040 ceph-mon[39581]: osdmap e58: 8 total, 8 up, 8 in 2024-04-29T14:05:40.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:39 smithi121 ceph-mon[44533]: from='client.14445 v1:172.21.15.194:0/4169992003' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:05:40.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:05:40.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:05:40.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:40.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:39 smithi121 ceph-mon[44533]: osdmap e58: 8 total, 8 up, 8 in 2024-04-29T14:05:41.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:40 smithi194 ceph-mon[44089]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:41.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:40 smithi040 ceph-mon[39581]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:41.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:40 smithi121 ceph-mon[44533]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-29T14:05:42.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/802942408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]: dispatch 2024-04-29T14:05:42.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:41 smithi194 ceph-mon[44089]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]: dispatch 2024-04-29T14:05:42.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:41 smithi194 ceph-mon[44089]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]': finished 2024-04-29T14:05:42.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:41 smithi194 ceph-mon[44089]: osdmap e59: 9 total, 8 up, 9 in 2024-04-29T14:05:42.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:41 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:42.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/802942408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]: dispatch 2024-04-29T14:05:42.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:41 smithi040 ceph-mon[39581]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]: dispatch 2024-04-29T14:05:42.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:41 smithi040 ceph-mon[39581]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]': finished 2024-04-29T14:05:42.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:41 smithi040 ceph-mon[39581]: osdmap e59: 9 total, 8 up, 9 in 2024-04-29T14:05:42.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:41 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:42.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/802942408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]: dispatch 2024-04-29T14:05:42.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:41 smithi121 ceph-mon[44533]: from='client.24337 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]: dispatch 2024-04-29T14:05:42.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:41 smithi121 ceph-mon[44533]: from='client.24337 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de"}]': finished 2024-04-29T14:05:42.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:41 smithi121 ceph-mon[44533]: osdmap e59: 9 total, 8 up, 9 in 2024-04-29T14:05:42.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:41 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:43.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:42 smithi040 ceph-mon[39581]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:43.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/2935478782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:05:43.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:42 smithi121 ceph-mon[44533]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:43.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/2935478782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:05:43.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:42 smithi194 ceph-mon[44089]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:43.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/2935478782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:05:45.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:44 smithi040 ceph-mon[39581]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:45.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:45.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:45.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:44 smithi121 ceph-mon[44533]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:45.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:45.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:45.309 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:44 smithi194 ceph-mon[44089]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:45.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:45.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:46.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:45 smithi040 ceph-mon[39581]: osdmap e60: 9 total, 8 up, 9 in 2024-04-29T14:05:46.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:46.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:45 smithi121 ceph-mon[44533]: osdmap e60: 9 total, 8 up, 9 in 2024-04-29T14:05:46.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:45 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:46.194 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:45 smithi194 ceph-mon[44089]: osdmap e60: 9 total, 8 up, 9 in 2024-04-29T14:05:46.194 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:45 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:47.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:46 smithi040 ceph-mon[39581]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:47.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:46 smithi121 ceph-mon[44533]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:47.222 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:46 smithi194 ceph-mon[44089]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:48.088 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:47 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-29T14:05:48.089 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:47 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:48.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:47 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-29T14:05:48.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:47 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:48.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:47 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-29T14:05:48.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:47 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:48.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:48 smithi194 ceph-mon[44089]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:48.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:48 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:48.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:48 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:48.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:48 smithi194 ceph-mon[44089]: Deploying daemon osd.8 on smithi194 2024-04-29T14:05:48.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:48 smithi194 ceph-mon[44089]: osdmap e61: 9 total, 8 up, 9 in 2024-04-29T14:05:48.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:48 smithi040 ceph-mon[39581]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:48 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:48 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:48 smithi040 ceph-mon[39581]: Deploying daemon osd.8 on smithi194 2024-04-29T14:05:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:48 smithi040 ceph-mon[39581]: osdmap e61: 9 total, 8 up, 9 in 2024-04-29T14:05:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:48 smithi121 ceph-mon[44533]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:48 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:48 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:48 smithi121 ceph-mon[44533]: Deploying daemon osd.8 on smithi194 2024-04-29T14:05:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:48 smithi121 ceph-mon[44533]: osdmap e61: 9 total, 8 up, 9 in 2024-04-29T14:05:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:51.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:50 smithi040 ceph-mon[39581]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:51.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:50 smithi121 ceph-mon[44533]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:50 smithi194 ceph-mon[44089]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:53.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:52 smithi040 ceph-mon[39581]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:53.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:52 smithi121 ceph-mon[44533]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:53.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:52 smithi194 ceph-mon[44089]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:54.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:53 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:54.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:53 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:54.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:53 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:53 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:53 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:53 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:54.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:53 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:54.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:53 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:54.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:53 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:55.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:54 smithi194 ceph-mon[44089]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:55.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:54 smithi040 ceph-mon[39581]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:55.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:54 smithi121 ceph-mon[44533]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:56.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:56.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:56.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:56.409 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: from='osd.8 v1:172.21.15.194:6800/2130074108' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:05:56.409 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:56 smithi194 ceph-mon[44089]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:05:56.443 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 8 on host 'smithi194' 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: from='osd.8 v1:172.21.15.194:6800/2130074108' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:05:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:56 smithi040 ceph-mon[39581]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:05:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:05:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:05:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: from='osd.8 v1:172.21.15.194:6800/2130074108' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:05:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:56 smithi121 ceph-mon[44533]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:05:57.168 DEBUG:teuthology.orchestra.run.smithi194:osd.8> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.8.service 2024-04-29T14:05:57.170 INFO:tasks.cephadm:Deploying osd.9 on smithi194 with /dev/vg_nvme/lv_3... 2024-04-29T14:05:57.170 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-29T14:05:57.201 INFO:journalctl@ceph.osd.8.smithi194.stdout:-- Logs begin at Mon 2024-04-29 13:48:49 UTC. -- 2024-04-29T14:05:57.409 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[50001]: 2024-04-29T14:05:57.257+0000 7ffa6b55b700 -1 osd.8 0 waiting for initial osdmap 2024-04-29T14:05:57.429 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[50001]: 2024-04-29T14:05:57.257+0000 7ffa62cd9700 -1 osd.8 0 failed to load OSD map for epoch 10, got 0 bytes 2024-04-29T14:05:57.429 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[50001]: 2024-04-29T14:05:57.265+0000 7ffa6734e700 -1 osd.8 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:05:57.429 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-29T14:05:57.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: osdmap e62: 9 total, 8 up, 9 in 2024-04-29T14:05:57.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: from='osd.8 v1:172.21.15.194:6800/2130074108' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:05:57.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:57.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:05:57.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:57.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:57.430 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: osdmap e62: 9 total, 8 up, 9 in 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: from='osd.8 v1:172.21.15.194:6800/2130074108' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:57.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-29T14:05:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: osdmap e62: 9 total, 8 up, 9 in 2024-04-29T14:05:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: from='osd.8 v1:172.21.15.194:6800/2130074108' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:05:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:05:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:05:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:05:57.932 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:05:58.276 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:58 smithi194 ceph-mon[44089]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:05:58.276 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:58 smithi194 ceph-mon[44089]: osdmap e63: 9 total, 8 up, 9 in 2024-04-29T14:05:58.276 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:58.276 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:58.276 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:58 smithi194 ceph-mon[44089]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:58.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:58 smithi040 ceph-mon[39581]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:05:58.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:58 smithi040 ceph-mon[39581]: osdmap e63: 9 total, 8 up, 9 in 2024-04-29T14:05:58.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:58.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:58.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:58 smithi040 ceph-mon[39581]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:58.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:58 smithi121 ceph-mon[44533]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:05:58.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:58 smithi121 ceph-mon[44533]: osdmap e63: 9 total, 8 up, 9 in 2024-04-29T14:05:58.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:58.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:58.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:58 smithi121 ceph-mon[44533]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-04-29T14:05:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:05:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:05:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:59 smithi194 ceph-mon[44089]: osd.8 v1:172.21.15.194:6800/2130074108 boot 2024-04-29T14:05:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:59 smithi194 ceph-mon[44089]: osdmap e64: 9 total, 9 up, 9 in 2024-04-29T14:05:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:05:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:59.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:05:59.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:05:59.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:59.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:59.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:59 smithi040 ceph-mon[39581]: osd.8 v1:172.21.15.194:6800/2130074108 boot 2024-04-29T14:05:59.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:59 smithi040 ceph-mon[39581]: osdmap e64: 9 total, 9 up, 9 in 2024-04-29T14:05:59.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:05:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:05:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:05:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:05:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:05:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:59 smithi121 ceph-mon[44533]: osd.8 v1:172.21.15.194:6800/2130074108 boot 2024-04-29T14:05:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:59 smithi121 ceph-mon[44533]: osdmap e64: 9 total, 9 up, 9 in 2024-04-29T14:05:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:05:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:06:00.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:00 smithi194 ceph-mon[44089]: osdmap e65: 9 total, 9 up, 9 in 2024-04-29T14:06:00.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:00 smithi194 ceph-mon[44089]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:00.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:00 smithi040 ceph-mon[39581]: osdmap e65: 9 total, 9 up, 9 in 2024-04-29T14:06:00.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:00 smithi040 ceph-mon[39581]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:00.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:00 smithi121 ceph-mon[44533]: osdmap e65: 9 total, 9 up, 9 in 2024-04-29T14:06:00.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:00 smithi121 ceph-mon[44533]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:00.901 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-29T14:06:00.902 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-29T14:06:00.902 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-29T14:06:00.902 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-29T14:06:00.902 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112758 s, 930 MB/s 2024-04-29T14:06:00.902 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-29T14:06:01.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: osdmap e66: 9 total, 9 up, 9 in 2024-04-29T14:06:01.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:01.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:01.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:01.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:01 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.610 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-04-29T14:06:01.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: osdmap e66: 9 total, 9 up, 9 in 2024-04-29T14:06:01.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:01.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:01.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:01.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:01 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: osdmap e66: 9 total, 9 up, 9 in 2024-04-29T14:06:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:01.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:01.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:01.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:01 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:01.873 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:06:03.028 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:02 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi194 2024-04-29T14:06:03.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:02 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi194 to 21218M 2024-04-29T14:06:03.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:02 smithi194 ceph-mon[44089]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:03.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:02 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi194 2024-04-29T14:06:03.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:02 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi194 to 21218M 2024-04-29T14:06:03.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:02 smithi040 ceph-mon[39581]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:02 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi194 2024-04-29T14:06:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:02 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi194 to 21218M 2024-04-29T14:06:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:02 smithi121 ceph-mon[44533]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:04.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:04.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:04.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:04.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:04.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:04.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:04.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:04.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:04.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:05.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:04 smithi040 ceph-mon[39581]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:05.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:04 smithi040 ceph-mon[39581]: from='client.24358 v1:172.21.15.194:0/118426111' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:05.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:04 smithi121 ceph-mon[44533]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:05.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:04 smithi121 ceph-mon[44533]: from='client.24358 v1:172.21.15.194:0/118426111' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:05.230 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:04 smithi194 ceph-mon[44089]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:05.230 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:04 smithi194 ceph-mon[44089]: from='client.24358 v1:172.21.15.194:0/118426111' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:06 smithi040 ceph-mon[39581]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-29T14:06:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/924318931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]: dispatch 2024-04-29T14:06:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:06 smithi040 ceph-mon[39581]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]: dispatch 2024-04-29T14:06:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:06 smithi040 ceph-mon[39581]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]': finished 2024-04-29T14:06:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:06 smithi040 ceph-mon[39581]: osdmap e67: 10 total, 9 up, 10 in 2024-04-29T14:06:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:07.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:06 smithi121 ceph-mon[44533]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-29T14:06:07.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/924318931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]: dispatch 2024-04-29T14:06:07.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:06 smithi121 ceph-mon[44533]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]: dispatch 2024-04-29T14:06:07.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:06 smithi121 ceph-mon[44533]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]': finished 2024-04-29T14:06:07.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:06 smithi121 ceph-mon[44533]: osdmap e67: 10 total, 9 up, 10 in 2024-04-29T14:06:07.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:06 smithi194 ceph-mon[44089]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-29T14:06:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/924318931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]: dispatch 2024-04-29T14:06:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:06 smithi194 ceph-mon[44089]: from='client.24364 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]: dispatch 2024-04-29T14:06:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:06 smithi194 ceph-mon[44089]: from='client.24364 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0"}]': finished 2024-04-29T14:06:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:06 smithi194 ceph-mon[44089]: osdmap e67: 10 total, 9 up, 10 in 2024-04-29T14:06:07.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:08.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/1724827490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:06:08.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:06:08.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:06:08.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/1724827490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:06:08.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:06:08.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:06:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/1724827490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:06:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:06:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:06:09.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:08 smithi040 ceph-mon[39581]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:09.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:08 smithi040 ceph-mon[39581]: osdmap e68: 10 total, 9 up, 10 in 2024-04-29T14:06:09.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:09.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:08 smithi121 ceph-mon[44533]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:09.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:08 smithi121 ceph-mon[44533]: osdmap e68: 10 total, 9 up, 10 in 2024-04-29T14:06:09.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:09.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:08 smithi194 ceph-mon[44089]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:09.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:08 smithi194 ceph-mon[44089]: osdmap e68: 10 total, 9 up, 10 in 2024-04-29T14:06:09.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:11.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:10 smithi194 ceph-mon[44089]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:11.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:10 smithi040 ceph-mon[39581]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:11.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:10 smithi121 ceph-mon[44533]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:12.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:11 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:06:12.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:11 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:06:12.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:11 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:06:12.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:11 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:06:12.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:11 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:06:12.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:11 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:06:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:12 smithi040 ceph-mon[39581]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:12 smithi040 ceph-mon[39581]: osdmap e69: 10 total, 9 up, 10 in 2024-04-29T14:06:13.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:12 smithi121 ceph-mon[44533]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:12 smithi121 ceph-mon[44533]: osdmap e69: 10 total, 9 up, 10 in 2024-04-29T14:06:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:12 smithi194 ceph-mon[44089]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:06:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:12 smithi194 ceph-mon[44089]: osdmap e69: 10 total, 9 up, 10 in 2024-04-29T14:06:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-29T14:06:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:13 smithi040 ceph-mon[39581]: Deploying daemon osd.9 on smithi194 2024-04-29T14:06:14.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-29T14:06:14.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:14.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:13 smithi121 ceph-mon[44533]: Deploying daemon osd.9 on smithi194 2024-04-29T14:06:14.178 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-29T14:06:14.178 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:14.178 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:13 smithi194 ceph-mon[44089]: Deploying daemon osd.9 on smithi194 2024-04-29T14:06:15.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:14 smithi194 ceph-mon[44089]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:15.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:14 smithi040 ceph-mon[39581]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:15.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:14 smithi121 ceph-mon[44533]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:16 smithi040 ceph-mon[39581]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:17.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:16 smithi121 ceph-mon[44533]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:17.129 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:16 smithi194 ceph-mon[44089]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:19.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:18 smithi194 ceph-mon[44089]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:19.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:19.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:19.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:18 smithi040 ceph-mon[39581]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:18 smithi121 ceph-mon[44533]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:21.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:20 smithi194 ceph-mon[44089]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:21.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:20 smithi040 ceph-mon[39581]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:21.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:20 smithi121 ceph-mon[44533]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:22.314 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 9 on host 'smithi194' 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: from='osd.9 v1:172.21.15.194:6804/453882699' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: from='osd.9 v1:172.21.15.194:6804/453882699' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:22.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:22.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:22.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: from='osd.9 v1:172.21.15.194:6804/453882699' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:06:22.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:06:22.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:22.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:22.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:23.008 DEBUG:teuthology.orchestra.run.smithi194:osd.9> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.9.service 2024-04-29T14:06:23.010 INFO:tasks.cephadm:Deploying osd.10 on smithi194 with /dev/vg_nvme/lv_2... 2024-04-29T14:06:23.011 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-29T14:06:23.040 INFO:journalctl@ceph.osd.9.smithi194.stdout:-- Logs begin at Mon 2024-04-29 13:48:49 UTC. -- 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: osdmap e70: 10 total, 9 up, 10 in 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: from='osd.9 v1:172.21.15.194:6804/453882699' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:23.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:23.249 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[57740]: 2024-04-29T14:06:23.209+0000 7f2d43804700 -1 osd.9 0 waiting for initial osdmap 2024-04-29T14:06:23.249 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[57740]: 2024-04-29T14:06:23.209+0000 7f2d3af82700 -1 osd.9 0 failed to load OSD map for epoch 10, got 0 bytes 2024-04-29T14:06:23.249 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[57740]: 2024-04-29T14:06:23.218+0000 7f2d3f5f7700 -1 osd.9 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: osdmap e70: 10 total, 9 up, 10 in 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: from='osd.9 v1:172.21.15.194:6804/453882699' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:23.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-29T14:06:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: osdmap e70: 10 total, 9 up, 10 in 2024-04-29T14:06:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: from='osd.9 v1:172.21.15.194:6804/453882699' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:23.764 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:06:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:24 smithi194 ceph-mon[44089]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:06:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:24 smithi194 ceph-mon[44089]: osdmap e71: 10 total, 9 up, 10 in 2024-04-29T14:06:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:24 smithi194 ceph-mon[44089]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:24.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:24 smithi040 ceph-mon[39581]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:06:24.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:24 smithi040 ceph-mon[39581]: osdmap e71: 10 total, 9 up, 10 in 2024-04-29T14:06:24.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:24.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:24 smithi040 ceph-mon[39581]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:24.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:24 smithi121 ceph-mon[44533]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:06:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:24 smithi121 ceph-mon[44533]: osdmap e71: 10 total, 9 up, 10 in 2024-04-29T14:06:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:24 smithi121 ceph-mon[44533]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-04-29T14:06:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:06:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:06:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:06:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:25 smithi194 ceph-mon[44089]: osd.9 v1:172.21.15.194:6804/453882699 boot 2024-04-29T14:06:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:25 smithi194 ceph-mon[44089]: osdmap e72: 10 total, 10 up, 10 in 2024-04-29T14:06:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:25.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:06:25.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:06:25.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:06:25.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:25 smithi040 ceph-mon[39581]: osd.9 v1:172.21.15.194:6804/453882699 boot 2024-04-29T14:06:25.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:25 smithi040 ceph-mon[39581]: osdmap e72: 10 total, 10 up, 10 in 2024-04-29T14:06:25.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:25.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:06:25.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:06:25.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:06:25.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:25 smithi121 ceph-mon[44533]: osd.9 v1:172.21.15.194:6804/453882699 boot 2024-04-29T14:06:25.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:25 smithi121 ceph-mon[44533]: osdmap e72: 10 total, 10 up, 10 in 2024-04-29T14:06:25.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:06:26.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:26 smithi194 ceph-mon[44089]: osdmap e73: 10 total, 10 up, 10 in 2024-04-29T14:06:26.546 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:26 smithi194 ceph-mon[44089]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:26.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:26 smithi040 ceph-mon[39581]: osdmap e73: 10 total, 10 up, 10 in 2024-04-29T14:06:26.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:26 smithi040 ceph-mon[39581]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:26.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:26 smithi121 ceph-mon[44533]: osdmap e73: 10 total, 10 up, 10 in 2024-04-29T14:06:26.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:26 smithi121 ceph-mon[44533]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:27.506 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-29T14:06:27.507 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-29T14:06:27.507 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-29T14:06:27.507 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-29T14:06:27.507 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111673 s, 939 MB/s 2024-04-29T14:06:27.507 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-29T14:06:27.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi194 2024-04-29T14:06:27.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:27.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:27.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi194 to 10609M 2024-04-29T14:06:27.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:27.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:27.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:06:27.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi194 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi194 to 10609M 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:27.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:27.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:06:27.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:06:27.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi194 2024-04-29T14:06:27.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:27.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:27.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi194 to 10609M 2024-04-29T14:06:27.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:27.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:27.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:27.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:06:27.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:06:28.112 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-04-29T14:06:28.373 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:06:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:28 smithi194 ceph-mon[44089]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:28 smithi194 ceph-mon[44089]: osdmap e74: 10 total, 10 up, 10 in 2024-04-29T14:06:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:06:28.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:06:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:28 smithi040 ceph-mon[39581]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:28 smithi040 ceph-mon[39581]: osdmap e74: 10 total, 10 up, 10 in 2024-04-29T14:06:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:06:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:06:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:28 smithi121 ceph-mon[44533]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:28 smithi121 ceph-mon[44533]: osdmap e74: 10 total, 10 up, 10 in 2024-04-29T14:06:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:06:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:06:30.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:29 smithi194 ceph-mon[44089]: osdmap e75: 10 total, 10 up, 10 in 2024-04-29T14:06:30.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:29 smithi040 ceph-mon[39581]: osdmap e75: 10 total, 10 up, 10 in 2024-04-29T14:06:30.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:29 smithi121 ceph-mon[44533]: osdmap e75: 10 total, 10 up, 10 in 2024-04-29T14:06:31.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:30 smithi040 ceph-mon[39581]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:31.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:31.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:31.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:31.121 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:30 smithi194 ceph-mon[44089]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:31.121 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:31.121 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:31.121 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:31.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:30 smithi121 ceph-mon[44533]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:31.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:31.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:31.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:32.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:31 smithi194 ceph-mon[44089]: from='client.24385 v1:172.21.15.194:0/277765016' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:32.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:31 smithi040 ceph-mon[39581]: from='client.24385 v1:172.21.15.194:0/277765016' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:32.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:31 smithi121 ceph-mon[44533]: from='client.24385 v1:172.21.15.194:0/277765016' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:33.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:32 smithi040 ceph-mon[39581]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:33.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:32 smithi121 ceph-mon[44533]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:33.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:32 smithi194 ceph-mon[44089]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:34.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/1839400051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]: dispatch 2024-04-29T14:06:34.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:33 smithi040 ceph-mon[39581]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]: dispatch 2024-04-29T14:06:34.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:33 smithi040 ceph-mon[39581]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]': finished 2024-04-29T14:06:34.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:33 smithi040 ceph-mon[39581]: osdmap e76: 11 total, 10 up, 11 in 2024-04-29T14:06:34.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:34.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/4157330674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:06:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/1839400051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]: dispatch 2024-04-29T14:06:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:33 smithi121 ceph-mon[44533]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]: dispatch 2024-04-29T14:06:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:33 smithi121 ceph-mon[44533]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]': finished 2024-04-29T14:06:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:33 smithi121 ceph-mon[44533]: osdmap e76: 11 total, 10 up, 11 in 2024-04-29T14:06:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:34.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/4157330674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:06:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/1839400051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]: dispatch 2024-04-29T14:06:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:33 smithi194 ceph-mon[44089]: from='client.24391 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]: dispatch 2024-04-29T14:06:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:33 smithi194 ceph-mon[44089]: from='client.24391 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0"}]': finished 2024-04-29T14:06:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:33 smithi194 ceph-mon[44089]: osdmap e76: 11 total, 10 up, 11 in 2024-04-29T14:06:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:34.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/4157330674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:06:35.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:34 smithi040 ceph-mon[39581]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:35.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:34 smithi121 ceph-mon[44533]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:34 smithi194 ceph-mon[44089]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:37.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:36 smithi040 ceph-mon[39581]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:37.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:36 smithi121 ceph-mon[44533]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:37.164 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:36 smithi194 ceph-mon[44089]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:38 smithi121 ceph-mon[44533]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:39.133 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:38 smithi194 ceph-mon[44089]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:38 smithi040 ceph-mon[39581]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:40.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-29T14:06:40.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:40.136 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-29T14:06:40.137 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:40.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-29T14:06:40.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:41.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:40 smithi121 ceph-mon[44533]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:41.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:40 smithi121 ceph-mon[44533]: Deploying daemon osd.10 on smithi194 2024-04-29T14:06:41.186 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:40 smithi194 ceph-mon[44089]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:41.186 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:40 smithi194 ceph-mon[44089]: Deploying daemon osd.10 on smithi194 2024-04-29T14:06:41.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:40 smithi040 ceph-mon[39581]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:41.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:40 smithi040 ceph-mon[39581]: Deploying daemon osd.10 on smithi194 2024-04-29T14:06:43.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:42 smithi121 ceph-mon[44533]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:43.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:42 smithi194 ceph-mon[44089]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:43.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:42 smithi040 ceph-mon[39581]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:45.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:44 smithi194 ceph-mon[44089]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:44 smithi040 ceph-mon[39581]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:45.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:44 smithi121 ceph-mon[44533]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:46 smithi040 ceph-mon[39581]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:46 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:46 smithi121 ceph-mon[44533]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:46 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:46 smithi194 ceph-mon[44089]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:46 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:48.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:48 smithi194 ceph-mon[44089]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:48.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:48 smithi040 ceph-mon[39581]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:48.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:48 smithi121 ceph-mon[44533]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:49.110 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 10 on host 'smithi194' 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: osdmap e77: 11 total, 10 up, 11 in 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='osd.10 v1:172.21.15.194:6808/1320459116' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:49.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:49.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: osdmap e77: 11 total, 10 up, 11 in 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='osd.10 v1:172.21.15.194:6808/1320459116' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:49.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:06:49.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:06:49.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: osdmap e77: 11 total, 10 up, 11 in 2024-04-29T14:06:49.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:49.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='osd.10 v1:172.21.15.194:6808/1320459116' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-29T14:06:49.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-29T14:06:49.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:49.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:06:49.885 DEBUG:teuthology.orchestra.run.smithi194:osd.10> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.10.service 2024-04-29T14:06:49.887 INFO:tasks.cephadm:Deploying osd.11 on smithi194 with /dev/vg_nvme/lv_1... 2024-04-29T14:06:49.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-29T14:06:49.915 INFO:journalctl@ceph.osd.10.smithi194.stdout:-- Logs begin at Mon 2024-04-29 13:48:49 UTC. -- 2024-04-29T14:06:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-mon[44089]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-mon[44089]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-29T14:06:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-mon[44089]: osdmap e78: 11 total, 10 up, 11 in 2024-04-29T14:06:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-mon[44089]: from='osd.10 v1:172.21.15.194:6808/1320459116' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-mon[44089]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-mon[44089]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:06:50.818 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T14:06:50.446+0000 7f5af0f97700 -1 osd.10 0 waiting for initial osdmap 2024-04-29T14:06:50.818 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T14:06:50.446+0000 7f5ae8715700 -1 osd.10 0 failed to load OSD map for epoch 22, got 0 bytes 2024-04-29T14:06:50.819 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 14:06:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T14:06:50.454+0000 7f5aecd8a700 -1 osd.10 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:06:50.861 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:06:50.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:50 smithi040 ceph-mon[39581]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:50.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:50 smithi040 ceph-mon[39581]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-29T14:06:50.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:50 smithi040 ceph-mon[39581]: osdmap e78: 11 total, 10 up, 11 in 2024-04-29T14:06:50.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:50 smithi040 ceph-mon[39581]: from='osd.10 v1:172.21.15.194:6808/1320459116' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:50.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:50.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:50 smithi040 ceph-mon[39581]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:50.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:50 smithi040 ceph-mon[39581]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:06:50.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:50 smithi121 ceph-mon[44533]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:50.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:50 smithi121 ceph-mon[44533]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-29T14:06:50.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:50 smithi121 ceph-mon[44533]: osdmap e78: 11 total, 10 up, 11 in 2024-04-29T14:06:50.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:50 smithi121 ceph-mon[44533]: from='osd.10 v1:172.21.15.194:6808/1320459116' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:50.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:50.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:50 smithi121 ceph-mon[44533]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:06:50.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:50 smithi121 ceph-mon[44533]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:06:51.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:51 smithi194 ceph-mon[44089]: osdmap e79: 11 total, 10 up, 11 in 2024-04-29T14:06:51.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:51.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:51.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:51 smithi040 ceph-mon[39581]: osdmap e79: 11 total, 10 up, 11 in 2024-04-29T14:06:51.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:51.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:51.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:51 smithi121 ceph-mon[44533]: osdmap e79: 11 total, 10 up, 11 in 2024-04-29T14:06:51.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:51.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:52.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:06:52.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:06:52.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:06:52.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:52 smithi194 ceph-mon[44089]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:52.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:52 smithi194 ceph-mon[44089]: osd.10 v1:172.21.15.194:6808/1320459116 boot 2024-04-29T14:06:52.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:52 smithi194 ceph-mon[44089]: osdmap e80: 11 total, 11 up, 11 in 2024-04-29T14:06:52.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:52.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:06:52.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:06:52.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:06:52.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:52 smithi040 ceph-mon[39581]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:52.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:52 smithi040 ceph-mon[39581]: osd.10 v1:172.21.15.194:6808/1320459116 boot 2024-04-29T14:06:52.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:52 smithi040 ceph-mon[39581]: osdmap e80: 11 total, 11 up, 11 in 2024-04-29T14:06:52.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:52.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:06:52.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:06:52.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:06:52.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:52 smithi121 ceph-mon[44533]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-04-29T14:06:52.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:52 smithi121 ceph-mon[44533]: osd.10 v1:172.21.15.194:6808/1320459116 boot 2024-04-29T14:06:52.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:52 smithi121 ceph-mon[44533]: osdmap e80: 11 total, 11 up, 11 in 2024-04-29T14:06:52.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-29T14:06:53.627 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-29T14:06:53.627 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-29T14:06:53.627 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-29T14:06:53.628 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-29T14:06:53.628 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127044 s, 825 MB/s 2024-04-29T14:06:53.628 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-29T14:06:53.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:53 smithi194 ceph-mon[44089]: osdmap e81: 11 total, 11 up, 11 in 2024-04-29T14:06:53.910 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:53 smithi121 ceph-mon[44533]: osdmap e81: 11 total, 11 up, 11 in 2024-04-29T14:06:53.910 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:53 smithi040 ceph-mon[39581]: osdmap e81: 11 total, 11 up, 11 in 2024-04-29T14:06:54.471 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-04-29T14:06:54.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:54 smithi194 ceph-mon[44089]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:54.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:54 smithi040 ceph-mon[39581]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:54 smithi121 ceph-mon[44533]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:55.177 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:06:56.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi194 2024-04-29T14:06:56.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi194 to 7072M 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:56 smithi194 ceph-mon[44089]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi194 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi194 to 7072M 2024-04-29T14:06:56.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:56 smithi040 ceph-mon[39581]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi194 2024-04-29T14:06:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi194 to 7072M 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:06:56.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:56 smithi121 ceph-mon[44533]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:58.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:58.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:58.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:58.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:58.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:58.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:58.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-29T14:06:58.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-29T14:06:58.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:06:59.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:58 smithi194 ceph-mon[44089]: from='client.24412 v1:172.21.15.194:0/4284704762' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:59.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:58 smithi194 ceph-mon[44089]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:59.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:58 smithi040 ceph-mon[39581]: from='client.24412 v1:172.21.15.194:0/4284704762' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:59.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:58 smithi040 ceph-mon[39581]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:06:59.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:58 smithi121 ceph-mon[44533]: from='client.24412 v1:172.21.15.194:0/4284704762' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:06:59.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:58 smithi121 ceph-mon[44533]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:00.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:00.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:00.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:06:59 smithi040 ceph-mon[39581]: osdmap e82: 11 total, 11 up, 11 in 2024-04-29T14:07:00.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:00.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:00.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:06:59 smithi121 ceph-mon[44533]: osdmap e82: 11 total, 11 up, 11 in 2024-04-29T14:07:00.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:06:59 smithi194 ceph-mon[44089]: osdmap e82: 11 total, 11 up, 11 in 2024-04-29T14:07:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:00 smithi040 ceph-mon[39581]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/3687524989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]: dispatch 2024-04-29T14:07:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:00 smithi040 ceph-mon[39581]: from='client.24418 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]: dispatch 2024-04-29T14:07:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:00 smithi040 ceph-mon[39581]: from='client.24418 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]': finished 2024-04-29T14:07:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:00 smithi040 ceph-mon[39581]: osdmap e83: 12 total, 11 up, 12 in 2024-04-29T14:07:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:01.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/3970997001' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:07:01.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:00 smithi121 ceph-mon[44533]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:01.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/3687524989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]: dispatch 2024-04-29T14:07:01.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:00 smithi121 ceph-mon[44533]: from='client.24418 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]: dispatch 2024-04-29T14:07:01.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:00 smithi121 ceph-mon[44533]: from='client.24418 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]': finished 2024-04-29T14:07:01.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:00 smithi121 ceph-mon[44533]: osdmap e83: 12 total, 11 up, 12 in 2024-04-29T14:07:01.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:01.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/3970997001' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:07:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:00 smithi194 ceph-mon[44089]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/3687524989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]: dispatch 2024-04-29T14:07:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:00 smithi194 ceph-mon[44089]: from='client.24418 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]: dispatch 2024-04-29T14:07:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:00 smithi194 ceph-mon[44089]: from='client.24418 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f"}]': finished 2024-04-29T14:07:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:00 smithi194 ceph-mon[44089]: osdmap e83: 12 total, 11 up, 12 in 2024-04-29T14:07:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/3970997001' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-29T14:07:02.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:02.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:02.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:02 smithi194 ceph-mon[44089]: osdmap e84: 12 total, 11 up, 12 in 2024-04-29T14:07:02.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:02 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:02.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:02 smithi194 ceph-mon[44089]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:02.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:02.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:02.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:02 smithi040 ceph-mon[39581]: osdmap e84: 12 total, 11 up, 12 in 2024-04-29T14:07:02.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:02.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:02 smithi040 ceph-mon[39581]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:02.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:02.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:02.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:02 smithi121 ceph-mon[44533]: osdmap e84: 12 total, 11 up, 12 in 2024-04-29T14:07:02.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:02 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:02.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:02 smithi121 ceph-mon[44533]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:03.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:03 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:07:03.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:03 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:07:03.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:03 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:07:03.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:03 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:07:03.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:03 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:07:03.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:03 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:07:04.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:04 smithi194 ceph-mon[44089]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:04.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:04 smithi040 ceph-mon[39581]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:04.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:04 smithi121 ceph-mon[44533]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=2793268025,config=2903391811,health=3897446751,kv=1928160232,logm=2913155229}) 2024-04-29T14:07:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3650767072}) 2024-04-29T14:07:06.603 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156684960}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2484128270}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718715582}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799259475}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=501202429}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47864973}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=933830176}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718187244}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384624138}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795335457}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200381234}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181252812}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=306346990}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814492342}) 2024-04-29T14:07:06.604 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=86,mdsmap=3,mgr=11} crc {logm=1033646498,mdsmap=2840952586,mgr=3942164832}) 2024-04-29T14:07:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=79} crc {mgr=3735722144,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1449096553,mon_config_key=3868661724}) 2024-04-29T14:07:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025839822}) 2024-04-29T14:07:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=191712714,monmap=455069592,osd_metadata=2474369729}) 2024-04-29T14:07:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osdmap=96} crc {osd_metadata=1938386097,osd_pg_creating=1587152217,osdmap=3169732804}) 2024-04-29T14:07:06.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=9} crc {osdmap=2164835518}) 2024-04-29T14:07:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=2793268025,config=2903391811,health=3897446751,kv=1928160232,logm=2913155229}) 2024-04-29T14:07:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3650767072}) 2024-04-29T14:07:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156684960}) 2024-04-29T14:07:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2484128270}) 2024-04-29T14:07:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718715582}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799259475}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=501202429}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47864973}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=933830176}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718187244}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384624138}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795335457}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200381234}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181252812}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=306346990}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814492342}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=86,mdsmap=3,mgr=11} crc {logm=1033646498,mdsmap=2840952586,mgr=3942164832}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=79} crc {mgr=3735722144,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1449096553,mon_config_key=3868661724}) 2024-04-29T14:07:06.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025839822}) 2024-04-29T14:07:06.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=191712714,monmap=455069592,osd_metadata=2474369729}) 2024-04-29T14:07:06.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osdmap=96} crc {osd_metadata=1938386097,osd_pg_creating=1587152217,osdmap=3169732804}) 2024-04-29T14:07:06.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=9} crc {osdmap=2164835518}) 2024-04-29T14:07:06.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:06.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=23,config=2,health=12,kv=53,logm=10} crc {auth=2793268025,config=2903391811,health=3897446751,kv=1928160232,logm=2913155229}) 2024-04-29T14:07:06.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3650767072}) 2024-04-29T14:07:06.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156684960}) 2024-04-29T14:07:06.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2484128270}) 2024-04-29T14:07:06.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718715582}) 2024-04-29T14:07:06.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799259475}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=501202429}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=47864973}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=933830176}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718187244}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384624138}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795335457}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200381234}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181252812}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=306346990}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814492342}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=86,mdsmap=3,mgr=11} crc {logm=1033646498,mdsmap=2840952586,mgr=3942164832}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=79} crc {mgr=3735722144,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1449096553,mon_config_key=3868661724}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025839822}) 2024-04-29T14:07:06.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=191712714,monmap=455069592,osd_metadata=2474369729}) 2024-04-29T14:07:06.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osdmap=96} crc {osd_metadata=1938386097,osd_pg_creating=1587152217,osdmap=3169732804}) 2024-04-29T14:07:06.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=9} crc {osdmap=2164835518}) 2024-04-29T14:07:07.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-29T14:07:07.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:07.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:07 smithi194 ceph-mon[44089]: Deploying daemon osd.11 on smithi194 2024-04-29T14:07:07.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-29T14:07:07.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:07.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:07 smithi040 ceph-mon[39581]: Deploying daemon osd.11 on smithi194 2024-04-29T14:07:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-29T14:07:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:07 smithi121 ceph-mon[44533]: Deploying daemon osd.11 on smithi194 2024-04-29T14:07:08.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:08 smithi194 ceph-mon[44089]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:08.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:08 smithi040 ceph-mon[39581]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:08.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:08 smithi121 ceph-mon[44533]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:10.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:10 smithi194 ceph-mon[44089]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:10.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:10 smithi040 ceph-mon[39581]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:10.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:10 smithi121 ceph-mon[44533]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:12.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:12 smithi194 ceph-mon[44089]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:12.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:12.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:12.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:12.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:12 smithi040 ceph-mon[39581]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:12.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:12.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:12.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:12.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:12 smithi121 ceph-mon[44533]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:12.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:12.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:12.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:14.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:14 smithi194 ceph-mon[44089]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:14.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:14 smithi040 ceph-mon[39581]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:14.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:14 smithi121 ceph-mon[44533]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:15.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:15.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:15.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: osdmap e85: 12 total, 11 up, 12 in 2024-04-29T14:07:15.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:15.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='osd.11 v1:172.21.15.194:6812/20977082' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-29T14:07:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:15 smithi194 ceph-mon[44089]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-29T14:07:15.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: osdmap e85: 12 total, 11 up, 12 in 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='osd.11 v1:172.21.15.194:6812/20977082' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-29T14:07:15.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:15 smithi040 ceph-mon[39581]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: osdmap e85: 12 total, 11 up, 12 in 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:15.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:15.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:15.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='osd.11 v1:172.21.15.194:6812/20977082' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-29T14:07:15.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:15 smithi121 ceph-mon[44533]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-29T14:07:15.951 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 11 on host 'smithi194' 2024-04-29T14:07:16.695 DEBUG:teuthology.orchestra.run.smithi194:osd.11> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.11.service 2024-04-29T14:07:16.697 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-29T14:07:16.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-29T14:07:16.729 INFO:journalctl@ceph.osd.11.smithi194.stdout:-- Logs begin at Mon 2024-04-29 13:48:49 UTC. -- 2024-04-29T14:07:16.987 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: from='osd.11 v1:172.21.15.194:6812/20977082' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: osdmap e86: 12 total, 11 up, 12 in 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:16 smithi040 ceph-mon[39581]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: from='osd.11 v1:172.21.15.194:6812/20977082' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: osdmap e86: 12 total, 11 up, 12 in 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:16 smithi194 ceph-mon[44089]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:07:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-29T14:07:17.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: from='osd.11 v1:172.21.15.194:6812/20977082' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:07:17.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: osdmap e86: 12 total, 11 up, 12 in 2024-04-29T14:07:17.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:17.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:16 smithi121 ceph-mon[44533]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:07:17.815 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 14:07:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T14:07:17.616+0000 7fcdb00db700 -1 osd.11 0 waiting for initial osdmap 2024-04-29T14:07:17.815 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 14:07:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T14:07:17.617+0000 7fcda3851700 -1 osd.11 0 failed to load OSD map for epoch 33, got 0 bytes 2024-04-29T14:07:17.815 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 14:07:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T14:07:17.624+0000 7fcda7ec6700 -1 osd.11 87 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:07:18.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:18 smithi194 ceph-mon[44089]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:07:18.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:18 smithi194 ceph-mon[44089]: osdmap e87: 12 total, 11 up, 12 in 2024-04-29T14:07:18.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:18.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:18.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:18 smithi194 ceph-mon[44089]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:18.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:18 smithi194 ceph-mon[44089]: from='osd.11 ' entity='osd.11' 2024-04-29T14:07:18.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:18 smithi040 ceph-mon[39581]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:07:18.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:18 smithi040 ceph-mon[39581]: osdmap e87: 12 total, 11 up, 12 in 2024-04-29T14:07:18.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:18.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:18.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:18 smithi040 ceph-mon[39581]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:18.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:18 smithi040 ceph-mon[39581]: from='osd.11 ' entity='osd.11' 2024-04-29T14:07:18.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:18 smithi121 ceph-mon[44533]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-29T14:07:18.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:18 smithi121 ceph-mon[44533]: osdmap e87: 12 total, 11 up, 12 in 2024-04-29T14:07:18.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:18.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:18.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:18 smithi121 ceph-mon[44533]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-04-29T14:07:18.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:18 smithi121 ceph-mon[44533]: from='osd.11 ' entity='osd.11' 2024-04-29T14:07:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:07:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:07:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:07:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: osd.11 v1:172.21.15.194:6812/20977082 boot 2024-04-29T14:07:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: osdmap e88: 12 total, 12 up, 12 in 2024-04-29T14:07:19.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:19.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3997128482' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-29T14:07:19.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:19.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:07:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:07:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: osd.11 v1:172.21.15.194:6812/20977082 boot 2024-04-29T14:07:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: osdmap e88: 12 total, 12 up, 12 in 2024-04-29T14:07:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3997128482' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-29T14:07:19.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:19.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:19.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:07:19.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:07:19.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:19.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: osd.11 v1:172.21.15.194:6812/20977082 boot 2024-04-29T14:07:19.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: osdmap e88: 12 total, 12 up, 12 in 2024-04-29T14:07:19.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-29T14:07:19.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3997128482' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-29T14:07:19.959 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":88,"num_osds":12,"num_up_osds":12,"osd_up_since":1714399638,"num_in_osds":12,"osd_in_since":1714399620,"num_remapped_pgs":1} 2024-04-29T14:07:19.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T14:07:20.241 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:20.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:20.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:20.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:20.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:20.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: osdmap e89: 12 total, 12 up, 12 in 2024-04-29T14:07:20.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: Detected new or changed devices on smithi194 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: Adjusting osd_memory_target on smithi194 to 5304M 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:20.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:20.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:20.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:20.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: osdmap e89: 12 total, 12 up, 12 in 2024-04-29T14:07:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: Detected new or changed devices on smithi194 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: Adjusting osd_memory_target on smithi194 to 5304M 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:20.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:20.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: osdmap e89: 12 total, 12 up, 12 in 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: Detected new or changed devices on smithi194 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: Adjusting osd_memory_target on smithi194 to 5304M 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:07:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:20.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:21.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:21 smithi040 ceph-mon[39581]: osdmap e90: 12 total, 12 up, 12 in 2024-04-29T14:07:21.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:21 smithi121 ceph-mon[44533]: osdmap e90: 12 total, 12 up, 12 in 2024-04-29T14:07:22.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:21 smithi194 ceph-mon[44089]: osdmap e90: 12 total, 12 up, 12 in 2024-04-29T14:07:22.272 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:07:22.272 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T14:07:20.622452+0000","last_up_change":"2024-04-29T14:07:18.619339+0000","last_in_change":"2024-04-29T14:07:00.005557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":4178456629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":4178456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":4178456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":4178456629}]},"public_addr":"172.21.15.40:6805/4178456629","cluster_addr":"172.21.15.40:6806/4178456629","heartbeat_back_addr":"172.21.15.40:6808/4178456629","heartbeat_front_addr":"172.21.15.40:6807/4178456629","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3368423938}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3368423938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3368423938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3368423938}]},"public_addr":"172.21.15.40:6809/3368423938","cluster_addr":"172.21.15.40:6810/3368423938","heartbeat_back_addr":"172.21.15.40:6812/3368423938","heartbeat_front_addr":"172.21.15.40:6811/3368423938","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":3807513901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":3807513901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":3807513901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":3807513901}]},"public_addr":"172.21.15.121:6804/3807513901","cluster_addr":"172.21.15.121:6805/3807513901","heartbeat_back_addr":"172.21.15.121:6807/3807513901","heartbeat_front_addr":"172.21.15.121:6806/3807513901","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":2130074108}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":2130074108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":2130074108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":2130074108}]},"public_addr":"172.21.15.194:6800/2130074108","cluster_addr":"172.21.15.194:6801/2130074108","heartbeat_back_addr":"172.21.15.194:6803/2130074108","heartbeat_front_addr":"172.21.15.194:6802/2130074108","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":453882699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":453882699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":453882699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":453882699}]},"public_addr":"172.21.15.194:6804/453882699","cluster_addr":"172.21.15.194:6805/453882699","heartbeat_back_addr":"172.21.15.194:6807/453882699","heartbeat_front_addr":"172.21.15.194:6806/453882699","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:47.758404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:14.118351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:27.000303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:28.235216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:00.559509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:07.107560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:19.197460+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:58.499128+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:19.622523+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:22.559082+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:49.528252+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:16.413576+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T14:07:22.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:22 smithi040 ceph-mon[39581]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:22.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/568695440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:22.924 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-29T14:07:22.924 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-29T14:07:23.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:22 smithi121 ceph-mon[44533]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:23.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/568695440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:23.198 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:23.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:22 smithi194 ceph-mon[44089]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:23.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/568695440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:24.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:24 smithi040 ceph-mon[39581]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:25.039 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-29T14:07:25.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:24 smithi121 ceph-mon[44533]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:25.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:24 smithi194 ceph-mon[44089]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:25.723 INFO:tasks.cephadm:Setting up client nodes... 2024-04-29T14:07:25.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-29T14:07:26.005 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:26.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1425221917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-29T14:07:26.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1425221917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-29T14:07:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1425221917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-29T14:07:27.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:26 smithi040 ceph-mon[39581]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-29T14:07:27.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:26 smithi121 ceph-mon[44533]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-29T14:07:27.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:26 smithi194 ceph-mon[44089]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-29T14:07:28.180 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-29T14:07:28.180 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCgqS9m29r4ChAAaVOMaJwKZsZs1UNnzl5kXw== 2024-04-29T14:07:28.985 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-29T14:07:28.985 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-29T14:07:28.985 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-29T14:07:29.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-29T14:07:29.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/262686829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/262686829' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:29.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/262686829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/262686829' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:29.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:29.302 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/262686829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/262686829' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:07:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:07:31.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:30 smithi040 ceph-mon[39581]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-29T14:07:31.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:30 smithi121 ceph-mon[44533]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-29T14:07:31.297 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-04-29T14:07:31.298 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCjqS9m12/VERAA7V4wAE6vUS7cAvDI8gUkZA== 2024-04-29T14:07:31.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:30 smithi194 ceph-mon[44089]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-29T14:07:31.846 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-29T14:07:31.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-29T14:07:31.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-29T14:07:31.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-29T14:07:32.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.121:0/1020737330' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:32.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:31 smithi040 ceph-mon[39581]: from='client.34463 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:32.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:31 smithi040 ceph-mon[39581]: from='client.34463 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:32.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.121:0/1020737330' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:32.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:31 smithi121 ceph-mon[44533]: from='client.34463 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:32.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:31 smithi121 ceph-mon[44533]: from='client.34463 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:32.163 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:07:32.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.121:0/1020737330' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:32.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:31 smithi194 ceph-mon[44089]: from='client.34463 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:32.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:31 smithi194 ceph-mon[44089]: from='client.34463 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:33.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:32 smithi040 ceph-mon[39581]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-29T14:07:33.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:32 smithi121 ceph-mon[44533]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-29T14:07:33.156 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:32 smithi194 ceph-mon[44089]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-29T14:07:34.319 INFO:teuthology.orchestra.run.smithi194.stdout:[client.2] 2024-04-29T14:07:34.319 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQCmqS9md/X/EhAA34x89g9VB7wvLGE3xN3AEQ== 2024-04-29T14:07:35.010 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:34 smithi194 ceph-mon[44089]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-29T14:07:35.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.194:0/3266108798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:35.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:34 smithi194 ceph-mon[44089]: from='client.24466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:35.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:34 smithi194 ceph-mon[44089]: from='client.24466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:35.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:35.011 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:35.035 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-29T14:07:35.036 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-04-29T14:07:35.036 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-29T14:07:35.075 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-29T14:07:35.076 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-29T14:07:35.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-29T14:07:35.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:34 smithi040 ceph-mon[39581]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-29T14:07:35.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.194:0/3266108798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:35.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:34 smithi040 ceph-mon[39581]: from='client.24466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:35.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:34 smithi040 ceph-mon[39581]: from='client.24466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:35.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:35.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:35.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:34 smithi121 ceph-mon[44533]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-29T14:07:35.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.194:0/3266108798' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:35.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:34 smithi121 ceph-mon[44533]: from='client.24466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-29T14:07:35.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:34 smithi121 ceph-mon[44533]: from='client.24466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-29T14:07:35.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:35.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:35.365 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:36.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:36 smithi040 ceph-mon[39581]: osdmap e91: 12 total, 12 up, 12 in 2024-04-29T14:07:36.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:36 smithi040 ceph-mon[39581]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:36 smithi121 ceph-mon[44533]: osdmap e91: 12 total, 12 up, 12 in 2024-04-29T14:07:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:36 smithi121 ceph-mon[44533]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:36.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:36 smithi194 ceph-mon[44089]: osdmap e91: 12 total, 12 up, 12 in 2024-04-29T14:07:36.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:36 smithi194 ceph-mon[44089]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:07:38.225 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":15,"flags":0,"active_gid":14153,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":2764884391}]},"active_addr":"172.21.15.40:6800/2764884391","active_change":"2024-04-29T14:01:01.302709+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":34106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.40:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":3688166624}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2334054287}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":3256809955}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2433447113}]}]} 2024-04-29T14:07:38.229 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-29T14:07:38.229 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-29T14:07:38.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T14:07:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:38.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:38 smithi040 ceph-mon[39581]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:38.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/876263933' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-29T14:07:38.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:38 smithi194 ceph-mon[44089]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:38.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/876263933' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-29T14:07:38.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:38 smithi121 ceph-mon[44533]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:38.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/876263933' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-29T14:07:40.582 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:07:40.582 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T14:07:40.435264+0000","last_up_change":"2024-04-29T14:07:18.619339+0000","last_in_change":"2024-04-29T14:07:00.005557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":4178456629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":4178456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":4178456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":4178456629}]},"public_addr":"172.21.15.40:6805/4178456629","cluster_addr":"172.21.15.40:6806/4178456629","heartbeat_back_addr":"172.21.15.40:6808/4178456629","heartbeat_front_addr":"172.21.15.40:6807/4178456629","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3368423938}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3368423938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3368423938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3368423938}]},"public_addr":"172.21.15.40:6809/3368423938","cluster_addr":"172.21.15.40:6810/3368423938","heartbeat_back_addr":"172.21.15.40:6812/3368423938","heartbeat_front_addr":"172.21.15.40:6811/3368423938","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":3807513901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":3807513901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":3807513901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":3807513901}]},"public_addr":"172.21.15.121:6804/3807513901","cluster_addr":"172.21.15.121:6805/3807513901","heartbeat_back_addr":"172.21.15.121:6807/3807513901","heartbeat_front_addr":"172.21.15.121:6806/3807513901","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":2130074108}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":2130074108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":2130074108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":2130074108}]},"public_addr":"172.21.15.194:6800/2130074108","cluster_addr":"172.21.15.194:6801/2130074108","heartbeat_back_addr":"172.21.15.194:6803/2130074108","heartbeat_front_addr":"172.21.15.194:6802/2130074108","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":453882699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":453882699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":453882699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":453882699}]},"public_addr":"172.21.15.194:6804/453882699","cluster_addr":"172.21.15.194:6805/453882699","heartbeat_back_addr":"172.21.15.194:6807/453882699","heartbeat_front_addr":"172.21.15.194:6806/453882699","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:47.758404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:14.118351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:27.000303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:28.235216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:00.559509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:34.635071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:19.197460+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:58.499128+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:19.622523+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:39.581959+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:49.528252+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:16.413576+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T14:07:40.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:40 smithi194 ceph-mon[44089]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:40.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:40 smithi040 ceph-mon[39581]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:40.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:40 smithi121 ceph-mon[44533]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:41.307 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-29T14:07:41.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T14:07:41.588 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:41.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:41 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:41.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:41 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:41.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:41 smithi040 ceph-mon[39581]: osdmap e92: 12 total, 12 up, 12 in 2024-04-29T14:07:41.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4168467132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:41.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:41 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:41.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:41 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:41.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:41 smithi194 ceph-mon[44089]: osdmap e92: 12 total, 12 up, 12 in 2024-04-29T14:07:41.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4168467132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:41.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:41 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:41.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:41 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:41.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:41 smithi121 ceph-mon[44533]: osdmap e92: 12 total, 12 up, 12 in 2024-04-29T14:07:41.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4168467132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:43.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:42 smithi040 ceph-mon[39581]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:43.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:42 smithi121 ceph-mon[44533]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:43.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:42 smithi194 ceph-mon[44089]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:43.518 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:07:43.518 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T14:07:40.435264+0000","last_up_change":"2024-04-29T14:07:18.619339+0000","last_in_change":"2024-04-29T14:07:00.005557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":4178456629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":4178456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":4178456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":4178456629}]},"public_addr":"172.21.15.40:6805/4178456629","cluster_addr":"172.21.15.40:6806/4178456629","heartbeat_back_addr":"172.21.15.40:6808/4178456629","heartbeat_front_addr":"172.21.15.40:6807/4178456629","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3368423938}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3368423938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3368423938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3368423938}]},"public_addr":"172.21.15.40:6809/3368423938","cluster_addr":"172.21.15.40:6810/3368423938","heartbeat_back_addr":"172.21.15.40:6812/3368423938","heartbeat_front_addr":"172.21.15.40:6811/3368423938","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":3807513901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":3807513901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":3807513901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":3807513901}]},"public_addr":"172.21.15.121:6804/3807513901","cluster_addr":"172.21.15.121:6805/3807513901","heartbeat_back_addr":"172.21.15.121:6807/3807513901","heartbeat_front_addr":"172.21.15.121:6806/3807513901","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":2130074108}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":2130074108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":2130074108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":2130074108}]},"public_addr":"172.21.15.194:6800/2130074108","cluster_addr":"172.21.15.194:6801/2130074108","heartbeat_back_addr":"172.21.15.194:6803/2130074108","heartbeat_front_addr":"172.21.15.194:6802/2130074108","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":453882699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":453882699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":453882699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":453882699}]},"public_addr":"172.21.15.194:6804/453882699","cluster_addr":"172.21.15.194:6805/453882699","heartbeat_back_addr":"172.21.15.194:6807/453882699","heartbeat_front_addr":"172.21.15.194:6806/453882699","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:47.758404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:14.118351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:27.000303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:28.235216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:00.559509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:34.635071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:19.197460+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:58.499128+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:19.622523+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:39.581959+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:06:49.528252+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:07:16.413576+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T14:07:44.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1628948585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:44.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1628948585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:44.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-29T14:07:44.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-29T14:07:44.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-29T14:07:44.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-29T14:07:44.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-29T14:07:44.142 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-29T14:07:44.142 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-29T14:07:44.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-29T14:07:44.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-29T14:07:44.144 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-29T14:07:44.144 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-29T14:07:44.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-29T14:07:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1628948585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:07:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:44.938 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:44.997 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:44.997 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:44 smithi040 ceph-mon[39581]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:45.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:44 smithi121 ceph-mon[44533]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:45.199 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:44 smithi194 ceph-mon[44089]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:45.933 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.937 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:45.942 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:07:46.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:46.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:46.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:46.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:46.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:46.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:47.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:46 smithi040 ceph-mon[39581]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:47.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:46 smithi040 ceph-mon[39581]: osdmap e93: 12 total, 12 up, 12 in 2024-04-29T14:07:47.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:46 smithi121 ceph-mon[44533]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:47.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:46 smithi121 ceph-mon[44533]: osdmap e93: 12 total, 12 up, 12 in 2024-04-29T14:07:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:46 smithi194 ceph-mon[44089]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:46 smithi194 ceph-mon[44089]: osdmap e93: 12 total, 12 up, 12 in 2024-04-29T14:07:49.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:48 smithi040 ceph-mon[39581]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:49.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:48 smithi121 ceph-mon[44533]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:49.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:48 smithi194 ceph-mon[44089]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:50.985 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:50 smithi040 ceph-mon[39581]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:51.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:50 smithi121 ceph-mon[44533]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:50 smithi194 ceph-mon[44089]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:53.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:52 smithi121 ceph-mon[44533]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:53.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:52 smithi194 ceph-mon[44089]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:53.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:52 smithi040 ceph-mon[39581]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:54.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:54.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:54.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:54.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:54.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:55.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:54 smithi040 ceph-mon[39581]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:55.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:54 smithi040 ceph-mon[39581]: osdmap e94: 12 total, 12 up, 12 in 2024-04-29T14:07:55.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:54 smithi194 ceph-mon[44089]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:55.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:54 smithi194 ceph-mon[44089]: osdmap e94: 12 total, 12 up, 12 in 2024-04-29T14:07:55.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:54 smithi121 ceph-mon[44533]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:55.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:54 smithi121 ceph-mon[44533]: osdmap e94: 12 total, 12 up, 12 in 2024-04-29T14:07:56.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:56.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:56.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:55 smithi194 ceph-mon[44089]: osdmap e95: 12 total, 12 up, 12 in 2024-04-29T14:07:56.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:56.349 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:56.349 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:55 smithi040 ceph-mon[39581]: osdmap e95: 12 total, 12 up, 12 in 2024-04-29T14:07:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:55 smithi121 ceph-mon[44533]: osdmap e95: 12 total, 12 up, 12 in 2024-04-29T14:07:57.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:56 smithi194 ceph-mon[44089]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:57.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:56 smithi040 ceph-mon[39581]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:56 smithi121 ceph-mon[44533]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:58.170 INFO:teuthology.orchestra.run.smithi040.stdout:107374182450 2024-04-29T14:07:58.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-29T14:07:59.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:58 smithi194 ceph-mon[44089]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:59.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:07:59.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:07:59.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:58 smithi040 ceph-mon[39581]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:59.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:07:59.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:07:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:58 smithi121 ceph-mon[44533]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:07:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:07:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:07:59.720 INFO:teuthology.orchestra.run.smithi040.stdout:34359738434 2024-04-29T14:07:59.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-29T14:08:00.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:07:59 smithi040 ceph-mon[39581]: osdmap e96: 12 total, 12 up, 12 in 2024-04-29T14:08:00.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:07:59 smithi194 ceph-mon[44089]: osdmap e96: 12 total, 12 up, 12 in 2024-04-29T14:08:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:07:59 smithi121 ceph-mon[44533]: osdmap e96: 12 total, 12 up, 12 in 2024-04-29T14:08:01.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:00 smithi040 ceph-mon[39581]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:00 smithi194 ceph-mon[44089]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:01.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:00 smithi121 ceph-mon[44533]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:02 smithi040 ceph-mon[39581]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:03.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:02 smithi194 ceph-mon[44089]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:02 smithi121 ceph-mon[44533]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:03.654 INFO:teuthology.orchestra.run.smithi040.stdout:154618822696 2024-04-29T14:08:03.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-29T14:08:03.951 INFO:teuthology.orchestra.run.smithi040.stdout:274877906970 2024-04-29T14:08:03.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-29T14:08:04.416 INFO:teuthology.orchestra.run.smithi040.stdout:55834574910 2024-04-29T14:08:04.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-29T14:08:04.847 INFO:teuthology.orchestra.run.smithi040.stdout:309237645334 2024-04-29T14:08:04.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-29T14:08:05.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:04 smithi040 ceph-mon[39581]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:04 smithi194 ceph-mon[44089]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:04 smithi121 ceph-mon[44533]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:05.382 INFO:teuthology.orchestra.run.smithi040.stdout:343597383696 2024-04-29T14:08:05.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-29T14:08:05.758 INFO:teuthology.orchestra.run.smithi040.stdout:85899345977 2024-04-29T14:08:05.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-29T14:08:07.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:06 smithi040 ceph-mon[39581]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:07.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:06 smithi194 ceph-mon[44089]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:06 smithi121 ceph-mon[44533]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:08.045 INFO:teuthology.orchestra.run.smithi040.stdout:377957122061 2024-04-29T14:08:08.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-29T14:08:09.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:08 smithi040 ceph-mon[39581]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:08 smithi194 ceph-mon[44089]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:08 smithi121 ceph-mon[44533]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:09.555 INFO:teuthology.orchestra.run.smithi040.stdout:193273528358 2024-04-29T14:08:09.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-29T14:08:10.007 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:10.008 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:10.008 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:10.008 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:10.011 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:10.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:09 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:10.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:09 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:10.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:09 smithi040 ceph-mon[39581]: osdmap e97: 12 total, 12 up, 12 in 2024-04-29T14:08:10.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:09 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:10.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:09 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:10.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:09 smithi194 ceph-mon[44089]: osdmap e97: 12 total, 12 up, 12 in 2024-04-29T14:08:10.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:09 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:10.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:09 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:10.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:09 smithi121 ceph-mon[44533]: osdmap e97: 12 total, 12 up, 12 in 2024-04-29T14:08:10.600 INFO:teuthology.orchestra.run.smithi040.stdout:231928234016 2024-04-29T14:08:10.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-29T14:08:11.179 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:10 smithi040 ceph-mon[39581]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:10 smithi194 ceph-mon[44089]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:10 smithi121 ceph-mon[44533]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:12.389 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:13.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:12 smithi040 ceph-mon[39581]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:13.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:12 smithi194 ceph-mon[44089]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:13.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:12 smithi121 ceph-mon[44533]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:15.230 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:14 smithi040 ceph-mon[39581]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:15.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:14 smithi194 ceph-mon[44089]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:14 smithi121 ceph-mon[44533]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:15 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:16.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:15 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:16.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1718430652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-29T14:08:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:15 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:15 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:16.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1718430652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-29T14:08:16.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:15 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:16.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:15 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:16.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1718430652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-29T14:08:16.465 INFO:teuthology.orchestra.run.smithi040.stdout:133143986225 2024-04-29T14:08:16.466 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-29T14:08:17.007 INFO:teuthology.orchestra.run.smithi040.stdout:343597383699 2024-04-29T14:08:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:16 smithi040 ceph-mon[39581]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:16 smithi040 ceph-mon[39581]: osdmap e98: 12 total, 12 up, 12 in 2024-04-29T14:08:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:16 smithi194 ceph-mon[44089]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:16 smithi194 ceph-mon[44089]: osdmap e98: 12 total, 12 up, 12 in 2024-04-29T14:08:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:16 smithi121 ceph-mon[44533]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:16 smithi121 ceph-mon[44533]: osdmap e98: 12 total, 12 up, 12 in 2024-04-29T14:08:18.687 INFO:teuthology.orchestra.run.smithi040.stdout:34359738439 2024-04-29T14:08:19.056 INFO:teuthology.orchestra.run.smithi040.stdout:154618822700 2024-04-29T14:08:19.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:18 smithi040 ceph-mon[39581]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:19.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3249869535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-29T14:08:19.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3213718894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-29T14:08:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:18 smithi194 ceph-mon[44089]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3249869535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-29T14:08:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3213718894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-29T14:08:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:18 smithi121 ceph-mon[44533]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3249869535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-29T14:08:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3213718894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-29T14:08:19.830 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822696 got 154618822700 for osd.5 2024-04-29T14:08:19.830 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:20.638 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383696 got 343597383699 for osd.10 2024-04-29T14:08:20.638 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:20 smithi040 ceph-mon[39581]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:20 smithi194 ceph-mon[44089]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:20 smithi121 ceph-mon[44533]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:21.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738434 got 34359738439 for osd.0 2024-04-29T14:08:21.440 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:21.876 INFO:teuthology.orchestra.run.smithi040.stdout:55834574913 2024-04-29T14:08:22.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:21 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:08:22.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:21 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:08:22.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3608673529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-29T14:08:22.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:21 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:08:22.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:21 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:08:22.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3608673529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-29T14:08:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:21 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:08:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:21 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:08:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3608673529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-29T14:08:22.734 INFO:teuthology.orchestra.run.smithi040.stdout:107374182457 2024-04-29T14:08:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:22 smithi194 ceph-mon[44089]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:23.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/157028862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-29T14:08:23.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:22 smithi040 ceph-mon[39581]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:23.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/157028862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-29T14:08:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:22 smithi121 ceph-mon[44533]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:23.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/157028862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-29T14:08:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:24 smithi194 ceph-mon[44089]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:24 smithi040 ceph-mon[39581]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:25.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:24 smithi121 ceph-mon[44533]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:25.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182450 got 107374182457 for osd.3 2024-04-29T14:08:25.724 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:26.148 INFO:teuthology.orchestra.run.smithi040.stdout:309237645340 2024-04-29T14:08:26.332 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:26.364 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574910 got 55834574913 for osd.1 2024-04-29T14:08:26.364 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:27.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:26 smithi040 ceph-mon[39581]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:27.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:27.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:27.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:26 smithi040 ceph-mon[39581]: osdmap e99: 12 total, 12 up, 12 in 2024-04-29T14:08:27.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1013022799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-29T14:08:27.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:26 smithi194 ceph-mon[44089]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:27.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:27.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:27.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:26 smithi194 ceph-mon[44089]: osdmap e99: 12 total, 12 up, 12 in 2024-04-29T14:08:27.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1013022799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-29T14:08:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:26 smithi121 ceph-mon[44533]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:26 smithi121 ceph-mon[44533]: osdmap e99: 12 total, 12 up, 12 in 2024-04-29T14:08:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1013022799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-29T14:08:27.862 INFO:teuthology.orchestra.run.smithi040.stdout:85899345981 2024-04-29T14:08:28.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2707003181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-29T14:08:28.159 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645334 got 309237645340 for osd.9 2024-04-29T14:08:28.159 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:28.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2707003181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-29T14:08:28.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2707003181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-29T14:08:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:28 smithi194 ceph-mon[44089]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:29.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:28 smithi040 ceph-mon[39581]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:29.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:29.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:28 smithi121 ceph-mon[44533]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:29.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:30.269 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:30.269 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:30.269 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:30.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:30.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:30.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:30.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:30.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:30.795 INFO:teuthology.orchestra.run.smithi040.stdout:377957122065 2024-04-29T14:08:31.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:30 smithi040 ceph-mon[39581]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:31.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/48761005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-29T14:08:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:30 smithi194 ceph-mon[44089]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:31.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/48761005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-29T14:08:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:30 smithi121 ceph-mon[44533]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/48761005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-29T14:08:31.525 INFO:tasks.cephadm.ceph_manager.ceph:need seq 85899345977 got 85899345981 for osd.2 2024-04-29T14:08:31.526 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:32.164 INFO:teuthology.orchestra.run.smithi040.stdout:193273528362 2024-04-29T14:08:32.382 INFO:teuthology.orchestra.run.smithi040.stdout:274877906977 2024-04-29T14:08:32.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:32.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:32.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:32 smithi194 ceph-mon[44089]: osdmap e100: 12 total, 12 up, 12 in 2024-04-29T14:08:32.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:32 smithi194 ceph-mon[44089]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:32.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1948537004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-29T14:08:32.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:32.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:32.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:32 smithi040 ceph-mon[39581]: osdmap e100: 12 total, 12 up, 12 in 2024-04-29T14:08:32.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:32 smithi040 ceph-mon[39581]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:32.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1948537004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-29T14:08:32.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:32.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:32.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:32 smithi121 ceph-mon[44533]: osdmap e100: 12 total, 12 up, 12 in 2024-04-29T14:08:32.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:32 smithi121 ceph-mon[44533]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:32.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1948537004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-29T14:08:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/450047110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-29T14:08:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/450047110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-29T14:08:33.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/450047110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-29T14:08:34.172 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528358 got 193273528362 for osd.6 2024-04-29T14:08:34.173 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:34.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:34 smithi040 ceph-mon[39581]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:34.519 INFO:tasks.cephadm.ceph_manager.ceph:need seq 377957122061 got 377957122065 for osd.11 2024-04-29T14:08:34.520 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:34 smithi194 ceph-mon[44089]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:34.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:34 smithi121 ceph-mon[44533]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:35.405 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906970 got 274877906977 for osd.8 2024-04-29T14:08:35.406 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:35.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:35.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:35.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:36.078 INFO:teuthology.orchestra.run.smithi040.stdout:133143986230 2024-04-29T14:08:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:36 smithi040 ceph-mon[39581]: osdmap e101: 12 total, 12 up, 12 in 2024-04-29T14:08:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:36 smithi040 ceph-mon[39581]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:36.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/58559972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-29T14:08:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:36 smithi194 ceph-mon[44089]: osdmap e101: 12 total, 12 up, 12 in 2024-04-29T14:08:36.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:36 smithi194 ceph-mon[44089]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:36.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/58559972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-29T14:08:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:36 smithi121 ceph-mon[44533]: osdmap e101: 12 total, 12 up, 12 in 2024-04-29T14:08:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:36 smithi121 ceph-mon[44533]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/58559972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-29T14:08:36.898 INFO:teuthology.orchestra.run.smithi040.stdout:231928234022 2024-04-29T14:08:37.015 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986225 got 133143986230 for osd.4 2024-04-29T14:08:37.015 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:37.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1176565304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-29T14:08:37.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1176565304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-29T14:08:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1176565304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-29T14:08:37.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234016 got 231928234022 for osd.7 2024-04-29T14:08:37.668 DEBUG:teuthology.parallel:result is None 2024-04-29T14:08:37.669 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:08:37.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:08:37.949 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:38.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:38 smithi194 ceph-mon[44089]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:38.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:38 smithi040 ceph-mon[39581]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:38 smithi121 ceph-mon[44533]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:39.842 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:08:39.844 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:08:40.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:40 smithi040 ceph-mon[39581]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:40.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:40 smithi040 ceph-mon[39581]: from='client.14745 v1:172.21.15.40:0/3057811712' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:40.479 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-04-29T14:08:39.393188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":337000,"kb_used_data":8600,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604824,"statfs":{"total":1151940427776,"available":1151595339776,"internally_reserved":0,"allocated":8806400,"data_stored":4556074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.194915"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":347,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-29T14:08:31.849393+0000","last_change":"2024-04-29T14:08:20.464185+0000","last_active":"2024-04-29T14:08:31.849393+0000","last_peered":"2024-04-29T14:08:31.849393+0000","last_clean":"2024-04-29T14:08:31.849393+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:08:31.849393+0000","last_undegraded":"2024-04-29T14:08:31.849393+0000","last_fullsized":"2024-04-29T14:08:31.849393+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:08:20.464057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:08:20.464057+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:09:29.677814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1171456,"data_stored":817624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645342,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717508,"statfs":{"total":95995035648,"available":95966728192,"internally_reserved":0,"allocated":638976,"data_stored":288716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877906979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234022,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1171456,"data_stored":817644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717508,"statfs":{"total":95995035648,"available":95966728192,"internally_reserved":0,"allocated":638976,"data_stored":288716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713336,"statfs":{"total":95995035648,"available":95962456064,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717508,"statfs":{"total":95995035648,"available":95966728192,"internally_reserved":0,"allocated":638976,"data_stored":288716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717508,"statfs":{"total":95995035648,"available":95966728192,"internally_reserved":0,"allocated":638976,"data_stored":288716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:08:40.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:08:40.761 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:40.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:40 smithi194 ceph-mon[44089]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:40.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:40 smithi194 ceph-mon[44089]: from='client.14745 v1:172.21.15.40:0/3057811712' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:40.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:40 smithi121 ceph-mon[44533]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:40.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:40 smithi121 ceph-mon[44533]: from='client.14745 v1:172.21.15.40:0/3057811712' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:42.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:42 smithi040 ceph-mon[39581]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:42.763 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:08:42.765 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:08:42.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:42 smithi194 ceph-mon[44089]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:42.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:42 smithi121 ceph-mon[44533]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:43.570 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-29T14:08:41.393605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":337072,"kb_used_data":8672,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604752,"statfs":{"total":1151940427776,"available":1151595266048,"internally_reserved":0,"allocated":8880128,"data_stored":4589740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001920"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":349,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-29T14:08:35.823566+0000","last_change":"2024-04-29T14:08:20.464185+0000","last_active":"2024-04-29T14:08:35.823566+0000","last_peered":"2024-04-29T14:08:35.823566+0000","last_clean":"2024-04-29T14:08:35.823566+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:08:35.823566+0000","last_undegraded":"2024-04-29T14:08:35.823566+0000","last_fullsized":"2024-04-29T14:08:35.823566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:08:20.464057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:08:20.464057+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:09:29.677814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":823235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645343,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877906979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234023,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":823255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713336,"statfs":{"total":95995035648,"available":95962456064,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:08:43.571 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:08:43.571 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-29T14:08:43.571 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-29T14:08:43.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-29T14:08:43.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:43 smithi194 ceph-mon[44089]: from='client.14751 v1:172.21.15.40:0/3181250967' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:43.851 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:43.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:43 smithi040 ceph-mon[39581]: from='client.14751 v1:172.21.15.40:0/3181250967' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:43.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:43 smithi121 ceph-mon[44533]: from='client.14751 v1:172.21.15.40:0/3181250967' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:44.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:44 smithi194 ceph-mon[44089]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:44.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:44 smithi040 ceph-mon[39581]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:44.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:44 smithi121 ceph-mon[44533]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:08:45.923 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-29T14:08:46.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:46 smithi040 ceph-mon[39581]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:46.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3607424508' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-29T14:08:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:46.752 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-29T14:08:46.752 INFO:tasks.cephadm:Setup complete, yielding 2024-04-29T14:08:46.752 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-29T14:08:46.765 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-29T14:08:46.765 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-pereman2-buffer-space'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '7fe685c922bae3b52755209e35feddd972749cff'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7fe685c922bae3b52755209e35feddd972749cff'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-pereman2-buffer-space', 'sha1': '7fe685c922bae3b52755209e35feddd972749cff'}} 2024-04-29T14:08:46.765 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-29T14:08:46.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-29T14:08:46.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:46 smithi194 ceph-mon[44089]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3607424508' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-29T14:08:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:46.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:46.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:46 smithi121 ceph-mon[44533]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:46.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3607424508' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-29T14:08:46.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:46.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:47.047 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:47.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:47 smithi040 ceph-mon[39581]: osdmap e102: 12 total, 12 up, 12 in 2024-04-29T14:08:47.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:47 smithi194 ceph-mon[44089]: osdmap e102: 12 total, 12 up, 12 in 2024-04-29T14:08:47.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:47 smithi121 ceph-mon[44533]: osdmap e102: 12 total, 12 up, 12 in 2024-04-29T14:08:49.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:48 smithi040 ceph-mon[39581]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:49.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:48 smithi121 ceph-mon[44533]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:48 smithi194 ceph-mon[44089]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:49.815 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-29T14:08:49.815 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:08:49.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:08:50.094 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:50 smithi194 ceph-mon[44089]: from='client.14763 ' entity='client.admin' 2024-04-29T14:08:50.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:50.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:50.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:50.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:50.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:50 smithi194 ceph-mon[44089]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:50.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:50 smithi040 ceph-mon[39581]: from='client.14763 ' entity='client.admin' 2024-04-29T14:08:50.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:50.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:50.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:50.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:50.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:50 smithi040 ceph-mon[39581]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:50 smithi121 ceph-mon[44533]: from='client.14763 ' entity='client.admin' 2024-04-29T14:08:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:50 smithi121 ceph-mon[44533]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:51.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:51.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:51.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:51.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:52.098 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:08:52.100 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:08:52.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:52 smithi040 ceph-mon[39581]: osdmap e103: 12 total, 12 up, 12 in 2024-04-29T14:08:52.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:52 smithi040 ceph-mon[39581]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:52.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:52 smithi194 ceph-mon[44089]: osdmap e103: 12 total, 12 up, 12 in 2024-04-29T14:08:52.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:52 smithi194 ceph-mon[44089]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:52 smithi121 ceph-mon[44533]: osdmap e103: 12 total, 12 up, 12 in 2024-04-29T14:08:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:52 smithi121 ceph-mon[44533]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:52.848 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-04-29T14:08:51.395833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":337192,"kb_used_data":8792,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604632,"statfs":{"total":1151940427776,"available":1151595143168,"internally_reserved":0,"allocated":9003008,"data_stored":4645850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001828"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":351,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-29T14:08:46.980401+0000","last_change":"2024-04-29T14:08:20.464185+0000","last_active":"2024-04-29T14:08:46.980401+0000","last_peered":"2024-04-29T14:08:46.980401+0000","last_clean":"2024-04-29T14:08:46.980401+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:08:46.980401+0000","last_undegraded":"2024-04-29T14:08:46.980401+0000","last_fullsized":"2024-04-29T14:08:46.980401+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:08:20.464057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:08:20.464057+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:09:29.677814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1196032,"data_stored":828846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383706,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":651264,"data_stored":294307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877906981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234025,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1196032,"data_stored":828866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528367,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713336,"statfs":{"total":95995035648,"available":95962456064,"internally_reserved":0,"allocated":651264,"data_stored":294327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822707,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:08:52.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:08:53.127 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:53.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:53 smithi194 ceph-mon[44089]: from='client.14769 v1:172.21.15.40:0/2784686799' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:53.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:53 smithi040 ceph-mon[39581]: from='client.14769 v1:172.21.15.40:0/2784686799' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:53.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:53 smithi121 ceph-mon[44533]: from='client.14769 v1:172.21.15.40:0/2784686799' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:54 smithi040 ceph-mon[39581]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:08:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:08:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:08:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:54 smithi194 ceph-mon[44089]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:08:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:54 smithi121 ceph-mon[44533]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:55.265 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:08:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:08:56.044 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-29T14:08:53.396216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":337228,"kb_used_data":8828,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604596,"statfs":{"total":1151940427776,"available":1151595106304,"internally_reserved":0,"allocated":9039872,"data_stored":4662683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001663"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":351,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-29T14:08:46.980401+0000","last_change":"2024-04-29T14:08:20.464185+0000","last_active":"2024-04-29T14:08:46.980401+0000","last_peered":"2024-04-29T14:08:46.980401+0000","last_clean":"2024-04-29T14:08:46.980401+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:08:46.980401+0000","last_undegraded":"2024-04-29T14:08:46.980401+0000","last_fullsized":"2024-04-29T14:08:46.980401+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:08:20.464057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:08:20.464057+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:09:29.677814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1196032,"data_stored":828846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383707,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877906981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234025,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1196032,"data_stored":828866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528367,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345987,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31840,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713312,"statfs":{"total":95995035648,"available":95962431488,"internally_reserved":0,"allocated":675840,"data_stored":305549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822707,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":299938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:08:56.045 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:08:56.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump 2024-04-29T14:08:56.322 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:08:56.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:56 smithi194 ceph-mon[44089]: from='client.14775 v1:172.21.15.40:0/3598059996' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:56.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:56 smithi194 ceph-mon[44089]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:56.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:56 smithi040 ceph-mon[39581]: from='client.14775 v1:172.21.15.40:0/3598059996' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:56.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:56 smithi040 ceph-mon[39581]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:56.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:56 smithi121 ceph-mon[44533]: from='client.14775 v1:172.21.15.40:0/3598059996' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:08:56.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:56 smithi121 ceph-mon[44533]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:57 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:08:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:57 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:08:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:57 smithi040 ceph-mon[39581]: osdmap e104: 12 total, 12 up, 12 in 2024-04-29T14:08:57.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:57 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:08:57.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:57 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:08:57.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:57 smithi194 ceph-mon[44089]: osdmap e104: 12 total, 12 up, 12 in 2024-04-29T14:08:57.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:57 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:08:57.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:57 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:08:57.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:57 smithi121 ceph-mon[44533]: osdmap e104: 12 total, 12 up, 12 in 2024-04-29T14:08:58.401 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 104 2024-04-29T14:08:58.401 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T14:08:58.401 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-04-29T14:00:01.568926+0000 2024-04-29T14:08:58.401 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-04-29T14:08:56.448711+0000 2024-04-29T14:08:58.401 INFO:teuthology.orchestra.run.smithi040.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-29T14:08:58.401 INFO:teuthology.orchestra.run.smithi040.stdout:crush_version 26 2024-04-29T14:08:58.401 INFO:teuthology.orchestra.run.smithi040.stdout:full_ratio 0.95 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:backfillfull_ratio 0.9 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:nearfull_ratio 0.85 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:require_min_compat_client luminous 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:min_compat_client jewel 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:require_osd_release squid 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:stretch_mode_enabled false 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:max_osd 12 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 up in weight 1 up_from 8 up_thru 56 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6801/4255137508 v1:172.21.15.40:6802/4255137508 exists,up 967eec9c-202c-4f98-a22a-a67db85bd462 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6805/4178456629 v1:172.21.15.40:6806/4178456629 exists,up 7f52729c-855b-465c-b134-8326f99f1422 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6809/3368423938 v1:172.21.15.40:6810/3368423938 exists,up 372baf31-26e1-4865-a272-898e7ff0c2f3 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6813/4012292460 v1:172.21.15.40:6814/4012292460 exists,up 975c2fbc-115b-4950-88ef-b0b2ad778765 2024-04-29T14:08:58.402 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6800/3987960551 v1:172.21.15.121:6801/3987960551 exists,up 6aa13ecc-0959-4915-a788-03c4739da269 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 up in weight 1 up_from 36 up_thru 39 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6804/3807513901 v1:172.21.15.121:6805/3807513901 exists,up 6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 up in weight 1 up_from 45 up_thru 48 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6808/1755659098 v1:172.21.15.121:6809/1755659098 exists,up 40ecb14f-7b61-45d2-b48c-0a374912edcf 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 up in weight 1 up_from 54 up_thru 89 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6812/82319362 v1:172.21.15.121:6813/82319362 exists,up 38e15653-4a3d-4543-8240-e17d95bdfe27 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 up in weight 1 up_from 64 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6800/2130074108 v1:172.21.15.194:6801/2130074108 exists,up d93f2b45-6c82-4ead-9a66-e86f9e83e0de 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 up in weight 1 up_from 72 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6804/453882699 v1:172.21.15.194:6805/453882699 exists,up a720d016-7cf7-4755-a25b-4d081c1f13a0 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6808/1320459116 v1:172.21.15.194:6809/1320459116 exists,up 87c864f5-90a1-48e9-bca8-65cf404bc2a0 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 up in weight 1 up_from 88 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6812/20977082 v1:172.21.15.194:6813/20977082 exists,up 5c85066b-5780-4393-be30-0033b9d2dd1f 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3316779772 expires 2024-04-30T14:01:01.302559+0000 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/2003707927 expires 2024-04-30T14:00:29.409543+0000 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/2722668908 expires 2024-04-30T14:00:29.409543+0000 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/2004115894 expires 2024-04-30T14:00:29.409543+0000 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3681930020 expires 2024-04-30T14:01:01.302559+0000 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/3860863692 expires 2024-04-30T14:01:01.302559+0000 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3976042573 expires 2024-04-30T14:00:29.409543+0000 2024-04-29T14:08:58.403 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/1925168828 expires 2024-04-30T14:01:01.302559+0000 2024-04-29T14:08:58.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:58 smithi040 ceph-mon[39581]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:58.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:08:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2477586695' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-29T14:08:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:58 smithi194 ceph-mon[44089]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:58.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:08:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2477586695' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-29T14:08:58.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:58 smithi121 ceph-mon[44533]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:08:58.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:08:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2477586695' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-29T14:08:59.082 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 56 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6801/4255137508 v1:172.21.15.40:6802/4255137508 exists,up 967eec9c-202c-4f98-a22a-a67db85bd462', 'osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6805/4178456629 v1:172.21.15.40:6806/4178456629 exists,up 7f52729c-855b-465c-b134-8326f99f1422', 'osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6809/3368423938 v1:172.21.15.40:6810/3368423938 exists,up 372baf31-26e1-4865-a272-898e7ff0c2f3', 'osd.3 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6813/4012292460 v1:172.21.15.40:6814/4012292460 exists,up 975c2fbc-115b-4950-88ef-b0b2ad778765', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6800/3987960551 v1:172.21.15.121:6801/3987960551 exists,up 6aa13ecc-0959-4915-a788-03c4739da269', 'osd.5 up in weight 1 up_from 36 up_thru 39 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6804/3807513901 v1:172.21.15.121:6805/3807513901 exists,up 6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'osd.6 up in weight 1 up_from 45 up_thru 48 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6808/1755659098 v1:172.21.15.121:6809/1755659098 exists,up 40ecb14f-7b61-45d2-b48c-0a374912edcf', 'osd.7 up in weight 1 up_from 54 up_thru 89 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6812/82319362 v1:172.21.15.121:6813/82319362 exists,up 38e15653-4a3d-4543-8240-e17d95bdfe27', 'osd.8 up in weight 1 up_from 64 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6800/2130074108 v1:172.21.15.194:6801/2130074108 exists,up d93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'osd.9 up in weight 1 up_from 72 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6804/453882699 v1:172.21.15.194:6805/453882699 exists,up a720d016-7cf7-4755-a25b-4d081c1f13a0', 'osd.10 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6808/1320459116 v1:172.21.15.194:6809/1320459116 exists,up 87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'osd.11 up in weight 1 up_from 88 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.194:6812/20977082 v1:172.21.15.194:6813/20977082 exists,up 5c85066b-5780-4393-be30-0033b9d2dd1f'] 2024-04-29T14:08:59.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon mon.a config show 2024-04-29T14:08:59.360 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:00.792 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:00 smithi040 ceph-mon[39581]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:00.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:00 smithi194 ceph-mon[44089]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:00 smithi121 ceph-mon[44533]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:01.082 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:09:01.082 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mon.a", 2024-04-29T14:09:01.082 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster": "ceph", 2024-04-29T14:09:01.082 INFO:teuthology.orchestra.run.smithi040.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-29T14:09:01.082 INFO:teuthology.orchestra.run.smithi040.stdout: "admin_socket_mode": "", 2024-04-29T14:09:01.082 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-29T14:09:01.082 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_client_required": "cephx, none", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_cluster_required": "cephx", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_debug": "true", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_service_required": "cephx", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_supported": "", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio": "true", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio_poll_ms": "250", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio_reap_max": "16", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_async_discard_threads": "0", 2024-04-29T14:09:01.083 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_block_size": "4096", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_aio": "false", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_inflight_ios": "false", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_enable_discard": "false", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_flock_retry": "3", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_inject_crash": "0", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_ioring": "false", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_ioring_hipri": "false", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-29T14:09:01.084 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_type": "", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_alloc_size": "1048576", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_allocator": "hybrid", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_buffered_io": "true", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_check_for_zeros": "false", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_compact_log_sync": "false", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-29T14:09:01.085 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_max_log_runway": "4194304", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_max_prefetch": "1048576", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_min_flush_size": "524288", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_min_log_runway": "1048576", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_replay_recovery": "false", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_sync_write": "false", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_allocation_from_file": "true", 2024-04-29T14:09:01.086 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_allocator": "hybrid", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_create": "true", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_db_create": "false", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_db_path": "", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_db_size": "0", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_path": "", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_preallocate_file": "false", 2024-04-29T14:09:01.087 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_size": "107374182400", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_wal_create": "false", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_wal_path": "", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_wal_size": "100663296", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs": "true", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_autotune": "true", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-29T14:09:01.088 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_size": "0", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_type": "2q", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_clone_cow": "true", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-29T14:09:01.089 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_mode": "none", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_csum_type": "crc32c", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_freelist": "false", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-29T14:09:01.090 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_small_allocations": "0", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_default_buffered_read": "true", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_default_buffered_write": "false", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-29T14:09:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fail_eio": "false", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-29T14:09:01.092 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mount": "false", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_umount": "false", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_ignore_data_csum": "false", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-29T14:09:01.093 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_log_op_age": "5.000000", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_alloc_size": "0", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_blob_size": "0", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_deferred_txc": "32", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_min_alloc_size": "0", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_nid_prealloc": "1024", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-29T14:09:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-29T14:09:01.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-29T14:09:01.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-29T14:09:01.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_retry_disk_reads": "3", 2024-04-29T14:09:01.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_cf": "true", 2024-04-29T14:09:01.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-29T14:09:01.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-29T14:09:01.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-29T14:09:01.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-29T14:09:01.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-29T14:09:01.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-29T14:09:01.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_mem": "512", 2024-04-29T14:09:01.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-29T14:09:01.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-29T14:09:01.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_tracing": "false", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-29T14:09:01.097 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_zero_block_detection": "false", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-29T14:09:01.098 INFO:teuthology.orchestra.run.smithi040.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_cluster_require_signatures": "false", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_cluster_require_version": "2", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_require_signatures": "false", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_require_version": "2", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_service_require_signatures": "false", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_service_require_version": "2", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_sign_messages": "true", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "chdir": "", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_acl_type": "", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_asio_thread_count": "2", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_cache_mid": "0.750000", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_cache_size": "16384", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_caps_release_delay": "5", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_check_pool_perm": "true", 2024-04-29T14:09:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_debug_force_sync_read": "false", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_debug_getattr_caps": "false", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_debug_inject_tick_delay": "0", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_die_on_failed_remount": "false", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_dirsize_rbytes": "true", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_force_lazyio": "false", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_fs": "", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_inject_release_failure": "false", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_max_inline_size": "4096", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mds_namespace": "", 2024-04-29T14:09:01.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_metadata": "", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mount_gid": "-1", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mount_timeout": "300", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mount_uid": "-1", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mountpoint": "/", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_notify_timeout": "10", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc": "true", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_max_dirty": "104857600", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_max_objects": "1000", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_size": "209715200", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_target_dirty": "8388608", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_permissions": "true", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_quota": "true", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_quota_df": "true", 2024-04-29T14:09:01.101 INFO:teuthology.orchestra.run.smithi040.stdout: "client_readahead_max_bytes": "0", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_readahead_max_periods": "4", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_readahead_min": "131072", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_reconnect_stale": "false", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_shutdown_timeout": "30", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_snapdir": ".snap", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_tick_interval": "1", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_trace": "", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_try_dentry_invalidate": "false", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_use_faked_inos": "false", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_use_random_mds": "false", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_graylog": "false", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_graylog_port": "12201", 2024-04-29T14:09:01.102 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_monitors": "default=true", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_syslog": "false", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_syslog_level": "info", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "-", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_network": "", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_network_interface": "", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zlib_isal": "false", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zlib_level": "5", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zlib_winsize": "-15", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zstd_level": "1", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cda992d224e2e50c726e715240eee84b5e97f0c57be048f859f062f583c4d452", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_alien_op_num_threads": "6", 2024-04-29T14:09:01.103 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_osd_stat_interval": "0", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_seastar_cpu_cores": "", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location_hook": "", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location_hook_timeout": "10", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "daemonize": "false", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "daos_pool": "tank", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_alienstore": "0/5", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_allow_any_pool_priority": "false", 2024-04-29T14:09:01.104 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_asok": "1/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_asok_assert_abort": "false", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_asserts_on_shutdown": "true", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_auth": "1/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_bdev": "1/3", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_bluefs": "1/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_bluestore": "1/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_buffer": "0/1", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_ceph_dedup": "0/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_ceph_exporter": "1/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_cephfs_mirror": "0/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_cephsqlite": "0/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_client": "0/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_compressor": "1/5", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_context": "0/1", 2024-04-29T14:09:01.105 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_crimson_interrupt": "0/5", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_crush": "1/1", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_crypto": "1/5", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_cyanstore": "0/5", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_deliberately_leak_memory": "false", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_disable_randomized_ping": "false", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_dpdk": "1/5", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_eventtrace": "1/5", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_filer": "0/1", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_filestore": "1/3", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_finisher": "1/1", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_fuse": "1/5", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_heartbeat_testing_span": "0", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_heartbeatmap": "1/5", 2024-04-29T14:09:01.106 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_javaclient": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_journal": "1/3", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_journaler": "0/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_kstore": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_lockdep": "0/1", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mclock": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_balancer": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_locker": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_log": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_log_expire": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_migrator": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_quiesce": "3/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_memstore": "1/5", 2024-04-29T14:09:01.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mgr": "2/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mgrc": "1/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mon": "20/20", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_monc": "0/10", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_ms": "1/1", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_none": "0/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_objclass": "0/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_objectcacher": "0/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_objecter": "0/1", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_optracker": "0/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_osd": "1/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_paxos": "20/20", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_perfcounter": "1/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_prioritycache": "1/5", 2024-04-29T14:09:01.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rados": "0/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd": "0/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd_mirror": "0/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd_pwl": "0/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd_replay": "0/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_refs": "0/0", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_reserver": "1/1", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw": "1/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_access": "1/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_datacache": "1/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_dbstore": "1/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_flight": "1/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_notification": "1/5", 2024-04-29T14:09:01.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_sync": "1/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rocksdb": "4/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_backref": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_cache": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_cleaner": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_device": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_epm": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_journal": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_lba": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_odata": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_omap": "0/5", 2024-04-29T14:09:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_onode": "0/5", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_t": "0/5", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_tm": "0/5", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_striper": "0/1", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_test": "0/5", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_throttle": "1/1", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_timer": "0/1", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_tp": "0/5", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_trace": "1/5", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "device_failure_prediction_mode": "none", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_graylog": "false", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_journald": "false", 2024-04-29T14:09:01.111 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_stderr": "true", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_syslog": "false", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event_tracing": "false", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_addr": "0.0.0.0", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_http_port": "9926", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_prio_limit": "5", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_sort_metrics": "true", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_stats_period": "5", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "fake_statfs_for_testing": "0", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "fatal_signal_handlers": "true", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "filer_max_purge_ops": "10", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "filer_max_truncate_ops": "128", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_apply_finisher_threads": "1", 2024-04-29T14:09:01.112 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_blackhole": "false", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_btrfs_clone_range": "true", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_btrfs_snap": "true", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_caller_concurrency": "10", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_collect_device_partition_information": "true", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_commit_timeout": "600.000000", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_debug_inject_read_err": "false", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_debug_omap_check": "false", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_debug_verify_split": "false", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_dump_file": "", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fadvise": "true", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fail_eio": "true", 2024-04-29T14:09:01.113 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fd_cache_shards": "16", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fd_cache_size": "128", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fiemap": "false", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fiemap_threshold": "4096", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_inject_stall": "0", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_journal_parallel": "false", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_journal_trailing": "false", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_journal_writeahead": "false", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_kill_at": "0", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-29T14:09:01.114 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs": "0", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size": "0", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_merge_threshold": "-10", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_odsync_write": "false", 2024-04-29T14:09:01.115 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_omap_backend": "rocksdb", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_omap_backend_path": "", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_op_thread_timeout": "60", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_op_threads": "2", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_punch_hole": "false", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-29T14:09:01.116 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_ops": "50", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_seek_data_hole": "false", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_sloppy_crc": "false", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_splice": "false", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_split_multiple": "2", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_split_rand_factor": "20", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_update_to": "1000", 2024-04-29T14:09:01.117 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_enable": "true", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_xfs_extsize": "false", 2024-04-29T14:09:01.118 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_zfs_snap": "false", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fio_dir": "/tmp/fio", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_allow_other": "true", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_atomic_o_trunc": "true", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_big_writes": "true", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_debug": "false", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_default_permissions": "false", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_disable_pagecache": "false", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_max_write": "0", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_multithreaded": "true", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_require_active_mds": "true", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_set_user_groups": "true", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_splice_move": "true", 2024-04-29T14:09:01.119 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_splice_read": "true", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_splice_write": "true", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_use_invalidate_cb": "true", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "gss_target_name": "ceph", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_file": "", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_inject_failure": "0", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_interval": "5", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "host": "smithi040", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-29T14:09:01.120 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "inject_early_sigterm": "false", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "jaeger_agent_port": "6799", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "jaeger_tracing_enable": "false", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_aio": "true", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_align_min_size": "65536", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_block_align": "true", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_block_size": "4096", 2024-04-29T14:09:01.121 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_dio": "true", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_discard": "false", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_force_aio": "false", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_ignore_corruption": "false", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_max_write_bytes": "10485760", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_max_write_entries": "100", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_replay_from": "0", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_write_header_frequency": "0", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_zero_on_create": "false", 2024-04-29T14:09:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: "journaler_prefetch_periods": "10", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "journaler_prezero_periods": "5", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "journaler_write_head_interval": "15", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "keyfile": "", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_backend": "rocksdb", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_default_stripe_size": "65536", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_fsck_on_mount": "false", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_max_bytes": "67108864", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_max_ops": "512", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_nid_prealloc": "1024", 2024-04-29T14:09:01.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_onode_map_size": "1024", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_sync_submit_transaction": "false", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_sync_transaction": "false", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "librados_thread_count": "2", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "lockdep": "false", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "lockdep_force_backtrace": "false", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_coarse_timestamps": "true", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_flush_on_exit": "false", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_graylog_host": "127.0.0.1", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_graylog_port": "12201", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_max_new": "1000", 2024-04-29T14:09:01.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_max_recent": "10000", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "log_stderr_prefix": "", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "log_stop_at_utilization": "0.970000", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": "true", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_graylog": "false", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_journald": "false", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_stderr": "false", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_syslog": "false", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max_rotating_auth_attempts": "10", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_action_on_write_error": "1", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_alternate_name_max": "8192", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_asio_thread_count": "2", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_export_pin": "true", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_dirs": "true", 2024-04-29T14:09:01.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_interval": "5", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_interval": "10", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_max": "-1", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_max_until": "-1", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_merge_size": "50", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_midchunk": "0.300000", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_min_start": "0.200000", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_minchunk": "0.001000", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_mode": "0", 2024-04-29T14:09:01.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_need_max": "1.200000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_need_min": "0.800000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_overload_epochs": "2", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_bits": "3", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_size": "10000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_target_decay": "10.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_beacon_grace": "15.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_beacon_interval": "4.000000", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-29T14:09:01.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_mid": "0.700000", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_quiesce_delay": "0", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_release_free_interval": "10", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_reservation": "0.050000", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_trim_interval": "1", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_trim_threshold": "262144", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-29T14:09:01.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_client_prealloc_inos": "1000", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_connect_bootstrapping": "false", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_damage_table_max_entries": "10000", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_auth_pins": "false", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_frag": "false", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_scatterstat": "false", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_subtrees": "false", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_decay_halflife": "5.000000", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_default_dir_hash": "2", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_defer_session_stale": "true", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_deny_all_reconnect": "false", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_keys_per_op": "16384", 2024-04-29T14:09:01.129 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_max_commit_size": "10", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_max_entries": "0", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_prefetch": "true", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_on_map": "false", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_early_reply": "true", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_enable_op_tracker": "true", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_enforce_unique_name": "true", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_random": "true", 2024-04-29T14:09:01.130 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_health_summarize_threshold": "10", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_health_dummy": "false", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_migrator_session_race": "false", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-29T14:09:01.131 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_join_fs": "", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_journal_format": "1", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_create_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_export_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_import_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_journal_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_journal_expire_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_journal_replay_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_link_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_mdstable_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_openc_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_rename_at": "0", 2024-04-29T14:09:01.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_shutdown_at": "0", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_event_large_threshold": "524288", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_events_per_segment": "1024", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_max_events": "-1", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_max_segments": "128", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_pause": "false", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_segment_size": "0", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_trim_threshold": "128", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-29T14:09:01.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_warn_factor": "2.000000", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_caps_per_client": "1048576", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_completed_flushes": "100000", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_completed_requests": "100000", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_export_size": "20971520", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_file_recover": "32", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_purge_files": "64", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_purge_ops": "8192", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_snaps_per_dir": "100", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_metrics_update_interval": "2", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_min_caps_per_client": "100", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_min_caps_working_set": "10000", 2024-04-29T14:09:01.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_numa_node": "-1", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_complaint_time": "30.000000", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_duration": "600", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_size": "20", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_slow_op_size": "20", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_log_threshold": "5", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_ping_grace": "15", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_ping_interval": "5", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_max_caps": "30000", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-29T14:09:01.135 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_warning_threshold": "262144", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_replay_interval": "1.000000", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_root_ino_gid": "0", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_root_ino_uid": "0", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-29T14:09:01.136 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_shutdown_check": "0", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_skip_ino": "0", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_snap_max_uid": "4294967294", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_snap_min_uid": "0", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_snap_rstat": "false", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_standby_replay_damaged": "false", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_symlink_recovery": "true", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-29T14:09:01.137 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_thrash_exports": "0", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_thrash_fragments": "0", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_tick_interval": "5.000000", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_valgrind_exit": "false", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_verify_backtrace": "1", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_verify_scatter": "false", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_wipe_sessions": "false", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool_debug": "false", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_device_bytes": "1073741824", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_page_set": "false", 2024-04-29T14:09:01.138 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_page_size": "65536", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_client_bytes": "134217728", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_client_messages": "512", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_disabled_modules": "", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_enable_op_tracker": "true", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_max_pg_creating": "1024", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_max_pg_num_change": "128", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mds_bytes": "134217728", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mds_messages": "128", 2024-04-29T14:09:01.139 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mon_bytes": "134217728", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mon_messages": "128", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_num_op_tracker_shard": "32", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_op_complaint_time": "30.000000", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_op_history_duration": "600", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_op_history_size": "20", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_op_history_slow_op_size": "20", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_op_log_threshold": "5", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_osd_bytes": "536870912", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_osd_messages": "8192", 2024-04-29T14:09:01.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_pool": "true", 2024-04-29T14:09:01.141 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-29T14:09:01.141 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_standby_modules": "true", 2024-04-29T14:09:01.141 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_stats_period": "5", 2024-04-29T14:09:01.141 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_stats_threshold": "5", 2024-04-29T14:09:01.141 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_tick_period": "2", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_allow_pool_delete": "true", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_allow_pool_size_one": "false", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_auth_validate_all_caps": "true", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_bytes": "104857600", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_directed_command_retry": "5", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-29T14:09:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_parallel": "3", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_log_interval": "1.000000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_ping_interval": "10.000000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_target_rank": "-1", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_level": "debug", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_file": "true", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-29T14:09:01.143 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_journald": "true", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_compact_on_bootstrap": "false", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_compact_on_start": "false", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_compact_on_trim": "true", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cpu_threads": "4", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_crush_min_required_version": "hammer", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_daemon_bytes": "419430400", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-29T14:09:01.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data_avail_crit": "5", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data_avail_warn": "5", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data_size_warn": "16106127360", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_dump_json": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_dump_transactions": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_extra_checks": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_require_reef": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_require_squid": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-29T14:09:01.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_down_mkfs_grace": "60", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_down_uptime_grace": "60", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_election_default_strategy": "1", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_election_timeout": "5.000000", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_elector_ping_divisor": "2", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_enable_op_tracker": "true", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_fake_pool_delete": "false", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_force_quorum_join": "false", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_globalid_prealloc": "10000", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_detail_to_clog": "true", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_log_update_period": "5", 2024-04-29T14:09:01.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_max_detail": "50", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_to_clog": "true", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_to_clog_interval": "600", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_host": "172.21.15.40:6789/0", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_host_override": "", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_initial_members": "", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-29T14:09:01.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_lease": "5.000000", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_log_full_interval": "50", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_log_max": "10000", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_log_max_summary": "50", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_log_epochs": "500", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_osd": "10000", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_pg_per_osd": "10000", 2024-04-29T14:09:01.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_pool_pg_num": "65536", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mds_force_trim_to": "0", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mds_skip_sanity": "false", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_memory_autotune": "true", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_memory_target": "2147483648", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_beacon_grace": "30", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_digest_period": "5", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_inactive_grace": "60", 2024-04-29T14:09:01.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_min_osdmap_epochs": "50", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_complaint_time": "30", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_duration": "600", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_size": "20", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_slow_op_size": "20", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_log_threshold": "5", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_auto_mark_in": "false", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-29T14:09:01.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_cache_size": "500", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_down_out_interval": "600", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_force_trim_to": "0", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-29T14:09:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_min_down_reporters": "2", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_report_timeout": "900", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-29T14:09:01.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_stuck_threshold": "60", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_probe_timeout": "2.000000", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_max_change": "0.050000", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_max_osds": "4", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-29T14:09:01.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_interval": "300", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_max_keys": "100", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_timeout": "300", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_session_timeout": "300", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_smart_report_timeout": "5", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stat_smooth_intervals": "6", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_pool_min_size": "2", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_pool_size": "4", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-29T14:09:01.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_debug": "false", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_provider_kill_at": "0", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_requester_kill_at": "0", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_timeout": "60.000000", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_target_pg_per_osd": "100", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_tick_interval": "5", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_timecheck_interval": "300.000000", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_use_min_delay_socket": "false", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_older_version_delay": "604800", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-29T14:09:01.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_misplaced": "false", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_older_version": "true", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-29T14:09:01.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": "", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-29T14:09:01.157 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_tracing_enabled": "false", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_op_threads": "3", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_cm": "false", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_device_name": "", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_dscp": "96", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_local_gid": "", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_port_num": "1", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-29T14:09:01.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_sl": "3", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_support_srq": "true", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_type": "ib", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_reap_threshold": "5", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_before_connect": "false", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_ipv4": "true", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_ipv6": "false", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_msgr1": "true", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_msgr2": "false", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_port_max": "7568", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_port_min": "6800", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-29T14:09:01.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_retry_count": "3", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_retry_delay": "5", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_client": "false", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_mds": "false", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_mgr": "false", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_mon": "false", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_osd": "false", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_client_mode": "crc secure", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_cluster_mode": "crc secure", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_cluster_type": "", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_compress_secure": "false", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_connection_idle_timeout": "900", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_connection_ready_timeout": "10", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_crc_data": "true", 2024-04-29T14:09:01.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_crc_header": "true", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_bad_msg": "false", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_bug": "true", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_old_message": "true", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_skipped_message": "false", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_coremask": "0xF", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_enable_tso": "true", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_hugepages": "", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-29T14:09:01.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_lro": "true", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_memory_channel": "4", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_pmd": "", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_port_id": "0", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dump_on_send": "false", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_initial_backoff": "0.200000", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_delay_max": "1.000000", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_delay_type": "", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-29T14:09:01.162 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_network_congestion": "0", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_socket_failures": "2500", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_learn_addr_from_peer": "true", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_max_accept_failures": "4", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_max_backoff": "15.000000", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_mon_client_mode": "secure crc", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_mon_service_mode": "secure crc", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_osd_compress_min_size": "1024", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_osd_compress_mode": "none", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_pq_min_cost": "65536", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_public_type": "", 2024-04-29T14:09:01.163 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_service_mode": "crc secure", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_listen_backlog": "512", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_nodelay": "true", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_rcvbuf": "0", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_type": "async+posix", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "no_config_file": "false", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_completion_locks_per_session": "32", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_inflight_ops": "1024", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_tick_interval": "5.000000", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_timeout": "10.000000", 2024-04-29T14:09:01.164 INFO:teuthology.orchestra.run.smithi040.stdout: "objectstore_blackhole": "false", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "openssl_engine_opts": "", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_delay_time": "5.000000", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_hist_halflife": "1000", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_max_low_ops": "2", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_max_ops": "4", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_slop": "0.020000", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_asio_thread_count": "2", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_async_recovery_min_cost": "100", 2024-04-29T14:09:01.165 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backfill_scan_max": "512", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backfill_scan_min": "64", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backoff_on_degraded": "false", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backoff_on_peering": "false", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backoff_on_unfound": "true", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_beacon_report_interval": "300", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_duration": "30", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_max_block_size": "67108864", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_blkin_trace_all": "false", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-29T14:09:01.166 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_check_for_log_corruption": "false", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_update_on_start": "true", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_message_cap": "256", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_message_size_cap": "524288000", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_op_priority": "63", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_watch_timeout": "30", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_command_max_records": "256", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_compact_on_start": "false", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-29T14:09:01.167 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_update_on_start": "true", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_update_weight_set": "true", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_feed_pullee": "-1", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_misdirected_ops": "false", 2024-04-29T14:09:01.168 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_no_acting_change": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_no_purge_strays": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_op_order": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_shutdown": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_trim_objects": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_snaps": "false", 2024-04-29T14:09:01.169 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_keys": "1024", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_stride": "524288", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_default_notify_timeout": "30", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep": "0.000000", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-29T14:09:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_discard_disconnected_ops": "true", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_enable_op_tracker": "true", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_info": "true", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_shutdown": "true", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_function_tracing": "false", 2024-04-29T14:09:01.171 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_grace": "20", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_interval": "6", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_min_peers": "10", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_min_size": "2000", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_stale": "600", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_hit_set_max_size": "100000", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_hit_set_min_size": "1000", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-29T14:09:01.172 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_journal_size": "5120", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_kill_backfill_at": "0", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_cache_size": "50", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_dedup": "true", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_message_max": "40", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_share_max_epochs": "40", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_attr_name_len": "100", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_attr_size": "0", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_backfills": "1", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_markdown_count": "5", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_markdown_period": "600", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_object_name_len": "2048", 2024-04-29T14:09:01.173 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_object_namespace_len": "256", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_object_size": "134217728", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pg_blocked_by": "16", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pg_log_entries": "10000", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pgls": "1024", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_push_cost": "8388608", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_push_objects": "10", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_scrubs": "3", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_trimming_pgs": "2", 2024-04-29T14:09:01.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_write_op_reply_len": "64", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_write_size": "90", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_profile": "balanced", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-29T14:09:01.175 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_base": "805306368", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_cache_min": "134217728", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_target": "4294967296", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_target_autotune": "false", 2024-04-29T14:09:01.176 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_min_pg_log_entries": "250", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_min_recovery_priority": "0", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_report_interval": "5", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_num_cache_shards": "32", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_num_op_tracker_shard": "32", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_numa_auto_affinity": "true", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_numa_node": "-1", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_numa_prefer_iface": "true", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-29T14:09:01.177 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objecter_finishers": "1", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objectstore": "bluestore", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objectstore_fuse": "false", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objectstore_tracing": "false", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_complaint_time": "30.000000", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_duration": "600", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_size": "20", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_slow_op_size": "20", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_log_threshold": "5", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_shards": "0", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_shards_hdd": "5", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_shards_ssd": "8", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-29T14:09:01.178 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_pq_min_cost": "65536", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_queue_cut_off": "high", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_thread_timeout": "15", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_open_classes_on_start": "true", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_os_flags": "0", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_peering_op_priority": "255", 2024-04-29T14:09:01.179 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_delete_cost": "1048576", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_delete_priority": "5", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_log_trim_max": "10000", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_log_trim_min": "100", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-29T14:09:01.180 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_crimson": "false", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flags": "0", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_min_size": "2", 2024-04-29T14:09:01.181 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_pg_num": "32", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_pgp_num": "0", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_read_ratio": "70", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_size": "2", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_type": "replicated", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_push_per_object_cost": "1000", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recover_clone_overlap": "true", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_cost": "20971520", 2024-04-29T14:09:01.182 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_active": "0", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_single_start": "1", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_op_priority": "3", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_priority": "5", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep": "0.000000", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-29T14:09:01.183 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_repair_during_recovery": "false", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_requested_scrub_priority": "120", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_auto_repair": "false", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_begin_hour": "0", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_begin_week_day": "0", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_chunk_max": "25", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_chunk_min": "5", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_cost": "52428800", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_during_recovery": "false", 2024-04-29T14:09:01.184 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_end_hour": "0", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_end_week_day": "0", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_event_cost": "4096", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_invalid_stats": "true", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_max_preemptions": "5", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_priority": "5", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_sleep": "0.000000", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-29T14:09:01.185 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_skip_data_digest": "false", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_smart_report_timeout": "5", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_cost": "1048576", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_priority": "5", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-29T14:09:01.186 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_target_transaction_size": "30", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tracing": "false", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_use_stale_snap": "false", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osdc_blkin_trace_all": "false", 2024-04-29T14:09:01.187 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_kill_at": "0", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_max_join_drift": "10", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_min": "500", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_min_wait": "0.050000", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_propose_interval": "1.000000", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_service_trim_max": "500", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_service_trim_min": "10", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_stash_full_interval": "25", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_trim_max": "500", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_trim_min": "250", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "perf": "true", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "pid_file": "", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-29T14:09:01.188 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "-", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrv": "", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "public_bind_addr": "-", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "public_network": "", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "public_network_interface": "", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "qat_compressor_enabled": "false", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "qat_compressor_session_max_number": "256", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "rados_mon_op_timeout": "0", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "rados_osd_op_timeout": "0", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "rados_tracing": "false", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_atime_update_interval": "60", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-29T14:09:01.189 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_balance_parent_reads": "false", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_balance_snap_reads": "false", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_blkin_trace_all": "false", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache": "true", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_policy": "writearound", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_size": "33554432", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_clone_copy_on_read": "false", 2024-04-29T14:09:01.190 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_compression_hint": "none", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_concurrent_management_ops": "10", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_clone_format": "auto", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_data_pool": "", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_features": "61", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_format": "2", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_map_options": "", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_order": "22", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_pool": "rbd", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_stripe_count": "0", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_stripe_unit": "0", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-29T14:09:01.191 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_enable_alloc_hint": "true", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_io_scheduler": "simple", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_order": "24", 2024-04-29T14:09:01.192 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_pool": "", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_splay_width": "4", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_localize_parent_reads": "false", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_localize_snap_reads": "false", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-29T14:09:01.193 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-29T14:09:01.194 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mtime_update_interval": "60", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_non_blocking_aio": "true", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_op_thread_timeout": "60", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_op_threads": "1", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_parent_cache_enabled": "false", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_plugins": "", 2024-04-29T14:09:01.195 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_bps_burst": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_bps_limit": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_exclude_ops": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_iops_burst": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_iops_limit": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-29T14:09:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_read_from_replica_policy": "default", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_skip_partial_discard": "true", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_tracing": "false", 2024-04-29T14:09:01.197 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_validate_names": "true", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_validate_pool": "true", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "restapi_base_url": "", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "restapi_log_level": "", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_account_default_quota_max_size": "-1", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_acl_grants_max_num": "100", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_admin_entry": "admin", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_backend_store": "rados", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_barbican_url": "", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_beast_enable_async": "true", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_counters_cache": "false", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-29T14:09:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cache_enabled": "true", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cache_expiry_interval": "900", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cache_lru_size": "25000", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_config_store": "rados", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_content_length_compat": "false", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_copy_obj_progress": "true", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-29T14:09:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cors_rules_max_num": "100", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cross_domain_policy": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_addr": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_password": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_username": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_require_ssl": "true", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-29T14:09:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_addr": "", 2024-04-29T14:09:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_auth": "token", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_namespace": "", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_prefix": "", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_token_file": "", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_buffersize": "524288", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_low_speed_time": "300", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-29T14:09:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-29T14:09:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_port": "6379", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_changes_size": "1000", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_num_shards": "128", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_window": "30", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-29T14:09:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-29T14:09:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dns_name": "", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dns_s3website_name": "", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dynamic_resharding": "true", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_gc_threads": "true", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_lc_threads": "true", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_ops_log": "false", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_quota_threads": "true", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_static_website": "false", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_usage_log": "false", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enforce_swift_acls": "true", 2024-04-29T14:09:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_exit_timeout_secs": "120", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_expose_bucket": "false", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_extended_http_attrs": "", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_filter": "none", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_frontends": "beast port=7480", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_deferred": "50", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_objs": "32", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_processor_period": "3600", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-29T14:09:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_init_timeout": "300", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_kafka_connection_idle": "300", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_domain": "", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_password": "", 2024-04-29T14:09:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_password_path": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_project": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_tenant": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_token": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_token_path": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_user": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_api_version": "2", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_domain": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_password": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_project": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_user": "", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-29T14:09:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_url": "", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_debug_interval": "-1", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_lock_max_time": "90", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_objs": "32", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_rules": "1000", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_worker": "3", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_thread_delay": "0", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_dnattr": "uid", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-29T14:09:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_searchfilter": "", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_uri": "", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_http_headers": "", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_object_name_utc": "false", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-29T14:09:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_attr_name_len": "0", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_attr_size": "0", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_chunk_size": "4194304", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_listing_results": "1000", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_notify_retries": "10", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_put_param_size": "1048576", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_put_size": "5368709120", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_slo_entries": "1000", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_md_log_max_shards": "64", 2024-04-29T14:09:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_mp_lock_max_time": "600", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_max_gc": "300", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-29T14:09:01.213 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_num_async_rados_threads": "32", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_num_control_oids": "8", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_numa_node": "-1", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_objexp_chunk_size": "100", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_objexp_gc_interval": "600", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-29T14:09:01.214 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_op_thread_timeout": "600", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_op_tracing": "false", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_opa_token": "", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_opa_url": "", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_opa_verify_ssl": "true", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_rados": "false", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_socket_path": "", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-29T14:09:01.215 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_push_interval": "2.000000", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_lanes": "3", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_partitions": "3", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_print_continue": "true", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-29T14:09:01.216 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rados_tracing": "false", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_realm": "", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_realm_id": "", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_region": "", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_request_uri": "", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_batch_size": "64", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-29T14:09:01.217 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_max_aio": "128", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_num_logs": "16", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_thread_interval": "600", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_resolve_cname": "false", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_run_sync_thread": "true", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-29T14:09:01.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_scheduler_type": "throttler", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_script_uri": "", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_service_provider_name": "", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_client_id": "", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_client_secret": "", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_entry": "sts", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_key": "sts", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_min_session_duration": "900", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_token_introspection_url": "", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_account_in_url": "false", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_auth_entry": "auth", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_auth_url": "", 2024-04-29T14:09:01.219 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_custom_header": "", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_need_stats": "true", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_tenant_name": "", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_token_expiration": "86400", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_url": "", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_url_prefix": "swift", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_lease_period": "120", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-29T14:09:01.220 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_thread_pool_size": "512", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_comment": "", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_createby": "", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_encoding": "", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_flag": "false", 2024-04-29T14:09:01.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_origin": "", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_tracker": "", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_trust_forwarded_https": "false", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_max_shards": "32", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_max_user_shards": "1", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_use_opa_authz": "false", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_counters_cache": "false", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-29T14:09:01.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_max_buckets": "1000", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_policies_max_num": "100", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_unique_email": "true", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_verify_ssl": "true", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zone": "", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zone_id": "", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zonegroup": "", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zonegroup_id": "", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-29T14:09:01.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rgwlc_auto_session_clear": "true", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_block_size": "4096", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_size": "536870912", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-29T14:09:01.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_index_type": "binary_search", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_partition_filters": "false", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_perf": "false", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "run_dir": "/var/run/ceph", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_block_create": "true", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_cache_lru_size": "67108864", 2024-04-29T14:09:01.225 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_default_max_object_size": "16777216", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_device_size": "53687091200", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_batch_capacity": "16", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_main_device_type": "SSD", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-29T14:09:01.226 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_segment_size": "67108864", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "set_keepcaps": "false", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "setgroup": "ceph", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "setuser": "ceph", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "setuser_match_path": "", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "thp": "false", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "threadpool_default_timeout": "60", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout: "throttler_perf_counter": "true" 2024-04-29T14:09:01.227 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:09:01.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-29T14:09:02.023 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:02.505 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:02 smithi040 ceph-mon[39581]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:02.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:02 smithi194 ceph-mon[44089]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:02 smithi121 ceph-mon[44533]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:03.786 INFO:teuthology.orchestra.run.smithi040.stdout:mon.a: {} 2024-04-29T14:09:03.797 INFO:teuthology.orchestra.run.smithi040.stdout:mon.c: {} 2024-04-29T14:09:03.808 INFO:teuthology.orchestra.run.smithi040.stdout:mon.b: {} 2024-04-29T14:09:03.809 INFO:teuthology.orchestra.run.smithi040.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-29T14:09:03.809 INFO:teuthology.orchestra.run.smithi040.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-29T14:09:03.809 INFO:teuthology.orchestra.run.smithi040.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-29T14:09:04.441 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-29T14:09:04.452 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:09:04.453 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:09:04.453 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-29T14:09:04.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-04-29T14:09:04.454 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-29T14:09:04.454 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:09:04.454 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:09:04.454 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:09:04.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:09:04.455 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-29T14:09:04.455 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:04.455 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-29T14:09:04.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:09:04.456 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-29T14:09:04.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:09:04.457 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-29T14:09:04.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:09:04.491 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:04.514 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:04.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:04.618 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:04.649 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:04.669 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:04.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:04.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:04.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:04.805 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:04 smithi121 ceph-mon[44533]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:04.805 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/80954518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:09:04.808 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:04.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:04 smithi194 ceph-mon[44089]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:04.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/80954518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:09:04.820 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:04.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:04.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:04 smithi040 ceph-mon[39581]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:04.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/80954518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T14:09:04.865 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:04.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:04.963 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:04.983 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:04.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:05.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:05.140 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:05.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:05.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:05.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:05.274 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:05.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:05.416 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:05.448 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:05.469 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:05.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:05.574 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:05.605 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:05.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:05.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:05.625 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:05.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:05.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:05.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:05.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:05.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:05.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:05.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:05.963 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:05.994 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:06.015 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:06.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:06.119 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:06.147 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:06.166 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:06.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:06.270 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:06.298 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:06.317 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:06.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:06.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:06.453 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:06.481 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:06.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:06 smithi194 ceph-mon[44089]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:06 smithi194 ceph-mon[44089]: osdmap e105: 12 total, 12 up, 12 in 2024-04-29T14:09:06.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:06.610 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:06 smithi040 ceph-mon[39581]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:06 smithi040 ceph-mon[39581]: osdmap e105: 12 total, 12 up, 12 in 2024-04-29T14:09:06.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:06.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:06.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:06.786 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:06.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:06.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:06 smithi121 ceph-mon[44533]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:06.941 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:06 smithi121 ceph-mon[44533]: osdmap e105: 12 total, 12 up, 12 in 2024-04-29T14:09:06.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:06.969 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:08.875 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:08.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:08.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:09.012 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:09.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:08 smithi040 ceph-mon[39581]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:09.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:09.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:09.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:08 smithi121 ceph-mon[44533]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:09.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:09.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:08 smithi194 ceph-mon[44089]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:09.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:09.394 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:09.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:09.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:09.530 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:09.551 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:09.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:09.656 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:09.688 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:09.709 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:09.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:09.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:10.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:09 smithi194 ceph-mon[44089]: osdmap e106: 12 total, 12 up, 12 in 2024-04-29T14:09:10.120 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:09:10.120 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:09:10.120 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:09:10.120 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:09:10.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:09 smithi121 ceph-mon[44533]: osdmap e106: 12 total, 12 up, 12 in 2024-04-29T14:09:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:09 smithi040 ceph-mon[39581]: osdmap e106: 12 total, 12 up, 12 in 2024-04-29T14:09:10.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:09:11.034 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_0' created 2024-04-29T14:09:11.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:10 smithi121 ceph-mon[44533]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:11.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1807278787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-29T14:09:11.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:10 smithi040 ceph-mon[39581]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:11.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1807278787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-29T14:09:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:10 smithi194 ceph-mon[44089]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:11.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1807278787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-29T14:09:11.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 108, 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:09:11.889468+0000", 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:07:18.619339+0000", 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:07:00.005557+0000", 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:09:12.047 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5799936, 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 26, 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 2, 2024-04-29T14:09:12.048 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:09:12.049 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:09:12.050 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:09:12.051 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:09:12.052 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:09:12.053 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:09:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 11.760000228881836, 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 11.760000228881836, 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.17000000178813934, 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 1 2024-04-29T14:09:12.055 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 2, 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_0", 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:09:10.356811+0000", 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 32769, 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool,creating", 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:09:12.056 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:09:12.057 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "107", 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:09:12.058 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:09:12.059 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:09:12.060 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": {}, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.25, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 2.25, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 1, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2.25, 2024-04-29T14:09:12.061 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2.25, 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 1 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.062 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 56, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:09:12.063 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.064 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:09:12.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.067 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:09:12.069 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.070 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:09:12.071 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 20, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.072 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:09:12.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.074 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3368423938", 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3368423938", 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3368423938", 2024-04-29T14:09:12.075 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3368423938", 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.076 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 0, 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.077 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:09:12.078 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:09:12.079 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.080 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 0, 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.081 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:09:12.082 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:09:12.083 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:09:12.084 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 36, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.086 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/3807513901", 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/3807513901", 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3807513901", 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/3807513901", 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.088 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.089 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.090 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:09:12.091 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:09:12.092 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.093 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:09:12.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.096 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:09:12.097 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 64, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.099 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:09:12.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/2130074108", 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/2130074108", 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/2130074108", 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/2130074108", 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:09:12.101 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.102 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.104 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 0, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.106 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.107 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.108 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:09:12.109 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:09:12.110 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 107, 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.111 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:09:12.112 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:09:12.113 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:12.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:07:57.981521+0000", 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:08:15.142873+0000", 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:09:12.116 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:07:54.783307+0000", 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:09:12.117 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:08:55.575826+0000", 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.118 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:08:25.668576+0000", 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:09:04.504934+0000", 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:08:46.007394+0000", 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.120 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:09:10.720929+0000", 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.121 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:08:34.608462+0000", 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:08:50.340341+0000", 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:09:08.440385+0000", 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:09:12.123 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:08:30.814175+0000", 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:09:12.124 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:09:12.125 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:09:12.126 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:09:12.127 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:09:12.127 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:12.127 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:09:12.179 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:11 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:12.180 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:11 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:12.180 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1807278787' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-29T14:09:12.180 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:11 smithi040 ceph-mon[39581]: osdmap e107: 12 total, 12 up, 12 in 2024-04-29T14:09:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:12.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:12.297 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:12.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:11 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:12.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:11 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:12.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1807278787' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-29T14:09:12.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:11 smithi194 ceph-mon[44089]: osdmap e107: 12 total, 12 up, 12 in 2024-04-29T14:09:12.324 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:12.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:11 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:11 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:12.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1807278787' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-29T14:09:12.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:11 smithi121 ceph-mon[44533]: osdmap e107: 12 total, 12 up, 12 in 2024-04-29T14:09:12.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:12.500 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:12.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:12.624 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:12.657 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:12.677 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:12.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:12.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:12.784 INFO:tasks.thrashosds.thrasher:j is {'epoch': 108, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:09:11.889468+0000', 'last_up_change': '2024-04-29T14:07:18.619339+0000', 'last_in_change': '2024-04-29T14:07:00.005557+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-29T14:09:10.356811+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3368423938}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3368423938}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3368423938}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3368423938}]}, 'public_addr': '172.21.15.40:6809/3368423938', 'cluster_addr': '172.21.15.40:6810/3368423938', 'heartbeat_back_addr': '172.21.15.40:6812/3368423938', 'heartbeat_front_addr': '172.21.15.40:6811/3368423938', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 3807513901}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3807513901}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3807513901}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 3807513901}]}, 'public_addr': '172.21.15.121:6804/3807513901', 'cluster_addr': '172.21.15.121:6805/3807513901', 'heartbeat_back_addr': '172.21.15.121:6807/3807513901', 'heartbeat_front_addr': '172.21.15.121:6806/3807513901', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 64, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 2130074108}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 2130074108}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 2130074108}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 2130074108}]}, 'public_addr': '172.21.15.194:6800/2130074108', 'cluster_addr': '172.21.15.194:6801/2130074108', 'heartbeat_back_addr': '172.21.15.194:6803/2130074108', 'heartbeat_front_addr': '172.21.15.194:6802/2130074108', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:07:57.981521+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:08:15.142873+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:07:54.783307+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:08:55.575826+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:08:25.668576+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:09:04.504934+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:08:46.007394+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:09:10.720929+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:08:34.608462+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:08:50.340341+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:09:08.440385+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:08:30.814175+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:09:12.785 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-29T14:09:12.814 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:12.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:12.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:12.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:13.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:12 smithi040 ceph-mon[39581]: pgmap v342: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:13.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:12 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:09:13.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:12 smithi040 ceph-mon[39581]: osdmap e108: 12 total, 12 up, 12 in 2024-04-29T14:09:13.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/177753197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:09:13.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:12 smithi194 ceph-mon[44089]: pgmap v342: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:13.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:12 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:09:13.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:12 smithi194 ceph-mon[44089]: osdmap e108: 12 total, 12 up, 12 in 2024-04-29T14:09:13.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/177753197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:09:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:12 smithi121 ceph-mon[44533]: pgmap v342: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:12 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:09:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:12 smithi121 ceph-mon[44533]: osdmap e108: 12 total, 12 up, 12 in 2024-04-29T14:09:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/177753197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:09:13.363 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:09:13.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:09:13.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:09:13.422 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:09:13.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:09:13.464 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:09:13.485 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:09:13.505 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:09:13.526 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:09:13.546 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:09:13.566 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:09:13.587 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:09:13.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:09:13.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:09:13.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:09:13.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:09:13.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:09:13.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:09:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:09:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:09:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:09:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:09:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:09:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:09:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:13.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:13.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:14.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:14.054 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:14.077 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:14.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:14.161 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:13 smithi040 ceph-mon[39581]: osdmap e109: 12 total, 12 up, 12 in 2024-04-29T14:09:14.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3224914379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:14.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.180 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:14.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:13 smithi194 ceph-mon[44089]: osdmap e109: 12 total, 12 up, 12 in 2024-04-29T14:09:14.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3224914379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:14.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.208 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.210 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:14.230 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:14.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:14.334 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:14.362 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:14.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:13 smithi121 ceph-mon[44533]: osdmap e109: 12 total, 12 up, 12 in 2024-04-29T14:09:14.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3224914379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:14.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:14.381 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:14.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:14.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:14.514 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:14.533 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:14.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:14.637 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:14.665 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:14.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:14.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:14.883 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:14.883 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:14.891 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:14.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:14.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:14.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:15.126 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:15.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: pgmap v345: 17 pgs: 2 creating+activating, 3 creating+peering, 1 active+clean, 11 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:15.259 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: pgmap v345: 17 pgs: 2 creating+activating, 3 creating+peering, 1 active+clean, 11 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:15.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.261 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.262 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:15.283 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:15.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: pgmap v345: 17 pgs: 2 creating+activating, 3 creating+peering, 1 active+clean, 11 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:15.387 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:15.417 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:15.437 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:15.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:15.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:15.571 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:15.591 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:15.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:15.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:15.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:15.914 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:09:15.974 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:16.006 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:16.027 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:16.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:16.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:16.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/830620064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:09:16.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:15 smithi194 ceph-mon[44089]: from='client.24706 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:09:16.161 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:16.182 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:16.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:16.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/830620064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:09:16.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:15 smithi040 ceph-mon[39581]: from='client.24706 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:09:16.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:16.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/830620064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:09:16.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:15 smithi121 ceph-mon[44533]: from='client.24706 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:09:16.317 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:16.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:16.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:16.472 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:16.492 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:16.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:16.598 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:16.624 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:09:16.627 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:16.647 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:16.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:16.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:16.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:16.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:16.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:17.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:16 smithi121 ceph-mon[44533]: pgmap v346: 17 pgs: 12 active+clean, 2 creating+activating, 3 creating+peering; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:17.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:16 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:17.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:16 smithi121 ceph-mon[44533]: from='client.24706 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:09:17.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:16 smithi121 ceph-mon[44533]: osdmap e110: 12 total, 12 up, 12 in 2024-04-29T14:09:17.185 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:17.188 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:16 smithi040 ceph-mon[39581]: pgmap v346: 17 pgs: 12 active+clean, 2 creating+activating, 3 creating+peering; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:17.188 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:16 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:17.188 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:16 smithi040 ceph-mon[39581]: from='client.24706 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:09:17.189 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:16 smithi040 ceph-mon[39581]: osdmap e110: 12 total, 12 up, 12 in 2024-04-29T14:09:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:17.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:17.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:16 smithi194 ceph-mon[44089]: pgmap v346: 17 pgs: 12 active+clean, 2 creating+activating, 3 creating+peering; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:16 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:16 smithi194 ceph-mon[44089]: from='client.24706 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:09:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:16 smithi194 ceph-mon[44089]: osdmap e110: 12 total, 12 up, 12 in 2024-04-29T14:09:17.319 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:17.418 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:17.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:17.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:17.555 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:17.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:17.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:17.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:17.712 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:17.786 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:09:17.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:09:17.786 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-29T14:09:17.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-04-29T14:09:17.863 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:17.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:17.954 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:09:17.955 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:09:17.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:18.022 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:18.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:18.126 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:18.158 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:18.179 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:18.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:18.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:18.310 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:18.505 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:09:18.509 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:18.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:18.576 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:18.613 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:18.645 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:18.666 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:18.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:18.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:18.802 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:18.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:18.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:18.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:18.956 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:19.054 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:19.057 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:18 smithi040 ceph-mon[39581]: pgmap v348: 17 pgs: 17 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:19.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:18 smithi121 ceph-mon[44533]: pgmap v348: 17 pgs: 17 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:19.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:19.190 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:19.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:19.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:19.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:19.215 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:09:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:18 smithi194 ceph-mon[44089]: pgmap v348: 17 pgs: 17 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:19.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:19.343 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:19.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:19.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:19.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:19.499 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:19.520 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:19.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:19.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:19.675 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:19.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:19.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:19.821 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:19.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:19.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:19.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:19.977 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:19.999 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:20.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:20.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:20.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:20.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:20.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:19 smithi121 ceph-mon[44533]: osdmap e111: 12 total, 12 up, 12 in 2024-04-29T14:09:20.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3949457319' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:09:20.130 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:20.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:20.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:20.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:19 smithi040 ceph-mon[39581]: osdmap e111: 12 total, 12 up, 12 in 2024-04-29T14:09:20.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3949457319' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:09:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:20.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:20.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:19 smithi194 ceph-mon[44089]: osdmap e111: 12 total, 12 up, 12 in 2024-04-29T14:09:20.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3949457319' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:09:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-29T14:09:20.562 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:21.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:20 smithi040 ceph-mon[39581]: pgmap v350: 17 pgs: 17 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:21.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3949457319' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:09:21.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:20 smithi040 ceph-mon[39581]: osdmap e112: 12 total, 12 up, 12 in 2024-04-29T14:09:21.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1151578282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:21.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:21.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:21.243 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:20 smithi194 ceph-mon[44089]: pgmap v350: 17 pgs: 17 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:21.243 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3949457319' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:09:21.243 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:20 smithi194 ceph-mon[44089]: osdmap e112: 12 total, 12 up, 12 in 2024-04-29T14:09:21.244 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1151578282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:21.245 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:21.266 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:21.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:21.284 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:09:21.284 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:09:21.284 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:09:21.284 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:09:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:09:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:09:21.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:20 smithi121 ceph-mon[44533]: pgmap v350: 17 pgs: 17 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3949457319' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:09:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:20 smithi121 ceph-mon[44533]: osdmap e112: 12 total, 12 up, 12 in 2024-04-29T14:09:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1151578282' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:21.402 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:21.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:21.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:21.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:21.556 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:21.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:21.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:21.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:21.708 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:21.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:21.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:21.766 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:21.831 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:21.863 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:21.884 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:21.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:21.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:22.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:22.019 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:22.019 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:22.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:22.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:22.107 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-29T14:09:22.109 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:09:22.109 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:09:22.139 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:22.166 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:22.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:22.168 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:22.189 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:22.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:22.225 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:22.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:22.322 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:22.341 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:22.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:22.354 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:09:22.354 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:09:22.354 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_79000 2024-04-29T14:09:22.355 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:09:22.355 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:09:22.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:22.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:22.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:22.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:22.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:22.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:22.945 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:23.049 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:23.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:23.155 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:23.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:23.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:22 smithi194 ceph-mon[44089]: pgmap v352: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:23.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:23.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:23.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:22 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:09:23.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:22 smithi194 ceph-mon[44089]: osdmap e113: 12 total, 12 up, 12 in 2024-04-29T14:09:23.195 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:23.221 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:23.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:23.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:23.355 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 16 5803 5787 361.668 361.688 0.00294378 0.00276022 2024-04-29T14:09:23.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:23.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:22 smithi040 ceph-mon[39581]: pgmap v352: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:23.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:23.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:22 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:09:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:22 smithi040 ceph-mon[39581]: osdmap e113: 12 total, 12 up, 12 in 2024-04-29T14:09:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:22 smithi121 ceph-mon[44533]: pgmap v352: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:23.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:23.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:22 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:09:23.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:22 smithi121 ceph-mon[44533]: osdmap e113: 12 total, 12 up, 12 in 2024-04-29T14:09:23.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:23.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:23.686 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:23.725 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:24.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:09:24.355 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 16 11227 11211 350.325 339 0.00282731 0.00285149 2024-04-29T14:09:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:24 smithi194 ceph-mon[44089]: pgmap v354: 17 pgs: 17 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:25.355 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 16 15142 15126 315.105 244.688 0.00245541 0.00317069 2024-04-29T14:09:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:24 smithi040 ceph-mon[39581]: pgmap v354: 17 pgs: 17 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:25.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:24 smithi121 ceph-mon[44533]: pgmap v354: 17 pgs: 17 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:09:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:25.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:25.490 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:25.532 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:25.557 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:25.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:25.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:25.701 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:25.732 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:25.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:25.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:26.356 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 16 19723 19707 307.834 286.312 0.00257409 0.00324601 2024-04-29T14:09:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:26.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:26.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:26.990 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:27.015 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:27.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:27.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:26 smithi040 ceph-mon[39581]: pgmap v355: 17 pgs: 17 active+clean; 576 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:09:27.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:27.181 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:27.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:27.229 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:27.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:26 smithi194 ceph-mon[44089]: pgmap v355: 17 pgs: 17 active+clean; 576 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:09:27.328 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:27.359 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 16 23591 23575 294.615 241.75 0.00203093 0.00339031 2024-04-29T14:09:27.364 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:26 smithi121 ceph-mon[44533]: pgmap v355: 17 pgs: 17 active+clean; 576 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:09:27.386 INFO:teuthology.orchestra.run.smithi040.stdout:moved 0 / 34 (0%) 2024-04-29T14:09:27.386 INFO:teuthology.orchestra.run.smithi040.stdout:avg 2.83333 2024-04-29T14:09:27.386 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.06719 -> 1.06719 (expected baseline 1.61159) 2024-04-29T14:09:27.386 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.4 with 1 -> 1 pgs (0.352941 -> 0.352941 * mean) 2024-04-29T14:09:27.386 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.1 with 4 -> 4 pgs (1.41176 -> 1.41176 * mean) 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 0.05 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0011 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0012 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 weight 1.0000 -> 0.9500 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 weight 1.0000 -> 0.9500 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 weight 1.0000 -> 0.9500 2024-04-29T14:09:27.387 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 1.0000 -> 0.9500 2024-04-29T14:09:27.390 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:27.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:27.398 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:09:27.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:27.526 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:27.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:27.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:27.613 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:09:27 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:09:27.386+0000 7f4cb5abe700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-04-29T14:09:27.653 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:27.688 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:27.714 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:27.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:27.819 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:27.866 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:27.893 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:27.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:27.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3429477335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:27.951 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:27 smithi040 ceph-mon[39581]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:27.970 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:09:27.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:28.029 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:28.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:28.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:28.155 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:09:28.197 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:28.229 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:09:28.232 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3429477335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:28.232 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:27 smithi194 ceph-mon[44089]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:28.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:09:28.333 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:28.359 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 15 27333 27318 284.375 233.938 0.00399764 0.00351449 2024-04-29T14:09:28.375 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:28.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3429477335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:28.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:27 smithi121 ceph-mon[44533]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:28.400 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:28.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:28.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:28.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:28.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:28.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:28.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:28.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:28.714 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:29.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: pgmap v356: 17 pgs: 17 active+clean; 974 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:09:29.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:29.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:09:29.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: osdmap e114: 12 total, 12 up, 12 in 2024-04-29T14:09:29.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:09:29.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:09:29.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:09:29.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:09:29.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:09:29.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:09:29.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:28 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:09:29.059 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-29T14:09:29.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-04-29T14:09:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: pgmap v356: 17 pgs: 17 active+clean; 974 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:09:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: osdmap e114: 12 total, 12 up, 12 in 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:09:29.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:28 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:09:29.359 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 16 31112 31096 277.482 236.125 0.00549296 0.00360188 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: pgmap v356: 17 pgs: 17 active+clean; 974 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: osdmap e114: 12 total, 12 up, 12 in 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:09:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:28 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:09:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:29.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:29.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:29.480 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:29.524 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:29.550 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:29.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:29.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:29.911 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:29.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:30.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:30.046 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:30.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:30.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:30.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:30.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:30.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:22946 2.4 e113/107 2:2e530d1c:::benchmark_data_smithi194_79000_object22945:head v 113'1394, mlcod=113'1389)", 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:27.166333+0000", 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.1559390970000001, 2024-04-29T14:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00964222, 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166333+0000", 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166333+0000", 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166333+0000", 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.51e-07 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166354+0000", 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1381000000000001e-05 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166361+0000", 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3180000000000003e-06 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166366+0000", 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2220000000000001e-06 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.167308+0000", 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00094194799999999996 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.167329+0000", 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1027999999999999e-05 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.175926+0000", 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0085966180000000003 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.175975+0000", 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9153999999999997e-05 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:22955 2.5 e113/107 2:a24e4c0d:::benchmark_data_smithi194_79000_object22954:head v 113'1400, mlcod=113'1398)", 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:27.166970+0000", 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.1553016349999998, 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0088722469999999998, 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166970+0000", 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166970+0000", 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.166972+0000", 2024-04-29T14:09:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.739e-06 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.167039+0000", 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7602999999999996e-05 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.167060+0000", 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0356999999999998e-05 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.167067+0000", 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7219999999999999e-06 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.167177+0000", 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109849 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.167198+0000", 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0718000000000001e-05 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.175791+0000", 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0085926630000000004 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.175842+0000", 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1595999999999999e-05 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:23316 2.c e113/107 2:38d063b5:::benchmark_data_smithi194_79000_object23315:head v 113'1442, mlcod=113'1440)", 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:27.278596+0000", 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.0436754709999998, 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010269018, 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278596+0000", 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278596+0000", 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278599+0000", 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.424e-06 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278642+0000", 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3594000000000003e-05 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278662+0000", 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9888e-05 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278674+0000", 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1562999999999999e-05 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278765+0000", 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1423999999999997e-05 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.278791+0000", 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5616e-05 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.288801+0000", 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010010249000000001 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:27.288865+0000", 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4259999999999998e-05 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28677 2.c e114/107 2:3f8e8b37:::benchmark_data_smithi194_79000_object28676:head v 114'1751, mlcod=114'1749)", 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.705866+0000", 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.616405587, 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025515797, 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.705866+0000", 2024-04-29T14:09:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.705866+0000", 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.705867+0000", 2024-04-29T14:09:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.378e-06 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708761+0000", 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0028937540000000001 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708781+0000", 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9564000000000001e-05 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708790+0000", 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4190000000000008e-06 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708993+0000", 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020249800000000001 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.709015+0000", 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2336000000000001e-05 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731306+0000", 2024-04-29T14:09:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022290977999999999 2024-04-29T14:09:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731382+0000", 2024-04-29T14:09:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5870000000000004e-05 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28682 2.4 e114/107 2:26359c48:::benchmark_data_smithi194_79000_object28681:head v 114'1742, mlcod=114'1740)", 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.708831+0000", 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6134406669999999, 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022584582999999998, 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708831+0000", 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708831+0000", 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708832+0000", 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5209999999999999e-06 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708855+0000", 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2269e-05 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708864+0000", 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8240000000000006e-06 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708870+0000", 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0820000000000004e-06 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.709493+0000", 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062347800000000001 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.709516+0000", 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2432000000000001e-05 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731386+0000", 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021870239999999999 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731416+0000", 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9737e-05 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28684 2.4 e114/107 2:23bc09c5:::benchmark_data_smithi194_79000_object28683:head v 114'1743, mlcod=114'1740)", 2024-04-29T14:09:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.708879+0000", 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.613392277, 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022565710999999999, 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708879+0000", 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708879+0000", 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708880+0000", 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8999999999999995e-07 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708906+0000", 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5641999999999999e-05 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708913+0000", 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2629999999999997e-06 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.708919+0000", 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8250000000000001e-06 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.712050+0000", 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0031312470000000002 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.712072+0000", 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2011e-05 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731418+0000", 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019345800999999999 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731445+0000", 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7132e-05 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28703 2.c e114/107 2:375103e8:::benchmark_data_smithi194_79000_object28702:head v 114'1752, mlcod=114'1749)", 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.715414+0000", 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6068573079999999, 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016140946, 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715414+0000", 2024-04-29T14:09:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715414+0000", 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715417+0000", 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6510000000000001e-06 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715566+0000", 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014889800000000001 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715574+0000", 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7780000000000007e-06 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715579+0000", 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1030000000000001e-06 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.716671+0000", 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010918919999999999 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.716691+0000", 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0384999999999999e-05 2024-04-29T14:09:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731515+0000", 2024-04-29T14:09:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014823777999999999 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731555+0000", 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0460999999999998e-05 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28702 2.4 e114/107 2:236382af:::benchmark_data_smithi194_79000_object28701:head v 114'1744, mlcod=114'1740)", 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.715446+0000", 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6068252219999999, 2024-04-29T14:09:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016021117000000001, 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715446+0000", 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715446+0000", 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715447+0000", 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5499999999999997e-07 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715484+0000", 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6874e-05 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715502+0000", 2024-04-29T14:09:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7961e-05 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715511+0000", 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5949999999999999e-06 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715653+0000", 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000142492 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.715669+0000", 2024-04-29T14:09:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.592e-05 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731447+0000", 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015777623000000001 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731467+0000", 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0897e-05 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28707 2.4 e114/107 2:26f0b044:::benchmark_data_smithi194_79000_object28706:head v 114'1745, mlcod=114'1740)", 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.718292+0000", 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6039800660000001, 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013294457000000001, 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718292+0000", 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718292+0000", 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718293+0000", 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.886e-06 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718325+0000", 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1550000000000001e-05 2024-04-29T14:09:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718351+0000", 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6318000000000001e-05 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718361+0000", 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0009e-05 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718419+0000", 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7766000000000002e-05 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718439+0000", 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0302e-05 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731558+0000", 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013118306999999999 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731586+0000", 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8319000000000001e-05 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28711 2.4 e114/107 2:20f1a470:::benchmark_data_smithi194_79000_object28710:head v 114'1746, mlcod=114'1740)", 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.718693+0000", 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6035785580000002, 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012920341, 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718693+0000", 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718693+0000", 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718695+0000", 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-06 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718718+0000", 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3008999999999999e-05 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718730+0000", 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2213e-05 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.718737+0000", 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6789999999999999e-06 2024-04-29T14:09:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719456+0000", 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071979599999999998 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719480+0000", 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3730000000000001e-05 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731589+0000", 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012108961999999999 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731613+0000", 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4389999999999999e-05 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28713 2.5 e114/107 2:a2958fba:::benchmark_data_smithi194_79000_object28712:head v 114'1757, mlcod=114'1755)", 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.719460+0000", 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6028115569999999, 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01227194, 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719460+0000", 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719460+0000", 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719461+0000", 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.362e-06 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719503+0000", 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1151e-05 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719513+0000", 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0689e-05 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719520+0000", 2024-04-29T14:09:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8449999999999997e-06 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719653+0000", 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013277300000000001 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719671+0000", 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7787000000000001e-05 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731693+0000", 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012022295000000001 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731732+0000", 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9038000000000001e-05 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28714 2.5 e114/107 2:ac822730:::benchmark_data_smithi194_79000_object28713:head v 114'1758, mlcod=114'1755)", 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.719540+0000", 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6027312760000001, 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012221998, 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719540+0000", 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719540+0000", 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719541+0000", 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1999999999999999e-07 2024-04-29T14:09:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719581+0000", 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9712000000000002e-05 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719592+0000", 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1148000000000001e-05 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.719597+0000", 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3890000000000004e-06 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.720497+0000", 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089983700000000001 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.720514+0000", 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6602000000000001e-05 2024-04-29T14:09:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731735+0000", 2024-04-29T14:09:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011221200000000001 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731762+0000", 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7489999999999999e-05 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:28719 2.c e114/107 2:3b9dd735:::benchmark_data_smithi194_79000_object28718:head v 114'1753, mlcod=114'1749)", 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:28.721798+0000", 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.6004739429999999, 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098460449999999995, 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721798+0000", 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721798+0000", 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721807+0000", 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4169999999999998e-06 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721882+0000", 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5092999999999994e-05 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721898+0000", 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5579000000000001e-05 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721907+0000", 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7690000000000002e-06 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721969+0000", 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2230000000000006e-05 2024-04-29T14:09:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.721992+0000", 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3728000000000001e-05 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731618+0000", 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0096252590000000006 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:28.731644+0000", 2024-04-29T14:09:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.597e-05 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:31193 2.c e114/107 2:3f5e5b9c:::benchmark_data_smithi194_79000_object31192:head v 114'1909, mlcod=114'1907)", 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:29.390847+0000", 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.931424209, 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012050046999999999, 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.390847+0000", 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.390847+0000", 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.390850+0000", 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1799999999999999e-06 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.390892+0000", 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2670000000000003e-05 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.390911+0000", 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8394e-05 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.390921+0000", 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0801e-05 2024-04-29T14:09:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.391027+0000", 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000105073 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.391051+0000", 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4878000000000001e-05 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.402842+0000", 2024-04-29T14:09:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011791062999999999 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.402897+0000", 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4988000000000003e-05 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:31200 2.c e114/107 2:3eb10c6c:::benchmark_data_smithi194_79000_object31199:head v 114'1910, mlcod=114'1907)", 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:29.393932+0000", 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.92833917899999996, 2024-04-29T14:09:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01110677, 2024-04-29T14:09:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.393932+0000", 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.393932+0000", 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.393934+0000", 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.685e-06 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.393969+0000", 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4431999999999998e-05 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.393985+0000", 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6144e-05 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.393994+0000", 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-06 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.394093+0000", 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8479000000000004e-05 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.394116+0000", 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.319e-05 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.404991+0000", 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010875473 2024-04-29T14:09:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.405039+0000", 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8007000000000003e-05 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:31207 2.c e114/107 2:3014e454:::benchmark_data_smithi194_79000_object31206:head v 114'1911, mlcod=114'1907)", 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:29.395420+0000", 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.92685209599999996, 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0096512690000000005, 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395420+0000", 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395420+0000", 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395421+0000", 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2550000000000001e-06 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395451+0000", 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0184000000000001e-05 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395465+0000", 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4289e-05 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395473+0000", 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0839999999999992e-06 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395563+0000", 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9506000000000007e-05 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.395585+0000", 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2317999999999999e-05 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.405043+0000", 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094574459999999996 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.405071+0000", 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8187e-05 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:31208 2.4 e114/107 2:28260a0d:::benchmark_data_smithi194_79000_object31207:head v 114'1897, mlcod=114'1895)", 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:29.396573+0000", 2024-04-29T14:09:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.92569888600000005, 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091801609999999992, 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396573+0000", 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396573+0000", 2024-04-29T14:09:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396574+0000", 2024-04-29T14:09:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3769999999999999e-06 2024-04-29T14:09:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396600+0000", 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5922999999999999e-05 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396610+0000", 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5170000000000005e-06 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396617+0000", 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0790000000000003e-06 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396832+0000", 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021530900000000001 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.396853+0000", 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1316999999999999e-05 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.405703+0000", 2024-04-29T14:09:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.008849272 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.405753+0000", 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0367000000000001e-05 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:31423 2.4 e114/107 2:2a56cea6:::benchmark_data_smithi194_79000_object31422:head v 114'1908, mlcod=114'1904)", 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:29.459619+0000", 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.86265259599999999, 2024-04-29T14:09:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0088967960000000002, 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.459619+0000", 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.459619+0000", 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.459620+0000", 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8100000000000001e-07 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.459648+0000", 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8036999999999999e-05 2024-04-29T14:09:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.459658+0000", 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6840000000000008e-06 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.459665+0000", 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1799999999999999e-06 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.460855+0000", 2024-04-29T14:09:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011906149999999999 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.460873+0000", 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7697999999999999e-05 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.468460+0000", 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0075870069999999998 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.468516+0000", 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5693999999999999e-05 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:31485 2.5 e114/107 2:a87f44bd:::benchmark_data_smithi194_79000_object31484:head v 114'1924, mlcod=114'1922)", 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:29.482925+0000", 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.83934658699999998, 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010108628, 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.482925+0000", 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.482925+0000", 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.482927+0000", 2024-04-29T14:09:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.854e-06 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483035+0000", 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010861000000000001 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483055+0000", 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9196000000000001e-05 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483065+0000", 2024-04-29T14:09:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8309999999999996e-06 2024-04-29T14:09:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483114+0000", 2024-04-29T14:09:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9908999999999999e-05 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483130+0000", 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5549e-05 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.492974+0000", 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098440000000000003 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.493034+0000", 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9679000000000002e-05 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:31487 2.5 e114/107 2:a4f3cde6:::benchmark_data_smithi194_79000_object31486:head v 114'1925, mlcod=114'1922)", 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:29.483087+0000", 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.83918461899999997, 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0099736740000000001, 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:09:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483087+0000", 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483087+0000", 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:09:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.483088+0000", 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8100000000000001e-07 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.484205+0000", 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001116906 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.484214+0000", 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8119999999999997e-06 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.484221+0000", 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1879999999999996e-06 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.484348+0000", 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012747999999999999 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.484369+0000", 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0712000000000001e-05 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.493037+0000", 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086681599999999994 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:29.493061+0000", 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3535000000000001e-05 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:09:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:09:30.417 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:09:30.417 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:30.418 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 16 34605 34589 270.088 218.312 0.00324636 0.00370029 2024-04-29T14:09:30.464 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:30.490 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:30.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:30.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:30.642 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:30.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:09:30.903 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:30.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:31.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:31.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:31.193 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:30 smithi040 ceph-mon[39581]: pgmap v358: 17 pgs: 17 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.41k op/s 2024-04-29T14:09:31.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:31.306 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:30 smithi194 ceph-mon[44089]: pgmap v358: 17 pgs: 17 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.41k op/s 2024-04-29T14:09:31.359 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 16 37821 37805 262.396 201 0.00473288 0.00380903 2024-04-29T14:09:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:30 smithi121 ceph-mon[44533]: pgmap v358: 17 pgs: 17 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.41k op/s 2024-04-29T14:09:31.416 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:31.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:31.522 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:31.564 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:31.594 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:31.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:31.620 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:09:31.659 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:09:31.694 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:09:31.698 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:31.725 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:09:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:31.764 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:09:31.810 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:09:31.844 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:09:31.874 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:09:31.909 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:09:31.952 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:09:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:31.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:32.002 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/959605485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:32.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:09:32.004 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:32.007 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:09:32.024 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:32.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:32.063 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:09:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:09:32.094 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:32.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:32.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:32.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/959605485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:32.258 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:32.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:32.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:32.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/959605485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:32.359 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 16 40735 40719 254.371 182.125 0.00380581 0.00392982 2024-04-29T14:09:32.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:32.411 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:32.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:32.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:32.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:32.611 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:32.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:32.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:32.755 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:32.780 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:32.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:32.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:32.920 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:32.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:32.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:33.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:33.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: pgmap v359: 17 pgs: 17 active+clean; 2.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 MiB/s wr, 3.41k op/s 2024-04-29T14:09:33.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: pgmap v359: 17 pgs: 17 active+clean; 2.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 MiB/s wr, 3.41k op/s 2024-04-29T14:09:33.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.092 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.092 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.092 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.092 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.092 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.095 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:33.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:33.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:33.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 43837 43821 248.872 193.875 0.00238353 0.00401591 2024-04-29T14:09:33.361 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:33.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: pgmap v359: 17 pgs: 17 active+clean; 2.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 MiB/s wr, 3.41k op/s 2024-04-29T14:09:33.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:34.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 46919 46903 244.185 192.625 0.00496739 0.00409317 2024-04-29T14:09:34.626 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:34.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:34.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:34.765 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:34.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:34.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:34.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:35.021 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:35.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:35.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:34 smithi121 ceph-mon[44533]: pgmap v360: 17 pgs: 17 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-04-29T14:09:35.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:09:35.168 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:35.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:09:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:35.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:09:35.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:35.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:34 smithi194 ceph-mon[44089]: pgmap v360: 17 pgs: 17 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-04-29T14:09:35.324 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:35.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:34 smithi040 ceph-mon[39581]: pgmap v360: 17 pgs: 17 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-04-29T14:09:35.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:35.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:35.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 16 49832 49816 239.407 182.062 0.00246487 0.00417545 2024-04-29T14:09:35.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:35.477 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:35.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:35.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:35.769 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:35.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:35.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:35.912 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:35.941 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:35.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:36.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:36.079 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:36.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:36.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:36.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 16 52424 52408 233.876 162 0.00694286 0.00427439 2024-04-29T14:09:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:36.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:36.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:36.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:36 smithi040 ceph-mon[39581]: pgmap v361: 17 pgs: 17 active+clean; 2.8 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 225 MiB/s wr, 3.60k op/s 2024-04-29T14:09:36.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2791641497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:09:36.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:36 smithi040 ceph-mon[39581]: from='client.34751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:09:36.981 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]: dispatch 2024-04-29T14:09:36.981 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:36 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]: dispatch 2024-04-29T14:09:36.991 INFO:teuthology.orchestra.run.smithi040.stdout:moved 2 / 34 (5.88235%) 2024-04-29T14:09:36.991 INFO:teuthology.orchestra.run.smithi040.stdout:avg 2.83333 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.06719 -> 1.06719 (expected baseline 1.61159) 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.4 with 1 -> 2 pgs (0.352941 -> 0.705882 * mean) 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.1 with 4 -> 4 pgs (1.41176 -> 1.41176 * mean) 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0051 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0057 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 weight 1.0000 -> 0.6640 2024-04-29T14:09:36.992 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 weight 1.0000 -> 0.7824 2024-04-29T14:09:36.993 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 weight 1.0000 -> 0.7888 2024-04-29T14:09:36.993 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 1.0000 -> 0.7947 2024-04-29T14:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:09:37.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:09:37.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:36 smithi194 ceph-mon[44089]: pgmap v361: 17 pgs: 17 active+clean; 2.8 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 225 MiB/s wr, 3.60k op/s 2024-04-29T14:09:37.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2791641497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:09:37.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:36 smithi194 ceph-mon[44089]: from='client.34751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:09:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]: dispatch 2024-04-29T14:09:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:36 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]: dispatch 2024-04-29T14:09:37.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 16 54266 54250 225.961 115.125 0.0019633 0.0043261 2024-04-29T14:09:37.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:36 smithi121 ceph-mon[44533]: pgmap v361: 17 pgs: 17 active+clean; 2.8 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 225 MiB/s wr, 3.60k op/s 2024-04-29T14:09:37.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2791641497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:09:37.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:36 smithi121 ceph-mon[44533]: from='client.34751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:09:37.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]: dispatch 2024-04-29T14:09:37.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:36 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]: dispatch 2024-04-29T14:09:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:38.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:37 smithi040 ceph-mon[39581]: from='client.14862 v1:172.21.15.40:0/4282183709' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:09:38.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:37 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:38.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:37 smithi040 ceph-mon[39581]: from='client.34751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:09:38.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:37 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]': finished 2024-04-29T14:09:38.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:37 smithi040 ceph-mon[39581]: osdmap e115: 12 total, 12 up, 12 in 2024-04-29T14:09:38.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:37 smithi194 ceph-mon[44089]: from='client.14862 v1:172.21.15.40:0/4282183709' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:09:38.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:37 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:38.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:37 smithi194 ceph-mon[44089]: from='client.34751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:09:38.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:37 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]': finished 2024-04-29T14:09:38.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:37 smithi194 ceph-mon[44089]: osdmap e115: 12 total, 12 up, 12 in 2024-04-29T14:09:38.329 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:38.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:38.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 16 54266 54250 211.841 0 - 0.0043261 2024-04-29T14:09:38.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:37 smithi121 ceph-mon[44533]: from='client.14862 v1:172.21.15.40:0/4282183709' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:09:38.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:37 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:09:38.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:37 smithi121 ceph-mon[44533]: from='client.34751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:09:38.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:37 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'51693','7':'52079','8':'43516','9':'51274'}"}]': finished 2024-04-29T14:09:38.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:37 smithi121 ceph-mon[44533]: osdmap e115: 12 total, 12 up, 12 in 2024-04-29T14:09:38.433 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:38.465 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:38.487 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:38.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:38.591 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:09:38.620 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:38.640 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:09:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:38.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:09:38.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:38.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:38.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:38.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:38.923 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:38.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:39.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:39.075 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:39.094 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:39.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:39.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:39 smithi121 ceph-mon[44533]: pgmap v363: 17 pgs: 17 active+clean; 3.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 227 MiB/s wr, 3.62k op/s 2024-04-29T14:09:39.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:39 smithi121 ceph-mon[44533]: osdmap e116: 12 total, 12 up, 12 in 2024-04-29T14:09:39.198 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:39.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:39 smithi040 ceph-mon[39581]: pgmap v363: 17 pgs: 17 active+clean; 3.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 227 MiB/s wr, 3.62k op/s 2024-04-29T14:09:39.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:39 smithi040 ceph-mon[39581]: osdmap e116: 12 total, 12 up, 12 in 2024-04-29T14:09:39.226 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:39.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:39.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:39 smithi194 ceph-mon[44089]: pgmap v363: 17 pgs: 17 active+clean; 3.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 227 MiB/s wr, 3.62k op/s 2024-04-29T14:09:39.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:39 smithi194 ceph-mon[44089]: osdmap e116: 12 total, 12 up, 12 in 2024-04-29T14:09:39.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 54440 54424 200.022 5.4375 0.00254006 0.00465013 2024-04-29T14:09:39.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:39.678 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-29T14:09:39.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-04-29T14:09:39.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:09:40.139 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:40.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:40.245 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:40.274 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:40.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 54440 54424 188.912 0 - 0.00465013 2024-04-29T14:09:40.475 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:09:40.486 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:09:40.506 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:09:40.525 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:09:40.545 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:09:40.565 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:09:40.585 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:09:40.605 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:09:40.626 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:09:40.646 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:09:40.667 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:09:40.687 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:09:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:09:40.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:09:40.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:09:40.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:09:40.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:09:40.723 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:40.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:40.737 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:09:40.737 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:09:40.737 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:09:40.737 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:09:40.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:40.857 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:40.955 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:40.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:41.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:41.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:41.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:09:41.253 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:41 smithi121 ceph-mon[44533]: pgmap v365: 17 pgs: 17 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-29T14:09:41.253 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/327519286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:41.254 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.254 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.254 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.257 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:41.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:09:41.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:41.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:09:41.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:41 smithi194 ceph-mon[44089]: pgmap v365: 17 pgs: 17 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-29T14:09:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/327519286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:41 smithi040 ceph-mon[39581]: pgmap v365: 17 pgs: 17 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-04-29T14:09:41.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/327519286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:41.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:41.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 55197 55181 181.461 23.6562 0.0148835 0.005509 2024-04-29T14:09:41.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:41.418 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:41.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:41.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:41.547 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:09:41.590 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:41.678 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:09:41.682 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:41.692 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:41.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:09:41.795 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:41.838 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:42.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.301 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.301 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.301 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.301 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.360 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:09:42.367510 min lat: 0.00107894 max lat: 4.06786 avg lat: 0.00557817 2024-04-29T14:09:42.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:09:42.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 15 57372 57357 179.188 136 0.00274969 0.00557817 2024-04-29T14:09:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:42.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:42.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:42.850 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:43.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:43 smithi040 ceph-mon[39581]: pgmap v366: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s; 3334/109052 objects degraded (3.057%); 3083/109052 objects misplaced (2.827%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:09:43.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:43 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 3334/109052 objects degraded (3.057%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:09:43.127 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:43.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:43.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:43.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:43 smithi194 ceph-mon[44089]: pgmap v366: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s; 3334/109052 objects degraded (3.057%); 3083/109052 objects misplaced (2.827%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:09:43.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:43 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 3334/109052 objects degraded (3.057%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:09:43.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 59635 59619 177.387 141.375 0.00394721 0.00563504 2024-04-29T14:09:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:43 smithi121 ceph-mon[44533]: pgmap v366: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s; 3334/109052 objects degraded (3.057%); 3083/109052 objects misplaced (2.827%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:09:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:43 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 3334/109052 objects degraded (3.057%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:09:43.649 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:43.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:43.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:09:43.756 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:43.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:43.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:43.931 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:43.971 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:44.003 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:44.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:44.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:44.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:09:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:44.280 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:44.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:44.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 15 61711 61696 175.225 129.812 0.0130543 0.00570461 2024-04-29T14:09:44.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:44.459 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:44.574 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:09:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:44.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:09:44.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:44.742 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:44.900 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:44.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:45.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:45.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:45 smithi040 ceph-mon[39581]: pgmap v367: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 3334/109666 objects degraded (3.040%); 3083/109666 objects misplaced (2.811%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:09:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:45 smithi194 ceph-mon[44089]: pgmap v367: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 3334/109666 objects degraded (3.040%); 3083/109666 objects misplaced (2.811%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:09:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:45.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:45.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 63571 63555 172.658 116.188 0.00295494 0.00578984 2024-04-29T14:09:45.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:45 smithi121 ceph-mon[44533]: pgmap v367: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 3334/109666 objects degraded (3.040%); 3083/109666 objects misplaced (2.811%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:09:45.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:09:45.484 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:45.509 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:09:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:45.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:09:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:45.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:09:45.679 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:moved 4 / 34 (11.7647%) 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:avg 2.83333 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.06719 -> 0.986013 (expected baseline 1.61159) 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.6 with 1 -> 2 pgs (0.352941 -> 0.705882 * mean) 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.7 with 5 -> 4 pgs (1.76471 -> 1.41176 * mean) 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 1 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0071 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0078 2024-04-29T14:09:46.009 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 weight 0.6640 -> 0.4686 2024-04-29T14:09:46.010 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 weight 0.7888 -> 0.5862 2024-04-29T14:09:46.010 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 0.7947 -> 0.5964 2024-04-29T14:09:46.010 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 weight 0.7824 -> 0.6562 2024-04-29T14:09:46.013 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:09:46.035 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:09:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4034397697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:46 smithi194 ceph-mon[44089]: from='client.24754 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:46.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 16 65594 65578 170.732 126.438 0.00618697 0.00585556 2024-04-29T14:09:46.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4034397697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:46.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:46 smithi040 ceph-mon[39581]: from='client.24754 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:46.363 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:09:46 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:09:46.007+0000 7f4cb5abe700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-04-29T14:09:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4034397697' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:46 smithi121 ceph-mon[44533]: from='client.24754 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:09:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:47 smithi194 ceph-mon[44089]: pgmap v368: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3334/120140 objects degraded (2.775%); 2536/120140 objects misplaced (2.111%); 6.1 MiB/s, 98 objects/s recovering 2024-04-29T14:09:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:47 smithi194 ceph-mon[44089]: from='client.14874 v1:172.21.15.40:0/3646159290' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:09:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:47 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:09:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:47 smithi194 ceph-mon[44089]: from='client.24754 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:09:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:47 smithi194 ceph-mon[44089]: osdmap e117: 12 total, 12 up, 12 in 2024-04-29T14:09:47.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 15 67785 67770 169.383 137 0.00706477 0.00590209 2024-04-29T14:09:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:47 smithi040 ceph-mon[39581]: pgmap v368: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3334/120140 objects degraded (2.775%); 2536/120140 objects misplaced (2.111%); 6.1 MiB/s, 98 objects/s recovering 2024-04-29T14:09:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:47 smithi040 ceph-mon[39581]: from='client.14874 v1:172.21.15.40:0/3646159290' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:09:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:47 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:09:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:47 smithi040 ceph-mon[39581]: from='client.24754 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:09:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:47 smithi040 ceph-mon[39581]: osdmap e117: 12 total, 12 up, 12 in 2024-04-29T14:09:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:47 smithi121 ceph-mon[44533]: pgmap v368: 17 pgs: 1 active+recovering, 1 active+recovery_wait+degraded, 15 active+clean; 3.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3334/120140 objects degraded (2.775%); 2536/120140 objects misplaced (2.111%); 6.1 MiB/s, 98 objects/s recovering 2024-04-29T14:09:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:47 smithi121 ceph-mon[44533]: from='client.14874 v1:172.21.15.40:0/3646159290' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:09:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:47 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:09:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:47 smithi121 ceph-mon[44533]: from='client.24754 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:09:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:47 smithi121 ceph-mon[44533]: osdmap e117: 12 total, 12 up, 12 in 2024-04-29T14:09:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:09:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:47.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:09:47.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:47.701 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:48.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:48 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:48.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:48 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:48.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:48 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:48.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:48 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:48.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:48 smithi040 ceph-mon[39581]: osdmap e118: 12 total, 12 up, 12 in 2024-04-29T14:09:48.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:48 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:48.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:48 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:48.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:48 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:48.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:48 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:48.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:48 smithi194 ceph-mon[44089]: osdmap e118: 12 total, 12 up, 12 in 2024-04-29T14:09:48.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 69649 69633 167.346 116.438 0.00677379 0.00597364 2024-04-29T14:09:48.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:48 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:48.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:48 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:48.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:48 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:48.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:48 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:48.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:48 smithi121 ceph-mon[44533]: osdmap e118: 12 total, 12 up, 12 in 2024-04-29T14:09:48.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:48.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:48.491 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-29T14:09:48.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-04-29T14:09:48.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:09:48.576 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:48.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:09:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:48.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:09:48.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:48.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:48.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:48.956 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:48.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:48.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:49.085 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:09:49.118 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:49.143 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:09:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:49.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:09:49.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:49.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:49 smithi040 ceph-mon[39581]: pgmap v370: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s; 5546/128002 objects misplaced (4.333%); 7.6 MiB/s, 122 objects/s recovering 2024-04-29T14:09:49.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:49 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3334/120140 objects degraded (2.775%), 1 pg degraded) 2024-04-29T14:09:49.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:49 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:09:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:49.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:49 smithi194 ceph-mon[44089]: pgmap v370: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s; 5546/128002 objects misplaced (4.333%); 7.6 MiB/s, 122 objects/s recovering 2024-04-29T14:09:49.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:49 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3334/120140 objects degraded (2.775%), 1 pg degraded) 2024-04-29T14:09:49.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:49 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:09:49.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 16 71481 71465 165.389 114.5 0.0074315 0.00604476 2024-04-29T14:09:49.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:49 smithi121 ceph-mon[44533]: pgmap v370: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s; 5546/128002 objects misplaced (4.333%); 7.6 MiB/s, 122 objects/s recovering 2024-04-29T14:09:49.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:49 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3334/120140 objects degraded (2.775%), 1 pg degraded) 2024-04-29T14:09:49.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:49 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:09:49.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:09:49.811 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:09:49.846 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:09:49.882 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:09:49.923 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:09:49.958 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:09:49.996 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:09:50.031 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:09:50.066 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:09:50.101 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:09:50.134 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:09:50.160 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:09:50.171 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:50.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:50.183 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:09:50.220 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:09:50.229 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:09:50.229 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:09:50.229 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:09:50.229 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:09:50.229 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:09:50.229 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:09:50.229 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:09:50.230 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:09:50.230 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:09:50.230 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:09:50.230 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:09:50.230 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:09:50.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:09:50.309 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:50 smithi194 ceph-mon[44089]: pgmap v372: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 5546/129170 objects misplaced (4.294%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:09:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/477592902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:50.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:09:50.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:50 smithi121 ceph-mon[44533]: pgmap v372: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 5546/129170 objects misplaced (4.294%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:09:50.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/477592902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:50.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:09:50.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 16 73250 73234 163.431 110.562 0.0094199 0.00611707 2024-04-29T14:09:50.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:50 smithi040 ceph-mon[39581]: pgmap v372: 17 pgs: 2 active+recovering, 15 active+clean; 3.9 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 5546/129170 objects misplaced (4.294%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:09:50.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/477592902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:09:50.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:09:50.494 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 75231 75215 162.064 123.812 0.0119822 0.00616902 2024-04-29T14:09:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:52.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 16 77183 77167 160.729 122 0.00312798 0.00622006 2024-04-29T14:09:52.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:52.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:52 smithi040 ceph-mon[39581]: pgmap v373: 17 pgs: 2 active+recovering, 15 active+clean; 4.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s; 4682/145130 objects misplaced (3.226%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:09:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:52 smithi121 ceph-mon[44533]: pgmap v373: 17 pgs: 2 active+recovering, 15 active+clean; 4.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s; 4682/145130 objects misplaced (3.226%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:09:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:09:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:52 smithi194 ceph-mon[44089]: pgmap v373: 17 pgs: 2 active+recovering, 15 active+clean; 4.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s; 4682/145130 objects misplaced (3.226%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:09:53.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 16 79027 79011 159.262 115.25 0.00891006 0.00627723 2024-04-29T14:09:54.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 81068 81052 158.271 127.562 0.00360973 0.006317 2024-04-29T14:09:54.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:54 smithi194 ceph-mon[44089]: pgmap v374: 17 pgs: 2 active+recovering, 15 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.70k op/s; 4682/147428 objects misplaced (3.176%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:09:54.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:54 smithi040 ceph-mon[39581]: pgmap v374: 17 pgs: 2 active+recovering, 15 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.70k op/s; 4682/147428 objects misplaced (3.176%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:09:54.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:54 smithi121 ceph-mon[44533]: pgmap v374: 17 pgs: 2 active+recovering, 15 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.70k op/s; 4682/147428 objects misplaced (3.176%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:09:55.361 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 82974 82958 157.085 119.125 0.00872996 0.0063646 2024-04-29T14:09:56.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 16 84785 84769 155.793 113.188 0.00312436 0.00641516 2024-04-29T14:09:56.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:56 smithi194 ceph-mon[44089]: pgmap v375: 17 pgs: 2 active+recovering, 15 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.02k op/s; 4133/159492 objects misplaced (2.591%); 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:09:56.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:56 smithi040 ceph-mon[39581]: pgmap v375: 17 pgs: 2 active+recovering, 15 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.02k op/s; 4133/159492 objects misplaced (2.591%); 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:09:56.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:56 smithi121 ceph-mon[44533]: pgmap v375: 17 pgs: 2 active+recovering, 15 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.02k op/s; 4133/159492 objects misplaced (2.591%); 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:09:57.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 86706 86690 154.772 120.062 0.00560314 0.00645975 2024-04-29T14:09:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:09:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:09:57.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:09:57.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:09:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:09:58.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 88561 88545 153.693 115.938 0.00695422 0.00650492 2024-04-29T14:09:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:58 smithi194 ceph-mon[44089]: pgmap v376: 17 pgs: 2 active+recovering, 15 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s; 3770/166588 objects misplaced (2.263%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:09:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:09:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:09:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:58 smithi040 ceph-mon[39581]: pgmap v376: 17 pgs: 2 active+recovering, 15 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s; 3770/166588 objects misplaced (2.263%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:09:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:09:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:09:58.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:58 smithi121 ceph-mon[44533]: pgmap v376: 17 pgs: 2 active+recovering, 15 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s; 3770/166588 objects misplaced (2.263%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:09:58.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:09:58.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:09:59.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 90293 90277 152.465 108.25 0.0164522 0.00655759 2024-04-29T14:09:59.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:09:59.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:09:59 smithi194 ceph-mon[44089]: osdmap e119: 12 total, 12 up, 12 in 2024-04-29T14:09:59.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:09:59 smithi040 ceph-mon[39581]: osdmap e119: 12 total, 12 up, 12 in 2024-04-29T14:09:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:09:59 smithi121 ceph-mon[44533]: osdmap e119: 12 total, 12 up, 12 in 2024-04-29T14:10:00.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 92004 91988 151.267 106.938 0.0090609 0.00660943 2024-04-29T14:10:00.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:00 smithi194 ceph-mon[44089]: pgmap v378: 17 pgs: 2 active+recovering, 15 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.92k op/s; 3770/167680 objects misplaced (2.248%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:10:00.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:00 smithi194 ceph-mon[44089]: overall HEALTH_OK 2024-04-29T14:10:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:00 smithi040 ceph-mon[39581]: pgmap v378: 17 pgs: 2 active+recovering, 15 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.92k op/s; 3770/167680 objects misplaced (2.248%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:10:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:00 smithi040 ceph-mon[39581]: overall HEALTH_OK 2024-04-29T14:10:00.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:00 smithi121 ceph-mon[44533]: pgmap v378: 17 pgs: 2 active+recovering, 15 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.92k op/s; 3770/167680 objects misplaced (2.248%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:10:00.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:00 smithi121 ceph-mon[44533]: overall HEALTH_OK 2024-04-29T14:10:00.985 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:01.069 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:01.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:01.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:01.244 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:01.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:01.363 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 93655 93639 150.034 103.188 0.00421617 0.00666411 2024-04-29T14:10:01.375 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:01.424 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:01.572 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:01.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:01.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:01.935 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:10:01.935 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:10:02.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:10:02.369155 min lat: 0.00107894 max lat: 4.06786 avg lat: 0.00671395 2024-04-29T14:10:02.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:10:02.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 95326 95310 148.894 104.438 0.00560755 0.00671395 2024-04-29T14:10:02.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:02 smithi194 ceph-mon[44089]: pgmap v379: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s; 2758/182490 objects misplaced (1.511%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:10:02.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:02.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:02.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:02 smithi040 ceph-mon[39581]: pgmap v379: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s; 2758/182490 objects misplaced (1.511%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:10:02.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:02.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:02.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:02 smithi121 ceph-mon[44533]: pgmap v379: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s; 2758/182490 objects misplaced (1.511%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:10:02.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:02.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:03.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 97055 97039 147.898 108.062 0.0144461 0.00675958 2024-04-29T14:10:03.514 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:03.778 INFO:teuthology.orchestra.run.smithi040.stdout:moved 0 / 34 (0%) 2024-04-29T14:10:03.778 INFO:teuthology.orchestra.run.smithi040.stdout:avg 2.83333 2024-04-29T14:10:03.778 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.06719 -> 1.06719 (expected baseline 1.61159) 2024-04-29T14:10:03.778 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.6 with 1 -> 1 pgs (0.352941 -> 0.352941 * mean) 2024-04-29T14:10:03.778 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.7 with 5 -> 5 pgs (1.76471 -> 1.76471 * mean) 2024-04-29T14:10:03.778 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:10:03.778 INFO:teuthology.orchestra.run.smithi040.stdout:oload 1000 2024-04-29T14:10:03.779 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:10:03.779 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:10:03.779 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0111 2024-04-29T14:10:03.779 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.1110 2024-04-29T14:10:03.785 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:10:03.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:03 smithi194 ceph-mon[44089]: osdmap e120: 12 total, 12 up, 12 in 2024-04-29T14:10:03.815 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:03.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:03.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:03 smithi040 ceph-mon[39581]: osdmap e120: 12 total, 12 up, 12 in 2024-04-29T14:10:03.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:10:03.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:03 smithi121 ceph-mon[44533]: osdmap e120: 12 total, 12 up, 12 in 2024-04-29T14:10:03.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:03.957 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:03.981 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:03.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:04.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:04.147 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:04.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:04.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:04.282 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:04.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:04.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:04.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 16 98624 98608 146.711 98.0625 0.00404783 0.00681437 2024-04-29T14:10:04.408 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:04.447 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:04.481 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:04.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:04.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:04 smithi121 ceph-mon[44533]: pgmap v381: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 2758/184692 objects misplaced (1.493%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:10:04.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:04 smithi121 ceph-mon[44533]: from='client.34778 v1:172.21.15.40:0/3865821678' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:10:04.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:04 smithi040 ceph-mon[39581]: pgmap v381: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 2758/184692 objects misplaced (1.493%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:10:04.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:04 smithi040 ceph-mon[39581]: from='client.34778 v1:172.21.15.40:0/3865821678' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:10:04.588 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:04.624 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:04.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:04.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:04.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:04.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:04 smithi194 ceph-mon[44089]: pgmap v381: 17 pgs: 2 active+recovering, 15 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 2758/184692 objects misplaced (1.493%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:10:04.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:04 smithi194 ceph-mon[44089]: from='client.34778 v1:172.21.15.40:0/3865821678' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:10:04.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:04.822 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:04.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:04.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:04.987 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-29T14:10:04.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-04-29T14:10:05.013 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:05.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:05.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:05.240 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:05.265 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:05.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:05.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 15 100237 100222 145.645 100.875 0.00336034 0.00686382 2024-04-29T14:10:05.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:05.421 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:06.191 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:06.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:06.210 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:06.247 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:06.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:10:06.335 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:06.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 15 101797 101782 144.551 97.5 0.00585513 0.00691657 2024-04-29T14:10:06.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:10:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:06.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:10:06.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:10:06.499 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:06.524 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:10:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:06.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:10:06.591 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:10:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:06 smithi121 ceph-mon[44533]: pgmap v382: 17 pgs: 2 active+recovering, 15 active+clean; 5.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s; 1917/194950 objects misplaced (0.983%); 14 MiB/s, 231 objects/s recovering 2024-04-29T14:10:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:06 smithi121 ceph-mon[44533]: osdmap e121: 12 total, 12 up, 12 in 2024-04-29T14:10:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3504663386' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:10:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:06 smithi121 ceph-mon[44533]: from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:10:06.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:06.658 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:06.686 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:06.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:06.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:06.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:06.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:06.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:06 smithi194 ceph-mon[44089]: pgmap v382: 17 pgs: 2 active+recovering, 15 active+clean; 5.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s; 1917/194950 objects misplaced (0.983%); 14 MiB/s, 231 objects/s recovering 2024-04-29T14:10:06.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:06 smithi194 ceph-mon[44089]: osdmap e121: 12 total, 12 up, 12 in 2024-04-29T14:10:06.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3504663386' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:10:06.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:06 smithi194 ceph-mon[44089]: from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:10:06.821 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:06.847 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:06.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:06.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:06.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:06 smithi040 ceph-mon[39581]: pgmap v382: 17 pgs: 2 active+recovering, 15 active+clean; 5.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s; 1917/194950 objects misplaced (0.983%); 14 MiB/s, 231 objects/s recovering 2024-04-29T14:10:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:06 smithi040 ceph-mon[39581]: osdmap e121: 12 total, 12 up, 12 in 2024-04-29T14:10:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3504663386' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:10:06.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:06 smithi040 ceph-mon[39581]: from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:10:06.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:07.009 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:07.362 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 103429 103413 143.604 101.938 0.00624976 0.00696241 2024-04-29T14:10:07.783 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:07.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:07.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:07.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:10:07.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: osdmap e122: 12 total, 12 up, 12 in 2024-04-29T14:10:07.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:07.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:07.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: osdmap e123: 12 total, 12 up, 12 in 2024-04-29T14:10:07.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2338358502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:07.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:07.834 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:07.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: osdmap e122: 12 total, 12 up, 12 in 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: osdmap e123: 12 total, 12 up, 12 in 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2338358502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:07.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:07.893 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:10:07.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:10:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:10:07.921 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:07.921 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:10:07.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: osdmap e122: 12 total, 12 up, 12 in 2024-04-29T14:10:07.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:07.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:07.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: osdmap e123: 12 total, 12 up, 12 in 2024-04-29T14:10:07.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2338358502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:07.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:07.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:07.923 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:07.951 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:07.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:08.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:10:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:10:08.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:08.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:10:08.228 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:08.259 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:08.284 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:08.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:08.363 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 105106 105090 142.76 104.812 0.00702505 0.00700323 2024-04-29T14:10:08.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:08.431 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:08.462 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:08.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:08.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:10:08.603 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:08.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:10:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:08.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:10:08.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:08.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: pgmap v386: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s; 1516/201098 objects misplaced (0.754%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:10:08.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.799 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:08.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: pgmap v386: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s; 1516/201098 objects misplaced (0.754%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:10:08.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:08.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: pgmap v386: 17 pgs: 2 active+recovering, 15 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s; 1516/201098 objects misplaced (0.754%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:10:09.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:09.363 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 106788 106772 141.96 105.125 0.00776314 0.00704276 2024-04-29T14:10:09.473 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:09.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:09.581 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:09.620 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:09.645 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:09.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:09.752 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:09.788 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:09.814 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:09.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:09.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:09.967 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:10.363 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 15 108472 108457 141.196 105.312 0.0120162 0.00708112 2024-04-29T14:10:10.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:10:10.635 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:10 smithi040 ceph-mon[39581]: pgmap v387: 17 pgs: 2 active+recovering, 15 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 1516/202136 objects misplaced (0.750%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:10:10.639 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:10.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:10.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:10.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:10 smithi121 ceph-mon[44533]: pgmap v387: 17 pgs: 2 active+recovering, 15 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 1516/202136 objects misplaced (0.750%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:10:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:10.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:11.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:11.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:10 smithi194 ceph-mon[44089]: pgmap v387: 17 pgs: 2 active+recovering, 15 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 1516/202136 objects misplaced (0.750%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:10:11.073 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:11.162 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:11.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:10:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:11.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:11.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:11.363 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 109823 109807 140.036 84.375 0.00694345 0.00713897 2024-04-29T14:10:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:10:11.414 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:10:11.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:10:11.415 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:10:11.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.415 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12809 2.6 2:6808fb47:::benchmark_data_smithi194_79000_object12808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:11.415 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.670785+0000", 2024-04-29T14:10:11.415 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 46.574756958000002, 2024-04-29T14:10:11.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054130532000000002, 2024-04-29T14:10:11.415 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.416 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.416 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.416 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.416 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12809 2024-04-29T14:10:11.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.417 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670785+0000", 2024-04-29T14:10:11.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670785+0000", 2024-04-29T14:10:11.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670787+0000", 2024-04-29T14:10:11.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3480000000000001e-06 2024-04-29T14:10:11.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670882+0000", 2024-04-29T14:10:11.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5150999999999998e-05 2024-04-29T14:10:11.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670893+0000", 2024-04-29T14:10:11.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1243999999999999e-05 2024-04-29T14:10:11.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670898+0000", 2024-04-29T14:10:11.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2599999999999996e-06 2024-04-29T14:10:11.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670932+0000", 2024-04-29T14:10:11.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3772999999999997e-05 2024-04-29T14:10:11.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671003+0000", 2024-04-29T14:10:11.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0900000000000002e-05 2024-04-29T14:10:11.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 8", 2024-04-29T14:10:11.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671079+0000", 2024-04-29T14:10:11.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6499000000000001e-05 2024-04-29T14:10:11.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673120+0000", 2024-04-29T14:10:11.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020410670000000001 2024-04-29T14:10:11.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724808+0000", 2024-04-29T14:10:11.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051687342999999997 2024-04-29T14:10:11.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724868+0000", 2024-04-29T14:10:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0705000000000002e-05 2024-04-29T14:10:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724916+0000", 2024-04-29T14:10:11.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7241999999999999e-05 2024-04-29T14:10:11.428 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.428 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.428 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12810 2.6 2:66b8be45:::benchmark_data_smithi194_79000_object12809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.670915+0000", 2024-04-29T14:10:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 46.574627571000001, 2024-04-29T14:10:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054135546, 2024-04-29T14:10:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12810 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670915+0000", 2024-04-29T14:10:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670915+0000", 2024-04-29T14:10:11.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670915+0000", 2024-04-29T14:10:11.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1900000000000002e-07 2024-04-29T14:10:11.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670956+0000", 2024-04-29T14:10:11.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1270999999999998e-05 2024-04-29T14:10:11.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670962+0000", 2024-04-29T14:10:11.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4260000000000002e-06 2024-04-29T14:10:11.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670966+0000", 2024-04-29T14:10:11.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9380000000000002e-06 2024-04-29T14:10:11.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671412+0000", 2024-04-29T14:10:11.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044643100000000001 2024-04-29T14:10:11.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671493+0000", 2024-04-29T14:10:11.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0353999999999997e-05 2024-04-29T14:10:11.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 8", 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671568+0000", 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5694999999999999e-05 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724757+0000", 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053189073000000003 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724969+0000", 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000211444 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725016+0000", 2024-04-29T14:10:11.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6740000000000003e-05 2024-04-29T14:10:11.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725050+0000", 2024-04-29T14:10:11.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4554999999999997e-05 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12811 2.6 2:6a427467:::benchmark_data_smithi194_79000_object12810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.670976+0000", 2024-04-29T14:10:11.440 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 46.574566515999997, 2024-04-29T14:10:11.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054205036999999998, 2024-04-29T14:10:11.441 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.441 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.441 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.441 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.441 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.442 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.442 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12811 2024-04-29T14:10:11.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.442 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670976+0000", 2024-04-29T14:10:11.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670976+0000", 2024-04-29T14:10:11.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.670976+0000", 2024-04-29T14:10:11.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5900000000000002e-07 2024-04-29T14:10:11.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671049+0000", 2024-04-29T14:10:11.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2490000000000006e-05 2024-04-29T14:10:11.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671068+0000", 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9252000000000001e-05 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671074+0000", 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1229999999999997e-06 2024-04-29T14:10:11.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671888+0000", 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00081376500000000004 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671965+0000", 2024-04-29T14:10:11.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7460999999999996e-05 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 8", 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672062+0000", 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6618000000000002e-05 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724763+0000", 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052701694 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725094+0000", 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00033019800000000003 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725139+0000", 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5745000000000003e-05 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725181+0000", 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1232000000000001e-05 2024-04-29T14:10:11.449 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12812 2.6 2:6aa843b6:::benchmark_data_smithi194_79000_object12811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.671082+0000", 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 46.574460137000003, 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054245266, 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.450 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12812 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671082+0000", 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671082+0000", 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671083+0000", 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4399999999999999e-07 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671116+0000", 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3297000000000003e-05 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671124+0000", 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1669999999999999e-06 2024-04-29T14:10:11.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.671130+0000", 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2830000000000001e-06 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672382+0000", 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001252647 2024-04-29T14:10:11.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672538+0000", 2024-04-29T14:10:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015553400000000001 2024-04-29T14:10:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 8", 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672676+0000", 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000138558 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724927+0000", 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052251038999999999 2024-04-29T14:10:11.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725241+0000", 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031332700000000002 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725292+0000", 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1164e-05 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725327+0000", 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5405999999999999e-05 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12820 2.6 2:63913e05:::benchmark_data_smithi194_79000_object12819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.672662+0000", 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 46.572879802000003, 2024-04-29T14:10:11.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052801672000000001, 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12820 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672662+0000", 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672662+0000", 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672663+0000", 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1659999999999999e-06 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672696+0000", 2024-04-29T14:10:11.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2205000000000001e-05 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672703+0000", 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9360000000000002e-06 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.672712+0000", 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8490000000000001e-06 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673167+0000", 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00045447899999999998 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673296+0000", 2024-04-29T14:10:11.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012902399999999999 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 8", 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673392+0000", 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5920000000000003e-05 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724932+0000", 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051540526000000003 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725374+0000", 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044173299999999998 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725419+0000", 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5037999999999998e-05 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725464+0000", 2024-04-29T14:10:11.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4796000000000003e-05 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12825 2.6 2:6762d7e4:::benchmark_data_smithi194_79000_object12824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.673011+0000", 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 46.572531040000001, 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054509706999999998, 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.465 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12825 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673011+0000", 2024-04-29T14:10:11.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673011+0000", 2024-04-29T14:10:11.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673012+0000", 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2920000000000001e-06 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673074+0000", 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1799999999999998e-05 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673087+0000", 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2670999999999999e-05 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673095+0000", 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7279999999999995e-06 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673779+0000", 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00068489400000000002 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.673887+0000", 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107496 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 8", 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.674079+0000", 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192387 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.725572+0000", 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051492774999999998 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.727443+0000", 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018706409999999999 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.727486+0000", 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3098e-05 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.727521+0000", 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4925000000000002e-05 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.472 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68545 2.4 2:22502e78:::benchmark_data_smithi194_79000_object68544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.750062+0000", 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.495479967000001, 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20905254600000001, 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.473 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68545 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750062+0000", 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750062+0000", 2024-04-29T14:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750064+0000", 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.381e-06 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750095+0000", 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1191e-05 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750104+0000", 2024-04-29T14:10:11.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9199999999999993e-06 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750115+0000", 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1025e-05 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750171+0000", 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6585000000000001e-05 2024-04-29T14:10:11.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750298+0000", 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012642400000000001 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.750436+0000", 2024-04-29T14:10:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013812199999999999 2024-04-29T14:10:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.753959+0000", 2024-04-29T14:10:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035235190000000001 2024-04-29T14:10:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.959037+0000", 2024-04-29T14:10:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20507795500000001 2024-04-29T14:10:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.959083+0000", 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5818000000000001e-05 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.959115+0000", 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1606000000000002e-05 2024-04-29T14:10:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68565 2.4 2:2a8e3cfb:::benchmark_data_smithi194_79000_object68564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.758212+0000", 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.487330348, 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.200979306, 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68565 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758212+0000", 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758212+0000", 2024-04-29T14:10:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758214+0000", 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0209999999999998e-06 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758241+0000", 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7279000000000001e-05 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758254+0000", 2024-04-29T14:10:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3328e-05 2024-04-29T14:10:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758262+0000", 2024-04-29T14:10:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9410000000000007e-06 2024-04-29T14:10:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758337+0000", 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5007999999999997e-05 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758464+0000", 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012656000000000001 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.758600+0000", 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013606299999999999 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.763297+0000", 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046973529999999996 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.959142+0000", 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19584433000000001 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.959170+0000", 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8464000000000001e-05 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.959191+0000", 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0959e-05 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68577 2.4 2:2354d368:::benchmark_data_smithi194_79000_object68576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.766070+0000", 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.4794725, 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19477202599999999, 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.489 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68577 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766070+0000", 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766070+0000", 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766071+0000", 2024-04-29T14:10:11.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.511e-06 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766131+0000", 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0044999999999999e-05 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766142+0000", 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1113e-05 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766148+0000", 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8220000000000003e-06 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766208+0000", 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9997000000000002e-05 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766342+0000", 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134215 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.766477+0000", 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134964 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.768732+0000", 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022542920000000002 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.960773+0000", 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19204174500000001 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.960813+0000", 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9816e-05 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.960842+0000", 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8506000000000001e-05 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68599 2.4 2:20f78048:::benchmark_data_smithi194_79000_object68598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.776682+0000", 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.468859896000001, 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18772065800000001, 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68599 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776682+0000", 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776682+0000", 2024-04-29T14:10:11.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776690+0000", 2024-04-29T14:10:11.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4919999999999996e-06 2024-04-29T14:10:11.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776772+0000", 2024-04-29T14:10:11.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2650000000000003e-05 2024-04-29T14:10:11.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776784+0000", 2024-04-29T14:10:11.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.163e-05 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776792+0000", 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7780000000000007e-06 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776847+0000", 2024-04-29T14:10:11.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5124000000000003e-05 2024-04-29T14:10:11.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.776932+0000", 2024-04-29T14:10:11.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5514999999999997e-05 2024-04-29T14:10:11.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.777023+0000", 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0187999999999998e-05 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.778774+0000", 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017516999999999999 2024-04-29T14:10:11.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964286+0000", 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.185511234 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964344+0000", 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8835000000000002e-05 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964403+0000", 2024-04-29T14:10:11.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8511999999999998e-05 2024-04-29T14:10:11.505 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.505 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.505 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.505 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68606 2.4 2:2fa51e11:::benchmark_data_smithi194_79000_object68605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.506 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.779271+0000", 2024-04-29T14:10:11.506 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.466271304999999, 2024-04-29T14:10:11.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.185305201, 2024-04-29T14:10:11.506 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.506 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.506 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.506 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68606 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779271+0000", 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779271+0000", 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779273+0000", 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8819999999999999e-06 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779327+0000", 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4781999999999998e-05 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779339+0000", 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1059e-05 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779347+0000", 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7159999999999992e-06 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779400+0000", 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3057999999999999e-05 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779529+0000", 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012860000000000001 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.779679+0000", 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000150178 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.781619+0000", 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019397590000000001 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964478+0000", 2024-04-29T14:10:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.182858938 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964532+0000", 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3913999999999999e-05 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964576+0000", 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4314999999999998e-05 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68619 2.4 2:2479b083:::benchmark_data_smithi194_79000_object68618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.786275+0000", 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.459267284999999, 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.178463761, 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.513 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68619 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786275+0000", 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786275+0000", 2024-04-29T14:10:11.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786276+0000", 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6619999999999999e-06 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786306+0000", 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9011000000000001e-05 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786318+0000", 2024-04-29T14:10:11.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2086000000000001e-05 2024-04-29T14:10:11.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786326+0000", 2024-04-29T14:10:11.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8750000000000006e-06 2024-04-29T14:10:11.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786369+0000", 2024-04-29T14:10:11.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2620000000000002e-05 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786469+0000", 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9535000000000002e-05 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.786576+0000", 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010713999999999999 2024-04-29T14:10:11.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.788286+0000", 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017097429999999999 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964640+0000", 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.17635446799999999 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964697+0000", 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7416000000000001e-05 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.964739+0000", 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1205000000000003e-05 2024-04-29T14:10:11.520 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68629 2.4 2:2fcff026:::benchmark_data_smithi194_79000_object68628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.791093+0000", 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.454449082, 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.176559149, 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.521 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68629 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791093+0000", 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791093+0000", 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791095+0000", 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.889e-06 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791153+0000", 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7856000000000003e-05 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791167+0000", 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4226000000000001e-05 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791175+0000", 2024-04-29T14:10:11.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4270000000000008e-06 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791234+0000", 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8709999999999999e-05 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791371+0000", 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136723 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.791511+0000", 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014010499999999999 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.793557+0000", 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020462309999999999 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967546+0000", 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.173988702 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967606+0000", 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0414000000000001e-05 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967652+0000", 2024-04-29T14:10:11.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5865999999999998e-05 2024-04-29T14:10:11.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.528 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.528 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68642 2.4 2:2c317a75:::benchmark_data_smithi194_79000_object68641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.529 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.800776+0000", 2024-04-29T14:10:11.529 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.444766308999998, 2024-04-29T14:10:11.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.16703593, 2024-04-29T14:10:11.529 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.529 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.530 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.530 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.530 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.530 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68642 2024-04-29T14:10:11.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.530 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800776+0000", 2024-04-29T14:10:11.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800776+0000", 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800777+0000", 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.531e-06 2024-04-29T14:10:11.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800805+0000", 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7654e-05 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800816+0000", 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0984e-05 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800822+0000", 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1249999999999998e-06 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800872+0000", 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9626000000000002e-05 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.800996+0000", 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000124678 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.801105+0000", 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108607 2024-04-29T14:10:11.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.805109+0000", 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0040039050000000003 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967712+0000", 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.162603313 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967772+0000", 2024-04-29T14:10:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0137999999999999e-05 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967812+0000", 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9369000000000002e-05 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68654 2.4 2:2e0610f1:::benchmark_data_smithi194_79000_object68653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.808827+0000", 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.436715571000001, 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15913481099999999, 2024-04-29T14:10:11.536 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68654 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.808827+0000", 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.808827+0000", 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.808828+0000", 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3540000000000001e-06 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.808860+0000", 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2422999999999998e-05 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.808873+0000", 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2447e-05 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.808881+0000", 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7840000000000004e-06 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.808944+0000", 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3410000000000004e-05 2024-04-29T14:10:11.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.809059+0000", 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011480399999999999 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.809227+0000", 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000168572 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.811509+0000", 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022818309999999998 2024-04-29T14:10:11.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967865+0000", 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15635626799999999 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967916+0000", 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0676000000000001e-05 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.967961+0000", 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5241999999999998e-05 2024-04-29T14:10:11.541 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68661 2.4 2:2e046159:::benchmark_data_smithi194_79000_object68660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.812640+0000", 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.432902008999999, 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15784182099999999, 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.542 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68661 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812640+0000", 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812640+0000", 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812641+0000", 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.381e-06 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812696+0000", 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4716999999999998e-05 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812707+0000", 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1161e-05 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812716+0000", 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3690000000000007e-06 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812770+0000", 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4320000000000002e-05 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.812886+0000", 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011636799999999999 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.813016+0000", 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012967400000000001 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.816830+0000", 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0038143460000000001 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.970358+0000", 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.153527673 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.970430+0000", 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1839000000000001e-05 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.970482+0000", 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1972999999999999e-05 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68665 2.4 2:275b1d81:::benchmark_data_smithi194_79000_object68664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.813179+0000", 2024-04-29T14:10:11.547 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.432362881, 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15746795199999999, 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68665 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.813179+0000", 2024-04-29T14:10:11.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.813179+0000", 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.813181+0000", 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7400000000000001e-06 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.813225+0000", 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4347000000000001e-05 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.813237+0000", 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1623e-05 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.813245+0000", 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1859999999999992e-06 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.815033+0000", 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017882779999999999 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.815232+0000", 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019885200000000001 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.815332+0000", 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9265000000000006e-05 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.816986+0000", 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001654804 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.970550+0000", 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15356376699999999 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.970606+0000", 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5927000000000002e-05 2024-04-29T14:10:11.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.970647+0000", 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1162999999999999e-05 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68677 2.4 2:271b2d08:::benchmark_data_smithi194_79000_object68676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.822290+0000", 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.423252207000001, 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.150840111, 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68677 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822290+0000", 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822290+0000", 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822290+0000", 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1600000000000001e-07 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822331+0000", 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0972e-05 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822336+0000", 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6430000000000003e-06 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822340+0000", 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6689999999999999e-06 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822384+0000", 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4162999999999997e-05 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822506+0000", 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121826 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.822653+0000", 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014712500000000001 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.824360+0000", 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001707477 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.973019+0000", 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.14865900600000001 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.973088+0000", 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8695000000000006e-05 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.973130+0000", 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2018999999999999e-05 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.558 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68697 2.4 2:2771f04a:::benchmark_data_smithi194_79000_object68696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.840633+0000", 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.404909033999999, 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.132649934, 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68697 2024-04-29T14:10:11.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840633+0000", 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840633+0000", 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840636+0000", 2024-04-29T14:10:11.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4260000000000002e-06 2024-04-29T14:10:11.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840675+0000", 2024-04-29T14:10:11.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9975000000000003e-05 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840691+0000", 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5124e-05 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840699+0000", 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4060000000000005e-06 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840763+0000", 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3613000000000002e-05 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.840892+0000", 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.841051+0000", 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015992600000000001 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.843170+0000", 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021189680000000002 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.973193+0000", 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130022583 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.973246+0000", 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2658000000000003e-05 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.973283+0000", 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7339999999999998e-05 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:68752 2.4 2:28f3fa91:::benchmark_data_smithi194_79000_object68751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e118)", 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:47.903635+0000", 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 23.341906797, 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.073566063000000001, 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 68752 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903635+0000", 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903635+0000", 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903638+0000", 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3080000000000002e-06 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903682+0000", 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4304000000000001e-05 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903694+0000", 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2214e-05 2024-04-29T14:10:11.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903701+0000", 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0729999999999999e-06 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903771+0000", 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9777000000000006e-05 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.903925+0000", 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015371199999999999 2024-04-29T14:10:11.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.904094+0000", 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016942699999999999 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.905820+0000", 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017256179999999999 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.977125+0000", 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.071305215000000005 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.977160+0000", 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4572e-05 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:47.977201+0000", 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1842999999999999e-05 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:10:11.610 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:11.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:moved 0 / 34 (0%) 2024-04-29T14:10:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:avg 2.83333 2024-04-29T14:10:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.06719 -> 1.06719 (expected baseline 1.61159) 2024-04-29T14:10:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.6 with 1 -> 1 pgs (0.352941 -> 0.352941 * mean) 2024-04-29T14:10:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.7 with 5 -> 5 pgs (1.76471 -> 1.76471 * mean) 2024-04-29T14:10:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:10:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:oload 1000 2024-04-29T14:10:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:10:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:10:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0129 2024-04-29T14:10:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.1294 2024-04-29T14:10:11.696 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:10:11.718 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:11.752 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:11.781 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:11.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:11.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:11.923 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:11.950 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:11.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:12.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:12.089 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:12.113 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:12.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:12.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:10:12.219 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:12.248 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:12.274 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:12.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:12.363 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 111285 111269 139.063 91.375 0.0129163 0.00718895 2024-04-29T14:10:12.383 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:12.414 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:12.436 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:12.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:12.440 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:12.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:12.577 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:12.599 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:12.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:12.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:12.736 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:12.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:12.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:12.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:12.949 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:13.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:12 smithi121 ceph-mon[44533]: pgmap v388: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s; 1078/215390 objects misplaced (0.500%); 8.7 MiB/s, 139 objects/s recovering 2024-04-29T14:10:13.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:12 smithi121 ceph-mon[44533]: from='client.24784 v1:172.21.15.40:0/1124481905' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:10:13.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:12 smithi121 ceph-mon[44533]: osdmap e124: 12 total, 12 up, 12 in 2024-04-29T14:10:13.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:12 smithi040 ceph-mon[39581]: pgmap v388: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s; 1078/215390 objects misplaced (0.500%); 8.7 MiB/s, 139 objects/s recovering 2024-04-29T14:10:13.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:12 smithi040 ceph-mon[39581]: from='client.24784 v1:172.21.15.40:0/1124481905' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:10:13.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:12 smithi040 ceph-mon[39581]: osdmap e124: 12 total, 12 up, 12 in 2024-04-29T14:10:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:12 smithi194 ceph-mon[44089]: pgmap v388: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s; 1078/215390 objects misplaced (0.500%); 8.7 MiB/s, 139 objects/s recovering 2024-04-29T14:10:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:12 smithi194 ceph-mon[44089]: from='client.24784 v1:172.21.15.40:0/1124481905' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:10:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:12 smithi194 ceph-mon[44089]: osdmap e124: 12 total, 12 up, 12 in 2024-04-29T14:10:13.365 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 112806 112790 138.195 95.0625 0.00335145 0.00723441 2024-04-29T14:10:14.148 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:14.365 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 114400 114384 137.453 99.625 0.0188764 0.00727353 2024-04-29T14:10:15.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:14 smithi040 ceph-mon[39581]: pgmap v390: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 1078/217298 objects misplaced (0.496%); 4.0 MiB/s, 63 objects/s recovering 2024-04-29T14:10:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:14 smithi194 ceph-mon[44089]: pgmap v390: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 1078/217298 objects misplaced (0.496%); 4.0 MiB/s, 63 objects/s recovering 2024-04-29T14:10:15.365 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 16 116070 116054 136.829 104.375 0.00523381 0.00730642 2024-04-29T14:10:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:14 smithi121 ceph-mon[44533]: pgmap v390: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 1078/217298 objects misplaced (0.496%); 4.0 MiB/s, 63 objects/s recovering 2024-04-29T14:10:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:15.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:15.539 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:15.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:15.811 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:15.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:15.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:15.947 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:15.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:15.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:16.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:16.240 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:16.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:16.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:16.365 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 117560 117544 136.019 93.125 0.0035817 0.00735003 2024-04-29T14:10:16.394 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:17.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:16 smithi040 ceph-mon[39581]: pgmap v391: 17 pgs: 1 active+recovering, 16 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s; 1078/226800 objects misplaced (0.475%); 3.2 MiB/s, 51 objects/s recovering 2024-04-29T14:10:17.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:16 smithi040 ceph-mon[39581]: osdmap e125: 12 total, 12 up, 12 in 2024-04-29T14:10:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:16 smithi194 ceph-mon[44089]: pgmap v391: 17 pgs: 1 active+recovering, 16 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s; 1078/226800 objects misplaced (0.475%); 3.2 MiB/s, 51 objects/s recovering 2024-04-29T14:10:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:16 smithi194 ceph-mon[44089]: osdmap e125: 12 total, 12 up, 12 in 2024-04-29T14:10:17.365 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 119022 119006 135.208 91.375 0.00344236 0.00739419 2024-04-29T14:10:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:16 smithi121 ceph-mon[44533]: pgmap v391: 17 pgs: 1 active+recovering, 16 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s; 1078/226800 objects misplaced (0.475%); 3.2 MiB/s, 51 objects/s recovering 2024-04-29T14:10:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:16 smithi121 ceph-mon[44533]: osdmap e125: 12 total, 12 up, 12 in 2024-04-29T14:10:17.466 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:17.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:17.573 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:17.612 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:17.634 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:17.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:17.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:17.805 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:17.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:17.937 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:10:17.947 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:10:17.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:10:17.947 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-29T14:10:17.947 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:10:17.947 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:10:17.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-04-29T14:10:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:10:17.996 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:10:18.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:18.026 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:10:18.026 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:10:18.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.026 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:119861 2.1 2:8bcb2a97:::benchmark_data_smithi194_79000_object119860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-04-29T14:10:18.026 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:17.958544+0000", 2024-04-29T14:10:18.026 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.018581032000000001, 2024-04-29T14:10:18.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018615406000000001, 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119861 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958544+0000", 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958544+0000", 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958545+0000", 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3680000000000001e-06 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958813+0000", 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026726400000000002 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958823+0000", 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0383e-05 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958832+0000", 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8349999999999993e-06 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958949+0000", 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117559 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.958955+0000", 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.327e-06 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964711+0000", 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057563220000000003 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964838+0000", 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012653100000000001 2024-04-29T14:10:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964971+0000", 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013374599999999999 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976409+0000", 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011437416000000001 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.031 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:119867 2.1 2:8f72a786:::benchmark_data_smithi194_79000_object119866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:17.964206+0000", 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.012918717, 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013059779000000001, 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119867 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:18.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964206+0000", 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964206+0000", 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964209+0000", 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2520000000000002e-06 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964349+0000", 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014006899999999999 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964360+0000", 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0827e-05 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.964368+0000", 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6389999999999998e-06 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.968468+0000", 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0040995199999999997 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.968593+0000", 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125401 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.968689+0000", 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6358000000000001e-05 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.977037+0000", 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0083474229999999996 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:119882 2.e 2:7ee265d8:::benchmark_data_smithi194_79000_object119881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:17.976689+0000", 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.00043593599999999999, 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00070940200000000004, 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:10:18.036 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119882 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976689+0000", 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976689+0000", 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976691+0000", 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0439999999999998e-06 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976767+0000", 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6171e-05 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976778+0000", 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0117e-05 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976786+0000", 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3979999999999999e-06 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.976859+0000", 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3094000000000002e-05 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.977002+0000", 2024-04-29T14:10:18.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000142836 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.977138+0000", 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136482 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 3 2024-04-29T14:10:18.040 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:10:18.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:10:18.072 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:18.079 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:10:18.096 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:18.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:18.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1473171555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:18.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3556290542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:18.125 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:10:18.167 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:10:18.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:18.208 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:10:18.229 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1473171555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:18.229 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3556290542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:18.230 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:18.245 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:10:18.253 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:18.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:18.302 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:10:18.338 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:10:18.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:18.365 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 120386 120370 134.316 85.25 0.00414904 0.007444 2024-04-29T14:10:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1473171555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3556290542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:18.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:10:18.413 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:10:18.417 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:10:18.439 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:10:18.440 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:10:18.440 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:10:18.440 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:10:18.440 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: pgmap v393: 17 pgs: 1 active+recovering, 16 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s; 633/232718 objects misplaced (0.272%); 6.9 MiB/s, 110 objects/s recovering 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1473171555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osdmap e126: 12 total, 12 up, 12 in 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: pgmap v393: 17 pgs: 1 active+recovering, 16 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s; 633/232718 objects misplaced (0.272%); 6.9 MiB/s, 110 objects/s recovering 2024-04-29T14:10:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1473171555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osdmap e126: 12 total, 12 up, 12 in 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.365 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 16 121914 121898 133.634 95.5 0.00644506 0.00748182 2024-04-29T14:10:19.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: pgmap v393: 17 pgs: 1 active+recovering, 16 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s; 633/232718 objects misplaced (0.272%); 6.9 MiB/s, 110 objects/s recovering 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1473171555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osdmap e126: 12 total, 12 up, 12 in 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:10:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:19.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:19.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:19.819 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:19.850 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:19.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:19.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:19.964 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:20.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:20.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:20.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:20.291 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:20.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:20.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:20.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 123391 123375 132.922 92.3125 0.0126856 0.00752174 2024-04-29T14:10:20.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:10:20.451 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:20.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:10:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:20.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:10:20.507 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:20.582 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:20.614 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:20.639 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:20.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:20.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:10:20.787 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:20.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:10:20.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:20.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:10:20.920 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:20.957 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:20.981 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:20.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:21.088 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:20 smithi040 ceph-mon[39581]: pgmap v395: 17 pgs: 1 active+recovering, 16 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s; 633/233630 objects misplaced (0.271%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:10:21.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:20 smithi121 ceph-mon[44533]: pgmap v395: 17 pgs: 1 active+recovering, 16 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s; 633/233630 objects misplaced (0.271%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:10:21.133 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:21.166 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:21.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:21.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:10:21.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:21.299 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:20 smithi194 ceph-mon[44089]: pgmap v395: 17 pgs: 1 active+recovering, 16 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s; 633/233630 objects misplaced (0.271%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:10:21.321 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:21.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:21.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 124874 124858 132.24 92.6875 0.00797702 0.00756027 2024-04-29T14:10:21.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:21.460 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:21.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:21.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:21.593 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:21.641 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:21.682 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:21.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:21.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:21.849 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:22.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:10:22.325 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:22.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:22.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:10:22.372787 min lat: 0.00107894 max lat: 4.06786 avg lat: 0.00759803 2024-04-29T14:10:22.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:10:22.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 126346 126330 131.569 92 0.0141628 0.00759803 2024-04-29T14:10:22.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:10:22.463 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:22.486 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:10:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:22.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:10:22.592 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:22.626 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:22.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:22.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:22.814 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:22.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:22 smithi040 ceph-mon[39581]: pgmap v396: 17 pgs: 1 active+recovering, 16 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s; 90/242898 objects misplaced (0.037%); 7.7 MiB/s, 123 objects/s recovering 2024-04-29T14:10:23.303 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:10:23.303 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-29T14:10:23.303 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:10:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:22 smithi194 ceph-mon[44089]: pgmap v396: 17 pgs: 1 active+recovering, 16 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s; 90/242898 objects misplaced (0.037%); 7.7 MiB/s, 123 objects/s recovering 2024-04-29T14:10:23.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 127824 127808 130.927 92.375 0.0115693 0.00763656 2024-04-29T14:10:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:22 smithi121 ceph-mon[44533]: pgmap v396: 17 pgs: 1 active+recovering, 16 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s; 90/242898 objects misplaced (0.037%); 7.7 MiB/s, 123 objects/s recovering 2024-04-29T14:10:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:23.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:23.586 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:23.627 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:23.650 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:23.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:23.754 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:23.793 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:23.832 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:23.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:23.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:23.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:23.971 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:23.995 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:24.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:24.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:24.131 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:24.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:24.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:24.263 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:24.323 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:24.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 16 129332 129316 130.335 94.25 0.00826446 0.00767065 2024-04-29T14:10:24.415 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:24.416 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:24.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:24.478 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:24.508 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:10:24.508 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:10:24.527 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:24.562 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:24.584 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:24.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:24.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:24.723 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:24.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:24.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:24.988 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:24 smithi121 ceph-mon[44533]: pgmap v397: 17 pgs: 1 active+recovering, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 90/247544 objects misplaced (0.036%); 7.7 MiB/s, 123 objects/s recovering 2024-04-29T14:10:25.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:24 smithi040 ceph-mon[39581]: pgmap v397: 17 pgs: 1 active+recovering, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 90/247544 objects misplaced (0.036%); 7.7 MiB/s, 123 objects/s recovering 2024-04-29T14:10:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:24 smithi194 ceph-mon[44089]: pgmap v397: 17 pgs: 1 active+recovering, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 90/247544 objects misplaced (0.036%); 7.7 MiB/s, 123 objects/s recovering 2024-04-29T14:10:25.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 130762 130746 129.685 89.375 0.0233132 0.00770962 2024-04-29T14:10:26.239 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:26.239 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:26.240 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:26 smithi040 ceph-mon[39581]: osdmap e127: 12 total, 12 up, 12 in 2024-04-29T14:10:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:26 smithi194 ceph-mon[44089]: osdmap e127: 12 total, 12 up, 12 in 2024-04-29T14:10:26.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 16 132279 132263 129.14 94.8125 0.00584971 0.00774204 2024-04-29T14:10:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:26 smithi121 ceph-mon[44533]: osdmap e127: 12 total, 12 up, 12 in 2024-04-29T14:10:27.014 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:27 smithi040 ceph-mon[39581]: pgmap v399: 17 pgs: 1 active+recovering, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s; 90/256720 objects misplaced (0.035%); 4.2 MiB/s, 67 objects/s recovering 2024-04-29T14:10:27.014 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:27 smithi040 ceph-mon[39581]: osdmap e128: 12 total, 12 up, 12 in 2024-04-29T14:10:27.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:27 smithi194 ceph-mon[44089]: pgmap v399: 17 pgs: 1 active+recovering, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s; 90/256720 objects misplaced (0.035%); 4.2 MiB/s, 67 objects/s recovering 2024-04-29T14:10:27.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:27 smithi194 ceph-mon[44089]: osdmap e128: 12 total, 12 up, 12 in 2024-04-29T14:10:27.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:10:27.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 133816 133800 128.631 96.0625 0.00677749 0.00777273 2024-04-29T14:10:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:27 smithi121 ceph-mon[44533]: pgmap v399: 17 pgs: 1 active+recovering, 16 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s; 90/256720 objects misplaced (0.035%); 4.2 MiB/s, 67 objects/s recovering 2024-04-29T14:10:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:27 smithi121 ceph-mon[44533]: osdmap e128: 12 total, 12 up, 12 in 2024-04-29T14:10:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:27.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:27.631 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:10:27.657 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:10:27.686 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:10:27.700 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:27.718 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:10:27.739 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:27.754 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:10:27.768 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:27.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:27.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:10:27.824 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:10:27.862 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:10:27.872 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:27.899 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:10:27.907 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:27.931 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:27.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:27.980 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:10:27.980 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:10:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:10:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:10:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:10:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:10:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:10:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:10:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:10:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:10:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:10:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:10:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:10:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:10:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:10:28.034 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3906699610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:28.034 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:28.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:28.077 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3906699610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:28.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 16 135172 135156 127.966 84.75 0.017297 0.00781299 2024-04-29T14:10:28.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3906699610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:28.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.165 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:10:29.166 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:10:29.166 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-29T14:10:29.166 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:10:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: pgmap v401: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s; 4.9 MiB/s, 79 objects/s recovering 2024-04-29T14:10:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:10:29.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 16 136481 136465 127.277 81.8125 0.0052358 0.00785569 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: pgmap v401: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s; 4.9 MiB/s, 79 objects/s recovering 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:10:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: pgmap v401: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s; 4.9 MiB/s, 79 objects/s recovering 2024-04-29T14:10:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:10:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:30 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:30 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:30 smithi194 ceph-mon[44089]: osdmap e129: 12 total, 12 up, 12 in 2024-04-29T14:10:30.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:30 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:30.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:30 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:30.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:30 smithi040 ceph-mon[39581]: osdmap e129: 12 total, 12 up, 12 in 2024-04-29T14:10:30.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 137902 137886 126.711 88.8125 0.00690753 0.00789079 2024-04-29T14:10:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:30 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:30 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:30.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:30 smithi121 ceph-mon[44533]: osdmap e129: 12 total, 12 up, 12 in 2024-04-29T14:10:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:31 smithi194 ceph-mon[44089]: pgmap v403: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 959 KiB/s, 14 objects/s recovering 2024-04-29T14:10:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:10:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:10:31.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:10:31.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:31 smithi040 ceph-mon[39581]: pgmap v403: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 959 KiB/s, 14 objects/s recovering 2024-04-29T14:10:31.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:10:31.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:10:31.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:10:31.366 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 139217 139201 126.066 82.1875 0.00817282 0.00793093 2024-04-29T14:10:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:31 smithi121 ceph-mon[44533]: pgmap v403: 17 pgs: 17 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 959 KiB/s, 14 objects/s recovering 2024-04-29T14:10:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:10:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:10:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:10:32.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 140651 140635 125.545 89.625 0.0127996 0.00796403 2024-04-29T14:10:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:32 smithi194 ceph-mon[44089]: pgmap v404: 17 pgs: 17 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s; 893 KiB/s, 13 objects/s recovering 2024-04-29T14:10:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:32 smithi194 ceph-mon[44089]: osdmap e130: 12 total, 12 up, 12 in 2024-04-29T14:10:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:32 smithi040 ceph-mon[39581]: pgmap v404: 17 pgs: 17 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s; 893 KiB/s, 13 objects/s recovering 2024-04-29T14:10:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:32 smithi040 ceph-mon[39581]: osdmap e130: 12 total, 12 up, 12 in 2024-04-29T14:10:33.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 15 142101 142086 125.054 90.6875 0.00859385 0.00799497 2024-04-29T14:10:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:32 smithi121 ceph-mon[44533]: pgmap v404: 17 pgs: 17 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s; 893 KiB/s, 13 objects/s recovering 2024-04-29T14:10:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:32 smithi121 ceph-mon[44533]: osdmap e130: 12 total, 12 up, 12 in 2024-04-29T14:10:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:33.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:33.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:33.919 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:34.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 143459 143443 124.495 84.8125 0.00420292 0.0080311 2024-04-29T14:10:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:34 smithi194 ceph-mon[44089]: pgmap v406: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-29T14:10:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:34 smithi040 ceph-mon[39581]: pgmap v406: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-29T14:10:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:35.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 144864 144848 123.993 87.8125 0.0121618 0.00806349 2024-04-29T14:10:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:34 smithi121 ceph-mon[44533]: pgmap v406: 17 pgs: 17 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-29T14:10:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:36.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:36.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:36.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:35 smithi194 ceph-mon[44089]: osdmap e131: 12 total, 12 up, 12 in 2024-04-29T14:10:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:35 smithi040 ceph-mon[39581]: osdmap e131: 12 total, 12 up, 12 in 2024-04-29T14:10:36.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 146295 146279 123.526 89.4375 0.0112939 0.00809426 2024-04-29T14:10:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:35 smithi121 ceph-mon[44533]: osdmap e131: 12 total, 12 up, 12 in 2024-04-29T14:10:37.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:36 smithi194 ceph-mon[44089]: pgmap v408: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:10:37.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:36 smithi194 ceph-mon[44089]: osdmap e132: 12 total, 12 up, 12 in 2024-04-29T14:10:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:36 smithi040 ceph-mon[39581]: pgmap v408: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:10:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:37 smithi040 ceph-mon[39581]: osdmap e132: 12 total, 12 up, 12 in 2024-04-29T14:10:37.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 15 147562 147547 122.936 79.25 0.00678667 0.00813304 2024-04-29T14:10:37.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:36 smithi121 ceph-mon[44533]: pgmap v408: 17 pgs: 17 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:10:37.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:36 smithi121 ceph-mon[44533]: osdmap e132: 12 total, 12 up, 12 in 2024-04-29T14:10:37.906 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:37.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:38.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:10:38.049 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:38.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:10:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:38.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:10:38.168 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:38.168 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:38.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:38.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:38.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:38.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 148820 148804 122.352 78.5625 0.00780356 0.00817203 2024-04-29T14:10:38.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:38.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:38.490 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:38.492 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:38.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:38.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:38.644 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:38.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:38.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:38.797 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:38.822 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:38.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:38.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:10:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:38.983 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:10:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:38.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:10:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:10:39.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:38 smithi040 ceph-mon[39581]: pgmap v410: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:10:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:38 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:10:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:38 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:10:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:38 smithi040 ceph-mon[39581]: osdmap e133: 12 total, 12 up, 12 in 2024-04-29T14:10:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3738385418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:10:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:38 smithi040 ceph-mon[39581]: from='client.34814 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:10:39.122 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:39.144 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:39.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:39.253 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:39.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:38 smithi194 ceph-mon[44089]: pgmap v410: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:10:39.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:38 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:10:39.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:38 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:10:39.289 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:38 smithi194 ceph-mon[44089]: osdmap e133: 12 total, 12 up, 12 in 2024-04-29T14:10:39.289 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3738385418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:10:39.289 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:38 smithi194 ceph-mon[44089]: from='client.34814 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:10:39.290 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:39.317 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:39.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:39.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 150018 150002 121.735 74.875 0.0100027 0.00821341 2024-04-29T14:10:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:38 smithi121 ceph-mon[44533]: pgmap v410: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:10:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:38 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:10:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:38 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:10:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:38 smithi121 ceph-mon[44533]: osdmap e133: 12 total, 12 up, 12 in 2024-04-29T14:10:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3738385418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:10:39.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:38 smithi121 ceph-mon[44533]: from='client.34814 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:10:39.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:39.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:10:40.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:40 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:40.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:40 smithi040 ceph-mon[39581]: from='client.34814 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:10:40.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:40 smithi040 ceph-mon[39581]: osdmap e134: 12 total, 12 up, 12 in 2024-04-29T14:10:40.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:40 smithi040 ceph-mon[39581]: 2.2 scrub starts 2024-04-29T14:10:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:40 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:40 smithi194 ceph-mon[44089]: from='client.34814 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:10:40.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:40 smithi194 ceph-mon[44089]: osdmap e134: 12 total, 12 up, 12 in 2024-04-29T14:10:40.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:40 smithi194 ceph-mon[44089]: 2.2 scrub starts 2024-04-29T14:10:40.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 151276 151260 121.182 78.625 0.00763446 0.00825067 2024-04-29T14:10:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:40 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:10:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:40 smithi121 ceph-mon[44533]: from='client.34814 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:10:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:40 smithi121 ceph-mon[44533]: osdmap e134: 12 total, 12 up, 12 in 2024-04-29T14:10:40.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:40 smithi121 ceph-mon[44533]: 2.2 scrub starts 2024-04-29T14:10:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:41 smithi040 ceph-mon[39581]: pgmap v413: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-29T14:10:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:41 smithi040 ceph-mon[39581]: 2.4 scrub starts 2024-04-29T14:10:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:41 smithi040 ceph-mon[39581]: 2.2 scrub ok 2024-04-29T14:10:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:41 smithi040 ceph-mon[39581]: 2.5 scrub starts 2024-04-29T14:10:41.251 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:41.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:41.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:41 smithi194 ceph-mon[44089]: pgmap v413: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-29T14:10:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:41 smithi194 ceph-mon[44089]: 2.4 scrub starts 2024-04-29T14:10:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:41 smithi194 ceph-mon[44089]: 2.2 scrub ok 2024-04-29T14:10:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:41 smithi194 ceph-mon[44089]: 2.5 scrub starts 2024-04-29T14:10:41.357 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:41.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 152350 152334 120.498 67.125 0.0232108 0.00829704 2024-04-29T14:10:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:41 smithi121 ceph-mon[44533]: pgmap v413: 17 pgs: 17 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-29T14:10:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:41 smithi121 ceph-mon[44533]: 2.4 scrub starts 2024-04-29T14:10:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:41 smithi121 ceph-mon[44533]: 2.2 scrub ok 2024-04-29T14:10:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:41 smithi121 ceph-mon[44533]: 2.5 scrub starts 2024-04-29T14:10:41.426 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:41.677 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:41.682 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:41.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:41.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:41.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:10:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:42.226 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:42 smithi040 ceph-mon[39581]: 2.f scrub starts 2024-04-29T14:10:42.226 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:42 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:10:42.226 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:42 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:10:42.226 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:42 smithi040 ceph-mon[39581]: 2.d scrub starts 2024-04-29T14:10:42.226 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:42 smithi040 ceph-mon[39581]: 2.3 scrub starts 2024-04-29T14:10:42.226 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:42 smithi040 ceph-mon[39581]: 2.b scrub starts 2024-04-29T14:10:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:42 smithi194 ceph-mon[44089]: 2.f scrub starts 2024-04-29T14:10:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:42 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:10:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:42 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:10:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:42 smithi194 ceph-mon[44089]: 2.d scrub starts 2024-04-29T14:10:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:42 smithi194 ceph-mon[44089]: 2.3 scrub starts 2024-04-29T14:10:42.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:42 smithi194 ceph-mon[44089]: 2.b scrub starts 2024-04-29T14:10:42.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:10:42.374420 min lat: 0.00107894 max lat: 4.06786 avg lat: 0.00835341 2024-04-29T14:10:42.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:10:42.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 153232 153216 119.681 55.125 0.0123748 0.00835341 2024-04-29T14:10:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:42 smithi121 ceph-mon[44533]: 2.f scrub starts 2024-04-29T14:10:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:42 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:10:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:42 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:10:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:42 smithi121 ceph-mon[44533]: 2.d scrub starts 2024-04-29T14:10:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:42 smithi121 ceph-mon[44533]: 2.3 scrub starts 2024-04-29T14:10:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:42 smithi121 ceph-mon[44533]: 2.b scrub starts 2024-04-29T14:10:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:42.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:42.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:42.602 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:42.639 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:42.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:42.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:42.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:42.792 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:42.886 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:10:42.899 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:10:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:42.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:43.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: pgmap v414: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.8 scrub starts 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.f scrub ok 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.e scrub starts 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.c scrub starts 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.d scrub ok 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.5 scrub ok 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.4 scrub ok 2024-04-29T14:10:43.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:43 smithi194 ceph-mon[44089]: 2.3 scrub ok 2024-04-29T14:10:43.058 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:43.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:10:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:43.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:10:43.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:43.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: pgmap v414: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.8 scrub starts 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.f scrub ok 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.e scrub starts 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.c scrub starts 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.d scrub ok 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.5 scrub ok 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.4 scrub ok 2024-04-29T14:10:43.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:43 smithi040 ceph-mon[39581]: 2.3 scrub ok 2024-04-29T14:10:43.253 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:43.368 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 154147 154131 118.909 57.1875 0.00644594 0.00840009 2024-04-29T14:10:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: pgmap v414: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.8 scrub starts 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.f scrub ok 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.e scrub starts 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.c scrub starts 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.d scrub ok 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.5 scrub ok 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.4 scrub ok 2024-04-29T14:10:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:43 smithi121 ceph-mon[44533]: 2.3 scrub ok 2024-04-29T14:10:44.009 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:44.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:44.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:44.148 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:44.174 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:44.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:44.201 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-04-29T14:10:41.447770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9803924000,"num_objects":149589,"num_object_clones":0,"num_object_copies":299178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149589,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":149768,"num_write_kb":9576506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6703,"num_bytes_recovered":440879232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149724,"ondisk_log_size":149724,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":20138636,"kb_used_data":19434116,"kb_used_omap":18,"kb_used_meta":704365,"kb_avail":1104803188,"statfs":{"total":1151940427776,"available":1131318464512,"internally_reserved":0,"allocated":19900534784,"data_stored":19894247602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":721270074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,6,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":461570048,"num_objects":7043,"num_object_clones":0,"num_object_copies":14086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7043,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.008168"},"pg_stats":[{"pgid":"2.f","version":"134'9288","reported_seq":9351,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:39.276568+0000","last_change":"2024-04-29T14:09:11.906218+0000","last_active":"2024-04-29T14:10:39.276568+0000","last_peered":"2024-04-29T14:10:39.276568+0000","last_clean":"2024-04-29T14:10:39.276568+0000","last_became_active":"2024-04-29T14:09:11.905834+0000","last_became_peered":"2024-04-29T14:09:11.905834+0000","last_unstale":"2024-04-29T14:10:39.276568+0000","last_undegraded":"2024-04-29T14:10:39.276568+0000","last_fullsized":"2024-04-29T14:10:39.276568+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9288,"log_dups_size":0,"ondisk_log_size":9288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":608698368,"num_objects":9288,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9288,"num_write_kb":594432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"131'9081","reported_seq":9137,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-29T14:10:36.435808+0000","last_change":"2024-04-29T14:09:11.900649+0000","last_active":"2024-04-29T14:10:36.435808+0000","last_peered":"2024-04-29T14:10:36.435808+0000","last_clean":"2024-04-29T14:10:36.435808+0000","last_became_active":"2024-04-29T14:09:11.900365+0000","last_became_peered":"2024-04-29T14:09:11.900365+0000","last_unstale":"2024-04-29T14:10:36.435808+0000","last_undegraded":"2024-04-29T14:10:36.435808+0000","last_fullsized":"2024-04-29T14:10:36.435808+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9081,"log_dups_size":0,"ondisk_log_size":9081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:37.175569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":595132416,"num_objects":9081,"num_object_clones":0,"num_object_copies":18162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":581184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"134'9198","reported_seq":9318,"reported_epoch":134,"state":"active+clean+scrubbing","last_fresh":"2024-04-29T14:10:39.816723+0000","last_change":"2024-04-29T14:10:38.992316+0000","last_active":"2024-04-29T14:10:39.816723+0000","last_peered":"2024-04-29T14:10:39.816723+0000","last_clean":"2024-04-29T14:10:39.816723+0000","last_became_active":"2024-04-29T14:09:11.906623+0000","last_became_peered":"2024-04-29T14:09:11.906623+0000","last_unstale":"2024-04-29T14:10:39.816723+0000","last_undegraded":"2024-04-29T14:10:39.816723+0000","last_fullsized":"2024-04-29T14:10:39.816723+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":5499,"log_size":9198,"log_dups_size":0,"ondisk_log_size":9198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":602800128,"num_objects":9198,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9198,"num_write_kb":588672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"132'9202","reported_seq":9261,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-29T14:10:37.009985+0000","last_change":"2024-04-29T14:09:12.586046+0000","last_active":"2024-04-29T14:10:37.009985+0000","last_peered":"2024-04-29T14:10:37.009985+0000","last_clean":"2024-04-29T14:10:37.009985+0000","last_became_active":"2024-04-29T14:09:12.585706+0000","last_became_peered":"2024-04-29T14:09:12.585706+0000","last_unstale":"2024-04-29T14:10:37.009985+0000","last_undegraded":"2024-04-29T14:10:37.009985+0000","last_fullsized":"2024-04-29T14:10:37.009985+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9202,"log_dups_size":0,"ondisk_log_size":9202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:40.132870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":603062272,"num_objects":9202,"num_object_clones":0,"num_object_copies":18404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9202,"num_write_kb":588928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"134'9463","reported_seq":9526,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:39.805384+0000","last_change":"2024-04-29T14:09:11.907503+0000","last_active":"2024-04-29T14:10:39.805384+0000","last_peered":"2024-04-29T14:10:39.805384+0000","last_clean":"2024-04-29T14:10:39.805384+0000","last_became_active":"2024-04-29T14:09:11.907211+0000","last_became_peered":"2024-04-29T14:09:11.907211+0000","last_unstale":"2024-04-29T14:10:39.805384+0000","last_undegraded":"2024-04-29T14:10:39.805384+0000","last_fullsized":"2024-04-29T14:10:39.805384+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":620167168,"num_objects":9463,"num_object_clones":0,"num_object_copies":18926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9463,"num_write_kb":605632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":416,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:38.981598+0000","last_change":"2024-04-29T14:08:20.464185+0000","last_active":"2024-04-29T14:10:38.981598+0000","last_peered":"2024-04-29T14:10:38.981598+0000","last_clean":"2024-04-29T14:10:38.981598+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:10:38.981598+0000","last_undegraded":"2024-04-29T14:10:38.981598+0000","last_fullsized":"2024-04-29T14:10:38.981598+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:08:20.464057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:08:20.464057+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"134'9613","reported_seq":9675,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:40.080835+0000","last_change":"2024-04-29T14:09:12.486690+0000","last_active":"2024-04-29T14:10:40.080835+0000","last_peered":"2024-04-29T14:10:40.080835+0000","last_clean":"2024-04-29T14:10:40.080835+0000","last_became_active":"2024-04-29T14:09:12.486361+0000","last_became_peered":"2024-04-29T14:09:12.486361+0000","last_unstale":"2024-04-29T14:10:40.080835+0000","last_undegraded":"2024-04-29T14:10:40.080835+0000","last_fullsized":"2024-04-29T14:10:40.080835+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:42.467931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":629997568,"num_objects":9613,"num_object_clones":0,"num_object_copies":19226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":615232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"134'9430","reported_seq":16202,"reported_epoch":134,"state":"active+clean+scrubbing","last_fresh":"2024-04-29T14:10:41.072148+0000","last_change":"2024-04-29T14:10:40.516009+0000","last_active":"2024-04-29T14:10:41.072148+0000","last_peered":"2024-04-29T14:10:41.072148+0000","last_clean":"2024-04-29T14:10:41.072148+0000","last_became_active":"2024-04-29T14:09:40.988856+0000","last_became_peered":"2024-04-29T14:09:40.988856+0000","last_unstale":"2024-04-29T14:10:41.072148+0000","last_undegraded":"2024-04-29T14:10:41.072148+0000","last_fullsized":"2024-04-29T14:10:41.072148+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":2899,"log_size":9430,"log_dups_size":0,"ondisk_log_size":9430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":618004480,"num_objects":9430,"num_object_clones":0,"num_object_copies":18860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9430,"num_write_kb":603520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3334,"num_bytes_recovered":218497024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"131'9019","reported_seq":9076,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-29T14:10:36.431963+0000","last_change":"2024-04-29T14:09:11.905197+0000","last_active":"2024-04-29T14:10:36.431963+0000","last_peered":"2024-04-29T14:10:36.431963+0000","last_clean":"2024-04-29T14:10:36.431963+0000","last_became_active":"2024-04-29T14:09:11.904732+0000","last_became_peered":"2024-04-29T14:09:11.904732+0000","last_unstale":"2024-04-29T14:10:36.431963+0000","last_undegraded":"2024-04-29T14:10:36.431963+0000","last_fullsized":"2024-04-29T14:10:36.431963+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:37.178319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":591069184,"num_objects":9019,"num_object_clones":0,"num_object_copies":18038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9019,"num_write_kb":577216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"134'9483","reported_seq":9546,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:40.086918+0000","last_change":"2024-04-29T14:09:12.486703+0000","last_active":"2024-04-29T14:10:40.086918+0000","last_peered":"2024-04-29T14:10:40.086918+0000","last_clean":"2024-04-29T14:10:40.086918+0000","last_became_active":"2024-04-29T14:09:12.486357+0000","last_became_peered":"2024-04-29T14:09:12.486357+0000","last_unstale":"2024-04-29T14:10:40.086918+0000","last_undegraded":"2024-04-29T14:10:40.086918+0000","last_fullsized":"2024-04-29T14:10:40.086918+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:40.476942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621477888,"num_objects":9483,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":606912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"134'9428","reported_seq":9491,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:40.070602+0000","last_change":"2024-04-29T14:09:11.907158+0000","last_active":"2024-04-29T14:10:40.070602+0000","last_peered":"2024-04-29T14:10:40.070602+0000","last_clean":"2024-04-29T14:10:40.070602+0000","last_became_active":"2024-04-29T14:09:11.906869+0000","last_became_peered":"2024-04-29T14:09:11.906869+0000","last_unstale":"2024-04-29T14:10:40.070602+0000","last_undegraded":"2024-04-29T14:10:40.070602+0000","last_fullsized":"2024-04-29T14:10:40.070602+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9428,"log_dups_size":0,"ondisk_log_size":9428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617873408,"num_objects":9428,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":603392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"131'9201","reported_seq":9258,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-29T14:10:36.432278+0000","last_change":"2024-04-29T14:09:11.906581+0000","last_active":"2024-04-29T14:10:36.432278+0000","last_peered":"2024-04-29T14:10:36.432278+0000","last_clean":"2024-04-29T14:10:36.432278+0000","last_became_active":"2024-04-29T14:09:11.906295+0000","last_became_peered":"2024-04-29T14:09:11.906295+0000","last_unstale":"2024-04-29T14:10:36.432278+0000","last_undegraded":"2024-04-29T14:10:36.432278+0000","last_fullsized":"2024-04-29T14:10:36.432278+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9201,"log_dups_size":0,"ondisk_log_size":9201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:37.178359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":602996736,"num_objects":9201,"num_object_clones":0,"num_object_copies":18402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9201,"num_write_kb":588864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"134'9565","reported_seq":9628,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:40.220969+0000","last_change":"2024-04-29T14:09:12.586361+0000","last_active":"2024-04-29T14:10:40.220969+0000","last_peered":"2024-04-29T14:10:40.220969+0000","last_clean":"2024-04-29T14:10:40.220969+0000","last_became_active":"2024-04-29T14:09:12.586059+0000","last_became_peered":"2024-04-29T14:09:12.586059+0000","last_unstale":"2024-04-29T14:10:40.220969+0000","last_undegraded":"2024-04-29T14:10:40.220969+0000","last_fullsized":"2024-04-29T14:10:40.220969+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":626851840,"num_objects":9565,"num_object_clones":0,"num_object_copies":19130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":612160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"134'9421","reported_seq":16213,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:40.199842+0000","last_change":"2024-04-29T14:10:05.269323+0000","last_active":"2024-04-29T14:10:40.199842+0000","last_peered":"2024-04-29T14:10:40.199842+0000","last_clean":"2024-04-29T14:10:40.199842+0000","last_became_active":"2024-04-29T14:09:38.488374+0000","last_became_peered":"2024-04-29T14:09:38.488374+0000","last_unstale":"2024-04-29T14:10:40.199842+0000","last_undegraded":"2024-04-29T14:10:40.199842+0000","last_fullsized":"2024-04-29T14:10:40.199842+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9421,"log_dups_size":0,"ondisk_log_size":9421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:40.247165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617414656,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9421,"num_write_kb":602944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3361,"num_bytes_recovered":220266496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.c","version":"134'9408","reported_seq":9471,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:40.244372+0000","last_change":"2024-04-29T14:09:11.904889+0000","last_active":"2024-04-29T14:10:40.244372+0000","last_peered":"2024-04-29T14:10:40.244372+0000","last_clean":"2024-04-29T14:10:40.244372+0000","last_became_active":"2024-04-29T14:09:11.904500+0000","last_became_peered":"2024-04-29T14:09:11.904500+0000","last_unstale":"2024-04-29T14:10:40.244372+0000","last_undegraded":"2024-04-29T14:10:40.244372+0000","last_fullsized":"2024-04-29T14:10:40.244372+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9408,"log_dups_size":0,"ondisk_log_size":9408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:40.247186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":616562688,"num_objects":9408,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9408,"num_write_kb":602112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"134'9577","reported_seq":9639,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-29T14:10:40.227243+0000","last_change":"2024-04-29T14:09:12.100785+0000","last_active":"2024-04-29T14:10:40.227243+0000","last_peered":"2024-04-29T14:10:40.227243+0000","last_clean":"2024-04-29T14:10:40.227243+0000","last_became_active":"2024-04-29T14:09:12.100466+0000","last_became_peered":"2024-04-29T14:09:12.100466+0000","last_unstale":"2024-04-29T14:10:40.227243+0000","last_undegraded":"2024-04-29T14:10:40.227243+0000","last_fullsized":"2024-04-29T14:10:40.227243+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9577,"log_dups_size":0,"ondisk_log_size":9577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":627638272,"num_objects":9577,"num_object_clones":0,"num_object_copies":19154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":612928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"132'9210","reported_seq":9269,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-29T14:10:36.954372+0000","last_change":"2024-04-29T14:09:12.586053+0000","last_active":"2024-04-29T14:10:36.954372+0000","last_peered":"2024-04-29T14:10:36.954372+0000","last_clean":"2024-04-29T14:10:36.954372+0000","last_became_active":"2024-04-29T14:09:12.585722+0000","last_became_peered":"2024-04-29T14:09:12.585722+0000","last_unstale":"2024-04-29T14:10:36.954372+0000","last_undegraded":"2024-04-29T14:10:36.954372+0000","last_fullsized":"2024-04-29T14:10:36.954372+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:09:10.878262+0000","last_clean_scrub_stamp":"2024-04-29T14:09:10.878262+0000","objects_scrubbed":0,"log_size":9210,"log_dups_size":0,"ondisk_log_size":9210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:40.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":603586560,"num_objects":9210,"num_object_clones":0,"num_object_copies":18420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9210,"num_write_kb":589440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":9803333632,"num_objects":149587,"num_object_clones":0,"num_object_copies":299174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":149587,"num_write_kb":9573568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6695,"num_bytes_recovered":438763520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19961741312,"data_stored":19961741312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149587,"ondisk_log_size":149587,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122092,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1894152,"kb_used_data":1818560,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91851000,"statfs":{"total":95995035648,"available":94055424000,"internally_reserved":0,"allocated":1862205440,"data_stored":1861673785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383729,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1249256,"kb_used_data":1195872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92495896,"statfs":{"total":95995035648,"available":94715797504,"internally_reserved":0,"allocated":1224572928,"data_stored":1224057449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645367,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1864968,"kb_used_data":1811576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":91880184,"statfs":{"total":95995035648,"available":94085308416,"internally_reserved":0,"allocated":1855053824,"data_stored":1854525741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877907003,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1571972,"kb_used_data":1514424,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92173180,"statfs":{"total":95995035648,"available":94385336320,"internally_reserved":0,"allocated":1550770176,"data_stored":1550242093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234047,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2486092,"kb_used_data":2418816,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91259060,"statfs":{"total":95995035648,"available":93449277440,"internally_reserved":0,"allocated":2476867584,"data_stored":2476335949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528389,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":634884,"kb_used_data":603704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93110268,"statfs":{"total":95995035648,"available":95344914432,"internally_reserved":0,"allocated":618192896,"data_stored":617664813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2377376,"kb_used_data":2314260,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91367776,"statfs":{"total":95995035648,"available":93560602624,"internally_reserved":0,"allocated":2369802240,"data_stored":2369293143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1871168,"kb_used_data":1808056,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91873984,"statfs":{"total":95995035648,"available":94078959616,"internally_reserved":0,"allocated":1851449344,"data_stored":1850921261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346008,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1814560,"kb_used_data":1757012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91930592,"statfs":{"total":95995035648,"available":94136926208,"internally_reserved":0,"allocated":1799180288,"data_stored":1798671191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851060,"kb_used_data":1793516,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91894092,"statfs":{"total":95995035648,"available":94099550208,"internally_reserved":0,"allocated":1836560384,"data_stored":1836038663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986256,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1282752,"kb_used_data":1211320,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92462400,"statfs":{"total":95995035648,"available":94681497600,"internally_reserved":0,"allocated":1240391680,"data_stored":1239863597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822729,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240396,"kb_used_data":1187000,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92504756,"statfs":{"total":95995035648,"available":94724870144,"internally_reserved":0,"allocated":1215488000,"data_stored":1214959917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1857814528,"data_stored":1857814528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2376204288,"data_stored":2376204288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1811415040,"data_stored":1811415040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1839398912,"data_stored":1839398912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1248067584,"data_stored":1248067584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1222967296,"data_stored":1222967296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":621477888,"data_stored":621477888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2488139776,"data_stored":2488139776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1550057472,"data_stored":1550057472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1860435968,"data_stored":1860435968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1224802304,"data_stored":1224802304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1860960256,"data_stored":1860960256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:10:44.203 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-29T14:10:44.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-04-29T14:10:44.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:44 smithi040 ceph-mon[39581]: 2.7 scrub starts 2024-04-29T14:10:44.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:44 smithi040 ceph-mon[39581]: 2.b scrub ok 2024-04-29T14:10:44.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:44 smithi040 ceph-mon[39581]: from='client.24796 v1:172.21.15.40:0/3978348597' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:10:44.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:44 smithi040 ceph-mon[39581]: 2.8 scrub ok 2024-04-29T14:10:44.269 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:44 smithi040 ceph-mon[39581]: 2.1 scrub starts 2024-04-29T14:10:44.280 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:44.311 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:44 smithi194 ceph-mon[44089]: 2.7 scrub starts 2024-04-29T14:10:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:44 smithi194 ceph-mon[44089]: 2.b scrub ok 2024-04-29T14:10:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:44 smithi194 ceph-mon[44089]: from='client.24796 v1:172.21.15.40:0/3978348597' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:10:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:44 smithi194 ceph-mon[44089]: 2.8 scrub ok 2024-04-29T14:10:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:44 smithi194 ceph-mon[44089]: 2.1 scrub starts 2024-04-29T14:10:44.332 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:44.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:44.359 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12845 2.e 2:7ea11cf9:::benchmark_data_smithi194_79000_object12844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.678348+0000", 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.683048968999998, 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050245286, 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12845 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678348+0000", 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678348+0000", 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678349+0000", 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.186e-06 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678432+0000", 2024-04-29T14:10:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2658999999999996e-05 2024-04-29T14:10:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678439+0000", 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3359999999999997e-06 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678444+0000", 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9840000000000001e-06 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678477+0000", 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2919000000000001e-05 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724390+0000", 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045912717999999998 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724542+0000", 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000151946 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.727597+0000", 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00305537 2024-04-29T14:10:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.728504+0000", 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00090714500000000004 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.728556+0000", 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1363000000000003e-05 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.728593+0000", 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7660000000000002e-05 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:12847 2.e 2:77423bba:::benchmark_data_smithi194_79000_object12846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.678481+0000", 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.682915895999997, 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050383626000000001, 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 12847 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678481+0000", 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678481+0000", 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678482+0000", 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6799999999999999e-07 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.678544+0000", 2024-04-29T14:10:44.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1859000000000005e-05 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724410+0000", 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045866562 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.724420+0000", 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6569999999999992e-06 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.726212+0000", 2024-04-29T14:10:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017922630000000001 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.726316+0000", 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104171 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.726451+0000", 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013518499999999999 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.727602+0000", 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011509599999999999 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.728782+0000", 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001179156 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.728827+0000", 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5936000000000002e-05 2024-04-29T14:10:44.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.728865+0000", 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7109e-05 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:13413 2.1 2:8913d936:::benchmark_data_smithi194_79000_object13412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.832298+0000", 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.529098454999996, 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.056525308000000003, 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.376 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 13413 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.832298+0000", 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.832298+0000", 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.832300+0000", 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0509999999999999e-06 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.832510+0000", 2024-04-29T14:10:44.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020925199999999999 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.832521+0000", 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1717e-05 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.832529+0000", 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6839999999999996e-06 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.832572+0000", 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3259e-05 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886392+0000", 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053819482000000002 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886534+0000", 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014265999999999999 2024-04-29T14:10:44.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888549+0000", 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020143919999999998 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888731+0000", 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018232599999999999 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888784+0000", 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2905999999999998e-05 2024-04-29T14:10:44.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888824+0000", 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9579000000000002e-05 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:13416 2.1 2:8ea58778:::benchmark_data_smithi194_79000_object13415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.833377+0000", 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.528019791000006, 2024-04-29T14:10:44.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.055525524999999999, 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 13416 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.833377+0000", 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.833377+0000", 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.833378+0000", 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.435e-06 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.833518+0000", 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013934 2024-04-29T14:10:44.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886319+0000", 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052800962 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886329+0000", 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0402e-05 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.887175+0000", 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084614399999999995 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.887286+0000", 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011081099999999999 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.887425+0000", 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013898099999999999 2024-04-29T14:10:44.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888610+0000", 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001185296 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888832+0000", 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000222065 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888871+0000", 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8631000000000003e-05 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888903+0000", 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1458000000000003e-05 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:13415 2.3 e113/107 2:cedb9834:::benchmark_data_smithi194_79000_object13414:head v 113'868, mlcod=113'866)", 2024-04-29T14:10:44.388 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:09:24.833971+0000", 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.527426289000005, 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054606790000000002, 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.833971+0000", 2024-04-29T14:10:44.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.833971+0000", 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.833972+0000", 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4109999999999999e-06 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.834039+0000", 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7447000000000007e-05 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886303+0000", 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052264007000000001 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886322+0000", 2024-04-29T14:10:44.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8206000000000002e-05 2024-04-29T14:10:44.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886380+0000", 2024-04-29T14:10:44.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8084000000000002e-05 2024-04-29T14:10:44.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.886404+0000", 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4185999999999998e-05 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888531+0000", 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021266639999999999 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:09:24.888577+0000", 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6785e-05 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:135526 2.e 2:7addc222:::benchmark_data_smithi194_79000_object135525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:28.654511+0000", 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.70688599, 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.070847214000000006, 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.395 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 135526 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.654511+0000", 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.654511+0000", 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.654512+0000", 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3600000000000002e-07 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.654544+0000", 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2693e-05 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.654552+0000", 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9859999999999994e-06 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.654559+0000", 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6850000000000004e-06 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.657887+0000", 2024-04-29T14:10:44.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0033280649999999998 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658028+0000", 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140426 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658172+0000", 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014430700000000001 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.709100+0000", 2024-04-29T14:10:44.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050928441999999997 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725230+0000", 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016129811000000001 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725296+0000", 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6144999999999998e-05 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725358+0000", 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1817999999999997e-05 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:135532 2.e 2:7cf757d0:::benchmark_data_smithi194_79000_object135531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:28.658234+0000", 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.703162502, 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.067222187000000003, 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 135532 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658234+0000", 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658234+0000", 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658236+0000", 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9039999999999999e-06 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658268+0000", 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2020000000000002e-05 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658281+0000", 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2386e-05 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.658289+0000", 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3899999999999993e-06 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.661965+0000", 2024-04-29T14:10:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036761749999999998 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.662112+0000", 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000147174 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.662271+0000", 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000158479 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.713914+0000", 2024-04-29T14:10:44.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051643452999999999 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725370+0000", 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011455686 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725415+0000", 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4835e-05 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725457+0000", 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1684999999999998e-05 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:135539 2.3 e128/107 2:c8857c64:::benchmark_data_smithi194_79000_object135538:head v 128'8529, mlcod=128'8527)", 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:28.666780+0000", 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.694616767999999, 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052786654000000002, 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.407 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.666780+0000", 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.666780+0000", 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.666782+0000", 2024-04-29T14:10:44.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5200000000000001e-06 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.666817+0000", 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5080000000000003e-05 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.710939+0000", 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.044122110999999999 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.710950+0000", 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1222e-05 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.711909+0000", 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00095913200000000002 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.711930+0000", 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0588e-05 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.719526+0000", 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0075958320000000003 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.719567+0000", 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1168999999999998e-05 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.411 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:135548 2.e 2:70a933ed:::benchmark_data_smithi194_79000_object135547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:28.673481+0000", 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.687916009, 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052212133000000001, 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 135548 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.673481+0000", 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.673481+0000", 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.673483+0000", 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.464e-06 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.673592+0000", 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108472 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.709092+0000", 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035499977000000002 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.709128+0000", 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5849999999999997e-05 2024-04-29T14:10:44.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.709263+0000", 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000135604 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.709484+0000", 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022033600000000001 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.709640+0000", 2024-04-29T14:10:44.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015634000000000001 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725517+0000", 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015876939 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725602+0000", 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5351000000000003e-05 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725650+0000", 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7851e-05 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:28.725693+0000", 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2948999999999998e-05 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:137765 2.e 2:761dfd6f:::benchmark_data_smithi194_79000_object137764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e129)", 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:30.244303+0000", 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.117094152, 2024-04-29T14:10:44.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.056592505000000001, 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 137765 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244303+0000", 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244303+0000", 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244304+0000", 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3200000000000001e-06 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244356+0000", 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2302999999999997e-05 2024-04-29T14:10:44.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244376+0000", 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9633e-05 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244404+0000", 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8102999999999998e-05 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244452+0000", 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7681e-05 2024-04-29T14:10:44.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244549+0000", 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7540000000000005e-05 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.244681+0000", 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013179899999999999 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300658+0000", 2024-04-29T14:10:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.055977236 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300787+0000", 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0001284 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300845+0000", 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8687000000000002e-05 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300895+0000", 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9802999999999997e-05 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:137770 2.e 2:70dccb76:::benchmark_data_smithi194_79000_object137769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e129)", 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:30.247773+0000", 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.113624107, 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053293334999999997, 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.423 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 137770 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.247773+0000", 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.247773+0000", 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.247776+0000", 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9069999999999999e-06 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.247814+0000", 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8784e-05 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.247830+0000", 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5404999999999999e-05 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.247839+0000", 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9309999999999997e-06 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248476+0000", 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063734299999999996 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248624+0000", 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014776700000000001 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248776+0000", 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015245100000000001 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300666+0000", 2024-04-29T14:10:44.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051889207 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300990+0000", 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032494199999999999 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.301042+0000", 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1190000000000003e-05 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.301066+0000", 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4408000000000001e-05 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(client.24727.0:137765 2.e e129/107 ondisk, result = 0)", 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:30.248857+0000", 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.112539919, 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052046017999999999, 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248857+0000", 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248857+0000", 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248858+0000", 2024-04-29T14:10:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1800000000000004e-07 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248862+0000", 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9559999999999999e-06 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248872+0000", 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0365e-05 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.248879+0000", 2024-04-29T14:10:44.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9800000000000001e-06 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300772+0000", 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051892487000000001 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300781+0000", 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9439999999999994e-06 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300903+0000", 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000122368 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:137778 2.e 2:765acda6:::benchmark_data_smithi194_79000_object137777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e129)", 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:30.255223+0000", 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.106173548999999, 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052141501999999999, 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.433 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 137778 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.255223+0000", 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.255223+0000", 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.255225+0000", 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9649999999999998e-06 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.255285+0000", 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9475999999999998e-05 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300638+0000", 2024-04-29T14:10:44.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045353752999999997 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.300653+0000", 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4481000000000001e-05 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.301888+0000", 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001234967 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.302000+0000", 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112543 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.302129+0000", 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000128081 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.306869+0000", 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0047405420000000004 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.307252+0000", 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038288500000000002 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.307317+0000", 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4739999999999993e-05 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.307365+0000", 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8068999999999997e-05 2024-04-29T14:10:44.438 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:138282 2.e 2:779f0014:::benchmark_data_smithi194_79000_object138281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e129)", 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:30.617894+0000", 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 13.74350259, 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050554977000000001, 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.439 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 138282 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.617894+0000", 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.617894+0000", 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.617896+0000", 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.482e-06 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.617939+0000", 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3503e-05 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.617949+0000", 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.022e-05 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.617957+0000", 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3529999999999997e-06 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.625317+0000", 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.007360442 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.625431+0000", 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011412399999999999 2024-04-29T14:10:44.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.625587+0000", 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015606300000000001 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.629164+0000", 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035767500000000001 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.668349+0000", 2024-04-29T14:10:44.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039184724999999997 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.668403+0000", 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4128000000000001e-05 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.668449+0000", 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6186999999999997e-05 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:141623 2.1 2:8ca05f03:::benchmark_data_smithi194_79000_object141622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:33.028859+0000", 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.33253755, 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054627022999999997, 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 141623 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.445 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.028859+0000", 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.028859+0000", 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.028861+0000", 2024-04-29T14:10:44.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4929999999999999e-06 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.028973+0000", 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112138 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.028982+0000", 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4269999999999997e-06 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.028989+0000", 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8669999999999996e-06 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.050019+0000", 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021030187999999998 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.050171+0000", 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015178899999999999 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.050338+0000", 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000167174 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.059030+0000", 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.008691364 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.083384+0000", 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024354252999999999 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.083439+0000", 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5112999999999999e-05 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.083486+0000", 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7216999999999999e-05 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:141624 2.1 2:881170b6:::benchmark_data_smithi194_79000_object141623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-04-29T14:10:44.450 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:33.029007+0000", 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.332389894, 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.055947859000000003, 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 141624 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.029007+0000", 2024-04-29T14:10:44.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.029007+0000", 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.029008+0000", 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3600000000000003e-07 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.029040+0000", 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2642999999999999e-05 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.029047+0000", 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6710000000000002e-06 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.029052+0000", 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.293e-06 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.054555+0000", 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025502351999999999 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.054693+0000", 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000137861 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.054827+0000", 2024-04-29T14:10:44.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134592 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.059307+0000", 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044799380000000001 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.084871+0000", 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025563563000000001 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.084917+0000", 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6291000000000001e-05 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:33.084955+0000", 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7919000000000001e-05 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:144725 2.e 2:7e043aff:::benchmark_data_smithi194_79000_object144724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:35.262426+0000", 2024-04-29T14:10:44.456 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.0989707820000003, 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051546651999999998, 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 144725 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.262426+0000", 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.262426+0000", 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.262427+0000", 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1710000000000001e-06 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.262497+0000", 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9821999999999997e-05 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.262503+0000", 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4180000000000002e-06 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.262508+0000", 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0890000000000002e-06 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.295431+0000", 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032923239 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.295545+0000", 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000113826 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.295675+0000", 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013031800000000001 2024-04-29T14:10:44.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.304533+0000", 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0088582279999999992 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.313879+0000", 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0093458410000000006 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.313931+0000", 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2454999999999998e-05 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:35.313973+0000", 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1245000000000002e-05 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:154030 2.e 2:7ae7faf2:::benchmark_data_smithi194_79000_object154029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:43.204581+0000", 2024-04-29T14:10:44.462 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.156815352, 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054408749999999999, 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154030 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.204581+0000", 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.204581+0000", 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.204585+0000", 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.204626+0000", 2024-04-29T14:10:44.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0559999999999998e-05 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.204641+0000", 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4871e-05 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.204652+0000", 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1347e-05 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.206634+0000", 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019817250000000002 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.206730+0000", 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6600000000000003e-05 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.206892+0000", 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016156800000000001 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.244173+0000", 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037280817000000001 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.258864+0000", 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014690964000000001 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.258925+0000", 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1100999999999997e-05 2024-04-29T14:10:44.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.258990+0000", 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5604000000000004e-05 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:154031 2.e 2:76ff5956:::benchmark_data_smithi194_79000_object154030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:43.207097+0000", 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.1542994120000001, 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052025555000000001, 2024-04-29T14:10:44.468 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154031 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.207097+0000", 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.207097+0000", 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.207100+0000", 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.74e-06 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.207142+0000", 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1480000000000003e-05 2024-04-29T14:10:44.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.207153+0000", 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1004e-05 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.207162+0000", 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8060000000000006e-06 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.212205+0000", 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050422710000000001 2024-04-29T14:10:44.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.220490+0000", 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082857109999999994 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.220671+0000", 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018069000000000001 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.244178+0000", 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023506721000000001 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.259031+0000", 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014853151 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.259079+0000", 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7729000000000003e-05 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.259123+0000", 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4252000000000003e-05 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:154462 2.1 2:880c41b0:::benchmark_data_smithi194_79000_object154461:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:43.702257+0000", 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.65913951900000001, 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051223440000000002, 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:10:44.474 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154462 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702257+0000", 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702257+0000", 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:10:44.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702260+0000", 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8289999999999999e-06 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702379+0000", 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119237 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702390+0000", 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0662000000000001e-05 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702397+0000", 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3479999999999998e-06 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702457+0000", 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0000000000000002e-05 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702590+0000", 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000133036 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.702734+0000", 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014364099999999999 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.728343+0000", 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025608801 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.753379+0000", 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025035973 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.753427+0000", 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8334000000000002e-05 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.753481+0000", 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3579000000000003e-05 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:10:44.479 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:10:44.480 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:44.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:44 smithi121 ceph-mon[44533]: 2.7 scrub starts 2024-04-29T14:10:44.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:44 smithi121 ceph-mon[44533]: 2.b scrub ok 2024-04-29T14:10:44.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:44 smithi121 ceph-mon[44533]: from='client.24796 v1:172.21.15.40:0/3978348597' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:10:44.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:44 smithi121 ceph-mon[44533]: 2.8 scrub ok 2024-04-29T14:10:44.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:44 smithi121 ceph-mon[44533]: 2.1 scrub starts 2024-04-29T14:10:44.481 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 155159 155143 118.231 63.25 0.00437267 0.00845582 2024-04-29T14:10:44.527 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:44.557 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:44.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:44.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:44.679 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:44.925 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:44.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:45.030 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:10:45.062 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:45.098 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:10:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:45.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:10:45.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:45 smithi040 ceph-mon[39581]: pgmap v415: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-29T14:10:45.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:45 smithi040 ceph-mon[39581]: 2.0 scrub starts 2024-04-29T14:10:45.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:45 smithi040 ceph-mon[39581]: 2.7 scrub ok 2024-04-29T14:10:45.185 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:45 smithi040 ceph-mon[39581]: 2.c scrub ok 2024-04-29T14:10:45.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:45.267 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:45 smithi194 ceph-mon[44089]: pgmap v415: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-29T14:10:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:45 smithi194 ceph-mon[44089]: 2.0 scrub starts 2024-04-29T14:10:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:45 smithi194 ceph-mon[44089]: 2.7 scrub ok 2024-04-29T14:10:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:45 smithi194 ceph-mon[44089]: 2.c scrub ok 2024-04-29T14:10:45.367 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 15 156317 156302 117.679 72.4375 0.0050349 0.00849648 2024-04-29T14:10:45.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:45 smithi121 ceph-mon[44533]: pgmap v415: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-29T14:10:45.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:45 smithi121 ceph-mon[44533]: 2.0 scrub starts 2024-04-29T14:10:45.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:45 smithi121 ceph-mon[44533]: 2.7 scrub ok 2024-04-29T14:10:45.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:45 smithi121 ceph-mon[44533]: 2.c scrub ok 2024-04-29T14:10:45.689 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:10:45.727 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:10:45.766 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:10:45.819 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:10:45.848 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:10:45.880 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:10:45.915 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:10:45.954 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:10:45.993 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:10:46.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:10:46.074 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:10:46.107 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:10:46.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:46 smithi040 ceph-mon[39581]: 2.e scrub ok 2024-04-29T14:10:46.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:46 smithi040 ceph-mon[39581]: 2.0 scrub ok 2024-04-29T14:10:46.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/516379028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:46.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:46.114 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:10:46.114 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:10:46.114 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:10:46.114 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:10:46.114 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:10:46.114 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:10:46.115 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:10:46.115 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:10:46.115 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:10:46.115 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:10:46.115 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:10:46.115 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:10:46.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 157586 157570 117.22 79.25 0.0129242 0.00852964 2024-04-29T14:10:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:46 smithi121 ceph-mon[44533]: 2.e scrub ok 2024-04-29T14:10:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:46 smithi121 ceph-mon[44533]: 2.0 scrub ok 2024-04-29T14:10:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/516379028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:46.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:46 smithi194 ceph-mon[44089]: 2.e scrub ok 2024-04-29T14:10:46.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:46 smithi194 ceph-mon[44089]: 2.0 scrub ok 2024-04-29T14:10:46.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/516379028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:46.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: 2.1 scrub ok 2024-04-29T14:10:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: pgmap v416: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:10:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 16 158854 158838 116.773 79.25 0.00719042 0.00856234 2024-04-29T14:10:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: 2.1 scrub ok 2024-04-29T14:10:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: pgmap v416: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:10:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:47.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: 2.1 scrub ok 2024-04-29T14:10:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: pgmap v416: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:10:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:47.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:10:47.680 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:47.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:10:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:47.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:10:47.808 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:47.837 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:47.864 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:47.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:47.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:48.059 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:48.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:48 smithi194 ceph-mon[44089]: 2.6 scrub starts 2024-04-29T14:10:48.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:48 smithi194 ceph-mon[44089]: 2.a scrub starts 2024-04-29T14:10:48.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:48 smithi194 ceph-mon[44089]: pgmap v417: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-29T14:10:48.140 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:48.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:48.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:10:48.247 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:48.287 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:48.322 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:48.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:48 smithi040 ceph-mon[39581]: 2.6 scrub starts 2024-04-29T14:10:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:48 smithi040 ceph-mon[39581]: 2.a scrub starts 2024-04-29T14:10:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:48 smithi040 ceph-mon[39581]: pgmap v417: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-29T14:10:48.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 159897 159881 116.173 65.1875 0.0087035 0.00860623 2024-04-29T14:10:48.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:48 smithi121 ceph-mon[44533]: 2.6 scrub starts 2024-04-29T14:10:48.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:48 smithi121 ceph-mon[44533]: 2.a scrub starts 2024-04-29T14:10:48.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:48 smithi121 ceph-mon[44533]: pgmap v417: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-29T14:10:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:48.428 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:48.461 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:48.488 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:48.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:48.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:48.680 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:48.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:48.788 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:48.818 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:48.843 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:48.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:48.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:49.124 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2859009396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:49.124 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:49 smithi194 ceph-mon[44089]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:49.147 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:10:49.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:10:49.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2859009396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:49.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:49 smithi040 ceph-mon[39581]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:49.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 160962 160946 115.603 66.5625 0.00931058 0.00864896 2024-04-29T14:10:49.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2859009396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:49.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:49 smithi121 ceph-mon[44533]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:10:49.550 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:49.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:49.594 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:49.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:49.715 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:50.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 162126 162110 115.116 72.75 0.0107162 0.0086856 2024-04-29T14:10:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:50 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:10:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:50 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:10:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:50 smithi040 ceph-mon[39581]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:10:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:50 smithi040 ceph-mon[39581]: osdmap e135: 12 total, 12 up, 12 in 2024-04-29T14:10:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:50 smithi040 ceph-mon[39581]: 2.6 scrub ok 2024-04-29T14:10:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:50 smithi040 ceph-mon[39581]: 2.a scrub ok 2024-04-29T14:10:50.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:50 smithi040 ceph-mon[39581]: pgmap v419: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:10:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:50 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:10:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:50 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:10:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:50 smithi194 ceph-mon[44089]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:10:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:50 smithi194 ceph-mon[44089]: osdmap e135: 12 total, 12 up, 12 in 2024-04-29T14:10:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:50 smithi194 ceph-mon[44089]: 2.6 scrub ok 2024-04-29T14:10:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:50 smithi194 ceph-mon[44089]: 2.a scrub ok 2024-04-29T14:10:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:50 smithi194 ceph-mon[44089]: pgmap v419: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:10:50.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:50 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:10:50.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:50 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:10:50.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:50 smithi121 ceph-mon[44533]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:10:50.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:50 smithi121 ceph-mon[44533]: osdmap e135: 12 total, 12 up, 12 in 2024-04-29T14:10:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:50 smithi121 ceph-mon[44533]: 2.6 scrub ok 2024-04-29T14:10:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:50 smithi121 ceph-mon[44533]: 2.a scrub ok 2024-04-29T14:10:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:50 smithi121 ceph-mon[44533]: pgmap v419: 17 pgs: 17 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:10:51.274 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:51 smithi040 ceph-mon[39581]: 2.9 deep-scrub starts 2024-04-29T14:10:51.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 163194 163178 114.573 66.75 0.00954375 0.00872533 2024-04-29T14:10:51.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:51 smithi194 ceph-mon[44089]: 2.9 deep-scrub starts 2024-04-29T14:10:51.612 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:51.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:51.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:51 smithi121 ceph-mon[44533]: 2.9 deep-scrub starts 2024-04-29T14:10:51.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:51.770 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:52.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:52.182 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 2 pg_num to 26 2024-04-29T14:10:52.219 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:52.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:52.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:52 smithi040 ceph-mon[39581]: pgmap v420: 17 pgs: 17 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:10:52.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1039058052' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:10:52.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:52 smithi040 ceph-mon[39581]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:10:52.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 90 15 164258 164243 114.039 66.5625 0.0181151 0.00876535 2024-04-29T14:10:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:52.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:52.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:10:52.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:10:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:52 smithi194 ceph-mon[44089]: pgmap v420: 17 pgs: 17 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:10:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1039058052' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:10:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:52 smithi194 ceph-mon[44089]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:10:52.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:52 smithi121 ceph-mon[44533]: pgmap v420: 17 pgs: 17 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:10:52.579 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1039058052' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:10:52.579 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:52 smithi121 ceph-mon[44533]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:10:52.580 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:52.601 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:10:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:52.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:10:52.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:10:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:53.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:53 smithi040 ceph-mon[39581]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-04-29T14:10:53.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:53 smithi040 ceph-mon[39581]: osdmap e136: 12 total, 12 up, 12 in 2024-04-29T14:10:53.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 90.0455 2024-04-29T14:10:53.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 164258 2024-04-29T14:10:53.369 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 114.01 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 62.0906 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 361.688 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 0 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1824 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 993.458 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 5787 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 0 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.00876969 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.0323592 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 4.06786 2024-04-29T14:10:53.370 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.00107894 2024-04-29T14:10:53.376 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:10:53.413 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:10:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:53.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:10:53.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:53.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:53 smithi194 ceph-mon[44089]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-04-29T14:10:53.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:53 smithi194 ceph-mon[44089]: osdmap e136: 12 total, 12 up, 12 in 2024-04-29T14:10:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:53 smithi121 ceph-mon[44533]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-04-29T14:10:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:53 smithi121 ceph-mon[44533]: osdmap e136: 12 total, 12 up, 12 in 2024-04-29T14:10:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:54.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:54.166 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:54.253 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:54.378 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:54 smithi040 ceph-mon[39581]: pgmap v422: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-04-29T14:10:54.379 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:54.379 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:54 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:54 smithi194 ceph-mon[44089]: pgmap v422: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-04-29T14:10:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:54.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:54 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:54.595 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:10:54.595 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:10:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:54 smithi121 ceph-mon[44533]: pgmap v422: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-04-29T14:10:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:54 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:55.133 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:55.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:55.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:10:55.190 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:55.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:10:55.266 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:55.290 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:10:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:55.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:10:55.395 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:55.427 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:55 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-29T14:10:55.427 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:55 smithi194 ceph-mon[44089]: osdmap e137: 12 total, 12 up, 12 in 2024-04-29T14:10:55.429 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:55.461 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:55.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:55.481 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:55 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-29T14:10:55.481 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:55 smithi040 ceph-mon[39581]: osdmap e137: 12 total, 12 up, 12 in 2024-04-29T14:10:55.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:10:55.613 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:55 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-29T14:10:55.613 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:55 smithi121 ceph-mon[44533]: osdmap e137: 12 total, 12 up, 12 in 2024-04-29T14:10:55.615 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:55.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:10:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:55.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:10:55.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:10:55.784 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:55.813 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:10:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:55.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:10:55.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:55.924 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:10:55.970 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:10:56.023 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:56.035 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:10:56.090 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:10:56.118 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:56.123 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:10:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:56.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:56.159 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:10:56.207 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:10:56.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:56.245 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:10:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:56.287 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:10:56.313 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:10:56.347 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:10:56.371 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:10:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:10:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:56 smithi040 ceph-mon[39581]: osdmap e138: 12 total, 12 up, 12 in 2024-04-29T14:10:56.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:56 smithi040 ceph-mon[39581]: 2.8 scrub starts 2024-04-29T14:10:56.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:56 smithi040 ceph-mon[39581]: pgmap v425: 27 pgs: 5 peering, 10 unknown, 1 active+clean+scrubbing+deep, 11 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 73 MiB/s wr, 1.25k op/s 2024-04-29T14:10:56.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:56.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:56 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:56.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:56 smithi040 ceph-mon[39581]: 2.5 scrub starts 2024-04-29T14:10:56.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2033881293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:56 smithi194 ceph-mon[44089]: osdmap e138: 12 total, 12 up, 12 in 2024-04-29T14:10:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:56 smithi194 ceph-mon[44089]: 2.8 scrub starts 2024-04-29T14:10:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:56 smithi194 ceph-mon[44089]: pgmap v425: 27 pgs: 5 peering, 10 unknown, 1 active+clean+scrubbing+deep, 11 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 73 MiB/s wr, 1.25k op/s 2024-04-29T14:10:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:56 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:56 smithi194 ceph-mon[44089]: 2.5 scrub starts 2024-04-29T14:10:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2033881293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:56.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:56 smithi121 ceph-mon[44533]: osdmap e138: 12 total, 12 up, 12 in 2024-04-29T14:10:56.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:56 smithi121 ceph-mon[44533]: 2.8 scrub starts 2024-04-29T14:10:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:56 smithi121 ceph-mon[44533]: pgmap v425: 27 pgs: 5 peering, 10 unknown, 1 active+clean+scrubbing+deep, 11 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 73 MiB/s wr, 1.25k op/s 2024-04-29T14:10:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:56 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:10:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:56 smithi121 ceph-mon[44533]: 2.5 scrub starts 2024-04-29T14:10:56.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2033881293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:10:57.077 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:10:57.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.1 scrub starts 2024-04-29T14:10:57.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.7 scrub starts 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.2 scrub starts 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.7 scrub ok 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.8 scrub ok 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.308 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osdmap e139: 12 total, 12 up, 12 in 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.6 scrub starts 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.5 scrub ok 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.0 deep-scrub starts 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: 2.2 scrub ok 2024-04-29T14:10:57.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:57 smithi040 ceph-mon[39581]: osdmap e140: 12 total, 12 up, 12 in 2024-04-29T14:10:57.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.1 scrub starts 2024-04-29T14:10:57.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.7 scrub starts 2024-04-29T14:10:57.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.2 scrub starts 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.7 scrub ok 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.8 scrub ok 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osdmap e139: 12 total, 12 up, 12 in 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.6 scrub starts 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.5 scrub ok 2024-04-29T14:10:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.0 deep-scrub starts 2024-04-29T14:10:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: 2.2 scrub ok 2024-04-29T14:10:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:57 smithi194 ceph-mon[44089]: osdmap e140: 12 total, 12 up, 12 in 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.1 scrub starts 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.7 scrub starts 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.2 scrub starts 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.7 scrub ok 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.8 scrub ok 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osdmap e139: 12 total, 12 up, 12 in 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.6 scrub starts 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.5 scrub ok 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.0 deep-scrub starts 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: 2.2 scrub ok 2024-04-29T14:10:57.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:57 smithi121 ceph-mon[44533]: osdmap e140: 12 total, 12 up, 12 in 2024-04-29T14:10:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:57.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:57.828 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:57.855 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:57.875 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:57.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:57.980 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:58.017 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:58.043 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:58.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:58.146 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:10:58.192 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:58.219 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:10:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:58.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:10:58.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:10:58.373 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: 2.1 scrub ok 2024-04-29T14:10:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: 2.17 deep-scrub starts 2024-04-29T14:10:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: 2.6 scrub ok 2024-04-29T14:10:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: 2.3 scrub starts 2024-04-29T14:10:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: 2.9 deep-scrub starts 2024-04-29T14:10:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: 2.3 scrub ok 2024-04-29T14:10:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: pgmap v428: 27 pgs: 2 remapped+peering, 2 active+clean+scrubbing, 5 peering, 8 unknown, 10 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 0 op/s 2024-04-29T14:10:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:58 smithi194 ceph-mon[44089]: osdmap e141: 12 total, 12 up, 12 in 2024-04-29T14:10:58.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: 2.1 scrub ok 2024-04-29T14:10:58.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: 2.17 deep-scrub starts 2024-04-29T14:10:58.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: 2.6 scrub ok 2024-04-29T14:10:58.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: 2.3 scrub starts 2024-04-29T14:10:58.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: 2.9 deep-scrub starts 2024-04-29T14:10:58.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: 2.3 scrub ok 2024-04-29T14:10:58.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: pgmap v428: 27 pgs: 2 remapped+peering, 2 active+clean+scrubbing, 5 peering, 8 unknown, 10 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 0 op/s 2024-04-29T14:10:58.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:58 smithi040 ceph-mon[39581]: osdmap e141: 12 total, 12 up, 12 in 2024-04-29T14:10:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: 2.1 scrub ok 2024-04-29T14:10:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: 2.17 deep-scrub starts 2024-04-29T14:10:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: 2.6 scrub ok 2024-04-29T14:10:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: 2.3 scrub starts 2024-04-29T14:10:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: 2.9 deep-scrub starts 2024-04-29T14:10:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: 2.3 scrub ok 2024-04-29T14:10:58.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: pgmap v428: 27 pgs: 2 remapped+peering, 2 active+clean+scrubbing, 5 peering, 8 unknown, 10 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 0 op/s 2024-04-29T14:10:58.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:58 smithi121 ceph-mon[44533]: osdmap e141: 12 total, 12 up, 12 in 2024-04-29T14:10:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:10:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:58.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:10:59.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:59.112 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:59.148 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:10:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:59.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:10:59.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:10:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:10:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:59.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.11 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10398 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.19 starting backfill to osd.3 from (0'0,0'0] MAX to 138'11166 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.12 starting backfill to osd.3 from (0'0,0'0] MAX to 138'10196 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.13 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10390 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.16 starting backfill to osd.5 from (0'0,0'0] MAX to 138'10166 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.14 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10289 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.15 starting backfill to osd.1 from (0'0,0'0] MAX to 138'10277 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.18 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10381 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.10 starting backfill to osd.1 from (0'0,0'0] MAX to 138'10542 2024-04-29T14:10:59.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.10 starting backfill to osd.8 from (0'0,0'0] MAX to 138'10542 2024-04-29T14:10:59.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.11 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10398 2024-04-29T14:10:59.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.14 starting backfill to osd.9 from (0'0,0'0] MAX to 138'10289 2024-04-29T14:10:59.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.15 starting backfill to osd.5 from (0'0,0'0] MAX to 138'10277 2024-04-29T14:10:59.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.12 starting backfill to osd.10 from (0'0,0'0] MAX to 138'10196 2024-04-29T14:10:59.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.19 starting backfill to osd.6 from (0'0,0'0] MAX to 138'11166 2024-04-29T14:10:59.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.16 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10166 2024-04-29T14:10:59.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:10:59 smithi040 ceph-mon[39581]: 2.13 starting backfill to osd.9 from (0'0,0'0] MAX to 138'10390 2024-04-29T14:10:59.540 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:10:59.541 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:10:59.541 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-29T14:10:59.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-04-29T14:10:59.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:10:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.11 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10398 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.19 starting backfill to osd.3 from (0'0,0'0] MAX to 138'11166 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.12 starting backfill to osd.3 from (0'0,0'0] MAX to 138'10196 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.13 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10390 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.16 starting backfill to osd.5 from (0'0,0'0] MAX to 138'10166 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.14 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10289 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.15 starting backfill to osd.1 from (0'0,0'0] MAX to 138'10277 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.18 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10381 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.10 starting backfill to osd.1 from (0'0,0'0] MAX to 138'10542 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.10 starting backfill to osd.8 from (0'0,0'0] MAX to 138'10542 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.11 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10398 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.14 starting backfill to osd.9 from (0'0,0'0] MAX to 138'10289 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.15 starting backfill to osd.5 from (0'0,0'0] MAX to 138'10277 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.12 starting backfill to osd.10 from (0'0,0'0] MAX to 138'10196 2024-04-29T14:10:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.19 starting backfill to osd.6 from (0'0,0'0] MAX to 138'11166 2024-04-29T14:10:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.16 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10166 2024-04-29T14:10:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:10:59 smithi194 ceph-mon[44089]: 2.13 starting backfill to osd.9 from (0'0,0'0] MAX to 138'10390 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.11 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10398 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.19 starting backfill to osd.3 from (0'0,0'0] MAX to 138'11166 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.12 starting backfill to osd.3 from (0'0,0'0] MAX to 138'10196 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.13 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10390 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.16 starting backfill to osd.5 from (0'0,0'0] MAX to 138'10166 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.14 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10289 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.15 starting backfill to osd.1 from (0'0,0'0] MAX to 138'10277 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.18 starting backfill to osd.4 from (0'0,0'0] MAX to 138'10381 2024-04-29T14:10:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.10 starting backfill to osd.1 from (0'0,0'0] MAX to 138'10542 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.10 starting backfill to osd.8 from (0'0,0'0] MAX to 138'10542 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.11 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10398 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.14 starting backfill to osd.9 from (0'0,0'0] MAX to 138'10289 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.15 starting backfill to osd.5 from (0'0,0'0] MAX to 138'10277 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.12 starting backfill to osd.10 from (0'0,0'0] MAX to 138'10196 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.19 starting backfill to osd.6 from (0'0,0'0] MAX to 138'11166 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.16 starting backfill to osd.6 from (0'0,0'0] MAX to 138'10166 2024-04-29T14:10:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:10:59 smithi121 ceph-mon[44533]: 2.13 starting backfill to osd.9 from (0'0,0'0] MAX to 138'10390 2024-04-29T14:10:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:59.816 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:10:59.816 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:10:59.816 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:10:59.816 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:10:59.816 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:10:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:10:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:10:59.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:10:59.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:10:59.965 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:10:59.996 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:11:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:00.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:11:00.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:11:00.152 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:00.181 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:11:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:00.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:11:00.287 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:11:00.354 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:00.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:00 smithi040 ceph-mon[39581]: pgmap v430: 27 pgs: 2 remapped+peering, 2 active+clean+scrubbing, 5 peering, 8 unknown, 10 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 239 B/s rd, 0 op/s 2024-04-29T14:11:00.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:00 smithi194 ceph-mon[44089]: pgmap v430: 27 pgs: 2 remapped+peering, 2 active+clean+scrubbing, 5 peering, 8 unknown, 10 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 239 B/s rd, 0 op/s 2024-04-29T14:11:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:11:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:00.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:11:00.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:11:00.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:00 smithi121 ceph-mon[44533]: pgmap v430: 27 pgs: 2 remapped+peering, 2 active+clean+scrubbing, 5 peering, 8 unknown, 10 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 239 B/s rd, 0 op/s 2024-04-29T14:11:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:00.684 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:11:00.726 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:00.756 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:11:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:00.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:11:00.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:11:00.916 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:00.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:11:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:00.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:11:00.999 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:11:01.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:01.103 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:01.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:01.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:11:01.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:01.308 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3109897547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:11:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:01 smithi194 ceph-mon[44089]: from='client.24814 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:11:01.600 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:01.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:01.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3109897547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:11:01.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:01 smithi040 ceph-mon[39581]: from='client.24814 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:11:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3109897547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:11:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:01 smithi121 ceph-mon[44533]: from='client.24814 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:11:01.711 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:01.744 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:01.772 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:01.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:01.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:11:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:01.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:11:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:01.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:11:02.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:11:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:11:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:02.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:11:02.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:02.286 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:02.338 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:02.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:02.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:02.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:02.492 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:02.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:02.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:02.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:02 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:02.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:02 smithi121 ceph-mon[44533]: from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:11:02.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:02 smithi121 ceph-mon[44533]: osdmap e142: 12 total, 12 up, 12 in 2024-04-29T14:11:02.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:02 smithi121 ceph-mon[44533]: pgmap v432: 27 pgs: 2 active+clean+scrubbing+deep, 2 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 3 unknown, 15 active+clean; 9.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32896/302114 objects misplaced (10.889%) 2024-04-29T14:11:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:02 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:02 smithi194 ceph-mon[44089]: from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:11:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:02 smithi194 ceph-mon[44089]: osdmap e142: 12 total, 12 up, 12 in 2024-04-29T14:11:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:02 smithi194 ceph-mon[44089]: pgmap v432: 27 pgs: 2 active+clean+scrubbing+deep, 2 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 3 unknown, 15 active+clean; 9.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32896/302114 objects misplaced (10.889%) 2024-04-29T14:11:02.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:02 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:02.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:02 smithi040 ceph-mon[39581]: from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:11:02.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:02 smithi040 ceph-mon[39581]: osdmap e142: 12 total, 12 up, 12 in 2024-04-29T14:11:02.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:02 smithi040 ceph-mon[39581]: pgmap v432: 27 pgs: 2 active+clean+scrubbing+deep, 2 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 3 unknown, 15 active+clean; 9.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32896/302114 objects misplaced (10.889%) 2024-04-29T14:11:02.639 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:02.695 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:02.761 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:02.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:02.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:03.030 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:03.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:03.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:11:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:11:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:03.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:11:03.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:03.404 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:03.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:03.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:04.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:11:04.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:04 smithi040 ceph-mon[39581]: pgmap v433: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78512/320162 objects misplaced (24.523%); 33 MiB/s, 525 objects/s recovering 2024-04-29T14:11:04.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:04 smithi194 ceph-mon[44089]: pgmap v433: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78512/320162 objects misplaced (24.523%); 33 MiB/s, 525 objects/s recovering 2024-04-29T14:11:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:04 smithi121 ceph-mon[44533]: pgmap v433: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78512/320162 objects misplaced (24.523%); 33 MiB/s, 525 objects/s recovering 2024-04-29T14:11:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:05.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:06.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:11:06.084 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:06.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:11:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:06.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:11:06.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:11:06.265 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:06.298 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:11:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:06.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:11:06.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:06.439 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:06.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:06.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:11:06.610 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:06.639 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:11:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:06.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:11:06.746 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:11:06.784 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:06 smithi194 ceph-mon[44089]: pgmap v434: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 76388/316100 objects misplaced (24.166%); 40 MiB/s, 641 objects/s recovering 2024-04-29T14:11:06.787 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:06.814 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:11:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:06.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:11:06.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:06 smithi040 ceph-mon[39581]: pgmap v434: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 76388/316100 objects misplaced (24.166%); 40 MiB/s, 641 objects/s recovering 2024-04-29T14:11:06.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:06 smithi121 ceph-mon[44533]: pgmap v434: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 76388/316100 objects misplaced (24.166%); 40 MiB/s, 641 objects/s recovering 2024-04-29T14:11:06.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:06.975 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:07.239 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:11:07.241 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:11:07.241 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:11:07.242 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:11:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:08 smithi194 ceph-mon[44089]: pgmap v435: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65890/313048 objects misplaced (21.048%); 63 MiB/s, 1.01k objects/s recovering 2024-04-29T14:11:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:08 smithi040 ceph-mon[39581]: pgmap v435: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65890/313048 objects misplaced (21.048%); 63 MiB/s, 1.01k objects/s recovering 2024-04-29T14:11:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:08 smithi121 ceph-mon[44533]: pgmap v435: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65890/313048 objects misplaced (21.048%); 63 MiB/s, 1.01k objects/s recovering 2024-04-29T14:11:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:10 smithi194 ceph-mon[44089]: pgmap v436: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65890/312648 objects misplaced (21.075%); 61 MiB/s, 969 objects/s recovering 2024-04-29T14:11:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:10 smithi040 ceph-mon[39581]: pgmap v436: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65890/312648 objects misplaced (21.075%); 61 MiB/s, 969 objects/s recovering 2024-04-29T14:11:11.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:10 smithi121 ceph-mon[44533]: pgmap v436: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65890/312648 objects misplaced (21.075%); 61 MiB/s, 969 objects/s recovering 2024-04-29T14:11:13.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:12 smithi194 ceph-mon[44089]: pgmap v437: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 1.17k op/s; 60475/308778 objects misplaced (19.585%); 70 MiB/s, 1.12k objects/s recovering 2024-04-29T14:11:13.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:12 smithi194 ceph-mon[44089]: 2.17 deep-scrub ok 2024-04-29T14:11:13.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:12 smithi040 ceph-mon[39581]: pgmap v437: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 1.17k op/s; 60475/308778 objects misplaced (19.585%); 70 MiB/s, 1.12k objects/s recovering 2024-04-29T14:11:13.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:12 smithi040 ceph-mon[39581]: 2.17 deep-scrub ok 2024-04-29T14:11:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:12 smithi121 ceph-mon[44533]: pgmap v437: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 1.17k op/s; 60475/308778 objects misplaced (19.585%); 70 MiB/s, 1.12k objects/s recovering 2024-04-29T14:11:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:12 smithi121 ceph-mon[44533]: 2.17 deep-scrub ok 2024-04-29T14:11:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:14 smithi194 ceph-mon[44089]: pgmap v438: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 1.08k op/s; 53005/306624 objects misplaced (17.287%); 77 MiB/s, 1.24k objects/s recovering 2024-04-29T14:11:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:14 smithi194 ceph-mon[44089]: osdmap e143: 12 total, 12 up, 12 in 2024-04-29T14:11:15.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:14 smithi040 ceph-mon[39581]: pgmap v438: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 1.08k op/s; 53005/306624 objects misplaced (17.287%); 77 MiB/s, 1.24k objects/s recovering 2024-04-29T14:11:15.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:14 smithi040 ceph-mon[39581]: osdmap e143: 12 total, 12 up, 12 in 2024-04-29T14:11:15.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:14 smithi121 ceph-mon[44533]: pgmap v438: 27 pgs: 3 active+clean+scrubbing+deep, 9 active+remapped+backfilling, 15 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 1.08k op/s; 53005/306624 objects misplaced (17.287%); 77 MiB/s, 1.24k objects/s recovering 2024-04-29T14:11:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:14 smithi121 ceph-mon[44533]: osdmap e143: 12 total, 12 up, 12 in 2024-04-29T14:11:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:15 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:15 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:15 smithi194 ceph-mon[44089]: osdmap e144: 12 total, 12 up, 12 in 2024-04-29T14:11:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:15 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:15 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:15 smithi040 ceph-mon[39581]: osdmap e144: 12 total, 12 up, 12 in 2024-04-29T14:11:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:15 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:15 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:15 smithi121 ceph-mon[44533]: osdmap e144: 12 total, 12 up, 12 in 2024-04-29T14:11:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:16.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:16.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:11:16.844 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:16.866 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:11:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:16.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:11:16.969 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:17.003 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:17.029 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:17.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:17.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:16 smithi121 ceph-mon[44533]: pgmap v441: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51406/302876 objects misplaced (16.973%); 61 MiB/s, 982 objects/s recovering 2024-04-29T14:11:17.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:17.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:17.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:17.170 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:17.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:17.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:17.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:16 smithi194 ceph-mon[44089]: pgmap v441: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51406/302876 objects misplaced (16.973%); 61 MiB/s, 982 objects/s recovering 2024-04-29T14:11:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:17.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:16 smithi040 ceph-mon[39581]: pgmap v441: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51406/302876 objects misplaced (16.973%); 61 MiB/s, 982 objects/s recovering 2024-04-29T14:11:17.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:17.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:17.357 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:17.816 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:17.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:17.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:11:17.961 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:17.984 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:11:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:17.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:11:18.088 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:11:18.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:18 smithi040 ceph-mon[39581]: osdmap e145: 12 total, 12 up, 12 in 2024-04-29T14:11:18.122 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:18.151 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:11:18.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:18.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:11:18.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:18 smithi194 ceph-mon[44089]: osdmap e145: 12 total, 12 up, 12 in 2024-04-29T14:11:18.334 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:18 smithi121 ceph-mon[44533]: osdmap e145: 12 total, 12 up, 12 in 2024-04-29T14:11:19.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:19 smithi040 ceph-mon[39581]: pgmap v443: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40636/300220 objects misplaced (13.535%); 101 MiB/s, 1.61k objects/s recovering 2024-04-29T14:11:19.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:19 smithi040 ceph-mon[39581]: 2.0 deep-scrub ok 2024-04-29T14:11:19.215 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:19.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:11:19.243 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:11:19.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-04-29T14:11:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:19 smithi194 ceph-mon[44089]: pgmap v443: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40636/300220 objects misplaced (13.535%); 101 MiB/s, 1.61k objects/s recovering 2024-04-29T14:11:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:19 smithi194 ceph-mon[44089]: 2.0 deep-scrub ok 2024-04-29T14:11:19.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:11:19.353 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:19.375 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:11:19.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:19 smithi121 ceph-mon[44533]: pgmap v443: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40636/300220 objects misplaced (13.535%); 101 MiB/s, 1.61k objects/s recovering 2024-04-29T14:11:19.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:19 smithi121 ceph-mon[44533]: 2.0 deep-scrub ok 2024-04-29T14:11:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:19.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:11:19.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:20.319 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:20.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:109766 2.1 e123/107 2:8d09ee08:::benchmark_data_smithi194_79000_object109765:head v 123'6850, mlcod=123'6848)", 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:11.312952+0000", 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 69.061654937, 2024-04-29T14:11:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033452510999999997, 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.312952+0000", 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.312952+0000", 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.312953+0000", 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3300000000000001e-07 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.312978+0000", 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5666000000000001e-05 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.312985+0000", 2024-04-29T14:11:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4880000000000004e-06 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.312995+0000", 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0315e-05 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.316103+0000", 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003107963 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.316122+0000", 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9479000000000001e-05 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.346343+0000", 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030220795000000002 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.346404+0000", 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1072000000000001e-05 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:109773 2.e e123/107 2:77951913:::benchmark_data_smithi194_79000_object109772:head v 123'6851, mlcod=123'6848)", 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:11.320378+0000", 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 69.054228688999999, 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039145534000000003, 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.320378+0000", 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.320378+0000", 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.320381+0000", 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.88e-06 2024-04-29T14:11:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.320412+0000", 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0938000000000001e-05 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.320427+0000", 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5085999999999999e-05 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.320436+0000", 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5619999999999993e-06 2024-04-29T14:11:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.335066+0000", 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01462923 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.335085+0000", 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9218000000000001e-05 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.359459+0000", 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024374064000000001 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:11.359524+0000", 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4555999999999997e-05 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:119670 2.1 e125/107 2:89386983:::benchmark_data_smithi194_79000_object119669:head v 125'7476, mlcod=125'7473)", 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:17.812819+0000", 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.561788112000002, 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030347348999999999, 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.812819+0000", 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.812819+0000", 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.812821+0000", 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.407e-06 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.812851+0000", 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0233999999999999e-05 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.812871+0000", 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9261000000000001e-05 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.812881+0000", 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0017000000000001e-05 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.813013+0000", 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000132605 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.813037+0000", 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3623000000000001e-05 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.843115+0000", 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030078608999999999 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:17.843166+0000", 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0593000000000002e-05 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:138319 2.e e129/107 2:7575b2bf:::benchmark_data_smithi194_79000_object138318:head v 129'8640, mlcod=129'8635)", 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:30.650559+0000", 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.724047802999998, 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032846684000000001, 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.387 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.650559+0000", 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.650559+0000", 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.650560+0000", 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.265e-06 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.650584+0000", 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4298e-05 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.650596+0000", 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1761e-05 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.650604+0000", 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1419999999999993e-06 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.655656+0000", 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519270000000003 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.655681+0000", 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5041999999999998e-05 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.683374+0000", 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027692517 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:30.683406+0000", 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1732e-05 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:148254 2.e e132/107 2:7d2aa2e7:::benchmark_data_smithi194_79000_object148253:head v 132'9290, mlcod=132'9287)", 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.920924+0000", 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 42.453682567000001, 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030930451000000001, 2024-04-29T14:11:20.391 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.920924+0000", 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.920924+0000", 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.920926+0000", 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9649999999999998e-06 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.920952+0000", 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5791000000000001e-05 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.920966+0000", 2024-04-29T14:11:20.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.920975+0000", 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9670000000000007e-06 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.921050+0000", 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4612999999999999e-05 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.921072+0000", 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2495000000000001e-05 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.951802+0000", 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030730285 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.951855+0000", 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2151999999999998e-05 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:154030 2.e e134/107 2:7ae7faf2:::benchmark_data_smithi194_79000_object154029:head v 134'9648, mlcod=134'9645)", 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:43.212440+0000", 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 37.162166251000002, 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045196266999999998, 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.212440+0000", 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.212440+0000", 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.212442+0000", 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7120000000000001e-06 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.212469+0000", 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-05 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.212480+0000", 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1528999999999999e-05 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.212488+0000", 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4889999999999998e-06 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226680+0000", 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014192371 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226718+0000", 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8015000000000001e-05 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.257606+0000", 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030887781 2024-04-29T14:11:20.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.257637+0000", 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0700000000000001e-05 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:154031 2.e e134/107 2:76ff5956:::benchmark_data_smithi194_79000_object154030:head v 134'9649, mlcod=134'9645)", 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:43.226506+0000", 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 37.148100702999997, 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031158558999999999, 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.400 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226506+0000", 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226506+0000", 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226507+0000", 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4799999999999998e-07 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226549+0000", 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2661999999999999e-05 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226556+0000", 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0419999999999997e-06 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.226562+0000", 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7180000000000001e-06 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.233430+0000", 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0068678389999999997 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.233452+0000", 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1923e-05 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.257642+0000", 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024189925000000001 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:43.257665+0000", 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2901999999999999e-05 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.404 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:155349 2.e e134/107 2:7858a573:::benchmark_data_smithi194_79000_object155348:head v 134'9728, mlcod=134'9726)", 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:44.545813+0000", 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.828794121000001, 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033418069000000002, 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.545813+0000", 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.545813+0000", 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.545815+0000", 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6309999999999999e-06 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.545841+0000", 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5454000000000001e-05 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.545859+0000", 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8172000000000002e-05 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.545868+0000", 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0499999999999997e-06 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.550211+0000", 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043434600000000004 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.550225+0000", 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3891e-05 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.579194+0000", 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028969017999999999 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.579231+0000", 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6393000000000002e-05 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:155565 2.1 e134/107 2:83b0c1ac:::benchmark_data_smithi194_79000_object155564:head v 134'9728, mlcod=134'9725)", 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:44.737051+0000", 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.637555583999998, 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039024757, 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.737051+0000", 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.737051+0000", 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.737052+0000", 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0240000000000001e-06 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.737072+0000", 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9352e-05 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.737080+0000", 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4309999999999994e-06 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.737085+0000", 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8960000000000002e-06 2024-04-29T14:11:20.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743905+0000", 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0068197029999999999 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743927+0000", 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2306e-05 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.776031+0000", 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032104274000000002 2024-04-29T14:11:20.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.776076+0000", 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4771000000000002e-05 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:155568 2.1 e134/107 2:8158bcc6:::benchmark_data_smithi194_79000_object155567:head v 134'9729, mlcod=134'9726)", 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:44.743811+0000", 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.630795825999996, 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032285032999999998, 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.413 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743811+0000", 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743811+0000", 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743812+0000", 2024-04-29T14:11:20.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4000000000000001e-07 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743836+0000", 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4377999999999999e-05 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743842+0000", 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6869999999999999e-06 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.743845+0000", 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.58e-06 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.747563+0000", 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037175009999999998 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.747579+0000", 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5845000000000002e-05 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.776081+0000", 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028502438000000001 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.776096+0000", 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4963999999999999e-05 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:155576 2.1 e134/107 2:82a40ea7:::benchmark_data_smithi194_79000_object155575:head v 134'9730, mlcod=134'9726)", 2024-04-29T14:11:20.417 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:44.745389+0000", 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.629217676000003, 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030720391, 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.745389+0000", 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.745389+0000", 2024-04-29T14:11:20.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.745391+0000", 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0080000000000001e-06 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.745417+0000", 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5658999999999999e-05 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.745429+0000", 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2131e-05 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.745436+0000", 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4710000000000001e-06 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.752215+0000", 2024-04-29T14:11:20.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0067789069999999998 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.752239+0000", 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3476e-05 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.776097+0000", 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023858572000000002 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:44.776109+0000", 2024-04-29T14:11:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:156239 2.1 e134/107 2:8fab5b6e:::benchmark_data_smithi194_79000_object156238:head v 134'9770, mlcod=134'9767)", 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:45.323546+0000", 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.051060857000003, 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031139592000000001, 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323546+0000", 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323546+0000", 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323548+0000", 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3499999999999999e-06 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323585+0000", 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6922999999999999e-05 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323604+0000", 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9063e-05 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323615+0000", 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0380000000000001e-05 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323671+0000", 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6338000000000002e-05 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323693+0000", 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1759999999999998e-05 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.354644+0000", 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030951106999999999 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.354685+0000", 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1671000000000001e-05 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:156240 2.1 e134/107 2:84930638:::benchmark_data_smithi194_79000_object156239:head v 134'9771, mlcod=134'9767)", 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:45.323679+0000", 2024-04-29T14:11:20.426 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.050927266000002, 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031035018000000001, 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323679+0000", 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323679+0000", 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323680+0000", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2799999999999996e-07 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323704+0000", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3595999999999999e-05 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323709+0000", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8150000000000002e-06 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.323714+0000", 2024-04-29T14:11:20.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8629999999999999e-06 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.328529+0000", 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048148649999999998 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.328620+0000", 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1143000000000005e-05 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.354690+0000", 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026069610999999999 2024-04-29T14:11:20.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:45.354714+0000", 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4496999999999999e-05 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.431 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:166350 2.e e138/107 2:7c9cde67:::benchmark_data_smithi194_79000_object2089:head v 138'10445, mlcod=138'10438)", 2024-04-29T14:11:20.431 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:56.164114+0000", 2024-04-29T14:11:20.431 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 24.210492367000001, 2024-04-29T14:11:20.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032172776, 2024-04-29T14:11:20.431 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.164114+0000", 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.164114+0000", 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.164115+0000", 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1699999999999998e-07 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.164120+0000", 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.203e-06 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.164128+0000", 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7270000000000007e-06 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.164149+0000", 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1328999999999999e-05 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.179509+0000", 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015359378999999999 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.179528+0000", 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9656999999999998e-05 2024-04-29T14:11:20.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.196237+0000", 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016708712000000001 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:56.196287+0000", 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0151999999999997e-05 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:166969 2.a e141/107 2:5bd6ed06:::benchmark_data_smithi194_79000_object2708:head v 141'10522, mlcod=141'10519)", 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:58.748343+0000", 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.626263939000001, 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029402860999999999, 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.748343+0000", 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.748343+0000", 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.748344+0000", 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.158e-06 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.748355+0000", 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1548e-05 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.748365+0000", 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6139999999999998e-06 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.748372+0000", 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299999999999997e-06 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.749936+0000", 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0015640750000000001 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.749956+0000", 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9675999999999999e-05 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.777690+0000", 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027733969000000001 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.777746+0000", 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5890999999999997e-05 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:181967 2.a e145/107 2:5d948896:::benchmark_data_smithi194_79000_object17706:head v 145'11509, mlcod=145'11507)", 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:20.279086+0000", 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.095520338999999996, 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045260166999999997, 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279086+0000", 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279086+0000", 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279088+0000", 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.103e-06 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279099+0000", 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0849999999999999e-05 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279120+0000", 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0893000000000001e-05 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279134+0000", 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3611999999999999e-05 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279213+0000", 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9101e-05 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.279236+0000", 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2589e-05 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324312+0000", 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045076299 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324347+0000", 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.472e-05 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.443 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:181955 2.12 e145/140 2:4c3e28a7:::benchmark_data_smithi194_79000_object17694:head v 145'10677, mlcod=145'10675)", 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:20.280760+0000", 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.093846567000000006, 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043545669000000002, 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280760+0000", 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280760+0000", 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280762+0000", 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.759e-06 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280772+0000", 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6809999999999993e-06 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280790+0000", 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8507000000000001e-05 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280799+0000", 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3989999999999993e-06 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.281803+0000", 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010031860000000001 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.281829+0000", 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6035000000000001e-05 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324252+0000", 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.042423547999999998 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324306+0000", 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3554000000000003e-05 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:181958 2.12 e145/140 2:48d50730:::benchmark_data_smithi194_79000_object17697:head v 145'10678, mlcod=145'10675)", 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:20.280822+0000", 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.093785038000000001, 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043552991999999999, 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:20.448 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280822+0000", 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280822+0000", 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280822+0000", 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8899999999999998e-07 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280828+0000", 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6010000000000002e-06 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280834+0000", 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8799999999999996e-06 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280840+0000", 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.677e-06 2024-04-29T14:11:20.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.284063+0000", 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003223105 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.284083+0000", 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0707e-05 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324349+0000", 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040265227000000001 2024-04-29T14:11:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324375+0000", 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6006000000000001e-05 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "pg_backfill(progress 2.12 e 145/145 lb 2:4d4d1545:::benchmark_data_smithi194_79000_object111294:head)", 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:20.280846+0000", 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.093760261999999997, 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034474644999999998, 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280846+0000", 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280846+0000", 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280847+0000", 2024-04-29T14:11:20.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6000000000000003e-07 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280859+0000", 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1813e-05 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280866+0000", 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4900000000000003e-06 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280873+0000", 2024-04-29T14:11:20.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9120000000000001e-06 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.315275+0000", 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034401726000000001 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.315292+0000", 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6685e-05 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.315321+0000", 2024-04-29T14:11:20.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9359000000000001e-05 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(2.12 145/140 [PushOp(2:4d4d1ae3:::benchmark_data_smithi194_79000_object146412:head, version: 131'8956, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4d4d1ae3:::benchmark_data_smithi194_79000_object146412:head@131'8956, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:20.280882+0000", 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.093725063999999997, 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043427146, 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280882+0000", 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280882+0000", 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280882+0000", 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1199999999999998e-07 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280914+0000", 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2363999999999997e-05 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280929+0000", 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4647000000000001e-05 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.280945+0000", 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5699999999999999e-05 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324033+0000", 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043087827000000002 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324048+0000", 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5064e-05 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.324309+0000", 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026113200000000002 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.459 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.460 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:11:20.460 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:11:20.460 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:11:20.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:20.491 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:20.515 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:20.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:20.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:20.668 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:20.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:20.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:20.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:20.888 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:21.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:21 smithi040 ceph-mon[39581]: pgmap v444: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40636/299788 objects misplaced (13.555%); 66 MiB/s, 1.05k objects/s recovering 2024-04-29T14:11:21.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:21 smithi040 ceph-mon[39581]: 2.9 deep-scrub ok 2024-04-29T14:11:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:21 smithi194 ceph-mon[44089]: pgmap v444: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40636/299788 objects misplaced (13.555%); 66 MiB/s, 1.05k objects/s recovering 2024-04-29T14:11:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:21 smithi194 ceph-mon[44089]: 2.9 deep-scrub ok 2024-04-29T14:11:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:21 smithi121 ceph-mon[44533]: pgmap v444: 27 pgs: 2 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 17 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40636/299788 objects misplaced (13.555%); 66 MiB/s, 1.05k objects/s recovering 2024-04-29T14:11:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:21 smithi121 ceph-mon[44533]: 2.9 deep-scrub ok 2024-04-29T14:11:21.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:11:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:21.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:21.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:21.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:11:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:11:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:11:21.735 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:11:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:11:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:11:22.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1197021975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:22.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1197021975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:22.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1197021975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:22.829 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:22.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:22.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:11:22.981 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: pgmap v445: 27 pgs: 1 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 18 active+clean; 9.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37120/295178 objects misplaced (12.575%); 65 MiB/s, 1.04k objects/s recovering 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:11:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:23.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:11:23.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: pgmap v445: 27 pgs: 1 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 18 active+clean; 9.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37120/295178 objects misplaced (12.575%); 65 MiB/s, 1.04k objects/s recovering 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.332 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.332 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.332 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: pgmap v445: 27 pgs: 1 active+clean+scrubbing+deep, 8 active+remapped+backfilling, 18 active+clean; 9.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37120/295178 objects misplaced (12.575%); 65 MiB/s, 1.04k objects/s recovering 2024-04-29T14:11:23.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:23.335 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:23.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:23.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:23.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:23.666 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:23.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:23.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:23.857 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:11:23.857 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:11:23.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:23.964 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:24.051 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:24.052 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:24.052 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:24 smithi040 ceph-mon[39581]: osdmap e146: 12 total, 12 up, 12 in 2024-04-29T14:11:24.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:24.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:24.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:24.232 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:24.232 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:24.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:24 smithi121 ceph-mon[44533]: osdmap e146: 12 total, 12 up, 12 in 2024-04-29T14:11:24.234 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:24.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:24.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:24 smithi194 ceph-mon[44089]: osdmap e146: 12 total, 12 up, 12 in 2024-04-29T14:11:24.375 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:11:24.401 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:24.420 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:11:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:24.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:11:24.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:24.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:24.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:11:24.696 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:11:24.729 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:24.754 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:11:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:24.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:11:24.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:11:24.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:11:24.942 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:25.032 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:25.060 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:11:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:25.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:11:25.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:11:25.217 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:25.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:11:25.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:25 smithi121 ceph-mon[44533]: pgmap v447: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31288/292256 objects misplaced (10.706%); 72 MiB/s, 1.14k objects/s recovering 2024-04-29T14:11:25.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:25.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:25.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:25 smithi121 ceph-mon[44533]: osdmap e147: 12 total, 12 up, 12 in 2024-04-29T14:11:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:25.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:11:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:25 smithi040 ceph-mon[39581]: pgmap v447: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31288/292256 objects misplaced (10.706%); 72 MiB/s, 1.14k objects/s recovering 2024-04-29T14:11:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:25 smithi040 ceph-mon[39581]: osdmap e147: 12 total, 12 up, 12 in 2024-04-29T14:11:25.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:25.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:25 smithi194 ceph-mon[44089]: pgmap v447: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31288/292256 objects misplaced (10.706%); 72 MiB/s, 1.14k objects/s recovering 2024-04-29T14:11:25.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:25.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:25.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:25 smithi194 ceph-mon[44089]: osdmap e147: 12 total, 12 up, 12 in 2024-04-29T14:11:25.466 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:25.492 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:25.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:25.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:25.660 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:26.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:26.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:26.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:26 smithi040 ceph-mon[39581]: osdmap e148: 12 total, 12 up, 12 in 2024-04-29T14:11:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:26 smithi121 ceph-mon[44533]: osdmap e148: 12 total, 12 up, 12 in 2024-04-29T14:11:26.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:26.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:26 smithi194 ceph-mon[44089]: osdmap e148: 12 total, 12 up, 12 in 2024-04-29T14:11:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:11:27.065 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:11:27.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:27.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:27.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:27 smithi040 ceph-mon[39581]: pgmap v450: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29964/286002 objects misplaced (10.477%); 49 MiB/s, 775 objects/s recovering 2024-04-29T14:11:27.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:27 smithi040 ceph-mon[39581]: osdmap e149: 12 total, 12 up, 12 in 2024-04-29T14:11:27.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:27 smithi040 ceph-mon[39581]: osdmap e150: 12 total, 12 up, 12 in 2024-04-29T14:11:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:27 smithi121 ceph-mon[44533]: pgmap v450: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29964/286002 objects misplaced (10.477%); 49 MiB/s, 775 objects/s recovering 2024-04-29T14:11:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:27 smithi121 ceph-mon[44533]: osdmap e149: 12 total, 12 up, 12 in 2024-04-29T14:11:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:27 smithi121 ceph-mon[44533]: osdmap e150: 12 total, 12 up, 12 in 2024-04-29T14:11:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:27 smithi194 ceph-mon[44089]: pgmap v450: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29964/286002 objects misplaced (10.477%); 49 MiB/s, 775 objects/s recovering 2024-04-29T14:11:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:27 smithi194 ceph-mon[44089]: osdmap e149: 12 total, 12 up, 12 in 2024-04-29T14:11:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:27 smithi194 ceph-mon[44089]: osdmap e150: 12 total, 12 up, 12 in 2024-04-29T14:11:27.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:11:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:28.241 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:28.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:28.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:11:28.391 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:11:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:28.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:11:28.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:28.582 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:28.613 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:28.654 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:28.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:11:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:11:28.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:29.050 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:29.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:29.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:11:29.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:11:29.194 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:29.227 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:11:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:29.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:11:29.330 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:11:29.359 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:29.389 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:11:29.392 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:29 smithi194 ceph-mon[44089]: pgmap v453: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 22078/281146 objects misplaced (7.853%); 64 MiB/s, 1.02k objects/s recovering 2024-04-29T14:11:29.393 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/474594190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:11:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:29.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:11:29.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:29 smithi121 ceph-mon[44533]: pgmap v453: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 22078/281146 objects misplaced (7.853%); 64 MiB/s, 1.02k objects/s recovering 2024-04-29T14:11:29.412 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/474594190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:11:29.495 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:29.541 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:29 smithi040 ceph-mon[39581]: pgmap v453: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 22078/281146 objects misplaced (7.853%); 64 MiB/s, 1.02k objects/s recovering 2024-04-29T14:11:29.541 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/474594190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:11:29.549 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:29.639 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:29.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:29.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:11:29.692 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:11:29.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-04-29T14:11:29.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:29.820 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:30.021 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:30.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:11:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:11:30.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:30.316 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:30.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:30.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:30 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:30.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/474594190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:11:30.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:30 smithi121 ceph-mon[44533]: osdmap e151: 12 total, 12 up, 12 in 2024-04-29T14:11:30.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:30 smithi121 ceph-mon[44533]: pgmap v455: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 22078/280480 objects misplaced (7.872%); 58 MiB/s, 932 objects/s recovering 2024-04-29T14:11:30.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:11:30.464 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:30.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:30 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:30.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/474594190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:11:30.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:30 smithi040 ceph-mon[39581]: osdmap e151: 12 total, 12 up, 12 in 2024-04-29T14:11:30.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:30 smithi040 ceph-mon[39581]: pgmap v455: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 22078/280480 objects misplaced (7.872%); 58 MiB/s, 932 objects/s recovering 2024-04-29T14:11:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:30 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/474594190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:11:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:30 smithi194 ceph-mon[44089]: osdmap e151: 12 total, 12 up, 12 in 2024-04-29T14:11:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:30 smithi194 ceph-mon[44089]: pgmap v455: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 22078/280480 objects misplaced (7.872%); 58 MiB/s, 932 objects/s recovering 2024-04-29T14:11:30.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:11:30.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:30.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:11:30.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:11:30.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:30.912 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:30.941 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:30.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:31.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:11:31.090 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:31.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:11:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:31.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:11:31.200 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:31.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:31.278 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:11:31.382 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:31.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:31.446 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:31.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:31.546 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:11:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:31.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:31.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:11:31.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:31.708 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:32.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:32.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:32.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:32.253 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:32.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:32.363 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:11:32.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: osdmap e152: 12 total, 12 up, 12 in 2024-04-29T14:11:32.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:11:32.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:11:32.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:11:32.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:11:32.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:32.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:32.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:32 smithi040 ceph-mon[39581]: osdmap e153: 12 total, 12 up, 12 in 2024-04-29T14:11:32.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: osdmap e152: 12 total, 12 up, 12 in 2024-04-29T14:11:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:11:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:11:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:11:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:11:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:32 smithi121 ceph-mon[44533]: osdmap e153: 12 total, 12 up, 12 in 2024-04-29T14:11:32.394 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:32.421 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:11:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:32.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:11:32.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:11:32.479 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:11:32.512 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:11:32.525 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:32.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: osdmap e152: 12 total, 12 up, 12 in 2024-04-29T14:11:32.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:11:32.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:11:32.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:11:32.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:11:32.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:32.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:32.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:32 smithi194 ceph-mon[44089]: osdmap e153: 12 total, 12 up, 12 in 2024-04-29T14:11:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:32.552 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:11:32.572 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:32.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:32.594 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:11:32.630 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:11:32.661 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:11:32.675 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:11:32.694 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:11:32.702 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:32.721 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:11:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:32.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:11:32.726 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:11:32.760 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:11:32.794 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:11:32.824 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:11:32.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:11:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:11:32.859 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:32.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:11:32.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:32.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:32.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:11:33.021 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:33.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:11:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:33.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:11:33.156 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:11:33.156 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:11:33.156 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:11:33.156 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:11:33.212 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:11:33.243 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:33.269 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:11:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:33.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:11:33.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:11:33.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2764917162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:33.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:33.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:33 smithi121 ceph-mon[44533]: osdmap e154: 12 total, 12 up, 12 in 2024-04-29T14:11:33.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2764917162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:33 smithi040 ceph-mon[39581]: osdmap e154: 12 total, 12 up, 12 in 2024-04-29T14:11:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2764917162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:33.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:33 smithi194 ceph-mon[44089]: osdmap e154: 12 total, 12 up, 12 in 2024-04-29T14:11:33.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:11:34.179 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: pgmap v460: 27 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 19 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 15840/270900 objects misplaced (5.847%); 41 MiB/s, 657 objects/s recovering 2024-04-29T14:11:34.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:11:34.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:11:34.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:11:34.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:34.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:34 smithi040 ceph-mon[39581]: osdmap e155: 12 total, 12 up, 12 in 2024-04-29T14:11:34.286 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:11:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:34.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:11:34.391 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:34.419 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:34.441 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:34.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: pgmap v460: 27 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 19 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 15840/270900 objects misplaced (5.847%); 41 MiB/s, 657 objects/s recovering 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:34.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:34 smithi121 ceph-mon[44533]: osdmap e155: 12 total, 12 up, 12 in 2024-04-29T14:11:34.479 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:34.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: pgmap v460: 27 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 19 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 15840/270900 objects misplaced (5.847%); 41 MiB/s, 657 objects/s recovering 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:11:34.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:34 smithi194 ceph-mon[44089]: osdmap e155: 12 total, 12 up, 12 in 2024-04-29T14:11:34.578 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:34.591 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:11:34.591 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:11:34.592 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:11:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:11:35.603 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:11:35.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-04-29T14:11:35.634 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:35.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:35.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:11:35.776 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:35.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:11:35.985 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:11:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:35.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:11:36.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:36.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:35 smithi121 ceph-mon[44533]: osdmap e156: 12 total, 12 up, 12 in 2024-04-29T14:11:36.150 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:36.152 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:11:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:11:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:11:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:11:36.153 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:11:36.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:35 smithi194 ceph-mon[44089]: osdmap e156: 12 total, 12 up, 12 in 2024-04-29T14:11:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:35 smithi040 ceph-mon[39581]: osdmap e156: 12 total, 12 up, 12 in 2024-04-29T14:11:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:36.521 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:36.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:36.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:11:36.667 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:36.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:11:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:36.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:11:36.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:11:36.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:37.038 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:37.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:37.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:36 smithi040 ceph-mon[39581]: pgmap v463: 27 pgs: 4 active+remapped+backfilling, 23 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 10916/265734 objects misplaced (4.108%); 60 MiB/s, 953 objects/s recovering 2024-04-29T14:11:37.072 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:36 smithi040 ceph-mon[39581]: osdmap e157: 12 total, 12 up, 12 in 2024-04-29T14:11:37.072 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:37.072 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:37.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:37.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:36 smithi121 ceph-mon[44533]: pgmap v463: 27 pgs: 4 active+remapped+backfilling, 23 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 10916/265734 objects misplaced (4.108%); 60 MiB/s, 953 objects/s recovering 2024-04-29T14:11:37.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:36 smithi121 ceph-mon[44533]: osdmap e157: 12 total, 12 up, 12 in 2024-04-29T14:11:37.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:37.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:37.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:37.228 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:36 smithi194 ceph-mon[44089]: pgmap v463: 27 pgs: 4 active+remapped+backfilling, 23 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 10916/265734 objects misplaced (4.108%); 60 MiB/s, 953 objects/s recovering 2024-04-29T14:11:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:36 smithi194 ceph-mon[44089]: osdmap e157: 12 total, 12 up, 12 in 2024-04-29T14:11:37.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:37.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:37.318 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:37.670 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:37.958 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:11:38.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:37 smithi040 ceph-mon[39581]: osdmap e158: 12 total, 12 up, 12 in 2024-04-29T14:11:38.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/997096668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:11:38.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:37 smithi121 ceph-mon[44533]: osdmap e158: 12 total, 12 up, 12 in 2024-04-29T14:11:38.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/997096668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:11:38.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:37 smithi194 ceph-mon[44089]: osdmap e158: 12 total, 12 up, 12 in 2024-04-29T14:11:38.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/997096668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:11:39.000 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:39.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:39.105 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:38 smithi040 ceph-mon[39581]: pgmap v466: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 6948/263734 objects misplaced (2.634%); 68 MiB/s, 1.09k objects/s recovering 2024-04-29T14:11:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:38 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/997096668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:11:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:38 smithi040 ceph-mon[39581]: osdmap e159: 12 total, 12 up, 12 in 2024-04-29T14:11:39.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:38 smithi040 ceph-mon[39581]: 2.13 scrub starts 2024-04-29T14:11:39.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:38 smithi121 ceph-mon[44533]: pgmap v466: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 6948/263734 objects misplaced (2.634%); 68 MiB/s, 1.09k objects/s recovering 2024-04-29T14:11:39.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:38 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/997096668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:11:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:38 smithi121 ceph-mon[44533]: osdmap e159: 12 total, 12 up, 12 in 2024-04-29T14:11:39.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:38 smithi121 ceph-mon[44533]: 2.13 scrub starts 2024-04-29T14:11:39.140 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:39.169 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:39.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:39.256 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:11:39.256 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:11:39.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:11:39.256 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14802 2.2 2:4cff85a0:::benchmark_data_smithi194_79000_object14801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.146831+0000", 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.112995883, 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20478190399999999, 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14802 2024-04-29T14:11:39.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146831+0000", 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146831+0000", 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146832+0000", 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5900000000000005e-07 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146990+0000", 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015835799999999999 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147000+0000", 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6080000000000002e-06 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147007+0000", 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8190000000000003e-06 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147051+0000", 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4249999999999998e-05 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147135+0000", 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3642999999999999e-05 2024-04-29T14:11:39.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147225+0000", 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9920000000000006e-05 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.148359+0000", 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001134501 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.351534+0000", 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20317476800000001 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.351570+0000", 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000002e-05 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.351613+0000", 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3328000000000003e-05 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14803 2.2 2:455db739:::benchmark_data_smithi194_79000_object14802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.147539+0000", 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.11228822800001, 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20464069300000001, 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.263 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14803 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147539+0000", 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147539+0000", 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147540+0000", 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1739999999999999e-06 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147597+0000", 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6879000000000002e-05 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147607+0000", 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7210000000000006e-06 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147612+0000", 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6999999999999996e-06 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147729+0000", 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117219 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147839+0000", 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010945100000000001 2024-04-29T14:11:39.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147973+0000", 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013422600000000001 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.148522+0000", 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054840499999999996 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352103+0000", 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20358119599999999 2024-04-29T14:11:39.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352137+0000", 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4239999999999997e-05 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352179+0000", 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2481999999999997e-05 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.268 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14822 2.2 2:41dfc9f4:::benchmark_data_smithi194_79000_object14821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.151076+0000", 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.108751318, 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201209203, 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14822 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151076+0000", 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151076+0000", 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151077+0000", 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.314e-06 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151160+0000", 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3131000000000001e-05 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151170+0000", 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5449999999999992e-06 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151175+0000", 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6300000000000003e-06 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151220+0000", 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4917999999999999e-05 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151345+0000", 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012466500000000001 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151486+0000", 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014131899999999999 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.152019+0000", 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00053267199999999998 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352224+0000", 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20020465600000001 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352259+0000", 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5544999999999999e-05 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352285+0000", 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5808000000000001e-05 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14832 2.2 2:450d2096:::benchmark_data_smithi194_79000_object14831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.153431+0000", 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.10639601700001, 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199426829, 2024-04-29T14:11:39.274 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14832 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153431+0000", 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153431+0000", 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153438+0000", 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.691e-06 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153504+0000", 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6223000000000006e-05 2024-04-29T14:11:39.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153513+0000", 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4590000000000004e-06 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153519+0000", 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8109999999999999e-06 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153570+0000", 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1270999999999998e-05 2024-04-29T14:11:39.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153678+0000", 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010785600000000001 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153791+0000", 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112497 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.154367+0000", 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057592799999999997 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352798+0000", 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.198431259 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352832+0000", 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4286999999999999e-05 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352858+0000", 2024-04-29T14:11:39.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5547000000000001e-05 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14843 2.2 2:4e70e968:::benchmark_data_smithi194_79000_object14842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.155295+0000", 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.10453237300001, 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.197668766, 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14843 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155295+0000", 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155295+0000", 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155296+0000", 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.074e-06 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155373+0000", 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6859000000000004e-05 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155383+0000", 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0020000000000001e-05 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155391+0000", 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0330000000000008e-06 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155428+0000", 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7508000000000001e-05 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155504+0000", 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6210000000000004e-05 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155600+0000", 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5660000000000002e-05 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.156065+0000", 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046479099999999999 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352893+0000", 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.196827797 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352926+0000", 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3729999999999997e-05 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352963+0000", 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7083999999999999e-05 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14879 2.2 2:4435437c:::benchmark_data_smithi194_79000_object14878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.285 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.163548+0000", 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.09627923599999, 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.190350727, 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14879 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163548+0000", 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163548+0000", 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163555+0000", 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8990000000000004e-06 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163649+0000", 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4238000000000001e-05 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163662+0000", 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3091000000000001e-05 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163670+0000", 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6969999999999993e-06 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163712+0000", 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2293000000000003e-05 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163811+0000", 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8529000000000005e-05 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.163897+0000", 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6242000000000005e-05 2024-04-29T14:11:39.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.164379+0000", 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048265100000000001 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.353835+0000", 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.189455189 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.353864+0000", 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9791e-05 2024-04-29T14:11:39.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.353899+0000", 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4106999999999997e-05 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14938 2.2 2:4f85fd28:::benchmark_data_smithi194_79000_object14937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.179388+0000", 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.08043911600001, 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17461587100000001, 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14938 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179388+0000", 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179388+0000", 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179389+0000", 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3090000000000001e-06 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179446+0000", 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6329000000000002e-05 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179455+0000", 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5400000000000001e-06 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179463+0000", 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7819999999999994e-06 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179505+0000", 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2205e-05 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179584+0000", 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8640999999999995e-05 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.179669+0000", 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5406000000000002e-05 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.180182+0000", 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051332799999999996 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.353942+0000", 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17375989 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.353978+0000", 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5777e-05 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354004+0000", 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5664e-05 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14952 2.2 2:4e9655c0:::benchmark_data_smithi194_79000_object14951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.183414+0000", 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.076413508, 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17121537100000001, 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14952 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.297 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183414+0000", 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183414+0000", 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183417+0000", 2024-04-29T14:11:39.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9009999999999996e-06 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183535+0000", 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118022 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183543+0000", 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6140000000000003e-06 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183548+0000", 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2089999999999996e-06 2024-04-29T14:11:39.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183583+0000", 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4419999999999999e-05 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183659+0000", 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6174000000000006e-05 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.183742+0000", 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3344999999999996e-05 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184225+0000", 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048284599999999999 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354549+0000", 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17032349499999999 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354605+0000", 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6480000000000001e-05 2024-04-29T14:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354629+0000", 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3864999999999999e-05 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14957 2.2 2:4db01287:::benchmark_data_smithi194_79000_object14956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.184278+0000", 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.07554904, 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17045490099999999, 2024-04-29T14:11:39.302 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14957 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184278+0000", 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184278+0000", 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184279+0000", 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-06 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184361+0000", 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2354000000000005e-05 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184371+0000", 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1239999999999994e-06 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184376+0000", 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7810000000000002e-06 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184418+0000", 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1576999999999999e-05 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184494+0000", 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6495999999999994e-05 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.184616+0000", 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122034 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.185112+0000", 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049564100000000001 2024-04-29T14:11:39.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354672+0000", 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16955969500000001 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354705+0000", 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3303000000000003e-05 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354733+0000", 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7841e-05 2024-04-29T14:11:39.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14970 2.2 2:49cb47e5:::benchmark_data_smithi194_79000_object14969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.187744+0000", 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.072083499, 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.167099792, 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.308 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14970 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.187744+0000", 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.187744+0000", 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.187745+0000", 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.553e-06 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.187871+0000", 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125685 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.187882+0000", 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1469e-05 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.187890+0000", 2024-04-29T14:11:39.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5669999999999997e-06 2024-04-29T14:11:39.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.187943+0000", 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3353999999999998e-05 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.188043+0000", 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9518000000000005e-05 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.188125+0000", 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2106000000000003e-05 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.188580+0000", 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045511999999999999 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354761+0000", 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16618127799999999 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354810+0000", 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8340999999999997e-05 2024-04-29T14:11:39.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354843+0000", 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3800999999999997e-05 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14979 2.2 2:4e613006:::benchmark_data_smithi194_79000_object14978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.190867+0000", 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.068960487, 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16417366999999999, 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.316 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14979 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.190867+0000", 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.190867+0000", 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.190868+0000", 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.257e-06 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.190923+0000", 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5374000000000002e-05 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.190933+0000", 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7459999999999995e-06 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.190939+0000", 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8390000000000003e-06 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.190982+0000", 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2834999999999999e-05 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.191102+0000", 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120147 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.191195+0000", 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3270000000000007e-05 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.191663+0000", 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046791999999999998 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.354990+0000", 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16332712199999999 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355018+0000", 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8308000000000001e-05 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355040+0000", 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1852e-05 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.321 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14992 2.2 2:4f10cfb9:::benchmark_data_smithi194_79000_object14991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.194431+0000", 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.06539584399999, 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.160741301, 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.322 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14992 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194431+0000", 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194431+0000", 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194433+0000", 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.423e-06 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194481+0000", 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8899000000000001e-05 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194491+0000", 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4830000000000005e-06 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194498+0000", 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2790000000000001e-06 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194558+0000", 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9896999999999999e-05 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194674+0000", 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011633600000000001 2024-04-29T14:11:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.194815+0000", 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064000000000001 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.195301+0000", 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048611900000000003 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355084+0000", 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15978261399999999 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355135+0000", 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1004999999999997e-05 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355172+0000", 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7605999999999999e-05 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:15038 2.2 2:47d3d782:::benchmark_data_smithi194_79000_object15037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.213355+0000", 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.04647213699999, 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14260956899999999, 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 15038 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213355+0000", 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213355+0000", 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213358+0000", 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8559999999999998e-06 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213404+0000", 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5939999999999997e-05 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213435+0000", 2024-04-29T14:11:39.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1458000000000003e-05 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213458+0000", 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2628000000000001e-05 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213498+0000", 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9833999999999998e-05 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213573+0000", 2024-04-29T14:11:39.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5640000000000001e-05 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.213658+0000", 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5106999999999997e-05 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.214161+0000", 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050285399999999995 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355870+0000", 2024-04-29T14:11:39.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.141708743 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355924+0000", 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3668999999999997e-05 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.355964+0000", 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0840000000000002e-05 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:15047 2.2 2:47bc9d89:::benchmark_data_smithi194_79000_object15046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.217729+0000", 2024-04-29T14:11:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.042098065, 2024-04-29T14:11:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13834044400000001, 2024-04-29T14:11:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 15047 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217729+0000", 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217729+0000", 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217730+0000", 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3549999999999999e-06 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217783+0000", 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2386000000000003e-05 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217793+0000", 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0196e-05 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217801+0000", 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217844+0000", 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3223999999999998e-05 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.217920+0000", 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6477000000000007e-05 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.218003+0000", 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2478999999999994e-05 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.218446+0000", 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00044323499999999999 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356004+0000", 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13755821300000001 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356037+0000", 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2879000000000001e-05 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356069+0000", 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2289000000000002e-05 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:15065 2.2 2:4e92918e:::benchmark_data_smithi194_79000_object15064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.230809+0000", 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.029017974, 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12539982299999999, 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 15065 2024-04-29T14:11:39.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.230809+0000", 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.230809+0000", 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.230810+0000", 2024-04-29T14:11:39.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2500000000000001e-06 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.230910+0000", 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100147 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.230920+0000", 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9799999999999993e-06 2024-04-29T14:11:39.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.230928+0000", 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6799999999999993e-06 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.230968+0000", 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9694000000000003e-05 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.231045+0000", 2024-04-29T14:11:39.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7674000000000002e-05 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.231138+0000", 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2225999999999994e-05 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.231637+0000", 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049906400000000002 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356114+0000", 2024-04-29T14:11:39.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12447675699999999 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356163+0000", 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8984999999999999e-05 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356209+0000", 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6366000000000003e-05 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:15093 2.2 2:4f20f438:::benchmark_data_smithi194_79000_object15092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.268149+0000", 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 133.99167782399999, 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.08821938, 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 15093 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268149+0000", 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268149+0000", 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268150+0000", 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5799999999999998e-07 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268206+0000", 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5822000000000002e-05 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268216+0000", 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0097e-05 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268223+0000", 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5799999999999997e-06 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268257+0000", 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4496000000000003e-05 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268335+0000", 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8133999999999994e-05 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268418+0000", 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2731000000000005e-05 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.268913+0000", 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049480100000000001 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356272+0000", 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.087359589000000001 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356330+0000", 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7522000000000003e-05 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.356369+0000", 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8649999999999998e-05 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:166924 2.3 2:c7723801:::benchmark_data_smithi194_79000_object2663:head [delete] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-29T14:11:39.351 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:10:58.639422+0000", 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.620404831000002, 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.086855853999999996, 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 166924 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639422+0000", 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639422+0000", 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639424+0000", 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9290000000000001e-06 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639430+0000", 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.976e-06 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639442+0000", 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1732999999999999e-05 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639450+0000", 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9859999999999994e-06 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639509+0000", 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8869999999999997e-05 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639609+0000", 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010058800000000001 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.639971+0000", 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036174200000000002 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.644477+0000", 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045058490000000001 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.726177+0000", 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.081700637000000007 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.726232+0000", 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4679000000000003e-05 2024-04-29T14:11:39.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.726278+0000", 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5865000000000002e-05 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:166927 2.3 2:c66c9899:::benchmark_data_smithi194_79000_object2666:head [delete] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:10:58.640396+0000", 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.619431040999999, 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.086015964, 2024-04-29T14:11:39.357 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 166927 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.640396+0000", 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.640396+0000", 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.640398+0000", 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6330000000000001e-06 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.640411+0000", 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3423e-05 2024-04-29T14:11:39.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.640421+0000", 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8919999999999995e-06 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.640429+0000", 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7300000000000005e-06 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.641859+0000", 2024-04-29T14:11:39.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014305640000000001 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.641963+0000", 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010374600000000001 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.642338+0000", 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037506699999999998 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.644573+0000", 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022349620000000001 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.726320+0000", 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.081747185999999999 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.726367+0000", 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6851000000000003e-05 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:58.726412+0000", 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4910000000000002e-05 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:176714 2.2 2:40492d7b:::benchmark_data_smithi194_79000_object12453:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:12.895916+0000", 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.363911219999999, 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10310322399999999, 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 176714 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.895916+0000", 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.895916+0000", 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.895918+0000", 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9439999999999999e-06 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.895924+0000", 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7000000000000002e-06 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.895937+0000", 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2320000000000001e-05 2024-04-29T14:11:39.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.895944+0000", 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8900000000000001e-06 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.895990+0000", 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6297000000000001e-05 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.896083+0000", 2024-04-29T14:11:39.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2848000000000003e-05 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.896440+0000", 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035674799999999999 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.898194+0000", 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017546790000000001 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.998934+0000", 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100739737 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.998971+0000", 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7021999999999999e-05 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:12.999019+0000", 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8038999999999999e-05 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:178579 2.2 2:451a6094:::benchmark_data_smithi194_79000_object14318:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:16.159067+0000", 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.100760246, 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101588992, 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178579 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159067+0000", 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159067+0000", 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159069+0000", 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5749999999999999e-06 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159077+0000", 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.763e-06 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159094+0000", 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6997999999999999e-05 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159104+0000", 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7189999999999996e-06 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159165+0000", 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1211999999999996e-05 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159265+0000", 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9778000000000006e-05 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.159686+0000", 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042162200000000002 2024-04-29T14:11:39.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.161976+0000", 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022892839999999999 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.260543+0000", 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098567117999999995 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.260605+0000", 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2212000000000007e-05 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:39.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:16.260656+0000", 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0711000000000003e-05 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:39.374 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:11:39.374 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:39.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:38 smithi194 ceph-mon[44089]: pgmap v466: 27 pgs: 3 active+remapped+backfilling, 24 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 6948/263734 objects misplaced (2.634%); 68 MiB/s, 1.09k objects/s recovering 2024-04-29T14:11:39.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:38 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:11:39.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/997096668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:11:39.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:38 smithi194 ceph-mon[44089]: osdmap e159: 12 total, 12 up, 12 in 2024-04-29T14:11:39.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:38 smithi194 ceph-mon[44089]: 2.13 scrub starts 2024-04-29T14:11:39.414 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:39.442 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:39.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:39.546 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:39.590 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:39.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:11:40.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:39 smithi194 ceph-mon[44089]: 2.13 scrub ok 2024-04-29T14:11:40.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:39 smithi194 ceph-mon[44089]: 2.16 scrub starts 2024-04-29T14:11:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:11:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:11:40.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:39 smithi040 ceph-mon[39581]: 2.13 scrub ok 2024-04-29T14:11:40.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:39 smithi040 ceph-mon[39581]: 2.16 scrub starts 2024-04-29T14:11:40.365 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:40.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:40.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:39 smithi121 ceph-mon[44533]: 2.13 scrub ok 2024-04-29T14:11:40.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:39 smithi121 ceph-mon[44533]: 2.16 scrub starts 2024-04-29T14:11:40.425 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:40.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:11:40.506 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:40.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:11:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:40.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:11:40.975 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:41.060 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:40 smithi121 ceph-mon[44533]: pgmap v468: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 5902/262736 objects misplaced (2.246%); 63 MiB/s, 1.01k objects/s recovering 2024-04-29T14:11:41.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:40 smithi121 ceph-mon[44533]: 2.16 scrub ok 2024-04-29T14:11:41.089 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:41.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:40 smithi040 ceph-mon[39581]: pgmap v468: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 5902/262736 objects misplaced (2.246%); 63 MiB/s, 1.01k objects/s recovering 2024-04-29T14:11:41.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:40 smithi040 ceph-mon[39581]: 2.16 scrub ok 2024-04-29T14:11:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:41.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:41.203 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:11:41.247 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:40 smithi194 ceph-mon[44089]: pgmap v468: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 5902/262736 objects misplaced (2.246%); 63 MiB/s, 1.01k objects/s recovering 2024-04-29T14:11:41.247 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:40 smithi194 ceph-mon[44089]: 2.16 scrub ok 2024-04-29T14:11:41.252 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:41.302 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:11:41.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:41.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:11:41.410 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:41.451 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:11:41.772 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:11:41.772 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-04-29T14:11:41.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:11:42.010 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:42.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:42.122 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:11:42.162 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:42.188 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:11:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:42.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:11:42.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:42 smithi121 ceph-mon[44533]: 2.12 scrub starts 2024-04-29T14:11:42.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:11:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:41 smithi194 ceph-mon[44089]: 2.12 scrub starts 2024-04-29T14:11:42.332 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:42.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:41 smithi040 ceph-mon[39581]: 2.12 scrub starts 2024-04-29T14:11:42.419 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:11:42.455 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:11:42.484 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:11:42.516 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:11:42.533 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:11:42.533 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:206124 2.17 2:e837d039:::benchmark_data_smithi194_79000_object41863:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:42.526980+0000", 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.011086575, 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011115053999999999, 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:42.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 206124 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.526980+0000", 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.526980+0000", 2024-04-29T14:11:42.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.526982+0000", 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.646e-06 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.526987+0000", 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000001e-06 2024-04-29T14:11:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.526996+0000", 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4029999999999996e-06 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.527004+0000", 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2649999999999999e-06 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.527050+0000", 2024-04-29T14:11:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6616999999999998e-05 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.527132+0000", 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1670999999999998e-05 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.527495+0000", 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000363486 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530309+0000", 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028132000000000001 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:206127 2.7 2:e2dc581b:::benchmark_data_smithi194_79000_object41866:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-29T14:11:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:42.530894+0000", 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0071726769999999997, 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072917950000000002, 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 206127 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530894+0000", 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530894+0000", 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530896+0000", 2024-04-29T14:11:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7069999999999999e-06 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530900+0000", 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8960000000000002e-06 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530910+0000", 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3859999999999996e-06 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530918+0000", 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1389999999999995e-06 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530969+0000", 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0618999999999998e-05 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.531093+0000", 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012491499999999999 2024-04-29T14:11:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.531575+0000", 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048170999999999999 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.535056+0000", 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034811529999999999 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:206129 2.17 2:edc82038:::benchmark_data_smithi194_79000_object41868:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:42.530933+0000", 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.007133571, 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073358329999999999, 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 206129 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530933+0000", 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530933+0000", 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530934+0000", 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3799999999999997e-07 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530937+0000", 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.993e-06 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530942+0000", 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4419999999999997e-06 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530948+0000", 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7869999999999998e-06 2024-04-29T14:11:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.530984+0000", 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6106000000000003e-05 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.531094+0000", 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109923 2024-04-29T14:11:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.531562+0000", 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046817099999999997 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.535201+0000", 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036385990000000002 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:206136 2.7 2:e6630e3c:::benchmark_data_smithi194_79000_object41875:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:42.537092+0000", 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00097424899999999995, 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012583239999999999, 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:11:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:11:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:11:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:11:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:11:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 206136 2024-04-29T14:11:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:11:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537092+0000", 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537092+0000", 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537094+0000", 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.978e-06 2024-04-29T14:11:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537099+0000", 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3100000000000002e-06 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537110+0000", 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0984e-05 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537115+0000", 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.096e-06 2024-04-29T14:11:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537152+0000", 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6891999999999998e-05 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:11:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537234+0000", 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1996999999999994e-05 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:42.537620+0000", 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038663199999999998 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 4 2024-04-29T14:11:42.555 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:11:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:11:42.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:11:42.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:11:42.591 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:11:42.621 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:11:42.653 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:11:42.657 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:42.696 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:11:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:42.727 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:11:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:42.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:42.770 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:11:42.810 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:11:42.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:11:42.850 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:11:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:11:42.856 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:11:42.889 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:42.914 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:11:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:42.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:11:42.996 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:43 smithi040 ceph-mon[39581]: pgmap v469: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 4134/254296 objects misplaced (1.626%) 2024-04-29T14:11:42.997 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:43 smithi040 ceph-mon[39581]: 2.12 scrub ok 2024-04-29T14:11:42.997 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4095450970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:43.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:43.054 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:43.191 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:11:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:43.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:11:43.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:43 smithi121 ceph-mon[44533]: pgmap v469: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 4134/254296 objects misplaced (1.626%) 2024-04-29T14:11:43.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:43 smithi121 ceph-mon[44533]: 2.12 scrub ok 2024-04-29T14:11:43.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4095450970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:43.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.205 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:11:43.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:43.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:43 smithi194 ceph-mon[44089]: pgmap v469: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 7.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 4134/254296 objects misplaced (1.626%) 2024-04-29T14:11:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:43 smithi194 ceph-mon[44089]: 2.12 scrub ok 2024-04-29T14:11:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4095450970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:11:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:43.337 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:43.619 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:44 smithi040 ceph-mon[39581]: 2.18 scrub starts 2024-04-29T14:11:44.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:44 smithi121 ceph-mon[44533]: 2.18 scrub starts 2024-04-29T14:11:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:11:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:44 smithi194 ceph-mon[44089]: 2.18 scrub starts 2024-04-29T14:11:45.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:45 smithi121 ceph-mon[44533]: pgmap v470: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 7.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2338/250276 objects misplaced (0.934%) 2024-04-29T14:11:45.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:45 smithi121 ceph-mon[44533]: 2.18 scrub ok 2024-04-29T14:11:45.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:45 smithi121 ceph-mon[44533]: 2.10 scrub starts 2024-04-29T14:11:45.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:45 smithi121 ceph-mon[44533]: osdmap e160: 12 total, 12 up, 12 in 2024-04-29T14:11:45.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:11:45.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:45 smithi194 ceph-mon[44089]: pgmap v470: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 7.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2338/250276 objects misplaced (0.934%) 2024-04-29T14:11:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:45 smithi194 ceph-mon[44089]: 2.18 scrub ok 2024-04-29T14:11:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:45 smithi194 ceph-mon[44089]: 2.10 scrub starts 2024-04-29T14:11:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:45 smithi194 ceph-mon[44089]: osdmap e160: 12 total, 12 up, 12 in 2024-04-29T14:11:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:11:45.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:45 smithi040 ceph-mon[39581]: pgmap v470: 27 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 24 active+clean; 7.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2338/250276 objects misplaced (0.934%) 2024-04-29T14:11:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:45 smithi040 ceph-mon[39581]: 2.18 scrub ok 2024-04-29T14:11:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:45 smithi040 ceph-mon[39581]: 2.10 scrub starts 2024-04-29T14:11:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:45 smithi040 ceph-mon[39581]: osdmap e160: 12 total, 12 up, 12 in 2024-04-29T14:11:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:11:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:45.411 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:11:45.411 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:11:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:46 smithi194 ceph-mon[44089]: osdmap e161: 12 total, 12 up, 12 in 2024-04-29T14:11:46.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:46 smithi040 ceph-mon[39581]: osdmap e161: 12 total, 12 up, 12 in 2024-04-29T14:11:46.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:46 smithi121 ceph-mon[44533]: osdmap e161: 12 total, 12 up, 12 in 2024-04-29T14:11:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: 2.10 scrub starts 2024-04-29T14:11:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: pgmap v473: 27 pgs: 1 peering, 1 active+remapped+backfilling, 25 active+clean; 7.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1376/241888 objects misplaced (0.569%) 2024-04-29T14:11:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: 2.11 scrub starts 2024-04-29T14:11:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: 2.10 scrub ok 2024-04-29T14:11:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: osdmap e162: 12 total, 12 up, 12 in 2024-04-29T14:11:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: 2.11 scrub ok 2024-04-29T14:11:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: 2.19 scrub starts 2024-04-29T14:11:47.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:47 smithi194 ceph-mon[44089]: osdmap e163: 12 total, 12 up, 12 in 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: 2.10 scrub starts 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: pgmap v473: 27 pgs: 1 peering, 1 active+remapped+backfilling, 25 active+clean; 7.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1376/241888 objects misplaced (0.569%) 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: 2.11 scrub starts 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: 2.10 scrub ok 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: osdmap e162: 12 total, 12 up, 12 in 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: 2.11 scrub ok 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: 2.19 scrub starts 2024-04-29T14:11:47.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:47 smithi040 ceph-mon[39581]: osdmap e163: 12 total, 12 up, 12 in 2024-04-29T14:11:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: 2.10 scrub starts 2024-04-29T14:11:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: pgmap v473: 27 pgs: 1 peering, 1 active+remapped+backfilling, 25 active+clean; 7.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1376/241888 objects misplaced (0.569%) 2024-04-29T14:11:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: 2.11 scrub starts 2024-04-29T14:11:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: 2.10 scrub ok 2024-04-29T14:11:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: osdmap e162: 12 total, 12 up, 12 in 2024-04-29T14:11:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: 2.11 scrub ok 2024-04-29T14:11:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: 2.19 scrub starts 2024-04-29T14:11:47.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:47 smithi121 ceph-mon[44533]: osdmap e163: 12 total, 12 up, 12 in 2024-04-29T14:11:48.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:48 smithi194 ceph-mon[44089]: 2.4 scrub starts 2024-04-29T14:11:48.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:48 smithi194 ceph-mon[44089]: 2.19 scrub ok 2024-04-29T14:11:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:48 smithi040 ceph-mon[39581]: 2.4 scrub starts 2024-04-29T14:11:48.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:48 smithi040 ceph-mon[39581]: 2.19 scrub ok 2024-04-29T14:11:48.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:48 smithi121 ceph-mon[44533]: 2.4 scrub starts 2024-04-29T14:11:48.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:48 smithi121 ceph-mon[44533]: 2.19 scrub ok 2024-04-29T14:11:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:49 smithi194 ceph-mon[44089]: pgmap v476: 27 pgs: 1 peering, 1 active+remapped+backfilling, 25 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1376/239280 objects misplaced (0.575%) 2024-04-29T14:11:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:49 smithi194 ceph-mon[44089]: 2.4 scrub ok 2024-04-29T14:11:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:49 smithi194 ceph-mon[44089]: 2.15 scrub starts 2024-04-29T14:11:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:49 smithi040 ceph-mon[39581]: pgmap v476: 27 pgs: 1 peering, 1 active+remapped+backfilling, 25 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1376/239280 objects misplaced (0.575%) 2024-04-29T14:11:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:49 smithi040 ceph-mon[39581]: 2.4 scrub ok 2024-04-29T14:11:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:49 smithi040 ceph-mon[39581]: 2.15 scrub starts 2024-04-29T14:11:49.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:49 smithi121 ceph-mon[44533]: pgmap v476: 27 pgs: 1 peering, 1 active+remapped+backfilling, 25 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1376/239280 objects misplaced (0.575%) 2024-04-29T14:11:49.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:49 smithi121 ceph-mon[44533]: 2.4 scrub ok 2024-04-29T14:11:49.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:49 smithi121 ceph-mon[44533]: 2.15 scrub starts 2024-04-29T14:11:49.811 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:49.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:49.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:50.038 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:50 smithi040 ceph-mon[39581]: 2.15 scrub ok 2024-04-29T14:11:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:50 smithi194 ceph-mon[44089]: 2.15 scrub ok 2024-04-29T14:11:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:50 smithi121 ceph-mon[44533]: 2.15 scrub ok 2024-04-29T14:11:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:51 smithi194 ceph-mon[44089]: 2.f scrub starts 2024-04-29T14:11:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:51 smithi194 ceph-mon[44089]: pgmap v477: 27 pgs: 27 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:51 smithi194 ceph-mon[44089]: 2.14 scrub starts 2024-04-29T14:11:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:51 smithi194 ceph-mon[44089]: 2.14 scrub ok 2024-04-29T14:11:51.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:51 smithi040 ceph-mon[39581]: 2.f scrub starts 2024-04-29T14:11:51.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:51 smithi040 ceph-mon[39581]: pgmap v477: 27 pgs: 27 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:51.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:51 smithi040 ceph-mon[39581]: 2.14 scrub starts 2024-04-29T14:11:51.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:51 smithi040 ceph-mon[39581]: 2.14 scrub ok 2024-04-29T14:11:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:51 smithi121 ceph-mon[44533]: 2.f scrub starts 2024-04-29T14:11:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:51 smithi121 ceph-mon[44533]: pgmap v477: 27 pgs: 27 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:51 smithi121 ceph-mon[44533]: 2.14 scrub starts 2024-04-29T14:11:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:51 smithi121 ceph-mon[44533]: 2.14 scrub ok 2024-04-29T14:11:52.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:52 smithi194 ceph-mon[44089]: 2.f scrub ok 2024-04-29T14:11:52.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:52 smithi040 ceph-mon[39581]: 2.f scrub ok 2024-04-29T14:11:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:52 smithi121 ceph-mon[44533]: 2.f scrub ok 2024-04-29T14:11:52.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:11:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:53 smithi194 ceph-mon[44089]: pgmap v478: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:11:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:53 smithi194 ceph-mon[44089]: osdmap e164: 12 total, 12 up, 12 in 2024-04-29T14:11:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:53 smithi040 ceph-mon[39581]: pgmap v478: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:53.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:11:53.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:53 smithi040 ceph-mon[39581]: osdmap e164: 12 total, 12 up, 12 in 2024-04-29T14:11:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:53 smithi121 ceph-mon[44533]: pgmap v478: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:11:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:53 smithi121 ceph-mon[44533]: osdmap e164: 12 total, 12 up, 12 in 2024-04-29T14:11:55.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:55 smithi121 ceph-mon[44533]: pgmap v480: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:55 smithi194 ceph-mon[44089]: pgmap v480: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:55.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:55 smithi040 ceph-mon[39581]: pgmap v480: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:55.859 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:11:55.859 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:11:56.400 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:56.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:56.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:11:56.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:11:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:57.031 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:11:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:57 smithi194 ceph-mon[44089]: pgmap v481: 27 pgs: 27 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:57 smithi194 ceph-mon[44089]: osdmap e165: 12 total, 12 up, 12 in 2024-04-29T14:11:57.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:57 smithi121 ceph-mon[44533]: pgmap v481: 27 pgs: 27 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:57.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:57 smithi121 ceph-mon[44533]: osdmap e165: 12 total, 12 up, 12 in 2024-04-29T14:11:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:57 smithi040 ceph-mon[39581]: pgmap v481: 27 pgs: 27 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:57 smithi040 ceph-mon[39581]: osdmap e165: 12 total, 12 up, 12 in 2024-04-29T14:11:57.482 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:11:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:57.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:11:57.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:11:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:58.490 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:11:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:11:58.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:11:58.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:11:58.627 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:11:59.047 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:11:59.047 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:11:59.061 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:11:59 smithi121 ceph-mon[44533]: pgmap v483: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:59.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:11:59 smithi194 ceph-mon[44089]: pgmap v483: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:11:59 smithi040 ceph-mon[39581]: pgmap v483: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:11:59.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:12:00.181 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:12:00.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:12:00.585 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:00.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:00.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:00.740 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:01.071 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:01 smithi121 ceph-mon[44533]: pgmap v484: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:12:01.248 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:01 smithi040 ceph-mon[39581]: pgmap v484: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:12:01.499 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:01.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:01.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:01 smithi194 ceph-mon[44089]: pgmap v484: 27 pgs: 27 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T14:12:01.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:01.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:01.784 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:01.785 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:01.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:01.918 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:02.000 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:02.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:02.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:02.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:02.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:02 smithi121 ceph-mon[44533]: osdmap e166: 12 total, 12 up, 12 in 2024-04-29T14:12:02.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:02 smithi121 ceph-mon[44533]: 2.c scrub starts 2024-04-29T14:12:02.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:02 smithi121 ceph-mon[44533]: osdmap e167: 12 total, 12 up, 12 in 2024-04-29T14:12:02.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4069099254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:02.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:02 smithi121 ceph-mon[44533]: from='client.34892 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:02.108 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:12:02.108 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:12:02.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:02.155 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:02.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:02.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:02.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:02 smithi040 ceph-mon[39581]: osdmap e166: 12 total, 12 up, 12 in 2024-04-29T14:12:02.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:02 smithi040 ceph-mon[39581]: 2.c scrub starts 2024-04-29T14:12:02.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:02 smithi040 ceph-mon[39581]: osdmap e167: 12 total, 12 up, 12 in 2024-04-29T14:12:02.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4069099254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:02.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:02 smithi040 ceph-mon[39581]: from='client.34892 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:02.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:02.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:02.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:02.507 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:02 smithi194 ceph-mon[44089]: osdmap e166: 12 total, 12 up, 12 in 2024-04-29T14:12:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:02 smithi194 ceph-mon[44089]: 2.c scrub starts 2024-04-29T14:12:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:02 smithi194 ceph-mon[44089]: osdmap e167: 12 total, 12 up, 12 in 2024-04-29T14:12:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4069099254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:02 smithi194 ceph-mon[44089]: from='client.34892 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:02.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:02.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:02.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:12:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:02.931 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:02.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-04-29T14:12:02.943 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:12:02.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:03.014 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:03.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:03.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:03.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:03 smithi040 ceph-mon[39581]: pgmap v486: 27 pgs: 27 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-04-29T14:12:03.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:03 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:12:03.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:03 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:12:03.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:03 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:12:03.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:03 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:12:03.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:03 smithi040 ceph-mon[39581]: from='client.34892 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:03.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:03 smithi040 ceph-mon[39581]: osdmap e168: 12 total, 12 up, 12 in 2024-04-29T14:12:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:03.290 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:12:03.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:03 smithi121 ceph-mon[44533]: pgmap v486: 27 pgs: 27 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-04-29T14:12:03.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:03 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:12:03.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:03 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:12:03.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:03 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:12:03.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:03 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:12:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:03 smithi121 ceph-mon[44533]: from='client.34892 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:03 smithi121 ceph-mon[44533]: osdmap e168: 12 total, 12 up, 12 in 2024-04-29T14:12:03.401 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:03.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:03.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:03.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:03 smithi194 ceph-mon[44089]: pgmap v486: 27 pgs: 27 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-04-29T14:12:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:03 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:12:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:03 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:12:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:03 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:12:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:03 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:12:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:03 smithi194 ceph-mon[44089]: from='client.34892 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:03 smithi194 ceph-mon[44089]: osdmap e168: 12 total, 12 up, 12 in 2024-04-29T14:12:03.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:03.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:03.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:03.733 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:04.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:04 smithi040 ceph-mon[39581]: 2.c scrub ok 2024-04-29T14:12:04.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:04 smithi040 ceph-mon[39581]: 2.a scrub starts 2024-04-29T14:12:04.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:04 smithi121 ceph-mon[44533]: 2.c scrub ok 2024-04-29T14:12:04.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:04 smithi121 ceph-mon[44533]: 2.a scrub starts 2024-04-29T14:12:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:04.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:04.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:04.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:04 smithi194 ceph-mon[44089]: 2.c scrub ok 2024-04-29T14:12:04.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:04 smithi194 ceph-mon[44089]: 2.a scrub starts 2024-04-29T14:12:04.579 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:04.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:04.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:04.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:04.710 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:04.764 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:04.796 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:04.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:04.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:04.939 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:05.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:05.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:05.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:05.195 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:12:05.195 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:12:05.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:05 smithi121 ceph-mon[44533]: pgmap v489: 27 pgs: 27 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:12:05.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:05.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:05.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:05.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:05.220 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:05 smithi121 ceph-mon[44533]: osdmap e169: 12 total, 12 up, 12 in 2024-04-29T14:12:05.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:12:05.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:12:05.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:12:05.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:12:05.310 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:12:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:05.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:05.351 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:12:05.379 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:12:05.414 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:12:05.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:05.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:12:05.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:05 smithi040 ceph-mon[39581]: pgmap v489: 27 pgs: 27 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:12:05.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:05.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:05.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:05.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:05.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:05 smithi040 ceph-mon[39581]: osdmap e169: 12 total, 12 up, 12 in 2024-04-29T14:12:05.476 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:05.483 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:12:05.517 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:12:05.558 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:12:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:05 smithi194 ceph-mon[44089]: pgmap v489: 27 pgs: 27 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:12:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:05 smithi194 ceph-mon[44089]: osdmap e169: 12 total, 12 up, 12 in 2024-04-29T14:12:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:12:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:12:05.984 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: 2.a scrub ok 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2459335769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: 2.3 scrub starts 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=2411002639,config=359415709,health=3897446751,kv=3654757942}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376419625}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661053108}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456855466}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636169133}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3692069078}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2591622971}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449948806}) 2024-04-29T14:12:06.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=894624788}) 2024-04-29T14:12:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2265496886}) 2024-04-29T14:12:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125718995}) 2024-04-29T14:12:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=433877393}) 2024-04-29T14:12:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363435229}) 2024-04-29T14:12:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980218543}) 2024-04-29T14:12:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791228274}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=437423348}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1241852641}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247944718}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606283083}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3535929901}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1530643108}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1704146204}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310999506}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575130651}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564983612}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1531640449}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2904670712}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=40,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=19} crc {logm=1672990820,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=879431884,mon_config_key=979177252}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1356091532}) 2024-04-29T14:12:06.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2189652826}) 2024-04-29T14:12:06.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=329114274,monmap=455069592,osd_metadata=1205174225,osd_pg_creating=2180537972,osdmap=689376737}) 2024-04-29T14:12:06.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=342826299}) 2024-04-29T14:12:06.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=33} crc {osdmap=3773484416}) 2024-04-29T14:12:06.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: 2.a scrub ok 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2459335769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: 2.3 scrub starts 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=2411002639,config=359415709,health=3897446751,kv=3654757942}) 2024-04-29T14:12:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376419625}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661053108}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456855466}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636169133}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3692069078}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2591622971}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449948806}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=894624788}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2265496886}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125718995}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=433877393}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363435229}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980218543}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791228274}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=437423348}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1241852641}) 2024-04-29T14:12:06.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247944718}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606283083}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3535929901}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1530643108}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1704146204}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310999506}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575130651}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564983612}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1531640449}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2904670712}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=40,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=19} crc {logm=1672990820,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=879431884,mon_config_key=979177252}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1356091532}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2189652826}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=329114274,monmap=455069592,osd_metadata=1205174225,osd_pg_creating=2180537972,osdmap=689376737}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=342826299}) 2024-04-29T14:12:06.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=33} crc {osdmap=3773484416}) 2024-04-29T14:12:06.393 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:06.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: 2.a scrub ok 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2459335769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: 2.3 scrub starts 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=2411002639,config=359415709,health=3897446751,kv=3654757942}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376419625}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661053108}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456855466}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636169133}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3692069078}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2591622971}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449948806}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=894624788}) 2024-04-29T14:12:06.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2265496886}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125718995}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=433877393}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363435229}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980218543}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791228274}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=437423348}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1241852641}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247944718}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606283083}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3535929901}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1530643108}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1704146204}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310999506}) 2024-04-29T14:12:06.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575130651}) 2024-04-29T14:12:06.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564983612}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1531640449}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2904670712}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=40,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=19} crc {logm=1672990820,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=879431884,mon_config_key=979177252}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1356091532}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2189652826}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=329114274,monmap=455069592,osd_metadata=1205174225,osd_pg_creating=2180537972,osdmap=689376737}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=342826299}) 2024-04-29T14:12:06.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=33} crc {osdmap=3773484416}) 2024-04-29T14:12:06.535 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:06.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:06.649 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:06.687 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:07.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:07.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:07 smithi040 ceph-mon[39581]: pgmap v491: 27 pgs: 27 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.66k op/s 2024-04-29T14:12:07.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:07 smithi040 ceph-mon[39581]: 2.b scrub starts 2024-04-29T14:12:07.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:07.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:07 smithi040 ceph-mon[39581]: 2.3 scrub ok 2024-04-29T14:12:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:07 smithi194 ceph-mon[44089]: pgmap v491: 27 pgs: 27 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.66k op/s 2024-04-29T14:12:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:07 smithi194 ceph-mon[44089]: 2.b scrub starts 2024-04-29T14:12:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:07 smithi194 ceph-mon[44089]: 2.3 scrub ok 2024-04-29T14:12:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:07 smithi121 ceph-mon[44533]: pgmap v491: 27 pgs: 27 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.66k op/s 2024-04-29T14:12:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:07 smithi121 ceph-mon[44533]: 2.b scrub starts 2024-04-29T14:12:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:07 smithi121 ceph-mon[44533]: 2.3 scrub ok 2024-04-29T14:12:07.815 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:08.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:08 smithi040 ceph-mon[39581]: 2.e scrub starts 2024-04-29T14:12:08.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:08 smithi040 ceph-mon[39581]: 2.b scrub ok 2024-04-29T14:12:08.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:08 smithi040 ceph-mon[39581]: pgmap v492: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.48k op/s 2024-04-29T14:12:08.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:08 smithi194 ceph-mon[44089]: 2.e scrub starts 2024-04-29T14:12:08.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:08 smithi194 ceph-mon[44089]: 2.b scrub ok 2024-04-29T14:12:08.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:08 smithi194 ceph-mon[44089]: pgmap v492: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.48k op/s 2024-04-29T14:12:08.577 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:12:08.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:08 smithi121 ceph-mon[44533]: 2.e scrub starts 2024-04-29T14:12:08.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:08 smithi121 ceph-mon[44533]: 2.b scrub ok 2024-04-29T14:12:08.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:08 smithi121 ceph-mon[44533]: pgmap v492: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.48k op/s 2024-04-29T14:12:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:08.952 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:08.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-04-29T14:12:08.954 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:08.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:09.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:09.091 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:09.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:09.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: 2.e scrub ok 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:12:09.200 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:12:09.201 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:09.274 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: 2.e scrub ok 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:12:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: 2.e scrub ok 2024-04-29T14:12:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:12:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:12:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:12:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:12:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:12:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:12:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:12:09.479 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:09.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:09.590 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:09.624 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:10.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:10 smithi040 ceph-mon[39581]: pgmap v493: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:12:10.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:10 smithi194 ceph-mon[44089]: pgmap v493: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:12:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:10 smithi121 ceph-mon[44533]: pgmap v493: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:12:10.774 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:10.774 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:12:10.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.774 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:245536 2.10 2:08de6323:::benchmark_data_smithi194_79000_object81275:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:12:10.774 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:10.774255+0000", 2024-04-29T14:12:10.774 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0059841299999999998, 2024-04-29T14:12:10.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060158520000000004, 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 245536 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774255+0000", 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774255+0000", 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774258+0000", 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3470000000000001e-06 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774265+0000", 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0169999999999999e-06 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774277+0000", 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2551999999999999e-05 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774287+0000", 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2580000000000001e-06 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774393+0000", 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106592 2024-04-29T14:12:10.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.774515+0000", 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000121407 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.775004+0000", 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00048989699999999999 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:10.779778+0000", 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047736610000000002 2024-04-29T14:12:10.778 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:10.779 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:10.779 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:10.779 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:10.779 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:12:10.779 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 1 2024-04-29T14:12:10.779 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:11.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:11.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:12:11.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:11.233 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:11.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:11.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:11 smithi040 ceph-mon[39581]: 2.d scrub starts 2024-04-29T14:12:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3405218415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:11 smithi040 ceph-mon[39581]: from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:11.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:11.531 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:11.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:11.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:11 smithi194 ceph-mon[44089]: 2.d scrub starts 2024-04-29T14:12:11.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3405218415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:11.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:11 smithi194 ceph-mon[44089]: from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:11.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:11 smithi121 ceph-mon[44533]: 2.d scrub starts 2024-04-29T14:12:11.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3405218415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:11.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:11 smithi121 ceph-mon[44533]: from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:11.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:11.693 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:12:11.836 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:11.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:11.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:11.982 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:12.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:12.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:12.091 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:12.112 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:12.161 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:12.262 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:12.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:12.368 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:12 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:12 smithi121 ceph-mon[44533]: from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:12:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:12 smithi121 ceph-mon[44533]: osdmap e170: 12 total, 12 up, 12 in 2024-04-29T14:12:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:12 smithi121 ceph-mon[44533]: pgmap v495: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:12:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:12 smithi121 ceph-mon[44533]: osdmap e171: 12 total, 12 up, 12 in 2024-04-29T14:12:12.425 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:12 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:12.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:12 smithi194 ceph-mon[44089]: from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:12:12.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:12 smithi194 ceph-mon[44089]: osdmap e170: 12 total, 12 up, 12 in 2024-04-29T14:12:12.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:12 smithi194 ceph-mon[44089]: pgmap v495: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:12:12.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:12 smithi194 ceph-mon[44089]: osdmap e171: 12 total, 12 up, 12 in 2024-04-29T14:12:12.428 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:12.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:12 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:12.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:12 smithi040 ceph-mon[39581]: from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:12:12.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:12 smithi040 ceph-mon[39581]: osdmap e170: 12 total, 12 up, 12 in 2024-04-29T14:12:12.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:12 smithi040 ceph-mon[39581]: pgmap v495: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:12:12.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:12 smithi040 ceph-mon[39581]: osdmap e171: 12 total, 12 up, 12 in 2024-04-29T14:12:12.458 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:12.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:12.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:12.621 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:12.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:12.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:12.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:12.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:12.885 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:12.921 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:12.983 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:12.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:13.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:13.143 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:13.213 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:13.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:13.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:13.363 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:13.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:13.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:13.494 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:13.544 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:13.697 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:13.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:13.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:13.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:13.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:13.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:13.998 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:14.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:14.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:14.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:14.176 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:14.246 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:14.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:14.336 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:12:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:12:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:12:14.336 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:14.353 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:14.397 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:14.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:12:14.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:14 smithi194 ceph-mon[44089]: pgmap v497: 27 pgs: 27 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:12:14.806 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:14 smithi040 ceph-mon[39581]: pgmap v497: 27 pgs: 27 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:12:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:14.839 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:14.839 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-04-29T14:12:14.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:14 smithi121 ceph-mon[44533]: pgmap v497: 27 pgs: 27 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:12:14.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:12:14.931 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:12:14.973 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:12:15.007 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:12:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:15.045 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:12:15.064 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:15.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:15.082 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:12:15.103 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:12:15.111 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:12:15.146 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:12:15.168 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:15.198 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:12:15.234 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:12:15.248 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:15.261 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:12:15.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:15.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:15.294 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:12:15.296 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:12:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:12:15.386 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:15.446 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:15.475 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:15.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:15.544 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:15.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:15.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3777280304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:15.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.619 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:15.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:15.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:15.752 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:15.786 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:15.789 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3777280304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:15.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.817 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:15.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:15.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3777280304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:15.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:15.925 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:15.971 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:16.436 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:16.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:16.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:16.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:16.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:16.707 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:16.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:16 smithi194 ceph-mon[44089]: pgmap v498: 27 pgs: 27 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.51k op/s 2024-04-29T14:12:16.753 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:16.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:16 smithi040 ceph-mon[39581]: pgmap v498: 27 pgs: 27 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.51k op/s 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:16.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:16 smithi121 ceph-mon[44533]: pgmap v498: 27 pgs: 27 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.51k op/s 2024-04-29T14:12:17.124 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:17.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:17.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:17.289 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:17.403 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:17.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:17.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:17.550 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:17.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:17.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:17.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:17.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:17.698 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:17.716 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:17.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:17.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:17.844 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:12:17.844 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:12:17.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:17.885 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:17.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:18.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:18.073 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:18.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:18.179 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:18.236 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:18.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:18 smithi194 ceph-mon[44089]: pgmap v499: 27 pgs: 27 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.60k op/s 2024-04-29T14:12:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2918116545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:18 smithi194 ceph-mon[44089]: from='client.34928 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:18.578 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:12:18.696 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:18.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:18.733 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:12:18.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:18.833 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:18 smithi121 ceph-mon[44533]: pgmap v499: 27 pgs: 27 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.60k op/s 2024-04-29T14:12:18.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2918116545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:18.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:18 smithi121 ceph-mon[44533]: from='client.34928 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:18.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:18 smithi040 ceph-mon[39581]: pgmap v499: 27 pgs: 27 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.60k op/s 2024-04-29T14:12:18.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2918116545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:18.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:18 smithi040 ceph-mon[39581]: from='client.34928 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:18.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:18.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:18.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:19.000 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:19.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:19.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:19.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:19.167 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:19.190 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:19.206 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:12:19.710 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:19 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:19.710 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:19 smithi194 ceph-mon[44089]: from='client.34928 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:19.710 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:19 smithi194 ceph-mon[44089]: osdmap e172: 12 total, 12 up, 12 in 2024-04-29T14:12:19.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:19 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:19.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:19 smithi040 ceph-mon[39581]: from='client.34928 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:19.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:19 smithi040 ceph-mon[39581]: osdmap e172: 12 total, 12 up, 12 in 2024-04-29T14:12:19.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:19 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:19.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:19 smithi121 ceph-mon[44533]: from='client.34928 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:19.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:19 smithi121 ceph-mon[44533]: osdmap e172: 12 total, 12 up, 12 in 2024-04-29T14:12:19.947 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:19.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:20.057 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:20.114 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:20.573 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:20.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:20.682 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:20.716 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:20.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:20 smithi194 ceph-mon[44089]: pgmap v501: 27 pgs: 27 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:12:20.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:20 smithi040 ceph-mon[39581]: pgmap v501: 27 pgs: 27 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:12:20.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:20 smithi121 ceph-mon[44533]: pgmap v501: 27 pgs: 27 active+clean; 4.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:12:20.979 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:20.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:21.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:21.131 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:21.156 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:21.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:21.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:21.265 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:12:21.265 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:12:21.291 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:21.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:21.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:21.423 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:21.471 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:21.538 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:21.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:21.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:21.678 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:21.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:12:22.035 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:12:22.421 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:22.830 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:22.830 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:22.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-04-29T14:12:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:22.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:22.939 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:22.956 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:22 smithi194 ceph-mon[44089]: pgmap v502: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s 2024-04-29T14:12:22.956 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:22 smithi194 ceph-mon[44089]: osdmap e173: 12 total, 12 up, 12 in 2024-04-29T14:12:22.990 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:23.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:22 smithi040 ceph-mon[39581]: pgmap v502: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s 2024-04-29T14:12:23.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:22 smithi040 ceph-mon[39581]: osdmap e173: 12 total, 12 up, 12 in 2024-04-29T14:12:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:22 smithi121 ceph-mon[44533]: pgmap v502: 27 pgs: 27 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s 2024-04-29T14:12:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:22 smithi121 ceph-mon[44533]: osdmap e173: 12 total, 12 up, 12 in 2024-04-29T14:12:23.472 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:23.846 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:23.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:23.952 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:24.010 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:24.090 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:24.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:24.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:24.245 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:24.304 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:24.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:24.410 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:12:24.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:24.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:12:24.471 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:24.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:12:24.502 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:12:24.529 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:12:24.547 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:12:24.547 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:12:24.554 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:12:24.589 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:12:24.619 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:12:24.659 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:12:24.688 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:12:24.719 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:12:24.732 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:24.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:24.756 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:12:24.775 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:12:24.775 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:12:24.775 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:12:24.775 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:12:24.775 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:12:24.776 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:12:24.776 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:12:24.776 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:12:24.776 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:12:24.776 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:12:24.776 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:12:24.776 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:12:24.848 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:24.902 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:24.957 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:24.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:25.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: pgmap v504: 27 pgs: 27 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: osdmap e174: 12 total, 12 up, 12 in 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1099594589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:25.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:25.285 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: pgmap v504: 27 pgs: 27 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:12:25.285 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:25.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:25.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: osdmap e174: 12 total, 12 up, 12 in 2024-04-29T14:12:25.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1099594589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:25.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.318 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:25.330 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:12:25.364 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:25.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: pgmap v504: 27 pgs: 27 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: osdmap e174: 12 total, 12 up, 12 in 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1099594589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.388 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:25.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:25.493 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:25.545 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:25.582 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:25.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:25.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:25.718 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:25.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:25.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:25.761 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:25.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:25.900 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:25.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:25.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:25.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.042 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.044 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:26.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:26.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:26.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:26.226 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:27.164 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:26 smithi194 ceph-mon[44089]: pgmap v506: 27 pgs: 27 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.72k op/s 2024-04-29T14:12:27.165 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:27.165 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:27.165 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:26 smithi194 ceph-mon[44089]: osdmap e175: 12 total, 12 up, 12 in 2024-04-29T14:12:27.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:26 smithi040 ceph-mon[39581]: pgmap v506: 27 pgs: 27 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.72k op/s 2024-04-29T14:12:27.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:27.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:27.232 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:26 smithi040 ceph-mon[39581]: osdmap e175: 12 total, 12 up, 12 in 2024-04-29T14:12:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:26 smithi121 ceph-mon[44533]: pgmap v506: 27 pgs: 27 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.72k op/s 2024-04-29T14:12:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:27.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:26 smithi121 ceph-mon[44533]: osdmap e175: 12 total, 12 up, 12 in 2024-04-29T14:12:27.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:12:27.868 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:27.868 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:12:27.868 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:12:27.868 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:12:27.868 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.868 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:166433 2.18 2:1e1aa8a7:::benchmark_data_smithi194_79000_object2172:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e140)", 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:10:57.916117+0000", 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 89.953138022000005, 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16973026099999999, 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 166433 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916117+0000", 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916117+0000", 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916119+0000", 2024-04-29T14:12:27.870 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6740000000000001e-06 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916127+0000", 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.379e-06 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916140+0000", 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3718e-05 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.871 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916157+0000", 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6636000000000001e-05 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916239+0000", 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2072999999999998e-05 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:57.916268+0000", 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8795000000000001e-05 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.020655+0000", 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.10438676600000001 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.020815+0000", 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000160812 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.020958+0000", 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014209499999999999 2024-04-29T14:12:27.873 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.023320+0000", 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023623820000000001 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.028361+0000", 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050415269999999996 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.085656+0000", 2024-04-29T14:12:27.874 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.057294229000000002 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.085731+0000", 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5783000000000003e-05 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:58.085847+0000", 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115392 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.875 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172177 2.d 2:b45a969b:::benchmark_data_smithi194_79000_object7916:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.892064+0000", 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.977190144000005, 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.236031834, 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172177 2024-04-29T14:12:27.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892064+0000", 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892064+0000", 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892066+0000", 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.968e-06 2024-04-29T14:12:27.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892072+0000", 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3809999999999998e-06 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892084+0000", 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2311e-05 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892092+0000", 2024-04-29T14:12:27.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1990000000000006e-06 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892151+0000", 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8669999999999999e-05 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892282+0000", 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013109299999999999 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.892410+0000", 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000128119 2024-04-29T14:12:27.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.901580+0000", 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0091694940000000003 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128028+0000", 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.22644804599999999 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128068+0000", 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0092000000000002e-05 2024-04-29T14:12:27.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128096+0000", 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8461000000000001e-05 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172180 2.d 2:b2eed99c:::benchmark_data_smithi194_79000_object7919:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.895882+0000", 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.973372412000003, 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23234322900000001, 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.881 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172180 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.895882+0000", 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.895882+0000", 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.895884+0000", 2024-04-29T14:12:27.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1519999999999999e-06 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.895889+0000", 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9620000000000001e-06 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.895898+0000", 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0699999999999996e-06 2024-04-29T14:12:27.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.895904+0000", 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9290000000000003e-06 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.896903+0000", 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099909299999999994 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.897030+0000", 2024-04-29T14:12:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012653400000000001 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.897151+0000", 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012128799999999999 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.901737+0000", 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045858069999999999 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128132+0000", 2024-04-29T14:12:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.22639546099999999 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128177+0000", 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4353e-05 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128225+0000", 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8579999999999999e-05 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172202 2.18 2:1b23e356:::benchmark_data_smithi194_79000_object7941:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.927929+0000", 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.941325843000001, 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17514848099999999, 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172202 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.888 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.927929+0000", 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.927929+0000", 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.927931+0000", 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0080000000000001e-06 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.927937+0000", 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4910000000000002e-06 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.927950+0000", 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2886e-05 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.927958+0000", 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4340000000000003e-06 2024-04-29T14:12:27.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.928012+0000", 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4710000000000003e-05 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.928114+0000", 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101486 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.928425+0000", 2024-04-29T14:12:27.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031140200000000001 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.930190+0000", 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017644609999999999 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.999727+0000", 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.069537236000000002 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.102937+0000", 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.10321034699999999 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.102978+0000", 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0946999999999999e-05 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.103077+0000", 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9073000000000005e-05 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172208 2.d 2:b96b2aab:::benchmark_data_smithi194_79000_object7947:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.937307+0000", 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.931947385000001, 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.191040291, 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172208 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.894 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937307+0000", 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937307+0000", 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937309+0000", 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.029e-06 2024-04-29T14:12:27.895 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937315+0000", 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6180000000000003e-06 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937324+0000", 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2380000000000003e-06 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937331+0000", 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6549999999999998e-06 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937385+0000", 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4245e-05 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937517+0000", 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000131693 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.937656+0000", 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000139677 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.945496+0000", 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078391550000000004 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128276+0000", 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18278079899999999 2024-04-29T14:12:27.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128311+0000", 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4671000000000001e-05 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128347+0000", 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6511000000000003e-05 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.899 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172210 2.d 2:b22fd145:::benchmark_data_smithi194_79000_object7949:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.941211+0000", 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.928043463999998, 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.187281635, 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172210 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.900 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941211+0000", 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941211+0000", 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941213+0000", 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.295e-06 2024-04-29T14:12:27.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941220+0000", 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7050000000000002e-06 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941234+0000", 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4056999999999999e-05 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941242+0000", 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2230000000000007e-06 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941747+0000", 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050447100000000004 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.941880+0000", 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000133254 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.942018+0000", 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000137959 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.945623+0000", 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036044779999999999 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128401+0000", 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.182777983 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128452+0000", 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1677e-05 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128493+0000", 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0533e-05 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172213 2.18 2:1da138b4:::benchmark_data_smithi194_79000_object7952:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.947370+0000", 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.921884929000001, 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.155948687, 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172213 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947370+0000", 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947370+0000", 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947372+0000", 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2970000000000002e-06 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947378+0000", 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7830000000000004e-06 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947390+0000", 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2658e-05 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947398+0000", 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6699999999999994e-06 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947443+0000", 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4984999999999997e-05 2024-04-29T14:12:27.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947568+0000", 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012519000000000001 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.947849+0000", 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00028050299999999998 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.949232+0000", 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0013831169999999999 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.000003+0000", 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.050771412000000002 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.103146+0000", 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.103142978 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.103210+0000", 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3966000000000004e-05 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.103318+0000", 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010812800000000001 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172237 2.d 2:bfe306fd:::benchmark_data_smithi194_79000_object7976:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.911 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:06.008882+0000", 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.860372910999999, 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11973974499999999, 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172237 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.008882+0000", 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.008882+0000", 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.008884+0000", 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3640000000000002e-06 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.008890+0000", 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.144e-06 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.008906+0000", 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5622999999999999e-05 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.008914+0000", 2024-04-29T14:12:27.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.507e-06 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.008977+0000", 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2729000000000002e-05 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.009086+0000", 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010911 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.009187+0000", 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101211 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.013811+0000", 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046236929999999999 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128537+0000", 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.114726131 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128585+0000", 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7926000000000002e-05 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128621+0000", 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6307000000000003e-05 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:172243 2.d 2:b6efe779:::benchmark_data_smithi194_79000_object7982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.917 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:06.015195+0000", 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 81.854059950000007, 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11356764599999999, 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172243 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015195+0000", 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015195+0000", 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015197+0000", 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4169999999999999e-06 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015204+0000", 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6510000000000004e-06 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015217+0000", 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3789e-05 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015226+0000", 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0290000000000005e-06 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015282+0000", 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6907000000000003e-05 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015399+0000", 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011692999999999999 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.015499+0000", 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9465999999999999e-05 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.020786+0000", 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005287478 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128671+0000", 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.107884625 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128725+0000", 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4434000000000001e-05 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:06.128762+0000", 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6919999999999999e-05 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:173834 2.18 2:1fe16fab:::benchmark_data_smithi194_79000_object9573:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.541109+0000", 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 79.328145367000005, 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.100697012, 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.923 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173834 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541109+0000", 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541109+0000", 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541112+0000", 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.283e-06 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541118+0000", 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3130000000000004e-06 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541130+0000", 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2636999999999999e-05 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541139+0000", 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1179999999999992e-06 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541201+0000", 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2763999999999997e-05 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541344+0000", 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014262699999999999 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.541669+0000", 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00032492399999999998 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.544965+0000", 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003296153 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.546403+0000", 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014375830000000001 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.641660+0000", 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095257235999999995 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.641696+0000", 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6380000000000001e-05 2024-04-29T14:12:27.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.641806+0000", 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109994 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:173836 2.18 2:1d10cdab:::benchmark_data_smithi194_79000_object9575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.543294+0000", 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 79.325960365, 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.100278825, 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.929 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173836 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543294+0000", 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543294+0000", 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543296+0000", 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8109999999999999e-06 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543302+0000", 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5979999999999996e-06 2024-04-29T14:12:27.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543310+0000", 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1790000000000008e-06 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543318+0000", 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1610000000000003e-06 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543554+0000", 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000236183 2024-04-29T14:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543616+0000", 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1588999999999996e-05 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.543843+0000", 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00022676400000000001 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.546376+0000", 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002533918 2024-04-29T14:12:27.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.548353+0000", 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001976827 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.643406+0000", 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095052906000000006 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.643465+0000", 2024-04-29T14:12:27.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9162e-05 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.643573+0000", 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010772699999999999 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174040 2.18 2:1c784e9b:::benchmark_data_smithi194_79000_object9779:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.821454+0000", 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 79.047801084, 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.099632997000000001, 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.935 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174040 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821454+0000", 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821454+0000", 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821455+0000", 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.809e-06 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821461+0000", 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.054e-06 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821474+0000", 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.223e-05 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821481+0000", 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8229999999999995e-06 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821533+0000", 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1594000000000001e-05 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.821670+0000", 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013694799999999999 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.822089+0000", 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00041880299999999998 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.823459+0000", 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001370088 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.826178+0000", 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027193819999999998 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.920927+0000", 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.094748649000000004 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.920991+0000", 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3932999999999999e-05 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.921086+0000", 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5684000000000001e-05 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174509 2.18 2:1f94f580:::benchmark_data_smithi194_79000_object10248:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:09.696305+0000", 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 78.172949240999998, 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.102363282, 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.941 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174509 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696305+0000", 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696305+0000", 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696307+0000", 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.756e-06 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696312+0000", 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.792e-06 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696323+0000", 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0905e-05 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696330+0000", 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9349999999999997e-06 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696366+0000", 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6488e-05 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696472+0000", 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106257 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.696825+0000", 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000352896 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.698755+0000", 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001929507 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.708140+0000", 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093854300000000002 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.798564+0000", 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.090424161000000003 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.798603+0000", 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8587000000000002e-05 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.798669+0000", 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5568000000000006e-05 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174520 2.18 2:1a18adcf:::benchmark_data_smithi194_79000_object10259:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:09.705202+0000", 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 78.164052119000004, 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.103091027, 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174520 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705202+0000", 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705202+0000", 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705203+0000", 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0500000000000002e-07 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705206+0000", 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8779999999999998e-06 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705213+0000", 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1189999999999999e-06 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705220+0000", 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1130000000000003e-06 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705264+0000", 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3489000000000003e-05 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705379+0000", 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115074 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.705765+0000", 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00038567499999999998 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.707509+0000", 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001744156 2024-04-29T14:12:27.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.710011+0000", 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025025960000000002 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.808154+0000", 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.098142170000000001 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.808192+0000", 2024-04-29T14:12:27.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8761999999999999e-05 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.808293+0000", 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010119 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174818 2.18 2:1c96320e:::benchmark_data_smithi194_79000_object10557:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:10.087557+0000", 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 77.781697652000005, 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.104370876, 2024-04-29T14:12:27.953 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174818 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.087557+0000", 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.087557+0000", 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.087559+0000", 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7409999999999999e-06 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.087564+0000", 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9280000000000001e-06 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.087573+0000", 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.397e-06 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.087580+0000", 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1679999999999998e-06 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.090661+0000", 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030813199999999998 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.090808+0000", 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000147007 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.091162+0000", 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00035340799999999999 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.094057+0000", 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002895225 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.094087+0000", 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9396000000000001e-05 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.191785+0000", 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.097698117000000001 2024-04-29T14:12:27.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.191821+0000", 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6616000000000003e-05 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.191928+0000", 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010655299999999999 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.959 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:175435 2.18 2:1ef25c75:::benchmark_data_smithi194_79000_object11174:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:11.070118+0000", 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 76.799136610000005, 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.116713727, 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.960 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 175435 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070118+0000", 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070118+0000", 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070120+0000", 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4480000000000001e-06 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070128+0000", 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8269999999999998e-06 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070145+0000", 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6492000000000001e-05 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070154+0000", 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9579999999999996e-06 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070215+0000", 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1167999999999994e-05 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070370+0000", 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000155131 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.070757+0000", 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00038663199999999998 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.072259+0000", 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015026760000000001 2024-04-29T14:12:27.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.077012+0000", 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004752371 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.186709+0000", 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.109697763 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.186766+0000", 2024-04-29T14:12:27.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6910999999999998e-05 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.186832+0000", 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5350000000000003e-05 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:175622 2.18 2:1ab89bfa:::benchmark_data_smithi194_79000_object11361:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:11.387816+0000", 2024-04-29T14:12:27.966 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 76.481438888, 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.106889463, 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 175622 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.387816+0000", 2024-04-29T14:12:27.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.387816+0000", 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.387818+0000", 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2469999999999998e-06 2024-04-29T14:12:27.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.387823+0000", 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1580000000000004e-06 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.387832+0000", 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2890000000000003e-06 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.387839+0000", 2024-04-29T14:12:27.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3690000000000003e-06 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.387885+0000", 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5933000000000002e-05 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.388009+0000", 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012382699999999999 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.388290+0000", 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000281978 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.389478+0000", 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011871659999999999 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.390485+0000", 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010068410000000001 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.494513+0000", 2024-04-29T14:12:27.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.10402839999999999 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.494580+0000", 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7556000000000002e-05 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:11.494705+0000", 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012469900000000001 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:176043 2.18 2:1ef90eea:::benchmark_data_smithi194_79000_object11782:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:12.013508+0000", 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 75.855746971000002, 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.101457134, 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 176043 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.013508+0000", 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.013508+0000", 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.013509+0000", 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.793e-06 2024-04-29T14:12:27.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.013514+0000", 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9729999999999997e-06 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.013525+0000", 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0698e-05 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.013532+0000", 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8449999999999997e-06 2024-04-29T14:12:27.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.014575+0000", 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010434439999999999 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.014707+0000", 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013124299999999999 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.015071+0000", 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036413 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.016651+0000", 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015797929999999999 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.018083+0000", 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001432022 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.114813+0000", 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.096730606999999996 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.114855+0000", 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1668000000000002e-05 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.114965+0000", 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109918 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.978 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:176396 2.18 2:1c844130:::benchmark_data_smithi194_79000_object12135:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:12.464682+0000", 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 75.404572647999998, 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.101826446, 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 176396 2024-04-29T14:12:27.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464682+0000", 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464682+0000", 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464684+0000", 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.029e-06 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464690+0000", 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7030000000000003e-06 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464701+0000", 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0922e-05 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464707+0000", 2024-04-29T14:12:27.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5209999999999999e-06 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464759+0000", 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1557999999999997e-05 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.464888+0000", 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000129178 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.465267+0000", 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037934999999999998 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.467037+0000", 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001769863 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.468472+0000", 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014347920000000001 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.566322+0000", 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.097850443999999995 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.566398+0000", 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5641000000000003e-05 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:12.566508+0000", 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110445 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.984 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:176979 2.18 2:1875ff51:::benchmark_data_smithi194_79000_object12718:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:13.314403+0000", 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 74.554851507999999, 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.102790588, 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 176979 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.985 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314403+0000", 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314403+0000", 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314405+0000", 2024-04-29T14:12:27.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2019999999999998e-06 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314411+0000", 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0059999999999998e-06 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314422+0000", 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0635000000000001e-05 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:27.987 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314429+0000", 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2579999999999998e-06 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314769+0000", 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000339334 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.314873+0000", 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000104817 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4,7", 2024-04-29T14:12:27.988 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.315231+0000", 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00035747200000000001 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.317040+0000", 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018089250000000001 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.317902+0000", 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00086270800000000001 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.417020+0000", 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.099117254000000002 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.417084+0000", 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4221999999999996e-05 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:13.417194+0000", 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109755 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.990 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.991 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.991 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:27.991 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:27.991 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:12:28.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2444744716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:12:28.252 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2444744716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:12:28.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2444744716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:12:28.666 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:12:29.085 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:28 smithi194 ceph-mon[44089]: pgmap v508: 27 pgs: 27 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s 2024-04-29T14:12:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:28 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2444744716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:12:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:28 smithi194 ceph-mon[44089]: osdmap e176: 12 total, 12 up, 12 in 2024-04-29T14:12:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:28 smithi194 ceph-mon[44089]: 2.d scrub starts 2024-04-29T14:12:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:28 smithi194 ceph-mon[44089]: 2.5 scrub starts 2024-04-29T14:12:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:28 smithi194 ceph-mon[44089]: 2.5 scrub ok 2024-04-29T14:12:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:28 smithi040 ceph-mon[39581]: pgmap v508: 27 pgs: 27 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s 2024-04-29T14:12:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:28 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2444744716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:12:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:28 smithi040 ceph-mon[39581]: osdmap e176: 12 total, 12 up, 12 in 2024-04-29T14:12:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:28 smithi040 ceph-mon[39581]: 2.d scrub starts 2024-04-29T14:12:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:28 smithi040 ceph-mon[39581]: 2.5 scrub starts 2024-04-29T14:12:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:28 smithi040 ceph-mon[39581]: 2.5 scrub ok 2024-04-29T14:12:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:28 smithi121 ceph-mon[44533]: pgmap v508: 27 pgs: 27 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s 2024-04-29T14:12:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:28 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2444744716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:12:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:28 smithi121 ceph-mon[44533]: osdmap e176: 12 total, 12 up, 12 in 2024-04-29T14:12:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:28 smithi121 ceph-mon[44533]: 2.d scrub starts 2024-04-29T14:12:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:28 smithi121 ceph-mon[44533]: 2.5 scrub starts 2024-04-29T14:12:29.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:28 smithi121 ceph-mon[44533]: 2.5 scrub ok 2024-04-29T14:12:30.296 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:30.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:29 smithi040 ceph-mon[39581]: 2.1 scrub starts 2024-04-29T14:12:30.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:29 smithi040 ceph-mon[39581]: 2.d scrub ok 2024-04-29T14:12:30.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:29 smithi040 ceph-mon[39581]: 2.1 scrub ok 2024-04-29T14:12:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:30.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:29 smithi194 ceph-mon[44089]: 2.1 scrub starts 2024-04-29T14:12:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:29 smithi194 ceph-mon[44089]: 2.d scrub ok 2024-04-29T14:12:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:29 smithi194 ceph-mon[44089]: 2.1 scrub ok 2024-04-29T14:12:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:29 smithi121 ceph-mon[44533]: 2.1 scrub starts 2024-04-29T14:12:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:29 smithi121 ceph-mon[44533]: 2.d scrub ok 2024-04-29T14:12:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:29 smithi121 ceph-mon[44533]: 2.1 scrub ok 2024-04-29T14:12:30.408 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:30.451 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:30.737 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:30.821 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:30.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:30.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:30.991 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:30.994 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:31 smithi194 ceph-mon[44089]: pgmap v510: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:12:30.994 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:31 smithi194 ceph-mon[44089]: 2.8 scrub starts 2024-04-29T14:12:30.994 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:31 smithi194 ceph-mon[44089]: 2.6 scrub starts 2024-04-29T14:12:30.994 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:31 smithi194 ceph-mon[44089]: 2.8 scrub ok 2024-04-29T14:12:30.994 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:31 smithi194 ceph-mon[44089]: 2.6 scrub ok 2024-04-29T14:12:31.021 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:31.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:31.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:31.186 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:30 smithi040 ceph-mon[39581]: pgmap v510: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:12:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:30 smithi040 ceph-mon[39581]: 2.8 scrub starts 2024-04-29T14:12:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:30 smithi040 ceph-mon[39581]: 2.6 scrub starts 2024-04-29T14:12:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:30 smithi040 ceph-mon[39581]: 2.8 scrub ok 2024-04-29T14:12:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:30 smithi040 ceph-mon[39581]: 2.6 scrub ok 2024-04-29T14:12:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:31 smithi121 ceph-mon[44533]: pgmap v510: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:12:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:31 smithi121 ceph-mon[44533]: 2.8 scrub starts 2024-04-29T14:12:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:31 smithi121 ceph-mon[44533]: 2.6 scrub starts 2024-04-29T14:12:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:31 smithi121 ceph-mon[44533]: 2.8 scrub ok 2024-04-29T14:12:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:31 smithi121 ceph-mon[44533]: 2.6 scrub ok 2024-04-29T14:12:31.444 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:31.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:31.475 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:31.475 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:31.480 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:31.480 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:12:31.480 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.480 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:274273 2.12 2:4897d8af:::benchmark_data_smithi194_79000_object110012:head [delete] snapc 0=[] ondisk+write+known_if_redirected e176)", 2024-04-29T14:12:31.480 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:31.480687+0000", 2024-04-29T14:12:31.480 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.002772591, 2024-04-29T14:12:31.480 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027983169999999998, 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 274273 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.481 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480687+0000", 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480687+0000", 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480688+0000", 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.685e-06 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480693+0000", 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0459999999999996e-06 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:31.483 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480704+0000", 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0912e-05 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480710+0000", 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.049e-06 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480776+0000", 2024-04-29T14:12:31.484 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5085000000000005e-05 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.480865+0000", 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9087999999999998e-05 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:31.481328+0000", 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00046311600000000002 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:12:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 1 2024-04-29T14:12:31.486 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:31.552 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:31.616 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:31.986 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:31.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:32.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:32.136 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:32.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:32.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:32.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:32.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:32.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:32.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:32 smithi121 ceph-mon[44533]: osdmap e177: 12 total, 12 up, 12 in 2024-04-29T14:12:32.301 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:32.313 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:32.313 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:32.313 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:32 smithi040 ceph-mon[39581]: osdmap e177: 12 total, 12 up, 12 in 2024-04-29T14:12:32.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:32.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:32.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:32 smithi194 ceph-mon[44089]: osdmap e177: 12 total, 12 up, 12 in 2024-04-29T14:12:32.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:32.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:32.345 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:32.345 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-04-29T14:12:32.352 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:12:32.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:32.467 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:32.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:32.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:32.598 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:32.652 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:32.691 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:32.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:32.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:12:32.799 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:32.800 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:32.842 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:32.977 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:32.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:33.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:33.111 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:33.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:33.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:33 smithi121 ceph-mon[44533]: pgmap v511: 27 pgs: 27 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.57k op/s 2024-04-29T14:12:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:33 smithi121 ceph-mon[44533]: 2.7 scrub starts 2024-04-29T14:12:33.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:33 smithi121 ceph-mon[44533]: 2.7 scrub ok 2024-04-29T14:12:33.268 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:33.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:33.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:33.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:33 smithi194 ceph-mon[44089]: pgmap v511: 27 pgs: 27 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.57k op/s 2024-04-29T14:12:33.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:33 smithi194 ceph-mon[44089]: 2.7 scrub starts 2024-04-29T14:12:33.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:33 smithi194 ceph-mon[44089]: 2.7 scrub ok 2024-04-29T14:12:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:33 smithi040 ceph-mon[39581]: pgmap v511: 27 pgs: 27 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.57k op/s 2024-04-29T14:12:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:33 smithi040 ceph-mon[39581]: 2.7 scrub starts 2024-04-29T14:12:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:33 smithi040 ceph-mon[39581]: 2.7 scrub ok 2024-04-29T14:12:33.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:33.453 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:33.584 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:33.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:33.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:33.694 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:33.720 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:33.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:33.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:33.782 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:12:33.815 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:12:33.843 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:12:33.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:33.880 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:12:33.898 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:33.917 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:12:33.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:12:33.987 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:12:34.011 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:34.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:34.017 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:12:34.056 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:12:34.091 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:12:34.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:34.123 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:12:34.153 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:34 smithi194 ceph-mon[44089]: 2.2 scrub starts 2024-04-29T14:12:34.153 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:34 smithi194 ceph-mon[44089]: 2.2 scrub ok 2024-04-29T14:12:34.153 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4203185104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:34.171 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:12:34.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:34 smithi040 ceph-mon[39581]: 2.2 scrub starts 2024-04-29T14:12:34.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:34 smithi040 ceph-mon[39581]: 2.2 scrub ok 2024-04-29T14:12:34.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4203185104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:34.182 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:12:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:12:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:12:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:12:34.184 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:12:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:34 smithi121 ceph-mon[44533]: 2.2 scrub starts 2024-04-29T14:12:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:34 smithi121 ceph-mon[44533]: 2.2 scrub ok 2024-04-29T14:12:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4203185104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:34.645 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:34.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:34.754 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:34.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:34.983 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:35.029 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:35.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: pgmap v513: 27 pgs: 27 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:12:35.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: pgmap v513: 27 pgs: 27 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: pgmap v513: 27 pgs: 27 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:37.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:37 smithi040 ceph-mon[39581]: pgmap v514: 27 pgs: 27 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.46k op/s 2024-04-29T14:12:37.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:37 smithi040 ceph-mon[39581]: 2.17 scrub starts 2024-04-29T14:12:37.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:37.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:37.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:37 smithi040 ceph-mon[39581]: osdmap e178: 12 total, 12 up, 12 in 2024-04-29T14:12:37.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:37 smithi040 ceph-mon[39581]: 2.17 scrub ok 2024-04-29T14:12:37.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:37 smithi040 ceph-mon[39581]: osdmap e179: 12 total, 12 up, 12 in 2024-04-29T14:12:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:37 smithi194 ceph-mon[44089]: pgmap v514: 27 pgs: 27 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.46k op/s 2024-04-29T14:12:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:37 smithi194 ceph-mon[44089]: 2.17 scrub starts 2024-04-29T14:12:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:37 smithi194 ceph-mon[44089]: osdmap e178: 12 total, 12 up, 12 in 2024-04-29T14:12:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:37 smithi194 ceph-mon[44089]: 2.17 scrub ok 2024-04-29T14:12:37.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:37 smithi194 ceph-mon[44089]: osdmap e179: 12 total, 12 up, 12 in 2024-04-29T14:12:37.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:37 smithi121 ceph-mon[44533]: pgmap v514: 27 pgs: 27 active+clean; 3.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.46k op/s 2024-04-29T14:12:37.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:37 smithi121 ceph-mon[44533]: 2.17 scrub starts 2024-04-29T14:12:37.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:37.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:37.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:37 smithi121 ceph-mon[44533]: osdmap e178: 12 total, 12 up, 12 in 2024-04-29T14:12:37.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:37 smithi121 ceph-mon[44533]: 2.17 scrub ok 2024-04-29T14:12:37.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:37 smithi121 ceph-mon[44533]: osdmap e179: 12 total, 12 up, 12 in 2024-04-29T14:12:37.379 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:12:37.380 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:12:39.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:39 smithi040 ceph-mon[39581]: pgmap v517: 27 pgs: 27 active+clean; 3.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:12:39.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:39 smithi194 ceph-mon[44089]: pgmap v517: 27 pgs: 27 active+clean; 3.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:12:39.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:39 smithi121 ceph-mon[44533]: pgmap v517: 27 pgs: 27 active+clean; 3.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:12:39.446 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:12:39.602 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:39.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:39.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:12:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:12:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:39.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:12:39.869 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:39.903 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:40.101 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:40.102 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:40 smithi194 ceph-mon[44089]: 2.0 scrub starts 2024-04-29T14:12:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:40 smithi194 ceph-mon[44089]: 2.0 scrub ok 2024-04-29T14:12:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2765048750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:40 smithi194 ceph-mon[44089]: from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:40.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:40 smithi040 ceph-mon[39581]: 2.0 scrub starts 2024-04-29T14:12:40.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:40 smithi040 ceph-mon[39581]: 2.0 scrub ok 2024-04-29T14:12:40.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:40.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:40.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2765048750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:40.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:40 smithi040 ceph-mon[39581]: from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:40 smithi121 ceph-mon[44533]: 2.0 scrub starts 2024-04-29T14:12:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:40 smithi121 ceph-mon[44533]: 2.0 scrub ok 2024-04-29T14:12:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2765048750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:40 smithi121 ceph-mon[44533]: from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:12:40.856 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:40.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:40.974 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:41.010 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:41 smithi194 ceph-mon[44089]: pgmap v518: 27 pgs: 27 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:12:41.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:41 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:12:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:41 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:12:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:41 smithi194 ceph-mon[44089]: from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:41.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:41 smithi194 ceph-mon[44089]: osdmap e180: 12 total, 12 up, 12 in 2024-04-29T14:12:41.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:41 smithi040 ceph-mon[39581]: pgmap v518: 27 pgs: 27 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:12:41.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:41 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:12:41.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:41 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:12:41.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:41 smithi040 ceph-mon[39581]: from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:41.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:41 smithi040 ceph-mon[39581]: osdmap e180: 12 total, 12 up, 12 in 2024-04-29T14:12:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:41 smithi121 ceph-mon[44533]: pgmap v518: 27 pgs: 27 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:12:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:41 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:12:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:41 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:12:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:41 smithi121 ceph-mon[44533]: from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:12:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:41 smithi121 ceph-mon[44533]: osdmap e180: 12 total, 12 up, 12 in 2024-04-29T14:12:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:41.470 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:41.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-04-29T14:12:42.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:12:42.583 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:42.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:42.687 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:42.741 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:42.768 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:42.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:42.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:42.928 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:42.956 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:42.964 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:42.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:43.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:43.118 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:43.229 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:43.233 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:42 smithi040 ceph-mon[39581]: pgmap v520: 27 pgs: 27 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:12:43.233 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:42 smithi040 ceph-mon[39581]: osdmap e181: 12 total, 12 up, 12 in 2024-04-29T14:12:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:43.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:43.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:42 smithi194 ceph-mon[44089]: pgmap v520: 27 pgs: 27 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:12:43.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:42 smithi194 ceph-mon[44089]: osdmap e181: 12 total, 12 up, 12 in 2024-04-29T14:12:43.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:43.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:42 smithi121 ceph-mon[44533]: pgmap v520: 27 pgs: 27 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:12:43.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:42 smithi121 ceph-mon[44533]: osdmap e181: 12 total, 12 up, 12 in 2024-04-29T14:12:43.368 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:43.391 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:43.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:43.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:43.522 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:43.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:43.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:43.663 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:43.712 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:43.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:12:43.853 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:43.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:43.958 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:43.997 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:44.196 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:12:44.231 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:12:44.259 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:12:44.295 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:12:44.333 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:12:44.363 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:12:44.394 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:12:44.418 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:12:44.446 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:12:44.473 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:44.474 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:12:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:44.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:44.503 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:12:44.551 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:12:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:12:44.579 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:12:44.608 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:44.749 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:12:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:44.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:12:44.856 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:44.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:44.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:44.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:45.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:45.066 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:45.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: pgmap v522: 27 pgs: 27 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:12:45.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3338236524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:45.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:45.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:13548 2.17 e142/137 2:ed650c75:::benchmark_data_smithi194_79000_object80113:head v 142'13359, mlcod=142'13350)", 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.504424+0000", 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.594205677000005, 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.055948502999999997, 2024-04-29T14:12:45.107 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.504424+0000", 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.504424+0000", 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.504425+0000", 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1459999999999999e-06 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.504432+0000", 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6189999999999996e-06 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.504440+0000", 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4409999999999993e-06 2024-04-29T14:12:45.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.504451+0000", 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1188e-05 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.505355+0000", 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00090435999999999999 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.505371+0000", 2024-04-29T14:12:45.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5701000000000001e-05 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560312+0000", 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.054941228000000002 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560372+0000", 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.982e-05 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:13549 2.17 e142/137 2:ed659ecb:::benchmark_data_smithi194_79000_object71575:head v 142'13360, mlcod=142'13350)", 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.507364+0000", 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.591265570000004, 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.053041525999999999, 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.507364+0000", 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.507364+0000", 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.507365+0000", 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.28e-06 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.507382+0000", 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6966999999999999e-05 2024-04-29T14:12:45.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.507392+0000", 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5400000000000001e-06 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.507401+0000", 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5920000000000007e-06 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.508139+0000", 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00073778699999999995 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.508156+0000", 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7235e-05 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560377+0000", 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.052220289000000003 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560405+0000", 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8836e-05 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:13550 2.17 e142/137 2:ed65ad71:::benchmark_data_smithi194_79000_object137271:head v 142'13361, mlcod=142'13350)", 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.510286+0000", 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.588343346000002, 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.050145323999999998, 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.116 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510286+0000", 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510286+0000", 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510287+0000", 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3400000000000001e-06 2024-04-29T14:12:45.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510296+0000", 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7350000000000002e-06 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510306+0000", 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0113e-05 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510318+0000", 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.185e-05 2024-04-29T14:12:45.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510959+0000", 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064107599999999997 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.510982+0000", 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2921e-05 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560408+0000", 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.049425425000000002 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560431+0000", 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3864000000000001e-05 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:171927 2.0 e142/137 2:00e7a6ed:::benchmark_data_smithi194_79000_object7666:head v 142'12957, mlcod=142'12953)", 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:05.511667+0000", 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.586962107999994, 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.059787972000000002, 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.120 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.511667+0000", 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.511667+0000", 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.511668+0000", 2024-04-29T14:12:45.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1659999999999999e-06 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.511675+0000", 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2450000000000003e-06 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560346+0000", 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.048671107999999998 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560354+0000", 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0250000000000002e-06 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560560+0000", 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00020604999999999999 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.560573+0000", 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3298e-05 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.571429+0000", 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010855806000000001 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:05.571455+0000", 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6274e-05 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:14260 2.17 e142/137 2:ee9dfc95:::benchmark_data_smithi194_79000_object62195:head v 142'14015, mlcod=142'13995)", 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312147+0000", 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786482724999999, 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.048600045000000001, 2024-04-29T14:12:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312147+0000", 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312147+0000", 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312147+0000", 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8099999999999998e-07 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312151+0000", 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8010000000000001e-06 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312156+0000", 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.938e-06 2024-04-29T14:12:45.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312160+0000", 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2350000000000001e-06 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.352411+0000", 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.040251441999999998 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.352423+0000", 2024-04-29T14:12:45.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.147e-05 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.360704+0000", 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082810390000000005 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.360747+0000", 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2738999999999999e-05 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:14261 2.17 e142/137 2:ee9e300e:::benchmark_data_smithi194_79000_object104355:head v 142'14016, mlcod=142'13995)", 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312166+0000", 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786463624000007, 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.053397068999999998, 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312166+0000", 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312166+0000", 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312166+0000", 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2000000000000001e-07 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312173+0000", 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6220000000000003e-06 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312178+0000", 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1619999999999999e-06 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312182+0000", 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2089999999999999e-06 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.354336+0000", 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042153439000000001 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.354349+0000", 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3828e-05 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.365527+0000", 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011177463 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.365563+0000", 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6025999999999997e-05 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.132 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:14262 2.17 e142/137 2:ee9ec436:::benchmark_data_smithi194_79000_object50506:head v 142'14017, mlcod=142'13995)", 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312188+0000", 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786441584000002, 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.053418172, 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312188+0000", 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312188+0000", 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312188+0000", 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9999999999999998e-07 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312192+0000", 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5810000000000001e-06 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312197+0000", 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2669999999999997e-06 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312201+0000", 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1810000000000003e-06 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.356640+0000", 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.044438404000000001 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.356658+0000", 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-05 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.365566+0000", 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089076859999999997 2024-04-29T14:12:45.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.365606+0000", 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9833000000000003e-05 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:14263 2.17 e142/137 2:ee9f271a:::benchmark_data_smithi194_79000_object92346:head v 142'14018, mlcod=142'13995)", 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312207+0000", 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786422741999999, 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.057684687999999998, 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.137 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312207+0000", 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312207+0000", 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312207+0000", 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8200000000000001e-07 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312214+0000", 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8639999999999998e-06 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312218+0000", 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4000000000000002e-06 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312222+0000", 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1200000000000004e-06 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.358574+0000", 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.046351746999999999 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.358595+0000", 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0682e-05 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369865+0000", 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011269639999999999 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369891+0000", 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6852999999999999e-05 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.6.0:14264 2.17 e142/137 2:ee9f8593:::benchmark_data_smithi194_79000_object108819:head v 142'14019, mlcod=142'13995)", 2024-04-29T14:12:45.141 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312228+0000", 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786401445999999, 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.057683476999999997, 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312228+0000", 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312228+0000", 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312228+0000", 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9000000000000002e-07 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312233+0000", 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1540000000000004e-06 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312238+0000", 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1399999999999999e-06 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312242+0000", 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.561e-06 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.360786+0000", 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.048543469999999998 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.360804+0000", 2024-04-29T14:12:45.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8805e-05 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369894+0000", 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009089158 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369911+0000", 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7799e-05 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173610 2.17 e142/137 2:ea6ae847:::benchmark_data_smithi194_79000_object9349:head v 142'14020, mlcod=142'13995)", 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312248+0000", 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786381844000005, 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.057679098999999998, 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312248+0000", 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312248+0000", 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312248+0000", 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5100000000000001e-07 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312254+0000", 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4219999999999997e-06 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312260+0000", 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5110000000000003e-06 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312264+0000", 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3390000000000003e-06 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.363200+0000", 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.050935963000000001 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.363216+0000", 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369913+0000", 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0066977130000000001 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369927+0000", 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3431000000000001e-05 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.149 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173617 2.17 e142/137 2:ea4c2311:::benchmark_data_smithi194_79000_object9356:head v 142'14021, mlcod=142'13995)", 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312270+0000", 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786359693999998, 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.059836752, 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312270+0000", 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312270+0000", 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312270+0000", 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7800000000000002e-07 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312274+0000", 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5130000000000001e-06 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312279+0000", 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1669999999999998e-06 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312283+0000", 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2239999999999997e-06 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.365653+0000", 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.053370421000000001 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.365673+0000", 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9927000000000001e-05 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372075+0000", 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064016380000000003 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372106+0000", 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1483999999999999e-05 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173625 2.17 e142/137 2:ed81edb2:::benchmark_data_smithi194_79000_object9364:head v 142'14022, mlcod=142'13995)", 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312288+0000", 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786340924000001, 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.062660669000000002, 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.154 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312288+0000", 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312288+0000", 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312289+0000", 2024-04-29T14:12:45.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7300000000000002e-07 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312293+0000", 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9380000000000002e-06 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312298+0000", 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6080000000000003e-06 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312303+0000", 2024-04-29T14:12:45.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5290000000000002e-06 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.367815+0000", 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.055512430000000001 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.367836+0000", 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.035e-05 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.374903+0000", 2024-04-29T14:12:45.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070671939999999997 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.374949+0000", 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6247e-05 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173636 2.7 e142/137 2:e5bc6543:::benchmark_data_smithi194_79000_object9375:head v 142'10593, mlcod=142'10591)", 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312309+0000", 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786320716999995, 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.059823306, 2024-04-29T14:12:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312309+0000", 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312309+0000", 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312309+0000", 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.89e-07 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312315+0000", 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1369999999999999e-06 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312321+0000", 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5430000000000001e-06 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312325+0000", 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3100000000000002e-06 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.367798+0000", 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.055472723000000002 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.367810+0000", 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2332e-05 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372110+0000", 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043002049999999997 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372132+0000", 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1667000000000001e-05 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.162 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173632 2.17 e142/137 2:ebef1b31:::benchmark_data_smithi194_79000_object9371:head v 142'14023, mlcod=142'13995)", 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312332+0000", 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786297021999999, 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.083033188999999993, 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312332+0000", 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312332+0000", 2024-04-29T14:12:45.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312333+0000", 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312336+0000", 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.529e-06 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312342+0000", 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3360000000000002e-06 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312346+0000", 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2560000000000004e-06 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369965+0000", 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.057619531000000002 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369982+0000", 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6753999999999999e-05 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.395324+0000", 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025341977000000002 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.395366+0000", 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1391999999999999e-05 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173640 2.7 e142/137 2:e581d5ea:::benchmark_data_smithi194_79000_object9379:head v 142'10594, mlcod=142'10591)", 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312351+0000", 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786278198000005, 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.064959344000000002, 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312351+0000", 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312351+0000", 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312352+0000", 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6100000000000002e-07 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312355+0000", 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3809999999999999e-06 2024-04-29T14:12:45.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312360+0000", 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1989999999999997e-06 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312365+0000", 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3889999999999998e-06 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369946+0000", 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.057581833999999998 2024-04-29T14:12:45.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.369959+0000", 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2218000000000001e-05 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.377275+0000", 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007316159 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.377311+0000", 2024-04-29T14:12:45.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5803000000000003e-05 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173648 2.17 e142/137 2:eb9a7570:::benchmark_data_smithi194_79000_object9387:head v 142'14024, mlcod=142'13996)", 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312370+0000", 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786259582, 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.086771748999999995, 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312370+0000", 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312370+0000", 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312370+0000", 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4499999999999998e-07 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312377+0000", 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5740000000000001e-06 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312382+0000", 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4260000000000002e-06 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312387+0000", 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3980000000000001e-06 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372145+0000", 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.059758881999999999 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372163+0000", 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7974000000000001e-05 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.399096+0000", 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.026932716999999998 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.399142+0000", 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5432999999999997e-05 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173673 2.7 e142/137 2:e4271adc:::benchmark_data_smithi194_79000_object9412:head v 142'10595, mlcod=142'10591)", 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312392+0000", 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786237256999996, 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.083020624000000001, 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.175 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312392+0000", 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312392+0000", 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312393+0000", 2024-04-29T14:12:45.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6899999999999998e-07 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312396+0000", 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.771e-06 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312401+0000", 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4009999999999999e-06 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312405+0000", 2024-04-29T14:12:45.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5020000000000003e-06 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372149+0000", 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.059743326999999999 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.372160+0000", 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1883e-05 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.395373+0000", 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023212199999999999 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.395413+0000", 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0170999999999999e-05 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.24727.0:173705 2.7 e142/137 2:e46e9efa:::benchmark_data_smithi194_79000_object9444:head v 142'10596, mlcod=142'10591)", 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:08.312410+0000", 2024-04-29T14:12:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.786218984000001, 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.086769810000000003, 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312410+0000", 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312410+0000", 2024-04-29T14:12:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312411+0000", 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.96e-07 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312417+0000", 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3790000000000003e-06 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312423+0000", 2024-04-29T14:12:45.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7849999999999997e-06 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.312427+0000", 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2239999999999997e-06 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.374924+0000", 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.062496472999999997 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.374950+0000", 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6084999999999998e-05 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.399145+0000", 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.024195326 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:08.399180+0000", 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5142000000000003e-05 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:196098 2.12 2:4963869d:::benchmark_data_smithi194_79000_object31837:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:34.941005+0000", 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.157624162000005, 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.050586735000000001, 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 196098 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.184 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941005+0000", 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941005+0000", 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941006+0000", 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1699999999999997e-07 2024-04-29T14:12:45.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941009+0000", 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.649e-06 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941020+0000", 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.114e-05 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941026+0000", 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0430000000000004e-06 2024-04-29T14:12:45.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941352+0000", 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00032623100000000001 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941356+0000", 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9509999999999999e-06 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.975620+0000", 2024-04-29T14:12:45.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.034263821999999999 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.975736+0000", 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116209 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.975992+0000", 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000256445 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.979956+0000", 2024-04-29T14:12:45.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039637709999999996 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.991491+0000", 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011535248999999999 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.991546+0000", 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5010999999999999e-05 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.991592+0000", 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5497000000000002e-05 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:196110 2.12 2:480e8aad:::benchmark_data_smithi194_79000_object31849:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:34.941044+0000", 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.157585853, 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.050664659000000001, 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:45.190 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 196110 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941044+0000", 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941044+0000", 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:45.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941044+0000", 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7599999999999996e-07 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941047+0000", 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.712e-06 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941053+0000", 2024-04-29T14:12:45.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2040000000000002e-06 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941059+0000", 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7849999999999997e-06 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941369+0000", 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030975300000000001 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.941372+0000", 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3500000000000001e-06 2024-04-29T14:12:45.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.978001+0000", 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.036628708000000003 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.978110+0000", 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108739 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.978433+0000", 2024-04-29T14:12:45.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00032319400000000001 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.981744+0000", 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0033107869999999999 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.991629+0000", 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0098857900000000002 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.991674+0000", 2024-04-29T14:12:45.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4947000000000002e-05 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:34.991708+0000", 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3914000000000001e-05 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:45.196 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:45.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:45.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: pgmap v522: 27 pgs: 27 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:12:45.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3338236524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:45.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.239 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.239 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.239 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.239 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.239 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.239 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.239 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:45.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:45.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: pgmap v522: 27 pgs: 27 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3338236524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:45.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:45.407 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:45.434 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:45.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:45.538 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:45.588 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:45.856 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:45.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:45.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:12:45.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:46.004 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:46.030 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:46.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:46.118 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:46.118 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:46.118 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:46.136 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:46.177 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.177 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.177 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.179 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:46.217 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:46.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:46.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:46.345 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:46.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:46.383 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:47.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:46 smithi040 ceph-mon[39581]: pgmap v523: 27 pgs: 27 active+clean; 2.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.48k op/s 2024-04-29T14:12:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:47.287 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:47.287 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-04-29T14:12:47.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:46 smithi194 ceph-mon[44089]: pgmap v523: 27 pgs: 27 active+clean; 2.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.48k op/s 2024-04-29T14:12:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:46 smithi121 ceph-mon[44533]: pgmap v523: 27 pgs: 27 active+clean; 2.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.48k op/s 2024-04-29T14:12:47.962 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:47.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:48.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:48.259 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:48.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:48.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:48.394 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:48.431 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:48.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:48.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:48.536 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:296898 2.b 2:df977ef1:::benchmark_data_smithi194_79000_object132637:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:48.524078+0000", 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.017523878999999999, 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017548660000000001, 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296898 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524078+0000", 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524078+0000", 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524079+0000", 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.33e-07 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524082+0000", 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1499999999999999e-06 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524089+0000", 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5529999999999998e-06 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524094+0000", 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4070000000000001e-06 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.527099+0000", 2024-04-29T14:12:48.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030050279999999999 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.527222+0000", 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012261100000000001 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.527344+0000", 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012205999999999999 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:48.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.537203+0000", 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0098593239999999992 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:296903 2.b 2:d327cf7f:::benchmark_data_smithi194_79000_object132642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:48.524273+0000", 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.017328698, 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017407855999999999, 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:12:48.544 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296903 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524273+0000", 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:48.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524273+0000", 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524274+0000", 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.99e-07 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524277+0000", 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7779999999999999e-06 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524282+0000", 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3399999999999997e-06 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.524287+0000", 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9989999999999999e-06 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.532287+0000", 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079999870000000001 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.532413+0000", 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126234 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.532508+0000", 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4708000000000002e-05 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:296905 2.b 2:d2d45406:::benchmark_data_smithi194_79000_object132644:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:12:48.548 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:48.533534+0000", 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0080675509999999992, 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082060380000000006, 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296905 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.533534+0000", 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:48.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.533534+0000", 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.533537+0000", 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1519999999999999e-06 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.533542+0000", 2024-04-29T14:12:48.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3689999999999998e-06 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.533552+0000", 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0413e-05 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.533560+0000", 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4540000000000001e-06 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.539776+0000", 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062160430000000001 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.539918+0000", 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014197600000000001 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:48.540049+0000", 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013155900000000001 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:48.552 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:48.553 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:48.553 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:12:48.553 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 3 2024-04-29T14:12:48.553 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:48.587 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:48.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:48.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:48.626 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:48.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:12:48.766 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:48.806 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:12:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:48.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:12:48.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:12:48.911 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:48.968 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:49.024 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:12:49.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:49 smithi121 ceph-mon[44533]: pgmap v524: 27 pgs: 27 active+clean; 2.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:12:49.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1797335494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:49.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:49 smithi121 ceph-mon[44533]: from='client.34973 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:49.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:49 smithi194 ceph-mon[44089]: pgmap v524: 27 pgs: 27 active+clean; 2.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:12:49.289 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1797335494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:49.289 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:49 smithi194 ceph-mon[44089]: from='client.34973 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:49.324 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:49.327 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:12:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:49.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:49.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:49 smithi040 ceph-mon[39581]: pgmap v524: 27 pgs: 27 active+clean; 2.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:12:49.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1797335494' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:49.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:49 smithi040 ceph-mon[39581]: from='client.34973 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:12:49.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:49.471 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:49.510 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:49.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:49.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:49.659 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:49.734 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:49.740 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:49.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:49.848 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:49.887 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:49.990 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:49.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:50.099 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:50.145 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:50.145 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:50 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:50.145 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:50 smithi194 ceph-mon[44089]: from='client.34973 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:12:50.145 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:50 smithi194 ceph-mon[44089]: osdmap e182: 12 total, 12 up, 12 in 2024-04-29T14:12:50.278 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:50.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:50.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:50 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:50.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:50 smithi040 ceph-mon[39581]: from='client.34973 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:12:50.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:50 smithi040 ceph-mon[39581]: osdmap e182: 12 total, 12 up, 12 in 2024-04-29T14:12:50.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:50 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:12:50.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:50 smithi121 ceph-mon[44533]: from='client.34973 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:12:50.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:50 smithi121 ceph-mon[44533]: osdmap e182: 12 total, 12 up, 12 in 2024-04-29T14:12:50.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:50.412 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:50.419 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:50.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:50.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:50.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:50.588 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:50.802 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:50.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:50.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:51.025 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:51.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:51 smithi040 ceph-mon[39581]: pgmap v526: 27 pgs: 27 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:12:51.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:51.180 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:51.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:51.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:51.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:51 smithi194 ceph-mon[44089]: pgmap v526: 27 pgs: 27 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:12:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:51 smithi121 ceph-mon[44533]: pgmap v526: 27 pgs: 27 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:12:51.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:12:51.430 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:51.512 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:12:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:51.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:12:51.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:51.739 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:51.739 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:12:51.739 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:12:51.739 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:12:51.739 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:51.798 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:51.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:51.903 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:51.948 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:52.004 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:52.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:52.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:52.120 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:52.120 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:52.120 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:52 smithi040 ceph-mon[39581]: osdmap e183: 12 total, 12 up, 12 in 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:bench [] [] [] [] 2024-04-29T14:12:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T14:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T14:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T14:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-04-29T14:12:52.181 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:52.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:52.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:52.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:52 smithi194 ceph-mon[44089]: osdmap e183: 12 total, 12 up, 12 in 2024-04-29T14:12:52.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:52.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:52.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:52 smithi121 ceph-mon[44533]: osdmap e183: 12 total, 12 up, 12 in 2024-04-29T14:12:52.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:12:52.568 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:12:52.952 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:12:53.303 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:53 smithi194 ceph-mon[44089]: pgmap v528: 27 pgs: 27 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s 2024-04-29T14:12:53.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:53 smithi040 ceph-mon[39581]: pgmap v528: 27 pgs: 27 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s 2024-04-29T14:12:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:53 smithi121 ceph-mon[44533]: pgmap v528: 27 pgs: 27 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s 2024-04-29T14:12:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:12:54.247 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-29T14:12:54.460 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:54.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:54.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:12:54.597 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:54.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:12:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:54.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:12:54.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:54.785 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:54.989 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:12:55.012 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:12:55.012 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:12:55.012 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:12:55.012 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:12:55.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:145699 2.a 2:54538e2e:::benchmark_data_smithi194_79000_object145698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:10:35.948031+0000", 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 139.066840727, 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068809756999999999, 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 145699 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.013 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948031+0000", 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948031+0000", 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948033+0000", 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0619999999999999e-06 2024-04-29T14:12:55.014 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948057+0000", 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4094999999999999e-05 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948068+0000", 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1e-05 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948076+0000", 2024-04-29T14:12:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1880000000000002e-06 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948132+0000", 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6405e-05 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948313+0000", 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018028599999999999 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.016 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948421+0000", 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108488 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.011352+0000", 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.062930629000000002 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.016742+0000", 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053906689999999998 2024-04-29T14:12:55.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.016803+0000", 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0155000000000003e-05 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.016840+0000", 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7780000000000001e-05 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:145700 2.a 2:5d653a2c:::benchmark_data_smithi194_79000_object145699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:10:35.948098+0000", 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 139.066773662, 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068884301999999994, 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 145700 2024-04-29T14:12:55.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948098+0000", 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948098+0000", 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948099+0000", 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9100000000000002e-07 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948119+0000", 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0412999999999999e-05 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948126+0000", 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.827e-06 2024-04-29T14:12:55.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.948131+0000", 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.699e-06 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.952616+0000", 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044849019999999998 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.952750+0000", 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013396099999999999 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.952886+0000", 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000135659 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.016675+0000", 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.063788826000000007 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.016896+0000", 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00022096299999999999 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.016945+0000", 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8829000000000003e-05 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.016982+0000", 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7332000000000001e-05 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.024 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:145720 2.a 2:5ed99eb2:::benchmark_data_smithi194_79000_object145719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:10:35.957946+0000", 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 139.05692544199999, 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068079597000000006, 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 145720 2024-04-29T14:12:55.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.957946+0000", 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.957946+0000", 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.957948+0000", 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8339999999999999e-06 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:35.957970+0000", 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.268e-05 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.006879+0000", 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.048908563000000002 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.006895+0000", 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5583999999999999e-05 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.017248+0000", 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010353014000000001 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.017363+0000", 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115616 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.017481+0000", 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011803000000000001 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.025942+0000", 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084612560000000003 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.025953+0000", 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0762e-05 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.025998+0000", 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5274000000000001e-05 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:10:36.026025+0000", 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6984000000000001e-05 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174343 2.c 2:350cbba0:::benchmark_data_smithi194_79000_object10082:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:09.426124+0000", 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 105.58874763199999, 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.073445014000000003, 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174343 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.031 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.426124+0000", 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.426124+0000", 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.426125+0000", 2024-04-29T14:12:55.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5540000000000001e-06 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.426131+0000", 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.694e-06 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.426140+0000", 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5489999999999995e-06 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.426146+0000", 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0279999999999997e-06 2024-04-29T14:12:55.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.429259+0000", 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031121439999999998 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.429375+0000", 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116484 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.429489+0000", 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000113806 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.437960+0000", 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084712060000000002 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.499472+0000", 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.061511580000000003 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.499522+0000", 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0420999999999997e-05 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.499569+0000", 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6548000000000002e-05 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174351 2.c 2:315d41e4:::benchmark_data_smithi194_79000_object10090:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:55.036 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:09.433498+0000", 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 105.581373361, 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.072452830999999995, 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174351 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.433498+0000", 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.433498+0000", 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.433499+0000", 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3999999999999999e-06 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.038 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.433504+0000", 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2950000000000004e-06 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.433514+0000", 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0329000000000001e-05 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.433521+0000", 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0570000000000004e-06 2024-04-29T14:12:55.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.434087+0000", 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00056632999999999998 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.434175+0000", 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8026000000000001e-05 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.434242+0000", 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6402000000000006e-05 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.438069+0000", 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038275409999999998 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.505873+0000", 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.067804133000000003 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.505910+0000", 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6062000000000001e-05 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.505951+0000", 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1255999999999999e-05 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174358 2.c 2:3fd0370e:::benchmark_data_smithi194_79000_object10097:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:09.444185+0000", 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 105.57068596400001, 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.069125470999999994, 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.043 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174358 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444185+0000", 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444185+0000", 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444187+0000", 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9589999999999998e-06 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444194+0000", 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6880000000000002e-06 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444206+0000", 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2342e-05 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444214+0000", 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9009999999999994e-06 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444268+0000", 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3312000000000001e-05 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444354+0000", 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6354999999999996e-05 2024-04-29T14:12:55.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.444439+0000", 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5222999999999994e-05 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.452009+0000", 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075703539999999996 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.513208+0000", 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.061198228 2024-04-29T14:12:55.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.513265+0000", 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7627999999999998e-05 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.513311+0000", 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5481e-05 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174359 2.c 2:399288d8:::benchmark_data_smithi194_79000_object10098:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:09.446471+0000", 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 105.56840026899999, 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.066988751999999999, 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174359 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.446471+0000", 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.446471+0000", 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.446473+0000", 2024-04-29T14:12:55.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.578e-06 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.446477+0000", 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5519999999999998e-06 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.446488+0000", 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1270000000000001e-05 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.446496+0000", 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8529999999999992e-06 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.448291+0000", 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001794468 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.448401+0000", 2024-04-29T14:12:55.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110319 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.448487+0000", 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6048e-05 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.452136+0000", 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003649149 2024-04-29T14:12:55.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.513367+0000", 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.061230677999999997 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.513417+0000", 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0130999999999998e-05 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:09.513460+0000", 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2706000000000001e-05 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:174912 2.c 2:3d0bb231:::benchmark_data_smithi194_79000_object10651:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:10.248862+0000", 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 104.76600881, 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.073784701999999994, 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.055 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174912 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.248862+0000", 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.248862+0000", 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.248865+0000", 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4310000000000001e-06 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.248872+0000", 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8000000000000001e-06 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.248887+0000", 2024-04-29T14:12:55.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5537999999999999e-05 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.248896+0000", 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5660000000000007e-06 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.248960+0000", 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3946000000000001e-05 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.249092+0000", 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013221700000000001 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.249216+0000", 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012355299999999999 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.253918+0000", 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047029419999999999 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.322549+0000", 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068630234999999998 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.322603+0000", 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3872000000000002e-05 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:10.322647+0000", 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4601999999999997e-05 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.060 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:177958 2.c 2:39328c3a:::benchmark_data_smithi194_79000_object13697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.100022+0000", 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.914849167, 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.089554231999999998, 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 177958 2024-04-29T14:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100022+0000", 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100022+0000", 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100024+0000", 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8190000000000001e-06 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100029+0000", 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7890000000000002e-06 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100038+0000", 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2720000000000003e-06 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100046+0000", 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6359999999999994e-06 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100084+0000", 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7916000000000001e-05 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100173+0000", 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9429e-05 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100247+0000", 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3813999999999995e-05 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.103657+0000", 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034102529999999998 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189500+0000", 2024-04-29T14:12:55.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.085843025000000003 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189540+0000", 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9620999999999999e-05 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189576+0000", 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6658e-05 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.066 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:177960 2.c 2:3d353946:::benchmark_data_smithi194_79000_object13699:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.108689+0000", 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.906182291999997, 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.081150963000000007, 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 177960 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.067 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108689+0000", 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108689+0000", 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108692+0000", 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9349999999999999e-06 2024-04-29T14:12:55.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108700+0000", 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5210000000000002e-06 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108724+0000", 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3376000000000001e-05 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108735+0000", 2024-04-29T14:12:55.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.152e-05 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108803+0000", 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7999999999999999e-05 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108928+0000", 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012510900000000001 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.109025+0000", 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6453000000000006e-05 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.112726+0000", 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003701513 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189788+0000", 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077061438999999995 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189817+0000", 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8873999999999998e-05 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189840+0000", 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3223000000000001e-05 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.072 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178139 2.c 2:3c2bb968:::benchmark_data_smithi194_79000_object13878:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.422554+0000", 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.592317120999994, 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.084318003000000002, 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178139 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422554+0000", 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422554+0000", 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422556+0000", 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8870000000000001e-06 2024-04-29T14:12:55.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422561+0000", 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8579999999999999e-06 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422573+0000", 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.17e-05 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422580+0000", 2024-04-29T14:12:55.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6760000000000007e-06 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422635+0000", 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4447999999999998e-05 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422725+0000", 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0698999999999993e-05 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422793+0000", 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7597999999999999e-05 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.426593+0000", 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037997199999999999 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.506762+0000", 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.080169293000000003 2024-04-29T14:12:55.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.506821+0000", 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8925000000000002e-05 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.506872+0000", 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1199000000000002e-05 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.079 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178446 2.c 2:3e923c02:::benchmark_data_smithi194_79000_object14185:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.079 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.913729+0000", 2024-04-29T14:12:55.079 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 99.101142166000002, 2024-04-29T14:12:55.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.074333147000000002, 2024-04-29T14:12:55.079 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.079 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.079 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178446 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913729+0000", 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913729+0000", 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913731+0000", 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.685e-06 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913736+0000", 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8749999999999999e-06 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913746+0000", 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0078000000000001e-05 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913753+0000", 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5650000000000004e-06 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913810+0000", 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6873000000000002e-05 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913926+0000", 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115635 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.914038+0000", 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011199100000000001 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.917954+0000", 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039161899999999999 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.987989+0000", 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.070034866000000001 2024-04-29T14:12:55.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.988027+0000", 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8210999999999997e-05 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.988062+0000", 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5178000000000001e-05 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178566 2.c 2:356259ea:::benchmark_data_smithi194_79000_object14305:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.144836+0000", 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 98.870035164000001, 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.112011359, 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178566 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144836+0000", 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144836+0000", 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144840+0000", 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4419999999999998e-06 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144848+0000", 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5429999999999994e-06 2024-04-29T14:12:55.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144867+0000", 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8550999999999999e-05 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144877+0000", 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0023999999999999e-05 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144945+0000", 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8017999999999998e-05 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.145075+0000", 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000130123 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.145161+0000", 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5828000000000006e-05 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.148891+0000", 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037306879999999998 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256736+0000", 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.1078446 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256794+0000", 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8264000000000003e-05 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256847+0000", 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3278e-05 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178571 2.c 2:31f7de83:::benchmark_data_smithi194_79000_object14310:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.152329+0000", 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 98.862542253000001, 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.104702769, 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.093 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178571 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152329+0000", 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152329+0000", 2024-04-29T14:12:55.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152332+0000", 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8559999999999998e-06 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152340+0000", 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1170000000000004e-06 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152357+0000", 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6594000000000001e-05 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152364+0000", 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6720000000000004e-06 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152428+0000", 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3874999999999994e-05 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152571+0000", 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014308600000000001 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152690+0000", 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118478 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.157030+0000", 2024-04-29T14:12:55.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043403169999999998 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256935+0000", 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.099905026999999993 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256989+0000", 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3486000000000003e-05 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.257032+0000", 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3260999999999998e-05 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178587 2.c 2:31bf17b8:::benchmark_data_smithi194_79000_object14326:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.166595+0000", 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 98.848276377000005, 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.104067166, 2024-04-29T14:12:55.099 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178587 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166595+0000", 2024-04-29T14:12:55.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166595+0000", 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166597+0000", 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8050000000000001e-06 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166602+0000", 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2959999999999998e-06 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166613+0000", 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1117e-05 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166619+0000", 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0129999999999999e-06 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166665+0000", 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5890999999999998e-05 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166759+0000", 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4129999999999995e-05 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166869+0000", 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109855 2024-04-29T14:12:55.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.174687+0000", 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078177899999999998 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270559+0000", 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095871737999999998 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270618+0000", 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9011000000000001e-05 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270662+0000", 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4520000000000001e-05 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178589 2.c 2:378ec238:::benchmark_data_smithi194_79000_object14328:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.168750+0000", 2024-04-29T14:12:55.105 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 98.846120987000006, 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.102036601, 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178589 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168750+0000", 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168750+0000", 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168752+0000", 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168757+0000", 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3909999999999997e-06 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168767+0000", 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0197000000000001e-05 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168775+0000", 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5129999999999999e-06 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.171154+0000", 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023789229999999998 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.171245+0000", 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1543000000000001e-05 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.171341+0000", 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5366999999999997e-05 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.174846+0000", 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035054539999999999 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270710+0000", 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095863956 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270752+0000", 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1834e-05 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270787+0000", 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4900999999999997e-05 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:239815 2.c 2:3e12e13d:::benchmark_data_smithi194_79000_object75554:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 48.418896844000002, 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077752630000000003, 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.111 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 239815 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:12:55.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595977+0000", 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2730000000000001e-06 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595984+0000", 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0559999999999999e-06 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595996+0000", 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2215000000000001e-05 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596005+0000", 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8249999999999994e-06 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596065+0000", 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0183000000000003e-05 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596172+0000", 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106702 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596280+0000", 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108251 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.605356+0000", 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090762619999999999 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673606+0000", 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068249290000000004 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673668+0000", 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2724000000000005e-05 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673727+0000", 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8848999999999998e-05 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.116 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:239816 2.c 2:3d6c3553:::benchmark_data_smithi194_79000_object75555:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 48.418848034, 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077792867000000002, 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.117 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 239816 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596024+0000", 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9400000000000004e-07 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596028+0000", 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6440000000000002e-06 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596034+0000", 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596041+0000", 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8210000000000004e-06 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601153+0000", 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051123899999999996 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601279+0000", 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012538299999999999 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601396+0000", 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117223 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.605374+0000", 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039785719999999997 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673769+0000", 2024-04-29T14:12:55.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068395101999999999 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673795+0000", 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.584e-05 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673816+0000", 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0874e-05 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:239838 2.a 2:5a447e17:::benchmark_data_smithi194_79000_object75577:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:06.611101+0000", 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 48.403770377999997, 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.073582199000000001, 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 239838 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.611101+0000", 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.611101+0000", 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.611105+0000", 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7720000000000001e-06 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.611116+0000", 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1260000000000001e-05 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.664564+0000", 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.053447733999999997 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.664582+0000", 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8743e-05 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.666563+0000", 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019810230000000002 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.666674+0000", 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110943 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.666789+0000", 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011506300000000001 2024-04-29T14:12:55.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.684510+0000", 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017721010999999998 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.684584+0000", 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3307999999999997e-05 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.684631+0000", 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7218000000000001e-05 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.684683+0000", 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2123999999999998e-05 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:270094 2.c 2:30a00e67:::benchmark_data_smithi194_79000_object105833:head [delete] snapc 0=[] ondisk+write+known_if_redirected e176)", 2024-04-29T14:12:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:28.352840+0000", 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.662031767999999, 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.071396684000000002, 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 270094 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.352840+0000", 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.352840+0000", 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.352842+0000", 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.418e-06 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.352848+0000", 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4060000000000002e-06 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.352862+0000", 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3816e-05 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.352871+0000", 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2229999999999996e-06 2024-04-29T14:12:55.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.352966+0000", 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4475999999999995e-05 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.353079+0000", 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112926 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:12:55.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.353189+0000", 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010992 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.357564+0000", 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043751140000000003 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.424158+0000", 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.066594048000000003 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.424197+0000", 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9514000000000001e-05 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:28.424236+0000", 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8822999999999997e-05 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:12:55.134 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:12:55.135 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:12:55.135 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:12:55.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:55 smithi040 ceph-mon[39581]: pgmap v529: 27 pgs: 27 active+clean; 1.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-04-29T14:12:55.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:12:55.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:12:55.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:55 smithi040 ceph-mon[39581]: osdmap e184: 12 total, 12 up, 12 in 2024-04-29T14:12:55.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2492739100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:55.136 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:12:55.136 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:12:55.150 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:12:55.190 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:12:55.220 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:12:55.260 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:12:55.291 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:12:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:55 smithi194 ceph-mon[44089]: pgmap v529: 27 pgs: 27 active+clean; 1.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-04-29T14:12:55.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:12:55.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:12:55.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:55 smithi194 ceph-mon[44089]: osdmap e184: 12 total, 12 up, 12 in 2024-04-29T14:12:55.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2492739100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:55.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:12:55.362 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:12:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:12:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:12:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:55.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:55.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:55 smithi121 ceph-mon[44533]: pgmap v529: 27 pgs: 27 active+clean; 1.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-04-29T14:12:55.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:12:55.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:12:55.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:55 smithi121 ceph-mon[44533]: osdmap e184: 12 total, 12 up, 12 in 2024-04-29T14:12:55.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2492739100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:12:55.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:55.519 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:55.775 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:12:55.777 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:55.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:55.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:56.068 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.068 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:56.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:56.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:12:56.464 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:56.494 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:12:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:56.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:12:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:56.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:56.648 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:56.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:12:56.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:12:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:57.070 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:12:57.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:57.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:12:57.177 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:57.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:57.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:57.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:57 smithi040 ceph-mon[39581]: pgmap v531: 27 pgs: 27 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.69k op/s 2024-04-29T14:12:57.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:57 smithi040 ceph-mon[39581]: osdmap e185: 12 total, 12 up, 12 in 2024-04-29T14:12:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:57 smithi121 ceph-mon[44533]: pgmap v531: 27 pgs: 27 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.69k op/s 2024-04-29T14:12:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:57 smithi121 ceph-mon[44533]: osdmap e185: 12 total, 12 up, 12 in 2024-04-29T14:12:57.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:57.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:12:57.427 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:57 smithi194 ceph-mon[44089]: pgmap v531: 27 pgs: 27 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.69k op/s 2024-04-29T14:12:57.427 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:57 smithi194 ceph-mon[44089]: osdmap e185: 12 total, 12 up, 12 in 2024-04-29T14:12:57.432 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:57.460 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:57.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:57.567 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:12:57.622 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:57.644 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:12:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:57.647 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:12:57.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:57.803 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:57.835 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:57.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:57.944 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:57.988 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:58.013 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:58.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:58.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:12:58.163 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:58.190 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:12:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:58.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:12:58.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:12:58.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:58.516 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:58.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:58.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:12:58.692 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:58.745 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:12:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:58.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:12:58.856 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:12:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:59.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:12:59.133 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:12:59.174 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:59.201 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:12:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:12:59.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:12:59.248 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:12:59.249 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:12:59.249 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:12:59.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 1 2024-04-29T14:12:59.286 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:12:59 smithi040 ceph-mon[39581]: pgmap v533: 27 pgs: 27 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-29T14:12:59.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:12:59.354 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:12:59.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:12:59 smithi121 ceph-mon[44533]: pgmap v533: 27 pgs: 27 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-29T14:12:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:12:59 smithi194 ceph-mon[44089]: pgmap v533: 27 pgs: 27 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-29T14:12:59.754 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:12:59.755 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:12:59.980 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:00.014 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:00.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:00.099 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:13:00.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:00.158 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:00.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:00.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:00.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:00.327 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:00.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:00.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:00.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1987721961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:00.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:00 smithi121 ceph-mon[44533]: from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1987721961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:00 smithi040 ceph-mon[39581]: from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:00.457 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:00.495 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1987721961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:00.495 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:00 smithi194 ceph-mon[44089]: from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:00.504 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:00.534 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:00.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:00.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:00.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:13:00.694 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:01.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:01 smithi040 ceph-mon[39581]: pgmap v534: 27 pgs: 27 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:13:01.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:01 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:01.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:01 smithi040 ceph-mon[39581]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:01.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:01 smithi040 ceph-mon[39581]: osdmap e186: 12 total, 12 up, 12 in 2024-04-29T14:13:01.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:01 smithi121 ceph-mon[44533]: pgmap v534: 27 pgs: 27 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:13:01.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:01 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:01.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:01 smithi121 ceph-mon[44533]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:01.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:01 smithi121 ceph-mon[44533]: osdmap e186: 12 total, 12 up, 12 in 2024-04-29T14:13:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:01 smithi194 ceph-mon[44089]: pgmap v534: 27 pgs: 27 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:13:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:01 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:01 smithi194 ceph-mon[44089]: from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:01 smithi194 ceph-mon[44089]: osdmap e186: 12 total, 12 up, 12 in 2024-04-29T14:13:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:01.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:01.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:01.812 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:01.843 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:01.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:01.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:01.984 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:02.011 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:02.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:02.118 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:02.159 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:02.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:02.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:02.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:02.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:02.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:02.516 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:02.529 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:02.552 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:02.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:02.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:02.696 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:02.834 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:02.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:02.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:02.985 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:02 smithi040 ceph-mon[39581]: pgmap v536: 27 pgs: 27 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.52k op/s 2024-04-29T14:13:02.986 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:02 smithi040 ceph-mon[39581]: osdmap e187: 12 total, 12 up, 12 in 2024-04-29T14:13:03.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:02 smithi194 ceph-mon[44089]: pgmap v536: 27 pgs: 27 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.52k op/s 2024-04-29T14:13:03.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:02 smithi194 ceph-mon[44089]: osdmap e187: 12 total, 12 up, 12 in 2024-04-29T14:13:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:02 smithi121 ceph-mon[44533]: pgmap v536: 27 pgs: 27 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.52k op/s 2024-04-29T14:13:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:02 smithi121 ceph-mon[44533]: osdmap e187: 12 total, 12 up, 12 in 2024-04-29T14:13:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:03.746 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:03.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:03.765 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:13:03.804 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:13:03.842 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:13:03.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:03.867 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:13:03.889 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:03.897 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:13:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:03.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:03.923 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:13:03.961 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:13:03.992 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:13:04.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:04.031 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:13:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:04.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:13:04.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:13:04.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:04.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:04.101 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:13:04.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1392014595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:04.139 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:13:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:13:04.142 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:13:04.142 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:13:04.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:04.242 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1392014595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:04.246 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:04.268 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:04.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:04.372 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:04.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1392014595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:04.413 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:04.453 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:04.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:04.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:04.618 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:04.884 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:04.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:04.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:05.032 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:05.056 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:05.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:05.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:05.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: pgmap v538: 27 pgs: 27 active+clean; 989 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osdmap e188: 12 total, 12 up, 12 in 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.260 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.260 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.261 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:05.298 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:05.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: pgmap v538: 27 pgs: 27 active+clean; 989 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:05.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osdmap e188: 12 total, 12 up, 12 in 2024-04-29T14:13:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: pgmap v538: 27 pgs: 27 active+clean; 989 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:13:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osdmap e188: 12 total, 12 up, 12 in 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:05.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:05.474 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:05.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:05 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:13:05.688+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e188 definitely_dead 0 2024-04-29T14:13:06.052 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.1. 2024-04-29T14:13:06.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/875137156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:13:06.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:06 smithi194 ceph-mon[44089]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:13:06.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:06 smithi194 ceph-mon[44089]: Client client.admin marked osd.1 out, while it was still marked up 2024-04-29T14:13:06.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/875137156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:13:06.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:06 smithi040 ceph-mon[39581]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:13:06.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:06 smithi040 ceph-mon[39581]: Client client.admin marked osd.1 out, while it was still marked up 2024-04-29T14:13:06.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/875137156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:13:06.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:06 smithi121 ceph-mon[44533]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:13:06.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:06 smithi121 ceph-mon[44533]: Client client.admin marked osd.1 out, while it was still marked up 2024-04-29T14:13:06.856 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:06.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:06.891 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:06.961 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:06.994 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:07.015 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:07.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:07.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:07 smithi040 ceph-mon[39581]: pgmap v540: 27 pgs: 27 active+clean; 716 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.68k op/s 2024-04-29T14:13:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:07 smithi040 ceph-mon[39581]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-29T14:13:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:07 smithi040 ceph-mon[39581]: osdmap e189: 12 total, 12 up, 11 in 2024-04-29T14:13:07.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:07 smithi040 ceph-mon[39581]: osdmap e190: 12 total, 12 up, 11 in 2024-04-29T14:13:07.119 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:07.150 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:07.170 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:07.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:07.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:07.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:07 smithi121 ceph-mon[44533]: pgmap v540: 27 pgs: 27 active+clean; 716 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.68k op/s 2024-04-29T14:13:07.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:07 smithi121 ceph-mon[44533]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-29T14:13:07.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:07 smithi121 ceph-mon[44533]: osdmap e189: 12 total, 12 up, 11 in 2024-04-29T14:13:07.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:07 smithi121 ceph-mon[44533]: osdmap e190: 12 total, 12 up, 11 in 2024-04-29T14:13:07.305 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:07.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:07 smithi194 ceph-mon[44089]: pgmap v540: 27 pgs: 27 active+clean; 716 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.68k op/s 2024-04-29T14:13:07.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:07 smithi194 ceph-mon[44089]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-29T14:13:07.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:07 smithi194 ceph-mon[44089]: osdmap e189: 12 total, 12 up, 11 in 2024-04-29T14:13:07.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:07 smithi194 ceph-mon[44089]: osdmap e190: 12 total, 12 up, 11 in 2024-04-29T14:13:07.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:07.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:07.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:07.465 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:07.815 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:07.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:07.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:13:07.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:07.951 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:07.977 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:07.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:08.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:08.109 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:08.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:08.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:08.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:08.277 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:08.933 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:08.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:08.991 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:13:08.991 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:13:08.991 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:13:08.991 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:13:08.991 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:13:09.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:09.092 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: pgmap v543: 27 pgs: 1 unknown, 26 active+clean; 638 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: 2.15 starting backfill to osd.8 from (0'0,0'0] MAX to 188'15033 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: 2.2 starting backfill to osd.6 from (0'0,0'0] MAX to 188'14941 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: osdmap e191: 12 total, 12 up, 11 in 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: 2.4 starting backfill to osd.10 from (0'0,0'0] MAX to 188'14974 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: 2.6 starting backfill to osd.3 from (0'0,0'0] MAX to 188'14844 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: 2.10 starting backfill to osd.9 from (0'0,0'0] MAX to 188'15316 2024-04-29T14:13:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:08 smithi194 ceph-mon[44089]: 2.9 starting backfill to osd.3 from (0'0,0'0] MAX to 188'19661 2024-04-29T14:13:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: pgmap v543: 27 pgs: 1 unknown, 26 active+clean; 638 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:13:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: 2.15 starting backfill to osd.8 from (0'0,0'0] MAX to 188'15033 2024-04-29T14:13:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: 2.2 starting backfill to osd.6 from (0'0,0'0] MAX to 188'14941 2024-04-29T14:13:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: osdmap e191: 12 total, 12 up, 11 in 2024-04-29T14:13:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: 2.4 starting backfill to osd.10 from (0'0,0'0] MAX to 188'14974 2024-04-29T14:13:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: 2.6 starting backfill to osd.3 from (0'0,0'0] MAX to 188'14844 2024-04-29T14:13:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: 2.10 starting backfill to osd.9 from (0'0,0'0] MAX to 188'15316 2024-04-29T14:13:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:08 smithi040 ceph-mon[39581]: 2.9 starting backfill to osd.3 from (0'0,0'0] MAX to 188'19661 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: pgmap v543: 27 pgs: 1 unknown, 26 active+clean; 638 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: 2.15 starting backfill to osd.8 from (0'0,0'0] MAX to 188'15033 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: 2.2 starting backfill to osd.6 from (0'0,0'0] MAX to 188'14941 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: osdmap e191: 12 total, 12 up, 11 in 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: 2.4 starting backfill to osd.10 from (0'0,0'0] MAX to 188'14974 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: 2.6 starting backfill to osd.3 from (0'0,0'0] MAX to 188'14844 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: 2.10 starting backfill to osd.9 from (0'0,0'0] MAX to 188'15316 2024-04-29T14:13:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:08 smithi121 ceph-mon[44533]: 2.9 starting backfill to osd.3 from (0'0,0'0] MAX to 188'19661 2024-04-29T14:13:09.429 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:09.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:09.534 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:09.586 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:09.609 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:09.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:09.714 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:09.756 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:09.785 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:09.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:09.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:09.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:09.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:09.984 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:09 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:10.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:10.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:09 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:10.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:09 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:11.248 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:10 smithi040 ceph-mon[39581]: pgmap v545: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 unknown, 24 active+clean; 588 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 37/18810 objects misplaced (0.197%); 5.4 MiB/s, 86 objects/s recovering 2024-04-29T14:13:11.249 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:11.249 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:11.249 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:10 smithi040 ceph-mon[39581]: osdmap e192: 12 total, 12 up, 11 in 2024-04-29T14:13:11.249 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:11.249 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:11.249 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:11.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:13:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:10 smithi194 ceph-mon[44089]: pgmap v545: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 unknown, 24 active+clean; 588 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 37/18810 objects misplaced (0.197%); 5.4 MiB/s, 86 objects/s recovering 2024-04-29T14:13:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:10 smithi194 ceph-mon[44089]: osdmap e192: 12 total, 12 up, 11 in 2024-04-29T14:13:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:10 smithi121 ceph-mon[44533]: pgmap v545: 27 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 unknown, 24 active+clean; 588 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 37/18810 objects misplaced (0.197%); 5.4 MiB/s, 86 objects/s recovering 2024-04-29T14:13:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:10 smithi121 ceph-mon[44533]: osdmap e192: 12 total, 12 up, 11 in 2024-04-29T14:13:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:11.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:11.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:11.754 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:11.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:11.886 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:11.892 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:13:11.892 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:13:11.892 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:13:11.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 6 2024-04-29T14:13:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:11.988 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:11.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:12.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:12.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:12 smithi121 ceph-mon[44533]: osdmap e193: 12 total, 12 up, 11 in 2024-04-29T14:13:12.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:12.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:12.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:12.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:12.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:12 smithi040 ceph-mon[39581]: osdmap e193: 12 total, 12 up, 11 in 2024-04-29T14:13:12.295 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:12.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:12 smithi194 ceph-mon[44089]: osdmap e193: 12 total, 12 up, 11 in 2024-04-29T14:13:12.651 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:12.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:12.669 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:12.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:13.027 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:13:13.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:13 smithi194 ceph-mon[44089]: pgmap v548: 27 pgs: 1 peering, 3 active+clean+remapped, 1 active+remapped+backfilling, 22 active+clean; 438 MiB data, 2.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 804 op/s; 37/14006 objects misplaced (0.264%); 20 MiB/s, 321 objects/s recovering 2024-04-29T14:13:13.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2385802007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:13:13.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:13 smithi040 ceph-mon[39581]: pgmap v548: 27 pgs: 1 peering, 3 active+clean+remapped, 1 active+remapped+backfilling, 22 active+clean; 438 MiB data, 2.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 804 op/s; 37/14006 objects misplaced (0.264%); 20 MiB/s, 321 objects/s recovering 2024-04-29T14:13:13.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2385802007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:13:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:13 smithi121 ceph-mon[44533]: pgmap v548: 27 pgs: 1 peering, 3 active+clean+remapped, 1 active+remapped+backfilling, 22 active+clean; 438 MiB data, 2.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 804 op/s; 37/14006 objects misplaced (0.264%); 20 MiB/s, 321 objects/s recovering 2024-04-29T14:13:13.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2385802007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:13:14.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:14 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:14.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2385802007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:13:14.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:14 smithi040 ceph-mon[39581]: osdmap e194: 12 total, 12 up, 11 in 2024-04-29T14:13:14.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:14 smithi040 ceph-mon[39581]: 2.15 scrub starts 2024-04-29T14:13:14.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:14 smithi040 ceph-mon[39581]: 2.15 scrub ok 2024-04-29T14:13:14.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:14 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:14.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2385802007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:13:14.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:14 smithi194 ceph-mon[44089]: osdmap e194: 12 total, 12 up, 11 in 2024-04-29T14:13:14.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:14 smithi194 ceph-mon[44089]: 2.15 scrub starts 2024-04-29T14:13:14.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:14 smithi194 ceph-mon[44089]: 2.15 scrub ok 2024-04-29T14:13:14.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:14 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:14.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2385802007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:13:14.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:14 smithi121 ceph-mon[44533]: osdmap e194: 12 total, 12 up, 11 in 2024-04-29T14:13:14.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:14 smithi121 ceph-mon[44533]: 2.15 scrub starts 2024-04-29T14:13:14.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:14 smithi121 ceph-mon[44533]: 2.15 scrub ok 2024-04-29T14:13:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:15.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:15.168 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:15.220 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:15.250 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:15.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:15 smithi194 ceph-mon[44089]: pgmap v550: 27 pgs: 1 peering, 3 active+clean+remapped, 1 active+remapped+backfilling, 22 active+clean; 411 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 37/13144 objects misplaced (0.281%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:13:15.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:15 smithi040 ceph-mon[39581]: pgmap v550: 27 pgs: 1 peering, 3 active+clean+remapped, 1 active+remapped+backfilling, 22 active+clean; 411 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 37/13144 objects misplaced (0.281%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:13:15.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:15 smithi121 ceph-mon[44533]: pgmap v550: 27 pgs: 1 peering, 3 active+clean+remapped, 1 active+remapped+backfilling, 22 active+clean; 411 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 37/13144 objects misplaced (0.281%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:13:15.399 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:15.478 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:15.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:15.584 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:15.615 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:15.750 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:15.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:15.807 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:13:15.834 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:13:15.854 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:13:15.856 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:15.887 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:13:15.898 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:15.918 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:13:15.935 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:15.942 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:13:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:15.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:15.979 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:13:16.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:13:16.034 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:13:16.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:16.076 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:13:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:16.106 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:13:16.135 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:13:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:13:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:13:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:13:16.184 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 164258 objects 2024-04-29T14:13:16.184 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :142.808 2024-04-29T14:13:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:16 smithi194 ceph-mon[44089]: osdmap e195: 12 total, 12 up, 11 in 2024-04-29T14:13:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:16 smithi194 ceph-mon[44089]: 2.13 scrub starts 2024-04-29T14:13:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:16 smithi194 ceph-mon[44089]: 2.13 scrub ok 2024-04-29T14:13:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4083514398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:16 smithi040 ceph-mon[39581]: osdmap e195: 12 total, 12 up, 11 in 2024-04-29T14:13:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:16 smithi040 ceph-mon[39581]: 2.13 scrub starts 2024-04-29T14:13:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:16 smithi040 ceph-mon[39581]: 2.13 scrub ok 2024-04-29T14:13:16.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4083514398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:16 smithi121 ceph-mon[44533]: osdmap e195: 12 total, 12 up, 11 in 2024-04-29T14:13:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:16 smithi121 ceph-mon[44533]: 2.13 scrub starts 2024-04-29T14:13:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:16 smithi121 ceph-mon[44533]: 2.13 scrub ok 2024-04-29T14:13:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4083514398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: pgmap v552: 27 pgs: 1 peering, 26 active+clean; 173 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 5.8 MiB/s, 93 objects/s recovering 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: 2.12 scrub starts 2024-04-29T14:13:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: 2.12 scrub ok 2024-04-29T14:13:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: 2.9 scrub starts 2024-04-29T14:13:17.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:17 smithi040 ceph-mon[39581]: 2.9 scrub ok 2024-04-29T14:13:17.206 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:13:17.206 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:13:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: pgmap v552: 27 pgs: 1 peering, 26 active+clean; 173 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 5.8 MiB/s, 93 objects/s recovering 2024-04-29T14:13:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: 2.12 scrub starts 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: 2.12 scrub ok 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: 2.9 scrub starts 2024-04-29T14:13:17.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:17 smithi194 ceph-mon[44089]: 2.9 scrub ok 2024-04-29T14:13:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: pgmap v552: 27 pgs: 1 peering, 26 active+clean; 173 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 5.8 MiB/s, 93 objects/s recovering 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: 2.12 scrub starts 2024-04-29T14:13:17.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: 2.12 scrub ok 2024-04-29T14:13:17.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: 2.9 scrub starts 2024-04-29T14:13:17.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:17 smithi121 ceph-mon[44533]: 2.9 scrub ok 2024-04-29T14:13:17.507 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:13:17.934 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:13:17.957 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:17.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:18.061 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:18.093 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:18.113 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:18.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:18.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:18.249 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:18.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.f scrub starts 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.f scrub ok 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.11 scrub starts 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.11 scrub ok 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.19 scrub starts 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.19 scrub ok 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.16 scrub starts 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.16 scrub ok 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.4 scrub starts 2024-04-29T14:13:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:18 smithi194 ceph-mon[44089]: 2.4 scrub ok 2024-04-29T14:13:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.f scrub starts 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.f scrub ok 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.11 scrub starts 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.11 scrub ok 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.19 scrub starts 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.19 scrub ok 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.16 scrub starts 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.16 scrub ok 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.4 scrub starts 2024-04-29T14:13:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:18 smithi040 ceph-mon[39581]: 2.4 scrub ok 2024-04-29T14:13:18.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.f scrub starts 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.f scrub ok 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.11 scrub starts 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.11 scrub ok 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.19 scrub starts 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.19 scrub ok 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.16 scrub starts 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.16 scrub ok 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.4 scrub starts 2024-04-29T14:13:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:18 smithi121 ceph-mon[44533]: 2.4 scrub ok 2024-04-29T14:13:18.404 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:18.946 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:18.953 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:18.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:19.057 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:19.176 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:19 smithi040 ceph-mon[39581]: pgmap v553: 27 pgs: 27 active+clean; 83 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 892 op/s; 2.6 MiB/s, 40 objects/s recovering 2024-04-29T14:13:19.176 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:19 smithi040 ceph-mon[39581]: 2.18 scrub starts 2024-04-29T14:13:19.176 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:19 smithi040 ceph-mon[39581]: 2.18 scrub ok 2024-04-29T14:13:19.188 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:19.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:19.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:19 smithi194 ceph-mon[44089]: pgmap v553: 27 pgs: 27 active+clean; 83 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 892 op/s; 2.6 MiB/s, 40 objects/s recovering 2024-04-29T14:13:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:19 smithi194 ceph-mon[44089]: 2.18 scrub starts 2024-04-29T14:13:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:19 smithi194 ceph-mon[44089]: 2.18 scrub ok 2024-04-29T14:13:19.321 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:19 smithi121 ceph-mon[44533]: pgmap v553: 27 pgs: 27 active+clean; 83 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 892 op/s; 2.6 MiB/s, 40 objects/s recovering 2024-04-29T14:13:19.321 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:19 smithi121 ceph-mon[44533]: 2.18 scrub starts 2024-04-29T14:13:19.321 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:19 smithi121 ceph-mon[44533]: 2.18 scrub ok 2024-04-29T14:13:19.323 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:19.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:19.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:19.448 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:19.480 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:19.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:19.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:19.658 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:19.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:19.762 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:19.794 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:19.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:20.057 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:20 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:13:20.016+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e195 definitely_dead 0 2024-04-29T14:13:20.077 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-29T14:13:20.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-29T14:13:20.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:20.124 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:20.144 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:20.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.6. 2024-04-29T14:13:20.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:20.280 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:20.300 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:20.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:20 smithi194 ceph-mon[44089]: 2.14 scrub starts 2024-04-29T14:13:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:20 smithi194 ceph-mon[44089]: 2.14 scrub ok 2024-04-29T14:13:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:20 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:13:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:20 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:13:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2239338318' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-29T14:13:20.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:20 smithi194 ceph-mon[44089]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-29T14:13:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:20 smithi040 ceph-mon[39581]: 2.14 scrub starts 2024-04-29T14:13:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:20 smithi040 ceph-mon[39581]: 2.14 scrub ok 2024-04-29T14:13:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:20 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:13:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:20 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:13:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2239338318' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-29T14:13:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:20 smithi040 ceph-mon[39581]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-29T14:13:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:20 smithi121 ceph-mon[44533]: 2.14 scrub starts 2024-04-29T14:13:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:20 smithi121 ceph-mon[44533]: 2.14 scrub ok 2024-04-29T14:13:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:20 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:13:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:20 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:13:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2239338318' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-29T14:13:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:20 smithi121 ceph-mon[44533]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-29T14:13:20.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:20.436 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:20.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:20.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:20.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:13:20.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:20.589 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:20.608 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:20.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:20.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:20.740 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:20.837 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:20.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:20.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:20.970 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:21.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: 2.10 scrub starts 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: 2.10 scrub ok 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: pgmap v554: 27 pgs: 27 active+clean; 48 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 288 KiB/s, 4 objects/s recovering 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: 2.b scrub starts 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: 2.b scrub ok 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2239338318' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: osdmap e196: 12 total, 12 up, 10 in 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: 2.3 scrub starts 2024-04-29T14:13:21.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:21 smithi040 ceph-mon[39581]: 2.3 scrub ok 2024-04-29T14:13:21.168 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:21.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:21.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: 2.10 scrub starts 2024-04-29T14:13:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: 2.10 scrub ok 2024-04-29T14:13:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: pgmap v554: 27 pgs: 27 active+clean; 48 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 288 KiB/s, 4 objects/s recovering 2024-04-29T14:13:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: 2.b scrub starts 2024-04-29T14:13:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: 2.b scrub ok 2024-04-29T14:13:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2239338318' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-29T14:13:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: osdmap e196: 12 total, 12 up, 10 in 2024-04-29T14:13:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: 2.3 scrub starts 2024-04-29T14:13:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:21 smithi121 ceph-mon[44533]: 2.3 scrub ok 2024-04-29T14:13:21.449 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:21.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:21.554 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: 2.10 scrub starts 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: 2.10 scrub ok 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: pgmap v554: 27 pgs: 27 active+clean; 48 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 288 KiB/s, 4 objects/s recovering 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: 2.b scrub starts 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: 2.b scrub ok 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2239338318' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: osdmap e196: 12 total, 12 up, 10 in 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: 2.3 scrub starts 2024-04-29T14:13:21.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:21 smithi194 ceph-mon[44089]: 2.3 scrub ok 2024-04-29T14:13:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:21.732 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:21.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:21.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:21.868 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:22.068 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: osdmap e197: 12 total, 12 up, 10 in 2024-04-29T14:13:22.068 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:13:22.068 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:13:22.068 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:13:22.068 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:22.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:22 smithi040 ceph-mon[39581]: osdmap e198: 12 total, 12 up, 10 in 2024-04-29T14:13:22.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: osdmap e197: 12 total, 12 up, 10 in 2024-04-29T14:13:22.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:13:22.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:13:22.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:13:22.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:22 smithi121 ceph-mon[44533]: osdmap e198: 12 total, 12 up, 10 in 2024-04-29T14:13:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: osdmap e197: 12 total, 12 up, 10 in 2024-04-29T14:13:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:13:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:13:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:13:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:13:22.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:22 smithi194 ceph-mon[44089]: osdmap e198: 12 total, 12 up, 10 in 2024-04-29T14:13:23.073 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:13:23.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: pgmap v557: 27 pgs: 2 unknown, 25 active+clean; 5.8 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 778 op/s 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: 2.19 starting backfill to osd.0 from (0'0,0'0] MAX to 195'16242 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: 2.17 starting backfill to osd.2 from (0'0,0'0] MAX to 195'19551 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 195'15530 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: 2.11 starting backfill to osd.9 from (0'0,0'0] MAX to 195'15452 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: 2.14 starting backfill to osd.5 from (0'0,0'0] MAX to 195'15424 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: 2.16 starting backfill to osd.11 from (0'0,0'0] MAX to 195'15195 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: 2.2 starting backfill to osd.9 from (0'0,0'0] MAX to 195'15236 2024-04-29T14:13:23.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3322097264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:23.335 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: pgmap v557: 27 pgs: 2 unknown, 25 active+clean; 5.8 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 778 op/s 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: 2.19 starting backfill to osd.0 from (0'0,0'0] MAX to 195'16242 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: 2.17 starting backfill to osd.2 from (0'0,0'0] MAX to 195'19551 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 195'15530 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: 2.11 starting backfill to osd.9 from (0'0,0'0] MAX to 195'15452 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: 2.14 starting backfill to osd.5 from (0'0,0'0] MAX to 195'15424 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: 2.16 starting backfill to osd.11 from (0'0,0'0] MAX to 195'15195 2024-04-29T14:13:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: 2.2 starting backfill to osd.9 from (0'0,0'0] MAX to 195'15236 2024-04-29T14:13:23.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3322097264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:23.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: pgmap v557: 27 pgs: 2 unknown, 25 active+clean; 5.8 MiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 778 op/s 2024-04-29T14:13:23.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: 2.19 starting backfill to osd.0 from (0'0,0'0] MAX to 195'16242 2024-04-29T14:13:23.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: 2.17 starting backfill to osd.2 from (0'0,0'0] MAX to 195'19551 2024-04-29T14:13:23.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: 2.7 starting backfill to osd.2 from (0'0,0'0] MAX to 195'15530 2024-04-29T14:13:23.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: 2.11 starting backfill to osd.9 from (0'0,0'0] MAX to 195'15452 2024-04-29T14:13:23.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: 2.14 starting backfill to osd.5 from (0'0,0'0] MAX to 195'15424 2024-04-29T14:13:23.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: 2.16 starting backfill to osd.11 from (0'0,0'0] MAX to 195'15195 2024-04-29T14:13:23.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: 2.2 starting backfill to osd.9 from (0'0,0'0] MAX to 195'15236 2024-04-29T14:13:23.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3322097264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:24.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:24.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:24.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:24.154 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:24.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:24.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:24.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:24 smithi040 ceph-mon[39581]: 2.c scrub starts 2024-04-29T14:13:24.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:24 smithi040 ceph-mon[39581]: 2.c scrub ok 2024-04-29T14:13:24.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:24 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:13:24.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:24 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:13:24.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3322097264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:24.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:24 smithi040 ceph-mon[39581]: osdmap e199: 12 total, 12 up, 10 in 2024-04-29T14:13:24.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:24 smithi040 ceph-mon[39581]: osdmap e200: 12 total, 12 up, 10 in 2024-04-29T14:13:24.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:24.306 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:24.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:24.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:24 smithi121 ceph-mon[44533]: 2.c scrub starts 2024-04-29T14:13:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:24 smithi121 ceph-mon[44533]: 2.c scrub ok 2024-04-29T14:13:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:24 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:13:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:24 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:13:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3322097264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:24 smithi121 ceph-mon[44533]: osdmap e199: 12 total, 12 up, 10 in 2024-04-29T14:13:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:24 smithi121 ceph-mon[44533]: osdmap e200: 12 total, 12 up, 10 in 2024-04-29T14:13:24.430 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:24.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:24 smithi194 ceph-mon[44089]: 2.c scrub starts 2024-04-29T14:13:24.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:24 smithi194 ceph-mon[44089]: 2.c scrub ok 2024-04-29T14:13:24.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:24 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:13:24.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:24 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:13:24.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3322097264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:24.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:24 smithi194 ceph-mon[44089]: osdmap e199: 12 total, 12 up, 10 in 2024-04-29T14:13:24.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:24 smithi194 ceph-mon[44089]: osdmap e200: 12 total, 12 up, 10 in 2024-04-29T14:13:24.462 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:24.483 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:24.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:24.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:24.614 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:24.633 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:24.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:24.737 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:24.768 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:24.822 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:24.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:24.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:24.958 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:24.979 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:24.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:25.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:25.110 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:25.128 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:25.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:25.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:25.258 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:25.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:25 smithi194 ceph-mon[44089]: pgmap v560: 27 pgs: 2 activating+remapped, 2 unknown, 23 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 187 op/s 2024-04-29T14:13:25.342 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:13:25.343 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:13:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:25 smithi040 ceph-mon[39581]: pgmap v560: 27 pgs: 2 activating+remapped, 2 unknown, 23 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 187 op/s 2024-04-29T14:13:25.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:25 smithi121 ceph-mon[44533]: pgmap v560: 27 pgs: 2 activating+remapped, 2 unknown, 23 active+clean; 577 KiB data, 1.8 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 187 op/s 2024-04-29T14:13:25.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:25.406 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:25.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:25.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:25.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:25.560 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:25.581 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:25.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:25.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:25.906 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:25.917 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:25.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:26.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:26.048 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:26.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:13:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:26.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:26.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:26.198 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:26.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:26.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:26.320 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:26.350 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:26 smithi194 ceph-mon[44089]: 2.a scrub starts 2024-04-29T14:13:26.350 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:26 smithi194 ceph-mon[44089]: 2.a scrub ok 2024-04-29T14:13:26.352 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:26 smithi040 ceph-mon[39581]: 2.a scrub starts 2024-04-29T14:13:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:26 smithi040 ceph-mon[39581]: 2.a scrub ok 2024-04-29T14:13:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:26 smithi121 ceph-mon[44533]: 2.a scrub starts 2024-04-29T14:13:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:26 smithi121 ceph-mon[44533]: 2.a scrub ok 2024-04-29T14:13:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:26.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:26.477 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:26.508 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:26.528 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:26.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:26.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:26.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:26.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:26.816 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:26.837 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:26.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:26.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:27.257 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:27 smithi040 ceph-mon[39581]: pgmap v562: 27 pgs: 1 peering, 2 activating+remapped, 1 unknown, 23 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 18 op/s 2024-04-29T14:13:27.257 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:27 smithi040 ceph-mon[39581]: 2.e scrub starts 2024-04-29T14:13:27.257 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:27 smithi040 ceph-mon[39581]: 2.e scrub ok 2024-04-29T14:13:27.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:13:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:27 smithi121 ceph-mon[44533]: pgmap v562: 27 pgs: 1 peering, 2 activating+remapped, 1 unknown, 23 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 18 op/s 2024-04-29T14:13:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:27 smithi121 ceph-mon[44533]: 2.e scrub starts 2024-04-29T14:13:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:27 smithi121 ceph-mon[44533]: 2.e scrub ok 2024-04-29T14:13:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:27.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:27 smithi194 ceph-mon[44089]: pgmap v562: 27 pgs: 1 peering, 2 activating+remapped, 1 unknown, 23 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 18 op/s 2024-04-29T14:13:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:27 smithi194 ceph-mon[44089]: 2.e scrub starts 2024-04-29T14:13:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:27 smithi194 ceph-mon[44089]: 2.e scrub ok 2024-04-29T14:13:27.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:27.686 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:27.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:27.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:27.820 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:28.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 10, 11] out_osds: [1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:13:28.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:13:28.375 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:13:28.806 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:28.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:28.824 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:28.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:28.939 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:29.089 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:29.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_0' removed 2024-04-29T14:13:29.194 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:29.220 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:29.242 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:29.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:29.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:29 smithi040 ceph-mon[39581]: pgmap v563: 27 pgs: 1 peering, 2 activating+remapped, 24 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 13 op/s 2024-04-29T14:13:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:29 smithi040 ceph-mon[39581]: osdmap e201: 12 total, 12 up, 10 in 2024-04-29T14:13:29.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3759153289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:13:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:29 smithi121 ceph-mon[44533]: pgmap v563: 27 pgs: 1 peering, 2 activating+remapped, 24 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 13 op/s 2024-04-29T14:13:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:29 smithi121 ceph-mon[44533]: osdmap e201: 12 total, 12 up, 10 in 2024-04-29T14:13:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3759153289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:13:29.381 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:29.481 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:29.482 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:29.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:29.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:29.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:29.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:29 smithi194 ceph-mon[44089]: pgmap v563: 27 pgs: 1 peering, 2 activating+remapped, 24 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail; 0 B/s wr, 13 op/s 2024-04-29T14:13:29.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:29 smithi194 ceph-mon[44089]: osdmap e201: 12 total, 12 up, 10 in 2024-04-29T14:13:29.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3759153289' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:13:29.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:30.096 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:13:30.097 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:13:30.097 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-29T14:13:30.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3759153289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: osdmap e202: 12 total, 12 up, 10 in 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2973158034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3759153289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:13:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: osdmap e202: 12 total, 12 up, 10 in 2024-04-29T14:13:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2973158034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:13:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3759153289' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: osdmap e202: 12 total, 12 up, 10 in 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2973158034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:30.634 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:31.065 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:31.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:31.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: pgmap v566: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: pgmap v566: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: pgmap v566: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:32.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:32.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:32.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:32 smithi194 ceph-mon[44089]: osdmap e203: 12 total, 12 up, 10 in 2024-04-29T14:13:32.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:32.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:32.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:32 smithi040 ceph-mon[39581]: osdmap e203: 12 total, 12 up, 10 in 2024-04-29T14:13:32.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:32.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:32.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:32 smithi121 ceph-mon[44533]: osdmap e203: 12 total, 12 up, 10 in 2024-04-29T14:13:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:32.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:32.756 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:32.789 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:32.812 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:32.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:32.916 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:32.948 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:32.968 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:32.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:33.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:33.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:33 smithi040 ceph-mon[39581]: pgmap v568: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:33.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:13:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:33 smithi194 ceph-mon[44089]: pgmap v568: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:33 smithi121 ceph-mon[44533]: pgmap v568: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:33.738 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:13:33.738 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:13:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:34.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:34 smithi194 ceph-mon[44089]: osdmap e204: 12 total, 12 up, 10 in 2024-04-29T14:13:34.591 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:34.591 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:34.591 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:34 smithi040 ceph-mon[39581]: osdmap e204: 12 total, 12 up, 10 in 2024-04-29T14:13:34.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:34.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:34.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:34 smithi121 ceph-mon[44533]: osdmap e204: 12 total, 12 up, 10 in 2024-04-29T14:13:35.248 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:35 smithi040 ceph-mon[39581]: pgmap v570: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:35.248 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:35 smithi040 ceph-mon[39581]: from='client.15105 v1:172.21.15.40:0/3735085260' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:13:35.491 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:35.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:35 smithi194 ceph-mon[44089]: pgmap v570: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:35 smithi194 ceph-mon[44089]: from='client.15105 v1:172.21.15.40:0/3735085260' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:13:35.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:35.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:35 smithi121 ceph-mon[44533]: pgmap v570: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:35.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:35 smithi121 ceph-mon[44533]: from='client.15105 v1:172.21.15.40:0/3735085260' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:13:35.629 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:35.650 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:35.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:35.757 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:35.786 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:35.805 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:35.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:35.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:35.941 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:35.952 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:13:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:13:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:13:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:13:35.952 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:13:36.153 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:13:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:36.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:36.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:36.327 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:36.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:36.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:36.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2567992193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:13:36.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:36.482 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:36.534 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:36.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2567992193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:13:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2567992193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:13:36.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:36.673 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:37 smithi040 ceph-mon[39581]: pgmap v571: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:37 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2567992193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:13:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:37 smithi040 ceph-mon[39581]: osdmap e205: 12 total, 12 up, 10 in 2024-04-29T14:13:37.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:37.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:37.453 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":570,"stamp":"2024-04-29T14:13:33.513808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":937451520,"kb_used":1705976,"kb_used_data":393252,"kb_used_omap":15,"kb_used_meta":1312560,"kb_avail":935745544,"statfs":{"total":959950356480,"available":958203437056,"internally_reserved":0,"allocated":402690048,"data_stored":398817667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15953,"internal_metadata":1344061871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-211244,"num_write_kb":-3950273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4859,"num_bytes_recovered":-318373888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412085"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":577,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-29T14:13:29.116036+0000","last_change":"2024-04-29T14:13:19.322008+0000","last_active":"2024-04-29T14:13:29.116036+0000","last_peered":"2024-04-29T14:13:29.116036+0000","last_clean":"2024-04-29T14:13:29.116036+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:13:29.116036+0000","last_undegraded":"2024-04-29T14:13:29.116036+0000","last_fullsized":"2024-04-29T14:13:29.116036+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:13:19.321844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:13:19.321844+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:14:32.984656+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108788,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636364,"statfs":{"total":95995035648,"available":95883636736,"internally_reserved":0,"allocated":1282048,"data_stored":891575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163384,"kb_used_data":37800,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93581768,"statfs":{"total":95995035648,"available":95827730432,"internally_reserved":0,"allocated":38707200,"data_stored":38313602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":128580032},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645401,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":172136,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93573016,"statfs":{"total":95995035648,"available":95818768384,"internally_reserved":0,"allocated":749568,"data_stored":362667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":175503808},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":8,"up_from":64,"seq":274877907038,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":156780,"kb_used_data":53404,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93588372,"statfs":{"total":95995035648,"available":95834492928,"internally_reserved":0,"allocated":54685696,"data_stored":54298795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":154612,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93590540,"statfs":{"total":95995035648,"available":95836712960,"internally_reserved":0,"allocated":1282048,"data_stored":891595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":157022666},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574976,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199276,"kb_used_data":105628,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93545876,"statfs":{"total":95995035648,"available":95790977024,"internally_reserved":0,"allocated":108163072,"data_stored":107776171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346043,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":112364,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":93632788,"statfs":{"total":95995035648,"available":95879974912,"internally_reserved":0,"allocated":749568,"data_stored":362667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114293189},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":209688,"kb_used_data":52172,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93535464,"statfs":{"total":95995035648,"available":95780315136,"internally_reserved":0,"allocated":53424128,"data_stored":53047677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":161282504},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986290,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204588,"kb_used_data":51228,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93540564,"statfs":{"total":95995035648,"available":95785537536,"internally_reserved":0,"allocated":52457472,"data_stored":52070571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822763,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224360,"kb_used_data":89052,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93520792,"statfs":{"total":95995035648,"available":95765291008,"internally_reserved":0,"allocated":91189248,"data_stored":90802347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:13:37.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:37.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:37 smithi194 ceph-mon[44089]: pgmap v571: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:37.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:37 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:37.559 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2567992193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:13:37.559 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:37 smithi194 ceph-mon[44089]: osdmap e205: 12 total, 12 up, 10 in 2024-04-29T14:13:37.560 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:37.581 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:37.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:37 smithi121 ceph-mon[44533]: pgmap v571: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:37 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2567992193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:13:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:37 smithi121 ceph-mon[44533]: osdmap e205: 12 total, 12 up, 10 in 2024-04-29T14:13:37.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:37.717 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:37.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:37.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:37.876 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:38.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:13:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:39.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:13:39.028 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:39.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:39.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:39.166 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:39.187 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:39.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:39.291 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:39.309 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:39 smithi040 ceph-mon[39581]: pgmap v573: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:39.322 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:39.343 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:39.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:39.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:39.477 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:39 smithi121 ceph-mon[44533]: pgmap v573: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:39.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:39.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:39 smithi194 ceph-mon[44089]: pgmap v573: 1 pgs: 1 active+clean; 577 KiB data, 1.6 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:39.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:39.655 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:39.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:39.759 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:39.791 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:39.812 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:39.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:39.916 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:39.946 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:39.966 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:39.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:40.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:40.102 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:41.052 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:41.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:41.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:41.186 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:41.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:41 smithi040 ceph-mon[39581]: pgmap v574: 1 pgs: 1 active+clean; 577 KiB data, 1.5 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:41.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:41.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:41.472 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:41 smithi121 ceph-mon[44533]: pgmap v574: 1 pgs: 1 active+clean; 577 KiB data, 1.5 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:41.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:41.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:41 smithi194 ceph-mon[44089]: pgmap v574: 1 pgs: 1 active+clean; 577 KiB data, 1.5 GiB used, 892 GiB / 894 GiB avail 2024-04-29T14:13:41.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:41.650 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:41.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:41.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:41.775 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:13:41.783 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:41.797 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:13:41.803 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:41.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:41.818 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:13:41.839 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:13:41.859 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:13:41.876 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:41.880 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:13:41.907 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:13:41.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:41.925 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:41.932 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:13:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:41.957 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:13:41.978 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:13:41.999 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:13:42.019 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:13:42.020 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:13:42.021 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:13:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:42.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_1' created 2024-04-29T14:13:42.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/962984596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:42.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:42.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1842501957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-29T14:13:42.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:42 smithi040 ceph-mon[39581]: from='client.35051 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-29T14:13:42.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:42.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:42.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:42.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/962984596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:42.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:42.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1842501957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-29T14:13:42.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:42 smithi121 ceph-mon[44533]: from='client.35051 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-29T14:13:42.434 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:42.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:42.454 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 10, 11] out_osds: [1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:13:42.454 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:13:42.454 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-29T14:13:42.454 INFO:tasks.thrashosds.thrasher:Setting osd 11 primary_affinity to 0.000000 2024-04-29T14:13:42.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 11 0 2024-04-29T14:13:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:42.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:42.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/962984596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1842501957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-29T14:13:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:42 smithi194 ceph-mon[44089]: from='client.35051 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-29T14:13:42.583 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:42.602 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:42.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:42.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:42.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:42.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:42.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:42.880 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:42.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:42.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:43.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:43.028 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:43.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:43.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:43.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:43.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:43.179 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:43.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:43.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:43.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:43.334 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.5 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: from='client.35051 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-29T14:13:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:43 smithi121 ceph-mon[44533]: osdmap e206: 12 total, 12 up, 10 in 2024-04-29T14:13:43.458 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:43.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.5 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: from='client.35051 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-29T14:13:43.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:43 smithi040 ceph-mon[39581]: osdmap e206: 12 total, 12 up, 10 in 2024-04-29T14:13:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:43.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:43.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:43.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.5 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:43.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: from='client.35051 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-29T14:13:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:43 smithi194 ceph-mon[44089]: osdmap e206: 12 total, 12 up, 10 in 2024-04-29T14:13:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:43.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:43.713 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:43.746 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:43.766 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:43.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:43.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:13:43.903 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:44.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:44 smithi194 ceph-mon[44089]: osdmap e207: 12 total, 12 up, 10 in 2024-04-29T14:13:44.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:44 smithi194 ceph-mon[44089]: pgmap v578: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.5 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:44.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:44 smithi194 ceph-mon[44089]: osdmap e208: 12 total, 12 up, 10 in 2024-04-29T14:13:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:44 smithi040 ceph-mon[39581]: osdmap e207: 12 total, 12 up, 10 in 2024-04-29T14:13:44.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:44 smithi040 ceph-mon[39581]: pgmap v578: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.5 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:44.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:44 smithi040 ceph-mon[39581]: osdmap e208: 12 total, 12 up, 10 in 2024-04-29T14:13:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:44 smithi121 ceph-mon[44533]: osdmap e207: 12 total, 12 up, 10 in 2024-04-29T14:13:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:44 smithi121 ceph-mon[44533]: pgmap v578: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.5 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:44 smithi121 ceph-mon[44533]: osdmap e208: 12 total, 12 up, 10 in 2024-04-29T14:13:45.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:45 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:13:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:13:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:45.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:13:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:45 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:13:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:45 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:13:45.644 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:13:45.644 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:13:45.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:45.692 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:45.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:45.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:45.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:45.846 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:45.866 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:45.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:45.970 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:46.024 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:46.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:46.128 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:46.157 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:46.177 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:46.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:46.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:46 smithi194 ceph-mon[44089]: osdmap e209: 12 total, 12 up, 10 in 2024-04-29T14:13:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:46.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:46 smithi194 ceph-mon[44089]: pgmap v581: 17 pgs: 6 unknown, 11 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:46.314 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:46.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:46.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:46 smithi040 ceph-mon[39581]: osdmap e209: 12 total, 12 up, 10 in 2024-04-29T14:13:46.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:46.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:46.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:46 smithi040 ceph-mon[39581]: pgmap v581: 17 pgs: 6 unknown, 11 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:46.318 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:46.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:46.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:13:46.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:46.494 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:46.494 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:46.494 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:46 smithi121 ceph-mon[44533]: osdmap e209: 12 total, 12 up, 10 in 2024-04-29T14:13:46.494 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:46.494 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:46.494 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:46 smithi121 ceph-mon[44533]: pgmap v581: 17 pgs: 6 unknown, 11 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:46.496 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:46.517 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:46.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:46.621 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:46.652 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:46.673 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:46.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:46.777 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:46.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:46.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-29T14:13:46.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:13:47.057 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:47.096 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:47.182 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:47.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:47.194 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:47.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:47.320 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:47.341 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:47.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:47.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:47 smithi040 ceph-mon[39581]: osdmap e210: 12 total, 12 up, 10 in 2024-04-29T14:13:47.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:13:47.473 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:47 smithi121 ceph-mon[44533]: osdmap e210: 12 total, 12 up, 10 in 2024-04-29T14:13:47.475 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:47.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:13:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:47.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:13:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:47 smithi194 ceph-mon[44089]: osdmap e210: 12 total, 12 up, 10 in 2024-04-29T14:13:47.599 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:47.631 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:47.651 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:47.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:47.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:47.785 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:47.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:47.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:47.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:47.939 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:48.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:48.166 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:48.190 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:13:48.199 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:48.220 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:48.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:48.324 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:48.354 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:48.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/798610865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:48 smithi040 ceph-mon[39581]: pgmap v583: 17 pgs: 17 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:48.373 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:48.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:48.478 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:48.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/798610865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:48.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:48 smithi194 ceph-mon[44089]: pgmap v583: 17 pgs: 17 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:48.506 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:48.525 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:48.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/798610865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:13:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:48 smithi121 ceph-mon[44533]: pgmap v583: 17 pgs: 17 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:48.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:48.661 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:48.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:48.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:48.786 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:48.816 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:48.836 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:48.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:48.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:48.973 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:48.993 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:48.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:49.098 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:49.128 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:49.149 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:49.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:49.253 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:49.281 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:49.300 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:49.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:49.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:49 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:49.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/798610865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:49.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:49 smithi121 ceph-mon[44533]: osdmap e211: 12 total, 12 up, 10 in 2024-04-29T14:13:49.404 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:49.429 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:49 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:49.429 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/798610865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:49.429 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:49 smithi194 ceph-mon[44089]: osdmap e211: 12 total, 12 up, 10 in 2024-04-29T14:13:49.431 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:49.450 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:49.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:49.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:49.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:49 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:13:49.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/798610865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:13:49.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:49 smithi040 ceph-mon[39581]: osdmap e211: 12 total, 12 up, 10 in 2024-04-29T14:13:49.593 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:49.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:49.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:49.758 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:49.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:49.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:49.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:49.910 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:49.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:50.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:50.062 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:50.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:50.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:50.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:50.213 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:50.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:50.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:50.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:50.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:50.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:50 smithi121 ceph-mon[44533]: osdmap e212: 12 total, 12 up, 10 in 2024-04-29T14:13:50.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:50 smithi121 ceph-mon[44533]: pgmap v586: 17 pgs: 17 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:50.362 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:50.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:50.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:50.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:50.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:50.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:50.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:50 smithi194 ceph-mon[44089]: osdmap e212: 12 total, 12 up, 10 in 2024-04-29T14:13:50.515 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:50 smithi194 ceph-mon[44089]: pgmap v586: 17 pgs: 17 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:50.517 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:50.539 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:50.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:50.574 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:13:50.574 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:13:50.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:50.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:50.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:50 smithi040 ceph-mon[39581]: osdmap e212: 12 total, 12 up, 10 in 2024-04-29T14:13:50.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:50 smithi040 ceph-mon[39581]: pgmap v586: 17 pgs: 17 active+clean; 577 KiB data, 1.4 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:50.643 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:50.672 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:50.692 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:50.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:50.750 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:50.796 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:50.824 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:50.844 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:50.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:50.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:50.978 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:50.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:51.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:51.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:51.131 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:51.150 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:51.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:51.205 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.11 primary-affinity to 0 (802) 2024-04-29T14:13:51.257 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:51.287 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:51.308 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:51.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:51.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3421197395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]: dispatch 2024-04-29T14:13:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3421197395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]: dispatch 2024-04-29T14:13:51.411 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:51.439 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:51.460 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:51.461 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3421197395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]: dispatch 2024-04-29T14:13:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:51.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:51.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:51.593 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:51.613 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:51.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:51.717 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:51.745 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:51.765 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:51.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:51.869 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:51.896 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:51.915 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:51.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:52.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:13:52.018 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:52.045 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:52.063 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:52.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:52.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:52.198 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:52.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:52.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:52.265 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3421197395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-04-29T14:13:52.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:52 smithi040 ceph-mon[39581]: osdmap e213: 12 total, 12 up, 10 in 2024-04-29T14:13:52.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:52 smithi040 ceph-mon[39581]: pgmap v588: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:52.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3421197395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-04-29T14:13:52.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:52 smithi194 ceph-mon[44089]: osdmap e213: 12 total, 12 up, 10 in 2024-04-29T14:13:52.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:52 smithi194 ceph-mon[44089]: pgmap v588: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:52.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:52.355 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:52.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:52.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:52.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:52.509 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:52.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3421197395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-04-29T14:13:52.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:52 smithi121 ceph-mon[44533]: osdmap e213: 12 total, 12 up, 10 in 2024-04-29T14:13:52.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:52 smithi121 ceph-mon[44533]: pgmap v588: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:52.699 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:52.708 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:52.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:52.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:52.841 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:52.938 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:52.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:53.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:53.074 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:53.095 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:53.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:53.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:13:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:53.253 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:13:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:53.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:13:53.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:53.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:53 smithi194 ceph-mon[44089]: osdmap e214: 12 total, 12 up, 10 in 2024-04-29T14:13:53.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:53.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:53.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:53 smithi040 ceph-mon[39581]: osdmap e214: 12 total, 12 up, 10 in 2024-04-29T14:13:53.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:53.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:53 smithi121 ceph-mon[44533]: osdmap e214: 12 total, 12 up, 10 in 2024-04-29T14:13:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:53.644 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:13:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:53.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:53.841 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:53.873 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:53.894 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:53.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:53.998 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:54.020 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:54.027 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:54.046 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:54.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:54.150 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:13:54.178 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:54.198 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:13:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:54.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:13:54.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:13:54.334 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:54.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:13:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:54.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:13:54.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:13:54.488 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:54 smithi121 ceph-mon[44533]: osdmap e215: 12 total, 12 up, 10 in 2024-04-29T14:13:54.488 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:54 smithi121 ceph-mon[44533]: pgmap v591: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:54.490 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:54.510 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:13:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:54.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:13:54.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:54 smithi040 ceph-mon[39581]: osdmap e215: 12 total, 12 up, 10 in 2024-04-29T14:13:54.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:54 smithi040 ceph-mon[39581]: pgmap v591: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:54 smithi194 ceph-mon[44089]: osdmap e215: 12 total, 12 up, 10 in 2024-04-29T14:13:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:54 smithi194 ceph-mon[44089]: pgmap v591: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:54.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:54.645 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:54.665 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:54.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:54.769 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:13:54.798 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:13:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:13:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:56.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:13:56.553 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:13:56.585 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:56.607 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:13:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:56.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:13:56.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:56 smithi040 ceph-mon[39581]: pgmap v592: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:56.711 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:13:56.740 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:56.759 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:13:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:56.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:13:56.864 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:13:56.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:56 smithi121 ceph-mon[44533]: pgmap v592: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:56.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:56 smithi194 ceph-mon[44089]: pgmap v592: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:56.892 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:56.912 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:13:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:56.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:13:57.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:13:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:57.734 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 10, 11] out_osds: [1, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:13:57.734 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:13:57.734 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-04-29T14:13:57.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 6 2024-04-29T14:13:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:13:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:58.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:13:58.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:58.172 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:13:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:13:58.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:13:58.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:13:58.305 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:13:58.309 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:13:58.309 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:13:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-29T14:13:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:58 smithi040 ceph-mon[39581]: pgmap v593: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:58 smithi040 ceph-mon[39581]: osdmap e216: 12 total, 12 up, 10 in 2024-04-29T14:13:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/497502803' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:13:58.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:58.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:58.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:58 smithi121 ceph-mon[44533]: pgmap v593: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:58.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:58 smithi121 ceph-mon[44533]: osdmap e216: 12 total, 12 up, 10 in 2024-04-29T14:13:58.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/497502803' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:13:59.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:13:59.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:13:59.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:58 smithi194 ceph-mon[44089]: pgmap v593: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:13:59.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:58 smithi194 ceph-mon[44089]: osdmap e216: 12 total, 12 up, 10 in 2024-04-29T14:13:59.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/497502803' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:13:59.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:13:59.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:13:59.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/497502803' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:13:59.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:59 smithi040 ceph-mon[39581]: osdmap e217: 12 total, 12 up, 10 in 2024-04-29T14:13:59.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:13:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3802399655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:59.843 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:13:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:13:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:13:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/497502803' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:13:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:59 smithi121 ceph-mon[44533]: osdmap e217: 12 total, 12 up, 10 in 2024-04-29T14:13:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:13:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3802399655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:13:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:13:59.922 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:13:59.923 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:13:59.923 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:13:59.923 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:13:59.923 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:14:00.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:00.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:00.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/497502803' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:14:00.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:59 smithi194 ceph-mon[44089]: osdmap e217: 12 total, 12 up, 10 in 2024-04-29T14:14:00.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:13:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3802399655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: pgmap v596: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: pgmap v596: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:00.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:14:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.061 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: pgmap v596: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:01.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:01.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:01.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:01.505 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:01.537 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:01.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:01.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:01.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:01.797 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:01.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:01.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:01.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.922 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:01.951 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.951 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:01.954 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:01.975 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:01.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:02.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:02.107 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-29T14:14:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:02.110 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:14:02.110 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:14:02.204 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:14:02.204 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:14:02.204 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_88551 2024-04-29T14:14:02.204 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:14:02.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:14:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:14:02.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:14:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:02 smithi040 ceph-mon[39581]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1326552726' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:02.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:02 smithi121 ceph-mon[44533]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:02.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1326552726' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:03.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:02 smithi194 ceph-mon[44089]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:03.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1326552726' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:03.204 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 15 6155 6140 383.73 383.75 0.00165289 0.00259946 2024-04-29T14:14:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:03.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:03.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:03.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:14:03.707 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:03.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:03 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:03.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1326552726' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:14:03.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:03 smithi040 ceph-mon[39581]: osdmap e218: 12 total, 12 up, 10 in 2024-04-29T14:14:03.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:03 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:03.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1326552726' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:14:03.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:03 smithi121 ceph-mon[44533]: osdmap e218: 12 total, 12 up, 10 in 2024-04-29T14:14:04.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:03 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:04.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1326552726' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:14:04.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:03 smithi194 ceph-mon[44089]: osdmap e218: 12 total, 12 up, 10 in 2024-04-29T14:14:04.204 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 16 11766 11750 367.164 350.625 0.00316283 0.00272031 2024-04-29T14:14:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:14:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:04.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:14:04.468 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:04.505 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:04.537 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:04.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:04.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:04 smithi040 ceph-mon[39581]: pgmap v599: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:04.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:04.680 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:04.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:04.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:04.813 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:04.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:04 smithi194 ceph-mon[44089]: pgmap v599: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:04.852 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:04 smithi121 ceph-mon[44533]: pgmap v599: 17 pgs: 17 active+clean; 577 KiB data, 1.3 GiB used, 893 GiB / 894 GiB avail 2024-04-29T14:14:04.880 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:04.884 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:04.985 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:05.021 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:05.047 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:05.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:05.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:05.193 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:05.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 15 16576 16561 344.997 300.688 0.00357537 0.00289608 2024-04-29T14:14:05.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:05 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:14:05.540+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e218 definitely_dead 0 2024-04-29T14:14:05.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/539304396' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-29T14:14:05.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:05 smithi040 ceph-mon[39581]: from='client.25069 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-29T14:14:05.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/539304396' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-29T14:14:05.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:05 smithi121 ceph-mon[44533]: from='client.25069 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-29T14:14:06.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/539304396' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-29T14:14:06.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:05 smithi194 ceph-mon[44089]: from='client.25069 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-29T14:14:06.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 16 18582 18566 290.07 125.312 0.00166187 0.00294498 2024-04-29T14:14:06.347 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:14:06.347 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:14:06.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:181740 2.13 2:cd4f6809:::benchmark_data_smithi194_79000_object17479:head [delete] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:19.986411+0000", 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 166.36475867199999, 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21154995300000001, 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.348 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181740 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986411+0000", 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986411+0000", 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986413+0000", 2024-04-29T14:14:06.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4890000000000001e-06 2024-04-29T14:14:06.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986417+0000", 2024-04-29T14:14:06.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6709999999999998e-06 2024-04-29T14:14:06.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986425+0000", 2024-04-29T14:14:06.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.588e-06 2024-04-29T14:14:06.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986432+0000", 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5729999999999996e-06 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986482+0000", 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0243e-05 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.986620+0000", 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000137899 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4,5,9", 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.987022+0000", 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040260099999999999 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991235+0000", 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042130379999999997 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991278+0000", 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2342000000000002e-05 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.995984+0000", 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004706099 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.197713+0000", 2024-04-29T14:14:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20172922099999999 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.197817+0000", 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000103608 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.197961+0000", 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014458099999999999 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.355 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:181746 2.13 2:c92a17e3:::benchmark_data_smithi194_79000_object17485:head [delete] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:19.991195+0000", 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 166.35997486799999, 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20701883500000001, 2024-04-29T14:14:06.356 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181746 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991195+0000", 2024-04-29T14:14:06.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991195+0000", 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991197+0000", 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5409999999999999e-06 2024-04-29T14:14:06.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991202+0000", 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1270000000000002e-06 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991210+0000", 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2490000000000001e-06 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991217+0000", 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6409999999999996e-06 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991347+0000", 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013034099999999999 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991476+0000", 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000128694 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4,5,9", 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.991873+0000", 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039757900000000001 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.995834+0000", 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039608220000000001 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.995887+0000", 2024-04-29T14:14:06.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3136000000000001e-05 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:19.995994+0000", 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107074 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.198020+0000", 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.202025911 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.198086+0000", 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5662999999999998e-05 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.198214+0000", 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000128057 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.363 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:181767 2.11 2:8c680f35:::benchmark_data_smithi194_79000_object17506:head [delete] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:20.018767+0000", 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 166.332402903, 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.17440602299999999, 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181767 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.018767+0000", 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.018767+0000", 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.018769+0000", 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.379e-06 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.018777+0000", 2024-04-29T14:14:06.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5390000000000002e-06 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.018795+0000", 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8088999999999999e-05 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.018805+0000", 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8889999999999997e-06 2024-04-29T14:14:06.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.018873+0000", 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8391000000000003e-05 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.019031+0000", 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015754399999999999 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3,4,6", 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.019568+0000", 2024-04-29T14:14:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053666499999999997 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.023969+0000", 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044015479999999999 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.026054+0000", 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020853830000000001 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.027307+0000", 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012522519999999999 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.192972+0000", 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.165665441 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.193046+0000", 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4152000000000004e-05 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.193173+0000", 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012675100000000001 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:181779 2.13 2:ce32db10:::benchmark_data_smithi194_79000_object17518:head [delete] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:20.031610+0000", 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 166.31956000900001, 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.16684258299999999, 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181779 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031610+0000", 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031610+0000", 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031613+0000", 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.253e-06 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031622+0000", 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0799999999999995e-06 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031644+0000", 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1685e-05 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031654+0000", 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8919999999999995e-06 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031728+0000", 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3659999999999996e-05 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.031857+0000", 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129324 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4,5,9", 2024-04-29T14:14:06.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.032164+0000", 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030764799999999999 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.034807+0000", 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026424 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.034822+0000", 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5318000000000001e-05 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.034860+0000", 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7511000000000001e-05 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.198277+0000", 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.163417216 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.198331+0000", 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4447999999999998e-05 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:20.198453+0000", 2024-04-29T14:14:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121148 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295237 2.e 2:7b91fc19:::benchmark_data_smithi194_79000_object130976:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.121832+0000", 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.229337838999996, 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22279016800000001, 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295237 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.121832+0000", 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.121832+0000", 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.121833+0000", 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4399999999999998e-07 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.121838+0000", 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7079999999999996e-06 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.121848+0000", 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0322e-05 2024-04-29T14:14:06.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.121856+0000", 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8320000000000006e-06 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.127451+0000", 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055950210000000004 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.127536+0000", 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5371999999999995e-05 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.127613+0000", 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6315000000000004e-05 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.140986+0000", 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013373184999999999 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.344502+0000", 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20351641200000001 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.344568+0000", 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5851000000000004e-05 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.344622+0000", 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4206000000000002e-05 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295249 2.e 2:791b67b0:::benchmark_data_smithi194_79000_object130988:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.131461+0000", 2024-04-29T14:14:06.381 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.219708995000005, 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21332417200000001, 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295249 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.131461+0000", 2024-04-29T14:14:06.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.131461+0000", 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.131463+0000", 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1040000000000001e-06 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.131469+0000", 2024-04-29T14:14:06.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0739999999999998e-06 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.131485+0000", 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6339999999999999e-05 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.131495+0000", 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3670000000000003e-06 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.132215+0000", 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072020099999999996 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.132342+0000", 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126651 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.132467+0000", 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012488700000000001 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.140996+0000", 2024-04-29T14:14:06.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0085291240000000008 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.344680+0000", 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20368429599999999 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.344735+0000", 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5319999999999999e-05 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.344785+0000", 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9808000000000002e-05 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295257 2.1 2:821de57b:::benchmark_data_smithi194_79000_object130996:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.138312+0000", 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.212857717000006, 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217155179, 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.387 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295257 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138312+0000", 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138312+0000", 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138314+0000", 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0310000000000001e-06 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138321+0000", 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3539999999999996e-06 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138332+0000", 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1222e-05 2024-04-29T14:14:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138341+0000", 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3419999999999997e-06 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138411+0000", 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0143999999999998e-05 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.138540+0000", 2024-04-29T14:14:06.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012896899999999999 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.139213+0000", 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067230700000000003 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143255+0000", 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0040427739999999998 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.355384+0000", 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21212861899999999 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.355421+0000", 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6538000000000002e-05 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.355467+0000", 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6879000000000003e-05 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.392 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295270 2.1 2:87a1201f:::benchmark_data_smithi194_79000_object131009:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.143912+0000", 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.207258246999999, 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214447572, 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.393 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295270 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143912+0000", 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143912+0000", 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143914+0000", 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4540000000000001e-06 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143919+0000", 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7779999999999998e-06 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143930+0000", 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0641e-05 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143936+0000", 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.212e-06 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143993+0000", 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7544999999999999e-05 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.144120+0000", 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012664100000000001 2024-04-29T14:14:06.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.144796+0000", 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0006759 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156324+0000", 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011528208999999999 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.358246+0000", 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.201922096 2024-04-29T14:14:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.358308+0000", 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.232e-05 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.358359+0000", 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0776000000000003e-05 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295271 2.e 2:7f8a98be:::benchmark_data_smithi194_79000_object131010:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.143953+0000", 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.207217377999996, 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217205441, 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295271 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143953+0000", 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143953+0000", 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143953+0000", 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4900000000000005e-07 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143957+0000", 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-06 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143970+0000", 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2931e-05 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.143976+0000", 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2920000000000001e-06 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.144030+0000", 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3699999999999997e-05 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.144156+0000", 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012665899999999999 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.144286+0000", 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012951600000000001 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.153025+0000", 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0087391269999999993 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.361058+0000", 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20803341 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.361115+0000", 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7046000000000002e-05 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.361158+0000", 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2589999999999997e-05 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295273 2.e 2:780302d5:::benchmark_data_smithi194_79000_object131012:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.146179+0000", 2024-04-29T14:14:06.404 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.204990590999998, 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223840602, 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295273 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.146179+0000", 2024-04-29T14:14:06.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.146179+0000", 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.146181+0000", 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8959999999999999e-06 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.146188+0000", 2024-04-29T14:14:06.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6039999999999998e-06 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.146201+0000", 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3362e-05 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.146210+0000", 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.789e-06 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.148032+0000", 2024-04-29T14:14:06.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001821958 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.148166+0000", 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013413699999999999 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.148292+0000", 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125587 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.153247+0000", 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049552550000000004 2024-04-29T14:14:06.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.369919+0000", 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216671894 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.369978+0000", 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9367999999999998e-05 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370020+0000", 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1752000000000003e-05 2024-04-29T14:14:06.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295280 2.1 2:8588681c:::benchmark_data_smithi194_79000_object131019:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.152867+0000", 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.198302744000003, 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21150637899999999, 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.410 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295280 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.152867+0000", 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.152867+0000", 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.152870+0000", 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9110000000000002e-06 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.152878+0000", 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9619999999999993e-06 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.152894+0000", 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5824e-05 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.152903+0000", 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5929999999999995e-06 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.152971+0000", 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7700999999999994e-05 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.153103+0000", 2024-04-29T14:14:06.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000131717 2024-04-29T14:14:06.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.153682+0000", 2024-04-29T14:14:06.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00057870000000000003 2024-04-29T14:14:06.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156473+0000", 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0027913 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364273+0000", 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20779998099999999 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364328+0000", 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4796000000000002e-05 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364374+0000", 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5893999999999998e-05 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295283 2.e 2:70b16dba:::benchmark_data_smithi194_79000_object131022:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.156705+0000", 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.194465034000004, 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.213455274, 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295283 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156705+0000", 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156705+0000", 2024-04-29T14:14:06.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156707+0000", 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.672e-06 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156711+0000", 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8600000000000001e-06 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156722+0000", 2024-04-29T14:14:06.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.077e-05 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156731+0000", 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0780000000000002e-06 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156804+0000", 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2308e-05 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156921+0000", 2024-04-29T14:14:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117662 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156990+0000", 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8763999999999994e-05 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.164145+0000", 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0071545339999999997 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370072+0000", 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20592756600000001 2024-04-29T14:14:06.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370121+0000", 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9347e-05 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370160+0000", 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8713e-05 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295284 2.1 2:832afe15:::benchmark_data_smithi194_79000_object131023:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.156771+0000", 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.194399207000004, 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20776761299999999, 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295284 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156771+0000", 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156771+0000", 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156772+0000", 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0400000000000005e-07 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156775+0000", 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3900000000000002e-06 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156782+0000", 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.52e-06 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156787+0000", 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2849999999999996e-06 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156827+0000", 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0034000000000003e-05 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.156910+0000", 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3338999999999996e-05 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.157271+0000", 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036065099999999997 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.160152+0000", 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002881045 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364432+0000", 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20428067499999999 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364493+0000", 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0161999999999998e-05 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364538+0000", 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5707999999999997e-05 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295288 2.e 2:7472cdc4:::benchmark_data_smithi194_79000_object131027:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.157291+0000", 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.193878689000002, 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.213401913, 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295288 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.157291+0000", 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.157291+0000", 2024-04-29T14:14:06.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.157293+0000", 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.505e-06 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.157298+0000", 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1830000000000002e-06 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.157308+0000", 2024-04-29T14:14:06.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0084999999999999e-05 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.157316+0000", 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7209999999999994e-06 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.160570+0000", 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032543870000000001 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.160654+0000", 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4016000000000003e-05 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.160759+0000", 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104818 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.164339+0000", 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035798280000000002 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370642+0000", 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206303668 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370670+0000", 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7861e-05 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370693+0000", 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2841e-05 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.432 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295293 2.1 2:8524773c:::benchmark_data_smithi194_79000_object131032:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.163810+0000", 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.187360040000002, 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.200855382, 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.433 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295293 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163810+0000", 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163810+0000", 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163812+0000", 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9230000000000001e-06 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163820+0000", 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9449999999999993e-06 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163830+0000", 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0652000000000001e-05 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163836+0000", 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0179999999999998e-06 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163887+0000", 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0881000000000003e-05 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.163979+0000", 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1978000000000006e-05 2024-04-29T14:14:06.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.164511+0000", 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053208499999999996 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.167428+0000", 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029161840000000001 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364578+0000", 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.197150136 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364629+0000", 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1051000000000003e-05 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.364665+0000", 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6529000000000002e-05 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.438 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295298 2.e 2:75ec2db5:::benchmark_data_smithi194_79000_object131037:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.169855+0000", 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.181315209000005, 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21682180300000001, 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295298 2024-04-29T14:14:06.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.169855+0000", 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.169855+0000", 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.169858+0000", 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.517e-06 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.169870+0000", 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1899000000000001e-05 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.169897+0000", 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6363000000000002e-05 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.169913+0000", 2024-04-29T14:14:06.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6183000000000001e-05 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.170011+0000", 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8797999999999998e-05 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.170098+0000", 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7006e-05 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.170169+0000", 2024-04-29T14:14:06.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0357000000000004e-05 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.174816+0000", 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046474669999999997 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386573+0000", 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211756956 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386625+0000", 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1316000000000001e-05 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386677+0000", 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1941000000000003e-05 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295318 2.1 2:8753935d:::benchmark_data_smithi194_79000_object131057:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.194180+0000", 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.156989658000001, 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.176001925, 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.444 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295318 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194180+0000", 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194180+0000", 2024-04-29T14:14:06.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194184+0000", 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5599999999999998e-06 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194194+0000", 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0396e-05 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194219+0000", 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.514e-05 2024-04-29T14:14:06.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194238+0000", 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8383e-05 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194351+0000", 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000113519 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.194529+0000", 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000177379 2024-04-29T14:14:06.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.195077+0000", 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000547884 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.197803+0000", 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002726947 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370096+0000", 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.17229245000000001 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370145+0000", 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8646999999999997e-05 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370182+0000", 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7620000000000002e-05 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.449 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295341 2.1 2:831a54ad:::benchmark_data_smithi194_79000_object131080:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.230581+0000", 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.120589011999996, 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.139717702, 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.450 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295341 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230581+0000", 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230581+0000", 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230583+0000", 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.543e-06 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230591+0000", 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9640000000000003e-06 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230607+0000", 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5330999999999999e-05 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230614+0000", 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1500000000000002e-06 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230675+0000", 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1416999999999998e-05 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.230779+0000", 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000103906 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.231348+0000", 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00056866800000000004 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.235034+0000", 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003685621 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370219+0000", 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13518567400000001 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370264+0000", 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4744999999999999e-05 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.370299+0000", 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4683e-05 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.455 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295346 2.e 2:7d97edc2:::benchmark_data_smithi194_79000_object131085:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.240816+0000", 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.110354221999998, 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.145965396, 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295346 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240816+0000", 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240816+0000", 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240818+0000", 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.801e-06 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240823+0000", 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4110000000000004e-06 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240834+0000", 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0844e-05 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240840+0000", 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.55e-06 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240891+0000", 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1165999999999998e-05 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240982+0000", 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0177000000000001e-05 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.241062+0000", 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0406000000000002e-05 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.246631+0000", 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055688140000000001 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386706+0000", 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.14007497899999999 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386734+0000", 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7827e-05 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386781+0000", 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7420999999999999e-05 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:295347 2.e 2:7631a79e:::benchmark_data_smithi194_79000_object131086:head [delete] snapc 0=[] ondisk+write+known_if_redirected e181)", 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:12:47.240876+0000", 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.110293661, 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.145994814, 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:06.461 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 295347 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240876+0000", 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240876+0000", 2024-04-29T14:14:06.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240877+0000", 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1939999999999999e-06 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240882+0000", 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2710000000000003e-06 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240889+0000", 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5700000000000004e-06 2024-04-29T14:14:06.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.240895+0000", 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4430000000000002e-06 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.244434+0000", 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035396400000000001 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.244514+0000", 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9374000000000003e-05 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.244582+0000", 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7879999999999994e-05 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.246785+0000", 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002203637 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386811+0000", 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.140025384 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386838+0000", 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7358000000000001e-05 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:12:47.386871+0000", 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3062999999999998e-05 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:06.466 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:14:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.6. 2024-04-29T14:14:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:06.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:06 smithi040 ceph-mon[39581]: pgmap v600: 17 pgs: 17 active+clean; 390 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 49 MiB/s wr, 783 op/s 2024-04-29T14:14:06.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:06 smithi040 ceph-mon[39581]: from='client.25069 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-29T14:14:06.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:06 smithi040 ceph-mon[39581]: osdmap e219: 12 total, 12 up, 11 in 2024-04-29T14:14:06.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:06 smithi040 ceph-mon[39581]: osdmap e220: 12 total, 12 up, 11 in 2024-04-29T14:14:06.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:06 smithi121 ceph-mon[44533]: pgmap v600: 17 pgs: 17 active+clean; 390 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 49 MiB/s wr, 783 op/s 2024-04-29T14:14:06.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:06 smithi121 ceph-mon[44533]: from='client.25069 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-29T14:14:06.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:06 smithi121 ceph-mon[44533]: osdmap e219: 12 total, 12 up, 11 in 2024-04-29T14:14:06.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:06 smithi121 ceph-mon[44533]: osdmap e220: 12 total, 12 up, 11 in 2024-04-29T14:14:07.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:06 smithi194 ceph-mon[44089]: pgmap v600: 17 pgs: 17 active+clean; 390 MiB data, 2.3 GiB used, 892 GiB / 894 GiB avail; 49 MiB/s wr, 783 op/s 2024-04-29T14:14:07.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:06 smithi194 ceph-mon[44089]: from='client.25069 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-29T14:14:07.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:06 smithi194 ceph-mon[44089]: osdmap e219: 12 total, 12 up, 11 in 2024-04-29T14:14:07.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:06 smithi194 ceph-mon[44089]: osdmap e220: 12 total, 12 up, 11 in 2024-04-29T14:14:07.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 16 18745 18729 234.09 10.1875 0.00150214 0.00326303 2024-04-29T14:14:07.580 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:07.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:07.595 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-04-29T14:14:07.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:07.730 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:07.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:07.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:07.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:07.907 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:07.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:07.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:08.037 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:08.068 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:08.093 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:08.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:08.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:14:08.155 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:07 smithi040 ceph-mon[39581]: osdmap e221: 12 total, 12 up, 11 in 2024-04-29T14:14:08.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:08.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 16 20557 20541 213.948 113.25 0.00162897 0.00467066 2024-04-29T14:14:08.241 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:07 smithi121 ceph-mon[44533]: osdmap e221: 12 total, 12 up, 11 in 2024-04-29T14:14:08.244 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:08.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:08.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:08.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:07 smithi194 ceph-mon[44089]: osdmap e221: 12 total, 12 up, 11 in 2024-04-29T14:14:08.380 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:08.440 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:08.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:08.522 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:14:08.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:08.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:08.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:08.770 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:08.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:08.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:08.903 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:08.941 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:09.073 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:09.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:09.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:09.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:08 smithi040 ceph-mon[39581]: pgmap v604: 17 pgs: 17 active+clean; 893 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-29T14:14:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:09.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 16 23348 23332 208.3 174.438 0.00176673 0.00470597 2024-04-29T14:14:09.259 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:09.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:09.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:08 smithi194 ceph-mon[44089]: pgmap v604: 17 pgs: 17 active+clean; 893 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-29T14:14:09.335 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:09.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:09.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:08 smithi121 ceph-mon[44533]: pgmap v604: 17 pgs: 17 active+clean; 893 MiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-29T14:14:09.386 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:09.416 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:10.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 16 26423 26407 206.284 192.188 0.00162801 0.00480654 2024-04-29T14:14:10.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:14:10.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:14:10.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:14:10.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:14:10.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:14:10.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:14:10.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:14:10.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:14:10.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:14:10.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:14:10.399 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:14:10.428 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:14:10.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:14:10.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:14:10.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:14:10.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:14:10.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:14:10.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:14:10.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:14:10.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:14:10.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:14:10.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:14:10.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:14:10.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:14:11.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 16 29182 29166 202.522 172.438 0.0060934 0.0049361 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: pgmap v605: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 183 MiB/s wr, 2.92k op/s; 631/35090 objects misplaced (1.798%); 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/767888504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: pgmap v605: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 183 MiB/s wr, 2.92k op/s; 631/35090 objects misplaced (1.798%); 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/767888504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: pgmap v605: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 183 MiB/s wr, 2.92k op/s; 631/35090 objects misplaced (1.798%); 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/767888504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:12.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 16 32472 32456 202.83 205.625 0.00334326 0.00492758 2024-04-29T14:14:12.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:12.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:12 smithi194 ceph-mon[44089]: osdmap e222: 12 total, 12 up, 11 in 2024-04-29T14:14:12.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:12.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:12 smithi040 ceph-mon[39581]: osdmap e222: 12 total, 12 up, 11 in 2024-04-29T14:14:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:12 smithi121 ceph-mon[44533]: osdmap e222: 12 total, 12 up, 11 in 2024-04-29T14:14:12.596 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 6] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:14:12.596 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:14:12.596 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 6] 2024-04-29T14:14:12.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 2 2024-04-29T14:14:12.839 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:12.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:12.948 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:12.982 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:13.008 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:13.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:13.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:13.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:13.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:13.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 35089 35073 199.259 163.562 0.00479708 0.00501669 2024-04-29T14:14:13.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:13 smithi194 ceph-mon[44089]: pgmap v606: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 1.6 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 207 MiB/s wr, 3.31k op/s; 1426/52138 objects misplaced (2.735%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:14:13.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:13 smithi040 ceph-mon[39581]: pgmap v606: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 1.6 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 207 MiB/s wr, 3.31k op/s; 1426/52138 objects misplaced (2.735%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:14:13.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:13 smithi121 ceph-mon[44533]: pgmap v606: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 1.6 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 207 MiB/s wr, 3.31k op/s; 1426/52138 objects misplaced (2.735%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:14:13.336 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:13.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:13.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:13.443 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:14:13.443 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:14:13.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:13.513 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:13.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:13.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:13.645 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:13.679 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:13.704 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:13.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:13.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:13.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:13.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:13.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:14.016 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:14:14.036 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:14.041 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1911732457' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:14.041 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:14 smithi040 ceph-mon[39581]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:14.062 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:14.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:14.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1911732457' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:14.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:14 smithi121 ceph-mon[44533]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:14.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:14:14.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 38069 38053 198.174 186.25 0.00212432 0.00504462 2024-04-29T14:14:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:14.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1911732457' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:14.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:14 smithi194 ceph-mon[44089]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:14.315 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:14:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:14.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:14:14.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:14.457 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:14.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:14.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:14.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:14.640 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:14.673 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:14.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:14.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:14.819 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:14.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:14.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:14.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:14.990 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:15.049 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:15.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:15.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:15 smithi121 ceph-mon[44533]: pgmap v608: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 1.6 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s; 1426/52138 objects misplaced (2.735%); 7.6 MiB/s, 122 objects/s recovering 2024-04-29T14:14:15.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:15 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:14:15.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:15 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:14:15.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:15 smithi121 ceph-mon[44533]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:15.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:15 smithi121 ceph-mon[44533]: osdmap e223: 12 total, 12 up, 11 in 2024-04-29T14:14:15.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:15.194 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:15.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 16 40978 40962 196.914 181.812 0.00219445 0.00507607 2024-04-29T14:14:15.221 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:15.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:15 smithi194 ceph-mon[44089]: pgmap v608: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 1.6 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s; 1426/52138 objects misplaced (2.735%); 7.6 MiB/s, 122 objects/s recovering 2024-04-29T14:14:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:15 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:14:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:15 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:14:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:15 smithi194 ceph-mon[44089]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:15.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:15 smithi194 ceph-mon[44089]: osdmap e223: 12 total, 12 up, 11 in 2024-04-29T14:14:15.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:15 smithi040 ceph-mon[39581]: pgmap v608: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 1.6 GiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s; 1426/52138 objects misplaced (2.735%); 7.6 MiB/s, 122 objects/s recovering 2024-04-29T14:14:15.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:15 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:14:15.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:15 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:14:15.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:15 smithi040 ceph-mon[39581]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:15.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:15 smithi040 ceph-mon[39581]: osdmap e223: 12 total, 12 up, 11 in 2024-04-29T14:14:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:15.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:15.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:15.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:14:15.578 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:15.612 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:15.639 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:15.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:15.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:15.788 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:15.814 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:15.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:15.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:15.970 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:16.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:16.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:16.205 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 15 43858 43843 195.709 180.062 0.00935677 0.0051077 2024-04-29T14:14:16.293 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:16.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:16.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:16.436 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:16.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:14:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:16.891 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:16.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:16.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:14:16.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:17 smithi040 ceph-mon[39581]: pgmap v610: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.0 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 149 MiB/s wr, 2.39k op/s; 795/66776 objects misplaced (1.191%); 12 MiB/s, 184 objects/s recovering 2024-04-29T14:14:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:17 smithi040 ceph-mon[39581]: osdmap e224: 12 total, 12 up, 11 in 2024-04-29T14:14:17.175 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:17.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:17.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 16 46336 46320 192.981 154.812 0.00161787 0.00512591 2024-04-29T14:14:17.282 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:17.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:17 smithi194 ceph-mon[44089]: pgmap v610: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.0 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 149 MiB/s wr, 2.39k op/s; 795/66776 objects misplaced (1.191%); 12 MiB/s, 184 objects/s recovering 2024-04-29T14:14:17.307 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:17 smithi194 ceph-mon[44089]: osdmap e224: 12 total, 12 up, 11 in 2024-04-29T14:14:17.309 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:17.328 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:17.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:17 smithi121 ceph-mon[44533]: pgmap v610: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.0 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 149 MiB/s wr, 2.39k op/s; 795/66776 objects misplaced (1.191%); 12 MiB/s, 184 objects/s recovering 2024-04-29T14:14:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:17 smithi121 ceph-mon[44533]: osdmap e224: 12 total, 12 up, 11 in 2024-04-29T14:14:17.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:14:17.461 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:17.718 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:17.910 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:14:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:17.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:14:18.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:18.050 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:18.084 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:18.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:18.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:14:18.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 16 46898 46882 183.115 35.125 0.0032425 0.00545977 2024-04-29T14:14:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:18.310 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:18.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:14:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:18.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:14:18.501 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:18.555 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:18.640 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:18.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:18.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:18.790 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:18.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:18.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:18 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:14:18.626+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e225 definitely_dead 0 2024-04-29T14:14:18.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:18.963 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.2. 2024-04-29T14:14:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:18 smithi121 ceph-mon[44533]: pgmap v612: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 154 MiB/s wr, 2.46k op/s; 104/81688 objects misplaced (0.127%); 14 MiB/s, 220 objects/s recovering 2024-04-29T14:14:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:18 smithi121 ceph-mon[44533]: osdmap e225: 12 total, 12 up, 11 in 2024-04-29T14:14:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2993221824' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:14:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:18 smithi121 ceph-mon[44533]: from='client.25087 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:14:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:18 smithi121 ceph-mon[44533]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-29T14:14:19.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 49668 49652 182.527 173.125 0.00461986 0.00547708 2024-04-29T14:14:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:18 smithi194 ceph-mon[44089]: pgmap v612: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 154 MiB/s wr, 2.46k op/s; 104/81688 objects misplaced (0.127%); 14 MiB/s, 220 objects/s recovering 2024-04-29T14:14:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:18 smithi194 ceph-mon[44089]: osdmap e225: 12 total, 12 up, 11 in 2024-04-29T14:14:19.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2993221824' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:14:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:18 smithi194 ceph-mon[44089]: from='client.25087 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:14:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:18 smithi194 ceph-mon[44089]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-29T14:14:19.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:18 smithi040 ceph-mon[39581]: pgmap v612: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 154 MiB/s wr, 2.46k op/s; 104/81688 objects misplaced (0.127%); 14 MiB/s, 220 objects/s recovering 2024-04-29T14:14:19.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:18 smithi040 ceph-mon[39581]: osdmap e225: 12 total, 12 up, 11 in 2024-04-29T14:14:19.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2993221824' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:14:19.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:18 smithi040 ceph-mon[39581]: from='client.25087 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:14:19.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:18 smithi040 ceph-mon[39581]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-29T14:14:19.819 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:19.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:19.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:19.966 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:19.995 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:20.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:20.102 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:20.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:20 smithi040 ceph-mon[39581]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-29T14:14:20.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:20 smithi040 ceph-mon[39581]: osdmap e226: 12 total, 12 up, 10 in 2024-04-29T14:14:20.137 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:20.162 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:20.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:20.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 52417 52401 181.931 171.812 0.00609314 0.00549491 2024-04-29T14:14:20.268 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:20.297 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:20 smithi194 ceph-mon[44089]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-29T14:14:20.297 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:20 smithi194 ceph-mon[44089]: osdmap e226: 12 total, 12 up, 10 in 2024-04-29T14:14:20.300 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:20.323 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:20.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:20 smithi121 ceph-mon[44533]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-29T14:14:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:20 smithi121 ceph-mon[44533]: osdmap e226: 12 total, 12 up, 10 in 2024-04-29T14:14:20.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:20.468 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:20.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:20.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:20.786 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:20.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:20.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:20.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:20.970 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:20.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:21.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:21.029 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:21 smithi121 ceph-mon[44533]: pgmap v615: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.7 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 168 MiB/s wr, 2.69k op/s; 104/87370 objects misplaced (0.119%); 7.8 MiB/s, 125 objects/s recovering 2024-04-29T14:14:21.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:14:21.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:14:21.030 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:21 smithi121 ceph-mon[44533]: osdmap e227: 12 total, 12 up, 10 in 2024-04-29T14:14:21.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:21.140 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:21 smithi040 ceph-mon[39581]: pgmap v615: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.7 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 168 MiB/s wr, 2.69k op/s; 104/87370 objects misplaced (0.119%); 7.8 MiB/s, 125 objects/s recovering 2024-04-29T14:14:21.140 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:14:21.141 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:14:21.141 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:21 smithi040 ceph-mon[39581]: osdmap e227: 12 total, 12 up, 10 in 2024-04-29T14:14:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:21.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 55075 55059 181.098 166.125 0.00692808 0.00552068 2024-04-29T14:14:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:21 smithi194 ceph-mon[44089]: pgmap v615: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 2.7 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 168 MiB/s wr, 2.69k op/s; 104/87370 objects misplaced (0.119%); 7.8 MiB/s, 125 objects/s recovering 2024-04-29T14:14:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:21.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:21 smithi194 ceph-mon[44089]: osdmap e227: 12 total, 12 up, 10 in 2024-04-29T14:14:21.599 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:21.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:21.719 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:21.756 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:21.785 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:21.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:21.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:21.942 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:21.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:22.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:22.119 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:22.184 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:22.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:22.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:22.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:14:22.213681 min lat: 0.00103764 max lat: 2.08667 avg lat: 0.00557419 2024-04-29T14:14:22.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:14:22.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 16 57412 57396 179.346 146.062 0.00374802 0.00557419 2024-04-29T14:14:22.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:22.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:14:22.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:22 smithi194 ceph-mon[44089]: osdmap e228: 12 total, 12 up, 10 in 2024-04-29T14:14:22.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:14:22.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:22 smithi121 ceph-mon[44533]: osdmap e228: 12 total, 12 up, 10 in 2024-04-29T14:14:22.342 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:22.348 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:14:22.348 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:14:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:14:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:22 smithi040 ceph-mon[39581]: osdmap e228: 12 total, 12 up, 10 in 2024-04-29T14:14:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:22.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:22.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:22.521 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:22.550 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:22.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:22.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:22.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:22.696 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:23.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:23.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:23.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:23.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 59950 59934 178.358 158.625 0.00256378 0.00560519 2024-04-29T14:14:23.213 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:23.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:23.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:23.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:23 smithi194 ceph-mon[44089]: pgmap v617: 17 pgs: 17 active+clean; 3.2 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 156 MiB/s wr, 2.49k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:14:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:23.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:23.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:23 smithi040 ceph-mon[39581]: pgmap v617: 17 pgs: 17 active+clean; 3.2 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 156 MiB/s wr, 2.49k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:14:23.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:14:23.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:14:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:23 smithi121 ceph-mon[44533]: pgmap v617: 17 pgs: 17 active+clean; 3.2 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 156 MiB/s wr, 2.49k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:14:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:14:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:14:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:23.480 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:23.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:23.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:14:23.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:23.655 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:23.794 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:23.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:23.900 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:23.935 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:23.962 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:23.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:24.066 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:24.102 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:24.126 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:24.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:24.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 16 62338 62322 177.035 149.25 0.00314971 0.0056475 2024-04-29T14:14:24.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:24.275 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:24.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:24.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:24.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:24.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:24.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:24.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:24.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:24.738 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:14:24.766 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:14:24.792 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:14:24.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:24.818 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:14:24.828 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:24.847 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:14:24.853 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:24.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:24.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:14:24.907 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:14:24.938 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:14:24.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:24.960 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:14:24.982 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:14:24.982 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:25.001 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:25.002 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:14:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:25.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:25.023 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:14:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:14:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:14:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:14:25.052 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:14:25.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:14:25.155 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:25.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 64286 64270 174.631 121.75 0.00557695 0.00572498 2024-04-29T14:14:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:25 smithi194 ceph-mon[44089]: pgmap v619: 17 pgs: 17 active+clean; 3.2 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 128 MiB/s wr, 2.04k op/s; 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:14:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2321742708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:14:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1592202427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:25 smithi040 ceph-mon[39581]: pgmap v619: 17 pgs: 17 active+clean; 3.2 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 128 MiB/s wr, 2.04k op/s; 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:14:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:14:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:14:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2321742708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:14:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1592202427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:25.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:25 smithi121 ceph-mon[44533]: pgmap v619: 17 pgs: 17 active+clean; 3.2 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 128 MiB/s wr, 2.04k op/s; 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:14:25.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:14:25.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:14:25.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2321742708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:14:25.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1592202427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:25.617 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:14:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:25.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:14:25.653 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 8, 9, 10, 11, 6] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:14:25.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:14:25.654 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:14:25.654 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-29T14:14:25.654 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2 2024-04-29T14:14:25.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:25.770 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:25.927 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:25.929 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:14:25 smithi040 systemd[1]: Stopping Ceph osd.2 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:14:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:25.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:26.031 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:26.071 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:26.206 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 16 66651 66635 173.513 147.812 0.00713359 0.00576171 2024-04-29T14:14:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2321742708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:14:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:26 smithi194 ceph-mon[44089]: osdmap e229: 12 total, 12 up, 10 in 2024-04-29T14:14:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2321742708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:14:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:26 smithi040 ceph-mon[39581]: osdmap e229: 12 total, 12 up, 10 in 2024-04-29T14:14:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2321742708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:14:26.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:26 smithi121 ceph-mon[44533]: osdmap e229: 12 total, 12 up, 10 in 2024-04-29T14:14:26.522 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:26.528 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:26.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:26.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:26.691 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:26.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:26.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:26.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:26.879 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:26.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:26.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:27.009 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:27.041 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:27.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:27 smithi121 ceph-mon[44533]: pgmap v621: 17 pgs: 17 active+clean; 3.6 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 161 MiB/s wr, 2.57k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:14:27.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:27 smithi121 ceph-mon[44533]: osdmap e230: 12 total, 12 up, 10 in 2024-04-29T14:14:27.066 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:27.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:27.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:27.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 16 68837 68821 172.037 136.625 0.0129973 0.00581073 2024-04-29T14:14:27.210 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:27.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:27.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:27.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:27 smithi194 ceph-mon[44089]: pgmap v621: 17 pgs: 17 active+clean; 3.6 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 161 MiB/s wr, 2.57k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:14:27.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:27 smithi194 ceph-mon[44089]: osdmap e230: 12 total, 12 up, 10 in 2024-04-29T14:14:27.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:27.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:27 smithi040 ceph-mon[39581]: pgmap v621: 17 pgs: 17 active+clean; 3.6 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 161 MiB/s wr, 2.57k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:14:27.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:27 smithi040 ceph-mon[39581]: osdmap e230: 12 total, 12 up, 10 in 2024-04-29T14:14:27.380 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:27.404 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:27.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:27.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:27.554 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:27.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:27.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:27.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:28.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 70953 70937 170.506 132.25 0.0102976 0.00586333 2024-04-29T14:14:28.485 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:28.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:28.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:14:28.589 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:28.627 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:28.652 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:28.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:28.756 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:28.790 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:28.816 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:28.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:28.862 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:14:28 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[73654]: 2024-04-29T14:14:28.508+0000 7f8fd32fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T14:14:28.863 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:14:28 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[73654]: 2024-04-29T14:14:28.508+0000 7f8fd32fd700 -1 osd.2 230 *** Got signal Terminated *** 2024-04-29T14:14:28.863 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:14:28 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[73654]: 2024-04-29T14:14:28.508+0000 7f8fd32fd700 -1 osd.2 230 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T14:14:28.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:29.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:29.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:29.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:29.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 16 73006 72990 168.943 128.312 0.00256234 0.00591591 2024-04-29T14:14:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:29 smithi040 ceph-mon[39581]: pgmap v623: 17 pgs: 17 active+clean; 3.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-29T14:14:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:29 smithi040 ceph-mon[39581]: osd.2 marked itself down and dead 2024-04-29T14:14:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:29 smithi121 ceph-mon[44533]: pgmap v623: 17 pgs: 17 active+clean; 3.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-29T14:14:29.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:29 smithi121 ceph-mon[44533]: osd.2 marked itself down and dead 2024-04-29T14:14:29.399 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:29.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:29.505 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:29.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:29 smithi194 ceph-mon[44089]: pgmap v623: 17 pgs: 17 active+clean; 3.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-29T14:14:29.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:29 smithi194 ceph-mon[44089]: osd.2 marked itself down and dead 2024-04-29T14:14:29.536 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:29.561 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:29.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:29.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:30.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 16 75064 75048 167.503 128.625 0.00585542 0.00596863 2024-04-29T14:14:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:30 smithi040 ceph-mon[39581]: osdmap e231: 12 total, 11 up, 10 in 2024-04-29T14:14:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:30 smithi121 ceph-mon[44533]: osdmap e231: 12 total, 11 up, 10 in 2024-04-29T14:14:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:30 smithi194 ceph-mon[44089]: osdmap e231: 12 total, 11 up, 10 in 2024-04-29T14:14:31.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 76671 76655 165.19 100.438 0.00989835 0.00605226 2024-04-29T14:14:31.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:31 smithi040 ceph-mon[39581]: pgmap v625: 17 pgs: 17 active+clean; 4.1 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-29T14:14:31.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:31 smithi121 ceph-mon[44533]: pgmap v625: 17 pgs: 17 active+clean; 4.1 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-29T14:14:31.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:14:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:31 smithi194 ceph-mon[44089]: pgmap v625: 17 pgs: 17 active+clean; 4.1 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-29T14:14:32.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 16 78752 78736 164.019 130.062 0.00361352 0.00609571 2024-04-29T14:14:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:14:32.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:14:32.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:32 smithi040 ceph-mon[39581]: osdmap e232: 12 total, 11 up, 10 in 2024-04-29T14:14:32.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:14:32.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:14:32.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:32 smithi121 ceph-mon[44533]: osdmap e232: 12 total, 11 up, 10 in 2024-04-29T14:14:32.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:32.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:32.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:32 smithi194 ceph-mon[44089]: osdmap e232: 12 total, 11 up, 10 in 2024-04-29T14:14:33.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 16 80792 80776 162.841 127.5 0.0073949 0.00613964 2024-04-29T14:14:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:33 smithi040 ceph-mon[39581]: pgmap v627: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-29T14:14:33.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:33 smithi121 ceph-mon[44533]: pgmap v627: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-29T14:14:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:33.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:33 smithi194 ceph-mon[44089]: pgmap v627: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-29T14:14:33.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:33.632 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:33.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:33.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:33.771 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:33.806 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:33.829 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:33.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:33.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:33.967 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:34.125 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:34.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:34.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 82658 82642 161.396 116.625 0.00404514 0.0061935 2024-04-29T14:14:34.208 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:34.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:34.271 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:14:33 smithi040 bash[127961]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2 2024-04-29T14:14:34.279 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:34.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:34.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:34.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:34.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:34.762 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:34.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:34.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:34.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:34.928 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:34.954 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:34.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:35.059 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:35.092 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:35.119 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:35.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:35.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 84498 84482 159.99 115 0.00550611 0.00624885 2024-04-29T14:14:35.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:35.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:35 smithi194 ceph-mon[44089]: pgmap v628: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-29T14:14:35.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:14:35.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:14:35.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.256 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:35.280 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:35.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:35.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:35 smithi040 ceph-mon[39581]: pgmap v628: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-29T14:14:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:14:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:14:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:35 smithi121 ceph-mon[44533]: pgmap v628: 17 pgs: 17 active+clean; 4.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-29T14:14:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:14:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:14:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:14:35.385 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:35.414 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:35.440 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:35.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:35.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:35.545 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:35.573 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:35.597 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:35.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:35.704 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:35.743 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:35.768 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:35.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:35.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:35.919 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:35.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:35.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:36.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:36.084 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:36.111 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:36.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:36 smithi040 ceph-mon[39581]: pgmap v629: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-04-29T14:14:36.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:36 smithi040 ceph-mon[39581]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-29T14:14:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:36.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:36.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 15 86415 86400 158.81 119.875 0.00713194 0.00629543 2024-04-29T14:14:36.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:36.249 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:36.273 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:36.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:36.377 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:36 smithi121 ceph-mon[44533]: pgmap v629: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-04-29T14:14:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:36 smithi121 ceph-mon[44533]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-29T14:14:36.414 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:36 smithi194 ceph-mon[44089]: pgmap v629: 17 pgs: 17 active+clean; 4.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-04-29T14:14:36.414 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:36 smithi194 ceph-mon[44089]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-29T14:14:36.417 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:36.440 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:36.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:36.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:36.608 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:36.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:36.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:36.742 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:36.765 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:36.777 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:36.804 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:36.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:36.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:36.954 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:36.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:37.088 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:37.122 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:37.146 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:37.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:37.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 88254 88238 157.554 114.875 0.00496865 0.00634543 2024-04-29T14:14:37.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:37.297 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:37.322 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:37.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:37.422 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.24727.0:319987 2.e e191/107 2:70ef11cd:::benchmark_data_smithi194_79000_object155726:head v 191'20068, mlcod=189'20066)", 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:13:08.167139+0000", 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 89.259162591000006, 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10639554599999999, 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167139+0000", 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167139+0000", 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167139+0000", 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4600000000000001e-07 2024-04-29T14:14:37.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167143+0000", 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1350000000000002e-06 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167157+0000", 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3794e-05 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167170+0000", 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2859e-05 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167233+0000", 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2599000000000002e-05 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.167250+0000", 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7041000000000002e-05 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.273479+0000", 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106229173 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.273534+0000", 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5699000000000003e-05 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:320002 2.19 2:9eb566ba:::benchmark_data_smithi194_79000_object155741:head [delete] snapc 0=[] ondisk+write+known_if_redirected e191)", 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:13:08.194279+0000", 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 89.232022869000005, 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.086101491000000002, 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 320002 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.428 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.194279+0000", 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.194279+0000", 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.194282+0000", 2024-04-29T14:14:37.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.067e-06 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.194292+0000", 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0217000000000001e-05 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.194309+0000", 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7034e-05 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.194321+0000", 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1677e-05 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.271924+0000", 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.077603602999999993 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.272106+0000", 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018144 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.273023+0000", 2024-04-29T14:14:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00091771099999999998 2024-04-29T14:14:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.276859+0000", 2024-04-29T14:14:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003835326 2024-04-29T14:14:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.280257+0000", 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003397897 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.280327+0000", 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0573000000000003e-05 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:13:08.280380+0000", 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2945999999999998e-05 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.434 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:74301 3.7 3:eed4540c:::benchmark_data_smithi194_88551_object74300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:29.837751+0000", 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 7.5885503530000005, 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.058583195999999997, 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 74301 2024-04-29T14:14:37.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.837751+0000", 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.837751+0000", 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.837753+0000", 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4169999999999999e-06 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.837823+0000", 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0352999999999995e-05 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.886983+0000", 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.049160504000000001 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.886996+0000", 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2118e-05 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.887105+0000", 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109448 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.887233+0000", 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012773 2024-04-29T14:14:37.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 11", 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.887375+0000", 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014228 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.894085+0000", 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0067100149999999997 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.896229+0000", 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021438400000000002 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.896303+0000", 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4169000000000001e-05 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:29.896334+0000", 2024-04-29T14:14:37.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1322000000000003e-05 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:75726 3.2 3:4705e249:::benchmark_data_smithi194_88551_object75725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.518624+0000", 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.9076774570000001, 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.066373576000000004, 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.441 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 75726 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518624+0000", 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518624+0000", 2024-04-29T14:14:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518625+0000", 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.466e-06 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518730+0000", 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104026 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518738+0000", 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0590000000000003e-06 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518752+0000", 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4781e-05 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518791+0000", 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8757999999999997e-05 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518870+0000", 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9246999999999995e-05 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518951+0000", 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0915999999999996e-05 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.582456+0000", 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.063505082000000004 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584926+0000", 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024696879999999998 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584967+0000", 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1321e-05 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584998+0000", 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0232000000000001e-05 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:75730 3.2 3:47d4c395:::benchmark_data_smithi194_88551_object75729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.518835+0000", 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.907466426, 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.066259117000000006, 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 75730 2024-04-29T14:14:37.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518835+0000", 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518835+0000", 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518836+0000", 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3069999999999999e-06 2024-04-29T14:14:37.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.518992+0000", 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000155458 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519000+0000", 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.174e-06 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519004+0000", 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1999999999999996e-06 2024-04-29T14:14:37.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.521156+0000", 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021516270000000001 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.521277+0000", 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121169 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.521401+0000", 2024-04-29T14:14:37.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012428800000000001 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.582463+0000", 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.061061987999999998 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585041+0000", 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002577242 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585070+0000", 2024-04-29T14:14:37.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9848e-05 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585094+0000", 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3816e-05 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:75731 3.2 3:4b5d1f09:::benchmark_data_smithi194_88551_object75730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.519014+0000", 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.907286977, 2024-04-29T14:14:37.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.066193293, 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 75731 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519014+0000", 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519014+0000", 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519015+0000", 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8999999999999996e-07 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519136+0000", 2024-04-29T14:14:37.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012044699999999999 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519144+0000", 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.303e-06 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519148+0000", 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2150000000000003e-06 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523081+0000", 2024-04-29T14:14:37.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039332799999999999 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523188+0000", 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000106807 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523319+0000", 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013052199999999999 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585010+0000", 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.061691575999999998 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585140+0000", 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012966 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585185+0000", 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5569999999999999e-05 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585208+0000", 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2323e-05 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:75732 3.2 3:4ecdcdc0:::benchmark_data_smithi194_88551_object75731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.519157+0000", 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.9071449510000003, 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.066139721999999998, 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.458 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 75732 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519157+0000", 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519157+0000", 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519157+0000", 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2500000000000004e-07 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519246+0000", 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9042999999999994e-05 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519252+0000", 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2190000000000003e-06 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.519255+0000", 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7529999999999999e-06 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.525112+0000", 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058565630000000004 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.525226+0000", 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114219 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.525326+0000", 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9699999999999998e-05 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585017+0000", 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.059690862999999997 2024-04-29T14:14:37.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585247+0000", 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000229912 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585275+0000", 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8486000000000001e-05 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585296+0000", 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1138999999999999e-05 2024-04-29T14:14:37.463 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:75743 3.2 3:4c04539d:::benchmark_data_smithi194_88551_object75742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.522385+0000", 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.9039166249999999, 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.062990766000000004, 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 75743 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.522385+0000", 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.522385+0000", 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.522387+0000", 2024-04-29T14:14:37.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.685e-06 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.522461+0000", 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4139000000000003e-05 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.522471+0000", 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8530000000000004e-06 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.522478+0000", 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.357e-06 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.580031+0000", 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.057553591000000001 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.580139+0000", 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107951 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.580284+0000", 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014423699999999999 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585215+0000", 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004931319 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585324+0000", 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108855 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585354+0000", 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0618999999999999e-05 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585376+0000", 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1160000000000001e-05 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.470 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.25063.0:75741 3.c e231/206 3:3140d813:::benchmark_data_smithi194_88551_object75740:head v 231'4732, mlcod=231'4730)", 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.523213+0000", 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.9030885279999996, 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.056741343, 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523213+0000", 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523213+0000", 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523214+0000", 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1349999999999999e-06 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523270+0000", 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6147000000000003e-05 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523277+0000", 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2579999999999998e-06 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523282+0000", 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7639999999999996e-06 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523321+0000", 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9164e-05 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.523334+0000", 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3064999999999999e-05 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.526104+0000", 2024-04-29T14:14:37.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0027690850000000001 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.579954+0000", 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053850725000000002 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.25063.0:75739 3.5 e231/206 3:ab8cd3cd:::benchmark_data_smithi194_88551_object75738:head v 231'4844, mlcod=231'4840)", 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.524894+0000", 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.9014077179999997, 2024-04-29T14:14:37.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.055125736000000002, 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.524894+0000", 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.524894+0000", 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.524895+0000", 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4810000000000001e-06 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.524953+0000", 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8063999999999998e-05 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.524963+0000", 2024-04-29T14:14:37.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.285e-06 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.524970+0000", 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9349999999999997e-06 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.525017+0000", 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7459000000000001e-05 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.525030+0000", 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2876e-05 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.579981+0000", 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054951541 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.580019+0000", 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8095e-05 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:75747 3.7 3:e4da63f7:::benchmark_data_smithi194_88551_object75746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.526412+0000", 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.8998895190000002, 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.058511581999999999, 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 75747 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.526412+0000", 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.526412+0000", 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.526414+0000", 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-06 2024-04-29T14:14:37.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.526465+0000", 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1304999999999998e-05 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.579961+0000", 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053495979999999999 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.579972+0000", 2024-04-29T14:14:37.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1165999999999999e-05 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.580028+0000", 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5988e-05 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.580137+0000", 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109298 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 11", 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.580280+0000", 2024-04-29T14:14:37.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000142435 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584635+0000", 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043548450000000004 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584842+0000", 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020726100000000001 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584875+0000", 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3099999999999998e-05 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584924+0000", 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8554999999999998e-05 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.485 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(client.25063.0:75726 3.2 e231/219 ondisk, result = 0)", 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.527991+0000", 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.8983108829999997, 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.057028011000000003, 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.527991+0000", 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.527991+0000", 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.527992+0000", 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3629999999999999e-06 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.527996+0000", 2024-04-29T14:14:37.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3880000000000002e-06 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.579923+0000", 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051926796999999997 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.579934+0000", 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1289e-05 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584913+0000", 2024-04-29T14:14:37.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049782460000000004 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.584920+0000", 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5440000000000001e-06 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.585019+0000", 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8383999999999999e-05 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:76155 3.2 3:4bcc1cca:::benchmark_data_smithi194_88551_object76154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.754864+0000", 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.6714370130000002, 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.057942220000000003, 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 76155 2024-04-29T14:14:37.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.754864+0000", 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.754864+0000", 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.754866+0000", 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.548e-06 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.754969+0000", 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000103119 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.754980+0000", 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0627e-05 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.754987+0000", 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3239999999999997e-06 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.755085+0000", 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7700000000000003e-05 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.755198+0000", 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112985 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.755317+0000", 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011904300000000001 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.757284+0000", 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019676670000000002 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812695+0000", 2024-04-29T14:14:37.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.055410612999999997 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812762+0000", 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7145999999999997e-05 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812807+0000", 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4447999999999999e-05 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.25063.0:76159 3.5 e231/206 3:a4a8830b:::benchmark_data_smithi194_88551_object76158:head v 231'4867, mlcod=231'4865)", 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.758029+0000", 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.6682722480000001, 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054698286999999998, 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758029+0000", 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758029+0000", 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758030+0000", 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8800000000000002e-07 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758065+0000", 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4900000000000001e-05 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758071+0000", 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5019999999999998e-06 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758077+0000", 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3000000000000001e-06 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758118+0000", 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1059000000000001e-05 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.758130+0000", 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2668e-05 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812684+0000", 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054553670999999998 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812728+0000", 2024-04-29T14:14:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3498999999999998e-05 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:76167 3.2 3:405b8809:::benchmark_data_smithi194_88551_object76166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.759426+0000", 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.6668756790000003, 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.054300255999999998, 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.500 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 76167 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759426+0000", 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759426+0000", 2024-04-29T14:14:37.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759427+0000", 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0550000000000001e-06 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759526+0000", 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8976999999999998e-05 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759535+0000", 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5580000000000006e-06 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759542+0000", 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0169999999999999e-06 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759591+0000", 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8369999999999999e-05 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759694+0000", 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010360499999999999 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.759829+0000", 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013503500000000001 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812855+0000", 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053025479 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.813643+0000", 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00078774499999999996 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.813690+0000", 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7011000000000001e-05 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.813726+0000", 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6403999999999999e-05 2024-04-29T14:14:37.505 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:76171 3.b 3:d857d42b:::benchmark_data_smithi194_88551_object76170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:30.762199+0000", 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.6641022379999999, 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.060576937999999997, 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.506 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 76171 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.762199+0000", 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.762199+0000", 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.762201+0000", 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9420000000000002e-06 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812809+0000", 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050607665000000003 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812819+0000", 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7019999999999996e-06 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.812826+0000", 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4379999999999998e-06 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.813127+0000", 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030061399999999999 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.813256+0000", 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012896399999999999 2024-04-29T14:14:37.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.813378+0000", 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000122225 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.816345+0000", 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002967136 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.822694+0000", 2024-04-29T14:14:37.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0063486710000000002 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.822728+0000", 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4119999999999999e-05 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:30.822776+0000", 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8461000000000002e-05 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.511 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:81693 3.2 3:48f76e29:::benchmark_data_smithi194_88551_object81692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e232)", 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:33.671768+0000", 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.7545339310000001, 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.067738582000000006, 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 81693 2024-04-29T14:14:37.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671768+0000", 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671768+0000", 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671769+0000", 2024-04-29T14:14:37.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.699e-06 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671802+0000", 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2873000000000002e-05 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671812+0000", 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0254999999999999e-05 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671820+0000", 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3629999999999996e-06 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671867+0000", 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6876999999999999e-05 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.671976+0000", 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010956299999999999 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:37.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.672095+0000", 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011909 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.674295+0000", 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002199346 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677417+0000", 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003122503 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677479+0000", 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2376e-05 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739506+0000", 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.062026637000000003 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:81700 3.7 3:e73f6ac9:::benchmark_data_smithi194_88551_object81699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e232)", 2024-04-29T14:14:37.517 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:33.675221+0000", 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.7510803689999999, 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.064607482999999993, 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 81700 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675221+0000", 2024-04-29T14:14:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675221+0000", 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675223+0000", 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.547e-06 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675275+0000", 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2290999999999998e-05 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675287+0000", 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2500000000000001e-05 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675295+0000", 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0700000000000001e-06 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675327+0000", 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2981000000000002e-05 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675403+0000", 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5306e-05 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 11", 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.675479+0000", 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6588999999999995e-05 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677292+0000", 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001812623 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739687+0000", 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.062394647999999997 2024-04-29T14:14:37.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739784+0000", 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7827999999999999e-05 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739829+0000", 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4100000000000001e-05 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(client.25063.0:81693 3.2 e232/219 ondisk, result = 0)", 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:33.677339+0000", 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.7489625389999999, 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.062192142999999998, 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677339+0000", 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677339+0000", 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677340+0000", 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1659999999999999e-06 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677345+0000", 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4229999999999998e-06 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677351+0000", 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6159999999999998e-06 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677356+0000", 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5920000000000002e-06 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677404+0000", 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7790000000000002e-05 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.677412+0000", 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9810000000000003e-06 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739531+0000", 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.062119575000000003 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(client.25063.0:81700 3.7 e232/206 ondisk, result = 0)", 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:14:33.679293+0000", 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.747008535, 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.060542547000000002, 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.679293+0000", 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.679293+0000", 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.679294+0000", 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.269e-06 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.679301+0000", 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1040000000000001e-06 2024-04-29T14:14:37.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739486+0000", 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.060184509999999997 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739510+0000", 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4363e-05 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739645+0000", 2024-04-29T14:14:37.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134759 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739668+0000", 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3263000000000001e-05 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:14:33.739835+0000", 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000167279 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:14:37.531 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:14:37.532 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:37.563 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:37.590 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:37.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:37.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:37.749 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:37.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:37.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:37.881 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:37.915 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:37.938 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:37.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:38.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:14:38.085 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:14:38.086 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:14:38.091 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:38.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:14:38.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:14:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:38.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:14:38.146 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:14:38.173 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:14:38.202 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:14:38.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 90021 90005 156.245 110.438 0.0069764 0.00639882 2024-04-29T14:14:38.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:38.230 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:14:38.262 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:14:38.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:38 smithi040 ceph-mon[39581]: pgmap v630: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-29T14:14:38.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4275863877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:38.280 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:38.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:14:38.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:38 smithi194 ceph-mon[44089]: pgmap v630: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-29T14:14:38.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4275863877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:38.328 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:14:38.356 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:14:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:14:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:14:38.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:38 smithi121 ceph-mon[44533]: pgmap v630: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-29T14:14:38.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4275863877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:38.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:38.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:38.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:38.552 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:38.613 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:38.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:38.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:38.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:38.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:38.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:38.937 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.120 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 91838 91822 155.091 113.562 0.00624156 0.0064463 2024-04-29T14:14:39.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:39.694 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:39.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:39.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:39.842 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:39.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:39.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:39.978 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:40.012 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:40.037 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:40.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:40.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:40.200 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:40.207 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 93647 93631 153.985 113.062 0.00604847 0.00649192 2024-04-29T14:14:40.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:40.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:40.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:40.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:40 smithi194 ceph-mon[44089]: pgmap v631: 17 pgs: 17 active+clean; 5.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-29T14:14:40.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:40.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:40.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:40 smithi040 ceph-mon[39581]: pgmap v631: 17 pgs: 17 active+clean; 5.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-29T14:14:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:40 smithi121 ceph-mon[44533]: pgmap v631: 17 pgs: 17 active+clean; 5.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-29T14:14:40.379 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:40.406 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:40.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:40.512 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:40.562 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:40.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:40.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:40.797 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:40.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:40.905 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:40.935 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:40.957 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:40.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:41.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:41.113 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:41.136 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:14:41.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:41.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:41.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 95331 95315 152.735 105.25 0.0125801 0.00654597 2024-04-29T14:14:41.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:41.287 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:41.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2421431016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:41.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2.service' 2024-04-29T14:14:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2421431016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:41.406 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2421431016' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:41.407 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:14:41 smithi040 systemd[1]: ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2.service: Succeeded. 2024-04-29T14:14:41.407 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:14:41 smithi040 systemd[1]: Stopped Ceph osd.2 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:14:41.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T14:14:41.457 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-29T14:14:41.457 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-04-29T14:14:41.457 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi040.front.sepia.ceph.com 2024-04-29T14:14:41.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.2 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-04-29T14:14:41.543 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:41.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:41.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:41.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:41.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:41.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:41.879 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:41.914 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:41.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:42.016 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.2/config 2024-04-29T14:14:42.020 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:42.049 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:42.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:14:42.075 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:42.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:42.181 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:42.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:14:42.215279 min lat: 0.00103764 max lat: 2.08667 avg lat: 0.00659586 2024-04-29T14:14:42.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:14:42.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 97036 97020 151.581 106.562 0.0141555 0.00659586 2024-04-29T14:14:42.212 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:42.234 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:42.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:42 smithi194 ceph-mon[44089]: pgmap v632: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-29T14:14:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:42 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2421431016' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:42 smithi194 ceph-mon[44089]: osdmap e233: 12 total, 11 up, 10 in 2024-04-29T14:14:42.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:42 smithi194 ceph-mon[44089]: osdmap e234: 12 total, 11 up, 10 in 2024-04-29T14:14:42.339 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:42 smithi040 ceph-mon[39581]: pgmap v632: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-29T14:14:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:42 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:42.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2421431016' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:42.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:42 smithi040 ceph-mon[39581]: osdmap e233: 12 total, 11 up, 10 in 2024-04-29T14:14:42.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:42 smithi040 ceph-mon[39581]: osdmap e234: 12 total, 11 up, 10 in 2024-04-29T14:14:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:42.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:42 smithi121 ceph-mon[44533]: pgmap v632: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-29T14:14:42.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:42 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:42.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2421431016' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:42.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:42 smithi121 ceph-mon[44533]: osdmap e233: 12 total, 11 up, 10 in 2024-04-29T14:14:42.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:42 smithi121 ceph-mon[44533]: osdmap e234: 12 total, 11 up, 10 in 2024-04-29T14:14:42.401 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:42.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:14:42.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:14:42.450 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:14:42.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:42.538 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:42.563 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:42.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:42.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:42.694 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:42.714 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:42.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:42.819 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:42.850 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:42.874 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:42.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:42.980 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:43.011 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:43.043 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:43.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:43.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:43.150 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:43.182 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:43.207 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:43.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 98550 98534 150.191 94.625 0.00605248 0.00665689 2024-04-29T14:14:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:43.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:43.313 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:44.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 15 100276 100261 149.185 107.938 0.00742158 0.00670178 2024-04-29T14:14:44.274 INFO:teuthology.orchestra.run.smithi040.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:14:44.311 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:44.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:44.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:44.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:14:44.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:44 smithi040 ceph-mon[39581]: pgmap v635: 17 pgs: 17 active+clean; 5.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:14:44.492 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:44 smithi121 ceph-mon[44533]: pgmap v635: 17 pgs: 17 active+clean; 5.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:14:44.501 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:44.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:44 smithi194 ceph-mon[44089]: pgmap v635: 17 pgs: 17 active+clean; 5.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:14:44.577 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:14:44.578 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:14:44.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:44.684 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:44.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:44.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:44.893 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:44.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:44.949 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:45.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:45.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:14:45.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 16 101841 101825 147.989 97.75 0.00775076 0.00675564 2024-04-29T14:14:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:45.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:45.320 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:45.351 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:45.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:14:45.373 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:45.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:45.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:45.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:45.539 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:45.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:45.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:45.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:45.738 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:45.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:45.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:45.805 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:14:45.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:45.911 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:46.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:46.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 15 103082 103067 146.389 77.625 0.0134599 0.00682985 2024-04-29T14:14:46.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:46.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:46.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:46.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:46 smithi121 ceph-mon[44533]: pgmap v636: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:14:46.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:46.523 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:46.525 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:46 smithi040 ceph-mon[39581]: pgmap v636: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:14:46.562 INFO:tasks.thrashosds.thrasher:No PGs found for osd.2 2024-04-29T14:14:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:46 smithi194 ceph-mon[44089]: pgmap v636: 17 pgs: 17 active+clean; 5.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:14:46.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:46.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:46.789 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:46.821 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:46.846 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:46.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:46.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:46.949 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:46.985 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:47.008 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:47.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:47.039 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:47.112 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:47.149 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:47.173 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:47.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:47.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 104654 104638 145.318 98.1875 0.00502495 0.00687994 2024-04-29T14:14:47.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:47.335 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:47.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:47.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:47.531 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:47.562 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:47.585 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:47.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:47.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:47.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:47.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:47.821 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:14:47.821 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:14:47.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:47.924 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:47.949 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:47.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:48.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:48.111 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:48.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 106188 106172 144.243 95.875 0.00679548 0.00693179 2024-04-29T14:14:48.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:48 smithi194 ceph-mon[44089]: pgmap v637: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-29T14:14:48.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2208862484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:48.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:48.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:48 smithi040 ceph-mon[39581]: pgmap v637: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-29T14:14:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2208862484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:48.538 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:48 smithi121 ceph-mon[44533]: pgmap v637: 17 pgs: 17 active+clean; 6.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-29T14:14:48.538 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2208862484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:48.539 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:48.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:14:48.654 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:14:48.654 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:14:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:14:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:14:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:48.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:48.978 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:49.007 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:49.029 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:49.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:14:49.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:49.170 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:49.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 107709 107693 143.196 95.0625 0.00860418 0.00698223 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osdmap e235: 12 total, 11 up, 10 in 2024-04-29T14:14:49.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:49.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.419 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.419 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.419 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.419 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osdmap e235: 12 total, 11 up, 10 in 2024-04-29T14:14:49.419 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:14:49.451 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:14:49.465 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:49.488 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:49.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:49.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osdmap e235: 12 total, 11 up, 10 in 2024-04-29T14:14:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:49.641 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:49.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:49.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:50.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:50.069 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:50.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:50.201 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:50.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 16 109260 109244 142.233 96.9375 0.00409243 0.00702919 2024-04-29T14:14:50.235 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:50.260 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:50.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:50.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:50 smithi121 ceph-mon[44533]: pgmap v639: 17 pgs: 17 active+clean; 6.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-04-29T14:14:50.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:50 smithi040 ceph-mon[39581]: pgmap v639: 17 pgs: 17 active+clean; 6.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-04-29T14:14:50.365 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:50.422 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:50.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:50.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:50.562 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:50 smithi194 ceph-mon[44089]: pgmap v639: 17 pgs: 17 active+clean; 6.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-04-29T14:14:50.600 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:50.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:50.704 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:50.734 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:50.757 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:50.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:50.862 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:50.896 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:50.920 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:50.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:51.001 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:14:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:14:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:14:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:14:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14802 2.2 2:4cff85a0:::benchmark_data_smithi194_79000_object14801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.146831+0000", 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 325.857992244, 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20478190399999999, 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14802 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146831+0000", 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146831+0000", 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146832+0000", 2024-04-29T14:14:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5900000000000005e-07 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.146990+0000", 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015835799999999999 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147000+0000", 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6080000000000002e-06 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147007+0000", 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8190000000000003e-06 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147051+0000", 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4249999999999998e-05 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147135+0000", 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3642999999999999e-05 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147225+0000", 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9920000000000006e-05 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.148359+0000", 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001134501 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.351534+0000", 2024-04-29T14:14:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20317476800000001 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.351570+0000", 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000002e-05 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.351613+0000", 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3328000000000003e-05 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14803 2.2 2:455db739:::benchmark_data_smithi194_79000_object14802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.147539+0000", 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 325.85728458900002, 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20464069300000001, 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14803 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147539+0000", 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147539+0000", 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147540+0000", 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1739999999999999e-06 2024-04-29T14:14:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147597+0000", 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6879000000000002e-05 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147607+0000", 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7210000000000006e-06 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147612+0000", 2024-04-29T14:14:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6999999999999996e-06 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147729+0000", 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117219 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147839+0000", 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010945100000000001 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:14:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.147973+0000", 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013422600000000001 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.148522+0000", 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054840499999999996 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352103+0000", 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20358119599999999 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352137+0000", 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4239999999999997e-05 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352179+0000", 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2481999999999997e-05 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14822 2.2 2:41dfc9f4:::benchmark_data_smithi194_79000_object14821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.151076+0000", 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 325.85374767899998, 2024-04-29T14:14:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201209203, 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14822 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151076+0000", 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151076+0000", 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151077+0000", 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.314e-06 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151160+0000", 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3131000000000001e-05 2024-04-29T14:14:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151170+0000", 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5449999999999992e-06 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151175+0000", 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6300000000000003e-06 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151220+0000", 2024-04-29T14:14:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4917999999999999e-05 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151345+0000", 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012466500000000001 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.151486+0000", 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014131899999999999 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.152019+0000", 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00053267199999999998 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352224+0000", 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20020465600000001 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352259+0000", 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5544999999999999e-05 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352285+0000", 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5808000000000001e-05 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14832 2.2 2:450d2096:::benchmark_data_smithi194_79000_object14831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.153431+0000", 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 325.85139237800001, 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199426829, 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14832 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153431+0000", 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153431+0000", 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153438+0000", 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.691e-06 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153504+0000", 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6223000000000006e-05 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153513+0000", 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4590000000000004e-06 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153519+0000", 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8109999999999999e-06 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153570+0000", 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1270999999999998e-05 2024-04-29T14:14:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153678+0000", 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010785600000000001 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.153791+0000", 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112497 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.154367+0000", 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057592799999999997 2024-04-29T14:14:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352798+0000", 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.198431259 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352832+0000", 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4286999999999999e-05 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352858+0000", 2024-04-29T14:14:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5547000000000001e-05 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:14843 2.2 2:4e70e968:::benchmark_data_smithi194_79000_object14842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:09:25.155295+0000", 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 325.84952873399999, 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.197668766, 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 14843 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155295+0000", 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155295+0000", 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155296+0000", 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.074e-06 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155373+0000", 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6859000000000004e-05 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155383+0000", 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0020000000000001e-05 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155391+0000", 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0330000000000008e-06 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155428+0000", 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7508000000000001e-05 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155504+0000", 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6210000000000004e-05 2024-04-29T14:14:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.155600+0000", 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5660000000000002e-05 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.156065+0000", 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046479099999999999 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352893+0000", 2024-04-29T14:14:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.196827797 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352926+0000", 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3729999999999997e-05 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:09:25.352963+0000", 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7083999999999999e-05 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319935 2.2 2.6dd562 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:06.092889+0000", 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.91193477500001, 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87845669999999998, 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319935 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092889+0000", 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092889+0000", 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092891+0000", 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.289e-06 2024-04-29T14:14:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092897+0000", 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7880000000000003e-06 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092907+0000", 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0686999999999999e-05 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092916+0000", 2024-04-29T14:14:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6619999999999994e-06 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092976+0000", 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0049000000000001e-05 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.092986+0000", 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5799999999999998e-06 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971298+0000", 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87831266799999996 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971345+0000", 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6977000000000001e-05 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319938 2.2 2.8ec7b962 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:06.093411+0000", 2024-04-29T14:14:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.911412328, 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87798137600000004, 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319938 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093411+0000", 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093411+0000", 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093412+0000", 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7499999999999998e-07 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093415+0000", 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.269e-06 2024-04-29T14:14:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093424+0000", 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1609999999999992e-06 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093431+0000", 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6850000000000004e-06 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093477+0000", 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5710000000000001e-05 2024-04-29T14:14:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.093485+0000", 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5059999999999995e-06 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971384+0000", 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87789858300000001 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971392+0000", 2024-04-29T14:14:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4870000000000002e-06 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319952 2.2 2.47832442 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:06.109725+0000", 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.895098448, 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.861705524, 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319952 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109725+0000", 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109725+0000", 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109727+0000", 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4549999999999998e-06 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109734+0000", 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6579999999999996e-06 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109747+0000", 2024-04-29T14:14:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3301e-05 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109757+0000", 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.008e-05 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109827+0000", 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9740000000000007e-05 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.109836+0000", 2024-04-29T14:14:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7450000000000001e-06 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971423+0000", 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.86158700499999996 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971430+0000", 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5399999999999998e-06 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319953 2.2 2.f22415c2 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:06.111429+0000", 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.89339483400001, 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.86004072399999998, 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319953 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111429+0000", 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111429+0000", 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111430+0000", 2024-04-29T14:14:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3740000000000001e-06 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111437+0000", 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6599999999999998e-06 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111446+0000", 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6949999999999995e-06 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111455+0000", 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8349999999999993e-06 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111509+0000", 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3523000000000002e-05 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.111516+0000", 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6860000000000006e-06 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971462+0000", 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85994551100000005 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971469+0000", 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4399999999999999e-06 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319955 2.2 2.928dba02 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:06.112966+0000", 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.891857567, 2024-04-29T14:14:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85854339700000004, 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319955 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.112966+0000", 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.112966+0000", 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.112967+0000", 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4699999999999999e-06 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.112972+0000", 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0719999999999999e-06 2024-04-29T14:14:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.112982+0000", 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5200000000000003e-06 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.112988+0000", 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.942e-06 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.113031+0000", 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-29T14:14:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.113037+0000", 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5359999999999998e-06 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971500+0000", 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85846307700000002 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971509+0000", 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7129999999999994e-06 2024-04-29T14:14:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319960 2.2 2.90bd2e62 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:06.118749+0000", 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.88607439, 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85279547200000005, 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319960 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118749+0000", 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118749+0000", 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118751+0000", 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.854e-06 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118758+0000", 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.939e-06 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118781+0000", 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2957000000000001e-05 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118788+0000", 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.849e-06 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118832+0000", 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4715999999999997e-05 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.118840+0000", 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5859999999999999e-06 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971538+0000", 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85269762800000004 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:06.971544+0000", 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9430000000000003e-06 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319935 2.2 2:46abb600:::benchmark_data_smithi194_79000_object155674:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e191)", 2024-04-29T14:14:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:08.001017+0000", 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.003806345, 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28288053800000001, 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319935 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001017+0000", 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001017+0000", 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001020+0000", 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.118e-06 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001029+0000", 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.329e-06 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001046+0000", 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6884999999999998e-05 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001058+0000", 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1667e-05 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001119+0000", 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1458999999999996e-05 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001128+0000", 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4290000000000001e-06 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.271240+0000", 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27011202400000001 2024-04-29T14:14:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.271387+0000", 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014669300000000001 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.271930+0000", 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054381799999999995 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.275460+0000", 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035291110000000001 2024-04-29T14:14:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.279423+0000", 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039635410000000001 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.283699+0000", 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042763260000000004 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.283778+0000", 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8565000000000004e-05 2024-04-29T14:14:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.283898+0000", 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119573 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319938 2.2 2:469de371:::benchmark_data_smithi194_79000_object155677:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e191)", 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:08.001087+0000", 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.00373687, 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28736461200000002, 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319938 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001087+0000", 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001087+0000", 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001087+0000", 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9100000000000003e-07 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001090+0000", 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0130000000000002e-06 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001095+0000", 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0180000000000001e-06 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001101+0000", 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3519999999999997e-06 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001157+0000", 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6777999999999997e-05 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001162+0000", 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9429999999999999e-06 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.275501+0000", 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27433834099999999 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.275579+0000", 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8330000000000004e-05 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.276058+0000", 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047887500000000003 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.283505+0000", 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074475460000000002 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.288174+0000", 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046688670000000002 2024-04-29T14:14:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.288349+0000", 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017473900000000001 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.288389+0000", 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0445000000000003e-05 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.288451+0000", 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1674000000000006e-05 2024-04-29T14:14:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319952 2.2 2:4224c1e2:::benchmark_data_smithi194_79000_object155691:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e191)", 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:08.001106+0000", 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.003717088, 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.296956216, 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319952 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001106+0000", 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001106+0000", 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001107+0000", 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0200000000000003e-07 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001109+0000", 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.182e-06 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001112+0000", 2024-04-29T14:14:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.309e-06 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001115+0000", 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1609999999999999e-06 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001198+0000", 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2615999999999996e-05 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001202+0000", 2024-04-29T14:14:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2379999999999999e-06 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.279492+0000", 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27828940000000002 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.279590+0000", 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8581999999999999e-05 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.280152+0000", 2024-04-29T14:14:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056190899999999998 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.283949+0000", 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037973759999999999 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.288183+0000", 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042336170000000003 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.297960+0000", 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097773740000000001 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.297997+0000", 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6409000000000003e-05 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298063+0000", 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5641000000000004e-05 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319953 2.2 2:43a8244f:::benchmark_data_smithi194_79000_object155692:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e191)", 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:08.001119+0000", 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.00370419799999, 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29711696700000001, 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319953 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001119+0000", 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001119+0000", 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001120+0000", 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3799999999999998e-07 2024-04-29T14:14:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001121+0000", 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4529999999999999e-06 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001124+0000", 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0800000000000002e-06 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001127+0000", 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1769999999999998e-06 2024-04-29T14:14:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001245+0000", 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011784400000000001 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001249+0000", 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8840000000000004e-06 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.283988+0000", 2024-04-29T14:14:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28273938700000001 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.284072+0000", 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3939999999999999e-05 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.284566+0000", 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049374100000000001 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.297919+0000", 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013352833 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.297933+0000", 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4324e-05 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298152+0000", 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021910199999999999 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298183+0000", 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0255000000000001e-05 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298236+0000", 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3609000000000001e-05 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319955 2.2 2:405db149:::benchmark_data_smithi194_79000_object155694:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e191)", 2024-04-29T14:14:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:08.001130+0000", 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.00369295500001, 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29726429999999998, 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319955 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001130+0000", 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001130+0000", 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001131+0000", 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9999999999999999e-07 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001132+0000", 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.389e-06 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001138+0000", 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6760000000000004e-06 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001141+0000", 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0400000000000001e-06 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001306+0000", 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016565899999999999 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001311+0000", 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.143e-06 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.288504+0000", 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28719324299999999 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.288623+0000", 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119234 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.289279+0000", 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065553599999999997 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298107+0000", 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088282610000000004 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298121+0000", 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4302e-05 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298313+0000", 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019148200000000001 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298342+0000", 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9292999999999999e-05 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298395+0000", 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2741999999999997e-05 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:319960 2.2 2:4674bd09:::benchmark_data_smithi194_79000_object155699:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e191)", 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:08.001150+0000", 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.00367300400001, 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29736504800000002, 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319960 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001150+0000", 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001150+0000", 2024-04-29T14:14:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001151+0000", 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.53e-07 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001152+0000", 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5859999999999999e-06 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001156+0000", 2024-04-29T14:14:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.495e-06 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001160+0000", 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8190000000000002e-06 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001328+0000", 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000167944 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:14:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.001331+0000", 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8589999999999998e-06 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.293142+0000", 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29181016599999998 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.293255+0000", 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113552 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-04-29T14:14:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.293957+0000", 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070144399999999996 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298272+0000", 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043149850000000003 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298282+0000", 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0879999999999999e-05 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298431+0000", 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014859399999999999 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298462+0000", 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1189000000000002e-05 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:08.298515+0000", 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3182e-05 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:320883 2.16 2:6821cdf7:::benchmark_data_smithi194_79000_object156622:head [delete] snapc 0=[] ondisk+write+known_if_redirected e191)", 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:09.073872+0000", 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.930950882, 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20769214, 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 320883 2024-04-29T14:14:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.073872+0000", 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.073872+0000", 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.073875+0000", 2024-04-29T14:14:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9160000000000001e-06 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.073882+0000", 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0679999999999999e-06 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.073898+0000", 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5973000000000001e-05 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.073907+0000", 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5990000000000002e-06 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.073972+0000", 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4583000000000002e-05 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.074059+0000", 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7225999999999995e-05 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.074641+0000", 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058170499999999996 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.078446+0000", 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003805148 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.281466+0000", 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203020012 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.281521+0000", 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5636000000000001e-05 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.281565+0000", 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3273999999999999e-05 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:320917 2.2 2:44a496b8:::benchmark_data_smithi194_79000_object156656:head [delete] snapc 0=[] ondisk+write+known_if_redirected e191)", 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:09.102706+0000", 2024-04-29T14:14:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.902117809, 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19377803399999999, 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 320917 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102706+0000", 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102706+0000", 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102706+0000", 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4099999999999999e-07 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102708+0000", 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.148e-06 2024-04-29T14:14:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102713+0000", 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8250000000000004e-06 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102718+0000", 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.685e-06 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102758+0000", 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0389000000000002e-05 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.102855+0000", 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7198e-05 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.103603+0000", 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074769699999999997 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.107672+0000", 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040692920000000004 2024-04-29T14:14:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.114565+0000", 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006892319 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.296366+0000", 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18180091600000001 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.296411+0000", 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5837999999999997e-05 2024-04-29T14:14:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.296484+0000", 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2186000000000003e-05 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:320920 2.16 2:6877689c:::benchmark_data_smithi194_79000_object156659:head [delete] snapc 0=[] ondisk+write+known_if_redirected e191)", 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:13:09.106842+0000", 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.89798145100001, 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19173105800000001, 2024-04-29T14:14:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 320920 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.106842+0000", 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.106842+0000", 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.106844+0000", 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.855e-06 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.106849+0000", 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1229999999999999e-06 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:14:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.106858+0000", 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4609999999999997e-06 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.106866+0000", 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4869999999999996e-06 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.106922+0000", 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6325e-05 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:14:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.107058+0000", 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000135544 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.107654+0000", 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059610299999999995 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.111299+0000", 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036447179999999999 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.298476+0000", 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18717737400000001 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.298529+0000", 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2638e-05 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:13:09.298573+0000", 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.443e-05 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:14:51.116 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:14:51.116 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:14:51.116 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:51.153 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:51.179 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:14:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:51.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:51.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 110757 110741 141.239 93.5625 0.00622945 0.00707898 2024-04-29T14:14:51.282 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:51.318 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:51.344 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:51.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:51.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/296905332' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:51.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:51 smithi121 ceph-mon[44533]: from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:51.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:51 smithi121 ceph-mon[44533]: pgmap v640: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:14:51.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:51.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:14:51.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/296905332' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:51.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:51 smithi040 ceph-mon[39581]: from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:51.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:51 smithi040 ceph-mon[39581]: pgmap v640: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:14:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:51.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 8, 9, 10, 11, 6] out_osds: [1, 2] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:14:51.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:14:51.566 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-29T14:14:51.566 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:14:51.566 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:14:51.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-04-29T14:14:51.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/296905332' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:51.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:51 smithi194 ceph-mon[44089]: from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:14:51.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:51 smithi194 ceph-mon[44089]: pgmap v640: 17 pgs: 17 active+clean; 6.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:14:51.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:14:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:52.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:14:52.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:52.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 112036 112020 140.013 79.9375 0.00503158 0.00714033 2024-04-29T14:14:52.253 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:52.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:52.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:52 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:52.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:52 smithi040 ceph-mon[39581]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:14:52.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:52 smithi040 ceph-mon[39581]: osdmap e236: 12 total, 11 up, 10 in 2024-04-29T14:14:52.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:52 smithi040 ceph-mon[39581]: 3.2 scrub starts 2024-04-29T14:14:52.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:52.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:52 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:52.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:52 smithi121 ceph-mon[44533]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:14:52.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:52 smithi121 ceph-mon[44533]: osdmap e236: 12 total, 11 up, 10 in 2024-04-29T14:14:52.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:52 smithi121 ceph-mon[44533]: 3.2 scrub starts 2024-04-29T14:14:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:52.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:52.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:52 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:14:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:52 smithi194 ceph-mon[44089]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:14:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:52 smithi194 ceph-mon[44089]: osdmap e236: 12 total, 11 up, 10 in 2024-04-29T14:14:52.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:52 smithi194 ceph-mon[44089]: 3.2 scrub starts 2024-04-29T14:14:52.570 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:52.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:52.755 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:52.949 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:14:52.993 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:14:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:52.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:14:53.096 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:53.129 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:53.151 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:53.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:53.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 113401 113385 138.94 85.3125 0.00381384 0.00719596 2024-04-29T14:14:53.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:53.305 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:53.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:53 smithi121 ceph-mon[44533]: 3.a scrub starts 2024-04-29T14:14:53.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:53 smithi121 ceph-mon[44533]: pgmap v642: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:14:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:53 smithi040 ceph-mon[39581]: 3.a scrub starts 2024-04-29T14:14:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:53 smithi040 ceph-mon[39581]: pgmap v642: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:14:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:53 smithi194 ceph-mon[44089]: 3.a scrub starts 2024-04-29T14:14:53.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:53 smithi194 ceph-mon[44089]: pgmap v642: 17 pgs: 17 active+clean; 6.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:14:54.078 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:14:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:54.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:54.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:14:54.160 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:14:54.189 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:14:54.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:54.208 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 114800 114784 137.95 87.4375 0.00985755 0.00724774 2024-04-29T14:14:54.238 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:14:54.240 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:54.275 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:14:54.326 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:14:54.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:54.359 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:14:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:54.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:54.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:54 smithi121 ceph-mon[44533]: 3.2 scrub ok 2024-04-29T14:14:54.366 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1898456574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:54.395 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:14:54.420 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:14:54.449 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:14:54.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:54.473 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:54.473 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:14:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:14:54.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:54 smithi040 ceph-mon[39581]: 3.2 scrub ok 2024-04-29T14:14:54.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1898456574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:54.514 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:54.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:54 smithi194 ceph-mon[44089]: 3.2 scrub ok 2024-04-29T14:14:54.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1898456574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:14:54.597 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:54.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:54.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:54.774 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:54.788 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:14:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-29T14:14:54.789 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:14:55.046 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:55.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:55.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:55.187 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:55.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 16 116049 116033 136.82 78.0625 0.0198377 0.00730725 2024-04-29T14:14:55.209 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:55.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:55.314 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:55.352 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: 3.a scrub ok 2024-04-29T14:14:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:14:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:14:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osdmap e237: 12 total, 11 up, 10 in 2024-04-29T14:14:55.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: 3.5 scrub starts 2024-04-29T14:14:55.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: pgmap v644: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:14:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:14:55.457 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:55.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:55.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: 3.a scrub ok 2024-04-29T14:14:55.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osdmap e237: 12 total, 11 up, 10 in 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: 3.5 scrub starts 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: pgmap v644: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:14:55.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:14:55.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:14:55.562 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:55.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: 3.a scrub ok 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osdmap e237: 12 total, 11 up, 10 in 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: 3.5 scrub starts 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:14:55.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: pgmap v644: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:14:55.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:14:55.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:14:55.591 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:55.616 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:55.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:55.720 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:55.752 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:55.774 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:55.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:55.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:55.915 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:55.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:14:55.950 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:14:56.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:56.199 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:56 smithi040 ceph-mon[39581]: 3.3 scrub starts 2024-04-29T14:14:56.199 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:56 smithi040 ceph-mon[39581]: osdmap e238: 12 total, 11 up, 10 in 2024-04-29T14:14:56.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 117266 117250 135.694 76.0625 0.00371826 0.00736829 2024-04-29T14:14:56.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:56.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:56.320 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:56.349 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:56.370 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:56.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:56 smithi121 ceph-mon[44533]: 3.3 scrub starts 2024-04-29T14:14:56.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:56 smithi121 ceph-mon[44533]: osdmap e238: 12 total, 11 up, 10 in 2024-04-29T14:14:56.409 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:14:56.410 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:14:56.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:56.514 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:56 smithi194 ceph-mon[44089]: 3.3 scrub starts 2024-04-29T14:14:56.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:56 smithi194 ceph-mon[44089]: osdmap e238: 12 total, 11 up, 10 in 2024-04-29T14:14:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:56.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:56.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:14:56.856 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:14:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:56.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:14:57.092 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:14:57.120 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:57.142 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:14:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:57.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:14:57.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:14:57.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 118614 118598 134.759 84.25 0.0183215 0.00741958 2024-04-29T14:14:57.245 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:57.275 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:57.308 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:57.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:57 smithi121 ceph-mon[44533]: 3.5 scrub ok 2024-04-29T14:14:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:57 smithi121 ceph-mon[44533]: pgmap v646: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-29T14:14:57.412 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:14:57.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:57 smithi194 ceph-mon[44089]: 3.5 scrub ok 2024-04-29T14:14:57.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:57 smithi194 ceph-mon[44089]: pgmap v646: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-29T14:14:57.441 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:57.464 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:14:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:57.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:14:57.568 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:14:57.598 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:57.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:57 smithi040 ceph-mon[39581]: 3.5 scrub ok 2024-04-29T14:14:57.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:57 smithi040 ceph-mon[39581]: pgmap v646: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-29T14:14:57.621 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:14:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:57.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:14:57.640 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:14:57.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:14:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:14:57.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:57.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:14:57.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:14:58.041 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:14:58.071 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:58.093 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:14:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:58.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:14:58.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:14:58.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 119906 119890 133.794 80.75 0.00891939 0.00747297 2024-04-29T14:14:58.230 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:14:58.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:58 smithi040 ceph-mon[39581]: 3.3 scrub ok 2024-04-29T14:14:58.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:58 smithi040 ceph-mon[39581]: 3.e scrub starts 2024-04-29T14:14:58.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3797172649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:58.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:58 smithi040 ceph-mon[39581]: from='client.25132 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:58.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:58 smithi194 ceph-mon[44089]: 3.3 scrub ok 2024-04-29T14:14:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:58 smithi194 ceph-mon[44089]: 3.e scrub starts 2024-04-29T14:14:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3797172649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:58 smithi194 ceph-mon[44089]: from='client.25132 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:58 smithi121 ceph-mon[44533]: 3.3 scrub ok 2024-04-29T14:14:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:58 smithi121 ceph-mon[44533]: 3.e scrub starts 2024-04-29T14:14:58.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3797172649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:58 smithi121 ceph-mon[44533]: from='client.25132 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:14:58.949 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:14:59.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:14:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:59.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:14:59.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:14:59.166 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:59.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 15 121275 121260 132.949 85.625 0.00673144 0.00752034 2024-04-29T14:14:59.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:59 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:14:59.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:59 smithi121 ceph-mon[44533]: from='client.25132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:59.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:59 smithi121 ceph-mon[44533]: osdmap e239: 12 total, 11 up, 10 in 2024-04-29T14:14:59.218 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:14:59 smithi121 ceph-mon[44533]: pgmap v648: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:14:59.276 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:14:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:59.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:14:59.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:14:59.435 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:59.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:14:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:14:59.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:14:59.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:59 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:14:59.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:59 smithi040 ceph-mon[39581]: from='client.25132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:59.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:59 smithi040 ceph-mon[39581]: osdmap e239: 12 total, 11 up, 10 in 2024-04-29T14:14:59.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:14:59 smithi040 ceph-mon[39581]: pgmap v648: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:14:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:59 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:14:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:59 smithi194 ceph-mon[44089]: from='client.25132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:14:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:59 smithi194 ceph-mon[44089]: osdmap e239: 12 total, 11 up, 10 in 2024-04-29T14:14:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:14:59 smithi194 ceph-mon[44089]: pgmap v648: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:14:59.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:14:59.638 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:14:59.662 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:14:59.662 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:00.096 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:00.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:00.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:00.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 122722 122706 132.215 90.375 0.0123423 0.0075622 2024-04-29T14:15:00.246 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:00.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:00 smithi121 ceph-mon[44533]: 3.e scrub ok 2024-04-29T14:15:00.336 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:00.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:00 smithi040 ceph-mon[39581]: 3.e scrub ok 2024-04-29T14:15:00.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:15:00.441 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:00.471 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:00.492 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:00.494 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:00 smithi194 ceph-mon[44089]: 3.e scrub ok 2024-04-29T14:15:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:00.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:00.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:00.652 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:00.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:00.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:00.784 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 8, 9, 10, 11, 6] out_osds: [1, 2] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:15:00.784 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:15:00.784 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:15:00.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:15:00.791 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:00.818 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:00.829 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:15:00.845 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:00.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:00.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:00.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:15:01.003 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:01.105 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:01.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:01.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 123998 123982 131.326 79.75 0.00948993 0.0076136 2024-04-29T14:15:01.215 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:01.240 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:01 smithi194 ceph-mon[44089]: 3.7 scrub starts 2024-04-29T14:15:01.240 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:01 smithi194 ceph-mon[44089]: pgmap v649: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-29T14:15:01.285 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:01.330 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:01 smithi040 ceph-mon[39581]: 3.7 scrub starts 2024-04-29T14:15:01.330 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:01 smithi040 ceph-mon[39581]: pgmap v649: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-29T14:15:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:01.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:01.341 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:15:01.380 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:15:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:01.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:01.432 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:15:01.485 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:15:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:01.530 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:15:01.559 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:15:01.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:15:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:01 smithi121 ceph-mon[44533]: 3.7 scrub starts 2024-04-29T14:15:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:01 smithi121 ceph-mon[44533]: pgmap v649: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-29T14:15:01.630 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:15:01.662 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:15:01.699 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:01.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:01.729 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:15:01.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:15:01.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:15:01.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:01.846 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:02.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:02.206 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:02.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:15:02.216888 min lat: 0.00103764 max lat: 2.08667 avg lat: 0.00766321 2024-04-29T14:15:02.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:15:02.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 125277 125261 130.469 79.9375 0.0259364 0.00766321 2024-04-29T14:15:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:02.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2129411891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: 3.0 scrub starts 2024-04-29T14:15:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:02 smithi040 ceph-mon[39581]: osdmap e240: 12 total, 11 up, 10 in 2024-04-29T14:15:02.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2129411891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:02.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: 3.0 scrub starts 2024-04-29T14:15:02.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:02 smithi121 ceph-mon[44533]: osdmap e240: 12 total, 11 up, 10 in 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2129411891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: 3.0 scrub starts 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:02 smithi194 ceph-mon[44089]: osdmap e240: 12 total, 11 up, 10 in 2024-04-29T14:15:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:15:02.588 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:15:02.996 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:03.027 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:03.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:03.086 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:15:03.086 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:15:03.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:15:03.086 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:166439 2.11 2.c7b87731 (undecoded) ondisk+write+known_if_redirected e139)", 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:10:56.243836+0000", 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 246.84449536299999, 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66351022999999998, 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.087 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:15:03.088 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.088 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.088 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.088 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 166439 2024-04-29T14:15:03.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.089 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243836+0000", 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243836+0000", 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243838+0000", 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4190000000000001e-06 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243842+0000", 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.143e-06 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243853+0000", 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.064e-05 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243861+0000", 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3100000000000001e-06 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243902+0000", 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1468999999999999e-05 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243908+0000", 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3310000000000003e-06 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.907335+0000", 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66342681599999997 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.907347+0000", 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2102e-05 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:172145 2.17 2:eca704d4:::benchmark_data_smithi194_79000_object7884:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.093 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:05.819419+0000", 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 237.26891282299999, 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12007738599999999, 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 172145 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.819419+0000", 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.819419+0000", 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.819421+0000", 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1349999999999999e-06 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.819427+0000", 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9510000000000003e-06 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.819437+0000", 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0219e-05 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.819445+0000", 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8460000000000008e-06 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.854549+0000", 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035104163000000001 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.854685+0000", 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013558900000000001 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.855263+0000", 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057835999999999999 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.862579+0000", 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073160739999999997 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.939404+0000", 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076824315000000004 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.939445+0000", 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0951999999999997e-05 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:05.939496+0000", 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1782e-05 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173448 2.17 2:eeb08944:::benchmark_data_smithi194_79000_object9187:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:07.900500+0000", 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 235.18783196300001, 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.163539032, 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.099 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173448 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.900500+0000", 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.900500+0000", 2024-04-29T14:15:03.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.900503+0000", 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7470000000000001e-06 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.900508+0000", 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6720000000000001e-06 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.900518+0000", 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0094e-05 2024-04-29T14:15:03.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.900526+0000", 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5329999999999997e-06 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.966217+0000", 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.065691319999999997 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.966301+0000", 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3430999999999995e-05 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.966813+0000", 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051202300000000001 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:07.971007+0000", 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041945139999999999 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.063955+0000", 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.092947973000000003 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.063991+0000", 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6327e-05 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.064039+0000", 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7398000000000003e-05 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173610 2.17 2:ea6ae847:::benchmark_data_smithi194_79000_object9349:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.95255469899999, 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.234702578, 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173610 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.105 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135779+0000", 2024-04-29T14:15:03.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2069999999999998e-06 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135785+0000", 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5849999999999999e-06 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135800+0000", 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5573000000000002e-05 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135809+0000", 2024-04-29T14:15:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2870000000000004e-06 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.170786+0000", 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034977716999999998 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.170861+0000", 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4418000000000005e-05 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.171114+0000", 2024-04-29T14:15:03.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025352699999999997 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184325+0000", 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01321051 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370366+0000", 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.186041027 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370424+0000", 2024-04-29T14:15:03.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7645999999999997e-05 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370480+0000", 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6081000000000001e-05 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173617 2.17 2:ea4c2311:::benchmark_data_smithi194_79000_object9356:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.943594209, 2024-04-29T14:15:03.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24862530199999999, 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173617 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144739+0000", 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8789999999999999e-06 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144745+0000", 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7599999999999999e-06 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144755+0000", 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7969999999999995e-06 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144761+0000", 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1140000000000002e-06 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.173690+0000", 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028928834 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.173816+0000", 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126067 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.174276+0000", 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045987699999999997 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184330+0000", 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010054110999999999 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393250+0000", 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208919664 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393302+0000", 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1860999999999997e-05 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393363+0000", 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1338000000000001e-05 2024-04-29T14:15:03.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173625 2.17 2:ed81edb2:::benchmark_data_smithi194_79000_object9364:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.93991794799999, 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24512054, 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.116 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173625 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148416+0000", 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.894e-06 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148421+0000", 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5319999999999997e-06 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148435+0000", 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3431000000000001e-05 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148443+0000", 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6719999999999993e-06 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.177311+0000", 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028867127999999999 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.177432+0000", 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121236 2024-04-29T14:15:03.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.178047+0000", 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061545999999999997 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187188+0000", 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091407750000000003 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393432+0000", 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20624394500000001 2024-04-29T14:15:03.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393478+0000", 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6011999999999999e-05 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393534+0000", 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6455000000000001e-05 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173632 2.17 2:ebef1b31:::benchmark_data_smithi194_79000_object9371:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.928939181, 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23825384199999999, 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173632 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159395+0000", 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0549999999999998e-06 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159401+0000", 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0569999999999998e-06 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159413+0000", 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2347999999999999e-05 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159422+0000", 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9299999999999992e-06 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180545+0000", 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021122809999999999 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180656+0000", 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011134 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181142+0000", 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000486031 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187196+0000", 2024-04-29T14:15:03.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060541850000000001 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397527+0000", 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21033081000000001 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397582+0000", 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5235999999999998e-05 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397646+0000", 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4040000000000003e-05 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173636 2.7 2:e5bc6543:::benchmark_data_smithi194_79000_object9375:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.928279737, 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23324684400000001, 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173636 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160053+0000", 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.294e-06 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160058+0000", 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4100000000000001e-06 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160068+0000", 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7569999999999999e-06 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160073+0000", 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8810000000000001e-06 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180528+0000", 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020454898999999999 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180612+0000", 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3422000000000002e-05 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181050+0000", 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000438391 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187192+0000", 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061421150000000001 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393229+0000", 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20603648399999999 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393264+0000", 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5549999999999997e-05 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393299+0000", 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4641000000000003e-05 2024-04-29T14:15:03.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173640 2.7 2:e581d5ea:::benchmark_data_smithi194_79000_object9379:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.92428773500001, 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22945062699999999, 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.133 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173640 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164046+0000", 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1409999999999999e-06 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164056+0000", 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008e-05 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164100+0000", 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3893000000000001e-05 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164109+0000", 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6889999999999993e-06 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184149+0000", 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020040235999999999 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184270+0000", 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120791 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184634+0000", 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036420799999999998 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187199+0000", 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025650199999999999 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393415+0000", 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20621562099999999 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393462+0000", 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7055999999999998e-05 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393495+0000", 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2963999999999998e-05 2024-04-29T14:15:03.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173648 2.17 2:eb9a7570:::benchmark_data_smithi194_79000_object9387:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.90701010199999, 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26978428700000001, 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.139 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173648 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181324+0000", 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.874e-06 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181330+0000", 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2210000000000002e-06 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181342+0000", 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2e-05 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181350+0000", 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.106e-06 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184353+0000", 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030029319999999998 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184447+0000", 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3710999999999998e-05 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184955+0000", 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050829300000000005 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187337+0000", 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00238265 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451020+0000", 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.263682898 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451050+0000", 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9578999999999999e-05 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451106+0000", 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6023000000000002e-05 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173673 2.7 2:e4271adc:::benchmark_data_smithi194_79000_object9412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.144 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.246965+0000", 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.841366849, 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15128623199999999, 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173673 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.246965+0000", 2024-04-29T14:15:03.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.246965+0000", 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.246967+0000", 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3870000000000002e-06 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.246974+0000", 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9029999999999998e-06 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.246989+0000", 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4609e-05 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.246997+0000", 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5390000000000007e-06 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.247059+0000", 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1529e-05 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.247202+0000", 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143416 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.247577+0000", 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037463299999999998 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.249340+0000", 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017629150000000001 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.398167+0000", 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14882757599999999 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.398219+0000", 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1138999999999999e-05 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.398251+0000", 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2586000000000003e-05 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173705 2.7 2:e46e9efa:::benchmark_data_smithi194_79000_object9444:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.308190+0000", 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 234.78014159, 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14285827000000001, 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.150 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173705 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308190+0000", 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308190+0000", 2024-04-29T14:15:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308192+0000", 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9609999999999999e-06 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308197+0000", 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8069999999999999e-06 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308205+0000", 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2090000000000005e-06 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308211+0000", 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.412e-06 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308257+0000", 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5946999999999999e-05 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308350+0000", 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3243000000000002e-05 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.308695+0000", 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034563600000000002 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.310403+0000", 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001707421 2024-04-29T14:15:03.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.450970+0000", 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14056764499999999 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451003+0000", 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2098000000000003e-05 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451048+0000", 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5890999999999998e-05 2024-04-29T14:15:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:174991 2.17 2:ed4199c0:::benchmark_data_smithi194_79000_object10730:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:10.385795+0000", 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.702536737, 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.120229431, 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 174991 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.385795+0000", 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.385795+0000", 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.385797+0000", 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.593e-06 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.385802+0000", 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0679999999999996e-06 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.385814+0000", 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.205e-05 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.385822+0000", 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9990000000000008e-06 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.407444+0000", 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021622355999999999 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.407587+0000", 2024-04-29T14:15:03.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014302299999999999 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.408223+0000", 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00063590200000000004 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.418209+0000", 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099858559999999996 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.505947+0000", 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.087737599999999999 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.505981+0000", 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4777999999999998e-05 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.506024+0000", 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3205999999999999e-05 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.161 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:174997 2.17 2:eedc6f8f:::benchmark_data_smithi194_79000_object10736:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:10.395031+0000", 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.69330048, 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.11115375299999999, 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.162 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 174997 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.395031+0000", 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.395031+0000", 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.395032+0000", 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4200000000000001e-07 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.395034+0000", 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0870000000000001e-06 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.395037+0000", 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-06 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.395043+0000", 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6749999999999999e-06 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.411049+0000", 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016006647999999998 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.411168+0000", 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118422 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.411620+0000", 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045197999999999999 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.506035+0000", 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.094415298999999994 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.506078+0000", 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2629999999999997e-05 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.506137+0000", 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8730999999999997e-05 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.506185+0000", 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8538999999999997e-05 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.167 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:175370 2.17 2:ebf266d6:::benchmark_data_smithi194_79000_object11109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:10.971223+0000", 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.117108312, 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12537055799999999, 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 175370 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.971223+0000", 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.971223+0000", 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.971225+0000", 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7209999999999999e-06 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.971230+0000", 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4519999999999999e-06 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.971239+0000", 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2669999999999995e-06 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.971245+0000", 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.489e-06 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.993755+0000", 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022509273 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.993874+0000", 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119536 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:10.994444+0000", 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056960599999999995 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.096418+0000", 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101974124 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.096474+0000", 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6514999999999997e-05 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.096527+0000", 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2837999999999998e-05 2024-04-29T14:15:03.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.096594+0000", 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6736999999999995e-05 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:175398 2.17 2:efd25725:::benchmark_data_smithi194_79000_object11137:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:11.022894+0000", 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.06543731299999, 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13144297899999999, 2024-04-29T14:15:03.173 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 175398 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.022894+0000", 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.022894+0000", 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.022897+0000", 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.022903+0000", 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9680000000000003e-06 2024-04-29T14:15:03.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.022911+0000", 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4800000000000001e-06 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.022920+0000", 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0360000000000006e-06 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.145779+0000", 2024-04-29T14:15:03.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122859654 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.145914+0000", 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013510700000000001 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.146496+0000", 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058141499999999995 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154212+0000", 2024-04-29T14:15:03.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077163450000000003 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154225+0000", 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2777e-05 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154270+0000", 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5513000000000003e-05 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154337+0000", 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7139999999999998e-05 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:175405 2.17 2:efd14ad5:::benchmark_data_smithi194_79000_object11144:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:11.031040+0000", 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.05729155399999, 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13103574300000001, 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.179 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 175405 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.031040+0000", 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.031040+0000", 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.031042+0000", 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.711e-06 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.031047+0000", 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1479999999999997e-06 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.031057+0000", 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7650000000000005e-06 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.031063+0000", 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5799999999999997e-06 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.150294+0000", 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.119230954 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.150423+0000", 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012847800000000001 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.151050+0000", 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062730000000000001 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.156690+0000", 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056394469999999997 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.161958+0000", 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052684619999999998 2024-04-29T14:15:03.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.162009+0000", 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1393000000000001e-05 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.162076+0000", 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6505000000000001e-05 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.184 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:175408 2.7 2:e4415554:::benchmark_data_smithi194_79000_object11147:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:11.037192+0000", 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.05113968000001, 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122033899, 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 175408 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.037192+0000", 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.037192+0000", 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.037194+0000", 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0540000000000002e-06 2024-04-29T14:15:03.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.037202+0000", 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7800000000000001e-06 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.037215+0000", 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2848e-05 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.037223+0000", 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4030000000000007e-06 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.150047+0000", 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.112823878 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.150158+0000", 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111271 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.150590+0000", 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043205400000000002 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154195+0000", 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036049329999999998 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.159129+0000", 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049341610000000003 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.159177+0000", 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7929000000000001e-05 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.159226+0000", 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8588000000000003e-05 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:175412 2.7 2:e0637da8:::benchmark_data_smithi194_79000_object11151:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:11.039227+0000", 2024-04-29T14:15:03.190 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.04910483500001, 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12519698300000001, 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 175412 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.039227+0000", 2024-04-29T14:15:03.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.039227+0000", 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.039228+0000", 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4720000000000001e-06 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.039234+0000", 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3920000000000002e-06 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.039243+0000", 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2310000000000002e-06 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.039250+0000", 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.55e-06 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154230+0000", 2024-04-29T14:15:03.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.114980586 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154346+0000", 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115413 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.154713+0000", 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036766899999999999 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.161927+0000", 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072132469999999999 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.164362+0000", 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024351709999999999 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.164395+0000", 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3621000000000002e-05 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.164424+0000", 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8631000000000001e-05 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:175424 2.7 2:e738610a:::benchmark_data_smithi194_79000_object11163:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:11.054897+0000", 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.033434871, 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.111626011, 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:03.196 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 175424 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.054897+0000", 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.054897+0000", 2024-04-29T14:15:03.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.054899+0000", 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.142e-06 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.054907+0000", 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.14e-06 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:03.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.054919+0000", 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1457999999999999e-05 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.054927+0000", 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4260000000000003e-06 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.156684+0000", 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101757008 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:03.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.156823+0000", 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013886599999999999 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.157195+0000", 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037195099999999998 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.161993+0000", 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047985780000000004 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.166439+0000", 2024-04-29T14:15:03.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044454220000000001 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.166483+0000", 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4437999999999997e-05 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:11.166523+0000", 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9582000000000002e-05 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:03.201 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:03.202 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:15:03.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:03.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 126566 126550 129.651 80.5625 0.00580315 0.00771153 2024-04-29T14:15:03.249 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:03.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: 3.7 scrub ok 2024-04-29T14:15:03.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.313 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: 3.c scrub starts 2024-04-29T14:15:03.313 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: 3.0 scrub ok 2024-04-29T14:15:03.313 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:03 smithi040 ceph-mon[39581]: pgmap v651: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:15:03.334 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:03.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: 3.7 scrub ok 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: 3.c scrub starts 2024-04-29T14:15:03.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: 3.0 scrub ok 2024-04-29T14:15:03.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:03 smithi121 ceph-mon[44533]: pgmap v651: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:15:03.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:03.494 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:03.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:03.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: 3.7 scrub ok 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: 3.c scrub starts 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: 3.0 scrub ok 2024-04-29T14:15:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:03 smithi194 ceph-mon[44089]: pgmap v651: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:15:03.631 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:03.660 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:03.684 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:03.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:03.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:03.819 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:03.844 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:03.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:03.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:15:03.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:04.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:04.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:04.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 15 127759 127744 128.763 74.625 0.0247897 0.00776499 2024-04-29T14:15:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:04.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:04.337 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:15:04.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:04.443 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:04.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:04 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:15:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:04 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:15:04.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:15:04.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:04.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:04 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:15:04.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:04 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:15:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:04.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:04.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:04 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:15:04.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:04 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:15:04.716 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:04.744 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:04.773 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:04.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:04.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:04.921 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:04.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:05.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:05.136 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:05.172 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:05.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:05.209 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 128950 128934 127.9 74.375 0.0101219 0.00781702 2024-04-29T14:15:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:05 smithi040 ceph-mon[39581]: 3.c scrub ok 2024-04-29T14:15:05.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:05 smithi040 ceph-mon[39581]: pgmap v652: 17 pgs: 17 active+clean; 7.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:15:05.277 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:15:05.277 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:15:05.277 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 241, 2024-04-29T14:15:05.277 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:15:05.253438+0000", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:14:29.066577+0000", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:14:19.024655+0000", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5799936, 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:15:05.278 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 26, 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 3, 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:15:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:15:05.280 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:15:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:15:05.282 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:15:05.283 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:15:05.284 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 11.109999656677246, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 11.109999656677246, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.18000000715255737, 2024-04-29T14:15:05.285 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 3, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_1", 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:13:42.111930+0000", 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:15:05.286 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:15:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "217", 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:15:05.288 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:15:05.289 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:15:05.290 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.059999942779541, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 2.059999942779541, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.81999999284744263, 2024-04-29T14:15:05.291 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 1.690000057220459, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 1.690000057220459, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:15:05.292 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.293 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.294 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.295 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:05.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.297 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.297 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:15:05.297 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:15:05.297 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.297 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:15:05.298 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.299 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.300 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.301 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.302 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.303 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 0, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 20, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 152, 2024-04-29T14:15:05.304 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.305 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3368423938 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3368423938", 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3368423938", 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3368423938", 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3368423938", 2024-04-29T14:15:05.307 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "exists" 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:15:05.308 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 219, 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:15:05.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:15:05.311 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.312 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 223, 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.313 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.315 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:15:05.316 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 36, 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 213, 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.318 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/3807513901", 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/3807513901", 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3807513901", 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/3807513901", 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.320 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 224, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:15:05.321 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:15:05.323 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.7946624755859375, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.325 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:15:05.328 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.66400146484375, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 64, 2024-04-29T14:15:05.329 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2130074108 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/2130074108", 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/2130074108", 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/2130074108", 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/2130074108", 2024-04-29T14:15:05.332 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.782379150390625, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:15:05.333 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 213, 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.334 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:15:05.335 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:15:05.336 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.337 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 199, 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:15:05.339 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:15:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:15:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:15:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:14:21.901931+0000", 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:15:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:13:57.057762+0000", 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:13:32.863656+0000", 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:14:55.097706+0000", 2024-04-29T14:15:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:15:05.095908+0000", 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:15:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:14:19.933476+0000", 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:15:04.414607+0000", 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:14:30.528622+0000", 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:13:45.510939+0000", 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:15:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:14:54.057296+0000", 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:14:24.069059+0000", 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:14:47.741702+0000", 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:15:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:15:05.357 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:15:05.390 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:05.416 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:05.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:05.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:05.563 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:05 smithi194 ceph-mon[44089]: 3.c scrub ok 2024-04-29T14:15:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:05 smithi194 ceph-mon[44089]: pgmap v652: 17 pgs: 17 active+clean; 7.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:15:05.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:05 smithi121 ceph-mon[44533]: 3.c scrub ok 2024-04-29T14:15:05.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:05 smithi121 ceph-mon[44533]: pgmap v652: 17 pgs: 17 active+clean; 7.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:15:06.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 16 130220 130204 127.142 79.375 0.0122021 0.00786353 2024-04-29T14:15:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:06 smithi121 ceph-mon[44533]: 3.d scrub starts 2024-04-29T14:15:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:06.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:06 smithi121 ceph-mon[44533]: osdmap e241: 12 total, 11 up, 10 in 2024-04-29T14:15:06.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1779668489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:15:06.374 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:15:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:15:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:130376 3.d 3:b3706239:::benchmark_data_smithi194_88551_object130375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e241)", 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:06.368825+0000", 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.012095059, 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012120878, 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 130376 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.368825+0000", 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.368825+0000", 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.368828+0000", 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1499999999999999e-06 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.368864+0000", 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6452999999999998e-05 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.368877+0000", 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2829e-05 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.368886+0000", 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1989999999999995e-06 2024-04-29T14:15:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.371894+0000", 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030081280000000001 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.372017+0000", 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123106 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.372141+0000", 2024-04-29T14:15:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012319100000000001 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:130385 3.d 3:b9b69ffc:::benchmark_data_smithi194_88551_object130384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e241)", 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:06.373627+0000", 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.007292771, 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073708740000000003, 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:15:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 130385 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.373627+0000", 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.373627+0000", 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.373630+0000", 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.796e-06 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.373668+0000", 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8729999999999997e-05 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.373681+0000", 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2652e-05 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.373691+0000", 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4490000000000005e-06 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.375615+0000", 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019248239999999999 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.375751+0000", 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013587699999999999 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.375927+0000", 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000175823 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:130394 3.f 3:f6cd3b3f:::benchmark_data_smithi194_88551_object130393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e241)", 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:06.376401+0000", 2024-04-29T14:15:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0045190780000000002, 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046411059999999999, 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 130394 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376401+0000", 2024-04-29T14:15:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376401+0000", 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376403+0000", 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3410000000000001e-06 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376430+0000", 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7279000000000001e-05 2024-04-29T14:15:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376444+0000", 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3967e-05 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376452+0000", 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9189999999999999e-06 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376511+0000", 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9244999999999999e-05 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376623+0000", 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112125 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.376749+0000", 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125943 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:15:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:06.380758+0000", 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040086380000000001 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 3 2024-04-29T14:15:06.388 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:15:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:06 smithi194 ceph-mon[44089]: 3.d scrub starts 2024-04-29T14:15:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:06.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:06.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:06 smithi194 ceph-mon[44089]: osdmap e241: 12 total, 11 up, 10 in 2024-04-29T14:15:06.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1779668489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:15:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:06 smithi040 ceph-mon[39581]: 3.d scrub starts 2024-04-29T14:15:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:06 smithi040 ceph-mon[39581]: osdmap e241: 12 total, 11 up, 10 in 2024-04-29T14:15:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1779668489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:15:07.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 131400 131384 126.32 73.75 0.00524381 0.00791523 2024-04-29T14:15:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:07 smithi194 ceph-mon[44089]: 3.1 scrub starts 2024-04-29T14:15:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:07 smithi194 ceph-mon[44089]: pgmap v654: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:15:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:07 smithi194 ceph-mon[44089]: osdmap e242: 12 total, 11 up, 10 in 2024-04-29T14:15:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:07 smithi040 ceph-mon[39581]: 3.1 scrub starts 2024-04-29T14:15:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:07 smithi040 ceph-mon[39581]: pgmap v654: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:15:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:07 smithi040 ceph-mon[39581]: osdmap e242: 12 total, 11 up, 10 in 2024-04-29T14:15:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:07 smithi121 ceph-mon[44533]: 3.1 scrub starts 2024-04-29T14:15:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:07 smithi121 ceph-mon[44533]: pgmap v654: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:15:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:07 smithi121 ceph-mon[44533]: osdmap e242: 12 total, 11 up, 10 in 2024-04-29T14:15:08.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 16 132601 132585 125.543 75.0625 0.00679333 0.00796011 2024-04-29T14:15:08.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:08 smithi121 ceph-mon[44533]: 3.d scrub ok 2024-04-29T14:15:08.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:08 smithi194 ceph-mon[44089]: 3.d scrub ok 2024-04-29T14:15:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:08 smithi040 ceph-mon[39581]: 3.d scrub ok 2024-04-29T14:15:09.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 16 133666 133650 124.663 66.5625 0.0054687 0.00802014 2024-04-29T14:15:09.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:09 smithi040 ceph-mon[39581]: 3.8 scrub starts 2024-04-29T14:15:09.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:09 smithi040 ceph-mon[39581]: 3.1 scrub ok 2024-04-29T14:15:09.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:09 smithi040 ceph-mon[39581]: pgmap v656: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-29T14:15:09.556 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:09 smithi194 ceph-mon[44089]: 3.8 scrub starts 2024-04-29T14:15:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:09 smithi194 ceph-mon[44089]: 3.1 scrub ok 2024-04-29T14:15:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:09 smithi194 ceph-mon[44089]: pgmap v656: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-29T14:15:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:09.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:09 smithi121 ceph-mon[44533]: 3.8 scrub starts 2024-04-29T14:15:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:09 smithi121 ceph-mon[44533]: 3.1 scrub ok 2024-04-29T14:15:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:09 smithi121 ceph-mon[44533]: pgmap v656: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-29T14:15:09.665 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:09.692 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:09.711 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:09.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:09.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:09.864 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:10.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 134549 134533 123.641 55.1875 0.0161913 0.00808495 2024-04-29T14:15:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:10 smithi194 ceph-mon[44089]: 3.f deep-scrub starts 2024-04-29T14:15:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:10 smithi194 ceph-mon[44089]: 3.b scrub starts 2024-04-29T14:15:10.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:10 smithi040 ceph-mon[39581]: 3.f deep-scrub starts 2024-04-29T14:15:10.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:10 smithi040 ceph-mon[39581]: 3.b scrub starts 2024-04-29T14:15:10.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:10 smithi121 ceph-mon[44533]: 3.f deep-scrub starts 2024-04-29T14:15:10.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:10 smithi121 ceph-mon[44533]: 3.b scrub starts 2024-04-29T14:15:11.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 135605 135589 122.806 66 0.00438467 0.00814078 2024-04-29T14:15:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:11 smithi194 ceph-mon[44089]: 3.8 scrub ok 2024-04-29T14:15:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:11 smithi194 ceph-mon[44089]: 3.b scrub ok 2024-04-29T14:15:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:11 smithi194 ceph-mon[44089]: pgmap v657: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:15:11.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:11 smithi040 ceph-mon[39581]: 3.8 scrub ok 2024-04-29T14:15:11.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:11 smithi040 ceph-mon[39581]: 3.b scrub ok 2024-04-29T14:15:11.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:11 smithi040 ceph-mon[39581]: pgmap v657: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:15:11.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:11 smithi121 ceph-mon[44533]: 3.8 scrub ok 2024-04-29T14:15:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:11 smithi121 ceph-mon[44533]: 3.b scrub ok 2024-04-29T14:15:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:11 smithi121 ceph-mon[44533]: pgmap v657: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:15:11.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:11.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:11.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:15:11.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:12.026 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:12.152 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:12.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:12.176 INFO:tasks.thrashosds.thrasher:j is {'epoch': 241, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:15:05.253438+0000', 'last_up_change': '2024-04-29T14:14:29.066577+0000', 'last_in_change': '2024-04-29T14:14:19.024655+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.18000000715255737, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-04-29T14:13:42.111930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 0.8199999928474426, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 152, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3368423938}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3368423938}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3368423938}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3368423938}]}, 'public_addr': '172.21.15.40:6809/3368423938', 'cluster_addr': '172.21.15.40:6810/3368423938', 'heartbeat_back_addr': '172.21.15.40:6812/3368423938', 'heartbeat_front_addr': '172.21.15.40:6811/3368423938', 'state': ['exists']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 219, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 213, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 3807513901}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3807513901}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3807513901}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 3807513901}]}, 'public_addr': '172.21.15.121:6804/3807513901', 'cluster_addr': '172.21.15.121:6805/3807513901', 'heartbeat_back_addr': '172.21.15.121:6807/3807513901', 'heartbeat_front_addr': '172.21.15.121:6806/3807513901', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 224, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 0.7946624755859375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 1, 'weight': 0.66400146484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 64, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 2130074108}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 2130074108}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 2130074108}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 2130074108}]}, 'public_addr': '172.21.15.194:6800/2130074108', 'cluster_addr': '172.21.15.194:6801/2130074108', 'heartbeat_back_addr': '172.21.15.194:6803/2130074108', 'heartbeat_front_addr': '172.21.15.194:6802/2130074108', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 0.782379150390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 213, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 199, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:14:21.901931+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:13:57.057762+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:13:32.863656+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:14:55.097706+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:15:05.095908+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:14:19.933476+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:15:04.414607+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:14:30.528622+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:13:45.510939+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:14:54.057296+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:14:24.069059+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:14:47.741702+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:15:12.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:15:12.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 136693 136677 122.023 68 0.00481327 0.00819101 2024-04-29T14:15:12.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:12.340 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:12.372 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:12.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:12.430 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:15:12.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:12.525 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:12.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:12.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:12.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:12.780 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:13.028 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:13 smithi121 ceph-mon[44533]: 3.4 scrub starts 2024-04-29T14:15:13.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 16 137624 137608 121.124 58.1875 0.00516666 0.00825404 2024-04-29T14:15:13.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:13 smithi194 ceph-mon[44089]: 3.4 scrub starts 2024-04-29T14:15:13.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:13 smithi040 ceph-mon[39581]: 3.4 scrub starts 2024-04-29T14:15:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:15:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:14.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 138626 138610 120.311 62.625 0.00827328 0.00830992 2024-04-29T14:15:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:14 smithi194 ceph-mon[44089]: pgmap v658: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:15:14.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/481505880' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:14.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:14 smithi194 ceph-mon[44089]: from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:14.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:14 smithi040 ceph-mon[39581]: pgmap v658: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:15:14.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/481505880' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:14.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:14 smithi040 ceph-mon[39581]: from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:14.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:14 smithi121 ceph-mon[44533]: pgmap v658: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:15:14.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/481505880' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:14.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:14 smithi121 ceph-mon[44533]: from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:14.850 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:14.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:14.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:14.984 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:15.007 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:15.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:15.112 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:15 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:15 smithi040 ceph-mon[39581]: from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:15:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:15 smithi040 ceph-mon[39581]: osdmap e243: 12 total, 11 up, 10 in 2024-04-29T14:15:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:15 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:15 smithi121 ceph-mon[44533]: from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:15:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:15 smithi121 ceph-mon[44533]: osdmap e243: 12 total, 11 up, 10 in 2024-04-29T14:15:15.141 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:15.162 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:15.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:15.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 139711 139695 119.592 67.8125 0.0148849 0.00835969 2024-04-29T14:15:15.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:15.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:15 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:15.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:15 smithi194 ceph-mon[44089]: from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:15:15.299 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:15 smithi194 ceph-mon[44089]: osdmap e243: 12 total, 11 up, 10 in 2024-04-29T14:15:15.300 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:15.325 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:15.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:15.429 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:15.458 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:15.478 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:15.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:15.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:15.632 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:15.699 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:15:15.722 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:15:15.749 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:15:15.781 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:15:15.805 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:15:15.834 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:15:15.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:15:15.883 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:15:15.905 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:15:15.929 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:15:15.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:15:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:15:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:15:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:15:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:15:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:15:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:15:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:15:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:15:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:15:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:15:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:15:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:15:16.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 140780 140764 118.879 66.8125 0.00471465 0.00840925 2024-04-29T14:15:16.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:16 smithi121 ceph-mon[44533]: pgmap v660: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 54 MiB/s wr, 858 op/s 2024-04-29T14:15:16.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4213862926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:16.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:16.289 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:16.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:16 smithi040 ceph-mon[39581]: pgmap v660: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 54 MiB/s wr, 858 op/s 2024-04-29T14:15:16.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4213862926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:16.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:16.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:16 smithi194 ceph-mon[44089]: pgmap v660: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 54 MiB/s wr, 858 op/s 2024-04-29T14:15:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4213862926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:16.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:16.455 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:16.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:16.582 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:16.582 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:16.645 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:16.674 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:16.699 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:16.702 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:16.803 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:16.830 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:16.857 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:16.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:16.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:15:16.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.053 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:17 smithi194 ceph-mon[44089]: osdmap e244: 12 total, 11 up, 10 in 2024-04-29T14:15:17.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 16 141849 141833 118.184 66.8125 0.0957731 0.00845898 2024-04-29T14:15:17.218 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:17.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:17 smithi121 ceph-mon[44533]: osdmap e244: 12 total, 11 up, 10 in 2024-04-29T14:15:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:15:17.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:15:17.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:15:17.290 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:17.322 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:17.357 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:17 smithi040 ceph-mon[39581]: osdmap e244: 12 total, 11 up, 10 in 2024-04-29T14:15:17.380 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:17.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:17.486 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:17.515 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:17.537 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:17.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:17.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:17.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:17.707 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:15:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:17.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:17.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:17.894 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:17.918 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:17.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:18.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:18.065 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:18.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:18.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:18.200 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:18.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 142863 142847 117.463 63.375 0.0152628 0.00850732 2024-04-29T14:15:18.230 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:18.256 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:18.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:18 smithi194 ceph-mon[44089]: pgmap v662: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-29T14:15:18.338 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:18 smithi040 ceph-mon[39581]: pgmap v662: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-29T14:15:18.350 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:18 smithi121 ceph-mon[44533]: pgmap v662: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-29T14:15:18.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:18.541 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:15:18.541 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:15:18.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:18.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:18.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:18.793 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:19.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:19.210 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 143922 143906 116.797 66.1875 0.00469609 0.00856054 2024-04-29T14:15:19.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:15:19.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:19.299 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:19.310 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2024-04-29T14:15:16.945563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9153544736,"num_objects":139665,"num_object_clones":0,"num_object_copies":279330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139665,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":139844,"num_write_kb":8941370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2278,"num_bytes_recovered":150882432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140699,"ondisk_log_size":140699,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":937451520,"kb_used":19753376,"kb_used_data":17831480,"kb_used_omap":15,"kb_used_meta":1920688,"kb_avail":917698144,"statfs":{"total":959950356480,"available":939722899456,"internally_reserved":0,"allocated":18259435520,"data_stored":18254354758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15952,"internal_metadata":1966784944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,3,12,4,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":633929728,"num_objects":9673,"num_object_clones":0,"num_object_copies":19346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9673,"num_write_kb":619072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002109"},"pg_stats":[{"pgid":"3.f","version":"243'9604","reported_seq":9740,"reported_epoch":243,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T14:15:15.609261+0000","last_change":"2024-04-29T14:15:08.658431+0000","last_active":"2024-04-29T14:15:15.609261+0000","last_peered":"2024-04-29T14:15:15.609261+0000","last_clean":"2024-04-29T14:15:15.609261+0000","last_became_active":"2024-04-29T14:13:43.169982+0000","last_became_peered":"2024-04-29T14:13:43.169982+0000","last_unstale":"2024-04-29T14:15:15.609261+0000","last_undegraded":"2024-04-29T14:15:15.609261+0000","last_fullsized":"2024-04-29T14:15:15.609261+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:13:42.154997+0000","objects_scrubbed":1261,"log_size":9604,"log_dups_size":0,"ondisk_log_size":9604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570490880,"num_objects":8705,"num_object_clones":0,"num_object_copies":17410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":557120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"243'8650","reported_seq":8802,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:16.012016+0000","last_change":"2024-04-29T14:14:58.773648+0000","last_active":"2024-04-29T14:15:16.012016+0000","last_peered":"2024-04-29T14:15:16.012016+0000","last_clean":"2024-04-29T14:15:16.012016+0000","last_became_active":"2024-04-29T14:13:52.443531+0000","last_became_peered":"2024-04-29T14:13:52.443531+0000","last_unstale":"2024-04-29T14:15:16.012016+0000","last_undegraded":"2024-04-29T14:15:16.012016+0000","last_fullsized":"2024-04-29T14:15:16.012016+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7424","last_scrub_stamp":"2024-04-29T14:14:58.773519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:14:58.773519+0000","objects_scrubbed":7341,"log_size":8650,"log_dups_size":0,"ondisk_log_size":8650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:21.392374+0000","scrub_duration":1736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566886400,"num_objects":8650,"num_object_clones":0,"num_object_copies":17300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":553600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"243'8874","reported_seq":9046,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:15.622716+0000","last_change":"2024-04-29T14:15:06.823786+0000","last_active":"2024-04-29T14:15:15.622716+0000","last_peered":"2024-04-29T14:15:15.622716+0000","last_clean":"2024-04-29T14:15:15.622716+0000","last_became_active":"2024-04-29T14:13:43.174780+0000","last_became_peered":"2024-04-29T14:13:43.174780+0000","last_unstale":"2024-04-29T14:15:15.622716+0000","last_undegraded":"2024-04-29T14:15:15.622716+0000","last_fullsized":"2024-04-29T14:15:15.622716+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'8297","last_scrub_stamp":"2024-04-29T14:15:06.823654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:15:06.823654+0000","objects_scrubbed":8206,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:18.778322+0000","scrub_duration":2156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581566464,"num_objects":8874,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8874,"num_write_kb":567936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"243'8741","reported_seq":8914,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:14.950211+0000","last_change":"2024-04-29T14:15:04.510976+0000","last_active":"2024-04-29T14:15:14.950211+0000","last_peered":"2024-04-29T14:15:14.950211+0000","last_clean":"2024-04-29T14:15:14.950211+0000","last_became_active":"2024-04-29T14:13:43.177266+0000","last_became_peered":"2024-04-29T14:13:43.177266+0000","last_unstale":"2024-04-29T14:15:14.950211+0000","last_undegraded":"2024-04-29T14:15:14.950211+0000","last_fullsized":"2024-04-29T14:15:14.950211+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8047","last_scrub_stamp":"2024-04-29T14:15:04.510841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:15:04.510841+0000","objects_scrubbed":7967,"log_size":8741,"log_dups_size":0,"ondisk_log_size":8741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:21.139332+0000","scrub_duration":2144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572850176,"num_objects":8741,"num_object_clones":0,"num_object_copies":17482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":559424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"242'8579","reported_seq":8750,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-29T14:15:13.620604+0000","last_change":"2024-04-29T14:15:10.400379+0000","last_active":"2024-04-29T14:15:13.620604+0000","last_peered":"2024-04-29T14:15:13.620604+0000","last_clean":"2024-04-29T14:15:13.620604+0000","last_became_active":"2024-04-29T14:13:43.175185+0000","last_became_peered":"2024-04-29T14:13:43.175185+0000","last_unstale":"2024-04-29T14:15:13.620604+0000","last_undegraded":"2024-04-29T14:15:13.620604+0000","last_fullsized":"2024-04-29T14:15:13.620604+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8393","last_scrub_stamp":"2024-04-29T14:15:10.400208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:15:10.400208+0000","objects_scrubbed":8368,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:24.598933+0000","scrub_duration":1233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562233344,"num_objects":8579,"num_object_clones":0,"num_object_copies":17158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8579,"num_write_kb":549056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"243'8788","reported_seq":11277,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:15.301499+0000","last_change":"2024-04-29T14:14:53.793853+0000","last_active":"2024-04-29T14:15:15.301499+0000","last_peered":"2024-04-29T14:15:15.301499+0000","last_clean":"2024-04-29T14:15:15.301499+0000","last_became_active":"2024-04-29T14:14:17.997407+0000","last_became_peered":"2024-04-29T14:14:17.997407+0000","last_unstale":"2024-04-29T14:15:15.301499+0000","last_undegraded":"2024-04-29T14:15:15.301499+0000","last_fullsized":"2024-04-29T14:15:15.301499+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7158","last_scrub_stamp":"2024-04-29T14:14:53.793605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:14:53.793605+0000","objects_scrubbed":7071,"log_size":8788,"log_dups_size":0,"ondisk_log_size":8788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:03.381668+0000","scrub_duration":2116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575930368,"num_objects":8788,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":562432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1159,"num_bytes_recovered":75956224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"243'8684","reported_seq":8767,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:14.273235+0000","last_change":"2024-04-29T14:13:43.170620+0000","last_active":"2024-04-29T14:15:14.273235+0000","last_peered":"2024-04-29T14:15:14.273235+0000","last_clean":"2024-04-29T14:15:14.273235+0000","last_became_active":"2024-04-29T14:13:43.170268+0000","last_became_peered":"2024-04-29T14:13:43.170268+0000","last_unstale":"2024-04-29T14:15:14.273235+0000","last_undegraded":"2024-04-29T14:15:14.273235+0000","last_fullsized":"2024-04-29T14:15:14.273235+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:13:42.154997+0000","objects_scrubbed":0,"log_size":8684,"log_dups_size":0,"ondisk_log_size":8684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569114624,"num_objects":8684,"num_object_clones":0,"num_object_copies":17368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":555776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"243'8948","reported_seq":9110,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:15.963282+0000","last_change":"2024-04-29T14:15:09.747612+0000","last_active":"2024-04-29T14:15:15.963282+0000","last_peered":"2024-04-29T14:15:15.963282+0000","last_clean":"2024-04-29T14:15:15.963282+0000","last_became_active":"2024-04-29T14:13:52.443412+0000","last_became_peered":"2024-04-29T14:13:52.443412+0000","last_unstale":"2024-04-29T14:15:15.963282+0000","last_undegraded":"2024-04-29T14:15:15.963282+0000","last_fullsized":"2024-04-29T14:15:15.963282+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8538","last_scrub_stamp":"2024-04-29T14:15:09.747403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:15:09.747403+0000","objects_scrubbed":8475,"log_size":8948,"log_dups_size":0,"ondisk_log_size":8948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:35.428417+0000","scrub_duration":1698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586416128,"num_objects":8948,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8948,"num_write_kb":572672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"242'8627","reported_seq":8792,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-29T14:15:13.624951+0000","last_change":"2024-04-29T14:15:01.730232+0000","last_active":"2024-04-29T14:15:13.624951+0000","last_peered":"2024-04-29T14:15:13.624951+0000","last_clean":"2024-04-29T14:15:13.624951+0000","last_became_active":"2024-04-29T14:13:43.174109+0000","last_became_peered":"2024-04-29T14:13:43.174109+0000","last_unstale":"2024-04-29T14:15:13.624951+0000","last_undegraded":"2024-04-29T14:15:13.624951+0000","last_fullsized":"2024-04-29T14:15:13.624951+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7797","last_scrub_stamp":"2024-04-29T14:15:01.729894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:15:01.729894+0000","objects_scrubbed":7737,"log_size":8627,"log_dups_size":0,"ondisk_log_size":8627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:03.293616+0000","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":565379072,"num_objects":8627,"num_object_clones":0,"num_object_copies":17254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8627,"num_write_kb":552128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"244'8769","reported_seq":8853,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-29T14:15:16.156691+0000","last_change":"2024-04-29T14:13:43.177973+0000","last_active":"2024-04-29T14:15:16.156691+0000","last_peered":"2024-04-29T14:15:16.156691+0000","last_clean":"2024-04-29T14:15:16.156691+0000","last_became_active":"2024-04-29T14:13:43.177667+0000","last_became_peered":"2024-04-29T14:13:43.177667+0000","last_unstale":"2024-04-29T14:15:16.156691+0000","last_undegraded":"2024-04-29T14:15:16.156691+0000","last_fullsized":"2024-04-29T14:15:16.156691+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:13:42.154997+0000","objects_scrubbed":0,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574685184,"num_objects":8769,"num_object_clones":0,"num_object_copies":17538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8769,"num_write_kb":561216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"243'8782","reported_seq":8948,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:14.952401+0000","last_change":"2024-04-29T14:14:56.086984+0000","last_active":"2024-04-29T14:15:14.952401+0000","last_peered":"2024-04-29T14:15:14.952401+0000","last_clean":"2024-04-29T14:15:14.952401+0000","last_became_active":"2024-04-29T14:13:43.177238+0000","last_became_peered":"2024-04-29T14:13:43.177238+0000","last_unstale":"2024-04-29T14:15:14.952401+0000","last_undegraded":"2024-04-29T14:15:14.952401+0000","last_fullsized":"2024-04-29T14:15:14.952401+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'7371","last_scrub_stamp":"2024-04-29T14:14:56.086795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:14:56.086795+0000","objects_scrubbed":7304,"log_size":8782,"log_dups_size":0,"ondisk_log_size":8782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:07.240454+0000","scrub_duration":1767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575537152,"num_objects":8782,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":562048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"243'8682","reported_seq":8827,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:14.952473+0000","last_change":"2024-04-29T14:15:14.166251+0000","last_active":"2024-04-29T14:15:14.952473+0000","last_peered":"2024-04-29T14:15:14.952473+0000","last_clean":"2024-04-29T14:15:14.952473+0000","last_became_active":"2024-04-29T14:13:52.443405+0000","last_became_peered":"2024-04-29T14:13:52.443405+0000","last_unstale":"2024-04-29T14:15:14.952473+0000","last_undegraded":"2024-04-29T14:15:14.952473+0000","last_fullsized":"2024-04-29T14:15:14.952473+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:13:42.154997+0000","objects_scrubbed":7149,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:15:19.133959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":568983552,"num_objects":8682,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":555648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"243'8760","reported_seq":8925,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:15.997433+0000","last_change":"2024-04-29T14:14:56.640955+0000","last_active":"2024-04-29T14:15:15.997433+0000","last_peered":"2024-04-29T14:15:15.997433+0000","last_clean":"2024-04-29T14:15:15.997433+0000","last_became_active":"2024-04-29T14:13:43.175151+0000","last_became_peered":"2024-04-29T14:13:43.175151+0000","last_unstale":"2024-04-29T14:15:15.997433+0000","last_undegraded":"2024-04-29T14:15:15.997433+0000","last_fullsized":"2024-04-29T14:15:15.997433+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'7355","last_scrub_stamp":"2024-04-29T14:14:56.640722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:14:56.640722+0000","objects_scrubbed":7286,"log_size":8760,"log_dups_size":0,"ondisk_log_size":8760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:22.512009+0000","scrub_duration":1570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574095360,"num_objects":8760,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8760,"num_write_kb":560640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"244'8763","reported_seq":8937,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-29T14:15:16.171554+0000","last_change":"2024-04-29T14:15:08.369373+0000","last_active":"2024-04-29T14:15:16.171554+0000","last_peered":"2024-04-29T14:15:16.171554+0000","last_clean":"2024-04-29T14:15:16.171554+0000","last_became_active":"2024-04-29T14:13:43.177685+0000","last_became_peered":"2024-04-29T14:13:43.177685+0000","last_unstale":"2024-04-29T14:15:16.171554+0000","last_undegraded":"2024-04-29T14:15:16.171554+0000","last_fullsized":"2024-04-29T14:15:16.171554+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8288","last_scrub_stamp":"2024-04-29T14:15:08.369244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:15:08.369244+0000","objects_scrubbed":8212,"log_size":8763,"log_dups_size":0,"ondisk_log_size":8763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:08.531867+0000","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574291968,"num_objects":8763,"num_object_clones":0,"num_object_copies":17526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8763,"num_write_kb":560832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"243'8595","reported_seq":8749,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:15.344701+0000","last_change":"2024-04-29T14:15:02.546953+0000","last_active":"2024-04-29T14:15:15.344701+0000","last_peered":"2024-04-29T14:15:15.344701+0000","last_clean":"2024-04-29T14:15:15.344701+0000","last_became_active":"2024-04-29T14:13:52.443296+0000","last_became_peered":"2024-04-29T14:13:52.443296+0000","last_unstale":"2024-04-29T14:15:15.344701+0000","last_undegraded":"2024-04-29T14:15:15.344701+0000","last_fullsized":"2024-04-29T14:15:15.344701+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'7744","last_scrub_stamp":"2024-04-29T14:15:02.546712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:15:02.546712+0000","objects_scrubbed":7694,"log_size":8595,"log_dups_size":0,"ondisk_log_size":8595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:13.206614+0000","scrub_duration":1132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":563281920,"num_objects":8595,"num_object_clones":0,"num_object_copies":17190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":550080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2","version":"242'8716","reported_seq":11104,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-29T14:15:13.584350+0000","last_change":"2024-04-29T14:14:53.235002+0000","last_active":"2024-04-29T14:15:13.584350+0000","last_peered":"2024-04-29T14:15:13.584350+0000","last_clean":"2024-04-29T14:15:13.584350+0000","last_became_active":"2024-04-29T14:14:06.644597+0000","last_became_peered":"2024-04-29T14:14:06.644597+0000","last_unstale":"2024-04-29T14:15:13.584350+0000","last_undegraded":"2024-04-29T14:15:13.584350+0000","last_fullsized":"2024-04-29T14:15:13.584350+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7147","last_scrub_stamp":"2024-04-29T14:14:53.234838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:14:53.234838+0000","objects_scrubbed":7050,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:11.266524+0000","scrub_duration":1861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":571211776,"num_objects":8716,"num_object_clones":0,"num_object_copies":17432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8716,"num_write_kb":557824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":72810496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":667,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-29T14:15:14.049453+0000","last_change":"2024-04-29T14:15:02.736465+0000","last_active":"2024-04-29T14:15:14.049453+0000","last_peered":"2024-04-29T14:15:14.049453+0000","last_clean":"2024-04-29T14:15:14.049453+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:15:14.049453+0000","last_undegraded":"2024-04-29T14:15:14.049453+0000","last_fullsized":"2024-04-29T14:15:14.049453+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:15:02.736312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:15:02.736312+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:16:27.328182+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":9152954368,"num_objects":139663,"num_object_clones":0,"num_object_copies":279326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139663,"num_write_kb":8938432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2270,"num_bytes_recovered":148766720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18321113088,"data_stored":18321113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140562,"ondisk_log_size":140562,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122147,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4143012,"kb_used_data":3904804,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":89602140,"statfs":{"total":95995035648,"available":91752591360,"internally_reserved":0,"allocated":3998519296,"data_stored":3998025976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":243726788},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383784,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":126676,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618476,"statfs":{"total":95995035648,"available":95865319424,"internally_reserved":0,"allocated":1118208,"data_stored":540371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":128580032},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741284,"kb_used_data":547676,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93003868,"statfs":{"total":95995035648,"available":95235960832,"internally_reserved":0,"allocated":560820224,"data_stored":560330476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":198244800},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877907059,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":685660,"kb_used_data":555740,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93059492,"statfs":{"total":95995035648,"available":95292919808,"internally_reserved":0,"allocated":569077760,"data_stored":568588012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234102,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1291236,"kb_used_data":1119652,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92453916,"statfs":{"total":95995035648,"available":94672809984,"internally_reserved":0,"allocated":1146523648,"data_stored":1146030348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,3,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1361692,"kb_used_data":1123484,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92383460,"statfs":{"total":95995035648,"available":94600663040,"internally_reserved":0,"allocated":1150447616,"data_stored":1149957868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243726790},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2384028,"kb_used_data":2236060,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91361124,"statfs":{"total":95995035648,"available":93553790976,"internally_reserved":0,"allocated":2289725440,"data_stored":2289235692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":85899346054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182540,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2986956,"kb_used_data":2761412,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":90758196,"statfs":{"total":95995035648,"available":92936392704,"internally_reserved":0,"allocated":2827685888,"data_stored":2827108071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":230947272},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986311,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3611484,"kb_used_data":3349660,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":90133668,"statfs":{"total":95995035648,"available":92296876032,"internally_reserved":0,"allocated":3430051840,"data_stored":3429562092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822784,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2421348,"kb_used_data":2231900,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":91323804,"statfs":{"total":95995035648,"available":93515575296,"internally_reserved":0,"allocated":2285465600,"data_stored":2284975852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2296381440,"data_stored":2296381440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2835480576,"data_stored":2835480576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3454468096,"data_stored":3454468096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2296643584,"data_stored":2296643584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1154940928,"data_stored":1154940928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1152188416,"data_stored":1152188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":569114624,"data_stored":569114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":563281920,"data_stored":563281920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3998613504,"data_stored":3998613504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:15:19.311 INFO:tasks.thrashosds.thrasher:Setting 3.d to [4, 1, 8, 3] 2024-04-29T14:15:19.312 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.d', '4', '1', '8', '3'] 2024-04-29T14:15:19.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 3.d 4 1 8 3 2024-04-29T14:15:19.432 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:19.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:19.537 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:19.570 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:19.592 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:19.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:19.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:19.743 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:19.784 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:19.784 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:19.785 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:19.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:19.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:19.942 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:20.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:20.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:20.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:20.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 144909 144893 116.091 61.6875 0.0116094 0.00861082 2024-04-29T14:15:20.230 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:20 smithi040 ceph-mon[39581]: from='client.15234 v1:172.21.15.40:0/817321249' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:15:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:20 smithi040 ceph-mon[39581]: pgmap v663: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 62 MiB/s wr, 984 op/s 2024-04-29T14:15:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:20.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:20 smithi121 ceph-mon[44533]: from='client.15234 v1:172.21.15.40:0/817321249' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:15:20.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:20 smithi121 ceph-mon[44533]: pgmap v663: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 62 MiB/s wr, 984 op/s 2024-04-29T14:15:20.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:20.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:20.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:20.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:20.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:15:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:20 smithi194 ceph-mon[44089]: from='client.15234 v1:172.21.15.40:0/817321249' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:15:20.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:20 smithi194 ceph-mon[44089]: pgmap v663: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 62 MiB/s wr, 984 op/s 2024-04-29T14:15:20.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:20.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:20.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:20.631 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:20.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:20.800 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:20.841 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:20.871 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:20.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:20.902 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:20.983 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:21.016 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:21.091 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:15:21.169 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:21.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:21.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 145954 145938 115.448 65.3125 0.0113142 0.00865992 2024-04-29T14:15:21.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:21.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:21 smithi040 ceph-mon[39581]: osdmap e245: 12 total, 11 up, 10 in 2024-04-29T14:15:21.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3353609963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:21.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:21 smithi040 ceph-mon[39581]: from='client.25168 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:21 smithi121 ceph-mon[44533]: osdmap e245: 12 total, 11 up, 10 in 2024-04-29T14:15:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3353609963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:21.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:21 smithi121 ceph-mon[44533]: from='client.25168 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:21.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:21 smithi194 ceph-mon[44089]: osdmap e245: 12 total, 11 up, 10 in 2024-04-29T14:15:21.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3353609963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:21.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:21 smithi194 ceph-mon[44089]: from='client.25168 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:21.529 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:22.065 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:22.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:22.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:22.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:15:22.218472 min lat: 0.00103764 max lat: 2.08667 avg lat: 0.00869454 2024-04-29T14:15:22.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:15:22.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 147217 147201 114.991 78.9375 0.00564215 0.00869454 2024-04-29T14:15:22.218 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:22.261 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:22.265 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:22 smithi194 ceph-mon[44089]: pgmap v665: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-29T14:15:22.265 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:22 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:22.265 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:22 smithi194 ceph-mon[44089]: from='client.25168 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:22.265 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:22 smithi194 ceph-mon[44089]: osdmap e246: 12 total, 11 up, 10 in 2024-04-29T14:15:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:22.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:22.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:22 smithi040 ceph-mon[39581]: pgmap v665: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-29T14:15:22.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:22 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:22.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:22 smithi040 ceph-mon[39581]: from='client.25168 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:22.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:22 smithi040 ceph-mon[39581]: osdmap e246: 12 total, 11 up, 10 in 2024-04-29T14:15:22.369 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:22 smithi121 ceph-mon[44533]: pgmap v665: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-29T14:15:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:22 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:22 smithi121 ceph-mon[44533]: from='client.25168 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:22 smithi121 ceph-mon[44533]: osdmap e246: 12 total, 11 up, 10 in 2024-04-29T14:15:22.401 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:22.658 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:22.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:22.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:22.807 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:22.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:22.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:22.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:22.975 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:22.975 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:22.985 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:23.011 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:23.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:23.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:23.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 148447 148431 114.521 76.875 0.00673171 0.00873095 2024-04-29T14:15:23.284 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:23.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:23.391 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:23.429 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:23.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:23.718 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:15:23.789 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:24.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:24.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:15:24.094 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:24.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:24.201 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:24.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 149681 149665 114.064 77.125 0.0138954 0.00876571 2024-04-29T14:15:24.349 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:24.349 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:24.349 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:24 smithi194 ceph-mon[44089]: pgmap v667: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 53 MiB/s wr, 843 op/s 2024-04-29T14:15:24.349 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:24 smithi194 ceph-mon[44089]: osdmap e247: 12 total, 11 up, 10 in 2024-04-29T14:15:24.349 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:24.349 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:24.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:24.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:24.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:24 smithi040 ceph-mon[39581]: pgmap v667: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 53 MiB/s wr, 843 op/s 2024-04-29T14:15:24.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:24 smithi040 ceph-mon[39581]: osdmap e247: 12 total, 11 up, 10 in 2024-04-29T14:15:24.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:24.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:24.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:24 smithi121 ceph-mon[44533]: pgmap v667: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 53 MiB/s wr, 843 op/s 2024-04-29T14:15:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:24 smithi121 ceph-mon[44533]: osdmap e247: 12 total, 11 up, 10 in 2024-04-29T14:15:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:24.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:24.509 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:24.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:24.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:24.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:24.687 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:15:24.710 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:15:24.710 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:24.738 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:15:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:24.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:24.776 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:15:24.814 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:15:24.841 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:15:24.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:24.870 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:15:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:24.905 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:15:24.906 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:24.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:24.934 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:24.937 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:15:24.971 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:15:25.006 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:15:25.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:15:25.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:15:25.006 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:15:25.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:15:25.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:15:25.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:15:25.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:15:25.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:15:25.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:15:25.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:15:25.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:15:25.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:15:25.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:25.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:25.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:25.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:25.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 15 150722 150707 113.475 65.125 0.0142263 0.00881115 2024-04-29T14:15:25.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:25 smithi121 ceph-mon[44533]: osdmap e248: 12 total, 11 up, 10 in 2024-04-29T14:15:25.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2914504762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:25.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:25.257 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:25.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:25 smithi040 ceph-mon[39581]: osdmap e248: 12 total, 11 up, 10 in 2024-04-29T14:15:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2914504762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:25.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:25.419 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:25 smithi194 ceph-mon[44089]: osdmap e248: 12 total, 11 up, 10 in 2024-04-29T14:15:25.419 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2914504762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:25.419 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:25.427 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:25.529 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:25.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:25.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:25.665 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:25.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:25.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:25.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:25.842 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:26.051 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:26.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:26.071 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:15:26.071 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:15:26.071 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:15:26.071 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:15:26.071 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:15:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:15:26.106 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:15:26.129 INFO:teuthology.orchestra.run.smithi040.stderr:set 3.d pg_upmap_items mapping to [4->1,8->3] 2024-04-29T14:15:26.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:26.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 151938 151922 113.028 75.9375 0.0133302 0.00884619 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: pgmap v670: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 55 MiB/s wr, 885 op/s 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3952640632' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [4, 1, 8, 3]}]: dispatch 2024-04-29T14:15:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: pgmap v670: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 55 MiB/s wr, 885 op/s 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3952640632' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [4, 1, 8, 3]}]: dispatch 2024-04-29T14:15:26.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: pgmap v670: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 55 MiB/s wr, 885 op/s 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:26.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3952640632' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [4, 1, 8, 3]}]: dispatch 2024-04-29T14:15:26.822 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:15:27.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 15 153169 153154 112.604 77 0.00943699 0.00887962 2024-04-29T14:15:27.214 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:27.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3952640632' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [4, 1, 8, 3]}]': finished 2024-04-29T14:15:27.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:27 smithi040 ceph-mon[39581]: osdmap e249: 12 total, 11 up, 10 in 2024-04-29T14:15:27.491 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3952640632' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [4, 1, 8, 3]}]': finished 2024-04-29T14:15:27.491 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:27 smithi194 ceph-mon[44089]: osdmap e249: 12 total, 11 up, 10 in 2024-04-29T14:15:27.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3952640632' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [4, 1, 8, 3]}]': finished 2024-04-29T14:15:27.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:27 smithi121 ceph-mon[44533]: osdmap e249: 12 total, 11 up, 10 in 2024-04-29T14:15:27.857 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:27.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:27.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:28.103 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:28.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:15:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:28.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:28.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:28.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 154395 154379 112.185 76.5625 0.00615301 0.00891244 2024-04-29T14:15:28.263 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:28.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:28.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:28.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:28 smithi040 ceph-mon[39581]: pgmap v672: 17 pgs: 17 active+clean; 9.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:15:28.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:28.435 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:28 smithi121 ceph-mon[44533]: pgmap v672: 17 pgs: 17 active+clean; 9.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:15:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:28.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:28.525 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:28.543 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:28 smithi194 ceph-mon[44089]: pgmap v672: 17 pgs: 17 active+clean; 9.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:15:28.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:28.610 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:28.785 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:28.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:28.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:28.941 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:28.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:28.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:29.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:29.116 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:29.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1353672020' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:15:29.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:29 smithi194 ceph-mon[44089]: from='client.25186 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:15:29.173 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:15:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:29.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:29.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 155568 155552 111.738 73.3125 0.0100133 0.00894812 2024-04-29T14:15:29.219 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:15:29.219 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:15:29.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:15:29.219 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:319893 2.10 2.877cf690 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:06.055893+0000", 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 143.16553502799999, 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.91288065600000001, 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.220 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 319893 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055893+0000", 2024-04-29T14:15:29.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055893+0000", 2024-04-29T14:15:29.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055895+0000", 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3630000000000001e-06 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055904+0000", 2024-04-29T14:15:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5979999999999997e-06 2024-04-29T14:15:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055917+0000", 2024-04-29T14:15:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2923e-05 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055926+0000", 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4450000000000002e-06 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055992+0000", 2024-04-29T14:15:29.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5813000000000002e-05 2024-04-29T14:15:29.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:15:29.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.056003+0000", 2024-04-29T14:15:29.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1532000000000001e-05 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968728+0000", 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.9127246 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968773+0000", 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5382e-05 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.228 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:319895 2.10 2.d2045630 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:15:29.229 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:06.055980+0000", 2024-04-29T14:15:29.229 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 143.16544723999999, 2024-04-29T14:15:29.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.91284100300000004, 2024-04-29T14:15:29.229 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.229 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.229 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:15:29.229 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 319895 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055980+0000", 2024-04-29T14:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055980+0000", 2024-04-29T14:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055981+0000", 2024-04-29T14:15:29.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0699999999999997e-07 2024-04-29T14:15:29.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055984+0000", 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.785e-06 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055988+0000", 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7400000000000004e-06 2024-04-29T14:15:29.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.055994+0000", 2024-04-29T14:15:29.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.818e-06 2024-04-29T14:15:29.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.056030+0000", 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5442999999999999e-05 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.056037+0000", 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9280000000000004e-06 2024-04-29T14:15:29.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968811+0000", 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.91277408199999999 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968821+0000", 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0699999999999999e-05 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:319923 2.10 2.89b850d0 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:15:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:06.077933+0000", 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 143.143494782, 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.89092813100000001, 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:15:29.238 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 319923 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.077933+0000", 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.077933+0000", 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.077934+0000", 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.404e-06 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.077939+0000", 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2109999999999997e-06 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.077949+0000", 2024-04-29T14:15:29.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9110000000000005e-06 2024-04-29T14:15:29.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.077957+0000", 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3050000000000004e-06 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.078008+0000", 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1022e-05 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.078015+0000", 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4479999999999997e-06 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968853+0000", 2024-04-29T14:15:29.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.89083748399999996 2024-04-29T14:15:29.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968861+0000", 2024-04-29T14:15:29.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3459999999999994e-06 2024-04-29T14:15:29.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.245 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.246 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.246 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:319965 2.10 2.37dba230 (undecoded) ondisk+write+known_if_redirected e189)", 2024-04-29T14:15:29.246 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:06.132925+0000", 2024-04-29T14:15:29.246 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 143.08850227299999, 2024-04-29T14:15:29.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83597666100000001, 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 319965 2024-04-29T14:15:29.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.132925+0000", 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.132925+0000", 2024-04-29T14:15:29.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.132928+0000", 2024-04-29T14:15:29.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5450000000000002e-06 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.132936+0000", 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7230000000000004e-06 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.132949+0000", 2024-04-29T14:15:29.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2876e-05 2024-04-29T14:15:29.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.132957+0000", 2024-04-29T14:15:29.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0290000000000005e-06 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.133012+0000", 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5009999999999997e-05 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.133021+0000", 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5119999999999997e-06 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968894+0000", 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83587256200000004 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:06.968902+0000", 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4039999999999995e-06 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:319969 2.6 2:61b1dea4:::benchmark_data_smithi194_79000_object155708:head [delete] snapc 0=[] ondisk+write+known_if_redirected e191)", 2024-04-29T14:15:29.253 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:08.139734+0000", 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 141.08169322000001, 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19178129099999999, 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 319969 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139734+0000", 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139734+0000", 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139736+0000", 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9740000000000001e-06 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139742+0000", 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6710000000000004e-06 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139752+0000", 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0028e-05 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139758+0000", 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000003e-06 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139815+0000", 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6588000000000001e-05 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.139827+0000", 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1742999999999999e-05 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.309061+0000", 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16923407900000001 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.309162+0000", 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101199 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1,3", 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.309743+0000", 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058094699999999995 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.322440+0000", 2024-04-29T14:15:29.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012697152 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.322485+0000", 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5012999999999997e-05 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.331350+0000", 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088646909999999992 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.331401+0000", 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0779999999999998e-05 2024-04-29T14:15:29.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:08.331516+0000", 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115176 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23178 3.9 3:971c24a9:::benchmark_data_smithi194_88551_object23177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.049295+0000", 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.172133115999998, 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20513878099999999, 2024-04-29T14:15:29.260 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23178 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049295+0000", 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049295+0000", 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049295+0000", 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6400000000000002e-07 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049311+0000", 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6317e-05 2024-04-29T14:15:29.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049316+0000", 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0829999999999997e-06 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049319+0000", 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4769999999999999e-06 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.050300+0000", 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00098083900000000006 2024-04-29T14:15:29.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.050391+0000", 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0938000000000002e-05 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.050527+0000", 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000135881 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053092+0000", 2024-04-29T14:15:29.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025657610000000002 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.254343+0000", 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.201251071 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.254389+0000", 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5373000000000001e-05 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.254433+0000", 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4477000000000001e-05 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23179 3.9 3:9cdc2169:::benchmark_data_smithi194_88551_object23178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.049605+0000", 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.171822813999995, 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20496076099999999, 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.266 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23179 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049605+0000", 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049605+0000", 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049606+0000", 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2669999999999999e-06 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049629+0000", 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2744000000000001e-05 2024-04-29T14:15:29.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049639+0000", 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9699999999999994e-06 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049646+0000", 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5329999999999997e-06 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.051166+0000", 2024-04-29T14:15:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015197419999999999 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.051293+0000", 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012723699999999999 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.051433+0000", 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013931300000000001 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053100+0000", 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016669300000000001 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.254474+0000", 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20137467100000001 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.254525+0000", 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0793e-05 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.254566+0000", 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0561e-05 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23180 3.9 3:9c961f06:::benchmark_data_smithi194_88551_object23179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.049660+0000", 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.171768002999997, 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20656461100000001, 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23180 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049660+0000", 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049660+0000", 2024-04-29T14:15:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049660+0000", 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.99e-07 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049675+0000", 2024-04-29T14:15:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4844999999999999e-05 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049680+0000", 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4390000000000002e-06 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.049683+0000", 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6990000000000001e-06 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.052239+0000", 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002555608 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.052363+0000", 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000124058 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.052452+0000", 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8931999999999995e-05 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053586+0000", 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011337280000000001 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.256161+0000", 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20257536900000001 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.256200+0000", 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8685e-05 2024-04-29T14:15:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.256224+0000", 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4649000000000001e-05 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23200 3.9 3:96420a02:::benchmark_data_smithi194_88551_object23199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.053944+0000", 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.167483695000001, 2024-04-29T14:15:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20318185, 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23200 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053944+0000", 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053944+0000", 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053946+0000", 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2299999999999998e-06 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053966+0000", 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0069999999999999e-05 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053976+0000", 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0125e-05 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.053983+0000", 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.319e-06 2024-04-29T14:15:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054033+0000", 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0194000000000001e-05 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054142+0000", 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108992 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054229+0000", 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6656000000000005e-05 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.055562+0000", 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0013339280000000001 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.257044+0000", 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20148128900000001 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.257096+0000", 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2114999999999998e-05 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.257126+0000", 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9932000000000001e-05 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23203 3.9 3:9f552434:::benchmark_data_smithi194_88551_object23202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.054465+0000", 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.166962534000007, 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20278627299999999, 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23203 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054465+0000", 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054465+0000", 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054467+0000", 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4729999999999999e-06 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054491+0000", 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.406e-05 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054500+0000", 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5240000000000006e-06 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054507+0000", 2024-04-29T14:15:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8009999999999998e-06 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.054869+0000", 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036192800000000002 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.055001+0000", 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000132275 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.055089+0000", 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7767000000000002e-05 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.055724+0000", 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063503699999999997 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.257162+0000", 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.201438224 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.257213+0000", 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0676000000000001e-05 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.257251+0000", 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8507999999999998e-05 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23219 3.9 3:96f85720:::benchmark_data_smithi194_88551_object23218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.060137+0000", 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.161291031000005, 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19839596100000001, 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23219 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060137+0000", 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060137+0000", 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060138+0000", 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8080000000000001e-06 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060159+0000", 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0343000000000002e-05 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060174+0000", 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4877e-05 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060183+0000", 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060225+0000", 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2645000000000003e-05 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060315+0000", 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0074000000000005e-05 2024-04-29T14:15:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.060402+0000", 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6311999999999996e-05 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.061126+0000", 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00072453100000000005 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.258425+0000", 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.197298896 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.258485+0000", 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9620000000000002e-05 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.258533+0000", 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7812000000000002e-05 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23228 3.9 3:975bd627:::benchmark_data_smithi194_88551_object23227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.062847+0000", 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.158580594, 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19582159199999999, 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23228 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.062847+0000", 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.062847+0000", 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.062849+0000", 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6920000000000001e-06 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.062890+0000", 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0970999999999998e-05 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.062902+0000", 2024-04-29T14:15:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1848000000000001e-05 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.062910+0000", 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9789999999999993e-06 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.062955+0000", 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4985999999999999e-05 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063038+0000", 2024-04-29T14:15:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3819999999999994e-05 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063171+0000", 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013215899999999999 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.064439+0000", 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001268438 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.258579+0000", 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19414015800000001 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.258634+0000", 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4895999999999998e-05 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.258669+0000", 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4644999999999998e-05 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23233 3.9 3:9ffde6eb:::benchmark_data_smithi194_88551_object23232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.063386+0000", 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.158042037000001, 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19673211600000001, 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23233 2024-04-29T14:15:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063386+0000", 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063386+0000", 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063387+0000", 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.206e-06 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063424+0000", 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7577000000000003e-05 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063433+0000", 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2519999999999999e-06 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063439+0000", 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7580000000000003e-06 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063811+0000", 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000371412 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063897+0000", 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6397999999999995e-05 2024-04-29T14:15:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.063998+0000", 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000100956 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.064517+0000", 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051833000000000001 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260014+0000", 2024-04-29T14:15:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.195497635 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260070+0000", 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6261999999999998e-05 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260118+0000", 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7330000000000003e-05 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23242 3.9 3:91b337fe:::benchmark_data_smithi194_88551_object23241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.066687+0000", 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.154740204000007, 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.193572942, 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23242 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066687+0000", 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066687+0000", 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066689+0000", 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3659999999999999e-06 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066731+0000", 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2460999999999999e-05 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066740+0000", 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8599999999999999e-06 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066747+0000", 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7499999999999997e-06 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066786+0000", 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8720000000000002e-05 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066862+0000", 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5903999999999997e-05 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.066946+0000", 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4302999999999996e-05 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.067521+0000", 2024-04-29T14:15:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00057481899999999996 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260181+0000", 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19266070900000001 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260235+0000", 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3446000000000003e-05 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260260+0000", 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5604000000000001e-05 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23259 3.9 3:9c447d11:::benchmark_data_smithi194_88551_object23258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.074741+0000", 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.146686372999994, 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18567112299999999, 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23259 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074741+0000", 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074741+0000", 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074743+0000", 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.781e-06 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074787+0000", 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4319e-05 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074801+0000", 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3519999999999999e-05 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074809+0000", 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1710000000000002e-06 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074862+0000", 2024-04-29T14:15:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3426e-05 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.074954+0000", 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1551000000000005e-05 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075048+0000", 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4399000000000002e-05 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.076331+0000", 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012830140000000001 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260317+0000", 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18398536200000001 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260374+0000", 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6764000000000001e-05 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260412+0000", 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8816000000000002e-05 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23261 3.9 3:99ba846a:::benchmark_data_smithi194_88551_object23260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.075278+0000", 2024-04-29T14:15:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.146149609000005, 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18527606699999999, 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23261 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075278+0000", 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075278+0000", 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075279+0000", 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2300000000000001e-06 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075326+0000", 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6859e-05 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075336+0000", 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0291e-05 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075343+0000", 2024-04-29T14:15:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0149999999999998e-06 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075606+0000", 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00026279999999999999 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075723+0000", 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116728 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.075868+0000", 2024-04-29T14:15:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014544499999999999 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.076463+0000", 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059447600000000003 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260479+0000", 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.184015866 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260521+0000", 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2549000000000002e-05 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.260554+0000", 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2808000000000002e-05 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23270 3.9 3:979c32ec:::benchmark_data_smithi194_88551_object23269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.078736+0000", 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.142691948999996, 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18323783499999999, 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23270 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078736+0000", 2024-04-29T14:15:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078736+0000", 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078737+0000", 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5540000000000001e-06 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078775+0000", 2024-04-29T14:15:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7929999999999998e-05 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078787+0000", 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1447999999999999e-05 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078795+0000", 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0900000000000005e-06 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078843+0000", 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8739999999999998e-05 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.078958+0000", 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011410500000000001 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.079054+0000", 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6390000000000004e-05 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.079602+0000", 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00054823999999999997 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.261910+0000", 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18230806299999999 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.261948+0000", 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7639999999999999e-05 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.261974+0000", 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5635000000000001e-05 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23283 3.9 3:981c708d:::benchmark_data_smithi194_88551_object23282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.085734+0000", 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.135693297000003, 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17634419000000001, 2024-04-29T14:15:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23283 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085734+0000", 2024-04-29T14:15:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085734+0000", 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085736+0000", 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3659999999999999e-06 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085772+0000", 2024-04-29T14:15:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5917999999999997e-05 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085781+0000", 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9719999999999998e-06 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085786+0000", 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4530000000000001e-06 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085836+0000", 2024-04-29T14:15:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0182999999999997e-05 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.085919+0000", 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.229e-05 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.086000+0000", 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1681000000000006e-05 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.086787+0000", 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078711999999999999 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.262010+0000", 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.175222773 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.262040+0000", 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9485e-05 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.262079+0000", 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8949000000000003e-05 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23297 3.9 3:9e59da6c:::benchmark_data_smithi194_88551_object23296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.095397+0000", 2024-04-29T14:15:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.126031034999997, 2024-04-29T14:15:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.166811125, 2024-04-29T14:15:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23297 2024-04-29T14:15:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095397+0000", 2024-04-29T14:15:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095397+0000", 2024-04-29T14:15:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095398+0000", 2024-04-29T14:15:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.77e-06 2024-04-29T14:15:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095436+0000", 2024-04-29T14:15:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8027e-05 2024-04-29T14:15:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095447+0000", 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0121e-05 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095453+0000", 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3069999999999999e-06 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095504+0000", 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0841999999999999e-05 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095594+0000", 2024-04-29T14:15:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0315999999999994e-05 2024-04-29T14:15:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.095680+0000", 2024-04-29T14:15:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5912e-05 2024-04-29T14:15:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.096527+0000", 2024-04-29T14:15:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00084736499999999999 2024-04-29T14:15:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.262123+0000", 2024-04-29T14:15:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.165595627 2024-04-29T14:15:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.262173+0000", 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9873000000000002e-05 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.262208+0000", 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4965000000000002e-05 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25063.0:23312 3.9 3:94d42c50:::benchmark_data_smithi194_88551_object23311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:14:09.109153+0000", 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 80.112274209999995, 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.154176595, 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25063", 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:15:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 23312 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109153+0000", 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109153+0000", 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:15:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109155+0000", 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5009999999999999e-06 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109200+0000", 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5049999999999997e-05 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109212+0000", 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1932e-05 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109220+0000", 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9279999999999993e-06 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109259+0000", 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9276999999999997e-05 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109344+0000", 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5047000000000001e-05 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 11", 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.109435+0000", 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0345000000000003e-05 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.110086+0000", 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000651565 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.263243+0000", 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15315649200000001 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.263294+0000", 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0924000000000002e-05 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:14:09.263330+0000", 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6534e-05 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:15:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:15:29.357 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:15:29.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1353672020' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:15:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:29 smithi040 ceph-mon[39581]: from='client.25186 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:15:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1353672020' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:15:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:29 smithi121 ceph-mon[44533]: from='client.25186 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:15:29.390 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:29.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:29.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:29.791 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:29.825 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:29.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:29.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:30.017 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:15:30.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:30.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:30.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:30.162 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:30.211 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 156775 156759 111.325 75.4375 0.00587504 0.00898123 2024-04-29T14:15:30.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:30 smithi194 ceph-mon[44089]: pgmap v673: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-29T14:15:30.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:30 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:30.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:30 smithi194 ceph-mon[44089]: from='client.25186 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:15:30.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:30 smithi194 ceph-mon[44089]: osdmap e250: 12 total, 11 up, 10 in 2024-04-29T14:15:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:30 smithi040 ceph-mon[39581]: pgmap v673: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-29T14:15:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:30 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:30.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:30 smithi040 ceph-mon[39581]: from='client.25186 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:15:30.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:30 smithi040 ceph-mon[39581]: osdmap e250: 12 total, 11 up, 10 in 2024-04-29T14:15:30.372 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:30 smithi121 ceph-mon[44533]: pgmap v673: 17 pgs: 17 active+clean; 9.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-29T14:15:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:30 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:30 smithi121 ceph-mon[44533]: from='client.25186 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:15:30.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:30 smithi121 ceph-mon[44533]: osdmap e250: 12 total, 11 up, 10 in 2024-04-29T14:15:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:30.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:30.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:30.764 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:30.894 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:30.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:31.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:31.046 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:31.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:31.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:31.178 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:31 smithi040 ceph-mon[39581]: 3.4 scrub starts 2024-04-29T14:15:31.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:31.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 157904 157888 110.867 70.5625 0.013502 0.00901856 2024-04-29T14:15:31.232 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:15:31.242 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:31.261 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:15:31.283 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:15:31.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:31.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:31 smithi121 ceph-mon[44533]: 3.4 scrub starts 2024-04-29T14:15:31.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:31.320 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:15:31.351 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:15:31.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:15:31.397 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:15:31.431 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:15:31.437 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:31.445 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:31 smithi194 ceph-mon[44089]: 3.4 scrub starts 2024-04-29T14:15:31.458 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:15:31.463 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:31.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:31.488 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:15:31.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:15:31.514 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:15:31.514 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:15:31.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:31.620 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:31.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:31.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:31.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:31.823 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:31.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:31.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:31.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:32.014 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:32.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:15:32.148 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:32.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:32.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 90 16 158940 158924 110.355 64.75 0.0277133 0.00905965 2024-04-29T14:15:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:15:32.218 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:15:32.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:32.283 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:32.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:32.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:32.413 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 8, 9, 10, 11, 6] out_osds: [1, 2] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:15:32.413 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:15:32.413 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 3, 4, 5, 7, 8, 9, 10, 11, 6] 2024-04-29T14:15:32.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 3 2024-04-29T14:15:32.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:32.430 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: pgmap v675: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1485729134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.441 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: pgmap v675: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1485729134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.459 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:32.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:32.476 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:32.477 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: pgmap v675: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1485729134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:32.563 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:32.594 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:32.722 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:32.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:32.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:32.854 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:33.207 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:33.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 90.0146 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 158940 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 110.357 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 59.0473 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 383.75 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 10.1875 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1765 2024-04-29T14:15:33.212 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 944.757 2024-04-29T14:15:33.213 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 6140 2024-04-29T14:15:33.213 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 163 2024-04-29T14:15:33.213 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.00906083 2024-04-29T14:15:33.213 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.022787 2024-04-29T14:15:33.213 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 2.08667 2024-04-29T14:15:33.213 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.00103764 2024-04-29T14:15:33.217 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:15:33.220 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:15:33.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:33.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:33.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:33 smithi040 ceph-mon[39581]: 3.f scrub starts 2024-04-29T14:15:33.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:33 smithi040 ceph-mon[39581]: 3.9 scrub starts 2024-04-29T14:15:33.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:33 smithi040 ceph-mon[39581]: 3.4 scrub ok 2024-04-29T14:15:33.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:33 smithi040 ceph-mon[39581]: pgmap v676: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:15:33.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:33 smithi040 ceph-mon[39581]: 3.9 scrub ok 2024-04-29T14:15:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:33.390 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:33.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:33.503 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:33.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:33.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:33 smithi194 ceph-mon[44089]: 3.f scrub starts 2024-04-29T14:15:33.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:33 smithi194 ceph-mon[44089]: 3.9 scrub starts 2024-04-29T14:15:33.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:33 smithi194 ceph-mon[44089]: 3.4 scrub ok 2024-04-29T14:15:33.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:33 smithi194 ceph-mon[44089]: pgmap v676: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:15:33.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:33 smithi194 ceph-mon[44089]: 3.9 scrub ok 2024-04-29T14:15:33.537 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:33.570 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:33.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:33 smithi121 ceph-mon[44533]: 3.f scrub starts 2024-04-29T14:15:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:33 smithi121 ceph-mon[44533]: 3.9 scrub starts 2024-04-29T14:15:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:33 smithi121 ceph-mon[44533]: 3.4 scrub ok 2024-04-29T14:15:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:33 smithi121 ceph-mon[44533]: pgmap v676: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:15:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:33 smithi121 ceph-mon[44533]: 3.9 scrub ok 2024-04-29T14:15:33.626 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:33.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:33.735 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:33.773 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:33.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:33.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:33.826 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:33.884 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:33.935 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:33.964 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:33.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:34.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:34.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:34.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:34.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:34.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:15:34.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:34.305 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:34.345 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:34 smithi040 ceph-mon[39581]: 3.f scrub ok 2024-04-29T14:15:34.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:34 smithi121 ceph-mon[44533]: 3.f scrub ok 2024-04-29T14:15:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:34 smithi194 ceph-mon[44089]: 3.f scrub ok 2024-04-29T14:15:34.664 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:34.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:34.769 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:34.799 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:15:35.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2574246334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:35 smithi194 ceph-mon[44089]: from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:35 smithi194 ceph-mon[44089]: pgmap v677: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 55 MiB/s wr, 902 op/s 2024-04-29T14:15:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:15:35.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2574246334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:35 smithi040 ceph-mon[39581]: from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:35 smithi040 ceph-mon[39581]: pgmap v677: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 55 MiB/s wr, 902 op/s 2024-04-29T14:15:35.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:15:35.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:35.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:35.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2574246334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:35.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:35 smithi121 ceph-mon[44533]: from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:35.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:35 smithi121 ceph-mon[44533]: pgmap v677: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 55 MiB/s wr, 902 op/s 2024-04-29T14:15:35.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:15:35.706 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:35.706 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:36.165 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:36 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:15:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:36 smithi040 ceph-mon[39581]: from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:36 smithi040 ceph-mon[39581]: osdmap e251: 12 total, 11 up, 10 in 2024-04-29T14:15:36.411 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:36 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:15:36.411 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:36 smithi194 ceph-mon[44089]: from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:36.411 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:36 smithi194 ceph-mon[44089]: osdmap e251: 12 total, 11 up, 10 in 2024-04-29T14:15:36.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:15:36.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:36 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:15:36.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:36 smithi121 ceph-mon[44533]: from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:36.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:36 smithi121 ceph-mon[44533]: osdmap e251: 12 total, 11 up, 10 in 2024-04-29T14:15:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:36.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:36.781 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:36.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:36.902 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:37.101 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:37.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:37.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:37.239 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:37.321 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:37.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:15:37.047+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e252 definitely_dead 0 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: osdmap e252: 12 total, 11 up, 10 in 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: pgmap v680: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 131 B/s rd, 57 MiB/s wr, 1.21k op/s 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2041982719' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:15:37.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:37 smithi040 ceph-mon[39581]: Client client.admin marked osd.3 out, while it was still marked up 2024-04-29T14:15:37.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:37.463 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: osdmap e252: 12 total, 11 up, 10 in 2024-04-29T14:15:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: pgmap v680: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 131 B/s rd, 57 MiB/s wr, 1.21k op/s 2024-04-29T14:15:37.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:37.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:37.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2041982719' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:15:37.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:37 smithi194 ceph-mon[44089]: Client client.admin marked osd.3 out, while it was still marked up 2024-04-29T14:15:37.565 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:37.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: osdmap e252: 12 total, 11 up, 10 in 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: pgmap v680: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 131 B/s rd, 57 MiB/s wr, 1.21k op/s 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2041982719' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:15:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:37 smithi121 ceph-mon[44533]: Client client.admin marked osd.3 out, while it was still marked up 2024-04-29T14:15:37.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:37.705 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:37.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:37.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:37.911 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.3. 2024-04-29T14:15:38.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:15:38.272 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:38.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:38.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:15:38.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:15:38.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:38 smithi040 ceph-mon[39581]: pgmap v681: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 153 B/s rd, 39 MiB/s wr, 964 op/s 2024-04-29T14:15:38.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:38 smithi040 ceph-mon[39581]: pgmap v682: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 219 B/s rd, 30 MiB/s wr, 970 op/s 2024-04-29T14:15:38.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:38.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2041982719' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-29T14:15:38.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:38 smithi040 ceph-mon[39581]: osdmap e253: 12 total, 11 up, 9 in 2024-04-29T14:15:38.376 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:38.404 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:38.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:15:38.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:15:38.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:38 smithi194 ceph-mon[44089]: pgmap v681: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 153 B/s rd, 39 MiB/s wr, 964 op/s 2024-04-29T14:15:38.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:38 smithi194 ceph-mon[44089]: pgmap v682: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 219 B/s rd, 30 MiB/s wr, 970 op/s 2024-04-29T14:15:38.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:38.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2041982719' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-29T14:15:38.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:38 smithi194 ceph-mon[44089]: osdmap e253: 12 total, 11 up, 9 in 2024-04-29T14:15:38.612 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:38.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:15:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:15:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:38 smithi121 ceph-mon[44533]: pgmap v681: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 153 B/s rd, 39 MiB/s wr, 964 op/s 2024-04-29T14:15:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:38 smithi121 ceph-mon[44533]: pgmap v682: 17 pgs: 17 active+clean; 9.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 219 B/s rd, 30 MiB/s wr, 970 op/s 2024-04-29T14:15:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:15:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2041982719' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-29T14:15:38.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:38 smithi121 ceph-mon[44533]: osdmap e253: 12 total, 11 up, 9 in 2024-04-29T14:15:38.717 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:38.747 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:38.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:15:38.889 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:15:38.895 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:38.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:38.908 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:15:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:15:38.948 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:15:38.968 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:15:38.989 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:15:38.999 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:39.009 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:15:39.023 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:39.023 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:39.029 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:39.032 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:15:39.049 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:39.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:39.052 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:15:39.072 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:15:39.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:15:39.074 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:15:39.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:39.180 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:39.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:39.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:39.304 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:15:39.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:39 smithi194 ceph-mon[44089]: 3.6 scrub starts 2024-04-29T14:15:39.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/583802018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:39.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:39.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:39 smithi194 ceph-mon[44089]: osdmap e254: 12 total, 11 up, 9 in 2024-04-29T14:15:39.332 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:39.351 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:15:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:39.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:15:39.461 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:39.486 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:39 smithi040 ceph-mon[39581]: 3.6 scrub starts 2024-04-29T14:15:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/583802018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:39.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:39 smithi040 ceph-mon[39581]: osdmap e254: 12 total, 11 up, 9 in 2024-04-29T14:15:39.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:39 smithi121 ceph-mon[44533]: 3.6 scrub starts 2024-04-29T14:15:39.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/583802018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:39.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:39.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:39 smithi121 ceph-mon[44533]: osdmap e254: 12 total, 11 up, 9 in 2024-04-29T14:15:39.637 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:39.684 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:39.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:39.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:15:39.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:39.818 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:39.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:39.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:39.970 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:39.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:39.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:40.001 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:40.092 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:40.119 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:40.317 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:40.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:40.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:40.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: 3.6 scrub ok 2024-04-29T14:15:40.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: pgmap v685: 17 pgs: 17 active+clean; 9.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 76 op/s 2024-04-29T14:15:40.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:40 smithi040 ceph-mon[39581]: osdmap e255: 12 total, 11 up, 9 in 2024-04-29T14:15:40.498 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:40.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:15:40.516 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: 3.6 scrub ok 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: pgmap v685: 17 pgs: 17 active+clean; 9.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 76 op/s 2024-04-29T14:15:40.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:40 smithi194 ceph-mon[44089]: osdmap e255: 12 total, 11 up, 9 in 2024-04-29T14:15:40.605 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: 3.6 scrub ok 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: pgmap v685: 17 pgs: 17 active+clean; 9.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 76 op/s 2024-04-29T14:15:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:40 smithi121 ceph-mon[44533]: osdmap e255: 12 total, 11 up, 9 in 2024-04-29T14:15:40.638 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:40.891 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:40.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:40.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:41.058 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:41.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:41.240 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:41 smithi040 ceph-mon[39581]: 3.7 starting backfill to osd.7 from (0'0,0'0] MAX to 252'10289 2024-04-29T14:15:41.241 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:41 smithi040 ceph-mon[39581]: 3.2 starting backfill to osd.4 from (0'0,0'0] MAX to 252'10410 2024-04-29T14:15:41.241 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:41 smithi040 ceph-mon[39581]: 3.5 starting backfill to osd.6 from (0'0,0'0] MAX to 252'10370 2024-04-29T14:15:41.241 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:41 smithi040 ceph-mon[39581]: 3.b starting backfill to osd.4 from (0'0,0'0] MAX to 252'10190 2024-04-29T14:15:41.241 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:41 smithi040 ceph-mon[39581]: 3.c starting backfill to osd.6 from (0'0,0'0] MAX to 252'10308 2024-04-29T14:15:41.275 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:41.306 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:41.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:41 smithi121 ceph-mon[44533]: 3.7 starting backfill to osd.7 from (0'0,0'0] MAX to 252'10289 2024-04-29T14:15:41.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:41 smithi121 ceph-mon[44533]: 3.2 starting backfill to osd.4 from (0'0,0'0] MAX to 252'10410 2024-04-29T14:15:41.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:41 smithi121 ceph-mon[44533]: 3.5 starting backfill to osd.6 from (0'0,0'0] MAX to 252'10370 2024-04-29T14:15:41.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:41 smithi121 ceph-mon[44533]: 3.b starting backfill to osd.4 from (0'0,0'0] MAX to 252'10190 2024-04-29T14:15:41.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:41 smithi121 ceph-mon[44533]: 3.c starting backfill to osd.6 from (0'0,0'0] MAX to 252'10308 2024-04-29T14:15:41.435 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:41.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:41.541 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:41 smithi194 ceph-mon[44089]: 3.7 starting backfill to osd.7 from (0'0,0'0] MAX to 252'10289 2024-04-29T14:15:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:41 smithi194 ceph-mon[44089]: 3.2 starting backfill to osd.4 from (0'0,0'0] MAX to 252'10410 2024-04-29T14:15:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:41 smithi194 ceph-mon[44089]: 3.5 starting backfill to osd.6 from (0'0,0'0] MAX to 252'10370 2024-04-29T14:15:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:41 smithi194 ceph-mon[44089]: 3.b starting backfill to osd.4 from (0'0,0'0] MAX to 252'10190 2024-04-29T14:15:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:41 smithi194 ceph-mon[44089]: 3.c starting backfill to osd.6 from (0'0,0'0] MAX to 252'10308 2024-04-29T14:15:41.570 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:41.756 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:41.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:41.865 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:41.900 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:41.929 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:41.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:42.033 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:42.066 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:42.084 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:15:42.084 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:15:42.084 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:15:42.084 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:15:42.090 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:42.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:42.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:42.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:42.252 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:42.252 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:15:42.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:42.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:42.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:42.442 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:42.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:42.495 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:42 smithi121 ceph-mon[44533]: pgmap v687: 17 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 13 active+clean; 9.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 19129/306642 objects misplaced (6.238%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:15:42.495 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1346613091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:42.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:42.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:42 smithi194 ceph-mon[44089]: pgmap v687: 17 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 13 active+clean; 9.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 19129/306642 objects misplaced (6.238%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:15:42.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1346613091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:42.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:15:42.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:42 smithi040 ceph-mon[39581]: pgmap v687: 17 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 13 active+clean; 9.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 19129/306642 objects misplaced (6.238%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:15:42.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1346613091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:15:42.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:42.660 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:42.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:42.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:42.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:42.861 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:42.878 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:15:42.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:42.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:43.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:43.090 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:43.128 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:43.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:43.206 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:43.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:43.263 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:43.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:43.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:43.418 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:43.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:43 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:43.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1346613091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:15:43.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:43 smithi194 ceph-mon[44089]: osdmap e256: 12 total, 11 up, 9 in 2024-04-29T14:15:43.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:43 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:43.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1346613091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:15:43.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:43 smithi040 ceph-mon[39581]: osdmap e256: 12 total, 11 up, 9 in 2024-04-29T14:15:43.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:43 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:43.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1346613091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:15:43.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:43 smithi121 ceph-mon[44533]: osdmap e256: 12 total, 11 up, 9 in 2024-04-29T14:15:44.388 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 8, 9, 10, 11, 6] out_osds: [1, 2, 3] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-29T14:15:44.389 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:15:44.389 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-29T14:15:44.389 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-04-29T14:15:44.389 DEBUG:teuthology.orchestra.run.smithi040:osd.2> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2.service 2024-04-29T14:15:44.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl start ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2 2024-04-29T14:15:44.425 INFO:journalctl@ceph.osd.2.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T14:15:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:44 smithi194 ceph-mon[44089]: pgmap v689: 17 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 13 active+clean; 9.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 597 op/s; 19129/306642 objects misplaced (6.238%); 1010 KiB/s, 15 objects/s recovering 2024-04-29T14:15:44.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:44 smithi040 ceph-mon[39581]: pgmap v689: 17 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 13 active+clean; 9.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 597 op/s; 19129/306642 objects misplaced (6.238%); 1010 KiB/s, 15 objects/s recovering 2024-04-29T14:15:44.613 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:44 smithi040 systemd[1]: Starting Ceph osd.2 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:15:44.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:44 smithi121 ceph-mon[44533]: pgmap v689: 17 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 13 active+clean; 9.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 597 op/s; 19129/306642 objects misplaced (6.238%); 1010 KiB/s, 15 objects/s recovering 2024-04-29T14:15:46.957 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:46 smithi040 ceph-mon[39581]: pgmap v690: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.2 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 872 op/s; 45362/301828 objects misplaced (15.029%); 16 MiB/s, 255 objects/s recovering 2024-04-29T14:15:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:46.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:47.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:46 smithi194 ceph-mon[44089]: pgmap v690: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.2 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 872 op/s; 45362/301828 objects misplaced (15.029%); 16 MiB/s, 255 objects/s recovering 2024-04-29T14:15:47.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:47.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:46 smithi121 ceph-mon[44533]: pgmap v690: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.2 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 872 op/s; 45362/301828 objects misplaced (15.029%); 16 MiB/s, 255 objects/s recovering 2024-04-29T14:15:47.126 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:47.128 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:47.161 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:47.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:47.262 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:47.294 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:47.547 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:47.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:47.656 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:47.690 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:47.744 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:47.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:47.851 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:47.883 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:47.887 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:15:47.887 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:15:47.887 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:15:47.887 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:15:47.887 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:15:47.904 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:47.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:48.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:48.060 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:48.091 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:48.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:48.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:48.248 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:48.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:48.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:48.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:48.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:48.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:15:48.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-29T14:15:48.673 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:48 smithi040 bash[135443]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-29T14:15:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:48.953 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:48.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:15:49.276 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:49 smithi194 ceph-mon[44089]: pgmap v691: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 44650/297228 objects misplaced (15.022%); 17 MiB/s, 264 objects/s recovering 2024-04-29T14:15:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:49 smithi040 ceph-mon[39581]: pgmap v691: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 44650/297228 objects misplaced (15.022%); 17 MiB/s, 264 objects/s recovering 2024-04-29T14:15:49.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:49 smithi121 ceph-mon[44533]: pgmap v691: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 44650/297228 objects misplaced (15.022%); 17 MiB/s, 264 objects/s recovering 2024-04-29T14:15:50.235 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:15:50.236 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:15:50.237 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:15:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2050002588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:50.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2050002588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:50.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2050002588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:50.514 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:50.639 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:50.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:50.750 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:50.782 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:50.835 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:50.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:50.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:50.993 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:50.993 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:51.012 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:15:51.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:15:51.042 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:51.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:51.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:51.198 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:51.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:51.305 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: pgmap v692: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 796 op/s; 43927/296142 objects misplaced (14.833%); 17 MiB/s, 268 objects/s recovering 2024-04-29T14:15:51.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.306 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: pgmap v692: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 796 op/s; 43927/296142 objects misplaced (14.833%); 17 MiB/s, 268 objects/s recovering 2024-04-29T14:15:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: pgmap v692: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 9.0 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 796 op/s; 43927/296142 objects misplaced (14.833%); 17 MiB/s, 268 objects/s recovering 2024-04-29T14:15:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:51.393 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:51.446 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:51.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:51.554 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:15:51.585 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:51.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:15:51.751 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:15:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:51.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:15:51.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:15:51.866 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:51.900 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:51.925 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:51.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:52.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:52.066 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:51 smithi040 bash[136013]: be9c642fae33b7f45c7c9925d237a38e51c1ac706f96e7d62404aa26fd03f28b 2024-04-29T14:15:52.066 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:51 smithi040 systemd[1]: Started Ceph osd.2 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:15:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:52.155 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:52.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:52.362 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:52 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[136028]: 2024-04-29T14:15:52.173+0000 7f983a61d7c0 -1 Falling back to public interface 2024-04-29T14:15:52.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:52.415 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:52.442 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:52.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:52.546 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:52.840 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:52.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:52.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:52.995 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:53.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:15:53.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:53.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:53.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:53 smithi194 ceph-mon[44089]: pgmap v693: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 41061/287912 objects misplaced (14.262%); 26 MiB/s, 422 objects/s recovering 2024-04-29T14:15:53.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:53.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:53.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:53 smithi194 ceph-mon[44089]: osdmap e257: 12 total, 11 up, 9 in 2024-04-29T14:15:53.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:53 smithi040 ceph-mon[39581]: pgmap v693: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 41061/287912 objects misplaced (14.262%); 26 MiB/s, 422 objects/s recovering 2024-04-29T14:15:53.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:53.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:53.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:53 smithi040 ceph-mon[39581]: osdmap e257: 12 total, 11 up, 9 in 2024-04-29T14:15:53.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:53.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:53.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:53.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:53 smithi121 ceph-mon[44533]: pgmap v693: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 41061/287912 objects misplaced (14.262%); 26 MiB/s, 422 objects/s recovering 2024-04-29T14:15:53.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:53.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:53.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:53 smithi121 ceph-mon[44533]: osdmap e257: 12 total, 11 up, 9 in 2024-04-29T14:15:53.393 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:53.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:53.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:53.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:15:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:15:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:53.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:15:53.769 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:53.805 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:53.906 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:53.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:54.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:15:54.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:54.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:54.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:54.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2128636042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:54.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:54 smithi194 ceph-mon[44089]: from='client.25222 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:54.274 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:54.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:54.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:54.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2128636042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:54.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:54 smithi121 ceph-mon[44533]: from='client.25222 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:54.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2128636042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:54.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:54 smithi040 ceph-mon[39581]: from='client.25222 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:15:54.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:15:54.481 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:54.517 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:15:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:54.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:15:54.623 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:15:54.656 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:55.076 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:15:55.111 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:15:55.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:55 smithi040 ceph-mon[39581]: pgmap v695: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 41061/287912 objects misplaced (14.262%); 26 MiB/s, 422 objects/s recovering 2024-04-29T14:15:55.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:55 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:55.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:55 smithi040 ceph-mon[39581]: from='client.25222 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:55.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:55 smithi040 ceph-mon[39581]: osdmap e258: 12 total, 11 up, 9 in 2024-04-29T14:15:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:55.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:15:55.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:55.288 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:55.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:55 smithi194 ceph-mon[44089]: pgmap v695: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 41061/287912 objects misplaced (14.262%); 26 MiB/s, 422 objects/s recovering 2024-04-29T14:15:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:55 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:55 smithi194 ceph-mon[44089]: from='client.25222 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:55 smithi194 ceph-mon[44089]: osdmap e258: 12 total, 11 up, 9 in 2024-04-29T14:15:55.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:55.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:55.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:55 smithi121 ceph-mon[44533]: pgmap v695: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.8 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 936 op/s; 41061/287912 objects misplaced (14.262%); 26 MiB/s, 422 objects/s recovering 2024-04-29T14:15:55.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:55 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:15:55.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:55 smithi121 ceph-mon[44533]: from='client.25222 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:15:55.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:55 smithi121 ceph-mon[44533]: osdmap e258: 12 total, 11 up, 9 in 2024-04-29T14:15:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:55.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:55.425 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:55.489 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:15:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:55.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:15:55.795 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:55.830 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:55.937 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:55.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:55.943 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:55 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[136028]: 2024-04-29T14:15:55.627+0000 7f983a61d7c0 -1 osd.2 230 log_to_monitors true 2024-04-29T14:15:56.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:15:56.080 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:56.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:56 smithi194 ceph-mon[44089]: from='osd.2 v1:172.21.15.40:6809/3416690575' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:15:56.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:56 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:15:56.362 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 14:15:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[136028]: 2024-04-29T14:15:56.092+0000 7f982ba75700 -1 osd.2 230 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:15:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:56 smithi040 ceph-mon[39581]: from='osd.2 v1:172.21.15.40:6809/3416690575' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:15:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:56 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:15:56.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:56 smithi121 ceph-mon[44533]: from='osd.2 v1:172.21.15.40:6809/3416690575' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:15:56.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:56 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-29T14:15:56.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:15:57.154 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:15:57.154 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:15:57.154 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:15:57.154 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:15:57.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:57 smithi194 ceph-mon[44089]: pgmap v697: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 878 op/s; 38354/283162 objects misplaced (13.545%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:15:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:57 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-29T14:15:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:57 smithi194 ceph-mon[44089]: osdmap e259: 12 total, 11 up, 9 in 2024-04-29T14:15:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:57 smithi194 ceph-mon[44089]: from='osd.2 v1:172.21.15.40:6809/3416690575' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:15:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:57 smithi194 ceph-mon[44089]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:15:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:15:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:57 smithi040 ceph-mon[39581]: pgmap v697: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 878 op/s; 38354/283162 objects misplaced (13.545%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:15:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:57 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-29T14:15:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:57 smithi040 ceph-mon[39581]: osdmap e259: 12 total, 11 up, 9 in 2024-04-29T14:15:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:57 smithi040 ceph-mon[39581]: from='osd.2 v1:172.21.15.40:6809/3416690575' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:15:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:57 smithi040 ceph-mon[39581]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:15:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:15:57.364 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:15:57.365 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:15:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:57 smithi121 ceph-mon[44533]: pgmap v697: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.6 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 878 op/s; 38354/283162 objects misplaced (13.545%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:15:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:57 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-29T14:15:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:57 smithi121 ceph-mon[44533]: osdmap e259: 12 total, 11 up, 9 in 2024-04-29T14:15:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:57 smithi121 ceph-mon[44533]: from='osd.2 v1:172.21.15.40:6809/3416690575' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:15:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:57 smithi121 ceph-mon[44533]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T14:15:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:15:57.950 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:15:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:57.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:15:58.056 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:15:58.089 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:58.129 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:15:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:58.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:15:58.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:15:58.212 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:58 smithi040 ceph-mon[39581]: osd.2 v1:172.21.15.40:6809/3416690575 boot 2024-04-29T14:15:58.212 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:58 smithi040 ceph-mon[39581]: osdmap e260: 12 total, 12 up, 9 in 2024-04-29T14:15:58.212 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:15:58.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:15:58.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:58.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:15:58.337 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:58 smithi121 ceph-mon[44533]: osd.2 v1:172.21.15.40:6809/3416690575 boot 2024-04-29T14:15:58.337 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:58 smithi121 ceph-mon[44533]: osdmap e260: 12 total, 12 up, 9 in 2024-04-29T14:15:58.337 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:15:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:58.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:15:58.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:15:58.486 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:58.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:15:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:58.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:15:58.531 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:15:58.532 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:15:58.545 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:15:58.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:58 smithi194 ceph-mon[44089]: osd.2 v1:172.21.15.40:6809/3416690575 boot 2024-04-29T14:15:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:58 smithi194 ceph-mon[44089]: osdmap e260: 12 total, 12 up, 9 in 2024-04-29T14:15:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-29T14:15:58.574 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:15:58.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:15:58.622 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:15:58.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:58.652 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:15:58.678 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:58.690 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:15:58.715 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:58.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:58.739 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:15:58.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:15:58.796 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:15:58.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:15:58.826 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:15:58.856 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:15:58.876 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:15:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:15:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:15:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:15:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:15:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:15:58.892 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:58.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:15:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:15:58.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:15:59.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:15:59.077 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:15:59.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:15:59.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:15:59.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: pgmap v700: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 842 op/s; 37655/278530 objects misplaced (13.519%); 23 MiB/s, 375 objects/s recovering 2024-04-29T14:15:59.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: osdmap e261: 12 total, 12 up, 9 in 2024-04-29T14:15:59.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/480982381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:59.106 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.107 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:15:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: pgmap v700: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 842 op/s; 37655/278530 objects misplaced (13.519%); 23 MiB/s, 375 objects/s recovering 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: osdmap e261: 12 total, 12 up, 9 in 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/480982381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:15:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: pgmap v700: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 842 op/s; 37655/278530 objects misplaced (13.519%); 23 MiB/s, 375 objects/s recovering 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: osdmap e261: 12 total, 12 up, 9 in 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/480982381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:15:59.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:15:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:00.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:00.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:00.288 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:00.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.311 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:00 smithi194 ceph-mon[44089]: osdmap e262: 12 total, 12 up, 9 in 2024-04-29T14:16:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:00 smithi040 ceph-mon[39581]: osdmap e262: 12 total, 12 up, 9 in 2024-04-29T14:16:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:00.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:00.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:00.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:00 smithi121 ceph-mon[44533]: osdmap e262: 12 total, 12 up, 9 in 2024-04-29T14:16:00.616 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:16:00.616 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:16:00.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:16:00.616 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:177958 2.c 2:39328c3a:::benchmark_data_smithi194_79000_object13697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.100022+0000", 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 285.520554296, 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.089554231999999998, 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 177958 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.617 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100022+0000", 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100022+0000", 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100024+0000", 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8190000000000001e-06 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100029+0000", 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7890000000000002e-06 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100038+0000", 2024-04-29T14:16:00.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2720000000000003e-06 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100046+0000", 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6359999999999994e-06 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100084+0000", 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7916000000000001e-05 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100173+0000", 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9429e-05 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.100247+0000", 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3813999999999995e-05 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.103657+0000", 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034102529999999998 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189500+0000", 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.085843025000000003 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189540+0000", 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9620999999999999e-05 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189576+0000", 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6658e-05 2024-04-29T14:16:00.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:177960 2.c 2:3d353946:::benchmark_data_smithi194_79000_object13699:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.108689+0000", 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 285.51188742099998, 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.081150963000000007, 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.623 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 177960 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108689+0000", 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108689+0000", 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108692+0000", 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9349999999999999e-06 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108700+0000", 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5210000000000002e-06 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108724+0000", 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3376000000000001e-05 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108735+0000", 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.152e-05 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108803+0000", 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7999999999999999e-05 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.108928+0000", 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012510900000000001 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.109025+0000", 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6453000000000006e-05 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.112726+0000", 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003701513 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189788+0000", 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077061438999999995 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189817+0000", 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8873999999999998e-05 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.189840+0000", 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3223000000000001e-05 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.628 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178139 2.c 2:3c2bb968:::benchmark_data_smithi194_79000_object13878:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.422554+0000", 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 285.19802225000001, 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.084318003000000002, 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178139 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422554+0000", 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422554+0000", 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422556+0000", 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8870000000000001e-06 2024-04-29T14:16:00.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422561+0000", 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8579999999999999e-06 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422573+0000", 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.17e-05 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422580+0000", 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6760000000000007e-06 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422635+0000", 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4447999999999998e-05 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422725+0000", 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0698999999999993e-05 2024-04-29T14:16:00.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.422793+0000", 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7597999999999999e-05 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.426593+0000", 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037997199999999999 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.506762+0000", 2024-04-29T14:16:00.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.080169293000000003 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.506821+0000", 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8925000000000002e-05 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.506872+0000", 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1199000000000002e-05 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178446 2.c 2:3e923c02:::benchmark_data_smithi194_79000_object14185:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:15.913729+0000", 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 284.70684729499999, 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.074333147000000002, 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178446 2024-04-29T14:16:00.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913729+0000", 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913729+0000", 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913731+0000", 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.685e-06 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913736+0000", 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8749999999999999e-06 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913746+0000", 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0078000000000001e-05 2024-04-29T14:16:00.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913753+0000", 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5650000000000004e-06 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913810+0000", 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6873000000000002e-05 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.913926+0000", 2024-04-29T14:16:00.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115635 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.914038+0000", 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011199100000000001 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.917954+0000", 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039161899999999999 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.987989+0000", 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.070034866000000001 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.988027+0000", 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8210999999999997e-05 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:15.988062+0000", 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5178000000000001e-05 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.640 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178566 2.c 2:356259ea:::benchmark_data_smithi194_79000_object14305:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.144836+0000", 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 284.475740293, 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.112011359, 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178566 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.641 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144836+0000", 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144836+0000", 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144840+0000", 2024-04-29T14:16:00.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4419999999999998e-06 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144848+0000", 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5429999999999994e-06 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144867+0000", 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8550999999999999e-05 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144877+0000", 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0023999999999999e-05 2024-04-29T14:16:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.144945+0000", 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8017999999999998e-05 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.145075+0000", 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000130123 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.145161+0000", 2024-04-29T14:16:00.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5828000000000006e-05 2024-04-29T14:16:00.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.148891+0000", 2024-04-29T14:16:00.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037306879999999998 2024-04-29T14:16:00.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256736+0000", 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.1078446 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256794+0000", 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8264000000000003e-05 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256847+0000", 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3278e-05 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178571 2.c 2:31f7de83:::benchmark_data_smithi194_79000_object14310:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.152329+0000", 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 284.46824738200002, 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.104702769, 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.647 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178571 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152329+0000", 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152329+0000", 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152332+0000", 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8559999999999998e-06 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152340+0000", 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1170000000000004e-06 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152357+0000", 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6594000000000001e-05 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152364+0000", 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6720000000000004e-06 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152428+0000", 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3874999999999994e-05 2024-04-29T14:16:00.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152571+0000", 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014308600000000001 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.152690+0000", 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118478 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.157030+0000", 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043403169999999998 2024-04-29T14:16:00.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256935+0000", 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.099905026999999993 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.256989+0000", 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3486000000000003e-05 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.257032+0000", 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3260999999999998e-05 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.652 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178587 2.c 2:31bf17b8:::benchmark_data_smithi194_79000_object14326:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.166595+0000", 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 284.45398150599999, 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.104067166, 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.653 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178587 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166595+0000", 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166595+0000", 2024-04-29T14:16:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166597+0000", 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8050000000000001e-06 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166602+0000", 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2959999999999998e-06 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166613+0000", 2024-04-29T14:16:00.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1117e-05 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166619+0000", 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0129999999999999e-06 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166665+0000", 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5890999999999998e-05 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166759+0000", 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4129999999999995e-05 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.166869+0000", 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109855 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.174687+0000", 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078177899999999998 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270559+0000", 2024-04-29T14:16:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095871737999999998 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270618+0000", 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9011000000000001e-05 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270662+0000", 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4520000000000001e-05 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.658 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:178589 2.c 2:378ec238:::benchmark_data_smithi194_79000_object14328:head [delete] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:11:16.168750+0000", 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 284.45182611600001, 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.102036601, 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 178589 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168750+0000", 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168750+0000", 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168752+0000", 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168757+0000", 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3909999999999997e-06 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168767+0000", 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0197000000000001e-05 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.168775+0000", 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5129999999999999e-06 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.171154+0000", 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023789229999999998 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.171245+0000", 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1543000000000001e-05 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.171341+0000", 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5366999999999997e-05 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.174846+0000", 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035054539999999999 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270710+0000", 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095863956 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270752+0000", 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1834e-05 2024-04-29T14:16:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:11:16.270787+0000", 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4900999999999997e-05 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:239815 2.c 2:3e12e13d:::benchmark_data_smithi194_79000_object75554:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 234.02460197299999, 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077752630000000003, 2024-04-29T14:16:00.664 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 239815 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:16:00.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595977+0000", 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2730000000000001e-06 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595984+0000", 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0559999999999999e-06 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595996+0000", 2024-04-29T14:16:00.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2215000000000001e-05 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596005+0000", 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8249999999999994e-06 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596065+0000", 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0183000000000003e-05 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596172+0000", 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106702 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596280+0000", 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108251 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.605356+0000", 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090762619999999999 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673606+0000", 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068249290000000004 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673668+0000", 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2724000000000005e-05 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673727+0000", 2024-04-29T14:16:00.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8848999999999998e-05 2024-04-29T14:16:00.671 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.671 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.671 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:239816 2.c 2:3d6c3553:::benchmark_data_smithi194_79000_object75555:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 234.02455316300001, 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077792867000000002, 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.672 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 239816 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596024+0000", 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9400000000000004e-07 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596028+0000", 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6440000000000002e-06 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596034+0000", 2024-04-29T14:16:00.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596041+0000", 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8210000000000004e-06 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601153+0000", 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051123899999999996 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601279+0000", 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012538299999999999 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601396+0000", 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117223 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.605374+0000", 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039785719999999997 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673769+0000", 2024-04-29T14:16:00.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068395101999999999 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673795+0000", 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.584e-05 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673816+0000", 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0874e-05 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323258 2.b 2:d0b65c6b:::benchmark_data_smithi194_79000_object158997:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.54736080699999, 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.231651628, 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323258 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4300000000000002e-07 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073219+0000", 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.066e-06 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073224+0000", 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8040000000000001e-06 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073230+0000", 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8089999999999998e-06 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073281+0000", 2024-04-29T14:16:00.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0677000000000003e-05 2024-04-29T14:16:00.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073356+0000", 2024-04-29T14:16:00.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5309999999999996e-05 2024-04-29T14:16:00.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073428+0000", 2024-04-29T14:16:00.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1680000000000005e-05 2024-04-29T14:16:00.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.077404+0000", 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039764600000000002 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304804+0000", 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.227399348 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304837+0000", 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3788999999999998e-05 2024-04-29T14:16:00.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304867+0000", 2024-04-29T14:16:00.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9941999999999999e-05 2024-04-29T14:16:00.685 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.685 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323282 2.b 2:de88de25:::benchmark_data_smithi194_79000_object159021:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.091895+0000", 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.52868175500001, 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21308433399999999, 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.686 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323282 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091895+0000", 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091895+0000", 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091897+0000", 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.84e-06 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091902+0000", 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5790000000000003e-06 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091913+0000", 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.118e-05 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091921+0000", 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8879999999999997e-06 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091990+0000", 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8392999999999994e-05 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.092076+0000", 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6626999999999996e-05 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.092163+0000", 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6722000000000001e-05 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.097159+0000", 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049962909999999999 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304912+0000", 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.207753146 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304944+0000", 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1189000000000002e-05 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304979+0000", 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5478999999999997e-05 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.691 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323296 2.b 2:d44dd007:::benchmark_data_smithi194_79000_object159035:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.099450+0000", 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.52112621200001, 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.205642045, 2024-04-29T14:16:00.692 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323296 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099450+0000", 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099450+0000", 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099453+0000", 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3319999999999999e-06 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099458+0000", 2024-04-29T14:16:00.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7629999999999997e-06 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099471+0000", 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2955999999999999e-05 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099478+0000", 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1239999999999999e-06 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099532+0000", 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3538999999999997e-05 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099613+0000", 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1308000000000001e-05 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099681+0000", 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7446000000000004e-05 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.104798+0000", 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051173939999999999 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.305030+0000", 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.200232206 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.305063+0000", 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2951999999999999e-05 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.305092+0000", 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9025000000000001e-05 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323321 2.b 2:d0365ce8:::benchmark_data_smithi194_79000_object159060:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.124090+0000", 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.49648662000001, 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18673873399999999, 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.698 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323321 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124090+0000", 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124090+0000", 2024-04-29T14:16:00.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124091+0000", 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.23e-07 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124094+0000", 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5250000000000001e-06 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124104+0000", 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0319000000000001e-05 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124112+0000", 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5310000000000004e-06 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.129158+0000", 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050463030000000002 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.129267+0000", 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108687 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.129374+0000", 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106903 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.138413+0000", 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090388220000000002 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310731+0000", 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17231806199999999 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310783+0000", 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2346999999999999e-05 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310829+0000", 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5512000000000001e-05 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323322 2.b 2:dc463b5d:::benchmark_data_smithi194_79000_object159061:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.49644985399999, 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.186836433, 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323322 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-04-29T14:16:00.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124131+0000", 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3950000000000001e-06 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124137+0000", 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2709999999999998e-06 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124143+0000", 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0689999999999998e-06 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.134072+0000", 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0099292919999999993 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.134206+0000", 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013343599999999999 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.134330+0000", 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012425500000000001 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.139426+0000", 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050959380000000004 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310888+0000", 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17146186099999999 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310920+0000", 2024-04-29T14:16:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2499999999999997e-05 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310963+0000", 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2784000000000002e-05 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323359 2.b 2:db19cab3:::benchmark_data_smithi194_79000_object159098:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.162598+0000", 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.45797860799999, 2024-04-29T14:16:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.148498665, 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323359 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162598+0000", 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162598+0000", 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162601+0000", 2024-04-29T14:16:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7760000000000002e-06 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162607+0000", 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7850000000000003e-06 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162621+0000", 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3834e-05 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162630+0000", 2024-04-29T14:16:00.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0499999999999997e-06 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162691+0000", 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0783999999999999e-05 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162805+0000", 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000113819 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162911+0000", 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106321 2024-04-29T14:16:00.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.167417+0000", 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045058579999999997 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311010+0000", 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14359249299999999 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311056+0000", 2024-04-29T14:16:00.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6869999999999997e-05 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311096+0000", 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0074999999999998e-05 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323378 2.b 2:d141cb3f:::benchmark_data_smithi194_79000_object159117:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.177760+0000", 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.44281678499999, 2024-04-29T14:16:00.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.133421761, 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323378 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177760+0000", 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177760+0000", 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177762+0000", 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0980000000000001e-06 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177767+0000", 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9200000000000003e-06 2024-04-29T14:16:00.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177775+0000", 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.704e-06 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177781+0000", 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7969999999999997e-06 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177832+0000", 2024-04-29T14:16:00.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0485999999999997e-05 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177926+0000", 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4827999999999994e-05 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178016+0000", 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9772999999999996e-05 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186701+0000", 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086849490000000008 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311126+0000", 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12442435 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311153+0000", 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7905000000000001e-05 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311181+0000", 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7951000000000001e-05 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323380 2.b 2:d8238550:::benchmark_data_smithi194_79000_object159119:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.178214+0000", 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.44236222399999, 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14135888199999999, 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.721 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323380 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178214+0000", 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178214+0000", 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178216+0000", 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3370000000000001e-06 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178220+0000", 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7350000000000004e-06 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178231+0000", 2024-04-29T14:16:00.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0457e-05 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178238+0000", 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9460000000000001e-06 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.181789+0000", 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003551612 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.181927+0000", 2024-04-29T14:16:00.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013803900000000001 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.182069+0000", 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014142899999999999 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186818+0000", 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004749479 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319502+0000", 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13268347899999999 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319543+0000", 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.176e-05 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319573+0000", 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9609000000000001e-05 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.726 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323388 2.b 2:ddd3457d:::benchmark_data_smithi194_79000_object159127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.186762+0000", 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.43381449099999, 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.132927094, 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323388 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186762+0000", 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186762+0000", 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186765+0000", 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1020000000000001e-06 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186774+0000", 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1130000000000007e-06 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186789+0000", 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5204e-05 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186799+0000", 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8549999999999997e-06 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186850+0000", 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0837000000000001e-05 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186976+0000", 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000125859 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.187100+0000", 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000124072 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.190963+0000", 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038628360000000001 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319614+0000", 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.128651668 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319649+0000", 2024-04-29T14:16:00.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4668000000000001e-05 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319689+0000", 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9879999999999998e-05 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323419 2.b 2:d676c126:::benchmark_data_smithi194_79000_object159158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:16:00.732 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.217619+0000", 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.402957324, 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.102206715, 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323419 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.733 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217619+0000", 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217619+0000", 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:16:00.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217620+0000", 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.372e-06 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217625+0000", 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9520000000000002e-06 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217637+0000", 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1093e-05 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217644+0000", 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2540000000000003e-06 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217698+0000", 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4710000000000003e-05 2024-04-29T14:16:00.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217783+0000", 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4277000000000006e-05 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217852+0000", 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9721999999999994e-05 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:16:00.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.221659+0000", 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038063200000000002 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319730+0000", 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.098070933999999998 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319774+0000", 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4733e-05 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:16:00.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319826+0000", 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1347999999999997e-05 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:16:00.739 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:16:00.740 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:00.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:00.849 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:00.879 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:00.905 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:00.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:01.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:01.048 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:01.279 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:01.287 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:01 smithi040 ceph-mon[39581]: pgmap v703: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 735 op/s; 36901/277276 objects misplaced (13.308%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:16:01.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:01.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:01 smithi194 ceph-mon[44089]: pgmap v703: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 735 op/s; 36901/277276 objects misplaced (13.308%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:16:01.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:01 smithi121 ceph-mon[44533]: pgmap v703: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 735 op/s; 36901/277276 objects misplaced (13.308%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:16:01.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:01.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:16:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:01.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:01.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:01.627 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:01.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:01.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:01.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:01.826 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:02.078 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:02.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:02.132 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:16:02.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:16:02.182 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:02.210 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:02.235 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:02.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:02.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:02.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3071295155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:16:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:02.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3071295155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:16:02.406 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:02.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:02.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:02.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3071295155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:16:02.570 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:02.602 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:02.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:02.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:02.747 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:02.772 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:02.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:02.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:02.939 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:02.954 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 8, 9, 10, 11, 6] out_osds: [1, 2, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-04-29T14:16:02.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-29T14:16:02.981 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:02.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:03.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:03.122 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:03.181 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:03.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:03.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:03.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:03 smithi121 ceph-mon[44533]: pgmap v704: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 34156/269102 objects misplaced (12.693%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:16:03.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:03 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:03.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3071295155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:16:03.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:03 smithi121 ceph-mon[44533]: osdmap e263: 12 total, 12 up, 9 in 2024-04-29T14:16:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:03 smithi040 ceph-mon[39581]: pgmap v704: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 34156/269102 objects misplaced (12.693%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:16:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:03 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3071295155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:16:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:03 smithi040 ceph-mon[39581]: osdmap e263: 12 total, 12 up, 9 in 2024-04-29T14:16:03.378 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:03.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:03.484 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:03.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:03 smithi194 ceph-mon[44089]: pgmap v704: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 34156/269102 objects misplaced (12.693%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:16:03.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:03 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:03.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3071295155' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:16:03.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:03 smithi194 ceph-mon[44089]: osdmap e263: 12 total, 12 up, 9 in 2024-04-29T14:16:03.512 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:03.533 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:03.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:03.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:03.668 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:03.925 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:03.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:03.963 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:04.031 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:04.064 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:04.276 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:04.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:04.381 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:04.411 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:04.438 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:04.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:04.543 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:04.574 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:04.593 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:04.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:04.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:04.731 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:04.751 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:04.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:04.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:05.045 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:05.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:05 smithi040 ceph-mon[39581]: pgmap v706: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 784 op/s; 34156/269102 objects misplaced (12.693%); 23 MiB/s, 370 objects/s recovering 2024-04-29T14:16:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:05 smithi194 ceph-mon[44089]: pgmap v706: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 784 op/s; 34156/269102 objects misplaced (12.693%); 23 MiB/s, 370 objects/s recovering 2024-04-29T14:16:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:05 smithi121 ceph-mon[44533]: pgmap v706: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 784 op/s; 34156/269102 objects misplaced (12.693%); 23 MiB/s, 370 objects/s recovering 2024-04-29T14:16:05.626 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:16:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:16:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:16:05.627 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:16:05.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:16:05.844 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:05.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:05.950 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:05.980 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:06.002 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:06.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:06.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:06.173 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:06.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:06.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:06.402 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:06.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:06.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:06.550 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:06.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:06.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:06.929 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:06.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:06.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:07.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:07.116 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:07.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:07.262 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:07.290 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:07.310 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:07.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:07 smithi121 ceph-mon[44533]: pgmap v707: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 31543/264564 objects misplaced (11.923%); 33 MiB/s, 522 objects/s recovering 2024-04-29T14:16:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:07 smithi121 ceph-mon[44533]: osdmap e264: 12 total, 12 up, 9 in 2024-04-29T14:16:07.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:07.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:07 smithi040 ceph-mon[39581]: pgmap v707: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 31543/264564 objects misplaced (11.923%); 33 MiB/s, 522 objects/s recovering 2024-04-29T14:16:07.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:07.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:07.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:07 smithi040 ceph-mon[39581]: osdmap e264: 12 total, 12 up, 9 in 2024-04-29T14:16:07.474 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:07.495 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:16:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:07.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:07 smithi194 ceph-mon[44089]: pgmap v707: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 8.1 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 927 op/s; 31543/264564 objects misplaced (11.923%); 33 MiB/s, 522 objects/s recovering 2024-04-29T14:16:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:07 smithi194 ceph-mon[44089]: osdmap e264: 12 total, 12 up, 9 in 2024-04-29T14:16:07.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:07.668 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:07.701 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:07.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:07.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:07.838 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:08.889 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:16:08.889 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:16:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:08.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:08.908 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:16:08.939 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:16:08.979 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:16:09.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:09.030 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:16:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:09.065 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:16:09.103 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:16:09.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:16:09.133 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:09.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:09.161 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:16:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:09.208 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:16:09.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:09.238 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:09.267 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:16:09.267 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:16:09.267 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:16:09.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:16:09.294 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:09.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:09.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:09.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:09 smithi040 ceph-mon[39581]: pgmap v709: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 30906/259742 objects misplaced (11.899%); 30 MiB/s, 479 objects/s recovering 2024-04-29T14:16:09.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/249184661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:09.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:09.452 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:09.478 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:09 smithi194 ceph-mon[44089]: pgmap v709: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 30906/259742 objects misplaced (11.899%); 30 MiB/s, 479 objects/s recovering 2024-04-29T14:16:09.478 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/249184661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:09.478 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:09.480 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:09.501 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:09.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:09.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:09.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:09 smithi121 ceph-mon[44533]: pgmap v709: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 30906/259742 objects misplaced (11.899%); 30 MiB/s, 479 objects/s recovering 2024-04-29T14:16:09.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/249184661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:09.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:09.636 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:09.661 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:09.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:09.765 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:09.793 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:09.829 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:09.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:09.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:09.977 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:10.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:10.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:10.110 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:10.140 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:10.159 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:10.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:10.186 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-04-29T14:16:10.263 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:10.291 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:10.315 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:10.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:10.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3096161841' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:16:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:10 smithi121 ceph-mon[44533]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:16:10.421 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:10.446 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3096161841' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:16:10.447 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:10 smithi194 ceph-mon[44089]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:16:10.448 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:10.470 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:10.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:10.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:10.604 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:10.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3096161841' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:16:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:10 smithi040 ceph-mon[39581]: from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:16:10.630 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:10.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:10.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:10.894 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:10.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:10.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:11.060 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:11.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:11.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:11.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:11.265 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:11.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:11.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:11.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:11 smithi040 ceph-mon[39581]: pgmap v710: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 657 op/s; 30203/258572 objects misplaced (11.681%); 20 MiB/s, 324 objects/s recovering 2024-04-29T14:16:11.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:11 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:11.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:11 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:11.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:11 smithi040 ceph-mon[39581]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:16:11.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:11 smithi040 ceph-mon[39581]: osdmap e265: 12 total, 12 up, 9 in 2024-04-29T14:16:11.345 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-29T14:16:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:11 smithi121 ceph-mon[44533]: pgmap v710: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 657 op/s; 30203/258572 objects misplaced (11.681%); 20 MiB/s, 324 objects/s recovering 2024-04-29T14:16:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:11 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:11 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:11 smithi121 ceph-mon[44533]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:16:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:11 smithi121 ceph-mon[44533]: osdmap e265: 12 total, 12 up, 9 in 2024-04-29T14:16:11.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:11 smithi194 ceph-mon[44089]: pgmap v710: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 657 op/s; 30203/258572 objects misplaced (11.681%); 20 MiB/s, 324 objects/s recovering 2024-04-29T14:16:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:11 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:11 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:11 smithi194 ceph-mon[44089]: from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:16:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:11 smithi194 ceph-mon[44089]: osdmap e265: 12 total, 12 up, 9 in 2024-04-29T14:16:11.581 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:11.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:11.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:11.742 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:12.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:12 smithi040 ceph-mon[39581]: pgmap v712: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 27767/250466 objects misplaced (11.086%); 33 MiB/s, 522 objects/s recovering 2024-04-29T14:16:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:12 smithi194 ceph-mon[44089]: pgmap v712: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 27767/250466 objects misplaced (11.086%); 33 MiB/s, 522 objects/s recovering 2024-04-29T14:16:12.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:12 smithi121 ceph-mon[44533]: pgmap v712: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 27767/250466 objects misplaced (11.086%); 33 MiB/s, 522 objects/s recovering 2024-04-29T14:16:12.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:16:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:13.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:13.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:13.536 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:13.560 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:13.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:13.664 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:13.693 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:13.715 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:13.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:13.733 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:16:13.733 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:16:13.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:13.859 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:13.992 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:13.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:14.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:14.141 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:14.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:14.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:14.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:16:15.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:14 smithi040 ceph-mon[39581]: pgmap v713: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 880 op/s; 27767/250466 objects misplaced (11.086%); 19 MiB/s, 308 objects/s recovering 2024-04-29T14:16:15.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1064940719' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:16:15.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:16:15.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:14 smithi194 ceph-mon[44089]: pgmap v713: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 880 op/s; 27767/250466 objects misplaced (11.086%); 19 MiB/s, 308 objects/s recovering 2024-04-29T14:16:15.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1064940719' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:16:15.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:14 smithi121 ceph-mon[44533]: pgmap v713: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.6 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 880 op/s; 27767/250466 objects misplaced (11.086%); 19 MiB/s, 308 objects/s recovering 2024-04-29T14:16:15.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1064940719' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:16:15.465 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:15.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:15.483 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:15.485 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:15.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:15.648 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:15.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:15.679 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:15 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:16:15.679 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1064940719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:16:15.679 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:15 smithi040 ceph-mon[39581]: osdmap e266: 12 total, 12 up, 9 in 2024-04-29T14:16:15.679 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:15 smithi040 ceph-mon[39581]: 3.0 scrub starts 2024-04-29T14:16:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:15.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:15.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:15.819 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:15.885 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:16.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:15 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:16:16.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1064940719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:16:16.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:15 smithi194 ceph-mon[44089]: osdmap e266: 12 total, 12 up, 9 in 2024-04-29T14:16:16.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:15 smithi194 ceph-mon[44089]: 3.0 scrub starts 2024-04-29T14:16:16.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:15 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:16:16.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1064940719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:16:16.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:15 smithi121 ceph-mon[44533]: osdmap e266: 12 total, 12 up, 9 in 2024-04-29T14:16:16.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:15 smithi121 ceph-mon[44533]: 3.0 scrub starts 2024-04-29T14:16:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:16.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:16.621 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:16.651 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:16.674 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:16.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:16.780 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:16.808 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:16.831 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:16.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:16.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:16 smithi040 ceph-mon[39581]: pgmap v715: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 875 op/s; 25304/245728 objects misplaced (10.298%); 29 MiB/s, 463 objects/s recovering 2024-04-29T14:16:16.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:16 smithi040 ceph-mon[39581]: 3.0 scrub ok 2024-04-29T14:16:16.937 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:16.962 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:16 smithi194 ceph-mon[44089]: pgmap v715: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 875 op/s; 25304/245728 objects misplaced (10.298%); 29 MiB/s, 463 objects/s recovering 2024-04-29T14:16:16.962 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:16 smithi194 ceph-mon[44089]: 3.0 scrub ok 2024-04-29T14:16:16.964 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:16.983 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:16.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:17.088 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:17.116 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:17.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:16 smithi121 ceph-mon[44533]: pgmap v715: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 875 op/s; 25304/245728 objects misplaced (10.298%); 29 MiB/s, 463 objects/s recovering 2024-04-29T14:16:17.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:16 smithi121 ceph-mon[44533]: 3.0 scrub ok 2024-04-29T14:16:17.138 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:17.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:17.245 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:17.274 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:17.294 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:17.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:17.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:17.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:16:17.476 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:17.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:17.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:17.609 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:17.637 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:17.659 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:17.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:17.762 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:17.809 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:18.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:18 smithi040 ceph-mon[39581]: pgmap v716: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 24676/241184 objects misplaced (10.231%); 29 MiB/s, 458 objects/s recovering 2024-04-29T14:16:19.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:18 smithi194 ceph-mon[44089]: pgmap v716: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 24676/241184 objects misplaced (10.231%); 29 MiB/s, 458 objects/s recovering 2024-04-29T14:16:19.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:18 smithi121 ceph-mon[44533]: pgmap v716: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 24676/241184 objects misplaced (10.231%); 29 MiB/s, 458 objects/s recovering 2024-04-29T14:16:21.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:20 smithi194 ceph-mon[44089]: pgmap v717: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 24010/240014 objects misplaced (10.004%); 22 MiB/s, 353 objects/s recovering 2024-04-29T14:16:21.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:20 smithi040 ceph-mon[39581]: pgmap v717: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 24010/240014 objects misplaced (10.004%); 22 MiB/s, 353 objects/s recovering 2024-04-29T14:16:21.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:20 smithi121 ceph-mon[44533]: pgmap v717: 17 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 11 active+clean; 7.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 24010/240014 objects misplaced (10.004%); 22 MiB/s, 353 objects/s recovering 2024-04-29T14:16:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:23 smithi194 ceph-mon[44089]: pgmap v718: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 21587/231952 objects misplaced (9.307%); 27 MiB/s, 432 objects/s recovering 2024-04-29T14:16:23.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:23 smithi040 ceph-mon[39581]: pgmap v718: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 21587/231952 objects misplaced (9.307%); 27 MiB/s, 432 objects/s recovering 2024-04-29T14:16:23.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:23 smithi121 ceph-mon[44533]: pgmap v718: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 21587/231952 objects misplaced (9.307%); 27 MiB/s, 432 objects/s recovering 2024-04-29T14:16:25.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:25 smithi040 ceph-mon[39581]: pgmap v719: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 21587/231952 objects misplaced (9.307%); 27 MiB/s, 432 objects/s recovering 2024-04-29T14:16:25.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:25.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:25.148 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:16:25.151 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:25.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:25.182 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:16:25.203 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:16:25.228 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:16:25.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:25.272 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:16:25.298 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:16:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:25 smithi194 ceph-mon[44089]: pgmap v719: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 21587/231952 objects misplaced (9.307%); 27 MiB/s, 432 objects/s recovering 2024-04-29T14:16:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:25.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:25.335 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:16:25.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:25 smithi121 ceph-mon[44533]: pgmap v719: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 7.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 21587/231952 objects misplaced (9.307%); 27 MiB/s, 432 objects/s recovering 2024-04-29T14:16:25.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:25.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:25.366 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:16:25.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:25.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:25.389 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:16:25.413 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:16:25.437 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:16:25.464 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:16:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:16:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:16:25.486 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:25.513 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:25.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:25.636 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:25.664 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:25.687 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:25.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:25.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:25.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:25.843 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:25.893 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:25.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:25.945 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:26.000 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:26.030 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.4 to 1 (10000) 2024-04-29T14:16:26.054 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:26.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osdmap e267: 12 total, 12 up, 9 in 2024-04-29T14:16:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2528364952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1431689794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osdmap e267: 12 total, 12 up, 9 in 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2528364952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1431689794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.325 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:26.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osdmap e267: 12 total, 12 up, 9 in 2024-04-29T14:16:26.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2528364952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:16:26.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:16:26.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1431689794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:26.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:26.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:26.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:26.465 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:26.488 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:26.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:26.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:26.669 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:26.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:26.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:26.776 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:26.802 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:26.829 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:16:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:26.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:26.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:27.245 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:27.245 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:27 smithi040 ceph-mon[39581]: pgmap v721: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 915 op/s; 19240/227416 objects misplaced (8.460%); 27 MiB/s, 436 objects/s recovering 2024-04-29T14:16:27.245 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:27 smithi040 ceph-mon[39581]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:16:27.245 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:27 smithi040 ceph-mon[39581]: osdmap e268: 12 total, 12 up, 9 in 2024-04-29T14:16:27.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:27.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:27 smithi121 ceph-mon[44533]: pgmap v721: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 915 op/s; 19240/227416 objects misplaced (8.460%); 27 MiB/s, 436 objects/s recovering 2024-04-29T14:16:27.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:27 smithi121 ceph-mon[44533]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:16:27.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:27 smithi121 ceph-mon[44533]: osdmap e268: 12 total, 12 up, 9 in 2024-04-29T14:16:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:27 smithi194 ceph-mon[44089]: pgmap v721: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 915 op/s; 19240/227416 objects misplaced (8.460%); 27 MiB/s, 436 objects/s recovering 2024-04-29T14:16:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:27 smithi194 ceph-mon[44089]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:16:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:27 smithi194 ceph-mon[44089]: osdmap e268: 12 total, 12 up, 9 in 2024-04-29T14:16:28.424 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:28.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:28.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-29T14:16:28.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:28.596 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:28.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:28.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:28.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:28.812 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:28.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:28.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:29.008 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:29.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:29.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:29.157 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:29.186 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:29.211 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:29.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:29.316 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:29.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:29 smithi194 ceph-mon[44089]: pgmap v723: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 18700/222728 objects misplaced (8.396%); 30 MiB/s, 476 objects/s recovering 2024-04-29T14:16:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:29 smithi040 ceph-mon[39581]: pgmap v723: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 18700/222728 objects misplaced (8.396%); 30 MiB/s, 476 objects/s recovering 2024-04-29T14:16:29.365 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:29.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:29 smithi121 ceph-mon[44533]: pgmap v723: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s; 18700/222728 objects misplaced (8.396%); 30 MiB/s, 476 objects/s recovering 2024-04-29T14:16:29.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:29.502 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:29.535 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:29.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:29.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:16:29.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:29.687 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:29.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:29.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:29.833 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:29.881 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:30.096 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:16:30.267 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:16:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3232337803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:16:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3232337803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:16:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:30.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:30.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3232337803' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:16:30.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:30.598 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:31.054 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:31.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:31.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:31.195 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:31.274 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:31.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:31 smithi040 ceph-mon[39581]: pgmap v724: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 18048/221624 objects misplaced (8.144%); 20 MiB/s, 325 objects/s recovering 2024-04-29T14:16:31.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:31 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:31.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3232337803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:16:31.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:31 smithi040 ceph-mon[39581]: osdmap e269: 12 total, 12 up, 9 in 2024-04-29T14:16:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:31.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:31.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:31 smithi121 ceph-mon[44533]: pgmap v724: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 18048/221624 objects misplaced (8.144%); 20 MiB/s, 325 objects/s recovering 2024-04-29T14:16:31.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:31 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:31.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3232337803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:16:31.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:31 smithi121 ceph-mon[44533]: osdmap e269: 12 total, 12 up, 9 in 2024-04-29T14:16:31.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:31.435 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:31.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:31.470 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:31.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:31 smithi194 ceph-mon[44089]: pgmap v724: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 645 op/s; 18048/221624 objects misplaced (8.144%); 20 MiB/s, 325 objects/s recovering 2024-04-29T14:16:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:31 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3232337803' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:16:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:31 smithi194 ceph-mon[44089]: osdmap e269: 12 total, 12 up, 9 in 2024-04-29T14:16:31.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:31.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:31.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:31.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:31.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:31.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:32.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:32.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:32.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:32.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:32.272 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:32.286 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:32.311 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:32.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:32.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:32.448 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:32.475 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:32.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:32.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:32.662 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:32.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:32.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:32.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:32.851 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:32.881 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:32.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:32.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:33.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:16:33.071 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:33.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:33 smithi040 ceph-mon[39581]: pgmap v726: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.32k op/s; 15841/213502 objects misplaced (7.420%); 35 MiB/s, 563 objects/s recovering 2024-04-29T14:16:33.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:33 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:33.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:33 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:33.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:33 smithi040 ceph-mon[39581]: osdmap e270: 12 total, 12 up, 9 in 2024-04-29T14:16:33.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:33.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:33.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:33.268 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:33.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:33.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:33.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:33 smithi121 ceph-mon[44533]: pgmap v726: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.32k op/s; 15841/213502 objects misplaced (7.420%); 35 MiB/s, 563 objects/s recovering 2024-04-29T14:16:33.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:33 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:33.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:33 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:33.332 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:33 smithi121 ceph-mon[44533]: osdmap e270: 12 total, 12 up, 9 in 2024-04-29T14:16:33.428 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:33.454 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:33 smithi194 ceph-mon[44089]: pgmap v726: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.32k op/s; 15841/213502 objects misplaced (7.420%); 35 MiB/s, 563 objects/s recovering 2024-04-29T14:16:33.454 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:33 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:33.454 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:33 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:33.454 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:33 smithi194 ceph-mon[44089]: osdmap e270: 12 total, 12 up, 9 in 2024-04-29T14:16:33.456 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:33.477 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:33.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:33.581 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:33.613 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:33.635 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:33.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:33.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:33.973 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:33.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:33.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:34.077 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:34.115 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:34.135 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:34.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:34.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:34.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:34.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:34.268 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:34.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:34.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:34.527 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:34.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:34.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:34.672 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:34.800 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:34.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:34.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:35.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:35.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:35.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:35.179 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:35.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:35 smithi040 ceph-mon[39581]: pgmap v728: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 840 op/s; 15841/213502 objects misplaced (7.420%); 21 MiB/s, 331 objects/s recovering 2024-04-29T14:16:35.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:35 smithi040 ceph-mon[39581]: osdmap e271: 12 total, 12 up, 9 in 2024-04-29T14:16:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:35 smithi121 ceph-mon[44533]: pgmap v728: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 840 op/s; 15841/213502 objects misplaced (7.420%); 21 MiB/s, 331 objects/s recovering 2024-04-29T14:16:35.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:35 smithi121 ceph-mon[44533]: osdmap e271: 12 total, 12 up, 9 in 2024-04-29T14:16:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:35 smithi194 ceph-mon[44089]: pgmap v728: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 840 op/s; 15841/213502 objects misplaced (7.420%); 21 MiB/s, 331 objects/s recovering 2024-04-29T14:16:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:35 smithi194 ceph-mon[44089]: osdmap e271: 12 total, 12 up, 9 in 2024-04-29T14:16:35.937 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:35.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:36.040 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:36.072 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:36.093 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:36.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:36.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:36.227 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:36.250 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:36.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:36.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/693538870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/693538870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.397 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:36.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:16:36.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:16:36.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:16:36.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:16:36.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:16:36.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:16:36.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:16:36.424 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:16:36.425 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:16:36.425 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:16:36.425 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:16:36.425 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:16:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/693538870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:36.647 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:36.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:36.881 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:36.909 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:36.929 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:36.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:37.033 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:37.060 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:37.081 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:37.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.5 to 1 (10000) 2024-04-29T14:16:37.187 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:37.213 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:37.213 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:37 smithi194 ceph-mon[44089]: pgmap v730: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 13766/208854 objects misplaced (6.591%); 33 MiB/s, 523 objects/s recovering 2024-04-29T14:16:37.213 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2931729000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:16:37.215 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:37.236 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:37.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:37.242 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:16:37.243 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:16:37.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:37.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:37.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:37 smithi040 ceph-mon[39581]: pgmap v730: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 13766/208854 objects misplaced (6.591%); 33 MiB/s, 523 objects/s recovering 2024-04-29T14:16:37.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2931729000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:16:37.374 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:37.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:37.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:37 smithi121 ceph-mon[44533]: pgmap v730: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 13766/208854 objects misplaced (6.591%); 33 MiB/s, 523 objects/s recovering 2024-04-29T14:16:37.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2931729000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:16:37.503 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:37.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:37.608 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:37.637 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:37.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:37.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:37.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:37.819 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:37.861 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:37.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:37.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:38.005 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:38.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:16:38.027 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:38.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:38.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:38.164 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:38.186 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:38.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:38.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:38.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2931729000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:16:38.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:38 smithi194 ceph-mon[44089]: osdmap e272: 12 total, 12 up, 9 in 2024-04-29T14:16:38.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:16:38.331 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2931729000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:16:38.331 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:38 smithi040 ceph-mon[39581]: osdmap e272: 12 total, 12 up, 9 in 2024-04-29T14:16:38.331 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:16:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:38.543 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:38.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:38.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2931729000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:16:38.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:38 smithi121 ceph-mon[44533]: osdmap e272: 12 total, 12 up, 9 in 2024-04-29T14:16:38.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:16:38.647 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:38.702 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:38.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:38.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:38.858 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:38.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:38.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:39.037 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:39.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:39.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:39.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:16:39.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:39.285 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:39.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:39.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:39.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:39.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-29T14:16:39.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:39.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:39 smithi040 ceph-mon[39581]: pgmap v732: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 775 op/s; 13208/204192 objects misplaced (6.468%); 21 MiB/s, 332 objects/s recovering 2024-04-29T14:16:39.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:16:39.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:16:39.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:16:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:39 smithi194 ceph-mon[44089]: pgmap v732: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 775 op/s; 13208/204192 objects misplaced (6.468%); 21 MiB/s, 332 objects/s recovering 2024-04-29T14:16:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:16:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:16:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:16:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:39 smithi121 ceph-mon[44533]: pgmap v732: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 775 op/s; 13208/204192 objects misplaced (6.468%); 21 MiB/s, 332 objects/s recovering 2024-04-29T14:16:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:16:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:16:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:16:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:39.844 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:39.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:39.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:39.949 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:39.982 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:40.004 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:40.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:40.108 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:40.138 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:40.162 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:40.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:40.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:40.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:40 smithi194 ceph-mon[44089]: pgmap v733: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 13208/204192 objects misplaced (6.468%); 19 MiB/s, 306 objects/s recovering 2024-04-29T14:16:40.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:40 smithi194 ceph-mon[44089]: pgmap v734: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 934 op/s; 13208/204192 objects misplaced (6.468%); 25 MiB/s, 400 objects/s recovering 2024-04-29T14:16:40.337 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:40 smithi040 ceph-mon[39581]: pgmap v733: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 13208/204192 objects misplaced (6.468%); 19 MiB/s, 306 objects/s recovering 2024-04-29T14:16:40.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:40 smithi040 ceph-mon[39581]: pgmap v734: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 934 op/s; 13208/204192 objects misplaced (6.468%); 25 MiB/s, 400 objects/s recovering 2024-04-29T14:16:40.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:40.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:40.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:40.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:40 smithi121 ceph-mon[44533]: pgmap v733: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 13208/204192 objects misplaced (6.468%); 19 MiB/s, 306 objects/s recovering 2024-04-29T14:16:40.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:40 smithi121 ceph-mon[44533]: pgmap v734: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.2 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 934 op/s; 13208/204192 objects misplaced (6.468%); 25 MiB/s, 400 objects/s recovering 2024-04-29T14:16:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:40.602 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:40.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:40.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:40.740 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:40.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:40.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:40.903 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:41.254 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:41.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:41.361 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:41.390 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:41.409 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:41.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:41.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:41.580 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:41.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:41.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:41.704 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:41.707 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:41.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:41.760 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:41.783 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:41.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:41.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:41.915 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:41.939 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:41.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:42.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:42.121 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:42.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:42.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:42.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:42.324 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:42 smithi040 ceph-mon[39581]: pgmap v735: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 11200/199700 objects misplaced (5.608%); 30 MiB/s, 479 objects/s recovering 2024-04-29T14:16:42.369 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:42.371 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:42 smithi121 ceph-mon[44533]: pgmap v735: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 11200/199700 objects misplaced (5.608%); 30 MiB/s, 479 objects/s recovering 2024-04-29T14:16:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:42.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:42.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:42.525 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:42 smithi194 ceph-mon[44089]: pgmap v735: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.1 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 11200/199700 objects misplaced (5.608%); 30 MiB/s, 479 objects/s recovering 2024-04-29T14:16:42.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:42.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:42.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:42.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:42.787 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:42.811 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:42.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:42.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:42.954 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:42.964 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:16:42.964 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:16:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:43 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:43.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:43 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:43.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:43 smithi194 ceph-mon[44089]: osdmap e273: 12 total, 12 up, 9 in 2024-04-29T14:16:43.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:43 smithi194 ceph-mon[44089]: pgmap v737: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 819 op/s; 10671/195196 objects misplaced (5.467%); 23 MiB/s, 363 objects/s recovering 2024-04-29T14:16:43.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:43 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:43.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:43 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:43.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:43 smithi040 ceph-mon[39581]: osdmap e273: 12 total, 12 up, 9 in 2024-04-29T14:16:43.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:43 smithi040 ceph-mon[39581]: pgmap v737: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 819 op/s; 10671/195196 objects misplaced (5.467%); 23 MiB/s, 363 objects/s recovering 2024-04-29T14:16:43.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:43 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:43.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:43 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:43.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:43 smithi121 ceph-mon[44533]: osdmap e273: 12 total, 12 up, 9 in 2024-04-29T14:16:43.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:43 smithi121 ceph-mon[44533]: pgmap v737: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 6.0 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 819 op/s; 10671/195196 objects misplaced (5.467%); 23 MiB/s, 363 objects/s recovering 2024-04-29T14:16:44.108 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:44.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:44.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:16:44.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:44.294 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:44.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:44.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:44.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:44.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:44 smithi040 ceph-mon[39581]: osdmap e274: 12 total, 12 up, 9 in 2024-04-29T14:16:44.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:44.508 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:44.508 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:44.508 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:44 smithi121 ceph-mon[44533]: osdmap e274: 12 total, 12 up, 9 in 2024-04-29T14:16:44.515 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:44.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:44.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:44 smithi194 ceph-mon[44089]: osdmap e274: 12 total, 12 up, 9 in 2024-04-29T14:16:44.646 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:44.677 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:44.699 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:44.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:44.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:44.833 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:44.854 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:44.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:44.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:45.023 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:45.079 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:45.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:45.185 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:16:45.215 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:45.235 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:16:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:45.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:16:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:16:45.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:45.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:45 smithi040 ceph-mon[39581]: pgmap v739: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.9 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 848 op/s; 10086/194006 objects misplaced (5.199%); 26 MiB/s, 419 objects/s recovering 2024-04-29T14:16:45.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3430385173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:16:45.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:45 smithi121 ceph-mon[44533]: pgmap v739: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.9 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 848 op/s; 10086/194006 objects misplaced (5.199%); 26 MiB/s, 419 objects/s recovering 2024-04-29T14:16:45.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3430385173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:16:45.411 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:45.456 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:45.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:45 smithi194 ceph-mon[44089]: pgmap v739: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.9 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 848 op/s; 10086/194006 objects misplaced (5.199%); 26 MiB/s, 419 objects/s recovering 2024-04-29T14:16:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3430385173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:16:45.571 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:45.598 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:45.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:45.728 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:45.758 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:45.780 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:45.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:45.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:45.934 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:45.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:45.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:46.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:46.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:46.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:46.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:46.310 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:46.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:46.542 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:46 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3430385173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:16:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:46 smithi194 ceph-mon[44089]: osdmap e275: 12 total, 12 up, 9 in 2024-04-29T14:16:46.569 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:46.591 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:46.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:46.608 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:16:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:46 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3430385173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:16:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:46 smithi040 ceph-mon[39581]: osdmap e275: 12 total, 12 up, 9 in 2024-04-29T14:16:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:46 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3430385173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:16:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:46 smithi121 ceph-mon[44533]: osdmap e275: 12 total, 12 up, 9 in 2024-04-29T14:16:46.634 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:16:46.658 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:16:46.685 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:16:46.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:46.716 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:16:46.742 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:16:46.775 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:16:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:46.810 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:16:46.822 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:46.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:46.839 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:16:46.867 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:16:46.895 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:16:46.923 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:16:46.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:46.926 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:16:46.927 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:16:46.954 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:46.979 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:46.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:47.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:47.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:47.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:47.237 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:47.290 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:47.318 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:47.341 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:47.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:47.445 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:47.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1209571811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:47.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:47 smithi040 ceph-mon[39581]: pgmap v741: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 8208/186014 objects misplaced (4.413%); 25 MiB/s, 405 objects/s recovering 2024-04-29T14:16:47.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1209571811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:47 smithi194 ceph-mon[44089]: pgmap v741: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 8208/186014 objects misplaced (4.413%); 25 MiB/s, 405 objects/s recovering 2024-04-29T14:16:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1209571811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:16:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:47 smithi121 ceph-mon[44533]: pgmap v741: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 8208/186014 objects misplaced (4.413%); 25 MiB/s, 405 objects/s recovering 2024-04-29T14:16:47.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:47.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:16:48.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:16:48.939 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:48.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:49.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:49.104 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:49.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:49.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:49.236 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:49.264 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.7 to 1 (10000) 2024-04-29T14:16:49.281 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:49.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:49 smithi121 ceph-mon[44533]: pgmap v742: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 8208/186014 objects misplaced (4.413%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:16:49.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/906461079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:16:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:49 smithi194 ceph-mon[44089]: pgmap v742: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 8208/186014 objects misplaced (4.413%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:16:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/906461079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:16:49.588 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:49.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:49 smithi040 ceph-mon[39581]: pgmap v742: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.7 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 715 op/s; 8208/186014 objects misplaced (4.413%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:16:49.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/906461079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:16:50.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-29T14:16:50.462 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:50.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:16:50.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:50.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:50.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/906461079' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:16:50.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:50 smithi040 ceph-mon[39581]: osdmap e276: 12 total, 12 up, 9 in 2024-04-29T14:16:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/906461079' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:16:50.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:50 smithi194 ceph-mon[44089]: osdmap e276: 12 total, 12 up, 9 in 2024-04-29T14:16:50.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:50.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:50.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:50.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/906461079' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:16:50.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:50 smithi121 ceph-mon[44533]: osdmap e276: 12 total, 12 up, 9 in 2024-04-29T14:16:50.616 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:50.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:50.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:50.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:16:50.767 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:50.802 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:50.823 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:50.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:50.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:51.022 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:51.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:51.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:51.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:51.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:51.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:51.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:16:51.426 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:51.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:16:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:51.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:16:51.554 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:146908 2.5 2:a25b6840:::benchmark_data_smithi194_79000_object146907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:36.859210+0000", 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.69907804399998, 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22006045599999999, 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.555 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 146908 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.859210+0000", 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.859210+0000", 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.859212+0000", 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9400000000000001e-06 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.859269+0000", 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7535999999999999e-05 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.859279+0000", 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0142e-05 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.859289+0000", 2024-04-29T14:16:51.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4599999999999992e-06 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.863400+0000", 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004111076 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.863543+0000", 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014269400000000001 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.863676+0000", 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000133048 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.871565+0000", 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0078889800000000003 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.079145+0000", 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.207580493 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.079209+0000", 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4024999999999998e-05 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.079270+0000", 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1062000000000006e-05 2024-04-29T14:16:51.561 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:146935 2.5 2:abc9a5f2:::benchmark_data_smithi194_79000_object146934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:36.879890+0000", 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.678397856, 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20311238700000001, 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.562 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 146935 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.879890+0000", 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.879890+0000", 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.879892+0000", 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4119999999999999e-06 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.879931+0000", 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8278000000000002e-05 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.879947+0000", 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6473e-05 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.879957+0000", 2024-04-29T14:16:51.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9680000000000001e-06 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.880026+0000", 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8800999999999994e-05 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.880195+0000", 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016867299999999999 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.880368+0000", 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017335000000000001 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.883987+0000", 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003618806 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.082940+0000", 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.198953203 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.082975+0000", 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4820000000000002e-05 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.083002+0000", 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7603e-05 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.568 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:146981 2.5 2:a8219c86:::benchmark_data_smithi194_79000_object146980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:36.914883+0000", 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.64340500499998, 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.173343423, 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.569 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 146981 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.914883+0000", 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.914883+0000", 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.914884+0000", 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3710000000000001e-06 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.914969+0000", 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4480000000000004e-05 2024-04-29T14:16:51.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.914978+0000", 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1600000000000004e-06 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.914983+0000", 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3580000000000002e-06 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.915039+0000", 2024-04-29T14:16:51.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6131000000000002e-05 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.915180+0000", 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014057800000000001 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.915340+0000", 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016037500000000001 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.918677+0000", 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0033367169999999999 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088140+0000", 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.16946312199999999 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088187+0000", 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7023e-05 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088226+0000", 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9107999999999999e-05 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.575 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147026 2.5 2:a3b80544:::benchmark_data_smithi194_79000_object147025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:36.950477+0000", 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.60781077799999, 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13785134399999999, 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147026 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.576 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950477+0000", 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950477+0000", 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950480+0000", 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6170000000000001e-06 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950554+0000", 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3934000000000001e-05 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950566+0000", 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2619999999999999e-05 2024-04-29T14:16:51.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950576+0000", 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4709999999999996e-06 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950639+0000", 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3183999999999997e-05 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950778+0000", 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013933399999999999 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.950912+0000", 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000133786 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.955236+0000", 2024-04-29T14:16:51.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043236450000000001 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088267+0000", 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13303183399999999 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088302+0000", 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4502000000000003e-05 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088328+0000", 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6417000000000001e-05 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147045 2.5 2:a8780b93:::benchmark_data_smithi194_79000_object147044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:36.964810+0000", 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.59347818200001, 2024-04-29T14:16:51.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.12361185600000001, 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147045 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.964810+0000", 2024-04-29T14:16:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.964810+0000", 2024-04-29T14:16:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.964812+0000", 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.508e-06 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.964858+0000", 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5655999999999998e-05 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.964870+0000", 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2174999999999999e-05 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.964877+0000", 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1770000000000001e-06 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.964927+0000", 2024-04-29T14:16:51.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9898000000000002e-05 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.965014+0000", 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6553999999999998e-05 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.965100+0000", 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6776999999999999e-05 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.968257+0000", 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0031564980000000002 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088358+0000", 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.120101344 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088387+0000", 2024-04-29T14:16:51.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8915999999999999e-05 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.088421+0000", 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4353000000000001e-05 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.589 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147056 2.5 2:a36b2670:::benchmark_data_smithi194_79000_object147055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:36.970041+0000", 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.58824636499997, 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.120807461, 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147056 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.590 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970041+0000", 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970041+0000", 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970043+0000", 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.04e-06 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970089+0000", 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5781000000000001e-05 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970098+0000", 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1249999999999999e-06 2024-04-29T14:16:51.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970104+0000", 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0549999999999996e-06 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970149+0000", 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4935000000000003e-05 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970239+0000", 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9531999999999996e-05 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.970333+0000", 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4085000000000005e-05 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:36.973582+0000", 2024-04-29T14:16:51.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032486949999999998 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.090752+0000", 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117170702 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.090807+0000", 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4274999999999998e-05 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.090849+0000", 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2236e-05 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147606 2.5 2:a8318952:::benchmark_data_smithi194_79000_object147605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.402518+0000", 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.15576953200002, 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21475043599999999, 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.596 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147606 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402518+0000", 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402518+0000", 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402520+0000", 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0190000000000001e-06 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402549+0000", 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9155000000000001e-05 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402566+0000", 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6164999999999999e-05 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402574+0000", 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4290000000000001e-06 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402633+0000", 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9240000000000002e-05 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402809+0000", 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000175296 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.402924+0000", 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115547 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.407023+0000", 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004099035 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.617201+0000", 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21017813699999999 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.617239+0000", 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7425000000000002e-05 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.617269+0000", 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9988000000000002e-05 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147616 2.5 2:ad7a2cfd:::benchmark_data_smithi194_79000_object147615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.409081+0000", 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.149206419, 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212331093, 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147616 2024-04-29T14:16:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409081+0000", 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409081+0000", 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409084+0000", 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3590000000000002e-06 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409122+0000", 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8519999999999997e-05 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409136+0000", 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4059e-05 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409144+0000", 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6660000000000008e-06 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409202+0000", 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8140000000000002e-05 2024-04-29T14:16:51.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409321+0000", 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118878 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.409447+0000", 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.413568+0000", 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041213919999999998 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621352+0000", 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20778339000000001 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621387+0000", 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4870999999999999e-05 2024-04-29T14:16:51.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621412+0000", 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5817999999999999e-05 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147624 2.5 2:a9d0daee:::benchmark_data_smithi194_79000_object147623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.417869+0000", 2024-04-29T14:16:51.609 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.14041893000001, 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203639717, 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147624 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.417869+0000", 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.417869+0000", 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.417871+0000", 2024-04-29T14:16:51.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1880000000000001e-06 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.417899+0000", 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8144999999999999e-05 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.417911+0000", 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1569000000000001e-05 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.417917+0000", 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6760000000000001e-06 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.417971+0000", 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3214999999999998e-05 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.418101+0000", 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.418248+0000", 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000147644 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.421996+0000", 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037474850000000001 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621455+0000", 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19945977300000001 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621486+0000", 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0592000000000001e-05 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621509+0000", 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2561999999999998e-05 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147632 2.5 2:a41ce634:::benchmark_data_smithi194_79000_object147631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.422308+0000", 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.135979857, 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19928252900000001, 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.616 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147632 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422308+0000", 2024-04-29T14:16:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422308+0000", 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422310+0000", 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8020000000000001e-06 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422370+0000", 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0565e-05 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422381+0000", 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0874e-05 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422390+0000", 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8100000000000004e-06 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422443+0000", 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2630000000000003e-05 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422566+0000", 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012385799999999999 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422763+0000", 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019660899999999999 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.429959+0000", 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0071959329999999998 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621540+0000", 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19158120000000001 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621569+0000", 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8670000000000002e-05 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.621590+0000", 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1577999999999999e-05 2024-04-29T14:16:51.622 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147633 2.5 2:a773283c:::benchmark_data_smithi194_79000_object147632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.422409+0000", 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.13587868500002, 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20423303800000001, 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.623 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147633 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422409+0000", 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422409+0000", 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422410+0000", 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5199999999999995e-07 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422445+0000", 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5485999999999999e-05 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422454+0000", 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6030000000000005e-06 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.422461+0000", 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1189999999999999e-06 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.426734+0000", 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042729380000000004 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.426870+0000", 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000135482 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.426977+0000", 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107192 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.430086+0000", 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0031093760000000001 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.626535+0000", 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19644846099999999 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.626592+0000", 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7160000000000002e-05 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.626642+0000", 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0368999999999998e-05 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147646 2.5 2:a5e97e9f:::benchmark_data_smithi194_79000_object147645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.629 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.431960+0000", 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.12632816000001, 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19482103200000001, 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147646 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.431960+0000", 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.431960+0000", 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.431962+0000", 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1459999999999999e-06 2024-04-29T14:16:51.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.432015+0000", 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3046e-05 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.432028+0000", 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3376e-05 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.432037+0000", 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5960000000000004e-06 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.432097+0000", 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0259e-05 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.432246+0000", 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014916800000000001 2024-04-29T14:16:51.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.432368+0000", 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000122247 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.436852+0000", 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00448359 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.626682+0000", 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18983007399999999 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.626733+0000", 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0893999999999998e-05 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.626781+0000", 2024-04-29T14:16:51.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7636000000000003e-05 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147681 2.5 2:a17bca09:::benchmark_data_smithi194_79000_object147680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.460201+0000", 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.09808647400001, 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.1750391, 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.636 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147681 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460201+0000", 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460201+0000", 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460204+0000", 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.385e-06 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460255+0000", 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1032999999999997e-05 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460275+0000", 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9958000000000001e-05 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460284+0000", 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7939999999999997e-06 2024-04-29T14:16:51.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460346+0000", 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1705999999999995e-05 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460496+0000", 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000150128 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.460658+0000", 2024-04-29T14:16:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000161741 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.464389+0000", 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037307439999999998 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.635107+0000", 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.17071783099999999 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.635186+0000", 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9179999999999997e-05 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.635240+0000", 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4599999999999999e-05 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:147733 2.5 2:ac993362:::benchmark_data_smithi194_79000_object147732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:37.503443+0000", 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 374.05484501000001, 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13195767899999999, 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 147733 2024-04-29T14:16:51.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503443+0000", 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503443+0000", 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503445+0000", 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6189999999999998e-06 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503482+0000", 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6270999999999999e-05 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503499+0000", 2024-04-29T14:16:51.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7621e-05 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503509+0000", 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6579999999999997e-06 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503574+0000", 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5304999999999999e-05 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503712+0000", 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000137995 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.503875+0000", 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000162337 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.507637+0000", 2024-04-29T14:16:51.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037624469999999999 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.635302+0000", 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.127665214 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.635358+0000", 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6166999999999999e-05 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:37.635400+0000", 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2045000000000001e-05 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:166399 2.14 2:2faf6f82:::benchmark_data_smithi194_79000_object2138:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e141)", 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:57.950124+0000", 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 353.60816394599999, 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10881165299999999, 2024-04-29T14:16:51.649 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 166399 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.950124+0000", 2024-04-29T14:16:51.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.950124+0000", 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.950128+0000", 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7850000000000002e-06 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.950138+0000", 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0807e-05 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.950157+0000", 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8471999999999999e-05 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.950171+0000", 2024-04-29T14:16:51.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3767999999999999e-05 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951674+0000", 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0015037679999999999 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951686+0000", 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1731e-05 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.023604+0000", 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.071918067000000002 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.023700+0000", 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6149000000000004e-05 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1,6,9", 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.023798+0000", 2024-04-29T14:16:51.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7560999999999996e-05 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.030423+0000", 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0066252339999999998 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.030607+0000", 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018425900000000001 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.031821+0000", 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012136219999999999 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.058764+0000", 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026943405 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.058814+0000", 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9716999999999998e-05 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.058935+0000", 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121308 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:166428 2.14 2:2aae1cad:::benchmark_data_smithi194_79000_object2167:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e141)", 2024-04-29T14:16:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:10:57.951360+0000", 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 353.606927429, 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839981899999999, 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 166428 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951360+0000", 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951360+0000", 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951361+0000", 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8999999999999996e-07 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951363+0000", 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4679999999999999e-06 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951369+0000", 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1540000000000001e-06 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951373+0000", 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7709999999999997e-06 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951785+0000", 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00041128200000000001 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:57.951791+0000", 2024-04-29T14:16:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3300000000000004e-06 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.028817+0000", 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.077025660999999995 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.028927+0000", 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000110079 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1,6,9", 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.029018+0000", 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0962000000000001e-05 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.030724+0000", 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001706431 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.030822+0000", 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7654999999999999e-05 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.033912+0000", 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030903509999999999 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.059581+0000", 2024-04-29T14:16:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025668691 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.059633+0000", 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1752000000000002e-05 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:10:58.059760+0000", 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127642 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:173561 2.15 2:ac0b3f24:::benchmark_data_smithi194_79000_object9300:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:08.059051+0000", 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 343.49923728499999, 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.168482454, 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 173561 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059051+0000", 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059051+0000", 2024-04-29T14:16:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059052+0000", 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.967e-06 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059058+0000", 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2519999999999999e-06 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059068+0000", 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0131e-05 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059075+0000", 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4370000000000001e-06 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059362+0000", 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028660099999999999 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059499+0000", 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013760500000000001 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1,4,5", 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.059928+0000", 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00042827000000000002 2024-04-29T14:16:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.062989+0000", 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030612489999999998 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.063318+0000", 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032858699999999998 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.065361+0000", 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020432559999999998 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.227319+0000", 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.161957815 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.227384+0000", 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5190000000000004e-05 2024-04-29T14:16:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.227533+0000", 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014909400000000001 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:173574 2.15 2:af6f1785:::benchmark_data_smithi194_79000_object9313:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:08.074964+0000", 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 343.48332374199998, 2024-04-29T14:16:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.152759957, 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 173574 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.074964+0000", 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.074964+0000", 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.074967+0000", 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6680000000000001e-06 2024-04-29T14:16:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.074973+0000", 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9109999999999998e-06 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.074986+0000", 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3229e-05 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.074995+0000", 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8920000000000006e-06 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.075059+0000", 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4527000000000001e-05 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.075174+0000", 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011448800000000001 2024-04-29T14:16:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1,4,5", 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.075530+0000", 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035575899999999998 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.077704+0000", 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021741019999999998 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.078655+0000", 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00095142700000000002 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.079765+0000", 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001110401 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.227591+0000", 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.147825228 2024-04-29T14:16:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.227644+0000", 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3718999999999998e-05 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:08.227724+0000", 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9605999999999996e-05 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.24727.0:180588 2.14 2:2cc7dc90:::benchmark_data_smithi194_79000_object16327:head [delete] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:11:18.660965+0000", 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 332.89732252599998, 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11489405, 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.24727", 2024-04-29T14:16:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 180588 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.660965+0000", 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.660965+0000", 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.660967+0000", 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0640000000000001e-06 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.660974+0000", 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4829999999999996e-06 2024-04-29T14:16:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.717690+0000", 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.056716562999999998 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.717704+0000", 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3852000000000001e-05 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.718236+0000", 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053160200000000005 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.718377+0000", 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014093299999999999 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1,6,9", 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.718785+0000", 2024-04-29T14:16:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040866399999999999 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.723985+0000", 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0051995740000000002 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.723999+0000", 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3944999999999999e-05 2024-04-29T14:16:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.724032+0000", 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3207000000000002e-05 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.775641+0000", 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051609109 2024-04-29T14:16:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.775728+0000", 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6446999999999994e-05 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:11:18.775859+0000", 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000131607 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:164487 3.b 3:dc72fc9d:::benchmark_data_smithi194_88551_object5544:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e254)", 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:15:40.054845+0000", 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 71.503442536999998, 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21303364699999999, 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:16:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164487 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054845+0000", 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054845+0000", 2024-04-29T14:16:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054848+0000", 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7709999999999998e-06 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054870+0000", 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2447999999999999e-05 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054884+0000", 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3417999999999999e-05 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054895+0000", 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1171e-05 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.055012+0000", 2024-04-29T14:16:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117259 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.055026+0000", 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3413e-05 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.265999+0000", 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21097299999999999 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:16:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.267879+0000", 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018801670000000001 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:16:51.694 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:16:51.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:51.695 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:51 smithi121 ceph-mon[44533]: pgmap v744: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 926 op/s; 6378/181466 objects misplaced (3.515%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:16:51.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:51 smithi194 ceph-mon[44089]: pgmap v744: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 926 op/s; 6378/181466 objects misplaced (3.515%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:16:51.696 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:51 smithi040 ceph-mon[39581]: pgmap v744: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 926 op/s; 6378/181466 objects misplaced (3.515%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:16:51.746 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:51.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:51.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:51.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:51.932 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:52.282 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:52.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:52.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:52.419 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:52.442 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:52.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:52.546 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:52.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:16:53.035 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:53.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:53.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:53.188 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:53.273 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:16:53.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:53.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:53.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:53.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:16:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:16:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:53.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:16:54.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:16:54.068 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:16:54.073 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:54.095 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:54 smithi040 ceph-mon[39581]: pgmap v745: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.4 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 5921/176818 objects misplaced (3.349%); 27 MiB/s, 431 objects/s recovering 2024-04-29T14:16:54.095 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1433400224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:16:54.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:54 smithi194 ceph-mon[44089]: pgmap v745: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.4 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 5921/176818 objects misplaced (3.349%); 27 MiB/s, 431 objects/s recovering 2024-04-29T14:16:54.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1433400224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:16:54.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:54 smithi121 ceph-mon[44533]: pgmap v745: 17 pgs: 5 active+remapped+backfilling, 12 active+clean; 5.4 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 5921/176818 objects misplaced (3.349%); 27 MiB/s, 431 objects/s recovering 2024-04-29T14:16:54.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1433400224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:16:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:54.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:16:55.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:55.067 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:55.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:55.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:55.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:55.249 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:55.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:55.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:55.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:55 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:55.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1433400224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:16:55.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:55 smithi194 ceph-mon[44089]: osdmap e277: 12 total, 12 up, 9 in 2024-04-29T14:16:55.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:55 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:55.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1433400224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:16:55.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:55 smithi040 ceph-mon[39581]: osdmap e277: 12 total, 12 up, 9 in 2024-04-29T14:16:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:55 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:16:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1433400224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:16:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:55 smithi121 ceph-mon[44533]: osdmap e277: 12 total, 12 up, 9 in 2024-04-29T14:16:55.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:16:55.422 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:55.450 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:16:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:55.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:16:55.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:55.603 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:55.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:55.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:55.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:55.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:55.805 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:55.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:55.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:55.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:16:55.983 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:56.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:56 smithi121 ceph-mon[44533]: pgmap v747: 17 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 12 active+clean; 5.4 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 640 op/s; 5440/175768 objects misplaced (3.095%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:16:56.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:56 smithi121 ceph-mon[44533]: 3.9 scrub starts 2024-04-29T14:16:56.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:56 smithi121 ceph-mon[44533]: osdmap e278: 12 total, 12 up, 9 in 2024-04-29T14:16:56.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:56 smithi121 ceph-mon[44533]: 3.9 scrub ok 2024-04-29T14:16:56.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:56 smithi040 ceph-mon[39581]: pgmap v747: 17 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 12 active+clean; 5.4 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 640 op/s; 5440/175768 objects misplaced (3.095%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:16:56.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:56 smithi040 ceph-mon[39581]: 3.9 scrub starts 2024-04-29T14:16:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:56 smithi040 ceph-mon[39581]: osdmap e278: 12 total, 12 up, 9 in 2024-04-29T14:16:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:56 smithi040 ceph-mon[39581]: 3.9 scrub ok 2024-04-29T14:16:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:16:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:56.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:16:56.537 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:16:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:56 smithi194 ceph-mon[44089]: pgmap v747: 17 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 12 active+clean; 5.4 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 640 op/s; 5440/175768 objects misplaced (3.095%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:16:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:56 smithi194 ceph-mon[44089]: 3.9 scrub starts 2024-04-29T14:16:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:56 smithi194 ceph-mon[44089]: osdmap e278: 12 total, 12 up, 9 in 2024-04-29T14:16:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:56 smithi194 ceph-mon[44089]: 3.9 scrub ok 2024-04-29T14:16:56.565 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:56.584 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:16:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:56.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:16:56.689 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:16:56.716 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:56.735 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:16:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:56.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:16:56.840 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:16:56.870 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:56.894 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:16:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:56.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:16:56.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:16:57.043 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:57.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:16:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:57.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:16:57.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:16:57.235 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:57.265 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:16:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:57.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:16:57.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:57 smithi194 ceph-mon[44089]: 3.7 scrub starts 2024-04-29T14:16:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:57 smithi040 ceph-mon[39581]: 3.7 scrub starts 2024-04-29T14:16:57.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:16:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:57 smithi121 ceph-mon[44533]: 3.7 scrub starts 2024-04-29T14:16:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:16:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:58 smithi040 ceph-mon[39581]: pgmap v749: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 2448/169170 objects misplaced (1.447%); 41 MiB/s, 649 objects/s recovering 2024-04-29T14:16:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:16:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:16:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:58 smithi040 ceph-mon[39581]: osdmap e279: 12 total, 12 up, 9 in 2024-04-29T14:16:58.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:16:58 smithi040 ceph-mon[39581]: 3.7 scrub ok 2024-04-29T14:16:58.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:58 smithi121 ceph-mon[44533]: pgmap v749: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 2448/169170 objects misplaced (1.447%); 41 MiB/s, 649 objects/s recovering 2024-04-29T14:16:58.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:16:58.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:16:58.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:58 smithi121 ceph-mon[44533]: osdmap e279: 12 total, 12 up, 9 in 2024-04-29T14:16:58.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:16:58 smithi121 ceph-mon[44533]: 3.7 scrub ok 2024-04-29T14:16:58.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:16:58.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:58 smithi194 ceph-mon[44089]: pgmap v749: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 2448/169170 objects misplaced (1.447%); 41 MiB/s, 649 objects/s recovering 2024-04-29T14:16:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:16:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:16:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:58 smithi194 ceph-mon[44089]: osdmap e279: 12 total, 12 up, 9 in 2024-04-29T14:16:58.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:16:58 smithi194 ceph-mon[44089]: 3.7 scrub ok 2024-04-29T14:16:59.908 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:16:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:16:59.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:00.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:00.041 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:00.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:00.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:00.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:00.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:00.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:00.343 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:00.347 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:00 smithi040 ceph-mon[39581]: pgmap v751: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 637 op/s; 2448/169170 objects misplaced (1.447%); 34 MiB/s, 546 objects/s recovering 2024-04-29T14:17:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:00 smithi040 ceph-mon[39581]: 3.d scrub starts 2024-04-29T14:17:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:00 smithi040 ceph-mon[39581]: 3.1 scrub starts 2024-04-29T14:17:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:00 smithi040 ceph-mon[39581]: osdmap e280: 12 total, 12 up, 9 in 2024-04-29T14:17:00.367 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:00 smithi194 ceph-mon[44089]: pgmap v751: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 637 op/s; 2448/169170 objects misplaced (1.447%); 34 MiB/s, 546 objects/s recovering 2024-04-29T14:17:00.367 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:00 smithi194 ceph-mon[44089]: 3.d scrub starts 2024-04-29T14:17:00.367 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:00 smithi194 ceph-mon[44089]: 3.1 scrub starts 2024-04-29T14:17:00.367 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:00 smithi194 ceph-mon[44089]: osdmap e280: 12 total, 12 up, 9 in 2024-04-29T14:17:00.369 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:00 smithi121 ceph-mon[44533]: pgmap v751: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.2 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 637 op/s; 2448/169170 objects misplaced (1.447%); 34 MiB/s, 546 objects/s recovering 2024-04-29T14:17:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:00 smithi121 ceph-mon[44533]: 3.d scrub starts 2024-04-29T14:17:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:00 smithi121 ceph-mon[44533]: 3.1 scrub starts 2024-04-29T14:17:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:00 smithi121 ceph-mon[44533]: osdmap e280: 12 total, 12 up, 9 in 2024-04-29T14:17:00.388 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:00.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:00.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:00.521 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:00.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:00.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:00.691 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:00.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:00.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:00.816 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:00.843 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:00.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:00.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:00.994 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:01.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:01.106 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.8 to 1 (10000) 2024-04-29T14:17:01.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:01.158 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:01.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:01.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:01.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:01 smithi040 ceph-mon[39581]: 3.d scrub ok 2024-04-29T14:17:01.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:01 smithi040 ceph-mon[39581]: 3.1 scrub ok 2024-04-29T14:17:01.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:01 smithi040 ceph-mon[39581]: 3.6 scrub starts 2024-04-29T14:17:01.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:01 smithi040 ceph-mon[39581]: osdmap e281: 12 total, 12 up, 9 in 2024-04-29T14:17:01.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:01 smithi040 ceph-mon[39581]: 3.6 scrub ok 2024-04-29T14:17:01.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3456080657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:17:01.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:01.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:01 smithi121 ceph-mon[44533]: 3.d scrub ok 2024-04-29T14:17:01.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:01 smithi121 ceph-mon[44533]: 3.1 scrub ok 2024-04-29T14:17:01.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:01 smithi121 ceph-mon[44533]: 3.6 scrub starts 2024-04-29T14:17:01.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:01 smithi121 ceph-mon[44533]: osdmap e281: 12 total, 12 up, 9 in 2024-04-29T14:17:01.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:01 smithi121 ceph-mon[44533]: 3.6 scrub ok 2024-04-29T14:17:01.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3456080657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:17:01.366 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:01.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:01.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:01.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:01.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:01 smithi194 ceph-mon[44089]: 3.d scrub ok 2024-04-29T14:17:01.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:01 smithi194 ceph-mon[44089]: 3.1 scrub ok 2024-04-29T14:17:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:01 smithi194 ceph-mon[44089]: 3.6 scrub starts 2024-04-29T14:17:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:01 smithi194 ceph-mon[44089]: osdmap e281: 12 total, 12 up, 9 in 2024-04-29T14:17:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:01 smithi194 ceph-mon[44089]: 3.6 scrub ok 2024-04-29T14:17:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3456080657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:17:01.593 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:01.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:01.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:01.747 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:01.773 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:01.794 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:01.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:01.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:01.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:01.992 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:01.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:02.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:02.132 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:02.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:02 smithi040 ceph-mon[39581]: pgmap v754: 17 pgs: 2 peering, 2 active+remapped+backfilling, 13 active+clean; 5.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 875 op/s; 612/166072 objects misplaced (0.369%); 52 MiB/s, 827 objects/s recovering 2024-04-29T14:17:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3456080657' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:17:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:02 smithi040 ceph-mon[39581]: osdmap e282: 12 total, 12 up, 9 in 2024-04-29T14:17:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:02 smithi040 ceph-mon[39581]: 3.5 scrub starts 2024-04-29T14:17:02.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:02 smithi040 ceph-mon[39581]: osdmap e283: 12 total, 12 up, 9 in 2024-04-29T14:17:02.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:02 smithi121 ceph-mon[44533]: pgmap v754: 17 pgs: 2 peering, 2 active+remapped+backfilling, 13 active+clean; 5.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 875 op/s; 612/166072 objects misplaced (0.369%); 52 MiB/s, 827 objects/s recovering 2024-04-29T14:17:02.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3456080657' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:17:02.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:02 smithi121 ceph-mon[44533]: osdmap e282: 12 total, 12 up, 9 in 2024-04-29T14:17:02.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:02 smithi121 ceph-mon[44533]: 3.5 scrub starts 2024-04-29T14:17:02.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:02 smithi121 ceph-mon[44533]: osdmap e283: 12 total, 12 up, 9 in 2024-04-29T14:17:02.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:02 smithi194 ceph-mon[44089]: pgmap v754: 17 pgs: 2 peering, 2 active+remapped+backfilling, 13 active+clean; 5.1 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 875 op/s; 612/166072 objects misplaced (0.369%); 52 MiB/s, 827 objects/s recovering 2024-04-29T14:17:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3456080657' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:17:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:02 smithi194 ceph-mon[44089]: osdmap e282: 12 total, 12 up, 9 in 2024-04-29T14:17:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:02 smithi194 ceph-mon[44089]: 3.5 scrub starts 2024-04-29T14:17:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:02 smithi194 ceph-mon[44089]: osdmap e283: 12 total, 12 up, 9 in 2024-04-29T14:17:03.076 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:17:03.076 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:17:03.079 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:17:03.083 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:03.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:03.106 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:17:03.132 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:17:03.156 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:17:03.177 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:17:03.194 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:03.202 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:17:03.209 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:17:03.209 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:17:03.224 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:03.226 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:17:03.247 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:17:03.248 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:03.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:03.270 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:17:03.291 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:17:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:17:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:17:03.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:03 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:17:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:03 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:17:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:03 smithi040 ceph-mon[39581]: 3.5 scrub ok 2024-04-29T14:17:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/729484810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:03 smithi040 ceph-mon[39581]: osdmap e284: 12 total, 12 up, 9 in 2024-04-29T14:17:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:03 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:17:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:03 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:17:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:03 smithi121 ceph-mon[44533]: 3.5 scrub ok 2024-04-29T14:17:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/729484810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:03 smithi121 ceph-mon[44533]: osdmap e284: 12 total, 12 up, 9 in 2024-04-29T14:17:03.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:03 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:17:03.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:03 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:17:03.381 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:03 smithi194 ceph-mon[44089]: 3.5 scrub ok 2024-04-29T14:17:03.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/729484810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:03.382 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:03 smithi194 ceph-mon[44089]: osdmap e284: 12 total, 12 up, 9 in 2024-04-29T14:17:03.384 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:03.406 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:03.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:03.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:03.588 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:03.626 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:03.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:03.736 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:03.765 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:03.784 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:03.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:03.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:04.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:04.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:04.123 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:04.129 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: pgmap v757: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 827 op/s; 444/162548 objects misplaced (0.273%); 30 MiB/s, 484 objects/s recovering 2024-04-29T14:17:04.129 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: 3.b scrub starts 2024-04-29T14:17:04.156 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:04.176 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:04.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:04.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:17:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:04.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:04.312 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:04.336 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:04.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:04.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: pgmap v757: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 827 op/s; 444/162548 objects misplaced (0.273%); 30 MiB/s, 484 objects/s recovering 2024-04-29T14:17:04.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: 3.b scrub starts 2024-04-29T14:17:04.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:04.503 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:04.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: pgmap v757: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 827 op/s; 444/162548 objects misplaced (0.273%); 30 MiB/s, 484 objects/s recovering 2024-04-29T14:17:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: 3.b scrub starts 2024-04-29T14:17:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:04.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:04.764 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:04.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:04.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:04.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:04.963 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:05.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:05.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:05.102 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-29T14:17:05.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:05.207 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:05.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:05.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:05.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:05.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:05 smithi040 ceph-mon[39581]: 3.b scrub ok 2024-04-29T14:17:05.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:05.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:05.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:05 smithi040 ceph-mon[39581]: 3.a scrub starts 2024-04-29T14:17:05.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:05 smithi040 ceph-mon[39581]: 3.e scrub starts 2024-04-29T14:17:05.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:05 smithi040 ceph-mon[39581]: osdmap e285: 12 total, 12 up, 9 in 2024-04-29T14:17:05.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:05 smithi121 ceph-mon[44533]: 3.b scrub ok 2024-04-29T14:17:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:05 smithi121 ceph-mon[44533]: 3.a scrub starts 2024-04-29T14:17:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:05 smithi121 ceph-mon[44533]: 3.e scrub starts 2024-04-29T14:17:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:05 smithi121 ceph-mon[44533]: osdmap e285: 12 total, 12 up, 9 in 2024-04-29T14:17:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:05.404 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:05.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:05.507 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:05.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:05.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:05 smithi194 ceph-mon[44089]: 3.b scrub ok 2024-04-29T14:17:05.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:05.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:05.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:05 smithi194 ceph-mon[44089]: 3.a scrub starts 2024-04-29T14:17:05.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:05 smithi194 ceph-mon[44089]: 3.e scrub starts 2024-04-29T14:17:05.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:05 smithi194 ceph-mon[44089]: osdmap e285: 12 total, 12 up, 9 in 2024-04-29T14:17:05.535 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:05.558 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:05.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:05.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:05.743 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:05.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:05.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:05.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:05.922 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:05.940 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:05.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:06.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:06.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:06.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:06.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:06.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: pgmap v760: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 444/162210 objects misplaced (0.274%); 2.6 MiB/s, 41 objects/s recovering 2024-04-29T14:17:06.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: 3.2 scrub starts 2024-04-29T14:17:06.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: 3.e scrub ok 2024-04-29T14:17:06.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: 3.4 scrub starts 2024-04-29T14:17:06.185 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=2007177597,config=359415709,health=3157097308,kv=2232674683}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=1928785552,logm=929333835}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331827136}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1519251859}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=416590300}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1905794633}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=371336566}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=166227369}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906921356}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4278348018}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2022939799}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3003982440}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2554333261}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343886275}) 2024-04-29T14:17:06.186 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772179157}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=308848057}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711189936}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4016559588}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3128899373}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561318033}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3691191902}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3900144842}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2974660750}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3259381857}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790885571}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182023808}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2911102847}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025607442}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071034022}) 2024-04-29T14:17:06.187 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209462993}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1659714910}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958158767}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=826950950}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380415503}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2043942563}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4239261272}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2502780961}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1157920440}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=30,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=28} crc {logm=2123635019,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1332329049,mon_config_key=1905223308}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1278573593}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1489470344}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=47,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=35} crc {mon_config_key=3176536425,monmap=455069592,osd_metadata=4221665743,osd_pg_creating=4179029199,osdmap=1216521796}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=848155763}) 2024-04-29T14:17:06.188 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:06 smithi121 ceph-mon[44533]: 3.c scrub ok 2024-04-29T14:17:06.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:06.302 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:06.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:06.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:06.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: pgmap v760: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 444/162210 objects misplaced (0.274%); 2.6 MiB/s, 41 objects/s recovering 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: 3.2 scrub starts 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: 3.e scrub ok 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: 3.4 scrub starts 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=2007177597,config=359415709,health=3157097308,kv=2232674683}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=1928785552,logm=929333835}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331827136}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1519251859}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=416590300}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1905794633}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=371336566}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=166227369}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906921356}) 2024-04-29T14:17:06.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4278348018}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2022939799}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3003982440}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2554333261}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343886275}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772179157}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=308848057}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711189936}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4016559588}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3128899373}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561318033}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3691191902}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3900144842}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2974660750}) 2024-04-29T14:17:06.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3259381857}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790885571}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182023808}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2911102847}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025607442}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071034022}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209462993}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1659714910}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958158767}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=826950950}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380415503}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2043942563}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4239261272}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2502780961}) 2024-04-29T14:17:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1157920440}) 2024-04-29T14:17:06.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=30,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=28} crc {logm=2123635019,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1332329049,mon_config_key=1905223308}) 2024-04-29T14:17:06.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1278573593}) 2024-04-29T14:17:06.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1489470344}) 2024-04-29T14:17:06.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=47,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=35} crc {mon_config_key=3176536425,monmap=455069592,osd_metadata=4221665743,osd_pg_creating=4179029199,osdmap=1216521796}) 2024-04-29T14:17:06.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=848155763}) 2024-04-29T14:17:06.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:06 smithi040 ceph-mon[39581]: 3.c scrub ok 2024-04-29T14:17:06.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:06.502 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: pgmap v760: 17 pgs: 2 peering, 1 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 444/162210 objects misplaced (0.274%); 2.6 MiB/s, 41 objects/s recovering 2024-04-29T14:17:06.502 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: 3.2 scrub starts 2024-04-29T14:17:06.502 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: 3.e scrub ok 2024-04-29T14:17:06.502 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: 3.4 scrub starts 2024-04-29T14:17:06.502 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=2007177597,config=359415709,health=3157097308,kv=2232674683}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=1928785552,logm=929333835}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331827136}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1519251859}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=416590300}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1905794633}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=371336566}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=166227369}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906921356}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4278348018}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2022939799}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3003982440}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2554333261}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343886275}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772179157}) 2024-04-29T14:17:06.503 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=308848057}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711189936}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4016559588}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3128899373}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561318033}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3691191902}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3900144842}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2974660750}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3259381857}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790885571}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182023808}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2911102847}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025607442}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071034022}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209462993}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1659714910}) 2024-04-29T14:17:06.504 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958158767}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=826950950}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380415503}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2043942563}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4239261272}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2502780961}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1157920440}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=30,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=28} crc {logm=2123635019,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1332329049,mon_config_key=1905223308}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1278573593}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1489470344}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=47,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=35} crc {mon_config_key=3176536425,monmap=455069592,osd_metadata=4221665743,osd_pg_creating=4179029199,osdmap=1216521796}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=848155763}) 2024-04-29T14:17:06.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:06 smithi194 ceph-mon[44089]: 3.c scrub ok 2024-04-29T14:17:06.506 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:06.526 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:06.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:06.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:06.658 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:06.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:17:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:07 smithi194 ceph-mon[44089]: 3.a scrub ok 2024-04-29T14:17:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:07 smithi194 ceph-mon[44089]: osdmap e286: 12 total, 12 up, 9 in 2024-04-29T14:17:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:07 smithi194 ceph-mon[44089]: pgmap v762: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 724 op/s; 6.1 MiB/s, 96 objects/s recovering 2024-04-29T14:17:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:07 smithi194 ceph-mon[44089]: 3.4 scrub ok 2024-04-29T14:17:07.608 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:07.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:07.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:07.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:07 smithi040 ceph-mon[39581]: 3.a scrub ok 2024-04-29T14:17:07.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:07 smithi040 ceph-mon[39581]: osdmap e286: 12 total, 12 up, 9 in 2024-04-29T14:17:07.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:07 smithi040 ceph-mon[39581]: pgmap v762: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 724 op/s; 6.1 MiB/s, 96 objects/s recovering 2024-04-29T14:17:07.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:07 smithi040 ceph-mon[39581]: 3.4 scrub ok 2024-04-29T14:17:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:07.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:07 smithi121 ceph-mon[44533]: 3.a scrub ok 2024-04-29T14:17:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:07 smithi121 ceph-mon[44533]: osdmap e286: 12 total, 12 up, 9 in 2024-04-29T14:17:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:07 smithi121 ceph-mon[44533]: pgmap v762: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 724 op/s; 6.1 MiB/s, 96 objects/s recovering 2024-04-29T14:17:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:07 smithi121 ceph-mon[44533]: 3.4 scrub ok 2024-04-29T14:17:07.713 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:07.743 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:07.944 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:07.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:08.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:08.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:08 smithi194 ceph-mon[44089]: 3.f scrub starts 2024-04-29T14:17:08.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:08 smithi194 ceph-mon[44089]: 3.2 scrub ok 2024-04-29T14:17:08.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:08 smithi194 ceph-mon[44089]: 3.3 scrub starts 2024-04-29T14:17:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:08 smithi040 ceph-mon[39581]: 3.f scrub starts 2024-04-29T14:17:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:08 smithi040 ceph-mon[39581]: 3.2 scrub ok 2024-04-29T14:17:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:08 smithi040 ceph-mon[39581]: 3.3 scrub starts 2024-04-29T14:17:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:08 smithi121 ceph-mon[44533]: 3.f scrub starts 2024-04-29T14:17:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:08 smithi121 ceph-mon[44533]: 3.2 scrub ok 2024-04-29T14:17:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:08 smithi121 ceph-mon[44533]: 3.3 scrub starts 2024-04-29T14:17:08.926 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:08.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:09.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:09.164 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:09.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:09.231 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:17:09.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:09.380 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:09.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:09.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:09.520 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:09 smithi040 ceph-mon[39581]: 3.f scrub ok 2024-04-29T14:17:09.520 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:09 smithi040 ceph-mon[39581]: 3.3 scrub ok 2024-04-29T14:17:09.520 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:09 smithi040 ceph-mon[39581]: pgmap v763: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 552 op/s; 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:17:09.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3422176042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:09.524 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:09 smithi194 ceph-mon[44089]: 3.f scrub ok 2024-04-29T14:17:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:09 smithi194 ceph-mon[44089]: 3.3 scrub ok 2024-04-29T14:17:09.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:09 smithi194 ceph-mon[44089]: pgmap v763: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 552 op/s; 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:17:09.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3422176042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:09 smithi121 ceph-mon[44533]: 3.f scrub ok 2024-04-29T14:17:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:09 smithi121 ceph-mon[44533]: 3.3 scrub ok 2024-04-29T14:17:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:09 smithi121 ceph-mon[44533]: pgmap v763: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 552 op/s; 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:17:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3422176042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:09.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:09.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:09.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:09.819 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:09.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:09.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:09.958 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:09.986 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:10.008 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:10.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:10.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:10.141 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:10.155 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:10.192 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:10.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.9 to 1 (10000) 2024-04-29T14:17:10.234 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:10 smithi194 ceph-mon[44089]: 3.8 scrub starts 2024-04-29T14:17:10.234 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:10 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:17:10.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:10.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:10 smithi040 ceph-mon[39581]: 3.8 scrub starts 2024-04-29T14:17:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:10 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:17:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3422176042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:10 smithi040 ceph-mon[39581]: osdmap e287: 12 total, 12 up, 9 in 2024-04-29T14:17:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2340428196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:17:10.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:10.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:10 smithi121 ceph-mon[44533]: 3.8 scrub starts 2024-04-29T14:17:10.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:10 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:17:10.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3422176042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:10.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:10 smithi121 ceph-mon[44533]: osdmap e287: 12 total, 12 up, 9 in 2024-04-29T14:17:10.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2340428196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:17:10.483 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:10.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:10.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3422176042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:10 smithi194 ceph-mon[44089]: osdmap e287: 12 total, 12 up, 9 in 2024-04-29T14:17:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2340428196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:17:10.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:11.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:11 smithi040 ceph-mon[39581]: 3.8 scrub ok 2024-04-29T14:17:11.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2340428196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:17:11.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:11 smithi040 ceph-mon[39581]: osdmap e288: 12 total, 12 up, 9 in 2024-04-29T14:17:11.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:11 smithi040 ceph-mon[39581]: pgmap v766: 17 pgs: 1 unknown, 16 active+clean; 4.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail 2024-04-29T14:17:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:11 smithi194 ceph-mon[44089]: 3.8 scrub ok 2024-04-29T14:17:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2340428196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:17:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:11 smithi194 ceph-mon[44089]: osdmap e288: 12 total, 12 up, 9 in 2024-04-29T14:17:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:11 smithi194 ceph-mon[44089]: pgmap v766: 17 pgs: 1 unknown, 16 active+clean; 4.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail 2024-04-29T14:17:11.587 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:11.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:11.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:11 smithi121 ceph-mon[44533]: 3.8 scrub ok 2024-04-29T14:17:11.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2340428196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:17:11.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:11 smithi121 ceph-mon[44533]: osdmap e288: 12 total, 12 up, 9 in 2024-04-29T14:17:11.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:11 smithi121 ceph-mon[44533]: pgmap v766: 17 pgs: 1 unknown, 16 active+clean; 4.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail 2024-04-29T14:17:11.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:11.971 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:11.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:12.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:17:12.075 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:12.107 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:12.128 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:12.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:12.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:12.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:12.260 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:12.282 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:12.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:12.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:12.515 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:12.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:12.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:12.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:12 smithi040 ceph-mon[39581]: osdmap e289: 12 total, 12 up, 9 in 2024-04-29T14:17:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:12.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:12.538 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:17:12.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:17:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:12 smithi194 ceph-mon[44089]: osdmap e289: 12 total, 12 up, 9 in 2024-04-29T14:17:12.577 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:17:12.596 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:17:12.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:17:12.618 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:12.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:12.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:12.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:12 smithi121 ceph-mon[44533]: osdmap e289: 12 total, 12 up, 9 in 2024-04-29T14:17:12.636 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:17:12.650 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:12.657 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:17:12.672 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:12.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:12.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:17:12.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:17:12.717 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:17:12.737 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:17:12.758 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:17:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:17:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:17:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:17:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:17:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:17:12.760 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:17:12.760 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:17:12.760 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:17:12.760 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:17:12.760 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:17:12.760 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:17:12.760 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:17:12.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:12.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-29T14:17:13.357 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: 3.2 starting backfill to osd.9 from (0'0,0'0] MAX to 287'15379 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: osdmap e290: 12 total, 12 up, 9 in 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2965406085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: pgmap v769: 17 pgs: 1 remapped+peering, 16 active+clean; 4.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:13.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:13.462 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:13.498 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:13.519 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:13.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: 3.2 starting backfill to osd.9 from (0'0,0'0] MAX to 287'15379 2024-04-29T14:17:13.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: osdmap e290: 12 total, 12 up, 9 in 2024-04-29T14:17:13.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2965406085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:13.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.525 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.525 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: pgmap v769: 17 pgs: 1 remapped+peering, 16 active+clean; 4.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-29T14:17:13.525 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.525 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:13.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:13.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:13.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: 3.2 starting backfill to osd.9 from (0'0,0'0] MAX to 287'15379 2024-04-29T14:17:13.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: osdmap e290: 12 total, 12 up, 9 in 2024-04-29T14:17:13.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2965406085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:13.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: pgmap v769: 17 pgs: 1 remapped+peering, 16 active+clean; 4.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-29T14:17:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:13.666 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:13.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:13.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:13.890 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:13.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:13.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:14.028 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:14.058 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:14.079 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:14.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:14.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:14.222 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:14.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:14.256 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:14.258 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.271 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:17:14.271 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:17:14.271 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:17:14.272 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:17:14.272 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:17:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.351 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:14.384 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:14.410 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:14.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:14.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:14.516 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:14.549 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:14.572 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:14.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:14.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:14.713 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:14.737 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:14.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:14.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:15 smithi194 ceph-mon[44089]: pgmap v770: 17 pgs: 1 remapped+peering, 16 active+clean; 4.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 804 op/s 2024-04-29T14:17:15.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:15 smithi040 ceph-mon[39581]: pgmap v770: 17 pgs: 1 remapped+peering, 16 active+clean; 4.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 804 op/s 2024-04-29T14:17:15.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:15.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:15 smithi121 ceph-mon[44533]: pgmap v770: 17 pgs: 1 remapped+peering, 16 active+clean; 4.5 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 804 op/s 2024-04-29T14:17:15.819 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:15.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:15.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:15.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:17:15.957 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:15.978 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:15.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:16.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:16.251 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:16.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:16.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:17.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:17.647 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:17:17.678 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:17.702 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:17.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:17.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:17.873 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:17.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:17.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:17 smithi040 ceph-mon[39581]: pgmap v771: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 932 op/s; 4084/141828 objects misplaced (2.880%); 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:17:17.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3651270028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:17:17.934 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:17:18.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:18.047 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:18.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:17 smithi121 ceph-mon[44533]: pgmap v771: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 932 op/s; 4084/141828 objects misplaced (2.880%); 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:17:18.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3651270028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:17:18.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:17 smithi194 ceph-mon[44089]: pgmap v771: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 932 op/s; 4084/141828 objects misplaced (2.880%); 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:17:18.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3651270028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:17:18.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:18.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:18.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:18.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:18.224 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:18.248 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:18.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:18.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:18.381 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:18.405 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:18.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:18.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:18.540 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:18.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:18.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:18.673 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:18 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3651270028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:18 smithi040 ceph-mon[39581]: osdmap e291: 12 total, 12 up, 9 in 2024-04-29T14:17:19.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:18 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:19.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3651270028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:17:19.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:18 smithi194 ceph-mon[44089]: osdmap e291: 12 total, 12 up, 9 in 2024-04-29T14:17:19.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:18 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:19.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3651270028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:17:19.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:18 smithi121 ceph-mon[44533]: osdmap e291: 12 total, 12 up, 9 in 2024-04-29T14:17:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:19.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:19.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:19.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:19.591 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.10 to 1 (10000) 2024-04-29T14:17:19.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:19 smithi040 ceph-mon[39581]: pgmap v773: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 85 MiB/s wr, 2.67k op/s; 4084/141828 objects misplaced (2.880%); 71 MiB/s, 1.14k objects/s recovering 2024-04-29T14:17:19.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3264482215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:17:19.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:19 smithi040 ceph-mon[39581]: from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:17:20.040 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:19 smithi121 ceph-mon[44533]: pgmap v773: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 85 MiB/s wr, 2.67k op/s; 4084/141828 objects misplaced (2.880%); 71 MiB/s, 1.14k objects/s recovering 2024-04-29T14:17:20.040 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3264482215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:17:20.040 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:19 smithi121 ceph-mon[44533]: from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:17:20.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:19 smithi194 ceph-mon[44089]: pgmap v773: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 85 MiB/s wr, 2.67k op/s; 4084/141828 objects misplaced (2.880%); 71 MiB/s, 1.14k objects/s recovering 2024-04-29T14:17:20.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3264482215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:17:20.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:19 smithi194 ceph-mon[44089]: from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:17:20.119 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:20.142 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:20.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:20.247 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:20.279 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:20.301 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:20.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:20.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:20.457 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:20.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:20.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:20.617 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:20.644 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:20.669 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:20.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:20.706 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:20 smithi121 ceph-mon[44533]: from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:17:20.706 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:20 smithi121 ceph-mon[44533]: osdmap e292: 12 total, 12 up, 9 in 2024-04-29T14:17:20.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:20.825 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:20.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:20.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:20.967 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:20.997 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:20 smithi194 ceph-mon[44089]: from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:17:20.997 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:20 smithi194 ceph-mon[44089]: osdmap e292: 12 total, 12 up, 9 in 2024-04-29T14:17:20.999 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:21.023 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:21.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:21.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:20 smithi040 ceph-mon[39581]: from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:17:21.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:20 smithi040 ceph-mon[39581]: osdmap e292: 12 total, 12 up, 9 in 2024-04-29T14:17:21.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:21.166 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:21.972 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:17:22.002 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:21 smithi040 ceph-mon[39581]: pgmap v775: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 4084/138146 objects misplaced (2.956%); 3.7 MiB/s, 58 objects/s recovering 2024-04-29T14:17:22.002 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:17:22.002 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:17:22.002 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:17:22.002 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:17:22.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:21 smithi194 ceph-mon[44089]: pgmap v775: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 4084/138146 objects misplaced (2.956%); 3.7 MiB/s, 58 objects/s recovering 2024-04-29T14:17:22.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:17:22.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:17:22.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:17:22.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:17:22.068 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:17:22.068 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:17:22.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:17:22.071 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:17:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:17:22.074 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:17:22.074 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:17:22.074 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:17:22.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:21 smithi121 ceph-mon[44533]: pgmap v775: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 590 op/s; 4084/138146 objects misplaced (2.956%); 3.7 MiB/s, 58 objects/s recovering 2024-04-29T14:17:22.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:17:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:17:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:17:22.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:17:22.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-29T14:17:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:22.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:22.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:22.670 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:22.699 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:22.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:22.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:22.836 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:22.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3742534347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:22.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:22.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:22.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:22.998 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:23.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3742534347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:23.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3742534347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.198 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:23.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:23.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:23.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:23.356 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:23.389 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:23.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:23.493 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:23.523 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:23.542 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:23.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:23.647 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:23.697 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:23.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:23.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:23.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: pgmap v776: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 942 op/s; 3191/132176 objects misplaced (2.414%); 8.7 MiB/s, 138 objects/s recovering 2024-04-29T14:17:23.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:23.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:23.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:17:23.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:17:23.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: pgmap v777: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 3191/132176 objects misplaced (2.414%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:17:23.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: pgmap v778: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 3191/132176 objects misplaced (2.414%); 7.9 MiB/s, 126 objects/s recovering 2024-04-29T14:17:23.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:23.819 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:17:23.819 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:17:23.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:23.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: pgmap v776: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 942 op/s; 3191/132176 objects misplaced (2.414%); 8.7 MiB/s, 138 objects/s recovering 2024-04-29T14:17:23.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:23.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:23.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:17:23.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:17:23.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: pgmap v777: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 3191/132176 objects misplaced (2.414%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:17:23.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: pgmap v778: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 3191/132176 objects misplaced (2.414%); 7.9 MiB/s, 126 objects/s recovering 2024-04-29T14:17:23.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:23.860 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:23.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:23.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:24.002 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:24.030 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:24.051 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:24.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:24.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:24.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: pgmap v776: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 942 op/s; 3191/132176 objects misplaced (2.414%); 8.7 MiB/s, 138 objects/s recovering 2024-04-29T14:17:24.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:24.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:24.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:17:24.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:17:24.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: pgmap v777: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 787 op/s; 3191/132176 objects misplaced (2.414%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:17:24.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: pgmap v778: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 3191/132176 objects misplaced (2.414%); 7.9 MiB/s, 126 objects/s recovering 2024-04-29T14:17:24.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:17:24.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:24.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:24.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:24.360 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:24.391 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:24.417 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:24.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:24.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:24.903 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:24.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:24.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:17:24.943 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:25.007 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:25.038 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:25.059 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:25.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:25.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:25.231 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:25.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:25.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:25.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:25.399 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:25.849 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:25.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:25.954 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:25.988 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:26.010 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:26.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:26.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:25 smithi194 ceph-mon[44089]: pgmap v779: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 834 op/s; 3191/131602 objects misplaced (2.425%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:17:26.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:26.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:26.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:25 smithi040 ceph-mon[39581]: pgmap v779: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 834 op/s; 3191/131602 objects misplaced (2.425%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:17:26.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:26.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:26.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:25 smithi121 ceph-mon[44533]: pgmap v779: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 4.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 834 op/s; 3191/131602 objects misplaced (2.425%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:17:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:26.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:26.142 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:26.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:17:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:26.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:26.598 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:26.665 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:26.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:26.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:26.800 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:26.825 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:26.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:26.930 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:26.962 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:26.983 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:26.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:27.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:26 smithi194 ceph-mon[44089]: osdmap e293: 12 total, 12 up, 9 in 2024-04-29T14:17:27.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:27.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:26 smithi040 ceph-mon[39581]: osdmap e293: 12 total, 12 up, 9 in 2024-04-29T14:17:27.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:26 smithi121 ceph-mon[44533]: osdmap e293: 12 total, 12 up, 9 in 2024-04-29T14:17:27.132 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:27.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:27.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:27.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:27.344 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:27.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:27.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:27.487 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:27.620 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:27.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:27.726 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:27.755 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:27.882 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:27.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:27.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:27.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:28.052 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:28.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:28.138 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:28 smithi040 ceph-mon[39581]: pgmap v781: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s; 2255/122626 objects misplaced (1.839%); 14 MiB/s, 221 objects/s recovering 2024-04-29T14:17:28.157 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:28.194 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:28.229 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:28.251 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:28.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:28.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:28 smithi194 ceph-mon[44089]: pgmap v781: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s; 2255/122626 objects misplaced (1.839%); 14 MiB/s, 221 objects/s recovering 2024-04-29T14:17:28.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:28.368 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:28.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:28 smithi121 ceph-mon[44533]: pgmap v781: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s; 2255/122626 objects misplaced (1.839%); 14 MiB/s, 221 objects/s recovering 2024-04-29T14:17:28.396 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:29.069 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:17:29.070 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.11 to 1 (10000) 2024-04-29T14:17:29.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/930675486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T14:17:29.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2900633155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/930675486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T14:17:29.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2900633155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/930675486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T14:17:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2900633155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:30.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:30 smithi194 ceph-mon[44089]: pgmap v782: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 779 op/s; 2255/122626 objects misplaced (1.839%); 7.4 MiB/s, 117 objects/s recovering 2024-04-29T14:17:30.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:30 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/930675486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T14:17:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2900633155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:30.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:30 smithi194 ceph-mon[44089]: osdmap e294: 12 total, 12 up, 9 in 2024-04-29T14:17:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:30 smithi040 ceph-mon[39581]: pgmap v782: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 779 op/s; 2255/122626 objects misplaced (1.839%); 7.4 MiB/s, 117 objects/s recovering 2024-04-29T14:17:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:30 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/930675486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T14:17:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2900633155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:30.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:30 smithi040 ceph-mon[39581]: osdmap e294: 12 total, 12 up, 9 in 2024-04-29T14:17:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:30 smithi121 ceph-mon[44533]: pgmap v782: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 779 op/s; 2255/122626 objects misplaced (1.839%); 7.4 MiB/s, 117 objects/s recovering 2024-04-29T14:17:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:30 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/930675486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T14:17:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2900633155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:30 smithi121 ceph-mon[44533]: osdmap e294: 12 total, 12 up, 9 in 2024-04-29T14:17:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:30.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:30.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-29T14:17:30.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:30.807 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:30.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:30.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:30.944 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:30.976 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:31.002 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:31.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:31.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:31 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:31.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:31 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:31.062 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:31 smithi121 ceph-mon[44533]: osdmap e295: 12 total, 12 up, 9 in 2024-04-29T14:17:31.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:31.147 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:31 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:31 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:31.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:31 smithi194 ceph-mon[44089]: osdmap e295: 12 total, 12 up, 9 in 2024-04-29T14:17:31.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:31 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:31.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:31 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:31.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:31 smithi040 ceph-mon[39581]: osdmap e295: 12 total, 12 up, 9 in 2024-04-29T14:17:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:32 smithi040 ceph-mon[39581]: pgmap v785: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 2255/119078 objects misplaced (1.894%); 7.5 MiB/s, 120 objects/s recovering 2024-04-29T14:17:32.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:32 smithi121 ceph-mon[44533]: pgmap v785: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 2255/119078 objects misplaced (1.894%); 7.5 MiB/s, 120 objects/s recovering 2024-04-29T14:17:32.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:32 smithi194 ceph-mon[44089]: pgmap v785: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.04k op/s; 2255/119078 objects misplaced (1.894%); 7.5 MiB/s, 120 objects/s recovering 2024-04-29T14:17:34.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:34 smithi040 ceph-mon[39581]: pgmap v786: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.5 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 991 op/s; 1397/113090 objects misplaced (1.235%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:17:34.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:34 smithi121 ceph-mon[44533]: pgmap v786: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.5 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 991 op/s; 1397/113090 objects misplaced (1.235%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:17:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:34 smithi194 ceph-mon[44089]: pgmap v786: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.5 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 991 op/s; 1397/113090 objects misplaced (1.235%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:17:34.746 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:35.630 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:35.633 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:17:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:35.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:35.653 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:17:35.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:17:35.693 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:17:35.725 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:17:35.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:35.756 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:17:35.782 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:17:35.795 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:35.814 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:17:35.835 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:17:35.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:35.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:35.859 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:17:35.879 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:17:35.905 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:17:35.907 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:17:35.908 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:17:35.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:35.996 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:36.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:36.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:36.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:36 smithi121 ceph-mon[44533]: pgmap v787: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.4 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 634 op/s; 1397/112466 objects misplaced (1.242%); 5.5 MiB/s, 88 objects/s recovering 2024-04-29T14:17:36.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3782447147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:36.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:36.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:17:36.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:36 smithi040 ceph-mon[39581]: pgmap v787: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.4 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 634 op/s; 1397/112466 objects misplaced (1.242%); 5.5 MiB/s, 88 objects/s recovering 2024-04-29T14:17:36.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3782447147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:36.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:36.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:36 smithi194 ceph-mon[44089]: pgmap v787: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.4 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 634 op/s; 1397/112466 objects misplaced (1.242%); 5.5 MiB/s, 88 objects/s recovering 2024-04-29T14:17:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3782447147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:36.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:36.604 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:36.957 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:36.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:37.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:37.096 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:37.234 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:17:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:17:37.234 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:17:37.234 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:17:37.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:37.647 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:37.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:37.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:37.783 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:37.861 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:37.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:37.975 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:38.007 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:38.031 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:38.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:38.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:38.203 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:38.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:38.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:38.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:38 smithi194 ceph-mon[44089]: pgmap v788: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 576/103486 objects misplaced (0.557%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:17:38.337 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:38 smithi040 ceph-mon[39581]: pgmap v788: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 576/103486 objects misplaced (0.557%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:17:38.362 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:38.369 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:17:38.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:38 smithi121 ceph-mon[44533]: pgmap v788: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 576/103486 objects misplaced (0.557%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:17:38.391 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:38.412 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:38.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:38.516 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:38.552 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:38.573 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:38.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:38.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:38.741 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:38.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:38.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:38.961 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:39.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:39.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:39.104 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.6 to 1 (10000) 2024-04-29T14:17:39.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:39.276 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:39.283 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1156545496' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:17:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:39.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1156545496' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:17:39.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:39.452 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:39.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:39.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1156545496' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:17:39.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:39.665 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:39.687 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:39.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:39.791 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:39.819 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:39.847 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:39.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:39.951 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:39.980 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:40.002 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:40.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:40.106 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:40.137 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:40.165 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:40.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:40.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:40.297 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:40.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:40.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:40.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:40 smithi040 ceph-mon[39581]: pgmap v789: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 989 op/s; 576/103486 objects misplaced (0.557%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:17:40.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:40.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:40.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1156545496' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:17:40.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:40 smithi040 ceph-mon[39581]: osdmap e296: 12 total, 12 up, 9 in 2024-04-29T14:17:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:40 smithi121 ceph-mon[44533]: pgmap v789: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 989 op/s; 576/103486 objects misplaced (0.557%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:17:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1156545496' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:17:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:40 smithi121 ceph-mon[44533]: osdmap e296: 12 total, 12 up, 9 in 2024-04-29T14:17:40.420 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:40.444 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:40 smithi194 ceph-mon[44089]: pgmap v789: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 3.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 989 op/s; 576/103486 objects misplaced (0.557%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:17:40.444 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:40.445 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:40.445 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1156545496' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:17:40.445 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:40 smithi194 ceph-mon[44089]: osdmap e296: 12 total, 12 up, 9 in 2024-04-29T14:17:40.446 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:40.464 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:40.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:40.503 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-29T14:17:40.503 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-29T14:17:40.503 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-29T14:17:40.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T14:17:40.568 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:40.594 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:40.612 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:40.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:40.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:40.742 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:40.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:40.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:40.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:40.890 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:40.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:40.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:40.980 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:40.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:17:41.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:41.038 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:41.057 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:41.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:41.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:41.188 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:41.287 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:41.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:41.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:41 smithi040 ceph-mon[39581]: osdmap e297: 12 total, 12 up, 9 in 2024-04-29T14:17:41.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:41 smithi194 ceph-mon[44089]: osdmap e297: 12 total, 12 up, 9 in 2024-04-29T14:17:41.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:41 smithi121 ceph-mon[44533]: osdmap e297: 12 total, 12 up, 9 in 2024-04-29T14:17:41.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:41.428 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:42.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:42 smithi121 ceph-mon[44533]: pgmap v792: 17 pgs: 1 peering, 16 active+clean; 3.0 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 852 op/s; 10 MiB/s, 159 objects/s recovering 2024-04-29T14:17:42.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:42 smithi121 ceph-mon[44533]: osdmap e298: 12 total, 12 up, 9 in 2024-04-29T14:17:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:42 smithi194 ceph-mon[44089]: pgmap v792: 17 pgs: 1 peering, 16 active+clean; 3.0 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 852 op/s; 10 MiB/s, 159 objects/s recovering 2024-04-29T14:17:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:42 smithi194 ceph-mon[44089]: osdmap e298: 12 total, 12 up, 9 in 2024-04-29T14:17:42.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:42 smithi040 ceph-mon[39581]: pgmap v792: 17 pgs: 1 peering, 16 active+clean; 3.0 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 852 op/s; 10 MiB/s, 159 objects/s recovering 2024-04-29T14:17:42.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:42 smithi040 ceph-mon[39581]: osdmap e298: 12 total, 12 up, 9 in 2024-04-29T14:17:42.752 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:42.753 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:42.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:42.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:43.141 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:17:43.271 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1061849687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:17:43.271 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:43 smithi040 ceph-mon[39581]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:17:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1061849687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:17:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:43 smithi194 ceph-mon[44089]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:17:43.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1061849687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:17:43.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:43 smithi121 ceph-mon[44533]: from='client.25372 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:17:43.881 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:43.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:43.985 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:44.017 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:44.045 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:44.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:44.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:44.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:44 smithi194 ceph-mon[44089]: pgmap v794: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 705 op/s; 5.7 MiB/s, 91 objects/s recovering 2024-04-29T14:17:44.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:44 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:44.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:44 smithi194 ceph-mon[44089]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:17:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:44 smithi194 ceph-mon[44089]: osdmap e299: 12 total, 12 up, 9 in 2024-04-29T14:17:44.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:44 smithi040 ceph-mon[39581]: pgmap v794: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 705 op/s; 5.7 MiB/s, 91 objects/s recovering 2024-04-29T14:17:44.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:44 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:44.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:44 smithi040 ceph-mon[39581]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:17:44.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:44 smithi040 ceph-mon[39581]: osdmap e299: 12 total, 12 up, 9 in 2024-04-29T14:17:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:44 smithi121 ceph-mon[44533]: pgmap v794: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 705 op/s; 5.7 MiB/s, 91 objects/s recovering 2024-04-29T14:17:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:44 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:17:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:44 smithi121 ceph-mon[44533]: from='client.25372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:17:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:44 smithi121 ceph-mon[44533]: osdmap e299: 12 total, 12 up, 9 in 2024-04-29T14:17:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:44.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:44.662 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:17:44.663 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:17:44.663 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:17:44.663 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:17:44.663 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:17:44.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:44.815 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:44.838 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:44.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:44.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:45.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:45.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:45.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:45.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:45.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:45.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:45.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:45.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:45.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:45.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:45.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:46.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:17:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:46 smithi194 ceph-mon[44089]: pgmap v796: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 8.1 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 797 op/s; 6.1 MiB/s, 98 objects/s recovering 2024-04-29T14:17:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:46 smithi194 ceph-mon[44089]: osdmap e300: 12 total, 12 up, 9 in 2024-04-29T14:17:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:46 smithi040 ceph-mon[39581]: pgmap v796: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 8.1 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 797 op/s; 6.1 MiB/s, 98 objects/s recovering 2024-04-29T14:17:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:46 smithi040 ceph-mon[39581]: osdmap e300: 12 total, 12 up, 9 in 2024-04-29T14:17:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:46 smithi121 ceph-mon[44533]: pgmap v796: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 8.1 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 797 op/s; 6.1 MiB/s, 98 objects/s recovering 2024-04-29T14:17:46.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:46 smithi121 ceph-mon[44533]: osdmap e300: 12 total, 12 up, 9 in 2024-04-29T14:17:46.814 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:46.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:46.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:47.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:47 smithi040 ceph-mon[39581]: 3.0 scrub starts 2024-04-29T14:17:47.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:47 smithi040 ceph-mon[39581]: 3.0 scrub ok 2024-04-29T14:17:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:47 smithi194 ceph-mon[44089]: 3.0 scrub starts 2024-04-29T14:17:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:47 smithi194 ceph-mon[44089]: 3.0 scrub ok 2024-04-29T14:17:47.626 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:17:47.626 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":300,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T14:17:45.148805+0000","last_up_change":"2024-04-29T14:15:57.071537+0000","last_in_change":"2024-04-29T14:15:38.029189+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.109999656677246,"score_stable":11.109999656677246,"optimal_score":0.18000000715255737,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.91666656732559204,"average_primary_affinity":1,"average_primary_affinity_weighted":0.92000001668930054}},{"pool":3,"pool_name":"unique_pool_1","create_time":"2024-04-29T14:13:42.111930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.0499999523162842,"score_stable":2.0499999523162842,"optimal_score":0.73000001907348633,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.91666656732559204,"average_primary_affinity":1,"average_primary_affinity_weighted":0.92000001668930054}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":280,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":4178456629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":4178456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":4178456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":4178456629}]},"public_addr":"172.21.15.40:6805/4178456629","cluster_addr":"172.21.15.40:6806/4178456629","heartbeat_back_addr":"172.21.15.40:6808/4178456629","heartbeat_front_addr":"172.21.15.40:6807/4178456629","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":230,"up_from":260,"up_thru":152,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3416690575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3416690575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3416690575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3416690575}]},"public_addr":"172.21.15.40:6809/3416690575","cluster_addr":"172.21.15.40:6810/3416690575","heartbeat_back_addr":"172.21.15.40:6812/3416690575","heartbeat_front_addr":"172.21.15.40:6811/3416690575","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":296,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":281,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":3807513901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":3807513901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":3807513901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":3807513901}]},"public_addr":"172.21.15.121:6804/3807513901","cluster_addr":"172.21.15.121:6805/3807513901","heartbeat_back_addr":"172.21.15.121:6807/3807513901","heartbeat_front_addr":"172.21.15.121:6806/3807513901","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":283,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":2130074108}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":2130074108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":2130074108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":2130074108}]},"public_addr":"172.21.15.194:6800/2130074108","cluster_addr":"172.21.15.194:6801/2130074108","heartbeat_back_addr":"172.21.15.194:6803/2130074108","heartbeat_front_addr":"172.21.15.194:6802/2130074108","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":297,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":453882699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":453882699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":453882699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":453882699}]},"public_addr":"172.21.15.194:6804/453882699","cluster_addr":"172.21.15.194:6805/453882699","heartbeat_back_addr":"172.21.15.194:6807/453882699","heartbeat_front_addr":"172.21.15.194:6806/453882699","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":199,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:16:42.953681+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":51693,"last_purged_snaps_scrub":"2024-04-29T14:16:49.086809+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-29T14:17:06.004209+0000","dead_epoch":230},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-29T14:17:03.987149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:16:33.149453+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:17:29.153736+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:17:10.871471+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:17:44.649038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:17:38.782831+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:17:25.296135+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:16:45.065148+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:16:56.927623+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T14:17:47.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:47 smithi121 ceph-mon[44533]: 3.0 scrub starts 2024-04-29T14:17:47.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:47 smithi121 ceph-mon[44533]: 3.0 scrub ok 2024-04-29T14:17:47.809 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:17:47.833 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:17:47.861 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:17:47.884 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:17:47.908 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:17:47.930 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:17:47.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:17:47.977 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:17:48.002 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:17:48.026 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:17:48.048 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:17:48.074 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:17:48.075 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:17:48.075 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:17:48.075 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:17:48.075 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:17:48.075 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:17:48.076 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:17:48.076 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:17:48.076 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:17:48.076 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:17:48.076 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:17:48.076 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:17:48.076 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:17:48.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:48 smithi040 ceph-mon[39581]: pgmap v798: 17 pgs: 17 active+clean; 2.6 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:48.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1954667347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:17:48.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2476090473' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:48.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:48.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:48 smithi194 ceph-mon[44089]: pgmap v798: 17 pgs: 17 active+clean; 2.6 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1954667347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:17:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2476090473' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:48.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:48 smithi121 ceph-mon[44533]: pgmap v798: 17 pgs: 17 active+clean; 2.6 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:48.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1954667347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:17:48.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2476090473' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:48.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:48.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:48.891 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-29T14:17:49.066 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.592 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.600 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:17:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:49.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:17:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:49.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:49.736 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:49.762 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:49.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:49.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:49.903 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:49.924 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:49.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:49.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:50.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:50.071 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:50.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:17:50.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:50 smithi040 ceph-mon[39581]: pgmap v799: 17 pgs: 17 active+clean; 2.6 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:50 smithi194 ceph-mon[44089]: pgmap v799: 17 pgs: 17 active+clean; 2.6 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:50.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:50 smithi121 ceph-mon[44533]: pgmap v799: 17 pgs: 17 active+clean; 2.6 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 671 op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:50.921 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:50.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:51.026 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:17:51.054 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:51.080 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:17:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:51.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:17:51.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:51.233 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:51.489 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:51.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:51.594 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:51.623 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:17:51.624 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:51.656 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:51.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:51.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:51.795 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:51.902 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:17:51.902 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:17:51.905 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:51.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:52.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:52.070 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:52.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:52.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:52.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:52.286 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:52.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:17:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:52.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:17:52.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:52 smithi121 ceph-mon[44533]: pgmap v800: 17 pgs: 17 active+clean; 2.5 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 834 op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:52.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:52 smithi040 ceph-mon[39581]: pgmap v800: 17 pgs: 17 active+clean; 2.5 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 834 op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:52.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:52.500 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:52.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:52.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:52 smithi194 ceph-mon[44089]: pgmap v800: 17 pgs: 17 active+clean; 2.5 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 834 op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:52.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:52.675 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:53.427 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:53.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:53.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:17:53.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:53 smithi121 ceph-mon[44533]: pgmap v801: 17 pgs: 17 active+clean; 2.3 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:53.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:53 smithi194 ceph-mon[44089]: pgmap v801: 17 pgs: 17 active+clean; 2.3 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:53.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:53 smithi040 ceph-mon[39581]: pgmap v801: 17 pgs: 17 active+clean; 2.3 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:53.629 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:53.669 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:53.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:53.779 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:53.887 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:17:54.014 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:54.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:54.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:54.150 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:54.232 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:54.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:54.336 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:54.365 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:54.472 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:17:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2089077242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:54.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2089077242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:54.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2089077242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:17:55.019 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:55.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:55.123 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:17:55.157 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:55.177 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:17:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:55.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:17:55.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:17:55.310 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:55.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:55 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:17:55.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2089077242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:55.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:55 smithi194 ceph-mon[44089]: osdmap e301: 12 total, 12 up, 9 in 2024-04-29T14:17:55.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:55 smithi194 ceph-mon[44089]: pgmap v803: 17 pgs: 17 active+clean; 2.3 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:55 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:17:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2089077242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:55 smithi040 ceph-mon[39581]: osdmap e301: 12 total, 12 up, 9 in 2024-04-29T14:17:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:55 smithi040 ceph-mon[39581]: pgmap v803: 17 pgs: 17 active+clean; 2.3 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:55.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:55 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:17:55.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2089077242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:17:55.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:55 smithi121 ceph-mon[44533]: osdmap e301: 12 total, 12 up, 9 in 2024-04-29T14:17:55.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:55 smithi121 ceph-mon[44533]: pgmap v803: 17 pgs: 17 active+clean; 2.3 GiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 0 B/s, 0 objects/s recovering 2024-04-29T14:17:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:17:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:55.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:17:55.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:55.844 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:55.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:55.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:56.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:56.068 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:56.096 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:17:56.096 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:17:56.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:56.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:56.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:56.246 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:56.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:56.383 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:17:56.413 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:56.437 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:17:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:56.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:17:56.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:56 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:56 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:56 smithi194 ceph-mon[44089]: osdmap e302: 12 total, 12 up, 9 in 2024-04-29T14:17:56.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:56 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:56.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:56 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:56.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:56 smithi040 ceph-mon[39581]: osdmap e302: 12 total, 12 up, 9 in 2024-04-29T14:17:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:56.614 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:17:56.639 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:17:56.660 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:17:56.681 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:17:56.714 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:17:56.743 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:17:56.770 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:17:56.798 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:17:56.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:56.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:56.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:56 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:56.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:56 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:56.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:56 smithi121 ceph-mon[44533]: osdmap e302: 12 total, 12 up, 9 in 2024-04-29T14:17:56.819 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:17:56.842 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:17:56.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:17:56.862 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:17:56.883 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:17:56.885 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:17:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:17:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:17:56.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:56.995 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:57.037 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:57.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:57.142 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:17:57.172 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:57.201 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:17:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:57.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:17:57.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:17:57.354 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:17:57.365 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:57.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:17:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:57.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:17:57.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:17:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:57.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1231345540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:57.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: pgmap v805: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-29T14:17:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:57 smithi040 ceph-mon[39581]: osdmap e303: 12 total, 12 up, 9 in 2024-04-29T14:17:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1231345540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: pgmap v805: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-29T14:17:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:57 smithi121 ceph-mon[44533]: osdmap e303: 12 total, 12 up, 9 in 2024-04-29T14:17:57.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:17:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:17:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:57.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:17:57.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1231345540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:17:57.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: pgmap v805: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-29T14:17:57.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:57.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:57 smithi194 ceph-mon[44089]: osdmap e303: 12 total, 12 up, 9 in 2024-04-29T14:17:57.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:57.940 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:57.963 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:57.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:57.993 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:17:58.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:17:58.142 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:58.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:17:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:58.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:17:58.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.846 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:58.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:17:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:58.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:58.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:17:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:17:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:59.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:17:59.474 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:17:59.474 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:17:59.474 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:17:59.474 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:17:59.474 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:17:59.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:17:59.626 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:17:59.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:59.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:59.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:17:59.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:17:59.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:59 smithi040 ceph-mon[39581]: osdmap e304: 12 total, 12 up, 9 in 2024-04-29T14:17:59.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:17:59 smithi040 ceph-mon[39581]: pgmap v808: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:17:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:17:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:17:59.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:17:59.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:59.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:59.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:17:59.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:17:59.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:59 smithi194 ceph-mon[44089]: osdmap e304: 12 total, 12 up, 9 in 2024-04-29T14:17:59.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:17:59 smithi194 ceph-mon[44089]: pgmap v808: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:17:59.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:17:59.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:17:59.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:59.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:59.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:17:59.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:17:59.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:59 smithi121 ceph-mon[44533]: osdmap e304: 12 total, 12 up, 9 in 2024-04-29T14:17:59.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:17:59 smithi121 ceph-mon[44533]: pgmap v808: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:17:59.901 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:00.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:18:00.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:00.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:00.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:00.438 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:00.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:00.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:00.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3242286257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:00.520 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:00 smithi121 ceph-mon[44533]: from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:18:00.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:00.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3242286257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:00.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:00 smithi040 ceph-mon[39581]: from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:00.657 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:00.691 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:00.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:00.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:00.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:00.802 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:00.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3242286257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:00.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:00 smithi194 ceph-mon[44089]: from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:00.824 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:00.846 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:00.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:00.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:01.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:01.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:01.465 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:01.496 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:01.518 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:01.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:01.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:01 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:01.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:01 smithi121 ceph-mon[44533]: from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:18:01.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:01 smithi121 ceph-mon[44533]: osdmap e305: 12 total, 12 up, 9 in 2024-04-29T14:18:01.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:01 smithi121 ceph-mon[44533]: pgmap v810: 17 pgs: 17 active+clean; 1.9 GiB data, 6.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 933 op/s 2024-04-29T14:18:01.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:01.720 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:01.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:01 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:01.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:01 smithi040 ceph-mon[39581]: from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:18:01.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:01 smithi040 ceph-mon[39581]: osdmap e305: 12 total, 12 up, 9 in 2024-04-29T14:18:01.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:01 smithi040 ceph-mon[39581]: pgmap v810: 17 pgs: 17 active+clean; 1.9 GiB data, 6.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 933 op/s 2024-04-29T14:18:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:01 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:01 smithi194 ceph-mon[44089]: from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:18:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:01 smithi194 ceph-mon[44089]: osdmap e305: 12 total, 12 up, 9 in 2024-04-29T14:18:01.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:01 smithi194 ceph-mon[44089]: pgmap v810: 17 pgs: 17 active+clean; 1.9 GiB data, 6.1 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 933 op/s 2024-04-29T14:18:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:01.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:02.031 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:02.113 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:02.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:02.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:02.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:02.534 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:02.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:02.638 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:02.686 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:02.689 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:02.710 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:02.712 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:02.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:02.729 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:02.749 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:02.777 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:02.802 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:02.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:02.827 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:02.856 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:02.870 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:02.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:18:02.901 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:18:02.924 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:18:02.945 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:18:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:18:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:18:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:18:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:18:02.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:02.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:03.049 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:03.049 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:03.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:03.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:03.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:03 smithi194 ceph-mon[44089]: osdmap e306: 12 total, 12 up, 9 in 2024-04-29T14:18:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1001736535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:03.337 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:03 smithi040 ceph-mon[39581]: osdmap e306: 12 total, 12 up, 9 in 2024-04-29T14:18:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1001736535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:03.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:03.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:03 smithi121 ceph-mon[44533]: osdmap e306: 12 total, 12 up, 9 in 2024-04-29T14:18:03.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1001736535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:03.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:03.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:03.496 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:03.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:18:03.850 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:03.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:03.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:04.040 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:04.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:04.089 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.089 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.089 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: pgmap v812: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 805 op/s 2024-04-29T14:18:04.089 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:04.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: pgmap v812: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 805 op/s 2024-04-29T14:18:04.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:18:04.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:04.223 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:04.242 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:04.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:04.346 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:04.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: pgmap v812: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 805 op/s 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.376 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:04.377 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:04.400 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:04.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:04.505 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:04.532 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:04.553 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:04.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:04.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:04.712 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:04.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:04.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:04.877 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:05.135 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:05.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:05.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:05.286 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:05.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:05.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:05.440 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:05.475 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:05.501 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:05.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:05.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:05.698 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:05.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:05.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:05.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:05.864 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:05.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:18:05.885 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:05.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:05.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:06.017 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:06.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:06 smithi194 ceph-mon[44089]: pgmap v813: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 781 op/s 2024-04-29T14:18:06.345 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:06.345 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:06.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:06 smithi040 ceph-mon[39581]: pgmap v813: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 781 op/s 2024-04-29T14:18:06.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:06 smithi121 ceph-mon[44533]: pgmap v813: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 781 op/s 2024-04-29T14:18:06.568 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:06.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:06.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:06.743 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:06.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:06.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:06.964 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:07.206 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:08.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:08 smithi194 ceph-mon[44089]: pgmap v814: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:18:08.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:08 smithi194 ceph-mon[44089]: osdmap e307: 12 total, 12 up, 9 in 2024-04-29T14:18:08.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:08 smithi040 ceph-mon[39581]: pgmap v814: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:18:08.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:08 smithi040 ceph-mon[39581]: osdmap e307: 12 total, 12 up, 9 in 2024-04-29T14:18:08.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:08 smithi121 ceph-mon[44533]: pgmap v814: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:18:08.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:08 smithi121 ceph-mon[44533]: osdmap e307: 12 total, 12 up, 9 in 2024-04-29T14:18:08.892 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:18:08.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:18:09.073 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:18:09.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1321584286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:09.357 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1321584286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:09.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:09.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1321584286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:09.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:10.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:10 smithi040 ceph-mon[39581]: pgmap v816: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:18:10.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:10 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:10.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1321584286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:10.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:10 smithi040 ceph-mon[39581]: osdmap e308: 12 total, 12 up, 9 in 2024-04-29T14:18:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:10 smithi194 ceph-mon[44089]: pgmap v816: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:18:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:10 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1321584286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:10.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:10 smithi194 ceph-mon[44089]: osdmap e308: 12 total, 12 up, 9 in 2024-04-29T14:18:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:10 smithi121 ceph-mon[44533]: pgmap v816: 17 pgs: 17 active+clean; 1.4 GiB data, 5.3 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:18:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:10 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:10.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1321584286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:10.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:10 smithi121 ceph-mon[44533]: osdmap e308: 12 total, 12 up, 9 in 2024-04-29T14:18:10.440 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:10.479 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:10.500 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:10.521 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:10.541 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:10.568 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:10.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:10.633 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:10.664 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:10.690 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:18:10.715 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:18:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:18:10.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:18:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:18:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:18:11.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/254652125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:11.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.282 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/254652125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:11.282 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.282 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.282 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.282 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.282 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.283 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/254652125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:11.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:18:12.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: pgmap v818: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 856 op/s 2024-04-29T14:18:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:12 smithi040 ceph-mon[39581]: osdmap e309: 12 total, 12 up, 9 in 2024-04-29T14:18:12.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: pgmap v818: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 856 op/s 2024-04-29T14:18:12.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:12 smithi121 ceph-mon[44533]: osdmap e309: 12 total, 12 up, 9 in 2024-04-29T14:18:12.548 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: pgmap v818: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 856 op/s 2024-04-29T14:18:12.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:12.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:12 smithi194 ceph-mon[44089]: osdmap e309: 12 total, 12 up, 9 in 2024-04-29T14:18:12.896 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:12.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:12.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:18:13.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:13.043 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:13.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:13.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:13.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:13.378 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:13.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:13.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:13.456 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:13.519 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:18:13.522 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:18:13.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:13.579 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:13.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:18:13.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:13.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:13.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:13.874 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:13.897 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:13.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:14.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:14.035 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:14 smithi194 ceph-mon[44089]: pgmap v820: 17 pgs: 17 active+clean; 1.1 GiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 804 op/s 2024-04-29T14:18:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:14.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:14 smithi040 ceph-mon[39581]: pgmap v820: 17 pgs: 17 active+clean; 1.1 GiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 804 op/s 2024-04-29T14:18:14.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:14.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:14.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:14.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:14.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:14 smithi121 ceph-mon[44533]: pgmap v820: 17 pgs: 17 active+clean; 1.1 GiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 804 op/s 2024-04-29T14:18:14.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:14.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:14.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:14.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:14.477 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":820,"stamp":"2024-04-29T14:18:12.747746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1226965560,"num_objects":18716,"num_object_clones":0,"num_object_copies":37432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18716,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":299349,"num_write_kb":10175099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41039,"num_bytes_recovered":2691057792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":8,"kb":843706368,"kb_used":4907748,"kb_used_data":2576516,"kb_used_omap":14,"kb_used_meta":2330993,"kb_avail":838798620,"statfs":{"total":863955320832,"available":858929786880,"internally_reserved":0,"allocated":2638352384,"data_stored":2634164656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14360,"internal_metadata":2386937832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-316276736,"num_objects":-4826,"num_object_clones":0,"num_object_copies":-9652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4826,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001072"},"pg_stats":[{"pgid":"3.f","version":"308'20144","reported_seq":20608,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.798080+0000","last_change":"2024-04-29T14:17:07.892573+0000","last_active":"2024-04-29T14:18:10.798080+0000","last_peered":"2024-04-29T14:18:10.798080+0000","last_clean":"2024-04-29T14:18:10.798080+0000","last_became_active":"2024-04-29T14:13:43.169982+0000","last_became_peered":"2024-04-29T14:13:43.169982+0000","last_unstale":"2024-04-29T14:18:10.798080+0000","last_undegraded":"2024-04-29T14:18:10.798080+0000","last_fullsized":"2024-04-29T14:18:10.798080+0000","mapping_epoch":206,"log_start":"244'10101","ondisk_log_start":"244'10101","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16603","last_scrub_stamp":"2024-04-29T14:17:07.892271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:07.892271+0000","objects_scrubbed":4693,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:12.504549+0000","scrub_duration":1107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":2240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18564,"num_write_kb":629888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"308'18413","reported_seq":18752,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:11.410289+0000","last_change":"2024-04-29T14:17:05.422450+0000","last_active":"2024-04-29T14:18:11.410289+0000","last_peered":"2024-04-29T14:18:11.410289+0000","last_clean":"2024-04-29T14:18:11.410289+0000","last_became_active":"2024-04-29T14:13:52.443531+0000","last_became_peered":"2024-04-29T14:13:52.443531+0000","last_unstale":"2024-04-29T14:18:11.410289+0000","last_undegraded":"2024-04-29T14:18:11.410289+0000","last_fullsized":"2024-04-29T14:18:11.410289+0000","mapping_epoch":213,"log_start":"242'8400","ondisk_log_start":"242'8400","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14769","last_scrub_stamp":"2024-04-29T14:17:05.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:05.422201+0000","objects_scrubbed":4768,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:12.036140+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71368704,"num_objects":1089,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18413,"num_write_kb":624064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"308'18920","reported_seq":19284,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.802257+0000","last_change":"2024-04-29T14:16:59.668689+0000","last_active":"2024-04-29T14:18:10.802257+0000","last_peered":"2024-04-29T14:18:10.802257+0000","last_clean":"2024-04-29T14:18:10.802257+0000","last_became_active":"2024-04-29T14:13:43.174780+0000","last_became_peered":"2024-04-29T14:13:43.174780+0000","last_unstale":"2024-04-29T14:18:10.802257+0000","last_undegraded":"2024-04-29T14:18:10.802257+0000","last_fullsized":"2024-04-29T14:18:10.802257+0000","mapping_epoch":206,"log_start":"243'8900","ondisk_log_start":"243'8900","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'14954","last_scrub_stamp":"2024-04-29T14:16:59.668535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:59.668535+0000","objects_scrubbed":5116,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:23.894891+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75038744,"num_objects":1146,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18920,"num_write_kb":642049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"308'18712","reported_seq":32908,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.273274+0000","last_change":"2024-04-29T14:17:05.994990+0000","last_active":"2024-04-29T14:18:10.273274+0000","last_peered":"2024-04-29T14:18:10.273274+0000","last_clean":"2024-04-29T14:18:10.273274+0000","last_became_active":"2024-04-29T14:17:01.115520+0000","last_became_peered":"2024-04-29T14:17:01.115520+0000","last_unstale":"2024-04-29T14:18:10.273274+0000","last_undegraded":"2024-04-29T14:18:10.273274+0000","last_fullsized":"2024-04-29T14:18:10.273274+0000","mapping_epoch":281,"log_start":"243'8700","ondisk_log_start":"243'8700","created":206,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15016","last_scrub_stamp":"2024-04-29T14:17:05.994807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:05.994807+0000","objects_scrubbed":4932,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":2368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18712,"num_write_kb":636672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6911,"num_bytes_recovered":452919296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"308'18596","reported_seq":32893,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.830607+0000","last_change":"2024-04-29T14:17:03.835268+0000","last_active":"2024-04-29T14:18:10.830607+0000","last_peered":"2024-04-29T14:18:10.830607+0000","last_clean":"2024-04-29T14:18:10.830607+0000","last_became_active":"2024-04-29T14:17:00.109521+0000","last_became_peered":"2024-04-29T14:17:00.109521+0000","last_unstale":"2024-04-29T14:18:10.830607+0000","last_undegraded":"2024-04-29T14:18:10.830607+0000","last_fullsized":"2024-04-29T14:18:10.830607+0000","mapping_epoch":280,"log_start":"242'8500","ondisk_log_start":"242'8500","created":206,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'14843","last_scrub_stamp":"2024-04-29T14:17:03.835159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:03.835159+0000","objects_scrubbed":4913,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:32.049771+0000","scrub_duration":815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74579968,"num_objects":1138,"num_object_clones":0,"num_object_copies":2276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18596,"num_write_kb":631488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":456261632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"308'18831","reported_seq":21511,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.465838+0000","last_change":"2024-04-29T14:17:06.021161+0000","last_active":"2024-04-29T14:18:10.465838+0000","last_peered":"2024-04-29T14:18:10.465838+0000","last_clean":"2024-04-29T14:18:10.465838+0000","last_became_active":"2024-04-29T14:14:17.997407+0000","last_became_peered":"2024-04-29T14:14:17.997407+0000","last_unstale":"2024-04-29T14:18:10.465838+0000","last_undegraded":"2024-04-29T14:18:10.465838+0000","last_fullsized":"2024-04-29T14:18:10.465838+0000","mapping_epoch":224,"log_start":"243'8800","ondisk_log_start":"243'8800","created":206,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15095","last_scrub_stamp":"2024-04-29T14:17:06.020918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:06.020918+0000","objects_scrubbed":5028,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:28.724750+0000","scrub_duration":2033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80805888,"num_objects":1233,"num_object_clones":0,"num_object_copies":2466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18831,"num_write_kb":642048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1159,"num_bytes_recovered":75956224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"308'18635","reported_seq":19018,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:09.427834+0000","last_change":"2024-04-29T14:16:55.139438+0000","last_active":"2024-04-29T14:18:09.427834+0000","last_peered":"2024-04-29T14:18:09.427834+0000","last_clean":"2024-04-29T14:18:09.427834+0000","last_became_active":"2024-04-29T14:13:43.170268+0000","last_became_peered":"2024-04-29T14:13:43.170268+0000","last_unstale":"2024-04-29T14:18:09.427834+0000","last_undegraded":"2024-04-29T14:18:09.427834+0000","last_fullsized":"2024-04-29T14:18:09.427834+0000","mapping_epoch":206,"log_start":"242'8600","ondisk_log_start":"242'8600","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14499","last_scrub_stamp":"2024-04-29T14:16:55.139275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:55.139275+0000","objects_scrubbed":5375,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:11.621951+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81199104,"num_objects":1239,"num_object_clones":0,"num_object_copies":2478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18635,"num_write_kb":635968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"308'19059","reported_seq":19407,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:11.462662+0000","last_change":"2024-04-29T14:17:10.141724+0000","last_active":"2024-04-29T14:18:11.462662+0000","last_peered":"2024-04-29T14:18:11.462662+0000","last_clean":"2024-04-29T14:18:11.462662+0000","last_became_active":"2024-04-29T14:13:52.443412+0000","last_became_peered":"2024-04-29T14:13:52.443412+0000","last_unstale":"2024-04-29T14:18:11.462662+0000","last_undegraded":"2024-04-29T14:18:11.462662+0000","last_fullsized":"2024-04-29T14:18:11.462662+0000","mapping_epoch":213,"log_start":"244'9000","ondisk_log_start":"244'9000","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'15477","last_scrub_stamp":"2024-04-29T14:17:10.141460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:10.141460+0000","objects_scrubbed":4737,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:24.008289+0000","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73990144,"num_objects":1129,"num_object_clones":0,"num_object_copies":2258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19059,"num_write_kb":646016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"308'18704","reported_seq":33603,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.812137+0000","last_change":"2024-04-29T14:16:57.186933+0000","last_active":"2024-04-29T14:18:10.812137+0000","last_peered":"2024-04-29T14:18:10.812137+0000","last_clean":"2024-04-29T14:18:10.812137+0000","last_became_active":"2024-04-29T14:16:55.076352+0000","last_became_peered":"2024-04-29T14:16:55.076352+0000","last_unstale":"2024-04-29T14:18:10.812137+0000","last_undegraded":"2024-04-29T14:18:10.812137+0000","last_fullsized":"2024-04-29T14:18:10.812137+0000","mapping_epoch":277,"log_start":"243'8700","ondisk_log_start":"243'8700","created":206,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'14662","last_scrub_stamp":"2024-04-29T14:16:57.186759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:57.186759+0000","objects_scrubbed":5239,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:14.897452+0000","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76152832,"num_objects":1162,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18704,"num_write_kb":635712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7266,"num_bytes_recovered":476184576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"308'18680","reported_seq":19056,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.981161+0000","last_change":"2024-04-29T14:17:00.452407+0000","last_active":"2024-04-29T14:18:10.981161+0000","last_peered":"2024-04-29T14:18:10.981161+0000","last_clean":"2024-04-29T14:18:10.981161+0000","last_became_active":"2024-04-29T14:13:43.177667+0000","last_became_peered":"2024-04-29T14:13:43.177667+0000","last_unstale":"2024-04-29T14:18:10.981161+0000","last_undegraded":"2024-04-29T14:18:10.981161+0000","last_fullsized":"2024-04-29T14:18:10.981161+0000","mapping_epoch":206,"log_start":"242'8600","ondisk_log_start":"242'8600","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'14770","last_scrub_stamp":"2024-04-29T14:17:00.452314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:00.452314+0000","objects_scrubbed":5100,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:15.951850+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":2360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18680,"num_write_kb":635520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"308'18811","reported_seq":33160,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.230477+0000","last_change":"2024-04-29T14:17:02.479037+0000","last_active":"2024-04-29T14:18:10.230477+0000","last_peered":"2024-04-29T14:18:10.230477+0000","last_clean":"2024-04-29T14:18:10.230477+0000","last_became_active":"2024-04-29T14:17:01.115039+0000","last_became_peered":"2024-04-29T14:17:01.115039+0000","last_unstale":"2024-04-29T14:18:10.230477+0000","last_undegraded":"2024-04-29T14:18:10.230477+0000","last_fullsized":"2024-04-29T14:18:10.230477+0000","mapping_epoch":281,"log_start":"243'8800","ondisk_log_start":"243'8800","created":206,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'15028","last_scrub_stamp":"2024-04-29T14:17:02.478924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:02.478924+0000","objects_scrubbed":4994,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:17.935624+0000","scrub_duration":1057,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":2382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18811,"num_write_kb":640064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6991,"num_bytes_recovered":458162176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"308'18561","reported_seq":19000,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.260059+0000","last_change":"2024-04-29T14:17:06.899700+0000","last_active":"2024-04-29T14:18:10.260059+0000","last_peered":"2024-04-29T14:18:10.260059+0000","last_clean":"2024-04-29T14:18:10.260059+0000","last_became_active":"2024-04-29T14:13:52.443405+0000","last_became_peered":"2024-04-29T14:13:52.443405+0000","last_unstale":"2024-04-29T14:18:10.260059+0000","last_undegraded":"2024-04-29T14:18:10.260059+0000","last_fullsized":"2024-04-29T14:18:10.260059+0000","mapping_epoch":213,"log_start":"242'8500","ondisk_log_start":"242'8500","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'14978","last_scrub_stamp":"2024-04-29T14:17:06.899506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:06.899506+0000","objects_scrubbed":4791,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:16.520983+0000","scrub_duration":1380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75956224,"num_objects":1159,"num_object_clones":0,"num_object_copies":2318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18561,"num_write_kb":631040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"308'18666","reported_seq":19017,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:11.359800+0000","last_change":"2024-04-29T14:17:08.402938+0000","last_active":"2024-04-29T14:18:11.359800+0000","last_peered":"2024-04-29T14:18:11.359800+0000","last_clean":"2024-04-29T14:18:11.359800+0000","last_became_active":"2024-04-29T14:13:43.175151+0000","last_became_peered":"2024-04-29T14:13:43.175151+0000","last_unstale":"2024-04-29T14:18:11.359800+0000","last_undegraded":"2024-04-29T14:18:11.359800+0000","last_fullsized":"2024-04-29T14:18:11.359800+0000","mapping_epoch":206,"log_start":"242'8600","ondisk_log_start":"242'8600","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15120","last_scrub_stamp":"2024-04-29T14:17:08.402720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:08.402720+0000","objects_scrubbed":4696,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:25.140805+0000","scrub_duration":1243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18666,"num_write_kb":632832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"308'18723","reported_seq":19086,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.947556+0000","last_change":"2024-04-29T14:16:59.767604+0000","last_active":"2024-04-29T14:18:10.947556+0000","last_peered":"2024-04-29T14:18:10.947556+0000","last_clean":"2024-04-29T14:18:10.947556+0000","last_became_active":"2024-04-29T14:13:43.177685+0000","last_became_peered":"2024-04-29T14:13:43.177685+0000","last_unstale":"2024-04-29T14:18:10.947556+0000","last_undegraded":"2024-04-29T14:18:10.947556+0000","last_fullsized":"2024-04-29T14:18:10.947556+0000","mapping_epoch":206,"log_start":"243'8700","ondisk_log_start":"243'8700","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'14808","last_scrub_stamp":"2024-04-29T14:16:59.767446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:59.767446+0000","objects_scrubbed":5144,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:29.151682+0000","scrub_duration":813,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80412672,"num_objects":1227,"num_object_clones":0,"num_object_copies":2454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18723,"num_write_kb":638400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"308'18379","reported_seq":18782,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.396571+0000","last_change":"2024-04-29T14:17:46.110171+0000","last_active":"2024-04-29T14:18:10.396571+0000","last_peered":"2024-04-29T14:18:10.396571+0000","last_clean":"2024-04-29T14:18:10.396571+0000","last_became_active":"2024-04-29T14:13:52.443296+0000","last_became_peered":"2024-04-29T14:13:52.443296+0000","last_unstale":"2024-04-29T14:18:10.396571+0000","last_undegraded":"2024-04-29T14:18:10.396571+0000","last_fullsized":"2024-04-29T14:18:10.396571+0000","mapping_epoch":213,"log_start":"242'8300","ondisk_log_start":"242'8300","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'16907","last_scrub_stamp":"2024-04-29T14:17:46.109978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:46.109978+0000","objects_scrubbed":2660,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:55.498890+0000","scrub_duration":399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135872,"num_objects":1177,"num_object_clones":0,"num_object_copies":2354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18379,"num_write_kb":625792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2","version":"308'18914","reported_seq":42780,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:10.441231+0000","last_change":"2024-04-29T14:17:41.286770+0000","last_active":"2024-04-29T14:18:10.441231+0000","last_peered":"2024-04-29T14:18:10.441231+0000","last_clean":"2024-04-29T14:18:10.441231+0000","last_became_active":"2024-04-29T14:17:41.160640+0000","last_became_peered":"2024-04-29T14:17:41.160640+0000","last_unstale":"2024-04-29T14:18:10.441231+0000","last_undegraded":"2024-04-29T14:18:10.441231+0000","last_fullsized":"2024-04-29T14:18:10.441231+0000","mapping_epoch":297,"log_start":"244'8900","ondisk_log_start":"244'8900","created":206,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15183","last_scrub_stamp":"2024-04-29T14:17:07.144742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:07.144742+0000","objects_scrubbed":5024,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:19.677399+0000","scrub_duration":2083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80609280,"num_objects":1230,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18914,"num_write_kb":644608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11742,"num_bytes_recovered":769458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":807,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-29T14:18:09.083544+0000","last_change":"2024-04-29T14:17:01.936340+0000","last_active":"2024-04-29T14:18:09.083544+0000","last_peered":"2024-04-29T14:18:09.083544+0000","last_clean":"2024-04-29T14:18:09.083544+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:18:09.083544+0000","last_undegraded":"2024-04-29T14:18:09.083544+0000","last_fullsized":"2024-04-29T14:18:09.083544+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:17:01.936201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:17:01.936201+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:30.492263+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1226375192,"num_objects":18714,"num_object_clones":0,"num_object_copies":37428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":299168,"num_write_kb":10172161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41031,"num_bytes_recovered":2688942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3656523776,"data_stored":3656515632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160647,"ondisk_log_size":160647,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122183,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826956,"kb_used_data":520896,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92918196,"statfs":{"total":95995035648,"available":95148232704,"internally_reserved":0,"allocated":533397504,"data_stored":532929626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":313391556},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":126468,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618684,"statfs":{"total":95995035648,"available":95865532416,"internally_reserved":0,"allocated":909312,"data_stored":444986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":128580032},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389764,"kb_used_data":160056,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93355388,"statfs":{"total":95995035648,"available":95595917312,"internally_reserved":0,"allocated":163897344,"data_stored":163433038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":235207104},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877907095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205236,"kb_used_data":83820,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93539916,"statfs":{"total":95995035648,"available":95784873984,"internally_reserved":0,"allocated":85831680,"data_stored":85373688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234137,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542864,"kb_used_data":232644,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93202288,"statfs":{"total":95995035648,"available":95439142912,"internally_reserved":0,"allocated":238227456,"data_stored":237755526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":317651402},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528479,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692548,"kb_used_data":310136,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93052604,"statfs":{"total":95995035648,"available":95285866496,"internally_reserved":0,"allocated":317579264,"data_stored":317114958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":498496,"kb_used_data":300728,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93246656,"statfs":{"total":95995035648,"available":95484575744,"internally_reserved":0,"allocated":307945472,"data_stored":307481166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691496989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182575,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986346,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1084164,"kb_used_data":655928,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92660988,"statfs":{"total":95995035648,"available":94884851712,"internally_reserved":0,"allocated":671670272,"data_stored":671205966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822819,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541252,"kb_used_data":311420,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93203900,"statfs":{"total":95995035648,"available":95440793600,"internally_reserved":0,"allocated":318894080,"data_stored":318425702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306511872,"data_stored":306511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1067712512,"data_stored":1067712512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":664272896,"data_stored":664272896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":308744192,"data_stored":308740120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":314900480,"data_stored":314900480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":224464896,"data_stored":224460824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":81199104,"data_stored":81199104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":157548544,"data_stored":157548544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":531169280,"data_stored":531169280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:18:14.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:18:14.485 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:14.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:14.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:14.901 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:14.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:14.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:15.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:15.075 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:15.174 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:15 smithi040 ceph-mon[39581]: from='client.15504 v1:172.21.15.40:0/4002543611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:18:15.174 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:15 smithi040 ceph-mon[39581]: osdmap e310: 12 total, 12 up, 9 in 2024-04-29T14:18:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:15.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:15.281 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:15.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:15 smithi121 ceph-mon[44533]: from='client.15504 v1:172.21.15.40:0/4002543611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:18:15.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:15 smithi121 ceph-mon[44533]: osdmap e310: 12 total, 12 up, 9 in 2024-04-29T14:18:15.310 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:15.335 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:15.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:15.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:15.508 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:15 smithi194 ceph-mon[44089]: from='client.15504 v1:172.21.15.40:0/4002543611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:18:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:15 smithi194 ceph-mon[44089]: osdmap e310: 12 total, 12 up, 9 in 2024-04-29T14:18:15.571 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:15.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:15.652 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:18:15.652 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:18:15.652 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:18:15.653 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:18:15.653 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:18:15.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:15.728 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:15.894 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:15.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:16.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:16.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:16 smithi040 ceph-mon[39581]: pgmap v822: 17 pgs: 17 active+clean; 1.1 GiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 852 op/s 2024-04-29T14:18:16.356 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:16 smithi121 ceph-mon[44533]: pgmap v822: 17 pgs: 17 active+clean; 1.1 GiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 852 op/s 2024-04-29T14:18:16.380 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:16.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:16.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:18:16.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:16.551 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:16 smithi194 ceph-mon[44089]: pgmap v822: 17 pgs: 17 active+clean; 1.1 GiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 852 op/s 2024-04-29T14:18:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:16.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:16.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:16.784 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:16.820 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:16.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:16.933 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:16.962 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:16.983 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:16.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:17.056 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:18:17.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:17.112 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2153851314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:18:17.112 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:17 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:17.112 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2153851314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:18:17.112 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:17 smithi121 ceph-mon[44533]: osdmap e311: 12 total, 12 up, 9 in 2024-04-29T14:18:17.138 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:17.194 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:17.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:17.216 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:17.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:17.237 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:17.258 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:17.291 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2153851314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:18:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:17 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2153851314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:18:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:17 smithi194 ceph-mon[44089]: osdmap e311: 12 total, 12 up, 9 in 2024-04-29T14:18:17.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:17.325 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:17.356 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2153851314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:18:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:17 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2153851314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:18:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:17 smithi040 ceph-mon[39581]: osdmap e311: 12 total, 12 up, 9 in 2024-04-29T14:18:17.374 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:17.384 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:17.386 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:17.412 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:18:17.416 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:17.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:17.439 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:18:17.460 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:18:17.484 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:18:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:18:17.486 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:18:17.486 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:18:17.486 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:18:17.486 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:18:17.519 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:17.550 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:17.589 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:17.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:17.692 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:17.723 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:17.745 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:17.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:17.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:17.907 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:17.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:17.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:18.092 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:18.126 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:18.155 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:18.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:18.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: pgmap v823: 17 pgs: 17 active+clean; 871 MiB data, 4.1 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-29T14:18:18.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/189352269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:18.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.266 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: pgmap v823: 17 pgs: 17 active+clean; 871 MiB data, 4.1 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/189352269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.301 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:18.401 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:18.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:18.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: pgmap v823: 17 pgs: 17 active+clean; 871 MiB data, 4.1 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-29T14:18:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/189352269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:18.610 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:18.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:18.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:18.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:18.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:19.018 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:19.032 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:19.032 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:19.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:19.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:19.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:19.215 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:19.286 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:19.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:19.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:19.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:19.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:19.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:18:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:19.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:19.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:19.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:18:19.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:20.066 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:20.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:20.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:20.201 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:20.224 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:20.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:20.299 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:20.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:20.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:20 smithi121 ceph-mon[44533]: pgmap v825: 17 pgs: 17 active+clean; 871 MiB data, 4.1 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 715 op/s 2024-04-29T14:18:20.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:20 smithi121 ceph-mon[44533]: 3.5 scrub starts 2024-04-29T14:18:20.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:20 smithi121 ceph-mon[44533]: 3.5 scrub ok 2024-04-29T14:18:20.398 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:20.484 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:20 smithi194 ceph-mon[44089]: pgmap v825: 17 pgs: 17 active+clean; 871 MiB data, 4.1 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 715 op/s 2024-04-29T14:18:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:20 smithi194 ceph-mon[44089]: 3.5 scrub starts 2024-04-29T14:18:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:20 smithi194 ceph-mon[44089]: 3.5 scrub ok 2024-04-29T14:18:20.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:20 smithi040 ceph-mon[39581]: pgmap v825: 17 pgs: 17 active+clean; 871 MiB data, 4.1 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 715 op/s 2024-04-29T14:18:20.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:20 smithi040 ceph-mon[39581]: 3.5 scrub starts 2024-04-29T14:18:20.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:20 smithi040 ceph-mon[39581]: 3.5 scrub ok 2024-04-29T14:18:20.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:20.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:20.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:18:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:18:20.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:20.781 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:20.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:20.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:20.925 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:20.955 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:20.980 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:20.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:21.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:21.123 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:21.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:21 smithi194 ceph-mon[44089]: 3.9 scrub starts 2024-04-29T14:18:21.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:21 smithi194 ceph-mon[44089]: 3.9 scrub ok 2024-04-29T14:18:21.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:21 smithi194 ceph-mon[44089]: 3.f scrub starts 2024-04-29T14:18:21.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:21 smithi194 ceph-mon[44089]: 3.f scrub ok 2024-04-29T14:18:21.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:21 smithi040 ceph-mon[39581]: 3.9 scrub starts 2024-04-29T14:18:21.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:21 smithi040 ceph-mon[39581]: 3.9 scrub ok 2024-04-29T14:18:21.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:21 smithi040 ceph-mon[39581]: 3.f scrub starts 2024-04-29T14:18:21.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:21 smithi040 ceph-mon[39581]: 3.f scrub ok 2024-04-29T14:18:21.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:18:21.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:21 smithi121 ceph-mon[44533]: 3.9 scrub starts 2024-04-29T14:18:21.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:21 smithi121 ceph-mon[44533]: 3.9 scrub ok 2024-04-29T14:18:21.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:21 smithi121 ceph-mon[44533]: 3.f scrub starts 2024-04-29T14:18:21.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:21 smithi121 ceph-mon[44533]: 3.f scrub ok 2024-04-29T14:18:21.871 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":825,"stamp":"2024-04-29T14:18:18.748791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":913113656,"num_objects":13927,"num_object_clones":0,"num_object_copies":27854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13927,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":304138,"num_write_kb":10175099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41039,"num_bytes_recovered":2691057792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":8,"kb":843706368,"kb_used":4273416,"kb_used_data":1958760,"kb_used_omap":14,"kb_used_meta":2314353,"kb_avail":839432952,"statfs":{"total":863955320832,"available":859579342848,"internally_reserved":0,"allocated":2005770240,"data_stored":2001463346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14360,"internal_metadata":2369898472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,6,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-313851904,"num_objects":-4789,"num_object_clones":0,"num_object_copies":-9578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.695876"},"pg_stats":[{"pgid":"3.f","version":"310'20440","reported_seq":20909,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.806184+0000","last_change":"2024-04-29T14:17:07.892573+0000","last_active":"2024-04-29T14:18:15.806184+0000","last_peered":"2024-04-29T14:18:15.806184+0000","last_clean":"2024-04-29T14:18:15.806184+0000","last_became_active":"2024-04-29T14:13:43.169982+0000","last_became_peered":"2024-04-29T14:13:43.169982+0000","last_unstale":"2024-04-29T14:18:15.806184+0000","last_undegraded":"2024-04-29T14:18:15.806184+0000","last_fullsized":"2024-04-29T14:18:15.806184+0000","mapping_epoch":206,"log_start":"244'10401","ondisk_log_start":"244'10401","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16603","last_scrub_stamp":"2024-04-29T14:17:07.892271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:07.892271+0000","objects_scrubbed":4693,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:17.888993+0000","scrub_duration":1107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54001664,"num_objects":824,"num_object_clones":0,"num_object_copies":1648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18860,"num_write_kb":629888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"310'18704","reported_seq":19047,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:16.442157+0000","last_change":"2024-04-29T14:17:05.422450+0000","last_active":"2024-04-29T14:18:16.442157+0000","last_peered":"2024-04-29T14:18:16.442157+0000","last_clean":"2024-04-29T14:18:16.442157+0000","last_became_active":"2024-04-29T14:13:52.443531+0000","last_became_peered":"2024-04-29T14:13:52.443531+0000","last_unstale":"2024-04-29T14:18:16.442157+0000","last_undegraded":"2024-04-29T14:18:16.442157+0000","last_fullsized":"2024-04-29T14:18:16.442157+0000","mapping_epoch":213,"log_start":"244'8700","ondisk_log_start":"244'8700","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14769","last_scrub_stamp":"2024-04-29T14:17:05.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:05.422201+0000","objects_scrubbed":4768,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:17.169825+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52297728,"num_objects":798,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18704,"num_write_kb":624064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"310'19206","reported_seq":19574,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.769178+0000","last_change":"2024-04-29T14:16:59.668689+0000","last_active":"2024-04-29T14:18:15.769178+0000","last_peered":"2024-04-29T14:18:15.769178+0000","last_clean":"2024-04-29T14:18:15.769178+0000","last_became_active":"2024-04-29T14:13:43.174780+0000","last_became_peered":"2024-04-29T14:13:43.174780+0000","last_unstale":"2024-04-29T14:18:15.769178+0000","last_undegraded":"2024-04-29T14:18:15.769178+0000","last_fullsized":"2024-04-29T14:18:15.769178+0000","mapping_epoch":206,"log_start":"245'9200","ondisk_log_start":"245'9200","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'14954","last_scrub_stamp":"2024-04-29T14:16:59.668535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:59.668535+0000","objects_scrubbed":5116,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:23.894891+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56295448,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19206,"num_write_kb":642049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"310'18994","reported_seq":33195,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.229812+0000","last_change":"2024-04-29T14:17:05.994990+0000","last_active":"2024-04-29T14:18:15.229812+0000","last_peered":"2024-04-29T14:18:15.229812+0000","last_clean":"2024-04-29T14:18:15.229812+0000","last_became_active":"2024-04-29T14:17:01.115520+0000","last_became_peered":"2024-04-29T14:17:01.115520+0000","last_unstale":"2024-04-29T14:18:15.229812+0000","last_undegraded":"2024-04-29T14:18:15.229812+0000","last_fullsized":"2024-04-29T14:18:15.229812+0000","mapping_epoch":281,"log_start":"244'8900","ondisk_log_start":"244'8900","created":206,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15016","last_scrub_stamp":"2024-04-29T14:17:05.994807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:05.994807+0000","objects_scrubbed":4932,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:15.693149+0000","scrub_duration":1542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59113472,"num_objects":902,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18994,"num_write_kb":636672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6911,"num_bytes_recovered":452919296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"310'18893","reported_seq":33194,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.945721+0000","last_change":"2024-04-29T14:17:03.835268+0000","last_active":"2024-04-29T14:18:15.945721+0000","last_peered":"2024-04-29T14:18:15.945721+0000","last_clean":"2024-04-29T14:18:15.945721+0000","last_became_active":"2024-04-29T14:17:00.109521+0000","last_became_peered":"2024-04-29T14:17:00.109521+0000","last_unstale":"2024-04-29T14:18:15.945721+0000","last_undegraded":"2024-04-29T14:18:15.945721+0000","last_fullsized":"2024-04-29T14:18:15.945721+0000","mapping_epoch":280,"log_start":"244'8800","ondisk_log_start":"244'8800","created":206,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'14843","last_scrub_stamp":"2024-04-29T14:17:03.835159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:03.835159+0000","objects_scrubbed":4913,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:32.049771+0000","scrub_duration":815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55115776,"num_objects":841,"num_object_clones":0,"num_object_copies":1682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18893,"num_write_kb":631488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6962,"num_bytes_recovered":456261632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"310'19150","reported_seq":21834,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.439594+0000","last_change":"2024-04-29T14:17:06.021161+0000","last_active":"2024-04-29T14:18:15.439594+0000","last_peered":"2024-04-29T14:18:15.439594+0000","last_clean":"2024-04-29T14:18:15.439594+0000","last_became_active":"2024-04-29T14:14:17.997407+0000","last_became_peered":"2024-04-29T14:14:17.997407+0000","last_unstale":"2024-04-29T14:18:15.439594+0000","last_undegraded":"2024-04-29T14:18:15.439594+0000","last_fullsized":"2024-04-29T14:18:15.439594+0000","mapping_epoch":224,"log_start":"244'9100","ondisk_log_start":"244'9100","created":206,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15095","last_scrub_stamp":"2024-04-29T14:17:06.020918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:06.020918+0000","objects_scrubbed":5028,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:28.724750+0000","scrub_duration":2033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59899904,"num_objects":914,"num_object_clones":0,"num_object_copies":1828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19150,"num_write_kb":642048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1159,"num_bytes_recovered":75956224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"310'18922","reported_seq":19309,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:14.410862+0000","last_change":"2024-04-29T14:16:55.139438+0000","last_active":"2024-04-29T14:18:14.410862+0000","last_peered":"2024-04-29T14:18:14.410862+0000","last_clean":"2024-04-29T14:18:14.410862+0000","last_became_active":"2024-04-29T14:13:43.170268+0000","last_became_peered":"2024-04-29T14:13:43.170268+0000","last_unstale":"2024-04-29T14:18:14.410862+0000","last_undegraded":"2024-04-29T14:18:14.410862+0000","last_fullsized":"2024-04-29T14:18:14.410862+0000","mapping_epoch":206,"log_start":"244'8900","ondisk_log_start":"244'8900","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14499","last_scrub_stamp":"2024-04-29T14:16:55.139275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:55.139275+0000","objects_scrubbed":5375,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:16.646435+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62390272,"num_objects":952,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18922,"num_write_kb":635968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"310'19358","reported_seq":19710,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:16.477971+0000","last_change":"2024-04-29T14:17:10.141724+0000","last_active":"2024-04-29T14:18:16.477971+0000","last_peered":"2024-04-29T14:18:16.477971+0000","last_clean":"2024-04-29T14:18:16.477971+0000","last_became_active":"2024-04-29T14:13:52.443412+0000","last_became_peered":"2024-04-29T14:13:52.443412+0000","last_unstale":"2024-04-29T14:18:16.477971+0000","last_undegraded":"2024-04-29T14:18:16.477971+0000","last_fullsized":"2024-04-29T14:18:16.477971+0000","mapping_epoch":213,"log_start":"246'9300","ondisk_log_start":"246'9300","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'15477","last_scrub_stamp":"2024-04-29T14:17:10.141460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:10.141460+0000","objects_scrubbed":4737,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:24.008289+0000","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54394880,"num_objects":830,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19358,"num_write_kb":646016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"310'19010","reported_seq":33912,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.795131+0000","last_change":"2024-04-29T14:16:57.186933+0000","last_active":"2024-04-29T14:18:15.795131+0000","last_peered":"2024-04-29T14:18:15.795131+0000","last_clean":"2024-04-29T14:18:15.795131+0000","last_became_active":"2024-04-29T14:16:55.076352+0000","last_became_peered":"2024-04-29T14:16:55.076352+0000","last_unstale":"2024-04-29T14:18:15.795131+0000","last_undegraded":"2024-04-29T14:18:15.795131+0000","last_fullsized":"2024-04-29T14:18:15.795131+0000","mapping_epoch":277,"log_start":"244'9000","ondisk_log_start":"244'9000","created":206,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'14662","last_scrub_stamp":"2024-04-29T14:16:57.186759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:57.186759+0000","objects_scrubbed":5239,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56098816,"num_objects":856,"num_object_clones":0,"num_object_copies":1712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19010,"num_write_kb":635712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7266,"num_bytes_recovered":476184576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"310'18987","reported_seq":19366,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.995347+0000","last_change":"2024-04-29T14:17:00.452407+0000","last_active":"2024-04-29T14:18:15.995347+0000","last_peered":"2024-04-29T14:18:15.995347+0000","last_clean":"2024-04-29T14:18:15.995347+0000","last_became_active":"2024-04-29T14:13:43.177667+0000","last_became_peered":"2024-04-29T14:13:43.177667+0000","last_unstale":"2024-04-29T14:18:15.995347+0000","last_undegraded":"2024-04-29T14:18:15.995347+0000","last_fullsized":"2024-04-29T14:18:15.995347+0000","mapping_epoch":206,"log_start":"244'8900","ondisk_log_start":"244'8900","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'14770","last_scrub_stamp":"2024-04-29T14:17:00.452314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:00.452314+0000","objects_scrubbed":5100,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57212928,"num_objects":873,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18987,"num_write_kb":635520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"310'19095","reported_seq":33448,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.282934+0000","last_change":"2024-04-29T14:17:02.479037+0000","last_active":"2024-04-29T14:18:15.282934+0000","last_peered":"2024-04-29T14:18:15.282934+0000","last_clean":"2024-04-29T14:18:15.282934+0000","last_became_active":"2024-04-29T14:17:01.115039+0000","last_became_peered":"2024-04-29T14:17:01.115039+0000","last_unstale":"2024-04-29T14:18:15.282934+0000","last_undegraded":"2024-04-29T14:18:15.282934+0000","last_fullsized":"2024-04-29T14:18:15.282934+0000","mapping_epoch":281,"log_start":"244'9000","ondisk_log_start":"244'9000","created":206,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'15028","last_scrub_stamp":"2024-04-29T14:17:02.478924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:02.478924+0000","objects_scrubbed":4994,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:17.935624+0000","scrub_duration":1057,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59441152,"num_objects":907,"num_object_clones":0,"num_object_copies":1814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19095,"num_write_kb":640064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6991,"num_bytes_recovered":458162176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"310'18840","reported_seq":19283,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.241758+0000","last_change":"2024-04-29T14:17:06.899700+0000","last_active":"2024-04-29T14:18:15.241758+0000","last_peered":"2024-04-29T14:18:15.241758+0000","last_clean":"2024-04-29T14:18:15.241758+0000","last_became_active":"2024-04-29T14:13:52.443405+0000","last_became_peered":"2024-04-29T14:13:52.443405+0000","last_unstale":"2024-04-29T14:18:15.241758+0000","last_undegraded":"2024-04-29T14:18:15.241758+0000","last_fullsized":"2024-04-29T14:18:15.241758+0000","mapping_epoch":213,"log_start":"244'8800","ondisk_log_start":"244'8800","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'14978","last_scrub_stamp":"2024-04-29T14:17:06.899506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:06.899506+0000","objects_scrubbed":4791,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:16.520983+0000","scrub_duration":1380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57671680,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18840,"num_write_kb":631040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"310'18961","reported_seq":19316,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:16.414875+0000","last_change":"2024-04-29T14:17:08.402938+0000","last_active":"2024-04-29T14:18:16.414875+0000","last_peered":"2024-04-29T14:18:16.414875+0000","last_clean":"2024-04-29T14:18:16.414875+0000","last_became_active":"2024-04-29T14:13:43.175151+0000","last_became_peered":"2024-04-29T14:13:43.175151+0000","last_unstale":"2024-04-29T14:18:16.414875+0000","last_undegraded":"2024-04-29T14:18:16.414875+0000","last_fullsized":"2024-04-29T14:18:16.414875+0000","mapping_epoch":206,"log_start":"244'8900","ondisk_log_start":"244'8900","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15120","last_scrub_stamp":"2024-04-29T14:17:08.402720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:08.402720+0000","objects_scrubbed":4696,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:25.140805+0000","scrub_duration":1243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":1630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18961,"num_write_kb":632832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"310'19045","reported_seq":19412,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.928062+0000","last_change":"2024-04-29T14:16:59.767604+0000","last_active":"2024-04-29T14:18:15.928062+0000","last_peered":"2024-04-29T14:18:15.928062+0000","last_clean":"2024-04-29T14:18:15.928062+0000","last_became_active":"2024-04-29T14:13:43.177685+0000","last_became_peered":"2024-04-29T14:13:43.177685+0000","last_unstale":"2024-04-29T14:18:15.928062+0000","last_undegraded":"2024-04-29T14:18:15.928062+0000","last_fullsized":"2024-04-29T14:18:15.928062+0000","mapping_epoch":206,"log_start":"244'9000","ondisk_log_start":"244'9000","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'14808","last_scrub_stamp":"2024-04-29T14:16:59.767446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:16:59.767446+0000","objects_scrubbed":5144,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:29.151682+0000","scrub_duration":813,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":1810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19045,"num_write_kb":638400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"310'18681","reported_seq":19088,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.450000+0000","last_change":"2024-04-29T14:17:46.110171+0000","last_active":"2024-04-29T14:18:15.450000+0000","last_peered":"2024-04-29T14:18:15.450000+0000","last_clean":"2024-04-29T14:18:15.450000+0000","last_became_active":"2024-04-29T14:13:52.443296+0000","last_became_peered":"2024-04-29T14:13:52.443296+0000","last_unstale":"2024-04-29T14:18:15.450000+0000","last_undegraded":"2024-04-29T14:18:15.450000+0000","last_fullsized":"2024-04-29T14:18:15.450000+0000","mapping_epoch":213,"log_start":"243'8600","ondisk_log_start":"243'8600","created":206,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'16907","last_scrub_stamp":"2024-04-29T14:17:46.109978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:46.109978+0000","objects_scrubbed":2660,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:55.498890+0000","scrub_duration":399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":1750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18681,"num_write_kb":625792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2","version":"310'19251","reported_seq":43121,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:15.433868+0000","last_change":"2024-04-29T14:17:41.286770+0000","last_active":"2024-04-29T14:18:15.433868+0000","last_peered":"2024-04-29T14:18:15.433868+0000","last_clean":"2024-04-29T14:18:15.433868+0000","last_became_active":"2024-04-29T14:17:41.160640+0000","last_became_peered":"2024-04-29T14:17:41.160640+0000","last_unstale":"2024-04-29T14:18:15.433868+0000","last_undegraded":"2024-04-29T14:18:15.433868+0000","last_fullsized":"2024-04-29T14:18:15.433868+0000","mapping_epoch":297,"log_start":"245'9200","ondisk_log_start":"245'9200","created":206,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15183","last_scrub_stamp":"2024-04-29T14:17:07.144742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:13:42.154997+0000","last_clean_scrub_stamp":"2024-04-29T14:17:07.144742+0000","objects_scrubbed":5024,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:19.677399+0000","scrub_duration":2083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58523648,"num_objects":893,"num_object_clones":0,"num_object_copies":1786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19251,"num_write_kb":644608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11742,"num_bytes_recovered":769458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":811,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-29T14:18:14.110697+0000","last_change":"2024-04-29T14:17:01.936340+0000","last_active":"2024-04-29T14:18:14.110697+0000","last_peered":"2024-04-29T14:18:14.110697+0000","last_clean":"2024-04-29T14:18:14.110697+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:18:14.110697+0000","last_undegraded":"2024-04-29T14:18:14.110697+0000","last_fullsized":"2024-04-29T14:18:14.110697+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:17:01.936201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:17:01.936201+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:18:30.492263+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":912523288,"num_objects":13925,"num_object_clones":0,"num_object_copies":27850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":303957,"num_write_kb":10172161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41031,"num_bytes_recovered":2688942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2922586112,"data_stored":2922577968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160836,"ondisk_log_size":160836,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122184,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":689764,"kb_used_data":392024,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93055388,"statfs":{"total":95995035648,"available":95288717312,"internally_reserved":0,"allocated":401432576,"data_stored":400952170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":304871876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383821,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":126504,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618648,"statfs":{"total":95995035648,"available":95865495552,"internally_reserved":0,"allocated":946176,"data_stored":463008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":128580032},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645458,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349272,"kb_used_data":119568,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93395880,"statfs":{"total":95995035648,"available":95637381120,"internally_reserved":0,"allocated":122437632,"data_stored":121960798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":235207104},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":64,"seq":274877907096,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186060,"kb_used_data":64644,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93559092,"statfs":{"total":95995035648,"available":95804510208,"internally_reserved":0,"allocated":66195456,"data_stored":65724836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234138,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471784,"kb_used_data":173980,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93273368,"statfs":{"total":95995035648,"available":95511928832,"internally_reserved":0,"allocated":178155520,"data_stored":177671062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":304871882},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528480,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":602972,"kb_used_data":233040,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93142180,"statfs":{"total":95995035648,"available":95377592320,"internally_reserved":0,"allocated":238632960,"data_stored":238156126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":378796486},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424412,"kb_used_data":222480,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93320740,"statfs":{"total":95995035648,"available":95560437760,"internally_reserved":0,"allocated":227819520,"data_stored":227342686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691496990,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182577,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950808,"kb_used_data":514192,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92794344,"statfs":{"total":95995035648,"available":95021408256,"internally_reserved":0,"allocated":526532608,"data_stored":526055774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822820,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471840,"kb_used_data":237908,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":93273312,"statfs":{"total":95995035648,"available":95511871488,"internally_reserved":0,"allocated":243617792,"data_stored":243136886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226295808,"data_stored":226295808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":969408512,"data_stored":969408512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":514916352,"data_stored":514916352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":233771008,"data_stored":233766936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":235667456,"data_stored":235667456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":166400000,"data_stored":166395928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":62390272,"data_stored":62390272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":115802112,"data_stored":115802112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":397934592,"data_stored":397934592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:18:21.872 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:18:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:21.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:21.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:22.046 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:22.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:22.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:22.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:22.247 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:22.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:22.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:22 smithi121 ceph-mon[44533]: from='client.25444 v1:172.21.15.40:0/2999116955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:18:22.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:22 smithi121 ceph-mon[44533]: pgmap v826: 17 pgs: 17 active+clean; 745 MiB data, 3.8 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 849 op/s 2024-04-29T14:18:22.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:22 smithi121 ceph-mon[44533]: 3.e scrub starts 2024-04-29T14:18:22.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:22 smithi121 ceph-mon[44533]: 3.e scrub ok 2024-04-29T14:18:22.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:22 smithi121 ceph-mon[44533]: 3.c scrub starts 2024-04-29T14:18:22.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:22 smithi121 ceph-mon[44533]: 3.c scrub ok 2024-04-29T14:18:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:22.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:22.401 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:22.401 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:22.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:22.533 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:22.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:22 smithi194 ceph-mon[44089]: from='client.25444 v1:172.21.15.40:0/2999116955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:18:22.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:22 smithi194 ceph-mon[44089]: pgmap v826: 17 pgs: 17 active+clean; 745 MiB data, 3.8 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 849 op/s 2024-04-29T14:18:22.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:22 smithi194 ceph-mon[44089]: 3.e scrub starts 2024-04-29T14:18:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:22 smithi194 ceph-mon[44089]: 3.e scrub ok 2024-04-29T14:18:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:22 smithi194 ceph-mon[44089]: 3.c scrub starts 2024-04-29T14:18:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:22 smithi194 ceph-mon[44089]: 3.c scrub ok 2024-04-29T14:18:22.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:22 smithi040 ceph-mon[39581]: from='client.25444 v1:172.21.15.40:0/2999116955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:18:22.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:22 smithi040 ceph-mon[39581]: pgmap v826: 17 pgs: 17 active+clean; 745 MiB data, 3.8 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 849 op/s 2024-04-29T14:18:22.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:22 smithi040 ceph-mon[39581]: 3.e scrub starts 2024-04-29T14:18:22.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:22 smithi040 ceph-mon[39581]: 3.e scrub ok 2024-04-29T14:18:22.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:22 smithi040 ceph-mon[39581]: 3.c scrub starts 2024-04-29T14:18:22.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:22 smithi040 ceph-mon[39581]: 3.c scrub ok 2024-04-29T14:18:22.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:22.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:22.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:22.874 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:23.162 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:23.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:23.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:18:23.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:23.303 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:23.324 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:23.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:23.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:23.458 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:23 smithi194 ceph-mon[44089]: 3.7 scrub starts 2024-04-29T14:18:23.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:23 smithi194 ceph-mon[44089]: 3.2 scrub starts 2024-04-29T14:18:23.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:23 smithi194 ceph-mon[44089]: 3.2 scrub ok 2024-04-29T14:18:23.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:23 smithi194 ceph-mon[44089]: 3.7 scrub ok 2024-04-29T14:18:23.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:18:23.461 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:23.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:23 smithi040 ceph-mon[39581]: 3.7 scrub starts 2024-04-29T14:18:23.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:23 smithi040 ceph-mon[39581]: 3.2 scrub starts 2024-04-29T14:18:23.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:23 smithi040 ceph-mon[39581]: 3.2 scrub ok 2024-04-29T14:18:23.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:23 smithi040 ceph-mon[39581]: 3.7 scrub ok 2024-04-29T14:18:23.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:18:23.487 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:23.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:23.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:23.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:23 smithi121 ceph-mon[44533]: 3.7 scrub starts 2024-04-29T14:18:23.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:23 smithi121 ceph-mon[44533]: 3.2 scrub starts 2024-04-29T14:18:23.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:23 smithi121 ceph-mon[44533]: 3.2 scrub ok 2024-04-29T14:18:23.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:23 smithi121 ceph-mon[44533]: 3.7 scrub ok 2024-04-29T14:18:23.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:18:23.669 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:23.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:23.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:23.876 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:24.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:24.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:24.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:24.112 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:24.140 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:24.165 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:24.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:24.205 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:18:24.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:24.302 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:24.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:24 smithi121 ceph-mon[44533]: pgmap v827: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:18:24.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/206387919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:24.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:18:24.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:18:24.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:18:24.326 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:24.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:24.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:24 smithi040 ceph-mon[39581]: pgmap v827: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:18:24.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/206387919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:24.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:18:24.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:18:24.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:18:24.427 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:24 smithi194 ceph-mon[44089]: pgmap v827: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:18:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/206387919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:18:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:18:24.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:18:24.458 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:24.481 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:24.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:24.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:24.658 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:24.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:24.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:24.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:24.903 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:24.923 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:24.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:25.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:25.061 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:25.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:25.114 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:25.135 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:25.156 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:25.178 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:25.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:25.210 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:25.233 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:25.240 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:25.268 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:25.297 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:25.322 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:18:25.344 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: pgmap v828: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: pgmap v829: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 655 op/s 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/206387919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: osdmap e312: 12 total, 12 up, 9 in 2024-04-29T14:18:25.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3190793298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:25.365 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: pgmap v828: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: pgmap v829: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 655 op/s 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/206387919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: osdmap e312: 12 total, 12 up, 9 in 2024-04-29T14:18:25.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3190793298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:25.385 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:18:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:18:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:18:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:18:25.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:25.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:25.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:25.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:25.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: pgmap v828: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:18:25.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: pgmap v829: 17 pgs: 17 active+clean; 585 MiB data, 3.5 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 655 op/s 2024-04-29T14:18:25.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:18:25.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/206387919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:25.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: osdmap e312: 12 total, 12 up, 9 in 2024-04-29T14:18:25.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3190793298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:25.572 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:25.601 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:25.624 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:25.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:25.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:25.812 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:25.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:25.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:25.842 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:18:25.842 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:18:25.842 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:18:25.842 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:18:25.842 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:18:25.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:25.965 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:26.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:26.236 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:26.263 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:26.286 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:26.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:26.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: pgmap v831: 17 pgs: 17 active+clean; 458 MiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 946 op/s 2024-04-29T14:18:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:26 smithi040 ceph-mon[39581]: 3.4 deep-scrub starts 2024-04-29T14:18:26.391 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:26.419 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:26.441 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:26.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:26.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: pgmap v831: 17 pgs: 17 active+clean; 458 MiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 946 op/s 2024-04-29T14:18:26.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:26 smithi194 ceph-mon[44089]: 3.4 deep-scrub starts 2024-04-29T14:18:26.567 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.567 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.567 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.567 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.567 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.567 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: pgmap v831: 17 pgs: 17 active+clean; 458 MiB data, 3.2 GiB used, 801 GiB / 805 GiB avail; 0 B/s wr, 946 op/s 2024-04-29T14:18:26.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:26 smithi121 ceph-mon[44533]: 3.4 deep-scrub starts 2024-04-29T14:18:26.575 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:26.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:26.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:18:26.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:26.762 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:26.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:26.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:26.907 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:26.936 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:26.960 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:26.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:26.997 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:27.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:27.100 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:27.122 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:27.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:27.213 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:27 smithi194 ceph-mon[44089]: 3.6 scrub starts 2024-04-29T14:18:27.214 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:27 smithi194 ceph-mon[44089]: 3.6 scrub ok 2024-04-29T14:18:27.214 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:27 smithi194 ceph-mon[44089]: osdmap e313: 12 total, 12 up, 9 in 2024-04-29T14:18:27.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:27.254 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:27.274 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:27.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:27.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:27.405 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:27.425 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:27.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:27.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:27.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:27 smithi040 ceph-mon[39581]: 3.6 scrub starts 2024-04-29T14:18:27.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:27 smithi040 ceph-mon[39581]: 3.6 scrub ok 2024-04-29T14:18:27.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:27 smithi040 ceph-mon[39581]: osdmap e313: 12 total, 12 up, 9 in 2024-04-29T14:18:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:27.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:27 smithi121 ceph-mon[44533]: 3.6 scrub starts 2024-04-29T14:18:27.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:27 smithi121 ceph-mon[44533]: 3.6 scrub ok 2024-04-29T14:18:27.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:27 smithi121 ceph-mon[44533]: osdmap e313: 12 total, 12 up, 9 in 2024-04-29T14:18:27.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:27.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:27.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:27.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:18:27.716 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:27.738 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:27.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:27.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:27.870 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:27.890 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:27.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:27.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:28.276 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:28.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:28.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:28.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:28 smithi040 ceph-mon[39581]: pgmap v833: 17 pgs: 17 active+clean; 298 MiB data, 3.0 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 921 op/s 2024-04-29T14:18:28.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:28 smithi040 ceph-mon[39581]: 3.4 deep-scrub ok 2024-04-29T14:18:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:28.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:28.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:28.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:28.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:28 smithi121 ceph-mon[44533]: pgmap v833: 17 pgs: 17 active+clean; 298 MiB data, 3.0 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 921 op/s 2024-04-29T14:18:28.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:28 smithi121 ceph-mon[44533]: 3.4 deep-scrub ok 2024-04-29T14:18:28.381 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:28.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:18:28.409 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:28.431 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:28.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:28.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:28 smithi194 ceph-mon[44089]: pgmap v833: 17 pgs: 17 active+clean; 298 MiB data, 3.0 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 921 op/s 2024-04-29T14:18:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:28 smithi194 ceph-mon[44089]: 3.4 deep-scrub ok 2024-04-29T14:18:28.566 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:28.920 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:28.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:29.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:29.052 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:29.061 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:18:29.061 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:18:29.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:18:29.061 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:18:29.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:166439 2.11 2.c7b87731 (undecoded) ondisk+write+known_if_redirected e139)", 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:10:56.243836+0000", 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 452.82194306500003, 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66351022999999998, 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 166439 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243836+0000", 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243836+0000", 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243838+0000", 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4190000000000001e-06 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243842+0000", 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.143e-06 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243853+0000", 2024-04-29T14:18:29.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.064e-05 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243861+0000", 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3100000000000001e-06 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243902+0000", 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1468999999999999e-05 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.243908+0000", 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3310000000000003e-06 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.907335+0000", 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66342681599999997 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:10:56.907347+0000", 2024-04-29T14:18:29.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2102e-05 2024-04-29T14:18:29.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.067 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.067 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173610 2.17 2:ea6ae847:::benchmark_data_smithi194_79000_object9349:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 440.93000240100002, 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.234702578, 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173610 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135779+0000", 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2069999999999998e-06 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135785+0000", 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5849999999999999e-06 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135800+0000", 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5573000000000002e-05 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135809+0000", 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2870000000000004e-06 2024-04-29T14:18:29.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.170786+0000", 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034977716999999998 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.170861+0000", 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4418000000000005e-05 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.171114+0000", 2024-04-29T14:18:29.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025352699999999997 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184325+0000", 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01321051 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370366+0000", 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.186041027 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370424+0000", 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7645999999999997e-05 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370480+0000", 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6081000000000001e-05 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173617 2.17 2:ea4c2311:::benchmark_data_smithi194_79000_object9356:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:18:29.073 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 440.92104191099997, 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24862530199999999, 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173617 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:18:29.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144739+0000", 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8789999999999999e-06 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144745+0000", 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7599999999999999e-06 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144755+0000", 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7969999999999995e-06 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144761+0000", 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1140000000000002e-06 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.173690+0000", 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028928834 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.173816+0000", 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126067 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.174276+0000", 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045987699999999997 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184330+0000", 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010054110999999999 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393250+0000", 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208919664 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393302+0000", 2024-04-29T14:18:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1860999999999997e-05 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393363+0000", 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1338000000000001e-05 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173625 2.17 2:ed81edb2:::benchmark_data_smithi194_79000_object9364:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 440.91736565000002, 2024-04-29T14:18:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24512054, 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173625 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148416+0000", 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.894e-06 2024-04-29T14:18:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148421+0000", 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5319999999999997e-06 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148435+0000", 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3431000000000001e-05 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148443+0000", 2024-04-29T14:18:29.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6719999999999993e-06 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.177311+0000", 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028867127999999999 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.177432+0000", 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121236 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:18:29.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.178047+0000", 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061545999999999997 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187188+0000", 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091407750000000003 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393432+0000", 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20624394500000001 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393478+0000", 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6011999999999999e-05 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393534+0000", 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6455000000000001e-05 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.085 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173632 2.17 2:ebef1b31:::benchmark_data_smithi194_79000_object9371:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 440.90638688299998, 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23825384199999999, 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173632 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159395+0000", 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0549999999999998e-06 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159401+0000", 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0569999999999998e-06 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159413+0000", 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2347999999999999e-05 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159422+0000", 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9299999999999992e-06 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180545+0000", 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021122809999999999 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180656+0000", 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011134 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181142+0000", 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000486031 2024-04-29T14:18:29.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187196+0000", 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060541850000000001 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397527+0000", 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21033081000000001 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397582+0000", 2024-04-29T14:18:29.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5235999999999998e-05 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397646+0000", 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4040000000000003e-05 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173636 2.7 2:e5bc6543:::benchmark_data_smithi194_79000_object9375:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 440.90572743899997, 2024-04-29T14:18:29.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23324684400000001, 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173636 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160053+0000", 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.294e-06 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160058+0000", 2024-04-29T14:18:29.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4100000000000001e-06 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160068+0000", 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7569999999999999e-06 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160073+0000", 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8810000000000001e-06 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180528+0000", 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020454898999999999 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180612+0000", 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3422000000000002e-05 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181050+0000", 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000438391 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187192+0000", 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061421150000000001 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393229+0000", 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20603648399999999 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393264+0000", 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5549999999999997e-05 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393299+0000", 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4641000000000003e-05 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173640 2.7 2:e581d5ea:::benchmark_data_smithi194_79000_object9379:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 440.90173543700001, 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22945062699999999, 2024-04-29T14:18:29.097 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173640 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164046+0000", 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1409999999999999e-06 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164056+0000", 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008e-05 2024-04-29T14:18:29.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164100+0000", 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3893000000000001e-05 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164109+0000", 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6889999999999993e-06 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184149+0000", 2024-04-29T14:18:29.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020040235999999999 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184270+0000", 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120791 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184634+0000", 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036420799999999998 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187199+0000", 2024-04-29T14:18:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025650199999999999 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393415+0000", 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20621562099999999 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393462+0000", 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7055999999999998e-05 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393495+0000", 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2963999999999998e-05 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173648 2.17 2:eb9a7570:::benchmark_data_smithi194_79000_object9387:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 440.88445780400002, 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26978428700000001, 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.103 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173648 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181324+0000", 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.874e-06 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181330+0000", 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2210000000000002e-06 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181342+0000", 2024-04-29T14:18:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2e-05 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181350+0000", 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.106e-06 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184353+0000", 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030029319999999998 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184447+0000", 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3710999999999998e-05 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184955+0000", 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050829300000000005 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187337+0000", 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00238265 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451020+0000", 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.263682898 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451050+0000", 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9578999999999999e-05 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451106+0000", 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6023000000000002e-05 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164478 3.2 3:4df1a902:::benchmark_data_smithi194_88551_object5535:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055547+0000", 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 169.01023234499999, 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20835846799999999, 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164478 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055547+0000", 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055547+0000", 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055550+0000", 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.875e-06 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055562+0000", 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2098e-05 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055574+0000", 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2119999999999999e-05 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055587+0000", 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.238e-05 2024-04-29T14:18:29.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055646+0000", 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9735000000000003e-05 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055659+0000", 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2143e-05 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.263814+0000", 2024-04-29T14:18:29.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20815571699999999 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.263906+0000", 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1399999999999999e-05 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164479 3.2 3:4e25dff6:::benchmark_data_smithi194_88551_object5536:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055608+0000", 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 169.01017119799999, 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20862729099999999, 2024-04-29T14:18:29.113 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164479 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055608+0000", 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055608+0000", 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055609+0000", 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1200000000000002e-07 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055612+0000", 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3230000000000002e-06 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055619+0000", 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7190000000000004e-06 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055625+0000", 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2190000000000001e-06 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055687+0000", 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1450000000000003e-05 2024-04-29T14:18:29.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055694+0000", 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1520000000000003e-06 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264161+0000", 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208467175 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264236+0000", 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4541000000000003e-05 2024-04-29T14:18:29.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.118 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.118 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164489 3.2 3:45db8f19:::benchmark_data_smithi194_88551_object5546:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:18:29.118 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055633+0000", 2024-04-29T14:18:29.118 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 169.01014665599999, 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20871237500000001, 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164489 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055633+0000", 2024-04-29T14:18:29.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055633+0000", 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055634+0000", 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7899999999999998e-07 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055636+0000", 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3709999999999998e-06 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055641+0000", 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7899999999999999e-06 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055646+0000", 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3800000000000002e-06 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055721+0000", 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4852999999999996e-05 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055728+0000", 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9009999999999997e-06 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264274+0000", 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208546641 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264345+0000", 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0759999999999993e-05 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.123 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164495 3.2 3:4824d7da:::benchmark_data_smithi194_88551_object5552:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055652+0000", 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 169.01012737299999, 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20880806800000001, 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.124 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164495 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055652+0000", 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055652+0000", 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055653+0000", 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5000000000000003e-07 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055655+0000", 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4449999999999999e-06 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055662+0000", 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7719999999999997e-06 2024-04-29T14:18:29.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055667+0000", 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3979999999999998e-06 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055778+0000", 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110528 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055787+0000", 2024-04-29T14:18:29.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5559999999999996e-06 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264382+0000", 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208594639 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264460+0000", 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.818e-05 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164502 3.2 3:4a6173d7:::benchmark_data_smithi194_88551_object5559:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055680+0000", 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 169.01009903900001, 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23313735499999999, 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164502 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055680+0000", 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055680+0000", 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055681+0000", 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8599999999999998e-07 2024-04-29T14:18:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055692+0000", 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.084e-05 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055698+0000", 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.28e-06 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055704+0000", 2024-04-29T14:18:29.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8370000000000002e-06 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055832+0000", 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012847100000000001 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055841+0000", 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7299999999999994e-06 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264650+0000", 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20880926299999999 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264786+0000", 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013555899999999999 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264924+0000", 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013797500000000001 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.274924+0000", 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010000136999999999 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.274975+0000", 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0387999999999999e-05 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288697+0000", 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013722695 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288736+0000", 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8791999999999997e-05 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288818+0000", 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1802e-05 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.135 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164503 3.2 3:4fa1a650:::benchmark_data_smithi194_88551_object5560:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055710+0000", 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 169.010069018, 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.233259412, 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164503 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055710+0000", 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055710+0000", 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055711+0000", 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299999999999996e-07 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055714+0000", 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.559e-06 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055719+0000", 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6529999999999999e-06 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055724+0000", 2024-04-29T14:18:29.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1200000000000001e-06 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055882+0000", 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000157415 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055890+0000", 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7510000000000008e-06 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.269776+0000", 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21388687300000001 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.269899+0000", 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122368 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.270030+0000", 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013152800000000001 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.274989+0000", 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049582719999999997 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275028+0000", 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9774999999999998e-05 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288853+0000", 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013824881000000001 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288888+0000", 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4275e-05 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288970+0000", 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2368999999999997e-05 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164512 3.2 3:40a769b0:::benchmark_data_smithi194_88551_object5569:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:18:29.142 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055730+0000", 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 169.010049458, 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.233439548, 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164512 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055730+0000", 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055730+0000", 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055731+0000", 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-07 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055733+0000", 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.278e-06 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055739+0000", 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0589999999999999e-06 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055744+0000", 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2839999999999997e-06 2024-04-29T14:18:29.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055916+0000", 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017157199999999999 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055924+0000", 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3059999999999998e-06 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275061+0000", 2024-04-29T14:18:29.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21913727699999999 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275176+0000", 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115135 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275327+0000", 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000150261 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288612+0000", 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01328532 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288650+0000", 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8164000000000003e-05 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.289023+0000", 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037293899999999997 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.289076+0000", 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2913e-05 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.289170+0000", 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3516000000000004e-05 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:182558 3.2 3:4b7f77b6:::benchmark_data_smithi194_88551_object23615:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-04-29T14:18:29.149 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:59.601230+0000", 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.464549019, 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.222199009, 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182558 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601230+0000", 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601230+0000", 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601232+0000", 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.579e-06 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601237+0000", 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1479999999999997e-06 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601249+0000", 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1958e-05 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601256+0000", 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1709999999999996e-06 2024-04-29T14:18:29.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601317+0000", 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1110000000000003e-05 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601441+0000", 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012336000000000001 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601557+0000", 2024-04-29T14:18:29.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001164 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.606861+0000", 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053038130000000001 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.608977+0000", 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021159239999999999 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823254+0000", 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21427707500000001 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823320+0000", 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6439999999999994e-05 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823429+0000", 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010903099999999999 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:182584 3.2 3:47496abe:::benchmark_data_smithi194_88551_object23641:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:59.623937+0000", 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.44184280499999, 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19972438200000001, 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182584 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.623937+0000", 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.623937+0000", 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.623939+0000", 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6199999999999999e-06 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.623947+0000", 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4889999999999998e-06 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.623960+0000", 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.362e-05 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.623970+0000", 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3270000000000007e-06 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.625693+0000", 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001723299 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.625850+0000", 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015666000000000001 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.626002+0000", 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015218 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.638574+0000", 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012572435 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.638588+0000", 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4090000000000001e-05 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823495+0000", 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18490627200000001 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823554+0000", 2024-04-29T14:18:29.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9169999999999998e-05 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823661+0000", 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010721999999999999 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:182586 3.2 3:49cb0fa1:::benchmark_data_smithi194_88551_object23643:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:59.629319+0000", 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.436460963, 2024-04-29T14:18:29.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20090831300000001, 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182586 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.629319+0000", 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.629319+0000", 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.629321+0000", 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7810000000000001e-06 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.629329+0000", 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2200000000000003e-06 2024-04-29T14:18:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.629342+0000", 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3574000000000001e-05 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.629352+0000", 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6549999999999999e-06 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.630946+0000", 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001594551 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.631054+0000", 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107738 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.631157+0000", 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010275100000000001 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.651543+0000", 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020386193 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.651587+0000", 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3757999999999997e-05 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.830051+0000", 2024-04-29T14:18:29.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17846399800000001 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.830118+0000", 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6768999999999997e-05 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.830227+0000", 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109325 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:182597 3.2 3:428e411d:::benchmark_data_smithi194_88551_object23654:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:59.640854+0000", 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.424925481, 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19877023499999999, 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182597 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640854+0000", 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640854+0000", 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640855+0000", 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3e-07 2024-04-29T14:18:29.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640858+0000", 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1530000000000001e-06 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640864+0000", 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5590000000000002e-06 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640870+0000", 2024-04-29T14:18:29.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.507e-06 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.646823+0000", 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059529919999999998 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.646970+0000", 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000147083 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.647114+0000", 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014374200000000001 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.655656+0000", 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085419669999999993 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.661144+0000", 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054887240000000004 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.839460+0000", 2024-04-29T14:18:29.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17831594100000001 2024-04-29T14:18:29.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.839516+0000", 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6168999999999997e-05 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.839624+0000", 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107768 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:182601 3.2 3:41cc4116:::benchmark_data_smithi194_88551_object23658:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:59.640893+0000", 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.42488640900001, 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19892627900000001, 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182601 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:18:29.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640893+0000", 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640893+0000", 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640893+0000", 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0699999999999998e-07 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640896+0000", 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.193e-06 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640903+0000", 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999993e-06 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.640908+0000", 2024-04-29T14:18:29.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6530000000000002e-06 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.651655+0000", 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010747403000000001 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.651737+0000", 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1185000000000003e-05 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.651830+0000", 2024-04-29T14:18:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3401999999999998e-05 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.661055+0000", 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092251 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.661406+0000", 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000351009 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:18:29.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.839661+0000", 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.178255196 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.839713+0000", 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1320999999999999e-05 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.839819+0000", 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106764 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:18:29.182 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:18:29.182 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:18:29.530 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:29.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:29 smithi121 ceph-mon[44533]: 3.a scrub starts 2024-04-29T14:18:29.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:29 smithi121 ceph-mon[44533]: 3.a scrub ok 2024-04-29T14:18:29.788 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:29 smithi040 ceph-mon[39581]: 3.a scrub starts 2024-04-29T14:18:29.788 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:29 smithi040 ceph-mon[39581]: 3.a scrub ok 2024-04-29T14:18:29.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:18:30.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:29 smithi194 ceph-mon[44089]: 3.a scrub starts 2024-04-29T14:18:30.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:29 smithi194 ceph-mon[44089]: 3.a scrub ok 2024-04-29T14:18:30.236 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:18:30.704 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:30.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:30.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:30.831 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:30 smithi121 ceph-mon[44533]: 3.d scrub starts 2024-04-29T14:18:30.831 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:30 smithi121 ceph-mon[44533]: 3.d scrub ok 2024-04-29T14:18:30.831 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:30 smithi121 ceph-mon[44533]: pgmap v834: 17 pgs: 17 active+clean; 280 MiB data, 3.0 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 813 op/s 2024-04-29T14:18:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:30.919 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:30 smithi040 ceph-mon[39581]: 3.d scrub starts 2024-04-29T14:18:30.919 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:30 smithi040 ceph-mon[39581]: 3.d scrub ok 2024-04-29T14:18:30.920 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:30 smithi040 ceph-mon[39581]: pgmap v834: 17 pgs: 17 active+clean; 280 MiB data, 3.0 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 813 op/s 2024-04-29T14:18:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:30.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:31.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:31.055 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:31.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:30 smithi194 ceph-mon[44089]: 3.d scrub starts 2024-04-29T14:18:31.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:30 smithi194 ceph-mon[44089]: 3.d scrub ok 2024-04-29T14:18:31.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:30 smithi194 ceph-mon[44089]: pgmap v834: 17 pgs: 17 active+clean; 280 MiB data, 3.0 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 813 op/s 2024-04-29T14:18:31.092 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 158940 objects 2024-04-29T14:18:31.092 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :177.875 2024-04-29T14:18:31.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:31.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:31.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:31.255 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:31.274 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:31.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:31.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:31.409 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:31.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:31.534 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:31.562 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:31.581 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:31.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:31.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:31.714 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:31.808 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3888484648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:31.809 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:31 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:31.809 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3888484648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:18:31.809 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:31 smithi121 ceph-mon[44533]: osdmap e314: 12 total, 12 up, 9 in 2024-04-29T14:18:31.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:31.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3888484648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:31.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:31 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:31.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3888484648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:18:31.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:31 smithi194 ceph-mon[44089]: osdmap e314: 12 total, 12 up, 9 in 2024-04-29T14:18:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:31.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:31.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3888484648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:18:31.846 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:31 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:31.846 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3888484648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:18:31.846 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:31 smithi040 ceph-mon[39581]: osdmap e314: 12 total, 12 up, 9 in 2024-04-29T14:18:31.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:31.944 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:31.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:31.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:32.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:32.089 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:18:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:18:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:18:32.089 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:18:32.097 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:32.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:32.239 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:32.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:32.410 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:32.438 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:32.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:32.458 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:32.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:32.465 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:32.484 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:32.503 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:32.523 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:32.543 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:32.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:32.563 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:32.582 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:32.591 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:32.602 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:18:32.622 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:18:32.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:18:32.665 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:18:32.666 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:18:32.666 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:18:32.666 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:18:32.667 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:18:32.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:32.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:32.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:18:32.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:32.866 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:32.919 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:32.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:33.023 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:33.061 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:33.078 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:33.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:33.188 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:33.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: pgmap v836: 17 pgs: 17 active+clean; 26 MiB data, 2.5 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osdmap e315: 12 total, 12 up, 9 in 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/176075222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: pgmap v836: 17 pgs: 17 active+clean; 26 MiB data, 2.5 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:18:33.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osdmap e315: 12 total, 12 up, 9 in 2024-04-29T14:18:33.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/176075222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:33.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.322 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: pgmap v836: 17 pgs: 17 active+clean; 26 MiB data, 2.5 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:18:33.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osdmap e315: 12 total, 12 up, 9 in 2024-04-29T14:18:33.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/176075222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:33.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:18:33.564 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:33.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:33.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:33.717 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:33.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:33.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:33.848 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:33.948 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:33.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:33.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:34.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:34.086 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:34.114 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:34.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:34.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:34.254 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:34.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:34.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:34.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:34.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:34.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:34.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:34.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:34.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:34.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:34.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:34.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:34.876 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:34.931 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:34.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:34.992 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:34.992 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:35.036 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:35.069 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:35.089 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:35.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:35.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:35.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:35 smithi040 ceph-mon[39581]: pgmap v838: 17 pgs: 17 active+clean; 26 MiB data, 2.5 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 653 op/s 2024-04-29T14:18:35.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:35 smithi040 ceph-mon[39581]: osdmap e316: 12 total, 12 up, 9 in 2024-04-29T14:18:35.223 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:35.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:35.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:35.317 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:35 smithi121 ceph-mon[44533]: pgmap v838: 17 pgs: 17 active+clean; 26 MiB data, 2.5 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 653 op/s 2024-04-29T14:18:35.317 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:35 smithi121 ceph-mon[44533]: osdmap e316: 12 total, 12 up, 9 in 2024-04-29T14:18:35.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:35.372 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:35.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:18:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:35 smithi194 ceph-mon[44089]: pgmap v838: 17 pgs: 17 active+clean; 26 MiB data, 2.5 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 653 op/s 2024-04-29T14:18:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:35 smithi194 ceph-mon[44089]: osdmap e316: 12 total, 12 up, 9 in 2024-04-29T14:18:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:35.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:35.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:18:35.724 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:35.756 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:35.777 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:35.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:35.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:35.783 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:35.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:35.910 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:35.950 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:18:36.009 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:36.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:36.081 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:18:36.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:36.138 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:36.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:36.273 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:36.306 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:36.327 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:36.329 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2342730284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:36.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2342730284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:36.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2342730284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:36.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:36.461 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:36.765 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-04-29T14:18:36.766 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-04-29T14:18:37.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:37 smithi121 ceph-mon[44533]: pgmap v840: 17 pgs: 17 active+clean; 3.8 MiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 736 op/s 2024-04-29T14:18:37.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:37 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:37.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2342730284' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:37.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:37 smithi121 ceph-mon[44533]: osdmap e317: 12 total, 12 up, 9 in 2024-04-29T14:18:37.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:37.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:37.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:37 smithi040 ceph-mon[39581]: pgmap v840: 17 pgs: 17 active+clean; 3.8 MiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 736 op/s 2024-04-29T14:18:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:37 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2342730284' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:37 smithi040 ceph-mon[39581]: osdmap e317: 12 total, 12 up, 9 in 2024-04-29T14:18:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:37.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:37.552 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:37.552 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:37 smithi194 ceph-mon[44089]: pgmap v840: 17 pgs: 17 active+clean; 3.8 MiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 736 op/s 2024-04-29T14:18:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:37 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2342730284' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:37 smithi194 ceph-mon[44089]: osdmap e317: 12 total, 12 up, 9 in 2024-04-29T14:18:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:37.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:38.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:38 smithi121 ceph-mon[44533]: osdmap e318: 12 total, 12 up, 9 in 2024-04-29T14:18:38.177 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:18:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:38.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:38 smithi040 ceph-mon[39581]: osdmap e318: 12 total, 12 up, 9 in 2024-04-29T14:18:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:38.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:38.447 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:18:38.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:38.494 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:38.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:38 smithi194 ceph-mon[44089]: osdmap e318: 12 total, 12 up, 9 in 2024-04-29T14:18:38.845 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:38.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:38.859 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:38.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:38.986 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:39.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:39.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:39.140 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:39.327 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:39 smithi040 ceph-mon[39581]: pgmap v843: 17 pgs: 17 active+clean; 577 KiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 71 op/s 2024-04-29T14:18:39.339 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:39.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:39 smithi121 ceph-mon[44533]: pgmap v843: 17 pgs: 17 active+clean; 577 KiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 71 op/s 2024-04-29T14:18:39.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:39.477 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:39.499 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:39.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:39 smithi194 ceph-mon[44089]: pgmap v843: 17 pgs: 17 active+clean; 577 KiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 71 op/s 2024-04-29T14:18:39.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:39.632 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:39.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:39.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:39.786 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:39.806 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:39.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:39.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:39.940 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:39.968 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:39.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:40.012 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:40.034 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:40.055 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:40.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:40.074 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:40.099 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:40.104 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:40.123 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:40.141 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:40.141 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:40.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:40.162 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:40.182 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:18:40.202 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:18:40.222 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:18:40.242 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:18:40.243 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:18:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:18:40.262 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:18:40.269 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1858326820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:40.289 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1858326820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:40.292 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:40.303 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:18:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:18:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:18:40.303 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:18:40.311 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:18:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:40.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:18:40.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:18:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1858326820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:40.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:40.450 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:40.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:40.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:40.573 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:40.600 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:40.619 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:40.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:40.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:40.749 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:41.097 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:41.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:41.124 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:18:41.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:41.234 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:41.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:41.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: pgmap v844: 17 pgs: 17 active+clean; 577 KiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 67 op/s 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:41.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: pgmap v844: 17 pgs: 17 active+clean; 577 KiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 67 op/s 2024-04-29T14:18:41.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.389 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:41.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:41.422 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:18:41.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:41.541 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:41.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: pgmap v844: 17 pgs: 17 active+clean; 577 KiB data, 2.4 GiB used, 802 GiB / 805 GiB avail; 0 B/s wr, 67 op/s 2024-04-29T14:18:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:41.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:41.667 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:41.695 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:41.717 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:41.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:41.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:41.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:41.872 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-29T14:18:41.872 INFO:tasks.thrashosds.thrasher:Killing osd 8, live_osds are [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-04-29T14:18:41.873 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-29T14:18:41.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.8 2024-04-29T14:18:42.063 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:18:41 smithi194 systemd[1]: Stopping Ceph osd.8 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:18:42.103 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:42.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:42.114 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_1' removed 2024-04-29T14:18:42.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:42.287 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:42.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:42.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:42.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:42.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:42.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:18:42.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:18:42.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:42 smithi040 ceph-mon[39581]: osdmap e319: 12 total, 12 up, 9 in 2024-04-29T14:18:42.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3895123426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:18:42.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:18:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:18:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:42 smithi121 ceph-mon[44533]: osdmap e319: 12 total, 12 up, 9 in 2024-04-29T14:18:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3895123426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:18:42.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:42.440 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:42.453 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:42.453 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:42.453 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:42.453 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:18:42.453 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:18:42.454 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-mon[44089]: osdmap e319: 12 total, 12 up, 9 in 2024-04-29T14:18:42.454 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3895123426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:18:42.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:42.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:42.562 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:42.594 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:42.752 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:42.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:42.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:42.861 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:18:42.861 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:18:42.862 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-04-29T14:18:42.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-04-29T14:18:42.884 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:42.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:42.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:43.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:43.037 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:43.063 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[50001]: 2024-04-29T14:18:42.937+0000 7ffa6c55d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T14:18:43.064 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[50001]: 2024-04-29T14:18:42.937+0000 7ffa6c55d700 -1 osd.8 320 *** Got signal Terminated *** 2024-04-29T14:18:43.064 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:18:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[50001]: 2024-04-29T14:18:42.937+0000 7ffa6c55d700 -1 osd.8 320 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T14:18:43.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3895123426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: osdmap e320: 12 total, 12 up, 9 in 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:18:43.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:18:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: pgmap v848: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: pgmap v849: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:18:43.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:43 smithi121 ceph-mon[44533]: osd.8 marked itself down and dead 2024-04-29T14:18:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:43.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:43.390 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3895123426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: osdmap e320: 12 total, 12 up, 9 in 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: pgmap v848: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: pgmap v849: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:18:43.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:43 smithi040 ceph-mon[39581]: osd.8 marked itself down and dead 2024-04-29T14:18:43.450 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:18:43.450 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:18:43.450 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:18:43.450 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:18:43.450 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:18:43.475 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3895123426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: osdmap e320: 12 total, 12 up, 9 in 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: pgmap v848: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: pgmap v849: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:18:43.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:43 smithi194 ceph-mon[44089]: osd.8 marked itself down and dead 2024-04-29T14:18:43.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:43.519 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:43.954 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:43.967 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:43.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:44.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:44.099 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:44.162 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:18:44.186 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:44 smithi040 ceph-mon[39581]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:18:44.186 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:44 smithi040 ceph-mon[39581]: osdmap e321: 12 total, 11 up, 9 in 2024-04-29T14:18:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:44.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:44.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:44.332 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:44.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:44 smithi121 ceph-mon[44533]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:18:44.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:44 smithi121 ceph-mon[44533]: osdmap e321: 12 total, 11 up, 9 in 2024-04-29T14:18:44.456 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:44 smithi194 ceph-mon[44089]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:18:44.456 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:44 smithi194 ceph-mon[44089]: osdmap e321: 12 total, 11 up, 9 in 2024-04-29T14:18:44.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:18:44 smithi194 bash[101427]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8 2024-04-29T14:18:44.831 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:18:45.060 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:45.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:45.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:45.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:45.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:45.322 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:45 smithi040 ceph-mon[39581]: pgmap v851: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:45.341 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:45 smithi194 ceph-mon[44089]: pgmap v851: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:45.371 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:45.373 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:45 smithi121 ceph-mon[44533]: pgmap v851: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:45.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:45.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:45.506 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:45.704 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:45.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:45.808 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:45.836 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:46.130 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:18:46.184 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:46.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:46.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:46.320 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:46.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:46.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:46.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3983971560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:18:46.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:46.472 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3983971560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:18:46.475 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:46.495 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:46.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:46.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3983971560' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:18:46.630 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:46.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:46.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:46.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:46.755 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:47.069 INFO:teuthology.orchestra.run.smithi194.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-29T14:18:47.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:47 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:47.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3983971560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:18:47.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:47 smithi194 ceph-mon[44089]: osdmap e322: 12 total, 11 up, 9 in 2024-04-29T14:18:47.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:47 smithi194 ceph-mon[44089]: pgmap v853: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:47.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:47 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:47.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3983971560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:18:47.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:47 smithi040 ceph-mon[39581]: osdmap e322: 12 total, 11 up, 9 in 2024-04-29T14:18:47.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:47 smithi040 ceph-mon[39581]: pgmap v853: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:47.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:47 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:18:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3983971560' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:18:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:47 smithi121 ceph-mon[44533]: osdmap e322: 12 total, 11 up, 9 in 2024-04-29T14:18:47.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:47 smithi121 ceph-mon[44533]: pgmap v853: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:47.636 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:47.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:47.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:48.140 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_2' created 2024-04-29T14:18:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:18:48.246 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:18:48.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:18:48.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:18:48.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3229670099' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-29T14:18:48.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2635055278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:48.569 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.8.service' 2024-04-29T14:18:48.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3229670099' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-29T14:18:48.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2635055278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:48.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T14:18:48.615 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-29T14:18:48.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd down 8 2024-04-29T14:18:48.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3229670099' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-29T14:18:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2635055278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:18:48.647 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:18:48.691 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:48.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:48.794 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:48.822 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:48.945 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:48.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:49.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:49.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3229670099' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osdmap e323: 12 total, 11 up, 9 in 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: pgmap v855: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:49.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.325 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:49.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:18:49.342 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:18:49.430 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3229670099' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osdmap e323: 12 total, 11 up, 9 in 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: pgmap v855: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:49.458 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.459 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:49.479 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:49.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:49.583 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3229670099' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-29T14:18:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osdmap e323: 12 total, 11 up, 9 in 2024-04-29T14:18:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: pgmap v855: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:18:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:49.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:49.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-04-29T14:18:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:49.737 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:49.764 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:49.887 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:49.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:49.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:50.018 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:50.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:50.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:50.205 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:50.227 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:50.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:18:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:50.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:50.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:50 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:18:50.255 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:50 smithi040 ceph-mon[39581]: osdmap e324: 12 total, 11 up, 9 in 2024-04-29T14:18:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:50 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:18:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:50 smithi194 ceph-mon[44089]: osdmap e324: 12 total, 11 up, 9 in 2024-04-29T14:18:50.331 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:50.360 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:50.484 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:50.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:50.496 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:50.496 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:50.517 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:50.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:50.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:50 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:18:50.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:50 smithi121 ceph-mon[44533]: osdmap e324: 12 total, 11 up, 9 in 2024-04-29T14:18:50.620 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:50.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:50.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:50.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:50.775 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:50.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:50.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:50.900 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:50.928 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:51.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:51.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:18:51.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:51.314 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:51.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:51.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:51.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:18:51.345 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:51 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:18:51.346 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:51 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:18:51.346 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:51 smithi040 ceph-mon[39581]: osdmap e325: 12 total, 11 up, 9 in 2024-04-29T14:18:51.346 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:51 smithi040 ceph-mon[39581]: pgmap v858: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:51.439 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:51.465 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:51 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:18:51.465 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:51 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:18:51.465 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:51 smithi194 ceph-mon[44089]: osdmap e325: 12 total, 11 up, 9 in 2024-04-29T14:18:51.465 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:51 smithi194 ceph-mon[44089]: pgmap v858: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:51.467 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:51.486 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:51.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:51.495 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:18:51.590 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:51.622 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:51.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:51 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:18:51.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:51 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:18:51.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:51 smithi121 ceph-mon[44533]: osdmap e325: 12 total, 11 up, 9 in 2024-04-29T14:18:51.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:51 smithi121 ceph-mon[44533]: pgmap v858: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:51.972 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:51.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:52.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:52.106 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:52.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:52.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:52.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:52.260 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:52.279 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:52.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:52.287 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:52.385 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:18:52.414 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8 is already down. 2024-04-29T14:18:52.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:52 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:18:52.442+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e325 definitely_dead 0 2024-04-29T14:18:52.763 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:18:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:52.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:18:52.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:52.902 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:52.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:53.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:53.056 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:53.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:53.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:53.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:53.207 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:53 smithi121 ceph-mon[44533]: pgmap v859: 17 pgs: 15 active+clean, 2 unknown; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:53.207 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3826075585' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["8"]}]: dispatch 2024-04-29T14:18:53.209 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:53.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:53.267 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:53.267 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:53.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:53 smithi194 ceph-mon[44089]: pgmap v859: 17 pgs: 15 active+clean, 2 unknown; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:53.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3826075585' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["8"]}]: dispatch 2024-04-29T14:18:53.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:53.358 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:53 smithi040 ceph-mon[39581]: pgmap v859: 17 pgs: 15 active+clean, 2 unknown; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3826075585' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["8"]}]: dispatch 2024-04-29T14:18:53.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:53.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:53.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:53.509 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:53.528 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:53.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:53.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:53.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:18:54.074 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-04-29T14:18:54.236 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:18:54.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/882670673' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:18:54.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/882670673' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:18:54.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/882670673' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:18:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/882670673' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:18:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:55 smithi194 ceph-mon[44089]: osdmap e326: 12 total, 11 up, 9 in 2024-04-29T14:18:55.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:55 smithi194 ceph-mon[44089]: pgmap v861: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:55.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/882670673' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:18:55.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:55 smithi040 ceph-mon[39581]: osdmap e326: 12 total, 11 up, 9 in 2024-04-29T14:18:55.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:55 smithi040 ceph-mon[39581]: pgmap v861: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:55.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/882670673' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:18:55.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:55 smithi121 ceph-mon[44533]: osdmap e326: 12 total, 11 up, 9 in 2024-04-29T14:18:55.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:55 smithi121 ceph-mon[44533]: pgmap v861: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:55.837 INFO:tasks.thrashosds.thrasher:Removing osd 8, in_osds are: [0, 4, 5, 7, 8, 9, 10, 11, 6] 2024-04-29T14:18:55.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 8 2024-04-29T14:18:55.922 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:55.922 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:56.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:56.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:56 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:18:56.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:18:56.345 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:56 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:18:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:56.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:56 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:18:56.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:18:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:56.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:18:56.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:18:56.501 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:18:56.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:56.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:18:56.646 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:18:56.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:56.766 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:56.866 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:56.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:56.877 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:18:56.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:18:56.999 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:18:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:57.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:18:57.178 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:18:57.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:57.207 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-04-29T14:18:57.209 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:18:57.209 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:18:57.236 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:57.257 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:57.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:57.262 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:57 smithi040 ceph-mon[39581]: pgmap v862: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:57.298 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:18:57.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:18:57.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_103039 2024-04-29T14:18:57.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:18:57.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:18:57.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:57 smithi194 ceph-mon[44089]: pgmap v862: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:57.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:18:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:57 smithi121 ceph-mon[44533]: pgmap v862: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 802 GiB / 805 GiB avail 2024-04-29T14:18:57.403 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:57.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:18:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:57.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:18:57.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:18:57.573 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:58.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 16 5108 5092 318.234 318.25 0.00285632 0.00313601 2024-04-29T14:18:58.324 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:18:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:58.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:18:58.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:18:58.441 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:18:58.455 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:58.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:18:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:58.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:18:58.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2294497708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:58.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:58 smithi194 ceph-mon[44089]: pgmap v863: 17 pgs: 17 active+clean; 11 MiB data, 2.3 GiB used, 802 GiB / 805 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-29T14:18:58.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:58.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2294497708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:58.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:58 smithi040 ceph-mon[39581]: pgmap v863: 17 pgs: 17 active+clean; 11 MiB data, 2.3 GiB used, 802 GiB / 805 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-29T14:18:58.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2294497708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:18:58.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:58 smithi121 ceph-mon[44533]: pgmap v863: 17 pgs: 17 active+clean; 11 MiB data, 2.3 GiB used, 802 GiB / 805 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-29T14:18:58.641 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:58.668 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:58.673 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:58.774 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:18:58.816 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:58.897 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:18:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:58.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:18:59.002 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:18:59.045 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:59.072 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:18:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:59.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:18:59.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:18:59.212 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:59.277 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:18:59.277 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:18:59.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 16 8751 8735 272.951 227.688 0.00323435 0.00366036 2024-04-29T14:18:59.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:59 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:18:59.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2294497708' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:59.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:18:59 smithi040 ceph-mon[39581]: osdmap e327: 12 total, 11 up, 9 in 2024-04-29T14:18:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:59 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:18:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2294497708' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:18:59 smithi194 ceph-mon[44089]: osdmap e327: 12 total, 11 up, 9 in 2024-04-29T14:18:59.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:59 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:18:59.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2294497708' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:18:59.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:18:59 smithi121 ceph-mon[44533]: osdmap e327: 12 total, 11 up, 9 in 2024-04-29T14:18:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:18:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:59.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:18:59.784 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:18:59.806 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:18:59.828 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:18:59.850 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:18:59.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:18:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:18:59.898 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:18:59.911 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:18:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:18:59.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:18:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:18:59.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:18:59.947 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:18:59.974 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:19:00.001 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:19:00.031 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:19:00.032 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:19:00.033 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:19:00.041 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:19:00.042 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:00.074 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:00.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:00.246 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:00.246 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:00.247 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3660136237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:00.247 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:00.247 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:00 smithi194 ceph-mon[44089]: pgmap v865: 17 pgs: 17 active+clean; 11 MiB data, 2.3 GiB used, 802 GiB / 805 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-29T14:19:00.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:00.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 16 13104 13088 272.648 272.062 0.00473214 0.0036645 2024-04-29T14:19:00.319 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:00.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:00.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3660136237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:00 smithi040 ceph-mon[39581]: pgmap v865: 17 pgs: 17 active+clean; 11 MiB data, 2.3 GiB used, 802 GiB / 805 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-29T14:19:00.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:00.464 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:19:00.494 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:00.528 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:00.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:00.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:00.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3660136237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:00.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:00.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:00 smithi121 ceph-mon[44533]: pgmap v865: 17 pgs: 17 active+clean; 11 MiB data, 2.3 GiB used, 802 GiB / 805 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-29T14:19:00.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:00.636 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:00.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:00 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:19:00.551+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e328 definitely_dead 0 2024-04-29T14:19:00.998 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:01.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:19:01.021 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:19:01.102 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:01.147 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.8. 2024-04-29T14:19:01.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 15 16972 16957 264.933 241.812 0.0024466 0.00377174 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: osdmap e328: 12 total, 11 up, 9 in 2024-04-29T14:19:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4042710514' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-04-29T14:19:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:01 smithi194 ceph-mon[44089]: Client client.admin marked osd.8 out, after it was down for 17 seconds 2024-04-29T14:19:01.356 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:01.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:01.461 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:01.502 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:01.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: osdmap e328: 12 total, 11 up, 9 in 2024-04-29T14:19:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4042710514' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-04-29T14:19:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:01 smithi040 ceph-mon[39581]: Client client.admin marked osd.8 out, after it was down for 17 seconds 2024-04-29T14:19:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: osdmap e328: 12 total, 11 up, 9 in 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4042710514' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-04-29T14:19:01.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:01 smithi121 ceph-mon[44533]: Client client.admin marked osd.8 out, after it was down for 17 seconds 2024-04-29T14:19:02.023 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.8 2024-04-29T14:19:02.023 INFO:tasks.thrashosds.thrasher:remote for osd 8 is ubuntu@smithi194.front.sepia.ceph.com 2024-04-29T14:19:02.023 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.8 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --op list-pgs 2024-04-29T14:19:02.057 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:02.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 16 20736 20720 258.979 235.188 0.00344967 0.00385814 2024-04-29T14:19:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:02 smithi194 ceph-mon[44089]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:19:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4042710514' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-04-29T14:19:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:02 smithi194 ceph-mon[44089]: osdmap e329: 12 total, 11 up, 8 in 2024-04-29T14:19:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:02 smithi194 ceph-mon[44089]: pgmap v868: 17 pgs: 17 active+clean; 839 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-29T14:19:02.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:02 smithi040 ceph-mon[39581]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:19:02.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4042710514' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-04-29T14:19:02.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:02 smithi040 ceph-mon[39581]: osdmap e329: 12 total, 11 up, 8 in 2024-04-29T14:19:02.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:02 smithi040 ceph-mon[39581]: pgmap v868: 17 pgs: 17 active+clean; 839 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-29T14:19:02.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:02 smithi121 ceph-mon[44533]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:19:02.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4042710514' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-04-29T14:19:02.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:02 smithi121 ceph-mon[44533]: osdmap e329: 12 total, 11 up, 8 in 2024-04-29T14:19:02.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:02 smithi121 ceph-mon[44533]: pgmap v868: 17 pgs: 17 active+clean; 839 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-29T14:19:02.948 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.8/config 2024-04-29T14:19:03.022 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:19:03.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 16 24136 24120 251.229 212.5 0.00414108 0.00397852 2024-04-29T14:19:03.655 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:03.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:03.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:03.904 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:04.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:04.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:04.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 16 27451 27435 244.935 207.188 0.0047329 0.00408068 2024-04-29T14:19:04.310 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:04.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:04 smithi194 ceph-mon[44089]: pgmap v869: 17 pgs: 17 active+clean; 992 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-04-29T14:19:04.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:04.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:04.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:04.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:04 smithi040 ceph-mon[39581]: pgmap v869: 17 pgs: 17 active+clean; 992 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-04-29T14:19:04.472 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:04.493 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:04.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:04.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:04.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:04 smithi121 ceph-mon[44533]: pgmap v869: 17 pgs: 17 active+clean; 992 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-04-29T14:19:04.644 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:04.776 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:04.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:04.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:04.932 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:04.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:04.960 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:19:04.960 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:19:04.960 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:19:04.960 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:19:04.960 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:19:05.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:05.079 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:05.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:05.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:05.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:05.242 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:05.257 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2757163327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:19:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:19:05.300 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 16 30560 30544 238.603 194.312 0.002976 0.00418862 2024-04-29T14:19:05.332 INFO:teuthology.orchestra.run.smithi194.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:19:05.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2757163327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:19:05.592 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:05.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2757163327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:19:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:05.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:05.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:05.741 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:05.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:05.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:05.871 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:05.912 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:06.022 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:06.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:06.069 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:19:06.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:06.162 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:06.236 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:06.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:06.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:06 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:06.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2757163327' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:19:06.263 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:06 smithi194 ceph-mon[44089]: osdmap e330: 12 total, 11 up, 8 in 2024-04-29T14:19:06.263 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:06 smithi194 ceph-mon[44089]: pgmap v871: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 287 MiB/s wr, 4.60k op/s 2024-04-29T14:19:06.300 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 15 33533 33518 232.726 185.875 0.00647011 0.00429473 2024-04-29T14:19:06.339 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:06.369 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:06.371 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:19:06.389 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:06.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:06.393 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:19:06.415 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:19:06.435 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:19:06.461 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:19:06.483 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:19:06.494 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:06.504 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:19:06.525 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:19:06.551 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:19:06.552 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:06.579 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:19:06.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:19:06.607 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:19:06.608 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:19:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:06 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2757163327' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:19:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:06 smithi040 ceph-mon[39581]: osdmap e330: 12 total, 11 up, 8 in 2024-04-29T14:19:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:06 smithi040 ceph-mon[39581]: pgmap v871: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 287 MiB/s wr, 4.60k op/s 2024-04-29T14:19:06.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:06 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:06.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2757163327' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:19:06.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:06 smithi121 ceph-mon[44533]: osdmap e330: 12 total, 11 up, 8 in 2024-04-29T14:19:06.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:06 smithi121 ceph-mon[44533]: pgmap v871: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 287 MiB/s wr, 4.60k op/s 2024-04-29T14:19:06.634 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:06.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:06.723 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:19:06.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:06.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:06.901 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:06.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:07.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:07.040 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:07.095 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:07.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:07.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:07.245 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:07.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:07.300 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 16 36018 36002 224.977 155.25 0.00345314 0.00444297 2024-04-29T14:19:07.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:07.426 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:07.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:07.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:07.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/186137307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/186137307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.587 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/186137307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:07.787 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:07.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:07.806 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:19:07.807 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:19:07.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:07.940 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:07.979 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:07.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:08.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:08.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:08.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:08.290 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:08.300 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 38800 38784 220.331 173.875 0.00580611 0.00453754 2024-04-29T14:19:08.310 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:08.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:08 smithi121 ceph-mon[44533]: osdmap e331: 12 total, 11 up, 8 in 2024-04-29T14:19:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:08 smithi121 ceph-mon[44533]: pgmap v873: 17 pgs: 17 active+clean; 2.0 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 204 MiB/s wr, 3.26k op/s 2024-04-29T14:19:08.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:08.462 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:08.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:08.543 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:08.543 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:08.543 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:08.543 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:08 smithi194 ceph-mon[44089]: osdmap e331: 12 total, 11 up, 8 in 2024-04-29T14:19:08.543 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:08 smithi194 ceph-mon[44089]: pgmap v873: 17 pgs: 17 active+clean; 2.0 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 204 MiB/s wr, 3.26k op/s 2024-04-29T14:19:08.566 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:08.608 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:08 smithi040 ceph-mon[39581]: osdmap e331: 12 total, 11 up, 8 in 2024-04-29T14:19:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:08 smithi040 ceph-mon[39581]: pgmap v873: 17 pgs: 17 active+clean; 2.0 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 204 MiB/s wr, 3.26k op/s 2024-04-29T14:19:08.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:19:08.957 INFO:tasks.thrashosds.thrasher:No PGs found for osd.8 2024-04-29T14:19:08.957 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-29T14:19:08.957 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:19:08.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:19:08.963 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:08.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:09.069 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:09.101 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:09.127 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:09.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:09.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:09.238 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:09.269 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:09.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:09.300 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 41475 41459 215.902 167.188 0.00583868 0.00462952 2024-04-29T14:19:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:09.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:09.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:09.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:09.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:09.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:09.592 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:19:09.593 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:19:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:09.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:19:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:09.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:09.997 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:10.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:10.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:10.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:10.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:10.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:10.269 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:10.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 16 44116 44100 211.99 165.062 0.00441684 0.00471553 2024-04-29T14:19:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:10 smithi194 ceph-mon[44089]: pgmap v874: 17 pgs: 17 active+clean; 2.0 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 150 MiB/s wr, 2.41k op/s 2024-04-29T14:19:10.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:10 smithi121 ceph-mon[44533]: pgmap v874: 17 pgs: 17 active+clean; 2.0 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 150 MiB/s wr, 2.41k op/s 2024-04-29T14:19:10.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:19:10.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:10 smithi040 ceph-mon[39581]: pgmap v874: 17 pgs: 17 active+clean; 2.0 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 150 MiB/s wr, 2.41k op/s 2024-04-29T14:19:10.723 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:10.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:10.827 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:10.869 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:10.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:11.002 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:11.051 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:11.079 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:11.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:11.183 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:11.223 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:11.249 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:11.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:11.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 16 46678 46662 208.285 160.125 0.00562969 0.00479922 2024-04-29T14:19:11.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:11.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:11.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:11.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:11.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:12.004 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:12.043 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:12.074 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:12.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:12.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:12.209 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:12.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 16 49227 49211 205.019 159.312 0.00734163 0.00487481 2024-04-29T14:19:12.334 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:12.337 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:12.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:12.339 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:12.439 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:12.479 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:12.503 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:12.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:12.608 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:12.637 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:12.660 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:12.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:12.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:12.805 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:12.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:12.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:12.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:12.990 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:13.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:13.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:13.077 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:13 smithi121 ceph-mon[44533]: pgmap v875: 17 pgs: 17 active+clean; 2.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 223 MiB/s wr, 3.56k op/s 2024-04-29T14:19:13.129 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:13.151 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:13 smithi040 ceph-mon[39581]: pgmap v875: 17 pgs: 17 active+clean; 2.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 223 MiB/s wr, 3.56k op/s 2024-04-29T14:19:13.169 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:13.201 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:13.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:13.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 16 51634 51618 201.607 150.438 0.00720735 0.0049582 2024-04-29T14:19:13.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:13.353 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:13.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:13.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:13.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:13.542 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:13 smithi194 ceph-mon[44089]: pgmap v875: 17 pgs: 17 active+clean; 2.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 223 MiB/s wr, 3.56k op/s 2024-04-29T14:19:13.581 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:13.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:13.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:13.736 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:13.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:13.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:13.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:13.906 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:14.156 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:14.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:14.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:14.291 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:14.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 53976 53960 198.358 146.375 0.00538527 0.00503975 2024-04-29T14:19:14.320 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:14.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:14.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:14.461 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:14.700 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:19:14.702 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:19:15.089 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:19:15.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 56316 56300 195.462 146.25 0.00276927 0.00511403 2024-04-29T14:19:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:15 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:15 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:15 smithi040 ceph-mon[39581]: osdmap e332: 12 total, 11 up, 8 in 2024-04-29T14:19:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:15 smithi040 ceph-mon[39581]: pgmap v877: 17 pgs: 17 active+clean; 2.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-29T14:19:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2517770736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:15 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:15 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:15 smithi121 ceph-mon[44533]: osdmap e332: 12 total, 11 up, 8 in 2024-04-29T14:19:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:15 smithi121 ceph-mon[44533]: pgmap v877: 17 pgs: 17 active+clean; 2.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-29T14:19:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2517770736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:15.515 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:15.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:15.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:15 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:15.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:15 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:15.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:15 smithi194 ceph-mon[44089]: osdmap e332: 12 total, 11 up, 8 in 2024-04-29T14:19:15.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:15 smithi194 ceph-mon[44089]: pgmap v877: 17 pgs: 17 active+clean; 2.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-29T14:19:15.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2517770736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:15.619 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:15.661 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:15.690 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:15.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:15.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:15.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:15.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:16.016 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:16.044 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:16.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:16.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:16 smithi121 ceph-mon[44533]: from='client.15636 v1:172.21.15.40:0/3206886879' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:19:16.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:16 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:16.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:16.215 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:16.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:16.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:16.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 58544 58528 192.503 139.25 0.00454564 0.00519255 2024-04-29T14:19:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:16 smithi194 ceph-mon[44089]: from='client.15636 v1:172.21.15.40:0/3206886879' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:19:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:16 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2517770736' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:16 smithi194 ceph-mon[44089]: osdmap e333: 12 total, 11 up, 8 in 2024-04-29T14:19:16.347 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:16.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:16 smithi040 ceph-mon[39581]: from='client.15636 v1:172.21.15.40:0/3206886879' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:19:16.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:16 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:16.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2517770736' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:16.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:16 smithi040 ceph-mon[39581]: osdmap e333: 12 total, 11 up, 8 in 2024-04-29T14:19:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2517770736' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:16 smithi121 ceph-mon[44533]: osdmap e333: 12 total, 11 up, 8 in 2024-04-29T14:19:16.391 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:16.421 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:16.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:16.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:16.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:16.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:16.718 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:16.760 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:16.788 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:16.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:16.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:16.930 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:16.957 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:16.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:17.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:17.094 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:19:17.095 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:19:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:17.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:19:17.309231 min lat: 0.00115367 max lat: 0.206219 avg lat: 0.00527119 2024-04-29T14:19:17.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:19:17.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 16 60711 60695 189.649 135.438 0.00500593 0.00527119 2024-04-29T14:19:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:17 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:17 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:17 smithi040 ceph-mon[39581]: osdmap e334: 12 total, 11 up, 8 in 2024-04-29T14:19:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:17 smithi040 ceph-mon[39581]: pgmap v880: 17 pgs: 17 active+clean; 3.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 225 MiB/s wr, 3.60k op/s 2024-04-29T14:19:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:17 smithi040 ceph-mon[39581]: osdmap e335: 12 total, 11 up, 8 in 2024-04-29T14:19:17.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:17 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:17.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:17 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:17 smithi121 ceph-mon[44533]: osdmap e334: 12 total, 11 up, 8 in 2024-04-29T14:19:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:17 smithi121 ceph-mon[44533]: pgmap v880: 17 pgs: 17 active+clean; 3.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 225 MiB/s wr, 3.60k op/s 2024-04-29T14:19:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:17 smithi121 ceph-mon[44533]: osdmap e335: 12 total, 11 up, 8 in 2024-04-29T14:19:17.524 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2024-04-29T14:19:14.189311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3013673504,"num_objects":45978,"num_object_clones":0,"num_object_copies":91956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45978,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":46157,"num_write_kb":2945402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46113,"ondisk_log_size":46113,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8150704,"kb_used_data":5768880,"kb_used_omap":12,"kb_used_meta":2381747,"kb_avail":741810512,"statfs":{"total":767960285184,"available":759613964288,"internally_reserved":0,"allocated":5907333120,"data_stored":5903954648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12765,"internal_metadata":2438909475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1193017344,"num_objects":18204,"num_object_clones":0,"num_object_copies":36408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18204,"num_write_kb":1165056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001400"},"pg_stats":[{"pgid":"4.f","version":"331'2845","reported_seq":2870,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.445372+0000","last_change":"2024-04-29T14:18:49.370708+0000","last_active":"2024-04-29T14:19:10.445372+0000","last_peered":"2024-04-29T14:19:10.445372+0000","last_clean":"2024-04-29T14:19:10.445372+0000","last_became_active":"2024-04-29T14:18:49.370395+0000","last_became_peered":"2024-04-29T14:18:49.370395+0000","last_unstale":"2024-04-29T14:19:10.445372+0000","last_undegraded":"2024-04-29T14:19:10.445372+0000","last_fullsized":"2024-04-29T14:19:10.445372+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2845,"log_dups_size":0,"ondisk_log_size":2845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.183106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186449920,"num_objects":2845,"num_object_clones":0,"num_object_copies":5690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2845,"num_write_kb":182080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"331'2922","reported_seq":2947,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:11.462215+0000","last_change":"2024-04-29T14:18:49.153002+0000","last_active":"2024-04-29T14:19:11.462215+0000","last_peered":"2024-04-29T14:19:11.462215+0000","last_clean":"2024-04-29T14:19:11.462215+0000","last_became_active":"2024-04-29T14:18:49.152699+0000","last_became_peered":"2024-04-29T14:18:49.152699+0000","last_unstale":"2024-04-29T14:19:11.462215+0000","last_undegraded":"2024-04-29T14:19:11.462215+0000","last_fullsized":"2024-04-29T14:19:11.462215+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2922,"log_dups_size":0,"ondisk_log_size":2922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:01.029124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191496192,"num_objects":2922,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":187008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"331'2957","reported_seq":2980,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:11.495980+0000","last_change":"2024-04-29T14:18:49.155985+0000","last_active":"2024-04-29T14:19:11.495980+0000","last_peered":"2024-04-29T14:19:11.495980+0000","last_clean":"2024-04-29T14:19:11.495980+0000","last_became_active":"2024-04-29T14:18:49.155690+0000","last_became_peered":"2024-04-29T14:18:49.155690+0000","last_unstale":"2024-04-29T14:19:11.495980+0000","last_undegraded":"2024-04-29T14:19:11.495980+0000","last_fullsized":"2024-04-29T14:19:11.495980+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2957,"log_dups_size":0,"ondisk_log_size":2957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:06.247246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193789952,"num_objects":2957,"num_object_clones":0,"num_object_copies":5914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":189248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"331'2915","reported_seq":2940,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:11.498370+0000","last_change":"2024-04-29T14:18:49.156461+0000","last_active":"2024-04-29T14:19:11.498370+0000","last_peered":"2024-04-29T14:19:11.498370+0000","last_clean":"2024-04-29T14:19:11.498370+0000","last_became_active":"2024-04-29T14:18:49.156158+0000","last_became_peered":"2024-04-29T14:18:49.156158+0000","last_unstale":"2024-04-29T14:19:11.498370+0000","last_undegraded":"2024-04-29T14:19:11.498370+0000","last_fullsized":"2024-04-29T14:19:11.498370+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:58.804262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":5830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2915,"num_write_kb":186560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"331'2923","reported_seq":2948,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:11.004196+0000","last_change":"2024-04-29T14:18:49.152425+0000","last_active":"2024-04-29T14:19:11.004196+0000","last_peered":"2024-04-29T14:19:11.004196+0000","last_clean":"2024-04-29T14:19:11.004196+0000","last_became_active":"2024-04-29T14:18:49.152097+0000","last_became_peered":"2024-04-29T14:18:49.152097+0000","last_unstale":"2024-04-29T14:19:11.004196+0000","last_undegraded":"2024-04-29T14:19:11.004196+0000","last_fullsized":"2024-04-29T14:19:11.004196+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:48.475752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191561728,"num_objects":2923,"num_object_clones":0,"num_object_copies":5846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":187072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"331'2759","reported_seq":2784,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.508008+0000","last_change":"2024-04-29T14:18:49.156745+0000","last_active":"2024-04-29T14:19:10.508008+0000","last_peered":"2024-04-29T14:19:10.508008+0000","last_clean":"2024-04-29T14:19:10.508008+0000","last_became_active":"2024-04-29T14:18:49.156474+0000","last_became_peered":"2024-04-29T14:18:49.156474+0000","last_unstale":"2024-04-29T14:19:10.508008+0000","last_undegraded":"2024-04-29T14:19:10.508008+0000","last_fullsized":"2024-04-29T14:19:10.508008+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2759,"log_dups_size":0,"ondisk_log_size":2759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.931388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180813824,"num_objects":2759,"num_object_clones":0,"num_object_copies":5518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":176576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"331'2770","reported_seq":2795,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.510992+0000","last_change":"2024-04-29T14:18:49.371513+0000","last_active":"2024-04-29T14:19:10.510992+0000","last_peered":"2024-04-29T14:19:10.510992+0000","last_clean":"2024-04-29T14:19:10.510992+0000","last_became_active":"2024-04-29T14:18:49.371178+0000","last_became_peered":"2024-04-29T14:18:49.371178+0000","last_unstale":"2024-04-29T14:19:10.510992+0000","last_undegraded":"2024-04-29T14:19:10.510992+0000","last_fullsized":"2024-04-29T14:19:10.510992+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.236063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181534720,"num_objects":2770,"num_object_clones":0,"num_object_copies":5540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":177280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"331'2819","reported_seq":2844,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.861113+0000","last_change":"2024-04-29T14:18:49.370740+0000","last_active":"2024-04-29T14:19:10.861113+0000","last_peered":"2024-04-29T14:19:10.861113+0000","last_clean":"2024-04-29T14:19:10.861113+0000","last_became_active":"2024-04-29T14:18:49.370541+0000","last_became_peered":"2024-04-29T14:18:49.370541+0000","last_unstale":"2024-04-29T14:19:10.861113+0000","last_undegraded":"2024-04-29T14:19:10.861113+0000","last_fullsized":"2024-04-29T14:19:10.861113+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2819,"log_dups_size":0,"ondisk_log_size":2819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:54.263953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184745984,"num_objects":2819,"num_object_clones":0,"num_object_copies":5638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2819,"num_write_kb":180416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"331'3053","reported_seq":3078,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:12.588600+0000","last_change":"2024-04-29T14:18:49.151635+0000","last_active":"2024-04-29T14:19:12.588600+0000","last_peered":"2024-04-29T14:19:12.588600+0000","last_clean":"2024-04-29T14:19:12.588600+0000","last_became_active":"2024-04-29T14:18:49.151382+0000","last_became_peered":"2024-04-29T14:18:49.151382+0000","last_unstale":"2024-04-29T14:19:12.588600+0000","last_undegraded":"2024-04-29T14:19:12.588600+0000","last_fullsized":"2024-04-29T14:19:12.588600+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":3053,"log_dups_size":0,"ondisk_log_size":3053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:56.402851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200081408,"num_objects":3053,"num_object_clones":0,"num_object_copies":6106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3053,"num_write_kb":195392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"331'2737","reported_seq":2762,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.449161+0000","last_change":"2024-04-29T14:18:49.157325+0000","last_active":"2024-04-29T14:19:10.449161+0000","last_peered":"2024-04-29T14:19:10.449161+0000","last_clean":"2024-04-29T14:19:10.449161+0000","last_became_active":"2024-04-29T14:18:49.157084+0000","last_became_peered":"2024-04-29T14:18:49.157084+0000","last_unstale":"2024-04-29T14:19:10.449161+0000","last_undegraded":"2024-04-29T14:19:10.449161+0000","last_fullsized":"2024-04-29T14:19:10.449161+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.092162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179372032,"num_objects":2737,"num_object_clones":0,"num_object_copies":5474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2737,"num_write_kb":175168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7],"acting":[9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"331'3044","reported_seq":3068,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:12.593343+0000","last_change":"2024-04-29T14:18:49.156653+0000","last_active":"2024-04-29T14:19:12.593343+0000","last_peered":"2024-04-29T14:19:12.593343+0000","last_clean":"2024-04-29T14:19:12.593343+0000","last_became_active":"2024-04-29T14:18:49.156484+0000","last_became_peered":"2024-04-29T14:18:49.156484+0000","last_unstale":"2024-04-29T14:19:12.593343+0000","last_undegraded":"2024-04-29T14:19:12.593343+0000","last_fullsized":"2024-04-29T14:19:12.593343+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":3044,"log_dups_size":0,"ondisk_log_size":3044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:54.124339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199491584,"num_objects":3044,"num_object_clones":0,"num_object_copies":6088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3044,"num_write_kb":194816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"331'2835","reported_seq":2860,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.854039+0000","last_change":"2024-04-29T14:18:49.371657+0000","last_active":"2024-04-29T14:19:10.854039+0000","last_peered":"2024-04-29T14:19:10.854039+0000","last_clean":"2024-04-29T14:19:10.854039+0000","last_became_active":"2024-04-29T14:18:49.371335+0000","last_became_peered":"2024-04-29T14:18:49.371335+0000","last_unstale":"2024-04-29T14:19:10.854039+0000","last_undegraded":"2024-04-29T14:19:10.854039+0000","last_fullsized":"2024-04-29T14:19:10.854039+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:56.673106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185794560,"num_objects":2835,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2835,"num_write_kb":181440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"331'2816","reported_seq":2841,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.435041+0000","last_change":"2024-04-29T14:18:49.158643+0000","last_active":"2024-04-29T14:19:10.435041+0000","last_peered":"2024-04-29T14:19:10.435041+0000","last_clean":"2024-04-29T14:19:10.435041+0000","last_became_active":"2024-04-29T14:18:49.158456+0000","last_became_peered":"2024-04-29T14:18:49.158456+0000","last_unstale":"2024-04-29T14:19:10.435041+0000","last_undegraded":"2024-04-29T14:19:10.435041+0000","last_fullsized":"2024-04-29T14:19:10.435041+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:13.613295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":2816,"num_object_clones":0,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2816,"num_write_kb":180224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"331'2885","reported_seq":2909,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.330850+0000","last_change":"2024-04-29T14:18:49.371803+0000","last_active":"2024-04-29T14:19:10.330850+0000","last_peered":"2024-04-29T14:19:10.330850+0000","last_clean":"2024-04-29T14:19:10.330850+0000","last_became_active":"2024-04-29T14:18:49.371468+0000","last_became_peered":"2024-04-29T14:18:49.371468+0000","last_unstale":"2024-04-29T14:19:10.330850+0000","last_undegraded":"2024-04-29T14:19:10.330850+0000","last_fullsized":"2024-04-29T14:19:10.330850+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2885,"log_dups_size":0,"ondisk_log_size":2885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:17.465842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189071360,"num_objects":2885,"num_object_clones":0,"num_object_copies":5770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":184640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"331'2924","reported_seq":2949,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.857478+0000","last_change":"2024-04-29T14:18:49.158273+0000","last_active":"2024-04-29T14:19:10.857478+0000","last_peered":"2024-04-29T14:19:10.857478+0000","last_clean":"2024-04-29T14:19:10.857478+0000","last_became_active":"2024-04-29T14:18:49.158015+0000","last_became_peered":"2024-04-29T14:18:49.158015+0000","last_unstale":"2024-04-29T14:19:10.857478+0000","last_undegraded":"2024-04-29T14:19:10.857478+0000","last_fullsized":"2024-04-29T14:19:10.857478+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2924,"log_dups_size":0,"ondisk_log_size":2924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:53.632841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191627264,"num_objects":2924,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":187136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"331'2772","reported_seq":2797,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:10.513160+0000","last_change":"2024-04-29T14:18:49.156792+0000","last_active":"2024-04-29T14:19:10.513160+0000","last_peered":"2024-04-29T14:19:10.513160+0000","last_clean":"2024-04-29T14:19:10.513160+0000","last_became_active":"2024-04-29T14:18:49.156512+0000","last_became_peered":"2024-04-29T14:18:49.156512+0000","last_unstale":"2024-04-29T14:19:10.513160+0000","last_undegraded":"2024-04-29T14:19:10.513160+0000","last_fullsized":"2024-04-29T14:19:10.513160+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":2772,"log_dups_size":0,"ondisk_log_size":2772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:08.011441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181665792,"num_objects":2772,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":177408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":861,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-29T14:19:07.294957+0000","last_change":"2024-04-29T14:18:49.855444+0000","last_active":"2024-04-29T14:19:07.294957+0000","last_peered":"2024-04-29T14:19:07.294957+0000","last_clean":"2024-04-29T14:19:07.294957+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:19:07.294957+0000","last_undegraded":"2024-04-29T14:19:07.294957+0000","last_fullsized":"2024-04-29T14:19:07.294957+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:18:49.855310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:18:49.855310+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:02.043752+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3013083136,"num_objects":45976,"num_object_clones":0,"num_object_copies":91952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45976,"num_write_kb":2942464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6034489344,"data_stored":6034489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45976,"ondisk_log_size":45976,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122195,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1223212,"kb_used_data":899108,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92521940,"statfs":{"total":95995035648,"available":94742466560,"internally_reserved":0,"allocated":920686592,"data_stored":920261604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":331872708},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383832,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1101476,"kb_used_data":939804,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92643676,"statfs":{"total":95995035648,"available":94867124224,"internally_reserved":0,"allocated":962359296,"data_stored":961937860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":165542336},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645469,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948580,"kb_used_data":700828,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92796572,"statfs":{"total":95995035648,"available":95023689728,"internally_reserved":0,"allocated":717647872,"data_stored":717226456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":253688256},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234149,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228588,"kb_used_data":904484,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92516564,"statfs":{"total":95995035648,"available":94736961536,"internally_reserved":0,"allocated":926191616,"data_stored":925766648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":331872714},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528491,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073508,"kb_used_data":685532,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92671644,"statfs":{"total":95995035648,"available":94895763456,"internally_reserved":0,"allocated":701984768,"data_stored":701563352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":397277638},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762340,"kb_used_data":528476,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":92982812,"statfs":{"total":95995035648,"available":95214399488,"internally_reserved":0,"allocated":541159424,"data_stored":540738008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986358,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197284,"kb_used_data":751004,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92547868,"statfs":{"total":95995035648,"available":94769016832,"internally_reserved":0,"allocated":769028096,"data_stored":768606680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822831,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615716,"kb_used_data":359644,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93129436,"statfs":{"total":95995035648,"available":95364542464,"internally_reserved":0,"allocated":368275456,"data_stored":367854040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":567738368,"data_stored":567738368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":771883008,"data_stored":771883008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":372768768,"data_stored":372768768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":724500480,"data_stored":724500480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":928972800,"data_stored":928972800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":732823552,"data_stored":732823552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1008992256,"data_stored":1008992256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":926810112,"data_stored":926810112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:19:17.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:19:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:17 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:17 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:17 smithi194 ceph-mon[44089]: osdmap e334: 12 total, 11 up, 8 in 2024-04-29T14:19:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:17 smithi194 ceph-mon[44089]: pgmap v880: 17 pgs: 17 active+clean; 3.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 225 MiB/s wr, 3.60k op/s 2024-04-29T14:19:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:17 smithi194 ceph-mon[44089]: osdmap e335: 12 total, 11 up, 8 in 2024-04-29T14:19:18.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:19:18.250 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:18.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:18.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 62887 62871 187.094 136 0.00519058 0.00534331 2024-04-29T14:19:18.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:18.389 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:18 smithi040 ceph-mon[39581]: pgmap v882: 17 pgs: 17 active+clean; 3.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-29T14:19:18.392 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:18.396 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:19:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:19:18.439 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:19:18.465 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:19:18.488 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:19:18.511 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:19:18.536 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:19:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:19:18.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:18 smithi194 ceph-mon[44089]: pgmap v882: 17 pgs: 17 active+clean; 3.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-29T14:19:18.584 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:19:18.591 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:18.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:19:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:18.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:18.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:18 smithi121 ceph-mon[44533]: pgmap v882: 17 pgs: 17 active+clean; 3.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-29T14:19:18.629 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:19:18.630 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:19:18.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:18.991 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:18.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:19.095 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:19.134 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:19.162 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:19.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:19.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/110001045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:19.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:19.270 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:19.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 16 65016 65000 184.638 133.062 0.00744672 0.00541417 2024-04-29T14:19:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:19.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:19.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:19.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:19.492 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:19.524 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:19.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:19.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/110001045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:19.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/110001045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:19.631 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:19.674 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:19.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:19.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:19.828 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:19:19.875 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:19:20.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 67074 67058 182.202 128.625 0.0043487 0.00548709 2024-04-29T14:19:20.339 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:20.339 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:20 smithi040 ceph-mon[39581]: pgmap v883: 17 pgs: 17 active+clean; 3.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-04-29T14:19:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:20 smithi194 ceph-mon[44089]: pgmap v883: 17 pgs: 17 active+clean; 3.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-04-29T14:19:20.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:20 smithi121 ceph-mon[44533]: pgmap v883: 17 pgs: 17 active+clean; 3.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-04-29T14:19:21.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:19:21.301 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 16 69083 69067 179.842 125.562 0.0119551 0.00555902 2024-04-29T14:19:21.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:19:22.222 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:22.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 15 71106 71091 177.708 126.5 0.00799887 0.00562573 2024-04-29T14:19:22.565 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:22.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:22.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:22.699 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:22.753 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:22.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:22.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:22.902 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:22.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:22.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:23.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:23.070 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:23.210 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:23.220 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:23.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:23.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 73084 73068 175.625 123.562 0.00288474 0.00569185 2024-04-29T14:19:23.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:23.349 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:23 smithi040 ceph-mon[39581]: osdmap e336: 12 total, 11 up, 8 in 2024-04-29T14:19:23.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:23 smithi040 ceph-mon[39581]: pgmap v885: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-04-29T14:19:23.352 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:23.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:23 smithi121 ceph-mon[44533]: osdmap e336: 12 total, 11 up, 8 in 2024-04-29T14:19:23.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:23 smithi121 ceph-mon[44533]: pgmap v885: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-04-29T14:19:23.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:23 smithi194 ceph-mon[44089]: osdmap e336: 12 total, 11 up, 8 in 2024-04-29T14:19:23.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:23 smithi194 ceph-mon[44089]: pgmap v885: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-04-29T14:19:24.002 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:24.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:24.107 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:24.151 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:24.179 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:24.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:24.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:24.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 16 75035 75019 173.636 121.938 0.0032105 0.00575762 2024-04-29T14:19:24.328 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:24.356 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:24.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:24.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:24.505 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:24.528 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:24.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:24 smithi121 ceph-mon[44533]: pgmap v886: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-29T14:19:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:24.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:24.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:24 smithi194 ceph-mon[44089]: pgmap v886: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-29T14:19:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:24 smithi040 ceph-mon[39581]: pgmap v886: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-29T14:19:24.633 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:24.660 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:24.680 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:24.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:24.765 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:19:24.766 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:19:24.766 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:19:24.766 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:19:24.766 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:19:24.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:24.820 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:19:24.915 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:19:25.170 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:25.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:25.274 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:25.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 15 76730 76715 171.22 106 0.0321885 0.00583833 2024-04-29T14:19:25.307 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:25.336 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:25.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:19:25.440 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:25.478 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:25.507 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:25.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:25.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/45463199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:19:25.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:25 smithi194 ceph-mon[44089]: from='client.35453 v1:172.21.15.40:0/149906122' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:19:25.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:25.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:25.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:25.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/45463199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:19:25.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:25 smithi040 ceph-mon[39581]: from='client.35453 v1:172.21.15.40:0/149906122' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:19:25.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:25.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:25.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/45463199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:19:25.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:25 smithi121 ceph-mon[44533]: from='client.35453 v1:172.21.15.40:0/149906122' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:19:25.664 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:25.691 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:25.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:25.797 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:26.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 78567 78551 169.273 114.75 0.00628598 0.00590585 2024-04-29T14:19:26.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:26 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:26.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/45463199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:19:26.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:26 smithi040 ceph-mon[39581]: osdmap e337: 12 total, 11 up, 8 in 2024-04-29T14:19:26.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:26 smithi040 ceph-mon[39581]: pgmap v888: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-29T14:19:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:26 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/45463199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:19:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:26 smithi194 ceph-mon[44089]: osdmap e337: 12 total, 11 up, 8 in 2024-04-29T14:19:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:26 smithi194 ceph-mon[44089]: pgmap v888: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-29T14:19:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:26.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:26.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:19:26.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:26 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:26.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/45463199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:19:26.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:26 smithi121 ceph-mon[44533]: osdmap e337: 12 total, 11 up, 8 in 2024-04-29T14:19:26.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:26 smithi121 ceph-mon[44533]: pgmap v888: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-29T14:19:26.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:26.722 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:26.753 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:26.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:26.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:26.908 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:26.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:27.033 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":886,"stamp":"2024-04-29T14:19:24.190951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4476174880,"num_objects":68294,"num_object_clones":0,"num_object_copies":136588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68294,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":68473,"num_write_kb":4373626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68429,"ondisk_log_size":68429,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11274368,"kb_used_data":8698172,"kb_used_omap":12,"kb_used_meta":2576115,"kb_avail":738686848,"statfs":{"total":767960285184,"available":756415332352,"internally_reserved":0,"allocated":8906928128,"data_stored":8903342094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12765,"internal_metadata":2637942307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":916652032,"num_objects":13987,"num_object_clones":0,"num_object_copies":27974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13987,"num_write_kb":895168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001342"},"pg_stats":[{"pgid":"4.f","version":"335'4213","reported_seq":4246,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.453677+0000","last_change":"2024-04-29T14:18:49.370708+0000","last_active":"2024-04-29T14:19:20.453677+0000","last_peered":"2024-04-29T14:19:20.453677+0000","last_clean":"2024-04-29T14:19:20.453677+0000","last_became_active":"2024-04-29T14:18:49.370395+0000","last_became_peered":"2024-04-29T14:18:49.370395+0000","last_unstale":"2024-04-29T14:19:20.453677+0000","last_undegraded":"2024-04-29T14:19:20.453677+0000","last_fullsized":"2024-04-29T14:19:20.453677+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.183106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276103168,"num_objects":4213,"num_object_clones":0,"num_object_copies":8426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4213,"num_write_kb":269632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"335'4208","reported_seq":4241,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.860688+0000","last_change":"2024-04-29T14:18:49.153002+0000","last_active":"2024-04-29T14:19:20.860688+0000","last_peered":"2024-04-29T14:19:20.860688+0000","last_clean":"2024-04-29T14:19:20.860688+0000","last_became_active":"2024-04-29T14:18:49.152699+0000","last_became_peered":"2024-04-29T14:18:49.152699+0000","last_unstale":"2024-04-29T14:19:20.860688+0000","last_undegraded":"2024-04-29T14:19:20.860688+0000","last_fullsized":"2024-04-29T14:19:20.860688+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:01.029124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275775488,"num_objects":4208,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4208,"num_write_kb":269312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"335'4219","reported_seq":4252,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.861289+0000","last_change":"2024-04-29T14:18:49.155985+0000","last_active":"2024-04-29T14:19:20.861289+0000","last_peered":"2024-04-29T14:19:20.861289+0000","last_clean":"2024-04-29T14:19:20.861289+0000","last_became_active":"2024-04-29T14:18:49.155690+0000","last_became_peered":"2024-04-29T14:18:49.155690+0000","last_unstale":"2024-04-29T14:19:20.861289+0000","last_undegraded":"2024-04-29T14:19:20.861289+0000","last_fullsized":"2024-04-29T14:19:20.861289+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:06.247246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276496384,"num_objects":4219,"num_object_clones":0,"num_object_copies":8438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4219,"num_write_kb":270016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"335'4301","reported_seq":4334,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.854849+0000","last_change":"2024-04-29T14:18:49.156461+0000","last_active":"2024-04-29T14:19:20.854849+0000","last_peered":"2024-04-29T14:19:20.854849+0000","last_clean":"2024-04-29T14:19:20.854849+0000","last_became_active":"2024-04-29T14:18:49.156158+0000","last_became_peered":"2024-04-29T14:18:49.156158+0000","last_unstale":"2024-04-29T14:19:20.854849+0000","last_undegraded":"2024-04-29T14:19:20.854849+0000","last_fullsized":"2024-04-29T14:19:20.854849+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:58.804262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281870336,"num_objects":4301,"num_object_clones":0,"num_object_copies":8602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4301,"num_write_kb":275264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"335'4367","reported_seq":4400,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.996332+0000","last_change":"2024-04-29T14:18:49.152425+0000","last_active":"2024-04-29T14:19:20.996332+0000","last_peered":"2024-04-29T14:19:20.996332+0000","last_clean":"2024-04-29T14:19:20.996332+0000","last_became_active":"2024-04-29T14:18:49.152097+0000","last_became_peered":"2024-04-29T14:18:49.152097+0000","last_unstale":"2024-04-29T14:19:20.996332+0000","last_undegraded":"2024-04-29T14:19:20.996332+0000","last_fullsized":"2024-04-29T14:19:20.996332+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:48.475752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286195712,"num_objects":4367,"num_object_clones":0,"num_object_copies":8734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4367,"num_write_kb":279488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"335'4188","reported_seq":4221,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.506952+0000","last_change":"2024-04-29T14:18:49.156745+0000","last_active":"2024-04-29T14:19:20.506952+0000","last_peered":"2024-04-29T14:19:20.506952+0000","last_clean":"2024-04-29T14:19:20.506952+0000","last_became_active":"2024-04-29T14:18:49.156474+0000","last_became_peered":"2024-04-29T14:18:49.156474+0000","last_unstale":"2024-04-29T14:19:20.506952+0000","last_undegraded":"2024-04-29T14:19:20.506952+0000","last_fullsized":"2024-04-29T14:19:20.506952+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.931388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274464768,"num_objects":4188,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4188,"num_write_kb":268032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"335'4139","reported_seq":4171,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.529378+0000","last_change":"2024-04-29T14:18:49.371513+0000","last_active":"2024-04-29T14:19:20.529378+0000","last_peered":"2024-04-29T14:19:20.529378+0000","last_clean":"2024-04-29T14:19:20.529378+0000","last_became_active":"2024-04-29T14:18:49.371178+0000","last_became_peered":"2024-04-29T14:18:49.371178+0000","last_unstale":"2024-04-29T14:19:20.529378+0000","last_undegraded":"2024-04-29T14:19:20.529378+0000","last_fullsized":"2024-04-29T14:19:20.529378+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.236063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271253504,"num_objects":4139,"num_object_clones":0,"num_object_copies":8278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4139,"num_write_kb":264896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"335'4276","reported_seq":4309,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.839711+0000","last_change":"2024-04-29T14:18:49.370740+0000","last_active":"2024-04-29T14:19:20.839711+0000","last_peered":"2024-04-29T14:19:20.839711+0000","last_clean":"2024-04-29T14:19:20.839711+0000","last_became_active":"2024-04-29T14:18:49.370541+0000","last_became_peered":"2024-04-29T14:18:49.370541+0000","last_unstale":"2024-04-29T14:19:20.839711+0000","last_undegraded":"2024-04-29T14:19:20.839711+0000","last_fullsized":"2024-04-29T14:19:20.839711+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:54.263953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280231936,"num_objects":4276,"num_object_clones":0,"num_object_copies":8552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":273664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"336'4453","reported_seq":4488,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-29T14:19:22.585611+0000","last_change":"2024-04-29T14:18:49.151635+0000","last_active":"2024-04-29T14:19:22.585611+0000","last_peered":"2024-04-29T14:19:22.585611+0000","last_clean":"2024-04-29T14:19:22.585611+0000","last_became_active":"2024-04-29T14:18:49.151382+0000","last_became_peered":"2024-04-29T14:18:49.151382+0000","last_unstale":"2024-04-29T14:19:22.585611+0000","last_undegraded":"2024-04-29T14:19:22.585611+0000","last_fullsized":"2024-04-29T14:19:22.585611+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:56.402851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291831808,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":284992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"335'4237","reported_seq":4269,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.453019+0000","last_change":"2024-04-29T14:18:49.157325+0000","last_active":"2024-04-29T14:19:20.453019+0000","last_peered":"2024-04-29T14:19:20.453019+0000","last_clean":"2024-04-29T14:19:20.453019+0000","last_became_active":"2024-04-29T14:18:49.157084+0000","last_became_peered":"2024-04-29T14:18:49.157084+0000","last_unstale":"2024-04-29T14:19:20.453019+0000","last_undegraded":"2024-04-29T14:19:20.453019+0000","last_fullsized":"2024-04-29T14:19:20.453019+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:49.092162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277676032,"num_objects":4237,"num_object_clones":0,"num_object_copies":8474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4237,"num_write_kb":271168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7],"acting":[9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"336'4285","reported_seq":4320,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-29T14:19:22.563104+0000","last_change":"2024-04-29T14:18:49.156653+0000","last_active":"2024-04-29T14:19:22.563104+0000","last_peered":"2024-04-29T14:19:22.563104+0000","last_clean":"2024-04-29T14:19:22.563104+0000","last_became_active":"2024-04-29T14:18:49.156484+0000","last_became_peered":"2024-04-29T14:18:49.156484+0000","last_unstale":"2024-04-29T14:19:22.563104+0000","last_undegraded":"2024-04-29T14:19:22.563104+0000","last_fullsized":"2024-04-29T14:19:22.563104+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:54.124339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280821760,"num_objects":4285,"num_object_clones":0,"num_object_copies":8570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4285,"num_write_kb":274240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"335'4267","reported_seq":4300,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.854843+0000","last_change":"2024-04-29T14:18:49.371657+0000","last_active":"2024-04-29T14:19:20.854843+0000","last_peered":"2024-04-29T14:19:20.854843+0000","last_clean":"2024-04-29T14:19:20.854843+0000","last_became_active":"2024-04-29T14:18:49.371335+0000","last_became_peered":"2024-04-29T14:18:49.371335+0000","last_unstale":"2024-04-29T14:19:20.854843+0000","last_undegraded":"2024-04-29T14:19:20.854843+0000","last_fullsized":"2024-04-29T14:19:20.854843+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:56.673106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279642112,"num_objects":4267,"num_object_clones":0,"num_object_copies":8534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4267,"num_write_kb":273088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"335'4272","reported_seq":4305,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.445916+0000","last_change":"2024-04-29T14:18:49.158643+0000","last_active":"2024-04-29T14:19:20.445916+0000","last_peered":"2024-04-29T14:19:20.445916+0000","last_clean":"2024-04-29T14:19:20.445916+0000","last_became_active":"2024-04-29T14:18:49.158456+0000","last_became_peered":"2024-04-29T14:18:49.158456+0000","last_unstale":"2024-04-29T14:19:20.445916+0000","last_undegraded":"2024-04-29T14:19:20.445916+0000","last_fullsized":"2024-04-29T14:19:20.445916+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:13.613295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279969792,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":273408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"335'4374","reported_seq":4407,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.343123+0000","last_change":"2024-04-29T14:18:49.371803+0000","last_active":"2024-04-29T14:19:20.343123+0000","last_peered":"2024-04-29T14:19:20.343123+0000","last_clean":"2024-04-29T14:19:20.343123+0000","last_became_active":"2024-04-29T14:18:49.371468+0000","last_became_peered":"2024-04-29T14:18:49.371468+0000","last_unstale":"2024-04-29T14:19:20.343123+0000","last_undegraded":"2024-04-29T14:19:20.343123+0000","last_fullsized":"2024-04-29T14:19:20.343123+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:17.465842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286654464,"num_objects":4374,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":279936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"335'4341","reported_seq":4374,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.872761+0000","last_change":"2024-04-29T14:18:49.158273+0000","last_active":"2024-04-29T14:19:20.872761+0000","last_peered":"2024-04-29T14:19:20.872761+0000","last_clean":"2024-04-29T14:19:20.872761+0000","last_became_active":"2024-04-29T14:18:49.158015+0000","last_became_peered":"2024-04-29T14:18:49.158015+0000","last_unstale":"2024-04-29T14:19:20.872761+0000","last_undegraded":"2024-04-29T14:19:20.872761+0000","last_fullsized":"2024-04-29T14:19:20.872761+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4341,"log_dups_size":0,"ondisk_log_size":4341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:53.632841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284491776,"num_objects":4341,"num_object_clones":0,"num_object_copies":8682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4341,"num_write_kb":277824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"335'4152","reported_seq":4185,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:20.517925+0000","last_change":"2024-04-29T14:18:49.156792+0000","last_active":"2024-04-29T14:19:20.517925+0000","last_peered":"2024-04-29T14:19:20.517925+0000","last_clean":"2024-04-29T14:19:20.517925+0000","last_became_active":"2024-04-29T14:18:49.156512+0000","last_became_peered":"2024-04-29T14:18:49.156512+0000","last_unstale":"2024-04-29T14:19:20.517925+0000","last_undegraded":"2024-04-29T14:19:20.517925+0000","last_fullsized":"2024-04-29T14:19:20.517925+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:18:48.136527+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:08.011441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272105472,"num_objects":4152,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4152,"num_write_kb":265728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":869,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-04-29T14:19:17.079073+0000","last_change":"2024-04-29T14:18:49.855444+0000","last_active":"2024-04-29T14:19:17.079073+0000","last_peered":"2024-04-29T14:19:17.079073+0000","last_clean":"2024-04-29T14:19:17.079073+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:19:17.079073+0000","last_undegraded":"2024-04-29T14:19:17.079073+0000","last_fullsized":"2024-04-29T14:19:17.079073+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:18:49.855310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:18:49.855310+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:20:02.043752+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4475584512,"num_objects":68292,"num_object_clones":0,"num_object_copies":136584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68292,"num_write_kb":4370688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8970698752,"data_stored":8970698752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68292,"ondisk_log_size":68292,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122197,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1670044,"kb_used_data":1354260,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92075108,"statfs":{"total":95995035648,"available":94284910592,"internally_reserved":0,"allocated":1386762240,"data_stored":1386312096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":323353028},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1562788,"kb_used_data":1409432,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92182364,"statfs":{"total":95995035648,"available":94394740736,"internally_reserved":0,"allocated":1443258368,"data_stored":1442805462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":157022656},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645471,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337876,"kb_used_data":1067916,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92407276,"statfs":{"total":95995035648,"available":94625050624,"internally_reserved":0,"allocated":1093545984,"data_stored":1093099412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":276429248},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234151,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1710876,"kb_used_data":1360404,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":92034276,"statfs":{"total":95995035648,"available":94243098624,"internally_reserved":0,"allocated":1393053696,"data_stored":1392603572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":358873546},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528493,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1519444,"kb_used_data":1055116,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92225708,"statfs":{"total":95995035648,"available":94439124992,"internally_reserved":0,"allocated":1080438784,"data_stored":1079992212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":475462086},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099668,"kb_used_data":807500,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92645484,"statfs":{"total":95995035648,"available":94868975616,"internally_reserved":0,"allocated":826880000,"data_stored":826433428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182590,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986360,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574676,"kb_used_data":1092300,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92170476,"statfs":{"total":95995035648,"available":94382567424,"internally_reserved":0,"allocated":1118515200,"data_stored":1118068628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822833,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798996,"kb_used_data":551244,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92946156,"statfs":{"total":95995035648,"available":95176863744,"internally_reserved":0,"allocated":564473856,"data_stored":564027284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":850526208,"data_stored":850526208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1119223808,"data_stored":1119223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":565772288,"data_stored":565772288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1097072640,"data_stored":1097072640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1394081792,"data_stored":1394081792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1106968576,"data_stored":1106968576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1444610048,"data_stored":1444610048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1392443392,"data_stored":1392443392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:19:27.034 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:19:27.034 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-29T14:19:27.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:27.091 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:27.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:27.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:27.234 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:27.285 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:27.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 15 80352 80337 167.351 111.625 0.0129609 0.00597368 2024-04-29T14:19:27.312 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:27.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:27.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:27.448 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:27.477 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:27.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:27.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:27.634 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:27.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:27.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:27.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:27.822 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:27.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:27.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:27.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:28.011 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:28.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:28.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:28.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:19:28.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:28.177 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:28.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 16 82114 82098 165.503 110.062 0.00629767 0.00604023 2024-04-29T14:19:28.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:28 smithi040 ceph-mon[39581]: osdmap e338: 12 total, 11 up, 8 in 2024-04-29T14:19:28.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:28.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:28.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:28 smithi121 ceph-mon[44533]: osdmap e338: 12 total, 11 up, 8 in 2024-04-29T14:19:28.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:28.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:28 smithi194 ceph-mon[44089]: osdmap e338: 12 total, 11 up, 8 in 2024-04-29T14:19:28.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:28.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:28.635 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:28.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:28.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:28.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:28.793 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:28.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:28.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:28.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:28.957 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:29.081 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:29.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:29.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:29.233 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:29.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:29.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:29.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 83866 83850 163.752 109.5 0.0118724 0.00610525 2024-04-29T14:19:29.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:29 smithi040 ceph-mon[39581]: osdmap e339: 12 total, 11 up, 8 in 2024-04-29T14:19:29.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:29 smithi040 ceph-mon[39581]: pgmap v891: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:19:29.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:29.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:29.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:29 smithi121 ceph-mon[44533]: osdmap e339: 12 total, 11 up, 8 in 2024-04-29T14:19:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:29 smithi121 ceph-mon[44533]: pgmap v891: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:19:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:29.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:29.416 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:29.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:29.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:29.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:29.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:29 smithi194 ceph-mon[44089]: osdmap e339: 12 total, 11 up, 8 in 2024-04-29T14:19:29.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:29 smithi194 ceph-mon[44089]: pgmap v891: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:19:29.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:29.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:29.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:19:29.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:19:29.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:19:29.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:19:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:29.722 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:19:29.746 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:19:29.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:19:29.789 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:19:29.817 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:19:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:19:29.860 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:19:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:19:30.287 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:30 smithi040 ceph-mon[39581]: osdmap e340: 12 total, 11 up, 8 in 2024-04-29T14:19:30.287 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/468232216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:30.287 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.287 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.287 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 85611 85595 162.095 109.062 0.0129788 0.00616741 2024-04-29T14:19:30.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:30 smithi121 ceph-mon[44533]: osdmap e340: 12 total, 11 up, 8 in 2024-04-29T14:19:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/468232216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:30 smithi194 ceph-mon[44089]: osdmap e340: 12 total, 11 up, 8 in 2024-04-29T14:19:30.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/468232216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.099 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.100 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.100 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.100 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:31 smithi040 ceph-mon[39581]: pgmap v893: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 122 MiB/s wr, 1.94k op/s 2024-04-29T14:19:31.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 16 87355 87339 160.533 109 0.00711735 0.00622782 2024-04-29T14:19:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:31 smithi121 ceph-mon[44533]: pgmap v893: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 122 MiB/s wr, 1.94k op/s 2024-04-29T14:19:31.392 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:19:31.392 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:31.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:31 smithi194 ceph-mon[44089]: pgmap v893: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 122 MiB/s wr, 1.94k op/s 2024-04-29T14:19:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:32.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:32.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 89079 89063 159.025 107.75 0.0141833 0.00628643 2024-04-29T14:19:32.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:32 smithi040 ceph-mon[39581]: osdmap e341: 12 total, 11 up, 8 in 2024-04-29T14:19:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:32 smithi040 ceph-mon[39581]: osdmap e342: 12 total, 11 up, 8 in 2024-04-29T14:19:32.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:32.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:32.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:32 smithi121 ceph-mon[44533]: osdmap e341: 12 total, 11 up, 8 in 2024-04-29T14:19:32.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:32 smithi121 ceph-mon[44533]: osdmap e342: 12 total, 11 up, 8 in 2024-04-29T14:19:32.377 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:32.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:32.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:32.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:32 smithi194 ceph-mon[44089]: osdmap e341: 12 total, 11 up, 8 in 2024-04-29T14:19:32.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:32 smithi194 ceph-mon[44089]: osdmap e342: 12 total, 11 up, 8 in 2024-04-29T14:19:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:19:32.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:19:32.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:19:33.228 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:33.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:33.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 90764 90748 157.533 105.312 0.00736704 0.00634622 2024-04-29T14:19:33.332 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:33.360 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:33 smithi040 ceph-mon[39581]: pgmap v896: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-29T14:19:33.363 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:33.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:33 smithi121 ceph-mon[44533]: pgmap v896: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-29T14:19:33.562 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:33 smithi194 ceph-mon[44089]: pgmap v896: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-29T14:19:33.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:33.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:33.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:33.699 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:33.799 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:33.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:33.904 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:33.933 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:34.060 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:34.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:34.123 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:19:34.164 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:34.201 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:34.233 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:34.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:34.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 92426 92410 156.082 103.875 0.00903415 0.00640554 2024-04-29T14:19:34.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:34.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/796992168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:34.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/796992168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:34.392 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:34.420 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:34.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:34.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/796992168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:34.582 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:34.608 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:34.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:34.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:19:34.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:34.759 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:34.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:34.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:34.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:34.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:34.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:35.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:35.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:35.138 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:35.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:35.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:35.291 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:35.302 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 93895 93879 154.391 91.8125 0.0162061 0.00647561 2024-04-29T14:19:35.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:35.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:35.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:35 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:19:35.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/796992168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:35.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:35 smithi040 ceph-mon[39581]: osdmap e343: 12 total, 11 up, 8 in 2024-04-29T14:19:35.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:35 smithi040 ceph-mon[39581]: pgmap v898: 17 pgs: 17 active+clean; 5.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-04-29T14:19:35.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:35 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:19:35.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/796992168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:35.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:35 smithi121 ceph-mon[44533]: osdmap e343: 12 total, 11 up, 8 in 2024-04-29T14:19:35.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:35 smithi121 ceph-mon[44533]: pgmap v898: 17 pgs: 17 active+clean; 5.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-04-29T14:19:35.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:35.469 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:35.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:35.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:35 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:19:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/796992168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:35 smithi194 ceph-mon[44089]: osdmap e343: 12 total, 11 up, 8 in 2024-04-29T14:19:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:35 smithi194 ceph-mon[44089]: pgmap v898: 17 pgs: 17 active+clean; 5.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-04-29T14:19:35.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:35.634 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:36.002 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:36.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:36.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:36.155 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:36.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:36.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:36.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:36.306 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 95504 95488 153.01 100.562 0.00343089 0.00653392 2024-04-29T14:19:36.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:36.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:36.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:36 smithi040 ceph-mon[39581]: osdmap e344: 12 total, 11 up, 8 in 2024-04-29T14:19:36.337 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:36.340 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:19:36.340 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:19:36.359 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:36.363 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:36.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:36.468 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:36.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:36.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:36.508 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:36 smithi194 ceph-mon[44089]: osdmap e344: 12 total, 11 up, 8 in 2024-04-29T14:19:36.510 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:36.541 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:36.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:36 smithi121 ceph-mon[44533]: osdmap e344: 12 total, 11 up, 8 in 2024-04-29T14:19:36.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:36.703 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:36.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:36.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:36.839 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:37.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:19:37.310877 min lat: 0.00115367 max lat: 0.220776 avg lat: 0.00659107 2024-04-29T14:19:37.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:19:37.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 97106 97090 151.688 100.125 0.00591202 0.00659107 2024-04-29T14:19:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:37 smithi194 ceph-mon[44089]: pgmap v900: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-04-29T14:19:37.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:37 smithi040 ceph-mon[39581]: pgmap v900: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-04-29T14:19:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:37 smithi121 ceph-mon[44533]: pgmap v900: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-04-29T14:19:37.720 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:37.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:37.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:19:37.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:19:37.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:37.871 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:37.907 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:37.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:38.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:38.046 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:38.197 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:38.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:38.258 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:38 smithi194 ceph-mon[44089]: pgmap v901: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-29T14:19:38.301 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:38.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 98709 98693 150.432 100.188 0.00734142 0.00664576 2024-04-29T14:19:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:38.346 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:38.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:38 smithi040 ceph-mon[39581]: pgmap v901: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-29T14:19:38.373 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:38.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:38.422 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:38.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:38.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:38 smithi121 ceph-mon[44533]: pgmap v901: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-29T14:19:38.528 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:38.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:38.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:38.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:38.694 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:38.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:38.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:38.884 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:38.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:39.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:39.062 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:39.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:39.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:39.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:39.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 15 100223 100208 149.104 94.6875 0.0111143 0.00670538 2024-04-29T14:19:39.392 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:39.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:39.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:39.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:39.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:39.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:39.605 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:19:39.625 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:19:39.647 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:19:39.669 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:19:39.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:39.698 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:19:39.727 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:19:39.739 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:39.753 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:19:39.774 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:39.775 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:19:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:39.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:39.797 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:19:39.818 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:19:39.842 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:19:39.842 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:19:39.842 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:19:39.843 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:19:39.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1409765527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:39.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:39.925 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:39.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:39.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:40.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:40.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1409765527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:40.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1409765527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:40.116 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:40.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:40.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:40.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:40.276 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:40.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 16 101740 101724 147.84 94.75 0.0151319 0.00676243 2024-04-29T14:19:40.316 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:40.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:40.430 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:40.470 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:40.498 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:40.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:40.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:40.633 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:40.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:40.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:40 smithi194 ceph-mon[44089]: pgmap v902: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-29T14:19:41.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:40 smithi040 ceph-mon[39581]: pgmap v902: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-29T14:19:41.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:40 smithi121 ceph-mon[44533]: pgmap v902: 17 pgs: 17 active+clean; 5.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-29T14:19:41.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 15 103141 103126 146.471 87.625 0.0172124 0.00682596 2024-04-29T14:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:41.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:41.490 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:41.537 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:41.561 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:41.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:41.667 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:41.715 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:41.741 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:41.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:41.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:41.848 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:41.888 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:41.914 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:41.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:42.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:42.063 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:42.063 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:42.063 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:42.066 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:42.093 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:42.093 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:42.093 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:42.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:42.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:42.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:42.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 104609 104593 145.254 91.6875 0.00895522 0.00688318 2024-04-29T14:19:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:19:42.575 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:19:42.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:42 smithi040 ceph-mon[39581]: pgmap v903: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-04-29T14:19:42.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:19:43.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:42 smithi194 ceph-mon[44089]: pgmap v903: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-04-29T14:19:43.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:19:43.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:42 smithi121 ceph-mon[44533]: pgmap v903: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-04-29T14:19:43.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:19:43.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 106001 105985 143.987 87 0.0062533 0.00694356 2024-04-29T14:19:43.581 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:43.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:43.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:43.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:19:43.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:19:43.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:43 smithi040 ceph-mon[39581]: pgmap v904: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:19:43.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:43 smithi040 ceph-mon[39581]: pgmap v905: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-29T14:19:43.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:19:43.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3614171965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:19:43.716 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:43.818 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:43.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:43.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:19:44.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:19:44.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:19:44.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:43 smithi194 ceph-mon[44089]: pgmap v904: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:19:44.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:43 smithi194 ceph-mon[44089]: pgmap v905: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-29T14:19:44.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:19:44.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3614171965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:19:44.123 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:19:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:19:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:19:44.123 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:19:44.123 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:19:44.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:19:44.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:19:44.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:43 smithi121 ceph-mon[44533]: pgmap v904: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:19:44.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:43 smithi121 ceph-mon[44533]: pgmap v905: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-29T14:19:44.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:19:44.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3614171965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:19:44.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 107483 107467 142.894 92.625 0.0143256 0.00699643 2024-04-29T14:19:44.575 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:19:44.602 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:44.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:44.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:44.759 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:44.795 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:44.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:44.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:44 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:44.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3614171965' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:19:44.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:44 smithi040 ceph-mon[39581]: osdmap e345: 12 total, 11 up, 8 in 2024-04-29T14:19:44.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:44.948 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:44 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:44.948 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3614171965' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:19:44.948 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:44 smithi121 ceph-mon[44533]: osdmap e345: 12 total, 11 up, 8 in 2024-04-29T14:19:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:44.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:45.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:44 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:45.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3614171965' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:19:45.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:44 smithi194 ceph-mon[44089]: osdmap e345: 12 total, 11 up, 8 in 2024-04-29T14:19:45.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:45.124 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:45.181 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:45.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:45.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:19:45.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:45.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 16 108947 108931 141.823 91.5 0.0072424 0.00704976 2024-04-29T14:19:45.328 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:45.671 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:45.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:45.693 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:45.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:45.834 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:45.870 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:45.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:45.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:45 smithi040 ceph-mon[39581]: pgmap v907: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:19:45.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:19:46.017 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:45 smithi121 ceph-mon[44533]: pgmap v907: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:19:46.020 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:46.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:19:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:46.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:19:46.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:45 smithi194 ceph-mon[44089]: pgmap v907: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:19:46.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:46.188 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:46.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 110394 110378 140.774 90.4375 0.0112508 0.00710178 2024-04-29T14:19:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:46.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:46.944 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:46.987 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:47.016 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:47.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:47.122 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:47.171 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:47.197 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:47.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:47.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:47.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 111835 111819 139.76 90.0625 0.0111597 0.00715361 2024-04-29T14:19:47.329 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:47.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:19:48.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:47 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:48.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:47 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:48.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:47 smithi194 ceph-mon[44089]: osdmap e346: 12 total, 11 up, 8 in 2024-04-29T14:19:48.066 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:47 smithi194 ceph-mon[44089]: pgmap v909: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:19:48.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:47 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:48.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:47 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:48.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:47 smithi040 ceph-mon[39581]: osdmap e346: 12 total, 11 up, 8 in 2024-04-29T14:19:48.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:47 smithi040 ceph-mon[39581]: pgmap v909: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:19:48.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:47 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:48.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:47 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:48.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:47 smithi121 ceph-mon[44533]: osdmap e346: 12 total, 11 up, 8 in 2024-04-29T14:19:48.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:47 smithi121 ceph-mon[44533]: pgmap v909: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:19:48.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 113027 113011 138.48 74.5 0.00939174 0.0072195 2024-04-29T14:19:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:48.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:48.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:48.620 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:48.870 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:19:48.890 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:19:48.913 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:19:48.943 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:19:48.969 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:48.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:48.980 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:19:49.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:19:49.029 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:19:49.053 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:19:49.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:49.075 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:19:49.098 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:19:49.123 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:49.123 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:19:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:19:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:19:49.148 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:49.153 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2493835880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:49.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:49.230 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:19:49.231 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:19:49.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2493835880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:49.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:49.295 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:49.303 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 114439 114423 137.514 88.25 0.00905132 0.00727056 2024-04-29T14:19:49.328 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:49.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:49.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2493835880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:49.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:49.467 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:49.662 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:49.669 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:49.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:49.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:49.825 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:49.856 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:49.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:49.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:49.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.988 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: pgmap v910: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:19:49.988 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.988 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.988 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.988 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.988 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:49.989 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:19:50.223 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:19:50.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 16 115835 115819 136.566 87.25 0.016053 0.00732084 2024-04-29T14:19:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: pgmap v910: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:19:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: pgmap v910: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:50.538 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:50.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:50.642 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:50.671 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:50.693 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:50.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:50.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:50.829 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:51.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:19:51.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 117142 117126 135.55 81.6875 0.0121193 0.00737571 2024-04-29T14:19:51.898 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:51.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:52.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:52.034 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:52.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:19:52.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:19:52.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:51 smithi040 ceph-mon[39581]: osdmap e347: 12 total, 11 up, 8 in 2024-04-29T14:19:52.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:51 smithi040 ceph-mon[39581]: pgmap v912: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-29T14:19:52.184 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:52.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:52.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:19:52.289 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:52.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 118523 118507 134.654 86.3125 0.0123816 0.00742472 2024-04-29T14:19:52.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:19:52.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:19:52.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:51 smithi194 ceph-mon[44089]: osdmap e347: 12 total, 11 up, 8 in 2024-04-29T14:19:52.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:51 smithi194 ceph-mon[44089]: pgmap v912: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-29T14:19:52.328 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:52.357 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:52.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:52.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:19:52.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:19:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:51 smithi121 ceph-mon[44533]: osdmap e347: 12 total, 11 up, 8 in 2024-04-29T14:19:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:51 smithi121 ceph-mon[44533]: pgmap v912: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-29T14:19:52.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:52.493 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:52.502 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:52.528 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:52.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:52.550 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:52.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:52.665 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:52.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:52.894 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:52.931 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:52.956 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:52.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:53.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:53.089 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:19:53.130 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:53.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:53.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:19:53.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:53 smithi040 ceph-mon[39581]: osdmap e348: 12 total, 11 up, 8 in 2024-04-29T14:19:53.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4219450389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:53.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 119849 119833 133.729 82.875 0.010244 0.0074765 2024-04-29T14:19:53.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:53 smithi194 ceph-mon[44089]: osdmap e348: 12 total, 11 up, 8 in 2024-04-29T14:19:53.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4219450389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:53 smithi121 ceph-mon[44533]: osdmap e348: 12 total, 11 up, 8 in 2024-04-29T14:19:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4219450389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:19:54.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 16 121181 121165 132.843 83.25 0.0175278 0.0075257 2024-04-29T14:19:54.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:54 smithi040 ceph-mon[39581]: pgmap v914: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:19:54.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:54 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:54.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4219450389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:54.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:54 smithi040 ceph-mon[39581]: osdmap e349: 12 total, 11 up, 8 in 2024-04-29T14:19:54.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:54 smithi121 ceph-mon[44533]: pgmap v914: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:19:54.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:54 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:54.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4219450389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:54.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:54 smithi121 ceph-mon[44533]: osdmap e349: 12 total, 11 up, 8 in 2024-04-29T14:19:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:54 smithi194 ceph-mon[44089]: pgmap v914: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:19:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:54 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:19:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4219450389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:19:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:54 smithi194 ceph-mon[44089]: osdmap e349: 12 total, 11 up, 8 in 2024-04-29T14:19:55.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:19:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:55.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:19:55.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:19:55.154 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:55.282 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:19:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:55.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:19:55.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 122549 122533 132.027 85.5 0.00476635 0.00757196 2024-04-29T14:19:55.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:19:55.445 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:55.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:19:55.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:55.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:19:55.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:55.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:55.683 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:19:55.683 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:19:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:55.686 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:19:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:55.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:55.707 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:19:55.730 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:19:55.756 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:19:55.783 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:19:55.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:55.806 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:19:55.827 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:55.829 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:19:55.855 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:19:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:55.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:19:55.877 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:19:55.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:19:55.878 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:19:55.961 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:55.998 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:56.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 123883 123867 131.202 83.375 0.0145646 0.00762004 2024-04-29T14:19:56.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:56 smithi040 ceph-mon[39581]: pgmap v916: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-29T14:19:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3952137677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:56 smithi121 ceph-mon[44533]: pgmap v916: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-29T14:19:56.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3952137677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:56.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.414 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:19:56.414 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:19:56.414 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:19:56.414 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:19:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:56 smithi194 ceph-mon[44089]: pgmap v916: 17 pgs: 17 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-29T14:19:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3952137677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:19:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:56.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:57.154 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:57.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:57.255 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:57.301 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:57.304 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:19:57.312660 min lat: 0.00115367 max lat: 0.225314 avg lat: 0.00766775 2024-04-29T14:19:57.305 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:19:57.305 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 125199 125183 130.386 82.25 0.0198307 0.00766775 2024-04-29T14:19:57.325 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:57.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:57.474 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:57.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:57.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:19:57.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:57.654 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:57.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:57.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:19:57.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:57.939 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:57.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:19:58.026 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:19:58.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:19:58.134 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:58.170 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:19:58.172 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:58.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:19:58.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:19:58.305 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 126496 126480 129.578 81.0625 0.00658061 0.00771548 2024-04-29T14:19:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:58.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:19:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:58 smithi040 ceph-mon[39581]: pgmap v917: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:19:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:19:58 smithi040 ceph-mon[39581]: osdmap e350: 12 total, 11 up, 8 in 2024-04-29T14:19:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:58.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:19:58.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:58 smithi121 ceph-mon[44533]: pgmap v917: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:19:58.365 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:19:58 smithi121 ceph-mon[44533]: osdmap e350: 12 total, 11 up, 8 in 2024-04-29T14:19:58.465 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:58.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:58 smithi194 ceph-mon[44089]: pgmap v917: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:19:58.506 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:19:58 smithi194 ceph-mon[44089]: osdmap e350: 12 total, 11 up, 8 in 2024-04-29T14:19:58.506 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:58.543 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:58.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:58.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:19:58.680 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:19:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:58.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:19:58.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:19:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:59.037 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:19:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:59.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:19:59.141 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:59.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:19:59.184 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:59.223 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:59.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:59.305 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 16 127735 127719 128.737 77.4375 0.00558496 0.00776664 2024-04-29T14:19:59.331 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:19:59.370 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:59.403 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:19:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:59.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:19:59.511 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:59.558 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:59.584 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:59.590 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:19:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:19:59.691 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:19:59.727 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:59.765 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:19:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:59.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:19:59.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:19:59.921 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:19:59.948 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:19:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:19:59.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:00.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:20:00.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:00.115 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:00.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:00.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:00.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:00.286 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:00 smithi040 ceph-mon[39581]: pgmap v919: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:20:00.286 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:00 smithi040 ceph-mon[39581]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub,nodeep-scrub flag(s) set 2024-04-29T14:20:00.286 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:00 smithi040 ceph-mon[39581]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-29T14:20:00.286 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:00 smithi040 ceph-mon[39581]: daemon osd.2 on smithi040 is in unknown state 2024-04-29T14:20:00.286 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:00 smithi040 ceph-mon[39581]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-29T14:20:00.305 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 128989 128973 127.937 78.375 0.00442704 0.0078149 2024-04-29T14:20:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:00 smithi194 ceph-mon[44089]: pgmap v919: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:20:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:00 smithi194 ceph-mon[44089]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub,nodeep-scrub flag(s) set 2024-04-29T14:20:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:00 smithi194 ceph-mon[44089]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-29T14:20:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:00 smithi194 ceph-mon[44089]: daemon osd.2 on smithi040 is in unknown state 2024-04-29T14:20:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:00 smithi194 ceph-mon[44089]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-29T14:20:00.315 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:00.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:00.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:00 smithi121 ceph-mon[44533]: pgmap v919: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:20:00.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:00 smithi121 ceph-mon[44533]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub,nodeep-scrub flag(s) set 2024-04-29T14:20:00.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:00 smithi121 ceph-mon[44533]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-29T14:20:00.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:00 smithi121 ceph-mon[44533]: daemon osd.2 on smithi040 is in unknown state 2024-04-29T14:20:00.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:00 smithi121 ceph-mon[44533]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-29T14:20:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:00.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:00.416 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:20:00.416 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:20:00.417 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:20:00.417 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:20:00.417 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:20:00.457 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:00.499 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:00.529 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:00.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:00.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:00.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:00.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:00.829 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:00.884 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:00.902 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:00.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:01.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:01.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:01.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:01.305 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 16 130237 130221 127.157 78 0.00380315 0.00786277 2024-04-29T14:20:01.328 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:01.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:01.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:01.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:01.509 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:01.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:01.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:01.647 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:01.692 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:01.724 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:01.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:01.755 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:01.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:20:01.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:01.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:02.034 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:02.076 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:02.104 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:02.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:02.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:02.308 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 131487 131471 126.396 78.125 0.00903684 0.00791052 2024-04-29T14:20:02.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:02 smithi040 ceph-mon[39581]: pgmap v920: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-29T14:20:02.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:02 smithi121 ceph-mon[44533]: pgmap v920: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-29T14:20:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:02.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:02.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:02 smithi194 ceph-mon[44089]: pgmap v920: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-29T14:20:02.593 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:02.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:02.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:02.725 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:02.760 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:02.787 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:02.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:02.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:02.933 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:02.971 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:02.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:03.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:03.115 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:03.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:03.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:03.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:03.308 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 16 132744 132728 125.671 78.5625 0.0185238 0.00795597 2024-04-29T14:20:03.332 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:03.378 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:03.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:03.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:03.514 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:04.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:04 smithi040 ceph-mon[39581]: pgmap v921: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-04-29T14:20:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:04.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:04.196 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:04.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:04.308 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 15 134016 134001 124.983 79.5625 0.01864 0.00799973 2024-04-29T14:20:04.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:04 smithi121 ceph-mon[44533]: pgmap v921: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-04-29T14:20:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:04.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:04.562 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:04.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:04 smithi194 ceph-mon[44089]: pgmap v921: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-04-29T14:20:04.616 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:04.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:04.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:04.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:04.799 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:04.830 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:04.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:04.937 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:04.969 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:05.045 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:20:05.045 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:20:05.045 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:20:05.046 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:20:05.051 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:20:05.068 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:05.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:05.074 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:20:05.101 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:20:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:20:05.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:20:05.156 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:20:05.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:05.181 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:20:05.204 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:20:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:20:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:20:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:20:05.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/635671364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:20:05.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2940050044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:05.218 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:05.246 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:05.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:05.309 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 135249 135233 124.277 77 0.0112272 0.00804491 2024-04-29T14:20:05.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:05.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/635671364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:20:05.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2940050044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:05.402 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:05.428 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:05.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:05.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/635671364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:20:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2940050044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:05.578 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:05.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:05.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:05.728 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:05.767 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:05.795 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:05.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:05.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:05.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:05.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:06.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:06.123 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:06.157 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:06.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:06.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:06.309 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 136413 136397 123.53 72.75 0.0160919 0.00809044 2024-04-29T14:20:06.317 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: pgmap v922: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:20:06.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.318 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:06.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/635671364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:20:06.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osdmap e351: 12 total, 11 up, 8 in 2024-04-29T14:20:06.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.321 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:06.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:06.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:06.514 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:06.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:06.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: pgmap v922: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:20:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/635671364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osdmap e351: 12 total, 11 up, 8 in 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: pgmap v922: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/635671364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osdmap e351: 12 total, 11 up, 8 in 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:06.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:06.685 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:06.709 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:06.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:06.818 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:06.875 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:06.904 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:06.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:07.012 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:07.035 INFO:tasks.thrashosds.thrasher:Reviving osd 8 2024-04-29T14:20:07.036 INFO:tasks.cephadm.osd.8:Restarting osd.8 (starting--it wasn't running)... 2024-04-29T14:20:07.036 DEBUG:teuthology.orchestra.run.smithi194:osd.8> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.8.service 2024-04-29T14:20:07.038 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl start ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.8 2024-04-29T14:20:07.043 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:07.088 INFO:journalctl@ceph.osd.8.smithi194.stdout:-- Logs begin at Mon 2024-04-29 13:48:49 UTC. -- 2024-04-29T14:20:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:07.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:07.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:07.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:07 smithi040 ceph-mon[39581]: 4.9 scrub starts 2024-04-29T14:20:07.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:07 smithi040 ceph-mon[39581]: 4.4 scrub starts 2024-04-29T14:20:07.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:07 smithi040 ceph-mon[39581]: 4.0 scrub starts 2024-04-29T14:20:07.303 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:07.309 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 137364 137348 122.614 59.4375 0.00779097 0.00815405 2024-04-29T14:20:07.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:07 smithi194 systemd[1]: Starting Ceph osd.8 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:20:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:07 smithi194 ceph-mon[44089]: 4.9 scrub starts 2024-04-29T14:20:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:07 smithi194 ceph-mon[44089]: 4.4 scrub starts 2024-04-29T14:20:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:07 smithi194 ceph-mon[44089]: 4.0 scrub starts 2024-04-29T14:20:07.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:07 smithi121 ceph-mon[44533]: 4.9 scrub starts 2024-04-29T14:20:07.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:07 smithi121 ceph-mon[44533]: 4.4 scrub starts 2024-04-29T14:20:07.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:07 smithi121 ceph-mon[44533]: 4.0 scrub starts 2024-04-29T14:20:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:20:07.686 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:20:08.046 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:20:08.046 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:20:08.054 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:08.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:08.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:08.187 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:08.309 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 16 138324 138308 121.733 60 0.0134708 0.00821306 2024-04-29T14:20:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:08.338 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:08 smithi040 ceph-mon[39581]: pgmap v924: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-04-29T14:20:08.338 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:08 smithi040 ceph-mon[39581]: 4.8 scrub starts 2024-04-29T14:20:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:08.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:08.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:08.443 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:08 smithi194 ceph-mon[44089]: pgmap v924: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-04-29T14:20:08.444 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:08 smithi194 ceph-mon[44089]: 4.8 scrub starts 2024-04-29T14:20:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:08.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:08 smithi121 ceph-mon[44533]: pgmap v924: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-04-29T14:20:08.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:08 smithi121 ceph-mon[44533]: 4.8 scrub starts 2024-04-29T14:20:08.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:20:08.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:20:08.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-29T14:20:08.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-29T14:20:08.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-8 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-29T14:20:08.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-29T14:20:08.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-29T14:20:08.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-8/block 2024-04-29T14:20:08.813 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-29T14:20:08.814 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:08 smithi194 bash[106274]: --> ceph-volume raw activate successful for osd ID: 8 2024-04-29T14:20:08.820 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:08.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:08.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:08.955 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:08.979 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:08.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:09.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:09.136 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:09.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:09.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:09.310 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 139291 139275 120.88 60.4375 0.0405799 0.00827088 2024-04-29T14:20:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:09 smithi194 ceph-mon[44089]: 4.4 scrub ok 2024-04-29T14:20:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:09 smithi194 ceph-mon[44089]: 4.9 scrub ok 2024-04-29T14:20:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:09 smithi194 ceph-mon[44089]: 4.a scrub starts 2024-04-29T14:20:09.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:09 smithi194 ceph-mon[44089]: 4.0 scrub ok 2024-04-29T14:20:09.352 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:09.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:09.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:09 smithi121 ceph-mon[44533]: 4.4 scrub ok 2024-04-29T14:20:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:09 smithi121 ceph-mon[44533]: 4.9 scrub ok 2024-04-29T14:20:09.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:09 smithi121 ceph-mon[44533]: 4.a scrub starts 2024-04-29T14:20:09.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:09 smithi121 ceph-mon[44533]: 4.0 scrub ok 2024-04-29T14:20:09.497 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:09.558 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:09.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:09 smithi040 ceph-mon[39581]: 4.4 scrub ok 2024-04-29T14:20:09.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:09 smithi040 ceph-mon[39581]: 4.9 scrub ok 2024-04-29T14:20:09.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:09 smithi040 ceph-mon[39581]: 4.a scrub starts 2024-04-29T14:20:09.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:09 smithi040 ceph-mon[39581]: 4.0 scrub ok 2024-04-29T14:20:09.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:09.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:09.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:20:09.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:09.840 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:09.878 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:09.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:09.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:20:09.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:10.019 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:10.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:10.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:10.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:10 smithi040 ceph-mon[39581]: 4.d scrub starts 2024-04-29T14:20:10.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:10 smithi040 ceph-mon[39581]: 4.5 scrub starts 2024-04-29T14:20:10.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:10 smithi040 ceph-mon[39581]: pgmap v925: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-29T14:20:10.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:10 smithi040 ceph-mon[39581]: 4.7 scrub starts 2024-04-29T14:20:10.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:10 smithi040 ceph-mon[39581]: 4.8 scrub ok 2024-04-29T14:20:10.302 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:10.311 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 140097 140081 119.913 50.375 0.0162943 0.00833717 2024-04-29T14:20:10.313 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:09 smithi194 bash[106529]: b3297f88305a33a0ba44e4b9704427877206e2b666ebddde19024ba8a19f3f2c 2024-04-29T14:20:10.313 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:09 smithi194 systemd[1]: Started Ceph osd.8 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:20:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:10 smithi194 ceph-mon[44089]: 4.d scrub starts 2024-04-29T14:20:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:10 smithi194 ceph-mon[44089]: 4.5 scrub starts 2024-04-29T14:20:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:10 smithi194 ceph-mon[44089]: pgmap v925: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-29T14:20:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:10 smithi194 ceph-mon[44089]: 4.7 scrub starts 2024-04-29T14:20:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:10 smithi194 ceph-mon[44089]: 4.8 scrub ok 2024-04-29T14:20:10.406 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:20:10.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:10 smithi121 ceph-mon[44533]: 4.d scrub starts 2024-04-29T14:20:10.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:10 smithi121 ceph-mon[44533]: 4.5 scrub starts 2024-04-29T14:20:10.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:10 smithi121 ceph-mon[44533]: pgmap v925: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-29T14:20:10.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:10 smithi121 ceph-mon[44533]: 4.7 scrub starts 2024-04-29T14:20:10.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:10 smithi121 ceph-mon[44533]: 4.8 scrub ok 2024-04-29T14:20:10.723 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T14:20:10.423+0000 7f238f5807c0 -1 Falling back to public interface 2024-04-29T14:20:11.030 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:11.054 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:11.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:11.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:11.192 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:11.266 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:11.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:11.311 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 141037 141021 119.086 58.75 0.0101981 0.00839575 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:11 smithi040 ceph-mon[39581]: 4.c scrub starts 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:11 smithi040 ceph-mon[39581]: 4.a scrub ok 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:11 smithi040 ceph-mon[39581]: 4.6 scrub starts 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:11 smithi040 ceph-mon[39581]: 4.d scrub ok 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:11 smithi040 ceph-mon[39581]: 4.7 scrub ok 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:11 smithi194 ceph-mon[44089]: 4.c scrub starts 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:11 smithi194 ceph-mon[44089]: 4.a scrub ok 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:11 smithi194 ceph-mon[44089]: 4.6 scrub starts 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:11 smithi194 ceph-mon[44089]: 4.d scrub ok 2024-04-29T14:20:11.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:11 smithi194 ceph-mon[44089]: 4.7 scrub ok 2024-04-29T14:20:11.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:11.423 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:11.461 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:11.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:11.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:11 smithi121 ceph-mon[44533]: 4.c scrub starts 2024-04-29T14:20:11.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:11 smithi121 ceph-mon[44533]: 4.a scrub ok 2024-04-29T14:20:11.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:11 smithi121 ceph-mon[44533]: 4.6 scrub starts 2024-04-29T14:20:11.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:11 smithi121 ceph-mon[44533]: 4.d scrub ok 2024-04-29T14:20:11.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:11 smithi121 ceph-mon[44533]: 4.7 scrub ok 2024-04-29T14:20:11.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:11.596 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:11.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:11.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:11.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:11.813 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:11.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:11.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:11.958 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:11.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:11.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:12.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:12.137 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:12.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:12.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:12.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:12.308 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:12.311 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 16 142021 142005 118.318 61.5 0.013348 0.00844991 2024-04-29T14:20:12.331 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:12.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:12.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:12.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:12 smithi121 ceph-mon[44533]: 4.5 scrub ok 2024-04-29T14:20:12.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:12 smithi121 ceph-mon[44533]: pgmap v926: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:20:12.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:12 smithi121 ceph-mon[44533]: 4.c scrub ok 2024-04-29T14:20:12.482 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:12.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:12.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:12 smithi194 ceph-mon[44089]: 4.5 scrub ok 2024-04-29T14:20:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:12 smithi194 ceph-mon[44089]: pgmap v926: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:20:12.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:12 smithi194 ceph-mon[44089]: 4.c scrub ok 2024-04-29T14:20:12.566 INFO:teuthology.orchestra.run.smithi194.stderr:no valid command found; 10 closest matches: 2024-04-29T14:20:12.566 INFO:teuthology.orchestra.run.smithi194.stderr:0 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:1 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:2 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:abort 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:assert 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:bluefs files list 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:bluefs stats 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:bluestore bluefs device info [] 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:config diff 2024-04-29T14:20:12.567 INFO:teuthology.orchestra.run.smithi194.stderr:admin_socket: invalid command 2024-04-29T14:20:12.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:12 smithi040 ceph-mon[39581]: 4.5 scrub ok 2024-04-29T14:20:12.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:12 smithi040 ceph-mon[39581]: pgmap v926: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:20:12.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:12 smithi040 ceph-mon[39581]: 4.c scrub ok 2024-04-29T14:20:12.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:12.658 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:12.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:12.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:12.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:12.820 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:13.311 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 143061 143045 117.617 65 0.0162721 0.00850068 2024-04-29T14:20:13.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:13 smithi194 ceph-mon[44089]: 4.6 scrub ok 2024-04-29T14:20:13.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2240027742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:13.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:13 smithi040 ceph-mon[39581]: 4.6 scrub ok 2024-04-29T14:20:13.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2240027742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:20:13.372 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-8, ['dump_ops_in_flight'] 2024-04-29T14:20:13.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:13 smithi121 ceph-mon[44533]: 4.6 scrub ok 2024-04-29T14:20:13.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2240027742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:13.930 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:20:14.171 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:14.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:14.196 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T14:20:13.830+0000 7f238f5807c0 -1 osd.8 320 log_to_monitors true 2024-04-29T14:20:14.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:14 smithi194 ceph-mon[44089]: 4.e scrub starts 2024-04-29T14:20:14.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:14 smithi194 ceph-mon[44089]: pgmap v927: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-29T14:20:14.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:14 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:20:14.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2240027742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:14.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:14 smithi194 ceph-mon[44089]: osdmap e352: 12 total, 11 up, 8 in 2024-04-29T14:20:14.197 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:14 smithi194 ceph-mon[44089]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:20:14.275 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:14.305 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:14.313 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 144126 144110 116.951 66.5625 0.00864631 0.00854874 2024-04-29T14:20:14.563 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 14:20:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T14:20:14.219+0000 7f23809d8700 -1 osd.8 320 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:20:14.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:14 smithi040 ceph-mon[39581]: 4.e scrub starts 2024-04-29T14:20:14.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:14 smithi040 ceph-mon[39581]: pgmap v927: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-29T14:20:14.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:14 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:20:14.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2240027742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:14.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:14 smithi040 ceph-mon[39581]: osdmap e352: 12 total, 11 up, 8 in 2024-04-29T14:20:14.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:14 smithi040 ceph-mon[39581]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:20:14.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:14 smithi121 ceph-mon[44533]: 4.e scrub starts 2024-04-29T14:20:14.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:14 smithi121 ceph-mon[44533]: pgmap v927: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-29T14:20:14.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:14 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:20:14.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2240027742' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:14.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:14 smithi121 ceph-mon[44533]: osdmap e352: 12 total, 11 up, 8 in 2024-04-29T14:20:14.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:14 smithi121 ceph-mon[44533]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-29T14:20:14.958 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:14.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:15.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:15.103 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:15.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:15.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:15 smithi040 ceph-mon[39581]: 4.b scrub starts 2024-04-29T14:20:15.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:15 smithi040 ceph-mon[39581]: 4.f scrub starts 2024-04-29T14:20:15.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:15 smithi040 ceph-mon[39581]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-29T14:20:15.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:15 smithi040 ceph-mon[39581]: osdmap e353: 12 total, 11 up, 8 in 2024-04-29T14:20:15.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:15 smithi040 ceph-mon[39581]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:20:15.208 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:20:15.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:15.272 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:15.301 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:15.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:15.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:20:15.313 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 145179 145163 116.296 65.8125 0.00924819 0.00859725 2024-04-29T14:20:15.329 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:20:15.353 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:20:15.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:15 smithi121 ceph-mon[44533]: 4.b scrub starts 2024-04-29T14:20:15.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:15 smithi121 ceph-mon[44533]: 4.f scrub starts 2024-04-29T14:20:15.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:15 smithi121 ceph-mon[44533]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-29T14:20:15.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:15 smithi121 ceph-mon[44533]: osdmap e353: 12 total, 11 up, 8 in 2024-04-29T14:20:15.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:15 smithi121 ceph-mon[44533]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:20:15.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:20:15.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:15 smithi121 ceph-mon[44533]: 4.e scrub ok 2024-04-29T14:20:15.376 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:20:15.401 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:20:15.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:20:15.436 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:15.458 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:20:15.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:20:15.503 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:20:15.526 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:20:15.540 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:15.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:15 smithi040 ceph-mon[39581]: 4.e scrub ok 2024-04-29T14:20:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:15.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:15.555 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:20:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:15 smithi194 ceph-mon[44089]: 4.b scrub starts 2024-04-29T14:20:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:15 smithi194 ceph-mon[44089]: 4.f scrub starts 2024-04-29T14:20:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:15 smithi194 ceph-mon[44089]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-29T14:20:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:15 smithi194 ceph-mon[44089]: osdmap e353: 12 total, 11 up, 8 in 2024-04-29T14:20:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:15 smithi194 ceph-mon[44089]: from='osd.8 v1:172.21.15.194:6800/165633203' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:20:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:20:15.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:15 smithi194 ceph-mon[44089]: 4.e scrub ok 2024-04-29T14:20:15.586 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:20:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:20:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:20:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:20:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:20:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:20:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:20:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:20:15.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:20:15.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:20:15.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:20:15.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:20:15.588 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:20:15.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:15.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:15.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:15.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:15.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:15.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:16.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:16.052 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:16.153 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:20:16.153 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:20:16.153 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:20:16.153 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:20:16.313 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 146206 146190 115.636 64.1875 0.0133218 0.00864621 2024-04-29T14:20:16.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: pgmap v930: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 43 MiB/s wr, 684 op/s 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: 4.2 scrub starts 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd.8 v1:172.21.15.194:6800/165633203 boot 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osdmap e354: 12 total, 12 up, 8 in 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/169270649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: 4.b scrub ok 2024-04-29T14:20:16.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: pgmap v930: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 43 MiB/s wr, 684 op/s 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: 4.2 scrub starts 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd.8 v1:172.21.15.194:6800/165633203 boot 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osdmap e354: 12 total, 12 up, 8 in 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/169270649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: 4.b scrub ok 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: pgmap v930: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 43 MiB/s wr, 684 op/s 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: 4.2 scrub starts 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd.8 v1:172.21.15.194:6800/165633203 boot 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osdmap e354: 12 total, 12 up, 8 in 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/169270649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: 4.b scrub ok 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:16.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:20:16.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:20:17.313 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:20:17.320990 min lat: 0.00115367 max lat: 0.236103 avg lat: 0.00869632 2024-04-29T14:20:17.313 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:20:17.313 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 147202 147186 114.969 62.25 0.00475447 0.00869632 2024-04-29T14:20:17.393 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:17.403 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:17.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:17.507 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:17.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:17 smithi040 ceph-mon[39581]: 4.f scrub ok 2024-04-29T14:20:17.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:17 smithi040 ceph-mon[39581]: osdmap e355: 12 total, 12 up, 8 in 2024-04-29T14:20:17.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:17 smithi040 ceph-mon[39581]: 4.3 deep-scrub starts 2024-04-29T14:20:17.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:17 smithi040 ceph-mon[39581]: pgmap v933: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:20:17.535 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:17.555 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:17.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:17.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:17 smithi194 ceph-mon[44089]: 4.f scrub ok 2024-04-29T14:20:17.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:17 smithi194 ceph-mon[44089]: osdmap e355: 12 total, 12 up, 8 in 2024-04-29T14:20:17.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:17 smithi194 ceph-mon[44089]: 4.3 deep-scrub starts 2024-04-29T14:20:17.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:17 smithi194 ceph-mon[44089]: pgmap v933: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:20:17.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:17 smithi121 ceph-mon[44533]: 4.f scrub ok 2024-04-29T14:20:17.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:17 smithi121 ceph-mon[44533]: osdmap e355: 12 total, 12 up, 8 in 2024-04-29T14:20:17.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:17 smithi121 ceph-mon[44533]: 4.3 deep-scrub starts 2024-04-29T14:20:17.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:17 smithi121 ceph-mon[44533]: pgmap v933: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:20:17.662 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:17.706 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:17.740 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:17.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:17.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:17.873 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:17.897 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:17.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:18.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:18.221 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:18.226 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:18.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:18.313 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 148205 148189 114.323 62.6875 0.0136921 0.00874536 2024-04-29T14:20:18.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:18.373 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:20:18.374 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:18.410 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:18.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:18.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:18 smithi040 ceph-mon[39581]: 4.2 scrub ok 2024-04-29T14:20:18.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:18.540 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:18 smithi194 ceph-mon[44089]: 4.2 scrub ok 2024-04-29T14:20:18.616 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:18.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:18.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:18 smithi121 ceph-mon[44533]: 4.2 scrub ok 2024-04-29T14:20:18.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:18.749 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:18.783 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:20:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:19.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:19.103 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:19.154 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:19.193 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:19.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:19.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:19.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:19.316 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 149123 149107 113.624 57.375 0.00967641 0.00879915 2024-04-29T14:20:19.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:19 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:20:19.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:19 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:20:19.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:19 smithi040 ceph-mon[39581]: pgmap v934: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 MiB/s wr, 878 op/s 2024-04-29T14:20:19.328 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:19.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:19 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:20:19.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:19 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:20:19.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:19 smithi194 ceph-mon[44089]: pgmap v934: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 MiB/s wr, 878 op/s 2024-04-29T14:20:19.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:20:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:19.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:19.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:19.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:19 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:20:19.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:19 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:20:19.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:19 smithi121 ceph-mon[44533]: pgmap v934: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 MiB/s wr, 878 op/s 2024-04-29T14:20:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:19.648 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:19.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:19.751 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:19.790 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:20.153 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:20.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:20.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:20.290 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:20.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 16 150068 150052 112.967 59.0625 0.0220891 0.00885006 2024-04-29T14:20:20.318 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:20.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:20.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:20:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:20.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3944493023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:20:20.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3944493023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:20:20.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3944493023' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:20:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:20.915 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:20.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:21.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:21.092 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:21.097 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:20:21.097 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:20:21.097 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:20:21.097 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:20:21.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:21.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:21.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:21.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:21.276 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:21.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:21.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:21.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 151001 150985 112.316 58.3125 0.00751069 0.0089014 2024-04-29T14:20:21.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:21.436 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:21 smithi194 ceph-mon[44089]: 4.1 scrub starts 2024-04-29T14:20:21.436 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:21 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:21.436 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3944493023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:20:21.436 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:21 smithi194 ceph-mon[44089]: osdmap e356: 12 total, 12 up, 8 in 2024-04-29T14:20:21.436 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:21 smithi194 ceph-mon[44089]: pgmap v936: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:20:21.454 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:21.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:21.584 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:21 smithi040 ceph-mon[39581]: 4.1 scrub starts 2024-04-29T14:20:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:21 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3944493023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:20:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:21 smithi040 ceph-mon[39581]: osdmap e356: 12 total, 12 up, 8 in 2024-04-29T14:20:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:21 smithi040 ceph-mon[39581]: pgmap v936: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:20:21.631 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:21.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:21 smithi121 ceph-mon[44533]: 4.1 scrub starts 2024-04-29T14:20:21.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:21 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:21.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3944493023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:20:21.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:21 smithi121 ceph-mon[44533]: osdmap e356: 12 total, 12 up, 8 in 2024-04-29T14:20:21.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:21 smithi121 ceph-mon[44533]: pgmap v936: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:20:21.984 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:20:21.985 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-29T14:20:21.985 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:20:21.985 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:20:21.985 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-29T14:20:21.993 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:21.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:22.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:22.130 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:22.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:22.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:22.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:22.301 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:22.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 16 151958 151942 111.698 59.8125 0.0197314 0.00895117 2024-04-29T14:20:22.408 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:22.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:22.444 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:20:22.444 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:20:22.444 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:20:22.444 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:20:22.444 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:20:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:22.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:22.550 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:22.585 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:22.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:22.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:22.735 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:22.758 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:22.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:22.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:22.920 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:22.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:22.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:23.055 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:23.091 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:23.123 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:23.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:23.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:23.272 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:23.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:23.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:23.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 152897 152881 111.082 58.6875 0.00534555 0.00899837 2024-04-29T14:20:23.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:23.496 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:23.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:23.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:23.666 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:23.710 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:23.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:23.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:20:23.817 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:23.851 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:23.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:23.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:20:23.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:24.018 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:24.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:24.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:24.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:24.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:24.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:24 smithi194 ceph-mon[44089]: pgmap v937: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:20:24.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 153871 153855 110.505 60.875 0.0104173 0.00904712 2024-04-29T14:20:24.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:24.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:24 smithi040 ceph-mon[39581]: pgmap v937: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:20:24.362 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:24.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:24 smithi121 ceph-mon[44533]: pgmap v937: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:20:24.393 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:24.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:24.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:24.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:24.623 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:20:24.646 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:20:24.667 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:20:24.689 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:20:24.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:24.717 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:20:24.740 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:20:24.761 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:20:24.761 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:24.782 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:20:24.785 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:24.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:24.805 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:20:24.826 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:20:24.848 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:20:24.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:20:24.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:20:24.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:20:24.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:20:24.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:20:24.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:20:24.871 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:20:24.871 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:20:24.871 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:20:24.871 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:20:24.871 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:20:24.871 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:20:24.871 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:20:24.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:24.923 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:24.958 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:24.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:25.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: osdmap e357: 12 total, 12 up, 8 in 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1812489251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.090 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.091 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:25.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:25.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:25.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: osdmap e357: 12 total, 12 up, 8 in 2024-04-29T14:20:25.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1812489251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:25.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:25.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:25.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: osdmap e357: 12 total, 12 up, 8 in 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1812489251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:25.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 154807 154791 109.914 58.5 0.0121222 0.00909568 2024-04-29T14:20:25.334 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:25.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:25.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:25.486 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:25.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:25.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:25.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:25.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:25.656 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:25.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:25.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:25.861 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:25.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:25.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:25.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:26.031 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:26.057 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:26.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:26.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:26.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:26 smithi121 ceph-mon[44533]: pgmap v939: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 MiB/s wr, 688 op/s 2024-04-29T14:20:26.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:26.219 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:26.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:26.284 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:26 smithi040 ceph-mon[39581]: pgmap v939: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 MiB/s wr, 688 op/s 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:26.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:26 smithi194 ceph-mon[44089]: pgmap v939: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 MiB/s wr, 688 op/s 2024-04-29T14:20:26.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 155754 155738 109.344 59.1875 0.0151163 0.00914297 2024-04-29T14:20:26.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:26.387 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:26.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:26.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:26.753 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:26.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:26.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:26.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:26.931 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:26.955 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:26.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:27.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:27.084 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:20:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-29T14:20:27.084 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:20:27.103 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:27.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:27.241 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:27.272 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:27.298 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:27.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:27.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 90.0178 2024-04-29T14:20:27.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 156680 2024-04-29T14:20:27.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:20:27.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:20:27.317 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 108.784 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 52.7289 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 318.25 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 50.375 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1740 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 843.663 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 5092 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 806 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.00919142 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.00839938 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 0.236103 2024-04-29T14:20:27.318 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.00115367 2024-04-29T14:20:27.325 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:20:27.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:27.463 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:27.486 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:27.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:27.599 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:27.638 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:27.670 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:27.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:27.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:27.814 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:27.848 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:27.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:27.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:27.992 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:28.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:28.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:28.130 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:28.171 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:28.198 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:28.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:28.246 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:28.303 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:28.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:20:28.339 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:28.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:28 smithi040 ceph-mon[39581]: pgmap v940: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-29T14:20:28.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:28 smithi121 ceph-mon[44533]: pgmap v940: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-29T14:20:28.377 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:28.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:28.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:28 smithi194 ceph-mon[44089]: pgmap v940: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-29T14:20:28.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:28.539 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:28.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:28.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:28.709 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:28.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:28.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:28.908 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:28.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:28.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:29.045 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:29.083 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:29.111 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:29.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:29.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:29.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:29.276 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:29.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:29.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:29.456 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:29.456 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:29.458 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:29.486 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:29.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:29.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:29.593 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:29.632 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:29.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:29.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:29.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:29.831 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:29.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:29.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:30.017 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:30.047 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:30.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:30.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:30 smithi121 ceph-mon[44533]: pgmap v941: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:20:30.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:30 smithi121 ceph-mon[44533]: osdmap e358: 12 total, 12 up, 8 in 2024-04-29T14:20:30.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:30 smithi040 ceph-mon[39581]: pgmap v941: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:20:30.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:30 smithi040 ceph-mon[39581]: osdmap e358: 12 total, 12 up, 8 in 2024-04-29T14:20:30.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:30.191 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:30.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:30.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:30.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:30 smithi194 ceph-mon[44089]: pgmap v941: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:20:30.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:30 smithi194 ceph-mon[44089]: osdmap e358: 12 total, 12 up, 8 in 2024-04-29T14:20:30.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:30.364 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:30.419 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:20:30.420 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:20:31.015 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:31.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:31.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:31.161 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:31.197 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:31.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:31.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:31.338 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:31.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:31.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:31.574 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:31.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:31.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:31.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:31.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:31.867 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:31.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:20:31.906 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:31.944 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:31.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:32.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:32.132 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:32.170 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:32.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:32.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:32.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:32 smithi194 ceph-mon[44089]: pgmap v943: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.11k op/s 2024-04-29T14:20:32.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:32.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:32.326 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:32.358 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:32.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:32.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:32.497 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:32.521 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:32.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:32.581 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:32 smithi121 ceph-mon[44533]: pgmap v943: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.11k op/s 2024-04-29T14:20:32.581 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:32.581 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:32.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:32 smithi040 ceph-mon[39581]: pgmap v943: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.11k op/s 2024-04-29T14:20:32.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:32.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:32.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:32.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:32.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:32.787 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 8] 2024-04-29T14:20:32.787 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:20:32.788 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-29T14:20:32.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:20:32.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:32.865 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:33.487 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:20:33.503 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:33.512 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:20:33.536 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:20:33.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:20:33.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:33 smithi194 ceph-mon[44089]: osdmap e359: 12 total, 12 up, 8 in 2024-04-29T14:20:33.586 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:20:33.606 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:20:33.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:33 smithi040 ceph-mon[39581]: osdmap e359: 12 total, 12 up, 8 in 2024-04-29T14:20:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:33 smithi121 ceph-mon[44533]: osdmap e359: 12 total, 12 up, 8 in 2024-04-29T14:20:33.628 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:20:33.654 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:20:33.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:20:33.698 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:20:33.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:33.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:33.725 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:20:33.760 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:20:33.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:20:33.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:20:33.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:20:33.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:20:33.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:33.849 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:34.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:34.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:20:34.175 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:34.243 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:34.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:34.351 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:34.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: pgmap v945: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.18k op/s 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3055565115' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: from='client.35513 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1128779514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.392 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:163304 4.c 4:329e7973:::benchmark_data_smithi194_103039_object6621:head [delete] snapc 0=[] ondisk+write+known_if_redirected e360)", 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:20:34.411945+0000", 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.012815195, 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012854042, 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163304 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.411945+0000", 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.411945+0000", 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.411946+0000", 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0500000000000002e-07 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.411948+0000", 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.875e-06 2024-04-29T14:20:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.411954+0000", 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5840000000000002e-06 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.411959+0000", 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0289999999999996e-06 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.416887+0000", 2024-04-29T14:20:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049276249999999997 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.417027+0000", 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014005500000000001 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.417163+0000", 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013595799999999999 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.422421+0000", 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052587800000000002 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:163306 4.d 4:bef0a842:::benchmark_data_smithi194_103039_object6623:head [delete] snapc 0=[] ondisk+write+known_if_redirected e360)", 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:20:34.416850+0000", 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0079099960000000007, 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080379920000000007, 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163306 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.416850+0000", 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.416850+0000", 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.416852+0000", 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7370000000000001e-06 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.416858+0000", 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.416871+0000", 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2526e-05 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.416879+0000", 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1750000000000005e-06 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.418186+0000", 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001307323 2024-04-29T14:20:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.418297+0000", 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011103999999999999 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.418418+0000", 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012102799999999999 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424535+0000", 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061166609999999998 2024-04-29T14:20:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25522.0:163309 4.b e360/323 4:d09e0aad:::benchmark_data_smithi194_103039_object6626:head v 360'10456, mlcod=360'10454)", 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:20:34.419810+0000", 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0049499959999999999, 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051622719999999999, 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419810+0000", 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419810+0000", 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419811+0000", 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.525e-06 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419819+0000", 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9370000000000004e-06 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419832+0000", 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2183e-05 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419840+0000", 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0059999999999992e-06 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419960+0000", 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120493 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.419981+0000", 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1195e-05 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25522.0:163313 4.b e360/323 4:dc8e7c2a:::benchmark_data_smithi194_103039_object6630:head v 360'10457, mlcod=360'10454)", 2024-04-29T14:20:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:20:34.424465+0000", 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00029455399999999999, 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056946000000000004, 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424465+0000", 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424465+0000", 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424467+0000", 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.189e-06 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424472+0000", 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.631e-06 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424482+0000", 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.009e-05 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424489+0000", 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2079999999999997e-06 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424550+0000", 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1951000000000004e-05 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:34.424575+0000", 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4088e-05 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 4 2024-04-29T14:20:34.439 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:20:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:34.440 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:34.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: pgmap v945: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.18k op/s 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3055565115' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: from='client.35513 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1128779514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: pgmap v945: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 1.18k op/s 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3055565115' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: from='client.35513 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1128779514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:34.671 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:34.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:34.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:34.810 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:35.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:20:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: from='client.35513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:35 smithi121 ceph-mon[44533]: osdmap e360: 12 total, 12 up, 8 in 2024-04-29T14:20:35.462 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: from='client.35513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:35.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:35 smithi040 ceph-mon[39581]: osdmap e360: 12 total, 12 up, 8 in 2024-04-29T14:20:35.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:35.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:35.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:35.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: from='client.35513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:35.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:35 smithi194 ceph-mon[44089]: osdmap e360: 12 total, 12 up, 8 in 2024-04-29T14:20:35.565 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:35.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:35.596 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:36 smithi040 ceph-mon[39581]: pgmap v947: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 170 B/s rd, 13 MiB/s wr, 778 op/s 2024-04-29T14:20:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:36.448 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:36.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:36 smithi121 ceph-mon[44533]: pgmap v947: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 170 B/s rd, 13 MiB/s wr, 778 op/s 2024-04-29T14:20:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:36.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:36 smithi194 ceph-mon[44089]: pgmap v947: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 170 B/s rd, 13 MiB/s wr, 778 op/s 2024-04-29T14:20:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:36.579 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:36.829 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:36.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:36.933 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:36.969 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:36.998 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:37.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:37.106 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:37.173 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:37.198 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:37.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:37.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:20:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:20:37.302 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:20:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 361, 2024-04-29T14:20:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:20:37.302 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:20:36.164632+0000", 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:20:15.206702+0000", 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:19:01.251051+0000", 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5806080, 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "nodeep-scrub", 2024-04-29T14:20:37.304 INFO:teuthology.orchestra.run.smithi040.stdout: "noscrub", 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 28, 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 4, 2024-04-29T14:20:37.305 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:20:37.306 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:20:37.307 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:20:37.309 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:20:37.310 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:20:37.311 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:20:37.312 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 11.109999656677246, 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 11.109999656677246, 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.18000000715255737, 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 4, 2024-04-29T14:20:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_2", 2024-04-29T14:20:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:18:47.160984+0000", 2024-04-29T14:20:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:20:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:20:37.314 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:20:37.315 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "326", 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:20:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:20:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:20:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.0499999523162842, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 2.0499999523162842, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.73000001907348633, 2024-04-29T14:20:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 1.5, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 1.5, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.328 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 152, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:37.329 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.330 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.334 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.336 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.338 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:37.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 36, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.341 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:20:37.342 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/3807513901", 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/3807513901", 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3807513901", 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/3807513901", 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.344 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.345 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.346 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.347 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.348 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.349 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.350 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:20:37.351 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:20:37.352 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:37.353 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.354 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.355 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.356 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.357 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.358 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.359 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.360 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:37.362 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.364 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.365 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:20:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.367 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.368 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:20:37.369 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.370 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:24.377589+0000", 2024-04-29T14:20:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:28.688717+0000", 2024-04-29T14:20:37.374 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:30.922830+0000", 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:34.875980+0000", 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:31.658336+0000", 2024-04-29T14:20:37.376 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:50.723211+0000", 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.377 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:27.419198+0000", 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:33.310804+0000", 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:14.794744+0000", 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.379 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:46.138814+0000", 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:35.646548+0000", 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:23.775639+0000", 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:20:37.381 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:20:37.382 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:20:37.383 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:37.384 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:20:37.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:37.385 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:37 smithi194 ceph-mon[44089]: osdmap e361: 12 total, 12 up, 8 in 2024-04-29T14:20:37.413 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:37.559 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:20:37.559 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:20:37.559 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:20:37.559 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:20:37.559 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:20:37.562 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:37.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:37 smithi040 ceph-mon[39581]: osdmap e361: 12 total, 12 up, 8 in 2024-04-29T14:20:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:37.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:37.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:37 smithi121 ceph-mon[44533]: osdmap e361: 12 total, 12 up, 8 in 2024-04-29T14:20:37.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:37.694 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:37.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:37.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:37.883 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:38.042 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:20:38.177 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:38 smithi040 ceph-mon[39581]: pgmap v949: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-04-29T14:20:38.177 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1788985760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:20:38.199 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:38.232 INFO:tasks.thrashosds.thrasher:j is {'epoch': 361, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:20:36.164632+0000', 'last_up_change': '2024-04-29T14:20:15.206702+0000', 'last_in_change': '2024-04-29T14:19:01.251051+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.18000000715255737, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-29T14:18:47.160984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '326', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.049999952316284, 'score_stable': 2.049999952316284, 'optimal_score': 0.7300000190734863, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 152, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 3807513901}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3807513901}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3807513901}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 3807513901}]}, 'public_addr': '172.21.15.121:6804/3807513901', 'cluster_addr': '172.21.15.121:6805/3807513901', 'heartbeat_back_addr': '172.21.15.121:6807/3807513901', 'heartbeat_front_addr': '172.21.15.121:6806/3807513901', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 206, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:19:24.377589+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:20:28.688717+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:19:30.922830+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:19:34.875980+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:31.658336+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:19:50.723211+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:19:27.419198+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:33.310804+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:20:14.794744+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:19:46.138814+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:35.646548+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:23.775639+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:20:38.233 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-29T14:20:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:38.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:38.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:38 smithi121 ceph-mon[44533]: pgmap v949: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-04-29T14:20:38.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1788985760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:20:38.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:20:38.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:38.412 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:38.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:38.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:38.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:38.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:38 smithi194 ceph-mon[44089]: pgmap v949: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-04-29T14:20:38.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1788985760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:20:38.606 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:38.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:38.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:38.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:38.774 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:39.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:39.052 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:39.129 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:39.164 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:39.198 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:39.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:39.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:39.351 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:40.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:40.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:40.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:40.214 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:40.250 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:40.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:40.333 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:40 smithi121 ceph-mon[44533]: pgmap v950: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 792 op/s 2024-04-29T14:20:40.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:40.400 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:40.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:40.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:40 smithi194 ceph-mon[44089]: pgmap v950: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 792 op/s 2024-04-29T14:20:40.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:40 smithi040 ceph-mon[39581]: pgmap v950: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 792 op/s 2024-04-29T14:20:40.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:40.652 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:40.700 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:20:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:20:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:20:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:20:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18495 3.a 3:5aa754d4:::benchmark_data_smithi194_88551_object18494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990240+0000", 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71319797199999, 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68929751399999994, 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18495 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990240+0000", 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990240+0000", 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990242+0000", 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5909999999999998e-06 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990321+0000", 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9048000000000006e-05 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990339+0000", 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7331000000000001e-05 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990348+0000", 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5799999999999998e-06 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990391+0000", 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2725999999999997e-05 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990398+0000", 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4150000000000002e-06 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.678954+0000", 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68855618299999999 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.679537+0000", 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058264000000000005 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18506 3.a 3:590c742d:::benchmark_data_smithi194_88551_object18505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990361+0000", 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71307641999999, 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69417462299999999, 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18506 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990361+0000", 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990361+0000", 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990362+0000", 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1939999999999999e-06 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990396+0000", 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3578999999999998e-05 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990402+0000", 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7529999999999998e-06 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990407+0000", 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9930000000000003e-06 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990430+0000", 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3125999999999998e-05 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990434+0000", 2024-04-29T14:20:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.794e-06 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680041+0000", 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68960765700000004 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680138+0000", 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6328000000000003e-05 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680245+0000", 2024-04-29T14:20:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107618 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.682280+0000", 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020350780000000001 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684069+0000", 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00178908 2024-04-29T14:20:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684129+0000", 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0080000000000001e-05 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684536+0000", 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040634299999999998 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18517 3.a 3:57592a66:::benchmark_data_smithi194_88551_object18516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990414+0000", 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.713023572, 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69492493700000002, 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18517 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990414+0000", 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990414+0000", 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990415+0000", 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9500000000000002e-07 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990447+0000", 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2835e-05 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990454+0000", 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.19e-06 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990457+0000", 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7060000000000002e-06 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990483+0000", 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5551e-05 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990490+0000", 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5899999999999996e-06 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680937+0000", 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69044763099999995 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681024+0000", 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7015000000000006e-05 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681135+0000", 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111004 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684013+0000", 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002877601 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684945+0000", 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093199899999999998 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685000+0000", 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5692999999999997e-05 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685339+0000", 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033842699999999998 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18531 3.a 3:50653578:::benchmark_data_smithi194_88551_object18530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990464+0000", 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71297352099998, 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69534631400000002, 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18531 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990464+0000", 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990464+0000", 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990465+0000", 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990514+0000", 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9667999999999999e-05 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990521+0000", 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7329999999999997e-06 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990525+0000", 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.907e-06 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990577+0000", 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1904000000000003e-05 2024-04-29T14:20:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990588+0000", 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0944999999999999e-05 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681648+0000", 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69106008100000005 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681729+0000", 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1100000000000006e-05 2024-04-29T14:20:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681845+0000", 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011582300000000001 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684895+0000", 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030497319999999999 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685386+0000", 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049112099999999996 2024-04-29T14:20:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685437+0000", 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1622000000000001e-05 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685810+0000", 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037302199999999998 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18535 3.a 3:59440a67:::benchmark_data_smithi194_88551_object18534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71289384400001, 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69569302700000002, 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18535 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6599999999999996e-07 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990581+0000", 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6721999999999998e-05 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990589+0000", 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4830000000000002e-06 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990594+0000", 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4060000000000004e-06 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990620+0000", 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5993e-05 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990624+0000", 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.683304+0000", 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69267976899999995 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.683413+0000", 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108596 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.683533+0000", 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012050299999999999 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685881+0000", 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023481589999999998 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685891+0000", 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4180000000000003e-06 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685922+0000", 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1084000000000003e-05 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686237+0000", 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031483699999999999 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18542 3.a 3:593cad59:::benchmark_data_smithi194_88551_object18541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990603+0000", 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.712834445, 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70048259899999998, 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18542 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990603+0000", 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990603+0000", 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990604+0000", 2024-04-29T14:20:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9100000000000003e-07 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990636+0000", 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2549000000000003e-05 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990643+0000", 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2820000000000002e-06 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990646+0000", 2024-04-29T14:20:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7780000000000001e-06 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990674+0000", 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7701000000000001e-05 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990678+0000", 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7500000000000001e-06 2024-04-29T14:20:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686278+0000", 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69559982399999998 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686374+0000", 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6582999999999993e-05 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686482+0000", 2024-04-29T14:20:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108087 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689376+0000", 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028935440000000001 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.690185+0000", 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080902800000000005 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.690242+0000", 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7210000000000003e-05 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691086+0000", 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00084347199999999995 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18546 3.a 3:5fb73ead:::benchmark_data_smithi194_88551_object18545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71278385300002, 2024-04-29T14:20:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70239883199999997, 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18546 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990699+0000", 2024-04-29T14:20:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4279000000000001e-05 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990704+0000", 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4099999999999999e-06 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990708+0000", 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3540000000000002e-06 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990733+0000", 2024-04-29T14:20:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4834999999999999e-05 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990737+0000", 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7629999999999998e-06 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687202+0000", 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69646489300000003 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687302+0000", 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100269 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687392+0000", 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759000000000006e-05 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691103+0000", 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037110260000000001 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.692192+0000", 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010887710000000001 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.692252+0000", 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0303999999999997e-05 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693052+0000", 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080051299999999998 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18554 3.a 3:55e068b6:::benchmark_data_smithi194_88551_object18553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990715+0000", 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71272239000001, 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70315993099999996, 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18554 2024-04-29T14:20:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990715+0000", 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990715+0000", 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990716+0000", 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.61e-07 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990747+0000", 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1173999999999997e-05 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990753+0000", 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9340000000000003e-06 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990756+0000", 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.535e-06 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990782+0000", 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5316e-05 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990788+0000", 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2210000000000002e-06 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687932+0000", 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69714447599999996 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688036+0000", 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010375200000000001 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688145+0000", 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108786 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691111+0000", 2024-04-29T14:20:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002965638 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693274+0000", 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021631530000000001 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693333+0000", 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9378e-05 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693875+0000", 2024-04-29T14:20:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000541907 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18557 3.a 3:5d1fe685:::benchmark_data_smithi194_88551_object18556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990763+0000", 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71267456099997, 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70364153100000004, 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18557 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990763+0000", 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990763+0000", 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990764+0000", 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1099999999999995e-07 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990844+0000", 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0158000000000001e-05 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990854+0000", 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9310000000000003e-06 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990862+0000", 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2439999999999993e-06 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990893+0000", 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0624999999999999e-05 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990896+0000", 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7720000000000001e-06 2024-04-29T14:20:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688636+0000", 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69773956299999995 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688757+0000", 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121134 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688916+0000", 2024-04-29T14:20:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159044 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693207+0000", 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042910320000000002 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693974+0000", 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00076646400000000001 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694028+0000", 2024-04-29T14:20:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4815999999999998e-05 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694404+0000", 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037613700000000002 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18582 3.a 3:56da5d7f:::benchmark_data_smithi194_88551_object18581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990877+0000", 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71256094099999, 2024-04-29T14:20:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70399896699999998, 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18582 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990877+0000", 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990877+0000", 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990878+0000", 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0499999999999999e-06 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990913+0000", 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4962999999999997e-05 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990920+0000", 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5070000000000003e-06 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990927+0000", 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6640000000000001e-06 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990973+0000", 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6453999999999999e-05 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990979+0000", 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4399999999999996e-06 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689422+0000", 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69844298299999996 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689537+0000", 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115591 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689683+0000", 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014627499999999999 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693215+0000", 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035312859999999998 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694459+0000", 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012442 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694506+0000", 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7375e-05 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694876+0000", 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036917900000000002 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18598 3.a 3:51a9ca23:::benchmark_data_smithi194_88551_object18597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990938+0000", 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71249952300002, 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70444663600000001, 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18598 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990938+0000", 2024-04-29T14:20:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990938+0000", 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990939+0000", 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7899999999999998e-07 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990975+0000", 2024-04-29T14:20:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5954000000000002e-05 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990981+0000", 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8569999999999997e-06 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990987+0000", 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991024+0000", 2024-04-29T14:20:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6860000000000003e-05 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991029+0000", 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4850000000000003e-06 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691156+0000", 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70012712399999999 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691304+0000", 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014863699999999999 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691541+0000", 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236987 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694419+0000", 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028776370000000002 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694938+0000", 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051860600000000001 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694994+0000", 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5971999999999999e-05 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.695385+0000", 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039107500000000001 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18616 3.a 3:57784d63:::benchmark_data_smithi194_88551_object18615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990997+0000", 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.712440012, 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72318601500000002, 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18616 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990997+0000", 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990997+0000", 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990998+0000", 2024-04-29T14:20:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5000000000000002e-07 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991030+0000", 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.167e-05 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991036+0000", 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.421e-06 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991042+0000", 2024-04-29T14:20:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3519999999999997e-06 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991074+0000", 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2289999999999997e-05 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991078+0000", 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8039999999999999e-06 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.698095+0000", 2024-04-29T14:20:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.707017699 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.698228+0000", 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013238900000000001 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.698409+0000", 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018093699999999999 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.702065+0000", 2024-04-29T14:20:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036564990000000001 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.713290+0000", 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011224622 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.713328+0000", 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7873000000000001e-05 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714184+0000", 2024-04-29T14:20:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00085570899999999996 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18624 3.a 3:5b1ca2c5:::benchmark_data_smithi194_88551_object18623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991051+0000", 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71238605799999, 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72377148300000005, 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18624 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991051+0000", 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991051+0000", 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991052+0000", 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6499999999999999e-07 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991106+0000", 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3702999999999997e-05 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991111+0000", 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.434e-06 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991117+0000", 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3539999999999999e-06 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991146+0000", 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9777999999999998e-05 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991150+0000", 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.681e-06 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699105+0000", 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.707954745 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699182+0000", 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7451000000000001e-05 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699282+0000", 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9840999999999995e-05 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705411+0000", 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061285130000000004 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714242+0000", 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088312289999999995 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714278+0000", 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6556e-05 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714823+0000", 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054453300000000004 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18640 3.a 3:5c4ff313:::benchmark_data_smithi194_88551_object18639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71231184099997, 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72425418900000005, 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18640 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9599999999999999e-07 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991176+0000", 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0062999999999998e-05 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991182+0000", 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2820000000000004e-06 2024-04-29T14:20:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991187+0000", 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2900000000000002e-06 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991218+0000", 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1297999999999997e-05 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991222+0000", 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5710000000000002e-06 2024-04-29T14:20:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699796+0000", 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70857407800000005 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699909+0000", 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011330799999999999 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700031+0000", 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012213 2024-04-29T14:20:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705420+0000", 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053887350000000004 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714880+0000", 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094600320000000002 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714915+0000", 2024-04-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4938999999999999e-05 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715380+0000", 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046476699999999999 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18720 3.a 3:5c930f07:::benchmark_data_smithi194_88551_object18719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991196+0000", 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71224114199998, 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.724726023, 2024-04-29T14:20:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18720 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991196+0000", 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991196+0000", 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991197+0000", 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5199999999999996e-07 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991248+0000", 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1104999999999999e-05 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991253+0000", 2024-04-29T14:20:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2909999999999998e-06 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991259+0000", 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3210000000000004e-06 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991289+0000", 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0627999999999999e-05 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991293+0000", 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700509+0000", 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70921634600000005 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700641+0000", 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013196700000000001 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700747+0000", 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010521199999999999 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705428+0000", 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046819299999999999 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715427+0000", 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099981389999999996 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715481+0000", 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4616e-05 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715922+0000", 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00044113900000000001 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18745 3.a 3:51ba897a:::benchmark_data_smithi194_88551_object18744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991268+0000", 2024-04-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 393.71216939599998, 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72519465900000002, 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18745 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991268+0000", 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991268+0000", 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991269+0000", 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6199999999999997e-07 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991315+0000", 2024-04-29T14:20:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6487999999999999e-05 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991320+0000", 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2039999999999996e-06 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991326+0000", 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7250000000000002e-06 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991357+0000", 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0629000000000001e-05 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991360+0000", 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.653e-06 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.701235+0000", 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.709874216 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.701400+0000", 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165317 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.701575+0000", 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017452599999999999 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705437+0000", 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038623939999999999 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715974+0000", 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010536943999999999 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.716017+0000", 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3154000000000001e-05 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.716463+0000", 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00044574700000000003 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:63888 3.e 3:74104bea:::benchmark_data_smithi194_88551_object63887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:24.890511+0000", 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 375.81292622900003, 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.209098494, 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 63888 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890511+0000", 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890511+0000", 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890513+0000", 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8020000000000001e-06 2024-04-29T14:20:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890562+0000", 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9156999999999997e-05 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890574+0000", 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1574e-05 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890581+0000", 2024-04-29T14:20:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2559999999999996e-06 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890693+0000", 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111537 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890801+0000", 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107928 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:20:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890948+0000", 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000146973 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.896888+0000", 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059409229999999999 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.099529+0000", 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20264083399999999 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.099565+0000", 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.612e-05 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.099610+0000", 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.439e-05 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:63893 3.8 3:166d9e62:::benchmark_data_smithi194_88551_object63892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:24.892888+0000", 2024-04-29T14:20:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 375.810549009, 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208170983, 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 63893 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892888+0000", 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892888+0000", 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892891+0000", 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0389999999999999e-06 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892917+0000", 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6872999999999998e-05 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892931+0000", 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3895e-05 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892939+0000", 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4159999999999998e-06 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892990+0000", 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0850000000000003e-05 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.893107+0000", 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117689 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.893253+0000", 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000145315 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.896902+0000", 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003648999 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.100970+0000", 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204068056 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.101019+0000", 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9650000000000001e-05 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.101059+0000", 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0200999999999997e-05 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:76274 3.e 3:78f8db03:::benchmark_data_smithi194_88551_object76273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:30.852615+0000", 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 369.85082223299997, 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21036295799999999, 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 76274 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.852615+0000", 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.852615+0000", 2024-04-29T14:20:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.852617+0000", 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4750000000000001e-06 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855259+0000", 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026425530000000002 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855271+0000", 2024-04-29T14:20:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1609e-05 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855279+0000", 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7950000000000008e-06 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855332+0000", 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3758999999999998e-05 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855436+0000", 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103514 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855554+0000", 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011763400000000001 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.857987+0000", 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002433068 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.062877+0000", 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20488995199999999 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.062935+0000", 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8443999999999998e-05 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.062978+0000", 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3155000000000003e-05 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:76286 3.8 3:10617bc5:::benchmark_data_smithi194_88551_object76285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:30.858407+0000", 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 369.84503050900003, 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.209131233, 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 76286 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858407+0000", 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858407+0000", 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858409+0000", 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-29T14:20:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858476+0000", 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7498999999999999e-05 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858487+0000", 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1069e-05 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858494+0000", 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.860292+0000", 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001797508 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.860416+0000", 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012426200000000001 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.860562+0000", 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014601699999999999 2024-04-29T14:20:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.864911+0000", 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043489059999999996 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.067425+0000", 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202513375 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.067493+0000", 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8307999999999997e-05 2024-04-29T14:20:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.067538+0000", 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5414000000000003e-05 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:40.822 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:20:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:40.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:40.859 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:20:40.881 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:20:40.900 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:20:40.923 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:20:40.925 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:40.951 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:20:40.978 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:20:40.978 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:41.002 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:41.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:20:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:41.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:41.028 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:20:41.049 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:20:41.080 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:20:41.102 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:20:41.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:41.122 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:20:41.123 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:20:41.123 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:20:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:20:41.158 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:41.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1306957309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:41.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1306957309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:41.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1306957309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:41.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:41.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:41.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:20:41.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:41.588 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:41.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:41.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:41.768 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:41.796 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:41.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:41.874 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:41.902 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:41.933 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:41.960 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:41.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:42.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:42.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:42.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:42.260 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:42.299 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:42.322 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:42.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: pgmap v951: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:42.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:42.433 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: pgmap v951: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:20:42.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:42.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:42.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: pgmap v951: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:42.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:42.712 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:42.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:42.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:42.847 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:42.884 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:42.920 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:42.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:43.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:43.056 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:43.076 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:43.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:43.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:20:43.234 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 8] 2024-04-29T14:20:43.234 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:20:43.234 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-29T14:20:43.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:20:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:43.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:43.359 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:43.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:43 smithi040 ceph-mon[39581]: osdmap e362: 12 total, 12 up, 8 in 2024-04-29T14:20:43.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/366021991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:20:43.398 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:43.429 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:43.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:43.538 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:43.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:43 smithi121 ceph-mon[44533]: osdmap e362: 12 total, 12 up, 8 in 2024-04-29T14:20:43.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/366021991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:20:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:43 smithi194 ceph-mon[44089]: osdmap e362: 12 total, 12 up, 8 in 2024-04-29T14:20:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/366021991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:20:43.588 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:43.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:43.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:43.775 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:43.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:43.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:43.949 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:20:43.949 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:20:43.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:44.015 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:44.016 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:44.042 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:44.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:44.146 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:44.181 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:44.207 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:44.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:44.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: pgmap v953: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/366021991' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: osdmap e363: 12 total, 12 up, 8 in 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: 4.1 scrub starts 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: pgmap v955: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 863 op/s 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: pgmap v956: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:20:44.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:44 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:20:44.322 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:44.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:20:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:44.375 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:44.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:44.479 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:44.517 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:44.544 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:44.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: pgmap v953: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/366021991' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: osdmap e363: 12 total, 12 up, 8 in 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: 4.1 scrub starts 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: pgmap v955: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 863 op/s 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: pgmap v956: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:20:44.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:44 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: pgmap v953: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/366021991' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: osdmap e363: 12 total, 12 up, 8 in 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: 4.1 scrub starts 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:20:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: pgmap v955: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 863 op/s 2024-04-29T14:20:44.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: pgmap v956: 17 pgs: 17 active+clean; 8.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:20:44.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:44 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:20:44.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:44.677 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:44.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:20:44.925 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:44.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:45.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:45.058 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:45.173 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:46.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:20:46.421 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:47.056 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:47 smithi121 ceph-mon[44533]: 4.1 scrub ok 2024-04-29T14:20:47.056 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:47 smithi121 ceph-mon[44533]: pgmap v957: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s 2024-04-29T14:20:47.056 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:47 smithi121 ceph-mon[44533]: 4.3 scrub starts 2024-04-29T14:20:47.155 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:20:47.155 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:20:47.209 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:47.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:47 smithi040 ceph-mon[39581]: 4.1 scrub ok 2024-04-29T14:20:47.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:47 smithi040 ceph-mon[39581]: pgmap v957: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s 2024-04-29T14:20:47.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:47 smithi040 ceph-mon[39581]: 4.3 scrub starts 2024-04-29T14:20:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:47.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:47.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:47 smithi194 ceph-mon[44089]: 4.1 scrub ok 2024-04-29T14:20:47.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:47 smithi194 ceph-mon[44089]: pgmap v957: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s 2024-04-29T14:20:47.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:47 smithi194 ceph-mon[44089]: 4.3 scrub starts 2024-04-29T14:20:47.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:47.638 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 363, 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:20:43.212868+0000", 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:20:15.206702+0000", 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:19:01.251051+0000", 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5804032, 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:20:47.639 INFO:teuthology.orchestra.run.smithi040.stdout: "nodeep-scrub", 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 28, 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 4, 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:20:47.641 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:20:47.642 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:20:47.643 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:20:47.644 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:20:47.645 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 11.109999656677246, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 11.109999656677246, 2024-04-29T14:20:47.646 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.18000000715255737, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 4, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_2", 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:18:47.160984+0000", 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:20:47.647 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:20:47.648 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "326", 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:20:47.649 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:20:47.650 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:20:47.651 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.0499999523162842, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 2.0499999523162842, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.73000001907348633, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 1.5, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 1.5, 2024-04-29T14:20:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.653 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.654 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:20:47.655 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:20:47.656 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.657 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.658 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:20:47.660 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:47.661 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 152, 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:47.662 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.663 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:47.665 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:20:47.666 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.669 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:47.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 36, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:20:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.675 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:20:47.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/3807513901", 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/3807513901", 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3807513901", 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/3807513901", 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:20:47.677 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.678 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.679 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:20:47.680 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.681 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:47.682 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.683 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.684 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:20:47.686 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.687 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.688 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.689 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.690 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:20:47.691 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.693 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:20:47.694 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.696 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.697 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.700 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:20:47.701 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.702 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:24.377589+0000", 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:28.688717+0000", 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.703 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:30.922830+0000", 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:34.875980+0000", 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:31.658336+0000", 2024-04-29T14:20:47.705 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:50.723211+0000", 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:20:47.706 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:41.496306+0000", 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:33.310804+0000", 2024-04-29T14:20:47.707 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:14.794744+0000", 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:20:47.708 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:19:46.138814+0000", 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.709 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:35.646548+0000", 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:23.775639+0000", 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:20:47.710 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:20:47.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:20:47.712 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:20:47.713 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:20:47.828 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:47.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:20:48.304 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:48.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1758844382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:20:48.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1758844382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:20:48.460 INFO:tasks.thrashosds.thrasher:j is {'epoch': 363, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:20:43.212868+0000', 'last_up_change': '2024-04-29T14:20:15.206702+0000', 'last_in_change': '2024-04-29T14:19:01.251051+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.18000000715255737, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-29T14:18:47.160984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '326', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.049999952316284, 'score_stable': 2.049999952316284, 'optimal_score': 0.7300000190734863, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 152, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 3807513901}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3807513901}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3807513901}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 3807513901}]}, 'public_addr': '172.21.15.121:6804/3807513901', 'cluster_addr': '172.21.15.121:6805/3807513901', 'heartbeat_back_addr': '172.21.15.121:6807/3807513901', 'heartbeat_front_addr': '172.21.15.121:6806/3807513901', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 206, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:19:24.377589+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:20:28.688717+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:19:30.922830+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:19:34.875980+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:31.658336+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:19:50.723211+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:41.496306+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:33.310804+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:20:14.794744+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:19:46.138814+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:35.646548+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:23.775639+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:20:48.460 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-29T14:20:48.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1758844382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:20:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:48.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:48.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:49.110 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:49.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:49.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:49.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:49 smithi121 ceph-mon[44533]: pgmap v958: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 993 op/s 2024-04-29T14:20:49.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:49 smithi121 ceph-mon[44533]: osdmap e364: 12 total, 12 up, 8 in 2024-04-29T14:20:49.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:49 smithi121 ceph-mon[44533]: 4.3 scrub ok 2024-04-29T14:20:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:49.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:49.216 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:49.264 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:49.310 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:49.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:49 smithi040 ceph-mon[39581]: pgmap v958: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 993 op/s 2024-04-29T14:20:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:49 smithi040 ceph-mon[39581]: osdmap e364: 12 total, 12 up, 8 in 2024-04-29T14:20:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:49 smithi040 ceph-mon[39581]: 4.3 scrub ok 2024-04-29T14:20:49.414 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:49.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:49.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:49.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:49 smithi194 ceph-mon[44089]: pgmap v958: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 993 op/s 2024-04-29T14:20:49.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:49 smithi194 ceph-mon[44089]: osdmap e364: 12 total, 12 up, 8 in 2024-04-29T14:20:49.449 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:49 smithi194 ceph-mon[44089]: 4.3 scrub ok 2024-04-29T14:20:49.451 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:49.491 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:49.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:49.594 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:49.648 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:49.677 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:49.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:49.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:49.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:49.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:50.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:20:50.034 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:50.193 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:20:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:50.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:20:50.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:50.334 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:20:50.334 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:50.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:50.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:50.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:50.751 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:51.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:20:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:51 smithi040 ceph-mon[39581]: pgmap v960: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 750 op/s 2024-04-29T14:20:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3804914785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:51.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:51.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:51.161 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:20:51.161 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:20:51.161 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:20:51.161 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:20:51.161 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:20:51.180 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:20:51.216 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:20:51.228 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:51.237 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:20:51.259 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:20:51.264 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:51.283 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:20:51.292 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:51.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:51.308 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:20:51.329 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:20:51.330 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:20:51.330 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:20:51.330 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:20:51.331 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:20:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:51 smithi121 ceph-mon[44533]: pgmap v960: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 750 op/s 2024-04-29T14:20:51.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3804914785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:51.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:51.455 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:51.491 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:51.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:51.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:51 smithi194 ceph-mon[44089]: pgmap v960: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 750 op/s 2024-04-29T14:20:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3804914785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:51.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:20:51.638 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:51.691 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:20:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:51.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:20:51.798 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:51.845 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:51.881 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:51.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:52.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:52.084 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:52.126 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.126 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.126 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.126 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.126 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.126 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:20:52.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3310594197' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3310594197' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3310594197' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:20:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:52.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:52.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:53.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:53.120 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:53.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:53.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:53.296 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:53.332 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:53.363 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:53.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:53.379 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:53 smithi194 ceph-mon[44089]: pgmap v961: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-29T14:20:53.379 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:53 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:20:53.379 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3310594197' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:53.379 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:53 smithi194 ceph-mon[44089]: osdmap e365: 12 total, 12 up, 8 in 2024-04-29T14:20:53.379 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:20:53.379 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:20:53.461 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 8] 2024-04-29T14:20:53.461 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:20:53.461 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-29T14:20:53.461 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:20:53.461 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:20:53.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-04-29T14:20:53.478 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:53.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:53 smithi040 ceph-mon[39581]: pgmap v961: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-29T14:20:53.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:53 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:20:53.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3310594197' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:53.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:53 smithi040 ceph-mon[39581]: osdmap e365: 12 total, 12 up, 8 in 2024-04-29T14:20:53.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:20:53.488 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:20:53.514 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:53.541 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:53.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:53.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:53 smithi121 ceph-mon[44533]: pgmap v961: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-29T14:20:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:53 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:20:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3310594197' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:20:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:53 smithi121 ceph-mon[44533]: osdmap e365: 12 total, 12 up, 8 in 2024-04-29T14:20:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:20:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:20:53.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:53.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:53.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:53.832 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:54.032 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:54.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:54.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:54.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:20:54.193 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:54.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:54.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:54.336 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:54.379 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:54.404 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:54.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:184362 4.5 4:ad95659f:::benchmark_data_smithi194_103039_object27679:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:20:54.418921+0000", 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0065519139999999998, 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065754799999999999, 2024-04-29T14:20:54.422 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 184362 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.418921+0000", 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.418921+0000", 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.418924+0000", 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5569999999999998e-06 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.418934+0000", 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.079e-05 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.418949+0000", 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.438e-05 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.418958+0000", 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7089999999999997e-06 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.419020+0000", 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1143999999999996e-05 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.419145+0000", 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012494399999999999 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.419265+0000", 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120084 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:184364 4.5 4.7c24c435 (undecoded) ondisk+write+known_if_redirected e366)", 2024-04-29T14:20:54.426 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:20:54.420869+0000", 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0046038169999999996, 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046765970000000002, 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 184364 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.420869+0000", 2024-04-29T14:20:54.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.420869+0000", 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.420871+0000", 2024-04-29T14:20:54.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4190000000000001e-06 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.420875+0000", 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9389999999999996e-06 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.420885+0000", 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6330000000000008e-06 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:20:54.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:20:54.420892+0000", 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1389999999999998e-06 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 2 2024-04-29T14:20:54.430 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:20:54.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:54 smithi121 ceph-mon[44533]: osdmap e366: 12 total, 12 up, 8 in 2024-04-29T14:20:54.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:54.552 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:54.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:54 smithi194 ceph-mon[44089]: osdmap e366: 12 total, 12 up, 8 in 2024-04-29T14:20:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:54 smithi040 ceph-mon[39581]: osdmap e366: 12 total, 12 up, 8 in 2024-04-29T14:20:54.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:54.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:54.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:54.828 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:54.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:54.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:20:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:54.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:54.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:55.028 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:55.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:55.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:55.213 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:55.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:20:55.291 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:55.336 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:55.363 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:55.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:55.456 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:55 smithi040 ceph-mon[39581]: pgmap v964: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-04-29T14:20:55.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:55.515 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:55 smithi121 ceph-mon[44533]: pgmap v964: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-04-29T14:20:55.526 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:55.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:55.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:55.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:55 smithi194 ceph-mon[44089]: pgmap v964: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-04-29T14:20:55.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:55.681 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:55.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:55.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:55.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:20:55.880 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:55.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:20:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:55.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:20:56.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:20:56.142 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:56.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:20:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:56.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:20:56.282 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:56.320 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:56.341 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:56.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:56.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:56.479 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:56.518 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:56.520 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:20:56.520 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "" 2024-04-29T14:20:56.520 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:20:56.520 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:20:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:56.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:56.624 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:20:56.682 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:56.709 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:20:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:56.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:20:56.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:20:56.846 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:57 smithi040 ceph-mon[39581]: pgmap v965: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:20:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:57 smithi121 ceph-mon[44533]: pgmap v965: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:20:57.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:57 smithi194 ceph-mon[44089]: pgmap v965: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:20:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:20:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:57.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:20:57.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:20:57.753 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:20:57.753 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:20:57.753 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:20:57.753 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:20:57.753 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:20:57.759 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:58.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:20:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:58.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:20:58.152 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:20:58.190 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:58.225 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:20:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:58.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:20:58.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:20:58.372 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:58.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2331524858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:58.409 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:20:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:58.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:20:58.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:20:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:20:58.519 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:20:58.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:20:58.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:20:58.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:20:58.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2331524858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:20:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:58.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:20:58.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2331524858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:20:58.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:20:58.760 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:58.792 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:20:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:58.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:20:58.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:20:58.927 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:20:59.003 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:20:59.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: pgmap v966: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:20:59.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:20:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: pgmap v966: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:20:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: pgmap v966: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:20:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:20:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:20:59.877 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:20:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:20:59.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:20:59.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:00.034 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:00.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:00.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:00.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:00.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:00 smithi121 ceph-mon[44533]: osdmap e367: 12 total, 12 up, 8 in 2024-04-29T14:21:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:00 smithi194 ceph-mon[44089]: osdmap e367: 12 total, 12 up, 8 in 2024-04-29T14:21:00.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:00.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:00.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:00.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:00 smithi040 ceph-mon[39581]: osdmap e367: 12 total, 12 up, 8 in 2024-04-29T14:21:01.222 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:01 smithi121 ceph-mon[44533]: pgmap v968: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-04-29T14:21:01.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:01 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:01.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:01 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:01.223 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/689087476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:21:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:21:01.314 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:21:01.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:01 smithi194 ceph-mon[44089]: pgmap v968: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-04-29T14:21:01.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:01 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:01.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:01 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:01.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/689087476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:21:01.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:01 smithi040 ceph-mon[39581]: pgmap v968: 17 pgs: 17 active+clean; 7.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-04-29T14:21:01.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:01 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:01.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:01 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:01.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/689087476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:21:02.036 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:02.509 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:02 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:02.509 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/689087476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:21:02.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:02 smithi040 ceph-mon[39581]: osdmap e368: 12 total, 12 up, 8 in 2024-04-29T14:21:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:02 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/689087476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:21:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:02 smithi194 ceph-mon[44089]: osdmap e368: 12 total, 12 up, 8 in 2024-04-29T14:21:02.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:02 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:02.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/689087476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:21:02.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:02 smithi121 ceph-mon[44533]: osdmap e368: 12 total, 12 up, 8 in 2024-04-29T14:21:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:03 smithi040 ceph-mon[39581]: pgmap v970: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-29T14:21:03.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:03 smithi194 ceph-mon[44089]: pgmap v970: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-29T14:21:03.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:03 smithi121 ceph-mon[44533]: pgmap v970: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-29T14:21:03.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:21:04.032 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:04.052 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:04.092 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:04.113 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:04.141 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:04.181 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:21:04.211 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:04.249 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:04.272 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:04.298 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:04.331 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:04.355 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:04.411 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:04.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:04.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:04.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:04 smithi194 ceph-mon[44089]: pgmap v971: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-04-29T14:21:04.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/263253358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:04.560 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:04.611 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:04 smithi040 ceph-mon[39581]: pgmap v971: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-04-29T14:21:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/263253358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:04.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:04.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:04 smithi121 ceph-mon[44533]: pgmap v971: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-04-29T14:21:04.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/263253358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:04.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:04.749 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:04.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:04.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:04.961 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:05.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:05.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:05.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:05.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:05.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:05.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:05.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 8] 2024-04-29T14:21:05.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:21:05.805 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 2, 8] 2024-04-29T14:21:05.806 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-29T14:21:05.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5 2024-04-29T14:21:05.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:06.125 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:21:05 smithi121 systemd[1]: Stopping Ceph osd.5 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:21:06.213 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:06.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:06.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:06.368 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:06.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:06.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:06.437 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:06 smithi121 ceph-mon[44533]: osdmap e369: 12 total, 12 up, 8 in 2024-04-29T14:21:06.438 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:06 smithi121 ceph-mon[44533]: pgmap v973: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:21:06.438 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:21:06 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[57024]: 2024-04-29T14:21:06.333+0000 7f7e345f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T14:21:06.438 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:21:06 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[57024]: 2024-04-29T14:21:06.333+0000 7f7e345f3700 -1 osd.5 369 *** Got signal Terminated *** 2024-04-29T14:21:06.438 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:21:06 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[57024]: 2024-04-29T14:21:06.333+0000 7f7e345f3700 -1 osd.5 369 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T14:21:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:06 smithi040 ceph-mon[39581]: osdmap e369: 12 total, 12 up, 8 in 2024-04-29T14:21:06.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:06 smithi040 ceph-mon[39581]: pgmap v973: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:21:06.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:06 smithi194 ceph-mon[44089]: osdmap e369: 12 total, 12 up, 8 in 2024-04-29T14:21:06.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:06 smithi194 ceph-mon[44089]: pgmap v973: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:21:06.565 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:06.603 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:06.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:06.620 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:21:06.620 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:21:06.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:21:06.620 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:21:06.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173610 2.17 2:ea6ae847:::benchmark_data_smithi194_79000_object9349:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 598.48417201699999, 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.234702578, 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173610 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135777+0000", 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135779+0000", 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2069999999999998e-06 2024-04-29T14:21:06.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135785+0000", 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5849999999999999e-06 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135800+0000", 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5573000000000002e-05 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.135809+0000", 2024-04-29T14:21:06.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2870000000000004e-06 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.170786+0000", 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034977716999999998 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.170861+0000", 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4418000000000005e-05 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.171114+0000", 2024-04-29T14:21:06.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025352699999999997 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184325+0000", 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01321051 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370366+0000", 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.186041027 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370424+0000", 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7645999999999997e-05 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.370480+0000", 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6081000000000001e-05 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173617 2.17 2:ea4c2311:::benchmark_data_smithi194_79000_object9356:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:21:06.626 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 598.47521152700006, 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24862530199999999, 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173617 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144738+0000", 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144739+0000", 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8789999999999999e-06 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144745+0000", 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7599999999999999e-06 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144755+0000", 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7969999999999995e-06 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.144761+0000", 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1140000000000002e-06 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.173690+0000", 2024-04-29T14:21:06.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028928834 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.173816+0000", 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126067 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.174276+0000", 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045987699999999997 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184330+0000", 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010054110999999999 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393250+0000", 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208919664 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393302+0000", 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1860999999999997e-05 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393363+0000", 2024-04-29T14:21:06.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1338000000000001e-05 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173625 2.17 2:ed81edb2:::benchmark_data_smithi194_79000_object9364:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 598.47153526600005, 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24512054, 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:21:06.632 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173625 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148414+0000", 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148416+0000", 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.894e-06 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148421+0000", 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5319999999999997e-06 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148435+0000", 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3431000000000001e-05 2024-04-29T14:21:06.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.148443+0000", 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6719999999999993e-06 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.177311+0000", 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028867127999999999 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.177432+0000", 2024-04-29T14:21:06.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121236 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.178047+0000", 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061545999999999997 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187188+0000", 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091407750000000003 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393432+0000", 2024-04-29T14:21:06.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20624394500000001 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393478+0000", 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6011999999999999e-05 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393534+0000", 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6455000000000001e-05 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173632 2.17 2:ebef1b31:::benchmark_data_smithi194_79000_object9371:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 598.46055649899995, 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23825384199999999, 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173632 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159393+0000", 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159395+0000", 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0549999999999998e-06 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159401+0000", 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0569999999999998e-06 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159413+0000", 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2347999999999999e-05 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.159422+0000", 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9299999999999992e-06 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180545+0000", 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021122809999999999 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180656+0000", 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011134 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181142+0000", 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000486031 2024-04-29T14:21:06.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187196+0000", 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060541850000000001 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397527+0000", 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21033081000000001 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397582+0000", 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5235999999999998e-05 2024-04-29T14:21:06.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.397646+0000", 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4040000000000003e-05 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173636 2.7 2:e5bc6543:::benchmark_data_smithi194_79000_object9375:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 598.45989705500006, 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23324684400000001, 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.643 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173636 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160052+0000", 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160053+0000", 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.294e-06 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160058+0000", 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4100000000000001e-06 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160068+0000", 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7569999999999999e-06 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.160073+0000", 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8810000000000001e-06 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180528+0000", 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020454898999999999 2024-04-29T14:21:06.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.180612+0000", 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3422000000000002e-05 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181050+0000", 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000438391 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187192+0000", 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061421150000000001 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393229+0000", 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20603648399999999 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393264+0000", 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5549999999999997e-05 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393299+0000", 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4641000000000003e-05 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173640 2.7 2:e581d5ea:::benchmark_data_smithi194_79000_object9379:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 598.45590505300004, 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22945062699999999, 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173640 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164044+0000", 2024-04-29T14:21:06.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164046+0000", 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1409999999999999e-06 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164056+0000", 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008e-05 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164100+0000", 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3893000000000001e-05 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.164109+0000", 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6889999999999993e-06 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184149+0000", 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020040235999999999 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184270+0000", 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120791 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184634+0000", 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036420799999999998 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187199+0000", 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025650199999999999 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393415+0000", 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20621562099999999 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393462+0000", 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7055999999999998e-05 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.393495+0000", 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2963999999999998e-05 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24727.0:173648 2.17 2:eb9a7570:::benchmark_data_smithi194_79000_object9387:head [delete] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 598.43862741999999, 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26978428700000001, 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24727", 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:06.655 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 173648 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181322+0000", 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181324+0000", 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.874e-06 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181330+0000", 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2210000000000002e-06 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181342+0000", 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2e-05 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.181350+0000", 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.106e-06 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184353+0000", 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030029319999999998 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184447+0000", 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3710999999999998e-05 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.184955+0000", 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050829300000000005 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.187337+0000", 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00238265 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451020+0000", 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.263682898 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451050+0000", 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9578999999999999e-05 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:11:08.451106+0000", 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6023000000000002e-05 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164479 3.2 3:4e25dff6:::benchmark_data_smithi194_88551_object5536:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:21:06.660 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055608+0000", 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 326.56434081399999, 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20862729099999999, 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164479 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055608+0000", 2024-04-29T14:21:06.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055608+0000", 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055609+0000", 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1200000000000002e-07 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055612+0000", 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3230000000000002e-06 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055619+0000", 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7190000000000004e-06 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055625+0000", 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2190000000000001e-06 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055687+0000", 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1450000000000003e-05 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055694+0000", 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1520000000000003e-06 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264161+0000", 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208467175 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264236+0000", 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4541000000000003e-05 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164489 3.2 3:45db8f19:::benchmark_data_smithi194_88551_object5546:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055633+0000", 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 326.56431627199999, 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20871237500000001, 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:21:06.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164489 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055633+0000", 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055633+0000", 2024-04-29T14:21:06.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055634+0000", 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7899999999999998e-07 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055636+0000", 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3709999999999998e-06 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055641+0000", 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7899999999999999e-06 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055646+0000", 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3800000000000002e-06 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055721+0000", 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4852999999999996e-05 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055728+0000", 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9009999999999997e-06 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264274+0000", 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208546641 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264345+0000", 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0759999999999993e-05 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164495 3.2 3:4824d7da:::benchmark_data_smithi194_88551_object5552:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055652+0000", 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 326.56429698900001, 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20880806800000001, 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164495 2024-04-29T14:21:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055652+0000", 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055652+0000", 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055653+0000", 2024-04-29T14:21:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5000000000000003e-07 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055655+0000", 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4449999999999999e-06 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055662+0000", 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7719999999999997e-06 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055667+0000", 2024-04-29T14:21:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3979999999999998e-06 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055778+0000", 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110528 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055787+0000", 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5559999999999996e-06 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264382+0000", 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208594639 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264460+0000", 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.818e-05 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164502 3.2 3:4a6173d7:::benchmark_data_smithi194_88551_object5559:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055680+0000", 2024-04-29T14:21:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 326.56426865499998, 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23313735499999999, 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164502 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055680+0000", 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055680+0000", 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055681+0000", 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8599999999999998e-07 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055692+0000", 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.084e-05 2024-04-29T14:21:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055698+0000", 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.28e-06 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055704+0000", 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8370000000000002e-06 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055832+0000", 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012847100000000001 2024-04-29T14:21:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055841+0000", 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7299999999999994e-06 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264650+0000", 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20880926299999999 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264786+0000", 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013555899999999999 2024-04-29T14:21:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.264924+0000", 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013797500000000001 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.274924+0000", 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010000136999999999 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.274975+0000", 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0387999999999999e-05 2024-04-29T14:21:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288697+0000", 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013722695 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288736+0000", 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8791999999999997e-05 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288818+0000", 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1802e-05 2024-04-29T14:21:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164503 3.2 3:4fa1a650:::benchmark_data_smithi194_88551_object5560:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055710+0000", 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 326.56423863399999, 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.233259412, 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:21:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164503 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055710+0000", 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055710+0000", 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055711+0000", 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299999999999996e-07 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055714+0000", 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.559e-06 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055719+0000", 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6529999999999999e-06 2024-04-29T14:21:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055724+0000", 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1200000000000001e-06 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055882+0000", 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000157415 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055890+0000", 2024-04-29T14:21:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7510000000000008e-06 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.269776+0000", 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21388687300000001 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.269899+0000", 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122368 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.270030+0000", 2024-04-29T14:21:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013152800000000001 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.274989+0000", 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049582719999999997 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275028+0000", 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9774999999999998e-05 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288853+0000", 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013824881000000001 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288888+0000", 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4275e-05 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288970+0000", 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2368999999999997e-05 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:164512 3.2 3:40a769b0:::benchmark_data_smithi194_88551_object5569:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e255)", 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:40.055730+0000", 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 326.56421907399999, 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.233439548, 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:21:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164512 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055730+0000", 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055730+0000", 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055731+0000", 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-07 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055733+0000", 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.278e-06 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055739+0000", 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0589999999999999e-06 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055744+0000", 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2839999999999997e-06 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055916+0000", 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017157199999999999 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.055924+0000", 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3059999999999998e-06 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275061+0000", 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21913727699999999 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275176+0000", 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115135 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.275327+0000", 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000150261 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288612+0000", 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01328532 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.288650+0000", 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8164000000000003e-05 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.289023+0000", 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037293899999999997 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.289076+0000", 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2913e-05 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:40.289170+0000", 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3516000000000004e-05 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:182558 3.2 3:4b7f77b6:::benchmark_data_smithi194_88551_object23615:head [delete] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-04-29T14:21:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:15:59.601230+0000", 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 307.01871863500003, 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.222199009, 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 182558 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601230+0000", 2024-04-29T14:21:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601230+0000", 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601232+0000", 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.579e-06 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601237+0000", 2024-04-29T14:21:06.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1479999999999997e-06 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601249+0000", 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1958e-05 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601256+0000", 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1709999999999996e-06 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601317+0000", 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1110000000000003e-05 2024-04-29T14:21:06.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601441+0000", 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012336000000000001 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.601557+0000", 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001164 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.606861+0000", 2024-04-29T14:21:06.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053038130000000001 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.608977+0000", 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021159239999999999 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823254+0000", 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21427707500000001 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823320+0000", 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6439999999999994e-05 2024-04-29T14:21:06.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:15:59.823429+0000", 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010903099999999999 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:112231 4.5 4:a54fe868:::benchmark_data_smithi194_103039_object112230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:19:47.604067+0000", 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.015882145000006, 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21402157999999999, 2024-04-29T14:21:06.700 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112231 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.604067+0000", 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.604067+0000", 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.604068+0000", 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2130000000000001e-06 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.604164+0000", 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6243999999999995e-05 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.604184+0000", 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9839000000000001e-05 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.604192+0000", 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.391e-06 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.607672+0000", 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003480225 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.607811+0000", 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013943399999999999 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.607940+0000", 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012885300000000001 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.615753+0000", 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078126399999999992 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.818007+0000", 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202253877 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.818050+0000", 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3556000000000001e-05 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.818089+0000", 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8308e-05 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:112239 4.5 4:a03b99ef:::benchmark_data_smithi194_103039_object112238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:19:47.611868+0000", 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.008080820000004, 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.217963194, 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:21:06.706 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112239 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.611868+0000", 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.611868+0000", 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.611869+0000", 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7999999999999995e-07 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.611905+0000", 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5800999999999998e-05 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.611911+0000", 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0179999999999998e-06 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.611916+0000", 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5550000000000002e-06 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.612151+0000", 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023519400000000001 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.612259+0000", 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107842 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.612379+0000", 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119734 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616009+0000", 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036297920000000002 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.829763+0000", 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21375439099999999 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.829803+0000", 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9863000000000001e-05 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.829831+0000", 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8424000000000001e-05 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:112251 4.5 4:ad0238e8:::benchmark_data_smithi194_103039_object112250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:19:47.616534+0000", 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.003414969999994, 2024-04-29T14:21:06.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22479059400000001, 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112251 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616534+0000", 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616534+0000", 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616536+0000", 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4529999999999999e-06 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616656+0000", 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120772 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616673+0000", 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6512e-05 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616681+0000", 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7000000000000008e-06 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616725+0000", 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4336999999999999e-05 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616858+0000", 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013270999999999999 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.616949+0000", 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1568000000000001e-05 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.619606+0000", 2024-04-29T14:21:06.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002656458 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841215+0000", 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.221609525 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841277+0000", 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1818999999999999e-05 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841325+0000", 2024-04-29T14:21:06.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.774e-05 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:112256 4.5 4:a8fa4a48:::benchmark_data_smithi194_103039_object112255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:19:47.623910+0000", 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.996039479000004, 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21754448100000001, 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112256 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.623910+0000", 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.623910+0000", 2024-04-29T14:21:06.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.623911+0000", 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.699e-06 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.624027+0000", 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116006 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.624041+0000", 2024-04-29T14:21:06.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3375e-05 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.624049+0000", 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3590000000000008e-06 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.624109+0000", 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0208999999999999e-05 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.624204+0000", 2024-04-29T14:21:06.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4380000000000001e-05 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.624336+0000", 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013241800000000001 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.627084+0000", 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027477539999999998 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841369+0000", 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.214284945 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841418+0000", 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9316e-05 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841454+0000", 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6019999999999997e-05 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:112262 4.5 4:aec421b1:::benchmark_data_smithi194_103039_object112261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e346)", 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:19:47.628195+0000", 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.991754280999999, 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21338773899999999, 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112262 2024-04-29T14:21:06.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628195+0000", 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628195+0000", 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628196+0000", 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2389999999999999e-06 2024-04-29T14:21:06.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628252+0000", 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5553000000000002e-05 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628264+0000", 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2813000000000001e-05 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628273+0000", 2024-04-29T14:21:06.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8829999999999995e-06 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628328+0000", 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5154000000000001e-05 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628421+0000", 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2434000000000003e-05 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.628505+0000", 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4363000000000005e-05 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.631990+0000", 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034848959999999999 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841496+0000", 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20950619600000001 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841546+0000", 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9531999999999999e-05 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:19:47.841583+0000", 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6675999999999999e-05 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:195471 4.9 4:934574c4:::benchmark_data_smithi194_103039_object38788:head [delete] snapc 0=[] ondisk+write+known_if_redirected e368)", 2024-04-29T14:21:06.728 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:21:04.759672+0000", 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.8602775409999999, 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060517989000000001, 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 195471 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759672+0000", 2024-04-29T14:21:06.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759672+0000", 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759673+0000", 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0380000000000001e-06 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759676+0000", 2024-04-29T14:21:06.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.861e-06 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759684+0000", 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0759999999999997e-06 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759689+0000", 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460000000000003e-06 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:21:06.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759726+0000", 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6179e-05 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759839+0000", 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112836 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.759937+0000", 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8356999999999994e-05 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:21:06.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.764570+0000", 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046335430000000004 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.820086+0000", 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055515238000000001 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.820145+0000", 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8872999999999997e-05 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:21:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:21:04.820190+0000", 2024-04-29T14:21:06.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5042e-05 2024-04-29T14:21:06.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.734 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:21:06.734 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:21:06.734 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:21:06.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:06.912 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:06.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:07.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:07.044 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:07.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:07.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:07.225 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:07.267 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:21:07.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:07 smithi040 ceph-mon[39581]: osd.5 marked itself down and dead 2024-04-29T14:21:07.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2336151423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:07.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:07 smithi121 ceph-mon[44533]: osd.5 marked itself down and dead 2024-04-29T14:21:07.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2336151423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:07.427 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:21:07.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:07.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:07.547 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:07.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:07 smithi194 ceph-mon[44089]: osd.5 marked itself down and dead 2024-04-29T14:21:07.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2336151423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:07.586 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:07.605 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:07.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:07.684 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:07.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:07.738 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:08.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:08.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:08.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:21:08.224 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:08.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:08 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:08.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:08 smithi040 ceph-mon[39581]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:21:08.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2336151423' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:08.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:08 smithi040 ceph-mon[39581]: osdmap e370: 12 total, 11 up, 8 in 2024-04-29T14:21:08.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:08 smithi040 ceph-mon[39581]: pgmap v975: 17 pgs: 1 stale+active+clean, 16 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-04-29T14:21:08.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:08.374 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:21:08 smithi121 bash[110448]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5 2024-04-29T14:21:08.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:08 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:08.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:08 smithi121 ceph-mon[44533]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:21:08.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2336151423' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:08.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:08 smithi121 ceph-mon[44533]: osdmap e370: 12 total, 11 up, 8 in 2024-04-29T14:21:08.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:08 smithi121 ceph-mon[44533]: pgmap v975: 17 pgs: 1 stale+active+clean, 16 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-04-29T14:21:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:08.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:08.477 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:08.478 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:08.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:08 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:08.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:08 smithi194 ceph-mon[44089]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:21:08.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2336151423' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:08.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:08 smithi194 ceph-mon[44089]: osdmap e370: 12 total, 11 up, 8 in 2024-04-29T14:21:08.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:08 smithi194 ceph-mon[44089]: pgmap v975: 17 pgs: 1 stale+active+clean, 16 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-04-29T14:21:08.506 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:08.525 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:08.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:08.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:08.677 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:08.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:08.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:08.809 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:08.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:08.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:08.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:09.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:09.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:09 smithi194 ceph-mon[44089]: osdmap e371: 12 total, 11 up, 8 in 2024-04-29T14:21:09.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:09.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:09.478 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:09.502 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:09 smithi040 ceph-mon[39581]: osdmap e371: 12 total, 11 up, 8 in 2024-04-29T14:21:09.504 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:09.581 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:21:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:21:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:21:09.581 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:21:09.611 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:09 smithi121 ceph-mon[44533]: osdmap e371: 12 total, 11 up, 8 in 2024-04-29T14:21:09.853 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:09.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:09.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5.service' 2024-04-29T14:21:09.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:09.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T14:21:09.960 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-29T14:21:09.960 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-04-29T14:21:09.960 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi121.front.sepia.ceph.com 2024-04-29T14:21:09.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-04-29T14:21:10.026 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:10.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:10.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:10.069 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:21:10.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:10.177 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:10.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:10.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:10.208 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:21:10.208 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:21:10.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:10.327 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:10.341 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:10 smithi040 ceph-mon[39581]: pgmap v977: 17 pgs: 1 stale+active+clean, 16 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-04-29T14:21:10.362 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:10.377 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:10.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:10.411 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:10.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:10 smithi121 ceph-mon[44533]: pgmap v977: 17 pgs: 1 stale+active+clean, 16 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-04-29T14:21:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:10.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:10.432 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:10.452 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:10.472 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:10.473 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:10.493 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:10.513 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:10.529 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:10.533 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:10.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:10 smithi194 ceph-mon[44089]: pgmap v977: 17 pgs: 1 stale+active+clean, 16 active+clean; 7.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-04-29T14:21:10.556 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:10.557 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:10.755 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:10.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:10.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:11.338 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:11.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:11.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4138391676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:11.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.413 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:21:11.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4138391676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.543 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:11.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4138391676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:11.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:11.679 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:11.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:11.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:11.909 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:11.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:12.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:12.381 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:21:12.382 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:21:12.430 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:21:12.430 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:21:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:12.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:12.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:12.779 INFO:teuthology.orchestra.run.smithi121.stdout:4.1 2024-04-29T14:21:12.779 INFO:teuthology.orchestra.run.smithi121.stdout:4.2 2024-04-29T14:21:12.803 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:12.822 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:12.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:12.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:12.958 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:12.977 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:12.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:12.988 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:21:13.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:13.110 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:13.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:13.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:13.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:13.251 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:13.260 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:13 smithi040 ceph-mon[39581]: pgmap v978: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:13.260 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:13 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:21:13.262 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:13.281 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:13.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:13 smithi194 ceph-mon[44089]: pgmap v978: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:13 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:21:13.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:13 smithi121 ceph-mon[44533]: pgmap v978: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:13.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:13 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:21:13.385 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:13.412 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:13.509 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:13.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:13.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:13.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:13.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:13.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 4.2 --file /var/log/ceph/exp.4.2.5 2024-04-29T14:21:13.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:13.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:13.924 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:21:13.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:13.956 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:14.028 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:14.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:14.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:21:14.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:14.158 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3095033794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:14 smithi194 ceph-mon[44089]: from='client.35573 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:14.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3095033794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:14.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:14 smithi040 ceph-mon[39581]: from='client.35573 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3095033794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:14 smithi121 ceph-mon[44533]: from='client.35573 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:14.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:21:14.460 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:14.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:14.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:14.594 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:14.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:14.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:14.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:14.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:14.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:14.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:14.909 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:14.929 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:14.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:14.999 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:21:14.999 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:21:15.033 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:15.059 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:21:15.061 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:15.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:15 smithi040 ceph-mon[39581]: pgmap v979: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 423 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:15.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:15 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:15.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:15 smithi040 ceph-mon[39581]: from='client.35573 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:21:15.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:15 smithi040 ceph-mon[39581]: osdmap e372: 12 total, 11 up, 8 in 2024-04-29T14:21:15.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:15 smithi121 ceph-mon[44533]: pgmap v979: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 423 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:15.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:15 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:15.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:15 smithi121 ceph-mon[44533]: from='client.35573 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:21:15.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:15 smithi121 ceph-mon[44533]: osdmap e372: 12 total, 11 up, 8 in 2024-04-29T14:21:15.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:15 smithi194 ceph-mon[44089]: pgmap v979: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 423 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:15.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:15 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:15.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:15 smithi194 ceph-mon[44089]: from='client.35573 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:21:15.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:15 smithi194 ceph-mon[44089]: osdmap e372: 12 total, 11 up, 8 in 2024-04-29T14:21:15.711 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:15.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:15.720 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:21:15.815 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:15.842 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:15.860 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:15.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:15.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:15.993 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:16.025 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:16.192 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:16.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:16.296 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:16.325 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:16.346 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:16.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:Exporting 4.2 info 4.2( v 369'12412 (331'2400,369'12412] local-lis/les=323/324 n=7198 ec=323/323 lis/c=323/323 les/c/f=324/324/0 sis=323) 2024-04-29T14:21:16.449 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:16.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40008f8d:::benchmark_data_smithi194_103039_object53885:head# 2024-04-29T14:21:16.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4000a56f:::benchmark_data_smithi194_103039_object100393:head# 2024-04-29T14:21:16.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4000cedc:::benchmark_data_smithi194_103039_object95179:head# 2024-04-29T14:21:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4001136b:::benchmark_data_smithi194_103039_object101004:head# 2024-04-29T14:21:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40012cac:::benchmark_data_smithi194_103039_object48225:head# 2024-04-29T14:21:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4002a611:::benchmark_data_smithi194_103039_object124037:head# 2024-04-29T14:21:16.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40034f76:::benchmark_data_smithi194_103039_object62540:head# 2024-04-29T14:21:16.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40049b3c:::benchmark_data_smithi194_103039_object45463:head# 2024-04-29T14:21:16.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4004caf8:::benchmark_data_smithi194_103039_object145811:head# 2024-04-29T14:21:16.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4004ec10:::benchmark_data_smithi194_103039_object121598:head# 2024-04-29T14:21:16.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400509e5:::benchmark_data_smithi194_103039_object86930:head# 2024-04-29T14:21:16.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4005586b:::benchmark_data_smithi194_103039_object52095:head# 2024-04-29T14:21:16.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40058371:::benchmark_data_smithi194_103039_object115442:head# 2024-04-29T14:21:16.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4005e590:::benchmark_data_smithi194_103039_object116004:head# 2024-04-29T14:21:16.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4006d085:::benchmark_data_smithi194_103039_object97077:head# 2024-04-29T14:21:16.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40071796:::benchmark_data_smithi194_103039_object115657:head# 2024-04-29T14:21:16.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40075e72:::benchmark_data_smithi194_103039_object47203:head# 2024-04-29T14:21:16.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4007b797:::benchmark_data_smithi194_103039_object47546:head# 2024-04-29T14:21:16.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4007c404:::benchmark_data_smithi194_103039_object125086:head# 2024-04-29T14:21:16.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4008bd5f:::benchmark_data_smithi194_103039_object61879:head# 2024-04-29T14:21:16.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4008c78e:::benchmark_data_smithi194_103039_object127502:head# 2024-04-29T14:21:16.476 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:16.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4008d114:::benchmark_data_smithi194_103039_object155618:head# 2024-04-29T14:21:16.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40091baa:::benchmark_data_smithi194_103039_object92011:head# 2024-04-29T14:21:16.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40097ef5:::benchmark_data_smithi194_103039_object138997:head# 2024-04-29T14:21:16.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4009bb39:::benchmark_data_smithi194_103039_object131244:head# 2024-04-29T14:21:16.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4009fa58:::benchmark_data_smithi194_103039_object97699:head# 2024-04-29T14:21:16.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400a3c51:::benchmark_data_smithi194_103039_object144285:head# 2024-04-29T14:21:16.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400b38f0:::benchmark_data_smithi194_103039_object45138:head# 2024-04-29T14:21:16.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400cc9bc:::benchmark_data_smithi194_103039_object107880:head# 2024-04-29T14:21:16.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400d3947:::benchmark_data_smithi194_103039_object82231:head# 2024-04-29T14:21:16.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400da435:::benchmark_data_smithi194_103039_object154795:head# 2024-04-29T14:21:16.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400dcf1b:::benchmark_data_smithi194_103039_object109359:head# 2024-04-29T14:21:16.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:400f6823:::benchmark_data_smithi194_103039_object82242:head# 2024-04-29T14:21:16.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40111fa5:::benchmark_data_smithi194_103039_object138578:head# 2024-04-29T14:21:16.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40115986:::benchmark_data_smithi194_103039_object84970:head# 2024-04-29T14:21:16.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4012759c:::benchmark_data_smithi194_103039_object107099:head# 2024-04-29T14:21:16.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4012fb74:::benchmark_data_smithi194_103039_object52943:head# 2024-04-29T14:21:16.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40143fe9:::benchmark_data_smithi194_103039_object69443:head# 2024-04-29T14:21:16.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401470f0:::benchmark_data_smithi194_103039_object123342:head# 2024-04-29T14:21:16.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4014a979:::benchmark_data_smithi194_103039_object150463:head# 2024-04-29T14:21:16.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4015b2af:::benchmark_data_smithi194_103039_object51369:head# 2024-04-29T14:21:16.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4016c215:::benchmark_data_smithi194_103039_object68627:head# 2024-04-29T14:21:16.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4016f240:::benchmark_data_smithi194_103039_object119595:head# 2024-04-29T14:21:16.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4016f9df:::benchmark_data_smithi194_103039_object128021:head# 2024-04-29T14:21:16.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40170408:::benchmark_data_smithi194_103039_object118058:head# 2024-04-29T14:21:16.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40182254:::benchmark_data_smithi194_103039_object108862:head# 2024-04-29T14:21:16.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401836b3:::benchmark_data_smithi194_103039_object97567:head# 2024-04-29T14:21:16.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401858e3:::benchmark_data_smithi194_103039_object129178:head# 2024-04-29T14:21:16.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40186707:::benchmark_data_smithi194_103039_object75459:head# 2024-04-29T14:21:16.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401967bb:::benchmark_data_smithi194_103039_object74218:head# 2024-04-29T14:21:16.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4019e36e:::benchmark_data_smithi194_103039_object59937:head# 2024-04-29T14:21:16.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401a8f9f:::benchmark_data_smithi194_103039_object72742:head# 2024-04-29T14:21:16.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401b7f7d:::benchmark_data_smithi194_103039_object142696:head# 2024-04-29T14:21:16.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401bda78:::benchmark_data_smithi194_103039_object95682:head# 2024-04-29T14:21:16.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401c1b12:::benchmark_data_smithi194_103039_object134175:head# 2024-04-29T14:21:16.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401c80ef:::benchmark_data_smithi194_103039_object135529:head# 2024-04-29T14:21:16.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401ce4bb:::benchmark_data_smithi194_103039_object65032:head# 2024-04-29T14:21:16.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:401d4f8c:::benchmark_data_smithi194_103039_object43324:head# 2024-04-29T14:21:16.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40209b53:::benchmark_data_smithi194_103039_object92757:head# 2024-04-29T14:21:16.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4022d098:::benchmark_data_smithi194_103039_object143576:head# 2024-04-29T14:21:16.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40231eab:::benchmark_data_smithi194_103039_object60966:head# 2024-04-29T14:21:16.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40232dd0:::benchmark_data_smithi194_103039_object150384:head# 2024-04-29T14:21:16.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4023794b:::benchmark_data_smithi194_103039_object94555:head# 2024-04-29T14:21:16.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402461fc:::benchmark_data_smithi194_103039_object121890:head# 2024-04-29T14:21:16.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40255ed6:::benchmark_data_smithi194_103039_object112007:head# 2024-04-29T14:21:16.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40258975:::benchmark_data_smithi194_103039_object156062:head# 2024-04-29T14:21:16.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4025c323:::benchmark_data_smithi194_103039_object149851:head# 2024-04-29T14:21:16.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4025ff77:::benchmark_data_smithi194_103039_object73390:head# 2024-04-29T14:21:16.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4026986d:::benchmark_data_smithi194_103039_object100474:head# 2024-04-29T14:21:16.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4026ba51:::benchmark_data_smithi194_103039_object129107:head# 2024-04-29T14:21:16.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4026dd73:::benchmark_data_smithi194_103039_object121295:head# 2024-04-29T14:21:16.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40274d0c:::benchmark_data_smithi194_103039_object97516:head# 2024-04-29T14:21:16.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40278c39:::benchmark_data_smithi194_103039_object133516:head# 2024-04-29T14:21:16.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4027b459:::benchmark_data_smithi194_103039_object92201:head# 2024-04-29T14:21:16.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4027f8ee:::benchmark_data_smithi194_103039_object143648:head# 2024-04-29T14:21:16.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402917fa:::benchmark_data_smithi194_103039_object45599:head# 2024-04-29T14:21:16.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402961d1:::benchmark_data_smithi194_103039_object107490:head# 2024-04-29T14:21:16.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4029bc9e:::benchmark_data_smithi194_103039_object45090:head# 2024-04-29T14:21:16.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402a8737:::benchmark_data_smithi194_103039_object135143:head# 2024-04-29T14:21:16.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402b2639:::benchmark_data_smithi194_103039_object42022:head# 2024-04-29T14:21:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402b66a9:::benchmark_data_smithi194_103039_object76413:head# 2024-04-29T14:21:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402b924b:::benchmark_data_smithi194_103039_object86141:head# 2024-04-29T14:21:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402bbbe1:::benchmark_data_smithi194_103039_object46515:head# 2024-04-29T14:21:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402d7bfa:::benchmark_data_smithi194_103039_object114926:head# 2024-04-29T14:21:16.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402f86f8:::benchmark_data_smithi194_103039_object128230:head# 2024-04-29T14:21:16.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402f9860:::benchmark_data_smithi194_103039_object76909:head# 2024-04-29T14:21:16.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:402f9b9e:::benchmark_data_smithi194_103039_object55243:head# 2024-04-29T14:21:16.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4030080a:::benchmark_data_smithi194_103039_object154785:head# 2024-04-29T14:21:16.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4030a955:::benchmark_data_smithi194_103039_object69373:head# 2024-04-29T14:21:16.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4030c102:::benchmark_data_smithi194_103039_object127510:head# 2024-04-29T14:21:16.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4030d3b0:::benchmark_data_smithi194_103039_object133930:head# 2024-04-29T14:21:16.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4030fb89:::benchmark_data_smithi194_103039_object96781:head# 2024-04-29T14:21:16.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40317c85:::benchmark_data_smithi194_103039_object43904:head# 2024-04-29T14:21:16.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40337993:::benchmark_data_smithi194_103039_object109263:head# 2024-04-29T14:21:16.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40339717:::benchmark_data_smithi194_103039_object75367:head# 2024-04-29T14:21:16.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40349f11:::benchmark_data_smithi194_103039_object66055:head# 2024-04-29T14:21:16.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40354541:::benchmark_data_smithi194_103039_object131353:head# 2024-04-29T14:21:16.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4035a24a:::benchmark_data_smithi194_103039_object97509:head# 2024-04-29T14:21:16.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4035ff77:::benchmark_data_smithi194_103039_object53942:head# 2024-04-29T14:21:16.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40360543:::benchmark_data_smithi194_103039_object98151:head# 2024-04-29T14:21:16.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403716c2:::benchmark_data_smithi194_103039_object134805:head# 2024-04-29T14:21:16.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40373158:::benchmark_data_smithi194_103039_object122964:head# 2024-04-29T14:21:16.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40378d19:::benchmark_data_smithi194_103039_object89041:head# 2024-04-29T14:21:16.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4038a4f7:::benchmark_data_smithi194_103039_object61896:head# 2024-04-29T14:21:16.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4038a539:::benchmark_data_smithi194_103039_object150045:head# 2024-04-29T14:21:16.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4039be8f:::benchmark_data_smithi194_103039_object95016:head# 2024-04-29T14:21:16.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403a56ef:::benchmark_data_smithi194_103039_object156110:head# 2024-04-29T14:21:16.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403aac5e:::benchmark_data_smithi194_103039_object57146:head# 2024-04-29T14:21:16.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403b3906:::benchmark_data_smithi194_103039_object115560:head# 2024-04-29T14:21:16.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403b588f:::benchmark_data_smithi194_103039_object131161:head# 2024-04-29T14:21:16.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403bad91:::benchmark_data_smithi194_103039_object80148:head# 2024-04-29T14:21:16.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403bfe87:::benchmark_data_smithi194_103039_object72646:head# 2024-04-29T14:21:16.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403c4aa8:::benchmark_data_smithi194_103039_object89202:head# 2024-04-29T14:21:16.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403cf35a:::benchmark_data_smithi194_103039_object137225:head# 2024-04-29T14:21:16.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403d9d95:::benchmark_data_smithi194_103039_object48607:head# 2024-04-29T14:21:16.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403df1d4:::benchmark_data_smithi194_103039_object53363:head# 2024-04-29T14:21:16.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403e0a64:::benchmark_data_smithi194_103039_object142892:head# 2024-04-29T14:21:16.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403e34f0:::benchmark_data_smithi194_103039_object67786:head# 2024-04-29T14:21:16.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:403f6667:::benchmark_data_smithi194_103039_object56222:head# 2024-04-29T14:21:16.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40402200:::benchmark_data_smithi194_103039_object145431:head# 2024-04-29T14:21:16.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40402743:::benchmark_data_smithi194_103039_object65611:head# 2024-04-29T14:21:16.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404051f2:::benchmark_data_smithi194_103039_object141666:head# 2024-04-29T14:21:16.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4041d032:::benchmark_data_smithi194_103039_object153062:head# 2024-04-29T14:21:16.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4041f5a2:::benchmark_data_smithi194_103039_object107577:head# 2024-04-29T14:21:16.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4042434e:::benchmark_data_smithi194_103039_object76036:head# 2024-04-29T14:21:16.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4043216a:::benchmark_data_smithi194_103039_object49606:head# 2024-04-29T14:21:16.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40434a03:::benchmark_data_smithi194_103039_object152622:head# 2024-04-29T14:21:16.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4044fd1b:::benchmark_data_smithi194_103039_object133533:head# 2024-04-29T14:21:16.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40453fa8:::benchmark_data_smithi194_103039_object112497:head# 2024-04-29T14:21:16.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40458959:::benchmark_data_smithi194_103039_object68096:head# 2024-04-29T14:21:16.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40459a18:::benchmark_data_smithi194_103039_object80568:head# 2024-04-29T14:21:16.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40467029:::benchmark_data_smithi194_103039_object57621:head# 2024-04-29T14:21:16.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40467ca1:::benchmark_data_smithi194_103039_object57809:head# 2024-04-29T14:21:16.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4046c13c:::benchmark_data_smithi194_103039_object129273:head# 2024-04-29T14:21:16.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40480360:::benchmark_data_smithi194_103039_object71546:head# 2024-04-29T14:21:16.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4048774b:::benchmark_data_smithi194_103039_object65542:head# 2024-04-29T14:21:16.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404b5d9f:::benchmark_data_smithi194_103039_object94672:head# 2024-04-29T14:21:16.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404bce9f:::benchmark_data_smithi194_103039_object85774:head# 2024-04-29T14:21:16.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404c1c70:::benchmark_data_smithi194_103039_object132664:head# 2024-04-29T14:21:16.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404c742b:::benchmark_data_smithi194_103039_object136750:head# 2024-04-29T14:21:16.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404c859b:::benchmark_data_smithi194_103039_object97891:head# 2024-04-29T14:21:16.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404ccdb9:::benchmark_data_smithi194_103039_object125050:head# 2024-04-29T14:21:16.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404e1d0e:::benchmark_data_smithi194_103039_object44970:head# 2024-04-29T14:21:16.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404e50b0:::benchmark_data_smithi194_103039_object103560:head# 2024-04-29T14:21:16.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404e6f4c:::benchmark_data_smithi194_103039_object107390:head# 2024-04-29T14:21:16.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404ecd84:::benchmark_data_smithi194_103039_object107638:head# 2024-04-29T14:21:16.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:404ee6fa:::benchmark_data_smithi194_103039_object143549:head# 2024-04-29T14:21:16.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40505c8c:::benchmark_data_smithi194_103039_object74044:head# 2024-04-29T14:21:16.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40507607:::benchmark_data_smithi194_103039_object65192:head# 2024-04-29T14:21:16.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4050d8c6:::benchmark_data_smithi194_103039_object76487:head# 2024-04-29T14:21:16.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40518792:::benchmark_data_smithi194_103039_object52012:head# 2024-04-29T14:21:16.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40543ae5:::benchmark_data_smithi194_103039_object94293:head# 2024-04-29T14:21:16.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405461e7:::benchmark_data_smithi194_103039_object127765:head# 2024-04-29T14:21:16.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4054aedc:::benchmark_data_smithi194_103039_object107322:head# 2024-04-29T14:21:16.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4054d89d:::benchmark_data_smithi194_103039_object77260:head# 2024-04-29T14:21:16.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40554582:::benchmark_data_smithi194_103039_object99450:head# 2024-04-29T14:21:16.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4056440c:::benchmark_data_smithi194_103039_object124675:head# 2024-04-29T14:21:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405668f6:::benchmark_data_smithi194_103039_object112003:head# 2024-04-29T14:21:16.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40568a80:::benchmark_data_smithi194_103039_object69511:head# 2024-04-29T14:21:16.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4056d7e8:::benchmark_data_smithi194_103039_object42392:head# 2024-04-29T14:21:16.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405710f8:::benchmark_data_smithi194_103039_object140478:head# 2024-04-29T14:21:16.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405731d1:::benchmark_data_smithi194_103039_object56818:head# 2024-04-29T14:21:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4057d506:::benchmark_data_smithi194_103039_object146782:head# 2024-04-29T14:21:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405a13ba:::benchmark_data_smithi194_103039_object126382:head# 2024-04-29T14:21:16.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405a2ead:::benchmark_data_smithi194_103039_object57634:head# 2024-04-29T14:21:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405a3842:::benchmark_data_smithi194_103039_object43407:head# 2024-04-29T14:21:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405a6ef7:::benchmark_data_smithi194_103039_object109400:head# 2024-04-29T14:21:16.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405acc9e:::benchmark_data_smithi194_103039_object90774:head# 2024-04-29T14:21:16.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405bb9e0:::benchmark_data_smithi194_103039_object106436:head# 2024-04-29T14:21:16.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405bc9c1:::benchmark_data_smithi194_103039_object85854:head# 2024-04-29T14:21:16.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405df3cd:::benchmark_data_smithi194_103039_object111486:head# 2024-04-29T14:21:16.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405f1a78:::benchmark_data_smithi194_103039_object71976:head# 2024-04-29T14:21:16.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405fbd2e:::benchmark_data_smithi194_103039_object42486:head# 2024-04-29T14:21:16.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405fc069:::benchmark_data_smithi194_103039_object103508:head# 2024-04-29T14:21:16.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:405ff56c:::benchmark_data_smithi194_103039_object53965:head# 2024-04-29T14:21:16.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406005cc:::benchmark_data_smithi194_103039_object126570:head# 2024-04-29T14:21:16.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4060622c:::benchmark_data_smithi194_103039_object109647:head# 2024-04-29T14:21:16.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40615569:::benchmark_data_smithi194_103039_object96241:head# 2024-04-29T14:21:16.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4061a4a1:::benchmark_data_smithi194_103039_object47396:head# 2024-04-29T14:21:16.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4062ed04:::benchmark_data_smithi194_103039_object57773:head# 2024-04-29T14:21:16.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4063b07f:::benchmark_data_smithi194_103039_object78672:head# 2024-04-29T14:21:16.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4064520b:::benchmark_data_smithi194_103039_object135639:head# 2024-04-29T14:21:16.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40646b62:::benchmark_data_smithi194_103039_object114960:head# 2024-04-29T14:21:16.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4065e6c7:::benchmark_data_smithi194_103039_object67691:head# 2024-04-29T14:21:16.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4065f8b3:::benchmark_data_smithi194_103039_object75809:head# 2024-04-29T14:21:16.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40665dc1:::benchmark_data_smithi194_103039_object73108:head# 2024-04-29T14:21:16.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4066cb7a:::benchmark_data_smithi194_103039_object132739:head# 2024-04-29T14:21:16.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4066d01d:::benchmark_data_smithi194_103039_object136694:head# 2024-04-29T14:21:16.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40672966:::benchmark_data_smithi194_103039_object88103:head# 2024-04-29T14:21:16.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4067b5ba:::benchmark_data_smithi194_103039_object89799:head# 2024-04-29T14:21:16.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4068e552:::benchmark_data_smithi194_103039_object69444:head# 2024-04-29T14:21:16.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406b2fd2:::benchmark_data_smithi194_103039_object119406:head# 2024-04-29T14:21:16.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406b64f8:::benchmark_data_smithi194_103039_object66069:head# 2024-04-29T14:21:16.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406ba2be:::benchmark_data_smithi194_103039_object47612:head# 2024-04-29T14:21:16.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406d4b40:::benchmark_data_smithi194_103039_object149137:head# 2024-04-29T14:21:16.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406d4c70:::benchmark_data_smithi194_103039_object101956:head# 2024-04-29T14:21:16.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406d9ed0:::benchmark_data_smithi194_103039_object152217:head# 2024-04-29T14:21:16.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406de59c:::benchmark_data_smithi194_103039_object51753:head# 2024-04-29T14:21:16.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406e55fd:::benchmark_data_smithi194_103039_object136004:head# 2024-04-29T14:21:16.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406eb686:::benchmark_data_smithi194_103039_object104082:head# 2024-04-29T14:21:16.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406ee4bc:::benchmark_data_smithi194_103039_object105817:head# 2024-04-29T14:21:16.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406f4832:::benchmark_data_smithi194_103039_object148967:head# 2024-04-29T14:21:16.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406f8f40:::benchmark_data_smithi194_103039_object154805:head# 2024-04-29T14:21:16.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:406f90d0:::benchmark_data_smithi194_103039_object100264:head# 2024-04-29T14:21:16.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4070407f:::benchmark_data_smithi194_103039_object82323:head# 2024-04-29T14:21:16.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40707790:::benchmark_data_smithi194_103039_object60907:head# 2024-04-29T14:21:16.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407085f0:::benchmark_data_smithi194_103039_object125550:head# 2024-04-29T14:21:16.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4070c675:::benchmark_data_smithi194_103039_object128189:head# 2024-04-29T14:21:16.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4071d347:::benchmark_data_smithi194_103039_object96391:head# 2024-04-29T14:21:16.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4072ce6a:::benchmark_data_smithi194_103039_object112641:head# 2024-04-29T14:21:16.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4073f1dc:::benchmark_data_smithi194_103039_object59620:head# 2024-04-29T14:21:16.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407498ff:::benchmark_data_smithi194_103039_object113912:head# 2024-04-29T14:21:16.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407663e2:::benchmark_data_smithi194_103039_object121887:head# 2024-04-29T14:21:16.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40779c39:::benchmark_data_smithi194_103039_object56143:head# 2024-04-29T14:21:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4077ede0:::benchmark_data_smithi194_103039_object139665:head# 2024-04-29T14:21:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40783368:::benchmark_data_smithi194_103039_object114720:head# 2024-04-29T14:21:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40785dd2:::benchmark_data_smithi194_103039_object64997:head# 2024-04-29T14:21:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4078be8d:::benchmark_data_smithi194_103039_object138245:head# 2024-04-29T14:21:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407b43be:::benchmark_data_smithi194_103039_object127526:head# 2024-04-29T14:21:16.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407c06f1:::benchmark_data_smithi194_103039_object73958:head# 2024-04-29T14:21:16.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407c10bb:::benchmark_data_smithi194_103039_object77324:head# 2024-04-29T14:21:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407c3a30:::benchmark_data_smithi194_103039_object94734:head# 2024-04-29T14:21:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407c7473:::benchmark_data_smithi194_103039_object123525:head# 2024-04-29T14:21:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407d08a7:::benchmark_data_smithi194_103039_object55691:head# 2024-04-29T14:21:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407d3243:::benchmark_data_smithi194_103039_object82184:head# 2024-04-29T14:21:16.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407dccb0:::benchmark_data_smithi194_103039_object74330:head# 2024-04-29T14:21:16.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:407eec6e:::benchmark_data_smithi194_103039_object63745:head# 2024-04-29T14:21:16.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40800aa7:::benchmark_data_smithi194_103039_object115242:head# 2024-04-29T14:21:16.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4080189e:::benchmark_data_smithi194_103039_object102062:head# 2024-04-29T14:21:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40802c93:::benchmark_data_smithi194_103039_object146510:head# 2024-04-29T14:21:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408073a7:::benchmark_data_smithi194_103039_object117185:head# 2024-04-29T14:21:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408093b1:::benchmark_data_smithi194_103039_object72552:head# 2024-04-29T14:21:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4080ce06:::benchmark_data_smithi194_103039_object132477:head# 2024-04-29T14:21:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4080ef8d:::benchmark_data_smithi194_103039_object89123:head# 2024-04-29T14:21:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4080fd1d:::benchmark_data_smithi194_103039_object99180:head# 2024-04-29T14:21:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408126b4:::benchmark_data_smithi194_103039_object127481:head# 2024-04-29T14:21:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4081fd10:::benchmark_data_smithi194_103039_object105285:head# 2024-04-29T14:21:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4082e439:::benchmark_data_smithi194_103039_object136367:head# 2024-04-29T14:21:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4082f61a:::benchmark_data_smithi194_103039_object114279:head# 2024-04-29T14:21:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40839736:::benchmark_data_smithi194_103039_object147008:head# 2024-04-29T14:21:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4083e29a:::benchmark_data_smithi194_103039_object135374:head# 2024-04-29T14:21:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4083ece7:::benchmark_data_smithi194_103039_object54547:head# 2024-04-29T14:21:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40840c47:::benchmark_data_smithi194_103039_object117065:head# 2024-04-29T14:21:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4084a82a:::benchmark_data_smithi194_103039_object99466:head# 2024-04-29T14:21:16.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4084fe07:::benchmark_data_smithi194_103039_object129233:head# 2024-04-29T14:21:16.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4086eaf6:::benchmark_data_smithi194_103039_object48264:head# 2024-04-29T14:21:16.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4087c8e0:::benchmark_data_smithi194_103039_object55586:head# 2024-04-29T14:21:16.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4088e781:::benchmark_data_smithi194_103039_object60611:head# 2024-04-29T14:21:16.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408917c3:::benchmark_data_smithi194_103039_object61629:head# 2024-04-29T14:21:16.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40892d26:::benchmark_data_smithi194_103039_object152583:head# 2024-04-29T14:21:16.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408a0529:::benchmark_data_smithi194_103039_object92128:head# 2024-04-29T14:21:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408a6d0f:::benchmark_data_smithi194_103039_object92086:head# 2024-04-29T14:21:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408b10d1:::benchmark_data_smithi194_103039_object67814:head# 2024-04-29T14:21:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408cc062:::benchmark_data_smithi194_103039_object156265:head# 2024-04-29T14:21:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408d66c8:::benchmark_data_smithi194_103039_object126733:head# 2024-04-29T14:21:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408d8c34:::benchmark_data_smithi194_103039_object96693:head# 2024-04-29T14:21:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408f05cf:::benchmark_data_smithi194_103039_object152050:head# 2024-04-29T14:21:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408f3c40:::benchmark_data_smithi194_103039_object152303:head# 2024-04-29T14:21:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408f68da:::benchmark_data_smithi194_103039_object59448:head# 2024-04-29T14:21:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408f6e83:::benchmark_data_smithi194_103039_object69505:head# 2024-04-29T14:21:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:408fae48:::benchmark_data_smithi194_103039_object79551:head# 2024-04-29T14:21:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40902e72:::benchmark_data_smithi194_103039_object73795:head# 2024-04-29T14:21:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4090bb6e:::benchmark_data_smithi194_103039_object46439:head# 2024-04-29T14:21:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409255bd:::benchmark_data_smithi194_103039_object128647:head# 2024-04-29T14:21:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4092a0ff:::benchmark_data_smithi194_103039_object140446:head# 2024-04-29T14:21:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40930250:::benchmark_data_smithi194_103039_object101959:head# 2024-04-29T14:21:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4093a5e0:::benchmark_data_smithi194_103039_object71643:head# 2024-04-29T14:21:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4094e197:::benchmark_data_smithi194_103039_object107608:head# 2024-04-29T14:21:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4095e89c:::benchmark_data_smithi194_103039_object118081:head# 2024-04-29T14:21:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4095f4f4:::benchmark_data_smithi194_103039_object54387:head# 2024-04-29T14:21:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409672cb:::benchmark_data_smithi194_103039_object70468:head# 2024-04-29T14:21:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4097d673:::benchmark_data_smithi194_103039_object76725:head# 2024-04-29T14:21:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40989d5d:::benchmark_data_smithi194_103039_object121463:head# 2024-04-29T14:21:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40996d50:::benchmark_data_smithi194_103039_object121761:head# 2024-04-29T14:21:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409a2ac2:::benchmark_data_smithi194_103039_object72972:head# 2024-04-29T14:21:16.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409a6f90:::benchmark_data_smithi194_103039_object132233:head# 2024-04-29T14:21:16.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409ac517:::benchmark_data_smithi194_103039_object93693:head# 2024-04-29T14:21:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409b2023:::benchmark_data_smithi194_103039_object75400:head# 2024-04-29T14:21:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409bbc3f:::benchmark_data_smithi194_103039_object112144:head# 2024-04-29T14:21:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409c28e4:::benchmark_data_smithi194_103039_object149701:head# 2024-04-29T14:21:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409c6ea8:::benchmark_data_smithi194_103039_object69533:head# 2024-04-29T14:21:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:409dddd9:::benchmark_data_smithi194_103039_object53506:head# 2024-04-29T14:21:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a051e8:::benchmark_data_smithi194_103039_object138747:head# 2024-04-29T14:21:16.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a24c48:::benchmark_data_smithi194_103039_object100689:head# 2024-04-29T14:21:16.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a5032e:::benchmark_data_smithi194_103039_object79709:head# 2024-04-29T14:21:16.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a57cc8:::benchmark_data_smithi194_103039_object81593:head# 2024-04-29T14:21:16.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a5fdf9:::benchmark_data_smithi194_103039_object72135:head# 2024-04-29T14:21:16.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a63e25:::benchmark_data_smithi194_103039_object64384:head# 2024-04-29T14:21:16.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a68860:::benchmark_data_smithi194_103039_object144579:head# 2024-04-29T14:21:16.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a6bfac:::benchmark_data_smithi194_103039_object119535:head# 2024-04-29T14:21:16.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a742ab:::benchmark_data_smithi194_103039_object43389:head# 2024-04-29T14:21:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a77790:::benchmark_data_smithi194_103039_object99795:head# 2024-04-29T14:21:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a7fd6d:::benchmark_data_smithi194_103039_object47218:head# 2024-04-29T14:21:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a8c81b:::benchmark_data_smithi194_103039_object67130:head# 2024-04-29T14:21:16.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a8d199:::benchmark_data_smithi194_103039_object126656:head# 2024-04-29T14:21:16.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a8ea4e:::benchmark_data_smithi194_103039_object144660:head# 2024-04-29T14:21:16.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40a9f5b1:::benchmark_data_smithi194_103039_object111564:head# 2024-04-29T14:21:16.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40aa3313:::benchmark_data_smithi194_103039_object53877:head# 2024-04-29T14:21:16.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ab5b0e:::benchmark_data_smithi194_103039_object98565:head# 2024-04-29T14:21:16.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40abc325:::benchmark_data_smithi194_103039_object139442:head# 2024-04-29T14:21:16.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ac87b6:::benchmark_data_smithi194_103039_object100478:head# 2024-04-29T14:21:16.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40aca3d9:::benchmark_data_smithi194_103039_object125139:head# 2024-04-29T14:21:16.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40acc45d:::benchmark_data_smithi194_103039_object128857:head# 2024-04-29T14:21:16.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ae57d1:::benchmark_data_smithi194_103039_object115690:head# 2024-04-29T14:21:16.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40afb1bc:::benchmark_data_smithi194_103039_object62283:head# 2024-04-29T14:21:16.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b07863:::benchmark_data_smithi194_103039_object130659:head# 2024-04-29T14:21:16.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b09840:::benchmark_data_smithi194_103039_object71736:head# 2024-04-29T14:21:16.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b0ac9d:::benchmark_data_smithi194_103039_object81415:head# 2024-04-29T14:21:16.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b13a38:::benchmark_data_smithi194_103039_object141286:head# 2024-04-29T14:21:16.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b176dc:::benchmark_data_smithi194_103039_object125909:head# 2024-04-29T14:21:16.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b1aa2b:::benchmark_data_smithi194_103039_object47990:head# 2024-04-29T14:21:16.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b1c543:::benchmark_data_smithi194_103039_object156061:head# 2024-04-29T14:21:16.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b1eef2:::benchmark_data_smithi194_103039_object106554:head# 2024-04-29T14:21:16.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b21eba:::benchmark_data_smithi194_103039_object101404:head# 2024-04-29T14:21:16.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b291f5:::benchmark_data_smithi194_103039_object95925:head# 2024-04-29T14:21:16.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b36d5a:::benchmark_data_smithi194_103039_object81784:head# 2024-04-29T14:21:16.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b392e5:::benchmark_data_smithi194_103039_object110636:head# 2024-04-29T14:21:16.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b3ed7a:::benchmark_data_smithi194_103039_object114209:head# 2024-04-29T14:21:16.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b68cec:::benchmark_data_smithi194_103039_object82946:head# 2024-04-29T14:21:16.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b6d951:::benchmark_data_smithi194_103039_object51774:head# 2024-04-29T14:21:16.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b719c1:::benchmark_data_smithi194_103039_object76401:head# 2024-04-29T14:21:16.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b7aa62:::benchmark_data_smithi194_103039_object136505:head# 2024-04-29T14:21:16.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b82750:::benchmark_data_smithi194_103039_object75678:head# 2024-04-29T14:21:16.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b871dd:::benchmark_data_smithi194_103039_object77807:head# 2024-04-29T14:21:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b8c92b:::benchmark_data_smithi194_103039_object75248:head# 2024-04-29T14:21:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b8d1df:::benchmark_data_smithi194_103039_object92945:head# 2024-04-29T14:21:16.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b950e0:::benchmark_data_smithi194_103039_object151244:head# 2024-04-29T14:21:16.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40b9b562:::benchmark_data_smithi194_103039_object111749:head# 2024-04-29T14:21:16.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ba66ad:::benchmark_data_smithi194_103039_object47566:head# 2024-04-29T14:21:16.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40bb193d:::benchmark_data_smithi194_103039_object133478:head# 2024-04-29T14:21:16.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40bb8229:::benchmark_data_smithi194_103039_object122190:head# 2024-04-29T14:21:16.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40bc0adf:::benchmark_data_smithi194_103039_object151524:head# 2024-04-29T14:21:16.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40bd020a:::benchmark_data_smithi194_103039_object92168:head# 2024-04-29T14:21:16.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40bdede9:::benchmark_data_smithi194_103039_object119217:head# 2024-04-29T14:21:16.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40be8927:::benchmark_data_smithi194_103039_object97647:head# 2024-04-29T14:21:16.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40bec734:::benchmark_data_smithi194_103039_object149730:head# 2024-04-29T14:21:16.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c01f87:::benchmark_data_smithi194_103039_object128512:head# 2024-04-29T14:21:16.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c07ab8:::benchmark_data_smithi194_103039_object134425:head# 2024-04-29T14:21:16.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c0fcec:::benchmark_data_smithi194_103039_object140571:head# 2024-04-29T14:21:16.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c20a41:::benchmark_data_smithi194_103039_object149834:head# 2024-04-29T14:21:16.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c32bd4:::benchmark_data_smithi194_103039_object105575:head# 2024-04-29T14:21:16.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c44e3f:::benchmark_data_smithi194_103039_object135801:head# 2024-04-29T14:21:16.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c56d0a:::benchmark_data_smithi194_103039_object99019:head# 2024-04-29T14:21:16.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c5c5ad:::benchmark_data_smithi194_103039_object112651:head# 2024-04-29T14:21:16.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c64a92:::benchmark_data_smithi194_103039_object130865:head# 2024-04-29T14:21:16.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c68a0f:::benchmark_data_smithi194_103039_object108444:head# 2024-04-29T14:21:16.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c7036a:::benchmark_data_smithi194_103039_object58162:head# 2024-04-29T14:21:16.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c71830:::benchmark_data_smithi194_103039_object77628:head# 2024-04-29T14:21:16.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c71c02:::benchmark_data_smithi194_103039_object87537:head# 2024-04-29T14:21:16.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c763a5:::benchmark_data_smithi194_103039_object109728:head# 2024-04-29T14:21:16.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c7e90f:::benchmark_data_smithi194_103039_object46812:head# 2024-04-29T14:21:16.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c817e6:::benchmark_data_smithi194_103039_object142306:head# 2024-04-29T14:21:16.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40c8c6fd:::benchmark_data_smithi194_103039_object146317:head# 2024-04-29T14:21:16.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40cb03b0:::benchmark_data_smithi194_103039_object128474:head# 2024-04-29T14:21:16.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ccabce:::benchmark_data_smithi194_103039_object141348:head# 2024-04-29T14:21:16.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ccacb9:::benchmark_data_smithi194_103039_object49986:head# 2024-04-29T14:21:16.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40cd07bd:::benchmark_data_smithi194_103039_object105792:head# 2024-04-29T14:21:16.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40cdc6bc:::benchmark_data_smithi194_103039_object77139:head# 2024-04-29T14:21:16.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ce0434:::benchmark_data_smithi194_103039_object113435:head# 2024-04-29T14:21:16.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ce1921:::benchmark_data_smithi194_103039_object109701:head# 2024-04-29T14:21:16.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ce4ede:::benchmark_data_smithi194_103039_object136272:head# 2024-04-29T14:21:16.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ce62a5:::benchmark_data_smithi194_103039_object86374:head# 2024-04-29T14:21:16.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ce7e3b:::benchmark_data_smithi194_103039_object48399:head# 2024-04-29T14:21:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40cebc60:::benchmark_data_smithi194_103039_object50175:head# 2024-04-29T14:21:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40cecf4f:::benchmark_data_smithi194_103039_object80948:head# 2024-04-29T14:21:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d0c068:::benchmark_data_smithi194_103039_object75714:head# 2024-04-29T14:21:16.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d18080:::benchmark_data_smithi194_103039_object94449:head# 2024-04-29T14:21:16.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d2c33c:::benchmark_data_smithi194_103039_object155201:head# 2024-04-29T14:21:16.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d42525:::benchmark_data_smithi194_103039_object47205:head# 2024-04-29T14:21:16.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d4d7ef:::benchmark_data_smithi194_103039_object118574:head# 2024-04-29T14:21:16.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d512a9:::benchmark_data_smithi194_103039_object96960:head# 2024-04-29T14:21:16.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d51aa2:::benchmark_data_smithi194_103039_object56625:head# 2024-04-29T14:21:16.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d546f3:::benchmark_data_smithi194_103039_object113436:head# 2024-04-29T14:21:16.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d555b0:::benchmark_data_smithi194_103039_object43222:head# 2024-04-29T14:21:16.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d82827:::benchmark_data_smithi194_103039_object42354:head# 2024-04-29T14:21:16.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40d8566c:::benchmark_data_smithi194_103039_object103435:head# 2024-04-29T14:21:16.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40da6069:::benchmark_data_smithi194_103039_object98678:head# 2024-04-29T14:21:16.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40da8c57:::benchmark_data_smithi194_103039_object118837:head# 2024-04-29T14:21:16.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40db9793:::benchmark_data_smithi194_103039_object81893:head# 2024-04-29T14:21:16.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40dbf7ae:::benchmark_data_smithi194_103039_object66998:head# 2024-04-29T14:21:16.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40dc2d3c:::benchmark_data_smithi194_103039_object67927:head# 2024-04-29T14:21:16.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40dc78b1:::benchmark_data_smithi194_103039_object109089:head# 2024-04-29T14:21:16.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40dc8e96:::benchmark_data_smithi194_103039_object88755:head# 2024-04-29T14:21:16.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40de665a:::benchmark_data_smithi194_103039_object143442:head# 2024-04-29T14:21:16.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e12df0:::benchmark_data_smithi194_103039_object101804:head# 2024-04-29T14:21:16.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e3223a:::benchmark_data_smithi194_103039_object107782:head# 2024-04-29T14:21:16.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e413e5:::benchmark_data_smithi194_103039_object83439:head# 2024-04-29T14:21:16.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e4b5ae:::benchmark_data_smithi194_103039_object143346:head# 2024-04-29T14:21:16.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e4c636:::benchmark_data_smithi194_103039_object117123:head# 2024-04-29T14:21:16.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e5269f:::benchmark_data_smithi194_103039_object75543:head# 2024-04-29T14:21:16.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e55bc6:::benchmark_data_smithi194_103039_object115210:head# 2024-04-29T14:21:16.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e566f9:::benchmark_data_smithi194_103039_object76261:head# 2024-04-29T14:21:16.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e69c4c:::benchmark_data_smithi194_103039_object153130:head# 2024-04-29T14:21:16.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e6ed9f:::benchmark_data_smithi194_103039_object123839:head# 2024-04-29T14:21:16.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e73cf6:::benchmark_data_smithi194_103039_object133910:head# 2024-04-29T14:21:16.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e74dc2:::benchmark_data_smithi194_103039_object75621:head# 2024-04-29T14:21:16.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e8ee54:::benchmark_data_smithi194_103039_object47617:head# 2024-04-29T14:21:16.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40e9107c:::benchmark_data_smithi194_103039_object57090:head# 2024-04-29T14:21:16.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ea102d:::benchmark_data_smithi194_103039_object106397:head# 2024-04-29T14:21:16.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ea63ff:::benchmark_data_smithi194_103039_object78758:head# 2024-04-29T14:21:16.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40eb111b:::benchmark_data_smithi194_103039_object134558:head# 2024-04-29T14:21:16.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40eb3f11:::benchmark_data_smithi194_103039_object86554:head# 2024-04-29T14:21:16.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ec234f:::benchmark_data_smithi194_103039_object58614:head# 2024-04-29T14:21:16.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ed14b7:::benchmark_data_smithi194_103039_object79841:head# 2024-04-29T14:21:16.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40edb056:::benchmark_data_smithi194_103039_object133040:head# 2024-04-29T14:21:16.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40edc54a:::benchmark_data_smithi194_103039_object135280:head# 2024-04-29T14:21:16.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40ee5fd7:::benchmark_data_smithi194_103039_object154491:head# 2024-04-29T14:21:16.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40eebcb2:::benchmark_data_smithi194_103039_object103192:head# 2024-04-29T14:21:16.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f04f7c:::benchmark_data_smithi194_103039_object153478:head# 2024-04-29T14:21:16.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f25216:::benchmark_data_smithi194_103039_object93685:head# 2024-04-29T14:21:16.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f264ac:::benchmark_data_smithi194_103039_object96369:head# 2024-04-29T14:21:16.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f2dcb5:::benchmark_data_smithi194_103039_object111456:head# 2024-04-29T14:21:16.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f2dcfd:::benchmark_data_smithi194_103039_object137231:head# 2024-04-29T14:21:16.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f35fc0:::benchmark_data_smithi194_103039_object105744:head# 2024-04-29T14:21:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f4a5a8:::benchmark_data_smithi194_103039_object74522:head# 2024-04-29T14:21:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f4b972:::benchmark_data_smithi194_103039_object112124:head# 2024-04-29T14:21:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f529ab:::benchmark_data_smithi194_103039_object152733:head# 2024-04-29T14:21:16.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f5d2fa:::benchmark_data_smithi194_103039_object73440:head# 2024-04-29T14:21:16.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f5ed16:::benchmark_data_smithi194_103039_object127814:head# 2024-04-29T14:21:16.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f61451:::benchmark_data_smithi194_103039_object120862:head# 2024-04-29T14:21:16.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f65ec7:::benchmark_data_smithi194_103039_object68414:head# 2024-04-29T14:21:16.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f811ac:::benchmark_data_smithi194_103039_object139392:head# 2024-04-29T14:21:16.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f86b57:::benchmark_data_smithi194_103039_object44172:head# 2024-04-29T14:21:16.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40f9b58e:::benchmark_data_smithi194_103039_object115616:head# 2024-04-29T14:21:16.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40fd1a22:::benchmark_data_smithi194_103039_object108799:head# 2024-04-29T14:21:16.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40fd6378:::benchmark_data_smithi194_103039_object61099:head# 2024-04-29T14:21:16.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40feb0ea:::benchmark_data_smithi194_103039_object102584:head# 2024-04-29T14:21:16.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40fefb14:::benchmark_data_smithi194_103039_object84610:head# 2024-04-29T14:21:16.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:40fff4be:::benchmark_data_smithi194_103039_object49902:head# 2024-04-29T14:21:16.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41001d3e:::benchmark_data_smithi194_103039_object122324:head# 2024-04-29T14:21:16.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41004cfa:::benchmark_data_smithi194_103039_object58299:head# 2024-04-29T14:21:16.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41006b07:::benchmark_data_smithi194_103039_object85267:head# 2024-04-29T14:21:16.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410143d3:::benchmark_data_smithi194_103039_object149241:head# 2024-04-29T14:21:16.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4101fd3a:::benchmark_data_smithi194_103039_object144946:head# 2024-04-29T14:21:16.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41021bf9:::benchmark_data_smithi194_103039_object121108:head# 2024-04-29T14:21:16.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410322b2:::benchmark_data_smithi194_103039_object103518:head# 2024-04-29T14:21:16.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410331a3:::benchmark_data_smithi194_103039_object46735:head# 2024-04-29T14:21:16.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4103b794:::benchmark_data_smithi194_103039_object119862:head# 2024-04-29T14:21:16.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41046ee2:::benchmark_data_smithi194_103039_object98649:head# 2024-04-29T14:21:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4104ef42:::benchmark_data_smithi194_103039_object67348:head# 2024-04-29T14:21:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410543d2:::benchmark_data_smithi194_103039_object105172:head# 2024-04-29T14:21:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41055530:::benchmark_data_smithi194_103039_object103726:head# 2024-04-29T14:21:16.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41059e9e:::benchmark_data_smithi194_103039_object77951:head# 2024-04-29T14:21:16.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4105eb8e:::benchmark_data_smithi194_103039_object58514:head# 2024-04-29T14:21:16.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410619e4:::benchmark_data_smithi194_103039_object111713:head# 2024-04-29T14:21:16.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41066de8:::benchmark_data_smithi194_103039_object87310:head# 2024-04-29T14:21:16.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4106cca8:::benchmark_data_smithi194_103039_object110875:head# 2024-04-29T14:21:16.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4106ea95:::benchmark_data_smithi194_103039_object66844:head# 2024-04-29T14:21:16.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410735b5:::benchmark_data_smithi194_103039_object138591:head# 2024-04-29T14:21:16.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4107e68f:::benchmark_data_smithi194_103039_object145591:head# 2024-04-29T14:21:16.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4108f79e:::benchmark_data_smithi194_103039_object145443:head# 2024-04-29T14:21:16.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41091872:::benchmark_data_smithi194_103039_object138759:head# 2024-04-29T14:21:16.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4109e8f2:::benchmark_data_smithi194_103039_object87663:head# 2024-04-29T14:21:16.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410a8418:::benchmark_data_smithi194_103039_object108888:head# 2024-04-29T14:21:16.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410affce:::benchmark_data_smithi194_103039_object51728:head# 2024-04-29T14:21:16.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410b1ed2:::benchmark_data_smithi194_103039_object98348:head# 2024-04-29T14:21:16.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410caf04:::benchmark_data_smithi194_103039_object112615:head# 2024-04-29T14:21:16.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410df4b9:::benchmark_data_smithi194_103039_object81391:head# 2024-04-29T14:21:16.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410e798c:::benchmark_data_smithi194_103039_object111568:head# 2024-04-29T14:21:16.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:410ed17a:::benchmark_data_smithi194_103039_object145506:head# 2024-04-29T14:21:16.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411019a8:::benchmark_data_smithi194_103039_object149689:head# 2024-04-29T14:21:16.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41120274:::benchmark_data_smithi194_103039_object47765:head# 2024-04-29T14:21:16.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41126428:::benchmark_data_smithi194_103039_object69650:head# 2024-04-29T14:21:16.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4113fe07:::benchmark_data_smithi194_103039_object83731:head# 2024-04-29T14:21:16.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41143d4d:::benchmark_data_smithi194_103039_object84239:head# 2024-04-29T14:21:16.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4114affd:::benchmark_data_smithi194_103039_object154479:head# 2024-04-29T14:21:16.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4116b8cc:::benchmark_data_smithi194_103039_object141768:head# 2024-04-29T14:21:16.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41173277:::benchmark_data_smithi194_103039_object80424:head# 2024-04-29T14:21:16.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4117ae07:::benchmark_data_smithi194_103039_object51366:head# 2024-04-29T14:21:16.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4118d9fa:::benchmark_data_smithi194_103039_object129731:head# 2024-04-29T14:21:16.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411929c4:::benchmark_data_smithi194_103039_object98773:head# 2024-04-29T14:21:16.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411932ed:::benchmark_data_smithi194_103039_object154618:head# 2024-04-29T14:21:16.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411aece1:::benchmark_data_smithi194_103039_object59283:head# 2024-04-29T14:21:16.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411c2c8c:::benchmark_data_smithi194_103039_object92941:head# 2024-04-29T14:21:16.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411c4ff7:::benchmark_data_smithi194_103039_object96665:head# 2024-04-29T14:21:16.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411cd196:::benchmark_data_smithi194_103039_object142717:head# 2024-04-29T14:21:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411e151c:::benchmark_data_smithi194_103039_object94062:head# 2024-04-29T14:21:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411e32a9:::benchmark_data_smithi194_103039_object44644:head# 2024-04-29T14:21:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411e4e35:::benchmark_data_smithi194_103039_object148623:head# 2024-04-29T14:21:16.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411e5242:::benchmark_data_smithi194_103039_object98252:head# 2024-04-29T14:21:16.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:411ec9de:::benchmark_data_smithi194_103039_object125379:head# 2024-04-29T14:21:16.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41203f31:::benchmark_data_smithi194_103039_object130752:head# 2024-04-29T14:21:16.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4120eb59:::benchmark_data_smithi194_103039_object142726:head# 2024-04-29T14:21:16.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41217c02:::benchmark_data_smithi194_103039_object57802:head# 2024-04-29T14:21:16.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41220a57:::benchmark_data_smithi194_103039_object66745:head# 2024-04-29T14:21:16.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4123957d:::benchmark_data_smithi194_103039_object108382:head# 2024-04-29T14:21:16.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41240d1d:::benchmark_data_smithi194_103039_object56576:head# 2024-04-29T14:21:16.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41241040:::benchmark_data_smithi194_103039_object93050:head# 2024-04-29T14:21:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412474fa:::benchmark_data_smithi194_103039_object116628:head# 2024-04-29T14:21:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41250d34:::benchmark_data_smithi194_103039_object84711:head# 2024-04-29T14:21:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4125677e:::benchmark_data_smithi194_103039_object78394:head# 2024-04-29T14:21:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4125a006:::benchmark_data_smithi194_103039_object129985:head# 2024-04-29T14:21:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412615f3:::benchmark_data_smithi194_103039_object95345:head# 2024-04-29T14:21:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4126b674:::benchmark_data_smithi194_103039_object122330:head# 2024-04-29T14:21:16.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41273e73:::benchmark_data_smithi194_103039_object47120:head# 2024-04-29T14:21:16.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412876c9:::benchmark_data_smithi194_103039_object72117:head# 2024-04-29T14:21:16.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4128e4db:::benchmark_data_smithi194_103039_object150408:head# 2024-04-29T14:21:16.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41299d59:::benchmark_data_smithi194_103039_object55200:head# 2024-04-29T14:21:16.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412b7a77:::benchmark_data_smithi194_103039_object47615:head# 2024-04-29T14:21:16.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412bfcb4:::benchmark_data_smithi194_103039_object86810:head# 2024-04-29T14:21:16.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412c6c4b:::benchmark_data_smithi194_103039_object147382:head# 2024-04-29T14:21:16.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412d126a:::benchmark_data_smithi194_103039_object88115:head# 2024-04-29T14:21:16.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412d1fb2:::benchmark_data_smithi194_103039_object51297:head# 2024-04-29T14:21:16.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412da889:::benchmark_data_smithi194_103039_object150716:head# 2024-04-29T14:21:16.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:412e913e:::benchmark_data_smithi194_103039_object128539:head# 2024-04-29T14:21:16.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41300c71:::benchmark_data_smithi194_103039_object64603:head# 2024-04-29T14:21:16.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4130e70d:::benchmark_data_smithi194_103039_object58807:head# 2024-04-29T14:21:16.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41316bf9:::benchmark_data_smithi194_103039_object42123:head# 2024-04-29T14:21:16.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4134f618:::benchmark_data_smithi194_103039_object61636:head# 2024-04-29T14:21:16.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4134f79c:::benchmark_data_smithi194_103039_object120101:head# 2024-04-29T14:21:16.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413657c0:::benchmark_data_smithi194_103039_object56159:head# 2024-04-29T14:21:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4136c94c:::benchmark_data_smithi194_103039_object44093:head# 2024-04-29T14:21:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4137162b:::benchmark_data_smithi194_103039_object84883:head# 2024-04-29T14:21:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41376406:::benchmark_data_smithi194_103039_object42117:head# 2024-04-29T14:21:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41379e63:::benchmark_data_smithi194_103039_object112950:head# 2024-04-29T14:21:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4137b098:::benchmark_data_smithi194_103039_object104074:head# 2024-04-29T14:21:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4139923a:::benchmark_data_smithi194_103039_object134731:head# 2024-04-29T14:21:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413a7e1a:::benchmark_data_smithi194_103039_object86919:head# 2024-04-29T14:21:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413acd84:::benchmark_data_smithi194_103039_object57229:head# 2024-04-29T14:21:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413b2637:::benchmark_data_smithi194_103039_object78846:head# 2024-04-29T14:21:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413be000:::benchmark_data_smithi194_103039_object134868:head# 2024-04-29T14:21:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413bfb64:::benchmark_data_smithi194_103039_object149974:head# 2024-04-29T14:21:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413c400a:::benchmark_data_smithi194_103039_object154620:head# 2024-04-29T14:21:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413c7802:::benchmark_data_smithi194_103039_object102219:head# 2024-04-29T14:21:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413cd5d8:::benchmark_data_smithi194_103039_object131679:head# 2024-04-29T14:21:16.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413dbe11:::benchmark_data_smithi194_103039_object56036:head# 2024-04-29T14:21:16.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413dcb5c:::benchmark_data_smithi194_103039_object89106:head# 2024-04-29T14:21:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413eba87:::benchmark_data_smithi194_103039_object122705:head# 2024-04-29T14:21:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:413f68ef:::benchmark_data_smithi194_103039_object93791:head# 2024-04-29T14:21:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4140ccf2:::benchmark_data_smithi194_103039_object107408:head# 2024-04-29T14:21:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41413fb9:::benchmark_data_smithi194_103039_object60196:head# 2024-04-29T14:21:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41417279:::benchmark_data_smithi194_103039_object113997:head# 2024-04-29T14:21:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4141d41b:::benchmark_data_smithi194_103039_object63069:head# 2024-04-29T14:21:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4142b8c7:::benchmark_data_smithi194_103039_object62135:head# 2024-04-29T14:21:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41442514:::benchmark_data_smithi194_103039_object91391:head# 2024-04-29T14:21:16.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4144ebbb:::benchmark_data_smithi194_103039_object121455:head# 2024-04-29T14:21:16.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4144fcff:::benchmark_data_smithi194_103039_object80876:head# 2024-04-29T14:21:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41464b08:::benchmark_data_smithi194_103039_object61739:head# 2024-04-29T14:21:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4146ea02:::benchmark_data_smithi194_103039_object79135:head# 2024-04-29T14:21:16.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4146fbb5:::benchmark_data_smithi194_103039_object130658:head# 2024-04-29T14:21:16.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41475189:::benchmark_data_smithi194_103039_object147874:head# 2024-04-29T14:21:16.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41476d05:::benchmark_data_smithi194_103039_object106853:head# 2024-04-29T14:21:16.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41478e3a:::benchmark_data_smithi194_103039_object53829:head# 2024-04-29T14:21:16.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41480401:::benchmark_data_smithi194_103039_object100398:head# 2024-04-29T14:21:16.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41486ccd:::benchmark_data_smithi194_103039_object94817:head# 2024-04-29T14:21:16.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4148c3e6:::benchmark_data_smithi194_103039_object66669:head# 2024-04-29T14:21:16.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4148f397:::benchmark_data_smithi194_103039_object107920:head# 2024-04-29T14:21:16.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41495134:::benchmark_data_smithi194_103039_object50526:head# 2024-04-29T14:21:16.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414a4006:::benchmark_data_smithi194_103039_object47491:head# 2024-04-29T14:21:16.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414b6294:::benchmark_data_smithi194_103039_object81425:head# 2024-04-29T14:21:16.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414b94ba:::benchmark_data_smithi194_103039_object76855:head# 2024-04-29T14:21:16.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414bde91:::benchmark_data_smithi194_103039_object77057:head# 2024-04-29T14:21:16.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414c2827:::benchmark_data_smithi194_103039_object105839:head# 2024-04-29T14:21:16.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414cf992:::benchmark_data_smithi194_103039_object153355:head# 2024-04-29T14:21:16.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414d0880:::benchmark_data_smithi194_103039_object140733:head# 2024-04-29T14:21:16.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414dc986:::benchmark_data_smithi194_103039_object143514:head# 2024-04-29T14:21:16.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414e263e:::benchmark_data_smithi194_103039_object107344:head# 2024-04-29T14:21:16.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:414fa4e0:::benchmark_data_smithi194_103039_object87141:head# 2024-04-29T14:21:16.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415001e2:::benchmark_data_smithi194_103039_object131365:head# 2024-04-29T14:21:16.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4150106d:::benchmark_data_smithi194_103039_object43509:head# 2024-04-29T14:21:16.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41504ae8:::benchmark_data_smithi194_103039_object99930:head# 2024-04-29T14:21:16.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4150588b:::benchmark_data_smithi194_103039_object139258:head# 2024-04-29T14:21:16.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4150b755:::benchmark_data_smithi194_103039_object51771:head# 2024-04-29T14:21:16.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41512339:::benchmark_data_smithi194_103039_object93454:head# 2024-04-29T14:21:16.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41517da3:::benchmark_data_smithi194_103039_object81624:head# 2024-04-29T14:21:16.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415289f8:::benchmark_data_smithi194_103039_object94003:head# 2024-04-29T14:21:16.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4152cb0b:::benchmark_data_smithi194_103039_object96276:head# 2024-04-29T14:21:16.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41531b18:::benchmark_data_smithi194_103039_object145637:head# 2024-04-29T14:21:16.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415326d3:::benchmark_data_smithi194_103039_object84993:head# 2024-04-29T14:21:16.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41533401:::benchmark_data_smithi194_103039_object142490:head# 2024-04-29T14:21:16.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4154235d:::benchmark_data_smithi194_103039_object147904:head# 2024-04-29T14:21:16.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4154657e:::benchmark_data_smithi194_103039_object73652:head# 2024-04-29T14:21:16.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4154671b:::benchmark_data_smithi194_103039_object83901:head# 2024-04-29T14:21:16.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4154f32a:::benchmark_data_smithi194_103039_object87989:head# 2024-04-29T14:21:16.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4155c2f8:::benchmark_data_smithi194_103039_object117585:head# 2024-04-29T14:21:16.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4155fa1c:::benchmark_data_smithi194_103039_object136518:head# 2024-04-29T14:21:16.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4156a0c6:::benchmark_data_smithi194_103039_object46796:head# 2024-04-29T14:21:16.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41577215:::benchmark_data_smithi194_103039_object45849:head# 2024-04-29T14:21:16.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4158eb34:::benchmark_data_smithi194_103039_object107678:head# 2024-04-29T14:21:16.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415ac80d:::benchmark_data_smithi194_103039_object51719:head# 2024-04-29T14:21:16.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415c473c:::benchmark_data_smithi194_103039_object46329:head# 2024-04-29T14:21:16.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415cb2aa:::benchmark_data_smithi194_103039_object114619:head# 2024-04-29T14:21:16.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415f7b91:::benchmark_data_smithi194_103039_object111316:head# 2024-04-29T14:21:16.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:415f8283:::benchmark_data_smithi194_103039_object109651:head# 2024-04-29T14:21:16.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4161f9e8:::benchmark_data_smithi194_103039_object73186:head# 2024-04-29T14:21:16.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41626bb8:::benchmark_data_smithi194_103039_object149233:head# 2024-04-29T14:21:16.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4162d1a2:::benchmark_data_smithi194_103039_object54116:head# 2024-04-29T14:21:16.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4163706b:::benchmark_data_smithi194_103039_object66775:head# 2024-04-29T14:21:16.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4164722b:::benchmark_data_smithi194_103039_object60055:head# 2024-04-29T14:21:16.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4164d949:::benchmark_data_smithi194_103039_object114840:head# 2024-04-29T14:21:16.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41659134:::benchmark_data_smithi194_103039_object51580:head# 2024-04-29T14:21:16.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4165ca4b:::benchmark_data_smithi194_103039_object149489:head# 2024-04-29T14:21:16.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4166a914:::benchmark_data_smithi194_103039_object80907:head# 2024-04-29T14:21:16.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4166db43:::benchmark_data_smithi194_103039_object55970:head# 2024-04-29T14:21:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41670d1e:::benchmark_data_smithi194_103039_object116089:head# 2024-04-29T14:21:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41676288:::benchmark_data_smithi194_103039_object56854:head# 2024-04-29T14:21:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4168958a:::benchmark_data_smithi194_103039_object78448:head# 2024-04-29T14:21:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4169533b:::benchmark_data_smithi194_103039_object78065:head# 2024-04-29T14:21:16.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41697b17:::benchmark_data_smithi194_103039_object95155:head# 2024-04-29T14:21:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416afbb6:::benchmark_data_smithi194_103039_object92389:head# 2024-04-29T14:21:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416b13cf:::benchmark_data_smithi194_103039_object104650:head# 2024-04-29T14:21:16.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416b71e1:::benchmark_data_smithi194_103039_object57403:head# 2024-04-29T14:21:16.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416b9741:::benchmark_data_smithi194_103039_object155035:head# 2024-04-29T14:21:16.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416d1f11:::benchmark_data_smithi194_103039_object55529:head# 2024-04-29T14:21:16.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416decbd:::benchmark_data_smithi194_103039_object42612:head# 2024-04-29T14:21:16.750 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:16.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416df529:::benchmark_data_smithi194_103039_object52146:head# 2024-04-29T14:21:16.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:416efd80:::benchmark_data_smithi194_103039_object119670:head# 2024-04-29T14:21:16.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4170282f:::benchmark_data_smithi194_103039_object55959:head# 2024-04-29T14:21:16.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4170f4e6:::benchmark_data_smithi194_103039_object156124:head# 2024-04-29T14:21:16.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417230df:::benchmark_data_smithi194_103039_object84705:head# 2024-04-29T14:21:16.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417260e8:::benchmark_data_smithi194_103039_object87174:head# 2024-04-29T14:21:16.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41726179:::benchmark_data_smithi194_103039_object57651:head# 2024-04-29T14:21:16.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4173401e:::benchmark_data_smithi194_103039_object148003:head# 2024-04-29T14:21:16.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4173e184:::benchmark_data_smithi194_103039_object82349:head# 2024-04-29T14:21:16.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4174e381:::benchmark_data_smithi194_103039_object116215:head# 2024-04-29T14:21:16.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41752db2:::benchmark_data_smithi194_103039_object151180:head# 2024-04-29T14:21:16.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4175ddbd:::benchmark_data_smithi194_103039_object146302:head# 2024-04-29T14:21:16.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41769007:::benchmark_data_smithi194_103039_object138583:head# 2024-04-29T14:21:16.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4177244a:::benchmark_data_smithi194_103039_object97652:head# 2024-04-29T14:21:16.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4177c35c:::benchmark_data_smithi194_103039_object50300:head# 2024-04-29T14:21:16.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41782960:::benchmark_data_smithi194_103039_object151386:head# 2024-04-29T14:21:16.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41784937:::benchmark_data_smithi194_103039_object97825:head# 2024-04-29T14:21:16.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4178da65:::benchmark_data_smithi194_103039_object82355:head# 2024-04-29T14:21:16.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4178f36a:::benchmark_data_smithi194_103039_object136912:head# 2024-04-29T14:21:16.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417926ec:::benchmark_data_smithi194_103039_object144790:head# 2024-04-29T14:21:16.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417994bc:::benchmark_data_smithi194_103039_object86357:head# 2024-04-29T14:21:16.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4179af4d:::benchmark_data_smithi194_103039_object115863:head# 2024-04-29T14:21:16.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417b6192:::benchmark_data_smithi194_103039_object145565:head# 2024-04-29T14:21:16.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417d4188:::benchmark_data_smithi194_103039_object117578:head# 2024-04-29T14:21:16.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417dd3a7:::benchmark_data_smithi194_103039_object125190:head# 2024-04-29T14:21:16.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417e588d:::benchmark_data_smithi194_103039_object90513:head# 2024-04-29T14:21:16.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417eca51:::benchmark_data_smithi194_103039_object125270:head# 2024-04-29T14:21:16.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417efb53:::benchmark_data_smithi194_103039_object141541:head# 2024-04-29T14:21:16.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:417ffa02:::benchmark_data_smithi194_103039_object47820:head# 2024-04-29T14:21:16.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41801063:::benchmark_data_smithi194_103039_object156565:head# 2024-04-29T14:21:16.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418058e2:::benchmark_data_smithi194_103039_object128498:head# 2024-04-29T14:21:16.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41808cc6:::benchmark_data_smithi194_103039_object74866:head# 2024-04-29T14:21:16.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4180a3d6:::benchmark_data_smithi194_103039_object86303:head# 2024-04-29T14:21:16.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41810f61:::benchmark_data_smithi194_103039_object103115:head# 2024-04-29T14:21:16.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4181533a:::benchmark_data_smithi194_103039_object86045:head# 2024-04-29T14:21:16.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41817192:::benchmark_data_smithi194_103039_object106860:head# 2024-04-29T14:21:16.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4181d6c4:::benchmark_data_smithi194_103039_object127980:head# 2024-04-29T14:21:16.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41821275:::benchmark_data_smithi194_103039_object77876:head# 2024-04-29T14:21:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41826dd4:::benchmark_data_smithi194_103039_object66159:head# 2024-04-29T14:21:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4182cfe9:::benchmark_data_smithi194_103039_object107998:head# 2024-04-29T14:21:16.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4183d066:::benchmark_data_smithi194_103039_object91069:head# 2024-04-29T14:21:16.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418431be:::benchmark_data_smithi194_103039_object42741:head# 2024-04-29T14:21:16.770 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41843b28:::benchmark_data_smithi194_103039_object120051:head# 2024-04-29T14:21:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41844844:::benchmark_data_smithi194_103039_object132329:head# 2024-04-29T14:21:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4184fb3c:::benchmark_data_smithi194_103039_object110419:head# 2024-04-29T14:21:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41852a56:::benchmark_data_smithi194_103039_object48215:head# 2024-04-29T14:21:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4185f6f4:::benchmark_data_smithi194_103039_object67157:head# 2024-04-29T14:21:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41879204:::benchmark_data_smithi194_103039_object76448:head# 2024-04-29T14:21:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41879d3e:::benchmark_data_smithi194_103039_object60865:head# 2024-04-29T14:21:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41896285:::benchmark_data_smithi194_103039_object68644:head# 2024-04-29T14:21:16.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4189abc2:::benchmark_data_smithi194_103039_object138835:head# 2024-04-29T14:21:16.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418a88ea:::benchmark_data_smithi194_103039_object135069:head# 2024-04-29T14:21:16.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418bca30:::benchmark_data_smithi194_103039_object79355:head# 2024-04-29T14:21:16.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418bd543:::benchmark_data_smithi194_103039_object99670:head# 2024-04-29T14:21:16.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418e8c5d:::benchmark_data_smithi194_103039_object52187:head# 2024-04-29T14:21:16.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418f0bff:::benchmark_data_smithi194_103039_object124315:head# 2024-04-29T14:21:16.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418f16db:::benchmark_data_smithi194_103039_object96959:head# 2024-04-29T14:21:16.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:418f4d77:::benchmark_data_smithi194_103039_object122058:head# 2024-04-29T14:21:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4190970e:::benchmark_data_smithi194_103039_object55988:head# 2024-04-29T14:21:16.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4191576c:::benchmark_data_smithi194_103039_object96964:head# 2024-04-29T14:21:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419211aa:::benchmark_data_smithi194_103039_object74247:head# 2024-04-29T14:21:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4192e60f:::benchmark_data_smithi194_103039_object86737:head# 2024-04-29T14:21:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4192f981:::benchmark_data_smithi194_103039_object116451:head# 2024-04-29T14:21:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41930a60:::benchmark_data_smithi194_103039_object55286:head# 2024-04-29T14:21:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41932fda:::benchmark_data_smithi194_103039_object45205:head# 2024-04-29T14:21:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419498e1:::benchmark_data_smithi194_103039_object85484:head# 2024-04-29T14:21:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41958d10:::benchmark_data_smithi194_103039_object43515:head# 2024-04-29T14:21:16.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4195fd1c:::benchmark_data_smithi194_103039_object82927:head# 2024-04-29T14:21:16.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4196fc60:::benchmark_data_smithi194_103039_object82553:head# 2024-04-29T14:21:16.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41988f99:::benchmark_data_smithi194_103039_object140639:head# 2024-04-29T14:21:16.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4198d37b:::benchmark_data_smithi194_103039_object107521:head# 2024-04-29T14:21:16.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419aa186:::benchmark_data_smithi194_103039_object140420:head# 2024-04-29T14:21:16.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419b39bb:::benchmark_data_smithi194_103039_object132583:head# 2024-04-29T14:21:16.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419b58e0:::benchmark_data_smithi194_103039_object48252:head# 2024-04-29T14:21:16.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419bde49:::benchmark_data_smithi194_103039_object152309:head# 2024-04-29T14:21:16.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419cf471:::benchmark_data_smithi194_103039_object42760:head# 2024-04-29T14:21:16.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419e17f7:::benchmark_data_smithi194_103039_object62137:head# 2024-04-29T14:21:16.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419e68eb:::benchmark_data_smithi194_103039_object127287:head# 2024-04-29T14:21:16.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419f23f6:::benchmark_data_smithi194_103039_object119961:head# 2024-04-29T14:21:16.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419f6a2a:::benchmark_data_smithi194_103039_object121004:head# 2024-04-29T14:21:16.789 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:16.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419f96d1:::benchmark_data_smithi194_103039_object68421:head# 2024-04-29T14:21:16.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:419fd194:::benchmark_data_smithi194_103039_object104033:head# 2024-04-29T14:21:16.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a14b6c:::benchmark_data_smithi194_103039_object50233:head# 2024-04-29T14:21:16.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a1b769:::benchmark_data_smithi194_103039_object74015:head# 2024-04-29T14:21:16.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a1bfef:::benchmark_data_smithi194_103039_object50712:head# 2024-04-29T14:21:16.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a255c3:::benchmark_data_smithi194_103039_object94457:head# 2024-04-29T14:21:16.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a2784f:::benchmark_data_smithi194_103039_object110666:head# 2024-04-29T14:21:16.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a2900b:::benchmark_data_smithi194_103039_object149781:head# 2024-04-29T14:21:16.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a2c861:::benchmark_data_smithi194_103039_object97264:head# 2024-04-29T14:21:16.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a32f1c:::benchmark_data_smithi194_103039_object109761:head# 2024-04-29T14:21:16.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a3b27c:::benchmark_data_smithi194_103039_object93180:head# 2024-04-29T14:21:16.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a3f055:::benchmark_data_smithi194_103039_object143154:head# 2024-04-29T14:21:16.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a40df2:::benchmark_data_smithi194_103039_object145242:head# 2024-04-29T14:21:16.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a4cc06:::benchmark_data_smithi194_103039_object147526:head# 2024-04-29T14:21:16.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a4ced4:::benchmark_data_smithi194_103039_object61270:head# 2024-04-29T14:21:16.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a4fb39:::benchmark_data_smithi194_103039_object125127:head# 2024-04-29T14:21:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a5d446:::benchmark_data_smithi194_103039_object138048:head# 2024-04-29T14:21:16.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a6325c:::benchmark_data_smithi194_103039_object124101:head# 2024-04-29T14:21:16.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a6624a:::benchmark_data_smithi194_103039_object49347:head# 2024-04-29T14:21:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a6afc0:::benchmark_data_smithi194_103039_object92841:head# 2024-04-29T14:21:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a71082:::benchmark_data_smithi194_103039_object69861:head# 2024-04-29T14:21:16.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a7eb2d:::benchmark_data_smithi194_103039_object100493:head# 2024-04-29T14:21:16.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a83b11:::benchmark_data_smithi194_103039_object142668:head# 2024-04-29T14:21:16.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a86529:::benchmark_data_smithi194_103039_object57714:head# 2024-04-29T14:21:16.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a87fbf:::benchmark_data_smithi194_103039_object42081:head# 2024-04-29T14:21:16.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41a9b1e7:::benchmark_data_smithi194_103039_object118088:head# 2024-04-29T14:21:16.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41aa2778:::benchmark_data_smithi194_103039_object129403:head# 2024-04-29T14:21:16.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41aa5bbe:::benchmark_data_smithi194_103039_object52504:head# 2024-04-29T14:21:16.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41aad390:::benchmark_data_smithi194_103039_object52795:head# 2024-04-29T14:21:16.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ab5278:::benchmark_data_smithi194_103039_object75999:head# 2024-04-29T14:21:16.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ab5f2a:::benchmark_data_smithi194_103039_object121717:head# 2024-04-29T14:21:16.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ac1034:::benchmark_data_smithi194_103039_object149463:head# 2024-04-29T14:21:16.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ac1b43:::benchmark_data_smithi194_103039_object91032:head# 2024-04-29T14:21:16.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ad393f:::benchmark_data_smithi194_103039_object55166:head# 2024-04-29T14:21:16.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41adc19a:::benchmark_data_smithi194_103039_object143728:head# 2024-04-29T14:21:16.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41add1ff:::benchmark_data_smithi194_103039_object55045:head# 2024-04-29T14:21:16.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41af2214:::benchmark_data_smithi194_103039_object68783:head# 2024-04-29T14:21:16.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41af8faf:::benchmark_data_smithi194_103039_object83565:head# 2024-04-29T14:21:16.808 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:16.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41afc8da:::benchmark_data_smithi194_103039_object134088:head# 2024-04-29T14:21:16.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41afe89b:::benchmark_data_smithi194_103039_object111572:head# 2024-04-29T14:21:16.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b02d6b:::benchmark_data_smithi194_103039_object45396:head# 2024-04-29T14:21:16.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b0efce:::benchmark_data_smithi194_103039_object146425:head# 2024-04-29T14:21:16.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b1217e:::benchmark_data_smithi194_103039_object74477:head# 2024-04-29T14:21:16.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b16ee0:::benchmark_data_smithi194_103039_object154804:head# 2024-04-29T14:21:16.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b1e599:::benchmark_data_smithi194_103039_object77152:head# 2024-04-29T14:21:16.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b2e7fa:::benchmark_data_smithi194_103039_object46847:head# 2024-04-29T14:21:16.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b3d0ac:::benchmark_data_smithi194_103039_object121013:head# 2024-04-29T14:21:16.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b42e97:::benchmark_data_smithi194_103039_object43914:head# 2024-04-29T14:21:16.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b45fa8:::benchmark_data_smithi194_103039_object127449:head# 2024-04-29T14:21:16.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b5408d:::benchmark_data_smithi194_103039_object95658:head# 2024-04-29T14:21:16.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b5cfaa:::benchmark_data_smithi194_103039_object121459:head# 2024-04-29T14:21:16.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b6dcd0:::benchmark_data_smithi194_103039_object70644:head# 2024-04-29T14:21:16.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b6e3df:::benchmark_data_smithi194_103039_object150412:head# 2024-04-29T14:21:16.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b7f35d:::benchmark_data_smithi194_103039_object143502:head# 2024-04-29T14:21:16.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b85db7:::benchmark_data_smithi194_103039_object106061:head# 2024-04-29T14:21:16.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b8921a:::benchmark_data_smithi194_103039_object124290:head# 2024-04-29T14:21:16.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b903c8:::benchmark_data_smithi194_103039_object80257:head# 2024-04-29T14:21:16.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41b9a538:::benchmark_data_smithi194_103039_object102158:head# 2024-04-29T14:21:16.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41bba957:::benchmark_data_smithi194_103039_object63531:head# 2024-04-29T14:21:16.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41bbd064:::benchmark_data_smithi194_103039_object89124:head# 2024-04-29T14:21:16.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41bc1cde:::benchmark_data_smithi194_103039_object148274:head# 2024-04-29T14:21:16.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41bd3f18:::benchmark_data_smithi194_103039_object147466:head# 2024-04-29T14:21:16.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41bd6ef3:::benchmark_data_smithi194_103039_object110827:head# 2024-04-29T14:21:16.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41bf5643:::benchmark_data_smithi194_103039_object141976:head# 2024-04-29T14:21:16.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c0287a:::benchmark_data_smithi194_103039_object69178:head# 2024-04-29T14:21:16.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c0bd3f:::benchmark_data_smithi194_103039_object63313:head# 2024-04-29T14:21:16.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c13a49:::benchmark_data_smithi194_103039_object147818:head# 2024-04-29T14:21:16.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c1b46e:::benchmark_data_smithi194_103039_object58719:head# 2024-04-29T14:21:16.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c215d1:::benchmark_data_smithi194_103039_object126722:head# 2024-04-29T14:21:16.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c24501:::benchmark_data_smithi194_103039_object114651:head# 2024-04-29T14:21:16.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c2b74e:::benchmark_data_smithi194_103039_object95754:head# 2024-04-29T14:21:16.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c3485a:::benchmark_data_smithi194_103039_object98393:head# 2024-04-29T14:21:16.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c39f80:::benchmark_data_smithi194_103039_object105263:head# 2024-04-29T14:21:16.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c3a493:::benchmark_data_smithi194_103039_object146503:head# 2024-04-29T14:21:16.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c433a5:::benchmark_data_smithi194_103039_object44142:head# 2024-04-29T14:21:16.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c441db:::benchmark_data_smithi194_103039_object117307:head# 2024-04-29T14:21:16.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c47503:::benchmark_data_smithi194_103039_object105934:head# 2024-04-29T14:21:16.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c5ba4e:::benchmark_data_smithi194_103039_object133419:head# 2024-04-29T14:21:16.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c708d7:::benchmark_data_smithi194_103039_object143690:head# 2024-04-29T14:21:16.828 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:16.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c7ab78:::benchmark_data_smithi194_103039_object112000:head# 2024-04-29T14:21:16.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c7ca89:::benchmark_data_smithi194_103039_object149922:head# 2024-04-29T14:21:16.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c8069e:::benchmark_data_smithi194_103039_object47888:head# 2024-04-29T14:21:16.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c8b8c4:::benchmark_data_smithi194_103039_object106311:head# 2024-04-29T14:21:16.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41c95d46:::benchmark_data_smithi194_103039_object78335:head# 2024-04-29T14:21:16.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41cd3e16:::benchmark_data_smithi194_103039_object62578:head# 2024-04-29T14:21:16.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41cd421b:::benchmark_data_smithi194_103039_object90311:head# 2024-04-29T14:21:16.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41cd5cdb:::benchmark_data_smithi194_103039_object63946:head# 2024-04-29T14:21:16.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ce1214:::benchmark_data_smithi194_103039_object98485:head# 2024-04-29T14:21:16.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41cf7534:::benchmark_data_smithi194_103039_object130718:head# 2024-04-29T14:21:16.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d06fbd:::benchmark_data_smithi194_103039_object80536:head# 2024-04-29T14:21:16.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d36d17:::benchmark_data_smithi194_103039_object43421:head# 2024-04-29T14:21:16.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d3a018:::benchmark_data_smithi194_103039_object101170:head# 2024-04-29T14:21:16.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d3d9ec:::benchmark_data_smithi194_103039_object112301:head# 2024-04-29T14:21:16.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d5039b:::benchmark_data_smithi194_103039_object153309:head# 2024-04-29T14:21:16.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d61cef:::benchmark_data_smithi194_103039_object44366:head# 2024-04-29T14:21:16.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d63763:::benchmark_data_smithi194_103039_object94612:head# 2024-04-29T14:21:16.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d6972f:::benchmark_data_smithi194_103039_object118744:head# 2024-04-29T14:21:16.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d69f8f:::benchmark_data_smithi194_103039_object110078:head# 2024-04-29T14:21:16.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d76357:::benchmark_data_smithi194_103039_object111309:head# 2024-04-29T14:21:16.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d77df4:::benchmark_data_smithi194_103039_object147810:head# 2024-04-29T14:21:16.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d79d88:::benchmark_data_smithi194_103039_object95157:head# 2024-04-29T14:21:16.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d868a5:::benchmark_data_smithi194_103039_object109892:head# 2024-04-29T14:21:16.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d8e005:::benchmark_data_smithi194_103039_object156342:head# 2024-04-29T14:21:16.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41d9e42d:::benchmark_data_smithi194_103039_object76759:head# 2024-04-29T14:21:16.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41dbae64:::benchmark_data_smithi194_103039_object131635:head# 2024-04-29T14:21:16.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41dd58eb:::benchmark_data_smithi194_103039_object49087:head# 2024-04-29T14:21:16.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ddb02d:::benchmark_data_smithi194_103039_object145968:head# 2024-04-29T14:21:16.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ddd1dc:::benchmark_data_smithi194_103039_object139710:head# 2024-04-29T14:21:16.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41df7819:::benchmark_data_smithi194_103039_object72130:head# 2024-04-29T14:21:16.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41df85aa:::benchmark_data_smithi194_103039_object53741:head# 2024-04-29T14:21:16.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41dfc16a:::benchmark_data_smithi194_103039_object128187:head# 2024-04-29T14:21:16.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e00196:::benchmark_data_smithi194_103039_object156006:head# 2024-04-29T14:21:16.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e05d5b:::benchmark_data_smithi194_103039_object151679:head# 2024-04-29T14:21:16.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e2012e:::benchmark_data_smithi194_103039_object129344:head# 2024-04-29T14:21:16.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e209d6:::benchmark_data_smithi194_103039_object133588:head# 2024-04-29T14:21:16.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e23f55:::benchmark_data_smithi194_103039_object91359:head# 2024-04-29T14:21:16.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e23ffb:::benchmark_data_smithi194_103039_object83730:head# 2024-04-29T14:21:16.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e27680:::benchmark_data_smithi194_103039_object124379:head# 2024-04-29T14:21:16.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e2c118:::benchmark_data_smithi194_103039_object139038:head# 2024-04-29T14:21:16.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e3a199:::benchmark_data_smithi194_103039_object145060:head# 2024-04-29T14:21:16.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e3bd11:::benchmark_data_smithi194_103039_object126776:head# 2024-04-29T14:21:16.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e43037:::benchmark_data_smithi194_103039_object50012:head# 2024-04-29T14:21:16.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e4528e:::benchmark_data_smithi194_103039_object46935:head# 2024-04-29T14:21:16.848 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:16.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e4c30d:::benchmark_data_smithi194_103039_object107716:head# 2024-04-29T14:21:16.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e5e57c:::benchmark_data_smithi194_103039_object77177:head# 2024-04-29T14:21:16.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e61b46:::benchmark_data_smithi194_103039_object65016:head# 2024-04-29T14:21:16.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e6c1d1:::benchmark_data_smithi194_103039_object96863:head# 2024-04-29T14:21:16.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e6e4a4:::benchmark_data_smithi194_103039_object107418:head# 2024-04-29T14:21:16.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e71600:::benchmark_data_smithi194_103039_object83385:head# 2024-04-29T14:21:16.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e90278:::benchmark_data_smithi194_103039_object96207:head# 2024-04-29T14:21:16.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e9150e:::benchmark_data_smithi194_103039_object63318:head# 2024-04-29T14:21:16.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41e98b57:::benchmark_data_smithi194_103039_object115157:head# 2024-04-29T14:21:16.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ea33ad:::benchmark_data_smithi194_103039_object143446:head# 2024-04-29T14:21:16.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ea5521:::benchmark_data_smithi194_103039_object128924:head# 2024-04-29T14:21:16.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ec368c:::benchmark_data_smithi194_103039_object78386:head# 2024-04-29T14:21:16.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ed0028:::benchmark_data_smithi194_103039_object47591:head# 2024-04-29T14:21:16.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ed64ef:::benchmark_data_smithi194_103039_object129024:head# 2024-04-29T14:21:16.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41edca5a:::benchmark_data_smithi194_103039_object79150:head# 2024-04-29T14:21:16.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41eeb3f2:::benchmark_data_smithi194_103039_object145305:head# 2024-04-29T14:21:16.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41efe5ab:::benchmark_data_smithi194_103039_object73972:head# 2024-04-29T14:21:16.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41efff49:::benchmark_data_smithi194_103039_object145533:head# 2024-04-29T14:21:16.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f0077e:::benchmark_data_smithi194_103039_object67569:head# 2024-04-29T14:21:16.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f030c0:::benchmark_data_smithi194_103039_object89417:head# 2024-04-29T14:21:16.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f049c6:::benchmark_data_smithi194_103039_object105097:head# 2024-04-29T14:21:16.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f082a4:::benchmark_data_smithi194_103039_object98689:head# 2024-04-29T14:21:16.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f0f650:::benchmark_data_smithi194_103039_object133151:head# 2024-04-29T14:21:16.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f13bcc:::benchmark_data_smithi194_103039_object127751:head# 2024-04-29T14:21:16.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f25777:::benchmark_data_smithi194_103039_object153104:head# 2024-04-29T14:21:16.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f35264:::benchmark_data_smithi194_103039_object109850:head# 2024-04-29T14:21:16.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f4b361:::benchmark_data_smithi194_103039_object81469:head# 2024-04-29T14:21:16.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f5f507:::benchmark_data_smithi194_103039_object101062:head# 2024-04-29T14:21:16.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f63fe0:::benchmark_data_smithi194_103039_object117727:head# 2024-04-29T14:21:16.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f6414a:::benchmark_data_smithi194_103039_object139681:head# 2024-04-29T14:21:16.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f6c37a:::benchmark_data_smithi194_103039_object109348:head# 2024-04-29T14:21:16.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f6cebe:::benchmark_data_smithi194_103039_object78128:head# 2024-04-29T14:21:16.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f772a7:::benchmark_data_smithi194_103039_object151982:head# 2024-04-29T14:21:16.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f7d16d:::benchmark_data_smithi194_103039_object115179:head# 2024-04-29T14:21:16.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f87f5a:::benchmark_data_smithi194_103039_object83153:head# 2024-04-29T14:21:16.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f8dac2:::benchmark_data_smithi194_103039_object71019:head# 2024-04-29T14:21:16.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41f9fd5c:::benchmark_data_smithi194_103039_object118247:head# 2024-04-29T14:21:16.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fa246d:::benchmark_data_smithi194_103039_object41964:head# 2024-04-29T14:21:16.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fb3c58:::benchmark_data_smithi194_103039_object97679:head# 2024-04-29T14:21:16.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fbfb0e:::benchmark_data_smithi194_103039_object65916:head# 2024-04-29T14:21:16.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fcacde:::benchmark_data_smithi194_103039_object103910:head# 2024-04-29T14:21:16.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fcb5c7:::benchmark_data_smithi194_103039_object76858:head# 2024-04-29T14:21:16.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fd45ce:::benchmark_data_smithi194_103039_object123748:head# 2024-04-29T14:21:16.867 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:16.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fd4622:::benchmark_data_smithi194_103039_object69358:head# 2024-04-29T14:21:16.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fdddb5:::benchmark_data_smithi194_103039_object59526:head# 2024-04-29T14:21:16.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41fe3cc1:::benchmark_data_smithi194_103039_object128300:head# 2024-04-29T14:21:16.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41feab62:::benchmark_data_smithi194_103039_object56525:head# 2024-04-29T14:21:16.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:41ffefce:::benchmark_data_smithi194_103039_object57641:head# 2024-04-29T14:21:16.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4201a98a:::benchmark_data_smithi194_103039_object114207:head# 2024-04-29T14:21:16.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4201ee43:::benchmark_data_smithi194_103039_object147201:head# 2024-04-29T14:21:16.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42027ef9:::benchmark_data_smithi194_103039_object88369:head# 2024-04-29T14:21:16.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420390df:::benchmark_data_smithi194_103039_object106066:head# 2024-04-29T14:21:16.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4203bd17:::benchmark_data_smithi194_103039_object73732:head# 2024-04-29T14:21:16.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4203c8bd:::benchmark_data_smithi194_103039_object135483:head# 2024-04-29T14:21:16.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4204423d:::benchmark_data_smithi194_103039_object102067:head# 2024-04-29T14:21:16.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42052501:::benchmark_data_smithi194_103039_object153132:head# 2024-04-29T14:21:16.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420532c4:::benchmark_data_smithi194_103039_object81583:head# 2024-04-29T14:21:16.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4205a61d:::benchmark_data_smithi194_103039_object55934:head# 2024-04-29T14:21:16.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42066561:::benchmark_data_smithi194_103039_object67417:head# 2024-04-29T14:21:16.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42086da7:::benchmark_data_smithi194_103039_object127074:head# 2024-04-29T14:21:16.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4208756a:::benchmark_data_smithi194_103039_object97978:head# 2024-04-29T14:21:16.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42096c92:::benchmark_data_smithi194_103039_object142784:head# 2024-04-29T14:21:16.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4209741b:::benchmark_data_smithi194_103039_object113910:head# 2024-04-29T14:21:16.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4209bafd:::benchmark_data_smithi194_103039_object115701:head# 2024-04-29T14:21:16.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4209bb74:::benchmark_data_smithi194_103039_object102841:head# 2024-04-29T14:21:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420a2b39:::benchmark_data_smithi194_103039_object61600:head# 2024-04-29T14:21:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420af68c:::benchmark_data_smithi194_103039_object145129:head# 2024-04-29T14:21:16.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420b3c8a:::benchmark_data_smithi194_103039_object146781:head# 2024-04-29T14:21:16.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420c93d8:::benchmark_data_smithi194_103039_object117471:head# 2024-04-29T14:21:16.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420ca3eb:::benchmark_data_smithi194_103039_object74875:head# 2024-04-29T14:21:16.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420cb275:::benchmark_data_smithi194_103039_object152804:head# 2024-04-29T14:21:16.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420dc29d:::benchmark_data_smithi194_103039_object51605:head# 2024-04-29T14:21:16.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420e79d3:::benchmark_data_smithi194_103039_object83706:head# 2024-04-29T14:21:16.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420eb593:::benchmark_data_smithi194_103039_object83220:head# 2024-04-29T14:21:16.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420f1bdc:::benchmark_data_smithi194_103039_object91670:head# 2024-04-29T14:21:16.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420f7814:::benchmark_data_smithi194_103039_object53284:head# 2024-04-29T14:21:16.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420f89b4:::benchmark_data_smithi194_103039_object65232:head# 2024-04-29T14:21:16.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:420feb0e:::benchmark_data_smithi194_103039_object152863:head# 2024-04-29T14:21:16.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42105ca4:::benchmark_data_smithi194_103039_object102590:head# 2024-04-29T14:21:16.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4210c4cb:::benchmark_data_smithi194_103039_object47594:head# 2024-04-29T14:21:16.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42112568:::benchmark_data_smithi194_103039_object152502:head# 2024-04-29T14:21:16.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42115d1c:::benchmark_data_smithi194_103039_object87551:head# 2024-04-29T14:21:16.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42135626:::benchmark_data_smithi194_103039_object74019:head# 2024-04-29T14:21:16.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4213aea3:::benchmark_data_smithi194_103039_object80502:head# 2024-04-29T14:21:16.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42144032:::benchmark_data_smithi194_103039_object63555:head# 2024-04-29T14:21:16.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421441fb:::benchmark_data_smithi194_103039_object92925:head# 2024-04-29T14:21:16.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4214f66b:::benchmark_data_smithi194_103039_object152725:head# 2024-04-29T14:21:16.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42155353:::benchmark_data_smithi194_103039_object142980:head# 2024-04-29T14:21:16.889 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:16.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4215c042:::benchmark_data_smithi194_103039_object149748:head# 2024-04-29T14:21:16.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4215fe52:::benchmark_data_smithi194_103039_object144793:head# 2024-04-29T14:21:16.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4216b3df:::benchmark_data_smithi194_103039_object127951:head# 2024-04-29T14:21:16.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4217a52c:::benchmark_data_smithi194_103039_object66172:head# 2024-04-29T14:21:16.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4218edc0:::benchmark_data_smithi194_103039_object110902:head# 2024-04-29T14:21:16.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4218f9fd:::benchmark_data_smithi194_103039_object128258:head# 2024-04-29T14:21:16.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4218fde2:::benchmark_data_smithi194_103039_object91993:head# 2024-04-29T14:21:16.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42198675:::benchmark_data_smithi194_103039_object66397:head# 2024-04-29T14:21:16.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4219e163:::benchmark_data_smithi194_103039_object55268:head# 2024-04-29T14:21:16.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4219e797:::benchmark_data_smithi194_103039_object128272:head# 2024-04-29T14:21:16.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421a28bf:::benchmark_data_smithi194_103039_object146107:head# 2024-04-29T14:21:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421abdd4:::benchmark_data_smithi194_103039_object58007:head# 2024-04-29T14:21:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421bb3ef:::benchmark_data_smithi194_103039_object87856:head# 2024-04-29T14:21:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421bd3b0:::benchmark_data_smithi194_103039_object107576:head# 2024-04-29T14:21:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421ce2fa:::benchmark_data_smithi194_103039_object53606:head# 2024-04-29T14:21:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421d0890:::benchmark_data_smithi194_103039_object131144:head# 2024-04-29T14:21:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421d70f8:::benchmark_data_smithi194_103039_object129803:head# 2024-04-29T14:21:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421d9885:::benchmark_data_smithi194_103039_object152533:head# 2024-04-29T14:21:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421e26a9:::benchmark_data_smithi194_103039_object49905:head# 2024-04-29T14:21:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421e40d5:::benchmark_data_smithi194_103039_object135894:head# 2024-04-29T14:21:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421f3827:::benchmark_data_smithi194_103039_object67700:head# 2024-04-29T14:21:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421fc446:::benchmark_data_smithi194_103039_object137577:head# 2024-04-29T14:21:16.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:421fe747:::benchmark_data_smithi194_103039_object148295:head# 2024-04-29T14:21:16.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4221db73:::benchmark_data_smithi194_103039_object60045:head# 2024-04-29T14:21:16.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4221f448:::benchmark_data_smithi194_103039_object55392:head# 2024-04-29T14:21:16.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42222476:::benchmark_data_smithi194_103039_object58036:head# 2024-04-29T14:21:16.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4223ccf4:::benchmark_data_smithi194_103039_object93948:head# 2024-04-29T14:21:16.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4224ef4c:::benchmark_data_smithi194_103039_object121112:head# 2024-04-29T14:21:16.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4225fa77:::benchmark_data_smithi194_103039_object47005:head# 2024-04-29T14:21:16.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4226e19a:::benchmark_data_smithi194_103039_object75936:head# 2024-04-29T14:21:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42278e58:::benchmark_data_smithi194_103039_object51544:head# 2024-04-29T14:21:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:422b61ee:::benchmark_data_smithi194_103039_object57900:head# 2024-04-29T14:21:16.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:422b6f73:::benchmark_data_smithi194_103039_object98250:head# 2024-04-29T14:21:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:422cab14:::benchmark_data_smithi194_103039_object139031:head# 2024-04-29T14:21:16.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:422d6789:::benchmark_data_smithi194_103039_object149742:head# 2024-04-29T14:21:16.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:422d9844:::benchmark_data_smithi194_103039_object135904:head# 2024-04-29T14:21:16.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:422e27d6:::benchmark_data_smithi194_103039_object77756:head# 2024-04-29T14:21:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:422efd82:::benchmark_data_smithi194_103039_object78591:head# 2024-04-29T14:21:16.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423089d8:::benchmark_data_smithi194_103039_object153694:head# 2024-04-29T14:21:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4230bea6:::benchmark_data_smithi194_103039_object42013:head# 2024-04-29T14:21:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42318649:::benchmark_data_smithi194_103039_object125470:head# 2024-04-29T14:21:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423254fb:::benchmark_data_smithi194_103039_object46259:head# 2024-04-29T14:21:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423258d9:::benchmark_data_smithi194_103039_object95051:head# 2024-04-29T14:21:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4232fb4d:::benchmark_data_smithi194_103039_object148605:head# 2024-04-29T14:21:16.909 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:16.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423343e7:::benchmark_data_smithi194_103039_object112432:head# 2024-04-29T14:21:16.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42335d85:::benchmark_data_smithi194_103039_object50888:head# 2024-04-29T14:21:16.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42364669:::benchmark_data_smithi194_103039_object102345:head# 2024-04-29T14:21:16.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4236f4cd:::benchmark_data_smithi194_103039_object140829:head# 2024-04-29T14:21:16.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4236fe0d:::benchmark_data_smithi194_103039_object151423:head# 2024-04-29T14:21:16.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4237b317:::benchmark_data_smithi194_103039_object148936:head# 2024-04-29T14:21:16.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4238f349:::benchmark_data_smithi194_103039_object74289:head# 2024-04-29T14:21:16.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42390c30:::benchmark_data_smithi194_103039_object64724:head# 2024-04-29T14:21:16.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4239eeee:::benchmark_data_smithi194_103039_object95239:head# 2024-04-29T14:21:16.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423ace11:::benchmark_data_smithi194_103039_object66058:head# 2024-04-29T14:21:16.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423bb50f:::benchmark_data_smithi194_103039_object91408:head# 2024-04-29T14:21:16.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423d89b5:::benchmark_data_smithi194_103039_object97274:head# 2024-04-29T14:21:16.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423d8f2b:::benchmark_data_smithi194_103039_object110682:head# 2024-04-29T14:21:16.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423dfb88:::benchmark_data_smithi194_103039_object51865:head# 2024-04-29T14:21:16.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423e1439:::benchmark_data_smithi194_103039_object54439:head# 2024-04-29T14:21:16.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423e6191:::benchmark_data_smithi194_103039_object111317:head# 2024-04-29T14:21:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423e857f:::benchmark_data_smithi194_103039_object126893:head# 2024-04-29T14:21:16.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423fd784:::benchmark_data_smithi194_103039_object141965:head# 2024-04-29T14:21:16.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:423ffb96:::benchmark_data_smithi194_103039_object123244:head# 2024-04-29T14:21:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42423218:::benchmark_data_smithi194_103039_object52539:head# 2024-04-29T14:21:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4242a895:::benchmark_data_smithi194_103039_object54477:head# 2024-04-29T14:21:16.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4243ab47:::benchmark_data_smithi194_103039_object97641:head# 2024-04-29T14:21:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42442c07:::benchmark_data_smithi194_103039_object98784:head# 2024-04-29T14:21:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4244c6de:::benchmark_data_smithi194_103039_object46787:head# 2024-04-29T14:21:16.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42456d53:::benchmark_data_smithi194_103039_object127012:head# 2024-04-29T14:21:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42459050:::benchmark_data_smithi194_103039_object73959:head# 2024-04-29T14:21:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42460abf:::benchmark_data_smithi194_103039_object44181:head# 2024-04-29T14:21:16.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42467296:::benchmark_data_smithi194_103039_object145887:head# 2024-04-29T14:21:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4246d23d:::benchmark_data_smithi194_103039_object47498:head# 2024-04-29T14:21:16.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42487c60:::benchmark_data_smithi194_103039_object107208:head# 2024-04-29T14:21:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4248ae43:::benchmark_data_smithi194_103039_object95709:head# 2024-04-29T14:21:16.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4248f842:::benchmark_data_smithi194_103039_object67332:head# 2024-04-29T14:21:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4249171c:::benchmark_data_smithi194_103039_object141527:head# 2024-04-29T14:21:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4249945d:::benchmark_data_smithi194_103039_object55285:head# 2024-04-29T14:21:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424a1595:::benchmark_data_smithi194_103039_object126415:head# 2024-04-29T14:21:16.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424aa72d:::benchmark_data_smithi194_103039_object156340:head# 2024-04-29T14:21:16.924 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424ac71c:::benchmark_data_smithi194_103039_object43749:head# 2024-04-29T14:21:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424add87:::benchmark_data_smithi194_103039_object135287:head# 2024-04-29T14:21:16.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424b5f9d:::benchmark_data_smithi194_103039_object89252:head# 2024-04-29T14:21:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424d14cf:::benchmark_data_smithi194_103039_object65452:head# 2024-04-29T14:21:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424d470a:::benchmark_data_smithi194_103039_object68029:head# 2024-04-29T14:21:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:16.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424e9c5a:::benchmark_data_smithi194_103039_object114479:head# 2024-04-29T14:21:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424ec288:::benchmark_data_smithi194_103039_object60891:head# 2024-04-29T14:21:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424ee558:::benchmark_data_smithi194_103039_object125337:head# 2024-04-29T14:21:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:424f5fa7:::benchmark_data_smithi194_103039_object106054:head# 2024-04-29T14:21:16.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425076c4:::benchmark_data_smithi194_103039_object153101:head# 2024-04-29T14:21:16.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:21:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425079af:::benchmark_data_smithi194_103039_object56194:head# 2024-04-29T14:21:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42508427:::benchmark_data_smithi194_103039_object122663:head# 2024-04-29T14:21:16.930 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:16.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4251b1fa:::benchmark_data_smithi194_103039_object110106:head# 2024-04-29T14:21:16.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4251de73:::benchmark_data_smithi194_103039_object58932:head# 2024-04-29T14:21:16.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4252fd59:::benchmark_data_smithi194_103039_object69589:head# 2024-04-29T14:21:16.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42546556:::benchmark_data_smithi194_103039_object59854:head# 2024-04-29T14:21:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425511e4:::benchmark_data_smithi194_103039_object50878:head# 2024-04-29T14:21:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4255a876:::benchmark_data_smithi194_103039_object96250:head# 2024-04-29T14:21:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42561c79:::benchmark_data_smithi194_103039_object129695:head# 2024-04-29T14:21:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42564da7:::benchmark_data_smithi194_103039_object45963:head# 2024-04-29T14:21:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425689e2:::benchmark_data_smithi194_103039_object103472:head# 2024-04-29T14:21:16.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42568a65:::benchmark_data_smithi194_103039_object130377:head# 2024-04-29T14:21:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42569638:::benchmark_data_smithi194_103039_object101839:head# 2024-04-29T14:21:16.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4258083d:::benchmark_data_smithi194_103039_object128805:head# 2024-04-29T14:21:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425892f0:::benchmark_data_smithi194_103039_object114997:head# 2024-04-29T14:21:16.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4259130d:::benchmark_data_smithi194_103039_object135997:head# 2024-04-29T14:21:16.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425a15f1:::benchmark_data_smithi194_103039_object130851:head# 2024-04-29T14:21:16.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425ab01f:::benchmark_data_smithi194_103039_object87117:head# 2024-04-29T14:21:16.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425ad0cc:::benchmark_data_smithi194_103039_object144489:head# 2024-04-29T14:21:16.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425bff90:::benchmark_data_smithi194_103039_object75338:head# 2024-04-29T14:21:16.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425c3c10:::benchmark_data_smithi194_103039_object127552:head# 2024-04-29T14:21:16.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425c8eb6:::benchmark_data_smithi194_103039_object156595:head# 2024-04-29T14:21:16.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425c92af:::benchmark_data_smithi194_103039_object138622:head# 2024-04-29T14:21:16.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425cf804:::benchmark_data_smithi194_103039_object156111:head# 2024-04-29T14:21:16.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425df45a:::benchmark_data_smithi194_103039_object92173:head# 2024-04-29T14:21:16.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425e1151:::benchmark_data_smithi194_103039_object112930:head# 2024-04-29T14:21:16.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425e2a3c:::benchmark_data_smithi194_103039_object55864:head# 2024-04-29T14:21:16.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425e5a3b:::benchmark_data_smithi194_103039_object65019:head# 2024-04-29T14:21:16.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425e8124:::benchmark_data_smithi194_103039_object107964:head# 2024-04-29T14:21:16.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425eac39:::benchmark_data_smithi194_103039_object125904:head# 2024-04-29T14:21:16.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425fa656:::benchmark_data_smithi194_103039_object122915:head# 2024-04-29T14:21:16.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:425fa9a7:::benchmark_data_smithi194_103039_object104728:head# 2024-04-29T14:21:16.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42611dac:::benchmark_data_smithi194_103039_object135019:head# 2024-04-29T14:21:16.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426314ee:::benchmark_data_smithi194_103039_object86315:head# 2024-04-29T14:21:16.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4263b3e0:::benchmark_data_smithi194_103039_object129998:head# 2024-04-29T14:21:16.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42643266:::benchmark_data_smithi194_103039_object98096:head# 2024-04-29T14:21:16.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42644ae8:::benchmark_data_smithi194_103039_object114145:head# 2024-04-29T14:21:16.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42647838:::benchmark_data_smithi194_103039_object114338:head# 2024-04-29T14:21:16.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4266d6e6:::benchmark_data_smithi194_103039_object142651:head# 2024-04-29T14:21:16.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42671881:::benchmark_data_smithi194_103039_object89899:head# 2024-04-29T14:21:16.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4268ea64:::benchmark_data_smithi194_103039_object135553:head# 2024-04-29T14:21:16.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42692966:::benchmark_data_smithi194_103039_object78291:head# 2024-04-29T14:21:16.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426992c6:::benchmark_data_smithi194_103039_object156057:head# 2024-04-29T14:21:16.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426a0a89:::benchmark_data_smithi194_103039_object107350:head# 2024-04-29T14:21:16.950 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:16.950 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:16.950 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:16.950 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:16.950 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:16.950 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:21:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426aba15:::benchmark_data_smithi194_103039_object97031:head# 2024-04-29T14:21:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426bd561:::benchmark_data_smithi194_103039_object97830:head# 2024-04-29T14:21:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426bf04a:::benchmark_data_smithi194_103039_object141667:head# 2024-04-29T14:21:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426c5803:::benchmark_data_smithi194_103039_object151112:head# 2024-04-29T14:21:16.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426c5978:::benchmark_data_smithi194_103039_object93410:head# 2024-04-29T14:21:16.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426ca503:::benchmark_data_smithi194_103039_object108344:head# 2024-04-29T14:21:16.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426ddf3e:::benchmark_data_smithi194_103039_object96124:head# 2024-04-29T14:21:16.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426e106f:::benchmark_data_smithi194_103039_object96630:head# 2024-04-29T14:21:16.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:426ebe14:::benchmark_data_smithi194_103039_object96280:head# 2024-04-29T14:21:16.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4270b286:::benchmark_data_smithi194_103039_object113814:head# 2024-04-29T14:21:16.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427129d4:::benchmark_data_smithi194_103039_object136606:head# 2024-04-29T14:21:16.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42715b91:::benchmark_data_smithi194_103039_object52091:head# 2024-04-29T14:21:16.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4272d4e2:::benchmark_data_smithi194_103039_object110458:head# 2024-04-29T14:21:16.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42732d56:::benchmark_data_smithi194_103039_object149287:head# 2024-04-29T14:21:16.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4273975e:::benchmark_data_smithi194_103039_object98764:head# 2024-04-29T14:21:16.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4273d9b1:::benchmark_data_smithi194_103039_object68208:head# 2024-04-29T14:21:16.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4274e6b6:::benchmark_data_smithi194_103039_object59244:head# 2024-04-29T14:21:16.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42750860:::benchmark_data_smithi194_103039_object151744:head# 2024-04-29T14:21:16.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427558ca:::benchmark_data_smithi194_103039_object70649:head# 2024-04-29T14:21:16.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4275ff0e:::benchmark_data_smithi194_103039_object56363:head# 2024-04-29T14:21:16.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4276984f:::benchmark_data_smithi194_103039_object94523:head# 2024-04-29T14:21:16.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4276a909:::benchmark_data_smithi194_103039_object147396:head# 2024-04-29T14:21:16.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427786dd:::benchmark_data_smithi194_103039_object72622:head# 2024-04-29T14:21:16.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4277a75d:::benchmark_data_smithi194_103039_object118455:head# 2024-04-29T14:21:16.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42780b47:::benchmark_data_smithi194_103039_object97741:head# 2024-04-29T14:21:16.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42787fcc:::benchmark_data_smithi194_103039_object76139:head# 2024-04-29T14:21:16.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42794cc6:::benchmark_data_smithi194_103039_object70475:head# 2024-04-29T14:21:16.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427baed2:::benchmark_data_smithi194_103039_object96744:head# 2024-04-29T14:21:16.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427bbe7c:::benchmark_data_smithi194_103039_object57290:head# 2024-04-29T14:21:16.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427d3721:::benchmark_data_smithi194_103039_object124478:head# 2024-04-29T14:21:16.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427deb20:::benchmark_data_smithi194_103039_object56277:head# 2024-04-29T14:21:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427e24a9:::benchmark_data_smithi194_103039_object45098:head# 2024-04-29T14:21:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:427e5e7a:::benchmark_data_smithi194_103039_object48586:head# 2024-04-29T14:21:16.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42806f76:::benchmark_data_smithi194_103039_object155143:head# 2024-04-29T14:21:16.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42808fef:::benchmark_data_smithi194_103039_object55025:head# 2024-04-29T14:21:16.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4280a66f:::benchmark_data_smithi194_103039_object153440:head# 2024-04-29T14:21:16.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42814b3d:::benchmark_data_smithi194_103039_object77809:head# 2024-04-29T14:21:16.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428162f1:::benchmark_data_smithi194_103039_object52075:head# 2024-04-29T14:21:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4281afc5:::benchmark_data_smithi194_103039_object88926:head# 2024-04-29T14:21:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42826a31:::benchmark_data_smithi194_103039_object130319:head# 2024-04-29T14:21:16.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4282bb15:::benchmark_data_smithi194_103039_object45612:head# 2024-04-29T14:21:16.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42838362:::benchmark_data_smithi194_103039_object127628:head# 2024-04-29T14:21:16.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4283db61:::benchmark_data_smithi194_103039_object52493:head# 2024-04-29T14:21:16.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42840da4:::benchmark_data_smithi194_103039_object91010:head# 2024-04-29T14:21:16.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4284843f:::benchmark_data_smithi194_103039_object114386:head# 2024-04-29T14:21:16.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42857cd5:::benchmark_data_smithi194_103039_object120393:head# 2024-04-29T14:21:16.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4286020d:::benchmark_data_smithi194_103039_object95590:head# 2024-04-29T14:21:16.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4286ad60:::benchmark_data_smithi194_103039_object133466:head# 2024-04-29T14:21:16.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4286d5a1:::benchmark_data_smithi194_103039_object49883:head# 2024-04-29T14:21:16.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42871b6a:::benchmark_data_smithi194_103039_object70215:head# 2024-04-29T14:21:16.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42874f5d:::benchmark_data_smithi194_103039_object42618:head# 2024-04-29T14:21:16.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42877450:::benchmark_data_smithi194_103039_object58426:head# 2024-04-29T14:21:16.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428865fb:::benchmark_data_smithi194_103039_object93383:head# 2024-04-29T14:21:16.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4288e5ac:::benchmark_data_smithi194_103039_object83666:head# 2024-04-29T14:21:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4288ee24:::benchmark_data_smithi194_103039_object119302:head# 2024-04-29T14:21:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428948c8:::benchmark_data_smithi194_103039_object117754:head# 2024-04-29T14:21:16.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4289963c:::benchmark_data_smithi194_103039_object119995:head# 2024-04-29T14:21:16.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4289a2f2:::benchmark_data_smithi194_103039_object94403:head# 2024-04-29T14:21:16.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428a3467:::benchmark_data_smithi194_103039_object74497:head# 2024-04-29T14:21:16.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428a4b49:::benchmark_data_smithi194_103039_object104447:head# 2024-04-29T14:21:16.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428b23db:::benchmark_data_smithi194_103039_object117616:head# 2024-04-29T14:21:16.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428c1685:::benchmark_data_smithi194_103039_object82268:head# 2024-04-29T14:21:16.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428c51b0:::benchmark_data_smithi194_103039_object113739:head# 2024-04-29T14:21:16.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428cb611:::benchmark_data_smithi194_103039_object89411:head# 2024-04-29T14:21:16.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428db44b:::benchmark_data_smithi194_103039_object135251:head# 2024-04-29T14:21:16.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428e335b:::benchmark_data_smithi194_103039_object73561:head# 2024-04-29T14:21:16.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428e384c:::benchmark_data_smithi194_103039_object107186:head# 2024-04-29T14:21:16.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428ed856:::benchmark_data_smithi194_103039_object52509:head# 2024-04-29T14:21:16.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428ee98d:::benchmark_data_smithi194_103039_object149259:head# 2024-04-29T14:21:16.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:428fe743:::benchmark_data_smithi194_103039_object63059:head# 2024-04-29T14:21:16.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42905540:::benchmark_data_smithi194_103039_object99716:head# 2024-04-29T14:21:16.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42912090:::benchmark_data_smithi194_103039_object61932:head# 2024-04-29T14:21:16.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4291d56d:::benchmark_data_smithi194_103039_object95158:head# 2024-04-29T14:21:16.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42932a02:::benchmark_data_smithi194_103039_object143965:head# 2024-04-29T14:21:16.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42945b60:::benchmark_data_smithi194_103039_object58903:head# 2024-04-29T14:21:16.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429511d4:::benchmark_data_smithi194_103039_object73601:head# 2024-04-29T14:21:16.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42951c94:::benchmark_data_smithi194_103039_object41931:head# 2024-04-29T14:21:16.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4296fe21:::benchmark_data_smithi194_103039_object116295:head# 2024-04-29T14:21:16.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4297b5a1:::benchmark_data_smithi194_103039_object130462:head# 2024-04-29T14:21:16.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4298d177:::benchmark_data_smithi194_103039_object136116:head# 2024-04-29T14:21:16.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42996dd8:::benchmark_data_smithi194_103039_object64838:head# 2024-04-29T14:21:16.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4299b35b:::benchmark_data_smithi194_103039_object106100:head# 2024-04-29T14:21:16.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429abc7c:::benchmark_data_smithi194_103039_object51851:head# 2024-04-29T14:21:16.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429b7107:::benchmark_data_smithi194_103039_object101718:head# 2024-04-29T14:21:16.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429bb9d5:::benchmark_data_smithi194_103039_object145338:head# 2024-04-29T14:21:16.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429c1e3b:::benchmark_data_smithi194_103039_object88658:head# 2024-04-29T14:21:16.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429c9625:::benchmark_data_smithi194_103039_object115936:head# 2024-04-29T14:21:16.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429ce48e:::benchmark_data_smithi194_103039_object141498:head# 2024-04-29T14:21:16.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429ce5a8:::benchmark_data_smithi194_103039_object81085:head# 2024-04-29T14:21:16.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429d3623:::benchmark_data_smithi194_103039_object98659:head# 2024-04-29T14:21:16.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429d743b:::benchmark_data_smithi194_103039_object138502:head# 2024-04-29T14:21:16.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429d7e5d:::benchmark_data_smithi194_103039_object90286:head# 2024-04-29T14:21:16.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429dea3a:::benchmark_data_smithi194_103039_object103532:head# 2024-04-29T14:21:16.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429df663:::benchmark_data_smithi194_103039_object135933:head# 2024-04-29T14:21:16.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429e80ca:::benchmark_data_smithi194_103039_object90606:head# 2024-04-29T14:21:16.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429e9a34:::benchmark_data_smithi194_103039_object50849:head# 2024-04-29T14:21:16.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429fc0eb:::benchmark_data_smithi194_103039_object60717:head# 2024-04-29T14:21:16.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429fc9c6:::benchmark_data_smithi194_103039_object144150:head# 2024-04-29T14:21:16.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:429fddad:::benchmark_data_smithi194_103039_object100184:head# 2024-04-29T14:21:16.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a02a7d:::benchmark_data_smithi194_103039_object96304:head# 2024-04-29T14:21:16.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a0bf7f:::benchmark_data_smithi194_103039_object144965:head# 2024-04-29T14:21:16.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a0c45a:::benchmark_data_smithi194_103039_object106072:head# 2024-04-29T14:21:16.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a0cefb:::benchmark_data_smithi194_103039_object97257:head# 2024-04-29T14:21:16.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a0d6de:::benchmark_data_smithi194_103039_object103318:head# 2024-04-29T14:21:16.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a13c14:::benchmark_data_smithi194_103039_object104012:head# 2024-04-29T14:21:16.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a14f25:::benchmark_data_smithi194_103039_object59496:head# 2024-04-29T14:21:16.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a18974:::benchmark_data_smithi194_103039_object56105:head# 2024-04-29T14:21:16.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a1f309:::benchmark_data_smithi194_103039_object63725:head# 2024-04-29T14:21:17.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a1fdca:::benchmark_data_smithi194_103039_object148897:head# 2024-04-29T14:21:17.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a24733:::benchmark_data_smithi194_103039_object81472:head# 2024-04-29T14:21:17.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a26e7f:::benchmark_data_smithi194_103039_object139709:head# 2024-04-29T14:21:17.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a3815d:::benchmark_data_smithi194_103039_object41698:head# 2024-04-29T14:21:17.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a39f6f:::benchmark_data_smithi194_103039_object126490:head# 2024-04-29T14:21:17.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a3d667:::benchmark_data_smithi194_103039_object152202:head# 2024-04-29T14:21:17.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a3e069:::benchmark_data_smithi194_103039_object131823:head# 2024-04-29T14:21:17.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a4c483:::benchmark_data_smithi194_103039_object49310:head# 2024-04-29T14:21:17.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a4d263:::benchmark_data_smithi194_103039_object146638:head# 2024-04-29T14:21:17.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a5126a:::benchmark_data_smithi194_103039_object123065:head# 2024-04-29T14:21:17.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a624c3:::benchmark_data_smithi194_103039_object65244:head# 2024-04-29T14:21:17.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a6310e:::benchmark_data_smithi194_103039_object113064:head# 2024-04-29T14:21:17.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a63b6b:::benchmark_data_smithi194_103039_object78615:head# 2024-04-29T14:21:17.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a65c03:::benchmark_data_smithi194_103039_object42327:head# 2024-04-29T14:21:17.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a6a172:::benchmark_data_smithi194_103039_object123717:head# 2024-04-29T14:21:17.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a71409:::benchmark_data_smithi194_103039_object132967:head# 2024-04-29T14:21:17.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a95544:::benchmark_data_smithi194_103039_object154031:head# 2024-04-29T14:21:17.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a9bed7:::benchmark_data_smithi194_103039_object47244:head# 2024-04-29T14:21:17.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a9e5ff:::benchmark_data_smithi194_103039_object129883:head# 2024-04-29T14:21:17.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42a9ef4d:::benchmark_data_smithi194_103039_object132293:head# 2024-04-29T14:21:17.009 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42aac679:::benchmark_data_smithi194_103039_object45931:head# 2024-04-29T14:21:17.009 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42abaa8d:::benchmark_data_smithi194_103039_object127026:head# 2024-04-29T14:21:17.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ac8423:::benchmark_data_smithi194_103039_object41700:head# 2024-04-29T14:21:17.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ad0da6:::benchmark_data_smithi194_103039_object54897:head# 2024-04-29T14:21:17.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42af0e6d:::benchmark_data_smithi194_103039_object102813:head# 2024-04-29T14:21:17.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42af7797:::benchmark_data_smithi194_103039_object112458:head# 2024-04-29T14:21:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b0361c:::benchmark_data_smithi194_103039_object88251:head# 2024-04-29T14:21:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b042e8:::benchmark_data_smithi194_103039_object47306:head# 2024-04-29T14:21:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b05fc6:::benchmark_data_smithi194_103039_object81532:head# 2024-04-29T14:21:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b0b4e6:::benchmark_data_smithi194_103039_object104632:head# 2024-04-29T14:21:17.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b10977:::benchmark_data_smithi194_103039_object98899:head# 2024-04-29T14:21:17.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b14771:::benchmark_data_smithi194_103039_object149739:head# 2024-04-29T14:21:17.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b3128d:::benchmark_data_smithi194_103039_object120417:head# 2024-04-29T14:21:17.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b3368f:::benchmark_data_smithi194_103039_object97318:head# 2024-04-29T14:21:17.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b42220:::benchmark_data_smithi194_103039_object65678:head# 2024-04-29T14:21:17.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b4eb78:::benchmark_data_smithi194_103039_object79924:head# 2024-04-29T14:21:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b5e1aa:::benchmark_data_smithi194_103039_object99230:head# 2024-04-29T14:21:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b6f31b:::benchmark_data_smithi194_103039_object98857:head# 2024-04-29T14:21:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b70338:::benchmark_data_smithi194_103039_object128005:head# 2024-04-29T14:21:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b744eb:::benchmark_data_smithi194_103039_object142902:head# 2024-04-29T14:21:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b8004c:::benchmark_data_smithi194_103039_object65956:head# 2024-04-29T14:21:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b8c3f2:::benchmark_data_smithi194_103039_object70487:head# 2024-04-29T14:21:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b90cd5:::benchmark_data_smithi194_103039_object134823:head# 2024-04-29T14:21:17.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b957f8:::benchmark_data_smithi194_103039_object155171:head# 2024-04-29T14:21:17.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b970a1:::benchmark_data_smithi194_103039_object90822:head# 2024-04-29T14:21:17.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b97f53:::benchmark_data_smithi194_103039_object112438:head# 2024-04-29T14:21:17.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42b9b142:::benchmark_data_smithi194_103039_object76183:head# 2024-04-29T14:21:17.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ba8f75:::benchmark_data_smithi194_103039_object97408:head# 2024-04-29T14:21:17.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42bac931:::benchmark_data_smithi194_103039_object42808:head# 2024-04-29T14:21:17.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42bb4208:::benchmark_data_smithi194_103039_object69507:head# 2024-04-29T14:21:17.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42bbc620:::benchmark_data_smithi194_103039_object44594:head# 2024-04-29T14:21:17.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42bbf794:::benchmark_data_smithi194_103039_object76380:head# 2024-04-29T14:21:17.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42bd91a1:::benchmark_data_smithi194_103039_object78427:head# 2024-04-29T14:21:17.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42be8811:::benchmark_data_smithi194_103039_object72194:head# 2024-04-29T14:21:17.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42bf0a6c:::benchmark_data_smithi194_103039_object52211:head# 2024-04-29T14:21:17.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42bfa7c9:::benchmark_data_smithi194_103039_object111350:head# 2024-04-29T14:21:17.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c04f64:::benchmark_data_smithi194_103039_object96757:head# 2024-04-29T14:21:17.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c13a00:::benchmark_data_smithi194_103039_object102225:head# 2024-04-29T14:21:17.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c18b2e:::benchmark_data_smithi194_103039_object83822:head# 2024-04-29T14:21:17.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c1954e:::benchmark_data_smithi194_103039_object64883:head# 2024-04-29T14:21:17.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:17.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c1e7cd:::benchmark_data_smithi194_103039_object87725:head# 2024-04-29T14:21:17.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c1f496:::benchmark_data_smithi194_103039_object122977:head# 2024-04-29T14:21:17.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c2ac36:::benchmark_data_smithi194_103039_object79224:head# 2024-04-29T14:21:17.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c3d909:::benchmark_data_smithi194_103039_object134407:head# 2024-04-29T14:21:17.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c5523e:::benchmark_data_smithi194_103039_object74050:head# 2024-04-29T14:21:17.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c560fa:::benchmark_data_smithi194_103039_object109861:head# 2024-04-29T14:21:17.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c5f683:::benchmark_data_smithi194_103039_object45615:head# 2024-04-29T14:21:17.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c68cef:::benchmark_data_smithi194_103039_object68693:head# 2024-04-29T14:21:17.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c73339:::benchmark_data_smithi194_103039_object70849:head# 2024-04-29T14:21:17.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c75235:::benchmark_data_smithi194_103039_object130958:head# 2024-04-29T14:21:17.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c81c48:::benchmark_data_smithi194_103039_object124885:head# 2024-04-29T14:21:17.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c8d59c:::benchmark_data_smithi194_103039_object102916:head# 2024-04-29T14:21:17.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c94b2b:::benchmark_data_smithi194_103039_object154982:head# 2024-04-29T14:21:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c956d0:::benchmark_data_smithi194_103039_object108278:head# 2024-04-29T14:21:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c9b4fc:::benchmark_data_smithi194_103039_object111882:head# 2024-04-29T14:21:17.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42c9bede:::benchmark_data_smithi194_103039_object139610:head# 2024-04-29T14:21:17.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ca02c7:::benchmark_data_smithi194_103039_object113725:head# 2024-04-29T14:21:17.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cac670:::benchmark_data_smithi194_103039_object121918:head# 2024-04-29T14:21:17.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cb3cd7:::benchmark_data_smithi194_103039_object123408:head# 2024-04-29T14:21:17.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cb756e:::benchmark_data_smithi194_103039_object149948:head# 2024-04-29T14:21:17.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cbdc2e:::benchmark_data_smithi194_103039_object127578:head# 2024-04-29T14:21:17.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cbfbdc:::benchmark_data_smithi194_103039_object48566:head# 2024-04-29T14:21:17.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cc1578:::benchmark_data_smithi194_103039_object150028:head# 2024-04-29T14:21:17.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cc8e53:::benchmark_data_smithi194_103039_object73410:head# 2024-04-29T14:21:17.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ccab46:::benchmark_data_smithi194_103039_object49051:head# 2024-04-29T14:21:17.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ccf596:::benchmark_data_smithi194_103039_object92195:head# 2024-04-29T14:21:17.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cd95d0:::benchmark_data_smithi194_103039_object118285:head# 2024-04-29T14:21:17.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42cdd17e:::benchmark_data_smithi194_103039_object110609:head# 2024-04-29T14:21:17.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ce740a:::benchmark_data_smithi194_103039_object122275:head# 2024-04-29T14:21:17.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d006e9:::benchmark_data_smithi194_103039_object43330:head# 2024-04-29T14:21:17.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d047ac:::benchmark_data_smithi194_103039_object140284:head# 2024-04-29T14:21:17.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d0c8ea:::benchmark_data_smithi194_103039_object147648:head# 2024-04-29T14:21:17.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d13b87:::benchmark_data_smithi194_103039_object116767:head# 2024-04-29T14:21:17.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d148b6:::benchmark_data_smithi194_103039_object127483:head# 2024-04-29T14:21:17.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d16f99:::benchmark_data_smithi194_103039_object79508:head# 2024-04-29T14:21:17.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d178b3:::benchmark_data_smithi194_103039_object153891:head# 2024-04-29T14:21:17.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d26e3e:::benchmark_data_smithi194_103039_object106820:head# 2024-04-29T14:21:17.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d39d16:::benchmark_data_smithi194_103039_object138531:head# 2024-04-29T14:21:17.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d44e24:::benchmark_data_smithi194_103039_object111073:head# 2024-04-29T14:21:17.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d74b97:::benchmark_data_smithi194_103039_object151041:head# 2024-04-29T14:21:17.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d79884:::benchmark_data_smithi194_103039_object102936:head# 2024-04-29T14:21:17.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d87240:::benchmark_data_smithi194_103039_object53281:head# 2024-04-29T14:21:17.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d8bb26:::benchmark_data_smithi194_103039_object145433:head# 2024-04-29T14:21:17.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d8c0fc:::benchmark_data_smithi194_103039_object113688:head# 2024-04-29T14:21:17.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42d9710e:::benchmark_data_smithi194_103039_object128165:head# 2024-04-29T14:21:17.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42da3f12:::benchmark_data_smithi194_103039_object111530:head# 2024-04-29T14:21:17.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42dbb006:::benchmark_data_smithi194_103039_object109099:head# 2024-04-29T14:21:17.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42dbb427:::benchmark_data_smithi194_103039_object72949:head# 2024-04-29T14:21:17.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42dd2320:::benchmark_data_smithi194_103039_object57475:head# 2024-04-29T14:21:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42dd2976:::benchmark_data_smithi194_103039_object76019:head# 2024-04-29T14:21:17.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42dd4ef1:::benchmark_data_smithi194_103039_object69234:head# 2024-04-29T14:21:17.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ddad90:::benchmark_data_smithi194_103039_object153248:head# 2024-04-29T14:21:17.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42dde2cc:::benchmark_data_smithi194_103039_object107736:head# 2024-04-29T14:21:17.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ddfd2d:::benchmark_data_smithi194_103039_object66356:head# 2024-04-29T14:21:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42de6290:::benchmark_data_smithi194_103039_object148421:head# 2024-04-29T14:21:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42de8877:::benchmark_data_smithi194_103039_object65700:head# 2024-04-29T14:21:17.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42dedbf8:::benchmark_data_smithi194_103039_object90451:head# 2024-04-29T14:21:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e038ae:::benchmark_data_smithi194_103039_object109742:head# 2024-04-29T14:21:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e04fa9:::benchmark_data_smithi194_103039_object137862:head# 2024-04-29T14:21:17.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e090bc:::benchmark_data_smithi194_103039_object90004:head# 2024-04-29T14:21:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e0ac64:::benchmark_data_smithi194_103039_object143695:head# 2024-04-29T14:21:17.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e21d01:::benchmark_data_smithi194_103039_object125172:head# 2024-04-29T14:21:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e331ef:::benchmark_data_smithi194_103039_object138779:head# 2024-04-29T14:21:17.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e43f9b:::benchmark_data_smithi194_103039_object94693:head# 2024-04-29T14:21:17.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e4f3a4:::benchmark_data_smithi194_103039_object84246:head# 2024-04-29T14:21:17.058 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:17.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e5a5de:::benchmark_data_smithi194_103039_object129454:head# 2024-04-29T14:21:17.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e6a58f:::benchmark_data_smithi194_103039_object112756:head# 2024-04-29T14:21:17.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e6ba9c:::benchmark_data_smithi194_103039_object101056:head# 2024-04-29T14:21:17.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e70c9a:::benchmark_data_smithi194_103039_object99062:head# 2024-04-29T14:21:17.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e7fc17:::benchmark_data_smithi194_103039_object63427:head# 2024-04-29T14:21:17.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e87b40:::benchmark_data_smithi194_103039_object50068:head# 2024-04-29T14:21:17.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e8966a:::benchmark_data_smithi194_103039_object112205:head# 2024-04-29T14:21:17.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e8b166:::benchmark_data_smithi194_103039_object49759:head# 2024-04-29T14:21:17.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e9705f:::benchmark_data_smithi194_103039_object156087:head# 2024-04-29T14:21:17.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42e9df4d:::benchmark_data_smithi194_103039_object139200:head# 2024-04-29T14:21:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ea9796:::benchmark_data_smithi194_103039_object143548:head# 2024-04-29T14:21:17.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42eae369:::benchmark_data_smithi194_103039_object123095:head# 2024-04-29T14:21:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42eae684:::benchmark_data_smithi194_103039_object73426:head# 2024-04-29T14:21:17.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42eb21b8:::benchmark_data_smithi194_103039_object97812:head# 2024-04-29T14:21:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42eb523c:::benchmark_data_smithi194_103039_object50378:head# 2024-04-29T14:21:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ebb973:::benchmark_data_smithi194_103039_object94124:head# 2024-04-29T14:21:17.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ebcc5c:::benchmark_data_smithi194_103039_object109608:head# 2024-04-29T14:21:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ebcfe2:::benchmark_data_smithi194_103039_object96129:head# 2024-04-29T14:21:17.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ebf526:::benchmark_data_smithi194_103039_object103257:head# 2024-04-29T14:21:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ec15fc:::benchmark_data_smithi194_103039_object118094:head# 2024-04-29T14:21:17.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ec3d65:::benchmark_data_smithi194_103039_object85000:head# 2024-04-29T14:21:17.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ec9604:::benchmark_data_smithi194_103039_object98227:head# 2024-04-29T14:21:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ed4ecd:::benchmark_data_smithi194_103039_object145025:head# 2024-04-29T14:21:17.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ed54e6:::benchmark_data_smithi194_103039_object130936:head# 2024-04-29T14:21:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ee983a:::benchmark_data_smithi194_103039_object117073:head# 2024-04-29T14:21:17.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42eea65a:::benchmark_data_smithi194_103039_object53333:head# 2024-04-29T14:21:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ef01e5:::benchmark_data_smithi194_103039_object78508:head# 2024-04-29T14:21:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ef9b11:::benchmark_data_smithi194_103039_object82939:head# 2024-04-29T14:21:17.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f1891b:::benchmark_data_smithi194_103039_object103151:head# 2024-04-29T14:21:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f403d8:::benchmark_data_smithi194_103039_object111370:head# 2024-04-29T14:21:17.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f47249:::benchmark_data_smithi194_103039_object61309:head# 2024-04-29T14:21:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f6be2a:::benchmark_data_smithi194_103039_object42115:head# 2024-04-29T14:21:17.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f6ef29:::benchmark_data_smithi194_103039_object94636:head# 2024-04-29T14:21:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f7511c:::benchmark_data_smithi194_103039_object79660:head# 2024-04-29T14:21:17.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f757ca:::benchmark_data_smithi194_103039_object52424:head# 2024-04-29T14:21:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f79bf0:::benchmark_data_smithi194_103039_object103182:head# 2024-04-29T14:21:17.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f7c5b6:::benchmark_data_smithi194_103039_object64154:head# 2024-04-29T14:21:17.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f7ea0f:::benchmark_data_smithi194_103039_object59009:head# 2024-04-29T14:21:17.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42f98961:::benchmark_data_smithi194_103039_object92969:head# 2024-04-29T14:21:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fae5d8:::benchmark_data_smithi194_103039_object52869:head# 2024-04-29T14:21:17.077 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fb0741:::benchmark_data_smithi194_103039_object74818:head# 2024-04-29T14:21:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fb537b:::benchmark_data_smithi194_103039_object137765:head# 2024-04-29T14:21:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fbe2e3:::benchmark_data_smithi194_103039_object84534:head# 2024-04-29T14:21:17.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fc9705:::benchmark_data_smithi194_103039_object92929:head# 2024-04-29T14:21:17.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fd889e:::benchmark_data_smithi194_103039_object83598:head# 2024-04-29T14:21:17.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fddb42:::benchmark_data_smithi194_103039_object156182:head# 2024-04-29T14:21:17.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fe1e87:::benchmark_data_smithi194_103039_object103983:head# 2024-04-29T14:21:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:17.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fe27bb:::benchmark_data_smithi194_103039_object141938:head# 2024-04-29T14:21:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fe47a5:::benchmark_data_smithi194_103039_object85503:head# 2024-04-29T14:21:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fe6f36:::benchmark_data_smithi194_103039_object136409:head# 2024-04-29T14:21:17.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42fec941:::benchmark_data_smithi194_103039_object70612:head# 2024-04-29T14:21:17.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ff004b:::benchmark_data_smithi194_103039_object113316:head# 2024-04-29T14:21:17.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ff00fd:::benchmark_data_smithi194_103039_object68375:head# 2024-04-29T14:21:17.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:42ffa380:::benchmark_data_smithi194_103039_object68953:head# 2024-04-29T14:21:17.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430041e6:::benchmark_data_smithi194_103039_object84215:head# 2024-04-29T14:21:17.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430062a1:::benchmark_data_smithi194_103039_object138061:head# 2024-04-29T14:21:17.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4300645d:::benchmark_data_smithi194_103039_object111881:head# 2024-04-29T14:21:17.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43008624:::benchmark_data_smithi194_103039_object92151:head# 2024-04-29T14:21:17.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43015935:::benchmark_data_smithi194_103039_object72060:head# 2024-04-29T14:21:17.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4301630a:::benchmark_data_smithi194_103039_object77203:head# 2024-04-29T14:21:17.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4304247c:::benchmark_data_smithi194_103039_object76365:head# 2024-04-29T14:21:17.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430469b2:::benchmark_data_smithi194_103039_object113262:head# 2024-04-29T14:21:17.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4304af89:::benchmark_data_smithi194_103039_object64618:head# 2024-04-29T14:21:17.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43050510:::benchmark_data_smithi194_103039_object140084:head# 2024-04-29T14:21:17.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43058663:::benchmark_data_smithi194_103039_object128295:head# 2024-04-29T14:21:17.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4306ba6a:::benchmark_data_smithi194_103039_object134659:head# 2024-04-29T14:21:17.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4306f6a7:::benchmark_data_smithi194_103039_object134795:head# 2024-04-29T14:21:17.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4307d2e0:::benchmark_data_smithi194_103039_object43108:head# 2024-04-29T14:21:17.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43085c9b:::benchmark_data_smithi194_103039_object140267:head# 2024-04-29T14:21:17.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43088627:::benchmark_data_smithi194_103039_object135060:head# 2024-04-29T14:21:17.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4308867a:::benchmark_data_smithi194_103039_object129847:head# 2024-04-29T14:21:17.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4308e29e:::benchmark_data_smithi194_103039_object119824:head# 2024-04-29T14:21:17.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4309d90f:::benchmark_data_smithi194_103039_object114670:head# 2024-04-29T14:21:17.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430a142c:::benchmark_data_smithi194_103039_object77947:head# 2024-04-29T14:21:17.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430a9a55:::benchmark_data_smithi194_103039_object89236:head# 2024-04-29T14:21:17.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430c022f:::benchmark_data_smithi194_103039_object66703:head# 2024-04-29T14:21:17.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430cebbc:::benchmark_data_smithi194_103039_object90487:head# 2024-04-29T14:21:17.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430d1f62:::benchmark_data_smithi194_103039_object125395:head# 2024-04-29T14:21:17.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430d3de0:::benchmark_data_smithi194_103039_object89098:head# 2024-04-29T14:21:17.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430d7886:::benchmark_data_smithi194_103039_object104755:head# 2024-04-29T14:21:17.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:430fb971:::benchmark_data_smithi194_103039_object57057:head# 2024-04-29T14:21:17.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4310078a:::benchmark_data_smithi194_103039_object91945:head# 2024-04-29T14:21:17.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431166b3:::benchmark_data_smithi194_103039_object95099:head# 2024-04-29T14:21:17.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4311a645:::benchmark_data_smithi194_103039_object103456:head# 2024-04-29T14:21:17.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43131c81:::benchmark_data_smithi194_103039_object129756:head# 2024-04-29T14:21:17.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431479fb:::benchmark_data_smithi194_103039_object127528:head# 2024-04-29T14:21:17.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43147d29:::benchmark_data_smithi194_103039_object102561:head# 2024-04-29T14:21:17.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4314c849:::benchmark_data_smithi194_103039_object48019:head# 2024-04-29T14:21:17.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4314d90a:::benchmark_data_smithi194_103039_object118191:head# 2024-04-29T14:21:17.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43151530:::benchmark_data_smithi194_103039_object142811:head# 2024-04-29T14:21:17.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43152a69:::benchmark_data_smithi194_103039_object129192:head# 2024-04-29T14:21:17.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4315878c:::benchmark_data_smithi194_103039_object99445:head# 2024-04-29T14:21:17.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4316700b:::benchmark_data_smithi194_103039_object101707:head# 2024-04-29T14:21:17.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4316b590:::benchmark_data_smithi194_103039_object41966:head# 2024-04-29T14:21:17.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4316c888:::benchmark_data_smithi194_103039_object144448:head# 2024-04-29T14:21:17.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43173d99:::benchmark_data_smithi194_103039_object99821:head# 2024-04-29T14:21:17.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4317d17d:::benchmark_data_smithi194_103039_object155200:head# 2024-04-29T14:21:17.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43180cba:::benchmark_data_smithi194_103039_object117377:head# 2024-04-29T14:21:17.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431827d9:::benchmark_data_smithi194_103039_object119520:head# 2024-04-29T14:21:17.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4318330b:::benchmark_data_smithi194_103039_object106422:head# 2024-04-29T14:21:17.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43185fcf:::benchmark_data_smithi194_103039_object61761:head# 2024-04-29T14:21:17.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4318eee1:::benchmark_data_smithi194_103039_object59555:head# 2024-04-29T14:21:17.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431936c4:::benchmark_data_smithi194_103039_object49491:head# 2024-04-29T14:21:17.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43197529:::benchmark_data_smithi194_103039_object143575:head# 2024-04-29T14:21:17.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43199a78:::benchmark_data_smithi194_103039_object147971:head# 2024-04-29T14:21:17.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4319d611:::benchmark_data_smithi194_103039_object58751:head# 2024-04-29T14:21:17.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431a9240:::benchmark_data_smithi194_103039_object69738:head# 2024-04-29T14:21:17.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431c2abf:::benchmark_data_smithi194_103039_object146333:head# 2024-04-29T14:21:17.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431c91b0:::benchmark_data_smithi194_103039_object64767:head# 2024-04-29T14:21:17.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431d0c9f:::benchmark_data_smithi194_103039_object105517:head# 2024-04-29T14:21:17.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431da2b4:::benchmark_data_smithi194_103039_object76486:head# 2024-04-29T14:21:17.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431db35d:::benchmark_data_smithi194_103039_object150935:head# 2024-04-29T14:21:17.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431df023:::benchmark_data_smithi194_103039_object91730:head# 2024-04-29T14:21:17.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431df905:::benchmark_data_smithi194_103039_object146428:head# 2024-04-29T14:21:17.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431dff2e:::benchmark_data_smithi194_103039_object44622:head# 2024-04-29T14:21:17.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431f3b02:::benchmark_data_smithi194_103039_object92005:head# 2024-04-29T14:21:17.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431f55e6:::benchmark_data_smithi194_103039_object119817:head# 2024-04-29T14:21:17.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:431ffaf5:::benchmark_data_smithi194_103039_object152562:head# 2024-04-29T14:21:17.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43207da2:::benchmark_data_smithi194_103039_object87988:head# 2024-04-29T14:21:17.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4320822e:::benchmark_data_smithi194_103039_object139461:head# 2024-04-29T14:21:17.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4320c78d:::benchmark_data_smithi194_103039_object98380:head# 2024-04-29T14:21:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4320c85c:::benchmark_data_smithi194_103039_object45438:head# 2024-04-29T14:21:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4321754c:::benchmark_data_smithi194_103039_object147488:head# 2024-04-29T14:21:17.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432398c3:::benchmark_data_smithi194_103039_object122565:head# 2024-04-29T14:21:17.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43244230:::benchmark_data_smithi194_103039_object126024:head# 2024-04-29T14:21:17.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4325ec56:::benchmark_data_smithi194_103039_object83670:head# 2024-04-29T14:21:17.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4325ed47:::benchmark_data_smithi194_103039_object137671:head# 2024-04-29T14:21:17.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4326a00f:::benchmark_data_smithi194_103039_object71747:head# 2024-04-29T14:21:17.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4328188d:::benchmark_data_smithi194_103039_object43793:head# 2024-04-29T14:21:17.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4328712f:::benchmark_data_smithi194_103039_object43638:head# 2024-04-29T14:21:17.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432ad43e:::benchmark_data_smithi194_103039_object100897:head# 2024-04-29T14:21:17.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432b7c28:::benchmark_data_smithi194_103039_object132529:head# 2024-04-29T14:21:17.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432beaa0:::benchmark_data_smithi194_103039_object80274:head# 2024-04-29T14:21:17.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432c4355:::benchmark_data_smithi194_103039_object123453:head# 2024-04-29T14:21:17.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432d921d:::benchmark_data_smithi194_103039_object117712:head# 2024-04-29T14:21:17.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432eb49e:::benchmark_data_smithi194_103039_object99222:head# 2024-04-29T14:21:17.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432fa4a6:::benchmark_data_smithi194_103039_object41738:head# 2024-04-29T14:21:17.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432fe1e9:::benchmark_data_smithi194_103039_object129745:head# 2024-04-29T14:21:17.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:432fe6d3:::benchmark_data_smithi194_103039_object149888:head# 2024-04-29T14:21:17.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43302432:::benchmark_data_smithi194_103039_object83150:head# 2024-04-29T14:21:17.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4330b668:::benchmark_data_smithi194_103039_object122063:head# 2024-04-29T14:21:17.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4330c77d:::benchmark_data_smithi194_103039_object101149:head# 2024-04-29T14:21:17.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43319b60:::benchmark_data_smithi194_103039_object128749:head# 2024-04-29T14:21:17.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43320377:::benchmark_data_smithi194_103039_object57145:head# 2024-04-29T14:21:17.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43323b35:::benchmark_data_smithi194_103039_object149095:head# 2024-04-29T14:21:17.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43328ae8:::benchmark_data_smithi194_103039_object63269:head# 2024-04-29T14:21:17.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43335d1e:::benchmark_data_smithi194_103039_object123263:head# 2024-04-29T14:21:17.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43337687:::benchmark_data_smithi194_103039_object139126:head# 2024-04-29T14:21:17.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43337b2d:::benchmark_data_smithi194_103039_object134968:head# 2024-04-29T14:21:17.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4333f7ce:::benchmark_data_smithi194_103039_object90285:head# 2024-04-29T14:21:17.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4334eec8:::benchmark_data_smithi194_103039_object77874:head# 2024-04-29T14:21:17.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433517b1:::benchmark_data_smithi194_103039_object120726:head# 2024-04-29T14:21:17.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43355672:::benchmark_data_smithi194_103039_object125087:head# 2024-04-29T14:21:17.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4335a2c6:::benchmark_data_smithi194_103039_object144508:head# 2024-04-29T14:21:17.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4335c802:::benchmark_data_smithi194_103039_object116114:head# 2024-04-29T14:21:17.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4336249d:::benchmark_data_smithi194_103039_object125507:head# 2024-04-29T14:21:17.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4336aabd:::benchmark_data_smithi194_103039_object79906:head# 2024-04-29T14:21:17.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4337edad:::benchmark_data_smithi194_103039_object140813:head# 2024-04-29T14:21:17.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4339631a:::benchmark_data_smithi194_103039_object75469:head# 2024-04-29T14:21:17.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433a5408:::benchmark_data_smithi194_103039_object138442:head# 2024-04-29T14:21:17.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433ac08d:::benchmark_data_smithi194_103039_object100466:head# 2024-04-29T14:21:17.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433b1b55:::benchmark_data_smithi194_103039_object156153:head# 2024-04-29T14:21:17.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433b444c:::benchmark_data_smithi194_103039_object147168:head# 2024-04-29T14:21:17.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433cebbe:::benchmark_data_smithi194_103039_object58563:head# 2024-04-29T14:21:17.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433d52ee:::benchmark_data_smithi194_103039_object66484:head# 2024-04-29T14:21:17.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433da039:::benchmark_data_smithi194_103039_object99947:head# 2024-04-29T14:21:17.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:433e205e:::benchmark_data_smithi194_103039_object105986:head# 2024-04-29T14:21:17.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43406171:::benchmark_data_smithi194_103039_object153878:head# 2024-04-29T14:21:17.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4340a549:::benchmark_data_smithi194_103039_object65923:head# 2024-04-29T14:21:17.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4340d78e:::benchmark_data_smithi194_103039_object91622:head# 2024-04-29T14:21:17.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43414ce6:::benchmark_data_smithi194_103039_object143003:head# 2024-04-29T14:21:17.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4342083b:::benchmark_data_smithi194_103039_object150634:head# 2024-04-29T14:21:17.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43428fa9:::benchmark_data_smithi194_103039_object140092:head# 2024-04-29T14:21:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43431098:::benchmark_data_smithi194_103039_object100621:head# 2024-04-29T14:21:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43435a6a:::benchmark_data_smithi194_103039_object49085:head# 2024-04-29T14:21:17.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4344a165:::benchmark_data_smithi194_103039_object94203:head# 2024-04-29T14:21:17.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4344d700:::benchmark_data_smithi194_103039_object137516:head# 2024-04-29T14:21:17.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4345122d:::benchmark_data_smithi194_103039_object47354:head# 2024-04-29T14:21:17.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434561cc:::benchmark_data_smithi194_103039_object50677:head# 2024-04-29T14:21:17.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4345ac7c:::benchmark_data_smithi194_103039_object135600:head# 2024-04-29T14:21:17.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434691f3:::benchmark_data_smithi194_103039_object109819:head# 2024-04-29T14:21:17.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43474611:::benchmark_data_smithi194_103039_object88612:head# 2024-04-29T14:21:17.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434764ad:::benchmark_data_smithi194_103039_object145621:head# 2024-04-29T14:21:17.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43483b6d:::benchmark_data_smithi194_103039_object81804:head# 2024-04-29T14:21:17.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43488774:::benchmark_data_smithi194_103039_object51676:head# 2024-04-29T14:21:17.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4348891d:::benchmark_data_smithi194_103039_object107684:head# 2024-04-29T14:21:17.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4348ad87:::benchmark_data_smithi194_103039_object104771:head# 2024-04-29T14:21:17.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4348f975:::benchmark_data_smithi194_103039_object125295:head# 2024-04-29T14:21:17.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434937d3:::benchmark_data_smithi194_103039_object106048:head# 2024-04-29T14:21:17.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4349e90c:::benchmark_data_smithi194_103039_object112285:head# 2024-04-29T14:21:17.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434a664e:::benchmark_data_smithi194_103039_object145785:head# 2024-04-29T14:21:17.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434a6c7f:::benchmark_data_smithi194_103039_object131767:head# 2024-04-29T14:21:17.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434a727c:::benchmark_data_smithi194_103039_object89655:head# 2024-04-29T14:21:17.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434b4734:::benchmark_data_smithi194_103039_object109693:head# 2024-04-29T14:21:17.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434b4c43:::benchmark_data_smithi194_103039_object115215:head# 2024-04-29T14:21:17.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434b5005:::benchmark_data_smithi194_103039_object60203:head# 2024-04-29T14:21:17.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434b5a60:::benchmark_data_smithi194_103039_object149998:head# 2024-04-29T14:21:17.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434bfb0d:::benchmark_data_smithi194_103039_object154250:head# 2024-04-29T14:21:17.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434cd02d:::benchmark_data_smithi194_103039_object154009:head# 2024-04-29T14:21:17.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434ce47c:::benchmark_data_smithi194_103039_object92798:head# 2024-04-29T14:21:17.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434d7b4c:::benchmark_data_smithi194_103039_object73121:head# 2024-04-29T14:21:17.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:434f5452:::benchmark_data_smithi194_103039_object135262:head# 2024-04-29T14:21:17.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4350127b:::benchmark_data_smithi194_103039_object41684:head# 2024-04-29T14:21:17.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4351366b:::benchmark_data_smithi194_103039_object136380:head# 2024-04-29T14:21:17.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43525416:::benchmark_data_smithi194_103039_object118546:head# 2024-04-29T14:21:17.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4352980b:::benchmark_data_smithi194_103039_object99120:head# 2024-04-29T14:21:17.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4352a34a:::benchmark_data_smithi194_103039_object50717:head# 2024-04-29T14:21:17.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435420d9:::benchmark_data_smithi194_103039_object99541:head# 2024-04-29T14:21:17.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435447b1:::benchmark_data_smithi194_103039_object79388:head# 2024-04-29T14:21:17.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43547212:::benchmark_data_smithi194_103039_object64624:head# 2024-04-29T14:21:17.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43557e4d:::benchmark_data_smithi194_103039_object119264:head# 2024-04-29T14:21:17.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43559cc8:::benchmark_data_smithi194_103039_object138704:head# 2024-04-29T14:21:17.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43568115:::benchmark_data_smithi194_103039_object43104:head# 2024-04-29T14:21:17.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43570bf0:::benchmark_data_smithi194_103039_object83238:head# 2024-04-29T14:21:17.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43570da8:::benchmark_data_smithi194_103039_object115352:head# 2024-04-29T14:21:17.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43577845:::benchmark_data_smithi194_103039_object81205:head# 2024-04-29T14:21:17.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43583886:::benchmark_data_smithi194_103039_object139887:head# 2024-04-29T14:21:17.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43589e04:::benchmark_data_smithi194_103039_object155602:head# 2024-04-29T14:21:17.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4358e113:::benchmark_data_smithi194_103039_object155236:head# 2024-04-29T14:21:17.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43593a0f:::benchmark_data_smithi194_103039_object53136:head# 2024-04-29T14:21:17.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4359cb35:::benchmark_data_smithi194_103039_object90806:head# 2024-04-29T14:21:17.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435ac4b2:::benchmark_data_smithi194_103039_object113161:head# 2024-04-29T14:21:17.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435b709a:::benchmark_data_smithi194_103039_object48218:head# 2024-04-29T14:21:17.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435b92c1:::benchmark_data_smithi194_103039_object147082:head# 2024-04-29T14:21:17.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435bd546:::benchmark_data_smithi194_103039_object87111:head# 2024-04-29T14:21:17.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435c72a2:::benchmark_data_smithi194_103039_object119038:head# 2024-04-29T14:21:17.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435d7280:::benchmark_data_smithi194_103039_object144322:head# 2024-04-29T14:21:17.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435dd825:::benchmark_data_smithi194_103039_object149074:head# 2024-04-29T14:21:17.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435f4e18:::benchmark_data_smithi194_103039_object154343:head# 2024-04-29T14:21:17.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:435fb4a3:::benchmark_data_smithi194_103039_object66096:head# 2024-04-29T14:21:17.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43600133:::benchmark_data_smithi194_103039_object112951:head# 2024-04-29T14:21:17.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436029c0:::benchmark_data_smithi194_103039_object76444:head# 2024-04-29T14:21:17.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4361d823:::benchmark_data_smithi194_103039_object142899:head# 2024-04-29T14:21:17.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4361f5ea:::benchmark_data_smithi194_103039_object45914:head# 2024-04-29T14:21:17.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4363744a:::benchmark_data_smithi194_103039_object122961:head# 2024-04-29T14:21:17.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43646fe1:::benchmark_data_smithi194_103039_object141588:head# 2024-04-29T14:21:17.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4364796c:::benchmark_data_smithi194_103039_object123830:head# 2024-04-29T14:21:17.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4364a4c4:::benchmark_data_smithi194_103039_object75363:head# 2024-04-29T14:21:17.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4364d49c:::benchmark_data_smithi194_103039_object113369:head# 2024-04-29T14:21:17.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43651c67:::benchmark_data_smithi194_103039_object57124:head# 2024-04-29T14:21:17.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4365dfa3:::benchmark_data_smithi194_103039_object127430:head# 2024-04-29T14:21:17.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43664cbd:::benchmark_data_smithi194_103039_object79856:head# 2024-04-29T14:21:17.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4366507e:::benchmark_data_smithi194_103039_object76783:head# 2024-04-29T14:21:17.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436652c6:::benchmark_data_smithi194_103039_object50361:head# 2024-04-29T14:21:17.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43687900:::benchmark_data_smithi194_103039_object50423:head# 2024-04-29T14:21:17.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4368b071:::benchmark_data_smithi194_103039_object154569:head# 2024-04-29T14:21:17.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4369f091:::benchmark_data_smithi194_103039_object145540:head# 2024-04-29T14:21:17.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436a0c04:::benchmark_data_smithi194_103039_object148581:head# 2024-04-29T14:21:17.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436ad08b:::benchmark_data_smithi194_103039_object54575:head# 2024-04-29T14:21:17.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436c2837:::benchmark_data_smithi194_103039_object84504:head# 2024-04-29T14:21:17.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436c6a83:::benchmark_data_smithi194_103039_object96064:head# 2024-04-29T14:21:17.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436ca071:::benchmark_data_smithi194_103039_object152542:head# 2024-04-29T14:21:17.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436cdff4:::benchmark_data_smithi194_103039_object80643:head# 2024-04-29T14:21:17.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436cf726:::benchmark_data_smithi194_103039_object48677:head# 2024-04-29T14:21:17.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436cff4d:::benchmark_data_smithi194_103039_object66051:head# 2024-04-29T14:21:17.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436d268f:::benchmark_data_smithi194_103039_object93525:head# 2024-04-29T14:21:17.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436dfcd3:::benchmark_data_smithi194_103039_object72114:head# 2024-04-29T14:21:17.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436e4dfa:::benchmark_data_smithi194_103039_object102543:head# 2024-04-29T14:21:17.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436e5d32:::benchmark_data_smithi194_103039_object141345:head# 2024-04-29T14:21:17.181 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:17.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436eca04:::benchmark_data_smithi194_103039_object47665:head# 2024-04-29T14:21:17.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436ee976:::benchmark_data_smithi194_103039_object47674:head# 2024-04-29T14:21:17.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436f0942:::benchmark_data_smithi194_103039_object41788:head# 2024-04-29T14:21:17.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:436f3ab5:::benchmark_data_smithi194_103039_object78059:head# 2024-04-29T14:21:17.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4370b1d8:::benchmark_data_smithi194_103039_object94041:head# 2024-04-29T14:21:17.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437115ab:::benchmark_data_smithi194_103039_object156092:head# 2024-04-29T14:21:17.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4371f104:::benchmark_data_smithi194_103039_object64856:head# 2024-04-29T14:21:17.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43724a62:::benchmark_data_smithi194_103039_object125083:head# 2024-04-29T14:21:17.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43725922:::benchmark_data_smithi194_103039_object109344:head# 2024-04-29T14:21:17.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43729058:::benchmark_data_smithi194_103039_object156188:head# 2024-04-29T14:21:17.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4373893b:::benchmark_data_smithi194_103039_object117876:head# 2024-04-29T14:21:17.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4373ee83:::benchmark_data_smithi194_103039_object119185:head# 2024-04-29T14:21:17.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4375b87d:::benchmark_data_smithi194_103039_object145688:head# 2024-04-29T14:21:17.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43771683:::benchmark_data_smithi194_103039_object91772:head# 2024-04-29T14:21:17.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437729b3:::benchmark_data_smithi194_103039_object137827:head# 2024-04-29T14:21:17.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43784334:::benchmark_data_smithi194_103039_object53154:head# 2024-04-29T14:21:17.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4378bc57:::benchmark_data_smithi194_103039_object134932:head# 2024-04-29T14:21:17.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43794c7a:::benchmark_data_smithi194_103039_object155547:head# 2024-04-29T14:21:17.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4379c510:::benchmark_data_smithi194_103039_object126564:head# 2024-04-29T14:21:17.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4379f2cb:::benchmark_data_smithi194_103039_object74143:head# 2024-04-29T14:21:17.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437a19a6:::benchmark_data_smithi194_103039_object152954:head# 2024-04-29T14:21:17.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437a1e31:::benchmark_data_smithi194_103039_object52043:head# 2024-04-29T14:21:17.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437c3db9:::benchmark_data_smithi194_103039_object86995:head# 2024-04-29T14:21:17.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437c9bbb:::benchmark_data_smithi194_103039_object102528:head# 2024-04-29T14:21:17.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437caa98:::benchmark_data_smithi194_103039_object63362:head# 2024-04-29T14:21:17.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437d726d:::benchmark_data_smithi194_103039_object82764:head# 2024-04-29T14:21:17.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437d7870:::benchmark_data_smithi194_103039_object58895:head# 2024-04-29T14:21:17.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437e7d7b:::benchmark_data_smithi194_103039_object51383:head# 2024-04-29T14:21:17.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437f0cb3:::benchmark_data_smithi194_103039_object81252:head# 2024-04-29T14:21:17.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:437f974f:::benchmark_data_smithi194_103039_object144880:head# 2024-04-29T14:21:17.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43800d5a:::benchmark_data_smithi194_103039_object57159:head# 2024-04-29T14:21:17.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4380bd5a:::benchmark_data_smithi194_103039_object101160:head# 2024-04-29T14:21:17.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4381ccf6:::benchmark_data_smithi194_103039_object129131:head# 2024-04-29T14:21:17.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43825b09:::benchmark_data_smithi194_103039_object98295:head# 2024-04-29T14:21:17.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43829640:::benchmark_data_smithi194_103039_object82977:head# 2024-04-29T14:21:17.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4382c5c9:::benchmark_data_smithi194_103039_object89268:head# 2024-04-29T14:21:17.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4383316f:::benchmark_data_smithi194_103039_object127353:head# 2024-04-29T14:21:17.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43841cbf:::benchmark_data_smithi194_103039_object114385:head# 2024-04-29T14:21:17.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43847ea0:::benchmark_data_smithi194_103039_object149799:head# 2024-04-29T14:21:17.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4384c682:::benchmark_data_smithi194_103039_object47320:head# 2024-04-29T14:21:17.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43858837:::benchmark_data_smithi194_103039_object92709:head# 2024-04-29T14:21:17.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4385b151:::benchmark_data_smithi194_103039_object142855:head# 2024-04-29T14:21:17.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4386643e:::benchmark_data_smithi194_103039_object80660:head# 2024-04-29T14:21:17.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43866daa:::benchmark_data_smithi194_103039_object45521:head# 2024-04-29T14:21:17.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4386e37b:::benchmark_data_smithi194_103039_object55685:head# 2024-04-29T14:21:17.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4386e43b:::benchmark_data_smithi194_103039_object124953:head# 2024-04-29T14:21:17.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438799d4:::benchmark_data_smithi194_103039_object83329:head# 2024-04-29T14:21:17.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4387c19c:::benchmark_data_smithi194_103039_object144934:head# 2024-04-29T14:21:17.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4387d455:::benchmark_data_smithi194_103039_object91477:head# 2024-04-29T14:21:17.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4387e36f:::benchmark_data_smithi194_103039_object110297:head# 2024-04-29T14:21:17.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4389805b:::benchmark_data_smithi194_103039_object115236:head# 2024-04-29T14:21:17.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4389c343:::benchmark_data_smithi194_103039_object137723:head# 2024-04-29T14:21:17.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438a5025:::benchmark_data_smithi194_103039_object59887:head# 2024-04-29T14:21:17.207 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:17 smithi194 ceph-mon[44089]: pgmap v981: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 181 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:17.207 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/141547992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:17.207 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:17.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438ac729:::benchmark_data_smithi194_103039_object42012:head# 2024-04-29T14:21:17.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438ae80d:::benchmark_data_smithi194_103039_object42318:head# 2024-04-29T14:21:17.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438b7a45:::benchmark_data_smithi194_103039_object99251:head# 2024-04-29T14:21:17.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438b8da3:::benchmark_data_smithi194_103039_object59716:head# 2024-04-29T14:21:17.209 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:17.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438be14c:::benchmark_data_smithi194_103039_object138979:head# 2024-04-29T14:21:17.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438c29da:::benchmark_data_smithi194_103039_object89009:head# 2024-04-29T14:21:17.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438c4cd2:::benchmark_data_smithi194_103039_object58057:head# 2024-04-29T14:21:17.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438c901f:::benchmark_data_smithi194_103039_object111750:head# 2024-04-29T14:21:17.211 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:17.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438d26d9:::benchmark_data_smithi194_103039_object48246:head# 2024-04-29T14:21:17.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438ded59:::benchmark_data_smithi194_103039_object48508:head# 2024-04-29T14:21:17.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438e16a7:::benchmark_data_smithi194_103039_object115814:head# 2024-04-29T14:21:17.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438e9f66:::benchmark_data_smithi194_103039_object91261:head# 2024-04-29T14:21:17.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438ee7db:::benchmark_data_smithi194_103039_object57129:head# 2024-04-29T14:21:17.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438f142b:::benchmark_data_smithi194_103039_object151596:head# 2024-04-29T14:21:17.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438f3d27:::benchmark_data_smithi194_103039_object47563:head# 2024-04-29T14:21:17.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438f7a6c:::benchmark_data_smithi194_103039_object58702:head# 2024-04-29T14:21:17.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438ffa83:::benchmark_data_smithi194_103039_object85962:head# 2024-04-29T14:21:17.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:438ffeb2:::benchmark_data_smithi194_103039_object60197:head# 2024-04-29T14:21:17.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43901a24:::benchmark_data_smithi194_103039_object100692:head# 2024-04-29T14:21:17.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4390d5d7:::benchmark_data_smithi194_103039_object106643:head# 2024-04-29T14:21:17.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:439173f9:::benchmark_data_smithi194_103039_object137317:head# 2024-04-29T14:21:17.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43918318:::benchmark_data_smithi194_103039_object145580:head# 2024-04-29T14:21:17.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:439213f1:::benchmark_data_smithi194_103039_object82213:head# 2024-04-29T14:21:17.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4392410e:::benchmark_data_smithi194_103039_object121664:head# 2024-04-29T14:21:17.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:439459a1:::benchmark_data_smithi194_103039_object149596:head# 2024-04-29T14:21:17.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43948dc2:::benchmark_data_smithi194_103039_object97859:head# 2024-04-29T14:21:17.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43949572:::benchmark_data_smithi194_103039_object59352:head# 2024-04-29T14:21:17.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43949f69:::benchmark_data_smithi194_103039_object134719:head# 2024-04-29T14:21:17.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4394b194:::benchmark_data_smithi194_103039_object99525:head# 2024-04-29T14:21:17.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4395aed9:::benchmark_data_smithi194_103039_object62342:head# 2024-04-29T14:21:17.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43960562:::benchmark_data_smithi194_103039_object65944:head# 2024-04-29T14:21:17.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43961d83:::benchmark_data_smithi194_103039_object97593:head# 2024-04-29T14:21:17.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4397a2a2:::benchmark_data_smithi194_103039_object114422:head# 2024-04-29T14:21:17.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43989e90:::benchmark_data_smithi194_103039_object116190:head# 2024-04-29T14:21:17.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4399297e:::benchmark_data_smithi194_103039_object137659:head# 2024-04-29T14:21:17.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43995ad0:::benchmark_data_smithi194_103039_object47586:head# 2024-04-29T14:21:17.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4399e4e3:::benchmark_data_smithi194_103039_object104182:head# 2024-04-29T14:21:17.225 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:439a6fa9:::benchmark_data_smithi194_103039_object150541:head# 2024-04-29T14:21:17.225 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:439b3de9:::benchmark_data_smithi194_103039_object78078:head# 2024-04-29T14:21:17.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:439d016b:::benchmark_data_smithi194_103039_object122889:head# 2024-04-29T14:21:17.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a0966e:::benchmark_data_smithi194_103039_object102650:head# 2024-04-29T14:21:17.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a31199:::benchmark_data_smithi194_103039_object56108:head# 2024-04-29T14:21:17.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a31ec1:::benchmark_data_smithi194_103039_object107277:head# 2024-04-29T14:21:17.228 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a36b3a:::benchmark_data_smithi194_103039_object42734:head# 2024-04-29T14:21:17.228 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a42dc2:::benchmark_data_smithi194_103039_object136485:head# 2024-04-29T14:21:17.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a52d77:::benchmark_data_smithi194_103039_object91832:head# 2024-04-29T14:21:17.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a55ca8:::benchmark_data_smithi194_103039_object68286:head# 2024-04-29T14:21:17.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a6431a:::benchmark_data_smithi194_103039_object43848:head# 2024-04-29T14:21:17.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a7b5e0:::benchmark_data_smithi194_103039_object77712:head# 2024-04-29T14:21:17.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a7d0aa:::benchmark_data_smithi194_103039_object60530:head# 2024-04-29T14:21:17.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a7d508:::benchmark_data_smithi194_103039_object116305:head# 2024-04-29T14:21:17.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a7fa34:::benchmark_data_smithi194_103039_object99992:head# 2024-04-29T14:21:17.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a80eea:::benchmark_data_smithi194_103039_object148634:head# 2024-04-29T14:21:17.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a898ba:::benchmark_data_smithi194_103039_object116774:head# 2024-04-29T14:21:17.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43a9c740:::benchmark_data_smithi194_103039_object95638:head# 2024-04-29T14:21:17.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43abe25d:::benchmark_data_smithi194_103039_object132438:head# 2024-04-29T14:21:17.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ac4767:::benchmark_data_smithi194_103039_object116484:head# 2024-04-29T14:21:17.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43acaf46:::benchmark_data_smithi194_103039_object125585:head# 2024-04-29T14:21:17.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43adf906:::benchmark_data_smithi194_103039_object87282:head# 2024-04-29T14:21:17.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ae6c34:::benchmark_data_smithi194_103039_object141533:head# 2024-04-29T14:21:17.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43aee911:::benchmark_data_smithi194_103039_object96094:head# 2024-04-29T14:21:17.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43aeed5e:::benchmark_data_smithi194_103039_object152788:head# 2024-04-29T14:21:17.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43af2867:::benchmark_data_smithi194_103039_object152475:head# 2024-04-29T14:21:17.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43af2d21:::benchmark_data_smithi194_103039_object107832:head# 2024-04-29T14:21:17.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43af4fc7:::benchmark_data_smithi194_103039_object124161:head# 2024-04-29T14:21:17.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43afc11c:::benchmark_data_smithi194_103039_object125705:head# 2024-04-29T14:21:17.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43afce1c:::benchmark_data_smithi194_103039_object77345:head# 2024-04-29T14:21:17.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b069bf:::benchmark_data_smithi194_103039_object90826:head# 2024-04-29T14:21:17.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b0e0e6:::benchmark_data_smithi194_103039_object42403:head# 2024-04-29T14:21:17.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b15bcd:::benchmark_data_smithi194_103039_object84581:head# 2024-04-29T14:21:17.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b194a6:::benchmark_data_smithi194_103039_object96545:head# 2024-04-29T14:21:17.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b24d9e:::benchmark_data_smithi194_103039_object111895:head# 2024-04-29T14:21:17.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b32b81:::benchmark_data_smithi194_103039_object101926:head# 2024-04-29T14:21:17.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b37984:::benchmark_data_smithi194_103039_object150431:head# 2024-04-29T14:21:17.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b4b457:::benchmark_data_smithi194_103039_object101138:head# 2024-04-29T14:21:17.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b6ad4c:::benchmark_data_smithi194_103039_object118895:head# 2024-04-29T14:21:17.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b6d498:::benchmark_data_smithi194_103039_object67436:head# 2024-04-29T14:21:17.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b74036:::benchmark_data_smithi194_103039_object99714:head# 2024-04-29T14:21:17.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b7f61f:::benchmark_data_smithi194_103039_object126005:head# 2024-04-29T14:21:17.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b7fb71:::benchmark_data_smithi194_103039_object57203:head# 2024-04-29T14:21:17.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b83d07:::benchmark_data_smithi194_103039_object48148:head# 2024-04-29T14:21:17.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b87039:::benchmark_data_smithi194_103039_object141051:head# 2024-04-29T14:21:17.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b8b9e6:::benchmark_data_smithi194_103039_object126556:head# 2024-04-29T14:21:17.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b92da2:::benchmark_data_smithi194_103039_object44095:head# 2024-04-29T14:21:17.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43b9a0fd:::benchmark_data_smithi194_103039_object63188:head# 2024-04-29T14:21:17.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43baa6e0:::benchmark_data_smithi194_103039_object77701:head# 2024-04-29T14:21:17.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43bb1e39:::benchmark_data_smithi194_103039_object105421:head# 2024-04-29T14:21:17.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43bc63a5:::benchmark_data_smithi194_103039_object47685:head# 2024-04-29T14:21:17.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43be05c1:::benchmark_data_smithi194_103039_object140273:head# 2024-04-29T14:21:17.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43be4e05:::benchmark_data_smithi194_103039_object94220:head# 2024-04-29T14:21:17.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43be9f1f:::benchmark_data_smithi194_103039_object67110:head# 2024-04-29T14:21:17.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43bf5222:::benchmark_data_smithi194_103039_object49070:head# 2024-04-29T14:21:17.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c2136a:::benchmark_data_smithi194_103039_object117110:head# 2024-04-29T14:21:17.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c271c7:::benchmark_data_smithi194_103039_object44270:head# 2024-04-29T14:21:17.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c28198:::benchmark_data_smithi194_103039_object61677:head# 2024-04-29T14:21:17.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c38905:::benchmark_data_smithi194_103039_object103122:head# 2024-04-29T14:21:17.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c40444:::benchmark_data_smithi194_103039_object152742:head# 2024-04-29T14:21:17.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c41e56:::benchmark_data_smithi194_103039_object131978:head# 2024-04-29T14:21:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c4541b:::benchmark_data_smithi194_103039_object70521:head# 2024-04-29T14:21:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c51472:::benchmark_data_smithi194_103039_object84662:head# 2024-04-29T14:21:17.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c581ef:::benchmark_data_smithi194_103039_object131396:head# 2024-04-29T14:21:17.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c5dff7:::benchmark_data_smithi194_103039_object148360:head# 2024-04-29T14:21:17.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c61457:::benchmark_data_smithi194_103039_object138953:head# 2024-04-29T14:21:17.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c6fc25:::benchmark_data_smithi194_103039_object70191:head# 2024-04-29T14:21:17.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c7c330:::benchmark_data_smithi194_103039_object103608:head# 2024-04-29T14:21:17.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c7f586:::benchmark_data_smithi194_103039_object140287:head# 2024-04-29T14:21:17.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c92a2a:::benchmark_data_smithi194_103039_object150902:head# 2024-04-29T14:21:17.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c944a7:::benchmark_data_smithi194_103039_object74229:head# 2024-04-29T14:21:17.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c97b23:::benchmark_data_smithi194_103039_object127359:head# 2024-04-29T14:21:17.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43c9d5cb:::benchmark_data_smithi194_103039_object143729:head# 2024-04-29T14:21:17.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43cacbeb:::benchmark_data_smithi194_103039_object73524:head# 2024-04-29T14:21:17.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43cb1ce9:::benchmark_data_smithi194_103039_object118368:head# 2024-04-29T14:21:17.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43cbc043:::benchmark_data_smithi194_103039_object52938:head# 2024-04-29T14:21:17.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43cccb21:::benchmark_data_smithi194_103039_object54953:head# 2024-04-29T14:21:17.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ccd85d:::benchmark_data_smithi194_103039_object125713:head# 2024-04-29T14:21:17.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43cdf29c:::benchmark_data_smithi194_103039_object138259:head# 2024-04-29T14:21:17.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43cff5b5:::benchmark_data_smithi194_103039_object153268:head# 2024-04-29T14:21:17.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d1ca6c:::benchmark_data_smithi194_103039_object142173:head# 2024-04-29T14:21:17.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d236a6:::benchmark_data_smithi194_103039_object149162:head# 2024-04-29T14:21:17.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d2acbb:::benchmark_data_smithi194_103039_object60215:head# 2024-04-29T14:21:17.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d39b1e:::benchmark_data_smithi194_103039_object83620:head# 2024-04-29T14:21:17.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d3da3f:::benchmark_data_smithi194_103039_object86709:head# 2024-04-29T14:21:17.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d406a2:::benchmark_data_smithi194_103039_object146643:head# 2024-04-29T14:21:17.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d42b28:::benchmark_data_smithi194_103039_object153758:head# 2024-04-29T14:21:17.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d458a1:::benchmark_data_smithi194_103039_object87569:head# 2024-04-29T14:21:17.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d4cce3:::benchmark_data_smithi194_103039_object47828:head# 2024-04-29T14:21:17.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d4de80:::benchmark_data_smithi194_103039_object136989:head# 2024-04-29T14:21:17.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d64b37:::benchmark_data_smithi194_103039_object149348:head# 2024-04-29T14:21:17.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d64be7:::benchmark_data_smithi194_103039_object137636:head# 2024-04-29T14:21:17.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d67718:::benchmark_data_smithi194_103039_object134674:head# 2024-04-29T14:21:17.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d6a5da:::benchmark_data_smithi194_103039_object154003:head# 2024-04-29T14:21:17.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d6b78b:::benchmark_data_smithi194_103039_object128200:head# 2024-04-29T14:21:17.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d7457c:::benchmark_data_smithi194_103039_object93873:head# 2024-04-29T14:21:17.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d74813:::benchmark_data_smithi194_103039_object156214:head# 2024-04-29T14:21:17.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d78ea9:::benchmark_data_smithi194_103039_object142692:head# 2024-04-29T14:21:17.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d7c32e:::benchmark_data_smithi194_103039_object82086:head# 2024-04-29T14:21:17.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d7e3a2:::benchmark_data_smithi194_103039_object86893:head# 2024-04-29T14:21:17.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d86814:::benchmark_data_smithi194_103039_object45068:head# 2024-04-29T14:21:17.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43d9a2d0:::benchmark_data_smithi194_103039_object130661:head# 2024-04-29T14:21:17.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43dab923:::benchmark_data_smithi194_103039_object137078:head# 2024-04-29T14:21:17.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43db003a:::benchmark_data_smithi194_103039_object135219:head# 2024-04-29T14:21:17.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43dc00c8:::benchmark_data_smithi194_103039_object42385:head# 2024-04-29T14:21:17.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43dc0470:::benchmark_data_smithi194_103039_object104432:head# 2024-04-29T14:21:17.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43dcc553:::benchmark_data_smithi194_103039_object134247:head# 2024-04-29T14:21:17.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ddc1fa:::benchmark_data_smithi194_103039_object111861:head# 2024-04-29T14:21:17.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ddee33:::benchmark_data_smithi194_103039_object89397:head# 2024-04-29T14:21:17.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43de1132:::benchmark_data_smithi194_103039_object136630:head# 2024-04-29T14:21:17.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43de4396:::benchmark_data_smithi194_103039_object125466:head# 2024-04-29T14:21:17.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43deb171:::benchmark_data_smithi194_103039_object55367:head# 2024-04-29T14:21:17.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43df250e:::benchmark_data_smithi194_103039_object145357:head# 2024-04-29T14:21:17.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43df60ee:::benchmark_data_smithi194_103039_object97632:head# 2024-04-29T14:21:17.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43dfdbfc:::benchmark_data_smithi194_103039_object104583:head# 2024-04-29T14:21:17.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e06184:::benchmark_data_smithi194_103039_object148473:head# 2024-04-29T14:21:17.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e0e770:::benchmark_data_smithi194_103039_object107537:head# 2024-04-29T14:21:17.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e0e946:::benchmark_data_smithi194_103039_object112008:head# 2024-04-29T14:21:17.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e15965:::benchmark_data_smithi194_103039_object136198:head# 2024-04-29T14:21:17.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e1e940:::benchmark_data_smithi194_103039_object108544:head# 2024-04-29T14:21:17.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e26f98:::benchmark_data_smithi194_103039_object77378:head# 2024-04-29T14:21:17.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e2855f:::benchmark_data_smithi194_103039_object105457:head# 2024-04-29T14:21:17.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e31884:::benchmark_data_smithi194_103039_object89760:head# 2024-04-29T14:21:17.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e48bae:::benchmark_data_smithi194_103039_object113350:head# 2024-04-29T14:21:17.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e65ff5:::benchmark_data_smithi194_103039_object113056:head# 2024-04-29T14:21:17.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e676a3:::benchmark_data_smithi194_103039_object86168:head# 2024-04-29T14:21:17.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e6d496:::benchmark_data_smithi194_103039_object61713:head# 2024-04-29T14:21:17.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43e89237:::benchmark_data_smithi194_103039_object48413:head# 2024-04-29T14:21:17.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ea9787:::benchmark_data_smithi194_103039_object108650:head# 2024-04-29T14:21:17.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ead394:::benchmark_data_smithi194_103039_object113354:head# 2024-04-29T14:21:17.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43eaeb4d:::benchmark_data_smithi194_103039_object46549:head# 2024-04-29T14:21:17.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ec1ee7:::benchmark_data_smithi194_103039_object94464:head# 2024-04-29T14:21:17.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ec5866:::benchmark_data_smithi194_103039_object99539:head# 2024-04-29T14:21:17.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ec5aa8:::benchmark_data_smithi194_103039_object149341:head# 2024-04-29T14:21:17.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ed1f61:::benchmark_data_smithi194_103039_object98897:head# 2024-04-29T14:21:17.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ed320b:::benchmark_data_smithi194_103039_object92473:head# 2024-04-29T14:21:17.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ed5ac3:::benchmark_data_smithi194_103039_object97870:head# 2024-04-29T14:21:17.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43edbb33:::benchmark_data_smithi194_103039_object65287:head# 2024-04-29T14:21:17.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43ef19dc:::benchmark_data_smithi194_103039_object91863:head# 2024-04-29T14:21:17.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f03df0:::benchmark_data_smithi194_103039_object120717:head# 2024-04-29T14:21:17.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f0b326:::benchmark_data_smithi194_103039_object152139:head# 2024-04-29T14:21:17.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f11d6a:::benchmark_data_smithi194_103039_object83510:head# 2024-04-29T14:21:17.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f2274c:::benchmark_data_smithi194_103039_object85575:head# 2024-04-29T14:21:17.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f242aa:::benchmark_data_smithi194_103039_object90246:head# 2024-04-29T14:21:17.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f2e6cc:::benchmark_data_smithi194_103039_object49390:head# 2024-04-29T14:21:17.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f3ddfd:::benchmark_data_smithi194_103039_object51209:head# 2024-04-29T14:21:17.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f4e496:::benchmark_data_smithi194_103039_object108811:head# 2024-04-29T14:21:17.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f682e8:::benchmark_data_smithi194_103039_object134172:head# 2024-04-29T14:21:17.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f749a1:::benchmark_data_smithi194_103039_object81254:head# 2024-04-29T14:21:17.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f7c7a3:::benchmark_data_smithi194_103039_object62441:head# 2024-04-29T14:21:17.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f884a3:::benchmark_data_smithi194_103039_object120889:head# 2024-04-29T14:21:17.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f8879a:::benchmark_data_smithi194_103039_object51039:head# 2024-04-29T14:21:17.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43f88ca2:::benchmark_data_smithi194_103039_object48338:head# 2024-04-29T14:21:17.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fa2aa3:::benchmark_data_smithi194_103039_object133244:head# 2024-04-29T14:21:17.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fb48c5:::benchmark_data_smithi194_103039_object89247:head# 2024-04-29T14:21:17.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fb5e74:::benchmark_data_smithi194_103039_object133585:head# 2024-04-29T14:21:17.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fbc543:::benchmark_data_smithi194_103039_object127811:head# 2024-04-29T14:21:17.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fc0d42:::benchmark_data_smithi194_103039_object60483:head# 2024-04-29T14:21:17.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fc5347:::benchmark_data_smithi194_103039_object88708:head# 2024-04-29T14:21:17.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fcb750:::benchmark_data_smithi194_103039_object96788:head# 2024-04-29T14:21:17.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fcf4b1:::benchmark_data_smithi194_103039_object136428:head# 2024-04-29T14:21:17.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fd39db:::benchmark_data_smithi194_103039_object129303:head# 2024-04-29T14:21:17.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:43fe57bc:::benchmark_data_smithi194_103039_object139209:head# 2024-04-29T14:21:17.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4400fa29:::benchmark_data_smithi194_103039_object56352:head# 2024-04-29T14:21:17.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4400fb69:::benchmark_data_smithi194_103039_object71769:head# 2024-04-29T14:21:17.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44011d0c:::benchmark_data_smithi194_103039_object46489:head# 2024-04-29T14:21:17.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4401ddf2:::benchmark_data_smithi194_103039_object81506:head# 2024-04-29T14:21:17.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44045a10:::benchmark_data_smithi194_103039_object68234:head# 2024-04-29T14:21:17.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44048aa4:::benchmark_data_smithi194_103039_object146310:head# 2024-04-29T14:21:17.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44049970:::benchmark_data_smithi194_103039_object151005:head# 2024-04-29T14:21:17.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4405656e:::benchmark_data_smithi194_103039_object125846:head# 2024-04-29T14:21:17.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44059bd8:::benchmark_data_smithi194_103039_object42595:head# 2024-04-29T14:21:17.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4405ecde:::benchmark_data_smithi194_103039_object97889:head# 2024-04-29T14:21:17.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44071d2b:::benchmark_data_smithi194_103039_object47730:head# 2024-04-29T14:21:17.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4407435d:::benchmark_data_smithi194_103039_object86323:head# 2024-04-29T14:21:17.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44083385:::benchmark_data_smithi194_103039_object134857:head# 2024-04-29T14:21:17.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44087a25:::benchmark_data_smithi194_103039_object120929:head# 2024-04-29T14:21:17.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4408afb3:::benchmark_data_smithi194_103039_object67594:head# 2024-04-29T14:21:17.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4408c1d7:::benchmark_data_smithi194_103039_object118400:head# 2024-04-29T14:21:17.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44091365:::benchmark_data_smithi194_103039_object43430:head# 2024-04-29T14:21:17.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440a1c61:::benchmark_data_smithi194_103039_object114777:head# 2024-04-29T14:21:17.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440aca2a:::benchmark_data_smithi194_103039_object145620:head# 2024-04-29T14:21:17.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440c5956:::benchmark_data_smithi194_103039_object76560:head# 2024-04-29T14:21:17.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440d700f:::benchmark_data_smithi194_103039_object84347:head# 2024-04-29T14:21:17.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440d9de3:::benchmark_data_smithi194_103039_object152604:head# 2024-04-29T14:21:17.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440e0797:::benchmark_data_smithi194_103039_object134636:head# 2024-04-29T14:21:17.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440e0cd3:::benchmark_data_smithi194_103039_object44388:head# 2024-04-29T14:21:17.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:440ef959:::benchmark_data_smithi194_103039_object71069:head# 2024-04-29T14:21:17.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441007bb:::benchmark_data_smithi194_103039_object130368:head# 2024-04-29T14:21:17.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441096b9:::benchmark_data_smithi194_103039_object116424:head# 2024-04-29T14:21:17.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4410a228:::benchmark_data_smithi194_103039_object85497:head# 2024-04-29T14:21:17.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4410d4e6:::benchmark_data_smithi194_103039_object138407:head# 2024-04-29T14:21:17.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4410d8dd:::benchmark_data_smithi194_103039_object87080:head# 2024-04-29T14:21:17.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441123f9:::benchmark_data_smithi194_103039_object129742:head# 2024-04-29T14:21:17.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44113cfb:::benchmark_data_smithi194_103039_object93314:head# 2024-04-29T14:21:17.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4411adf2:::benchmark_data_smithi194_103039_object129737:head# 2024-04-29T14:21:17.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4411dff9:::benchmark_data_smithi194_103039_object77163:head# 2024-04-29T14:21:17.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44120b23:::benchmark_data_smithi194_103039_object125175:head# 2024-04-29T14:21:17.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4413ddfb:::benchmark_data_smithi194_103039_object132267:head# 2024-04-29T14:21:17.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4414cc28:::benchmark_data_smithi194_103039_object77027:head# 2024-04-29T14:21:17.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44157358:::benchmark_data_smithi194_103039_object124369:head# 2024-04-29T14:21:17.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4415c400:::benchmark_data_smithi194_103039_object102817:head# 2024-04-29T14:21:17.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4416a011:::benchmark_data_smithi194_103039_object78831:head# 2024-04-29T14:21:17.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4417ebc9:::benchmark_data_smithi194_103039_object61254:head# 2024-04-29T14:21:17.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44181c03:::benchmark_data_smithi194_103039_object127125:head# 2024-04-29T14:21:17.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44192482:::benchmark_data_smithi194_103039_object63356:head# 2024-04-29T14:21:17.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4419683b:::benchmark_data_smithi194_103039_object98304:head# 2024-04-29T14:21:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4419c9fb:::benchmark_data_smithi194_103039_object110977:head# 2024-04-29T14:21:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4419e70e:::benchmark_data_smithi194_103039_object46352:head# 2024-04-29T14:21:17.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441a56aa:::benchmark_data_smithi194_103039_object131782:head# 2024-04-29T14:21:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441c2635:::benchmark_data_smithi194_103039_object151501:head# 2024-04-29T14:21:17.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441d5ad5:::benchmark_data_smithi194_103039_object42989:head# 2024-04-29T14:21:17.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441dbda7:::benchmark_data_smithi194_103039_object148555:head# 2024-04-29T14:21:17.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441e2bca:::benchmark_data_smithi194_103039_object63444:head# 2024-04-29T14:21:17.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:441f3a16:::benchmark_data_smithi194_103039_object72666:head# 2024-04-29T14:21:17.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44214f78:::benchmark_data_smithi194_103039_object128786:head# 2024-04-29T14:21:17.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44225907:::benchmark_data_smithi194_103039_object61860:head# 2024-04-29T14:21:17.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4422630f:::benchmark_data_smithi194_103039_object58772:head# 2024-04-29T14:21:17.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4422f7c8:::benchmark_data_smithi194_103039_object71491:head# 2024-04-29T14:21:17.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44230d25:::benchmark_data_smithi194_103039_object107381:head# 2024-04-29T14:21:17.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4424e4b6:::benchmark_data_smithi194_103039_object62203:head# 2024-04-29T14:21:17.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44256d76:::benchmark_data_smithi194_103039_object64758:head# 2024-04-29T14:21:17.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4425ed26:::benchmark_data_smithi194_103039_object79950:head# 2024-04-29T14:21:17.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4426faf0:::benchmark_data_smithi194_103039_object129465:head# 2024-04-29T14:21:17.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44271e78:::benchmark_data_smithi194_103039_object69275:head# 2024-04-29T14:21:17.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4427ceed:::benchmark_data_smithi194_103039_object106080:head# 2024-04-29T14:21:17.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4427e499:::benchmark_data_smithi194_103039_object110529:head# 2024-04-29T14:21:17.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44289d04:::benchmark_data_smithi194_103039_object63692:head# 2024-04-29T14:21:17.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4428c394:::benchmark_data_smithi194_103039_object70511:head# 2024-04-29T14:21:17.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4428c459:::benchmark_data_smithi194_103039_object51452:head# 2024-04-29T14:21:17.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4429a1da:::benchmark_data_smithi194_103039_object146238:head# 2024-04-29T14:21:17.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442a2f50:::benchmark_data_smithi194_103039_object51137:head# 2024-04-29T14:21:17.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442a6946:::benchmark_data_smithi194_103039_object135454:head# 2024-04-29T14:21:17.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442b0043:::benchmark_data_smithi194_103039_object75548:head# 2024-04-29T14:21:17.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442b7843:::benchmark_data_smithi194_103039_object123360:head# 2024-04-29T14:21:17.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442b92af:::benchmark_data_smithi194_103039_object86001:head# 2024-04-29T14:21:17.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442cd472:::benchmark_data_smithi194_103039_object64708:head# 2024-04-29T14:21:17.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442efc03:::benchmark_data_smithi194_103039_object138464:head# 2024-04-29T14:21:17.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442f30ee:::benchmark_data_smithi194_103039_object73379:head# 2024-04-29T14:21:17.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:442fa6ea:::benchmark_data_smithi194_103039_object86646:head# 2024-04-29T14:21:17.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44300c27:::benchmark_data_smithi194_103039_object87454:head# 2024-04-29T14:21:17.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4430a299:::benchmark_data_smithi194_103039_object87048:head# 2024-04-29T14:21:17.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4432ae5e:::benchmark_data_smithi194_103039_object129041:head# 2024-04-29T14:21:17.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44334989:::benchmark_data_smithi194_103039_object130372:head# 2024-04-29T14:21:17.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4433fe11:::benchmark_data_smithi194_103039_object64731:head# 2024-04-29T14:21:17.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44343edb:::benchmark_data_smithi194_103039_object58014:head# 2024-04-29T14:21:17.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4434f095:::benchmark_data_smithi194_103039_object59810:head# 2024-04-29T14:21:17.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44350ae6:::benchmark_data_smithi194_103039_object130139:head# 2024-04-29T14:21:17.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443536cb:::benchmark_data_smithi194_103039_object136486:head# 2024-04-29T14:21:17.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44353dd8:::benchmark_data_smithi194_103039_object104353:head# 2024-04-29T14:21:17.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4435533a:::benchmark_data_smithi194_103039_object43016:head# 2024-04-29T14:21:17.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44358b13:::benchmark_data_smithi194_103039_object47271:head# 2024-04-29T14:21:17.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4435b8b1:::benchmark_data_smithi194_103039_object75060:head# 2024-04-29T14:21:17.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4435f01b:::benchmark_data_smithi194_103039_object80467:head# 2024-04-29T14:21:17.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44365154:::benchmark_data_smithi194_103039_object56616:head# 2024-04-29T14:21:17.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4436da5c:::benchmark_data_smithi194_103039_object115427:head# 2024-04-29T14:21:17.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44392def:::benchmark_data_smithi194_103039_object117638:head# 2024-04-29T14:21:17.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44399c1e:::benchmark_data_smithi194_103039_object101468:head# 2024-04-29T14:21:17.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443a2436:::benchmark_data_smithi194_103039_object142763:head# 2024-04-29T14:21:17.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443a31b9:::benchmark_data_smithi194_103039_object63959:head# 2024-04-29T14:21:17.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443b369e:::benchmark_data_smithi194_103039_object102339:head# 2024-04-29T14:21:17.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443c4116:::benchmark_data_smithi194_103039_object154465:head# 2024-04-29T14:21:17.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443cb8d9:::benchmark_data_smithi194_103039_object45699:head# 2024-04-29T14:21:17.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443d8240:::benchmark_data_smithi194_103039_object103374:head# 2024-04-29T14:21:17.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443dcece:::benchmark_data_smithi194_103039_object129654:head# 2024-04-29T14:21:17.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443e4b6e:::benchmark_data_smithi194_103039_object113183:head# 2024-04-29T14:21:17.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443ec3a1:::benchmark_data_smithi194_103039_object84294:head# 2024-04-29T14:21:17.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443ed53a:::benchmark_data_smithi194_103039_object126615:head# 2024-04-29T14:21:17.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443ef481:::benchmark_data_smithi194_103039_object75603:head# 2024-04-29T14:21:17.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:443fab22:::benchmark_data_smithi194_103039_object52003:head# 2024-04-29T14:21:17.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44400162:::benchmark_data_smithi194_103039_object122730:head# 2024-04-29T14:21:17.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44400eeb:::benchmark_data_smithi194_103039_object55503:head# 2024-04-29T14:21:17.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44402b5b:::benchmark_data_smithi194_103039_object68005:head# 2024-04-29T14:21:17.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444089aa:::benchmark_data_smithi194_103039_object133199:head# 2024-04-29T14:21:17.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44425738:::benchmark_data_smithi194_103039_object80670:head# 2024-04-29T14:21:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44429ae0:::benchmark_data_smithi194_103039_object74416:head# 2024-04-29T14:21:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4442b8da:::benchmark_data_smithi194_103039_object120631:head# 2024-04-29T14:21:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44431ba1:::benchmark_data_smithi194_103039_object94082:head# 2024-04-29T14:21:17.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4443df2b:::benchmark_data_smithi194_103039_object122651:head# 2024-04-29T14:21:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44453a06:::benchmark_data_smithi194_103039_object115131:head# 2024-04-29T14:21:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44453c9c:::benchmark_data_smithi194_103039_object104265:head# 2024-04-29T14:21:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4445a654:::benchmark_data_smithi194_103039_object127342:head# 2024-04-29T14:21:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44460906:::benchmark_data_smithi194_103039_object69613:head# 2024-04-29T14:21:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44463986:::benchmark_data_smithi194_103039_object55262:head# 2024-04-29T14:21:17.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4446d2a2:::benchmark_data_smithi194_103039_object129444:head# 2024-04-29T14:21:17.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44470c09:::benchmark_data_smithi194_103039_object151533:head# 2024-04-29T14:21:17.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4447cc8d:::benchmark_data_smithi194_103039_object138764:head# 2024-04-29T14:21:17.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44486e39:::benchmark_data_smithi194_103039_object141216:head# 2024-04-29T14:21:17.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44489a36:::benchmark_data_smithi194_103039_object98866:head# 2024-04-29T14:21:17.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44498250:::benchmark_data_smithi194_103039_object50727:head# 2024-04-29T14:21:17.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4449ddc1:::benchmark_data_smithi194_103039_object149647:head# 2024-04-29T14:21:17.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4449e32a:::benchmark_data_smithi194_103039_object128304:head# 2024-04-29T14:21:17.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444a501c:::benchmark_data_smithi194_103039_object48791:head# 2024-04-29T14:21:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:17 smithi040 ceph-mon[39581]: pgmap v981: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 181 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/141547992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:17.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444b061f:::benchmark_data_smithi194_103039_object57210:head# 2024-04-29T14:21:17.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444c7c46:::benchmark_data_smithi194_103039_object155421:head# 2024-04-29T14:21:17.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444c7cfc:::benchmark_data_smithi194_103039_object82783:head# 2024-04-29T14:21:17.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444c8748:::benchmark_data_smithi194_103039_object58086:head# 2024-04-29T14:21:17.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444d239a:::benchmark_data_smithi194_103039_object124388:head# 2024-04-29T14:21:17.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444d548a:::benchmark_data_smithi194_103039_object146911:head# 2024-04-29T14:21:17.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444d81ab:::benchmark_data_smithi194_103039_object152181:head# 2024-04-29T14:21:17.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444dcbac:::benchmark_data_smithi194_103039_object149680:head# 2024-04-29T14:21:17.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444dd671:::benchmark_data_smithi194_103039_object123049:head# 2024-04-29T14:21:17.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444debea:::benchmark_data_smithi194_103039_object49589:head# 2024-04-29T14:21:17.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444df661:::benchmark_data_smithi194_103039_object96164:head# 2024-04-29T14:21:17.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444e0d14:::benchmark_data_smithi194_103039_object125091:head# 2024-04-29T14:21:17.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444e8b58:::benchmark_data_smithi194_103039_object111017:head# 2024-04-29T14:21:17.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444eae85:::benchmark_data_smithi194_103039_object66520:head# 2024-04-29T14:21:17.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444eecfd:::benchmark_data_smithi194_103039_object75523:head# 2024-04-29T14:21:17.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:444fff1c:::benchmark_data_smithi194_103039_object130475:head# 2024-04-29T14:21:17.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44500618:::benchmark_data_smithi194_103039_object151871:head# 2024-04-29T14:21:17.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44502678:::benchmark_data_smithi194_103039_object77957:head# 2024-04-29T14:21:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4450e735:::benchmark_data_smithi194_103039_object147731:head# 2024-04-29T14:21:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44511aa3:::benchmark_data_smithi194_103039_object98315:head# 2024-04-29T14:21:17.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4451a35c:::benchmark_data_smithi194_103039_object87410:head# 2024-04-29T14:21:17.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4452547b:::benchmark_data_smithi194_103039_object77459:head# 2024-04-29T14:21:17.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4452bd0d:::benchmark_data_smithi194_103039_object89952:head# 2024-04-29T14:21:17.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:445389ce:::benchmark_data_smithi194_103039_object88754:head# 2024-04-29T14:21:17.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4453b6af:::benchmark_data_smithi194_103039_object109455:head# 2024-04-29T14:21:17.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44546647:::benchmark_data_smithi194_103039_object59116:head# 2024-04-29T14:21:17.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:445628f4:::benchmark_data_smithi194_103039_object44749:head# 2024-04-29T14:21:17.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44565284:::benchmark_data_smithi194_103039_object62655:head# 2024-04-29T14:21:17.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:17 smithi121 ceph-mon[44533]: pgmap v981: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 181 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:17.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/141547992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:17.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:17.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44567386:::benchmark_data_smithi194_103039_object81864:head# 2024-04-29T14:21:17.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44575950:::benchmark_data_smithi194_103039_object119984:head# 2024-04-29T14:21:17.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44576f12:::benchmark_data_smithi194_103039_object134897:head# 2024-04-29T14:21:17.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44585cda:::benchmark_data_smithi194_103039_object104658:head# 2024-04-29T14:21:17.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4458e03e:::benchmark_data_smithi194_103039_object81986:head# 2024-04-29T14:21:17.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:445a566d:::benchmark_data_smithi194_103039_object60339:head# 2024-04-29T14:21:17.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:445bed92:::benchmark_data_smithi194_103039_object82424:head# 2024-04-29T14:21:17.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:445ccc97:::benchmark_data_smithi194_103039_object93537:head# 2024-04-29T14:21:17.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:445ce91b:::benchmark_data_smithi194_103039_object57234:head# 2024-04-29T14:21:17.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:445f733a:::benchmark_data_smithi194_103039_object97108:head# 2024-04-29T14:21:17.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44615bdc:::benchmark_data_smithi194_103039_object51727:head# 2024-04-29T14:21:17.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44617061:::benchmark_data_smithi194_103039_object70030:head# 2024-04-29T14:21:17.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4461e6dd:::benchmark_data_smithi194_103039_object66385:head# 2024-04-29T14:21:17.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4461e8af:::benchmark_data_smithi194_103039_object109150:head# 2024-04-29T14:21:17.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44625f09:::benchmark_data_smithi194_103039_object92660:head# 2024-04-29T14:21:17.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4462c789:::benchmark_data_smithi194_103039_object75930:head# 2024-04-29T14:21:17.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44633ff6:::benchmark_data_smithi194_103039_object111657:head# 2024-04-29T14:21:17.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4464593a:::benchmark_data_smithi194_103039_object44641:head# 2024-04-29T14:21:17.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4464af9e:::benchmark_data_smithi194_103039_object140600:head# 2024-04-29T14:21:17.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44650e19:::benchmark_data_smithi194_103039_object115471:head# 2024-04-29T14:21:17.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44654434:::benchmark_data_smithi194_103039_object147149:head# 2024-04-29T14:21:17.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4465dc38:::benchmark_data_smithi194_103039_object129735:head# 2024-04-29T14:21:17.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446622f9:::benchmark_data_smithi194_103039_object138626:head# 2024-04-29T14:21:17.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4466952c:::benchmark_data_smithi194_103039_object92065:head# 2024-04-29T14:21:17.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44678bf6:::benchmark_data_smithi194_103039_object148136:head# 2024-04-29T14:21:17.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4467eb65:::benchmark_data_smithi194_103039_object62690:head# 2024-04-29T14:21:17.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44689433:::benchmark_data_smithi194_103039_object50180:head# 2024-04-29T14:21:17.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44692040:::benchmark_data_smithi194_103039_object84801:head# 2024-04-29T14:21:17.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446a9cf5:::benchmark_data_smithi194_103039_object48331:head# 2024-04-29T14:21:17.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446b1683:::benchmark_data_smithi194_103039_object103840:head# 2024-04-29T14:21:17.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446d550f:::benchmark_data_smithi194_103039_object43308:head# 2024-04-29T14:21:17.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446d8ee9:::benchmark_data_smithi194_103039_object54730:head# 2024-04-29T14:21:17.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446e1637:::benchmark_data_smithi194_103039_object136265:head# 2024-04-29T14:21:17.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446f61a9:::benchmark_data_smithi194_103039_object109745:head# 2024-04-29T14:21:17.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:446ffa9e:::benchmark_data_smithi194_103039_object131100:head# 2024-04-29T14:21:17.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447086c5:::benchmark_data_smithi194_103039_object89673:head# 2024-04-29T14:21:17.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4471c28d:::benchmark_data_smithi194_103039_object146558:head# 2024-04-29T14:21:17.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44735969:::benchmark_data_smithi194_103039_object84327:head# 2024-04-29T14:21:17.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4473c7fb:::benchmark_data_smithi194_103039_object147400:head# 2024-04-29T14:21:17.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4473cb82:::benchmark_data_smithi194_103039_object147706:head# 2024-04-29T14:21:17.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4474746e:::benchmark_data_smithi194_103039_object45697:head# 2024-04-29T14:21:17.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4474d082:::benchmark_data_smithi194_103039_object147611:head# 2024-04-29T14:21:17.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4474e07b:::benchmark_data_smithi194_103039_object89303:head# 2024-04-29T14:21:17.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4475a9b1:::benchmark_data_smithi194_103039_object88600:head# 2024-04-29T14:21:17.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4476bd80:::benchmark_data_smithi194_103039_object58188:head# 2024-04-29T14:21:17.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447856b3:::benchmark_data_smithi194_103039_object128942:head# 2024-04-29T14:21:17.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44799ad9:::benchmark_data_smithi194_103039_object62235:head# 2024-04-29T14:21:17.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447a0b8d:::benchmark_data_smithi194_103039_object146443:head# 2024-04-29T14:21:17.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447a879c:::benchmark_data_smithi194_103039_object117567:head# 2024-04-29T14:21:17.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447abed9:::benchmark_data_smithi194_103039_object87852:head# 2024-04-29T14:21:17.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447ae714:::benchmark_data_smithi194_103039_object131063:head# 2024-04-29T14:21:17.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447c1412:::benchmark_data_smithi194_103039_object156301:head# 2024-04-29T14:21:17.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447c52f9:::benchmark_data_smithi194_103039_object132053:head# 2024-04-29T14:21:17.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447cba25:::benchmark_data_smithi194_103039_object127695:head# 2024-04-29T14:21:17.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447d8468:::benchmark_data_smithi194_103039_object91701:head# 2024-04-29T14:21:17.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447ed62a:::benchmark_data_smithi194_103039_object62818:head# 2024-04-29T14:21:17.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447edf7b:::benchmark_data_smithi194_103039_object112198:head# 2024-04-29T14:21:17.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447f137d:::benchmark_data_smithi194_103039_object71322:head# 2024-04-29T14:21:17.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:447fbe6c:::benchmark_data_smithi194_103039_object75061:head# 2024-04-29T14:21:17.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4480a82f:::benchmark_data_smithi194_103039_object87014:head# 2024-04-29T14:21:17.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4480e770:::benchmark_data_smithi194_103039_object81262:head# 2024-04-29T14:21:17.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4481abf8:::benchmark_data_smithi194_103039_object57307:head# 2024-04-29T14:21:17.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44833b79:::benchmark_data_smithi194_103039_object103303:head# 2024-04-29T14:21:17.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44834c15:::benchmark_data_smithi194_103039_object61924:head# 2024-04-29T14:21:17.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4483fce3:::benchmark_data_smithi194_103039_object125061:head# 2024-04-29T14:21:17.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4484906f:::benchmark_data_smithi194_103039_object110558:head# 2024-04-29T14:21:17.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4484d3ea:::benchmark_data_smithi194_103039_object82837:head# 2024-04-29T14:21:17.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44858563:::benchmark_data_smithi194_103039_object118072:head# 2024-04-29T14:21:17.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44859410:::benchmark_data_smithi194_103039_object99211:head# 2024-04-29T14:21:17.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44875bb9:::benchmark_data_smithi194_103039_object155438:head# 2024-04-29T14:21:17.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44879ccf:::benchmark_data_smithi194_103039_object90097:head# 2024-04-29T14:21:17.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4487a4a7:::benchmark_data_smithi194_103039_object107225:head# 2024-04-29T14:21:17.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44880116:::benchmark_data_smithi194_103039_object49432:head# 2024-04-29T14:21:17.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44882d55:::benchmark_data_smithi194_103039_object100936:head# 2024-04-29T14:21:17.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44885ac9:::benchmark_data_smithi194_103039_object75409:head# 2024-04-29T14:21:17.408 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:17.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44887a7e:::benchmark_data_smithi194_103039_object65083:head# 2024-04-29T14:21:17.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44887b9c:::benchmark_data_smithi194_103039_object48059:head# 2024-04-29T14:21:17.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4489312a:::benchmark_data_smithi194_103039_object52601:head# 2024-04-29T14:21:17.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44897afd:::benchmark_data_smithi194_103039_object115265:head# 2024-04-29T14:21:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:17.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448a0fad:::benchmark_data_smithi194_103039_object127461:head# 2024-04-29T14:21:17.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448a21b4:::benchmark_data_smithi194_103039_object71806:head# 2024-04-29T14:21:17.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448a5501:::benchmark_data_smithi194_103039_object106978:head# 2024-04-29T14:21:17.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:17.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448ad6a6:::benchmark_data_smithi194_103039_object127466:head# 2024-04-29T14:21:17.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448b9c41:::benchmark_data_smithi194_103039_object144871:head# 2024-04-29T14:21:17.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448c38fb:::benchmark_data_smithi194_103039_object57007:head# 2024-04-29T14:21:17.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448de5a8:::benchmark_data_smithi194_103039_object113299:head# 2024-04-29T14:21:17.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448e86da:::benchmark_data_smithi194_103039_object156606:head# 2024-04-29T14:21:17.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448f0d01:::benchmark_data_smithi194_103039_object111716:head# 2024-04-29T14:21:17.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448fca4f:::benchmark_data_smithi194_103039_object124585:head# 2024-04-29T14:21:17.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:448fd2f4:::benchmark_data_smithi194_103039_object83977:head# 2024-04-29T14:21:17.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44903a0f:::benchmark_data_smithi194_103039_object136888:head# 2024-04-29T14:21:17.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4491ad3a:::benchmark_data_smithi194_103039_object81240:head# 2024-04-29T14:21:17.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4491b08d:::benchmark_data_smithi194_103039_object81851:head# 2024-04-29T14:21:17.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449269f1:::benchmark_data_smithi194_103039_object141074:head# 2024-04-29T14:21:17.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44926b80:::benchmark_data_smithi194_103039_object67500:head# 2024-04-29T14:21:17.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4492700e:::benchmark_data_smithi194_103039_object137286:head# 2024-04-29T14:21:17.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4492a73c:::benchmark_data_smithi194_103039_object139134:head# 2024-04-29T14:21:17.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449301a7:::benchmark_data_smithi194_103039_object132144:head# 2024-04-29T14:21:17.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449335b7:::benchmark_data_smithi194_103039_object57078:head# 2024-04-29T14:21:17.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44933e74:::benchmark_data_smithi194_103039_object108287:head# 2024-04-29T14:21:17.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4493f0d2:::benchmark_data_smithi194_103039_object113828:head# 2024-04-29T14:21:17.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449471c4:::benchmark_data_smithi194_103039_object128274:head# 2024-04-29T14:21:17.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4494e907:::benchmark_data_smithi194_103039_object112767:head# 2024-04-29T14:21:17.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4495c3b6:::benchmark_data_smithi194_103039_object72611:head# 2024-04-29T14:21:17.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4496c685:::benchmark_data_smithi194_103039_object57943:head# 2024-04-29T14:21:17.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44970ff8:::benchmark_data_smithi194_103039_object86208:head# 2024-04-29T14:21:17.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4497211f:::benchmark_data_smithi194_103039_object79370:head# 2024-04-29T14:21:17.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44972aa7:::benchmark_data_smithi194_103039_object101734:head# 2024-04-29T14:21:17.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4497679c:::benchmark_data_smithi194_103039_object136839:head# 2024-04-29T14:21:17.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4497a6d0:::benchmark_data_smithi194_103039_object112139:head# 2024-04-29T14:21:17.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4498855c:::benchmark_data_smithi194_103039_object131683:head# 2024-04-29T14:21:17.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44989bbe:::benchmark_data_smithi194_103039_object64357:head# 2024-04-29T14:21:17.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4498b0b2:::benchmark_data_smithi194_103039_object149309:head# 2024-04-29T14:21:17.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4498bc05:::benchmark_data_smithi194_103039_object110132:head# 2024-04-29T14:21:17.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44991698:::benchmark_data_smithi194_103039_object153348:head# 2024-04-29T14:21:17.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44991e45:::benchmark_data_smithi194_103039_object91698:head# 2024-04-29T14:21:17.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44995164:::benchmark_data_smithi194_103039_object57059:head# 2024-04-29T14:21:17.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449b23a5:::benchmark_data_smithi194_103039_object112011:head# 2024-04-29T14:21:17.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449b9e5d:::benchmark_data_smithi194_103039_object90086:head# 2024-04-29T14:21:17.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449cb9ad:::benchmark_data_smithi194_103039_object153658:head# 2024-04-29T14:21:17.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449ce383:::benchmark_data_smithi194_103039_object50568:head# 2024-04-29T14:21:17.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449db605:::benchmark_data_smithi194_103039_object119789:head# 2024-04-29T14:21:17.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449fb1f6:::benchmark_data_smithi194_103039_object87843:head# 2024-04-29T14:21:17.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:449fcc12:::benchmark_data_smithi194_103039_object57807:head# 2024-04-29T14:21:17.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a17d87:::benchmark_data_smithi194_103039_object114858:head# 2024-04-29T14:21:17.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a1dee2:::benchmark_data_smithi194_103039_object108730:head# 2024-04-29T14:21:17.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a44569:::benchmark_data_smithi194_103039_object117204:head# 2024-04-29T14:21:17.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a62348:::benchmark_data_smithi194_103039_object82976:head# 2024-04-29T14:21:17.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a71256:::benchmark_data_smithi194_103039_object132533:head# 2024-04-29T14:21:17.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a7c2ea:::benchmark_data_smithi194_103039_object64227:head# 2024-04-29T14:21:17.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a814bd:::benchmark_data_smithi194_103039_object60096:head# 2024-04-29T14:21:17.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a82174:::benchmark_data_smithi194_103039_object58734:head# 2024-04-29T14:21:17.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a8878e:::benchmark_data_smithi194_103039_object71625:head# 2024-04-29T14:21:17.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a94b99:::benchmark_data_smithi194_103039_object130047:head# 2024-04-29T14:21:17.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a966af:::benchmark_data_smithi194_103039_object98492:head# 2024-04-29T14:21:17.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44a98318:::benchmark_data_smithi194_103039_object122067:head# 2024-04-29T14:21:17.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ab6242:::benchmark_data_smithi194_103039_object113143:head# 2024-04-29T14:21:17.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ac1fb6:::benchmark_data_smithi194_103039_object115170:head# 2024-04-29T14:21:17.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44acff76:::benchmark_data_smithi194_103039_object49195:head# 2024-04-29T14:21:17.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ad0cf2:::benchmark_data_smithi194_103039_object125998:head# 2024-04-29T14:21:17.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ad46f5:::benchmark_data_smithi194_103039_object113829:head# 2024-04-29T14:21:17.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ad8cbd:::benchmark_data_smithi194_103039_object154483:head# 2024-04-29T14:21:17.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44adaf72:::benchmark_data_smithi194_103039_object110606:head# 2024-04-29T14:21:17.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44adb7e1:::benchmark_data_smithi194_103039_object58781:head# 2024-04-29T14:21:17.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ae9b14:::benchmark_data_smithi194_103039_object104117:head# 2024-04-29T14:21:17.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44af6291:::benchmark_data_smithi194_103039_object140061:head# 2024-04-29T14:21:17.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44af7406:::benchmark_data_smithi194_103039_object58522:head# 2024-04-29T14:21:17.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b0406c:::benchmark_data_smithi194_103039_object51538:head# 2024-04-29T14:21:17.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b07201:::benchmark_data_smithi194_103039_object82371:head# 2024-04-29T14:21:17.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b0734b:::benchmark_data_smithi194_103039_object119017:head# 2024-04-29T14:21:17.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b13755:::benchmark_data_smithi194_103039_object72729:head# 2024-04-29T14:21:17.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b15611:::benchmark_data_smithi194_103039_object96277:head# 2024-04-29T14:21:17.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b28297:::benchmark_data_smithi194_103039_object127413:head# 2024-04-29T14:21:17.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b35866:::benchmark_data_smithi194_103039_object129766:head# 2024-04-29T14:21:17.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b3c96c:::benchmark_data_smithi194_103039_object129193:head# 2024-04-29T14:21:17.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b3d9aa:::benchmark_data_smithi194_103039_object100468:head# 2024-04-29T14:21:17.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b3f681:::benchmark_data_smithi194_103039_object93108:head# 2024-04-29T14:21:17.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b3f73f:::benchmark_data_smithi194_103039_object111091:head# 2024-04-29T14:21:17.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b48442:::benchmark_data_smithi194_103039_object115727:head# 2024-04-29T14:21:17.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b51b1e:::benchmark_data_smithi194_103039_object124927:head# 2024-04-29T14:21:17.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b5e72e:::benchmark_data_smithi194_103039_object63650:head# 2024-04-29T14:21:17.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b650f7:::benchmark_data_smithi194_103039_object79929:head# 2024-04-29T14:21:17.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b6b16e:::benchmark_data_smithi194_103039_object117126:head# 2024-04-29T14:21:17.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b746d6:::benchmark_data_smithi194_103039_object126946:head# 2024-04-29T14:21:17.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b76127:::benchmark_data_smithi194_103039_object62502:head# 2024-04-29T14:21:17.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44b9402e:::benchmark_data_smithi194_103039_object92630:head# 2024-04-29T14:21:17.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ba8d2f:::benchmark_data_smithi194_103039_object123391:head# 2024-04-29T14:21:17.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44bb60ac:::benchmark_data_smithi194_103039_object61292:head# 2024-04-29T14:21:17.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44bbffc5:::benchmark_data_smithi194_103039_object119758:head# 2024-04-29T14:21:17.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44bd7034:::benchmark_data_smithi194_103039_object44974:head# 2024-04-29T14:21:17.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44bd8293:::benchmark_data_smithi194_103039_object115109:head# 2024-04-29T14:21:17.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44be49d5:::benchmark_data_smithi194_103039_object72377:head# 2024-04-29T14:21:17.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44be51ea:::benchmark_data_smithi194_103039_object96450:head# 2024-04-29T14:21:17.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44be8864:::benchmark_data_smithi194_103039_object59966:head# 2024-04-29T14:21:17.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44bf4bbb:::benchmark_data_smithi194_103039_object115810:head# 2024-04-29T14:21:17.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44bf925b:::benchmark_data_smithi194_103039_object85799:head# 2024-04-29T14:21:17.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44bfe67c:::benchmark_data_smithi194_103039_object101736:head# 2024-04-29T14:21:17.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c004f8:::benchmark_data_smithi194_103039_object102482:head# 2024-04-29T14:21:17.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c017bb:::benchmark_data_smithi194_103039_object51099:head# 2024-04-29T14:21:17.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c0a1f2:::benchmark_data_smithi194_103039_object62231:head# 2024-04-29T14:21:17.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c163b0:::benchmark_data_smithi194_103039_object120217:head# 2024-04-29T14:21:17.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c1955c:::benchmark_data_smithi194_103039_object61161:head# 2024-04-29T14:21:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c1d135:::benchmark_data_smithi194_103039_object51214:head# 2024-04-29T14:21:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c2341b:::benchmark_data_smithi194_103039_object114417:head# 2024-04-29T14:21:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c24be5:::benchmark_data_smithi194_103039_object151939:head# 2024-04-29T14:21:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c34ede:::benchmark_data_smithi194_103039_object54842:head# 2024-04-29T14:21:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c450d6:::benchmark_data_smithi194_103039_object79337:head# 2024-04-29T14:21:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c4d0d6:::benchmark_data_smithi194_103039_object73894:head# 2024-04-29T14:21:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c905eb:::benchmark_data_smithi194_103039_object139950:head# 2024-04-29T14:21:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c99cd7:::benchmark_data_smithi194_103039_object102688:head# 2024-04-29T14:21:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44c9df62:::benchmark_data_smithi194_103039_object47951:head# 2024-04-29T14:21:17.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ca929e:::benchmark_data_smithi194_103039_object91267:head# 2024-04-29T14:21:17.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cb01f8:::benchmark_data_smithi194_103039_object130449:head# 2024-04-29T14:21:17.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cb4776:::benchmark_data_smithi194_103039_object113657:head# 2024-04-29T14:21:17.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cb705e:::benchmark_data_smithi194_103039_object55271:head# 2024-04-29T14:21:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cbbd6a:::benchmark_data_smithi194_103039_object133225:head# 2024-04-29T14:21:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cc3c60:::benchmark_data_smithi194_103039_object118591:head# 2024-04-29T14:21:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cc7746:::benchmark_data_smithi194_103039_object130404:head# 2024-04-29T14:21:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cc8460:::benchmark_data_smithi194_103039_object129410:head# 2024-04-29T14:21:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ccdd58:::benchmark_data_smithi194_103039_object61486:head# 2024-04-29T14:21:17.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cce33b:::benchmark_data_smithi194_103039_object78461:head# 2024-04-29T14:21:17.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ce1ecc:::benchmark_data_smithi194_103039_object53664:head# 2024-04-29T14:21:17.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cecdd6:::benchmark_data_smithi194_103039_object131581:head# 2024-04-29T14:21:17.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ced062:::benchmark_data_smithi194_103039_object122328:head# 2024-04-29T14:21:17.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ced32b:::benchmark_data_smithi194_103039_object142396:head# 2024-04-29T14:21:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cf021b:::benchmark_data_smithi194_103039_object118146:head# 2024-04-29T14:21:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cf1561:::benchmark_data_smithi194_103039_object60693:head# 2024-04-29T14:21:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cf19a6:::benchmark_data_smithi194_103039_object120075:head# 2024-04-29T14:21:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cfd2e2:::benchmark_data_smithi194_103039_object82310:head# 2024-04-29T14:21:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44cfe706:::benchmark_data_smithi194_103039_object54592:head# 2024-04-29T14:21:17.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d0d493:::benchmark_data_smithi194_103039_object94143:head# 2024-04-29T14:21:17.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d1210f:::benchmark_data_smithi194_103039_object135590:head# 2024-04-29T14:21:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d2f2e0:::benchmark_data_smithi194_103039_object155786:head# 2024-04-29T14:21:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d30dde:::benchmark_data_smithi194_103039_object132724:head# 2024-04-29T14:21:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d35a47:::benchmark_data_smithi194_103039_object146387:head# 2024-04-29T14:21:17.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d600e8:::benchmark_data_smithi194_103039_object49270:head# 2024-04-29T14:21:17.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d65b18:::benchmark_data_smithi194_103039_object107010:head# 2024-04-29T14:21:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d6fd06:::benchmark_data_smithi194_103039_object49512:head# 2024-04-29T14:21:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d7a12b:::benchmark_data_smithi194_103039_object102586:head# 2024-04-29T14:21:17.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d88c23:::benchmark_data_smithi194_103039_object123923:head# 2024-04-29T14:21:17.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44d9d56c:::benchmark_data_smithi194_103039_object121033:head# 2024-04-29T14:21:17.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44da15b8:::benchmark_data_smithi194_103039_object134834:head# 2024-04-29T14:21:17.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44da40b8:::benchmark_data_smithi194_103039_object134254:head# 2024-04-29T14:21:17.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dab1d0:::benchmark_data_smithi194_103039_object118664:head# 2024-04-29T14:21:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dab286:::benchmark_data_smithi194_103039_object148687:head# 2024-04-29T14:21:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dac221:::benchmark_data_smithi194_103039_object139752:head# 2024-04-29T14:21:17.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dafd21:::benchmark_data_smithi194_103039_object66302:head# 2024-04-29T14:21:17.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44db3d95:::benchmark_data_smithi194_103039_object57771:head# 2024-04-29T14:21:17.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dba2a6:::benchmark_data_smithi194_103039_object88582:head# 2024-04-29T14:21:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dc1083:::benchmark_data_smithi194_103039_object105573:head# 2024-04-29T14:21:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dc998a:::benchmark_data_smithi194_103039_object111633:head# 2024-04-29T14:21:17.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dca9ed:::benchmark_data_smithi194_103039_object51114:head# 2024-04-29T14:21:17.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dd26d9:::benchmark_data_smithi194_103039_object117484:head# 2024-04-29T14:21:17.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44dd72ab:::benchmark_data_smithi194_103039_object108308:head# 2024-04-29T14:21:17.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44df9c0c:::benchmark_data_smithi194_103039_object155365:head# 2024-04-29T14:21:17.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e00ab6:::benchmark_data_smithi194_103039_object88782:head# 2024-04-29T14:21:17.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e17af1:::benchmark_data_smithi194_103039_object55697:head# 2024-04-29T14:21:17.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e3e42b:::benchmark_data_smithi194_103039_object124247:head# 2024-04-29T14:21:17.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e413e8:::benchmark_data_smithi194_103039_object64220:head# 2024-04-29T14:21:17.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e44ef1:::benchmark_data_smithi194_103039_object95103:head# 2024-04-29T14:21:17.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e4e4ca:::benchmark_data_smithi194_103039_object42410:head# 2024-04-29T14:21:17.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e4e84d:::benchmark_data_smithi194_103039_object136971:head# 2024-04-29T14:21:17.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e6ba9a:::benchmark_data_smithi194_103039_object137287:head# 2024-04-29T14:21:17.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e76026:::benchmark_data_smithi194_103039_object104206:head# 2024-04-29T14:21:17.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e7a843:::benchmark_data_smithi194_103039_object45824:head# 2024-04-29T14:21:17.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e7e8cb:::benchmark_data_smithi194_103039_object142843:head# 2024-04-29T14:21:17.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e80144:::benchmark_data_smithi194_103039_object127803:head# 2024-04-29T14:21:17.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e88f70:::benchmark_data_smithi194_103039_object42033:head# 2024-04-29T14:21:17.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44e96dad:::benchmark_data_smithi194_103039_object144127:head# 2024-04-29T14:21:17.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ebdcce:::benchmark_data_smithi194_103039_object74888:head# 2024-04-29T14:21:17.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ec05a0:::benchmark_data_smithi194_103039_object50390:head# 2024-04-29T14:21:17.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ec51b6:::benchmark_data_smithi194_103039_object80513:head# 2024-04-29T14:21:17.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44edfadb:::benchmark_data_smithi194_103039_object138951:head# 2024-04-29T14:21:17.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ee1b45:::benchmark_data_smithi194_103039_object135685:head# 2024-04-29T14:21:17.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44eee7c8:::benchmark_data_smithi194_103039_object99699:head# 2024-04-29T14:21:17.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ef7888:::benchmark_data_smithi194_103039_object57079:head# 2024-04-29T14:21:17.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ef81d6:::benchmark_data_smithi194_103039_object138357:head# 2024-04-29T14:21:17.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f150eb:::benchmark_data_smithi194_103039_object120102:head# 2024-04-29T14:21:17.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f23ad6:::benchmark_data_smithi194_103039_object131270:head# 2024-04-29T14:21:17.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f425df:::benchmark_data_smithi194_103039_object66413:head# 2024-04-29T14:21:17.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f470c9:::benchmark_data_smithi194_103039_object148722:head# 2024-04-29T14:21:17.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f5542b:::benchmark_data_smithi194_103039_object58150:head# 2024-04-29T14:21:17.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f5afe1:::benchmark_data_smithi194_103039_object91643:head# 2024-04-29T14:21:17.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f60e39:::benchmark_data_smithi194_103039_object67309:head# 2024-04-29T14:21:17.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f67bcb:::benchmark_data_smithi194_103039_object109936:head# 2024-04-29T14:21:17.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f6c0c0:::benchmark_data_smithi194_103039_object97996:head# 2024-04-29T14:21:17.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f6f2af:::benchmark_data_smithi194_103039_object131884:head# 2024-04-29T14:21:17.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44f8b974:::benchmark_data_smithi194_103039_object60194:head# 2024-04-29T14:21:17.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fa2164:::benchmark_data_smithi194_103039_object74130:head# 2024-04-29T14:21:17.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fa7296:::benchmark_data_smithi194_103039_object48763:head# 2024-04-29T14:21:17.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fad9a6:::benchmark_data_smithi194_103039_object142515:head# 2024-04-29T14:21:17.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fb07f1:::benchmark_data_smithi194_103039_object140890:head# 2024-04-29T14:21:17.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fb18ee:::benchmark_data_smithi194_103039_object126453:head# 2024-04-29T14:21:17.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fb194a:::benchmark_data_smithi194_103039_object130124:head# 2024-04-29T14:21:17.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fb58bc:::benchmark_data_smithi194_103039_object123493:head# 2024-04-29T14:21:17.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fb828d:::benchmark_data_smithi194_103039_object84075:head# 2024-04-29T14:21:17.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fc5933:::benchmark_data_smithi194_103039_object73631:head# 2024-04-29T14:21:17.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fcdf07:::benchmark_data_smithi194_103039_object106836:head# 2024-04-29T14:21:17.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fd3ec4:::benchmark_data_smithi194_103039_object84437:head# 2024-04-29T14:21:17.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fe39e6:::benchmark_data_smithi194_103039_object88865:head# 2024-04-29T14:21:17.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fe4387:::benchmark_data_smithi194_103039_object138079:head# 2024-04-29T14:21:17.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fea3a6:::benchmark_data_smithi194_103039_object46183:head# 2024-04-29T14:21:17.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44feccdb:::benchmark_data_smithi194_103039_object74475:head# 2024-04-29T14:21:17.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fed20e:::benchmark_data_smithi194_103039_object78087:head# 2024-04-29T14:21:17.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ff5b8f:::benchmark_data_smithi194_103039_object146184:head# 2024-04-29T14:21:17.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ff795f:::benchmark_data_smithi194_103039_object91807:head# 2024-04-29T14:21:17.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ff8b4e:::benchmark_data_smithi194_103039_object111187:head# 2024-04-29T14:21:17.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ffbe28:::benchmark_data_smithi194_103039_object50464:head# 2024-04-29T14:21:17.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ffc3b2:::benchmark_data_smithi194_103039_object119183:head# 2024-04-29T14:21:17.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44ffe0dd:::benchmark_data_smithi194_103039_object137195:head# 2024-04-29T14:21:17.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:44fff163:::benchmark_data_smithi194_103039_object146095:head# 2024-04-29T14:21:17.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450009d2:::benchmark_data_smithi194_103039_object88859:head# 2024-04-29T14:21:17.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4500c321:::benchmark_data_smithi194_103039_object80030:head# 2024-04-29T14:21:17.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45010205:::benchmark_data_smithi194_103039_object123786:head# 2024-04-29T14:21:17.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450147cb:::benchmark_data_smithi194_103039_object68582:head# 2024-04-29T14:21:17.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450271bf:::benchmark_data_smithi194_103039_object89853:head# 2024-04-29T14:21:17.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4502a41e:::benchmark_data_smithi194_103039_object76800:head# 2024-04-29T14:21:17.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450451d0:::benchmark_data_smithi194_103039_object82448:head# 2024-04-29T14:21:17.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45054a23:::benchmark_data_smithi194_103039_object128363:head# 2024-04-29T14:21:17.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45069a52:::benchmark_data_smithi194_103039_object128012:head# 2024-04-29T14:21:17.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4506dc81:::benchmark_data_smithi194_103039_object132803:head# 2024-04-29T14:21:17.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45071264:::benchmark_data_smithi194_103039_object66995:head# 2024-04-29T14:21:17.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45075c6c:::benchmark_data_smithi194_103039_object113370:head# 2024-04-29T14:21:17.512 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:17.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4507f95d:::benchmark_data_smithi194_103039_object149789:head# 2024-04-29T14:21:17.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450833fe:::benchmark_data_smithi194_103039_object94799:head# 2024-04-29T14:21:17.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45088be2:::benchmark_data_smithi194_103039_object95884:head# 2024-04-29T14:21:17.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45094859:::benchmark_data_smithi194_103039_object52135:head# 2024-04-29T14:21:17.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4509e91d:::benchmark_data_smithi194_103039_object45453:head# 2024-04-29T14:21:17.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450a2ba5:::benchmark_data_smithi194_103039_object65987:head# 2024-04-29T14:21:17.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450a821e:::benchmark_data_smithi194_103039_object53304:head# 2024-04-29T14:21:17.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450b5674:::benchmark_data_smithi194_103039_object141380:head# 2024-04-29T14:21:17.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450d338a:::benchmark_data_smithi194_103039_object91590:head# 2024-04-29T14:21:17.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450d67db:::benchmark_data_smithi194_103039_object131103:head# 2024-04-29T14:21:17.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450e0261:::benchmark_data_smithi194_103039_object65681:head# 2024-04-29T14:21:17.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450eaa03:::benchmark_data_smithi194_103039_object41784:head# 2024-04-29T14:21:17.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450ed099:::benchmark_data_smithi194_103039_object71293:head# 2024-04-29T14:21:17.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:450f3f57:::benchmark_data_smithi194_103039_object131026:head# 2024-04-29T14:21:17.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451103bb:::benchmark_data_smithi194_103039_object126835:head# 2024-04-29T14:21:17.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45134761:::benchmark_data_smithi194_103039_object123270:head# 2024-04-29T14:21:17.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45138261:::benchmark_data_smithi194_103039_object142972:head# 2024-04-29T14:21:17.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45138c36:::benchmark_data_smithi194_103039_object109553:head# 2024-04-29T14:21:17.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451456d5:::benchmark_data_smithi194_103039_object96011:head# 2024-04-29T14:21:17.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45147838:::benchmark_data_smithi194_103039_object73515:head# 2024-04-29T14:21:17.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451479c1:::benchmark_data_smithi194_103039_object93747:head# 2024-04-29T14:21:17.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4514e6fe:::benchmark_data_smithi194_103039_object128273:head# 2024-04-29T14:21:17.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451591cf:::benchmark_data_smithi194_103039_object138203:head# 2024-04-29T14:21:17.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4515afee:::benchmark_data_smithi194_103039_object52275:head# 2024-04-29T14:21:17.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451658ee:::benchmark_data_smithi194_103039_object147536:head# 2024-04-29T14:21:17.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45166423:::benchmark_data_smithi194_103039_object89643:head# 2024-04-29T14:21:17.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4516d06a:::benchmark_data_smithi194_103039_object145597:head# 2024-04-29T14:21:17.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45173676:::benchmark_data_smithi194_103039_object78971:head# 2024-04-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45177b6a:::benchmark_data_smithi194_103039_object119298:head# 2024-04-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4517cc1b:::benchmark_data_smithi194_103039_object117615:head# 2024-04-29T14:21:17.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4518066b:::benchmark_data_smithi194_103039_object73461:head# 2024-04-29T14:21:17.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451818e4:::benchmark_data_smithi194_103039_object94788:head# 2024-04-29T14:21:17.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4519abf3:::benchmark_data_smithi194_103039_object57410:head# 2024-04-29T14:21:17.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4519f90a:::benchmark_data_smithi194_103039_object99160:head# 2024-04-29T14:21:17.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451a529b:::benchmark_data_smithi194_103039_object107962:head# 2024-04-29T14:21:17.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451ad202:::benchmark_data_smithi194_103039_object120489:head# 2024-04-29T14:21:17.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451afe8f:::benchmark_data_smithi194_103039_object92654:head# 2024-04-29T14:21:17.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451c4d18:::benchmark_data_smithi194_103039_object156018:head# 2024-04-29T14:21:17.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451c5cc6:::benchmark_data_smithi194_103039_object87598:head# 2024-04-29T14:21:17.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451ce517:::benchmark_data_smithi194_103039_object96292:head# 2024-04-29T14:21:17.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451ce867:::benchmark_data_smithi194_103039_object111489:head# 2024-04-29T14:21:17.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451d7123:::benchmark_data_smithi194_103039_object141973:head# 2024-04-29T14:21:17.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451dd288:::benchmark_data_smithi194_103039_object99904:head# 2024-04-29T14:21:17.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451edbab:::benchmark_data_smithi194_103039_object90319:head# 2024-04-29T14:21:17.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451f2b67:::benchmark_data_smithi194_103039_object113794:head# 2024-04-29T14:21:17.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:451f95a8:::benchmark_data_smithi194_103039_object68107:head# 2024-04-29T14:21:17.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4520086c:::benchmark_data_smithi194_103039_object80121:head# 2024-04-29T14:21:17.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4520760a:::benchmark_data_smithi194_103039_object136371:head# 2024-04-29T14:21:17.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452077e0:::benchmark_data_smithi194_103039_object140792:head# 2024-04-29T14:21:17.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45215269:::benchmark_data_smithi194_103039_object96449:head# 2024-04-29T14:21:17.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45228840:::benchmark_data_smithi194_103039_object75649:head# 2024-04-29T14:21:17.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4522aa9f:::benchmark_data_smithi194_103039_object108574:head# 2024-04-29T14:21:17.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4524e9b5:::benchmark_data_smithi194_103039_object46400:head# 2024-04-29T14:21:17.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45253115:::benchmark_data_smithi194_103039_object59542:head# 2024-04-29T14:21:17.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452666ac:::benchmark_data_smithi194_103039_object124668:head# 2024-04-29T14:21:17.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45287973:::benchmark_data_smithi194_103039_object138444:head# 2024-04-29T14:21:17.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45289c68:::benchmark_data_smithi194_103039_object100620:head# 2024-04-29T14:21:17.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4528ce73:::benchmark_data_smithi194_103039_object83303:head# 2024-04-29T14:21:17.540 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:17.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452a0643:::benchmark_data_smithi194_103039_object140643:head# 2024-04-29T14:21:17.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452b1c82:::benchmark_data_smithi194_103039_object57217:head# 2024-04-29T14:21:17.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452cb646:::benchmark_data_smithi194_103039_object140272:head# 2024-04-29T14:21:17.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452cc75d:::benchmark_data_smithi194_103039_object80259:head# 2024-04-29T14:21:17.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452cf711:::benchmark_data_smithi194_103039_object90012:head# 2024-04-29T14:21:17.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452d48c0:::benchmark_data_smithi194_103039_object61203:head# 2024-04-29T14:21:17.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452e9d37:::benchmark_data_smithi194_103039_object100577:head# 2024-04-29T14:21:17.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452f011a:::benchmark_data_smithi194_103039_object98570:head# 2024-04-29T14:21:17.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452f460a:::benchmark_data_smithi194_103039_object50746:head# 2024-04-29T14:21:17.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:452f8eea:::benchmark_data_smithi194_103039_object145534:head# 2024-04-29T14:21:17.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4530695c:::benchmark_data_smithi194_103039_object144388:head# 2024-04-29T14:21:17.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4531443a:::benchmark_data_smithi194_103039_object63197:head# 2024-04-29T14:21:17.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4531768b:::benchmark_data_smithi194_103039_object138716:head# 2024-04-29T14:21:17.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4531acb5:::benchmark_data_smithi194_103039_object118585:head# 2024-04-29T14:21:17.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4532a860:::benchmark_data_smithi194_103039_object119686:head# 2024-04-29T14:21:17.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453360ad:::benchmark_data_smithi194_103039_object113024:head# 2024-04-29T14:21:17.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4533b9ff:::benchmark_data_smithi194_103039_object125393:head# 2024-04-29T14:21:17.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4533c8d8:::benchmark_data_smithi194_103039_object119589:head# 2024-04-29T14:21:17.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45346daf:::benchmark_data_smithi194_103039_object60749:head# 2024-04-29T14:21:17.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4534787d:::benchmark_data_smithi194_103039_object90507:head# 2024-04-29T14:21:17.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45358f1d:::benchmark_data_smithi194_103039_object81150:head# 2024-04-29T14:21:17.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4535a8d6:::benchmark_data_smithi194_103039_object137463:head# 2024-04-29T14:21:17.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453647b7:::benchmark_data_smithi194_103039_object74974:head# 2024-04-29T14:21:17.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4536affb:::benchmark_data_smithi194_103039_object57393:head# 2024-04-29T14:21:17.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4537e03c:::benchmark_data_smithi194_103039_object87554:head# 2024-04-29T14:21:17.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4538436f:::benchmark_data_smithi194_103039_object81226:head# 2024-04-29T14:21:17.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4538a066:::benchmark_data_smithi194_103039_object84031:head# 2024-04-29T14:21:17.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4538fe96:::benchmark_data_smithi194_103039_object155042:head# 2024-04-29T14:21:17.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45395d03:::benchmark_data_smithi194_103039_object106821:head# 2024-04-29T14:21:17.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453afedc:::benchmark_data_smithi194_103039_object63366:head# 2024-04-29T14:21:17.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453b34f6:::benchmark_data_smithi194_103039_object119843:head# 2024-04-29T14:21:17.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453b4758:::benchmark_data_smithi194_103039_object92770:head# 2024-04-29T14:21:17.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453bab6c:::benchmark_data_smithi194_103039_object147569:head# 2024-04-29T14:21:17.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453cd96c:::benchmark_data_smithi194_103039_object89412:head# 2024-04-29T14:21:17.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453cf368:::benchmark_data_smithi194_103039_object51933:head# 2024-04-29T14:21:17.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453cff5a:::benchmark_data_smithi194_103039_object138579:head# 2024-04-29T14:21:17.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453d0b2e:::benchmark_data_smithi194_103039_object106613:head# 2024-04-29T14:21:17.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453d5a71:::benchmark_data_smithi194_103039_object106924:head# 2024-04-29T14:21:17.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453d9007:::benchmark_data_smithi194_103039_object72890:head# 2024-04-29T14:21:17.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:453ff1d3:::benchmark_data_smithi194_103039_object57243:head# 2024-04-29T14:21:17.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45407f0a:::benchmark_data_smithi194_103039_object65031:head# 2024-04-29T14:21:17.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45413b5d:::benchmark_data_smithi194_103039_object126139:head# 2024-04-29T14:21:17.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45414ec8:::benchmark_data_smithi194_103039_object125931:head# 2024-04-29T14:21:17.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454400a6:::benchmark_data_smithi194_103039_object87332:head# 2024-04-29T14:21:17.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45447cf0:::benchmark_data_smithi194_103039_object75401:head# 2024-04-29T14:21:17.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4544cff9:::benchmark_data_smithi194_103039_object55984:head# 2024-04-29T14:21:17.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4545565b:::benchmark_data_smithi194_103039_object109323:head# 2024-04-29T14:21:17.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454560fc:::benchmark_data_smithi194_103039_object65697:head# 2024-04-29T14:21:17.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4545bc05:::benchmark_data_smithi194_103039_object46869:head# 2024-04-29T14:21:17.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4545cb67:::benchmark_data_smithi194_103039_object135064:head# 2024-04-29T14:21:17.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454654b6:::benchmark_data_smithi194_103039_object52320:head# 2024-04-29T14:21:17.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45476bfe:::benchmark_data_smithi194_103039_object97982:head# 2024-04-29T14:21:17.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45499169:::benchmark_data_smithi194_103039_object113101:head# 2024-04-29T14:21:17.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454a92fa:::benchmark_data_smithi194_103039_object149593:head# 2024-04-29T14:21:17.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454b3ab0:::benchmark_data_smithi194_103039_object74222:head# 2024-04-29T14:21:17.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454b609b:::benchmark_data_smithi194_103039_object42203:head# 2024-04-29T14:21:17.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454cdac6:::benchmark_data_smithi194_103039_object98394:head# 2024-04-29T14:21:17.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454d3005:::benchmark_data_smithi194_103039_object69731:head# 2024-04-29T14:21:17.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454d903b:::benchmark_data_smithi194_103039_object83645:head# 2024-04-29T14:21:17.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454dfddb:::benchmark_data_smithi194_103039_object128589:head# 2024-04-29T14:21:17.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454dfe22:::benchmark_data_smithi194_103039_object124693:head# 2024-04-29T14:21:17.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:454ffd6e:::benchmark_data_smithi194_103039_object63635:head# 2024-04-29T14:21:17.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45508dc3:::benchmark_data_smithi194_103039_object99805:head# 2024-04-29T14:21:17.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4550a58a:::benchmark_data_smithi194_103039_object54726:head# 2024-04-29T14:21:17.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4552796a:::benchmark_data_smithi194_103039_object150900:head# 2024-04-29T14:21:17.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45537594:::benchmark_data_smithi194_103039_object71016:head# 2024-04-29T14:21:17.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45546440:::benchmark_data_smithi194_103039_object78975:head# 2024-04-29T14:21:17.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4556820e:::benchmark_data_smithi194_103039_object78287:head# 2024-04-29T14:21:17.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45568fca:::benchmark_data_smithi194_103039_object100070:head# 2024-04-29T14:21:17.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45580651:::benchmark_data_smithi194_103039_object121847:head# 2024-04-29T14:21:17.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455986a6:::benchmark_data_smithi194_103039_object130039:head# 2024-04-29T14:21:17.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455a3109:::benchmark_data_smithi194_103039_object98979:head# 2024-04-29T14:21:17.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455a5528:::benchmark_data_smithi194_103039_object62188:head# 2024-04-29T14:21:17.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455aca85:::benchmark_data_smithi194_103039_object143224:head# 2024-04-29T14:21:17.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455b1157:::benchmark_data_smithi194_103039_object92603:head# 2024-04-29T14:21:17.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455b384d:::benchmark_data_smithi194_103039_object85923:head# 2024-04-29T14:21:17.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455dd9c5:::benchmark_data_smithi194_103039_object54866:head# 2024-04-29T14:21:17.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455de3a4:::benchmark_data_smithi194_103039_object108372:head# 2024-04-29T14:21:17.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455df5fd:::benchmark_data_smithi194_103039_object131606:head# 2024-04-29T14:21:17.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455e42c5:::benchmark_data_smithi194_103039_object96751:head# 2024-04-29T14:21:17.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:455f3afb:::benchmark_data_smithi194_103039_object91528:head# 2024-04-29T14:21:17.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4561a42d:::benchmark_data_smithi194_103039_object142288:head# 2024-04-29T14:21:17.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4561fc15:::benchmark_data_smithi194_103039_object48135:head# 2024-04-29T14:21:17.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45621b1a:::benchmark_data_smithi194_103039_object81447:head# 2024-04-29T14:21:17.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45630b5e:::benchmark_data_smithi194_103039_object126878:head# 2024-04-29T14:21:17.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45644ffe:::benchmark_data_smithi194_103039_object151895:head# 2024-04-29T14:21:17.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456528cf:::benchmark_data_smithi194_103039_object124510:head# 2024-04-29T14:21:17.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4565b1d3:::benchmark_data_smithi194_103039_object88071:head# 2024-04-29T14:21:17.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4565f3e3:::benchmark_data_smithi194_103039_object47497:head# 2024-04-29T14:21:17.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45665ca9:::benchmark_data_smithi194_103039_object90114:head# 2024-04-29T14:21:17.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4566993e:::benchmark_data_smithi194_103039_object153221:head# 2024-04-29T14:21:17.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4566d155:::benchmark_data_smithi194_103039_object118311:head# 2024-04-29T14:21:17.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4567f535:::benchmark_data_smithi194_103039_object89878:head# 2024-04-29T14:21:17.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45690b1b:::benchmark_data_smithi194_103039_object123417:head# 2024-04-29T14:21:17.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45691bbc:::benchmark_data_smithi194_103039_object132548:head# 2024-04-29T14:21:17.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4569204e:::benchmark_data_smithi194_103039_object68055:head# 2024-04-29T14:21:17.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45694a99:::benchmark_data_smithi194_103039_object75145:head# 2024-04-29T14:21:17.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4569f5b5:::benchmark_data_smithi194_103039_object127941:head# 2024-04-29T14:21:17.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456a5c94:::benchmark_data_smithi194_103039_object122767:head# 2024-04-29T14:21:17.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456afa4d:::benchmark_data_smithi194_103039_object156232:head# 2024-04-29T14:21:17.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456b5dcf:::benchmark_data_smithi194_103039_object156538:head# 2024-04-29T14:21:17.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456c4bab:::benchmark_data_smithi194_103039_object75457:head# 2024-04-29T14:21:17.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456da0d2:::benchmark_data_smithi194_103039_object120219:head# 2024-04-29T14:21:17.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456e661f:::benchmark_data_smithi194_103039_object137891:head# 2024-04-29T14:21:17.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456e8c7e:::benchmark_data_smithi194_103039_object141947:head# 2024-04-29T14:21:17.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456eaa0e:::benchmark_data_smithi194_103039_object82203:head# 2024-04-29T14:21:17.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456ec210:::benchmark_data_smithi194_103039_object60363:head# 2024-04-29T14:21:17.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:456fd03e:::benchmark_data_smithi194_103039_object120540:head# 2024-04-29T14:21:17.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4570370c:::benchmark_data_smithi194_103039_object117673:head# 2024-04-29T14:21:17.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4570e1ac:::benchmark_data_smithi194_103039_object82533:head# 2024-04-29T14:21:17.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457233da:::benchmark_data_smithi194_103039_object133286:head# 2024-04-29T14:21:17.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4572f278:::benchmark_data_smithi194_103039_object47276:head# 2024-04-29T14:21:17.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4573577f:::benchmark_data_smithi194_103039_object126467:head# 2024-04-29T14:21:17.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4574c406:::benchmark_data_smithi194_103039_object94893:head# 2024-04-29T14:21:17.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457541cf:::benchmark_data_smithi194_103039_object49174:head# 2024-04-29T14:21:17.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45756f67:::benchmark_data_smithi194_103039_object61347:head# 2024-04-29T14:21:17.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457573bb:::benchmark_data_smithi194_103039_object50761:head# 2024-04-29T14:21:17.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4575a792:::benchmark_data_smithi194_103039_object111367:head# 2024-04-29T14:21:17.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4575fe38:::benchmark_data_smithi194_103039_object117079:head# 2024-04-29T14:21:17.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457641f0:::benchmark_data_smithi194_103039_object138162:head# 2024-04-29T14:21:17.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45765be0:::benchmark_data_smithi194_103039_object80735:head# 2024-04-29T14:21:17.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45785e78:::benchmark_data_smithi194_103039_object93342:head# 2024-04-29T14:21:17.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4578b1c6:::benchmark_data_smithi194_103039_object93490:head# 2024-04-29T14:21:17.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4578f811:::benchmark_data_smithi194_103039_object117360:head# 2024-04-29T14:21:17.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4579168a:::benchmark_data_smithi194_103039_object103800:head# 2024-04-29T14:21:17.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45796f98:::benchmark_data_smithi194_103039_object71931:head# 2024-04-29T14:21:17.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45799ee2:::benchmark_data_smithi194_103039_object110844:head# 2024-04-29T14:21:17.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457a4b1f:::benchmark_data_smithi194_103039_object128248:head# 2024-04-29T14:21:17.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457a57ef:::benchmark_data_smithi194_103039_object116615:head# 2024-04-29T14:21:17.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457a7970:::benchmark_data_smithi194_103039_object42250:head# 2024-04-29T14:21:17.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457ab536:::benchmark_data_smithi194_103039_object92879:head# 2024-04-29T14:21:17.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457b64b0:::benchmark_data_smithi194_103039_object109188:head# 2024-04-29T14:21:17.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457b7bc3:::benchmark_data_smithi194_103039_object103218:head# 2024-04-29T14:21:17.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457c820f:::benchmark_data_smithi194_103039_object59659:head# 2024-04-29T14:21:17.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457d1cde:::benchmark_data_smithi194_103039_object44637:head# 2024-04-29T14:21:17.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457da233:::benchmark_data_smithi194_103039_object97391:head# 2024-04-29T14:21:17.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457f4fda:::benchmark_data_smithi194_103039_object78860:head# 2024-04-29T14:21:17.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457f8127:::benchmark_data_smithi194_103039_object68461:head# 2024-04-29T14:21:17.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:457fec21:::benchmark_data_smithi194_103039_object96940:head# 2024-04-29T14:21:17.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4580b81c:::benchmark_data_smithi194_103039_object55618:head# 2024-04-29T14:21:17.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4580b892:::benchmark_data_smithi194_103039_object55036:head# 2024-04-29T14:21:17.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4580e4ee:::benchmark_data_smithi194_103039_object63250:head# 2024-04-29T14:21:17.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4580f361:::benchmark_data_smithi194_103039_object49389:head# 2024-04-29T14:21:17.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45832db5:::benchmark_data_smithi194_103039_object130202:head# 2024-04-29T14:21:17.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4583400b:::benchmark_data_smithi194_103039_object83647:head# 2024-04-29T14:21:17.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4583c115:::benchmark_data_smithi194_103039_object153213:head# 2024-04-29T14:21:17.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458404fd:::benchmark_data_smithi194_103039_object80457:head# 2024-04-29T14:21:17.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4584fb84:::benchmark_data_smithi194_103039_object105222:head# 2024-04-29T14:21:17.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458578dd:::benchmark_data_smithi194_103039_object97951:head# 2024-04-29T14:21:17.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4585c3d8:::benchmark_data_smithi194_103039_object72784:head# 2024-04-29T14:21:17.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4586acbd:::benchmark_data_smithi194_103039_object43557:head# 2024-04-29T14:21:17.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45872520:::benchmark_data_smithi194_103039_object68682:head# 2024-04-29T14:21:17.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458735cc:::benchmark_data_smithi194_103039_object86034:head# 2024-04-29T14:21:17.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45873df2:::benchmark_data_smithi194_103039_object105715:head# 2024-04-29T14:21:17.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45877486:::benchmark_data_smithi194_103039_object66319:head# 2024-04-29T14:21:17.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45883ad3:::benchmark_data_smithi194_103039_object43620:head# 2024-04-29T14:21:17.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45893df8:::benchmark_data_smithi194_103039_object134442:head# 2024-04-29T14:21:17.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4589fcd9:::benchmark_data_smithi194_103039_object62581:head# 2024-04-29T14:21:17.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458b23b7:::benchmark_data_smithi194_103039_object122547:head# 2024-04-29T14:21:17.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458c159d:::benchmark_data_smithi194_103039_object86939:head# 2024-04-29T14:21:17.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458da5c4:::benchmark_data_smithi194_103039_object153201:head# 2024-04-29T14:21:17.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458e1d97:::benchmark_data_smithi194_103039_object69927:head# 2024-04-29T14:21:17.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:458ea978:::benchmark_data_smithi194_103039_object63853:head# 2024-04-29T14:21:17.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459091fa:::benchmark_data_smithi194_103039_object78784:head# 2024-04-29T14:21:17.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4590c523:::benchmark_data_smithi194_103039_object55702:head# 2024-04-29T14:21:17.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45917735:::benchmark_data_smithi194_103039_object84867:head# 2024-04-29T14:21:17.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4591cfab:::benchmark_data_smithi194_103039_object50671:head# 2024-04-29T14:21:17.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4592f198:::benchmark_data_smithi194_103039_object153676:head# 2024-04-29T14:21:17.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4593a300:::benchmark_data_smithi194_103039_object71908:head# 2024-04-29T14:21:17.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45947b21:::benchmark_data_smithi194_103039_object45353:head# 2024-04-29T14:21:17.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4595509b:::benchmark_data_smithi194_103039_object103930:head# 2024-04-29T14:21:17.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45955675:::benchmark_data_smithi194_103039_object79323:head# 2024-04-29T14:21:17.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4595d2bb:::benchmark_data_smithi194_103039_object91566:head# 2024-04-29T14:21:17.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459643ae:::benchmark_data_smithi194_103039_object52153:head# 2024-04-29T14:21:17.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45966348:::benchmark_data_smithi194_103039_object82444:head# 2024-04-29T14:21:17.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45966d91:::benchmark_data_smithi194_103039_object115328:head# 2024-04-29T14:21:17.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4596867f:::benchmark_data_smithi194_103039_object108416:head# 2024-04-29T14:21:17.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4597a3e0:::benchmark_data_smithi194_103039_object147486:head# 2024-04-29T14:21:17.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4599b82d:::benchmark_data_smithi194_103039_object123726:head# 2024-04-29T14:21:17.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4599f1ec:::benchmark_data_smithi194_103039_object123120:head# 2024-04-29T14:21:17.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459a5566:::benchmark_data_smithi194_103039_object75041:head# 2024-04-29T14:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459ad71d:::benchmark_data_smithi194_103039_object55373:head# 2024-04-29T14:21:17.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459b2dac:::benchmark_data_smithi194_103039_object114554:head# 2024-04-29T14:21:17.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459bac8b:::benchmark_data_smithi194_103039_object72592:head# 2024-04-29T14:21:17.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459d97c0:::benchmark_data_smithi194_103039_object63115:head# 2024-04-29T14:21:17.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459f0aa3:::benchmark_data_smithi194_103039_object143260:head# 2024-04-29T14:21:17.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459f4076:::benchmark_data_smithi194_103039_object42179:head# 2024-04-29T14:21:17.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:459f98a3:::benchmark_data_smithi194_103039_object103858:head# 2024-04-29T14:21:17.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a0b83b:::benchmark_data_smithi194_103039_object62663:head# 2024-04-29T14:21:17.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a0b9df:::benchmark_data_smithi194_103039_object77997:head# 2024-04-29T14:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a0cd69:::benchmark_data_smithi194_103039_object122130:head# 2024-04-29T14:21:17.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a20053:::benchmark_data_smithi194_103039_object125642:head# 2024-04-29T14:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a2d879:::benchmark_data_smithi194_103039_object149765:head# 2024-04-29T14:21:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a46cb1:::benchmark_data_smithi194_103039_object49721:head# 2024-04-29T14:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a47831:::benchmark_data_smithi194_103039_object43684:head# 2024-04-29T14:21:17.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a4817c:::benchmark_data_smithi194_103039_object84379:head# 2024-04-29T14:21:17.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a4de15:::benchmark_data_smithi194_103039_object77041:head# 2024-04-29T14:21:17.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a506f1:::benchmark_data_smithi194_103039_object96211:head# 2024-04-29T14:21:17.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a62120:::benchmark_data_smithi194_103039_object147877:head# 2024-04-29T14:21:17.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a66ace:::benchmark_data_smithi194_103039_object95671:head# 2024-04-29T14:21:17.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a73dc4:::benchmark_data_smithi194_103039_object57769:head# 2024-04-29T14:21:17.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a7bce6:::benchmark_data_smithi194_103039_object72109:head# 2024-04-29T14:21:17.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a8e1d2:::benchmark_data_smithi194_103039_object112371:head# 2024-04-29T14:21:17.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a92480:::benchmark_data_smithi194_103039_object150415:head# 2024-04-29T14:21:17.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45a9b087:::benchmark_data_smithi194_103039_object49446:head# 2024-04-29T14:21:17.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45aa2e08:::benchmark_data_smithi194_103039_object63485:head# 2024-04-29T14:21:17.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ad027b:::benchmark_data_smithi194_103039_object152105:head# 2024-04-29T14:21:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45adb979:::benchmark_data_smithi194_103039_object54915:head# 2024-04-29T14:21:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45add23a:::benchmark_data_smithi194_103039_object87952:head# 2024-04-29T14:21:17.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ae5c75:::benchmark_data_smithi194_103039_object114467:head# 2024-04-29T14:21:17.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b010c8:::benchmark_data_smithi194_103039_object144312:head# 2024-04-29T14:21:17.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b02033:::benchmark_data_smithi194_103039_object110928:head# 2024-04-29T14:21:17.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b02b78:::benchmark_data_smithi194_103039_object141367:head# 2024-04-29T14:21:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b1a44e:::benchmark_data_smithi194_103039_object113469:head# 2024-04-29T14:21:17.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b1fd0b:::benchmark_data_smithi194_103039_object84946:head# 2024-04-29T14:21:17.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b33444:::benchmark_data_smithi194_103039_object41619:head# 2024-04-29T14:21:17.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b4286d:::benchmark_data_smithi194_103039_object122242:head# 2024-04-29T14:21:17.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b49f04:::benchmark_data_smithi194_103039_object142486:head# 2024-04-29T14:21:17.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b4fea2:::benchmark_data_smithi194_103039_object55406:head# 2024-04-29T14:21:17.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b55b2d:::benchmark_data_smithi194_103039_object107263:head# 2024-04-29T14:21:17.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b58bb6:::benchmark_data_smithi194_103039_object119721:head# 2024-04-29T14:21:17.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b5e620:::benchmark_data_smithi194_103039_object88981:head# 2024-04-29T14:21:17.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b758cb:::benchmark_data_smithi194_103039_object62339:head# 2024-04-29T14:21:17.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b86591:::benchmark_data_smithi194_103039_object67117:head# 2024-04-29T14:21:17.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45b979b4:::benchmark_data_smithi194_103039_object63826:head# 2024-04-29T14:21:17.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45bec464:::benchmark_data_smithi194_103039_object53791:head# 2024-04-29T14:21:17.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45bf2574:::benchmark_data_smithi194_103039_object83198:head# 2024-04-29T14:21:17.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45bf6fc5:::benchmark_data_smithi194_103039_object91482:head# 2024-04-29T14:21:17.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c01380:::benchmark_data_smithi194_103039_object88941:head# 2024-04-29T14:21:17.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c05499:::benchmark_data_smithi194_103039_object88474:head# 2024-04-29T14:21:17.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c11e0c:::benchmark_data_smithi194_103039_object155561:head# 2024-04-29T14:21:17.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c14ea7:::benchmark_data_smithi194_103039_object152714:head# 2024-04-29T14:21:17.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c1775b:::benchmark_data_smithi194_103039_object142744:head# 2024-04-29T14:21:17.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c1a790:::benchmark_data_smithi194_103039_object116989:head# 2024-04-29T14:21:17.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c28c74:::benchmark_data_smithi194_103039_object68679:head# 2024-04-29T14:21:17.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c46549:::benchmark_data_smithi194_103039_object112699:head# 2024-04-29T14:21:17.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c5372b:::benchmark_data_smithi194_103039_object83654:head# 2024-04-29T14:21:17.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c5675b:::benchmark_data_smithi194_103039_object98914:head# 2024-04-29T14:21:17.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c64325:::benchmark_data_smithi194_103039_object45450:head# 2024-04-29T14:21:17.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c7e6a2:::benchmark_data_smithi194_103039_object103602:head# 2024-04-29T14:21:17.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c88762:::benchmark_data_smithi194_103039_object134926:head# 2024-04-29T14:21:17.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c8a6c7:::benchmark_data_smithi194_103039_object75590:head# 2024-04-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c913aa:::benchmark_data_smithi194_103039_object65874:head# 2024-04-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c93692:::benchmark_data_smithi194_103039_object85155:head# 2024-04-29T14:21:17.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45c9d1a7:::benchmark_data_smithi194_103039_object94688:head# 2024-04-29T14:21:17.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ca2a6c:::benchmark_data_smithi194_103039_object137108:head# 2024-04-29T14:21:17.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ca7968:::benchmark_data_smithi194_103039_object152364:head# 2024-04-29T14:21:17.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45cbe818:::benchmark_data_smithi194_103039_object100422:head# 2024-04-29T14:21:17.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45cca9c2:::benchmark_data_smithi194_103039_object95719:head# 2024-04-29T14:21:17.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45cd495d:::benchmark_data_smithi194_103039_object119771:head# 2024-04-29T14:21:17.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45cd92aa:::benchmark_data_smithi194_103039_object55145:head# 2024-04-29T14:21:17.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45cdb9f7:::benchmark_data_smithi194_103039_object121566:head# 2024-04-29T14:21:17.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ce65a8:::benchmark_data_smithi194_103039_object61853:head# 2024-04-29T14:21:17.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ce74cb:::benchmark_data_smithi194_103039_object104492:head# 2024-04-29T14:21:17.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45cf01f5:::benchmark_data_smithi194_103039_object77590:head# 2024-04-29T14:21:17.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45cf23fe:::benchmark_data_smithi194_103039_object63867:head# 2024-04-29T14:21:17.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d0c155:::benchmark_data_smithi194_103039_object103003:head# 2024-04-29T14:21:17.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d111b9:::benchmark_data_smithi194_103039_object109940:head# 2024-04-29T14:21:17.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d42b47:::benchmark_data_smithi194_103039_object146496:head# 2024-04-29T14:21:17.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d56a5a:::benchmark_data_smithi194_103039_object99837:head# 2024-04-29T14:21:17.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d647ff:::benchmark_data_smithi194_103039_object148107:head# 2024-04-29T14:21:17.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d6e811:::benchmark_data_smithi194_103039_object129210:head# 2024-04-29T14:21:17.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d76723:::benchmark_data_smithi194_103039_object137220:head# 2024-04-29T14:21:17.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d7db29:::benchmark_data_smithi194_103039_object153235:head# 2024-04-29T14:21:17.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45d8f62c:::benchmark_data_smithi194_103039_object44718:head# 2024-04-29T14:21:17.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45da0205:::benchmark_data_smithi194_103039_object76843:head# 2024-04-29T14:21:17.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45da518c:::benchmark_data_smithi194_103039_object48963:head# 2024-04-29T14:21:17.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45da6657:::benchmark_data_smithi194_103039_object114077:head# 2024-04-29T14:21:17.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45db1335:::benchmark_data_smithi194_103039_object111185:head# 2024-04-29T14:21:17.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45dbbdde:::benchmark_data_smithi194_103039_object48187:head# 2024-04-29T14:21:17.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45dc3cdb:::benchmark_data_smithi194_103039_object49388:head# 2024-04-29T14:21:17.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45dcd8b7:::benchmark_data_smithi194_103039_object93576:head# 2024-04-29T14:21:17.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45dda427:::benchmark_data_smithi194_103039_object57629:head# 2024-04-29T14:21:17.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ddcf62:::benchmark_data_smithi194_103039_object100121:head# 2024-04-29T14:21:17.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ddedea:::benchmark_data_smithi194_103039_object95969:head# 2024-04-29T14:21:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45dff95e:::benchmark_data_smithi194_103039_object112419:head# 2024-04-29T14:21:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e010b1:::benchmark_data_smithi194_103039_object130278:head# 2024-04-29T14:21:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e02af6:::benchmark_data_smithi194_103039_object110386:head# 2024-04-29T14:21:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e0342f:::benchmark_data_smithi194_103039_object91120:head# 2024-04-29T14:21:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e0ba81:::benchmark_data_smithi194_103039_object91102:head# 2024-04-29T14:21:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e100c4:::benchmark_data_smithi194_103039_object132898:head# 2024-04-29T14:21:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e1d7cb:::benchmark_data_smithi194_103039_object103920:head# 2024-04-29T14:21:17.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e20b08:::benchmark_data_smithi194_103039_object76240:head# 2024-04-29T14:21:17.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e2103f:::benchmark_data_smithi194_103039_object118804:head# 2024-04-29T14:21:17.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e2661a:::benchmark_data_smithi194_103039_object82330:head# 2024-04-29T14:21:17.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e2bdcf:::benchmark_data_smithi194_103039_object69195:head# 2024-04-29T14:21:17.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e339b3:::benchmark_data_smithi194_103039_object148020:head# 2024-04-29T14:21:17.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e33be8:::benchmark_data_smithi194_103039_object70184:head# 2024-04-29T14:21:17.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e3ae20:::benchmark_data_smithi194_103039_object73217:head# 2024-04-29T14:21:17.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e4d802:::benchmark_data_smithi194_103039_object141226:head# 2024-04-29T14:21:17.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e52179:::benchmark_data_smithi194_103039_object65149:head# 2024-04-29T14:21:17.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e52a15:::benchmark_data_smithi194_103039_object152476:head# 2024-04-29T14:21:17.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e54a64:::benchmark_data_smithi194_103039_object69103:head# 2024-04-29T14:21:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e65a10:::benchmark_data_smithi194_103039_object152166:head# 2024-04-29T14:21:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e69e31:::benchmark_data_smithi194_103039_object143263:head# 2024-04-29T14:21:17.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45e93a8a:::benchmark_data_smithi194_103039_object99240:head# 2024-04-29T14:21:17.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ea9768:::benchmark_data_smithi194_103039_object51973:head# 2024-04-29T14:21:17.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45eba889:::benchmark_data_smithi194_103039_object57358:head# 2024-04-29T14:21:17.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ec595c:::benchmark_data_smithi194_103039_object119472:head# 2024-04-29T14:21:17.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ed2356:::benchmark_data_smithi194_103039_object59776:head# 2024-04-29T14:21:17.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ee1efe:::benchmark_data_smithi194_103039_object97247:head# 2024-04-29T14:21:17.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ee4286:::benchmark_data_smithi194_103039_object104863:head# 2024-04-29T14:21:17.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45eecf48:::benchmark_data_smithi194_103039_object73970:head# 2024-04-29T14:21:17.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45eed05b:::benchmark_data_smithi194_103039_object84646:head# 2024-04-29T14:21:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ef6d0c:::benchmark_data_smithi194_103039_object97664:head# 2024-04-29T14:21:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ef7a19:::benchmark_data_smithi194_103039_object88191:head# 2024-04-29T14:21:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ef9be8:::benchmark_data_smithi194_103039_object91409:head# 2024-04-29T14:21:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f0ea24:::benchmark_data_smithi194_103039_object114995:head# 2024-04-29T14:21:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f13043:::benchmark_data_smithi194_103039_object155410:head# 2024-04-29T14:21:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f15698:::benchmark_data_smithi194_103039_object54657:head# 2024-04-29T14:21:17.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f17329:::benchmark_data_smithi194_103039_object115059:head# 2024-04-29T14:21:17.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f29839:::benchmark_data_smithi194_103039_object114451:head# 2024-04-29T14:21:17.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f2b321:::benchmark_data_smithi194_103039_object113031:head# 2024-04-29T14:21:17.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f2b915:::benchmark_data_smithi194_103039_object147299:head# 2024-04-29T14:21:17.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f35939:::benchmark_data_smithi194_103039_object80893:head# 2024-04-29T14:21:17.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f4a6e5:::benchmark_data_smithi194_103039_object110575:head# 2024-04-29T14:21:17.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f4be86:::benchmark_data_smithi194_103039_object103108:head# 2024-04-29T14:21:17.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f5ab4a:::benchmark_data_smithi194_103039_object88829:head# 2024-04-29T14:21:17.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f5dd4e:::benchmark_data_smithi194_103039_object75343:head# 2024-04-29T14:21:17.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f5fb18:::benchmark_data_smithi194_103039_object87956:head# 2024-04-29T14:21:17.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f6b228:::benchmark_data_smithi194_103039_object108350:head# 2024-04-29T14:21:17.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f72a21:::benchmark_data_smithi194_103039_object101864:head# 2024-04-29T14:21:17.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f7ba43:::benchmark_data_smithi194_103039_object143854:head# 2024-04-29T14:21:17.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f7e2ec:::benchmark_data_smithi194_103039_object58326:head# 2024-04-29T14:21:17.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f881c6:::benchmark_data_smithi194_103039_object61231:head# 2024-04-29T14:21:17.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f93956:::benchmark_data_smithi194_103039_object91040:head# 2024-04-29T14:21:17.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f953a1:::benchmark_data_smithi194_103039_object92091:head# 2024-04-29T14:21:17.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f986ee:::benchmark_data_smithi194_103039_object106562:head# 2024-04-29T14:21:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f9a351:::benchmark_data_smithi194_103039_object142650:head# 2024-04-29T14:21:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45f9b5f2:::benchmark_data_smithi194_103039_object139663:head# 2024-04-29T14:21:17.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45faaabc:::benchmark_data_smithi194_103039_object74029:head# 2024-04-29T14:21:17.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45fac750:::benchmark_data_smithi194_103039_object42016:head# 2024-04-29T14:21:17.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45fb3bb9:::benchmark_data_smithi194_103039_object109335:head# 2024-04-29T14:21:17.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45fb529c:::benchmark_data_smithi194_103039_object91629:head# 2024-04-29T14:21:17.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45fc96bf:::benchmark_data_smithi194_103039_object66015:head# 2024-04-29T14:21:17.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45fd2fb7:::benchmark_data_smithi194_103039_object136395:head# 2024-04-29T14:21:17.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ff703d:::benchmark_data_smithi194_103039_object146134:head# 2024-04-29T14:21:17.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ff89a7:::benchmark_data_smithi194_103039_object154147:head# 2024-04-29T14:21:17.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ffba9d:::benchmark_data_smithi194_103039_object96047:head# 2024-04-29T14:21:17.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:45ffdb19:::benchmark_data_smithi194_103039_object67031:head# 2024-04-29T14:21:17.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460064ff:::benchmark_data_smithi194_103039_object70626:head# 2024-04-29T14:21:17.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4600faa9:::benchmark_data_smithi194_103039_object101135:head# 2024-04-29T14:21:17.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460114dd:::benchmark_data_smithi194_103039_object82963:head# 2024-04-29T14:21:17.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4601291a:::benchmark_data_smithi194_103039_object57303:head# 2024-04-29T14:21:17.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460191fa:::benchmark_data_smithi194_103039_object104890:head# 2024-04-29T14:21:17.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4601fdcc:::benchmark_data_smithi194_103039_object98125:head# 2024-04-29T14:21:17.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460247cd:::benchmark_data_smithi194_103039_object129126:head# 2024-04-29T14:21:17.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46028d32:::benchmark_data_smithi194_103039_object73547:head# 2024-04-29T14:21:17.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4602dc81:::benchmark_data_smithi194_103039_object147923:head# 2024-04-29T14:21:17.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460322de:::benchmark_data_smithi194_103039_object93533:head# 2024-04-29T14:21:17.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46039fe8:::benchmark_data_smithi194_103039_object138435:head# 2024-04-29T14:21:17.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4603b3b6:::benchmark_data_smithi194_103039_object46554:head# 2024-04-29T14:21:17.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4603b55f:::benchmark_data_smithi194_103039_object143999:head# 2024-04-29T14:21:17.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4604465f:::benchmark_data_smithi194_103039_object91291:head# 2024-04-29T14:21:17.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460458fc:::benchmark_data_smithi194_103039_object124908:head# 2024-04-29T14:21:17.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46054caf:::benchmark_data_smithi194_103039_object135773:head# 2024-04-29T14:21:17.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46055fbe:::benchmark_data_smithi194_103039_object135161:head# 2024-04-29T14:21:17.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46059601:::benchmark_data_smithi194_103039_object142091:head# 2024-04-29T14:21:17.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4605d9cc:::benchmark_data_smithi194_103039_object76157:head# 2024-04-29T14:21:17.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4606a60e:::benchmark_data_smithi194_103039_object147177:head# 2024-04-29T14:21:17.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46078325:::benchmark_data_smithi194_103039_object61690:head# 2024-04-29T14:21:17.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46083553:::benchmark_data_smithi194_103039_object114718:head# 2024-04-29T14:21:17.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46083a92:::benchmark_data_smithi194_103039_object92021:head# 2024-04-29T14:21:17.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46087280:::benchmark_data_smithi194_103039_object94121:head# 2024-04-29T14:21:17.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460913ae:::benchmark_data_smithi194_103039_object107842:head# 2024-04-29T14:21:17.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460922b5:::benchmark_data_smithi194_103039_object150438:head# 2024-04-29T14:21:17.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4609ac64:::benchmark_data_smithi194_103039_object137362:head# 2024-04-29T14:21:17.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4609d7dc:::benchmark_data_smithi194_103039_object127776:head# 2024-04-29T14:21:17.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460a579f:::benchmark_data_smithi194_103039_object90346:head# 2024-04-29T14:21:17.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460a8cea:::benchmark_data_smithi194_103039_object103258:head# 2024-04-29T14:21:17.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460a9e60:::benchmark_data_smithi194_103039_object131236:head# 2024-04-29T14:21:17.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460b6643:::benchmark_data_smithi194_103039_object46285:head# 2024-04-29T14:21:17.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460ca2c3:::benchmark_data_smithi194_103039_object146906:head# 2024-04-29T14:21:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460d1d91:::benchmark_data_smithi194_103039_object118342:head# 2024-04-29T14:21:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460e3625:::benchmark_data_smithi194_103039_object50255:head# 2024-04-29T14:21:17.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460e4a7e:::benchmark_data_smithi194_103039_object69189:head# 2024-04-29T14:21:17.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460efb10:::benchmark_data_smithi194_103039_object58711:head# 2024-04-29T14:21:17.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460f2de0:::benchmark_data_smithi194_103039_object53388:head# 2024-04-29T14:21:17.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460f81d4:::benchmark_data_smithi194_103039_object46155:head# 2024-04-29T14:21:17.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:460fa0db:::benchmark_data_smithi194_103039_object82641:head# 2024-04-29T14:21:17.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4611d102:::benchmark_data_smithi194_103039_object146687:head# 2024-04-29T14:21:17.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46120ebb:::benchmark_data_smithi194_103039_object69133:head# 2024-04-29T14:21:17.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4612be74:::benchmark_data_smithi194_103039_object121197:head# 2024-04-29T14:21:17.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4612df81:::benchmark_data_smithi194_103039_object76339:head# 2024-04-29T14:21:17.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4613971c:::benchmark_data_smithi194_103039_object144218:head# 2024-04-29T14:21:17.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4613aa9f:::benchmark_data_smithi194_103039_object132750:head# 2024-04-29T14:21:17.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4613bd3f:::benchmark_data_smithi194_103039_object85455:head# 2024-04-29T14:21:17.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4613d511:::benchmark_data_smithi194_103039_object155301:head# 2024-04-29T14:21:17.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4614231a:::benchmark_data_smithi194_103039_object80339:head# 2024-04-29T14:21:17.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4614e4d5:::benchmark_data_smithi194_103039_object110310:head# 2024-04-29T14:21:17.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4615a591:::benchmark_data_smithi194_103039_object106801:head# 2024-04-29T14:21:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4615da38:::benchmark_data_smithi194_103039_object53552:head# 2024-04-29T14:21:17.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4617913a:::benchmark_data_smithi194_103039_object116734:head# 2024-04-29T14:21:17.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4617f4d0:::benchmark_data_smithi194_103039_object48924:head# 2024-04-29T14:21:17.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461850b8:::benchmark_data_smithi194_103039_object149640:head# 2024-04-29T14:21:17.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461876e0:::benchmark_data_smithi194_103039_object79222:head# 2024-04-29T14:21:17.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461928a6:::benchmark_data_smithi194_103039_object76971:head# 2024-04-29T14:21:17.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46193a58:::benchmark_data_smithi194_103039_object133378:head# 2024-04-29T14:21:17.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46194a22:::benchmark_data_smithi194_103039_object84736:head# 2024-04-29T14:21:17.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46197335:::benchmark_data_smithi194_103039_object47061:head# 2024-04-29T14:21:17.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461a59f9:::benchmark_data_smithi194_103039_object116259:head# 2024-04-29T14:21:17.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461adea6:::benchmark_data_smithi194_103039_object92752:head# 2024-04-29T14:21:17.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461af6f2:::benchmark_data_smithi194_103039_object66326:head# 2024-04-29T14:21:17.738 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:17.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461b2146:::benchmark_data_smithi194_103039_object65333:head# 2024-04-29T14:21:17.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461babeb:::benchmark_data_smithi194_103039_object58978:head# 2024-04-29T14:21:17.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461c0528:::benchmark_data_smithi194_103039_object109284:head# 2024-04-29T14:21:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:17.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461c4e41:::benchmark_data_smithi194_103039_object134405:head# 2024-04-29T14:21:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461c51a9:::benchmark_data_smithi194_103039_object122362:head# 2024-04-29T14:21:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461e39f1:::benchmark_data_smithi194_103039_object116446:head# 2024-04-29T14:21:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461ee149:::benchmark_data_smithi194_103039_object79327:head# 2024-04-29T14:21:17.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461faaf0:::benchmark_data_smithi194_103039_object131833:head# 2024-04-29T14:21:17.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461fac10:::benchmark_data_smithi194_103039_object153381:head# 2024-04-29T14:21:17.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461fc856:::benchmark_data_smithi194_103039_object116388:head# 2024-04-29T14:21:17.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:461ff738:::benchmark_data_smithi194_103039_object107311:head# 2024-04-29T14:21:17.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46205b44:::benchmark_data_smithi194_103039_object54505:head# 2024-04-29T14:21:17.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4620e42a:::benchmark_data_smithi194_103039_object145319:head# 2024-04-29T14:21:17.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462321f6:::benchmark_data_smithi194_103039_object149873:head# 2024-04-29T14:21:17.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46243cbf:::benchmark_data_smithi194_103039_object73595:head# 2024-04-29T14:21:17.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4624424f:::benchmark_data_smithi194_103039_object143959:head# 2024-04-29T14:21:17.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4624cc30:::benchmark_data_smithi194_103039_object131808:head# 2024-04-29T14:21:17.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46252ea4:::benchmark_data_smithi194_103039_object53060:head# 2024-04-29T14:21:17.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4625ab7f:::benchmark_data_smithi194_103039_object107757:head# 2024-04-29T14:21:17.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4625bbc6:::benchmark_data_smithi194_103039_object106659:head# 2024-04-29T14:21:17.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46265a7e:::benchmark_data_smithi194_103039_object140642:head# 2024-04-29T14:21:17.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4626670b:::benchmark_data_smithi194_103039_object59639:head# 2024-04-29T14:21:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46279c4c:::benchmark_data_smithi194_103039_object108974:head# 2024-04-29T14:21:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46280f80:::benchmark_data_smithi194_103039_object67716:head# 2024-04-29T14:21:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462835bf:::benchmark_data_smithi194_103039_object107651:head# 2024-04-29T14:21:17.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46287429:::benchmark_data_smithi194_103039_object90356:head# 2024-04-29T14:21:17.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4629a1b2:::benchmark_data_smithi194_103039_object109018:head# 2024-04-29T14:21:17.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462b358d:::benchmark_data_smithi194_103039_object55250:head# 2024-04-29T14:21:17.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462c1674:::benchmark_data_smithi194_103039_object124472:head# 2024-04-29T14:21:17.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462cbbbd:::benchmark_data_smithi194_103039_object134177:head# 2024-04-29T14:21:17.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462ebadf:::benchmark_data_smithi194_103039_object73543:head# 2024-04-29T14:21:17.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462ef89f:::benchmark_data_smithi194_103039_object112456:head# 2024-04-29T14:21:17.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462f383e:::benchmark_data_smithi194_103039_object85058:head# 2024-04-29T14:21:17.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462f40c0:::benchmark_data_smithi194_103039_object81446:head# 2024-04-29T14:21:17.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462fab42:::benchmark_data_smithi194_103039_object99604:head# 2024-04-29T14:21:17.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:462fb11a:::benchmark_data_smithi194_103039_object98503:head# 2024-04-29T14:21:17.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46304907:::benchmark_data_smithi194_103039_object138696:head# 2024-04-29T14:21:17.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4630b8e4:::benchmark_data_smithi194_103039_object117233:head# 2024-04-29T14:21:17.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4630d1a3:::benchmark_data_smithi194_103039_object111263:head# 2024-04-29T14:21:17.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4631d1cf:::benchmark_data_smithi194_103039_object56603:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4632f51c:::benchmark_data_smithi194_103039_object120751:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4633c39c:::benchmark_data_smithi194_103039_object108958:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463496cd:::benchmark_data_smithi194_103039_object59591:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4634a4d7:::benchmark_data_smithi194_103039_object59013:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4634aa43:::benchmark_data_smithi194_103039_object105632:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4634bad7:::benchmark_data_smithi194_103039_object153447:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46377dd6:::benchmark_data_smithi194_103039_object136435:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46381a2d:::benchmark_data_smithi194_103039_object68868:head# 2024-04-29T14:21:17.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4638644e:::benchmark_data_smithi194_103039_object147508:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4638716a:::benchmark_data_smithi194_103039_object102292:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4638f814:::benchmark_data_smithi194_103039_object88162:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46396256:::benchmark_data_smithi194_103039_object71148:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4639e54c:::benchmark_data_smithi194_103039_object129936:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463a0dcd:::benchmark_data_smithi194_103039_object69148:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463a1257:::benchmark_data_smithi194_103039_object73993:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463b1201:::benchmark_data_smithi194_103039_object106268:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463b2b7d:::benchmark_data_smithi194_103039_object113624:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463be365:::benchmark_data_smithi194_103039_object133367:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463c4cf9:::benchmark_data_smithi194_103039_object139828:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463d7794:::benchmark_data_smithi194_103039_object134663:head# 2024-04-29T14:21:17.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463d8204:::benchmark_data_smithi194_103039_object118163:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463dfd68:::benchmark_data_smithi194_103039_object133674:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463e5dd3:::benchmark_data_smithi194_103039_object110020:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463f2c09:::benchmark_data_smithi194_103039_object156485:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:463fee02:::benchmark_data_smithi194_103039_object152195:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464042d5:::benchmark_data_smithi194_103039_object152944:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464095d4:::benchmark_data_smithi194_103039_object126229:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46409677:::benchmark_data_smithi194_103039_object130225:head# 2024-04-29T14:21:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4640f2ec:::benchmark_data_smithi194_103039_object105634:head# 2024-04-29T14:21:17.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4642b339:::benchmark_data_smithi194_103039_object132044:head# 2024-04-29T14:21:17.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46433135:::benchmark_data_smithi194_103039_object64418:head# 2024-04-29T14:21:17.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46439698:::benchmark_data_smithi194_103039_object143148:head# 2024-04-29T14:21:17.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464420e7:::benchmark_data_smithi194_103039_object106451:head# 2024-04-29T14:21:17.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46451ba7:::benchmark_data_smithi194_103039_object73184:head# 2024-04-29T14:21:17.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464568aa:::benchmark_data_smithi194_103039_object41806:head# 2024-04-29T14:21:17.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46457e3d:::benchmark_data_smithi194_103039_object47328:head# 2024-04-29T14:21:17.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46468a2a:::benchmark_data_smithi194_103039_object60935:head# 2024-04-29T14:21:17.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464772bb:::benchmark_data_smithi194_103039_object116572:head# 2024-04-29T14:21:17.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4647bda5:::benchmark_data_smithi194_103039_object120065:head# 2024-04-29T14:21:17.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46499a15:::benchmark_data_smithi194_103039_object99609:head# 2024-04-29T14:21:17.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4649c642:::benchmark_data_smithi194_103039_object123284:head# 2024-04-29T14:21:17.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4649cf71:::benchmark_data_smithi194_103039_object156498:head# 2024-04-29T14:21:17.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464ae407:::benchmark_data_smithi194_103039_object119476:head# 2024-04-29T14:21:17.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464b3e6c:::benchmark_data_smithi194_103039_object104923:head# 2024-04-29T14:21:17.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464b5bf0:::benchmark_data_smithi194_103039_object60221:head# 2024-04-29T14:21:17.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464ca905:::benchmark_data_smithi194_103039_object44402:head# 2024-04-29T14:21:17.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464d0980:::benchmark_data_smithi194_103039_object62360:head# 2024-04-29T14:21:17.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464d8041:::benchmark_data_smithi194_103039_object104073:head# 2024-04-29T14:21:17.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464e5cd3:::benchmark_data_smithi194_103039_object155625:head# 2024-04-29T14:21:17.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464e5dae:::benchmark_data_smithi194_103039_object120465:head# 2024-04-29T14:21:17.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464f3964:::benchmark_data_smithi194_103039_object68971:head# 2024-04-29T14:21:17.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464fc9a0:::benchmark_data_smithi194_103039_object63319:head# 2024-04-29T14:21:17.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:464fcc95:::benchmark_data_smithi194_103039_object41882:head# 2024-04-29T14:21:17.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46503bfc:::benchmark_data_smithi194_103039_object147139:head# 2024-04-29T14:21:17.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46507c2c:::benchmark_data_smithi194_103039_object124825:head# 2024-04-29T14:21:17.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46509c79:::benchmark_data_smithi194_103039_object73892:head# 2024-04-29T14:21:17.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46512e4b:::benchmark_data_smithi194_103039_object129623:head# 2024-04-29T14:21:17.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4651d4a5:::benchmark_data_smithi194_103039_object98339:head# 2024-04-29T14:21:17.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46523693:::benchmark_data_smithi194_103039_object151755:head# 2024-04-29T14:21:17.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465273a4:::benchmark_data_smithi194_103039_object53821:head# 2024-04-29T14:21:17.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46532530:::benchmark_data_smithi194_103039_object54373:head# 2024-04-29T14:21:17.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46533e47:::benchmark_data_smithi194_103039_object150359:head# 2024-04-29T14:21:17.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465431e9:::benchmark_data_smithi194_103039_object80078:head# 2024-04-29T14:21:17.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46546729:::benchmark_data_smithi194_103039_object80045:head# 2024-04-29T14:21:17.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4654aaf7:::benchmark_data_smithi194_103039_object55201:head# 2024-04-29T14:21:17.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4655e6ff:::benchmark_data_smithi194_103039_object107682:head# 2024-04-29T14:21:17.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46565cad:::benchmark_data_smithi194_103039_object67288:head# 2024-04-29T14:21:17.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46566f40:::benchmark_data_smithi194_103039_object112155:head# 2024-04-29T14:21:17.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4656a6ce:::benchmark_data_smithi194_103039_object70154:head# 2024-04-29T14:21:17.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465708ae:::benchmark_data_smithi194_103039_object98451:head# 2024-04-29T14:21:17.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46573079:::benchmark_data_smithi194_103039_object51292:head# 2024-04-29T14:21:17.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465817ce:::benchmark_data_smithi194_103039_object52941:head# 2024-04-29T14:21:17.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46586550:::benchmark_data_smithi194_103039_object86481:head# 2024-04-29T14:21:17.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4659346c:::benchmark_data_smithi194_103039_object60825:head# 2024-04-29T14:21:17.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4659f7c3:::benchmark_data_smithi194_103039_object85600:head# 2024-04-29T14:21:17.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465a73b3:::benchmark_data_smithi194_103039_object41847:head# 2024-04-29T14:21:17.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465a8e2f:::benchmark_data_smithi194_103039_object125620:head# 2024-04-29T14:21:17.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465ba320:::benchmark_data_smithi194_103039_object49577:head# 2024-04-29T14:21:17.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465c6f0a:::benchmark_data_smithi194_103039_object109874:head# 2024-04-29T14:21:17.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465d8ddb:::benchmark_data_smithi194_103039_object83419:head# 2024-04-29T14:21:17.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465d939f:::benchmark_data_smithi194_103039_object126745:head# 2024-04-29T14:21:17.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465dbf54:::benchmark_data_smithi194_103039_object135390:head# 2024-04-29T14:21:17.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465dfa2b:::benchmark_data_smithi194_103039_object155105:head# 2024-04-29T14:21:17.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465e4ee3:::benchmark_data_smithi194_103039_object45563:head# 2024-04-29T14:21:17.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465e85e6:::benchmark_data_smithi194_103039_object80135:head# 2024-04-29T14:21:17.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465e979e:::benchmark_data_smithi194_103039_object59783:head# 2024-04-29T14:21:17.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465ede58:::benchmark_data_smithi194_103039_object155075:head# 2024-04-29T14:21:17.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465ef60e:::benchmark_data_smithi194_103039_object95657:head# 2024-04-29T14:21:17.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465f6b8f:::benchmark_data_smithi194_103039_object103420:head# 2024-04-29T14:21:17.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:465fbf39:::benchmark_data_smithi194_103039_object129782:head# 2024-04-29T14:21:17.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46604b15:::benchmark_data_smithi194_103039_object146172:head# 2024-04-29T14:21:17.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4660739a:::benchmark_data_smithi194_103039_object67758:head# 2024-04-29T14:21:17.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466075fc:::benchmark_data_smithi194_103039_object109709:head# 2024-04-29T14:21:17.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4664279a:::benchmark_data_smithi194_103039_object118604:head# 2024-04-29T14:21:17.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46643e6e:::benchmark_data_smithi194_103039_object114243:head# 2024-04-29T14:21:17.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46645049:::benchmark_data_smithi194_103039_object42544:head# 2024-04-29T14:21:17.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4665c46f:::benchmark_data_smithi194_103039_object115997:head# 2024-04-29T14:21:17.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46664d28:::benchmark_data_smithi194_103039_object150526:head# 2024-04-29T14:21:17.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46664dae:::benchmark_data_smithi194_103039_object82663:head# 2024-04-29T14:21:17.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4667b317:::benchmark_data_smithi194_103039_object76289:head# 2024-04-29T14:21:17.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4667cfbf:::benchmark_data_smithi194_103039_object152000:head# 2024-04-29T14:21:17.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46680af5:::benchmark_data_smithi194_103039_object123705:head# 2024-04-29T14:21:17.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4668e461:::benchmark_data_smithi194_103039_object95267:head# 2024-04-29T14:21:17.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4669b92c:::benchmark_data_smithi194_103039_object52671:head# 2024-04-29T14:21:17.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4669eeac:::benchmark_data_smithi194_103039_object133721:head# 2024-04-29T14:21:17.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466ab4b7:::benchmark_data_smithi194_103039_object47661:head# 2024-04-29T14:21:17.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466b5533:::benchmark_data_smithi194_103039_object151564:head# 2024-04-29T14:21:17.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466bb3d1:::benchmark_data_smithi194_103039_object56111:head# 2024-04-29T14:21:17.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466d3791:::benchmark_data_smithi194_103039_object113968:head# 2024-04-29T14:21:17.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466e3ef9:::benchmark_data_smithi194_103039_object71810:head# 2024-04-29T14:21:17.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466e6749:::benchmark_data_smithi194_103039_object117854:head# 2024-04-29T14:21:17.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:466fae79:::benchmark_data_smithi194_103039_object86375:head# 2024-04-29T14:21:17.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46705c02:::benchmark_data_smithi194_103039_object85201:head# 2024-04-29T14:21:17.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4670f98c:::benchmark_data_smithi194_103039_object45945:head# 2024-04-29T14:21:17.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4671d7b2:::benchmark_data_smithi194_103039_object64841:head# 2024-04-29T14:21:17.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467327aa:::benchmark_data_smithi194_103039_object49702:head# 2024-04-29T14:21:17.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46739164:::benchmark_data_smithi194_103039_object49416:head# 2024-04-29T14:21:17.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4673e8de:::benchmark_data_smithi194_103039_object154477:head# 2024-04-29T14:21:17.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46752811:::benchmark_data_smithi194_103039_object145513:head# 2024-04-29T14:21:17.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46760598:::benchmark_data_smithi194_103039_object71512:head# 2024-04-29T14:21:17.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46761d3b:::benchmark_data_smithi194_103039_object121553:head# 2024-04-29T14:21:17.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46762e78:::benchmark_data_smithi194_103039_object68260:head# 2024-04-29T14:21:17.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46764a99:::benchmark_data_smithi194_103039_object125899:head# 2024-04-29T14:21:17.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4677032c:::benchmark_data_smithi194_103039_object105829:head# 2024-04-29T14:21:17.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467728ab:::benchmark_data_smithi194_103039_object63239:head# 2024-04-29T14:21:17.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467841de:::benchmark_data_smithi194_103039_object137662:head# 2024-04-29T14:21:17.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4678c393:::benchmark_data_smithi194_103039_object91973:head# 2024-04-29T14:21:17.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4678c5b5:::benchmark_data_smithi194_103039_object86843:head# 2024-04-29T14:21:17.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46797a51:::benchmark_data_smithi194_103039_object50661:head# 2024-04-29T14:21:17.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4679b482:::benchmark_data_smithi194_103039_object52870:head# 2024-04-29T14:21:17.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467a658b:::benchmark_data_smithi194_103039_object60567:head# 2024-04-29T14:21:17.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467acf94:::benchmark_data_smithi194_103039_object130282:head# 2024-04-29T14:21:17.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467aede1:::benchmark_data_smithi194_103039_object47044:head# 2024-04-29T14:21:17.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467d5076:::benchmark_data_smithi194_103039_object46911:head# 2024-04-29T14:21:17.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467dd57a:::benchmark_data_smithi194_103039_object128372:head# 2024-04-29T14:21:17.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:467f7a43:::benchmark_data_smithi194_103039_object93508:head# 2024-04-29T14:21:17.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4680b925:::benchmark_data_smithi194_103039_object101763:head# 2024-04-29T14:21:17.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4680d7ca:::benchmark_data_smithi194_103039_object112716:head# 2024-04-29T14:21:17.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4680d915:::benchmark_data_smithi194_103039_object151556:head# 2024-04-29T14:21:17.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4681cb51:::benchmark_data_smithi194_103039_object70622:head# 2024-04-29T14:21:17.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4684530e:::benchmark_data_smithi194_103039_object145071:head# 2024-04-29T14:21:17.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4684bf33:::benchmark_data_smithi194_103039_object53305:head# 2024-04-29T14:21:17.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46853aca:::benchmark_data_smithi194_103039_object65398:head# 2024-04-29T14:21:17.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46871095:::benchmark_data_smithi194_103039_object56734:head# 2024-04-29T14:21:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4687733e:::benchmark_data_smithi194_103039_object122286:head# 2024-04-29T14:21:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46877fcd:::benchmark_data_smithi194_103039_object42892:head# 2024-04-29T14:21:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4687a226:::benchmark_data_smithi194_103039_object95938:head# 2024-04-29T14:21:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4687cc89:::benchmark_data_smithi194_103039_object50258:head# 2024-04-29T14:21:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4687cf69:::benchmark_data_smithi194_103039_object94684:head# 2024-04-29T14:21:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468823eb:::benchmark_data_smithi194_103039_object108590:head# 2024-04-29T14:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46888b0c:::benchmark_data_smithi194_103039_object90158:head# 2024-04-29T14:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4688f6e9:::benchmark_data_smithi194_103039_object80497:head# 2024-04-29T14:21:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468949d8:::benchmark_data_smithi194_103039_object100534:head# 2024-04-29T14:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468b737b:::benchmark_data_smithi194_103039_object89737:head# 2024-04-29T14:21:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468c0beb:::benchmark_data_smithi194_103039_object73765:head# 2024-04-29T14:21:17.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468c0d59:::benchmark_data_smithi194_103039_object58785:head# 2024-04-29T14:21:17.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468c7fee:::benchmark_data_smithi194_103039_object63631:head# 2024-04-29T14:21:17.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468ce149:::benchmark_data_smithi194_103039_object111238:head# 2024-04-29T14:21:17.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468d3904:::benchmark_data_smithi194_103039_object62936:head# 2024-04-29T14:21:17.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468dadff:::benchmark_data_smithi194_103039_object150952:head# 2024-04-29T14:21:17.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468ea216:::benchmark_data_smithi194_103039_object122148:head# 2024-04-29T14:21:17.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468ed924:::benchmark_data_smithi194_103039_object49160:head# 2024-04-29T14:21:17.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468f8b29:::benchmark_data_smithi194_103039_object148710:head# 2024-04-29T14:21:17.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:468fbbf7:::benchmark_data_smithi194_103039_object60173:head# 2024-04-29T14:21:17.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469081d1:::benchmark_data_smithi194_103039_object42681:head# 2024-04-29T14:21:17.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46910dec:::benchmark_data_smithi194_103039_object107271:head# 2024-04-29T14:21:17.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4691738c:::benchmark_data_smithi194_103039_object148610:head# 2024-04-29T14:21:17.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46918e31:::benchmark_data_smithi194_103039_object60078:head# 2024-04-29T14:21:17.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46919ae5:::benchmark_data_smithi194_103039_object100598:head# 2024-04-29T14:21:17.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4691f856:::benchmark_data_smithi194_103039_object144330:head# 2024-04-29T14:21:17.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46921ff0:::benchmark_data_smithi194_103039_object103010:head# 2024-04-29T14:21:17.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4692222e:::benchmark_data_smithi194_103039_object72508:head# 2024-04-29T14:21:17.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46932126:::benchmark_data_smithi194_103039_object110087:head# 2024-04-29T14:21:17.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:17.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469422b9:::benchmark_data_smithi194_103039_object85847:head# 2024-04-29T14:21:17.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4694b0ed:::benchmark_data_smithi194_103039_object94598:head# 2024-04-29T14:21:17.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4694b406:::benchmark_data_smithi194_103039_object46699:head# 2024-04-29T14:21:17.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46952b43:::benchmark_data_smithi194_103039_object119849:head# 2024-04-29T14:21:17.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46953c7b:::benchmark_data_smithi194_103039_object120945:head# 2024-04-29T14:21:17.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46959957:::benchmark_data_smithi194_103039_object90482:head# 2024-04-29T14:21:17.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4695f0b6:::benchmark_data_smithi194_103039_object92313:head# 2024-04-29T14:21:17.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46960da8:::benchmark_data_smithi194_103039_object127595:head# 2024-04-29T14:21:17.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4696585b:::benchmark_data_smithi194_103039_object51178:head# 2024-04-29T14:21:17.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46974381:::benchmark_data_smithi194_103039_object68262:head# 2024-04-29T14:21:17.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4697891b:::benchmark_data_smithi194_103039_object109149:head# 2024-04-29T14:21:17.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4697db71:::benchmark_data_smithi194_103039_object114275:head# 2024-04-29T14:21:17.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4697ef10:::benchmark_data_smithi194_103039_object130479:head# 2024-04-29T14:21:17.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46985ee3:::benchmark_data_smithi194_103039_object55554:head# 2024-04-29T14:21:17.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46989d00:::benchmark_data_smithi194_103039_object137762:head# 2024-04-29T14:21:17.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469a826a:::benchmark_data_smithi194_103039_object90423:head# 2024-04-29T14:21:17.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469bed4b:::benchmark_data_smithi194_103039_object51483:head# 2024-04-29T14:21:17.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469d00a6:::benchmark_data_smithi194_103039_object62150:head# 2024-04-29T14:21:17.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469d8dbf:::benchmark_data_smithi194_103039_object97246:head# 2024-04-29T14:21:17.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469daf66:::benchmark_data_smithi194_103039_object138629:head# 2024-04-29T14:21:17.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469dc6da:::benchmark_data_smithi194_103039_object48003:head# 2024-04-29T14:21:17.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469e298e:::benchmark_data_smithi194_103039_object118290:head# 2024-04-29T14:21:17.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469e4c5d:::benchmark_data_smithi194_103039_object125853:head# 2024-04-29T14:21:17.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469e8ea6:::benchmark_data_smithi194_103039_object109746:head# 2024-04-29T14:21:17.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:469f8652:::benchmark_data_smithi194_103039_object67910:head# 2024-04-29T14:21:17.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a05530:::benchmark_data_smithi194_103039_object79260:head# 2024-04-29T14:21:17.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a0eb7c:::benchmark_data_smithi194_103039_object100641:head# 2024-04-29T14:21:17.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a10516:::benchmark_data_smithi194_103039_object110768:head# 2024-04-29T14:21:17.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a13a77:::benchmark_data_smithi194_103039_object97773:head# 2024-04-29T14:21:17.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a1461b:::benchmark_data_smithi194_103039_object116769:head# 2024-04-29T14:21:17.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a1a173:::benchmark_data_smithi194_103039_object44407:head# 2024-04-29T14:21:17.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a1e712:::benchmark_data_smithi194_103039_object58455:head# 2024-04-29T14:21:17.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a1fc02:::benchmark_data_smithi194_103039_object82609:head# 2024-04-29T14:21:17.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a38472:::benchmark_data_smithi194_103039_object89051:head# 2024-04-29T14:21:17.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a5594d:::benchmark_data_smithi194_103039_object117627:head# 2024-04-29T14:21:17.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a5cef4:::benchmark_data_smithi194_103039_object65772:head# 2024-04-29T14:21:17.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a5f4fb:::benchmark_data_smithi194_103039_object126603:head# 2024-04-29T14:21:17.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a709ec:::benchmark_data_smithi194_103039_object70655:head# 2024-04-29T14:21:17.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a8edb4:::benchmark_data_smithi194_103039_object151166:head# 2024-04-29T14:21:17.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a90300:::benchmark_data_smithi194_103039_object94750:head# 2024-04-29T14:21:17.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46a959ad:::benchmark_data_smithi194_103039_object152197:head# 2024-04-29T14:21:17.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46aa0dbe:::benchmark_data_smithi194_103039_object133642:head# 2024-04-29T14:21:17.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ac28cd:::benchmark_data_smithi194_103039_object153596:head# 2024-04-29T14:21:17.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ac34bb:::benchmark_data_smithi194_103039_object114663:head# 2024-04-29T14:21:17.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ac7ee9:::benchmark_data_smithi194_103039_object139443:head# 2024-04-29T14:21:17.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ad9126:::benchmark_data_smithi194_103039_object51986:head# 2024-04-29T14:21:17.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ad934f:::benchmark_data_smithi194_103039_object66564:head# 2024-04-29T14:21:17.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46adc748:::benchmark_data_smithi194_103039_object102707:head# 2024-04-29T14:21:17.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46adfd04:::benchmark_data_smithi194_103039_object42728:head# 2024-04-29T14:21:17.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ae0ad8:::benchmark_data_smithi194_103039_object135985:head# 2024-04-29T14:21:17.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ae11c9:::benchmark_data_smithi194_103039_object96215:head# 2024-04-29T14:21:17.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ae5b0e:::benchmark_data_smithi194_103039_object42019:head# 2024-04-29T14:21:17.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46af1889:::benchmark_data_smithi194_103039_object98471:head# 2024-04-29T14:21:17.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46af2d66:::benchmark_data_smithi194_103039_object145039:head# 2024-04-29T14:21:17.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b00ef6:::benchmark_data_smithi194_103039_object43852:head# 2024-04-29T14:21:17.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b05e5c:::benchmark_data_smithi194_103039_object59159:head# 2024-04-29T14:21:17.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b06472:::benchmark_data_smithi194_103039_object79922:head# 2024-04-29T14:21:17.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b15341:::benchmark_data_smithi194_103039_object42219:head# 2024-04-29T14:21:17.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b17416:::benchmark_data_smithi194_103039_object129769:head# 2024-04-29T14:21:17.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b18d3e:::benchmark_data_smithi194_103039_object132747:head# 2024-04-29T14:21:17.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b1cd44:::benchmark_data_smithi194_103039_object154104:head# 2024-04-29T14:21:17.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b2978c:::benchmark_data_smithi194_103039_object154445:head# 2024-04-29T14:21:17.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b3811f:::benchmark_data_smithi194_103039_object108784:head# 2024-04-29T14:21:17.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b41109:::benchmark_data_smithi194_103039_object125443:head# 2024-04-29T14:21:17.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b47b0c:::benchmark_data_smithi194_103039_object53484:head# 2024-04-29T14:21:17.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b4ddee:::benchmark_data_smithi194_103039_object41748:head# 2024-04-29T14:21:17.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b595f1:::benchmark_data_smithi194_103039_object135810:head# 2024-04-29T14:21:17.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b5c5f3:::benchmark_data_smithi194_103039_object102431:head# 2024-04-29T14:21:17.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b659b8:::benchmark_data_smithi194_103039_object135841:head# 2024-04-29T14:21:17.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b6b104:::benchmark_data_smithi194_103039_object42921:head# 2024-04-29T14:21:17.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b6fb0f:::benchmark_data_smithi194_103039_object151165:head# 2024-04-29T14:21:17.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b7fd3f:::benchmark_data_smithi194_103039_object121296:head# 2024-04-29T14:21:17.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b88f93:::benchmark_data_smithi194_103039_object86259:head# 2024-04-29T14:21:17.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b8ad9c:::benchmark_data_smithi194_103039_object150932:head# 2024-04-29T14:21:17.873 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:17.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46b8f7ec:::benchmark_data_smithi194_103039_object150008:head# 2024-04-29T14:21:17.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ba05a6:::benchmark_data_smithi194_103039_object73873:head# 2024-04-29T14:21:17.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ba10ad:::benchmark_data_smithi194_103039_object154873:head# 2024-04-29T14:21:17.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ba4438:::benchmark_data_smithi194_103039_object73549:head# 2024-04-29T14:21:17.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ba98df:::benchmark_data_smithi194_103039_object79232:head# 2024-04-29T14:21:17.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46bb85ff:::benchmark_data_smithi194_103039_object84029:head# 2024-04-29T14:21:17.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46bbcf22:::benchmark_data_smithi194_103039_object89931:head# 2024-04-29T14:21:17.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46bda9bd:::benchmark_data_smithi194_103039_object134450:head# 2024-04-29T14:21:17.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46be2623:::benchmark_data_smithi194_103039_object68606:head# 2024-04-29T14:21:17.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46bec81c:::benchmark_data_smithi194_103039_object131793:head# 2024-04-29T14:21:17.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46bfacce:::benchmark_data_smithi194_103039_object94264:head# 2024-04-29T14:21:17.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c001fc:::benchmark_data_smithi194_103039_object142036:head# 2024-04-29T14:21:17.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c0587f:::benchmark_data_smithi194_103039_object154249:head# 2024-04-29T14:21:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c0bd1a:::benchmark_data_smithi194_103039_object125451:head# 2024-04-29T14:21:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c0db97:::benchmark_data_smithi194_103039_object110193:head# 2024-04-29T14:21:17.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c1a756:::benchmark_data_smithi194_103039_object63341:head# 2024-04-29T14:21:17.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c285cf:::benchmark_data_smithi194_103039_object51836:head# 2024-04-29T14:21:17.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c2888a:::benchmark_data_smithi194_103039_object84099:head# 2024-04-29T14:21:17.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c30f23:::benchmark_data_smithi194_103039_object47393:head# 2024-04-29T14:21:17.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c3deb4:::benchmark_data_smithi194_103039_object93455:head# 2024-04-29T14:21:17.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c48ce8:::benchmark_data_smithi194_103039_object130605:head# 2024-04-29T14:21:17.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c4d062:::benchmark_data_smithi194_103039_object58443:head# 2024-04-29T14:21:17.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c7167e:::benchmark_data_smithi194_103039_object100535:head# 2024-04-29T14:21:17.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c81985:::benchmark_data_smithi194_103039_object95614:head# 2024-04-29T14:21:17.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c85504:::benchmark_data_smithi194_103039_object57887:head# 2024-04-29T14:21:17.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c92286:::benchmark_data_smithi194_103039_object122283:head# 2024-04-29T14:21:17.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46c9c8ac:::benchmark_data_smithi194_103039_object70948:head# 2024-04-29T14:21:17.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ca2c76:::benchmark_data_smithi194_103039_object79859:head# 2024-04-29T14:21:17.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46cc9c1c:::benchmark_data_smithi194_103039_object139278:head# 2024-04-29T14:21:17.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ccfa40:::benchmark_data_smithi194_103039_object125321:head# 2024-04-29T14:21:17.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46cdb281:::benchmark_data_smithi194_103039_object148776:head# 2024-04-29T14:21:17.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46cdca3c:::benchmark_data_smithi194_103039_object140773:head# 2024-04-29T14:21:17.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ce0c7f:::benchmark_data_smithi194_103039_object106845:head# 2024-04-29T14:21:17.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ce84ec:::benchmark_data_smithi194_103039_object89468:head# 2024-04-29T14:21:17.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46cec54b:::benchmark_data_smithi194_103039_object112565:head# 2024-04-29T14:21:17.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46cf5315:::benchmark_data_smithi194_103039_object107614:head# 2024-04-29T14:21:17.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46cf8030:::benchmark_data_smithi194_103039_object145912:head# 2024-04-29T14:21:17.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d0f6f6:::benchmark_data_smithi194_103039_object117628:head# 2024-04-29T14:21:17.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d13d78:::benchmark_data_smithi194_103039_object109978:head# 2024-04-29T14:21:17.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d15567:::benchmark_data_smithi194_103039_object75136:head# 2024-04-29T14:21:17.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d1f4a6:::benchmark_data_smithi194_103039_object85685:head# 2024-04-29T14:21:17.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d273de:::benchmark_data_smithi194_103039_object154541:head# 2024-04-29T14:21:17.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d2b21b:::benchmark_data_smithi194_103039_object50165:head# 2024-04-29T14:21:17.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d2ded1:::benchmark_data_smithi194_103039_object123891:head# 2024-04-29T14:21:17.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d42acb:::benchmark_data_smithi194_103039_object51805:head# 2024-04-29T14:21:17.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d4b65d:::benchmark_data_smithi194_103039_object151083:head# 2024-04-29T14:21:17.892 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:17.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d4e29f:::benchmark_data_smithi194_103039_object57138:head# 2024-04-29T14:21:17.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d5aac3:::benchmark_data_smithi194_103039_object44646:head# 2024-04-29T14:21:17.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d71c40:::benchmark_data_smithi194_103039_object121817:head# 2024-04-29T14:21:17.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d74eb9:::benchmark_data_smithi194_103039_object112017:head# 2024-04-29T14:21:17.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d7a7c0:::benchmark_data_smithi194_103039_object61223:head# 2024-04-29T14:21:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:17.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:17.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d82d6f:::benchmark_data_smithi194_103039_object115906:head# 2024-04-29T14:21:17.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d96826:::benchmark_data_smithi194_103039_object118867:head# 2024-04-29T14:21:17.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d97ea0:::benchmark_data_smithi194_103039_object155569:head# 2024-04-29T14:21:17.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46d98edc:::benchmark_data_smithi194_103039_object101031:head# 2024-04-29T14:21:17.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46da98b1:::benchmark_data_smithi194_103039_object104788:head# 2024-04-29T14:21:17.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46daaef6:::benchmark_data_smithi194_103039_object142187:head# 2024-04-29T14:21:17.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46db7c6d:::benchmark_data_smithi194_103039_object129408:head# 2024-04-29T14:21:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46dbae89:::benchmark_data_smithi194_103039_object148906:head# 2024-04-29T14:21:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46dbba24:::benchmark_data_smithi194_103039_object153370:head# 2024-04-29T14:21:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46dbe219:::benchmark_data_smithi194_103039_object138837:head# 2024-04-29T14:21:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46dbf3b0:::benchmark_data_smithi194_103039_object117984:head# 2024-04-29T14:21:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46dd60b0:::benchmark_data_smithi194_103039_object140239:head# 2024-04-29T14:21:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46de1c7c:::benchmark_data_smithi194_103039_object103809:head# 2024-04-29T14:21:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46de5c12:::benchmark_data_smithi194_103039_object46620:head# 2024-04-29T14:21:17.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46de76b8:::benchmark_data_smithi194_103039_object143556:head# 2024-04-29T14:21:17.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46dea559:::benchmark_data_smithi194_103039_object51521:head# 2024-04-29T14:21:17.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46df2c81:::benchmark_data_smithi194_103039_object95067:head# 2024-04-29T14:21:17.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e04417:::benchmark_data_smithi194_103039_object89190:head# 2024-04-29T14:21:17.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e0db90:::benchmark_data_smithi194_103039_object128748:head# 2024-04-29T14:21:17.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e145f6:::benchmark_data_smithi194_103039_object49514:head# 2024-04-29T14:21:17.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e19a1d:::benchmark_data_smithi194_103039_object90052:head# 2024-04-29T14:21:17.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e1a357:::benchmark_data_smithi194_103039_object66527:head# 2024-04-29T14:21:17.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e2629b:::benchmark_data_smithi194_103039_object92097:head# 2024-04-29T14:21:17.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e305d6:::benchmark_data_smithi194_103039_object84214:head# 2024-04-29T14:21:17.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e32f7c:::benchmark_data_smithi194_103039_object85906:head# 2024-04-29T14:21:17.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e330e7:::benchmark_data_smithi194_103039_object135311:head# 2024-04-29T14:21:17.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e3a5d2:::benchmark_data_smithi194_103039_object65528:head# 2024-04-29T14:21:17.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e3ff77:::benchmark_data_smithi194_103039_object124386:head# 2024-04-29T14:21:17.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e43382:::benchmark_data_smithi194_103039_object125920:head# 2024-04-29T14:21:17.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e4880a:::benchmark_data_smithi194_103039_object99802:head# 2024-04-29T14:21:17.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e4a23c:::benchmark_data_smithi194_103039_object55610:head# 2024-04-29T14:21:17.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e4af75:::benchmark_data_smithi194_103039_object93676:head# 2024-04-29T14:21:17.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e4ee11:::benchmark_data_smithi194_103039_object134759:head# 2024-04-29T14:21:17.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e6ee5b:::benchmark_data_smithi194_103039_object59046:head# 2024-04-29T14:21:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e6f6cd:::benchmark_data_smithi194_103039_object141674:head# 2024-04-29T14:21:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e7216d:::benchmark_data_smithi194_103039_object50956:head# 2024-04-29T14:21:17.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e76bd2:::benchmark_data_smithi194_103039_object99495:head# 2024-04-29T14:21:17.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e88127:::benchmark_data_smithi194_103039_object94957:head# 2024-04-29T14:21:17.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46e9407d:::benchmark_data_smithi194_103039_object128793:head# 2024-04-29T14:21:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46eaa3b8:::benchmark_data_smithi194_103039_object147003:head# 2024-04-29T14:21:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46eaf537:::benchmark_data_smithi194_103039_object94333:head# 2024-04-29T14:21:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46eb02b6:::benchmark_data_smithi194_103039_object155647:head# 2024-04-29T14:21:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ec114d:::benchmark_data_smithi194_103039_object121062:head# 2024-04-29T14:21:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ec707d:::benchmark_data_smithi194_103039_object71961:head# 2024-04-29T14:21:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ec8ff2:::benchmark_data_smithi194_103039_object41543:head# 2024-04-29T14:21:17.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ecfdae:::benchmark_data_smithi194_103039_object104415:head# 2024-04-29T14:21:17.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ed96c4:::benchmark_data_smithi194_103039_object105895:head# 2024-04-29T14:21:17.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46efe6c9:::benchmark_data_smithi194_103039_object107530:head# 2024-04-29T14:21:17.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f0d926:::benchmark_data_smithi194_103039_object142231:head# 2024-04-29T14:21:17.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f13c3c:::benchmark_data_smithi194_103039_object50749:head# 2024-04-29T14:21:17.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f2c35f:::benchmark_data_smithi194_103039_object144709:head# 2024-04-29T14:21:17.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f2d14b:::benchmark_data_smithi194_103039_object147284:head# 2024-04-29T14:21:17.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f30c53:::benchmark_data_smithi194_103039_object95043:head# 2024-04-29T14:21:17.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f37be4:::benchmark_data_smithi194_103039_object52809:head# 2024-04-29T14:21:17.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f3afbd:::benchmark_data_smithi194_103039_object119585:head# 2024-04-29T14:21:17.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f40c99:::benchmark_data_smithi194_103039_object87841:head# 2024-04-29T14:21:17.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f4314e:::benchmark_data_smithi194_103039_object100486:head# 2024-04-29T14:21:17.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f4a27b:::benchmark_data_smithi194_103039_object125412:head# 2024-04-29T14:21:17.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f4e0d2:::benchmark_data_smithi194_103039_object47757:head# 2024-04-29T14:21:17.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f5ab0a:::benchmark_data_smithi194_103039_object108495:head# 2024-04-29T14:21:17.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f63784:::benchmark_data_smithi194_103039_object69823:head# 2024-04-29T14:21:17.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f65b61:::benchmark_data_smithi194_103039_object95385:head# 2024-04-29T14:21:17.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f73fdb:::benchmark_data_smithi194_103039_object123289:head# 2024-04-29T14:21:17.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f74b28:::benchmark_data_smithi194_103039_object146250:head# 2024-04-29T14:21:17.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f87225:::benchmark_data_smithi194_103039_object122315:head# 2024-04-29T14:21:17.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f88613:::benchmark_data_smithi194_103039_object99014:head# 2024-04-29T14:21:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f8e3c8:::benchmark_data_smithi194_103039_object128779:head# 2024-04-29T14:21:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f9980a:::benchmark_data_smithi194_103039_object66531:head# 2024-04-29T14:21:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46f9dc33:::benchmark_data_smithi194_103039_object111915:head# 2024-04-29T14:21:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46fa46d2:::benchmark_data_smithi194_103039_object71032:head# 2024-04-29T14:21:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46fade80:::benchmark_data_smithi194_103039_object47163:head# 2024-04-29T14:21:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46fc3b2a:::benchmark_data_smithi194_103039_object142474:head# 2024-04-29T14:21:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46fc7dfd:::benchmark_data_smithi194_103039_object109712:head# 2024-04-29T14:21:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46fd4a0e:::benchmark_data_smithi194_103039_object154338:head# 2024-04-29T14:21:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46fe057f:::benchmark_data_smithi194_103039_object83967:head# 2024-04-29T14:21:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46feaed2:::benchmark_data_smithi194_103039_object91205:head# 2024-04-29T14:21:17.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46feccb5:::benchmark_data_smithi194_103039_object42920:head# 2024-04-29T14:21:17.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:46ff8671:::benchmark_data_smithi194_103039_object143020:head# 2024-04-29T14:21:17.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47006bc9:::benchmark_data_smithi194_103039_object101857:head# 2024-04-29T14:21:17.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4700932f:::benchmark_data_smithi194_103039_object111036:head# 2024-04-29T14:21:17.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4700afb4:::benchmark_data_smithi194_103039_object156068:head# 2024-04-29T14:21:17.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4701056f:::benchmark_data_smithi194_103039_object129517:head# 2024-04-29T14:21:17.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4701bfc3:::benchmark_data_smithi194_103039_object138081:head# 2024-04-29T14:21:17.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4701c8dc:::benchmark_data_smithi194_103039_object90157:head# 2024-04-29T14:21:17.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4702e8fe:::benchmark_data_smithi194_103039_object44363:head# 2024-04-29T14:21:17.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47038b5b:::benchmark_data_smithi194_103039_object66765:head# 2024-04-29T14:21:17.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4703968b:::benchmark_data_smithi194_103039_object92044:head# 2024-04-29T14:21:17.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4703f41b:::benchmark_data_smithi194_103039_object90480:head# 2024-04-29T14:21:17.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47047eb5:::benchmark_data_smithi194_103039_object108484:head# 2024-04-29T14:21:17.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47052971:::benchmark_data_smithi194_103039_object128415:head# 2024-04-29T14:21:17.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47062a19:::benchmark_data_smithi194_103039_object59829:head# 2024-04-29T14:21:17.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4706321b:::benchmark_data_smithi194_103039_object84371:head# 2024-04-29T14:21:17.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470642ee:::benchmark_data_smithi194_103039_object64812:head# 2024-04-29T14:21:17.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4707436d:::benchmark_data_smithi194_103039_object71337:head# 2024-04-29T14:21:17.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47085d2b:::benchmark_data_smithi194_103039_object146073:head# 2024-04-29T14:21:17.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470934f0:::benchmark_data_smithi194_103039_object97597:head# 2024-04-29T14:21:17.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47099d00:::benchmark_data_smithi194_103039_object97271:head# 2024-04-29T14:21:17.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470b7b5a:::benchmark_data_smithi194_103039_object116597:head# 2024-04-29T14:21:17.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470b8c92:::benchmark_data_smithi194_103039_object112781:head# 2024-04-29T14:21:17.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470c7c82:::benchmark_data_smithi194_103039_object91691:head# 2024-04-29T14:21:17.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470d0be0:::benchmark_data_smithi194_103039_object51030:head# 2024-04-29T14:21:17.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470ea0f2:::benchmark_data_smithi194_103039_object113007:head# 2024-04-29T14:21:17.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:470f060b:::benchmark_data_smithi194_103039_object90638:head# 2024-04-29T14:21:17.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47101444:::benchmark_data_smithi194_103039_object123945:head# 2024-04-29T14:21:17.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471126a3:::benchmark_data_smithi194_103039_object49888:head# 2024-04-29T14:21:17.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4713bb6b:::benchmark_data_smithi194_103039_object136170:head# 2024-04-29T14:21:17.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4713cc3f:::benchmark_data_smithi194_103039_object114375:head# 2024-04-29T14:21:17.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4713e7fb:::benchmark_data_smithi194_103039_object127137:head# 2024-04-29T14:21:17.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471495a1:::benchmark_data_smithi194_103039_object109906:head# 2024-04-29T14:21:17.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47155565:::benchmark_data_smithi194_103039_object116605:head# 2024-04-29T14:21:17.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4715603d:::benchmark_data_smithi194_103039_object93610:head# 2024-04-29T14:21:17.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4715d032:::benchmark_data_smithi194_103039_object110578:head# 2024-04-29T14:21:17.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4717d008:::benchmark_data_smithi194_103039_object94206:head# 2024-04-29T14:21:17.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47181328:::benchmark_data_smithi194_103039_object109059:head# 2024-04-29T14:21:17.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47188cad:::benchmark_data_smithi194_103039_object135714:head# 2024-04-29T14:21:17.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4718aa03:::benchmark_data_smithi194_103039_object57385:head# 2024-04-29T14:21:17.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4719234b:::benchmark_data_smithi194_103039_object122192:head# 2024-04-29T14:21:17.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471964e5:::benchmark_data_smithi194_103039_object99531:head# 2024-04-29T14:21:17.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471977ca:::benchmark_data_smithi194_103039_object115721:head# 2024-04-29T14:21:17.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4719b50e:::benchmark_data_smithi194_103039_object45148:head# 2024-04-29T14:21:17.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471a322f:::benchmark_data_smithi194_103039_object128136:head# 2024-04-29T14:21:17.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471a93e5:::benchmark_data_smithi194_103039_object107365:head# 2024-04-29T14:21:17.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471b77b5:::benchmark_data_smithi194_103039_object98044:head# 2024-04-29T14:21:17.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471bc18b:::benchmark_data_smithi194_103039_object64332:head# 2024-04-29T14:21:17.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471bf85f:::benchmark_data_smithi194_103039_object81496:head# 2024-04-29T14:21:17.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471cfa05:::benchmark_data_smithi194_103039_object57253:head# 2024-04-29T14:21:17.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471e4cac:::benchmark_data_smithi194_103039_object92861:head# 2024-04-29T14:21:17.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471e86bc:::benchmark_data_smithi194_103039_object96621:head# 2024-04-29T14:21:17.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471e955c:::benchmark_data_smithi194_103039_object109157:head# 2024-04-29T14:21:17.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471f742c:::benchmark_data_smithi194_103039_object135012:head# 2024-04-29T14:21:17.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:471ff668:::benchmark_data_smithi194_103039_object86951:head# 2024-04-29T14:21:17.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47201a95:::benchmark_data_smithi194_103039_object125328:head# 2024-04-29T14:21:17.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4720f574:::benchmark_data_smithi194_103039_object96488:head# 2024-04-29T14:21:17.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47215cfc:::benchmark_data_smithi194_103039_object102809:head# 2024-04-29T14:21:17.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472163fb:::benchmark_data_smithi194_103039_object112763:head# 2024-04-29T14:21:17.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4721a83d:::benchmark_data_smithi194_103039_object149788:head# 2024-04-29T14:21:17.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472320d7:::benchmark_data_smithi194_103039_object60038:head# 2024-04-29T14:21:17.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47237dc8:::benchmark_data_smithi194_103039_object139536:head# 2024-04-29T14:21:17.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472403c4:::benchmark_data_smithi194_103039_object42193:head# 2024-04-29T14:21:17.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4725157a:::benchmark_data_smithi194_103039_object42994:head# 2024-04-29T14:21:17.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47255a39:::benchmark_data_smithi194_103039_object70223:head# 2024-04-29T14:21:17.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4726a7c7:::benchmark_data_smithi194_103039_object65391:head# 2024-04-29T14:21:17.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47272314:::benchmark_data_smithi194_103039_object93138:head# 2024-04-29T14:21:17.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47280a1d:::benchmark_data_smithi194_103039_object86305:head# 2024-04-29T14:21:17.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4728cfda:::benchmark_data_smithi194_103039_object136178:head# 2024-04-29T14:21:17.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4729bf28:::benchmark_data_smithi194_103039_object71222:head# 2024-04-29T14:21:17.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472a7a5b:::benchmark_data_smithi194_103039_object89108:head# 2024-04-29T14:21:17.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472ab8b9:::benchmark_data_smithi194_103039_object99068:head# 2024-04-29T14:21:17.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472acd34:::benchmark_data_smithi194_103039_object86339:head# 2024-04-29T14:21:17.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472b90c4:::benchmark_data_smithi194_103039_object125591:head# 2024-04-29T14:21:17.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472bda95:::benchmark_data_smithi194_103039_object72888:head# 2024-04-29T14:21:17.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472c27f7:::benchmark_data_smithi194_103039_object105612:head# 2024-04-29T14:21:17.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472c3568:::benchmark_data_smithi194_103039_object133470:head# 2024-04-29T14:21:17.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472c7bef:::benchmark_data_smithi194_103039_object145512:head# 2024-04-29T14:21:17.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472e30d6:::benchmark_data_smithi194_103039_object130050:head# 2024-04-29T14:21:17.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472e47ca:::benchmark_data_smithi194_103039_object55274:head# 2024-04-29T14:21:17.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472edb24:::benchmark_data_smithi194_103039_object63926:head# 2024-04-29T14:21:17.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472f090b:::benchmark_data_smithi194_103039_object133122:head# 2024-04-29T14:21:17.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:472f53d2:::benchmark_data_smithi194_103039_object134564:head# 2024-04-29T14:21:17.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47310a39:::benchmark_data_smithi194_103039_object85222:head# 2024-04-29T14:21:17.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473114f9:::benchmark_data_smithi194_103039_object58929:head# 2024-04-29T14:21:17.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473164eb:::benchmark_data_smithi194_103039_object88465:head# 2024-04-29T14:21:17.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4731965e:::benchmark_data_smithi194_103039_object152734:head# 2024-04-29T14:21:17.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4734c393:::benchmark_data_smithi194_103039_object93937:head# 2024-04-29T14:21:17.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4735983c:::benchmark_data_smithi194_103039_object96110:head# 2024-04-29T14:21:17.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4736472e:::benchmark_data_smithi194_103039_object101678:head# 2024-04-29T14:21:17.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4736b1b6:::benchmark_data_smithi194_103039_object144356:head# 2024-04-29T14:21:17.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4736d38e:::benchmark_data_smithi194_103039_object56076:head# 2024-04-29T14:21:17.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4736f6cc:::benchmark_data_smithi194_103039_object87071:head# 2024-04-29T14:21:17.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4737b23d:::benchmark_data_smithi194_103039_object79040:head# 2024-04-29T14:21:17.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473819a2:::benchmark_data_smithi194_103039_object128397:head# 2024-04-29T14:21:17.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4738ec51:::benchmark_data_smithi194_103039_object85651:head# 2024-04-29T14:21:17.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4739d362:::benchmark_data_smithi194_103039_object113613:head# 2024-04-29T14:21:17.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473a2167:::benchmark_data_smithi194_103039_object115975:head# 2024-04-29T14:21:17.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473b17f1:::benchmark_data_smithi194_103039_object151495:head# 2024-04-29T14:21:17.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473b78eb:::benchmark_data_smithi194_103039_object131786:head# 2024-04-29T14:21:17.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473bbc16:::benchmark_data_smithi194_103039_object98414:head# 2024-04-29T14:21:17.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473cbfe8:::benchmark_data_smithi194_103039_object55857:head# 2024-04-29T14:21:17.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473ccdde:::benchmark_data_smithi194_103039_object104452:head# 2024-04-29T14:21:17.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473e193c:::benchmark_data_smithi194_103039_object69983:head# 2024-04-29T14:21:17.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473e486b:::benchmark_data_smithi194_103039_object130799:head# 2024-04-29T14:21:17.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473eaeb5:::benchmark_data_smithi194_103039_object61516:head# 2024-04-29T14:21:17.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473ec9f0:::benchmark_data_smithi194_103039_object46642:head# 2024-04-29T14:21:17.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:473fb5dd:::benchmark_data_smithi194_103039_object74040:head# 2024-04-29T14:21:17.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4740a11e:::benchmark_data_smithi194_103039_object144922:head# 2024-04-29T14:21:17.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47412924:::benchmark_data_smithi194_103039_object133482:head# 2024-04-29T14:21:17.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47419600:::benchmark_data_smithi194_103039_object144315:head# 2024-04-29T14:21:17.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474211a0:::benchmark_data_smithi194_103039_object100308:head# 2024-04-29T14:21:17.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474280f0:::benchmark_data_smithi194_103039_object112350:head# 2024-04-29T14:21:17.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4742cf1a:::benchmark_data_smithi194_103039_object49799:head# 2024-04-29T14:21:17.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474322a1:::benchmark_data_smithi194_103039_object76798:head# 2024-04-29T14:21:17.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47433c41:::benchmark_data_smithi194_103039_object131120:head# 2024-04-29T14:21:17.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4743a37b:::benchmark_data_smithi194_103039_object151973:head# 2024-04-29T14:21:17.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4743bb8f:::benchmark_data_smithi194_103039_object112328:head# 2024-04-29T14:21:17.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474407fd:::benchmark_data_smithi194_103039_object44078:head# 2024-04-29T14:21:17.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47442846:::benchmark_data_smithi194_103039_object105827:head# 2024-04-29T14:21:17.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4744ac21:::benchmark_data_smithi194_103039_object48961:head# 2024-04-29T14:21:17.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4745310b:::benchmark_data_smithi194_103039_object108893:head# 2024-04-29T14:21:17.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4745aaec:::benchmark_data_smithi194_103039_object156151:head# 2024-04-29T14:21:17.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4745b160:::benchmark_data_smithi194_103039_object52720:head# 2024-04-29T14:21:17.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47460aab:::benchmark_data_smithi194_103039_object102415:head# 2024-04-29T14:21:17.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47478d5f:::benchmark_data_smithi194_103039_object63345:head# 2024-04-29T14:21:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47479a42:::benchmark_data_smithi194_103039_object101922:head# 2024-04-29T14:21:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47493e7d:::benchmark_data_smithi194_103039_object100400:head# 2024-04-29T14:21:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47497824:::benchmark_data_smithi194_103039_object75106:head# 2024-04-29T14:21:17.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474ae770:::benchmark_data_smithi194_103039_object91725:head# 2024-04-29T14:21:17.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474b64a2:::benchmark_data_smithi194_103039_object49038:head# 2024-04-29T14:21:17.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474bcc4d:::benchmark_data_smithi194_103039_object92036:head# 2024-04-29T14:21:17.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474c9b8a:::benchmark_data_smithi194_103039_object117793:head# 2024-04-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474d025b:::benchmark_data_smithi194_103039_object78525:head# 2024-04-29T14:21:17.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474d79a8:::benchmark_data_smithi194_103039_object122831:head# 2024-04-29T14:21:17.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:474faf8d:::benchmark_data_smithi194_103039_object76761:head# 2024-04-29T14:21:17.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4750c235:::benchmark_data_smithi194_103039_object59158:head# 2024-04-29T14:21:17.996 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:17.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47529e47:::benchmark_data_smithi194_103039_object155631:head# 2024-04-29T14:21:17.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4752f083:::benchmark_data_smithi194_103039_object91852:head# 2024-04-29T14:21:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4753e733:::benchmark_data_smithi194_103039_object148703:head# 2024-04-29T14:21:17.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4753f5ff:::benchmark_data_smithi194_103039_object137697:head# 2024-04-29T14:21:17.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475411e7:::benchmark_data_smithi194_103039_object97167:head# 2024-04-29T14:21:17.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4755262f:::benchmark_data_smithi194_103039_object129667:head# 2024-04-29T14:21:17.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47556989:::benchmark_data_smithi194_103039_object144532:head# 2024-04-29T14:21:17.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47566627:::benchmark_data_smithi194_103039_object79943:head# 2024-04-29T14:21:18.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4757609f:::benchmark_data_smithi194_103039_object63132:head# 2024-04-29T14:21:18.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4757bbdf:::benchmark_data_smithi194_103039_object128889:head# 2024-04-29T14:21:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47586947:::benchmark_data_smithi194_103039_object100793:head# 2024-04-29T14:21:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47588846:::benchmark_data_smithi194_103039_object138137:head# 2024-04-29T14:21:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4758ef67:::benchmark_data_smithi194_103039_object45223:head# 2024-04-29T14:21:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4758f019:::benchmark_data_smithi194_103039_object100528:head# 2024-04-29T14:21:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47592427:::benchmark_data_smithi194_103039_object49363:head# 2024-04-29T14:21:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475a2abd:::benchmark_data_smithi194_103039_object78338:head# 2024-04-29T14:21:18.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475b0229:::benchmark_data_smithi194_103039_object137864:head# 2024-04-29T14:21:18.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475b2883:::benchmark_data_smithi194_103039_object68866:head# 2024-04-29T14:21:18.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475bfc85:::benchmark_data_smithi194_103039_object147376:head# 2024-04-29T14:21:18.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475c5320:::benchmark_data_smithi194_103039_object51077:head# 2024-04-29T14:21:18.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475c630f:::benchmark_data_smithi194_103039_object132042:head# 2024-04-29T14:21:18.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475d21bc:::benchmark_data_smithi194_103039_object120649:head# 2024-04-29T14:21:18.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475d303f:::benchmark_data_smithi194_103039_object152379:head# 2024-04-29T14:21:18.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:475e4dd8:::benchmark_data_smithi194_103039_object83108:head# 2024-04-29T14:21:18.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4760231f:::benchmark_data_smithi194_103039_object75253:head# 2024-04-29T14:21:18.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4761c813:::benchmark_data_smithi194_103039_object134919:head# 2024-04-29T14:21:18.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4762014a:::benchmark_data_smithi194_103039_object57488:head# 2024-04-29T14:21:18.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47622414:::benchmark_data_smithi194_103039_object63036:head# 2024-04-29T14:21:18.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47646fef:::benchmark_data_smithi194_103039_object128833:head# 2024-04-29T14:21:18.009 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47659c8e:::benchmark_data_smithi194_103039_object92706:head# 2024-04-29T14:21:18.009 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4765ad9c:::benchmark_data_smithi194_103039_object148594:head# 2024-04-29T14:21:18.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4765e5ef:::benchmark_data_smithi194_103039_object138948:head# 2024-04-29T14:21:18.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47675299:::benchmark_data_smithi194_103039_object143558:head# 2024-04-29T14:21:18.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4767530a:::benchmark_data_smithi194_103039_object148321:head# 2024-04-29T14:21:18.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4767b3b5:::benchmark_data_smithi194_103039_object56201:head# 2024-04-29T14:21:18.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4767e8bd:::benchmark_data_smithi194_103039_object92670:head# 2024-04-29T14:21:18.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4768f994:::benchmark_data_smithi194_103039_object91704:head# 2024-04-29T14:21:18.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47698631:::benchmark_data_smithi194_103039_object109199:head# 2024-04-29T14:21:18.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4769c87b:::benchmark_data_smithi194_103039_object91607:head# 2024-04-29T14:21:18.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476a2887:::benchmark_data_smithi194_103039_object98725:head# 2024-04-29T14:21:18.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476a8bee:::benchmark_data_smithi194_103039_object45580:head# 2024-04-29T14:21:18.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476b4a0b:::benchmark_data_smithi194_103039_object45409:head# 2024-04-29T14:21:18.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476b5e05:::benchmark_data_smithi194_103039_object53558:head# 2024-04-29T14:21:18.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476b9013:::benchmark_data_smithi194_103039_object88843:head# 2024-04-29T14:21:18.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476bbaf8:::benchmark_data_smithi194_103039_object117856:head# 2024-04-29T14:21:18.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476e6e9c:::benchmark_data_smithi194_103039_object46405:head# 2024-04-29T14:21:18.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476e777e:::benchmark_data_smithi194_103039_object96798:head# 2024-04-29T14:21:18.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:476fcfc5:::benchmark_data_smithi194_103039_object89565:head# 2024-04-29T14:21:18.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47701986:::benchmark_data_smithi194_103039_object145211:head# 2024-04-29T14:21:18.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47711354:::benchmark_data_smithi194_103039_object138345:head# 2024-04-29T14:21:18.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477152f4:::benchmark_data_smithi194_103039_object99087:head# 2024-04-29T14:21:18.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4771a75f:::benchmark_data_smithi194_103039_object55001:head# 2024-04-29T14:21:18.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47728f46:::benchmark_data_smithi194_103039_object129200:head# 2024-04-29T14:21:18.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4772c053:::benchmark_data_smithi194_103039_object76366:head# 2024-04-29T14:21:18.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47730614:::benchmark_data_smithi194_103039_object134401:head# 2024-04-29T14:21:18.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47745387:::benchmark_data_smithi194_103039_object147340:head# 2024-04-29T14:21:18.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4775988f:::benchmark_data_smithi194_103039_object115213:head# 2024-04-29T14:21:18.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47764d74:::benchmark_data_smithi194_103039_object119227:head# 2024-04-29T14:21:18.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47765cc9:::benchmark_data_smithi194_103039_object90419:head# 2024-04-29T14:21:18.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4776b0b5:::benchmark_data_smithi194_103039_object148614:head# 2024-04-29T14:21:18.024 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:18.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47773bae:::benchmark_data_smithi194_103039_object130596:head# 2024-04-29T14:21:18.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47777906:::benchmark_data_smithi194_103039_object69110:head# 2024-04-29T14:21:18.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47778ac6:::benchmark_data_smithi194_103039_object100907:head# 2024-04-29T14:21:18.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4778ff8c:::benchmark_data_smithi194_103039_object84544:head# 2024-04-29T14:21:18.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47797b85:::benchmark_data_smithi194_103039_object144944:head# 2024-04-29T14:21:18.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47797be7:::benchmark_data_smithi194_103039_object94705:head# 2024-04-29T14:21:18.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477a5a69:::benchmark_data_smithi194_103039_object73029:head# 2024-04-29T14:21:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477a73b5:::benchmark_data_smithi194_103039_object129880:head# 2024-04-29T14:21:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477a91bc:::benchmark_data_smithi194_103039_object97961:head# 2024-04-29T14:21:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477aab47:::benchmark_data_smithi194_103039_object156080:head# 2024-04-29T14:21:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477ac9f3:::benchmark_data_smithi194_103039_object96531:head# 2024-04-29T14:21:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477aecf4:::benchmark_data_smithi194_103039_object46758:head# 2024-04-29T14:21:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477b533e:::benchmark_data_smithi194_103039_object123973:head# 2024-04-29T14:21:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477bcf1b:::benchmark_data_smithi194_103039_object130700:head# 2024-04-29T14:21:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477ce7c4:::benchmark_data_smithi194_103039_object103795:head# 2024-04-29T14:21:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477e009e:::benchmark_data_smithi194_103039_object139815:head# 2024-04-29T14:21:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477e59f3:::benchmark_data_smithi194_103039_object79865:head# 2024-04-29T14:21:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477eb68f:::benchmark_data_smithi194_103039_object138907:head# 2024-04-29T14:21:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477f5a99:::benchmark_data_smithi194_103039_object131733:head# 2024-04-29T14:21:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477fa4d2:::benchmark_data_smithi194_103039_object56700:head# 2024-04-29T14:21:18.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:477fac16:::benchmark_data_smithi194_103039_object67262:head# 2024-04-29T14:21:18.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47803bef:::benchmark_data_smithi194_103039_object83998:head# 2024-04-29T14:21:18.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4780caed:::benchmark_data_smithi194_103039_object55614:head# 2024-04-29T14:21:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47810f3c:::benchmark_data_smithi194_103039_object136856:head# 2024-04-29T14:21:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4781817b:::benchmark_data_smithi194_103039_object105412:head# 2024-04-29T14:21:18.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47819cbc:::benchmark_data_smithi194_103039_object145934:head# 2024-04-29T14:21:18.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4781adf3:::benchmark_data_smithi194_103039_object133722:head# 2024-04-29T14:21:18.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47822f1a:::benchmark_data_smithi194_103039_object125817:head# 2024-04-29T14:21:18.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478286a0:::benchmark_data_smithi194_103039_object68871:head# 2024-04-29T14:21:18.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4782c392:::benchmark_data_smithi194_103039_object65191:head# 2024-04-29T14:21:18.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4785703b:::benchmark_data_smithi194_103039_object112801:head# 2024-04-29T14:21:18.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4785cd54:::benchmark_data_smithi194_103039_object104697:head# 2024-04-29T14:21:18.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4786e5f4:::benchmark_data_smithi194_103039_object134377:head# 2024-04-29T14:21:18.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4786ed8a:::benchmark_data_smithi194_103039_object62816:head# 2024-04-29T14:21:18.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47871a03:::benchmark_data_smithi194_103039_object139657:head# 2024-04-29T14:21:18.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478746ab:::benchmark_data_smithi194_103039_object54152:head# 2024-04-29T14:21:18.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4787a9a1:::benchmark_data_smithi194_103039_object155150:head# 2024-04-29T14:21:18.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4788b09f:::benchmark_data_smithi194_103039_object115136:head# 2024-04-29T14:21:18.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47891ff5:::benchmark_data_smithi194_103039_object96005:head# 2024-04-29T14:21:18.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47896416:::benchmark_data_smithi194_103039_object101784:head# 2024-04-29T14:21:18.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4789c935:::benchmark_data_smithi194_103039_object132888:head# 2024-04-29T14:21:18.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4789f22e:::benchmark_data_smithi194_103039_object105627:head# 2024-04-29T14:21:18.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478a1020:::benchmark_data_smithi194_103039_object75763:head# 2024-04-29T14:21:18.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478a1376:::benchmark_data_smithi194_103039_object145254:head# 2024-04-29T14:21:18.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478a9554:::benchmark_data_smithi194_103039_object134952:head# 2024-04-29T14:21:18.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478bf862:::benchmark_data_smithi194_103039_object65168:head# 2024-04-29T14:21:18.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478c95a8:::benchmark_data_smithi194_103039_object126405:head# 2024-04-29T14:21:18.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478c9db4:::benchmark_data_smithi194_103039_object144568:head# 2024-04-29T14:21:18.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478cc953:::benchmark_data_smithi194_103039_object88387:head# 2024-04-29T14:21:18.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478cf1fa:::benchmark_data_smithi194_103039_object156145:head# 2024-04-29T14:21:18.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478d0c0c:::benchmark_data_smithi194_103039_object134647:head# 2024-04-29T14:21:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478d2cf0:::benchmark_data_smithi194_103039_object137922:head# 2024-04-29T14:21:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478e5ea7:::benchmark_data_smithi194_103039_object51623:head# 2024-04-29T14:21:18.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478e62eb:::benchmark_data_smithi194_103039_object47037:head# 2024-04-29T14:21:18.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478f2811:::benchmark_data_smithi194_103039_object73573:head# 2024-04-29T14:21:18.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:478ffa8e:::benchmark_data_smithi194_103039_object51136:head# 2024-04-29T14:21:18.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47900524:::benchmark_data_smithi194_103039_object98985:head# 2024-04-29T14:21:18.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47906202:::benchmark_data_smithi194_103039_object55663:head# 2024-04-29T14:21:18.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4791026e:::benchmark_data_smithi194_103039_object46162:head# 2024-04-29T14:21:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479129d2:::benchmark_data_smithi194_103039_object61414:head# 2024-04-29T14:21:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479136b0:::benchmark_data_smithi194_103039_object80107:head# 2024-04-29T14:21:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4791cb2e:::benchmark_data_smithi194_103039_object148283:head# 2024-04-29T14:21:18.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47921a38:::benchmark_data_smithi194_103039_object95297:head# 2024-04-29T14:21:18.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4792c40c:::benchmark_data_smithi194_103039_object88096:head# 2024-04-29T14:21:18.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4792d698:::benchmark_data_smithi194_103039_object138470:head# 2024-04-29T14:21:18.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4793b9b7:::benchmark_data_smithi194_103039_object138918:head# 2024-04-29T14:21:18.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47957d6a:::benchmark_data_smithi194_103039_object47053:head# 2024-04-29T14:21:18.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47958058:::benchmark_data_smithi194_103039_object151408:head# 2024-04-29T14:21:18.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4795b95f:::benchmark_data_smithi194_103039_object45840:head# 2024-04-29T14:21:18.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47960a63:::benchmark_data_smithi194_103039_object76852:head# 2024-04-29T14:21:18.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47975dcf:::benchmark_data_smithi194_103039_object127725:head# 2024-04-29T14:21:18.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479781ae:::benchmark_data_smithi194_103039_object42790:head# 2024-04-29T14:21:18.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47988f57:::benchmark_data_smithi194_103039_object64040:head# 2024-04-29T14:21:18.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4798f43e:::benchmark_data_smithi194_103039_object67633:head# 2024-04-29T14:21:18.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4799b12b:::benchmark_data_smithi194_103039_object99491:head# 2024-04-29T14:21:18.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479abe89:::benchmark_data_smithi194_103039_object88175:head# 2024-04-29T14:21:18.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479bc6d1:::benchmark_data_smithi194_103039_object89376:head# 2024-04-29T14:21:18.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479ccf14:::benchmark_data_smithi194_103039_object43138:head# 2024-04-29T14:21:18.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479ce679:::benchmark_data_smithi194_103039_object140016:head# 2024-04-29T14:21:18.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479e687d:::benchmark_data_smithi194_103039_object74348:head# 2024-04-29T14:21:18.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:479e8a61:::benchmark_data_smithi194_103039_object139929:head# 2024-04-29T14:21:18.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a05ba7:::benchmark_data_smithi194_103039_object102476:head# 2024-04-29T14:21:18.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a1d603:::benchmark_data_smithi194_103039_object90334:head# 2024-04-29T14:21:18.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a28d48:::benchmark_data_smithi194_103039_object122944:head# 2024-04-29T14:21:18.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a2fa94:::benchmark_data_smithi194_103039_object88257:head# 2024-04-29T14:21:18.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a331c3:::benchmark_data_smithi194_103039_object81648:head# 2024-04-29T14:21:18.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a386e8:::benchmark_data_smithi194_103039_object125099:head# 2024-04-29T14:21:18.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a51d33:::benchmark_data_smithi194_103039_object144884:head# 2024-04-29T14:21:18.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a59f3c:::benchmark_data_smithi194_103039_object90243:head# 2024-04-29T14:21:18.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a684de:::benchmark_data_smithi194_103039_object45937:head# 2024-04-29T14:21:18.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a77698:::benchmark_data_smithi194_103039_object129710:head# 2024-04-29T14:21:18.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a8079e:::benchmark_data_smithi194_103039_object136896:head# 2024-04-29T14:21:18.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a854e4:::benchmark_data_smithi194_103039_object118833:head# 2024-04-29T14:21:18.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a8688f:::benchmark_data_smithi194_103039_object77648:head# 2024-04-29T14:21:18.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a8d284:::benchmark_data_smithi194_103039_object95841:head# 2024-04-29T14:21:18.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a90c03:::benchmark_data_smithi194_103039_object54118:head# 2024-04-29T14:21:18.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47a97e7d:::benchmark_data_smithi194_103039_object86148:head# 2024-04-29T14:21:18.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47aa7af7:::benchmark_data_smithi194_103039_object55103:head# 2024-04-29T14:21:18.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ab3746:::benchmark_data_smithi194_103039_object121667:head# 2024-04-29T14:21:18.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ab3ac9:::benchmark_data_smithi194_103039_object60158:head# 2024-04-29T14:21:18.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ab64c1:::benchmark_data_smithi194_103039_object64267:head# 2024-04-29T14:21:18.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47aba5a6:::benchmark_data_smithi194_103039_object136705:head# 2024-04-29T14:21:18.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ac0968:::benchmark_data_smithi194_103039_object90840:head# 2024-04-29T14:21:18.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ac1644:::benchmark_data_smithi194_103039_object81012:head# 2024-04-29T14:21:18.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ac165d:::benchmark_data_smithi194_103039_object128201:head# 2024-04-29T14:21:18.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ac604e:::benchmark_data_smithi194_103039_object111735:head# 2024-04-29T14:21:18.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ad7e83:::benchmark_data_smithi194_103039_object51079:head# 2024-04-29T14:21:18.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ae70a9:::benchmark_data_smithi194_103039_object136591:head# 2024-04-29T14:21:18.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47aed97e:::benchmark_data_smithi194_103039_object107148:head# 2024-04-29T14:21:18.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47af2d21:::benchmark_data_smithi194_103039_object145260:head# 2024-04-29T14:21:18.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b09db3:::benchmark_data_smithi194_103039_object107912:head# 2024-04-29T14:21:18.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b0e383:::benchmark_data_smithi194_103039_object153957:head# 2024-04-29T14:21:18.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b0e602:::benchmark_data_smithi194_103039_object48425:head# 2024-04-29T14:21:18.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b181dc:::benchmark_data_smithi194_103039_object113129:head# 2024-04-29T14:21:18.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b213e6:::benchmark_data_smithi194_103039_object89021:head# 2024-04-29T14:21:18.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b221c5:::benchmark_data_smithi194_103039_object56506:head# 2024-04-29T14:21:18.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b24eac:::benchmark_data_smithi194_103039_object89838:head# 2024-04-29T14:21:18.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b25d52:::benchmark_data_smithi194_103039_object47141:head# 2024-04-29T14:21:18.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b31485:::benchmark_data_smithi194_103039_object73212:head# 2024-04-29T14:21:18.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b3171a:::benchmark_data_smithi194_103039_object151933:head# 2024-04-29T14:21:18.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b37232:::benchmark_data_smithi194_103039_object140680:head# 2024-04-29T14:21:18.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b3ac3b:::benchmark_data_smithi194_103039_object123297:head# 2024-04-29T14:21:18.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b571eb:::benchmark_data_smithi194_103039_object130532:head# 2024-04-29T14:21:18.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b71ad8:::benchmark_data_smithi194_103039_object112945:head# 2024-04-29T14:21:18.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b721a4:::benchmark_data_smithi194_103039_object58408:head# 2024-04-29T14:21:18.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b7be7f:::benchmark_data_smithi194_103039_object80837:head# 2024-04-29T14:21:18.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47b8582b:::benchmark_data_smithi194_103039_object104978:head# 2024-04-29T14:21:18.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ba3fb2:::benchmark_data_smithi194_103039_object119411:head# 2024-04-29T14:21:18.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47baaf00:::benchmark_data_smithi194_103039_object122166:head# 2024-04-29T14:21:18.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bafb6c:::benchmark_data_smithi194_103039_object75551:head# 2024-04-29T14:21:18.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bb1c03:::benchmark_data_smithi194_103039_object55800:head# 2024-04-29T14:21:18.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bb5993:::benchmark_data_smithi194_103039_object51845:head# 2024-04-29T14:21:18.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bba3c1:::benchmark_data_smithi194_103039_object48196:head# 2024-04-29T14:21:18.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bc95fc:::benchmark_data_smithi194_103039_object152390:head# 2024-04-29T14:21:18.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bca265:::benchmark_data_smithi194_103039_object82491:head# 2024-04-29T14:21:18.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bd1776:::benchmark_data_smithi194_103039_object144524:head# 2024-04-29T14:21:18.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bd958d:::benchmark_data_smithi194_103039_object41506:head# 2024-04-29T14:21:18.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bdfeb6:::benchmark_data_smithi194_103039_object152371:head# 2024-04-29T14:21:18.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47be02f8:::benchmark_data_smithi194_103039_object142303:head# 2024-04-29T14:21:18.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47be4b70:::benchmark_data_smithi194_103039_object103017:head# 2024-04-29T14:21:18.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47be991c:::benchmark_data_smithi194_103039_object77754:head# 2024-04-29T14:21:18.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bf2416:::benchmark_data_smithi194_103039_object60451:head# 2024-04-29T14:21:18.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47bf987f:::benchmark_data_smithi194_103039_object146263:head# 2024-04-29T14:21:18.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c02587:::benchmark_data_smithi194_103039_object76443:head# 2024-04-29T14:21:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c2e498:::benchmark_data_smithi194_103039_object44118:head# 2024-04-29T14:21:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c3b4e3:::benchmark_data_smithi194_103039_object49066:head# 2024-04-29T14:21:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c4c062:::benchmark_data_smithi194_103039_object102980:head# 2024-04-29T14:21:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c7da64:::benchmark_data_smithi194_103039_object106152:head# 2024-04-29T14:21:18.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c890f7:::benchmark_data_smithi194_103039_object99021:head# 2024-04-29T14:21:18.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c9078b:::benchmark_data_smithi194_103039_object61759:head# 2024-04-29T14:21:18.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c91b64:::benchmark_data_smithi194_103039_object94265:head# 2024-04-29T14:21:18.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c93623:::benchmark_data_smithi194_103039_object76976:head# 2024-04-29T14:21:18.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c997a2:::benchmark_data_smithi194_103039_object60748:head# 2024-04-29T14:21:18.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47c9e1fc:::benchmark_data_smithi194_103039_object125179:head# 2024-04-29T14:21:18.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ca16e5:::benchmark_data_smithi194_103039_object97488:head# 2024-04-29T14:21:18.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ca202c:::benchmark_data_smithi194_103039_object155595:head# 2024-04-29T14:21:18.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ca8b4f:::benchmark_data_smithi194_103039_object83513:head# 2024-04-29T14:21:18.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47cae646:::benchmark_data_smithi194_103039_object80651:head# 2024-04-29T14:21:18.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47cb4490:::benchmark_data_smithi194_103039_object80830:head# 2024-04-29T14:21:18.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47cba397:::benchmark_data_smithi194_103039_object83137:head# 2024-04-29T14:21:18.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ccfc03:::benchmark_data_smithi194_103039_object138790:head# 2024-04-29T14:21:18.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47cddab7:::benchmark_data_smithi194_103039_object114500:head# 2024-04-29T14:21:18.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ceeeb3:::benchmark_data_smithi194_103039_object101662:head# 2024-04-29T14:21:18.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47cf2850:::benchmark_data_smithi194_103039_object151879:head# 2024-04-29T14:21:18.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d11422:::benchmark_data_smithi194_103039_object92708:head# 2024-04-29T14:21:18.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d2629a:::benchmark_data_smithi194_103039_object144543:head# 2024-04-29T14:21:18.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d2b226:::benchmark_data_smithi194_103039_object117475:head# 2024-04-29T14:21:18.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d2c8da:::benchmark_data_smithi194_103039_object97161:head# 2024-04-29T14:21:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d3fd64:::benchmark_data_smithi194_103039_object134696:head# 2024-04-29T14:21:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d4ae6b:::benchmark_data_smithi194_103039_object68136:head# 2024-04-29T14:21:18.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d688f7:::benchmark_data_smithi194_103039_object72299:head# 2024-04-29T14:21:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d6a82d:::benchmark_data_smithi194_103039_object103825:head# 2024-04-29T14:21:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d6bac7:::benchmark_data_smithi194_103039_object143094:head# 2024-04-29T14:21:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d6e35a:::benchmark_data_smithi194_103039_object114429:head# 2024-04-29T14:21:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d6f7b7:::benchmark_data_smithi194_103039_object103086:head# 2024-04-29T14:21:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d723b5:::benchmark_data_smithi194_103039_object116947:head# 2024-04-29T14:21:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d7c706:::benchmark_data_smithi194_103039_object84032:head# 2024-04-29T14:21:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d7f62c:::benchmark_data_smithi194_103039_object144457:head# 2024-04-29T14:21:18.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47d9b849:::benchmark_data_smithi194_103039_object138053:head# 2024-04-29T14:21:18.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47da5442:::benchmark_data_smithi194_103039_object68936:head# 2024-04-29T14:21:18.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47db7fc1:::benchmark_data_smithi194_103039_object76399:head# 2024-04-29T14:21:18.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47dbe904:::benchmark_data_smithi194_103039_object42700:head# 2024-04-29T14:21:18.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47dc515e:::benchmark_data_smithi194_103039_object73283:head# 2024-04-29T14:21:18.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47dc54d1:::benchmark_data_smithi194_103039_object89139:head# 2024-04-29T14:21:18.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47dfa624:::benchmark_data_smithi194_103039_object98539:head# 2024-04-29T14:21:18.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e0a855:::benchmark_data_smithi194_103039_object61663:head# 2024-04-29T14:21:18.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e0c96f:::benchmark_data_smithi194_103039_object128070:head# 2024-04-29T14:21:18.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e16ac4:::benchmark_data_smithi194_103039_object130892:head# 2024-04-29T14:21:18.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e1902a:::benchmark_data_smithi194_103039_object82500:head# 2024-04-29T14:21:18.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e30995:::benchmark_data_smithi194_103039_object144702:head# 2024-04-29T14:21:18.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e34490:::benchmark_data_smithi194_103039_object90147:head# 2024-04-29T14:21:18.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e3839b:::benchmark_data_smithi194_103039_object68738:head# 2024-04-29T14:21:18.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e4091f:::benchmark_data_smithi194_103039_object106117:head# 2024-04-29T14:21:18.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e44054:::benchmark_data_smithi194_103039_object151240:head# 2024-04-29T14:21:18.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e524eb:::benchmark_data_smithi194_103039_object49947:head# 2024-04-29T14:21:18.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e5717d:::benchmark_data_smithi194_103039_object114966:head# 2024-04-29T14:21:18.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e5d6b3:::benchmark_data_smithi194_103039_object102510:head# 2024-04-29T14:21:18.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e7dbbe:::benchmark_data_smithi194_103039_object48726:head# 2024-04-29T14:21:18.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e7f522:::benchmark_data_smithi194_103039_object42139:head# 2024-04-29T14:21:18.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e85ce0:::benchmark_data_smithi194_103039_object89486:head# 2024-04-29T14:21:18.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e8abba:::benchmark_data_smithi194_103039_object142298:head# 2024-04-29T14:21:18.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e8acb5:::benchmark_data_smithi194_103039_object137788:head# 2024-04-29T14:21:18.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e8ce4a:::benchmark_data_smithi194_103039_object143787:head# 2024-04-29T14:21:18.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e8e849:::benchmark_data_smithi194_103039_object134683:head# 2024-04-29T14:21:18.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e94ed1:::benchmark_data_smithi194_103039_object43596:head# 2024-04-29T14:21:18.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47e95fbd:::benchmark_data_smithi194_103039_object143879:head# 2024-04-29T14:21:18.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ea1e59:::benchmark_data_smithi194_103039_object45627:head# 2024-04-29T14:21:18.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47eb3fff:::benchmark_data_smithi194_103039_object94013:head# 2024-04-29T14:21:18.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47eba8f4:::benchmark_data_smithi194_103039_object131255:head# 2024-04-29T14:21:18.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ec7535:::benchmark_data_smithi194_103039_object150887:head# 2024-04-29T14:21:18.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47eca605:::benchmark_data_smithi194_103039_object136087:head# 2024-04-29T14:21:18.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ecdb41:::benchmark_data_smithi194_103039_object104760:head# 2024-04-29T14:21:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47ef1ab9:::benchmark_data_smithi194_103039_object88619:head# 2024-04-29T14:21:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47efa4cc:::benchmark_data_smithi194_103039_object112849:head# 2024-04-29T14:21:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47efb021:::benchmark_data_smithi194_103039_object95285:head# 2024-04-29T14:21:18.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f0860c:::benchmark_data_smithi194_103039_object72365:head# 2024-04-29T14:21:18.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f09ae5:::benchmark_data_smithi194_103039_object70297:head# 2024-04-29T14:21:18.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f0a34c:::benchmark_data_smithi194_103039_object48839:head# 2024-04-29T14:21:18.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f1070d:::benchmark_data_smithi194_103039_object141256:head# 2024-04-29T14:21:18.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f1d7dc:::benchmark_data_smithi194_103039_object61222:head# 2024-04-29T14:21:18.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f218b6:::benchmark_data_smithi194_103039_object116477:head# 2024-04-29T14:21:18.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f2209a:::benchmark_data_smithi194_103039_object89830:head# 2024-04-29T14:21:18.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f25112:::benchmark_data_smithi194_103039_object71258:head# 2024-04-29T14:21:18.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f348e7:::benchmark_data_smithi194_103039_object87157:head# 2024-04-29T14:21:18.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f35787:::benchmark_data_smithi194_103039_object57712:head# 2024-04-29T14:21:18.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f48192:::benchmark_data_smithi194_103039_object54675:head# 2024-04-29T14:21:18.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f48d71:::benchmark_data_smithi194_103039_object109767:head# 2024-04-29T14:21:18.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f4eefa:::benchmark_data_smithi194_103039_object72794:head# 2024-04-29T14:21:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f5312b:::benchmark_data_smithi194_103039_object138158:head# 2024-04-29T14:21:18.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f57001:::benchmark_data_smithi194_103039_object154929:head# 2024-04-29T14:21:18.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f654d3:::benchmark_data_smithi194_103039_object125120:head# 2024-04-29T14:21:18.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f6c7ae:::benchmark_data_smithi194_103039_object111212:head# 2024-04-29T14:21:18.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f79b67:::benchmark_data_smithi194_103039_object130948:head# 2024-04-29T14:21:18.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47f7f8d1:::benchmark_data_smithi194_103039_object49536:head# 2024-04-29T14:21:18.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47fb4ad5:::benchmark_data_smithi194_103039_object72558:head# 2024-04-29T14:21:18.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47fb5e70:::benchmark_data_smithi194_103039_object82682:head# 2024-04-29T14:21:18.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47fe56af:::benchmark_data_smithi194_103039_object144794:head# 2024-04-29T14:21:18.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:47fec677:::benchmark_data_smithi194_103039_object75746:head# 2024-04-29T14:21:18.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480016fc:::benchmark_data_smithi194_103039_object55197:head# 2024-04-29T14:21:18.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4800afbe:::benchmark_data_smithi194_103039_object108203:head# 2024-04-29T14:21:18.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48010f77:::benchmark_data_smithi194_103039_object63902:head# 2024-04-29T14:21:18.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48022efd:::benchmark_data_smithi194_103039_object46200:head# 2024-04-29T14:21:18.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48028896:::benchmark_data_smithi194_103039_object96784:head# 2024-04-29T14:21:18.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48029360:::benchmark_data_smithi194_103039_object67266:head# 2024-04-29T14:21:18.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48041be7:::benchmark_data_smithi194_103039_object124704:head# 2024-04-29T14:21:18.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480448c1:::benchmark_data_smithi194_103039_object63676:head# 2024-04-29T14:21:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48045f33:::benchmark_data_smithi194_103039_object134725:head# 2024-04-29T14:21:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4804b2cf:::benchmark_data_smithi194_103039_object68104:head# 2024-04-29T14:21:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48050b08:::benchmark_data_smithi194_103039_object120566:head# 2024-04-29T14:21:18.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4806d5e3:::benchmark_data_smithi194_103039_object84184:head# 2024-04-29T14:21:18.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4806ef6d:::benchmark_data_smithi194_103039_object123640:head# 2024-04-29T14:21:18.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48074b48:::benchmark_data_smithi194_103039_object67304:head# 2024-04-29T14:21:18.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4807552a:::benchmark_data_smithi194_103039_object110103:head# 2024-04-29T14:21:18.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48077c02:::benchmark_data_smithi194_103039_object98028:head# 2024-04-29T14:21:18.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48089e30:::benchmark_data_smithi194_103039_object73407:head# 2024-04-29T14:21:18.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4809ec3a:::benchmark_data_smithi194_103039_object155399:head# 2024-04-29T14:21:18.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480abdcb:::benchmark_data_smithi194_103039_object76071:head# 2024-04-29T14:21:18.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480b32cf:::benchmark_data_smithi194_103039_object89763:head# 2024-04-29T14:21:18.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480b44d6:::benchmark_data_smithi194_103039_object91949:head# 2024-04-29T14:21:18.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480be1cc:::benchmark_data_smithi194_103039_object138736:head# 2024-04-29T14:21:18.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480c1ada:::benchmark_data_smithi194_103039_object42576:head# 2024-04-29T14:21:18.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480c5ebb:::benchmark_data_smithi194_103039_object86350:head# 2024-04-29T14:21:18.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480c6743:::benchmark_data_smithi194_103039_object72205:head# 2024-04-29T14:21:18.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480cd66c:::benchmark_data_smithi194_103039_object89194:head# 2024-04-29T14:21:18.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480d7884:::benchmark_data_smithi194_103039_object54725:head# 2024-04-29T14:21:18.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480dfbf9:::benchmark_data_smithi194_103039_object126367:head# 2024-04-29T14:21:18.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480ea545:::benchmark_data_smithi194_103039_object91554:head# 2024-04-29T14:21:18.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480ea95d:::benchmark_data_smithi194_103039_object53199:head# 2024-04-29T14:21:18.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:480f2597:::benchmark_data_smithi194_103039_object89100:head# 2024-04-29T14:21:18.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48102ff4:::benchmark_data_smithi194_103039_object110927:head# 2024-04-29T14:21:18.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48117626:::benchmark_data_smithi194_103039_object83983:head# 2024-04-29T14:21:18.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48118715:::benchmark_data_smithi194_103039_object154434:head# 2024-04-29T14:21:18.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48121ddf:::benchmark_data_smithi194_103039_object55439:head# 2024-04-29T14:21:18.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4812eb97:::benchmark_data_smithi194_103039_object155453:head# 2024-04-29T14:21:18.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48138161:::benchmark_data_smithi194_103039_object42914:head# 2024-04-29T14:21:18.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48145068:::benchmark_data_smithi194_103039_object102686:head# 2024-04-29T14:21:18.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48147c16:::benchmark_data_smithi194_103039_object105011:head# 2024-04-29T14:21:18.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4814ab8e:::benchmark_data_smithi194_103039_object101665:head# 2024-04-29T14:21:18.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4814be3a:::benchmark_data_smithi194_103039_object111345:head# 2024-04-29T14:21:18.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4814d092:::benchmark_data_smithi194_103039_object87195:head# 2024-04-29T14:21:18.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4814f6cf:::benchmark_data_smithi194_103039_object145891:head# 2024-04-29T14:21:18.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48156deb:::benchmark_data_smithi194_103039_object134873:head# 2024-04-29T14:21:18.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48180eab:::benchmark_data_smithi194_103039_object82542:head# 2024-04-29T14:21:18.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4818d9ba:::benchmark_data_smithi194_103039_object126446:head# 2024-04-29T14:21:18.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4819dae8:::benchmark_data_smithi194_103039_object74722:head# 2024-04-29T14:21:18.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481a2cbc:::benchmark_data_smithi194_103039_object98307:head# 2024-04-29T14:21:18.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481ab710:::benchmark_data_smithi194_103039_object72654:head# 2024-04-29T14:21:18.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481b0367:::benchmark_data_smithi194_103039_object80833:head# 2024-04-29T14:21:18.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481b0be5:::benchmark_data_smithi194_103039_object89283:head# 2024-04-29T14:21:18.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481b7afe:::benchmark_data_smithi194_103039_object63763:head# 2024-04-29T14:21:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481bb7b7:::benchmark_data_smithi194_103039_object103264:head# 2024-04-29T14:21:18.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481e6d8f:::benchmark_data_smithi194_103039_object83377:head# 2024-04-29T14:21:18.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:481f0a77:::benchmark_data_smithi194_103039_object92182:head# 2024-04-29T14:21:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48203230:::benchmark_data_smithi194_103039_object114057:head# 2024-04-29T14:21:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4820c56c:::benchmark_data_smithi194_103039_object151156:head# 2024-04-29T14:21:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4820d3ea:::benchmark_data_smithi194_103039_object136049:head# 2024-04-29T14:21:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48211962:::benchmark_data_smithi194_103039_object48713:head# 2024-04-29T14:21:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48216300:::benchmark_data_smithi194_103039_object87101:head# 2024-04-29T14:21:18.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4822adc3:::benchmark_data_smithi194_103039_object103352:head# 2024-04-29T14:21:18.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4822b278:::benchmark_data_smithi194_103039_object42277:head# 2024-04-29T14:21:18.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48237c45:::benchmark_data_smithi194_103039_object55629:head# 2024-04-29T14:21:18.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48243593:::benchmark_data_smithi194_103039_object95153:head# 2024-04-29T14:21:18.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48245904:::benchmark_data_smithi194_103039_object62193:head# 2024-04-29T14:21:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482480a2:::benchmark_data_smithi194_103039_object109212:head# 2024-04-29T14:21:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4824f689:::benchmark_data_smithi194_103039_object136312:head# 2024-04-29T14:21:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4824f9c3:::benchmark_data_smithi194_103039_object44299:head# 2024-04-29T14:21:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482522c3:::benchmark_data_smithi194_103039_object111297:head# 2024-04-29T14:21:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4825485e:::benchmark_data_smithi194_103039_object136445:head# 2024-04-29T14:21:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4827c0c0:::benchmark_data_smithi194_103039_object68130:head# 2024-04-29T14:21:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4828032f:::benchmark_data_smithi194_103039_object86931:head# 2024-04-29T14:21:18.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482842be:::benchmark_data_smithi194_103039_object66916:head# 2024-04-29T14:21:18.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4829c3c0:::benchmark_data_smithi194_103039_object128844:head# 2024-04-29T14:21:18.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482a74cc:::benchmark_data_smithi194_103039_object143007:head# 2024-04-29T14:21:18.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482c2a8b:::benchmark_data_smithi194_103039_object43721:head# 2024-04-29T14:21:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482c93e7:::benchmark_data_smithi194_103039_object71048:head# 2024-04-29T14:21:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482d551e:::benchmark_data_smithi194_103039_object83415:head# 2024-04-29T14:21:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482ece5b:::benchmark_data_smithi194_103039_object85999:head# 2024-04-29T14:21:18.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482f3965:::benchmark_data_smithi194_103039_object143374:head# 2024-04-29T14:21:18.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482f8fda:::benchmark_data_smithi194_103039_object116866:head# 2024-04-29T14:21:18.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482fad09:::benchmark_data_smithi194_103039_object62697:head# 2024-04-29T14:21:18.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482fbb45:::benchmark_data_smithi194_103039_object51333:head# 2024-04-29T14:21:18.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:482fefa4:::benchmark_data_smithi194_103039_object130695:head# 2024-04-29T14:21:18.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48310f49:::benchmark_data_smithi194_103039_object126107:head# 2024-04-29T14:21:18.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483148da:::benchmark_data_smithi194_103039_object151186:head# 2024-04-29T14:21:18.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4831bff0:::benchmark_data_smithi194_103039_object74535:head# 2024-04-29T14:21:18.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4831fd63:::benchmark_data_smithi194_103039_object94712:head# 2024-04-29T14:21:18.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48322ed9:::benchmark_data_smithi194_103039_object50783:head# 2024-04-29T14:21:18.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4832a653:::benchmark_data_smithi194_103039_object107507:head# 2024-04-29T14:21:18.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4832f0f5:::benchmark_data_smithi194_103039_object135759:head# 2024-04-29T14:21:18.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48331d0d:::benchmark_data_smithi194_103039_object62614:head# 2024-04-29T14:21:18.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4834142b:::benchmark_data_smithi194_103039_object43348:head# 2024-04-29T14:21:18.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48348824:::benchmark_data_smithi194_103039_object113771:head# 2024-04-29T14:21:18.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48349da7:::benchmark_data_smithi194_103039_object49115:head# 2024-04-29T14:21:18.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4834e18f:::benchmark_data_smithi194_103039_object104686:head# 2024-04-29T14:21:18.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483507b8:::benchmark_data_smithi194_103039_object91406:head# 2024-04-29T14:21:18.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48351813:::benchmark_data_smithi194_103039_object65061:head# 2024-04-29T14:21:18.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4835eb5d:::benchmark_data_smithi194_103039_object120092:head# 2024-04-29T14:21:18.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4836f828:::benchmark_data_smithi194_103039_object138220:head# 2024-04-29T14:21:18.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4836f8f4:::benchmark_data_smithi194_103039_object50851:head# 2024-04-29T14:21:18.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48378b7d:::benchmark_data_smithi194_103039_object122943:head# 2024-04-29T14:21:18.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4837d315:::benchmark_data_smithi194_103039_object55487:head# 2024-04-29T14:21:18.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48380088:::benchmark_data_smithi194_103039_object118367:head# 2024-04-29T14:21:18.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48387363:::benchmark_data_smithi194_103039_object50375:head# 2024-04-29T14:21:18.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48393143:::benchmark_data_smithi194_103039_object106101:head# 2024-04-29T14:21:18.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483958fe:::benchmark_data_smithi194_103039_object48197:head# 2024-04-29T14:21:18.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4839db70:::benchmark_data_smithi194_103039_object65184:head# 2024-04-29T14:21:18.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4839e790:::benchmark_data_smithi194_103039_object71472:head# 2024-04-29T14:21:18.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4839fd5a:::benchmark_data_smithi194_103039_object108683:head# 2024-04-29T14:21:18.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483c715c:::benchmark_data_smithi194_103039_object78819:head# 2024-04-29T14:21:18.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483c9ea8:::benchmark_data_smithi194_103039_object154573:head# 2024-04-29T14:21:18.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483cc1ff:::benchmark_data_smithi194_103039_object59205:head# 2024-04-29T14:21:18.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483e8d17:::benchmark_data_smithi194_103039_object93825:head# 2024-04-29T14:21:18.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:483e995f:::benchmark_data_smithi194_103039_object111051:head# 2024-04-29T14:21:18.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4840497c:::benchmark_data_smithi194_103039_object143609:head# 2024-04-29T14:21:18.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48406f97:::benchmark_data_smithi194_103039_object52377:head# 2024-04-29T14:21:18.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484165b7:::benchmark_data_smithi194_103039_object50778:head# 2024-04-29T14:21:18.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4841ce1e:::benchmark_data_smithi194_103039_object119807:head# 2024-04-29T14:21:18.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4843f48e:::benchmark_data_smithi194_103039_object93518:head# 2024-04-29T14:21:18.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4843f741:::benchmark_data_smithi194_103039_object100980:head# 2024-04-29T14:21:18.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48441d35:::benchmark_data_smithi194_103039_object95147:head# 2024-04-29T14:21:18.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4844778c:::benchmark_data_smithi194_103039_object127750:head# 2024-04-29T14:21:18.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48452263:::benchmark_data_smithi194_103039_object55301:head# 2024-04-29T14:21:18.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4845fd91:::benchmark_data_smithi194_103039_object58982:head# 2024-04-29T14:21:18.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48463bc5:::benchmark_data_smithi194_103039_object76691:head# 2024-04-29T14:21:18.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4846f31d:::benchmark_data_smithi194_103039_object48745:head# 2024-04-29T14:21:18.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484770e6:::benchmark_data_smithi194_103039_object147257:head# 2024-04-29T14:21:18.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4848a636:::benchmark_data_smithi194_103039_object114092:head# 2024-04-29T14:21:18.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4849edc5:::benchmark_data_smithi194_103039_object99794:head# 2024-04-29T14:21:18.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484a09c4:::benchmark_data_smithi194_103039_object121836:head# 2024-04-29T14:21:18.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484a23be:::benchmark_data_smithi194_103039_object117139:head# 2024-04-29T14:21:18.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484bd121:::benchmark_data_smithi194_103039_object102878:head# 2024-04-29T14:21:18.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484c05fa:::benchmark_data_smithi194_103039_object107898:head# 2024-04-29T14:21:18.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484c7149:::benchmark_data_smithi194_103039_object125122:head# 2024-04-29T14:21:18.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484c71f0:::benchmark_data_smithi194_103039_object137480:head# 2024-04-29T14:21:18.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484e9d13:::benchmark_data_smithi194_103039_object56894:head# 2024-04-29T14:21:18.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484f5dc4:::benchmark_data_smithi194_103039_object91908:head# 2024-04-29T14:21:18.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:484f654d:::benchmark_data_smithi194_103039_object59217:head# 2024-04-29T14:21:18.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4850267a:::benchmark_data_smithi194_103039_object65633:head# 2024-04-29T14:21:18.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48503934:::benchmark_data_smithi194_103039_object46506:head# 2024-04-29T14:21:18.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485061c7:::benchmark_data_smithi194_103039_object88602:head# 2024-04-29T14:21:18.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4850830b:::benchmark_data_smithi194_103039_object101656:head# 2024-04-29T14:21:18.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48513118:::benchmark_data_smithi194_103039_object58441:head# 2024-04-29T14:21:18.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4851669d:::benchmark_data_smithi194_103039_object147479:head# 2024-04-29T14:21:18.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485189b4:::benchmark_data_smithi194_103039_object88988:head# 2024-04-29T14:21:18.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4851af26:::benchmark_data_smithi194_103039_object139271:head# 2024-04-29T14:21:18.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48527b1f:::benchmark_data_smithi194_103039_object124159:head# 2024-04-29T14:21:18.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4852a888:::benchmark_data_smithi194_103039_object51618:head# 2024-04-29T14:21:18.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485314ca:::benchmark_data_smithi194_103039_object82643:head# 2024-04-29T14:21:18.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48536c0b:::benchmark_data_smithi194_103039_object51622:head# 2024-04-29T14:21:18.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48541108:::benchmark_data_smithi194_103039_object103878:head# 2024-04-29T14:21:18.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48545f20:::benchmark_data_smithi194_103039_object109457:head# 2024-04-29T14:21:18.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4855974f:::benchmark_data_smithi194_103039_object70747:head# 2024-04-29T14:21:18.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4855a5ae:::benchmark_data_smithi194_103039_object93498:head# 2024-04-29T14:21:18.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4855a7e0:::benchmark_data_smithi194_103039_object73790:head# 2024-04-29T14:21:18.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4855c641:::benchmark_data_smithi194_103039_object103904:head# 2024-04-29T14:21:18.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48561f2e:::benchmark_data_smithi194_103039_object46032:head# 2024-04-29T14:21:18.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4856332b:::benchmark_data_smithi194_103039_object88520:head# 2024-04-29T14:21:18.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485721a2:::benchmark_data_smithi194_103039_object149785:head# 2024-04-29T14:21:18.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4857f32b:::benchmark_data_smithi194_103039_object43420:head# 2024-04-29T14:21:18.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4857feeb:::benchmark_data_smithi194_103039_object60645:head# 2024-04-29T14:21:18.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48586eb5:::benchmark_data_smithi194_103039_object113580:head# 2024-04-29T14:21:18.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48589eb8:::benchmark_data_smithi194_103039_object111417:head# 2024-04-29T14:21:18.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485904bd:::benchmark_data_smithi194_103039_object79031:head# 2024-04-29T14:21:18.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48595790:::benchmark_data_smithi194_103039_object146326:head# 2024-04-29T14:21:18.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4859fbad:::benchmark_data_smithi194_103039_object152382:head# 2024-04-29T14:21:18.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485aeeb7:::benchmark_data_smithi194_103039_object47848:head# 2024-04-29T14:21:18.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485ba758:::benchmark_data_smithi194_103039_object60574:head# 2024-04-29T14:21:18.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485c86f2:::benchmark_data_smithi194_103039_object125913:head# 2024-04-29T14:21:18.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485c972c:::benchmark_data_smithi194_103039_object129505:head# 2024-04-29T14:21:18.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:485dd370:::benchmark_data_smithi194_103039_object66799:head# 2024-04-29T14:21:18.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486045d1:::benchmark_data_smithi194_103039_object105343:head# 2024-04-29T14:21:18.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48606758:::benchmark_data_smithi194_103039_object83535:head# 2024-04-29T14:21:18.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48609c2c:::benchmark_data_smithi194_103039_object129719:head# 2024-04-29T14:21:18.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4860b39b:::benchmark_data_smithi194_103039_object110580:head# 2024-04-29T14:21:18.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4860d6db:::benchmark_data_smithi194_103039_object103917:head# 2024-04-29T14:21:18.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4861cedf:::benchmark_data_smithi194_103039_object134821:head# 2024-04-29T14:21:18.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4861d50a:::benchmark_data_smithi194_103039_object84566:head# 2024-04-29T14:21:18.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486506d7:::benchmark_data_smithi194_103039_object149061:head# 2024-04-29T14:21:18.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4865b437:::benchmark_data_smithi194_103039_object155136:head# 2024-04-29T14:21:18.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48663cba:::benchmark_data_smithi194_103039_object93416:head# 2024-04-29T14:21:18.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48669097:::benchmark_data_smithi194_103039_object59728:head# 2024-04-29T14:21:18.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4866a07f:::benchmark_data_smithi194_103039_object131820:head# 2024-04-29T14:21:18.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4866d686:::benchmark_data_smithi194_103039_object141702:head# 2024-04-29T14:21:18.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4866dbd8:::benchmark_data_smithi194_103039_object63552:head# 2024-04-29T14:21:18.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4867d82c:::benchmark_data_smithi194_103039_object42391:head# 2024-04-29T14:21:18.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4867dc08:::benchmark_data_smithi194_103039_object96457:head# 2024-04-29T14:21:18.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4868b019:::benchmark_data_smithi194_103039_object102635:head# 2024-04-29T14:21:18.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48690f59:::benchmark_data_smithi194_103039_object103155:head# 2024-04-29T14:21:18.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48696030:::benchmark_data_smithi194_103039_object69202:head# 2024-04-29T14:21:18.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486bc366:::benchmark_data_smithi194_103039_object77029:head# 2024-04-29T14:21:18.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486be452:::benchmark_data_smithi194_103039_object56271:head# 2024-04-29T14:21:18.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486dc02e:::benchmark_data_smithi194_103039_object99099:head# 2024-04-29T14:21:18.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486dd333:::benchmark_data_smithi194_103039_object69309:head# 2024-04-29T14:21:18.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486eb272:::benchmark_data_smithi194_103039_object65807:head# 2024-04-29T14:21:18.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486eb5d9:::benchmark_data_smithi194_103039_object139631:head# 2024-04-29T14:21:18.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486f891a:::benchmark_data_smithi194_103039_object99454:head# 2024-04-29T14:21:18.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:486f939d:::benchmark_data_smithi194_103039_object110860:head# 2024-04-29T14:21:18.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48703575:::benchmark_data_smithi194_103039_object55524:head# 2024-04-29T14:21:18.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487076c7:::benchmark_data_smithi194_103039_object57424:head# 2024-04-29T14:21:18.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48717ce2:::benchmark_data_smithi194_103039_object45462:head# 2024-04-29T14:21:18.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4871c1b3:::benchmark_data_smithi194_103039_object67853:head# 2024-04-29T14:21:18.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4871cdee:::benchmark_data_smithi194_103039_object130698:head# 2024-04-29T14:21:18.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4871e5d4:::benchmark_data_smithi194_103039_object84392:head# 2024-04-29T14:21:18.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48734c23:::benchmark_data_smithi194_103039_object104778:head# 2024-04-29T14:21:18.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487366a5:::benchmark_data_smithi194_103039_object47974:head# 2024-04-29T14:21:18.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48739768:::benchmark_data_smithi194_103039_object151799:head# 2024-04-29T14:21:18.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4873a2d9:::benchmark_data_smithi194_103039_object113467:head# 2024-04-29T14:21:18.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4873a796:::benchmark_data_smithi194_103039_object130217:head# 2024-04-29T14:21:18.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4874dd62:::benchmark_data_smithi194_103039_object83487:head# 2024-04-29T14:21:18.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4874e5d7:::benchmark_data_smithi194_103039_object149977:head# 2024-04-29T14:21:18.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4874e608:::benchmark_data_smithi194_103039_object72946:head# 2024-04-29T14:21:18.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4874ebac:::benchmark_data_smithi194_103039_object112333:head# 2024-04-29T14:21:18.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4874ec3a:::benchmark_data_smithi194_103039_object55918:head# 2024-04-29T14:21:18.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48755f62:::benchmark_data_smithi194_103039_object43503:head# 2024-04-29T14:21:18.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48760b5d:::benchmark_data_smithi194_103039_object44847:head# 2024-04-29T14:21:18.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48764ada:::benchmark_data_smithi194_103039_object56023:head# 2024-04-29T14:21:18.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48783a53:::benchmark_data_smithi194_103039_object135007:head# 2024-04-29T14:21:18.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4878ee6f:::benchmark_data_smithi194_103039_object81536:head# 2024-04-29T14:21:18.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487a22d6:::benchmark_data_smithi194_103039_object149442:head# 2024-04-29T14:21:18.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487a8efa:::benchmark_data_smithi194_103039_object66595:head# 2024-04-29T14:21:18.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487aa73b:::benchmark_data_smithi194_103039_object62024:head# 2024-04-29T14:21:18.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487b131c:::benchmark_data_smithi194_103039_object44662:head# 2024-04-29T14:21:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487b1986:::benchmark_data_smithi194_103039_object117429:head# 2024-04-29T14:21:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487b947a:::benchmark_data_smithi194_103039_object49760:head# 2024-04-29T14:21:18.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487c099b:::benchmark_data_smithi194_103039_object155593:head# 2024-04-29T14:21:18.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487c16d8:::benchmark_data_smithi194_103039_object52098:head# 2024-04-29T14:21:18.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487e3739:::benchmark_data_smithi194_103039_object120443:head# 2024-04-29T14:21:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487eff3e:::benchmark_data_smithi194_103039_object139099:head# 2024-04-29T14:21:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487f098a:::benchmark_data_smithi194_103039_object111189:head# 2024-04-29T14:21:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487f109f:::benchmark_data_smithi194_103039_object77105:head# 2024-04-29T14:21:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487f5f58:::benchmark_data_smithi194_103039_object80624:head# 2024-04-29T14:21:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487f7915:::benchmark_data_smithi194_103039_object46270:head# 2024-04-29T14:21:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487f847b:::benchmark_data_smithi194_103039_object118976:head# 2024-04-29T14:21:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:487fd5c7:::benchmark_data_smithi194_103039_object80186:head# 2024-04-29T14:21:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4880646c:::benchmark_data_smithi194_103039_object52563:head# 2024-04-29T14:21:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48808af0:::benchmark_data_smithi194_103039_object43145:head# 2024-04-29T14:21:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48809df4:::benchmark_data_smithi194_103039_object115621:head# 2024-04-29T14:21:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488143ea:::benchmark_data_smithi194_103039_object123187:head# 2024-04-29T14:21:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48819e02:::benchmark_data_smithi194_103039_object72018:head# 2024-04-29T14:21:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48841380:::benchmark_data_smithi194_103039_object131544:head# 2024-04-29T14:21:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4884705a:::benchmark_data_smithi194_103039_object123205:head# 2024-04-29T14:21:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4884b67d:::benchmark_data_smithi194_103039_object94134:head# 2024-04-29T14:21:18.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48860588:::benchmark_data_smithi194_103039_object131966:head# 2024-04-29T14:21:18.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488814e5:::benchmark_data_smithi194_103039_object75395:head# 2024-04-29T14:21:18.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48883e21:::benchmark_data_smithi194_103039_object133743:head# 2024-04-29T14:21:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4888db53:::benchmark_data_smithi194_103039_object51837:head# 2024-04-29T14:21:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4889e4fc:::benchmark_data_smithi194_103039_object148256:head# 2024-04-29T14:21:18.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488a5cae:::benchmark_data_smithi194_103039_object91694:head# 2024-04-29T14:21:18.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488ae99f:::benchmark_data_smithi194_103039_object109645:head# 2024-04-29T14:21:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488bbf63:::benchmark_data_smithi194_103039_object133538:head# 2024-04-29T14:21:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488bc57a:::benchmark_data_smithi194_103039_object134642:head# 2024-04-29T14:21:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488c4869:::benchmark_data_smithi194_103039_object55281:head# 2024-04-29T14:21:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488cc2be:::benchmark_data_smithi194_103039_object104394:head# 2024-04-29T14:21:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488ccf40:::benchmark_data_smithi194_103039_object56701:head# 2024-04-29T14:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488cd0f5:::benchmark_data_smithi194_103039_object62053:head# 2024-04-29T14:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488eb658:::benchmark_data_smithi194_103039_object122668:head# 2024-04-29T14:21:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488ee4fb:::benchmark_data_smithi194_103039_object140497:head# 2024-04-29T14:21:18.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:488ef117:::benchmark_data_smithi194_103039_object43257:head# 2024-04-29T14:21:18.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4890b303:::benchmark_data_smithi194_103039_object102933:head# 2024-04-29T14:21:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48918127:::benchmark_data_smithi194_103039_object102172:head# 2024-04-29T14:21:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48923325:::benchmark_data_smithi194_103039_object103344:head# 2024-04-29T14:21:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48929377:::benchmark_data_smithi194_103039_object155007:head# 2024-04-29T14:21:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48933ec5:::benchmark_data_smithi194_103039_object62592:head# 2024-04-29T14:21:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48935685:::benchmark_data_smithi194_103039_object78774:head# 2024-04-29T14:21:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48937fe7:::benchmark_data_smithi194_103039_object84734:head# 2024-04-29T14:21:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4893b4d5:::benchmark_data_smithi194_103039_object122421:head# 2024-04-29T14:21:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4895f46c:::benchmark_data_smithi194_103039_object114729:head# 2024-04-29T14:21:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4896591c:::benchmark_data_smithi194_103039_object89179:head# 2024-04-29T14:21:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4896b470:::benchmark_data_smithi194_103039_object127864:head# 2024-04-29T14:21:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4896e6e8:::benchmark_data_smithi194_103039_object103592:head# 2024-04-29T14:21:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4897dfcc:::benchmark_data_smithi194_103039_object140757:head# 2024-04-29T14:21:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4899928e:::benchmark_data_smithi194_103039_object82735:head# 2024-04-29T14:21:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489a2383:::benchmark_data_smithi194_103039_object98859:head# 2024-04-29T14:21:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489ae123:::benchmark_data_smithi194_103039_object110467:head# 2024-04-29T14:21:18.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489b1ca6:::benchmark_data_smithi194_103039_object146985:head# 2024-04-29T14:21:18.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489b1e69:::benchmark_data_smithi194_103039_object51849:head# 2024-04-29T14:21:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489b8c3b:::benchmark_data_smithi194_103039_object101229:head# 2024-04-29T14:21:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489c6468:::benchmark_data_smithi194_103039_object64551:head# 2024-04-29T14:21:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489c6f98:::benchmark_data_smithi194_103039_object83601:head# 2024-04-29T14:21:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489e50aa:::benchmark_data_smithi194_103039_object46454:head# 2024-04-29T14:21:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489e9a11:::benchmark_data_smithi194_103039_object112954:head# 2024-04-29T14:21:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:489f8ac0:::benchmark_data_smithi194_103039_object91440:head# 2024-04-29T14:21:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a05a83:::benchmark_data_smithi194_103039_object121156:head# 2024-04-29T14:21:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a1427f:::benchmark_data_smithi194_103039_object134233:head# 2024-04-29T14:21:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a1c922:::benchmark_data_smithi194_103039_object150474:head# 2024-04-29T14:21:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a2248a:::benchmark_data_smithi194_103039_object61822:head# 2024-04-29T14:21:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a27558:::benchmark_data_smithi194_103039_object86182:head# 2024-04-29T14:21:18.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a2f16a:::benchmark_data_smithi194_103039_object141179:head# 2024-04-29T14:21:18.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a37466:::benchmark_data_smithi194_103039_object43493:head# 2024-04-29T14:21:18.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a56b28:::benchmark_data_smithi194_103039_object114753:head# 2024-04-29T14:21:18.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a76b13:::benchmark_data_smithi194_103039_object61838:head# 2024-04-29T14:21:18.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a774ba:::benchmark_data_smithi194_103039_object98284:head# 2024-04-29T14:21:18.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a7a9c7:::benchmark_data_smithi194_103039_object86036:head# 2024-04-29T14:21:18.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a7b15e:::benchmark_data_smithi194_103039_object74401:head# 2024-04-29T14:21:18.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a90cb1:::benchmark_data_smithi194_103039_object64300:head# 2024-04-29T14:21:18.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a941dc:::benchmark_data_smithi194_103039_object126443:head# 2024-04-29T14:21:18.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a94a96:::benchmark_data_smithi194_103039_object140416:head# 2024-04-29T14:21:18.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48a9ae8a:::benchmark_data_smithi194_103039_object156585:head# 2024-04-29T14:21:18.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48aa19ab:::benchmark_data_smithi194_103039_object46749:head# 2024-04-29T14:21:18.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48aa72e9:::benchmark_data_smithi194_103039_object139991:head# 2024-04-29T14:21:18.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48aade03:::benchmark_data_smithi194_103039_object100109:head# 2024-04-29T14:21:18.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48acb2e1:::benchmark_data_smithi194_103039_object85498:head# 2024-04-29T14:21:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48acc351:::benchmark_data_smithi194_103039_object123220:head# 2024-04-29T14:21:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ad11a6:::benchmark_data_smithi194_103039_object96659:head# 2024-04-29T14:21:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ad5f98:::benchmark_data_smithi194_103039_object88334:head# 2024-04-29T14:21:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ad6e69:::benchmark_data_smithi194_103039_object109640:head# 2024-04-29T14:21:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48aec045:::benchmark_data_smithi194_103039_object59421:head# 2024-04-29T14:21:18.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48af133c:::benchmark_data_smithi194_103039_object52773:head# 2024-04-29T14:21:18.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48af55a2:::benchmark_data_smithi194_103039_object129909:head# 2024-04-29T14:21:18.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48afa78a:::benchmark_data_smithi194_103039_object102481:head# 2024-04-29T14:21:18.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b14ca3:::benchmark_data_smithi194_103039_object92916:head# 2024-04-29T14:21:18.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b18141:::benchmark_data_smithi194_103039_object97629:head# 2024-04-29T14:21:18.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b18cf7:::benchmark_data_smithi194_103039_object79396:head# 2024-04-29T14:21:18.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b1ac87:::benchmark_data_smithi194_103039_object48208:head# 2024-04-29T14:21:18.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b299cb:::benchmark_data_smithi194_103039_object75428:head# 2024-04-29T14:21:18.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b3898d:::benchmark_data_smithi194_103039_object123925:head# 2024-04-29T14:21:18.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b4a677:::benchmark_data_smithi194_103039_object44473:head# 2024-04-29T14:21:18.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b6b479:::benchmark_data_smithi194_103039_object115035:head# 2024-04-29T14:21:18.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b7b33a:::benchmark_data_smithi194_103039_object73850:head# 2024-04-29T14:21:18.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b81c8c:::benchmark_data_smithi194_103039_object135715:head# 2024-04-29T14:21:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b86afc:::benchmark_data_smithi194_103039_object60469:head# 2024-04-29T14:21:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b8f3cc:::benchmark_data_smithi194_103039_object126227:head# 2024-04-29T14:21:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b99a04:::benchmark_data_smithi194_103039_object43454:head# 2024-04-29T14:21:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48b9e0ba:::benchmark_data_smithi194_103039_object98084:head# 2024-04-29T14:21:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ba2ede:::benchmark_data_smithi194_103039_object98697:head# 2024-04-29T14:21:18.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ba830b:::benchmark_data_smithi194_103039_object92561:head# 2024-04-29T14:21:18.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bb4c1b:::benchmark_data_smithi194_103039_object153731:head# 2024-04-29T14:21:18.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bc0403:::benchmark_data_smithi194_103039_object59731:head# 2024-04-29T14:21:18.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bc4189:::benchmark_data_smithi194_103039_object46676:head# 2024-04-29T14:21:18.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bc5455:::benchmark_data_smithi194_103039_object156091:head# 2024-04-29T14:21:18.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bcca4a:::benchmark_data_smithi194_103039_object128314:head# 2024-04-29T14:21:18.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bcf13c:::benchmark_data_smithi194_103039_object43221:head# 2024-04-29T14:21:18.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bd7e12:::benchmark_data_smithi194_103039_object100943:head# 2024-04-29T14:21:18.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bda311:::benchmark_data_smithi194_103039_object131265:head# 2024-04-29T14:21:18.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48be37e7:::benchmark_data_smithi194_103039_object109217:head# 2024-04-29T14:21:18.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48be8a92:::benchmark_data_smithi194_103039_object79532:head# 2024-04-29T14:21:18.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48beb587:::benchmark_data_smithi194_103039_object94507:head# 2024-04-29T14:21:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bf51a9:::benchmark_data_smithi194_103039_object53675:head# 2024-04-29T14:21:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bfe6d3:::benchmark_data_smithi194_103039_object89392:head# 2024-04-29T14:21:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48bff397:::benchmark_data_smithi194_103039_object139693:head# 2024-04-29T14:21:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c16997:::benchmark_data_smithi194_103039_object52178:head# 2024-04-29T14:21:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c173ca:::benchmark_data_smithi194_103039_object53058:head# 2024-04-29T14:21:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c224d7:::benchmark_data_smithi194_103039_object97833:head# 2024-04-29T14:21:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c246f0:::benchmark_data_smithi194_103039_object102446:head# 2024-04-29T14:21:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c25a91:::benchmark_data_smithi194_103039_object105240:head# 2024-04-29T14:21:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c350c9:::benchmark_data_smithi194_103039_object131426:head# 2024-04-29T14:21:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c41b14:::benchmark_data_smithi194_103039_object113930:head# 2024-04-29T14:21:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c429f1:::benchmark_data_smithi194_103039_object127879:head# 2024-04-29T14:21:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c430be:::benchmark_data_smithi194_103039_object60366:head# 2024-04-29T14:21:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c4a006:::benchmark_data_smithi194_103039_object137290:head# 2024-04-29T14:21:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c4e435:::benchmark_data_smithi194_103039_object105987:head# 2024-04-29T14:21:18.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c79a64:::benchmark_data_smithi194_103039_object140277:head# 2024-04-29T14:21:18.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c79dc6:::benchmark_data_smithi194_103039_object103607:head# 2024-04-29T14:21:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c7a6b4:::benchmark_data_smithi194_103039_object66293:head# 2024-04-29T14:21:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c8cc0f:::benchmark_data_smithi194_103039_object69866:head# 2024-04-29T14:21:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c8d438:::benchmark_data_smithi194_103039_object139088:head# 2024-04-29T14:21:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48c98835:::benchmark_data_smithi194_103039_object121506:head# 2024-04-29T14:21:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ca42f9:::benchmark_data_smithi194_103039_object98698:head# 2024-04-29T14:21:18.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ca85e4:::benchmark_data_smithi194_103039_object99928:head# 2024-04-29T14:21:18.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48cb2c08:::benchmark_data_smithi194_103039_object42943:head# 2024-04-29T14:21:18.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48cb4ebe:::benchmark_data_smithi194_103039_object51756:head# 2024-04-29T14:21:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48cc7132:::benchmark_data_smithi194_103039_object151079:head# 2024-04-29T14:21:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ccb4ed:::benchmark_data_smithi194_103039_object66573:head# 2024-04-29T14:21:18.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48cd95b2:::benchmark_data_smithi194_103039_object128497:head# 2024-04-29T14:21:18.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ce5d65:::benchmark_data_smithi194_103039_object134620:head# 2024-04-29T14:21:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ceb580:::benchmark_data_smithi194_103039_object135117:head# 2024-04-29T14:21:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ced875:::benchmark_data_smithi194_103039_object116133:head# 2024-04-29T14:21:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ceeb53:::benchmark_data_smithi194_103039_object73177:head# 2024-04-29T14:21:18.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ceef66:::benchmark_data_smithi194_103039_object128912:head# 2024-04-29T14:21:18.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d09bde:::benchmark_data_smithi194_103039_object140647:head# 2024-04-29T14:21:18.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d0e2d1:::benchmark_data_smithi194_103039_object147043:head# 2024-04-29T14:21:18.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d1547a:::benchmark_data_smithi194_103039_object97160:head# 2024-04-29T14:21:18.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d1b3c6:::benchmark_data_smithi194_103039_object77608:head# 2024-04-29T14:21:18.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d25d64:::benchmark_data_smithi194_103039_object146202:head# 2024-04-29T14:21:18.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d266e0:::benchmark_data_smithi194_103039_object62889:head# 2024-04-29T14:21:18.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d3dfcc:::benchmark_data_smithi194_103039_object95889:head# 2024-04-29T14:21:18.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d3e366:::benchmark_data_smithi194_103039_object139855:head# 2024-04-29T14:21:18.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d426fe:::benchmark_data_smithi194_103039_object98439:head# 2024-04-29T14:21:18.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d45780:::benchmark_data_smithi194_103039_object77342:head# 2024-04-29T14:21:18.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d460f4:::benchmark_data_smithi194_103039_object114004:head# 2024-04-29T14:21:18.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d545e9:::benchmark_data_smithi194_103039_object92381:head# 2024-04-29T14:21:18.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d56ee9:::benchmark_data_smithi194_103039_object85079:head# 2024-04-29T14:21:18.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d62529:::benchmark_data_smithi194_103039_object81717:head# 2024-04-29T14:21:18.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d6e946:::benchmark_data_smithi194_103039_object75269:head# 2024-04-29T14:21:18.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d70853:::benchmark_data_smithi194_103039_object62115:head# 2024-04-29T14:21:18.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d7b472:::benchmark_data_smithi194_103039_object84762:head# 2024-04-29T14:21:18.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d7b53a:::benchmark_data_smithi194_103039_object114717:head# 2024-04-29T14:21:18.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d7e0fb:::benchmark_data_smithi194_103039_object135636:head# 2024-04-29T14:21:18.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d86636:::benchmark_data_smithi194_103039_object133550:head# 2024-04-29T14:21:18.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d989e9:::benchmark_data_smithi194_103039_object144808:head# 2024-04-29T14:21:18.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d9d3ce:::benchmark_data_smithi194_103039_object66572:head# 2024-04-29T14:21:18.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48d9de9f:::benchmark_data_smithi194_103039_object72766:head# 2024-04-29T14:21:18.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48dab77c:::benchmark_data_smithi194_103039_object48658:head# 2024-04-29T14:21:18.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48dadc3a:::benchmark_data_smithi194_103039_object66891:head# 2024-04-29T14:21:18.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48db0f2d:::benchmark_data_smithi194_103039_object132125:head# 2024-04-29T14:21:18.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48db861b:::benchmark_data_smithi194_103039_object96856:head# 2024-04-29T14:21:18.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48dc9f81:::benchmark_data_smithi194_103039_object150437:head# 2024-04-29T14:21:18.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48dcf98b:::benchmark_data_smithi194_103039_object88113:head# 2024-04-29T14:21:18.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48de0c6b:::benchmark_data_smithi194_103039_object95366:head# 2024-04-29T14:21:18.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48de3ed7:::benchmark_data_smithi194_103039_object146171:head# 2024-04-29T14:21:18.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48dedbb3:::benchmark_data_smithi194_103039_object152739:head# 2024-04-29T14:21:18.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48dfa1f1:::benchmark_data_smithi194_103039_object71154:head# 2024-04-29T14:21:18.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e02f5e:::benchmark_data_smithi194_103039_object77148:head# 2024-04-29T14:21:18.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e16987:::benchmark_data_smithi194_103039_object146851:head# 2024-04-29T14:21:18.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e19192:::benchmark_data_smithi194_103039_object142778:head# 2024-04-29T14:21:18.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e1b992:::benchmark_data_smithi194_103039_object135403:head# 2024-04-29T14:21:18.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e1fbd5:::benchmark_data_smithi194_103039_object68836:head# 2024-04-29T14:21:18.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e28e6d:::benchmark_data_smithi194_103039_object131716:head# 2024-04-29T14:21:18.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e2e0de:::benchmark_data_smithi194_103039_object82215:head# 2024-04-29T14:21:18.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e3824b:::benchmark_data_smithi194_103039_object131745:head# 2024-04-29T14:21:18.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e462ef:::benchmark_data_smithi194_103039_object102524:head# 2024-04-29T14:21:18.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e7cdbd:::benchmark_data_smithi194_103039_object64102:head# 2024-04-29T14:21:18.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e7d251:::benchmark_data_smithi194_103039_object143822:head# 2024-04-29T14:21:18.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e84f2c:::benchmark_data_smithi194_103039_object141196:head# 2024-04-29T14:21:18.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e8a917:::benchmark_data_smithi194_103039_object124797:head# 2024-04-29T14:21:18.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e8d87f:::benchmark_data_smithi194_103039_object118080:head# 2024-04-29T14:21:18.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48e95051:::benchmark_data_smithi194_103039_object128470:head# 2024-04-29T14:21:18.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48eb0886:::benchmark_data_smithi194_103039_object121307:head# 2024-04-29T14:21:18.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48eb5d45:::benchmark_data_smithi194_103039_object53668:head# 2024-04-29T14:21:18.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48eb6f6d:::benchmark_data_smithi194_103039_object137613:head# 2024-04-29T14:21:18.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ece246:::benchmark_data_smithi194_103039_object142132:head# 2024-04-29T14:21:18.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ed349e:::benchmark_data_smithi194_103039_object101191:head# 2024-04-29T14:21:18.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48edcb7f:::benchmark_data_smithi194_103039_object129521:head# 2024-04-29T14:21:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48edf272:::benchmark_data_smithi194_103039_object80256:head# 2024-04-29T14:21:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ef37d8:::benchmark_data_smithi194_103039_object148408:head# 2024-04-29T14:21:18.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ef966e:::benchmark_data_smithi194_103039_object55302:head# 2024-04-29T14:21:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48efa5a6:::benchmark_data_smithi194_103039_object53129:head# 2024-04-29T14:21:18.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f0148f:::benchmark_data_smithi194_103039_object84595:head# 2024-04-29T14:21:18.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f1abf4:::benchmark_data_smithi194_103039_object116162:head# 2024-04-29T14:21:18.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f1ec60:::benchmark_data_smithi194_103039_object155516:head# 2024-04-29T14:21:18.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f1ed1c:::benchmark_data_smithi194_103039_object56795:head# 2024-04-29T14:21:18.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f21f0b:::benchmark_data_smithi194_103039_object153205:head# 2024-04-29T14:21:18.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f29016:::benchmark_data_smithi194_103039_object74679:head# 2024-04-29T14:21:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f360f8:::benchmark_data_smithi194_103039_object65076:head# 2024-04-29T14:21:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f48006:::benchmark_data_smithi194_103039_object73477:head# 2024-04-29T14:21:18.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f48a18:::benchmark_data_smithi194_103039_object103735:head# 2024-04-29T14:21:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f53b1f:::benchmark_data_smithi194_103039_object141010:head# 2024-04-29T14:21:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f56314:::benchmark_data_smithi194_103039_object104459:head# 2024-04-29T14:21:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f5cef5:::benchmark_data_smithi194_103039_object127218:head# 2024-04-29T14:21:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f63529:::benchmark_data_smithi194_103039_object132822:head# 2024-04-29T14:21:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f638c8:::benchmark_data_smithi194_103039_object124877:head# 2024-04-29T14:21:18.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f6a92a:::benchmark_data_smithi194_103039_object44556:head# 2024-04-29T14:21:18.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f82d8d:::benchmark_data_smithi194_103039_object150834:head# 2024-04-29T14:21:18.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f85887:::benchmark_data_smithi194_103039_object145193:head# 2024-04-29T14:21:18.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f894bb:::benchmark_data_smithi194_103039_object73270:head# 2024-04-29T14:21:18.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f90c40:::benchmark_data_smithi194_103039_object115266:head# 2024-04-29T14:21:18.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f9252f:::benchmark_data_smithi194_103039_object92296:head# 2024-04-29T14:21:18.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f984c5:::benchmark_data_smithi194_103039_object59118:head# 2024-04-29T14:21:18.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f99ddb:::benchmark_data_smithi194_103039_object61377:head# 2024-04-29T14:21:18.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48f9e6a7:::benchmark_data_smithi194_103039_object115475:head# 2024-04-29T14:21:18.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48faca39:::benchmark_data_smithi194_103039_object156094:head# 2024-04-29T14:21:18.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48fb76e6:::benchmark_data_smithi194_103039_object73966:head# 2024-04-29T14:21:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48fd2756:::benchmark_data_smithi194_103039_object95036:head# 2024-04-29T14:21:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ff359c:::benchmark_data_smithi194_103039_object108035:head# 2024-04-29T14:21:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48ff575c:::benchmark_data_smithi194_103039_object48266:head# 2024-04-29T14:21:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:48fff605:::benchmark_data_smithi194_103039_object48845:head# 2024-04-29T14:21:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49016008:::benchmark_data_smithi194_103039_object85779:head# 2024-04-29T14:21:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490194af:::benchmark_data_smithi194_103039_object115739:head# 2024-04-29T14:21:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49024fd8:::benchmark_data_smithi194_103039_object62474:head# 2024-04-29T14:21:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49029c3e:::benchmark_data_smithi194_103039_object138978:head# 2024-04-29T14:21:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4903184a:::benchmark_data_smithi194_103039_object137594:head# 2024-04-29T14:21:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49033567:::benchmark_data_smithi194_103039_object141261:head# 2024-04-29T14:21:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49034678:::benchmark_data_smithi194_103039_object119975:head# 2024-04-29T14:21:18.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4903b025:::benchmark_data_smithi194_103039_object115233:head# 2024-04-29T14:21:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49045f93:::benchmark_data_smithi194_103039_object104900:head# 2024-04-29T14:21:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4904a760:::benchmark_data_smithi194_103039_object124515:head# 2024-04-29T14:21:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4904ce23:::benchmark_data_smithi194_103039_object96400:head# 2024-04-29T14:21:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49078e5f:::benchmark_data_smithi194_103039_object64044:head# 2024-04-29T14:21:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4907a46c:::benchmark_data_smithi194_103039_object54975:head# 2024-04-29T14:21:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4907c829:::benchmark_data_smithi194_103039_object154488:head# 2024-04-29T14:21:18.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4908bdc9:::benchmark_data_smithi194_103039_object75366:head# 2024-04-29T14:21:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4908f2a2:::benchmark_data_smithi194_103039_object59129:head# 2024-04-29T14:21:18.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490948a9:::benchmark_data_smithi194_103039_object50855:head# 2024-04-29T14:21:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490b67f9:::benchmark_data_smithi194_103039_object68315:head# 2024-04-29T14:21:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490c5ade:::benchmark_data_smithi194_103039_object148308:head# 2024-04-29T14:21:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490d322b:::benchmark_data_smithi194_103039_object127514:head# 2024-04-29T14:21:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490d7413:::benchmark_data_smithi194_103039_object79341:head# 2024-04-29T14:21:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490d7b05:::benchmark_data_smithi194_103039_object103685:head# 2024-04-29T14:21:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490ea528:::benchmark_data_smithi194_103039_object155202:head# 2024-04-29T14:21:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490ed5da:::benchmark_data_smithi194_103039_object60283:head# 2024-04-29T14:21:18.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490f531f:::benchmark_data_smithi194_103039_object156511:head# 2024-04-29T14:21:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490f7538:::benchmark_data_smithi194_103039_object90905:head# 2024-04-29T14:21:18.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:490fd89d:::benchmark_data_smithi194_103039_object121452:head# 2024-04-29T14:21:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49103d66:::benchmark_data_smithi194_103039_object52410:head# 2024-04-29T14:21:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4911b1ef:::benchmark_data_smithi194_103039_object57142:head# 2024-04-29T14:21:18.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49129131:::benchmark_data_smithi194_103039_object52270:head# 2024-04-29T14:21:18.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49132575:::benchmark_data_smithi194_103039_object88555:head# 2024-04-29T14:21:18.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4913475d:::benchmark_data_smithi194_103039_object114887:head# 2024-04-29T14:21:18.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4913a18f:::benchmark_data_smithi194_103039_object117593:head# 2024-04-29T14:21:18.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49148879:::benchmark_data_smithi194_103039_object73869:head# 2024-04-29T14:21:18.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4914b545:::benchmark_data_smithi194_103039_object149669:head# 2024-04-29T14:21:18.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49158d7c:::benchmark_data_smithi194_103039_object77891:head# 2024-04-29T14:21:18.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4915d975:::benchmark_data_smithi194_103039_object103822:head# 2024-04-29T14:21:18.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4915ed47:::benchmark_data_smithi194_103039_object105045:head# 2024-04-29T14:21:18.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491604b1:::benchmark_data_smithi194_103039_object142771:head# 2024-04-29T14:21:18.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49165c95:::benchmark_data_smithi194_103039_object86556:head# 2024-04-29T14:21:18.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4916c51b:::benchmark_data_smithi194_103039_object104042:head# 2024-04-29T14:21:18.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49177107:::benchmark_data_smithi194_103039_object121493:head# 2024-04-29T14:21:18.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4917ef53:::benchmark_data_smithi194_103039_object50856:head# 2024-04-29T14:21:18.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491812a4:::benchmark_data_smithi194_103039_object77623:head# 2024-04-29T14:21:18.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4918db76:::benchmark_data_smithi194_103039_object64563:head# 2024-04-29T14:21:18.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4919b668:::benchmark_data_smithi194_103039_object46384:head# 2024-04-29T14:21:18.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4919e8fa:::benchmark_data_smithi194_103039_object95089:head# 2024-04-29T14:21:18.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491adbbc:::benchmark_data_smithi194_103039_object103332:head# 2024-04-29T14:21:18.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491b8c17:::benchmark_data_smithi194_103039_object48394:head# 2024-04-29T14:21:18.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491bc401:::benchmark_data_smithi194_103039_object105037:head# 2024-04-29T14:21:18.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491bf864:::benchmark_data_smithi194_103039_object118203:head# 2024-04-29T14:21:18.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491c1490:::benchmark_data_smithi194_103039_object138152:head# 2024-04-29T14:21:18.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491d9b7a:::benchmark_data_smithi194_103039_object145392:head# 2024-04-29T14:21:18.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491db3e0:::benchmark_data_smithi194_103039_object101950:head# 2024-04-29T14:21:18.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491f7302:::benchmark_data_smithi194_103039_object148041:head# 2024-04-29T14:21:18.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491fa9a6:::benchmark_data_smithi194_103039_object136689:head# 2024-04-29T14:21:18.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:491fd41c:::benchmark_data_smithi194_103039_object107488:head# 2024-04-29T14:21:18.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4920e936:::benchmark_data_smithi194_103039_object97706:head# 2024-04-29T14:21:18.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4920ee45:::benchmark_data_smithi194_103039_object144431:head# 2024-04-29T14:21:18.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49223433:::benchmark_data_smithi194_103039_object125485:head# 2024-04-29T14:21:18.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49225981:::benchmark_data_smithi194_103039_object54881:head# 2024-04-29T14:21:18.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49237abc:::benchmark_data_smithi194_103039_object45995:head# 2024-04-29T14:21:18.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49238fdd:::benchmark_data_smithi194_103039_object110405:head# 2024-04-29T14:21:18.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49239c1b:::benchmark_data_smithi194_103039_object136707:head# 2024-04-29T14:21:18.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4923a6af:::benchmark_data_smithi194_103039_object97863:head# 2024-04-29T14:21:18.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4923edf5:::benchmark_data_smithi194_103039_object114877:head# 2024-04-29T14:21:18.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492406e0:::benchmark_data_smithi194_103039_object95961:head# 2024-04-29T14:21:18.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49240feb:::benchmark_data_smithi194_103039_object72361:head# 2024-04-29T14:21:18.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49247d0c:::benchmark_data_smithi194_103039_object76683:head# 2024-04-29T14:21:18.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492500b2:::benchmark_data_smithi194_103039_object92587:head# 2024-04-29T14:21:18.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4925a60d:::benchmark_data_smithi194_103039_object122537:head# 2024-04-29T14:21:18.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4925c03a:::benchmark_data_smithi194_103039_object141312:head# 2024-04-29T14:21:18.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49277ba9:::benchmark_data_smithi194_103039_object94129:head# 2024-04-29T14:21:18.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4927d016:::benchmark_data_smithi194_103039_object111582:head# 2024-04-29T14:21:18.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49288b26:::benchmark_data_smithi194_103039_object65804:head# 2024-04-29T14:21:18.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492988c8:::benchmark_data_smithi194_103039_object154699:head# 2024-04-29T14:21:18.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492a28d6:::benchmark_data_smithi194_103039_object112943:head# 2024-04-29T14:21:18.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492a4f6e:::benchmark_data_smithi194_103039_object94845:head# 2024-04-29T14:21:18.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492b7b14:::benchmark_data_smithi194_103039_object113412:head# 2024-04-29T14:21:18.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492bf5cf:::benchmark_data_smithi194_103039_object101100:head# 2024-04-29T14:21:18.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492c854b:::benchmark_data_smithi194_103039_object114205:head# 2024-04-29T14:21:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492cf821:::benchmark_data_smithi194_103039_object126029:head# 2024-04-29T14:21:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492da282:::benchmark_data_smithi194_103039_object44246:head# 2024-04-29T14:21:18.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492dd6b1:::benchmark_data_smithi194_103039_object121007:head# 2024-04-29T14:21:18.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492e959f:::benchmark_data_smithi194_103039_object102514:head# 2024-04-29T14:21:18.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492f4b42:::benchmark_data_smithi194_103039_object56823:head# 2024-04-29T14:21:18.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492f626a:::benchmark_data_smithi194_103039_object89678:head# 2024-04-29T14:21:18.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492f6713:::benchmark_data_smithi194_103039_object80841:head# 2024-04-29T14:21:18.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:492fdf5c:::benchmark_data_smithi194_103039_object151559:head# 2024-04-29T14:21:18.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4930610c:::benchmark_data_smithi194_103039_object124633:head# 2024-04-29T14:21:18.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4930cd9f:::benchmark_data_smithi194_103039_object129749:head# 2024-04-29T14:21:18.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49311e5c:::benchmark_data_smithi194_103039_object142860:head# 2024-04-29T14:21:18.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4931525b:::benchmark_data_smithi194_103039_object131039:head# 2024-04-29T14:21:18.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4931571e:::benchmark_data_smithi194_103039_object88825:head# 2024-04-29T14:21:18.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4931e728:::benchmark_data_smithi194_103039_object105482:head# 2024-04-29T14:21:18.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49326ec8:::benchmark_data_smithi194_103039_object81201:head# 2024-04-29T14:21:18.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49335904:::benchmark_data_smithi194_103039_object64661:head# 2024-04-29T14:21:18.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493385be:::benchmark_data_smithi194_103039_object71556:head# 2024-04-29T14:21:18.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4933ae12:::benchmark_data_smithi194_103039_object57835:head# 2024-04-29T14:21:18.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4933afca:::benchmark_data_smithi194_103039_object43544:head# 2024-04-29T14:21:18.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49349983:::benchmark_data_smithi194_103039_object94505:head# 2024-04-29T14:21:18.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4934b390:::benchmark_data_smithi194_103039_object60134:head# 2024-04-29T14:21:18.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4934d07d:::benchmark_data_smithi194_103039_object112004:head# 2024-04-29T14:21:18.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49361873:::benchmark_data_smithi194_103039_object67341:head# 2024-04-29T14:21:18.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493877d1:::benchmark_data_smithi194_103039_object91888:head# 2024-04-29T14:21:18.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4938a0f6:::benchmark_data_smithi194_103039_object98067:head# 2024-04-29T14:21:18.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4938b597:::benchmark_data_smithi194_103039_object88841:head# 2024-04-29T14:21:18.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4938c2f3:::benchmark_data_smithi194_103039_object63619:head# 2024-04-29T14:21:18.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49394915:::benchmark_data_smithi194_103039_object124648:head# 2024-04-29T14:21:18.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49399be6:::benchmark_data_smithi194_103039_object53841:head# 2024-04-29T14:21:18.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4939b2a6:::benchmark_data_smithi194_103039_object90489:head# 2024-04-29T14:21:18.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4939f5bc:::benchmark_data_smithi194_103039_object155304:head# 2024-04-29T14:21:18.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493a879a:::benchmark_data_smithi194_103039_object61957:head# 2024-04-29T14:21:18.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493b7b17:::benchmark_data_smithi194_103039_object52969:head# 2024-04-29T14:21:18.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493b956b:::benchmark_data_smithi194_103039_object132108:head# 2024-04-29T14:21:18.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493c8e70:::benchmark_data_smithi194_103039_object152693:head# 2024-04-29T14:21:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:18.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493ca6c2:::benchmark_data_smithi194_103039_object45430:head# 2024-04-29T14:21:18.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493cc3db:::benchmark_data_smithi194_103039_object46630:head# 2024-04-29T14:21:18.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493d4ad4:::benchmark_data_smithi194_103039_object86901:head# 2024-04-29T14:21:18.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493e1066:::benchmark_data_smithi194_103039_object55599:head# 2024-04-29T14:21:18.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493e49c6:::benchmark_data_smithi194_103039_object139035:head# 2024-04-29T14:21:18.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493e7f9e:::benchmark_data_smithi194_103039_object138919:head# 2024-04-29T14:21:18.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493ed72c:::benchmark_data_smithi194_103039_object117449:head# 2024-04-29T14:21:18.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:493f343f:::benchmark_data_smithi194_103039_object49147:head# 2024-04-29T14:21:18.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494040fc:::benchmark_data_smithi194_103039_object63754:head# 2024-04-29T14:21:18.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49406d01:::benchmark_data_smithi194_103039_object100803:head# 2024-04-29T14:21:18.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49413802:::benchmark_data_smithi194_103039_object62858:head# 2024-04-29T14:21:18.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49417a9a:::benchmark_data_smithi194_103039_object138157:head# 2024-04-29T14:21:18.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494228c5:::benchmark_data_smithi194_103039_object46251:head# 2024-04-29T14:21:18.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49423067:::benchmark_data_smithi194_103039_object57710:head# 2024-04-29T14:21:18.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49426f80:::benchmark_data_smithi194_103039_object48216:head# 2024-04-29T14:21:18.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4943203d:::benchmark_data_smithi194_103039_object105537:head# 2024-04-29T14:21:18.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4943d4fc:::benchmark_data_smithi194_103039_object96216:head# 2024-04-29T14:21:18.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4944e91f:::benchmark_data_smithi194_103039_object125236:head# 2024-04-29T14:21:18.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494576c5:::benchmark_data_smithi194_103039_object90289:head# 2024-04-29T14:21:18.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49457ca9:::benchmark_data_smithi194_103039_object115072:head# 2024-04-29T14:21:18.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4945cfd7:::benchmark_data_smithi194_103039_object79407:head# 2024-04-29T14:21:18.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49461798:::benchmark_data_smithi194_103039_object89586:head# 2024-04-29T14:21:18.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49461fb3:::benchmark_data_smithi194_103039_object71347:head# 2024-04-29T14:21:18.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4947ee91:::benchmark_data_smithi194_103039_object135686:head# 2024-04-29T14:21:18.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49493d7f:::benchmark_data_smithi194_103039_object80596:head# 2024-04-29T14:21:18.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4949b94a:::benchmark_data_smithi194_103039_object42348:head# 2024-04-29T14:21:18.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494a6c57:::benchmark_data_smithi194_103039_object58463:head# 2024-04-29T14:21:18.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494a8daa:::benchmark_data_smithi194_103039_object118478:head# 2024-04-29T14:21:18.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494abb09:::benchmark_data_smithi194_103039_object57016:head# 2024-04-29T14:21:18.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494b033e:::benchmark_data_smithi194_103039_object129866:head# 2024-04-29T14:21:18.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494b4516:::benchmark_data_smithi194_103039_object120139:head# 2024-04-29T14:21:18.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494b9825:::benchmark_data_smithi194_103039_object99994:head# 2024-04-29T14:21:18.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494bb878:::benchmark_data_smithi194_103039_object147973:head# 2024-04-29T14:21:18.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494becc3:::benchmark_data_smithi194_103039_object70636:head# 2024-04-29T14:21:18.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494c0df2:::benchmark_data_smithi194_103039_object101880:head# 2024-04-29T14:21:18.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494c2745:::benchmark_data_smithi194_103039_object122790:head# 2024-04-29T14:21:18.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494cd4a9:::benchmark_data_smithi194_103039_object48583:head# 2024-04-29T14:21:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494e2aa7:::benchmark_data_smithi194_103039_object130726:head# 2024-04-29T14:21:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494ee19a:::benchmark_data_smithi194_103039_object96609:head# 2024-04-29T14:21:18.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:494fd6bf:::benchmark_data_smithi194_103039_object109986:head# 2024-04-29T14:21:18.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49500e4d:::benchmark_data_smithi194_103039_object55402:head# 2024-04-29T14:21:18.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495087fc:::benchmark_data_smithi194_103039_object148668:head# 2024-04-29T14:21:18.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49512a74:::benchmark_data_smithi194_103039_object64792:head# 2024-04-29T14:21:18.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49515d7d:::benchmark_data_smithi194_103039_object154061:head# 2024-04-29T14:21:18.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49521637:::benchmark_data_smithi194_103039_object77468:head# 2024-04-29T14:21:18.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4952546d:::benchmark_data_smithi194_103039_object153148:head# 2024-04-29T14:21:18.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49527164:::benchmark_data_smithi194_103039_object137739:head# 2024-04-29T14:21:18.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49535f43:::benchmark_data_smithi194_103039_object43250:head# 2024-04-29T14:21:18.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4953a963:::benchmark_data_smithi194_103039_object44153:head# 2024-04-29T14:21:18.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49545027:::benchmark_data_smithi194_103039_object90479:head# 2024-04-29T14:21:18.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4954569c:::benchmark_data_smithi194_103039_object107770:head# 2024-04-29T14:21:18.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4955772e:::benchmark_data_smithi194_103039_object77791:head# 2024-04-29T14:21:18.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495619d1:::benchmark_data_smithi194_103039_object104341:head# 2024-04-29T14:21:18.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49565ad7:::benchmark_data_smithi194_103039_object102397:head# 2024-04-29T14:21:18.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49569a5b:::benchmark_data_smithi194_103039_object118497:head# 2024-04-29T14:21:18.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4956f703:::benchmark_data_smithi194_103039_object90042:head# 2024-04-29T14:21:18.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49582f13:::benchmark_data_smithi194_103039_object48606:head# 2024-04-29T14:21:18.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495844a7:::benchmark_data_smithi194_103039_object93456:head# 2024-04-29T14:21:18.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4958a9c6:::benchmark_data_smithi194_103039_object112863:head# 2024-04-29T14:21:18.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4959d31f:::benchmark_data_smithi194_103039_object84427:head# 2024-04-29T14:21:18.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4959ee3b:::benchmark_data_smithi194_103039_object53039:head# 2024-04-29T14:21:18.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4959efe1:::benchmark_data_smithi194_103039_object154154:head# 2024-04-29T14:21:18.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495a2af3:::benchmark_data_smithi194_103039_object127962:head# 2024-04-29T14:21:18.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495aef34:::benchmark_data_smithi194_103039_object127056:head# 2024-04-29T14:21:18.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495b2ab4:::benchmark_data_smithi194_103039_object102660:head# 2024-04-29T14:21:18.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495bd35c:::benchmark_data_smithi194_103039_object70579:head# 2024-04-29T14:21:18.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495c96cf:::benchmark_data_smithi194_103039_object155005:head# 2024-04-29T14:21:18.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495cab21:::benchmark_data_smithi194_103039_object135409:head# 2024-04-29T14:21:18.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495d479d:::benchmark_data_smithi194_103039_object99336:head# 2024-04-29T14:21:18.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495db073:::benchmark_data_smithi194_103039_object104052:head# 2024-04-29T14:21:18.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495e4520:::benchmark_data_smithi194_103039_object77479:head# 2024-04-29T14:21:18.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495ed8a6:::benchmark_data_smithi194_103039_object43341:head# 2024-04-29T14:21:18.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495f56a6:::benchmark_data_smithi194_103039_object57439:head# 2024-04-29T14:21:18.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:495fe4a0:::benchmark_data_smithi194_103039_object108690:head# 2024-04-29T14:21:18.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4960295a:::benchmark_data_smithi194_103039_object81431:head# 2024-04-29T14:21:18.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4961c5e1:::benchmark_data_smithi194_103039_object88712:head# 2024-04-29T14:21:18.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4961d70d:::benchmark_data_smithi194_103039_object61786:head# 2024-04-29T14:21:18.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4961dca9:::benchmark_data_smithi194_103039_object91579:head# 2024-04-29T14:21:18.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4961eb46:::benchmark_data_smithi194_103039_object137746:head# 2024-04-29T14:21:18.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49627eea:::benchmark_data_smithi194_103039_object130635:head# 2024-04-29T14:21:18.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4962984e:::benchmark_data_smithi194_103039_object42194:head# 2024-04-29T14:21:18.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49636879:::benchmark_data_smithi194_103039_object76000:head# 2024-04-29T14:21:18.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496389bc:::benchmark_data_smithi194_103039_object145447:head# 2024-04-29T14:21:18.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496451fe:::benchmark_data_smithi194_103039_object51937:head# 2024-04-29T14:21:18.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4964f313:::benchmark_data_smithi194_103039_object84090:head# 2024-04-29T14:21:18.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496537bf:::benchmark_data_smithi194_103039_object95187:head# 2024-04-29T14:21:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496654e9:::benchmark_data_smithi194_103039_object48270:head# 2024-04-29T14:21:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49666016:::benchmark_data_smithi194_103039_object145772:head# 2024-04-29T14:21:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49674543:::benchmark_data_smithi194_103039_object103133:head# 2024-04-29T14:21:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4967499a:::benchmark_data_smithi194_103039_object79117:head# 2024-04-29T14:21:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49677afd:::benchmark_data_smithi194_103039_object47623:head# 2024-04-29T14:21:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49678aa3:::benchmark_data_smithi194_103039_object51951:head# 2024-04-29T14:21:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49693f61:::benchmark_data_smithi194_103039_object71752:head# 2024-04-29T14:21:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4969f959:::benchmark_data_smithi194_103039_object130751:head# 2024-04-29T14:21:18.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496a1b70:::benchmark_data_smithi194_103039_object80534:head# 2024-04-29T14:21:18.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496a8496:::benchmark_data_smithi194_103039_object132819:head# 2024-04-29T14:21:18.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496adc74:::benchmark_data_smithi194_103039_object72373:head# 2024-04-29T14:21:18.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496c0457:::benchmark_data_smithi194_103039_object132932:head# 2024-04-29T14:21:18.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496c1639:::benchmark_data_smithi194_103039_object60711:head# 2024-04-29T14:21:18.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496c1c75:::benchmark_data_smithi194_103039_object78310:head# 2024-04-29T14:21:18.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496c4c77:::benchmark_data_smithi194_103039_object85852:head# 2024-04-29T14:21:18.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496cb60e:::benchmark_data_smithi194_103039_object66221:head# 2024-04-29T14:21:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496d8f83:::benchmark_data_smithi194_103039_object122963:head# 2024-04-29T14:21:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496dc34f:::benchmark_data_smithi194_103039_object77202:head# 2024-04-29T14:21:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496f299b:::benchmark_data_smithi194_103039_object151808:head# 2024-04-29T14:21:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:496f81fc:::benchmark_data_smithi194_103039_object62931:head# 2024-04-29T14:21:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497093b8:::benchmark_data_smithi194_103039_object112362:head# 2024-04-29T14:21:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4971425d:::benchmark_data_smithi194_103039_object48695:head# 2024-04-29T14:21:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497233fa:::benchmark_data_smithi194_103039_object74213:head# 2024-04-29T14:21:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4972a72f:::benchmark_data_smithi194_103039_object121102:head# 2024-04-29T14:21:18.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497341f0:::benchmark_data_smithi194_103039_object50882:head# 2024-04-29T14:21:18.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49736623:::benchmark_data_smithi194_103039_object47206:head# 2024-04-29T14:21:18.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4974448f:::benchmark_data_smithi194_103039_object57588:head# 2024-04-29T14:21:18.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49753148:::benchmark_data_smithi194_103039_object59763:head# 2024-04-29T14:21:18.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49757468:::benchmark_data_smithi194_103039_object60294:head# 2024-04-29T14:21:18.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497649a4:::benchmark_data_smithi194_103039_object117802:head# 2024-04-29T14:21:18.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49765a6c:::benchmark_data_smithi194_103039_object113943:head# 2024-04-29T14:21:18.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4976bfc5:::benchmark_data_smithi194_103039_object113531:head# 2024-04-29T14:21:18.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4977a177:::benchmark_data_smithi194_103039_object78886:head# 2024-04-29T14:21:18.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49780d76:::benchmark_data_smithi194_103039_object61401:head# 2024-04-29T14:21:18.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497980da:::benchmark_data_smithi194_103039_object83070:head# 2024-04-29T14:21:18.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497ad2de:::benchmark_data_smithi194_103039_object55138:head# 2024-04-29T14:21:18.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497c59db:::benchmark_data_smithi194_103039_object131227:head# 2024-04-29T14:21:18.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497d4dc3:::benchmark_data_smithi194_103039_object140366:head# 2024-04-29T14:21:18.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497e4f75:::benchmark_data_smithi194_103039_object71072:head# 2024-04-29T14:21:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:497e8988:::benchmark_data_smithi194_103039_object130498:head# 2024-04-29T14:21:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49804b1b:::benchmark_data_smithi194_103039_object148950:head# 2024-04-29T14:21:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49805a61:::benchmark_data_smithi194_103039_object78531:head# 2024-04-29T14:21:18.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49807e49:::benchmark_data_smithi194_103039_object131643:head# 2024-04-29T14:21:18.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4980e5d7:::benchmark_data_smithi194_103039_object116805:head# 2024-04-29T14:21:18.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4981a527:::benchmark_data_smithi194_103039_object76658:head# 2024-04-29T14:21:18.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4981f19e:::benchmark_data_smithi194_103039_object105000:head# 2024-04-29T14:21:18.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49830add:::benchmark_data_smithi194_103039_object44234:head# 2024-04-29T14:21:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49852e10:::benchmark_data_smithi194_103039_object110541:head# 2024-04-29T14:21:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498803d7:::benchmark_data_smithi194_103039_object102501:head# 2024-04-29T14:21:18.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49886bb5:::benchmark_data_smithi194_103039_object72167:head# 2024-04-29T14:21:18.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49891c00:::benchmark_data_smithi194_103039_object122700:head# 2024-04-29T14:21:18.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498a6428:::benchmark_data_smithi194_103039_object110314:head# 2024-04-29T14:21:18.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498afead:::benchmark_data_smithi194_103039_object130117:head# 2024-04-29T14:21:18.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498b3c94:::benchmark_data_smithi194_103039_object70992:head# 2024-04-29T14:21:18.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498b4347:::benchmark_data_smithi194_103039_object84536:head# 2024-04-29T14:21:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498bef43:::benchmark_data_smithi194_103039_object55007:head# 2024-04-29T14:21:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498c65b3:::benchmark_data_smithi194_103039_object94605:head# 2024-04-29T14:21:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498c80bd:::benchmark_data_smithi194_103039_object66886:head# 2024-04-29T14:21:18.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498c90f1:::benchmark_data_smithi194_103039_object141811:head# 2024-04-29T14:21:18.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498d2d4b:::benchmark_data_smithi194_103039_object67964:head# 2024-04-29T14:21:18.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498e5f7d:::benchmark_data_smithi194_103039_object53661:head# 2024-04-29T14:21:18.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498eea82:::benchmark_data_smithi194_103039_object126747:head# 2024-04-29T14:21:18.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498f02ef:::benchmark_data_smithi194_103039_object96033:head# 2024-04-29T14:21:18.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498f1d9a:::benchmark_data_smithi194_103039_object140473:head# 2024-04-29T14:21:18.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498f7c13:::benchmark_data_smithi194_103039_object82581:head# 2024-04-29T14:21:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498fd58c:::benchmark_data_smithi194_103039_object50437:head# 2024-04-29T14:21:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:498ff1f5:::benchmark_data_smithi194_103039_object93894:head# 2024-04-29T14:21:18.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49901eb5:::benchmark_data_smithi194_103039_object118178:head# 2024-04-29T14:21:18.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499073a8:::benchmark_data_smithi194_103039_object75823:head# 2024-04-29T14:21:18.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4990efc6:::benchmark_data_smithi194_103039_object137741:head# 2024-04-29T14:21:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49918e9c:::benchmark_data_smithi194_103039_object102859:head# 2024-04-29T14:21:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49923630:::benchmark_data_smithi194_103039_object61545:head# 2024-04-29T14:21:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4992973e:::benchmark_data_smithi194_103039_object127722:head# 2024-04-29T14:21:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4992e56e:::benchmark_data_smithi194_103039_object97606:head# 2024-04-29T14:21:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49930cce:::benchmark_data_smithi194_103039_object96322:head# 2024-04-29T14:21:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49934b1b:::benchmark_data_smithi194_103039_object62074:head# 2024-04-29T14:21:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4993957f:::benchmark_data_smithi194_103039_object115458:head# 2024-04-29T14:21:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4993ffdb:::benchmark_data_smithi194_103039_object99612:head# 2024-04-29T14:21:18.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499493d2:::benchmark_data_smithi194_103039_object98200:head# 2024-04-29T14:21:18.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4995567d:::benchmark_data_smithi194_103039_object102607:head# 2024-04-29T14:21:18.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49970db4:::benchmark_data_smithi194_103039_object61025:head# 2024-04-29T14:21:18.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4997477e:::benchmark_data_smithi194_103039_object50759:head# 2024-04-29T14:21:18.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49975122:::benchmark_data_smithi194_103039_object61250:head# 2024-04-29T14:21:18.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49986001:::benchmark_data_smithi194_103039_object59351:head# 2024-04-29T14:21:18.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49991ec2:::benchmark_data_smithi194_103039_object118524:head# 2024-04-29T14:21:18.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499966aa:::benchmark_data_smithi194_103039_object104101:head# 2024-04-29T14:21:18.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49997141:::benchmark_data_smithi194_103039_object43924:head# 2024-04-29T14:21:18.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4999f75b:::benchmark_data_smithi194_103039_object101180:head# 2024-04-29T14:21:18.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499a427d:::benchmark_data_smithi194_103039_object86355:head# 2024-04-29T14:21:18.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499ab8b1:::benchmark_data_smithi194_103039_object94092:head# 2024-04-29T14:21:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499afb5b:::benchmark_data_smithi194_103039_object119986:head# 2024-04-29T14:21:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499b17af:::benchmark_data_smithi194_103039_object98356:head# 2024-04-29T14:21:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499b4b49:::benchmark_data_smithi194_103039_object150940:head# 2024-04-29T14:21:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499b78f6:::benchmark_data_smithi194_103039_object83301:head# 2024-04-29T14:21:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499bb318:::benchmark_data_smithi194_103039_object146436:head# 2024-04-29T14:21:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499bcc4b:::benchmark_data_smithi194_103039_object89809:head# 2024-04-29T14:21:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499ca89a:::benchmark_data_smithi194_103039_object138381:head# 2024-04-29T14:21:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499dd5b6:::benchmark_data_smithi194_103039_object74191:head# 2024-04-29T14:21:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499df0be:::benchmark_data_smithi194_103039_object81807:head# 2024-04-29T14:21:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499e7b6a:::benchmark_data_smithi194_103039_object98642:head# 2024-04-29T14:21:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499f4748:::benchmark_data_smithi194_103039_object112102:head# 2024-04-29T14:21:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:499f55ac:::benchmark_data_smithi194_103039_object110225:head# 2024-04-29T14:21:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a077da:::benchmark_data_smithi194_103039_object113044:head# 2024-04-29T14:21:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a201ce:::benchmark_data_smithi194_103039_object139936:head# 2024-04-29T14:21:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a282b6:::benchmark_data_smithi194_103039_object50559:head# 2024-04-29T14:21:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a2907d:::benchmark_data_smithi194_103039_object62801:head# 2024-04-29T14:21:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a31690:::benchmark_data_smithi194_103039_object143486:head# 2024-04-29T14:21:18.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a39949:::benchmark_data_smithi194_103039_object46349:head# 2024-04-29T14:21:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a481d0:::benchmark_data_smithi194_103039_object58648:head# 2024-04-29T14:21:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a4b5ba:::benchmark_data_smithi194_103039_object102258:head# 2024-04-29T14:21:18.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a78760:::benchmark_data_smithi194_103039_object42675:head# 2024-04-29T14:21:18.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a7da78:::benchmark_data_smithi194_103039_object71623:head# 2024-04-29T14:21:18.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a7e02e:::benchmark_data_smithi194_103039_object54629:head# 2024-04-29T14:21:18.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a8536d:::benchmark_data_smithi194_103039_object89552:head# 2024-04-29T14:21:18.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49a885c8:::benchmark_data_smithi194_103039_object94835:head# 2024-04-29T14:21:18.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49aa9d2c:::benchmark_data_smithi194_103039_object92517:head# 2024-04-29T14:21:18.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ab7992:::benchmark_data_smithi194_103039_object50683:head# 2024-04-29T14:21:18.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ab9480:::benchmark_data_smithi194_103039_object88468:head# 2024-04-29T14:21:18.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49aba46f:::benchmark_data_smithi194_103039_object103345:head# 2024-04-29T14:21:18.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ac997f:::benchmark_data_smithi194_103039_object103380:head# 2024-04-29T14:21:18.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b00443:::benchmark_data_smithi194_103039_object113237:head# 2024-04-29T14:21:18.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b174bd:::benchmark_data_smithi194_103039_object65983:head# 2024-04-29T14:21:18.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b23128:::benchmark_data_smithi194_103039_object147572:head# 2024-04-29T14:21:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b23270:::benchmark_data_smithi194_103039_object89523:head# 2024-04-29T14:21:18.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b30f86:::benchmark_data_smithi194_103039_object59965:head# 2024-04-29T14:21:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b311d2:::benchmark_data_smithi194_103039_object140728:head# 2024-04-29T14:21:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b3bb16:::benchmark_data_smithi194_103039_object68165:head# 2024-04-29T14:21:18.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b47484:::benchmark_data_smithi194_103039_object50760:head# 2024-04-29T14:21:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b49c96:::benchmark_data_smithi194_103039_object107791:head# 2024-04-29T14:21:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b53751:::benchmark_data_smithi194_103039_object49271:head# 2024-04-29T14:21:18.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b5a1ee:::benchmark_data_smithi194_103039_object139287:head# 2024-04-29T14:21:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b60ea8:::benchmark_data_smithi194_103039_object105442:head# 2024-04-29T14:21:18.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b61745:::benchmark_data_smithi194_103039_object145569:head# 2024-04-29T14:21:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b6688a:::benchmark_data_smithi194_103039_object65961:head# 2024-04-29T14:21:18.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b697cc:::benchmark_data_smithi194_103039_object153643:head# 2024-04-29T14:21:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b6fa87:::benchmark_data_smithi194_103039_object110342:head# 2024-04-29T14:21:18.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b72a16:::benchmark_data_smithi194_103039_object70149:head# 2024-04-29T14:21:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49b96aed:::benchmark_data_smithi194_103039_object60151:head# 2024-04-29T14:21:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49bb6fbc:::benchmark_data_smithi194_103039_object154774:head# 2024-04-29T14:21:18.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49bb7ddc:::benchmark_data_smithi194_103039_object51362:head# 2024-04-29T14:21:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49bc0248:::benchmark_data_smithi194_103039_object101264:head# 2024-04-29T14:21:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49bd44f5:::benchmark_data_smithi194_103039_object81722:head# 2024-04-29T14:21:18.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49be1b3a:::benchmark_data_smithi194_103039_object151602:head# 2024-04-29T14:21:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49bf359a:::benchmark_data_smithi194_103039_object130823:head# 2024-04-29T14:21:18.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c06f75:::benchmark_data_smithi194_103039_object53529:head# 2024-04-29T14:21:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c0c09a:::benchmark_data_smithi194_103039_object71783:head# 2024-04-29T14:21:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c138e7:::benchmark_data_smithi194_103039_object81568:head# 2024-04-29T14:21:18.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c1adc3:::benchmark_data_smithi194_103039_object120882:head# 2024-04-29T14:21:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c240a8:::benchmark_data_smithi194_103039_object133260:head# 2024-04-29T14:21:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c31359:::benchmark_data_smithi194_103039_object80766:head# 2024-04-29T14:21:18.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c3b956:::benchmark_data_smithi194_103039_object69620:head# 2024-04-29T14:21:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c3eaf7:::benchmark_data_smithi194_103039_object56959:head# 2024-04-29T14:21:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c416e5:::benchmark_data_smithi194_103039_object73260:head# 2024-04-29T14:21:18.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c4c753:::benchmark_data_smithi194_103039_object135468:head# 2024-04-29T14:21:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c52f8e:::benchmark_data_smithi194_103039_object86255:head# 2024-04-29T14:21:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c533ed:::benchmark_data_smithi194_103039_object148536:head# 2024-04-29T14:21:18.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c5c479:::benchmark_data_smithi194_103039_object68771:head# 2024-04-29T14:21:18.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c61bef:::benchmark_data_smithi194_103039_object78793:head# 2024-04-29T14:21:18.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c61cdd:::benchmark_data_smithi194_103039_object62698:head# 2024-04-29T14:21:18.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c63dfd:::benchmark_data_smithi194_103039_object131378:head# 2024-04-29T14:21:18.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c6857b:::benchmark_data_smithi194_103039_object73418:head# 2024-04-29T14:21:18.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c72e89:::benchmark_data_smithi194_103039_object52316:head# 2024-04-29T14:21:18.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c8bd33:::benchmark_data_smithi194_103039_object76288:head# 2024-04-29T14:21:18.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c902e5:::benchmark_data_smithi194_103039_object135845:head# 2024-04-29T14:21:18.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49c9aa6a:::benchmark_data_smithi194_103039_object41816:head# 2024-04-29T14:21:18.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49cd3bb4:::benchmark_data_smithi194_103039_object102212:head# 2024-04-29T14:21:18.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49cd7a15:::benchmark_data_smithi194_103039_object89866:head# 2024-04-29T14:21:18.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49cedaea:::benchmark_data_smithi194_103039_object84179:head# 2024-04-29T14:21:18.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49cf21ec:::benchmark_data_smithi194_103039_object70203:head# 2024-04-29T14:21:18.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49cf3d8b:::benchmark_data_smithi194_103039_object124798:head# 2024-04-29T14:21:18.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49cf5c31:::benchmark_data_smithi194_103039_object99766:head# 2024-04-29T14:21:18.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49cfd053:::benchmark_data_smithi194_103039_object56416:head# 2024-04-29T14:21:18.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d190e4:::benchmark_data_smithi194_103039_object98337:head# 2024-04-29T14:21:18.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d222e5:::benchmark_data_smithi194_103039_object43913:head# 2024-04-29T14:21:18.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d2327f:::benchmark_data_smithi194_103039_object80553:head# 2024-04-29T14:21:18.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d2ca18:::benchmark_data_smithi194_103039_object84471:head# 2024-04-29T14:21:18.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d2df09:::benchmark_data_smithi194_103039_object62793:head# 2024-04-29T14:21:18.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d306a5:::benchmark_data_smithi194_103039_object115848:head# 2024-04-29T14:21:18.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d4658a:::benchmark_data_smithi194_103039_object97968:head# 2024-04-29T14:21:18.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d4750b:::benchmark_data_smithi194_103039_object96284:head# 2024-04-29T14:21:18.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d4ea99:::benchmark_data_smithi194_103039_object155555:head# 2024-04-29T14:21:18.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d4f104:::benchmark_data_smithi194_103039_object123166:head# 2024-04-29T14:21:18.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d5a95c:::benchmark_data_smithi194_103039_object117329:head# 2024-04-29T14:21:18.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d64bc2:::benchmark_data_smithi194_103039_object142113:head# 2024-04-29T14:21:18.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d65171:::benchmark_data_smithi194_103039_object77748:head# 2024-04-29T14:21:18.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d677e2:::benchmark_data_smithi194_103039_object128185:head# 2024-04-29T14:21:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d72ee5:::benchmark_data_smithi194_103039_object94186:head# 2024-04-29T14:21:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d8614e:::benchmark_data_smithi194_103039_object78269:head# 2024-04-29T14:21:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d8d3b4:::benchmark_data_smithi194_103039_object119180:head# 2024-04-29T14:21:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49d95259:::benchmark_data_smithi194_103039_object46988:head# 2024-04-29T14:21:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49da1cf9:::benchmark_data_smithi194_103039_object64684:head# 2024-04-29T14:21:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49db540e:::benchmark_data_smithi194_103039_object131511:head# 2024-04-29T14:21:18.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49dba5fa:::benchmark_data_smithi194_103039_object124517:head# 2024-04-29T14:21:18.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49dcd37e:::benchmark_data_smithi194_103039_object115067:head# 2024-04-29T14:21:18.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49dda284:::benchmark_data_smithi194_103039_object83315:head# 2024-04-29T14:21:18.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ddbc06:::benchmark_data_smithi194_103039_object74915:head# 2024-04-29T14:21:18.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49df5c2e:::benchmark_data_smithi194_103039_object138920:head# 2024-04-29T14:21:18.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49dfeeab:::benchmark_data_smithi194_103039_object150839:head# 2024-04-29T14:21:18.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49dffa56:::benchmark_data_smithi194_103039_object129503:head# 2024-04-29T14:21:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e10dd7:::benchmark_data_smithi194_103039_object85326:head# 2024-04-29T14:21:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e11a87:::benchmark_data_smithi194_103039_object130350:head# 2024-04-29T14:21:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e14d42:::benchmark_data_smithi194_103039_object57355:head# 2024-04-29T14:21:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e312a3:::benchmark_data_smithi194_103039_object52506:head# 2024-04-29T14:21:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e32254:::benchmark_data_smithi194_103039_object89081:head# 2024-04-29T14:21:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e36879:::benchmark_data_smithi194_103039_object59774:head# 2024-04-29T14:21:18.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e5337b:::benchmark_data_smithi194_103039_object154403:head# 2024-04-29T14:21:18.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e5447a:::benchmark_data_smithi194_103039_object92316:head# 2024-04-29T14:21:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e55783:::benchmark_data_smithi194_103039_object82796:head# 2024-04-29T14:21:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e5e401:::benchmark_data_smithi194_103039_object57735:head# 2024-04-29T14:21:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e65267:::benchmark_data_smithi194_103039_object118558:head# 2024-04-29T14:21:18.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e69bb9:::benchmark_data_smithi194_103039_object48373:head# 2024-04-29T14:21:18.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e7746d:::benchmark_data_smithi194_103039_object122990:head# 2024-04-29T14:21:18.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e83408:::benchmark_data_smithi194_103039_object67720:head# 2024-04-29T14:21:18.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49e9e803:::benchmark_data_smithi194_103039_object152280:head# 2024-04-29T14:21:18.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49eb00a6:::benchmark_data_smithi194_103039_object68488:head# 2024-04-29T14:21:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49eb1287:::benchmark_data_smithi194_103039_object145896:head# 2024-04-29T14:21:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ecf553:::benchmark_data_smithi194_103039_object81676:head# 2024-04-29T14:21:18.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ed0932:::benchmark_data_smithi194_103039_object155501:head# 2024-04-29T14:21:18.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ed0be4:::benchmark_data_smithi194_103039_object75036:head# 2024-04-29T14:21:18.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49eda1d0:::benchmark_data_smithi194_103039_object50314:head# 2024-04-29T14:21:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ef4466:::benchmark_data_smithi194_103039_object43815:head# 2024-04-29T14:21:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f04a05:::benchmark_data_smithi194_103039_object90898:head# 2024-04-29T14:21:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f13756:::benchmark_data_smithi194_103039_object116682:head# 2024-04-29T14:21:18.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f19810:::benchmark_data_smithi194_103039_object41834:head# 2024-04-29T14:21:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f1c2ce:::benchmark_data_smithi194_103039_object81423:head# 2024-04-29T14:21:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f2d669:::benchmark_data_smithi194_103039_object45121:head# 2024-04-29T14:21:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f32435:::benchmark_data_smithi194_103039_object62318:head# 2024-04-29T14:21:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f35344:::benchmark_data_smithi194_103039_object47523:head# 2024-04-29T14:21:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f3b817:::benchmark_data_smithi194_103039_object54815:head# 2024-04-29T14:21:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f4d60c:::benchmark_data_smithi194_103039_object66951:head# 2024-04-29T14:21:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f53217:::benchmark_data_smithi194_103039_object95730:head# 2024-04-29T14:21:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f675e3:::benchmark_data_smithi194_103039_object133620:head# 2024-04-29T14:21:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f6777d:::benchmark_data_smithi194_103039_object61218:head# 2024-04-29T14:21:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f7d1a4:::benchmark_data_smithi194_103039_object42408:head# 2024-04-29T14:21:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f85e2e:::benchmark_data_smithi194_103039_object91229:head# 2024-04-29T14:21:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f868dc:::benchmark_data_smithi194_103039_object80888:head# 2024-04-29T14:21:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f8c1c1:::benchmark_data_smithi194_103039_object132149:head# 2024-04-29T14:21:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f8f137:::benchmark_data_smithi194_103039_object98198:head# 2024-04-29T14:21:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49f9cfb1:::benchmark_data_smithi194_103039_object127697:head# 2024-04-29T14:21:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fa16fa:::benchmark_data_smithi194_103039_object58074:head# 2024-04-29T14:21:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fa8820:::benchmark_data_smithi194_103039_object65535:head# 2024-04-29T14:21:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fae67b:::benchmark_data_smithi194_103039_object119194:head# 2024-04-29T14:21:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fb15f1:::benchmark_data_smithi194_103039_object78340:head# 2024-04-29T14:21:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fb6012:::benchmark_data_smithi194_103039_object148034:head# 2024-04-29T14:21:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fbed3b:::benchmark_data_smithi194_103039_object70273:head# 2024-04-29T14:21:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fc8f6d:::benchmark_data_smithi194_103039_object146435:head# 2024-04-29T14:21:18.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fde8e8:::benchmark_data_smithi194_103039_object43604:head# 2024-04-29T14:21:18.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fdecb6:::benchmark_data_smithi194_103039_object110729:head# 2024-04-29T14:21:18.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fe0163:::benchmark_data_smithi194_103039_object154527:head# 2024-04-29T14:21:18.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fe7369:::benchmark_data_smithi194_103039_object63359:head# 2024-04-29T14:21:18.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fe9071:::benchmark_data_smithi194_103039_object104303:head# 2024-04-29T14:21:18.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fea8eb:::benchmark_data_smithi194_103039_object119342:head# 2024-04-29T14:21:18.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49fec1a6:::benchmark_data_smithi194_103039_object44864:head# 2024-04-29T14:21:18.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:49ff92fa:::benchmark_data_smithi194_103039_object86415:head# 2024-04-29T14:21:18.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a008ff4:::benchmark_data_smithi194_103039_object71743:head# 2024-04-29T14:21:18.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a031f1f:::benchmark_data_smithi194_103039_object98483:head# 2024-04-29T14:21:18.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a033431:::benchmark_data_smithi194_103039_object126115:head# 2024-04-29T14:21:18.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a034a40:::benchmark_data_smithi194_103039_object119236:head# 2024-04-29T14:21:18.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a03d053:::benchmark_data_smithi194_103039_object83652:head# 2024-04-29T14:21:18.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a043d5e:::benchmark_data_smithi194_103039_object65331:head# 2024-04-29T14:21:18.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a04b54e:::benchmark_data_smithi194_103039_object112182:head# 2024-04-29T14:21:18.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a04e6f0:::benchmark_data_smithi194_103039_object91950:head# 2024-04-29T14:21:18.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a04f0d9:::benchmark_data_smithi194_103039_object46858:head# 2024-04-29T14:21:18.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0536d8:::benchmark_data_smithi194_103039_object122599:head# 2024-04-29T14:21:18.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a05ab3d:::benchmark_data_smithi194_103039_object53568:head# 2024-04-29T14:21:18.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a060330:::benchmark_data_smithi194_103039_object153601:head# 2024-04-29T14:21:18.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a069221:::benchmark_data_smithi194_103039_object119014:head# 2024-04-29T14:21:18.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a06d5eb:::benchmark_data_smithi194_103039_object74270:head# 2024-04-29T14:21:18.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a06f84f:::benchmark_data_smithi194_103039_object137235:head# 2024-04-29T14:21:18.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a076b6f:::benchmark_data_smithi194_103039_object81499:head# 2024-04-29T14:21:18.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a077ed8:::benchmark_data_smithi194_103039_object129905:head# 2024-04-29T14:21:18.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a07c707:::benchmark_data_smithi194_103039_object94351:head# 2024-04-29T14:21:18.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a08a9b4:::benchmark_data_smithi194_103039_object101651:head# 2024-04-29T14:21:18.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a097894:::benchmark_data_smithi194_103039_object110605:head# 2024-04-29T14:21:18.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0a828c:::benchmark_data_smithi194_103039_object68450:head# 2024-04-29T14:21:18.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0a8e30:::benchmark_data_smithi194_103039_object116057:head# 2024-04-29T14:21:18.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0b7427:::benchmark_data_smithi194_103039_object82415:head# 2024-04-29T14:21:18.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0c696a:::benchmark_data_smithi194_103039_object133012:head# 2024-04-29T14:21:18.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0d5f73:::benchmark_data_smithi194_103039_object43182:head# 2024-04-29T14:21:18.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0e2fb2:::benchmark_data_smithi194_103039_object51239:head# 2024-04-29T14:21:18.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a0fe4ba:::benchmark_data_smithi194_103039_object84083:head# 2024-04-29T14:21:18.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a117984:::benchmark_data_smithi194_103039_object43879:head# 2024-04-29T14:21:18.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a11d7ee:::benchmark_data_smithi194_103039_object151034:head# 2024-04-29T14:21:18.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a11fc56:::benchmark_data_smithi194_103039_object128660:head# 2024-04-29T14:21:18.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a125975:::benchmark_data_smithi194_103039_object57645:head# 2024-04-29T14:21:18.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1365a7:::benchmark_data_smithi194_103039_object145097:head# 2024-04-29T14:21:18.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1384d9:::benchmark_data_smithi194_103039_object114586:head# 2024-04-29T14:21:18.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a13de29:::benchmark_data_smithi194_103039_object45501:head# 2024-04-29T14:21:18.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a15dd01:::benchmark_data_smithi194_103039_object86321:head# 2024-04-29T14:21:18.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a16c61f:::benchmark_data_smithi194_103039_object117994:head# 2024-04-29T14:21:18.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a17e310:::benchmark_data_smithi194_103039_object77240:head# 2024-04-29T14:21:18.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a17e3db:::benchmark_data_smithi194_103039_object72619:head# 2024-04-29T14:21:18.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a18ecca:::benchmark_data_smithi194_103039_object154600:head# 2024-04-29T14:21:18.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a18fb54:::benchmark_data_smithi194_103039_object126749:head# 2024-04-29T14:21:18.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1977b9:::benchmark_data_smithi194_103039_object54703:head# 2024-04-29T14:21:18.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1a4e3e:::benchmark_data_smithi194_103039_object77172:head# 2024-04-29T14:21:18.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1b3652:::benchmark_data_smithi194_103039_object68428:head# 2024-04-29T14:21:18.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1c2397:::benchmark_data_smithi194_103039_object45903:head# 2024-04-29T14:21:18.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1c2c50:::benchmark_data_smithi194_103039_object144726:head# 2024-04-29T14:21:18.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1c4bf4:::benchmark_data_smithi194_103039_object133257:head# 2024-04-29T14:21:18.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1c72d3:::benchmark_data_smithi194_103039_object135596:head# 2024-04-29T14:21:18.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1d25af:::benchmark_data_smithi194_103039_object125871:head# 2024-04-29T14:21:18.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1d53a1:::benchmark_data_smithi194_103039_object117875:head# 2024-04-29T14:21:18.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1d878b:::benchmark_data_smithi194_103039_object147079:head# 2024-04-29T14:21:18.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1da345:::benchmark_data_smithi194_103039_object137975:head# 2024-04-29T14:21:18.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1dd5b2:::benchmark_data_smithi194_103039_object137534:head# 2024-04-29T14:21:18.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1e033a:::benchmark_data_smithi194_103039_object120371:head# 2024-04-29T14:21:18.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1e7317:::benchmark_data_smithi194_103039_object142903:head# 2024-04-29T14:21:18.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1e8ab3:::benchmark_data_smithi194_103039_object135678:head# 2024-04-29T14:21:18.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1ed41e:::benchmark_data_smithi194_103039_object105460:head# 2024-04-29T14:21:18.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a1ef57a:::benchmark_data_smithi194_103039_object70323:head# 2024-04-29T14:21:18.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a20f392:::benchmark_data_smithi194_103039_object55161:head# 2024-04-29T14:21:18.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a21186b:::benchmark_data_smithi194_103039_object106669:head# 2024-04-29T14:21:18.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a215f08:::benchmark_data_smithi194_103039_object73613:head# 2024-04-29T14:21:18.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2241d4:::benchmark_data_smithi194_103039_object77126:head# 2024-04-29T14:21:18.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a230224:::benchmark_data_smithi194_103039_object119802:head# 2024-04-29T14:21:18.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a230ce5:::benchmark_data_smithi194_103039_object139011:head# 2024-04-29T14:21:18.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a23170f:::benchmark_data_smithi194_103039_object91049:head# 2024-04-29T14:21:18.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a234235:::benchmark_data_smithi194_103039_object143678:head# 2024-04-29T14:21:18.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a257ad4:::benchmark_data_smithi194_103039_object76686:head# 2024-04-29T14:21:18.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2610fe:::benchmark_data_smithi194_103039_object129065:head# 2024-04-29T14:21:18.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2670ff:::benchmark_data_smithi194_103039_object156656:head# 2024-04-29T14:21:18.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2692b5:::benchmark_data_smithi194_103039_object109204:head# 2024-04-29T14:21:18.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a272a7d:::benchmark_data_smithi194_103039_object55209:head# 2024-04-29T14:21:18.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a275da4:::benchmark_data_smithi194_103039_object111976:head# 2024-04-29T14:21:18.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a275e26:::benchmark_data_smithi194_103039_object57331:head# 2024-04-29T14:21:18.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a28a55a:::benchmark_data_smithi194_103039_object77079:head# 2024-04-29T14:21:18.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a295a51:::benchmark_data_smithi194_103039_object70338:head# 2024-04-29T14:21:18.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a297c20:::benchmark_data_smithi194_103039_object144721:head# 2024-04-29T14:21:18.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a298bb4:::benchmark_data_smithi194_103039_object52871:head# 2024-04-29T14:21:18.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2ac094:::benchmark_data_smithi194_103039_object136114:head# 2024-04-29T14:21:18.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2b261d:::benchmark_data_smithi194_103039_object154439:head# 2024-04-29T14:21:18.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2bbba2:::benchmark_data_smithi194_103039_object131952:head# 2024-04-29T14:21:18.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2c4f3b:::benchmark_data_smithi194_103039_object99047:head# 2024-04-29T14:21:18.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2c5f36:::benchmark_data_smithi194_103039_object108882:head# 2024-04-29T14:21:18.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2c842d:::benchmark_data_smithi194_103039_object86359:head# 2024-04-29T14:21:18.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2c8bc0:::benchmark_data_smithi194_103039_object79441:head# 2024-04-29T14:21:18.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2cb7fd:::benchmark_data_smithi194_103039_object81774:head# 2024-04-29T14:21:18.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2dbc19:::benchmark_data_smithi194_103039_object78223:head# 2024-04-29T14:21:18.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2ecbd0:::benchmark_data_smithi194_103039_object75369:head# 2024-04-29T14:21:18.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2ee10b:::benchmark_data_smithi194_103039_object57526:head# 2024-04-29T14:21:18.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a2f3a59:::benchmark_data_smithi194_103039_object79719:head# 2024-04-29T14:21:18.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a308a44:::benchmark_data_smithi194_103039_object77656:head# 2024-04-29T14:21:18.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a30a737:::benchmark_data_smithi194_103039_object69529:head# 2024-04-29T14:21:18.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a316682:::benchmark_data_smithi194_103039_object103893:head# 2024-04-29T14:21:18.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a31a368:::benchmark_data_smithi194_103039_object126915:head# 2024-04-29T14:21:18.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a329368:::benchmark_data_smithi194_103039_object151103:head# 2024-04-29T14:21:18.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a32ae7f:::benchmark_data_smithi194_103039_object126187:head# 2024-04-29T14:21:18.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a332e21:::benchmark_data_smithi194_103039_object52919:head# 2024-04-29T14:21:18.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a33ba94:::benchmark_data_smithi194_103039_object111506:head# 2024-04-29T14:21:18.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a33f003:::benchmark_data_smithi194_103039_object149934:head# 2024-04-29T14:21:18.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a343dee:::benchmark_data_smithi194_103039_object68447:head# 2024-04-29T14:21:18.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a354456:::benchmark_data_smithi194_103039_object112748:head# 2024-04-29T14:21:18.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a361175:::benchmark_data_smithi194_103039_object81741:head# 2024-04-29T14:21:18.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a36491d:::benchmark_data_smithi194_103039_object51529:head# 2024-04-29T14:21:18.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a36da29:::benchmark_data_smithi194_103039_object73337:head# 2024-04-29T14:21:18.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3899f5:::benchmark_data_smithi194_103039_object107262:head# 2024-04-29T14:21:18.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a38caca:::benchmark_data_smithi194_103039_object128777:head# 2024-04-29T14:21:18.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a396e20:::benchmark_data_smithi194_103039_object69154:head# 2024-04-29T14:21:18.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3a2a72:::benchmark_data_smithi194_103039_object80175:head# 2024-04-29T14:21:18.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3a73a5:::benchmark_data_smithi194_103039_object68717:head# 2024-04-29T14:21:18.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3c3047:::benchmark_data_smithi194_103039_object65508:head# 2024-04-29T14:21:18.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3c6a41:::benchmark_data_smithi194_103039_object56683:head# 2024-04-29T14:21:18.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3dc221:::benchmark_data_smithi194_103039_object127910:head# 2024-04-29T14:21:18.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3e44b9:::benchmark_data_smithi194_103039_object104032:head# 2024-04-29T14:21:18.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a3f12a1:::benchmark_data_smithi194_103039_object68657:head# 2024-04-29T14:21:18.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a409ebf:::benchmark_data_smithi194_103039_object64505:head# 2024-04-29T14:21:18.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a40ebc2:::benchmark_data_smithi194_103039_object43823:head# 2024-04-29T14:21:18.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a40f99f:::benchmark_data_smithi194_103039_object144531:head# 2024-04-29T14:21:18.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a427841:::benchmark_data_smithi194_103039_object55549:head# 2024-04-29T14:21:18.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a42c395:::benchmark_data_smithi194_103039_object58387:head# 2024-04-29T14:21:18.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a443264:::benchmark_data_smithi194_103039_object58574:head# 2024-04-29T14:21:18.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4555f1:::benchmark_data_smithi194_103039_object155681:head# 2024-04-29T14:21:18.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a456d25:::benchmark_data_smithi194_103039_object108677:head# 2024-04-29T14:21:18.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a459b44:::benchmark_data_smithi194_103039_object75629:head# 2024-04-29T14:21:18.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a46e294:::benchmark_data_smithi194_103039_object148622:head# 2024-04-29T14:21:18.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4702a8:::benchmark_data_smithi194_103039_object130710:head# 2024-04-29T14:21:18.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a472c44:::benchmark_data_smithi194_103039_object90481:head# 2024-04-29T14:21:18.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a482461:::benchmark_data_smithi194_103039_object90118:head# 2024-04-29T14:21:18.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a49a802:::benchmark_data_smithi194_103039_object148667:head# 2024-04-29T14:21:18.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a49b0a6:::benchmark_data_smithi194_103039_object148800:head# 2024-04-29T14:21:18.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4ab397:::benchmark_data_smithi194_103039_object82085:head# 2024-04-29T14:21:18.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4ac5ef:::benchmark_data_smithi194_103039_object132884:head# 2024-04-29T14:21:18.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4b2369:::benchmark_data_smithi194_103039_object89748:head# 2024-04-29T14:21:18.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4b70f3:::benchmark_data_smithi194_103039_object124508:head# 2024-04-29T14:21:18.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4b71d7:::benchmark_data_smithi194_103039_object133082:head# 2024-04-29T14:21:18.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4c77e0:::benchmark_data_smithi194_103039_object108959:head# 2024-04-29T14:21:18.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4dbaf7:::benchmark_data_smithi194_103039_object153890:head# 2024-04-29T14:21:18.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4dfaf9:::benchmark_data_smithi194_103039_object136617:head# 2024-04-29T14:21:18.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4f69f1:::benchmark_data_smithi194_103039_object94367:head# 2024-04-29T14:21:18.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a4f89cf:::benchmark_data_smithi194_103039_object145225:head# 2024-04-29T14:21:18.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a50c653:::benchmark_data_smithi194_103039_object128127:head# 2024-04-29T14:21:18.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a51792a:::benchmark_data_smithi194_103039_object154432:head# 2024-04-29T14:21:18.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5218a8:::benchmark_data_smithi194_103039_object118107:head# 2024-04-29T14:21:18.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a53bce8:::benchmark_data_smithi194_103039_object67917:head# 2024-04-29T14:21:18.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a545ff2:::benchmark_data_smithi194_103039_object127306:head# 2024-04-29T14:21:18.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5616ac:::benchmark_data_smithi194_103039_object42651:head# 2024-04-29T14:21:18.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5645e4:::benchmark_data_smithi194_103039_object120242:head# 2024-04-29T14:21:18.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a56eb58:::benchmark_data_smithi194_103039_object117364:head# 2024-04-29T14:21:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a57377e:::benchmark_data_smithi194_103039_object146703:head# 2024-04-29T14:21:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a577758:::benchmark_data_smithi194_103039_object76527:head# 2024-04-29T14:21:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a57bab0:::benchmark_data_smithi194_103039_object103394:head# 2024-04-29T14:21:18.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5821cd:::benchmark_data_smithi194_103039_object124046:head# 2024-04-29T14:21:18.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a589783:::benchmark_data_smithi194_103039_object119598:head# 2024-04-29T14:21:18.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a58aa4e:::benchmark_data_smithi194_103039_object104681:head# 2024-04-29T14:21:18.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a58c194:::benchmark_data_smithi194_103039_object58365:head# 2024-04-29T14:21:18.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a595aa3:::benchmark_data_smithi194_103039_object108209:head# 2024-04-29T14:21:18.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a59dec8:::benchmark_data_smithi194_103039_object147294:head# 2024-04-29T14:21:18.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a59f894:::benchmark_data_smithi194_103039_object132410:head# 2024-04-29T14:21:18.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5a0070:::benchmark_data_smithi194_103039_object106780:head# 2024-04-29T14:21:18.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5caaca:::benchmark_data_smithi194_103039_object128894:head# 2024-04-29T14:21:18.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5d7ec4:::benchmark_data_smithi194_103039_object155459:head# 2024-04-29T14:21:18.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a5ed153:::benchmark_data_smithi194_103039_object49227:head# 2024-04-29T14:21:18.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6001b6:::benchmark_data_smithi194_103039_object156016:head# 2024-04-29T14:21:18.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a60bd65:::benchmark_data_smithi194_103039_object62490:head# 2024-04-29T14:21:18.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a620968:::benchmark_data_smithi194_103039_object147606:head# 2024-04-29T14:21:18.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a623f90:::benchmark_data_smithi194_103039_object87087:head# 2024-04-29T14:21:18.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a628c2b:::benchmark_data_smithi194_103039_object146359:head# 2024-04-29T14:21:18.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a62bbb8:::benchmark_data_smithi194_103039_object53947:head# 2024-04-29T14:21:18.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a62f126:::benchmark_data_smithi194_103039_object82474:head# 2024-04-29T14:21:18.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6385a5:::benchmark_data_smithi194_103039_object75162:head# 2024-04-29T14:21:18.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a640475:::benchmark_data_smithi194_103039_object124734:head# 2024-04-29T14:21:18.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a650a19:::benchmark_data_smithi194_103039_object115618:head# 2024-04-29T14:21:18.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6604c5:::benchmark_data_smithi194_103039_object102243:head# 2024-04-29T14:21:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a666845:::benchmark_data_smithi194_103039_object44930:head# 2024-04-29T14:21:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a66c296:::benchmark_data_smithi194_103039_object56273:head# 2024-04-29T14:21:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a67295b:::benchmark_data_smithi194_103039_object50688:head# 2024-04-29T14:21:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a67a994:::benchmark_data_smithi194_103039_object77452:head# 2024-04-29T14:21:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a681181:::benchmark_data_smithi194_103039_object153407:head# 2024-04-29T14:21:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a68ffc8:::benchmark_data_smithi194_103039_object87379:head# 2024-04-29T14:21:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a697970:::benchmark_data_smithi194_103039_object118831:head# 2024-04-29T14:21:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a69bcc7:::benchmark_data_smithi194_103039_object51531:head# 2024-04-29T14:21:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a69fb25:::benchmark_data_smithi194_103039_object63103:head# 2024-04-29T14:21:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6a55cb:::benchmark_data_smithi194_103039_object149180:head# 2024-04-29T14:21:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6aa591:::benchmark_data_smithi194_103039_object119908:head# 2024-04-29T14:21:18.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6b1e7b:::benchmark_data_smithi194_103039_object155247:head# 2024-04-29T14:21:18.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6b5d3a:::benchmark_data_smithi194_103039_object151496:head# 2024-04-29T14:21:18.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6ba44a:::benchmark_data_smithi194_103039_object93852:head# 2024-04-29T14:21:18.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6bc36d:::benchmark_data_smithi194_103039_object52821:head# 2024-04-29T14:21:18.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6c3b6e:::benchmark_data_smithi194_103039_object115056:head# 2024-04-29T14:21:18.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6cbdbb:::benchmark_data_smithi194_103039_object52375:head# 2024-04-29T14:21:18.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6cdaa4:::benchmark_data_smithi194_103039_object96203:head# 2024-04-29T14:21:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6cdd7c:::benchmark_data_smithi194_103039_object77773:head# 2024-04-29T14:21:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6d92d3:::benchmark_data_smithi194_103039_object51335:head# 2024-04-29T14:21:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6e07a7:::benchmark_data_smithi194_103039_object96266:head# 2024-04-29T14:21:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6f39f5:::benchmark_data_smithi194_103039_object51215:head# 2024-04-29T14:21:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6f65d7:::benchmark_data_smithi194_103039_object90015:head# 2024-04-29T14:21:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a6f706d:::benchmark_data_smithi194_103039_object122291:head# 2024-04-29T14:21:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a70031f:::benchmark_data_smithi194_103039_object49767:head# 2024-04-29T14:21:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a701bbf:::benchmark_data_smithi194_103039_object51932:head# 2024-04-29T14:21:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a702cde:::benchmark_data_smithi194_103039_object59396:head# 2024-04-29T14:21:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a719b7e:::benchmark_data_smithi194_103039_object148247:head# 2024-04-29T14:21:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7214ee:::benchmark_data_smithi194_103039_object114710:head# 2024-04-29T14:21:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a73256f:::benchmark_data_smithi194_103039_object81172:head# 2024-04-29T14:21:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a733412:::benchmark_data_smithi194_103039_object99408:head# 2024-04-29T14:21:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a73dfa6:::benchmark_data_smithi194_103039_object132743:head# 2024-04-29T14:21:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a74d00e:::benchmark_data_smithi194_103039_object57612:head# 2024-04-29T14:21:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a75ad5e:::benchmark_data_smithi194_103039_object63256:head# 2024-04-29T14:21:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a75daa7:::benchmark_data_smithi194_103039_object45628:head# 2024-04-29T14:21:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7698ac:::benchmark_data_smithi194_103039_object53631:head# 2024-04-29T14:21:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a769d32:::benchmark_data_smithi194_103039_object98400:head# 2024-04-29T14:21:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a76b4b2:::benchmark_data_smithi194_103039_object115898:head# 2024-04-29T14:21:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a771945:::benchmark_data_smithi194_103039_object80306:head# 2024-04-29T14:21:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a77717a:::benchmark_data_smithi194_103039_object112534:head# 2024-04-29T14:21:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a78f846:::benchmark_data_smithi194_103039_object143922:head# 2024-04-29T14:21:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a79af7f:::benchmark_data_smithi194_103039_object79800:head# 2024-04-29T14:21:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a79d2f4:::benchmark_data_smithi194_103039_object142615:head# 2024-04-29T14:21:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7a5696:::benchmark_data_smithi194_103039_object123571:head# 2024-04-29T14:21:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7a776a:::benchmark_data_smithi194_103039_object120873:head# 2024-04-29T14:21:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7b140c:::benchmark_data_smithi194_103039_object122313:head# 2024-04-29T14:21:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7bdbed:::benchmark_data_smithi194_103039_object64339:head# 2024-04-29T14:21:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7d03eb:::benchmark_data_smithi194_103039_object138550:head# 2024-04-29T14:21:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7d2e92:::benchmark_data_smithi194_103039_object134785:head# 2024-04-29T14:21:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7def9d:::benchmark_data_smithi194_103039_object133384:head# 2024-04-29T14:21:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7f91d5:::benchmark_data_smithi194_103039_object133128:head# 2024-04-29T14:21:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7fe7b1:::benchmark_data_smithi194_103039_object126356:head# 2024-04-29T14:21:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a7ff06b:::benchmark_data_smithi194_103039_object143574:head# 2024-04-29T14:21:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a800ab4:::benchmark_data_smithi194_103039_object74979:head# 2024-04-29T14:21:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a806d15:::benchmark_data_smithi194_103039_object70080:head# 2024-04-29T14:21:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a815ae4:::benchmark_data_smithi194_103039_object75739:head# 2024-04-29T14:21:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8265ca:::benchmark_data_smithi194_103039_object144582:head# 2024-04-29T14:21:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a828531:::benchmark_data_smithi194_103039_object45901:head# 2024-04-29T14:21:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a828a82:::benchmark_data_smithi194_103039_object129393:head# 2024-04-29T14:21:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a82a46d:::benchmark_data_smithi194_103039_object48381:head# 2024-04-29T14:21:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a82e966:::benchmark_data_smithi194_103039_object87125:head# 2024-04-29T14:21:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a834433:::benchmark_data_smithi194_103039_object145659:head# 2024-04-29T14:21:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a837335:::benchmark_data_smithi194_103039_object115384:head# 2024-04-29T14:21:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a840cff:::benchmark_data_smithi194_103039_object117658:head# 2024-04-29T14:21:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a84355e:::benchmark_data_smithi194_103039_object75646:head# 2024-04-29T14:21:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a84faf3:::benchmark_data_smithi194_103039_object46953:head# 2024-04-29T14:21:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a85792b:::benchmark_data_smithi194_103039_object73902:head# 2024-04-29T14:21:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8618c6:::benchmark_data_smithi194_103039_object84072:head# 2024-04-29T14:21:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8732ae:::benchmark_data_smithi194_103039_object131940:head# 2024-04-29T14:21:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a87c37a:::benchmark_data_smithi194_103039_object87276:head# 2024-04-29T14:21:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8a6bd2:::benchmark_data_smithi194_103039_object87075:head# 2024-04-29T14:21:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8b8933:::benchmark_data_smithi194_103039_object94544:head# 2024-04-29T14:21:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8c3795:::benchmark_data_smithi194_103039_object134190:head# 2024-04-29T14:21:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8c7c34:::benchmark_data_smithi194_103039_object92032:head# 2024-04-29T14:21:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8d1fd7:::benchmark_data_smithi194_103039_object43974:head# 2024-04-29T14:21:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8d5c7e:::benchmark_data_smithi194_103039_object59060:head# 2024-04-29T14:21:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8d6a96:::benchmark_data_smithi194_103039_object96793:head# 2024-04-29T14:21:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8d800b:::benchmark_data_smithi194_103039_object61023:head# 2024-04-29T14:21:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8d846e:::benchmark_data_smithi194_103039_object71159:head# 2024-04-29T14:21:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8dd0bc:::benchmark_data_smithi194_103039_object62149:head# 2024-04-29T14:21:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8e0265:::benchmark_data_smithi194_103039_object141604:head# 2024-04-29T14:21:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8e14e6:::benchmark_data_smithi194_103039_object85707:head# 2024-04-29T14:21:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a8e5116:::benchmark_data_smithi194_103039_object132434:head# 2024-04-29T14:21:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a916dd1:::benchmark_data_smithi194_103039_object82161:head# 2024-04-29T14:21:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a917934:::benchmark_data_smithi194_103039_object52508:head# 2024-04-29T14:21:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a91c4e7:::benchmark_data_smithi194_103039_object54002:head# 2024-04-29T14:21:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a91e944:::benchmark_data_smithi194_103039_object76247:head# 2024-04-29T14:21:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a925e7c:::benchmark_data_smithi194_103039_object151377:head# 2024-04-29T14:21:18.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a92b77a:::benchmark_data_smithi194_103039_object105308:head# 2024-04-29T14:21:18.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9341a1:::benchmark_data_smithi194_103039_object73013:head# 2024-04-29T14:21:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a937d42:::benchmark_data_smithi194_103039_object102895:head# 2024-04-29T14:21:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a93f76b:::benchmark_data_smithi194_103039_object92992:head# 2024-04-29T14:21:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a96d19c:::benchmark_data_smithi194_103039_object89189:head# 2024-04-29T14:21:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a977fcd:::benchmark_data_smithi194_103039_object133638:head# 2024-04-29T14:21:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a97bda2:::benchmark_data_smithi194_103039_object135111:head# 2024-04-29T14:21:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a988913:::benchmark_data_smithi194_103039_object124828:head# 2024-04-29T14:21:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a98a721:::benchmark_data_smithi194_103039_object90848:head# 2024-04-29T14:21:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9a0cfa:::benchmark_data_smithi194_103039_object52619:head# 2024-04-29T14:21:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9a2918:::benchmark_data_smithi194_103039_object74281:head# 2024-04-29T14:21:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9a3a07:::benchmark_data_smithi194_103039_object54245:head# 2024-04-29T14:21:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9a9d28:::benchmark_data_smithi194_103039_object78682:head# 2024-04-29T14:21:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9b1005:::benchmark_data_smithi194_103039_object153302:head# 2024-04-29T14:21:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9b6edb:::benchmark_data_smithi194_103039_object115112:head# 2024-04-29T14:21:18.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9b7326:::benchmark_data_smithi194_103039_object77907:head# 2024-04-29T14:21:18.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9ba215:::benchmark_data_smithi194_103039_object151507:head# 2024-04-29T14:21:18.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9c0f6c:::benchmark_data_smithi194_103039_object55046:head# 2024-04-29T14:21:18.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9c1a3d:::benchmark_data_smithi194_103039_object148210:head# 2024-04-29T14:21:18.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9cc9d1:::benchmark_data_smithi194_103039_object76212:head# 2024-04-29T14:21:18.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9d1877:::benchmark_data_smithi194_103039_object43606:head# 2024-04-29T14:21:18.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9dfb23:::benchmark_data_smithi194_103039_object77938:head# 2024-04-29T14:21:18.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9e8293:::benchmark_data_smithi194_103039_object114076:head# 2024-04-29T14:21:18.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9ec1a2:::benchmark_data_smithi194_103039_object101759:head# 2024-04-29T14:21:18.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9ec7cc:::benchmark_data_smithi194_103039_object134687:head# 2024-04-29T14:21:18.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9f6ed3:::benchmark_data_smithi194_103039_object115986:head# 2024-04-29T14:21:18.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9fc468:::benchmark_data_smithi194_103039_object57374:head# 2024-04-29T14:21:18.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4a9fc731:::benchmark_data_smithi194_103039_object89089:head# 2024-04-29T14:21:18.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa0138a:::benchmark_data_smithi194_103039_object51315:head# 2024-04-29T14:21:18.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa06baa:::benchmark_data_smithi194_103039_object95050:head# 2024-04-29T14:21:18.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa123dc:::benchmark_data_smithi194_103039_object137898:head# 2024-04-29T14:21:18.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa20435:::benchmark_data_smithi194_103039_object73399:head# 2024-04-29T14:21:18.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa2c8a5:::benchmark_data_smithi194_103039_object125142:head# 2024-04-29T14:21:18.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa3203a:::benchmark_data_smithi194_103039_object97952:head# 2024-04-29T14:21:18.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa37447:::benchmark_data_smithi194_103039_object123435:head# 2024-04-29T14:21:18.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa411ba:::benchmark_data_smithi194_103039_object141506:head# 2024-04-29T14:21:18.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa4aee0:::benchmark_data_smithi194_103039_object60766:head# 2024-04-29T14:21:18.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa74278:::benchmark_data_smithi194_103039_object144811:head# 2024-04-29T14:21:18.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa888e8:::benchmark_data_smithi194_103039_object44782:head# 2024-04-29T14:21:18.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa9160e:::benchmark_data_smithi194_103039_object142754:head# 2024-04-29T14:21:18.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa91b6b:::benchmark_data_smithi194_103039_object54415:head# 2024-04-29T14:21:18.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aa93896:::benchmark_data_smithi194_103039_object111551:head# 2024-04-29T14:21:18.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aaa5a64:::benchmark_data_smithi194_103039_object66659:head# 2024-04-29T14:21:18.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aaab0b3:::benchmark_data_smithi194_103039_object59524:head# 2024-04-29T14:21:18.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aac1d1a:::benchmark_data_smithi194_103039_object115183:head# 2024-04-29T14:21:18.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aac4f92:::benchmark_data_smithi194_103039_object41611:head# 2024-04-29T14:21:18.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aac900f:::benchmark_data_smithi194_103039_object61410:head# 2024-04-29T14:21:18.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aad1692:::benchmark_data_smithi194_103039_object58686:head# 2024-04-29T14:21:18.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aad3a0e:::benchmark_data_smithi194_103039_object85458:head# 2024-04-29T14:21:18.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aadeb09:::benchmark_data_smithi194_103039_object130910:head# 2024-04-29T14:21:18.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aadfa35:::benchmark_data_smithi194_103039_object124605:head# 2024-04-29T14:21:18.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aae3994:::benchmark_data_smithi194_103039_object93433:head# 2024-04-29T14:21:18.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aae4f0f:::benchmark_data_smithi194_103039_object47519:head# 2024-04-29T14:21:18.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aaf030e:::benchmark_data_smithi194_103039_object103647:head# 2024-04-29T14:21:18.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aaf33ab:::benchmark_data_smithi194_103039_object80461:head# 2024-04-29T14:21:18.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab137f5:::benchmark_data_smithi194_103039_object115650:head# 2024-04-29T14:21:18.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab15013:::benchmark_data_smithi194_103039_object114161:head# 2024-04-29T14:21:18.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab15aa6:::benchmark_data_smithi194_103039_object119919:head# 2024-04-29T14:21:18.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab1d41d:::benchmark_data_smithi194_103039_object89472:head# 2024-04-29T14:21:18.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab297e4:::benchmark_data_smithi194_103039_object46936:head# 2024-04-29T14:21:18.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab32629:::benchmark_data_smithi194_103039_object57114:head# 2024-04-29T14:21:18.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab36a5d:::benchmark_data_smithi194_103039_object153883:head# 2024-04-29T14:21:18.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab38c98:::benchmark_data_smithi194_103039_object78820:head# 2024-04-29T14:21:18.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab42fc7:::benchmark_data_smithi194_103039_object87796:head# 2024-04-29T14:21:18.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab50b32:::benchmark_data_smithi194_103039_object130781:head# 2024-04-29T14:21:18.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab5300d:::benchmark_data_smithi194_103039_object77034:head# 2024-04-29T14:21:18.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab59f83:::benchmark_data_smithi194_103039_object45498:head# 2024-04-29T14:21:18.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab5a969:::benchmark_data_smithi194_103039_object149970:head# 2024-04-29T14:21:18.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab67c8e:::benchmark_data_smithi194_103039_object65562:head# 2024-04-29T14:21:18.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab6c949:::benchmark_data_smithi194_103039_object77129:head# 2024-04-29T14:21:18.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab6fd4e:::benchmark_data_smithi194_103039_object96285:head# 2024-04-29T14:21:18.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab75dc1:::benchmark_data_smithi194_103039_object88050:head# 2024-04-29T14:21:18.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab77b4f:::benchmark_data_smithi194_103039_object107487:head# 2024-04-29T14:21:18.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab78c2c:::benchmark_data_smithi194_103039_object81698:head# 2024-04-29T14:21:18.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab7c96f:::benchmark_data_smithi194_103039_object81169:head# 2024-04-29T14:21:18.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab7cc80:::benchmark_data_smithi194_103039_object143010:head# 2024-04-29T14:21:18.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab7fe6e:::benchmark_data_smithi194_103039_object93668:head# 2024-04-29T14:21:18.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab80cef:::benchmark_data_smithi194_103039_object53164:head# 2024-04-29T14:21:18.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab83ad8:::benchmark_data_smithi194_103039_object112058:head# 2024-04-29T14:21:18.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab87755:::benchmark_data_smithi194_103039_object85639:head# 2024-04-29T14:21:18.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab8bfd5:::benchmark_data_smithi194_103039_object66920:head# 2024-04-29T14:21:18.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab8e6d5:::benchmark_data_smithi194_103039_object87566:head# 2024-04-29T14:21:18.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ab99952:::benchmark_data_smithi194_103039_object65084:head# 2024-04-29T14:21:18.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4abb42e3:::benchmark_data_smithi194_103039_object95474:head# 2024-04-29T14:21:18.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4abd8914:::benchmark_data_smithi194_103039_object130741:head# 2024-04-29T14:21:18.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4abe060b:::benchmark_data_smithi194_103039_object150013:head# 2024-04-29T14:21:18.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4abe28cc:::benchmark_data_smithi194_103039_object52151:head# 2024-04-29T14:21:18.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4abf46b6:::benchmark_data_smithi194_103039_object143433:head# 2024-04-29T14:21:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4abf5b8d:::benchmark_data_smithi194_103039_object141865:head# 2024-04-29T14:21:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4abfb9cb:::benchmark_data_smithi194_103039_object76170:head# 2024-04-29T14:21:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac020ca:::benchmark_data_smithi194_103039_object151554:head# 2024-04-29T14:21:18.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac07659:::benchmark_data_smithi194_103039_object126624:head# 2024-04-29T14:21:18.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac09a03:::benchmark_data_smithi194_103039_object72734:head# 2024-04-29T14:21:18.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac0ee9c:::benchmark_data_smithi194_103039_object106539:head# 2024-04-29T14:21:18.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac17609:::benchmark_data_smithi194_103039_object53316:head# 2024-04-29T14:21:18.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac182c4:::benchmark_data_smithi194_103039_object111688:head# 2024-04-29T14:21:18.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac25456:::benchmark_data_smithi194_103039_object154251:head# 2024-04-29T14:21:18.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac2e46c:::benchmark_data_smithi194_103039_object51940:head# 2024-04-29T14:21:18.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac33a4c:::benchmark_data_smithi194_103039_object106998:head# 2024-04-29T14:21:18.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac54080:::benchmark_data_smithi194_103039_object62143:head# 2024-04-29T14:21:18.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac72ebf:::benchmark_data_smithi194_103039_object101817:head# 2024-04-29T14:21:18.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac79e7b:::benchmark_data_smithi194_103039_object117801:head# 2024-04-29T14:21:18.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ac7a641:::benchmark_data_smithi194_103039_object88317:head# 2024-04-29T14:21:18.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acaed55:::benchmark_data_smithi194_103039_object62675:head# 2024-04-29T14:21:18.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acaef1f:::benchmark_data_smithi194_103039_object60264:head# 2024-04-29T14:21:18.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acafb59:::benchmark_data_smithi194_103039_object98475:head# 2024-04-29T14:21:18.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acafde0:::benchmark_data_smithi194_103039_object96279:head# 2024-04-29T14:21:18.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acbc282:::benchmark_data_smithi194_103039_object116331:head# 2024-04-29T14:21:18.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acbffb4:::benchmark_data_smithi194_103039_object47065:head# 2024-04-29T14:21:18.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acc6d93:::benchmark_data_smithi194_103039_object129566:head# 2024-04-29T14:21:18.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acd2a3c:::benchmark_data_smithi194_103039_object138026:head# 2024-04-29T14:21:18.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acd2f1e:::benchmark_data_smithi194_103039_object65948:head# 2024-04-29T14:21:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acd47c8:::benchmark_data_smithi194_103039_object119980:head# 2024-04-29T14:21:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acd9503:::benchmark_data_smithi194_103039_object102154:head# 2024-04-29T14:21:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ace02b5:::benchmark_data_smithi194_103039_object59279:head# 2024-04-29T14:21:18.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ace2ae7:::benchmark_data_smithi194_103039_object144128:head# 2024-04-29T14:21:18.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ace30b9:::benchmark_data_smithi194_103039_object95545:head# 2024-04-29T14:21:18.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acf0a3f:::benchmark_data_smithi194_103039_object69786:head# 2024-04-29T14:21:18.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acfa910:::benchmark_data_smithi194_103039_object93955:head# 2024-04-29T14:21:18.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acfb315:::benchmark_data_smithi194_103039_object73285:head# 2024-04-29T14:21:18.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acfc0fb:::benchmark_data_smithi194_103039_object64879:head# 2024-04-29T14:21:18.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4acfe1a5:::benchmark_data_smithi194_103039_object55120:head# 2024-04-29T14:21:18.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad1d6f8:::benchmark_data_smithi194_103039_object124680:head# 2024-04-29T14:21:18.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad35d47:::benchmark_data_smithi194_103039_object98739:head# 2024-04-29T14:21:18.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad5ecaa:::benchmark_data_smithi194_103039_object120492:head# 2024-04-29T14:21:18.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad60061:::benchmark_data_smithi194_103039_object53828:head# 2024-04-29T14:21:18.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad6b1a4:::benchmark_data_smithi194_103039_object143238:head# 2024-04-29T14:21:18.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad7421e:::benchmark_data_smithi194_103039_object155920:head# 2024-04-29T14:21:18.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad74f2c:::benchmark_data_smithi194_103039_object56469:head# 2024-04-29T14:21:18.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad8a425:::benchmark_data_smithi194_103039_object59904:head# 2024-04-29T14:21:18.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad8b043:::benchmark_data_smithi194_103039_object51107:head# 2024-04-29T14:21:18.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ad9b0ae:::benchmark_data_smithi194_103039_object156014:head# 2024-04-29T14:21:18.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4adaca28:::benchmark_data_smithi194_103039_object52518:head# 2024-04-29T14:21:18.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4adb4812:::benchmark_data_smithi194_103039_object116251:head# 2024-04-29T14:21:18.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4adbe1ff:::benchmark_data_smithi194_103039_object83180:head# 2024-04-29T14:21:18.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4adc290c:::benchmark_data_smithi194_103039_object134604:head# 2024-04-29T14:21:18.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4adcef9d:::benchmark_data_smithi194_103039_object51830:head# 2024-04-29T14:21:18.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ade1aaf:::benchmark_data_smithi194_103039_object137730:head# 2024-04-29T14:21:18.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ade75c3:::benchmark_data_smithi194_103039_object55660:head# 2024-04-29T14:21:18.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ade7a2d:::benchmark_data_smithi194_103039_object126046:head# 2024-04-29T14:21:18.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4adeaada:::benchmark_data_smithi194_103039_object46961:head# 2024-04-29T14:21:18.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4adf7dfc:::benchmark_data_smithi194_103039_object68272:head# 2024-04-29T14:21:18.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae09a1c:::benchmark_data_smithi194_103039_object55248:head# 2024-04-29T14:21:18.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae0fdb0:::benchmark_data_smithi194_103039_object71862:head# 2024-04-29T14:21:18.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae126e5:::benchmark_data_smithi194_103039_object82359:head# 2024-04-29T14:21:18.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae12fbe:::benchmark_data_smithi194_103039_object97078:head# 2024-04-29T14:21:18.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae1707a:::benchmark_data_smithi194_103039_object144495:head# 2024-04-29T14:21:18.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae1b25b:::benchmark_data_smithi194_103039_object154673:head# 2024-04-29T14:21:18.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae29a2f:::benchmark_data_smithi194_103039_object45736:head# 2024-04-29T14:21:18.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae35c46:::benchmark_data_smithi194_103039_object80072:head# 2024-04-29T14:21:18.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae3e8c5:::benchmark_data_smithi194_103039_object68522:head# 2024-04-29T14:21:18.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae41849:::benchmark_data_smithi194_103039_object71381:head# 2024-04-29T14:21:18.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae4f369:::benchmark_data_smithi194_103039_object92347:head# 2024-04-29T14:21:18.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae6b346:::benchmark_data_smithi194_103039_object129489:head# 2024-04-29T14:21:18.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae7a10a:::benchmark_data_smithi194_103039_object151355:head# 2024-04-29T14:21:18.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae86924:::benchmark_data_smithi194_103039_object71709:head# 2024-04-29T14:21:18.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ae893d1:::benchmark_data_smithi194_103039_object114617:head# 2024-04-29T14:21:18.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aea23a2:::benchmark_data_smithi194_103039_object44858:head# 2024-04-29T14:21:18.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aea426d:::benchmark_data_smithi194_103039_object63648:head# 2024-04-29T14:21:18.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aea8586:::benchmark_data_smithi194_103039_object113780:head# 2024-04-29T14:21:18.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aeaadfe:::benchmark_data_smithi194_103039_object120823:head# 2024-04-29T14:21:18.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aeac02f:::benchmark_data_smithi194_103039_object88188:head# 2024-04-29T14:21:18.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aec4ebc:::benchmark_data_smithi194_103039_object104363:head# 2024-04-29T14:21:18.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aec890a:::benchmark_data_smithi194_103039_object81874:head# 2024-04-29T14:21:18.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aece699:::benchmark_data_smithi194_103039_object91734:head# 2024-04-29T14:21:18.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aed3080:::benchmark_data_smithi194_103039_object90641:head# 2024-04-29T14:21:18.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aee7f94:::benchmark_data_smithi194_103039_object72868:head# 2024-04-29T14:21:18.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4aefcfa9:::benchmark_data_smithi194_103039_object106721:head# 2024-04-29T14:21:18.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af0556a:::benchmark_data_smithi194_103039_object134695:head# 2024-04-29T14:21:18.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af06c7f:::benchmark_data_smithi194_103039_object148568:head# 2024-04-29T14:21:18.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af06e33:::benchmark_data_smithi194_103039_object131997:head# 2024-04-29T14:21:18.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af0b391:::benchmark_data_smithi194_103039_object43077:head# 2024-04-29T14:21:18.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af1a130:::benchmark_data_smithi194_103039_object151073:head# 2024-04-29T14:21:18.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af1c537:::benchmark_data_smithi194_103039_object75407:head# 2024-04-29T14:21:18.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af1fd48:::benchmark_data_smithi194_103039_object50503:head# 2024-04-29T14:21:18.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af25774:::benchmark_data_smithi194_103039_object119196:head# 2024-04-29T14:21:18.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af3583e:::benchmark_data_smithi194_103039_object79746:head# 2024-04-29T14:21:18.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af35dd9:::benchmark_data_smithi194_103039_object123257:head# 2024-04-29T14:21:18.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af3b713:::benchmark_data_smithi194_103039_object73834:head# 2024-04-29T14:21:18.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af426dc:::benchmark_data_smithi194_103039_object104823:head# 2024-04-29T14:21:18.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af431be:::benchmark_data_smithi194_103039_object88551:head# 2024-04-29T14:21:18.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af437c0:::benchmark_data_smithi194_103039_object54126:head# 2024-04-29T14:21:18.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af4e071:::benchmark_data_smithi194_103039_object48871:head# 2024-04-29T14:21:18.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af51ab7:::benchmark_data_smithi194_103039_object85856:head# 2024-04-29T14:21:18.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af51f39:::benchmark_data_smithi194_103039_object127784:head# 2024-04-29T14:21:18.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af56199:::benchmark_data_smithi194_103039_object56314:head# 2024-04-29T14:21:18.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af56f1b:::benchmark_data_smithi194_103039_object116456:head# 2024-04-29T14:21:18.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af685b2:::benchmark_data_smithi194_103039_object43799:head# 2024-04-29T14:21:18.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af68812:::benchmark_data_smithi194_103039_object134931:head# 2024-04-29T14:21:18.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af69472:::benchmark_data_smithi194_103039_object98691:head# 2024-04-29T14:21:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af7b313:::benchmark_data_smithi194_103039_object54654:head# 2024-04-29T14:21:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af7bc45:::benchmark_data_smithi194_103039_object74146:head# 2024-04-29T14:21:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af7f7e4:::benchmark_data_smithi194_103039_object103638:head# 2024-04-29T14:21:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af8c617:::benchmark_data_smithi194_103039_object45503:head# 2024-04-29T14:21:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af91b3c:::benchmark_data_smithi194_103039_object58873:head# 2024-04-29T14:21:18.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af935a3:::benchmark_data_smithi194_103039_object91478:head# 2024-04-29T14:21:18.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4af97dd9:::benchmark_data_smithi194_103039_object127230:head# 2024-04-29T14:21:18.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afa625d:::benchmark_data_smithi194_103039_object93710:head# 2024-04-29T14:21:18.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afa645b:::benchmark_data_smithi194_103039_object56442:head# 2024-04-29T14:21:18.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afb1ce1:::benchmark_data_smithi194_103039_object53125:head# 2024-04-29T14:21:18.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afbdd8d:::benchmark_data_smithi194_103039_object124272:head# 2024-04-29T14:21:18.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afbe6cf:::benchmark_data_smithi194_103039_object64023:head# 2024-04-29T14:21:18.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afc4496:::benchmark_data_smithi194_103039_object103769:head# 2024-04-29T14:21:18.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afcb494:::benchmark_data_smithi194_103039_object109654:head# 2024-04-29T14:21:18.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4afd032b:::benchmark_data_smithi194_103039_object60500:head# 2024-04-29T14:21:18.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4affa39c:::benchmark_data_smithi194_103039_object72861:head# 2024-04-29T14:21:18.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b00c333:::benchmark_data_smithi194_103039_object105928:head# 2024-04-29T14:21:18.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b017ed9:::benchmark_data_smithi194_103039_object150222:head# 2024-04-29T14:21:18.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b01e709:::benchmark_data_smithi194_103039_object106088:head# 2024-04-29T14:21:18.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b03e719:::benchmark_data_smithi194_103039_object46029:head# 2024-04-29T14:21:18.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b03ef21:::benchmark_data_smithi194_103039_object139295:head# 2024-04-29T14:21:18.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b042b16:::benchmark_data_smithi194_103039_object68962:head# 2024-04-29T14:21:18.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b04792e:::benchmark_data_smithi194_103039_object128243:head# 2024-04-29T14:21:18.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b05dbb1:::benchmark_data_smithi194_103039_object43247:head# 2024-04-29T14:21:18.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0696ac:::benchmark_data_smithi194_103039_object71117:head# 2024-04-29T14:21:18.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0697ec:::benchmark_data_smithi194_103039_object92552:head# 2024-04-29T14:21:18.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0734ed:::benchmark_data_smithi194_103039_object71542:head# 2024-04-29T14:21:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b087159:::benchmark_data_smithi194_103039_object70390:head# 2024-04-29T14:21:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b09a5bf:::benchmark_data_smithi194_103039_object104576:head# 2024-04-29T14:21:18.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0a6ae3:::benchmark_data_smithi194_103039_object145892:head# 2024-04-29T14:21:18.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0b3c43:::benchmark_data_smithi194_103039_object46564:head# 2024-04-29T14:21:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0bacb0:::benchmark_data_smithi194_103039_object42806:head# 2024-04-29T14:21:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0bad85:::benchmark_data_smithi194_103039_object43018:head# 2024-04-29T14:21:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0cbcbd:::benchmark_data_smithi194_103039_object48466:head# 2024-04-29T14:21:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0d1b30:::benchmark_data_smithi194_103039_object138127:head# 2024-04-29T14:21:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0d4f87:::benchmark_data_smithi194_103039_object53790:head# 2024-04-29T14:21:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0de6d5:::benchmark_data_smithi194_103039_object75166:head# 2024-04-29T14:21:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0eb1d5:::benchmark_data_smithi194_103039_object60961:head# 2024-04-29T14:21:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b0f3512:::benchmark_data_smithi194_103039_object43847:head# 2024-04-29T14:21:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b102058:::benchmark_data_smithi194_103039_object94242:head# 2024-04-29T14:21:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b106c55:::benchmark_data_smithi194_103039_object86699:head# 2024-04-29T14:21:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b10c153:::benchmark_data_smithi194_103039_object107503:head# 2024-04-29T14:21:18.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b111d06:::benchmark_data_smithi194_103039_object63225:head# 2024-04-29T14:21:18.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b11569a:::benchmark_data_smithi194_103039_object95787:head# 2024-04-29T14:21:18.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b11af15:::benchmark_data_smithi194_103039_object114491:head# 2024-04-29T14:21:18.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b121f04:::benchmark_data_smithi194_103039_object100532:head# 2024-04-29T14:21:18.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b154a3f:::benchmark_data_smithi194_103039_object152089:head# 2024-04-29T14:21:18.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b15c33a:::benchmark_data_smithi194_103039_object47652:head# 2024-04-29T14:21:18.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b162197:::benchmark_data_smithi194_103039_object93969:head# 2024-04-29T14:21:18.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b16ed30:::benchmark_data_smithi194_103039_object54806:head# 2024-04-29T14:21:18.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b170b15:::benchmark_data_smithi194_103039_object105105:head# 2024-04-29T14:21:18.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b18beff:::benchmark_data_smithi194_103039_object53008:head# 2024-04-29T14:21:18.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b19b2e3:::benchmark_data_smithi194_103039_object139172:head# 2024-04-29T14:21:18.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b19cdf8:::benchmark_data_smithi194_103039_object89053:head# 2024-04-29T14:21:18.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b1b5856:::benchmark_data_smithi194_103039_object60531:head# 2024-04-29T14:21:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b1c40bf:::benchmark_data_smithi194_103039_object71079:head# 2024-04-29T14:21:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b1cbf17:::benchmark_data_smithi194_103039_object111490:head# 2024-04-29T14:21:18.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b1d1659:::benchmark_data_smithi194_103039_object81679:head# 2024-04-29T14:21:18.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b1ed4c1:::benchmark_data_smithi194_103039_object69362:head# 2024-04-29T14:21:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b1edb31:::benchmark_data_smithi194_103039_object132101:head# 2024-04-29T14:21:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b20323e:::benchmark_data_smithi194_103039_object75970:head# 2024-04-29T14:21:18.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2132d5:::benchmark_data_smithi194_103039_object64131:head# 2024-04-29T14:21:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b213d73:::benchmark_data_smithi194_103039_object57698:head# 2024-04-29T14:21:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b21817c:::benchmark_data_smithi194_103039_object47921:head# 2024-04-29T14:21:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b21a11c:::benchmark_data_smithi194_103039_object86317:head# 2024-04-29T14:21:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b21d6e3:::benchmark_data_smithi194_103039_object118986:head# 2024-04-29T14:21:18.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b21fc4e:::benchmark_data_smithi194_103039_object85420:head# 2024-04-29T14:21:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b221396:::benchmark_data_smithi194_103039_object79963:head# 2024-04-29T14:21:18.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b227536:::benchmark_data_smithi194_103039_object154498:head# 2024-04-29T14:21:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2326c2:::benchmark_data_smithi194_103039_object81334:head# 2024-04-29T14:21:18.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b24a997:::benchmark_data_smithi194_103039_object65758:head# 2024-04-29T14:21:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b24b8e1:::benchmark_data_smithi194_103039_object42362:head# 2024-04-29T14:21:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2515c6:::benchmark_data_smithi194_103039_object57823:head# 2024-04-29T14:21:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2585e0:::benchmark_data_smithi194_103039_object107091:head# 2024-04-29T14:21:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2669c8:::benchmark_data_smithi194_103039_object81036:head# 2024-04-29T14:21:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b26a5ec:::benchmark_data_smithi194_103039_object78134:head# 2024-04-29T14:21:18.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b271fbf:::benchmark_data_smithi194_103039_object75506:head# 2024-04-29T14:21:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b275a7a:::benchmark_data_smithi194_103039_object143385:head# 2024-04-29T14:21:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b27fed0:::benchmark_data_smithi194_103039_object57760:head# 2024-04-29T14:21:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b287916:::benchmark_data_smithi194_103039_object130106:head# 2024-04-29T14:21:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b29f1ee:::benchmark_data_smithi194_103039_object56220:head# 2024-04-29T14:21:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2a4c7c:::benchmark_data_smithi194_103039_object149041:head# 2024-04-29T14:21:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2a8d98:::benchmark_data_smithi194_103039_object87866:head# 2024-04-29T14:21:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2a8e5c:::benchmark_data_smithi194_103039_object87822:head# 2024-04-29T14:21:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2ad71f:::benchmark_data_smithi194_103039_object64657:head# 2024-04-29T14:21:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2b0abe:::benchmark_data_smithi194_103039_object144242:head# 2024-04-29T14:21:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2b23f7:::benchmark_data_smithi194_103039_object142837:head# 2024-04-29T14:21:18.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2b3f6f:::benchmark_data_smithi194_103039_object51799:head# 2024-04-29T14:21:18.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2be807:::benchmark_data_smithi194_103039_object140370:head# 2024-04-29T14:21:18.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2c8046:::benchmark_data_smithi194_103039_object53052:head# 2024-04-29T14:21:18.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2cbefb:::benchmark_data_smithi194_103039_object143338:head# 2024-04-29T14:21:18.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2ce153:::benchmark_data_smithi194_103039_object41654:head# 2024-04-29T14:21:18.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2d414e:::benchmark_data_smithi194_103039_object135294:head# 2024-04-29T14:21:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b2f1c0c:::benchmark_data_smithi194_103039_object56727:head# 2024-04-29T14:21:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b30bc6d:::benchmark_data_smithi194_103039_object45131:head# 2024-04-29T14:21:18.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b345982:::benchmark_data_smithi194_103039_object58467:head# 2024-04-29T14:21:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b35b08c:::benchmark_data_smithi194_103039_object143189:head# 2024-04-29T14:21:18.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b36bb8f:::benchmark_data_smithi194_103039_object142033:head# 2024-04-29T14:21:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b37ee19:::benchmark_data_smithi194_103039_object63408:head# 2024-04-29T14:21:18.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b37ffb4:::benchmark_data_smithi194_103039_object102690:head# 2024-04-29T14:21:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b39177e:::benchmark_data_smithi194_103039_object73450:head# 2024-04-29T14:21:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b394be5:::benchmark_data_smithi194_103039_object119527:head# 2024-04-29T14:21:18.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3a051f:::benchmark_data_smithi194_103039_object141058:head# 2024-04-29T14:21:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3ae6f6:::benchmark_data_smithi194_103039_object95188:head# 2024-04-29T14:21:18.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3b327c:::benchmark_data_smithi194_103039_object150010:head# 2024-04-29T14:21:18.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3b5e72:::benchmark_data_smithi194_103039_object63092:head# 2024-04-29T14:21:18.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3bc1dd:::benchmark_data_smithi194_103039_object151561:head# 2024-04-29T14:21:18.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3c53ee:::benchmark_data_smithi194_103039_object76861:head# 2024-04-29T14:21:18.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3d332a:::benchmark_data_smithi194_103039_object141748:head# 2024-04-29T14:21:18.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3d59d0:::benchmark_data_smithi194_103039_object67829:head# 2024-04-29T14:21:18.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3e5320:::benchmark_data_smithi194_103039_object125776:head# 2024-04-29T14:21:18.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3e541d:::benchmark_data_smithi194_103039_object138539:head# 2024-04-29T14:21:18.717 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:21:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b3f75f5:::benchmark_data_smithi194_103039_object109665:head# 2024-04-29T14:21:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b428887:::benchmark_data_smithi194_103039_object136965:head# 2024-04-29T14:21:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b442917:::benchmark_data_smithi194_103039_object72215:head# 2024-04-29T14:21:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b459c74:::benchmark_data_smithi194_103039_object135682:head# 2024-04-29T14:21:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b45e38e:::benchmark_data_smithi194_103039_object137174:head# 2024-04-29T14:21:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b465a53:::benchmark_data_smithi194_103039_object49299:head# 2024-04-29T14:21:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b47fc6d:::benchmark_data_smithi194_103039_object100065:head# 2024-04-29T14:21:18.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b48b409:::benchmark_data_smithi194_103039_object44050:head# 2024-04-29T14:21:18.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b48b756:::benchmark_data_smithi194_103039_object99129:head# 2024-04-29T14:21:18.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b48d264:::benchmark_data_smithi194_103039_object133165:head# 2024-04-29T14:21:18.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b497b29:::benchmark_data_smithi194_103039_object147078:head# 2024-04-29T14:21:18.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b49f0d3:::benchmark_data_smithi194_103039_object82195:head# 2024-04-29T14:21:18.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b49f219:::benchmark_data_smithi194_103039_object155240:head# 2024-04-29T14:21:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b4b2892:::benchmark_data_smithi194_103039_object106815:head# 2024-04-29T14:21:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b4cac20:::benchmark_data_smithi194_103039_object118702:head# 2024-04-29T14:21:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b4d8ffa:::benchmark_data_smithi194_103039_object47525:head# 2024-04-29T14:21:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b4e3aa5:::benchmark_data_smithi194_103039_object143857:head# 2024-04-29T14:21:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b4ef4ed:::benchmark_data_smithi194_103039_object54356:head# 2024-04-29T14:21:18.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b50751d:::benchmark_data_smithi194_103039_object84535:head# 2024-04-29T14:21:18.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b507544:::benchmark_data_smithi194_103039_object94232:head# 2024-04-29T14:21:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b50bf89:::benchmark_data_smithi194_103039_object136419:head# 2024-04-29T14:21:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b543b8c:::benchmark_data_smithi194_103039_object92899:head# 2024-04-29T14:21:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b546181:::benchmark_data_smithi194_103039_object48946:head# 2024-04-29T14:21:18.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5497ff:::benchmark_data_smithi194_103039_object83421:head# 2024-04-29T14:21:18.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b54c913:::benchmark_data_smithi194_103039_object104214:head# 2024-04-29T14:21:18.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b54dbc0:::benchmark_data_smithi194_103039_object50660:head# 2024-04-29T14:21:18.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b54fd82:::benchmark_data_smithi194_103039_object87737:head# 2024-04-29T14:21:18.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b558487:::benchmark_data_smithi194_103039_object88689:head# 2024-04-29T14:21:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b573a8a:::benchmark_data_smithi194_103039_object89547:head# 2024-04-29T14:21:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b577c32:::benchmark_data_smithi194_103039_object108370:head# 2024-04-29T14:21:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b579de9:::benchmark_data_smithi194_103039_object129676:head# 2024-04-29T14:21:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b57b772:::benchmark_data_smithi194_103039_object53413:head# 2024-04-29T14:21:18.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5821fa:::benchmark_data_smithi194_103039_object49697:head# 2024-04-29T14:21:18.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b585501:::benchmark_data_smithi194_103039_object123474:head# 2024-04-29T14:21:18.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b585718:::benchmark_data_smithi194_103039_object136720:head# 2024-04-29T14:21:18.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b58815c:::benchmark_data_smithi194_103039_object83252:head# 2024-04-29T14:21:18.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b58e829:::benchmark_data_smithi194_103039_object44669:head# 2024-04-29T14:21:18.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b59f561:::benchmark_data_smithi194_103039_object131686:head# 2024-04-29T14:21:18.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5a35b2:::benchmark_data_smithi194_103039_object104158:head# 2024-04-29T14:21:18.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5ab155:::benchmark_data_smithi194_103039_object139873:head# 2024-04-29T14:21:18.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5b6940:::benchmark_data_smithi194_103039_object118541:head# 2024-04-29T14:21:18.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5ba0fe:::benchmark_data_smithi194_103039_object102462:head# 2024-04-29T14:21:18.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5ba69a:::benchmark_data_smithi194_103039_object147598:head# 2024-04-29T14:21:18.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5c33c1:::benchmark_data_smithi194_103039_object109040:head# 2024-04-29T14:21:18.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5cbe9c:::benchmark_data_smithi194_103039_object109572:head# 2024-04-29T14:21:18.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5cfa8f:::benchmark_data_smithi194_103039_object62479:head# 2024-04-29T14:21:18.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5da6be:::benchmark_data_smithi194_103039_object104857:head# 2024-04-29T14:21:18.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5dd525:::benchmark_data_smithi194_103039_object84918:head# 2024-04-29T14:21:18.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5e8557:::benchmark_data_smithi194_103039_object48950:head# 2024-04-29T14:21:18.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5ee8c6:::benchmark_data_smithi194_103039_object128543:head# 2024-04-29T14:21:18.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5fcd16:::benchmark_data_smithi194_103039_object103798:head# 2024-04-29T14:21:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b5ff58f:::benchmark_data_smithi194_103039_object91575:head# 2024-04-29T14:21:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b608c5a:::benchmark_data_smithi194_103039_object135023:head# 2024-04-29T14:21:18.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b61b5b5:::benchmark_data_smithi194_103039_object63455:head# 2024-04-29T14:21:18.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b61d513:::benchmark_data_smithi194_103039_object97213:head# 2024-04-29T14:21:18.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b621451:::benchmark_data_smithi194_103039_object62420:head# 2024-04-29T14:21:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6243c3:::benchmark_data_smithi194_103039_object44533:head# 2024-04-29T14:21:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6310e4:::benchmark_data_smithi194_103039_object111198:head# 2024-04-29T14:21:18.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b64491b:::benchmark_data_smithi194_103039_object156071:head# 2024-04-29T14:21:18.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b648e9a:::benchmark_data_smithi194_103039_object149268:head# 2024-04-29T14:21:18.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b64e8df:::benchmark_data_smithi194_103039_object146455:head# 2024-04-29T14:21:18.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b656ba2:::benchmark_data_smithi194_103039_object81669:head# 2024-04-29T14:21:18.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b661fef:::benchmark_data_smithi194_103039_object127718:head# 2024-04-29T14:21:18.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b672803:::benchmark_data_smithi194_103039_object143657:head# 2024-04-29T14:21:18.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b681d7f:::benchmark_data_smithi194_103039_object89492:head# 2024-04-29T14:21:18.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6956bd:::benchmark_data_smithi194_103039_object150370:head# 2024-04-29T14:21:18.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6a0a9d:::benchmark_data_smithi194_103039_object41876:head# 2024-04-29T14:21:18.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6a169f:::benchmark_data_smithi194_103039_object82383:head# 2024-04-29T14:21:18.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6b1c6a:::benchmark_data_smithi194_103039_object118323:head# 2024-04-29T14:21:18.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6bc58b:::benchmark_data_smithi194_103039_object75673:head# 2024-04-29T14:21:18.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6c21f3:::benchmark_data_smithi194_103039_object155603:head# 2024-04-29T14:21:18.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6cbc6d:::benchmark_data_smithi194_103039_object138553:head# 2024-04-29T14:21:18.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6cc9a7:::benchmark_data_smithi194_103039_object113966:head# 2024-04-29T14:21:18.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6ce41c:::benchmark_data_smithi194_103039_object112131:head# 2024-04-29T14:21:18.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6ceac0:::benchmark_data_smithi194_103039_object96820:head# 2024-04-29T14:21:18.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6d21af:::benchmark_data_smithi194_103039_object153162:head# 2024-04-29T14:21:18.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6e6fb1:::benchmark_data_smithi194_103039_object50384:head# 2024-04-29T14:21:18.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6ecbb6:::benchmark_data_smithi194_103039_object131810:head# 2024-04-29T14:21:18.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b6fb839:::benchmark_data_smithi194_103039_object60412:head# 2024-04-29T14:21:18.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b734ffb:::benchmark_data_smithi194_103039_object107849:head# 2024-04-29T14:21:18.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b744779:::benchmark_data_smithi194_103039_object46970:head# 2024-04-29T14:21:18.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b749f2a:::benchmark_data_smithi194_103039_object136353:head# 2024-04-29T14:21:18.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b74df3c:::benchmark_data_smithi194_103039_object71714:head# 2024-04-29T14:21:18.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b751dc9:::benchmark_data_smithi194_103039_object105335:head# 2024-04-29T14:21:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b75262c:::benchmark_data_smithi194_103039_object44184:head# 2024-04-29T14:21:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b75a27e:::benchmark_data_smithi194_103039_object92323:head# 2024-04-29T14:21:18.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7612ae:::benchmark_data_smithi194_103039_object120286:head# 2024-04-29T14:21:18.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b76378d:::benchmark_data_smithi194_103039_object117234:head# 2024-04-29T14:21:18.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b76f638:::benchmark_data_smithi194_103039_object80843:head# 2024-04-29T14:21:18.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7702c2:::benchmark_data_smithi194_103039_object75288:head# 2024-04-29T14:21:18.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b77059e:::benchmark_data_smithi194_103039_object124954:head# 2024-04-29T14:21:18.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b776cd8:::benchmark_data_smithi194_103039_object86029:head# 2024-04-29T14:21:18.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b78710b:::benchmark_data_smithi194_103039_object92534:head# 2024-04-29T14:21:18.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b78e1a5:::benchmark_data_smithi194_103039_object79463:head# 2024-04-29T14:21:18.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7928be:::benchmark_data_smithi194_103039_object65911:head# 2024-04-29T14:21:18.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b793874:::benchmark_data_smithi194_103039_object96636:head# 2024-04-29T14:21:18.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7b1366:::benchmark_data_smithi194_103039_object91493:head# 2024-04-29T14:21:18.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7c4bf8:::benchmark_data_smithi194_103039_object124183:head# 2024-04-29T14:21:18.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7cbb21:::benchmark_data_smithi194_103039_object84704:head# 2024-04-29T14:21:18.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7eaaf4:::benchmark_data_smithi194_103039_object110158:head# 2024-04-29T14:21:18.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b7f0e87:::benchmark_data_smithi194_103039_object50911:head# 2024-04-29T14:21:18.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8135bb:::benchmark_data_smithi194_103039_object99970:head# 2024-04-29T14:21:18.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b82150a:::benchmark_data_smithi194_103039_object141015:head# 2024-04-29T14:21:18.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b82434c:::benchmark_data_smithi194_103039_object58616:head# 2024-04-29T14:21:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b83ba3e:::benchmark_data_smithi194_103039_object96946:head# 2024-04-29T14:21:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b84cdc8:::benchmark_data_smithi194_103039_object57647:head# 2024-04-29T14:21:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b84e0d9:::benchmark_data_smithi194_103039_object120380:head# 2024-04-29T14:21:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b84f9fb:::benchmark_data_smithi194_103039_object133617:head# 2024-04-29T14:21:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b859150:::benchmark_data_smithi194_103039_object142575:head# 2024-04-29T14:21:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b86a600:::benchmark_data_smithi194_103039_object153764:head# 2024-04-29T14:21:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b86a73e:::benchmark_data_smithi194_103039_object153726:head# 2024-04-29T14:21:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b86f47f:::benchmark_data_smithi194_103039_object107437:head# 2024-04-29T14:21:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b872cbe:::benchmark_data_smithi194_103039_object47116:head# 2024-04-29T14:21:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b875fed:::benchmark_data_smithi194_103039_object154336:head# 2024-04-29T14:21:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b87619c:::benchmark_data_smithi194_103039_object136892:head# 2024-04-29T14:21:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b884e89:::benchmark_data_smithi194_103039_object142304:head# 2024-04-29T14:21:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b88ddbd:::benchmark_data_smithi194_103039_object115888:head# 2024-04-29T14:21:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8a6db5:::benchmark_data_smithi194_103039_object130337:head# 2024-04-29T14:21:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8b3c6c:::benchmark_data_smithi194_103039_object144785:head# 2024-04-29T14:21:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8b7b54:::benchmark_data_smithi194_103039_object142545:head# 2024-04-29T14:21:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8b8ace:::benchmark_data_smithi194_103039_object124502:head# 2024-04-29T14:21:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8bdd1f:::benchmark_data_smithi194_103039_object89933:head# 2024-04-29T14:21:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8c79dd:::benchmark_data_smithi194_103039_object103373:head# 2024-04-29T14:21:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8cf7b6:::benchmark_data_smithi194_103039_object116052:head# 2024-04-29T14:21:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8d7967:::benchmark_data_smithi194_103039_object143647:head# 2024-04-29T14:21:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8e2b2f:::benchmark_data_smithi194_103039_object146129:head# 2024-04-29T14:21:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b8fa27d:::benchmark_data_smithi194_103039_object94940:head# 2024-04-29T14:21:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9028dd:::benchmark_data_smithi194_103039_object125719:head# 2024-04-29T14:21:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b90948d:::benchmark_data_smithi194_103039_object152810:head# 2024-04-29T14:21:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9161f8:::benchmark_data_smithi194_103039_object115331:head# 2024-04-29T14:21:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b92362d:::benchmark_data_smithi194_103039_object72262:head# 2024-04-29T14:21:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9244a9:::benchmark_data_smithi194_103039_object80690:head# 2024-04-29T14:21:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b932f28:::benchmark_data_smithi194_103039_object59412:head# 2024-04-29T14:21:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b93c66c:::benchmark_data_smithi194_103039_object109766:head# 2024-04-29T14:21:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b93ee0b:::benchmark_data_smithi194_103039_object118959:head# 2024-04-29T14:21:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b94397b:::benchmark_data_smithi194_103039_object150784:head# 2024-04-29T14:21:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9489fe:::benchmark_data_smithi194_103039_object138893:head# 2024-04-29T14:21:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b952865:::benchmark_data_smithi194_103039_object146623:head# 2024-04-29T14:21:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:18.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:18.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b953bbb:::benchmark_data_smithi194_103039_object41646:head# 2024-04-29T14:21:18.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9640d1:::benchmark_data_smithi194_103039_object49175:head# 2024-04-29T14:21:18.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b96f1fc:::benchmark_data_smithi194_103039_object89213:head# 2024-04-29T14:21:18.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b96f807:::benchmark_data_smithi194_103039_object103033:head# 2024-04-29T14:21:18.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9757e1:::benchmark_data_smithi194_103039_object55287:head# 2024-04-29T14:21:18.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b97ac5f:::benchmark_data_smithi194_103039_object114881:head# 2024-04-29T14:21:18.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9878dd:::benchmark_data_smithi194_103039_object59193:head# 2024-04-29T14:21:18.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b988880:::benchmark_data_smithi194_103039_object123062:head# 2024-04-29T14:21:18.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b98d2f9:::benchmark_data_smithi194_103039_object137796:head# 2024-04-29T14:21:18.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b99a832:::benchmark_data_smithi194_103039_object51040:head# 2024-04-29T14:21:18.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b99a913:::benchmark_data_smithi194_103039_object83832:head# 2024-04-29T14:21:18.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b99b8fc:::benchmark_data_smithi194_103039_object78916:head# 2024-04-29T14:21:18.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b99ef53:::benchmark_data_smithi194_103039_object102157:head# 2024-04-29T14:21:18.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9a5f18:::benchmark_data_smithi194_103039_object66442:head# 2024-04-29T14:21:18.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9aa78a:::benchmark_data_smithi194_103039_object95519:head# 2024-04-29T14:21:18.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9b3f2b:::benchmark_data_smithi194_103039_object50436:head# 2024-04-29T14:21:18.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9b4016:::benchmark_data_smithi194_103039_object134041:head# 2024-04-29T14:21:18.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9c66c4:::benchmark_data_smithi194_103039_object123215:head# 2024-04-29T14:21:18.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9cc155:::benchmark_data_smithi194_103039_object102392:head# 2024-04-29T14:21:18.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9cde57:::benchmark_data_smithi194_103039_object113875:head# 2024-04-29T14:21:18.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9e217a:::benchmark_data_smithi194_103039_object154421:head# 2024-04-29T14:21:18.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9e2cf1:::benchmark_data_smithi194_103039_object121764:head# 2024-04-29T14:21:18.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9ea567:::benchmark_data_smithi194_103039_object44766:head# 2024-04-29T14:21:18.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9edeef:::benchmark_data_smithi194_103039_object120071:head# 2024-04-29T14:21:18.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9f3165:::benchmark_data_smithi194_103039_object137867:head# 2024-04-29T14:21:18.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4b9fc3c6:::benchmark_data_smithi194_103039_object56566:head# 2024-04-29T14:21:18.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba01f1b:::benchmark_data_smithi194_103039_object74530:head# 2024-04-29T14:21:18.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba2f9b2:::benchmark_data_smithi194_103039_object105637:head# 2024-04-29T14:21:18.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba2fc7c:::benchmark_data_smithi194_103039_object109963:head# 2024-04-29T14:21:18.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba32c59:::benchmark_data_smithi194_103039_object107057:head# 2024-04-29T14:21:18.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba38533:::benchmark_data_smithi194_103039_object72173:head# 2024-04-29T14:21:18.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba42cd8:::benchmark_data_smithi194_103039_object103288:head# 2024-04-29T14:21:18.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba45587:::benchmark_data_smithi194_103039_object156604:head# 2024-04-29T14:21:18.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba50449:::benchmark_data_smithi194_103039_object93965:head# 2024-04-29T14:21:18.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba5192e:::benchmark_data_smithi194_103039_object152200:head# 2024-04-29T14:21:18.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba5c665:::benchmark_data_smithi194_103039_object103098:head# 2024-04-29T14:21:18.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba7321a:::benchmark_data_smithi194_103039_object121338:head# 2024-04-29T14:21:18.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba73e86:::benchmark_data_smithi194_103039_object117504:head# 2024-04-29T14:21:18.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba8a019:::benchmark_data_smithi194_103039_object91534:head# 2024-04-29T14:21:18.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba95f6e:::benchmark_data_smithi194_103039_object104608:head# 2024-04-29T14:21:18.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ba97355:::benchmark_data_smithi194_103039_object146879:head# 2024-04-29T14:21:18.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4baa14b1:::benchmark_data_smithi194_103039_object146919:head# 2024-04-29T14:21:18.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4baa2688:::benchmark_data_smithi194_103039_object56130:head# 2024-04-29T14:21:18.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4baaca09:::benchmark_data_smithi194_103039_object72138:head# 2024-04-29T14:21:18.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4baad9d0:::benchmark_data_smithi194_103039_object146912:head# 2024-04-29T14:21:18.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bab1f7d:::benchmark_data_smithi194_103039_object142346:head# 2024-04-29T14:21:18.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bac61af:::benchmark_data_smithi194_103039_object60081:head# 2024-04-29T14:21:18.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bada45b:::benchmark_data_smithi194_103039_object85078:head# 2024-04-29T14:21:18.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4baff3bf:::benchmark_data_smithi194_103039_object65289:head# 2024-04-29T14:21:18.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb04328:::benchmark_data_smithi194_103039_object127362:head# 2024-04-29T14:21:18.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb26a75:::benchmark_data_smithi194_103039_object59370:head# 2024-04-29T14:21:18.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb2a1e6:::benchmark_data_smithi194_103039_object153903:head# 2024-04-29T14:21:18.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb3354a:::benchmark_data_smithi194_103039_object149145:head# 2024-04-29T14:21:18.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb339f3:::benchmark_data_smithi194_103039_object74632:head# 2024-04-29T14:21:18.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb36ff6:::benchmark_data_smithi194_103039_object117096:head# 2024-04-29T14:21:18.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb44733:::benchmark_data_smithi194_103039_object104861:head# 2024-04-29T14:21:18.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb4868c:::benchmark_data_smithi194_103039_object85998:head# 2024-04-29T14:21:18.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb7047d:::benchmark_data_smithi194_103039_object56263:head# 2024-04-29T14:21:18.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb72b67:::benchmark_data_smithi194_103039_object128244:head# 2024-04-29T14:21:18.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb8e940:::benchmark_data_smithi194_103039_object149744:head# 2024-04-29T14:21:18.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb9238c:::benchmark_data_smithi194_103039_object121492:head# 2024-04-29T14:21:18.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bb95821:::benchmark_data_smithi194_103039_object105438:head# 2024-04-29T14:21:18.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bba436c:::benchmark_data_smithi194_103039_object43597:head# 2024-04-29T14:21:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbaebef:::benchmark_data_smithi194_103039_object44084:head# 2024-04-29T14:21:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbaf1dc:::benchmark_data_smithi194_103039_object123020:head# 2024-04-29T14:21:18.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbb67da:::benchmark_data_smithi194_103039_object61724:head# 2024-04-29T14:21:18.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbbc5f9:::benchmark_data_smithi194_103039_object55829:head# 2024-04-29T14:21:18.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbc5b61:::benchmark_data_smithi194_103039_object135718:head# 2024-04-29T14:21:18.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbc8a81:::benchmark_data_smithi194_103039_object102780:head# 2024-04-29T14:21:18.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbcafe8:::benchmark_data_smithi194_103039_object92582:head# 2024-04-29T14:21:18.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbcb346:::benchmark_data_smithi194_103039_object102127:head# 2024-04-29T14:21:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbd0e1a:::benchmark_data_smithi194_103039_object69535:head# 2024-04-29T14:21:18.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbd1c7e:::benchmark_data_smithi194_103039_object73157:head# 2024-04-29T14:21:18.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbd1f22:::benchmark_data_smithi194_103039_object97240:head# 2024-04-29T14:21:18.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbd8939:::benchmark_data_smithi194_103039_object61618:head# 2024-04-29T14:21:18.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbe097b:::benchmark_data_smithi194_103039_object73310:head# 2024-04-29T14:21:18.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbe8db8:::benchmark_data_smithi194_103039_object89344:head# 2024-04-29T14:21:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbec6f8:::benchmark_data_smithi194_103039_object140049:head# 2024-04-29T14:21:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbf019d:::benchmark_data_smithi194_103039_object125186:head# 2024-04-29T14:21:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bbf80ed:::benchmark_data_smithi194_103039_object108638:head# 2024-04-29T14:21:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc06960:::benchmark_data_smithi194_103039_object71446:head# 2024-04-29T14:21:18.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc1b29a:::benchmark_data_smithi194_103039_object92052:head# 2024-04-29T14:21:18.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc1ff0e:::benchmark_data_smithi194_103039_object64143:head# 2024-04-29T14:21:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc23410:::benchmark_data_smithi194_103039_object149481:head# 2024-04-29T14:21:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc2565e:::benchmark_data_smithi194_103039_object128542:head# 2024-04-29T14:21:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc271e5:::benchmark_data_smithi194_103039_object156035:head# 2024-04-29T14:21:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc2a3b1:::benchmark_data_smithi194_103039_object86081:head# 2024-04-29T14:21:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc31a10:::benchmark_data_smithi194_103039_object60010:head# 2024-04-29T14:21:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc327cf:::benchmark_data_smithi194_103039_object131756:head# 2024-04-29T14:21:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc3d960:::benchmark_data_smithi194_103039_object152099:head# 2024-04-29T14:21:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc3e802:::benchmark_data_smithi194_103039_object99983:head# 2024-04-29T14:21:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc44832:::benchmark_data_smithi194_103039_object51540:head# 2024-04-29T14:21:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc4508d:::benchmark_data_smithi194_103039_object103901:head# 2024-04-29T14:21:18.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc51d14:::benchmark_data_smithi194_103039_object69108:head# 2024-04-29T14:21:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc57256:::benchmark_data_smithi194_103039_object121298:head# 2024-04-29T14:21:18.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc574a4:::benchmark_data_smithi194_103039_object129355:head# 2024-04-29T14:21:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc63f57:::benchmark_data_smithi194_103039_object78655:head# 2024-04-29T14:21:18.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc6a833:::benchmark_data_smithi194_103039_object57207:head# 2024-04-29T14:21:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc7fda5:::benchmark_data_smithi194_103039_object92468:head# 2024-04-29T14:21:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bc9dcde:::benchmark_data_smithi194_103039_object154608:head# 2024-04-29T14:21:18.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bcb8f55:::benchmark_data_smithi194_103039_object94730:head# 2024-04-29T14:21:18.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bcd2db3:::benchmark_data_smithi194_103039_object67502:head# 2024-04-29T14:21:18.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bcd3fc3:::benchmark_data_smithi194_103039_object108601:head# 2024-04-29T14:21:18.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bcdd477:::benchmark_data_smithi194_103039_object106661:head# 2024-04-29T14:21:18.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bcde77e:::benchmark_data_smithi194_103039_object58496:head# 2024-04-29T14:21:18.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bce8717:::benchmark_data_smithi194_103039_object124793:head# 2024-04-29T14:21:18.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bcfc745:::benchmark_data_smithi194_103039_object99997:head# 2024-04-29T14:21:18.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd06545:::benchmark_data_smithi194_103039_object105080:head# 2024-04-29T14:21:18.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd07158:::benchmark_data_smithi194_103039_object93506:head# 2024-04-29T14:21:18.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd0b8ec:::benchmark_data_smithi194_103039_object153443:head# 2024-04-29T14:21:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd0fbaf:::benchmark_data_smithi194_103039_object108115:head# 2024-04-29T14:21:18.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd17171:::benchmark_data_smithi194_103039_object55368:head# 2024-04-29T14:21:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd288ee:::benchmark_data_smithi194_103039_object147106:head# 2024-04-29T14:21:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd2c024:::benchmark_data_smithi194_103039_object82662:head# 2024-04-29T14:21:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd3203b:::benchmark_data_smithi194_103039_object147605:head# 2024-04-29T14:21:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd45a37:::benchmark_data_smithi194_103039_object141866:head# 2024-04-29T14:21:18.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd476d1:::benchmark_data_smithi194_103039_object111779:head# 2024-04-29T14:21:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd48ec9:::benchmark_data_smithi194_103039_object152522:head# 2024-04-29T14:21:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd52e9d:::benchmark_data_smithi194_103039_object101157:head# 2024-04-29T14:21:18.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd5b477:::benchmark_data_smithi194_103039_object67459:head# 2024-04-29T14:21:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd5f692:::benchmark_data_smithi194_103039_object140258:head# 2024-04-29T14:21:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd64425:::benchmark_data_smithi194_103039_object122301:head# 2024-04-29T14:21:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd66bcf:::benchmark_data_smithi194_103039_object79573:head# 2024-04-29T14:21:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd80139:::benchmark_data_smithi194_103039_object41627:head# 2024-04-29T14:21:18.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd815dd:::benchmark_data_smithi194_103039_object145451:head# 2024-04-29T14:21:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd83aaa:::benchmark_data_smithi194_103039_object152053:head# 2024-04-29T14:21:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd84ba2:::benchmark_data_smithi194_103039_object150012:head# 2024-04-29T14:21:18.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd9156d:::benchmark_data_smithi194_103039_object109779:head# 2024-04-29T14:21:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bd9aa23:::benchmark_data_smithi194_103039_object122798:head# 2024-04-29T14:21:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdab647:::benchmark_data_smithi194_103039_object125202:head# 2024-04-29T14:21:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdb2235:::benchmark_data_smithi194_103039_object55751:head# 2024-04-29T14:21:18.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdb423e:::benchmark_data_smithi194_103039_object47121:head# 2024-04-29T14:21:18.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdbaa5b:::benchmark_data_smithi194_103039_object107288:head# 2024-04-29T14:21:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdc36fe:::benchmark_data_smithi194_103039_object64557:head# 2024-04-29T14:21:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdc8216:::benchmark_data_smithi194_103039_object155776:head# 2024-04-29T14:21:18.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdc83b3:::benchmark_data_smithi194_103039_object145167:head# 2024-04-29T14:21:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdcaff1:::benchmark_data_smithi194_103039_object114043:head# 2024-04-29T14:21:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdd2b45:::benchmark_data_smithi194_103039_object74485:head# 2024-04-29T14:21:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bddd46d:::benchmark_data_smithi194_103039_object87842:head# 2024-04-29T14:21:18.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bdf133d:::benchmark_data_smithi194_103039_object151723:head# 2024-04-29T14:21:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be00d67:::benchmark_data_smithi194_103039_object79619:head# 2024-04-29T14:21:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be02d59:::benchmark_data_smithi194_103039_object98797:head# 2024-04-29T14:21:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be033ef:::benchmark_data_smithi194_103039_object123145:head# 2024-04-29T14:21:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be0db2d:::benchmark_data_smithi194_103039_object121736:head# 2024-04-29T14:21:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be0e4b5:::benchmark_data_smithi194_103039_object99773:head# 2024-04-29T14:21:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be1f5ee:::benchmark_data_smithi194_103039_object119114:head# 2024-04-29T14:21:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be3034f:::benchmark_data_smithi194_103039_object81145:head# 2024-04-29T14:21:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be4c745:::benchmark_data_smithi194_103039_object140765:head# 2024-04-29T14:21:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be4f358:::benchmark_data_smithi194_103039_object84440:head# 2024-04-29T14:21:18.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be56edd:::benchmark_data_smithi194_103039_object131888:head# 2024-04-29T14:21:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be5dcce:::benchmark_data_smithi194_103039_object92942:head# 2024-04-29T14:21:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be65517:::benchmark_data_smithi194_103039_object103035:head# 2024-04-29T14:21:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be6ea17:::benchmark_data_smithi194_103039_object152360:head# 2024-04-29T14:21:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be70b16:::benchmark_data_smithi194_103039_object90109:head# 2024-04-29T14:21:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be7270a:::benchmark_data_smithi194_103039_object125637:head# 2024-04-29T14:21:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be82fdb:::benchmark_data_smithi194_103039_object74834:head# 2024-04-29T14:21:18.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be83426:::benchmark_data_smithi194_103039_object139162:head# 2024-04-29T14:21:18.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4be89c62:::benchmark_data_smithi194_103039_object107504:head# 2024-04-29T14:21:18.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4beb4bdb:::benchmark_data_smithi194_103039_object53975:head# 2024-04-29T14:21:18.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bec1170:::benchmark_data_smithi194_103039_object108181:head# 2024-04-29T14:21:18.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bec415b:::benchmark_data_smithi194_103039_object141636:head# 2024-04-29T14:21:18.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4becbde9:::benchmark_data_smithi194_103039_object55750:head# 2024-04-29T14:21:18.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4becda46:::benchmark_data_smithi194_103039_object74991:head# 2024-04-29T14:21:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4beced7d:::benchmark_data_smithi194_103039_object115719:head# 2024-04-29T14:21:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bedbcbe:::benchmark_data_smithi194_103039_object126084:head# 2024-04-29T14:21:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4beded48:::benchmark_data_smithi194_103039_object48559:head# 2024-04-29T14:21:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bedf4da:::benchmark_data_smithi194_103039_object73774:head# 2024-04-29T14:21:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4beedd9a:::benchmark_data_smithi194_103039_object152588:head# 2024-04-29T14:21:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bef4546:::benchmark_data_smithi194_103039_object42463:head# 2024-04-29T14:21:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4befb1df:::benchmark_data_smithi194_103039_object51606:head# 2024-04-29T14:21:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4beff3d5:::benchmark_data_smithi194_103039_object78794:head# 2024-04-29T14:21:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf068e7:::benchmark_data_smithi194_103039_object83699:head# 2024-04-29T14:21:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf29e32:::benchmark_data_smithi194_103039_object88451:head# 2024-04-29T14:21:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf2d089:::benchmark_data_smithi194_103039_object113078:head# 2024-04-29T14:21:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf2eacc:::benchmark_data_smithi194_103039_object63023:head# 2024-04-29T14:21:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf4a7bf:::benchmark_data_smithi194_103039_object125152:head# 2024-04-29T14:21:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf4ab5a:::benchmark_data_smithi194_103039_object123847:head# 2024-04-29T14:21:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf52f8b:::benchmark_data_smithi194_103039_object129081:head# 2024-04-29T14:21:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf5c9f0:::benchmark_data_smithi194_103039_object72326:head# 2024-04-29T14:21:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf5d8e4:::benchmark_data_smithi194_103039_object124701:head# 2024-04-29T14:21:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf68d10:::benchmark_data_smithi194_103039_object65599:head# 2024-04-29T14:21:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf6acbf:::benchmark_data_smithi194_103039_object56491:head# 2024-04-29T14:21:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf73a1e:::benchmark_data_smithi194_103039_object62127:head# 2024-04-29T14:21:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf7b80f:::benchmark_data_smithi194_103039_object41546:head# 2024-04-29T14:21:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bf9f9c5:::benchmark_data_smithi194_103039_object103411:head# 2024-04-29T14:21:18.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bfa10d8:::benchmark_data_smithi194_103039_object47580:head# 2024-04-29T14:21:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bfac813:::benchmark_data_smithi194_103039_object117783:head# 2024-04-29T14:21:18.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bfb447e:::benchmark_data_smithi194_103039_object91825:head# 2024-04-29T14:21:18.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bfb50cb:::benchmark_data_smithi194_103039_object95700:head# 2024-04-29T14:21:18.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bfd061b:::benchmark_data_smithi194_103039_object43519:head# 2024-04-29T14:21:18.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bfd5352:::benchmark_data_smithi194_103039_object47679:head# 2024-04-29T14:21:18.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bfedd64:::benchmark_data_smithi194_103039_object61483:head# 2024-04-29T14:21:18.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4bffb2a8:::benchmark_data_smithi194_103039_object58482:head# 2024-04-29T14:21:18.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c006ad9:::benchmark_data_smithi194_103039_object87545:head# 2024-04-29T14:21:18.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c01c9eb:::benchmark_data_smithi194_103039_object96281:head# 2024-04-29T14:21:18.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c02989a:::benchmark_data_smithi194_103039_object71928:head# 2024-04-29T14:21:18.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c02991b:::benchmark_data_smithi194_103039_object57245:head# 2024-04-29T14:21:18.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c03957c:::benchmark_data_smithi194_103039_object104574:head# 2024-04-29T14:21:18.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c03cd5d:::benchmark_data_smithi194_103039_object59746:head# 2024-04-29T14:21:18.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c04152f:::benchmark_data_smithi194_103039_object140465:head# 2024-04-29T14:21:18.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0418d4:::benchmark_data_smithi194_103039_object102800:head# 2024-04-29T14:21:18.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c04247f:::benchmark_data_smithi194_103039_object105867:head# 2024-04-29T14:21:18.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c050362:::benchmark_data_smithi194_103039_object102798:head# 2024-04-29T14:21:18.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c056eb3:::benchmark_data_smithi194_103039_object106627:head# 2024-04-29T14:21:18.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c057727:::benchmark_data_smithi194_103039_object53989:head# 2024-04-29T14:21:18.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c05ba51:::benchmark_data_smithi194_103039_object122624:head# 2024-04-29T14:21:18.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c05e077:::benchmark_data_smithi194_103039_object73695:head# 2024-04-29T14:21:18.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c069454:::benchmark_data_smithi194_103039_object89998:head# 2024-04-29T14:21:18.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c07615a:::benchmark_data_smithi194_103039_object59835:head# 2024-04-29T14:21:18.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0775b2:::benchmark_data_smithi194_103039_object105964:head# 2024-04-29T14:21:18.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c07a5d9:::benchmark_data_smithi194_103039_object134274:head# 2024-04-29T14:21:18.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c07fd2f:::benchmark_data_smithi194_103039_object63147:head# 2024-04-29T14:21:18.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c083d8b:::benchmark_data_smithi194_103039_object76794:head# 2024-04-29T14:21:18.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c08c50f:::benchmark_data_smithi194_103039_object83683:head# 2024-04-29T14:21:18.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c09ce53:::benchmark_data_smithi194_103039_object138522:head# 2024-04-29T14:21:18.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0bb13e:::benchmark_data_smithi194_103039_object117734:head# 2024-04-29T14:21:18.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0c090d:::benchmark_data_smithi194_103039_object88487:head# 2024-04-29T14:21:18.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0c272c:::benchmark_data_smithi194_103039_object147301:head# 2024-04-29T14:21:18.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0c3741:::benchmark_data_smithi194_103039_object98328:head# 2024-04-29T14:21:18.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0c5dac:::benchmark_data_smithi194_103039_object121848:head# 2024-04-29T14:21:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0c64ae:::benchmark_data_smithi194_103039_object122345:head# 2024-04-29T14:21:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0cac88:::benchmark_data_smithi194_103039_object85505:head# 2024-04-29T14:21:18.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0cff68:::benchmark_data_smithi194_103039_object76863:head# 2024-04-29T14:21:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0d7fd1:::benchmark_data_smithi194_103039_object125124:head# 2024-04-29T14:21:18.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0e7fa5:::benchmark_data_smithi194_103039_object68110:head# 2024-04-29T14:21:18.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0f709e:::benchmark_data_smithi194_103039_object106204:head# 2024-04-29T14:21:18.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c0f8886:::benchmark_data_smithi194_103039_object44038:head# 2024-04-29T14:21:18.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c104a69:::benchmark_data_smithi194_103039_object65979:head# 2024-04-29T14:21:18.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c11f0a5:::benchmark_data_smithi194_103039_object143513:head# 2024-04-29T14:21:18.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c12a103:::benchmark_data_smithi194_103039_object73218:head# 2024-04-29T14:21:18.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c134fab:::benchmark_data_smithi194_103039_object53079:head# 2024-04-29T14:21:18.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c14d8ed:::benchmark_data_smithi194_103039_object67660:head# 2024-04-29T14:21:18.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:18.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c159523:::benchmark_data_smithi194_103039_object148470:head# 2024-04-29T14:21:18.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c15967c:::benchmark_data_smithi194_103039_object112882:head# 2024-04-29T14:21:18.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c15df89:::benchmark_data_smithi194_103039_object108853:head# 2024-04-29T14:21:18.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c16105a:::benchmark_data_smithi194_103039_object62618:head# 2024-04-29T14:21:18.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c163385:::benchmark_data_smithi194_103039_object129797:head# 2024-04-29T14:21:18.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c16c727:::benchmark_data_smithi194_103039_object134988:head# 2024-04-29T14:21:18.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c173e80:::benchmark_data_smithi194_103039_object120027:head# 2024-04-29T14:21:18.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c17954b:::benchmark_data_smithi194_103039_object65538:head# 2024-04-29T14:21:18.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1806f6:::benchmark_data_smithi194_103039_object47105:head# 2024-04-29T14:21:18.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c18dda5:::benchmark_data_smithi194_103039_object46088:head# 2024-04-29T14:21:18.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c18ed10:::benchmark_data_smithi194_103039_object52181:head# 2024-04-29T14:21:18.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c196814:::benchmark_data_smithi194_103039_object59465:head# 2024-04-29T14:21:18.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c19999b:::benchmark_data_smithi194_103039_object72105:head# 2024-04-29T14:21:18.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c19c452:::benchmark_data_smithi194_103039_object153208:head# 2024-04-29T14:21:18.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1b82ca:::benchmark_data_smithi194_103039_object147664:head# 2024-04-29T14:21:18.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1c1d40:::benchmark_data_smithi194_103039_object87306:head# 2024-04-29T14:21:18.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1cbe8d:::benchmark_data_smithi194_103039_object45078:head# 2024-04-29T14:21:18.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1cdd25:::benchmark_data_smithi194_103039_object151378:head# 2024-04-29T14:21:18.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1dc24e:::benchmark_data_smithi194_103039_object137593:head# 2024-04-29T14:21:18.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1de0fe:::benchmark_data_smithi194_103039_object119779:head# 2024-04-29T14:21:18.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1ec8e6:::benchmark_data_smithi194_103039_object58043:head# 2024-04-29T14:21:18.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1fd346:::benchmark_data_smithi194_103039_object92426:head# 2024-04-29T14:21:18.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c1ffbe9:::benchmark_data_smithi194_103039_object67116:head# 2024-04-29T14:21:18.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c20a458:::benchmark_data_smithi194_103039_object146511:head# 2024-04-29T14:21:18.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c21556c:::benchmark_data_smithi194_103039_object147938:head# 2024-04-29T14:21:18.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c219d23:::benchmark_data_smithi194_103039_object68509:head# 2024-04-29T14:21:18.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c219ff1:::benchmark_data_smithi194_103039_object136406:head# 2024-04-29T14:21:18.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c21eb64:::benchmark_data_smithi194_103039_object115356:head# 2024-04-29T14:21:18.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c22e639:::benchmark_data_smithi194_103039_object140655:head# 2024-04-29T14:21:18.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c238f64:::benchmark_data_smithi194_103039_object47211:head# 2024-04-29T14:21:18.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c243ea4:::benchmark_data_smithi194_103039_object106940:head# 2024-04-29T14:21:18.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c24ec4a:::benchmark_data_smithi194_103039_object147473:head# 2024-04-29T14:21:18.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c25c5e1:::benchmark_data_smithi194_103039_object42916:head# 2024-04-29T14:21:18.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c25fa66:::benchmark_data_smithi194_103039_object63467:head# 2024-04-29T14:21:18.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2662b3:::benchmark_data_smithi194_103039_object112707:head# 2024-04-29T14:21:18.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c26844d:::benchmark_data_smithi194_103039_object111709:head# 2024-04-29T14:21:18.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c26c623:::benchmark_data_smithi194_103039_object114563:head# 2024-04-29T14:21:18.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c26f722:::benchmark_data_smithi194_103039_object53430:head# 2024-04-29T14:21:18.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c27b38e:::benchmark_data_smithi194_103039_object90142:head# 2024-04-29T14:21:18.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c27f783:::benchmark_data_smithi194_103039_object132033:head# 2024-04-29T14:21:18.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c290dda:::benchmark_data_smithi194_103039_object106013:head# 2024-04-29T14:21:18.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c290ea2:::benchmark_data_smithi194_103039_object104989:head# 2024-04-29T14:21:18.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c291cb6:::benchmark_data_smithi194_103039_object55052:head# 2024-04-29T14:21:18.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c292ea3:::benchmark_data_smithi194_103039_object147052:head# 2024-04-29T14:21:18.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2aada4:::benchmark_data_smithi194_103039_object70463:head# 2024-04-29T14:21:18.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2be264:::benchmark_data_smithi194_103039_object97890:head# 2024-04-29T14:21:18.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2c031b:::benchmark_data_smithi194_103039_object111328:head# 2024-04-29T14:21:18.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2c99bb:::benchmark_data_smithi194_103039_object128603:head# 2024-04-29T14:21:18.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2cd3de:::benchmark_data_smithi194_103039_object97769:head# 2024-04-29T14:21:18.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2d6d79:::benchmark_data_smithi194_103039_object150110:head# 2024-04-29T14:21:18.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2e120d:::benchmark_data_smithi194_103039_object77992:head# 2024-04-29T14:21:18.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2efa6e:::benchmark_data_smithi194_103039_object114516:head# 2024-04-29T14:21:18.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c2f21ec:::benchmark_data_smithi194_103039_object86070:head# 2024-04-29T14:21:18.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3093e0:::benchmark_data_smithi194_103039_object47918:head# 2024-04-29T14:21:18.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c314ece:::benchmark_data_smithi194_103039_object125118:head# 2024-04-29T14:21:18.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c31fb05:::benchmark_data_smithi194_103039_object63642:head# 2024-04-29T14:21:18.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c32a051:::benchmark_data_smithi194_103039_object42560:head# 2024-04-29T14:21:18.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c32fabc:::benchmark_data_smithi194_103039_object122239:head# 2024-04-29T14:21:18.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3301c2:::benchmark_data_smithi194_103039_object74239:head# 2024-04-29T14:21:18.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c331177:::benchmark_data_smithi194_103039_object103403:head# 2024-04-29T14:21:18.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c33509f:::benchmark_data_smithi194_103039_object148779:head# 2024-04-29T14:21:18.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c341606:::benchmark_data_smithi194_103039_object124428:head# 2024-04-29T14:21:18.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3496d3:::benchmark_data_smithi194_103039_object140706:head# 2024-04-29T14:21:18.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c34f689:::benchmark_data_smithi194_103039_object150273:head# 2024-04-29T14:21:18.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c357117:::benchmark_data_smithi194_103039_object51267:head# 2024-04-29T14:21:18.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c35f99b:::benchmark_data_smithi194_103039_object148750:head# 2024-04-29T14:21:18.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c367a58:::benchmark_data_smithi194_103039_object156282:head# 2024-04-29T14:21:18.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c36cd78:::benchmark_data_smithi194_103039_object63771:head# 2024-04-29T14:21:18.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c36dcca:::benchmark_data_smithi194_103039_object151932:head# 2024-04-29T14:21:18.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3735e2:::benchmark_data_smithi194_103039_object108385:head# 2024-04-29T14:21:18.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c375f3e:::benchmark_data_smithi194_103039_object76145:head# 2024-04-29T14:21:18.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c37a4ec:::benchmark_data_smithi194_103039_object117400:head# 2024-04-29T14:21:18.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3848bb:::benchmark_data_smithi194_103039_object78872:head# 2024-04-29T14:21:18.907 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:18.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c38926c:::benchmark_data_smithi194_103039_object155820:head# 2024-04-29T14:21:18.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c38eaee:::benchmark_data_smithi194_103039_object102252:head# 2024-04-29T14:21:18.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c395f88:::benchmark_data_smithi194_103039_object123775:head# 2024-04-29T14:21:18.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c39868e:::benchmark_data_smithi194_103039_object51686:head# 2024-04-29T14:21:18.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3a9b4a:::benchmark_data_smithi194_103039_object85012:head# 2024-04-29T14:21:18.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3b9343:::benchmark_data_smithi194_103039_object69863:head# 2024-04-29T14:21:18.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3b9cf9:::benchmark_data_smithi194_103039_object145255:head# 2024-04-29T14:21:18.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3bc9c7:::benchmark_data_smithi194_103039_object98147:head# 2024-04-29T14:21:18.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3bc9fd:::benchmark_data_smithi194_103039_object68954:head# 2024-04-29T14:21:18.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3c5b06:::benchmark_data_smithi194_103039_object105114:head# 2024-04-29T14:21:18.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3c9c82:::benchmark_data_smithi194_103039_object93787:head# 2024-04-29T14:21:18.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3ca2a2:::benchmark_data_smithi194_103039_object90352:head# 2024-04-29T14:21:18.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3cd1fe:::benchmark_data_smithi194_103039_object148980:head# 2024-04-29T14:21:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3ce566:::benchmark_data_smithi194_103039_object49749:head# 2024-04-29T14:21:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3e2ec7:::benchmark_data_smithi194_103039_object103941:head# 2024-04-29T14:21:18.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3e7854:::benchmark_data_smithi194_103039_object100847:head# 2024-04-29T14:21:18.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3ecad8:::benchmark_data_smithi194_103039_object154996:head# 2024-04-29T14:21:18.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3ecc0b:::benchmark_data_smithi194_103039_object115104:head# 2024-04-29T14:21:18.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3ef5bf:::benchmark_data_smithi194_103039_object86485:head# 2024-04-29T14:21:18.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c3f508c:::benchmark_data_smithi194_103039_object113689:head# 2024-04-29T14:21:18.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c404850:::benchmark_data_smithi194_103039_object139439:head# 2024-04-29T14:21:18.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c40c0ef:::benchmark_data_smithi194_103039_object127974:head# 2024-04-29T14:21:18.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c41423a:::benchmark_data_smithi194_103039_object56784:head# 2024-04-29T14:21:18.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c415e46:::benchmark_data_smithi194_103039_object123932:head# 2024-04-29T14:21:18.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4198a3:::benchmark_data_smithi194_103039_object64261:head# 2024-04-29T14:21:18.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c419a34:::benchmark_data_smithi194_103039_object97786:head# 2024-04-29T14:21:18.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c41dcf6:::benchmark_data_smithi194_103039_object55880:head# 2024-04-29T14:21:18.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c420c30:::benchmark_data_smithi194_103039_object136601:head# 2024-04-29T14:21:18.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c42353d:::benchmark_data_smithi194_103039_object149721:head# 2024-04-29T14:21:18.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c431d76:::benchmark_data_smithi194_103039_object91598:head# 2024-04-29T14:21:18.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c43576b:::benchmark_data_smithi194_103039_object68661:head# 2024-04-29T14:21:18.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c435e9f:::benchmark_data_smithi194_103039_object131298:head# 2024-04-29T14:21:18.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c44ec9e:::benchmark_data_smithi194_103039_object49906:head# 2024-04-29T14:21:18.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c451372:::benchmark_data_smithi194_103039_object132104:head# 2024-04-29T14:21:18.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c458061:::benchmark_data_smithi194_103039_object149533:head# 2024-04-29T14:21:18.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c468089:::benchmark_data_smithi194_103039_object92760:head# 2024-04-29T14:21:18.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4769f3:::benchmark_data_smithi194_103039_object45732:head# 2024-04-29T14:21:18.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c483c46:::benchmark_data_smithi194_103039_object79790:head# 2024-04-29T14:21:18.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c483c9c:::benchmark_data_smithi194_103039_object47021:head# 2024-04-29T14:21:18.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4858d1:::benchmark_data_smithi194_103039_object100818:head# 2024-04-29T14:21:18.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c488c29:::benchmark_data_smithi194_103039_object115076:head# 2024-04-29T14:21:18.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c488e02:::benchmark_data_smithi194_103039_object117202:head# 2024-04-29T14:21:18.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c491d6b:::benchmark_data_smithi194_103039_object137747:head# 2024-04-29T14:21:18.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c49799d:::benchmark_data_smithi194_103039_object84232:head# 2024-04-29T14:21:18.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c49932b:::benchmark_data_smithi194_103039_object83353:head# 2024-04-29T14:21:18.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4b50bd:::benchmark_data_smithi194_103039_object61408:head# 2024-04-29T14:21:18.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4ba9e8:::benchmark_data_smithi194_103039_object136523:head# 2024-04-29T14:21:18.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4bd338:::benchmark_data_smithi194_103039_object111676:head# 2024-04-29T14:21:18.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4c1de7:::benchmark_data_smithi194_103039_object53581:head# 2024-04-29T14:21:18.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4c623d:::benchmark_data_smithi194_103039_object75110:head# 2024-04-29T14:21:18.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4e3eef:::benchmark_data_smithi194_103039_object109734:head# 2024-04-29T14:21:18.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4e6f22:::benchmark_data_smithi194_103039_object129872:head# 2024-04-29T14:21:18.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4ea04e:::benchmark_data_smithi194_103039_object56944:head# 2024-04-29T14:21:18.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4ee9b3:::benchmark_data_smithi194_103039_object107489:head# 2024-04-29T14:21:18.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4f25a0:::benchmark_data_smithi194_103039_object44429:head# 2024-04-29T14:21:18.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4f545a:::benchmark_data_smithi194_103039_object62368:head# 2024-04-29T14:21:18.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c4f66f4:::benchmark_data_smithi194_103039_object121412:head# 2024-04-29T14:21:18.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c510765:::benchmark_data_smithi194_103039_object45659:head# 2024-04-29T14:21:18.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c514e5d:::benchmark_data_smithi194_103039_object148084:head# 2024-04-29T14:21:18.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c519aaa:::benchmark_data_smithi194_103039_object136208:head# 2024-04-29T14:21:18.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c51adfe:::benchmark_data_smithi194_103039_object47455:head# 2024-04-29T14:21:18.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c523537:::benchmark_data_smithi194_103039_object44664:head# 2024-04-29T14:21:18.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c53776c:::benchmark_data_smithi194_103039_object83937:head# 2024-04-29T14:21:18.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c53ce0a:::benchmark_data_smithi194_103039_object146489:head# 2024-04-29T14:21:18.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5550b5:::benchmark_data_smithi194_103039_object66137:head# 2024-04-29T14:21:18.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5586da:::benchmark_data_smithi194_103039_object143544:head# 2024-04-29T14:21:18.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c565a1d:::benchmark_data_smithi194_103039_object118056:head# 2024-04-29T14:21:18.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c56cbfe:::benchmark_data_smithi194_103039_object138024:head# 2024-04-29T14:21:18.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c56fbc3:::benchmark_data_smithi194_103039_object47023:head# 2024-04-29T14:21:18.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c57475d:::benchmark_data_smithi194_103039_object72297:head# 2024-04-29T14:21:18.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c57ddfd:::benchmark_data_smithi194_103039_object63673:head# 2024-04-29T14:21:18.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c584857:::benchmark_data_smithi194_103039_object44419:head# 2024-04-29T14:21:18.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c58549d:::benchmark_data_smithi194_103039_object44034:head# 2024-04-29T14:21:18.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5a37ef:::benchmark_data_smithi194_103039_object71123:head# 2024-04-29T14:21:18.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5aff48:::benchmark_data_smithi194_103039_object99293:head# 2024-04-29T14:21:18.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5b376a:::benchmark_data_smithi194_103039_object122034:head# 2024-04-29T14:21:18.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5b4ead:::benchmark_data_smithi194_103039_object45465:head# 2024-04-29T14:21:18.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5b8bf0:::benchmark_data_smithi194_103039_object52802:head# 2024-04-29T14:21:18.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5c0bfa:::benchmark_data_smithi194_103039_object149413:head# 2024-04-29T14:21:18.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5c6ebb:::benchmark_data_smithi194_103039_object151446:head# 2024-04-29T14:21:18.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5dd600:::benchmark_data_smithi194_103039_object70836:head# 2024-04-29T14:21:18.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5ec56d:::benchmark_data_smithi194_103039_object48702:head# 2024-04-29T14:21:18.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5f3cbb:::benchmark_data_smithi194_103039_object93189:head# 2024-04-29T14:21:18.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c5fd14e:::benchmark_data_smithi194_103039_object141429:head# 2024-04-29T14:21:18.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6067e4:::benchmark_data_smithi194_103039_object155499:head# 2024-04-29T14:21:18.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c60e73f:::benchmark_data_smithi194_103039_object149523:head# 2024-04-29T14:21:18.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c615ab1:::benchmark_data_smithi194_103039_object52380:head# 2024-04-29T14:21:18.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c62ef5f:::benchmark_data_smithi194_103039_object98093:head# 2024-04-29T14:21:18.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c633afd:::benchmark_data_smithi194_103039_object76549:head# 2024-04-29T14:21:18.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c64bbde:::benchmark_data_smithi194_103039_object126237:head# 2024-04-29T14:21:18.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c65039e:::benchmark_data_smithi194_103039_object124287:head# 2024-04-29T14:21:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c65612d:::benchmark_data_smithi194_103039_object118737:head# 2024-04-29T14:21:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6584e2:::benchmark_data_smithi194_103039_object59628:head# 2024-04-29T14:21:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6738a5:::benchmark_data_smithi194_103039_object142393:head# 2024-04-29T14:21:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c679388:::benchmark_data_smithi194_103039_object142184:head# 2024-04-29T14:21:18.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c67940e:::benchmark_data_smithi194_103039_object52940:head# 2024-04-29T14:21:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c67a87b:::benchmark_data_smithi194_103039_object81193:head# 2024-04-29T14:21:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c684bea:::benchmark_data_smithi194_103039_object96650:head# 2024-04-29T14:21:18.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6867f5:::benchmark_data_smithi194_103039_object144637:head# 2024-04-29T14:21:18.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c68953e:::benchmark_data_smithi194_103039_object132478:head# 2024-04-29T14:21:18.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6943e7:::benchmark_data_smithi194_103039_object129723:head# 2024-04-29T14:21:18.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c695999:::benchmark_data_smithi194_103039_object107112:head# 2024-04-29T14:21:18.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6a2798:::benchmark_data_smithi194_103039_object88885:head# 2024-04-29T14:21:18.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6a532a:::benchmark_data_smithi194_103039_object46852:head# 2024-04-29T14:21:18.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6ae2eb:::benchmark_data_smithi194_103039_object154430:head# 2024-04-29T14:21:18.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6c2e64:::benchmark_data_smithi194_103039_object72710:head# 2024-04-29T14:21:18.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6c5cef:::benchmark_data_smithi194_103039_object152503:head# 2024-04-29T14:21:18.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6d71ca:::benchmark_data_smithi194_103039_object56856:head# 2024-04-29T14:21:18.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6dfab1:::benchmark_data_smithi194_103039_object44680:head# 2024-04-29T14:21:18.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6e0de1:::benchmark_data_smithi194_103039_object51980:head# 2024-04-29T14:21:18.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6e63f4:::benchmark_data_smithi194_103039_object130952:head# 2024-04-29T14:21:18.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c6ed195:::benchmark_data_smithi194_103039_object143296:head# 2024-04-29T14:21:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c702155:::benchmark_data_smithi194_103039_object124607:head# 2024-04-29T14:21:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c70e349:::benchmark_data_smithi194_103039_object74346:head# 2024-04-29T14:21:18.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c715cac:::benchmark_data_smithi194_103039_object107129:head# 2024-04-29T14:21:18.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7178ae:::benchmark_data_smithi194_103039_object69602:head# 2024-04-29T14:21:18.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c718fa6:::benchmark_data_smithi194_103039_object54039:head# 2024-04-29T14:21:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c719eba:::benchmark_data_smithi194_103039_object140242:head# 2024-04-29T14:21:18.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c71aad0:::benchmark_data_smithi194_103039_object150498:head# 2024-04-29T14:21:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7276a1:::benchmark_data_smithi194_103039_object119718:head# 2024-04-29T14:21:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c74f738:::benchmark_data_smithi194_103039_object41902:head# 2024-04-29T14:21:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c752503:::benchmark_data_smithi194_103039_object132116:head# 2024-04-29T14:21:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c75df4a:::benchmark_data_smithi194_103039_object89762:head# 2024-04-29T14:21:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c75e99c:::benchmark_data_smithi194_103039_object57826:head# 2024-04-29T14:21:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c76929f:::benchmark_data_smithi194_103039_object144311:head# 2024-04-29T14:21:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7698e4:::benchmark_data_smithi194_103039_object113378:head# 2024-04-29T14:21:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c76d875:::benchmark_data_smithi194_103039_object131921:head# 2024-04-29T14:21:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c76f134:::benchmark_data_smithi194_103039_object109523:head# 2024-04-29T14:21:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7741c1:::benchmark_data_smithi194_103039_object128961:head# 2024-04-29T14:21:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c775e72:::benchmark_data_smithi194_103039_object55609:head# 2024-04-29T14:21:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c77cd33:::benchmark_data_smithi194_103039_object75919:head# 2024-04-29T14:21:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c79261c:::benchmark_data_smithi194_103039_object152040:head# 2024-04-29T14:21:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7a0f98:::benchmark_data_smithi194_103039_object133290:head# 2024-04-29T14:21:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7a3d41:::benchmark_data_smithi194_103039_object137694:head# 2024-04-29T14:21:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7a4e37:::benchmark_data_smithi194_103039_object103292:head# 2024-04-29T14:21:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7b05ae:::benchmark_data_smithi194_103039_object93534:head# 2024-04-29T14:21:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7d44eb:::benchmark_data_smithi194_103039_object70678:head# 2024-04-29T14:21:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7d5852:::benchmark_data_smithi194_103039_object150856:head# 2024-04-29T14:21:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7e1964:::benchmark_data_smithi194_103039_object128949:head# 2024-04-29T14:21:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7e9d63:::benchmark_data_smithi194_103039_object48256:head# 2024-04-29T14:21:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7eedd1:::benchmark_data_smithi194_103039_object122786:head# 2024-04-29T14:21:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7f273f:::benchmark_data_smithi194_103039_object136996:head# 2024-04-29T14:21:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7f7178:::benchmark_data_smithi194_103039_object102031:head# 2024-04-29T14:21:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c7fb4a7:::benchmark_data_smithi194_103039_object148851:head# 2024-04-29T14:21:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c81bf42:::benchmark_data_smithi194_103039_object148437:head# 2024-04-29T14:21:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8280c0:::benchmark_data_smithi194_103039_object75291:head# 2024-04-29T14:21:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8389a3:::benchmark_data_smithi194_103039_object68318:head# 2024-04-29T14:21:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c83b99d:::benchmark_data_smithi194_103039_object120130:head# 2024-04-29T14:21:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c841be3:::benchmark_data_smithi194_103039_object149136:head# 2024-04-29T14:21:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8627be:::benchmark_data_smithi194_103039_object42218:head# 2024-04-29T14:21:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c86664a:::benchmark_data_smithi194_103039_object57851:head# 2024-04-29T14:21:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c86a684:::benchmark_data_smithi194_103039_object82182:head# 2024-04-29T14:21:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c86c9ef:::benchmark_data_smithi194_103039_object143770:head# 2024-04-29T14:21:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c872ac8:::benchmark_data_smithi194_103039_object153804:head# 2024-04-29T14:21:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c876272:::benchmark_data_smithi194_103039_object64141:head# 2024-04-29T14:21:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c876bc2:::benchmark_data_smithi194_103039_object77690:head# 2024-04-29T14:21:18.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c881375:::benchmark_data_smithi194_103039_object90896:head# 2024-04-29T14:21:18.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c88fca0:::benchmark_data_smithi194_103039_object65363:head# 2024-04-29T14:21:18.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c892ed9:::benchmark_data_smithi194_103039_object128760:head# 2024-04-29T14:21:18.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c896193:::benchmark_data_smithi194_103039_object67297:head# 2024-04-29T14:21:18.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8991e0:::benchmark_data_smithi194_103039_object106693:head# 2024-04-29T14:21:18.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8a5de0:::benchmark_data_smithi194_103039_object90706:head# 2024-04-29T14:21:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8a5fb1:::benchmark_data_smithi194_103039_object53336:head# 2024-04-29T14:21:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8a6512:::benchmark_data_smithi194_103039_object155529:head# 2024-04-29T14:21:18.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8b7c0b:::benchmark_data_smithi194_103039_object142777:head# 2024-04-29T14:21:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8b9de9:::benchmark_data_smithi194_103039_object41587:head# 2024-04-29T14:21:18.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8bc2cc:::benchmark_data_smithi194_103039_object41608:head# 2024-04-29T14:21:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8be08d:::benchmark_data_smithi194_103039_object75010:head# 2024-04-29T14:21:18.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8be612:::benchmark_data_smithi194_103039_object78466:head# 2024-04-29T14:21:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8c0f49:::benchmark_data_smithi194_103039_object84678:head# 2024-04-29T14:21:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8c16d8:::benchmark_data_smithi194_103039_object111405:head# 2024-04-29T14:21:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8c41bd:::benchmark_data_smithi194_103039_object125523:head# 2024-04-29T14:21:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8c51c0:::benchmark_data_smithi194_103039_object152115:head# 2024-04-29T14:21:18.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8c8040:::benchmark_data_smithi194_103039_object131302:head# 2024-04-29T14:21:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8c83c7:::benchmark_data_smithi194_103039_object136675:head# 2024-04-29T14:21:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8cc4d8:::benchmark_data_smithi194_103039_object142260:head# 2024-04-29T14:21:18.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8cf7d4:::benchmark_data_smithi194_103039_object66093:head# 2024-04-29T14:21:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8d4f0f:::benchmark_data_smithi194_103039_object74126:head# 2024-04-29T14:21:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8db586:::benchmark_data_smithi194_103039_object55459:head# 2024-04-29T14:21:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8dc9d1:::benchmark_data_smithi194_103039_object144089:head# 2024-04-29T14:21:18.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8e56de:::benchmark_data_smithi194_103039_object49644:head# 2024-04-29T14:21:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8e8c24:::benchmark_data_smithi194_103039_object61524:head# 2024-04-29T14:21:18.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8f80d8:::benchmark_data_smithi194_103039_object123718:head# 2024-04-29T14:21:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c8fa3b9:::benchmark_data_smithi194_103039_object71748:head# 2024-04-29T14:21:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c916a52:::benchmark_data_smithi194_103039_object82644:head# 2024-04-29T14:21:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c924a19:::benchmark_data_smithi194_103039_object139197:head# 2024-04-29T14:21:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c925d76:::benchmark_data_smithi194_103039_object125271:head# 2024-04-29T14:21:18.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c925dd3:::benchmark_data_smithi194_103039_object42221:head# 2024-04-29T14:21:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9330bf:::benchmark_data_smithi194_103039_object47086:head# 2024-04-29T14:21:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c93defb:::benchmark_data_smithi194_103039_object42273:head# 2024-04-29T14:21:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c94788b:::benchmark_data_smithi194_103039_object111018:head# 2024-04-29T14:21:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9492d6:::benchmark_data_smithi194_103039_object127762:head# 2024-04-29T14:21:18.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c951f11:::benchmark_data_smithi194_103039_object152952:head# 2024-04-29T14:21:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9536d8:::benchmark_data_smithi194_103039_object103391:head# 2024-04-29T14:21:18.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9646b1:::benchmark_data_smithi194_103039_object107431:head# 2024-04-29T14:21:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c985580:::benchmark_data_smithi194_103039_object84148:head# 2024-04-29T14:21:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c985b25:::benchmark_data_smithi194_103039_object80465:head# 2024-04-29T14:21:18.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9a264f:::benchmark_data_smithi194_103039_object116899:head# 2024-04-29T14:21:18.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9a8bb8:::benchmark_data_smithi194_103039_object100503:head# 2024-04-29T14:21:18.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9a8d19:::benchmark_data_smithi194_103039_object89225:head# 2024-04-29T14:21:18.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9b74a6:::benchmark_data_smithi194_103039_object49588:head# 2024-04-29T14:21:18.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9bf20f:::benchmark_data_smithi194_103039_object124846:head# 2024-04-29T14:21:18.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9cc9a0:::benchmark_data_smithi194_103039_object81239:head# 2024-04-29T14:21:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4c9cca4e:::benchmark_data_smithi194_103039_object51112:head# 2024-04-29T14:21:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca04cb7:::benchmark_data_smithi194_103039_object49820:head# 2024-04-29T14:21:18.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca15fc3:::benchmark_data_smithi194_103039_object54504:head# 2024-04-29T14:21:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca2046e:::benchmark_data_smithi194_103039_object113214:head# 2024-04-29T14:21:18.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca26055:::benchmark_data_smithi194_103039_object118320:head# 2024-04-29T14:21:19.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca2ae38:::benchmark_data_smithi194_103039_object142357:head# 2024-04-29T14:21:19.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca2fcf7:::benchmark_data_smithi194_103039_object76508:head# 2024-04-29T14:21:19.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca3738e:::benchmark_data_smithi194_103039_object79853:head# 2024-04-29T14:21:19.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca3d16e:::benchmark_data_smithi194_103039_object50945:head# 2024-04-29T14:21:19.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca4202a:::benchmark_data_smithi194_103039_object151404:head# 2024-04-29T14:21:19.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca4b671:::benchmark_data_smithi194_103039_object45356:head# 2024-04-29T14:21:19.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca50a80:::benchmark_data_smithi194_103039_object83583:head# 2024-04-29T14:21:19.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca5142e:::benchmark_data_smithi194_103039_object130554:head# 2024-04-29T14:21:19.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca5181f:::benchmark_data_smithi194_103039_object42230:head# 2024-04-29T14:21:19.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca54473:::benchmark_data_smithi194_103039_object61971:head# 2024-04-29T14:21:19.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca5456b:::benchmark_data_smithi194_103039_object139837:head# 2024-04-29T14:21:19.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca5d94d:::benchmark_data_smithi194_103039_object118775:head# 2024-04-29T14:21:19.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca6720e:::benchmark_data_smithi194_103039_object152236:head# 2024-04-29T14:21:19.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca6959d:::benchmark_data_smithi194_103039_object76014:head# 2024-04-29T14:21:19.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca695e4:::benchmark_data_smithi194_103039_object67183:head# 2024-04-29T14:21:19.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca6fe24:::benchmark_data_smithi194_103039_object96471:head# 2024-04-29T14:21:19.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca7c310:::benchmark_data_smithi194_103039_object48140:head# 2024-04-29T14:21:19.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca901b8:::benchmark_data_smithi194_103039_object77646:head# 2024-04-29T14:21:19.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ca98031:::benchmark_data_smithi194_103039_object109379:head# 2024-04-29T14:21:19.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4caa5925:::benchmark_data_smithi194_103039_object49110:head# 2024-04-29T14:21:19.009 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4caa93c0:::benchmark_data_smithi194_103039_object98324:head# 2024-04-29T14:21:19.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4caaec5e:::benchmark_data_smithi194_103039_object141855:head# 2024-04-29T14:21:19.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cab6333:::benchmark_data_smithi194_103039_object143461:head# 2024-04-29T14:21:19.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cab93d4:::benchmark_data_smithi194_103039_object63299:head# 2024-04-29T14:21:19.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cabacad:::benchmark_data_smithi194_103039_object73035:head# 2024-04-29T14:21:19.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cabc188:::benchmark_data_smithi194_103039_object64640:head# 2024-04-29T14:21:19.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cac80e7:::benchmark_data_smithi194_103039_object126842:head# 2024-04-29T14:21:19.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cad251a:::benchmark_data_smithi194_103039_object57860:head# 2024-04-29T14:21:19.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4caecac2:::benchmark_data_smithi194_103039_object89291:head# 2024-04-29T14:21:19.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb1e4eb:::benchmark_data_smithi194_103039_object58922:head# 2024-04-29T14:21:19.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb23d35:::benchmark_data_smithi194_103039_object140891:head# 2024-04-29T14:21:19.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb32b5e:::benchmark_data_smithi194_103039_object47575:head# 2024-04-29T14:21:19.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb49fa9:::benchmark_data_smithi194_103039_object143788:head# 2024-04-29T14:21:19.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb7bf68:::benchmark_data_smithi194_103039_object119627:head# 2024-04-29T14:21:19.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb8f1ca:::benchmark_data_smithi194_103039_object138208:head# 2024-04-29T14:21:19.016 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:19.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb920b8:::benchmark_data_smithi194_103039_object72527:head# 2024-04-29T14:21:19.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cb98c69:::benchmark_data_smithi194_103039_object76535:head# 2024-04-29T14:21:19.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbac124:::benchmark_data_smithi194_103039_object134000:head# 2024-04-29T14:21:19.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbb8558:::benchmark_data_smithi194_103039_object113053:head# 2024-04-29T14:21:19.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbbc8e2:::benchmark_data_smithi194_103039_object81156:head# 2024-04-29T14:21:19.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbbce5d:::benchmark_data_smithi194_103039_object139607:head# 2024-04-29T14:21:19.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbc6113:::benchmark_data_smithi194_103039_object91585:head# 2024-04-29T14:21:19.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbcc276:::benchmark_data_smithi194_103039_object125961:head# 2024-04-29T14:21:19.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbd321e:::benchmark_data_smithi194_103039_object106369:head# 2024-04-29T14:21:19.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbd709b:::benchmark_data_smithi194_103039_object156310:head# 2024-04-29T14:21:19.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbe821b:::benchmark_data_smithi194_103039_object52048:head# 2024-04-29T14:21:19.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbf0467:::benchmark_data_smithi194_103039_object64994:head# 2024-04-29T14:21:19.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cbf31c4:::benchmark_data_smithi194_103039_object83366:head# 2024-04-29T14:21:19.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc12798:::benchmark_data_smithi194_103039_object125595:head# 2024-04-29T14:21:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc1d797:::benchmark_data_smithi194_103039_object97514:head# 2024-04-29T14:21:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc1e61d:::benchmark_data_smithi194_103039_object81071:head# 2024-04-29T14:21:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc20795:::benchmark_data_smithi194_103039_object93025:head# 2024-04-29T14:21:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc3675b:::benchmark_data_smithi194_103039_object122610:head# 2024-04-29T14:21:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc3db10:::benchmark_data_smithi194_103039_object95470:head# 2024-04-29T14:21:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc48e61:::benchmark_data_smithi194_103039_object64498:head# 2024-04-29T14:21:19.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc4b2a1:::benchmark_data_smithi194_103039_object44956:head# 2024-04-29T14:21:19.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc4ef3a:::benchmark_data_smithi194_103039_object122337:head# 2024-04-29T14:21:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc5729a:::benchmark_data_smithi194_103039_object72882:head# 2024-04-29T14:21:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc62a28:::benchmark_data_smithi194_103039_object93037:head# 2024-04-29T14:21:19.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc80091:::benchmark_data_smithi194_103039_object66182:head# 2024-04-29T14:21:19.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc890e4:::benchmark_data_smithi194_103039_object101467:head# 2024-04-29T14:21:19.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc9223f:::benchmark_data_smithi194_103039_object124105:head# 2024-04-29T14:21:19.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc93c6d:::benchmark_data_smithi194_103039_object107473:head# 2024-04-29T14:21:19.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cc9c211:::benchmark_data_smithi194_103039_object78453:head# 2024-04-29T14:21:19.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cca1b6f:::benchmark_data_smithi194_103039_object96992:head# 2024-04-29T14:21:19.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cca3388:::benchmark_data_smithi194_103039_object81531:head# 2024-04-29T14:21:19.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccaeb4a:::benchmark_data_smithi194_103039_object115615:head# 2024-04-29T14:21:19.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccb615d:::benchmark_data_smithi194_103039_object110162:head# 2024-04-29T14:21:19.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccbe5f0:::benchmark_data_smithi194_103039_object79670:head# 2024-04-29T14:21:19.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccbeac4:::benchmark_data_smithi194_103039_object151821:head# 2024-04-29T14:21:19.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccc7125:::benchmark_data_smithi194_103039_object102969:head# 2024-04-29T14:21:19.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccccd3a:::benchmark_data_smithi194_103039_object44961:head# 2024-04-29T14:21:19.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccd2222:::benchmark_data_smithi194_103039_object133318:head# 2024-04-29T14:21:19.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccde230:::benchmark_data_smithi194_103039_object96428:head# 2024-04-29T14:21:19.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccf2a1f:::benchmark_data_smithi194_103039_object121494:head# 2024-04-29T14:21:19.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccfe4f7:::benchmark_data_smithi194_103039_object95993:head# 2024-04-29T14:21:19.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ccfe675:::benchmark_data_smithi194_103039_object67207:head# 2024-04-29T14:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd022bb:::benchmark_data_smithi194_103039_object154285:head# 2024-04-29T14:21:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd11612:::benchmark_data_smithi194_103039_object103857:head# 2024-04-29T14:21:19.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd13e0c:::benchmark_data_smithi194_103039_object144610:head# 2024-04-29T14:21:19.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd15ef9:::benchmark_data_smithi194_103039_object80872:head# 2024-04-29T14:21:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd19f0b:::benchmark_data_smithi194_103039_object64275:head# 2024-04-29T14:21:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd4aa7f:::benchmark_data_smithi194_103039_object92232:head# 2024-04-29T14:21:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd4c1ab:::benchmark_data_smithi194_103039_object136968:head# 2024-04-29T14:21:19.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd56777:::benchmark_data_smithi194_103039_object98006:head# 2024-04-29T14:21:19.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd680c3:::benchmark_data_smithi194_103039_object135336:head# 2024-04-29T14:21:19.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd6b76d:::benchmark_data_smithi194_103039_object74481:head# 2024-04-29T14:21:19.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd75cfb:::benchmark_data_smithi194_103039_object148562:head# 2024-04-29T14:21:19.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd812e3:::benchmark_data_smithi194_103039_object93908:head# 2024-04-29T14:21:19.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cd8d7ac:::benchmark_data_smithi194_103039_object57354:head# 2024-04-29T14:21:19.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cda00de:::benchmark_data_smithi194_103039_object71653:head# 2024-04-29T14:21:19.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cda4fd1:::benchmark_data_smithi194_103039_object64111:head# 2024-04-29T14:21:19.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cdb0055:::benchmark_data_smithi194_103039_object91425:head# 2024-04-29T14:21:19.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cdb4ddc:::benchmark_data_smithi194_103039_object91871:head# 2024-04-29T14:21:19.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cdc23ee:::benchmark_data_smithi194_103039_object136918:head# 2024-04-29T14:21:19.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cdc4114:::benchmark_data_smithi194_103039_object98911:head# 2024-04-29T14:21:19.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cdd1d5c:::benchmark_data_smithi194_103039_object145384:head# 2024-04-29T14:21:19.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cde0de7:::benchmark_data_smithi194_103039_object148752:head# 2024-04-29T14:21:19.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cde8ed9:::benchmark_data_smithi194_103039_object110881:head# 2024-04-29T14:21:19.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cde9ce9:::benchmark_data_smithi194_103039_object70412:head# 2024-04-29T14:21:19.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cdf6014:::benchmark_data_smithi194_103039_object53863:head# 2024-04-29T14:21:19.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cdfa452:::benchmark_data_smithi194_103039_object64552:head# 2024-04-29T14:21:19.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce088de:::benchmark_data_smithi194_103039_object81161:head# 2024-04-29T14:21:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce0e069:::benchmark_data_smithi194_103039_object62854:head# 2024-04-29T14:21:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce1a89a:::benchmark_data_smithi194_103039_object124724:head# 2024-04-29T14:21:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce2c6ef:::benchmark_data_smithi194_103039_object136870:head# 2024-04-29T14:21:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce47fe0:::benchmark_data_smithi194_103039_object68778:head# 2024-04-29T14:21:19.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce54e75:::benchmark_data_smithi194_103039_object119829:head# 2024-04-29T14:21:19.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce69e1b:::benchmark_data_smithi194_103039_object41751:head# 2024-04-29T14:21:19.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce813a9:::benchmark_data_smithi194_103039_object44790:head# 2024-04-29T14:21:19.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce83f97:::benchmark_data_smithi194_103039_object56889:head# 2024-04-29T14:21:19.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce8f7e8:::benchmark_data_smithi194_103039_object129991:head# 2024-04-29T14:21:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce91674:::benchmark_data_smithi194_103039_object96611:head# 2024-04-29T14:21:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ce930dc:::benchmark_data_smithi194_103039_object42628:head# 2024-04-29T14:21:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ceae545:::benchmark_data_smithi194_103039_object122094:head# 2024-04-29T14:21:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cebb991:::benchmark_data_smithi194_103039_object69292:head# 2024-04-29T14:21:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cec626b:::benchmark_data_smithi194_103039_object66813:head# 2024-04-29T14:21:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cedd77e:::benchmark_data_smithi194_103039_object155191:head# 2024-04-29T14:21:19.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cedde40:::benchmark_data_smithi194_103039_object72028:head# 2024-04-29T14:21:19.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cee77e2:::benchmark_data_smithi194_103039_object94677:head# 2024-04-29T14:21:19.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ceecc6d:::benchmark_data_smithi194_103039_object64139:head# 2024-04-29T14:21:19.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cef2bb4:::benchmark_data_smithi194_103039_object41777:head# 2024-04-29T14:21:19.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cef8bb0:::benchmark_data_smithi194_103039_object109236:head# 2024-04-29T14:21:19.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf105f6:::benchmark_data_smithi194_103039_object80354:head# 2024-04-29T14:21:19.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf13735:::benchmark_data_smithi194_103039_object64902:head# 2024-04-29T14:21:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf16817:::benchmark_data_smithi194_103039_object98335:head# 2024-04-29T14:21:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf1c5ac:::benchmark_data_smithi194_103039_object73835:head# 2024-04-29T14:21:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf1d87f:::benchmark_data_smithi194_103039_object94857:head# 2024-04-29T14:21:19.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf1e5aa:::benchmark_data_smithi194_103039_object48850:head# 2024-04-29T14:21:19.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf1f2cd:::benchmark_data_smithi194_103039_object99755:head# 2024-04-29T14:21:19.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf20ab7:::benchmark_data_smithi194_103039_object104007:head# 2024-04-29T14:21:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf22cba:::benchmark_data_smithi194_103039_object56538:head# 2024-04-29T14:21:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf22e9f:::benchmark_data_smithi194_103039_object88127:head# 2024-04-29T14:21:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf33462:::benchmark_data_smithi194_103039_object83062:head# 2024-04-29T14:21:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf350d3:::benchmark_data_smithi194_103039_object45645:head# 2024-04-29T14:21:19.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf36bda:::benchmark_data_smithi194_103039_object125934:head# 2024-04-29T14:21:19.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf50930:::benchmark_data_smithi194_103039_object122781:head# 2024-04-29T14:21:19.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf5bb27:::benchmark_data_smithi194_103039_object120467:head# 2024-04-29T14:21:19.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf6f890:::benchmark_data_smithi194_103039_object136756:head# 2024-04-29T14:21:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf7f5ba:::benchmark_data_smithi194_103039_object99401:head# 2024-04-29T14:21:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf8cfc8:::benchmark_data_smithi194_103039_object95274:head# 2024-04-29T14:21:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf8ffbe:::benchmark_data_smithi194_103039_object74515:head# 2024-04-29T14:21:19.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cf9b8e3:::benchmark_data_smithi194_103039_object78362:head# 2024-04-29T14:21:19.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cfaba94:::benchmark_data_smithi194_103039_object88618:head# 2024-04-29T14:21:19.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cfade4d:::benchmark_data_smithi194_103039_object137376:head# 2024-04-29T14:21:19.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cfb14a8:::benchmark_data_smithi194_103039_object110352:head# 2024-04-29T14:21:19.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cfb509c:::benchmark_data_smithi194_103039_object135152:head# 2024-04-29T14:21:19.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cfc251f:::benchmark_data_smithi194_103039_object123328:head# 2024-04-29T14:21:19.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cfd7b08:::benchmark_data_smithi194_103039_object79523:head# 2024-04-29T14:21:19.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cff922e:::benchmark_data_smithi194_103039_object52069:head# 2024-04-29T14:21:19.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4cff9d0d:::benchmark_data_smithi194_103039_object73864:head# 2024-04-29T14:21:19.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d007291:::benchmark_data_smithi194_103039_object77597:head# 2024-04-29T14:21:19.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0072e9:::benchmark_data_smithi194_103039_object113020:head# 2024-04-29T14:21:19.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d008880:::benchmark_data_smithi194_103039_object79725:head# 2024-04-29T14:21:19.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d018cec:::benchmark_data_smithi194_103039_object101851:head# 2024-04-29T14:21:19.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d02ee54:::benchmark_data_smithi194_103039_object94964:head# 2024-04-29T14:21:19.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d02fc30:::benchmark_data_smithi194_103039_object75696:head# 2024-04-29T14:21:19.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d03ec47:::benchmark_data_smithi194_103039_object138068:head# 2024-04-29T14:21:19.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0410a5:::benchmark_data_smithi194_103039_object148139:head# 2024-04-29T14:21:19.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d045496:::benchmark_data_smithi194_103039_object65371:head# 2024-04-29T14:21:19.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d049615:::benchmark_data_smithi194_103039_object45742:head# 2024-04-29T14:21:19.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d04bfa1:::benchmark_data_smithi194_103039_object130811:head# 2024-04-29T14:21:19.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d054d39:::benchmark_data_smithi194_103039_object49913:head# 2024-04-29T14:21:19.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0712e0:::benchmark_data_smithi194_103039_object55956:head# 2024-04-29T14:21:19.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d07a7e5:::benchmark_data_smithi194_103039_object96871:head# 2024-04-29T14:21:19.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d07cc93:::benchmark_data_smithi194_103039_object51415:head# 2024-04-29T14:21:19.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d08333c:::benchmark_data_smithi194_103039_object99398:head# 2024-04-29T14:21:19.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d087a32:::benchmark_data_smithi194_103039_object139916:head# 2024-04-29T14:21:19.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d089df2:::benchmark_data_smithi194_103039_object58392:head# 2024-04-29T14:21:19.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d09280a:::benchmark_data_smithi194_103039_object61471:head# 2024-04-29T14:21:19.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0a97b4:::benchmark_data_smithi194_103039_object134937:head# 2024-04-29T14:21:19.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0b48f7:::benchmark_data_smithi194_103039_object51290:head# 2024-04-29T14:21:19.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0bbd52:::benchmark_data_smithi194_103039_object103296:head# 2024-04-29T14:21:19.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0cd435:::benchmark_data_smithi194_103039_object96595:head# 2024-04-29T14:21:19.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0d50b8:::benchmark_data_smithi194_103039_object132210:head# 2024-04-29T14:21:19.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0d8517:::benchmark_data_smithi194_103039_object113854:head# 2024-04-29T14:21:19.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d0f7a99:::benchmark_data_smithi194_103039_object69773:head# 2024-04-29T14:21:19.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1078b4:::benchmark_data_smithi194_103039_object65050:head# 2024-04-29T14:21:19.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d10aa76:::benchmark_data_smithi194_103039_object78296:head# 2024-04-29T14:21:19.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d10c5ea:::benchmark_data_smithi194_103039_object107800:head# 2024-04-29T14:21:19.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d117c2a:::benchmark_data_smithi194_103039_object136190:head# 2024-04-29T14:21:19.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d119b09:::benchmark_data_smithi194_103039_object89672:head# 2024-04-29T14:21:19.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d11b763:::benchmark_data_smithi194_103039_object106043:head# 2024-04-29T14:21:19.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d12689d:::benchmark_data_smithi194_103039_object138371:head# 2024-04-29T14:21:19.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d127fbc:::benchmark_data_smithi194_103039_object136682:head# 2024-04-29T14:21:19.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d12f05f:::benchmark_data_smithi194_103039_object89729:head# 2024-04-29T14:21:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:21:19.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d13558a:::benchmark_data_smithi194_103039_object94111:head# 2024-04-29T14:21:19.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1507f2:::benchmark_data_smithi194_103039_object42113:head# 2024-04-29T14:21:19.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d162494:::benchmark_data_smithi194_103039_object100420:head# 2024-04-29T14:21:19.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d16492b:::benchmark_data_smithi194_103039_object151656:head# 2024-04-29T14:21:19.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:21:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d170ab6:::benchmark_data_smithi194_103039_object134575:head# 2024-04-29T14:21:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d187bd8:::benchmark_data_smithi194_103039_object150375:head# 2024-04-29T14:21:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d189bc5:::benchmark_data_smithi194_103039_object49357:head# 2024-04-29T14:21:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d19414a:::benchmark_data_smithi194_103039_object97976:head# 2024-04-29T14:21:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1a49ff:::benchmark_data_smithi194_103039_object77436:head# 2024-04-29T14:21:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1b0c24:::benchmark_data_smithi194_103039_object146397:head# 2024-04-29T14:21:19.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1c2c66:::benchmark_data_smithi194_103039_object131819:head# 2024-04-29T14:21:19.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1c9048:::benchmark_data_smithi194_103039_object130883:head# 2024-04-29T14:21:19.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1ce604:::benchmark_data_smithi194_103039_object81999:head# 2024-04-29T14:21:19.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1cffb0:::benchmark_data_smithi194_103039_object90240:head# 2024-04-29T14:21:19.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1d47f2:::benchmark_data_smithi194_103039_object67047:head# 2024-04-29T14:21:19.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1dcf2b:::benchmark_data_smithi194_103039_object143917:head# 2024-04-29T14:21:19.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1ddbc9:::benchmark_data_smithi194_103039_object57075:head# 2024-04-29T14:21:19.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1e5d33:::benchmark_data_smithi194_103039_object91379:head# 2024-04-29T14:21:19.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d1f7b55:::benchmark_data_smithi194_103039_object108464:head# 2024-04-29T14:21:19.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d202b78:::benchmark_data_smithi194_103039_object69555:head# 2024-04-29T14:21:19.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d2090fb:::benchmark_data_smithi194_103039_object84700:head# 2024-04-29T14:21:19.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d220116:::benchmark_data_smithi194_103039_object151919:head# 2024-04-29T14:21:19.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d222130:::benchmark_data_smithi194_103039_object47436:head# 2024-04-29T14:21:19.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d222382:::benchmark_data_smithi194_103039_object145412:head# 2024-04-29T14:21:19.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d23b018:::benchmark_data_smithi194_103039_object92643:head# 2024-04-29T14:21:19.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d251eb2:::benchmark_data_smithi194_103039_object100687:head# 2024-04-29T14:21:19.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d25328d:::benchmark_data_smithi194_103039_object64734:head# 2024-04-29T14:21:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d264556:::benchmark_data_smithi194_103039_object98934:head# 2024-04-29T14:21:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d267afd:::benchmark_data_smithi194_103039_object117492:head# 2024-04-29T14:21:19.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d277ad9:::benchmark_data_smithi194_103039_object80544:head# 2024-04-29T14:21:19.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d280a34:::benchmark_data_smithi194_103039_object133884:head# 2024-04-29T14:21:19.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d285a7c:::benchmark_data_smithi194_103039_object43891:head# 2024-04-29T14:21:19.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d2a1ce4:::benchmark_data_smithi194_103039_object60869:head# 2024-04-29T14:21:19.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d2a207a:::benchmark_data_smithi194_103039_object52455:head# 2024-04-29T14:21:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d2a5730:::benchmark_data_smithi194_103039_object54784:head# 2024-04-29T14:21:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d2aebf7:::benchmark_data_smithi194_103039_object114914:head# 2024-04-29T14:21:19.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d2b6ec4:::benchmark_data_smithi194_103039_object112745:head# 2024-04-29T14:21:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d307c57:::benchmark_data_smithi194_103039_object88944:head# 2024-04-29T14:21:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d31c29d:::benchmark_data_smithi194_103039_object53455:head# 2024-04-29T14:21:19.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d324aba:::benchmark_data_smithi194_103039_object41551:head# 2024-04-29T14:21:19.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d32770e:::benchmark_data_smithi194_103039_object128703:head# 2024-04-29T14:21:19.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d32d9a8:::benchmark_data_smithi194_103039_object68554:head# 2024-04-29T14:21:19.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d332ed5:::benchmark_data_smithi194_103039_object155234:head# 2024-04-29T14:21:19.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d344549:::benchmark_data_smithi194_103039_object110915:head# 2024-04-29T14:21:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:19.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3510eb:::benchmark_data_smithi194_103039_object156505:head# 2024-04-29T14:21:19.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d361fa7:::benchmark_data_smithi194_103039_object136132:head# 2024-04-29T14:21:19.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d368e97:::benchmark_data_smithi194_103039_object155584:head# 2024-04-29T14:21:19.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d371b5c:::benchmark_data_smithi194_103039_object140695:head# 2024-04-29T14:21:19.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d379330:::benchmark_data_smithi194_103039_object99653:head# 2024-04-29T14:21:19.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d37ab55:::benchmark_data_smithi194_103039_object146039:head# 2024-04-29T14:21:19.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d387d8b:::benchmark_data_smithi194_103039_object88139:head# 2024-04-29T14:21:19.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d393868:::benchmark_data_smithi194_103039_object105703:head# 2024-04-29T14:21:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:19.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:19.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3984a8:::benchmark_data_smithi194_103039_object72719:head# 2024-04-29T14:21:19.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d39be1b:::benchmark_data_smithi194_103039_object122677:head# 2024-04-29T14:21:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3acc14:::benchmark_data_smithi194_103039_object149759:head# 2024-04-29T14:21:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3bb53a:::benchmark_data_smithi194_103039_object52752:head# 2024-04-29T14:21:19.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3bf3bf:::benchmark_data_smithi194_103039_object106350:head# 2024-04-29T14:21:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3cb79e:::benchmark_data_smithi194_103039_object96362:head# 2024-04-29T14:21:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3cfc68:::benchmark_data_smithi194_103039_object111120:head# 2024-04-29T14:21:19.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3d772a:::benchmark_data_smithi194_103039_object101710:head# 2024-04-29T14:21:19.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3dfba0:::benchmark_data_smithi194_103039_object43582:head# 2024-04-29T14:21:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3e1295:::benchmark_data_smithi194_103039_object102599:head# 2024-04-29T14:21:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3f0517:::benchmark_data_smithi194_103039_object125922:head# 2024-04-29T14:21:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d3ff773:::benchmark_data_smithi194_103039_object75830:head# 2024-04-29T14:21:19.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d405fdb:::benchmark_data_smithi194_103039_object108981:head# 2024-04-29T14:21:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d40bbe9:::benchmark_data_smithi194_103039_object86152:head# 2024-04-29T14:21:19.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d414915:::benchmark_data_smithi194_103039_object61054:head# 2024-04-29T14:21:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d41855d:::benchmark_data_smithi194_103039_object84418:head# 2024-04-29T14:21:19.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4192e1:::benchmark_data_smithi194_103039_object87145:head# 2024-04-29T14:21:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d41b139:::benchmark_data_smithi194_103039_object94527:head# 2024-04-29T14:21:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d433113:::benchmark_data_smithi194_103039_object112208:head# 2024-04-29T14:21:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d43663c:::benchmark_data_smithi194_103039_object69098:head# 2024-04-29T14:21:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d44b5a1:::benchmark_data_smithi194_103039_object138939:head# 2024-04-29T14:21:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d46078a:::benchmark_data_smithi194_103039_object81974:head# 2024-04-29T14:21:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d47623b:::benchmark_data_smithi194_103039_object111981:head# 2024-04-29T14:21:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d47723a:::benchmark_data_smithi194_103039_object132817:head# 2024-04-29T14:21:19.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d486571:::benchmark_data_smithi194_103039_object122789:head# 2024-04-29T14:21:19.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d486781:::benchmark_data_smithi194_103039_object112148:head# 2024-04-29T14:21:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4887a2:::benchmark_data_smithi194_103039_object140337:head# 2024-04-29T14:21:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d48f94a:::benchmark_data_smithi194_103039_object144768:head# 2024-04-29T14:21:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4918c5:::benchmark_data_smithi194_103039_object139791:head# 2024-04-29T14:21:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4a0798:::benchmark_data_smithi194_103039_object43556:head# 2024-04-29T14:21:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4a7489:::benchmark_data_smithi194_103039_object129066:head# 2024-04-29T14:21:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4b25a7:::benchmark_data_smithi194_103039_object120349:head# 2024-04-29T14:21:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4b3b8b:::benchmark_data_smithi194_103039_object135417:head# 2024-04-29T14:21:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4b4769:::benchmark_data_smithi194_103039_object73222:head# 2024-04-29T14:21:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4bb62c:::benchmark_data_smithi194_103039_object91424:head# 2024-04-29T14:21:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4bdcde:::benchmark_data_smithi194_103039_object126778:head# 2024-04-29T14:21:19.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4c2f8c:::benchmark_data_smithi194_103039_object41796:head# 2024-04-29T14:21:19.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4d0af6:::benchmark_data_smithi194_103039_object51794:head# 2024-04-29T14:21:19.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4decf8:::benchmark_data_smithi194_103039_object86275:head# 2024-04-29T14:21:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4e5e1d:::benchmark_data_smithi194_103039_object134335:head# 2024-04-29T14:21:19.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4e8163:::benchmark_data_smithi194_103039_object143898:head# 2024-04-29T14:21:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4ebbaa:::benchmark_data_smithi194_103039_object126252:head# 2024-04-29T14:21:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4f0d7d:::benchmark_data_smithi194_103039_object81514:head# 2024-04-29T14:21:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4f2cef:::benchmark_data_smithi194_103039_object83199:head# 2024-04-29T14:21:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4f89ad:::benchmark_data_smithi194_103039_object99050:head# 2024-04-29T14:21:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4fd38f:::benchmark_data_smithi194_103039_object133790:head# 2024-04-29T14:21:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d4fef8c:::benchmark_data_smithi194_103039_object147421:head# 2024-04-29T14:21:19.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d503a92:::benchmark_data_smithi194_103039_object88654:head# 2024-04-29T14:21:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d507864:::benchmark_data_smithi194_103039_object85304:head# 2024-04-29T14:21:19.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d50dddb:::benchmark_data_smithi194_103039_object50637:head# 2024-04-29T14:21:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d50ec2d:::benchmark_data_smithi194_103039_object63148:head# 2024-04-29T14:21:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5121e4:::benchmark_data_smithi194_103039_object94701:head# 2024-04-29T14:21:19.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d51ffea:::benchmark_data_smithi194_103039_object111831:head# 2024-04-29T14:21:19.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d522cc5:::benchmark_data_smithi194_103039_object60107:head# 2024-04-29T14:21:19.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d535275:::benchmark_data_smithi194_103039_object56247:head# 2024-04-29T14:21:19.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d55452d:::benchmark_data_smithi194_103039_object146913:head# 2024-04-29T14:21:19.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d554622:::benchmark_data_smithi194_103039_object87800:head# 2024-04-29T14:21:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d55b7a7:::benchmark_data_smithi194_103039_object102663:head# 2024-04-29T14:21:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5607fb:::benchmark_data_smithi194_103039_object53134:head# 2024-04-29T14:21:19.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5618c8:::benchmark_data_smithi194_103039_object79230:head# 2024-04-29T14:21:19.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d565648:::benchmark_data_smithi194_103039_object101790:head# 2024-04-29T14:21:19.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d570d34:::benchmark_data_smithi194_103039_object43663:head# 2024-04-29T14:21:19.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d57a46b:::benchmark_data_smithi194_103039_object128595:head# 2024-04-29T14:21:19.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d57c3a7:::benchmark_data_smithi194_103039_object128973:head# 2024-04-29T14:21:19.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d596a83:::benchmark_data_smithi194_103039_object78683:head# 2024-04-29T14:21:19.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d59daef:::benchmark_data_smithi194_103039_object113213:head# 2024-04-29T14:21:19.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d59fc31:::benchmark_data_smithi194_103039_object103626:head# 2024-04-29T14:21:19.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5a9995:::benchmark_data_smithi194_103039_object58820:head# 2024-04-29T14:21:19.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5c03ec:::benchmark_data_smithi194_103039_object60696:head# 2024-04-29T14:21:19.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5c28c0:::benchmark_data_smithi194_103039_object46943:head# 2024-04-29T14:21:19.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5c8c98:::benchmark_data_smithi194_103039_object114096:head# 2024-04-29T14:21:19.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5c8fdf:::benchmark_data_smithi194_103039_object108597:head# 2024-04-29T14:21:19.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5d6149:::benchmark_data_smithi194_103039_object105718:head# 2024-04-29T14:21:19.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5db671:::benchmark_data_smithi194_103039_object137790:head# 2024-04-29T14:21:19.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5e52bb:::benchmark_data_smithi194_103039_object130364:head# 2024-04-29T14:21:19.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5e7634:::benchmark_data_smithi194_103039_object47337:head# 2024-04-29T14:21:19.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5f5806:::benchmark_data_smithi194_103039_object112167:head# 2024-04-29T14:21:19.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5f9bbe:::benchmark_data_smithi194_103039_object113446:head# 2024-04-29T14:21:19.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d5fc7a3:::benchmark_data_smithi194_103039_object83740:head# 2024-04-29T14:21:19.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d602977:::benchmark_data_smithi194_103039_object61071:head# 2024-04-29T14:21:19.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d60802b:::benchmark_data_smithi194_103039_object77262:head# 2024-04-29T14:21:19.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d60869f:::benchmark_data_smithi194_103039_object128919:head# 2024-04-29T14:21:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6088e6:::benchmark_data_smithi194_103039_object99112:head# 2024-04-29T14:21:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6163ed:::benchmark_data_smithi194_103039_object129906:head# 2024-04-29T14:21:19.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d618ee4:::benchmark_data_smithi194_103039_object80174:head# 2024-04-29T14:21:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6239a1:::benchmark_data_smithi194_103039_object134039:head# 2024-04-29T14:21:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6260d7:::benchmark_data_smithi194_103039_object61508:head# 2024-04-29T14:21:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d63beb6:::benchmark_data_smithi194_103039_object145221:head# 2024-04-29T14:21:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d65485b:::benchmark_data_smithi194_103039_object72575:head# 2024-04-29T14:21:19.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d656640:::benchmark_data_smithi194_103039_object129171:head# 2024-04-29T14:21:19.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d65988a:::benchmark_data_smithi194_103039_object153674:head# 2024-04-29T14:21:19.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d65ac40:::benchmark_data_smithi194_103039_object111165:head# 2024-04-29T14:21:19.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d66d4c4:::benchmark_data_smithi194_103039_object59514:head# 2024-04-29T14:21:19.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d66e082:::benchmark_data_smithi194_103039_object141823:head# 2024-04-29T14:21:19.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d674fdd:::benchmark_data_smithi194_103039_object75639:head# 2024-04-29T14:21:19.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d68e475:::benchmark_data_smithi194_103039_object120572:head# 2024-04-29T14:21:19.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6a0550:::benchmark_data_smithi194_103039_object69691:head# 2024-04-29T14:21:19.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6a41dd:::benchmark_data_smithi194_103039_object71799:head# 2024-04-29T14:21:19.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6a64b9:::benchmark_data_smithi194_103039_object87389:head# 2024-04-29T14:21:19.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6b20e7:::benchmark_data_smithi194_103039_object145738:head# 2024-04-29T14:21:19.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6b315f:::benchmark_data_smithi194_103039_object117722:head# 2024-04-29T14:21:19.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6b8dfe:::benchmark_data_smithi194_103039_object116711:head# 2024-04-29T14:21:19.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6c3bfd:::benchmark_data_smithi194_103039_object42324:head# 2024-04-29T14:21:19.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6c5039:::benchmark_data_smithi194_103039_object90335:head# 2024-04-29T14:21:19.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6d41a4:::benchmark_data_smithi194_103039_object43141:head# 2024-04-29T14:21:19.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6dea76:::benchmark_data_smithi194_103039_object60839:head# 2024-04-29T14:21:19.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6e3c80:::benchmark_data_smithi194_103039_object42554:head# 2024-04-29T14:21:19.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6e54c5:::benchmark_data_smithi194_103039_object64274:head# 2024-04-29T14:21:19.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6e6da7:::benchmark_data_smithi194_103039_object112549:head# 2024-04-29T14:21:19.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6ed9b8:::benchmark_data_smithi194_103039_object152759:head# 2024-04-29T14:21:19.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6eff46:::benchmark_data_smithi194_103039_object141425:head# 2024-04-29T14:21:19.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d6f1108:::benchmark_data_smithi194_103039_object75156:head# 2024-04-29T14:21:19.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d70b9f9:::benchmark_data_smithi194_103039_object64007:head# 2024-04-29T14:21:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d71169a:::benchmark_data_smithi194_103039_object79602:head# 2024-04-29T14:21:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d711ecf:::benchmark_data_smithi194_103039_object71212:head# 2024-04-29T14:21:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7387ed:::benchmark_data_smithi194_103039_object147796:head# 2024-04-29T14:21:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d756fe8:::benchmark_data_smithi194_103039_object135820:head# 2024-04-29T14:21:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7599b7:::benchmark_data_smithi194_103039_object76251:head# 2024-04-29T14:21:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d76aa4c:::benchmark_data_smithi194_103039_object82560:head# 2024-04-29T14:21:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d775fa2:::benchmark_data_smithi194_103039_object65553:head# 2024-04-29T14:21:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d779898:::benchmark_data_smithi194_103039_object44206:head# 2024-04-29T14:21:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d77d3c1:::benchmark_data_smithi194_103039_object89516:head# 2024-04-29T14:21:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7ae357:::benchmark_data_smithi194_103039_object148727:head# 2024-04-29T14:21:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7b37f0:::benchmark_data_smithi194_103039_object42707:head# 2024-04-29T14:21:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7b73be:::benchmark_data_smithi194_103039_object70713:head# 2024-04-29T14:21:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7b8d9a:::benchmark_data_smithi194_103039_object99479:head# 2024-04-29T14:21:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7ba4c8:::benchmark_data_smithi194_103039_object137366:head# 2024-04-29T14:21:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7beb79:::benchmark_data_smithi194_103039_object137737:head# 2024-04-29T14:21:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7d20a5:::benchmark_data_smithi194_103039_object74839:head# 2024-04-29T14:21:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7d6c33:::benchmark_data_smithi194_103039_object120780:head# 2024-04-29T14:21:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7d95b1:::benchmark_data_smithi194_103039_object99565:head# 2024-04-29T14:21:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7ddc9f:::benchmark_data_smithi194_103039_object128567:head# 2024-04-29T14:21:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7e326e:::benchmark_data_smithi194_103039_object52894:head# 2024-04-29T14:21:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7f1696:::benchmark_data_smithi194_103039_object46015:head# 2024-04-29T14:21:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7f3065:::benchmark_data_smithi194_103039_object121332:head# 2024-04-29T14:21:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7fa7ac:::benchmark_data_smithi194_103039_object143685:head# 2024-04-29T14:21:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d7fff5b:::benchmark_data_smithi194_103039_object113722:head# 2024-04-29T14:21:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d805723:::benchmark_data_smithi194_103039_object118569:head# 2024-04-29T14:21:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d805f08:::benchmark_data_smithi194_103039_object110756:head# 2024-04-29T14:21:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d80db2c:::benchmark_data_smithi194_103039_object140898:head# 2024-04-29T14:21:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d80dc1f:::benchmark_data_smithi194_103039_object93854:head# 2024-04-29T14:21:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d80fa5e:::benchmark_data_smithi194_103039_object142247:head# 2024-04-29T14:21:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d82895a:::benchmark_data_smithi194_103039_object117210:head# 2024-04-29T14:21:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d82ba90:::benchmark_data_smithi194_103039_object130323:head# 2024-04-29T14:21:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d830c1d:::benchmark_data_smithi194_103039_object120412:head# 2024-04-29T14:21:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d83eeca:::benchmark_data_smithi194_103039_object86569:head# 2024-04-29T14:21:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8480cf:::benchmark_data_smithi194_103039_object149866:head# 2024-04-29T14:21:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d84bb76:::benchmark_data_smithi194_103039_object117900:head# 2024-04-29T14:21:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d84e25d:::benchmark_data_smithi194_103039_object153568:head# 2024-04-29T14:21:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d858167:::benchmark_data_smithi194_103039_object105918:head# 2024-04-29T14:21:19.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d864c16:::benchmark_data_smithi194_103039_object76206:head# 2024-04-29T14:21:19.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d88c9a7:::benchmark_data_smithi194_103039_object149185:head# 2024-04-29T14:21:19.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8a4238:::benchmark_data_smithi194_103039_object104248:head# 2024-04-29T14:21:19.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8a4861:::benchmark_data_smithi194_103039_object112470:head# 2024-04-29T14:21:19.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8becd0:::benchmark_data_smithi194_103039_object55742:head# 2024-04-29T14:21:19.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8bf16c:::benchmark_data_smithi194_103039_object152160:head# 2024-04-29T14:21:19.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8c6e62:::benchmark_data_smithi194_103039_object51780:head# 2024-04-29T14:21:19.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8c77e7:::benchmark_data_smithi194_103039_object100462:head# 2024-04-29T14:21:19.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8c803d:::benchmark_data_smithi194_103039_object71926:head# 2024-04-29T14:21:19.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8dd2c0:::benchmark_data_smithi194_103039_object118521:head# 2024-04-29T14:21:19.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8eb64b:::benchmark_data_smithi194_103039_object119496:head# 2024-04-29T14:21:19.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d8ebcbe:::benchmark_data_smithi194_103039_object135269:head# 2024-04-29T14:21:19.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d901066:::benchmark_data_smithi194_103039_object105260:head# 2024-04-29T14:21:19.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d90193c:::benchmark_data_smithi194_103039_object140085:head# 2024-04-29T14:21:19.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d90d844:::benchmark_data_smithi194_103039_object104109:head# 2024-04-29T14:21:19.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d913458:::benchmark_data_smithi194_103039_object104113:head# 2024-04-29T14:21:19.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d91ebb9:::benchmark_data_smithi194_103039_object146460:head# 2024-04-29T14:21:19.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d924e2d:::benchmark_data_smithi194_103039_object126771:head# 2024-04-29T14:21:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d928b59:::benchmark_data_smithi194_103039_object49305:head# 2024-04-29T14:21:19.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d92f7aa:::benchmark_data_smithi194_103039_object97984:head# 2024-04-29T14:21:19.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d938471:::benchmark_data_smithi194_103039_object78124:head# 2024-04-29T14:21:19.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9388d8:::benchmark_data_smithi194_103039_object147436:head# 2024-04-29T14:21:19.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d943850:::benchmark_data_smithi194_103039_object117564:head# 2024-04-29T14:21:19.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d947f53:::benchmark_data_smithi194_103039_object60245:head# 2024-04-29T14:21:19.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d954ed4:::benchmark_data_smithi194_103039_object119833:head# 2024-04-29T14:21:19.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d95765e:::benchmark_data_smithi194_103039_object110964:head# 2024-04-29T14:21:19.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d964553:::benchmark_data_smithi194_103039_object125902:head# 2024-04-29T14:21:19.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d964bf9:::benchmark_data_smithi194_103039_object140451:head# 2024-04-29T14:21:19.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9667b3:::benchmark_data_smithi194_103039_object107742:head# 2024-04-29T14:21:19.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d96bac2:::benchmark_data_smithi194_103039_object89091:head# 2024-04-29T14:21:19.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d972dc5:::benchmark_data_smithi194_103039_object69905:head# 2024-04-29T14:21:19.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d972fb3:::benchmark_data_smithi194_103039_object138195:head# 2024-04-29T14:21:19.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d976146:::benchmark_data_smithi194_103039_object141423:head# 2024-04-29T14:21:19.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d98473e:::benchmark_data_smithi194_103039_object61706:head# 2024-04-29T14:21:19.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d98babf:::benchmark_data_smithi194_103039_object52560:head# 2024-04-29T14:21:19.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d98dc0c:::benchmark_data_smithi194_103039_object122605:head# 2024-04-29T14:21:19.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d99a670:::benchmark_data_smithi194_103039_object52846:head# 2024-04-29T14:21:19.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d99c0ea:::benchmark_data_smithi194_103039_object71525:head# 2024-04-29T14:21:19.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9a0668:::benchmark_data_smithi194_103039_object117526:head# 2024-04-29T14:21:19.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9b0951:::benchmark_data_smithi194_103039_object60538:head# 2024-04-29T14:21:19.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9ccbaa:::benchmark_data_smithi194_103039_object94979:head# 2024-04-29T14:21:19.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9e0c84:::benchmark_data_smithi194_103039_object151125:head# 2024-04-29T14:21:19.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9e4b12:::benchmark_data_smithi194_103039_object109828:head# 2024-04-29T14:21:19.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9e4bf1:::benchmark_data_smithi194_103039_object114972:head# 2024-04-29T14:21:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9e511b:::benchmark_data_smithi194_103039_object95448:head# 2024-04-29T14:21:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9ea9ef:::benchmark_data_smithi194_103039_object77595:head# 2024-04-29T14:21:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9f4f49:::benchmark_data_smithi194_103039_object107435:head# 2024-04-29T14:21:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9f5042:::benchmark_data_smithi194_103039_object133245:head# 2024-04-29T14:21:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4d9f63e8:::benchmark_data_smithi194_103039_object142824:head# 2024-04-29T14:21:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da01c0a:::benchmark_data_smithi194_103039_object153892:head# 2024-04-29T14:21:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da12aef:::benchmark_data_smithi194_103039_object113572:head# 2024-04-29T14:21:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da1d2a9:::benchmark_data_smithi194_103039_object140173:head# 2024-04-29T14:21:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da2716f:::benchmark_data_smithi194_103039_object139667:head# 2024-04-29T14:21:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da29666:::benchmark_data_smithi194_103039_object143694:head# 2024-04-29T14:21:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da33302:::benchmark_data_smithi194_103039_object42242:head# 2024-04-29T14:21:19.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da3bc6d:::benchmark_data_smithi194_103039_object63202:head# 2024-04-29T14:21:19.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da416cc:::benchmark_data_smithi194_103039_object146101:head# 2024-04-29T14:21:19.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da517ec:::benchmark_data_smithi194_103039_object64565:head# 2024-04-29T14:21:19.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da518af:::benchmark_data_smithi194_103039_object58144:head# 2024-04-29T14:21:19.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da5e1a4:::benchmark_data_smithi194_103039_object69453:head# 2024-04-29T14:21:19.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da6058f:::benchmark_data_smithi194_103039_object147727:head# 2024-04-29T14:21:19.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da6e899:::benchmark_data_smithi194_103039_object90676:head# 2024-04-29T14:21:19.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da7449f:::benchmark_data_smithi194_103039_object111991:head# 2024-04-29T14:21:19.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da93087:::benchmark_data_smithi194_103039_object72956:head# 2024-04-29T14:21:19.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da96e2c:::benchmark_data_smithi194_103039_object86160:head# 2024-04-29T14:21:19.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4da99640:::benchmark_data_smithi194_103039_object54088:head# 2024-04-29T14:21:19.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4daae5fc:::benchmark_data_smithi194_103039_object79916:head# 2024-04-29T14:21:19.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4daae875:::benchmark_data_smithi194_103039_object105050:head# 2024-04-29T14:21:19.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dab5a65:::benchmark_data_smithi194_103039_object117013:head# 2024-04-29T14:21:19.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dac5b6a:::benchmark_data_smithi194_103039_object66833:head# 2024-04-29T14:21:19.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dace321:::benchmark_data_smithi194_103039_object136196:head# 2024-04-29T14:21:19.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dad4a50:::benchmark_data_smithi194_103039_object104806:head# 2024-04-29T14:21:19.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dadd15f:::benchmark_data_smithi194_103039_object93843:head# 2024-04-29T14:21:19.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dae3eee:::benchmark_data_smithi194_103039_object123591:head# 2024-04-29T14:21:19.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dae6e73:::benchmark_data_smithi194_103039_object132911:head# 2024-04-29T14:21:19.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4daeb8bb:::benchmark_data_smithi194_103039_object44508:head# 2024-04-29T14:21:19.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4daed58d:::benchmark_data_smithi194_103039_object113881:head# 2024-04-29T14:21:19.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:19.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dafa7fa:::benchmark_data_smithi194_103039_object142632:head# 2024-04-29T14:21:19.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db108b4:::benchmark_data_smithi194_103039_object88472:head# 2024-04-29T14:21:19.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db18900:::benchmark_data_smithi194_103039_object68630:head# 2024-04-29T14:21:19.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db1f5a2:::benchmark_data_smithi194_103039_object155784:head# 2024-04-29T14:21:19.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db2275f:::benchmark_data_smithi194_103039_object144655:head# 2024-04-29T14:21:19.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db22f67:::benchmark_data_smithi194_103039_object78705:head# 2024-04-29T14:21:19.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db23c85:::benchmark_data_smithi194_103039_object66475:head# 2024-04-29T14:21:19.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db43803:::benchmark_data_smithi194_103039_object55658:head# 2024-04-29T14:21:19.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db4f794:::benchmark_data_smithi194_103039_object50219:head# 2024-04-29T14:21:19.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db53b44:::benchmark_data_smithi194_103039_object68432:head# 2024-04-29T14:21:19.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db56ffb:::benchmark_data_smithi194_103039_object72605:head# 2024-04-29T14:21:19.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db5aa1d:::benchmark_data_smithi194_103039_object87797:head# 2024-04-29T14:21:19.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db5c7c3:::benchmark_data_smithi194_103039_object89815:head# 2024-04-29T14:21:19.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db64f07:::benchmark_data_smithi194_103039_object144729:head# 2024-04-29T14:21:19.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db6fc8c:::benchmark_data_smithi194_103039_object134296:head# 2024-04-29T14:21:19.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db7823a:::benchmark_data_smithi194_103039_object43981:head# 2024-04-29T14:21:19.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db789f8:::benchmark_data_smithi194_103039_object83375:head# 2024-04-29T14:21:19.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db7aad9:::benchmark_data_smithi194_103039_object118071:head# 2024-04-29T14:21:19.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db7cbb2:::benchmark_data_smithi194_103039_object55411:head# 2024-04-29T14:21:19.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db8b28e:::benchmark_data_smithi194_103039_object61120:head# 2024-04-29T14:21:19.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db8e887:::benchmark_data_smithi194_103039_object90184:head# 2024-04-29T14:21:19.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4db9c84c:::benchmark_data_smithi194_103039_object146776:head# 2024-04-29T14:21:19.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dba3660:::benchmark_data_smithi194_103039_object59799:head# 2024-04-29T14:21:19.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dba65df:::benchmark_data_smithi194_103039_object120141:head# 2024-04-29T14:21:19.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbae343:::benchmark_data_smithi194_103039_object73202:head# 2024-04-29T14:21:19.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbb7e01:::benchmark_data_smithi194_103039_object119588:head# 2024-04-29T14:21:19.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbb8b4b:::benchmark_data_smithi194_103039_object131530:head# 2024-04-29T14:21:19.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbb9990:::benchmark_data_smithi194_103039_object64383:head# 2024-04-29T14:21:19.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbc2070:::benchmark_data_smithi194_103039_object97945:head# 2024-04-29T14:21:19.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbc39b6:::benchmark_data_smithi194_103039_object112797:head# 2024-04-29T14:21:19.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbca621:::benchmark_data_smithi194_103039_object42388:head# 2024-04-29T14:21:19.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbd9307:::benchmark_data_smithi194_103039_object96461:head# 2024-04-29T14:21:19.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbe33bd:::benchmark_data_smithi194_103039_object137399:head# 2024-04-29T14:21:19.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbe4039:::benchmark_data_smithi194_103039_object91405:head# 2024-04-29T14:21:19.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbe5dfc:::benchmark_data_smithi194_103039_object147929:head# 2024-04-29T14:21:19.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dbf0d65:::benchmark_data_smithi194_103039_object112569:head# 2024-04-29T14:21:19.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc03eef:::benchmark_data_smithi194_103039_object43249:head# 2024-04-29T14:21:19.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc24424:::benchmark_data_smithi194_103039_object147947:head# 2024-04-29T14:21:19.225 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc29cc5:::benchmark_data_smithi194_103039_object123791:head# 2024-04-29T14:21:19.225 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc30610:::benchmark_data_smithi194_103039_object136101:head# 2024-04-29T14:21:19.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc37b35:::benchmark_data_smithi194_103039_object130960:head# 2024-04-29T14:21:19.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc4696a:::benchmark_data_smithi194_103039_object102621:head# 2024-04-29T14:21:19.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc47ed0:::benchmark_data_smithi194_103039_object61919:head# 2024-04-29T14:21:19.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc48185:::benchmark_data_smithi194_103039_object105965:head# 2024-04-29T14:21:19.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc834f4:::benchmark_data_smithi194_103039_object151632:head# 2024-04-29T14:21:19.228 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc973d8:::benchmark_data_smithi194_103039_object83780:head# 2024-04-29T14:21:19.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dc97d17:::benchmark_data_smithi194_103039_object154840:head# 2024-04-29T14:21:19.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcaec9f:::benchmark_data_smithi194_103039_object149534:head# 2024-04-29T14:21:19.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcb0a78:::benchmark_data_smithi194_103039_object66036:head# 2024-04-29T14:21:19.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcb7ec8:::benchmark_data_smithi194_103039_object48183:head# 2024-04-29T14:21:19.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcbde92:::benchmark_data_smithi194_103039_object89993:head# 2024-04-29T14:21:19.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcc6c2d:::benchmark_data_smithi194_103039_object133414:head# 2024-04-29T14:21:19.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcc80e2:::benchmark_data_smithi194_103039_object86506:head# 2024-04-29T14:21:19.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dccd87e:::benchmark_data_smithi194_103039_object54494:head# 2024-04-29T14:21:19.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcd47b0:::benchmark_data_smithi194_103039_object56970:head# 2024-04-29T14:21:19.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcd74be:::benchmark_data_smithi194_103039_object115597:head# 2024-04-29T14:21:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcda681:::benchmark_data_smithi194_103039_object44537:head# 2024-04-29T14:21:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dceae6f:::benchmark_data_smithi194_103039_object110049:head# 2024-04-29T14:21:19.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcf51fb:::benchmark_data_smithi194_103039_object119783:head# 2024-04-29T14:21:19.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dcf986f:::benchmark_data_smithi194_103039_object131721:head# 2024-04-29T14:21:19.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd06d21:::benchmark_data_smithi194_103039_object59828:head# 2024-04-29T14:21:19.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd0b690:::benchmark_data_smithi194_103039_object115799:head# 2024-04-29T14:21:19.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd1357b:::benchmark_data_smithi194_103039_object114607:head# 2024-04-29T14:21:19.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd2ba2d:::benchmark_data_smithi194_103039_object146700:head# 2024-04-29T14:21:19.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd5149f:::benchmark_data_smithi194_103039_object71617:head# 2024-04-29T14:21:19.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd5fd39:::benchmark_data_smithi194_103039_object68115:head# 2024-04-29T14:21:19.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd61fd7:::benchmark_data_smithi194_103039_object113082:head# 2024-04-29T14:21:19.237 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:19.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd63788:::benchmark_data_smithi194_103039_object45607:head# 2024-04-29T14:21:19.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd6cce4:::benchmark_data_smithi194_103039_object156643:head# 2024-04-29T14:21:19.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd6e603:::benchmark_data_smithi194_103039_object90024:head# 2024-04-29T14:21:19.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd85bac:::benchmark_data_smithi194_103039_object129987:head# 2024-04-29T14:21:19.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd97b04:::benchmark_data_smithi194_103039_object97432:head# 2024-04-29T14:21:19.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dd9f9bc:::benchmark_data_smithi194_103039_object107054:head# 2024-04-29T14:21:19.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dda6921:::benchmark_data_smithi194_103039_object56739:head# 2024-04-29T14:21:19.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dda6a61:::benchmark_data_smithi194_103039_object64698:head# 2024-04-29T14:21:19.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddac57d:::benchmark_data_smithi194_103039_object50720:head# 2024-04-29T14:21:19.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddae22c:::benchmark_data_smithi194_103039_object49650:head# 2024-04-29T14:21:19.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddb1392:::benchmark_data_smithi194_103039_object85801:head# 2024-04-29T14:21:19.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddb74ec:::benchmark_data_smithi194_103039_object119321:head# 2024-04-29T14:21:19.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddc2301:::benchmark_data_smithi194_103039_object41541:head# 2024-04-29T14:21:19.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddc8560:::benchmark_data_smithi194_103039_object142055:head# 2024-04-29T14:21:19.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddd17ab:::benchmark_data_smithi194_103039_object135442:head# 2024-04-29T14:21:19.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddd33f3:::benchmark_data_smithi194_103039_object117840:head# 2024-04-29T14:21:19.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddd57d9:::benchmark_data_smithi194_103039_object76524:head# 2024-04-29T14:21:19.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dddb2d7:::benchmark_data_smithi194_103039_object67085:head# 2024-04-29T14:21:19.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddf0d4c:::benchmark_data_smithi194_103039_object121595:head# 2024-04-29T14:21:19.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddfcd25:::benchmark_data_smithi194_103039_object86010:head# 2024-04-29T14:21:19.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ddfe5a3:::benchmark_data_smithi194_103039_object46470:head# 2024-04-29T14:21:19.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de0e4f1:::benchmark_data_smithi194_103039_object79403:head# 2024-04-29T14:21:19.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de134d6:::benchmark_data_smithi194_103039_object75891:head# 2024-04-29T14:21:19.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de14096:::benchmark_data_smithi194_103039_object153036:head# 2024-04-29T14:21:19.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de1b8a3:::benchmark_data_smithi194_103039_object47089:head# 2024-04-29T14:21:19.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de42ca7:::benchmark_data_smithi194_103039_object132300:head# 2024-04-29T14:21:19.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de5243c:::benchmark_data_smithi194_103039_object44197:head# 2024-04-29T14:21:19.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de6623b:::benchmark_data_smithi194_103039_object77667:head# 2024-04-29T14:21:19.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de684f7:::benchmark_data_smithi194_103039_object45500:head# 2024-04-29T14:21:19.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de6ed5e:::benchmark_data_smithi194_103039_object42065:head# 2024-04-29T14:21:19.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de97697:::benchmark_data_smithi194_103039_object132110:head# 2024-04-29T14:21:19.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4de9c07b:::benchmark_data_smithi194_103039_object86496:head# 2024-04-29T14:21:19.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dea7488:::benchmark_data_smithi194_103039_object125629:head# 2024-04-29T14:21:19.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4deaa63d:::benchmark_data_smithi194_103039_object64929:head# 2024-04-29T14:21:19.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4debca1a:::benchmark_data_smithi194_103039_object142552:head# 2024-04-29T14:21:19.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dec558e:::benchmark_data_smithi194_103039_object55079:head# 2024-04-29T14:21:19.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4decd1a0:::benchmark_data_smithi194_103039_object115808:head# 2024-04-29T14:21:19.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ded6080:::benchmark_data_smithi194_103039_object132192:head# 2024-04-29T14:21:19.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dee914a:::benchmark_data_smithi194_103039_object106040:head# 2024-04-29T14:21:19.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4deed677:::benchmark_data_smithi194_103039_object110690:head# 2024-04-29T14:21:19.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4deee535:::benchmark_data_smithi194_103039_object136192:head# 2024-04-29T14:21:19.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4deff9ab:::benchmark_data_smithi194_103039_object71114:head# 2024-04-29T14:21:19.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df0ce17:::benchmark_data_smithi194_103039_object127589:head# 2024-04-29T14:21:19.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df107be:::benchmark_data_smithi194_103039_object124344:head# 2024-04-29T14:21:19.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df22851:::benchmark_data_smithi194_103039_object110273:head# 2024-04-29T14:21:19.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df23e06:::benchmark_data_smithi194_103039_object131456:head# 2024-04-29T14:21:19.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df34bd1:::benchmark_data_smithi194_103039_object50591:head# 2024-04-29T14:21:19.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df3af62:::benchmark_data_smithi194_103039_object119715:head# 2024-04-29T14:21:19.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df3de09:::benchmark_data_smithi194_103039_object141979:head# 2024-04-29T14:21:19.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df3fd8a:::benchmark_data_smithi194_103039_object52932:head# 2024-04-29T14:21:19.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df3fd8c:::benchmark_data_smithi194_103039_object103936:head# 2024-04-29T14:21:19.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:19.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df43303:::benchmark_data_smithi194_103039_object149052:head# 2024-04-29T14:21:19.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df47b70:::benchmark_data_smithi194_103039_object148663:head# 2024-04-29T14:21:19.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df4e70b:::benchmark_data_smithi194_103039_object92531:head# 2024-04-29T14:21:19.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df506b3:::benchmark_data_smithi194_103039_object109973:head# 2024-04-29T14:21:19.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df59032:::benchmark_data_smithi194_103039_object94616:head# 2024-04-29T14:21:19.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df5a5ea:::benchmark_data_smithi194_103039_object132978:head# 2024-04-29T14:21:19.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df5d320:::benchmark_data_smithi194_103039_object59182:head# 2024-04-29T14:21:19.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df61c5a:::benchmark_data_smithi194_103039_object86469:head# 2024-04-29T14:21:19.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df643ae:::benchmark_data_smithi194_103039_object50385:head# 2024-04-29T14:21:19.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df65fab:::benchmark_data_smithi194_103039_object95292:head# 2024-04-29T14:21:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:19.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:19.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df8e294:::benchmark_data_smithi194_103039_object93636:head# 2024-04-29T14:21:19.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df931eb:::benchmark_data_smithi194_103039_object42478:head# 2024-04-29T14:21:19.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4df98581:::benchmark_data_smithi194_103039_object103513:head# 2024-04-29T14:21:19.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dfa1109:::benchmark_data_smithi194_103039_object66387:head# 2024-04-29T14:21:19.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dfb24ea:::benchmark_data_smithi194_103039_object138239:head# 2024-04-29T14:21:19.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dfc401e:::benchmark_data_smithi194_103039_object109365:head# 2024-04-29T14:21:19.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dfd53c0:::benchmark_data_smithi194_103039_object105451:head# 2024-04-29T14:21:19.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dfe0c97:::benchmark_data_smithi194_103039_object152498:head# 2024-04-29T14:21:19.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dfe4400:::benchmark_data_smithi194_103039_object56431:head# 2024-04-29T14:21:19.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4dfee00b:::benchmark_data_smithi194_103039_object60379:head# 2024-04-29T14:21:19.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e002979:::benchmark_data_smithi194_103039_object84091:head# 2024-04-29T14:21:19.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e009972:::benchmark_data_smithi194_103039_object107020:head# 2024-04-29T14:21:19.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0115de:::benchmark_data_smithi194_103039_object85510:head# 2024-04-29T14:21:19.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e016429:::benchmark_data_smithi194_103039_object55245:head# 2024-04-29T14:21:19.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e01744b:::benchmark_data_smithi194_103039_object137268:head# 2024-04-29T14:21:19.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e01fb5a:::benchmark_data_smithi194_103039_object120683:head# 2024-04-29T14:21:19.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e028f6a:::benchmark_data_smithi194_103039_object123427:head# 2024-04-29T14:21:19.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e039d22:::benchmark_data_smithi194_103039_object66330:head# 2024-04-29T14:21:19.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e042d49:::benchmark_data_smithi194_103039_object112985:head# 2024-04-29T14:21:19.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e04b3df:::benchmark_data_smithi194_103039_object54379:head# 2024-04-29T14:21:19.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e052da0:::benchmark_data_smithi194_103039_object109281:head# 2024-04-29T14:21:19.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e063f67:::benchmark_data_smithi194_103039_object57019:head# 2024-04-29T14:21:19.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e068051:::benchmark_data_smithi194_103039_object93303:head# 2024-04-29T14:21:19.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e07ed57:::benchmark_data_smithi194_103039_object137055:head# 2024-04-29T14:21:19.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e080c2c:::benchmark_data_smithi194_103039_object132982:head# 2024-04-29T14:21:19.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e089f6b:::benchmark_data_smithi194_103039_object130073:head# 2024-04-29T14:21:19.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e08e3d3:::benchmark_data_smithi194_103039_object54284:head# 2024-04-29T14:21:19.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e08fd5f:::benchmark_data_smithi194_103039_object72188:head# 2024-04-29T14:21:19.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e09280a:::benchmark_data_smithi194_103039_object140126:head# 2024-04-29T14:21:19.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e096297:::benchmark_data_smithi194_103039_object62961:head# 2024-04-29T14:21:19.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e09a315:::benchmark_data_smithi194_103039_object94354:head# 2024-04-29T14:21:19.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0a3116:::benchmark_data_smithi194_103039_object81225:head# 2024-04-29T14:21:19.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0aed3f:::benchmark_data_smithi194_103039_object146149:head# 2024-04-29T14:21:19.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0b132e:::benchmark_data_smithi194_103039_object120716:head# 2024-04-29T14:21:19.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0b7810:::benchmark_data_smithi194_103039_object134414:head# 2024-04-29T14:21:19.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0b9a81:::benchmark_data_smithi194_103039_object50048:head# 2024-04-29T14:21:19.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0d3372:::benchmark_data_smithi194_103039_object113838:head# 2024-04-29T14:21:19.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0d53c4:::benchmark_data_smithi194_103039_object62349:head# 2024-04-29T14:21:19.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0d6841:::benchmark_data_smithi194_103039_object144364:head# 2024-04-29T14:21:19.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0ddb14:::benchmark_data_smithi194_103039_object116252:head# 2024-04-29T14:21:19.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0e2fe4:::benchmark_data_smithi194_103039_object109564:head# 2024-04-29T14:21:19.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0e529f:::benchmark_data_smithi194_103039_object145595:head# 2024-04-29T14:21:19.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0e8c52:::benchmark_data_smithi194_103039_object107693:head# 2024-04-29T14:21:19.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0eeb94:::benchmark_data_smithi194_103039_object145350:head# 2024-04-29T14:21:19.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0f1536:::benchmark_data_smithi194_103039_object134382:head# 2024-04-29T14:21:19.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0f725a:::benchmark_data_smithi194_103039_object94831:head# 2024-04-29T14:21:19.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e0f9d36:::benchmark_data_smithi194_103039_object50418:head# 2024-04-29T14:21:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e10632b:::benchmark_data_smithi194_103039_object118200:head# 2024-04-29T14:21:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e10cfa3:::benchmark_data_smithi194_103039_object80419:head# 2024-04-29T14:21:19.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e115f54:::benchmark_data_smithi194_103039_object137107:head# 2024-04-29T14:21:19.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e12f2ef:::benchmark_data_smithi194_103039_object43853:head# 2024-04-29T14:21:19.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e12f997:::benchmark_data_smithi194_103039_object58880:head# 2024-04-29T14:21:19.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e13b57c:::benchmark_data_smithi194_103039_object82655:head# 2024-04-29T14:21:19.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e13c9ff:::benchmark_data_smithi194_103039_object46690:head# 2024-04-29T14:21:19.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e13dff2:::benchmark_data_smithi194_103039_object71380:head# 2024-04-29T14:21:19.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e13e053:::benchmark_data_smithi194_103039_object86785:head# 2024-04-29T14:21:19.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e141c97:::benchmark_data_smithi194_103039_object137379:head# 2024-04-29T14:21:19.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e141e83:::benchmark_data_smithi194_103039_object123381:head# 2024-04-29T14:21:19.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e146f2c:::benchmark_data_smithi194_103039_object97657:head# 2024-04-29T14:21:19.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e15142a:::benchmark_data_smithi194_103039_object119558:head# 2024-04-29T14:21:19.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1564dd:::benchmark_data_smithi194_103039_object109102:head# 2024-04-29T14:21:19.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e15d664:::benchmark_data_smithi194_103039_object92971:head# 2024-04-29T14:21:19.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e17c2b8:::benchmark_data_smithi194_103039_object46327:head# 2024-04-29T14:21:19.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1882f7:::benchmark_data_smithi194_103039_object51090:head# 2024-04-29T14:21:19.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e18b340:::benchmark_data_smithi194_103039_object141760:head# 2024-04-29T14:21:19.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1972d6:::benchmark_data_smithi194_103039_object76439:head# 2024-04-29T14:21:19.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e19b7c7:::benchmark_data_smithi194_103039_object145543:head# 2024-04-29T14:21:19.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e19d680:::benchmark_data_smithi194_103039_object64829:head# 2024-04-29T14:21:19.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1a4994:::benchmark_data_smithi194_103039_object61255:head# 2024-04-29T14:21:19.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1a965e:::benchmark_data_smithi194_103039_object145740:head# 2024-04-29T14:21:19.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1ac1dd:::benchmark_data_smithi194_103039_object112268:head# 2024-04-29T14:21:19.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1ba51f:::benchmark_data_smithi194_103039_object87876:head# 2024-04-29T14:21:19.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1c4e6c:::benchmark_data_smithi194_103039_object120014:head# 2024-04-29T14:21:19.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1c8f3c:::benchmark_data_smithi194_103039_object119522:head# 2024-04-29T14:21:19.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1dbe3a:::benchmark_data_smithi194_103039_object60940:head# 2024-04-29T14:21:19.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1e22ab:::benchmark_data_smithi194_103039_object122652:head# 2024-04-29T14:21:19.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1e53ac:::benchmark_data_smithi194_103039_object109337:head# 2024-04-29T14:21:19.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1f16a4:::benchmark_data_smithi194_103039_object106674:head# 2024-04-29T14:21:19.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e1fa166:::benchmark_data_smithi194_103039_object155641:head# 2024-04-29T14:21:19.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e202010:::benchmark_data_smithi194_103039_object65506:head# 2024-04-29T14:21:19.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2072f5:::benchmark_data_smithi194_103039_object115636:head# 2024-04-29T14:21:19.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e216e71:::benchmark_data_smithi194_103039_object134810:head# 2024-04-29T14:21:19.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2270ea:::benchmark_data_smithi194_103039_object64434:head# 2024-04-29T14:21:19.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e228892:::benchmark_data_smithi194_103039_object67347:head# 2024-04-29T14:21:19.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e22c50f:::benchmark_data_smithi194_103039_object67261:head# 2024-04-29T14:21:19.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e24faf2:::benchmark_data_smithi194_103039_object136404:head# 2024-04-29T14:21:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e25caa5:::benchmark_data_smithi194_103039_object102025:head# 2024-04-29T14:21:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e26154e:::benchmark_data_smithi194_103039_object125058:head# 2024-04-29T14:21:19.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e264e42:::benchmark_data_smithi194_103039_object129079:head# 2024-04-29T14:21:19.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e26cafc:::benchmark_data_smithi194_103039_object95937:head# 2024-04-29T14:21:19.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e26cd2c:::benchmark_data_smithi194_103039_object151172:head# 2024-04-29T14:21:19.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2829e4:::benchmark_data_smithi194_103039_object110818:head# 2024-04-29T14:21:19.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2870de:::benchmark_data_smithi194_103039_object130148:head# 2024-04-29T14:21:19.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e295573:::benchmark_data_smithi194_103039_object134292:head# 2024-04-29T14:21:19.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e29df5d:::benchmark_data_smithi194_103039_object109196:head# 2024-04-29T14:21:19.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e29eb95:::benchmark_data_smithi194_103039_object63159:head# 2024-04-29T14:21:19.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2af6bf:::benchmark_data_smithi194_103039_object89579:head# 2024-04-29T14:21:19.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2b1d48:::benchmark_data_smithi194_103039_object87756:head# 2024-04-29T14:21:19.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2cc4ba:::benchmark_data_smithi194_103039_object112680:head# 2024-04-29T14:21:19.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2d9d53:::benchmark_data_smithi194_103039_object52820:head# 2024-04-29T14:21:19.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2e4a91:::benchmark_data_smithi194_103039_object46945:head# 2024-04-29T14:21:19.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2e64de:::benchmark_data_smithi194_103039_object53848:head# 2024-04-29T14:21:19.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2fb1e5:::benchmark_data_smithi194_103039_object77438:head# 2024-04-29T14:21:19.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2fd9eb:::benchmark_data_smithi194_103039_object145052:head# 2024-04-29T14:21:19.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e2fe395:::benchmark_data_smithi194_103039_object48520:head# 2024-04-29T14:21:19.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e30197c:::benchmark_data_smithi194_103039_object146244:head# 2024-04-29T14:21:19.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e304067:::benchmark_data_smithi194_103039_object70997:head# 2024-04-29T14:21:19.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e306571:::benchmark_data_smithi194_103039_object103988:head# 2024-04-29T14:21:19.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e311e7f:::benchmark_data_smithi194_103039_object132074:head# 2024-04-29T14:21:19.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e313f98:::benchmark_data_smithi194_103039_object116939:head# 2024-04-29T14:21:19.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e315ca5:::benchmark_data_smithi194_103039_object45344:head# 2024-04-29T14:21:19.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3178b4:::benchmark_data_smithi194_103039_object124089:head# 2024-04-29T14:21:19.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e31a548:::benchmark_data_smithi194_103039_object64869:head# 2024-04-29T14:21:19.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3261a3:::benchmark_data_smithi194_103039_object109050:head# 2024-04-29T14:21:19.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e328250:::benchmark_data_smithi194_103039_object99149:head# 2024-04-29T14:21:19.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3304bc:::benchmark_data_smithi194_103039_object99119:head# 2024-04-29T14:21:19.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e33af22:::benchmark_data_smithi194_103039_object63168:head# 2024-04-29T14:21:19.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3429c8:::benchmark_data_smithi194_103039_object127201:head# 2024-04-29T14:21:19.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e34787d:::benchmark_data_smithi194_103039_object42418:head# 2024-04-29T14:21:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e348e68:::benchmark_data_smithi194_103039_object98770:head# 2024-04-29T14:21:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e349a67:::benchmark_data_smithi194_103039_object86428:head# 2024-04-29T14:21:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e351008:::benchmark_data_smithi194_103039_object146959:head# 2024-04-29T14:21:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3627f2:::benchmark_data_smithi194_103039_object55827:head# 2024-04-29T14:21:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e38057c:::benchmark_data_smithi194_103039_object130037:head# 2024-04-29T14:21:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e388215:::benchmark_data_smithi194_103039_object106715:head# 2024-04-29T14:21:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3890b4:::benchmark_data_smithi194_103039_object137095:head# 2024-04-29T14:21:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e38dcbc:::benchmark_data_smithi194_103039_object72294:head# 2024-04-29T14:21:19.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e39f911:::benchmark_data_smithi194_103039_object146096:head# 2024-04-29T14:21:19.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3a22e8:::benchmark_data_smithi194_103039_object68931:head# 2024-04-29T14:21:19.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3af0a9:::benchmark_data_smithi194_103039_object108583:head# 2024-04-29T14:21:19.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3cf714:::benchmark_data_smithi194_103039_object106385:head# 2024-04-29T14:21:19.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3d6385:::benchmark_data_smithi194_103039_object94901:head# 2024-04-29T14:21:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3d9ef6:::benchmark_data_smithi194_103039_object60060:head# 2024-04-29T14:21:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3ded79:::benchmark_data_smithi194_103039_object86179:head# 2024-04-29T14:21:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3e0c05:::benchmark_data_smithi194_103039_object105507:head# 2024-04-29T14:21:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3eef1d:::benchmark_data_smithi194_103039_object91209:head# 2024-04-29T14:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3f0de5:::benchmark_data_smithi194_103039_object88435:head# 2024-04-29T14:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3f2891:::benchmark_data_smithi194_103039_object72217:head# 2024-04-29T14:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3f61b2:::benchmark_data_smithi194_103039_object87924:head# 2024-04-29T14:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3ff9aa:::benchmark_data_smithi194_103039_object142316:head# 2024-04-29T14:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e3ffdfd:::benchmark_data_smithi194_103039_object121253:head# 2024-04-29T14:21:19.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e40fa33:::benchmark_data_smithi194_103039_object136026:head# 2024-04-29T14:21:19.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e415b65:::benchmark_data_smithi194_103039_object84670:head# 2024-04-29T14:21:19.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e41c002:::benchmark_data_smithi194_103039_object63407:head# 2024-04-29T14:21:19.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e41ebc2:::benchmark_data_smithi194_103039_object120594:head# 2024-04-29T14:21:19.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e450968:::benchmark_data_smithi194_103039_object85622:head# 2024-04-29T14:21:19.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e46c56c:::benchmark_data_smithi194_103039_object136487:head# 2024-04-29T14:21:19.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e483178:::benchmark_data_smithi194_103039_object114507:head# 2024-04-29T14:21:19.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:19 smithi040 ceph-mon[39581]: pgmap v982: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 149 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:19.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:19 smithi040 ceph-mon[39581]: 4.0 scrub starts 2024-04-29T14:21:19.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:19 smithi040 ceph-mon[39581]: 4.0 scrub ok 2024-04-29T14:21:19.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e484261:::benchmark_data_smithi194_103039_object113067:head# 2024-04-29T14:21:19.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4946a5:::benchmark_data_smithi194_103039_object114542:head# 2024-04-29T14:21:19.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e49ed5b:::benchmark_data_smithi194_103039_object69918:head# 2024-04-29T14:21:19.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4ac18d:::benchmark_data_smithi194_103039_object127247:head# 2024-04-29T14:21:19.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4b30ca:::benchmark_data_smithi194_103039_object136728:head# 2024-04-29T14:21:19.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4bba57:::benchmark_data_smithi194_103039_object74856:head# 2024-04-29T14:21:19.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4c41f3:::benchmark_data_smithi194_103039_object53626:head# 2024-04-29T14:21:19.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4c5096:::benchmark_data_smithi194_103039_object147524:head# 2024-04-29T14:21:19.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4d3765:::benchmark_data_smithi194_103039_object77990:head# 2024-04-29T14:21:19.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4d507e:::benchmark_data_smithi194_103039_object142629:head# 2024-04-29T14:21:19.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4e599f:::benchmark_data_smithi194_103039_object71173:head# 2024-04-29T14:21:19.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e4f276e:::benchmark_data_smithi194_103039_object153795:head# 2024-04-29T14:21:19.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e502ef2:::benchmark_data_smithi194_103039_object87798:head# 2024-04-29T14:21:19.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e503a02:::benchmark_data_smithi194_103039_object111637:head# 2024-04-29T14:21:19.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e50a2f2:::benchmark_data_smithi194_103039_object93466:head# 2024-04-29T14:21:19.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e50dac8:::benchmark_data_smithi194_103039_object124550:head# 2024-04-29T14:21:19.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e51ff1b:::benchmark_data_smithi194_103039_object72904:head# 2024-04-29T14:21:19.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e524209:::benchmark_data_smithi194_103039_object119867:head# 2024-04-29T14:21:19.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5249a6:::benchmark_data_smithi194_103039_object124789:head# 2024-04-29T14:21:19.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e527223:::benchmark_data_smithi194_103039_object113630:head# 2024-04-29T14:21:19.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e52907b:::benchmark_data_smithi194_103039_object79720:head# 2024-04-29T14:21:19.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5294e0:::benchmark_data_smithi194_103039_object49981:head# 2024-04-29T14:21:19.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e530f1d:::benchmark_data_smithi194_103039_object128722:head# 2024-04-29T14:21:19.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e534a41:::benchmark_data_smithi194_103039_object95230:head# 2024-04-29T14:21:19.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5371cb:::benchmark_data_smithi194_103039_object50433:head# 2024-04-29T14:21:19.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e540673:::benchmark_data_smithi194_103039_object142661:head# 2024-04-29T14:21:19.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5459fe:::benchmark_data_smithi194_103039_object99977:head# 2024-04-29T14:21:19.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e556293:::benchmark_data_smithi194_103039_object64692:head# 2024-04-29T14:21:19.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e55985b:::benchmark_data_smithi194_103039_object102906:head# 2024-04-29T14:21:19.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e570806:::benchmark_data_smithi194_103039_object55991:head# 2024-04-29T14:21:19.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e572178:::benchmark_data_smithi194_103039_object105338:head# 2024-04-29T14:21:19.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5779be:::benchmark_data_smithi194_103039_object155217:head# 2024-04-29T14:21:19.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e59e67c:::benchmark_data_smithi194_103039_object131803:head# 2024-04-29T14:21:19.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5a0dcc:::benchmark_data_smithi194_103039_object41534:head# 2024-04-29T14:21:19.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5a0f7e:::benchmark_data_smithi194_103039_object88080:head# 2024-04-29T14:21:19.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5a2d09:::benchmark_data_smithi194_103039_object80358:head# 2024-04-29T14:21:19.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5a4384:::benchmark_data_smithi194_103039_object71958:head# 2024-04-29T14:21:19.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5bb745:::benchmark_data_smithi194_103039_object156108:head# 2024-04-29T14:21:19.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5c082d:::benchmark_data_smithi194_103039_object117989:head# 2024-04-29T14:21:19.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5c97d6:::benchmark_data_smithi194_103039_object104748:head# 2024-04-29T14:21:19.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5d8f4b:::benchmark_data_smithi194_103039_object140236:head# 2024-04-29T14:21:19.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5ed9a3:::benchmark_data_smithi194_103039_object86680:head# 2024-04-29T14:21:19.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e5f2a25:::benchmark_data_smithi194_103039_object69121:head# 2024-04-29T14:21:19.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6048dc:::benchmark_data_smithi194_103039_object149134:head# 2024-04-29T14:21:19.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e606042:::benchmark_data_smithi194_103039_object154650:head# 2024-04-29T14:21:19.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e607249:::benchmark_data_smithi194_103039_object104435:head# 2024-04-29T14:21:19.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e616cca:::benchmark_data_smithi194_103039_object82128:head# 2024-04-29T14:21:19.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e61aa7f:::benchmark_data_smithi194_103039_object101681:head# 2024-04-29T14:21:19.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e61f2f8:::benchmark_data_smithi194_103039_object153571:head# 2024-04-29T14:21:19.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e63ca94:::benchmark_data_smithi194_103039_object124396:head# 2024-04-29T14:21:19.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e65a185:::benchmark_data_smithi194_103039_object86508:head# 2024-04-29T14:21:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6682d1:::benchmark_data_smithi194_103039_object60338:head# 2024-04-29T14:21:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e66a6af:::benchmark_data_smithi194_103039_object132558:head# 2024-04-29T14:21:19.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e66d790:::benchmark_data_smithi194_103039_object97043:head# 2024-04-29T14:21:19.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6790e1:::benchmark_data_smithi194_103039_object109989:head# 2024-04-29T14:21:19.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e67c468:::benchmark_data_smithi194_103039_object77942:head# 2024-04-29T14:21:19.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e67d332:::benchmark_data_smithi194_103039_object110232:head# 2024-04-29T14:21:19.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6850ad:::benchmark_data_smithi194_103039_object84820:head# 2024-04-29T14:21:19.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6888d9:::benchmark_data_smithi194_103039_object54589:head# 2024-04-29T14:21:19.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e68c720:::benchmark_data_smithi194_103039_object41537:head# 2024-04-29T14:21:19.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e68e79f:::benchmark_data_smithi194_103039_object138642:head# 2024-04-29T14:21:19.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e695397:::benchmark_data_smithi194_103039_object108490:head# 2024-04-29T14:21:19.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e69cf57:::benchmark_data_smithi194_103039_object152165:head# 2024-04-29T14:21:19.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6aa84b:::benchmark_data_smithi194_103039_object47172:head# 2024-04-29T14:21:19.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6bd7f0:::benchmark_data_smithi194_103039_object121160:head# 2024-04-29T14:21:19.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6c1142:::benchmark_data_smithi194_103039_object57658:head# 2024-04-29T14:21:19.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6c3470:::benchmark_data_smithi194_103039_object126891:head# 2024-04-29T14:21:19.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6c9ecb:::benchmark_data_smithi194_103039_object60876:head# 2024-04-29T14:21:19.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6cf505:::benchmark_data_smithi194_103039_object50022:head# 2024-04-29T14:21:19.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6d11bc:::benchmark_data_smithi194_103039_object93028:head# 2024-04-29T14:21:19.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6d347d:::benchmark_data_smithi194_103039_object86292:head# 2024-04-29T14:21:19.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6e7334:::benchmark_data_smithi194_103039_object55229:head# 2024-04-29T14:21:19.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6e7727:::benchmark_data_smithi194_103039_object91901:head# 2024-04-29T14:21:19.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6e94e7:::benchmark_data_smithi194_103039_object105759:head# 2024-04-29T14:21:19.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e6fc17c:::benchmark_data_smithi194_103039_object103762:head# 2024-04-29T14:21:19.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e701051:::benchmark_data_smithi194_103039_object48874:head# 2024-04-29T14:21:19.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e714134:::benchmark_data_smithi194_103039_object115291:head# 2024-04-29T14:21:19.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e71c430:::benchmark_data_smithi194_103039_object131868:head# 2024-04-29T14:21:19.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e73dd47:::benchmark_data_smithi194_103039_object54470:head# 2024-04-29T14:21:19.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e753935:::benchmark_data_smithi194_103039_object146506:head# 2024-04-29T14:21:19.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e758c68:::benchmark_data_smithi194_103039_object43874:head# 2024-04-29T14:21:19.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e75d38e:::benchmark_data_smithi194_103039_object114739:head# 2024-04-29T14:21:19.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e7610c3:::benchmark_data_smithi194_103039_object47966:head# 2024-04-29T14:21:19.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e761bfc:::benchmark_data_smithi194_103039_object42902:head# 2024-04-29T14:21:19.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e762b3a:::benchmark_data_smithi194_103039_object81127:head# 2024-04-29T14:21:19.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e772eda:::benchmark_data_smithi194_103039_object105104:head# 2024-04-29T14:21:19.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e785455:::benchmark_data_smithi194_103039_object74618:head# 2024-04-29T14:21:19.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e787a61:::benchmark_data_smithi194_103039_object56848:head# 2024-04-29T14:21:19.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e787d73:::benchmark_data_smithi194_103039_object58906:head# 2024-04-29T14:21:19.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e797ec9:::benchmark_data_smithi194_103039_object110218:head# 2024-04-29T14:21:19.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e7b1e07:::benchmark_data_smithi194_103039_object135798:head# 2024-04-29T14:21:19.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e7b64d7:::benchmark_data_smithi194_103039_object108328:head# 2024-04-29T14:21:19.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:19.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e7ecc48:::benchmark_data_smithi194_103039_object59545:head# 2024-04-29T14:21:19.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e7f65da:::benchmark_data_smithi194_103039_object85939:head# 2024-04-29T14:21:19.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e7f7f8d:::benchmark_data_smithi194_103039_object110289:head# 2024-04-29T14:21:19.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e804cb6:::benchmark_data_smithi194_103039_object102654:head# 2024-04-29T14:21:19.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e813409:::benchmark_data_smithi194_103039_object50161:head# 2024-04-29T14:21:19.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e813646:::benchmark_data_smithi194_103039_object68837:head# 2024-04-29T14:21:19.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e81545b:::benchmark_data_smithi194_103039_object148824:head# 2024-04-29T14:21:19.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e81779c:::benchmark_data_smithi194_103039_object67220:head# 2024-04-29T14:21:19.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8243bd:::benchmark_data_smithi194_103039_object142497:head# 2024-04-29T14:21:19.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e825611:::benchmark_data_smithi194_103039_object123674:head# 2024-04-29T14:21:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e835d87:::benchmark_data_smithi194_103039_object70126:head# 2024-04-29T14:21:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8367d8:::benchmark_data_smithi194_103039_object76423:head# 2024-04-29T14:21:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8376fd:::benchmark_data_smithi194_103039_object154960:head# 2024-04-29T14:21:19.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e853283:::benchmark_data_smithi194_103039_object77973:head# 2024-04-29T14:21:19.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e86f2ff:::benchmark_data_smithi194_103039_object120622:head# 2024-04-29T14:21:19.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e876068:::benchmark_data_smithi194_103039_object151131:head# 2024-04-29T14:21:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8771f7:::benchmark_data_smithi194_103039_object52759:head# 2024-04-29T14:21:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e877b69:::benchmark_data_smithi194_103039_object49958:head# 2024-04-29T14:21:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e882699:::benchmark_data_smithi194_103039_object129830:head# 2024-04-29T14:21:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e884194:::benchmark_data_smithi194_103039_object51574:head# 2024-04-29T14:21:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e889e0e:::benchmark_data_smithi194_103039_object142544:head# 2024-04-29T14:21:19.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e898758:::benchmark_data_smithi194_103039_object63192:head# 2024-04-29T14:21:19.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e89bc3a:::benchmark_data_smithi194_103039_object123565:head# 2024-04-29T14:21:19.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e89cb07:::benchmark_data_smithi194_103039_object138419:head# 2024-04-29T14:21:19.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:19 smithi121 ceph-mon[44533]: pgmap v982: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 149 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:19.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:19 smithi121 ceph-mon[44533]: 4.0 scrub starts 2024-04-29T14:21:19.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:19 smithi121 ceph-mon[44533]: 4.0 scrub ok 2024-04-29T14:21:19.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e89ecc0:::benchmark_data_smithi194_103039_object55480:head# 2024-04-29T14:21:19.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e89eed4:::benchmark_data_smithi194_103039_object62118:head# 2024-04-29T14:21:19.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8a3e4b:::benchmark_data_smithi194_103039_object151784:head# 2024-04-29T14:21:19.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8a9338:::benchmark_data_smithi194_103039_object149360:head# 2024-04-29T14:21:19.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8c26b8:::benchmark_data_smithi194_103039_object89686:head# 2024-04-29T14:21:19.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8cf539:::benchmark_data_smithi194_103039_object41875:head# 2024-04-29T14:21:19.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8d2543:::benchmark_data_smithi194_103039_object138455:head# 2024-04-29T14:21:19.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8d4133:::benchmark_data_smithi194_103039_object130620:head# 2024-04-29T14:21:19.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e8d77e8:::benchmark_data_smithi194_103039_object75683:head# 2024-04-29T14:21:19.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9034ea:::benchmark_data_smithi194_103039_object49150:head# 2024-04-29T14:21:19.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e903c5f:::benchmark_data_smithi194_103039_object118132:head# 2024-04-29T14:21:19.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9053d2:::benchmark_data_smithi194_103039_object89793:head# 2024-04-29T14:21:19.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e90d9a1:::benchmark_data_smithi194_103039_object43658:head# 2024-04-29T14:21:19.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e92d6b2:::benchmark_data_smithi194_103039_object42028:head# 2024-04-29T14:21:19.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e959806:::benchmark_data_smithi194_103039_object66263:head# 2024-04-29T14:21:19.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e96112e:::benchmark_data_smithi194_103039_object88566:head# 2024-04-29T14:21:19.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e961b2b:::benchmark_data_smithi194_103039_object59487:head# 2024-04-29T14:21:19.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e96a4a3:::benchmark_data_smithi194_103039_object50266:head# 2024-04-29T14:21:19.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e96ab48:::benchmark_data_smithi194_103039_object64310:head# 2024-04-29T14:21:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e985154:::benchmark_data_smithi194_103039_object122162:head# 2024-04-29T14:21:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9b694d:::benchmark_data_smithi194_103039_object131275:head# 2024-04-29T14:21:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9c225f:::benchmark_data_smithi194_103039_object81256:head# 2024-04-29T14:21:19.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9c3189:::benchmark_data_smithi194_103039_object50454:head# 2024-04-29T14:21:19.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9c3f70:::benchmark_data_smithi194_103039_object146652:head# 2024-04-29T14:21:19.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9c647f:::benchmark_data_smithi194_103039_object114340:head# 2024-04-29T14:21:19.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9c9e1d:::benchmark_data_smithi194_103039_object48854:head# 2024-04-29T14:21:19.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9cdb60:::benchmark_data_smithi194_103039_object107737:head# 2024-04-29T14:21:19.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9d08e7:::benchmark_data_smithi194_103039_object81314:head# 2024-04-29T14:21:19.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9d58a2:::benchmark_data_smithi194_103039_object110703:head# 2024-04-29T14:21:19.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9d5e62:::benchmark_data_smithi194_103039_object73908:head# 2024-04-29T14:21:19.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9ea940:::benchmark_data_smithi194_103039_object145281:head# 2024-04-29T14:21:19.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4e9f7001:::benchmark_data_smithi194_103039_object62864:head# 2024-04-29T14:21:19.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea17370:::benchmark_data_smithi194_103039_object105171:head# 2024-04-29T14:21:19.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea17e93:::benchmark_data_smithi194_103039_object153768:head# 2024-04-29T14:21:19.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea2398c:::benchmark_data_smithi194_103039_object142747:head# 2024-04-29T14:21:19.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea2ba97:::benchmark_data_smithi194_103039_object52278:head# 2024-04-29T14:21:19.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea2d75f:::benchmark_data_smithi194_103039_object92402:head# 2024-04-29T14:21:19.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea32fab:::benchmark_data_smithi194_103039_object149007:head# 2024-04-29T14:21:19.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea3efb0:::benchmark_data_smithi194_103039_object85985:head# 2024-04-29T14:21:19.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea4fbb4:::benchmark_data_smithi194_103039_object81176:head# 2024-04-29T14:21:19.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea5a6bc:::benchmark_data_smithi194_103039_object146665:head# 2024-04-29T14:21:19.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea5b881:::benchmark_data_smithi194_103039_object115047:head# 2024-04-29T14:21:19.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea5cf22:::benchmark_data_smithi194_103039_object68440:head# 2024-04-29T14:21:19.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea5d37b:::benchmark_data_smithi194_103039_object117206:head# 2024-04-29T14:21:19.392 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea66367:::benchmark_data_smithi194_103039_object104421:head# 2024-04-29T14:21:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea6c3b0:::benchmark_data_smithi194_103039_object82884:head# 2024-04-29T14:21:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea6cec2:::benchmark_data_smithi194_103039_object79243:head# 2024-04-29T14:21:19.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea6db3a:::benchmark_data_smithi194_103039_object94392:head# 2024-04-29T14:21:19.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea708d1:::benchmark_data_smithi194_103039_object46011:head# 2024-04-29T14:21:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea715cb:::benchmark_data_smithi194_103039_object123357:head# 2024-04-29T14:21:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea859c5:::benchmark_data_smithi194_103039_object141185:head# 2024-04-29T14:21:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea947d7:::benchmark_data_smithi194_103039_object153836:head# 2024-04-29T14:21:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ea99f28:::benchmark_data_smithi194_103039_object55352:head# 2024-04-29T14:21:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eaabd45:::benchmark_data_smithi194_103039_object134934:head# 2024-04-29T14:21:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eab1bf2:::benchmark_data_smithi194_103039_object64237:head# 2024-04-29T14:21:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eab44d3:::benchmark_data_smithi194_103039_object64319:head# 2024-04-29T14:21:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eac031e:::benchmark_data_smithi194_103039_object50341:head# 2024-04-29T14:21:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eac7e6b:::benchmark_data_smithi194_103039_object70817:head# 2024-04-29T14:21:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eac8824:::benchmark_data_smithi194_103039_object112325:head# 2024-04-29T14:21:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eaca1ba:::benchmark_data_smithi194_103039_object72442:head# 2024-04-29T14:21:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eae14ce:::benchmark_data_smithi194_103039_object129381:head# 2024-04-29T14:21:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eaf5c23:::benchmark_data_smithi194_103039_object106547:head# 2024-04-29T14:21:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eafe2a4:::benchmark_data_smithi194_103039_object129117:head# 2024-04-29T14:21:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb04333:::benchmark_data_smithi194_103039_object73530:head# 2024-04-29T14:21:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb06e25:::benchmark_data_smithi194_103039_object146817:head# 2024-04-29T14:21:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb0a8cc:::benchmark_data_smithi194_103039_object54847:head# 2024-04-29T14:21:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb13dc7:::benchmark_data_smithi194_103039_object83771:head# 2024-04-29T14:21:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb14a1b:::benchmark_data_smithi194_103039_object88388:head# 2024-04-29T14:21:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb1b278:::benchmark_data_smithi194_103039_object76474:head# 2024-04-29T14:21:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb22aa9:::benchmark_data_smithi194_103039_object63820:head# 2024-04-29T14:21:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb23324:::benchmark_data_smithi194_103039_object75304:head# 2024-04-29T14:21:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb2b6bf:::benchmark_data_smithi194_103039_object63200:head# 2024-04-29T14:21:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb2f6b7:::benchmark_data_smithi194_103039_object126868:head# 2024-04-29T14:21:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb3b34a:::benchmark_data_smithi194_103039_object114211:head# 2024-04-29T14:21:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb3d9aa:::benchmark_data_smithi194_103039_object63954:head# 2024-04-29T14:21:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb3ef74:::benchmark_data_smithi194_103039_object87754:head# 2024-04-29T14:21:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb46dde:::benchmark_data_smithi194_103039_object56029:head# 2024-04-29T14:21:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb4d38d:::benchmark_data_smithi194_103039_object108462:head# 2024-04-29T14:21:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb56179:::benchmark_data_smithi194_103039_object154687:head# 2024-04-29T14:21:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb564e5:::benchmark_data_smithi194_103039_object95191:head# 2024-04-29T14:21:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb59304:::benchmark_data_smithi194_103039_object97918:head# 2024-04-29T14:21:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb5b6d4:::benchmark_data_smithi194_103039_object144463:head# 2024-04-29T14:21:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb5bda4:::benchmark_data_smithi194_103039_object72484:head# 2024-04-29T14:21:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb630ab:::benchmark_data_smithi194_103039_object115046:head# 2024-04-29T14:21:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb66ea0:::benchmark_data_smithi194_103039_object128110:head# 2024-04-29T14:21:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb67812:::benchmark_data_smithi194_103039_object61498:head# 2024-04-29T14:21:19.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb6efb3:::benchmark_data_smithi194_103039_object72503:head# 2024-04-29T14:21:19.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb79de7:::benchmark_data_smithi194_103039_object95609:head# 2024-04-29T14:21:19.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb7c730:::benchmark_data_smithi194_103039_object68968:head# 2024-04-29T14:21:19.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb7d8b1:::benchmark_data_smithi194_103039_object104637:head# 2024-04-29T14:21:19.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb86ab0:::benchmark_data_smithi194_103039_object97121:head# 2024-04-29T14:21:19.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:19.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb94fe2:::benchmark_data_smithi194_103039_object111280:head# 2024-04-29T14:21:19.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eb9de2a:::benchmark_data_smithi194_103039_object136036:head# 2024-04-29T14:21:19.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eba475d:::benchmark_data_smithi194_103039_object130759:head# 2024-04-29T14:21:19.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebafe66:::benchmark_data_smithi194_103039_object95220:head# 2024-04-29T14:21:19.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebb44e8:::benchmark_data_smithi194_103039_object154088:head# 2024-04-29T14:21:19.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebc44ac:::benchmark_data_smithi194_103039_object72156:head# 2024-04-29T14:21:19.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:19.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:19.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebc6ae3:::benchmark_data_smithi194_103039_object135623:head# 2024-04-29T14:21:19.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebc89f6:::benchmark_data_smithi194_103039_object54630:head# 2024-04-29T14:21:19.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebd377d:::benchmark_data_smithi194_103039_object121751:head# 2024-04-29T14:21:19.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebe9351:::benchmark_data_smithi194_103039_object77168:head# 2024-04-29T14:21:19.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebed62f:::benchmark_data_smithi194_103039_object96555:head# 2024-04-29T14:21:19.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebfa414:::benchmark_data_smithi194_103039_object49036:head# 2024-04-29T14:21:19.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ebff851:::benchmark_data_smithi194_103039_object130170:head# 2024-04-29T14:21:19.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec0796f:::benchmark_data_smithi194_103039_object95790:head# 2024-04-29T14:21:19.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec0bdc4:::benchmark_data_smithi194_103039_object97874:head# 2024-04-29T14:21:19.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec1c5b9:::benchmark_data_smithi194_103039_object51092:head# 2024-04-29T14:21:19.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec29b4a:::benchmark_data_smithi194_103039_object61413:head# 2024-04-29T14:21:19.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec410e2:::benchmark_data_smithi194_103039_object153324:head# 2024-04-29T14:21:19.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec459b8:::benchmark_data_smithi194_103039_object156517:head# 2024-04-29T14:21:19.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec5d6ed:::benchmark_data_smithi194_103039_object102441:head# 2024-04-29T14:21:19.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec616d3:::benchmark_data_smithi194_103039_object148106:head# 2024-04-29T14:21:19.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec626ff:::benchmark_data_smithi194_103039_object150056:head# 2024-04-29T14:21:19.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec6821d:::benchmark_data_smithi194_103039_object76209:head# 2024-04-29T14:21:19.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec6ba3e:::benchmark_data_smithi194_103039_object90295:head# 2024-04-29T14:21:19.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec77a80:::benchmark_data_smithi194_103039_object49813:head# 2024-04-29T14:21:19.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec7d678:::benchmark_data_smithi194_103039_object120711:head# 2024-04-29T14:21:19.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ec89749:::benchmark_data_smithi194_103039_object48188:head# 2024-04-29T14:21:19.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eca6f46:::benchmark_data_smithi194_103039_object47840:head# 2024-04-29T14:21:19.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecaca0c:::benchmark_data_smithi194_103039_object87217:head# 2024-04-29T14:21:19.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecb0e7a:::benchmark_data_smithi194_103039_object135248:head# 2024-04-29T14:21:19.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecc0f76:::benchmark_data_smithi194_103039_object118282:head# 2024-04-29T14:21:19.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecc13de:::benchmark_data_smithi194_103039_object81002:head# 2024-04-29T14:21:19.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecd2e65:::benchmark_data_smithi194_103039_object47944:head# 2024-04-29T14:21:19.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecda2a1:::benchmark_data_smithi194_103039_object73345:head# 2024-04-29T14:21:19.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ece4e42:::benchmark_data_smithi194_103039_object106857:head# 2024-04-29T14:21:19.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ece6102:::benchmark_data_smithi194_103039_object153820:head# 2024-04-29T14:21:19.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ece8892:::benchmark_data_smithi194_103039_object120169:head# 2024-04-29T14:21:19.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecee055:::benchmark_data_smithi194_103039_object65419:head# 2024-04-29T14:21:19.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecf3b44:::benchmark_data_smithi194_103039_object125187:head# 2024-04-29T14:21:19.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecf3eff:::benchmark_data_smithi194_103039_object117217:head# 2024-04-29T14:21:19.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ecf48a6:::benchmark_data_smithi194_103039_object152386:head# 2024-04-29T14:21:19.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed0092a:::benchmark_data_smithi194_103039_object144674:head# 2024-04-29T14:21:19.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed06ff7:::benchmark_data_smithi194_103039_object88765:head# 2024-04-29T14:21:19.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed122fa:::benchmark_data_smithi194_103039_object116006:head# 2024-04-29T14:21:19.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed1b517:::benchmark_data_smithi194_103039_object79570:head# 2024-04-29T14:21:19.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed24b53:::benchmark_data_smithi194_103039_object130065:head# 2024-04-29T14:21:19.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed2a32a:::benchmark_data_smithi194_103039_object53998:head# 2024-04-29T14:21:19.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed2c75c:::benchmark_data_smithi194_103039_object125009:head# 2024-04-29T14:21:19.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed2f468:::benchmark_data_smithi194_103039_object74873:head# 2024-04-29T14:21:19.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed4b095:::benchmark_data_smithi194_103039_object71602:head# 2024-04-29T14:21:19.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed54162:::benchmark_data_smithi194_103039_object53682:head# 2024-04-29T14:21:19.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed60734:::benchmark_data_smithi194_103039_object122769:head# 2024-04-29T14:21:19.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed713d1:::benchmark_data_smithi194_103039_object149246:head# 2024-04-29T14:21:19.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed75b90:::benchmark_data_smithi194_103039_object87760:head# 2024-04-29T14:21:19.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed7cd05:::benchmark_data_smithi194_103039_object103913:head# 2024-04-29T14:21:19.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed817f3:::benchmark_data_smithi194_103039_object152662:head# 2024-04-29T14:21:19.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed880fb:::benchmark_data_smithi194_103039_object71840:head# 2024-04-29T14:21:19.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed90c81:::benchmark_data_smithi194_103039_object128428:head# 2024-04-29T14:21:19.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ed9c763:::benchmark_data_smithi194_103039_object70381:head# 2024-04-29T14:21:19.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eda7b8a:::benchmark_data_smithi194_103039_object85982:head# 2024-04-29T14:21:19.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eda8d43:::benchmark_data_smithi194_103039_object149485:head# 2024-04-29T14:21:19.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edc4e79:::benchmark_data_smithi194_103039_object144314:head# 2024-04-29T14:21:19.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edcccc1:::benchmark_data_smithi194_103039_object146139:head# 2024-04-29T14:21:19.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edd2353:::benchmark_data_smithi194_103039_object71540:head# 2024-04-29T14:21:19.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eddb905:::benchmark_data_smithi194_103039_object69038:head# 2024-04-29T14:21:19.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ede504c:::benchmark_data_smithi194_103039_object109007:head# 2024-04-29T14:21:19.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ede8d48:::benchmark_data_smithi194_103039_object129610:head# 2024-04-29T14:21:19.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edeffa1:::benchmark_data_smithi194_103039_object49447:head# 2024-04-29T14:21:19.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edf11af:::benchmark_data_smithi194_103039_object50252:head# 2024-04-29T14:21:19.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edfce03:::benchmark_data_smithi194_103039_object134654:head# 2024-04-29T14:21:19.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edfdd05:::benchmark_data_smithi194_103039_object134159:head# 2024-04-29T14:21:19.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edfe9f8:::benchmark_data_smithi194_103039_object104212:head# 2024-04-29T14:21:19.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4edfec6b:::benchmark_data_smithi194_103039_object64537:head# 2024-04-29T14:21:19.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee0056d:::benchmark_data_smithi194_103039_object63474:head# 2024-04-29T14:21:19.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee014b6:::benchmark_data_smithi194_103039_object45557:head# 2024-04-29T14:21:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee068e0:::benchmark_data_smithi194_103039_object97702:head# 2024-04-29T14:21:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee0c3e7:::benchmark_data_smithi194_103039_object127612:head# 2024-04-29T14:21:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee1894f:::benchmark_data_smithi194_103039_object52913:head# 2024-04-29T14:21:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee3f92e:::benchmark_data_smithi194_103039_object50885:head# 2024-04-29T14:21:19.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee421e3:::benchmark_data_smithi194_103039_object150494:head# 2024-04-29T14:21:19.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee5f7a0:::benchmark_data_smithi194_103039_object116953:head# 2024-04-29T14:21:19.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee6306d:::benchmark_data_smithi194_103039_object58693:head# 2024-04-29T14:21:19.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee65a27:::benchmark_data_smithi194_103039_object144095:head# 2024-04-29T14:21:19.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee66b6f:::benchmark_data_smithi194_103039_object116140:head# 2024-04-29T14:21:19.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee75eb5:::benchmark_data_smithi194_103039_object155947:head# 2024-04-29T14:21:19.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee81e89:::benchmark_data_smithi194_103039_object96405:head# 2024-04-29T14:21:19.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee86a4f:::benchmark_data_smithi194_103039_object75179:head# 2024-04-29T14:21:19.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee92e46:::benchmark_data_smithi194_103039_object113818:head# 2024-04-29T14:21:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee965c2:::benchmark_data_smithi194_103039_object49723:head# 2024-04-29T14:21:19.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ee9af6d:::benchmark_data_smithi194_103039_object133158:head# 2024-04-29T14:21:19.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eea0e28:::benchmark_data_smithi194_103039_object141335:head# 2024-04-29T14:21:19.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eea7b9f:::benchmark_data_smithi194_103039_object70896:head# 2024-04-29T14:21:19.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eea9750:::benchmark_data_smithi194_103039_object108041:head# 2024-04-29T14:21:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eeae5c9:::benchmark_data_smithi194_103039_object44405:head# 2024-04-29T14:21:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eec7592:::benchmark_data_smithi194_103039_object154701:head# 2024-04-29T14:21:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eec9e1c:::benchmark_data_smithi194_103039_object117652:head# 2024-04-29T14:21:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eecaf76:::benchmark_data_smithi194_103039_object109969:head# 2024-04-29T14:21:19.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eecfcbb:::benchmark_data_smithi194_103039_object62167:head# 2024-04-29T14:21:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eedcd3d:::benchmark_data_smithi194_103039_object49937:head# 2024-04-29T14:21:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eedf912:::benchmark_data_smithi194_103039_object140620:head# 2024-04-29T14:21:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eee164e:::benchmark_data_smithi194_103039_object144989:head# 2024-04-29T14:21:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eeee025:::benchmark_data_smithi194_103039_object60602:head# 2024-04-29T14:21:19.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef140e8:::benchmark_data_smithi194_103039_object67565:head# 2024-04-29T14:21:19.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef18fe4:::benchmark_data_smithi194_103039_object73220:head# 2024-04-29T14:21:19.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef204ac:::benchmark_data_smithi194_103039_object83464:head# 2024-04-29T14:21:19.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef451e9:::benchmark_data_smithi194_103039_object72494:head# 2024-04-29T14:21:19.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef499c8:::benchmark_data_smithi194_103039_object105870:head# 2024-04-29T14:21:19.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef50345:::benchmark_data_smithi194_103039_object118105:head# 2024-04-29T14:21:19.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef57552:::benchmark_data_smithi194_103039_object67382:head# 2024-04-29T14:21:19.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef5ce52:::benchmark_data_smithi194_103039_object136163:head# 2024-04-29T14:21:19.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef614c9:::benchmark_data_smithi194_103039_object127041:head# 2024-04-29T14:21:19.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef6734a:::benchmark_data_smithi194_103039_object86332:head# 2024-04-29T14:21:19.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef73637:::benchmark_data_smithi194_103039_object110630:head# 2024-04-29T14:21:19.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef77eda:::benchmark_data_smithi194_103039_object43103:head# 2024-04-29T14:21:19.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef7b6a9:::benchmark_data_smithi194_103039_object137585:head# 2024-04-29T14:21:19.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef809b4:::benchmark_data_smithi194_103039_object132211:head# 2024-04-29T14:21:19.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef8e092:::benchmark_data_smithi194_103039_object70844:head# 2024-04-29T14:21:19.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef907d5:::benchmark_data_smithi194_103039_object120321:head# 2024-04-29T14:21:19.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef9387c:::benchmark_data_smithi194_103039_object97749:head# 2024-04-29T14:21:19.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef97277:::benchmark_data_smithi194_103039_object45740:head# 2024-04-29T14:21:19.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef9c1ef:::benchmark_data_smithi194_103039_object111533:head# 2024-04-29T14:21:19.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ef9e779:::benchmark_data_smithi194_103039_object65921:head# 2024-04-29T14:21:19.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efa21ec:::benchmark_data_smithi194_103039_object63357:head# 2024-04-29T14:21:19.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efa9c7d:::benchmark_data_smithi194_103039_object68565:head# 2024-04-29T14:21:19.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efafe18:::benchmark_data_smithi194_103039_object110744:head# 2024-04-29T14:21:19.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efb1910:::benchmark_data_smithi194_103039_object109945:head# 2024-04-29T14:21:19.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efb1e3b:::benchmark_data_smithi194_103039_object44013:head# 2024-04-29T14:21:19.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efb34ca:::benchmark_data_smithi194_103039_object98056:head# 2024-04-29T14:21:19.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efb6e12:::benchmark_data_smithi194_103039_object106806:head# 2024-04-29T14:21:19.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efbacc8:::benchmark_data_smithi194_103039_object99728:head# 2024-04-29T14:21:19.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efbb709:::benchmark_data_smithi194_103039_object45080:head# 2024-04-29T14:21:19.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efe00f9:::benchmark_data_smithi194_103039_object98618:head# 2024-04-29T14:21:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efe69e0:::benchmark_data_smithi194_103039_object99285:head# 2024-04-29T14:21:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4efea75c:::benchmark_data_smithi194_103039_object136210:head# 2024-04-29T14:21:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4eff1a2d:::benchmark_data_smithi194_103039_object121204:head# 2024-04-29T14:21:19.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4effafe8:::benchmark_data_smithi194_103039_object156020:head# 2024-04-29T14:21:19.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4effb8e9:::benchmark_data_smithi194_103039_object76205:head# 2024-04-29T14:21:19.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f00fa32:::benchmark_data_smithi194_103039_object82597:head# 2024-04-29T14:21:19.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f02105e:::benchmark_data_smithi194_103039_object75397:head# 2024-04-29T14:21:19.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f021e10:::benchmark_data_smithi194_103039_object72485:head# 2024-04-29T14:21:19.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f02e317:::benchmark_data_smithi194_103039_object92737:head# 2024-04-29T14:21:19.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f02fe55:::benchmark_data_smithi194_103039_object125043:head# 2024-04-29T14:21:19.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f03110d:::benchmark_data_smithi194_103039_object60447:head# 2024-04-29T14:21:19.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f03f2fc:::benchmark_data_smithi194_103039_object122908:head# 2024-04-29T14:21:19.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0407b5:::benchmark_data_smithi194_103039_object101021:head# 2024-04-29T14:21:19.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f044712:::benchmark_data_smithi194_103039_object101720:head# 2024-04-29T14:21:19.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f04691e:::benchmark_data_smithi194_103039_object125707:head# 2024-04-29T14:21:19.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f04a8ee:::benchmark_data_smithi194_103039_object80607:head# 2024-04-29T14:21:19.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f052683:::benchmark_data_smithi194_103039_object80576:head# 2024-04-29T14:21:19.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f057248:::benchmark_data_smithi194_103039_object113793:head# 2024-04-29T14:21:19.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f05da98:::benchmark_data_smithi194_103039_object104616:head# 2024-04-29T14:21:19.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f06bc0a:::benchmark_data_smithi194_103039_object104600:head# 2024-04-29T14:21:19.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f06cd5a:::benchmark_data_smithi194_103039_object64231:head# 2024-04-29T14:21:19.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f06f4c2:::benchmark_data_smithi194_103039_object149889:head# 2024-04-29T14:21:19.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f08e5e6:::benchmark_data_smithi194_103039_object133015:head# 2024-04-29T14:21:19.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f09811c:::benchmark_data_smithi194_103039_object104766:head# 2024-04-29T14:21:19.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f098ed6:::benchmark_data_smithi194_103039_object131662:head# 2024-04-29T14:21:19.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0aaa96:::benchmark_data_smithi194_103039_object131428:head# 2024-04-29T14:21:19.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0b2be4:::benchmark_data_smithi194_103039_object124520:head# 2024-04-29T14:21:19.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0bd920:::benchmark_data_smithi194_103039_object109079:head# 2024-04-29T14:21:19.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0c02d5:::benchmark_data_smithi194_103039_object149909:head# 2024-04-29T14:21:19.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0c9b9f:::benchmark_data_smithi194_103039_object53481:head# 2024-04-29T14:21:19.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0d114d:::benchmark_data_smithi194_103039_object56926:head# 2024-04-29T14:21:19.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0d17a7:::benchmark_data_smithi194_103039_object96118:head# 2024-04-29T14:21:19.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0dbed1:::benchmark_data_smithi194_103039_object44892:head# 2024-04-29T14:21:19.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0ddc10:::benchmark_data_smithi194_103039_object153888:head# 2024-04-29T14:21:19.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0df613:::benchmark_data_smithi194_103039_object44764:head# 2024-04-29T14:21:19.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0e1d1a:::benchmark_data_smithi194_103039_object154258:head# 2024-04-29T14:21:19.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f0e9ea2:::benchmark_data_smithi194_103039_object65925:head# 2024-04-29T14:21:19.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f10a983:::benchmark_data_smithi194_103039_object154437:head# 2024-04-29T14:21:19.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f10e087:::benchmark_data_smithi194_103039_object130939:head# 2024-04-29T14:21:19.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f12cf31:::benchmark_data_smithi194_103039_object128232:head# 2024-04-29T14:21:19.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f134644:::benchmark_data_smithi194_103039_object148212:head# 2024-04-29T14:21:19.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1374c2:::benchmark_data_smithi194_103039_object100784:head# 2024-04-29T14:21:19.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f157ec6:::benchmark_data_smithi194_103039_object54634:head# 2024-04-29T14:21:19.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f164947:::benchmark_data_smithi194_103039_object100479:head# 2024-04-29T14:21:19.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1712f7:::benchmark_data_smithi194_103039_object111851:head# 2024-04-29T14:21:19.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f183ccd:::benchmark_data_smithi194_103039_object103491:head# 2024-04-29T14:21:19.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f19b296:::benchmark_data_smithi194_103039_object122229:head# 2024-04-29T14:21:19.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1a0859:::benchmark_data_smithi194_103039_object86684:head# 2024-04-29T14:21:19.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1acfcd:::benchmark_data_smithi194_103039_object75988:head# 2024-04-29T14:21:19.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1aedbb:::benchmark_data_smithi194_103039_object51948:head# 2024-04-29T14:21:19.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1becde:::benchmark_data_smithi194_103039_object62428:head# 2024-04-29T14:21:19.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1c232b:::benchmark_data_smithi194_103039_object53614:head# 2024-04-29T14:21:19.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1dd54b:::benchmark_data_smithi194_103039_object122302:head# 2024-04-29T14:21:19.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1e294c:::benchmark_data_smithi194_103039_object136086:head# 2024-04-29T14:21:19.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1f4d00:::benchmark_data_smithi194_103039_object133828:head# 2024-04-29T14:21:19.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f1f9a30:::benchmark_data_smithi194_103039_object69491:head# 2024-04-29T14:21:19.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f208902:::benchmark_data_smithi194_103039_object121172:head# 2024-04-29T14:21:19.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f210802:::benchmark_data_smithi194_103039_object92372:head# 2024-04-29T14:21:19.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f214bb2:::benchmark_data_smithi194_103039_object95425:head# 2024-04-29T14:21:19.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f21bd03:::benchmark_data_smithi194_103039_object66731:head# 2024-04-29T14:21:19.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f22813a:::benchmark_data_smithi194_103039_object122902:head# 2024-04-29T14:21:19.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f237bfd:::benchmark_data_smithi194_103039_object151703:head# 2024-04-29T14:21:19.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f24628c:::benchmark_data_smithi194_103039_object106055:head# 2024-04-29T14:21:19.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f251496:::benchmark_data_smithi194_103039_object75719:head# 2024-04-29T14:21:19.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f256d2c:::benchmark_data_smithi194_103039_object91264:head# 2024-04-29T14:21:19.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f25b157:::benchmark_data_smithi194_103039_object88381:head# 2024-04-29T14:21:19.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f260ad7:::benchmark_data_smithi194_103039_object135842:head# 2024-04-29T14:21:19.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f263d3c:::benchmark_data_smithi194_103039_object84599:head# 2024-04-29T14:21:19.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f268e84:::benchmark_data_smithi194_103039_object43197:head# 2024-04-29T14:21:19.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f28ccc2:::benchmark_data_smithi194_103039_object137113:head# 2024-04-29T14:21:19.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f28d3ad:::benchmark_data_smithi194_103039_object58733:head# 2024-04-29T14:21:19.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f29a6dc:::benchmark_data_smithi194_103039_object53616:head# 2024-04-29T14:21:19.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f29ac62:::benchmark_data_smithi194_103039_object124292:head# 2024-04-29T14:21:19.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2b0ef5:::benchmark_data_smithi194_103039_object119546:head# 2024-04-29T14:21:19.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2c41a8:::benchmark_data_smithi194_103039_object146283:head# 2024-04-29T14:21:19.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2cf334:::benchmark_data_smithi194_103039_object75271:head# 2024-04-29T14:21:19.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2d327e:::benchmark_data_smithi194_103039_object56573:head# 2024-04-29T14:21:19.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2d8a99:::benchmark_data_smithi194_103039_object50403:head# 2024-04-29T14:21:19.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2dcf98:::benchmark_data_smithi194_103039_object153988:head# 2024-04-29T14:21:19.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2e6c3e:::benchmark_data_smithi194_103039_object90095:head# 2024-04-29T14:21:19.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2ec8d0:::benchmark_data_smithi194_103039_object48454:head# 2024-04-29T14:21:19.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2f5df5:::benchmark_data_smithi194_103039_object91700:head# 2024-04-29T14:21:19.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f2fe808:::benchmark_data_smithi194_103039_object121368:head# 2024-04-29T14:21:19.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f310550:::benchmark_data_smithi194_103039_object101552:head# 2024-04-29T14:21:19.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3128d7:::benchmark_data_smithi194_103039_object100533:head# 2024-04-29T14:21:19.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f327dfa:::benchmark_data_smithi194_103039_object154466:head# 2024-04-29T14:21:19.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f331fb1:::benchmark_data_smithi194_103039_object143014:head# 2024-04-29T14:21:19.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f338e85:::benchmark_data_smithi194_103039_object103186:head# 2024-04-29T14:21:19.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3394c7:::benchmark_data_smithi194_103039_object135633:head# 2024-04-29T14:21:19.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f33ac2c:::benchmark_data_smithi194_103039_object129397:head# 2024-04-29T14:21:19.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f34cced:::benchmark_data_smithi194_103039_object85398:head# 2024-04-29T14:21:19.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f34ff75:::benchmark_data_smithi194_103039_object112283:head# 2024-04-29T14:21:19.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f359f82:::benchmark_data_smithi194_103039_object95422:head# 2024-04-29T14:21:19.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f35fabe:::benchmark_data_smithi194_103039_object92172:head# 2024-04-29T14:21:19.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3662cd:::benchmark_data_smithi194_103039_object58026:head# 2024-04-29T14:21:19.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3688ec:::benchmark_data_smithi194_103039_object90861:head# 2024-04-29T14:21:19.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f37bcb4:::benchmark_data_smithi194_103039_object112819:head# 2024-04-29T14:21:19.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f37cc6f:::benchmark_data_smithi194_103039_object151634:head# 2024-04-29T14:21:19.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f37fa33:::benchmark_data_smithi194_103039_object71740:head# 2024-04-29T14:21:19.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f385978:::benchmark_data_smithi194_103039_object60948:head# 2024-04-29T14:21:19.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f38df2b:::benchmark_data_smithi194_103039_object108186:head# 2024-04-29T14:21:19.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3ac4ba:::benchmark_data_smithi194_103039_object115807:head# 2024-04-29T14:21:19.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3ac546:::benchmark_data_smithi194_103039_object138503:head# 2024-04-29T14:21:19.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3b4070:::benchmark_data_smithi194_103039_object128887:head# 2024-04-29T14:21:19.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3e4ecb:::benchmark_data_smithi194_103039_object103343:head# 2024-04-29T14:21:19.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3eed2d:::benchmark_data_smithi194_103039_object129628:head# 2024-04-29T14:21:19.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f3f5986:::benchmark_data_smithi194_103039_object71031:head# 2024-04-29T14:21:19.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f407385:::benchmark_data_smithi194_103039_object137232:head# 2024-04-29T14:21:19.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f40ee01:::benchmark_data_smithi194_103039_object82174:head# 2024-04-29T14:21:19.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f412ffb:::benchmark_data_smithi194_103039_object82175:head# 2024-04-29T14:21:19.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f41d724:::benchmark_data_smithi194_103039_object121838:head# 2024-04-29T14:21:19.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f420430:::benchmark_data_smithi194_103039_object95795:head# 2024-04-29T14:21:19.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f425d7b:::benchmark_data_smithi194_103039_object69318:head# 2024-04-29T14:21:19.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f43f5d2:::benchmark_data_smithi194_103039_object100983:head# 2024-04-29T14:21:19.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f44379f:::benchmark_data_smithi194_103039_object123573:head# 2024-04-29T14:21:19.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f448f00:::benchmark_data_smithi194_103039_object145503:head# 2024-04-29T14:21:19.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f449c92:::benchmark_data_smithi194_103039_object130942:head# 2024-04-29T14:21:19.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f46127d:::benchmark_data_smithi194_103039_object144772:head# 2024-04-29T14:21:19.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f46e7a5:::benchmark_data_smithi194_103039_object62586:head# 2024-04-29T14:21:19.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f470d41:::benchmark_data_smithi194_103039_object52649:head# 2024-04-29T14:21:19.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f474194:::benchmark_data_smithi194_103039_object105188:head# 2024-04-29T14:21:19.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4765c9:::benchmark_data_smithi194_103039_object98522:head# 2024-04-29T14:21:19.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f476faa:::benchmark_data_smithi194_103039_object86797:head# 2024-04-29T14:21:19.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f480990:::benchmark_data_smithi194_103039_object136716:head# 2024-04-29T14:21:19.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f48c283:::benchmark_data_smithi194_103039_object90084:head# 2024-04-29T14:21:19.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f49a4d8:::benchmark_data_smithi194_103039_object126929:head# 2024-04-29T14:21:19.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:19.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4afe8a:::benchmark_data_smithi194_103039_object72851:head# 2024-04-29T14:21:19.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4c13b7:::benchmark_data_smithi194_103039_object80435:head# 2024-04-29T14:21:19.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4c30f7:::benchmark_data_smithi194_103039_object56080:head# 2024-04-29T14:21:19.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4d9a8f:::benchmark_data_smithi194_103039_object133430:head# 2024-04-29T14:21:19.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4e8949:::benchmark_data_smithi194_103039_object97489:head# 2024-04-29T14:21:19.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4ea377:::benchmark_data_smithi194_103039_object63576:head# 2024-04-29T14:21:19.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4edddf:::benchmark_data_smithi194_103039_object129218:head# 2024-04-29T14:21:19.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4f2c23:::benchmark_data_smithi194_103039_object76298:head# 2024-04-29T14:21:19.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4f664f:::benchmark_data_smithi194_103039_object53710:head# 2024-04-29T14:21:19.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f4fe7cb:::benchmark_data_smithi194_103039_object144953:head# 2024-04-29T14:21:19.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5026a9:::benchmark_data_smithi194_103039_object93749:head# 2024-04-29T14:21:19.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f50c472:::benchmark_data_smithi194_103039_object150826:head# 2024-04-29T14:21:19.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f513ca7:::benchmark_data_smithi194_103039_object73653:head# 2024-04-29T14:21:19.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f528454:::benchmark_data_smithi194_103039_object105811:head# 2024-04-29T14:21:19.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f52a56f:::benchmark_data_smithi194_103039_object62639:head# 2024-04-29T14:21:19.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f52e090:::benchmark_data_smithi194_103039_object134500:head# 2024-04-29T14:21:19.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f53a842:::benchmark_data_smithi194_103039_object116235:head# 2024-04-29T14:21:19.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f53e8b2:::benchmark_data_smithi194_103039_object88547:head# 2024-04-29T14:21:19.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f540b3d:::benchmark_data_smithi194_103039_object133652:head# 2024-04-29T14:21:19.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5455c8:::benchmark_data_smithi194_103039_object86435:head# 2024-04-29T14:21:19.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f55289d:::benchmark_data_smithi194_103039_object146291:head# 2024-04-29T14:21:19.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f554195:::benchmark_data_smithi194_103039_object67673:head# 2024-04-29T14:21:19.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f56d8c6:::benchmark_data_smithi194_103039_object142734:head# 2024-04-29T14:21:19.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f56f293:::benchmark_data_smithi194_103039_object71998:head# 2024-04-29T14:21:19.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f57dd50:::benchmark_data_smithi194_103039_object44309:head# 2024-04-29T14:21:19.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5883bf:::benchmark_data_smithi194_103039_object69561:head# 2024-04-29T14:21:19.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5889fb:::benchmark_data_smithi194_103039_object77494:head# 2024-04-29T14:21:19.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f58d340:::benchmark_data_smithi194_103039_object46730:head# 2024-04-29T14:21:19.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f594525:::benchmark_data_smithi194_103039_object127888:head# 2024-04-29T14:21:19.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f59751d:::benchmark_data_smithi194_103039_object120284:head# 2024-04-29T14:21:19.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5a2560:::benchmark_data_smithi194_103039_object123106:head# 2024-04-29T14:21:19.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5a70eb:::benchmark_data_smithi194_103039_object58061:head# 2024-04-29T14:21:19.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5aae50:::benchmark_data_smithi194_103039_object99581:head# 2024-04-29T14:21:19.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5afacc:::benchmark_data_smithi194_103039_object69045:head# 2024-04-29T14:21:19.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5b2ec2:::benchmark_data_smithi194_103039_object136865:head# 2024-04-29T14:21:19.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5bdf47:::benchmark_data_smithi194_103039_object92070:head# 2024-04-29T14:21:19.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5d0e9f:::benchmark_data_smithi194_103039_object124684:head# 2024-04-29T14:21:19.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f5d9913:::benchmark_data_smithi194_103039_object88975:head# 2024-04-29T14:21:19.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f604110:::benchmark_data_smithi194_103039_object144560:head# 2024-04-29T14:21:19.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f609d00:::benchmark_data_smithi194_103039_object103950:head# 2024-04-29T14:21:19.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f615128:::benchmark_data_smithi194_103039_object120229:head# 2024-04-29T14:21:19.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6156d8:::benchmark_data_smithi194_103039_object80880:head# 2024-04-29T14:21:19.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f61d579:::benchmark_data_smithi194_103039_object86830:head# 2024-04-29T14:21:19.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f656108:::benchmark_data_smithi194_103039_object129319:head# 2024-04-29T14:21:19.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f66920a:::benchmark_data_smithi194_103039_object95986:head# 2024-04-29T14:21:19.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f66b8de:::benchmark_data_smithi194_103039_object139410:head# 2024-04-29T14:21:19.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f66c8ae:::benchmark_data_smithi194_103039_object55338:head# 2024-04-29T14:21:19.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f674614:::benchmark_data_smithi194_103039_object67854:head# 2024-04-29T14:21:19.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f676006:::benchmark_data_smithi194_103039_object100861:head# 2024-04-29T14:21:19.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f677482:::benchmark_data_smithi194_103039_object41695:head# 2024-04-29T14:21:19.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f67c4b9:::benchmark_data_smithi194_103039_object67664:head# 2024-04-29T14:21:19.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f683e35:::benchmark_data_smithi194_103039_object56511:head# 2024-04-29T14:21:19.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f68e67a:::benchmark_data_smithi194_103039_object69073:head# 2024-04-29T14:21:19.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6b0c05:::benchmark_data_smithi194_103039_object56417:head# 2024-04-29T14:21:19.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6b167b:::benchmark_data_smithi194_103039_object81291:head# 2024-04-29T14:21:19.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6b3989:::benchmark_data_smithi194_103039_object50007:head# 2024-04-29T14:21:19.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6c854b:::benchmark_data_smithi194_103039_object99645:head# 2024-04-29T14:21:19.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6cb893:::benchmark_data_smithi194_103039_object69730:head# 2024-04-29T14:21:19.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6d82a2:::benchmark_data_smithi194_103039_object109330:head# 2024-04-29T14:21:19.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6e5dc7:::benchmark_data_smithi194_103039_object135654:head# 2024-04-29T14:21:19.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6e9221:::benchmark_data_smithi194_103039_object104713:head# 2024-04-29T14:21:19.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6ef15d:::benchmark_data_smithi194_103039_object113069:head# 2024-04-29T14:21:19.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6f06e4:::benchmark_data_smithi194_103039_object90276:head# 2024-04-29T14:21:19.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6f76b0:::benchmark_data_smithi194_103039_object113791:head# 2024-04-29T14:21:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f6f89d1:::benchmark_data_smithi194_103039_object149056:head# 2024-04-29T14:21:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7013dc:::benchmark_data_smithi194_103039_object92824:head# 2024-04-29T14:21:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f70879f:::benchmark_data_smithi194_103039_object104420:head# 2024-04-29T14:21:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f709502:::benchmark_data_smithi194_103039_object116641:head# 2024-04-29T14:21:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f729393:::benchmark_data_smithi194_103039_object139130:head# 2024-04-29T14:21:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7341e6:::benchmark_data_smithi194_103039_object48833:head# 2024-04-29T14:21:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7490d4:::benchmark_data_smithi194_103039_object88889:head# 2024-04-29T14:21:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f74fc6d:::benchmark_data_smithi194_103039_object66873:head# 2024-04-29T14:21:19.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:19 smithi194 ceph-mon[44089]: pgmap v982: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 149 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:19.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:19 smithi194 ceph-mon[44089]: 4.0 scrub starts 2024-04-29T14:21:19.547 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:19 smithi194 ceph-mon[44089]: 4.0 scrub ok 2024-04-29T14:21:19.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f75a800:::benchmark_data_smithi194_103039_object124659:head# 2024-04-29T14:21:19.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f768275:::benchmark_data_smithi194_103039_object113866:head# 2024-04-29T14:21:19.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f769b6d:::benchmark_data_smithi194_103039_object137888:head# 2024-04-29T14:21:19.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f78c688:::benchmark_data_smithi194_103039_object142532:head# 2024-04-29T14:21:19.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f78d43d:::benchmark_data_smithi194_103039_object95751:head# 2024-04-29T14:21:19.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f791db4:::benchmark_data_smithi194_103039_object77277:head# 2024-04-29T14:21:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:19.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7ab338:::benchmark_data_smithi194_103039_object90614:head# 2024-04-29T14:21:19.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7b53df:::benchmark_data_smithi194_103039_object101323:head# 2024-04-29T14:21:19.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7c8c01:::benchmark_data_smithi194_103039_object154561:head# 2024-04-29T14:21:19.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7d08a1:::benchmark_data_smithi194_103039_object127858:head# 2024-04-29T14:21:19.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7f37ef:::benchmark_data_smithi194_103039_object80801:head# 2024-04-29T14:21:19.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7f4d9b:::benchmark_data_smithi194_103039_object42553:head# 2024-04-29T14:21:19.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7f69c0:::benchmark_data_smithi194_103039_object84336:head# 2024-04-29T14:21:19.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f7f6b9c:::benchmark_data_smithi194_103039_object96458:head# 2024-04-29T14:21:19.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f80ca6a:::benchmark_data_smithi194_103039_object49688:head# 2024-04-29T14:21:19.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f816b13:::benchmark_data_smithi194_103039_object149048:head# 2024-04-29T14:21:19.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f81c52a:::benchmark_data_smithi194_103039_object103259:head# 2024-04-29T14:21:19.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f83be02:::benchmark_data_smithi194_103039_object119922:head# 2024-04-29T14:21:19.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f83d190:::benchmark_data_smithi194_103039_object123027:head# 2024-04-29T14:21:19.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f858c50:::benchmark_data_smithi194_103039_object147431:head# 2024-04-29T14:21:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f85927b:::benchmark_data_smithi194_103039_object104535:head# 2024-04-29T14:21:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f85fad7:::benchmark_data_smithi194_103039_object46644:head# 2024-04-29T14:21:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f86187a:::benchmark_data_smithi194_103039_object128173:head# 2024-04-29T14:21:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8687d8:::benchmark_data_smithi194_103039_object153402:head# 2024-04-29T14:21:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f86aa7d:::benchmark_data_smithi194_103039_object44563:head# 2024-04-29T14:21:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f86d548:::benchmark_data_smithi194_103039_object94892:head# 2024-04-29T14:21:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f86ecf2:::benchmark_data_smithi194_103039_object93611:head# 2024-04-29T14:21:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f875076:::benchmark_data_smithi194_103039_object96966:head# 2024-04-29T14:21:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f87c9b2:::benchmark_data_smithi194_103039_object119422:head# 2024-04-29T14:21:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8807e2:::benchmark_data_smithi194_103039_object150586:head# 2024-04-29T14:21:19.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f88b5c1:::benchmark_data_smithi194_103039_object92726:head# 2024-04-29T14:21:19.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8904bf:::benchmark_data_smithi194_103039_object91940:head# 2024-04-29T14:21:19.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f89eb72:::benchmark_data_smithi194_103039_object52959:head# 2024-04-29T14:21:19.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8a803e:::benchmark_data_smithi194_103039_object74376:head# 2024-04-29T14:21:19.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8bca4f:::benchmark_data_smithi194_103039_object90665:head# 2024-04-29T14:21:19.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8c0b6c:::benchmark_data_smithi194_103039_object65074:head# 2024-04-29T14:21:19.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8d6d2f:::benchmark_data_smithi194_103039_object113690:head# 2024-04-29T14:21:19.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8d7242:::benchmark_data_smithi194_103039_object113220:head# 2024-04-29T14:21:19.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8ef6ad:::benchmark_data_smithi194_103039_object136662:head# 2024-04-29T14:21:19.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8f116b:::benchmark_data_smithi194_103039_object90580:head# 2024-04-29T14:21:19.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f8f2de4:::benchmark_data_smithi194_103039_object60132:head# 2024-04-29T14:21:19.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9063e2:::benchmark_data_smithi194_103039_object70867:head# 2024-04-29T14:21:19.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f907936:::benchmark_data_smithi194_103039_object107022:head# 2024-04-29T14:21:19.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9310f4:::benchmark_data_smithi194_103039_object97045:head# 2024-04-29T14:21:19.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f938125:::benchmark_data_smithi194_103039_object131838:head# 2024-04-29T14:21:19.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9454bb:::benchmark_data_smithi194_103039_object78475:head# 2024-04-29T14:21:19.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f947333:::benchmark_data_smithi194_103039_object134969:head# 2024-04-29T14:21:19.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f95b3df:::benchmark_data_smithi194_103039_object136949:head# 2024-04-29T14:21:19.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f95edf4:::benchmark_data_smithi194_103039_object152387:head# 2024-04-29T14:21:19.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f97a7b9:::benchmark_data_smithi194_103039_object155801:head# 2024-04-29T14:21:19.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f98d929:::benchmark_data_smithi194_103039_object133358:head# 2024-04-29T14:21:19.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f993894:::benchmark_data_smithi194_103039_object110494:head# 2024-04-29T14:21:19.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f995bb6:::benchmark_data_smithi194_103039_object97319:head# 2024-04-29T14:21:19.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9a1549:::benchmark_data_smithi194_103039_object81444:head# 2024-04-29T14:21:19.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9a5d47:::benchmark_data_smithi194_103039_object99935:head# 2024-04-29T14:21:19.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9ab68d:::benchmark_data_smithi194_103039_object115478:head# 2024-04-29T14:21:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:19.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9b75db:::benchmark_data_smithi194_103039_object104166:head# 2024-04-29T14:21:19.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9bcbbe:::benchmark_data_smithi194_103039_object116433:head# 2024-04-29T14:21:19.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9c1b4b:::benchmark_data_smithi194_103039_object139956:head# 2024-04-29T14:21:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9c29d5:::benchmark_data_smithi194_103039_object132099:head# 2024-04-29T14:21:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9cc639:::benchmark_data_smithi194_103039_object136867:head# 2024-04-29T14:21:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9d679d:::benchmark_data_smithi194_103039_object128074:head# 2024-04-29T14:21:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:19.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9ec3ed:::benchmark_data_smithi194_103039_object56694:head# 2024-04-29T14:21:19.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4f9ee6f0:::benchmark_data_smithi194_103039_object61447:head# 2024-04-29T14:21:19.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa00e0d:::benchmark_data_smithi194_103039_object92292:head# 2024-04-29T14:21:19.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:19.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa03618:::benchmark_data_smithi194_103039_object61551:head# 2024-04-29T14:21:19.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa07e7b:::benchmark_data_smithi194_103039_object63263:head# 2024-04-29T14:21:19.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa39ed2:::benchmark_data_smithi194_103039_object110094:head# 2024-04-29T14:21:19.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa45af8:::benchmark_data_smithi194_103039_object144280:head# 2024-04-29T14:21:19.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa501a5:::benchmark_data_smithi194_103039_object124002:head# 2024-04-29T14:21:19.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa553c2:::benchmark_data_smithi194_103039_object60775:head# 2024-04-29T14:21:19.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa67a95:::benchmark_data_smithi194_103039_object104483:head# 2024-04-29T14:21:19.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa690fe:::benchmark_data_smithi194_103039_object68369:head# 2024-04-29T14:21:19.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa6b129:::benchmark_data_smithi194_103039_object71778:head# 2024-04-29T14:21:19.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa77083:::benchmark_data_smithi194_103039_object52264:head# 2024-04-29T14:21:19.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fa98d43:::benchmark_data_smithi194_103039_object152013:head# 2024-04-29T14:21:19.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4faa6ddc:::benchmark_data_smithi194_103039_object59318:head# 2024-04-29T14:21:19.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fac1381:::benchmark_data_smithi194_103039_object114028:head# 2024-04-29T14:21:19.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fac1bbb:::benchmark_data_smithi194_103039_object156323:head# 2024-04-29T14:21:19.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fade5d9:::benchmark_data_smithi194_103039_object115826:head# 2024-04-29T14:21:19.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fae0f1c:::benchmark_data_smithi194_103039_object124525:head# 2024-04-29T14:21:19.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fae2b85:::benchmark_data_smithi194_103039_object78534:head# 2024-04-29T14:21:19.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fae35a4:::benchmark_data_smithi194_103039_object102320:head# 2024-04-29T14:21:19.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fae8f41:::benchmark_data_smithi194_103039_object87948:head# 2024-04-29T14:21:19.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4faeb464:::benchmark_data_smithi194_103039_object151150:head# 2024-04-29T14:21:19.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4faf0f17:::benchmark_data_smithi194_103039_object108587:head# 2024-04-29T14:21:19.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4faf44df:::benchmark_data_smithi194_103039_object133805:head# 2024-04-29T14:21:19.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fafbd44:::benchmark_data_smithi194_103039_object110718:head# 2024-04-29T14:21:19.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb1ab58:::benchmark_data_smithi194_103039_object46218:head# 2024-04-29T14:21:19.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb1e29b:::benchmark_data_smithi194_103039_object47572:head# 2024-04-29T14:21:19.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb1f25b:::benchmark_data_smithi194_103039_object136042:head# 2024-04-29T14:21:19.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb2d875:::benchmark_data_smithi194_103039_object69014:head# 2024-04-29T14:21:19.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb2f5e1:::benchmark_data_smithi194_103039_object149508:head# 2024-04-29T14:21:19.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb32e64:::benchmark_data_smithi194_103039_object149994:head# 2024-04-29T14:21:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb366b3:::benchmark_data_smithi194_103039_object45617:head# 2024-04-29T14:21:19.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb418c6:::benchmark_data_smithi194_103039_object61262:head# 2024-04-29T14:21:19.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb4a2b3:::benchmark_data_smithi194_103039_object51968:head# 2024-04-29T14:21:19.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb52d4b:::benchmark_data_smithi194_103039_object44165:head# 2024-04-29T14:21:19.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb5490b:::benchmark_data_smithi194_103039_object122399:head# 2024-04-29T14:21:19.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb5af1d:::benchmark_data_smithi194_103039_object67992:head# 2024-04-29T14:21:19.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb779d5:::benchmark_data_smithi194_103039_object142671:head# 2024-04-29T14:21:19.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb7b08d:::benchmark_data_smithi194_103039_object153869:head# 2024-04-29T14:21:19.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb82e96:::benchmark_data_smithi194_103039_object149918:head# 2024-04-29T14:21:19.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb8619e:::benchmark_data_smithi194_103039_object71039:head# 2024-04-29T14:21:19.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb862f9:::benchmark_data_smithi194_103039_object134171:head# 2024-04-29T14:21:19.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb89f31:::benchmark_data_smithi194_103039_object106406:head# 2024-04-29T14:21:19.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb8cdaf:::benchmark_data_smithi194_103039_object50173:head# 2024-04-29T14:21:19.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb8fc2c:::benchmark_data_smithi194_103039_object79539:head# 2024-04-29T14:21:19.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fb9aeb9:::benchmark_data_smithi194_103039_object134320:head# 2024-04-29T14:21:19.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fba3aa5:::benchmark_data_smithi194_103039_object142479:head# 2024-04-29T14:21:19.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fba6d4e:::benchmark_data_smithi194_103039_object55922:head# 2024-04-29T14:21:19.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fba8298:::benchmark_data_smithi194_103039_object77319:head# 2024-04-29T14:21:19.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbb7cb7:::benchmark_data_smithi194_103039_object41946:head# 2024-04-29T14:21:19.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbbee2d:::benchmark_data_smithi194_103039_object53280:head# 2024-04-29T14:21:19.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbc0631:::benchmark_data_smithi194_103039_object123466:head# 2024-04-29T14:21:19.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbc25e5:::benchmark_data_smithi194_103039_object48129:head# 2024-04-29T14:21:19.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbca021:::benchmark_data_smithi194_103039_object53677:head# 2024-04-29T14:21:19.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbdd7fb:::benchmark_data_smithi194_103039_object148738:head# 2024-04-29T14:21:19.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbdeb9e:::benchmark_data_smithi194_103039_object135213:head# 2024-04-29T14:21:19.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbdfcbd:::benchmark_data_smithi194_103039_object148282:head# 2024-04-29T14:21:19.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbe0e14:::benchmark_data_smithi194_103039_object73634:head# 2024-04-29T14:21:19.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbe3043:::benchmark_data_smithi194_103039_object141450:head# 2024-04-29T14:21:19.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbfb5a0:::benchmark_data_smithi194_103039_object116129:head# 2024-04-29T14:21:19.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fbfc760:::benchmark_data_smithi194_103039_object100597:head# 2024-04-29T14:21:19.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc0c0f4:::benchmark_data_smithi194_103039_object105096:head# 2024-04-29T14:21:19.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc166ed:::benchmark_data_smithi194_103039_object62171:head# 2024-04-29T14:21:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc1ba2a:::benchmark_data_smithi194_103039_object83726:head# 2024-04-29T14:21:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc1f8bd:::benchmark_data_smithi194_103039_object147959:head# 2024-04-29T14:21:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc206fc:::benchmark_data_smithi194_103039_object103531:head# 2024-04-29T14:21:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc278db:::benchmark_data_smithi194_103039_object122613:head# 2024-04-29T14:21:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc41e52:::benchmark_data_smithi194_103039_object147457:head# 2024-04-29T14:21:19.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc52d89:::benchmark_data_smithi194_103039_object62404:head# 2024-04-29T14:21:19.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc60a42:::benchmark_data_smithi194_103039_object69091:head# 2024-04-29T14:21:19.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc62c5c:::benchmark_data_smithi194_103039_object74196:head# 2024-04-29T14:21:19.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc6500d:::benchmark_data_smithi194_103039_object81382:head# 2024-04-29T14:21:19.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc66d97:::benchmark_data_smithi194_103039_object82280:head# 2024-04-29T14:21:19.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc70267:::benchmark_data_smithi194_103039_object120500:head# 2024-04-29T14:21:19.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc996e4:::benchmark_data_smithi194_103039_object55371:head# 2024-04-29T14:21:19.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fc99c07:::benchmark_data_smithi194_103039_object46801:head# 2024-04-29T14:21:19.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fca5e2e:::benchmark_data_smithi194_103039_object69141:head# 2024-04-29T14:21:19.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fcab795:::benchmark_data_smithi194_103039_object132417:head# 2024-04-29T14:21:19.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fcabe53:::benchmark_data_smithi194_103039_object61034:head# 2024-04-29T14:21:19.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fcade97:::benchmark_data_smithi194_103039_object128014:head# 2024-04-29T14:21:19.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fcbf7bc:::benchmark_data_smithi194_103039_object120033:head# 2024-04-29T14:21:19.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fcc627a:::benchmark_data_smithi194_103039_object49651:head# 2024-04-29T14:21:19.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fccf465:::benchmark_data_smithi194_103039_object69614:head# 2024-04-29T14:21:19.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fcd1907:::benchmark_data_smithi194_103039_object91816:head# 2024-04-29T14:21:19.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fcf3bd3:::benchmark_data_smithi194_103039_object82427:head# 2024-04-29T14:21:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd0935b:::benchmark_data_smithi194_103039_object111736:head# 2024-04-29T14:21:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd19636:::benchmark_data_smithi194_103039_object77664:head# 2024-04-29T14:21:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd1cc97:::benchmark_data_smithi194_103039_object109033:head# 2024-04-29T14:21:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd292f2:::benchmark_data_smithi194_103039_object152094:head# 2024-04-29T14:21:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd3abcb:::benchmark_data_smithi194_103039_object79896:head# 2024-04-29T14:21:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd3f849:::benchmark_data_smithi194_103039_object69623:head# 2024-04-29T14:21:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd4aa03:::benchmark_data_smithi194_103039_object102033:head# 2024-04-29T14:21:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd62317:::benchmark_data_smithi194_103039_object102523:head# 2024-04-29T14:21:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd6bdbc:::benchmark_data_smithi194_103039_object96032:head# 2024-04-29T14:21:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd6ca25:::benchmark_data_smithi194_103039_object146084:head# 2024-04-29T14:21:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd755df:::benchmark_data_smithi194_103039_object124491:head# 2024-04-29T14:21:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd7eb4d:::benchmark_data_smithi194_103039_object69806:head# 2024-04-29T14:21:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd80388:::benchmark_data_smithi194_103039_object77980:head# 2024-04-29T14:21:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd81ebf:::benchmark_data_smithi194_103039_object66081:head# 2024-04-29T14:21:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd8c70a:::benchmark_data_smithi194_103039_object76911:head# 2024-04-29T14:21:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd8fecf:::benchmark_data_smithi194_103039_object127822:head# 2024-04-29T14:21:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fd9648b:::benchmark_data_smithi194_103039_object148686:head# 2024-04-29T14:21:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fda64eb:::benchmark_data_smithi194_103039_object111419:head# 2024-04-29T14:21:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdaa826:::benchmark_data_smithi194_103039_object154716:head# 2024-04-29T14:21:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdaaed6:::benchmark_data_smithi194_103039_object97153:head# 2024-04-29T14:21:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdad276:::benchmark_data_smithi194_103039_object51935:head# 2024-04-29T14:21:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdb3a4a:::benchmark_data_smithi194_103039_object148552:head# 2024-04-29T14:21:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdb4526:::benchmark_data_smithi194_103039_object107839:head# 2024-04-29T14:21:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdc0dc7:::benchmark_data_smithi194_103039_object127503:head# 2024-04-29T14:21:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdcbc55:::benchmark_data_smithi194_103039_object94785:head# 2024-04-29T14:21:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdce387:::benchmark_data_smithi194_103039_object66523:head# 2024-04-29T14:21:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdd101a:::benchmark_data_smithi194_103039_object57004:head# 2024-04-29T14:21:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdd912b:::benchmark_data_smithi194_103039_object54191:head# 2024-04-29T14:21:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fddc342:::benchmark_data_smithi194_103039_object45129:head# 2024-04-29T14:21:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fde1c9f:::benchmark_data_smithi194_103039_object145373:head# 2024-04-29T14:21:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fde4dd3:::benchmark_data_smithi194_103039_object62104:head# 2024-04-29T14:21:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fde5ef1:::benchmark_data_smithi194_103039_object42338:head# 2024-04-29T14:21:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fde9982:::benchmark_data_smithi194_103039_object81847:head# 2024-04-29T14:21:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdf1934:::benchmark_data_smithi194_103039_object133702:head# 2024-04-29T14:21:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fdfe6e5:::benchmark_data_smithi194_103039_object73360:head# 2024-04-29T14:21:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe0eb32:::benchmark_data_smithi194_103039_object140076:head# 2024-04-29T14:21:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe14a21:::benchmark_data_smithi194_103039_object109937:head# 2024-04-29T14:21:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe37c61:::benchmark_data_smithi194_103039_object53297:head# 2024-04-29T14:21:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe3d2e8:::benchmark_data_smithi194_103039_object127841:head# 2024-04-29T14:21:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe40cf9:::benchmark_data_smithi194_103039_object115587:head# 2024-04-29T14:21:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe4190e:::benchmark_data_smithi194_103039_object43196:head# 2024-04-29T14:21:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe447ca:::benchmark_data_smithi194_103039_object109303:head# 2024-04-29T14:21:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe4db9a:::benchmark_data_smithi194_103039_object112137:head# 2024-04-29T14:21:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe5fa7e:::benchmark_data_smithi194_103039_object131975:head# 2024-04-29T14:21:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe6396b:::benchmark_data_smithi194_103039_object122618:head# 2024-04-29T14:21:19.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe67b49:::benchmark_data_smithi194_103039_object107951:head# 2024-04-29T14:21:19.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe71c03:::benchmark_data_smithi194_103039_object73153:head# 2024-04-29T14:21:19.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe7c7b3:::benchmark_data_smithi194_103039_object122584:head# 2024-04-29T14:21:19.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe7ed38:::benchmark_data_smithi194_103039_object78669:head# 2024-04-29T14:21:19.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe86669:::benchmark_data_smithi194_103039_object44739:head# 2024-04-29T14:21:19.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe869d0:::benchmark_data_smithi194_103039_object72076:head# 2024-04-29T14:21:19.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe8958c:::benchmark_data_smithi194_103039_object68485:head# 2024-04-29T14:21:19.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe91209:::benchmark_data_smithi194_103039_object128706:head# 2024-04-29T14:21:19.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe924b2:::benchmark_data_smithi194_103039_object44886:head# 2024-04-29T14:21:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe93333:::benchmark_data_smithi194_103039_object154373:head# 2024-04-29T14:21:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fe9e624:::benchmark_data_smithi194_103039_object153185:head# 2024-04-29T14:21:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fea223c:::benchmark_data_smithi194_103039_object151004:head# 2024-04-29T14:21:19.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fea74ef:::benchmark_data_smithi194_103039_object110776:head# 2024-04-29T14:21:19.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fea927b:::benchmark_data_smithi194_103039_object64016:head# 2024-04-29T14:21:19.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4feab7b4:::benchmark_data_smithi194_103039_object142803:head# 2024-04-29T14:21:19.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4feadba4:::benchmark_data_smithi194_103039_object139527:head# 2024-04-29T14:21:19.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4feb1bd3:::benchmark_data_smithi194_103039_object41903:head# 2024-04-29T14:21:19.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4feb69ef:::benchmark_data_smithi194_103039_object81103:head# 2024-04-29T14:21:19.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4feba904:::benchmark_data_smithi194_103039_object71549:head# 2024-04-29T14:21:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4febfcbb:::benchmark_data_smithi194_103039_object136502:head# 2024-04-29T14:21:19.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fec4f36:::benchmark_data_smithi194_103039_object105269:head# 2024-04-29T14:21:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fecb7cf:::benchmark_data_smithi194_103039_object109864:head# 2024-04-29T14:21:19.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fed2486:::benchmark_data_smithi194_103039_object125608:head# 2024-04-29T14:21:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fee6c13:::benchmark_data_smithi194_103039_object105456:head# 2024-04-29T14:21:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fee9177:::benchmark_data_smithi194_103039_object110043:head# 2024-04-29T14:21:19.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4feef06f:::benchmark_data_smithi194_103039_object121971:head# 2024-04-29T14:21:19.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fef344f:::benchmark_data_smithi194_103039_object131288:head# 2024-04-29T14:21:19.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4fef5062:::benchmark_data_smithi194_103039_object118607:head# 2024-04-29T14:21:19.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff0432e:::benchmark_data_smithi194_103039_object58435:head# 2024-04-29T14:21:19.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff0ae55:::benchmark_data_smithi194_103039_object151254:head# 2024-04-29T14:21:19.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff0dacf:::benchmark_data_smithi194_103039_object134269:head# 2024-04-29T14:21:19.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff13694:::benchmark_data_smithi194_103039_object84476:head# 2024-04-29T14:21:19.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff14d6c:::benchmark_data_smithi194_103039_object121604:head# 2024-04-29T14:21:19.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff2e0c0:::benchmark_data_smithi194_103039_object120767:head# 2024-04-29T14:21:19.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff3f988:::benchmark_data_smithi194_103039_object72474:head# 2024-04-29T14:21:19.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff42495:::benchmark_data_smithi194_103039_object123792:head# 2024-04-29T14:21:19.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff4d2e5:::benchmark_data_smithi194_103039_object147567:head# 2024-04-29T14:21:19.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff52bec:::benchmark_data_smithi194_103039_object127780:head# 2024-04-29T14:21:19.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff5aac1:::benchmark_data_smithi194_103039_object118891:head# 2024-04-29T14:21:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff5ba67:::benchmark_data_smithi194_103039_object122791:head# 2024-04-29T14:21:19.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff63006:::benchmark_data_smithi194_103039_object53250:head# 2024-04-29T14:21:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff6ab19:::benchmark_data_smithi194_103039_object143740:head# 2024-04-29T14:21:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff77f44:::benchmark_data_smithi194_103039_object136118:head# 2024-04-29T14:21:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff7d236:::benchmark_data_smithi194_103039_object136209:head# 2024-04-29T14:21:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff92fdb:::benchmark_data_smithi194_103039_object143497:head# 2024-04-29T14:21:19.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ff9525d:::benchmark_data_smithi194_103039_object51159:head# 2024-04-29T14:21:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffa3aaf:::benchmark_data_smithi194_103039_object126475:head# 2024-04-29T14:21:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffaf9c9:::benchmark_data_smithi194_103039_object73060:head# 2024-04-29T14:21:19.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffb90e4:::benchmark_data_smithi194_103039_object153483:head# 2024-04-29T14:21:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffbcd84:::benchmark_data_smithi194_103039_object118051:head# 2024-04-29T14:21:19.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffc0a90:::benchmark_data_smithi194_103039_object154133:head# 2024-04-29T14:21:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffd0920:::benchmark_data_smithi194_103039_object146447:head# 2024-04-29T14:21:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffdb618:::benchmark_data_smithi194_103039_object96594:head# 2024-04-29T14:21:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffdd806:::benchmark_data_smithi194_103039_object116604:head# 2024-04-29T14:21:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffddf9e:::benchmark_data_smithi194_103039_object140436:head# 2024-04-29T14:21:19.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:4ffefd35:::benchmark_data_smithi194_103039_object76222:head# 2024-04-29T14:21:19.650 INFO:teuthology.orchestra.run.smithi121.stderr:Export successful 2024-04-29T14:21:19.674 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:19.702 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:19.825 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:19.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:19.929 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:19.955 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:21:20.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:20 smithi121 ceph-mon[44533]: 4.d scrub starts 2024-04-29T14:21:20.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/382510828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:20.303 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:20.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:20.387 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:20 smithi040 ceph-mon[39581]: 4.d scrub starts 2024-04-29T14:21:20.387 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/382510828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:20.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:20.437 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:20 smithi194 ceph-mon[44089]: 4.d scrub starts 2024-04-29T14:21:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/382510828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:20.579 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:20.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:20.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 4.2 2024-04-29T14:21:20.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:20.708 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:20.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:20.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:20.831 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:21:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:20.897 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:20.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:20.944 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:21:20.944 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:21:20.944 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:21:20.944 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:21:20.944 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:21:21.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:21.037 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:21.059 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:21.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:21.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:21:21.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:21.194 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:21.217 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:21.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:21.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:21 smithi194 ceph-mon[44089]: 4.d scrub ok 2024-04-29T14:21:21.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:21 smithi194 ceph-mon[44089]: pgmap v983: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 145 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:21.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:21 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:21:21.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/382510828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:21.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:21 smithi194 ceph-mon[44089]: osdmap e373: 12 total, 11 up, 8 in 2024-04-29T14:21:21.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:21.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:21 smithi040 ceph-mon[39581]: 4.d scrub ok 2024-04-29T14:21:21.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:21 smithi040 ceph-mon[39581]: pgmap v983: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 145 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:21.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:21 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:21:21.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/382510828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:21.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:21 smithi040 ceph-mon[39581]: osdmap e373: 12 total, 11 up, 8 in 2024-04-29T14:21:21.350 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:21.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:21.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:21 smithi121 ceph-mon[44533]: 4.d scrub ok 2024-04-29T14:21:21.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:21 smithi121 ceph-mon[44533]: pgmap v983: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 145 op/s; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:21.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:21 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:21:21.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/382510828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:21.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:21 smithi121 ceph-mon[44533]: osdmap e373: 12 total, 11 up, 8 in 2024-04-29T14:21:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:21.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:21.503 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:21.651 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:21.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:21.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:21:21.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:21.782 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:21.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:21.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:21.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:21.937 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:21.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:21.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:21.982 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:22.064 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:22.093 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:22.192 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:22.193 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:21:22.193 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: pgmap v985: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: pgmap v986: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: 4.e scrub starts 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:22.194 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:22.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:21:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:21:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:21:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:21:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:21:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:21:22.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: pgmap v985: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:22.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: pgmap v986: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:22.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:21:22.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: 4.e scrub starts 2024-04-29T14:21:22.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:22.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:22.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:22.308 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:21:22.323 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:22.480 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:21:22.480 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:21:22.480 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: pgmap v985: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: pgmap v986: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: 4.e scrub starts 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:22.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:22.671 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:22.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:22.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:22.762 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:21:22.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:22.834 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:22.933 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:23.612 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:23 smithi194 ceph-mon[44089]: 4.e scrub ok 2024-04-29T14:21:23.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:23 smithi194 ceph-mon[44089]: osdmap e374: 12 total, 11 up, 8 in 2024-04-29T14:21:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:23.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:23.614 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:23.614 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:23.614 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:23.614 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:23.615 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:23.615 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:23.615 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:23.615 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:23.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:23 smithi040 ceph-mon[39581]: 4.e scrub ok 2024-04-29T14:21:23.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:23 smithi040 ceph-mon[39581]: osdmap e374: 12 total, 11 up, 8 in 2024-04-29T14:21:23.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:23.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:23.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:23.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:23.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:23 smithi121 ceph-mon[44533]: 4.e scrub ok 2024-04-29T14:21:23.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:23 smithi121 ceph-mon[44533]: osdmap e374: 12 total, 11 up, 8 in 2024-04-29T14:21:23.714 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:23.749 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:23.767 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:23.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:23.773 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:21:23.773 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:21:23.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:23.900 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:24.047 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:24.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:24.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:24.182 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:24.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:24.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:24.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: pgmap v988: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/453700021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.341 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:24.504 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:21:24.601 INFO:teuthology.orchestra.run.smithi121.stdout: marking collection for removal 2024-04-29T14:21:24.601 INFO:teuthology.orchestra.run.smithi121.stdout:setting '_remove' omap key 2024-04-29T14:21:24.610 INFO:teuthology.orchestra.run.smithi121.stdout:finish_remove_pgs 4.2_head removing 4.2 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: pgmap v988: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/453700021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: pgmap v988: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:24.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/453700021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:24.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:24.716 INFO:teuthology.orchestra.run.smithi121.stdout:Remove successful 2024-04-29T14:21:24.760 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:21:24.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:21:25.191 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:25.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:25.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:25.323 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:25.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:25.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:25.526 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:25.556 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:25.575 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:25.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:25.678 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:25.704 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:25.731 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:25.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:25.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.2.5 2024-04-29T14:21:25.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:25.869 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:25.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:25.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:25.976 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:21:25.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:26.023 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:26.043 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:26.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:26.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:26.174 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:21:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:26 smithi040 ceph-mon[39581]: 4.9 scrub starts 2024-04-29T14:21:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:26 smithi040 ceph-mon[39581]: pgmap v989: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:26 smithi040 ceph-mon[39581]: 4.9 scrub ok 2024-04-29T14:21:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2497393862' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:21:26.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:26 smithi194 ceph-mon[44089]: 4.9 scrub starts 2024-04-29T14:21:26.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:26 smithi194 ceph-mon[44089]: pgmap v989: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:26.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:26 smithi194 ceph-mon[44089]: 4.9 scrub ok 2024-04-29T14:21:26.377 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2497393862' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:21:26.508 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:26 smithi121 ceph-mon[44533]: 4.9 scrub starts 2024-04-29T14:21:26.509 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:26 smithi121 ceph-mon[44533]: pgmap v989: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:26.509 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:26 smithi121 ceph-mon[44533]: 4.9 scrub ok 2024-04-29T14:21:26.509 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2497393862' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:21:26.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:26.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:26.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:26.653 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:26.660 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:21:26.661 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:21:26.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:26.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:26.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:26.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:26.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:21:27.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:27.086 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:27.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:27.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:27.264 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:27.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:27.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:27 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:27.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2497393862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:21:27.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:27 smithi194 ceph-mon[44089]: osdmap e375: 12 total, 11 up, 8 in 2024-04-29T14:21:27.367 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:27.389 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:21:27.396 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:27.417 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:27.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:27.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:27 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:27.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2497393862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:21:27.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:27 smithi040 ceph-mon[39581]: osdmap e375: 12 total, 11 up, 8 in 2024-04-29T14:21:27.521 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:27.547 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:27.560 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:27 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:27.560 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2497393862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:21:27.560 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:27 smithi121 ceph-mon[44533]: osdmap e375: 12 total, 11 up, 8 in 2024-04-29T14:21:27.569 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:27.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:27.611 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:21:27.665 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:27.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:27.702 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:27.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:27.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:27.932 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:27.963 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:28.087 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:28.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:28.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:28.219 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:28.227 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:28 smithi194 ceph-mon[44089]: pgmap v991: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:28.271 INFO:teuthology.orchestra.run.smithi121.stdout:get_pg_num_history pg_num_history pg_num_history(e369 pg_nums {1={14=1},3={206=16},4={323=16}} deleted_pools 320,3) 2024-04-29T14:21:28.271 INFO:teuthology.orchestra.run.smithi121.stdout:Importing pgid 4.2 2024-04-29T14:21:28.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40008f8d:::benchmark_data_smithi194_103039_object53885:head# 2024-04-29T14:21:28.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4000a56f:::benchmark_data_smithi194_103039_object100393:head# 2024-04-29T14:21:28.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4000cedc:::benchmark_data_smithi194_103039_object95179:head# 2024-04-29T14:21:28.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4001136b:::benchmark_data_smithi194_103039_object101004:head# 2024-04-29T14:21:28.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40012cac:::benchmark_data_smithi194_103039_object48225:head# 2024-04-29T14:21:28.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4002a611:::benchmark_data_smithi194_103039_object124037:head# 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40034f76:::benchmark_data_smithi194_103039_object62540:head# 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40049b3c:::benchmark_data_smithi194_103039_object45463:head# 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4004caf8:::benchmark_data_smithi194_103039_object145811:head# 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4004ec10:::benchmark_data_smithi194_103039_object121598:head# 2024-04-29T14:21:28.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400509e5:::benchmark_data_smithi194_103039_object86930:head# 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4005586b:::benchmark_data_smithi194_103039_object52095:head# 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40058371:::benchmark_data_smithi194_103039_object115442:head# 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4005e590:::benchmark_data_smithi194_103039_object116004:head# 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4006d085:::benchmark_data_smithi194_103039_object97077:head# 2024-04-29T14:21:28.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40071796:::benchmark_data_smithi194_103039_object115657:head# 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40075e72:::benchmark_data_smithi194_103039_object47203:head# 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4007b797:::benchmark_data_smithi194_103039_object47546:head# 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4007c404:::benchmark_data_smithi194_103039_object125086:head# 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4008bd5f:::benchmark_data_smithi194_103039_object61879:head# 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4008c78e:::benchmark_data_smithi194_103039_object127502:head# 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4008d114:::benchmark_data_smithi194_103039_object155618:head# 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40091baa:::benchmark_data_smithi194_103039_object92011:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40097ef5:::benchmark_data_smithi194_103039_object138997:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4009bb39:::benchmark_data_smithi194_103039_object131244:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4009fa58:::benchmark_data_smithi194_103039_object97699:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400a3c51:::benchmark_data_smithi194_103039_object144285:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400b38f0:::benchmark_data_smithi194_103039_object45138:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400cc9bc:::benchmark_data_smithi194_103039_object107880:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400d3947:::benchmark_data_smithi194_103039_object82231:head# 2024-04-29T14:21:28.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400da435:::benchmark_data_smithi194_103039_object154795:head# 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400dcf1b:::benchmark_data_smithi194_103039_object109359:head# 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:400f6823:::benchmark_data_smithi194_103039_object82242:head# 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40111fa5:::benchmark_data_smithi194_103039_object138578:head# 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40115986:::benchmark_data_smithi194_103039_object84970:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4012759c:::benchmark_data_smithi194_103039_object107099:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4012fb74:::benchmark_data_smithi194_103039_object52943:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40143fe9:::benchmark_data_smithi194_103039_object69443:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401470f0:::benchmark_data_smithi194_103039_object123342:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4014a979:::benchmark_data_smithi194_103039_object150463:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4015b2af:::benchmark_data_smithi194_103039_object51369:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4016c215:::benchmark_data_smithi194_103039_object68627:head# 2024-04-29T14:21:28.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4016f240:::benchmark_data_smithi194_103039_object119595:head# 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4016f9df:::benchmark_data_smithi194_103039_object128021:head# 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40170408:::benchmark_data_smithi194_103039_object118058:head# 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40182254:::benchmark_data_smithi194_103039_object108862:head# 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401836b3:::benchmark_data_smithi194_103039_object97567:head# 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401858e3:::benchmark_data_smithi194_103039_object129178:head# 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40186707:::benchmark_data_smithi194_103039_object75459:head# 2024-04-29T14:21:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401967bb:::benchmark_data_smithi194_103039_object74218:head# 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4019e36e:::benchmark_data_smithi194_103039_object59937:head# 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401a8f9f:::benchmark_data_smithi194_103039_object72742:head# 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401b7f7d:::benchmark_data_smithi194_103039_object142696:head# 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401bda78:::benchmark_data_smithi194_103039_object95682:head# 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401c1b12:::benchmark_data_smithi194_103039_object134175:head# 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401c80ef:::benchmark_data_smithi194_103039_object135529:head# 2024-04-29T14:21:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401ce4bb:::benchmark_data_smithi194_103039_object65032:head# 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:401d4f8c:::benchmark_data_smithi194_103039_object43324:head# 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40209b53:::benchmark_data_smithi194_103039_object92757:head# 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4022d098:::benchmark_data_smithi194_103039_object143576:head# 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40231eab:::benchmark_data_smithi194_103039_object60966:head# 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40232dd0:::benchmark_data_smithi194_103039_object150384:head# 2024-04-29T14:21:28.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4023794b:::benchmark_data_smithi194_103039_object94555:head# 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402461fc:::benchmark_data_smithi194_103039_object121890:head# 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40255ed6:::benchmark_data_smithi194_103039_object112007:head# 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40258975:::benchmark_data_smithi194_103039_object156062:head# 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4025c323:::benchmark_data_smithi194_103039_object149851:head# 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4025ff77:::benchmark_data_smithi194_103039_object73390:head# 2024-04-29T14:21:28.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4026986d:::benchmark_data_smithi194_103039_object100474:head# 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4026ba51:::benchmark_data_smithi194_103039_object129107:head# 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4026dd73:::benchmark_data_smithi194_103039_object121295:head# 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40274d0c:::benchmark_data_smithi194_103039_object97516:head# 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40278c39:::benchmark_data_smithi194_103039_object133516:head# 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4027b459:::benchmark_data_smithi194_103039_object92201:head# 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4027f8ee:::benchmark_data_smithi194_103039_object143648:head# 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402917fa:::benchmark_data_smithi194_103039_object45599:head# 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402961d1:::benchmark_data_smithi194_103039_object107490:head# 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4029bc9e:::benchmark_data_smithi194_103039_object45090:head# 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402a8737:::benchmark_data_smithi194_103039_object135143:head# 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402b2639:::benchmark_data_smithi194_103039_object42022:head# 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402b66a9:::benchmark_data_smithi194_103039_object76413:head# 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402b924b:::benchmark_data_smithi194_103039_object86141:head# 2024-04-29T14:21:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402bbbe1:::benchmark_data_smithi194_103039_object46515:head# 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402d7bfa:::benchmark_data_smithi194_103039_object114926:head# 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402f86f8:::benchmark_data_smithi194_103039_object128230:head# 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402f9860:::benchmark_data_smithi194_103039_object76909:head# 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:402f9b9e:::benchmark_data_smithi194_103039_object55243:head# 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4030080a:::benchmark_data_smithi194_103039_object154785:head# 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4030a955:::benchmark_data_smithi194_103039_object69373:head# 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4030c102:::benchmark_data_smithi194_103039_object127510:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4030d3b0:::benchmark_data_smithi194_103039_object133930:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4030fb89:::benchmark_data_smithi194_103039_object96781:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40317c85:::benchmark_data_smithi194_103039_object43904:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40337993:::benchmark_data_smithi194_103039_object109263:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40339717:::benchmark_data_smithi194_103039_object75367:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40349f11:::benchmark_data_smithi194_103039_object66055:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40354541:::benchmark_data_smithi194_103039_object131353:head# 2024-04-29T14:21:28.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4035a24a:::benchmark_data_smithi194_103039_object97509:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4035ff77:::benchmark_data_smithi194_103039_object53942:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40360543:::benchmark_data_smithi194_103039_object98151:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403716c2:::benchmark_data_smithi194_103039_object134805:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40373158:::benchmark_data_smithi194_103039_object122964:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40378d19:::benchmark_data_smithi194_103039_object89041:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4038a4f7:::benchmark_data_smithi194_103039_object61896:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4038a539:::benchmark_data_smithi194_103039_object150045:head# 2024-04-29T14:21:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4039be8f:::benchmark_data_smithi194_103039_object95016:head# 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403a56ef:::benchmark_data_smithi194_103039_object156110:head# 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403aac5e:::benchmark_data_smithi194_103039_object57146:head# 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403b3906:::benchmark_data_smithi194_103039_object115560:head# 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403b588f:::benchmark_data_smithi194_103039_object131161:head# 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403bad91:::benchmark_data_smithi194_103039_object80148:head# 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403bfe87:::benchmark_data_smithi194_103039_object72646:head# 2024-04-29T14:21:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403c4aa8:::benchmark_data_smithi194_103039_object89202:head# 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403cf35a:::benchmark_data_smithi194_103039_object137225:head# 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403d9d95:::benchmark_data_smithi194_103039_object48607:head# 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403df1d4:::benchmark_data_smithi194_103039_object53363:head# 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403e0a64:::benchmark_data_smithi194_103039_object142892:head# 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403e34f0:::benchmark_data_smithi194_103039_object67786:head# 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:403f6667:::benchmark_data_smithi194_103039_object56222:head# 2024-04-29T14:21:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40402200:::benchmark_data_smithi194_103039_object145431:head# 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40402743:::benchmark_data_smithi194_103039_object65611:head# 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404051f2:::benchmark_data_smithi194_103039_object141666:head# 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4041d032:::benchmark_data_smithi194_103039_object153062:head# 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4041f5a2:::benchmark_data_smithi194_103039_object107577:head# 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4042434e:::benchmark_data_smithi194_103039_object76036:head# 2024-04-29T14:21:28.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4043216a:::benchmark_data_smithi194_103039_object49606:head# 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40434a03:::benchmark_data_smithi194_103039_object152622:head# 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4044fd1b:::benchmark_data_smithi194_103039_object133533:head# 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40453fa8:::benchmark_data_smithi194_103039_object112497:head# 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40458959:::benchmark_data_smithi194_103039_object68096:head# 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40459a18:::benchmark_data_smithi194_103039_object80568:head# 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40467029:::benchmark_data_smithi194_103039_object57621:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40467ca1:::benchmark_data_smithi194_103039_object57809:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4046c13c:::benchmark_data_smithi194_103039_object129273:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40480360:::benchmark_data_smithi194_103039_object71546:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4048774b:::benchmark_data_smithi194_103039_object65542:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404b5d9f:::benchmark_data_smithi194_103039_object94672:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404bce9f:::benchmark_data_smithi194_103039_object85774:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404c1c70:::benchmark_data_smithi194_103039_object132664:head# 2024-04-29T14:21:28.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404c742b:::benchmark_data_smithi194_103039_object136750:head# 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404c859b:::benchmark_data_smithi194_103039_object97891:head# 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404ccdb9:::benchmark_data_smithi194_103039_object125050:head# 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404e1d0e:::benchmark_data_smithi194_103039_object44970:head# 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404e50b0:::benchmark_data_smithi194_103039_object103560:head# 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404e6f4c:::benchmark_data_smithi194_103039_object107390:head# 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404ecd84:::benchmark_data_smithi194_103039_object107638:head# 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:404ee6fa:::benchmark_data_smithi194_103039_object143549:head# 2024-04-29T14:21:28.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40505c8c:::benchmark_data_smithi194_103039_object74044:head# 2024-04-29T14:21:28.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:28 smithi040 ceph-mon[39581]: pgmap v991: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:28.403 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40507607:::benchmark_data_smithi194_103039_object65192:head# 2024-04-29T14:21:28.403 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4050d8c6:::benchmark_data_smithi194_103039_object76487:head# 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40518792:::benchmark_data_smithi194_103039_object52012:head# 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40543ae5:::benchmark_data_smithi194_103039_object94293:head# 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405461e7:::benchmark_data_smithi194_103039_object127765:head# 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4054aedc:::benchmark_data_smithi194_103039_object107322:head# 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4054d89d:::benchmark_data_smithi194_103039_object77260:head# 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40554582:::benchmark_data_smithi194_103039_object99450:head# 2024-04-29T14:21:28.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4056440c:::benchmark_data_smithi194_103039_object124675:head# 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405668f6:::benchmark_data_smithi194_103039_object112003:head# 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40568a80:::benchmark_data_smithi194_103039_object69511:head# 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4056d7e8:::benchmark_data_smithi194_103039_object42392:head# 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405710f8:::benchmark_data_smithi194_103039_object140478:head# 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405731d1:::benchmark_data_smithi194_103039_object56818:head# 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4057d506:::benchmark_data_smithi194_103039_object146782:head# 2024-04-29T14:21:28.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405a13ba:::benchmark_data_smithi194_103039_object126382:head# 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405a2ead:::benchmark_data_smithi194_103039_object57634:head# 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405a3842:::benchmark_data_smithi194_103039_object43407:head# 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405a6ef7:::benchmark_data_smithi194_103039_object109400:head# 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405acc9e:::benchmark_data_smithi194_103039_object90774:head# 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405bb9e0:::benchmark_data_smithi194_103039_object106436:head# 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405bc9c1:::benchmark_data_smithi194_103039_object85854:head# 2024-04-29T14:21:28.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405df3cd:::benchmark_data_smithi194_103039_object111486:head# 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405f1a78:::benchmark_data_smithi194_103039_object71976:head# 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405fbd2e:::benchmark_data_smithi194_103039_object42486:head# 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405fc069:::benchmark_data_smithi194_103039_object103508:head# 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:405ff56c:::benchmark_data_smithi194_103039_object53965:head# 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406005cc:::benchmark_data_smithi194_103039_object126570:head# 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4060622c:::benchmark_data_smithi194_103039_object109647:head# 2024-04-29T14:21:28.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40615569:::benchmark_data_smithi194_103039_object96241:head# 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4061a4a1:::benchmark_data_smithi194_103039_object47396:head# 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4062ed04:::benchmark_data_smithi194_103039_object57773:head# 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4063b07f:::benchmark_data_smithi194_103039_object78672:head# 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4064520b:::benchmark_data_smithi194_103039_object135639:head# 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40646b62:::benchmark_data_smithi194_103039_object114960:head# 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4065e6c7:::benchmark_data_smithi194_103039_object67691:head# 2024-04-29T14:21:28.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4065f8b3:::benchmark_data_smithi194_103039_object75809:head# 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40665dc1:::benchmark_data_smithi194_103039_object73108:head# 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4066cb7a:::benchmark_data_smithi194_103039_object132739:head# 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4066d01d:::benchmark_data_smithi194_103039_object136694:head# 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40672966:::benchmark_data_smithi194_103039_object88103:head# 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4067b5ba:::benchmark_data_smithi194_103039_object89799:head# 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4068e552:::benchmark_data_smithi194_103039_object69444:head# 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406b2fd2:::benchmark_data_smithi194_103039_object119406:head# 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406b64f8:::benchmark_data_smithi194_103039_object66069:head# 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406ba2be:::benchmark_data_smithi194_103039_object47612:head# 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406d4b40:::benchmark_data_smithi194_103039_object149137:head# 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406d4c70:::benchmark_data_smithi194_103039_object101956:head# 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406d9ed0:::benchmark_data_smithi194_103039_object152217:head# 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406de59c:::benchmark_data_smithi194_103039_object51753:head# 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406e55fd:::benchmark_data_smithi194_103039_object136004:head# 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406eb686:::benchmark_data_smithi194_103039_object104082:head# 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406ee4bc:::benchmark_data_smithi194_103039_object105817:head# 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406f4832:::benchmark_data_smithi194_103039_object148967:head# 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406f8f40:::benchmark_data_smithi194_103039_object154805:head# 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:406f90d0:::benchmark_data_smithi194_103039_object100264:head# 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4070407f:::benchmark_data_smithi194_103039_object82323:head# 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40707790:::benchmark_data_smithi194_103039_object60907:head# 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407085f0:::benchmark_data_smithi194_103039_object125550:head# 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4070c675:::benchmark_data_smithi194_103039_object128189:head# 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4071d347:::benchmark_data_smithi194_103039_object96391:head# 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4072ce6a:::benchmark_data_smithi194_103039_object112641:head# 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4073f1dc:::benchmark_data_smithi194_103039_object59620:head# 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407498ff:::benchmark_data_smithi194_103039_object113912:head# 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407663e2:::benchmark_data_smithi194_103039_object121887:head# 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40779c39:::benchmark_data_smithi194_103039_object56143:head# 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4077ede0:::benchmark_data_smithi194_103039_object139665:head# 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40783368:::benchmark_data_smithi194_103039_object114720:head# 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40785dd2:::benchmark_data_smithi194_103039_object64997:head# 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4078be8d:::benchmark_data_smithi194_103039_object138245:head# 2024-04-29T14:21:28.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407b43be:::benchmark_data_smithi194_103039_object127526:head# 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407c06f1:::benchmark_data_smithi194_103039_object73958:head# 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407c10bb:::benchmark_data_smithi194_103039_object77324:head# 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407c3a30:::benchmark_data_smithi194_103039_object94734:head# 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407c7473:::benchmark_data_smithi194_103039_object123525:head# 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407d08a7:::benchmark_data_smithi194_103039_object55691:head# 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407d3243:::benchmark_data_smithi194_103039_object82184:head# 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407dccb0:::benchmark_data_smithi194_103039_object74330:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:407eec6e:::benchmark_data_smithi194_103039_object63745:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40800aa7:::benchmark_data_smithi194_103039_object115242:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4080189e:::benchmark_data_smithi194_103039_object102062:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40802c93:::benchmark_data_smithi194_103039_object146510:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408073a7:::benchmark_data_smithi194_103039_object117185:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408093b1:::benchmark_data_smithi194_103039_object72552:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4080ce06:::benchmark_data_smithi194_103039_object132477:head# 2024-04-29T14:21:28.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4080ef8d:::benchmark_data_smithi194_103039_object89123:head# 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4080fd1d:::benchmark_data_smithi194_103039_object99180:head# 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408126b4:::benchmark_data_smithi194_103039_object127481:head# 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4081fd10:::benchmark_data_smithi194_103039_object105285:head# 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4082e439:::benchmark_data_smithi194_103039_object136367:head# 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4082f61a:::benchmark_data_smithi194_103039_object114279:head# 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40839736:::benchmark_data_smithi194_103039_object147008:head# 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4083e29a:::benchmark_data_smithi194_103039_object135374:head# 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4083ece7:::benchmark_data_smithi194_103039_object54547:head# 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40840c47:::benchmark_data_smithi194_103039_object117065:head# 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4084a82a:::benchmark_data_smithi194_103039_object99466:head# 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4084fe07:::benchmark_data_smithi194_103039_object129233:head# 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4086eaf6:::benchmark_data_smithi194_103039_object48264:head# 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4087c8e0:::benchmark_data_smithi194_103039_object55586:head# 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4088e781:::benchmark_data_smithi194_103039_object60611:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408917c3:::benchmark_data_smithi194_103039_object61629:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40892d26:::benchmark_data_smithi194_103039_object152583:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408a0529:::benchmark_data_smithi194_103039_object92128:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408a6d0f:::benchmark_data_smithi194_103039_object92086:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408b10d1:::benchmark_data_smithi194_103039_object67814:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408cc062:::benchmark_data_smithi194_103039_object156265:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408d66c8:::benchmark_data_smithi194_103039_object126733:head# 2024-04-29T14:21:28.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408d8c34:::benchmark_data_smithi194_103039_object96693:head# 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408f05cf:::benchmark_data_smithi194_103039_object152050:head# 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408f3c40:::benchmark_data_smithi194_103039_object152303:head# 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408f68da:::benchmark_data_smithi194_103039_object59448:head# 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408f6e83:::benchmark_data_smithi194_103039_object69505:head# 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:408fae48:::benchmark_data_smithi194_103039_object79551:head# 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40902e72:::benchmark_data_smithi194_103039_object73795:head# 2024-04-29T14:21:28.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4090bb6e:::benchmark_data_smithi194_103039_object46439:head# 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409255bd:::benchmark_data_smithi194_103039_object128647:head# 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4092a0ff:::benchmark_data_smithi194_103039_object140446:head# 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40930250:::benchmark_data_smithi194_103039_object101959:head# 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4093a5e0:::benchmark_data_smithi194_103039_object71643:head# 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4094e197:::benchmark_data_smithi194_103039_object107608:head# 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4095e89c:::benchmark_data_smithi194_103039_object118081:head# 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4095f4f4:::benchmark_data_smithi194_103039_object54387:head# 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409672cb:::benchmark_data_smithi194_103039_object70468:head# 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4097d673:::benchmark_data_smithi194_103039_object76725:head# 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40989d5d:::benchmark_data_smithi194_103039_object121463:head# 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40996d50:::benchmark_data_smithi194_103039_object121761:head# 2024-04-29T14:21:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409a2ac2:::benchmark_data_smithi194_103039_object72972:head# 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409a6f90:::benchmark_data_smithi194_103039_object132233:head# 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409ac517:::benchmark_data_smithi194_103039_object93693:head# 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409b2023:::benchmark_data_smithi194_103039_object75400:head# 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409bbc3f:::benchmark_data_smithi194_103039_object112144:head# 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409c28e4:::benchmark_data_smithi194_103039_object149701:head# 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409c6ea8:::benchmark_data_smithi194_103039_object69533:head# 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:409dddd9:::benchmark_data_smithi194_103039_object53506:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a051e8:::benchmark_data_smithi194_103039_object138747:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a24c48:::benchmark_data_smithi194_103039_object100689:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a5032e:::benchmark_data_smithi194_103039_object79709:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a57cc8:::benchmark_data_smithi194_103039_object81593:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a5fdf9:::benchmark_data_smithi194_103039_object72135:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a63e25:::benchmark_data_smithi194_103039_object64384:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a68860:::benchmark_data_smithi194_103039_object144579:head# 2024-04-29T14:21:28.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a6bfac:::benchmark_data_smithi194_103039_object119535:head# 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a742ab:::benchmark_data_smithi194_103039_object43389:head# 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a77790:::benchmark_data_smithi194_103039_object99795:head# 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a7fd6d:::benchmark_data_smithi194_103039_object47218:head# 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a8c81b:::benchmark_data_smithi194_103039_object67130:head# 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a8d199:::benchmark_data_smithi194_103039_object126656:head# 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a8ea4e:::benchmark_data_smithi194_103039_object144660:head# 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40a9f5b1:::benchmark_data_smithi194_103039_object111564:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40aa3313:::benchmark_data_smithi194_103039_object53877:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ab5b0e:::benchmark_data_smithi194_103039_object98565:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40abc325:::benchmark_data_smithi194_103039_object139442:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ac87b6:::benchmark_data_smithi194_103039_object100478:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40aca3d9:::benchmark_data_smithi194_103039_object125139:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40acc45d:::benchmark_data_smithi194_103039_object128857:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ae57d1:::benchmark_data_smithi194_103039_object115690:head# 2024-04-29T14:21:28.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40afb1bc:::benchmark_data_smithi194_103039_object62283:head# 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b07863:::benchmark_data_smithi194_103039_object130659:head# 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b09840:::benchmark_data_smithi194_103039_object71736:head# 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b0ac9d:::benchmark_data_smithi194_103039_object81415:head# 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b13a38:::benchmark_data_smithi194_103039_object141286:head# 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b176dc:::benchmark_data_smithi194_103039_object125909:head# 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b1aa2b:::benchmark_data_smithi194_103039_object47990:head# 2024-04-29T14:21:28.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b1c543:::benchmark_data_smithi194_103039_object156061:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b1eef2:::benchmark_data_smithi194_103039_object106554:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b21eba:::benchmark_data_smithi194_103039_object101404:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b291f5:::benchmark_data_smithi194_103039_object95925:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b36d5a:::benchmark_data_smithi194_103039_object81784:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b392e5:::benchmark_data_smithi194_103039_object110636:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b3ed7a:::benchmark_data_smithi194_103039_object114209:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b68cec:::benchmark_data_smithi194_103039_object82946:head# 2024-04-29T14:21:28.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b6d951:::benchmark_data_smithi194_103039_object51774:head# 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b719c1:::benchmark_data_smithi194_103039_object76401:head# 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b7aa62:::benchmark_data_smithi194_103039_object136505:head# 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b82750:::benchmark_data_smithi194_103039_object75678:head# 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b871dd:::benchmark_data_smithi194_103039_object77807:head# 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b8c92b:::benchmark_data_smithi194_103039_object75248:head# 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b8d1df:::benchmark_data_smithi194_103039_object92945:head# 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b950e0:::benchmark_data_smithi194_103039_object151244:head# 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40b9b562:::benchmark_data_smithi194_103039_object111749:head# 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ba66ad:::benchmark_data_smithi194_103039_object47566:head# 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40bb193d:::benchmark_data_smithi194_103039_object133478:head# 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40bb8229:::benchmark_data_smithi194_103039_object122190:head# 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40bc0adf:::benchmark_data_smithi194_103039_object151524:head# 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40bd020a:::benchmark_data_smithi194_103039_object92168:head# 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40bdede9:::benchmark_data_smithi194_103039_object119217:head# 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40be8927:::benchmark_data_smithi194_103039_object97647:head# 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40bec734:::benchmark_data_smithi194_103039_object149730:head# 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c01f87:::benchmark_data_smithi194_103039_object128512:head# 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c07ab8:::benchmark_data_smithi194_103039_object134425:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c0fcec:::benchmark_data_smithi194_103039_object140571:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c20a41:::benchmark_data_smithi194_103039_object149834:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c32bd4:::benchmark_data_smithi194_103039_object105575:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c44e3f:::benchmark_data_smithi194_103039_object135801:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c56d0a:::benchmark_data_smithi194_103039_object99019:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c5c5ad:::benchmark_data_smithi194_103039_object112651:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c64a92:::benchmark_data_smithi194_103039_object130865:head# 2024-04-29T14:21:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c68a0f:::benchmark_data_smithi194_103039_object108444:head# 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c7036a:::benchmark_data_smithi194_103039_object58162:head# 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c71830:::benchmark_data_smithi194_103039_object77628:head# 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c71c02:::benchmark_data_smithi194_103039_object87537:head# 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c763a5:::benchmark_data_smithi194_103039_object109728:head# 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c7e90f:::benchmark_data_smithi194_103039_object46812:head# 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c817e6:::benchmark_data_smithi194_103039_object142306:head# 2024-04-29T14:21:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40c8c6fd:::benchmark_data_smithi194_103039_object146317:head# 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40cb03b0:::benchmark_data_smithi194_103039_object128474:head# 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ccabce:::benchmark_data_smithi194_103039_object141348:head# 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ccacb9:::benchmark_data_smithi194_103039_object49986:head# 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40cd07bd:::benchmark_data_smithi194_103039_object105792:head# 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40cdc6bc:::benchmark_data_smithi194_103039_object77139:head# 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ce0434:::benchmark_data_smithi194_103039_object113435:head# 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ce1921:::benchmark_data_smithi194_103039_object109701:head# 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ce4ede:::benchmark_data_smithi194_103039_object136272:head# 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ce62a5:::benchmark_data_smithi194_103039_object86374:head# 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ce7e3b:::benchmark_data_smithi194_103039_object48399:head# 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40cebc60:::benchmark_data_smithi194_103039_object50175:head# 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40cecf4f:::benchmark_data_smithi194_103039_object80948:head# 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d0c068:::benchmark_data_smithi194_103039_object75714:head# 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d18080:::benchmark_data_smithi194_103039_object94449:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d2c33c:::benchmark_data_smithi194_103039_object155201:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d42525:::benchmark_data_smithi194_103039_object47205:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d4d7ef:::benchmark_data_smithi194_103039_object118574:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d512a9:::benchmark_data_smithi194_103039_object96960:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d51aa2:::benchmark_data_smithi194_103039_object56625:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d546f3:::benchmark_data_smithi194_103039_object113436:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d555b0:::benchmark_data_smithi194_103039_object43222:head# 2024-04-29T14:21:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d82827:::benchmark_data_smithi194_103039_object42354:head# 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40d8566c:::benchmark_data_smithi194_103039_object103435:head# 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40da6069:::benchmark_data_smithi194_103039_object98678:head# 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40da8c57:::benchmark_data_smithi194_103039_object118837:head# 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40db9793:::benchmark_data_smithi194_103039_object81893:head# 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40dbf7ae:::benchmark_data_smithi194_103039_object66998:head# 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40dc2d3c:::benchmark_data_smithi194_103039_object67927:head# 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40dc78b1:::benchmark_data_smithi194_103039_object109089:head# 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40dc8e96:::benchmark_data_smithi194_103039_object88755:head# 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40de665a:::benchmark_data_smithi194_103039_object143442:head# 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e12df0:::benchmark_data_smithi194_103039_object101804:head# 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e3223a:::benchmark_data_smithi194_103039_object107782:head# 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e413e5:::benchmark_data_smithi194_103039_object83439:head# 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e4b5ae:::benchmark_data_smithi194_103039_object143346:head# 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e4c636:::benchmark_data_smithi194_103039_object117123:head# 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e5269f:::benchmark_data_smithi194_103039_object75543:head# 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e55bc6:::benchmark_data_smithi194_103039_object115210:head# 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e566f9:::benchmark_data_smithi194_103039_object76261:head# 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e69c4c:::benchmark_data_smithi194_103039_object153130:head# 2024-04-29T14:21:28.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e6ed9f:::benchmark_data_smithi194_103039_object123839:head# 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e73cf6:::benchmark_data_smithi194_103039_object133910:head# 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e74dc2:::benchmark_data_smithi194_103039_object75621:head# 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e8ee54:::benchmark_data_smithi194_103039_object47617:head# 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40e9107c:::benchmark_data_smithi194_103039_object57090:head# 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ea102d:::benchmark_data_smithi194_103039_object106397:head# 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ea63ff:::benchmark_data_smithi194_103039_object78758:head# 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40eb111b:::benchmark_data_smithi194_103039_object134558:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40eb3f11:::benchmark_data_smithi194_103039_object86554:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ec234f:::benchmark_data_smithi194_103039_object58614:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ed14b7:::benchmark_data_smithi194_103039_object79841:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40edb056:::benchmark_data_smithi194_103039_object133040:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40edc54a:::benchmark_data_smithi194_103039_object135280:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40ee5fd7:::benchmark_data_smithi194_103039_object154491:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40eebcb2:::benchmark_data_smithi194_103039_object103192:head# 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f04f7c:::benchmark_data_smithi194_103039_object153478:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f25216:::benchmark_data_smithi194_103039_object93685:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f264ac:::benchmark_data_smithi194_103039_object96369:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f2dcb5:::benchmark_data_smithi194_103039_object111456:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f2dcfd:::benchmark_data_smithi194_103039_object137231:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f35fc0:::benchmark_data_smithi194_103039_object105744:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f4a5a8:::benchmark_data_smithi194_103039_object74522:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f4b972:::benchmark_data_smithi194_103039_object112124:head# 2024-04-29T14:21:28.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f529ab:::benchmark_data_smithi194_103039_object152733:head# 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f5d2fa:::benchmark_data_smithi194_103039_object73440:head# 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f5ed16:::benchmark_data_smithi194_103039_object127814:head# 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f61451:::benchmark_data_smithi194_103039_object120862:head# 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f65ec7:::benchmark_data_smithi194_103039_object68414:head# 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f811ac:::benchmark_data_smithi194_103039_object139392:head# 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f86b57:::benchmark_data_smithi194_103039_object44172:head# 2024-04-29T14:21:28.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40f9b58e:::benchmark_data_smithi194_103039_object115616:head# 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40fd1a22:::benchmark_data_smithi194_103039_object108799:head# 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40fd6378:::benchmark_data_smithi194_103039_object61099:head# 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40feb0ea:::benchmark_data_smithi194_103039_object102584:head# 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40fefb14:::benchmark_data_smithi194_103039_object84610:head# 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:40fff4be:::benchmark_data_smithi194_103039_object49902:head# 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41001d3e:::benchmark_data_smithi194_103039_object122324:head# 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41004cfa:::benchmark_data_smithi194_103039_object58299:head# 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41006b07:::benchmark_data_smithi194_103039_object85267:head# 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410143d3:::benchmark_data_smithi194_103039_object149241:head# 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4101fd3a:::benchmark_data_smithi194_103039_object144946:head# 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41021bf9:::benchmark_data_smithi194_103039_object121108:head# 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410322b2:::benchmark_data_smithi194_103039_object103518:head# 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410331a3:::benchmark_data_smithi194_103039_object46735:head# 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4103b794:::benchmark_data_smithi194_103039_object119862:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41046ee2:::benchmark_data_smithi194_103039_object98649:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4104ef42:::benchmark_data_smithi194_103039_object67348:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410543d2:::benchmark_data_smithi194_103039_object105172:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41055530:::benchmark_data_smithi194_103039_object103726:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41059e9e:::benchmark_data_smithi194_103039_object77951:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4105eb8e:::benchmark_data_smithi194_103039_object58514:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410619e4:::benchmark_data_smithi194_103039_object111713:head# 2024-04-29T14:21:28.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41066de8:::benchmark_data_smithi194_103039_object87310:head# 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4106cca8:::benchmark_data_smithi194_103039_object110875:head# 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4106ea95:::benchmark_data_smithi194_103039_object66844:head# 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410735b5:::benchmark_data_smithi194_103039_object138591:head# 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4107e68f:::benchmark_data_smithi194_103039_object145591:head# 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4108f79e:::benchmark_data_smithi194_103039_object145443:head# 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41091872:::benchmark_data_smithi194_103039_object138759:head# 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4109e8f2:::benchmark_data_smithi194_103039_object87663:head# 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410a8418:::benchmark_data_smithi194_103039_object108888:head# 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410affce:::benchmark_data_smithi194_103039_object51728:head# 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410b1ed2:::benchmark_data_smithi194_103039_object98348:head# 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410caf04:::benchmark_data_smithi194_103039_object112615:head# 2024-04-29T14:21:28.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410df4b9:::benchmark_data_smithi194_103039_object81391:head# 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410e798c:::benchmark_data_smithi194_103039_object111568:head# 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:410ed17a:::benchmark_data_smithi194_103039_object145506:head# 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411019a8:::benchmark_data_smithi194_103039_object149689:head# 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41120274:::benchmark_data_smithi194_103039_object47765:head# 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41126428:::benchmark_data_smithi194_103039_object69650:head# 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4113fe07:::benchmark_data_smithi194_103039_object83731:head# 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41143d4d:::benchmark_data_smithi194_103039_object84239:head# 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4114affd:::benchmark_data_smithi194_103039_object154479:head# 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4116b8cc:::benchmark_data_smithi194_103039_object141768:head# 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41173277:::benchmark_data_smithi194_103039_object80424:head# 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4117ae07:::benchmark_data_smithi194_103039_object51366:head# 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4118d9fa:::benchmark_data_smithi194_103039_object129731:head# 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411929c4:::benchmark_data_smithi194_103039_object98773:head# 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411932ed:::benchmark_data_smithi194_103039_object154618:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411aece1:::benchmark_data_smithi194_103039_object59283:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411c2c8c:::benchmark_data_smithi194_103039_object92941:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411c4ff7:::benchmark_data_smithi194_103039_object96665:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411cd196:::benchmark_data_smithi194_103039_object142717:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411e151c:::benchmark_data_smithi194_103039_object94062:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411e32a9:::benchmark_data_smithi194_103039_object44644:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411e4e35:::benchmark_data_smithi194_103039_object148623:head# 2024-04-29T14:21:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411e5242:::benchmark_data_smithi194_103039_object98252:head# 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:411ec9de:::benchmark_data_smithi194_103039_object125379:head# 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41203f31:::benchmark_data_smithi194_103039_object130752:head# 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4120eb59:::benchmark_data_smithi194_103039_object142726:head# 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41217c02:::benchmark_data_smithi194_103039_object57802:head# 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41220a57:::benchmark_data_smithi194_103039_object66745:head# 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4123957d:::benchmark_data_smithi194_103039_object108382:head# 2024-04-29T14:21:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41240d1d:::benchmark_data_smithi194_103039_object56576:head# 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41241040:::benchmark_data_smithi194_103039_object93050:head# 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412474fa:::benchmark_data_smithi194_103039_object116628:head# 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41250d34:::benchmark_data_smithi194_103039_object84711:head# 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4125677e:::benchmark_data_smithi194_103039_object78394:head# 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4125a006:::benchmark_data_smithi194_103039_object129985:head# 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412615f3:::benchmark_data_smithi194_103039_object95345:head# 2024-04-29T14:21:28.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4126b674:::benchmark_data_smithi194_103039_object122330:head# 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41273e73:::benchmark_data_smithi194_103039_object47120:head# 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412876c9:::benchmark_data_smithi194_103039_object72117:head# 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4128e4db:::benchmark_data_smithi194_103039_object150408:head# 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41299d59:::benchmark_data_smithi194_103039_object55200:head# 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412b7a77:::benchmark_data_smithi194_103039_object47615:head# 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412bfcb4:::benchmark_data_smithi194_103039_object86810:head# 2024-04-29T14:21:28.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412c6c4b:::benchmark_data_smithi194_103039_object147382:head# 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412d126a:::benchmark_data_smithi194_103039_object88115:head# 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412d1fb2:::benchmark_data_smithi194_103039_object51297:head# 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412da889:::benchmark_data_smithi194_103039_object150716:head# 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:412e913e:::benchmark_data_smithi194_103039_object128539:head# 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41300c71:::benchmark_data_smithi194_103039_object64603:head# 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4130e70d:::benchmark_data_smithi194_103039_object58807:head# 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41316bf9:::benchmark_data_smithi194_103039_object42123:head# 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4134f618:::benchmark_data_smithi194_103039_object61636:head# 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4134f79c:::benchmark_data_smithi194_103039_object120101:head# 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413657c0:::benchmark_data_smithi194_103039_object56159:head# 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4136c94c:::benchmark_data_smithi194_103039_object44093:head# 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4137162b:::benchmark_data_smithi194_103039_object84883:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41376406:::benchmark_data_smithi194_103039_object42117:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41379e63:::benchmark_data_smithi194_103039_object112950:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4137b098:::benchmark_data_smithi194_103039_object104074:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4139923a:::benchmark_data_smithi194_103039_object134731:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413a7e1a:::benchmark_data_smithi194_103039_object86919:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413acd84:::benchmark_data_smithi194_103039_object57229:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413b2637:::benchmark_data_smithi194_103039_object78846:head# 2024-04-29T14:21:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413be000:::benchmark_data_smithi194_103039_object134868:head# 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413bfb64:::benchmark_data_smithi194_103039_object149974:head# 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413c400a:::benchmark_data_smithi194_103039_object154620:head# 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413c7802:::benchmark_data_smithi194_103039_object102219:head# 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413cd5d8:::benchmark_data_smithi194_103039_object131679:head# 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413dbe11:::benchmark_data_smithi194_103039_object56036:head# 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413dcb5c:::benchmark_data_smithi194_103039_object89106:head# 2024-04-29T14:21:28.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413eba87:::benchmark_data_smithi194_103039_object122705:head# 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:413f68ef:::benchmark_data_smithi194_103039_object93791:head# 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4140ccf2:::benchmark_data_smithi194_103039_object107408:head# 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41413fb9:::benchmark_data_smithi194_103039_object60196:head# 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41417279:::benchmark_data_smithi194_103039_object113997:head# 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4141d41b:::benchmark_data_smithi194_103039_object63069:head# 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4142b8c7:::benchmark_data_smithi194_103039_object62135:head# 2024-04-29T14:21:28.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41442514:::benchmark_data_smithi194_103039_object91391:head# 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4144ebbb:::benchmark_data_smithi194_103039_object121455:head# 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4144fcff:::benchmark_data_smithi194_103039_object80876:head# 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41464b08:::benchmark_data_smithi194_103039_object61739:head# 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4146ea02:::benchmark_data_smithi194_103039_object79135:head# 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4146fbb5:::benchmark_data_smithi194_103039_object130658:head# 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41475189:::benchmark_data_smithi194_103039_object147874:head# 2024-04-29T14:21:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41476d05:::benchmark_data_smithi194_103039_object106853:head# 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41478e3a:::benchmark_data_smithi194_103039_object53829:head# 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41480401:::benchmark_data_smithi194_103039_object100398:head# 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41486ccd:::benchmark_data_smithi194_103039_object94817:head# 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4148c3e6:::benchmark_data_smithi194_103039_object66669:head# 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4148f397:::benchmark_data_smithi194_103039_object107920:head# 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41495134:::benchmark_data_smithi194_103039_object50526:head# 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414a4006:::benchmark_data_smithi194_103039_object47491:head# 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414b6294:::benchmark_data_smithi194_103039_object81425:head# 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414b94ba:::benchmark_data_smithi194_103039_object76855:head# 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414bde91:::benchmark_data_smithi194_103039_object77057:head# 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414c2827:::benchmark_data_smithi194_103039_object105839:head# 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414cf992:::benchmark_data_smithi194_103039_object153355:head# 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414d0880:::benchmark_data_smithi194_103039_object140733:head# 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414dc986:::benchmark_data_smithi194_103039_object143514:head# 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414e263e:::benchmark_data_smithi194_103039_object107344:head# 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:414fa4e0:::benchmark_data_smithi194_103039_object87141:head# 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415001e2:::benchmark_data_smithi194_103039_object131365:head# 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4150106d:::benchmark_data_smithi194_103039_object43509:head# 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41504ae8:::benchmark_data_smithi194_103039_object99930:head# 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4150588b:::benchmark_data_smithi194_103039_object139258:head# 2024-04-29T14:21:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4150b755:::benchmark_data_smithi194_103039_object51771:head# 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41512339:::benchmark_data_smithi194_103039_object93454:head# 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41517da3:::benchmark_data_smithi194_103039_object81624:head# 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415289f8:::benchmark_data_smithi194_103039_object94003:head# 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4152cb0b:::benchmark_data_smithi194_103039_object96276:head# 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41531b18:::benchmark_data_smithi194_103039_object145637:head# 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415326d3:::benchmark_data_smithi194_103039_object84993:head# 2024-04-29T14:21:28.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41533401:::benchmark_data_smithi194_103039_object142490:head# 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4154235d:::benchmark_data_smithi194_103039_object147904:head# 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4154657e:::benchmark_data_smithi194_103039_object73652:head# 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4154671b:::benchmark_data_smithi194_103039_object83901:head# 2024-04-29T14:21:28.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4154f32a:::benchmark_data_smithi194_103039_object87989:head# 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4155c2f8:::benchmark_data_smithi194_103039_object117585:head# 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4155fa1c:::benchmark_data_smithi194_103039_object136518:head# 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4156a0c6:::benchmark_data_smithi194_103039_object46796:head# 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41577215:::benchmark_data_smithi194_103039_object45849:head# 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4158eb34:::benchmark_data_smithi194_103039_object107678:head# 2024-04-29T14:21:28.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415ac80d:::benchmark_data_smithi194_103039_object51719:head# 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415c473c:::benchmark_data_smithi194_103039_object46329:head# 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415cb2aa:::benchmark_data_smithi194_103039_object114619:head# 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415f7b91:::benchmark_data_smithi194_103039_object111316:head# 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:415f8283:::benchmark_data_smithi194_103039_object109651:head# 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4161f9e8:::benchmark_data_smithi194_103039_object73186:head# 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41626bb8:::benchmark_data_smithi194_103039_object149233:head# 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4162d1a2:::benchmark_data_smithi194_103039_object54116:head# 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4163706b:::benchmark_data_smithi194_103039_object66775:head# 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4164722b:::benchmark_data_smithi194_103039_object60055:head# 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4164d949:::benchmark_data_smithi194_103039_object114840:head# 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41659134:::benchmark_data_smithi194_103039_object51580:head# 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4165ca4b:::benchmark_data_smithi194_103039_object149489:head# 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4166a914:::benchmark_data_smithi194_103039_object80907:head# 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4166db43:::benchmark_data_smithi194_103039_object55970:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41670d1e:::benchmark_data_smithi194_103039_object116089:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41676288:::benchmark_data_smithi194_103039_object56854:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4168958a:::benchmark_data_smithi194_103039_object78448:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4169533b:::benchmark_data_smithi194_103039_object78065:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41697b17:::benchmark_data_smithi194_103039_object95155:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416afbb6:::benchmark_data_smithi194_103039_object92389:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416b13cf:::benchmark_data_smithi194_103039_object104650:head# 2024-04-29T14:21:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416b71e1:::benchmark_data_smithi194_103039_object57403:head# 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416b9741:::benchmark_data_smithi194_103039_object155035:head# 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416d1f11:::benchmark_data_smithi194_103039_object55529:head# 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416decbd:::benchmark_data_smithi194_103039_object42612:head# 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416df529:::benchmark_data_smithi194_103039_object52146:head# 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:416efd80:::benchmark_data_smithi194_103039_object119670:head# 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4170282f:::benchmark_data_smithi194_103039_object55959:head# 2024-04-29T14:21:28.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4170f4e6:::benchmark_data_smithi194_103039_object156124:head# 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417230df:::benchmark_data_smithi194_103039_object84705:head# 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417260e8:::benchmark_data_smithi194_103039_object87174:head# 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41726179:::benchmark_data_smithi194_103039_object57651:head# 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4173401e:::benchmark_data_smithi194_103039_object148003:head# 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4173e184:::benchmark_data_smithi194_103039_object82349:head# 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4174e381:::benchmark_data_smithi194_103039_object116215:head# 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41752db2:::benchmark_data_smithi194_103039_object151180:head# 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4175ddbd:::benchmark_data_smithi194_103039_object146302:head# 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41769007:::benchmark_data_smithi194_103039_object138583:head# 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4177244a:::benchmark_data_smithi194_103039_object97652:head# 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4177c35c:::benchmark_data_smithi194_103039_object50300:head# 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41782960:::benchmark_data_smithi194_103039_object151386:head# 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41784937:::benchmark_data_smithi194_103039_object97825:head# 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4178da65:::benchmark_data_smithi194_103039_object82355:head# 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4178f36a:::benchmark_data_smithi194_103039_object136912:head# 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417926ec:::benchmark_data_smithi194_103039_object144790:head# 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417994bc:::benchmark_data_smithi194_103039_object86357:head# 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4179af4d:::benchmark_data_smithi194_103039_object115863:head# 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417b6192:::benchmark_data_smithi194_103039_object145565:head# 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417d4188:::benchmark_data_smithi194_103039_object117578:head# 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417dd3a7:::benchmark_data_smithi194_103039_object125190:head# 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417e588d:::benchmark_data_smithi194_103039_object90513:head# 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417eca51:::benchmark_data_smithi194_103039_object125270:head# 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417efb53:::benchmark_data_smithi194_103039_object141541:head# 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:417ffa02:::benchmark_data_smithi194_103039_object47820:head# 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41801063:::benchmark_data_smithi194_103039_object156565:head# 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418058e2:::benchmark_data_smithi194_103039_object128498:head# 2024-04-29T14:21:28.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41808cc6:::benchmark_data_smithi194_103039_object74866:head# 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4180a3d6:::benchmark_data_smithi194_103039_object86303:head# 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41810f61:::benchmark_data_smithi194_103039_object103115:head# 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4181533a:::benchmark_data_smithi194_103039_object86045:head# 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41817192:::benchmark_data_smithi194_103039_object106860:head# 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4181d6c4:::benchmark_data_smithi194_103039_object127980:head# 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41821275:::benchmark_data_smithi194_103039_object77876:head# 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41826dd4:::benchmark_data_smithi194_103039_object66159:head# 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4182cfe9:::benchmark_data_smithi194_103039_object107998:head# 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4183d066:::benchmark_data_smithi194_103039_object91069:head# 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418431be:::benchmark_data_smithi194_103039_object42741:head# 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41843b28:::benchmark_data_smithi194_103039_object120051:head# 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41844844:::benchmark_data_smithi194_103039_object132329:head# 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4184fb3c:::benchmark_data_smithi194_103039_object110419:head# 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41852a56:::benchmark_data_smithi194_103039_object48215:head# 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4185f6f4:::benchmark_data_smithi194_103039_object67157:head# 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41879204:::benchmark_data_smithi194_103039_object76448:head# 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41879d3e:::benchmark_data_smithi194_103039_object60865:head# 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41896285:::benchmark_data_smithi194_103039_object68644:head# 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4189abc2:::benchmark_data_smithi194_103039_object138835:head# 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418a88ea:::benchmark_data_smithi194_103039_object135069:head# 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418bca30:::benchmark_data_smithi194_103039_object79355:head# 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418bd543:::benchmark_data_smithi194_103039_object99670:head# 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418e8c5d:::benchmark_data_smithi194_103039_object52187:head# 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418f0bff:::benchmark_data_smithi194_103039_object124315:head# 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418f16db:::benchmark_data_smithi194_103039_object96959:head# 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:418f4d77:::benchmark_data_smithi194_103039_object122058:head# 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4190970e:::benchmark_data_smithi194_103039_object55988:head# 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4191576c:::benchmark_data_smithi194_103039_object96964:head# 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419211aa:::benchmark_data_smithi194_103039_object74247:head# 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4192e60f:::benchmark_data_smithi194_103039_object86737:head# 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4192f981:::benchmark_data_smithi194_103039_object116451:head# 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41930a60:::benchmark_data_smithi194_103039_object55286:head# 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41932fda:::benchmark_data_smithi194_103039_object45205:head# 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419498e1:::benchmark_data_smithi194_103039_object85484:head# 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41958d10:::benchmark_data_smithi194_103039_object43515:head# 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4195fd1c:::benchmark_data_smithi194_103039_object82927:head# 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4196fc60:::benchmark_data_smithi194_103039_object82553:head# 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41988f99:::benchmark_data_smithi194_103039_object140639:head# 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4198d37b:::benchmark_data_smithi194_103039_object107521:head# 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419aa186:::benchmark_data_smithi194_103039_object140420:head# 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419b39bb:::benchmark_data_smithi194_103039_object132583:head# 2024-04-29T14:21:28.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419b58e0:::benchmark_data_smithi194_103039_object48252:head# 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419bde49:::benchmark_data_smithi194_103039_object152309:head# 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419cf471:::benchmark_data_smithi194_103039_object42760:head# 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419e17f7:::benchmark_data_smithi194_103039_object62137:head# 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419e68eb:::benchmark_data_smithi194_103039_object127287:head# 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419f23f6:::benchmark_data_smithi194_103039_object119961:head# 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419f6a2a:::benchmark_data_smithi194_103039_object121004:head# 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419f96d1:::benchmark_data_smithi194_103039_object68421:head# 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:419fd194:::benchmark_data_smithi194_103039_object104033:head# 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a14b6c:::benchmark_data_smithi194_103039_object50233:head# 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a1b769:::benchmark_data_smithi194_103039_object74015:head# 2024-04-29T14:21:28.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a1bfef:::benchmark_data_smithi194_103039_object50712:head# 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a255c3:::benchmark_data_smithi194_103039_object94457:head# 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a2784f:::benchmark_data_smithi194_103039_object110666:head# 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a2900b:::benchmark_data_smithi194_103039_object149781:head# 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a2c861:::benchmark_data_smithi194_103039_object97264:head# 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a32f1c:::benchmark_data_smithi194_103039_object109761:head# 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a3b27c:::benchmark_data_smithi194_103039_object93180:head# 2024-04-29T14:21:28.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a3f055:::benchmark_data_smithi194_103039_object143154:head# 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a40df2:::benchmark_data_smithi194_103039_object145242:head# 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a4cc06:::benchmark_data_smithi194_103039_object147526:head# 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a4ced4:::benchmark_data_smithi194_103039_object61270:head# 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a4fb39:::benchmark_data_smithi194_103039_object125127:head# 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a5d446:::benchmark_data_smithi194_103039_object138048:head# 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a6325c:::benchmark_data_smithi194_103039_object124101:head# 2024-04-29T14:21:28.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a6624a:::benchmark_data_smithi194_103039_object49347:head# 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a6afc0:::benchmark_data_smithi194_103039_object92841:head# 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a71082:::benchmark_data_smithi194_103039_object69861:head# 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a7eb2d:::benchmark_data_smithi194_103039_object100493:head# 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a83b11:::benchmark_data_smithi194_103039_object142668:head# 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a86529:::benchmark_data_smithi194_103039_object57714:head# 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a87fbf:::benchmark_data_smithi194_103039_object42081:head# 2024-04-29T14:21:28.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41a9b1e7:::benchmark_data_smithi194_103039_object118088:head# 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41aa2778:::benchmark_data_smithi194_103039_object129403:head# 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41aa5bbe:::benchmark_data_smithi194_103039_object52504:head# 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41aad390:::benchmark_data_smithi194_103039_object52795:head# 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ab5278:::benchmark_data_smithi194_103039_object75999:head# 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ab5f2a:::benchmark_data_smithi194_103039_object121717:head# 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ac1034:::benchmark_data_smithi194_103039_object149463:head# 2024-04-29T14:21:28.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ac1b43:::benchmark_data_smithi194_103039_object91032:head# 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ad393f:::benchmark_data_smithi194_103039_object55166:head# 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41adc19a:::benchmark_data_smithi194_103039_object143728:head# 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41add1ff:::benchmark_data_smithi194_103039_object55045:head# 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41af2214:::benchmark_data_smithi194_103039_object68783:head# 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41af8faf:::benchmark_data_smithi194_103039_object83565:head# 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41afc8da:::benchmark_data_smithi194_103039_object134088:head# 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41afe89b:::benchmark_data_smithi194_103039_object111572:head# 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b02d6b:::benchmark_data_smithi194_103039_object45396:head# 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b0efce:::benchmark_data_smithi194_103039_object146425:head# 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b1217e:::benchmark_data_smithi194_103039_object74477:head# 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b16ee0:::benchmark_data_smithi194_103039_object154804:head# 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b1e599:::benchmark_data_smithi194_103039_object77152:head# 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b2e7fa:::benchmark_data_smithi194_103039_object46847:head# 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b3d0ac:::benchmark_data_smithi194_103039_object121013:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b42e97:::benchmark_data_smithi194_103039_object43914:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b45fa8:::benchmark_data_smithi194_103039_object127449:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b5408d:::benchmark_data_smithi194_103039_object95658:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b5cfaa:::benchmark_data_smithi194_103039_object121459:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b6dcd0:::benchmark_data_smithi194_103039_object70644:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b6e3df:::benchmark_data_smithi194_103039_object150412:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b7f35d:::benchmark_data_smithi194_103039_object143502:head# 2024-04-29T14:21:28.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b85db7:::benchmark_data_smithi194_103039_object106061:head# 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b8921a:::benchmark_data_smithi194_103039_object124290:head# 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b903c8:::benchmark_data_smithi194_103039_object80257:head# 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41b9a538:::benchmark_data_smithi194_103039_object102158:head# 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41bba957:::benchmark_data_smithi194_103039_object63531:head# 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41bbd064:::benchmark_data_smithi194_103039_object89124:head# 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41bc1cde:::benchmark_data_smithi194_103039_object148274:head# 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41bd3f18:::benchmark_data_smithi194_103039_object147466:head# 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41bd6ef3:::benchmark_data_smithi194_103039_object110827:head# 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41bf5643:::benchmark_data_smithi194_103039_object141976:head# 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c0287a:::benchmark_data_smithi194_103039_object69178:head# 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c0bd3f:::benchmark_data_smithi194_103039_object63313:head# 2024-04-29T14:21:28.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c13a49:::benchmark_data_smithi194_103039_object147818:head# 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c1b46e:::benchmark_data_smithi194_103039_object58719:head# 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c215d1:::benchmark_data_smithi194_103039_object126722:head# 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c24501:::benchmark_data_smithi194_103039_object114651:head# 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c2b74e:::benchmark_data_smithi194_103039_object95754:head# 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c3485a:::benchmark_data_smithi194_103039_object98393:head# 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c39f80:::benchmark_data_smithi194_103039_object105263:head# 2024-04-29T14:21:28.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c3a493:::benchmark_data_smithi194_103039_object146503:head# 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c433a5:::benchmark_data_smithi194_103039_object44142:head# 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c441db:::benchmark_data_smithi194_103039_object117307:head# 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c47503:::benchmark_data_smithi194_103039_object105934:head# 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c5ba4e:::benchmark_data_smithi194_103039_object133419:head# 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c708d7:::benchmark_data_smithi194_103039_object143690:head# 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c7ab78:::benchmark_data_smithi194_103039_object112000:head# 2024-04-29T14:21:28.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c7ca89:::benchmark_data_smithi194_103039_object149922:head# 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c8069e:::benchmark_data_smithi194_103039_object47888:head# 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c8b8c4:::benchmark_data_smithi194_103039_object106311:head# 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41c95d46:::benchmark_data_smithi194_103039_object78335:head# 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41cd3e16:::benchmark_data_smithi194_103039_object62578:head# 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41cd421b:::benchmark_data_smithi194_103039_object90311:head# 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41cd5cdb:::benchmark_data_smithi194_103039_object63946:head# 2024-04-29T14:21:28.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ce1214:::benchmark_data_smithi194_103039_object98485:head# 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41cf7534:::benchmark_data_smithi194_103039_object130718:head# 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d06fbd:::benchmark_data_smithi194_103039_object80536:head# 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d36d17:::benchmark_data_smithi194_103039_object43421:head# 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d3a018:::benchmark_data_smithi194_103039_object101170:head# 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d3d9ec:::benchmark_data_smithi194_103039_object112301:head# 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d5039b:::benchmark_data_smithi194_103039_object153309:head# 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d61cef:::benchmark_data_smithi194_103039_object44366:head# 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d63763:::benchmark_data_smithi194_103039_object94612:head# 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d6972f:::benchmark_data_smithi194_103039_object118744:head# 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d69f8f:::benchmark_data_smithi194_103039_object110078:head# 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d76357:::benchmark_data_smithi194_103039_object111309:head# 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d77df4:::benchmark_data_smithi194_103039_object147810:head# 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d79d88:::benchmark_data_smithi194_103039_object95157:head# 2024-04-29T14:21:28.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d868a5:::benchmark_data_smithi194_103039_object109892:head# 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d8e005:::benchmark_data_smithi194_103039_object156342:head# 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41d9e42d:::benchmark_data_smithi194_103039_object76759:head# 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41dbae64:::benchmark_data_smithi194_103039_object131635:head# 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41dd58eb:::benchmark_data_smithi194_103039_object49087:head# 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ddb02d:::benchmark_data_smithi194_103039_object145968:head# 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ddd1dc:::benchmark_data_smithi194_103039_object139710:head# 2024-04-29T14:21:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41df7819:::benchmark_data_smithi194_103039_object72130:head# 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41df85aa:::benchmark_data_smithi194_103039_object53741:head# 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41dfc16a:::benchmark_data_smithi194_103039_object128187:head# 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e00196:::benchmark_data_smithi194_103039_object156006:head# 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e05d5b:::benchmark_data_smithi194_103039_object151679:head# 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e2012e:::benchmark_data_smithi194_103039_object129344:head# 2024-04-29T14:21:28.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e209d6:::benchmark_data_smithi194_103039_object133588:head# 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e23f55:::benchmark_data_smithi194_103039_object91359:head# 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e23ffb:::benchmark_data_smithi194_103039_object83730:head# 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e27680:::benchmark_data_smithi194_103039_object124379:head# 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e2c118:::benchmark_data_smithi194_103039_object139038:head# 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e3a199:::benchmark_data_smithi194_103039_object145060:head# 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e3bd11:::benchmark_data_smithi194_103039_object126776:head# 2024-04-29T14:21:28.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e43037:::benchmark_data_smithi194_103039_object50012:head# 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e4528e:::benchmark_data_smithi194_103039_object46935:head# 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e4c30d:::benchmark_data_smithi194_103039_object107716:head# 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e5e57c:::benchmark_data_smithi194_103039_object77177:head# 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e61b46:::benchmark_data_smithi194_103039_object65016:head# 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e6c1d1:::benchmark_data_smithi194_103039_object96863:head# 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e6e4a4:::benchmark_data_smithi194_103039_object107418:head# 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e71600:::benchmark_data_smithi194_103039_object83385:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e90278:::benchmark_data_smithi194_103039_object96207:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e9150e:::benchmark_data_smithi194_103039_object63318:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41e98b57:::benchmark_data_smithi194_103039_object115157:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ea33ad:::benchmark_data_smithi194_103039_object143446:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ea5521:::benchmark_data_smithi194_103039_object128924:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ec368c:::benchmark_data_smithi194_103039_object78386:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ed0028:::benchmark_data_smithi194_103039_object47591:head# 2024-04-29T14:21:28.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ed64ef:::benchmark_data_smithi194_103039_object129024:head# 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41edca5a:::benchmark_data_smithi194_103039_object79150:head# 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41eeb3f2:::benchmark_data_smithi194_103039_object145305:head# 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41efe5ab:::benchmark_data_smithi194_103039_object73972:head# 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41efff49:::benchmark_data_smithi194_103039_object145533:head# 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f0077e:::benchmark_data_smithi194_103039_object67569:head# 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f030c0:::benchmark_data_smithi194_103039_object89417:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f049c6:::benchmark_data_smithi194_103039_object105097:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f082a4:::benchmark_data_smithi194_103039_object98689:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f0f650:::benchmark_data_smithi194_103039_object133151:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f13bcc:::benchmark_data_smithi194_103039_object127751:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f25777:::benchmark_data_smithi194_103039_object153104:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f35264:::benchmark_data_smithi194_103039_object109850:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f4b361:::benchmark_data_smithi194_103039_object81469:head# 2024-04-29T14:21:28.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f5f507:::benchmark_data_smithi194_103039_object101062:head# 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f63fe0:::benchmark_data_smithi194_103039_object117727:head# 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f6414a:::benchmark_data_smithi194_103039_object139681:head# 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f6c37a:::benchmark_data_smithi194_103039_object109348:head# 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f6cebe:::benchmark_data_smithi194_103039_object78128:head# 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f772a7:::benchmark_data_smithi194_103039_object151982:head# 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f7d16d:::benchmark_data_smithi194_103039_object115179:head# 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f87f5a:::benchmark_data_smithi194_103039_object83153:head# 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f8dac2:::benchmark_data_smithi194_103039_object71019:head# 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41f9fd5c:::benchmark_data_smithi194_103039_object118247:head# 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fa246d:::benchmark_data_smithi194_103039_object41964:head# 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fb3c58:::benchmark_data_smithi194_103039_object97679:head# 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fbfb0e:::benchmark_data_smithi194_103039_object65916:head# 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fcacde:::benchmark_data_smithi194_103039_object103910:head# 2024-04-29T14:21:28.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fcb5c7:::benchmark_data_smithi194_103039_object76858:head# 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fd45ce:::benchmark_data_smithi194_103039_object123748:head# 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fd4622:::benchmark_data_smithi194_103039_object69358:head# 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fdddb5:::benchmark_data_smithi194_103039_object59526:head# 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41fe3cc1:::benchmark_data_smithi194_103039_object128300:head# 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41feab62:::benchmark_data_smithi194_103039_object56525:head# 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:41ffefce:::benchmark_data_smithi194_103039_object57641:head# 2024-04-29T14:21:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4201a98a:::benchmark_data_smithi194_103039_object114207:head# 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4201ee43:::benchmark_data_smithi194_103039_object147201:head# 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42027ef9:::benchmark_data_smithi194_103039_object88369:head# 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420390df:::benchmark_data_smithi194_103039_object106066:head# 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4203bd17:::benchmark_data_smithi194_103039_object73732:head# 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4203c8bd:::benchmark_data_smithi194_103039_object135483:head# 2024-04-29T14:21:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4204423d:::benchmark_data_smithi194_103039_object102067:head# 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42052501:::benchmark_data_smithi194_103039_object153132:head# 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420532c4:::benchmark_data_smithi194_103039_object81583:head# 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4205a61d:::benchmark_data_smithi194_103039_object55934:head# 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42066561:::benchmark_data_smithi194_103039_object67417:head# 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42086da7:::benchmark_data_smithi194_103039_object127074:head# 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4208756a:::benchmark_data_smithi194_103039_object97978:head# 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42096c92:::benchmark_data_smithi194_103039_object142784:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4209741b:::benchmark_data_smithi194_103039_object113910:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4209bafd:::benchmark_data_smithi194_103039_object115701:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4209bb74:::benchmark_data_smithi194_103039_object102841:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420a2b39:::benchmark_data_smithi194_103039_object61600:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420af68c:::benchmark_data_smithi194_103039_object145129:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420b3c8a:::benchmark_data_smithi194_103039_object146781:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420c93d8:::benchmark_data_smithi194_103039_object117471:head# 2024-04-29T14:21:28.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420ca3eb:::benchmark_data_smithi194_103039_object74875:head# 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420cb275:::benchmark_data_smithi194_103039_object152804:head# 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420dc29d:::benchmark_data_smithi194_103039_object51605:head# 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420e79d3:::benchmark_data_smithi194_103039_object83706:head# 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420eb593:::benchmark_data_smithi194_103039_object83220:head# 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420f1bdc:::benchmark_data_smithi194_103039_object91670:head# 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420f7814:::benchmark_data_smithi194_103039_object53284:head# 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420f89b4:::benchmark_data_smithi194_103039_object65232:head# 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:420feb0e:::benchmark_data_smithi194_103039_object152863:head# 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42105ca4:::benchmark_data_smithi194_103039_object102590:head# 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4210c4cb:::benchmark_data_smithi194_103039_object47594:head# 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42112568:::benchmark_data_smithi194_103039_object152502:head# 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42115d1c:::benchmark_data_smithi194_103039_object87551:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42135626:::benchmark_data_smithi194_103039_object74019:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4213aea3:::benchmark_data_smithi194_103039_object80502:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42144032:::benchmark_data_smithi194_103039_object63555:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421441fb:::benchmark_data_smithi194_103039_object92925:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4214f66b:::benchmark_data_smithi194_103039_object152725:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42155353:::benchmark_data_smithi194_103039_object142980:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4215c042:::benchmark_data_smithi194_103039_object149748:head# 2024-04-29T14:21:28.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4215fe52:::benchmark_data_smithi194_103039_object144793:head# 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4216b3df:::benchmark_data_smithi194_103039_object127951:head# 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4217a52c:::benchmark_data_smithi194_103039_object66172:head# 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4218edc0:::benchmark_data_smithi194_103039_object110902:head# 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4218f9fd:::benchmark_data_smithi194_103039_object128258:head# 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4218fde2:::benchmark_data_smithi194_103039_object91993:head# 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42198675:::benchmark_data_smithi194_103039_object66397:head# 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4219e163:::benchmark_data_smithi194_103039_object55268:head# 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4219e797:::benchmark_data_smithi194_103039_object128272:head# 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421a28bf:::benchmark_data_smithi194_103039_object146107:head# 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421abdd4:::benchmark_data_smithi194_103039_object58007:head# 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421bb3ef:::benchmark_data_smithi194_103039_object87856:head# 2024-04-29T14:21:28.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421bd3b0:::benchmark_data_smithi194_103039_object107576:head# 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421ce2fa:::benchmark_data_smithi194_103039_object53606:head# 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421d0890:::benchmark_data_smithi194_103039_object131144:head# 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421d70f8:::benchmark_data_smithi194_103039_object129803:head# 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421d9885:::benchmark_data_smithi194_103039_object152533:head# 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421e26a9:::benchmark_data_smithi194_103039_object49905:head# 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421e40d5:::benchmark_data_smithi194_103039_object135894:head# 2024-04-29T14:21:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421f3827:::benchmark_data_smithi194_103039_object67700:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421fc446:::benchmark_data_smithi194_103039_object137577:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:421fe747:::benchmark_data_smithi194_103039_object148295:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4221db73:::benchmark_data_smithi194_103039_object60045:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4221f448:::benchmark_data_smithi194_103039_object55392:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42222476:::benchmark_data_smithi194_103039_object58036:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4223ccf4:::benchmark_data_smithi194_103039_object93948:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4224ef4c:::benchmark_data_smithi194_103039_object121112:head# 2024-04-29T14:21:28.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4225fa77:::benchmark_data_smithi194_103039_object47005:head# 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4226e19a:::benchmark_data_smithi194_103039_object75936:head# 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42278e58:::benchmark_data_smithi194_103039_object51544:head# 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:422b61ee:::benchmark_data_smithi194_103039_object57900:head# 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:422b6f73:::benchmark_data_smithi194_103039_object98250:head# 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:422cab14:::benchmark_data_smithi194_103039_object139031:head# 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:422d6789:::benchmark_data_smithi194_103039_object149742:head# 2024-04-29T14:21:28.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:422d9844:::benchmark_data_smithi194_103039_object135904:head# 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:422e27d6:::benchmark_data_smithi194_103039_object77756:head# 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:422efd82:::benchmark_data_smithi194_103039_object78591:head# 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423089d8:::benchmark_data_smithi194_103039_object153694:head# 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4230bea6:::benchmark_data_smithi194_103039_object42013:head# 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42318649:::benchmark_data_smithi194_103039_object125470:head# 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423254fb:::benchmark_data_smithi194_103039_object46259:head# 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423258d9:::benchmark_data_smithi194_103039_object95051:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4232fb4d:::benchmark_data_smithi194_103039_object148605:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423343e7:::benchmark_data_smithi194_103039_object112432:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42335d85:::benchmark_data_smithi194_103039_object50888:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42364669:::benchmark_data_smithi194_103039_object102345:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4236f4cd:::benchmark_data_smithi194_103039_object140829:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4236fe0d:::benchmark_data_smithi194_103039_object151423:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4237b317:::benchmark_data_smithi194_103039_object148936:head# 2024-04-29T14:21:28.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4238f349:::benchmark_data_smithi194_103039_object74289:head# 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42390c30:::benchmark_data_smithi194_103039_object64724:head# 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4239eeee:::benchmark_data_smithi194_103039_object95239:head# 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423ace11:::benchmark_data_smithi194_103039_object66058:head# 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423bb50f:::benchmark_data_smithi194_103039_object91408:head# 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423d89b5:::benchmark_data_smithi194_103039_object97274:head# 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423d8f2b:::benchmark_data_smithi194_103039_object110682:head# 2024-04-29T14:21:28.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423dfb88:::benchmark_data_smithi194_103039_object51865:head# 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423e1439:::benchmark_data_smithi194_103039_object54439:head# 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423e6191:::benchmark_data_smithi194_103039_object111317:head# 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423e857f:::benchmark_data_smithi194_103039_object126893:head# 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423fd784:::benchmark_data_smithi194_103039_object141965:head# 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:423ffb96:::benchmark_data_smithi194_103039_object123244:head# 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42423218:::benchmark_data_smithi194_103039_object52539:head# 2024-04-29T14:21:28.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4242a895:::benchmark_data_smithi194_103039_object54477:head# 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4243ab47:::benchmark_data_smithi194_103039_object97641:head# 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42442c07:::benchmark_data_smithi194_103039_object98784:head# 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4244c6de:::benchmark_data_smithi194_103039_object46787:head# 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42456d53:::benchmark_data_smithi194_103039_object127012:head# 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42459050:::benchmark_data_smithi194_103039_object73959:head# 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42460abf:::benchmark_data_smithi194_103039_object44181:head# 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42467296:::benchmark_data_smithi194_103039_object145887:head# 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4246d23d:::benchmark_data_smithi194_103039_object47498:head# 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42487c60:::benchmark_data_smithi194_103039_object107208:head# 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4248ae43:::benchmark_data_smithi194_103039_object95709:head# 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4248f842:::benchmark_data_smithi194_103039_object67332:head# 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4249171c:::benchmark_data_smithi194_103039_object141527:head# 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4249945d:::benchmark_data_smithi194_103039_object55285:head# 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424a1595:::benchmark_data_smithi194_103039_object126415:head# 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424aa72d:::benchmark_data_smithi194_103039_object156340:head# 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424ac71c:::benchmark_data_smithi194_103039_object43749:head# 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424add87:::benchmark_data_smithi194_103039_object135287:head# 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424b5f9d:::benchmark_data_smithi194_103039_object89252:head# 2024-04-29T14:21:28.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424d14cf:::benchmark_data_smithi194_103039_object65452:head# 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424d470a:::benchmark_data_smithi194_103039_object68029:head# 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424e9c5a:::benchmark_data_smithi194_103039_object114479:head# 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424ec288:::benchmark_data_smithi194_103039_object60891:head# 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424ee558:::benchmark_data_smithi194_103039_object125337:head# 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:424f5fa7:::benchmark_data_smithi194_103039_object106054:head# 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425076c4:::benchmark_data_smithi194_103039_object153101:head# 2024-04-29T14:21:28.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425079af:::benchmark_data_smithi194_103039_object56194:head# 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42508427:::benchmark_data_smithi194_103039_object122663:head# 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4251b1fa:::benchmark_data_smithi194_103039_object110106:head# 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4251de73:::benchmark_data_smithi194_103039_object58932:head# 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4252fd59:::benchmark_data_smithi194_103039_object69589:head# 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42546556:::benchmark_data_smithi194_103039_object59854:head# 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425511e4:::benchmark_data_smithi194_103039_object50878:head# 2024-04-29T14:21:28.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4255a876:::benchmark_data_smithi194_103039_object96250:head# 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42561c79:::benchmark_data_smithi194_103039_object129695:head# 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42564da7:::benchmark_data_smithi194_103039_object45963:head# 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425689e2:::benchmark_data_smithi194_103039_object103472:head# 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42568a65:::benchmark_data_smithi194_103039_object130377:head# 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42569638:::benchmark_data_smithi194_103039_object101839:head# 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4258083d:::benchmark_data_smithi194_103039_object128805:head# 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425892f0:::benchmark_data_smithi194_103039_object114997:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4259130d:::benchmark_data_smithi194_103039_object135997:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425a15f1:::benchmark_data_smithi194_103039_object130851:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425ab01f:::benchmark_data_smithi194_103039_object87117:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425ad0cc:::benchmark_data_smithi194_103039_object144489:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425bff90:::benchmark_data_smithi194_103039_object75338:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425c3c10:::benchmark_data_smithi194_103039_object127552:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425c8eb6:::benchmark_data_smithi194_103039_object156595:head# 2024-04-29T14:21:28.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425c92af:::benchmark_data_smithi194_103039_object138622:head# 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425cf804:::benchmark_data_smithi194_103039_object156111:head# 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425df45a:::benchmark_data_smithi194_103039_object92173:head# 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425e1151:::benchmark_data_smithi194_103039_object112930:head# 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425e2a3c:::benchmark_data_smithi194_103039_object55864:head# 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425e5a3b:::benchmark_data_smithi194_103039_object65019:head# 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425e8124:::benchmark_data_smithi194_103039_object107964:head# 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425eac39:::benchmark_data_smithi194_103039_object125904:head# 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425fa656:::benchmark_data_smithi194_103039_object122915:head# 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:425fa9a7:::benchmark_data_smithi194_103039_object104728:head# 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42611dac:::benchmark_data_smithi194_103039_object135019:head# 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426314ee:::benchmark_data_smithi194_103039_object86315:head# 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4263b3e0:::benchmark_data_smithi194_103039_object129998:head# 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42643266:::benchmark_data_smithi194_103039_object98096:head# 2024-04-29T14:21:28.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42644ae8:::benchmark_data_smithi194_103039_object114145:head# 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42647838:::benchmark_data_smithi194_103039_object114338:head# 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4266d6e6:::benchmark_data_smithi194_103039_object142651:head# 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42671881:::benchmark_data_smithi194_103039_object89899:head# 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4268ea64:::benchmark_data_smithi194_103039_object135553:head# 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42692966:::benchmark_data_smithi194_103039_object78291:head# 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426992c6:::benchmark_data_smithi194_103039_object156057:head# 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426a0a89:::benchmark_data_smithi194_103039_object107350:head# 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426aba15:::benchmark_data_smithi194_103039_object97031:head# 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426bd561:::benchmark_data_smithi194_103039_object97830:head# 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426bf04a:::benchmark_data_smithi194_103039_object141667:head# 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426c5803:::benchmark_data_smithi194_103039_object151112:head# 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426c5978:::benchmark_data_smithi194_103039_object93410:head# 2024-04-29T14:21:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426ca503:::benchmark_data_smithi194_103039_object108344:head# 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426ddf3e:::benchmark_data_smithi194_103039_object96124:head# 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426e106f:::benchmark_data_smithi194_103039_object96630:head# 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:426ebe14:::benchmark_data_smithi194_103039_object96280:head# 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4270b286:::benchmark_data_smithi194_103039_object113814:head# 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427129d4:::benchmark_data_smithi194_103039_object136606:head# 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42715b91:::benchmark_data_smithi194_103039_object52091:head# 2024-04-29T14:21:28.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4272d4e2:::benchmark_data_smithi194_103039_object110458:head# 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42732d56:::benchmark_data_smithi194_103039_object149287:head# 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4273975e:::benchmark_data_smithi194_103039_object98764:head# 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4273d9b1:::benchmark_data_smithi194_103039_object68208:head# 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4274e6b6:::benchmark_data_smithi194_103039_object59244:head# 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42750860:::benchmark_data_smithi194_103039_object151744:head# 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427558ca:::benchmark_data_smithi194_103039_object70649:head# 2024-04-29T14:21:28.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4275ff0e:::benchmark_data_smithi194_103039_object56363:head# 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4276984f:::benchmark_data_smithi194_103039_object94523:head# 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4276a909:::benchmark_data_smithi194_103039_object147396:head# 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427786dd:::benchmark_data_smithi194_103039_object72622:head# 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4277a75d:::benchmark_data_smithi194_103039_object118455:head# 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42780b47:::benchmark_data_smithi194_103039_object97741:head# 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42787fcc:::benchmark_data_smithi194_103039_object76139:head# 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42794cc6:::benchmark_data_smithi194_103039_object70475:head# 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427baed2:::benchmark_data_smithi194_103039_object96744:head# 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427bbe7c:::benchmark_data_smithi194_103039_object57290:head# 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427d3721:::benchmark_data_smithi194_103039_object124478:head# 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427deb20:::benchmark_data_smithi194_103039_object56277:head# 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427e24a9:::benchmark_data_smithi194_103039_object45098:head# 2024-04-29T14:21:28.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:427e5e7a:::benchmark_data_smithi194_103039_object48586:head# 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42806f76:::benchmark_data_smithi194_103039_object155143:head# 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42808fef:::benchmark_data_smithi194_103039_object55025:head# 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4280a66f:::benchmark_data_smithi194_103039_object153440:head# 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42814b3d:::benchmark_data_smithi194_103039_object77809:head# 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428162f1:::benchmark_data_smithi194_103039_object52075:head# 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4281afc5:::benchmark_data_smithi194_103039_object88926:head# 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42826a31:::benchmark_data_smithi194_103039_object130319:head# 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4282bb15:::benchmark_data_smithi194_103039_object45612:head# 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42838362:::benchmark_data_smithi194_103039_object127628:head# 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4283db61:::benchmark_data_smithi194_103039_object52493:head# 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42840da4:::benchmark_data_smithi194_103039_object91010:head# 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4284843f:::benchmark_data_smithi194_103039_object114386:head# 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42857cd5:::benchmark_data_smithi194_103039_object120393:head# 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4286020d:::benchmark_data_smithi194_103039_object95590:head# 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4286ad60:::benchmark_data_smithi194_103039_object133466:head# 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4286d5a1:::benchmark_data_smithi194_103039_object49883:head# 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42871b6a:::benchmark_data_smithi194_103039_object70215:head# 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42874f5d:::benchmark_data_smithi194_103039_object42618:head# 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42877450:::benchmark_data_smithi194_103039_object58426:head# 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428865fb:::benchmark_data_smithi194_103039_object93383:head# 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4288e5ac:::benchmark_data_smithi194_103039_object83666:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4288ee24:::benchmark_data_smithi194_103039_object119302:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428948c8:::benchmark_data_smithi194_103039_object117754:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4289963c:::benchmark_data_smithi194_103039_object119995:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4289a2f2:::benchmark_data_smithi194_103039_object94403:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428a3467:::benchmark_data_smithi194_103039_object74497:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428a4b49:::benchmark_data_smithi194_103039_object104447:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428b23db:::benchmark_data_smithi194_103039_object117616:head# 2024-04-29T14:21:28.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428c1685:::benchmark_data_smithi194_103039_object82268:head# 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428c51b0:::benchmark_data_smithi194_103039_object113739:head# 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428cb611:::benchmark_data_smithi194_103039_object89411:head# 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428db44b:::benchmark_data_smithi194_103039_object135251:head# 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428e335b:::benchmark_data_smithi194_103039_object73561:head# 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428e384c:::benchmark_data_smithi194_103039_object107186:head# 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428ed856:::benchmark_data_smithi194_103039_object52509:head# 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428ee98d:::benchmark_data_smithi194_103039_object149259:head# 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:428fe743:::benchmark_data_smithi194_103039_object63059:head# 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42905540:::benchmark_data_smithi194_103039_object99716:head# 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42912090:::benchmark_data_smithi194_103039_object61932:head# 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4291d56d:::benchmark_data_smithi194_103039_object95158:head# 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42932a02:::benchmark_data_smithi194_103039_object143965:head# 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42945b60:::benchmark_data_smithi194_103039_object58903:head# 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429511d4:::benchmark_data_smithi194_103039_object73601:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42951c94:::benchmark_data_smithi194_103039_object41931:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4296fe21:::benchmark_data_smithi194_103039_object116295:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4297b5a1:::benchmark_data_smithi194_103039_object130462:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4298d177:::benchmark_data_smithi194_103039_object136116:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42996dd8:::benchmark_data_smithi194_103039_object64838:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4299b35b:::benchmark_data_smithi194_103039_object106100:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429abc7c:::benchmark_data_smithi194_103039_object51851:head# 2024-04-29T14:21:28.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429b7107:::benchmark_data_smithi194_103039_object101718:head# 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429bb9d5:::benchmark_data_smithi194_103039_object145338:head# 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429c1e3b:::benchmark_data_smithi194_103039_object88658:head# 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429c9625:::benchmark_data_smithi194_103039_object115936:head# 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429ce48e:::benchmark_data_smithi194_103039_object141498:head# 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429ce5a8:::benchmark_data_smithi194_103039_object81085:head# 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429d3623:::benchmark_data_smithi194_103039_object98659:head# 2024-04-29T14:21:28.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429d743b:::benchmark_data_smithi194_103039_object138502:head# 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429d7e5d:::benchmark_data_smithi194_103039_object90286:head# 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429dea3a:::benchmark_data_smithi194_103039_object103532:head# 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429df663:::benchmark_data_smithi194_103039_object135933:head# 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429e80ca:::benchmark_data_smithi194_103039_object90606:head# 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429e9a34:::benchmark_data_smithi194_103039_object50849:head# 2024-04-29T14:21:28.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429fc0eb:::benchmark_data_smithi194_103039_object60717:head# 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429fc9c6:::benchmark_data_smithi194_103039_object144150:head# 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:429fddad:::benchmark_data_smithi194_103039_object100184:head# 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a02a7d:::benchmark_data_smithi194_103039_object96304:head# 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a0bf7f:::benchmark_data_smithi194_103039_object144965:head# 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a0c45a:::benchmark_data_smithi194_103039_object106072:head# 2024-04-29T14:21:28.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a0cefb:::benchmark_data_smithi194_103039_object97257:head# 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a0d6de:::benchmark_data_smithi194_103039_object103318:head# 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a13c14:::benchmark_data_smithi194_103039_object104012:head# 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a14f25:::benchmark_data_smithi194_103039_object59496:head# 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a18974:::benchmark_data_smithi194_103039_object56105:head# 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a1f309:::benchmark_data_smithi194_103039_object63725:head# 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a1fdca:::benchmark_data_smithi194_103039_object148897:head# 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a24733:::benchmark_data_smithi194_103039_object81472:head# 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a26e7f:::benchmark_data_smithi194_103039_object139709:head# 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a3815d:::benchmark_data_smithi194_103039_object41698:head# 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a39f6f:::benchmark_data_smithi194_103039_object126490:head# 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a3d667:::benchmark_data_smithi194_103039_object152202:head# 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a3e069:::benchmark_data_smithi194_103039_object131823:head# 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a4c483:::benchmark_data_smithi194_103039_object49310:head# 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a4d263:::benchmark_data_smithi194_103039_object146638:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a5126a:::benchmark_data_smithi194_103039_object123065:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a624c3:::benchmark_data_smithi194_103039_object65244:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a6310e:::benchmark_data_smithi194_103039_object113064:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a63b6b:::benchmark_data_smithi194_103039_object78615:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a65c03:::benchmark_data_smithi194_103039_object42327:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a6a172:::benchmark_data_smithi194_103039_object123717:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a71409:::benchmark_data_smithi194_103039_object132967:head# 2024-04-29T14:21:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a95544:::benchmark_data_smithi194_103039_object154031:head# 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a9bed7:::benchmark_data_smithi194_103039_object47244:head# 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a9e5ff:::benchmark_data_smithi194_103039_object129883:head# 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42a9ef4d:::benchmark_data_smithi194_103039_object132293:head# 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42aac679:::benchmark_data_smithi194_103039_object45931:head# 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42abaa8d:::benchmark_data_smithi194_103039_object127026:head# 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ac8423:::benchmark_data_smithi194_103039_object41700:head# 2024-04-29T14:21:28.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ad0da6:::benchmark_data_smithi194_103039_object54897:head# 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42af0e6d:::benchmark_data_smithi194_103039_object102813:head# 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42af7797:::benchmark_data_smithi194_103039_object112458:head# 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b0361c:::benchmark_data_smithi194_103039_object88251:head# 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b042e8:::benchmark_data_smithi194_103039_object47306:head# 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b05fc6:::benchmark_data_smithi194_103039_object81532:head# 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b0b4e6:::benchmark_data_smithi194_103039_object104632:head# 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b10977:::benchmark_data_smithi194_103039_object98899:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b14771:::benchmark_data_smithi194_103039_object149739:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b3128d:::benchmark_data_smithi194_103039_object120417:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b3368f:::benchmark_data_smithi194_103039_object97318:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b42220:::benchmark_data_smithi194_103039_object65678:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b4eb78:::benchmark_data_smithi194_103039_object79924:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b5e1aa:::benchmark_data_smithi194_103039_object99230:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b6f31b:::benchmark_data_smithi194_103039_object98857:head# 2024-04-29T14:21:28.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b70338:::benchmark_data_smithi194_103039_object128005:head# 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b744eb:::benchmark_data_smithi194_103039_object142902:head# 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b8004c:::benchmark_data_smithi194_103039_object65956:head# 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b8c3f2:::benchmark_data_smithi194_103039_object70487:head# 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b90cd5:::benchmark_data_smithi194_103039_object134823:head# 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b957f8:::benchmark_data_smithi194_103039_object155171:head# 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b970a1:::benchmark_data_smithi194_103039_object90822:head# 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b97f53:::benchmark_data_smithi194_103039_object112438:head# 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42b9b142:::benchmark_data_smithi194_103039_object76183:head# 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ba8f75:::benchmark_data_smithi194_103039_object97408:head# 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42bac931:::benchmark_data_smithi194_103039_object42808:head# 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42bb4208:::benchmark_data_smithi194_103039_object69507:head# 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42bbc620:::benchmark_data_smithi194_103039_object44594:head# 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42bbf794:::benchmark_data_smithi194_103039_object76380:head# 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42bd91a1:::benchmark_data_smithi194_103039_object78427:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42be8811:::benchmark_data_smithi194_103039_object72194:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42bf0a6c:::benchmark_data_smithi194_103039_object52211:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42bfa7c9:::benchmark_data_smithi194_103039_object111350:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c04f64:::benchmark_data_smithi194_103039_object96757:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c13a00:::benchmark_data_smithi194_103039_object102225:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c18b2e:::benchmark_data_smithi194_103039_object83822:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c1954e:::benchmark_data_smithi194_103039_object64883:head# 2024-04-29T14:21:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c1e7cd:::benchmark_data_smithi194_103039_object87725:head# 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c1f496:::benchmark_data_smithi194_103039_object122977:head# 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c2ac36:::benchmark_data_smithi194_103039_object79224:head# 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c3d909:::benchmark_data_smithi194_103039_object134407:head# 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c5523e:::benchmark_data_smithi194_103039_object74050:head# 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c560fa:::benchmark_data_smithi194_103039_object109861:head# 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c5f683:::benchmark_data_smithi194_103039_object45615:head# 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c68cef:::benchmark_data_smithi194_103039_object68693:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c73339:::benchmark_data_smithi194_103039_object70849:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c75235:::benchmark_data_smithi194_103039_object130958:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c81c48:::benchmark_data_smithi194_103039_object124885:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c8d59c:::benchmark_data_smithi194_103039_object102916:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c94b2b:::benchmark_data_smithi194_103039_object154982:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c956d0:::benchmark_data_smithi194_103039_object108278:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c9b4fc:::benchmark_data_smithi194_103039_object111882:head# 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42c9bede:::benchmark_data_smithi194_103039_object139610:head# 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ca02c7:::benchmark_data_smithi194_103039_object113725:head# 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cac670:::benchmark_data_smithi194_103039_object121918:head# 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cb3cd7:::benchmark_data_smithi194_103039_object123408:head# 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cb756e:::benchmark_data_smithi194_103039_object149948:head# 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cbdc2e:::benchmark_data_smithi194_103039_object127578:head# 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cbfbdc:::benchmark_data_smithi194_103039_object48566:head# 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cc1578:::benchmark_data_smithi194_103039_object150028:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cc8e53:::benchmark_data_smithi194_103039_object73410:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ccab46:::benchmark_data_smithi194_103039_object49051:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ccf596:::benchmark_data_smithi194_103039_object92195:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cd95d0:::benchmark_data_smithi194_103039_object118285:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42cdd17e:::benchmark_data_smithi194_103039_object110609:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ce740a:::benchmark_data_smithi194_103039_object122275:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d006e9:::benchmark_data_smithi194_103039_object43330:head# 2024-04-29T14:21:28.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d047ac:::benchmark_data_smithi194_103039_object140284:head# 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d0c8ea:::benchmark_data_smithi194_103039_object147648:head# 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d13b87:::benchmark_data_smithi194_103039_object116767:head# 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d148b6:::benchmark_data_smithi194_103039_object127483:head# 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d16f99:::benchmark_data_smithi194_103039_object79508:head# 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d178b3:::benchmark_data_smithi194_103039_object153891:head# 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d26e3e:::benchmark_data_smithi194_103039_object106820:head# 2024-04-29T14:21:28.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d39d16:::benchmark_data_smithi194_103039_object138531:head# 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d44e24:::benchmark_data_smithi194_103039_object111073:head# 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d74b97:::benchmark_data_smithi194_103039_object151041:head# 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d79884:::benchmark_data_smithi194_103039_object102936:head# 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d87240:::benchmark_data_smithi194_103039_object53281:head# 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d8bb26:::benchmark_data_smithi194_103039_object145433:head# 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d8c0fc:::benchmark_data_smithi194_103039_object113688:head# 2024-04-29T14:21:28.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42d9710e:::benchmark_data_smithi194_103039_object128165:head# 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42da3f12:::benchmark_data_smithi194_103039_object111530:head# 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42dbb006:::benchmark_data_smithi194_103039_object109099:head# 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42dbb427:::benchmark_data_smithi194_103039_object72949:head# 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42dd2320:::benchmark_data_smithi194_103039_object57475:head# 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42dd2976:::benchmark_data_smithi194_103039_object76019:head# 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42dd4ef1:::benchmark_data_smithi194_103039_object69234:head# 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ddad90:::benchmark_data_smithi194_103039_object153248:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42dde2cc:::benchmark_data_smithi194_103039_object107736:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ddfd2d:::benchmark_data_smithi194_103039_object66356:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42de6290:::benchmark_data_smithi194_103039_object148421:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42de8877:::benchmark_data_smithi194_103039_object65700:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42dedbf8:::benchmark_data_smithi194_103039_object90451:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e038ae:::benchmark_data_smithi194_103039_object109742:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e04fa9:::benchmark_data_smithi194_103039_object137862:head# 2024-04-29T14:21:28.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e090bc:::benchmark_data_smithi194_103039_object90004:head# 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e0ac64:::benchmark_data_smithi194_103039_object143695:head# 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e21d01:::benchmark_data_smithi194_103039_object125172:head# 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e331ef:::benchmark_data_smithi194_103039_object138779:head# 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e43f9b:::benchmark_data_smithi194_103039_object94693:head# 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e4f3a4:::benchmark_data_smithi194_103039_object84246:head# 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e5a5de:::benchmark_data_smithi194_103039_object129454:head# 2024-04-29T14:21:28.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e6a58f:::benchmark_data_smithi194_103039_object112756:head# 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e6ba9c:::benchmark_data_smithi194_103039_object101056:head# 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e70c9a:::benchmark_data_smithi194_103039_object99062:head# 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e7fc17:::benchmark_data_smithi194_103039_object63427:head# 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e87b40:::benchmark_data_smithi194_103039_object50068:head# 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e8966a:::benchmark_data_smithi194_103039_object112205:head# 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e8b166:::benchmark_data_smithi194_103039_object49759:head# 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e9705f:::benchmark_data_smithi194_103039_object156087:head# 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42e9df4d:::benchmark_data_smithi194_103039_object139200:head# 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ea9796:::benchmark_data_smithi194_103039_object143548:head# 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42eae369:::benchmark_data_smithi194_103039_object123095:head# 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42eae684:::benchmark_data_smithi194_103039_object73426:head# 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42eb21b8:::benchmark_data_smithi194_103039_object97812:head# 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42eb523c:::benchmark_data_smithi194_103039_object50378:head# 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ebb973:::benchmark_data_smithi194_103039_object94124:head# 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ebcc5c:::benchmark_data_smithi194_103039_object109608:head# 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ebcfe2:::benchmark_data_smithi194_103039_object96129:head# 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ebf526:::benchmark_data_smithi194_103039_object103257:head# 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ec15fc:::benchmark_data_smithi194_103039_object118094:head# 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ec3d65:::benchmark_data_smithi194_103039_object85000:head# 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ec9604:::benchmark_data_smithi194_103039_object98227:head# 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ed4ecd:::benchmark_data_smithi194_103039_object145025:head# 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ed54e6:::benchmark_data_smithi194_103039_object130936:head# 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ee983a:::benchmark_data_smithi194_103039_object117073:head# 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42eea65a:::benchmark_data_smithi194_103039_object53333:head# 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ef01e5:::benchmark_data_smithi194_103039_object78508:head# 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ef9b11:::benchmark_data_smithi194_103039_object82939:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f1891b:::benchmark_data_smithi194_103039_object103151:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f403d8:::benchmark_data_smithi194_103039_object111370:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f47249:::benchmark_data_smithi194_103039_object61309:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f6be2a:::benchmark_data_smithi194_103039_object42115:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f6ef29:::benchmark_data_smithi194_103039_object94636:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f7511c:::benchmark_data_smithi194_103039_object79660:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f757ca:::benchmark_data_smithi194_103039_object52424:head# 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f79bf0:::benchmark_data_smithi194_103039_object103182:head# 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f7c5b6:::benchmark_data_smithi194_103039_object64154:head# 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f7ea0f:::benchmark_data_smithi194_103039_object59009:head# 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42f98961:::benchmark_data_smithi194_103039_object92969:head# 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fae5d8:::benchmark_data_smithi194_103039_object52869:head# 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fb0741:::benchmark_data_smithi194_103039_object74818:head# 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fb537b:::benchmark_data_smithi194_103039_object137765:head# 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fbe2e3:::benchmark_data_smithi194_103039_object84534:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fc9705:::benchmark_data_smithi194_103039_object92929:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fd889e:::benchmark_data_smithi194_103039_object83598:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fddb42:::benchmark_data_smithi194_103039_object156182:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fe1e87:::benchmark_data_smithi194_103039_object103983:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fe27bb:::benchmark_data_smithi194_103039_object141938:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fe47a5:::benchmark_data_smithi194_103039_object85503:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fe6f36:::benchmark_data_smithi194_103039_object136409:head# 2024-04-29T14:21:28.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42fec941:::benchmark_data_smithi194_103039_object70612:head# 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ff004b:::benchmark_data_smithi194_103039_object113316:head# 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ff00fd:::benchmark_data_smithi194_103039_object68375:head# 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:42ffa380:::benchmark_data_smithi194_103039_object68953:head# 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430041e6:::benchmark_data_smithi194_103039_object84215:head# 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430062a1:::benchmark_data_smithi194_103039_object138061:head# 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4300645d:::benchmark_data_smithi194_103039_object111881:head# 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43008624:::benchmark_data_smithi194_103039_object92151:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43015935:::benchmark_data_smithi194_103039_object72060:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4301630a:::benchmark_data_smithi194_103039_object77203:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4304247c:::benchmark_data_smithi194_103039_object76365:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430469b2:::benchmark_data_smithi194_103039_object113262:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4304af89:::benchmark_data_smithi194_103039_object64618:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43050510:::benchmark_data_smithi194_103039_object140084:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43058663:::benchmark_data_smithi194_103039_object128295:head# 2024-04-29T14:21:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4306ba6a:::benchmark_data_smithi194_103039_object134659:head# 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4306f6a7:::benchmark_data_smithi194_103039_object134795:head# 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4307d2e0:::benchmark_data_smithi194_103039_object43108:head# 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43085c9b:::benchmark_data_smithi194_103039_object140267:head# 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43088627:::benchmark_data_smithi194_103039_object135060:head# 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4308867a:::benchmark_data_smithi194_103039_object129847:head# 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4308e29e:::benchmark_data_smithi194_103039_object119824:head# 2024-04-29T14:21:28.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4309d90f:::benchmark_data_smithi194_103039_object114670:head# 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430a142c:::benchmark_data_smithi194_103039_object77947:head# 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430a9a55:::benchmark_data_smithi194_103039_object89236:head# 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430c022f:::benchmark_data_smithi194_103039_object66703:head# 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430cebbc:::benchmark_data_smithi194_103039_object90487:head# 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430d1f62:::benchmark_data_smithi194_103039_object125395:head# 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430d3de0:::benchmark_data_smithi194_103039_object89098:head# 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430d7886:::benchmark_data_smithi194_103039_object104755:head# 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:430fb971:::benchmark_data_smithi194_103039_object57057:head# 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4310078a:::benchmark_data_smithi194_103039_object91945:head# 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431166b3:::benchmark_data_smithi194_103039_object95099:head# 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4311a645:::benchmark_data_smithi194_103039_object103456:head# 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43131c81:::benchmark_data_smithi194_103039_object129756:head# 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431479fb:::benchmark_data_smithi194_103039_object127528:head# 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43147d29:::benchmark_data_smithi194_103039_object102561:head# 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4314c849:::benchmark_data_smithi194_103039_object48019:head# 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4314d90a:::benchmark_data_smithi194_103039_object118191:head# 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43151530:::benchmark_data_smithi194_103039_object142811:head# 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43152a69:::benchmark_data_smithi194_103039_object129192:head# 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4315878c:::benchmark_data_smithi194_103039_object99445:head# 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4316700b:::benchmark_data_smithi194_103039_object101707:head# 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4316b590:::benchmark_data_smithi194_103039_object41966:head# 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4316c888:::benchmark_data_smithi194_103039_object144448:head# 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43173d99:::benchmark_data_smithi194_103039_object99821:head# 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4317d17d:::benchmark_data_smithi194_103039_object155200:head# 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43180cba:::benchmark_data_smithi194_103039_object117377:head# 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431827d9:::benchmark_data_smithi194_103039_object119520:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4318330b:::benchmark_data_smithi194_103039_object106422:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43185fcf:::benchmark_data_smithi194_103039_object61761:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4318eee1:::benchmark_data_smithi194_103039_object59555:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431936c4:::benchmark_data_smithi194_103039_object49491:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43197529:::benchmark_data_smithi194_103039_object143575:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43199a78:::benchmark_data_smithi194_103039_object147971:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4319d611:::benchmark_data_smithi194_103039_object58751:head# 2024-04-29T14:21:28.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431a9240:::benchmark_data_smithi194_103039_object69738:head# 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431c2abf:::benchmark_data_smithi194_103039_object146333:head# 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431c91b0:::benchmark_data_smithi194_103039_object64767:head# 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431d0c9f:::benchmark_data_smithi194_103039_object105517:head# 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431da2b4:::benchmark_data_smithi194_103039_object76486:head# 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431db35d:::benchmark_data_smithi194_103039_object150935:head# 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431df023:::benchmark_data_smithi194_103039_object91730:head# 2024-04-29T14:21:28.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431df905:::benchmark_data_smithi194_103039_object146428:head# 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431dff2e:::benchmark_data_smithi194_103039_object44622:head# 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431f3b02:::benchmark_data_smithi194_103039_object92005:head# 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431f55e6:::benchmark_data_smithi194_103039_object119817:head# 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:431ffaf5:::benchmark_data_smithi194_103039_object152562:head# 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43207da2:::benchmark_data_smithi194_103039_object87988:head# 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4320822e:::benchmark_data_smithi194_103039_object139461:head# 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4320c78d:::benchmark_data_smithi194_103039_object98380:head# 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4320c85c:::benchmark_data_smithi194_103039_object45438:head# 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4321754c:::benchmark_data_smithi194_103039_object147488:head# 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432398c3:::benchmark_data_smithi194_103039_object122565:head# 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43244230:::benchmark_data_smithi194_103039_object126024:head# 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4325ec56:::benchmark_data_smithi194_103039_object83670:head# 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4325ed47:::benchmark_data_smithi194_103039_object137671:head# 2024-04-29T14:21:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4326a00f:::benchmark_data_smithi194_103039_object71747:head# 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4328188d:::benchmark_data_smithi194_103039_object43793:head# 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4328712f:::benchmark_data_smithi194_103039_object43638:head# 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432ad43e:::benchmark_data_smithi194_103039_object100897:head# 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432b7c28:::benchmark_data_smithi194_103039_object132529:head# 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432beaa0:::benchmark_data_smithi194_103039_object80274:head# 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432c4355:::benchmark_data_smithi194_103039_object123453:head# 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432d921d:::benchmark_data_smithi194_103039_object117712:head# 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432eb49e:::benchmark_data_smithi194_103039_object99222:head# 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432fa4a6:::benchmark_data_smithi194_103039_object41738:head# 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432fe1e9:::benchmark_data_smithi194_103039_object129745:head# 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:432fe6d3:::benchmark_data_smithi194_103039_object149888:head# 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43302432:::benchmark_data_smithi194_103039_object83150:head# 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4330b668:::benchmark_data_smithi194_103039_object122063:head# 2024-04-29T14:21:28.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4330c77d:::benchmark_data_smithi194_103039_object101149:head# 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43319b60:::benchmark_data_smithi194_103039_object128749:head# 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43320377:::benchmark_data_smithi194_103039_object57145:head# 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43323b35:::benchmark_data_smithi194_103039_object149095:head# 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43328ae8:::benchmark_data_smithi194_103039_object63269:head# 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43335d1e:::benchmark_data_smithi194_103039_object123263:head# 2024-04-29T14:21:28.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43337687:::benchmark_data_smithi194_103039_object139126:head# 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43337b2d:::benchmark_data_smithi194_103039_object134968:head# 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4333f7ce:::benchmark_data_smithi194_103039_object90285:head# 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4334eec8:::benchmark_data_smithi194_103039_object77874:head# 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433517b1:::benchmark_data_smithi194_103039_object120726:head# 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43355672:::benchmark_data_smithi194_103039_object125087:head# 2024-04-29T14:21:28.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4335a2c6:::benchmark_data_smithi194_103039_object144508:head# 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4335c802:::benchmark_data_smithi194_103039_object116114:head# 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4336249d:::benchmark_data_smithi194_103039_object125507:head# 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4336aabd:::benchmark_data_smithi194_103039_object79906:head# 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4337edad:::benchmark_data_smithi194_103039_object140813:head# 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4339631a:::benchmark_data_smithi194_103039_object75469:head# 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433a5408:::benchmark_data_smithi194_103039_object138442:head# 2024-04-29T14:21:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433ac08d:::benchmark_data_smithi194_103039_object100466:head# 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433b1b55:::benchmark_data_smithi194_103039_object156153:head# 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433b444c:::benchmark_data_smithi194_103039_object147168:head# 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433cebbe:::benchmark_data_smithi194_103039_object58563:head# 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433d52ee:::benchmark_data_smithi194_103039_object66484:head# 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433da039:::benchmark_data_smithi194_103039_object99947:head# 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:433e205e:::benchmark_data_smithi194_103039_object105986:head# 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43406171:::benchmark_data_smithi194_103039_object153878:head# 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4340a549:::benchmark_data_smithi194_103039_object65923:head# 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4340d78e:::benchmark_data_smithi194_103039_object91622:head# 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43414ce6:::benchmark_data_smithi194_103039_object143003:head# 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4342083b:::benchmark_data_smithi194_103039_object150634:head# 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43428fa9:::benchmark_data_smithi194_103039_object140092:head# 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43431098:::benchmark_data_smithi194_103039_object100621:head# 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43435a6a:::benchmark_data_smithi194_103039_object49085:head# 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4344a165:::benchmark_data_smithi194_103039_object94203:head# 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4344d700:::benchmark_data_smithi194_103039_object137516:head# 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4345122d:::benchmark_data_smithi194_103039_object47354:head# 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434561cc:::benchmark_data_smithi194_103039_object50677:head# 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4345ac7c:::benchmark_data_smithi194_103039_object135600:head# 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434691f3:::benchmark_data_smithi194_103039_object109819:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43474611:::benchmark_data_smithi194_103039_object88612:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434764ad:::benchmark_data_smithi194_103039_object145621:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43483b6d:::benchmark_data_smithi194_103039_object81804:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43488774:::benchmark_data_smithi194_103039_object51676:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4348891d:::benchmark_data_smithi194_103039_object107684:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4348ad87:::benchmark_data_smithi194_103039_object104771:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4348f975:::benchmark_data_smithi194_103039_object125295:head# 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434937d3:::benchmark_data_smithi194_103039_object106048:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4349e90c:::benchmark_data_smithi194_103039_object112285:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434a664e:::benchmark_data_smithi194_103039_object145785:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434a6c7f:::benchmark_data_smithi194_103039_object131767:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434a727c:::benchmark_data_smithi194_103039_object89655:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434b4734:::benchmark_data_smithi194_103039_object109693:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434b4c43:::benchmark_data_smithi194_103039_object115215:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434b5005:::benchmark_data_smithi194_103039_object60203:head# 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434b5a60:::benchmark_data_smithi194_103039_object149998:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434bfb0d:::benchmark_data_smithi194_103039_object154250:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434cd02d:::benchmark_data_smithi194_103039_object154009:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434ce47c:::benchmark_data_smithi194_103039_object92798:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434d7b4c:::benchmark_data_smithi194_103039_object73121:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:434f5452:::benchmark_data_smithi194_103039_object135262:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4350127b:::benchmark_data_smithi194_103039_object41684:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4351366b:::benchmark_data_smithi194_103039_object136380:head# 2024-04-29T14:21:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43525416:::benchmark_data_smithi194_103039_object118546:head# 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4352980b:::benchmark_data_smithi194_103039_object99120:head# 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4352a34a:::benchmark_data_smithi194_103039_object50717:head# 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435420d9:::benchmark_data_smithi194_103039_object99541:head# 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435447b1:::benchmark_data_smithi194_103039_object79388:head# 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43547212:::benchmark_data_smithi194_103039_object64624:head# 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43557e4d:::benchmark_data_smithi194_103039_object119264:head# 2024-04-29T14:21:28.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43559cc8:::benchmark_data_smithi194_103039_object138704:head# 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43568115:::benchmark_data_smithi194_103039_object43104:head# 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43570bf0:::benchmark_data_smithi194_103039_object83238:head# 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43570da8:::benchmark_data_smithi194_103039_object115352:head# 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43577845:::benchmark_data_smithi194_103039_object81205:head# 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43583886:::benchmark_data_smithi194_103039_object139887:head# 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43589e04:::benchmark_data_smithi194_103039_object155602:head# 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4358e113:::benchmark_data_smithi194_103039_object155236:head# 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43593a0f:::benchmark_data_smithi194_103039_object53136:head# 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4359cb35:::benchmark_data_smithi194_103039_object90806:head# 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435ac4b2:::benchmark_data_smithi194_103039_object113161:head# 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435b709a:::benchmark_data_smithi194_103039_object48218:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435b92c1:::benchmark_data_smithi194_103039_object147082:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435bd546:::benchmark_data_smithi194_103039_object87111:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435c72a2:::benchmark_data_smithi194_103039_object119038:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435d7280:::benchmark_data_smithi194_103039_object144322:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435dd825:::benchmark_data_smithi194_103039_object149074:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435f4e18:::benchmark_data_smithi194_103039_object154343:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:435fb4a3:::benchmark_data_smithi194_103039_object66096:head# 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43600133:::benchmark_data_smithi194_103039_object112951:head# 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436029c0:::benchmark_data_smithi194_103039_object76444:head# 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4361d823:::benchmark_data_smithi194_103039_object142899:head# 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4361f5ea:::benchmark_data_smithi194_103039_object45914:head# 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4363744a:::benchmark_data_smithi194_103039_object122961:head# 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43646fe1:::benchmark_data_smithi194_103039_object141588:head# 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4364796c:::benchmark_data_smithi194_103039_object123830:head# 2024-04-29T14:21:28.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4364a4c4:::benchmark_data_smithi194_103039_object75363:head# 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4364d49c:::benchmark_data_smithi194_103039_object113369:head# 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43651c67:::benchmark_data_smithi194_103039_object57124:head# 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4365dfa3:::benchmark_data_smithi194_103039_object127430:head# 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43664cbd:::benchmark_data_smithi194_103039_object79856:head# 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4366507e:::benchmark_data_smithi194_103039_object76783:head# 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436652c6:::benchmark_data_smithi194_103039_object50361:head# 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43687900:::benchmark_data_smithi194_103039_object50423:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4368b071:::benchmark_data_smithi194_103039_object154569:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4369f091:::benchmark_data_smithi194_103039_object145540:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436a0c04:::benchmark_data_smithi194_103039_object148581:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436ad08b:::benchmark_data_smithi194_103039_object54575:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436c2837:::benchmark_data_smithi194_103039_object84504:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436c6a83:::benchmark_data_smithi194_103039_object96064:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436ca071:::benchmark_data_smithi194_103039_object152542:head# 2024-04-29T14:21:28.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436cdff4:::benchmark_data_smithi194_103039_object80643:head# 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436cf726:::benchmark_data_smithi194_103039_object48677:head# 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436cff4d:::benchmark_data_smithi194_103039_object66051:head# 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436d268f:::benchmark_data_smithi194_103039_object93525:head# 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436dfcd3:::benchmark_data_smithi194_103039_object72114:head# 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436e4dfa:::benchmark_data_smithi194_103039_object102543:head# 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436e5d32:::benchmark_data_smithi194_103039_object141345:head# 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436eca04:::benchmark_data_smithi194_103039_object47665:head# 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436ee976:::benchmark_data_smithi194_103039_object47674:head# 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436f0942:::benchmark_data_smithi194_103039_object41788:head# 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:436f3ab5:::benchmark_data_smithi194_103039_object78059:head# 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4370b1d8:::benchmark_data_smithi194_103039_object94041:head# 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437115ab:::benchmark_data_smithi194_103039_object156092:head# 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4371f104:::benchmark_data_smithi194_103039_object64856:head# 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43724a62:::benchmark_data_smithi194_103039_object125083:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43725922:::benchmark_data_smithi194_103039_object109344:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43729058:::benchmark_data_smithi194_103039_object156188:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4373893b:::benchmark_data_smithi194_103039_object117876:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4373ee83:::benchmark_data_smithi194_103039_object119185:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4375b87d:::benchmark_data_smithi194_103039_object145688:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43771683:::benchmark_data_smithi194_103039_object91772:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437729b3:::benchmark_data_smithi194_103039_object137827:head# 2024-04-29T14:21:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43784334:::benchmark_data_smithi194_103039_object53154:head# 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4378bc57:::benchmark_data_smithi194_103039_object134932:head# 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43794c7a:::benchmark_data_smithi194_103039_object155547:head# 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4379c510:::benchmark_data_smithi194_103039_object126564:head# 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4379f2cb:::benchmark_data_smithi194_103039_object74143:head# 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437a19a6:::benchmark_data_smithi194_103039_object152954:head# 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437a1e31:::benchmark_data_smithi194_103039_object52043:head# 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437c3db9:::benchmark_data_smithi194_103039_object86995:head# 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437c9bbb:::benchmark_data_smithi194_103039_object102528:head# 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437caa98:::benchmark_data_smithi194_103039_object63362:head# 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437d726d:::benchmark_data_smithi194_103039_object82764:head# 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437d7870:::benchmark_data_smithi194_103039_object58895:head# 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437e7d7b:::benchmark_data_smithi194_103039_object51383:head# 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437f0cb3:::benchmark_data_smithi194_103039_object81252:head# 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:437f974f:::benchmark_data_smithi194_103039_object144880:head# 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43800d5a:::benchmark_data_smithi194_103039_object57159:head# 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4380bd5a:::benchmark_data_smithi194_103039_object101160:head# 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4381ccf6:::benchmark_data_smithi194_103039_object129131:head# 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43825b09:::benchmark_data_smithi194_103039_object98295:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43829640:::benchmark_data_smithi194_103039_object82977:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4382c5c9:::benchmark_data_smithi194_103039_object89268:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4383316f:::benchmark_data_smithi194_103039_object127353:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43841cbf:::benchmark_data_smithi194_103039_object114385:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43847ea0:::benchmark_data_smithi194_103039_object149799:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4384c682:::benchmark_data_smithi194_103039_object47320:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43858837:::benchmark_data_smithi194_103039_object92709:head# 2024-04-29T14:21:28.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4385b151:::benchmark_data_smithi194_103039_object142855:head# 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4386643e:::benchmark_data_smithi194_103039_object80660:head# 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43866daa:::benchmark_data_smithi194_103039_object45521:head# 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4386e37b:::benchmark_data_smithi194_103039_object55685:head# 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4386e43b:::benchmark_data_smithi194_103039_object124953:head# 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438799d4:::benchmark_data_smithi194_103039_object83329:head# 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4387c19c:::benchmark_data_smithi194_103039_object144934:head# 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4387d455:::benchmark_data_smithi194_103039_object91477:head# 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4387e36f:::benchmark_data_smithi194_103039_object110297:head# 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4389805b:::benchmark_data_smithi194_103039_object115236:head# 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4389c343:::benchmark_data_smithi194_103039_object137723:head# 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438a5025:::benchmark_data_smithi194_103039_object59887:head# 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438ac729:::benchmark_data_smithi194_103039_object42012:head# 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438ae80d:::benchmark_data_smithi194_103039_object42318:head# 2024-04-29T14:21:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438b7a45:::benchmark_data_smithi194_103039_object99251:head# 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438b8da3:::benchmark_data_smithi194_103039_object59716:head# 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438be14c:::benchmark_data_smithi194_103039_object138979:head# 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438c29da:::benchmark_data_smithi194_103039_object89009:head# 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438c4cd2:::benchmark_data_smithi194_103039_object58057:head# 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438c901f:::benchmark_data_smithi194_103039_object111750:head# 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438d26d9:::benchmark_data_smithi194_103039_object48246:head# 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438ded59:::benchmark_data_smithi194_103039_object48508:head# 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438e16a7:::benchmark_data_smithi194_103039_object115814:head# 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438e9f66:::benchmark_data_smithi194_103039_object91261:head# 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438ee7db:::benchmark_data_smithi194_103039_object57129:head# 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438f142b:::benchmark_data_smithi194_103039_object151596:head# 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438f3d27:::benchmark_data_smithi194_103039_object47563:head# 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438f7a6c:::benchmark_data_smithi194_103039_object58702:head# 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438ffa83:::benchmark_data_smithi194_103039_object85962:head# 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:438ffeb2:::benchmark_data_smithi194_103039_object60197:head# 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43901a24:::benchmark_data_smithi194_103039_object100692:head# 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4390d5d7:::benchmark_data_smithi194_103039_object106643:head# 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:439173f9:::benchmark_data_smithi194_103039_object137317:head# 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43918318:::benchmark_data_smithi194_103039_object145580:head# 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:439213f1:::benchmark_data_smithi194_103039_object82213:head# 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4392410e:::benchmark_data_smithi194_103039_object121664:head# 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:439459a1:::benchmark_data_smithi194_103039_object149596:head# 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43948dc2:::benchmark_data_smithi194_103039_object97859:head# 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43949572:::benchmark_data_smithi194_103039_object59352:head# 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43949f69:::benchmark_data_smithi194_103039_object134719:head# 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4394b194:::benchmark_data_smithi194_103039_object99525:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4395aed9:::benchmark_data_smithi194_103039_object62342:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43960562:::benchmark_data_smithi194_103039_object65944:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43961d83:::benchmark_data_smithi194_103039_object97593:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4397a2a2:::benchmark_data_smithi194_103039_object114422:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43989e90:::benchmark_data_smithi194_103039_object116190:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4399297e:::benchmark_data_smithi194_103039_object137659:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43995ad0:::benchmark_data_smithi194_103039_object47586:head# 2024-04-29T14:21:28.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4399e4e3:::benchmark_data_smithi194_103039_object104182:head# 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:439a6fa9:::benchmark_data_smithi194_103039_object150541:head# 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:439b3de9:::benchmark_data_smithi194_103039_object78078:head# 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:439d016b:::benchmark_data_smithi194_103039_object122889:head# 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a0966e:::benchmark_data_smithi194_103039_object102650:head# 2024-04-29T14:21:28.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:28.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:28 smithi121 ceph-mon[44533]: pgmap v991: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:28.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:28.907 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:28.948 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:28.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:21:29.047 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:29.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:29.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:29.203 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:29.229 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:29.244 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:29.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:29.248 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:29.268 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:29.282 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:29.289 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:29.315 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:29.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/765842163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:29.340 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:29.347 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:29.355 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:29.376 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:29.389 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:29.396 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:29.416 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:29.436 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:29.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/765842163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:29.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a31199:::benchmark_data_smithi194_103039_object56108:head# 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a31ec1:::benchmark_data_smithi194_103039_object107277:head# 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a36b3a:::benchmark_data_smithi194_103039_object42734:head# 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a42dc2:::benchmark_data_smithi194_103039_object136485:head# 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a52d77:::benchmark_data_smithi194_103039_object91832:head# 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a55ca8:::benchmark_data_smithi194_103039_object68286:head# 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a6431a:::benchmark_data_smithi194_103039_object43848:head# 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a7b5e0:::benchmark_data_smithi194_103039_object77712:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a7d0aa:::benchmark_data_smithi194_103039_object60530:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a7d508:::benchmark_data_smithi194_103039_object116305:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a7fa34:::benchmark_data_smithi194_103039_object99992:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a80eea:::benchmark_data_smithi194_103039_object148634:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a898ba:::benchmark_data_smithi194_103039_object116774:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43a9c740:::benchmark_data_smithi194_103039_object95638:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43abe25d:::benchmark_data_smithi194_103039_object132438:head# 2024-04-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ac4767:::benchmark_data_smithi194_103039_object116484:head# 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43acaf46:::benchmark_data_smithi194_103039_object125585:head# 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43adf906:::benchmark_data_smithi194_103039_object87282:head# 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ae6c34:::benchmark_data_smithi194_103039_object141533:head# 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43aee911:::benchmark_data_smithi194_103039_object96094:head# 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43aeed5e:::benchmark_data_smithi194_103039_object152788:head# 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43af2867:::benchmark_data_smithi194_103039_object152475:head# 2024-04-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43af2d21:::benchmark_data_smithi194_103039_object107832:head# 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43af4fc7:::benchmark_data_smithi194_103039_object124161:head# 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43afc11c:::benchmark_data_smithi194_103039_object125705:head# 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43afce1c:::benchmark_data_smithi194_103039_object77345:head# 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b069bf:::benchmark_data_smithi194_103039_object90826:head# 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b0e0e6:::benchmark_data_smithi194_103039_object42403:head# 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b15bcd:::benchmark_data_smithi194_103039_object84581:head# 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b194a6:::benchmark_data_smithi194_103039_object96545:head# 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b24d9e:::benchmark_data_smithi194_103039_object111895:head# 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b32b81:::benchmark_data_smithi194_103039_object101926:head# 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b37984:::benchmark_data_smithi194_103039_object150431:head# 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b4b457:::benchmark_data_smithi194_103039_object101138:head# 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b6ad4c:::benchmark_data_smithi194_103039_object118895:head# 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b6d498:::benchmark_data_smithi194_103039_object67436:head# 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b74036:::benchmark_data_smithi194_103039_object99714:head# 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b7f61f:::benchmark_data_smithi194_103039_object126005:head# 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b7fb71:::benchmark_data_smithi194_103039_object57203:head# 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b83d07:::benchmark_data_smithi194_103039_object48148:head# 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b87039:::benchmark_data_smithi194_103039_object141051:head# 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b8b9e6:::benchmark_data_smithi194_103039_object126556:head# 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b92da2:::benchmark_data_smithi194_103039_object44095:head# 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43b9a0fd:::benchmark_data_smithi194_103039_object63188:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43baa6e0:::benchmark_data_smithi194_103039_object77701:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43bb1e39:::benchmark_data_smithi194_103039_object105421:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43bc63a5:::benchmark_data_smithi194_103039_object47685:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43be05c1:::benchmark_data_smithi194_103039_object140273:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43be4e05:::benchmark_data_smithi194_103039_object94220:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43be9f1f:::benchmark_data_smithi194_103039_object67110:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43bf5222:::benchmark_data_smithi194_103039_object49070:head# 2024-04-29T14:21:29.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c2136a:::benchmark_data_smithi194_103039_object117110:head# 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c271c7:::benchmark_data_smithi194_103039_object44270:head# 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c28198:::benchmark_data_smithi194_103039_object61677:head# 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c38905:::benchmark_data_smithi194_103039_object103122:head# 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c40444:::benchmark_data_smithi194_103039_object152742:head# 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c41e56:::benchmark_data_smithi194_103039_object131978:head# 2024-04-29T14:21:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c4541b:::benchmark_data_smithi194_103039_object70521:head# 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c51472:::benchmark_data_smithi194_103039_object84662:head# 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c581ef:::benchmark_data_smithi194_103039_object131396:head# 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c5dff7:::benchmark_data_smithi194_103039_object148360:head# 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c61457:::benchmark_data_smithi194_103039_object138953:head# 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c6fc25:::benchmark_data_smithi194_103039_object70191:head# 2024-04-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c7c330:::benchmark_data_smithi194_103039_object103608:head# 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c7f586:::benchmark_data_smithi194_103039_object140287:head# 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c92a2a:::benchmark_data_smithi194_103039_object150902:head# 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c944a7:::benchmark_data_smithi194_103039_object74229:head# 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c97b23:::benchmark_data_smithi194_103039_object127359:head# 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43c9d5cb:::benchmark_data_smithi194_103039_object143729:head# 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43cacbeb:::benchmark_data_smithi194_103039_object73524:head# 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43cb1ce9:::benchmark_data_smithi194_103039_object118368:head# 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43cbc043:::benchmark_data_smithi194_103039_object52938:head# 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43cccb21:::benchmark_data_smithi194_103039_object54953:head# 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ccd85d:::benchmark_data_smithi194_103039_object125713:head# 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43cdf29c:::benchmark_data_smithi194_103039_object138259:head# 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43cff5b5:::benchmark_data_smithi194_103039_object153268:head# 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d1ca6c:::benchmark_data_smithi194_103039_object142173:head# 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d236a6:::benchmark_data_smithi194_103039_object149162:head# 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d2acbb:::benchmark_data_smithi194_103039_object60215:head# 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d39b1e:::benchmark_data_smithi194_103039_object83620:head# 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d3da3f:::benchmark_data_smithi194_103039_object86709:head# 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d406a2:::benchmark_data_smithi194_103039_object146643:head# 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d42b28:::benchmark_data_smithi194_103039_object153758:head# 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d458a1:::benchmark_data_smithi194_103039_object87569:head# 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d4cce3:::benchmark_data_smithi194_103039_object47828:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d4de80:::benchmark_data_smithi194_103039_object136989:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d64b37:::benchmark_data_smithi194_103039_object149348:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d64be7:::benchmark_data_smithi194_103039_object137636:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d67718:::benchmark_data_smithi194_103039_object134674:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d6a5da:::benchmark_data_smithi194_103039_object154003:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d6b78b:::benchmark_data_smithi194_103039_object128200:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d7457c:::benchmark_data_smithi194_103039_object93873:head# 2024-04-29T14:21:29.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d74813:::benchmark_data_smithi194_103039_object156214:head# 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d78ea9:::benchmark_data_smithi194_103039_object142692:head# 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d7c32e:::benchmark_data_smithi194_103039_object82086:head# 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d7e3a2:::benchmark_data_smithi194_103039_object86893:head# 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d86814:::benchmark_data_smithi194_103039_object45068:head# 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43d9a2d0:::benchmark_data_smithi194_103039_object130661:head# 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43dab923:::benchmark_data_smithi194_103039_object137078:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43db003a:::benchmark_data_smithi194_103039_object135219:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43dc00c8:::benchmark_data_smithi194_103039_object42385:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43dc0470:::benchmark_data_smithi194_103039_object104432:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43dcc553:::benchmark_data_smithi194_103039_object134247:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ddc1fa:::benchmark_data_smithi194_103039_object111861:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ddee33:::benchmark_data_smithi194_103039_object89397:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43de1132:::benchmark_data_smithi194_103039_object136630:head# 2024-04-29T14:21:29.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43de4396:::benchmark_data_smithi194_103039_object125466:head# 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43deb171:::benchmark_data_smithi194_103039_object55367:head# 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43df250e:::benchmark_data_smithi194_103039_object145357:head# 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43df60ee:::benchmark_data_smithi194_103039_object97632:head# 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43dfdbfc:::benchmark_data_smithi194_103039_object104583:head# 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e06184:::benchmark_data_smithi194_103039_object148473:head# 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e0e770:::benchmark_data_smithi194_103039_object107537:head# 2024-04-29T14:21:29.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e0e946:::benchmark_data_smithi194_103039_object112008:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e15965:::benchmark_data_smithi194_103039_object136198:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e1e940:::benchmark_data_smithi194_103039_object108544:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e26f98:::benchmark_data_smithi194_103039_object77378:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e2855f:::benchmark_data_smithi194_103039_object105457:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e31884:::benchmark_data_smithi194_103039_object89760:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e48bae:::benchmark_data_smithi194_103039_object113350:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e65ff5:::benchmark_data_smithi194_103039_object113056:head# 2024-04-29T14:21:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e676a3:::benchmark_data_smithi194_103039_object86168:head# 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e6d496:::benchmark_data_smithi194_103039_object61713:head# 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43e89237:::benchmark_data_smithi194_103039_object48413:head# 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ea9787:::benchmark_data_smithi194_103039_object108650:head# 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ead394:::benchmark_data_smithi194_103039_object113354:head# 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43eaeb4d:::benchmark_data_smithi194_103039_object46549:head# 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ec1ee7:::benchmark_data_smithi194_103039_object94464:head# 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ec5866:::benchmark_data_smithi194_103039_object99539:head# 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ec5aa8:::benchmark_data_smithi194_103039_object149341:head# 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ed1f61:::benchmark_data_smithi194_103039_object98897:head# 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ed320b:::benchmark_data_smithi194_103039_object92473:head# 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ed5ac3:::benchmark_data_smithi194_103039_object97870:head# 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43edbb33:::benchmark_data_smithi194_103039_object65287:head# 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43ef19dc:::benchmark_data_smithi194_103039_object91863:head# 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f03df0:::benchmark_data_smithi194_103039_object120717:head# 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f0b326:::benchmark_data_smithi194_103039_object152139:head# 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f11d6a:::benchmark_data_smithi194_103039_object83510:head# 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f2274c:::benchmark_data_smithi194_103039_object85575:head# 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f242aa:::benchmark_data_smithi194_103039_object90246:head# 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f2e6cc:::benchmark_data_smithi194_103039_object49390:head# 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f3ddfd:::benchmark_data_smithi194_103039_object51209:head# 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f4e496:::benchmark_data_smithi194_103039_object108811:head# 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f682e8:::benchmark_data_smithi194_103039_object134172:head# 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f749a1:::benchmark_data_smithi194_103039_object81254:head# 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f7c7a3:::benchmark_data_smithi194_103039_object62441:head# 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f884a3:::benchmark_data_smithi194_103039_object120889:head# 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f8879a:::benchmark_data_smithi194_103039_object51039:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43f88ca2:::benchmark_data_smithi194_103039_object48338:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fa2aa3:::benchmark_data_smithi194_103039_object133244:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fb48c5:::benchmark_data_smithi194_103039_object89247:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fb5e74:::benchmark_data_smithi194_103039_object133585:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fbc543:::benchmark_data_smithi194_103039_object127811:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fc0d42:::benchmark_data_smithi194_103039_object60483:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fc5347:::benchmark_data_smithi194_103039_object88708:head# 2024-04-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fcb750:::benchmark_data_smithi194_103039_object96788:head# 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fcf4b1:::benchmark_data_smithi194_103039_object136428:head# 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fd39db:::benchmark_data_smithi194_103039_object129303:head# 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:43fe57bc:::benchmark_data_smithi194_103039_object139209:head# 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4400fa29:::benchmark_data_smithi194_103039_object56352:head# 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4400fb69:::benchmark_data_smithi194_103039_object71769:head# 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44011d0c:::benchmark_data_smithi194_103039_object46489:head# 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4401ddf2:::benchmark_data_smithi194_103039_object81506:head# 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44045a10:::benchmark_data_smithi194_103039_object68234:head# 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44048aa4:::benchmark_data_smithi194_103039_object146310:head# 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44049970:::benchmark_data_smithi194_103039_object151005:head# 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4405656e:::benchmark_data_smithi194_103039_object125846:head# 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44059bd8:::benchmark_data_smithi194_103039_object42595:head# 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4405ecde:::benchmark_data_smithi194_103039_object97889:head# 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44071d2b:::benchmark_data_smithi194_103039_object47730:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4407435d:::benchmark_data_smithi194_103039_object86323:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44083385:::benchmark_data_smithi194_103039_object134857:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44087a25:::benchmark_data_smithi194_103039_object120929:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4408afb3:::benchmark_data_smithi194_103039_object67594:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4408c1d7:::benchmark_data_smithi194_103039_object118400:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44091365:::benchmark_data_smithi194_103039_object43430:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440a1c61:::benchmark_data_smithi194_103039_object114777:head# 2024-04-29T14:21:29.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440aca2a:::benchmark_data_smithi194_103039_object145620:head# 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440c5956:::benchmark_data_smithi194_103039_object76560:head# 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440d700f:::benchmark_data_smithi194_103039_object84347:head# 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440d9de3:::benchmark_data_smithi194_103039_object152604:head# 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440e0797:::benchmark_data_smithi194_103039_object134636:head# 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440e0cd3:::benchmark_data_smithi194_103039_object44388:head# 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:440ef959:::benchmark_data_smithi194_103039_object71069:head# 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441007bb:::benchmark_data_smithi194_103039_object130368:head# 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441096b9:::benchmark_data_smithi194_103039_object116424:head# 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4410a228:::benchmark_data_smithi194_103039_object85497:head# 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4410d4e6:::benchmark_data_smithi194_103039_object138407:head# 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4410d8dd:::benchmark_data_smithi194_103039_object87080:head# 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441123f9:::benchmark_data_smithi194_103039_object129742:head# 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44113cfb:::benchmark_data_smithi194_103039_object93314:head# 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4411adf2:::benchmark_data_smithi194_103039_object129737:head# 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4411dff9:::benchmark_data_smithi194_103039_object77163:head# 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44120b23:::benchmark_data_smithi194_103039_object125175:head# 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4413ddfb:::benchmark_data_smithi194_103039_object132267:head# 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4414cc28:::benchmark_data_smithi194_103039_object77027:head# 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44157358:::benchmark_data_smithi194_103039_object124369:head# 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4415c400:::benchmark_data_smithi194_103039_object102817:head# 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4416a011:::benchmark_data_smithi194_103039_object78831:head# 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4417ebc9:::benchmark_data_smithi194_103039_object61254:head# 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44181c03:::benchmark_data_smithi194_103039_object127125:head# 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44192482:::benchmark_data_smithi194_103039_object63356:head# 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4419683b:::benchmark_data_smithi194_103039_object98304:head# 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4419c9fb:::benchmark_data_smithi194_103039_object110977:head# 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4419e70e:::benchmark_data_smithi194_103039_object46352:head# 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441a56aa:::benchmark_data_smithi194_103039_object131782:head# 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441c2635:::benchmark_data_smithi194_103039_object151501:head# 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441d5ad5:::benchmark_data_smithi194_103039_object42989:head# 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441dbda7:::benchmark_data_smithi194_103039_object148555:head# 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441e2bca:::benchmark_data_smithi194_103039_object63444:head# 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:441f3a16:::benchmark_data_smithi194_103039_object72666:head# 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44214f78:::benchmark_data_smithi194_103039_object128786:head# 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44225907:::benchmark_data_smithi194_103039_object61860:head# 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4422630f:::benchmark_data_smithi194_103039_object58772:head# 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4422f7c8:::benchmark_data_smithi194_103039_object71491:head# 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44230d25:::benchmark_data_smithi194_103039_object107381:head# 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4424e4b6:::benchmark_data_smithi194_103039_object62203:head# 2024-04-29T14:21:29.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44256d76:::benchmark_data_smithi194_103039_object64758:head# 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4425ed26:::benchmark_data_smithi194_103039_object79950:head# 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4426faf0:::benchmark_data_smithi194_103039_object129465:head# 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44271e78:::benchmark_data_smithi194_103039_object69275:head# 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4427ceed:::benchmark_data_smithi194_103039_object106080:head# 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4427e499:::benchmark_data_smithi194_103039_object110529:head# 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44289d04:::benchmark_data_smithi194_103039_object63692:head# 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4428c394:::benchmark_data_smithi194_103039_object70511:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4428c459:::benchmark_data_smithi194_103039_object51452:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4429a1da:::benchmark_data_smithi194_103039_object146238:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442a2f50:::benchmark_data_smithi194_103039_object51137:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442a6946:::benchmark_data_smithi194_103039_object135454:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442b0043:::benchmark_data_smithi194_103039_object75548:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442b7843:::benchmark_data_smithi194_103039_object123360:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442b92af:::benchmark_data_smithi194_103039_object86001:head# 2024-04-29T14:21:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442cd472:::benchmark_data_smithi194_103039_object64708:head# 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442efc03:::benchmark_data_smithi194_103039_object138464:head# 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442f30ee:::benchmark_data_smithi194_103039_object73379:head# 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:442fa6ea:::benchmark_data_smithi194_103039_object86646:head# 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44300c27:::benchmark_data_smithi194_103039_object87454:head# 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4430a299:::benchmark_data_smithi194_103039_object87048:head# 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4432ae5e:::benchmark_data_smithi194_103039_object129041:head# 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44334989:::benchmark_data_smithi194_103039_object130372:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4433fe11:::benchmark_data_smithi194_103039_object64731:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44343edb:::benchmark_data_smithi194_103039_object58014:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4434f095:::benchmark_data_smithi194_103039_object59810:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44350ae6:::benchmark_data_smithi194_103039_object130139:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443536cb:::benchmark_data_smithi194_103039_object136486:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44353dd8:::benchmark_data_smithi194_103039_object104353:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4435533a:::benchmark_data_smithi194_103039_object43016:head# 2024-04-29T14:21:29.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44358b13:::benchmark_data_smithi194_103039_object47271:head# 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4435b8b1:::benchmark_data_smithi194_103039_object75060:head# 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4435f01b:::benchmark_data_smithi194_103039_object80467:head# 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44365154:::benchmark_data_smithi194_103039_object56616:head# 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4436da5c:::benchmark_data_smithi194_103039_object115427:head# 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44392def:::benchmark_data_smithi194_103039_object117638:head# 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44399c1e:::benchmark_data_smithi194_103039_object101468:head# 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443a2436:::benchmark_data_smithi194_103039_object142763:head# 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443a31b9:::benchmark_data_smithi194_103039_object63959:head# 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443b369e:::benchmark_data_smithi194_103039_object102339:head# 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443c4116:::benchmark_data_smithi194_103039_object154465:head# 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443cb8d9:::benchmark_data_smithi194_103039_object45699:head# 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443d8240:::benchmark_data_smithi194_103039_object103374:head# 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443dcece:::benchmark_data_smithi194_103039_object129654:head# 2024-04-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443e4b6e:::benchmark_data_smithi194_103039_object113183:head# 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443ec3a1:::benchmark_data_smithi194_103039_object84294:head# 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443ed53a:::benchmark_data_smithi194_103039_object126615:head# 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443ef481:::benchmark_data_smithi194_103039_object75603:head# 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:443fab22:::benchmark_data_smithi194_103039_object52003:head# 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44400162:::benchmark_data_smithi194_103039_object122730:head# 2024-04-29T14:21:29.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44400eeb:::benchmark_data_smithi194_103039_object55503:head# 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44402b5b:::benchmark_data_smithi194_103039_object68005:head# 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444089aa:::benchmark_data_smithi194_103039_object133199:head# 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44425738:::benchmark_data_smithi194_103039_object80670:head# 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44429ae0:::benchmark_data_smithi194_103039_object74416:head# 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4442b8da:::benchmark_data_smithi194_103039_object120631:head# 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44431ba1:::benchmark_data_smithi194_103039_object94082:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4443df2b:::benchmark_data_smithi194_103039_object122651:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44453a06:::benchmark_data_smithi194_103039_object115131:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44453c9c:::benchmark_data_smithi194_103039_object104265:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4445a654:::benchmark_data_smithi194_103039_object127342:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44460906:::benchmark_data_smithi194_103039_object69613:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44463986:::benchmark_data_smithi194_103039_object55262:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4446d2a2:::benchmark_data_smithi194_103039_object129444:head# 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44470c09:::benchmark_data_smithi194_103039_object151533:head# 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4447cc8d:::benchmark_data_smithi194_103039_object138764:head# 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44486e39:::benchmark_data_smithi194_103039_object141216:head# 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44489a36:::benchmark_data_smithi194_103039_object98866:head# 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44498250:::benchmark_data_smithi194_103039_object50727:head# 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4449ddc1:::benchmark_data_smithi194_103039_object149647:head# 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4449e32a:::benchmark_data_smithi194_103039_object128304:head# 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444a501c:::benchmark_data_smithi194_103039_object48791:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444b061f:::benchmark_data_smithi194_103039_object57210:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444c7c46:::benchmark_data_smithi194_103039_object155421:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444c7cfc:::benchmark_data_smithi194_103039_object82783:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444c8748:::benchmark_data_smithi194_103039_object58086:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444d239a:::benchmark_data_smithi194_103039_object124388:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444d548a:::benchmark_data_smithi194_103039_object146911:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444d81ab:::benchmark_data_smithi194_103039_object152181:head# 2024-04-29T14:21:29.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444dcbac:::benchmark_data_smithi194_103039_object149680:head# 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444dd671:::benchmark_data_smithi194_103039_object123049:head# 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444debea:::benchmark_data_smithi194_103039_object49589:head# 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444df661:::benchmark_data_smithi194_103039_object96164:head# 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444e0d14:::benchmark_data_smithi194_103039_object125091:head# 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444e8b58:::benchmark_data_smithi194_103039_object111017:head# 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444eae85:::benchmark_data_smithi194_103039_object66520:head# 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444eecfd:::benchmark_data_smithi194_103039_object75523:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:444fff1c:::benchmark_data_smithi194_103039_object130475:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44500618:::benchmark_data_smithi194_103039_object151871:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44502678:::benchmark_data_smithi194_103039_object77957:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4450e735:::benchmark_data_smithi194_103039_object147731:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44511aa3:::benchmark_data_smithi194_103039_object98315:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4451a35c:::benchmark_data_smithi194_103039_object87410:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4452547b:::benchmark_data_smithi194_103039_object77459:head# 2024-04-29T14:21:29.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4452bd0d:::benchmark_data_smithi194_103039_object89952:head# 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:445389ce:::benchmark_data_smithi194_103039_object88754:head# 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4453b6af:::benchmark_data_smithi194_103039_object109455:head# 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44546647:::benchmark_data_smithi194_103039_object59116:head# 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:445628f4:::benchmark_data_smithi194_103039_object44749:head# 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44565284:::benchmark_data_smithi194_103039_object62655:head# 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44567386:::benchmark_data_smithi194_103039_object81864:head# 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44575950:::benchmark_data_smithi194_103039_object119984:head# 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44576f12:::benchmark_data_smithi194_103039_object134897:head# 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44585cda:::benchmark_data_smithi194_103039_object104658:head# 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4458e03e:::benchmark_data_smithi194_103039_object81986:head# 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:445a566d:::benchmark_data_smithi194_103039_object60339:head# 2024-04-29T14:21:29.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:445bed92:::benchmark_data_smithi194_103039_object82424:head# 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:445ccc97:::benchmark_data_smithi194_103039_object93537:head# 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:445ce91b:::benchmark_data_smithi194_103039_object57234:head# 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:445f733a:::benchmark_data_smithi194_103039_object97108:head# 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44615bdc:::benchmark_data_smithi194_103039_object51727:head# 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44617061:::benchmark_data_smithi194_103039_object70030:head# 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4461e6dd:::benchmark_data_smithi194_103039_object66385:head# 2024-04-29T14:21:29.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4461e8af:::benchmark_data_smithi194_103039_object109150:head# 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44625f09:::benchmark_data_smithi194_103039_object92660:head# 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4462c789:::benchmark_data_smithi194_103039_object75930:head# 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44633ff6:::benchmark_data_smithi194_103039_object111657:head# 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4464593a:::benchmark_data_smithi194_103039_object44641:head# 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4464af9e:::benchmark_data_smithi194_103039_object140600:head# 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44650e19:::benchmark_data_smithi194_103039_object115471:head# 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44654434:::benchmark_data_smithi194_103039_object147149:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4465dc38:::benchmark_data_smithi194_103039_object129735:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446622f9:::benchmark_data_smithi194_103039_object138626:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4466952c:::benchmark_data_smithi194_103039_object92065:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44678bf6:::benchmark_data_smithi194_103039_object148136:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4467eb65:::benchmark_data_smithi194_103039_object62690:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44689433:::benchmark_data_smithi194_103039_object50180:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44692040:::benchmark_data_smithi194_103039_object84801:head# 2024-04-29T14:21:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446a9cf5:::benchmark_data_smithi194_103039_object48331:head# 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446b1683:::benchmark_data_smithi194_103039_object103840:head# 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446d550f:::benchmark_data_smithi194_103039_object43308:head# 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446d8ee9:::benchmark_data_smithi194_103039_object54730:head# 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446e1637:::benchmark_data_smithi194_103039_object136265:head# 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446f61a9:::benchmark_data_smithi194_103039_object109745:head# 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:446ffa9e:::benchmark_data_smithi194_103039_object131100:head# 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447086c5:::benchmark_data_smithi194_103039_object89673:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4471c28d:::benchmark_data_smithi194_103039_object146558:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44735969:::benchmark_data_smithi194_103039_object84327:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4473c7fb:::benchmark_data_smithi194_103039_object147400:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4473cb82:::benchmark_data_smithi194_103039_object147706:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4474746e:::benchmark_data_smithi194_103039_object45697:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4474d082:::benchmark_data_smithi194_103039_object147611:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4474e07b:::benchmark_data_smithi194_103039_object89303:head# 2024-04-29T14:21:29.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4475a9b1:::benchmark_data_smithi194_103039_object88600:head# 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4476bd80:::benchmark_data_smithi194_103039_object58188:head# 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447856b3:::benchmark_data_smithi194_103039_object128942:head# 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44799ad9:::benchmark_data_smithi194_103039_object62235:head# 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447a0b8d:::benchmark_data_smithi194_103039_object146443:head# 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447a879c:::benchmark_data_smithi194_103039_object117567:head# 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447abed9:::benchmark_data_smithi194_103039_object87852:head# 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447ae714:::benchmark_data_smithi194_103039_object131063:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447c1412:::benchmark_data_smithi194_103039_object156301:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447c52f9:::benchmark_data_smithi194_103039_object132053:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447cba25:::benchmark_data_smithi194_103039_object127695:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447d8468:::benchmark_data_smithi194_103039_object91701:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447ed62a:::benchmark_data_smithi194_103039_object62818:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447edf7b:::benchmark_data_smithi194_103039_object112198:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447f137d:::benchmark_data_smithi194_103039_object71322:head# 2024-04-29T14:21:29.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:447fbe6c:::benchmark_data_smithi194_103039_object75061:head# 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4480a82f:::benchmark_data_smithi194_103039_object87014:head# 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4480e770:::benchmark_data_smithi194_103039_object81262:head# 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4481abf8:::benchmark_data_smithi194_103039_object57307:head# 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44833b79:::benchmark_data_smithi194_103039_object103303:head# 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44834c15:::benchmark_data_smithi194_103039_object61924:head# 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4483fce3:::benchmark_data_smithi194_103039_object125061:head# 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4484906f:::benchmark_data_smithi194_103039_object110558:head# 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4484d3ea:::benchmark_data_smithi194_103039_object82837:head# 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44858563:::benchmark_data_smithi194_103039_object118072:head# 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44859410:::benchmark_data_smithi194_103039_object99211:head# 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44875bb9:::benchmark_data_smithi194_103039_object155438:head# 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44879ccf:::benchmark_data_smithi194_103039_object90097:head# 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4487a4a7:::benchmark_data_smithi194_103039_object107225:head# 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44880116:::benchmark_data_smithi194_103039_object49432:head# 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44882d55:::benchmark_data_smithi194_103039_object100936:head# 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44885ac9:::benchmark_data_smithi194_103039_object75409:head# 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44887a7e:::benchmark_data_smithi194_103039_object65083:head# 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44887b9c:::benchmark_data_smithi194_103039_object48059:head# 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4489312a:::benchmark_data_smithi194_103039_object52601:head# 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44897afd:::benchmark_data_smithi194_103039_object115265:head# 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448a0fad:::benchmark_data_smithi194_103039_object127461:head# 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448a21b4:::benchmark_data_smithi194_103039_object71806:head# 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448a5501:::benchmark_data_smithi194_103039_object106978:head# 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448ad6a6:::benchmark_data_smithi194_103039_object127466:head# 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448b9c41:::benchmark_data_smithi194_103039_object144871:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448c38fb:::benchmark_data_smithi194_103039_object57007:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448de5a8:::benchmark_data_smithi194_103039_object113299:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448e86da:::benchmark_data_smithi194_103039_object156606:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448f0d01:::benchmark_data_smithi194_103039_object111716:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448fca4f:::benchmark_data_smithi194_103039_object124585:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:448fd2f4:::benchmark_data_smithi194_103039_object83977:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44903a0f:::benchmark_data_smithi194_103039_object136888:head# 2024-04-29T14:21:29.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4491ad3a:::benchmark_data_smithi194_103039_object81240:head# 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4491b08d:::benchmark_data_smithi194_103039_object81851:head# 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449269f1:::benchmark_data_smithi194_103039_object141074:head# 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44926b80:::benchmark_data_smithi194_103039_object67500:head# 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4492700e:::benchmark_data_smithi194_103039_object137286:head# 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4492a73c:::benchmark_data_smithi194_103039_object139134:head# 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449301a7:::benchmark_data_smithi194_103039_object132144:head# 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449335b7:::benchmark_data_smithi194_103039_object57078:head# 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44933e74:::benchmark_data_smithi194_103039_object108287:head# 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4493f0d2:::benchmark_data_smithi194_103039_object113828:head# 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449471c4:::benchmark_data_smithi194_103039_object128274:head# 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4494e907:::benchmark_data_smithi194_103039_object112767:head# 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4495c3b6:::benchmark_data_smithi194_103039_object72611:head# 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4496c685:::benchmark_data_smithi194_103039_object57943:head# 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44970ff8:::benchmark_data_smithi194_103039_object86208:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4497211f:::benchmark_data_smithi194_103039_object79370:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44972aa7:::benchmark_data_smithi194_103039_object101734:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4497679c:::benchmark_data_smithi194_103039_object136839:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4497a6d0:::benchmark_data_smithi194_103039_object112139:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4498855c:::benchmark_data_smithi194_103039_object131683:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44989bbe:::benchmark_data_smithi194_103039_object64357:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4498b0b2:::benchmark_data_smithi194_103039_object149309:head# 2024-04-29T14:21:29.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4498bc05:::benchmark_data_smithi194_103039_object110132:head# 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44991698:::benchmark_data_smithi194_103039_object153348:head# 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44991e45:::benchmark_data_smithi194_103039_object91698:head# 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44995164:::benchmark_data_smithi194_103039_object57059:head# 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449b23a5:::benchmark_data_smithi194_103039_object112011:head# 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449b9e5d:::benchmark_data_smithi194_103039_object90086:head# 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449cb9ad:::benchmark_data_smithi194_103039_object153658:head# 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449ce383:::benchmark_data_smithi194_103039_object50568:head# 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449db605:::benchmark_data_smithi194_103039_object119789:head# 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449fb1f6:::benchmark_data_smithi194_103039_object87843:head# 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:449fcc12:::benchmark_data_smithi194_103039_object57807:head# 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a17d87:::benchmark_data_smithi194_103039_object114858:head# 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a1dee2:::benchmark_data_smithi194_103039_object108730:head# 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a44569:::benchmark_data_smithi194_103039_object117204:head# 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a62348:::benchmark_data_smithi194_103039_object82976:head# 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a71256:::benchmark_data_smithi194_103039_object132533:head# 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a7c2ea:::benchmark_data_smithi194_103039_object64227:head# 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a814bd:::benchmark_data_smithi194_103039_object60096:head# 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a82174:::benchmark_data_smithi194_103039_object58734:head# 2024-04-29T14:21:29.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a8878e:::benchmark_data_smithi194_103039_object71625:head# 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a94b99:::benchmark_data_smithi194_103039_object130047:head# 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a966af:::benchmark_data_smithi194_103039_object98492:head# 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44a98318:::benchmark_data_smithi194_103039_object122067:head# 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ab6242:::benchmark_data_smithi194_103039_object113143:head# 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ac1fb6:::benchmark_data_smithi194_103039_object115170:head# 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44acff76:::benchmark_data_smithi194_103039_object49195:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ad0cf2:::benchmark_data_smithi194_103039_object125998:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ad46f5:::benchmark_data_smithi194_103039_object113829:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ad8cbd:::benchmark_data_smithi194_103039_object154483:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44adaf72:::benchmark_data_smithi194_103039_object110606:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44adb7e1:::benchmark_data_smithi194_103039_object58781:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ae9b14:::benchmark_data_smithi194_103039_object104117:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44af6291:::benchmark_data_smithi194_103039_object140061:head# 2024-04-29T14:21:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44af7406:::benchmark_data_smithi194_103039_object58522:head# 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b0406c:::benchmark_data_smithi194_103039_object51538:head# 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b07201:::benchmark_data_smithi194_103039_object82371:head# 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b0734b:::benchmark_data_smithi194_103039_object119017:head# 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b13755:::benchmark_data_smithi194_103039_object72729:head# 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b15611:::benchmark_data_smithi194_103039_object96277:head# 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b28297:::benchmark_data_smithi194_103039_object127413:head# 2024-04-29T14:21:29.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b35866:::benchmark_data_smithi194_103039_object129766:head# 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b3c96c:::benchmark_data_smithi194_103039_object129193:head# 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b3d9aa:::benchmark_data_smithi194_103039_object100468:head# 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b3f681:::benchmark_data_smithi194_103039_object93108:head# 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b3f73f:::benchmark_data_smithi194_103039_object111091:head# 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b48442:::benchmark_data_smithi194_103039_object115727:head# 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b51b1e:::benchmark_data_smithi194_103039_object124927:head# 2024-04-29T14:21:29.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b5e72e:::benchmark_data_smithi194_103039_object63650:head# 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b650f7:::benchmark_data_smithi194_103039_object79929:head# 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b6b16e:::benchmark_data_smithi194_103039_object117126:head# 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b746d6:::benchmark_data_smithi194_103039_object126946:head# 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b76127:::benchmark_data_smithi194_103039_object62502:head# 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44b9402e:::benchmark_data_smithi194_103039_object92630:head# 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ba8d2f:::benchmark_data_smithi194_103039_object123391:head# 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44bb60ac:::benchmark_data_smithi194_103039_object61292:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44bbffc5:::benchmark_data_smithi194_103039_object119758:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44bd7034:::benchmark_data_smithi194_103039_object44974:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44bd8293:::benchmark_data_smithi194_103039_object115109:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44be49d5:::benchmark_data_smithi194_103039_object72377:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44be51ea:::benchmark_data_smithi194_103039_object96450:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44be8864:::benchmark_data_smithi194_103039_object59966:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44bf4bbb:::benchmark_data_smithi194_103039_object115810:head# 2024-04-29T14:21:29.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44bf925b:::benchmark_data_smithi194_103039_object85799:head# 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44bfe67c:::benchmark_data_smithi194_103039_object101736:head# 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c004f8:::benchmark_data_smithi194_103039_object102482:head# 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c017bb:::benchmark_data_smithi194_103039_object51099:head# 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c0a1f2:::benchmark_data_smithi194_103039_object62231:head# 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c163b0:::benchmark_data_smithi194_103039_object120217:head# 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c1955c:::benchmark_data_smithi194_103039_object61161:head# 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c1d135:::benchmark_data_smithi194_103039_object51214:head# 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c2341b:::benchmark_data_smithi194_103039_object114417:head# 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c24be5:::benchmark_data_smithi194_103039_object151939:head# 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c34ede:::benchmark_data_smithi194_103039_object54842:head# 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c450d6:::benchmark_data_smithi194_103039_object79337:head# 2024-04-29T14:21:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c4d0d6:::benchmark_data_smithi194_103039_object73894:head# 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c905eb:::benchmark_data_smithi194_103039_object139950:head# 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c99cd7:::benchmark_data_smithi194_103039_object102688:head# 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44c9df62:::benchmark_data_smithi194_103039_object47951:head# 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ca929e:::benchmark_data_smithi194_103039_object91267:head# 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cb01f8:::benchmark_data_smithi194_103039_object130449:head# 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cb4776:::benchmark_data_smithi194_103039_object113657:head# 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cb705e:::benchmark_data_smithi194_103039_object55271:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cbbd6a:::benchmark_data_smithi194_103039_object133225:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cc3c60:::benchmark_data_smithi194_103039_object118591:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cc7746:::benchmark_data_smithi194_103039_object130404:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cc8460:::benchmark_data_smithi194_103039_object129410:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ccdd58:::benchmark_data_smithi194_103039_object61486:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cce33b:::benchmark_data_smithi194_103039_object78461:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ce1ecc:::benchmark_data_smithi194_103039_object53664:head# 2024-04-29T14:21:29.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cecdd6:::benchmark_data_smithi194_103039_object131581:head# 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ced062:::benchmark_data_smithi194_103039_object122328:head# 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ced32b:::benchmark_data_smithi194_103039_object142396:head# 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cf021b:::benchmark_data_smithi194_103039_object118146:head# 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cf1561:::benchmark_data_smithi194_103039_object60693:head# 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cf19a6:::benchmark_data_smithi194_103039_object120075:head# 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cfd2e2:::benchmark_data_smithi194_103039_object82310:head# 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44cfe706:::benchmark_data_smithi194_103039_object54592:head# 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d0d493:::benchmark_data_smithi194_103039_object94143:head# 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d1210f:::benchmark_data_smithi194_103039_object135590:head# 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d2f2e0:::benchmark_data_smithi194_103039_object155786:head# 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d30dde:::benchmark_data_smithi194_103039_object132724:head# 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d35a47:::benchmark_data_smithi194_103039_object146387:head# 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d600e8:::benchmark_data_smithi194_103039_object49270:head# 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d65b18:::benchmark_data_smithi194_103039_object107010:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d6fd06:::benchmark_data_smithi194_103039_object49512:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d7a12b:::benchmark_data_smithi194_103039_object102586:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d88c23:::benchmark_data_smithi194_103039_object123923:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44d9d56c:::benchmark_data_smithi194_103039_object121033:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44da15b8:::benchmark_data_smithi194_103039_object134834:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44da40b8:::benchmark_data_smithi194_103039_object134254:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dab1d0:::benchmark_data_smithi194_103039_object118664:head# 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dab286:::benchmark_data_smithi194_103039_object148687:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dac221:::benchmark_data_smithi194_103039_object139752:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dafd21:::benchmark_data_smithi194_103039_object66302:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44db3d95:::benchmark_data_smithi194_103039_object57771:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dba2a6:::benchmark_data_smithi194_103039_object88582:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dc1083:::benchmark_data_smithi194_103039_object105573:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dc998a:::benchmark_data_smithi194_103039_object111633:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dca9ed:::benchmark_data_smithi194_103039_object51114:head# 2024-04-29T14:21:29.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dd26d9:::benchmark_data_smithi194_103039_object117484:head# 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44dd72ab:::benchmark_data_smithi194_103039_object108308:head# 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44df9c0c:::benchmark_data_smithi194_103039_object155365:head# 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e00ab6:::benchmark_data_smithi194_103039_object88782:head# 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e17af1:::benchmark_data_smithi194_103039_object55697:head# 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e3e42b:::benchmark_data_smithi194_103039_object124247:head# 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e413e8:::benchmark_data_smithi194_103039_object64220:head# 2024-04-29T14:21:29.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e44ef1:::benchmark_data_smithi194_103039_object95103:head# 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e4e4ca:::benchmark_data_smithi194_103039_object42410:head# 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e4e84d:::benchmark_data_smithi194_103039_object136971:head# 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e6ba9a:::benchmark_data_smithi194_103039_object137287:head# 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e76026:::benchmark_data_smithi194_103039_object104206:head# 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e7a843:::benchmark_data_smithi194_103039_object45824:head# 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e7e8cb:::benchmark_data_smithi194_103039_object142843:head# 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e80144:::benchmark_data_smithi194_103039_object127803:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e88f70:::benchmark_data_smithi194_103039_object42033:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44e96dad:::benchmark_data_smithi194_103039_object144127:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ebdcce:::benchmark_data_smithi194_103039_object74888:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ec05a0:::benchmark_data_smithi194_103039_object50390:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ec51b6:::benchmark_data_smithi194_103039_object80513:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44edfadb:::benchmark_data_smithi194_103039_object138951:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ee1b45:::benchmark_data_smithi194_103039_object135685:head# 2024-04-29T14:21:29.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44eee7c8:::benchmark_data_smithi194_103039_object99699:head# 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ef7888:::benchmark_data_smithi194_103039_object57079:head# 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ef81d6:::benchmark_data_smithi194_103039_object138357:head# 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f150eb:::benchmark_data_smithi194_103039_object120102:head# 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f23ad6:::benchmark_data_smithi194_103039_object131270:head# 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f425df:::benchmark_data_smithi194_103039_object66413:head# 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f470c9:::benchmark_data_smithi194_103039_object148722:head# 2024-04-29T14:21:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f5542b:::benchmark_data_smithi194_103039_object58150:head# 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f5afe1:::benchmark_data_smithi194_103039_object91643:head# 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f60e39:::benchmark_data_smithi194_103039_object67309:head# 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f67bcb:::benchmark_data_smithi194_103039_object109936:head# 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f6c0c0:::benchmark_data_smithi194_103039_object97996:head# 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f6f2af:::benchmark_data_smithi194_103039_object131884:head# 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44f8b974:::benchmark_data_smithi194_103039_object60194:head# 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fa2164:::benchmark_data_smithi194_103039_object74130:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fa7296:::benchmark_data_smithi194_103039_object48763:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fad9a6:::benchmark_data_smithi194_103039_object142515:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fb07f1:::benchmark_data_smithi194_103039_object140890:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fb18ee:::benchmark_data_smithi194_103039_object126453:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fb194a:::benchmark_data_smithi194_103039_object130124:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fb58bc:::benchmark_data_smithi194_103039_object123493:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fb828d:::benchmark_data_smithi194_103039_object84075:head# 2024-04-29T14:21:29.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fc5933:::benchmark_data_smithi194_103039_object73631:head# 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fcdf07:::benchmark_data_smithi194_103039_object106836:head# 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fd3ec4:::benchmark_data_smithi194_103039_object84437:head# 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fe39e6:::benchmark_data_smithi194_103039_object88865:head# 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fe4387:::benchmark_data_smithi194_103039_object138079:head# 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fea3a6:::benchmark_data_smithi194_103039_object46183:head# 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44feccdb:::benchmark_data_smithi194_103039_object74475:head# 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fed20e:::benchmark_data_smithi194_103039_object78087:head# 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ff5b8f:::benchmark_data_smithi194_103039_object146184:head# 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ff795f:::benchmark_data_smithi194_103039_object91807:head# 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ff8b4e:::benchmark_data_smithi194_103039_object111187:head# 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ffbe28:::benchmark_data_smithi194_103039_object50464:head# 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ffc3b2:::benchmark_data_smithi194_103039_object119183:head# 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44ffe0dd:::benchmark_data_smithi194_103039_object137195:head# 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:44fff163:::benchmark_data_smithi194_103039_object146095:head# 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450009d2:::benchmark_data_smithi194_103039_object88859:head# 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4500c321:::benchmark_data_smithi194_103039_object80030:head# 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45010205:::benchmark_data_smithi194_103039_object123786:head# 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450147cb:::benchmark_data_smithi194_103039_object68582:head# 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450271bf:::benchmark_data_smithi194_103039_object89853:head# 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4502a41e:::benchmark_data_smithi194_103039_object76800:head# 2024-04-29T14:21:29.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450451d0:::benchmark_data_smithi194_103039_object82448:head# 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45054a23:::benchmark_data_smithi194_103039_object128363:head# 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45069a52:::benchmark_data_smithi194_103039_object128012:head# 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4506dc81:::benchmark_data_smithi194_103039_object132803:head# 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45071264:::benchmark_data_smithi194_103039_object66995:head# 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45075c6c:::benchmark_data_smithi194_103039_object113370:head# 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4507f95d:::benchmark_data_smithi194_103039_object149789:head# 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450833fe:::benchmark_data_smithi194_103039_object94799:head# 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45088be2:::benchmark_data_smithi194_103039_object95884:head# 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45094859:::benchmark_data_smithi194_103039_object52135:head# 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4509e91d:::benchmark_data_smithi194_103039_object45453:head# 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450a2ba5:::benchmark_data_smithi194_103039_object65987:head# 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450a821e:::benchmark_data_smithi194_103039_object53304:head# 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450b5674:::benchmark_data_smithi194_103039_object141380:head# 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450d338a:::benchmark_data_smithi194_103039_object91590:head# 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450d67db:::benchmark_data_smithi194_103039_object131103:head# 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450e0261:::benchmark_data_smithi194_103039_object65681:head# 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450eaa03:::benchmark_data_smithi194_103039_object41784:head# 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450ed099:::benchmark_data_smithi194_103039_object71293:head# 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:450f3f57:::benchmark_data_smithi194_103039_object131026:head# 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451103bb:::benchmark_data_smithi194_103039_object126835:head# 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45134761:::benchmark_data_smithi194_103039_object123270:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45138261:::benchmark_data_smithi194_103039_object142972:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45138c36:::benchmark_data_smithi194_103039_object109553:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451456d5:::benchmark_data_smithi194_103039_object96011:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45147838:::benchmark_data_smithi194_103039_object73515:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451479c1:::benchmark_data_smithi194_103039_object93747:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4514e6fe:::benchmark_data_smithi194_103039_object128273:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451591cf:::benchmark_data_smithi194_103039_object138203:head# 2024-04-29T14:21:29.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4515afee:::benchmark_data_smithi194_103039_object52275:head# 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451658ee:::benchmark_data_smithi194_103039_object147536:head# 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45166423:::benchmark_data_smithi194_103039_object89643:head# 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4516d06a:::benchmark_data_smithi194_103039_object145597:head# 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45173676:::benchmark_data_smithi194_103039_object78971:head# 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45177b6a:::benchmark_data_smithi194_103039_object119298:head# 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4517cc1b:::benchmark_data_smithi194_103039_object117615:head# 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4518066b:::benchmark_data_smithi194_103039_object73461:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451818e4:::benchmark_data_smithi194_103039_object94788:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4519abf3:::benchmark_data_smithi194_103039_object57410:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4519f90a:::benchmark_data_smithi194_103039_object99160:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451a529b:::benchmark_data_smithi194_103039_object107962:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451ad202:::benchmark_data_smithi194_103039_object120489:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451afe8f:::benchmark_data_smithi194_103039_object92654:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451c4d18:::benchmark_data_smithi194_103039_object156018:head# 2024-04-29T14:21:29.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451c5cc6:::benchmark_data_smithi194_103039_object87598:head# 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451ce517:::benchmark_data_smithi194_103039_object96292:head# 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451ce867:::benchmark_data_smithi194_103039_object111489:head# 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451d7123:::benchmark_data_smithi194_103039_object141973:head# 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451dd288:::benchmark_data_smithi194_103039_object99904:head# 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451edbab:::benchmark_data_smithi194_103039_object90319:head# 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451f2b67:::benchmark_data_smithi194_103039_object113794:head# 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:451f95a8:::benchmark_data_smithi194_103039_object68107:head# 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4520086c:::benchmark_data_smithi194_103039_object80121:head# 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4520760a:::benchmark_data_smithi194_103039_object136371:head# 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452077e0:::benchmark_data_smithi194_103039_object140792:head# 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45215269:::benchmark_data_smithi194_103039_object96449:head# 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45228840:::benchmark_data_smithi194_103039_object75649:head# 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4522aa9f:::benchmark_data_smithi194_103039_object108574:head# 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4524e9b5:::benchmark_data_smithi194_103039_object46400:head# 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45253115:::benchmark_data_smithi194_103039_object59542:head# 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452666ac:::benchmark_data_smithi194_103039_object124668:head# 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45287973:::benchmark_data_smithi194_103039_object138444:head# 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45289c68:::benchmark_data_smithi194_103039_object100620:head# 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4528ce73:::benchmark_data_smithi194_103039_object83303:head# 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452a0643:::benchmark_data_smithi194_103039_object140643:head# 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452b1c82:::benchmark_data_smithi194_103039_object57217:head# 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452cb646:::benchmark_data_smithi194_103039_object140272:head# 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452cc75d:::benchmark_data_smithi194_103039_object80259:head# 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452cf711:::benchmark_data_smithi194_103039_object90012:head# 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452d48c0:::benchmark_data_smithi194_103039_object61203:head# 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452e9d37:::benchmark_data_smithi194_103039_object100577:head# 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452f011a:::benchmark_data_smithi194_103039_object98570:head# 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452f460a:::benchmark_data_smithi194_103039_object50746:head# 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:452f8eea:::benchmark_data_smithi194_103039_object145534:head# 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4530695c:::benchmark_data_smithi194_103039_object144388:head# 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4531443a:::benchmark_data_smithi194_103039_object63197:head# 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4531768b:::benchmark_data_smithi194_103039_object138716:head# 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4531acb5:::benchmark_data_smithi194_103039_object118585:head# 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4532a860:::benchmark_data_smithi194_103039_object119686:head# 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453360ad:::benchmark_data_smithi194_103039_object113024:head# 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4533b9ff:::benchmark_data_smithi194_103039_object125393:head# 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4533c8d8:::benchmark_data_smithi194_103039_object119589:head# 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45346daf:::benchmark_data_smithi194_103039_object60749:head# 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4534787d:::benchmark_data_smithi194_103039_object90507:head# 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45358f1d:::benchmark_data_smithi194_103039_object81150:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4535a8d6:::benchmark_data_smithi194_103039_object137463:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453647b7:::benchmark_data_smithi194_103039_object74974:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4536affb:::benchmark_data_smithi194_103039_object57393:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4537e03c:::benchmark_data_smithi194_103039_object87554:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4538436f:::benchmark_data_smithi194_103039_object81226:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4538a066:::benchmark_data_smithi194_103039_object84031:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4538fe96:::benchmark_data_smithi194_103039_object155042:head# 2024-04-29T14:21:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45395d03:::benchmark_data_smithi194_103039_object106821:head# 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453afedc:::benchmark_data_smithi194_103039_object63366:head# 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453b34f6:::benchmark_data_smithi194_103039_object119843:head# 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453b4758:::benchmark_data_smithi194_103039_object92770:head# 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453bab6c:::benchmark_data_smithi194_103039_object147569:head# 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453cd96c:::benchmark_data_smithi194_103039_object89412:head# 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453cf368:::benchmark_data_smithi194_103039_object51933:head# 2024-04-29T14:21:29.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453cff5a:::benchmark_data_smithi194_103039_object138579:head# 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453d0b2e:::benchmark_data_smithi194_103039_object106613:head# 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453d5a71:::benchmark_data_smithi194_103039_object106924:head# 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453d9007:::benchmark_data_smithi194_103039_object72890:head# 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:453ff1d3:::benchmark_data_smithi194_103039_object57243:head# 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45407f0a:::benchmark_data_smithi194_103039_object65031:head# 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45413b5d:::benchmark_data_smithi194_103039_object126139:head# 2024-04-29T14:21:29.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45414ec8:::benchmark_data_smithi194_103039_object125931:head# 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454400a6:::benchmark_data_smithi194_103039_object87332:head# 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45447cf0:::benchmark_data_smithi194_103039_object75401:head# 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4544cff9:::benchmark_data_smithi194_103039_object55984:head# 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4545565b:::benchmark_data_smithi194_103039_object109323:head# 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454560fc:::benchmark_data_smithi194_103039_object65697:head# 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4545bc05:::benchmark_data_smithi194_103039_object46869:head# 2024-04-29T14:21:29.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4545cb67:::benchmark_data_smithi194_103039_object135064:head# 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454654b6:::benchmark_data_smithi194_103039_object52320:head# 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45476bfe:::benchmark_data_smithi194_103039_object97982:head# 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45499169:::benchmark_data_smithi194_103039_object113101:head# 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454a92fa:::benchmark_data_smithi194_103039_object149593:head# 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454b3ab0:::benchmark_data_smithi194_103039_object74222:head# 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454b609b:::benchmark_data_smithi194_103039_object42203:head# 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454cdac6:::benchmark_data_smithi194_103039_object98394:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454d3005:::benchmark_data_smithi194_103039_object69731:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454d903b:::benchmark_data_smithi194_103039_object83645:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454dfddb:::benchmark_data_smithi194_103039_object128589:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454dfe22:::benchmark_data_smithi194_103039_object124693:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:454ffd6e:::benchmark_data_smithi194_103039_object63635:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45508dc3:::benchmark_data_smithi194_103039_object99805:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4550a58a:::benchmark_data_smithi194_103039_object54726:head# 2024-04-29T14:21:29.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4552796a:::benchmark_data_smithi194_103039_object150900:head# 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45537594:::benchmark_data_smithi194_103039_object71016:head# 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45546440:::benchmark_data_smithi194_103039_object78975:head# 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4556820e:::benchmark_data_smithi194_103039_object78287:head# 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45568fca:::benchmark_data_smithi194_103039_object100070:head# 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45580651:::benchmark_data_smithi194_103039_object121847:head# 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455986a6:::benchmark_data_smithi194_103039_object130039:head# 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455a3109:::benchmark_data_smithi194_103039_object98979:head# 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455a5528:::benchmark_data_smithi194_103039_object62188:head# 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455aca85:::benchmark_data_smithi194_103039_object143224:head# 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455b1157:::benchmark_data_smithi194_103039_object92603:head# 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455b384d:::benchmark_data_smithi194_103039_object85923:head# 2024-04-29T14:21:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455dd9c5:::benchmark_data_smithi194_103039_object54866:head# 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455de3a4:::benchmark_data_smithi194_103039_object108372:head# 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455df5fd:::benchmark_data_smithi194_103039_object131606:head# 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455e42c5:::benchmark_data_smithi194_103039_object96751:head# 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:455f3afb:::benchmark_data_smithi194_103039_object91528:head# 2024-04-29T14:21:29.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4561a42d:::benchmark_data_smithi194_103039_object142288:head# 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4561fc15:::benchmark_data_smithi194_103039_object48135:head# 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45621b1a:::benchmark_data_smithi194_103039_object81447:head# 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45630b5e:::benchmark_data_smithi194_103039_object126878:head# 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45644ffe:::benchmark_data_smithi194_103039_object151895:head# 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456528cf:::benchmark_data_smithi194_103039_object124510:head# 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4565b1d3:::benchmark_data_smithi194_103039_object88071:head# 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4565f3e3:::benchmark_data_smithi194_103039_object47497:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45665ca9:::benchmark_data_smithi194_103039_object90114:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4566993e:::benchmark_data_smithi194_103039_object153221:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4566d155:::benchmark_data_smithi194_103039_object118311:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4567f535:::benchmark_data_smithi194_103039_object89878:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45690b1b:::benchmark_data_smithi194_103039_object123417:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45691bbc:::benchmark_data_smithi194_103039_object132548:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4569204e:::benchmark_data_smithi194_103039_object68055:head# 2024-04-29T14:21:29.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45694a99:::benchmark_data_smithi194_103039_object75145:head# 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4569f5b5:::benchmark_data_smithi194_103039_object127941:head# 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456a5c94:::benchmark_data_smithi194_103039_object122767:head# 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456afa4d:::benchmark_data_smithi194_103039_object156232:head# 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456b5dcf:::benchmark_data_smithi194_103039_object156538:head# 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456c4bab:::benchmark_data_smithi194_103039_object75457:head# 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456da0d2:::benchmark_data_smithi194_103039_object120219:head# 2024-04-29T14:21:29.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456e661f:::benchmark_data_smithi194_103039_object137891:head# 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456e8c7e:::benchmark_data_smithi194_103039_object141947:head# 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456eaa0e:::benchmark_data_smithi194_103039_object82203:head# 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456ec210:::benchmark_data_smithi194_103039_object60363:head# 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:456fd03e:::benchmark_data_smithi194_103039_object120540:head# 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4570370c:::benchmark_data_smithi194_103039_object117673:head# 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4570e1ac:::benchmark_data_smithi194_103039_object82533:head# 2024-04-29T14:21:29.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457233da:::benchmark_data_smithi194_103039_object133286:head# 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4572f278:::benchmark_data_smithi194_103039_object47276:head# 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4573577f:::benchmark_data_smithi194_103039_object126467:head# 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4574c406:::benchmark_data_smithi194_103039_object94893:head# 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457541cf:::benchmark_data_smithi194_103039_object49174:head# 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45756f67:::benchmark_data_smithi194_103039_object61347:head# 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457573bb:::benchmark_data_smithi194_103039_object50761:head# 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4575a792:::benchmark_data_smithi194_103039_object111367:head# 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4575fe38:::benchmark_data_smithi194_103039_object117079:head# 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457641f0:::benchmark_data_smithi194_103039_object138162:head# 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45765be0:::benchmark_data_smithi194_103039_object80735:head# 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45785e78:::benchmark_data_smithi194_103039_object93342:head# 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4578b1c6:::benchmark_data_smithi194_103039_object93490:head# 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4578f811:::benchmark_data_smithi194_103039_object117360:head# 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4579168a:::benchmark_data_smithi194_103039_object103800:head# 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45796f98:::benchmark_data_smithi194_103039_object71931:head# 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45799ee2:::benchmark_data_smithi194_103039_object110844:head# 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457a4b1f:::benchmark_data_smithi194_103039_object128248:head# 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457a57ef:::benchmark_data_smithi194_103039_object116615:head# 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457a7970:::benchmark_data_smithi194_103039_object42250:head# 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457ab536:::benchmark_data_smithi194_103039_object92879:head# 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457b64b0:::benchmark_data_smithi194_103039_object109188:head# 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457b7bc3:::benchmark_data_smithi194_103039_object103218:head# 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457c820f:::benchmark_data_smithi194_103039_object59659:head# 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457d1cde:::benchmark_data_smithi194_103039_object44637:head# 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457da233:::benchmark_data_smithi194_103039_object97391:head# 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457f4fda:::benchmark_data_smithi194_103039_object78860:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457f8127:::benchmark_data_smithi194_103039_object68461:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:457fec21:::benchmark_data_smithi194_103039_object96940:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4580b81c:::benchmark_data_smithi194_103039_object55618:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4580b892:::benchmark_data_smithi194_103039_object55036:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4580e4ee:::benchmark_data_smithi194_103039_object63250:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4580f361:::benchmark_data_smithi194_103039_object49389:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45832db5:::benchmark_data_smithi194_103039_object130202:head# 2024-04-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4583400b:::benchmark_data_smithi194_103039_object83647:head# 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4583c115:::benchmark_data_smithi194_103039_object153213:head# 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458404fd:::benchmark_data_smithi194_103039_object80457:head# 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4584fb84:::benchmark_data_smithi194_103039_object105222:head# 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458578dd:::benchmark_data_smithi194_103039_object97951:head# 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4585c3d8:::benchmark_data_smithi194_103039_object72784:head# 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4586acbd:::benchmark_data_smithi194_103039_object43557:head# 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45872520:::benchmark_data_smithi194_103039_object68682:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458735cc:::benchmark_data_smithi194_103039_object86034:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45873df2:::benchmark_data_smithi194_103039_object105715:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45877486:::benchmark_data_smithi194_103039_object66319:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45883ad3:::benchmark_data_smithi194_103039_object43620:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45893df8:::benchmark_data_smithi194_103039_object134442:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4589fcd9:::benchmark_data_smithi194_103039_object62581:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458b23b7:::benchmark_data_smithi194_103039_object122547:head# 2024-04-29T14:21:29.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458c159d:::benchmark_data_smithi194_103039_object86939:head# 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458da5c4:::benchmark_data_smithi194_103039_object153201:head# 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458e1d97:::benchmark_data_smithi194_103039_object69927:head# 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:458ea978:::benchmark_data_smithi194_103039_object63853:head# 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459091fa:::benchmark_data_smithi194_103039_object78784:head# 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4590c523:::benchmark_data_smithi194_103039_object55702:head# 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45917735:::benchmark_data_smithi194_103039_object84867:head# 2024-04-29T14:21:29.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4591cfab:::benchmark_data_smithi194_103039_object50671:head# 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4592f198:::benchmark_data_smithi194_103039_object153676:head# 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4593a300:::benchmark_data_smithi194_103039_object71908:head# 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45947b21:::benchmark_data_smithi194_103039_object45353:head# 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4595509b:::benchmark_data_smithi194_103039_object103930:head# 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45955675:::benchmark_data_smithi194_103039_object79323:head# 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4595d2bb:::benchmark_data_smithi194_103039_object91566:head# 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459643ae:::benchmark_data_smithi194_103039_object52153:head# 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45966348:::benchmark_data_smithi194_103039_object82444:head# 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45966d91:::benchmark_data_smithi194_103039_object115328:head# 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4596867f:::benchmark_data_smithi194_103039_object108416:head# 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4597a3e0:::benchmark_data_smithi194_103039_object147486:head# 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4599b82d:::benchmark_data_smithi194_103039_object123726:head# 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4599f1ec:::benchmark_data_smithi194_103039_object123120:head# 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459a5566:::benchmark_data_smithi194_103039_object75041:head# 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459ad71d:::benchmark_data_smithi194_103039_object55373:head# 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459b2dac:::benchmark_data_smithi194_103039_object114554:head# 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459bac8b:::benchmark_data_smithi194_103039_object72592:head# 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459d97c0:::benchmark_data_smithi194_103039_object63115:head# 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459f0aa3:::benchmark_data_smithi194_103039_object143260:head# 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459f4076:::benchmark_data_smithi194_103039_object42179:head# 2024-04-29T14:21:29.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:459f98a3:::benchmark_data_smithi194_103039_object103858:head# 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a0b83b:::benchmark_data_smithi194_103039_object62663:head# 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a0b9df:::benchmark_data_smithi194_103039_object77997:head# 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a0cd69:::benchmark_data_smithi194_103039_object122130:head# 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a20053:::benchmark_data_smithi194_103039_object125642:head# 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a2d879:::benchmark_data_smithi194_103039_object149765:head# 2024-04-29T14:21:29.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a46cb1:::benchmark_data_smithi194_103039_object49721:head# 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a47831:::benchmark_data_smithi194_103039_object43684:head# 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a4817c:::benchmark_data_smithi194_103039_object84379:head# 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a4de15:::benchmark_data_smithi194_103039_object77041:head# 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a506f1:::benchmark_data_smithi194_103039_object96211:head# 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a62120:::benchmark_data_smithi194_103039_object147877:head# 2024-04-29T14:21:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a66ace:::benchmark_data_smithi194_103039_object95671:head# 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a73dc4:::benchmark_data_smithi194_103039_object57769:head# 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a7bce6:::benchmark_data_smithi194_103039_object72109:head# 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a8e1d2:::benchmark_data_smithi194_103039_object112371:head# 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a92480:::benchmark_data_smithi194_103039_object150415:head# 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45a9b087:::benchmark_data_smithi194_103039_object49446:head# 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45aa2e08:::benchmark_data_smithi194_103039_object63485:head# 2024-04-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ad027b:::benchmark_data_smithi194_103039_object152105:head# 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45adb979:::benchmark_data_smithi194_103039_object54915:head# 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45add23a:::benchmark_data_smithi194_103039_object87952:head# 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ae5c75:::benchmark_data_smithi194_103039_object114467:head# 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b010c8:::benchmark_data_smithi194_103039_object144312:head# 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b02033:::benchmark_data_smithi194_103039_object110928:head# 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b02b78:::benchmark_data_smithi194_103039_object141367:head# 2024-04-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b1a44e:::benchmark_data_smithi194_103039_object113469:head# 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b1fd0b:::benchmark_data_smithi194_103039_object84946:head# 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b33444:::benchmark_data_smithi194_103039_object41619:head# 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b4286d:::benchmark_data_smithi194_103039_object122242:head# 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b49f04:::benchmark_data_smithi194_103039_object142486:head# 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b4fea2:::benchmark_data_smithi194_103039_object55406:head# 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b55b2d:::benchmark_data_smithi194_103039_object107263:head# 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b58bb6:::benchmark_data_smithi194_103039_object119721:head# 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b5e620:::benchmark_data_smithi194_103039_object88981:head# 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b758cb:::benchmark_data_smithi194_103039_object62339:head# 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b86591:::benchmark_data_smithi194_103039_object67117:head# 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45b979b4:::benchmark_data_smithi194_103039_object63826:head# 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45bec464:::benchmark_data_smithi194_103039_object53791:head# 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45bf2574:::benchmark_data_smithi194_103039_object83198:head# 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45bf6fc5:::benchmark_data_smithi194_103039_object91482:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c01380:::benchmark_data_smithi194_103039_object88941:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c05499:::benchmark_data_smithi194_103039_object88474:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c11e0c:::benchmark_data_smithi194_103039_object155561:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c14ea7:::benchmark_data_smithi194_103039_object152714:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c1775b:::benchmark_data_smithi194_103039_object142744:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c1a790:::benchmark_data_smithi194_103039_object116989:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c28c74:::benchmark_data_smithi194_103039_object68679:head# 2024-04-29T14:21:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c46549:::benchmark_data_smithi194_103039_object112699:head# 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c5372b:::benchmark_data_smithi194_103039_object83654:head# 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c5675b:::benchmark_data_smithi194_103039_object98914:head# 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c64325:::benchmark_data_smithi194_103039_object45450:head# 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c7e6a2:::benchmark_data_smithi194_103039_object103602:head# 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c88762:::benchmark_data_smithi194_103039_object134926:head# 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c8a6c7:::benchmark_data_smithi194_103039_object75590:head# 2024-04-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c913aa:::benchmark_data_smithi194_103039_object65874:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c93692:::benchmark_data_smithi194_103039_object85155:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45c9d1a7:::benchmark_data_smithi194_103039_object94688:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ca2a6c:::benchmark_data_smithi194_103039_object137108:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ca7968:::benchmark_data_smithi194_103039_object152364:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45cbe818:::benchmark_data_smithi194_103039_object100422:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45cca9c2:::benchmark_data_smithi194_103039_object95719:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45cd495d:::benchmark_data_smithi194_103039_object119771:head# 2024-04-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45cd92aa:::benchmark_data_smithi194_103039_object55145:head# 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45cdb9f7:::benchmark_data_smithi194_103039_object121566:head# 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ce65a8:::benchmark_data_smithi194_103039_object61853:head# 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ce74cb:::benchmark_data_smithi194_103039_object104492:head# 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45cf01f5:::benchmark_data_smithi194_103039_object77590:head# 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45cf23fe:::benchmark_data_smithi194_103039_object63867:head# 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d0c155:::benchmark_data_smithi194_103039_object103003:head# 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d111b9:::benchmark_data_smithi194_103039_object109940:head# 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d42b47:::benchmark_data_smithi194_103039_object146496:head# 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d56a5a:::benchmark_data_smithi194_103039_object99837:head# 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d647ff:::benchmark_data_smithi194_103039_object148107:head# 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d6e811:::benchmark_data_smithi194_103039_object129210:head# 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d76723:::benchmark_data_smithi194_103039_object137220:head# 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d7db29:::benchmark_data_smithi194_103039_object153235:head# 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45d8f62c:::benchmark_data_smithi194_103039_object44718:head# 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45da0205:::benchmark_data_smithi194_103039_object76843:head# 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45da518c:::benchmark_data_smithi194_103039_object48963:head# 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45da6657:::benchmark_data_smithi194_103039_object114077:head# 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45db1335:::benchmark_data_smithi194_103039_object111185:head# 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45dbbdde:::benchmark_data_smithi194_103039_object48187:head# 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45dc3cdb:::benchmark_data_smithi194_103039_object49388:head# 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45dcd8b7:::benchmark_data_smithi194_103039_object93576:head# 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45dda427:::benchmark_data_smithi194_103039_object57629:head# 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ddcf62:::benchmark_data_smithi194_103039_object100121:head# 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ddedea:::benchmark_data_smithi194_103039_object95969:head# 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45dff95e:::benchmark_data_smithi194_103039_object112419:head# 2024-04-29T14:21:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e010b1:::benchmark_data_smithi194_103039_object130278:head# 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e02af6:::benchmark_data_smithi194_103039_object110386:head# 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e0342f:::benchmark_data_smithi194_103039_object91120:head# 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e0ba81:::benchmark_data_smithi194_103039_object91102:head# 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e100c4:::benchmark_data_smithi194_103039_object132898:head# 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e1d7cb:::benchmark_data_smithi194_103039_object103920:head# 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e20b08:::benchmark_data_smithi194_103039_object76240:head# 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e2103f:::benchmark_data_smithi194_103039_object118804:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e2661a:::benchmark_data_smithi194_103039_object82330:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e2bdcf:::benchmark_data_smithi194_103039_object69195:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e339b3:::benchmark_data_smithi194_103039_object148020:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e33be8:::benchmark_data_smithi194_103039_object70184:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e3ae20:::benchmark_data_smithi194_103039_object73217:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e4d802:::benchmark_data_smithi194_103039_object141226:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e52179:::benchmark_data_smithi194_103039_object65149:head# 2024-04-29T14:21:29.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e52a15:::benchmark_data_smithi194_103039_object152476:head# 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e54a64:::benchmark_data_smithi194_103039_object69103:head# 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e65a10:::benchmark_data_smithi194_103039_object152166:head# 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e69e31:::benchmark_data_smithi194_103039_object143263:head# 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45e93a8a:::benchmark_data_smithi194_103039_object99240:head# 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ea9768:::benchmark_data_smithi194_103039_object51973:head# 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45eba889:::benchmark_data_smithi194_103039_object57358:head# 2024-04-29T14:21:29.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ec595c:::benchmark_data_smithi194_103039_object119472:head# 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ed2356:::benchmark_data_smithi194_103039_object59776:head# 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ee1efe:::benchmark_data_smithi194_103039_object97247:head# 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ee4286:::benchmark_data_smithi194_103039_object104863:head# 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45eecf48:::benchmark_data_smithi194_103039_object73970:head# 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45eed05b:::benchmark_data_smithi194_103039_object84646:head# 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ef6d0c:::benchmark_data_smithi194_103039_object97664:head# 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ef7a19:::benchmark_data_smithi194_103039_object88191:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ef9be8:::benchmark_data_smithi194_103039_object91409:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f0ea24:::benchmark_data_smithi194_103039_object114995:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f13043:::benchmark_data_smithi194_103039_object155410:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f15698:::benchmark_data_smithi194_103039_object54657:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f17329:::benchmark_data_smithi194_103039_object115059:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f29839:::benchmark_data_smithi194_103039_object114451:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f2b321:::benchmark_data_smithi194_103039_object113031:head# 2024-04-29T14:21:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f2b915:::benchmark_data_smithi194_103039_object147299:head# 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f35939:::benchmark_data_smithi194_103039_object80893:head# 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f4a6e5:::benchmark_data_smithi194_103039_object110575:head# 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f4be86:::benchmark_data_smithi194_103039_object103108:head# 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f5ab4a:::benchmark_data_smithi194_103039_object88829:head# 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f5dd4e:::benchmark_data_smithi194_103039_object75343:head# 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f5fb18:::benchmark_data_smithi194_103039_object87956:head# 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f6b228:::benchmark_data_smithi194_103039_object108350:head# 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f72a21:::benchmark_data_smithi194_103039_object101864:head# 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f7ba43:::benchmark_data_smithi194_103039_object143854:head# 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f7e2ec:::benchmark_data_smithi194_103039_object58326:head# 2024-04-29T14:21:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f881c6:::benchmark_data_smithi194_103039_object61231:head# 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f93956:::benchmark_data_smithi194_103039_object91040:head# 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f953a1:::benchmark_data_smithi194_103039_object92091:head# 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f986ee:::benchmark_data_smithi194_103039_object106562:head# 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f9a351:::benchmark_data_smithi194_103039_object142650:head# 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45f9b5f2:::benchmark_data_smithi194_103039_object139663:head# 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45faaabc:::benchmark_data_smithi194_103039_object74029:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45fac750:::benchmark_data_smithi194_103039_object42016:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45fb3bb9:::benchmark_data_smithi194_103039_object109335:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45fb529c:::benchmark_data_smithi194_103039_object91629:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45fc96bf:::benchmark_data_smithi194_103039_object66015:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45fd2fb7:::benchmark_data_smithi194_103039_object136395:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ff703d:::benchmark_data_smithi194_103039_object146134:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ff89a7:::benchmark_data_smithi194_103039_object154147:head# 2024-04-29T14:21:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ffba9d:::benchmark_data_smithi194_103039_object96047:head# 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:45ffdb19:::benchmark_data_smithi194_103039_object67031:head# 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460064ff:::benchmark_data_smithi194_103039_object70626:head# 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4600faa9:::benchmark_data_smithi194_103039_object101135:head# 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460114dd:::benchmark_data_smithi194_103039_object82963:head# 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4601291a:::benchmark_data_smithi194_103039_object57303:head# 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460191fa:::benchmark_data_smithi194_103039_object104890:head# 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4601fdcc:::benchmark_data_smithi194_103039_object98125:head# 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460247cd:::benchmark_data_smithi194_103039_object129126:head# 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46028d32:::benchmark_data_smithi194_103039_object73547:head# 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4602dc81:::benchmark_data_smithi194_103039_object147923:head# 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460322de:::benchmark_data_smithi194_103039_object93533:head# 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46039fe8:::benchmark_data_smithi194_103039_object138435:head# 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4603b3b6:::benchmark_data_smithi194_103039_object46554:head# 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4603b55f:::benchmark_data_smithi194_103039_object143999:head# 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4604465f:::benchmark_data_smithi194_103039_object91291:head# 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460458fc:::benchmark_data_smithi194_103039_object124908:head# 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46054caf:::benchmark_data_smithi194_103039_object135773:head# 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46055fbe:::benchmark_data_smithi194_103039_object135161:head# 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46059601:::benchmark_data_smithi194_103039_object142091:head# 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4605d9cc:::benchmark_data_smithi194_103039_object76157:head# 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4606a60e:::benchmark_data_smithi194_103039_object147177:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46078325:::benchmark_data_smithi194_103039_object61690:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46083553:::benchmark_data_smithi194_103039_object114718:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46083a92:::benchmark_data_smithi194_103039_object92021:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46087280:::benchmark_data_smithi194_103039_object94121:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460913ae:::benchmark_data_smithi194_103039_object107842:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460922b5:::benchmark_data_smithi194_103039_object150438:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4609ac64:::benchmark_data_smithi194_103039_object137362:head# 2024-04-29T14:21:29.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4609d7dc:::benchmark_data_smithi194_103039_object127776:head# 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460a579f:::benchmark_data_smithi194_103039_object90346:head# 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460a8cea:::benchmark_data_smithi194_103039_object103258:head# 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460a9e60:::benchmark_data_smithi194_103039_object131236:head# 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460b6643:::benchmark_data_smithi194_103039_object46285:head# 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460ca2c3:::benchmark_data_smithi194_103039_object146906:head# 2024-04-29T14:21:29.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460d1d91:::benchmark_data_smithi194_103039_object118342:head# 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460e3625:::benchmark_data_smithi194_103039_object50255:head# 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460e4a7e:::benchmark_data_smithi194_103039_object69189:head# 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460efb10:::benchmark_data_smithi194_103039_object58711:head# 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460f2de0:::benchmark_data_smithi194_103039_object53388:head# 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460f81d4:::benchmark_data_smithi194_103039_object46155:head# 2024-04-29T14:21:29.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:460fa0db:::benchmark_data_smithi194_103039_object82641:head# 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4611d102:::benchmark_data_smithi194_103039_object146687:head# 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46120ebb:::benchmark_data_smithi194_103039_object69133:head# 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4612be74:::benchmark_data_smithi194_103039_object121197:head# 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4612df81:::benchmark_data_smithi194_103039_object76339:head# 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4613971c:::benchmark_data_smithi194_103039_object144218:head# 2024-04-29T14:21:29.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4613aa9f:::benchmark_data_smithi194_103039_object132750:head# 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4613bd3f:::benchmark_data_smithi194_103039_object85455:head# 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4613d511:::benchmark_data_smithi194_103039_object155301:head# 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4614231a:::benchmark_data_smithi194_103039_object80339:head# 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4614e4d5:::benchmark_data_smithi194_103039_object110310:head# 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4615a591:::benchmark_data_smithi194_103039_object106801:head# 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4615da38:::benchmark_data_smithi194_103039_object53552:head# 2024-04-29T14:21:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4617913a:::benchmark_data_smithi194_103039_object116734:head# 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4617f4d0:::benchmark_data_smithi194_103039_object48924:head# 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461850b8:::benchmark_data_smithi194_103039_object149640:head# 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461876e0:::benchmark_data_smithi194_103039_object79222:head# 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461928a6:::benchmark_data_smithi194_103039_object76971:head# 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46193a58:::benchmark_data_smithi194_103039_object133378:head# 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46194a22:::benchmark_data_smithi194_103039_object84736:head# 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46197335:::benchmark_data_smithi194_103039_object47061:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461a59f9:::benchmark_data_smithi194_103039_object116259:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461adea6:::benchmark_data_smithi194_103039_object92752:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461af6f2:::benchmark_data_smithi194_103039_object66326:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461b2146:::benchmark_data_smithi194_103039_object65333:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461babeb:::benchmark_data_smithi194_103039_object58978:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461c0528:::benchmark_data_smithi194_103039_object109284:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461c4e41:::benchmark_data_smithi194_103039_object134405:head# 2024-04-29T14:21:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461c51a9:::benchmark_data_smithi194_103039_object122362:head# 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461e39f1:::benchmark_data_smithi194_103039_object116446:head# 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461ee149:::benchmark_data_smithi194_103039_object79327:head# 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461faaf0:::benchmark_data_smithi194_103039_object131833:head# 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461fac10:::benchmark_data_smithi194_103039_object153381:head# 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461fc856:::benchmark_data_smithi194_103039_object116388:head# 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:461ff738:::benchmark_data_smithi194_103039_object107311:head# 2024-04-29T14:21:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46205b44:::benchmark_data_smithi194_103039_object54505:head# 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4620e42a:::benchmark_data_smithi194_103039_object145319:head# 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462321f6:::benchmark_data_smithi194_103039_object149873:head# 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46243cbf:::benchmark_data_smithi194_103039_object73595:head# 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4624424f:::benchmark_data_smithi194_103039_object143959:head# 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4624cc30:::benchmark_data_smithi194_103039_object131808:head# 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46252ea4:::benchmark_data_smithi194_103039_object53060:head# 2024-04-29T14:21:29.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4625ab7f:::benchmark_data_smithi194_103039_object107757:head# 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4625bbc6:::benchmark_data_smithi194_103039_object106659:head# 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46265a7e:::benchmark_data_smithi194_103039_object140642:head# 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4626670b:::benchmark_data_smithi194_103039_object59639:head# 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46279c4c:::benchmark_data_smithi194_103039_object108974:head# 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46280f80:::benchmark_data_smithi194_103039_object67716:head# 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462835bf:::benchmark_data_smithi194_103039_object107651:head# 2024-04-29T14:21:29.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46287429:::benchmark_data_smithi194_103039_object90356:head# 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4629a1b2:::benchmark_data_smithi194_103039_object109018:head# 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462b358d:::benchmark_data_smithi194_103039_object55250:head# 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462c1674:::benchmark_data_smithi194_103039_object124472:head# 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462cbbbd:::benchmark_data_smithi194_103039_object134177:head# 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462ebadf:::benchmark_data_smithi194_103039_object73543:head# 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462ef89f:::benchmark_data_smithi194_103039_object112456:head# 2024-04-29T14:21:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462f383e:::benchmark_data_smithi194_103039_object85058:head# 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462f40c0:::benchmark_data_smithi194_103039_object81446:head# 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462fab42:::benchmark_data_smithi194_103039_object99604:head# 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:462fb11a:::benchmark_data_smithi194_103039_object98503:head# 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46304907:::benchmark_data_smithi194_103039_object138696:head# 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4630b8e4:::benchmark_data_smithi194_103039_object117233:head# 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4630d1a3:::benchmark_data_smithi194_103039_object111263:head# 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4631d1cf:::benchmark_data_smithi194_103039_object56603:head# 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4632f51c:::benchmark_data_smithi194_103039_object120751:head# 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4633c39c:::benchmark_data_smithi194_103039_object108958:head# 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463496cd:::benchmark_data_smithi194_103039_object59591:head# 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4634a4d7:::benchmark_data_smithi194_103039_object59013:head# 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4634aa43:::benchmark_data_smithi194_103039_object105632:head# 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4634bad7:::benchmark_data_smithi194_103039_object153447:head# 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46377dd6:::benchmark_data_smithi194_103039_object136435:head# 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46381a2d:::benchmark_data_smithi194_103039_object68868:head# 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4638644e:::benchmark_data_smithi194_103039_object147508:head# 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4638716a:::benchmark_data_smithi194_103039_object102292:head# 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4638f814:::benchmark_data_smithi194_103039_object88162:head# 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46396256:::benchmark_data_smithi194_103039_object71148:head# 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4639e54c:::benchmark_data_smithi194_103039_object129936:head# 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463a0dcd:::benchmark_data_smithi194_103039_object69148:head# 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463a1257:::benchmark_data_smithi194_103039_object73993:head# 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463b1201:::benchmark_data_smithi194_103039_object106268:head# 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463b2b7d:::benchmark_data_smithi194_103039_object113624:head# 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463be365:::benchmark_data_smithi194_103039_object133367:head# 2024-04-29T14:21:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463c4cf9:::benchmark_data_smithi194_103039_object139828:head# 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463d7794:::benchmark_data_smithi194_103039_object134663:head# 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463d8204:::benchmark_data_smithi194_103039_object118163:head# 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463dfd68:::benchmark_data_smithi194_103039_object133674:head# 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463e5dd3:::benchmark_data_smithi194_103039_object110020:head# 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463f2c09:::benchmark_data_smithi194_103039_object156485:head# 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:463fee02:::benchmark_data_smithi194_103039_object152195:head# 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464042d5:::benchmark_data_smithi194_103039_object152944:head# 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464095d4:::benchmark_data_smithi194_103039_object126229:head# 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46409677:::benchmark_data_smithi194_103039_object130225:head# 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4640f2ec:::benchmark_data_smithi194_103039_object105634:head# 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4642b339:::benchmark_data_smithi194_103039_object132044:head# 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46433135:::benchmark_data_smithi194_103039_object64418:head# 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46439698:::benchmark_data_smithi194_103039_object143148:head# 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464420e7:::benchmark_data_smithi194_103039_object106451:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46451ba7:::benchmark_data_smithi194_103039_object73184:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464568aa:::benchmark_data_smithi194_103039_object41806:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46457e3d:::benchmark_data_smithi194_103039_object47328:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46468a2a:::benchmark_data_smithi194_103039_object60935:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464772bb:::benchmark_data_smithi194_103039_object116572:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4647bda5:::benchmark_data_smithi194_103039_object120065:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46499a15:::benchmark_data_smithi194_103039_object99609:head# 2024-04-29T14:21:29.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4649c642:::benchmark_data_smithi194_103039_object123284:head# 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4649cf71:::benchmark_data_smithi194_103039_object156498:head# 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464ae407:::benchmark_data_smithi194_103039_object119476:head# 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464b3e6c:::benchmark_data_smithi194_103039_object104923:head# 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464b5bf0:::benchmark_data_smithi194_103039_object60221:head# 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464ca905:::benchmark_data_smithi194_103039_object44402:head# 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464d0980:::benchmark_data_smithi194_103039_object62360:head# 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464d8041:::benchmark_data_smithi194_103039_object104073:head# 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464e5cd3:::benchmark_data_smithi194_103039_object155625:head# 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464e5dae:::benchmark_data_smithi194_103039_object120465:head# 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464f3964:::benchmark_data_smithi194_103039_object68971:head# 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464fc9a0:::benchmark_data_smithi194_103039_object63319:head# 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:464fcc95:::benchmark_data_smithi194_103039_object41882:head# 2024-04-29T14:21:29.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46503bfc:::benchmark_data_smithi194_103039_object147139:head# 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46507c2c:::benchmark_data_smithi194_103039_object124825:head# 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46509c79:::benchmark_data_smithi194_103039_object73892:head# 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46512e4b:::benchmark_data_smithi194_103039_object129623:head# 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4651d4a5:::benchmark_data_smithi194_103039_object98339:head# 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46523693:::benchmark_data_smithi194_103039_object151755:head# 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465273a4:::benchmark_data_smithi194_103039_object53821:head# 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46532530:::benchmark_data_smithi194_103039_object54373:head# 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46533e47:::benchmark_data_smithi194_103039_object150359:head# 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465431e9:::benchmark_data_smithi194_103039_object80078:head# 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46546729:::benchmark_data_smithi194_103039_object80045:head# 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4654aaf7:::benchmark_data_smithi194_103039_object55201:head# 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4655e6ff:::benchmark_data_smithi194_103039_object107682:head# 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46565cad:::benchmark_data_smithi194_103039_object67288:head# 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46566f40:::benchmark_data_smithi194_103039_object112155:head# 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4656a6ce:::benchmark_data_smithi194_103039_object70154:head# 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465708ae:::benchmark_data_smithi194_103039_object98451:head# 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46573079:::benchmark_data_smithi194_103039_object51292:head# 2024-04-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465817ce:::benchmark_data_smithi194_103039_object52941:head# 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46586550:::benchmark_data_smithi194_103039_object86481:head# 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4659346c:::benchmark_data_smithi194_103039_object60825:head# 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4659f7c3:::benchmark_data_smithi194_103039_object85600:head# 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465a73b3:::benchmark_data_smithi194_103039_object41847:head# 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465a8e2f:::benchmark_data_smithi194_103039_object125620:head# 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465ba320:::benchmark_data_smithi194_103039_object49577:head# 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465c6f0a:::benchmark_data_smithi194_103039_object109874:head# 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465d8ddb:::benchmark_data_smithi194_103039_object83419:head# 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465d939f:::benchmark_data_smithi194_103039_object126745:head# 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465dbf54:::benchmark_data_smithi194_103039_object135390:head# 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465dfa2b:::benchmark_data_smithi194_103039_object155105:head# 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465e4ee3:::benchmark_data_smithi194_103039_object45563:head# 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465e85e6:::benchmark_data_smithi194_103039_object80135:head# 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465e979e:::benchmark_data_smithi194_103039_object59783:head# 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465ede58:::benchmark_data_smithi194_103039_object155075:head# 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465ef60e:::benchmark_data_smithi194_103039_object95657:head# 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465f6b8f:::benchmark_data_smithi194_103039_object103420:head# 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:465fbf39:::benchmark_data_smithi194_103039_object129782:head# 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46604b15:::benchmark_data_smithi194_103039_object146172:head# 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4660739a:::benchmark_data_smithi194_103039_object67758:head# 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466075fc:::benchmark_data_smithi194_103039_object109709:head# 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4664279a:::benchmark_data_smithi194_103039_object118604:head# 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46643e6e:::benchmark_data_smithi194_103039_object114243:head# 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46645049:::benchmark_data_smithi194_103039_object42544:head# 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4665c46f:::benchmark_data_smithi194_103039_object115997:head# 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46664d28:::benchmark_data_smithi194_103039_object150526:head# 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46664dae:::benchmark_data_smithi194_103039_object82663:head# 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4667b317:::benchmark_data_smithi194_103039_object76289:head# 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4667cfbf:::benchmark_data_smithi194_103039_object152000:head# 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46680af5:::benchmark_data_smithi194_103039_object123705:head# 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4668e461:::benchmark_data_smithi194_103039_object95267:head# 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4669b92c:::benchmark_data_smithi194_103039_object52671:head# 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4669eeac:::benchmark_data_smithi194_103039_object133721:head# 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466ab4b7:::benchmark_data_smithi194_103039_object47661:head# 2024-04-29T14:21:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466b5533:::benchmark_data_smithi194_103039_object151564:head# 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466bb3d1:::benchmark_data_smithi194_103039_object56111:head# 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466d3791:::benchmark_data_smithi194_103039_object113968:head# 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466e3ef9:::benchmark_data_smithi194_103039_object71810:head# 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466e6749:::benchmark_data_smithi194_103039_object117854:head# 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:466fae79:::benchmark_data_smithi194_103039_object86375:head# 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46705c02:::benchmark_data_smithi194_103039_object85201:head# 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4670f98c:::benchmark_data_smithi194_103039_object45945:head# 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4671d7b2:::benchmark_data_smithi194_103039_object64841:head# 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467327aa:::benchmark_data_smithi194_103039_object49702:head# 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46739164:::benchmark_data_smithi194_103039_object49416:head# 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4673e8de:::benchmark_data_smithi194_103039_object154477:head# 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46752811:::benchmark_data_smithi194_103039_object145513:head# 2024-04-29T14:21:29.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46760598:::benchmark_data_smithi194_103039_object71512:head# 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46761d3b:::benchmark_data_smithi194_103039_object121553:head# 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46762e78:::benchmark_data_smithi194_103039_object68260:head# 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46764a99:::benchmark_data_smithi194_103039_object125899:head# 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4677032c:::benchmark_data_smithi194_103039_object105829:head# 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467728ab:::benchmark_data_smithi194_103039_object63239:head# 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467841de:::benchmark_data_smithi194_103039_object137662:head# 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4678c393:::benchmark_data_smithi194_103039_object91973:head# 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4678c5b5:::benchmark_data_smithi194_103039_object86843:head# 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46797a51:::benchmark_data_smithi194_103039_object50661:head# 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4679b482:::benchmark_data_smithi194_103039_object52870:head# 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467a658b:::benchmark_data_smithi194_103039_object60567:head# 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467acf94:::benchmark_data_smithi194_103039_object130282:head# 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467aede1:::benchmark_data_smithi194_103039_object47044:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467d5076:::benchmark_data_smithi194_103039_object46911:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467dd57a:::benchmark_data_smithi194_103039_object128372:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:467f7a43:::benchmark_data_smithi194_103039_object93508:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4680b925:::benchmark_data_smithi194_103039_object101763:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4680d7ca:::benchmark_data_smithi194_103039_object112716:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4680d915:::benchmark_data_smithi194_103039_object151556:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4681cb51:::benchmark_data_smithi194_103039_object70622:head# 2024-04-29T14:21:29.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4684530e:::benchmark_data_smithi194_103039_object145071:head# 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4684bf33:::benchmark_data_smithi194_103039_object53305:head# 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46853aca:::benchmark_data_smithi194_103039_object65398:head# 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46871095:::benchmark_data_smithi194_103039_object56734:head# 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4687733e:::benchmark_data_smithi194_103039_object122286:head# 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46877fcd:::benchmark_data_smithi194_103039_object42892:head# 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4687a226:::benchmark_data_smithi194_103039_object95938:head# 2024-04-29T14:21:29.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4687cc89:::benchmark_data_smithi194_103039_object50258:head# 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4687cf69:::benchmark_data_smithi194_103039_object94684:head# 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468823eb:::benchmark_data_smithi194_103039_object108590:head# 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46888b0c:::benchmark_data_smithi194_103039_object90158:head# 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4688f6e9:::benchmark_data_smithi194_103039_object80497:head# 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468949d8:::benchmark_data_smithi194_103039_object100534:head# 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468b737b:::benchmark_data_smithi194_103039_object89737:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468c0beb:::benchmark_data_smithi194_103039_object73765:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468c0d59:::benchmark_data_smithi194_103039_object58785:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468c7fee:::benchmark_data_smithi194_103039_object63631:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468ce149:::benchmark_data_smithi194_103039_object111238:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468d3904:::benchmark_data_smithi194_103039_object62936:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468dadff:::benchmark_data_smithi194_103039_object150952:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468ea216:::benchmark_data_smithi194_103039_object122148:head# 2024-04-29T14:21:29.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468ed924:::benchmark_data_smithi194_103039_object49160:head# 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468f8b29:::benchmark_data_smithi194_103039_object148710:head# 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:468fbbf7:::benchmark_data_smithi194_103039_object60173:head# 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469081d1:::benchmark_data_smithi194_103039_object42681:head# 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46910dec:::benchmark_data_smithi194_103039_object107271:head# 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4691738c:::benchmark_data_smithi194_103039_object148610:head# 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46918e31:::benchmark_data_smithi194_103039_object60078:head# 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46919ae5:::benchmark_data_smithi194_103039_object100598:head# 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4691f856:::benchmark_data_smithi194_103039_object144330:head# 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46921ff0:::benchmark_data_smithi194_103039_object103010:head# 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4692222e:::benchmark_data_smithi194_103039_object72508:head# 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46932126:::benchmark_data_smithi194_103039_object110087:head# 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469422b9:::benchmark_data_smithi194_103039_object85847:head# 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4694b0ed:::benchmark_data_smithi194_103039_object94598:head# 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4694b406:::benchmark_data_smithi194_103039_object46699:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46952b43:::benchmark_data_smithi194_103039_object119849:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46953c7b:::benchmark_data_smithi194_103039_object120945:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46959957:::benchmark_data_smithi194_103039_object90482:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4695f0b6:::benchmark_data_smithi194_103039_object92313:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46960da8:::benchmark_data_smithi194_103039_object127595:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4696585b:::benchmark_data_smithi194_103039_object51178:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46974381:::benchmark_data_smithi194_103039_object68262:head# 2024-04-29T14:21:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4697891b:::benchmark_data_smithi194_103039_object109149:head# 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4697db71:::benchmark_data_smithi194_103039_object114275:head# 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4697ef10:::benchmark_data_smithi194_103039_object130479:head# 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46985ee3:::benchmark_data_smithi194_103039_object55554:head# 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46989d00:::benchmark_data_smithi194_103039_object137762:head# 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469a826a:::benchmark_data_smithi194_103039_object90423:head# 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469bed4b:::benchmark_data_smithi194_103039_object51483:head# 2024-04-29T14:21:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469d00a6:::benchmark_data_smithi194_103039_object62150:head# 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469d8dbf:::benchmark_data_smithi194_103039_object97246:head# 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469daf66:::benchmark_data_smithi194_103039_object138629:head# 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469dc6da:::benchmark_data_smithi194_103039_object48003:head# 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469e298e:::benchmark_data_smithi194_103039_object118290:head# 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469e4c5d:::benchmark_data_smithi194_103039_object125853:head# 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469e8ea6:::benchmark_data_smithi194_103039_object109746:head# 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:469f8652:::benchmark_data_smithi194_103039_object67910:head# 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a05530:::benchmark_data_smithi194_103039_object79260:head# 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a0eb7c:::benchmark_data_smithi194_103039_object100641:head# 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a10516:::benchmark_data_smithi194_103039_object110768:head# 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a13a77:::benchmark_data_smithi194_103039_object97773:head# 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a1461b:::benchmark_data_smithi194_103039_object116769:head# 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a1a173:::benchmark_data_smithi194_103039_object44407:head# 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a1e712:::benchmark_data_smithi194_103039_object58455:head# 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a1fc02:::benchmark_data_smithi194_103039_object82609:head# 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a38472:::benchmark_data_smithi194_103039_object89051:head# 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a5594d:::benchmark_data_smithi194_103039_object117627:head# 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a5cef4:::benchmark_data_smithi194_103039_object65772:head# 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a5f4fb:::benchmark_data_smithi194_103039_object126603:head# 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a709ec:::benchmark_data_smithi194_103039_object70655:head# 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a8edb4:::benchmark_data_smithi194_103039_object151166:head# 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a90300:::benchmark_data_smithi194_103039_object94750:head# 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46a959ad:::benchmark_data_smithi194_103039_object152197:head# 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46aa0dbe:::benchmark_data_smithi194_103039_object133642:head# 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ac28cd:::benchmark_data_smithi194_103039_object153596:head# 2024-04-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ac34bb:::benchmark_data_smithi194_103039_object114663:head# 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ac7ee9:::benchmark_data_smithi194_103039_object139443:head# 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ad9126:::benchmark_data_smithi194_103039_object51986:head# 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ad934f:::benchmark_data_smithi194_103039_object66564:head# 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46adc748:::benchmark_data_smithi194_103039_object102707:head# 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46adfd04:::benchmark_data_smithi194_103039_object42728:head# 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ae0ad8:::benchmark_data_smithi194_103039_object135985:head# 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ae11c9:::benchmark_data_smithi194_103039_object96215:head# 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ae5b0e:::benchmark_data_smithi194_103039_object42019:head# 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46af1889:::benchmark_data_smithi194_103039_object98471:head# 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46af2d66:::benchmark_data_smithi194_103039_object145039:head# 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b00ef6:::benchmark_data_smithi194_103039_object43852:head# 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b05e5c:::benchmark_data_smithi194_103039_object59159:head# 2024-04-29T14:21:29.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b06472:::benchmark_data_smithi194_103039_object79922:head# 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b15341:::benchmark_data_smithi194_103039_object42219:head# 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b17416:::benchmark_data_smithi194_103039_object129769:head# 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b18d3e:::benchmark_data_smithi194_103039_object132747:head# 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b1cd44:::benchmark_data_smithi194_103039_object154104:head# 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b2978c:::benchmark_data_smithi194_103039_object154445:head# 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b3811f:::benchmark_data_smithi194_103039_object108784:head# 2024-04-29T14:21:29.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b41109:::benchmark_data_smithi194_103039_object125443:head# 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b47b0c:::benchmark_data_smithi194_103039_object53484:head# 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b4ddee:::benchmark_data_smithi194_103039_object41748:head# 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b595f1:::benchmark_data_smithi194_103039_object135810:head# 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b5c5f3:::benchmark_data_smithi194_103039_object102431:head# 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b659b8:::benchmark_data_smithi194_103039_object135841:head# 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b6b104:::benchmark_data_smithi194_103039_object42921:head# 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b6fb0f:::benchmark_data_smithi194_103039_object151165:head# 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b7fd3f:::benchmark_data_smithi194_103039_object121296:head# 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b88f93:::benchmark_data_smithi194_103039_object86259:head# 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b8ad9c:::benchmark_data_smithi194_103039_object150932:head# 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46b8f7ec:::benchmark_data_smithi194_103039_object150008:head# 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ba05a6:::benchmark_data_smithi194_103039_object73873:head# 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ba10ad:::benchmark_data_smithi194_103039_object154873:head# 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ba4438:::benchmark_data_smithi194_103039_object73549:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ba98df:::benchmark_data_smithi194_103039_object79232:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46bb85ff:::benchmark_data_smithi194_103039_object84029:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46bbcf22:::benchmark_data_smithi194_103039_object89931:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46bda9bd:::benchmark_data_smithi194_103039_object134450:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46be2623:::benchmark_data_smithi194_103039_object68606:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46bec81c:::benchmark_data_smithi194_103039_object131793:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46bfacce:::benchmark_data_smithi194_103039_object94264:head# 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c001fc:::benchmark_data_smithi194_103039_object142036:head# 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c0587f:::benchmark_data_smithi194_103039_object154249:head# 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c0bd1a:::benchmark_data_smithi194_103039_object125451:head# 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c0db97:::benchmark_data_smithi194_103039_object110193:head# 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c1a756:::benchmark_data_smithi194_103039_object63341:head# 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c285cf:::benchmark_data_smithi194_103039_object51836:head# 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c2888a:::benchmark_data_smithi194_103039_object84099:head# 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c30f23:::benchmark_data_smithi194_103039_object47393:head# 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c3deb4:::benchmark_data_smithi194_103039_object93455:head# 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c48ce8:::benchmark_data_smithi194_103039_object130605:head# 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c4d062:::benchmark_data_smithi194_103039_object58443:head# 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c7167e:::benchmark_data_smithi194_103039_object100535:head# 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c81985:::benchmark_data_smithi194_103039_object95614:head# 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c85504:::benchmark_data_smithi194_103039_object57887:head# 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c92286:::benchmark_data_smithi194_103039_object122283:head# 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46c9c8ac:::benchmark_data_smithi194_103039_object70948:head# 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ca2c76:::benchmark_data_smithi194_103039_object79859:head# 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46cc9c1c:::benchmark_data_smithi194_103039_object139278:head# 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ccfa40:::benchmark_data_smithi194_103039_object125321:head# 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46cdb281:::benchmark_data_smithi194_103039_object148776:head# 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46cdca3c:::benchmark_data_smithi194_103039_object140773:head# 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ce0c7f:::benchmark_data_smithi194_103039_object106845:head# 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ce84ec:::benchmark_data_smithi194_103039_object89468:head# 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46cec54b:::benchmark_data_smithi194_103039_object112565:head# 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46cf5315:::benchmark_data_smithi194_103039_object107614:head# 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46cf8030:::benchmark_data_smithi194_103039_object145912:head# 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d0f6f6:::benchmark_data_smithi194_103039_object117628:head# 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d13d78:::benchmark_data_smithi194_103039_object109978:head# 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d15567:::benchmark_data_smithi194_103039_object75136:head# 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d1f4a6:::benchmark_data_smithi194_103039_object85685:head# 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d273de:::benchmark_data_smithi194_103039_object154541:head# 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d2b21b:::benchmark_data_smithi194_103039_object50165:head# 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d2ded1:::benchmark_data_smithi194_103039_object123891:head# 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d42acb:::benchmark_data_smithi194_103039_object51805:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d4b65d:::benchmark_data_smithi194_103039_object151083:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d4e29f:::benchmark_data_smithi194_103039_object57138:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d5aac3:::benchmark_data_smithi194_103039_object44646:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d71c40:::benchmark_data_smithi194_103039_object121817:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d74eb9:::benchmark_data_smithi194_103039_object112017:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d7a7c0:::benchmark_data_smithi194_103039_object61223:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d82d6f:::benchmark_data_smithi194_103039_object115906:head# 2024-04-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d96826:::benchmark_data_smithi194_103039_object118867:head# 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d97ea0:::benchmark_data_smithi194_103039_object155569:head# 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46d98edc:::benchmark_data_smithi194_103039_object101031:head# 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46da98b1:::benchmark_data_smithi194_103039_object104788:head# 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46daaef6:::benchmark_data_smithi194_103039_object142187:head# 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46db7c6d:::benchmark_data_smithi194_103039_object129408:head# 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46dbae89:::benchmark_data_smithi194_103039_object148906:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46dbba24:::benchmark_data_smithi194_103039_object153370:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46dbe219:::benchmark_data_smithi194_103039_object138837:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46dbf3b0:::benchmark_data_smithi194_103039_object117984:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46dd60b0:::benchmark_data_smithi194_103039_object140239:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46de1c7c:::benchmark_data_smithi194_103039_object103809:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46de5c12:::benchmark_data_smithi194_103039_object46620:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46de76b8:::benchmark_data_smithi194_103039_object143556:head# 2024-04-29T14:21:29.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46dea559:::benchmark_data_smithi194_103039_object51521:head# 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46df2c81:::benchmark_data_smithi194_103039_object95067:head# 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e04417:::benchmark_data_smithi194_103039_object89190:head# 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e0db90:::benchmark_data_smithi194_103039_object128748:head# 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e145f6:::benchmark_data_smithi194_103039_object49514:head# 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e19a1d:::benchmark_data_smithi194_103039_object90052:head# 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e1a357:::benchmark_data_smithi194_103039_object66527:head# 2024-04-29T14:21:29.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e2629b:::benchmark_data_smithi194_103039_object92097:head# 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e305d6:::benchmark_data_smithi194_103039_object84214:head# 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e32f7c:::benchmark_data_smithi194_103039_object85906:head# 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e330e7:::benchmark_data_smithi194_103039_object135311:head# 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e3a5d2:::benchmark_data_smithi194_103039_object65528:head# 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e3ff77:::benchmark_data_smithi194_103039_object124386:head# 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e43382:::benchmark_data_smithi194_103039_object125920:head# 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e4880a:::benchmark_data_smithi194_103039_object99802:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e4a23c:::benchmark_data_smithi194_103039_object55610:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e4af75:::benchmark_data_smithi194_103039_object93676:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e4ee11:::benchmark_data_smithi194_103039_object134759:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e6ee5b:::benchmark_data_smithi194_103039_object59046:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e6f6cd:::benchmark_data_smithi194_103039_object141674:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e7216d:::benchmark_data_smithi194_103039_object50956:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e76bd2:::benchmark_data_smithi194_103039_object99495:head# 2024-04-29T14:21:29.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e88127:::benchmark_data_smithi194_103039_object94957:head# 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46e9407d:::benchmark_data_smithi194_103039_object128793:head# 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46eaa3b8:::benchmark_data_smithi194_103039_object147003:head# 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46eaf537:::benchmark_data_smithi194_103039_object94333:head# 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46eb02b6:::benchmark_data_smithi194_103039_object155647:head# 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ec114d:::benchmark_data_smithi194_103039_object121062:head# 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ec707d:::benchmark_data_smithi194_103039_object71961:head# 2024-04-29T14:21:29.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ec8ff2:::benchmark_data_smithi194_103039_object41543:head# 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ecfdae:::benchmark_data_smithi194_103039_object104415:head# 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ed96c4:::benchmark_data_smithi194_103039_object105895:head# 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46efe6c9:::benchmark_data_smithi194_103039_object107530:head# 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f0d926:::benchmark_data_smithi194_103039_object142231:head# 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f13c3c:::benchmark_data_smithi194_103039_object50749:head# 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f2c35f:::benchmark_data_smithi194_103039_object144709:head# 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f2d14b:::benchmark_data_smithi194_103039_object147284:head# 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f30c53:::benchmark_data_smithi194_103039_object95043:head# 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f37be4:::benchmark_data_smithi194_103039_object52809:head# 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f3afbd:::benchmark_data_smithi194_103039_object119585:head# 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f40c99:::benchmark_data_smithi194_103039_object87841:head# 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f4314e:::benchmark_data_smithi194_103039_object100486:head# 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f4a27b:::benchmark_data_smithi194_103039_object125412:head# 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f4e0d2:::benchmark_data_smithi194_103039_object47757:head# 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f5ab0a:::benchmark_data_smithi194_103039_object108495:head# 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f63784:::benchmark_data_smithi194_103039_object69823:head# 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f65b61:::benchmark_data_smithi194_103039_object95385:head# 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f73fdb:::benchmark_data_smithi194_103039_object123289:head# 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f74b28:::benchmark_data_smithi194_103039_object146250:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f87225:::benchmark_data_smithi194_103039_object122315:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f88613:::benchmark_data_smithi194_103039_object99014:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f8e3c8:::benchmark_data_smithi194_103039_object128779:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f9980a:::benchmark_data_smithi194_103039_object66531:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46f9dc33:::benchmark_data_smithi194_103039_object111915:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46fa46d2:::benchmark_data_smithi194_103039_object71032:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46fade80:::benchmark_data_smithi194_103039_object47163:head# 2024-04-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46fc3b2a:::benchmark_data_smithi194_103039_object142474:head# 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46fc7dfd:::benchmark_data_smithi194_103039_object109712:head# 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46fd4a0e:::benchmark_data_smithi194_103039_object154338:head# 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46fe057f:::benchmark_data_smithi194_103039_object83967:head# 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46feaed2:::benchmark_data_smithi194_103039_object91205:head# 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46feccb5:::benchmark_data_smithi194_103039_object42920:head# 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:46ff8671:::benchmark_data_smithi194_103039_object143020:head# 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47006bc9:::benchmark_data_smithi194_103039_object101857:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4700932f:::benchmark_data_smithi194_103039_object111036:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4700afb4:::benchmark_data_smithi194_103039_object156068:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4701056f:::benchmark_data_smithi194_103039_object129517:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4701bfc3:::benchmark_data_smithi194_103039_object138081:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4701c8dc:::benchmark_data_smithi194_103039_object90157:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4702e8fe:::benchmark_data_smithi194_103039_object44363:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47038b5b:::benchmark_data_smithi194_103039_object66765:head# 2024-04-29T14:21:29.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4703968b:::benchmark_data_smithi194_103039_object92044:head# 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4703f41b:::benchmark_data_smithi194_103039_object90480:head# 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47047eb5:::benchmark_data_smithi194_103039_object108484:head# 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47052971:::benchmark_data_smithi194_103039_object128415:head# 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47062a19:::benchmark_data_smithi194_103039_object59829:head# 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4706321b:::benchmark_data_smithi194_103039_object84371:head# 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470642ee:::benchmark_data_smithi194_103039_object64812:head# 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4707436d:::benchmark_data_smithi194_103039_object71337:head# 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47085d2b:::benchmark_data_smithi194_103039_object146073:head# 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470934f0:::benchmark_data_smithi194_103039_object97597:head# 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47099d00:::benchmark_data_smithi194_103039_object97271:head# 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470b7b5a:::benchmark_data_smithi194_103039_object116597:head# 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470b8c92:::benchmark_data_smithi194_103039_object112781:head# 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470c7c82:::benchmark_data_smithi194_103039_object91691:head# 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470d0be0:::benchmark_data_smithi194_103039_object51030:head# 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470ea0f2:::benchmark_data_smithi194_103039_object113007:head# 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:470f060b:::benchmark_data_smithi194_103039_object90638:head# 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47101444:::benchmark_data_smithi194_103039_object123945:head# 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471126a3:::benchmark_data_smithi194_103039_object49888:head# 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4713bb6b:::benchmark_data_smithi194_103039_object136170:head# 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4713cc3f:::benchmark_data_smithi194_103039_object114375:head# 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4713e7fb:::benchmark_data_smithi194_103039_object127137:head# 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471495a1:::benchmark_data_smithi194_103039_object109906:head# 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47155565:::benchmark_data_smithi194_103039_object116605:head# 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4715603d:::benchmark_data_smithi194_103039_object93610:head# 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4715d032:::benchmark_data_smithi194_103039_object110578:head# 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4717d008:::benchmark_data_smithi194_103039_object94206:head# 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47181328:::benchmark_data_smithi194_103039_object109059:head# 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.731 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47188cad:::benchmark_data_smithi194_103039_object135714:head# 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4718aa03:::benchmark_data_smithi194_103039_object57385:head# 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4719234b:::benchmark_data_smithi194_103039_object122192:head# 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471964e5:::benchmark_data_smithi194_103039_object99531:head# 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471977ca:::benchmark_data_smithi194_103039_object115721:head# 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4719b50e:::benchmark_data_smithi194_103039_object45148:head# 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471a322f:::benchmark_data_smithi194_103039_object128136:head# 2024-04-29T14:21:29.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471a93e5:::benchmark_data_smithi194_103039_object107365:head# 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471b77b5:::benchmark_data_smithi194_103039_object98044:head# 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471bc18b:::benchmark_data_smithi194_103039_object64332:head# 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471bf85f:::benchmark_data_smithi194_103039_object81496:head# 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471cfa05:::benchmark_data_smithi194_103039_object57253:head# 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471e4cac:::benchmark_data_smithi194_103039_object92861:head# 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471e86bc:::benchmark_data_smithi194_103039_object96621:head# 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471e955c:::benchmark_data_smithi194_103039_object109157:head# 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471f742c:::benchmark_data_smithi194_103039_object135012:head# 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:471ff668:::benchmark_data_smithi194_103039_object86951:head# 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47201a95:::benchmark_data_smithi194_103039_object125328:head# 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4720f574:::benchmark_data_smithi194_103039_object96488:head# 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47215cfc:::benchmark_data_smithi194_103039_object102809:head# 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472163fb:::benchmark_data_smithi194_103039_object112763:head# 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4721a83d:::benchmark_data_smithi194_103039_object149788:head# 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472320d7:::benchmark_data_smithi194_103039_object60038:head# 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47237dc8:::benchmark_data_smithi194_103039_object139536:head# 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472403c4:::benchmark_data_smithi194_103039_object42193:head# 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4725157a:::benchmark_data_smithi194_103039_object42994:head# 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47255a39:::benchmark_data_smithi194_103039_object70223:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4726a7c7:::benchmark_data_smithi194_103039_object65391:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47272314:::benchmark_data_smithi194_103039_object93138:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47280a1d:::benchmark_data_smithi194_103039_object86305:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4728cfda:::benchmark_data_smithi194_103039_object136178:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4729bf28:::benchmark_data_smithi194_103039_object71222:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472a7a5b:::benchmark_data_smithi194_103039_object89108:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472ab8b9:::benchmark_data_smithi194_103039_object99068:head# 2024-04-29T14:21:29.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472acd34:::benchmark_data_smithi194_103039_object86339:head# 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472b90c4:::benchmark_data_smithi194_103039_object125591:head# 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472bda95:::benchmark_data_smithi194_103039_object72888:head# 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472c27f7:::benchmark_data_smithi194_103039_object105612:head# 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472c3568:::benchmark_data_smithi194_103039_object133470:head# 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472c7bef:::benchmark_data_smithi194_103039_object145512:head# 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472e30d6:::benchmark_data_smithi194_103039_object130050:head# 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472e47ca:::benchmark_data_smithi194_103039_object55274:head# 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472edb24:::benchmark_data_smithi194_103039_object63926:head# 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472f090b:::benchmark_data_smithi194_103039_object133122:head# 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:472f53d2:::benchmark_data_smithi194_103039_object134564:head# 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47310a39:::benchmark_data_smithi194_103039_object85222:head# 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473114f9:::benchmark_data_smithi194_103039_object58929:head# 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473164eb:::benchmark_data_smithi194_103039_object88465:head# 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4731965e:::benchmark_data_smithi194_103039_object152734:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4734c393:::benchmark_data_smithi194_103039_object93937:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4735983c:::benchmark_data_smithi194_103039_object96110:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4736472e:::benchmark_data_smithi194_103039_object101678:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4736b1b6:::benchmark_data_smithi194_103039_object144356:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4736d38e:::benchmark_data_smithi194_103039_object56076:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4736f6cc:::benchmark_data_smithi194_103039_object87071:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4737b23d:::benchmark_data_smithi194_103039_object79040:head# 2024-04-29T14:21:29.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473819a2:::benchmark_data_smithi194_103039_object128397:head# 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4738ec51:::benchmark_data_smithi194_103039_object85651:head# 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4739d362:::benchmark_data_smithi194_103039_object113613:head# 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473a2167:::benchmark_data_smithi194_103039_object115975:head# 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473b17f1:::benchmark_data_smithi194_103039_object151495:head# 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473b78eb:::benchmark_data_smithi194_103039_object131786:head# 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473bbc16:::benchmark_data_smithi194_103039_object98414:head# 2024-04-29T14:21:29.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473cbfe8:::benchmark_data_smithi194_103039_object55857:head# 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473ccdde:::benchmark_data_smithi194_103039_object104452:head# 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473e193c:::benchmark_data_smithi194_103039_object69983:head# 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473e486b:::benchmark_data_smithi194_103039_object130799:head# 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473eaeb5:::benchmark_data_smithi194_103039_object61516:head# 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473ec9f0:::benchmark_data_smithi194_103039_object46642:head# 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:473fb5dd:::benchmark_data_smithi194_103039_object74040:head# 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4740a11e:::benchmark_data_smithi194_103039_object144922:head# 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47412924:::benchmark_data_smithi194_103039_object133482:head# 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47419600:::benchmark_data_smithi194_103039_object144315:head# 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474211a0:::benchmark_data_smithi194_103039_object100308:head# 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474280f0:::benchmark_data_smithi194_103039_object112350:head# 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4742cf1a:::benchmark_data_smithi194_103039_object49799:head# 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474322a1:::benchmark_data_smithi194_103039_object76798:head# 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47433c41:::benchmark_data_smithi194_103039_object131120:head# 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4743a37b:::benchmark_data_smithi194_103039_object151973:head# 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4743bb8f:::benchmark_data_smithi194_103039_object112328:head# 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474407fd:::benchmark_data_smithi194_103039_object44078:head# 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47442846:::benchmark_data_smithi194_103039_object105827:head# 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4744ac21:::benchmark_data_smithi194_103039_object48961:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4745310b:::benchmark_data_smithi194_103039_object108893:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4745aaec:::benchmark_data_smithi194_103039_object156151:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4745b160:::benchmark_data_smithi194_103039_object52720:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47460aab:::benchmark_data_smithi194_103039_object102415:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47478d5f:::benchmark_data_smithi194_103039_object63345:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47479a42:::benchmark_data_smithi194_103039_object101922:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47493e7d:::benchmark_data_smithi194_103039_object100400:head# 2024-04-29T14:21:29.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47497824:::benchmark_data_smithi194_103039_object75106:head# 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474ae770:::benchmark_data_smithi194_103039_object91725:head# 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474b64a2:::benchmark_data_smithi194_103039_object49038:head# 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474bcc4d:::benchmark_data_smithi194_103039_object92036:head# 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474c9b8a:::benchmark_data_smithi194_103039_object117793:head# 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474d025b:::benchmark_data_smithi194_103039_object78525:head# 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474d79a8:::benchmark_data_smithi194_103039_object122831:head# 2024-04-29T14:21:29.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:474faf8d:::benchmark_data_smithi194_103039_object76761:head# 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4750c235:::benchmark_data_smithi194_103039_object59158:head# 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47529e47:::benchmark_data_smithi194_103039_object155631:head# 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4752f083:::benchmark_data_smithi194_103039_object91852:head# 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4753e733:::benchmark_data_smithi194_103039_object148703:head# 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4753f5ff:::benchmark_data_smithi194_103039_object137697:head# 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475411e7:::benchmark_data_smithi194_103039_object97167:head# 2024-04-29T14:21:29.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4755262f:::benchmark_data_smithi194_103039_object129667:head# 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47556989:::benchmark_data_smithi194_103039_object144532:head# 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47566627:::benchmark_data_smithi194_103039_object79943:head# 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4757609f:::benchmark_data_smithi194_103039_object63132:head# 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4757bbdf:::benchmark_data_smithi194_103039_object128889:head# 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47586947:::benchmark_data_smithi194_103039_object100793:head# 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47588846:::benchmark_data_smithi194_103039_object138137:head# 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.747 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4758ef67:::benchmark_data_smithi194_103039_object45223:head# 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4758f019:::benchmark_data_smithi194_103039_object100528:head# 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47592427:::benchmark_data_smithi194_103039_object49363:head# 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475a2abd:::benchmark_data_smithi194_103039_object78338:head# 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475b0229:::benchmark_data_smithi194_103039_object137864:head# 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475b2883:::benchmark_data_smithi194_103039_object68866:head# 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475bfc85:::benchmark_data_smithi194_103039_object147376:head# 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475c5320:::benchmark_data_smithi194_103039_object51077:head# 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475c630f:::benchmark_data_smithi194_103039_object132042:head# 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475d21bc:::benchmark_data_smithi194_103039_object120649:head# 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475d303f:::benchmark_data_smithi194_103039_object152379:head# 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:475e4dd8:::benchmark_data_smithi194_103039_object83108:head# 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4760231f:::benchmark_data_smithi194_103039_object75253:head# 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4761c813:::benchmark_data_smithi194_103039_object134919:head# 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4762014a:::benchmark_data_smithi194_103039_object57488:head# 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47622414:::benchmark_data_smithi194_103039_object63036:head# 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47646fef:::benchmark_data_smithi194_103039_object128833:head# 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47659c8e:::benchmark_data_smithi194_103039_object92706:head# 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4765ad9c:::benchmark_data_smithi194_103039_object148594:head# 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4765e5ef:::benchmark_data_smithi194_103039_object138948:head# 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47675299:::benchmark_data_smithi194_103039_object143558:head# 2024-04-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4767530a:::benchmark_data_smithi194_103039_object148321:head# 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4767b3b5:::benchmark_data_smithi194_103039_object56201:head# 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4767e8bd:::benchmark_data_smithi194_103039_object92670:head# 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4768f994:::benchmark_data_smithi194_103039_object91704:head# 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47698631:::benchmark_data_smithi194_103039_object109199:head# 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4769c87b:::benchmark_data_smithi194_103039_object91607:head# 2024-04-29T14:21:29.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476a2887:::benchmark_data_smithi194_103039_object98725:head# 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476a8bee:::benchmark_data_smithi194_103039_object45580:head# 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476b4a0b:::benchmark_data_smithi194_103039_object45409:head# 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476b5e05:::benchmark_data_smithi194_103039_object53558:head# 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476b9013:::benchmark_data_smithi194_103039_object88843:head# 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476bbaf8:::benchmark_data_smithi194_103039_object117856:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476e6e9c:::benchmark_data_smithi194_103039_object46405:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476e777e:::benchmark_data_smithi194_103039_object96798:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:476fcfc5:::benchmark_data_smithi194_103039_object89565:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47701986:::benchmark_data_smithi194_103039_object145211:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47711354:::benchmark_data_smithi194_103039_object138345:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477152f4:::benchmark_data_smithi194_103039_object99087:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4771a75f:::benchmark_data_smithi194_103039_object55001:head# 2024-04-29T14:21:29.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47728f46:::benchmark_data_smithi194_103039_object129200:head# 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4772c053:::benchmark_data_smithi194_103039_object76366:head# 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47730614:::benchmark_data_smithi194_103039_object134401:head# 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47745387:::benchmark_data_smithi194_103039_object147340:head# 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4775988f:::benchmark_data_smithi194_103039_object115213:head# 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47764d74:::benchmark_data_smithi194_103039_object119227:head# 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47765cc9:::benchmark_data_smithi194_103039_object90419:head# 2024-04-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4776b0b5:::benchmark_data_smithi194_103039_object148614:head# 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47773bae:::benchmark_data_smithi194_103039_object130596:head# 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47777906:::benchmark_data_smithi194_103039_object69110:head# 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47778ac6:::benchmark_data_smithi194_103039_object100907:head# 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4778ff8c:::benchmark_data_smithi194_103039_object84544:head# 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47797b85:::benchmark_data_smithi194_103039_object144944:head# 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47797be7:::benchmark_data_smithi194_103039_object94705:head# 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477a5a69:::benchmark_data_smithi194_103039_object73029:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477a73b5:::benchmark_data_smithi194_103039_object129880:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477a91bc:::benchmark_data_smithi194_103039_object97961:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477aab47:::benchmark_data_smithi194_103039_object156080:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477ac9f3:::benchmark_data_smithi194_103039_object96531:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477aecf4:::benchmark_data_smithi194_103039_object46758:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477b533e:::benchmark_data_smithi194_103039_object123973:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477bcf1b:::benchmark_data_smithi194_103039_object130700:head# 2024-04-29T14:21:29.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477ce7c4:::benchmark_data_smithi194_103039_object103795:head# 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477e009e:::benchmark_data_smithi194_103039_object139815:head# 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477e59f3:::benchmark_data_smithi194_103039_object79865:head# 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477eb68f:::benchmark_data_smithi194_103039_object138907:head# 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477f5a99:::benchmark_data_smithi194_103039_object131733:head# 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477fa4d2:::benchmark_data_smithi194_103039_object56700:head# 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:477fac16:::benchmark_data_smithi194_103039_object67262:head# 2024-04-29T14:21:29.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47803bef:::benchmark_data_smithi194_103039_object83998:head# 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4780caed:::benchmark_data_smithi194_103039_object55614:head# 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47810f3c:::benchmark_data_smithi194_103039_object136856:head# 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4781817b:::benchmark_data_smithi194_103039_object105412:head# 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47819cbc:::benchmark_data_smithi194_103039_object145934:head# 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4781adf3:::benchmark_data_smithi194_103039_object133722:head# 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47822f1a:::benchmark_data_smithi194_103039_object125817:head# 2024-04-29T14:21:29.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478286a0:::benchmark_data_smithi194_103039_object68871:head# 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4782c392:::benchmark_data_smithi194_103039_object65191:head# 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4785703b:::benchmark_data_smithi194_103039_object112801:head# 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4785cd54:::benchmark_data_smithi194_103039_object104697:head# 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4786e5f4:::benchmark_data_smithi194_103039_object134377:head# 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4786ed8a:::benchmark_data_smithi194_103039_object62816:head# 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47871a03:::benchmark_data_smithi194_103039_object139657:head# 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478746ab:::benchmark_data_smithi194_103039_object54152:head# 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4787a9a1:::benchmark_data_smithi194_103039_object155150:head# 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4788b09f:::benchmark_data_smithi194_103039_object115136:head# 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47891ff5:::benchmark_data_smithi194_103039_object96005:head# 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47896416:::benchmark_data_smithi194_103039_object101784:head# 2024-04-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4789c935:::benchmark_data_smithi194_103039_object132888:head# 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4789f22e:::benchmark_data_smithi194_103039_object105627:head# 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478a1020:::benchmark_data_smithi194_103039_object75763:head# 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478a1376:::benchmark_data_smithi194_103039_object145254:head# 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478a9554:::benchmark_data_smithi194_103039_object134952:head# 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478bf862:::benchmark_data_smithi194_103039_object65168:head# 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478c95a8:::benchmark_data_smithi194_103039_object126405:head# 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478c9db4:::benchmark_data_smithi194_103039_object144568:head# 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478cc953:::benchmark_data_smithi194_103039_object88387:head# 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478cf1fa:::benchmark_data_smithi194_103039_object156145:head# 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478d0c0c:::benchmark_data_smithi194_103039_object134647:head# 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478d2cf0:::benchmark_data_smithi194_103039_object137922:head# 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478e5ea7:::benchmark_data_smithi194_103039_object51623:head# 2024-04-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478e62eb:::benchmark_data_smithi194_103039_object47037:head# 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478f2811:::benchmark_data_smithi194_103039_object73573:head# 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:478ffa8e:::benchmark_data_smithi194_103039_object51136:head# 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47900524:::benchmark_data_smithi194_103039_object98985:head# 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47906202:::benchmark_data_smithi194_103039_object55663:head# 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4791026e:::benchmark_data_smithi194_103039_object46162:head# 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479129d2:::benchmark_data_smithi194_103039_object61414:head# 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479136b0:::benchmark_data_smithi194_103039_object80107:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4791cb2e:::benchmark_data_smithi194_103039_object148283:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47921a38:::benchmark_data_smithi194_103039_object95297:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4792c40c:::benchmark_data_smithi194_103039_object88096:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4792d698:::benchmark_data_smithi194_103039_object138470:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4793b9b7:::benchmark_data_smithi194_103039_object138918:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47957d6a:::benchmark_data_smithi194_103039_object47053:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47958058:::benchmark_data_smithi194_103039_object151408:head# 2024-04-29T14:21:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4795b95f:::benchmark_data_smithi194_103039_object45840:head# 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47960a63:::benchmark_data_smithi194_103039_object76852:head# 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47975dcf:::benchmark_data_smithi194_103039_object127725:head# 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479781ae:::benchmark_data_smithi194_103039_object42790:head# 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47988f57:::benchmark_data_smithi194_103039_object64040:head# 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4798f43e:::benchmark_data_smithi194_103039_object67633:head# 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4799b12b:::benchmark_data_smithi194_103039_object99491:head# 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479abe89:::benchmark_data_smithi194_103039_object88175:head# 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479bc6d1:::benchmark_data_smithi194_103039_object89376:head# 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479ccf14:::benchmark_data_smithi194_103039_object43138:head# 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479ce679:::benchmark_data_smithi194_103039_object140016:head# 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479e687d:::benchmark_data_smithi194_103039_object74348:head# 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:479e8a61:::benchmark_data_smithi194_103039_object139929:head# 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a05ba7:::benchmark_data_smithi194_103039_object102476:head# 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a1d603:::benchmark_data_smithi194_103039_object90334:head# 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a28d48:::benchmark_data_smithi194_103039_object122944:head# 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a2fa94:::benchmark_data_smithi194_103039_object88257:head# 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a331c3:::benchmark_data_smithi194_103039_object81648:head# 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a386e8:::benchmark_data_smithi194_103039_object125099:head# 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a51d33:::benchmark_data_smithi194_103039_object144884:head# 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a59f3c:::benchmark_data_smithi194_103039_object90243:head# 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a684de:::benchmark_data_smithi194_103039_object45937:head# 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a77698:::benchmark_data_smithi194_103039_object129710:head# 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a8079e:::benchmark_data_smithi194_103039_object136896:head# 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a854e4:::benchmark_data_smithi194_103039_object118833:head# 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a8688f:::benchmark_data_smithi194_103039_object77648:head# 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a8d284:::benchmark_data_smithi194_103039_object95841:head# 2024-04-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a90c03:::benchmark_data_smithi194_103039_object54118:head# 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47a97e7d:::benchmark_data_smithi194_103039_object86148:head# 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47aa7af7:::benchmark_data_smithi194_103039_object55103:head# 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ab3746:::benchmark_data_smithi194_103039_object121667:head# 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ab3ac9:::benchmark_data_smithi194_103039_object60158:head# 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ab64c1:::benchmark_data_smithi194_103039_object64267:head# 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47aba5a6:::benchmark_data_smithi194_103039_object136705:head# 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ac0968:::benchmark_data_smithi194_103039_object90840:head# 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ac1644:::benchmark_data_smithi194_103039_object81012:head# 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ac165d:::benchmark_data_smithi194_103039_object128201:head# 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ac604e:::benchmark_data_smithi194_103039_object111735:head# 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ad7e83:::benchmark_data_smithi194_103039_object51079:head# 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ae70a9:::benchmark_data_smithi194_103039_object136591:head# 2024-04-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47aed97e:::benchmark_data_smithi194_103039_object107148:head# 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47af2d21:::benchmark_data_smithi194_103039_object145260:head# 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b09db3:::benchmark_data_smithi194_103039_object107912:head# 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b0e383:::benchmark_data_smithi194_103039_object153957:head# 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b0e602:::benchmark_data_smithi194_103039_object48425:head# 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b181dc:::benchmark_data_smithi194_103039_object113129:head# 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b213e6:::benchmark_data_smithi194_103039_object89021:head# 2024-04-29T14:21:29.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b221c5:::benchmark_data_smithi194_103039_object56506:head# 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b24eac:::benchmark_data_smithi194_103039_object89838:head# 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b25d52:::benchmark_data_smithi194_103039_object47141:head# 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b31485:::benchmark_data_smithi194_103039_object73212:head# 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b3171a:::benchmark_data_smithi194_103039_object151933:head# 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b37232:::benchmark_data_smithi194_103039_object140680:head# 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b3ac3b:::benchmark_data_smithi194_103039_object123297:head# 2024-04-29T14:21:29.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b571eb:::benchmark_data_smithi194_103039_object130532:head# 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b71ad8:::benchmark_data_smithi194_103039_object112945:head# 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b721a4:::benchmark_data_smithi194_103039_object58408:head# 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b7be7f:::benchmark_data_smithi194_103039_object80837:head# 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47b8582b:::benchmark_data_smithi194_103039_object104978:head# 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ba3fb2:::benchmark_data_smithi194_103039_object119411:head# 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47baaf00:::benchmark_data_smithi194_103039_object122166:head# 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bafb6c:::benchmark_data_smithi194_103039_object75551:head# 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bb1c03:::benchmark_data_smithi194_103039_object55800:head# 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bb5993:::benchmark_data_smithi194_103039_object51845:head# 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bba3c1:::benchmark_data_smithi194_103039_object48196:head# 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bc95fc:::benchmark_data_smithi194_103039_object152390:head# 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bca265:::benchmark_data_smithi194_103039_object82491:head# 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bd1776:::benchmark_data_smithi194_103039_object144524:head# 2024-04-29T14:21:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bd958d:::benchmark_data_smithi194_103039_object41506:head# 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bdfeb6:::benchmark_data_smithi194_103039_object152371:head# 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47be02f8:::benchmark_data_smithi194_103039_object142303:head# 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47be4b70:::benchmark_data_smithi194_103039_object103017:head# 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47be991c:::benchmark_data_smithi194_103039_object77754:head# 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bf2416:::benchmark_data_smithi194_103039_object60451:head# 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47bf987f:::benchmark_data_smithi194_103039_object146263:head# 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c02587:::benchmark_data_smithi194_103039_object76443:head# 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c2e498:::benchmark_data_smithi194_103039_object44118:head# 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c3b4e3:::benchmark_data_smithi194_103039_object49066:head# 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c4c062:::benchmark_data_smithi194_103039_object102980:head# 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c7da64:::benchmark_data_smithi194_103039_object106152:head# 2024-04-29T14:21:29.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c890f7:::benchmark_data_smithi194_103039_object99021:head# 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c9078b:::benchmark_data_smithi194_103039_object61759:head# 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c91b64:::benchmark_data_smithi194_103039_object94265:head# 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c93623:::benchmark_data_smithi194_103039_object76976:head# 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c997a2:::benchmark_data_smithi194_103039_object60748:head# 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47c9e1fc:::benchmark_data_smithi194_103039_object125179:head# 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ca16e5:::benchmark_data_smithi194_103039_object97488:head# 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ca202c:::benchmark_data_smithi194_103039_object155595:head# 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ca8b4f:::benchmark_data_smithi194_103039_object83513:head# 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47cae646:::benchmark_data_smithi194_103039_object80651:head# 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47cb4490:::benchmark_data_smithi194_103039_object80830:head# 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47cba397:::benchmark_data_smithi194_103039_object83137:head# 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ccfc03:::benchmark_data_smithi194_103039_object138790:head# 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47cddab7:::benchmark_data_smithi194_103039_object114500:head# 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ceeeb3:::benchmark_data_smithi194_103039_object101662:head# 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47cf2850:::benchmark_data_smithi194_103039_object151879:head# 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d11422:::benchmark_data_smithi194_103039_object92708:head# 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d2629a:::benchmark_data_smithi194_103039_object144543:head# 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d2b226:::benchmark_data_smithi194_103039_object117475:head# 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d2c8da:::benchmark_data_smithi194_103039_object97161:head# 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d3fd64:::benchmark_data_smithi194_103039_object134696:head# 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d4ae6b:::benchmark_data_smithi194_103039_object68136:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d688f7:::benchmark_data_smithi194_103039_object72299:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d6a82d:::benchmark_data_smithi194_103039_object103825:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d6bac7:::benchmark_data_smithi194_103039_object143094:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d6e35a:::benchmark_data_smithi194_103039_object114429:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d6f7b7:::benchmark_data_smithi194_103039_object103086:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d723b5:::benchmark_data_smithi194_103039_object116947:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d7c706:::benchmark_data_smithi194_103039_object84032:head# 2024-04-29T14:21:29.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d7f62c:::benchmark_data_smithi194_103039_object144457:head# 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47d9b849:::benchmark_data_smithi194_103039_object138053:head# 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47da5442:::benchmark_data_smithi194_103039_object68936:head# 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47db7fc1:::benchmark_data_smithi194_103039_object76399:head# 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47dbe904:::benchmark_data_smithi194_103039_object42700:head# 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47dc515e:::benchmark_data_smithi194_103039_object73283:head# 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47dc54d1:::benchmark_data_smithi194_103039_object89139:head# 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47dfa624:::benchmark_data_smithi194_103039_object98539:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e0a855:::benchmark_data_smithi194_103039_object61663:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e0c96f:::benchmark_data_smithi194_103039_object128070:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e16ac4:::benchmark_data_smithi194_103039_object130892:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e1902a:::benchmark_data_smithi194_103039_object82500:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e30995:::benchmark_data_smithi194_103039_object144702:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e34490:::benchmark_data_smithi194_103039_object90147:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e3839b:::benchmark_data_smithi194_103039_object68738:head# 2024-04-29T14:21:29.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e4091f:::benchmark_data_smithi194_103039_object106117:head# 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e44054:::benchmark_data_smithi194_103039_object151240:head# 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e524eb:::benchmark_data_smithi194_103039_object49947:head# 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e5717d:::benchmark_data_smithi194_103039_object114966:head# 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e5d6b3:::benchmark_data_smithi194_103039_object102510:head# 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e7dbbe:::benchmark_data_smithi194_103039_object48726:head# 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e7f522:::benchmark_data_smithi194_103039_object42139:head# 2024-04-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e85ce0:::benchmark_data_smithi194_103039_object89486:head# 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e8abba:::benchmark_data_smithi194_103039_object142298:head# 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e8acb5:::benchmark_data_smithi194_103039_object137788:head# 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e8ce4a:::benchmark_data_smithi194_103039_object143787:head# 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e8e849:::benchmark_data_smithi194_103039_object134683:head# 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e94ed1:::benchmark_data_smithi194_103039_object43596:head# 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47e95fbd:::benchmark_data_smithi194_103039_object143879:head# 2024-04-29T14:21:29.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ea1e59:::benchmark_data_smithi194_103039_object45627:head# 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47eb3fff:::benchmark_data_smithi194_103039_object94013:head# 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47eba8f4:::benchmark_data_smithi194_103039_object131255:head# 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ec7535:::benchmark_data_smithi194_103039_object150887:head# 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47eca605:::benchmark_data_smithi194_103039_object136087:head# 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ecdb41:::benchmark_data_smithi194_103039_object104760:head# 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47ef1ab9:::benchmark_data_smithi194_103039_object88619:head# 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47efa4cc:::benchmark_data_smithi194_103039_object112849:head# 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47efb021:::benchmark_data_smithi194_103039_object95285:head# 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f0860c:::benchmark_data_smithi194_103039_object72365:head# 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f09ae5:::benchmark_data_smithi194_103039_object70297:head# 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f0a34c:::benchmark_data_smithi194_103039_object48839:head# 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f1070d:::benchmark_data_smithi194_103039_object141256:head# 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f1d7dc:::benchmark_data_smithi194_103039_object61222:head# 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f218b6:::benchmark_data_smithi194_103039_object116477:head# 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f2209a:::benchmark_data_smithi194_103039_object89830:head# 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f25112:::benchmark_data_smithi194_103039_object71258:head# 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f348e7:::benchmark_data_smithi194_103039_object87157:head# 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f35787:::benchmark_data_smithi194_103039_object57712:head# 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f48192:::benchmark_data_smithi194_103039_object54675:head# 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f48d71:::benchmark_data_smithi194_103039_object109767:head# 2024-04-29T14:21:29.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f4eefa:::benchmark_data_smithi194_103039_object72794:head# 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f5312b:::benchmark_data_smithi194_103039_object138158:head# 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f57001:::benchmark_data_smithi194_103039_object154929:head# 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f654d3:::benchmark_data_smithi194_103039_object125120:head# 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f6c7ae:::benchmark_data_smithi194_103039_object111212:head# 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f79b67:::benchmark_data_smithi194_103039_object130948:head# 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47f7f8d1:::benchmark_data_smithi194_103039_object49536:head# 2024-04-29T14:21:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47fb4ad5:::benchmark_data_smithi194_103039_object72558:head# 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47fb5e70:::benchmark_data_smithi194_103039_object82682:head# 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47fe56af:::benchmark_data_smithi194_103039_object144794:head# 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:47fec677:::benchmark_data_smithi194_103039_object75746:head# 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480016fc:::benchmark_data_smithi194_103039_object55197:head# 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4800afbe:::benchmark_data_smithi194_103039_object108203:head# 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48010f77:::benchmark_data_smithi194_103039_object63902:head# 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48022efd:::benchmark_data_smithi194_103039_object46200:head# 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48028896:::benchmark_data_smithi194_103039_object96784:head# 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48029360:::benchmark_data_smithi194_103039_object67266:head# 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48041be7:::benchmark_data_smithi194_103039_object124704:head# 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480448c1:::benchmark_data_smithi194_103039_object63676:head# 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48045f33:::benchmark_data_smithi194_103039_object134725:head# 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4804b2cf:::benchmark_data_smithi194_103039_object68104:head# 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48050b08:::benchmark_data_smithi194_103039_object120566:head# 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4806d5e3:::benchmark_data_smithi194_103039_object84184:head# 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4806ef6d:::benchmark_data_smithi194_103039_object123640:head# 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48074b48:::benchmark_data_smithi194_103039_object67304:head# 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4807552a:::benchmark_data_smithi194_103039_object110103:head# 2024-04-29T14:21:29.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48077c02:::benchmark_data_smithi194_103039_object98028:head# 2024-04-29T14:21:29.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48089e30:::benchmark_data_smithi194_103039_object73407:head# 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4809ec3a:::benchmark_data_smithi194_103039_object155399:head# 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480abdcb:::benchmark_data_smithi194_103039_object76071:head# 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480b32cf:::benchmark_data_smithi194_103039_object89763:head# 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480b44d6:::benchmark_data_smithi194_103039_object91949:head# 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480be1cc:::benchmark_data_smithi194_103039_object138736:head# 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480c1ada:::benchmark_data_smithi194_103039_object42576:head# 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480c5ebb:::benchmark_data_smithi194_103039_object86350:head# 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480c6743:::benchmark_data_smithi194_103039_object72205:head# 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480cd66c:::benchmark_data_smithi194_103039_object89194:head# 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480d7884:::benchmark_data_smithi194_103039_object54725:head# 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480dfbf9:::benchmark_data_smithi194_103039_object126367:head# 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480ea545:::benchmark_data_smithi194_103039_object91554:head# 2024-04-29T14:21:29.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480ea95d:::benchmark_data_smithi194_103039_object53199:head# 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:480f2597:::benchmark_data_smithi194_103039_object89100:head# 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48102ff4:::benchmark_data_smithi194_103039_object110927:head# 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48117626:::benchmark_data_smithi194_103039_object83983:head# 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48118715:::benchmark_data_smithi194_103039_object154434:head# 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48121ddf:::benchmark_data_smithi194_103039_object55439:head# 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4812eb97:::benchmark_data_smithi194_103039_object155453:head# 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48138161:::benchmark_data_smithi194_103039_object42914:head# 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48145068:::benchmark_data_smithi194_103039_object102686:head# 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48147c16:::benchmark_data_smithi194_103039_object105011:head# 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4814ab8e:::benchmark_data_smithi194_103039_object101665:head# 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4814be3a:::benchmark_data_smithi194_103039_object111345:head# 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4814d092:::benchmark_data_smithi194_103039_object87195:head# 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4814f6cf:::benchmark_data_smithi194_103039_object145891:head# 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48156deb:::benchmark_data_smithi194_103039_object134873:head# 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48180eab:::benchmark_data_smithi194_103039_object82542:head# 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4818d9ba:::benchmark_data_smithi194_103039_object126446:head# 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4819dae8:::benchmark_data_smithi194_103039_object74722:head# 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481a2cbc:::benchmark_data_smithi194_103039_object98307:head# 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481ab710:::benchmark_data_smithi194_103039_object72654:head# 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481b0367:::benchmark_data_smithi194_103039_object80833:head# 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481b0be5:::benchmark_data_smithi194_103039_object89283:head# 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481b7afe:::benchmark_data_smithi194_103039_object63763:head# 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481bb7b7:::benchmark_data_smithi194_103039_object103264:head# 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481e6d8f:::benchmark_data_smithi194_103039_object83377:head# 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:481f0a77:::benchmark_data_smithi194_103039_object92182:head# 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48203230:::benchmark_data_smithi194_103039_object114057:head# 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4820c56c:::benchmark_data_smithi194_103039_object151156:head# 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4820d3ea:::benchmark_data_smithi194_103039_object136049:head# 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48211962:::benchmark_data_smithi194_103039_object48713:head# 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48216300:::benchmark_data_smithi194_103039_object87101:head# 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4822adc3:::benchmark_data_smithi194_103039_object103352:head# 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4822b278:::benchmark_data_smithi194_103039_object42277:head# 2024-04-29T14:21:29.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48237c45:::benchmark_data_smithi194_103039_object55629:head# 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48243593:::benchmark_data_smithi194_103039_object95153:head# 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48245904:::benchmark_data_smithi194_103039_object62193:head# 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482480a2:::benchmark_data_smithi194_103039_object109212:head# 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4824f689:::benchmark_data_smithi194_103039_object136312:head# 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4824f9c3:::benchmark_data_smithi194_103039_object44299:head# 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482522c3:::benchmark_data_smithi194_103039_object111297:head# 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4825485e:::benchmark_data_smithi194_103039_object136445:head# 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4827c0c0:::benchmark_data_smithi194_103039_object68130:head# 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4828032f:::benchmark_data_smithi194_103039_object86931:head# 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482842be:::benchmark_data_smithi194_103039_object66916:head# 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4829c3c0:::benchmark_data_smithi194_103039_object128844:head# 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482a74cc:::benchmark_data_smithi194_103039_object143007:head# 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482c2a8b:::benchmark_data_smithi194_103039_object43721:head# 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482c93e7:::benchmark_data_smithi194_103039_object71048:head# 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482d551e:::benchmark_data_smithi194_103039_object83415:head# 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482ece5b:::benchmark_data_smithi194_103039_object85999:head# 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482f3965:::benchmark_data_smithi194_103039_object143374:head# 2024-04-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482f8fda:::benchmark_data_smithi194_103039_object116866:head# 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482fad09:::benchmark_data_smithi194_103039_object62697:head# 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482fbb45:::benchmark_data_smithi194_103039_object51333:head# 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:482fefa4:::benchmark_data_smithi194_103039_object130695:head# 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48310f49:::benchmark_data_smithi194_103039_object126107:head# 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483148da:::benchmark_data_smithi194_103039_object151186:head# 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4831bff0:::benchmark_data_smithi194_103039_object74535:head# 2024-04-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4831fd63:::benchmark_data_smithi194_103039_object94712:head# 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48322ed9:::benchmark_data_smithi194_103039_object50783:head# 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4832a653:::benchmark_data_smithi194_103039_object107507:head# 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4832f0f5:::benchmark_data_smithi194_103039_object135759:head# 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48331d0d:::benchmark_data_smithi194_103039_object62614:head# 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4834142b:::benchmark_data_smithi194_103039_object43348:head# 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48348824:::benchmark_data_smithi194_103039_object113771:head# 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48349da7:::benchmark_data_smithi194_103039_object49115:head# 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4834e18f:::benchmark_data_smithi194_103039_object104686:head# 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483507b8:::benchmark_data_smithi194_103039_object91406:head# 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48351813:::benchmark_data_smithi194_103039_object65061:head# 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4835eb5d:::benchmark_data_smithi194_103039_object120092:head# 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4836f828:::benchmark_data_smithi194_103039_object138220:head# 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4836f8f4:::benchmark_data_smithi194_103039_object50851:head# 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48378b7d:::benchmark_data_smithi194_103039_object122943:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4837d315:::benchmark_data_smithi194_103039_object55487:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48380088:::benchmark_data_smithi194_103039_object118367:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48387363:::benchmark_data_smithi194_103039_object50375:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48393143:::benchmark_data_smithi194_103039_object106101:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483958fe:::benchmark_data_smithi194_103039_object48197:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4839db70:::benchmark_data_smithi194_103039_object65184:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4839e790:::benchmark_data_smithi194_103039_object71472:head# 2024-04-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4839fd5a:::benchmark_data_smithi194_103039_object108683:head# 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483c715c:::benchmark_data_smithi194_103039_object78819:head# 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483c9ea8:::benchmark_data_smithi194_103039_object154573:head# 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483cc1ff:::benchmark_data_smithi194_103039_object59205:head# 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483e8d17:::benchmark_data_smithi194_103039_object93825:head# 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:483e995f:::benchmark_data_smithi194_103039_object111051:head# 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4840497c:::benchmark_data_smithi194_103039_object143609:head# 2024-04-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48406f97:::benchmark_data_smithi194_103039_object52377:head# 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484165b7:::benchmark_data_smithi194_103039_object50778:head# 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4841ce1e:::benchmark_data_smithi194_103039_object119807:head# 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4843f48e:::benchmark_data_smithi194_103039_object93518:head# 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4843f741:::benchmark_data_smithi194_103039_object100980:head# 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48441d35:::benchmark_data_smithi194_103039_object95147:head# 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4844778c:::benchmark_data_smithi194_103039_object127750:head# 2024-04-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48452263:::benchmark_data_smithi194_103039_object55301:head# 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4845fd91:::benchmark_data_smithi194_103039_object58982:head# 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48463bc5:::benchmark_data_smithi194_103039_object76691:head# 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4846f31d:::benchmark_data_smithi194_103039_object48745:head# 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484770e6:::benchmark_data_smithi194_103039_object147257:head# 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4848a636:::benchmark_data_smithi194_103039_object114092:head# 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4849edc5:::benchmark_data_smithi194_103039_object99794:head# 2024-04-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484a09c4:::benchmark_data_smithi194_103039_object121836:head# 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484a23be:::benchmark_data_smithi194_103039_object117139:head# 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484bd121:::benchmark_data_smithi194_103039_object102878:head# 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484c05fa:::benchmark_data_smithi194_103039_object107898:head# 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484c7149:::benchmark_data_smithi194_103039_object125122:head# 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484c71f0:::benchmark_data_smithi194_103039_object137480:head# 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484e9d13:::benchmark_data_smithi194_103039_object56894:head# 2024-04-29T14:21:29.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484f5dc4:::benchmark_data_smithi194_103039_object91908:head# 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:484f654d:::benchmark_data_smithi194_103039_object59217:head# 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4850267a:::benchmark_data_smithi194_103039_object65633:head# 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48503934:::benchmark_data_smithi194_103039_object46506:head# 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485061c7:::benchmark_data_smithi194_103039_object88602:head# 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4850830b:::benchmark_data_smithi194_103039_object101656:head# 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48513118:::benchmark_data_smithi194_103039_object58441:head# 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4851669d:::benchmark_data_smithi194_103039_object147479:head# 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485189b4:::benchmark_data_smithi194_103039_object88988:head# 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4851af26:::benchmark_data_smithi194_103039_object139271:head# 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48527b1f:::benchmark_data_smithi194_103039_object124159:head# 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4852a888:::benchmark_data_smithi194_103039_object51618:head# 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485314ca:::benchmark_data_smithi194_103039_object82643:head# 2024-04-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48536c0b:::benchmark_data_smithi194_103039_object51622:head# 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48541108:::benchmark_data_smithi194_103039_object103878:head# 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48545f20:::benchmark_data_smithi194_103039_object109457:head# 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4855974f:::benchmark_data_smithi194_103039_object70747:head# 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4855a5ae:::benchmark_data_smithi194_103039_object93498:head# 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4855a7e0:::benchmark_data_smithi194_103039_object73790:head# 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4855c641:::benchmark_data_smithi194_103039_object103904:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48561f2e:::benchmark_data_smithi194_103039_object46032:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4856332b:::benchmark_data_smithi194_103039_object88520:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485721a2:::benchmark_data_smithi194_103039_object149785:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4857f32b:::benchmark_data_smithi194_103039_object43420:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4857feeb:::benchmark_data_smithi194_103039_object60645:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48586eb5:::benchmark_data_smithi194_103039_object113580:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48589eb8:::benchmark_data_smithi194_103039_object111417:head# 2024-04-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485904bd:::benchmark_data_smithi194_103039_object79031:head# 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48595790:::benchmark_data_smithi194_103039_object146326:head# 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4859fbad:::benchmark_data_smithi194_103039_object152382:head# 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485aeeb7:::benchmark_data_smithi194_103039_object47848:head# 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485ba758:::benchmark_data_smithi194_103039_object60574:head# 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485c86f2:::benchmark_data_smithi194_103039_object125913:head# 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485c972c:::benchmark_data_smithi194_103039_object129505:head# 2024-04-29T14:21:29.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:485dd370:::benchmark_data_smithi194_103039_object66799:head# 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486045d1:::benchmark_data_smithi194_103039_object105343:head# 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48606758:::benchmark_data_smithi194_103039_object83535:head# 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48609c2c:::benchmark_data_smithi194_103039_object129719:head# 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4860b39b:::benchmark_data_smithi194_103039_object110580:head# 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4860d6db:::benchmark_data_smithi194_103039_object103917:head# 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4861cedf:::benchmark_data_smithi194_103039_object134821:head# 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4861d50a:::benchmark_data_smithi194_103039_object84566:head# 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486506d7:::benchmark_data_smithi194_103039_object149061:head# 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4865b437:::benchmark_data_smithi194_103039_object155136:head# 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48663cba:::benchmark_data_smithi194_103039_object93416:head# 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48669097:::benchmark_data_smithi194_103039_object59728:head# 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4866a07f:::benchmark_data_smithi194_103039_object131820:head# 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4866d686:::benchmark_data_smithi194_103039_object141702:head# 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4866dbd8:::benchmark_data_smithi194_103039_object63552:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4867d82c:::benchmark_data_smithi194_103039_object42391:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4867dc08:::benchmark_data_smithi194_103039_object96457:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4868b019:::benchmark_data_smithi194_103039_object102635:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48690f59:::benchmark_data_smithi194_103039_object103155:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48696030:::benchmark_data_smithi194_103039_object69202:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486bc366:::benchmark_data_smithi194_103039_object77029:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486be452:::benchmark_data_smithi194_103039_object56271:head# 2024-04-29T14:21:29.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486dc02e:::benchmark_data_smithi194_103039_object99099:head# 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486dd333:::benchmark_data_smithi194_103039_object69309:head# 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486eb272:::benchmark_data_smithi194_103039_object65807:head# 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486eb5d9:::benchmark_data_smithi194_103039_object139631:head# 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486f891a:::benchmark_data_smithi194_103039_object99454:head# 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:486f939d:::benchmark_data_smithi194_103039_object110860:head# 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48703575:::benchmark_data_smithi194_103039_object55524:head# 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487076c7:::benchmark_data_smithi194_103039_object57424:head# 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48717ce2:::benchmark_data_smithi194_103039_object45462:head# 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4871c1b3:::benchmark_data_smithi194_103039_object67853:head# 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4871cdee:::benchmark_data_smithi194_103039_object130698:head# 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4871e5d4:::benchmark_data_smithi194_103039_object84392:head# 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48734c23:::benchmark_data_smithi194_103039_object104778:head# 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487366a5:::benchmark_data_smithi194_103039_object47974:head# 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48739768:::benchmark_data_smithi194_103039_object151799:head# 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4873a2d9:::benchmark_data_smithi194_103039_object113467:head# 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4873a796:::benchmark_data_smithi194_103039_object130217:head# 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4874dd62:::benchmark_data_smithi194_103039_object83487:head# 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4874e5d7:::benchmark_data_smithi194_103039_object149977:head# 2024-04-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4874e608:::benchmark_data_smithi194_103039_object72946:head# 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4874ebac:::benchmark_data_smithi194_103039_object112333:head# 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4874ec3a:::benchmark_data_smithi194_103039_object55918:head# 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48755f62:::benchmark_data_smithi194_103039_object43503:head# 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48760b5d:::benchmark_data_smithi194_103039_object44847:head# 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48764ada:::benchmark_data_smithi194_103039_object56023:head# 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48783a53:::benchmark_data_smithi194_103039_object135007:head# 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4878ee6f:::benchmark_data_smithi194_103039_object81536:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487a22d6:::benchmark_data_smithi194_103039_object149442:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487a8efa:::benchmark_data_smithi194_103039_object66595:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487aa73b:::benchmark_data_smithi194_103039_object62024:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487b131c:::benchmark_data_smithi194_103039_object44662:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487b1986:::benchmark_data_smithi194_103039_object117429:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487b947a:::benchmark_data_smithi194_103039_object49760:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487c099b:::benchmark_data_smithi194_103039_object155593:head# 2024-04-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487c16d8:::benchmark_data_smithi194_103039_object52098:head# 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487e3739:::benchmark_data_smithi194_103039_object120443:head# 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487eff3e:::benchmark_data_smithi194_103039_object139099:head# 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487f098a:::benchmark_data_smithi194_103039_object111189:head# 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487f109f:::benchmark_data_smithi194_103039_object77105:head# 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487f5f58:::benchmark_data_smithi194_103039_object80624:head# 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487f7915:::benchmark_data_smithi194_103039_object46270:head# 2024-04-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487f847b:::benchmark_data_smithi194_103039_object118976:head# 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:487fd5c7:::benchmark_data_smithi194_103039_object80186:head# 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4880646c:::benchmark_data_smithi194_103039_object52563:head# 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48808af0:::benchmark_data_smithi194_103039_object43145:head# 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48809df4:::benchmark_data_smithi194_103039_object115621:head# 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488143ea:::benchmark_data_smithi194_103039_object123187:head# 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48819e02:::benchmark_data_smithi194_103039_object72018:head# 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48841380:::benchmark_data_smithi194_103039_object131544:head# 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4884705a:::benchmark_data_smithi194_103039_object123205:head# 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4884b67d:::benchmark_data_smithi194_103039_object94134:head# 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48860588:::benchmark_data_smithi194_103039_object131966:head# 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488814e5:::benchmark_data_smithi194_103039_object75395:head# 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48883e21:::benchmark_data_smithi194_103039_object133743:head# 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4888db53:::benchmark_data_smithi194_103039_object51837:head# 2024-04-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4889e4fc:::benchmark_data_smithi194_103039_object148256:head# 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488a5cae:::benchmark_data_smithi194_103039_object91694:head# 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488ae99f:::benchmark_data_smithi194_103039_object109645:head# 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488bbf63:::benchmark_data_smithi194_103039_object133538:head# 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488bc57a:::benchmark_data_smithi194_103039_object134642:head# 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488c4869:::benchmark_data_smithi194_103039_object55281:head# 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488cc2be:::benchmark_data_smithi194_103039_object104394:head# 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488ccf40:::benchmark_data_smithi194_103039_object56701:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488cd0f5:::benchmark_data_smithi194_103039_object62053:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488eb658:::benchmark_data_smithi194_103039_object122668:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488ee4fb:::benchmark_data_smithi194_103039_object140497:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:488ef117:::benchmark_data_smithi194_103039_object43257:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4890b303:::benchmark_data_smithi194_103039_object102933:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48918127:::benchmark_data_smithi194_103039_object102172:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48923325:::benchmark_data_smithi194_103039_object103344:head# 2024-04-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48929377:::benchmark_data_smithi194_103039_object155007:head# 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48933ec5:::benchmark_data_smithi194_103039_object62592:head# 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48935685:::benchmark_data_smithi194_103039_object78774:head# 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48937fe7:::benchmark_data_smithi194_103039_object84734:head# 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4893b4d5:::benchmark_data_smithi194_103039_object122421:head# 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4895f46c:::benchmark_data_smithi194_103039_object114729:head# 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4896591c:::benchmark_data_smithi194_103039_object89179:head# 2024-04-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4896b470:::benchmark_data_smithi194_103039_object127864:head# 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4896e6e8:::benchmark_data_smithi194_103039_object103592:head# 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4897dfcc:::benchmark_data_smithi194_103039_object140757:head# 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4899928e:::benchmark_data_smithi194_103039_object82735:head# 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489a2383:::benchmark_data_smithi194_103039_object98859:head# 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489ae123:::benchmark_data_smithi194_103039_object110467:head# 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489b1ca6:::benchmark_data_smithi194_103039_object146985:head# 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489b1e69:::benchmark_data_smithi194_103039_object51849:head# 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489b8c3b:::benchmark_data_smithi194_103039_object101229:head# 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489c6468:::benchmark_data_smithi194_103039_object64551:head# 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489c6f98:::benchmark_data_smithi194_103039_object83601:head# 2024-04-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489e50aa:::benchmark_data_smithi194_103039_object46454:head# 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489e9a11:::benchmark_data_smithi194_103039_object112954:head# 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:489f8ac0:::benchmark_data_smithi194_103039_object91440:head# 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a05a83:::benchmark_data_smithi194_103039_object121156:head# 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a1427f:::benchmark_data_smithi194_103039_object134233:head# 2024-04-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a1c922:::benchmark_data_smithi194_103039_object150474:head# 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a2248a:::benchmark_data_smithi194_103039_object61822:head# 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a27558:::benchmark_data_smithi194_103039_object86182:head# 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a2f16a:::benchmark_data_smithi194_103039_object141179:head# 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a37466:::benchmark_data_smithi194_103039_object43493:head# 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a56b28:::benchmark_data_smithi194_103039_object114753:head# 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a76b13:::benchmark_data_smithi194_103039_object61838:head# 2024-04-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a774ba:::benchmark_data_smithi194_103039_object98284:head# 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a7a9c7:::benchmark_data_smithi194_103039_object86036:head# 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a7b15e:::benchmark_data_smithi194_103039_object74401:head# 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a90cb1:::benchmark_data_smithi194_103039_object64300:head# 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a941dc:::benchmark_data_smithi194_103039_object126443:head# 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a94a96:::benchmark_data_smithi194_103039_object140416:head# 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48a9ae8a:::benchmark_data_smithi194_103039_object156585:head# 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48aa19ab:::benchmark_data_smithi194_103039_object46749:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48aa72e9:::benchmark_data_smithi194_103039_object139991:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48aade03:::benchmark_data_smithi194_103039_object100109:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48acb2e1:::benchmark_data_smithi194_103039_object85498:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48acc351:::benchmark_data_smithi194_103039_object123220:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ad11a6:::benchmark_data_smithi194_103039_object96659:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ad5f98:::benchmark_data_smithi194_103039_object88334:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ad6e69:::benchmark_data_smithi194_103039_object109640:head# 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48aec045:::benchmark_data_smithi194_103039_object59421:head# 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48af133c:::benchmark_data_smithi194_103039_object52773:head# 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48af55a2:::benchmark_data_smithi194_103039_object129909:head# 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48afa78a:::benchmark_data_smithi194_103039_object102481:head# 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b14ca3:::benchmark_data_smithi194_103039_object92916:head# 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b18141:::benchmark_data_smithi194_103039_object97629:head# 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b18cf7:::benchmark_data_smithi194_103039_object79396:head# 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b1ac87:::benchmark_data_smithi194_103039_object48208:head# 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b299cb:::benchmark_data_smithi194_103039_object75428:head# 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b3898d:::benchmark_data_smithi194_103039_object123925:head# 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b4a677:::benchmark_data_smithi194_103039_object44473:head# 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b6b479:::benchmark_data_smithi194_103039_object115035:head# 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b7b33a:::benchmark_data_smithi194_103039_object73850:head# 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b81c8c:::benchmark_data_smithi194_103039_object135715:head# 2024-04-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b86afc:::benchmark_data_smithi194_103039_object60469:head# 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b8f3cc:::benchmark_data_smithi194_103039_object126227:head# 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b99a04:::benchmark_data_smithi194_103039_object43454:head# 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48b9e0ba:::benchmark_data_smithi194_103039_object98084:head# 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ba2ede:::benchmark_data_smithi194_103039_object98697:head# 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ba830b:::benchmark_data_smithi194_103039_object92561:head# 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bb4c1b:::benchmark_data_smithi194_103039_object153731:head# 2024-04-29T14:21:29.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bc0403:::benchmark_data_smithi194_103039_object59731:head# 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bc4189:::benchmark_data_smithi194_103039_object46676:head# 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bc5455:::benchmark_data_smithi194_103039_object156091:head# 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bcca4a:::benchmark_data_smithi194_103039_object128314:head# 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bcf13c:::benchmark_data_smithi194_103039_object43221:head# 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bd7e12:::benchmark_data_smithi194_103039_object100943:head# 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bda311:::benchmark_data_smithi194_103039_object131265:head# 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48be37e7:::benchmark_data_smithi194_103039_object109217:head# 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48be8a92:::benchmark_data_smithi194_103039_object79532:head# 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48beb587:::benchmark_data_smithi194_103039_object94507:head# 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bf51a9:::benchmark_data_smithi194_103039_object53675:head# 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bfe6d3:::benchmark_data_smithi194_103039_object89392:head# 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48bff397:::benchmark_data_smithi194_103039_object139693:head# 2024-04-29T14:21:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c16997:::benchmark_data_smithi194_103039_object52178:head# 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c173ca:::benchmark_data_smithi194_103039_object53058:head# 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c224d7:::benchmark_data_smithi194_103039_object97833:head# 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c246f0:::benchmark_data_smithi194_103039_object102446:head# 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c25a91:::benchmark_data_smithi194_103039_object105240:head# 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c350c9:::benchmark_data_smithi194_103039_object131426:head# 2024-04-29T14:21:29.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c41b14:::benchmark_data_smithi194_103039_object113930:head# 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c429f1:::benchmark_data_smithi194_103039_object127879:head# 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c430be:::benchmark_data_smithi194_103039_object60366:head# 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c4a006:::benchmark_data_smithi194_103039_object137290:head# 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c4e435:::benchmark_data_smithi194_103039_object105987:head# 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c79a64:::benchmark_data_smithi194_103039_object140277:head# 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c79dc6:::benchmark_data_smithi194_103039_object103607:head# 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c7a6b4:::benchmark_data_smithi194_103039_object66293:head# 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c8cc0f:::benchmark_data_smithi194_103039_object69866:head# 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c8d438:::benchmark_data_smithi194_103039_object139088:head# 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48c98835:::benchmark_data_smithi194_103039_object121506:head# 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ca42f9:::benchmark_data_smithi194_103039_object98698:head# 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ca85e4:::benchmark_data_smithi194_103039_object99928:head# 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48cb2c08:::benchmark_data_smithi194_103039_object42943:head# 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48cb4ebe:::benchmark_data_smithi194_103039_object51756:head# 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48cc7132:::benchmark_data_smithi194_103039_object151079:head# 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ccb4ed:::benchmark_data_smithi194_103039_object66573:head# 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48cd95b2:::benchmark_data_smithi194_103039_object128497:head# 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ce5d65:::benchmark_data_smithi194_103039_object134620:head# 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ceb580:::benchmark_data_smithi194_103039_object135117:head# 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ced875:::benchmark_data_smithi194_103039_object116133:head# 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ceeb53:::benchmark_data_smithi194_103039_object73177:head# 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ceef66:::benchmark_data_smithi194_103039_object128912:head# 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d09bde:::benchmark_data_smithi194_103039_object140647:head# 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d0e2d1:::benchmark_data_smithi194_103039_object147043:head# 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d1547a:::benchmark_data_smithi194_103039_object97160:head# 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d1b3c6:::benchmark_data_smithi194_103039_object77608:head# 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d25d64:::benchmark_data_smithi194_103039_object146202:head# 2024-04-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d266e0:::benchmark_data_smithi194_103039_object62889:head# 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d3dfcc:::benchmark_data_smithi194_103039_object95889:head# 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d3e366:::benchmark_data_smithi194_103039_object139855:head# 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d426fe:::benchmark_data_smithi194_103039_object98439:head# 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d45780:::benchmark_data_smithi194_103039_object77342:head# 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d460f4:::benchmark_data_smithi194_103039_object114004:head# 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d545e9:::benchmark_data_smithi194_103039_object92381:head# 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d56ee9:::benchmark_data_smithi194_103039_object85079:head# 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d62529:::benchmark_data_smithi194_103039_object81717:head# 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d6e946:::benchmark_data_smithi194_103039_object75269:head# 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d70853:::benchmark_data_smithi194_103039_object62115:head# 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d7b472:::benchmark_data_smithi194_103039_object84762:head# 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d7b53a:::benchmark_data_smithi194_103039_object114717:head# 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d7e0fb:::benchmark_data_smithi194_103039_object135636:head# 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d86636:::benchmark_data_smithi194_103039_object133550:head# 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d989e9:::benchmark_data_smithi194_103039_object144808:head# 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d9d3ce:::benchmark_data_smithi194_103039_object66572:head# 2024-04-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48d9de9f:::benchmark_data_smithi194_103039_object72766:head# 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48dab77c:::benchmark_data_smithi194_103039_object48658:head# 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48dadc3a:::benchmark_data_smithi194_103039_object66891:head# 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48db0f2d:::benchmark_data_smithi194_103039_object132125:head# 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48db861b:::benchmark_data_smithi194_103039_object96856:head# 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48dc9f81:::benchmark_data_smithi194_103039_object150437:head# 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48dcf98b:::benchmark_data_smithi194_103039_object88113:head# 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48de0c6b:::benchmark_data_smithi194_103039_object95366:head# 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48de3ed7:::benchmark_data_smithi194_103039_object146171:head# 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48dedbb3:::benchmark_data_smithi194_103039_object152739:head# 2024-04-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48dfa1f1:::benchmark_data_smithi194_103039_object71154:head# 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e02f5e:::benchmark_data_smithi194_103039_object77148:head# 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e16987:::benchmark_data_smithi194_103039_object146851:head# 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e19192:::benchmark_data_smithi194_103039_object142778:head# 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e1b992:::benchmark_data_smithi194_103039_object135403:head# 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e1fbd5:::benchmark_data_smithi194_103039_object68836:head# 2024-04-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e28e6d:::benchmark_data_smithi194_103039_object131716:head# 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e2e0de:::benchmark_data_smithi194_103039_object82215:head# 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e3824b:::benchmark_data_smithi194_103039_object131745:head# 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e462ef:::benchmark_data_smithi194_103039_object102524:head# 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e7cdbd:::benchmark_data_smithi194_103039_object64102:head# 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e7d251:::benchmark_data_smithi194_103039_object143822:head# 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e84f2c:::benchmark_data_smithi194_103039_object141196:head# 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e8a917:::benchmark_data_smithi194_103039_object124797:head# 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e8d87f:::benchmark_data_smithi194_103039_object118080:head# 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48e95051:::benchmark_data_smithi194_103039_object128470:head# 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48eb0886:::benchmark_data_smithi194_103039_object121307:head# 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48eb5d45:::benchmark_data_smithi194_103039_object53668:head# 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48eb6f6d:::benchmark_data_smithi194_103039_object137613:head# 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ece246:::benchmark_data_smithi194_103039_object142132:head# 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ed349e:::benchmark_data_smithi194_103039_object101191:head# 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48edcb7f:::benchmark_data_smithi194_103039_object129521:head# 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48edf272:::benchmark_data_smithi194_103039_object80256:head# 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ef37d8:::benchmark_data_smithi194_103039_object148408:head# 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ef966e:::benchmark_data_smithi194_103039_object55302:head# 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48efa5a6:::benchmark_data_smithi194_103039_object53129:head# 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f0148f:::benchmark_data_smithi194_103039_object84595:head# 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f1abf4:::benchmark_data_smithi194_103039_object116162:head# 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f1ec60:::benchmark_data_smithi194_103039_object155516:head# 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f1ed1c:::benchmark_data_smithi194_103039_object56795:head# 2024-04-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f21f0b:::benchmark_data_smithi194_103039_object153205:head# 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f29016:::benchmark_data_smithi194_103039_object74679:head# 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f360f8:::benchmark_data_smithi194_103039_object65076:head# 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f48006:::benchmark_data_smithi194_103039_object73477:head# 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f48a18:::benchmark_data_smithi194_103039_object103735:head# 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.857 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f53b1f:::benchmark_data_smithi194_103039_object141010:head# 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f56314:::benchmark_data_smithi194_103039_object104459:head# 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f5cef5:::benchmark_data_smithi194_103039_object127218:head# 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f63529:::benchmark_data_smithi194_103039_object132822:head# 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f638c8:::benchmark_data_smithi194_103039_object124877:head# 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f6a92a:::benchmark_data_smithi194_103039_object44556:head# 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f82d8d:::benchmark_data_smithi194_103039_object150834:head# 2024-04-29T14:21:29.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f85887:::benchmark_data_smithi194_103039_object145193:head# 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f894bb:::benchmark_data_smithi194_103039_object73270:head# 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f90c40:::benchmark_data_smithi194_103039_object115266:head# 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f9252f:::benchmark_data_smithi194_103039_object92296:head# 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f984c5:::benchmark_data_smithi194_103039_object59118:head# 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f99ddb:::benchmark_data_smithi194_103039_object61377:head# 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48f9e6a7:::benchmark_data_smithi194_103039_object115475:head# 2024-04-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48faca39:::benchmark_data_smithi194_103039_object156094:head# 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48fb76e6:::benchmark_data_smithi194_103039_object73966:head# 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48fd2756:::benchmark_data_smithi194_103039_object95036:head# 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ff359c:::benchmark_data_smithi194_103039_object108035:head# 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48ff575c:::benchmark_data_smithi194_103039_object48266:head# 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:48fff605:::benchmark_data_smithi194_103039_object48845:head# 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49016008:::benchmark_data_smithi194_103039_object85779:head# 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.860 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490194af:::benchmark_data_smithi194_103039_object115739:head# 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49024fd8:::benchmark_data_smithi194_103039_object62474:head# 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49029c3e:::benchmark_data_smithi194_103039_object138978:head# 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4903184a:::benchmark_data_smithi194_103039_object137594:head# 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49033567:::benchmark_data_smithi194_103039_object141261:head# 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49034678:::benchmark_data_smithi194_103039_object119975:head# 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4903b025:::benchmark_data_smithi194_103039_object115233:head# 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49045f93:::benchmark_data_smithi194_103039_object104900:head# 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4904a760:::benchmark_data_smithi194_103039_object124515:head# 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4904ce23:::benchmark_data_smithi194_103039_object96400:head# 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49078e5f:::benchmark_data_smithi194_103039_object64044:head# 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4907a46c:::benchmark_data_smithi194_103039_object54975:head# 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4907c829:::benchmark_data_smithi194_103039_object154488:head# 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4908bdc9:::benchmark_data_smithi194_103039_object75366:head# 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4908f2a2:::benchmark_data_smithi194_103039_object59129:head# 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490948a9:::benchmark_data_smithi194_103039_object50855:head# 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490b67f9:::benchmark_data_smithi194_103039_object68315:head# 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490c5ade:::benchmark_data_smithi194_103039_object148308:head# 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490d322b:::benchmark_data_smithi194_103039_object127514:head# 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490d7413:::benchmark_data_smithi194_103039_object79341:head# 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490d7b05:::benchmark_data_smithi194_103039_object103685:head# 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490ea528:::benchmark_data_smithi194_103039_object155202:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490ed5da:::benchmark_data_smithi194_103039_object60283:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490f531f:::benchmark_data_smithi194_103039_object156511:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490f7538:::benchmark_data_smithi194_103039_object90905:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:490fd89d:::benchmark_data_smithi194_103039_object121452:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49103d66:::benchmark_data_smithi194_103039_object52410:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4911b1ef:::benchmark_data_smithi194_103039_object57142:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49129131:::benchmark_data_smithi194_103039_object52270:head# 2024-04-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49132575:::benchmark_data_smithi194_103039_object88555:head# 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4913475d:::benchmark_data_smithi194_103039_object114887:head# 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4913a18f:::benchmark_data_smithi194_103039_object117593:head# 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49148879:::benchmark_data_smithi194_103039_object73869:head# 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4914b545:::benchmark_data_smithi194_103039_object149669:head# 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49158d7c:::benchmark_data_smithi194_103039_object77891:head# 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4915d975:::benchmark_data_smithi194_103039_object103822:head# 2024-04-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4915ed47:::benchmark_data_smithi194_103039_object105045:head# 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491604b1:::benchmark_data_smithi194_103039_object142771:head# 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49165c95:::benchmark_data_smithi194_103039_object86556:head# 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4916c51b:::benchmark_data_smithi194_103039_object104042:head# 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49177107:::benchmark_data_smithi194_103039_object121493:head# 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4917ef53:::benchmark_data_smithi194_103039_object50856:head# 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491812a4:::benchmark_data_smithi194_103039_object77623:head# 2024-04-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4918db76:::benchmark_data_smithi194_103039_object64563:head# 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4919b668:::benchmark_data_smithi194_103039_object46384:head# 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4919e8fa:::benchmark_data_smithi194_103039_object95089:head# 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491adbbc:::benchmark_data_smithi194_103039_object103332:head# 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491b8c17:::benchmark_data_smithi194_103039_object48394:head# 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491bc401:::benchmark_data_smithi194_103039_object105037:head# 2024-04-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491bf864:::benchmark_data_smithi194_103039_object118203:head# 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491c1490:::benchmark_data_smithi194_103039_object138152:head# 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491d9b7a:::benchmark_data_smithi194_103039_object145392:head# 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491db3e0:::benchmark_data_smithi194_103039_object101950:head# 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491f7302:::benchmark_data_smithi194_103039_object148041:head# 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491fa9a6:::benchmark_data_smithi194_103039_object136689:head# 2024-04-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:491fd41c:::benchmark_data_smithi194_103039_object107488:head# 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4920e936:::benchmark_data_smithi194_103039_object97706:head# 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4920ee45:::benchmark_data_smithi194_103039_object144431:head# 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49223433:::benchmark_data_smithi194_103039_object125485:head# 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49225981:::benchmark_data_smithi194_103039_object54881:head# 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49237abc:::benchmark_data_smithi194_103039_object45995:head# 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49238fdd:::benchmark_data_smithi194_103039_object110405:head# 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49239c1b:::benchmark_data_smithi194_103039_object136707:head# 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4923a6af:::benchmark_data_smithi194_103039_object97863:head# 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4923edf5:::benchmark_data_smithi194_103039_object114877:head# 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492406e0:::benchmark_data_smithi194_103039_object95961:head# 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49240feb:::benchmark_data_smithi194_103039_object72361:head# 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49247d0c:::benchmark_data_smithi194_103039_object76683:head# 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492500b2:::benchmark_data_smithi194_103039_object92587:head# 2024-04-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4925a60d:::benchmark_data_smithi194_103039_object122537:head# 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4925c03a:::benchmark_data_smithi194_103039_object141312:head# 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49277ba9:::benchmark_data_smithi194_103039_object94129:head# 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4927d016:::benchmark_data_smithi194_103039_object111582:head# 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49288b26:::benchmark_data_smithi194_103039_object65804:head# 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492988c8:::benchmark_data_smithi194_103039_object154699:head# 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492a28d6:::benchmark_data_smithi194_103039_object112943:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492a4f6e:::benchmark_data_smithi194_103039_object94845:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492b7b14:::benchmark_data_smithi194_103039_object113412:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492bf5cf:::benchmark_data_smithi194_103039_object101100:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492c854b:::benchmark_data_smithi194_103039_object114205:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492cf821:::benchmark_data_smithi194_103039_object126029:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492da282:::benchmark_data_smithi194_103039_object44246:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492dd6b1:::benchmark_data_smithi194_103039_object121007:head# 2024-04-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492e959f:::benchmark_data_smithi194_103039_object102514:head# 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492f4b42:::benchmark_data_smithi194_103039_object56823:head# 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492f626a:::benchmark_data_smithi194_103039_object89678:head# 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492f6713:::benchmark_data_smithi194_103039_object80841:head# 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:492fdf5c:::benchmark_data_smithi194_103039_object151559:head# 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4930610c:::benchmark_data_smithi194_103039_object124633:head# 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4930cd9f:::benchmark_data_smithi194_103039_object129749:head# 2024-04-29T14:21:29.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49311e5c:::benchmark_data_smithi194_103039_object142860:head# 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4931525b:::benchmark_data_smithi194_103039_object131039:head# 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4931571e:::benchmark_data_smithi194_103039_object88825:head# 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4931e728:::benchmark_data_smithi194_103039_object105482:head# 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49326ec8:::benchmark_data_smithi194_103039_object81201:head# 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49335904:::benchmark_data_smithi194_103039_object64661:head# 2024-04-29T14:21:29.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493385be:::benchmark_data_smithi194_103039_object71556:head# 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4933ae12:::benchmark_data_smithi194_103039_object57835:head# 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4933afca:::benchmark_data_smithi194_103039_object43544:head# 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49349983:::benchmark_data_smithi194_103039_object94505:head# 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4934b390:::benchmark_data_smithi194_103039_object60134:head# 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4934d07d:::benchmark_data_smithi194_103039_object112004:head# 2024-04-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49361873:::benchmark_data_smithi194_103039_object67341:head# 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493877d1:::benchmark_data_smithi194_103039_object91888:head# 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4938a0f6:::benchmark_data_smithi194_103039_object98067:head# 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4938b597:::benchmark_data_smithi194_103039_object88841:head# 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4938c2f3:::benchmark_data_smithi194_103039_object63619:head# 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49394915:::benchmark_data_smithi194_103039_object124648:head# 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49399be6:::benchmark_data_smithi194_103039_object53841:head# 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4939b2a6:::benchmark_data_smithi194_103039_object90489:head# 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4939f5bc:::benchmark_data_smithi194_103039_object155304:head# 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493a879a:::benchmark_data_smithi194_103039_object61957:head# 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493b7b17:::benchmark_data_smithi194_103039_object52969:head# 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493b956b:::benchmark_data_smithi194_103039_object132108:head# 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493c8e70:::benchmark_data_smithi194_103039_object152693:head# 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493ca6c2:::benchmark_data_smithi194_103039_object45430:head# 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493cc3db:::benchmark_data_smithi194_103039_object46630:head# 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493d4ad4:::benchmark_data_smithi194_103039_object86901:head# 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493e1066:::benchmark_data_smithi194_103039_object55599:head# 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493e49c6:::benchmark_data_smithi194_103039_object139035:head# 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493e7f9e:::benchmark_data_smithi194_103039_object138919:head# 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493ed72c:::benchmark_data_smithi194_103039_object117449:head# 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:493f343f:::benchmark_data_smithi194_103039_object49147:head# 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494040fc:::benchmark_data_smithi194_103039_object63754:head# 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49406d01:::benchmark_data_smithi194_103039_object100803:head# 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49413802:::benchmark_data_smithi194_103039_object62858:head# 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49417a9a:::benchmark_data_smithi194_103039_object138157:head# 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494228c5:::benchmark_data_smithi194_103039_object46251:head# 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49423067:::benchmark_data_smithi194_103039_object57710:head# 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49426f80:::benchmark_data_smithi194_103039_object48216:head# 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4943203d:::benchmark_data_smithi194_103039_object105537:head# 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4943d4fc:::benchmark_data_smithi194_103039_object96216:head# 2024-04-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4944e91f:::benchmark_data_smithi194_103039_object125236:head# 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494576c5:::benchmark_data_smithi194_103039_object90289:head# 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49457ca9:::benchmark_data_smithi194_103039_object115072:head# 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4945cfd7:::benchmark_data_smithi194_103039_object79407:head# 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49461798:::benchmark_data_smithi194_103039_object89586:head# 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49461fb3:::benchmark_data_smithi194_103039_object71347:head# 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4947ee91:::benchmark_data_smithi194_103039_object135686:head# 2024-04-29T14:21:29.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49493d7f:::benchmark_data_smithi194_103039_object80596:head# 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4949b94a:::benchmark_data_smithi194_103039_object42348:head# 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494a6c57:::benchmark_data_smithi194_103039_object58463:head# 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494a8daa:::benchmark_data_smithi194_103039_object118478:head# 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494abb09:::benchmark_data_smithi194_103039_object57016:head# 2024-04-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494b033e:::benchmark_data_smithi194_103039_object129866:head# 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494b4516:::benchmark_data_smithi194_103039_object120139:head# 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494b9825:::benchmark_data_smithi194_103039_object99994:head# 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494bb878:::benchmark_data_smithi194_103039_object147973:head# 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494becc3:::benchmark_data_smithi194_103039_object70636:head# 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494c0df2:::benchmark_data_smithi194_103039_object101880:head# 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494c2745:::benchmark_data_smithi194_103039_object122790:head# 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494cd4a9:::benchmark_data_smithi194_103039_object48583:head# 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494e2aa7:::benchmark_data_smithi194_103039_object130726:head# 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494ee19a:::benchmark_data_smithi194_103039_object96609:head# 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:494fd6bf:::benchmark_data_smithi194_103039_object109986:head# 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49500e4d:::benchmark_data_smithi194_103039_object55402:head# 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495087fc:::benchmark_data_smithi194_103039_object148668:head# 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49512a74:::benchmark_data_smithi194_103039_object64792:head# 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49515d7d:::benchmark_data_smithi194_103039_object154061:head# 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49521637:::benchmark_data_smithi194_103039_object77468:head# 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4952546d:::benchmark_data_smithi194_103039_object153148:head# 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49527164:::benchmark_data_smithi194_103039_object137739:head# 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49535f43:::benchmark_data_smithi194_103039_object43250:head# 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4953a963:::benchmark_data_smithi194_103039_object44153:head# 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49545027:::benchmark_data_smithi194_103039_object90479:head# 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4954569c:::benchmark_data_smithi194_103039_object107770:head# 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4955772e:::benchmark_data_smithi194_103039_object77791:head# 2024-04-29T14:21:29.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495619d1:::benchmark_data_smithi194_103039_object104341:head# 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49565ad7:::benchmark_data_smithi194_103039_object102397:head# 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49569a5b:::benchmark_data_smithi194_103039_object118497:head# 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4956f703:::benchmark_data_smithi194_103039_object90042:head# 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49582f13:::benchmark_data_smithi194_103039_object48606:head# 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495844a7:::benchmark_data_smithi194_103039_object93456:head# 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4958a9c6:::benchmark_data_smithi194_103039_object112863:head# 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4959d31f:::benchmark_data_smithi194_103039_object84427:head# 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4959ee3b:::benchmark_data_smithi194_103039_object53039:head# 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4959efe1:::benchmark_data_smithi194_103039_object154154:head# 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495a2af3:::benchmark_data_smithi194_103039_object127962:head# 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495aef34:::benchmark_data_smithi194_103039_object127056:head# 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495b2ab4:::benchmark_data_smithi194_103039_object102660:head# 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495bd35c:::benchmark_data_smithi194_103039_object70579:head# 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495c96cf:::benchmark_data_smithi194_103039_object155005:head# 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495cab21:::benchmark_data_smithi194_103039_object135409:head# 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495d479d:::benchmark_data_smithi194_103039_object99336:head# 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495db073:::benchmark_data_smithi194_103039_object104052:head# 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495e4520:::benchmark_data_smithi194_103039_object77479:head# 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495ed8a6:::benchmark_data_smithi194_103039_object43341:head# 2024-04-29T14:21:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495f56a6:::benchmark_data_smithi194_103039_object57439:head# 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:495fe4a0:::benchmark_data_smithi194_103039_object108690:head# 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4960295a:::benchmark_data_smithi194_103039_object81431:head# 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4961c5e1:::benchmark_data_smithi194_103039_object88712:head# 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4961d70d:::benchmark_data_smithi194_103039_object61786:head# 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4961dca9:::benchmark_data_smithi194_103039_object91579:head# 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4961eb46:::benchmark_data_smithi194_103039_object137746:head# 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49627eea:::benchmark_data_smithi194_103039_object130635:head# 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4962984e:::benchmark_data_smithi194_103039_object42194:head# 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49636879:::benchmark_data_smithi194_103039_object76000:head# 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496389bc:::benchmark_data_smithi194_103039_object145447:head# 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496451fe:::benchmark_data_smithi194_103039_object51937:head# 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4964f313:::benchmark_data_smithi194_103039_object84090:head# 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496537bf:::benchmark_data_smithi194_103039_object95187:head# 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496654e9:::benchmark_data_smithi194_103039_object48270:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49666016:::benchmark_data_smithi194_103039_object145772:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49674543:::benchmark_data_smithi194_103039_object103133:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4967499a:::benchmark_data_smithi194_103039_object79117:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49677afd:::benchmark_data_smithi194_103039_object47623:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49678aa3:::benchmark_data_smithi194_103039_object51951:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49693f61:::benchmark_data_smithi194_103039_object71752:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4969f959:::benchmark_data_smithi194_103039_object130751:head# 2024-04-29T14:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496a1b70:::benchmark_data_smithi194_103039_object80534:head# 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496a8496:::benchmark_data_smithi194_103039_object132819:head# 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496adc74:::benchmark_data_smithi194_103039_object72373:head# 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496c0457:::benchmark_data_smithi194_103039_object132932:head# 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496c1639:::benchmark_data_smithi194_103039_object60711:head# 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496c1c75:::benchmark_data_smithi194_103039_object78310:head# 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496c4c77:::benchmark_data_smithi194_103039_object85852:head# 2024-04-29T14:21:29.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496cb60e:::benchmark_data_smithi194_103039_object66221:head# 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496d8f83:::benchmark_data_smithi194_103039_object122963:head# 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496dc34f:::benchmark_data_smithi194_103039_object77202:head# 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496f299b:::benchmark_data_smithi194_103039_object151808:head# 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:496f81fc:::benchmark_data_smithi194_103039_object62931:head# 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497093b8:::benchmark_data_smithi194_103039_object112362:head# 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4971425d:::benchmark_data_smithi194_103039_object48695:head# 2024-04-29T14:21:29.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497233fa:::benchmark_data_smithi194_103039_object74213:head# 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4972a72f:::benchmark_data_smithi194_103039_object121102:head# 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497341f0:::benchmark_data_smithi194_103039_object50882:head# 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49736623:::benchmark_data_smithi194_103039_object47206:head# 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4974448f:::benchmark_data_smithi194_103039_object57588:head# 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49753148:::benchmark_data_smithi194_103039_object59763:head# 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49757468:::benchmark_data_smithi194_103039_object60294:head# 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497649a4:::benchmark_data_smithi194_103039_object117802:head# 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49765a6c:::benchmark_data_smithi194_103039_object113943:head# 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4976bfc5:::benchmark_data_smithi194_103039_object113531:head# 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4977a177:::benchmark_data_smithi194_103039_object78886:head# 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49780d76:::benchmark_data_smithi194_103039_object61401:head# 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497980da:::benchmark_data_smithi194_103039_object83070:head# 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497ad2de:::benchmark_data_smithi194_103039_object55138:head# 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497c59db:::benchmark_data_smithi194_103039_object131227:head# 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497d4dc3:::benchmark_data_smithi194_103039_object140366:head# 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497e4f75:::benchmark_data_smithi194_103039_object71072:head# 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:497e8988:::benchmark_data_smithi194_103039_object130498:head# 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49804b1b:::benchmark_data_smithi194_103039_object148950:head# 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49805a61:::benchmark_data_smithi194_103039_object78531:head# 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49807e49:::benchmark_data_smithi194_103039_object131643:head# 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4980e5d7:::benchmark_data_smithi194_103039_object116805:head# 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4981a527:::benchmark_data_smithi194_103039_object76658:head# 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4981f19e:::benchmark_data_smithi194_103039_object105000:head# 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49830add:::benchmark_data_smithi194_103039_object44234:head# 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49852e10:::benchmark_data_smithi194_103039_object110541:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498803d7:::benchmark_data_smithi194_103039_object102501:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49886bb5:::benchmark_data_smithi194_103039_object72167:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49891c00:::benchmark_data_smithi194_103039_object122700:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498a6428:::benchmark_data_smithi194_103039_object110314:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498afead:::benchmark_data_smithi194_103039_object130117:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498b3c94:::benchmark_data_smithi194_103039_object70992:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498b4347:::benchmark_data_smithi194_103039_object84536:head# 2024-04-29T14:21:29.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498bef43:::benchmark_data_smithi194_103039_object55007:head# 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498c65b3:::benchmark_data_smithi194_103039_object94605:head# 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498c80bd:::benchmark_data_smithi194_103039_object66886:head# 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498c90f1:::benchmark_data_smithi194_103039_object141811:head# 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498d2d4b:::benchmark_data_smithi194_103039_object67964:head# 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498e5f7d:::benchmark_data_smithi194_103039_object53661:head# 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498eea82:::benchmark_data_smithi194_103039_object126747:head# 2024-04-29T14:21:29.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498f02ef:::benchmark_data_smithi194_103039_object96033:head# 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498f1d9a:::benchmark_data_smithi194_103039_object140473:head# 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498f7c13:::benchmark_data_smithi194_103039_object82581:head# 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498fd58c:::benchmark_data_smithi194_103039_object50437:head# 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:498ff1f5:::benchmark_data_smithi194_103039_object93894:head# 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49901eb5:::benchmark_data_smithi194_103039_object118178:head# 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499073a8:::benchmark_data_smithi194_103039_object75823:head# 2024-04-29T14:21:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4990efc6:::benchmark_data_smithi194_103039_object137741:head# 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49918e9c:::benchmark_data_smithi194_103039_object102859:head# 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49923630:::benchmark_data_smithi194_103039_object61545:head# 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4992973e:::benchmark_data_smithi194_103039_object127722:head# 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4992e56e:::benchmark_data_smithi194_103039_object97606:head# 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49930cce:::benchmark_data_smithi194_103039_object96322:head# 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49934b1b:::benchmark_data_smithi194_103039_object62074:head# 2024-04-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4993957f:::benchmark_data_smithi194_103039_object115458:head# 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4993ffdb:::benchmark_data_smithi194_103039_object99612:head# 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499493d2:::benchmark_data_smithi194_103039_object98200:head# 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4995567d:::benchmark_data_smithi194_103039_object102607:head# 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49970db4:::benchmark_data_smithi194_103039_object61025:head# 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4997477e:::benchmark_data_smithi194_103039_object50759:head# 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49975122:::benchmark_data_smithi194_103039_object61250:head# 2024-04-29T14:21:29.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49986001:::benchmark_data_smithi194_103039_object59351:head# 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49991ec2:::benchmark_data_smithi194_103039_object118524:head# 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499966aa:::benchmark_data_smithi194_103039_object104101:head# 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49997141:::benchmark_data_smithi194_103039_object43924:head# 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4999f75b:::benchmark_data_smithi194_103039_object101180:head# 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499a427d:::benchmark_data_smithi194_103039_object86355:head# 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499ab8b1:::benchmark_data_smithi194_103039_object94092:head# 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499afb5b:::benchmark_data_smithi194_103039_object119986:head# 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499b17af:::benchmark_data_smithi194_103039_object98356:head# 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499b4b49:::benchmark_data_smithi194_103039_object150940:head# 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499b78f6:::benchmark_data_smithi194_103039_object83301:head# 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499bb318:::benchmark_data_smithi194_103039_object146436:head# 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499bcc4b:::benchmark_data_smithi194_103039_object89809:head# 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499ca89a:::benchmark_data_smithi194_103039_object138381:head# 2024-04-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499dd5b6:::benchmark_data_smithi194_103039_object74191:head# 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499df0be:::benchmark_data_smithi194_103039_object81807:head# 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499e7b6a:::benchmark_data_smithi194_103039_object98642:head# 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499f4748:::benchmark_data_smithi194_103039_object112102:head# 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:499f55ac:::benchmark_data_smithi194_103039_object110225:head# 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a077da:::benchmark_data_smithi194_103039_object113044:head# 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a201ce:::benchmark_data_smithi194_103039_object139936:head# 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a282b6:::benchmark_data_smithi194_103039_object50559:head# 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a2907d:::benchmark_data_smithi194_103039_object62801:head# 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a31690:::benchmark_data_smithi194_103039_object143486:head# 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a39949:::benchmark_data_smithi194_103039_object46349:head# 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a481d0:::benchmark_data_smithi194_103039_object58648:head# 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a4b5ba:::benchmark_data_smithi194_103039_object102258:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a78760:::benchmark_data_smithi194_103039_object42675:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a7da78:::benchmark_data_smithi194_103039_object71623:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a7e02e:::benchmark_data_smithi194_103039_object54629:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a8536d:::benchmark_data_smithi194_103039_object89552:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49a885c8:::benchmark_data_smithi194_103039_object94835:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49aa9d2c:::benchmark_data_smithi194_103039_object92517:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ab7992:::benchmark_data_smithi194_103039_object50683:head# 2024-04-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ab9480:::benchmark_data_smithi194_103039_object88468:head# 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49aba46f:::benchmark_data_smithi194_103039_object103345:head# 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ac997f:::benchmark_data_smithi194_103039_object103380:head# 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b00443:::benchmark_data_smithi194_103039_object113237:head# 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b174bd:::benchmark_data_smithi194_103039_object65983:head# 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b23128:::benchmark_data_smithi194_103039_object147572:head# 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b23270:::benchmark_data_smithi194_103039_object89523:head# 2024-04-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b30f86:::benchmark_data_smithi194_103039_object59965:head# 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b311d2:::benchmark_data_smithi194_103039_object140728:head# 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b3bb16:::benchmark_data_smithi194_103039_object68165:head# 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b47484:::benchmark_data_smithi194_103039_object50760:head# 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b49c96:::benchmark_data_smithi194_103039_object107791:head# 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b53751:::benchmark_data_smithi194_103039_object49271:head# 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b5a1ee:::benchmark_data_smithi194_103039_object139287:head# 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.911 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b60ea8:::benchmark_data_smithi194_103039_object105442:head# 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b61745:::benchmark_data_smithi194_103039_object145569:head# 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b6688a:::benchmark_data_smithi194_103039_object65961:head# 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b697cc:::benchmark_data_smithi194_103039_object153643:head# 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b6fa87:::benchmark_data_smithi194_103039_object110342:head# 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b72a16:::benchmark_data_smithi194_103039_object70149:head# 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49b96aed:::benchmark_data_smithi194_103039_object60151:head# 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49bb6fbc:::benchmark_data_smithi194_103039_object154774:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49bb7ddc:::benchmark_data_smithi194_103039_object51362:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49bc0248:::benchmark_data_smithi194_103039_object101264:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49bd44f5:::benchmark_data_smithi194_103039_object81722:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49be1b3a:::benchmark_data_smithi194_103039_object151602:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49bf359a:::benchmark_data_smithi194_103039_object130823:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c06f75:::benchmark_data_smithi194_103039_object53529:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c0c09a:::benchmark_data_smithi194_103039_object71783:head# 2024-04-29T14:21:29.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c138e7:::benchmark_data_smithi194_103039_object81568:head# 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c1adc3:::benchmark_data_smithi194_103039_object120882:head# 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c240a8:::benchmark_data_smithi194_103039_object133260:head# 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c31359:::benchmark_data_smithi194_103039_object80766:head# 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c3b956:::benchmark_data_smithi194_103039_object69620:head# 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c3eaf7:::benchmark_data_smithi194_103039_object56959:head# 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c416e5:::benchmark_data_smithi194_103039_object73260:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c4c753:::benchmark_data_smithi194_103039_object135468:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c52f8e:::benchmark_data_smithi194_103039_object86255:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c533ed:::benchmark_data_smithi194_103039_object148536:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c5c479:::benchmark_data_smithi194_103039_object68771:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c61bef:::benchmark_data_smithi194_103039_object78793:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c61cdd:::benchmark_data_smithi194_103039_object62698:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c63dfd:::benchmark_data_smithi194_103039_object131378:head# 2024-04-29T14:21:29.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c6857b:::benchmark_data_smithi194_103039_object73418:head# 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c72e89:::benchmark_data_smithi194_103039_object52316:head# 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c8bd33:::benchmark_data_smithi194_103039_object76288:head# 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c902e5:::benchmark_data_smithi194_103039_object135845:head# 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49c9aa6a:::benchmark_data_smithi194_103039_object41816:head# 2024-04-29T14:21:29.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49cd3bb4:::benchmark_data_smithi194_103039_object102212:head# 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49cd7a15:::benchmark_data_smithi194_103039_object89866:head# 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49cedaea:::benchmark_data_smithi194_103039_object84179:head# 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49cf21ec:::benchmark_data_smithi194_103039_object70203:head# 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49cf3d8b:::benchmark_data_smithi194_103039_object124798:head# 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49cf5c31:::benchmark_data_smithi194_103039_object99766:head# 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49cfd053:::benchmark_data_smithi194_103039_object56416:head# 2024-04-29T14:21:29.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d190e4:::benchmark_data_smithi194_103039_object98337:head# 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d222e5:::benchmark_data_smithi194_103039_object43913:head# 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d2327f:::benchmark_data_smithi194_103039_object80553:head# 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d2ca18:::benchmark_data_smithi194_103039_object84471:head# 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d2df09:::benchmark_data_smithi194_103039_object62793:head# 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d306a5:::benchmark_data_smithi194_103039_object115848:head# 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d4658a:::benchmark_data_smithi194_103039_object97968:head# 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d4750b:::benchmark_data_smithi194_103039_object96284:head# 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d4ea99:::benchmark_data_smithi194_103039_object155555:head# 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d4f104:::benchmark_data_smithi194_103039_object123166:head# 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d5a95c:::benchmark_data_smithi194_103039_object117329:head# 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d64bc2:::benchmark_data_smithi194_103039_object142113:head# 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d65171:::benchmark_data_smithi194_103039_object77748:head# 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d677e2:::benchmark_data_smithi194_103039_object128185:head# 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d72ee5:::benchmark_data_smithi194_103039_object94186:head# 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d8614e:::benchmark_data_smithi194_103039_object78269:head# 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d8d3b4:::benchmark_data_smithi194_103039_object119180:head# 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49d95259:::benchmark_data_smithi194_103039_object46988:head# 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49da1cf9:::benchmark_data_smithi194_103039_object64684:head# 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49db540e:::benchmark_data_smithi194_103039_object131511:head# 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49dba5fa:::benchmark_data_smithi194_103039_object124517:head# 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49dcd37e:::benchmark_data_smithi194_103039_object115067:head# 2024-04-29T14:21:29.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49dda284:::benchmark_data_smithi194_103039_object83315:head# 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ddbc06:::benchmark_data_smithi194_103039_object74915:head# 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49df5c2e:::benchmark_data_smithi194_103039_object138920:head# 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49dfeeab:::benchmark_data_smithi194_103039_object150839:head# 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49dffa56:::benchmark_data_smithi194_103039_object129503:head# 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e10dd7:::benchmark_data_smithi194_103039_object85326:head# 2024-04-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e11a87:::benchmark_data_smithi194_103039_object130350:head# 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e14d42:::benchmark_data_smithi194_103039_object57355:head# 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e312a3:::benchmark_data_smithi194_103039_object52506:head# 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e32254:::benchmark_data_smithi194_103039_object89081:head# 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e36879:::benchmark_data_smithi194_103039_object59774:head# 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e5337b:::benchmark_data_smithi194_103039_object154403:head# 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e5447a:::benchmark_data_smithi194_103039_object92316:head# 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e55783:::benchmark_data_smithi194_103039_object82796:head# 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e5e401:::benchmark_data_smithi194_103039_object57735:head# 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e65267:::benchmark_data_smithi194_103039_object118558:head# 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e69bb9:::benchmark_data_smithi194_103039_object48373:head# 2024-04-29T14:21:29.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e7746d:::benchmark_data_smithi194_103039_object122990:head# 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e83408:::benchmark_data_smithi194_103039_object67720:head# 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49e9e803:::benchmark_data_smithi194_103039_object152280:head# 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49eb00a6:::benchmark_data_smithi194_103039_object68488:head# 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49eb1287:::benchmark_data_smithi194_103039_object145896:head# 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ecf553:::benchmark_data_smithi194_103039_object81676:head# 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ed0932:::benchmark_data_smithi194_103039_object155501:head# 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ed0be4:::benchmark_data_smithi194_103039_object75036:head# 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49eda1d0:::benchmark_data_smithi194_103039_object50314:head# 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ef4466:::benchmark_data_smithi194_103039_object43815:head# 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f04a05:::benchmark_data_smithi194_103039_object90898:head# 2024-04-29T14:21:29.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f13756:::benchmark_data_smithi194_103039_object116682:head# 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f19810:::benchmark_data_smithi194_103039_object41834:head# 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f1c2ce:::benchmark_data_smithi194_103039_object81423:head# 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f2d669:::benchmark_data_smithi194_103039_object45121:head# 2024-04-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f32435:::benchmark_data_smithi194_103039_object62318:head# 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f35344:::benchmark_data_smithi194_103039_object47523:head# 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f3b817:::benchmark_data_smithi194_103039_object54815:head# 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f4d60c:::benchmark_data_smithi194_103039_object66951:head# 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f53217:::benchmark_data_smithi194_103039_object95730:head# 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f675e3:::benchmark_data_smithi194_103039_object133620:head# 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f6777d:::benchmark_data_smithi194_103039_object61218:head# 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f7d1a4:::benchmark_data_smithi194_103039_object42408:head# 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f85e2e:::benchmark_data_smithi194_103039_object91229:head# 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f868dc:::benchmark_data_smithi194_103039_object80888:head# 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f8c1c1:::benchmark_data_smithi194_103039_object132149:head# 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f8f137:::benchmark_data_smithi194_103039_object98198:head# 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49f9cfb1:::benchmark_data_smithi194_103039_object127697:head# 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fa16fa:::benchmark_data_smithi194_103039_object58074:head# 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fa8820:::benchmark_data_smithi194_103039_object65535:head# 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fae67b:::benchmark_data_smithi194_103039_object119194:head# 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fb15f1:::benchmark_data_smithi194_103039_object78340:head# 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fb6012:::benchmark_data_smithi194_103039_object148034:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fbed3b:::benchmark_data_smithi194_103039_object70273:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fc8f6d:::benchmark_data_smithi194_103039_object146435:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fde8e8:::benchmark_data_smithi194_103039_object43604:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fdecb6:::benchmark_data_smithi194_103039_object110729:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fe0163:::benchmark_data_smithi194_103039_object154527:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fe7369:::benchmark_data_smithi194_103039_object63359:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fe9071:::benchmark_data_smithi194_103039_object104303:head# 2024-04-29T14:21:29.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fea8eb:::benchmark_data_smithi194_103039_object119342:head# 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49fec1a6:::benchmark_data_smithi194_103039_object44864:head# 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:49ff92fa:::benchmark_data_smithi194_103039_object86415:head# 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a008ff4:::benchmark_data_smithi194_103039_object71743:head# 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a031f1f:::benchmark_data_smithi194_103039_object98483:head# 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a033431:::benchmark_data_smithi194_103039_object126115:head# 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a034a40:::benchmark_data_smithi194_103039_object119236:head# 2024-04-29T14:21:29.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a03d053:::benchmark_data_smithi194_103039_object83652:head# 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a043d5e:::benchmark_data_smithi194_103039_object65331:head# 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a04b54e:::benchmark_data_smithi194_103039_object112182:head# 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a04e6f0:::benchmark_data_smithi194_103039_object91950:head# 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a04f0d9:::benchmark_data_smithi194_103039_object46858:head# 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0536d8:::benchmark_data_smithi194_103039_object122599:head# 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a05ab3d:::benchmark_data_smithi194_103039_object53568:head# 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a060330:::benchmark_data_smithi194_103039_object153601:head# 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a069221:::benchmark_data_smithi194_103039_object119014:head# 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a06d5eb:::benchmark_data_smithi194_103039_object74270:head# 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a06f84f:::benchmark_data_smithi194_103039_object137235:head# 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a076b6f:::benchmark_data_smithi194_103039_object81499:head# 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a077ed8:::benchmark_data_smithi194_103039_object129905:head# 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a07c707:::benchmark_data_smithi194_103039_object94351:head# 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a08a9b4:::benchmark_data_smithi194_103039_object101651:head# 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a097894:::benchmark_data_smithi194_103039_object110605:head# 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0a828c:::benchmark_data_smithi194_103039_object68450:head# 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0a8e30:::benchmark_data_smithi194_103039_object116057:head# 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0b7427:::benchmark_data_smithi194_103039_object82415:head# 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0c696a:::benchmark_data_smithi194_103039_object133012:head# 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0d5f73:::benchmark_data_smithi194_103039_object43182:head# 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0e2fb2:::benchmark_data_smithi194_103039_object51239:head# 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a0fe4ba:::benchmark_data_smithi194_103039_object84083:head# 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a117984:::benchmark_data_smithi194_103039_object43879:head# 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a11d7ee:::benchmark_data_smithi194_103039_object151034:head# 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a11fc56:::benchmark_data_smithi194_103039_object128660:head# 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a125975:::benchmark_data_smithi194_103039_object57645:head# 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1365a7:::benchmark_data_smithi194_103039_object145097:head# 2024-04-29T14:21:29.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1384d9:::benchmark_data_smithi194_103039_object114586:head# 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a13de29:::benchmark_data_smithi194_103039_object45501:head# 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a15dd01:::benchmark_data_smithi194_103039_object86321:head# 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a16c61f:::benchmark_data_smithi194_103039_object117994:head# 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a17e310:::benchmark_data_smithi194_103039_object77240:head# 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a17e3db:::benchmark_data_smithi194_103039_object72619:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a18ecca:::benchmark_data_smithi194_103039_object154600:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a18fb54:::benchmark_data_smithi194_103039_object126749:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1977b9:::benchmark_data_smithi194_103039_object54703:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1a4e3e:::benchmark_data_smithi194_103039_object77172:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1b3652:::benchmark_data_smithi194_103039_object68428:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1c2397:::benchmark_data_smithi194_103039_object45903:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1c2c50:::benchmark_data_smithi194_103039_object144726:head# 2024-04-29T14:21:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1c4bf4:::benchmark_data_smithi194_103039_object133257:head# 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1c72d3:::benchmark_data_smithi194_103039_object135596:head# 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1d25af:::benchmark_data_smithi194_103039_object125871:head# 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1d53a1:::benchmark_data_smithi194_103039_object117875:head# 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1d878b:::benchmark_data_smithi194_103039_object147079:head# 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1da345:::benchmark_data_smithi194_103039_object137975:head# 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1dd5b2:::benchmark_data_smithi194_103039_object137534:head# 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1e033a:::benchmark_data_smithi194_103039_object120371:head# 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1e7317:::benchmark_data_smithi194_103039_object142903:head# 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1e8ab3:::benchmark_data_smithi194_103039_object135678:head# 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1ed41e:::benchmark_data_smithi194_103039_object105460:head# 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a1ef57a:::benchmark_data_smithi194_103039_object70323:head# 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a20f392:::benchmark_data_smithi194_103039_object55161:head# 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a21186b:::benchmark_data_smithi194_103039_object106669:head# 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a215f08:::benchmark_data_smithi194_103039_object73613:head# 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2241d4:::benchmark_data_smithi194_103039_object77126:head# 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a230224:::benchmark_data_smithi194_103039_object119802:head# 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a230ce5:::benchmark_data_smithi194_103039_object139011:head# 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a23170f:::benchmark_data_smithi194_103039_object91049:head# 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a234235:::benchmark_data_smithi194_103039_object143678:head# 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a257ad4:::benchmark_data_smithi194_103039_object76686:head# 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2610fe:::benchmark_data_smithi194_103039_object129065:head# 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2670ff:::benchmark_data_smithi194_103039_object156656:head# 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2692b5:::benchmark_data_smithi194_103039_object109204:head# 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a272a7d:::benchmark_data_smithi194_103039_object55209:head# 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a275da4:::benchmark_data_smithi194_103039_object111976:head# 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a275e26:::benchmark_data_smithi194_103039_object57331:head# 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a28a55a:::benchmark_data_smithi194_103039_object77079:head# 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a295a51:::benchmark_data_smithi194_103039_object70338:head# 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a297c20:::benchmark_data_smithi194_103039_object144721:head# 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a298bb4:::benchmark_data_smithi194_103039_object52871:head# 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2ac094:::benchmark_data_smithi194_103039_object136114:head# 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2b261d:::benchmark_data_smithi194_103039_object154439:head# 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2bbba2:::benchmark_data_smithi194_103039_object131952:head# 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2c4f3b:::benchmark_data_smithi194_103039_object99047:head# 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2c5f36:::benchmark_data_smithi194_103039_object108882:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2c842d:::benchmark_data_smithi194_103039_object86359:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2c8bc0:::benchmark_data_smithi194_103039_object79441:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2cb7fd:::benchmark_data_smithi194_103039_object81774:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2dbc19:::benchmark_data_smithi194_103039_object78223:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2ecbd0:::benchmark_data_smithi194_103039_object75369:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2ee10b:::benchmark_data_smithi194_103039_object57526:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a2f3a59:::benchmark_data_smithi194_103039_object79719:head# 2024-04-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a308a44:::benchmark_data_smithi194_103039_object77656:head# 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a30a737:::benchmark_data_smithi194_103039_object69529:head# 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a316682:::benchmark_data_smithi194_103039_object103893:head# 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a31a368:::benchmark_data_smithi194_103039_object126915:head# 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a329368:::benchmark_data_smithi194_103039_object151103:head# 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a32ae7f:::benchmark_data_smithi194_103039_object126187:head# 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a332e21:::benchmark_data_smithi194_103039_object52919:head# 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a33ba94:::benchmark_data_smithi194_103039_object111506:head# 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a33f003:::benchmark_data_smithi194_103039_object149934:head# 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a343dee:::benchmark_data_smithi194_103039_object68447:head# 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a354456:::benchmark_data_smithi194_103039_object112748:head# 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a361175:::benchmark_data_smithi194_103039_object81741:head# 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a36491d:::benchmark_data_smithi194_103039_object51529:head# 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a36da29:::benchmark_data_smithi194_103039_object73337:head# 2024-04-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3899f5:::benchmark_data_smithi194_103039_object107262:head# 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a38caca:::benchmark_data_smithi194_103039_object128777:head# 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a396e20:::benchmark_data_smithi194_103039_object69154:head# 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3a2a72:::benchmark_data_smithi194_103039_object80175:head# 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3a73a5:::benchmark_data_smithi194_103039_object68717:head# 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3c3047:::benchmark_data_smithi194_103039_object65508:head# 2024-04-29T14:21:29.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3c6a41:::benchmark_data_smithi194_103039_object56683:head# 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3dc221:::benchmark_data_smithi194_103039_object127910:head# 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3e44b9:::benchmark_data_smithi194_103039_object104032:head# 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a3f12a1:::benchmark_data_smithi194_103039_object68657:head# 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a409ebf:::benchmark_data_smithi194_103039_object64505:head# 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a40ebc2:::benchmark_data_smithi194_103039_object43823:head# 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a40f99f:::benchmark_data_smithi194_103039_object144531:head# 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a427841:::benchmark_data_smithi194_103039_object55549:head# 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a42c395:::benchmark_data_smithi194_103039_object58387:head# 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a443264:::benchmark_data_smithi194_103039_object58574:head# 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4555f1:::benchmark_data_smithi194_103039_object155681:head# 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a456d25:::benchmark_data_smithi194_103039_object108677:head# 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a459b44:::benchmark_data_smithi194_103039_object75629:head# 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a46e294:::benchmark_data_smithi194_103039_object148622:head# 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4702a8:::benchmark_data_smithi194_103039_object130710:head# 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a472c44:::benchmark_data_smithi194_103039_object90481:head# 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a482461:::benchmark_data_smithi194_103039_object90118:head# 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a49a802:::benchmark_data_smithi194_103039_object148667:head# 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a49b0a6:::benchmark_data_smithi194_103039_object148800:head# 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4ab397:::benchmark_data_smithi194_103039_object82085:head# 2024-04-29T14:21:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4ac5ef:::benchmark_data_smithi194_103039_object132884:head# 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4b2369:::benchmark_data_smithi194_103039_object89748:head# 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4b70f3:::benchmark_data_smithi194_103039_object124508:head# 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4b71d7:::benchmark_data_smithi194_103039_object133082:head# 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4c77e0:::benchmark_data_smithi194_103039_object108959:head# 2024-04-29T14:21:29.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4dbaf7:::benchmark_data_smithi194_103039_object153890:head# 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4dfaf9:::benchmark_data_smithi194_103039_object136617:head# 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4f69f1:::benchmark_data_smithi194_103039_object94367:head# 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a4f89cf:::benchmark_data_smithi194_103039_object145225:head# 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a50c653:::benchmark_data_smithi194_103039_object128127:head# 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a51792a:::benchmark_data_smithi194_103039_object154432:head# 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5218a8:::benchmark_data_smithi194_103039_object118107:head# 2024-04-29T14:21:29.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a53bce8:::benchmark_data_smithi194_103039_object67917:head# 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a545ff2:::benchmark_data_smithi194_103039_object127306:head# 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5616ac:::benchmark_data_smithi194_103039_object42651:head# 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5645e4:::benchmark_data_smithi194_103039_object120242:head# 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a56eb58:::benchmark_data_smithi194_103039_object117364:head# 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a57377e:::benchmark_data_smithi194_103039_object146703:head# 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a577758:::benchmark_data_smithi194_103039_object76527:head# 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a57bab0:::benchmark_data_smithi194_103039_object103394:head# 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5821cd:::benchmark_data_smithi194_103039_object124046:head# 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a589783:::benchmark_data_smithi194_103039_object119598:head# 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a58aa4e:::benchmark_data_smithi194_103039_object104681:head# 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a58c194:::benchmark_data_smithi194_103039_object58365:head# 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a595aa3:::benchmark_data_smithi194_103039_object108209:head# 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a59dec8:::benchmark_data_smithi194_103039_object147294:head# 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a59f894:::benchmark_data_smithi194_103039_object132410:head# 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5a0070:::benchmark_data_smithi194_103039_object106780:head# 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5caaca:::benchmark_data_smithi194_103039_object128894:head# 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5d7ec4:::benchmark_data_smithi194_103039_object155459:head# 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a5ed153:::benchmark_data_smithi194_103039_object49227:head# 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6001b6:::benchmark_data_smithi194_103039_object156016:head# 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a60bd65:::benchmark_data_smithi194_103039_object62490:head# 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a620968:::benchmark_data_smithi194_103039_object147606:head# 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a623f90:::benchmark_data_smithi194_103039_object87087:head# 2024-04-29T14:21:29.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a628c2b:::benchmark_data_smithi194_103039_object146359:head# 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a62bbb8:::benchmark_data_smithi194_103039_object53947:head# 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a62f126:::benchmark_data_smithi194_103039_object82474:head# 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6385a5:::benchmark_data_smithi194_103039_object75162:head# 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a640475:::benchmark_data_smithi194_103039_object124734:head# 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a650a19:::benchmark_data_smithi194_103039_object115618:head# 2024-04-29T14:21:29.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6604c5:::benchmark_data_smithi194_103039_object102243:head# 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a666845:::benchmark_data_smithi194_103039_object44930:head# 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a66c296:::benchmark_data_smithi194_103039_object56273:head# 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a67295b:::benchmark_data_smithi194_103039_object50688:head# 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a67a994:::benchmark_data_smithi194_103039_object77452:head# 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a681181:::benchmark_data_smithi194_103039_object153407:head# 2024-04-29T14:21:29.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a68ffc8:::benchmark_data_smithi194_103039_object87379:head# 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a697970:::benchmark_data_smithi194_103039_object118831:head# 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a69bcc7:::benchmark_data_smithi194_103039_object51531:head# 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a69fb25:::benchmark_data_smithi194_103039_object63103:head# 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6a55cb:::benchmark_data_smithi194_103039_object149180:head# 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6aa591:::benchmark_data_smithi194_103039_object119908:head# 2024-04-29T14:21:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6b1e7b:::benchmark_data_smithi194_103039_object155247:head# 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6b5d3a:::benchmark_data_smithi194_103039_object151496:head# 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6ba44a:::benchmark_data_smithi194_103039_object93852:head# 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6bc36d:::benchmark_data_smithi194_103039_object52821:head# 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6c3b6e:::benchmark_data_smithi194_103039_object115056:head# 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6cbdbb:::benchmark_data_smithi194_103039_object52375:head# 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6cdaa4:::benchmark_data_smithi194_103039_object96203:head# 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6cdd7c:::benchmark_data_smithi194_103039_object77773:head# 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6d92d3:::benchmark_data_smithi194_103039_object51335:head# 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6e07a7:::benchmark_data_smithi194_103039_object96266:head# 2024-04-29T14:21:29.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6f39f5:::benchmark_data_smithi194_103039_object51215:head# 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6f65d7:::benchmark_data_smithi194_103039_object90015:head# 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a6f706d:::benchmark_data_smithi194_103039_object122291:head# 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a70031f:::benchmark_data_smithi194_103039_object49767:head# 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a701bbf:::benchmark_data_smithi194_103039_object51932:head# 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a702cde:::benchmark_data_smithi194_103039_object59396:head# 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a719b7e:::benchmark_data_smithi194_103039_object148247:head# 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7214ee:::benchmark_data_smithi194_103039_object114710:head# 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a73256f:::benchmark_data_smithi194_103039_object81172:head# 2024-04-29T14:21:29.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a733412:::benchmark_data_smithi194_103039_object99408:head# 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a73dfa6:::benchmark_data_smithi194_103039_object132743:head# 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a74d00e:::benchmark_data_smithi194_103039_object57612:head# 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a75ad5e:::benchmark_data_smithi194_103039_object63256:head# 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a75daa7:::benchmark_data_smithi194_103039_object45628:head# 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7698ac:::benchmark_data_smithi194_103039_object53631:head# 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a769d32:::benchmark_data_smithi194_103039_object98400:head# 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a76b4b2:::benchmark_data_smithi194_103039_object115898:head# 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a771945:::benchmark_data_smithi194_103039_object80306:head# 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a77717a:::benchmark_data_smithi194_103039_object112534:head# 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a78f846:::benchmark_data_smithi194_103039_object143922:head# 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a79af7f:::benchmark_data_smithi194_103039_object79800:head# 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a79d2f4:::benchmark_data_smithi194_103039_object142615:head# 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7a5696:::benchmark_data_smithi194_103039_object123571:head# 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7a776a:::benchmark_data_smithi194_103039_object120873:head# 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7b140c:::benchmark_data_smithi194_103039_object122313:head# 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7bdbed:::benchmark_data_smithi194_103039_object64339:head# 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7d03eb:::benchmark_data_smithi194_103039_object138550:head# 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7d2e92:::benchmark_data_smithi194_103039_object134785:head# 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7def9d:::benchmark_data_smithi194_103039_object133384:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7f91d5:::benchmark_data_smithi194_103039_object133128:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7fe7b1:::benchmark_data_smithi194_103039_object126356:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a7ff06b:::benchmark_data_smithi194_103039_object143574:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a800ab4:::benchmark_data_smithi194_103039_object74979:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a806d15:::benchmark_data_smithi194_103039_object70080:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a815ae4:::benchmark_data_smithi194_103039_object75739:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8265ca:::benchmark_data_smithi194_103039_object144582:head# 2024-04-29T14:21:29.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a828531:::benchmark_data_smithi194_103039_object45901:head# 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a828a82:::benchmark_data_smithi194_103039_object129393:head# 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a82a46d:::benchmark_data_smithi194_103039_object48381:head# 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a82e966:::benchmark_data_smithi194_103039_object87125:head# 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a834433:::benchmark_data_smithi194_103039_object145659:head# 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a837335:::benchmark_data_smithi194_103039_object115384:head# 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a840cff:::benchmark_data_smithi194_103039_object117658:head# 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a84355e:::benchmark_data_smithi194_103039_object75646:head# 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a84faf3:::benchmark_data_smithi194_103039_object46953:head# 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a85792b:::benchmark_data_smithi194_103039_object73902:head# 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8618c6:::benchmark_data_smithi194_103039_object84072:head# 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8732ae:::benchmark_data_smithi194_103039_object131940:head# 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a87c37a:::benchmark_data_smithi194_103039_object87276:head# 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8a6bd2:::benchmark_data_smithi194_103039_object87075:head# 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8b8933:::benchmark_data_smithi194_103039_object94544:head# 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8c3795:::benchmark_data_smithi194_103039_object134190:head# 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8c7c34:::benchmark_data_smithi194_103039_object92032:head# 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8d1fd7:::benchmark_data_smithi194_103039_object43974:head# 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8d5c7e:::benchmark_data_smithi194_103039_object59060:head# 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8d6a96:::benchmark_data_smithi194_103039_object96793:head# 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8d800b:::benchmark_data_smithi194_103039_object61023:head# 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8d846e:::benchmark_data_smithi194_103039_object71159:head# 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8dd0bc:::benchmark_data_smithi194_103039_object62149:head# 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8e0265:::benchmark_data_smithi194_103039_object141604:head# 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8e14e6:::benchmark_data_smithi194_103039_object85707:head# 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a8e5116:::benchmark_data_smithi194_103039_object132434:head# 2024-04-29T14:21:29.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a916dd1:::benchmark_data_smithi194_103039_object82161:head# 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a917934:::benchmark_data_smithi194_103039_object52508:head# 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a91c4e7:::benchmark_data_smithi194_103039_object54002:head# 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a91e944:::benchmark_data_smithi194_103039_object76247:head# 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a925e7c:::benchmark_data_smithi194_103039_object151377:head# 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a92b77a:::benchmark_data_smithi194_103039_object105308:head# 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9341a1:::benchmark_data_smithi194_103039_object73013:head# 2024-04-29T14:21:29.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a937d42:::benchmark_data_smithi194_103039_object102895:head# 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a93f76b:::benchmark_data_smithi194_103039_object92992:head# 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a96d19c:::benchmark_data_smithi194_103039_object89189:head# 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a977fcd:::benchmark_data_smithi194_103039_object133638:head# 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a97bda2:::benchmark_data_smithi194_103039_object135111:head# 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a988913:::benchmark_data_smithi194_103039_object124828:head# 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a98a721:::benchmark_data_smithi194_103039_object90848:head# 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9a0cfa:::benchmark_data_smithi194_103039_object52619:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9a2918:::benchmark_data_smithi194_103039_object74281:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9a3a07:::benchmark_data_smithi194_103039_object54245:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9a9d28:::benchmark_data_smithi194_103039_object78682:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9b1005:::benchmark_data_smithi194_103039_object153302:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9b6edb:::benchmark_data_smithi194_103039_object115112:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9b7326:::benchmark_data_smithi194_103039_object77907:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9ba215:::benchmark_data_smithi194_103039_object151507:head# 2024-04-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9c0f6c:::benchmark_data_smithi194_103039_object55046:head# 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9c1a3d:::benchmark_data_smithi194_103039_object148210:head# 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9cc9d1:::benchmark_data_smithi194_103039_object76212:head# 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9d1877:::benchmark_data_smithi194_103039_object43606:head# 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9dfb23:::benchmark_data_smithi194_103039_object77938:head# 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9e8293:::benchmark_data_smithi194_103039_object114076:head# 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9ec1a2:::benchmark_data_smithi194_103039_object101759:head# 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9ec7cc:::benchmark_data_smithi194_103039_object134687:head# 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9f6ed3:::benchmark_data_smithi194_103039_object115986:head# 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9fc468:::benchmark_data_smithi194_103039_object57374:head# 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4a9fc731:::benchmark_data_smithi194_103039_object89089:head# 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa0138a:::benchmark_data_smithi194_103039_object51315:head# 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa06baa:::benchmark_data_smithi194_103039_object95050:head# 2024-04-29T14:21:29.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa123dc:::benchmark_data_smithi194_103039_object137898:head# 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa20435:::benchmark_data_smithi194_103039_object73399:head# 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa2c8a5:::benchmark_data_smithi194_103039_object125142:head# 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa3203a:::benchmark_data_smithi194_103039_object97952:head# 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa37447:::benchmark_data_smithi194_103039_object123435:head# 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa411ba:::benchmark_data_smithi194_103039_object141506:head# 2024-04-29T14:21:29.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa4aee0:::benchmark_data_smithi194_103039_object60766:head# 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa74278:::benchmark_data_smithi194_103039_object144811:head# 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa888e8:::benchmark_data_smithi194_103039_object44782:head# 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa9160e:::benchmark_data_smithi194_103039_object142754:head# 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa91b6b:::benchmark_data_smithi194_103039_object54415:head# 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aa93896:::benchmark_data_smithi194_103039_object111551:head# 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aaa5a64:::benchmark_data_smithi194_103039_object66659:head# 2024-04-29T14:21:29.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aaab0b3:::benchmark_data_smithi194_103039_object59524:head# 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aac1d1a:::benchmark_data_smithi194_103039_object115183:head# 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aac4f92:::benchmark_data_smithi194_103039_object41611:head# 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aac900f:::benchmark_data_smithi194_103039_object61410:head# 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aad1692:::benchmark_data_smithi194_103039_object58686:head# 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aad3a0e:::benchmark_data_smithi194_103039_object85458:head# 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aadeb09:::benchmark_data_smithi194_103039_object130910:head# 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aadfa35:::benchmark_data_smithi194_103039_object124605:head# 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aae3994:::benchmark_data_smithi194_103039_object93433:head# 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aae4f0f:::benchmark_data_smithi194_103039_object47519:head# 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aaf030e:::benchmark_data_smithi194_103039_object103647:head# 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aaf33ab:::benchmark_data_smithi194_103039_object80461:head# 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab137f5:::benchmark_data_smithi194_103039_object115650:head# 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab15013:::benchmark_data_smithi194_103039_object114161:head# 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab15aa6:::benchmark_data_smithi194_103039_object119919:head# 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab1d41d:::benchmark_data_smithi194_103039_object89472:head# 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab297e4:::benchmark_data_smithi194_103039_object46936:head# 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab32629:::benchmark_data_smithi194_103039_object57114:head# 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab36a5d:::benchmark_data_smithi194_103039_object153883:head# 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab38c98:::benchmark_data_smithi194_103039_object78820:head# 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab42fc7:::benchmark_data_smithi194_103039_object87796:head# 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab50b32:::benchmark_data_smithi194_103039_object130781:head# 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab5300d:::benchmark_data_smithi194_103039_object77034:head# 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab59f83:::benchmark_data_smithi194_103039_object45498:head# 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab5a969:::benchmark_data_smithi194_103039_object149970:head# 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab67c8e:::benchmark_data_smithi194_103039_object65562:head# 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab6c949:::benchmark_data_smithi194_103039_object77129:head# 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab6fd4e:::benchmark_data_smithi194_103039_object96285:head# 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab75dc1:::benchmark_data_smithi194_103039_object88050:head# 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab77b4f:::benchmark_data_smithi194_103039_object107487:head# 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab78c2c:::benchmark_data_smithi194_103039_object81698:head# 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab7c96f:::benchmark_data_smithi194_103039_object81169:head# 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab7cc80:::benchmark_data_smithi194_103039_object143010:head# 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab7fe6e:::benchmark_data_smithi194_103039_object93668:head# 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab80cef:::benchmark_data_smithi194_103039_object53164:head# 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab83ad8:::benchmark_data_smithi194_103039_object112058:head# 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab87755:::benchmark_data_smithi194_103039_object85639:head# 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab8bfd5:::benchmark_data_smithi194_103039_object66920:head# 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab8e6d5:::benchmark_data_smithi194_103039_object87566:head# 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ab99952:::benchmark_data_smithi194_103039_object65084:head# 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4abb42e3:::benchmark_data_smithi194_103039_object95474:head# 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4abd8914:::benchmark_data_smithi194_103039_object130741:head# 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4abe060b:::benchmark_data_smithi194_103039_object150013:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4abe28cc:::benchmark_data_smithi194_103039_object52151:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4abf46b6:::benchmark_data_smithi194_103039_object143433:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4abf5b8d:::benchmark_data_smithi194_103039_object141865:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4abfb9cb:::benchmark_data_smithi194_103039_object76170:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac020ca:::benchmark_data_smithi194_103039_object151554:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac07659:::benchmark_data_smithi194_103039_object126624:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac09a03:::benchmark_data_smithi194_103039_object72734:head# 2024-04-29T14:21:29.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac0ee9c:::benchmark_data_smithi194_103039_object106539:head# 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac17609:::benchmark_data_smithi194_103039_object53316:head# 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac182c4:::benchmark_data_smithi194_103039_object111688:head# 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac25456:::benchmark_data_smithi194_103039_object154251:head# 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac2e46c:::benchmark_data_smithi194_103039_object51940:head# 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac33a4c:::benchmark_data_smithi194_103039_object106998:head# 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac54080:::benchmark_data_smithi194_103039_object62143:head# 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac72ebf:::benchmark_data_smithi194_103039_object101817:head# 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac79e7b:::benchmark_data_smithi194_103039_object117801:head# 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ac7a641:::benchmark_data_smithi194_103039_object88317:head# 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acaed55:::benchmark_data_smithi194_103039_object62675:head# 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acaef1f:::benchmark_data_smithi194_103039_object60264:head# 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acafb59:::benchmark_data_smithi194_103039_object98475:head# 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acafde0:::benchmark_data_smithi194_103039_object96279:head# 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acbc282:::benchmark_data_smithi194_103039_object116331:head# 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acbffb4:::benchmark_data_smithi194_103039_object47065:head# 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acc6d93:::benchmark_data_smithi194_103039_object129566:head# 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acd2a3c:::benchmark_data_smithi194_103039_object138026:head# 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acd2f1e:::benchmark_data_smithi194_103039_object65948:head# 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acd47c8:::benchmark_data_smithi194_103039_object119980:head# 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acd9503:::benchmark_data_smithi194_103039_object102154:head# 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ace02b5:::benchmark_data_smithi194_103039_object59279:head# 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ace2ae7:::benchmark_data_smithi194_103039_object144128:head# 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ace30b9:::benchmark_data_smithi194_103039_object95545:head# 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acf0a3f:::benchmark_data_smithi194_103039_object69786:head# 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acfa910:::benchmark_data_smithi194_103039_object93955:head# 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acfb315:::benchmark_data_smithi194_103039_object73285:head# 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acfc0fb:::benchmark_data_smithi194_103039_object64879:head# 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4acfe1a5:::benchmark_data_smithi194_103039_object55120:head# 2024-04-29T14:21:29.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad1d6f8:::benchmark_data_smithi194_103039_object124680:head# 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad35d47:::benchmark_data_smithi194_103039_object98739:head# 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad5ecaa:::benchmark_data_smithi194_103039_object120492:head# 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad60061:::benchmark_data_smithi194_103039_object53828:head# 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad6b1a4:::benchmark_data_smithi194_103039_object143238:head# 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad7421e:::benchmark_data_smithi194_103039_object155920:head# 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad74f2c:::benchmark_data_smithi194_103039_object56469:head# 2024-04-29T14:21:29.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad8a425:::benchmark_data_smithi194_103039_object59904:head# 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad8b043:::benchmark_data_smithi194_103039_object51107:head# 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ad9b0ae:::benchmark_data_smithi194_103039_object156014:head# 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4adaca28:::benchmark_data_smithi194_103039_object52518:head# 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4adb4812:::benchmark_data_smithi194_103039_object116251:head# 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4adbe1ff:::benchmark_data_smithi194_103039_object83180:head# 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4adc290c:::benchmark_data_smithi194_103039_object134604:head# 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4adcef9d:::benchmark_data_smithi194_103039_object51830:head# 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ade1aaf:::benchmark_data_smithi194_103039_object137730:head# 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ade75c3:::benchmark_data_smithi194_103039_object55660:head# 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ade7a2d:::benchmark_data_smithi194_103039_object126046:head# 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4adeaada:::benchmark_data_smithi194_103039_object46961:head# 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4adf7dfc:::benchmark_data_smithi194_103039_object68272:head# 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae09a1c:::benchmark_data_smithi194_103039_object55248:head# 2024-04-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae0fdb0:::benchmark_data_smithi194_103039_object71862:head# 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae126e5:::benchmark_data_smithi194_103039_object82359:head# 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae12fbe:::benchmark_data_smithi194_103039_object97078:head# 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae1707a:::benchmark_data_smithi194_103039_object144495:head# 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae1b25b:::benchmark_data_smithi194_103039_object154673:head# 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae29a2f:::benchmark_data_smithi194_103039_object45736:head# 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae35c46:::benchmark_data_smithi194_103039_object80072:head# 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae3e8c5:::benchmark_data_smithi194_103039_object68522:head# 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae41849:::benchmark_data_smithi194_103039_object71381:head# 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae4f369:::benchmark_data_smithi194_103039_object92347:head# 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae6b346:::benchmark_data_smithi194_103039_object129489:head# 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae7a10a:::benchmark_data_smithi194_103039_object151355:head# 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae86924:::benchmark_data_smithi194_103039_object71709:head# 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.995 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ae893d1:::benchmark_data_smithi194_103039_object114617:head# 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aea23a2:::benchmark_data_smithi194_103039_object44858:head# 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aea426d:::benchmark_data_smithi194_103039_object63648:head# 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aea8586:::benchmark_data_smithi194_103039_object113780:head# 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aeaadfe:::benchmark_data_smithi194_103039_object120823:head# 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aeac02f:::benchmark_data_smithi194_103039_object88188:head# 2024-04-29T14:21:29.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aec4ebc:::benchmark_data_smithi194_103039_object104363:head# 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aec890a:::benchmark_data_smithi194_103039_object81874:head# 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aece699:::benchmark_data_smithi194_103039_object91734:head# 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aed3080:::benchmark_data_smithi194_103039_object90641:head# 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aee7f94:::benchmark_data_smithi194_103039_object72868:head# 2024-04-29T14:21:29.997 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4aefcfa9:::benchmark_data_smithi194_103039_object106721:head# 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af0556a:::benchmark_data_smithi194_103039_object134695:head# 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af06c7f:::benchmark_data_smithi194_103039_object148568:head# 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af06e33:::benchmark_data_smithi194_103039_object131997:head# 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af0b391:::benchmark_data_smithi194_103039_object43077:head# 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af1a130:::benchmark_data_smithi194_103039_object151073:head# 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af1c537:::benchmark_data_smithi194_103039_object75407:head# 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af1fd48:::benchmark_data_smithi194_103039_object50503:head# 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af25774:::benchmark_data_smithi194_103039_object119196:head# 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af3583e:::benchmark_data_smithi194_103039_object79746:head# 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af35dd9:::benchmark_data_smithi194_103039_object123257:head# 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af3b713:::benchmark_data_smithi194_103039_object73834:head# 2024-04-29T14:21:29.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af426dc:::benchmark_data_smithi194_103039_object104823:head# 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af431be:::benchmark_data_smithi194_103039_object88551:head# 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af437c0:::benchmark_data_smithi194_103039_object54126:head# 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af4e071:::benchmark_data_smithi194_103039_object48871:head# 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af51ab7:::benchmark_data_smithi194_103039_object85856:head# 2024-04-29T14:21:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af51f39:::benchmark_data_smithi194_103039_object127784:head# 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af56199:::benchmark_data_smithi194_103039_object56314:head# 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af56f1b:::benchmark_data_smithi194_103039_object116456:head# 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af685b2:::benchmark_data_smithi194_103039_object43799:head# 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af68812:::benchmark_data_smithi194_103039_object134931:head# 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af69472:::benchmark_data_smithi194_103039_object98691:head# 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af7b313:::benchmark_data_smithi194_103039_object54654:head# 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af7bc45:::benchmark_data_smithi194_103039_object74146:head# 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af7f7e4:::benchmark_data_smithi194_103039_object103638:head# 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af8c617:::benchmark_data_smithi194_103039_object45503:head# 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af91b3c:::benchmark_data_smithi194_103039_object58873:head# 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af935a3:::benchmark_data_smithi194_103039_object91478:head# 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4af97dd9:::benchmark_data_smithi194_103039_object127230:head# 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afa625d:::benchmark_data_smithi194_103039_object93710:head# 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afa645b:::benchmark_data_smithi194_103039_object56442:head# 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afb1ce1:::benchmark_data_smithi194_103039_object53125:head# 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afbdd8d:::benchmark_data_smithi194_103039_object124272:head# 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afbe6cf:::benchmark_data_smithi194_103039_object64023:head# 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afc4496:::benchmark_data_smithi194_103039_object103769:head# 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afcb494:::benchmark_data_smithi194_103039_object109654:head# 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4afd032b:::benchmark_data_smithi194_103039_object60500:head# 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4affa39c:::benchmark_data_smithi194_103039_object72861:head# 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b00c333:::benchmark_data_smithi194_103039_object105928:head# 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b017ed9:::benchmark_data_smithi194_103039_object150222:head# 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b01e709:::benchmark_data_smithi194_103039_object106088:head# 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b03e719:::benchmark_data_smithi194_103039_object46029:head# 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b03ef21:::benchmark_data_smithi194_103039_object139295:head# 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b042b16:::benchmark_data_smithi194_103039_object68962:head# 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b04792e:::benchmark_data_smithi194_103039_object128243:head# 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b05dbb1:::benchmark_data_smithi194_103039_object43247:head# 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0696ac:::benchmark_data_smithi194_103039_object71117:head# 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0697ec:::benchmark_data_smithi194_103039_object92552:head# 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0734ed:::benchmark_data_smithi194_103039_object71542:head# 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b087159:::benchmark_data_smithi194_103039_object70390:head# 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b09a5bf:::benchmark_data_smithi194_103039_object104576:head# 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0a6ae3:::benchmark_data_smithi194_103039_object145892:head# 2024-04-29T14:21:30.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0b3c43:::benchmark_data_smithi194_103039_object46564:head# 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0bacb0:::benchmark_data_smithi194_103039_object42806:head# 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0bad85:::benchmark_data_smithi194_103039_object43018:head# 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0cbcbd:::benchmark_data_smithi194_103039_object48466:head# 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0d1b30:::benchmark_data_smithi194_103039_object138127:head# 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0d4f87:::benchmark_data_smithi194_103039_object53790:head# 2024-04-29T14:21:30.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0de6d5:::benchmark_data_smithi194_103039_object75166:head# 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0eb1d5:::benchmark_data_smithi194_103039_object60961:head# 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b0f3512:::benchmark_data_smithi194_103039_object43847:head# 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b102058:::benchmark_data_smithi194_103039_object94242:head# 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b106c55:::benchmark_data_smithi194_103039_object86699:head# 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b10c153:::benchmark_data_smithi194_103039_object107503:head# 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b111d06:::benchmark_data_smithi194_103039_object63225:head# 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b11569a:::benchmark_data_smithi194_103039_object95787:head# 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b11af15:::benchmark_data_smithi194_103039_object114491:head# 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b121f04:::benchmark_data_smithi194_103039_object100532:head# 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b154a3f:::benchmark_data_smithi194_103039_object152089:head# 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b15c33a:::benchmark_data_smithi194_103039_object47652:head# 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b162197:::benchmark_data_smithi194_103039_object93969:head# 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b16ed30:::benchmark_data_smithi194_103039_object54806:head# 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b170b15:::benchmark_data_smithi194_103039_object105105:head# 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b18beff:::benchmark_data_smithi194_103039_object53008:head# 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b19b2e3:::benchmark_data_smithi194_103039_object139172:head# 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b19cdf8:::benchmark_data_smithi194_103039_object89053:head# 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b1b5856:::benchmark_data_smithi194_103039_object60531:head# 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b1c40bf:::benchmark_data_smithi194_103039_object71079:head# 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b1cbf17:::benchmark_data_smithi194_103039_object111490:head# 2024-04-29T14:21:30.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b1d1659:::benchmark_data_smithi194_103039_object81679:head# 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b1ed4c1:::benchmark_data_smithi194_103039_object69362:head# 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b1edb31:::benchmark_data_smithi194_103039_object132101:head# 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b20323e:::benchmark_data_smithi194_103039_object75970:head# 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2132d5:::benchmark_data_smithi194_103039_object64131:head# 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b213d73:::benchmark_data_smithi194_103039_object57698:head# 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b21817c:::benchmark_data_smithi194_103039_object47921:head# 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b21a11c:::benchmark_data_smithi194_103039_object86317:head# 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b21d6e3:::benchmark_data_smithi194_103039_object118986:head# 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b21fc4e:::benchmark_data_smithi194_103039_object85420:head# 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b221396:::benchmark_data_smithi194_103039_object79963:head# 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b227536:::benchmark_data_smithi194_103039_object154498:head# 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2326c2:::benchmark_data_smithi194_103039_object81334:head# 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b24a997:::benchmark_data_smithi194_103039_object65758:head# 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b24b8e1:::benchmark_data_smithi194_103039_object42362:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2515c6:::benchmark_data_smithi194_103039_object57823:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2585e0:::benchmark_data_smithi194_103039_object107091:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2669c8:::benchmark_data_smithi194_103039_object81036:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b26a5ec:::benchmark_data_smithi194_103039_object78134:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b271fbf:::benchmark_data_smithi194_103039_object75506:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b275a7a:::benchmark_data_smithi194_103039_object143385:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b27fed0:::benchmark_data_smithi194_103039_object57760:head# 2024-04-29T14:21:30.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b287916:::benchmark_data_smithi194_103039_object130106:head# 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b29f1ee:::benchmark_data_smithi194_103039_object56220:head# 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2a4c7c:::benchmark_data_smithi194_103039_object149041:head# 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2a8d98:::benchmark_data_smithi194_103039_object87866:head# 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2a8e5c:::benchmark_data_smithi194_103039_object87822:head# 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2ad71f:::benchmark_data_smithi194_103039_object64657:head# 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2b0abe:::benchmark_data_smithi194_103039_object144242:head# 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2b23f7:::benchmark_data_smithi194_103039_object142837:head# 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2b3f6f:::benchmark_data_smithi194_103039_object51799:head# 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2be807:::benchmark_data_smithi194_103039_object140370:head# 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2c8046:::benchmark_data_smithi194_103039_object53052:head# 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2cbefb:::benchmark_data_smithi194_103039_object143338:head# 2024-04-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2ce153:::benchmark_data_smithi194_103039_object41654:head# 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2d414e:::benchmark_data_smithi194_103039_object135294:head# 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b2f1c0c:::benchmark_data_smithi194_103039_object56727:head# 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b30bc6d:::benchmark_data_smithi194_103039_object45131:head# 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b345982:::benchmark_data_smithi194_103039_object58467:head# 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b35b08c:::benchmark_data_smithi194_103039_object143189:head# 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b36bb8f:::benchmark_data_smithi194_103039_object142033:head# 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b37ee19:::benchmark_data_smithi194_103039_object63408:head# 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b37ffb4:::benchmark_data_smithi194_103039_object102690:head# 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b39177e:::benchmark_data_smithi194_103039_object73450:head# 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b394be5:::benchmark_data_smithi194_103039_object119527:head# 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3a051f:::benchmark_data_smithi194_103039_object141058:head# 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3ae6f6:::benchmark_data_smithi194_103039_object95188:head# 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3b327c:::benchmark_data_smithi194_103039_object150010:head# 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3b5e72:::benchmark_data_smithi194_103039_object63092:head# 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3bc1dd:::benchmark_data_smithi194_103039_object151561:head# 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3c53ee:::benchmark_data_smithi194_103039_object76861:head# 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3d332a:::benchmark_data_smithi194_103039_object141748:head# 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3d59d0:::benchmark_data_smithi194_103039_object67829:head# 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3e5320:::benchmark_data_smithi194_103039_object125776:head# 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3e541d:::benchmark_data_smithi194_103039_object138539:head# 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b3f75f5:::benchmark_data_smithi194_103039_object109665:head# 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b428887:::benchmark_data_smithi194_103039_object136965:head# 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b442917:::benchmark_data_smithi194_103039_object72215:head# 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b459c74:::benchmark_data_smithi194_103039_object135682:head# 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b45e38e:::benchmark_data_smithi194_103039_object137174:head# 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b465a53:::benchmark_data_smithi194_103039_object49299:head# 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b47fc6d:::benchmark_data_smithi194_103039_object100065:head# 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b48b409:::benchmark_data_smithi194_103039_object44050:head# 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b48b756:::benchmark_data_smithi194_103039_object99129:head# 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b48d264:::benchmark_data_smithi194_103039_object133165:head# 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b497b29:::benchmark_data_smithi194_103039_object147078:head# 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b49f0d3:::benchmark_data_smithi194_103039_object82195:head# 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b49f219:::benchmark_data_smithi194_103039_object155240:head# 2024-04-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b4b2892:::benchmark_data_smithi194_103039_object106815:head# 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b4cac20:::benchmark_data_smithi194_103039_object118702:head# 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b4d8ffa:::benchmark_data_smithi194_103039_object47525:head# 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b4e3aa5:::benchmark_data_smithi194_103039_object143857:head# 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b4ef4ed:::benchmark_data_smithi194_103039_object54356:head# 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b50751d:::benchmark_data_smithi194_103039_object84535:head# 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b507544:::benchmark_data_smithi194_103039_object94232:head# 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b50bf89:::benchmark_data_smithi194_103039_object136419:head# 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b543b8c:::benchmark_data_smithi194_103039_object92899:head# 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b546181:::benchmark_data_smithi194_103039_object48946:head# 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5497ff:::benchmark_data_smithi194_103039_object83421:head# 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b54c913:::benchmark_data_smithi194_103039_object104214:head# 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b54dbc0:::benchmark_data_smithi194_103039_object50660:head# 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b54fd82:::benchmark_data_smithi194_103039_object87737:head# 2024-04-29T14:21:30.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b558487:::benchmark_data_smithi194_103039_object88689:head# 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b573a8a:::benchmark_data_smithi194_103039_object89547:head# 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b577c32:::benchmark_data_smithi194_103039_object108370:head# 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b579de9:::benchmark_data_smithi194_103039_object129676:head# 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b57b772:::benchmark_data_smithi194_103039_object53413:head# 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5821fa:::benchmark_data_smithi194_103039_object49697:head# 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b585501:::benchmark_data_smithi194_103039_object123474:head# 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b585718:::benchmark_data_smithi194_103039_object136720:head# 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b58815c:::benchmark_data_smithi194_103039_object83252:head# 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b58e829:::benchmark_data_smithi194_103039_object44669:head# 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b59f561:::benchmark_data_smithi194_103039_object131686:head# 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5a35b2:::benchmark_data_smithi194_103039_object104158:head# 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5ab155:::benchmark_data_smithi194_103039_object139873:head# 2024-04-29T14:21:30.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5b6940:::benchmark_data_smithi194_103039_object118541:head# 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5ba0fe:::benchmark_data_smithi194_103039_object102462:head# 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5ba69a:::benchmark_data_smithi194_103039_object147598:head# 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5c33c1:::benchmark_data_smithi194_103039_object109040:head# 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5cbe9c:::benchmark_data_smithi194_103039_object109572:head# 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5cfa8f:::benchmark_data_smithi194_103039_object62479:head# 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5da6be:::benchmark_data_smithi194_103039_object104857:head# 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5dd525:::benchmark_data_smithi194_103039_object84918:head# 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5e8557:::benchmark_data_smithi194_103039_object48950:head# 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5ee8c6:::benchmark_data_smithi194_103039_object128543:head# 2024-04-29T14:21:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5fcd16:::benchmark_data_smithi194_103039_object103798:head# 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b5ff58f:::benchmark_data_smithi194_103039_object91575:head# 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b608c5a:::benchmark_data_smithi194_103039_object135023:head# 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b61b5b5:::benchmark_data_smithi194_103039_object63455:head# 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b61d513:::benchmark_data_smithi194_103039_object97213:head# 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b621451:::benchmark_data_smithi194_103039_object62420:head# 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6243c3:::benchmark_data_smithi194_103039_object44533:head# 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6310e4:::benchmark_data_smithi194_103039_object111198:head# 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b64491b:::benchmark_data_smithi194_103039_object156071:head# 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b648e9a:::benchmark_data_smithi194_103039_object149268:head# 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b64e8df:::benchmark_data_smithi194_103039_object146455:head# 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b656ba2:::benchmark_data_smithi194_103039_object81669:head# 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b661fef:::benchmark_data_smithi194_103039_object127718:head# 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b672803:::benchmark_data_smithi194_103039_object143657:head# 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b681d7f:::benchmark_data_smithi194_103039_object89492:head# 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6956bd:::benchmark_data_smithi194_103039_object150370:head# 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6a0a9d:::benchmark_data_smithi194_103039_object41876:head# 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6a169f:::benchmark_data_smithi194_103039_object82383:head# 2024-04-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6b1c6a:::benchmark_data_smithi194_103039_object118323:head# 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6bc58b:::benchmark_data_smithi194_103039_object75673:head# 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6c21f3:::benchmark_data_smithi194_103039_object155603:head# 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6cbc6d:::benchmark_data_smithi194_103039_object138553:head# 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6cc9a7:::benchmark_data_smithi194_103039_object113966:head# 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6ce41c:::benchmark_data_smithi194_103039_object112131:head# 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6ceac0:::benchmark_data_smithi194_103039_object96820:head# 2024-04-29T14:21:30.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6d21af:::benchmark_data_smithi194_103039_object153162:head# 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6e6fb1:::benchmark_data_smithi194_103039_object50384:head# 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6ecbb6:::benchmark_data_smithi194_103039_object131810:head# 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b6fb839:::benchmark_data_smithi194_103039_object60412:head# 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b734ffb:::benchmark_data_smithi194_103039_object107849:head# 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b744779:::benchmark_data_smithi194_103039_object46970:head# 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b749f2a:::benchmark_data_smithi194_103039_object136353:head# 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b74df3c:::benchmark_data_smithi194_103039_object71714:head# 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b751dc9:::benchmark_data_smithi194_103039_object105335:head# 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b75262c:::benchmark_data_smithi194_103039_object44184:head# 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b75a27e:::benchmark_data_smithi194_103039_object92323:head# 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7612ae:::benchmark_data_smithi194_103039_object120286:head# 2024-04-29T14:21:30.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b76378d:::benchmark_data_smithi194_103039_object117234:head# 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b76f638:::benchmark_data_smithi194_103039_object80843:head# 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7702c2:::benchmark_data_smithi194_103039_object75288:head# 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b77059e:::benchmark_data_smithi194_103039_object124954:head# 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b776cd8:::benchmark_data_smithi194_103039_object86029:head# 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b78710b:::benchmark_data_smithi194_103039_object92534:head# 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b78e1a5:::benchmark_data_smithi194_103039_object79463:head# 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7928be:::benchmark_data_smithi194_103039_object65911:head# 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b793874:::benchmark_data_smithi194_103039_object96636:head# 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7b1366:::benchmark_data_smithi194_103039_object91493:head# 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7c4bf8:::benchmark_data_smithi194_103039_object124183:head# 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7cbb21:::benchmark_data_smithi194_103039_object84704:head# 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7eaaf4:::benchmark_data_smithi194_103039_object110158:head# 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b7f0e87:::benchmark_data_smithi194_103039_object50911:head# 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8135bb:::benchmark_data_smithi194_103039_object99970:head# 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b82150a:::benchmark_data_smithi194_103039_object141015:head# 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b82434c:::benchmark_data_smithi194_103039_object58616:head# 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b83ba3e:::benchmark_data_smithi194_103039_object96946:head# 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b84cdc8:::benchmark_data_smithi194_103039_object57647:head# 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b84e0d9:::benchmark_data_smithi194_103039_object120380:head# 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b84f9fb:::benchmark_data_smithi194_103039_object133617:head# 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b859150:::benchmark_data_smithi194_103039_object142575:head# 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b86a600:::benchmark_data_smithi194_103039_object153764:head# 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b86a73e:::benchmark_data_smithi194_103039_object153726:head# 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b86f47f:::benchmark_data_smithi194_103039_object107437:head# 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b872cbe:::benchmark_data_smithi194_103039_object47116:head# 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b875fed:::benchmark_data_smithi194_103039_object154336:head# 2024-04-29T14:21:30.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b87619c:::benchmark_data_smithi194_103039_object136892:head# 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b884e89:::benchmark_data_smithi194_103039_object142304:head# 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b88ddbd:::benchmark_data_smithi194_103039_object115888:head# 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8a6db5:::benchmark_data_smithi194_103039_object130337:head# 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8b3c6c:::benchmark_data_smithi194_103039_object144785:head# 2024-04-29T14:21:30.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8b7b54:::benchmark_data_smithi194_103039_object142545:head# 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8b8ace:::benchmark_data_smithi194_103039_object124502:head# 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8bdd1f:::benchmark_data_smithi194_103039_object89933:head# 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8c79dd:::benchmark_data_smithi194_103039_object103373:head# 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8cf7b6:::benchmark_data_smithi194_103039_object116052:head# 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8d7967:::benchmark_data_smithi194_103039_object143647:head# 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8e2b2f:::benchmark_data_smithi194_103039_object146129:head# 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b8fa27d:::benchmark_data_smithi194_103039_object94940:head# 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9028dd:::benchmark_data_smithi194_103039_object125719:head# 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b90948d:::benchmark_data_smithi194_103039_object152810:head# 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9161f8:::benchmark_data_smithi194_103039_object115331:head# 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b92362d:::benchmark_data_smithi194_103039_object72262:head# 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9244a9:::benchmark_data_smithi194_103039_object80690:head# 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b932f28:::benchmark_data_smithi194_103039_object59412:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b93c66c:::benchmark_data_smithi194_103039_object109766:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b93ee0b:::benchmark_data_smithi194_103039_object118959:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b94397b:::benchmark_data_smithi194_103039_object150784:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9489fe:::benchmark_data_smithi194_103039_object138893:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b952865:::benchmark_data_smithi194_103039_object146623:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b953bbb:::benchmark_data_smithi194_103039_object41646:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9640d1:::benchmark_data_smithi194_103039_object49175:head# 2024-04-29T14:21:30.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b96f1fc:::benchmark_data_smithi194_103039_object89213:head# 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b96f807:::benchmark_data_smithi194_103039_object103033:head# 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9757e1:::benchmark_data_smithi194_103039_object55287:head# 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b97ac5f:::benchmark_data_smithi194_103039_object114881:head# 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9878dd:::benchmark_data_smithi194_103039_object59193:head# 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b988880:::benchmark_data_smithi194_103039_object123062:head# 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b98d2f9:::benchmark_data_smithi194_103039_object137796:head# 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b99a832:::benchmark_data_smithi194_103039_object51040:head# 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b99a913:::benchmark_data_smithi194_103039_object83832:head# 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b99b8fc:::benchmark_data_smithi194_103039_object78916:head# 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b99ef53:::benchmark_data_smithi194_103039_object102157:head# 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9a5f18:::benchmark_data_smithi194_103039_object66442:head# 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9aa78a:::benchmark_data_smithi194_103039_object95519:head# 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9b3f2b:::benchmark_data_smithi194_103039_object50436:head# 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9b4016:::benchmark_data_smithi194_103039_object134041:head# 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9c66c4:::benchmark_data_smithi194_103039_object123215:head# 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9cc155:::benchmark_data_smithi194_103039_object102392:head# 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9cde57:::benchmark_data_smithi194_103039_object113875:head# 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9e217a:::benchmark_data_smithi194_103039_object154421:head# 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9e2cf1:::benchmark_data_smithi194_103039_object121764:head# 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9ea567:::benchmark_data_smithi194_103039_object44766:head# 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9edeef:::benchmark_data_smithi194_103039_object120071:head# 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9f3165:::benchmark_data_smithi194_103039_object137867:head# 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4b9fc3c6:::benchmark_data_smithi194_103039_object56566:head# 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba01f1b:::benchmark_data_smithi194_103039_object74530:head# 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba2f9b2:::benchmark_data_smithi194_103039_object105637:head# 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba2fc7c:::benchmark_data_smithi194_103039_object109963:head# 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba32c59:::benchmark_data_smithi194_103039_object107057:head# 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba38533:::benchmark_data_smithi194_103039_object72173:head# 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba42cd8:::benchmark_data_smithi194_103039_object103288:head# 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba45587:::benchmark_data_smithi194_103039_object156604:head# 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba50449:::benchmark_data_smithi194_103039_object93965:head# 2024-04-29T14:21:30.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba5192e:::benchmark_data_smithi194_103039_object152200:head# 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba5c665:::benchmark_data_smithi194_103039_object103098:head# 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba7321a:::benchmark_data_smithi194_103039_object121338:head# 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba73e86:::benchmark_data_smithi194_103039_object117504:head# 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba8a019:::benchmark_data_smithi194_103039_object91534:head# 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba95f6e:::benchmark_data_smithi194_103039_object104608:head# 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ba97355:::benchmark_data_smithi194_103039_object146879:head# 2024-04-29T14:21:30.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4baa14b1:::benchmark_data_smithi194_103039_object146919:head# 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4baa2688:::benchmark_data_smithi194_103039_object56130:head# 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4baaca09:::benchmark_data_smithi194_103039_object72138:head# 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4baad9d0:::benchmark_data_smithi194_103039_object146912:head# 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bab1f7d:::benchmark_data_smithi194_103039_object142346:head# 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bac61af:::benchmark_data_smithi194_103039_object60081:head# 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bada45b:::benchmark_data_smithi194_103039_object85078:head# 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4baff3bf:::benchmark_data_smithi194_103039_object65289:head# 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb04328:::benchmark_data_smithi194_103039_object127362:head# 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb26a75:::benchmark_data_smithi194_103039_object59370:head# 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb2a1e6:::benchmark_data_smithi194_103039_object153903:head# 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb3354a:::benchmark_data_smithi194_103039_object149145:head# 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb339f3:::benchmark_data_smithi194_103039_object74632:head# 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb36ff6:::benchmark_data_smithi194_103039_object117096:head# 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb44733:::benchmark_data_smithi194_103039_object104861:head# 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb4868c:::benchmark_data_smithi194_103039_object85998:head# 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb7047d:::benchmark_data_smithi194_103039_object56263:head# 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb72b67:::benchmark_data_smithi194_103039_object128244:head# 2024-04-29T14:21:30.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb8e940:::benchmark_data_smithi194_103039_object149744:head# 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb9238c:::benchmark_data_smithi194_103039_object121492:head# 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bb95821:::benchmark_data_smithi194_103039_object105438:head# 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bba436c:::benchmark_data_smithi194_103039_object43597:head# 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbaebef:::benchmark_data_smithi194_103039_object44084:head# 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbaf1dc:::benchmark_data_smithi194_103039_object123020:head# 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbb67da:::benchmark_data_smithi194_103039_object61724:head# 2024-04-29T14:21:30.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbbc5f9:::benchmark_data_smithi194_103039_object55829:head# 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbc5b61:::benchmark_data_smithi194_103039_object135718:head# 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbc8a81:::benchmark_data_smithi194_103039_object102780:head# 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbcafe8:::benchmark_data_smithi194_103039_object92582:head# 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbcb346:::benchmark_data_smithi194_103039_object102127:head# 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbd0e1a:::benchmark_data_smithi194_103039_object69535:head# 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbd1c7e:::benchmark_data_smithi194_103039_object73157:head# 2024-04-29T14:21:30.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbd1f22:::benchmark_data_smithi194_103039_object97240:head# 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbd8939:::benchmark_data_smithi194_103039_object61618:head# 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbe097b:::benchmark_data_smithi194_103039_object73310:head# 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbe8db8:::benchmark_data_smithi194_103039_object89344:head# 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbec6f8:::benchmark_data_smithi194_103039_object140049:head# 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbf019d:::benchmark_data_smithi194_103039_object125186:head# 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bbf80ed:::benchmark_data_smithi194_103039_object108638:head# 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc06960:::benchmark_data_smithi194_103039_object71446:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc1b29a:::benchmark_data_smithi194_103039_object92052:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc1ff0e:::benchmark_data_smithi194_103039_object64143:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc23410:::benchmark_data_smithi194_103039_object149481:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc2565e:::benchmark_data_smithi194_103039_object128542:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc271e5:::benchmark_data_smithi194_103039_object156035:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc2a3b1:::benchmark_data_smithi194_103039_object86081:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc31a10:::benchmark_data_smithi194_103039_object60010:head# 2024-04-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc327cf:::benchmark_data_smithi194_103039_object131756:head# 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc3d960:::benchmark_data_smithi194_103039_object152099:head# 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc3e802:::benchmark_data_smithi194_103039_object99983:head# 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc44832:::benchmark_data_smithi194_103039_object51540:head# 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc4508d:::benchmark_data_smithi194_103039_object103901:head# 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc51d14:::benchmark_data_smithi194_103039_object69108:head# 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc57256:::benchmark_data_smithi194_103039_object121298:head# 2024-04-29T14:21:30.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc574a4:::benchmark_data_smithi194_103039_object129355:head# 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc63f57:::benchmark_data_smithi194_103039_object78655:head# 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc6a833:::benchmark_data_smithi194_103039_object57207:head# 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc7fda5:::benchmark_data_smithi194_103039_object92468:head# 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bc9dcde:::benchmark_data_smithi194_103039_object154608:head# 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bcb8f55:::benchmark_data_smithi194_103039_object94730:head# 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bcd2db3:::benchmark_data_smithi194_103039_object67502:head# 2024-04-29T14:21:30.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bcd3fc3:::benchmark_data_smithi194_103039_object108601:head# 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bcdd477:::benchmark_data_smithi194_103039_object106661:head# 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bcde77e:::benchmark_data_smithi194_103039_object58496:head# 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bce8717:::benchmark_data_smithi194_103039_object124793:head# 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bcfc745:::benchmark_data_smithi194_103039_object99997:head# 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd06545:::benchmark_data_smithi194_103039_object105080:head# 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd07158:::benchmark_data_smithi194_103039_object93506:head# 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd0b8ec:::benchmark_data_smithi194_103039_object153443:head# 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd0fbaf:::benchmark_data_smithi194_103039_object108115:head# 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd17171:::benchmark_data_smithi194_103039_object55368:head# 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd288ee:::benchmark_data_smithi194_103039_object147106:head# 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd2c024:::benchmark_data_smithi194_103039_object82662:head# 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd3203b:::benchmark_data_smithi194_103039_object147605:head# 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd45a37:::benchmark_data_smithi194_103039_object141866:head# 2024-04-29T14:21:30.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd476d1:::benchmark_data_smithi194_103039_object111779:head# 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd48ec9:::benchmark_data_smithi194_103039_object152522:head# 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd52e9d:::benchmark_data_smithi194_103039_object101157:head# 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd5b477:::benchmark_data_smithi194_103039_object67459:head# 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd5f692:::benchmark_data_smithi194_103039_object140258:head# 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd64425:::benchmark_data_smithi194_103039_object122301:head# 2024-04-29T14:21:30.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd66bcf:::benchmark_data_smithi194_103039_object79573:head# 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd80139:::benchmark_data_smithi194_103039_object41627:head# 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd815dd:::benchmark_data_smithi194_103039_object145451:head# 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd83aaa:::benchmark_data_smithi194_103039_object152053:head# 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd84ba2:::benchmark_data_smithi194_103039_object150012:head# 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd9156d:::benchmark_data_smithi194_103039_object109779:head# 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bd9aa23:::benchmark_data_smithi194_103039_object122798:head# 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdab647:::benchmark_data_smithi194_103039_object125202:head# 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdb2235:::benchmark_data_smithi194_103039_object55751:head# 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdb423e:::benchmark_data_smithi194_103039_object47121:head# 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdbaa5b:::benchmark_data_smithi194_103039_object107288:head# 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdc36fe:::benchmark_data_smithi194_103039_object64557:head# 2024-04-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdc8216:::benchmark_data_smithi194_103039_object155776:head# 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdc83b3:::benchmark_data_smithi194_103039_object145167:head# 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdcaff1:::benchmark_data_smithi194_103039_object114043:head# 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdd2b45:::benchmark_data_smithi194_103039_object74485:head# 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bddd46d:::benchmark_data_smithi194_103039_object87842:head# 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bdf133d:::benchmark_data_smithi194_103039_object151723:head# 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be00d67:::benchmark_data_smithi194_103039_object79619:head# 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be02d59:::benchmark_data_smithi194_103039_object98797:head# 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be033ef:::benchmark_data_smithi194_103039_object123145:head# 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be0db2d:::benchmark_data_smithi194_103039_object121736:head# 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be0e4b5:::benchmark_data_smithi194_103039_object99773:head# 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be1f5ee:::benchmark_data_smithi194_103039_object119114:head# 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be3034f:::benchmark_data_smithi194_103039_object81145:head# 2024-04-29T14:21:30.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be4c745:::benchmark_data_smithi194_103039_object140765:head# 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be4f358:::benchmark_data_smithi194_103039_object84440:head# 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be56edd:::benchmark_data_smithi194_103039_object131888:head# 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be5dcce:::benchmark_data_smithi194_103039_object92942:head# 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be65517:::benchmark_data_smithi194_103039_object103035:head# 2024-04-29T14:21:30.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be6ea17:::benchmark_data_smithi194_103039_object152360:head# 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be70b16:::benchmark_data_smithi194_103039_object90109:head# 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be7270a:::benchmark_data_smithi194_103039_object125637:head# 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be82fdb:::benchmark_data_smithi194_103039_object74834:head# 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be83426:::benchmark_data_smithi194_103039_object139162:head# 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4be89c62:::benchmark_data_smithi194_103039_object107504:head# 2024-04-29T14:21:30.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4beb4bdb:::benchmark_data_smithi194_103039_object53975:head# 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bec1170:::benchmark_data_smithi194_103039_object108181:head# 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bec415b:::benchmark_data_smithi194_103039_object141636:head# 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4becbde9:::benchmark_data_smithi194_103039_object55750:head# 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4becda46:::benchmark_data_smithi194_103039_object74991:head# 2024-04-29T14:21:30.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4beced7d:::benchmark_data_smithi194_103039_object115719:head# 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bedbcbe:::benchmark_data_smithi194_103039_object126084:head# 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4beded48:::benchmark_data_smithi194_103039_object48559:head# 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bedf4da:::benchmark_data_smithi194_103039_object73774:head# 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4beedd9a:::benchmark_data_smithi194_103039_object152588:head# 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bef4546:::benchmark_data_smithi194_103039_object42463:head# 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4befb1df:::benchmark_data_smithi194_103039_object51606:head# 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4beff3d5:::benchmark_data_smithi194_103039_object78794:head# 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf068e7:::benchmark_data_smithi194_103039_object83699:head# 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf29e32:::benchmark_data_smithi194_103039_object88451:head# 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf2d089:::benchmark_data_smithi194_103039_object113078:head# 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf2eacc:::benchmark_data_smithi194_103039_object63023:head# 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf4a7bf:::benchmark_data_smithi194_103039_object125152:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf4ab5a:::benchmark_data_smithi194_103039_object123847:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf52f8b:::benchmark_data_smithi194_103039_object129081:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf5c9f0:::benchmark_data_smithi194_103039_object72326:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf5d8e4:::benchmark_data_smithi194_103039_object124701:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf68d10:::benchmark_data_smithi194_103039_object65599:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf6acbf:::benchmark_data_smithi194_103039_object56491:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf73a1e:::benchmark_data_smithi194_103039_object62127:head# 2024-04-29T14:21:30.072 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf7b80f:::benchmark_data_smithi194_103039_object41546:head# 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bf9f9c5:::benchmark_data_smithi194_103039_object103411:head# 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bfa10d8:::benchmark_data_smithi194_103039_object47580:head# 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bfac813:::benchmark_data_smithi194_103039_object117783:head# 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bfb447e:::benchmark_data_smithi194_103039_object91825:head# 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bfb50cb:::benchmark_data_smithi194_103039_object95700:head# 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bfd061b:::benchmark_data_smithi194_103039_object43519:head# 2024-04-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bfd5352:::benchmark_data_smithi194_103039_object47679:head# 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bfedd64:::benchmark_data_smithi194_103039_object61483:head# 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4bffb2a8:::benchmark_data_smithi194_103039_object58482:head# 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c006ad9:::benchmark_data_smithi194_103039_object87545:head# 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c01c9eb:::benchmark_data_smithi194_103039_object96281:head# 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c02989a:::benchmark_data_smithi194_103039_object71928:head# 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c02991b:::benchmark_data_smithi194_103039_object57245:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c03957c:::benchmark_data_smithi194_103039_object104574:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c03cd5d:::benchmark_data_smithi194_103039_object59746:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c04152f:::benchmark_data_smithi194_103039_object140465:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0418d4:::benchmark_data_smithi194_103039_object102800:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c04247f:::benchmark_data_smithi194_103039_object105867:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c050362:::benchmark_data_smithi194_103039_object102798:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c056eb3:::benchmark_data_smithi194_103039_object106627:head# 2024-04-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c057727:::benchmark_data_smithi194_103039_object53989:head# 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c05ba51:::benchmark_data_smithi194_103039_object122624:head# 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c05e077:::benchmark_data_smithi194_103039_object73695:head# 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c069454:::benchmark_data_smithi194_103039_object89998:head# 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c07615a:::benchmark_data_smithi194_103039_object59835:head# 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0775b2:::benchmark_data_smithi194_103039_object105964:head# 2024-04-29T14:21:30.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c07a5d9:::benchmark_data_smithi194_103039_object134274:head# 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c07fd2f:::benchmark_data_smithi194_103039_object63147:head# 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c083d8b:::benchmark_data_smithi194_103039_object76794:head# 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c08c50f:::benchmark_data_smithi194_103039_object83683:head# 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c09ce53:::benchmark_data_smithi194_103039_object138522:head# 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0bb13e:::benchmark_data_smithi194_103039_object117734:head# 2024-04-29T14:21:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0c090d:::benchmark_data_smithi194_103039_object88487:head# 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0c272c:::benchmark_data_smithi194_103039_object147301:head# 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0c3741:::benchmark_data_smithi194_103039_object98328:head# 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0c5dac:::benchmark_data_smithi194_103039_object121848:head# 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0c64ae:::benchmark_data_smithi194_103039_object122345:head# 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0cac88:::benchmark_data_smithi194_103039_object85505:head# 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0cff68:::benchmark_data_smithi194_103039_object76863:head# 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0d7fd1:::benchmark_data_smithi194_103039_object125124:head# 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0e7fa5:::benchmark_data_smithi194_103039_object68110:head# 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0f709e:::benchmark_data_smithi194_103039_object106204:head# 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c0f8886:::benchmark_data_smithi194_103039_object44038:head# 2024-04-29T14:21:30.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c104a69:::benchmark_data_smithi194_103039_object65979:head# 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c11f0a5:::benchmark_data_smithi194_103039_object143513:head# 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c12a103:::benchmark_data_smithi194_103039_object73218:head# 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c134fab:::benchmark_data_smithi194_103039_object53079:head# 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c14d8ed:::benchmark_data_smithi194_103039_object67660:head# 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c159523:::benchmark_data_smithi194_103039_object148470:head# 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c15967c:::benchmark_data_smithi194_103039_object112882:head# 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c15df89:::benchmark_data_smithi194_103039_object108853:head# 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c16105a:::benchmark_data_smithi194_103039_object62618:head# 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c163385:::benchmark_data_smithi194_103039_object129797:head# 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c16c727:::benchmark_data_smithi194_103039_object134988:head# 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c173e80:::benchmark_data_smithi194_103039_object120027:head# 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c17954b:::benchmark_data_smithi194_103039_object65538:head# 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1806f6:::benchmark_data_smithi194_103039_object47105:head# 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c18dda5:::benchmark_data_smithi194_103039_object46088:head# 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c18ed10:::benchmark_data_smithi194_103039_object52181:head# 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c196814:::benchmark_data_smithi194_103039_object59465:head# 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c19999b:::benchmark_data_smithi194_103039_object72105:head# 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c19c452:::benchmark_data_smithi194_103039_object153208:head# 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1b82ca:::benchmark_data_smithi194_103039_object147664:head# 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1c1d40:::benchmark_data_smithi194_103039_object87306:head# 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1cbe8d:::benchmark_data_smithi194_103039_object45078:head# 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1cdd25:::benchmark_data_smithi194_103039_object151378:head# 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1dc24e:::benchmark_data_smithi194_103039_object137593:head# 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1de0fe:::benchmark_data_smithi194_103039_object119779:head# 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1ec8e6:::benchmark_data_smithi194_103039_object58043:head# 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1fd346:::benchmark_data_smithi194_103039_object92426:head# 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c1ffbe9:::benchmark_data_smithi194_103039_object67116:head# 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c20a458:::benchmark_data_smithi194_103039_object146511:head# 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c21556c:::benchmark_data_smithi194_103039_object147938:head# 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c219d23:::benchmark_data_smithi194_103039_object68509:head# 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c219ff1:::benchmark_data_smithi194_103039_object136406:head# 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c21eb64:::benchmark_data_smithi194_103039_object115356:head# 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c22e639:::benchmark_data_smithi194_103039_object140655:head# 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c238f64:::benchmark_data_smithi194_103039_object47211:head# 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c243ea4:::benchmark_data_smithi194_103039_object106940:head# 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c24ec4a:::benchmark_data_smithi194_103039_object147473:head# 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c25c5e1:::benchmark_data_smithi194_103039_object42916:head# 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c25fa66:::benchmark_data_smithi194_103039_object63467:head# 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2662b3:::benchmark_data_smithi194_103039_object112707:head# 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c26844d:::benchmark_data_smithi194_103039_object111709:head# 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c26c623:::benchmark_data_smithi194_103039_object114563:head# 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c26f722:::benchmark_data_smithi194_103039_object53430:head# 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c27b38e:::benchmark_data_smithi194_103039_object90142:head# 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c27f783:::benchmark_data_smithi194_103039_object132033:head# 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c290dda:::benchmark_data_smithi194_103039_object106013:head# 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c290ea2:::benchmark_data_smithi194_103039_object104989:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c291cb6:::benchmark_data_smithi194_103039_object55052:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c292ea3:::benchmark_data_smithi194_103039_object147052:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2aada4:::benchmark_data_smithi194_103039_object70463:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2be264:::benchmark_data_smithi194_103039_object97890:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2c031b:::benchmark_data_smithi194_103039_object111328:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2c99bb:::benchmark_data_smithi194_103039_object128603:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2cd3de:::benchmark_data_smithi194_103039_object97769:head# 2024-04-29T14:21:30.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2d6d79:::benchmark_data_smithi194_103039_object150110:head# 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2e120d:::benchmark_data_smithi194_103039_object77992:head# 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2efa6e:::benchmark_data_smithi194_103039_object114516:head# 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c2f21ec:::benchmark_data_smithi194_103039_object86070:head# 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3093e0:::benchmark_data_smithi194_103039_object47918:head# 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c314ece:::benchmark_data_smithi194_103039_object125118:head# 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c31fb05:::benchmark_data_smithi194_103039_object63642:head# 2024-04-29T14:21:30.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c32a051:::benchmark_data_smithi194_103039_object42560:head# 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c32fabc:::benchmark_data_smithi194_103039_object122239:head# 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3301c2:::benchmark_data_smithi194_103039_object74239:head# 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c331177:::benchmark_data_smithi194_103039_object103403:head# 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c33509f:::benchmark_data_smithi194_103039_object148779:head# 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c341606:::benchmark_data_smithi194_103039_object124428:head# 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3496d3:::benchmark_data_smithi194_103039_object140706:head# 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c34f689:::benchmark_data_smithi194_103039_object150273:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c357117:::benchmark_data_smithi194_103039_object51267:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c35f99b:::benchmark_data_smithi194_103039_object148750:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c367a58:::benchmark_data_smithi194_103039_object156282:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c36cd78:::benchmark_data_smithi194_103039_object63771:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c36dcca:::benchmark_data_smithi194_103039_object151932:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3735e2:::benchmark_data_smithi194_103039_object108385:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c375f3e:::benchmark_data_smithi194_103039_object76145:head# 2024-04-29T14:21:30.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c37a4ec:::benchmark_data_smithi194_103039_object117400:head# 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3848bb:::benchmark_data_smithi194_103039_object78872:head# 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c38926c:::benchmark_data_smithi194_103039_object155820:head# 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c38eaee:::benchmark_data_smithi194_103039_object102252:head# 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c395f88:::benchmark_data_smithi194_103039_object123775:head# 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c39868e:::benchmark_data_smithi194_103039_object51686:head# 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3a9b4a:::benchmark_data_smithi194_103039_object85012:head# 2024-04-29T14:21:30.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3b9343:::benchmark_data_smithi194_103039_object69863:head# 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3b9cf9:::benchmark_data_smithi194_103039_object145255:head# 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3bc9c7:::benchmark_data_smithi194_103039_object98147:head# 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3bc9fd:::benchmark_data_smithi194_103039_object68954:head# 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3c5b06:::benchmark_data_smithi194_103039_object105114:head# 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3c9c82:::benchmark_data_smithi194_103039_object93787:head# 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3ca2a2:::benchmark_data_smithi194_103039_object90352:head# 2024-04-29T14:21:30.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3cd1fe:::benchmark_data_smithi194_103039_object148980:head# 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3ce566:::benchmark_data_smithi194_103039_object49749:head# 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3e2ec7:::benchmark_data_smithi194_103039_object103941:head# 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3e7854:::benchmark_data_smithi194_103039_object100847:head# 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3ecad8:::benchmark_data_smithi194_103039_object154996:head# 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3ecc0b:::benchmark_data_smithi194_103039_object115104:head# 2024-04-29T14:21:30.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3ef5bf:::benchmark_data_smithi194_103039_object86485:head# 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c3f508c:::benchmark_data_smithi194_103039_object113689:head# 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c404850:::benchmark_data_smithi194_103039_object139439:head# 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c40c0ef:::benchmark_data_smithi194_103039_object127974:head# 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c41423a:::benchmark_data_smithi194_103039_object56784:head# 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c415e46:::benchmark_data_smithi194_103039_object123932:head# 2024-04-29T14:21:30.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4198a3:::benchmark_data_smithi194_103039_object64261:head# 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c419a34:::benchmark_data_smithi194_103039_object97786:head# 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c41dcf6:::benchmark_data_smithi194_103039_object55880:head# 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c420c30:::benchmark_data_smithi194_103039_object136601:head# 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c42353d:::benchmark_data_smithi194_103039_object149721:head# 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c431d76:::benchmark_data_smithi194_103039_object91598:head# 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c43576b:::benchmark_data_smithi194_103039_object68661:head# 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c435e9f:::benchmark_data_smithi194_103039_object131298:head# 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c44ec9e:::benchmark_data_smithi194_103039_object49906:head# 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c451372:::benchmark_data_smithi194_103039_object132104:head# 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c458061:::benchmark_data_smithi194_103039_object149533:head# 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c468089:::benchmark_data_smithi194_103039_object92760:head# 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4769f3:::benchmark_data_smithi194_103039_object45732:head# 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c483c46:::benchmark_data_smithi194_103039_object79790:head# 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c483c9c:::benchmark_data_smithi194_103039_object47021:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4858d1:::benchmark_data_smithi194_103039_object100818:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c488c29:::benchmark_data_smithi194_103039_object115076:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c488e02:::benchmark_data_smithi194_103039_object117202:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c491d6b:::benchmark_data_smithi194_103039_object137747:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c49799d:::benchmark_data_smithi194_103039_object84232:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c49932b:::benchmark_data_smithi194_103039_object83353:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4b50bd:::benchmark_data_smithi194_103039_object61408:head# 2024-04-29T14:21:30.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4ba9e8:::benchmark_data_smithi194_103039_object136523:head# 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4bd338:::benchmark_data_smithi194_103039_object111676:head# 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4c1de7:::benchmark_data_smithi194_103039_object53581:head# 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4c623d:::benchmark_data_smithi194_103039_object75110:head# 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4e3eef:::benchmark_data_smithi194_103039_object109734:head# 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4e6f22:::benchmark_data_smithi194_103039_object129872:head# 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4ea04e:::benchmark_data_smithi194_103039_object56944:head# 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4ee9b3:::benchmark_data_smithi194_103039_object107489:head# 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4f25a0:::benchmark_data_smithi194_103039_object44429:head# 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4f545a:::benchmark_data_smithi194_103039_object62368:head# 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c4f66f4:::benchmark_data_smithi194_103039_object121412:head# 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c510765:::benchmark_data_smithi194_103039_object45659:head# 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c514e5d:::benchmark_data_smithi194_103039_object148084:head# 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c519aaa:::benchmark_data_smithi194_103039_object136208:head# 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c51adfe:::benchmark_data_smithi194_103039_object47455:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c523537:::benchmark_data_smithi194_103039_object44664:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c53776c:::benchmark_data_smithi194_103039_object83937:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c53ce0a:::benchmark_data_smithi194_103039_object146489:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5550b5:::benchmark_data_smithi194_103039_object66137:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5586da:::benchmark_data_smithi194_103039_object143544:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c565a1d:::benchmark_data_smithi194_103039_object118056:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c56cbfe:::benchmark_data_smithi194_103039_object138024:head# 2024-04-29T14:21:30.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c56fbc3:::benchmark_data_smithi194_103039_object47023:head# 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c57475d:::benchmark_data_smithi194_103039_object72297:head# 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c57ddfd:::benchmark_data_smithi194_103039_object63673:head# 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c584857:::benchmark_data_smithi194_103039_object44419:head# 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c58549d:::benchmark_data_smithi194_103039_object44034:head# 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5a37ef:::benchmark_data_smithi194_103039_object71123:head# 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5aff48:::benchmark_data_smithi194_103039_object99293:head# 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5b376a:::benchmark_data_smithi194_103039_object122034:head# 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5b4ead:::benchmark_data_smithi194_103039_object45465:head# 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5b8bf0:::benchmark_data_smithi194_103039_object52802:head# 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5c0bfa:::benchmark_data_smithi194_103039_object149413:head# 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5c6ebb:::benchmark_data_smithi194_103039_object151446:head# 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5dd600:::benchmark_data_smithi194_103039_object70836:head# 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5ec56d:::benchmark_data_smithi194_103039_object48702:head# 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5f3cbb:::benchmark_data_smithi194_103039_object93189:head# 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c5fd14e:::benchmark_data_smithi194_103039_object141429:head# 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6067e4:::benchmark_data_smithi194_103039_object155499:head# 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c60e73f:::benchmark_data_smithi194_103039_object149523:head# 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c615ab1:::benchmark_data_smithi194_103039_object52380:head# 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c62ef5f:::benchmark_data_smithi194_103039_object98093:head# 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c633afd:::benchmark_data_smithi194_103039_object76549:head# 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c64bbde:::benchmark_data_smithi194_103039_object126237:head# 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c65039e:::benchmark_data_smithi194_103039_object124287:head# 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c65612d:::benchmark_data_smithi194_103039_object118737:head# 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6584e2:::benchmark_data_smithi194_103039_object59628:head# 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6738a5:::benchmark_data_smithi194_103039_object142393:head# 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c679388:::benchmark_data_smithi194_103039_object142184:head# 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c67940e:::benchmark_data_smithi194_103039_object52940:head# 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c67a87b:::benchmark_data_smithi194_103039_object81193:head# 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c684bea:::benchmark_data_smithi194_103039_object96650:head# 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6867f5:::benchmark_data_smithi194_103039_object144637:head# 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c68953e:::benchmark_data_smithi194_103039_object132478:head# 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6943e7:::benchmark_data_smithi194_103039_object129723:head# 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c695999:::benchmark_data_smithi194_103039_object107112:head# 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6a2798:::benchmark_data_smithi194_103039_object88885:head# 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6a532a:::benchmark_data_smithi194_103039_object46852:head# 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6ae2eb:::benchmark_data_smithi194_103039_object154430:head# 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6c2e64:::benchmark_data_smithi194_103039_object72710:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6c5cef:::benchmark_data_smithi194_103039_object152503:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6d71ca:::benchmark_data_smithi194_103039_object56856:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6dfab1:::benchmark_data_smithi194_103039_object44680:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6e0de1:::benchmark_data_smithi194_103039_object51980:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6e63f4:::benchmark_data_smithi194_103039_object130952:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c6ed195:::benchmark_data_smithi194_103039_object143296:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c702155:::benchmark_data_smithi194_103039_object124607:head# 2024-04-29T14:21:30.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c70e349:::benchmark_data_smithi194_103039_object74346:head# 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c715cac:::benchmark_data_smithi194_103039_object107129:head# 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7178ae:::benchmark_data_smithi194_103039_object69602:head# 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c718fa6:::benchmark_data_smithi194_103039_object54039:head# 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c719eba:::benchmark_data_smithi194_103039_object140242:head# 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c71aad0:::benchmark_data_smithi194_103039_object150498:head# 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7276a1:::benchmark_data_smithi194_103039_object119718:head# 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c74f738:::benchmark_data_smithi194_103039_object41902:head# 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c752503:::benchmark_data_smithi194_103039_object132116:head# 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c75df4a:::benchmark_data_smithi194_103039_object89762:head# 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c75e99c:::benchmark_data_smithi194_103039_object57826:head# 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c76929f:::benchmark_data_smithi194_103039_object144311:head# 2024-04-29T14:21:30.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7698e4:::benchmark_data_smithi194_103039_object113378:head# 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c76d875:::benchmark_data_smithi194_103039_object131921:head# 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c76f134:::benchmark_data_smithi194_103039_object109523:head# 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7741c1:::benchmark_data_smithi194_103039_object128961:head# 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c775e72:::benchmark_data_smithi194_103039_object55609:head# 2024-04-29T14:21:30.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c77cd33:::benchmark_data_smithi194_103039_object75919:head# 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c79261c:::benchmark_data_smithi194_103039_object152040:head# 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7a0f98:::benchmark_data_smithi194_103039_object133290:head# 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7a3d41:::benchmark_data_smithi194_103039_object137694:head# 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7a4e37:::benchmark_data_smithi194_103039_object103292:head# 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7b05ae:::benchmark_data_smithi194_103039_object93534:head# 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7d44eb:::benchmark_data_smithi194_103039_object70678:head# 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7d5852:::benchmark_data_smithi194_103039_object150856:head# 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7e1964:::benchmark_data_smithi194_103039_object128949:head# 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7e9d63:::benchmark_data_smithi194_103039_object48256:head# 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7eedd1:::benchmark_data_smithi194_103039_object122786:head# 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7f273f:::benchmark_data_smithi194_103039_object136996:head# 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7f7178:::benchmark_data_smithi194_103039_object102031:head# 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c7fb4a7:::benchmark_data_smithi194_103039_object148851:head# 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c81bf42:::benchmark_data_smithi194_103039_object148437:head# 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8280c0:::benchmark_data_smithi194_103039_object75291:head# 2024-04-29T14:21:30.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8389a3:::benchmark_data_smithi194_103039_object68318:head# 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c83b99d:::benchmark_data_smithi194_103039_object120130:head# 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c841be3:::benchmark_data_smithi194_103039_object149136:head# 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8627be:::benchmark_data_smithi194_103039_object42218:head# 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c86664a:::benchmark_data_smithi194_103039_object57851:head# 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c86a684:::benchmark_data_smithi194_103039_object82182:head# 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c86c9ef:::benchmark_data_smithi194_103039_object143770:head# 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c872ac8:::benchmark_data_smithi194_103039_object153804:head# 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c876272:::benchmark_data_smithi194_103039_object64141:head# 2024-04-29T14:21:30.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c876bc2:::benchmark_data_smithi194_103039_object77690:head# 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c881375:::benchmark_data_smithi194_103039_object90896:head# 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c88fca0:::benchmark_data_smithi194_103039_object65363:head# 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c892ed9:::benchmark_data_smithi194_103039_object128760:head# 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c896193:::benchmark_data_smithi194_103039_object67297:head# 2024-04-29T14:21:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8991e0:::benchmark_data_smithi194_103039_object106693:head# 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8a5de0:::benchmark_data_smithi194_103039_object90706:head# 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8a5fb1:::benchmark_data_smithi194_103039_object53336:head# 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8a6512:::benchmark_data_smithi194_103039_object155529:head# 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8b7c0b:::benchmark_data_smithi194_103039_object142777:head# 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8b9de9:::benchmark_data_smithi194_103039_object41587:head# 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8bc2cc:::benchmark_data_smithi194_103039_object41608:head# 2024-04-29T14:21:30.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8be08d:::benchmark_data_smithi194_103039_object75010:head# 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8be612:::benchmark_data_smithi194_103039_object78466:head# 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8c0f49:::benchmark_data_smithi194_103039_object84678:head# 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8c16d8:::benchmark_data_smithi194_103039_object111405:head# 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8c41bd:::benchmark_data_smithi194_103039_object125523:head# 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8c51c0:::benchmark_data_smithi194_103039_object152115:head# 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8c8040:::benchmark_data_smithi194_103039_object131302:head# 2024-04-29T14:21:30.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8c83c7:::benchmark_data_smithi194_103039_object136675:head# 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8cc4d8:::benchmark_data_smithi194_103039_object142260:head# 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8cf7d4:::benchmark_data_smithi194_103039_object66093:head# 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8d4f0f:::benchmark_data_smithi194_103039_object74126:head# 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8db586:::benchmark_data_smithi194_103039_object55459:head# 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8dc9d1:::benchmark_data_smithi194_103039_object144089:head# 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8e56de:::benchmark_data_smithi194_103039_object49644:head# 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8e8c24:::benchmark_data_smithi194_103039_object61524:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8f80d8:::benchmark_data_smithi194_103039_object123718:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c8fa3b9:::benchmark_data_smithi194_103039_object71748:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c916a52:::benchmark_data_smithi194_103039_object82644:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c924a19:::benchmark_data_smithi194_103039_object139197:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c925d76:::benchmark_data_smithi194_103039_object125271:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c925dd3:::benchmark_data_smithi194_103039_object42221:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9330bf:::benchmark_data_smithi194_103039_object47086:head# 2024-04-29T14:21:30.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c93defb:::benchmark_data_smithi194_103039_object42273:head# 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c94788b:::benchmark_data_smithi194_103039_object111018:head# 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9492d6:::benchmark_data_smithi194_103039_object127762:head# 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c951f11:::benchmark_data_smithi194_103039_object152952:head# 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9536d8:::benchmark_data_smithi194_103039_object103391:head# 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9646b1:::benchmark_data_smithi194_103039_object107431:head# 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c985580:::benchmark_data_smithi194_103039_object84148:head# 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c985b25:::benchmark_data_smithi194_103039_object80465:head# 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9a264f:::benchmark_data_smithi194_103039_object116899:head# 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9a8bb8:::benchmark_data_smithi194_103039_object100503:head# 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9a8d19:::benchmark_data_smithi194_103039_object89225:head# 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9b74a6:::benchmark_data_smithi194_103039_object49588:head# 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9bf20f:::benchmark_data_smithi194_103039_object124846:head# 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9cc9a0:::benchmark_data_smithi194_103039_object81239:head# 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4c9cca4e:::benchmark_data_smithi194_103039_object51112:head# 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca04cb7:::benchmark_data_smithi194_103039_object49820:head# 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca15fc3:::benchmark_data_smithi194_103039_object54504:head# 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca2046e:::benchmark_data_smithi194_103039_object113214:head# 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca26055:::benchmark_data_smithi194_103039_object118320:head# 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca2ae38:::benchmark_data_smithi194_103039_object142357:head# 2024-04-29T14:21:30.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca2fcf7:::benchmark_data_smithi194_103039_object76508:head# 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca3738e:::benchmark_data_smithi194_103039_object79853:head# 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca3d16e:::benchmark_data_smithi194_103039_object50945:head# 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca4202a:::benchmark_data_smithi194_103039_object151404:head# 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca4b671:::benchmark_data_smithi194_103039_object45356:head# 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca50a80:::benchmark_data_smithi194_103039_object83583:head# 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca5142e:::benchmark_data_smithi194_103039_object130554:head# 2024-04-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca5181f:::benchmark_data_smithi194_103039_object42230:head# 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca54473:::benchmark_data_smithi194_103039_object61971:head# 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca5456b:::benchmark_data_smithi194_103039_object139837:head# 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca5d94d:::benchmark_data_smithi194_103039_object118775:head# 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca6720e:::benchmark_data_smithi194_103039_object152236:head# 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca6959d:::benchmark_data_smithi194_103039_object76014:head# 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca695e4:::benchmark_data_smithi194_103039_object67183:head# 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca6fe24:::benchmark_data_smithi194_103039_object96471:head# 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca7c310:::benchmark_data_smithi194_103039_object48140:head# 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca901b8:::benchmark_data_smithi194_103039_object77646:head# 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ca98031:::benchmark_data_smithi194_103039_object109379:head# 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4caa5925:::benchmark_data_smithi194_103039_object49110:head# 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4caa93c0:::benchmark_data_smithi194_103039_object98324:head# 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4caaec5e:::benchmark_data_smithi194_103039_object141855:head# 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cab6333:::benchmark_data_smithi194_103039_object143461:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cab93d4:::benchmark_data_smithi194_103039_object63299:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cabacad:::benchmark_data_smithi194_103039_object73035:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cabc188:::benchmark_data_smithi194_103039_object64640:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cac80e7:::benchmark_data_smithi194_103039_object126842:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cad251a:::benchmark_data_smithi194_103039_object57860:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4caecac2:::benchmark_data_smithi194_103039_object89291:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb1e4eb:::benchmark_data_smithi194_103039_object58922:head# 2024-04-29T14:21:30.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb23d35:::benchmark_data_smithi194_103039_object140891:head# 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb32b5e:::benchmark_data_smithi194_103039_object47575:head# 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb49fa9:::benchmark_data_smithi194_103039_object143788:head# 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb7bf68:::benchmark_data_smithi194_103039_object119627:head# 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb8f1ca:::benchmark_data_smithi194_103039_object138208:head# 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb920b8:::benchmark_data_smithi194_103039_object72527:head# 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cb98c69:::benchmark_data_smithi194_103039_object76535:head# 2024-04-29T14:21:30.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbac124:::benchmark_data_smithi194_103039_object134000:head# 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbb8558:::benchmark_data_smithi194_103039_object113053:head# 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbbc8e2:::benchmark_data_smithi194_103039_object81156:head# 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbbce5d:::benchmark_data_smithi194_103039_object139607:head# 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbc6113:::benchmark_data_smithi194_103039_object91585:head# 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbcc276:::benchmark_data_smithi194_103039_object125961:head# 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbd321e:::benchmark_data_smithi194_103039_object106369:head# 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbd709b:::benchmark_data_smithi194_103039_object156310:head# 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbe821b:::benchmark_data_smithi194_103039_object52048:head# 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbf0467:::benchmark_data_smithi194_103039_object64994:head# 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cbf31c4:::benchmark_data_smithi194_103039_object83366:head# 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc12798:::benchmark_data_smithi194_103039_object125595:head# 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc1d797:::benchmark_data_smithi194_103039_object97514:head# 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc1e61d:::benchmark_data_smithi194_103039_object81071:head# 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc20795:::benchmark_data_smithi194_103039_object93025:head# 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc3675b:::benchmark_data_smithi194_103039_object122610:head# 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc3db10:::benchmark_data_smithi194_103039_object95470:head# 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc48e61:::benchmark_data_smithi194_103039_object64498:head# 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc4b2a1:::benchmark_data_smithi194_103039_object44956:head# 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc4ef3a:::benchmark_data_smithi194_103039_object122337:head# 2024-04-29T14:21:30.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc5729a:::benchmark_data_smithi194_103039_object72882:head# 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc62a28:::benchmark_data_smithi194_103039_object93037:head# 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc80091:::benchmark_data_smithi194_103039_object66182:head# 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc890e4:::benchmark_data_smithi194_103039_object101467:head# 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc9223f:::benchmark_data_smithi194_103039_object124105:head# 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc93c6d:::benchmark_data_smithi194_103039_object107473:head# 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cc9c211:::benchmark_data_smithi194_103039_object78453:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cca1b6f:::benchmark_data_smithi194_103039_object96992:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cca3388:::benchmark_data_smithi194_103039_object81531:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccaeb4a:::benchmark_data_smithi194_103039_object115615:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccb615d:::benchmark_data_smithi194_103039_object110162:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccbe5f0:::benchmark_data_smithi194_103039_object79670:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccbeac4:::benchmark_data_smithi194_103039_object151821:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccc7125:::benchmark_data_smithi194_103039_object102969:head# 2024-04-29T14:21:30.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccccd3a:::benchmark_data_smithi194_103039_object44961:head# 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccd2222:::benchmark_data_smithi194_103039_object133318:head# 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccde230:::benchmark_data_smithi194_103039_object96428:head# 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccf2a1f:::benchmark_data_smithi194_103039_object121494:head# 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccfe4f7:::benchmark_data_smithi194_103039_object95993:head# 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ccfe675:::benchmark_data_smithi194_103039_object67207:head# 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd022bb:::benchmark_data_smithi194_103039_object154285:head# 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd11612:::benchmark_data_smithi194_103039_object103857:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd13e0c:::benchmark_data_smithi194_103039_object144610:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd15ef9:::benchmark_data_smithi194_103039_object80872:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd19f0b:::benchmark_data_smithi194_103039_object64275:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd4aa7f:::benchmark_data_smithi194_103039_object92232:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd4c1ab:::benchmark_data_smithi194_103039_object136968:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd56777:::benchmark_data_smithi194_103039_object98006:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd680c3:::benchmark_data_smithi194_103039_object135336:head# 2024-04-29T14:21:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd6b76d:::benchmark_data_smithi194_103039_object74481:head# 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd75cfb:::benchmark_data_smithi194_103039_object148562:head# 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd812e3:::benchmark_data_smithi194_103039_object93908:head# 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cd8d7ac:::benchmark_data_smithi194_103039_object57354:head# 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cda00de:::benchmark_data_smithi194_103039_object71653:head# 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cda4fd1:::benchmark_data_smithi194_103039_object64111:head# 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cdb0055:::benchmark_data_smithi194_103039_object91425:head# 2024-04-29T14:21:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cdb4ddc:::benchmark_data_smithi194_103039_object91871:head# 2024-04-29T14:21:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cdc23ee:::benchmark_data_smithi194_103039_object136918:head# 2024-04-29T14:21:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cdc4114:::benchmark_data_smithi194_103039_object98911:head# 2024-04-29T14:21:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.138 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:21:30.138 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:21:30.138 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:21:30.138 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:21:30.138 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:21:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:30.140 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:30.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:30.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cdd1d5c:::benchmark_data_smithi194_103039_object145384:head# 2024-04-29T14:21:30.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cde0de7:::benchmark_data_smithi194_103039_object148752:head# 2024-04-29T14:21:30.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cde8ed9:::benchmark_data_smithi194_103039_object110881:head# 2024-04-29T14:21:30.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cde9ce9:::benchmark_data_smithi194_103039_object70412:head# 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cdf6014:::benchmark_data_smithi194_103039_object53863:head# 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cdfa452:::benchmark_data_smithi194_103039_object64552:head# 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce088de:::benchmark_data_smithi194_103039_object81161:head# 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce0e069:::benchmark_data_smithi194_103039_object62854:head# 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce1a89a:::benchmark_data_smithi194_103039_object124724:head# 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce2c6ef:::benchmark_data_smithi194_103039_object136870:head# 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce47fe0:::benchmark_data_smithi194_103039_object68778:head# 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce54e75:::benchmark_data_smithi194_103039_object119829:head# 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce69e1b:::benchmark_data_smithi194_103039_object41751:head# 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce813a9:::benchmark_data_smithi194_103039_object44790:head# 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce83f97:::benchmark_data_smithi194_103039_object56889:head# 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce8f7e8:::benchmark_data_smithi194_103039_object129991:head# 2024-04-29T14:21:30.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce91674:::benchmark_data_smithi194_103039_object96611:head# 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ce930dc:::benchmark_data_smithi194_103039_object42628:head# 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ceae545:::benchmark_data_smithi194_103039_object122094:head# 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cebb991:::benchmark_data_smithi194_103039_object69292:head# 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cec626b:::benchmark_data_smithi194_103039_object66813:head# 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cedd77e:::benchmark_data_smithi194_103039_object155191:head# 2024-04-29T14:21:30.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cedde40:::benchmark_data_smithi194_103039_object72028:head# 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cee77e2:::benchmark_data_smithi194_103039_object94677:head# 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ceecc6d:::benchmark_data_smithi194_103039_object64139:head# 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cef2bb4:::benchmark_data_smithi194_103039_object41777:head# 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cef8bb0:::benchmark_data_smithi194_103039_object109236:head# 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf105f6:::benchmark_data_smithi194_103039_object80354:head# 2024-04-29T14:21:30.240 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf13735:::benchmark_data_smithi194_103039_object64902:head# 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf16817:::benchmark_data_smithi194_103039_object98335:head# 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf1c5ac:::benchmark_data_smithi194_103039_object73835:head# 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf1d87f:::benchmark_data_smithi194_103039_object94857:head# 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf1e5aa:::benchmark_data_smithi194_103039_object48850:head# 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf1f2cd:::benchmark_data_smithi194_103039_object99755:head# 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf20ab7:::benchmark_data_smithi194_103039_object104007:head# 2024-04-29T14:21:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf22cba:::benchmark_data_smithi194_103039_object56538:head# 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf22e9f:::benchmark_data_smithi194_103039_object88127:head# 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf33462:::benchmark_data_smithi194_103039_object83062:head# 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf350d3:::benchmark_data_smithi194_103039_object45645:head# 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf36bda:::benchmark_data_smithi194_103039_object125934:head# 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf50930:::benchmark_data_smithi194_103039_object122781:head# 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf5bb27:::benchmark_data_smithi194_103039_object120467:head# 2024-04-29T14:21:30.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf6f890:::benchmark_data_smithi194_103039_object136756:head# 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf7f5ba:::benchmark_data_smithi194_103039_object99401:head# 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf8cfc8:::benchmark_data_smithi194_103039_object95274:head# 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf8ffbe:::benchmark_data_smithi194_103039_object74515:head# 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cf9b8e3:::benchmark_data_smithi194_103039_object78362:head# 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cfaba94:::benchmark_data_smithi194_103039_object88618:head# 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cfade4d:::benchmark_data_smithi194_103039_object137376:head# 2024-04-29T14:21:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cfb14a8:::benchmark_data_smithi194_103039_object110352:head# 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cfb509c:::benchmark_data_smithi194_103039_object135152:head# 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cfc251f:::benchmark_data_smithi194_103039_object123328:head# 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cfd7b08:::benchmark_data_smithi194_103039_object79523:head# 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cff922e:::benchmark_data_smithi194_103039_object52069:head# 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4cff9d0d:::benchmark_data_smithi194_103039_object73864:head# 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d007291:::benchmark_data_smithi194_103039_object77597:head# 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0072e9:::benchmark_data_smithi194_103039_object113020:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d008880:::benchmark_data_smithi194_103039_object79725:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d018cec:::benchmark_data_smithi194_103039_object101851:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d02ee54:::benchmark_data_smithi194_103039_object94964:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d02fc30:::benchmark_data_smithi194_103039_object75696:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d03ec47:::benchmark_data_smithi194_103039_object138068:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0410a5:::benchmark_data_smithi194_103039_object148139:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d045496:::benchmark_data_smithi194_103039_object65371:head# 2024-04-29T14:21:30.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d049615:::benchmark_data_smithi194_103039_object45742:head# 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d04bfa1:::benchmark_data_smithi194_103039_object130811:head# 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d054d39:::benchmark_data_smithi194_103039_object49913:head# 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0712e0:::benchmark_data_smithi194_103039_object55956:head# 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d07a7e5:::benchmark_data_smithi194_103039_object96871:head# 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d07cc93:::benchmark_data_smithi194_103039_object51415:head# 2024-04-29T14:21:30.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d08333c:::benchmark_data_smithi194_103039_object99398:head# 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d087a32:::benchmark_data_smithi194_103039_object139916:head# 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d089df2:::benchmark_data_smithi194_103039_object58392:head# 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d09280a:::benchmark_data_smithi194_103039_object61471:head# 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0a97b4:::benchmark_data_smithi194_103039_object134937:head# 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0b48f7:::benchmark_data_smithi194_103039_object51290:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0bbd52:::benchmark_data_smithi194_103039_object103296:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0cd435:::benchmark_data_smithi194_103039_object96595:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0d50b8:::benchmark_data_smithi194_103039_object132210:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0d8517:::benchmark_data_smithi194_103039_object113854:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d0f7a99:::benchmark_data_smithi194_103039_object69773:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1078b4:::benchmark_data_smithi194_103039_object65050:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d10aa76:::benchmark_data_smithi194_103039_object78296:head# 2024-04-29T14:21:30.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d10c5ea:::benchmark_data_smithi194_103039_object107800:head# 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d117c2a:::benchmark_data_smithi194_103039_object136190:head# 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d119b09:::benchmark_data_smithi194_103039_object89672:head# 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d11b763:::benchmark_data_smithi194_103039_object106043:head# 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d12689d:::benchmark_data_smithi194_103039_object138371:head# 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d127fbc:::benchmark_data_smithi194_103039_object136682:head# 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d12f05f:::benchmark_data_smithi194_103039_object89729:head# 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d13558a:::benchmark_data_smithi194_103039_object94111:head# 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1507f2:::benchmark_data_smithi194_103039_object42113:head# 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d162494:::benchmark_data_smithi194_103039_object100420:head# 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d16492b:::benchmark_data_smithi194_103039_object151656:head# 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d170ab6:::benchmark_data_smithi194_103039_object134575:head# 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d187bd8:::benchmark_data_smithi194_103039_object150375:head# 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d189bc5:::benchmark_data_smithi194_103039_object49357:head# 2024-04-29T14:21:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d19414a:::benchmark_data_smithi194_103039_object97976:head# 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1a49ff:::benchmark_data_smithi194_103039_object77436:head# 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1b0c24:::benchmark_data_smithi194_103039_object146397:head# 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1c2c66:::benchmark_data_smithi194_103039_object131819:head# 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1c9048:::benchmark_data_smithi194_103039_object130883:head# 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1ce604:::benchmark_data_smithi194_103039_object81999:head# 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1cffb0:::benchmark_data_smithi194_103039_object90240:head# 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1d47f2:::benchmark_data_smithi194_103039_object67047:head# 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1dcf2b:::benchmark_data_smithi194_103039_object143917:head# 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1ddbc9:::benchmark_data_smithi194_103039_object57075:head# 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1e5d33:::benchmark_data_smithi194_103039_object91379:head# 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d1f7b55:::benchmark_data_smithi194_103039_object108464:head# 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d202b78:::benchmark_data_smithi194_103039_object69555:head# 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d2090fb:::benchmark_data_smithi194_103039_object84700:head# 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d220116:::benchmark_data_smithi194_103039_object151919:head# 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d222130:::benchmark_data_smithi194_103039_object47436:head# 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d222382:::benchmark_data_smithi194_103039_object145412:head# 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d23b018:::benchmark_data_smithi194_103039_object92643:head# 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d251eb2:::benchmark_data_smithi194_103039_object100687:head# 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d25328d:::benchmark_data_smithi194_103039_object64734:head# 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d264556:::benchmark_data_smithi194_103039_object98934:head# 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d267afd:::benchmark_data_smithi194_103039_object117492:head# 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d277ad9:::benchmark_data_smithi194_103039_object80544:head# 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d280a34:::benchmark_data_smithi194_103039_object133884:head# 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d285a7c:::benchmark_data_smithi194_103039_object43891:head# 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d2a1ce4:::benchmark_data_smithi194_103039_object60869:head# 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d2a207a:::benchmark_data_smithi194_103039_object52455:head# 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d2a5730:::benchmark_data_smithi194_103039_object54784:head# 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d2aebf7:::benchmark_data_smithi194_103039_object114914:head# 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d2b6ec4:::benchmark_data_smithi194_103039_object112745:head# 2024-04-29T14:21:30.257 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d307c57:::benchmark_data_smithi194_103039_object88944:head# 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d31c29d:::benchmark_data_smithi194_103039_object53455:head# 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d324aba:::benchmark_data_smithi194_103039_object41551:head# 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d32770e:::benchmark_data_smithi194_103039_object128703:head# 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d32d9a8:::benchmark_data_smithi194_103039_object68554:head# 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d332ed5:::benchmark_data_smithi194_103039_object155234:head# 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d344549:::benchmark_data_smithi194_103039_object110915:head# 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3510eb:::benchmark_data_smithi194_103039_object156505:head# 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d361fa7:::benchmark_data_smithi194_103039_object136132:head# 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d368e97:::benchmark_data_smithi194_103039_object155584:head# 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d371b5c:::benchmark_data_smithi194_103039_object140695:head# 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d379330:::benchmark_data_smithi194_103039_object99653:head# 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d37ab55:::benchmark_data_smithi194_103039_object146039:head# 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d387d8b:::benchmark_data_smithi194_103039_object88139:head# 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d393868:::benchmark_data_smithi194_103039_object105703:head# 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3984a8:::benchmark_data_smithi194_103039_object72719:head# 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d39be1b:::benchmark_data_smithi194_103039_object122677:head# 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3acc14:::benchmark_data_smithi194_103039_object149759:head# 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3bb53a:::benchmark_data_smithi194_103039_object52752:head# 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3bf3bf:::benchmark_data_smithi194_103039_object106350:head# 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3cb79e:::benchmark_data_smithi194_103039_object96362:head# 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3cfc68:::benchmark_data_smithi194_103039_object111120:head# 2024-04-29T14:21:30.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3d772a:::benchmark_data_smithi194_103039_object101710:head# 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3dfba0:::benchmark_data_smithi194_103039_object43582:head# 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3e1295:::benchmark_data_smithi194_103039_object102599:head# 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3f0517:::benchmark_data_smithi194_103039_object125922:head# 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d3ff773:::benchmark_data_smithi194_103039_object75830:head# 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d405fdb:::benchmark_data_smithi194_103039_object108981:head# 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d40bbe9:::benchmark_data_smithi194_103039_object86152:head# 2024-04-29T14:21:30.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d414915:::benchmark_data_smithi194_103039_object61054:head# 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d41855d:::benchmark_data_smithi194_103039_object84418:head# 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4192e1:::benchmark_data_smithi194_103039_object87145:head# 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d41b139:::benchmark_data_smithi194_103039_object94527:head# 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d433113:::benchmark_data_smithi194_103039_object112208:head# 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d43663c:::benchmark_data_smithi194_103039_object69098:head# 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d44b5a1:::benchmark_data_smithi194_103039_object138939:head# 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d46078a:::benchmark_data_smithi194_103039_object81974:head# 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d47623b:::benchmark_data_smithi194_103039_object111981:head# 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d47723a:::benchmark_data_smithi194_103039_object132817:head# 2024-04-29T14:21:30.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d486571:::benchmark_data_smithi194_103039_object122789:head# 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d486781:::benchmark_data_smithi194_103039_object112148:head# 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4887a2:::benchmark_data_smithi194_103039_object140337:head# 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d48f94a:::benchmark_data_smithi194_103039_object144768:head# 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4918c5:::benchmark_data_smithi194_103039_object139791:head# 2024-04-29T14:21:30.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4a0798:::benchmark_data_smithi194_103039_object43556:head# 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4a7489:::benchmark_data_smithi194_103039_object129066:head# 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4b25a7:::benchmark_data_smithi194_103039_object120349:head# 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4b3b8b:::benchmark_data_smithi194_103039_object135417:head# 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4b4769:::benchmark_data_smithi194_103039_object73222:head# 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4bb62c:::benchmark_data_smithi194_103039_object91424:head# 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4bdcde:::benchmark_data_smithi194_103039_object126778:head# 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4c2f8c:::benchmark_data_smithi194_103039_object41796:head# 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4d0af6:::benchmark_data_smithi194_103039_object51794:head# 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4decf8:::benchmark_data_smithi194_103039_object86275:head# 2024-04-29T14:21:30.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4e5e1d:::benchmark_data_smithi194_103039_object134335:head# 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4e8163:::benchmark_data_smithi194_103039_object143898:head# 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4ebbaa:::benchmark_data_smithi194_103039_object126252:head# 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4f0d7d:::benchmark_data_smithi194_103039_object81514:head# 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4f2cef:::benchmark_data_smithi194_103039_object83199:head# 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4f89ad:::benchmark_data_smithi194_103039_object99050:head# 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4fd38f:::benchmark_data_smithi194_103039_object133790:head# 2024-04-29T14:21:30.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d4fef8c:::benchmark_data_smithi194_103039_object147421:head# 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d503a92:::benchmark_data_smithi194_103039_object88654:head# 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d507864:::benchmark_data_smithi194_103039_object85304:head# 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d50dddb:::benchmark_data_smithi194_103039_object50637:head# 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d50ec2d:::benchmark_data_smithi194_103039_object63148:head# 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5121e4:::benchmark_data_smithi194_103039_object94701:head# 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d51ffea:::benchmark_data_smithi194_103039_object111831:head# 2024-04-29T14:21:30.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d522cc5:::benchmark_data_smithi194_103039_object60107:head# 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d535275:::benchmark_data_smithi194_103039_object56247:head# 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d55452d:::benchmark_data_smithi194_103039_object146913:head# 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d554622:::benchmark_data_smithi194_103039_object87800:head# 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d55b7a7:::benchmark_data_smithi194_103039_object102663:head# 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5607fb:::benchmark_data_smithi194_103039_object53134:head# 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5618c8:::benchmark_data_smithi194_103039_object79230:head# 2024-04-29T14:21:30.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d565648:::benchmark_data_smithi194_103039_object101790:head# 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d570d34:::benchmark_data_smithi194_103039_object43663:head# 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d57a46b:::benchmark_data_smithi194_103039_object128595:head# 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d57c3a7:::benchmark_data_smithi194_103039_object128973:head# 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d596a83:::benchmark_data_smithi194_103039_object78683:head# 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d59daef:::benchmark_data_smithi194_103039_object113213:head# 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d59fc31:::benchmark_data_smithi194_103039_object103626:head# 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5a9995:::benchmark_data_smithi194_103039_object58820:head# 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5c03ec:::benchmark_data_smithi194_103039_object60696:head# 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5c28c0:::benchmark_data_smithi194_103039_object46943:head# 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5c8c98:::benchmark_data_smithi194_103039_object114096:head# 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5c8fdf:::benchmark_data_smithi194_103039_object108597:head# 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5d6149:::benchmark_data_smithi194_103039_object105718:head# 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5db671:::benchmark_data_smithi194_103039_object137790:head# 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5e52bb:::benchmark_data_smithi194_103039_object130364:head# 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5e7634:::benchmark_data_smithi194_103039_object47337:head# 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5f5806:::benchmark_data_smithi194_103039_object112167:head# 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5f9bbe:::benchmark_data_smithi194_103039_object113446:head# 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d5fc7a3:::benchmark_data_smithi194_103039_object83740:head# 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d602977:::benchmark_data_smithi194_103039_object61071:head# 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d60802b:::benchmark_data_smithi194_103039_object77262:head# 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d60869f:::benchmark_data_smithi194_103039_object128919:head# 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6088e6:::benchmark_data_smithi194_103039_object99112:head# 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6163ed:::benchmark_data_smithi194_103039_object129906:head# 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d618ee4:::benchmark_data_smithi194_103039_object80174:head# 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6239a1:::benchmark_data_smithi194_103039_object134039:head# 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6260d7:::benchmark_data_smithi194_103039_object61508:head# 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d63beb6:::benchmark_data_smithi194_103039_object145221:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d65485b:::benchmark_data_smithi194_103039_object72575:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d656640:::benchmark_data_smithi194_103039_object129171:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d65988a:::benchmark_data_smithi194_103039_object153674:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d65ac40:::benchmark_data_smithi194_103039_object111165:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d66d4c4:::benchmark_data_smithi194_103039_object59514:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d66e082:::benchmark_data_smithi194_103039_object141823:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d674fdd:::benchmark_data_smithi194_103039_object75639:head# 2024-04-29T14:21:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d68e475:::benchmark_data_smithi194_103039_object120572:head# 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6a0550:::benchmark_data_smithi194_103039_object69691:head# 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6a41dd:::benchmark_data_smithi194_103039_object71799:head# 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6a64b9:::benchmark_data_smithi194_103039_object87389:head# 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6b20e7:::benchmark_data_smithi194_103039_object145738:head# 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6b315f:::benchmark_data_smithi194_103039_object117722:head# 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6b8dfe:::benchmark_data_smithi194_103039_object116711:head# 2024-04-29T14:21:30.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6c3bfd:::benchmark_data_smithi194_103039_object42324:head# 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6c5039:::benchmark_data_smithi194_103039_object90335:head# 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6d41a4:::benchmark_data_smithi194_103039_object43141:head# 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6dea76:::benchmark_data_smithi194_103039_object60839:head# 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6e3c80:::benchmark_data_smithi194_103039_object42554:head# 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6e54c5:::benchmark_data_smithi194_103039_object64274:head# 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6e6da7:::benchmark_data_smithi194_103039_object112549:head# 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6ed9b8:::benchmark_data_smithi194_103039_object152759:head# 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6eff46:::benchmark_data_smithi194_103039_object141425:head# 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d6f1108:::benchmark_data_smithi194_103039_object75156:head# 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d70b9f9:::benchmark_data_smithi194_103039_object64007:head# 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d71169a:::benchmark_data_smithi194_103039_object79602:head# 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d711ecf:::benchmark_data_smithi194_103039_object71212:head# 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7387ed:::benchmark_data_smithi194_103039_object147796:head# 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d756fe8:::benchmark_data_smithi194_103039_object135820:head# 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7599b7:::benchmark_data_smithi194_103039_object76251:head# 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d76aa4c:::benchmark_data_smithi194_103039_object82560:head# 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d775fa2:::benchmark_data_smithi194_103039_object65553:head# 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d779898:::benchmark_data_smithi194_103039_object44206:head# 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d77d3c1:::benchmark_data_smithi194_103039_object89516:head# 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7ae357:::benchmark_data_smithi194_103039_object148727:head# 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7b37f0:::benchmark_data_smithi194_103039_object42707:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7b73be:::benchmark_data_smithi194_103039_object70713:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7b8d9a:::benchmark_data_smithi194_103039_object99479:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7ba4c8:::benchmark_data_smithi194_103039_object137366:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7beb79:::benchmark_data_smithi194_103039_object137737:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7d20a5:::benchmark_data_smithi194_103039_object74839:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7d6c33:::benchmark_data_smithi194_103039_object120780:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7d95b1:::benchmark_data_smithi194_103039_object99565:head# 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7ddc9f:::benchmark_data_smithi194_103039_object128567:head# 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7e326e:::benchmark_data_smithi194_103039_object52894:head# 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7f1696:::benchmark_data_smithi194_103039_object46015:head# 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7f3065:::benchmark_data_smithi194_103039_object121332:head# 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7fa7ac:::benchmark_data_smithi194_103039_object143685:head# 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d7fff5b:::benchmark_data_smithi194_103039_object113722:head# 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d805723:::benchmark_data_smithi194_103039_object118569:head# 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d805f08:::benchmark_data_smithi194_103039_object110756:head# 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d80db2c:::benchmark_data_smithi194_103039_object140898:head# 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d80dc1f:::benchmark_data_smithi194_103039_object93854:head# 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d80fa5e:::benchmark_data_smithi194_103039_object142247:head# 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d82895a:::benchmark_data_smithi194_103039_object117210:head# 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d82ba90:::benchmark_data_smithi194_103039_object130323:head# 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d830c1d:::benchmark_data_smithi194_103039_object120412:head# 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d83eeca:::benchmark_data_smithi194_103039_object86569:head# 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8480cf:::benchmark_data_smithi194_103039_object149866:head# 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d84bb76:::benchmark_data_smithi194_103039_object117900:head# 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d84e25d:::benchmark_data_smithi194_103039_object153568:head# 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d858167:::benchmark_data_smithi194_103039_object105918:head# 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d864c16:::benchmark_data_smithi194_103039_object76206:head# 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d88c9a7:::benchmark_data_smithi194_103039_object149185:head# 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8a4238:::benchmark_data_smithi194_103039_object104248:head# 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8a4861:::benchmark_data_smithi194_103039_object112470:head# 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8becd0:::benchmark_data_smithi194_103039_object55742:head# 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8bf16c:::benchmark_data_smithi194_103039_object152160:head# 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8c6e62:::benchmark_data_smithi194_103039_object51780:head# 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8c77e7:::benchmark_data_smithi194_103039_object100462:head# 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8c803d:::benchmark_data_smithi194_103039_object71926:head# 2024-04-29T14:21:30.284 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8dd2c0:::benchmark_data_smithi194_103039_object118521:head# 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8eb64b:::benchmark_data_smithi194_103039_object119496:head# 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d8ebcbe:::benchmark_data_smithi194_103039_object135269:head# 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d901066:::benchmark_data_smithi194_103039_object105260:head# 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d90193c:::benchmark_data_smithi194_103039_object140085:head# 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d90d844:::benchmark_data_smithi194_103039_object104109:head# 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d913458:::benchmark_data_smithi194_103039_object104113:head# 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d91ebb9:::benchmark_data_smithi194_103039_object146460:head# 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d924e2d:::benchmark_data_smithi194_103039_object126771:head# 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d928b59:::benchmark_data_smithi194_103039_object49305:head# 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d92f7aa:::benchmark_data_smithi194_103039_object97984:head# 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d938471:::benchmark_data_smithi194_103039_object78124:head# 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9388d8:::benchmark_data_smithi194_103039_object147436:head# 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d943850:::benchmark_data_smithi194_103039_object117564:head# 2024-04-29T14:21:30.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d947f53:::benchmark_data_smithi194_103039_object60245:head# 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d954ed4:::benchmark_data_smithi194_103039_object119833:head# 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d95765e:::benchmark_data_smithi194_103039_object110964:head# 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d964553:::benchmark_data_smithi194_103039_object125902:head# 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d964bf9:::benchmark_data_smithi194_103039_object140451:head# 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9667b3:::benchmark_data_smithi194_103039_object107742:head# 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d96bac2:::benchmark_data_smithi194_103039_object89091:head# 2024-04-29T14:21:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d972dc5:::benchmark_data_smithi194_103039_object69905:head# 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d972fb3:::benchmark_data_smithi194_103039_object138195:head# 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d976146:::benchmark_data_smithi194_103039_object141423:head# 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d98473e:::benchmark_data_smithi194_103039_object61706:head# 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d98babf:::benchmark_data_smithi194_103039_object52560:head# 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d98dc0c:::benchmark_data_smithi194_103039_object122605:head# 2024-04-29T14:21:30.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d99a670:::benchmark_data_smithi194_103039_object52846:head# 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d99c0ea:::benchmark_data_smithi194_103039_object71525:head# 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9a0668:::benchmark_data_smithi194_103039_object117526:head# 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9b0951:::benchmark_data_smithi194_103039_object60538:head# 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9ccbaa:::benchmark_data_smithi194_103039_object94979:head# 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9e0c84:::benchmark_data_smithi194_103039_object151125:head# 2024-04-29T14:21:30.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9e4b12:::benchmark_data_smithi194_103039_object109828:head# 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9e4bf1:::benchmark_data_smithi194_103039_object114972:head# 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9e511b:::benchmark_data_smithi194_103039_object95448:head# 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9ea9ef:::benchmark_data_smithi194_103039_object77595:head# 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9f4f49:::benchmark_data_smithi194_103039_object107435:head# 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9f5042:::benchmark_data_smithi194_103039_object133245:head# 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4d9f63e8:::benchmark_data_smithi194_103039_object142824:head# 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da01c0a:::benchmark_data_smithi194_103039_object153892:head# 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da12aef:::benchmark_data_smithi194_103039_object113572:head# 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da1d2a9:::benchmark_data_smithi194_103039_object140173:head# 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da2716f:::benchmark_data_smithi194_103039_object139667:head# 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da29666:::benchmark_data_smithi194_103039_object143694:head# 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da33302:::benchmark_data_smithi194_103039_object42242:head# 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da3bc6d:::benchmark_data_smithi194_103039_object63202:head# 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da416cc:::benchmark_data_smithi194_103039_object146101:head# 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da517ec:::benchmark_data_smithi194_103039_object64565:head# 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da518af:::benchmark_data_smithi194_103039_object58144:head# 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da5e1a4:::benchmark_data_smithi194_103039_object69453:head# 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da6058f:::benchmark_data_smithi194_103039_object147727:head# 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da6e899:::benchmark_data_smithi194_103039_object90676:head# 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da7449f:::benchmark_data_smithi194_103039_object111991:head# 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da93087:::benchmark_data_smithi194_103039_object72956:head# 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da96e2c:::benchmark_data_smithi194_103039_object86160:head# 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4da99640:::benchmark_data_smithi194_103039_object54088:head# 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4daae5fc:::benchmark_data_smithi194_103039_object79916:head# 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4daae875:::benchmark_data_smithi194_103039_object105050:head# 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dab5a65:::benchmark_data_smithi194_103039_object117013:head# 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dac5b6a:::benchmark_data_smithi194_103039_object66833:head# 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dace321:::benchmark_data_smithi194_103039_object136196:head# 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dad4a50:::benchmark_data_smithi194_103039_object104806:head# 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dadd15f:::benchmark_data_smithi194_103039_object93843:head# 2024-04-29T14:21:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dae3eee:::benchmark_data_smithi194_103039_object123591:head# 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dae6e73:::benchmark_data_smithi194_103039_object132911:head# 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4daeb8bb:::benchmark_data_smithi194_103039_object44508:head# 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4daed58d:::benchmark_data_smithi194_103039_object113881:head# 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dafa7fa:::benchmark_data_smithi194_103039_object142632:head# 2024-04-29T14:21:30.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db108b4:::benchmark_data_smithi194_103039_object88472:head# 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db18900:::benchmark_data_smithi194_103039_object68630:head# 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db1f5a2:::benchmark_data_smithi194_103039_object155784:head# 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db2275f:::benchmark_data_smithi194_103039_object144655:head# 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db22f67:::benchmark_data_smithi194_103039_object78705:head# 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db23c85:::benchmark_data_smithi194_103039_object66475:head# 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db43803:::benchmark_data_smithi194_103039_object55658:head# 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db4f794:::benchmark_data_smithi194_103039_object50219:head# 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db53b44:::benchmark_data_smithi194_103039_object68432:head# 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db56ffb:::benchmark_data_smithi194_103039_object72605:head# 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db5aa1d:::benchmark_data_smithi194_103039_object87797:head# 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db5c7c3:::benchmark_data_smithi194_103039_object89815:head# 2024-04-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db64f07:::benchmark_data_smithi194_103039_object144729:head# 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db6fc8c:::benchmark_data_smithi194_103039_object134296:head# 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db7823a:::benchmark_data_smithi194_103039_object43981:head# 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db789f8:::benchmark_data_smithi194_103039_object83375:head# 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db7aad9:::benchmark_data_smithi194_103039_object118071:head# 2024-04-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db7cbb2:::benchmark_data_smithi194_103039_object55411:head# 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db8b28e:::benchmark_data_smithi194_103039_object61120:head# 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db8e887:::benchmark_data_smithi194_103039_object90184:head# 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4db9c84c:::benchmark_data_smithi194_103039_object146776:head# 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dba3660:::benchmark_data_smithi194_103039_object59799:head# 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dba65df:::benchmark_data_smithi194_103039_object120141:head# 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbae343:::benchmark_data_smithi194_103039_object73202:head# 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbb7e01:::benchmark_data_smithi194_103039_object119588:head# 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbb8b4b:::benchmark_data_smithi194_103039_object131530:head# 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbb9990:::benchmark_data_smithi194_103039_object64383:head# 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbc2070:::benchmark_data_smithi194_103039_object97945:head# 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbc39b6:::benchmark_data_smithi194_103039_object112797:head# 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbca621:::benchmark_data_smithi194_103039_object42388:head# 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbd9307:::benchmark_data_smithi194_103039_object96461:head# 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbe33bd:::benchmark_data_smithi194_103039_object137399:head# 2024-04-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbe4039:::benchmark_data_smithi194_103039_object91405:head# 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbe5dfc:::benchmark_data_smithi194_103039_object147929:head# 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dbf0d65:::benchmark_data_smithi194_103039_object112569:head# 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc03eef:::benchmark_data_smithi194_103039_object43249:head# 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc24424:::benchmark_data_smithi194_103039_object147947:head# 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc29cc5:::benchmark_data_smithi194_103039_object123791:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc30610:::benchmark_data_smithi194_103039_object136101:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc37b35:::benchmark_data_smithi194_103039_object130960:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc4696a:::benchmark_data_smithi194_103039_object102621:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc47ed0:::benchmark_data_smithi194_103039_object61919:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc48185:::benchmark_data_smithi194_103039_object105965:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc834f4:::benchmark_data_smithi194_103039_object151632:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc973d8:::benchmark_data_smithi194_103039_object83780:head# 2024-04-29T14:21:30.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dc97d17:::benchmark_data_smithi194_103039_object154840:head# 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcaec9f:::benchmark_data_smithi194_103039_object149534:head# 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcb0a78:::benchmark_data_smithi194_103039_object66036:head# 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcb7ec8:::benchmark_data_smithi194_103039_object48183:head# 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcbde92:::benchmark_data_smithi194_103039_object89993:head# 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcc6c2d:::benchmark_data_smithi194_103039_object133414:head# 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcc80e2:::benchmark_data_smithi194_103039_object86506:head# 2024-04-29T14:21:30.304 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dccd87e:::benchmark_data_smithi194_103039_object54494:head# 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcd47b0:::benchmark_data_smithi194_103039_object56970:head# 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcd74be:::benchmark_data_smithi194_103039_object115597:head# 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcda681:::benchmark_data_smithi194_103039_object44537:head# 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dceae6f:::benchmark_data_smithi194_103039_object110049:head# 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcf51fb:::benchmark_data_smithi194_103039_object119783:head# 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dcf986f:::benchmark_data_smithi194_103039_object131721:head# 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd06d21:::benchmark_data_smithi194_103039_object59828:head# 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd0b690:::benchmark_data_smithi194_103039_object115799:head# 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd1357b:::benchmark_data_smithi194_103039_object114607:head# 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd2ba2d:::benchmark_data_smithi194_103039_object146700:head# 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd5149f:::benchmark_data_smithi194_103039_object71617:head# 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd5fd39:::benchmark_data_smithi194_103039_object68115:head# 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd61fd7:::benchmark_data_smithi194_103039_object113082:head# 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd63788:::benchmark_data_smithi194_103039_object45607:head# 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd6cce4:::benchmark_data_smithi194_103039_object156643:head# 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd6e603:::benchmark_data_smithi194_103039_object90024:head# 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd85bac:::benchmark_data_smithi194_103039_object129987:head# 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd97b04:::benchmark_data_smithi194_103039_object97432:head# 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dd9f9bc:::benchmark_data_smithi194_103039_object107054:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dda6921:::benchmark_data_smithi194_103039_object56739:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dda6a61:::benchmark_data_smithi194_103039_object64698:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddac57d:::benchmark_data_smithi194_103039_object50720:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddae22c:::benchmark_data_smithi194_103039_object49650:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddb1392:::benchmark_data_smithi194_103039_object85801:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddb74ec:::benchmark_data_smithi194_103039_object119321:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddc2301:::benchmark_data_smithi194_103039_object41541:head# 2024-04-29T14:21:30.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddc8560:::benchmark_data_smithi194_103039_object142055:head# 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddd17ab:::benchmark_data_smithi194_103039_object135442:head# 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddd33f3:::benchmark_data_smithi194_103039_object117840:head# 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddd57d9:::benchmark_data_smithi194_103039_object76524:head# 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dddb2d7:::benchmark_data_smithi194_103039_object67085:head# 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddf0d4c:::benchmark_data_smithi194_103039_object121595:head# 2024-04-29T14:21:30.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddfcd25:::benchmark_data_smithi194_103039_object86010:head# 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ddfe5a3:::benchmark_data_smithi194_103039_object46470:head# 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de0e4f1:::benchmark_data_smithi194_103039_object79403:head# 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de134d6:::benchmark_data_smithi194_103039_object75891:head# 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de14096:::benchmark_data_smithi194_103039_object153036:head# 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de1b8a3:::benchmark_data_smithi194_103039_object47089:head# 2024-04-29T14:21:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de42ca7:::benchmark_data_smithi194_103039_object132300:head# 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de5243c:::benchmark_data_smithi194_103039_object44197:head# 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de6623b:::benchmark_data_smithi194_103039_object77667:head# 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de684f7:::benchmark_data_smithi194_103039_object45500:head# 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de6ed5e:::benchmark_data_smithi194_103039_object42065:head# 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de97697:::benchmark_data_smithi194_103039_object132110:head# 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4de9c07b:::benchmark_data_smithi194_103039_object86496:head# 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dea7488:::benchmark_data_smithi194_103039_object125629:head# 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4deaa63d:::benchmark_data_smithi194_103039_object64929:head# 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4debca1a:::benchmark_data_smithi194_103039_object142552:head# 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dec558e:::benchmark_data_smithi194_103039_object55079:head# 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4decd1a0:::benchmark_data_smithi194_103039_object115808:head# 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ded6080:::benchmark_data_smithi194_103039_object132192:head# 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dee914a:::benchmark_data_smithi194_103039_object106040:head# 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4deed677:::benchmark_data_smithi194_103039_object110690:head# 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4deee535:::benchmark_data_smithi194_103039_object136192:head# 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4deff9ab:::benchmark_data_smithi194_103039_object71114:head# 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df0ce17:::benchmark_data_smithi194_103039_object127589:head# 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df107be:::benchmark_data_smithi194_103039_object124344:head# 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df22851:::benchmark_data_smithi194_103039_object110273:head# 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df23e06:::benchmark_data_smithi194_103039_object131456:head# 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df34bd1:::benchmark_data_smithi194_103039_object50591:head# 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df3af62:::benchmark_data_smithi194_103039_object119715:head# 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df3de09:::benchmark_data_smithi194_103039_object141979:head# 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df3fd8a:::benchmark_data_smithi194_103039_object52932:head# 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df3fd8c:::benchmark_data_smithi194_103039_object103936:head# 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df43303:::benchmark_data_smithi194_103039_object149052:head# 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df47b70:::benchmark_data_smithi194_103039_object148663:head# 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df4e70b:::benchmark_data_smithi194_103039_object92531:head# 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df506b3:::benchmark_data_smithi194_103039_object109973:head# 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df59032:::benchmark_data_smithi194_103039_object94616:head# 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df5a5ea:::benchmark_data_smithi194_103039_object132978:head# 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df5d320:::benchmark_data_smithi194_103039_object59182:head# 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df61c5a:::benchmark_data_smithi194_103039_object86469:head# 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df643ae:::benchmark_data_smithi194_103039_object50385:head# 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df65fab:::benchmark_data_smithi194_103039_object95292:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df8e294:::benchmark_data_smithi194_103039_object93636:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df931eb:::benchmark_data_smithi194_103039_object42478:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4df98581:::benchmark_data_smithi194_103039_object103513:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dfa1109:::benchmark_data_smithi194_103039_object66387:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dfb24ea:::benchmark_data_smithi194_103039_object138239:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dfc401e:::benchmark_data_smithi194_103039_object109365:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dfd53c0:::benchmark_data_smithi194_103039_object105451:head# 2024-04-29T14:21:30.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dfe0c97:::benchmark_data_smithi194_103039_object152498:head# 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dfe4400:::benchmark_data_smithi194_103039_object56431:head# 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4dfee00b:::benchmark_data_smithi194_103039_object60379:head# 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e002979:::benchmark_data_smithi194_103039_object84091:head# 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e009972:::benchmark_data_smithi194_103039_object107020:head# 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0115de:::benchmark_data_smithi194_103039_object85510:head# 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e016429:::benchmark_data_smithi194_103039_object55245:head# 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e01744b:::benchmark_data_smithi194_103039_object137268:head# 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e01fb5a:::benchmark_data_smithi194_103039_object120683:head# 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e028f6a:::benchmark_data_smithi194_103039_object123427:head# 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e039d22:::benchmark_data_smithi194_103039_object66330:head# 2024-04-29T14:21:30.318 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e042d49:::benchmark_data_smithi194_103039_object112985:head# 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e04b3df:::benchmark_data_smithi194_103039_object54379:head# 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e052da0:::benchmark_data_smithi194_103039_object109281:head# 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e063f67:::benchmark_data_smithi194_103039_object57019:head# 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e068051:::benchmark_data_smithi194_103039_object93303:head# 2024-04-29T14:21:30.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e07ed57:::benchmark_data_smithi194_103039_object137055:head# 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e080c2c:::benchmark_data_smithi194_103039_object132982:head# 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e089f6b:::benchmark_data_smithi194_103039_object130073:head# 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e08e3d3:::benchmark_data_smithi194_103039_object54284:head# 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e08fd5f:::benchmark_data_smithi194_103039_object72188:head# 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e09280a:::benchmark_data_smithi194_103039_object140126:head# 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e096297:::benchmark_data_smithi194_103039_object62961:head# 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e09a315:::benchmark_data_smithi194_103039_object94354:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0a3116:::benchmark_data_smithi194_103039_object81225:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0aed3f:::benchmark_data_smithi194_103039_object146149:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0b132e:::benchmark_data_smithi194_103039_object120716:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0b7810:::benchmark_data_smithi194_103039_object134414:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0b9a81:::benchmark_data_smithi194_103039_object50048:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0d3372:::benchmark_data_smithi194_103039_object113838:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0d53c4:::benchmark_data_smithi194_103039_object62349:head# 2024-04-29T14:21:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0d6841:::benchmark_data_smithi194_103039_object144364:head# 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0ddb14:::benchmark_data_smithi194_103039_object116252:head# 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0e2fe4:::benchmark_data_smithi194_103039_object109564:head# 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0e529f:::benchmark_data_smithi194_103039_object145595:head# 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0e8c52:::benchmark_data_smithi194_103039_object107693:head# 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0eeb94:::benchmark_data_smithi194_103039_object145350:head# 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0f1536:::benchmark_data_smithi194_103039_object134382:head# 2024-04-29T14:21:30.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0f725a:::benchmark_data_smithi194_103039_object94831:head# 2024-04-29T14:21:30.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/765842163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:30.324 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:21:30.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: pgmap v992: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.328 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:21:30.358 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:21:30.358 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:30.377 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:30.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:30.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e0f9d36:::benchmark_data_smithi194_103039_object50418:head# 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e10632b:::benchmark_data_smithi194_103039_object118200:head# 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e10cfa3:::benchmark_data_smithi194_103039_object80419:head# 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e115f54:::benchmark_data_smithi194_103039_object137107:head# 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e12f2ef:::benchmark_data_smithi194_103039_object43853:head# 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e12f997:::benchmark_data_smithi194_103039_object58880:head# 2024-04-29T14:21:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e13b57c:::benchmark_data_smithi194_103039_object82655:head# 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e13c9ff:::benchmark_data_smithi194_103039_object46690:head# 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e13dff2:::benchmark_data_smithi194_103039_object71380:head# 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e13e053:::benchmark_data_smithi194_103039_object86785:head# 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e141c97:::benchmark_data_smithi194_103039_object137379:head# 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e141e83:::benchmark_data_smithi194_103039_object123381:head# 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e146f2c:::benchmark_data_smithi194_103039_object97657:head# 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e15142a:::benchmark_data_smithi194_103039_object119558:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1564dd:::benchmark_data_smithi194_103039_object109102:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e15d664:::benchmark_data_smithi194_103039_object92971:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e17c2b8:::benchmark_data_smithi194_103039_object46327:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1882f7:::benchmark_data_smithi194_103039_object51090:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e18b340:::benchmark_data_smithi194_103039_object141760:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1972d6:::benchmark_data_smithi194_103039_object76439:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e19b7c7:::benchmark_data_smithi194_103039_object145543:head# 2024-04-29T14:21:30.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e19d680:::benchmark_data_smithi194_103039_object64829:head# 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1a4994:::benchmark_data_smithi194_103039_object61255:head# 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1a965e:::benchmark_data_smithi194_103039_object145740:head# 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1ac1dd:::benchmark_data_smithi194_103039_object112268:head# 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1ba51f:::benchmark_data_smithi194_103039_object87876:head# 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1c4e6c:::benchmark_data_smithi194_103039_object120014:head# 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1c8f3c:::benchmark_data_smithi194_103039_object119522:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1dbe3a:::benchmark_data_smithi194_103039_object60940:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1e22ab:::benchmark_data_smithi194_103039_object122652:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1e53ac:::benchmark_data_smithi194_103039_object109337:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1f16a4:::benchmark_data_smithi194_103039_object106674:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e1fa166:::benchmark_data_smithi194_103039_object155641:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e202010:::benchmark_data_smithi194_103039_object65506:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2072f5:::benchmark_data_smithi194_103039_object115636:head# 2024-04-29T14:21:30.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e216e71:::benchmark_data_smithi194_103039_object134810:head# 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2270ea:::benchmark_data_smithi194_103039_object64434:head# 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e228892:::benchmark_data_smithi194_103039_object67347:head# 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e22c50f:::benchmark_data_smithi194_103039_object67261:head# 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e24faf2:::benchmark_data_smithi194_103039_object136404:head# 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e25caa5:::benchmark_data_smithi194_103039_object102025:head# 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e26154e:::benchmark_data_smithi194_103039_object125058:head# 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e264e42:::benchmark_data_smithi194_103039_object129079:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e26cafc:::benchmark_data_smithi194_103039_object95937:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e26cd2c:::benchmark_data_smithi194_103039_object151172:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2829e4:::benchmark_data_smithi194_103039_object110818:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2870de:::benchmark_data_smithi194_103039_object130148:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e295573:::benchmark_data_smithi194_103039_object134292:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e29df5d:::benchmark_data_smithi194_103039_object109196:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e29eb95:::benchmark_data_smithi194_103039_object63159:head# 2024-04-29T14:21:30.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2af6bf:::benchmark_data_smithi194_103039_object89579:head# 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2b1d48:::benchmark_data_smithi194_103039_object87756:head# 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2cc4ba:::benchmark_data_smithi194_103039_object112680:head# 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2d9d53:::benchmark_data_smithi194_103039_object52820:head# 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2e4a91:::benchmark_data_smithi194_103039_object46945:head# 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2e64de:::benchmark_data_smithi194_103039_object53848:head# 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2fb1e5:::benchmark_data_smithi194_103039_object77438:head# 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2fd9eb:::benchmark_data_smithi194_103039_object145052:head# 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e2fe395:::benchmark_data_smithi194_103039_object48520:head# 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e30197c:::benchmark_data_smithi194_103039_object146244:head# 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e304067:::benchmark_data_smithi194_103039_object70997:head# 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e306571:::benchmark_data_smithi194_103039_object103988:head# 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e311e7f:::benchmark_data_smithi194_103039_object132074:head# 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e313f98:::benchmark_data_smithi194_103039_object116939:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e315ca5:::benchmark_data_smithi194_103039_object45344:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3178b4:::benchmark_data_smithi194_103039_object124089:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e31a548:::benchmark_data_smithi194_103039_object64869:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3261a3:::benchmark_data_smithi194_103039_object109050:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e328250:::benchmark_data_smithi194_103039_object99149:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3304bc:::benchmark_data_smithi194_103039_object99119:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e33af22:::benchmark_data_smithi194_103039_object63168:head# 2024-04-29T14:21:30.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3429c8:::benchmark_data_smithi194_103039_object127201:head# 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e34787d:::benchmark_data_smithi194_103039_object42418:head# 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e348e68:::benchmark_data_smithi194_103039_object98770:head# 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e349a67:::benchmark_data_smithi194_103039_object86428:head# 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e351008:::benchmark_data_smithi194_103039_object146959:head# 2024-04-29T14:21:30.477 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3627f2:::benchmark_data_smithi194_103039_object55827:head# 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e38057c:::benchmark_data_smithi194_103039_object130037:head# 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e388215:::benchmark_data_smithi194_103039_object106715:head# 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3890b4:::benchmark_data_smithi194_103039_object137095:head# 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e38dcbc:::benchmark_data_smithi194_103039_object72294:head# 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e39f911:::benchmark_data_smithi194_103039_object146096:head# 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3a22e8:::benchmark_data_smithi194_103039_object68931:head# 2024-04-29T14:21:30.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3af0a9:::benchmark_data_smithi194_103039_object108583:head# 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3cf714:::benchmark_data_smithi194_103039_object106385:head# 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3d6385:::benchmark_data_smithi194_103039_object94901:head# 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3d9ef6:::benchmark_data_smithi194_103039_object60060:head# 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3ded79:::benchmark_data_smithi194_103039_object86179:head# 2024-04-29T14:21:30.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3e0c05:::benchmark_data_smithi194_103039_object105507:head# 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3eef1d:::benchmark_data_smithi194_103039_object91209:head# 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3f0de5:::benchmark_data_smithi194_103039_object88435:head# 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3f2891:::benchmark_data_smithi194_103039_object72217:head# 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3f61b2:::benchmark_data_smithi194_103039_object87924:head# 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3ff9aa:::benchmark_data_smithi194_103039_object142316:head# 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e3ffdfd:::benchmark_data_smithi194_103039_object121253:head# 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e40fa33:::benchmark_data_smithi194_103039_object136026:head# 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e415b65:::benchmark_data_smithi194_103039_object84670:head# 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e41c002:::benchmark_data_smithi194_103039_object63407:head# 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e41ebc2:::benchmark_data_smithi194_103039_object120594:head# 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e450968:::benchmark_data_smithi194_103039_object85622:head# 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e46c56c:::benchmark_data_smithi194_103039_object136487:head# 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e483178:::benchmark_data_smithi194_103039_object114507:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e484261:::benchmark_data_smithi194_103039_object113067:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4946a5:::benchmark_data_smithi194_103039_object114542:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e49ed5b:::benchmark_data_smithi194_103039_object69918:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4ac18d:::benchmark_data_smithi194_103039_object127247:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4b30ca:::benchmark_data_smithi194_103039_object136728:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4bba57:::benchmark_data_smithi194_103039_object74856:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4c41f3:::benchmark_data_smithi194_103039_object53626:head# 2024-04-29T14:21:30.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4c5096:::benchmark_data_smithi194_103039_object147524:head# 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4d3765:::benchmark_data_smithi194_103039_object77990:head# 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4d507e:::benchmark_data_smithi194_103039_object142629:head# 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4e599f:::benchmark_data_smithi194_103039_object71173:head# 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e4f276e:::benchmark_data_smithi194_103039_object153795:head# 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e502ef2:::benchmark_data_smithi194_103039_object87798:head# 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e503a02:::benchmark_data_smithi194_103039_object111637:head# 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e50a2f2:::benchmark_data_smithi194_103039_object93466:head# 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e50dac8:::benchmark_data_smithi194_103039_object124550:head# 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e51ff1b:::benchmark_data_smithi194_103039_object72904:head# 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e524209:::benchmark_data_smithi194_103039_object119867:head# 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5249a6:::benchmark_data_smithi194_103039_object124789:head# 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e527223:::benchmark_data_smithi194_103039_object113630:head# 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e52907b:::benchmark_data_smithi194_103039_object79720:head# 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5294e0:::benchmark_data_smithi194_103039_object49981:head# 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e530f1d:::benchmark_data_smithi194_103039_object128722:head# 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e534a41:::benchmark_data_smithi194_103039_object95230:head# 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5371cb:::benchmark_data_smithi194_103039_object50433:head# 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e540673:::benchmark_data_smithi194_103039_object142661:head# 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5459fe:::benchmark_data_smithi194_103039_object99977:head# 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e556293:::benchmark_data_smithi194_103039_object64692:head# 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e55985b:::benchmark_data_smithi194_103039_object102906:head# 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e570806:::benchmark_data_smithi194_103039_object55991:head# 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e572178:::benchmark_data_smithi194_103039_object105338:head# 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5779be:::benchmark_data_smithi194_103039_object155217:head# 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e59e67c:::benchmark_data_smithi194_103039_object131803:head# 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5a0dcc:::benchmark_data_smithi194_103039_object41534:head# 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5a0f7e:::benchmark_data_smithi194_103039_object88080:head# 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5a2d09:::benchmark_data_smithi194_103039_object80358:head# 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5a4384:::benchmark_data_smithi194_103039_object71958:head# 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5bb745:::benchmark_data_smithi194_103039_object156108:head# 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5c082d:::benchmark_data_smithi194_103039_object117989:head# 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5c97d6:::benchmark_data_smithi194_103039_object104748:head# 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5d8f4b:::benchmark_data_smithi194_103039_object140236:head# 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5ed9a3:::benchmark_data_smithi194_103039_object86680:head# 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e5f2a25:::benchmark_data_smithi194_103039_object69121:head# 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6048dc:::benchmark_data_smithi194_103039_object149134:head# 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e606042:::benchmark_data_smithi194_103039_object154650:head# 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e607249:::benchmark_data_smithi194_103039_object104435:head# 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e616cca:::benchmark_data_smithi194_103039_object82128:head# 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e61aa7f:::benchmark_data_smithi194_103039_object101681:head# 2024-04-29T14:21:30.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e61f2f8:::benchmark_data_smithi194_103039_object153571:head# 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e63ca94:::benchmark_data_smithi194_103039_object124396:head# 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e65a185:::benchmark_data_smithi194_103039_object86508:head# 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6682d1:::benchmark_data_smithi194_103039_object60338:head# 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e66a6af:::benchmark_data_smithi194_103039_object132558:head# 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e66d790:::benchmark_data_smithi194_103039_object97043:head# 2024-04-29T14:21:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6790e1:::benchmark_data_smithi194_103039_object109989:head# 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e67c468:::benchmark_data_smithi194_103039_object77942:head# 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e67d332:::benchmark_data_smithi194_103039_object110232:head# 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6850ad:::benchmark_data_smithi194_103039_object84820:head# 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6888d9:::benchmark_data_smithi194_103039_object54589:head# 2024-04-29T14:21:30.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e68c720:::benchmark_data_smithi194_103039_object41537:head# 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e68e79f:::benchmark_data_smithi194_103039_object138642:head# 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e695397:::benchmark_data_smithi194_103039_object108490:head# 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e69cf57:::benchmark_data_smithi194_103039_object152165:head# 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6aa84b:::benchmark_data_smithi194_103039_object47172:head# 2024-04-29T14:21:30.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6bd7f0:::benchmark_data_smithi194_103039_object121160:head# 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6c1142:::benchmark_data_smithi194_103039_object57658:head# 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6c3470:::benchmark_data_smithi194_103039_object126891:head# 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6c9ecb:::benchmark_data_smithi194_103039_object60876:head# 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6cf505:::benchmark_data_smithi194_103039_object50022:head# 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6d11bc:::benchmark_data_smithi194_103039_object93028:head# 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6d347d:::benchmark_data_smithi194_103039_object86292:head# 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6e7334:::benchmark_data_smithi194_103039_object55229:head# 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6e7727:::benchmark_data_smithi194_103039_object91901:head# 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6e94e7:::benchmark_data_smithi194_103039_object105759:head# 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e6fc17c:::benchmark_data_smithi194_103039_object103762:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e701051:::benchmark_data_smithi194_103039_object48874:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e714134:::benchmark_data_smithi194_103039_object115291:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e71c430:::benchmark_data_smithi194_103039_object131868:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e73dd47:::benchmark_data_smithi194_103039_object54470:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e753935:::benchmark_data_smithi194_103039_object146506:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e758c68:::benchmark_data_smithi194_103039_object43874:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e75d38e:::benchmark_data_smithi194_103039_object114739:head# 2024-04-29T14:21:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e7610c3:::benchmark_data_smithi194_103039_object47966:head# 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e761bfc:::benchmark_data_smithi194_103039_object42902:head# 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e762b3a:::benchmark_data_smithi194_103039_object81127:head# 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e772eda:::benchmark_data_smithi194_103039_object105104:head# 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e785455:::benchmark_data_smithi194_103039_object74618:head# 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e787a61:::benchmark_data_smithi194_103039_object56848:head# 2024-04-29T14:21:30.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e787d73:::benchmark_data_smithi194_103039_object58906:head# 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e797ec9:::benchmark_data_smithi194_103039_object110218:head# 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e7b1e07:::benchmark_data_smithi194_103039_object135798:head# 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e7b64d7:::benchmark_data_smithi194_103039_object108328:head# 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e7ecc48:::benchmark_data_smithi194_103039_object59545:head# 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e7f65da:::benchmark_data_smithi194_103039_object85939:head# 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e7f7f8d:::benchmark_data_smithi194_103039_object110289:head# 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e804cb6:::benchmark_data_smithi194_103039_object102654:head# 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e813409:::benchmark_data_smithi194_103039_object50161:head# 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e813646:::benchmark_data_smithi194_103039_object68837:head# 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e81545b:::benchmark_data_smithi194_103039_object148824:head# 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e81779c:::benchmark_data_smithi194_103039_object67220:head# 2024-04-29T14:21:30.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8243bd:::benchmark_data_smithi194_103039_object142497:head# 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e825611:::benchmark_data_smithi194_103039_object123674:head# 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e835d87:::benchmark_data_smithi194_103039_object70126:head# 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8367d8:::benchmark_data_smithi194_103039_object76423:head# 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8376fd:::benchmark_data_smithi194_103039_object154960:head# 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e853283:::benchmark_data_smithi194_103039_object77973:head# 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e86f2ff:::benchmark_data_smithi194_103039_object120622:head# 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e876068:::benchmark_data_smithi194_103039_object151131:head# 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8771f7:::benchmark_data_smithi194_103039_object52759:head# 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e877b69:::benchmark_data_smithi194_103039_object49958:head# 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e882699:::benchmark_data_smithi194_103039_object129830:head# 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e884194:::benchmark_data_smithi194_103039_object51574:head# 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e889e0e:::benchmark_data_smithi194_103039_object142544:head# 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e898758:::benchmark_data_smithi194_103039_object63192:head# 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e89bc3a:::benchmark_data_smithi194_103039_object123565:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e89cb07:::benchmark_data_smithi194_103039_object138419:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e89ecc0:::benchmark_data_smithi194_103039_object55480:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e89eed4:::benchmark_data_smithi194_103039_object62118:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8a3e4b:::benchmark_data_smithi194_103039_object151784:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8a9338:::benchmark_data_smithi194_103039_object149360:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8c26b8:::benchmark_data_smithi194_103039_object89686:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8cf539:::benchmark_data_smithi194_103039_object41875:head# 2024-04-29T14:21:30.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8d2543:::benchmark_data_smithi194_103039_object138455:head# 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8d4133:::benchmark_data_smithi194_103039_object130620:head# 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e8d77e8:::benchmark_data_smithi194_103039_object75683:head# 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9034ea:::benchmark_data_smithi194_103039_object49150:head# 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e903c5f:::benchmark_data_smithi194_103039_object118132:head# 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9053d2:::benchmark_data_smithi194_103039_object89793:head# 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e90d9a1:::benchmark_data_smithi194_103039_object43658:head# 2024-04-29T14:21:30.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e92d6b2:::benchmark_data_smithi194_103039_object42028:head# 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e959806:::benchmark_data_smithi194_103039_object66263:head# 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e96112e:::benchmark_data_smithi194_103039_object88566:head# 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e961b2b:::benchmark_data_smithi194_103039_object59487:head# 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e96a4a3:::benchmark_data_smithi194_103039_object50266:head# 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e96ab48:::benchmark_data_smithi194_103039_object64310:head# 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e985154:::benchmark_data_smithi194_103039_object122162:head# 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9b694d:::benchmark_data_smithi194_103039_object131275:head# 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9c225f:::benchmark_data_smithi194_103039_object81256:head# 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9c3189:::benchmark_data_smithi194_103039_object50454:head# 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9c3f70:::benchmark_data_smithi194_103039_object146652:head# 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9c647f:::benchmark_data_smithi194_103039_object114340:head# 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9c9e1d:::benchmark_data_smithi194_103039_object48854:head# 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9cdb60:::benchmark_data_smithi194_103039_object107737:head# 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9d08e7:::benchmark_data_smithi194_103039_object81314:head# 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9d58a2:::benchmark_data_smithi194_103039_object110703:head# 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9d5e62:::benchmark_data_smithi194_103039_object73908:head# 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9ea940:::benchmark_data_smithi194_103039_object145281:head# 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4e9f7001:::benchmark_data_smithi194_103039_object62864:head# 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea17370:::benchmark_data_smithi194_103039_object105171:head# 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea17e93:::benchmark_data_smithi194_103039_object153768:head# 2024-04-29T14:21:30.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea2398c:::benchmark_data_smithi194_103039_object142747:head# 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea2ba97:::benchmark_data_smithi194_103039_object52278:head# 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea2d75f:::benchmark_data_smithi194_103039_object92402:head# 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea32fab:::benchmark_data_smithi194_103039_object149007:head# 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea3efb0:::benchmark_data_smithi194_103039_object85985:head# 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea4fbb4:::benchmark_data_smithi194_103039_object81176:head# 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea5a6bc:::benchmark_data_smithi194_103039_object146665:head# 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea5b881:::benchmark_data_smithi194_103039_object115047:head# 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea5cf22:::benchmark_data_smithi194_103039_object68440:head# 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea5d37b:::benchmark_data_smithi194_103039_object117206:head# 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea66367:::benchmark_data_smithi194_103039_object104421:head# 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea6c3b0:::benchmark_data_smithi194_103039_object82884:head# 2024-04-29T14:21:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea6cec2:::benchmark_data_smithi194_103039_object79243:head# 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea6db3a:::benchmark_data_smithi194_103039_object94392:head# 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea708d1:::benchmark_data_smithi194_103039_object46011:head# 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea715cb:::benchmark_data_smithi194_103039_object123357:head# 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea859c5:::benchmark_data_smithi194_103039_object141185:head# 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea947d7:::benchmark_data_smithi194_103039_object153836:head# 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ea99f28:::benchmark_data_smithi194_103039_object55352:head# 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eaabd45:::benchmark_data_smithi194_103039_object134934:head# 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eab1bf2:::benchmark_data_smithi194_103039_object64237:head# 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eab44d3:::benchmark_data_smithi194_103039_object64319:head# 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eac031e:::benchmark_data_smithi194_103039_object50341:head# 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eac7e6b:::benchmark_data_smithi194_103039_object70817:head# 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eac8824:::benchmark_data_smithi194_103039_object112325:head# 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eaca1ba:::benchmark_data_smithi194_103039_object72442:head# 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eae14ce:::benchmark_data_smithi194_103039_object129381:head# 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eaf5c23:::benchmark_data_smithi194_103039_object106547:head# 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eafe2a4:::benchmark_data_smithi194_103039_object129117:head# 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb04333:::benchmark_data_smithi194_103039_object73530:head# 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb06e25:::benchmark_data_smithi194_103039_object146817:head# 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb0a8cc:::benchmark_data_smithi194_103039_object54847:head# 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb13dc7:::benchmark_data_smithi194_103039_object83771:head# 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb14a1b:::benchmark_data_smithi194_103039_object88388:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb1b278:::benchmark_data_smithi194_103039_object76474:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb22aa9:::benchmark_data_smithi194_103039_object63820:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb23324:::benchmark_data_smithi194_103039_object75304:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb2b6bf:::benchmark_data_smithi194_103039_object63200:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb2f6b7:::benchmark_data_smithi194_103039_object126868:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb3b34a:::benchmark_data_smithi194_103039_object114211:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb3d9aa:::benchmark_data_smithi194_103039_object63954:head# 2024-04-29T14:21:30.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb3ef74:::benchmark_data_smithi194_103039_object87754:head# 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb46dde:::benchmark_data_smithi194_103039_object56029:head# 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb4d38d:::benchmark_data_smithi194_103039_object108462:head# 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb56179:::benchmark_data_smithi194_103039_object154687:head# 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb564e5:::benchmark_data_smithi194_103039_object95191:head# 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb59304:::benchmark_data_smithi194_103039_object97918:head# 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb5b6d4:::benchmark_data_smithi194_103039_object144463:head# 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb5bda4:::benchmark_data_smithi194_103039_object72484:head# 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb630ab:::benchmark_data_smithi194_103039_object115046:head# 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb66ea0:::benchmark_data_smithi194_103039_object128110:head# 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb67812:::benchmark_data_smithi194_103039_object61498:head# 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb6efb3:::benchmark_data_smithi194_103039_object72503:head# 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb79de7:::benchmark_data_smithi194_103039_object95609:head# 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb7c730:::benchmark_data_smithi194_103039_object68968:head# 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb7d8b1:::benchmark_data_smithi194_103039_object104637:head# 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb86ab0:::benchmark_data_smithi194_103039_object97121:head# 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb94fe2:::benchmark_data_smithi194_103039_object111280:head# 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eb9de2a:::benchmark_data_smithi194_103039_object136036:head# 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eba475d:::benchmark_data_smithi194_103039_object130759:head# 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebafe66:::benchmark_data_smithi194_103039_object95220:head# 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebb44e8:::benchmark_data_smithi194_103039_object154088:head# 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebc44ac:::benchmark_data_smithi194_103039_object72156:head# 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebc6ae3:::benchmark_data_smithi194_103039_object135623:head# 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebc89f6:::benchmark_data_smithi194_103039_object54630:head# 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebd377d:::benchmark_data_smithi194_103039_object121751:head# 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebe9351:::benchmark_data_smithi194_103039_object77168:head# 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebed62f:::benchmark_data_smithi194_103039_object96555:head# 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebfa414:::benchmark_data_smithi194_103039_object49036:head# 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ebff851:::benchmark_data_smithi194_103039_object130170:head# 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec0796f:::benchmark_data_smithi194_103039_object95790:head# 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec0bdc4:::benchmark_data_smithi194_103039_object97874:head# 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec1c5b9:::benchmark_data_smithi194_103039_object51092:head# 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec29b4a:::benchmark_data_smithi194_103039_object61413:head# 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec410e2:::benchmark_data_smithi194_103039_object153324:head# 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec459b8:::benchmark_data_smithi194_103039_object156517:head# 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec5d6ed:::benchmark_data_smithi194_103039_object102441:head# 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec616d3:::benchmark_data_smithi194_103039_object148106:head# 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec626ff:::benchmark_data_smithi194_103039_object150056:head# 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec6821d:::benchmark_data_smithi194_103039_object76209:head# 2024-04-29T14:21:30.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec6ba3e:::benchmark_data_smithi194_103039_object90295:head# 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec77a80:::benchmark_data_smithi194_103039_object49813:head# 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec7d678:::benchmark_data_smithi194_103039_object120711:head# 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ec89749:::benchmark_data_smithi194_103039_object48188:head# 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eca6f46:::benchmark_data_smithi194_103039_object47840:head# 2024-04-29T14:21:30.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecaca0c:::benchmark_data_smithi194_103039_object87217:head# 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecb0e7a:::benchmark_data_smithi194_103039_object135248:head# 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecc0f76:::benchmark_data_smithi194_103039_object118282:head# 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecc13de:::benchmark_data_smithi194_103039_object81002:head# 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecd2e65:::benchmark_data_smithi194_103039_object47944:head# 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecda2a1:::benchmark_data_smithi194_103039_object73345:head# 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ece4e42:::benchmark_data_smithi194_103039_object106857:head# 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ece6102:::benchmark_data_smithi194_103039_object153820:head# 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ece8892:::benchmark_data_smithi194_103039_object120169:head# 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecee055:::benchmark_data_smithi194_103039_object65419:head# 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecf3b44:::benchmark_data_smithi194_103039_object125187:head# 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecf3eff:::benchmark_data_smithi194_103039_object117217:head# 2024-04-29T14:21:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ecf48a6:::benchmark_data_smithi194_103039_object152386:head# 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed0092a:::benchmark_data_smithi194_103039_object144674:head# 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed06ff7:::benchmark_data_smithi194_103039_object88765:head# 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed122fa:::benchmark_data_smithi194_103039_object116006:head# 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed1b517:::benchmark_data_smithi194_103039_object79570:head# 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed24b53:::benchmark_data_smithi194_103039_object130065:head# 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed2a32a:::benchmark_data_smithi194_103039_object53998:head# 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed2c75c:::benchmark_data_smithi194_103039_object125009:head# 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed2f468:::benchmark_data_smithi194_103039_object74873:head# 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed4b095:::benchmark_data_smithi194_103039_object71602:head# 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed54162:::benchmark_data_smithi194_103039_object53682:head# 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed60734:::benchmark_data_smithi194_103039_object122769:head# 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed713d1:::benchmark_data_smithi194_103039_object149246:head# 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed75b90:::benchmark_data_smithi194_103039_object87760:head# 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed7cd05:::benchmark_data_smithi194_103039_object103913:head# 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed817f3:::benchmark_data_smithi194_103039_object152662:head# 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed880fb:::benchmark_data_smithi194_103039_object71840:head# 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed90c81:::benchmark_data_smithi194_103039_object128428:head# 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ed9c763:::benchmark_data_smithi194_103039_object70381:head# 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eda7b8a:::benchmark_data_smithi194_103039_object85982:head# 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eda8d43:::benchmark_data_smithi194_103039_object149485:head# 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edc4e79:::benchmark_data_smithi194_103039_object144314:head# 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edcccc1:::benchmark_data_smithi194_103039_object146139:head# 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edd2353:::benchmark_data_smithi194_103039_object71540:head# 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eddb905:::benchmark_data_smithi194_103039_object69038:head# 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ede504c:::benchmark_data_smithi194_103039_object109007:head# 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ede8d48:::benchmark_data_smithi194_103039_object129610:head# 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edeffa1:::benchmark_data_smithi194_103039_object49447:head# 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edf11af:::benchmark_data_smithi194_103039_object50252:head# 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edfce03:::benchmark_data_smithi194_103039_object134654:head# 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edfdd05:::benchmark_data_smithi194_103039_object134159:head# 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edfe9f8:::benchmark_data_smithi194_103039_object104212:head# 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4edfec6b:::benchmark_data_smithi194_103039_object64537:head# 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee0056d:::benchmark_data_smithi194_103039_object63474:head# 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee014b6:::benchmark_data_smithi194_103039_object45557:head# 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee068e0:::benchmark_data_smithi194_103039_object97702:head# 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee0c3e7:::benchmark_data_smithi194_103039_object127612:head# 2024-04-29T14:21:30.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee1894f:::benchmark_data_smithi194_103039_object52913:head# 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee3f92e:::benchmark_data_smithi194_103039_object50885:head# 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee421e3:::benchmark_data_smithi194_103039_object150494:head# 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee5f7a0:::benchmark_data_smithi194_103039_object116953:head# 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee6306d:::benchmark_data_smithi194_103039_object58693:head# 2024-04-29T14:21:30.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee65a27:::benchmark_data_smithi194_103039_object144095:head# 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee66b6f:::benchmark_data_smithi194_103039_object116140:head# 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee75eb5:::benchmark_data_smithi194_103039_object155947:head# 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee81e89:::benchmark_data_smithi194_103039_object96405:head# 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee86a4f:::benchmark_data_smithi194_103039_object75179:head# 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee92e46:::benchmark_data_smithi194_103039_object113818:head# 2024-04-29T14:21:30.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee965c2:::benchmark_data_smithi194_103039_object49723:head# 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ee9af6d:::benchmark_data_smithi194_103039_object133158:head# 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eea0e28:::benchmark_data_smithi194_103039_object141335:head# 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eea7b9f:::benchmark_data_smithi194_103039_object70896:head# 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eea9750:::benchmark_data_smithi194_103039_object108041:head# 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eeae5c9:::benchmark_data_smithi194_103039_object44405:head# 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eec7592:::benchmark_data_smithi194_103039_object154701:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eec9e1c:::benchmark_data_smithi194_103039_object117652:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eecaf76:::benchmark_data_smithi194_103039_object109969:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eecfcbb:::benchmark_data_smithi194_103039_object62167:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eedcd3d:::benchmark_data_smithi194_103039_object49937:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eedf912:::benchmark_data_smithi194_103039_object140620:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eee164e:::benchmark_data_smithi194_103039_object144989:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eeee025:::benchmark_data_smithi194_103039_object60602:head# 2024-04-29T14:21:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef140e8:::benchmark_data_smithi194_103039_object67565:head# 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef18fe4:::benchmark_data_smithi194_103039_object73220:head# 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef204ac:::benchmark_data_smithi194_103039_object83464:head# 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef451e9:::benchmark_data_smithi194_103039_object72494:head# 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef499c8:::benchmark_data_smithi194_103039_object105870:head# 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef50345:::benchmark_data_smithi194_103039_object118105:head# 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef57552:::benchmark_data_smithi194_103039_object67382:head# 2024-04-29T14:21:30.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef5ce52:::benchmark_data_smithi194_103039_object136163:head# 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef614c9:::benchmark_data_smithi194_103039_object127041:head# 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef6734a:::benchmark_data_smithi194_103039_object86332:head# 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef73637:::benchmark_data_smithi194_103039_object110630:head# 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef77eda:::benchmark_data_smithi194_103039_object43103:head# 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef7b6a9:::benchmark_data_smithi194_103039_object137585:head# 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef809b4:::benchmark_data_smithi194_103039_object132211:head# 2024-04-29T14:21:30.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef8e092:::benchmark_data_smithi194_103039_object70844:head# 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef907d5:::benchmark_data_smithi194_103039_object120321:head# 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef9387c:::benchmark_data_smithi194_103039_object97749:head# 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef97277:::benchmark_data_smithi194_103039_object45740:head# 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef9c1ef:::benchmark_data_smithi194_103039_object111533:head# 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ef9e779:::benchmark_data_smithi194_103039_object65921:head# 2024-04-29T14:21:30.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efa21ec:::benchmark_data_smithi194_103039_object63357:head# 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efa9c7d:::benchmark_data_smithi194_103039_object68565:head# 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efafe18:::benchmark_data_smithi194_103039_object110744:head# 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efb1910:::benchmark_data_smithi194_103039_object109945:head# 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efb1e3b:::benchmark_data_smithi194_103039_object44013:head# 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efb34ca:::benchmark_data_smithi194_103039_object98056:head# 2024-04-29T14:21:30.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efb6e12:::benchmark_data_smithi194_103039_object106806:head# 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efbacc8:::benchmark_data_smithi194_103039_object99728:head# 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efbb709:::benchmark_data_smithi194_103039_object45080:head# 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efe00f9:::benchmark_data_smithi194_103039_object98618:head# 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efe69e0:::benchmark_data_smithi194_103039_object99285:head# 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4efea75c:::benchmark_data_smithi194_103039_object136210:head# 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4eff1a2d:::benchmark_data_smithi194_103039_object121204:head# 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4effafe8:::benchmark_data_smithi194_103039_object156020:head# 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4effb8e9:::benchmark_data_smithi194_103039_object76205:head# 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f00fa32:::benchmark_data_smithi194_103039_object82597:head# 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f02105e:::benchmark_data_smithi194_103039_object75397:head# 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f021e10:::benchmark_data_smithi194_103039_object72485:head# 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f02e317:::benchmark_data_smithi194_103039_object92737:head# 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f02fe55:::benchmark_data_smithi194_103039_object125043:head# 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f03110d:::benchmark_data_smithi194_103039_object60447:head# 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f03f2fc:::benchmark_data_smithi194_103039_object122908:head# 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0407b5:::benchmark_data_smithi194_103039_object101021:head# 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f044712:::benchmark_data_smithi194_103039_object101720:head# 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f04691e:::benchmark_data_smithi194_103039_object125707:head# 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f04a8ee:::benchmark_data_smithi194_103039_object80607:head# 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f052683:::benchmark_data_smithi194_103039_object80576:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f057248:::benchmark_data_smithi194_103039_object113793:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f05da98:::benchmark_data_smithi194_103039_object104616:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f06bc0a:::benchmark_data_smithi194_103039_object104600:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f06cd5a:::benchmark_data_smithi194_103039_object64231:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f06f4c2:::benchmark_data_smithi194_103039_object149889:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f08e5e6:::benchmark_data_smithi194_103039_object133015:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f09811c:::benchmark_data_smithi194_103039_object104766:head# 2024-04-29T14:21:30.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f098ed6:::benchmark_data_smithi194_103039_object131662:head# 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0aaa96:::benchmark_data_smithi194_103039_object131428:head# 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0b2be4:::benchmark_data_smithi194_103039_object124520:head# 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0bd920:::benchmark_data_smithi194_103039_object109079:head# 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0c02d5:::benchmark_data_smithi194_103039_object149909:head# 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0c9b9f:::benchmark_data_smithi194_103039_object53481:head# 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0d114d:::benchmark_data_smithi194_103039_object56926:head# 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0d17a7:::benchmark_data_smithi194_103039_object96118:head# 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0dbed1:::benchmark_data_smithi194_103039_object44892:head# 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0ddc10:::benchmark_data_smithi194_103039_object153888:head# 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0df613:::benchmark_data_smithi194_103039_object44764:head# 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0e1d1a:::benchmark_data_smithi194_103039_object154258:head# 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f0e9ea2:::benchmark_data_smithi194_103039_object65925:head# 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f10a983:::benchmark_data_smithi194_103039_object154437:head# 2024-04-29T14:21:30.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f10e087:::benchmark_data_smithi194_103039_object130939:head# 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f12cf31:::benchmark_data_smithi194_103039_object128232:head# 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f134644:::benchmark_data_smithi194_103039_object148212:head# 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1374c2:::benchmark_data_smithi194_103039_object100784:head# 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f157ec6:::benchmark_data_smithi194_103039_object54634:head# 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f164947:::benchmark_data_smithi194_103039_object100479:head# 2024-04-29T14:21:30.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1712f7:::benchmark_data_smithi194_103039_object111851:head# 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f183ccd:::benchmark_data_smithi194_103039_object103491:head# 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f19b296:::benchmark_data_smithi194_103039_object122229:head# 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1a0859:::benchmark_data_smithi194_103039_object86684:head# 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1acfcd:::benchmark_data_smithi194_103039_object75988:head# 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1aedbb:::benchmark_data_smithi194_103039_object51948:head# 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1becde:::benchmark_data_smithi194_103039_object62428:head# 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1c232b:::benchmark_data_smithi194_103039_object53614:head# 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1dd54b:::benchmark_data_smithi194_103039_object122302:head# 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1e294c:::benchmark_data_smithi194_103039_object136086:head# 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1f4d00:::benchmark_data_smithi194_103039_object133828:head# 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f1f9a30:::benchmark_data_smithi194_103039_object69491:head# 2024-04-29T14:21:30.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f208902:::benchmark_data_smithi194_103039_object121172:head# 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f210802:::benchmark_data_smithi194_103039_object92372:head# 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f214bb2:::benchmark_data_smithi194_103039_object95425:head# 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f21bd03:::benchmark_data_smithi194_103039_object66731:head# 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f22813a:::benchmark_data_smithi194_103039_object122902:head# 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f237bfd:::benchmark_data_smithi194_103039_object151703:head# 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f24628c:::benchmark_data_smithi194_103039_object106055:head# 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f251496:::benchmark_data_smithi194_103039_object75719:head# 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f256d2c:::benchmark_data_smithi194_103039_object91264:head# 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f25b157:::benchmark_data_smithi194_103039_object88381:head# 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f260ad7:::benchmark_data_smithi194_103039_object135842:head# 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f263d3c:::benchmark_data_smithi194_103039_object84599:head# 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f268e84:::benchmark_data_smithi194_103039_object43197:head# 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f28ccc2:::benchmark_data_smithi194_103039_object137113:head# 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f28d3ad:::benchmark_data_smithi194_103039_object58733:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f29a6dc:::benchmark_data_smithi194_103039_object53616:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f29ac62:::benchmark_data_smithi194_103039_object124292:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2b0ef5:::benchmark_data_smithi194_103039_object119546:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2c41a8:::benchmark_data_smithi194_103039_object146283:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2cf334:::benchmark_data_smithi194_103039_object75271:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2d327e:::benchmark_data_smithi194_103039_object56573:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2d8a99:::benchmark_data_smithi194_103039_object50403:head# 2024-04-29T14:21:30.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2dcf98:::benchmark_data_smithi194_103039_object153988:head# 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2e6c3e:::benchmark_data_smithi194_103039_object90095:head# 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2ec8d0:::benchmark_data_smithi194_103039_object48454:head# 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2f5df5:::benchmark_data_smithi194_103039_object91700:head# 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f2fe808:::benchmark_data_smithi194_103039_object121368:head# 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f310550:::benchmark_data_smithi194_103039_object101552:head# 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3128d7:::benchmark_data_smithi194_103039_object100533:head# 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f327dfa:::benchmark_data_smithi194_103039_object154466:head# 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f331fb1:::benchmark_data_smithi194_103039_object143014:head# 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f338e85:::benchmark_data_smithi194_103039_object103186:head# 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3394c7:::benchmark_data_smithi194_103039_object135633:head# 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f33ac2c:::benchmark_data_smithi194_103039_object129397:head# 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f34cced:::benchmark_data_smithi194_103039_object85398:head# 2024-04-29T14:21:30.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f34ff75:::benchmark_data_smithi194_103039_object112283:head# 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f359f82:::benchmark_data_smithi194_103039_object95422:head# 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f35fabe:::benchmark_data_smithi194_103039_object92172:head# 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3662cd:::benchmark_data_smithi194_103039_object58026:head# 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3688ec:::benchmark_data_smithi194_103039_object90861:head# 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f37bcb4:::benchmark_data_smithi194_103039_object112819:head# 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f37cc6f:::benchmark_data_smithi194_103039_object151634:head# 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f37fa33:::benchmark_data_smithi194_103039_object71740:head# 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f385978:::benchmark_data_smithi194_103039_object60948:head# 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f38df2b:::benchmark_data_smithi194_103039_object108186:head# 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3ac4ba:::benchmark_data_smithi194_103039_object115807:head# 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3ac546:::benchmark_data_smithi194_103039_object138503:head# 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3b4070:::benchmark_data_smithi194_103039_object128887:head# 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3e4ecb:::benchmark_data_smithi194_103039_object103343:head# 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3eed2d:::benchmark_data_smithi194_103039_object129628:head# 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f3f5986:::benchmark_data_smithi194_103039_object71031:head# 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f407385:::benchmark_data_smithi194_103039_object137232:head# 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.552 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f40ee01:::benchmark_data_smithi194_103039_object82174:head# 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f412ffb:::benchmark_data_smithi194_103039_object82175:head# 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f41d724:::benchmark_data_smithi194_103039_object121838:head# 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f420430:::benchmark_data_smithi194_103039_object95795:head# 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f425d7b:::benchmark_data_smithi194_103039_object69318:head# 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f43f5d2:::benchmark_data_smithi194_103039_object100983:head# 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f44379f:::benchmark_data_smithi194_103039_object123573:head# 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f448f00:::benchmark_data_smithi194_103039_object145503:head# 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f449c92:::benchmark_data_smithi194_103039_object130942:head# 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f46127d:::benchmark_data_smithi194_103039_object144772:head# 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f46e7a5:::benchmark_data_smithi194_103039_object62586:head# 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f470d41:::benchmark_data_smithi194_103039_object52649:head# 2024-04-29T14:21:30.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f474194:::benchmark_data_smithi194_103039_object105188:head# 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4765c9:::benchmark_data_smithi194_103039_object98522:head# 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f476faa:::benchmark_data_smithi194_103039_object86797:head# 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f480990:::benchmark_data_smithi194_103039_object136716:head# 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f48c283:::benchmark_data_smithi194_103039_object90084:head# 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f49a4d8:::benchmark_data_smithi194_103039_object126929:head# 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4afe8a:::benchmark_data_smithi194_103039_object72851:head# 2024-04-29T14:21:30.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4c13b7:::benchmark_data_smithi194_103039_object80435:head# 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4c30f7:::benchmark_data_smithi194_103039_object56080:head# 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4d9a8f:::benchmark_data_smithi194_103039_object133430:head# 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4e8949:::benchmark_data_smithi194_103039_object97489:head# 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4ea377:::benchmark_data_smithi194_103039_object63576:head# 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4edddf:::benchmark_data_smithi194_103039_object129218:head# 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4f2c23:::benchmark_data_smithi194_103039_object76298:head# 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4f664f:::benchmark_data_smithi194_103039_object53710:head# 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f4fe7cb:::benchmark_data_smithi194_103039_object144953:head# 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5026a9:::benchmark_data_smithi194_103039_object93749:head# 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f50c472:::benchmark_data_smithi194_103039_object150826:head# 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f513ca7:::benchmark_data_smithi194_103039_object73653:head# 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f528454:::benchmark_data_smithi194_103039_object105811:head# 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f52a56f:::benchmark_data_smithi194_103039_object62639:head# 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f52e090:::benchmark_data_smithi194_103039_object134500:head# 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f53a842:::benchmark_data_smithi194_103039_object116235:head# 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f53e8b2:::benchmark_data_smithi194_103039_object88547:head# 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f540b3d:::benchmark_data_smithi194_103039_object133652:head# 2024-04-29T14:21:30.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5455c8:::benchmark_data_smithi194_103039_object86435:head# 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f55289d:::benchmark_data_smithi194_103039_object146291:head# 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f554195:::benchmark_data_smithi194_103039_object67673:head# 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f56d8c6:::benchmark_data_smithi194_103039_object142734:head# 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f56f293:::benchmark_data_smithi194_103039_object71998:head# 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f57dd50:::benchmark_data_smithi194_103039_object44309:head# 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5883bf:::benchmark_data_smithi194_103039_object69561:head# 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5889fb:::benchmark_data_smithi194_103039_object77494:head# 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f58d340:::benchmark_data_smithi194_103039_object46730:head# 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f594525:::benchmark_data_smithi194_103039_object127888:head# 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f59751d:::benchmark_data_smithi194_103039_object120284:head# 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5a2560:::benchmark_data_smithi194_103039_object123106:head# 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5a70eb:::benchmark_data_smithi194_103039_object58061:head# 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5aae50:::benchmark_data_smithi194_103039_object99581:head# 2024-04-29T14:21:30.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5afacc:::benchmark_data_smithi194_103039_object69045:head# 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5b2ec2:::benchmark_data_smithi194_103039_object136865:head# 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5bdf47:::benchmark_data_smithi194_103039_object92070:head# 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5d0e9f:::benchmark_data_smithi194_103039_object124684:head# 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f5d9913:::benchmark_data_smithi194_103039_object88975:head# 2024-04-29T14:21:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f604110:::benchmark_data_smithi194_103039_object144560:head# 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f609d00:::benchmark_data_smithi194_103039_object103950:head# 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f615128:::benchmark_data_smithi194_103039_object120229:head# 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6156d8:::benchmark_data_smithi194_103039_object80880:head# 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f61d579:::benchmark_data_smithi194_103039_object86830:head# 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f656108:::benchmark_data_smithi194_103039_object129319:head# 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f66920a:::benchmark_data_smithi194_103039_object95986:head# 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f66b8de:::benchmark_data_smithi194_103039_object139410:head# 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f66c8ae:::benchmark_data_smithi194_103039_object55338:head# 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f674614:::benchmark_data_smithi194_103039_object67854:head# 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f676006:::benchmark_data_smithi194_103039_object100861:head# 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f677482:::benchmark_data_smithi194_103039_object41695:head# 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f67c4b9:::benchmark_data_smithi194_103039_object67664:head# 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f683e35:::benchmark_data_smithi194_103039_object56511:head# 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f68e67a:::benchmark_data_smithi194_103039_object69073:head# 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6b0c05:::benchmark_data_smithi194_103039_object56417:head# 2024-04-29T14:21:30.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6b167b:::benchmark_data_smithi194_103039_object81291:head# 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6b3989:::benchmark_data_smithi194_103039_object50007:head# 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6c854b:::benchmark_data_smithi194_103039_object99645:head# 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6cb893:::benchmark_data_smithi194_103039_object69730:head# 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6d82a2:::benchmark_data_smithi194_103039_object109330:head# 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6e5dc7:::benchmark_data_smithi194_103039_object135654:head# 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6e9221:::benchmark_data_smithi194_103039_object104713:head# 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6ef15d:::benchmark_data_smithi194_103039_object113069:head# 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6f06e4:::benchmark_data_smithi194_103039_object90276:head# 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6f76b0:::benchmark_data_smithi194_103039_object113791:head# 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f6f89d1:::benchmark_data_smithi194_103039_object149056:head# 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7013dc:::benchmark_data_smithi194_103039_object92824:head# 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f70879f:::benchmark_data_smithi194_103039_object104420:head# 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f709502:::benchmark_data_smithi194_103039_object116641:head# 2024-04-29T14:21:30.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f729393:::benchmark_data_smithi194_103039_object139130:head# 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7341e6:::benchmark_data_smithi194_103039_object48833:head# 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7490d4:::benchmark_data_smithi194_103039_object88889:head# 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f74fc6d:::benchmark_data_smithi194_103039_object66873:head# 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f75a800:::benchmark_data_smithi194_103039_object124659:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f768275:::benchmark_data_smithi194_103039_object113866:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f769b6d:::benchmark_data_smithi194_103039_object137888:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f78c688:::benchmark_data_smithi194_103039_object142532:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f78d43d:::benchmark_data_smithi194_103039_object95751:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f791db4:::benchmark_data_smithi194_103039_object77277:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7ab338:::benchmark_data_smithi194_103039_object90614:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7b53df:::benchmark_data_smithi194_103039_object101323:head# 2024-04-29T14:21:30.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7c8c01:::benchmark_data_smithi194_103039_object154561:head# 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7d08a1:::benchmark_data_smithi194_103039_object127858:head# 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7f37ef:::benchmark_data_smithi194_103039_object80801:head# 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7f4d9b:::benchmark_data_smithi194_103039_object42553:head# 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7f69c0:::benchmark_data_smithi194_103039_object84336:head# 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f7f6b9c:::benchmark_data_smithi194_103039_object96458:head# 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f80ca6a:::benchmark_data_smithi194_103039_object49688:head# 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f816b13:::benchmark_data_smithi194_103039_object149048:head# 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f81c52a:::benchmark_data_smithi194_103039_object103259:head# 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f83be02:::benchmark_data_smithi194_103039_object119922:head# 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f83d190:::benchmark_data_smithi194_103039_object123027:head# 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f858c50:::benchmark_data_smithi194_103039_object147431:head# 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f85927b:::benchmark_data_smithi194_103039_object104535:head# 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f85fad7:::benchmark_data_smithi194_103039_object46644:head# 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f86187a:::benchmark_data_smithi194_103039_object128173:head# 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8687d8:::benchmark_data_smithi194_103039_object153402:head# 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f86aa7d:::benchmark_data_smithi194_103039_object44563:head# 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f86d548:::benchmark_data_smithi194_103039_object94892:head# 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f86ecf2:::benchmark_data_smithi194_103039_object93611:head# 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f875076:::benchmark_data_smithi194_103039_object96966:head# 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f87c9b2:::benchmark_data_smithi194_103039_object119422:head# 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8807e2:::benchmark_data_smithi194_103039_object150586:head# 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f88b5c1:::benchmark_data_smithi194_103039_object92726:head# 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8904bf:::benchmark_data_smithi194_103039_object91940:head# 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f89eb72:::benchmark_data_smithi194_103039_object52959:head# 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8a803e:::benchmark_data_smithi194_103039_object74376:head# 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8bca4f:::benchmark_data_smithi194_103039_object90665:head# 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8c0b6c:::benchmark_data_smithi194_103039_object65074:head# 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8d6d2f:::benchmark_data_smithi194_103039_object113690:head# 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8d7242:::benchmark_data_smithi194_103039_object113220:head# 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8ef6ad:::benchmark_data_smithi194_103039_object136662:head# 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8f116b:::benchmark_data_smithi194_103039_object90580:head# 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f8f2de4:::benchmark_data_smithi194_103039_object60132:head# 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9063e2:::benchmark_data_smithi194_103039_object70867:head# 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f907936:::benchmark_data_smithi194_103039_object107022:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9310f4:::benchmark_data_smithi194_103039_object97045:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f938125:::benchmark_data_smithi194_103039_object131838:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9454bb:::benchmark_data_smithi194_103039_object78475:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f947333:::benchmark_data_smithi194_103039_object134969:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f95b3df:::benchmark_data_smithi194_103039_object136949:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f95edf4:::benchmark_data_smithi194_103039_object152387:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f97a7b9:::benchmark_data_smithi194_103039_object155801:head# 2024-04-29T14:21:30.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f98d929:::benchmark_data_smithi194_103039_object133358:head# 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f993894:::benchmark_data_smithi194_103039_object110494:head# 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f995bb6:::benchmark_data_smithi194_103039_object97319:head# 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9a1549:::benchmark_data_smithi194_103039_object81444:head# 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9a5d47:::benchmark_data_smithi194_103039_object99935:head# 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9ab68d:::benchmark_data_smithi194_103039_object115478:head# 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9b75db:::benchmark_data_smithi194_103039_object104166:head# 2024-04-29T14:21:30.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9bcbbe:::benchmark_data_smithi194_103039_object116433:head# 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9c1b4b:::benchmark_data_smithi194_103039_object139956:head# 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9c29d5:::benchmark_data_smithi194_103039_object132099:head# 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9cc639:::benchmark_data_smithi194_103039_object136867:head# 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9d679d:::benchmark_data_smithi194_103039_object128074:head# 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9ec3ed:::benchmark_data_smithi194_103039_object56694:head# 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4f9ee6f0:::benchmark_data_smithi194_103039_object61447:head# 2024-04-29T14:21:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa00e0d:::benchmark_data_smithi194_103039_object92292:head# 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa03618:::benchmark_data_smithi194_103039_object61551:head# 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa07e7b:::benchmark_data_smithi194_103039_object63263:head# 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa39ed2:::benchmark_data_smithi194_103039_object110094:head# 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa45af8:::benchmark_data_smithi194_103039_object144280:head# 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa501a5:::benchmark_data_smithi194_103039_object124002:head# 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa553c2:::benchmark_data_smithi194_103039_object60775:head# 2024-04-29T14:21:30.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa67a95:::benchmark_data_smithi194_103039_object104483:head# 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa690fe:::benchmark_data_smithi194_103039_object68369:head# 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa6b129:::benchmark_data_smithi194_103039_object71778:head# 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa77083:::benchmark_data_smithi194_103039_object52264:head# 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fa98d43:::benchmark_data_smithi194_103039_object152013:head# 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4faa6ddc:::benchmark_data_smithi194_103039_object59318:head# 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fac1381:::benchmark_data_smithi194_103039_object114028:head# 2024-04-29T14:21:30.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fac1bbb:::benchmark_data_smithi194_103039_object156323:head# 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fade5d9:::benchmark_data_smithi194_103039_object115826:head# 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fae0f1c:::benchmark_data_smithi194_103039_object124525:head# 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fae2b85:::benchmark_data_smithi194_103039_object78534:head# 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fae35a4:::benchmark_data_smithi194_103039_object102320:head# 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fae8f41:::benchmark_data_smithi194_103039_object87948:head# 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4faeb464:::benchmark_data_smithi194_103039_object151150:head# 2024-04-29T14:21:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4faf0f17:::benchmark_data_smithi194_103039_object108587:head# 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4faf44df:::benchmark_data_smithi194_103039_object133805:head# 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fafbd44:::benchmark_data_smithi194_103039_object110718:head# 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb1ab58:::benchmark_data_smithi194_103039_object46218:head# 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb1e29b:::benchmark_data_smithi194_103039_object47572:head# 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb1f25b:::benchmark_data_smithi194_103039_object136042:head# 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb2d875:::benchmark_data_smithi194_103039_object69014:head# 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb2f5e1:::benchmark_data_smithi194_103039_object149508:head# 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb32e64:::benchmark_data_smithi194_103039_object149994:head# 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb366b3:::benchmark_data_smithi194_103039_object45617:head# 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb418c6:::benchmark_data_smithi194_103039_object61262:head# 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb4a2b3:::benchmark_data_smithi194_103039_object51968:head# 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb52d4b:::benchmark_data_smithi194_103039_object44165:head# 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb5490b:::benchmark_data_smithi194_103039_object122399:head# 2024-04-29T14:21:30.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb5af1d:::benchmark_data_smithi194_103039_object67992:head# 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb779d5:::benchmark_data_smithi194_103039_object142671:head# 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb7b08d:::benchmark_data_smithi194_103039_object153869:head# 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb82e96:::benchmark_data_smithi194_103039_object149918:head# 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb8619e:::benchmark_data_smithi194_103039_object71039:head# 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb862f9:::benchmark_data_smithi194_103039_object134171:head# 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb89f31:::benchmark_data_smithi194_103039_object106406:head# 2024-04-29T14:21:30.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb8cdaf:::benchmark_data_smithi194_103039_object50173:head# 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb8fc2c:::benchmark_data_smithi194_103039_object79539:head# 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fb9aeb9:::benchmark_data_smithi194_103039_object134320:head# 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fba3aa5:::benchmark_data_smithi194_103039_object142479:head# 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fba6d4e:::benchmark_data_smithi194_103039_object55922:head# 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fba8298:::benchmark_data_smithi194_103039_object77319:head# 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbb7cb7:::benchmark_data_smithi194_103039_object41946:head# 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbbee2d:::benchmark_data_smithi194_103039_object53280:head# 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbc0631:::benchmark_data_smithi194_103039_object123466:head# 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbc25e5:::benchmark_data_smithi194_103039_object48129:head# 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbca021:::benchmark_data_smithi194_103039_object53677:head# 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbdd7fb:::benchmark_data_smithi194_103039_object148738:head# 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbdeb9e:::benchmark_data_smithi194_103039_object135213:head# 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbdfcbd:::benchmark_data_smithi194_103039_object148282:head# 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbe0e14:::benchmark_data_smithi194_103039_object73634:head# 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbe3043:::benchmark_data_smithi194_103039_object141450:head# 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbfb5a0:::benchmark_data_smithi194_103039_object116129:head# 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fbfc760:::benchmark_data_smithi194_103039_object100597:head# 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc0c0f4:::benchmark_data_smithi194_103039_object105096:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc166ed:::benchmark_data_smithi194_103039_object62171:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc1ba2a:::benchmark_data_smithi194_103039_object83726:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc1f8bd:::benchmark_data_smithi194_103039_object147959:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc206fc:::benchmark_data_smithi194_103039_object103531:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc278db:::benchmark_data_smithi194_103039_object122613:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc41e52:::benchmark_data_smithi194_103039_object147457:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc52d89:::benchmark_data_smithi194_103039_object62404:head# 2024-04-29T14:21:30.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc60a42:::benchmark_data_smithi194_103039_object69091:head# 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc62c5c:::benchmark_data_smithi194_103039_object74196:head# 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc6500d:::benchmark_data_smithi194_103039_object81382:head# 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc66d97:::benchmark_data_smithi194_103039_object82280:head# 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc70267:::benchmark_data_smithi194_103039_object120500:head# 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc996e4:::benchmark_data_smithi194_103039_object55371:head# 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fc99c07:::benchmark_data_smithi194_103039_object46801:head# 2024-04-29T14:21:30.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fca5e2e:::benchmark_data_smithi194_103039_object69141:head# 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fcab795:::benchmark_data_smithi194_103039_object132417:head# 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fcabe53:::benchmark_data_smithi194_103039_object61034:head# 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fcade97:::benchmark_data_smithi194_103039_object128014:head# 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fcbf7bc:::benchmark_data_smithi194_103039_object120033:head# 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fcc627a:::benchmark_data_smithi194_103039_object49651:head# 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fccf465:::benchmark_data_smithi194_103039_object69614:head# 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fcd1907:::benchmark_data_smithi194_103039_object91816:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fcf3bd3:::benchmark_data_smithi194_103039_object82427:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd0935b:::benchmark_data_smithi194_103039_object111736:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd19636:::benchmark_data_smithi194_103039_object77664:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd1cc97:::benchmark_data_smithi194_103039_object109033:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd292f2:::benchmark_data_smithi194_103039_object152094:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd3abcb:::benchmark_data_smithi194_103039_object79896:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd3f849:::benchmark_data_smithi194_103039_object69623:head# 2024-04-29T14:21:30.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd4aa03:::benchmark_data_smithi194_103039_object102033:head# 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd62317:::benchmark_data_smithi194_103039_object102523:head# 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd6bdbc:::benchmark_data_smithi194_103039_object96032:head# 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd6ca25:::benchmark_data_smithi194_103039_object146084:head# 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd755df:::benchmark_data_smithi194_103039_object124491:head# 2024-04-29T14:21:30.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd7eb4d:::benchmark_data_smithi194_103039_object69806:head# 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd80388:::benchmark_data_smithi194_103039_object77980:head# 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd81ebf:::benchmark_data_smithi194_103039_object66081:head# 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd8c70a:::benchmark_data_smithi194_103039_object76911:head# 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd8fecf:::benchmark_data_smithi194_103039_object127822:head# 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fd9648b:::benchmark_data_smithi194_103039_object148686:head# 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fda64eb:::benchmark_data_smithi194_103039_object111419:head# 2024-04-29T14:21:30.592 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdaa826:::benchmark_data_smithi194_103039_object154716:head# 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdaaed6:::benchmark_data_smithi194_103039_object97153:head# 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdad276:::benchmark_data_smithi194_103039_object51935:head# 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdb3a4a:::benchmark_data_smithi194_103039_object148552:head# 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdb4526:::benchmark_data_smithi194_103039_object107839:head# 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdc0dc7:::benchmark_data_smithi194_103039_object127503:head# 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdcbc55:::benchmark_data_smithi194_103039_object94785:head# 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdce387:::benchmark_data_smithi194_103039_object66523:head# 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdd101a:::benchmark_data_smithi194_103039_object57004:head# 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdd912b:::benchmark_data_smithi194_103039_object54191:head# 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fddc342:::benchmark_data_smithi194_103039_object45129:head# 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fde1c9f:::benchmark_data_smithi194_103039_object145373:head# 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fde4dd3:::benchmark_data_smithi194_103039_object62104:head# 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fde5ef1:::benchmark_data_smithi194_103039_object42338:head# 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fde9982:::benchmark_data_smithi194_103039_object81847:head# 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdf1934:::benchmark_data_smithi194_103039_object133702:head# 2024-04-29T14:21:30.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fdfe6e5:::benchmark_data_smithi194_103039_object73360:head# 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe0eb32:::benchmark_data_smithi194_103039_object140076:head# 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe14a21:::benchmark_data_smithi194_103039_object109937:head# 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe37c61:::benchmark_data_smithi194_103039_object53297:head# 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe3d2e8:::benchmark_data_smithi194_103039_object127841:head# 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe40cf9:::benchmark_data_smithi194_103039_object115587:head# 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe4190e:::benchmark_data_smithi194_103039_object43196:head# 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe447ca:::benchmark_data_smithi194_103039_object109303:head# 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe4db9a:::benchmark_data_smithi194_103039_object112137:head# 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe5fa7e:::benchmark_data_smithi194_103039_object131975:head# 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe6396b:::benchmark_data_smithi194_103039_object122618:head# 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe67b49:::benchmark_data_smithi194_103039_object107951:head# 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe71c03:::benchmark_data_smithi194_103039_object73153:head# 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe7c7b3:::benchmark_data_smithi194_103039_object122584:head# 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe7ed38:::benchmark_data_smithi194_103039_object78669:head# 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe86669:::benchmark_data_smithi194_103039_object44739:head# 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe869d0:::benchmark_data_smithi194_103039_object72076:head# 2024-04-29T14:21:30.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe8958c:::benchmark_data_smithi194_103039_object68485:head# 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe91209:::benchmark_data_smithi194_103039_object128706:head# 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe924b2:::benchmark_data_smithi194_103039_object44886:head# 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe93333:::benchmark_data_smithi194_103039_object154373:head# 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fe9e624:::benchmark_data_smithi194_103039_object153185:head# 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fea223c:::benchmark_data_smithi194_103039_object151004:head# 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fea74ef:::benchmark_data_smithi194_103039_object110776:head# 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fea927b:::benchmark_data_smithi194_103039_object64016:head# 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4feab7b4:::benchmark_data_smithi194_103039_object142803:head# 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4feadba4:::benchmark_data_smithi194_103039_object139527:head# 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4feb1bd3:::benchmark_data_smithi194_103039_object41903:head# 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4feb69ef:::benchmark_data_smithi194_103039_object81103:head# 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4feba904:::benchmark_data_smithi194_103039_object71549:head# 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4febfcbb:::benchmark_data_smithi194_103039_object136502:head# 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fec4f36:::benchmark_data_smithi194_103039_object105269:head# 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fecb7cf:::benchmark_data_smithi194_103039_object109864:head# 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fed2486:::benchmark_data_smithi194_103039_object125608:head# 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fee6c13:::benchmark_data_smithi194_103039_object105456:head# 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fee9177:::benchmark_data_smithi194_103039_object110043:head# 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4feef06f:::benchmark_data_smithi194_103039_object121971:head# 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fef344f:::benchmark_data_smithi194_103039_object131288:head# 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4fef5062:::benchmark_data_smithi194_103039_object118607:head# 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff0432e:::benchmark_data_smithi194_103039_object58435:head# 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff0ae55:::benchmark_data_smithi194_103039_object151254:head# 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff0dacf:::benchmark_data_smithi194_103039_object134269:head# 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff13694:::benchmark_data_smithi194_103039_object84476:head# 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff14d6c:::benchmark_data_smithi194_103039_object121604:head# 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff2e0c0:::benchmark_data_smithi194_103039_object120767:head# 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff3f988:::benchmark_data_smithi194_103039_object72474:head# 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff42495:::benchmark_data_smithi194_103039_object123792:head# 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff4d2e5:::benchmark_data_smithi194_103039_object147567:head# 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff52bec:::benchmark_data_smithi194_103039_object127780:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff5aac1:::benchmark_data_smithi194_103039_object118891:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff5ba67:::benchmark_data_smithi194_103039_object122791:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff63006:::benchmark_data_smithi194_103039_object53250:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff6ab19:::benchmark_data_smithi194_103039_object143740:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff77f44:::benchmark_data_smithi194_103039_object136118:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff7d236:::benchmark_data_smithi194_103039_object136209:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff92fdb:::benchmark_data_smithi194_103039_object143497:head# 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ff9525d:::benchmark_data_smithi194_103039_object51159:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffa3aaf:::benchmark_data_smithi194_103039_object126475:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffaf9c9:::benchmark_data_smithi194_103039_object73060:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffb90e4:::benchmark_data_smithi194_103039_object153483:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffbcd84:::benchmark_data_smithi194_103039_object118051:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffc0a90:::benchmark_data_smithi194_103039_object154133:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffd0920:::benchmark_data_smithi194_103039_object146447:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffdb618:::benchmark_data_smithi194_103039_object96594:head# 2024-04-29T14:21:30.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffdd806:::benchmark_data_smithi194_103039_object116604:head# 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffddf9e:::benchmark_data_smithi194_103039_object140436:head# 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:4ffefd35:::benchmark_data_smithi194_103039_object76222:head# 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:write_pg epoch 369 info 4.2( v 369'12412 (331'2400,369'12412] local-lis/les=323/324 n=7198 ec=323/323 lis/c=323/323 les/c/f=324/324/0 sis=323) 2024-04-29T14:21:30.607 INFO:teuthology.orchestra.run.smithi121.stdout:Import successful 2024-04-29T14:21:30.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: pgmap v992: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: pgmap v992: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:30.633 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:30.693 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:30.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:30.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:30.829 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:30.882 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:30.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:30.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/exp.4.2.5 2024-04-29T14:21:30.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:31.035 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:31.056 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:31.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:31.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:31.186 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:31.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:31.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:31.338 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:31.437 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:31.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:31.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:31.569 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:31.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:31.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:31.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:31.720 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:31.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:31.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:31.871 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:31.923 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:31.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:32.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:32.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:32.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:32.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:32.206 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:32.211 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:21:32.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:32.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:32.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:32.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:21:32.363 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:32.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:32 smithi121 ceph-mon[44533]: pgmap v993: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:32.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3652993336' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:32.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:32.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:32.502 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:21:32.502 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:21:32.502 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:21:32.502 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:239815 2.c 2:3e12e13d:::benchmark_data_smithi194_79000_object75554:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 565.91224438400002, 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077752630000000003, 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 239815 2024-04-29T14:21:32.503 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595974+0000", 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595977+0000", 2024-04-29T14:21:32.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2730000000000001e-06 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595984+0000", 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0559999999999999e-06 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.595996+0000", 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2215000000000001e-05 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596005+0000", 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8249999999999994e-06 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596065+0000", 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0183000000000003e-05 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596172+0000", 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106702 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:21:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596280+0000", 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108251 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.605356+0000", 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090762619999999999 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673606+0000", 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068249290000000004 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673668+0000", 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2724000000000005e-05 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673727+0000", 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8848999999999998e-05 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:239816 2.c 2:3d6c3553:::benchmark_data_smithi194_79000_object75555:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:21:32.508 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 565.91219557399995, 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077792867000000002, 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 239816 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596023+0000", 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596024+0000", 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9400000000000004e-07 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596028+0000", 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6440000000000002e-06 2024-04-29T14:21:32.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596034+0000", 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.596041+0000", 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8210000000000004e-06 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601153+0000", 2024-04-29T14:21:32.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051123899999999996 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601279+0000", 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012538299999999999 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.601396+0000", 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117223 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.605374+0000", 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039785719999999997 2024-04-29T14:21:32.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673769+0000", 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.068395101999999999 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673795+0000", 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.584e-05 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:12:06.673816+0000", 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0874e-05 2024-04-29T14:21:32.513 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323258 2.b 2:d0b65c6b:::benchmark_data_smithi194_79000_object158997:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.43500321800002, 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.231651628, 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.514 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323258 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073216+0000", 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4300000000000002e-07 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073219+0000", 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.066e-06 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073224+0000", 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8040000000000001e-06 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073230+0000", 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8089999999999998e-06 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073281+0000", 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0677000000000003e-05 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073356+0000", 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5309999999999996e-05 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.073428+0000", 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1680000000000005e-05 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.077404+0000", 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039764600000000002 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304804+0000", 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.227399348 2024-04-29T14:21:32.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304837+0000", 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3788999999999998e-05 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304867+0000", 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9941999999999999e-05 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.519 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323282 2.b 2:de88de25:::benchmark_data_smithi194_79000_object159021:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.091895+0000", 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.41632416599998, 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21308433399999999, 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323282 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.520 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091895+0000", 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091895+0000", 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091897+0000", 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.84e-06 2024-04-29T14:21:32.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091902+0000", 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5790000000000003e-06 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091913+0000", 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.118e-05 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091921+0000", 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8879999999999997e-06 2024-04-29T14:21:32.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.091990+0000", 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8392999999999994e-05 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.092076+0000", 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6626999999999996e-05 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.092163+0000", 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6722000000000001e-05 2024-04-29T14:21:32.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.097159+0000", 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049962909999999999 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304912+0000", 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.207753146 2024-04-29T14:21:32.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304944+0000", 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1189000000000002e-05 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.304979+0000", 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5478999999999997e-05 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323296 2.b 2:d44dd007:::benchmark_data_smithi194_79000_object159035:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.099450+0000", 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.40876862300001, 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.205642045, 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323296 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099450+0000", 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099450+0000", 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099453+0000", 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3319999999999999e-06 2024-04-29T14:21:32.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099458+0000", 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7629999999999997e-06 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099471+0000", 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2955999999999999e-05 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099478+0000", 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1239999999999999e-06 2024-04-29T14:21:32.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099532+0000", 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3538999999999997e-05 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099613+0000", 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1308000000000001e-05 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.099681+0000", 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7446000000000004e-05 2024-04-29T14:21:32.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.104798+0000", 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051173939999999999 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.305030+0000", 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.200232206 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.305063+0000", 2024-04-29T14:21:32.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2951999999999999e-05 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.305092+0000", 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9025000000000001e-05 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323321 2.b 2:d0365ce8:::benchmark_data_smithi194_79000_object159060:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.124090+0000", 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.38412903099999, 2024-04-29T14:21:32.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18673873399999999, 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323321 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124090+0000", 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124090+0000", 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124091+0000", 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.23e-07 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124094+0000", 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5250000000000001e-06 2024-04-29T14:21:32.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124104+0000", 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0319000000000001e-05 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124112+0000", 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5310000000000004e-06 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.129158+0000", 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050463030000000002 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.129267+0000", 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108687 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.129374+0000", 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106903 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.138413+0000", 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090388220000000002 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310731+0000", 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17231806199999999 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310783+0000", 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2346999999999999e-05 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310829+0000", 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5512000000000001e-05 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323322 2.b 2:dc463b5d:::benchmark_data_smithi194_79000_object159061:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.38409226499999, 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.186836433, 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.537 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323322 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124127+0000", 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124131+0000", 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3950000000000001e-06 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124137+0000", 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2709999999999998e-06 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.124143+0000", 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0689999999999998e-06 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.134072+0000", 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0099292919999999993 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.134206+0000", 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013343599999999999 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.134330+0000", 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012425500000000001 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.139426+0000", 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050959380000000004 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310888+0000", 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17146186099999999 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310920+0000", 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2499999999999997e-05 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.310963+0000", 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2784000000000002e-05 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.542 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323359 2.b 2:db19cab3:::benchmark_data_smithi194_79000_object159098:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.162598+0000", 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.34562101900002, 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.148498665, 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.543 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323359 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162598+0000", 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162598+0000", 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162601+0000", 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7760000000000002e-06 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162607+0000", 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7850000000000003e-06 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162621+0000", 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3834e-05 2024-04-29T14:21:32.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162630+0000", 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0499999999999997e-06 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162691+0000", 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0783999999999999e-05 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162805+0000", 2024-04-29T14:21:32.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000113819 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.162911+0000", 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106321 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.167417+0000", 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045058579999999997 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311010+0000", 2024-04-29T14:21:32.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14359249299999999 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311056+0000", 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6869999999999997e-05 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311096+0000", 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0074999999999998e-05 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323378 2.b 2:d141cb3f:::benchmark_data_smithi194_79000_object159117:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.177760+0000", 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.33045919599999, 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.133421761, 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323378 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177760+0000", 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177760+0000", 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177762+0000", 2024-04-29T14:21:32.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0980000000000001e-06 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177767+0000", 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9200000000000003e-06 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177775+0000", 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.704e-06 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177781+0000", 2024-04-29T14:21:32.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7969999999999997e-06 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177832+0000", 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0485999999999997e-05 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.177926+0000", 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4827999999999994e-05 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178016+0000", 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9772999999999996e-05 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186701+0000", 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086849490000000008 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311126+0000", 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12442435 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311153+0000", 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7905000000000001e-05 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.311181+0000", 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7951000000000001e-05 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323380 2.b 2:d8238550:::benchmark_data_smithi194_79000_object159119:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.178214+0000", 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.33000463500002, 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14135888199999999, 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323380 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178214+0000", 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178214+0000", 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178216+0000", 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3370000000000001e-06 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178220+0000", 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7350000000000004e-06 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178231+0000", 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0457e-05 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.178238+0000", 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9460000000000001e-06 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.181789+0000", 2024-04-29T14:21:32.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003551612 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.181927+0000", 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013803900000000001 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.182069+0000", 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014142899999999999 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186818+0000", 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004749479 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319502+0000", 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13268347899999999 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319543+0000", 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.176e-05 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319573+0000", 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9609000000000001e-05 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.561 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323388 2.b 2:ddd3457d:::benchmark_data_smithi194_79000_object159127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.186762+0000", 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.32145690200002, 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.132927094, 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.562 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323388 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186762+0000", 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186762+0000", 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186765+0000", 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1020000000000001e-06 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186774+0000", 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1130000000000007e-06 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186789+0000", 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5204e-05 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186799+0000", 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8549999999999997e-06 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186850+0000", 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0837000000000001e-05 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.186976+0000", 2024-04-29T14:21:32.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000125859 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.187100+0000", 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000124072 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.190963+0000", 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038628360000000001 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319614+0000", 2024-04-29T14:21:32.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.128651668 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319649+0000", 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4668000000000001e-05 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319689+0000", 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9879999999999998e-05 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.24727.0:323419 2.b 2:d676c126:::benchmark_data_smithi194_79000_object159158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e193)", 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:13:12.217619+0000", 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 500.290599735, 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.102206715, 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.24727", 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1433516624", 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 323419 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217619+0000", 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217619+0000", 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217620+0000", 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.372e-06 2024-04-29T14:21:32.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217625+0000", 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9520000000000002e-06 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217637+0000", 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1093e-05 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217644+0000", 2024-04-29T14:21:32.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2540000000000003e-06 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217698+0000", 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4710000000000003e-05 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217783+0000", 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4277000000000006e-05 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:21:32.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.217852+0000", 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9721999999999994e-05 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.221659+0000", 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038063200000000002 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319730+0000", 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.098070933999999998 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319774+0000", 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4733e-05 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:13:12.319826+0000", 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1347999999999997e-05 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(osd.7.0:112647 4.3 e355/323 4:c12040d4:::benchmark_data_smithi194_103039_object85888:head v 355'9869, mlcod=355'9857)", 2024-04-29T14:21:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:20:20.184968+0000", 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 72.323251079000002, 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.077031323999999998, 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.184968+0000", 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.184968+0000", 2024-04-29T14:21:32.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.184968+0000", 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.68e-07 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.184973+0000", 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6059999999999996e-06 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:21:32.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.184978+0000", 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2249999999999999e-06 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.184983+0000", 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4190000000000004e-06 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.191047+0000", 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006064308 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:21:32.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.191068+0000", 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.084e-05 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.261943+0000", 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.070874793000000005 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:20:20.261999+0000", 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6465000000000002e-05 2024-04-29T14:21:32.578 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.579 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.579 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.579 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:21:32.579 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:21:32.579 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:21:32.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:32.580 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:32 smithi040 ceph-mon[39581]: pgmap v993: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:32.580 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3652993336' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:32.581 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:32 smithi194 ceph-mon[44089]: pgmap v993: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:32.581 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3652993336' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:32.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:33.064 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:33.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:33.072 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:33.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:21:33.168 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:33.199 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:33.219 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:33.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:33.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:33.303 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:33 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:33.303 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3652993336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:33.303 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:33 smithi040 ceph-mon[39581]: osdmap e376: 12 total, 11 up, 8 in 2024-04-29T14:21:33.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:33.354 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:33.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:33.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:33.508 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:33 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:33.509 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3652993336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:33.509 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:33 smithi121 ceph-mon[44533]: osdmap e376: 12 total, 11 up, 8 in 2024-04-29T14:21:33.511 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:33.534 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:33.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:33.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:33 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:33.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3652993336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:33 smithi194 ceph-mon[44089]: osdmap e376: 12 total, 11 up, 8 in 2024-04-29T14:21:33.636 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:33.666 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:33.685 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:33.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:33.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:33.825 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:33.826 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:33.846 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:33.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:33.952 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:33.980 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:34.000 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:34.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:34.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:34.132 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:34.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:34 smithi194 ceph-mon[44089]: pgmap v995: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:34.383 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:34 smithi040 ceph-mon[39581]: pgmap v995: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:34.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:34.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:34.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:34.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:34 smithi121 ceph-mon[44533]: pgmap v995: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:34.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:34.719 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:34.917 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:34.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:35.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:21:35.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:35.053 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:35.074 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:35.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:35.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:35.179 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:35.209 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:35.230 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:35.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:35.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3704207784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:35.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:35.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3704207784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:35.366 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:35.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:35.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:35.484 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:35.484 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:35.484 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:35.484 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:35.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:35.521 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3704207784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:35.770 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:35.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:35.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:35.905 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:35.926 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:35.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:36.031 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:36.072 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [5] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8] 2024-04-29T14:21:36.072 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:21:36.073 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:21:36.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:21:36.084 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:36.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:36.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:36.219 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:36.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:36.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:36.344 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:36.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: pgmap v996: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.370 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: pgmap v996: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: pgmap v996: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:21:36.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:21:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:37.512 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:21:37.513 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:21:38.220 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:38.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:38.326 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:38.358 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:38.380 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:38.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:38.484 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:38.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:38 smithi194 ceph-mon[44089]: pgmap v997: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:38.514 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:38.535 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:38.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:38.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:38.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:38 smithi040 ceph-mon[39581]: pgmap v997: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:38.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:38 smithi121 ceph-mon[44533]: pgmap v997: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:38.639 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:38.667 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:38.686 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:38.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:38.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:21:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:38.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:21:38.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:38.818 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:38.838 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:38.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:38.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:38.974 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:38.995 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:38.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:39.099 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:39.128 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:39.147 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:39.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:39.251 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:39.279 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:39.299 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:39.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:39.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:39 smithi040 ceph-mon[39581]: pgmap v998: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:39.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:39.431 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:39 smithi194 ceph-mon[44089]: pgmap v998: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:39.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:39 smithi121 ceph-mon[44533]: pgmap v998: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:40.946 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:21:41.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2195898696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:41.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2195898696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:41.372 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:41.372 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:41.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2195898696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:21:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:41.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:41.486 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:41.519 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:41.539 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:41.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:41.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:41.677 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:41.776 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:41.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:41.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:41.912 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:41.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:41.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:42.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:42.230 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:41 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:42.230 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2195898696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:21:42.230 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:41 smithi040 ceph-mon[39581]: osdmap e377: 12 total, 11 up, 8 in 2024-04-29T14:21:42.230 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:41 smithi040 ceph-mon[39581]: pgmap v1000: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:42.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:41 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:42.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2195898696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:21:42.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:41 smithi194 ceph-mon[44089]: osdmap e377: 12 total, 11 up, 8 in 2024-04-29T14:21:42.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:41 smithi194 ceph-mon[44089]: pgmap v1000: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:42.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:41 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:21:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2195898696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:21:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:41 smithi121 ceph-mon[44533]: osdmap e377: 12 total, 11 up, 8 in 2024-04-29T14:21:42.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:41 smithi121 ceph-mon[44533]: pgmap v1000: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:43 smithi194 ceph-mon[44089]: pgmap v1001: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:43.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:43 smithi040 ceph-mon[39581]: pgmap v1001: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:43.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:43 smithi121 ceph-mon[44533]: pgmap v1001: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:43.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:21:43.815 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:43.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:43.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:44 smithi194 ceph-mon[44089]: 4.7 scrub starts 2024-04-29T14:21:44.601 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:44.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:44 smithi040 ceph-mon[39581]: 4.7 scrub starts 2024-04-29T14:21:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:44.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:44.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:44 smithi121 ceph-mon[44533]: 4.7 scrub starts 2024-04-29T14:21:44.636 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:21:44.636 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:21:44.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:44.732 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:44.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:44.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 378, 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:21:44.280061+0000", 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:21:07.248070+0000", 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:19:01.251051+0000", 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5804032, 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:21:44.781 INFO:teuthology.orchestra.run.smithi040.stdout: "nodeep-scrub", 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 28, 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 4, 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:21:44.782 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:21:44.783 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:21:44.784 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:21:44.785 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:21:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:21:44.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 11.109999656677246, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 11.109999656677246, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.18000000715255737, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-29T14:21:44.788 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 4, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_2", 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:18:47.160984+0000", 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:21:44.789 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:21:44.790 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "326", 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:21:44.791 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:21:44.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:21:44.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "error_message": "pool 4 has some PGs where all OSDs are with primary_affinity 0 (at least pg 4.1), read balance score may not be reliable", 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 0, 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 0 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.794 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:21:44.796 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:21:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.799 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:21:44.800 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:21:44.801 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:21:44.802 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 152, 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.803 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.804 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:21:44.805 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:21:44.806 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.808 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.809 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.810 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:21:44.811 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.812 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 0, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 36, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 370, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:21:44.816 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3807513901 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/3807513901", 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/3807513901", 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3807513901", 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/3807513901", 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "exists" 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:21:44.817 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.818 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.819 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.820 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:21:44.822 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.823 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:21:44.824 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.825 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.826 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:21:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:21:44.828 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.829 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 370, 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.832 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:21:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.834 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.835 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 370, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:21:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:21:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:21:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:52.378880+0000", 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:43.967990+0000", 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:58.830949+0000", 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:47.950960+0000", 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:31.658336+0000", 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:04.787468+0000", 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 369 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:41.496306+0000", 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:33.310804+0000", 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:21.723586+0000", 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:00.238782+0000", 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:21:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:20:35.646548+0000", 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:21:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:43.691636+0000", 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:21:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:21:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:21:44.851 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:21:44.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:44.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:45.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:45.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:45.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:45.536 INFO:tasks.thrashosds.thrasher:j is {'epoch': 378, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:21:44.280061+0000', 'last_up_change': '2024-04-29T14:21:07.248070+0000', 'last_in_change': '2024-04-29T14:19:01.251051+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.18000000715255737, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-29T14:18:47.160984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '326', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'error_message': 'pool 4 has some PGs where all OSDs are with primary_affinity 0 (at least pg 4.1), read balance score may not be reliable', 'score_acting': 0, 'score_stable': 0}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 152, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 36, 'up_thru': 323, 'down_at': 370, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 3807513901}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3807513901}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3807513901}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 3807513901}]}, 'public_addr': '172.21.15.121:6804/3807513901', 'cluster_addr': '172.21.15.121:6805/3807513901', 'heartbeat_back_addr': '172.21.15.121:6807/3807513901', 'heartbeat_front_addr': '172.21.15.121:6806/3807513901', 'state': ['exists']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 206, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:52.378880+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:21:43.967990+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:20:58.830949+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:20:47.950960+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:31.658336+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:21:04.787468+0000', 'dead_epoch': 369}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:41.496306+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:33.310804+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:21:21.723586+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:21:00.238782+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:20:35.646548+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:21:43.691636+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:21:45.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: 4.7 scrub ok 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: 4.c scrub starts 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: osdmap e378: 12 total, 11 up, 8 in 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1175892822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:21:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:45 smithi194 ceph-mon[44089]: pgmap v1003: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:45.600 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: 4.7 scrub ok 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: 4.c scrub starts 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: osdmap e378: 12 total, 11 up, 8 in 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1175892822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:21:45.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:45 smithi040 ceph-mon[39581]: pgmap v1003: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:45.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:45.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:45.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: 4.7 scrub ok 2024-04-29T14:21:45.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:45.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:45.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: 4.c scrub starts 2024-04-29T14:21:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: osdmap e378: 12 total, 11 up, 8 in 2024-04-29T14:21:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1175892822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:21:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:45 smithi121 ceph-mon[44533]: pgmap v1003: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:46.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:46 smithi040 ceph-mon[39581]: 4.c scrub ok 2024-04-29T14:21:46.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:46 smithi040 ceph-mon[39581]: 4.f scrub starts 2024-04-29T14:21:46.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:46 smithi040 ceph-mon[39581]: 4.4 scrub starts 2024-04-29T14:21:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:46 smithi194 ceph-mon[44089]: 4.c scrub ok 2024-04-29T14:21:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:46 smithi194 ceph-mon[44089]: 4.f scrub starts 2024-04-29T14:21:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:46 smithi194 ceph-mon[44089]: 4.4 scrub starts 2024-04-29T14:21:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:46 smithi121 ceph-mon[44533]: 4.c scrub ok 2024-04-29T14:21:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:46 smithi121 ceph-mon[44533]: 4.f scrub starts 2024-04-29T14:21:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:46 smithi121 ceph-mon[44533]: 4.4 scrub starts 2024-04-29T14:21:46.725 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:46.745 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:46.765 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:46.785 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:46.806 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:46.827 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:46.848 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:46.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:46.890 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:46.910 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:46.931 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:46.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: 4.f scrub ok 2024-04-29T14:21:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: 4.4 scrub ok 2024-04-29T14:21:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: 4.5 scrub starts 2024-04-29T14:21:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: 4.8 scrub starts 2024-04-29T14:21:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: 4.5 scrub ok 2024-04-29T14:21:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1279547231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: 4.8 scrub ok 2024-04-29T14:21:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:47 smithi194 ceph-mon[44089]: pgmap v1004: 17 pgs: 1 active+clean+scrubbing, 2 undersized+degraded+peered, 14 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: 4.f scrub ok 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: 4.4 scrub ok 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: 4.5 scrub starts 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: 4.8 scrub starts 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: 4.5 scrub ok 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1279547231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: 4.8 scrub ok 2024-04-29T14:21:47.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:47 smithi040 ceph-mon[39581]: pgmap v1004: 17 pgs: 1 active+clean+scrubbing, 2 undersized+degraded+peered, 14 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: 4.f scrub ok 2024-04-29T14:21:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: 4.4 scrub ok 2024-04-29T14:21:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: 4.5 scrub starts 2024-04-29T14:21:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: 4.8 scrub starts 2024-04-29T14:21:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: 4.5 scrub ok 2024-04-29T14:21:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1279547231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:47.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: 4.8 scrub ok 2024-04-29T14:21:47.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:47 smithi121 ceph-mon[44533]: pgmap v1004: 17 pgs: 1 active+clean+scrubbing, 2 undersized+degraded+peered, 14 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:21:47.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:21:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:47.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:47.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:21:47.860 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:47.892 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:47.913 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:47.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:48.017 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:48.046 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:48.066 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:48.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:48.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:48.201 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:48.222 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:48.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:48.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:48.355 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:48.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:48.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:48.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:48.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:48 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:48.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:48 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:48.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:48.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:48 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:49.008 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:49.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:49.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:49.146 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:49.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:49.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:49.300 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:49.321 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:49.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:49.425 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:49.450 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:49 smithi194 ceph-mon[44089]: osdmap e379: 12 total, 11 up, 8 in 2024-04-29T14:21:49.450 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:49 smithi194 ceph-mon[44089]: pgmap v1006: 17 pgs: 1 active+clean+scrubbing, 2 undersized+degraded+peered, 14 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:49.454 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:49.471 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:49.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:49.574 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:21:49.601 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:49.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:49 smithi040 ceph-mon[39581]: osdmap e379: 12 total, 11 up, 8 in 2024-04-29T14:21:49.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:49 smithi040 ceph-mon[39581]: pgmap v1006: 17 pgs: 1 active+clean+scrubbing, 2 undersized+degraded+peered, 14 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:49.620 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:21:49.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:49.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:21:49.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:49 smithi121 ceph-mon[44533]: osdmap e379: 12 total, 11 up, 8 in 2024-04-29T14:21:49.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:49 smithi121 ceph-mon[44533]: pgmap v1006: 17 pgs: 1 active+clean+scrubbing, 2 undersized+degraded+peered, 14 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:49.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:49.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:21:49.755 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:49.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:49.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:49.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:49.909 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:49.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:50.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:50.321 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:50 smithi040 ceph-mon[39581]: 4.b scrub starts 2024-04-29T14:21:50.321 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:50 smithi040 ceph-mon[39581]: 4.b scrub ok 2024-04-29T14:21:50.385 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:50 smithi194 ceph-mon[44089]: 4.b scrub starts 2024-04-29T14:21:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:50 smithi194 ceph-mon[44089]: 4.b scrub ok 2024-04-29T14:21:50.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:50 smithi121 ceph-mon[44533]: 4.b scrub starts 2024-04-29T14:21:50.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:50 smithi121 ceph-mon[44533]: 4.b scrub ok 2024-04-29T14:21:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:21:51.506 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:21:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:51 smithi194 ceph-mon[44089]: pgmap v1007: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:51.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:51 smithi040 ceph-mon[39581]: pgmap v1007: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:51.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:51 smithi121 ceph-mon[44533]: pgmap v1007: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:52.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:52 smithi040 ceph-mon[39581]: 4.6 scrub starts 2024-04-29T14:21:52.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:52 smithi040 ceph-mon[39581]: from='client.15894 v1:172.21.15.40:0/3464821054' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:21:52.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:52 smithi121 ceph-mon[44533]: 4.6 scrub starts 2024-04-29T14:21:52.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:52 smithi121 ceph-mon[44533]: from='client.15894 v1:172.21.15.40:0/3464821054' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:21:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:52.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:52 smithi194 ceph-mon[44089]: 4.6 scrub starts 2024-04-29T14:21:52.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:52 smithi194 ceph-mon[44089]: from='client.15894 v1:172.21.15.40:0/3464821054' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:21:53.214 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:53.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:53.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:53.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:21:53.319 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:21:53.346 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:53.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:21:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:53.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:21:53.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:53.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: 4.6 scrub ok 2024-04-29T14:21:53.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:21:53.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:21:53.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/119804729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:53.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:21:53.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:21:53.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:21:53.498 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:53 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:21:53.500 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:53.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: 4.6 scrub ok 2024-04-29T14:21:53.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:21:53.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:21:53.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/119804729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:53.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:21:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:21:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:21:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:53 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:21:53.811 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1007,"stamp":"2024-04-29T14:21:51.187434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7546864184,"num_objects":115150,"num_object_clones":0,"num_object_copies":230300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115150,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":198395,"num_write_kb":10030459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160981,"ondisk_log_size":160981,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17729288,"kb_used_data":14761580,"kb_used_omap":12,"kb_used_meta":2967603,"kb_avail":732231928,"statfs":{"total":767960285184,"available":749805494272,"internally_reserved":0,"allocated":15115857920,"data_stored":15112774891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12765,"internal_metadata":3038826019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001815"},"pg_stats":[{"pgid":"4.f","version":"370'12469","reported_seq":12770,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.310247+0000","last_change":"2024-04-29T14:21:46.009086+0000","last_active":"2024-04-29T14:21:48.310247+0000","last_peered":"2024-04-29T14:21:48.310247+0000","last_clean":"2024-04-29T14:21:48.310247+0000","last_became_active":"2024-04-29T14:18:49.370395+0000","last_became_peered":"2024-04-29T14:18:49.370395+0000","last_unstale":"2024-04-29T14:21:48.310247+0000","last_undegraded":"2024-04-29T14:21:48.310247+0000","last_fullsized":"2024-04-29T14:21:48.310247+0000","mapping_epoch":323,"log_start":"331'2400","ondisk_log_start":"331'2400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12469","last_scrub_stamp":"2024-04-29T14:21:46.008916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:46.008916+0000","objects_scrubbed":7173,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:04.016084+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470089728,"num_objects":7173,"num_object_clones":0,"num_object_copies":14346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12469,"num_write_kb":628544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"370'12350","reported_seq":12651,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.307300+0000","last_change":"2024-04-29T14:21:21.878717+0000","last_active":"2024-04-29T14:21:48.307300+0000","last_peered":"2024-04-29T14:21:48.307300+0000","last_clean":"2024-04-29T14:21:48.307300+0000","last_became_active":"2024-04-29T14:18:49.152699+0000","last_became_peered":"2024-04-29T14:18:49.152699+0000","last_unstale":"2024-04-29T14:21:48.307300+0000","last_undegraded":"2024-04-29T14:21:48.307300+0000","last_fullsized":"2024-04-29T14:21:48.307300+0000","mapping_epoch":323,"log_start":"331'2300","ondisk_log_start":"331'2300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12350","last_scrub_stamp":"2024-04-29T14:21:21.878602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:21.878602+0000","objects_scrubbed":7262,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:22:51.528365+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475922432,"num_objects":7262,"num_object_clones":0,"num_object_copies":14524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12350,"num_write_kb":627584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"370'12486","reported_seq":12785,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.307311+0000","last_change":"2024-04-29T14:21:19.844364+0000","last_active":"2024-04-29T14:21:48.307311+0000","last_peered":"2024-04-29T14:21:48.307311+0000","last_clean":"2024-04-29T14:21:48.307311+0000","last_became_active":"2024-04-29T14:18:49.155690+0000","last_became_peered":"2024-04-29T14:18:49.155690+0000","last_unstale":"2024-04-29T14:21:48.307311+0000","last_undegraded":"2024-04-29T14:21:48.307311+0000","last_fullsized":"2024-04-29T14:21:48.307311+0000","mapping_epoch":323,"log_start":"331'2400","ondisk_log_start":"331'2400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12486","last_scrub_stamp":"2024-04-29T14:21:19.844220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:19.844220+0000","objects_scrubbed":7278,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:22:44.701458+0000","scrub_duration":599,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476905496,"num_objects":7278,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12486,"num_write_kb":632385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"370'12380","reported_seq":12678,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.307312+0000","last_change":"2024-04-29T14:21:44.740246+0000","last_active":"2024-04-29T14:21:48.307312+0000","last_peered":"2024-04-29T14:21:48.307312+0000","last_clean":"2024-04-29T14:21:48.307312+0000","last_became_active":"2024-04-29T14:18:49.156158+0000","last_became_peered":"2024-04-29T14:18:49.156158+0000","last_unstale":"2024-04-29T14:21:48.307312+0000","last_undegraded":"2024-04-29T14:21:48.307312+0000","last_fullsized":"2024-04-29T14:21:48.307312+0000","mapping_epoch":323,"log_start":"331'2300","ondisk_log_start":"331'2300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12380","last_scrub_stamp":"2024-04-29T14:21:44.740055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:44.740055+0000","objects_scrubbed":7272,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:22:57.637928+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476577792,"num_objects":7272,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":628864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"370'12656","reported_seq":12959,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:49.484191+0000","last_change":"2024-04-29T14:21:49.484255+0000","last_active":"2024-04-29T14:21:49.484191+0000","last_peered":"2024-04-29T14:21:49.484191+0000","last_clean":"2024-04-29T14:21:49.484191+0000","last_became_active":"2024-04-29T14:18:49.152097+0000","last_became_peered":"2024-04-29T14:18:49.152097+0000","last_unstale":"2024-04-29T14:21:49.484191+0000","last_undegraded":"2024-04-29T14:21:49.484191+0000","last_fullsized":"2024-04-29T14:21:49.484191+0000","mapping_epoch":323,"log_start":"331'2600","ondisk_log_start":"331'2600","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12656","last_scrub_stamp":"2024-04-29T14:21:49.484118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:49.484118+0000","objects_scrubbed":7362,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:15.923754+0000","scrub_duration":628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482476032,"num_objects":7362,"num_object_clones":0,"num_object_copies":14724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12656,"num_write_kb":640576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"370'12343","reported_seq":12559,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.382594+0000","last_change":"2024-04-29T14:20:09.832596+0000","last_active":"2024-04-29T14:21:48.382594+0000","last_peered":"2024-04-29T14:21:48.382594+0000","last_clean":"2024-04-29T14:21:48.382594+0000","last_became_active":"2024-04-29T14:18:49.156474+0000","last_became_peered":"2024-04-29T14:18:49.156474+0000","last_unstale":"2024-04-29T14:21:48.382594+0000","last_undegraded":"2024-04-29T14:21:48.382594+0000","last_fullsized":"2024-04-29T14:21:48.382594+0000","mapping_epoch":323,"log_start":"331'2300","ondisk_log_start":"331'2300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'8698","last_scrub_stamp":"2024-04-29T14:20:09.832334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:20:09.832334+0000","objects_scrubbed":8645,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471924736,"num_objects":7201,"num_object_clones":0,"num_object_copies":14402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12343,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"370'12311","reported_seq":12606,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.382594+0000","last_change":"2024-04-29T14:21:25.446462+0000","last_active":"2024-04-29T14:21:48.382594+0000","last_peered":"2024-04-29T14:21:48.382594+0000","last_clean":"2024-04-29T14:21:48.382594+0000","last_became_active":"2024-04-29T14:18:49.371178+0000","last_became_peered":"2024-04-29T14:18:49.371178+0000","last_unstale":"2024-04-29T14:21:48.382594+0000","last_undegraded":"2024-04-29T14:21:48.382594+0000","last_fullsized":"2024-04-29T14:21:48.382594+0000","mapping_epoch":323,"log_start":"331'2300","ondisk_log_start":"331'2300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12311","last_scrub_stamp":"2024-04-29T14:21:25.446261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:25.446261+0000","objects_scrubbed":7179,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:22:35.525521+0000","scrub_duration":585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470482944,"num_objects":7179,"num_object_clones":0,"num_object_copies":14358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12311,"num_write_kb":623680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"370'12370","reported_seq":12666,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.972713+0000","last_change":"2024-04-29T14:21:46.778153+0000","last_active":"2024-04-29T14:21:48.972713+0000","last_peered":"2024-04-29T14:21:48.972713+0000","last_clean":"2024-04-29T14:21:48.972713+0000","last_became_active":"2024-04-29T14:18:49.370541+0000","last_became_peered":"2024-04-29T14:18:49.370541+0000","last_unstale":"2024-04-29T14:21:48.972713+0000","last_undegraded":"2024-04-29T14:21:48.972713+0000","last_fullsized":"2024-04-29T14:21:48.972713+0000","mapping_epoch":323,"log_start":"331'2300","ondisk_log_start":"331'2300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12370","last_scrub_stamp":"2024-04-29T14:21:46.777975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:46.777975+0000","objects_scrubbed":7196,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:09.308038+0000","scrub_duration":605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471597056,"num_objects":7196,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12370,"num_write_kb":626112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"370'12296","reported_seq":12591,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-04-29T14:21:45.358350+0000","last_change":"2024-04-29T14:21:43.967168+0000","last_active":"2024-04-29T14:21:45.358350+0000","last_peered":"2024-04-29T14:21:45.358350+0000","last_clean":"2024-04-29T14:21:45.358350+0000","last_became_active":"2024-04-29T14:18:49.151382+0000","last_became_peered":"2024-04-29T14:18:49.151382+0000","last_unstale":"2024-04-29T14:21:45.358350+0000","last_undegraded":"2024-04-29T14:21:45.358350+0000","last_fullsized":"2024-04-29T14:21:45.358350+0000","mapping_epoch":323,"log_start":"330'2200","ondisk_log_start":"330'2200","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12296","last_scrub_stamp":"2024-04-29T14:21:43.966992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:43.966992+0000","objects_scrubbed":7200,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:11.384006+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471859200,"num_objects":7200,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12296,"num_write_kb":623872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"370'12433","reported_seq":12652,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.310237+0000","last_change":"2024-04-29T14:20:11.946110+0000","last_active":"2024-04-29T14:21:48.310237+0000","last_peered":"2024-04-29T14:21:48.310237+0000","last_clean":"2024-04-29T14:21:48.310237+0000","last_became_active":"2024-04-29T14:18:49.157084+0000","last_became_peered":"2024-04-29T14:18:49.157084+0000","last_unstale":"2024-04-29T14:21:48.310237+0000","last_undegraded":"2024-04-29T14:21:48.310237+0000","last_fullsized":"2024-04-29T14:21:48.310237+0000","mapping_epoch":323,"log_start":"331'2400","ondisk_log_start":"331'2400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'8949","last_scrub_stamp":"2024-04-29T14:20:11.945873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:20:11.945873+0000","objects_scrubbed":8894,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480968704,"num_objects":7339,"num_object_clones":0,"num_object_copies":14678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12433,"num_write_kb":632704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7],"acting":[9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"370'11966","reported_seq":12254,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-04-29T14:21:46.039306+0000","last_change":"2024-04-29T14:21:46.039338+0000","last_active":"2024-04-29T14:21:46.039306+0000","last_peered":"2024-04-29T14:21:46.039306+0000","last_clean":"2024-04-29T14:21:46.039306+0000","last_became_active":"2024-04-29T14:18:49.156484+0000","last_became_peered":"2024-04-29T14:18:49.156484+0000","last_unstale":"2024-04-29T14:21:46.039306+0000","last_undegraded":"2024-04-29T14:21:46.039306+0000","last_fullsized":"2024-04-29T14:21:46.039306+0000","mapping_epoch":323,"log_start":"330'1900","ondisk_log_start":"330'1900","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'11966","last_scrub_stamp":"2024-04-29T14:21:46.039226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:46.039226+0000","objects_scrubbed":6928,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:22:53.058177+0000","scrub_duration":681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454033408,"num_objects":6928,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11966,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"370'14737","reported_seq":15073,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.972726+0000","last_change":"2024-04-29T14:20:48.273884+0000","last_active":"2024-04-29T14:21:48.972726+0000","last_peered":"2024-04-29T14:21:48.972726+0000","last_clean":"2024-04-29T14:21:48.972726+0000","last_became_active":"2024-04-29T14:18:49.371335+0000","last_became_peered":"2024-04-29T14:18:49.371335+0000","last_unstale":"2024-04-29T14:21:48.972726+0000","last_undegraded":"2024-04-29T14:21:48.972726+0000","last_fullsized":"2024-04-29T14:21:48.972726+0000","mapping_epoch":323,"log_start":"336'4700","ondisk_log_start":"336'4700","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'13479","last_scrub_stamp":"2024-04-29T14:20:48.273648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:20:48.273648+0000","objects_scrubbed":8508,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:21:51.266754+0000","scrub_duration":2046,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470351872,"num_objects":7177,"num_object_clones":0,"num_object_copies":14354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":626688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"369'12412","reported_seq":12636,"reported_epoch":379,"state":"undersized+degraded+peered","last_fresh":"2024-04-29T14:21:48.310237+0000","last_change":"2024-04-29T14:21:08.282228+0000","last_active":"2024-04-29T14:21:07.279755+0000","last_peered":"2024-04-29T14:21:48.310237+0000","last_clean":"2024-04-29T14:21:07.253125+0000","last_became_active":"2024-04-29T14:18:49.158456+0000","last_became_peered":"2024-04-29T14:21:08.282228+0000","last_unstale":"2024-04-29T14:21:48.310237+0000","last_undegraded":"2024-04-29T14:21:08.281851+0000","last_fullsized":"2024-04-29T14:21:08.281740+0000","mapping_epoch":370,"log_start":"331'2400","ondisk_log_start":"331'2400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'9216","last_scrub_stamp":"2024-04-29T14:20:16.920768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:20:16.920768+0000","objects_scrubbed":9154,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:21:50.284430+0000","scrub_duration":1726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471728128,"num_objects":7198,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":627520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":7198}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"369'12534","reported_seq":12766,"reported_epoch":379,"state":"undersized+degraded+peered","last_fresh":"2024-04-29T14:21:48.309606+0000","last_change":"2024-04-29T14:21:08.285927+0000","last_active":"2024-04-29T14:21:07.275309+0000","last_peered":"2024-04-29T14:21:48.309606+0000","last_clean":"2024-04-29T14:21:07.262284+0000","last_became_active":"2024-04-29T14:18:49.371468+0000","last_became_peered":"2024-04-29T14:21:08.285927+0000","last_unstale":"2024-04-29T14:21:48.309606+0000","last_undegraded":"2024-04-29T14:21:08.285485+0000","last_fullsized":"2024-04-29T14:21:08.285395+0000","mapping_epoch":370,"log_start":"331'2500","ondisk_log_start":"331'2500","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'11031","last_scrub_stamp":"2024-04-29T14:20:45.230006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:20:45.230006+0000","objects_scrubbed":8676,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:21:57.826734+0000","scrub_duration":1767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467795968,"num_objects":7138,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12534,"num_write_kb":629504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[{"shards":"11","objects":7138}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"370'12416","reported_seq":12709,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.972725+0000","last_change":"2024-04-29T14:21:18.742529+0000","last_active":"2024-04-29T14:21:48.972725+0000","last_peered":"2024-04-29T14:21:48.972725+0000","last_clean":"2024-04-29T14:21:48.972725+0000","last_became_active":"2024-04-29T14:18:49.158015+0000","last_became_peered":"2024-04-29T14:18:49.158015+0000","last_unstale":"2024-04-29T14:21:48.972725+0000","last_undegraded":"2024-04-29T14:21:48.972725+0000","last_fullsized":"2024-04-29T14:21:48.972725+0000","mapping_epoch":323,"log_start":"331'2400","ondisk_log_start":"331'2400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12416","last_scrub_stamp":"2024-04-29T14:21:18.742364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:18.742364+0000","objects_scrubbed":7086,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:22:36.885033+0000","scrub_duration":572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464388096,"num_objects":7086,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":624064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"370'12385","reported_seq":12681,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.382595+0000","last_change":"2024-04-29T14:21:46.642204+0000","last_active":"2024-04-29T14:21:48.382595+0000","last_peered":"2024-04-29T14:21:48.382595+0000","last_clean":"2024-04-29T14:21:48.382595+0000","last_became_active":"2024-04-29T14:18:49.156512+0000","last_became_peered":"2024-04-29T14:18:49.156512+0000","last_unstale":"2024-04-29T14:21:48.382595+0000","last_undegraded":"2024-04-29T14:21:48.382595+0000","last_fullsized":"2024-04-29T14:21:48.382595+0000","mapping_epoch":323,"log_start":"331'2300","ondisk_log_start":"331'2300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12385","last_scrub_stamp":"2024-04-29T14:21:46.642080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:46.642080+0000","objects_scrubbed":7159,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:22:53.172215+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469172224,"num_objects":7159,"num_object_clones":0,"num_object_copies":14318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":967,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-29T14:21:48.972857+0000","last_change":"2024-04-29T14:20:18.736993+0000","last_active":"2024-04-29T14:21:48.972857+0000","last_peered":"2024-04-29T14:21:48.972857+0000","last_clean":"2024-04-29T14:21:48.972857+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:21:48.972857+0000","last_undegraded":"2024-04-29T14:21:48.972857+0000","last_fullsized":"2024-04-29T14:21:48.972857+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:20:18.736901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:20:18.736901+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7546273816,"num_objects":115148,"num_object_clones":0,"num_object_copies":230296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115148,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198214,"num_write_kb":10027521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15108087808,"data_stored":15108079664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":30,"acting":30,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122227,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2696936,"kb_used_data":2296476,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91048216,"statfs":{"total":95995035648,"available":93233373184,"internally_reserved":0,"allocated":2351591424,"data_stored":2351203644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":410057156},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2490640,"kb_used_data":2288712,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":91254512,"statfs":{"total":95995035648,"available":93444620288,"internally_reserved":0,"allocated":2343641088,"data_stored":2343263124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":206764480},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645501,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2202212,"kb_used_data":1855896,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91542940,"statfs":{"total":95995035648,"available":93739970560,"internally_reserved":0,"allocated":1900437504,"data_stored":1900049224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234182,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2665960,"kb_used_data":2287708,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91079192,"statfs":{"total":95995035648,"available":93265092608,"internally_reserved":0,"allocated":2342612992,"data_stored":2342225232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2340896,"kb_used_data":1844628,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91404256,"statfs":{"total":95995035648,"available":93597958144,"internally_reserved":0,"allocated":1888899072,"data_stored":1888514864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738602,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714464,"kb_used_data":1386196,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92030688,"statfs":{"total":95995035648,"available":94239424512,"internally_reserved":0,"allocated":1419464704,"data_stored":1419080496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182619,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986390,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2386276,"kb_used_data":1867800,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91358876,"statfs":{"total":95995035648,"available":93551489024,"internally_reserved":0,"allocated":1912627200,"data_stored":1912238920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1231904,"kb_used_data":934164,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92513248,"statfs":{"total":95995035648,"available":94733565952,"internally_reserved":0,"allocated":956583936,"data_stored":956199387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1418723328,"data_stored":1418723328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1911885824,"data_stored":1911881752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":955056128,"data_stored":955056128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1888157696,"data_stored":1888157696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2341339136,"data_stored":2341339136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1899696128,"data_stored":1899692056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2342912000,"data_stored":2342912000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2350317568,"data_stored":2350317568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:21:53.813 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [8, 10, 9, 7] 2024-04-29T14:21:53.813 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.0', '8', '10', '9', '7'] 2024-04-29T14:21:53.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 1.0 8 10 9 7 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: 4.6 scrub ok 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/119804729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:21:53.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:53 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:21:54.019 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:21:54.048 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:21:54.077 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:21:54.098 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:21:54.196 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:21:54.196 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:21:54.196 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:21:54.196 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:21:54.202 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:21:54.224 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:21:54.244 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:21:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:21:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:21:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:21:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:21:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:21:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:21:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:21:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:21:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:21:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:21:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:21:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:21:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:54.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:54.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/119804729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: osdmap e380: 12 total, 11 up, 8 in 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3403122467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:54.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:54.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:54.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:21:54.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/119804729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: osdmap e380: 12 total, 11 up, 8 in 2024-04-29T14:21:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3403122467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/119804729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: osdmap e380: 12 total, 11 up, 8 in 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3403122467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:21:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:54.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:55.331 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:55.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:55.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:55.464 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: 4.a scrub starts 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: 4.3 scrub starts 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: osdmap e381: 12 total, 11 up, 8 in 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: 4.3 scrub ok 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:21:55.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: pgmap v1011: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:55.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: pgmap v1012: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:55.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:55.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:55.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: 4.a scrub starts 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: 4.3 scrub starts 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: osdmap e381: 12 total, 11 up, 8 in 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: 4.3 scrub ok 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: pgmap v1011: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: pgmap v1012: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:55.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:55 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:55.919 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: 4.a scrub starts 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.949 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: 4.3 scrub starts 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: osdmap e381: 12 total, 11 up, 8 in 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: 4.3 scrub ok 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: pgmap v1011: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: pgmap v1012: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:21:55.951 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:55.973 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:55.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:55.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:56.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:21:56.106 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:56.858 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:21:56.860 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:56 smithi040 ceph-mon[39581]: 4.a scrub ok 2024-04-29T14:21:56.860 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:56 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:21:56.860 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:56 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:21:56.860 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:56 smithi040 ceph-mon[39581]: osdmap e382: 12 total, 11 up, 8 in 2024-04-29T14:21:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:56.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:21:56.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:56 smithi121 ceph-mon[44533]: 4.a scrub ok 2024-04-29T14:21:56.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:56 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:21:56.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:56 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:21:56.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:56 smithi121 ceph-mon[44533]: osdmap e382: 12 total, 11 up, 8 in 2024-04-29T14:21:56.962 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:56.992 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:56 smithi194 ceph-mon[44089]: 4.a scrub ok 2024-04-29T14:21:56.993 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:56 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:21:56.993 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:56 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:21:56.993 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:56 smithi194 ceph-mon[44089]: osdmap e382: 12 total, 11 up, 8 in 2024-04-29T14:21:56.994 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:21:56.994 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:21:56.995 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:57.016 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:57.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:57.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:21:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:21:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:57.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:21:57.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:57.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:57.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:57.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:57.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:57.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:57.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:57.816 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:21:57.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:21:57.848 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:21:57.870 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:21:57.870 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:21:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:57.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:21:57.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:21:58.003 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:58.024 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:21:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:58.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:21:58.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:21:58.160 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:58.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:21:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:58.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:21:58.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:58.315 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:58.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:58.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:58 smithi040 ceph-mon[39581]: pgmap v1014: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:58.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:58 smithi121 ceph-mon[44533]: pgmap v1014: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:58.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:21:58.469 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:21:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:58.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:21:58.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:58 smithi194 ceph-mon[44089]: pgmap v1014: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:21:58.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:58.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:58.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:21:58.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:21:58.841 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:58.862 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:21:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:58.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:21:58.954 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:21:58.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:21:58.993 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:21:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:set 1.0 pg_upmap_items mapping to [8->10,9->7] 2024-04-29T14:21:59.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:21:59 smithi040 ceph-mon[39581]: from='client.15912 v1:172.21.15.40:0/2788168858' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [8, 10, 9, 7]}]: dispatch 2024-04-29T14:21:59.259 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:21:59.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:21:59 smithi121 ceph-mon[44533]: from='client.15912 v1:172.21.15.40:0/2788168858' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [8, 10, 9, 7]}]: dispatch 2024-04-29T14:21:59.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:21:59 smithi194 ceph-mon[44089]: from='client.15912 v1:172.21.15.40:0/2788168858' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [8, 10, 9, 7]}]: dispatch 2024-04-29T14:21:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:21:59.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:59.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:21:59.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:21:59.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:21:59.962 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:21:59.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:21:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:21:59.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:00.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:00.138 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:00.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:00.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:00.270 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:00 smithi121 ceph-mon[44533]: pgmap v1015: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:00 smithi121 ceph-mon[44533]: from='client.15912 v1:172.21.15.40:0/2788168858' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [8, 10, 9, 7]}]': finished 2024-04-29T14:22:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:00 smithi121 ceph-mon[44533]: osdmap e383: 12 total, 11 up, 8 in 2024-04-29T14:22:00.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:00 smithi194 ceph-mon[44089]: pgmap v1015: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:00.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:00 smithi194 ceph-mon[44089]: from='client.15912 v1:172.21.15.40:0/2788168858' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [8, 10, 9, 7]}]': finished 2024-04-29T14:22:00.426 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:00 smithi194 ceph-mon[44089]: osdmap e383: 12 total, 11 up, 8 in 2024-04-29T14:22:00.467 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:00.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:00.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:00 smithi040 ceph-mon[39581]: pgmap v1015: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:00.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:00 smithi040 ceph-mon[39581]: from='client.15912 v1:172.21.15.40:0/2788168858' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [8, 10, 9, 7]}]': finished 2024-04-29T14:22:00.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:00 smithi040 ceph-mon[39581]: osdmap e383: 12 total, 11 up, 8 in 2024-04-29T14:22:00.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:00.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:00.912 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:00.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:00.982 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:01.001 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:01.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:01.025 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:22:01.025 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:22:01.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:01.157 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:01.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:01.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:01.291 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:01.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:01.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:01.443 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:01.861 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:22:01.876 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:01.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:01.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:02.083 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:02.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:02.123 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:22:02.138 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:22:02.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:02.216 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:02 smithi194 ceph-mon[44089]: pgmap v1017: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/788884626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:22:02.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:02 smithi040 ceph-mon[39581]: pgmap v1017: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:02.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/788884626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:22:02.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:02 smithi121 ceph-mon[44533]: pgmap v1017: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:02.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/788884626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:22:02.414 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:02.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:02.518 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:02.547 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:02.566 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:02.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:02.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:02.696 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:02.748 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:02.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:02.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:02.883 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:02.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:02.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:03.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:03.039 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:03 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/788884626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:22:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:03 smithi040 ceph-mon[39581]: osdmap e384: 12 total, 11 up, 8 in 2024-04-29T14:22:03.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:03 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:03.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/788884626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:22:03.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:03 smithi121 ceph-mon[44533]: osdmap e384: 12 total, 11 up, 8 in 2024-04-29T14:22:03.488 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:03.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:03.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:03 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:03.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/788884626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:22:03.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:03 smithi194 ceph-mon[44089]: osdmap e384: 12 total, 11 up, 8 in 2024-04-29T14:22:03.592 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:03.620 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:03.818 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:03.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:03.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:03.954 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:03.975 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:03.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:04.009 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:22:04.028 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:22:04.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:22:04.067 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:22:04.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:04.087 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:22:04.102 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:22:04.102 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:22:04.107 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:04.108 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:22:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:04.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:04.131 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:22:04.152 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:22:04.171 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:22:04.192 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:22:04.212 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:22:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:22:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:22:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:22:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:22:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:22:04.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:04.233 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:04 smithi040 ceph-mon[39581]: pgmap v1019: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:04.234 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4270530644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:04.283 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:04.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:04.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:04 smithi121 ceph-mon[44533]: pgmap v1019: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:04.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4270530644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:04.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:04.417 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:04.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:04.443 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:04 smithi194 ceph-mon[44089]: pgmap v1019: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:04.443 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4270530644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:04.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:04.572 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:04.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:04.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:04.697 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:04.727 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:04.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:04.792 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:22:04.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:22:04.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:22:04.900 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:04.921 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:04.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:05.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:05.056 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:05.076 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:05.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:05.090 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:22:05.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:05.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.211 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.212 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.213 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:05.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:05.338 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:05.354 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [5] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8] 2024-04-29T14:22:05.354 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:22:05.355 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-04-29T14:22:05.355 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-04-29T14:22:05.355 DEBUG:teuthology.orchestra.run.smithi121:osd.5> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5.service 2024-04-29T14:22:05.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl start ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5 2024-04-29T14:22:05.367 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:05.387 INFO:journalctl@ceph.osd.5.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:22:05.388 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.390 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.391 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.391 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.391 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:05.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:05.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:22:05.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.491 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:05.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:05.544 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:05.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:05.624 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:05 smithi121 systemd[1]: Starting Ceph osd.5 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:22:05.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:05.675 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:05.697 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:05.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:05.802 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:05.829 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:06.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:06.133 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: pgmap v1020: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: osdmap e385: 12 total, 11 up, 8 in 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=53,logm=1} crc {auth=2995697159,config=359415709,health=1091798053,kv=3585853274,logm=1404913468}) 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530826868}) 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4249369625}) 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454928464}) 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1728110384}) 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707541192}) 2024-04-29T14:22:06.158 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517201456}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950468761}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539755239}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776364308}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460100659}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419840721}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4090106845}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567956863}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219923055}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178331337}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007707767}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833277447}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=141091887}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091381347}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129964790}) 2024-04-29T14:22:06.159 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940079484}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899389752}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169875580}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=320946690}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597725171}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205505587}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241366737}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224727425}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1013704270}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2108061596}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2374596705}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518779019}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865675883}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210713602}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156358688}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161595278}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3688096762}) 2024-04-29T14:22:06.160 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783940886}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384435165}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1613531942}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825705797}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265803210}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916183333}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1896805587}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430439930}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309889895}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=111764570}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4163107622}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=2} crc {logm=2942141979,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=608560165,mon_config_key=1964985202}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2320865557}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1030335798}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=3843852346,monmap=455069592,osd_metadata=401180102,osd_pg_creating=469659463,osdmap=2065198245}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1638942680}) 2024-04-29T14:22:06.161 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=831353484}) 2024-04-29T14:22:06.162 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:06.181 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:06.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:06.285 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:06.312 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: pgmap v1020: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: osdmap e385: 12 total, 11 up, 8 in 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=53,logm=1} crc {auth=2995697159,config=359415709,health=1091798053,kv=3585853274,logm=1404913468}) 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530826868}) 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4249369625}) 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454928464}) 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1728110384}) 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707541192}) 2024-04-29T14:22:06.415 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517201456}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950468761}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539755239}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776364308}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460100659}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419840721}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4090106845}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567956863}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219923055}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178331337}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007707767}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833277447}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=141091887}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091381347}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129964790}) 2024-04-29T14:22:06.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940079484}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899389752}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169875580}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=320946690}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597725171}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205505587}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241366737}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224727425}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1013704270}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2108061596}) 2024-04-29T14:22:06.417 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-29T14:22:06.418 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-29T14:22:06.418 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-29T14:22:06.418 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-29T14:22:06.418 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-29T14:22:06.418 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-29T14:22:06.418 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-29T14:22:06.418 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:06 smithi121 bash[114896]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-29T14:22:06.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2374596705}) 2024-04-29T14:22:06.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518779019}) 2024-04-29T14:22:06.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865675883}) 2024-04-29T14:22:06.418 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210713602}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156358688}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161595278}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3688096762}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783940886}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384435165}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1613531942}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825705797}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265803210}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916183333}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1896805587}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430439930}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309889895}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=111764570}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4163107622}) 2024-04-29T14:22:06.419 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=2} crc {logm=2942141979,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=608560165,mon_config_key=1964985202}) 2024-04-29T14:22:06.420 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2320865557}) 2024-04-29T14:22:06.420 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1030335798}) 2024-04-29T14:22:06.420 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=3843852346,monmap=455069592,osd_metadata=401180102,osd_pg_creating=469659463,osdmap=2065198245}) 2024-04-29T14:22:06.420 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1638942680}) 2024-04-29T14:22:06.420 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=831353484}) 2024-04-29T14:22:06.510 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:06.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:06.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: pgmap v1020: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: osdmap e385: 12 total, 11 up, 8 in 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=53,logm=1} crc {auth=2995697159,config=359415709,health=1091798053,kv=3585853274,logm=1404913468}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530826868}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4249369625}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454928464}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1728110384}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707541192}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517201456}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950468761}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539755239}) 2024-04-29T14:22:06.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776364308}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460100659}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419840721}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4090106845}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567956863}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219923055}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178331337}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007707767}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833277447}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=141091887}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091381347}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129964790}) 2024-04-29T14:22:06.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940079484}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899389752}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169875580}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=320946690}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597725171}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205505587}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241366737}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224727425}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1013704270}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2108061596}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2374596705}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518779019}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2865675883}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210713602}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156358688}) 2024-04-29T14:22:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161595278}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3688096762}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783940886}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384435165}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1613531942}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825705797}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265803210}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916183333}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1896805587}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430439930}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309889895}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=111764570}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4163107622}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=2} crc {logm=2942141979,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=608560165,mon_config_key=1964985202}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2320865557}) 2024-04-29T14:22:06.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1030335798}) 2024-04-29T14:22:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=3843852346,monmap=455069592,osd_metadata=401180102,osd_pg_creating=469659463,osdmap=2065198245}) 2024-04-29T14:22:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1638942680}) 2024-04-29T14:22:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=831353484}) 2024-04-29T14:22:06.617 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:06.646 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:06.770 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:22:06.771 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:22:06.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:22:06.771 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [] 2024-04-29T14:22:06.771 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:22:06.794 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:06.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:06.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:06.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:22:06.927 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:07.000 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:07.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:07.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:07.135 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:07.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:07.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:07.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:22:07.397 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:07.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:22:07.597 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:07.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:07.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:07.705 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:07.730 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:07.748 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:07.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:07.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:07.858 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:22:07.874 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:07 smithi121 bash[115150]: 35f5b04112f83680832844491bbbb6fb8a4f824649d1c9f17103d0cc157a35de 2024-04-29T14:22:07.874 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:07 smithi121 systemd[1]: Started Ceph osd.5 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:22:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:08.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:08.132 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:08 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[115165]: 2024-04-29T14:22:08.037+0000 7f0b6f6c87c0 -1 Falling back to public interface 2024-04-29T14:22:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:08.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:08.140 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:22:08.234 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:08.261 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:08 smithi040 ceph-mon[39581]: pgmap v1022: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:08.261 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:08.261 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:08.261 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/283893071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:08.263 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:08.283 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:08.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:08.387 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:08.413 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:08 smithi194 ceph-mon[44089]: pgmap v1022: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:08.413 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:08.413 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:08.413 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/283893071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:08.415 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:08.434 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:08.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:08.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:08.569 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:08.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:08 smithi121 ceph-mon[44533]: pgmap v1022: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:08.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:08.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:08.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/283893071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:09 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/283893071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:09 smithi040 ceph-mon[39581]: osdmap e386: 12 total, 11 up, 8 in 2024-04-29T14:22:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:09 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/283893071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:09 smithi121 ceph-mon[44533]: osdmap e386: 12 total, 11 up, 8 in 2024-04-29T14:22:09.419 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:09.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:09.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:09.558 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:09 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/283893071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:09 smithi194 ceph-mon[44089]: osdmap e386: 12 total, 11 up, 8 in 2024-04-29T14:22:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:10 smithi040 ceph-mon[39581]: pgmap v1024: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:10 smithi040 ceph-mon[39581]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-04-29T14:22:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:10 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-29T14:22:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:10 smithi194 ceph-mon[44089]: pgmap v1024: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:10 smithi194 ceph-mon[44089]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-04-29T14:22:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:10 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-29T14:22:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:10 smithi121 ceph-mon[44533]: pgmap v1024: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:10 smithi121 ceph-mon[44533]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-04-29T14:22:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:10 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-29T14:22:11.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:11.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:11.112 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:11.145 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:11.167 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:11.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:11.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:11.300 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:11 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[115165]: 2024-04-29T14:22:11.132+0000 7f0b6f6c87c0 -1 osd.5 369 log_to_monitors true 2024-04-29T14:22:11.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:11 smithi121 ceph-mon[44533]: from='osd.5 v1:172.21.15.121:6804/2586630078' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:22:11.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:11 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:22:11.303 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:11.402 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:no valid command found; 10 closest matches: 2024-04-29T14:22:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:0 2024-04-29T14:22:11.404 INFO:teuthology.orchestra.run.smithi121.stderr:1 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:2 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:abort 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:assert 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:bluefs debug_inject_read_zeros 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:bluefs files list 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:bluefs stats 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:bluestore allocator dump block 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:bluestore allocator fragmentation block 2024-04-29T14:22:11.405 INFO:teuthology.orchestra.run.smithi121.stderr:admin_socket: invalid command 2024-04-29T14:22:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:11.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:11.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:11.511 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:22:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:22:11.511 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:22:11.511 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:22:11.515 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:11 smithi040 ceph-mon[39581]: from='osd.5 v1:172.21.15.121:6804/2586630078' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:22:11.515 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:11 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:22:11.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:11 smithi194 ceph-mon[44089]: from='osd.5 v1:172.21.15.121:6804/2586630078' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:22:11.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:11 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:22:11.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:11.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:22:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:22:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:22:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:22:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:22:11.659 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:22:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:22:11.700 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:22:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:22:11.740 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:22:11.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:11.761 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:22:11.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:22:11.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:22:11.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:22:11.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:22:11.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:22:11.761 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:22:11.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:22:11.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:22:11.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:22:11.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:22:11.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:22:11.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:22:11.769 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:11.874 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:22:11 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[115165]: 2024-04-29T14:22:11.594+0000 7f0b5cb18700 -1 osd.5 369 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:22:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:22:12.107 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-04-29T14:22:12.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:12.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:12.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:12.249 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:12.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:12.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:12.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:12.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: pgmap v1025: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:12.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:22:12.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osdmap e387: 12 total, 11 up, 8 in 2024-04-29T14:22:12.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: from='osd.5 v1:172.21.15.121:6804/2586630078' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2339319822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.400 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: pgmap v1025: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:12.400 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:22:12.400 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osdmap e387: 12 total, 11 up, 8 in 2024-04-29T14:22:12.400 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: from='osd.5 v1:172.21.15.121:6804/2586630078' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:22:12.400 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2339319822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.402 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:12.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: pgmap v1025: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:12.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:22:12.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osdmap e387: 12 total, 11 up, 8 in 2024-04-29T14:22:12.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: from='osd.5 v1:172.21.15.121:6804/2586630078' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2339319822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:12.423 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:12.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:22:12.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:22:12.466 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:22:12.527 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:12.556 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:12.576 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:12.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:12.680 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:12.709 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:12.729 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:12.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:12.833 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:12.860 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:12.879 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:12.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:12.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:22:12.982 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:12.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:13.014 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:13.262 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:13.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:13.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:22:13.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: osd.5 v1:172.21.15.121:6804/2586630078 boot 2024-04-29T14:22:13.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: osdmap e388: 12 total, 12 up, 8 in 2024-04-29T14:22:13.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:22:13.398 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:13.420 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:13.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:13.476 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.476 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:22:13.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: osd.5 v1:172.21.15.121:6804/2586630078 boot 2024-04-29T14:22:13.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: osdmap e388: 12 total, 12 up, 8 in 2024-04-29T14:22:13.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:22:13.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:13.552 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: osd.5 v1:172.21.15.121:6804/2586630078 boot 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: osdmap e388: 12 total, 12 up, 8 in 2024-04-29T14:22:13.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:22:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:14.444 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:22:14.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:14.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:14.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:14 smithi040 ceph-mon[39581]: pgmap v1028: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:14.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:14 smithi040 ceph-mon[39581]: osdmap e389: 12 total, 12 up, 8 in 2024-04-29T14:22:14.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:14.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:14.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:14 smithi194 ceph-mon[44089]: pgmap v1028: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:14.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:14 smithi194 ceph-mon[44089]: osdmap e389: 12 total, 12 up, 8 in 2024-04-29T14:22:14.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:14.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:14.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:14 smithi121 ceph-mon[44533]: pgmap v1028: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:14.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:14 smithi121 ceph-mon[44533]: osdmap e389: 12 total, 12 up, 8 in 2024-04-29T14:22:14.802 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:14.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:14.908 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:14.956 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:14.988 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:14.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:15.094 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:15.144 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:15.171 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:15.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:15.281 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:15.345 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:15.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:15.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:15.481 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:15.502 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:15.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:15.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:15.634 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:15.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:15.765 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:15.805 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:15.832 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:15.838 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:15.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:15.987 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:16.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:16.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:16.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:16.165 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:22:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:16.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:16.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:16.281 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:16 smithi040 ceph-mon[39581]: pgmap v1030: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:16.281 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/536348716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:22:16.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:16.337 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:16 smithi121 ceph-mon[44533]: pgmap v1030: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/536348716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:22:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:16 smithi194 ceph-mon[44089]: pgmap v1030: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 7.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 14336/230300 objects degraded (6.225%) 2024-04-29T14:22:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/536348716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:22:17.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:22:17.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:17 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/536348716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:22:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:17 smithi040 ceph-mon[39581]: osdmap e390: 12 total, 12 up, 8 in 2024-04-29T14:22:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:17.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:17.555 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:22:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:17 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/536348716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:22:17.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:17 smithi194 ceph-mon[44089]: osdmap e390: 12 total, 12 up, 8 in 2024-04-29T14:22:17.594 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:17.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:17 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:17.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/536348716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:22:17.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:17 smithi121 ceph-mon[44533]: osdmap e390: 12 total, 12 up, 8 in 2024-04-29T14:22:17.662 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:17.695 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:17.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:17.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:17.851 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:17.878 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:17.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:17.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:18.015 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:18 smithi040 ceph-mon[39581]: pgmap v1032: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 360 op/s 2024-04-29T14:22:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:18 smithi040 ceph-mon[39581]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-04-29T14:22:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:18 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded, 2 pgs undersized) 2024-04-29T14:22:18.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:18 smithi040 ceph-mon[39581]: 4.2 scrub starts 2024-04-29T14:22:18.433 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:22:18.433 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:22:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:18 smithi194 ceph-mon[44089]: pgmap v1032: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 360 op/s 2024-04-29T14:22:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:18 smithi194 ceph-mon[44089]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-04-29T14:22:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:18 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded, 2 pgs undersized) 2024-04-29T14:22:18.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:18 smithi194 ceph-mon[44089]: 4.2 scrub starts 2024-04-29T14:22:18.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:18 smithi121 ceph-mon[44533]: pgmap v1032: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 360 op/s 2024-04-29T14:22:18.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:18 smithi121 ceph-mon[44533]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-04-29T14:22:18.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:18 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14336/230300 objects degraded (6.225%), 2 pgs degraded, 2 pgs undersized) 2024-04-29T14:22:18.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:18 smithi121 ceph-mon[44533]: 4.2 scrub starts 2024-04-29T14:22:19.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:22:19.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:19 smithi040 ceph-mon[39581]: 4.2 scrub ok 2024-04-29T14:22:19.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:19 smithi040 ceph-mon[39581]: 4.1 scrub starts 2024-04-29T14:22:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:19 smithi194 ceph-mon[44089]: 4.2 scrub ok 2024-04-29T14:22:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:19 smithi194 ceph-mon[44089]: 4.1 scrub starts 2024-04-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:22:19.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:19 smithi121 ceph-mon[44533]: 4.2 scrub ok 2024-04-29T14:22:19.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:19 smithi121 ceph-mon[44533]: 4.1 scrub starts 2024-04-29T14:22:19.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:22:19.966 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:19.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:20.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:20.100 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:20.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:20.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:20.278 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:22:20.285 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:20 smithi121 ceph-mon[44533]: pgmap v1033: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 416 op/s 2024-04-29T14:22:20.289 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:20.299 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:20.303 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:22:20.335 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:22:20.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:20.356 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:22:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:20.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:20.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:20 smithi040 ceph-mon[39581]: pgmap v1033: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 416 op/s 2024-04-29T14:22:20.387 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:22:20.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:22:20.455 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:22:20.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:20.480 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:22:20.501 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:20.502 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:22:20.525 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:22:20.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:20.547 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:22:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:20.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:20 smithi194 ceph-mon[44089]: pgmap v1033: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 416 op/s 2024-04-29T14:22:20.569 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:22:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:22:20.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:20.686 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:20.715 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:20.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:20.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:20.867 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:20.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:20.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:21.007 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:21.038 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:21.239 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:21.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:21.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:21.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2624499546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:21.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: 4.1 scrub ok 2024-04-29T14:22:21.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2624499546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: 4.1 scrub ok 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2624499546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: 4.1 scrub ok 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:21.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:22.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:22.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:22 smithi040 ceph-mon[39581]: pgmap v1034: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 574 op/s 2024-04-29T14:22:22.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:22.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:22.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:22.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:22.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:22 smithi194 ceph-mon[44089]: pgmap v1034: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 574 op/s 2024-04-29T14:22:22.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:22.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:22.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:22 smithi121 ceph-mon[44533]: pgmap v1034: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 574 op/s 2024-04-29T14:22:22.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:22.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:22.585 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:22.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:22.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:22.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:22.765 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:22.793 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:22.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:22.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:22.940 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:22.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:23.085 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:23.118 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:23.147 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:23.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:23.193 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:23 smithi121 ceph-mon[44533]: osdmap e391: 12 total, 12 up, 8 in 2024-04-29T14:22:23.193 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3468963487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:23.196 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:22:23.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:22:23.343 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:23.371 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:22:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:23.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:22:23.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:23.494 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:23 smithi040 ceph-mon[39581]: osdmap e391: 12 total, 12 up, 8 in 2024-04-29T14:22:23.494 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3468963487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:23.515 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:23.545 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:23.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:23.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:23 smithi194 ceph-mon[44089]: osdmap e391: 12 total, 12 up, 8 in 2024-04-29T14:22:23.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3468963487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:23.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:23.681 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:23.710 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:23.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:23.814 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:23.852 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:23.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:23.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:24.022 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:24.172 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:24.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:24.195 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:24 smithi194 ceph-mon[44089]: pgmap v1036: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 993 op/s 2024-04-29T14:22:24.195 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:24 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:22:24.195 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3468963487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:24.195 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:24 smithi194 ceph-mon[44089]: osdmap e392: 12 total, 12 up, 8 in 2024-04-29T14:22:24.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:24.327 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:24.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:24.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:24.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:24.481 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:24 smithi040 ceph-mon[39581]: pgmap v1036: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 993 op/s 2024-04-29T14:22:24.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:24 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:22:24.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3468963487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:24.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:24 smithi040 ceph-mon[39581]: osdmap e392: 12 total, 12 up, 8 in 2024-04-29T14:22:24.484 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:24.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:24 smithi121 ceph-mon[44533]: pgmap v1036: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 993 op/s 2024-04-29T14:22:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:24 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:22:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3468963487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:24.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:24 smithi121 ceph-mon[44533]: osdmap e392: 12 total, 12 up, 8 in 2024-04-29T14:22:25.034 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:25.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:25.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:25.180 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:25.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:25.297 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:22:25.297 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25063.0:164487 3.b 3:dc72fc9d:::benchmark_data_smithi194_88551_object5544:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e254)", 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:15:40.054845+0000", 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 405.24797186199999, 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21303364699999999, 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25063", 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164487 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054845+0000", 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054845+0000", 2024-04-29T14:22:25.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054848+0000", 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7709999999999998e-06 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054870+0000", 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2447999999999999e-05 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054884+0000", 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3417999999999999e-05 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.054895+0000", 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1171e-05 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.055012+0000", 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117259 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:22:25.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.055026+0000", 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3413e-05 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.265999+0000", 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21097299999999999 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:15:40.267879+0000", 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018801670000000001 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.302 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:167486 4.b 4:d9d0eddb:::benchmark_data_smithi194_103039_object10803:head [delete] snapc 0=[] ondisk+write+known_if_redirected e361)", 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:38.236864+0000", 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 107.06595272200001, 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030726046999999999, 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 167486 2024-04-29T14:22:25.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236864+0000", 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236864+0000", 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236869+0000", 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.459e-06 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236883+0000", 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4581e-05 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236911+0000", 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7821999999999999e-05 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236926+0000", 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4476e-05 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.237013+0000", 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6828000000000003e-05 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.237180+0000", 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016765500000000001 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.237331+0000", 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015088300000000001 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.241733+0000", 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044023939999999996 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.267517+0000", 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025783897 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.267556+0000", 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8210999999999997e-05 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.267590+0000", 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4841000000000001e-05 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:170115 4.b 4:d8cb76b9:::benchmark_data_smithi194_103039_object13432:head [delete] snapc 0=[] ondisk+write+known_if_redirected e361)", 2024-04-29T14:22:25.308 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:40.707663+0000", 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 104.595153803, 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030995383000000001, 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 170115 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707663+0000", 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707663+0000", 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707664+0000", 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.08e-06 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707669+0000", 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7840000000000003e-06 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707677+0000", 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2099999999999993e-06 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707685+0000", 2024-04-29T14:22:25.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5399999999999998e-06 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.715617+0000", 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0079318849999999996 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.715721+0000", 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104033 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.715815+0000", 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4254999999999998e-05 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.726361+0000", 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010545974 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.738596+0000", 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012235108999999999 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.738629+0000", 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2626999999999998e-05 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.738659+0000", 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9886000000000002e-05 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181319 4.b 4:daa64077:::benchmark_data_smithi194_103039_object24636:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:22:25.314 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.530760+0000", 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.772057547000003, 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030546224, 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181319 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530760+0000", 2024-04-29T14:22:25.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530760+0000", 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530761+0000", 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7370000000000001e-06 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530766+0000", 2024-04-29T14:22:25.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6820000000000002e-06 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530776+0000", 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0054000000000001e-05 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530782+0000", 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2500000000000003e-06 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530822+0000", 2024-04-29T14:22:25.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9823000000000001e-05 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530903+0000", 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0618999999999995e-05 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530973+0000", 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0166000000000005e-05 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540450+0000", 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094768140000000001 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561164+0000", 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020713850999999998 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561249+0000", 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5235999999999995e-05 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561306+0000", 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6991999999999999e-05 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181322 4.b 4:d4feca94:::benchmark_data_smithi194_103039_object24639:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.772024414000001, 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030675788999999998, 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.320 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181322 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.39e-07 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530795+0000", 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7969999999999999e-06 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530798+0000", 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3529999999999999e-06 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530803+0000", 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8709999999999996e-06 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.533825+0000", 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030218530000000001 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.533896+0000", 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0837e-05 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.533977+0000", 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0865000000000006e-05 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540455+0000", 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.006478765 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561367+0000", 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020911619999999999 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561424+0000", 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7213999999999998e-05 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561468+0000", 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4175000000000003e-05 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.325 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181323 4.b 4:d92ad292:::benchmark_data_smithi194_103039_object24640:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.772009175999997, 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030732281, 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181323 2024-04-29T14:22:25.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0100000000000001e-07 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530810+0000", 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.545e-06 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530813+0000", 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8059999999999999e-06 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530815+0000", 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8870000000000001e-06 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.536080+0000", 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0052646100000000003 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.536144+0000", 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4354000000000001e-05 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.536218+0000", 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3238000000000006e-05 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540460+0000", 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042421070000000002 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561494+0000", 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021033759999999999 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561520+0000", 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6631e-05 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561540+0000", 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0041999999999999e-05 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181330 4.b 4:de33ed73:::benchmark_data_smithi194_103039_object24647:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:22:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.767177656000001, 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039659663999999997, 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181330 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535641+0000", 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.248e-06 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535645+0000", 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4429999999999996e-06 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535655+0000", 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0348e-05 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535662+0000", 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0970000000000002e-06 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.537976+0000", 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023144060000000002 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.538102+0000", 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012558 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.538229+0000", 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126928 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540630+0000", 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024012769999999998 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575087+0000", 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034457152999999997 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575253+0000", 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000166441 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575299+0000", 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5742999999999999e-05 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:182054 4.b 4:db2ee909:::benchmark_data_smithi194_103039_object25371:head [delete] snapc 0=[] ondisk+write+known_if_redirected e365)", 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:52.229066+0000", 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.073750818999997, 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030512937, 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.337 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 182054 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229066+0000", 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229066+0000", 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229068+0000", 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4449999999999999e-06 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229073+0000", 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7720000000000002e-06 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229082+0000", 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2839999999999995e-06 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229088+0000", 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8939999999999998e-06 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229134+0000", 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6604999999999998e-05 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229247+0000", 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011307900000000001 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229356+0000", 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108612 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.241626+0000", 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012269535 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259495+0000", 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017869511000000001 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259529+0000", 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3865000000000002e-05 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259579+0000", 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0334999999999998e-05 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:182056 4.b 4:d06152d6:::benchmark_data_smithi194_103039_object25373:head [delete] snapc 0=[] ondisk+write+known_if_redirected e365)", 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:52.229123+0000", 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.073693894000002, 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030576231999999998, 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 182056 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229123+0000", 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229123+0000", 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229124+0000", 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4099999999999997e-07 2024-04-29T14:22:25.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229127+0000", 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1480000000000002e-06 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229133+0000", 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2299999999999996e-06 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229137+0000", 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9670000000000003e-06 2024-04-29T14:22:25.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.233218+0000", 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0040804880000000002 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.233302+0000", 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3850000000000005e-05 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.233374+0000", 2024-04-29T14:22:25.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2560999999999999e-05 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.241628+0000", 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082541520000000007 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259631+0000", 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018003008000000001 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259662+0000", 2024-04-29T14:22:25.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0796000000000001e-05 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259699+0000", 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7190999999999997e-05 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:182057 4.b 4:d01586c9:::benchmark_data_smithi194_103039_object25374:head [delete] snapc 0=[] ondisk+write+known_if_redirected e365)", 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:22:25.348 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.073662475000006, 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030692338999999999, 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 182057 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2200000000000004e-07 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229158+0000", 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.249e-06 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229162+0000", 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0090000000000001e-06 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229165+0000", 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3119999999999998e-06 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.235695+0000", 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065303940000000001 2024-04-29T14:22:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.235766+0000", 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0659000000000002e-05 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.235842+0000", 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5726e-05 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.241631+0000", 2024-04-29T14:22:25.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057892839999999996 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259763+0000", 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018132514999999998 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259810+0000", 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6683e-05 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259847+0000", 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6885999999999999e-05 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187965 4.b 4:d7e0b0fd:::benchmark_data_smithi194_103039_object31282:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 87.481459994999994, 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045912334999999999, 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.356 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187965 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821359+0000", 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5039999999999999e-06 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821363+0000", 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4939999999999997e-06 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821373+0000", 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0001e-05 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821380+0000", 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9589999999999998e-06 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821426+0000", 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5896999999999998e-05 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821500+0000", 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3790999999999999e-05 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821568+0000", 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8112000000000001e-05 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.823920+0000", 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023521409999999999 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867185+0000", 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043265190000000002 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867231+0000", 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5726999999999998e-05 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867269+0000", 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8519000000000002e-05 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.361 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187971 4.b 4:d0a42069:::benchmark_data_smithi194_103039_object31288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 87.471328893000006, 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035888784999999999, 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187971 2024-04-29T14:22:25.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831492+0000", 2024-04-29T14:22:25.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3139999999999999e-06 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831502+0000", 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0713000000000001e-05 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831529+0000", 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6893000000000002e-05 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831548+0000", 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8697000000000001e-05 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831712+0000", 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016401100000000001 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831816+0000", 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104293 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831888+0000", 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1792000000000007e-05 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836180+0000", 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004292293 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867316+0000", 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031135787000000002 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867351+0000", 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4573999999999998e-05 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867377+0000", 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6418e-05 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187972 4.b 4:d50db589:::benchmark_data_smithi194_103039_object31289:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:22:25.367 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 87.47124298, 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035895613999999999, 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187972 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831575+0000", 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.24e-06 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831580+0000", 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1359999999999999e-06 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831589+0000", 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5740000000000002e-06 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831602+0000", 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2585999999999999e-05 2024-04-29T14:22:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834234+0000", 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002631839 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834303+0000", 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9339999999999997e-05 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834370+0000", 2024-04-29T14:22:25.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.703e-05 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.838064+0000", 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036939109999999998 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867419+0000", 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029355322999999999 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867447+0000", 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8164999999999999e-05 2024-04-29T14:22:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867470+0000", 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.247e-05 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187977 4.b 4:d9b4303c:::benchmark_data_smithi194_103039_object31294:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.834981+0000", 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 87.467835772000001, 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032568039, 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.373 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187977 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834981+0000", 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834981+0000", 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834983+0000", 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3999999999999999e-06 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834987+0000", 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5109999999999997e-06 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834998+0000", 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0461e-05 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.835010+0000", 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2283000000000001e-05 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836200+0000", 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001189697 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836266+0000", 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6630000000000004e-05 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836331+0000", 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4391000000000001e-05 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.838220+0000", 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018889740000000001 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867501+0000", 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029281411 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867529+0000", 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7426000000000001e-05 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867549+0000", 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0854999999999999e-05 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.378 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:195494 4.b 4:d756e1a7:::benchmark_data_smithi194_103039_object38811:head [delete] snapc 0=[] ondisk+write+known_if_redirected e368)", 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 80.521000963000006, 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039141214000000001, 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 195494 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.379 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781818+0000", 2024-04-29T14:22:25.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.916e-06 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781824+0000", 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5049999999999998e-06 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781834+0000", 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0494e-05 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781842+0000", 2024-04-29T14:22:25.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0260000000000007e-06 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781908+0000", 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5474000000000004e-05 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.782012+0000", 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104421 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.782099+0000", 2024-04-29T14:22:25.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7496000000000004e-05 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.785711+0000", 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036118610000000001 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820846+0000", 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035134336000000002 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820907+0000", 2024-04-29T14:22:25.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1050999999999995e-05 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820957+0000", 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0633999999999997e-05 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:197308 4.b 4:d0f24d33:::benchmark_data_smithi194_103039_object40625:head [delete] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:21:06.497279+0000", 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.805538545999994, 2024-04-29T14:22:25.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034682491000000003, 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 197308 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497279+0000", 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497279+0000", 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497282+0000", 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.473e-06 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497293+0000", 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0947e-05 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497318+0000", 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5439000000000002e-05 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497333+0000", 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4732999999999999e-05 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497474+0000", 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014126899999999999 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497671+0000", 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019645700000000001 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497822+0000", 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015072999999999999 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.501363+0000", 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035410039999999999 2024-04-29T14:22:25.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.531859+0000", 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030496743 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.531914+0000", 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4864999999999997e-05 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.531961+0000", 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6830999999999999e-05 2024-04-29T14:22:25.389 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:197940 4.b 4:d4c58e28:::benchmark_data_smithi194_103039_object41257:head [delete] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.227064510999995, 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033304070999999998, 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.390 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 197940 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9800000000000003e-07 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075756+0000", 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1580000000000001e-06 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075766+0000", 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2429999999999994e-06 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075770+0000", 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1189999999999999e-06 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.080827+0000", 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050571269999999998 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.080906+0000", 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8968999999999996e-05 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.080977+0000", 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0797999999999995e-05 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.083109+0000", 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021327780000000001 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.108949+0000", 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.02583999 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.109007+0000", 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7181e-05 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.109057+0000", 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0009999999999997e-05 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205500 4.b 4:dcf18273:::benchmark_data_smithi194_103039_object48817:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:22:25.395 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.9371828500000001, 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038420901, 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205500 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365635+0000", 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8099999999999998e-07 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365637+0000", 2024-04-29T14:22:25.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.889e-06 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365640+0000", 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.619e-06 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365644+0000", 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4359999999999998e-06 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372228+0000", 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065845260000000003 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372328+0000", 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100299 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372423+0000", 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4629999999999994e-05 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.383904+0000", 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011481358000000001 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.403987+0000", 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020082472000000001 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404022+0000", 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4737999999999998e-05 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404055+0000", 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3553000000000002e-05 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205501 4.b 4:d60cc47a:::benchmark_data_smithi194_103039_object48818:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.937169226, 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038509798999999997, 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205501 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9999999999999999e-07 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365650+0000", 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.545e-06 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365653+0000", 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.456e-06 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365656+0000", 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9390000000000002e-06 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.375824+0000", 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010167756 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.375966+0000", 2024-04-29T14:22:25.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014242200000000001 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.376106+0000", 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140068 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.387047+0000", 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01094083 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404096+0000", 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01704864 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404128+0000", 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2452e-05 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404158+0000", 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9391e-05 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.406 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205503 4.b 4:d23be51c:::benchmark_data_smithi194_103039_object48820:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.9356178740000001, 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037041931, 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205503 2024-04-29T14:22:25.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367200+0000", 2024-04-29T14:22:25.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0580000000000001e-06 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367205+0000", 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6e-06 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367214+0000", 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5249999999999994e-06 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367220+0000", 2024-04-29T14:22:25.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6200000000000004e-06 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380152+0000", 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012932018 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380291+0000", 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013889600000000001 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380429+0000", 2024-04-29T14:22:25.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000138541 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.390770+0000", 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010340066 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404193+0000", 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013423403 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404221+0000", 2024-04-29T14:22:25.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7872e-05 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404241+0000", 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0332000000000001e-05 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:25.412 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:22:25.412 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:25.413 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:25.413 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:22:25.413 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:22:25.413 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-04-29T14:22:25.413 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:22:25.413 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:22:25.413 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-04-29T14:22:25.414 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:25.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:25.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:25.434 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:25.434 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:25.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:25.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:25.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:22:25.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:25.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:25.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:25.749 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:25.814 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:22:26.137 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:22:26.199 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:26.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:26.305 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:26.351 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:26.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:26 smithi040 ceph-mon[39581]: pgmap v1038: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 736 op/s 2024-04-29T14:22:26.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:26 smithi040 ceph-mon[39581]: osdmap e393: 12 total, 12 up, 8 in 2024-04-29T14:22:26.380 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:26.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:26.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:26.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:26.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:26 smithi121 ceph-mon[44533]: pgmap v1038: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 736 op/s 2024-04-29T14:22:26.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:26 smithi121 ceph-mon[44533]: osdmap e393: 12 total, 12 up, 8 in 2024-04-29T14:22:26.529 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:26.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:26 smithi194 ceph-mon[44089]: pgmap v1038: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 736 op/s 2024-04-29T14:22:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:26 smithi194 ceph-mon[44089]: osdmap e393: 12 total, 12 up, 8 in 2024-04-29T14:22:26.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:26.792 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:26.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:26.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:22:26.896 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:27.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:27.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:27.146 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:27.183 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:27.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:27.280 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:27.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:27.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:27.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:27 smithi194 ceph-mon[44089]: osdmap e394: 12 total, 12 up, 8 in 2024-04-29T14:22:27.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:27.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:27.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:27.470 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:27.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:27.503 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:27.504 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:27.504 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:27 smithi121 ceph-mon[44533]: osdmap e394: 12 total, 12 up, 8 in 2024-04-29T14:22:27.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:27.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:27.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:27 smithi040 ceph-mon[39581]: osdmap e394: 12 total, 12 up, 8 in 2024-04-29T14:22:27.604 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:27.686 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:27.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:27.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:27.857 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:27.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:27.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:27.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:28.030 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:22:28.030 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-29T14:22:28.030 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:22:28.057 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:28.084 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:28.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:28.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:28.237 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:28.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:28.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:28.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:28 smithi194 ceph-mon[44089]: pgmap v1041: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:22:28.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:28.474 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:28.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:28.481 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:28 smithi040 ceph-mon[39581]: pgmap v1041: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:22:28.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:28.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:28 smithi121 ceph-mon[44533]: pgmap v1041: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-29T14:22:28.642 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:28.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:28.791 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:28.841 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:28.891 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:28.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:28.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:29.053 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:29.086 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:29.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:29.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:29.223 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:29.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:29 smithi040 ceph-mon[39581]: pgmap v1042: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-04-29T14:22:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:29.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:29 smithi121 ceph-mon[44533]: pgmap v1042: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-04-29T14:22:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:29.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:29.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:29 smithi194 ceph-mon[44089]: pgmap v1042: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-04-29T14:22:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:29.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:29.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:29.709 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:29.764 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:29.787 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:29.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:29.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:29.927 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:30.126 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:30.127 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:22:30.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:30.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:30.151 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:22:30.170 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:22:30.193 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:22:30.217 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:22:30.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:30.244 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:22:30.266 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:30.281 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:22:30.311 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:22:30.337 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:22:30.356 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:22:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:30.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:30.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:22:30.412 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:22:30.413 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:22:30.414 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:22:30.414 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:22:30.414 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:22:30.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:30.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:30 smithi121 ceph-mon[44533]: osdmap e395: 12 total, 12 up, 8 in 2024-04-29T14:22:30.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2889525190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:30.540 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:30.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:30 smithi040 ceph-mon[39581]: osdmap e395: 12 total, 12 up, 8 in 2024-04-29T14:22:30.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2889525190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:30 smithi194 ceph-mon[44089]: osdmap e395: 12 total, 12 up, 8 in 2024-04-29T14:22:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2889525190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:30.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:30.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:30.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:31.078 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:22:31.079 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:22:31.275 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:31.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:31.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:31.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:31.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:31 smithi194 ceph-mon[44089]: pgmap v1044: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-29T14:22:31.579 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:31 smithi040 ceph-mon[39581]: pgmap v1044: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-29T14:22:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:31 smithi121 ceph-mon[44533]: pgmap v1044: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s 2024-04-29T14:22:31.627 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:31.655 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:31.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:31.767 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:31.807 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:31.835 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:31.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:31.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:31.968 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:31.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:32.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:32.128 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:32.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:32.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:32.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:32.309 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:32.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:32.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:32.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:32.481 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:32.504 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:32.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:32.609 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:32.653 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:32.678 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:32.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:32.783 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:32.822 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:32.851 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:32.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:32.920 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3705056281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:22:32.923 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:22:32.955 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:33.000 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:33.032 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:33.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:33.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:22:33.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:33.171 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3705056281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:22:33.174 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:33.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3705056281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:22:33.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:33.406 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:33.462 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:33.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:33.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:33.604 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:33.817 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:22:33.817 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:22:33.817 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-29T14:22:33.817 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-04-29T14:22:33.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 0 0 2024-04-29T14:22:33.853 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:33.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:33.869 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:33.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:33.994 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:34.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:22:34.020 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:34.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:34.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:33 smithi040 ceph-mon[39581]: pgmap v1045: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-29T14:22:34.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:33 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:34.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3705056281' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:22:34.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:33 smithi040 ceph-mon[39581]: osdmap e396: 12 total, 12 up, 8 in 2024-04-29T14:22:34.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:34.159 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:34.193 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:34.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:34.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:34.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:33 smithi194 ceph-mon[44089]: pgmap v1045: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-29T14:22:34.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:33 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:34.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3705056281' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:22:34.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:33 smithi194 ceph-mon[44089]: osdmap e396: 12 total, 12 up, 8 in 2024-04-29T14:22:34.326 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:34.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:33 smithi121 ceph-mon[44533]: pgmap v1045: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-29T14:22:34.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:33 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:34.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3705056281' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:22:34.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:33 smithi121 ceph-mon[44533]: osdmap e396: 12 total, 12 up, 8 in 2024-04-29T14:22:34.776 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:34.778 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:34.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:34.879 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:34.929 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:34.961 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:34.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:35.069 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:35.102 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:35.126 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:35.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:35.233 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:35.386 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:35.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:35.408 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:35.490 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:35.536 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:35.575 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:35.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:35.685 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:35.725 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:35.751 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:35.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:35.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:35.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:35.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:22:35.890 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:35.945 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:35.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:35.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:35 smithi040 ceph-mon[39581]: pgmap v1047: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 779 op/s 2024-04-29T14:22:36.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:36.088 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:36.125 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:36.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:36.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:36.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:35 smithi194 ceph-mon[44089]: pgmap v1047: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 779 op/s 2024-04-29T14:22:36.261 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:36.284 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:36.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:36.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:35 smithi121 ceph-mon[44533]: pgmap v1047: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 779 op/s 2024-04-29T14:22:36.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:36.420 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:37.273 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:37.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:37.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:37.409 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:37.961 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:37.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:38.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:38.109 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:38.143 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:38.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:38.249 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:38.281 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:38.311 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:38.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:38.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:38 smithi040 ceph-mon[39581]: pgmap v1048: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:22:38.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:38 smithi121 ceph-mon[44533]: pgmap v1048: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:22:38.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:38.453 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:38.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:38 smithi194 ceph-mon[44089]: pgmap v1048: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:22:38.481 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:38.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:38.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:38.618 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:38.759 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:22:38.798 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:22:38.820 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:22:38.847 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:22:38.874 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:22:38.899 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:22:38.923 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:22:38.949 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:22:38.968 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:22:38.974 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:38.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:38.993 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:22:39.017 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:22:39.018 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:22:39.018 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:22:39.018 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:22:39.018 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:22:39.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:22:39.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:39.118 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:39.145 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:39.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:39.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:39.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1206477246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:39.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:39.283 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1206477246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1206477246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:39.642 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:22:39.642 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:22:40.123 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.0 primary-affinity to 0 (802) 2024-04-29T14:22:40.506 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.506 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: pgmap v1049: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1312981976' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: pgmap v1049: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1312981976' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-29T14:22:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: pgmap v1049: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1312981976' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-29T14:22:41.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1312981976' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-04-29T14:22:41.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:41 smithi040 ceph-mon[39581]: osdmap e397: 12 total, 12 up, 8 in 2024-04-29T14:22:41.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:22:41.469 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:41.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:41.493 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1312981976' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-04-29T14:22:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:41 smithi194 ceph-mon[44089]: osdmap e397: 12 total, 12 up, 8 in 2024-04-29T14:22:41.574 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:41.614 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1312981976' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-04-29T14:22:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:41 smithi121 ceph-mon[44533]: osdmap e397: 12 total, 12 up, 8 in 2024-04-29T14:22:41.647 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:41.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:41.754 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:41.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:42.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:42.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:42.291 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:42.334 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:42.361 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:42 smithi040 ceph-mon[39581]: pgmap v1051: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-04-29T14:22:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:42 smithi040 ceph-mon[39581]: osdmap e398: 12 total, 12 up, 8 in 2024-04-29T14:22:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:42.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:42.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:22:42.521 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:42.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:42 smithi121 ceph-mon[44533]: pgmap v1051: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-04-29T14:22:42.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:42 smithi121 ceph-mon[44533]: osdmap e398: 12 total, 12 up, 8 in 2024-04-29T14:22:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:42 smithi194 ceph-mon[44089]: pgmap v1051: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-04-29T14:22:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:42 smithi194 ceph-mon[44089]: osdmap e398: 12 total, 12 up, 8 in 2024-04-29T14:22:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:22:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:42.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:22:42.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:42.705 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:42.736 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:42.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:42.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:42.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:22:43.322 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:43.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:43.405 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:43.406 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:43.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:43.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:43.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:43.829 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:43.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:43.933 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:43.970 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:44.000 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:44.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:44.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:44.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:44.263 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:44 smithi040 ceph-mon[39581]: pgmap v1053: 17 pgs: 1 peering, 16 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-29T14:22:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:44 smithi194 ceph-mon[44089]: pgmap v1053: 17 pgs: 1 peering, 16 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-29T14:22:44.590 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:44.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:44.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:44 smithi121 ceph-mon[44533]: pgmap v1053: 17 pgs: 1 peering, 16 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-29T14:22:44.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:22:44.737 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:22:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:44.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:22:44.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:44.939 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:44.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:45.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:45.089 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:45.116 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:45.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:45.163 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:22:45.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:45.288 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:45.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:45.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4267804951' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:45.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:45.425 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:45.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4267804951' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4267804951' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:22:45.775 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:45.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:45.879 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:45.930 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:45.958 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:45.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:46.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:46.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:46.253 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:46.286 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:46 smithi194 ceph-mon[44089]: pgmap v1054: 17 pgs: 1 peering, 16 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 699 op/s 2024-04-29T14:22:46.287 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:46 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:46.287 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4267804951' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:46.287 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:46 smithi194 ceph-mon[44089]: osdmap e399: 12 total, 12 up, 8 in 2024-04-29T14:22:46.287 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:46.333 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:46.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:46.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:46 smithi040 ceph-mon[39581]: pgmap v1054: 17 pgs: 1 peering, 16 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 699 op/s 2024-04-29T14:22:46.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:46 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:46.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4267804951' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:46.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:46 smithi040 ceph-mon[39581]: osdmap e399: 12 total, 12 up, 8 in 2024-04-29T14:22:46.364 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:22:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:22:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:22:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:22:46.364 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:22:46.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:46 smithi121 ceph-mon[44533]: pgmap v1054: 17 pgs: 1 peering, 16 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 699 op/s 2024-04-29T14:22:46.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:46 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:46.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4267804951' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:22:46.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:46 smithi121 ceph-mon[44533]: osdmap e399: 12 total, 12 up, 8 in 2024-04-29T14:22:46.437 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:46.471 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:46.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:46.700 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:46.740 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:46.772 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:46.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:46.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:46.935 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:46.955 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:22:46.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:22:46.955 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:22:46.956 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:22:46.978 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:46.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:47.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:22:47.154 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:47.192 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:22:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:47.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:22:47.296 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:47.339 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:47.368 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:47.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:47.478 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:47.707 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:47.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:47.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:22:47.862 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:22:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:47.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:22:47.962 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:22:47.984 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:22:47.996 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:48.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:22:48.024 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:22:48.033 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:48.045 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:22:48.057 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:48.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:48.070 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:22:48.094 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:22:48.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:22:48.136 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:22:48.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:22:48.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:48.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:22:48.205 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:22:48.207 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:22:48.208 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:22:48.208 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:22:48.208 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:48.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:48.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:48.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:48.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:48 smithi194 ceph-mon[44089]: pgmap v1056: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-29T14:22:48.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3560807071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:48.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:48.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:48.396 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:48.425 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:48.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:48.529 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:48.572 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:48.598 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:48.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:48.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:48.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:48 smithi040 ceph-mon[39581]: pgmap v1056: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-29T14:22:48.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3560807071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:48.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:48.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:48 smithi121 ceph-mon[44533]: pgmap v1056: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-29T14:22:48.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3560807071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:22:48.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:48.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:22:48.733 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:49.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:22:49.082 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:22:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:49.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:22:49.186 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:49.231 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:49.259 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:49.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osdmap e400: 12 total, 12 up, 8 in 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:49.395 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:49.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:22:49.526 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:49.543 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:49.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osdmap e400: 12 total, 12 up, 8 in 2024-04-29T14:22:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osdmap e400: 12 total, 12 up, 8 in 2024-04-29T14:22:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:22:49.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:22:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:49.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:22:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:49.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:22:49.853 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:49.888 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:49.933 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:49.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:50.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:22:50.084 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:50.114 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:22:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:50.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:22:50.187 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:50 smithi194 ceph-mon[44089]: pgmap v1058: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 782 op/s 2024-04-29T14:22:50.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:50.250 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:50.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:50 smithi040 ceph-mon[39581]: pgmap v1058: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 782 op/s 2024-04-29T14:22:50.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:50 smithi121 ceph-mon[44533]: pgmap v1058: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 782 op/s 2024-04-29T14:22:50.958 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:51.229 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:51.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:51.334 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:51.367 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:51.465 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:51.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:51.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:22:51.572 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:51.615 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:51.647 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:51.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:51.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:51.806 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:51.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:51.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:51.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:51.963 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:52.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:52 smithi040 ceph-mon[39581]: pgmap v1059: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 916 op/s 2024-04-29T14:22:52.512 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:52.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:52 smithi194 ceph-mon[44089]: pgmap v1059: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 916 op/s 2024-04-29T14:22:52.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:52.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:52 smithi121 ceph-mon[44533]: pgmap v1059: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 916 op/s 2024-04-29T14:22:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:52.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:53.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:53.130 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:53.579 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:53.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:53.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:53.689 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:22:53.740 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:53.780 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:22:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:53.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:22:53.884 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:22:53.931 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:53.959 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:22:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:53.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:22:54.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:54.204 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:54 smithi040 ceph-mon[39581]: pgmap v1060: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:22:54.204 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:54 smithi040 ceph-mon[39581]: osdmap e401: 12 total, 12 up, 8 in 2024-04-29T14:22:54.204 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:54.204 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:54.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:54 smithi194 ceph-mon[44089]: pgmap v1060: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:22:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:54 smithi194 ceph-mon[44089]: osdmap e401: 12 total, 12 up, 8 in 2024-04-29T14:22:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:54.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:54 smithi121 ceph-mon[44533]: pgmap v1060: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:22:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:54 smithi121 ceph-mon[44533]: osdmap e401: 12 total, 12 up, 8 in 2024-04-29T14:22:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:55.224 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:55.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:55.276 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:22:55.276 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:22:55.328 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:22:55.385 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:55.422 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:22:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:55.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:22:55.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:22:55.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:55 smithi194 ceph-mon[44089]: osdmap e402: 12 total, 12 up, 8 in 2024-04-29T14:22:55.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:55 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:22:55.564 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:55 smithi121 ceph-mon[44533]: osdmap e402: 12 total, 12 up, 8 in 2024-04-29T14:22:55.564 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:55 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:22:55.568 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:55.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:22:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:55 smithi040 ceph-mon[39581]: osdmap e402: 12 total, 12 up, 8 in 2024-04-29T14:22:55.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:55 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:22:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:55.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:22:55.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:22:55.775 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:55.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:22:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:55.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:22:55.907 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:55.950 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:55.975 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:55.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:56.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:56.110 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:56.330 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:22:56.330 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:22:56.330 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 402, 2024-04-29T14:22:56.330 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:22:56.330 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:22:56.330 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:22:54.209244+0000", 2024-04-29T14:22:56.330 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:22:12.596258+0000", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:19:01.251051+0000", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5806080, 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "nodeep-scrub", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "noscrub", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:22:56.331 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 29, 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 4, 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:22:56.332 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:22:56.333 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:22:56.334 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:22:56.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:22:56.336 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:22:56.337 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:22:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:22:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:22:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:22:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:22:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:22:56.338 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:22:56.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 10, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 10, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.20000000298023224, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.341 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 4, 2024-04-29T14:22:56.341 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_2", 2024-04-29T14:22:56.341 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:18:47.160984+0000", 2024-04-29T14:22:56.341 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:22:56.341 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:22:56.341 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:22:56.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "326", 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:22:56.343 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:22:56.344 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:22:56.345 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.5, 2024-04-29T14:22:56.346 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 2.5, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.80000001192092896, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:22:56.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:22:56.350 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:22:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.352 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.353 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:22:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:22:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 152, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.357 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:22:56.358 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:22:56.359 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:22:56.360 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:22:56.362 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:22:56.363 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 397, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.364 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.366 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:22:56.367 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 36, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 369, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 388, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 388, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 370, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:22:56.368 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:22:56.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/2586630078", 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/2586630078", 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/2586630078", 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/2586630078", 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:22:56.371 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.372 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.373 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:22:56.374 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.375 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:22:56.376 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.380 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:22:56.382 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.383 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 388, 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.384 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:22:56.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:22:56.386 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.387 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.388 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.389 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:22:56.390 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:22:56.391 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 370, 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:22:56.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:22:56.394 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:07.530956+0000", 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:53.643707+0000", 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.396 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:28.535076+0000", 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:04.827323+0000", 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.397 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:53.818758+0000", 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.398 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:26.135475+0000", 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 369 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:52.220678+0000", 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:53.058668+0000", 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.400 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:24.665708+0000", 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:21.483233+0000", 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.401 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:21:55.478767+0000", 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:22:56.402 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:47.912536+0000", 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:22:56.403 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:22:56.404 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:22:56.405 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:22:56.406 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:56 smithi040 ceph-mon[39581]: pgmap v1063: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 831 op/s 2024-04-29T14:22:56.406 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:56.406 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:56.406 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1491050713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:22:56.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:56 smithi194 ceph-mon[44089]: pgmap v1063: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 831 op/s 2024-04-29T14:22:56.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:56.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:56.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1491050713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:22:56.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:56 smithi121 ceph-mon[44533]: pgmap v1063: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 831 op/s 2024-04-29T14:22:56.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:56.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:56.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1491050713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:22:56.860 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:56.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:56.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:22:57.015 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:57.051 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:22:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:57.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:22:57.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:57.188 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:22:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1615304993' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:22:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:22:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:22:57.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:57 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1615304993' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:22:57.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:22:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:22:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:57 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:57.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:22:57.561 INFO:tasks.thrashosds.thrasher:j is {'epoch': 402, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:22:54.209244+0000', 'last_up_change': '2024-04-29T14:22:12.596258+0000', 'last_in_change': '2024-04-29T14:19:01.251051+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-29T14:18:47.160984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '326', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.800000011920929, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 152, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 397, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 36, 'last_clean_end': 369, 'up_from': 388, 'up_thru': 388, 'down_at': 370, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 2586630078}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2586630078}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2586630078}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2586630078}]}, 'public_addr': '172.21.15.121:6804/2586630078', 'cluster_addr': '172.21.15.121:6805/2586630078', 'heartbeat_back_addr': '172.21.15.121:6807/2586630078', 'heartbeat_front_addr': '172.21.15.121:6806/2586630078', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 206, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 388, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:07.530956+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:22:53.643707+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:22:28.535076+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:22:04.827323+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:21:53.818758+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:26.135475+0000', 'dead_epoch': 369}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:52.220678+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:21:53.058668+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:22:24.665708+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:21.483233+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:21:55.478767+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:47.912536+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:22:57.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:22:57.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1615304993' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:22:57.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:22:57.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:22:57.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:57 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:22:57.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:58.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:58.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:58.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:58 smithi040 ceph-mon[39581]: pgmap v1064: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:22:58.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:58 smithi040 ceph-mon[39581]: pgmap v1065: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-29T14:22:58.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:58 smithi040 ceph-mon[39581]: pgmap v1066: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:22:58.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:58 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:58.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1615304993' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:22:58.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:58 smithi040 ceph-mon[39581]: osdmap e403: 12 total, 12 up, 8 in 2024-04-29T14:22:58.553 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:58 smithi194 ceph-mon[44089]: pgmap v1064: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:22:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:58 smithi194 ceph-mon[44089]: pgmap v1065: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-29T14:22:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:58 smithi194 ceph-mon[44089]: pgmap v1066: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:22:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:58 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1615304993' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:22:58.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:58 smithi194 ceph-mon[44089]: osdmap e403: 12 total, 12 up, 8 in 2024-04-29T14:22:58.627 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:58 smithi121 ceph-mon[44533]: pgmap v1064: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:22:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:58 smithi121 ceph-mon[44533]: pgmap v1065: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-29T14:22:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:58 smithi121 ceph-mon[44533]: pgmap v1066: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:22:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:58 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:22:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1615304993' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:22:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:58 smithi121 ceph-mon[44533]: osdmap e403: 12 total, 12 up, 8 in 2024-04-29T14:22:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:58.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:58.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:22:58.763 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:59.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:22:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:22:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:59.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:22:59.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:22:59.287 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:59.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:22:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:59.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:22:59.356 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:22:59.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:22:59.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:22:59.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:59 smithi040 ceph-mon[39581]: 4.9 scrub starts 2024-04-29T14:22:59.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:59 smithi040 ceph-mon[39581]: 4.6 scrub starts 2024-04-29T14:22:59.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:59 smithi040 ceph-mon[39581]: osdmap e404: 12 total, 12 up, 8 in 2024-04-29T14:22:59.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:22:59 smithi040 ceph-mon[39581]: 4.0 scrub starts 2024-04-29T14:22:59.417 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:22:59.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:22:59.440 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:22:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:59.463 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:22:59.489 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:22:59.517 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:22:59.543 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:22:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:22:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:22:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:59 smithi194 ceph-mon[44089]: 4.9 scrub starts 2024-04-29T14:22:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:59 smithi194 ceph-mon[44089]: 4.6 scrub starts 2024-04-29T14:22:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:59 smithi194 ceph-mon[44089]: osdmap e404: 12 total, 12 up, 8 in 2024-04-29T14:22:59.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:22:59 smithi194 ceph-mon[44089]: 4.0 scrub starts 2024-04-29T14:22:59.568 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:22:59.596 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:22:59.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:22:59.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:22:59.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:22:59.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:59 smithi121 ceph-mon[44533]: 4.9 scrub starts 2024-04-29T14:22:59.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:59 smithi121 ceph-mon[44533]: 4.6 scrub starts 2024-04-29T14:22:59.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:59 smithi121 ceph-mon[44533]: osdmap e404: 12 total, 12 up, 8 in 2024-04-29T14:22:59.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:22:59 smithi121 ceph-mon[44533]: 4.0 scrub starts 2024-04-29T14:22:59.641 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:22:59.661 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:22:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:59.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:22:59.666 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:22:59.688 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:22:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:22:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:22:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:22:59.766 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:22:59.794 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:22:59.942 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:22:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:22:59.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:00.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:00.076 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:00.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:00.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:00.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:00.228 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: 4.6 scrub ok 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: 4.9 scrub ok 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: pgmap v1069: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: 4.5 scrub starts 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: 4.d scrub starts 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4232835731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: 4.0 scrub ok 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: 4.6 scrub ok 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: 4.9 scrub ok 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: pgmap v1069: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: 4.5 scrub starts 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: 4.d scrub starts 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4232835731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: 4.0 scrub ok 2024-04-29T14:23:00.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:23:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: 4.6 scrub ok 2024-04-29T14:23:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: 4.9 scrub ok 2024-04-29T14:23:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: pgmap v1069: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:23:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: 4.5 scrub starts 2024-04-29T14:23:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: 4.d scrub starts 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4232835731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: 4.0 scrub ok 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:00.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:01.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:01.042 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:01.070 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:01.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:01.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:01.222 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:01.259 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:01.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:01 smithi194 ceph-mon[44089]: 4.5 scrub ok 2024-04-29T14:23:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:01 smithi194 ceph-mon[44089]: 4.d scrub ok 2024-04-29T14:23:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:01 smithi194 ceph-mon[44089]: 4.e scrub starts 2024-04-29T14:23:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:01 smithi194 ceph-mon[44089]: 4.e scrub ok 2024-04-29T14:23:01.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:01 smithi194 ceph-mon[44089]: pgmap v1070: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 881 op/s 2024-04-29T14:23:01.364 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:01.407 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:01.430 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:01.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:01.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:01.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:01 smithi040 ceph-mon[39581]: 4.5 scrub ok 2024-04-29T14:23:01.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:01 smithi040 ceph-mon[39581]: 4.d scrub ok 2024-04-29T14:23:01.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:01 smithi040 ceph-mon[39581]: 4.e scrub starts 2024-04-29T14:23:01.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:01 smithi040 ceph-mon[39581]: 4.e scrub ok 2024-04-29T14:23:01.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:01 smithi040 ceph-mon[39581]: pgmap v1070: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 881 op/s 2024-04-29T14:23:01.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:01.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:01.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:01 smithi121 ceph-mon[44533]: 4.5 scrub ok 2024-04-29T14:23:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:01 smithi121 ceph-mon[44533]: 4.d scrub ok 2024-04-29T14:23:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:01 smithi121 ceph-mon[44533]: 4.e scrub starts 2024-04-29T14:23:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:01 smithi121 ceph-mon[44533]: 4.e scrub ok 2024-04-29T14:23:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:01 smithi121 ceph-mon[44533]: pgmap v1070: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 881 op/s 2024-04-29T14:23:02.088 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:02.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:02.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:23:02.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:02.329 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:23:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:02.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:23:02.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:02 smithi040 ceph-mon[39581]: 4.4 scrub starts 2024-04-29T14:23:02.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:02 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:23:02.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:02 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:23:02.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:02.498 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:02.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:02.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:02 smithi121 ceph-mon[44533]: 4.4 scrub starts 2024-04-29T14:23:02.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:02 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:23:02.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:02 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:23:02.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:02 smithi194 ceph-mon[44089]: 4.4 scrub starts 2024-04-29T14:23:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:02 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:23:02.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:02 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:23:02.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:02.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:23:03.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:03 smithi194 ceph-mon[44089]: 4.4 scrub ok 2024-04-29T14:23:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:03 smithi194 ceph-mon[44089]: 4.c scrub starts 2024-04-29T14:23:03.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:03 smithi194 ceph-mon[44089]: pgmap v1071: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-04-29T14:23:03.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:03 smithi040 ceph-mon[39581]: 4.4 scrub ok 2024-04-29T14:23:03.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:03 smithi040 ceph-mon[39581]: 4.c scrub starts 2024-04-29T14:23:03.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:03 smithi040 ceph-mon[39581]: pgmap v1071: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-04-29T14:23:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:03 smithi121 ceph-mon[44533]: 4.4 scrub ok 2024-04-29T14:23:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:03 smithi121 ceph-mon[44533]: 4.c scrub starts 2024-04-29T14:23:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:03 smithi121 ceph-mon[44533]: pgmap v1071: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-04-29T14:23:03.982 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:23:03.984 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:23:04.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:04 smithi194 ceph-mon[44089]: 4.c scrub ok 2024-04-29T14:23:04.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:04 smithi194 ceph-mon[44089]: from='client.16020 v1:172.21.15.40:0/1089406842' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:23:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:04 smithi040 ceph-mon[39581]: 4.c scrub ok 2024-04-29T14:23:04.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:04 smithi040 ceph-mon[39581]: from='client.16020 v1:172.21.15.40:0/1089406842' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:23:04.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:04 smithi121 ceph-mon[44533]: 4.c scrub ok 2024-04-29T14:23:04.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:04 smithi121 ceph-mon[44533]: from='client.16020 v1:172.21.15.40:0/1089406842' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:23:04.778 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1071,"stamp":"2024-04-29T14:23:03.084745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4221895224,"num_objects":64415,"num_object_clones":0,"num_object_copies":128830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64415,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":249130,"num_write_kb":10030459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161016,"ondisk_log_size":161016,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11085180,"kb_used_data":8289400,"kb_used_omap":222,"kb_used_meta":2795425,"kb_avail":738876036,"statfs":{"total":767960285184,"available":756609060864,"internally_reserved":0,"allocated":8488345600,"data_stored":8485166192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":227868,"internal_metadata":2862515684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-355270656,"num_objects":-5421,"num_object_clones":0,"num_object_copies":-10842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001183"},"pg_stats":[{"pgid":"4.f","version":"404'15596","reported_seq":15947,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:00.575048+0000","last_change":"2024-04-29T14:21:46.009086+0000","last_active":"2024-04-29T14:23:00.575048+0000","last_peered":"2024-04-29T14:23:00.575048+0000","last_clean":"2024-04-29T14:23:00.575048+0000","last_became_active":"2024-04-29T14:18:49.370395+0000","last_became_peered":"2024-04-29T14:18:49.370395+0000","last_unstale":"2024-04-29T14:23:00.575048+0000","last_undegraded":"2024-04-29T14:23:00.575048+0000","last_fullsized":"2024-04-29T14:23:00.575048+0000","mapping_epoch":323,"log_start":"341'5500","ondisk_log_start":"341'5500","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12469","last_scrub_stamp":"2024-04-29T14:21:46.008916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:46.008916+0000","objects_scrubbed":7173,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:04.016084+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265158656,"num_objects":4046,"num_object_clones":0,"num_object_copies":8092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15596,"num_write_kb":628544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"404'15559","reported_seq":15957,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.033119+0000","last_change":"2024-04-29T14:23:00.699458+0000","last_active":"2024-04-29T14:23:01.033119+0000","last_peered":"2024-04-29T14:23:01.033119+0000","last_clean":"2024-04-29T14:23:01.033119+0000","last_became_active":"2024-04-29T14:18:49.152699+0000","last_became_peered":"2024-04-29T14:18:49.152699+0000","last_unstale":"2024-04-29T14:23:01.033119+0000","last_undegraded":"2024-04-29T14:23:01.033119+0000","last_fullsized":"2024-04-29T14:23:01.033119+0000","mapping_epoch":323,"log_start":"341'5500","ondisk_log_start":"341'5500","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15536","last_scrub_stamp":"2024-04-29T14:23:00.699185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:00.699185+0000","objects_scrubbed":4092,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:13.273646+0000","scrub_duration":475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265617408,"num_objects":4053,"num_object_clones":0,"num_object_copies":8106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15559,"num_write_kb":627584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"404'15664","reported_seq":16059,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.037659+0000","last_change":"2024-04-29T14:23:00.094580+0000","last_active":"2024-04-29T14:23:01.037659+0000","last_peered":"2024-04-29T14:23:01.037659+0000","last_clean":"2024-04-29T14:23:01.037659+0000","last_became_active":"2024-04-29T14:18:49.155690+0000","last_became_peered":"2024-04-29T14:18:49.155690+0000","last_unstale":"2024-04-29T14:23:01.037659+0000","last_undegraded":"2024-04-29T14:23:01.037659+0000","last_fullsized":"2024-04-29T14:23:01.037659+0000","mapping_epoch":323,"log_start":"342'5600","ondisk_log_start":"342'5600","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15614","last_scrub_stamp":"2024-04-29T14:23:00.094315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:00.094315+0000","objects_scrubbed":4176,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:03.040952+0000","scrub_duration":836,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268632088,"num_objects":4100,"num_object_clones":0,"num_object_copies":8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15664,"num_write_kb":632385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"404'15631","reported_seq":15979,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.028158+0000","last_change":"2024-04-29T14:21:44.740246+0000","last_active":"2024-04-29T14:23:01.028158+0000","last_peered":"2024-04-29T14:23:01.028158+0000","last_clean":"2024-04-29T14:23:01.028158+0000","last_became_active":"2024-04-29T14:18:49.156158+0000","last_became_peered":"2024-04-29T14:18:49.156158+0000","last_unstale":"2024-04-29T14:23:01.028158+0000","last_undegraded":"2024-04-29T14:23:01.028158+0000","last_fullsized":"2024-04-29T14:23:01.028158+0000","mapping_epoch":323,"log_start":"342'5600","ondisk_log_start":"342'5600","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12380","last_scrub_stamp":"2024-04-29T14:21:44.740055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:44.740055+0000","objects_scrubbed":7272,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15631,"num_write_kb":628864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"404'15912","reported_seq":16268,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.011169+0000","last_change":"2024-04-29T14:22:41.172125+0000","last_active":"2024-04-29T14:23:01.011169+0000","last_peered":"2024-04-29T14:23:01.011169+0000","last_clean":"2024-04-29T14:23:01.011169+0000","last_became_active":"2024-04-29T14:22:41.142887+0000","last_became_peered":"2024-04-29T14:22:41.142887+0000","last_unstale":"2024-04-29T14:23:01.011169+0000","last_undegraded":"2024-04-29T14:23:01.011169+0000","last_fullsized":"2024-04-29T14:23:01.011169+0000","mapping_epoch":397,"log_start":"343'5900","ondisk_log_start":"343'5900","created":323,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12656","last_scrub_stamp":"2024-04-29T14:21:49.484118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:49.484118+0000","objects_scrubbed":7362,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:12.851748+0000","scrub_duration":628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269090816,"num_objects":4106,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15912,"num_write_kb":640576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"404'15438","reported_seq":15783,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:00.075314+0000","last_change":"2024-04-29T14:21:54.688108+0000","last_active":"2024-04-29T14:23:00.075314+0000","last_peered":"2024-04-29T14:23:00.075314+0000","last_clean":"2024-04-29T14:23:00.075314+0000","last_became_active":"2024-04-29T14:18:49.156474+0000","last_became_peered":"2024-04-29T14:18:49.156474+0000","last_unstale":"2024-04-29T14:23:00.075314+0000","last_undegraded":"2024-04-29T14:23:00.075314+0000","last_fullsized":"2024-04-29T14:23:00.075314+0000","mapping_epoch":323,"log_start":"341'5400","ondisk_log_start":"341'5400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12343","last_scrub_stamp":"2024-04-29T14:21:54.687953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:54.687953+0000","objects_scrubbed":7201,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:15.986846+0000","scrub_duration":602,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269090816,"num_objects":4106,"num_object_clones":0,"num_object_copies":8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15438,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"404'15438","reported_seq":15832,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:22:59.982938+0000","last_change":"2024-04-29T14:22:58.989877+0000","last_active":"2024-04-29T14:22:59.982938+0000","last_peered":"2024-04-29T14:22:59.982938+0000","last_clean":"2024-04-29T14:22:59.982938+0000","last_became_active":"2024-04-29T14:18:49.371178+0000","last_became_peered":"2024-04-29T14:18:49.371178+0000","last_unstale":"2024-04-29T14:22:59.982938+0000","last_undegraded":"2024-04-29T14:22:59.982938+0000","last_fullsized":"2024-04-29T14:22:59.982938+0000","mapping_epoch":323,"log_start":"340'5400","ondisk_log_start":"340'5400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15374","last_scrub_stamp":"2024-04-29T14:22:58.989714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:58.989714+0000","objects_scrubbed":4131,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:01.626190+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265551872,"num_objects":4052,"num_object_clones":0,"num_object_copies":8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15438,"num_write_kb":623680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"404'15549","reported_seq":15894,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.062498+0000","last_change":"2024-04-29T14:21:46.778153+0000","last_active":"2024-04-29T14:23:01.062498+0000","last_peered":"2024-04-29T14:23:01.062498+0000","last_clean":"2024-04-29T14:23:01.062498+0000","last_became_active":"2024-04-29T14:18:49.370541+0000","last_became_peered":"2024-04-29T14:18:49.370541+0000","last_unstale":"2024-04-29T14:23:01.062498+0000","last_undegraded":"2024-04-29T14:23:01.062498+0000","last_fullsized":"2024-04-29T14:23:01.062498+0000","mapping_epoch":323,"log_start":"341'5500","ondisk_log_start":"341'5500","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12370","last_scrub_stamp":"2024-04-29T14:21:46.777975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:46.777975+0000","objects_scrubbed":7196,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:09.308038+0000","scrub_duration":605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263258112,"num_objects":4017,"num_object_clones":0,"num_object_copies":8034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15549,"num_write_kb":626112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"404'15568","reported_seq":15914,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:02.733659+0000","last_change":"2024-04-29T14:21:43.967168+0000","last_active":"2024-04-29T14:23:02.733659+0000","last_peered":"2024-04-29T14:23:02.733659+0000","last_clean":"2024-04-29T14:23:02.733659+0000","last_became_active":"2024-04-29T14:18:49.151382+0000","last_became_peered":"2024-04-29T14:18:49.151382+0000","last_unstale":"2024-04-29T14:23:02.733659+0000","last_undegraded":"2024-04-29T14:23:02.733659+0000","last_fullsized":"2024-04-29T14:23:02.733659+0000","mapping_epoch":323,"log_start":"342'5500","ondisk_log_start":"342'5500","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12296","last_scrub_stamp":"2024-04-29T14:21:43.966992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:43.966992+0000","objects_scrubbed":7200,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:11.384006+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257425408,"num_objects":3928,"num_object_clones":0,"num_object_copies":7856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15568,"num_write_kb":623872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"404'15693","reported_seq":16095,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:00.609567+0000","last_change":"2024-04-29T14:22:58.908796+0000","last_active":"2024-04-29T14:23:00.609567+0000","last_peered":"2024-04-29T14:23:00.609567+0000","last_clean":"2024-04-29T14:23:00.609567+0000","last_became_active":"2024-04-29T14:18:49.157084+0000","last_became_peered":"2024-04-29T14:18:49.157084+0000","last_unstale":"2024-04-29T14:23:00.609567+0000","last_undegraded":"2024-04-29T14:23:00.609567+0000","last_fullsized":"2024-04-29T14:23:00.609567+0000","mapping_epoch":323,"log_start":"341'5600","ondisk_log_start":"341'5600","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15612","last_scrub_stamp":"2024-04-29T14:22:58.908548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:58.908548+0000","objects_scrubbed":4178,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:01.580330+0000","scrub_duration":792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267321344,"num_objects":4079,"num_object_clones":0,"num_object_copies":8158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15693,"num_write_kb":632704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7],"acting":[9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"404'15046","reported_seq":15435,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:02.711290+0000","last_change":"2024-04-29T14:23:01.894806+0000","last_active":"2024-04-29T14:23:02.711290+0000","last_peered":"2024-04-29T14:23:02.711290+0000","last_clean":"2024-04-29T14:23:02.711290+0000","last_became_active":"2024-04-29T14:18:49.156484+0000","last_became_peered":"2024-04-29T14:18:49.156484+0000","last_unstale":"2024-04-29T14:23:02.711290+0000","last_undegraded":"2024-04-29T14:23:02.711290+0000","last_fullsized":"2024-04-29T14:23:02.711290+0000","mapping_epoch":323,"log_start":"340'5000","ondisk_log_start":"340'5000","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14993","last_scrub_stamp":"2024-04-29T14:23:01.894626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:01.894626+0000","objects_scrubbed":3926,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:06.815555+0000","scrub_duration":829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252182528,"num_objects":3848,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15046,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"404'17944","reported_seq":18409,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.028420+0000","last_change":"2024-04-29T14:21:54.780846+0000","last_active":"2024-04-29T14:23:01.028420+0000","last_peered":"2024-04-29T14:23:01.028420+0000","last_clean":"2024-04-29T14:23:01.028420+0000","last_became_active":"2024-04-29T14:18:49.371335+0000","last_became_peered":"2024-04-29T14:18:49.371335+0000","last_unstale":"2024-04-29T14:23:01.028420+0000","last_undegraded":"2024-04-29T14:23:01.028420+0000","last_fullsized":"2024-04-29T14:23:01.028420+0000","mapping_epoch":323,"log_start":"350'7900","ondisk_log_start":"350'7900","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'14737","last_scrub_stamp":"2024-04-29T14:21:54.780714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:54.780714+0000","objects_scrubbed":7177,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:18.490218+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260177920,"num_objects":3970,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15614,"num_write_kb":626688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"404'15551","reported_seq":15908,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:00.595449+0000","last_change":"2024-04-29T14:22:18.591338+0000","last_active":"2024-04-29T14:23:00.595449+0000","last_peered":"2024-04-29T14:23:00.595449+0000","last_clean":"2024-04-29T14:23:00.595449+0000","last_became_active":"2024-04-29T14:22:13.613928+0000","last_became_peered":"2024-04-29T14:22:13.613928+0000","last_unstale":"2024-04-29T14:23:00.595449+0000","last_undegraded":"2024-04-29T14:23:00.595449+0000","last_fullsized":"2024-04-29T14:23:00.595449+0000","mapping_epoch":388,"log_start":"341'5500","ondisk_log_start":"341'5500","created":323,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'12766","last_scrub_stamp":"2024-04-29T14:22:18.591115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:18.591115+0000","objects_scrubbed":6887,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:42.749342+0000","scrub_duration":1374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266010624,"num_objects":4059,"num_object_clones":0,"num_object_copies":8118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15551,"num_write_kb":627520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"404'15691","reported_seq":16014,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.126887+0000","last_change":"2024-04-29T14:22:20.297495+0000","last_active":"2024-04-29T14:23:01.126887+0000","last_peered":"2024-04-29T14:23:01.126887+0000","last_clean":"2024-04-29T14:23:01.126887+0000","last_became_active":"2024-04-29T14:22:13.607474+0000","last_became_peered":"2024-04-29T14:22:13.607474+0000","last_unstale":"2024-04-29T14:23:01.126887+0000","last_undegraded":"2024-04-29T14:23:01.126887+0000","last_fullsized":"2024-04-29T14:23:01.126887+0000","mapping_epoch":388,"log_start":"341'5600","ondisk_log_start":"341'5600","created":323,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'13020","last_scrub_stamp":"2024-04-29T14:22:20.297257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:20.297257+0000","objects_scrubbed":6680,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:29.891325+0000","scrub_duration":1237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260898816,"num_objects":3981,"num_object_clones":0,"num_object_copies":7962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15691,"num_write_kb":629504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"404'15555","reported_seq":15946,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:01.044945+0000","last_change":"2024-04-29T14:22:59.405191+0000","last_active":"2024-04-29T14:23:01.044945+0000","last_peered":"2024-04-29T14:23:01.044945+0000","last_clean":"2024-04-29T14:23:01.044945+0000","last_became_active":"2024-04-29T14:18:49.158015+0000","last_became_peered":"2024-04-29T14:18:49.158015+0000","last_unstale":"2024-04-29T14:23:01.044945+0000","last_undegraded":"2024-04-29T14:23:01.044945+0000","last_fullsized":"2024-04-29T14:23:01.044945+0000","mapping_epoch":323,"log_start":"340'5500","ondisk_log_start":"340'5500","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15489","last_scrub_stamp":"2024-04-29T14:22:59.404935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:59.404935+0000","objects_scrubbed":4030,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:02.608640+0000","scrub_duration":903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258670592,"num_objects":3947,"num_object_clones":0,"num_object_copies":7894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15555,"num_write_kb":624064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"404'15444","reported_seq":15839,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:23:00.041784+0000","last_change":"2024-04-29T14:22:59.972343+0000","last_active":"2024-04-29T14:23:00.041784+0000","last_peered":"2024-04-29T14:23:00.041784+0000","last_clean":"2024-04-29T14:23:00.041784+0000","last_became_active":"2024-04-29T14:18:49.156512+0000","last_became_peered":"2024-04-29T14:18:49.156512+0000","last_unstale":"2024-04-29T14:23:00.041784+0000","last_undegraded":"2024-04-29T14:23:00.041784+0000","last_fullsized":"2024-04-29T14:23:00.041784+0000","mapping_epoch":323,"log_start":"341'5400","ondisk_log_start":"341'5400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15439","last_scrub_stamp":"2024-04-29T14:22:59.972178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:59.972178+0000","objects_scrubbed":4122,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:26.200777+0000","scrub_duration":865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268697600,"num_objects":4100,"num_object_clones":0,"num_object_copies":8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15444,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1024,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-04-29T14:22:58.241715+0000","last_change":"2024-04-29T14:21:52.193218+0000","last_active":"2024-04-29T14:22:58.241715+0000","last_peered":"2024-04-29T14:22:58.241715+0000","last_clean":"2024-04-29T14:22:58.241715+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:22:58.241715+0000","last_undegraded":"2024-04-29T14:22:58.241715+0000","last_fullsized":"2024-04-29T14:22:58.241715+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:21:52.193031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:21:52.193031+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:00.532202+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4221304856,"num_objects":64413,"num_object_clones":0,"num_object_copies":128826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64413,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248949,"num_write_kb":10027521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8431149056,"data_stored":8431140912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":212282,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122241,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1688956,"kb_used_data":1292660,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92056196,"statfs":{"total":95995035648,"available":94265544704,"internally_reserved":0,"allocated":1323683840,"data_stored":1323284753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":405797316},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383878,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504436,"kb_used_data":1270508,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":92240716,"statfs":{"total":95995035648,"available":94454493184,"internally_reserved":0,"allocated":1301000192,"data_stored":1300604657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":239466944},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645515,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1394172,"kb_used_data":1047856,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92350980,"statfs":{"total":95995035648,"available":94567403520,"internally_reserved":0,"allocated":1073004544,"data_stored":1072604957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234196,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1697084,"kb_used_data":1291060,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92048068,"statfs":{"total":95995035648,"available":94257221632,"internally_reserved":0,"allocated":1322045440,"data_stored":1321646373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557940,"kb_used_data":1045036,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92187212,"statfs":{"total":95995035648,"available":94399705088,"internally_reserved":0,"allocated":1070116864,"data_stored":1069721349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":525203910},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1091316,"kb_used_data":775532,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92653836,"statfs":{"total":95995035648,"available":94877528064,"internally_reserved":0,"allocated":794144768,"data_stored":793749253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182633,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1584312,"kb_used_data":1053360,"kb_used_omap":1,"kb_used_meta":530942,"kb_avail":92160840,"statfs":{"total":95995035648,"available":94372700160,"internally_reserved":0,"allocated":1078640640,"data_stored":1078241053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":543685061},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":388,"seq":1666447310859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":566964,"kb_used_data":513388,"kb_used_omap":211,"kb_used_meta":53356,"kb_avail":93178188,"statfs":{"total":95995035648,"available":95414464512,"internally_reserved":0,"allocated":525709312,"data_stored":525313797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":216698,"internal_metadata":54636934},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":791543808,"data_stored":791543808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1066733568,"data_stored":1066729496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":523698176,"data_stored":523698176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":212282,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1069219840,"data_stored":1069219840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1306525696,"data_stored":1306525696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1069682688,"data_stored":1069678616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1284702208,"data_stored":1284702208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1319043072,"data_stored":1319043072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:23:04.779 INFO:tasks.thrashosds.thrasher:Setting 4.3 to [11, 4, 3, 10] 2024-04-29T14:23:04.779 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.3', '11', '4', '3', '10'] 2024-04-29T14:23:04.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 4.3 11 4 3 10 2024-04-29T14:23:05.014 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:05.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:05.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:05.440 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:23:05.440 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:23:05.440 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:23:05.440 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:23:05.440 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:23:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:05 smithi194 ceph-mon[44089]: 4.f scrub starts 2024-04-29T14:23:05.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:05 smithi194 ceph-mon[44089]: pgmap v1072: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-04-29T14:23:05.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:05 smithi040 ceph-mon[39581]: 4.f scrub starts 2024-04-29T14:23:05.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:05 smithi040 ceph-mon[39581]: pgmap v1072: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-04-29T14:23:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:05 smithi121 ceph-mon[44533]: 4.f scrub starts 2024-04-29T14:23:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:05 smithi121 ceph-mon[44533]: pgmap v1072: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-04-29T14:23:05.652 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:05.901 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:05.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:06.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:06.051 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:06.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:06.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:06.193 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:06.249 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:06.267 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:06.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:06.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:06.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:06 smithi121 ceph-mon[44533]: 4.f scrub ok 2024-04-29T14:23:06.427 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:06.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:06.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:06 smithi194 ceph-mon[44089]: 4.f scrub ok 2024-04-29T14:23:06.577 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:06.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:06 smithi040 ceph-mon[39581]: 4.f scrub ok 2024-04-29T14:23:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:06.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:23:06.844 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:06.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:06.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:06.981 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:07.038 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:07.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:07.142 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:07.203 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:07.243 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:07.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:23:07.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:07.382 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3644846895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:07.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:07 smithi194 ceph-mon[44089]: pgmap v1073: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:23:07.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3644846895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:07.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:07 smithi040 ceph-mon[39581]: pgmap v1073: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:23:07.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3644846895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:07.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:07 smithi121 ceph-mon[44533]: pgmap v1073: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:23:07.750 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:08.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:08 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:23:08.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3644846895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:08.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:08 smithi194 ceph-mon[44089]: osdmap e405: 12 total, 12 up, 8 in 2024-04-29T14:23:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:08 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:23:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3644846895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:08.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:08 smithi040 ceph-mon[39581]: osdmap e405: 12 total, 12 up, 8 in 2024-04-29T14:23:08.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:08 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:23:08.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3644846895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:08.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:08 smithi121 ceph-mon[44533]: osdmap e405: 12 total, 12 up, 8 in 2024-04-29T14:23:09.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:09 smithi040 ceph-mon[39581]: pgmap v1075: 17 pgs: 17 active+clean; 3.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-04-29T14:23:09.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:09 smithi121 ceph-mon[44533]: pgmap v1075: 17 pgs: 17 active+clean; 3.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-04-29T14:23:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:09.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:09.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:09 smithi194 ceph-mon[44089]: pgmap v1075: 17 pgs: 17 active+clean; 3.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-04-29T14:23:09.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:09.999 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:10.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:10.108 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:10.165 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:10.198 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:10.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:10.304 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:10.338 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:23:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:10.358 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:23:10.378 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:23:10.386 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:10.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:10.401 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:23:10.431 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:23:10.457 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:23:10.481 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:23:10.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:10.503 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:23:10.528 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:23:10.536 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:10.554 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:23:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:10 smithi194 ceph-mon[44089]: 4.8 scrub starts 2024-04-29T14:23:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2847464396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:10.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:10.577 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:23:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:23:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:23:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:23:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:23:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:23:10.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:10 smithi040 ceph-mon[39581]: 4.8 scrub starts 2024-04-29T14:23:10.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2847464396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:10.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:10 smithi121 ceph-mon[44533]: 4.8 scrub starts 2024-04-29T14:23:10.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2847464396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:10.667 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:10.702 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:10.733 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:10.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:10.838 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:10.875 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:10.904 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:10.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:10.991 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:11.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:11.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:11.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:11.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: 4.8 scrub ok 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:11 smithi040 ceph-mon[39581]: pgmap v1076: 17 pgs: 17 active+clean; 3.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-29T14:23:11.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: 4.8 scrub ok 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:11 smithi121 ceph-mon[44533]: pgmap v1076: 17 pgs: 17 active+clean; 3.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-29T14:23:11.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: 4.8 scrub ok 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:11.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:11 smithi194 ceph-mon[44089]: pgmap v1076: 17 pgs: 17 active+clean; 3.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-29T14:23:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:11.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:11.945 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:11.976 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:12.006 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:12.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:12.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:12.146 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:12.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:12.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:12.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:12.304 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:12.853 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:12.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:12.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:12.995 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:13.032 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:13.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:13.137 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:13.177 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:13.200 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:13.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:13.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:13.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:13.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:13.488 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:13.515 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:13.547 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:13.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:13.591 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:13.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:13.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:13.974 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:14.024 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:14.054 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:14.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:14.149 INFO:teuthology.orchestra.run.smithi040.stderr:set 4.3 pg_upmap_items mapping to [11->4,3->10] 2024-04-29T14:23:14.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:14.196 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: 4.b scrub starts 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: pgmap v1077: 17 pgs: 17 active+clean; 3.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: osdmap e406: 12 total, 12 up, 8 in 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: 4.b scrub ok 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2753903385' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]: dispatch 2024-04-29T14:23:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:14 smithi194 ceph-mon[44089]: from='client.25885 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]: dispatch 2024-04-29T14:23:14.353 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:14.355 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:14.355 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:14.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: 4.b scrub starts 2024-04-29T14:23:14.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: pgmap v1077: 17 pgs: 17 active+clean; 3.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-29T14:23:14.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: osdmap e406: 12 total, 12 up, 8 in 2024-04-29T14:23:14.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: 4.b scrub ok 2024-04-29T14:23:14.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2753903385' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]: dispatch 2024-04-29T14:23:14.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:14 smithi040 ceph-mon[39581]: from='client.25885 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]: dispatch 2024-04-29T14:23:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:14.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: 4.b scrub starts 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: pgmap v1077: 17 pgs: 17 active+clean; 3.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: osdmap e406: 12 total, 12 up, 8 in 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: 4.b scrub ok 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2753903385' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]: dispatch 2024-04-29T14:23:14.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:14 smithi121 ceph-mon[44533]: from='client.25885 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]: dispatch 2024-04-29T14:23:14.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:14.486 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:14.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:14.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:14.897 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:15.050 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:15.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:15.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:15.169 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:23:15.169 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:23:15.181 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:15 smithi040 ceph-mon[39581]: from='client.25885 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]': finished 2024-04-29T14:23:15.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:15 smithi040 ceph-mon[39581]: osdmap e407: 12 total, 12 up, 8 in 2024-04-29T14:23:15.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:23:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:15 smithi194 ceph-mon[44089]: from='client.25885 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]': finished 2024-04-29T14:23:15.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:15 smithi194 ceph-mon[44089]: osdmap e407: 12 total, 12 up, 8 in 2024-04-29T14:23:15.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:15 smithi121 ceph-mon[44533]: from='client.25885 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [11, 4, 3, 10]}]': finished 2024-04-29T14:23:15.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:15 smithi121 ceph-mon[44533]: osdmap e407: 12 total, 12 up, 8 in 2024-04-29T14:23:15.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:23:15.930 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:15.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:16.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:16.061 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:16.114 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:16.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:16.204 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:16.218 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:16.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:16.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:16.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:16 smithi040 ceph-mon[39581]: pgmap v1080: 17 pgs: 17 active+clean; 3.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-04-29T14:23:16.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:16 smithi040 ceph-mon[39581]: osdmap e408: 12 total, 12 up, 8 in 2024-04-29T14:23:16.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:16.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:16.246 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:16 smithi121 ceph-mon[44533]: pgmap v1080: 17 pgs: 17 active+clean; 3.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-04-29T14:23:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:16 smithi121 ceph-mon[44533]: osdmap e408: 12 total, 12 up, 8 in 2024-04-29T14:23:16.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:16.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:16 smithi194 ceph-mon[44089]: pgmap v1080: 17 pgs: 17 active+clean; 3.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-04-29T14:23:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:16 smithi194 ceph-mon[44089]: osdmap e408: 12 total, 12 up, 8 in 2024-04-29T14:23:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:16.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:17.195 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:17.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:17.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:17.371 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:17 smithi121 ceph-mon[44533]: 4.3 starting backfill to osd.4 from (0'0,0'0] MAX to 406'18838 2024-04-29T14:23:17.371 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:17 smithi121 ceph-mon[44533]: osdmap e409: 12 total, 12 up, 8 in 2024-04-29T14:23:17.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:17 smithi040 ceph-mon[39581]: 4.3 starting backfill to osd.4 from (0'0,0'0] MAX to 406'18838 2024-04-29T14:23:17.493 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:17 smithi040 ceph-mon[39581]: osdmap e409: 12 total, 12 up, 8 in 2024-04-29T14:23:17.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:17.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:17 smithi194 ceph-mon[44089]: 4.3 starting backfill to osd.4 from (0'0,0'0] MAX to 406'18838 2024-04-29T14:23:17.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:17 smithi194 ceph-mon[44089]: osdmap e409: 12 total, 12 up, 8 in 2024-04-29T14:23:17.616 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:17.652 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:17.679 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:17.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:17.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:17.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:17.873 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:17.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:17.983 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:18.032 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:18.067 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:18.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:18.174 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:23:18.175 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:18.224 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:18.250 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:18.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:18.255 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:23:18.255 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:23:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:18 smithi194 ceph-mon[44089]: pgmap v1083: 17 pgs: 1 remapped+peering, 16 active+clean; 3.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 849 op/s 2024-04-29T14:23:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:18.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/188925810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:23:18.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:18.413 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:18.508 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:18.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:18 smithi121 ceph-mon[44533]: pgmap v1083: 17 pgs: 1 remapped+peering, 16 active+clean; 3.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 849 op/s 2024-04-29T14:23:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:18.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/188925810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:23:18.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:18 smithi040 ceph-mon[39581]: pgmap v1083: 17 pgs: 1 remapped+peering, 16 active+clean; 3.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 849 op/s 2024-04-29T14:23:18.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:18.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:18.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/188925810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:23:18.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:18.660 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:18.685 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:18.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:18.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:18.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:18.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:19.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:23:19.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:19.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:19 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:19.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/188925810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:23:19.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:19 smithi040 ceph-mon[39581]: osdmap e410: 12 total, 12 up, 8 in 2024-04-29T14:23:19.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:19 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:19.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/188925810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:23:19.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:19 smithi121 ceph-mon[44533]: osdmap e410: 12 total, 12 up, 8 in 2024-04-29T14:23:19.463 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:19.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:19.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:19 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/188925810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:23:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:19 smithi194 ceph-mon[44089]: osdmap e410: 12 total, 12 up, 8 in 2024-04-29T14:23:19.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:19.672 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:23:19.672 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:23:19.672 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:23:19.672 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:23:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:23:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:23:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:23:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:23:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:23:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:23:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:23:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:23:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:19.942 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:19.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:20.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:20.104 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: pgmap v1085: 17 pgs: 1 remapped+peering, 16 active+clean; 3.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: from='client.16047 v1:172.21.15.40:0/3842138090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: from='client.16047 v1:172.21.15.40:0/3842138090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: pgmap v1085: 17 pgs: 1 remapped+peering, 16 active+clean; 3.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-04-29T14:23:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: from='client.16047 v1:172.21.15.40:0/3842138090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: from='client.16047 v1:172.21.15.40:0/3842138090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:20.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:20.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:20.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:20.509 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:20.530 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:20.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: pgmap v1085: 17 pgs: 1 remapped+peering, 16 active+clean; 3.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: from='client.16047 v1:172.21.15.40:0/3842138090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: from='client.16047 v1:172.21.15.40:0/3842138090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:20.633 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:20.669 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:20.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:20.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:20.838 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:20.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:20.962 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:21.003 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:21.039 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:21.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:21.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:21.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:21.230 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:21.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:21.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:21.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:21.382 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:21.436 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:21.456 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:21.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:21.558 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:23:21.558 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:23:21.559 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-29T14:23:21.559 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:23:21.559 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:21.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:23:21.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:21.593 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:21.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:21.650 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:23:21.650 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:23:21.650 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:23:21.650 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:23:21.651 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:23:21.719 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:21.759 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:21.792 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:21.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:21.882 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:21.900 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:21.935 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:21.972 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:21.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:22.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:22.132 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:22.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:22 smithi121 ceph-mon[44533]: pgmap v1086: 17 pgs: 1 remapped+peering, 16 active+clean; 2.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 987 op/s 2024-04-29T14:23:22.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:22 smithi040 ceph-mon[39581]: pgmap v1086: 17 pgs: 1 remapped+peering, 16 active+clean; 2.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 987 op/s 2024-04-29T14:23:22.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:22.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:22.518 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:23:22.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:23:22.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:22 smithi194 ceph-mon[44089]: pgmap v1086: 17 pgs: 1 remapped+peering, 16 active+clean; 2.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 987 op/s 2024-04-29T14:23:22.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:22.648 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:22.679 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:22.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:22.781 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:22.816 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:22.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:22.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:23:23.009 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:23.010 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:23.037 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:23:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:23.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:23:23.142 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:23.181 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:23.205 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:23.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:23.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:23.341 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:23.595 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:23.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:23.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:23.735 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:23.772 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:23.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:23.876 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:23.915 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:23.939 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:23.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:24.046 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:24.101 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:24.128 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:24.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:24.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:24.147 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:24.232 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:24.274 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:24.299 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:24.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:24.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:24 smithi121 ceph-mon[44533]: pgmap v1087: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 2451/92264 objects misplaced (2.657%); 3.1 MiB/s, 49 objects/s recovering 2024-04-29T14:23:24.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:24.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:24 smithi040 ceph-mon[39581]: pgmap v1087: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 2451/92264 objects misplaced (2.657%); 3.1 MiB/s, 49 objects/s recovering 2024-04-29T14:23:24.528 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:24.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:24 smithi194 ceph-mon[44089]: pgmap v1087: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 2451/92264 objects misplaced (2.657%); 3.1 MiB/s, 49 objects/s recovering 2024-04-29T14:23:24.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:24.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:24.852 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:24.882 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:24.909 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:24.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:25.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:25.049 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18495 3.a 3:5aa754d4:::benchmark_data_smithi194_88551_object18494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990240+0000", 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15817627700005, 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68929751399999994, 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.148 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18495 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990240+0000", 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990240+0000", 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990242+0000", 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5909999999999998e-06 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990321+0000", 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9048000000000006e-05 2024-04-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990339+0000", 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7331000000000001e-05 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990348+0000", 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5799999999999998e-06 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990391+0000", 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2725999999999997e-05 2024-04-29T14:23:25.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990398+0000", 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4150000000000002e-06 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.678954+0000", 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68855618299999999 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.679537+0000", 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058264000000000005 2024-04-29T14:23:25.152 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18506 3.a 3:590c742d:::benchmark_data_smithi194_88551_object18505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990361+0000", 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15805472500006, 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69417462299999999, 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.153 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18506 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990361+0000", 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990361+0000", 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990362+0000", 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1939999999999999e-06 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990396+0000", 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3578999999999998e-05 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990402+0000", 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7529999999999998e-06 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990407+0000", 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9930000000000003e-06 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990430+0000", 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3125999999999998e-05 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990434+0000", 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.794e-06 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680041+0000", 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68960765700000004 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680138+0000", 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6328000000000003e-05 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680245+0000", 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107618 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.682280+0000", 2024-04-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020350780000000001 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684069+0000", 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00178908 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684129+0000", 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0080000000000001e-05 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684536+0000", 2024-04-29T14:23:25.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040634299999999998 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18517 3.a 3:57592a66:::benchmark_data_smithi194_88551_object18516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990414+0000", 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15800187699995, 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69492493700000002, 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18517 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990414+0000", 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990414+0000", 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990415+0000", 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9500000000000002e-07 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990447+0000", 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2835e-05 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990454+0000", 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.19e-06 2024-04-29T14:23:25.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990457+0000", 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7060000000000002e-06 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990483+0000", 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5551e-05 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990490+0000", 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5899999999999996e-06 2024-04-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.680937+0000", 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69044763099999995 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681024+0000", 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7015000000000006e-05 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681135+0000", 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111004 2024-04-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684013+0000", 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002877601 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684945+0000", 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093199899999999998 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685000+0000", 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5692999999999997e-05 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685339+0000", 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033842699999999998 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18531 3.a 3:50653578:::benchmark_data_smithi194_88551_object18530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990464+0000", 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15795182600004, 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69534631400000002, 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.165 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18531 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990464+0000", 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990464+0000", 2024-04-29T14:23:25.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990465+0000", 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990514+0000", 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9667999999999999e-05 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990521+0000", 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7329999999999997e-06 2024-04-29T14:23:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990525+0000", 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.907e-06 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990577+0000", 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1904000000000003e-05 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990588+0000", 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0944999999999999e-05 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681648+0000", 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69106008100000005 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681729+0000", 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1100000000000006e-05 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.681845+0000", 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011582300000000001 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.684895+0000", 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030497319999999999 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685386+0000", 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049112099999999996 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685437+0000", 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1622000000000001e-05 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685810+0000", 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037302199999999998 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18535 3.a 3:59440a67:::benchmark_data_smithi194_88551_object18534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15787214900001, 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69569302700000002, 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18535 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990544+0000", 2024-04-29T14:23:25.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6599999999999996e-07 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990581+0000", 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6721999999999998e-05 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990589+0000", 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4830000000000002e-06 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990594+0000", 2024-04-29T14:23:25.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4060000000000004e-06 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990620+0000", 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5993e-05 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990624+0000", 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.683304+0000", 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69267976899999995 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.683413+0000", 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108596 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.683533+0000", 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012050299999999999 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685881+0000", 2024-04-29T14:23:25.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023481589999999998 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685891+0000", 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4180000000000003e-06 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.685922+0000", 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1084000000000003e-05 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686237+0000", 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031483699999999999 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.176 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18542 3.a 3:593cad59:::benchmark_data_smithi194_88551_object18541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990603+0000", 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15781274999995, 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70048259899999998, 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18542 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.177 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990603+0000", 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990603+0000", 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990604+0000", 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9100000000000003e-07 2024-04-29T14:23:25.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990636+0000", 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2549000000000003e-05 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990643+0000", 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2820000000000002e-06 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990646+0000", 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7780000000000001e-06 2024-04-29T14:23:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990674+0000", 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7701000000000001e-05 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990678+0000", 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7500000000000001e-06 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686278+0000", 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69559982399999998 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686374+0000", 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6582999999999993e-05 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.686482+0000", 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108087 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689376+0000", 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028935440000000001 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.690185+0000", 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080902800000000005 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.690242+0000", 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7210000000000003e-05 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691086+0000", 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00084347199999999995 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18546 3.a 3:5fb73ead:::benchmark_data_smithi194_88551_object18545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15776215799997, 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70239883199999997, 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18546 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990654+0000", 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5600000000000005e-07 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990699+0000", 2024-04-29T14:23:25.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4279000000000001e-05 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990704+0000", 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4099999999999999e-06 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990708+0000", 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3540000000000002e-06 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990733+0000", 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4834999999999999e-05 2024-04-29T14:23:25.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990737+0000", 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7629999999999998e-06 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687202+0000", 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69646489300000003 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687302+0000", 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100269 2024-04-29T14:23:25.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687392+0000", 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759000000000006e-05 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691103+0000", 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037110260000000001 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.692192+0000", 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010887710000000001 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.692252+0000", 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0303999999999997e-05 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693052+0000", 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080051299999999998 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.188 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18554 3.a 3:55e068b6:::benchmark_data_smithi194_88551_object18553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990715+0000", 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15770069500002, 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70315993099999996, 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18554 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990715+0000", 2024-04-29T14:23:25.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990715+0000", 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990716+0000", 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.61e-07 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990747+0000", 2024-04-29T14:23:25.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1173999999999997e-05 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990753+0000", 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9340000000000003e-06 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990756+0000", 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.535e-06 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990782+0000", 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5316e-05 2024-04-29T14:23:25.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990788+0000", 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2210000000000002e-06 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.687932+0000", 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69714447599999996 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688036+0000", 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010375200000000001 2024-04-29T14:23:25.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688145+0000", 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108786 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691111+0000", 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002965638 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693274+0000", 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021631530000000001 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693333+0000", 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9378e-05 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693875+0000", 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000541907 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18557 3.a 3:5d1fe685:::benchmark_data_smithi194_88551_object18556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990763+0000", 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15765286600003, 2024-04-29T14:23:25.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70364153100000004, 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18557 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990763+0000", 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990763+0000", 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990764+0000", 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1099999999999995e-07 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990844+0000", 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0158000000000001e-05 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990854+0000", 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9310000000000003e-06 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990862+0000", 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2439999999999993e-06 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990893+0000", 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0624999999999999e-05 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990896+0000", 2024-04-29T14:23:25.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7720000000000001e-06 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688636+0000", 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69773956299999995 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688757+0000", 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121134 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.688916+0000", 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159044 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693207+0000", 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042910320000000002 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693974+0000", 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00076646400000000001 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694028+0000", 2024-04-29T14:23:25.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4815999999999998e-05 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694404+0000", 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037613700000000002 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18582 3.a 3:56da5d7f:::benchmark_data_smithi194_88551_object18581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e221)", 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990877+0000", 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15753924600006, 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70399896699999998, 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.200 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18582 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990877+0000", 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990877+0000", 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990878+0000", 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0499999999999999e-06 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990913+0000", 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4962999999999997e-05 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990920+0000", 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5070000000000003e-06 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990927+0000", 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6640000000000001e-06 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990973+0000", 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6453999999999999e-05 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990979+0000", 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4399999999999996e-06 2024-04-29T14:23:25.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689422+0000", 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69844298299999996 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689537+0000", 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115591 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.689683+0000", 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014627499999999999 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.693215+0000", 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035312859999999998 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694459+0000", 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012442 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694506+0000", 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7375e-05 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694876+0000", 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036917900000000002 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18598 3.a 3:51a9ca23:::benchmark_data_smithi194_88551_object18597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990938+0000", 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15747782799997, 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70444663600000001, 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18598 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990938+0000", 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990938+0000", 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990939+0000", 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7899999999999998e-07 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990975+0000", 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5954000000000002e-05 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990981+0000", 2024-04-29T14:23:25.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8569999999999997e-06 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990987+0000", 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991024+0000", 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6860000000000003e-05 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991029+0000", 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4850000000000003e-06 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691156+0000", 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70012712399999999 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691304+0000", 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014863699999999999 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.691541+0000", 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236987 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694419+0000", 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028776370000000002 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694938+0000", 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051860600000000001 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.694994+0000", 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5971999999999999e-05 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.695385+0000", 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039107500000000001 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18616 3.a 3:57784d63:::benchmark_data_smithi194_88551_object18615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.990997+0000", 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15741831699995, 2024-04-29T14:23:25.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72318601500000002, 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18616 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990997+0000", 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990997+0000", 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.990998+0000", 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5000000000000002e-07 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991030+0000", 2024-04-29T14:23:25.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.167e-05 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991036+0000", 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.421e-06 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991042+0000", 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3519999999999997e-06 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991074+0000", 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2289999999999997e-05 2024-04-29T14:23:25.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991078+0000", 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8039999999999999e-06 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.698095+0000", 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.707017699 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.698228+0000", 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013238900000000001 2024-04-29T14:23:25.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.698409+0000", 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018093699999999999 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.702065+0000", 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036564990000000001 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.713290+0000", 2024-04-29T14:23:25.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011224622 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.713328+0000", 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7873000000000001e-05 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714184+0000", 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00085570899999999996 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18624 3.a 3:5b1ca2c5:::benchmark_data_smithi194_88551_object18623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991051+0000", 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15736436300006, 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72377148300000005, 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18624 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991051+0000", 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991051+0000", 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991052+0000", 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6499999999999999e-07 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991106+0000", 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3702999999999997e-05 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991111+0000", 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.434e-06 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991117+0000", 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3539999999999999e-06 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991146+0000", 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9777999999999998e-05 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991150+0000", 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.681e-06 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699105+0000", 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.707954745 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699182+0000", 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7451000000000001e-05 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699282+0000", 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9840999999999995e-05 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705411+0000", 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061285130000000004 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714242+0000", 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088312289999999995 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714278+0000", 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6556e-05 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714823+0000", 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054453300000000004 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.224 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18640 3.a 3:5c4ff313:::benchmark_data_smithi194_88551_object18639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15729014600004, 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72425418900000005, 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18640 2024-04-29T14:23:25.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991126+0000", 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9599999999999999e-07 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991176+0000", 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0062999999999998e-05 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991182+0000", 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2820000000000004e-06 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991187+0000", 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2900000000000002e-06 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991218+0000", 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1297999999999997e-05 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991222+0000", 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5710000000000002e-06 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699796+0000", 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70857407800000005 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.699909+0000", 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011330799999999999 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700031+0000", 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012213 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705420+0000", 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053887350000000004 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714880+0000", 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094600320000000002 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.714915+0000", 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4938999999999999e-05 2024-04-29T14:23:25.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715380+0000", 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046476699999999999 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18720 3.a 3:5c930f07:::benchmark_data_smithi194_88551_object18719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991196+0000", 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15721944699999, 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.724726023, 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.231 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18720 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991196+0000", 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991196+0000", 2024-04-29T14:23:25.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991197+0000", 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5199999999999996e-07 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991248+0000", 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1104999999999999e-05 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991253+0000", 2024-04-29T14:23:25.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2909999999999998e-06 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991259+0000", 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3210000000000004e-06 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991289+0000", 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0627999999999999e-05 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991293+0000", 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700509+0000", 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70921634600000005 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700641+0000", 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013196700000000001 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.700747+0000", 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010521199999999999 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705428+0000", 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046819299999999999 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715427+0000", 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099981389999999996 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715481+0000", 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4616e-05 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715922+0000", 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00044113900000000001 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:18745 3.a 3:51ba897a:::benchmark_data_smithi194_88551_object18744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:06.991268+0000", 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 558.15714770099999, 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72519465900000002, 2024-04-29T14:23:25.237 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 18745 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991268+0000", 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991268+0000", 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991269+0000", 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6199999999999997e-07 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991315+0000", 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6487999999999999e-05 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991320+0000", 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2039999999999996e-06 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991326+0000", 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7250000000000002e-06 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991357+0000", 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0629000000000001e-05 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:06.991360+0000", 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.653e-06 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.701235+0000", 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.709874216 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.701400+0000", 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165317 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.701575+0000", 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017452599999999999 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.705437+0000", 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038623939999999999 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.715974+0000", 2024-04-29T14:23:25.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010536943999999999 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.716017+0000", 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3154000000000001e-05 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:07.716463+0000", 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00044574700000000003 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:63888 3.e 3:74104bea:::benchmark_data_smithi194_88551_object63887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:24.890511+0000", 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 540.25790453399998, 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.209098494, 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.244 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 63888 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890511+0000", 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890511+0000", 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890513+0000", 2024-04-29T14:23:25.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8020000000000001e-06 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890562+0000", 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9156999999999997e-05 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890574+0000", 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1574e-05 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890581+0000", 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2559999999999996e-06 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890693+0000", 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111537 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890801+0000", 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107928 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:23:25.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.890948+0000", 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000146973 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.896888+0000", 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059409229999999999 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.099529+0000", 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20264083399999999 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.099565+0000", 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.612e-05 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.099610+0000", 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.439e-05 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:63893 3.8 3:166d9e62:::benchmark_data_smithi194_88551_object63892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:24.892888+0000", 2024-04-29T14:23:25.249 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 540.25552731400001, 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208170983, 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 63893 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892888+0000", 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892888+0000", 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892891+0000", 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0389999999999999e-06 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892917+0000", 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6872999999999998e-05 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892931+0000", 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3895e-05 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892939+0000", 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4159999999999998e-06 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.892990+0000", 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0850000000000003e-05 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.893107+0000", 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117689 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.893253+0000", 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000145315 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:24.896902+0000", 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003648999 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.100970+0000", 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204068056 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.101019+0000", 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9650000000000001e-05 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:25.101059+0000", 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0200999999999997e-05 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:76274 3.e 3:78f8db03:::benchmark_data_smithi194_88551_object76273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:30.852615+0000", 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 534.29580053799998, 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21036295799999999, 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.255 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 76274 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.852615+0000", 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.852615+0000", 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.852617+0000", 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4750000000000001e-06 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855259+0000", 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026425530000000002 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855271+0000", 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1609e-05 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855279+0000", 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7950000000000008e-06 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855332+0000", 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3758999999999998e-05 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855436+0000", 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103514 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.855554+0000", 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011763400000000001 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.857987+0000", 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002433068 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.062877+0000", 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20488995199999999 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.062935+0000", 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8443999999999998e-05 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.062978+0000", 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3155000000000003e-05 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25063.0:76286 3.8 3:10617bc5:::benchmark_data_smithi194_88551_object76285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-04-29T14:23:25.260 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:14:30.858407+0000", 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 534.29000881399998, 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.209131233, 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25063", 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1052405861", 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 76286 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858407+0000", 2024-04-29T14:23:25.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858407+0000", 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858409+0000", 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858476+0000", 2024-04-29T14:23:25.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7498999999999999e-05 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858487+0000", 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1069e-05 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.858494+0000", 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.860292+0000", 2024-04-29T14:23:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001797508 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.860416+0000", 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012426200000000001 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.860562+0000", 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014601699999999999 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:30.864911+0000", 2024-04-29T14:23:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043489059999999996 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.067425+0000", 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202513375 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.067493+0000", 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8307999999999997e-05 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:14:31.067538+0000", 2024-04-29T14:23:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5414000000000003e-05 2024-04-29T14:23:25.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:25.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:25.266 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:23:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:25.267 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:25.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:25.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:25.412 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:25.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:25.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:23:25.612 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:25.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:23:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:26.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:23:26.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:23:26.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 411, 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:23:26.209035+0000", 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:22:12.596258+0000", 2024-04-29T14:23:26.365 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:19:01.251051+0000", 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5806080, 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "nodeep-scrub", 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "noscrub", 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 29, 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:23:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 4, 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "luminous", 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:23:26.367 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:23:26.368 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:23:26.369 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:23:26.370 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:23:26.371 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:23:26.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 10, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 10, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.20000000298023224, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.373 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 4, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_2", 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:18:47.160984+0000", 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:23:26.374 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.375 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "326", 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:23:26.376 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:23:26.377 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.5, 2024-04-29T14:23:26.378 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 3.130000114440918, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.80000001192092896, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2.5, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.379 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.380 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.383 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:23:26.384 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.385 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:23:26.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:23:26.387 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 152, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.388 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.389 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:23:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:23:26.391 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.392 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.393 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:23:26.394 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.395 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 407, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.399 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 36, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 369, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 388, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 388, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 370, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.401 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.402 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2586630078 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/2586630078", 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/2586630078", 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/2586630078", 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/2586630078", 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.404 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.405 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.406 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 408, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.408 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.410 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.412 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.413 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:23:26.414 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:23:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 388, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:23:26.416 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:23:26.419 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.420 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.421 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.422 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:23:26.423 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 370, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.424 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.425 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:23:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:23:26.427 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:23:17.332311+0000", 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:23:26.428 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:53.643707+0000", 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.429 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:28.535076+0000", 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:23:14.808960+0000", 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.430 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:23:13.041888+0000", 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.431 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:26.135475+0000", 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 369 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:52.220678+0000", 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:23:15.542483+0000", 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.433 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:24.665708+0000", 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:21.483233+0000", 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:58.060054+0000", 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:23:26.435 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:22:47.912536+0000", 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [ 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "4.3", 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "mappings": [ 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "from": 11, 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: "to": 4 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.436 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [ 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "4.3", 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: 7, 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: 11 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:23:26.437 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:23:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:23:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:23:26.440 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:23:26.440 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:23:26.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:26 smithi121 ceph-mon[44533]: pgmap v1088: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 493 op/s; 2451/92264 objects misplaced (2.657%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:23:26.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1182566021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:26.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:26 smithi040 ceph-mon[39581]: pgmap v1088: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 493 op/s; 2451/92264 objects misplaced (2.657%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:23:26.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1182566021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:26.540 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:26 smithi194 ceph-mon[44089]: pgmap v1088: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 493 op/s; 2451/92264 objects misplaced (2.657%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:23:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1182566021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:27.189 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:27.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:27.215 INFO:tasks.thrashosds.thrasher:j is {'epoch': 411, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:23:26.209035+0000', 'last_up_change': '2024-04-29T14:22:12.596258+0000', 'last_in_change': '2024-04-29T14:19:01.251051+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-29T14:18:47.160984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '326', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 3.130000114440918, 'optimal_score': 0.800000011920929, 'raw_score_acting': 2, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 152, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 407, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 36, 'last_clean_end': 369, 'up_from': 388, 'up_thru': 388, 'down_at': 370, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 2586630078}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2586630078}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2586630078}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2586630078}]}, 'public_addr': '172.21.15.121:6804/2586630078', 'cluster_addr': '172.21.15.121:6805/2586630078', 'heartbeat_back_addr': '172.21.15.121:6807/2586630078', 'heartbeat_front_addr': '172.21.15.121:6806/2586630078', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 408, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 206, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 388, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:23:17.332311+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:22:53.643707+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:22:28.535076+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:23:14.808960+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:23:13.041888+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:26.135475+0000', 'dead_epoch': 369}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:52.220678+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:23:15.542483+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:22:24.665708+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:21.483233+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:58.060054+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:22:47.912536+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.3', 'mappings': [{'from': 11, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.3', 'osds': [7, 11]}], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:23:27.215 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:23:27.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:27.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:27 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:27.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1182566021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:27.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:27 smithi121 ceph-mon[44533]: osdmap e411: 12 total, 12 up, 8 in 2024-04-29T14:23:27.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3308343301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:23:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:27.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:27 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:27.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1182566021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:27.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:27 smithi040 ceph-mon[39581]: osdmap e411: 12 total, 12 up, 8 in 2024-04-29T14:23:27.497 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3308343301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:23:27.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:27 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1182566021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:27 smithi194 ceph-mon[44089]: osdmap e411: 12 total, 12 up, 8 in 2024-04-29T14:23:27.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3308343301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:27.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:27.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:28.036 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:28.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:28.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:28.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:28.212 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:28.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:28 smithi121 ceph-mon[44533]: pgmap v1090: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 1710/83480 objects misplaced (2.048%); 6.5 MiB/s, 103 objects/s recovering 2024-04-29T14:23:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:28.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:28 smithi040 ceph-mon[39581]: pgmap v1090: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 1710/83480 objects misplaced (2.048%); 6.5 MiB/s, 103 objects/s recovering 2024-04-29T14:23:28.491 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:28 smithi194 ceph-mon[44089]: pgmap v1090: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 1710/83480 objects misplaced (2.048%); 6.5 MiB/s, 103 objects/s recovering 2024-04-29T14:23:28.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:28.611 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:28.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:28.759 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:275094 4.a 4:56b0301b:::benchmark_data_smithi194_103039_object118411:head [delete] snapc 0=[] ondisk+write+known_if_redirected e411)", 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:28.745876+0000", 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.010315749000000001, 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010347943, 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:23:28.760 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 275094 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.745876+0000", 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.745876+0000", 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:28.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.745878+0000", 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1880000000000001e-06 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.745886+0000", 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3129999999999999e-06 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.745902+0000", 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5679e-05 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.745910+0000", 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.748336+0000", 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002425909 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.751037+0000", 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027012160000000002 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.751167+0000", 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012976799999999999 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:275104 4.a 4:506d95c9:::benchmark_data_smithi194_103039_object118421:head [delete] snapc 0=[] ondisk+write+known_if_redirected e411)", 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:28.754796+0000", 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.001395688, 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001509933, 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:23:28.764 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 275104 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754796+0000", 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754796+0000", 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754798+0000", 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2079999999999999e-06 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754808+0000", 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4360000000000008e-06 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754821+0000", 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3294e-05 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754829+0000", 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2519999999999999e-06 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755681+0000", 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000852103 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755803+0000", 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012176999999999999 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755913+0000", 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010964900000000001 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:275105 4.a 4.8f557bea (undecoded) ondisk+write+known_if_redirected e411)", 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:28.754844+0000", 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.001348085, 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015330470000000001, 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:23:28.768 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 275105 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754844+0000", 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754844+0000", 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754844+0000", 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-07 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754847+0000", 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7439999999999999e-06 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754853+0000", 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2879999999999998e-06 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.754858+0000", 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8919999999999999e-06 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.25522.0:275094 4.a e411/323)", 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:28.755848+0000", 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.000343796, 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058019300000000001, 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:28.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755848+0000", 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755848+0000", 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755850+0000", 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6750000000000001e-06 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755855+0000", 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6370000000000004e-06 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755865+0000", 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6390000000000004e-06 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755872+0000", 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5780000000000001e-06 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25522.0:275099 4.c e411/323 4:3917c23e:::benchmark_data_smithi194_103039_object118416:head v 411'17245, mlcod=411'17241)", 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:28.755891+0000", 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00030104600000000002, 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058757799999999995, 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755891+0000", 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755891+0000", 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755900+0000", 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1840000000000005e-06 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755907+0000", 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2910000000000002e-06 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:23:28.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755915+0000", 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9400000000000002e-06 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755921+0000", 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0900000000000001e-06 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755970+0000", 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8835999999999998e-05 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:23:28.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:28.755991+0000", 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1067999999999999e-05 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 5 2024-04-29T14:23:28.777 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:23:28.791 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:28.821 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:28.847 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:28.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:28.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:29.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:23:29.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:29.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:29.679 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:29.707 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:29.731 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:29.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:29.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:29.847 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:23:29.892 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:29.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:29.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:30.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:23:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:30.178 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:23:30.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:23:30.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: pgmap v1091: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 1710/82198 objects misplaced (2.080%); 5.8 MiB/s, 92 objects/s recovering 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1885053861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.237 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:23:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:30.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:23:30.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: pgmap v1091: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 1710/82198 objects misplaced (2.080%); 5.8 MiB/s, 92 objects/s recovering 2024-04-29T14:23:30.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1885053861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:30.469 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:30.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: pgmap v1091: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 1710/82198 objects misplaced (2.080%); 5.8 MiB/s, 92 objects/s recovering 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1885053861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:30.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:30.670 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:31.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:31.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:31.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:31.181 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:31.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:31.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:31 smithi121 ceph-mon[44533]: pgmap v1092: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 1710/75822 objects misplaced (2.255%); 5.8 MiB/s, 92 objects/s recovering 2024-04-29T14:23:31.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:31.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:31.526 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:31.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:31 smithi194 ceph-mon[44089]: pgmap v1092: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 1710/75822 objects misplaced (2.255%); 5.8 MiB/s, 92 objects/s recovering 2024-04-29T14:23:31.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:31.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:31 smithi040 ceph-mon[39581]: pgmap v1092: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 1710/75822 objects misplaced (2.255%); 5.8 MiB/s, 92 objects/s recovering 2024-04-29T14:23:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:31.780 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:31.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:23:31.808 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:23:31.883 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:31.922 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:31.964 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:31.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:32.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:32.135 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:23:32.135 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:23:32.140 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:32.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:32.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:32.250 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:32 smithi194 ceph-mon[44089]: from='client.16068 v1:172.21.15.40:0/2363484068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:23:32.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:32.344 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:32.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:32.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:32.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:32.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:32 smithi121 ceph-mon[44533]: from='client.16068 v1:172.21.15.40:0/2363484068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:23:32.545 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:32.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:32 smithi040 ceph-mon[39581]: from='client.16068 v1:172.21.15.40:0/2363484068' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:23:32.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:23:33.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:33.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:33.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:23:33.113 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:33.144 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:33.257 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:33 smithi121 ceph-mon[44533]: pgmap v1093: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 1063/72140 objects misplaced (1.474%); 6.3 MiB/s, 101 objects/s recovering 2024-04-29T14:23:33.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:33 smithi040 ceph-mon[39581]: pgmap v1093: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 1063/72140 objects misplaced (1.474%); 6.3 MiB/s, 101 objects/s recovering 2024-04-29T14:23:33.441 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:33.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:33 smithi194 ceph-mon[44089]: pgmap v1093: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 1063/72140 objects misplaced (1.474%); 6.3 MiB/s, 101 objects/s recovering 2024-04-29T14:23:33.608 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1092,"stamp":"2024-04-29T14:23:31.090308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2484929080,"num_objects":37911,"num_object_clones":0,"num_object_copies":75822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1710,"num_objects_unfound":0,"num_objects_dirty":37911,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":275634,"num_write_kb":10030459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":933,"num_bytes_recovered":62670976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161020,"ondisk_log_size":161020,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7745796,"kb_used_data":4956800,"kb_used_omap":222,"kb_used_meta":2788513,"kb_avail":742215420,"statfs":{"total":767960285184,"available":760028590080,"internally_reserved":0,"allocated":5075763200,"data_stored":5072741272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":227868,"internal_metadata":2855437796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-646905856,"num_objects":-9871,"num_object_clones":0,"num_object_copies":-19742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1710,"num_objects_unfound":0,"num_objects_dirty":-9871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9871,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":925,"num_bytes_recovered":60555264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001974"},"pg_stats":[{"pgid":"4.f","version":"411'17362","reported_seq":17773,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:30.220185+0000","last_change":"2024-04-29T14:23:04.730509+0000","last_active":"2024-04-29T14:23:30.220185+0000","last_peered":"2024-04-29T14:23:30.220185+0000","last_clean":"2024-04-29T14:23:30.220185+0000","last_became_active":"2024-04-29T14:18:49.370395+0000","last_became_peered":"2024-04-29T14:18:49.370395+0000","last_unstale":"2024-04-29T14:23:30.220185+0000","last_undegraded":"2024-04-29T14:23:30.220185+0000","last_fullsized":"2024-04-29T14:23:30.220185+0000","mapping_epoch":323,"log_start":"346'7300","ondisk_log_start":"346'7300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15845","last_scrub_stamp":"2024-04-29T14:23:04.730222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:04.730222+0000","objects_scrubbed":3820,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:22.261116+0000","scrub_duration":649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149422080,"num_objects":2280,"num_object_clones":0,"num_object_copies":4560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17362,"num_write_kb":628544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"411'17374","reported_seq":17787,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:30.993134+0000","last_change":"2024-04-29T14:23:00.699458+0000","last_active":"2024-04-29T14:23:30.993134+0000","last_peered":"2024-04-29T14:23:30.993134+0000","last_clean":"2024-04-29T14:23:30.993134+0000","last_became_active":"2024-04-29T14:18:49.152699+0000","last_became_peered":"2024-04-29T14:18:49.152699+0000","last_unstale":"2024-04-29T14:23:30.993134+0000","last_undegraded":"2024-04-29T14:23:30.993134+0000","last_fullsized":"2024-04-29T14:23:30.993134+0000","mapping_epoch":323,"log_start":"347'7300","ondisk_log_start":"347'7300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15536","last_scrub_stamp":"2024-04-29T14:23:00.699185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:00.699185+0000","objects_scrubbed":4092,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:13.273646+0000","scrub_duration":475,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17374,"num_write_kb":627584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"411'17480","reported_seq":17893,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:31.059465+0000","last_change":"2024-04-29T14:23:00.094580+0000","last_active":"2024-04-29T14:23:31.059465+0000","last_peered":"2024-04-29T14:23:31.059465+0000","last_clean":"2024-04-29T14:23:31.059465+0000","last_became_active":"2024-04-29T14:18:49.155690+0000","last_became_peered":"2024-04-29T14:18:49.155690+0000","last_unstale":"2024-04-29T14:23:31.059465+0000","last_undegraded":"2024-04-29T14:23:31.059465+0000","last_fullsized":"2024-04-29T14:23:31.059465+0000","mapping_epoch":323,"log_start":"347'7400","ondisk_log_start":"347'7400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15614","last_scrub_stamp":"2024-04-29T14:23:00.094315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:00.094315+0000","objects_scrubbed":4176,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:03.040952+0000","scrub_duration":836,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149618712,"num_objects":2284,"num_object_clones":0,"num_object_copies":4568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17480,"num_write_kb":632385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"411'17385","reported_seq":17795,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:31.022144+0000","last_change":"2024-04-29T14:23:03.139449+0000","last_active":"2024-04-29T14:23:31.022144+0000","last_peered":"2024-04-29T14:23:31.022144+0000","last_clean":"2024-04-29T14:23:31.022144+0000","last_became_active":"2024-04-29T14:18:49.156158+0000","last_became_peered":"2024-04-29T14:18:49.156158+0000","last_unstale":"2024-04-29T14:23:31.022144+0000","last_undegraded":"2024-04-29T14:23:31.022144+0000","last_fullsized":"2024-04-29T14:23:31.022144+0000","mapping_epoch":323,"log_start":"347'7300","ondisk_log_start":"347'7300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15757","last_scrub_stamp":"2024-04-29T14:23:03.139310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:03.139310+0000","objects_scrubbed":3920,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:11.760827+0000","scrub_duration":891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148570112,"num_objects":2267,"num_object_clones":0,"num_object_copies":4534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17385,"num_write_kb":628864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"411'17737","reported_seq":18147,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:31.048227+0000","last_change":"2024-04-29T14:23:13.641912+0000","last_active":"2024-04-29T14:23:31.048227+0000","last_peered":"2024-04-29T14:23:31.048227+0000","last_clean":"2024-04-29T14:23:31.048227+0000","last_became_active":"2024-04-29T14:22:41.142887+0000","last_became_peered":"2024-04-29T14:22:41.142887+0000","last_unstale":"2024-04-29T14:23:31.048227+0000","last_undegraded":"2024-04-29T14:23:31.048227+0000","last_fullsized":"2024-04-29T14:23:31.048227+0000","mapping_epoch":397,"log_start":"349'7700","ondisk_log_start":"349'7700","created":323,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'16829","last_scrub_stamp":"2024-04-29T14:23:13.641816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:13.641816+0000","objects_scrubbed":3207,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:38.097338+0000","scrub_duration":576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149487616,"num_objects":2281,"num_object_clones":0,"num_object_copies":4562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17737,"num_write_kb":640576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"411'17182","reported_seq":17542,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:30.164357+0000","last_change":"2024-04-29T14:21:54.688108+0000","last_active":"2024-04-29T14:23:30.164357+0000","last_peered":"2024-04-29T14:23:30.164357+0000","last_clean":"2024-04-29T14:23:30.164357+0000","last_became_active":"2024-04-29T14:18:49.156474+0000","last_became_peered":"2024-04-29T14:18:49.156474+0000","last_unstale":"2024-04-29T14:23:30.164357+0000","last_undegraded":"2024-04-29T14:23:30.164357+0000","last_fullsized":"2024-04-29T14:23:30.164357+0000","mapping_epoch":323,"log_start":"346'7100","ondisk_log_start":"346'7100","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12343","last_scrub_stamp":"2024-04-29T14:21:54.687953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:54.687953+0000","objects_scrubbed":7201,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:30.947698+0000","scrub_duration":602,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154796032,"num_objects":2362,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17182,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"411'17224","reported_seq":17633,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:30.155852+0000","last_change":"2024-04-29T14:22:58.989877+0000","last_active":"2024-04-29T14:23:30.155852+0000","last_peered":"2024-04-29T14:23:30.155852+0000","last_clean":"2024-04-29T14:23:30.155852+0000","last_became_active":"2024-04-29T14:18:49.371178+0000","last_became_peered":"2024-04-29T14:18:49.371178+0000","last_unstale":"2024-04-29T14:23:30.155852+0000","last_undegraded":"2024-04-29T14:23:30.155852+0000","last_fullsized":"2024-04-29T14:23:30.155852+0000","mapping_epoch":323,"log_start":"346'7200","ondisk_log_start":"346'7200","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15374","last_scrub_stamp":"2024-04-29T14:22:58.989714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:58.989714+0000","objects_scrubbed":4131,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:01.626190+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148504576,"num_objects":2266,"num_object_clones":0,"num_object_copies":4532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17224,"num_write_kb":623680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"410'16982","reported_seq":17383,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-04-29T14:23:26.075121+0000","last_change":"2024-04-29T14:23:10.436926+0000","last_active":"2024-04-29T14:23:26.075121+0000","last_peered":"2024-04-29T14:23:26.075121+0000","last_clean":"2024-04-29T14:23:26.075121+0000","last_became_active":"2024-04-29T14:18:49.370541+0000","last_became_peered":"2024-04-29T14:18:49.370541+0000","last_unstale":"2024-04-29T14:23:26.075121+0000","last_undegraded":"2024-04-29T14:23:26.075121+0000","last_fullsized":"2024-04-29T14:23:26.075121+0000","mapping_epoch":323,"log_start":"345'6900","ondisk_log_start":"345'6900","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'16182","last_scrub_stamp":"2024-04-29T14:23:10.436763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:10.436763+0000","objects_scrubbed":3424,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:16.646262+0000","scrub_duration":934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169345024,"num_objects":2584,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16982,"num_write_kb":626112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"411'17026","reported_seq":17387,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:27.748667+0000","last_change":"2024-04-29T14:21:43.967168+0000","last_active":"2024-04-29T14:23:27.748667+0000","last_peered":"2024-04-29T14:23:27.748667+0000","last_clean":"2024-04-29T14:23:27.748667+0000","last_became_active":"2024-04-29T14:18:49.151382+0000","last_became_peered":"2024-04-29T14:18:49.151382+0000","last_unstale":"2024-04-29T14:23:27.748667+0000","last_undegraded":"2024-04-29T14:23:27.748667+0000","last_fullsized":"2024-04-29T14:23:27.748667+0000","mapping_epoch":323,"log_start":"346'7000","ondisk_log_start":"346'7000","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'12296","last_scrub_stamp":"2024-04-29T14:21:43.966992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:43.966992+0000","objects_scrubbed":7200,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:29.088749+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":4940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17026,"num_write_kb":623872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"411'17481","reported_seq":17897,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:30.214068+0000","last_change":"2024-04-29T14:22:58.908796+0000","last_active":"2024-04-29T14:23:30.214068+0000","last_peered":"2024-04-29T14:23:30.214068+0000","last_clean":"2024-04-29T14:23:30.214068+0000","last_became_active":"2024-04-29T14:18:49.157084+0000","last_became_peered":"2024-04-29T14:18:49.157084+0000","last_unstale":"2024-04-29T14:23:30.214068+0000","last_undegraded":"2024-04-29T14:23:30.214068+0000","last_fullsized":"2024-04-29T14:23:30.214068+0000","mapping_epoch":323,"log_start":"347'7400","ondisk_log_start":"347'7400","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15612","last_scrub_stamp":"2024-04-29T14:22:58.908548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:58.908548+0000","objects_scrubbed":4178,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:01.580330+0000","scrub_duration":792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150142976,"num_objects":2291,"num_object_clones":0,"num_object_copies":4582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17481,"num_write_kb":632704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7],"acting":[9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"411'16513","reported_seq":16916,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:27.698292+0000","last_change":"2024-04-29T14:23:01.894806+0000","last_active":"2024-04-29T14:23:27.698292+0000","last_peered":"2024-04-29T14:23:27.698292+0000","last_clean":"2024-04-29T14:23:27.698292+0000","last_became_active":"2024-04-29T14:18:49.156484+0000","last_became_peered":"2024-04-29T14:18:49.156484+0000","last_unstale":"2024-04-29T14:23:27.698292+0000","last_undegraded":"2024-04-29T14:23:27.698292+0000","last_fullsized":"2024-04-29T14:23:27.698292+0000","mapping_epoch":323,"log_start":"345'6500","ondisk_log_start":"345'6500","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14993","last_scrub_stamp":"2024-04-29T14:23:01.894626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:23:01.894626+0000","objects_scrubbed":3926,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:06.815555+0000","scrub_duration":829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156041216,"num_objects":2381,"num_object_clones":0,"num_object_copies":4762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16513,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"410'19376","reported_seq":21709,"reported_epoch":410,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:23:26.126291+0000","last_change":"2024-04-29T14:23:17.109804+0000","last_active":"2024-04-29T14:23:26.126291+0000","last_peered":"2024-04-29T14:23:26.126291+0000","last_clean":"2024-04-29T14:23:14.157510+0000","last_became_active":"2024-04-29T14:23:17.001762+0000","last_became_peered":"2024-04-29T14:23:17.001762+0000","last_unstale":"2024-04-29T14:23:26.126291+0000","last_undegraded":"2024-04-29T14:23:26.126291+0000","last_fullsized":"2024-04-29T14:23:26.126291+0000","mapping_epoch":408,"log_start":"355'9300","ondisk_log_start":"355'9300","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'14737","last_scrub_stamp":"2024-04-29T14:21:54.780714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:21:54.780714+0000","objects_scrubbed":7177,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330368,"num_objects":2538,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1710,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17046,"num_write_kb":626688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":925,"num_bytes_recovered":60555264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[7,11],"avail_no_missing":["7","11"],"object_location_counts":[{"shards":"7,11","objects":2538}],"blocked_by":[],"up_primary":4,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"411'17358","reported_seq":17729,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:30.238191+0000","last_change":"2024-04-29T14:22:18.591338+0000","last_active":"2024-04-29T14:23:30.238191+0000","last_peered":"2024-04-29T14:23:30.238191+0000","last_clean":"2024-04-29T14:23:30.238191+0000","last_became_active":"2024-04-29T14:22:13.613928+0000","last_became_peered":"2024-04-29T14:22:13.613928+0000","last_unstale":"2024-04-29T14:23:30.238191+0000","last_undegraded":"2024-04-29T14:23:30.238191+0000","last_fullsized":"2024-04-29T14:23:30.238191+0000","mapping_epoch":388,"log_start":"346'7300","ondisk_log_start":"346'7300","created":323,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'12766","last_scrub_stamp":"2024-04-29T14:22:18.591115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:18.591115+0000","objects_scrubbed":6887,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:42.749342+0000","scrub_duration":1374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147587072,"num_objects":2252,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17358,"num_write_kb":627520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"410'17140","reported_seq":17474,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-04-29T14:23:26.170028+0000","last_change":"2024-04-29T14:22:20.297495+0000","last_active":"2024-04-29T14:23:26.170028+0000","last_peered":"2024-04-29T14:23:26.170028+0000","last_clean":"2024-04-29T14:23:26.170028+0000","last_became_active":"2024-04-29T14:22:13.607474+0000","last_became_peered":"2024-04-29T14:22:13.607474+0000","last_unstale":"2024-04-29T14:23:26.170028+0000","last_undegraded":"2024-04-29T14:23:26.170028+0000","last_fullsized":"2024-04-29T14:23:26.170028+0000","mapping_epoch":388,"log_start":"346'7100","ondisk_log_start":"346'7100","created":323,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'13020","last_scrub_stamp":"2024-04-29T14:22:20.297257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:20.297257+0000","objects_scrubbed":6680,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:23:29.891325+0000","scrub_duration":1237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165937152,"num_objects":2532,"num_object_clones":0,"num_object_copies":5064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17140,"num_write_kb":629504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"410'16935","reported_seq":17338,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-04-29T14:23:26.103889+0000","last_change":"2024-04-29T14:22:59.405191+0000","last_active":"2024-04-29T14:23:26.103889+0000","last_peered":"2024-04-29T14:23:26.103889+0000","last_clean":"2024-04-29T14:23:26.103889+0000","last_became_active":"2024-04-29T14:18:49.158015+0000","last_became_peered":"2024-04-29T14:18:49.158015+0000","last_unstale":"2024-04-29T14:23:26.103889+0000","last_undegraded":"2024-04-29T14:23:26.103889+0000","last_fullsized":"2024-04-29T14:23:26.103889+0000","mapping_epoch":323,"log_start":"346'6900","ondisk_log_start":"346'6900","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15489","last_scrub_stamp":"2024-04-29T14:22:59.404935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:59.404935+0000","objects_scrubbed":4030,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:02.608640+0000","scrub_duration":903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":5134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16935,"num_write_kb":624064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"411'17228","reported_seq":17635,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-04-29T14:23:30.182221+0000","last_change":"2024-04-29T14:22:59.972343+0000","last_active":"2024-04-29T14:23:30.182221+0000","last_peered":"2024-04-29T14:23:30.182221+0000","last_clean":"2024-04-29T14:23:30.182221+0000","last_became_active":"2024-04-29T14:18:49.156512+0000","last_became_peered":"2024-04-29T14:18:49.156512+0000","last_unstale":"2024-04-29T14:23:30.182221+0000","last_undegraded":"2024-04-29T14:23:30.182221+0000","last_fullsized":"2024-04-29T14:23:30.182221+0000","mapping_epoch":323,"log_start":"346'7200","ondisk_log_start":"346'7200","created":323,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15439","last_scrub_stamp":"2024-04-29T14:22:59.972178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:18:48.136527+0000","last_clean_scrub_stamp":"2024-04-29T14:22:59.972178+0000","objects_scrubbed":4122,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:26.200777+0000","scrub_duration":865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151781376,"num_objects":2316,"num_object_clones":0,"num_object_copies":4632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17228,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1044,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-04-29T14:23:18.201592+0000","last_change":"2024-04-29T14:23:01.538258+0000","last_active":"2024-04-29T14:23:18.201592+0000","last_peered":"2024-04-29T14:23:18.201592+0000","last_clean":"2024-04-29T14:23:18.201592+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:23:18.201592+0000","last_undegraded":"2024-04-29T14:23:18.201592+0000","last_fullsized":"2024-04-29T14:23:18.201592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:23:01.538129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:23:01.538129+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:24:25.882602+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2484338712,"num_objects":37909,"num_object_clones":0,"num_object_copies":75818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1710,"num_objects_unfound":0,"num_objects_dirty":37909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275453,"num_write_kb":10027521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":925,"num_bytes_recovered":60555264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5028651008,"data_stored":5028642864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":212282,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122247,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1125656,"kb_used_data":729360,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92619496,"statfs":{"total":95995035648,"available":94842363904,"internally_reserved":0,"allocated":746864640,"data_stored":746483674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":405797316},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383883,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1030804,"kb_used_data":801096,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92714348,"statfs":{"total":95995035648,"available":94939492352,"internally_reserved":0,"allocated":820322304,"data_stored":819944890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":235207104},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645521,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931220,"kb_used_data":589068,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92813932,"statfs":{"total":95995035648,"available":95041466368,"internally_reserved":0,"allocated":603205632,"data_stored":602824166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234201,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1236812,"kb_used_data":822468,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92508340,"statfs":{"total":95995035648,"available":94728540160,"internally_reserved":0,"allocated":842207232,"data_stored":841832542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528543,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102416,"kb_used_data":597832,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92642736,"statfs":{"total":95995035648,"available":94866161664,"internally_reserved":0,"allocated":612179968,"data_stored":611802574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516684230},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738622,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758096,"kb_used_data":438152,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92987056,"statfs":{"total":95995035648,"available":95218745344,"internally_reserved":0,"allocated":448667648,"data_stored":448290254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182639,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986410,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1189268,"kb_used_data":656716,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92555884,"statfs":{"total":95995035648,"available":94777225216,"internally_reserved":0,"allocated":672477184,"data_stored":672095718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":388,"seq":1666447310864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371524,"kb_used_data":322108,"kb_used_omap":211,"kb_used_meta":49196,"kb_avail":93373628,"statfs":{"total":95995035648,"available":95614595072,"internally_reserved":0,"allocated":329838592,"data_stored":329467454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":216698,"internal_metadata":50377094},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":444792832,"data_stored":444792832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":670699520,"data_stored":670695448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":328597504,"data_stored":328597504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":212282,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":606928896,"data_stored":606928896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":833421312,"data_stored":833421312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":600576000,"data_stored":600571928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":804388864,"data_stored":804388864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":739246080,"data_stored":739246080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:23:33.609 INFO:tasks.thrashosds.thrasher:Setting 4.a to [6, 11] 2024-04-29T14:23:33.610 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.a', '6', '11'] 2024-04-29T14:23:33.610 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 4.a 6 11 2024-04-29T14:23:33.612 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:33.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:33.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:33.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:33.756 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:33.784 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:33.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:33.890 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:33.918 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:33.942 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:33.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:34.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:34.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:34.485 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:23:34.491 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2441792419' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:23:34.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2441792419' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:23:34.861 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2441792419' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:23:35.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:35.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:35.217 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:35.261 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:35.288 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:35.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:35.399 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:35.458 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:35.486 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:35.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:35.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:35.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:35 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:35.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2441792419' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:23:35.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:35 smithi040 ceph-mon[39581]: osdmap e412: 12 total, 12 up, 8 in 2024-04-29T14:23:35.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:35.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:35.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:35 smithi040 ceph-mon[39581]: pgmap v1095: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 638 op/s; 1063/72140 objects misplaced (1.474%); 3.4 MiB/s, 54 objects/s recovering 2024-04-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:35.751 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:35.753 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:35.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:35.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:35 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:35.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2441792419' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:23:35.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:35 smithi194 ceph-mon[44089]: osdmap e412: 12 total, 12 up, 8 in 2024-04-29T14:23:35.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:35.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:35.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:35 smithi194 ceph-mon[44089]: pgmap v1095: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 638 op/s; 1063/72140 objects misplaced (1.474%); 3.4 MiB/s, 54 objects/s recovering 2024-04-29T14:23:35.823 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:23:35.823 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:23:35.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:35 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:35.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2441792419' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:23:35.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:35 smithi121 ceph-mon[44533]: osdmap e412: 12 total, 12 up, 8 in 2024-04-29T14:23:35.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:35.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:35.851 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:35 smithi121 ceph-mon[44533]: pgmap v1095: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 2.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 638 op/s; 1063/72140 objects misplaced (1.474%); 3.4 MiB/s, 54 objects/s recovering 2024-04-29T14:23:35.858 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:35.887 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:36.011 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:36.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:36.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:36.147 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:36.167 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:36.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:36.274 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:36.302 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:36.331 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:36.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:36.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:36.468 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:36.499 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:36.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:36 smithi121 ceph-mon[44533]: 4.7 scrub starts 2024-04-29T14:23:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:36 smithi121 ceph-mon[44533]: 4.7 scrub ok 2024-04-29T14:23:36.513 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:36 smithi121 ceph-mon[44533]: osdmap e413: 12 total, 12 up, 8 in 2024-04-29T14:23:36.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:36 smithi121 ceph-mon[44533]: 4.1 scrub starts 2024-04-29T14:23:36.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:36 smithi194 ceph-mon[44089]: 4.7 scrub starts 2024-04-29T14:23:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:36 smithi194 ceph-mon[44089]: 4.7 scrub ok 2024-04-29T14:23:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:36 smithi194 ceph-mon[44089]: osdmap e413: 12 total, 12 up, 8 in 2024-04-29T14:23:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:36 smithi194 ceph-mon[44089]: 4.1 scrub starts 2024-04-29T14:23:36.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:36.633 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:36.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:23:36.809 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:36.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:36 smithi040 ceph-mon[39581]: 4.7 scrub starts 2024-04-29T14:23:36.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:36 smithi040 ceph-mon[39581]: 4.7 scrub ok 2024-04-29T14:23:36.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:36 smithi040 ceph-mon[39581]: osdmap e413: 12 total, 12 up, 8 in 2024-04-29T14:23:36.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:36 smithi040 ceph-mon[39581]: 4.1 scrub starts 2024-04-29T14:23:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:36.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:36.921 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:36.959 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:36.982 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:36.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:37.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:37.154 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:37.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:37.481 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:37.512 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:37.544 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:37.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:37.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:37.693 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:37.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:37.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:37.909 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:38.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:38.363 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:38.393 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:38 smithi194 ceph-mon[44089]: 4.1 scrub ok 2024-04-29T14:23:38.393 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:38 smithi194 ceph-mon[44089]: pgmap v1097: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 461/63322 objects misplaced (0.728%); 7.5 MiB/s, 119 objects/s recovering 2024-04-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:38.426 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:38.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:38.526 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:23:38.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:38.550 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:23:38.571 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:23:38.579 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:38 smithi121 ceph-mon[44533]: 4.1 scrub ok 2024-04-29T14:23:38.579 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:38 smithi121 ceph-mon[44533]: pgmap v1097: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 461/63322 objects misplaced (0.728%); 7.5 MiB/s, 119 objects/s recovering 2024-04-29T14:23:38.591 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:38.592 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:23:38.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:38 smithi040 ceph-mon[39581]: 4.1 scrub ok 2024-04-29T14:23:38.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:38 smithi040 ceph-mon[39581]: pgmap v1097: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 461/63322 objects misplaced (0.728%); 7.5 MiB/s, 119 objects/s recovering 2024-04-29T14:23:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:38.642 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:23:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:38.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:38.676 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:23:38.695 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:23:38.723 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:23:38.743 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:23:38.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:38.766 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:23:38.788 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:23:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:38.809 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:23:38.810 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:23:38.811 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:23:38.811 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:23:38.811 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:23:39.151 INFO:teuthology.orchestra.run.smithi040.stderr:set 4.a pg_upmap mapping to [6,11] 2024-04-29T14:23:39.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:39.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:39.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:39.303 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:23:39.310 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/48376627' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [6, 11]}]: dispatch 2024-04-29T14:23:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2465831937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: 4.a scrub starts 2024-04-29T14:23:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.445 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:23:39.446 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:23:39.446 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:23:39.446 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/48376627' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [6, 11]}]: dispatch 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2465831937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: 4.a scrub starts 2024-04-29T14:23:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/48376627' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [6, 11]}]: dispatch 2024-04-29T14:23:39.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2465831937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: 4.a scrub starts 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:39.863 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:39.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:39.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:39.995 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:40.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:23:40.359 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:40.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: pgmap v1098: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 461/62122 objects misplaced (0.742%); 7.5 MiB/s, 119 objects/s recovering 2024-04-29T14:23:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: 4.a scrub ok 2024-04-29T14:23:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/48376627' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [6, 11]}]': finished 2024-04-29T14:23:40.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:40 smithi040 ceph-mon[39581]: osdmap e414: 12 total, 12 up, 8 in 2024-04-29T14:23:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: pgmap v1098: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 461/62122 objects misplaced (0.742%); 7.5 MiB/s, 119 objects/s recovering 2024-04-29T14:23:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: 4.a scrub ok 2024-04-29T14:23:40.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/48376627' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [6, 11]}]': finished 2024-04-29T14:23:40.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:40 smithi121 ceph-mon[44533]: osdmap e414: 12 total, 12 up, 8 in 2024-04-29T14:23:40.445 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:40.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:40.549 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: pgmap v1098: 17 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 15 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 461/62122 objects misplaced (0.742%); 7.5 MiB/s, 119 objects/s recovering 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: 4.a scrub ok 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/48376627' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [6, 11]}]': finished 2024-04-29T14:23:40.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:40 smithi194 ceph-mon[44089]: osdmap e414: 12 total, 12 up, 8 in 2024-04-29T14:23:40.580 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:40.602 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:40.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:40.706 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:40.738 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:40.757 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:40.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:40.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:23:40.891 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:41.139 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:23:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:41.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:23:41.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:41.271 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:41.295 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:41.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:41.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:41 smithi040 ceph-mon[39581]: osdmap e415: 12 total, 12 up, 8 in 2024-04-29T14:23:41.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:41.431 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:41.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:41 smithi194 ceph-mon[44089]: osdmap e415: 12 total, 12 up, 8 in 2024-04-29T14:23:41.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:41 smithi121 ceph-mon[44533]: osdmap e415: 12 total, 12 up, 8 in 2024-04-29T14:23:41.635 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:41.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:41.682 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:23:41.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:41.974 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:41.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:42.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:23:42.111 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:42.134 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:23:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:42.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:23:42.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:42.301 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:42.306 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:42 smithi121 ceph-mon[44533]: pgmap v1101: 17 pgs: 1 remapped+peering, 1 peering, 1 active+clean+scrubbing, 14 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s; 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:23:42.307 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:42 smithi121 ceph-mon[44533]: osdmap e416: 12 total, 12 up, 8 in 2024-04-29T14:23:42.307 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:42 smithi121 ceph-mon[44533]: 4.a starting backfill to osd.11 from (0'0,0'0] MAX to 413'17771 2024-04-29T14:23:42.307 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:42 smithi121 ceph-mon[44533]: 4.3 scrub starts 2024-04-29T14:23:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:42.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:42 smithi040 ceph-mon[39581]: pgmap v1101: 17 pgs: 1 remapped+peering, 1 peering, 1 active+clean+scrubbing, 14 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s; 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:23:42.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:42 smithi040 ceph-mon[39581]: osdmap e416: 12 total, 12 up, 8 in 2024-04-29T14:23:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:42 smithi040 ceph-mon[39581]: 4.a starting backfill to osd.11 from (0'0,0'0] MAX to 413'17771 2024-04-29T14:23:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:42 smithi040 ceph-mon[39581]: 4.3 scrub starts 2024-04-29T14:23:42.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:42.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:42.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:42.421 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:23:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:23:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:23:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:23:42.422 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:23:42.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:42.521 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:42.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:42.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:42 smithi194 ceph-mon[44089]: pgmap v1101: 17 pgs: 1 remapped+peering, 1 peering, 1 active+clean+scrubbing, 14 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s; 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:23:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:42 smithi194 ceph-mon[44089]: osdmap e416: 12 total, 12 up, 8 in 2024-04-29T14:23:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:42 smithi194 ceph-mon[44089]: 4.a starting backfill to osd.11 from (0'0,0'0] MAX to 413'17771 2024-04-29T14:23:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:42 smithi194 ceph-mon[44089]: 4.3 scrub starts 2024-04-29T14:23:42.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:43.179 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:23:43.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:23:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:43 smithi194 ceph-mon[44089]: 4.3 scrub ok 2024-04-29T14:23:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3433686286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:43.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:43 smithi040 ceph-mon[39581]: 4.3 scrub ok 2024-04-29T14:23:43.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3433686286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:43.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:43 smithi121 ceph-mon[44533]: 4.3 scrub ok 2024-04-29T14:23:43.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3433686286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:23:44.058 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:44.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:44.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:44.244 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:44.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:44.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:44 smithi121 ceph-mon[44533]: pgmap v1103: 17 pgs: 1 remapped+peering, 1 peering, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 4.4 MiB/s, 71 objects/s recovering 2024-04-29T14:23:44.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:44.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:44.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:44 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:23:44.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3433686286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:44.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:44 smithi121 ceph-mon[44533]: osdmap e417: 12 total, 12 up, 8 in 2024-04-29T14:23:44.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:44.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:44 smithi040 ceph-mon[39581]: pgmap v1103: 17 pgs: 1 remapped+peering, 1 peering, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 4.4 MiB/s, 71 objects/s recovering 2024-04-29T14:23:44.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:44.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:44.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:44 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:23:44.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3433686286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:44.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:44 smithi040 ceph-mon[39581]: osdmap e417: 12 total, 12 up, 8 in 2024-04-29T14:23:44.418 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:44 smithi194 ceph-mon[44089]: pgmap v1103: 17 pgs: 1 remapped+peering, 1 peering, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 4.4 MiB/s, 71 objects/s recovering 2024-04-29T14:23:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:44 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:23:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3433686286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:23:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:44 smithi194 ceph-mon[44089]: osdmap e417: 12 total, 12 up, 8 in 2024-04-29T14:23:45.302 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:23:45.322 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:23:45.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:23:45.364 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:23:45.389 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:23:45.411 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:23:45.437 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:23:45.447 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 5, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:23:45.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-29T14:23:45.459 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:23:45.484 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:23:45.511 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:23:45.533 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:23:45.555 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:23:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:23:46.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:23:46.413 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:46.452 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: pgmap v1105: 17 pgs: 1 remapped+peering, 1 peering, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 4.5 MiB/s, 71 objects/s recovering 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1206785066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: pgmap v1105: 17 pgs: 1 remapped+peering, 1 peering, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 4.5 MiB/s, 71 objects/s recovering 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1206785066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: pgmap v1105: 17 pgs: 1 remapped+peering, 1 peering, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 4.5 MiB/s, 71 objects/s recovering 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1206785066' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:46.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:47.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:47.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:47.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:48.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:23:48.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:48 smithi194 ceph-mon[44089]: pgmap v1106: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1036/47776 objects misplaced (2.168%); 8.6 MiB/s, 138 objects/s recovering 2024-04-29T14:23:48.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:48 smithi040 ceph-mon[39581]: pgmap v1106: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1036/47776 objects misplaced (2.168%); 8.6 MiB/s, 138 objects/s recovering 2024-04-29T14:23:48.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:48 smithi121 ceph-mon[44533]: pgmap v1106: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.5 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1036/47776 objects misplaced (2.168%); 8.6 MiB/s, 138 objects/s recovering 2024-04-29T14:23:49.786 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:50.201 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:23:50.215 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-04-29T14:23:50.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:50 smithi040 ceph-mon[39581]: pgmap v1107: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.4 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 1036/46498 objects misplaced (2.228%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:23:50.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4090966572' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:23:50.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:50 smithi040 ceph-mon[39581]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:23:50.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3503933340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:23:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:50 smithi194 ceph-mon[44089]: pgmap v1107: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.4 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 1036/46498 objects misplaced (2.228%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:23:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4090966572' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:23:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:50 smithi194 ceph-mon[44089]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:23:50.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3503933340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:23:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:50 smithi121 ceph-mon[44533]: pgmap v1107: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.4 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 1036/46498 objects misplaced (2.228%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:23:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4090966572' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:23:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:50 smithi121 ceph-mon[44533]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:23:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3503933340' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:23:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:51 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:51 smithi194 ceph-mon[44089]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:23:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3503933340' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:23:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:51 smithi194 ceph-mon[44089]: osdmap e418: 12 total, 12 up, 8 in 2024-04-29T14:23:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:51.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:51.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:51 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:51.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:51 smithi040 ceph-mon[39581]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:23:51.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3503933340' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:23:51.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:51 smithi040 ceph-mon[39581]: osdmap e418: 12 total, 12 up, 8 in 2024-04-29T14:23:51.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:51.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:51 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:23:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:51 smithi121 ceph-mon[44533]: from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:23:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3503933340' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:23:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:51 smithi121 ceph-mon[44533]: osdmap e418: 12 total, 12 up, 8 in 2024-04-29T14:23:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:52.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-29T14:23:52.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:52.490 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:52 smithi121 ceph-mon[44533]: pgmap v1109: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 169/42866 objects misplaced (0.394%); 9.8 MiB/s, 157 objects/s recovering 2024-04-29T14:23:52.490 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:52 smithi121 ceph-mon[44533]: osdmap e419: 12 total, 12 up, 8 in 2024-04-29T14:23:52.490 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:52 smithi121 ceph-mon[44533]: osdmap e420: 12 total, 12 up, 8 in 2024-04-29T14:23:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:52.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:52 smithi194 ceph-mon[44089]: pgmap v1109: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 169/42866 objects misplaced (0.394%); 9.8 MiB/s, 157 objects/s recovering 2024-04-29T14:23:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:52 smithi194 ceph-mon[44089]: osdmap e419: 12 total, 12 up, 8 in 2024-04-29T14:23:52.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:52 smithi194 ceph-mon[44089]: osdmap e420: 12 total, 12 up, 8 in 2024-04-29T14:23:52.591 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:23:52.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:52 smithi040 ceph-mon[39581]: pgmap v1109: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 169/42866 objects misplaced (0.394%); 9.8 MiB/s, 157 objects/s recovering 2024-04-29T14:23:52.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:52 smithi040 ceph-mon[39581]: osdmap e419: 12 total, 12 up, 8 in 2024-04-29T14:23:52.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:52 smithi040 ceph-mon[39581]: osdmap e420: 12 total, 12 up, 8 in 2024-04-29T14:23:52.617 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:52.635 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:23:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:52.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:23:52.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:52.765 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:52.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:53.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:53.033 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:23:53.034 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:23:53.034 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:53.053 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:53.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:53.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:53.188 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:23:53.190 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:53.219 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:23:53.240 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:23:53.260 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:23:53.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:23:53.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:53.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:53.307 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:53.308 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:53.308 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:53.308 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:53.308 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:53 smithi121 ceph-mon[44533]: osdmap e421: 12 total, 12 up, 8 in 2024-04-29T14:23:53.308 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3756220353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:53.311 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:23:53.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:23:53.388 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:23:53.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:23:53.416 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:23:53.439 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:23:53.463 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:23:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:23:53.466 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:23:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:53.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:53.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:53.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:53.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:53.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:53 smithi040 ceph-mon[39581]: osdmap e421: 12 total, 12 up, 8 in 2024-04-29T14:23:53.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3756220353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:53 smithi194 ceph-mon[44089]: osdmap e421: 12 total, 12 up, 8 in 2024-04-29T14:23:53.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3756220353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:23:53.825 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:23:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:53.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:23:53.839 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:23:53.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:53.968 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:54.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:54.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:54.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:54.155 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:54.187 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:54.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:54.230 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:23:54.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: pgmap v1112: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 912 op/s; 169/36826 objects misplaced (0.459%); 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.527 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: pgmap v1112: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 912 op/s; 169/36826 objects misplaced (0.459%); 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:54.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:54.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: pgmap v1112: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 912 op/s; 169/36826 objects misplaced (0.459%); 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:23:54.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:23:54.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:54.660 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:54.684 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:54.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:23:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:54.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:54.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:23:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:54.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:23:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:54.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:23:54.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:55.098 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:55.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:55.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:55.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:55.569 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:55.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:55.683 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:55.732 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:55.763 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:55.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:55.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:56.100 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:56.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:56.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:56.259 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:56.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:56.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:56 smithi194 ceph-mon[44089]: pgmap v1114: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 169/36826 objects misplaced (0.459%); 9.3 MiB/s, 149 objects/s recovering 2024-04-29T14:23:56.361 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:23:56.361 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:23:56.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:299714 4.6 4:6dc382ad:::benchmark_data_smithi194_103039_object143031:head [delete] snapc 0=[] ondisk+write+known_if_redirected e421)", 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:56.362938+0000", 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0057671709999999998, 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058014779999999997, 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 299714 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:23:56.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.362938+0000", 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.362938+0000", 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.362940+0000", 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.531e-06 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.362945+0000", 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4820000000000002e-06 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.362961+0000", 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5536000000000002e-05 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.362969+0000", 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1300000000000001e-06 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365159+0000", 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021894169999999999 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365271+0000", 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000111971 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365376+0000", 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105146 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.25522.0:299717 4.d e421/323 4:b000dac7:::benchmark_data_smithi194_103039_object143034:head v 421'18887, mlcod=421'18885)", 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:56.365149+0000", 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.00355683, 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003674329, 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365149+0000", 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365149+0000", 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365150+0000", 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3149999999999999e-06 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365158+0000", 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7659999999999998e-06 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365166+0000", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7570000000000004e-06 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365173+0000", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0119999999999999e-06 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365233+0000", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0507000000000002e-05 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.365256+0000", 2024-04-29T14:23:56.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2390000000000001e-05 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:299697 4.2 4:4f331fb1:::benchmark_data_smithi194_103039_object143014:head [delete] snapc 0=[] ondisk+write+known_if_redirected e421)", 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:56.346508+0000", 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.022198105999999999, 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022358111, 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:23:56.369 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 299697 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.346508+0000", 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.346508+0000", 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.346508+0000", 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6199999999999998e-07 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.346511+0000", 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7149999999999998e-06 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.346544+0000", 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2775999999999999e-05 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.346552+0000", 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3229999999999998e-06 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355457+0000", 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089051410000000001 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355557+0000", 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000100028 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355669+0000", 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011215800000000001 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360497+0000", 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048281920000000002 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:299703 4.2 4:46ff8671:::benchmark_data_smithi194_103039_object143020:head [delete] snapc 0=[] ondisk+write+known_if_redirected e421)", 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:56.355537+0000", 2024-04-29T14:23:56.373 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.013168371999999999, 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01339601, 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 299703 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355537+0000", 2024-04-29T14:23:56.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355537+0000", 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355539+0000", 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0669999999999999e-06 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355545+0000", 2024-04-29T14:23:56.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1129999999999997e-06 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355556+0000", 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0696000000000001e-05 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.355564+0000", 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0739999999999993e-06 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360763+0000", 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051984359999999999 2024-04-29T14:23:56.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360865+0000", 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010271499999999999 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360969+0000", 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103871 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:299708 4.6 4:6d0819ab:::benchmark_data_smithi194_103039_object143025:head [delete] snapc 0=[] ondisk+write+known_if_redirected e421)", 2024-04-29T14:23:56.377 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:56.360181+0000", 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0085248110000000002, 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088168050000000005, 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 299708 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360181+0000", 2024-04-29T14:23:56.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360181+0000", 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360183+0000", 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7349999999999999e-06 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360188+0000", 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3809999999999998e-06 2024-04-29T14:23:56.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360197+0000", 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8540000000000003e-06 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360205+0000", 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8140000000000001e-06 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360246+0000", 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1647000000000003e-05 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360362+0000", 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011612000000000001 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:56.360474+0000", 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011188699999999999 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 5 2024-04-29T14:23:56.381 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:23:56.382 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:56 smithi040 ceph-mon[39581]: pgmap v1114: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 169/36826 objects misplaced (0.459%); 9.3 MiB/s, 149 objects/s recovering 2024-04-29T14:23:56.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:56.445 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:56.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:56.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:56.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:56.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:56.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:56 smithi121 ceph-mon[44533]: pgmap v1114: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 1.1 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 169/36826 objects misplaced (0.459%); 9.3 MiB/s, 149 objects/s recovering 2024-04-29T14:23:56.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:56.858 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:23:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:56.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:23:56.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:57.032 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:57.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:57.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:57.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:23:57.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:23:57.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:23:57.217 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:57.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:23:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:57.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:23:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.4 to 1 (10000) 2024-04-29T14:23:57.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/993401782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:23:57.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:57 smithi194 ceph-mon[44089]: pgmap v1115: 17 pgs: 17 active+clean; 914 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-29T14:23:57.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:23:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/993401782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:23:57.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:57 smithi121 ceph-mon[44533]: pgmap v1115: 17 pgs: 17 active+clean; 914 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-29T14:23:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:23:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:57.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:23:57.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/993401782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:23:57.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:57 smithi040 ceph-mon[39581]: pgmap v1115: 17 pgs: 17 active+clean; 914 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-29T14:23:57.500 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:23:57.517 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:23:57.567 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:57.826 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:23:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:57.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:23:57.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:57.983 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:58.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:58.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:58.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:58.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:23:58.159 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:58.256 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:23:58.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:23:58.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/993401782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:23:58.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:58 smithi121 ceph-mon[44533]: osdmap e422: 12 total, 12 up, 8 in 2024-04-29T14:23:58.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:23:58.532 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:23:58.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:23:58.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/993401782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:23:58.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:58 smithi194 ceph-mon[44089]: osdmap e422: 12 total, 12 up, 8 in 2024-04-29T14:23:58.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:23:58.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:23:58.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:23:58.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/993401782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:23:58.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:58 smithi040 ceph-mon[39581]: osdmap e422: 12 total, 12 up, 8 in 2024-04-29T14:23:58.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:23:58.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-29T14:23:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:23:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:58.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:23:58.669 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:23:58.714 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:23:58.761 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:59.025 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:23:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:59.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:23:59.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:23:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:59.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:23:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:59.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:23:59.259 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:59 smithi194 ceph-mon[44089]: from='client.16128 v1:172.21.15.40:0/3224170437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:59.259 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:23:59.259 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:23:59.259 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:59 smithi194 ceph-mon[44089]: pgmap v1117: 17 pgs: 17 active+clean; 877 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:23:59.259 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:59 smithi194 ceph-mon[44089]: pgmap v1118: 17 pgs: 17 active+clean; 877 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:23:59.259 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:23:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:23:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:23:59.312 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:23:59.359 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:59.495 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:23:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:23:59.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:23:59.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:59 smithi040 ceph-mon[39581]: from='client.16128 v1:172.21.15.40:0/3224170437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:59.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:23:59.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:23:59.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:59 smithi040 ceph-mon[39581]: pgmap v1117: 17 pgs: 17 active+clean; 877 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:23:59.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:59 smithi040 ceph-mon[39581]: pgmap v1118: 17 pgs: 17 active+clean; 877 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:23:59.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:23:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:23:59.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:23:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:59 smithi121 ceph-mon[44533]: from='client.16128 v1:172.21.15.40:0/3224170437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:23:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:23:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:23:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:59 smithi121 ceph-mon[44533]: pgmap v1117: 17 pgs: 17 active+clean; 877 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:23:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:59 smithi121 ceph-mon[44533]: pgmap v1118: 17 pgs: 17 active+clean; 877 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:23:59.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:23:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:23:59.634 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:23:59.685 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:23:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:23:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:23:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:23:59.685 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:24:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:00 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:00 smithi194 ceph-mon[44089]: from='client.16128 v1:172.21.15.40:0/3224170437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:00 smithi194 ceph-mon[44089]: osdmap e423: 12 total, 12 up, 8 in 2024-04-29T14:24:00.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:00 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:00.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:00 smithi040 ceph-mon[39581]: from='client.16128 v1:172.21.15.40:0/3224170437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:00.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:00 smithi040 ceph-mon[39581]: osdmap e423: 12 total, 12 up, 8 in 2024-04-29T14:24:00.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:00 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:00.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:00 smithi121 ceph-mon[44533]: from='client.16128 v1:172.21.15.40:0/3224170437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:00.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:00 smithi121 ceph-mon[44533]: osdmap e423: 12 total, 12 up, 8 in 2024-04-29T14:24:00.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:24:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:01.075 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:01.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:01.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:01.326 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:01.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:01 smithi194 ceph-mon[44089]: pgmap v1120: 17 pgs: 17 active+clean; 751 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:24:01.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:01 smithi040 ceph-mon[39581]: pgmap v1120: 17 pgs: 17 active+clean; 751 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:24:01.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:01 smithi121 ceph-mon[44533]: pgmap v1120: 17 pgs: 17 active+clean; 751 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:24:01.781 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:01.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:01.886 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:01.914 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:24:01.917 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:01.940 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:24:01.961 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:24:01.981 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:24:02.006 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:24:02.019 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:02.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:02.029 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:24:02.064 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:24:02.087 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:24:02.108 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:24:02.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:02.137 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:24:02.159 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:24:02.160 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:02.184 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:24:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:24:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:24:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:24:02.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/366018501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:02.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:02.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/366018501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:02.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:02.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/366018501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:02.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:02.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:24:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:03 smithi194 ceph-mon[44089]: pgmap v1121: 17 pgs: 17 active+clean; 590 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 915 op/s 2024-04-29T14:24:03.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:03 smithi040 ceph-mon[39581]: pgmap v1121: 17 pgs: 17 active+clean; 590 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 915 op/s 2024-04-29T14:24:03.398 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:24:03.398 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:24:03.410 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:03.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:03.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:03.562 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:03.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:03 smithi121 ceph-mon[44533]: pgmap v1121: 17 pgs: 17 active+clean; 590 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 915 op/s 2024-04-29T14:24:03.665 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:03.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:03.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:03.858 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:03.900 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:03.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:04.006 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:04.050 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:04.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:04.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:24:04.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.5 to 1 (10000) 2024-04-29T14:24:04.301 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:04.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:04.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2804759769' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:24:04.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:04.441 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:04.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:04.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:04.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2804759769' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:24:04.573 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:04.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2804759769' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:24:04.669 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:04.706 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:04.846 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:04.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:04.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:05.003 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:05.110 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:05.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:05.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:24:05.222 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:05.253 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:05.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2804759769' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:24:05.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:05 smithi040 ceph-mon[39581]: osdmap e424: 12 total, 12 up, 8 in 2024-04-29T14:24:05.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:05 smithi040 ceph-mon[39581]: pgmap v1123: 17 pgs: 17 active+clean; 550 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 872 op/s 2024-04-29T14:24:05.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-29T14:24:05.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2804759769' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:24:05.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:05 smithi194 ceph-mon[44089]: osdmap e424: 12 total, 12 up, 8 in 2024-04-29T14:24:05.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:05 smithi194 ceph-mon[44089]: pgmap v1123: 17 pgs: 17 active+clean; 550 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 872 op/s 2024-04-29T14:24:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2804759769' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:24:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:05 smithi121 ceph-mon[44533]: osdmap e424: 12 total, 12 up, 8 in 2024-04-29T14:24:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:05 smithi121 ceph-mon[44533]: pgmap v1123: 17 pgs: 17 active+clean; 550 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 872 op/s 2024-04-29T14:24:05.809 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:05.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:05.920 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:05.958 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:06.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:24:06.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:24:06.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:06 smithi040 ceph-mon[39581]: osdmap e425: 12 total, 12 up, 8 in 2024-04-29T14:24:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:24:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:24:06.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:06 smithi194 ceph-mon[44089]: osdmap e425: 12 total, 12 up, 8 in 2024-04-29T14:24:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:24:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:24:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:06 smithi121 ceph-mon[44533]: osdmap e425: 12 total, 12 up, 8 in 2024-04-29T14:24:07.931 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:07.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:08.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:08.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:08.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:08.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:08.231 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:08.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:08 smithi194 ceph-mon[44089]: pgmap v1125: 17 pgs: 17 active+clean; 257 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:24:08.265 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:08.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:08 smithi040 ceph-mon[39581]: pgmap v1125: 17 pgs: 17 active+clean; 257 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:24:08.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:08 smithi121 ceph-mon[44533]: pgmap v1125: 17 pgs: 17 active+clean; 257 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-04-29T14:24:08.808 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:24:08.828 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:24:08.850 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:24:08.870 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:24:08.895 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:24:08.918 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:24:08.939 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:24:08.962 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:24:08.986 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:24:09.009 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:24:09.030 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:24:09.051 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:24:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:24:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:24:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:24:09.155 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:24:09.289 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 156680 objects 2024-04-29T14:24:09.289 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :221.964 2024-04-29T14:24:09.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/615522363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:24:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/327236540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:09.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/615522363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:24:09.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/327236540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:09.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/615522363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:24:09.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/327236540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:09.624 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:09.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:09.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:09.762 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:10.012 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:10.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:10.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:10.119 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:24:10.119 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:24:10.119 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:24:10.119 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:24:10.146 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: pgmap v1126: 17 pgs: 17 active+clean; 218 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/615522363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: osdmap e426: 12 total, 12 up, 8 in 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: 4.9 scrub starts 2024-04-29T14:24:10.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:10 smithi040 ceph-mon[39581]: 4.9 scrub ok 2024-04-29T14:24:10.495 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:10.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: pgmap v1126: 17 pgs: 17 active+clean; 218 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/615522363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: osdmap e426: 12 total, 12 up, 8 in 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: 4.9 scrub starts 2024-04-29T14:24:10.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:10 smithi194 ceph-mon[44089]: 4.9 scrub ok 2024-04-29T14:24:10.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: pgmap v1126: 17 pgs: 17 active+clean; 218 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T14:24:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/615522363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: osdmap e426: 12 total, 12 up, 8 in 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: 4.9 scrub starts 2024-04-29T14:24:10.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:10 smithi121 ceph-mon[44533]: 4.9 scrub ok 2024-04-29T14:24:10.631 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:10.652 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:10.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:10.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:24:10.756 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:10.783 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:10.803 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:10.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:10.907 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:10.935 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:10.947 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:10.956 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:10.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:11.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:11.090 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:11.109 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:11.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:11.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:11.240 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:11.259 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:11.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:11.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:11.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:11.390 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:11.409 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:11.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:11.513 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:11.538 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:12.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:24:12.157 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.7 to 1 (10000) 2024-04-29T14:24:12.187 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:12.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:12.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:12.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:12.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:12.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:12 smithi040 ceph-mon[39581]: pgmap v1128: 17 pgs: 17 active+clean; 119 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T14:24:12.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3612485803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:24:12.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:12 smithi040 ceph-mon[39581]: 4.d scrub starts 2024-04-29T14:24:12.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:12 smithi040 ceph-mon[39581]: 4.d scrub ok 2024-04-29T14:24:12.446 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:12 smithi194 ceph-mon[44089]: pgmap v1128: 17 pgs: 17 active+clean; 119 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T14:24:12.446 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3612485803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:24:12.446 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:12 smithi194 ceph-mon[44089]: 4.d scrub starts 2024-04-29T14:24:12.446 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:12 smithi194 ceph-mon[44089]: 4.d scrub ok 2024-04-29T14:24:12.448 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:24:12.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:12.486 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:12 smithi121 ceph-mon[44533]: pgmap v1128: 17 pgs: 17 active+clean; 119 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T14:24:12.486 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3612485803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:24:12.486 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:12 smithi121 ceph-mon[44533]: 4.d scrub starts 2024-04-29T14:24:12.486 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:12 smithi121 ceph-mon[44533]: 4.d scrub ok 2024-04-29T14:24:12.488 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:12.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:12.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:12.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:12.640 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:12.662 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:24:12.662 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:24:12.662 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:24:12.662 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:24:12.662 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:24:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:12.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:12.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:12.792 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:12.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:12.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:12.914 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:12.944 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:12.951 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:12.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-29T14:24:13.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3612485803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:24:13.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:13 smithi040 ceph-mon[39581]: osdmap e427: 12 total, 12 up, 8 in 2024-04-29T14:24:13.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:13 smithi040 ceph-mon[39581]: 4.c scrub starts 2024-04-29T14:24:13.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:13 smithi040 ceph-mon[39581]: 4.c scrub ok 2024-04-29T14:24:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3612485803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:24:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:13 smithi194 ceph-mon[44089]: osdmap e427: 12 total, 12 up, 8 in 2024-04-29T14:24:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:13 smithi194 ceph-mon[44089]: 4.c scrub starts 2024-04-29T14:24:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:13 smithi194 ceph-mon[44089]: 4.c scrub ok 2024-04-29T14:24:13.371 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:13.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:13.387 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:24:13.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:13.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3612485803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:24:13.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:13 smithi121 ceph-mon[44533]: osdmap e427: 12 total, 12 up, 8 in 2024-04-29T14:24:13.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:13 smithi121 ceph-mon[44533]: 4.c scrub starts 2024-04-29T14:24:13.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:13 smithi121 ceph-mon[44533]: 4.c scrub ok 2024-04-29T14:24:13.508 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:13.530 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:13.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:13.634 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:13.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:13.685 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:13.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:13.691 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:13.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:13.821 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:13.920 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:13.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:14.023 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:14.053 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:14.149 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:14.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:14.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:14 smithi194 ceph-mon[44089]: pgmap v1130: 17 pgs: 17 active+clean; 15 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-04-29T14:24:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:24:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:24:14.435 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:14 smithi040 ceph-mon[39581]: pgmap v1130: 17 pgs: 17 active+clean; 15 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-04-29T14:24:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:24:14.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:24:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:14.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:14.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:14.571 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:14 smithi121 ceph-mon[44533]: pgmap v1130: 17 pgs: 17 active+clean; 15 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-04-29T14:24:14.571 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:24:14.571 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:24:14.573 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:14.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:14.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:14.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:14.733 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:14.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:14.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:14.857 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:14.888 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:14.910 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:14.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:14.960 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-04-29T14:24:14.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-04-29T14:24:15.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:15.080 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:15.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:15.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:15.242 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:15.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:15.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:75318 4.7 4:e7d3c9ac:::benchmark_data_smithi194_103039_object75317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:19:24.471701+0000", 2024-04-29T14:24:15.351 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 290.885063876, 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.212201377, 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 75318 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471701+0000", 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471701+0000", 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471702+0000", 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2860000000000001e-06 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471731+0000", 2024-04-29T14:24:15.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9090000000000001e-05 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471742+0000", 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0978999999999999e-05 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471750+0000", 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9489999999999996e-06 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471809+0000", 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9110999999999997e-05 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.471928+0000", 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011945100000000001 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.472061+0000", 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013222200000000001 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.474581+0000", 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025206450000000002 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.683823+0000", 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20924194700000001 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.683865+0000", 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1646000000000001e-05 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.683902+0000", 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7051000000000002e-05 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:75338 4.7 4:e2dd1a78:::benchmark_data_smithi194_103039_object75337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:19:24.480810+0000", 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 290.87595393800001, 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20549305200000001, 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.357 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 75338 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480810+0000", 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480810+0000", 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480811+0000", 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.95e-07 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480842+0000", 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0929000000000002e-05 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480849+0000", 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7320000000000001e-06 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480855+0000", 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8710000000000002e-06 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480896+0000", 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1170000000000001e-05 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.480987+0000", 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0600999999999995e-05 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.481115+0000", 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000128734 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.483315+0000", 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021990410000000001 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.686182+0000", 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.202867724 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.686247+0000", 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5152000000000002e-05 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.686303+0000", 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6103000000000001e-05 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:75346 4.7 4:e14bd354:::benchmark_data_smithi194_103039_object75345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:19:24.486468+0000", 2024-04-29T14:24:15.362 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 290.870296215, 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19997963499999999, 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 75346 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486468+0000", 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486468+0000", 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486470+0000", 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.872e-06 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486492+0000", 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2120000000000002e-05 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486502+0000", 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0108e-05 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486510+0000", 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5449999999999998e-06 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486564+0000", 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3872999999999997e-05 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486678+0000", 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000114613 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.486811+0000", 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000132943 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.489217+0000", 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002406006 2024-04-29T14:24:15.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.686350+0000", 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19713316 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.686406+0000", 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5727999999999999e-05 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.686448+0000", 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1666999999999999e-05 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.367 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:75356 4.7 4:ec4049c6:::benchmark_data_smithi194_103039_object75355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:19:24.495307+0000", 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 290.86145711400002, 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19429601199999999, 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.368 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 75356 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495307+0000", 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495307+0000", 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495309+0000", 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0980000000000001e-06 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495334+0000", 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.48e-05 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495347+0000", 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2699e-05 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495355+0000", 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5450000000000003e-06 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495418+0000", 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2615999999999998e-05 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495524+0000", 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105567 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:24:15.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.495615+0000", 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1608999999999997e-05 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.497505+0000", 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018898649999999999 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.689530+0000", 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19202524600000001 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.689568+0000", 2024-04-29T14:24:15.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8050999999999999e-05 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.689603+0000", 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4916000000000002e-05 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:75416 4.7 4:e129f3ba:::benchmark_data_smithi194_103039_object75415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:19:24.534382+0000", 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 290.82238188399998, 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15546775099999999, 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.373 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 75416 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534382+0000", 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534382+0000", 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534384+0000", 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.53e-06 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534406+0000", 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1583999999999999e-05 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534415+0000", 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6509999999999996e-06 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534423+0000", 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9619999999999993e-06 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534481+0000", 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8081000000000002e-05 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534622+0000", 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014047699999999999 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.534778+0000", 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015630099999999999 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.539506+0000", 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047276519999999997 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.689758+0000", 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15025211299999999 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.689810+0000", 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2213000000000003e-05 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:19:24.689850+0000", 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0187e-05 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253029 4.4 4:22f4d079:::benchmark_data_smithi194_103039_object96346:head [delete] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.888527+0000", 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.468237826000006, 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.230891454, 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253029 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888527+0000", 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888527+0000", 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888527+0000", 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2600000000000001e-07 2024-04-29T14:24:15.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888530+0000", 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5739999999999998e-06 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888536+0000", 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3709999999999996e-06 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888542+0000", 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2320000000000001e-06 2024-04-29T14:24:15.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888588+0000", 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6152999999999997e-05 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888711+0000", 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012314199999999999 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.888834+0000", 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012302299999999999 2024-04-29T14:24:15.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893097+0000", 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042632130000000001 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119306+0000", 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.22620850100000001 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119353+0000", 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7401000000000002e-05 2024-04-29T14:24:15.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119418+0000", 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4918000000000004e-05 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253031 4.4 4:206eb9d6:::benchmark_data_smithi194_103039_object96348:head [delete] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.893139+0000", 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.463625183000005, 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.22644992799999999, 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253031 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893139+0000", 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893139+0000", 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893141+0000", 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.198e-06 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893148+0000", 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2999999999999998e-06 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893157+0000", 2024-04-29T14:24:15.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7140000000000005e-06 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893166+0000", 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3250000000000008e-06 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893218+0000", 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2169999999999997e-05 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893346+0000", 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012853100000000001 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.893469+0000", 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000122915 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.898374+0000", 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049046999999999997 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119476+0000", 2024-04-29T14:24:15.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.22110233900000001 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119526+0000", 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9565999999999999e-05 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119589+0000", 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3170000000000007e-05 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253044 4.4 4:215bf6d5:::benchmark_data_smithi194_103039_object96361:head [delete] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.905757+0000", 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.451007289000003, 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21397592100000001, 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253044 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905757+0000", 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905757+0000", 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905759+0000", 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.012e-06 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905765+0000", 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2400000000000004e-06 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905778+0000", 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2975e-05 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905787+0000", 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5150000000000006e-06 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905840+0000", 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3572999999999997e-05 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.905951+0000", 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011082300000000001 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.906543+0000", 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059213799999999997 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.910758+0000", 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042150369999999996 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119634+0000", 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20887568200000001 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119674+0000", 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9919000000000002e-05 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119733+0000", 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9006999999999999e-05 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253104 4.4 4:2a7396d0:::benchmark_data_smithi194_103039_object96421:head [delete] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.959939+0000", 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.396825504999995, 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.164133802, 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.395 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253104 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.959939+0000", 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.959939+0000", 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.959941+0000", 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1210000000000001e-06 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.959948+0000", 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7900000000000002e-06 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.959960+0000", 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2094e-05 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.959968+0000", 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5059999999999995e-06 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.960017+0000", 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8959000000000003e-05 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.960114+0000", 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7038000000000002e-05 2024-04-29T14:24:15.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.960194+0000", 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9227000000000006e-05 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.964040+0000", 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038468830000000002 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124000+0000", 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15995989899999999 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124031+0000", 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0855999999999997e-05 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124073+0000", 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1428999999999999e-05 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253107 4.4 4:2964a813:::benchmark_data_smithi194_103039_object96424:head [delete] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-29T14:24:15.400 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.965515+0000", 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.391249713999997, 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15869739699999999, 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253107 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.401 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965515+0000", 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965515+0000", 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965517+0000", 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9549999999999999e-06 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965523+0000", 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.032e-06 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965534+0000", 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1528999999999999e-05 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965542+0000", 2024-04-29T14:24:15.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8790000000000003e-06 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965607+0000", 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4554000000000006e-05 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965752+0000", 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014496800000000001 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.965884+0000", 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013229600000000001 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.970192+0000", 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043076490000000002 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124121+0000", 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.153928974 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124161+0000", 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0673999999999997e-05 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124212+0000", 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0887000000000003e-05 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253123 4.4 4:26925a69:::benchmark_data_smithi194_103039_object96440:head [delete] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-29T14:24:15.406 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.979595+0000", 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.377169758999997, 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.153063277, 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253123 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979595+0000", 2024-04-29T14:24:15.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979595+0000", 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979596+0000", 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7880000000000001e-06 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979602+0000", 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4979999999999997e-06 2024-04-29T14:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979613+0000", 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1296e-05 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979619+0000", 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9070000000000004e-06 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979672+0000", 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3143000000000003e-05 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979809+0000", 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013684 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.979927+0000", 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117921 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.983620+0000", 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003693441 2024-04-29T14:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.132564+0000", 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14894343199999999 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.132607+0000", 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2901000000000001e-05 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.132658+0000", 2024-04-29T14:24:15.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1109999999999997e-05 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253029 4.4 4:22f4d079:::benchmark_data_smithi194_103039_object96346:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.990371+0000", 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.366393158999998, 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12899854599999999, 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.412 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253029 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990371+0000", 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990371+0000", 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.413 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990372+0000", 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1489999999999999e-06 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990377+0000", 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.656e-06 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990389+0000", 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1551e-05 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.414 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990396+0000", 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7849999999999992e-06 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990462+0000", 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5526999999999998e-05 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990522+0000", 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9750000000000002e-05 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.415 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119370+0000", 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12884812800000001 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253031 4.4 4:206eb9d6:::benchmark_data_smithi194_103039_object96348:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.990412+0000", 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.366352250999995, 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.129128677, 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.416 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253031 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990412+0000", 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990412+0000", 2024-04-29T14:24:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990413+0000", 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.61e-07 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990416+0000", 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7580000000000001e-06 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990429+0000", 2024-04-29T14:24:15.418 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3713000000000001e-05 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990435+0000", 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7359999999999998e-06 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990545+0000", 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110281 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.990592+0000", 2024-04-29T14:24:15.419 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6663999999999999e-05 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119541+0000", 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.128948864 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253029 4.4 4:22f4d079:::benchmark_data_smithi194_103039_object96346:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.995422+0000", 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.361342563999997, 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.123950371, 2024-04-29T14:24:15.420 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253029 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995422+0000", 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.421 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995422+0000", 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995424+0000", 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5160000000000001e-06 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995430+0000", 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9309999999999996e-06 2024-04-29T14:24:15.422 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995443+0000", 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2700999999999999e-05 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995452+0000", 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8570000000000001e-06 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995511+0000", 2024-04-29T14:24:15.423 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9129999999999998e-05 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995571+0000", 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0047999999999998e-05 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119372+0000", 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12380118800000001 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.424 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253031 4.4 4:206eb9d6:::benchmark_data_smithi194_103039_object96348:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.995468+0000", 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.361296484999997, 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.124078167, 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253031 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.425 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995468+0000", 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995468+0000", 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995468+0000", 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0100000000000005e-07 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995471+0000", 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8449999999999999e-06 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995478+0000", 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0729999999999999e-06 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995484+0000", 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3959999999999997e-06 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.427 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995603+0000", 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000119442 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995650+0000", 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6635999999999999e-05 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119546+0000", 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.123896174 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.428 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253044 4.4 4:215bf6d5:::benchmark_data_smithi194_103039_object96361:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.995507+0000", 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.361257198999994, 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12417541, 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253044 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.429 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995507+0000", 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995507+0000", 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995508+0000", 2024-04-29T14:24:15.430 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7000000000000005e-07 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995510+0000", 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4439999999999998e-06 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995516+0000", 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3170000000000001e-06 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.431 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995520+0000", 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7809999999999996e-06 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995675+0000", 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000154415 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995734+0000", 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9719999999999997e-05 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.432 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.119683+0000", 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.123948163 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253104 4.4 4:2a7396d0:::benchmark_data_smithi194_103039_object96421:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.995526+0000", 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.361238787000005, 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12851484399999999, 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.433 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253104 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995526+0000", 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995526+0000", 2024-04-29T14:24:15.434 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995526+0000", 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9200000000000001e-07 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995528+0000", 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9759999999999998e-06 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995533+0000", 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5979999999999999e-06 2024-04-29T14:24:15.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995538+0000", 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0030000000000002e-06 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995761+0000", 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00022324500000000001 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995808+0000", 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6751e-05 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.436 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124040+0000", 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12823267899999999 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253107 4.4 4:2964a813:::benchmark_data_smithi194_103039_object96424:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.995542+0000", 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.361221938, 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.128628883, 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.437 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253107 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995542+0000", 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.438 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995542+0000", 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995543+0000", 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.89e-07 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995545+0000", 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9980000000000002e-06 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.439 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995550+0000", 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6949999999999999e-06 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995554+0000", 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5090000000000004e-06 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995832+0000", 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00027808500000000002 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.440 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995877+0000", 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4585000000000001e-05 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.124171+0000", 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12829452199999999 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253123 4.4 4:26925a69:::benchmark_data_smithi194_103039_object96440:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e404)", 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.995558+0000", 2024-04-29T14:24:15.441 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.361205968999997, 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.137064086, 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253123 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995558+0000", 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.442 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995558+0000", 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995559+0000", 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7099999999999997e-07 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995561+0000", 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.792e-06 2024-04-29T14:24:15.443 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995565+0000", 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5059999999999997e-06 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995570+0000", 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4530000000000004e-06 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995902+0000", 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00033197000000000001 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.444 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.995948+0000", 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6153999999999999e-05 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.132622+0000", 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13667483999999999 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.25522.0:253139 4.4 4:212bf21b:::benchmark_data_smithi194_103039_object96456:head [delete] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-29T14:24:15.445 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:23:04.996103+0000", 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.360660937000006, 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14514654499999999, 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.25522", 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 253139 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996103+0000", 2024-04-29T14:24:15.446 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996103+0000", 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996104+0000", 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6199999999999996e-07 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996109+0000", 2024-04-29T14:24:15.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2390000000000004e-06 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996119+0000", 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996126+0000", 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.793e-06 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996169+0000", 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2648000000000002e-05 2024-04-29T14:24:15.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996276+0000", 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107587 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:04.996387+0000", 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011076100000000001 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.001097+0000", 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047103020000000004 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.141169+0000", 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14007159599999999 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.141205+0000", 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5706e-05 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:23:05.141250+0000", 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5225999999999997e-05 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.450 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.451 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.451 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:24:15.451 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:24:15.451 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:15 smithi040 ceph-mon[39581]: 4.4 scrub starts 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:15 smithi040 ceph-mon[39581]: 4.4 scrub ok 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:15 smithi040 ceph-mon[39581]: osdmap e428: 12 total, 12 up, 8 in 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:15 smithi040 ceph-mon[39581]: 4.0 scrub starts 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:15 smithi040 ceph-mon[39581]: 4.0 scrub ok 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:15 smithi194 ceph-mon[44089]: 4.4 scrub starts 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:15 smithi194 ceph-mon[44089]: 4.4 scrub ok 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:15 smithi194 ceph-mon[44089]: osdmap e428: 12 total, 12 up, 8 in 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:15 smithi194 ceph-mon[44089]: 4.0 scrub starts 2024-04-29T14:24:15.452 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:15 smithi194 ceph-mon[44089]: 4.0 scrub ok 2024-04-29T14:24:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:15.453 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:15.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:15.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:15.554 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:15.585 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:15.603 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:15.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:15.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:15 smithi121 ceph-mon[44533]: 4.4 scrub starts 2024-04-29T14:24:15.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:15 smithi121 ceph-mon[44533]: 4.4 scrub ok 2024-04-29T14:24:15.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:15 smithi121 ceph-mon[44533]: osdmap e428: 12 total, 12 up, 8 in 2024-04-29T14:24:15.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:15 smithi121 ceph-mon[44533]: 4.0 scrub starts 2024-04-29T14:24:15.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:15 smithi121 ceph-mon[44533]: 4.0 scrub ok 2024-04-29T14:24:15.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:15.740 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:15.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:15.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:15.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:15.895 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:15.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:16.020 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:16.048 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:16.071 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:16.075 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:16.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:16.102 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:24:16.179 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:24:16.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:16.209 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:16.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:16.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:16.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:16 smithi040 ceph-mon[39581]: pgmap v1132: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 580 op/s 2024-04-29T14:24:16.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/331678015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:24:16.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:16 smithi040 ceph-mon[39581]: 4.e scrub starts 2024-04-29T14:24:16.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:16 smithi040 ceph-mon[39581]: 4.e scrub ok 2024-04-29T14:24:16.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:16 smithi194 ceph-mon[44089]: pgmap v1132: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 580 op/s 2024-04-29T14:24:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/331678015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:24:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:16 smithi194 ceph-mon[44089]: 4.e scrub starts 2024-04-29T14:24:16.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:16 smithi194 ceph-mon[44089]: 4.e scrub ok 2024-04-29T14:24:16.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:16.360 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:16.379 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:16.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:16.407 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:16.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:16.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:16 smithi121 ceph-mon[44533]: pgmap v1132: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 580 op/s 2024-04-29T14:24:16.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/331678015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:24:16.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:16 smithi121 ceph-mon[44533]: 4.e scrub starts 2024-04-29T14:24:16.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:16 smithi121 ceph-mon[44533]: 4.e scrub ok 2024-04-29T14:24:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:17 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:24:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/331678015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:17 smithi194 ceph-mon[44089]: osdmap e429: 12 total, 12 up, 8 in 2024-04-29T14:24:17.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:17 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:24:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/331678015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:17 smithi040 ceph-mon[39581]: osdmap e429: 12 total, 12 up, 8 in 2024-04-29T14:24:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:17.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:17.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:17 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:24:17.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/331678015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:17.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:17 smithi121 ceph-mon[44533]: osdmap e429: 12 total, 12 up, 8 in 2024-04-29T14:24:17.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:17.702 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:18.147 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:24:18.147 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:24:18.147 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:24:18.148 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:24:18.449 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:18 smithi040 ceph-mon[39581]: pgmap v1134: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 317 op/s 2024-04-29T14:24:18.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:18 smithi040 ceph-mon[39581]: 4.2 scrub starts 2024-04-29T14:24:18.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:18 smithi040 ceph-mon[39581]: 4.2 scrub ok 2024-04-29T14:24:18.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:18 smithi194 ceph-mon[44089]: pgmap v1134: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 317 op/s 2024-04-29T14:24:18.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:18 smithi194 ceph-mon[44089]: 4.2 scrub starts 2024-04-29T14:24:18.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:18 smithi194 ceph-mon[44089]: 4.2 scrub ok 2024-04-29T14:24:18.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:18 smithi121 ceph-mon[44533]: pgmap v1134: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 317 op/s 2024-04-29T14:24:18.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:18 smithi121 ceph-mon[44533]: 4.2 scrub starts 2024-04-29T14:24:18.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:18 smithi121 ceph-mon[44533]: 4.2 scrub ok 2024-04-29T14:24:18.652 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:18.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:18.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:18.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:24:18.787 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:18.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:24:19.078 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:19.192 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:24:19.213 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:24:19.232 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:24:19.246 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.9 to 1 (10000) 2024-04-29T14:24:19.261 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:24:19.281 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:24:19.302 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:24:19.323 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:24:19.343 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:24:19.363 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:24:19.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:24:19.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:24:19.425 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:24:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:24:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:24:19.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:19 smithi040 ceph-mon[39581]: 4.6 scrub starts 2024-04-29T14:24:19.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:19 smithi040 ceph-mon[39581]: 4.6 scrub ok 2024-04-29T14:24:19.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3891186077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:19 smithi194 ceph-mon[44089]: 4.6 scrub starts 2024-04-29T14:24:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:19 smithi194 ceph-mon[44089]: 4.6 scrub ok 2024-04-29T14:24:19.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3891186077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:19.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:19 smithi121 ceph-mon[44533]: 4.6 scrub starts 2024-04-29T14:24:19.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:19 smithi121 ceph-mon[44533]: 4.6 scrub ok 2024-04-29T14:24:19.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3891186077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:20.521 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: pgmap v1135: 17 pgs: 17 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 34 op/s 2024-04-29T14:24:20.521 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/224694020' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:24:20.521 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osdmap e430: 12 total, 12 up, 8 in 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: pgmap v1135: 17 pgs: 17 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 34 op/s 2024-04-29T14:24:20.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/224694020' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:24:20.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osdmap e430: 12 total, 12 up, 8 in 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.539 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:20.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:20.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: pgmap v1135: 17 pgs: 17 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 34 op/s 2024-04-29T14:24:20.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/224694020' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:24:20.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:24:20.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osdmap e430: 12 total, 12 up, 8 in 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:20.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:20.675 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:20.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:20.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:20.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:20.828 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:24:20.828 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:24:20.831 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:20.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:20.955 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:20.985 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:21.086 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:21.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:21.224 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_2' removed 2024-04-29T14:24:21.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:21.556 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1114386034' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:24:21.580 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:24:21.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:21.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1114386034' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:24:21.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:21.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1114386034' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:24:21.839 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:21.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:21.896 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-29T14:24:21.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:22.007 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:22.134 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:22.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:22.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:22.270 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:22.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:22 smithi194 ceph-mon[44089]: pgmap v1137: 17 pgs: 17 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:22.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1114386034' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:24:22.408 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:22 smithi194 ceph-mon[44089]: osdmap e431: 12 total, 12 up, 8 in 2024-04-29T14:24:22.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:22 smithi040 ceph-mon[39581]: pgmap v1137: 17 pgs: 17 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:22.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1114386034' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:24:22.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:22 smithi040 ceph-mon[39581]: osdmap e431: 12 total, 12 up, 8 in 2024-04-29T14:24:22.452 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:24:22.453 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:24:22.453 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-04-29T14:24:22.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-04-29T14:24:22.469 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:22.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:22.502 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:22.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:22.599 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:22.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:22 smithi121 ceph-mon[44533]: pgmap v1137: 17 pgs: 17 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:22.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1114386034' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:24:22.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:22 smithi121 ceph-mon[44533]: osdmap e431: 12 total, 12 up, 8 in 2024-04-29T14:24:22.805 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:22.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:22.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:22.938 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:22.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:22.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:22.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:23.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:23.088 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:23.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:24:23.342 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:23.342 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:23.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:23.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:23.474 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:23.547 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:23.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:23.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:23.699 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:23.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:23.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:23.829 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:24:23.830 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:24:23.834 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:23.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:23.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:23.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:23.984 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:24.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:24.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:24.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:24 smithi040 ceph-mon[39581]: pgmap v1139: 1 pgs: 1 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:24.218 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:24.486 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:24 smithi194 ceph-mon[44089]: pgmap v1139: 1 pgs: 1 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:24.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:24:24.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:24 smithi121 ceph-mon[44533]: pgmap v1139: 1 pgs: 1 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:24.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:24.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:24.803 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:24.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:24.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:24.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:24.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:24.956 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:25.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:25.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:26 smithi194 ceph-mon[44089]: pgmap v1140: 1 pgs: 1 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:26.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:26 smithi040 ceph-mon[39581]: pgmap v1140: 1 pgs: 1 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:26.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:26 smithi121 ceph-mon[44533]: pgmap v1140: 1 pgs: 1 active+clean; 577 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:27.588 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:27.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:27.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:27.728 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:27.878 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:27.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:27.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:28.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:28.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:28.140 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:28.174 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:28.195 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:28.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:28.300 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:28.330 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:28.349 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:28.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:28.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:28 smithi121 ceph-mon[44533]: pgmap v1141: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:28.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:28.485 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:28.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:28.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:28.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:28 smithi194 ceph-mon[44089]: pgmap v1141: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:28.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:28.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:28 smithi040 ceph-mon[39581]: pgmap v1141: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:28.638 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:28.655 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:28.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:28.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:28.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:28.787 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:28.806 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:28.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:28.910 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:28.936 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:28.954 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:28.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:29.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:29.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:29.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:29.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:29.218 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.10 to 1 (10000) 2024-04-29T14:24:29.248 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:29.248 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:24:29.248 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_3' created 2024-04-29T14:24:29.261 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:29.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:29.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:29 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:24:29.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:29 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:24:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1198613678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-29T14:24:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:29 smithi040 ceph-mon[39581]: from='client.26002 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-29T14:24:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1675646583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:24:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:29 smithi040 ceph-mon[39581]: from='client.26008 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:24:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1024362674' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:24:29.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:29 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:24:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:29 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:24:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1198613678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-29T14:24:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:29 smithi121 ceph-mon[44533]: from='client.26002 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-29T14:24:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1675646583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:24:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:29 smithi121 ceph-mon[44533]: from='client.26008 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:24:29.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1024362674' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:24:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:29 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:24:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:29 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:24:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1198613678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-29T14:24:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:29 smithi194 ceph-mon[44089]: from='client.26002 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-29T14:24:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1675646583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:24:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:29 smithi194 ceph-mon[44089]: from='client.26008 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:24:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1024362674' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:24:30.245 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:30.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:30.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:30.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:30 smithi040 ceph-mon[39581]: pgmap v1142: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:30.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:30 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:30.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:30 smithi040 ceph-mon[39581]: from='client.26002 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-29T14:24:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:30 smithi040 ceph-mon[39581]: from='client.26008 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:24:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1024362674' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:24:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:30 smithi040 ceph-mon[39581]: osdmap e432: 12 total, 12 up, 8 in 2024-04-29T14:24:30.378 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:24:30.382 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:30.394 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:24:30.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:30.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:30.413 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:24:30.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:24:30.453 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:24:30.474 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:24:30.497 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:24:30.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:30.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:24:30.537 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:30 smithi194 ceph-mon[44089]: pgmap v1142: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:30.538 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:30 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:30.538 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:30 smithi194 ceph-mon[44089]: from='client.26002 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-29T14:24:30.538 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:30 smithi194 ceph-mon[44089]: from='client.26008 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:24:30.538 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1024362674' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:24:30.538 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:30 smithi194 ceph-mon[44089]: osdmap e432: 12 total, 12 up, 8 in 2024-04-29T14:24:30.539 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:24:30.540 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:30.560 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:24:30.562 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:30.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:30.581 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:24:30.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:24:30.609 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:24:30.609 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:24:30.609 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:24:30.610 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:24:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:30 smithi121 ceph-mon[44533]: pgmap v1142: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:30 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:30 smithi121 ceph-mon[44533]: from='client.26002 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-29T14:24:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:30 smithi121 ceph-mon[44533]: from='client.26008 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:24:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1024362674' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:24:30.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:30 smithi121 ceph-mon[44533]: osdmap e432: 12 total, 12 up, 8 in 2024-04-29T14:24:30.666 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:30.695 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:30.714 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:30.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:30.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:30.887 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:30.888 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:24:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:24:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:24:30.888 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:24:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:30.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:30.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:31.020 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:31.093 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:31.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:31.198 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:31.234 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:31.255 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:31.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:31.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:31.391 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:31.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:31.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:31.516 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:31.544 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osdmap e433: 12 total, 12 up, 8 in 2024-04-29T14:24:31.544 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1671916582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.547 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:31.567 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:31.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:31.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osdmap e433: 12 total, 12 up, 8 in 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1671916582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osdmap e433: 12 total, 12 up, 8 in 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1671916582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:31.671 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:31.699 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:31.719 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:31.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:31.823 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:31.850 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:31.868 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:31.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:31.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:31.999 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:32.019 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:32.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:32.123 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:32.150 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:32.168 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:32.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:32.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:32.308 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:32.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:32 smithi194 ceph-mon[44089]: pgmap v1145: 17 pgs: 5 creating+activating, 11 unknown, 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:32.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:32 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:24:32.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:32 smithi194 ceph-mon[44089]: osdmap e434: 12 total, 12 up, 8 in 2024-04-29T14:24:32.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:32 smithi040 ceph-mon[39581]: pgmap v1145: 17 pgs: 5 creating+activating, 11 unknown, 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:32.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:32 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:24:32.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:32 smithi040 ceph-mon[39581]: osdmap e434: 12 total, 12 up, 8 in 2024-04-29T14:24:32.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:32 smithi121 ceph-mon[44533]: pgmap v1145: 17 pgs: 5 creating+activating, 11 unknown, 1 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:32.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:32 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:24:32.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:32 smithi121 ceph-mon[44533]: osdmap e434: 12 total, 12 up, 8 in 2024-04-29T14:24:32.660 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:32.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:32.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-04-29T14:24:32.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:32.799 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:32.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:32.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:32.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:32.955 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:32.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:32.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:32.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-29T14:24:33.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:33.109 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:33.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:33.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:33.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:33.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:33.262 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:33.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:33.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:33.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:33.413 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:33.536 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:33.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:33.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:24:33.640 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:33.670 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:33.674 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:24:33.794 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:33.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:33.900 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:33.930 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:34.371 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:34 smithi194 ceph-mon[44089]: pgmap v1147: 17 pgs: 9 active+clean, 5 creating+activating, 3 unknown; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:24:34.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:24:34.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:34 smithi040 ceph-mon[39581]: pgmap v1147: 17 pgs: 9 active+clean, 5 creating+activating, 3 unknown; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:34.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:24:34.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:24:34.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:34 smithi121 ceph-mon[44533]: pgmap v1147: 17 pgs: 9 active+clean, 5 creating+activating, 3 unknown; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:34.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:24:34.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:24:34.679 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:34.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:34.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:34.810 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:35.512 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:35.512 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:24:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:24:35.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:35 smithi194 ceph-mon[44089]: osdmap e435: 12 total, 12 up, 8 in 2024-04-29T14:24:35.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:24:35.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:24:35.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:35 smithi040 ceph-mon[39581]: osdmap e435: 12 total, 12 up, 8 in 2024-04-29T14:24:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:24:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:24:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:35 smithi121 ceph-mon[44533]: osdmap e435: 12 total, 12 up, 8 in 2024-04-29T14:24:35.868 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:35.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:35.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:36.005 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:36.026 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:36.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:36.130 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:36.161 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:36.182 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:36.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:36.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:36.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:36 smithi194 ceph-mon[44089]: pgmap v1149: 17 pgs: 12 active+clean, 5 creating+activating; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:36.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:24:36.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:24:36.316 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:36.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:24:36.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:36 smithi040 ceph-mon[39581]: pgmap v1149: 17 pgs: 12 active+clean, 5 creating+activating; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:36.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:24:36.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:24:36.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:36 smithi121 ceph-mon[44533]: pgmap v1149: 17 pgs: 12 active+clean, 5 creating+activating; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:36.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:24:36.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:24:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:37 smithi194 ceph-mon[44089]: osdmap e436: 12 total, 12 up, 8 in 2024-04-29T14:24:37.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:37 smithi194 ceph-mon[44089]: pgmap v1151: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:37.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:37 smithi040 ceph-mon[39581]: osdmap e436: 12 total, 12 up, 8 in 2024-04-29T14:24:37.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:37 smithi040 ceph-mon[39581]: pgmap v1151: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:37.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:37 smithi121 ceph-mon[44533]: osdmap e436: 12 total, 12 up, 8 in 2024-04-29T14:24:37.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:37 smithi121 ceph-mon[44533]: pgmap v1151: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:38.067 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:38.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:38.190 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:38.222 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:38.244 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:38.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:38.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:38.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:38.378 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:38.398 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:38.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:38.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:38.533 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:39.266 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-04-29T14:24:39.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:39 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:24:39.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:39 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:24:39.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:39 smithi040 ceph-mon[39581]: osdmap e437: 12 total, 12 up, 8 in 2024-04-29T14:24:39.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/524843067' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:24:39.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:39 smithi040 ceph-mon[39581]: pgmap v1153: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:39 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:24:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:39 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:24:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:39 smithi194 ceph-mon[44089]: osdmap e437: 12 total, 12 up, 8 in 2024-04-29T14:24:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/524843067' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:24:39.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:39 smithi194 ceph-mon[44089]: pgmap v1153: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:39.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:39 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:24:39.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:39 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:24:39.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:39 smithi121 ceph-mon[44533]: osdmap e437: 12 total, 12 up, 8 in 2024-04-29T14:24:39.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/524843067' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:24:39.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:39 smithi121 ceph-mon[44533]: pgmap v1153: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:40.084 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:40.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:40.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:40.221 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:40.242 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:40.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:40.273 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.11 to 1 (10000) 2024-04-29T14:24:40.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:40.375 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:40.395 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:40.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:40.498 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:40.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/524843067' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:24:40.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:40 smithi194 ceph-mon[44089]: osdmap e438: 12 total, 12 up, 8 in 2024-04-29T14:24:40.524 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3653841811' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T14:24:40.526 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:40.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:40.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/524843067' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:24:40.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:40 smithi040 ceph-mon[39581]: osdmap e438: 12 total, 12 up, 8 in 2024-04-29T14:24:40.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3653841811' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T14:24:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/524843067' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:24:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:40 smithi121 ceph-mon[44533]: osdmap e438: 12 total, 12 up, 8 in 2024-04-29T14:24:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3653841811' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T14:24:40.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:40.682 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:40.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:40.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:24:40.799 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:24:40.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:40.813 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:24:40.815 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:24:40.816 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:24:40.816 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:24:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:40.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:40.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:40.960 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:40.988 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:41.008 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:41.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:41.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:41.143 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:41.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:41.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:41.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:41.299 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:41.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:41.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:41.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3653841811' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osdmap e439: 12 total, 12 up, 8 in 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4257551048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.450 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.451 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.451 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.451 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.451 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:41 smithi040 ceph-mon[39581]: pgmap v1156: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:41.453 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3653841811' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osdmap e439: 12 total, 12 up, 8 in 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4257551048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:41 smithi194 ceph-mon[44089]: pgmap v1156: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3653841811' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T14:24:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osdmap e439: 12 total, 12 up, 8 in 2024-04-29T14:24:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4257551048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:41.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:41 smithi121 ceph-mon[44533]: pgmap v1156: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:42.151 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-04-29T14:24:42.153 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:24:42.153 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:24:42.249 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:24:42.249 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:24:42.249 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_121512 2024-04-29T14:24:42.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:24:42.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:24:42.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:42 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:24:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:42.402 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:24:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:42.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:24:42.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-29T14:24:42.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:42.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:42 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:24:42.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:42.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:42 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:24:42.749 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:42.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:42.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:42.903 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:42.927 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:42.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:43.032 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:43.071 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:43.099 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:43.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:43.204 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:43.241 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:43.249 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 16 4460 4444 277.734 277.75 0.00236071 0.0035946 2024-04-29T14:24:43.268 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:43.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:43.273 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:24:43.273 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:24:43.287 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:43 smithi121 ceph-mon[44533]: pgmap v1157: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:43.372 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:43.405 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:43 smithi194 ceph-mon[44089]: pgmap v1157: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:43.433 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:43.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:43.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:43.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:43 smithi040 ceph-mon[39581]: pgmap v1157: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-04-29T14:24:43.573 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:44.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:44.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:24:44.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:44.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 15 8700 8685 271.387 265.062 0.00365901 0.00368066 2024-04-29T14:24:44.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:44.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:44.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:44.409 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:44.790 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:44.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:44.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:24:44.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:44.939 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:45.068 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:45.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:45.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:45.207 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:45.236 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:45.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:45.244 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:45.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 16 13300 13284 276.73 287.438 0.00185372 0.00360972 2024-04-29T14:24:45.340 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:45.366 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:45.384 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:45.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:45.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:45.830 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:24:46.073 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:46.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:45 smithi040 ceph-mon[39581]: pgmap v1158: 17 pgs: 17 active+clean; 23 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s wr, 55 op/s 2024-04-29T14:24:46.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4189472483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:24:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:46.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:46.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:45 smithi121 ceph-mon[44533]: pgmap v1158: 17 pgs: 17 active+clean; 23 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s wr, 55 op/s 2024-04-29T14:24:46.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4189472483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:24:46.216 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:46.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 16 16561 16545 258.496 203.812 0.0022671 0.00386515 2024-04-29T14:24:46.251 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:45 smithi194 ceph-mon[44089]: pgmap v1158: 17 pgs: 17 active+clean; 23 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 MiB/s wr, 55 op/s 2024-04-29T14:24:46.251 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4189472483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:24:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:46.280 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:46.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:46.385 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:46.446 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:46.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:46.550 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:46.584 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:46.607 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:46.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:46.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:46.750 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:46.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:46.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:46 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:46.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4189472483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:46.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:46 smithi194 ceph-mon[44089]: osdmap e440: 12 total, 12 up, 8 in 2024-04-29T14:24:46.885 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:46.948 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:46.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:47.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:47.092 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:46 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:47.092 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4189472483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:47.092 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:46 smithi121 ceph-mon[44533]: osdmap e440: 12 total, 12 up, 8 in 2024-04-29T14:24:47.097 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:47.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:46 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:47.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4189472483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:24:47.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:46 smithi040 ceph-mon[39581]: osdmap e440: 12 total, 12 up, 8 in 2024-04-29T14:24:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:47.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:47.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:47.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 16 20455 20439 255.468 243.375 0.00452358 0.00391177 2024-04-29T14:24:47.275 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:47.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:47.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:47.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:47.446 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:47.487 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:47.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:47.592 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:47.628 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:47.652 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:47.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:47.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:47.807 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:47.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:47.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:47.945 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:47.977 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:47 smithi194 ceph-mon[44089]: pgmap v1160: 17 pgs: 17 active+clean; 798 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-29T14:24:47.980 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:48.008 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:48.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:48.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:47 smithi040 ceph-mon[39581]: pgmap v1160: 17 pgs: 17 active+clean; 798 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-29T14:24:48.116 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:48.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:47 smithi121 ceph-mon[44533]: pgmap v1160: 17 pgs: 17 active+clean; 798 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-29T14:24:48.148 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:48.175 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:48.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:48.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 15 24097 24082 250.835 227.688 0.00605682 0.00398378 2024-04-29T14:24:48.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:48.326 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:48.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:48.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:24:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:48.519 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:24:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:48.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:24:48.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:48.670 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:48.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:48.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:48.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:48.842 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:48.857 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.6 to 1 (10000) 2024-04-29T14:24:48.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3949312399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:24:49.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3949312399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:24:49.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 15 27549 27534 245.821 215.75 0.00530367 0.00406661 2024-04-29T14:24:49.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3949312399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:24:49.675 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:49.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:49.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:49.827 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:49.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:49.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:49 smithi040 ceph-mon[39581]: pgmap v1161: 17 pgs: 17 active+clean; 1.0 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:24:49.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3949312399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:24:49.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:49 smithi040 ceph-mon[39581]: osdmap e441: 12 total, 12 up, 8 in 2024-04-29T14:24:49.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:50.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:50.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:50.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:24:50.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:49 smithi121 ceph-mon[44533]: pgmap v1161: 17 pgs: 17 active+clean; 1.0 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:24:50.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3949312399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:24:50.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:49 smithi121 ceph-mon[44533]: osdmap e441: 12 total, 12 up, 8 in 2024-04-29T14:24:50.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:50.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 16 30838 30822 240.779 205.5 0.00188952 0.00415097 2024-04-29T14:24:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:49 smithi194 ceph-mon[44089]: pgmap v1161: 17 pgs: 17 active+clean; 1.0 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:24:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3949312399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:24:50.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:49 smithi194 ceph-mon[44089]: osdmap e441: 12 total, 12 up, 8 in 2024-04-29T14:24:50.480 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-29T14:24:50.481 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-29T14:24:50.481 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-29T14:24:50.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T14:24:51.227 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:51.233 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:51.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:51.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 15 34059 34044 236.398 201.375 0.00265464 0.00422808 2024-04-29T14:24:51.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:51.381 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:51.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:51.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:51.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:24:51.560 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:51.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:24:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:51.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:24:51.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:51.726 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:51.754 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:51.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:51.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:51.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:51 smithi121 ceph-mon[44533]: pgmap v1163: 17 pgs: 17 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-04-29T14:24:51.908 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:51.939 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:51.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:51.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:51 smithi040 ceph-mon[39581]: pgmap v1163: 17 pgs: 17 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-04-29T14:24:52.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:24:52.062 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:51 smithi194 ceph-mon[44089]: pgmap v1163: 17 pgs: 17 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-04-29T14:24:52.083 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:52.110 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:24:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:52.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:24:52.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:52.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 16 36973 36957 230.963 182.062 0.00564147 0.00432767 2024-04-29T14:24:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:52.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:52.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:24:52.456 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:52.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:24:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:52.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:24:52.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:24:52.636 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:52.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:24:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:52.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:24:52.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:52.784 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:24:52.784 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:24:52.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:24:52.784 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:24:52.784 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:24:52.799 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:52.822 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:52.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:52.868 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:24:52.868 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:24:52.868 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:24:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:24:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:24:52.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:52.960 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:52.984 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:52.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osdmap e442: 12 total, 12 up, 8 in 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osdmap e443: 12 total, 12 up, 8 in 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1572680988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:53.123 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:53.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:24:53.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:24:53.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osdmap e442: 12 total, 12 up, 8 in 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osdmap e443: 12 total, 12 up, 8 in 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1572680988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.145 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:53.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:53.250 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:53.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 39769 39753 225.852 174.75 0.00873628 0.00442549 2024-04-29T14:24:53.282 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:53.308 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:53.309 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:53.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:24:53.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osdmap e442: 12 total, 12 up, 8 in 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osdmap e443: 12 total, 12 up, 8 in 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1572680988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:53.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:53.448 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:53.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:53.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:24:54.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:54.036 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:54.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.120 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:53 smithi040 ceph-mon[39581]: pgmap v1166: 17 pgs: 17 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-04-29T14:24:54.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:53 smithi121 ceph-mon[44533]: pgmap v1166: 17 pgs: 17 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-04-29T14:24:54.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 42577 42561 221.654 175.5 0.00370551 0.00450909 2024-04-29T14:24:54.284 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:24:54.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:24:54.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:53 smithi194 ceph-mon[44089]: pgmap v1166: 17 pgs: 17 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-04-29T14:24:55.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 16 44800 44784 215.291 138.938 0.00378861 0.00464297 2024-04-29T14:24:55.290 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:55.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:55.397 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:24:55.434 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:55.947 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:55 smithi040 ceph-mon[39581]: pgmap v1167: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 180 MiB/s wr, 2.89k op/s 2024-04-29T14:24:56.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:55 smithi121 ceph-mon[44533]: pgmap v1167: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 180 MiB/s wr, 2.89k op/s 2024-04-29T14:24:56.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 16 47429 47413 211.648 164.312 0.00483703 0.00472315 2024-04-29T14:24:56.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:55 smithi194 ceph-mon[44089]: pgmap v1167: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 180 MiB/s wr, 2.89k op/s 2024-04-29T14:24:57.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 16 49804 49788 207.433 148.438 0.00193352 0.0048109 2024-04-29T14:24:57.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:24:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:24:58.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:57 smithi040 ceph-mon[39581]: pgmap v1168: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 224 MiB/s wr, 3.58k op/s 2024-04-29T14:24:58.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:57 smithi040 ceph-mon[39581]: from='client.16254 v1:172.21.15.40:0/1851664325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:24:58.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:57 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:58.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:57 smithi040 ceph-mon[39581]: from='client.16254 v1:172.21.15.40:0/1851664325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:24:58.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:57 smithi040 ceph-mon[39581]: osdmap e444: 12 total, 12 up, 8 in 2024-04-29T14:24:58.179 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:57 smithi194 ceph-mon[44089]: pgmap v1168: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 224 MiB/s wr, 3.58k op/s 2024-04-29T14:24:58.179 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:57 smithi194 ceph-mon[44089]: from='client.16254 v1:172.21.15.40:0/1851664325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:24:58.179 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:57 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:58.179 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:57 smithi194 ceph-mon[44089]: from='client.16254 v1:172.21.15.40:0/1851664325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:24:58.179 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:57 smithi194 ceph-mon[44089]: osdmap e444: 12 total, 12 up, 8 in 2024-04-29T14:24:58.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 15 51980 51965 202.972 136.062 0.00259492 0.00492511 2024-04-29T14:24:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:24:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:58.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:24:58.356 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:24:58.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:57 smithi121 ceph-mon[44533]: pgmap v1168: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 224 MiB/s wr, 3.58k op/s 2024-04-29T14:24:58.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:57 smithi121 ceph-mon[44533]: from='client.16254 v1:172.21.15.40:0/1851664325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:24:58.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:57 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:24:58.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:57 smithi121 ceph-mon[44533]: from='client.16254 v1:172.21.15.40:0/1851664325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:24:58.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:57 smithi121 ceph-mon[44533]: osdmap e444: 12 total, 12 up, 8 in 2024-04-29T14:24:58.388 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:58.409 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:24:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:58.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:24:58.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:24:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:24:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:58.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:24:58.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:58.760 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:24:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:59.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:24:59.129 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:24:59.171 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:59.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 53921 53905 198.164 121.25 0.00558186 0.00504441 2024-04-29T14:24:59.354 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:24:59.426 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:24:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:59.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:24:59.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:24:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:24:59.600 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:24:59.600 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":444,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T14:24:57.154087+0000","last_up_change":"2024-04-29T14:22:12.596258+0000","last_in_change":"2024-04-29T14:19:01.251051+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":10,"score_stable":10,"optimal_score":0.20000000298023224,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2024-04-29T14:24:28.811762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"438","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 5 has some PGs where all OSDs are with primary_affinity 0 (at least pg 5.8), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":323,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":4178456629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":4178456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":4178456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":4178456629}]},"public_addr":"172.21.15.40:6805/4178456629","cluster_addr":"172.21.15.40:6806/4178456629","heartbeat_back_addr":"172.21.15.40:6808/4178456629","heartbeat_front_addr":"172.21.15.40:6807/4178456629","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":230,"up_from":260,"up_thru":152,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3416690575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3416690575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3416690575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3416690575}]},"public_addr":"172.21.15.40:6809/3416690575","cluster_addr":"172.21.15.40:6810/3416690575","heartbeat_back_addr":"172.21.15.40:6812/3416690575","heartbeat_front_addr":"172.21.15.40:6811/3416690575","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":432,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":369,"up_from":388,"up_thru":432,"down_at":370,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2586630078}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2586630078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2586630078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2586630078}]},"public_addr":"172.21.15.121:6804/2586630078","cluster_addr":"172.21.15.121:6805/2586630078","heartbeat_back_addr":"172.21.15.121:6807/2586630078","heartbeat_front_addr":"172.21.15.121:6806/2586630078","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":432,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":432,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":64,"last_clean_end":320,"up_from":354,"up_thru":206,"down_at":321,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":165633203}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":165633203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":165633203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":165633203}]},"public_addr":"172.21.15.194:6800/165633203","cluster_addr":"172.21.15.194:6801/165633203","heartbeat_back_addr":"172.21.15.194:6803/165633203","heartbeat_front_addr":"172.21.15.194:6802/165633203","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":432,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":453882699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":453882699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":453882699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":453882699}]},"public_addr":"172.21.15.194:6804/453882699","cluster_addr":"172.21.15.194:6805/453882699","heartbeat_back_addr":"172.21.15.194:6807/453882699","heartbeat_front_addr":"172.21.15.194:6806/453882699","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":432,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":432,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:24:38.006240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":51693,"last_purged_snaps_scrub":"2024-04-29T14:24:13.248344+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-29T14:23:50.451188+0000","dead_epoch":230},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-29T14:24:35.757203+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:24:34.028960+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:23:52.479324+0000","dead_epoch":369},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:23:57.128982+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:24:33.477856+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-29T14:24:51.725200+0000","dead_epoch":320},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:23:43.100359+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:24:05.142783+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:23:51.962294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T14:24:59.640 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:24:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:24:59.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:24:59.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:24:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:00.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 56069 56053 194.612 134.25 0.0082044 0.00513667 2024-04-29T14:25:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:59 smithi194 ceph-mon[44089]: pgmap v1170: 17 pgs: 17 active+clean; 2.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-04-29T14:25:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:25:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:24:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1054504117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:25:00.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:59 smithi040 ceph-mon[39581]: pgmap v1170: 17 pgs: 17 active+clean; 2.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-04-29T14:25:00.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:25:00.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:24:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1054504117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:25:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:59 smithi121 ceph-mon[44533]: pgmap v1170: 17 pgs: 17 active+clean; 2.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-04-29T14:25:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:25:00.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:24:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1054504117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:25:00.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:25:00.829 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:25:00.829 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:25:00.829 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:25:00.829 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:25:00.929 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-29T14:25:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:00.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:01.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:01.142 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:01.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:01.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:01.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 58134 58118 191.162 129.062 0.0102902 0.00522919 2024-04-29T14:25:01.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:01.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:01.542 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:01.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:01.620 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:25:01.628 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:01.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:01.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:01.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:01.793 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:01.819 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:01.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:01.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:01 smithi194 ceph-mon[44089]: pgmap v1171: 17 pgs: 17 active+clean; 3.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-29T14:25:01.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:01 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:25:01.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:01 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:25:01.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:01.949 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:25:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:02.017 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:02.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:02.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:02.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:01 smithi121 ceph-mon[44533]: pgmap v1171: 17 pgs: 17 active+clean; 3.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-29T14:25:02.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:01 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:25:02.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:01 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:25:02.182 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:02.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:02.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:02.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:25:02.259724 min lat: 0.00099482 max lat: 0.215835 avg lat: 0.00532682 2024-04-29T14:25:02.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:25:02.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 16 60074 60058 187.666 121.25 0.00649596 0.00532682 2024-04-29T14:25:02.316 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:02.348 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:02.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:01 smithi040 ceph-mon[39581]: pgmap v1171: 17 pgs: 17 active+clean; 3.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-04-29T14:25:02.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:01 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:25:02.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:01 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:25:02.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:25:02.467 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:25:02.476 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:02.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:02.488 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:25:02.513 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:25:02.546 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:25:02.572 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:25:02.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:02.602 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:25:02.620 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:02.629 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:25:02.652 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:25:02.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:02.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:02.679 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:25:02.702 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:25:02.725 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:25:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:25:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:25:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:25:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:25:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:25:02.757 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:02.790 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:02.818 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:02.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:02.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:02.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:02.969 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:03.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osdmap e445: 12 total, 12 up, 8 in 2024-04-29T14:25:03.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osdmap e446: 12 total, 12 up, 8 in 2024-04-29T14:25:03.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3294828737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:03.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osdmap e445: 12 total, 12 up, 8 in 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osdmap e446: 12 total, 12 up, 8 in 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3294828737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:03.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osdmap e445: 12 total, 12 up, 8 in 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osdmap e446: 12 total, 12 up, 8 in 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3294828737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.151 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:03.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:03.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 62181 62165 185 131.688 0.00611955 0.00540354 2024-04-29T14:25:03.300 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:03.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:03.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:03.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:03.478 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:03.509 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:03.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:03.615 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:03.654 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:03.729 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:03.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:03.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:03.869 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:04.004 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:04.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.007 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:03 smithi194 ceph-mon[44089]: pgmap v1174: 17 pgs: 17 active+clean; 3.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-29T14:25:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:04.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:04.060 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:25:04.060 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:25:04.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:03 smithi040 ceph-mon[39581]: pgmap v1174: 17 pgs: 17 active+clean; 3.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-29T14:25:04.144 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:04.201 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:04.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:04.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 16 64219 64203 182.38 127.375 0.0120523 0.00548161 2024-04-29T14:25:04.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:04.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:04.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:03 smithi121 ceph-mon[44533]: pgmap v1174: 17 pgs: 17 active+clean; 3.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-29T14:25:04.356 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:04.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:04.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:04.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:04.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:04.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:04.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:04.728 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:04.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:04.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:04.819 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:25:04.860 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:04.898 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:04.931 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:04.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:05.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:05.068 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:05.178 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:25:05.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 66203 66187 179.841 124 0.0048769 0.00555879 2024-04-29T14:25:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:05.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:05.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:05.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:05.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:05.862 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:05.894 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:05.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:06.003 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:06.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:05 smithi194 ceph-mon[44089]: pgmap v1175: 17 pgs: 17 active+clean; 3.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-29T14:25:06.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:06.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:06.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3988535354' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:06.036 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:06.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:05 smithi121 ceph-mon[44533]: pgmap v1175: 17 pgs: 17 active+clean; 3.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-29T14:25:06.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:06.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:06.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3988535354' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:06.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 16 68163 68147 177.452 122.5 0.00360977 0.00563141 2024-04-29T14:25:06.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:05 smithi040 ceph-mon[39581]: pgmap v1175: 17 pgs: 17 active+clean; 3.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-29T14:25:06.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:06.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:06.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3988535354' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:06.425 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:25:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:06.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:06.594 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:06.628 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:06.777 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:06.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:06.883 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:06.917 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:06.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:25:07.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:25:07.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:25:07.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:06 smithi194 ceph-mon[44089]: pgmap v1176: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:25:07.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:06 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:07.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:06 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:25:07.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3988535354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:07.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:06 smithi194 ceph-mon[44089]: osdmap e447: 12 total, 12 up, 8 in 2024-04-29T14:25:07.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:25:07.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:25:07.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:06 smithi040 ceph-mon[39581]: pgmap v1176: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:25:07.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:06 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:07.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:06 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:25:07.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3988535354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:07.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:06 smithi040 ceph-mon[39581]: osdmap e447: 12 total, 12 up, 8 in 2024-04-29T14:25:07.138 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:25:07.138 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:25:07.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 16 69781 69765 174.398 101.125 0.00702992 0.00573189 2024-04-29T14:25:07.270 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:07.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:07.375 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:25:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:25:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:06 smithi121 ceph-mon[44533]: pgmap v1176: 17 pgs: 17 active+clean; 3.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-04-29T14:25:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:06 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:25:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:06 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:25:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3988535354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:07.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:06 smithi121 ceph-mon[44533]: osdmap e447: 12 total, 12 up, 8 in 2024-04-29T14:25:07.415 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:07.521 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:07.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:07.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:07.681 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:07.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:07.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:07.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:07.863 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:07.901 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:07.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:07.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:25:07.957 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:07 smithi121 ceph-mon[44533]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-29T14:25:07.957 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:07 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:25:08.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:08.191 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:08.193 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:07 smithi040 ceph-mon[39581]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-29T14:25:08.193 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:07 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:25:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:08.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:08.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 71603 71587 172.07 113.875 0.00240906 0.00580917 2024-04-29T14:25:08.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:08.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:07 smithi194 ceph-mon[44089]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-29T14:25:08.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:07 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:25:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:08.391 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:25:08.777 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:08.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:08.883 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:08.914 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:08.934 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:08.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:09.038 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:09.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:08 smithi040 ceph-mon[39581]: pgmap v1178: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-04-29T14:25:09.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:25:09.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:25:09.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:25:09.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:25:09.068 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:09.090 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:09.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:09.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:09.226 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:09.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 15 73175 73160 169.338 98.3125 0.00547278 0.00590374 2024-04-29T14:25:09.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:25:09.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:08 smithi194 ceph-mon[44089]: pgmap v1178: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-04-29T14:25:09.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:25:09.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:25:09.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:25:09.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:25:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:09.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:09.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:25:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:08 smithi121 ceph-mon[44533]: pgmap v1178: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-04-29T14:25:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:25:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:25:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:25:09.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:25:09.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:25:09.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:25:09.429 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:25:09.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:09.469 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:09.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:25:09.490 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:09.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:09.523 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:25:09.555 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:25:09.583 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:25:09.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:09.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:25:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:09.641 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:09.647 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:25:09.662 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:09.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:09.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:09.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:25:09.693 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:25:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:25:09.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:09.837 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:09.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:09.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:09.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:09.920 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osdmap e448: 12 total, 12 up, 8 in 2024-04-29T14:25:09.920 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3548397562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:10.010 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:10.043 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:10.067 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:10.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:10.104 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osdmap e448: 12 total, 12 up, 8 in 2024-04-29T14:25:10.104 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3548397562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:10.104 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.104 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.104 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:10.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:10.244 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:10.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:10.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 16 74857 74841 167.042 105.062 0.00362964 0.00598453 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osdmap e448: 12 total, 12 up, 8 in 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3548397562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:10.347 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:10.378 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:10.398 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:10.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:10.502 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:10.533 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:10.555 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:10.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:10.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:10.793 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:11.101 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:11.101 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:11.101 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:10 smithi040 ceph-mon[39581]: pgmap v1180: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-29T14:25:11.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:11.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:11.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:10 smithi121 ceph-mon[44533]: pgmap v1180: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-29T14:25:11.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 76532 76516 164.892 104.688 0.00233162 0.00606283 2024-04-29T14:25:11.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:11.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:11.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:10 smithi194 ceph-mon[44089]: pgmap v1180: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-29T14:25:11.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:11.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:11.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:11.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:11.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:11.962 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:11.994 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:12.018 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:12.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:12.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:12.176 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:12.210 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:12.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:12.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 16 78181 78165 162.831 103.062 0.00385293 0.00613979 2024-04-29T14:25:12.298 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:25:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:25:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:78255 5.f 5.f4e1508f (undecoded) ondisk+write+known_if_redirected e449)", 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:12.303606+0000", 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0020486300000000001, 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020730689999999999, 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 78255 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:25:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.303606+0000", 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.303606+0000", 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.303608+0000", 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7770000000000001e-06 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.303704+0000", 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5937e-05 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.303716+0000", 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1829e-05 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.303723+0000", 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9990000000000002e-06 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:78259 5.6 5:67eb2418:::benchmark_data_smithi194_121512_object78258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e449)", 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:12.304337+0000", 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.001317663, 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014175209999999999, 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 78259 2024-04-29T14:25:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.304337+0000", 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.304337+0000", 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:25:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.304338+0000", 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.164e-06 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.304379+0000", 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1032999999999998e-05 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.304391+0000", 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1776e-05 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:25:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.304398+0000", 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0740000000000004e-06 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305149+0000", 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075055300000000005 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305236+0000", 2024-04-29T14:25:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7201999999999996e-05 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305328+0000", 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1729000000000002e-05 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26047.0:78257 5.4 e449/432 5:2887afdc:::benchmark_data_smithi194_121512_object78256:head v 449'4840, mlcod=449'4837)", 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:12.305143+0000", 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00051169300000000002, 2024-04-29T14:25:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070052600000000003, 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305143+0000", 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305143+0000", 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305144+0000", 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.325e-06 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305179+0000", 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4813e-05 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305189+0000", 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6549999999999999e-06 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305196+0000", 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8449999999999997e-06 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305300+0000", 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010453100000000001 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305321+0000", 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0560999999999998e-05 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:25:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:78244 5.f 5:fdfcd7dd:::benchmark_data_smithi194_121512_object78243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e449)", 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:12.296889+0000", 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0087659869999999994, 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090218290000000003, 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 78244 2024-04-29T14:25:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.296889+0000", 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.296889+0000", 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.296889+0000", 2024-04-29T14:25:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6000000000000003e-07 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.296920+0000", 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.296928+0000", 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1480000000000006e-06 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.296934+0000", 2024-04-29T14:25:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9839999999999998e-06 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.302832+0000", 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058980400000000002 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.302946+0000", 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114196 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.303069+0000", 2024-04-29T14:25:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012297999999999999 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:78254 5.f 5:f0111659:::benchmark_data_smithi194_121512_object78253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e449)", 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:12.300394+0000", 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0052607089999999997, 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055919979999999999, 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:25:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 78254 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.300394+0000", 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.300394+0000", 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.300396+0000", 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8560000000000001e-06 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.300429+0000", 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2959000000000001e-05 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.300444+0000", 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5044e-05 2024-04-29T14:25:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.300453+0000", 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9360000000000005e-06 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305473+0000", 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050197710000000001 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305593+0000", 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012077400000000001 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:25:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:12.305733+0000", 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014006000000000001 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 5 2024-04-29T14:25:12.318 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:25:12.318 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:12.355 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:12.383 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:12.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:12.487 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:12.519 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:12.545 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:12.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:12.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:12.686 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:12.947 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:25:12.947 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:25:12.947 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:12 smithi121 ceph-mon[44533]: osdmap e449: 12 total, 12 up, 8 in 2024-04-29T14:25:12.947 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:12 smithi121 ceph-mon[44533]: pgmap v1182: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-04-29T14:25:12.947 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/441363857' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:25:12.947 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:25:13.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:25:13.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:13.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:13.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:25:13.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:13.219 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:13.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 16 79849 79833 160.941 104.25 0.00478501 0.0062115 2024-04-29T14:25:13.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:13.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:13.304 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:25:13.304 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:25:13.304 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:12 smithi040 ceph-mon[39581]: osdmap e449: 12 total, 12 up, 8 in 2024-04-29T14:25:13.304 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:12 smithi040 ceph-mon[39581]: pgmap v1182: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-04-29T14:25:13.304 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/441363857' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:25:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:25:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:25:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:12 smithi194 ceph-mon[44089]: osdmap e449: 12 total, 12 up, 8 in 2024-04-29T14:25:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:12 smithi194 ceph-mon[44089]: pgmap v1182: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-04-29T14:25:13.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/441363857' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:25:13.361 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:13.393 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:13.416 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:13.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:13.520 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:13.541 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:25:13.552 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:13.573 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:13.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:13.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:13.733 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:13.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:13.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:13.874 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:13.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:13.962 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:14.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:13 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/441363857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:25:14.114 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:13 smithi040 ceph-mon[39581]: osdmap e450: 12 total, 12 up, 8 in 2024-04-29T14:25:14.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:13 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:14.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/441363857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:25:14.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:13 smithi121 ceph-mon[44533]: osdmap e450: 12 total, 12 up, 8 in 2024-04-29T14:25:14.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 81416 81400 158.972 97.9375 0.00253478 0.00628003 2024-04-29T14:25:14.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:13 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/441363857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:25:14.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:13 smithi194 ceph-mon[44089]: osdmap e450: 12 total, 12 up, 8 in 2024-04-29T14:25:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:14.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:15.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:15.074 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:15.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:15 smithi040 ceph-mon[39581]: pgmap v1184: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-29T14:25:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:15.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:15.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 83042 83026 157.234 101.625 0.00334175 0.00635803 2024-04-29T14:25:15.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:15.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:15 smithi194 ceph-mon[44089]: pgmap v1184: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-29T14:25:15.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:15 smithi121 ceph-mon[44533]: pgmap v1184: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-29T14:25:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:15.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:15.616 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:15.649 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:15.671 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:15.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:15.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:25:15.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:15.824 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:25:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:15.863 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:25:15.903 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:25:15.944 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:25:16.011 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:25:16.036 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:25:16.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1809927218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:16.120 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:25:16.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1809927218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:16.149 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:25:16.178 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:25:16.203 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:25:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:25:16.252 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:25:16.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 16 84602 84586 155.477 97.5 0.00365349 0.00642952 2024-04-29T14:25:16.253 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:25:16.253 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:25:16.253 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:25:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:25:16.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1809927218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:16.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:16.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:16.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:17.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: pgmap v1185: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 86251 86235 153.979 103.062 0.00279592 0.00649251 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: pgmap v1185: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:25:17.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: pgmap v1185: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:17.573 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:25:17.573 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:25:18.106 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:18.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:18.213 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:18.243 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:18.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 87866 87850 152.505 100.938 0.00581748 0.00655548 2024-04-29T14:25:18.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:18 smithi194 ceph-mon[44089]: osdmap e451: 12 total, 12 up, 8 in 2024-04-29T14:25:18.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:18 smithi040 ceph-mon[39581]: osdmap e451: 12 total, 12 up, 8 in 2024-04-29T14:25:18.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:18 smithi121 ceph-mon[44533]: osdmap e451: 12 total, 12 up, 8 in 2024-04-29T14:25:18.443 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:18.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:18.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:18.629 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:19.047 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:25:19.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:19.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:19.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:25:19.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 89496 89480 151.137 101.875 0.00940033 0.00661481 2024-04-29T14:25:19.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:19.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:19 smithi194 ceph-mon[44089]: pgmap v1187: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:25:19.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1306303276' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:19.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:19 smithi194 ceph-mon[44089]: from='client.26104 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:19.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:19 smithi121 ceph-mon[44533]: pgmap v1187: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:25:19.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1306303276' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:19.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:19 smithi121 ceph-mon[44533]: from='client.26104 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:19.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:19 smithi040 ceph-mon[39581]: pgmap v1187: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:25:19.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1306303276' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:19.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:19 smithi040 ceph-mon[39581]: from='client.26104 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:19.362 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:19.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:19.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:19.544 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:19.727 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:25:19.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:25:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:19.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:20.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:20.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:20 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:20.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:20 smithi040 ceph-mon[39581]: from='client.26104 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:20.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:20 smithi040 ceph-mon[39581]: osdmap e452: 12 total, 12 up, 8 in 2024-04-29T14:25:20.054 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:20.067 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:20.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:20.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:20.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:20.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 90975 90959 149.592 92.4375 0.00742132 0.00668316 2024-04-29T14:25:20.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:20 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:20.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:20 smithi194 ceph-mon[44089]: from='client.26104 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:20.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:20 smithi194 ceph-mon[44089]: osdmap e452: 12 total, 12 up, 8 in 2024-04-29T14:25:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:20 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:20 smithi121 ceph-mon[44533]: from='client.26104 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:20 smithi121 ceph-mon[44533]: osdmap e452: 12 total, 12 up, 8 in 2024-04-29T14:25:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:20.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:20.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:20.690 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:20.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:20.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:20.840 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:20.870 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:20.893 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:20.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:20.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:25:20.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:25:20.999 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:21.029 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:21.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 92312 92296 147.898 83.5625 0.0219582 0.0067599 2024-04-29T14:25:21.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:21 smithi194 ceph-mon[44089]: pgmap v1189: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-29T14:25:21.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:21 smithi040 ceph-mon[39581]: pgmap v1189: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-29T14:25:21.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:21 smithi121 ceph-mon[44533]: pgmap v1189: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-29T14:25:21.385 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:21.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:21.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:21.539 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:21.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:21.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:21.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:21.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:25:21.899 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:22.007 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:22.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:22.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:22.147 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:22.179 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:22.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:22.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:25:22.261284 min lat: 0.00099482 max lat: 0.215835 avg lat: 0.00682189 2024-04-29T14:25:22.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:25:22.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 93815 93799 146.549 93.9375 0.025583 0.00682189 2024-04-29T14:25:22.258 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:25:22.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:22.289 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:25:22.315 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:25:22.321 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:22.337 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:25:22.372 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:25:22.398 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:25:22.398 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:25:22.401 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:25:22.435 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:25:22.461 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:25:22.486 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:25:22.510 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:25:22.535 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:25:22.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:25:22.558 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:25:22.558 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:25:22.558 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:25:22.558 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:25:22.558 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:25:22.558 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:25:22.559 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:25:22.559 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:25:22.559 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:25:22.559 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:25:22.559 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:25:22.559 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:25:22.873 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:22.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:22.978 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:23.010 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:23.032 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:23.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:23.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:23.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 95180 95164 145.056 85.3125 0.00846178 0.00689221 2024-04-29T14:25:23.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: pgmap v1190: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osdmap e453: 12 total, 12 up, 8 in 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2401103095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: pgmap v1190: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:25:23.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osdmap e453: 12 total, 12 up, 8 in 2024-04-29T14:25:23.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2401103095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:23.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: pgmap v1190: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-04-29T14:25:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osdmap e453: 12 total, 12 up, 8 in 2024-04-29T14:25:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2401103095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:23.922 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:23.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:24.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:25:24.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:24.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:24.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:24.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 16 96730 96714 143.908 96.875 0.00415744 0.0069471 2024-04-29T14:25:24.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:24.363 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:24.369 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:24.420 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:24.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:24.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:24.457 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:24.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:24.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:24.565 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:24.588 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:24.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:24.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:24.740 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:24.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:24.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:24.935 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:24.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:25.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:25.070 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:25.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:25.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:25 smithi040 ceph-mon[39581]: pgmap v1192: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:25:25.249 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:25.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 16 98283 98267 142.819 97.0625 0.0039455 0.0070002 2024-04-29T14:25:25.279 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:25.304 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:25.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:25.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:25.442 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:25:25.449 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:25:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:25.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:25.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:25 smithi194 ceph-mon[44089]: pgmap v1192: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:25:25.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:25.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:25:25.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:25 smithi121 ceph-mon[44533]: pgmap v1192: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-29T14:25:25.637 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:25.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:25.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:25.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:25.834 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:25.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:25.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:26.013 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:26.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:26.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:26.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 16 99829 99813 141.769 96.625 0.00435727 0.00705254 2024-04-29T14:25:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:26.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:26.858 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1192,"stamp":"2024-04-29T14:25:23.954615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6051922464,"num_objects":92338,"num_object_clones":0,"num_object_copies":184676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92338,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":92517,"num_write_kb":5912442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92473,"ondisk_log_size":92473,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15043384,"kb_used_data":11747868,"kb_used_omap":222,"kb_used_meta":3295201,"kb_avail":734917832,"statfs":{"total":767960285184,"available":752555859968,"internally_reserved":0,"allocated":12029816832,"data_stored":12025811270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":227868,"internal_metadata":3374286308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":602669056,"num_objects":9196,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9196,"num_write_kb":588544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.930546"},"pg_stats":[{"pgid":"5.f","version":"452'5946","reported_seq":5993,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.272129+0000","last_change":"2024-04-29T14:24:30.237056+0000","last_active":"2024-04-29T14:25:21.272129+0000","last_peered":"2024-04-29T14:25:21.272129+0000","last_clean":"2024-04-29T14:25:21.272129+0000","last_became_active":"2024-04-29T14:24:30.236756+0000","last_became_peered":"2024-04-29T14:24:30.236756+0000","last_unstale":"2024-04-29T14:25:21.272129+0000","last_undegraded":"2024-04-29T14:25:21.272129+0000","last_fullsized":"2024-04-29T14:25:21.272129+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:41.678068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389677056,"num_objects":5946,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":380544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"452'5747","reported_seq":5796,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:20.283367+0000","last_change":"2024-04-29T14:24:30.769658+0000","last_active":"2024-04-29T14:25:20.283367+0000","last_peered":"2024-04-29T14:25:20.283367+0000","last_clean":"2024-04-29T14:25:20.283367+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:25:20.283367+0000","last_undegraded":"2024-04-29T14:25:20.283367+0000","last_fullsized":"2024-04-29T14:25:20.283367+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:30.774248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376635392,"num_objects":5747,"num_object_clones":0,"num_object_copies":11494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5747,"num_write_kb":367808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"452'5791","reported_seq":5839,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.245119+0000","last_change":"2024-04-29T14:24:30.235557+0000","last_active":"2024-04-29T14:25:21.245119+0000","last_peered":"2024-04-29T14:25:21.245119+0000","last_clean":"2024-04-29T14:25:21.245119+0000","last_became_active":"2024-04-29T14:24:30.235200+0000","last_became_peered":"2024-04-29T14:24:30.235200+0000","last_unstale":"2024-04-29T14:25:21.245119+0000","last_undegraded":"2024-04-29T14:25:21.245119+0000","last_fullsized":"2024-04-29T14:25:21.245119+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:33.771684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379518976,"num_objects":5791,"num_object_clones":0,"num_object_copies":11582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5791,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"452'5798","reported_seq":5847,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.042909+0000","last_change":"2024-04-29T14:24:30.769384+0000","last_active":"2024-04-29T14:25:21.042909+0000","last_peered":"2024-04-29T14:25:21.042909+0000","last_clean":"2024-04-29T14:25:21.042909+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:25:21.042909+0000","last_undegraded":"2024-04-29T14:25:21.042909+0000","last_fullsized":"2024-04-29T14:25:21.042909+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:59.201118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379977728,"num_objects":5798,"num_object_clones":0,"num_object_copies":11596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":371072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"452'5740","reported_seq":5789,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:20.303554+0000","last_change":"2024-04-29T14:24:30.769484+0000","last_active":"2024-04-29T14:25:20.303554+0000","last_peered":"2024-04-29T14:25:20.303554+0000","last_clean":"2024-04-29T14:25:20.303554+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:25:20.303554+0000","last_undegraded":"2024-04-29T14:25:20.303554+0000","last_fullsized":"2024-04-29T14:25:20.303554+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5740,"log_dups_size":0,"ondisk_log_size":5740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:36.313674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376176640,"num_objects":5740,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":367360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"453'5869","reported_seq":5920,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-04-29T14:25:22.799098+0000","last_change":"2024-04-29T14:24:30.236183+0000","last_active":"2024-04-29T14:25:22.799098+0000","last_peered":"2024-04-29T14:25:22.799098+0000","last_clean":"2024-04-29T14:25:22.799098+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:25:22.799098+0000","last_undegraded":"2024-04-29T14:25:22.799098+0000","last_fullsized":"2024-04-29T14:25:22.799098+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:44.730993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384630784,"num_objects":5869,"num_object_clones":0,"num_object_copies":11738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":375616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"452'5724","reported_seq":5773,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.092716+0000","last_change":"2024-04-29T14:24:30.235709+0000","last_active":"2024-04-29T14:25:21.092716+0000","last_peered":"2024-04-29T14:25:21.092716+0000","last_clean":"2024-04-29T14:25:21.092716+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:25:21.092716+0000","last_undegraded":"2024-04-29T14:25:21.092716+0000","last_fullsized":"2024-04-29T14:25:21.092716+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:39.178651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375128064,"num_objects":5724,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5724,"num_write_kb":366336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"452'5738","reported_seq":5787,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:20.739218+0000","last_change":"2024-04-29T14:24:30.769412+0000","last_active":"2024-04-29T14:25:20.739218+0000","last_peered":"2024-04-29T14:25:20.739218+0000","last_clean":"2024-04-29T14:25:20.739218+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:25:20.739218+0000","last_undegraded":"2024-04-29T14:25:20.739218+0000","last_fullsized":"2024-04-29T14:25:20.739218+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:30.928704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376045568,"num_objects":5738,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":367232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"452'5819","reported_seq":5867,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.106208+0000","last_change":"2024-04-29T14:24:30.233714+0000","last_active":"2024-04-29T14:25:21.106208+0000","last_peered":"2024-04-29T14:25:21.106208+0000","last_clean":"2024-04-29T14:25:21.106208+0000","last_became_active":"2024-04-29T14:24:30.233285+0000","last_became_peered":"2024-04-29T14:24:30.233285+0000","last_unstale":"2024-04-29T14:25:21.106208+0000","last_undegraded":"2024-04-29T14:25:21.106208+0000","last_fullsized":"2024-04-29T14:25:21.106208+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:34.461794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381353984,"num_objects":5819,"num_object_clones":0,"num_object_copies":11638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":372416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"452'5751","reported_seq":5797,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.054819+0000","last_change":"2024-04-29T14:24:30.233856+0000","last_active":"2024-04-29T14:25:21.054819+0000","last_peered":"2024-04-29T14:25:21.054819+0000","last_clean":"2024-04-29T14:25:21.054819+0000","last_became_active":"2024-04-29T14:24:30.233553+0000","last_became_peered":"2024-04-29T14:24:30.233553+0000","last_unstale":"2024-04-29T14:25:21.054819+0000","last_undegraded":"2024-04-29T14:25:21.054819+0000","last_fullsized":"2024-04-29T14:25:21.054819+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:43.588762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376897536,"num_objects":5751,"num_object_clones":0,"num_object_copies":11502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5751,"num_write_kb":368064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"452'5782","reported_seq":5831,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.096194+0000","last_change":"2024-04-29T14:24:30.769371+0000","last_active":"2024-04-29T14:25:21.096194+0000","last_peered":"2024-04-29T14:25:21.096194+0000","last_clean":"2024-04-29T14:25:21.096194+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:25:21.096194+0000","last_undegraded":"2024-04-29T14:25:21.096194+0000","last_fullsized":"2024-04-29T14:25:21.096194+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5782,"log_dups_size":0,"ondisk_log_size":5782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:42.330273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378929152,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5782,"num_write_kb":370048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"452'5703","reported_seq":5751,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:20.252423+0000","last_change":"2024-04-29T14:24:30.769402+0000","last_active":"2024-04-29T14:25:20.252423+0000","last_peered":"2024-04-29T14:25:20.252423+0000","last_clean":"2024-04-29T14:25:20.252423+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:25:20.252423+0000","last_undegraded":"2024-04-29T14:25:20.252423+0000","last_fullsized":"2024-04-29T14:25:20.252423+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:46.142650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373751808,"num_objects":5703,"num_object_clones":0,"num_object_copies":11406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":364992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"452'5627","reported_seq":5675,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:20.277922+0000","last_change":"2024-04-29T14:24:30.769398+0000","last_active":"2024-04-29T14:25:20.277922+0000","last_peered":"2024-04-29T14:25:20.277922+0000","last_clean":"2024-04-29T14:25:20.277922+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:25:20.277922+0000","last_undegraded":"2024-04-29T14:25:20.277922+0000","last_fullsized":"2024-04-29T14:25:20.277922+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:48.469572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368771072,"num_objects":5627,"num_object_clones":0,"num_object_copies":11254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":360128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"452'5619","reported_seq":5668,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:21.101859+0000","last_change":"2024-04-29T14:24:30.237079+0000","last_active":"2024-04-29T14:25:21.101859+0000","last_peered":"2024-04-29T14:25:21.101859+0000","last_clean":"2024-04-29T14:25:21.101859+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:25:21.101859+0000","last_undegraded":"2024-04-29T14:25:21.101859+0000","last_fullsized":"2024-04-29T14:25:21.101859+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5619,"log_dups_size":0,"ondisk_log_size":5619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:50.988056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368246784,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5619,"num_write_kb":359616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"453'5816","reported_seq":5867,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-04-29T14:25:22.776014+0000","last_change":"2024-04-29T14:24:30.236266+0000","last_active":"2024-04-29T14:25:22.776014+0000","last_peered":"2024-04-29T14:25:22.776014+0000","last_clean":"2024-04-29T14:25:22.776014+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:25:22.776014+0000","last_undegraded":"2024-04-29T14:25:22.776014+0000","last_fullsized":"2024-04-29T14:25:22.776014+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5816,"log_dups_size":0,"ondisk_log_size":5816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:38.258787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381157376,"num_objects":5816,"num_object_clones":0,"num_object_copies":11632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5816,"num_write_kb":372224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"453'5866","reported_seq":5917,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-04-29T14:25:22.793233+0000","last_change":"2024-04-29T14:24:30.236750+0000","last_active":"2024-04-29T14:25:22.793233+0000","last_peered":"2024-04-29T14:25:22.793233+0000","last_clean":"2024-04-29T14:25:22.793233+0000","last_became_active":"2024-04-29T14:24:30.236451+0000","last_became_peered":"2024-04-29T14:24:30.236451+0000","last_unstale":"2024-04-29T14:25:22.793233+0000","last_undegraded":"2024-04-29T14:25:22.793233+0000","last_fullsized":"2024-04-29T14:25:22.793233+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":5866,"log_dups_size":0,"ondisk_log_size":5866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:50.113461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384434176,"num_objects":5866,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":375424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1137,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-04-29T14:25:19.058216+0000","last_change":"2024-04-29T14:24:28.576580+0000","last_active":"2024-04-29T14:25:19.058216+0000","last_peered":"2024-04-29T14:25:19.058216+0000","last_clean":"2024-04-29T14:25:19.058216+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:25:19.058216+0000","last_undegraded":"2024-04-29T14:25:19.058216+0000","last_fullsized":"2024-04-29T14:25:19.058216+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:24:28.576475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:24:28.576475+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:30.609381+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6051332096,"num_objects":92336,"num_object_clones":0,"num_object_copies":184672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92336,"num_write_kb":5909504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12084903936,"data_stored":12084903936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92336,"ondisk_log_size":92336,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1222404,"kb_used_data":735864,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92522748,"statfs":{"total":95995035648,"available":94743293952,"internally_reserved":0,"allocated":753524736,"data_stored":753022151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1443784,"kb_used_data":1123836,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92301368,"statfs":{"total":95995035648,"available":94516600832,"internally_reserved":0,"allocated":1150808064,"data_stored":1150302717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645543,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475264,"kb_used_data":1092848,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92269888,"statfs":{"total":95995035648,"available":94484365312,"internally_reserved":0,"allocated":1119076352,"data_stored":1118577339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":391576000},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234224,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1543684,"kb_used_data":1089080,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92201468,"statfs":{"total":95995035648,"available":94414303232,"internally_reserved":0,"allocated":1115217920,"data_stored":1114715355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":465500618},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528565,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1614588,"kb_used_data":1069744,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92130564,"statfs":{"total":95995035648,"available":94341697536,"internally_reserved":0,"allocated":1095417856,"data_stored":1094918843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575118,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738644,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2963256,"kb_used_data":2585008,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":90781896,"statfs":{"total":95995035648,"available":92960661504,"internally_reserved":0,"allocated":2647048192,"data_stored":2646549179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182662,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2842808,"kb_used_data":2207728,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":90902344,"statfs":{"total":95995035648,"available":93084000256,"internally_reserved":0,"allocated":2260713472,"data_stored":2260214459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":388,"seq":1666447310887,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1937596,"kb_used_data":1843760,"kb_used_omap":211,"kb_used_meta":93612,"kb_avail":91807556,"statfs":{"total":95995035648,"available":94010937344,"internally_reserved":0,"allocated":1888010240,"data_stored":1887511227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":216698,"internal_metadata":95859078},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2654011392,"data_stored":2654011392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2263941120,"data_stored":2263941120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1902444544,"data_stored":1902444544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1111490560,"data_stored":1111490560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1120075776,"data_stored":1120075776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1129512960,"data_stored":1129512960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1150418944,"data_stored":1150418944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":753008640,"data_stored":753008640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:25:26.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:25:26.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:26.985 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:27.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:27.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:27.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:27.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:27.244 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:27 smithi121 ceph-mon[44533]: from='client.26116 v1:172.21.15.40:0/1427975662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:25:27.244 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:27 smithi121 ceph-mon[44533]: pgmap v1193: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:25:27.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:27.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 101326 101310 140.697 93.5625 0.00884885 0.00710599 2024-04-29T14:25:27.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:27.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:27 smithi040 ceph-mon[39581]: from='client.26116 v1:172.21.15.40:0/1427975662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:25:27.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:27 smithi040 ceph-mon[39581]: pgmap v1193: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:25:27.392 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:27.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:27.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:27.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:27.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:27 smithi194 ceph-mon[44089]: from='client.26116 v1:172.21.15.40:0/1427975662' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:25:27.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:27 smithi194 ceph-mon[44089]: pgmap v1193: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:25:27.591 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:27.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:27.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:27.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:28.117 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:28.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:28.212 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:25:28.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:28.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 102798 102782 139.638 92 0.00844617 0.00715967 2024-04-29T14:25:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:28.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:25:28.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:25:28.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1035950430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:25:28.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:25:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:25:28.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1035950430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:25:28.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:25:28.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:25:28.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1035950430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:25:29.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 104321 104305 138.693 95.1875 0.0120984 0.00720844 2024-04-29T14:25:29.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:29 smithi194 ceph-mon[44089]: pgmap v1194: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-29T14:25:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:29 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1035950430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:25:29.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:29 smithi194 ceph-mon[44089]: osdmap e454: 12 total, 12 up, 8 in 2024-04-29T14:25:29.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:29 smithi040 ceph-mon[39581]: pgmap v1194: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-29T14:25:29.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:29 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:29.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1035950430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:25:29.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:29 smithi040 ceph-mon[39581]: osdmap e454: 12 total, 12 up, 8 in 2024-04-29T14:25:29.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:29 smithi121 ceph-mon[44533]: pgmap v1194: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-29T14:25:29.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:29 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:29.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1035950430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:25:29.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:29 smithi121 ceph-mon[44533]: osdmap e454: 12 total, 12 up, 8 in 2024-04-29T14:25:30.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 16 105846 105830 137.789 95.3125 0.0177742 0.00725592 2024-04-29T14:25:30.307 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:30.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:30.343 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:25:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:25:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:105948 5.2 e454/432 5:4f1fc0c9:::benchmark_data_smithi194_121512_object105947:head v 454'6532, mlcod=454'6530)", 2024-04-29T14:25:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:25:30.349042+0000", 2024-04-29T14:25:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.001730037, 2024-04-29T14:25:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017508210000000001, 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349042+0000", 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349042+0000", 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349043+0000", 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.469e-06 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349074+0000", 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0941000000000001e-05 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349082+0000", 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8990000000000001e-06 2024-04-29T14:25:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349087+0000", 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8930000000000004e-06 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349144+0000", 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6986e-05 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:30.349168+0000", 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3388e-05 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-04-29T14:25:30.347 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:25:30.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:30.450 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:30.656 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:30.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:30.759 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:30.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:30.818 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:30.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:30.924 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:30.958 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:30.992 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:30.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:31.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:31.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:31.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:31.223 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:31.223 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:31.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 107272 107256 136.795 89.125 0.00837592 0.00730881 2024-04-29T14:25:31.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:31.328 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:31.352 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:31.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:31.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:31.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:31 smithi040 ceph-mon[39581]: pgmap v1196: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:25:31.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:31 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:25:31.490 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:31.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:31 smithi194 ceph-mon[44089]: pgmap v1196: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:25:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:31 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:25:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:31 smithi121 ceph-mon[44533]: pgmap v1196: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:25:31.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:31 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:25:31.641 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:31.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:31.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:25:31.745 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:31.776 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:31.801 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:31.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:31.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:31.965 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:31.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:32.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:32.160 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:32.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:32.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:32.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 108699 108683 135.843 89.1875 0.0125382 0.00735979 2024-04-29T14:25:32.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:32.324 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:32.480 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:32 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:25:32.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:32 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:25:32.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:32 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:25:32.775 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:32.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:32.878 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:32.908 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:32.931 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:32.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:32.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:33.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:33.093 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:33.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:33.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:25:33.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:33.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 110115 110099 134.915 88.5 0.0214675 0.00741072 2024-04-29T14:25:33.311 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:33.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:33.355 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:33 smithi121 ceph-mon[44533]: pgmap v1197: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:25:33.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:33.483 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:33 smithi194 ceph-mon[44089]: pgmap v1197: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:25:33.486 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:33.513 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:33.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:33.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:33 smithi040 ceph-mon[39581]: pgmap v1197: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:25:33.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:33.645 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:33.675 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:33.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:33.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:33.811 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:33.937 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:33.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:34.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:34.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:34.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:34.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:34.234 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:34 smithi040 ceph-mon[39581]: 5.e scrub starts 2024-04-29T14:25:34.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:34.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 111498 111482 133.982 86.4375 0.00738647 0.0074626 2024-04-29T14:25:34.269 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:34.292 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:34.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:34.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:34 smithi121 ceph-mon[44533]: 5.e scrub starts 2024-04-29T14:25:34.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:34.499 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:34.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:34.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:34.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:34 smithi194 ceph-mon[44089]: 5.e scrub starts 2024-04-29T14:25:34.641 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:34.669 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:34.692 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:34.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:34.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:34.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:34.992 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:35.020 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:35.045 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:35.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:35.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:25:35.151 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:25:35.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:35.174 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:25:35.187 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:35.200 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:25:35.210 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:35.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:35.233 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:25:35.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 16 112775 112759 132.96 79.8125 0.0212598 0.00751948 2024-04-29T14:25:35.261 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:25:35.290 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:25:35.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:35 smithi194 ceph-mon[44089]: pgmap v1198: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:25:35.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:35 smithi194 ceph-mon[44089]: 5.e scrub ok 2024-04-29T14:25:35.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3078644051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:35.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:25:35.348 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:35.349 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:25:35.375 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:25:35.401 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:25:35.427 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:25:35.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:25:35.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:25:35.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:25:35.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:25:35.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:25:35.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:25:35.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:25:35.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:25:35.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:25:35.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:25:35.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:25:35.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:25:35.449 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:35 smithi040 ceph-mon[39581]: pgmap v1198: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:25:35.449 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:35 smithi040 ceph-mon[39581]: 5.e scrub ok 2024-04-29T14:25:35.449 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3078644051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:35 smithi121 ceph-mon[44533]: pgmap v1198: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:25:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:35 smithi121 ceph-mon[44533]: 5.e scrub ok 2024-04-29T14:25:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3078644051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:35.700 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:35.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:35.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:35.843 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:36.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:36.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:36.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:36.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 113937 113921 131.843 72.625 0.00643276 0.00758295 2024-04-29T14:25:36.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:36.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:36.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:36.421 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:36.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: 5.d scrub starts 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: 5.d scrub starts 2024-04-29T14:25:36.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: 5.d scrub starts 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:37.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 115331 115315 131.029 87.125 0.0111176 0.00763048 2024-04-29T14:25:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:25:37.533 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:25:37.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:37 smithi040 ceph-mon[39581]: pgmap v1199: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-29T14:25:37.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:37 smithi040 ceph-mon[39581]: 5.d scrub ok 2024-04-29T14:25:37.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:37 smithi194 ceph-mon[44089]: pgmap v1199: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-29T14:25:37.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:37 smithi194 ceph-mon[44089]: 5.d scrub ok 2024-04-29T14:25:37.575 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:37.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:37 smithi121 ceph-mon[44533]: pgmap v1199: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-29T14:25:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:37 smithi121 ceph-mon[44533]: 5.d scrub ok 2024-04-29T14:25:37.685 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:37.713 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:37.737 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:37.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:37.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:37.873 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:37.897 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:37.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:38.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:38.046 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:38.116 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:38.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:38.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:38.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 116684 116668 130.2 84.5625 0.0045119 0.0076791 2024-04-29T14:25:38.261 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:38.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:38 smithi040 ceph-mon[39581]: from='client.26134 v1:172.21.15.40:0/2409468038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:25:38.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:38 smithi040 ceph-mon[39581]: pgmap v1200: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:25:38.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:38 smithi121 ceph-mon[44533]: from='client.26134 v1:172.21.15.40:0/2409468038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:25:38.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:38 smithi121 ceph-mon[44533]: pgmap v1200: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:25:38.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:38 smithi194 ceph-mon[44089]: from='client.26134 v1:172.21.15.40:0/2409468038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:25:38.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:38 smithi194 ceph-mon[44089]: pgmap v1200: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:25:38.797 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1199,"stamp":"2024-04-29T14:25:35.956713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7170163232,"num_objects":109401,"num_object_clones":0,"num_object_copies":218802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109401,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":109580,"num_write_kb":7004474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109536,"ondisk_log_size":109536,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17178600,"kb_used_data":13845424,"kb_used_omap":222,"kb_used_meta":3332705,"kb_avail":732782616,"statfs":{"total":767960285184,"available":750369398784,"internally_reserved":0,"allocated":14177714176,"data_stored":14174356968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":227868,"internal_metadata":3412690404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,5,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":970391552,"num_objects":14807,"num_object_clones":0,"num_object_copies":29614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14807,"num_write_kb":947648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001794"},"pg_stats":[{"pgid":"5.f","version":"454'6916","reported_seq":6968,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.239327+0000","last_change":"2024-04-29T14:24:30.237056+0000","last_active":"2024-04-29T14:25:31.239327+0000","last_peered":"2024-04-29T14:25:31.239327+0000","last_clean":"2024-04-29T14:25:31.239327+0000","last_became_active":"2024-04-29T14:24:30.236756+0000","last_became_peered":"2024-04-29T14:24:30.236756+0000","last_unstale":"2024-04-29T14:25:31.239327+0000","last_undegraded":"2024-04-29T14:25:31.239327+0000","last_fullsized":"2024-04-29T14:25:31.239327+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:41.678068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453246976,"num_objects":6916,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"454'7132","reported_seq":7263,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:35.293259+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:25:35.293259+0000","last_peered":"2024-04-29T14:25:35.293259+0000","last_clean":"2024-04-29T14:25:35.293259+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:25:35.293259+0000","last_undegraded":"2024-04-29T14:25:35.293259+0000","last_fullsized":"2024-04-29T14:25:35.293259+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:26:51.238655+0000","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467402752,"num_objects":7132,"num_object_clones":0,"num_object_copies":14264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7132,"num_write_kb":456448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"454'6731","reported_seq":6783,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.211362+0000","last_change":"2024-04-29T14:24:30.235557+0000","last_active":"2024-04-29T14:25:31.211362+0000","last_peered":"2024-04-29T14:25:31.211362+0000","last_clean":"2024-04-29T14:25:31.211362+0000","last_became_active":"2024-04-29T14:24:30.235200+0000","last_became_peered":"2024-04-29T14:24:30.235200+0000","last_unstale":"2024-04-29T14:25:31.211362+0000","last_undegraded":"2024-04-29T14:25:31.211362+0000","last_fullsized":"2024-04-29T14:25:31.211362+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6731,"log_dups_size":0,"ondisk_log_size":6731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:33.771684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441122816,"num_objects":6731,"num_object_clones":0,"num_object_copies":13462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6731,"num_write_kb":430784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11],"acting":[5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"454'6716","reported_seq":6769,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.204139+0000","last_change":"2024-04-29T14:24:30.769384+0000","last_active":"2024-04-29T14:25:31.204139+0000","last_peered":"2024-04-29T14:25:31.204139+0000","last_clean":"2024-04-29T14:25:31.204139+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:25:31.204139+0000","last_undegraded":"2024-04-29T14:25:31.204139+0000","last_fullsized":"2024-04-29T14:25:31.204139+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:59.201118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":429824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"454'7063","reported_seq":7116,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:35.264461+0000","last_change":"2024-04-29T14:24:30.769484+0000","last_active":"2024-04-29T14:25:35.264461+0000","last_peered":"2024-04-29T14:25:35.264461+0000","last_clean":"2024-04-29T14:25:35.264461+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:25:35.264461+0000","last_undegraded":"2024-04-29T14:25:35.264461+0000","last_fullsized":"2024-04-29T14:25:35.264461+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":7063,"log_dups_size":0,"ondisk_log_size":7063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:36.313674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462880768,"num_objects":7063,"num_object_clones":0,"num_object_copies":14126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7063,"num_write_kb":452032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"454'6757","reported_seq":6809,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:32.796571+0000","last_change":"2024-04-29T14:24:30.236183+0000","last_active":"2024-04-29T14:25:32.796571+0000","last_peered":"2024-04-29T14:25:32.796571+0000","last_clean":"2024-04-29T14:25:32.796571+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:25:32.796571+0000","last_undegraded":"2024-04-29T14:25:32.796571+0000","last_fullsized":"2024-04-29T14:25:32.796571+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6757,"log_dups_size":0,"ondisk_log_size":6757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:44.730993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442826752,"num_objects":6757,"num_object_clones":0,"num_object_copies":13514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6757,"num_write_kb":432448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"454'6629","reported_seq":6681,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.209110+0000","last_change":"2024-04-29T14:24:30.235709+0000","last_active":"2024-04-29T14:25:31.209110+0000","last_peered":"2024-04-29T14:25:31.209110+0000","last_clean":"2024-04-29T14:25:31.209110+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:25:31.209110+0000","last_undegraded":"2024-04-29T14:25:31.209110+0000","last_fullsized":"2024-04-29T14:25:31.209110+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6629,"log_dups_size":0,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:39.178651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":13258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":424256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"454'7091","reported_seq":7144,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:35.746302+0000","last_change":"2024-04-29T14:24:30.769412+0000","last_active":"2024-04-29T14:25:35.746302+0000","last_peered":"2024-04-29T14:25:35.746302+0000","last_clean":"2024-04-29T14:25:35.746302+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:25:35.746302+0000","last_undegraded":"2024-04-29T14:25:35.746302+0000","last_fullsized":"2024-04-29T14:25:35.746302+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":7091,"log_dups_size":0,"ondisk_log_size":7091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464715776,"num_objects":7091,"num_object_clones":0,"num_object_copies":14182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7091,"num_write_kb":453824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"454'6752","reported_seq":6804,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.256685+0000","last_change":"2024-04-29T14:24:30.233714+0000","last_active":"2024-04-29T14:25:31.256685+0000","last_peered":"2024-04-29T14:25:31.256685+0000","last_clean":"2024-04-29T14:25:31.256685+0000","last_became_active":"2024-04-29T14:24:30.233285+0000","last_became_peered":"2024-04-29T14:24:30.233285+0000","last_unstale":"2024-04-29T14:25:31.256685+0000","last_undegraded":"2024-04-29T14:25:31.256685+0000","last_fullsized":"2024-04-29T14:25:31.256685+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6752,"log_dups_size":0,"ondisk_log_size":6752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:34.461794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442499072,"num_objects":6752,"num_object_clones":0,"num_object_copies":13504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":432128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"454'6663","reported_seq":6715,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.235713+0000","last_change":"2024-04-29T14:24:30.233856+0000","last_active":"2024-04-29T14:25:31.235713+0000","last_peered":"2024-04-29T14:25:31.235713+0000","last_clean":"2024-04-29T14:25:31.235713+0000","last_became_active":"2024-04-29T14:24:30.233553+0000","last_became_peered":"2024-04-29T14:24:30.233553+0000","last_unstale":"2024-04-29T14:25:31.235713+0000","last_undegraded":"2024-04-29T14:25:31.235713+0000","last_fullsized":"2024-04-29T14:25:31.235713+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:43.588762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436666368,"num_objects":6663,"num_object_clones":0,"num_object_copies":13326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6663,"num_write_kb":426432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"454'6702","reported_seq":6755,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.196224+0000","last_change":"2024-04-29T14:24:30.769371+0000","last_active":"2024-04-29T14:25:31.196224+0000","last_peered":"2024-04-29T14:25:31.196224+0000","last_clean":"2024-04-29T14:25:31.196224+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:25:31.196224+0000","last_undegraded":"2024-04-29T14:25:31.196224+0000","last_fullsized":"2024-04-29T14:25:31.196224+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6702,"log_dups_size":0,"ondisk_log_size":6702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:42.330273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439222272,"num_objects":6702,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6702,"num_write_kb":428928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"454'7052","reported_seq":7105,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:35.230565+0000","last_change":"2024-04-29T14:24:30.769402+0000","last_active":"2024-04-29T14:25:35.230565+0000","last_peered":"2024-04-29T14:25:35.230565+0000","last_clean":"2024-04-29T14:25:35.230565+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:25:35.230565+0000","last_undegraded":"2024-04-29T14:25:35.230565+0000","last_fullsized":"2024-04-29T14:25:35.230565+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:46.142650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462159872,"num_objects":7052,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7052,"num_write_kb":451328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"454'6956","reported_seq":7009,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:35.271021+0000","last_change":"2024-04-29T14:24:30.769398+0000","last_active":"2024-04-29T14:25:35.271021+0000","last_peered":"2024-04-29T14:25:35.271021+0000","last_clean":"2024-04-29T14:25:35.271021+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:25:35.271021+0000","last_undegraded":"2024-04-29T14:25:35.271021+0000","last_fullsized":"2024-04-29T14:25:35.271021+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:48.469572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455868416,"num_objects":6956,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":445184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"454'6557","reported_seq":6610,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:31.184419+0000","last_change":"2024-04-29T14:24:30.237079+0000","last_active":"2024-04-29T14:25:31.184419+0000","last_peered":"2024-04-29T14:25:31.184419+0000","last_clean":"2024-04-29T14:25:31.184419+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:25:31.184419+0000","last_undegraded":"2024-04-29T14:25:31.184419+0000","last_fullsized":"2024-04-29T14:25:31.184419+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6557,"log_dups_size":0,"ondisk_log_size":6557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:50.988056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429719552,"num_objects":6557,"num_object_clones":0,"num_object_copies":13114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6557,"num_write_kb":419648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"454'6837","reported_seq":6887,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:32.823924+0000","last_change":"2024-04-29T14:24:30.236266+0000","last_active":"2024-04-29T14:25:32.823924+0000","last_peered":"2024-04-29T14:25:32.823924+0000","last_clean":"2024-04-29T14:25:32.823924+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:25:32.823924+0000","last_undegraded":"2024-04-29T14:25:32.823924+0000","last_fullsized":"2024-04-29T14:25:32.823924+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6837,"log_dups_size":0,"ondisk_log_size":6837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:38.258787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448069632,"num_objects":6837,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6837,"num_write_kb":437568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"454'6845","reported_seq":6898,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:32.788263+0000","last_change":"2024-04-29T14:24:30.236750+0000","last_active":"2024-04-29T14:25:32.788263+0000","last_peered":"2024-04-29T14:25:32.788263+0000","last_clean":"2024-04-29T14:25:32.788263+0000","last_became_active":"2024-04-29T14:24:30.236451+0000","last_became_peered":"2024-04-29T14:24:30.236451+0000","last_unstale":"2024-04-29T14:25:32.788263+0000","last_undegraded":"2024-04-29T14:25:32.788263+0000","last_fullsized":"2024-04-29T14:25:32.788263+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:25:50.113461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448593920,"num_objects":6845,"num_object_clones":0,"num_object_copies":13690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6845,"num_write_kb":438080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1149,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-04-29T14:25:30.927963+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:25:30.927963+0000","last_peered":"2024-04-29T14:25:30.927963+0000","last_clean":"2024-04-29T14:25:30.927963+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:25:30.927963+0000","last_undegraded":"2024-04-29T14:25:30.927963+0000","last_fullsized":"2024-04-29T14:25:30.927963+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:26:34.154198+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":7169572864,"num_objects":109399,"num_object_clones":0,"num_object_copies":218798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109399,"num_write_kb":7001536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14259847168,"data_stored":14259847168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109399,"ondisk_log_size":109399,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122272,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382820,"kb_used_data":900444,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92362332,"statfs":{"total":95995035648,"available":94579027968,"internally_reserved":0,"allocated":922054656,"data_stored":921632326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621276,"kb_used_data":1293012,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92123876,"statfs":{"total":95995035648,"available":94334849024,"internally_reserved":0,"allocated":1324044288,"data_stored":1323625510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1754652,"kb_used_data":1354196,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91990500,"statfs":{"total":95995035648,"available":94198272000,"internally_reserved":0,"allocated":1386696704,"data_stored":1386277946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234226,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1711652,"kb_used_data":1265372,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92033500,"statfs":{"total":95995035648,"available":94242304000,"internally_reserved":0,"allocated":1295740928,"data_stored":1295318618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":456980938},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528568,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1896156,"kb_used_data":1333268,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":91848996,"statfs":{"total":95995035648,"available":94053371904,"internally_reserved":0,"allocated":1365266432,"data_stored":1364847674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":576387526},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575120,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738646,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3359516,"kb_used_data":2989588,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":90385636,"statfs":{"total":95995035648,"available":92554891264,"internally_reserved":0,"allocated":3061338112,"data_stored":3060919354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986434,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3197148,"kb_used_data":2562068,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":90548004,"statfs":{"total":95995035648,"available":92721156096,"internally_reserved":0,"allocated":2623557632,"data_stored":2623138874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":388,"seq":1666447310889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2255380,"kb_used_data":2147476,"kb_used_omap":211,"kb_used_meta":107500,"kb_avail":91489772,"statfs":{"total":95995035648,"available":93685526528,"internally_reserved":0,"allocated":2199015424,"data_stored":2198596666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":216698,"internal_metadata":110080390},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3071410176,"data_stored":3071410176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2634350592,"data_stored":2634350592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2213806080,"data_stored":2213806080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1376518144,"data_stored":1376518144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1300692992,"data_stored":1300692992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1394933760,"data_stored":1394933760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1339490304,"data_stored":1339490304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":928645120,"data_stored":928645120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:25:38.799 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:25:38.814 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:38.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:38.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:25:38.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:38.985 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:39.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:39.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:39.133 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:39.167 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:39.241 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:39.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:39.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 16 117955 117939 129.309 79.4375 0.0164559 0.0077316 2024-04-29T14:25:39.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:39.378 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:39.404 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:25:39.404 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:25:40.001 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:40.236 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:40.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:40.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 119200 119184 128.421 77.8125 0.0226281 0.00778488 2024-04-29T14:25:40.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:40.375 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:40.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:40.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:40.516 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:40 smithi040 ceph-mon[39581]: 5.7 scrub starts 2024-04-29T14:25:40.516 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:25:40.516 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:25:40.516 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:40 smithi040 ceph-mon[39581]: 5.8 scrub starts 2024-04-29T14:25:40.516 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:40 smithi040 ceph-mon[39581]: osdmap e455: 12 total, 12 up, 8 in 2024-04-29T14:25:40.516 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:40 smithi040 ceph-mon[39581]: pgmap v1202: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:25:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:40 smithi194 ceph-mon[44089]: 5.7 scrub starts 2024-04-29T14:25:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:25:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:25:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:40 smithi194 ceph-mon[44089]: 5.8 scrub starts 2024-04-29T14:25:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:40 smithi194 ceph-mon[44089]: osdmap e455: 12 total, 12 up, 8 in 2024-04-29T14:25:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:40 smithi194 ceph-mon[44089]: pgmap v1202: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:25:40.537 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:40.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:40 smithi121 ceph-mon[44533]: 5.7 scrub starts 2024-04-29T14:25:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:25:40.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:25:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:40 smithi121 ceph-mon[44533]: 5.8 scrub starts 2024-04-29T14:25:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:40 smithi121 ceph-mon[44533]: osdmap e455: 12 total, 12 up, 8 in 2024-04-29T14:25:40.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:40 smithi121 ceph-mon[44533]: pgmap v1202: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:25:40.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:40.697 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:41.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 120528 120512 127.651 83 0.0112619 0.00783205 2024-04-29T14:25:41.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:41 smithi194 ceph-mon[44089]: 5.8 scrub ok 2024-04-29T14:25:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:41 smithi194 ceph-mon[44089]: 5.7 scrub ok 2024-04-29T14:25:41.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:41 smithi040 ceph-mon[39581]: 5.8 scrub ok 2024-04-29T14:25:41.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:41 smithi040 ceph-mon[39581]: 5.7 scrub ok 2024-04-29T14:25:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:41 smithi121 ceph-mon[44533]: 5.8 scrub ok 2024-04-29T14:25:41.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:41 smithi121 ceph-mon[44533]: 5.7 scrub ok 2024-04-29T14:25:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:42.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:25:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:42.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:42.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:42.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:25:42.262800 min lat: 0.00099482 max lat: 0.218319 avg lat: 0.00787765 2024-04-29T14:25:42.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:25:42.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 121869 121853 126.92 83.8125 0.00790432 0.00787765 2024-04-29T14:25:42.288 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:25:42.402 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3940684892' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:42.402 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:42 smithi040 ceph-mon[39581]: pgmap v1203: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-29T14:25:42.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3940684892' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:42.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:42 smithi194 ceph-mon[44089]: pgmap v1203: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-29T14:25:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3940684892' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:42 smithi121 ceph-mon[44533]: pgmap v1203: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-29T14:25:42.821 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:42.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:42.852 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:25:42.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:25:42.899 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:25:42.924 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:25:42.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:42.949 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:25:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:42.979 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:25:43.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:25:43.022 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:43.023 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:25:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:43.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:43.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:25:43.071 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:25:43.095 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:25:43.118 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:25:43.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:25:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:25:43.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:43.181 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:43.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:43.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:43.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 123007 122991 126.005 71.125 0.0127622 0.00793373 2024-04-29T14:25:43.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:43.408 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:43 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:25:43.408 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:43 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:25:43.408 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3940684892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:43.408 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:43 smithi040 ceph-mon[39581]: osdmap e456: 12 total, 12 up, 8 in 2024-04-29T14:25:43.408 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:43 smithi040 ceph-mon[39581]: 5.f scrub starts 2024-04-29T14:25:43.409 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3040521052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:43.409 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:43.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:43 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:25:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:43 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:25:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3940684892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:43 smithi194 ceph-mon[44089]: osdmap e456: 12 total, 12 up, 8 in 2024-04-29T14:25:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:43 smithi194 ceph-mon[44089]: 5.f scrub starts 2024-04-29T14:25:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3040521052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:43.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:43.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:43.620 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:43.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:43 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:25:43.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:43 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:25:43.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3940684892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:25:43.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:43 smithi121 ceph-mon[44533]: osdmap e456: 12 total, 12 up, 8 in 2024-04-29T14:25:43.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:43 smithi121 ceph-mon[44533]: 5.f scrub starts 2024-04-29T14:25:43.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3040521052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:43.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:43.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:43.775 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:43.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:43.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:43.922 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:43.944 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:43.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:44.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:44.085 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:44.111 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:44.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:44.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:44.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 16 124018 124002 124.992 63.1875 0.0285039 0.00799815 2024-04-29T14:25:44.271 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:44.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:44.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:44.417 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:44.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: 5.0 deep-scrub starts 2024-04-29T14:25:44.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:44 smithi040 ceph-mon[39581]: pgmap v1205: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-29T14:25:44.449 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:44.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: 5.0 deep-scrub starts 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:44 smithi194 ceph-mon[44089]: pgmap v1205: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-29T14:25:44.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:44.608 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: 5.0 deep-scrub starts 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:44.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:44 smithi121 ceph-mon[44533]: pgmap v1205: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-29T14:25:44.685 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:44.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:44.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:44.830 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:44.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:44.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:44.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:44.996 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:45.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 125071 125055 124.053 65.8125 0.0108491 0.00805905 2024-04-29T14:25:45.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:25:45.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:45 smithi040 ceph-mon[39581]: 5.b scrub starts 2024-04-29T14:25:45.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:45 smithi040 ceph-mon[39581]: 5.f scrub ok 2024-04-29T14:25:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:45 smithi194 ceph-mon[44089]: 5.b scrub starts 2024-04-29T14:25:45.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:45 smithi194 ceph-mon[44089]: 5.f scrub ok 2024-04-29T14:25:45.626 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:45.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:45.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:45.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:45 smithi121 ceph-mon[44533]: 5.b scrub starts 2024-04-29T14:25:45.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:45 smithi121 ceph-mon[44533]: 5.f scrub ok 2024-04-29T14:25:45.733 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:45.766 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:45.787 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:45.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:45.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:45.921 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:45.943 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:45.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:46.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:46.085 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:46.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:25:46.221 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:25:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:25:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:25:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:25:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:167486 4.b 4:d9d0eddb:::benchmark_data_smithi194_103039_object10803:head [delete] snapc 0=[] ondisk+write+known_if_redirected e361)", 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:38.236864+0000", 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 307.98929955, 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030726046999999999, 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 167486 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236864+0000", 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236864+0000", 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236869+0000", 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.459e-06 2024-04-29T14:25:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236883+0000", 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4581e-05 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236911+0000", 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7821999999999999e-05 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.236926+0000", 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4476e-05 2024-04-29T14:25:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.237013+0000", 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6828000000000003e-05 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.237180+0000", 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016765500000000001 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.237331+0000", 2024-04-29T14:25:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015088300000000001 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.241733+0000", 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044023939999999996 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.267517+0000", 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025783897 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.267556+0000", 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8210999999999997e-05 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:38.267590+0000", 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4841000000000001e-05 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:170115 4.b 4:d8cb76b9:::benchmark_data_smithi194_103039_object13432:head [delete] snapc 0=[] ondisk+write+known_if_redirected e361)", 2024-04-29T14:25:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:40.707663+0000", 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 305.51850063099999, 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030995383000000001, 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 170115 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707663+0000", 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707663+0000", 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707664+0000", 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.08e-06 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707669+0000", 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7840000000000003e-06 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707677+0000", 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2099999999999993e-06 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.707685+0000", 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5399999999999998e-06 2024-04-29T14:25:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.715617+0000", 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0079318849999999996 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.715721+0000", 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104033 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.715815+0000", 2024-04-29T14:25:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4254999999999998e-05 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.726361+0000", 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010545974 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.738596+0000", 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012235108999999999 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.738629+0000", 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2626999999999998e-05 2024-04-29T14:25:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:40.738659+0000", 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9886000000000002e-05 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181319 4.b 4:daa64077:::benchmark_data_smithi194_103039_object24636:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.530760+0000", 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 294.69540437500001, 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030546224, 2024-04-29T14:25:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181319 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530760+0000", 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530760+0000", 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530761+0000", 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7370000000000001e-06 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530766+0000", 2024-04-29T14:25:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6820000000000002e-06 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530776+0000", 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0054000000000001e-05 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530782+0000", 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2500000000000003e-06 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530822+0000", 2024-04-29T14:25:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9823000000000001e-05 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530903+0000", 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0618999999999995e-05 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530973+0000", 2024-04-29T14:25:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0166000000000005e-05 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540450+0000", 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094768140000000001 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561164+0000", 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020713850999999998 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561249+0000", 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5235999999999995e-05 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561306+0000", 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6991999999999999e-05 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181322 4.b 4:d4feca94:::benchmark_data_smithi194_103039_object24639:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 294.69537124200002, 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030675788999999998, 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181322 2024-04-29T14:25:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530793+0000", 2024-04-29T14:25:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.39e-07 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530795+0000", 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7969999999999999e-06 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530798+0000", 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3529999999999999e-06 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530803+0000", 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8709999999999996e-06 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.533825+0000", 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030218530000000001 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.533896+0000", 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0837e-05 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.533977+0000", 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0865000000000006e-05 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540455+0000", 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.006478765 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561367+0000", 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020911619999999999 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561424+0000", 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7213999999999998e-05 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561468+0000", 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4175000000000003e-05 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181323 4.b 4:d92ad292:::benchmark_data_smithi194_103039_object24640:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 294.69535600400002, 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030732281, 2024-04-29T14:25:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181323 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:25:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530808+0000", 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0100000000000001e-07 2024-04-29T14:25:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530810+0000", 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.545e-06 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530813+0000", 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8059999999999999e-06 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.530815+0000", 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8870000000000001e-06 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.536080+0000", 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0052646100000000003 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.536144+0000", 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4354000000000001e-05 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.536218+0000", 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3238000000000006e-05 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540460+0000", 2024-04-29T14:25:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042421070000000002 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561494+0000", 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021033759999999999 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561520+0000", 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6631e-05 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.561540+0000", 2024-04-29T14:25:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0041999999999999e-05 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181330 4.b 4:de33ed73:::benchmark_data_smithi194_103039_object24647:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 294.69052448399998, 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039659663999999997, 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181330 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535641+0000", 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.248e-06 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535645+0000", 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4429999999999996e-06 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535655+0000", 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0348e-05 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535662+0000", 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0970000000000002e-06 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.537976+0000", 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023144060000000002 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.538102+0000", 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012558 2024-04-29T14:25:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.538229+0000", 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126928 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540630+0000", 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024012769999999998 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575087+0000", 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034457152999999997 2024-04-29T14:25:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575253+0000", 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000166441 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575299+0000", 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5742999999999999e-05 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:182056 4.b 4:d06152d6:::benchmark_data_smithi194_103039_object25373:head [delete] snapc 0=[] ondisk+write+known_if_redirected e365)", 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:52.229123+0000", 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 293.99704072200001, 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030576231999999998, 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 182056 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229123+0000", 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229123+0000", 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229124+0000", 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4099999999999997e-07 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229127+0000", 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1480000000000002e-06 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229133+0000", 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2299999999999996e-06 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229137+0000", 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9670000000000003e-06 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.233218+0000", 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0040804880000000002 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.233302+0000", 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3850000000000005e-05 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.233374+0000", 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2560999999999999e-05 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.241628+0000", 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082541520000000007 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259631+0000", 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018003008000000001 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259662+0000", 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0796000000000001e-05 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259699+0000", 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7190999999999997e-05 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:182057 4.b 4:d01586c9:::benchmark_data_smithi194_103039_object25374:head [delete] snapc 0=[] ondisk+write+known_if_redirected e365)", 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 293.99700930300003, 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030692338999999999, 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 182057 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:25:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229155+0000", 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2200000000000004e-07 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229158+0000", 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.249e-06 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229162+0000", 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0090000000000001e-06 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.229165+0000", 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3119999999999998e-06 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.235695+0000", 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065303940000000001 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.235766+0000", 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0659000000000002e-05 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.235842+0000", 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5726e-05 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.241631+0000", 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057892839999999996 2024-04-29T14:25:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259763+0000", 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018132514999999998 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259810+0000", 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6683e-05 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:52.259847+0000", 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6885999999999999e-05 2024-04-29T14:25:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187965 4.b 4:d7e0b0fd:::benchmark_data_smithi194_103039_object31282:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 288.404806823, 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045912334999999999, 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187965 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821359+0000", 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5039999999999999e-06 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821363+0000", 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4939999999999997e-06 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821373+0000", 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0001e-05 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821380+0000", 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9589999999999998e-06 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821426+0000", 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5896999999999998e-05 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821500+0000", 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3790999999999999e-05 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821568+0000", 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8112000000000001e-05 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.823920+0000", 2024-04-29T14:25:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023521409999999999 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867185+0000", 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043265190000000002 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867231+0000", 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5726999999999998e-05 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867269+0000", 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8519000000000002e-05 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187971 4.b 4:d0a42069:::benchmark_data_smithi194_103039_object31288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 288.394675721, 2024-04-29T14:25:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035888784999999999, 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187971 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831492+0000", 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3139999999999999e-06 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831502+0000", 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0713000000000001e-05 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831529+0000", 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6893000000000002e-05 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831548+0000", 2024-04-29T14:25:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8697000000000001e-05 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831712+0000", 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016401100000000001 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831816+0000", 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104293 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831888+0000", 2024-04-29T14:25:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1792000000000007e-05 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836180+0000", 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004292293 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867316+0000", 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031135787000000002 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867351+0000", 2024-04-29T14:25:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4573999999999998e-05 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867377+0000", 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6418e-05 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187972 4.b 4:d50db589:::benchmark_data_smithi194_103039_object31289:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 288.39458980799998, 2024-04-29T14:25:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035895613999999999, 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187972 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831575+0000", 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.24e-06 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831580+0000", 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1359999999999999e-06 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831589+0000", 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5740000000000002e-06 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831602+0000", 2024-04-29T14:25:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2585999999999999e-05 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834234+0000", 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002631839 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834303+0000", 2024-04-29T14:25:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9339999999999997e-05 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834370+0000", 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.703e-05 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.838064+0000", 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036939109999999998 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867419+0000", 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029355322999999999 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867447+0000", 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8164999999999999e-05 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867470+0000", 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.247e-05 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187977 4.b 4:d9b4303c:::benchmark_data_smithi194_103039_object31294:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.834981+0000", 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 288.39118259999998, 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032568039, 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187977 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834981+0000", 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834981+0000", 2024-04-29T14:25:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834983+0000", 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3999999999999999e-06 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834987+0000", 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5109999999999997e-06 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834998+0000", 2024-04-29T14:25:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0461e-05 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.835010+0000", 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2283000000000001e-05 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836200+0000", 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001189697 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836266+0000", 2024-04-29T14:25:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6630000000000004e-05 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836331+0000", 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4391000000000001e-05 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.838220+0000", 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018889740000000001 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867501+0000", 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029281411 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867529+0000", 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7426000000000001e-05 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867549+0000", 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0854999999999999e-05 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:195494 4.b 4:d756e1a7:::benchmark_data_smithi194_103039_object38811:head [delete] snapc 0=[] ondisk+write+known_if_redirected e368)", 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 281.44434779099998, 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039141214000000001, 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 195494 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781818+0000", 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.916e-06 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781824+0000", 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5049999999999998e-06 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781834+0000", 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0494e-05 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781842+0000", 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0260000000000007e-06 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781908+0000", 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5474000000000004e-05 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.782012+0000", 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104421 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.782099+0000", 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7496000000000004e-05 2024-04-29T14:25:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.785711+0000", 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036118610000000001 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820846+0000", 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035134336000000002 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820907+0000", 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1050999999999995e-05 2024-04-29T14:25:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820957+0000", 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0633999999999997e-05 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:197308 4.b 4:d0f24d33:::benchmark_data_smithi194_103039_object40625:head [delete] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:21:06.497279+0000", 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 279.72888537400001, 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034682491000000003, 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 197308 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497279+0000", 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497279+0000", 2024-04-29T14:25:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497282+0000", 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.473e-06 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497293+0000", 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0947e-05 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497318+0000", 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5439000000000002e-05 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497333+0000", 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4732999999999999e-05 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497474+0000", 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014126899999999999 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497671+0000", 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019645700000000001 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.497822+0000", 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015072999999999999 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.501363+0000", 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035410039999999999 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.531859+0000", 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030496743 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.531914+0000", 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4864999999999997e-05 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:06.531961+0000", 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6830999999999999e-05 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:197940 4.b 4:d4c58e28:::benchmark_data_smithi194_103039_object41257:head [delete] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 279.15041133900002, 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033304070999999998, 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 197940 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075753+0000", 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9800000000000003e-07 2024-04-29T14:25:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075756+0000", 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1580000000000001e-06 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075766+0000", 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2429999999999994e-06 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.075770+0000", 2024-04-29T14:25:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1189999999999999e-06 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.080827+0000", 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050571269999999998 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.080906+0000", 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8968999999999996e-05 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.080977+0000", 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0797999999999995e-05 2024-04-29T14:25:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.083109+0000", 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021327780000000001 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.108949+0000", 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.02583999 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.109007+0000", 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7181e-05 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:07.109057+0000", 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0009999999999997e-05 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205500 4.b 4:dcf18273:::benchmark_data_smithi194_103039_object48817:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 205.86052967800001, 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038420901, 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205500 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365635+0000", 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8099999999999998e-07 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365637+0000", 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.889e-06 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365640+0000", 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.619e-06 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365644+0000", 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4359999999999998e-06 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372228+0000", 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065845260000000003 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372328+0000", 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100299 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372423+0000", 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4629999999999994e-05 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.383904+0000", 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011481358000000001 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.403987+0000", 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020082472000000001 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404022+0000", 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4737999999999998e-05 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404055+0000", 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3553000000000002e-05 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205501 4.b 4:d60cc47a:::benchmark_data_smithi194_103039_object48818:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 205.86051605399999, 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038509798999999997, 2024-04-29T14:25:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205501 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:25:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9999999999999999e-07 2024-04-29T14:25:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365650+0000", 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.545e-06 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365653+0000", 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.456e-06 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365656+0000", 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9390000000000002e-06 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.375824+0000", 2024-04-29T14:25:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010167756 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.375966+0000", 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014242200000000001 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.376106+0000", 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140068 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.387047+0000", 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01094083 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404096+0000", 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01704864 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404128+0000", 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2452e-05 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404158+0000", 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9391e-05 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205503 4.b 4:d23be51c:::benchmark_data_smithi194_103039_object48820:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 205.85896470200001, 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037041931, 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205503 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367200+0000", 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0580000000000001e-06 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367205+0000", 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6e-06 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367214+0000", 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5249999999999994e-06 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367220+0000", 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6200000000000004e-06 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380152+0000", 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012932018 2024-04-29T14:25:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380291+0000", 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013889600000000001 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380429+0000", 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000138541 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.390770+0000", 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010340066 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404193+0000", 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013423403 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404221+0000", 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7872e-05 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404241+0000", 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0332000000000001e-05 2024-04-29T14:25:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:214667 4.b 4:dce03977:::benchmark_data_smithi194_103039_object57984:head [delete] snapc 0=[] ondisk+write+known_if_redirected e394)", 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:28.624504+0000", 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 197.60166031200001, 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032501269999999999, 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:25:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 214667 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.624504+0000", 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.624504+0000", 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.624504+0000", 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7199999999999999e-07 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.624507+0000", 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3690000000000001e-06 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.624511+0000", 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7659999999999998e-06 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.624515+0000", 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4929999999999998e-06 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.633267+0000", 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0087519969999999992 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.633346+0000", 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9278999999999998e-05 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:25:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.633428+0000", 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2389e-05 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.635599+0000", 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00217022 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.656944+0000", 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021345557000000001 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.656975+0000", 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0514e-05 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:28.657005+0000", 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0114e-05 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:125390 5.5 e456/432 5:a1575df9:::benchmark_data_smithi194_121512_object125389:head v 456'7832, mlcod=456'7830)", 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:25:45.551175+0000", 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.67498907900000005, 2024-04-29T14:25:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014181614, 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551175+0000", 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551175+0000", 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:25:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551176+0000", 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.24e-06 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551238+0000", 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1727e-05 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551249+0000", 2024-04-29T14:25:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1168999999999999e-05 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551256+0000", 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5599999999999999e-06 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551308+0000", 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2383999999999999e-05 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.551320+0000", 2024-04-29T14:25:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2428e-05 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.565312+0000", 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013991271 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:25:45.565356+0000", 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4835e-05 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:25:46.340 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:25:46.341 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 16 126218 126202 123.234 71.6875 0.0126196 0.00811279 2024-04-29T14:25:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:46.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:46.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:46.476 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:46.497 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:46.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:46.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:46.606 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:46 smithi040 ceph-mon[39581]: 5.b scrub ok 2024-04-29T14:25:46.606 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:46 smithi040 ceph-mon[39581]: pgmap v1206: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-29T14:25:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:46 smithi121 ceph-mon[44533]: 5.b scrub ok 2024-04-29T14:25:46.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:46 smithi121 ceph-mon[44533]: pgmap v1206: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-29T14:25:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:46.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:46 smithi194 ceph-mon[44089]: 5.b scrub ok 2024-04-29T14:25:46.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:46 smithi194 ceph-mon[44089]: pgmap v1206: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-29T14:25:46.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:25:47.087 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:47.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:47.195 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:47.225 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:47.250 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:47.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:47.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 127274 127258 122.354 66 0.00887466 0.00817171 2024-04-29T14:25:47.352 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:25:47.379 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:47.400 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:25:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:47.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:25:47.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:47 smithi040 ceph-mon[39581]: 5.5 scrub starts 2024-04-29T14:25:47.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:47 smithi040 ceph-mon[39581]: 5.9 scrub starts 2024-04-29T14:25:47.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:47 smithi194 ceph-mon[44089]: 5.5 scrub starts 2024-04-29T14:25:47.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:47 smithi194 ceph-mon[44089]: 5.9 scrub starts 2024-04-29T14:25:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:47 smithi121 ceph-mon[44533]: 5.5 scrub starts 2024-04-29T14:25:47.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:47 smithi121 ceph-mon[44533]: 5.9 scrub starts 2024-04-29T14:25:47.743 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:47.781 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:48.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 15 128068 128053 121.253 49.6875 0.0069736 0.00824508 2024-04-29T14:25:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:48.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:48.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:48.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:48 smithi040 ceph-mon[39581]: 5.6 scrub starts 2024-04-29T14:25:48.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:48 smithi040 ceph-mon[39581]: pgmap v1207: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-29T14:25:48.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:48 smithi121 ceph-mon[44533]: 5.6 scrub starts 2024-04-29T14:25:48.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:48 smithi121 ceph-mon[44533]: pgmap v1207: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-29T14:25:48.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:48 smithi194 ceph-mon[44089]: 5.6 scrub starts 2024-04-29T14:25:48.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:48 smithi194 ceph-mon[44089]: pgmap v1207: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-29T14:25:49.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 16 128972 128956 120.285 56.4375 0.00482751 0.00831094 2024-04-29T14:25:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:49.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:49.481 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:49.521 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:49.547 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:49.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:49.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:49 smithi040 ceph-mon[39581]: 5.5 scrub ok 2024-04-29T14:25:49.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:49 smithi040 ceph-mon[39581]: 5.9 scrub ok 2024-04-29T14:25:49.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:49 smithi040 ceph-mon[39581]: 5.6 scrub ok 2024-04-29T14:25:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:49 smithi121 ceph-mon[44533]: 5.5 scrub ok 2024-04-29T14:25:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:49 smithi121 ceph-mon[44533]: 5.9 scrub ok 2024-04-29T14:25:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:49 smithi121 ceph-mon[44533]: 5.6 scrub ok 2024-04-29T14:25:49.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:49.681 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:49.703 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:49.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:49.807 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:49.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:49 smithi194 ceph-mon[44089]: 5.5 scrub ok 2024-04-29T14:25:49.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:49 smithi194 ceph-mon[44089]: 5.9 scrub ok 2024-04-29T14:25:49.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:49 smithi194 ceph-mon[44089]: 5.6 scrub ok 2024-04-29T14:25:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:50.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 129932 129916 119.399 60 0.0120505 0.00837218 2024-04-29T14:25:50.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:50 smithi040 ceph-mon[39581]: 5.a deep-scrub starts 2024-04-29T14:25:50.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:50 smithi040 ceph-mon[39581]: pgmap v1208: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:25:50.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:50 smithi121 ceph-mon[44533]: 5.a deep-scrub starts 2024-04-29T14:25:50.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:50 smithi121 ceph-mon[44533]: pgmap v1208: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:25:50.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:50 smithi194 ceph-mon[44089]: 5.a deep-scrub starts 2024-04-29T14:25:50.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:50 smithi194 ceph-mon[44089]: pgmap v1208: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-29T14:25:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:51.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:51.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 130809 130793 118.463 54.8125 0.0178807 0.0084389 2024-04-29T14:25:51.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:51.371 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:25:51.384 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:51.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:51.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:51.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:51.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:51 smithi040 ceph-mon[39581]: 5.3 scrub starts 2024-04-29T14:25:51.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3844907660' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:25:51.561 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:51.585 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:51.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:51.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:51 smithi121 ceph-mon[44533]: 5.3 scrub starts 2024-04-29T14:25:51.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3844907660' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:25:51.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:51.739 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:51.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:51.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:51.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:51 smithi194 ceph-mon[44089]: 5.3 scrub starts 2024-04-29T14:25:51.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3844907660' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:25:51.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:51.939 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:51.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:51.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:52.052 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:25:52.052 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:25:52.052 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:25:52.052 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:25:52.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:52.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:52.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:52.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 131698 131682 117.564 55.5625 0.0112447 0.00850227 2024-04-29T14:25:52.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:52.309 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:52.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:52 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:52.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3844907660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:25:52.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:52 smithi040 ceph-mon[39581]: osdmap e457: 12 total, 12 up, 8 in 2024-04-29T14:25:52.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:52 smithi040 ceph-mon[39581]: pgmap v1210: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:25:52.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:52 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:52.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3844907660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:25:52.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:52 smithi121 ceph-mon[44533]: osdmap e457: 12 total, 12 up, 8 in 2024-04-29T14:25:52.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:52 smithi121 ceph-mon[44533]: pgmap v1210: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:25:52.757 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:25:52.780 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:25:52.807 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:25:52.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:52 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:25:52.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3844907660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:25:52.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:52 smithi194 ceph-mon[44089]: osdmap e457: 12 total, 12 up, 8 in 2024-04-29T14:25:52.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:52 smithi194 ceph-mon[44089]: pgmap v1210: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:25:52.831 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:25:52.859 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:25:52.861 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:52.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:52.893 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:25:52.918 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:25:52.946 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:25:52.968 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:25:52.972 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:52.995 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:25:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:53.023 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:25:53.048 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:25:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:25:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:25:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:25:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:25:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:25:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:25:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:25:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:25:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:25:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:25:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:25:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:25:53.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 16 132656 132640 116.751 59.875 0.0148998 0.00856257 2024-04-29T14:25:53.554 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:53.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3144208266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:53.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:53.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:53.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:53.659 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:53.688 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3144208266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:53.688 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:53.688 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:53.690 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:53.719 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:53.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:53.823 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:25:53.856 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:53.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3144208266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:25:53.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:53.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:53.882 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:25:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:53.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:25:53.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:54.021 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:54.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 133449 133433 115.818 49.5625 0.00430812 0.00861068 2024-04-29T14:25:54.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:25:54.572 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:54.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:54.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:54.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:54 smithi121 ceph-mon[44533]: pgmap v1211: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-29T14:25:54.703 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:54.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:54.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:54 smithi194 ceph-mon[44089]: pgmap v1211: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-29T14:25:54.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:54.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:25:54.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:54 smithi040 ceph-mon[39581]: pgmap v1211: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-29T14:25:54.858 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:55.106 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:55.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:55.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:55.240 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:55.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 133462 133446 114.243 0.8125 0.00385668 0.008611 2024-04-29T14:25:55.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:25:55.312 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:55.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:55.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:55.444 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:55.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:25:56.172 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:56.194 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:56.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:56.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 133462 133446 112.699 0 - 0.008611 2024-04-29T14:25:56.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:25:56.327 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:25:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:56.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:25:56.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:56.477 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:56.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:56.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:56.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:57.011 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:57.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:57.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:57.142 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:57.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:57.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:57.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 16 133462 133446 111.196 0 - 0.008611 2024-04-29T14:25:57.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:25:57.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:57 smithi040 ceph-mon[39581]: pgmap v1212: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s 2024-04-29T14:25:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:57.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:57 smithi194 ceph-mon[44089]: pgmap v1212: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s 2024-04-29T14:25:57.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:57 smithi121 ceph-mon[44533]: pgmap v1212: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s 2024-04-29T14:25:57.481 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:25:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:57.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:25:57.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:57.613 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:57.699 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:25:57.715 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:57.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:57.751 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:25:57.751 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:25:57.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:25:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:58.047 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:25:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:58.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:25:58.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:25:58.179 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:58.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 133462 133446 109.733 0 - 0.008611 2024-04-29T14:25:58.800 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-29T14:25:58.800 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:25:58.800 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-29T14:25:58.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5 2024-04-29T14:25:58.907 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:25:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:58.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:25:59.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:25:59.038 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:59.057 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:25:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:59.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:25:59.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:59 smithi121 ceph-mon[44533]: pgmap v1213: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 45 MiB/s wr, 721 op/s 2024-04-29T14:25:59.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:25:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2299169355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:59.125 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:25:58 smithi121 systemd[1]: Stopping Ceph osd.5 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:25:59.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:25:59.188 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:59.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:25:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:59.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:25:59.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 133462 133446 108.308 0 - 0.008611 2024-04-29T14:25:59.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:25:59.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:59 smithi194 ceph-mon[44089]: pgmap v1213: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 45 MiB/s wr, 721 op/s 2024-04-29T14:25:59.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:25:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2299169355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:59.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:59 smithi040 ceph-mon[39581]: pgmap v1213: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 45 MiB/s wr, 721 op/s 2024-04-29T14:25:59.340 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:25:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2299169355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:25:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:25:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:59.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:25:59.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:25:59.546 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:25:59 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[115165]: 2024-04-29T14:25:59.206+0000 7f0b65d2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T14:25:59.547 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:25:59 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[115165]: 2024-04-29T14:25:59.206+0000 7f0b65d2f700 -1 osd.5 457 *** Got signal Terminated *** 2024-04-29T14:25:59.547 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:25:59 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[115165]: 2024-04-29T14:25:59.206+0000 7f0b65d2f700 -1 osd.5 457 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T14:25:59.549 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:25:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:59.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:25:59.673 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:25:59.702 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:59.721 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:25:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:59.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:25:59.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:25:59.854 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:25:59.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:25:59.874 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:25:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:25:59.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:25:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:25:59.978 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:00.007 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:00.028 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:00.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:00.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:00.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:00 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:00.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2299169355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:00.111 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:00 smithi040 ceph-mon[39581]: osdmap e458: 12 total, 12 up, 8 in 2024-04-29T14:26:00.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:00 smithi040 ceph-mon[39581]: osd.5 marked itself down and dead 2024-04-29T14:26:00.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:00 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:00.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2299169355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:00.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:00 smithi121 ceph-mon[44533]: osdmap e458: 12 total, 12 up, 8 in 2024-04-29T14:26:00.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:00 smithi121 ceph-mon[44533]: osd.5 marked itself down and dead 2024-04-29T14:26:00.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:00.159 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:00.178 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:00.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:00.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 133462 133446 106.919 0 - 0.008611 2024-04-29T14:26:00.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:00.310 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:00 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:00.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2299169355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:00 smithi194 ceph-mon[44089]: osdmap e458: 12 total, 12 up, 8 in 2024-04-29T14:26:00.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:00 smithi194 ceph-mon[44089]: osd.5 marked itself down and dead 2024-04-29T14:26:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:00.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:00.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:00.688 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:00.696 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:01.016 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:26:00 smithi121 bash[125531]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5 2024-04-29T14:26:01.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 133462 133446 105.566 0 - 0.008611 2024-04-29T14:26:01.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:01 smithi194 ceph-mon[44089]: pgmap v1215: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 25 MiB/s wr, 398 op/s 2024-04-29T14:26:01.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:01 smithi194 ceph-mon[44089]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:26:01.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:01 smithi194 ceph-mon[44089]: osdmap e459: 12 total, 11 up, 8 in 2024-04-29T14:26:01.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:01 smithi040 ceph-mon[39581]: pgmap v1215: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 25 MiB/s wr, 398 op/s 2024-04-29T14:26:01.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:01 smithi040 ceph-mon[39581]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:26:01.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:01 smithi040 ceph-mon[39581]: osdmap e459: 12 total, 11 up, 8 in 2024-04-29T14:26:01.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:01 smithi121 ceph-mon[44533]: pgmap v1215: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 25 MiB/s wr, 398 op/s 2024-04-29T14:26:01.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:01 smithi121 ceph-mon[44533]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:26:01.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:01 smithi121 ceph-mon[44533]: osdmap e459: 12 total, 11 up, 8 in 2024-04-29T14:26:01.947 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:01.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:01.962 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:01.982 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:02.001 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:02.021 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:02.042 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:02.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:26:02.069 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:02.082 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:02.094 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:02.112 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:02.156 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:02.173 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:26:02.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:26:02.180 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:02.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:02.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:02 smithi040 ceph-mon[39581]: osdmap e460: 12 total, 11 up, 8 in 2024-04-29T14:26:02.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:02.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:02.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1622007512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:02.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:26:02.264652 min lat: 0.00099482 max lat: 0.218319 avg lat: 0.008611 2024-04-29T14:26:02.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:26:02.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 133462 133446 104.246 0 - 0.008611 2024-04-29T14:26:02.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:02.303 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:02 smithi194 ceph-mon[44089]: osdmap e460: 12 total, 11 up, 8 in 2024-04-29T14:26:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:02.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1622007512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:02.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:02 smithi121 ceph-mon[44533]: osdmap e460: 12 total, 11 up, 8 in 2024-04-29T14:26:02.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:02.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:02.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1622007512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:02.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:26:02.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5.service' 2024-04-29T14:26:02.852 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:02.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:02.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T14:26:02.863 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-29T14:26:02.863 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd down 5 2024-04-29T14:26:02.956 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:02.984 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: pgmap v1218: 17 pgs: 2 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 2 active+clean+scrubbing+deep, 11 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 16 MiB/s wr, 254 op/s; 16848/266896 objects degraded (6.313%) 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 16848/266896 objects degraded (6.313%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osdmap e461: 12 total, 11 up, 8 in 2024-04-29T14:26:03.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:03.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:03.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: pgmap v1218: 17 pgs: 2 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 2 active+clean+scrubbing+deep, 11 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 16 MiB/s wr, 254 op/s; 16848/266896 objects degraded (6.313%) 2024-04-29T14:26:03.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 16848/266896 objects degraded (6.313%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:03.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osdmap e461: 12 total, 11 up, 8 in 2024-04-29T14:26:03.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:03.126 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:03.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 133462 133446 102.959 0 - 0.008611 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: pgmap v1218: 17 pgs: 2 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 2 active+clean+scrubbing+deep, 11 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 16 MiB/s wr, 254 op/s; 16848/266896 objects degraded (6.313%) 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 16848/266896 objects degraded (6.313%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osdmap e461: 12 total, 11 up, 8 in 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:03.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:03.734 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:03.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:03.743 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:03.745 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:03.839 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:03.868 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:03.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:04.070 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:04.101 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:04.122 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:04.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:04.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:04.239 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:04 smithi040 ceph-mon[39581]: osdmap e462: 12 total, 11 up, 8 in 2024-04-29T14:26:04.239 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:04.239 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:04.255 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:04.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 133462 133446 101.703 0 - 0.008611 2024-04-29T14:26:04.275 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:04.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:04.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:04 smithi194 ceph-mon[44089]: osdmap e462: 12 total, 11 up, 8 in 2024-04-29T14:26:04.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:04.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:04.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:04 smithi121 ceph-mon[44533]: osdmap e462: 12 total, 11 up, 8 in 2024-04-29T14:26:04.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:04.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:04.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:04.407 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:04.427 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:04.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:04.531 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:26:04.558 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:04.578 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:26:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:04.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:26:04.594 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:26:04.594 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:26:04.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:26:04.594 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:26:04.594 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:26:04.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:04.726 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:04.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:04.751 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:04.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:05.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:05 smithi040 ceph-mon[39581]: pgmap v1221: 17 pgs: 3 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 25196/266896 objects degraded (9.440%) 2024-04-29T14:26:05.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:05 smithi040 ceph-mon[39581]: osdmap e463: 12 total, 11 up, 8 in 2024-04-29T14:26:05.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 16 133462 133446 100.478 0 - 0.008611 2024-04-29T14:26:05.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:05 smithi194 ceph-mon[44089]: pgmap v1221: 17 pgs: 3 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 25196/266896 objects degraded (9.440%) 2024-04-29T14:26:05.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:05 smithi194 ceph-mon[44089]: osdmap e463: 12 total, 11 up, 8 in 2024-04-29T14:26:05.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:05 smithi121 ceph-mon[44533]: pgmap v1221: 17 pgs: 3 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 25196/266896 objects degraded (9.440%) 2024-04-29T14:26:05.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:05 smithi121 ceph-mon[44533]: osdmap e463: 12 total, 11 up, 8 in 2024-04-29T14:26:05.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:26:05.810 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:05.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:05.915 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:05.948 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:05.969 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:05.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:06.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:06.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 133462 133446 99.2819 0 - 0.008611 2024-04-29T14:26:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:06.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:06.402 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:06.431 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:06.530 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:06.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:06.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:26:06.667 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:06.692 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:26:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:06.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:26:06.797 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:26:06.824 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:06.842 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:26:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:06.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:26:06.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:06.977 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:07.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 16 133462 133446 98.1139 0 - 0.008611 2024-04-29T14:26:07.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:07.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:07.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:07 smithi040 ceph-mon[39581]: pgmap v1223: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:07.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:07 smithi040 ceph-mon[39581]: osdmap e464: 12 total, 11 up, 8 in 2024-04-29T14:26:07.291 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:26:07.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:07.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:07.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:07 smithi194 ceph-mon[44089]: pgmap v1223: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:07.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:07 smithi194 ceph-mon[44089]: osdmap e464: 12 total, 11 up, 8 in 2024-04-29T14:26:07.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:26:07.327 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:07.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:07.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:07.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:07.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:07 smithi121 ceph-mon[44533]: pgmap v1223: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:07.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:07 smithi121 ceph-mon[44533]: osdmap e464: 12 total, 11 up, 8 in 2024-04-29T14:26:07.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:26:07.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:07.458 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:07.477 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:07.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:07.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:07.609 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:07.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:07.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:26:07.880 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:26:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:07.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:26:08.006 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:26:08.038 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:08.059 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:26:08.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:26:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:08.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:26:08.074 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:26:08.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:08.190 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:08.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 133462 133446 96.973 0 - 0.008611 2024-04-29T14:26:08.288 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:08.290 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2776200039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:26:08.290 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:08 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 41941/266896 objects degraded (15.714%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:08.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:08.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2776200039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:26:08.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:08 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 41941/266896 objects degraded (15.714%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:08.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2776200039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:26:08.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:08 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 41941/266896 objects degraded (15.714%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:08.392 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:08.423 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:08.445 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:08.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:08.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:08.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:26:08.578 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:08.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:26:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:08.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:26:08.617 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:08.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:08.734 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:08.755 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:08.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:08.774 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5 is already down. 2024-04-29T14:26:08.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:08.926 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:08.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:09.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:09.061 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:09.067 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:08 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:26:08.778+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e465 definitely_dead 0 2024-04-29T14:26:09.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:09.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:09.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:09.224 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:09.245 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:09.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:09.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 133462 133446 95.8582 0 - 0.008611 2024-04-29T14:26:09.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:09.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:09 smithi040 ceph-mon[39581]: pgmap v1225: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:09.361 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:09 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2776200039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:26:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:09 smithi040 ceph-mon[39581]: osdmap e465: 12 total, 11 up, 8 in 2024-04-29T14:26:09.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1226698497' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["5"]}]: dispatch 2024-04-29T14:26:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:09 smithi121 ceph-mon[44533]: pgmap v1225: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:09 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2776200039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:26:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:09 smithi121 ceph-mon[44533]: osdmap e465: 12 total, 11 up, 8 in 2024-04-29T14:26:09.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1226698497' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["5"]}]: dispatch 2024-04-29T14:26:09.378 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:09 smithi194 ceph-mon[44089]: pgmap v1225: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:09 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2776200039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:26:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:09 smithi194 ceph-mon[44089]: osdmap e465: 12 total, 11 up, 8 in 2024-04-29T14:26:09.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1226698497' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["5"]}]: dispatch 2024-04-29T14:26:09.615 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 4, 5, 7, 9, 10, 11, 6] 2024-04-29T14:26:09.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 5 2024-04-29T14:26:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:09.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:09.684 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:09.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:10.125 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:10.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:10.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:10.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:26:10.256 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:10.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 133462 133446 94.7688 0 - 0.008611 2024-04-29T14:26:10.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:26:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:10.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:26:10.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:10.409 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:10.429 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:10.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:10.533 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:10.564 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:10.585 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:10.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:10.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:26:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:10.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:26:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:10.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:26:10.840 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:10.873 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:10.894 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:10.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:10.952 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:26:10.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:10.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:10.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:10 smithi040 ceph-mon[39581]: pgmap v1227: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:10.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:26:10.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:26:10.980 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:10.997 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:11.027 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:11.027 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:11.027 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:10 smithi194 ceph-mon[44089]: pgmap v1227: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:11.027 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:26:11.027 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:26:11.027 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:11.029 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:11.050 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:11.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:11.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:11.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:11.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:10 smithi121 ceph-mon[44533]: pgmap v1227: 17 pgs: 5 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:11.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:26:11.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:26:11.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:26:11.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:26:11.184 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:11.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 133462 133446 93.704 0 - 0.008611 2024-04-29T14:26:11.432 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:26:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:11.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:26:11.521 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:11.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:11.563 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:11.989 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:11 smithi040 ceph-mon[39581]: 5.1 scrub starts 2024-04-29T14:26:12.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:11 smithi194 ceph-mon[44089]: 5.1 scrub starts 2024-04-29T14:26:12.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:11 smithi121 ceph-mon[44533]: 5.1 scrub starts 2024-04-29T14:26:12.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 90 16 133462 133446 92.6628 0 - 0.008611 2024-04-29T14:26:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:12 smithi040 ceph-mon[39581]: 5.1 scrub ok 2024-04-29T14:26:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:12 smithi040 ceph-mon[39581]: osdmap e466: 12 total, 11 up, 8 in 2024-04-29T14:26:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:12 smithi040 ceph-mon[39581]: pgmap v1229: 17 pgs: 1 active+clean+scrubbing, 5 undersized+degraded+peered, 11 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:13.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:13.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:13.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:12 smithi194 ceph-mon[44089]: 5.1 scrub ok 2024-04-29T14:26:13.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:12 smithi194 ceph-mon[44089]: osdmap e466: 12 total, 11 up, 8 in 2024-04-29T14:26:13.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:12 smithi194 ceph-mon[44089]: pgmap v1229: 17 pgs: 1 active+clean+scrubbing, 5 undersized+degraded+peered, 11 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:13.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:13.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:13.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:12 smithi121 ceph-mon[44533]: 5.1 scrub ok 2024-04-29T14:26:13.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:12 smithi121 ceph-mon[44533]: osdmap e466: 12 total, 11 up, 8 in 2024-04-29T14:26:13.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:12 smithi121 ceph-mon[44533]: pgmap v1229: 17 pgs: 1 active+clean+scrubbing, 5 undersized+degraded+peered, 11 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:13.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 91 16 133462 133446 91.6445 0 - 0.008611 2024-04-29T14:26:13.900 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:13 smithi040 ceph-mon[39581]: 5.0 scrub starts 2024-04-29T14:26:13.901 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2537790010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:13.901 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:14.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:13 smithi194 ceph-mon[44089]: 5.0 scrub starts 2024-04-29T14:26:14.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2537790010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:14.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:14.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:13 smithi121 ceph-mon[44533]: 5.0 scrub starts 2024-04-29T14:26:14.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2537790010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:14.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:14.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 92 16 133462 133446 90.6483 0 - 0.008611 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:14.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:14.434 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:14.435 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:14.435 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:14.435 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:14.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:14.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:26:14.510+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e466 definitely_dead 0 2024-04-29T14:26:14.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.5. 2024-04-29T14:26:14.782 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:15.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: 5.0 scrub ok 2024-04-29T14:26:15.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: 5.a scrub starts 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: pgmap v1230: 17 pgs: 2 active+clean+scrubbing, 5 undersized+degraded+peered, 10 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2366920513' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-29T14:26:15.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:14 smithi194 ceph-mon[44089]: Client client.admin marked osd.5 out, after it was down for 14 seconds 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: 5.0 scrub ok 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: 5.a scrub starts 2024-04-29T14:26:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: pgmap v1230: 17 pgs: 2 active+clean+scrubbing, 5 undersized+degraded+peered, 10 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2366920513' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-29T14:26:15.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:14 smithi040 ceph-mon[39581]: Client client.admin marked osd.5 out, after it was down for 14 seconds 2024-04-29T14:26:15.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: 5.0 scrub ok 2024-04-29T14:26:15.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: 5.a scrub starts 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: pgmap v1230: 17 pgs: 2 active+clean+scrubbing, 5 undersized+degraded+peered, 10 active+clean; 8.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 41941/266896 objects degraded (15.714%) 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2366920513' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-29T14:26:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:14 smithi121 ceph-mon[44533]: Client client.admin marked osd.5 out, after it was down for 14 seconds 2024-04-29T14:26:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:15.195 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:15.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 93 16 133462 133446 89.6736 0 - 0.008611 2024-04-29T14:26:15.272 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:26:15.272 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:26:15.632 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:15.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:15.736 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:15.792 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:15.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:15.900 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:26:15.929 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:15.949 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:26:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:15.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:26:16.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:16.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:15 smithi194 ceph-mon[44089]: 5.c scrub starts 2024-04-29T14:26:16.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:15 smithi194 ceph-mon[44089]: 5.a scrub ok 2024-04-29T14:26:16.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:15 smithi194 ceph-mon[44089]: 5.c scrub ok 2024-04-29T14:26:16.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:15 smithi194 ceph-mon[44089]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:26:16.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2366920513' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-29T14:26:16.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:15 smithi194 ceph-mon[44089]: osdmap e467: 12 total, 11 up, 7 in 2024-04-29T14:26:16.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:15 smithi040 ceph-mon[39581]: 5.c scrub starts 2024-04-29T14:26:16.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:15 smithi040 ceph-mon[39581]: 5.a scrub ok 2024-04-29T14:26:16.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:15 smithi040 ceph-mon[39581]: 5.c scrub ok 2024-04-29T14:26:16.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:15 smithi040 ceph-mon[39581]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:26:16.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2366920513' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-29T14:26:16.079 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:15 smithi040 ceph-mon[39581]: osdmap e467: 12 total, 11 up, 7 in 2024-04-29T14:26:16.081 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:16.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:15 smithi121 ceph-mon[44533]: 5.c scrub starts 2024-04-29T14:26:16.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:15 smithi121 ceph-mon[44533]: 5.a scrub ok 2024-04-29T14:26:16.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:15 smithi121 ceph-mon[44533]: 5.c scrub ok 2024-04-29T14:26:16.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:15 smithi121 ceph-mon[44533]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:26:16.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2366920513' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-29T14:26:16.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:15 smithi121 ceph-mon[44533]: osdmap e467: 12 total, 11 up, 7 in 2024-04-29T14:26:16.251 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:16.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 93.9598 2024-04-29T14:26:16.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 133463 2024-04-29T14:26:16.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:26:16.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 88.7767 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 68.6502 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 287.438 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 0 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1420 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 1098.4 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 4599 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 0 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.0112623 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.24225 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 22.315 2024-04-29T14:26:16.259 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.00099482 2024-04-29T14:26:16.281 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-04-29T14:26:16.281 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi121.front.sepia.ceph.com 2024-04-29T14:26:16.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-04-29T14:26:16.513 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:26:17.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:26:17.041 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:16 smithi040 ceph-mon[39581]: osdmap e468: 12 total, 11 up, 7 in 2024-04-29T14:26:17.041 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:16 smithi040 ceph-mon[39581]: pgmap v1233: 17 pgs: 2 active+clean+scrubbing, 1 peering, 4 undersized+degraded+peered, 10 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 33542/266896 objects degraded (12.567%) 2024-04-29T14:26:17.041 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:16 smithi040 ceph-mon[39581]: osdmap e469: 12 total, 11 up, 7 in 2024-04-29T14:26:17.044 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:17.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:17.048 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:26:17.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:16 smithi194 ceph-mon[44089]: osdmap e468: 12 total, 11 up, 7 in 2024-04-29T14:26:17.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:16 smithi194 ceph-mon[44089]: pgmap v1233: 17 pgs: 2 active+clean+scrubbing, 1 peering, 4 undersized+degraded+peered, 10 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 33542/266896 objects degraded (12.567%) 2024-04-29T14:26:17.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:16 smithi194 ceph-mon[44089]: osdmap e469: 12 total, 11 up, 7 in 2024-04-29T14:26:17.129 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:16 smithi121 ceph-mon[44533]: osdmap e468: 12 total, 11 up, 7 in 2024-04-29T14:26:17.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:16 smithi121 ceph-mon[44533]: pgmap v1233: 17 pgs: 2 active+clean+scrubbing, 1 peering, 4 undersized+degraded+peered, 10 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 33542/266896 objects degraded (12.567%) 2024-04-29T14:26:17.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:16 smithi121 ceph-mon[44533]: osdmap e469: 12 total, 11 up, 7 in 2024-04-29T14:26:17.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:17.180 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:17.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:26:18.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:17 smithi194 ceph-mon[44089]: 5.3 scrub starts 2024-04-29T14:26:18.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:17 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 33542/266896 objects degraded (12.567%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:18.063 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:17 smithi040 ceph-mon[39581]: 5.3 scrub starts 2024-04-29T14:26:18.064 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:17 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 33542/266896 objects degraded (12.567%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:18.099 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:18.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:17 smithi121 ceph-mon[44533]: 5.3 scrub starts 2024-04-29T14:26:18.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:17 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 33542/266896 objects degraded (12.567%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:18.134 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:18.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:18.240 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:18.305 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:18.337 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:18.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:18.444 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:18.579 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:18.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:18.683 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:18.714 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:19.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:18 smithi194 ceph-mon[44089]: 5.3 scrub ok 2024-04-29T14:26:19.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:18 smithi194 ceph-mon[44089]: pgmap v1235: 17 pgs: 3 active+recovering+degraded, 1 peering, 1 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 181 KiB/s wr, 18 op/s; 33458/266746 objects degraded (12.543%); 896 KiB/s, 13 objects/s recovering 2024-04-29T14:26:19.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:19.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:19.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:18 smithi040 ceph-mon[39581]: 5.3 scrub ok 2024-04-29T14:26:19.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:18 smithi040 ceph-mon[39581]: pgmap v1235: 17 pgs: 3 active+recovering+degraded, 1 peering, 1 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 181 KiB/s wr, 18 op/s; 33458/266746 objects degraded (12.543%); 896 KiB/s, 13 objects/s recovering 2024-04-29T14:26:19.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:19.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:19.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:18 smithi121 ceph-mon[44533]: 5.3 scrub ok 2024-04-29T14:26:19.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:18 smithi121 ceph-mon[44533]: pgmap v1235: 17 pgs: 3 active+recovering+degraded, 1 peering, 1 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 181 KiB/s wr, 18 op/s; 33458/266746 objects degraded (12.543%); 896 KiB/s, 13 objects/s recovering 2024-04-29T14:26:19.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:19.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:19.267 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:19.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:19.302 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:19.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:19.413 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:19.468 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:19.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:19.573 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:19.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:19.735 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:19.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:19.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:19 smithi194 ceph-mon[44089]: 5.2 scrub starts 2024-04-29T14:26:19.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:19 smithi194 ceph-mon[44089]: osdmap e470: 12 total, 11 up, 7 in 2024-04-29T14:26:19.798 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2742585323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:19.807 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:26:19.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:19.861 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:19 smithi040 ceph-mon[39581]: 5.2 scrub starts 2024-04-29T14:26:19.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:19 smithi040 ceph-mon[39581]: osdmap e470: 12 total, 11 up, 7 in 2024-04-29T14:26:19.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2742585323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:20.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:19 smithi121 ceph-mon[44533]: 5.2 scrub starts 2024-04-29T14:26:20.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:19 smithi121 ceph-mon[44533]: osdmap e470: 12 total, 11 up, 7 in 2024-04-29T14:26:20.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2742585323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:20.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:20.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:20.861 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:21.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:20 smithi194 ceph-mon[44089]: 5.2 scrub ok 2024-04-29T14:26:21.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:20 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:26:21.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2742585323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:21.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:20 smithi194 ceph-mon[44089]: osdmap e471: 12 total, 11 up, 7 in 2024-04-29T14:26:21.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:20 smithi194 ceph-mon[44089]: pgmap v1238: 17 pgs: 3 active+recovering+degraded, 1 peering, 1 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 259 KiB/s wr, 25 op/s; 33458/266746 objects degraded (12.543%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:26:21.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:20 smithi040 ceph-mon[39581]: 5.2 scrub ok 2024-04-29T14:26:21.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:20 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:26:21.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2742585323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:21.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:20 smithi040 ceph-mon[39581]: osdmap e471: 12 total, 11 up, 7 in 2024-04-29T14:26:21.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:20 smithi040 ceph-mon[39581]: pgmap v1238: 17 pgs: 3 active+recovering+degraded, 1 peering, 1 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 259 KiB/s wr, 25 op/s; 33458/266746 objects degraded (12.543%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:26:21.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:20 smithi121 ceph-mon[44533]: 5.2 scrub ok 2024-04-29T14:26:21.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:20 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:26:21.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2742585323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:21.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:20 smithi121 ceph-mon[44533]: osdmap e471: 12 total, 11 up, 7 in 2024-04-29T14:26:21.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:20 smithi121 ceph-mon[44533]: pgmap v1238: 17 pgs: 3 active+recovering+degraded, 1 peering, 1 undersized+degraded+peered, 12 active+clean; 8.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 259 KiB/s wr, 25 op/s; 33458/266746 objects degraded (12.543%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:26:21.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:21.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:21.826 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:21.885 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:21.915 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:21.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:22.023 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:22.065 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:22.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1555190444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:22.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1555190444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:22.326 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:22.327 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:22.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1555190444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:22.617 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:22.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:22.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:22.753 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:22.878 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:22.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:22.983 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:23.022 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:23.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:23.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:26:23.191 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:26:23.192 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:26:23.192 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:26:23.192 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:26:23.192 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:26:23.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: pgmap v1239: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 8.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 170 B/s rd, 181 KiB/s wr, 329 op/s; 41245/263014 objects degraded (15.682%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:26:23.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.203 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:22 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 33458/266746 objects degraded (12.543%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:23.206 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: pgmap v1239: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 8.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 170 B/s rd, 181 KiB/s wr, 329 op/s; 41245/263014 objects degraded (15.682%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:22 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 33458/266746 objects degraded (12.543%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: pgmap v1239: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 8.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 170 B/s rd, 181 KiB/s wr, 329 op/s; 41245/263014 objects degraded (15.682%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:26:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:23.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:22 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 33458/266746 objects degraded (12.543%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:23.363 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:26:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:23.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:24.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:26:24.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:24.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:24.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:24.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:24.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:24.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:24.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:26:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:26:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:24.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:26:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:24.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:24.703 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:24.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:24.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:24.846 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:24.882 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:24.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:25.009 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:26:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:25.083 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:26:25.087 INFO:teuthology.orchestra.run.smithi121.stdout:5.d 2024-04-29T14:26:25.087 INFO:teuthology.orchestra.run.smithi121.stdout:5.7 2024-04-29T14:26:25.087 INFO:teuthology.orchestra.run.smithi121.stdout:5.f 2024-04-29T14:26:25.087 INFO:teuthology.orchestra.run.smithi121.stdout:5.4 2024-04-29T14:26:25.087 INFO:teuthology.orchestra.run.smithi121.stdout:5.6 2024-04-29T14:26:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:25.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:26:25.093 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:24 smithi040 ceph-mon[39581]: pgmap v1240: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 8.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 142 B/s rd, 142 B/s wr, 351 op/s; 41097/261884 objects degraded (15.693%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:26:25.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:24 smithi121 ceph-mon[44533]: pgmap v1240: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 8.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 142 B/s rd, 142 B/s wr, 351 op/s; 41097/261884 objects degraded (15.693%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:26:25.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:26:25.231 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:26:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:25.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:26:25.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:24 smithi194 ceph-mon[44089]: pgmap v1240: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 8.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 142 B/s rd, 142 B/s wr, 351 op/s; 41097/261884 objects degraded (15.693%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:26:25.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:26:25.404 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:25.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:26:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:25.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:26:25.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:25.582 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:25.805 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:25.834 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:25.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:25.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:26:25.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:25.974 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:26.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:26.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:26.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:26.161 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:27.029 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:26:27.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:27 smithi121 ceph-mon[44533]: pgmap v1241: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 127 B/s rd, 127 B/s wr, 463 op/s; 40728/259334 objects degraded (15.705%); 8.8 MiB/s, 141 objects/s recovering 2024-04-29T14:26:27.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3264841354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:27.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:27 smithi121 ceph-mon[44533]: from='client.26209 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:27.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:27 smithi194 ceph-mon[44089]: pgmap v1241: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 127 B/s rd, 127 B/s wr, 463 op/s; 40728/259334 objects degraded (15.705%); 8.8 MiB/s, 141 objects/s recovering 2024-04-29T14:26:27.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3264841354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:27.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:27 smithi194 ceph-mon[44089]: from='client.26209 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:27.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 5.6 --file /var/log/ceph/exp.5.6.5 2024-04-29T14:26:27.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:27 smithi040 ceph-mon[39581]: pgmap v1241: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 127 B/s rd, 127 B/s wr, 463 op/s; 40728/259334 objects degraded (15.705%); 8.8 MiB/s, 141 objects/s recovering 2024-04-29T14:26:27.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3264841354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:27.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:27 smithi040 ceph-mon[39581]: from='client.26209 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:27.615 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:27.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:27.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:27.752 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:27.763 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:26:27.880 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:27.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:27.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:28.057 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:28.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:28.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:28.197 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:28.214 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:26:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:26:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:26:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:26:28.215 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:26:28.227 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:28.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:28.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:28 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:28.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:28 smithi040 ceph-mon[39581]: from='client.26209 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:26:28.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:28 smithi040 ceph-mon[39581]: osdmap e472: 12 total, 11 up, 7 in 2024-04-29T14:26:28.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:28 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 40728/259334 objects degraded (15.705%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:28.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:28 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:28.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:28 smithi194 ceph-mon[44089]: from='client.26209 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:26:28.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:28 smithi194 ceph-mon[44089]: osdmap e472: 12 total, 11 up, 7 in 2024-04-29T14:26:28.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:28 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 40728/259334 objects degraded (15.705%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:28.332 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:28.360 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:28.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:28 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:28.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:28 smithi121 ceph-mon[44533]: from='client.26209 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:26:28.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:28 smithi121 ceph-mon[44533]: osdmap e472: 12 total, 11 up, 7 in 2024-04-29T14:26:28.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:28 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 40728/259334 objects degraded (15.705%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:28.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:28.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:28.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:28.570 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:28.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:28.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:28.808 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:28.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:28.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:29.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:26:29.084 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:29.119 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:26:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:29.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:26:29.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:29.285 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:29.334 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:29.339 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:29 smithi194 ceph-mon[44089]: pgmap v1243: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 125 B/s rd, 125 B/s wr, 584 op/s; 40382/257228 objects degraded (15.699%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:26:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:29.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:29.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:26:29.464 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:29 smithi121 ceph-mon[44533]: pgmap v1243: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 125 B/s rd, 125 B/s wr, 584 op/s; 40382/257228 objects degraded (15.699%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:26:29.466 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:29.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:29 smithi040 ceph-mon[39581]: pgmap v1243: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 125 B/s rd, 125 B/s wr, 584 op/s; 40382/257228 objects degraded (15.699%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:26:29.712 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:29.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:26:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:26:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:29.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:26:29.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:29.998 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:30.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:30.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:30.133 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:30.262 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:30 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:30.262 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:30 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:30.263 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:30 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:30.263 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:30 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:30.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:30 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:30.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:30 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:30.374 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:30 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:30.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:30 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:30 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:30 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:30 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:30.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:30 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:31.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:26:31.300 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:26:31.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:31.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:31.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:31.442 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:31.492 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:31 smithi121 ceph-mon[44533]: pgmap v1244: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 102 B/s rd, 102 B/s wr, 531 op/s; 40192/256124 objects degraded (15.692%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:26:31.492 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:31 smithi121 ceph-mon[44533]: osdmap e473: 12 total, 11 up, 7 in 2024-04-29T14:26:31.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:31.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:31.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:31 smithi040 ceph-mon[39581]: pgmap v1244: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 102 B/s rd, 102 B/s wr, 531 op/s; 40192/256124 objects degraded (15.692%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:26:31.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:31 smithi040 ceph-mon[39581]: osdmap e473: 12 total, 11 up, 7 in 2024-04-29T14:26:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:31 smithi194 ceph-mon[44089]: pgmap v1244: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 102 B/s rd, 102 B/s wr, 531 op/s; 40192/256124 objects degraded (15.692%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:26:31.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:31 smithi194 ceph-mon[44089]: osdmap e473: 12 total, 11 up, 7 in 2024-04-29T14:26:31.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:26:31.642 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:31.831 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:26:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:31.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:26:31.863 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:31.886 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:31.908 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:31.935 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:31.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:31.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:31.976 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:31.978 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:32.008 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:32.030 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:32.054 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:32.077 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:32.082 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:32.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:32.144 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:26:32.185 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:26:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:32.267 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:26:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:32.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:26:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3008201702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:32.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:32 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 40192/256124 objects degraded (15.692%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:32.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3008201702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:32.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:32 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 40192/256124 objects degraded (15.692%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:32.375 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:26:32.405 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:32.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3008201702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:32.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:32 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 40192/256124 objects degraded (15.692%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:33.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: pgmap v1246: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 39495/251372 objects degraded (15.712%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: pgmap v1246: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 39495/251372 objects degraded (15.712%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: pgmap v1246: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 39495/251372 objects degraded (15.712%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:33.959 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:26:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:33.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:26:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:33.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:34.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:34.093 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:34.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:34.548 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:34.581 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:34.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:34.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:34.883 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:34.945 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:34.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:35.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:35.084 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:35.106 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:35.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:35.212 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:26:35.229 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:35 smithi194 ceph-mon[44089]: pgmap v1247: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 564 op/s; 39317/250306 objects degraded (15.708%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:26:35.231 INFO:teuthology.orchestra.run.smithi121.stderr:Exporting 5.6 info 5.6( v 457'8382 (0'0,457'8382] local-lis/les=432/433 n=8382 ec=432/432 lis/c=432/432 les/c/f=433/433/0 sis=432) 2024-04-29T14:26:35.276 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:35.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60001ed0:::benchmark_data_smithi194_121512_object24470:head# 2024-04-29T14:26:35.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6000a506:::benchmark_data_smithi194_121512_object73522:head# 2024-04-29T14:26:35.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6000eb80:::benchmark_data_smithi194_121512_object30226:head# 2024-04-29T14:26:35.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6001080f:::benchmark_data_smithi194_121512_object78388:head# 2024-04-29T14:26:35.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60028613:::benchmark_data_smithi194_121512_object45583:head# 2024-04-29T14:26:35.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6002aa12:::benchmark_data_smithi194_121512_object71361:head# 2024-04-29T14:26:35.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6003031b:::benchmark_data_smithi194_121512_object98011:head# 2024-04-29T14:26:35.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6004218c:::benchmark_data_smithi194_121512_object14532:head# 2024-04-29T14:26:35.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60042ddd:::benchmark_data_smithi194_121512_object47941:head# 2024-04-29T14:26:35.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6004519a:::benchmark_data_smithi194_121512_object69753:head# 2024-04-29T14:26:35.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60049131:::benchmark_data_smithi194_121512_object127112:head# 2024-04-29T14:26:35.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6004f0ff:::benchmark_data_smithi194_121512_object121400:head# 2024-04-29T14:26:35.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600538b9:::benchmark_data_smithi194_121512_object4360:head# 2024-04-29T14:26:35.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60063bd2:::benchmark_data_smithi194_121512_object47110:head# 2024-04-29T14:26:35.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60064e30:::benchmark_data_smithi194_121512_object43763:head# 2024-04-29T14:26:35.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60074edf:::benchmark_data_smithi194_121512_object82297:head# 2024-04-29T14:26:35.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6007589e:::benchmark_data_smithi194_121512_object73634:head# 2024-04-29T14:26:35.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6007d98d:::benchmark_data_smithi194_121512_object118990:head# 2024-04-29T14:26:35.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600855e0:::benchmark_data_smithi194_121512_object83524:head# 2024-04-29T14:26:35.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600b1280:::benchmark_data_smithi194_121512_object105085:head# 2024-04-29T14:26:35.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600b8e01:::benchmark_data_smithi194_121512_object123610:head# 2024-04-29T14:26:35.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600dd11d:::benchmark_data_smithi194_121512_object76597:head# 2024-04-29T14:26:35.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600df137:::benchmark_data_smithi194_121512_object81861:head# 2024-04-29T14:26:35.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600e0973:::benchmark_data_smithi194_121512_object81877:head# 2024-04-29T14:26:35.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600e5d43:::benchmark_data_smithi194_121512_object124695:head# 2024-04-29T14:26:35.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600e646e:::benchmark_data_smithi194_121512_object59793:head# 2024-04-29T14:26:35.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600e92ef:::benchmark_data_smithi194_121512_object6904:head# 2024-04-29T14:26:35.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:600ef351:::benchmark_data_smithi194_121512_object833:head# 2024-04-29T14:26:35.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6010f4eb:::benchmark_data_smithi194_121512_object27410:head# 2024-04-29T14:26:35.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60114e11:::benchmark_data_smithi194_121512_object14919:head# 2024-04-29T14:26:35.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6011593d:::benchmark_data_smithi194_121512_object58807:head# 2024-04-29T14:26:35.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60116a40:::benchmark_data_smithi194_121512_object83691:head# 2024-04-29T14:26:35.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601204fe:::benchmark_data_smithi194_121512_object27226:head# 2024-04-29T14:26:35.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60122e35:::benchmark_data_smithi194_121512_object52362:head# 2024-04-29T14:26:35.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60123e1c:::benchmark_data_smithi194_121512_object9504:head# 2024-04-29T14:26:35.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60138350:::benchmark_data_smithi194_121512_object93164:head# 2024-04-29T14:26:35.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6013cc60:::benchmark_data_smithi194_121512_object32954:head# 2024-04-29T14:26:35.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6013e97e:::benchmark_data_smithi194_121512_object113602:head# 2024-04-29T14:26:35.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60142d6e:::benchmark_data_smithi194_121512_object97725:head# 2024-04-29T14:26:35.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60143892:::benchmark_data_smithi194_121512_object12706:head# 2024-04-29T14:26:35.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601512fc:::benchmark_data_smithi194_121512_object87962:head# 2024-04-29T14:26:35.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60157b71:::benchmark_data_smithi194_121512_object132573:head# 2024-04-29T14:26:35.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6015bb54:::benchmark_data_smithi194_121512_object68143:head# 2024-04-29T14:26:35.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6016eef1:::benchmark_data_smithi194_121512_object95236:head# 2024-04-29T14:26:35.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6017228f:::benchmark_data_smithi194_121512_object27501:head# 2024-04-29T14:26:35.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60179ebc:::benchmark_data_smithi194_121512_object99439:head# 2024-04-29T14:26:35.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601820b4:::benchmark_data_smithi194_121512_object41161:head# 2024-04-29T14:26:35.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60197e9c:::benchmark_data_smithi194_121512_object16862:head# 2024-04-29T14:26:35.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601ae4af:::benchmark_data_smithi194_121512_object4313:head# 2024-04-29T14:26:35.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601b1e91:::benchmark_data_smithi194_121512_object118363:head# 2024-04-29T14:26:35.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601bb5df:::benchmark_data_smithi194_121512_object1031:head# 2024-04-29T14:26:35.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601cafd8:::benchmark_data_smithi194_121512_object99000:head# 2024-04-29T14:26:35.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601e0cc4:::benchmark_data_smithi194_121512_object90368:head# 2024-04-29T14:26:35.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601e259f:::benchmark_data_smithi194_121512_object26976:head# 2024-04-29T14:26:35.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601f1b83:::benchmark_data_smithi194_121512_object69058:head# 2024-04-29T14:26:35.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:601f600d:::benchmark_data_smithi194_121512_object119925:head# 2024-04-29T14:26:35.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6021b1c1:::benchmark_data_smithi194_121512_object86794:head# 2024-04-29T14:26:35.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6022701b:::benchmark_data_smithi194_121512_object115765:head# 2024-04-29T14:26:35.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60227c9a:::benchmark_data_smithi194_121512_object120838:head# 2024-04-29T14:26:35.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6022a1a6:::benchmark_data_smithi194_121512_object93875:head# 2024-04-29T14:26:35.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6022d939:::benchmark_data_smithi194_121512_object92603:head# 2024-04-29T14:26:35.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6023419a:::benchmark_data_smithi194_121512_object41922:head# 2024-04-29T14:26:35.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6023cc94:::benchmark_data_smithi194_121512_object30709:head# 2024-04-29T14:26:35.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6023de63:::benchmark_data_smithi194_121512_object72728:head# 2024-04-29T14:26:35.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6023ff5a:::benchmark_data_smithi194_121512_object7379:head# 2024-04-29T14:26:35.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60246553:::benchmark_data_smithi194_121512_object113303:head# 2024-04-29T14:26:35.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60252bb7:::benchmark_data_smithi194_121512_object65096:head# 2024-04-29T14:26:35.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60252e1e:::benchmark_data_smithi194_121512_object124130:head# 2024-04-29T14:26:35.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602644bc:::benchmark_data_smithi194_121512_object56286:head# 2024-04-29T14:26:35.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6026b460:::benchmark_data_smithi194_121512_object118360:head# 2024-04-29T14:26:35.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602719a3:::benchmark_data_smithi194_121512_object71043:head# 2024-04-29T14:26:35.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60273a38:::benchmark_data_smithi194_121512_object83910:head# 2024-04-29T14:26:35.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60277d2e:::benchmark_data_smithi194_121512_object123537:head# 2024-04-29T14:26:35.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6027df36:::benchmark_data_smithi194_121512_object25457:head# 2024-04-29T14:26:35.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6028a1a5:::benchmark_data_smithi194_121512_object81426:head# 2024-04-29T14:26:35.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6028c098:::benchmark_data_smithi194_121512_object71238:head# 2024-04-29T14:26:35.330 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:26:35.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6029f301:::benchmark_data_smithi194_121512_object10067:head# 2024-04-29T14:26:35.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602aaa54:::benchmark_data_smithi194_121512_object96596:head# 2024-04-29T14:26:35.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602aadb2:::benchmark_data_smithi194_121512_object99279:head# 2024-04-29T14:26:35.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602b19d0:::benchmark_data_smithi194_121512_object87833:head# 2024-04-29T14:26:35.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602bcc05:::benchmark_data_smithi194_121512_object38365:head# 2024-04-29T14:26:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:35.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:26:35.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602bd279:::benchmark_data_smithi194_121512_object71703:head# 2024-04-29T14:26:35.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602c37ab:::benchmark_data_smithi194_121512_object37715:head# 2024-04-29T14:26:35.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602cf058:::benchmark_data_smithi194_121512_object129962:head# 2024-04-29T14:26:35.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602d1bbb:::benchmark_data_smithi194_121512_object7326:head# 2024-04-29T14:26:35.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602de7ef:::benchmark_data_smithi194_121512_object6415:head# 2024-04-29T14:26:35.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602ded4d:::benchmark_data_smithi194_121512_object53889:head# 2024-04-29T14:26:35.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602def05:::benchmark_data_smithi194_121512_object43903:head# 2024-04-29T14:26:35.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602e54e9:::benchmark_data_smithi194_121512_object116609:head# 2024-04-29T14:26:35.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602e65bc:::benchmark_data_smithi194_121512_object51161:head# 2024-04-29T14:26:35.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602ece5e:::benchmark_data_smithi194_121512_object48668:head# 2024-04-29T14:26:35.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:602f14b9:::benchmark_data_smithi194_121512_object127704:head# 2024-04-29T14:26:35.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60309085:::benchmark_data_smithi194_121512_object71950:head# 2024-04-29T14:26:35.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6030e435:::benchmark_data_smithi194_121512_object123704:head# 2024-04-29T14:26:35.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60311c96:::benchmark_data_smithi194_121512_object102932:head# 2024-04-29T14:26:35.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60312de7:::benchmark_data_smithi194_121512_object119494:head# 2024-04-29T14:26:35.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60314d80:::benchmark_data_smithi194_121512_object47377:head# 2024-04-29T14:26:35.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6031ff13:::benchmark_data_smithi194_121512_object117718:head# 2024-04-29T14:26:35.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6032edae:::benchmark_data_smithi194_121512_object90328:head# 2024-04-29T14:26:35.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60335707:::benchmark_data_smithi194_121512_object66207:head# 2024-04-29T14:26:35.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6033591e:::benchmark_data_smithi194_121512_object56089:head# 2024-04-29T14:26:35.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60336818:::benchmark_data_smithi194_121512_object125856:head# 2024-04-29T14:26:35.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60340fbb:::benchmark_data_smithi194_121512_object10494:head# 2024-04-29T14:26:35.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60343949:::benchmark_data_smithi194_121512_object53124:head# 2024-04-29T14:26:35.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60345a56:::benchmark_data_smithi194_121512_object46420:head# 2024-04-29T14:26:35.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6034adad:::benchmark_data_smithi194_121512_object35407:head# 2024-04-29T14:26:35.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6034b087:::benchmark_data_smithi194_121512_object101005:head# 2024-04-29T14:26:35.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6036400b:::benchmark_data_smithi194_121512_object132962:head# 2024-04-29T14:26:35.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603649c8:::benchmark_data_smithi194_121512_object96620:head# 2024-04-29T14:26:35.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6036749b:::benchmark_data_smithi194_121512_object128259:head# 2024-04-29T14:26:35.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6036b899:::benchmark_data_smithi194_121512_object26969:head# 2024-04-29T14:26:35.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6036c528:::benchmark_data_smithi194_121512_object30200:head# 2024-04-29T14:26:35.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603733d6:::benchmark_data_smithi194_121512_object118012:head# 2024-04-29T14:26:35.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6037b033:::benchmark_data_smithi194_121512_object101083:head# 2024-04-29T14:26:35.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6037f30e:::benchmark_data_smithi194_121512_object71755:head# 2024-04-29T14:26:35.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6037f95a:::benchmark_data_smithi194_121512_object22610:head# 2024-04-29T14:26:35.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6038637e:::benchmark_data_smithi194_121512_object94895:head# 2024-04-29T14:26:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6038d5aa:::benchmark_data_smithi194_121512_object5815:head# 2024-04-29T14:26:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6038f1d6:::benchmark_data_smithi194_121512_object29760:head# 2024-04-29T14:26:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60391419:::benchmark_data_smithi194_121512_object102126:head# 2024-04-29T14:26:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603982f6:::benchmark_data_smithi194_121512_object22933:head# 2024-04-29T14:26:35.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6039b209:::benchmark_data_smithi194_121512_object65081:head# 2024-04-29T14:26:35.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6039cda0:::benchmark_data_smithi194_121512_object113982:head# 2024-04-29T14:26:35.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603a2355:::benchmark_data_smithi194_121512_object64898:head# 2024-04-29T14:26:35.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603abecb:::benchmark_data_smithi194_121512_object70910:head# 2024-04-29T14:26:35.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603adbb7:::benchmark_data_smithi194_121512_object73117:head# 2024-04-29T14:26:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603ae62c:::benchmark_data_smithi194_121512_object92507:head# 2024-04-29T14:26:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603bd947:::benchmark_data_smithi194_121512_object120483:head# 2024-04-29T14:26:35.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603be836:::benchmark_data_smithi194_121512_object92591:head# 2024-04-29T14:26:35.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603c73de:::benchmark_data_smithi194_121512_object108719:head# 2024-04-29T14:26:35.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603c7588:::benchmark_data_smithi194_121512_object51339:head# 2024-04-29T14:26:35.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:35 smithi040 ceph-mon[39581]: pgmap v1247: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 564 op/s; 39317/250306 objects degraded (15.708%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603c81ce:::benchmark_data_smithi194_121512_object99852:head# 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603d2050:::benchmark_data_smithi194_121512_object50688:head# 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603dc609:::benchmark_data_smithi194_121512_object115748:head# 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603e5d39:::benchmark_data_smithi194_121512_object9451:head# 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603f1228:::benchmark_data_smithi194_121512_object78930:head# 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603f25cf:::benchmark_data_smithi194_121512_object34462:head# 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603f37a2:::benchmark_data_smithi194_121512_object9387:head# 2024-04-29T14:26:35.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603f4196:::benchmark_data_smithi194_121512_object118040:head# 2024-04-29T14:26:35.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:603fff34:::benchmark_data_smithi194_121512_object92380:head# 2024-04-29T14:26:35.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60400a01:::benchmark_data_smithi194_121512_object9045:head# 2024-04-29T14:26:35.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60401670:::benchmark_data_smithi194_121512_object24657:head# 2024-04-29T14:26:35.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60402a21:::benchmark_data_smithi194_121512_object31025:head# 2024-04-29T14:26:35.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60406954:::benchmark_data_smithi194_121512_object72109:head# 2024-04-29T14:26:35.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60408026:::benchmark_data_smithi194_121512_object6549:head# 2024-04-29T14:26:35.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60408980:::benchmark_data_smithi194_121512_object57917:head# 2024-04-29T14:26:35.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60411417:::benchmark_data_smithi194_121512_object98457:head# 2024-04-29T14:26:35.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60415e7c:::benchmark_data_smithi194_121512_object72786:head# 2024-04-29T14:26:35.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6041647f:::benchmark_data_smithi194_121512_object14167:head# 2024-04-29T14:26:35.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60418c7d:::benchmark_data_smithi194_121512_object93593:head# 2024-04-29T14:26:35.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6041b38f:::benchmark_data_smithi194_121512_object112745:head# 2024-04-29T14:26:35.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6041b808:::benchmark_data_smithi194_121512_object54183:head# 2024-04-29T14:26:35.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6041db43:::benchmark_data_smithi194_121512_object70444:head# 2024-04-29T14:26:35.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60435e15:::benchmark_data_smithi194_121512_object86469:head# 2024-04-29T14:26:35.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6043c9a6:::benchmark_data_smithi194_121512_object111667:head# 2024-04-29T14:26:35.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6044063c:::benchmark_data_smithi194_121512_object57702:head# 2024-04-29T14:26:35.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604421ea:::benchmark_data_smithi194_121512_object5282:head# 2024-04-29T14:26:35.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60447130:::benchmark_data_smithi194_121512_object26528:head# 2024-04-29T14:26:35.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6044b7ca:::benchmark_data_smithi194_121512_object20582:head# 2024-04-29T14:26:35.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6044d09c:::benchmark_data_smithi194_121512_object24823:head# 2024-04-29T14:26:35.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6046e968:::benchmark_data_smithi194_121512_object18595:head# 2024-04-29T14:26:35.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604705a8:::benchmark_data_smithi194_121512_object83800:head# 2024-04-29T14:26:35.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6047a365:::benchmark_data_smithi194_121512_object40019:head# 2024-04-29T14:26:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6047b6af:::benchmark_data_smithi194_121512_object75783:head# 2024-04-29T14:26:35.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6047e9ef:::benchmark_data_smithi194_121512_object112262:head# 2024-04-29T14:26:35.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604867de:::benchmark_data_smithi194_121512_object115718:head# 2024-04-29T14:26:35.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6048f64f:::benchmark_data_smithi194_121512_object97939:head# 2024-04-29T14:26:35.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604945f0:::benchmark_data_smithi194_121512_object128660:head# 2024-04-29T14:26:35.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604980ce:::benchmark_data_smithi194_121512_object96784:head# 2024-04-29T14:26:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6049eb94:::benchmark_data_smithi194_121512_object53633:head# 2024-04-29T14:26:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604af895:::benchmark_data_smithi194_121512_object56009:head# 2024-04-29T14:26:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604cbb59:::benchmark_data_smithi194_121512_object9484:head# 2024-04-29T14:26:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604d3e45:::benchmark_data_smithi194_121512_object27996:head# 2024-04-29T14:26:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604f31a9:::benchmark_data_smithi194_121512_object133282:head# 2024-04-29T14:26:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:604f860b:::benchmark_data_smithi194_121512_object50887:head# 2024-04-29T14:26:35.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6050e208:::benchmark_data_smithi194_121512_object112247:head# 2024-04-29T14:26:35.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6051590b:::benchmark_data_smithi194_121512_object96296:head# 2024-04-29T14:26:35.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60515c45:::benchmark_data_smithi194_121512_object39152:head# 2024-04-29T14:26:35.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6051a015:::benchmark_data_smithi194_121512_object59297:head# 2024-04-29T14:26:35.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6051e268:::benchmark_data_smithi194_121512_object41487:head# 2024-04-29T14:26:35.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605200b5:::benchmark_data_smithi194_121512_object6914:head# 2024-04-29T14:26:35.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60524077:::benchmark_data_smithi194_121512_object71600:head# 2024-04-29T14:26:35.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60544b54:::benchmark_data_smithi194_121512_object39767:head# 2024-04-29T14:26:35.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605475e1:::benchmark_data_smithi194_121512_object132355:head# 2024-04-29T14:26:35.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6054e205:::benchmark_data_smithi194_121512_object58999:head# 2024-04-29T14:26:35.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60552f35:::benchmark_data_smithi194_121512_object34364:head# 2024-04-29T14:26:35.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6055adfd:::benchmark_data_smithi194_121512_object103207:head# 2024-04-29T14:26:35.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6055fe87:::benchmark_data_smithi194_121512_object49371:head# 2024-04-29T14:26:35.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60561dc9:::benchmark_data_smithi194_121512_object86843:head# 2024-04-29T14:26:35.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60574b9e:::benchmark_data_smithi194_121512_object100765:head# 2024-04-29T14:26:35.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60580be6:::benchmark_data_smithi194_121512_object111539:head# 2024-04-29T14:26:35.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60582677:::benchmark_data_smithi194_121512_object8966:head# 2024-04-29T14:26:35.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60582ad6:::benchmark_data_smithi194_121512_object31878:head# 2024-04-29T14:26:35.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6059302c:::benchmark_data_smithi194_121512_object38297:head# 2024-04-29T14:26:35.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605982e2:::benchmark_data_smithi194_121512_object93765:head# 2024-04-29T14:26:35.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6059ce4d:::benchmark_data_smithi194_121512_object125868:head# 2024-04-29T14:26:35.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605a0e20:::benchmark_data_smithi194_121512_object132055:head# 2024-04-29T14:26:35.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605a9ea8:::benchmark_data_smithi194_121512_object88173:head# 2024-04-29T14:26:35.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605c7658:::benchmark_data_smithi194_121512_object37338:head# 2024-04-29T14:26:35.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605c97ae:::benchmark_data_smithi194_121512_object64205:head# 2024-04-29T14:26:35.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605e26ff:::benchmark_data_smithi194_121512_object54518:head# 2024-04-29T14:26:35.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605efcff:::benchmark_data_smithi194_121512_object36828:head# 2024-04-29T14:26:35.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605f2f7a:::benchmark_data_smithi194_121512_object79755:head# 2024-04-29T14:26:35.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605f51ef:::benchmark_data_smithi194_121512_object80992:head# 2024-04-29T14:26:35.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:605f6f8b:::benchmark_data_smithi194_121512_object9487:head# 2024-04-29T14:26:35.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60608483:::benchmark_data_smithi194_121512_object19577:head# 2024-04-29T14:26:35.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606122a4:::benchmark_data_smithi194_121512_object74273:head# 2024-04-29T14:26:35.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60621065:::benchmark_data_smithi194_121512_object121899:head# 2024-04-29T14:26:35.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6062526e:::benchmark_data_smithi194_121512_object42753:head# 2024-04-29T14:26:35.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60635c3b:::benchmark_data_smithi194_121512_object45725:head# 2024-04-29T14:26:35.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60637a30:::benchmark_data_smithi194_121512_object132980:head# 2024-04-29T14:26:35.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6063863d:::benchmark_data_smithi194_121512_object122131:head# 2024-04-29T14:26:35.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6066000d:::benchmark_data_smithi194_121512_object112168:head# 2024-04-29T14:26:35.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606603e2:::benchmark_data_smithi194_121512_object34119:head# 2024-04-29T14:26:35.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60660805:::benchmark_data_smithi194_121512_object22698:head# 2024-04-29T14:26:35.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60663461:::benchmark_data_smithi194_121512_object111987:head# 2024-04-29T14:26:35.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60666ecc:::benchmark_data_smithi194_121512_object12343:head# 2024-04-29T14:26:35.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60667fe4:::benchmark_data_smithi194_121512_object11068:head# 2024-04-29T14:26:35.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6066b7e0:::benchmark_data_smithi194_121512_object7445:head# 2024-04-29T14:26:35.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6067681f:::benchmark_data_smithi194_121512_object34066:head# 2024-04-29T14:26:35.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60689bcc:::benchmark_data_smithi194_121512_object23027:head# 2024-04-29T14:26:35.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6068fb69:::benchmark_data_smithi194_121512_object886:head# 2024-04-29T14:26:35.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60696f25:::benchmark_data_smithi194_121512_object69329:head# 2024-04-29T14:26:35.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606a0156:::benchmark_data_smithi194_121512_object19231:head# 2024-04-29T14:26:35.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606af595:::benchmark_data_smithi194_121512_object72668:head# 2024-04-29T14:26:35.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606b3657:::benchmark_data_smithi194_121512_object64989:head# 2024-04-29T14:26:35.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606bb67a:::benchmark_data_smithi194_121512_object35792:head# 2024-04-29T14:26:35.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606bcf39:::benchmark_data_smithi194_121512_object27123:head# 2024-04-29T14:26:35.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606c4cc0:::benchmark_data_smithi194_121512_object97670:head# 2024-04-29T14:26:35.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606ca393:::benchmark_data_smithi194_121512_object111275:head# 2024-04-29T14:26:35.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606e0ef5:::benchmark_data_smithi194_121512_object3281:head# 2024-04-29T14:26:35.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606e61b2:::benchmark_data_smithi194_121512_object9681:head# 2024-04-29T14:26:35.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606e7d06:::benchmark_data_smithi194_121512_object13414:head# 2024-04-29T14:26:35.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606ee524:::benchmark_data_smithi194_121512_object39327:head# 2024-04-29T14:26:35.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606fc9f3:::benchmark_data_smithi194_121512_object128751:head# 2024-04-29T14:26:35.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:606ff705:::benchmark_data_smithi194_121512_object76578:head# 2024-04-29T14:26:35.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60723580:::benchmark_data_smithi194_121512_object84677:head# 2024-04-29T14:26:35.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6072d363:::benchmark_data_smithi194_121512_object101476:head# 2024-04-29T14:26:35.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6073cf87:::benchmark_data_smithi194_121512_object94190:head# 2024-04-29T14:26:35.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6075321a:::benchmark_data_smithi194_121512_object99956:head# 2024-04-29T14:26:35.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60757b1f:::benchmark_data_smithi194_121512_object69803:head# 2024-04-29T14:26:35.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6075ea7f:::benchmark_data_smithi194_121512_object88628:head# 2024-04-29T14:26:35.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607635d5:::benchmark_data_smithi194_121512_object5326:head# 2024-04-29T14:26:35.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60764c21:::benchmark_data_smithi194_121512_object101391:head# 2024-04-29T14:26:35.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60765280:::benchmark_data_smithi194_121512_object88964:head# 2024-04-29T14:26:35.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6076cfa0:::benchmark_data_smithi194_121512_object88105:head# 2024-04-29T14:26:35.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6077c843:::benchmark_data_smithi194_121512_object124913:head# 2024-04-29T14:26:35.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6077d5d0:::benchmark_data_smithi194_121512_object28624:head# 2024-04-29T14:26:35.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607838a6:::benchmark_data_smithi194_121512_object73055:head# 2024-04-29T14:26:35.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60787bda:::benchmark_data_smithi194_121512_object73393:head# 2024-04-29T14:26:35.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607896e6:::benchmark_data_smithi194_121512_object95767:head# 2024-04-29T14:26:35.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60797acc:::benchmark_data_smithi194_121512_object110:head# 2024-04-29T14:26:35.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60799bf7:::benchmark_data_smithi194_121512_object107242:head# 2024-04-29T14:26:35.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6079b9cc:::benchmark_data_smithi194_121512_object65544:head# 2024-04-29T14:26:35.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607c077c:::benchmark_data_smithi194_121512_object21800:head# 2024-04-29T14:26:35.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607c2df2:::benchmark_data_smithi194_121512_object57569:head# 2024-04-29T14:26:35.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607dceb2:::benchmark_data_smithi194_121512_object10370:head# 2024-04-29T14:26:35.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607ddcc2:::benchmark_data_smithi194_121512_object46122:head# 2024-04-29T14:26:35.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607e2473:::benchmark_data_smithi194_121512_object60450:head# 2024-04-29T14:26:35.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607e3c56:::benchmark_data_smithi194_121512_object89803:head# 2024-04-29T14:26:35.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607e6c20:::benchmark_data_smithi194_121512_object132589:head# 2024-04-29T14:26:35.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607e86cc:::benchmark_data_smithi194_121512_object97517:head# 2024-04-29T14:26:35.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607e9bd8:::benchmark_data_smithi194_121512_object76177:head# 2024-04-29T14:26:35.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607ed2bb:::benchmark_data_smithi194_121512_object31519:head# 2024-04-29T14:26:35.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607f19e0:::benchmark_data_smithi194_121512_object10647:head# 2024-04-29T14:26:35.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:607fc49e:::benchmark_data_smithi194_121512_object113584:head# 2024-04-29T14:26:35.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60801edb:::benchmark_data_smithi194_121512_object116689:head# 2024-04-29T14:26:35.433 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:35.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60804bf6:::benchmark_data_smithi194_121512_object73633:head# 2024-04-29T14:26:35.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608104a8:::benchmark_data_smithi194_121512_object69597:head# 2024-04-29T14:26:35.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60821635:::benchmark_data_smithi194_121512_object75898:head# 2024-04-29T14:26:35.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60826d68:::benchmark_data_smithi194_121512_object128742:head# 2024-04-29T14:26:35.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6084d5ae:::benchmark_data_smithi194_121512_object30651:head# 2024-04-29T14:26:35.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6085087b:::benchmark_data_smithi194_121512_object125968:head# 2024-04-29T14:26:35.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6085552e:::benchmark_data_smithi194_121512_object102648:head# 2024-04-29T14:26:35.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6085a8b8:::benchmark_data_smithi194_121512_object126029:head# 2024-04-29T14:26:35.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60867899:::benchmark_data_smithi194_121512_object36139:head# 2024-04-29T14:26:35.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6086caf8:::benchmark_data_smithi194_121512_object82748:head# 2024-04-29T14:26:35.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6087aa2f:::benchmark_data_smithi194_121512_object31751:head# 2024-04-29T14:26:35.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6087f548:::benchmark_data_smithi194_121512_object41768:head# 2024-04-29T14:26:35.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60883fb2:::benchmark_data_smithi194_121512_object87778:head# 2024-04-29T14:26:35.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608871af:::benchmark_data_smithi194_121512_object103356:head# 2024-04-29T14:26:35.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6089b311:::benchmark_data_smithi194_121512_object36147:head# 2024-04-29T14:26:35.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608a2678:::benchmark_data_smithi194_121512_object19190:head# 2024-04-29T14:26:35.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608a300f:::benchmark_data_smithi194_121512_object6194:head# 2024-04-29T14:26:35.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608a98d0:::benchmark_data_smithi194_121512_object128652:head# 2024-04-29T14:26:35.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608afc43:::benchmark_data_smithi194_121512_object83038:head# 2024-04-29T14:26:35.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608bb840:::benchmark_data_smithi194_121512_object84213:head# 2024-04-29T14:26:35.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608bfe14:::benchmark_data_smithi194_121512_object65284:head# 2024-04-29T14:26:35.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608d3ae3:::benchmark_data_smithi194_121512_object41449:head# 2024-04-29T14:26:35.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608d7d90:::benchmark_data_smithi194_121512_object10306:head# 2024-04-29T14:26:35.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608d960b:::benchmark_data_smithi194_121512_object63596:head# 2024-04-29T14:26:35.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608dd361:::benchmark_data_smithi194_121512_object71515:head# 2024-04-29T14:26:35.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608e434a:::benchmark_data_smithi194_121512_object80837:head# 2024-04-29T14:26:35.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608e7e5b:::benchmark_data_smithi194_121512_object92306:head# 2024-04-29T14:26:35.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608e8f83:::benchmark_data_smithi194_121512_object56226:head# 2024-04-29T14:26:35.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608ec516:::benchmark_data_smithi194_121512_object69324:head# 2024-04-29T14:26:35.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608ed8f9:::benchmark_data_smithi194_121512_object11387:head# 2024-04-29T14:26:35.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608ee51c:::benchmark_data_smithi194_121512_object52102:head# 2024-04-29T14:26:35.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608f1fe5:::benchmark_data_smithi194_121512_object65216:head# 2024-04-29T14:26:35.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:608f95d0:::benchmark_data_smithi194_121512_object67478:head# 2024-04-29T14:26:35.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609043e4:::benchmark_data_smithi194_121512_object54825:head# 2024-04-29T14:26:35.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6090f9ad:::benchmark_data_smithi194_121512_object22305:head# 2024-04-29T14:26:35.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6091034b:::benchmark_data_smithi194_121512_object37519:head# 2024-04-29T14:26:35.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60919788:::benchmark_data_smithi194_121512_object102808:head# 2024-04-29T14:26:35.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60932e35:::benchmark_data_smithi194_121512_object52685:head# 2024-04-29T14:26:35.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609334f9:::benchmark_data_smithi194_121512_object106769:head# 2024-04-29T14:26:35.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609344b1:::benchmark_data_smithi194_121512_object60733:head# 2024-04-29T14:26:35.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6094b845:::benchmark_data_smithi194_121512_object91111:head# 2024-04-29T14:26:35.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609521c4:::benchmark_data_smithi194_121512_object77205:head# 2024-04-29T14:26:35.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60953afe:::benchmark_data_smithi194_121512_object12218:head# 2024-04-29T14:26:35.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6095b424:::benchmark_data_smithi194_121512_object15630:head# 2024-04-29T14:26:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609635d5:::benchmark_data_smithi194_121512_object129136:head# 2024-04-29T14:26:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6097ea13:::benchmark_data_smithi194_121512_object61313:head# 2024-04-29T14:26:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6098099d:::benchmark_data_smithi194_121512_object47701:head# 2024-04-29T14:26:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6098a99a:::benchmark_data_smithi194_121512_object96672:head# 2024-04-29T14:26:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60994443:::benchmark_data_smithi194_121512_object129903:head# 2024-04-29T14:26:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609a1198:::benchmark_data_smithi194_121512_object43937:head# 2024-04-29T14:26:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609abed9:::benchmark_data_smithi194_121512_object115552:head# 2024-04-29T14:26:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609b240f:::benchmark_data_smithi194_121512_object53626:head# 2024-04-29T14:26:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609b25eb:::benchmark_data_smithi194_121512_object121345:head# 2024-04-29T14:26:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609b948d:::benchmark_data_smithi194_121512_object17338:head# 2024-04-29T14:26:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609ec101:::benchmark_data_smithi194_121512_object121347:head# 2024-04-29T14:26:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609f5013:::benchmark_data_smithi194_121512_object73837:head# 2024-04-29T14:26:35.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:609fa3d3:::benchmark_data_smithi194_121512_object36024:head# 2024-04-29T14:26:35.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a04623:::benchmark_data_smithi194_121512_object50852:head# 2024-04-29T14:26:35.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a07e97:::benchmark_data_smithi194_121512_object93792:head# 2024-04-29T14:26:35.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a11cbf:::benchmark_data_smithi194_121512_object124190:head# 2024-04-29T14:26:35.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a2437a:::benchmark_data_smithi194_121512_object67561:head# 2024-04-29T14:26:35.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a268a3:::benchmark_data_smithi194_121512_object1457:head# 2024-04-29T14:26:35.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a417d0:::benchmark_data_smithi194_121512_object132155:head# 2024-04-29T14:26:35.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a4afbe:::benchmark_data_smithi194_121512_object37181:head# 2024-04-29T14:26:35.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a4b237:::benchmark_data_smithi194_121512_object46713:head# 2024-04-29T14:26:35.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a4c2e6:::benchmark_data_smithi194_121512_object96480:head# 2024-04-29T14:26:35.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a4ff0b:::benchmark_data_smithi194_121512_object12190:head# 2024-04-29T14:26:35.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a56505:::benchmark_data_smithi194_121512_object53657:head# 2024-04-29T14:26:35.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a569ca:::benchmark_data_smithi194_121512_object66930:head# 2024-04-29T14:26:35.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a56d3c:::benchmark_data_smithi194_121512_object115228:head# 2024-04-29T14:26:35.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a62d36:::benchmark_data_smithi194_121512_object30991:head# 2024-04-29T14:26:35.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a75cfd:::benchmark_data_smithi194_121512_object66192:head# 2024-04-29T14:26:35.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a8c470:::benchmark_data_smithi194_121512_object15633:head# 2024-04-29T14:26:35.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a8cc48:::benchmark_data_smithi194_121512_object7702:head# 2024-04-29T14:26:35.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a90615:::benchmark_data_smithi194_121512_object62903:head# 2024-04-29T14:26:35.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a9643c:::benchmark_data_smithi194_121512_object116764:head# 2024-04-29T14:26:35.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60a9c7f3:::benchmark_data_smithi194_121512_object129354:head# 2024-04-29T14:26:35.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60aa8907:::benchmark_data_smithi194_121512_object54483:head# 2024-04-29T14:26:35.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60aaa317:::benchmark_data_smithi194_121512_object50975:head# 2024-04-29T14:26:35.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ab75db:::benchmark_data_smithi194_121512_object131069:head# 2024-04-29T14:26:35.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ab7620:::benchmark_data_smithi194_121512_object105773:head# 2024-04-29T14:26:35.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60abb941:::benchmark_data_smithi194_121512_object107828:head# 2024-04-29T14:26:35.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ac0dbc:::benchmark_data_smithi194_121512_object44139:head# 2024-04-29T14:26:35.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ac1069:::benchmark_data_smithi194_121512_object99299:head# 2024-04-29T14:26:35.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60acf79f:::benchmark_data_smithi194_121512_object111291:head# 2024-04-29T14:26:35.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ad11b7:::benchmark_data_smithi194_121512_object24444:head# 2024-04-29T14:26:35.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ad5bbd:::benchmark_data_smithi194_121512_object63878:head# 2024-04-29T14:26:35.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ade9c6:::benchmark_data_smithi194_121512_object18443:head# 2024-04-29T14:26:35.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60adeefb:::benchmark_data_smithi194_121512_object33712:head# 2024-04-29T14:26:35.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ae2146:::benchmark_data_smithi194_121512_object37411:head# 2024-04-29T14:26:35.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60aec211:::benchmark_data_smithi194_121512_object50226:head# 2024-04-29T14:26:35.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60aecb83:::benchmark_data_smithi194_121512_object85208:head# 2024-04-29T14:26:35.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b0779a:::benchmark_data_smithi194_121512_object104865:head# 2024-04-29T14:26:35.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b08c60:::benchmark_data_smithi194_121512_object19287:head# 2024-04-29T14:26:35.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b09549:::benchmark_data_smithi194_121512_object690:head# 2024-04-29T14:26:35.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b18f5f:::benchmark_data_smithi194_121512_object55752:head# 2024-04-29T14:26:35.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b1be2f:::benchmark_data_smithi194_121512_object45034:head# 2024-04-29T14:26:35.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b1beb5:::benchmark_data_smithi194_121512_object64371:head# 2024-04-29T14:26:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b249c4:::benchmark_data_smithi194_121512_object106183:head# 2024-04-29T14:26:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b33aa1:::benchmark_data_smithi194_121512_object111566:head# 2024-04-29T14:26:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b3c403:::benchmark_data_smithi194_121512_object23702:head# 2024-04-29T14:26:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b422a3:::benchmark_data_smithi194_121512_object16322:head# 2024-04-29T14:26:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b4ce85:::benchmark_data_smithi194_121512_object24988:head# 2024-04-29T14:26:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b4d3ee:::benchmark_data_smithi194_121512_object31443:head# 2024-04-29T14:26:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b5c1a4:::benchmark_data_smithi194_121512_object115427:head# 2024-04-29T14:26:35.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b6170c:::benchmark_data_smithi194_121512_object113175:head# 2024-04-29T14:26:35.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b74204:::benchmark_data_smithi194_121512_object80898:head# 2024-04-29T14:26:35.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b74f8c:::benchmark_data_smithi194_121512_object124044:head# 2024-04-29T14:26:35.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b75074:::benchmark_data_smithi194_121512_object89451:head# 2024-04-29T14:26:35.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b78f13:::benchmark_data_smithi194_121512_object42287:head# 2024-04-29T14:26:35.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b80cc7:::benchmark_data_smithi194_121512_object81673:head# 2024-04-29T14:26:35.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b81135:::benchmark_data_smithi194_121512_object62668:head# 2024-04-29T14:26:35.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60b86d79:::benchmark_data_smithi194_121512_object3539:head# 2024-04-29T14:26:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ba8428:::benchmark_data_smithi194_121512_object60966:head# 2024-04-29T14:26:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bb1d0d:::benchmark_data_smithi194_121512_object26789:head# 2024-04-29T14:26:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bb2084:::benchmark_data_smithi194_121512_object32061:head# 2024-04-29T14:26:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bbc925:::benchmark_data_smithi194_121512_object61062:head# 2024-04-29T14:26:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bd5bd8:::benchmark_data_smithi194_121512_object75508:head# 2024-04-29T14:26:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bdcbb9:::benchmark_data_smithi194_121512_object11538:head# 2024-04-29T14:26:35.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bdcc77:::benchmark_data_smithi194_121512_object81359:head# 2024-04-29T14:26:35.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bddcd6:::benchmark_data_smithi194_121512_object56711:head# 2024-04-29T14:26:35.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bebe3d:::benchmark_data_smithi194_121512_object42791:head# 2024-04-29T14:26:35.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bf0ba1:::benchmark_data_smithi194_121512_object132185:head# 2024-04-29T14:26:35.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60bf8743:::benchmark_data_smithi194_121512_object78357:head# 2024-04-29T14:26:35.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c0af7e:::benchmark_data_smithi194_121512_object133031:head# 2024-04-29T14:26:35.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c21e3c:::benchmark_data_smithi194_121512_object80922:head# 2024-04-29T14:26:35.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c25b60:::benchmark_data_smithi194_121512_object130003:head# 2024-04-29T14:26:35.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c2f22f:::benchmark_data_smithi194_121512_object110964:head# 2024-04-29T14:26:35.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c33d2e:::benchmark_data_smithi194_121512_object96338:head# 2024-04-29T14:26:35.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c355e8:::benchmark_data_smithi194_121512_object1331:head# 2024-04-29T14:26:35.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c3d6f3:::benchmark_data_smithi194_121512_object88922:head# 2024-04-29T14:26:35.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c3d764:::benchmark_data_smithi194_121512_object51047:head# 2024-04-29T14:26:35.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c3e6bb:::benchmark_data_smithi194_121512_object114700:head# 2024-04-29T14:26:35.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c40a42:::benchmark_data_smithi194_121512_object108428:head# 2024-04-29T14:26:35.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c41f99:::benchmark_data_smithi194_121512_object74100:head# 2024-04-29T14:26:35.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c4f256:::benchmark_data_smithi194_121512_object119679:head# 2024-04-29T14:26:35.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c5484c:::benchmark_data_smithi194_121512_object53722:head# 2024-04-29T14:26:35.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c5cc06:::benchmark_data_smithi194_121512_object94195:head# 2024-04-29T14:26:35.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c6017b:::benchmark_data_smithi194_121512_object24857:head# 2024-04-29T14:26:35.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c6ec09:::benchmark_data_smithi194_121512_object91058:head# 2024-04-29T14:26:35.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c6f62d:::benchmark_data_smithi194_121512_object62365:head# 2024-04-29T14:26:35.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c73f6d:::benchmark_data_smithi194_121512_object104135:head# 2024-04-29T14:26:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c74298:::benchmark_data_smithi194_121512_object46006:head# 2024-04-29T14:26:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c7bf57:::benchmark_data_smithi194_121512_object67524:head# 2024-04-29T14:26:35.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c850d5:::benchmark_data_smithi194_121512_object11139:head# 2024-04-29T14:26:35.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c8d6d9:::benchmark_data_smithi194_121512_object84664:head# 2024-04-29T14:26:35.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c90b8c:::benchmark_data_smithi194_121512_object100997:head# 2024-04-29T14:26:35.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c92c1d:::benchmark_data_smithi194_121512_object22693:head# 2024-04-29T14:26:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c9768b:::benchmark_data_smithi194_121512_object58976:head# 2024-04-29T14:26:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60c9e60f:::benchmark_data_smithi194_121512_object94953:head# 2024-04-29T14:26:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ca6f3a:::benchmark_data_smithi194_121512_object88467:head# 2024-04-29T14:26:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cb3b6d:::benchmark_data_smithi194_121512_object118373:head# 2024-04-29T14:26:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cb92d8:::benchmark_data_smithi194_121512_object60850:head# 2024-04-29T14:26:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cbe3e0:::benchmark_data_smithi194_121512_object49558:head# 2024-04-29T14:26:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cc7157:::benchmark_data_smithi194_121512_object59042:head# 2024-04-29T14:26:35.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cc990a:::benchmark_data_smithi194_121512_object127827:head# 2024-04-29T14:26:35.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cdbc6b:::benchmark_data_smithi194_121512_object68:head# 2024-04-29T14:26:35.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cdbf44:::benchmark_data_smithi194_121512_object105369:head# 2024-04-29T14:26:35.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ce1adb:::benchmark_data_smithi194_121512_object127354:head# 2024-04-29T14:26:35.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ce1e60:::benchmark_data_smithi194_121512_object13938:head# 2024-04-29T14:26:35.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ce9322:::benchmark_data_smithi194_121512_object48649:head# 2024-04-29T14:26:35.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ceb8de:::benchmark_data_smithi194_121512_object30072:head# 2024-04-29T14:26:35.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cee45f:::benchmark_data_smithi194_121512_object126619:head# 2024-04-29T14:26:35.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cfa323:::benchmark_data_smithi194_121512_object43404:head# 2024-04-29T14:26:35.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60cfc8ca:::benchmark_data_smithi194_121512_object104285:head# 2024-04-29T14:26:35.526 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d0cfed:::benchmark_data_smithi194_121512_object119499:head# 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d15d14:::benchmark_data_smithi194_121512_object10613:head# 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d20aca:::benchmark_data_smithi194_121512_object39205:head# 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d28f26:::benchmark_data_smithi194_121512_object15758:head# 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d458d1:::benchmark_data_smithi194_121512_object125663:head# 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d4e082:::benchmark_data_smithi194_121512_object125347:head# 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d5147c:::benchmark_data_smithi194_121512_object80711:head# 2024-04-29T14:26:35.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d54066:::benchmark_data_smithi194_121512_object11755:head# 2024-04-29T14:26:35.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d54a09:::benchmark_data_smithi194_121512_object102854:head# 2024-04-29T14:26:35.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d58919:::benchmark_data_smithi194_121512_object78815:head# 2024-04-29T14:26:35.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d595ab:::benchmark_data_smithi194_121512_object131982:head# 2024-04-29T14:26:35.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d5e118:::benchmark_data_smithi194_121512_object110715:head# 2024-04-29T14:26:35.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d62005:::benchmark_data_smithi194_121512_object132506:head# 2024-04-29T14:26:35.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d6373f:::benchmark_data_smithi194_121512_object94840:head# 2024-04-29T14:26:35.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d6b2a0:::benchmark_data_smithi194_121512_object79297:head# 2024-04-29T14:26:35.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d72625:::benchmark_data_smithi194_121512_object112573:head# 2024-04-29T14:26:35.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d735c7:::benchmark_data_smithi194_121512_object51502:head# 2024-04-29T14:26:35.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d739c8:::benchmark_data_smithi194_121512_object60381:head# 2024-04-29T14:26:35.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d86fcc:::benchmark_data_smithi194_121512_object74868:head# 2024-04-29T14:26:35.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d88b7c:::benchmark_data_smithi194_121512_object75637:head# 2024-04-29T14:26:35.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60d8aa71:::benchmark_data_smithi194_121512_object12134:head# 2024-04-29T14:26:35.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60da45ad:::benchmark_data_smithi194_121512_object121500:head# 2024-04-29T14:26:35.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60da52ea:::benchmark_data_smithi194_121512_object51371:head# 2024-04-29T14:26:35.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60db8d5c:::benchmark_data_smithi194_121512_object56539:head# 2024-04-29T14:26:35.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60dce35e:::benchmark_data_smithi194_121512_object89912:head# 2024-04-29T14:26:35.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60dce6e8:::benchmark_data_smithi194_121512_object74294:head# 2024-04-29T14:26:35.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60dd26e5:::benchmark_data_smithi194_121512_object113631:head# 2024-04-29T14:26:35.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60dddcb5:::benchmark_data_smithi194_121512_object5521:head# 2024-04-29T14:26:35.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60de355a:::benchmark_data_smithi194_121512_object96589:head# 2024-04-29T14:26:35.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60de79fa:::benchmark_data_smithi194_121512_object113256:head# 2024-04-29T14:26:35.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60dfa9ec:::benchmark_data_smithi194_121512_object69093:head# 2024-04-29T14:26:35.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60dfb8c5:::benchmark_data_smithi194_121512_object112606:head# 2024-04-29T14:26:35.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e035f2:::benchmark_data_smithi194_121512_object101635:head# 2024-04-29T14:26:35.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e0a073:::benchmark_data_smithi194_121512_object8919:head# 2024-04-29T14:26:35.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e0c09c:::benchmark_data_smithi194_121512_object101776:head# 2024-04-29T14:26:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e0ee7f:::benchmark_data_smithi194_121512_object81208:head# 2024-04-29T14:26:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e24203:::benchmark_data_smithi194_121512_object68004:head# 2024-04-29T14:26:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e29b8e:::benchmark_data_smithi194_121512_object98492:head# 2024-04-29T14:26:35.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e29bba:::benchmark_data_smithi194_121512_object22849:head# 2024-04-29T14:26:35.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e2f27f:::benchmark_data_smithi194_121512_object42392:head# 2024-04-29T14:26:35.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e3bf9b:::benchmark_data_smithi194_121512_object117278:head# 2024-04-29T14:26:35.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e3f8a8:::benchmark_data_smithi194_121512_object103729:head# 2024-04-29T14:26:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e437e3:::benchmark_data_smithi194_121512_object7150:head# 2024-04-29T14:26:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e45090:::benchmark_data_smithi194_121512_object110136:head# 2024-04-29T14:26:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e5349d:::benchmark_data_smithi194_121512_object52668:head# 2024-04-29T14:26:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e54b2e:::benchmark_data_smithi194_121512_object112296:head# 2024-04-29T14:26:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e649e1:::benchmark_data_smithi194_121512_object111519:head# 2024-04-29T14:26:35.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e656b3:::benchmark_data_smithi194_121512_object47233:head# 2024-04-29T14:26:35.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e66cc3:::benchmark_data_smithi194_121512_object39247:head# 2024-04-29T14:26:35.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e75c5d:::benchmark_data_smithi194_121512_object89066:head# 2024-04-29T14:26:35.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e8b580:::benchmark_data_smithi194_121512_object47921:head# 2024-04-29T14:26:35.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e8c5b9:::benchmark_data_smithi194_121512_object131832:head# 2024-04-29T14:26:35.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60e95542:::benchmark_data_smithi194_121512_object9592:head# 2024-04-29T14:26:35.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ec164c:::benchmark_data_smithi194_121512_object46746:head# 2024-04-29T14:26:35.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ec9bf5:::benchmark_data_smithi194_121512_object53821:head# 2024-04-29T14:26:35.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ecb6d4:::benchmark_data_smithi194_121512_object48747:head# 2024-04-29T14:26:35.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60edc851:::benchmark_data_smithi194_121512_object37901:head# 2024-04-29T14:26:35.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ee3549:::benchmark_data_smithi194_121512_object48201:head# 2024-04-29T14:26:35.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ee5f44:::benchmark_data_smithi194_121512_object68250:head# 2024-04-29T14:26:35.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60eec1c3:::benchmark_data_smithi194_121512_object62051:head# 2024-04-29T14:26:35.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60efb8a3:::benchmark_data_smithi194_121512_object104566:head# 2024-04-29T14:26:35.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f03709:::benchmark_data_smithi194_121512_object46596:head# 2024-04-29T14:26:35.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f1be93:::benchmark_data_smithi194_121512_object112832:head# 2024-04-29T14:26:35.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f2993a:::benchmark_data_smithi194_121512_object21405:head# 2024-04-29T14:26:35.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f2ffda:::benchmark_data_smithi194_121512_object37070:head# 2024-04-29T14:26:35.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f3348e:::benchmark_data_smithi194_121512_object111320:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f3bb70:::benchmark_data_smithi194_121512_object34623:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f4a4d6:::benchmark_data_smithi194_121512_object82057:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f6d0c2:::benchmark_data_smithi194_121512_object16046:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f7ca46:::benchmark_data_smithi194_121512_object21828:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f7ec03:::benchmark_data_smithi194_121512_object95954:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f90377:::benchmark_data_smithi194_121512_object124476:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f94c6c:::benchmark_data_smithi194_121512_object44998:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f96a6d:::benchmark_data_smithi194_121512_object12891:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f9a57c:::benchmark_data_smithi194_121512_object28598:head# 2024-04-29T14:26:35.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60f9c802:::benchmark_data_smithi194_121512_object3644:head# 2024-04-29T14:26:35.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60fb3a2a:::benchmark_data_smithi194_121512_object123553:head# 2024-04-29T14:26:35.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60fc4bf1:::benchmark_data_smithi194_121512_object98672:head# 2024-04-29T14:26:35.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60fc5ef0:::benchmark_data_smithi194_121512_object10813:head# 2024-04-29T14:26:35.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60fca358:::benchmark_data_smithi194_121512_object78787:head# 2024-04-29T14:26:35.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60fd1648:::benchmark_data_smithi194_121512_object21496:head# 2024-04-29T14:26:35.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ff88ad:::benchmark_data_smithi194_121512_object91468:head# 2024-04-29T14:26:35.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:60ffa60e:::benchmark_data_smithi194_121512_object85399:head# 2024-04-29T14:26:35.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6100353a:::benchmark_data_smithi194_121512_object16390:head# 2024-04-29T14:26:35.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61007484:::benchmark_data_smithi194_121512_object92209:head# 2024-04-29T14:26:35.564 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6100bb1f:::benchmark_data_smithi194_121512_object132914:head# 2024-04-29T14:26:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6100eb9d:::benchmark_data_smithi194_121512_object77891:head# 2024-04-29T14:26:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610110e1:::benchmark_data_smithi194_121512_object120957:head# 2024-04-29T14:26:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610195ce:::benchmark_data_smithi194_121512_object44718:head# 2024-04-29T14:26:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6101b1b7:::benchmark_data_smithi194_121512_object77682:head# 2024-04-29T14:26:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6101eddd:::benchmark_data_smithi194_121512_object101080:head# 2024-04-29T14:26:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6102077c:::benchmark_data_smithi194_121512_object77101:head# 2024-04-29T14:26:35.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610324d8:::benchmark_data_smithi194_121512_object82826:head# 2024-04-29T14:26:35.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6105a913:::benchmark_data_smithi194_121512_object9072:head# 2024-04-29T14:26:35.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6106cd5c:::benchmark_data_smithi194_121512_object70704:head# 2024-04-29T14:26:35.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6106d661:::benchmark_data_smithi194_121512_object35550:head# 2024-04-29T14:26:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:35.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61075f77:::benchmark_data_smithi194_121512_object118233:head# 2024-04-29T14:26:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61078eb7:::benchmark_data_smithi194_121512_object30054:head# 2024-04-29T14:26:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6107bd22:::benchmark_data_smithi194_121512_object126674:head# 2024-04-29T14:26:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6107eb09:::benchmark_data_smithi194_121512_object57356:head# 2024-04-29T14:26:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610a590f:::benchmark_data_smithi194_121512_object132899:head# 2024-04-29T14:26:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610a6fc5:::benchmark_data_smithi194_121512_object20101:head# 2024-04-29T14:26:35.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610ab1f0:::benchmark_data_smithi194_121512_object91995:head# 2024-04-29T14:26:35.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610b320c:::benchmark_data_smithi194_121512_object36732:head# 2024-04-29T14:26:35.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610b6688:::benchmark_data_smithi194_121512_object64502:head# 2024-04-29T14:26:35.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610b6990:::benchmark_data_smithi194_121512_object95387:head# 2024-04-29T14:26:35.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610c552b:::benchmark_data_smithi194_121512_object90316:head# 2024-04-29T14:26:35.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610c99ae:::benchmark_data_smithi194_121512_object41194:head# 2024-04-29T14:26:35.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610ccb43:::benchmark_data_smithi194_121512_object40440:head# 2024-04-29T14:26:35.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610d408f:::benchmark_data_smithi194_121512_object129130:head# 2024-04-29T14:26:35.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610dd529:::benchmark_data_smithi194_121512_object109054:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610de88d:::benchmark_data_smithi194_121512_object7931:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610f097d:::benchmark_data_smithi194_121512_object82606:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:610ffeaf:::benchmark_data_smithi194_121512_object80399:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61100468:::benchmark_data_smithi194_121512_object100117:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61101782:::benchmark_data_smithi194_121512_object118683:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6110cbb3:::benchmark_data_smithi194_121512_object72558:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611170dc:::benchmark_data_smithi194_121512_object99137:head# 2024-04-29T14:26:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6111fc40:::benchmark_data_smithi194_121512_object42484:head# 2024-04-29T14:26:35.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6111fe56:::benchmark_data_smithi194_121512_object80844:head# 2024-04-29T14:26:35.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611202fa:::benchmark_data_smithi194_121512_object52623:head# 2024-04-29T14:26:35.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6112790c:::benchmark_data_smithi194_121512_object14411:head# 2024-04-29T14:26:35.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61136a95:::benchmark_data_smithi194_121512_object41507:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61138fb3:::benchmark_data_smithi194_121512_object20504:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6113d9bd:::benchmark_data_smithi194_121512_object13829:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611409ce:::benchmark_data_smithi194_121512_object39926:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61149ca4:::benchmark_data_smithi194_121512_object1758:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6114d68c:::benchmark_data_smithi194_121512_object92409:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6114fff3:::benchmark_data_smithi194_121512_object55941:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6115a044:::benchmark_data_smithi194_121512_object21978:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6116aa05:::benchmark_data_smithi194_121512_object122666:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61170103:::benchmark_data_smithi194_121512_object19394:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61173395:::benchmark_data_smithi194_121512_object14194:head# 2024-04-29T14:26:35.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61173462:::benchmark_data_smithi194_121512_object75543:head# 2024-04-29T14:26:35.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61179624:::benchmark_data_smithi194_121512_object86856:head# 2024-04-29T14:26:35.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611a7375:::benchmark_data_smithi194_121512_object124669:head# 2024-04-29T14:26:35.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611b0a68:::benchmark_data_smithi194_121512_object70261:head# 2024-04-29T14:26:35.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611c6e74:::benchmark_data_smithi194_121512_object82342:head# 2024-04-29T14:26:35.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611d52d5:::benchmark_data_smithi194_121512_object57793:head# 2024-04-29T14:26:35.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611ddcda:::benchmark_data_smithi194_121512_object19552:head# 2024-04-29T14:26:35.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611dfd24:::benchmark_data_smithi194_121512_object7663:head# 2024-04-29T14:26:35.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611ea034:::benchmark_data_smithi194_121512_object111968:head# 2024-04-29T14:26:35.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611f15a0:::benchmark_data_smithi194_121512_object117143:head# 2024-04-29T14:26:35.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:611ff2cc:::benchmark_data_smithi194_121512_object73612:head# 2024-04-29T14:26:35.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612016dd:::benchmark_data_smithi194_121512_object37858:head# 2024-04-29T14:26:35.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612075fe:::benchmark_data_smithi194_121512_object128493:head# 2024-04-29T14:26:35.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6120e0ef:::benchmark_data_smithi194_121512_object107949:head# 2024-04-29T14:26:35.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6121d48a:::benchmark_data_smithi194_121512_object34015:head# 2024-04-29T14:26:35.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61222f94:::benchmark_data_smithi194_121512_object82019:head# 2024-04-29T14:26:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6122427b:::benchmark_data_smithi194_121512_object27969:head# 2024-04-29T14:26:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61226bb1:::benchmark_data_smithi194_121512_object47490:head# 2024-04-29T14:26:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61232455:::benchmark_data_smithi194_121512_object50751:head# 2024-04-29T14:26:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61232b86:::benchmark_data_smithi194_121512_object13914:head# 2024-04-29T14:26:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6123524d:::benchmark_data_smithi194_121512_object112196:head# 2024-04-29T14:26:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6123c6a6:::benchmark_data_smithi194_121512_object55883:head# 2024-04-29T14:26:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6123cf4c:::benchmark_data_smithi194_121512_object2351:head# 2024-04-29T14:26:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6123f33b:::benchmark_data_smithi194_121512_object43916:head# 2024-04-29T14:26:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61241349:::benchmark_data_smithi194_121512_object12081:head# 2024-04-29T14:26:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61241bb3:::benchmark_data_smithi194_121512_object32541:head# 2024-04-29T14:26:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6124b2a6:::benchmark_data_smithi194_121512_object15181:head# 2024-04-29T14:26:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61254977:::benchmark_data_smithi194_121512_object67452:head# 2024-04-29T14:26:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6125a0c6:::benchmark_data_smithi194_121512_object65286:head# 2024-04-29T14:26:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6125dfd7:::benchmark_data_smithi194_121512_object32518:head# 2024-04-29T14:26:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61263995:::benchmark_data_smithi194_121512_object127156:head# 2024-04-29T14:26:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612693b7:::benchmark_data_smithi194_121512_object121066:head# 2024-04-29T14:26:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6127c10f:::benchmark_data_smithi194_121512_object123895:head# 2024-04-29T14:26:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612811a2:::benchmark_data_smithi194_121512_object25739:head# 2024-04-29T14:26:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61281ef5:::benchmark_data_smithi194_121512_object92550:head# 2024-04-29T14:26:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61284199:::benchmark_data_smithi194_121512_object65826:head# 2024-04-29T14:26:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6129f9ea:::benchmark_data_smithi194_121512_object34371:head# 2024-04-29T14:26:35.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612a961e:::benchmark_data_smithi194_121512_object48110:head# 2024-04-29T14:26:35.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612ae5ed:::benchmark_data_smithi194_121512_object7009:head# 2024-04-29T14:26:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612b0401:::benchmark_data_smithi194_121512_object83304:head# 2024-04-29T14:26:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612b7e01:::benchmark_data_smithi194_121512_object9927:head# 2024-04-29T14:26:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612bcb78:::benchmark_data_smithi194_121512_object31282:head# 2024-04-29T14:26:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612bd829:::benchmark_data_smithi194_121512_object131957:head# 2024-04-29T14:26:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612befe2:::benchmark_data_smithi194_121512_object3514:head# 2024-04-29T14:26:35.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612c4b9b:::benchmark_data_smithi194_121512_object11853:head# 2024-04-29T14:26:35.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612c84d9:::benchmark_data_smithi194_121512_object844:head# 2024-04-29T14:26:35.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612ca6f4:::benchmark_data_smithi194_121512_object119465:head# 2024-04-29T14:26:35.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612ce852:::benchmark_data_smithi194_121512_object44768:head# 2024-04-29T14:26:35.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612d5f52:::benchmark_data_smithi194_121512_object105502:head# 2024-04-29T14:26:35.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612e1f3a:::benchmark_data_smithi194_121512_object24538:head# 2024-04-29T14:26:35.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612e5261:::benchmark_data_smithi194_121512_object100373:head# 2024-04-29T14:26:35.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612e654d:::benchmark_data_smithi194_121512_object86752:head# 2024-04-29T14:26:35.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:35 smithi121 ceph-mon[44533]: pgmap v1247: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 564 op/s; 39317/250306 objects degraded (15.708%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:26:35.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612e750c:::benchmark_data_smithi194_121512_object128105:head# 2024-04-29T14:26:35.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612e9761:::benchmark_data_smithi194_121512_object8879:head# 2024-04-29T14:26:35.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612f82a6:::benchmark_data_smithi194_121512_object133048:head# 2024-04-29T14:26:35.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612fa1ea:::benchmark_data_smithi194_121512_object86540:head# 2024-04-29T14:26:35.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612fb788:::benchmark_data_smithi194_121512_object97778:head# 2024-04-29T14:26:35.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:612fe535:::benchmark_data_smithi194_121512_object110153:head# 2024-04-29T14:26:35.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61300674:::benchmark_data_smithi194_121512_object9508:head# 2024-04-29T14:26:35.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61303100:::benchmark_data_smithi194_121512_object84981:head# 2024-04-29T14:26:35.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61308d38:::benchmark_data_smithi194_121512_object44709:head# 2024-04-29T14:26:35.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6130a034:::benchmark_data_smithi194_121512_object8432:head# 2024-04-29T14:26:35.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613156da:::benchmark_data_smithi194_121512_object118254:head# 2024-04-29T14:26:35.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61322fea:::benchmark_data_smithi194_121512_object124204:head# 2024-04-29T14:26:35.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6132818d:::benchmark_data_smithi194_121512_object124984:head# 2024-04-29T14:26:35.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61329f9f:::benchmark_data_smithi194_121512_object70993:head# 2024-04-29T14:26:35.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6132f50b:::benchmark_data_smithi194_121512_object116028:head# 2024-04-29T14:26:35.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61330b64:::benchmark_data_smithi194_121512_object129685:head# 2024-04-29T14:26:35.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613312f3:::benchmark_data_smithi194_121512_object64087:head# 2024-04-29T14:26:35.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61334d0c:::benchmark_data_smithi194_121512_object127356:head# 2024-04-29T14:26:35.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6133b5c2:::benchmark_data_smithi194_121512_object67161:head# 2024-04-29T14:26:35.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6134df02:::benchmark_data_smithi194_121512_object108138:head# 2024-04-29T14:26:35.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6134fcf4:::benchmark_data_smithi194_121512_object122722:head# 2024-04-29T14:26:35.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6135216b:::benchmark_data_smithi194_121512_object86691:head# 2024-04-29T14:26:35.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6135d3d3:::benchmark_data_smithi194_121512_object82163:head# 2024-04-29T14:26:35.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6135dde4:::benchmark_data_smithi194_121512_object8542:head# 2024-04-29T14:26:35.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6135f5ad:::benchmark_data_smithi194_121512_object84159:head# 2024-04-29T14:26:35.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6137348a:::benchmark_data_smithi194_121512_object121308:head# 2024-04-29T14:26:35.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6137b1c0:::benchmark_data_smithi194_121512_object35771:head# 2024-04-29T14:26:35.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6137cd98:::benchmark_data_smithi194_121512_object11785:head# 2024-04-29T14:26:35.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6137ce15:::benchmark_data_smithi194_121512_object45720:head# 2024-04-29T14:26:35.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6137dd9a:::benchmark_data_smithi194_121512_object48631:head# 2024-04-29T14:26:35.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6137e51b:::benchmark_data_smithi194_121512_object79065:head# 2024-04-29T14:26:35.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6137e6b1:::benchmark_data_smithi194_121512_object89166:head# 2024-04-29T14:26:35.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613800d7:::benchmark_data_smithi194_121512_object120646:head# 2024-04-29T14:26:35.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613815a6:::benchmark_data_smithi194_121512_object120453:head# 2024-04-29T14:26:35.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61399bbc:::benchmark_data_smithi194_121512_object115633:head# 2024-04-29T14:26:35.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613a412e:::benchmark_data_smithi194_121512_object62053:head# 2024-04-29T14:26:35.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613a5769:::benchmark_data_smithi194_121512_object111293:head# 2024-04-29T14:26:35.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613a9244:::benchmark_data_smithi194_121512_object119628:head# 2024-04-29T14:26:35.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613a97e9:::benchmark_data_smithi194_121512_object87119:head# 2024-04-29T14:26:35.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613b83e9:::benchmark_data_smithi194_121512_object118244:head# 2024-04-29T14:26:35.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613b97fa:::benchmark_data_smithi194_121512_object102694:head# 2024-04-29T14:26:35.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613e125f:::benchmark_data_smithi194_121512_object99708:head# 2024-04-29T14:26:35.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613e1c9f:::benchmark_data_smithi194_121512_object20623:head# 2024-04-29T14:26:35.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613e8b5d:::benchmark_data_smithi194_121512_object110755:head# 2024-04-29T14:26:35.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613ee82c:::benchmark_data_smithi194_121512_object40571:head# 2024-04-29T14:26:35.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613f36d0:::benchmark_data_smithi194_121512_object73450:head# 2024-04-29T14:26:35.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613fe3d2:::benchmark_data_smithi194_121512_object10688:head# 2024-04-29T14:26:35.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:613ff546:::benchmark_data_smithi194_121512_object114261:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61408e70:::benchmark_data_smithi194_121512_object124425:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61411643:::benchmark_data_smithi194_121512_object106336:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6141b428:::benchmark_data_smithi194_121512_object51375:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6141c37a:::benchmark_data_smithi194_121512_object27795:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61420f9c:::benchmark_data_smithi194_121512_object47846:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614243e1:::benchmark_data_smithi194_121512_object51426:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6142b64c:::benchmark_data_smithi194_121512_object88792:head# 2024-04-29T14:26:35.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61433c6a:::benchmark_data_smithi194_121512_object18672:head# 2024-04-29T14:26:35.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:26:35.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614368c6:::benchmark_data_smithi194_121512_object120740:head# 2024-04-29T14:26:35.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6145ff8b:::benchmark_data_smithi194_121512_object1053:head# 2024-04-29T14:26:35.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61460845:::benchmark_data_smithi194_121512_object209:head# 2024-04-29T14:26:35.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6146a13e:::benchmark_data_smithi194_121512_object21081:head# 2024-04-29T14:26:35.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6147616f:::benchmark_data_smithi194_121512_object111795:head# 2024-04-29T14:26:35.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6147b19d:::benchmark_data_smithi194_121512_object126765:head# 2024-04-29T14:26:35.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61485250:::benchmark_data_smithi194_121512_object39627:head# 2024-04-29T14:26:35.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6148e5ab:::benchmark_data_smithi194_121512_object57621:head# 2024-04-29T14:26:35.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61490e79:::benchmark_data_smithi194_121512_object26743:head# 2024-04-29T14:26:35.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61492b3c:::benchmark_data_smithi194_121512_object94487:head# 2024-04-29T14:26:35.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614a907d:::benchmark_data_smithi194_121512_object115539:head# 2024-04-29T14:26:35.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614b5a19:::benchmark_data_smithi194_121512_object62249:head# 2024-04-29T14:26:35.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614c7d44:::benchmark_data_smithi194_121512_object34181:head# 2024-04-29T14:26:35.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614d0ce2:::benchmark_data_smithi194_121512_object105786:head# 2024-04-29T14:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614d12f9:::benchmark_data_smithi194_121512_object4600:head# 2024-04-29T14:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614e9fac:::benchmark_data_smithi194_121512_object68502:head# 2024-04-29T14:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614ee247:::benchmark_data_smithi194_121512_object122348:head# 2024-04-29T14:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614fa7dd:::benchmark_data_smithi194_121512_object73882:head# 2024-04-29T14:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614fb61f:::benchmark_data_smithi194_121512_object19224:head# 2024-04-29T14:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614fc236:::benchmark_data_smithi194_121512_object8208:head# 2024-04-29T14:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:614fdfb4:::benchmark_data_smithi194_121512_object123413:head# 2024-04-29T14:26:35.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61500d8c:::benchmark_data_smithi194_121512_object116055:head# 2024-04-29T14:26:35.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61505e41:::benchmark_data_smithi194_121512_object5195:head# 2024-04-29T14:26:35.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6150c2ff:::benchmark_data_smithi194_121512_object75208:head# 2024-04-29T14:26:35.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6150ca7e:::benchmark_data_smithi194_121512_object124070:head# 2024-04-29T14:26:35.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61513a10:::benchmark_data_smithi194_121512_object83197:head# 2024-04-29T14:26:35.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61519920:::benchmark_data_smithi194_121512_object36691:head# 2024-04-29T14:26:35.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61536135:::benchmark_data_smithi194_121512_object55074:head# 2024-04-29T14:26:35.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61538619:::benchmark_data_smithi194_121512_object16232:head# 2024-04-29T14:26:35.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61544991:::benchmark_data_smithi194_121512_object119566:head# 2024-04-29T14:26:35.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61551bda:::benchmark_data_smithi194_121512_object116456:head# 2024-04-29T14:26:35.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61570f3f:::benchmark_data_smithi194_121512_object123696:head# 2024-04-29T14:26:35.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615733e2:::benchmark_data_smithi194_121512_object30238:head# 2024-04-29T14:26:35.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615828b3:::benchmark_data_smithi194_121512_object9007:head# 2024-04-29T14:26:35.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6158893a:::benchmark_data_smithi194_121512_object20392:head# 2024-04-29T14:26:35.701 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6158d276:::benchmark_data_smithi194_121512_object355:head# 2024-04-29T14:26:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6158fe25:::benchmark_data_smithi194_121512_object76610:head# 2024-04-29T14:26:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615923f7:::benchmark_data_smithi194_121512_object112313:head# 2024-04-29T14:26:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61594d93:::benchmark_data_smithi194_121512_object85467:head# 2024-04-29T14:26:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615957ef:::benchmark_data_smithi194_121512_object8104:head# 2024-04-29T14:26:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6159b3a0:::benchmark_data_smithi194_121512_object96626:head# 2024-04-29T14:26:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615a2836:::benchmark_data_smithi194_121512_object101018:head# 2024-04-29T14:26:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615dc1d6:::benchmark_data_smithi194_121512_object23296:head# 2024-04-29T14:26:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615e6d3b:::benchmark_data_smithi194_121512_object67192:head# 2024-04-29T14:26:35.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615edfa4:::benchmark_data_smithi194_121512_object21605:head# 2024-04-29T14:26:35.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615f5528:::benchmark_data_smithi194_121512_object11135:head# 2024-04-29T14:26:35.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:615f64da:::benchmark_data_smithi194_121512_object112600:head# 2024-04-29T14:26:35.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61601215:::benchmark_data_smithi194_121512_object87932:head# 2024-04-29T14:26:35.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61602df4:::benchmark_data_smithi194_121512_object101572:head# 2024-04-29T14:26:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:26:35.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6160cb7f:::benchmark_data_smithi194_121512_object26955:head# 2024-04-29T14:26:35.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61616e98:::benchmark_data_smithi194_121512_object68332:head# 2024-04-29T14:26:35.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616186be:::benchmark_data_smithi194_121512_object78083:head# 2024-04-29T14:26:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:35.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:26:35.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61619a42:::benchmark_data_smithi194_121512_object68153:head# 2024-04-29T14:26:35.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6163e29a:::benchmark_data_smithi194_121512_object130135:head# 2024-04-29T14:26:35.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61649f77:::benchmark_data_smithi194_121512_object51252:head# 2024-04-29T14:26:35.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6165f7e3:::benchmark_data_smithi194_121512_object125587:head# 2024-04-29T14:26:35.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61663fe6:::benchmark_data_smithi194_121512_object93071:head# 2024-04-29T14:26:35.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61667706:::benchmark_data_smithi194_121512_object22311:head# 2024-04-29T14:26:35.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61667e92:::benchmark_data_smithi194_121512_object115930:head# 2024-04-29T14:26:35.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6166b7bf:::benchmark_data_smithi194_121512_object89753:head# 2024-04-29T14:26:35.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61694720:::benchmark_data_smithi194_121512_object3255:head# 2024-04-29T14:26:35.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616a168e:::benchmark_data_smithi194_121512_object109529:head# 2024-04-29T14:26:35.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616a987e:::benchmark_data_smithi194_121512_object129617:head# 2024-04-29T14:26:35.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616aa055:::benchmark_data_smithi194_121512_object42006:head# 2024-04-29T14:26:35.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616c60e4:::benchmark_data_smithi194_121512_object127415:head# 2024-04-29T14:26:35.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616cb148:::benchmark_data_smithi194_121512_object40183:head# 2024-04-29T14:26:35.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616d6c7e:::benchmark_data_smithi194_121512_object39250:head# 2024-04-29T14:26:35.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616dd8d6:::benchmark_data_smithi194_121512_object34593:head# 2024-04-29T14:26:35.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616ea03e:::benchmark_data_smithi194_121512_object95024:head# 2024-04-29T14:26:35.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616edaeb:::benchmark_data_smithi194_121512_object68112:head# 2024-04-29T14:26:35.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616f0ce3:::benchmark_data_smithi194_121512_object9950:head# 2024-04-29T14:26:35.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616f22a0:::benchmark_data_smithi194_121512_object77777:head# 2024-04-29T14:26:35.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:616f53a5:::benchmark_data_smithi194_121512_object53959:head# 2024-04-29T14:26:35.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617039ca:::benchmark_data_smithi194_121512_object39147:head# 2024-04-29T14:26:35.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6170af8b:::benchmark_data_smithi194_121512_object49153:head# 2024-04-29T14:26:35.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61712540:::benchmark_data_smithi194_121512_object131987:head# 2024-04-29T14:26:35.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6171473c:::benchmark_data_smithi194_121512_object97995:head# 2024-04-29T14:26:35.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617179d8:::benchmark_data_smithi194_121512_object53714:head# 2024-04-29T14:26:35.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6171d5fd:::benchmark_data_smithi194_121512_object125459:head# 2024-04-29T14:26:35.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6172c646:::benchmark_data_smithi194_121512_object109394:head# 2024-04-29T14:26:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61730ee2:::benchmark_data_smithi194_121512_object119391:head# 2024-04-29T14:26:35.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6173d67b:::benchmark_data_smithi194_121512_object10934:head# 2024-04-29T14:26:35.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61746772:::benchmark_data_smithi194_121512_object91949:head# 2024-04-29T14:26:35.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6174f532:::benchmark_data_smithi194_121512_object106290:head# 2024-04-29T14:26:35.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617505c7:::benchmark_data_smithi194_121512_object91092:head# 2024-04-29T14:26:35.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61750902:::benchmark_data_smithi194_121512_object14547:head# 2024-04-29T14:26:35.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6175337f:::benchmark_data_smithi194_121512_object124077:head# 2024-04-29T14:26:35.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61763a22:::benchmark_data_smithi194_121512_object74120:head# 2024-04-29T14:26:35.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6176846f:::benchmark_data_smithi194_121512_object46119:head# 2024-04-29T14:26:35.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6177deb2:::benchmark_data_smithi194_121512_object132518:head# 2024-04-29T14:26:35.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61782626:::benchmark_data_smithi194_121512_object46484:head# 2024-04-29T14:26:35.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61782706:::benchmark_data_smithi194_121512_object130058:head# 2024-04-29T14:26:35.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61783abb:::benchmark_data_smithi194_121512_object28552:head# 2024-04-29T14:26:35.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61784712:::benchmark_data_smithi194_121512_object126474:head# 2024-04-29T14:26:35.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61784ecb:::benchmark_data_smithi194_121512_object106307:head# 2024-04-29T14:26:35.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61787fcd:::benchmark_data_smithi194_121512_object74207:head# 2024-04-29T14:26:35.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6179d36b:::benchmark_data_smithi194_121512_object27374:head# 2024-04-29T14:26:35.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617a4338:::benchmark_data_smithi194_121512_object92544:head# 2024-04-29T14:26:35.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617ae57b:::benchmark_data_smithi194_121512_object68272:head# 2024-04-29T14:26:35.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617af5b1:::benchmark_data_smithi194_121512_object89938:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617b38ed:::benchmark_data_smithi194_121512_object26896:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617b7488:::benchmark_data_smithi194_121512_object36028:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617c6058:::benchmark_data_smithi194_121512_object7100:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617c62e8:::benchmark_data_smithi194_121512_object73501:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617ca6b4:::benchmark_data_smithi194_121512_object110111:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617cab8b:::benchmark_data_smithi194_121512_object21596:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617ce145:::benchmark_data_smithi194_121512_object110343:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617d83b3:::benchmark_data_smithi194_121512_object94493:head# 2024-04-29T14:26:35.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617edc91:::benchmark_data_smithi194_121512_object123700:head# 2024-04-29T14:26:35.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617f7654:::benchmark_data_smithi194_121512_object128507:head# 2024-04-29T14:26:35.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:617fd8a1:::benchmark_data_smithi194_121512_object310:head# 2024-04-29T14:26:35.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61801654:::benchmark_data_smithi194_121512_object122588:head# 2024-04-29T14:26:35.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61803ae1:::benchmark_data_smithi194_121512_object57204:head# 2024-04-29T14:26:35.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61812003:::benchmark_data_smithi194_121512_object114958:head# 2024-04-29T14:26:35.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61820e09:::benchmark_data_smithi194_121512_object111525:head# 2024-04-29T14:26:35.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61834240:::benchmark_data_smithi194_121512_object133019:head# 2024-04-29T14:26:35.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61835b98:::benchmark_data_smithi194_121512_object15523:head# 2024-04-29T14:26:35.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6184e52d:::benchmark_data_smithi194_121512_object19207:head# 2024-04-29T14:26:35.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6186562c:::benchmark_data_smithi194_121512_object106961:head# 2024-04-29T14:26:35.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6187b5d0:::benchmark_data_smithi194_121512_object18474:head# 2024-04-29T14:26:35.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61882280:::benchmark_data_smithi194_121512_object73345:head# 2024-04-29T14:26:35.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61882dec:::benchmark_data_smithi194_121512_object99466:head# 2024-04-29T14:26:35.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61886628:::benchmark_data_smithi194_121512_object78792:head# 2024-04-29T14:26:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61888597:::benchmark_data_smithi194_121512_object6805:head# 2024-04-29T14:26:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61888691:::benchmark_data_smithi194_121512_object92562:head# 2024-04-29T14:26:35.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618887c3:::benchmark_data_smithi194_121512_object71035:head# 2024-04-29T14:26:35.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61889954:::benchmark_data_smithi194_121512_object4421:head# 2024-04-29T14:26:35.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61890cd7:::benchmark_data_smithi194_121512_object25264:head# 2024-04-29T14:26:35.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61891f9f:::benchmark_data_smithi194_121512_object123861:head# 2024-04-29T14:26:35.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618961dc:::benchmark_data_smithi194_121512_object60261:head# 2024-04-29T14:26:35.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6189c5b5:::benchmark_data_smithi194_121512_object5899:head# 2024-04-29T14:26:35.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618b9c49:::benchmark_data_smithi194_121512_object39110:head# 2024-04-29T14:26:35.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618c25f1:::benchmark_data_smithi194_121512_object119804:head# 2024-04-29T14:26:35.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618c2faa:::benchmark_data_smithi194_121512_object125232:head# 2024-04-29T14:26:35.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618c619e:::benchmark_data_smithi194_121512_object111776:head# 2024-04-29T14:26:35.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618c8f98:::benchmark_data_smithi194_121512_object10245:head# 2024-04-29T14:26:35.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618cfd56:::benchmark_data_smithi194_121512_object8426:head# 2024-04-29T14:26:35.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618d0dcc:::benchmark_data_smithi194_121512_object51326:head# 2024-04-29T14:26:35.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618dc2d9:::benchmark_data_smithi194_121512_object32461:head# 2024-04-29T14:26:35.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618ece4d:::benchmark_data_smithi194_121512_object55603:head# 2024-04-29T14:26:35.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618efea0:::benchmark_data_smithi194_121512_object37498:head# 2024-04-29T14:26:35.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618f5baa:::benchmark_data_smithi194_121512_object20154:head# 2024-04-29T14:26:35.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618f9a51:::benchmark_data_smithi194_121512_object34743:head# 2024-04-29T14:26:35.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618fdbfa:::benchmark_data_smithi194_121512_object64683:head# 2024-04-29T14:26:35.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:618fddf4:::benchmark_data_smithi194_121512_object15658:head# 2024-04-29T14:26:35.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61903cd0:::benchmark_data_smithi194_121512_object28724:head# 2024-04-29T14:26:35.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6191c0e4:::benchmark_data_smithi194_121512_object79007:head# 2024-04-29T14:26:35.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6191cd44:::benchmark_data_smithi194_121512_object41667:head# 2024-04-29T14:26:35.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6192849c:::benchmark_data_smithi194_121512_object81221:head# 2024-04-29T14:26:35.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61929874:::benchmark_data_smithi194_121512_object104225:head# 2024-04-29T14:26:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6193005c:::benchmark_data_smithi194_121512_object61298:head# 2024-04-29T14:26:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619336ff:::benchmark_data_smithi194_121512_object123551:head# 2024-04-29T14:26:35.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6193524d:::benchmark_data_smithi194_121512_object127970:head# 2024-04-29T14:26:35.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6194a92e:::benchmark_data_smithi194_121512_object29021:head# 2024-04-29T14:26:35.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61958865:::benchmark_data_smithi194_121512_object17435:head# 2024-04-29T14:26:35.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6195f12b:::benchmark_data_smithi194_121512_object52919:head# 2024-04-29T14:26:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619620fc:::benchmark_data_smithi194_121512_object70533:head# 2024-04-29T14:26:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61969168:::benchmark_data_smithi194_121512_object49069:head# 2024-04-29T14:26:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61969ee6:::benchmark_data_smithi194_121512_object118258:head# 2024-04-29T14:26:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61971142:::benchmark_data_smithi194_121512_object46268:head# 2024-04-29T14:26:35.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619725e9:::benchmark_data_smithi194_121512_object22174:head# 2024-04-29T14:26:35.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6197ebe9:::benchmark_data_smithi194_121512_object9546:head# 2024-04-29T14:26:35.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619861f0:::benchmark_data_smithi194_121512_object63443:head# 2024-04-29T14:26:35.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6199124e:::benchmark_data_smithi194_121512_object86399:head# 2024-04-29T14:26:35.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61997841:::benchmark_data_smithi194_121512_object72967:head# 2024-04-29T14:26:35.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61999a73:::benchmark_data_smithi194_121512_object18718:head# 2024-04-29T14:26:35.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6199d0e6:::benchmark_data_smithi194_121512_object74275:head# 2024-04-29T14:26:35.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6199d176:::benchmark_data_smithi194_121512_object115195:head# 2024-04-29T14:26:35.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619ae05f:::benchmark_data_smithi194_121512_object126114:head# 2024-04-29T14:26:35.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619afae2:::benchmark_data_smithi194_121512_object33864:head# 2024-04-29T14:26:35.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619d10ea:::benchmark_data_smithi194_121512_object127007:head# 2024-04-29T14:26:35.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619d29e2:::benchmark_data_smithi194_121512_object64457:head# 2024-04-29T14:26:35.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619d9394:::benchmark_data_smithi194_121512_object69327:head# 2024-04-29T14:26:35.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619db6ea:::benchmark_data_smithi194_121512_object61329:head# 2024-04-29T14:26:35.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619e2f56:::benchmark_data_smithi194_121512_object119260:head# 2024-04-29T14:26:35.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619e71be:::benchmark_data_smithi194_121512_object69563:head# 2024-04-29T14:26:35.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619eae22:::benchmark_data_smithi194_121512_object73035:head# 2024-04-29T14:26:35.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619f36e6:::benchmark_data_smithi194_121512_object50247:head# 2024-04-29T14:26:35.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:619f8f27:::benchmark_data_smithi194_121512_object112753:head# 2024-04-29T14:26:35.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a0107c:::benchmark_data_smithi194_121512_object127680:head# 2024-04-29T14:26:35.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a01761:::benchmark_data_smithi194_121512_object90162:head# 2024-04-29T14:26:35.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a0dcd5:::benchmark_data_smithi194_121512_object30047:head# 2024-04-29T14:26:35.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a0df67:::benchmark_data_smithi194_121512_object28725:head# 2024-04-29T14:26:35.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a2310f:::benchmark_data_smithi194_121512_object122872:head# 2024-04-29T14:26:35.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a2a074:::benchmark_data_smithi194_121512_object50754:head# 2024-04-29T14:26:35.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a2c513:::benchmark_data_smithi194_121512_object105444:head# 2024-04-29T14:26:35.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a3a8f1:::benchmark_data_smithi194_121512_object38321:head# 2024-04-29T14:26:35.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a518c4:::benchmark_data_smithi194_121512_object98959:head# 2024-04-29T14:26:35.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a52999:::benchmark_data_smithi194_121512_object53516:head# 2024-04-29T14:26:35.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a5f740:::benchmark_data_smithi194_121512_object103971:head# 2024-04-29T14:26:35.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a695e5:::benchmark_data_smithi194_121512_object19451:head# 2024-04-29T14:26:35.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a7ef17:::benchmark_data_smithi194_121512_object83379:head# 2024-04-29T14:26:35.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a89c94:::benchmark_data_smithi194_121512_object69982:head# 2024-04-29T14:26:35.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a8f58b:::benchmark_data_smithi194_121512_object76238:head# 2024-04-29T14:26:35.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61a9dbff:::benchmark_data_smithi194_121512_object62017:head# 2024-04-29T14:26:35.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61aa279e:::benchmark_data_smithi194_121512_object125306:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61aaebae:::benchmark_data_smithi194_121512_object19267:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ab388a:::benchmark_data_smithi194_121512_object28783:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ab6641:::benchmark_data_smithi194_121512_object90181:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ab7999:::benchmark_data_smithi194_121512_object129920:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ab8351:::benchmark_data_smithi194_121512_object67490:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61abb341:::benchmark_data_smithi194_121512_object610:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61abf247:::benchmark_data_smithi194_121512_object61253:head# 2024-04-29T14:26:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ac02a6:::benchmark_data_smithi194_121512_object124502:head# 2024-04-29T14:26:35.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ac4224:::benchmark_data_smithi194_121512_object100915:head# 2024-04-29T14:26:35.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ac5edd:::benchmark_data_smithi194_121512_object92589:head# 2024-04-29T14:26:35.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ac983b:::benchmark_data_smithi194_121512_object5217:head# 2024-04-29T14:26:35.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61acd565:::benchmark_data_smithi194_121512_object110214:head# 2024-04-29T14:26:35.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ad170a:::benchmark_data_smithi194_121512_object111440:head# 2024-04-29T14:26:35.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ad4f00:::benchmark_data_smithi194_121512_object115796:head# 2024-04-29T14:26:35.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ae7adb:::benchmark_data_smithi194_121512_object74534:head# 2024-04-29T14:26:35.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61af0dde:::benchmark_data_smithi194_121512_object119403:head# 2024-04-29T14:26:35.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61af3d36:::benchmark_data_smithi194_121512_object15136:head# 2024-04-29T14:26:35.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b0415f:::benchmark_data_smithi194_121512_object9051:head# 2024-04-29T14:26:35.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b0d7cd:::benchmark_data_smithi194_121512_object104268:head# 2024-04-29T14:26:35.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b0e64e:::benchmark_data_smithi194_121512_object57483:head# 2024-04-29T14:26:35.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b1b7f9:::benchmark_data_smithi194_121512_object57714:head# 2024-04-29T14:26:35.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b1d4f0:::benchmark_data_smithi194_121512_object6140:head# 2024-04-29T14:26:35.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b1d95e:::benchmark_data_smithi194_121512_object129243:head# 2024-04-29T14:26:35.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b23695:::benchmark_data_smithi194_121512_object18246:head# 2024-04-29T14:26:35.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b26062:::benchmark_data_smithi194_121512_object66309:head# 2024-04-29T14:26:35.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b294d0:::benchmark_data_smithi194_121512_object16870:head# 2024-04-29T14:26:35.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b2fdc2:::benchmark_data_smithi194_121512_object79538:head# 2024-04-29T14:26:35.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b3de95:::benchmark_data_smithi194_121512_object14497:head# 2024-04-29T14:26:35.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b3e032:::benchmark_data_smithi194_121512_object111791:head# 2024-04-29T14:26:35.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b50d81:::benchmark_data_smithi194_121512_object44323:head# 2024-04-29T14:26:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b5635b:::benchmark_data_smithi194_121512_object18860:head# 2024-04-29T14:26:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b5ff27:::benchmark_data_smithi194_121512_object98249:head# 2024-04-29T14:26:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b61e15:::benchmark_data_smithi194_121512_object76105:head# 2024-04-29T14:26:35.828 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:35.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b69499:::benchmark_data_smithi194_121512_object14430:head# 2024-04-29T14:26:35.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b6be74:::benchmark_data_smithi194_121512_object82142:head# 2024-04-29T14:26:35.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b785dd:::benchmark_data_smithi194_121512_object60016:head# 2024-04-29T14:26:35.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b7b368:::benchmark_data_smithi194_121512_object120529:head# 2024-04-29T14:26:35.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b81472:::benchmark_data_smithi194_121512_object9140:head# 2024-04-29T14:26:35.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b8be05:::benchmark_data_smithi194_121512_object15388:head# 2024-04-29T14:26:35.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b8d94e:::benchmark_data_smithi194_121512_object52554:head# 2024-04-29T14:26:35.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b924ce:::benchmark_data_smithi194_121512_object32475:head# 2024-04-29T14:26:35.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61b9c10d:::benchmark_data_smithi194_121512_object94656:head# 2024-04-29T14:26:35.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61baada7:::benchmark_data_smithi194_121512_object111343:head# 2024-04-29T14:26:35.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bb3feb:::benchmark_data_smithi194_121512_object9732:head# 2024-04-29T14:26:35.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bbf97c:::benchmark_data_smithi194_121512_object47884:head# 2024-04-29T14:26:35.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bbfb47:::benchmark_data_smithi194_121512_object4596:head# 2024-04-29T14:26:35.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bc0a8b:::benchmark_data_smithi194_121512_object132624:head# 2024-04-29T14:26:35.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bc54ab:::benchmark_data_smithi194_121512_object91594:head# 2024-04-29T14:26:35.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bc5dd7:::benchmark_data_smithi194_121512_object40304:head# 2024-04-29T14:26:35.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bd1001:::benchmark_data_smithi194_121512_object25652:head# 2024-04-29T14:26:35.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bd187b:::benchmark_data_smithi194_121512_object29091:head# 2024-04-29T14:26:35.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bf0126:::benchmark_data_smithi194_121512_object73871:head# 2024-04-29T14:26:35.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61bfe14d:::benchmark_data_smithi194_121512_object7404:head# 2024-04-29T14:26:35.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c0d03f:::benchmark_data_smithi194_121512_object131230:head# 2024-04-29T14:26:35.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c19ae1:::benchmark_data_smithi194_121512_object73456:head# 2024-04-29T14:26:35.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c23b09:::benchmark_data_smithi194_121512_object8847:head# 2024-04-29T14:26:35.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c2ddb7:::benchmark_data_smithi194_121512_object55656:head# 2024-04-29T14:26:35.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c344b1:::benchmark_data_smithi194_121512_object63074:head# 2024-04-29T14:26:35.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c37488:::benchmark_data_smithi194_121512_object61838:head# 2024-04-29T14:26:35.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c40c61:::benchmark_data_smithi194_121512_object127775:head# 2024-04-29T14:26:35.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c49ce3:::benchmark_data_smithi194_121512_object62057:head# 2024-04-29T14:26:35.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c59928:::benchmark_data_smithi194_121512_object15221:head# 2024-04-29T14:26:35.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c5ee4e:::benchmark_data_smithi194_121512_object100239:head# 2024-04-29T14:26:35.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c67b77:::benchmark_data_smithi194_121512_object108291:head# 2024-04-29T14:26:35.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c6a89b:::benchmark_data_smithi194_121512_object3578:head# 2024-04-29T14:26:35.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c6ed74:::benchmark_data_smithi194_121512_object75805:head# 2024-04-29T14:26:35.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c76517:::benchmark_data_smithi194_121512_object99853:head# 2024-04-29T14:26:35.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c77ae1:::benchmark_data_smithi194_121512_object19303:head# 2024-04-29T14:26:35.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c7f0cd:::benchmark_data_smithi194_121512_object37751:head# 2024-04-29T14:26:35.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c8cfff:::benchmark_data_smithi194_121512_object72877:head# 2024-04-29T14:26:35.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c9494d:::benchmark_data_smithi194_121512_object55329:head# 2024-04-29T14:26:35.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61c9a306:::benchmark_data_smithi194_121512_object35753:head# 2024-04-29T14:26:35.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ca7313:::benchmark_data_smithi194_121512_object25778:head# 2024-04-29T14:26:35.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61cbcb72:::benchmark_data_smithi194_121512_object30749:head# 2024-04-29T14:26:35.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61cc8775:::benchmark_data_smithi194_121512_object68119:head# 2024-04-29T14:26:35.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61cc8fe0:::benchmark_data_smithi194_121512_object8091:head# 2024-04-29T14:26:35.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61cd8c09:::benchmark_data_smithi194_121512_object72770:head# 2024-04-29T14:26:35.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61cdceb3:::benchmark_data_smithi194_121512_object22652:head# 2024-04-29T14:26:35.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ce6320:::benchmark_data_smithi194_121512_object100891:head# 2024-04-29T14:26:35.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ce7a5b:::benchmark_data_smithi194_121512_object116840:head# 2024-04-29T14:26:35.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61cf1823:::benchmark_data_smithi194_121512_object110704:head# 2024-04-29T14:26:35.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61cfc6a0:::benchmark_data_smithi194_121512_object31180:head# 2024-04-29T14:26:35.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d03d81:::benchmark_data_smithi194_121512_object90032:head# 2024-04-29T14:26:35.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d09683:::benchmark_data_smithi194_121512_object60977:head# 2024-04-29T14:26:35.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d0ae47:::benchmark_data_smithi194_121512_object72988:head# 2024-04-29T14:26:35.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d1f2ea:::benchmark_data_smithi194_121512_object104471:head# 2024-04-29T14:26:35.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d25484:::benchmark_data_smithi194_121512_object78987:head# 2024-04-29T14:26:35.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d32c3a:::benchmark_data_smithi194_121512_object91467:head# 2024-04-29T14:26:35.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d33567:::benchmark_data_smithi194_121512_object19160:head# 2024-04-29T14:26:35.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d3809d:::benchmark_data_smithi194_121512_object4789:head# 2024-04-29T14:26:35.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d3d9e3:::benchmark_data_smithi194_121512_object120761:head# 2024-04-29T14:26:35.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d3daf5:::benchmark_data_smithi194_121512_object125137:head# 2024-04-29T14:26:35.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d48664:::benchmark_data_smithi194_121512_object36945:head# 2024-04-29T14:26:35.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d50ee2:::benchmark_data_smithi194_121512_object37820:head# 2024-04-29T14:26:35.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d5133c:::benchmark_data_smithi194_121512_object8093:head# 2024-04-29T14:26:35.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d74bff:::benchmark_data_smithi194_121512_object22080:head# 2024-04-29T14:26:35.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d74da4:::benchmark_data_smithi194_121512_object69301:head# 2024-04-29T14:26:35.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d8b700:::benchmark_data_smithi194_121512_object70731:head# 2024-04-29T14:26:35.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61d97c6b:::benchmark_data_smithi194_121512_object109403:head# 2024-04-29T14:26:35.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61da8df0:::benchmark_data_smithi194_121512_object42991:head# 2024-04-29T14:26:35.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61dacc3d:::benchmark_data_smithi194_121512_object114245:head# 2024-04-29T14:26:35.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61db1d15:::benchmark_data_smithi194_121512_object342:head# 2024-04-29T14:26:35.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61db95db:::benchmark_data_smithi194_121512_object31247:head# 2024-04-29T14:26:35.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61db9b0a:::benchmark_data_smithi194_121512_object53178:head# 2024-04-29T14:26:35.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61dc10b4:::benchmark_data_smithi194_121512_object5707:head# 2024-04-29T14:26:35.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61dc1aed:::benchmark_data_smithi194_121512_object35758:head# 2024-04-29T14:26:35.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61dc7203:::benchmark_data_smithi194_121512_object104253:head# 2024-04-29T14:26:35.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61dcc92c:::benchmark_data_smithi194_121512_object23132:head# 2024-04-29T14:26:35.865 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61dd2d2e:::benchmark_data_smithi194_121512_object76044:head# 2024-04-29T14:26:35.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61dd6cd4:::benchmark_data_smithi194_121512_object92480:head# 2024-04-29T14:26:35.866 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61df3065:::benchmark_data_smithi194_121512_object62810:head# 2024-04-29T14:26:35.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e00a09:::benchmark_data_smithi194_121512_object72226:head# 2024-04-29T14:26:35.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e05253:::benchmark_data_smithi194_121512_object48995:head# 2024-04-29T14:26:35.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e0c601:::benchmark_data_smithi194_121512_object103797:head# 2024-04-29T14:26:35.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e0edbe:::benchmark_data_smithi194_121512_object119092:head# 2024-04-29T14:26:35.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e1b371:::benchmark_data_smithi194_121512_object107362:head# 2024-04-29T14:26:35.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e1b823:::benchmark_data_smithi194_121512_object37547:head# 2024-04-29T14:26:35.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e34b30:::benchmark_data_smithi194_121512_object39174:head# 2024-04-29T14:26:35.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e357cd:::benchmark_data_smithi194_121512_object87747:head# 2024-04-29T14:26:35.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e57bdb:::benchmark_data_smithi194_121512_object25482:head# 2024-04-29T14:26:35.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e58c3e:::benchmark_data_smithi194_121512_object9996:head# 2024-04-29T14:26:35.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e61e98:::benchmark_data_smithi194_121512_object124325:head# 2024-04-29T14:26:35.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e64163:::benchmark_data_smithi194_121512_object36751:head# 2024-04-29T14:26:35.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e66b88:::benchmark_data_smithi194_121512_object118609:head# 2024-04-29T14:26:35.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e75ea3:::benchmark_data_smithi194_121512_object62275:head# 2024-04-29T14:26:35.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e7e4c0:::benchmark_data_smithi194_121512_object32163:head# 2024-04-29T14:26:35.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e813f3:::benchmark_data_smithi194_121512_object56784:head# 2024-04-29T14:26:35.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e85caa:::benchmark_data_smithi194_121512_object112924:head# 2024-04-29T14:26:35.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e8724f:::benchmark_data_smithi194_121512_object127161:head# 2024-04-29T14:26:35.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61e96304:::benchmark_data_smithi194_121512_object70565:head# 2024-04-29T14:26:35.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ea2b4f:::benchmark_data_smithi194_121512_object115689:head# 2024-04-29T14:26:35.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61eaa665:::benchmark_data_smithi194_121512_object3269:head# 2024-04-29T14:26:35.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61eb7363:::benchmark_data_smithi194_121512_object122772:head# 2024-04-29T14:26:35.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61eba579:::benchmark_data_smithi194_121512_object118734:head# 2024-04-29T14:26:35.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ebf73e:::benchmark_data_smithi194_121512_object12626:head# 2024-04-29T14:26:35.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ec860d:::benchmark_data_smithi194_121512_object92134:head# 2024-04-29T14:26:35.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ecc230:::benchmark_data_smithi194_121512_object79332:head# 2024-04-29T14:26:35.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ecd25d:::benchmark_data_smithi194_121512_object132666:head# 2024-04-29T14:26:35.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ecf933:::benchmark_data_smithi194_121512_object123633:head# 2024-04-29T14:26:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ed45ed:::benchmark_data_smithi194_121512_object45656:head# 2024-04-29T14:26:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61edc3b3:::benchmark_data_smithi194_121512_object66338:head# 2024-04-29T14:26:35.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61edc3df:::benchmark_data_smithi194_121512_object62304:head# 2024-04-29T14:26:35.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ee1e88:::benchmark_data_smithi194_121512_object106847:head# 2024-04-29T14:26:35.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ee5f81:::benchmark_data_smithi194_121512_object100308:head# 2024-04-29T14:26:35.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ef173a:::benchmark_data_smithi194_121512_object128661:head# 2024-04-29T14:26:35.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f056eb:::benchmark_data_smithi194_121512_object31620:head# 2024-04-29T14:26:35.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f0992b:::benchmark_data_smithi194_121512_object5666:head# 2024-04-29T14:26:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f0b66e:::benchmark_data_smithi194_121512_object921:head# 2024-04-29T14:26:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f11c99:::benchmark_data_smithi194_121512_object85447:head# 2024-04-29T14:26:35.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f2002d:::benchmark_data_smithi194_121512_object16159:head# 2024-04-29T14:26:35.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f28821:::benchmark_data_smithi194_121512_object89913:head# 2024-04-29T14:26:35.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f29700:::benchmark_data_smithi194_121512_object15734:head# 2024-04-29T14:26:35.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f35749:::benchmark_data_smithi194_121512_object119183:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f63f0c:::benchmark_data_smithi194_121512_object71224:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f7f336:::benchmark_data_smithi194_121512_object89231:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f8ad69:::benchmark_data_smithi194_121512_object5931:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f99928:::benchmark_data_smithi194_121512_object87505:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61f9f16a:::benchmark_data_smithi194_121512_object19304:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fa0b07:::benchmark_data_smithi194_121512_object96319:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fa62f4:::benchmark_data_smithi194_121512_object53817:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fa94a2:::benchmark_data_smithi194_121512_object103229:head# 2024-04-29T14:26:35.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61faada0:::benchmark_data_smithi194_121512_object122925:head# 2024-04-29T14:26:35.891 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:35.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fcc1b1:::benchmark_data_smithi194_121512_object81251:head# 2024-04-29T14:26:35.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fd66ee:::benchmark_data_smithi194_121512_object55861:head# 2024-04-29T14:26:35.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fda5b3:::benchmark_data_smithi194_121512_object86350:head# 2024-04-29T14:26:35.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fe79aa:::benchmark_data_smithi194_121512_object56630:head# 2024-04-29T14:26:35.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61fef512:::benchmark_data_smithi194_121512_object127335:head# 2024-04-29T14:26:35.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:61ffa8e1:::benchmark_data_smithi194_121512_object105339:head# 2024-04-29T14:26:35.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62004f14:::benchmark_data_smithi194_121512_object114076:head# 2024-04-29T14:26:35.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6200657a:::benchmark_data_smithi194_121512_object34915:head# 2024-04-29T14:26:35.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62006709:::benchmark_data_smithi194_121512_object48101:head# 2024-04-29T14:26:35.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6200afef:::benchmark_data_smithi194_121512_object128860:head# 2024-04-29T14:26:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6200ed52:::benchmark_data_smithi194_121512_object114735:head# 2024-04-29T14:26:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620121e2:::benchmark_data_smithi194_121512_object16025:head# 2024-04-29T14:26:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6201adbf:::benchmark_data_smithi194_121512_object5728:head# 2024-04-29T14:26:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6201d91e:::benchmark_data_smithi194_121512_object77844:head# 2024-04-29T14:26:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6201e60e:::benchmark_data_smithi194_121512_object43043:head# 2024-04-29T14:26:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6203055b:::benchmark_data_smithi194_121512_object90337:head# 2024-04-29T14:26:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6203663b:::benchmark_data_smithi194_121512_object130022:head# 2024-04-29T14:26:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6203ba12:::benchmark_data_smithi194_121512_object88513:head# 2024-04-29T14:26:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6203dc5b:::benchmark_data_smithi194_121512_object41859:head# 2024-04-29T14:26:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62049a43:::benchmark_data_smithi194_121512_object40010:head# 2024-04-29T14:26:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6204dc61:::benchmark_data_smithi194_121512_object13644:head# 2024-04-29T14:26:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6204fee6:::benchmark_data_smithi194_121512_object98215:head# 2024-04-29T14:26:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62058c3e:::benchmark_data_smithi194_121512_object100422:head# 2024-04-29T14:26:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620597c1:::benchmark_data_smithi194_121512_object53501:head# 2024-04-29T14:26:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6205c9d4:::benchmark_data_smithi194_121512_object117290:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6205f243:::benchmark_data_smithi194_121512_object83571:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6205f7be:::benchmark_data_smithi194_121512_object126427:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6206a0f4:::benchmark_data_smithi194_121512_object43494:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6206a9e1:::benchmark_data_smithi194_121512_object53313:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6206c7d4:::benchmark_data_smithi194_121512_object38303:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62082e52:::benchmark_data_smithi194_121512_object95222:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62085841:::benchmark_data_smithi194_121512_object96468:head# 2024-04-29T14:26:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6208e071:::benchmark_data_smithi194_121512_object12582:head# 2024-04-29T14:26:35.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6208f1ae:::benchmark_data_smithi194_121512_object108703:head# 2024-04-29T14:26:35.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62092d0e:::benchmark_data_smithi194_121512_object129740:head# 2024-04-29T14:26:35.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620948b4:::benchmark_data_smithi194_121512_object19502:head# 2024-04-29T14:26:35.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620972aa:::benchmark_data_smithi194_121512_object21003:head# 2024-04-29T14:26:35.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620a815f:::benchmark_data_smithi194_121512_object29568:head# 2024-04-29T14:26:35.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620af14a:::benchmark_data_smithi194_121512_object80200:head# 2024-04-29T14:26:35.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620b1887:::benchmark_data_smithi194_121512_object43944:head# 2024-04-29T14:26:35.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620b6958:::benchmark_data_smithi194_121512_object32976:head# 2024-04-29T14:26:35.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620c3cb2:::benchmark_data_smithi194_121512_object109666:head# 2024-04-29T14:26:35.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620cba16:::benchmark_data_smithi194_121512_object91911:head# 2024-04-29T14:26:35.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620cfd90:::benchmark_data_smithi194_121512_object96032:head# 2024-04-29T14:26:35.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620dc4c6:::benchmark_data_smithi194_121512_object129273:head# 2024-04-29T14:26:35.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620dd0d6:::benchmark_data_smithi194_121512_object51055:head# 2024-04-29T14:26:35.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620df572:::benchmark_data_smithi194_121512_object7423:head# 2024-04-29T14:26:35.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620e6fd0:::benchmark_data_smithi194_121512_object1625:head# 2024-04-29T14:26:35.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620eb69a:::benchmark_data_smithi194_121512_object111283:head# 2024-04-29T14:26:35.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620ee9a4:::benchmark_data_smithi194_121512_object124561:head# 2024-04-29T14:26:35.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620f0294:::benchmark_data_smithi194_121512_object107908:head# 2024-04-29T14:26:35.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:620f3a5a:::benchmark_data_smithi194_121512_object75281:head# 2024-04-29T14:26:35.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621024de:::benchmark_data_smithi194_121512_object41633:head# 2024-04-29T14:26:35.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6210344a:::benchmark_data_smithi194_121512_object103390:head# 2024-04-29T14:26:35.919 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:35.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62105ef5:::benchmark_data_smithi194_121512_object78674:head# 2024-04-29T14:26:35.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62106dc8:::benchmark_data_smithi194_121512_object6062:head# 2024-04-29T14:26:35.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6210a671:::benchmark_data_smithi194_121512_object73496:head# 2024-04-29T14:26:35.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6211b01c:::benchmark_data_smithi194_121512_object31148:head# 2024-04-29T14:26:35.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6211be32:::benchmark_data_smithi194_121512_object87299:head# 2024-04-29T14:26:35.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62128d90:::benchmark_data_smithi194_121512_object22871:head# 2024-04-29T14:26:35.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62129cb5:::benchmark_data_smithi194_121512_object126049:head# 2024-04-29T14:26:35.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6212bfe7:::benchmark_data_smithi194_121512_object127850:head# 2024-04-29T14:26:35.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62131dcc:::benchmark_data_smithi194_121512_object100279:head# 2024-04-29T14:26:35.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62156223:::benchmark_data_smithi194_121512_object55839:head# 2024-04-29T14:26:35.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6215a067:::benchmark_data_smithi194_121512_object7786:head# 2024-04-29T14:26:35.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621619aa:::benchmark_data_smithi194_121512_object47034:head# 2024-04-29T14:26:35.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62178778:::benchmark_data_smithi194_121512_object51404:head# 2024-04-29T14:26:35.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6217db94:::benchmark_data_smithi194_121512_object18635:head# 2024-04-29T14:26:35.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6217e763:::benchmark_data_smithi194_121512_object88717:head# 2024-04-29T14:26:35.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6218b222:::benchmark_data_smithi194_121512_object47504:head# 2024-04-29T14:26:35.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6218e457:::benchmark_data_smithi194_121512_object118322:head# 2024-04-29T14:26:35.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621911b5:::benchmark_data_smithi194_121512_object117698:head# 2024-04-29T14:26:35.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62196d32:::benchmark_data_smithi194_121512_object6998:head# 2024-04-29T14:26:35.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621a5763:::benchmark_data_smithi194_121512_object114300:head# 2024-04-29T14:26:35.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621a900e:::benchmark_data_smithi194_121512_object101994:head# 2024-04-29T14:26:35.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621ac1a2:::benchmark_data_smithi194_121512_object62065:head# 2024-04-29T14:26:35.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621b61cb:::benchmark_data_smithi194_121512_object105671:head# 2024-04-29T14:26:35.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621c3687:::benchmark_data_smithi194_121512_object97873:head# 2024-04-29T14:26:35.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621c4993:::benchmark_data_smithi194_121512_object75598:head# 2024-04-29T14:26:35.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621cc2f2:::benchmark_data_smithi194_121512_object65296:head# 2024-04-29T14:26:35.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621cefe9:::benchmark_data_smithi194_121512_object75480:head# 2024-04-29T14:26:35.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621d0a0c:::benchmark_data_smithi194_121512_object10012:head# 2024-04-29T14:26:35.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621d77e0:::benchmark_data_smithi194_121512_object82224:head# 2024-04-29T14:26:35.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621e2c06:::benchmark_data_smithi194_121512_object1544:head# 2024-04-29T14:26:35.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621ed0de:::benchmark_data_smithi194_121512_object14239:head# 2024-04-29T14:26:35.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621f0290:::benchmark_data_smithi194_121512_object125332:head# 2024-04-29T14:26:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:35.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:35.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621f673a:::benchmark_data_smithi194_121512_object86099:head# 2024-04-29T14:26:35.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:621fd012:::benchmark_data_smithi194_121512_object87442:head# 2024-04-29T14:26:35.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6220e15b:::benchmark_data_smithi194_121512_object80310:head# 2024-04-29T14:26:35.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6221b025:::benchmark_data_smithi194_121512_object85250:head# 2024-04-29T14:26:35.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62227908:::benchmark_data_smithi194_121512_object120679:head# 2024-04-29T14:26:35.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622354f4:::benchmark_data_smithi194_121512_object68184:head# 2024-04-29T14:26:35.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6223baa6:::benchmark_data_smithi194_121512_object4888:head# 2024-04-29T14:26:35.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6224ab57:::benchmark_data_smithi194_121512_object29375:head# 2024-04-29T14:26:35.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6224fe8e:::benchmark_data_smithi194_121512_object111455:head# 2024-04-29T14:26:35.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62262598:::benchmark_data_smithi194_121512_object55635:head# 2024-04-29T14:26:35.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62263ae9:::benchmark_data_smithi194_121512_object38950:head# 2024-04-29T14:26:35.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6226cf39:::benchmark_data_smithi194_121512_object107306:head# 2024-04-29T14:26:35.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62278f64:::benchmark_data_smithi194_121512_object53186:head# 2024-04-29T14:26:35.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62283ae8:::benchmark_data_smithi194_121512_object125288:head# 2024-04-29T14:26:35.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62287cb2:::benchmark_data_smithi194_121512_object24270:head# 2024-04-29T14:26:35.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62292e31:::benchmark_data_smithi194_121512_object72801:head# 2024-04-29T14:26:35.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62297135:::benchmark_data_smithi194_121512_object63604:head# 2024-04-29T14:26:35.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6229962b:::benchmark_data_smithi194_121512_object25497:head# 2024-04-29T14:26:35.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6229e204:::benchmark_data_smithi194_121512_object104552:head# 2024-04-29T14:26:35.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622a247b:::benchmark_data_smithi194_121512_object48965:head# 2024-04-29T14:26:35.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622b87e9:::benchmark_data_smithi194_121512_object110379:head# 2024-04-29T14:26:35.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622be8fb:::benchmark_data_smithi194_121512_object3621:head# 2024-04-29T14:26:35.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622c0d23:::benchmark_data_smithi194_121512_object81291:head# 2024-04-29T14:26:35.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622c96d1:::benchmark_data_smithi194_121512_object77209:head# 2024-04-29T14:26:35.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622ddf46:::benchmark_data_smithi194_121512_object90633:head# 2024-04-29T14:26:35.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622e0fa3:::benchmark_data_smithi194_121512_object61709:head# 2024-04-29T14:26:35.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622e0fe3:::benchmark_data_smithi194_121512_object56476:head# 2024-04-29T14:26:35.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622ed649:::benchmark_data_smithi194_121512_object126863:head# 2024-04-29T14:26:35.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:622f71f1:::benchmark_data_smithi194_121512_object39001:head# 2024-04-29T14:26:35.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6230116d:::benchmark_data_smithi194_121512_object31352:head# 2024-04-29T14:26:35.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6230b589:::benchmark_data_smithi194_121512_object15143:head# 2024-04-29T14:26:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6230da57:::benchmark_data_smithi194_121512_object86880:head# 2024-04-29T14:26:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62312b42:::benchmark_data_smithi194_121512_object5078:head# 2024-04-29T14:26:35.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62323f93:::benchmark_data_smithi194_121512_object35348:head# 2024-04-29T14:26:35.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62348937:::benchmark_data_smithi194_121512_object33975:head# 2024-04-29T14:26:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62349ba0:::benchmark_data_smithi194_121512_object89393:head# 2024-04-29T14:26:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623757da:::benchmark_data_smithi194_121512_object56766:head# 2024-04-29T14:26:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6238e420:::benchmark_data_smithi194_121512_object6625:head# 2024-04-29T14:26:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623b000a:::benchmark_data_smithi194_121512_object14827:head# 2024-04-29T14:26:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623b251c:::benchmark_data_smithi194_121512_object130896:head# 2024-04-29T14:26:35.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623d4f82:::benchmark_data_smithi194_121512_object420:head# 2024-04-29T14:26:35.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623e35cb:::benchmark_data_smithi194_121512_object64941:head# 2024-04-29T14:26:35.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623e5114:::benchmark_data_smithi194_121512_object129338:head# 2024-04-29T14:26:35.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623eba47:::benchmark_data_smithi194_121512_object73824:head# 2024-04-29T14:26:35.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623f358b:::benchmark_data_smithi194_121512_object28845:head# 2024-04-29T14:26:35.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623f4dfa:::benchmark_data_smithi194_121512_object83116:head# 2024-04-29T14:26:35.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:623f5cd7:::benchmark_data_smithi194_121512_object53658:head# 2024-04-29T14:26:35.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62401323:::benchmark_data_smithi194_121512_object113894:head# 2024-04-29T14:26:35.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62404de0:::benchmark_data_smithi194_121512_object90161:head# 2024-04-29T14:26:35.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62409d98:::benchmark_data_smithi194_121512_object17245:head# 2024-04-29T14:26:35.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6240b285:::benchmark_data_smithi194_121512_object56069:head# 2024-04-29T14:26:35.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62415a36:::benchmark_data_smithi194_121512_object80294:head# 2024-04-29T14:26:35.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62425762:::benchmark_data_smithi194_121512_object8955:head# 2024-04-29T14:26:35.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6243fd1e:::benchmark_data_smithi194_121512_object129177:head# 2024-04-29T14:26:35.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62449f43:::benchmark_data_smithi194_121512_object76958:head# 2024-04-29T14:26:35.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6244b1e5:::benchmark_data_smithi194_121512_object100093:head# 2024-04-29T14:26:35.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6245efd9:::benchmark_data_smithi194_121512_object105835:head# 2024-04-29T14:26:35.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624678e9:::benchmark_data_smithi194_121512_object14307:head# 2024-04-29T14:26:35.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62468795:::benchmark_data_smithi194_121512_object65885:head# 2024-04-29T14:26:35.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6246fb0a:::benchmark_data_smithi194_121512_object74323:head# 2024-04-29T14:26:35.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6247d51d:::benchmark_data_smithi194_121512_object86689:head# 2024-04-29T14:26:35.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6249681b:::benchmark_data_smithi194_121512_object24066:head# 2024-04-29T14:26:35.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6249846a:::benchmark_data_smithi194_121512_object28113:head# 2024-04-29T14:26:35.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624990c4:::benchmark_data_smithi194_121512_object59878:head# 2024-04-29T14:26:35.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6249d44d:::benchmark_data_smithi194_121512_object38311:head# 2024-04-29T14:26:35.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6249fc48:::benchmark_data_smithi194_121512_object81621:head# 2024-04-29T14:26:35.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6249ff6c:::benchmark_data_smithi194_121512_object24968:head# 2024-04-29T14:26:35.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624aca0d:::benchmark_data_smithi194_121512_object16433:head# 2024-04-29T14:26:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624b00a9:::benchmark_data_smithi194_121512_object20629:head# 2024-04-29T14:26:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624b322e:::benchmark_data_smithi194_121512_object55830:head# 2024-04-29T14:26:35.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624c1a8d:::benchmark_data_smithi194_121512_object58236:head# 2024-04-29T14:26:35.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624c8f81:::benchmark_data_smithi194_121512_object107685:head# 2024-04-29T14:26:35.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624cae3f:::benchmark_data_smithi194_121512_object49202:head# 2024-04-29T14:26:35.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624fc082:::benchmark_data_smithi194_121512_object82655:head# 2024-04-29T14:26:35.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:624ffc11:::benchmark_data_smithi194_121512_object82656:head# 2024-04-29T14:26:35.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625017c1:::benchmark_data_smithi194_121512_object56277:head# 2024-04-29T14:26:35.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62504f3d:::benchmark_data_smithi194_121512_object8905:head# 2024-04-29T14:26:35.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6250645f:::benchmark_data_smithi194_121512_object128777:head# 2024-04-29T14:26:35.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6250d709:::benchmark_data_smithi194_121512_object3679:head# 2024-04-29T14:26:35.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62514b27:::benchmark_data_smithi194_121512_object80750:head# 2024-04-29T14:26:35.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62516b0d:::benchmark_data_smithi194_121512_object20381:head# 2024-04-29T14:26:35.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62536a0d:::benchmark_data_smithi194_121512_object72175:head# 2024-04-29T14:26:35.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6253c732:::benchmark_data_smithi194_121512_object94408:head# 2024-04-29T14:26:35.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625471c1:::benchmark_data_smithi194_121512_object129115:head# 2024-04-29T14:26:35.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6254d5f6:::benchmark_data_smithi194_121512_object46668:head# 2024-04-29T14:26:35.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6254f821:::benchmark_data_smithi194_121512_object43368:head# 2024-04-29T14:26:35.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625510c6:::benchmark_data_smithi194_121512_object123880:head# 2024-04-29T14:26:35.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6255263f:::benchmark_data_smithi194_121512_object115775:head# 2024-04-29T14:26:35.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6255271f:::benchmark_data_smithi194_121512_object57454:head# 2024-04-29T14:26:35.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6255648e:::benchmark_data_smithi194_121512_object23378:head# 2024-04-29T14:26:35.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62562411:::benchmark_data_smithi194_121512_object105250:head# 2024-04-29T14:26:35.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625640d3:::benchmark_data_smithi194_121512_object69160:head# 2024-04-29T14:26:35.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62567d61:::benchmark_data_smithi194_121512_object38725:head# 2024-04-29T14:26:35.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6256c777:::benchmark_data_smithi194_121512_object103539:head# 2024-04-29T14:26:35.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62580bd2:::benchmark_data_smithi194_121512_object105699:head# 2024-04-29T14:26:35.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6258a831:::benchmark_data_smithi194_121512_object121236:head# 2024-04-29T14:26:35.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6259446f:::benchmark_data_smithi194_121512_object60277:head# 2024-04-29T14:26:35.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625a3574:::benchmark_data_smithi194_121512_object37391:head# 2024-04-29T14:26:35.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625adb4b:::benchmark_data_smithi194_121512_object111966:head# 2024-04-29T14:26:35.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625b632c:::benchmark_data_smithi194_121512_object97859:head# 2024-04-29T14:26:35.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625cea85:::benchmark_data_smithi194_121512_object93558:head# 2024-04-29T14:26:35.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625e26d3:::benchmark_data_smithi194_121512_object101567:head# 2024-04-29T14:26:35.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625e7034:::benchmark_data_smithi194_121512_object94783:head# 2024-04-29T14:26:35.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625e8ddc:::benchmark_data_smithi194_121512_object8610:head# 2024-04-29T14:26:35.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:26:35.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625eb2ef:::benchmark_data_smithi194_121512_object72585:head# 2024-04-29T14:26:35.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625eff6e:::benchmark_data_smithi194_121512_object121046:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:625f8b91:::benchmark_data_smithi194_121512_object16183:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62607544:::benchmark_data_smithi194_121512_object49415:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6260ad02:::benchmark_data_smithi194_121512_object103544:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62612198:::benchmark_data_smithi194_121512_object52921:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626148ee:::benchmark_data_smithi194_121512_object39260:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6261a403:::benchmark_data_smithi194_121512_object84182:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6261e4f8:::benchmark_data_smithi194_121512_object33658:head# 2024-04-29T14:26:35.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6265c0a9:::benchmark_data_smithi194_121512_object123046:head# 2024-04-29T14:26:36.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6265d713:::benchmark_data_smithi194_121512_object25721:head# 2024-04-29T14:26:36.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6266224d:::benchmark_data_smithi194_121512_object130529:head# 2024-04-29T14:26:36.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62672317:::benchmark_data_smithi194_121512_object67800:head# 2024-04-29T14:26:36.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62680155:::benchmark_data_smithi194_121512_object35934:head# 2024-04-29T14:26:36.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6268866f:::benchmark_data_smithi194_121512_object28669:head# 2024-04-29T14:26:36.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6268946f:::benchmark_data_smithi194_121512_object69104:head# 2024-04-29T14:26:36.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626a1ddc:::benchmark_data_smithi194_121512_object88707:head# 2024-04-29T14:26:36.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626a9f42:::benchmark_data_smithi194_121512_object41535:head# 2024-04-29T14:26:36.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626bebe6:::benchmark_data_smithi194_121512_object86756:head# 2024-04-29T14:26:36.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626c963b:::benchmark_data_smithi194_121512_object67632:head# 2024-04-29T14:26:36.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626ce58e:::benchmark_data_smithi194_121512_object116595:head# 2024-04-29T14:26:36.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626d6cf6:::benchmark_data_smithi194_121512_object15396:head# 2024-04-29T14:26:36.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626d77f0:::benchmark_data_smithi194_121512_object110026:head# 2024-04-29T14:26:36.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626d97d6:::benchmark_data_smithi194_121512_object105870:head# 2024-04-29T14:26:36.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626e5bd4:::benchmark_data_smithi194_121512_object43455:head# 2024-04-29T14:26:36.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626f6a43:::benchmark_data_smithi194_121512_object35038:head# 2024-04-29T14:26:36.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:626f859d:::benchmark_data_smithi194_121512_object20958:head# 2024-04-29T14:26:36.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62708b03:::benchmark_data_smithi194_121512_object62402:head# 2024-04-29T14:26:36.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62709c26:::benchmark_data_smithi194_121512_object14973:head# 2024-04-29T14:26:36.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6270baa0:::benchmark_data_smithi194_121512_object93046:head# 2024-04-29T14:26:36.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62710ae8:::benchmark_data_smithi194_121512_object114112:head# 2024-04-29T14:26:36.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62715f92:::benchmark_data_smithi194_121512_object54685:head# 2024-04-29T14:26:36.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62721e47:::benchmark_data_smithi194_121512_object100957:head# 2024-04-29T14:26:36.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62722a31:::benchmark_data_smithi194_121512_object55016:head# 2024-04-29T14:26:36.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6272700b:::benchmark_data_smithi194_121512_object53603:head# 2024-04-29T14:26:36.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62728307:::benchmark_data_smithi194_121512_object7586:head# 2024-04-29T14:26:36.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6272c8af:::benchmark_data_smithi194_121512_object40604:head# 2024-04-29T14:26:36.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6273e1a7:::benchmark_data_smithi194_121512_object51479:head# 2024-04-29T14:26:36.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62740f92:::benchmark_data_smithi194_121512_object19673:head# 2024-04-29T14:26:36.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62742426:::benchmark_data_smithi194_121512_object21917:head# 2024-04-29T14:26:36.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627463f1:::benchmark_data_smithi194_121512_object32134:head# 2024-04-29T14:26:36.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62747dcb:::benchmark_data_smithi194_121512_object123112:head# 2024-04-29T14:26:36.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6274a4f1:::benchmark_data_smithi194_121512_object90146:head# 2024-04-29T14:26:36.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6274bbd0:::benchmark_data_smithi194_121512_object110203:head# 2024-04-29T14:26:36.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62752559:::benchmark_data_smithi194_121512_object58052:head# 2024-04-29T14:26:36.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62753a3d:::benchmark_data_smithi194_121512_object74947:head# 2024-04-29T14:26:36.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62756fa6:::benchmark_data_smithi194_121512_object42234:head# 2024-04-29T14:26:36.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6276f01e:::benchmark_data_smithi194_121512_object85310:head# 2024-04-29T14:26:36.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6276f7f8:::benchmark_data_smithi194_121512_object35379:head# 2024-04-29T14:26:36.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627798b8:::benchmark_data_smithi194_121512_object34621:head# 2024-04-29T14:26:36.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627830d3:::benchmark_data_smithi194_121512_object61322:head# 2024-04-29T14:26:36.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6279bf04:::benchmark_data_smithi194_121512_object100738:head# 2024-04-29T14:26:36.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6279ffce:::benchmark_data_smithi194_121512_object16530:head# 2024-04-29T14:26:36.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627a360a:::benchmark_data_smithi194_121512_object1761:head# 2024-04-29T14:26:36.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627b07c3:::benchmark_data_smithi194_121512_object1793:head# 2024-04-29T14:26:36.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627bb2a8:::benchmark_data_smithi194_121512_object111485:head# 2024-04-29T14:26:36.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627bb36b:::benchmark_data_smithi194_121512_object112465:head# 2024-04-29T14:26:36.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627c2c1a:::benchmark_data_smithi194_121512_object130266:head# 2024-04-29T14:26:36.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627c4869:::benchmark_data_smithi194_121512_object83099:head# 2024-04-29T14:26:36.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627d85b6:::benchmark_data_smithi194_121512_object102489:head# 2024-04-29T14:26:36.034 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:36.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627f3e4b:::benchmark_data_smithi194_121512_object118488:head# 2024-04-29T14:26:36.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627fa4dd:::benchmark_data_smithi194_121512_object4472:head# 2024-04-29T14:26:36.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:627fdcc6:::benchmark_data_smithi194_121512_object50807:head# 2024-04-29T14:26:36.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62806c33:::benchmark_data_smithi194_121512_object110096:head# 2024-04-29T14:26:36.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6280a00f:::benchmark_data_smithi194_121512_object84010:head# 2024-04-29T14:26:36.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6280a016:::benchmark_data_smithi194_121512_object4839:head# 2024-04-29T14:26:36.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6280acb7:::benchmark_data_smithi194_121512_object13964:head# 2024-04-29T14:26:36.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62819c51:::benchmark_data_smithi194_121512_object114630:head# 2024-04-29T14:26:36.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62824be5:::benchmark_data_smithi194_121512_object20546:head# 2024-04-29T14:26:36.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62828f80:::benchmark_data_smithi194_121512_object16075:head# 2024-04-29T14:26:36.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6282f6f8:::benchmark_data_smithi194_121512_object59767:head# 2024-04-29T14:26:36.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62835d95:::benchmark_data_smithi194_121512_object7263:head# 2024-04-29T14:26:36.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62838cc2:::benchmark_data_smithi194_121512_object40144:head# 2024-04-29T14:26:36.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6283a219:::benchmark_data_smithi194_121512_object104204:head# 2024-04-29T14:26:36.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6284075f:::benchmark_data_smithi194_121512_object45959:head# 2024-04-29T14:26:36.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628415f2:::benchmark_data_smithi194_121512_object130175:head# 2024-04-29T14:26:36.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628440e5:::benchmark_data_smithi194_121512_object61450:head# 2024-04-29T14:26:36.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62849932:::benchmark_data_smithi194_121512_object27325:head# 2024-04-29T14:26:36.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62857bef:::benchmark_data_smithi194_121512_object117622:head# 2024-04-29T14:26:36.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628660dc:::benchmark_data_smithi194_121512_object132083:head# 2024-04-29T14:26:36.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62868720:::benchmark_data_smithi194_121512_object76522:head# 2024-04-29T14:26:36.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6286a2e9:::benchmark_data_smithi194_121512_object37433:head# 2024-04-29T14:26:36.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62872dd8:::benchmark_data_smithi194_121512_object824:head# 2024-04-29T14:26:36.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62880795:::benchmark_data_smithi194_121512_object84720:head# 2024-04-29T14:26:36.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6288378b:::benchmark_data_smithi194_121512_object87143:head# 2024-04-29T14:26:36.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6288d6a8:::benchmark_data_smithi194_121512_object86413:head# 2024-04-29T14:26:36.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62892d70:::benchmark_data_smithi194_121512_object84362:head# 2024-04-29T14:26:36.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62897243:::benchmark_data_smithi194_121512_object122367:head# 2024-04-29T14:26:36.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62898898:::benchmark_data_smithi194_121512_object6368:head# 2024-04-29T14:26:36.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628b1aa8:::benchmark_data_smithi194_121512_object121230:head# 2024-04-29T14:26:36.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628c553c:::benchmark_data_smithi194_121512_object78236:head# 2024-04-29T14:26:36.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628c5ae6:::benchmark_data_smithi194_121512_object21210:head# 2024-04-29T14:26:36.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628d5783:::benchmark_data_smithi194_121512_object114814:head# 2024-04-29T14:26:36.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628d7f46:::benchmark_data_smithi194_121512_object34657:head# 2024-04-29T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628dc938:::benchmark_data_smithi194_121512_object114062:head# 2024-04-29T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628e5251:::benchmark_data_smithi194_121512_object34843:head# 2024-04-29T14:26:36.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628e80b1:::benchmark_data_smithi194_121512_object21642:head# 2024-04-29T14:26:36.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628efd50:::benchmark_data_smithi194_121512_object83020:head# 2024-04-29T14:26:36.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628f3a08:::benchmark_data_smithi194_121512_object15712:head# 2024-04-29T14:26:36.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628f8c1a:::benchmark_data_smithi194_121512_object69135:head# 2024-04-29T14:26:36.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628fab2a:::benchmark_data_smithi194_121512_object118153:head# 2024-04-29T14:26:36.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:628fdf0a:::benchmark_data_smithi194_121512_object65550:head# 2024-04-29T14:26:36.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62900528:::benchmark_data_smithi194_121512_object7913:head# 2024-04-29T14:26:36.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62908b4c:::benchmark_data_smithi194_121512_object78333:head# 2024-04-29T14:26:36.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6290c623:::benchmark_data_smithi194_121512_object96214:head# 2024-04-29T14:26:36.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6290f320:::benchmark_data_smithi194_121512_object64287:head# 2024-04-29T14:26:36.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629181e2:::benchmark_data_smithi194_121512_object34712:head# 2024-04-29T14:26:36.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629193b1:::benchmark_data_smithi194_121512_object107400:head# 2024-04-29T14:26:36.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62921ed9:::benchmark_data_smithi194_121512_object38841:head# 2024-04-29T14:26:36.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629318a4:::benchmark_data_smithi194_121512_object109539:head# 2024-04-29T14:26:36.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629327bb:::benchmark_data_smithi194_121512_object4170:head# 2024-04-29T14:26:36.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6293521e:::benchmark_data_smithi194_121512_object116190:head# 2024-04-29T14:26:36.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6293555f:::benchmark_data_smithi194_121512_object17161:head# 2024-04-29T14:26:36.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6294cfda:::benchmark_data_smithi194_121512_object28165:head# 2024-04-29T14:26:36.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6294e7a2:::benchmark_data_smithi194_121512_object2591:head# 2024-04-29T14:26:36.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629529da:::benchmark_data_smithi194_121512_object35481:head# 2024-04-29T14:26:36.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6296006a:::benchmark_data_smithi194_121512_object56397:head# 2024-04-29T14:26:36.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62975adb:::benchmark_data_smithi194_121512_object3467:head# 2024-04-29T14:26:36.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629796ee:::benchmark_data_smithi194_121512_object21398:head# 2024-04-29T14:26:36.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6297d0b8:::benchmark_data_smithi194_121512_object17320:head# 2024-04-29T14:26:36.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62980500:::benchmark_data_smithi194_121512_object69946:head# 2024-04-29T14:26:36.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6299b3ac:::benchmark_data_smithi194_121512_object38754:head# 2024-04-29T14:26:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:36.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629b1323:::benchmark_data_smithi194_121512_object36692:head# 2024-04-29T14:26:36.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629b1ad3:::benchmark_data_smithi194_121512_object22023:head# 2024-04-29T14:26:36.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629b751c:::benchmark_data_smithi194_121512_object125865:head# 2024-04-29T14:26:36.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629bc224:::benchmark_data_smithi194_121512_object87010:head# 2024-04-29T14:26:36.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629bf5e0:::benchmark_data_smithi194_121512_object4599:head# 2024-04-29T14:26:36.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629c57c3:::benchmark_data_smithi194_121512_object68731:head# 2024-04-29T14:26:36.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629d847c:::benchmark_data_smithi194_121512_object71531:head# 2024-04-29T14:26:36.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629ddad1:::benchmark_data_smithi194_121512_object71365:head# 2024-04-29T14:26:36.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629e3ed1:::benchmark_data_smithi194_121512_object61320:head# 2024-04-29T14:26:36.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629e60ec:::benchmark_data_smithi194_121512_object31772:head# 2024-04-29T14:26:36.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629e65a7:::benchmark_data_smithi194_121512_object65759:head# 2024-04-29T14:26:36.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629e715d:::benchmark_data_smithi194_121512_object13235:head# 2024-04-29T14:26:36.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629fb4cb:::benchmark_data_smithi194_121512_object99993:head# 2024-04-29T14:26:36.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:629fd2b3:::benchmark_data_smithi194_121512_object131479:head# 2024-04-29T14:26:36.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a0c9b6:::benchmark_data_smithi194_121512_object116475:head# 2024-04-29T14:26:36.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a12279:::benchmark_data_smithi194_121512_object93021:head# 2024-04-29T14:26:36.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a24927:::benchmark_data_smithi194_121512_object2537:head# 2024-04-29T14:26:36.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a2525a:::benchmark_data_smithi194_121512_object86391:head# 2024-04-29T14:26:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a2801b:::benchmark_data_smithi194_121512_object39728:head# 2024-04-29T14:26:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a2fbb7:::benchmark_data_smithi194_121512_object103168:head# 2024-04-29T14:26:36.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a311f5:::benchmark_data_smithi194_121512_object47686:head# 2024-04-29T14:26:36.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a327cb:::benchmark_data_smithi194_121512_object70961:head# 2024-04-29T14:26:36.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a36d6c:::benchmark_data_smithi194_121512_object64988:head# 2024-04-29T14:26:36.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a398a6:::benchmark_data_smithi194_121512_object124383:head# 2024-04-29T14:26:36.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a4ac83:::benchmark_data_smithi194_121512_object109073:head# 2024-04-29T14:26:36.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a4afb8:::benchmark_data_smithi194_121512_object132121:head# 2024-04-29T14:26:36.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a5983e:::benchmark_data_smithi194_121512_object97193:head# 2024-04-29T14:26:36.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a5df38:::benchmark_data_smithi194_121512_object105875:head# 2024-04-29T14:26:36.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a6aa86:::benchmark_data_smithi194_121512_object17633:head# 2024-04-29T14:26:36.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a6d23b:::benchmark_data_smithi194_121512_object66470:head# 2024-04-29T14:26:36.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a8342a:::benchmark_data_smithi194_121512_object25629:head# 2024-04-29T14:26:36.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a8ade1:::benchmark_data_smithi194_121512_object17906:head# 2024-04-29T14:26:36.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a8eaf9:::benchmark_data_smithi194_121512_object74387:head# 2024-04-29T14:26:36.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a95d85:::benchmark_data_smithi194_121512_object28719:head# 2024-04-29T14:26:36.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62a98e34:::benchmark_data_smithi194_121512_object17524:head# 2024-04-29T14:26:36.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62aa9058:::benchmark_data_smithi194_121512_object69290:head# 2024-04-29T14:26:36.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ab5465:::benchmark_data_smithi194_121512_object8392:head# 2024-04-29T14:26:36.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62abacd2:::benchmark_data_smithi194_121512_object92706:head# 2024-04-29T14:26:36.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62abd08a:::benchmark_data_smithi194_121512_object124116:head# 2024-04-29T14:26:36.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ac0b32:::benchmark_data_smithi194_121512_object35826:head# 2024-04-29T14:26:36.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ac6657:::benchmark_data_smithi194_121512_object34160:head# 2024-04-29T14:26:36.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ad63e8:::benchmark_data_smithi194_121512_object7534:head# 2024-04-29T14:26:36.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ae5178:::benchmark_data_smithi194_121512_object40755:head# 2024-04-29T14:26:36.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62af4ba3:::benchmark_data_smithi194_121512_object38944:head# 2024-04-29T14:26:36.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62af7925:::benchmark_data_smithi194_121512_object62502:head# 2024-04-29T14:26:36.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b057af:::benchmark_data_smithi194_121512_object98270:head# 2024-04-29T14:26:36.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b0e23d:::benchmark_data_smithi194_121512_object117067:head# 2024-04-29T14:26:36.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b197da:::benchmark_data_smithi194_121512_object117439:head# 2024-04-29T14:26:36.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b33881:::benchmark_data_smithi194_121512_object91005:head# 2024-04-29T14:26:36.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b46166:::benchmark_data_smithi194_121512_object96498:head# 2024-04-29T14:26:36.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b48a8d:::benchmark_data_smithi194_121512_object127255:head# 2024-04-29T14:26:36.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b4e42c:::benchmark_data_smithi194_121512_object97611:head# 2024-04-29T14:26:36.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b4f49b:::benchmark_data_smithi194_121512_object45945:head# 2024-04-29T14:26:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:36.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b4fcb6:::benchmark_data_smithi194_121512_object37938:head# 2024-04-29T14:26:36.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b55b07:::benchmark_data_smithi194_121512_object49924:head# 2024-04-29T14:26:36.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b57a78:::benchmark_data_smithi194_121512_object30309:head# 2024-04-29T14:26:36.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b5835e:::benchmark_data_smithi194_121512_object84831:head# 2024-04-29T14:26:36.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b5dcc9:::benchmark_data_smithi194_121512_object10185:head# 2024-04-29T14:26:36.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b68a03:::benchmark_data_smithi194_121512_object125061:head# 2024-04-29T14:26:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:36.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:36.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b743e2:::benchmark_data_smithi194_121512_object17811:head# 2024-04-29T14:26:36.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b7b638:::benchmark_data_smithi194_121512_object26286:head# 2024-04-29T14:26:36.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b8644c:::benchmark_data_smithi194_121512_object94798:head# 2024-04-29T14:26:36.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b88665:::benchmark_data_smithi194_121512_object121544:head# 2024-04-29T14:26:36.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b8cfa1:::benchmark_data_smithi194_121512_object78495:head# 2024-04-29T14:26:36.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b9356d:::benchmark_data_smithi194_121512_object5090:head# 2024-04-29T14:26:36.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b94269:::benchmark_data_smithi194_121512_object19700:head# 2024-04-29T14:26:36.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62b9c628:::benchmark_data_smithi194_121512_object72499:head# 2024-04-29T14:26:36.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ba72f4:::benchmark_data_smithi194_121512_object24377:head# 2024-04-29T14:26:36.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ba898c:::benchmark_data_smithi194_121512_object24311:head# 2024-04-29T14:26:36.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62bb5f16:::benchmark_data_smithi194_121512_object19890:head# 2024-04-29T14:26:36.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62bc577f:::benchmark_data_smithi194_121512_object58127:head# 2024-04-29T14:26:36.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62bc7eea:::benchmark_data_smithi194_121512_object23686:head# 2024-04-29T14:26:36.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62bcf7b8:::benchmark_data_smithi194_121512_object17466:head# 2024-04-29T14:26:36.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62bd6cdd:::benchmark_data_smithi194_121512_object98374:head# 2024-04-29T14:26:36.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62bdf7ed:::benchmark_data_smithi194_121512_object115295:head# 2024-04-29T14:26:36.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62beb2d8:::benchmark_data_smithi194_121512_object35293:head# 2024-04-29T14:26:36.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62bedb0a:::benchmark_data_smithi194_121512_object82657:head# 2024-04-29T14:26:36.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c00cc1:::benchmark_data_smithi194_121512_object33806:head# 2024-04-29T14:26:36.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c05d70:::benchmark_data_smithi194_121512_object43975:head# 2024-04-29T14:26:36.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c0a342:::benchmark_data_smithi194_121512_object127324:head# 2024-04-29T14:26:36.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c11de4:::benchmark_data_smithi194_121512_object116895:head# 2024-04-29T14:26:36.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c13dd1:::benchmark_data_smithi194_121512_object69606:head# 2024-04-29T14:26:36.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c15a62:::benchmark_data_smithi194_121512_object1768:head# 2024-04-29T14:26:36.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c18404:::benchmark_data_smithi194_121512_object131182:head# 2024-04-29T14:26:36.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c18f90:::benchmark_data_smithi194_121512_object128645:head# 2024-04-29T14:26:36.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c1a0c1:::benchmark_data_smithi194_121512_object58685:head# 2024-04-29T14:26:36.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c219cf:::benchmark_data_smithi194_121512_object132537:head# 2024-04-29T14:26:36.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c233f2:::benchmark_data_smithi194_121512_object15082:head# 2024-04-29T14:26:36.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c244e7:::benchmark_data_smithi194_121512_object36859:head# 2024-04-29T14:26:36.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c2bc1a:::benchmark_data_smithi194_121512_object86153:head# 2024-04-29T14:26:36.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c2e3e1:::benchmark_data_smithi194_121512_object36243:head# 2024-04-29T14:26:36.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c312bd:::benchmark_data_smithi194_121512_object13590:head# 2024-04-29T14:26:36.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c3c434:::benchmark_data_smithi194_121512_object98437:head# 2024-04-29T14:26:36.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c42fd0:::benchmark_data_smithi194_121512_object132308:head# 2024-04-29T14:26:36.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c482b4:::benchmark_data_smithi194_121512_object123816:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c54a8d:::benchmark_data_smithi194_121512_object115808:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c670b1:::benchmark_data_smithi194_121512_object61985:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c899ea:::benchmark_data_smithi194_121512_object55335:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62c987dc:::benchmark_data_smithi194_121512_object12660:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62cc229d:::benchmark_data_smithi194_121512_object62331:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62cc27b3:::benchmark_data_smithi194_121512_object94216:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62cd1cec:::benchmark_data_smithi194_121512_object95963:head# 2024-04-29T14:26:36.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ce4537:::benchmark_data_smithi194_121512_object71018:head# 2024-04-29T14:26:36.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62cfca34:::benchmark_data_smithi194_121512_object117335:head# 2024-04-29T14:26:36.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62cffc56:::benchmark_data_smithi194_121512_object73323:head# 2024-04-29T14:26:36.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d16054:::benchmark_data_smithi194_121512_object88766:head# 2024-04-29T14:26:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d161cd:::benchmark_data_smithi194_121512_object107492:head# 2024-04-29T14:26:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d1ae9f:::benchmark_data_smithi194_121512_object102472:head# 2024-04-29T14:26:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d1c4c6:::benchmark_data_smithi194_121512_object50458:head# 2024-04-29T14:26:36.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d1f208:::benchmark_data_smithi194_121512_object6408:head# 2024-04-29T14:26:36.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d2051c:::benchmark_data_smithi194_121512_object62192:head# 2024-04-29T14:26:36.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d2df60:::benchmark_data_smithi194_121512_object112821:head# 2024-04-29T14:26:36.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d2e1f0:::benchmark_data_smithi194_121512_object48010:head# 2024-04-29T14:26:36.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d2ef6e:::benchmark_data_smithi194_121512_object77729:head# 2024-04-29T14:26:36.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d333e1:::benchmark_data_smithi194_121512_object64907:head# 2024-04-29T14:26:36.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d3a3ff:::benchmark_data_smithi194_121512_object84917:head# 2024-04-29T14:26:36.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d40ef4:::benchmark_data_smithi194_121512_object128598:head# 2024-04-29T14:26:36.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d56781:::benchmark_data_smithi194_121512_object27945:head# 2024-04-29T14:26:36.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d5a490:::benchmark_data_smithi194_121512_object8643:head# 2024-04-29T14:26:36.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d5d0d4:::benchmark_data_smithi194_121512_object18232:head# 2024-04-29T14:26:36.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d61ee2:::benchmark_data_smithi194_121512_object88107:head# 2024-04-29T14:26:36.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d62011:::benchmark_data_smithi194_121512_object26122:head# 2024-04-29T14:26:36.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d67b48:::benchmark_data_smithi194_121512_object98856:head# 2024-04-29T14:26:36.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d680aa:::benchmark_data_smithi194_121512_object109056:head# 2024-04-29T14:26:36.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d738ba:::benchmark_data_smithi194_121512_object2832:head# 2024-04-29T14:26:36.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d7a62f:::benchmark_data_smithi194_121512_object33035:head# 2024-04-29T14:26:36.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d7e22e:::benchmark_data_smithi194_121512_object82117:head# 2024-04-29T14:26:36.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d8204c:::benchmark_data_smithi194_121512_object55276:head# 2024-04-29T14:26:36.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d97cd9:::benchmark_data_smithi194_121512_object94356:head# 2024-04-29T14:26:36.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62d9ccd5:::benchmark_data_smithi194_121512_object7658:head# 2024-04-29T14:26:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62da0903:::benchmark_data_smithi194_121512_object97633:head# 2024-04-29T14:26:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62da318e:::benchmark_data_smithi194_121512_object102841:head# 2024-04-29T14:26:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62da95b8:::benchmark_data_smithi194_121512_object48348:head# 2024-04-29T14:26:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62db0341:::benchmark_data_smithi194_121512_object3241:head# 2024-04-29T14:26:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62db10f6:::benchmark_data_smithi194_121512_object123626:head# 2024-04-29T14:26:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62dbf65a:::benchmark_data_smithi194_121512_object10363:head# 2024-04-29T14:26:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62dc72c3:::benchmark_data_smithi194_121512_object71290:head# 2024-04-29T14:26:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62dd94e0:::benchmark_data_smithi194_121512_object4305:head# 2024-04-29T14:26:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ded6e0:::benchmark_data_smithi194_121512_object94675:head# 2024-04-29T14:26:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62dfa07a:::benchmark_data_smithi194_121512_object80933:head# 2024-04-29T14:26:36.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e03ef0:::benchmark_data_smithi194_121512_object25967:head# 2024-04-29T14:26:36.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e05ce0:::benchmark_data_smithi194_121512_object50842:head# 2024-04-29T14:26:36.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e10e02:::benchmark_data_smithi194_121512_object48120:head# 2024-04-29T14:26:36.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e1561f:::benchmark_data_smithi194_121512_object59185:head# 2024-04-29T14:26:36.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e226ec:::benchmark_data_smithi194_121512_object78344:head# 2024-04-29T14:26:36.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e23bde:::benchmark_data_smithi194_121512_object127445:head# 2024-04-29T14:26:36.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e282a9:::benchmark_data_smithi194_121512_object117576:head# 2024-04-29T14:26:36.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e28ea3:::benchmark_data_smithi194_121512_object2713:head# 2024-04-29T14:26:36.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e2b496:::benchmark_data_smithi194_121512_object60213:head# 2024-04-29T14:26:36.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e60129:::benchmark_data_smithi194_121512_object55643:head# 2024-04-29T14:26:36.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e6b341:::benchmark_data_smithi194_121512_object126998:head# 2024-04-29T14:26:36.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e6c71b:::benchmark_data_smithi194_121512_object115484:head# 2024-04-29T14:26:36.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e6e994:::benchmark_data_smithi194_121512_object132419:head# 2024-04-29T14:26:36.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e71b8d:::benchmark_data_smithi194_121512_object104921:head# 2024-04-29T14:26:36.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e794bb:::benchmark_data_smithi194_121512_object128921:head# 2024-04-29T14:26:36.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e8f741:::benchmark_data_smithi194_121512_object94447:head# 2024-04-29T14:26:36.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62e92cc9:::benchmark_data_smithi194_121512_object13377:head# 2024-04-29T14:26:36.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ea731e:::benchmark_data_smithi194_121512_object85795:head# 2024-04-29T14:26:36.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ebe374:::benchmark_data_smithi194_121512_object100910:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ecb9ac:::benchmark_data_smithi194_121512_object78156:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62eccf9e:::benchmark_data_smithi194_121512_object10657:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ed5adf:::benchmark_data_smithi194_121512_object81776:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62eda8b9:::benchmark_data_smithi194_121512_object22892:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62edab1d:::benchmark_data_smithi194_121512_object79186:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ee0691:::benchmark_data_smithi194_121512_object124872:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ee24f4:::benchmark_data_smithi194_121512_object72165:head# 2024-04-29T14:26:36.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ef0ff6:::benchmark_data_smithi194_121512_object113182:head# 2024-04-29T14:26:36.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ef45b9:::benchmark_data_smithi194_121512_object41984:head# 2024-04-29T14:26:36.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62efa4ed:::benchmark_data_smithi194_121512_object8057:head# 2024-04-29T14:26:36.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62efcc2d:::benchmark_data_smithi194_121512_object71140:head# 2024-04-29T14:26:36.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f03ee4:::benchmark_data_smithi194_121512_object95212:head# 2024-04-29T14:26:36.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f04358:::benchmark_data_smithi194_121512_object127378:head# 2024-04-29T14:26:36.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f085ab:::benchmark_data_smithi194_121512_object80900:head# 2024-04-29T14:26:36.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f0a7e9:::benchmark_data_smithi194_121512_object61208:head# 2024-04-29T14:26:36.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f177f1:::benchmark_data_smithi194_121512_object126258:head# 2024-04-29T14:26:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f284cc:::benchmark_data_smithi194_121512_object27082:head# 2024-04-29T14:26:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f29102:::benchmark_data_smithi194_121512_object113117:head# 2024-04-29T14:26:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f38971:::benchmark_data_smithi194_121512_object111297:head# 2024-04-29T14:26:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f3c172:::benchmark_data_smithi194_121512_object14295:head# 2024-04-29T14:26:36.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f48810:::benchmark_data_smithi194_121512_object85950:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f4b86f:::benchmark_data_smithi194_121512_object87504:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f50bb7:::benchmark_data_smithi194_121512_object27557:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f62d50:::benchmark_data_smithi194_121512_object40720:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f6a53e:::benchmark_data_smithi194_121512_object94926:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f6d0bc:::benchmark_data_smithi194_121512_object81741:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f6f685:::benchmark_data_smithi194_121512_object15514:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f6fde2:::benchmark_data_smithi194_121512_object132118:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f70232:::benchmark_data_smithi194_121512_object129672:head# 2024-04-29T14:26:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f7050a:::benchmark_data_smithi194_121512_object87721:head# 2024-04-29T14:26:36.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f717f3:::benchmark_data_smithi194_121512_object76827:head# 2024-04-29T14:26:36.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f7b255:::benchmark_data_smithi194_121512_object91674:head# 2024-04-29T14:26:36.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f80154:::benchmark_data_smithi194_121512_object109809:head# 2024-04-29T14:26:36.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f82fba:::benchmark_data_smithi194_121512_object8833:head# 2024-04-29T14:26:36.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62f85558:::benchmark_data_smithi194_121512_object15916:head# 2024-04-29T14:26:36.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fa6033:::benchmark_data_smithi194_121512_object58122:head# 2024-04-29T14:26:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fac8f9:::benchmark_data_smithi194_121512_object14103:head# 2024-04-29T14:26:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fafda2:::benchmark_data_smithi194_121512_object26766:head# 2024-04-29T14:26:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fb203d:::benchmark_data_smithi194_121512_object50916:head# 2024-04-29T14:26:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fb21ed:::benchmark_data_smithi194_121512_object9230:head# 2024-04-29T14:26:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fb3b2e:::benchmark_data_smithi194_121512_object85963:head# 2024-04-29T14:26:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fbcace:::benchmark_data_smithi194_121512_object53852:head# 2024-04-29T14:26:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fbea09:::benchmark_data_smithi194_121512_object100932:head# 2024-04-29T14:26:36.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fc1ee3:::benchmark_data_smithi194_121512_object9039:head# 2024-04-29T14:26:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fc8ac8:::benchmark_data_smithi194_121512_object94463:head# 2024-04-29T14:26:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fc8ce0:::benchmark_data_smithi194_121512_object43727:head# 2024-04-29T14:26:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fcccd2:::benchmark_data_smithi194_121512_object29607:head# 2024-04-29T14:26:36.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fd0c82:::benchmark_data_smithi194_121512_object71631:head# 2024-04-29T14:26:36.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fd593d:::benchmark_data_smithi194_121512_object121910:head# 2024-04-29T14:26:36.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fde262:::benchmark_data_smithi194_121512_object7938:head# 2024-04-29T14:26:36.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fe0eaa:::benchmark_data_smithi194_121512_object104662:head# 2024-04-29T14:26:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62fe1407:::benchmark_data_smithi194_121512_object48688:head# 2024-04-29T14:26:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ff6177:::benchmark_data_smithi194_121512_object108433:head# 2024-04-29T14:26:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:62ffb8f8:::benchmark_data_smithi194_121512_object75419:head# 2024-04-29T14:26:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63000955:::benchmark_data_smithi194_121512_object51238:head# 2024-04-29T14:26:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6300edb4:::benchmark_data_smithi194_121512_object94478:head# 2024-04-29T14:26:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63031c39:::benchmark_data_smithi194_121512_object106998:head# 2024-04-29T14:26:36.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63037a0a:::benchmark_data_smithi194_121512_object44905:head# 2024-04-29T14:26:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6304ad82:::benchmark_data_smithi194_121512_object21085:head# 2024-04-29T14:26:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6304c67c:::benchmark_data_smithi194_121512_object57640:head# 2024-04-29T14:26:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630527c6:::benchmark_data_smithi194_121512_object34245:head# 2024-04-29T14:26:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63056322:::benchmark_data_smithi194_121512_object58242:head# 2024-04-29T14:26:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6306fcea:::benchmark_data_smithi194_121512_object104281:head# 2024-04-29T14:26:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63078ed8:::benchmark_data_smithi194_121512_object80861:head# 2024-04-29T14:26:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63096109:::benchmark_data_smithi194_121512_object124974:head# 2024-04-29T14:26:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6309c768:::benchmark_data_smithi194_121512_object65942:head# 2024-04-29T14:26:36.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630a15c2:::benchmark_data_smithi194_121512_object37016:head# 2024-04-29T14:26:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630a6924:::benchmark_data_smithi194_121512_object27556:head# 2024-04-29T14:26:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630b090b:::benchmark_data_smithi194_121512_object43764:head# 2024-04-29T14:26:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630b3b86:::benchmark_data_smithi194_121512_object5000:head# 2024-04-29T14:26:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630bf406:::benchmark_data_smithi194_121512_object63359:head# 2024-04-29T14:26:36.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630c31bd:::benchmark_data_smithi194_121512_object46161:head# 2024-04-29T14:26:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630d00ed:::benchmark_data_smithi194_121512_object99756:head# 2024-04-29T14:26:36.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630db6be:::benchmark_data_smithi194_121512_object80781:head# 2024-04-29T14:26:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630ebc37:::benchmark_data_smithi194_121512_object68116:head# 2024-04-29T14:26:36.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630f512f:::benchmark_data_smithi194_121512_object68779:head# 2024-04-29T14:26:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:630f74f7:::benchmark_data_smithi194_121512_object100406:head# 2024-04-29T14:26:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6310889d:::benchmark_data_smithi194_121512_object30582:head# 2024-04-29T14:26:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6310a3a1:::benchmark_data_smithi194_121512_object96145:head# 2024-04-29T14:26:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6310ceb9:::benchmark_data_smithi194_121512_object32303:head# 2024-04-29T14:26:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63110914:::benchmark_data_smithi194_121512_object15890:head# 2024-04-29T14:26:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63112646:::benchmark_data_smithi194_121512_object128286:head# 2024-04-29T14:26:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63122170:::benchmark_data_smithi194_121512_object53154:head# 2024-04-29T14:26:36.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6313279d:::benchmark_data_smithi194_121512_object5324:head# 2024-04-29T14:26:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6313366b:::benchmark_data_smithi194_121512_object82594:head# 2024-04-29T14:26:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6314828e:::benchmark_data_smithi194_121512_object118178:head# 2024-04-29T14:26:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631489af:::benchmark_data_smithi194_121512_object10888:head# 2024-04-29T14:26:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6314c7fd:::benchmark_data_smithi194_121512_object32493:head# 2024-04-29T14:26:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63152da1:::benchmark_data_smithi194_121512_object58917:head# 2024-04-29T14:26:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63152ed1:::benchmark_data_smithi194_121512_object103905:head# 2024-04-29T14:26:36.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63159941:::benchmark_data_smithi194_121512_object81413:head# 2024-04-29T14:26:36.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6315aa35:::benchmark_data_smithi194_121512_object11452:head# 2024-04-29T14:26:36.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6316fb3b:::benchmark_data_smithi194_121512_object8636:head# 2024-04-29T14:26:36.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63179126:::benchmark_data_smithi194_121512_object110020:head# 2024-04-29T14:26:36.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6318ebba:::benchmark_data_smithi194_121512_object115891:head# 2024-04-29T14:26:36.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631911e3:::benchmark_data_smithi194_121512_object63395:head# 2024-04-29T14:26:36.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63197bfc:::benchmark_data_smithi194_121512_object110852:head# 2024-04-29T14:26:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631a09ad:::benchmark_data_smithi194_121512_object81567:head# 2024-04-29T14:26:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631aed62:::benchmark_data_smithi194_121512_object67238:head# 2024-04-29T14:26:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631bc011:::benchmark_data_smithi194_121512_object129614:head# 2024-04-29T14:26:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631cc97c:::benchmark_data_smithi194_121512_object61784:head# 2024-04-29T14:26:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631cd7bc:::benchmark_data_smithi194_121512_object76730:head# 2024-04-29T14:26:36.195 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631d773e:::benchmark_data_smithi194_121512_object59451:head# 2024-04-29T14:26:36.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631dde59:::benchmark_data_smithi194_121512_object123890:head# 2024-04-29T14:26:36.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631e079e:::benchmark_data_smithi194_121512_object29334:head# 2024-04-29T14:26:36.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631e359e:::benchmark_data_smithi194_121512_object37549:head# 2024-04-29T14:26:36.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:631efb0f:::benchmark_data_smithi194_121512_object73437:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6320367e:::benchmark_data_smithi194_121512_object53926:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6321756a:::benchmark_data_smithi194_121512_object52464:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63219ec7:::benchmark_data_smithi194_121512_object132267:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6321d835:::benchmark_data_smithi194_121512_object67442:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632213b3:::benchmark_data_smithi194_121512_object35697:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632262b9:::benchmark_data_smithi194_121512_object126704:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6322f43f:::benchmark_data_smithi194_121512_object23971:head# 2024-04-29T14:26:36.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632375c8:::benchmark_data_smithi194_121512_object56252:head# 2024-04-29T14:26:36.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632378e3:::benchmark_data_smithi194_121512_object110459:head# 2024-04-29T14:26:36.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632380ad:::benchmark_data_smithi194_121512_object113579:head# 2024-04-29T14:26:36.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6323e692:::benchmark_data_smithi194_121512_object87362:head# 2024-04-29T14:26:36.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632483b7:::benchmark_data_smithi194_121512_object9759:head# 2024-04-29T14:26:36.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632514f6:::benchmark_data_smithi194_121512_object43833:head# 2024-04-29T14:26:36.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632618d7:::benchmark_data_smithi194_121512_object34846:head# 2024-04-29T14:26:36.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63263ba2:::benchmark_data_smithi194_121512_object124608:head# 2024-04-29T14:26:36.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63266e0d:::benchmark_data_smithi194_121512_object85019:head# 2024-04-29T14:26:36.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63272649:::benchmark_data_smithi194_121512_object109973:head# 2024-04-29T14:26:36.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63277230:::benchmark_data_smithi194_121512_object115771:head# 2024-04-29T14:26:36.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63281e65:::benchmark_data_smithi194_121512_object29237:head# 2024-04-29T14:26:36.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63287f13:::benchmark_data_smithi194_121512_object66979:head# 2024-04-29T14:26:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63289aff:::benchmark_data_smithi194_121512_object58282:head# 2024-04-29T14:26:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6329a665:::benchmark_data_smithi194_121512_object97915:head# 2024-04-29T14:26:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632ab229:::benchmark_data_smithi194_121512_object40836:head# 2024-04-29T14:26:36.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632b0153:::benchmark_data_smithi194_121512_object60267:head# 2024-04-29T14:26:36.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632b73ed:::benchmark_data_smithi194_121512_object54540:head# 2024-04-29T14:26:36.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632b9cea:::benchmark_data_smithi194_121512_object79607:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632c3884:::benchmark_data_smithi194_121512_object4740:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632cc1d7:::benchmark_data_smithi194_121512_object70901:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632dca0f:::benchmark_data_smithi194_121512_object92427:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632dd10b:::benchmark_data_smithi194_121512_object37684:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632de8a4:::benchmark_data_smithi194_121512_object46862:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632e6ae9:::benchmark_data_smithi194_121512_object78440:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632f7df4:::benchmark_data_smithi194_121512_object97861:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:632fc79c:::benchmark_data_smithi194_121512_object118025:head# 2024-04-29T14:26:36.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6330b789:::benchmark_data_smithi194_121512_object58173:head# 2024-04-29T14:26:36.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63310b04:::benchmark_data_smithi194_121512_object130674:head# 2024-04-29T14:26:36.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63317c4f:::benchmark_data_smithi194_121512_object63591:head# 2024-04-29T14:26:36.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6331f872:::benchmark_data_smithi194_121512_object93072:head# 2024-04-29T14:26:36.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63321ec0:::benchmark_data_smithi194_121512_object97581:head# 2024-04-29T14:26:36.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6332d8ca:::benchmark_data_smithi194_121512_object67359:head# 2024-04-29T14:26:36.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6334df91:::benchmark_data_smithi194_121512_object36772:head# 2024-04-29T14:26:36.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63351c75:::benchmark_data_smithi194_121512_object27753:head# 2024-04-29T14:26:36.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6335a8ca:::benchmark_data_smithi194_121512_object122227:head# 2024-04-29T14:26:36.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63363107:::benchmark_data_smithi194_121512_object46832:head# 2024-04-29T14:26:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63367abc:::benchmark_data_smithi194_121512_object109025:head# 2024-04-29T14:26:36.227 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3241230645' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:36.227 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:36.227 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:36.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6336c79e:::benchmark_data_smithi194_121512_object97753:head# 2024-04-29T14:26:36.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63376115:::benchmark_data_smithi194_121512_object81560:head# 2024-04-29T14:26:36.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63378e6b:::benchmark_data_smithi194_121512_object113645:head# 2024-04-29T14:26:36.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63389b2e:::benchmark_data_smithi194_121512_object122306:head# 2024-04-29T14:26:36.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6338b459:::benchmark_data_smithi194_121512_object69497:head# 2024-04-29T14:26:36.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6338d139:::benchmark_data_smithi194_121512_object100636:head# 2024-04-29T14:26:36.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6338e8ad:::benchmark_data_smithi194_121512_object99835:head# 2024-04-29T14:26:36.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633a1023:::benchmark_data_smithi194_121512_object79158:head# 2024-04-29T14:26:36.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633a8b94:::benchmark_data_smithi194_121512_object111901:head# 2024-04-29T14:26:36.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633ac809:::benchmark_data_smithi194_121512_object63639:head# 2024-04-29T14:26:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:26:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633af042:::benchmark_data_smithi194_121512_object95561:head# 2024-04-29T14:26:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633b0f99:::benchmark_data_smithi194_121512_object90875:head# 2024-04-29T14:26:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633b75ba:::benchmark_data_smithi194_121512_object106103:head# 2024-04-29T14:26:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633bf2f9:::benchmark_data_smithi194_121512_object69377:head# 2024-04-29T14:26:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633c1723:::benchmark_data_smithi194_121512_object131584:head# 2024-04-29T14:26:36.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633c7a60:::benchmark_data_smithi194_121512_object92925:head# 2024-04-29T14:26:36.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633cf6ba:::benchmark_data_smithi194_121512_object96807:head# 2024-04-29T14:26:36.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633e7962:::benchmark_data_smithi194_121512_object105893:head# 2024-04-29T14:26:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633f84e0:::benchmark_data_smithi194_121512_object122263:head# 2024-04-29T14:26:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633f8a1d:::benchmark_data_smithi194_121512_object62720:head# 2024-04-29T14:26:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633fd514:::benchmark_data_smithi194_121512_object120732:head# 2024-04-29T14:26:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:633fef8d:::benchmark_data_smithi194_121512_object63970:head# 2024-04-29T14:26:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6340f4b4:::benchmark_data_smithi194_121512_object66690:head# 2024-04-29T14:26:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6341626b:::benchmark_data_smithi194_121512_object96063:head# 2024-04-29T14:26:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6341a83a:::benchmark_data_smithi194_121512_object112951:head# 2024-04-29T14:26:36.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6341b2f5:::benchmark_data_smithi194_121512_object8595:head# 2024-04-29T14:26:36.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6341e494:::benchmark_data_smithi194_121512_object52223:head# 2024-04-29T14:26:36.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6342856d:::benchmark_data_smithi194_121512_object86465:head# 2024-04-29T14:26:36.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6343ac54:::benchmark_data_smithi194_121512_object132796:head# 2024-04-29T14:26:36.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63448093:::benchmark_data_smithi194_121512_object94661:head# 2024-04-29T14:26:36.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6344a292:::benchmark_data_smithi194_121512_object78427:head# 2024-04-29T14:26:36.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63450954:::benchmark_data_smithi194_121512_object109565:head# 2024-04-29T14:26:36.248 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:36.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63451a19:::benchmark_data_smithi194_121512_object54445:head# 2024-04-29T14:26:36.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6345d1ca:::benchmark_data_smithi194_121512_object123709:head# 2024-04-29T14:26:36.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63460aed:::benchmark_data_smithi194_121512_object77762:head# 2024-04-29T14:26:36.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634621d4:::benchmark_data_smithi194_121512_object47871:head# 2024-04-29T14:26:36.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63465333:::benchmark_data_smithi194_121512_object113721:head# 2024-04-29T14:26:36.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6346ac81:::benchmark_data_smithi194_121512_object73644:head# 2024-04-29T14:26:36.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6346b2c5:::benchmark_data_smithi194_121512_object23046:head# 2024-04-29T14:26:36.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6346dcf8:::benchmark_data_smithi194_121512_object2340:head# 2024-04-29T14:26:36.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63475150:::benchmark_data_smithi194_121512_object98624:head# 2024-04-29T14:26:36.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634775e0:::benchmark_data_smithi194_121512_object115527:head# 2024-04-29T14:26:36.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6349dbf9:::benchmark_data_smithi194_121512_object99133:head# 2024-04-29T14:26:36.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634ada4a:::benchmark_data_smithi194_121512_object49629:head# 2024-04-29T14:26:36.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634b3d7b:::benchmark_data_smithi194_121512_object34412:head# 2024-04-29T14:26:36.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634bd299:::benchmark_data_smithi194_121512_object92295:head# 2024-04-29T14:26:36.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634c3edd:::benchmark_data_smithi194_121512_object18782:head# 2024-04-29T14:26:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634cd29b:::benchmark_data_smithi194_121512_object131806:head# 2024-04-29T14:26:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634d178a:::benchmark_data_smithi194_121512_object98883:head# 2024-04-29T14:26:36.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634d81fb:::benchmark_data_smithi194_121512_object76510:head# 2024-04-29T14:26:36.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634dca65:::benchmark_data_smithi194_121512_object111176:head# 2024-04-29T14:26:36.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634dcd93:::benchmark_data_smithi194_121512_object11563:head# 2024-04-29T14:26:36.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634e422c:::benchmark_data_smithi194_121512_object72519:head# 2024-04-29T14:26:36.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634f445a:::benchmark_data_smithi194_121512_object89724:head# 2024-04-29T14:26:36.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:634f8449:::benchmark_data_smithi194_121512_object7433:head# 2024-04-29T14:26:36.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6350100b:::benchmark_data_smithi194_121512_object107762:head# 2024-04-29T14:26:36.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63503d37:::benchmark_data_smithi194_121512_object81028:head# 2024-04-29T14:26:36.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635040c1:::benchmark_data_smithi194_121512_object40158:head# 2024-04-29T14:26:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63510e69:::benchmark_data_smithi194_121512_object29005:head# 2024-04-29T14:26:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635182e7:::benchmark_data_smithi194_121512_object124080:head# 2024-04-29T14:26:36.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6351ca16:::benchmark_data_smithi194_121512_object74342:head# 2024-04-29T14:26:36.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63521ba4:::benchmark_data_smithi194_121512_object84952:head# 2024-04-29T14:26:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63529baa:::benchmark_data_smithi194_121512_object17695:head# 2024-04-29T14:26:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6352eccc:::benchmark_data_smithi194_121512_object130395:head# 2024-04-29T14:26:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63532ce6:::benchmark_data_smithi194_121512_object92867:head# 2024-04-29T14:26:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6353dc7b:::benchmark_data_smithi194_121512_object22323:head# 2024-04-29T14:26:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6354a67e:::benchmark_data_smithi194_121512_object45765:head# 2024-04-29T14:26:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6354c9e1:::benchmark_data_smithi194_121512_object90389:head# 2024-04-29T14:26:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635569a2:::benchmark_data_smithi194_121512_object93122:head# 2024-04-29T14:26:36.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63559cbb:::benchmark_data_smithi194_121512_object121493:head# 2024-04-29T14:26:36.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6355e5ae:::benchmark_data_smithi194_121512_object53594:head# 2024-04-29T14:26:36.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6355f367:::benchmark_data_smithi194_121512_object43546:head# 2024-04-29T14:26:36.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6356f277:::benchmark_data_smithi194_121512_object24616:head# 2024-04-29T14:26:36.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63576233:::benchmark_data_smithi194_121512_object127052:head# 2024-04-29T14:26:36.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63576df4:::benchmark_data_smithi194_121512_object73257:head# 2024-04-29T14:26:36.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635801d7:::benchmark_data_smithi194_121512_object11149:head# 2024-04-29T14:26:36.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63582201:::benchmark_data_smithi194_121512_object34401:head# 2024-04-29T14:26:36.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63591efb:::benchmark_data_smithi194_121512_object83096:head# 2024-04-29T14:26:36.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6359624f:::benchmark_data_smithi194_121512_object104177:head# 2024-04-29T14:26:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6359bc55:::benchmark_data_smithi194_121512_object22197:head# 2024-04-29T14:26:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635a6cb8:::benchmark_data_smithi194_121512_object122403:head# 2024-04-29T14:26:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635ab7a2:::benchmark_data_smithi194_121512_object39394:head# 2024-04-29T14:26:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635ac381:::benchmark_data_smithi194_121512_object51140:head# 2024-04-29T14:26:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635af05b:::benchmark_data_smithi194_121512_object103254:head# 2024-04-29T14:26:36.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635afcfb:::benchmark_data_smithi194_121512_object126117:head# 2024-04-29T14:26:36.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635bbb8f:::benchmark_data_smithi194_121512_object106933:head# 2024-04-29T14:26:36.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635c0b2e:::benchmark_data_smithi194_121512_object39583:head# 2024-04-29T14:26:36.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635c28a9:::benchmark_data_smithi194_121512_object123250:head# 2024-04-29T14:26:36.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635ccd9a:::benchmark_data_smithi194_121512_object47816:head# 2024-04-29T14:26:36.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635d85e0:::benchmark_data_smithi194_121512_object77966:head# 2024-04-29T14:26:36.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635d9ac1:::benchmark_data_smithi194_121512_object98850:head# 2024-04-29T14:26:36.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635dbb4e:::benchmark_data_smithi194_121512_object61606:head# 2024-04-29T14:26:36.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635dd63b:::benchmark_data_smithi194_121512_object41046:head# 2024-04-29T14:26:36.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635de5ce:::benchmark_data_smithi194_121512_object114153:head# 2024-04-29T14:26:36.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635e0433:::benchmark_data_smithi194_121512_object5245:head# 2024-04-29T14:26:36.282 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:36.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635e269b:::benchmark_data_smithi194_121512_object104689:head# 2024-04-29T14:26:36.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635e5c83:::benchmark_data_smithi194_121512_object112382:head# 2024-04-29T14:26:36.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:635f109f:::benchmark_data_smithi194_121512_object108092:head# 2024-04-29T14:26:36.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636056c4:::benchmark_data_smithi194_121512_object62979:head# 2024-04-29T14:26:36.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6360d63b:::benchmark_data_smithi194_121512_object74899:head# 2024-04-29T14:26:36.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636359d2:::benchmark_data_smithi194_121512_object80488:head# 2024-04-29T14:26:36.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63641bcb:::benchmark_data_smithi194_121512_object29219:head# 2024-04-29T14:26:36.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63647420:::benchmark_data_smithi194_121512_object52990:head# 2024-04-29T14:26:36.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63647c77:::benchmark_data_smithi194_121512_object67080:head# 2024-04-29T14:26:36.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63652430:::benchmark_data_smithi194_121512_object18630:head# 2024-04-29T14:26:36.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63655f85:::benchmark_data_smithi194_121512_object38434:head# 2024-04-29T14:26:36.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63657612:::benchmark_data_smithi194_121512_object67889:head# 2024-04-29T14:26:36.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6365eb27:::benchmark_data_smithi194_121512_object77875:head# 2024-04-29T14:26:36.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63670f65:::benchmark_data_smithi194_121512_object51303:head# 2024-04-29T14:26:36.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6367c911:::benchmark_data_smithi194_121512_object77840:head# 2024-04-29T14:26:36.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63692c4a:::benchmark_data_smithi194_121512_object82261:head# 2024-04-29T14:26:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:36.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63694365:::benchmark_data_smithi194_121512_object53747:head# 2024-04-29T14:26:36.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636963e5:::benchmark_data_smithi194_121512_object121639:head# 2024-04-29T14:26:36.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:36.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636a42e7:::benchmark_data_smithi194_121512_object115107:head# 2024-04-29T14:26:36.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636ae4e8:::benchmark_data_smithi194_121512_object7057:head# 2024-04-29T14:26:36.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636b9781:::benchmark_data_smithi194_121512_object23518:head# 2024-04-29T14:26:36.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636bbeb8:::benchmark_data_smithi194_121512_object27755:head# 2024-04-29T14:26:36.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636caaff:::benchmark_data_smithi194_121512_object57249:head# 2024-04-29T14:26:36.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636cbc8e:::benchmark_data_smithi194_121512_object31188:head# 2024-04-29T14:26:36.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636dcf90:::benchmark_data_smithi194_121512_object61355:head# 2024-04-29T14:26:36.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636e03e2:::benchmark_data_smithi194_121512_object14351:head# 2024-04-29T14:26:36.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636e3ab8:::benchmark_data_smithi194_121512_object101175:head# 2024-04-29T14:26:36.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636e51c6:::benchmark_data_smithi194_121512_object60594:head# 2024-04-29T14:26:36.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636e5339:::benchmark_data_smithi194_121512_object81718:head# 2024-04-29T14:26:36.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:636e5ee4:::benchmark_data_smithi194_121512_object33591:head# 2024-04-29T14:26:36.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6370ba14:::benchmark_data_smithi194_121512_object10693:head# 2024-04-29T14:26:36.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6370d797:::benchmark_data_smithi194_121512_object63131:head# 2024-04-29T14:26:36.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6371621c:::benchmark_data_smithi194_121512_object96773:head# 2024-04-29T14:26:36.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6372b105:::benchmark_data_smithi194_121512_object30304:head# 2024-04-29T14:26:36.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63732ecd:::benchmark_data_smithi194_121512_object130390:head# 2024-04-29T14:26:36.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637375f4:::benchmark_data_smithi194_121512_object62497:head# 2024-04-29T14:26:36.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6373b3a9:::benchmark_data_smithi194_121512_object120680:head# 2024-04-29T14:26:36.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63745c6a:::benchmark_data_smithi194_121512_object76998:head# 2024-04-29T14:26:36.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6374ba71:::benchmark_data_smithi194_121512_object105879:head# 2024-04-29T14:26:36.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6375d811:::benchmark_data_smithi194_121512_object4769:head# 2024-04-29T14:26:36.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63766770:::benchmark_data_smithi194_121512_object71040:head# 2024-04-29T14:26:36.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63767724:::benchmark_data_smithi194_121512_object66827:head# 2024-04-29T14:26:36.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637717ad:::benchmark_data_smithi194_121512_object1958:head# 2024-04-29T14:26:36.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637798e3:::benchmark_data_smithi194_121512_object106297:head# 2024-04-29T14:26:36.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63781ac2:::benchmark_data_smithi194_121512_object113804:head# 2024-04-29T14:26:36.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6378db22:::benchmark_data_smithi194_121512_object48066:head# 2024-04-29T14:26:36.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637944f9:::benchmark_data_smithi194_121512_object10482:head# 2024-04-29T14:26:36.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6379924b:::benchmark_data_smithi194_121512_object107795:head# 2024-04-29T14:26:36.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6379afb7:::benchmark_data_smithi194_121512_object93241:head# 2024-04-29T14:26:36.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6379e3b7:::benchmark_data_smithi194_121512_object31830:head# 2024-04-29T14:26:36.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6379e495:::benchmark_data_smithi194_121512_object71368:head# 2024-04-29T14:26:36.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637a1dfb:::benchmark_data_smithi194_121512_object4412:head# 2024-04-29T14:26:36.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637a3bd8:::benchmark_data_smithi194_121512_object44935:head# 2024-04-29T14:26:36.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637b7411:::benchmark_data_smithi194_121512_object30936:head# 2024-04-29T14:26:36.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637b9967:::benchmark_data_smithi194_121512_object31147:head# 2024-04-29T14:26:36.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637c51f4:::benchmark_data_smithi194_121512_object133237:head# 2024-04-29T14:26:36.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637ca89a:::benchmark_data_smithi194_121512_object80448:head# 2024-04-29T14:26:36.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637da7df:::benchmark_data_smithi194_121512_object11011:head# 2024-04-29T14:26:36.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637db6ff:::benchmark_data_smithi194_121512_object81013:head# 2024-04-29T14:26:36.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637ed515:::benchmark_data_smithi194_121512_object27993:head# 2024-04-29T14:26:36.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637f28b7:::benchmark_data_smithi194_121512_object69884:head# 2024-04-29T14:26:36.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637f6e60:::benchmark_data_smithi194_121512_object60564:head# 2024-04-29T14:26:36.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:637ffa0c:::benchmark_data_smithi194_121512_object102103:head# 2024-04-29T14:26:36.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6381497b:::benchmark_data_smithi194_121512_object28582:head# 2024-04-29T14:26:36.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63815c3a:::benchmark_data_smithi194_121512_object81585:head# 2024-04-29T14:26:36.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63816aff:::benchmark_data_smithi194_121512_object75085:head# 2024-04-29T14:26:36.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638181f2:::benchmark_data_smithi194_121512_object28946:head# 2024-04-29T14:26:36.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6382121b:::benchmark_data_smithi194_121512_object131380:head# 2024-04-29T14:26:36.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63824d4f:::benchmark_data_smithi194_121512_object90619:head# 2024-04-29T14:26:36.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63837725:::benchmark_data_smithi194_121512_object99928:head# 2024-04-29T14:26:36.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6383c676:::benchmark_data_smithi194_121512_object24890:head# 2024-04-29T14:26:36.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6383fe40:::benchmark_data_smithi194_121512_object28281:head# 2024-04-29T14:26:36.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6384025d:::benchmark_data_smithi194_121512_object80022:head# 2024-04-29T14:26:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63840837:::benchmark_data_smithi194_121512_object82495:head# 2024-04-29T14:26:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6384b154:::benchmark_data_smithi194_121512_object128547:head# 2024-04-29T14:26:36.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6384b1a2:::benchmark_data_smithi194_121512_object90460:head# 2024-04-29T14:26:36.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6384b9ad:::benchmark_data_smithi194_121512_object88834:head# 2024-04-29T14:26:36.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6384d5cd:::benchmark_data_smithi194_121512_object14286:head# 2024-04-29T14:26:36.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6384ecdb:::benchmark_data_smithi194_121512_object126189:head# 2024-04-29T14:26:36.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6384f9ff:::benchmark_data_smithi194_121512_object40300:head# 2024-04-29T14:26:36.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638549bc:::benchmark_data_smithi194_121512_object53266:head# 2024-04-29T14:26:36.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63860a36:::benchmark_data_smithi194_121512_object21335:head# 2024-04-29T14:26:36.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6386d937:::benchmark_data_smithi194_121512_object91135:head# 2024-04-29T14:26:36.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63877d59:::benchmark_data_smithi194_121512_object35183:head# 2024-04-29T14:26:36.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6387ece3:::benchmark_data_smithi194_121512_object92740:head# 2024-04-29T14:26:36.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638862ae:::benchmark_data_smithi194_121512_object9001:head# 2024-04-29T14:26:36.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63890918:::benchmark_data_smithi194_121512_object116139:head# 2024-04-29T14:26:36.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638976d1:::benchmark_data_smithi194_121512_object118498:head# 2024-04-29T14:26:36.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638b225b:::benchmark_data_smithi194_121512_object92842:head# 2024-04-29T14:26:36.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638cd8ba:::benchmark_data_smithi194_121512_object63404:head# 2024-04-29T14:26:36.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638d1b23:::benchmark_data_smithi194_121512_object54449:head# 2024-04-29T14:26:36.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638ea9ea:::benchmark_data_smithi194_121512_object41000:head# 2024-04-29T14:26:36.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638eae4f:::benchmark_data_smithi194_121512_object50244:head# 2024-04-29T14:26:36.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:638f7f3c:::benchmark_data_smithi194_121512_object68164:head# 2024-04-29T14:26:36.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63901115:::benchmark_data_smithi194_121512_object17490:head# 2024-04-29T14:26:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63903f4e:::benchmark_data_smithi194_121512_object89880:head# 2024-04-29T14:26:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639066f3:::benchmark_data_smithi194_121512_object125542:head# 2024-04-29T14:26:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6391c2df:::benchmark_data_smithi194_121512_object9233:head# 2024-04-29T14:26:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63926047:::benchmark_data_smithi194_121512_object9384:head# 2024-04-29T14:26:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6392fccd:::benchmark_data_smithi194_121512_object43444:head# 2024-04-29T14:26:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639304d5:::benchmark_data_smithi194_121512_object58711:head# 2024-04-29T14:26:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6393150e:::benchmark_data_smithi194_121512_object106885:head# 2024-04-29T14:26:36.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63938292:::benchmark_data_smithi194_121512_object118636:head# 2024-04-29T14:26:36.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6393e9a3:::benchmark_data_smithi194_121512_object29011:head# 2024-04-29T14:26:36.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6393eb59:::benchmark_data_smithi194_121512_object10551:head# 2024-04-29T14:26:36.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63949075:::benchmark_data_smithi194_121512_object103488:head# 2024-04-29T14:26:36.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6394a215:::benchmark_data_smithi194_121512_object91830:head# 2024-04-29T14:26:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6395d225:::benchmark_data_smithi194_121512_object14310:head# 2024-04-29T14:26:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6395f8c1:::benchmark_data_smithi194_121512_object13701:head# 2024-04-29T14:26:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63960695:::benchmark_data_smithi194_121512_object67668:head# 2024-04-29T14:26:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6397637d:::benchmark_data_smithi194_121512_object84998:head# 2024-04-29T14:26:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6398f24c:::benchmark_data_smithi194_121512_object20978:head# 2024-04-29T14:26:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6398f69f:::benchmark_data_smithi194_121512_object81603:head# 2024-04-29T14:26:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639907be:::benchmark_data_smithi194_121512_object94351:head# 2024-04-29T14:26:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6399a434:::benchmark_data_smithi194_121512_object52621:head# 2024-04-29T14:26:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639b3164:::benchmark_data_smithi194_121512_object79852:head# 2024-04-29T14:26:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639b97ad:::benchmark_data_smithi194_121512_object84810:head# 2024-04-29T14:26:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639b9c2a:::benchmark_data_smithi194_121512_object108148:head# 2024-04-29T14:26:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639ba6ae:::benchmark_data_smithi194_121512_object57163:head# 2024-04-29T14:26:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639bbae8:::benchmark_data_smithi194_121512_object47441:head# 2024-04-29T14:26:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639bc07a:::benchmark_data_smithi194_121512_object101325:head# 2024-04-29T14:26:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3241230645' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:36.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639c96b8:::benchmark_data_smithi194_121512_object68512:head# 2024-04-29T14:26:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639cf621:::benchmark_data_smithi194_121512_object35425:head# 2024-04-29T14:26:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639d20c1:::benchmark_data_smithi194_121512_object8654:head# 2024-04-29T14:26:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639d5891:::benchmark_data_smithi194_121512_object69857:head# 2024-04-29T14:26:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639d5fc2:::benchmark_data_smithi194_121512_object24750:head# 2024-04-29T14:26:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639e2250:::benchmark_data_smithi194_121512_object111852:head# 2024-04-29T14:26:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639ef868:::benchmark_data_smithi194_121512_object5745:head# 2024-04-29T14:26:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639f2888:::benchmark_data_smithi194_121512_object34878:head# 2024-04-29T14:26:36.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:639f51a7:::benchmark_data_smithi194_121512_object1772:head# 2024-04-29T14:26:36.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a074ed:::benchmark_data_smithi194_121512_object16665:head# 2024-04-29T14:26:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a10225:::benchmark_data_smithi194_121512_object79494:head# 2024-04-29T14:26:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a21831:::benchmark_data_smithi194_121512_object54211:head# 2024-04-29T14:26:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a323aa:::benchmark_data_smithi194_121512_object112946:head# 2024-04-29T14:26:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a3e4a0:::benchmark_data_smithi194_121512_object75663:head# 2024-04-29T14:26:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a46a0a:::benchmark_data_smithi194_121512_object116166:head# 2024-04-29T14:26:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a48264:::benchmark_data_smithi194_121512_object109036:head# 2024-04-29T14:26:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a4ba2c:::benchmark_data_smithi194_121512_object112361:head# 2024-04-29T14:26:36.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a524f3:::benchmark_data_smithi194_121512_object94037:head# 2024-04-29T14:26:36.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a55f64:::benchmark_data_smithi194_121512_object45556:head# 2024-04-29T14:26:36.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a643e1:::benchmark_data_smithi194_121512_object7208:head# 2024-04-29T14:26:36.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a6d407:::benchmark_data_smithi194_121512_object43220:head# 2024-04-29T14:26:36.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a6d738:::benchmark_data_smithi194_121512_object51039:head# 2024-04-29T14:26:36.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a812ba:::benchmark_data_smithi194_121512_object5774:head# 2024-04-29T14:26:36.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a83114:::benchmark_data_smithi194_121512_object50179:head# 2024-04-29T14:26:36.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a8ad34:::benchmark_data_smithi194_121512_object99105:head# 2024-04-29T14:26:36.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a8eea2:::benchmark_data_smithi194_121512_object90139:head# 2024-04-29T14:26:36.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a8f065:::benchmark_data_smithi194_121512_object69063:head# 2024-04-29T14:26:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a97698:::benchmark_data_smithi194_121512_object58429:head# 2024-04-29T14:26:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63a9e884:::benchmark_data_smithi194_121512_object103420:head# 2024-04-29T14:26:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63aa0776:::benchmark_data_smithi194_121512_object19340:head# 2024-04-29T14:26:36.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ab8ccf:::benchmark_data_smithi194_121512_object109765:head# 2024-04-29T14:26:36.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63abd129:::benchmark_data_smithi194_121512_object93779:head# 2024-04-29T14:26:36.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ac0bd5:::benchmark_data_smithi194_121512_object78938:head# 2024-04-29T14:26:36.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ac0fcb:::benchmark_data_smithi194_121512_object127469:head# 2024-04-29T14:26:36.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ac85a1:::benchmark_data_smithi194_121512_object116424:head# 2024-04-29T14:26:36.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ad48cc:::benchmark_data_smithi194_121512_object39955:head# 2024-04-29T14:26:36.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ad4aa0:::benchmark_data_smithi194_121512_object67206:head# 2024-04-29T14:26:36.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ad63f4:::benchmark_data_smithi194_121512_object11991:head# 2024-04-29T14:26:36.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63adbc26:::benchmark_data_smithi194_121512_object121932:head# 2024-04-29T14:26:36.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ae3cc8:::benchmark_data_smithi194_121512_object78972:head# 2024-04-29T14:26:36.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ae4e1f:::benchmark_data_smithi194_121512_object105144:head# 2024-04-29T14:26:36.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63af4135:::benchmark_data_smithi194_121512_object25488:head# 2024-04-29T14:26:36.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63af596b:::benchmark_data_smithi194_121512_object16744:head# 2024-04-29T14:26:36.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63afa618:::benchmark_data_smithi194_121512_object74858:head# 2024-04-29T14:26:36.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b1219b:::benchmark_data_smithi194_121512_object25690:head# 2024-04-29T14:26:36.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b12948:::benchmark_data_smithi194_121512_object21464:head# 2024-04-29T14:26:36.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b173f1:::benchmark_data_smithi194_121512_object27145:head# 2024-04-29T14:26:36.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b1d6f4:::benchmark_data_smithi194_121512_object42133:head# 2024-04-29T14:26:36.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b1f25b:::benchmark_data_smithi194_121512_object127355:head# 2024-04-29T14:26:36.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b2a856:::benchmark_data_smithi194_121512_object66093:head# 2024-04-29T14:26:36.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b2be51:::benchmark_data_smithi194_121512_object47225:head# 2024-04-29T14:26:36.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b38a95:::benchmark_data_smithi194_121512_object87875:head# 2024-04-29T14:26:36.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b498de:::benchmark_data_smithi194_121512_object122421:head# 2024-04-29T14:26:36.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b4a196:::benchmark_data_smithi194_121512_object62168:head# 2024-04-29T14:26:36.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b51ecf:::benchmark_data_smithi194_121512_object98162:head# 2024-04-29T14:26:36.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b6ec4f:::benchmark_data_smithi194_121512_object26566:head# 2024-04-29T14:26:36.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b75168:::benchmark_data_smithi194_121512_object79632:head# 2024-04-29T14:26:36.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b85a23:::benchmark_data_smithi194_121512_object8140:head# 2024-04-29T14:26:36.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b93957:::benchmark_data_smithi194_121512_object23556:head# 2024-04-29T14:26:36.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63b9471b:::benchmark_data_smithi194_121512_object116400:head# 2024-04-29T14:26:36.393 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:36.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bac310:::benchmark_data_smithi194_121512_object73250:head# 2024-04-29T14:26:36.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bb0909:::benchmark_data_smithi194_121512_object21606:head# 2024-04-29T14:26:36.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bb1416:::benchmark_data_smithi194_121512_object59894:head# 2024-04-29T14:26:36.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bb3293:::benchmark_data_smithi194_121512_object130830:head# 2024-04-29T14:26:36.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bb6876:::benchmark_data_smithi194_121512_object1805:head# 2024-04-29T14:26:36.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bbc0a7:::benchmark_data_smithi194_121512_object77319:head# 2024-04-29T14:26:36.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bbd185:::benchmark_data_smithi194_121512_object40629:head# 2024-04-29T14:26:36.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bbfadb:::benchmark_data_smithi194_121512_object25834:head# 2024-04-29T14:26:36.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bc87a2:::benchmark_data_smithi194_121512_object37377:head# 2024-04-29T14:26:36.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bd0fed:::benchmark_data_smithi194_121512_object84787:head# 2024-04-29T14:26:36.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bd404e:::benchmark_data_smithi194_121512_object22401:head# 2024-04-29T14:26:36.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bd748d:::benchmark_data_smithi194_121512_object117830:head# 2024-04-29T14:26:36.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bdec19:::benchmark_data_smithi194_121512_object46659:head# 2024-04-29T14:26:36.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63be9ebd:::benchmark_data_smithi194_121512_object96370:head# 2024-04-29T14:26:36.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63bf0f38:::benchmark_data_smithi194_121512_object97052:head# 2024-04-29T14:26:36.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c02e2b:::benchmark_data_smithi194_121512_object52628:head# 2024-04-29T14:26:36.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c070bc:::benchmark_data_smithi194_121512_object89072:head# 2024-04-29T14:26:36.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c096cd:::benchmark_data_smithi194_121512_object571:head# 2024-04-29T14:26:36.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c0b3c1:::benchmark_data_smithi194_121512_object36342:head# 2024-04-29T14:26:36.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c23b46:::benchmark_data_smithi194_121512_object39088:head# 2024-04-29T14:26:36.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c2f52a:::benchmark_data_smithi194_121512_object119935:head# 2024-04-29T14:26:36.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c335b7:::benchmark_data_smithi194_121512_object8606:head# 2024-04-29T14:26:36.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c36182:::benchmark_data_smithi194_121512_object22665:head# 2024-04-29T14:26:36.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c3aec8:::benchmark_data_smithi194_121512_object18457:head# 2024-04-29T14:26:36.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c3c309:::benchmark_data_smithi194_121512_object13890:head# 2024-04-29T14:26:36.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c3e9bc:::benchmark_data_smithi194_121512_object92770:head# 2024-04-29T14:26:36.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c494d7:::benchmark_data_smithi194_121512_object80798:head# 2024-04-29T14:26:36.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c57d74:::benchmark_data_smithi194_121512_object20364:head# 2024-04-29T14:26:36.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c580bb:::benchmark_data_smithi194_121512_object108140:head# 2024-04-29T14:26:36.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c645bd:::benchmark_data_smithi194_121512_object21105:head# 2024-04-29T14:26:36.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c73512:::benchmark_data_smithi194_121512_object126168:head# 2024-04-29T14:26:36.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c78c5a:::benchmark_data_smithi194_121512_object39616:head# 2024-04-29T14:26:36.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c82e87:::benchmark_data_smithi194_121512_object11500:head# 2024-04-29T14:26:36.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c83ac4:::benchmark_data_smithi194_121512_object92632:head# 2024-04-29T14:26:36.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c8a935:::benchmark_data_smithi194_121512_object55400:head# 2024-04-29T14:26:36.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c9654b:::benchmark_data_smithi194_121512_object66157:head# 2024-04-29T14:26:36.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c976ff:::benchmark_data_smithi194_121512_object42963:head# 2024-04-29T14:26:36.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63c9bb9d:::benchmark_data_smithi194_121512_object55963:head# 2024-04-29T14:26:36.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ca0ad7:::benchmark_data_smithi194_121512_object130959:head# 2024-04-29T14:26:36.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ca1a41:::benchmark_data_smithi194_121512_object99027:head# 2024-04-29T14:26:36.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cb08cc:::benchmark_data_smithi194_121512_object7995:head# 2024-04-29T14:26:36.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cc7510:::benchmark_data_smithi194_121512_object16921:head# 2024-04-29T14:26:36.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cc7e94:::benchmark_data_smithi194_121512_object90451:head# 2024-04-29T14:26:36.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cd08be:::benchmark_data_smithi194_121512_object21877:head# 2024-04-29T14:26:36.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cd6f16:::benchmark_data_smithi194_121512_object90458:head# 2024-04-29T14:26:36.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cd829b:::benchmark_data_smithi194_121512_object6370:head# 2024-04-29T14:26:36.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ce075e:::benchmark_data_smithi194_121512_object100955:head# 2024-04-29T14:26:36.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ce31af:::benchmark_data_smithi194_121512_object55084:head# 2024-04-29T14:26:36.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ceb044:::benchmark_data_smithi194_121512_object126444:head# 2024-04-29T14:26:36.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cf454e:::benchmark_data_smithi194_121512_object116548:head# 2024-04-29T14:26:36.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cf7cdc:::benchmark_data_smithi194_121512_object77943:head# 2024-04-29T14:26:36.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63cf7d98:::benchmark_data_smithi194_121512_object47008:head# 2024-04-29T14:26:36.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d06dc2:::benchmark_data_smithi194_121512_object122594:head# 2024-04-29T14:26:36.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d14dbd:::benchmark_data_smithi194_121512_object127645:head# 2024-04-29T14:26:36.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d27937:::benchmark_data_smithi194_121512_object84863:head# 2024-04-29T14:26:36.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d324d8:::benchmark_data_smithi194_121512_object61239:head# 2024-04-29T14:26:36.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d36376:::benchmark_data_smithi194_121512_object83433:head# 2024-04-29T14:26:36.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d3d001:::benchmark_data_smithi194_121512_object45655:head# 2024-04-29T14:26:36.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d3f815:::benchmark_data_smithi194_121512_object77931:head# 2024-04-29T14:26:36.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d4043a:::benchmark_data_smithi194_121512_object96774:head# 2024-04-29T14:26:36.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d536ff:::benchmark_data_smithi194_121512_object42949:head# 2024-04-29T14:26:36.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d54e9a:::benchmark_data_smithi194_121512_object37558:head# 2024-04-29T14:26:36.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d58539:::benchmark_data_smithi194_121512_object91696:head# 2024-04-29T14:26:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d5d565:::benchmark_data_smithi194_121512_object60587:head# 2024-04-29T14:26:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d66a4b:::benchmark_data_smithi194_121512_object23805:head# 2024-04-29T14:26:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d6f63f:::benchmark_data_smithi194_121512_object16678:head# 2024-04-29T14:26:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d7137b:::benchmark_data_smithi194_121512_object62121:head# 2024-04-29T14:26:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d74e8a:::benchmark_data_smithi194_121512_object115844:head# 2024-04-29T14:26:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d78bdc:::benchmark_data_smithi194_121512_object16266:head# 2024-04-29T14:26:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d7a232:::benchmark_data_smithi194_121512_object65914:head# 2024-04-29T14:26:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d7df08:::benchmark_data_smithi194_121512_object1078:head# 2024-04-29T14:26:36.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d863dc:::benchmark_data_smithi194_121512_object109035:head# 2024-04-29T14:26:36.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d896ad:::benchmark_data_smithi194_121512_object74002:head# 2024-04-29T14:26:36.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d8ac30:::benchmark_data_smithi194_121512_object50221:head# 2024-04-29T14:26:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63d9fb45:::benchmark_data_smithi194_121512_object53862:head# 2024-04-29T14:26:36.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63da0e70:::benchmark_data_smithi194_121512_object128329:head# 2024-04-29T14:26:36.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63da51fa:::benchmark_data_smithi194_121512_object92592:head# 2024-04-29T14:26:36.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63da65ad:::benchmark_data_smithi194_121512_object126865:head# 2024-04-29T14:26:36.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63da8c2c:::benchmark_data_smithi194_121512_object44240:head# 2024-04-29T14:26:36.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63daa0ae:::benchmark_data_smithi194_121512_object81817:head# 2024-04-29T14:26:36.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63db7c4c:::benchmark_data_smithi194_121512_object129649:head# 2024-04-29T14:26:36.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63dbb850:::benchmark_data_smithi194_121512_object33567:head# 2024-04-29T14:26:36.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63dbd6e8:::benchmark_data_smithi194_121512_object109451:head# 2024-04-29T14:26:36.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63dc9b69:::benchmark_data_smithi194_121512_object78571:head# 2024-04-29T14:26:36.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63dcc9c3:::benchmark_data_smithi194_121512_object123127:head# 2024-04-29T14:26:36.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63dd2e05:::benchmark_data_smithi194_121512_object126730:head# 2024-04-29T14:26:36.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e16033:::benchmark_data_smithi194_121512_object109096:head# 2024-04-29T14:26:36.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e2699d:::benchmark_data_smithi194_121512_object111991:head# 2024-04-29T14:26:36.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e2b7e9:::benchmark_data_smithi194_121512_object73704:head# 2024-04-29T14:26:36.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e3422d:::benchmark_data_smithi194_121512_object9193:head# 2024-04-29T14:26:36.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e38655:::benchmark_data_smithi194_121512_object38523:head# 2024-04-29T14:26:36.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e3b630:::benchmark_data_smithi194_121512_object87733:head# 2024-04-29T14:26:36.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e40034:::benchmark_data_smithi194_121512_object39948:head# 2024-04-29T14:26:36.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e4659d:::benchmark_data_smithi194_121512_object113823:head# 2024-04-29T14:26:36.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e4ecb0:::benchmark_data_smithi194_121512_object23948:head# 2024-04-29T14:26:36.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e4f370:::benchmark_data_smithi194_121512_object97278:head# 2024-04-29T14:26:36.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e4fc23:::benchmark_data_smithi194_121512_object65661:head# 2024-04-29T14:26:36.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e502a4:::benchmark_data_smithi194_121512_object126331:head# 2024-04-29T14:26:36.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e64865:::benchmark_data_smithi194_121512_object108011:head# 2024-04-29T14:26:36.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e7c234:::benchmark_data_smithi194_121512_object22668:head# 2024-04-29T14:26:36.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e84739:::benchmark_data_smithi194_121512_object120770:head# 2024-04-29T14:26:36.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e84fde:::benchmark_data_smithi194_121512_object6462:head# 2024-04-29T14:26:36.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e85dd0:::benchmark_data_smithi194_121512_object108530:head# 2024-04-29T14:26:36.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e8d6a7:::benchmark_data_smithi194_121512_object22447:head# 2024-04-29T14:26:36.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e965d2:::benchmark_data_smithi194_121512_object35844:head# 2024-04-29T14:26:36.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e9a02d:::benchmark_data_smithi194_121512_object96411:head# 2024-04-29T14:26:36.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63e9d2a7:::benchmark_data_smithi194_121512_object21384:head# 2024-04-29T14:26:36.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ea1a0b:::benchmark_data_smithi194_121512_object32249:head# 2024-04-29T14:26:36.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ea88ba:::benchmark_data_smithi194_121512_object81165:head# 2024-04-29T14:26:36.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ea943c:::benchmark_data_smithi194_121512_object79290:head# 2024-04-29T14:26:36.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63eaa8d2:::benchmark_data_smithi194_121512_object39590:head# 2024-04-29T14:26:36.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63eac077:::benchmark_data_smithi194_121512_object77276:head# 2024-04-29T14:26:36.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63eb7822:::benchmark_data_smithi194_121512_object100992:head# 2024-04-29T14:26:36.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ebfb2e:::benchmark_data_smithi194_121512_object125448:head# 2024-04-29T14:26:36.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ec26e6:::benchmark_data_smithi194_121512_object125446:head# 2024-04-29T14:26:36.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ef394a:::benchmark_data_smithi194_121512_object12924:head# 2024-04-29T14:26:36.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ef8cf3:::benchmark_data_smithi194_121512_object29082:head# 2024-04-29T14:26:36.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f0ab2d:::benchmark_data_smithi194_121512_object79440:head# 2024-04-29T14:26:36.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f0ff8e:::benchmark_data_smithi194_121512_object44607:head# 2024-04-29T14:26:36.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f18198:::benchmark_data_smithi194_121512_object118212:head# 2024-04-29T14:26:36.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f29cf0:::benchmark_data_smithi194_121512_object8518:head# 2024-04-29T14:26:36.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f306dd:::benchmark_data_smithi194_121512_object102222:head# 2024-04-29T14:26:36.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f34a3b:::benchmark_data_smithi194_121512_object19149:head# 2024-04-29T14:26:36.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f382de:::benchmark_data_smithi194_121512_object64602:head# 2024-04-29T14:26:36.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f3d589:::benchmark_data_smithi194_121512_object124648:head# 2024-04-29T14:26:36.446 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:36.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f46a8c:::benchmark_data_smithi194_121512_object44963:head# 2024-04-29T14:26:36.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f55046:::benchmark_data_smithi194_121512_object81958:head# 2024-04-29T14:26:36.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f55746:::benchmark_data_smithi194_121512_object74505:head# 2024-04-29T14:26:36.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f63c8f:::benchmark_data_smithi194_121512_object40202:head# 2024-04-29T14:26:36.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f7bc69:::benchmark_data_smithi194_121512_object8177:head# 2024-04-29T14:26:36.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f7ed91:::benchmark_data_smithi194_121512_object96516:head# 2024-04-29T14:26:36.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63f96547:::benchmark_data_smithi194_121512_object108730:head# 2024-04-29T14:26:36.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fa02c6:::benchmark_data_smithi194_121512_object49723:head# 2024-04-29T14:26:36.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fa35f7:::benchmark_data_smithi194_121512_object103310:head# 2024-04-29T14:26:36.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fa405f:::benchmark_data_smithi194_121512_object99419:head# 2024-04-29T14:26:36.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fa97a1:::benchmark_data_smithi194_121512_object43336:head# 2024-04-29T14:26:36.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fab3a7:::benchmark_data_smithi194_121512_object107841:head# 2024-04-29T14:26:36.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fba485:::benchmark_data_smithi194_121512_object63147:head# 2024-04-29T14:26:36.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fbb905:::benchmark_data_smithi194_121512_object92205:head# 2024-04-29T14:26:36.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fcae6e:::benchmark_data_smithi194_121512_object62293:head# 2024-04-29T14:26:36.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fd1dcc:::benchmark_data_smithi194_121512_object99604:head# 2024-04-29T14:26:36.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fd9c0e:::benchmark_data_smithi194_121512_object94670:head# 2024-04-29T14:26:36.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fe4889:::benchmark_data_smithi194_121512_object71715:head# 2024-04-29T14:26:36.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fe5b06:::benchmark_data_smithi194_121512_object62829:head# 2024-04-29T14:26:36.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fe85a5:::benchmark_data_smithi194_121512_object16016:head# 2024-04-29T14:26:36.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fed532:::benchmark_data_smithi194_121512_object42887:head# 2024-04-29T14:26:36.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ff2b6a:::benchmark_data_smithi194_121512_object97354:head# 2024-04-29T14:26:36.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63ff5f79:::benchmark_data_smithi194_121512_object63218:head# 2024-04-29T14:26:36.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fff8e8:::benchmark_data_smithi194_121512_object26866:head# 2024-04-29T14:26:36.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:63fffeac:::benchmark_data_smithi194_121512_object92376:head# 2024-04-29T14:26:36.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64003101:::benchmark_data_smithi194_121512_object42359:head# 2024-04-29T14:26:36.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640054e4:::benchmark_data_smithi194_121512_object87251:head# 2024-04-29T14:26:36.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6401251f:::benchmark_data_smithi194_121512_object24800:head# 2024-04-29T14:26:36.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64019614:::benchmark_data_smithi194_121512_object53392:head# 2024-04-29T14:26:36.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6401b09f:::benchmark_data_smithi194_121512_object102961:head# 2024-04-29T14:26:36.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64021394:::benchmark_data_smithi194_121512_object91406:head# 2024-04-29T14:26:36.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64022415:::benchmark_data_smithi194_121512_object10589:head# 2024-04-29T14:26:36.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6402931c:::benchmark_data_smithi194_121512_object74456:head# 2024-04-29T14:26:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64045ad3:::benchmark_data_smithi194_121512_object43391:head# 2024-04-29T14:26:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64049d1f:::benchmark_data_smithi194_121512_object58996:head# 2024-04-29T14:26:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6404e5f6:::benchmark_data_smithi194_121512_object101003:head# 2024-04-29T14:26:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640562ae:::benchmark_data_smithi194_121512_object107763:head# 2024-04-29T14:26:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640617fc:::benchmark_data_smithi194_121512_object54000:head# 2024-04-29T14:26:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64069e1c:::benchmark_data_smithi194_121512_object129294:head# 2024-04-29T14:26:36.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6406b78a:::benchmark_data_smithi194_121512_object2659:head# 2024-04-29T14:26:36.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6406c033:::benchmark_data_smithi194_121512_object87853:head# 2024-04-29T14:26:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6407263f:::benchmark_data_smithi194_121512_object76770:head# 2024-04-29T14:26:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640764ba:::benchmark_data_smithi194_121512_object63304:head# 2024-04-29T14:26:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64079245:::benchmark_data_smithi194_121512_object65407:head# 2024-04-29T14:26:36.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6407b836:::benchmark_data_smithi194_121512_object27237:head# 2024-04-29T14:26:36.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6408d545:::benchmark_data_smithi194_121512_object127616:head# 2024-04-29T14:26:36.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6409c4bc:::benchmark_data_smithi194_121512_object60655:head# 2024-04-29T14:26:36.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640a124a:::benchmark_data_smithi194_121512_object60186:head# 2024-04-29T14:26:36.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640a4c23:::benchmark_data_smithi194_121512_object119251:head# 2024-04-29T14:26:36.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640a8f6e:::benchmark_data_smithi194_121512_object62117:head# 2024-04-29T14:26:36.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640ae9cd:::benchmark_data_smithi194_121512_object63034:head# 2024-04-29T14:26:36.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640b1132:::benchmark_data_smithi194_121512_object86858:head# 2024-04-29T14:26:36.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640b2e9b:::benchmark_data_smithi194_121512_object133448:head# 2024-04-29T14:26:36.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640b74e0:::benchmark_data_smithi194_121512_object81270:head# 2024-04-29T14:26:36.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640d646c:::benchmark_data_smithi194_121512_object29593:head# 2024-04-29T14:26:36.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640d8b28:::benchmark_data_smithi194_121512_object129457:head# 2024-04-29T14:26:36.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640dc43b:::benchmark_data_smithi194_121512_object100622:head# 2024-04-29T14:26:36.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640df3f3:::benchmark_data_smithi194_121512_object108065:head# 2024-04-29T14:26:36.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640df452:::benchmark_data_smithi194_121512_object130265:head# 2024-04-29T14:26:36.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640e858e:::benchmark_data_smithi194_121512_object24420:head# 2024-04-29T14:26:36.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640f3e43:::benchmark_data_smithi194_121512_object866:head# 2024-04-29T14:26:36.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640fbc9b:::benchmark_data_smithi194_121512_object120452:head# 2024-04-29T14:26:36.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:640fe840:::benchmark_data_smithi194_121512_object67353:head# 2024-04-29T14:26:36.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6410d6de:::benchmark_data_smithi194_121512_object119103:head# 2024-04-29T14:26:36.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641182c0:::benchmark_data_smithi194_121512_object56121:head# 2024-04-29T14:26:36.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641221b4:::benchmark_data_smithi194_121512_object75505:head# 2024-04-29T14:26:36.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64123dfe:::benchmark_data_smithi194_121512_object139:head# 2024-04-29T14:26:36.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6412fd8c:::benchmark_data_smithi194_121512_object10009:head# 2024-04-29T14:26:36.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6413866a:::benchmark_data_smithi194_121512_object81485:head# 2024-04-29T14:26:36.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6415492b:::benchmark_data_smithi194_121512_object73705:head# 2024-04-29T14:26:36.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6415e8f6:::benchmark_data_smithi194_121512_object42726:head# 2024-04-29T14:26:36.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6416074d:::benchmark_data_smithi194_121512_object26923:head# 2024-04-29T14:26:36.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641609c4:::benchmark_data_smithi194_121512_object132596:head# 2024-04-29T14:26:36.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6417914e:::benchmark_data_smithi194_121512_object20795:head# 2024-04-29T14:26:36.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64179786:::benchmark_data_smithi194_121512_object79368:head# 2024-04-29T14:26:36.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6418824f:::benchmark_data_smithi194_121512_object95117:head# 2024-04-29T14:26:36.484 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:36.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6418e5ce:::benchmark_data_smithi194_121512_object38513:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64193f10:::benchmark_data_smithi194_121512_object68985:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6419ffe8:::benchmark_data_smithi194_121512_object67755:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641a9b5c:::benchmark_data_smithi194_121512_object82310:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641ab7df:::benchmark_data_smithi194_121512_object32239:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641ac837:::benchmark_data_smithi194_121512_object40226:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641ae881:::benchmark_data_smithi194_121512_object57449:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641b4d4d:::benchmark_data_smithi194_121512_object124393:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641b55e7:::benchmark_data_smithi194_121512_object117209:head# 2024-04-29T14:26:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641b9257:::benchmark_data_smithi194_121512_object267:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641bb064:::benchmark_data_smithi194_121512_object64582:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641c93aa:::benchmark_data_smithi194_121512_object37156:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641d2b21:::benchmark_data_smithi194_121512_object18687:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641da77a:::benchmark_data_smithi194_121512_object79088:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641db815:::benchmark_data_smithi194_121512_object117165:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641e4175:::benchmark_data_smithi194_121512_object27513:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641ee982:::benchmark_data_smithi194_121512_object19461:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641eea47:::benchmark_data_smithi194_121512_object26487:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641f01be:::benchmark_data_smithi194_121512_object56172:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:641fe5e1:::benchmark_data_smithi194_121512_object69651:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642038d4:::benchmark_data_smithi194_121512_object74839:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6420e173:::benchmark_data_smithi194_121512_object10275:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64212e98:::benchmark_data_smithi194_121512_object46543:head# 2024-04-29T14:26:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64214391:::benchmark_data_smithi194_121512_object33710:head# 2024-04-29T14:26:36.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64223601:::benchmark_data_smithi194_121512_object119246:head# 2024-04-29T14:26:36.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6422ee4b:::benchmark_data_smithi194_121512_object85867:head# 2024-04-29T14:26:36.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64244341:::benchmark_data_smithi194_121512_object29727:head# 2024-04-29T14:26:36.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6424b5b2:::benchmark_data_smithi194_121512_object50914:head# 2024-04-29T14:26:36.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6424d0ed:::benchmark_data_smithi194_121512_object76063:head# 2024-04-29T14:26:36.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6425195e:::benchmark_data_smithi194_121512_object11039:head# 2024-04-29T14:26:36.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642583df:::benchmark_data_smithi194_121512_object52645:head# 2024-04-29T14:26:36.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6425d04e:::benchmark_data_smithi194_121512_object126746:head# 2024-04-29T14:26:36.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642610d9:::benchmark_data_smithi194_121512_object27281:head# 2024-04-29T14:26:36.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6426c8fc:::benchmark_data_smithi194_121512_object85667:head# 2024-04-29T14:26:36.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6426ed1b:::benchmark_data_smithi194_121512_object86783:head# 2024-04-29T14:26:36.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6427639e:::benchmark_data_smithi194_121512_object21231:head# 2024-04-29T14:26:36.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6428b2a1:::benchmark_data_smithi194_121512_object19131:head# 2024-04-29T14:26:36.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6429aea6:::benchmark_data_smithi194_121512_object87667:head# 2024-04-29T14:26:36.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6429e3c9:::benchmark_data_smithi194_121512_object45011:head# 2024-04-29T14:26:36.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642a088a:::benchmark_data_smithi194_121512_object65416:head# 2024-04-29T14:26:36.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642a92d6:::benchmark_data_smithi194_121512_object127726:head# 2024-04-29T14:26:36.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642ae337:::benchmark_data_smithi194_121512_object31422:head# 2024-04-29T14:26:36.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642b2d43:::benchmark_data_smithi194_121512_object5443:head# 2024-04-29T14:26:36.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642bee38:::benchmark_data_smithi194_121512_object49843:head# 2024-04-29T14:26:36.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642c1af0:::benchmark_data_smithi194_121512_object21173:head# 2024-04-29T14:26:36.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642c2f8e:::benchmark_data_smithi194_121512_object91937:head# 2024-04-29T14:26:36.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642c8adc:::benchmark_data_smithi194_121512_object46652:head# 2024-04-29T14:26:36.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642ced8e:::benchmark_data_smithi194_121512_object13841:head# 2024-04-29T14:26:36.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642e3653:::benchmark_data_smithi194_121512_object22033:head# 2024-04-29T14:26:36.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642f215d:::benchmark_data_smithi194_121512_object44915:head# 2024-04-29T14:26:36.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:642fed45:::benchmark_data_smithi194_121512_object15339:head# 2024-04-29T14:26:36.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64307f04:::benchmark_data_smithi194_121512_object69081:head# 2024-04-29T14:26:36.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64314dda:::benchmark_data_smithi194_121512_object49007:head# 2024-04-29T14:26:36.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64318bf4:::benchmark_data_smithi194_121512_object57678:head# 2024-04-29T14:26:36.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6431b266:::benchmark_data_smithi194_121512_object6113:head# 2024-04-29T14:26:36.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6431d7ca:::benchmark_data_smithi194_121512_object64814:head# 2024-04-29T14:26:36.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643264ed:::benchmark_data_smithi194_121512_object106658:head# 2024-04-29T14:26:36.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6432dc69:::benchmark_data_smithi194_121512_object33212:head# 2024-04-29T14:26:36.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6433e137:::benchmark_data_smithi194_121512_object40021:head# 2024-04-29T14:26:36.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64345eb5:::benchmark_data_smithi194_121512_object89329:head# 2024-04-29T14:26:36.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64349014:::benchmark_data_smithi194_121512_object26408:head# 2024-04-29T14:26:36.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6434d4ea:::benchmark_data_smithi194_121512_object49136:head# 2024-04-29T14:26:36.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6435a644:::benchmark_data_smithi194_121512_object68533:head# 2024-04-29T14:26:36.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64368bf5:::benchmark_data_smithi194_121512_object88211:head# 2024-04-29T14:26:36.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6436e584:::benchmark_data_smithi194_121512_object117463:head# 2024-04-29T14:26:36.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64372690:::benchmark_data_smithi194_121512_object36115:head# 2024-04-29T14:26:36.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6437312b:::benchmark_data_smithi194_121512_object63421:head# 2024-04-29T14:26:36.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6438caa6:::benchmark_data_smithi194_121512_object46782:head# 2024-04-29T14:26:36.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6439aaae:::benchmark_data_smithi194_121512_object31567:head# 2024-04-29T14:26:36.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6439e865:::benchmark_data_smithi194_121512_object129731:head# 2024-04-29T14:26:36.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643bdfc3:::benchmark_data_smithi194_121512_object51560:head# 2024-04-29T14:26:36.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643c41e5:::benchmark_data_smithi194_121512_object16950:head# 2024-04-29T14:26:36.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643c4bff:::benchmark_data_smithi194_121512_object113047:head# 2024-04-29T14:26:36.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643c50ed:::benchmark_data_smithi194_121512_object92576:head# 2024-04-29T14:26:36.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643d637b:::benchmark_data_smithi194_121512_object126471:head# 2024-04-29T14:26:36.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643d67c0:::benchmark_data_smithi194_121512_object81671:head# 2024-04-29T14:26:36.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643db7a9:::benchmark_data_smithi194_121512_object112958:head# 2024-04-29T14:26:36.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643dec51:::benchmark_data_smithi194_121512_object16969:head# 2024-04-29T14:26:36.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643decd5:::benchmark_data_smithi194_121512_object96749:head# 2024-04-29T14:26:36.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643e10aa:::benchmark_data_smithi194_121512_object18084:head# 2024-04-29T14:26:36.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643e6a73:::benchmark_data_smithi194_121512_object129278:head# 2024-04-29T14:26:36.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643e91b7:::benchmark_data_smithi194_121512_object115410:head# 2024-04-29T14:26:36.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:643f0a79:::benchmark_data_smithi194_121512_object17834:head# 2024-04-29T14:26:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64403b25:::benchmark_data_smithi194_121512_object116759:head# 2024-04-29T14:26:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6440a1b1:::benchmark_data_smithi194_121512_object89171:head# 2024-04-29T14:26:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6441d828:::benchmark_data_smithi194_121512_object128788:head# 2024-04-29T14:26:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644228e0:::benchmark_data_smithi194_121512_object113610:head# 2024-04-29T14:26:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64425044:::benchmark_data_smithi194_121512_object92786:head# 2024-04-29T14:26:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6442db8f:::benchmark_data_smithi194_121512_object58311:head# 2024-04-29T14:26:36.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6442f95a:::benchmark_data_smithi194_121512_object126796:head# 2024-04-29T14:26:36.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64435c9f:::benchmark_data_smithi194_121512_object22901:head# 2024-04-29T14:26:36.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6444b3d6:::benchmark_data_smithi194_121512_object76363:head# 2024-04-29T14:26:36.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64459300:::benchmark_data_smithi194_121512_object72743:head# 2024-04-29T14:26:36.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64459632:::benchmark_data_smithi194_121512_object102389:head# 2024-04-29T14:26:36.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6445a3b8:::benchmark_data_smithi194_121512_object102734:head# 2024-04-29T14:26:36.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6445d65d:::benchmark_data_smithi194_121512_object74334:head# 2024-04-29T14:26:36.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64460515:::benchmark_data_smithi194_121512_object113114:head# 2024-04-29T14:26:36.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644689e6:::benchmark_data_smithi194_121512_object10760:head# 2024-04-29T14:26:36.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6446f8f4:::benchmark_data_smithi194_121512_object62659:head# 2024-04-29T14:26:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644720bb:::benchmark_data_smithi194_121512_object87681:head# 2024-04-29T14:26:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6447ac92:::benchmark_data_smithi194_121512_object40580:head# 2024-04-29T14:26:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644807c1:::benchmark_data_smithi194_121512_object10458:head# 2024-04-29T14:26:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64480acd:::benchmark_data_smithi194_121512_object46128:head# 2024-04-29T14:26:36.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64481810:::benchmark_data_smithi194_121512_object13382:head# 2024-04-29T14:26:36.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64495f59:::benchmark_data_smithi194_121512_object123429:head# 2024-04-29T14:26:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64497695:::benchmark_data_smithi194_121512_object11992:head# 2024-04-29T14:26:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644a919a:::benchmark_data_smithi194_121512_object119976:head# 2024-04-29T14:26:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644b5dd2:::benchmark_data_smithi194_121512_object57589:head# 2024-04-29T14:26:36.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644bb079:::benchmark_data_smithi194_121512_object90801:head# 2024-04-29T14:26:36.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644bc5ac:::benchmark_data_smithi194_121512_object31872:head# 2024-04-29T14:26:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644caefa:::benchmark_data_smithi194_121512_object34863:head# 2024-04-29T14:26:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644ce91a:::benchmark_data_smithi194_121512_object57208:head# 2024-04-29T14:26:36.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644d114d:::benchmark_data_smithi194_121512_object98987:head# 2024-04-29T14:26:36.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644de59b:::benchmark_data_smithi194_121512_object32803:head# 2024-04-29T14:26:36.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644f4c7c:::benchmark_data_smithi194_121512_object107963:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:644f5ae9:::benchmark_data_smithi194_121512_object127340:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6450b04d:::benchmark_data_smithi194_121512_object14659:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6450d14d:::benchmark_data_smithi194_121512_object46786:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6450f185:::benchmark_data_smithi194_121512_object114803:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64517461:::benchmark_data_smithi194_121512_object53163:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6451cf35:::benchmark_data_smithi194_121512_object94437:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6452b08b:::benchmark_data_smithi194_121512_object44087:head# 2024-04-29T14:26:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6452efb7:::benchmark_data_smithi194_121512_object113015:head# 2024-04-29T14:26:36.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6452f59d:::benchmark_data_smithi194_121512_object65297:head# 2024-04-29T14:26:36.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64536074:::benchmark_data_smithi194_121512_object68881:head# 2024-04-29T14:26:36.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645360d8:::benchmark_data_smithi194_121512_object86717:head# 2024-04-29T14:26:36.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6453e2fe:::benchmark_data_smithi194_121512_object67534:head# 2024-04-29T14:26:36.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64540cc0:::benchmark_data_smithi194_121512_object123134:head# 2024-04-29T14:26:36.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64545c46:::benchmark_data_smithi194_121512_object40303:head# 2024-04-29T14:26:36.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6454f5a0:::benchmark_data_smithi194_121512_object19623:head# 2024-04-29T14:26:36.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645525ce:::benchmark_data_smithi194_121512_object16321:head# 2024-04-29T14:26:36.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645585b8:::benchmark_data_smithi194_121512_object76984:head# 2024-04-29T14:26:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6455d3a7:::benchmark_data_smithi194_121512_object64207:head# 2024-04-29T14:26:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645622de:::benchmark_data_smithi194_121512_object45117:head# 2024-04-29T14:26:36.552 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6456679e:::benchmark_data_smithi194_121512_object66772:head# 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64567654:::benchmark_data_smithi194_121512_object27555:head# 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6456e536:::benchmark_data_smithi194_121512_object26381:head# 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6457640a:::benchmark_data_smithi194_121512_object120169:head# 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6457ba70:::benchmark_data_smithi194_121512_object23618:head# 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64585eed:::benchmark_data_smithi194_121512_object25563:head# 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6458d5c1:::benchmark_data_smithi194_121512_object95238:head# 2024-04-29T14:26:36.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645900fb:::benchmark_data_smithi194_121512_object93961:head# 2024-04-29T14:26:36.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6459554c:::benchmark_data_smithi194_121512_object26970:head# 2024-04-29T14:26:36.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645ad179:::benchmark_data_smithi194_121512_object55907:head# 2024-04-29T14:26:36.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645aedd7:::benchmark_data_smithi194_121512_object43098:head# 2024-04-29T14:26:36.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645d2210:::benchmark_data_smithi194_121512_object32517:head# 2024-04-29T14:26:36.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645d6d0c:::benchmark_data_smithi194_121512_object73678:head# 2024-04-29T14:26:36.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645d80fc:::benchmark_data_smithi194_121512_object17891:head# 2024-04-29T14:26:36.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645da42d:::benchmark_data_smithi194_121512_object43143:head# 2024-04-29T14:26:36.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645e9413:::benchmark_data_smithi194_121512_object117392:head# 2024-04-29T14:26:36.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645eb110:::benchmark_data_smithi194_121512_object105499:head# 2024-04-29T14:26:36.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645f2c5e:::benchmark_data_smithi194_121512_object28882:head# 2024-04-29T14:26:36.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645f7a59:::benchmark_data_smithi194_121512_object44490:head# 2024-04-29T14:26:36.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:645feae5:::benchmark_data_smithi194_121512_object946:head# 2024-04-29T14:26:36.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6460674e:::benchmark_data_smithi194_121512_object116290:head# 2024-04-29T14:26:36.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6460ac74:::benchmark_data_smithi194_121512_object45635:head# 2024-04-29T14:26:36.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6460b098:::benchmark_data_smithi194_121512_object116861:head# 2024-04-29T14:26:36.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646125b3:::benchmark_data_smithi194_121512_object27856:head# 2024-04-29T14:26:36.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6462596e:::benchmark_data_smithi194_121512_object76552:head# 2024-04-29T14:26:36.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6462bad6:::benchmark_data_smithi194_121512_object132345:head# 2024-04-29T14:26:36.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6462c42d:::benchmark_data_smithi194_121512_object71440:head# 2024-04-29T14:26:36.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64631085:::benchmark_data_smithi194_121512_object132544:head# 2024-04-29T14:26:36.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64631c5e:::benchmark_data_smithi194_121512_object104060:head# 2024-04-29T14:26:36.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6463922e:::benchmark_data_smithi194_121512_object34787:head# 2024-04-29T14:26:36.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6464972c:::benchmark_data_smithi194_121512_object62091:head# 2024-04-29T14:26:36.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6464b5a3:::benchmark_data_smithi194_121512_object77772:head# 2024-04-29T14:26:36.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646582a8:::benchmark_data_smithi194_121512_object95929:head# 2024-04-29T14:26:36.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64660868:::benchmark_data_smithi194_121512_object30494:head# 2024-04-29T14:26:36.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6466e1bd:::benchmark_data_smithi194_121512_object63580:head# 2024-04-29T14:26:36.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6467aad0:::benchmark_data_smithi194_121512_object58342:head# 2024-04-29T14:26:36.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6467b91c:::benchmark_data_smithi194_121512_object38484:head# 2024-04-29T14:26:36.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6467d992:::benchmark_data_smithi194_121512_object53198:head# 2024-04-29T14:26:36.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6468566d:::benchmark_data_smithi194_121512_object82608:head# 2024-04-29T14:26:36.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6468bb1a:::benchmark_data_smithi194_121512_object83563:head# 2024-04-29T14:26:36.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6469759c:::benchmark_data_smithi194_121512_object74318:head# 2024-04-29T14:26:36.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64699a78:::benchmark_data_smithi194_121512_object72171:head# 2024-04-29T14:26:36.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6469e99e:::benchmark_data_smithi194_121512_object118251:head# 2024-04-29T14:26:36.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646a0e15:::benchmark_data_smithi194_121512_object111505:head# 2024-04-29T14:26:36.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646a66bd:::benchmark_data_smithi194_121512_object6013:head# 2024-04-29T14:26:36.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646a8a1b:::benchmark_data_smithi194_121512_object93234:head# 2024-04-29T14:26:36.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646a90b9:::benchmark_data_smithi194_121512_object95720:head# 2024-04-29T14:26:36.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646b57cb:::benchmark_data_smithi194_121512_object10839:head# 2024-04-29T14:26:36.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646b6a07:::benchmark_data_smithi194_121512_object77479:head# 2024-04-29T14:26:36.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646bd90c:::benchmark_data_smithi194_121512_object53704:head# 2024-04-29T14:26:36.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646d1dd0:::benchmark_data_smithi194_121512_object16821:head# 2024-04-29T14:26:36.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646e1105:::benchmark_data_smithi194_121512_object20613:head# 2024-04-29T14:26:36.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646e8fe4:::benchmark_data_smithi194_121512_object19609:head# 2024-04-29T14:26:36.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646ec62d:::benchmark_data_smithi194_121512_object51314:head# 2024-04-29T14:26:36.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:646ffbe9:::benchmark_data_smithi194_121512_object55538:head# 2024-04-29T14:26:36.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6471fb8c:::benchmark_data_smithi194_121512_object2183:head# 2024-04-29T14:26:36.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6472a443:::benchmark_data_smithi194_121512_object116384:head# 2024-04-29T14:26:36.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6473395e:::benchmark_data_smithi194_121512_object27541:head# 2024-04-29T14:26:36.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64733df4:::benchmark_data_smithi194_121512_object19766:head# 2024-04-29T14:26:36.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64749781:::benchmark_data_smithi194_121512_object126512:head# 2024-04-29T14:26:36.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6474d681:::benchmark_data_smithi194_121512_object121486:head# 2024-04-29T14:26:36.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64758844:::benchmark_data_smithi194_121512_object4380:head# 2024-04-29T14:26:36.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6475f607:::benchmark_data_smithi194_121512_object4438:head# 2024-04-29T14:26:36.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64761183:::benchmark_data_smithi194_121512_object52643:head# 2024-04-29T14:26:36.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64762474:::benchmark_data_smithi194_121512_object83921:head# 2024-04-29T14:26:36.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6476684e:::benchmark_data_smithi194_121512_object13215:head# 2024-04-29T14:26:36.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6476d4e9:::benchmark_data_smithi194_121512_object66753:head# 2024-04-29T14:26:36.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64782421:::benchmark_data_smithi194_121512_object79100:head# 2024-04-29T14:26:36.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6478428a:::benchmark_data_smithi194_121512_object14439:head# 2024-04-29T14:26:36.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64787d11:::benchmark_data_smithi194_121512_object130322:head# 2024-04-29T14:26:36.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6478b395:::benchmark_data_smithi194_121512_object81395:head# 2024-04-29T14:26:36.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64790a6c:::benchmark_data_smithi194_121512_object61452:head# 2024-04-29T14:26:36.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:36.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64795628:::benchmark_data_smithi194_121512_object45585:head# 2024-04-29T14:26:36.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6479fe46:::benchmark_data_smithi194_121512_object40541:head# 2024-04-29T14:26:36.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647a49b0:::benchmark_data_smithi194_121512_object7548:head# 2024-04-29T14:26:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647b06d0:::benchmark_data_smithi194_121512_object58142:head# 2024-04-29T14:26:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647b23e4:::benchmark_data_smithi194_121512_object836:head# 2024-04-29T14:26:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647be61b:::benchmark_data_smithi194_121512_object72649:head# 2024-04-29T14:26:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647be748:::benchmark_data_smithi194_121512_object94710:head# 2024-04-29T14:26:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647d283c:::benchmark_data_smithi194_121512_object93015:head# 2024-04-29T14:26:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647dff06:::benchmark_data_smithi194_121512_object11941:head# 2024-04-29T14:26:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647e7151:::benchmark_data_smithi194_121512_object26925:head# 2024-04-29T14:26:36.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647e9043:::benchmark_data_smithi194_121512_object3369:head# 2024-04-29T14:26:36.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647e9aa3:::benchmark_data_smithi194_121512_object42718:head# 2024-04-29T14:26:36.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647f4e35:::benchmark_data_smithi194_121512_object40247:head# 2024-04-29T14:26:36.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647f63f0:::benchmark_data_smithi194_121512_object38285:head# 2024-04-29T14:26:36.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:647ff2b7:::benchmark_data_smithi194_121512_object41513:head# 2024-04-29T14:26:36.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64807e41:::benchmark_data_smithi194_121512_object123540:head# 2024-04-29T14:26:36.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6480d3e5:::benchmark_data_smithi194_121512_object74772:head# 2024-04-29T14:26:36.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6480fd4c:::benchmark_data_smithi194_121512_object110269:head# 2024-04-29T14:26:36.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6482583d:::benchmark_data_smithi194_121512_object80769:head# 2024-04-29T14:26:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6482f65d:::benchmark_data_smithi194_121512_object123088:head# 2024-04-29T14:26:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6483230c:::benchmark_data_smithi194_121512_object47808:head# 2024-04-29T14:26:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64832b28:::benchmark_data_smithi194_121512_object74248:head# 2024-04-29T14:26:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6483b9c0:::benchmark_data_smithi194_121512_object35685:head# 2024-04-29T14:26:36.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6483bb14:::benchmark_data_smithi194_121512_object12041:head# 2024-04-29T14:26:36.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64848272:::benchmark_data_smithi194_121512_object115086:head# 2024-04-29T14:26:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64869218:::benchmark_data_smithi194_121512_object122210:head# 2024-04-29T14:26:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6486f806:::benchmark_data_smithi194_121512_object66425:head# 2024-04-29T14:26:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64871c14:::benchmark_data_smithi194_121512_object21122:head# 2024-04-29T14:26:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64877491:::benchmark_data_smithi194_121512_object35423:head# 2024-04-29T14:26:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648778fe:::benchmark_data_smithi194_121512_object121178:head# 2024-04-29T14:26:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64881d1c:::benchmark_data_smithi194_121512_object41478:head# 2024-04-29T14:26:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6488398c:::benchmark_data_smithi194_121512_object93003:head# 2024-04-29T14:26:36.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64894bb9:::benchmark_data_smithi194_121512_object124334:head# 2024-04-29T14:26:36.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648a176c:::benchmark_data_smithi194_121512_object113189:head# 2024-04-29T14:26:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648a530a:::benchmark_data_smithi194_121512_object11322:head# 2024-04-29T14:26:36.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648a72a5:::benchmark_data_smithi194_121512_object107397:head# 2024-04-29T14:26:36.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648b3802:::benchmark_data_smithi194_121512_object2312:head# 2024-04-29T14:26:36.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648bd1b4:::benchmark_data_smithi194_121512_object117774:head# 2024-04-29T14:26:36.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648c6a3e:::benchmark_data_smithi194_121512_object53620:head# 2024-04-29T14:26:36.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648c9e79:::benchmark_data_smithi194_121512_object123420:head# 2024-04-29T14:26:36.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648cad46:::benchmark_data_smithi194_121512_object6230:head# 2024-04-29T14:26:36.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648e1812:::benchmark_data_smithi194_121512_object78525:head# 2024-04-29T14:26:36.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648e758a:::benchmark_data_smithi194_121512_object20757:head# 2024-04-29T14:26:36.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648ed44e:::benchmark_data_smithi194_121512_object53320:head# 2024-04-29T14:26:36.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648f357f:::benchmark_data_smithi194_121512_object108310:head# 2024-04-29T14:26:36.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:648f6703:::benchmark_data_smithi194_121512_object56223:head# 2024-04-29T14:26:36.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649022e8:::benchmark_data_smithi194_121512_object12602:head# 2024-04-29T14:26:36.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6491d6e2:::benchmark_data_smithi194_121512_object20011:head# 2024-04-29T14:26:36.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6492a6e1:::benchmark_data_smithi194_121512_object76933:head# 2024-04-29T14:26:36.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64943bce:::benchmark_data_smithi194_121512_object120853:head# 2024-04-29T14:26:36.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6494e0cf:::benchmark_data_smithi194_121512_object18185:head# 2024-04-29T14:26:36.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64952977:::benchmark_data_smithi194_121512_object91701:head# 2024-04-29T14:26:36.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64956b18:::benchmark_data_smithi194_121512_object22739:head# 2024-04-29T14:26:36.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64956e9e:::benchmark_data_smithi194_121512_object22046:head# 2024-04-29T14:26:36.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649589f0:::benchmark_data_smithi194_121512_object32362:head# 2024-04-29T14:26:36.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6495bff7:::benchmark_data_smithi194_121512_object47295:head# 2024-04-29T14:26:36.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6495ca2e:::benchmark_data_smithi194_121512_object10565:head# 2024-04-29T14:26:36.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6496b46a:::benchmark_data_smithi194_121512_object13171:head# 2024-04-29T14:26:36.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64974975:::benchmark_data_smithi194_121512_object55570:head# 2024-04-29T14:26:36.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6497914d:::benchmark_data_smithi194_121512_object85721:head# 2024-04-29T14:26:36.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649793b2:::benchmark_data_smithi194_121512_object108726:head# 2024-04-29T14:26:36.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6497b293:::benchmark_data_smithi194_121512_object54655:head# 2024-04-29T14:26:36.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6497cf2f:::benchmark_data_smithi194_121512_object115528:head# 2024-04-29T14:26:36.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6497f3e3:::benchmark_data_smithi194_121512_object52719:head# 2024-04-29T14:26:36.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64981070:::benchmark_data_smithi194_121512_object15651:head# 2024-04-29T14:26:36.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6498d267:::benchmark_data_smithi194_121512_object86250:head# 2024-04-29T14:26:36.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6498fdcd:::benchmark_data_smithi194_121512_object129124:head# 2024-04-29T14:26:36.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64991caa:::benchmark_data_smithi194_121512_object122532:head# 2024-04-29T14:26:36.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64998963:::benchmark_data_smithi194_121512_object21051:head# 2024-04-29T14:26:36.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6499e5c4:::benchmark_data_smithi194_121512_object89073:head# 2024-04-29T14:26:36.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6499ed15:::benchmark_data_smithi194_121512_object67060:head# 2024-04-29T14:26:36.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649b3bf4:::benchmark_data_smithi194_121512_object10013:head# 2024-04-29T14:26:36.620 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:36.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649b984f:::benchmark_data_smithi194_121512_object123678:head# 2024-04-29T14:26:36.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649cdd97:::benchmark_data_smithi194_121512_object131372:head# 2024-04-29T14:26:36.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649def01:::benchmark_data_smithi194_121512_object132892:head# 2024-04-29T14:26:36.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649f0891:::benchmark_data_smithi194_121512_object119318:head# 2024-04-29T14:26:36.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649fb035:::benchmark_data_smithi194_121512_object83833:head# 2024-04-29T14:26:36.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649fc864:::benchmark_data_smithi194_121512_object48100:head# 2024-04-29T14:26:36.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:649fcd84:::benchmark_data_smithi194_121512_object52310:head# 2024-04-29T14:26:36.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a03219:::benchmark_data_smithi194_121512_object7884:head# 2024-04-29T14:26:36.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a0327c:::benchmark_data_smithi194_121512_object53396:head# 2024-04-29T14:26:36.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a0e5ba:::benchmark_data_smithi194_121512_object83032:head# 2024-04-29T14:26:36.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a16b91:::benchmark_data_smithi194_121512_object70811:head# 2024-04-29T14:26:36.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a1c941:::benchmark_data_smithi194_121512_object78177:head# 2024-04-29T14:26:36.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a28854:::benchmark_data_smithi194_121512_object31351:head# 2024-04-29T14:26:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3241230645' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:36.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a2a2f1:::benchmark_data_smithi194_121512_object28909:head# 2024-04-29T14:26:36.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a2eca1:::benchmark_data_smithi194_121512_object49268:head# 2024-04-29T14:26:36.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a3a84a:::benchmark_data_smithi194_121512_object62422:head# 2024-04-29T14:26:36.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a3c99d:::benchmark_data_smithi194_121512_object67811:head# 2024-04-29T14:26:36.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a4b261:::benchmark_data_smithi194_121512_object65863:head# 2024-04-29T14:26:36.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a4fe2f:::benchmark_data_smithi194_121512_object23898:head# 2024-04-29T14:26:36.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a5069d:::benchmark_data_smithi194_121512_object21326:head# 2024-04-29T14:26:36.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a51f0e:::benchmark_data_smithi194_121512_object115586:head# 2024-04-29T14:26:36.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a5cbb9:::benchmark_data_smithi194_121512_object58773:head# 2024-04-29T14:26:36.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a5e684:::benchmark_data_smithi194_121512_object36149:head# 2024-04-29T14:26:36.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a5f634:::benchmark_data_smithi194_121512_object129406:head# 2024-04-29T14:26:36.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a61cb7:::benchmark_data_smithi194_121512_object94450:head# 2024-04-29T14:26:36.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a658c1:::benchmark_data_smithi194_121512_object75940:head# 2024-04-29T14:26:36.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a662f8:::benchmark_data_smithi194_121512_object117206:head# 2024-04-29T14:26:36.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a73f8e:::benchmark_data_smithi194_121512_object63828:head# 2024-04-29T14:26:36.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a76f6b:::benchmark_data_smithi194_121512_object31253:head# 2024-04-29T14:26:36.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a7f4c8:::benchmark_data_smithi194_121512_object100922:head# 2024-04-29T14:26:36.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a83c92:::benchmark_data_smithi194_121512_object62738:head# 2024-04-29T14:26:36.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a84c29:::benchmark_data_smithi194_121512_object39298:head# 2024-04-29T14:26:36.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a8bb63:::benchmark_data_smithi194_121512_object10046:head# 2024-04-29T14:26:36.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a8c117:::benchmark_data_smithi194_121512_object15404:head# 2024-04-29T14:26:36.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64a945ef:::benchmark_data_smithi194_121512_object43942:head# 2024-04-29T14:26:36.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64aa1297:::benchmark_data_smithi194_121512_object61895:head# 2024-04-29T14:26:36.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64aa3dce:::benchmark_data_smithi194_121512_object146:head# 2024-04-29T14:26:36.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64aa66a1:::benchmark_data_smithi194_121512_object131587:head# 2024-04-29T14:26:36.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64aa7668:::benchmark_data_smithi194_121512_object37201:head# 2024-04-29T14:26:36.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64aadc27:::benchmark_data_smithi194_121512_object118347:head# 2024-04-29T14:26:36.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ab20d0:::benchmark_data_smithi194_121512_object104114:head# 2024-04-29T14:26:36.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64abed7d:::benchmark_data_smithi194_121512_object35867:head# 2024-04-29T14:26:36.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ad50e1:::benchmark_data_smithi194_121512_object2393:head# 2024-04-29T14:26:36.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64aea7f2:::benchmark_data_smithi194_121512_object9596:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b058c8:::benchmark_data_smithi194_121512_object53102:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b2089c:::benchmark_data_smithi194_121512_object1481:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b2b60e:::benchmark_data_smithi194_121512_object59353:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b2cc8f:::benchmark_data_smithi194_121512_object119131:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b35f72:::benchmark_data_smithi194_121512_object107041:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b3e2af:::benchmark_data_smithi194_121512_object44467:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b4769d:::benchmark_data_smithi194_121512_object24376:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b6799e:::benchmark_data_smithi194_121512_object59359:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b74373:::benchmark_data_smithi194_121512_object32511:head# 2024-04-29T14:26:36.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b7e39b:::benchmark_data_smithi194_121512_object131030:head# 2024-04-29T14:26:36.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b83bbb:::benchmark_data_smithi194_121512_object63968:head# 2024-04-29T14:26:36.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b88f8f:::benchmark_data_smithi194_121512_object54943:head# 2024-04-29T14:26:36.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b8bf80:::benchmark_data_smithi194_121512_object39476:head# 2024-04-29T14:26:36.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b8e821:::benchmark_data_smithi194_121512_object116925:head# 2024-04-29T14:26:36.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b9356c:::benchmark_data_smithi194_121512_object11682:head# 2024-04-29T14:26:36.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b9bfba:::benchmark_data_smithi194_121512_object99138:head# 2024-04-29T14:26:36.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b9e122:::benchmark_data_smithi194_121512_object94245:head# 2024-04-29T14:26:36.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64b9f49e:::benchmark_data_smithi194_121512_object32330:head# 2024-04-29T14:26:36.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ba3275:::benchmark_data_smithi194_121512_object44535:head# 2024-04-29T14:26:36.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ba57f2:::benchmark_data_smithi194_121512_object596:head# 2024-04-29T14:26:36.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bae39e:::benchmark_data_smithi194_121512_object31194:head# 2024-04-29T14:26:36.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bae909:::benchmark_data_smithi194_121512_object11130:head# 2024-04-29T14:26:36.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64baf579:::benchmark_data_smithi194_121512_object128473:head# 2024-04-29T14:26:36.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bb776b:::benchmark_data_smithi194_121512_object5035:head# 2024-04-29T14:26:36.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bc2376:::benchmark_data_smithi194_121512_object113758:head# 2024-04-29T14:26:36.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bd36d5:::benchmark_data_smithi194_121512_object119317:head# 2024-04-29T14:26:36.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bd7d5d:::benchmark_data_smithi194_121512_object101537:head# 2024-04-29T14:26:36.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64be6281:::benchmark_data_smithi194_121512_object70818:head# 2024-04-29T14:26:36.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bf0d65:::benchmark_data_smithi194_121512_object98177:head# 2024-04-29T14:26:36.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bf3c15:::benchmark_data_smithi194_121512_object121276:head# 2024-04-29T14:26:36.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bfc15a:::benchmark_data_smithi194_121512_object14978:head# 2024-04-29T14:26:36.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64bfd939:::benchmark_data_smithi194_121512_object19052:head# 2024-04-29T14:26:36.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c015e6:::benchmark_data_smithi194_121512_object100423:head# 2024-04-29T14:26:36.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c02d72:::benchmark_data_smithi194_121512_object10895:head# 2024-04-29T14:26:36.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c0a349:::benchmark_data_smithi194_121512_object78587:head# 2024-04-29T14:26:36.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c241da:::benchmark_data_smithi194_121512_object24222:head# 2024-04-29T14:26:36.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c2b000:::benchmark_data_smithi194_121512_object2009:head# 2024-04-29T14:26:36.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c4072d:::benchmark_data_smithi194_121512_object56434:head# 2024-04-29T14:26:36.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c498ba:::benchmark_data_smithi194_121512_object65304:head# 2024-04-29T14:26:36.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c4c2d9:::benchmark_data_smithi194_121512_object25509:head# 2024-04-29T14:26:36.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c4cf9d:::benchmark_data_smithi194_121512_object66316:head# 2024-04-29T14:26:36.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c52022:::benchmark_data_smithi194_121512_object39801:head# 2024-04-29T14:26:36.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c5bfaf:::benchmark_data_smithi194_121512_object79045:head# 2024-04-29T14:26:36.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c5f00a:::benchmark_data_smithi194_121512_object110156:head# 2024-04-29T14:26:36.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c6101d:::benchmark_data_smithi194_121512_object81407:head# 2024-04-29T14:26:36.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c6547c:::benchmark_data_smithi194_121512_object7746:head# 2024-04-29T14:26:36.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c6bdaa:::benchmark_data_smithi194_121512_object16009:head# 2024-04-29T14:26:36.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c6c7e1:::benchmark_data_smithi194_121512_object111680:head# 2024-04-29T14:26:36.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c89f38:::benchmark_data_smithi194_121512_object28199:head# 2024-04-29T14:26:36.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c8cfc5:::benchmark_data_smithi194_121512_object61605:head# 2024-04-29T14:26:36.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64c9cb32:::benchmark_data_smithi194_121512_object88419:head# 2024-04-29T14:26:36.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ca331e:::benchmark_data_smithi194_121512_object105866:head# 2024-04-29T14:26:36.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64cb5bc3:::benchmark_data_smithi194_121512_object15828:head# 2024-04-29T14:26:36.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64cbc2c9:::benchmark_data_smithi194_121512_object22825:head# 2024-04-29T14:26:36.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64cc634b:::benchmark_data_smithi194_121512_object104403:head# 2024-04-29T14:26:36.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ce02c1:::benchmark_data_smithi194_121512_object49705:head# 2024-04-29T14:26:36.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64cf4a5e:::benchmark_data_smithi194_121512_object123972:head# 2024-04-29T14:26:36.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64cf5f13:::benchmark_data_smithi194_121512_object73102:head# 2024-04-29T14:26:36.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64cf814c:::benchmark_data_smithi194_121512_object49990:head# 2024-04-29T14:26:36.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d0a1ac:::benchmark_data_smithi194_121512_object33398:head# 2024-04-29T14:26:36.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d11621:::benchmark_data_smithi194_121512_object60889:head# 2024-04-29T14:26:36.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d1249c:::benchmark_data_smithi194_121512_object48529:head# 2024-04-29T14:26:36.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d188b3:::benchmark_data_smithi194_121512_object133125:head# 2024-04-29T14:26:36.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d1f119:::benchmark_data_smithi194_121512_object50388:head# 2024-04-29T14:26:36.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d20e5b:::benchmark_data_smithi194_121512_object81004:head# 2024-04-29T14:26:36.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d2999d:::benchmark_data_smithi194_121512_object6385:head# 2024-04-29T14:26:36.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d39d80:::benchmark_data_smithi194_121512_object25432:head# 2024-04-29T14:26:36.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d45afb:::benchmark_data_smithi194_121512_object65678:head# 2024-04-29T14:26:36.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d5596e:::benchmark_data_smithi194_121512_object121720:head# 2024-04-29T14:26:36.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d5fa09:::benchmark_data_smithi194_121512_object47947:head# 2024-04-29T14:26:36.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d7f71d:::benchmark_data_smithi194_121512_object11751:head# 2024-04-29T14:26:36.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d8916f:::benchmark_data_smithi194_121512_object46046:head# 2024-04-29T14:26:36.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d89af4:::benchmark_data_smithi194_121512_object90471:head# 2024-04-29T14:26:36.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d8ed83:::benchmark_data_smithi194_121512_object42799:head# 2024-04-29T14:26:36.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d966e4:::benchmark_data_smithi194_121512_object22108:head# 2024-04-29T14:26:36.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d96720:::benchmark_data_smithi194_121512_object83262:head# 2024-04-29T14:26:36.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d97c33:::benchmark_data_smithi194_121512_object6862:head# 2024-04-29T14:26:36.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64d989fc:::benchmark_data_smithi194_121512_object82098:head# 2024-04-29T14:26:36.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64da88fc:::benchmark_data_smithi194_121512_object92455:head# 2024-04-29T14:26:36.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64daa7a2:::benchmark_data_smithi194_121512_object81023:head# 2024-04-29T14:26:36.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64dad584:::benchmark_data_smithi194_121512_object56832:head# 2024-04-29T14:26:36.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64dae713:::benchmark_data_smithi194_121512_object106972:head# 2024-04-29T14:26:36.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64db6b5a:::benchmark_data_smithi194_121512_object79085:head# 2024-04-29T14:26:36.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64db8c23:::benchmark_data_smithi194_121512_object84468:head# 2024-04-29T14:26:36.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64dc773f:::benchmark_data_smithi194_121512_object2982:head# 2024-04-29T14:26:36.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64dda657:::benchmark_data_smithi194_121512_object27826:head# 2024-04-29T14:26:36.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64dde1a4:::benchmark_data_smithi194_121512_object15963:head# 2024-04-29T14:26:36.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64de026a:::benchmark_data_smithi194_121512_object54488:head# 2024-04-29T14:26:36.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64dfdb5e:::benchmark_data_smithi194_121512_object478:head# 2024-04-29T14:26:36.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e008e3:::benchmark_data_smithi194_121512_object67179:head# 2024-04-29T14:26:36.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e0aa58:::benchmark_data_smithi194_121512_object7727:head# 2024-04-29T14:26:36.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e0b334:::benchmark_data_smithi194_121512_object44764:head# 2024-04-29T14:26:36.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e2019d:::benchmark_data_smithi194_121512_object133307:head# 2024-04-29T14:26:36.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e22a40:::benchmark_data_smithi194_121512_object69583:head# 2024-04-29T14:26:36.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e22dd3:::benchmark_data_smithi194_121512_object124650:head# 2024-04-29T14:26:36.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e2bacf:::benchmark_data_smithi194_121512_object91896:head# 2024-04-29T14:26:36.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e35e8e:::benchmark_data_smithi194_121512_object30003:head# 2024-04-29T14:26:36.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e35f3d:::benchmark_data_smithi194_121512_object129241:head# 2024-04-29T14:26:36.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e39d96:::benchmark_data_smithi194_121512_object47748:head# 2024-04-29T14:26:36.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e3d109:::benchmark_data_smithi194_121512_object63066:head# 2024-04-29T14:26:36.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e3d2fa:::benchmark_data_smithi194_121512_object8351:head# 2024-04-29T14:26:36.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e7479e:::benchmark_data_smithi194_121512_object57442:head# 2024-04-29T14:26:36.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e76843:::benchmark_data_smithi194_121512_object124500:head# 2024-04-29T14:26:36.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e79c08:::benchmark_data_smithi194_121512_object64972:head# 2024-04-29T14:26:36.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e7b119:::benchmark_data_smithi194_121512_object37465:head# 2024-04-29T14:26:36.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e82f80:::benchmark_data_smithi194_121512_object90079:head# 2024-04-29T14:26:36.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e8afd4:::benchmark_data_smithi194_121512_object96325:head# 2024-04-29T14:26:36.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e8c7cf:::benchmark_data_smithi194_121512_object16417:head# 2024-04-29T14:26:36.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e8cb80:::benchmark_data_smithi194_121512_object5539:head# 2024-04-29T14:26:36.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e8e49b:::benchmark_data_smithi194_121512_object94008:head# 2024-04-29T14:26:36.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64e9d1c8:::benchmark_data_smithi194_121512_object30623:head# 2024-04-29T14:26:36.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ea33f3:::benchmark_data_smithi194_121512_object3563:head# 2024-04-29T14:26:36.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ea557a:::benchmark_data_smithi194_121512_object130656:head# 2024-04-29T14:26:36.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64eaa1d9:::benchmark_data_smithi194_121512_object39882:head# 2024-04-29T14:26:36.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64eb16b1:::benchmark_data_smithi194_121512_object4852:head# 2024-04-29T14:26:36.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ec5c6b:::benchmark_data_smithi194_121512_object62765:head# 2024-04-29T14:26:36.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ec6aea:::benchmark_data_smithi194_121512_object92484:head# 2024-04-29T14:26:36.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ecd1bb:::benchmark_data_smithi194_121512_object52599:head# 2024-04-29T14:26:36.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ee6332:::benchmark_data_smithi194_121512_object97759:head# 2024-04-29T14:26:36.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64eeab54:::benchmark_data_smithi194_121512_object14747:head# 2024-04-29T14:26:36.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64eeaff8:::benchmark_data_smithi194_121512_object31405:head# 2024-04-29T14:26:36.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64eeb3f5:::benchmark_data_smithi194_121512_object132807:head# 2024-04-29T14:26:36.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f07e83:::benchmark_data_smithi194_121512_object10938:head# 2024-04-29T14:26:36.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f0fd0a:::benchmark_data_smithi194_121512_object45251:head# 2024-04-29T14:26:36.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f1057a:::benchmark_data_smithi194_121512_object39436:head# 2024-04-29T14:26:36.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f1d4c1:::benchmark_data_smithi194_121512_object53563:head# 2024-04-29T14:26:36.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f20584:::benchmark_data_smithi194_121512_object44367:head# 2024-04-29T14:26:36.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f2a5ad:::benchmark_data_smithi194_121512_object79140:head# 2024-04-29T14:26:36.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f36017:::benchmark_data_smithi194_121512_object51009:head# 2024-04-29T14:26:36.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f3a7c7:::benchmark_data_smithi194_121512_object24462:head# 2024-04-29T14:26:36.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f3e36b:::benchmark_data_smithi194_121512_object37697:head# 2024-04-29T14:26:36.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f3eeb2:::benchmark_data_smithi194_121512_object105882:head# 2024-04-29T14:26:36.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f422f2:::benchmark_data_smithi194_121512_object67993:head# 2024-04-29T14:26:36.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f48f55:::benchmark_data_smithi194_121512_object4326:head# 2024-04-29T14:26:36.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f627b7:::benchmark_data_smithi194_121512_object95722:head# 2024-04-29T14:26:36.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f65ca4:::benchmark_data_smithi194_121512_object102446:head# 2024-04-29T14:26:36.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f6a596:::benchmark_data_smithi194_121512_object83993:head# 2024-04-29T14:26:36.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f6e811:::benchmark_data_smithi194_121512_object25667:head# 2024-04-29T14:26:36.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f7c136:::benchmark_data_smithi194_121512_object88824:head# 2024-04-29T14:26:36.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f8a44d:::benchmark_data_smithi194_121512_object32591:head# 2024-04-29T14:26:36.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f92132:::benchmark_data_smithi194_121512_object10123:head# 2024-04-29T14:26:36.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f964ca:::benchmark_data_smithi194_121512_object50907:head# 2024-04-29T14:26:36.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f999c8:::benchmark_data_smithi194_121512_object28578:head# 2024-04-29T14:26:36.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64f9d478:::benchmark_data_smithi194_121512_object117055:head# 2024-04-29T14:26:36.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fa54f8:::benchmark_data_smithi194_121512_object131478:head# 2024-04-29T14:26:36.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fb4d2e:::benchmark_data_smithi194_121512_object131811:head# 2024-04-29T14:26:36.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fb9c1e:::benchmark_data_smithi194_121512_object115864:head# 2024-04-29T14:26:36.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fbbb3e:::benchmark_data_smithi194_121512_object32414:head# 2024-04-29T14:26:36.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fd2c79:::benchmark_data_smithi194_121512_object50216:head# 2024-04-29T14:26:36.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fd6797:::benchmark_data_smithi194_121512_object126605:head# 2024-04-29T14:26:36.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fdd286:::benchmark_data_smithi194_121512_object51443:head# 2024-04-29T14:26:36.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64fea2e8:::benchmark_data_smithi194_121512_object46347:head# 2024-04-29T14:26:36.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:64ffda6d:::benchmark_data_smithi194_121512_object34859:head# 2024-04-29T14:26:36.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650036ef:::benchmark_data_smithi194_121512_object123177:head# 2024-04-29T14:26:36.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6500e0ca:::benchmark_data_smithi194_121512_object128216:head# 2024-04-29T14:26:36.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6501206c:::benchmark_data_smithi194_121512_object118416:head# 2024-04-29T14:26:36.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65024f1f:::benchmark_data_smithi194_121512_object38580:head# 2024-04-29T14:26:36.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650258bf:::benchmark_data_smithi194_121512_object14613:head# 2024-04-29T14:26:36.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65029371:::benchmark_data_smithi194_121512_object131518:head# 2024-04-29T14:26:36.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6502ece0:::benchmark_data_smithi194_121512_object16829:head# 2024-04-29T14:26:36.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6502efbf:::benchmark_data_smithi194_121512_object22465:head# 2024-04-29T14:26:36.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65030a9d:::benchmark_data_smithi194_121512_object124740:head# 2024-04-29T14:26:36.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65034a86:::benchmark_data_smithi194_121512_object24114:head# 2024-04-29T14:26:36.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650350dc:::benchmark_data_smithi194_121512_object77948:head# 2024-04-29T14:26:36.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65037fda:::benchmark_data_smithi194_121512_object100125:head# 2024-04-29T14:26:36.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6504cba2:::benchmark_data_smithi194_121512_object126689:head# 2024-04-29T14:26:36.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6504f24a:::benchmark_data_smithi194_121512_object41940:head# 2024-04-29T14:26:36.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650545d3:::benchmark_data_smithi194_121512_object21066:head# 2024-04-29T14:26:36.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65067073:::benchmark_data_smithi194_121512_object30871:head# 2024-04-29T14:26:36.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65067413:::benchmark_data_smithi194_121512_object68558:head# 2024-04-29T14:26:36.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6506ceff:::benchmark_data_smithi194_121512_object57188:head# 2024-04-29T14:26:36.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650713bd:::benchmark_data_smithi194_121512_object83466:head# 2024-04-29T14:26:36.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650774fb:::benchmark_data_smithi194_121512_object115444:head# 2024-04-29T14:26:36.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650778eb:::benchmark_data_smithi194_121512_object65379:head# 2024-04-29T14:26:36.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6507e198:::benchmark_data_smithi194_121512_object70276:head# 2024-04-29T14:26:36.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6507edcd:::benchmark_data_smithi194_121512_object17690:head# 2024-04-29T14:26:36.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650b23cf:::benchmark_data_smithi194_121512_object131343:head# 2024-04-29T14:26:36.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650b5289:::benchmark_data_smithi194_121512_object94574:head# 2024-04-29T14:26:36.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650b7861:::benchmark_data_smithi194_121512_object118617:head# 2024-04-29T14:26:36.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650b976d:::benchmark_data_smithi194_121512_object42806:head# 2024-04-29T14:26:36.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650c3f0e:::benchmark_data_smithi194_121512_object127638:head# 2024-04-29T14:26:36.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650c7080:::benchmark_data_smithi194_121512_object88765:head# 2024-04-29T14:26:36.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650ce4d9:::benchmark_data_smithi194_121512_object713:head# 2024-04-29T14:26:36.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650dba8b:::benchmark_data_smithi194_121512_object3324:head# 2024-04-29T14:26:36.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650e2722:::benchmark_data_smithi194_121512_object113646:head# 2024-04-29T14:26:36.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650e378e:::benchmark_data_smithi194_121512_object99326:head# 2024-04-29T14:26:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650e3ade:::benchmark_data_smithi194_121512_object88678:head# 2024-04-29T14:26:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650e5160:::benchmark_data_smithi194_121512_object73556:head# 2024-04-29T14:26:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650ea54e:::benchmark_data_smithi194_121512_object110605:head# 2024-04-29T14:26:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650f6150:::benchmark_data_smithi194_121512_object27368:head# 2024-04-29T14:26:36.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:650fbe2a:::benchmark_data_smithi194_121512_object32806:head# 2024-04-29T14:26:36.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6510ead9:::benchmark_data_smithi194_121512_object98234:head# 2024-04-29T14:26:36.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65118ec7:::benchmark_data_smithi194_121512_object42163:head# 2024-04-29T14:26:36.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6512018d:::benchmark_data_smithi194_121512_object55205:head# 2024-04-29T14:26:36.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65123891:::benchmark_data_smithi194_121512_object129765:head# 2024-04-29T14:26:36.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6512f846:::benchmark_data_smithi194_121512_object92528:head# 2024-04-29T14:26:36.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65133f63:::benchmark_data_smithi194_121512_object78741:head# 2024-04-29T14:26:36.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6513899b:::benchmark_data_smithi194_121512_object29889:head# 2024-04-29T14:26:36.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65145440:::benchmark_data_smithi194_121512_object19819:head# 2024-04-29T14:26:36.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6515b3f2:::benchmark_data_smithi194_121512_object10658:head# 2024-04-29T14:26:36.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651615ac:::benchmark_data_smithi194_121512_object123199:head# 2024-04-29T14:26:36.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65166705:::benchmark_data_smithi194_121512_object86382:head# 2024-04-29T14:26:36.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6516edc0:::benchmark_data_smithi194_121512_object49539:head# 2024-04-29T14:26:36.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65180e58:::benchmark_data_smithi194_121512_object12510:head# 2024-04-29T14:26:36.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65184205:::benchmark_data_smithi194_121512_object78493:head# 2024-04-29T14:26:36.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6518d194:::benchmark_data_smithi194_121512_object69524:head# 2024-04-29T14:26:36.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65195d52:::benchmark_data_smithi194_121512_object46155:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6519eea9:::benchmark_data_smithi194_121512_object128834:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651a04aa:::benchmark_data_smithi194_121512_object97868:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651a6427:::benchmark_data_smithi194_121512_object56043:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651a8937:::benchmark_data_smithi194_121512_object128729:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651bb7b7:::benchmark_data_smithi194_121512_object113483:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651cf157:::benchmark_data_smithi194_121512_object14909:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651d0f59:::benchmark_data_smithi194_121512_object69298:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651df2e1:::benchmark_data_smithi194_121512_object40050:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:651f3933:::benchmark_data_smithi194_121512_object72650:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652006dc:::benchmark_data_smithi194_121512_object39495:head# 2024-04-29T14:26:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6520bc1f:::benchmark_data_smithi194_121512_object92194:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6520cf21:::benchmark_data_smithi194_121512_object125951:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65216d4c:::benchmark_data_smithi194_121512_object109962:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65221fe9:::benchmark_data_smithi194_121512_object92398:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65224e1a:::benchmark_data_smithi194_121512_object36990:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65226ff0:::benchmark_data_smithi194_121512_object19214:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65227b63:::benchmark_data_smithi194_121512_object65083:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6522bf4f:::benchmark_data_smithi194_121512_object70894:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65231ecd:::benchmark_data_smithi194_121512_object2107:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65237692:::benchmark_data_smithi194_121512_object119136:head# 2024-04-29T14:26:36.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6523e945:::benchmark_data_smithi194_121512_object26933:head# 2024-04-29T14:26:36.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65259799:::benchmark_data_smithi194_121512_object34193:head# 2024-04-29T14:26:36.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65259bd3:::benchmark_data_smithi194_121512_object113777:head# 2024-04-29T14:26:36.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6525e20e:::benchmark_data_smithi194_121512_object79892:head# 2024-04-29T14:26:36.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65262e81:::benchmark_data_smithi194_121512_object104761:head# 2024-04-29T14:26:36.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65267957:::benchmark_data_smithi194_121512_object129998:head# 2024-04-29T14:26:36.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652871ed:::benchmark_data_smithi194_121512_object83585:head# 2024-04-29T14:26:36.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65290d49:::benchmark_data_smithi194_121512_object3244:head# 2024-04-29T14:26:36.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652a59c6:::benchmark_data_smithi194_121512_object39478:head# 2024-04-29T14:26:36.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652ab123:::benchmark_data_smithi194_121512_object18140:head# 2024-04-29T14:26:36.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652ae3aa:::benchmark_data_smithi194_121512_object117982:head# 2024-04-29T14:26:36.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652b9c2b:::benchmark_data_smithi194_121512_object17128:head# 2024-04-29T14:26:36.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652bcdbe:::benchmark_data_smithi194_121512_object124672:head# 2024-04-29T14:26:36.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652bff0c:::benchmark_data_smithi194_121512_object62948:head# 2024-04-29T14:26:36.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652c4df8:::benchmark_data_smithi194_121512_object52515:head# 2024-04-29T14:26:36.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652c58a4:::benchmark_data_smithi194_121512_object11584:head# 2024-04-29T14:26:36.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652cdee4:::benchmark_data_smithi194_121512_object73049:head# 2024-04-29T14:26:36.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652ceb91:::benchmark_data_smithi194_121512_object79633:head# 2024-04-29T14:26:36.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652d55ba:::benchmark_data_smithi194_121512_object121433:head# 2024-04-29T14:26:36.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652dca78:::benchmark_data_smithi194_121512_object63330:head# 2024-04-29T14:26:36.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652dd719:::benchmark_data_smithi194_121512_object19597:head# 2024-04-29T14:26:36.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652df6e5:::benchmark_data_smithi194_121512_object88832:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652dfb98:::benchmark_data_smithi194_121512_object23967:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652ecf16:::benchmark_data_smithi194_121512_object102671:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652f523b:::benchmark_data_smithi194_121512_object80465:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:652f6393:::benchmark_data_smithi194_121512_object104834:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65327c90:::benchmark_data_smithi194_121512_object40254:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6532db55:::benchmark_data_smithi194_121512_object115102:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6532fa09:::benchmark_data_smithi194_121512_object49905:head# 2024-04-29T14:26:36.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65333cb0:::benchmark_data_smithi194_121512_object57705:head# 2024-04-29T14:26:36.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65336f54:::benchmark_data_smithi194_121512_object43467:head# 2024-04-29T14:26:36.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6533cf24:::benchmark_data_smithi194_121512_object75992:head# 2024-04-29T14:26:36.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653411f9:::benchmark_data_smithi194_121512_object15110:head# 2024-04-29T14:26:36.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653422df:::benchmark_data_smithi194_121512_object23725:head# 2024-04-29T14:26:36.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65345147:::benchmark_data_smithi194_121512_object36309:head# 2024-04-29T14:26:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6535ddd3:::benchmark_data_smithi194_121512_object97346:head# 2024-04-29T14:26:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6535e1c4:::benchmark_data_smithi194_121512_object80835:head# 2024-04-29T14:26:36.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65363ffc:::benchmark_data_smithi194_121512_object125713:head# 2024-04-29T14:26:36.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65368cf7:::benchmark_data_smithi194_121512_object26586:head# 2024-04-29T14:26:36.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65373689:::benchmark_data_smithi194_121512_object60730:head# 2024-04-29T14:26:36.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65379122:::benchmark_data_smithi194_121512_object98979:head# 2024-04-29T14:26:36.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6538a588:::benchmark_data_smithi194_121512_object42847:head# 2024-04-29T14:26:36.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6538d508:::benchmark_data_smithi194_121512_object30844:head# 2024-04-29T14:26:36.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65398a1a:::benchmark_data_smithi194_121512_object107901:head# 2024-04-29T14:26:36.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65398a92:::benchmark_data_smithi194_121512_object33445:head# 2024-04-29T14:26:36.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6539ab11:::benchmark_data_smithi194_121512_object23836:head# 2024-04-29T14:26:36.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653a011e:::benchmark_data_smithi194_121512_object18845:head# 2024-04-29T14:26:36.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653aca17:::benchmark_data_smithi194_121512_object122577:head# 2024-04-29T14:26:36.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653aea7b:::benchmark_data_smithi194_121512_object63210:head# 2024-04-29T14:26:36.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653b1c2a:::benchmark_data_smithi194_121512_object8789:head# 2024-04-29T14:26:36.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653bfc0a:::benchmark_data_smithi194_121512_object26917:head# 2024-04-29T14:26:36.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653c1e45:::benchmark_data_smithi194_121512_object40216:head# 2024-04-29T14:26:36.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653cde72:::benchmark_data_smithi194_121512_object20119:head# 2024-04-29T14:26:36.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653d1886:::benchmark_data_smithi194_121512_object63879:head# 2024-04-29T14:26:36.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653d98ca:::benchmark_data_smithi194_121512_object107843:head# 2024-04-29T14:26:36.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653ed6eb:::benchmark_data_smithi194_121512_object54665:head# 2024-04-29T14:26:36.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:653ff84e:::benchmark_data_smithi194_121512_object111019:head# 2024-04-29T14:26:36.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65416669:::benchmark_data_smithi194_121512_object38653:head# 2024-04-29T14:26:36.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65416fd5:::benchmark_data_smithi194_121512_object118807:head# 2024-04-29T14:26:36.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65437458:::benchmark_data_smithi194_121512_object126872:head# 2024-04-29T14:26:36.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6543f45e:::benchmark_data_smithi194_121512_object113895:head# 2024-04-29T14:26:36.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65446598:::benchmark_data_smithi194_121512_object78992:head# 2024-04-29T14:26:36.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6544e878:::benchmark_data_smithi194_121512_object60952:head# 2024-04-29T14:26:36.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65458327:::benchmark_data_smithi194_121512_object128854:head# 2024-04-29T14:26:36.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65474a6d:::benchmark_data_smithi194_121512_object6481:head# 2024-04-29T14:26:36.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6547675a:::benchmark_data_smithi194_121512_object38115:head# 2024-04-29T14:26:36.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6547a3a2:::benchmark_data_smithi194_121512_object106291:head# 2024-04-29T14:26:36.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6548bb46:::benchmark_data_smithi194_121512_object76144:head# 2024-04-29T14:26:36.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6549e83a:::benchmark_data_smithi194_121512_object61537:head# 2024-04-29T14:26:36.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654a6058:::benchmark_data_smithi194_121512_object64766:head# 2024-04-29T14:26:36.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654aab2a:::benchmark_data_smithi194_121512_object84972:head# 2024-04-29T14:26:36.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654b1bc8:::benchmark_data_smithi194_121512_object58430:head# 2024-04-29T14:26:36.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654b5aeb:::benchmark_data_smithi194_121512_object92147:head# 2024-04-29T14:26:36.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654b7149:::benchmark_data_smithi194_121512_object83289:head# 2024-04-29T14:26:36.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654bb09e:::benchmark_data_smithi194_121512_object88121:head# 2024-04-29T14:26:36.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654be810:::benchmark_data_smithi194_121512_object9020:head# 2024-04-29T14:26:36.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654c1340:::benchmark_data_smithi194_121512_object17630:head# 2024-04-29T14:26:36.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654c57da:::benchmark_data_smithi194_121512_object2426:head# 2024-04-29T14:26:36.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654c7cf9:::benchmark_data_smithi194_121512_object63361:head# 2024-04-29T14:26:36.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654d50c5:::benchmark_data_smithi194_121512_object78041:head# 2024-04-29T14:26:36.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654de0fa:::benchmark_data_smithi194_121512_object89247:head# 2024-04-29T14:26:36.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654df28f:::benchmark_data_smithi194_121512_object70589:head# 2024-04-29T14:26:36.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654e23cb:::benchmark_data_smithi194_121512_object4120:head# 2024-04-29T14:26:36.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654e8cdd:::benchmark_data_smithi194_121512_object102127:head# 2024-04-29T14:26:36.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654f8123:::benchmark_data_smithi194_121512_object90524:head# 2024-04-29T14:26:36.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:654f996a:::benchmark_data_smithi194_121512_object129548:head# 2024-04-29T14:26:36.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6550628d:::benchmark_data_smithi194_121512_object115435:head# 2024-04-29T14:26:36.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6550f501:::benchmark_data_smithi194_121512_object1742:head# 2024-04-29T14:26:36.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65529934:::benchmark_data_smithi194_121512_object54636:head# 2024-04-29T14:26:36.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6552d827:::benchmark_data_smithi194_121512_object101402:head# 2024-04-29T14:26:36.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65534070:::benchmark_data_smithi194_121512_object13897:head# 2024-04-29T14:26:36.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6553827d:::benchmark_data_smithi194_121512_object89180:head# 2024-04-29T14:26:36.818 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:36.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655411f5:::benchmark_data_smithi194_121512_object119926:head# 2024-04-29T14:26:36.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6554ed27:::benchmark_data_smithi194_121512_object59854:head# 2024-04-29T14:26:36.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6555adcd:::benchmark_data_smithi194_121512_object93944:head# 2024-04-29T14:26:36.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6555d3b0:::benchmark_data_smithi194_121512_object103839:head# 2024-04-29T14:26:36.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65560ca7:::benchmark_data_smithi194_121512_object11046:head# 2024-04-29T14:26:36.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6556480b:::benchmark_data_smithi194_121512_object104939:head# 2024-04-29T14:26:36.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6556507c:::benchmark_data_smithi194_121512_object126905:head# 2024-04-29T14:26:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:36.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:36.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6556fd8a:::benchmark_data_smithi194_121512_object121738:head# 2024-04-29T14:26:36.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655738b9:::benchmark_data_smithi194_121512_object130876:head# 2024-04-29T14:26:36.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65574867:::benchmark_data_smithi194_121512_object117422:head# 2024-04-29T14:26:36.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65582fbb:::benchmark_data_smithi194_121512_object122747:head# 2024-04-29T14:26:36.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65588fbb:::benchmark_data_smithi194_121512_object89616:head# 2024-04-29T14:26:36.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6558c803:::benchmark_data_smithi194_121512_object35067:head# 2024-04-29T14:26:36.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6559ec1a:::benchmark_data_smithi194_121512_object124770:head# 2024-04-29T14:26:36.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6559fd2a:::benchmark_data_smithi194_121512_object97478:head# 2024-04-29T14:26:36.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655a9c45:::benchmark_data_smithi194_121512_object98510:head# 2024-04-29T14:26:36.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655acce7:::benchmark_data_smithi194_121512_object56852:head# 2024-04-29T14:26:36.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655b13c9:::benchmark_data_smithi194_121512_object109994:head# 2024-04-29T14:26:36.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655b4fcd:::benchmark_data_smithi194_121512_object98296:head# 2024-04-29T14:26:36.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655c0b33:::benchmark_data_smithi194_121512_object109180:head# 2024-04-29T14:26:36.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655c97da:::benchmark_data_smithi194_121512_object23232:head# 2024-04-29T14:26:36.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655caedb:::benchmark_data_smithi194_121512_object12917:head# 2024-04-29T14:26:36.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655cfb28:::benchmark_data_smithi194_121512_object70050:head# 2024-04-29T14:26:36.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655d4d92:::benchmark_data_smithi194_121512_object114232:head# 2024-04-29T14:26:36.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655d6de0:::benchmark_data_smithi194_121512_object84996:head# 2024-04-29T14:26:36.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655d7cc9:::benchmark_data_smithi194_121512_object109178:head# 2024-04-29T14:26:36.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655d8ac1:::benchmark_data_smithi194_121512_object105764:head# 2024-04-29T14:26:36.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655f201b:::benchmark_data_smithi194_121512_object97627:head# 2024-04-29T14:26:36.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655f4dd9:::benchmark_data_smithi194_121512_object67748:head# 2024-04-29T14:26:36.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:655ff704:::benchmark_data_smithi194_121512_object82555:head# 2024-04-29T14:26:36.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65605f54:::benchmark_data_smithi194_121512_object9091:head# 2024-04-29T14:26:36.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6560fb65:::benchmark_data_smithi194_121512_object122919:head# 2024-04-29T14:26:36.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65614d53:::benchmark_data_smithi194_121512_object124549:head# 2024-04-29T14:26:36.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65614fba:::benchmark_data_smithi194_121512_object85796:head# 2024-04-29T14:26:36.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6561b82e:::benchmark_data_smithi194_121512_object131225:head# 2024-04-29T14:26:36.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6561e73b:::benchmark_data_smithi194_121512_object1172:head# 2024-04-29T14:26:36.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65624ad6:::benchmark_data_smithi194_121512_object120012:head# 2024-04-29T14:26:36.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6562928e:::benchmark_data_smithi194_121512_object69571:head# 2024-04-29T14:26:36.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65629e50:::benchmark_data_smithi194_121512_object53602:head# 2024-04-29T14:26:36.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6563847f:::benchmark_data_smithi194_121512_object124429:head# 2024-04-29T14:26:36.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6563928b:::benchmark_data_smithi194_121512_object56792:head# 2024-04-29T14:26:36.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6564049d:::benchmark_data_smithi194_121512_object37124:head# 2024-04-29T14:26:36.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656470e0:::benchmark_data_smithi194_121512_object128878:head# 2024-04-29T14:26:36.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6564a7df:::benchmark_data_smithi194_121512_object116578:head# 2024-04-29T14:26:36.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6565575d:::benchmark_data_smithi194_121512_object51630:head# 2024-04-29T14:26:36.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65657e3d:::benchmark_data_smithi194_121512_object121752:head# 2024-04-29T14:26:36.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6566881f:::benchmark_data_smithi194_121512_object109480:head# 2024-04-29T14:26:36.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65669d39:::benchmark_data_smithi194_121512_object23526:head# 2024-04-29T14:26:36.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6566a54b:::benchmark_data_smithi194_121512_object1581:head# 2024-04-29T14:26:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6566f122:::benchmark_data_smithi194_121512_object95825:head# 2024-04-29T14:26:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6566f5c4:::benchmark_data_smithi194_121512_object91249:head# 2024-04-29T14:26:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65674053:::benchmark_data_smithi194_121512_object122059:head# 2024-04-29T14:26:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65685227:::benchmark_data_smithi194_121512_object53398:head# 2024-04-29T14:26:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65689bd5:::benchmark_data_smithi194_121512_object118202:head# 2024-04-29T14:26:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6568ea66:::benchmark_data_smithi194_121512_object61671:head# 2024-04-29T14:26:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6568f031:::benchmark_data_smithi194_121512_object25302:head# 2024-04-29T14:26:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65691a3c:::benchmark_data_smithi194_121512_object121038:head# 2024-04-29T14:26:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6569a6c4:::benchmark_data_smithi194_121512_object20932:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6569d5f9:::benchmark_data_smithi194_121512_object39733:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6569f6ad:::benchmark_data_smithi194_121512_object119695:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656a6707:::benchmark_data_smithi194_121512_object7442:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656b0178:::benchmark_data_smithi194_121512_object45772:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656b0499:::benchmark_data_smithi194_121512_object56845:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656b7cc7:::benchmark_data_smithi194_121512_object64147:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656ba6c5:::benchmark_data_smithi194_121512_object40061:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656c1476:::benchmark_data_smithi194_121512_object122290:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656d13a4:::benchmark_data_smithi194_121512_object127121:head# 2024-04-29T14:26:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656ef0cc:::benchmark_data_smithi194_121512_object123246:head# 2024-04-29T14:26:36.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656f1bd5:::benchmark_data_smithi194_121512_object132552:head# 2024-04-29T14:26:36.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656ff8a5:::benchmark_data_smithi194_121512_object30811:head# 2024-04-29T14:26:36.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:656ff989:::benchmark_data_smithi194_121512_object100302:head# 2024-04-29T14:26:36.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6570640d:::benchmark_data_smithi194_121512_object29051:head# 2024-04-29T14:26:36.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6571b642:::benchmark_data_smithi194_121512_object86381:head# 2024-04-29T14:26:36.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6571ef96:::benchmark_data_smithi194_121512_object90564:head# 2024-04-29T14:26:36.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6572c09b:::benchmark_data_smithi194_121512_object13923:head# 2024-04-29T14:26:36.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6572dcce:::benchmark_data_smithi194_121512_object68009:head# 2024-04-29T14:26:36.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6572e137:::benchmark_data_smithi194_121512_object87647:head# 2024-04-29T14:26:36.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65731e85:::benchmark_data_smithi194_121512_object88580:head# 2024-04-29T14:26:36.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6573f562:::benchmark_data_smithi194_121512_object95296:head# 2024-04-29T14:26:36.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65748a76:::benchmark_data_smithi194_121512_object17982:head# 2024-04-29T14:26:36.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6574a58e:::benchmark_data_smithi194_121512_object66808:head# 2024-04-29T14:26:36.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6574d1ce:::benchmark_data_smithi194_121512_object17648:head# 2024-04-29T14:26:36.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6575349e:::benchmark_data_smithi194_121512_object115743:head# 2024-04-29T14:26:36.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657556eb:::benchmark_data_smithi194_121512_object108385:head# 2024-04-29T14:26:36.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6575878b:::benchmark_data_smithi194_121512_object266:head# 2024-04-29T14:26:36.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65759c2a:::benchmark_data_smithi194_121512_object30264:head# 2024-04-29T14:26:36.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6575e843:::benchmark_data_smithi194_121512_object88113:head# 2024-04-29T14:26:36.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65760b5b:::benchmark_data_smithi194_121512_object8298:head# 2024-04-29T14:26:36.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65766303:::benchmark_data_smithi194_121512_object71369:head# 2024-04-29T14:26:36.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6576ccd7:::benchmark_data_smithi194_121512_object102071:head# 2024-04-29T14:26:36.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6577102e:::benchmark_data_smithi194_121512_object28890:head# 2024-04-29T14:26:36.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6577120e:::benchmark_data_smithi194_121512_object129049:head# 2024-04-29T14:26:36.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65782c04:::benchmark_data_smithi194_121512_object133383:head# 2024-04-29T14:26:36.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6578613c:::benchmark_data_smithi194_121512_object73574:head# 2024-04-29T14:26:36.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6579bed7:::benchmark_data_smithi194_121512_object118027:head# 2024-04-29T14:26:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657a2262:::benchmark_data_smithi194_121512_object30834:head# 2024-04-29T14:26:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657b7e18:::benchmark_data_smithi194_121512_object97790:head# 2024-04-29T14:26:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657bb644:::benchmark_data_smithi194_121512_object55403:head# 2024-04-29T14:26:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657cae0e:::benchmark_data_smithi194_121512_object111561:head# 2024-04-29T14:26:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657d598a:::benchmark_data_smithi194_121512_object72057:head# 2024-04-29T14:26:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657e2ba3:::benchmark_data_smithi194_121512_object1020:head# 2024-04-29T14:26:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657ea150:::benchmark_data_smithi194_121512_object52639:head# 2024-04-29T14:26:36.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657ede44:::benchmark_data_smithi194_121512_object51712:head# 2024-04-29T14:26:36.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657f5b83:::benchmark_data_smithi194_121512_object35337:head# 2024-04-29T14:26:36.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657f66bc:::benchmark_data_smithi194_121512_object45384:head# 2024-04-29T14:26:36.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657f77e6:::benchmark_data_smithi194_121512_object105415:head# 2024-04-29T14:26:36.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657f79b8:::benchmark_data_smithi194_121512_object131330:head# 2024-04-29T14:26:36.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657f7d68:::benchmark_data_smithi194_121512_object75231:head# 2024-04-29T14:26:36.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:657fb8a7:::benchmark_data_smithi194_121512_object85463:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6580764e:::benchmark_data_smithi194_121512_object110574:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6581f8b7:::benchmark_data_smithi194_121512_object9982:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6581fe18:::benchmark_data_smithi194_121512_object117352:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6582006c:::benchmark_data_smithi194_121512_object24529:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65821f36:::benchmark_data_smithi194_121512_object23416:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65828f44:::benchmark_data_smithi194_121512_object88400:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6582b39d:::benchmark_data_smithi194_121512_object67711:head# 2024-04-29T14:26:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65852e04:::benchmark_data_smithi194_121512_object8382:head# 2024-04-29T14:26:36.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65859c91:::benchmark_data_smithi194_121512_object18726:head# 2024-04-29T14:26:36.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6585c7d5:::benchmark_data_smithi194_121512_object95096:head# 2024-04-29T14:26:36.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6585cf66:::benchmark_data_smithi194_121512_object82130:head# 2024-04-29T14:26:36.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65867276:::benchmark_data_smithi194_121512_object114353:head# 2024-04-29T14:26:36.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658718f6:::benchmark_data_smithi194_121512_object98284:head# 2024-04-29T14:26:36.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65877410:::benchmark_data_smithi194_121512_object35667:head# 2024-04-29T14:26:36.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65884dbf:::benchmark_data_smithi194_121512_object114403:head# 2024-04-29T14:26:36.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65889451:::benchmark_data_smithi194_121512_object5291:head# 2024-04-29T14:26:36.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6588c2a6:::benchmark_data_smithi194_121512_object40433:head# 2024-04-29T14:26:36.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6589b08b:::benchmark_data_smithi194_121512_object16011:head# 2024-04-29T14:26:36.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658a82d0:::benchmark_data_smithi194_121512_object28950:head# 2024-04-29T14:26:36.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658aaaad:::benchmark_data_smithi194_121512_object81753:head# 2024-04-29T14:26:36.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658aeba5:::benchmark_data_smithi194_121512_object93343:head# 2024-04-29T14:26:36.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658b8b80:::benchmark_data_smithi194_121512_object94635:head# 2024-04-29T14:26:36.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658baa49:::benchmark_data_smithi194_121512_object2802:head# 2024-04-29T14:26:36.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658cdd98:::benchmark_data_smithi194_121512_object106484:head# 2024-04-29T14:26:36.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658d07f1:::benchmark_data_smithi194_121512_object51190:head# 2024-04-29T14:26:36.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658d5820:::benchmark_data_smithi194_121512_object108542:head# 2024-04-29T14:26:36.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658d6fb4:::benchmark_data_smithi194_121512_object110879:head# 2024-04-29T14:26:36.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658def58:::benchmark_data_smithi194_121512_object132574:head# 2024-04-29T14:26:36.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658e1f6a:::benchmark_data_smithi194_121512_object118904:head# 2024-04-29T14:26:36.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:658faef1:::benchmark_data_smithi194_121512_object51390:head# 2024-04-29T14:26:36.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6590026d:::benchmark_data_smithi194_121512_object43556:head# 2024-04-29T14:26:36.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659058f2:::benchmark_data_smithi194_121512_object28322:head# 2024-04-29T14:26:36.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6590ac29:::benchmark_data_smithi194_121512_object58821:head# 2024-04-29T14:26:36.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6590d33d:::benchmark_data_smithi194_121512_object7161:head# 2024-04-29T14:26:36.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6590e802:::benchmark_data_smithi194_121512_object87662:head# 2024-04-29T14:26:36.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65910cd9:::benchmark_data_smithi194_121512_object53878:head# 2024-04-29T14:26:36.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659144d8:::benchmark_data_smithi194_121512_object66049:head# 2024-04-29T14:26:36.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65916a97:::benchmark_data_smithi194_121512_object41073:head# 2024-04-29T14:26:36.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65918b71:::benchmark_data_smithi194_121512_object38121:head# 2024-04-29T14:26:36.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6591a2e0:::benchmark_data_smithi194_121512_object104394:head# 2024-04-29T14:26:36.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6592919b:::benchmark_data_smithi194_121512_object93566:head# 2024-04-29T14:26:36.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6592bde4:::benchmark_data_smithi194_121512_object43132:head# 2024-04-29T14:26:36.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6592bfff:::benchmark_data_smithi194_121512_object43711:head# 2024-04-29T14:26:36.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6593c150:::benchmark_data_smithi194_121512_object35701:head# 2024-04-29T14:26:36.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6594674c:::benchmark_data_smithi194_121512_object121893:head# 2024-04-29T14:26:36.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6594f3da:::benchmark_data_smithi194_121512_object42635:head# 2024-04-29T14:26:36.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65954f9e:::benchmark_data_smithi194_121512_object100704:head# 2024-04-29T14:26:36.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65958fa7:::benchmark_data_smithi194_121512_object55504:head# 2024-04-29T14:26:36.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65960032:::benchmark_data_smithi194_121512_object3448:head# 2024-04-29T14:26:36.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65961fa4:::benchmark_data_smithi194_121512_object27342:head# 2024-04-29T14:26:36.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659688c2:::benchmark_data_smithi194_121512_object53068:head# 2024-04-29T14:26:36.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65973388:::benchmark_data_smithi194_121512_object107283:head# 2024-04-29T14:26:36.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6597ec07:::benchmark_data_smithi194_121512_object33490:head# 2024-04-29T14:26:36.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65985c9a:::benchmark_data_smithi194_121512_object55717:head# 2024-04-29T14:26:36.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659865e6:::benchmark_data_smithi194_121512_object14653:head# 2024-04-29T14:26:36.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65989fd6:::benchmark_data_smithi194_121512_object117336:head# 2024-04-29T14:26:36.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659916cf:::benchmark_data_smithi194_121512_object42507:head# 2024-04-29T14:26:36.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65993b6b:::benchmark_data_smithi194_121512_object131639:head# 2024-04-29T14:26:36.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65993ef1:::benchmark_data_smithi194_121512_object63472:head# 2024-04-29T14:26:36.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6599e422:::benchmark_data_smithi194_121512_object44374:head# 2024-04-29T14:26:36.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659a1276:::benchmark_data_smithi194_121512_object24441:head# 2024-04-29T14:26:36.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659c2dad:::benchmark_data_smithi194_121512_object75395:head# 2024-04-29T14:26:36.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659c44b4:::benchmark_data_smithi194_121512_object30137:head# 2024-04-29T14:26:36.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659d6015:::benchmark_data_smithi194_121512_object130688:head# 2024-04-29T14:26:36.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659da3f4:::benchmark_data_smithi194_121512_object6366:head# 2024-04-29T14:26:36.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659e1a4e:::benchmark_data_smithi194_121512_object53120:head# 2024-04-29T14:26:36.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659e2a60:::benchmark_data_smithi194_121512_object32938:head# 2024-04-29T14:26:36.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659e6fa6:::benchmark_data_smithi194_121512_object24489:head# 2024-04-29T14:26:36.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659e89c7:::benchmark_data_smithi194_121512_object72516:head# 2024-04-29T14:26:36.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659f707e:::benchmark_data_smithi194_121512_object93116:head# 2024-04-29T14:26:36.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:659fe6d9:::benchmark_data_smithi194_121512_object59098:head# 2024-04-29T14:26:36.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a03f2f:::benchmark_data_smithi194_121512_object129019:head# 2024-04-29T14:26:36.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a0523c:::benchmark_data_smithi194_121512_object74211:head# 2024-04-29T14:26:36.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a0c8a2:::benchmark_data_smithi194_121512_object15130:head# 2024-04-29T14:26:36.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a15e43:::benchmark_data_smithi194_121512_object115898:head# 2024-04-29T14:26:36.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a1661c:::benchmark_data_smithi194_121512_object21397:head# 2024-04-29T14:26:36.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a18642:::benchmark_data_smithi194_121512_object41632:head# 2024-04-29T14:26:36.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a1d3ca:::benchmark_data_smithi194_121512_object23842:head# 2024-04-29T14:26:36.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:36.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a30667:::benchmark_data_smithi194_121512_object115901:head# 2024-04-29T14:26:36.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a33653:::benchmark_data_smithi194_121512_object39382:head# 2024-04-29T14:26:36.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a3d243:::benchmark_data_smithi194_121512_object97441:head# 2024-04-29T14:26:36.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a3f447:::benchmark_data_smithi194_121512_object92542:head# 2024-04-29T14:26:36.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a3f8ba:::benchmark_data_smithi194_121512_object31089:head# 2024-04-29T14:26:36.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a4652c:::benchmark_data_smithi194_121512_object42276:head# 2024-04-29T14:26:36.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a56169:::benchmark_data_smithi194_121512_object11973:head# 2024-04-29T14:26:36.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a5955f:::benchmark_data_smithi194_121512_object20136:head# 2024-04-29T14:26:36.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a5b691:::benchmark_data_smithi194_121512_object97021:head# 2024-04-29T14:26:36.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a5c87f:::benchmark_data_smithi194_121512_object82840:head# 2024-04-29T14:26:36.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a5cb3e:::benchmark_data_smithi194_121512_object125079:head# 2024-04-29T14:26:36.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a67d78:::benchmark_data_smithi194_121512_object26900:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a6b2c7:::benchmark_data_smithi194_121512_object90587:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a6f9b3:::benchmark_data_smithi194_121512_object19963:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a7c2e7:::benchmark_data_smithi194_121512_object118068:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a82c89:::benchmark_data_smithi194_121512_object127857:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a8709d:::benchmark_data_smithi194_121512_object93693:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a8961e:::benchmark_data_smithi194_121512_object110367:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65a9d9eb:::benchmark_data_smithi194_121512_object2850:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65aa3a65:::benchmark_data_smithi194_121512_object17437:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65aa3e48:::benchmark_data_smithi194_121512_object14510:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65abf29e:::benchmark_data_smithi194_121512_object26762:head# 2024-04-29T14:26:36.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ae9660:::benchmark_data_smithi194_121512_object130824:head# 2024-04-29T14:26:36.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ae9c2c:::benchmark_data_smithi194_121512_object98902:head# 2024-04-29T14:26:36.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65af104b:::benchmark_data_smithi194_121512_object78743:head# 2024-04-29T14:26:36.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65af19d3:::benchmark_data_smithi194_121512_object65237:head# 2024-04-29T14:26:36.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65af7309:::benchmark_data_smithi194_121512_object14302:head# 2024-04-29T14:26:36.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65afa600:::benchmark_data_smithi194_121512_object41939:head# 2024-04-29T14:26:36.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65afc4df:::benchmark_data_smithi194_121512_object104355:head# 2024-04-29T14:26:36.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65afe071:::benchmark_data_smithi194_121512_object70373:head# 2024-04-29T14:26:36.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b01bbf:::benchmark_data_smithi194_121512_object105244:head# 2024-04-29T14:26:36.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b07019:::benchmark_data_smithi194_121512_object13623:head# 2024-04-29T14:26:36.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b0d033:::benchmark_data_smithi194_121512_object29469:head# 2024-04-29T14:26:36.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b0de2f:::benchmark_data_smithi194_121512_object13006:head# 2024-04-29T14:26:36.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b10d07:::benchmark_data_smithi194_121512_object55854:head# 2024-04-29T14:26:36.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b1b1bc:::benchmark_data_smithi194_121512_object27367:head# 2024-04-29T14:26:36.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b1cf7e:::benchmark_data_smithi194_121512_object12722:head# 2024-04-29T14:26:36.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b28d53:::benchmark_data_smithi194_121512_object28530:head# 2024-04-29T14:26:36.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b2dba5:::benchmark_data_smithi194_121512_object95429:head# 2024-04-29T14:26:36.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b2e2ef:::benchmark_data_smithi194_121512_object105842:head# 2024-04-29T14:26:36.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b3082b:::benchmark_data_smithi194_121512_object65622:head# 2024-04-29T14:26:36.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b31a84:::benchmark_data_smithi194_121512_object102474:head# 2024-04-29T14:26:36.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b34fd0:::benchmark_data_smithi194_121512_object15982:head# 2024-04-29T14:26:36.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b3dd62:::benchmark_data_smithi194_121512_object109154:head# 2024-04-29T14:26:36.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b3eba9:::benchmark_data_smithi194_121512_object9605:head# 2024-04-29T14:26:36.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b46bc7:::benchmark_data_smithi194_121512_object21499:head# 2024-04-29T14:26:36.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b48b0e:::benchmark_data_smithi194_121512_object2328:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b4daa3:::benchmark_data_smithi194_121512_object93853:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b4daee:::benchmark_data_smithi194_121512_object93740:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b643bf:::benchmark_data_smithi194_121512_object26548:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b7cb66:::benchmark_data_smithi194_121512_object80569:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b859c2:::benchmark_data_smithi194_121512_object102546:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65b9fa86:::benchmark_data_smithi194_121512_object51289:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ba6f2c:::benchmark_data_smithi194_121512_object10330:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ba90ec:::benchmark_data_smithi194_121512_object61400:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bad191:::benchmark_data_smithi194_121512_object78891:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bb6235:::benchmark_data_smithi194_121512_object83127:head# 2024-04-29T14:26:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bb81bd:::benchmark_data_smithi194_121512_object26248:head# 2024-04-29T14:26:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bb96a3:::benchmark_data_smithi194_121512_object35179:head# 2024-04-29T14:26:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bba451:::benchmark_data_smithi194_121512_object78782:head# 2024-04-29T14:26:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bbca23:::benchmark_data_smithi194_121512_object84368:head# 2024-04-29T14:26:36.952 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:36.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bc525d:::benchmark_data_smithi194_121512_object79988:head# 2024-04-29T14:26:36.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bc6de5:::benchmark_data_smithi194_121512_object34299:head# 2024-04-29T14:26:36.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bd426c:::benchmark_data_smithi194_121512_object95776:head# 2024-04-29T14:26:36.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bd4f0a:::benchmark_data_smithi194_121512_object7227:head# 2024-04-29T14:26:36.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bdd3e1:::benchmark_data_smithi194_121512_object72064:head# 2024-04-29T14:26:36.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bed5c2:::benchmark_data_smithi194_121512_object101554:head# 2024-04-29T14:26:36.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65bf161d:::benchmark_data_smithi194_121512_object115533:head# 2024-04-29T14:26:36.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c00e38:::benchmark_data_smithi194_121512_object7414:head# 2024-04-29T14:26:36.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c02a39:::benchmark_data_smithi194_121512_object36556:head# 2024-04-29T14:26:36.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c0beae:::benchmark_data_smithi194_121512_object63154:head# 2024-04-29T14:26:36.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c33702:::benchmark_data_smithi194_121512_object45680:head# 2024-04-29T14:26:36.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c37bd0:::benchmark_data_smithi194_121512_object11918:head# 2024-04-29T14:26:36.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c39a81:::benchmark_data_smithi194_121512_object80828:head# 2024-04-29T14:26:36.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c3ea2f:::benchmark_data_smithi194_121512_object43750:head# 2024-04-29T14:26:36.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c63c6e:::benchmark_data_smithi194_121512_object33039:head# 2024-04-29T14:26:36.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c6443f:::benchmark_data_smithi194_121512_object27790:head# 2024-04-29T14:26:36.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c78184:::benchmark_data_smithi194_121512_object108241:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c7ea3b:::benchmark_data_smithi194_121512_object10896:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c7f52a:::benchmark_data_smithi194_121512_object22135:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c88282:::benchmark_data_smithi194_121512_object20556:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65c9fcd3:::benchmark_data_smithi194_121512_object9055:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ca11bf:::benchmark_data_smithi194_121512_object27140:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ca879c:::benchmark_data_smithi194_121512_object112090:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65caaf7a:::benchmark_data_smithi194_121512_object119952:head# 2024-04-29T14:26:36.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cb0d60:::benchmark_data_smithi194_121512_object24867:head# 2024-04-29T14:26:36.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ccb619:::benchmark_data_smithi194_121512_object99418:head# 2024-04-29T14:26:36.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ccd1d7:::benchmark_data_smithi194_121512_object129381:head# 2024-04-29T14:26:36.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cd05ec:::benchmark_data_smithi194_121512_object14403:head# 2024-04-29T14:26:36.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cd3a52:::benchmark_data_smithi194_121512_object106746:head# 2024-04-29T14:26:36.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cd4595:::benchmark_data_smithi194_121512_object89597:head# 2024-04-29T14:26:36.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cd5ca8:::benchmark_data_smithi194_121512_object69811:head# 2024-04-29T14:26:36.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cd81b3:::benchmark_data_smithi194_121512_object55421:head# 2024-04-29T14:26:36.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ce98c3:::benchmark_data_smithi194_121512_object93030:head# 2024-04-29T14:26:36.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ceaa88:::benchmark_data_smithi194_121512_object34610:head# 2024-04-29T14:26:36.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cf31e4:::benchmark_data_smithi194_121512_object21824:head# 2024-04-29T14:26:36.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cf3846:::benchmark_data_smithi194_121512_object14026:head# 2024-04-29T14:26:36.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cf5791:::benchmark_data_smithi194_121512_object97777:head# 2024-04-29T14:26:36.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cf68d0:::benchmark_data_smithi194_121512_object61123:head# 2024-04-29T14:26:36.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65cfb12b:::benchmark_data_smithi194_121512_object62081:head# 2024-04-29T14:26:36.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d0a575:::benchmark_data_smithi194_121512_object45776:head# 2024-04-29T14:26:36.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d0ca6a:::benchmark_data_smithi194_121512_object26705:head# 2024-04-29T14:26:36.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d1e851:::benchmark_data_smithi194_121512_object32632:head# 2024-04-29T14:26:36.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d22f02:::benchmark_data_smithi194_121512_object48285:head# 2024-04-29T14:26:36.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d45854:::benchmark_data_smithi194_121512_object74877:head# 2024-04-29T14:26:36.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d52a61:::benchmark_data_smithi194_121512_object95717:head# 2024-04-29T14:26:36.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d5e902:::benchmark_data_smithi194_121512_object93057:head# 2024-04-29T14:26:36.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d72a05:::benchmark_data_smithi194_121512_object56787:head# 2024-04-29T14:26:36.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d7472c:::benchmark_data_smithi194_121512_object95336:head# 2024-04-29T14:26:36.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d7edaa:::benchmark_data_smithi194_121512_object129715:head# 2024-04-29T14:26:36.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d8fdfe:::benchmark_data_smithi194_121512_object8279:head# 2024-04-29T14:26:36.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d97df8:::benchmark_data_smithi194_121512_object50896:head# 2024-04-29T14:26:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d99776:::benchmark_data_smithi194_121512_object73820:head# 2024-04-29T14:26:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65d9b27f:::benchmark_data_smithi194_121512_object111534:head# 2024-04-29T14:26:36.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65daa285:::benchmark_data_smithi194_121512_object83013:head# 2024-04-29T14:26:36.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65dabed7:::benchmark_data_smithi194_121512_object12234:head# 2024-04-29T14:26:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65daf3aa:::benchmark_data_smithi194_121512_object108080:head# 2024-04-29T14:26:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65db5956:::benchmark_data_smithi194_121512_object119924:head# 2024-04-29T14:26:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65db5ba8:::benchmark_data_smithi194_121512_object47161:head# 2024-04-29T14:26:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65dbac1c:::benchmark_data_smithi194_121512_object25101:head# 2024-04-29T14:26:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65dc0ffe:::benchmark_data_smithi194_121512_object58114:head# 2024-04-29T14:26:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65dc732d:::benchmark_data_smithi194_121512_object130845:head# 2024-04-29T14:26:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65dddb34:::benchmark_data_smithi194_121512_object129565:head# 2024-04-29T14:26:36.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65dde095:::benchmark_data_smithi194_121512_object77428:head# 2024-04-29T14:26:36.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65de64d2:::benchmark_data_smithi194_121512_object114659:head# 2024-04-29T14:26:36.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e053a4:::benchmark_data_smithi194_121512_object64137:head# 2024-04-29T14:26:36.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e05689:::benchmark_data_smithi194_121512_object62318:head# 2024-04-29T14:26:36.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e1918f:::benchmark_data_smithi194_121512_object93326:head# 2024-04-29T14:26:36.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e1cd24:::benchmark_data_smithi194_121512_object12374:head# 2024-04-29T14:26:36.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e24785:::benchmark_data_smithi194_121512_object80702:head# 2024-04-29T14:26:36.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e2b630:::benchmark_data_smithi194_121512_object88061:head# 2024-04-29T14:26:36.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e2f4f4:::benchmark_data_smithi194_121512_object100178:head# 2024-04-29T14:26:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e32c63:::benchmark_data_smithi194_121512_object127835:head# 2024-04-29T14:26:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e35ff1:::benchmark_data_smithi194_121512_object56336:head# 2024-04-29T14:26:36.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e36810:::benchmark_data_smithi194_121512_object107739:head# 2024-04-29T14:26:36.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e47781:::benchmark_data_smithi194_121512_object60705:head# 2024-04-29T14:26:36.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e4ae28:::benchmark_data_smithi194_121512_object130801:head# 2024-04-29T14:26:36.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e4b2ef:::benchmark_data_smithi194_121512_object89233:head# 2024-04-29T14:26:36.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e4c4ef:::benchmark_data_smithi194_121512_object73684:head# 2024-04-29T14:26:36.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e4ccdd:::benchmark_data_smithi194_121512_object131762:head# 2024-04-29T14:26:36.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e53e02:::benchmark_data_smithi194_121512_object124618:head# 2024-04-29T14:26:36.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e5534e:::benchmark_data_smithi194_121512_object26416:head# 2024-04-29T14:26:36.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e58eec:::benchmark_data_smithi194_121512_object94264:head# 2024-04-29T14:26:36.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e73d2a:::benchmark_data_smithi194_121512_object67466:head# 2024-04-29T14:26:36.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e76e35:::benchmark_data_smithi194_121512_object94646:head# 2024-04-29T14:26:36.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e87731:::benchmark_data_smithi194_121512_object77331:head# 2024-04-29T14:26:36.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e88f1c:::benchmark_data_smithi194_121512_object25854:head# 2024-04-29T14:26:36.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e8b98b:::benchmark_data_smithi194_121512_object133326:head# 2024-04-29T14:26:36.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e8e39c:::benchmark_data_smithi194_121512_object13036:head# 2024-04-29T14:26:36.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e949ed:::benchmark_data_smithi194_121512_object50859:head# 2024-04-29T14:26:37.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e96b39:::benchmark_data_smithi194_121512_object8558:head# 2024-04-29T14:26:37.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65e9c52c:::benchmark_data_smithi194_121512_object9480:head# 2024-04-29T14:26:37.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ea076a:::benchmark_data_smithi194_121512_object86966:head# 2024-04-29T14:26:37.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65eb5c5c:::benchmark_data_smithi194_121512_object48488:head# 2024-04-29T14:26:37.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65eb6629:::benchmark_data_smithi194_121512_object46635:head# 2024-04-29T14:26:37.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ebcee0:::benchmark_data_smithi194_121512_object6929:head# 2024-04-29T14:26:37.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ec0b9a:::benchmark_data_smithi194_121512_object114979:head# 2024-04-29T14:26:37.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ec79ae:::benchmark_data_smithi194_121512_object95227:head# 2024-04-29T14:26:37.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ecb919:::benchmark_data_smithi194_121512_object83751:head# 2024-04-29T14:26:37.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ed762c:::benchmark_data_smithi194_121512_object22026:head# 2024-04-29T14:26:37.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ed7942:::benchmark_data_smithi194_121512_object18597:head# 2024-04-29T14:26:37.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ee11c1:::benchmark_data_smithi194_121512_object50103:head# 2024-04-29T14:26:37.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ee906f:::benchmark_data_smithi194_121512_object10030:head# 2024-04-29T14:26:37.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65eefed3:::benchmark_data_smithi194_121512_object124636:head# 2024-04-29T14:26:37.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ef850e:::benchmark_data_smithi194_121512_object14188:head# 2024-04-29T14:26:37.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65efb9bf:::benchmark_data_smithi194_121512_object54929:head# 2024-04-29T14:26:37.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f008df:::benchmark_data_smithi194_121512_object85414:head# 2024-04-29T14:26:37.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f043c7:::benchmark_data_smithi194_121512_object24034:head# 2024-04-29T14:26:37.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f0f42a:::benchmark_data_smithi194_121512_object131191:head# 2024-04-29T14:26:37.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f1c4c2:::benchmark_data_smithi194_121512_object38792:head# 2024-04-29T14:26:37.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f33eec:::benchmark_data_smithi194_121512_object27664:head# 2024-04-29T14:26:37.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f3ae4d:::benchmark_data_smithi194_121512_object112129:head# 2024-04-29T14:26:37.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f3b442:::benchmark_data_smithi194_121512_object88853:head# 2024-04-29T14:26:37.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f40861:::benchmark_data_smithi194_121512_object48279:head# 2024-04-29T14:26:37.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f4b7f6:::benchmark_data_smithi194_121512_object46198:head# 2024-04-29T14:26:37.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f50d9c:::benchmark_data_smithi194_121512_object43344:head# 2024-04-29T14:26:37.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f5f8cc:::benchmark_data_smithi194_121512_object86595:head# 2024-04-29T14:26:37.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f666f1:::benchmark_data_smithi194_121512_object19386:head# 2024-04-29T14:26:37.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f7d6b7:::benchmark_data_smithi194_121512_object33959:head# 2024-04-29T14:26:37.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f821fd:::benchmark_data_smithi194_121512_object66144:head# 2024-04-29T14:26:37.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f84e8b:::benchmark_data_smithi194_121512_object35192:head# 2024-04-29T14:26:37.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65f89df8:::benchmark_data_smithi194_121512_object133248:head# 2024-04-29T14:26:37.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fa16ff:::benchmark_data_smithi194_121512_object122801:head# 2024-04-29T14:26:37.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fabc95:::benchmark_data_smithi194_121512_object34625:head# 2024-04-29T14:26:37.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fb1629:::benchmark_data_smithi194_121512_object85698:head# 2024-04-29T14:26:37.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fb94a8:::benchmark_data_smithi194_121512_object7360:head# 2024-04-29T14:26:37.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fc8d42:::benchmark_data_smithi194_121512_object71696:head# 2024-04-29T14:26:37.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fd5f50:::benchmark_data_smithi194_121512_object55218:head# 2024-04-29T14:26:37.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fe294e:::benchmark_data_smithi194_121512_object97443:head# 2024-04-29T14:26:37.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fe923e:::benchmark_data_smithi194_121512_object32984:head# 2024-04-29T14:26:37.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ff0992:::benchmark_data_smithi194_121512_object112843:head# 2024-04-29T14:26:37.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65ffef52:::benchmark_data_smithi194_121512_object124391:head# 2024-04-29T14:26:37.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:65fffb7e:::benchmark_data_smithi194_121512_object65401:head# 2024-04-29T14:26:37.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66004278:::benchmark_data_smithi194_121512_object24233:head# 2024-04-29T14:26:37.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66005234:::benchmark_data_smithi194_121512_object95409:head# 2024-04-29T14:26:37.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66011701:::benchmark_data_smithi194_121512_object58138:head# 2024-04-29T14:26:37.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6601cffb:::benchmark_data_smithi194_121512_object125844:head# 2024-04-29T14:26:37.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6601f0c9:::benchmark_data_smithi194_121512_object95059:head# 2024-04-29T14:26:37.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660216b8:::benchmark_data_smithi194_121512_object44123:head# 2024-04-29T14:26:37.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6602fa6e:::benchmark_data_smithi194_121512_object125158:head# 2024-04-29T14:26:37.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660391ec:::benchmark_data_smithi194_121512_object94715:head# 2024-04-29T14:26:37.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66046d99:::benchmark_data_smithi194_121512_object31847:head# 2024-04-29T14:26:37.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6604f73a:::benchmark_data_smithi194_121512_object128503:head# 2024-04-29T14:26:37.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6605f629:::benchmark_data_smithi194_121512_object129562:head# 2024-04-29T14:26:37.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6606ef2f:::benchmark_data_smithi194_121512_object124968:head# 2024-04-29T14:26:37.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66073e65:::benchmark_data_smithi194_121512_object131564:head# 2024-04-29T14:26:37.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660799ef:::benchmark_data_smithi194_121512_object82445:head# 2024-04-29T14:26:37.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6607d93a:::benchmark_data_smithi194_121512_object47830:head# 2024-04-29T14:26:37.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660803d5:::benchmark_data_smithi194_121512_object40939:head# 2024-04-29T14:26:37.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66081873:::benchmark_data_smithi194_121512_object35052:head# 2024-04-29T14:26:37.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6608cfd1:::benchmark_data_smithi194_121512_object70422:head# 2024-04-29T14:26:37.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6608e382:::benchmark_data_smithi194_121512_object92723:head# 2024-04-29T14:26:37.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6608ed32:::benchmark_data_smithi194_121512_object81966:head# 2024-04-29T14:26:37.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6608f0ac:::benchmark_data_smithi194_121512_object64165:head# 2024-04-29T14:26:37.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660995c6:::benchmark_data_smithi194_121512_object123054:head# 2024-04-29T14:26:37.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660996a8:::benchmark_data_smithi194_121512_object71764:head# 2024-04-29T14:26:37.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660ab413:::benchmark_data_smithi194_121512_object14832:head# 2024-04-29T14:26:37.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660b3c18:::benchmark_data_smithi194_121512_object119034:head# 2024-04-29T14:26:37.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660b775b:::benchmark_data_smithi194_121512_object38982:head# 2024-04-29T14:26:37.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660c121e:::benchmark_data_smithi194_121512_object84486:head# 2024-04-29T14:26:37.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660c21f9:::benchmark_data_smithi194_121512_object30283:head# 2024-04-29T14:26:37.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660c22e1:::benchmark_data_smithi194_121512_object101415:head# 2024-04-29T14:26:37.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660c48be:::benchmark_data_smithi194_121512_object83586:head# 2024-04-29T14:26:37.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660c9045:::benchmark_data_smithi194_121512_object45862:head# 2024-04-29T14:26:37.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660d15c5:::benchmark_data_smithi194_121512_object18713:head# 2024-04-29T14:26:37.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660de4ca:::benchmark_data_smithi194_121512_object116994:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660edbee:::benchmark_data_smithi194_121512_object100425:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:660f9f7e:::benchmark_data_smithi194_121512_object103027:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66100b9e:::benchmark_data_smithi194_121512_object55524:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6611688a:::benchmark_data_smithi194_121512_object130426:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6611835a:::benchmark_data_smithi194_121512_object21719:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6611cd9b:::benchmark_data_smithi194_121512_object131724:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6611f3d4:::benchmark_data_smithi194_121512_object87218:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6612e537:::benchmark_data_smithi194_121512_object45761:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6614e786:::benchmark_data_smithi194_121512_object42632:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6614f192:::benchmark_data_smithi194_121512_object110669:head# 2024-04-29T14:26:37.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66152641:::benchmark_data_smithi194_121512_object115872:head# 2024-04-29T14:26:37.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66164689:::benchmark_data_smithi194_121512_object3230:head# 2024-04-29T14:26:37.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66170a75:::benchmark_data_smithi194_121512_object114455:head# 2024-04-29T14:26:37.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661740c1:::benchmark_data_smithi194_121512_object117271:head# 2024-04-29T14:26:37.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66179036:::benchmark_data_smithi194_121512_object121097:head# 2024-04-29T14:26:37.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66183b34:::benchmark_data_smithi194_121512_object23859:head# 2024-04-29T14:26:37.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6618c372:::benchmark_data_smithi194_121512_object45590:head# 2024-04-29T14:26:37.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6619eaa2:::benchmark_data_smithi194_121512_object97964:head# 2024-04-29T14:26:37.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661a1bcf:::benchmark_data_smithi194_121512_object23743:head# 2024-04-29T14:26:37.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661a7173:::benchmark_data_smithi194_121512_object108098:head# 2024-04-29T14:26:37.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661b38cc:::benchmark_data_smithi194_121512_object32305:head# 2024-04-29T14:26:37.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661b683b:::benchmark_data_smithi194_121512_object65007:head# 2024-04-29T14:26:37.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661be344:::benchmark_data_smithi194_121512_object81139:head# 2024-04-29T14:26:37.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661c1650:::benchmark_data_smithi194_121512_object79751:head# 2024-04-29T14:26:37.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661cde5b:::benchmark_data_smithi194_121512_object28767:head# 2024-04-29T14:26:37.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661d5327:::benchmark_data_smithi194_121512_object102986:head# 2024-04-29T14:26:37.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661efd17:::benchmark_data_smithi194_121512_object122802:head# 2024-04-29T14:26:37.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661fbfd9:::benchmark_data_smithi194_121512_object123345:head# 2024-04-29T14:26:37.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:661ff19d:::benchmark_data_smithi194_121512_object125255:head# 2024-04-29T14:26:37.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662028f8:::benchmark_data_smithi194_121512_object128859:head# 2024-04-29T14:26:37.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66203829:::benchmark_data_smithi194_121512_object39711:head# 2024-04-29T14:26:37.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6620d27d:::benchmark_data_smithi194_121512_object43386:head# 2024-04-29T14:26:37.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6620fa1c:::benchmark_data_smithi194_121512_object74199:head# 2024-04-29T14:26:37.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6621481f:::benchmark_data_smithi194_121512_object105125:head# 2024-04-29T14:26:37.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662152ad:::benchmark_data_smithi194_121512_object19604:head# 2024-04-29T14:26:37.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66221fe1:::benchmark_data_smithi194_121512_object25698:head# 2024-04-29T14:26:37.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6622609b:::benchmark_data_smithi194_121512_object116078:head# 2024-04-29T14:26:37.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662319ef:::benchmark_data_smithi194_121512_object80474:head# 2024-04-29T14:26:37.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6625dcef:::benchmark_data_smithi194_121512_object113168:head# 2024-04-29T14:26:37.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6626a72c:::benchmark_data_smithi194_121512_object61416:head# 2024-04-29T14:26:37.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6626bc9a:::benchmark_data_smithi194_121512_object72240:head# 2024-04-29T14:26:37.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66270e2f:::benchmark_data_smithi194_121512_object69368:head# 2024-04-29T14:26:37.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66277e62:::benchmark_data_smithi194_121512_object16149:head# 2024-04-29T14:26:37.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66281702:::benchmark_data_smithi194_121512_object37232:head# 2024-04-29T14:26:37.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66284b82:::benchmark_data_smithi194_121512_object50743:head# 2024-04-29T14:26:37.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6628a7fc:::benchmark_data_smithi194_121512_object112596:head# 2024-04-29T14:26:37.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6628f71a:::benchmark_data_smithi194_121512_object59892:head# 2024-04-29T14:26:37.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662aed06:::benchmark_data_smithi194_121512_object64143:head# 2024-04-29T14:26:37.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662afe68:::benchmark_data_smithi194_121512_object116526:head# 2024-04-29T14:26:37.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662b49f2:::benchmark_data_smithi194_121512_object37088:head# 2024-04-29T14:26:37.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662bc5e4:::benchmark_data_smithi194_121512_object92349:head# 2024-04-29T14:26:37.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662c7275:::benchmark_data_smithi194_121512_object61641:head# 2024-04-29T14:26:37.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662cbf6f:::benchmark_data_smithi194_121512_object102844:head# 2024-04-29T14:26:37.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662cc8be:::benchmark_data_smithi194_121512_object124121:head# 2024-04-29T14:26:37.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662de40b:::benchmark_data_smithi194_121512_object50016:head# 2024-04-29T14:26:37.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662e0c58:::benchmark_data_smithi194_121512_object73237:head# 2024-04-29T14:26:37.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662f294a:::benchmark_data_smithi194_121512_object36565:head# 2024-04-29T14:26:37.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662fb3a5:::benchmark_data_smithi194_121512_object61362:head# 2024-04-29T14:26:37.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:662fd33e:::benchmark_data_smithi194_121512_object107480:head# 2024-04-29T14:26:37.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66309dde:::benchmark_data_smithi194_121512_object76836:head# 2024-04-29T14:26:37.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66331ce1:::benchmark_data_smithi194_121512_object45397:head# 2024-04-29T14:26:37.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663343cb:::benchmark_data_smithi194_121512_object64717:head# 2024-04-29T14:26:37.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663527d5:::benchmark_data_smithi194_121512_object110803:head# 2024-04-29T14:26:37.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6635e017:::benchmark_data_smithi194_121512_object5112:head# 2024-04-29T14:26:37.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6636384d:::benchmark_data_smithi194_121512_object98013:head# 2024-04-29T14:26:37.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663652ef:::benchmark_data_smithi194_121512_object25504:head# 2024-04-29T14:26:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6636dca2:::benchmark_data_smithi194_121512_object97682:head# 2024-04-29T14:26:37.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66377f42:::benchmark_data_smithi194_121512_object122203:head# 2024-04-29T14:26:37.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6637826a:::benchmark_data_smithi194_121512_object123238:head# 2024-04-29T14:26:37.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6638da1f:::benchmark_data_smithi194_121512_object22539:head# 2024-04-29T14:26:37.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663901ea:::benchmark_data_smithi194_121512_object108628:head# 2024-04-29T14:26:37.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663a6cb3:::benchmark_data_smithi194_121512_object1939:head# 2024-04-29T14:26:37.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663ab0e1:::benchmark_data_smithi194_121512_object79580:head# 2024-04-29T14:26:37.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663ad323:::benchmark_data_smithi194_121512_object63400:head# 2024-04-29T14:26:37.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663ae05f:::benchmark_data_smithi194_121512_object64104:head# 2024-04-29T14:26:37.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663ae74e:::benchmark_data_smithi194_121512_object17966:head# 2024-04-29T14:26:37.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663bbb2a:::benchmark_data_smithi194_121512_object3234:head# 2024-04-29T14:26:37.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663c6025:::benchmark_data_smithi194_121512_object78954:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663cc1aa:::benchmark_data_smithi194_121512_object129088:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663e09bc:::benchmark_data_smithi194_121512_object48178:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663e5b38:::benchmark_data_smithi194_121512_object48338:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663e6799:::benchmark_data_smithi194_121512_object30346:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663f2168:::benchmark_data_smithi194_121512_object107323:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663f31b3:::benchmark_data_smithi194_121512_object119066:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663f3384:::benchmark_data_smithi194_121512_object52897:head# 2024-04-29T14:26:37.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663f6ce4:::benchmark_data_smithi194_121512_object131252:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:663f8d6f:::benchmark_data_smithi194_121512_object28325:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6640271e:::benchmark_data_smithi194_121512_object117750:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66405bb8:::benchmark_data_smithi194_121512_object79700:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66409054:::benchmark_data_smithi194_121512_object42016:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6640959b:::benchmark_data_smithi194_121512_object77877:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664127f3:::benchmark_data_smithi194_121512_object113765:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66419182:::benchmark_data_smithi194_121512_object5999:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6641f96b:::benchmark_data_smithi194_121512_object35396:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66424af3:::benchmark_data_smithi194_121512_object35443:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664306b8:::benchmark_data_smithi194_121512_object54627:head# 2024-04-29T14:26:37.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664590a2:::benchmark_data_smithi194_121512_object53445:head# 2024-04-29T14:26:37.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6646975f:::benchmark_data_smithi194_121512_object30325:head# 2024-04-29T14:26:37.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66489e9f:::benchmark_data_smithi194_121512_object118170:head# 2024-04-29T14:26:37.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664973ee:::benchmark_data_smithi194_121512_object93354:head# 2024-04-29T14:26:37.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6649aa1f:::benchmark_data_smithi194_121512_object19624:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664ac707:::benchmark_data_smithi194_121512_object49482:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664b5250:::benchmark_data_smithi194_121512_object33413:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664bfc25:::benchmark_data_smithi194_121512_object127412:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664c1e9f:::benchmark_data_smithi194_121512_object44140:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664c860c:::benchmark_data_smithi194_121512_object6823:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664cbe31:::benchmark_data_smithi194_121512_object63812:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664cf879:::benchmark_data_smithi194_121512_object19925:head# 2024-04-29T14:26:37.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664cfc79:::benchmark_data_smithi194_121512_object132154:head# 2024-04-29T14:26:37.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664d33d9:::benchmark_data_smithi194_121512_object67115:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664d5bfe:::benchmark_data_smithi194_121512_object31959:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664d7184:::benchmark_data_smithi194_121512_object5306:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664e52f4:::benchmark_data_smithi194_121512_object18406:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:664fd313:::benchmark_data_smithi194_121512_object132379:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66515ea6:::benchmark_data_smithi194_121512_object108115:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66515ea9:::benchmark_data_smithi194_121512_object132763:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6651b202:::benchmark_data_smithi194_121512_object107212:head# 2024-04-29T14:26:37.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6651d8aa:::benchmark_data_smithi194_121512_object26823:head# 2024-04-29T14:26:37.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66543d35:::benchmark_data_smithi194_121512_object28081:head# 2024-04-29T14:26:37.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6655dc1e:::benchmark_data_smithi194_121512_object50088:head# 2024-04-29T14:26:37.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66566f24:::benchmark_data_smithi194_121512_object25886:head# 2024-04-29T14:26:37.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665696cd:::benchmark_data_smithi194_121512_object51683:head# 2024-04-29T14:26:37.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6656a30e:::benchmark_data_smithi194_121512_object105840:head# 2024-04-29T14:26:37.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66575227:::benchmark_data_smithi194_121512_object119406:head# 2024-04-29T14:26:37.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66580de1:::benchmark_data_smithi194_121512_object95501:head# 2024-04-29T14:26:37.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665815c0:::benchmark_data_smithi194_121512_object29988:head# 2024-04-29T14:26:37.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665909e4:::benchmark_data_smithi194_121512_object26529:head# 2024-04-29T14:26:37.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66592490:::benchmark_data_smithi194_121512_object128400:head# 2024-04-29T14:26:37.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6659bb11:::benchmark_data_smithi194_121512_object69589:head# 2024-04-29T14:26:37.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665a7625:::benchmark_data_smithi194_121512_object84934:head# 2024-04-29T14:26:37.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665a93f7:::benchmark_data_smithi194_121512_object6752:head# 2024-04-29T14:26:37.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665b1fba:::benchmark_data_smithi194_121512_object73885:head# 2024-04-29T14:26:37.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665b3216:::benchmark_data_smithi194_121512_object33384:head# 2024-04-29T14:26:37.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665b33c5:::benchmark_data_smithi194_121512_object27228:head# 2024-04-29T14:26:37.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665bb433:::benchmark_data_smithi194_121512_object4427:head# 2024-04-29T14:26:37.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665bdb3f:::benchmark_data_smithi194_121512_object70035:head# 2024-04-29T14:26:37.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665bfd8d:::benchmark_data_smithi194_121512_object107547:head# 2024-04-29T14:26:37.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665c870b:::benchmark_data_smithi194_121512_object86148:head# 2024-04-29T14:26:37.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665ce3f9:::benchmark_data_smithi194_121512_object4774:head# 2024-04-29T14:26:37.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665cf903:::benchmark_data_smithi194_121512_object48138:head# 2024-04-29T14:26:37.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665d2192:::benchmark_data_smithi194_121512_object63807:head# 2024-04-29T14:26:37.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665d521f:::benchmark_data_smithi194_121512_object90762:head# 2024-04-29T14:26:37.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665dff5c:::benchmark_data_smithi194_121512_object87772:head# 2024-04-29T14:26:37.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665e1963:::benchmark_data_smithi194_121512_object124827:head# 2024-04-29T14:26:37.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665e7a11:::benchmark_data_smithi194_121512_object97458:head# 2024-04-29T14:26:37.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:665edc96:::benchmark_data_smithi194_121512_object108316:head# 2024-04-29T14:26:37.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666148fc:::benchmark_data_smithi194_121512_object29973:head# 2024-04-29T14:26:37.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66614ff4:::benchmark_data_smithi194_121512_object22799:head# 2024-04-29T14:26:37.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6661799a:::benchmark_data_smithi194_121512_object13360:head# 2024-04-29T14:26:37.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6661d1bf:::benchmark_data_smithi194_121512_object29368:head# 2024-04-29T14:26:37.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666215f1:::benchmark_data_smithi194_121512_object13445:head# 2024-04-29T14:26:37.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6662b1b9:::benchmark_data_smithi194_121512_object127370:head# 2024-04-29T14:26:37.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66653a96:::benchmark_data_smithi194_121512_object14677:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66669582:::benchmark_data_smithi194_121512_object90499:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6666c064:::benchmark_data_smithi194_121512_object8341:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6666dbe5:::benchmark_data_smithi194_121512_object19356:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6666e677:::benchmark_data_smithi194_121512_object26832:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66678ba4:::benchmark_data_smithi194_121512_object125052:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6667a8b7:::benchmark_data_smithi194_121512_object757:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6667ec19:::benchmark_data_smithi194_121512_object1688:head# 2024-04-29T14:26:37.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666807e2:::benchmark_data_smithi194_121512_object271:head# 2024-04-29T14:26:37.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66687db8:::benchmark_data_smithi194_121512_object18781:head# 2024-04-29T14:26:37.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6668a350:::benchmark_data_smithi194_121512_object98603:head# 2024-04-29T14:26:37.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6668eb79:::benchmark_data_smithi194_121512_object76179:head# 2024-04-29T14:26:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666a0e0f:::benchmark_data_smithi194_121512_object24110:head# 2024-04-29T14:26:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666a2f9d:::benchmark_data_smithi194_121512_object48081:head# 2024-04-29T14:26:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666a44f1:::benchmark_data_smithi194_121512_object77965:head# 2024-04-29T14:26:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666b60ce:::benchmark_data_smithi194_121512_object31088:head# 2024-04-29T14:26:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666b6abc:::benchmark_data_smithi194_121512_object97274:head# 2024-04-29T14:26:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666c2d05:::benchmark_data_smithi194_121512_object101411:head# 2024-04-29T14:26:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666c3360:::benchmark_data_smithi194_121512_object46909:head# 2024-04-29T14:26:37.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666c97e6:::benchmark_data_smithi194_121512_object4196:head# 2024-04-29T14:26:37.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666c9b1f:::benchmark_data_smithi194_121512_object130391:head# 2024-04-29T14:26:37.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666cf06d:::benchmark_data_smithi194_121512_object120500:head# 2024-04-29T14:26:37.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666d2887:::benchmark_data_smithi194_121512_object60936:head# 2024-04-29T14:26:37.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666d7a5e:::benchmark_data_smithi194_121512_object81710:head# 2024-04-29T14:26:37.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666f41ef:::benchmark_data_smithi194_121512_object85768:head# 2024-04-29T14:26:37.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:666fe42a:::benchmark_data_smithi194_121512_object63064:head# 2024-04-29T14:26:37.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66704db3:::benchmark_data_smithi194_121512_object50127:head# 2024-04-29T14:26:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6670709f:::benchmark_data_smithi194_121512_object2073:head# 2024-04-29T14:26:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66709a3c:::benchmark_data_smithi194_121512_object61069:head# 2024-04-29T14:26:37.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66725813:::benchmark_data_smithi194_121512_object20448:head# 2024-04-29T14:26:37.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6672c8ec:::benchmark_data_smithi194_121512_object39623:head# 2024-04-29T14:26:37.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667513ab:::benchmark_data_smithi194_121512_object46790:head# 2024-04-29T14:26:37.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66751b2e:::benchmark_data_smithi194_121512_object108578:head# 2024-04-29T14:26:37.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667596b4:::benchmark_data_smithi194_121512_object20931:head# 2024-04-29T14:26:37.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6675fe44:::benchmark_data_smithi194_121512_object86259:head# 2024-04-29T14:26:37.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667620d7:::benchmark_data_smithi194_121512_object263:head# 2024-04-29T14:26:37.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66765798:::benchmark_data_smithi194_121512_object31006:head# 2024-04-29T14:26:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667699e0:::benchmark_data_smithi194_121512_object62105:head# 2024-04-29T14:26:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6677533a:::benchmark_data_smithi194_121512_object113229:head# 2024-04-29T14:26:37.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667759b0:::benchmark_data_smithi194_121512_object89290:head# 2024-04-29T14:26:37.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667893a3:::benchmark_data_smithi194_121512_object13546:head# 2024-04-29T14:26:37.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66789b1b:::benchmark_data_smithi194_121512_object77582:head# 2024-04-29T14:26:37.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6678c6f8:::benchmark_data_smithi194_121512_object79104:head# 2024-04-29T14:26:37.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667967ec:::benchmark_data_smithi194_121512_object31873:head# 2024-04-29T14:26:37.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6679e060:::benchmark_data_smithi194_121512_object66203:head# 2024-04-29T14:26:37.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667a26b5:::benchmark_data_smithi194_121512_object100460:head# 2024-04-29T14:26:37.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667abec1:::benchmark_data_smithi194_121512_object38772:head# 2024-04-29T14:26:37.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667b89f6:::benchmark_data_smithi194_121512_object85243:head# 2024-04-29T14:26:37.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667bd463:::benchmark_data_smithi194_121512_object83729:head# 2024-04-29T14:26:37.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667d2be9:::benchmark_data_smithi194_121512_object37576:head# 2024-04-29T14:26:37.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667eebd0:::benchmark_data_smithi194_121512_object132017:head# 2024-04-29T14:26:37.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:667ffff6:::benchmark_data_smithi194_121512_object18972:head# 2024-04-29T14:26:37.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66817ff7:::benchmark_data_smithi194_121512_object124449:head# 2024-04-29T14:26:37.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6681c380:::benchmark_data_smithi194_121512_object52182:head# 2024-04-29T14:26:37.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6681dac1:::benchmark_data_smithi194_121512_object114882:head# 2024-04-29T14:26:37.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6681f407:::benchmark_data_smithi194_121512_object130883:head# 2024-04-29T14:26:37.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6682320c:::benchmark_data_smithi194_121512_object96612:head# 2024-04-29T14:26:37.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66835ef1:::benchmark_data_smithi194_121512_object118224:head# 2024-04-29T14:26:37.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6683a2c3:::benchmark_data_smithi194_121512_object115256:head# 2024-04-29T14:26:37.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6683f94d:::benchmark_data_smithi194_121512_object88368:head# 2024-04-29T14:26:37.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6685334f:::benchmark_data_smithi194_121512_object82808:head# 2024-04-29T14:26:37.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6685801e:::benchmark_data_smithi194_121512_object71135:head# 2024-04-29T14:26:37.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668647fe:::benchmark_data_smithi194_121512_object111865:head# 2024-04-29T14:26:37.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668697be:::benchmark_data_smithi194_121512_object1080:head# 2024-04-29T14:26:37.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6686e960:::benchmark_data_smithi194_121512_object69372:head# 2024-04-29T14:26:37.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668704d0:::benchmark_data_smithi194_121512_object80336:head# 2024-04-29T14:26:37.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668756bf:::benchmark_data_smithi194_121512_object64762:head# 2024-04-29T14:26:37.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66878d30:::benchmark_data_smithi194_121512_object14388:head# 2024-04-29T14:26:37.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668818c9:::benchmark_data_smithi194_121512_object102402:head# 2024-04-29T14:26:37.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66891bca:::benchmark_data_smithi194_121512_object119256:head# 2024-04-29T14:26:37.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6689fded:::benchmark_data_smithi194_121512_object42268:head# 2024-04-29T14:26:37.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668a3e15:::benchmark_data_smithi194_121512_object100834:head# 2024-04-29T14:26:37.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668ae6f3:::benchmark_data_smithi194_121512_object52238:head# 2024-04-29T14:26:37.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668aeb57:::benchmark_data_smithi194_121512_object111663:head# 2024-04-29T14:26:37.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668b3d9d:::benchmark_data_smithi194_121512_object55930:head# 2024-04-29T14:26:37.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668b9c9a:::benchmark_data_smithi194_121512_object69025:head# 2024-04-29T14:26:37.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668bca0b:::benchmark_data_smithi194_121512_object5753:head# 2024-04-29T14:26:37.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668c02e4:::benchmark_data_smithi194_121512_object86941:head# 2024-04-29T14:26:37.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668c6f26:::benchmark_data_smithi194_121512_object45229:head# 2024-04-29T14:26:37.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668caaed:::benchmark_data_smithi194_121512_object100959:head# 2024-04-29T14:26:37.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668dd92b:::benchmark_data_smithi194_121512_object42405:head# 2024-04-29T14:26:37.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668df4a4:::benchmark_data_smithi194_121512_object69052:head# 2024-04-29T14:26:37.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668eb091:::benchmark_data_smithi194_121512_object102868:head# 2024-04-29T14:26:37.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668f046f:::benchmark_data_smithi194_121512_object36753:head# 2024-04-29T14:26:37.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668fdbaa:::benchmark_data_smithi194_121512_object51262:head# 2024-04-29T14:26:37.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:668ff146:::benchmark_data_smithi194_121512_object69049:head# 2024-04-29T14:26:37.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6690094a:::benchmark_data_smithi194_121512_object108563:head# 2024-04-29T14:26:37.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66905a91:::benchmark_data_smithi194_121512_object3342:head# 2024-04-29T14:26:37.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6690975e:::benchmark_data_smithi194_121512_object57568:head# 2024-04-29T14:26:37.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66927c70:::benchmark_data_smithi194_121512_object23307:head# 2024-04-29T14:26:37.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66937411:::benchmark_data_smithi194_121512_object116729:head# 2024-04-29T14:26:37.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6693d544:::benchmark_data_smithi194_121512_object21832:head# 2024-04-29T14:26:37.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66942503:::benchmark_data_smithi194_121512_object21279:head# 2024-04-29T14:26:37.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66955616:::benchmark_data_smithi194_121512_object43800:head# 2024-04-29T14:26:37.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6695ac0a:::benchmark_data_smithi194_121512_object63453:head# 2024-04-29T14:26:37.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6696146f:::benchmark_data_smithi194_121512_object112157:head# 2024-04-29T14:26:37.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669632c3:::benchmark_data_smithi194_121512_object38198:head# 2024-04-29T14:26:37.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6696734a:::benchmark_data_smithi194_121512_object119028:head# 2024-04-29T14:26:37.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6696d93b:::benchmark_data_smithi194_121512_object37203:head# 2024-04-29T14:26:37.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66970887:::benchmark_data_smithi194_121512_object20413:head# 2024-04-29T14:26:37.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6697c850:::benchmark_data_smithi194_121512_object79774:head# 2024-04-29T14:26:37.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6698c2da:::benchmark_data_smithi194_121512_object97001:head# 2024-04-29T14:26:37.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669976d8:::benchmark_data_smithi194_121512_object121269:head# 2024-04-29T14:26:37.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669afacd:::benchmark_data_smithi194_121512_object49217:head# 2024-04-29T14:26:37.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669b03e8:::benchmark_data_smithi194_121512_object15626:head# 2024-04-29T14:26:37.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669b5e66:::benchmark_data_smithi194_121512_object63826:head# 2024-04-29T14:26:37.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669c2e13:::benchmark_data_smithi194_121512_object3356:head# 2024-04-29T14:26:37.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669c7372:::benchmark_data_smithi194_121512_object119078:head# 2024-04-29T14:26:37.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669ce14d:::benchmark_data_smithi194_121512_object47986:head# 2024-04-29T14:26:37.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669dcde2:::benchmark_data_smithi194_121512_object100739:head# 2024-04-29T14:26:37.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669e0b2c:::benchmark_data_smithi194_121512_object10338:head# 2024-04-29T14:26:37.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669f1553:::benchmark_data_smithi194_121512_object83181:head# 2024-04-29T14:26:37.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669fb6b4:::benchmark_data_smithi194_121512_object63329:head# 2024-04-29T14:26:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:669fffdb:::benchmark_data_smithi194_121512_object6358:head# 2024-04-29T14:26:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a01c77:::benchmark_data_smithi194_121512_object68146:head# 2024-04-29T14:26:37.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a0b3b7:::benchmark_data_smithi194_121512_object52823:head# 2024-04-29T14:26:37.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a1b5e8:::benchmark_data_smithi194_121512_object118708:head# 2024-04-29T14:26:37.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a1c282:::benchmark_data_smithi194_121512_object41758:head# 2024-04-29T14:26:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a1ee68:::benchmark_data_smithi194_121512_object84557:head# 2024-04-29T14:26:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a505cd:::benchmark_data_smithi194_121512_object126193:head# 2024-04-29T14:26:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a604f8:::benchmark_data_smithi194_121512_object117933:head# 2024-04-29T14:26:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a68cbd:::benchmark_data_smithi194_121512_object60390:head# 2024-04-29T14:26:37.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a6e80a:::benchmark_data_smithi194_121512_object25446:head# 2024-04-29T14:26:37.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a79980:::benchmark_data_smithi194_121512_object41573:head# 2024-04-29T14:26:37.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a7cc9e:::benchmark_data_smithi194_121512_object83254:head# 2024-04-29T14:26:37.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a7ed47:::benchmark_data_smithi194_121512_object49564:head# 2024-04-29T14:26:37.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a8402a:::benchmark_data_smithi194_121512_object64792:head# 2024-04-29T14:26:37.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a8efea:::benchmark_data_smithi194_121512_object82264:head# 2024-04-29T14:26:37.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66a8f720:::benchmark_data_smithi194_121512_object6517:head# 2024-04-29T14:26:37.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ab74e2:::benchmark_data_smithi194_121512_object121953:head# 2024-04-29T14:26:37.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ab75ec:::benchmark_data_smithi194_121512_object83368:head# 2024-04-29T14:26:37.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66abc9c2:::benchmark_data_smithi194_121512_object118032:head# 2024-04-29T14:26:37.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ac5c1b:::benchmark_data_smithi194_121512_object20134:head# 2024-04-29T14:26:37.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ac9d75:::benchmark_data_smithi194_121512_object25394:head# 2024-04-29T14:26:37.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ace734:::benchmark_data_smithi194_121512_object47857:head# 2024-04-29T14:26:37.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ad1b27:::benchmark_data_smithi194_121512_object38257:head# 2024-04-29T14:26:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66adbd8b:::benchmark_data_smithi194_121512_object3254:head# 2024-04-29T14:26:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ade7b6:::benchmark_data_smithi194_121512_object18437:head# 2024-04-29T14:26:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ae21d5:::benchmark_data_smithi194_121512_object108331:head# 2024-04-29T14:26:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ae3972:::benchmark_data_smithi194_121512_object107488:head# 2024-04-29T14:26:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ae5aa4:::benchmark_data_smithi194_121512_object98729:head# 2024-04-29T14:26:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66af1805:::benchmark_data_smithi194_121512_object22332:head# 2024-04-29T14:26:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66af1c5e:::benchmark_data_smithi194_121512_object8226:head# 2024-04-29T14:26:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66afcd35:::benchmark_data_smithi194_121512_object30713:head# 2024-04-29T14:26:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66afd7dd:::benchmark_data_smithi194_121512_object79735:head# 2024-04-29T14:26:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b0d528:::benchmark_data_smithi194_121512_object89922:head# 2024-04-29T14:26:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b18dc4:::benchmark_data_smithi194_121512_object97271:head# 2024-04-29T14:26:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b27602:::benchmark_data_smithi194_121512_object17486:head# 2024-04-29T14:26:37.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b2c0a7:::benchmark_data_smithi194_121512_object32456:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b2e653:::benchmark_data_smithi194_121512_object13517:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b32e88:::benchmark_data_smithi194_121512_object58778:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b33419:::benchmark_data_smithi194_121512_object48422:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b35901:::benchmark_data_smithi194_121512_object64868:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b421fc:::benchmark_data_smithi194_121512_object120789:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b4596a:::benchmark_data_smithi194_121512_object33942:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b53299:::benchmark_data_smithi194_121512_object66401:head# 2024-04-29T14:26:37.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b569ef:::benchmark_data_smithi194_121512_object1486:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b58500:::benchmark_data_smithi194_121512_object19523:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b6c009:::benchmark_data_smithi194_121512_object48936:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b70657:::benchmark_data_smithi194_121512_object8102:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b7100b:::benchmark_data_smithi194_121512_object100689:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b72a60:::benchmark_data_smithi194_121512_object111174:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b72b6b:::benchmark_data_smithi194_121512_object49975:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b7bdd1:::benchmark_data_smithi194_121512_object77812:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b7e413:::benchmark_data_smithi194_121512_object79964:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b81548:::benchmark_data_smithi194_121512_object53356:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b841f1:::benchmark_data_smithi194_121512_object24490:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b8c1fa:::benchmark_data_smithi194_121512_object88679:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b8c688:::benchmark_data_smithi194_121512_object125810:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b95fa0:::benchmark_data_smithi194_121512_object8582:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66b97193:::benchmark_data_smithi194_121512_object101787:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bb9936:::benchmark_data_smithi194_121512_object65244:head# 2024-04-29T14:26:37.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bcc2b6:::benchmark_data_smithi194_121512_object14138:head# 2024-04-29T14:26:37.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bcfbb9:::benchmark_data_smithi194_121512_object13447:head# 2024-04-29T14:26:37.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bd6b07:::benchmark_data_smithi194_121512_object74428:head# 2024-04-29T14:26:37.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bde04d:::benchmark_data_smithi194_121512_object107010:head# 2024-04-29T14:26:37.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66be5201:::benchmark_data_smithi194_121512_object128872:head# 2024-04-29T14:26:37.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bec755:::benchmark_data_smithi194_121512_object34356:head# 2024-04-29T14:26:37.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bf1470:::benchmark_data_smithi194_121512_object56666:head# 2024-04-29T14:26:37.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bf45f2:::benchmark_data_smithi194_121512_object19086:head# 2024-04-29T14:26:37.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66bf8fe5:::benchmark_data_smithi194_121512_object65348:head# 2024-04-29T14:26:37.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c218da:::benchmark_data_smithi194_121512_object129494:head# 2024-04-29T14:26:37.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c256a7:::benchmark_data_smithi194_121512_object107710:head# 2024-04-29T14:26:37.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c2c822:::benchmark_data_smithi194_121512_object97466:head# 2024-04-29T14:26:37.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c3c1e5:::benchmark_data_smithi194_121512_object131562:head# 2024-04-29T14:26:37.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c40fce:::benchmark_data_smithi194_121512_object21170:head# 2024-04-29T14:26:37.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c50a61:::benchmark_data_smithi194_121512_object95818:head# 2024-04-29T14:26:37.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c5690d:::benchmark_data_smithi194_121512_object2686:head# 2024-04-29T14:26:37.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c58568:::benchmark_data_smithi194_121512_object99367:head# 2024-04-29T14:26:37.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c60756:::benchmark_data_smithi194_121512_object104520:head# 2024-04-29T14:26:37.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c6250a:::benchmark_data_smithi194_121512_object113898:head# 2024-04-29T14:26:37.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c7b6f6:::benchmark_data_smithi194_121512_object26365:head# 2024-04-29T14:26:37.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c7c3d8:::benchmark_data_smithi194_121512_object20409:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c8bf95:::benchmark_data_smithi194_121512_object26450:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c9a068:::benchmark_data_smithi194_121512_object103996:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66c9ea94:::benchmark_data_smithi194_121512_object30953:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cab4bb:::benchmark_data_smithi194_121512_object36885:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cacafa:::benchmark_data_smithi194_121512_object124346:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cb4201:::benchmark_data_smithi194_121512_object121678:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cc0850:::benchmark_data_smithi194_121512_object66438:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cc6b80:::benchmark_data_smithi194_121512_object132584:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cc8d30:::benchmark_data_smithi194_121512_object13098:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ccfdcf:::benchmark_data_smithi194_121512_object93340:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cd8dc6:::benchmark_data_smithi194_121512_object128697:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ce7f01:::benchmark_data_smithi194_121512_object111091:head# 2024-04-29T14:26:37.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cf08a7:::benchmark_data_smithi194_121512_object121289:head# 2024-04-29T14:26:37.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cf3620:::benchmark_data_smithi194_121512_object125946:head# 2024-04-29T14:26:37.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cf7861:::benchmark_data_smithi194_121512_object16913:head# 2024-04-29T14:26:37.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cf95c9:::benchmark_data_smithi194_121512_object117027:head# 2024-04-29T14:26:37.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66cf9b95:::benchmark_data_smithi194_121512_object99641:head# 2024-04-29T14:26:37.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d063d5:::benchmark_data_smithi194_121512_object65820:head# 2024-04-29T14:26:37.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d0c434:::benchmark_data_smithi194_121512_object25343:head# 2024-04-29T14:26:37.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d1a736:::benchmark_data_smithi194_121512_object127546:head# 2024-04-29T14:26:37.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d1f08c:::benchmark_data_smithi194_121512_object72603:head# 2024-04-29T14:26:37.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d3684b:::benchmark_data_smithi194_121512_object86697:head# 2024-04-29T14:26:37.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d3bb91:::benchmark_data_smithi194_121512_object122671:head# 2024-04-29T14:26:37.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d623f5:::benchmark_data_smithi194_121512_object77722:head# 2024-04-29T14:26:37.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d7bee5:::benchmark_data_smithi194_121512_object1061:head# 2024-04-29T14:26:37.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d87d48:::benchmark_data_smithi194_121512_object31980:head# 2024-04-29T14:26:37.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d8d827:::benchmark_data_smithi194_121512_object18432:head# 2024-04-29T14:26:37.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d8efd9:::benchmark_data_smithi194_121512_object17830:head# 2024-04-29T14:26:37.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d8fa04:::benchmark_data_smithi194_121512_object57307:head# 2024-04-29T14:26:37.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66d9f851:::benchmark_data_smithi194_121512_object98232:head# 2024-04-29T14:26:37.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66da0c86:::benchmark_data_smithi194_121512_object31093:head# 2024-04-29T14:26:37.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66da18a3:::benchmark_data_smithi194_121512_object44941:head# 2024-04-29T14:26:37.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66da3708:::benchmark_data_smithi194_121512_object24279:head# 2024-04-29T14:26:37.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66da72e8:::benchmark_data_smithi194_121512_object19062:head# 2024-04-29T14:26:37.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66daad6b:::benchmark_data_smithi194_121512_object79242:head# 2024-04-29T14:26:37.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66dad7cc:::benchmark_data_smithi194_121512_object97078:head# 2024-04-29T14:26:37.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66dae16b:::benchmark_data_smithi194_121512_object39465:head# 2024-04-29T14:26:37.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66db27e0:::benchmark_data_smithi194_121512_object47946:head# 2024-04-29T14:26:37.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66db45f8:::benchmark_data_smithi194_121512_object35312:head# 2024-04-29T14:26:37.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66db9925:::benchmark_data_smithi194_121512_object75747:head# 2024-04-29T14:26:37.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66dc7ee7:::benchmark_data_smithi194_121512_object74039:head# 2024-04-29T14:26:37.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66dce764:::benchmark_data_smithi194_121512_object59507:head# 2024-04-29T14:26:37.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66dd233b:::benchmark_data_smithi194_121512_object23511:head# 2024-04-29T14:26:37.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ddd0aa:::benchmark_data_smithi194_121512_object123619:head# 2024-04-29T14:26:37.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ddd4ef:::benchmark_data_smithi194_121512_object21691:head# 2024-04-29T14:26:37.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ddec82:::benchmark_data_smithi194_121512_object95744:head# 2024-04-29T14:26:37.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66de0861:::benchmark_data_smithi194_121512_object124143:head# 2024-04-29T14:26:37.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66df2790:::benchmark_data_smithi194_121512_object40990:head# 2024-04-29T14:26:37.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66dfadbf:::benchmark_data_smithi194_121512_object77751:head# 2024-04-29T14:26:37.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e00189:::benchmark_data_smithi194_121512_object59118:head# 2024-04-29T14:26:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e00f01:::benchmark_data_smithi194_121512_object102777:head# 2024-04-29T14:26:37.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e02d45:::benchmark_data_smithi194_121512_object70356:head# 2024-04-29T14:26:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e05613:::benchmark_data_smithi194_121512_object45447:head# 2024-04-29T14:26:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e095c0:::benchmark_data_smithi194_121512_object31777:head# 2024-04-29T14:26:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e10601:::benchmark_data_smithi194_121512_object82614:head# 2024-04-29T14:26:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e16b64:::benchmark_data_smithi194_121512_object89157:head# 2024-04-29T14:26:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e1deb8:::benchmark_data_smithi194_121512_object25177:head# 2024-04-29T14:26:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e34a21:::benchmark_data_smithi194_121512_object63259:head# 2024-04-29T14:26:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e35f1d:::benchmark_data_smithi194_121512_object120663:head# 2024-04-29T14:26:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e6356b:::benchmark_data_smithi194_121512_object115717:head# 2024-04-29T14:26:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e64990:::benchmark_data_smithi194_121512_object23860:head# 2024-04-29T14:26:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66e8f0ce:::benchmark_data_smithi194_121512_object23980:head# 2024-04-29T14:26:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ea3c80:::benchmark_data_smithi194_121512_object131821:head# 2024-04-29T14:26:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66eb5cb0:::benchmark_data_smithi194_121512_object93001:head# 2024-04-29T14:26:37.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66eb86c1:::benchmark_data_smithi194_121512_object124748:head# 2024-04-29T14:26:37.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66eba125:::benchmark_data_smithi194_121512_object25143:head# 2024-04-29T14:26:37.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ec0b7e:::benchmark_data_smithi194_121512_object80258:head# 2024-04-29T14:26:37.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ed12a9:::benchmark_data_smithi194_121512_object28771:head# 2024-04-29T14:26:37.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66ed48a3:::benchmark_data_smithi194_121512_object101217:head# 2024-04-29T14:26:37.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66eddb52:::benchmark_data_smithi194_121512_object61119:head# 2024-04-29T14:26:37.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66eea894:::benchmark_data_smithi194_121512_object106180:head# 2024-04-29T14:26:37.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66eec5ab:::benchmark_data_smithi194_121512_object34791:head# 2024-04-29T14:26:37.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66eefd2c:::benchmark_data_smithi194_121512_object76503:head# 2024-04-29T14:26:37.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66efcb12:::benchmark_data_smithi194_121512_object81961:head# 2024-04-29T14:26:37.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f0b47b:::benchmark_data_smithi194_121512_object69387:head# 2024-04-29T14:26:37.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f1ab3e:::benchmark_data_smithi194_121512_object61917:head# 2024-04-29T14:26:37.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f2001d:::benchmark_data_smithi194_121512_object120435:head# 2024-04-29T14:26:37.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f2de0d:::benchmark_data_smithi194_121512_object2336:head# 2024-04-29T14:26:37.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f2eda1:::benchmark_data_smithi194_121512_object60087:head# 2024-04-29T14:26:37.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f354f5:::benchmark_data_smithi194_121512_object94983:head# 2024-04-29T14:26:37.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f3a6a0:::benchmark_data_smithi194_121512_object63215:head# 2024-04-29T14:26:37.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f3a85c:::benchmark_data_smithi194_121512_object114258:head# 2024-04-29T14:26:37.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f3b1c1:::benchmark_data_smithi194_121512_object84248:head# 2024-04-29T14:26:37.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f43a07:::benchmark_data_smithi194_121512_object108613:head# 2024-04-29T14:26:37.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f4c572:::benchmark_data_smithi194_121512_object12991:head# 2024-04-29T14:26:37.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f53678:::benchmark_data_smithi194_121512_object42973:head# 2024-04-29T14:26:37.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f5a71b:::benchmark_data_smithi194_121512_object72727:head# 2024-04-29T14:26:37.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f5fe66:::benchmark_data_smithi194_121512_object120840:head# 2024-04-29T14:26:37.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f62b91:::benchmark_data_smithi194_121512_object126524:head# 2024-04-29T14:26:37.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f64ee0:::benchmark_data_smithi194_121512_object59587:head# 2024-04-29T14:26:37.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f7015a:::benchmark_data_smithi194_121512_object78314:head# 2024-04-29T14:26:37.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f7410d:::benchmark_data_smithi194_121512_object80371:head# 2024-04-29T14:26:37.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f7cc82:::benchmark_data_smithi194_121512_object7505:head# 2024-04-29T14:26:37.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f97ce4:::benchmark_data_smithi194_121512_object41716:head# 2024-04-29T14:26:37.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66f9f40b:::benchmark_data_smithi194_121512_object63922:head# 2024-04-29T14:26:37.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fb1945:::benchmark_data_smithi194_121512_object124232:head# 2024-04-29T14:26:37.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fbaf4d:::benchmark_data_smithi194_121512_object82737:head# 2024-04-29T14:26:37.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fbc0ef:::benchmark_data_smithi194_121512_object129257:head# 2024-04-29T14:26:37.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fc2216:::benchmark_data_smithi194_121512_object101688:head# 2024-04-29T14:26:37.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fc5902:::benchmark_data_smithi194_121512_object93217:head# 2024-04-29T14:26:37.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fc7900:::benchmark_data_smithi194_121512_object116327:head# 2024-04-29T14:26:37.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fc8049:::benchmark_data_smithi194_121512_object92660:head# 2024-04-29T14:26:37.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fd06db:::benchmark_data_smithi194_121512_object105260:head# 2024-04-29T14:26:37.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fd71ee:::benchmark_data_smithi194_121512_object66130:head# 2024-04-29T14:26:37.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fd7fc2:::benchmark_data_smithi194_121512_object16063:head# 2024-04-29T14:26:37.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fe35df:::benchmark_data_smithi194_121512_object98950:head# 2024-04-29T14:26:37.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:66fe69cf:::benchmark_data_smithi194_121512_object48347:head# 2024-04-29T14:26:37.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670054dd:::benchmark_data_smithi194_121512_object10624:head# 2024-04-29T14:26:37.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67006631:::benchmark_data_smithi194_121512_object101289:head# 2024-04-29T14:26:37.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6700cc70:::benchmark_data_smithi194_121512_object108808:head# 2024-04-29T14:26:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67021b0e:::benchmark_data_smithi194_121512_object103814:head# 2024-04-29T14:26:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67022e74:::benchmark_data_smithi194_121512_object76206:head# 2024-04-29T14:26:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6702b2c3:::benchmark_data_smithi194_121512_object92716:head# 2024-04-29T14:26:37.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6702e60d:::benchmark_data_smithi194_121512_object16822:head# 2024-04-29T14:26:37.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67048856:::benchmark_data_smithi194_121512_object36721:head# 2024-04-29T14:26:37.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67054fc6:::benchmark_data_smithi194_121512_object36576:head# 2024-04-29T14:26:37.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6705ecc1:::benchmark_data_smithi194_121512_object33459:head# 2024-04-29T14:26:37.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670611d2:::benchmark_data_smithi194_121512_object115128:head# 2024-04-29T14:26:37.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6706f05f:::benchmark_data_smithi194_121512_object82085:head# 2024-04-29T14:26:37.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67071b8c:::benchmark_data_smithi194_121512_object96128:head# 2024-04-29T14:26:37.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670749b4:::benchmark_data_smithi194_121512_object70302:head# 2024-04-29T14:26:37.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67076b12:::benchmark_data_smithi194_121512_object69557:head# 2024-04-29T14:26:37.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6707b686:::benchmark_data_smithi194_121512_object6110:head# 2024-04-29T14:26:37.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67080f35:::benchmark_data_smithi194_121512_object90315:head# 2024-04-29T14:26:37.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67082e61:::benchmark_data_smithi194_121512_object14108:head# 2024-04-29T14:26:37.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670844f5:::benchmark_data_smithi194_121512_object89992:head# 2024-04-29T14:26:37.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67088bb3:::benchmark_data_smithi194_121512_object58258:head# 2024-04-29T14:26:37.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6708d240:::benchmark_data_smithi194_121512_object29700:head# 2024-04-29T14:26:37.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670a3722:::benchmark_data_smithi194_121512_object23002:head# 2024-04-29T14:26:37.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670a5944:::benchmark_data_smithi194_121512_object131165:head# 2024-04-29T14:26:37.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670ab510:::benchmark_data_smithi194_121512_object81247:head# 2024-04-29T14:26:37.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670b247a:::benchmark_data_smithi194_121512_object111490:head# 2024-04-29T14:26:37.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670b535a:::benchmark_data_smithi194_121512_object112739:head# 2024-04-29T14:26:37.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670b5907:::benchmark_data_smithi194_121512_object68561:head# 2024-04-29T14:26:37.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670b9852:::benchmark_data_smithi194_121512_object91947:head# 2024-04-29T14:26:37.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670c14f6:::benchmark_data_smithi194_121512_object91517:head# 2024-04-29T14:26:37.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670c845e:::benchmark_data_smithi194_121512_object84745:head# 2024-04-29T14:26:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670d3582:::benchmark_data_smithi194_121512_object112188:head# 2024-04-29T14:26:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670d9552:::benchmark_data_smithi194_121512_object2084:head# 2024-04-29T14:26:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670dd819:::benchmark_data_smithi194_121512_object91176:head# 2024-04-29T14:26:37.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670f2731:::benchmark_data_smithi194_121512_object17972:head# 2024-04-29T14:26:37.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:670f7121:::benchmark_data_smithi194_121512_object22067:head# 2024-04-29T14:26:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67102393:::benchmark_data_smithi194_121512_object109934:head# 2024-04-29T14:26:37.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67122f95:::benchmark_data_smithi194_121512_object71805:head# 2024-04-29T14:26:37.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67125c89:::benchmark_data_smithi194_121512_object120149:head# 2024-04-29T14:26:37.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671265c0:::benchmark_data_smithi194_121512_object128987:head# 2024-04-29T14:26:37.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6712703f:::benchmark_data_smithi194_121512_object10989:head# 2024-04-29T14:26:37.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67149c09:::benchmark_data_smithi194_121512_object96398:head# 2024-04-29T14:26:37.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6714ffdb:::benchmark_data_smithi194_121512_object72033:head# 2024-04-29T14:26:37.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67154bf8:::benchmark_data_smithi194_121512_object124966:head# 2024-04-29T14:26:37.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67158acd:::benchmark_data_smithi194_121512_object97631:head# 2024-04-29T14:26:37.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67158c15:::benchmark_data_smithi194_121512_object8746:head# 2024-04-29T14:26:37.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6715ba81:::benchmark_data_smithi194_121512_object6896:head# 2024-04-29T14:26:37.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67162b44:::benchmark_data_smithi194_121512_object82428:head# 2024-04-29T14:26:37.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6716f775:::benchmark_data_smithi194_121512_object13085:head# 2024-04-29T14:26:37.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6717386b:::benchmark_data_smithi194_121512_object86492:head# 2024-04-29T14:26:37.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6717f483:::benchmark_data_smithi194_121512_object18350:head# 2024-04-29T14:26:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67186b0e:::benchmark_data_smithi194_121512_object23062:head# 2024-04-29T14:26:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67187aa7:::benchmark_data_smithi194_121512_object77459:head# 2024-04-29T14:26:37.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6719c1f7:::benchmark_data_smithi194_121512_object96356:head# 2024-04-29T14:26:37.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6719fc2e:::benchmark_data_smithi194_121512_object93106:head# 2024-04-29T14:26:37.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671aaa03:::benchmark_data_smithi194_121512_object18758:head# 2024-04-29T14:26:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671b8a64:::benchmark_data_smithi194_121512_object86424:head# 2024-04-29T14:26:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671be89e:::benchmark_data_smithi194_121512_object17060:head# 2024-04-29T14:26:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671c5584:::benchmark_data_smithi194_121512_object107115:head# 2024-04-29T14:26:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671cd58b:::benchmark_data_smithi194_121512_object55114:head# 2024-04-29T14:26:37.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671d424e:::benchmark_data_smithi194_121512_object66950:head# 2024-04-29T14:26:37.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671d8fe5:::benchmark_data_smithi194_121512_object91150:head# 2024-04-29T14:26:37.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671de008:::benchmark_data_smithi194_121512_object132723:head# 2024-04-29T14:26:37.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671e02d7:::benchmark_data_smithi194_121512_object70667:head# 2024-04-29T14:26:37.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671e092e:::benchmark_data_smithi194_121512_object128233:head# 2024-04-29T14:26:37.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671e0a0c:::benchmark_data_smithi194_121512_object77570:head# 2024-04-29T14:26:37.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671e8f8e:::benchmark_data_smithi194_121512_object14986:head# 2024-04-29T14:26:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671ea205:::benchmark_data_smithi194_121512_object91564:head# 2024-04-29T14:26:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671edbf8:::benchmark_data_smithi194_121512_object86812:head# 2024-04-29T14:26:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671f7252:::benchmark_data_smithi194_121512_object29901:head# 2024-04-29T14:26:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:671f7a1e:::benchmark_data_smithi194_121512_object80268:head# 2024-04-29T14:26:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67207a9a:::benchmark_data_smithi194_121512_object15450:head# 2024-04-29T14:26:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6720ca6e:::benchmark_data_smithi194_121512_object89106:head# 2024-04-29T14:26:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67212ebf:::benchmark_data_smithi194_121512_object54872:head# 2024-04-29T14:26:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6721fc58:::benchmark_data_smithi194_121512_object21979:head# 2024-04-29T14:26:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672240e6:::benchmark_data_smithi194_121512_object51868:head# 2024-04-29T14:26:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67225e34:::benchmark_data_smithi194_121512_object68511:head# 2024-04-29T14:26:37.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67227261:::benchmark_data_smithi194_121512_object56529:head# 2024-04-29T14:26:37.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6722829d:::benchmark_data_smithi194_121512_object26550:head# 2024-04-29T14:26:37.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6722d929:::benchmark_data_smithi194_121512_object132048:head# 2024-04-29T14:26:37.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67232ed7:::benchmark_data_smithi194_121512_object74955:head# 2024-04-29T14:26:37.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6723deb0:::benchmark_data_smithi194_121512_object13626:head# 2024-04-29T14:26:37.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6723fa64:::benchmark_data_smithi194_121512_object10203:head# 2024-04-29T14:26:37.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67248295:::benchmark_data_smithi194_121512_object7112:head# 2024-04-29T14:26:37.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672501d0:::benchmark_data_smithi194_121512_object101909:head# 2024-04-29T14:26:37.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6725e9a3:::benchmark_data_smithi194_121512_object10441:head# 2024-04-29T14:26:37.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6726dd8c:::benchmark_data_smithi194_121512_object11328:head# 2024-04-29T14:26:37.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67270cf5:::benchmark_data_smithi194_121512_object115679:head# 2024-04-29T14:26:37.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6727570d:::benchmark_data_smithi194_121512_object90432:head# 2024-04-29T14:26:37.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6727a589:::benchmark_data_smithi194_121512_object79098:head# 2024-04-29T14:26:37.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67280d33:::benchmark_data_smithi194_121512_object97284:head# 2024-04-29T14:26:37.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67282517:::benchmark_data_smithi194_121512_object39820:head# 2024-04-29T14:26:37.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67289ce9:::benchmark_data_smithi194_121512_object96736:head# 2024-04-29T14:26:37.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6729059e:::benchmark_data_smithi194_121512_object57522:head# 2024-04-29T14:26:37.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672a535d:::benchmark_data_smithi194_121512_object114272:head# 2024-04-29T14:26:37.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672a8608:::benchmark_data_smithi194_121512_object57503:head# 2024-04-29T14:26:37.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672b6fc9:::benchmark_data_smithi194_121512_object63880:head# 2024-04-29T14:26:37.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672b8fb3:::benchmark_data_smithi194_121512_object16206:head# 2024-04-29T14:26:37.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672b9e28:::benchmark_data_smithi194_121512_object129550:head# 2024-04-29T14:26:37.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672ba4cb:::benchmark_data_smithi194_121512_object19988:head# 2024-04-29T14:26:37.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672bcefc:::benchmark_data_smithi194_121512_object38888:head# 2024-04-29T14:26:37.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672bf550:::benchmark_data_smithi194_121512_object54034:head# 2024-04-29T14:26:37.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672c6e45:::benchmark_data_smithi194_121512_object114299:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672c7852:::benchmark_data_smithi194_121512_object111254:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672c7bb2:::benchmark_data_smithi194_121512_object126385:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672d331c:::benchmark_data_smithi194_121512_object115643:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672d6475:::benchmark_data_smithi194_121512_object120223:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672e0e4f:::benchmark_data_smithi194_121512_object119544:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672e7852:::benchmark_data_smithi194_121512_object110350:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672ea111:::benchmark_data_smithi194_121512_object107813:head# 2024-04-29T14:26:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672efe7d:::benchmark_data_smithi194_121512_object106752:head# 2024-04-29T14:26:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:672f578b:::benchmark_data_smithi194_121512_object22220:head# 2024-04-29T14:26:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673041f8:::benchmark_data_smithi194_121512_object19843:head# 2024-04-29T14:26:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67308460:::benchmark_data_smithi194_121512_object79921:head# 2024-04-29T14:26:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6730d099:::benchmark_data_smithi194_121512_object121993:head# 2024-04-29T14:26:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6730e14a:::benchmark_data_smithi194_121512_object73693:head# 2024-04-29T14:26:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67332cef:::benchmark_data_smithi194_121512_object50143:head# 2024-04-29T14:26:37.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67335774:::benchmark_data_smithi194_121512_object111855:head# 2024-04-29T14:26:37.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67336673:::benchmark_data_smithi194_121512_object2376:head# 2024-04-29T14:26:37.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67338da6:::benchmark_data_smithi194_121512_object65144:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67349c00:::benchmark_data_smithi194_121512_object128999:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6734e6b5:::benchmark_data_smithi194_121512_object58354:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673562ee:::benchmark_data_smithi194_121512_object30367:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6736b731:::benchmark_data_smithi194_121512_object132697:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6736bb15:::benchmark_data_smithi194_121512_object464:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673732f5:::benchmark_data_smithi194_121512_object62616:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6737ff52:::benchmark_data_smithi194_121512_object54265:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67383335:::benchmark_data_smithi194_121512_object56738:head# 2024-04-29T14:26:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67390156:::benchmark_data_smithi194_121512_object46193:head# 2024-04-29T14:26:37.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67391768:::benchmark_data_smithi194_121512_object33001:head# 2024-04-29T14:26:37.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6739551c:::benchmark_data_smithi194_121512_object55374:head# 2024-04-29T14:26:37.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67399449:::benchmark_data_smithi194_121512_object4417:head# 2024-04-29T14:26:37.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6739dff2:::benchmark_data_smithi194_121512_object65327:head# 2024-04-29T14:26:37.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673a0143:::benchmark_data_smithi194_121512_object19421:head# 2024-04-29T14:26:37.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673a212a:::benchmark_data_smithi194_121512_object18641:head# 2024-04-29T14:26:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673a539c:::benchmark_data_smithi194_121512_object92364:head# 2024-04-29T14:26:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673a70e1:::benchmark_data_smithi194_121512_object35727:head# 2024-04-29T14:26:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673ac7ef:::benchmark_data_smithi194_121512_object95386:head# 2024-04-29T14:26:37.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673b3d5d:::benchmark_data_smithi194_121512_object26665:head# 2024-04-29T14:26:37.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673c0766:::benchmark_data_smithi194_121512_object79415:head# 2024-04-29T14:26:37.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673c9dc6:::benchmark_data_smithi194_121512_object102187:head# 2024-04-29T14:26:37.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673d0bbb:::benchmark_data_smithi194_121512_object55733:head# 2024-04-29T14:26:37.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673e65dd:::benchmark_data_smithi194_121512_object17078:head# 2024-04-29T14:26:37.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:673e79c6:::benchmark_data_smithi194_121512_object32065:head# 2024-04-29T14:26:37.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67406785:::benchmark_data_smithi194_121512_object131393:head# 2024-04-29T14:26:37.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6740a9aa:::benchmark_data_smithi194_121512_object31509:head# 2024-04-29T14:26:37.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6740ee9b:::benchmark_data_smithi194_121512_object68121:head# 2024-04-29T14:26:37.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6742e9e8:::benchmark_data_smithi194_121512_object73818:head# 2024-04-29T14:26:37.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67432774:::benchmark_data_smithi194_121512_object130179:head# 2024-04-29T14:26:37.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67450038:::benchmark_data_smithi194_121512_object45482:head# 2024-04-29T14:26:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6745e949:::benchmark_data_smithi194_121512_object83792:head# 2024-04-29T14:26:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67461310:::benchmark_data_smithi194_121512_object41639:head# 2024-04-29T14:26:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67461666:::benchmark_data_smithi194_121512_object47147:head# 2024-04-29T14:26:37.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6747b99b:::benchmark_data_smithi194_121512_object98846:head# 2024-04-29T14:26:37.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6749de7f:::benchmark_data_smithi194_121512_object71966:head# 2024-04-29T14:26:37.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674a0da0:::benchmark_data_smithi194_121512_object52484:head# 2024-04-29T14:26:37.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674acad1:::benchmark_data_smithi194_121512_object17778:head# 2024-04-29T14:26:37.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674b02af:::benchmark_data_smithi194_121512_object130516:head# 2024-04-29T14:26:37.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674b0e65:::benchmark_data_smithi194_121512_object27738:head# 2024-04-29T14:26:37.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674b0f7d:::benchmark_data_smithi194_121512_object26293:head# 2024-04-29T14:26:37.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674b4355:::benchmark_data_smithi194_121512_object114902:head# 2024-04-29T14:26:37.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674b678e:::benchmark_data_smithi194_121512_object90577:head# 2024-04-29T14:26:37.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674b855c:::benchmark_data_smithi194_121512_object32765:head# 2024-04-29T14:26:37.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674c2cde:::benchmark_data_smithi194_121512_object22659:head# 2024-04-29T14:26:37.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674c5781:::benchmark_data_smithi194_121512_object121727:head# 2024-04-29T14:26:37.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674c84a2:::benchmark_data_smithi194_121512_object100453:head# 2024-04-29T14:26:37.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674d47c3:::benchmark_data_smithi194_121512_object17252:head# 2024-04-29T14:26:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674da98e:::benchmark_data_smithi194_121512_object61633:head# 2024-04-29T14:26:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674dabfd:::benchmark_data_smithi194_121512_object115886:head# 2024-04-29T14:26:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674dd1af:::benchmark_data_smithi194_121512_object75883:head# 2024-04-29T14:26:37.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674e8432:::benchmark_data_smithi194_121512_object24611:head# 2024-04-29T14:26:37.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674e8c8a:::benchmark_data_smithi194_121512_object86012:head# 2024-04-29T14:26:37.406 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:37.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674f8690:::benchmark_data_smithi194_121512_object107625:head# 2024-04-29T14:26:37.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:674fbf15:::benchmark_data_smithi194_121512_object85740:head# 2024-04-29T14:26:37.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67505742:::benchmark_data_smithi194_121512_object20545:head# 2024-04-29T14:26:37.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6750b203:::benchmark_data_smithi194_121512_object114369:head# 2024-04-29T14:26:37.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6752273b:::benchmark_data_smithi194_121512_object13690:head# 2024-04-29T14:26:37.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675247ae:::benchmark_data_smithi194_121512_object107328:head# 2024-04-29T14:26:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:37.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:37.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675261c4:::benchmark_data_smithi194_121512_object51714:head# 2024-04-29T14:26:37.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675339f0:::benchmark_data_smithi194_121512_object20235:head# 2024-04-29T14:26:37.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67536055:::benchmark_data_smithi194_121512_object100576:head# 2024-04-29T14:26:37.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675378e0:::benchmark_data_smithi194_121512_object83837:head# 2024-04-29T14:26:37.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6753c3f8:::benchmark_data_smithi194_121512_object16269:head# 2024-04-29T14:26:37.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67548994:::benchmark_data_smithi194_121512_object88808:head# 2024-04-29T14:26:37.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675663e6:::benchmark_data_smithi194_121512_object71470:head# 2024-04-29T14:26:37.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6756c547:::benchmark_data_smithi194_121512_object96932:head# 2024-04-29T14:26:37.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675810d9:::benchmark_data_smithi194_121512_object123443:head# 2024-04-29T14:26:37.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675853c4:::benchmark_data_smithi194_121512_object46179:head# 2024-04-29T14:26:37.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67588752:::benchmark_data_smithi194_121512_object124671:head# 2024-04-29T14:26:37.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67590e33:::benchmark_data_smithi194_121512_object75586:head# 2024-04-29T14:26:37.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675955a8:::benchmark_data_smithi194_121512_object74555:head# 2024-04-29T14:26:37.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6759929f:::benchmark_data_smithi194_121512_object48006:head# 2024-04-29T14:26:37.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675abe19:::benchmark_data_smithi194_121512_object78503:head# 2024-04-29T14:26:37.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675b4564:::benchmark_data_smithi194_121512_object126333:head# 2024-04-29T14:26:37.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675bb8a8:::benchmark_data_smithi194_121512_object130828:head# 2024-04-29T14:26:37.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675bfddf:::benchmark_data_smithi194_121512_object129546:head# 2024-04-29T14:26:37.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675c2486:::benchmark_data_smithi194_121512_object59765:head# 2024-04-29T14:26:37.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675c6199:::benchmark_data_smithi194_121512_object110971:head# 2024-04-29T14:26:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675dd97c:::benchmark_data_smithi194_121512_object127201:head# 2024-04-29T14:26:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675df9f8:::benchmark_data_smithi194_121512_object111653:head# 2024-04-29T14:26:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675e51c5:::benchmark_data_smithi194_121512_object60039:head# 2024-04-29T14:26:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675e9b9a:::benchmark_data_smithi194_121512_object53949:head# 2024-04-29T14:26:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675ec270:::benchmark_data_smithi194_121512_object76130:head# 2024-04-29T14:26:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675eff1b:::benchmark_data_smithi194_121512_object56305:head# 2024-04-29T14:26:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:675f7dce:::benchmark_data_smithi194_121512_object38552:head# 2024-04-29T14:26:37.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6760f2ec:::benchmark_data_smithi194_121512_object18752:head# 2024-04-29T14:26:37.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67620862:::benchmark_data_smithi194_121512_object65856:head# 2024-04-29T14:26:37.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67621614:::benchmark_data_smithi194_121512_object130260:head# 2024-04-29T14:26:37.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67622aeb:::benchmark_data_smithi194_121512_object623:head# 2024-04-29T14:26:37.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676267e5:::benchmark_data_smithi194_121512_object111486:head# 2024-04-29T14:26:37.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6762b62c:::benchmark_data_smithi194_121512_object45125:head# 2024-04-29T14:26:37.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6762bcd9:::benchmark_data_smithi194_121512_object119098:head# 2024-04-29T14:26:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67633c6a:::benchmark_data_smithi194_121512_object50321:head# 2024-04-29T14:26:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6764a8e2:::benchmark_data_smithi194_121512_object98937:head# 2024-04-29T14:26:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6764c118:::benchmark_data_smithi194_121512_object105639:head# 2024-04-29T14:26:37.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6765d039:::benchmark_data_smithi194_121512_object59956:head# 2024-04-29T14:26:37.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67669818:::benchmark_data_smithi194_121512_object71438:head# 2024-04-29T14:26:37.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67695d54:::benchmark_data_smithi194_121512_object3543:head# 2024-04-29T14:26:37.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676a5df1:::benchmark_data_smithi194_121512_object61190:head# 2024-04-29T14:26:37.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676a9168:::benchmark_data_smithi194_121512_object45626:head# 2024-04-29T14:26:37.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676a9bde:::benchmark_data_smithi194_121512_object42834:head# 2024-04-29T14:26:37.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676acb26:::benchmark_data_smithi194_121512_object40609:head# 2024-04-29T14:26:37.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676acf0d:::benchmark_data_smithi194_121512_object50565:head# 2024-04-29T14:26:37.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676b2d7b:::benchmark_data_smithi194_121512_object109709:head# 2024-04-29T14:26:37.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676c5135:::benchmark_data_smithi194_121512_object2511:head# 2024-04-29T14:26:37.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676c6d6d:::benchmark_data_smithi194_121512_object117806:head# 2024-04-29T14:26:37.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676c8712:::benchmark_data_smithi194_121512_object119101:head# 2024-04-29T14:26:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676e51c5:::benchmark_data_smithi194_121512_object130673:head# 2024-04-29T14:26:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676ef593:::benchmark_data_smithi194_121512_object7697:head# 2024-04-29T14:26:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676f163c:::benchmark_data_smithi194_121512_object5500:head# 2024-04-29T14:26:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:676ff616:::benchmark_data_smithi194_121512_object118673:head# 2024-04-29T14:26:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67702f88:::benchmark_data_smithi194_121512_object53809:head# 2024-04-29T14:26:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6770a249:::benchmark_data_smithi194_121512_object133053:head# 2024-04-29T14:26:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6770c7fc:::benchmark_data_smithi194_121512_object47454:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6771624b:::benchmark_data_smithi194_121512_object14739:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677164b0:::benchmark_data_smithi194_121512_object64791:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67727f4a:::benchmark_data_smithi194_121512_object20307:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6772d58f:::benchmark_data_smithi194_121512_object9827:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6773b8bd:::benchmark_data_smithi194_121512_object33633:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6773fd4c:::benchmark_data_smithi194_121512_object67650:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67750266:::benchmark_data_smithi194_121512_object111993:head# 2024-04-29T14:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67755552:::benchmark_data_smithi194_121512_object129569:head# 2024-04-29T14:26:37.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6775fefe:::benchmark_data_smithi194_121512_object101181:head# 2024-04-29T14:26:37.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677684ce:::benchmark_data_smithi194_121512_object65970:head# 2024-04-29T14:26:37.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677703f6:::benchmark_data_smithi194_121512_object73950:head# 2024-04-29T14:26:37.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67774ada:::benchmark_data_smithi194_121512_object1047:head# 2024-04-29T14:26:37.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6777b422:::benchmark_data_smithi194_121512_object40549:head# 2024-04-29T14:26:37.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6778e892:::benchmark_data_smithi194_121512_object86836:head# 2024-04-29T14:26:37.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677a6764:::benchmark_data_smithi194_121512_object99773:head# 2024-04-29T14:26:37.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677b5031:::benchmark_data_smithi194_121512_object24121:head# 2024-04-29T14:26:37.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677bd119:::benchmark_data_smithi194_121512_object54440:head# 2024-04-29T14:26:37.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677d1f24:::benchmark_data_smithi194_121512_object65125:head# 2024-04-29T14:26:37.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677db6a6:::benchmark_data_smithi194_121512_object114162:head# 2024-04-29T14:26:37.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677dc407:::benchmark_data_smithi194_121512_object84805:head# 2024-04-29T14:26:37.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677eaf76:::benchmark_data_smithi194_121512_object85634:head# 2024-04-29T14:26:37.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677ecdb3:::benchmark_data_smithi194_121512_object22102:head# 2024-04-29T14:26:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:677efb14:::benchmark_data_smithi194_121512_object107878:head# 2024-04-29T14:26:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67800590:::benchmark_data_smithi194_121512_object65989:head# 2024-04-29T14:26:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6780e3de:::benchmark_data_smithi194_121512_object116862:head# 2024-04-29T14:26:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678103d7:::benchmark_data_smithi194_121512_object117289:head# 2024-04-29T14:26:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67814f96:::benchmark_data_smithi194_121512_object26949:head# 2024-04-29T14:26:37.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6781a6a8:::benchmark_data_smithi194_121512_object38293:head# 2024-04-29T14:26:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67823cf5:::benchmark_data_smithi194_121512_object113108:head# 2024-04-29T14:26:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678266ec:::benchmark_data_smithi194_121512_object72542:head# 2024-04-29T14:26:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6782a20f:::benchmark_data_smithi194_121512_object15733:head# 2024-04-29T14:26:37.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6782b1b3:::benchmark_data_smithi194_121512_object104825:head# 2024-04-29T14:26:37.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6782d5bb:::benchmark_data_smithi194_121512_object15270:head# 2024-04-29T14:26:37.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67834d17:::benchmark_data_smithi194_121512_object43823:head# 2024-04-29T14:26:37.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6783a780:::benchmark_data_smithi194_121512_object58137:head# 2024-04-29T14:26:37.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6783a78e:::benchmark_data_smithi194_121512_object85847:head# 2024-04-29T14:26:37.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6783da59:::benchmark_data_smithi194_121512_object78292:head# 2024-04-29T14:26:37.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6784547d:::benchmark_data_smithi194_121512_object133177:head# 2024-04-29T14:26:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6784b5cd:::benchmark_data_smithi194_121512_object77862:head# 2024-04-29T14:26:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678563ce:::benchmark_data_smithi194_121512_object48518:head# 2024-04-29T14:26:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6785ca50:::benchmark_data_smithi194_121512_object94364:head# 2024-04-29T14:26:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6785f64e:::benchmark_data_smithi194_121512_object89127:head# 2024-04-29T14:26:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678617f7:::benchmark_data_smithi194_121512_object22890:head# 2024-04-29T14:26:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678633a3:::benchmark_data_smithi194_121512_object118089:head# 2024-04-29T14:26:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67864d7a:::benchmark_data_smithi194_121512_object3407:head# 2024-04-29T14:26:37.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6787348e:::benchmark_data_smithi194_121512_object106056:head# 2024-04-29T14:26:37.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6787f19c:::benchmark_data_smithi194_121512_object96619:head# 2024-04-29T14:26:37.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6787f998:::benchmark_data_smithi194_121512_object18555:head# 2024-04-29T14:26:37.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6788437a:::benchmark_data_smithi194_121512_object118574:head# 2024-04-29T14:26:37.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6788991e:::benchmark_data_smithi194_121512_object110259:head# 2024-04-29T14:26:37.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6789c7b8:::benchmark_data_smithi194_121512_object86395:head# 2024-04-29T14:26:37.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678a96af:::benchmark_data_smithi194_121512_object97294:head# 2024-04-29T14:26:37.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678aa99b:::benchmark_data_smithi194_121512_object77501:head# 2024-04-29T14:26:37.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678abe00:::benchmark_data_smithi194_121512_object57089:head# 2024-04-29T14:26:37.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678bbe97:::benchmark_data_smithi194_121512_object131004:head# 2024-04-29T14:26:37.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678be556:::benchmark_data_smithi194_121512_object68161:head# 2024-04-29T14:26:37.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678d7b22:::benchmark_data_smithi194_121512_object120446:head# 2024-04-29T14:26:37.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678dea63:::benchmark_data_smithi194_121512_object120978:head# 2024-04-29T14:26:37.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678ece68:::benchmark_data_smithi194_121512_object51622:head# 2024-04-29T14:26:37.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678f36e1:::benchmark_data_smithi194_121512_object113587:head# 2024-04-29T14:26:37.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:678f9547:::benchmark_data_smithi194_121512_object132978:head# 2024-04-29T14:26:37.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6790cfee:::benchmark_data_smithi194_121512_object75765:head# 2024-04-29T14:26:37.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6790f947:::benchmark_data_smithi194_121512_object1849:head# 2024-04-29T14:26:37.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6791b54d:::benchmark_data_smithi194_121512_object110955:head# 2024-04-29T14:26:37.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6791f7c7:::benchmark_data_smithi194_121512_object86601:head# 2024-04-29T14:26:37.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67920137:::benchmark_data_smithi194_121512_object91991:head# 2024-04-29T14:26:37.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67923f7f:::benchmark_data_smithi194_121512_object35916:head# 2024-04-29T14:26:37.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67926618:::benchmark_data_smithi194_121512_object42766:head# 2024-04-29T14:26:37.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6792ce21:::benchmark_data_smithi194_121512_object9941:head# 2024-04-29T14:26:37.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6792e69f:::benchmark_data_smithi194_121512_object38780:head# 2024-04-29T14:26:37.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67938f41:::benchmark_data_smithi194_121512_object49885:head# 2024-04-29T14:26:37.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67939995:::benchmark_data_smithi194_121512_object1517:head# 2024-04-29T14:26:37.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67945fe2:::benchmark_data_smithi194_121512_object76211:head# 2024-04-29T14:26:37.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67949a90:::benchmark_data_smithi194_121512_object26007:head# 2024-04-29T14:26:37.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67954419:::benchmark_data_smithi194_121512_object10002:head# 2024-04-29T14:26:37.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67961dfe:::benchmark_data_smithi194_121512_object80398:head# 2024-04-29T14:26:37.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6796e0e8:::benchmark_data_smithi194_121512_object62218:head# 2024-04-29T14:26:37.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679758c8:::benchmark_data_smithi194_121512_object13542:head# 2024-04-29T14:26:37.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6797777d:::benchmark_data_smithi194_121512_object69226:head# 2024-04-29T14:26:37.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6797b390:::benchmark_data_smithi194_121512_object60234:head# 2024-04-29T14:26:37.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6797e3f1:::benchmark_data_smithi194_121512_object7258:head# 2024-04-29T14:26:37.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6798cd25:::benchmark_data_smithi194_121512_object68620:head# 2024-04-29T14:26:37.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679965f0:::benchmark_data_smithi194_121512_object56455:head# 2024-04-29T14:26:37.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6799807b:::benchmark_data_smithi194_121512_object18374:head# 2024-04-29T14:26:37.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6799838c:::benchmark_data_smithi194_121512_object46020:head# 2024-04-29T14:26:37.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6799ec85:::benchmark_data_smithi194_121512_object110504:head# 2024-04-29T14:26:37.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679a119d:::benchmark_data_smithi194_121512_object50556:head# 2024-04-29T14:26:37.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679c105e:::benchmark_data_smithi194_121512_object110527:head# 2024-04-29T14:26:37.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679d47ea:::benchmark_data_smithi194_121512_object16453:head# 2024-04-29T14:26:37.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679d5942:::benchmark_data_smithi194_121512_object125160:head# 2024-04-29T14:26:37.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679d86e2:::benchmark_data_smithi194_121512_object120371:head# 2024-04-29T14:26:37.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679daa04:::benchmark_data_smithi194_121512_object43338:head# 2024-04-29T14:26:37.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679e8fb4:::benchmark_data_smithi194_121512_object3698:head# 2024-04-29T14:26:37.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679ec7c5:::benchmark_data_smithi194_121512_object88822:head# 2024-04-29T14:26:37.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679f672f:::benchmark_data_smithi194_121512_object100579:head# 2024-04-29T14:26:37.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:679fd35f:::benchmark_data_smithi194_121512_object122908:head# 2024-04-29T14:26:37.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a00f63:::benchmark_data_smithi194_121512_object72238:head# 2024-04-29T14:26:37.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a031e5:::benchmark_data_smithi194_121512_object43366:head# 2024-04-29T14:26:37.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a07c7e:::benchmark_data_smithi194_121512_object62403:head# 2024-04-29T14:26:37.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a08206:::benchmark_data_smithi194_121512_object125436:head# 2024-04-29T14:26:37.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a0baaa:::benchmark_data_smithi194_121512_object83642:head# 2024-04-29T14:26:37.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a11ff1:::benchmark_data_smithi194_121512_object95453:head# 2024-04-29T14:26:37.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a139d7:::benchmark_data_smithi194_121512_object4936:head# 2024-04-29T14:26:37.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a13be9:::benchmark_data_smithi194_121512_object42063:head# 2024-04-29T14:26:37.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a198eb:::benchmark_data_smithi194_121512_object54666:head# 2024-04-29T14:26:37.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a1d743:::benchmark_data_smithi194_121512_object52344:head# 2024-04-29T14:26:37.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a1e572:::benchmark_data_smithi194_121512_object5020:head# 2024-04-29T14:26:37.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a260c7:::benchmark_data_smithi194_121512_object9526:head# 2024-04-29T14:26:37.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a276ac:::benchmark_data_smithi194_121512_object63943:head# 2024-04-29T14:26:37.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a2e681:::benchmark_data_smithi194_121512_object92670:head# 2024-04-29T14:26:37.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a315db:::benchmark_data_smithi194_121512_object28752:head# 2024-04-29T14:26:37.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a36232:::benchmark_data_smithi194_121512_object93355:head# 2024-04-29T14:26:37.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a37bdf:::benchmark_data_smithi194_121512_object117519:head# 2024-04-29T14:26:37.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a389e7:::benchmark_data_smithi194_121512_object81865:head# 2024-04-29T14:26:37.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a49b98:::benchmark_data_smithi194_121512_object107809:head# 2024-04-29T14:26:37.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a52e10:::benchmark_data_smithi194_121512_object60329:head# 2024-04-29T14:26:37.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a64da4:::benchmark_data_smithi194_121512_object124588:head# 2024-04-29T14:26:37.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a6816d:::benchmark_data_smithi194_121512_object41864:head# 2024-04-29T14:26:37.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a70ed9:::benchmark_data_smithi194_121512_object86792:head# 2024-04-29T14:26:37.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a72452:::benchmark_data_smithi194_121512_object112234:head# 2024-04-29T14:26:37.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a8b115:::benchmark_data_smithi194_121512_object121741:head# 2024-04-29T14:26:37.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67a8e182:::benchmark_data_smithi194_121512_object29631:head# 2024-04-29T14:26:37.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67aa1735:::benchmark_data_smithi194_121512_object94689:head# 2024-04-29T14:26:37.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ac3936:::benchmark_data_smithi194_121512_object56098:head# 2024-04-29T14:26:37.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ad32f1:::benchmark_data_smithi194_121512_object94400:head# 2024-04-29T14:26:37.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ae695b:::benchmark_data_smithi194_121512_object45116:head# 2024-04-29T14:26:37.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ae9175:::benchmark_data_smithi194_121512_object55729:head# 2024-04-29T14:26:37.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b115a6:::benchmark_data_smithi194_121512_object102315:head# 2024-04-29T14:26:37.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b13176:::benchmark_data_smithi194_121512_object50303:head# 2024-04-29T14:26:37.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b154bb:::benchmark_data_smithi194_121512_object15251:head# 2024-04-29T14:26:37.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b16d4a:::benchmark_data_smithi194_121512_object30641:head# 2024-04-29T14:26:37.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b19523:::benchmark_data_smithi194_121512_object101482:head# 2024-04-29T14:26:37.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b1d615:::benchmark_data_smithi194_121512_object10277:head# 2024-04-29T14:26:37.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b20196:::benchmark_data_smithi194_121512_object93109:head# 2024-04-29T14:26:37.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b23192:::benchmark_data_smithi194_121512_object106652:head# 2024-04-29T14:26:37.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b356ef:::benchmark_data_smithi194_121512_object37741:head# 2024-04-29T14:26:37.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b394c7:::benchmark_data_smithi194_121512_object75574:head# 2024-04-29T14:26:37.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b42cad:::benchmark_data_smithi194_121512_object22602:head# 2024-04-29T14:26:37.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b42ce3:::benchmark_data_smithi194_121512_object92840:head# 2024-04-29T14:26:37.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b4cb6e:::benchmark_data_smithi194_121512_object57036:head# 2024-04-29T14:26:37.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b67677:::benchmark_data_smithi194_121512_object52302:head# 2024-04-29T14:26:37.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b7760a:::benchmark_data_smithi194_121512_object119616:head# 2024-04-29T14:26:37.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b83bfd:::benchmark_data_smithi194_121512_object71796:head# 2024-04-29T14:26:37.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b98e4b:::benchmark_data_smithi194_121512_object38251:head# 2024-04-29T14:26:37.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67b9ec93:::benchmark_data_smithi194_121512_object7314:head# 2024-04-29T14:26:37.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ba5fb3:::benchmark_data_smithi194_121512_object19189:head# 2024-04-29T14:26:37.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bad79b:::benchmark_data_smithi194_121512_object2810:head# 2024-04-29T14:26:37.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bb7c76:::benchmark_data_smithi194_121512_object82402:head# 2024-04-29T14:26:37.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bc09fa:::benchmark_data_smithi194_121512_object18109:head# 2024-04-29T14:26:37.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bc27ec:::benchmark_data_smithi194_121512_object106302:head# 2024-04-29T14:26:37.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bc40fc:::benchmark_data_smithi194_121512_object92792:head# 2024-04-29T14:26:37.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bc7dca:::benchmark_data_smithi194_121512_object18456:head# 2024-04-29T14:26:37.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bc9ea4:::benchmark_data_smithi194_121512_object21422:head# 2024-04-29T14:26:37.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67be0cda:::benchmark_data_smithi194_121512_object124486:head# 2024-04-29T14:26:37.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67be23c3:::benchmark_data_smithi194_121512_object98765:head# 2024-04-29T14:26:37.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67be9fb5:::benchmark_data_smithi194_121512_object121779:head# 2024-04-29T14:26:37.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bef82c:::benchmark_data_smithi194_121512_object89598:head# 2024-04-29T14:26:37.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67bf75b2:::benchmark_data_smithi194_121512_object11822:head# 2024-04-29T14:26:37.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c04e1f:::benchmark_data_smithi194_121512_object38374:head# 2024-04-29T14:26:37.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c0c2ac:::benchmark_data_smithi194_121512_object96266:head# 2024-04-29T14:26:37.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c0cd3e:::benchmark_data_smithi194_121512_object119500:head# 2024-04-29T14:26:37.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c123d2:::benchmark_data_smithi194_121512_object22827:head# 2024-04-29T14:26:37.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c1264c:::benchmark_data_smithi194_121512_object20531:head# 2024-04-29T14:26:37.509 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:37.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c16299:::benchmark_data_smithi194_121512_object76638:head# 2024-04-29T14:26:37.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c215d6:::benchmark_data_smithi194_121512_object26412:head# 2024-04-29T14:26:37.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c21df7:::benchmark_data_smithi194_121512_object38182:head# 2024-04-29T14:26:37.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c23989:::benchmark_data_smithi194_121512_object93469:head# 2024-04-29T14:26:37.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c24d73:::benchmark_data_smithi194_121512_object78246:head# 2024-04-29T14:26:37.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c254a7:::benchmark_data_smithi194_121512_object108880:head# 2024-04-29T14:26:37.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c271a1:::benchmark_data_smithi194_121512_object39131:head# 2024-04-29T14:26:37.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c3dc17:::benchmark_data_smithi194_121512_object113350:head# 2024-04-29T14:26:37.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c45061:::benchmark_data_smithi194_121512_object103057:head# 2024-04-29T14:26:37.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c59cd4:::benchmark_data_smithi194_121512_object25468:head# 2024-04-29T14:26:37.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c5a6c1:::benchmark_data_smithi194_121512_object105961:head# 2024-04-29T14:26:37.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c62ec3:::benchmark_data_smithi194_121512_object126091:head# 2024-04-29T14:26:37.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c67748:::benchmark_data_smithi194_121512_object72497:head# 2024-04-29T14:26:37.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c67b20:::benchmark_data_smithi194_121512_object32794:head# 2024-04-29T14:26:37.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c7b941:::benchmark_data_smithi194_121512_object93111:head# 2024-04-29T14:26:37.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c88baa:::benchmark_data_smithi194_121512_object95592:head# 2024-04-29T14:26:37.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c936eb:::benchmark_data_smithi194_121512_object28573:head# 2024-04-29T14:26:37.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67c9c510:::benchmark_data_smithi194_121512_object58672:head# 2024-04-29T14:26:37.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ca5d96:::benchmark_data_smithi194_121512_object17413:head# 2024-04-29T14:26:37.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ca9343:::benchmark_data_smithi194_121512_object132585:head# 2024-04-29T14:26:37.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67cb68ae:::benchmark_data_smithi194_121512_object113601:head# 2024-04-29T14:26:37.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67cb9e63:::benchmark_data_smithi194_121512_object113541:head# 2024-04-29T14:26:37.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67cd86bf:::benchmark_data_smithi194_121512_object78074:head# 2024-04-29T14:26:37.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67cec881:::benchmark_data_smithi194_121512_object57329:head# 2024-04-29T14:26:37.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67cf57d8:::benchmark_data_smithi194_121512_object94027:head# 2024-04-29T14:26:37.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67cf95b4:::benchmark_data_smithi194_121512_object67177:head# 2024-04-29T14:26:37.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d083d9:::benchmark_data_smithi194_121512_object20887:head# 2024-04-29T14:26:37.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d08482:::benchmark_data_smithi194_121512_object73317:head# 2024-04-29T14:26:37.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d09f59:::benchmark_data_smithi194_121512_object63433:head# 2024-04-29T14:26:37.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d0abae:::benchmark_data_smithi194_121512_object17196:head# 2024-04-29T14:26:37.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d0fb82:::benchmark_data_smithi194_121512_object106946:head# 2024-04-29T14:26:37.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d112a2:::benchmark_data_smithi194_121512_object94115:head# 2024-04-29T14:26:37.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d118fc:::benchmark_data_smithi194_121512_object62385:head# 2024-04-29T14:26:37.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d27723:::benchmark_data_smithi194_121512_object53450:head# 2024-04-29T14:26:37.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d36b78:::benchmark_data_smithi194_121512_object10945:head# 2024-04-29T14:26:37.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d37af0:::benchmark_data_smithi194_121512_object113007:head# 2024-04-29T14:26:37.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d593b5:::benchmark_data_smithi194_121512_object12738:head# 2024-04-29T14:26:37.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d67627:::benchmark_data_smithi194_121512_object46240:head# 2024-04-29T14:26:37.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d72001:::benchmark_data_smithi194_121512_object61516:head# 2024-04-29T14:26:37.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d75300:::benchmark_data_smithi194_121512_object131589:head# 2024-04-29T14:26:37.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d80fe7:::benchmark_data_smithi194_121512_object83145:head# 2024-04-29T14:26:37.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d82372:::benchmark_data_smithi194_121512_object40324:head# 2024-04-29T14:26:37.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d8a764:::benchmark_data_smithi194_121512_object26023:head# 2024-04-29T14:26:37.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67d9f60c:::benchmark_data_smithi194_121512_object45307:head# 2024-04-29T14:26:37.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67da01d4:::benchmark_data_smithi194_121512_object75108:head# 2024-04-29T14:26:37.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67db5402:::benchmark_data_smithi194_121512_object57361:head# 2024-04-29T14:26:37.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67db9ac6:::benchmark_data_smithi194_121512_object108091:head# 2024-04-29T14:26:37.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67dba27d:::benchmark_data_smithi194_121512_object108837:head# 2024-04-29T14:26:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67dbb705:::benchmark_data_smithi194_121512_object86755:head# 2024-04-29T14:26:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67dc9cfe:::benchmark_data_smithi194_121512_object19248:head# 2024-04-29T14:26:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67dca028:::benchmark_data_smithi194_121512_object54403:head# 2024-04-29T14:26:37.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67dd0d01:::benchmark_data_smithi194_121512_object31835:head# 2024-04-29T14:26:37.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67dd324d:::benchmark_data_smithi194_121512_object5778:head# 2024-04-29T14:26:37.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67dd45e9:::benchmark_data_smithi194_121512_object1998:head# 2024-04-29T14:26:37.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ddbec8:::benchmark_data_smithi194_121512_object124772:head# 2024-04-29T14:26:37.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67de3b09:::benchmark_data_smithi194_121512_object5252:head# 2024-04-29T14:26:37.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e0087a:::benchmark_data_smithi194_121512_object35079:head# 2024-04-29T14:26:37.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e0a9b2:::benchmark_data_smithi194_121512_object58256:head# 2024-04-29T14:26:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e12335:::benchmark_data_smithi194_121512_object56435:head# 2024-04-29T14:26:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e1a953:::benchmark_data_smithi194_121512_object35428:head# 2024-04-29T14:26:37.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e1b30d:::benchmark_data_smithi194_121512_object109628:head# 2024-04-29T14:26:37.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e271f6:::benchmark_data_smithi194_121512_object89791:head# 2024-04-29T14:26:37.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e476f3:::benchmark_data_smithi194_121512_object116991:head# 2024-04-29T14:26:37.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e494cb:::benchmark_data_smithi194_121512_object29632:head# 2024-04-29T14:26:37.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e4a6a8:::benchmark_data_smithi194_121512_object86625:head# 2024-04-29T14:26:37.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e6aa90:::benchmark_data_smithi194_121512_object113739:head# 2024-04-29T14:26:37.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e849d7:::benchmark_data_smithi194_121512_object108681:head# 2024-04-29T14:26:37.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e87007:::benchmark_data_smithi194_121512_object93817:head# 2024-04-29T14:26:37.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e8c9c3:::benchmark_data_smithi194_121512_object8768:head# 2024-04-29T14:26:37.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e8e8ce:::benchmark_data_smithi194_121512_object91711:head# 2024-04-29T14:26:37.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e943fe:::benchmark_data_smithi194_121512_object40349:head# 2024-04-29T14:26:37.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67e9f9e0:::benchmark_data_smithi194_121512_object62415:head# 2024-04-29T14:26:37.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67eab210:::benchmark_data_smithi194_121512_object89408:head# 2024-04-29T14:26:37.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67eaf1e2:::benchmark_data_smithi194_121512_object57427:head# 2024-04-29T14:26:37.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67eb2418:::benchmark_data_smithi194_121512_object78258:head# 2024-04-29T14:26:37.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67eb3fbd:::benchmark_data_smithi194_121512_object28849:head# 2024-04-29T14:26:37.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67eb9e06:::benchmark_data_smithi194_121512_object130237:head# 2024-04-29T14:26:37.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ebcb52:::benchmark_data_smithi194_121512_object15116:head# 2024-04-29T14:26:37.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ec1127:::benchmark_data_smithi194_121512_object35285:head# 2024-04-29T14:26:37.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ed89c4:::benchmark_data_smithi194_121512_object8950:head# 2024-04-29T14:26:37.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67efa4ed:::benchmark_data_smithi194_121512_object101282:head# 2024-04-29T14:26:37.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f0cf62:::benchmark_data_smithi194_121512_object16964:head# 2024-04-29T14:26:37.547 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:37.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f0f862:::benchmark_data_smithi194_121512_object15695:head# 2024-04-29T14:26:37.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f23f6f:::benchmark_data_smithi194_121512_object76296:head# 2024-04-29T14:26:37.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f32a1e:::benchmark_data_smithi194_121512_object61172:head# 2024-04-29T14:26:37.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:37 smithi194 ceph-mon[44089]: pgmap v1248: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 38967/247944 objects degraded (15.716%); 9.9 MiB/s, 158 objects/s recovering 2024-04-29T14:26:37.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:37 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:37.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3241230645' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:37.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:37 smithi194 ceph-mon[44089]: osdmap e474: 12 total, 11 up, 7 in 2024-04-29T14:26:37.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:37.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:37.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:37 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 38967/247944 objects degraded (15.716%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:37.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f6b2df:::benchmark_data_smithi194_121512_object52948:head# 2024-04-29T14:26:37.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f7001e:::benchmark_data_smithi194_121512_object23369:head# 2024-04-29T14:26:37.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f7efe8:::benchmark_data_smithi194_121512_object10740:head# 2024-04-29T14:26:37.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f98bea:::benchmark_data_smithi194_121512_object97487:head# 2024-04-29T14:26:37.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f99e04:::benchmark_data_smithi194_121512_object2731:head# 2024-04-29T14:26:37.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67f9d647:::benchmark_data_smithi194_121512_object111915:head# 2024-04-29T14:26:37.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fb1613:::benchmark_data_smithi194_121512_object129363:head# 2024-04-29T14:26:37.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fb331e:::benchmark_data_smithi194_121512_object83377:head# 2024-04-29T14:26:37.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fb8b3f:::benchmark_data_smithi194_121512_object64619:head# 2024-04-29T14:26:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fbd859:::benchmark_data_smithi194_121512_object118031:head# 2024-04-29T14:26:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fbeda2:::benchmark_data_smithi194_121512_object38972:head# 2024-04-29T14:26:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fc2498:::benchmark_data_smithi194_121512_object91016:head# 2024-04-29T14:26:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fc4ca1:::benchmark_data_smithi194_121512_object26253:head# 2024-04-29T14:26:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fcab31:::benchmark_data_smithi194_121512_object75076:head# 2024-04-29T14:26:37.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fd107b:::benchmark_data_smithi194_121512_object87023:head# 2024-04-29T14:26:37.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fda63b:::benchmark_data_smithi194_121512_object36343:head# 2024-04-29T14:26:37.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fde024:::benchmark_data_smithi194_121512_object113991:head# 2024-04-29T14:26:37.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fde851:::benchmark_data_smithi194_121512_object74215:head# 2024-04-29T14:26:37.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67fe0f49:::benchmark_data_smithi194_121512_object120431:head# 2024-04-29T14:26:37.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:67ffbf99:::benchmark_data_smithi194_121512_object90627:head# 2024-04-29T14:26:37.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68000ddd:::benchmark_data_smithi194_121512_object92948:head# 2024-04-29T14:26:37.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68001fb1:::benchmark_data_smithi194_121512_object96523:head# 2024-04-29T14:26:37.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6800ac4a:::benchmark_data_smithi194_121512_object83632:head# 2024-04-29T14:26:37.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6800b420:::benchmark_data_smithi194_121512_object126918:head# 2024-04-29T14:26:37.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68016356:::benchmark_data_smithi194_121512_object51677:head# 2024-04-29T14:26:37.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68018d97:::benchmark_data_smithi194_121512_object97962:head# 2024-04-29T14:26:37.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6801af2f:::benchmark_data_smithi194_121512_object7790:head# 2024-04-29T14:26:37.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6801c86a:::benchmark_data_smithi194_121512_object37044:head# 2024-04-29T14:26:37.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68033226:::benchmark_data_smithi194_121512_object67568:head# 2024-04-29T14:26:37.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680406c8:::benchmark_data_smithi194_121512_object127404:head# 2024-04-29T14:26:37.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68041bc0:::benchmark_data_smithi194_121512_object58668:head# 2024-04-29T14:26:37.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68043135:::benchmark_data_smithi194_121512_object56460:head# 2024-04-29T14:26:37.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6804b15f:::benchmark_data_smithi194_121512_object29639:head# 2024-04-29T14:26:37.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6804f2f8:::benchmark_data_smithi194_121512_object94753:head# 2024-04-29T14:26:37.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6805118a:::benchmark_data_smithi194_121512_object48903:head# 2024-04-29T14:26:37.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68054053:::benchmark_data_smithi194_121512_object71551:head# 2024-04-29T14:26:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6805c978:::benchmark_data_smithi194_121512_object115242:head# 2024-04-29T14:26:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68061749:::benchmark_data_smithi194_121512_object79058:head# 2024-04-29T14:26:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68067cbc:::benchmark_data_smithi194_121512_object52958:head# 2024-04-29T14:26:37.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68068c7b:::benchmark_data_smithi194_121512_object87296:head# 2024-04-29T14:26:37.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68072a0d:::benchmark_data_smithi194_121512_object79561:head# 2024-04-29T14:26:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6807f594:::benchmark_data_smithi194_121512_object77611:head# 2024-04-29T14:26:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6808358b:::benchmark_data_smithi194_121512_object49267:head# 2024-04-29T14:26:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68089337:::benchmark_data_smithi194_121512_object75880:head# 2024-04-29T14:26:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680941e4:::benchmark_data_smithi194_121512_object115543:head# 2024-04-29T14:26:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68096ee8:::benchmark_data_smithi194_121512_object45601:head# 2024-04-29T14:26:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6809a73e:::benchmark_data_smithi194_121512_object48192:head# 2024-04-29T14:26:37.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6809b031:::benchmark_data_smithi194_121512_object11377:head# 2024-04-29T14:26:37.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680a4699:::benchmark_data_smithi194_121512_object54140:head# 2024-04-29T14:26:37.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680ac8cf:::benchmark_data_smithi194_121512_object30098:head# 2024-04-29T14:26:37.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680bbcc0:::benchmark_data_smithi194_121512_object57581:head# 2024-04-29T14:26:37.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680c17b1:::benchmark_data_smithi194_121512_object18169:head# 2024-04-29T14:26:37.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680c60dd:::benchmark_data_smithi194_121512_object67611:head# 2024-04-29T14:26:37.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680cd3fd:::benchmark_data_smithi194_121512_object103308:head# 2024-04-29T14:26:37.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680cf62c:::benchmark_data_smithi194_121512_object17203:head# 2024-04-29T14:26:37.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:680ecec1:::benchmark_data_smithi194_121512_object10971:head# 2024-04-29T14:26:37.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68101bd0:::benchmark_data_smithi194_121512_object109892:head# 2024-04-29T14:26:37.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68105f73:::benchmark_data_smithi194_121512_object83695:head# 2024-04-29T14:26:37.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68106b64:::benchmark_data_smithi194_121512_object103336:head# 2024-04-29T14:26:37.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6810f013:::benchmark_data_smithi194_121512_object121189:head# 2024-04-29T14:26:37.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68112441:::benchmark_data_smithi194_121512_object100107:head# 2024-04-29T14:26:37.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68129cb5:::benchmark_data_smithi194_121512_object6573:head# 2024-04-29T14:26:37.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6812ab4f:::benchmark_data_smithi194_121512_object80721:head# 2024-04-29T14:26:37.576 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:37.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68130715:::benchmark_data_smithi194_121512_object24627:head# 2024-04-29T14:26:37.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6814029f:::benchmark_data_smithi194_121512_object30073:head# 2024-04-29T14:26:37.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681498dd:::benchmark_data_smithi194_121512_object46482:head# 2024-04-29T14:26:37.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6814c0ee:::benchmark_data_smithi194_121512_object67483:head# 2024-04-29T14:26:37.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6815a375:::benchmark_data_smithi194_121512_object54696:head# 2024-04-29T14:26:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:37.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:37.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68170186:::benchmark_data_smithi194_121512_object63817:head# 2024-04-29T14:26:37.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68172f74:::benchmark_data_smithi194_121512_object36241:head# 2024-04-29T14:26:37.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68180a3d:::benchmark_data_smithi194_121512_object82050:head# 2024-04-29T14:26:37.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6818a14a:::benchmark_data_smithi194_121512_object31084:head# 2024-04-29T14:26:37.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6819a2e8:::benchmark_data_smithi194_121512_object87352:head# 2024-04-29T14:26:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6819c1be:::benchmark_data_smithi194_121512_object80881:head# 2024-04-29T14:26:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681a3448:::benchmark_data_smithi194_121512_object72336:head# 2024-04-29T14:26:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681bb299:::benchmark_data_smithi194_121512_object66391:head# 2024-04-29T14:26:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681cf7c8:::benchmark_data_smithi194_121512_object48767:head# 2024-04-29T14:26:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681d810c:::benchmark_data_smithi194_121512_object130185:head# 2024-04-29T14:26:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681ef027:::benchmark_data_smithi194_121512_object46726:head# 2024-04-29T14:26:37.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681f8e5c:::benchmark_data_smithi194_121512_object101930:head# 2024-04-29T14:26:37.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681f9346:::benchmark_data_smithi194_121512_object122137:head# 2024-04-29T14:26:37.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681fa203:::benchmark_data_smithi194_121512_object9631:head# 2024-04-29T14:26:37.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:681fda6e:::benchmark_data_smithi194_121512_object59893:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682032c5:::benchmark_data_smithi194_121512_object13021:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6820dbfb:::benchmark_data_smithi194_121512_object101577:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68227573:::benchmark_data_smithi194_121512_object9201:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6822b73a:::benchmark_data_smithi194_121512_object29273:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6822c6d7:::benchmark_data_smithi194_121512_object127956:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6822fd48:::benchmark_data_smithi194_121512_object99863:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6823eeee:::benchmark_data_smithi194_121512_object31143:head# 2024-04-29T14:26:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6824605d:::benchmark_data_smithi194_121512_object40878:head# 2024-04-29T14:26:37.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68250005:::benchmark_data_smithi194_121512_object103845:head# 2024-04-29T14:26:37.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6825ba8a:::benchmark_data_smithi194_121512_object7775:head# 2024-04-29T14:26:37.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682620dc:::benchmark_data_smithi194_121512_object130980:head# 2024-04-29T14:26:37.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68268815:::benchmark_data_smithi194_121512_object113853:head# 2024-04-29T14:26:37.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6826a518:::benchmark_data_smithi194_121512_object70774:head# 2024-04-29T14:26:37.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6826c0db:::benchmark_data_smithi194_121512_object82055:head# 2024-04-29T14:26:37.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6826df23:::benchmark_data_smithi194_121512_object127988:head# 2024-04-29T14:26:37.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682747c1:::benchmark_data_smithi194_121512_object7214:head# 2024-04-29T14:26:37.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68276f4d:::benchmark_data_smithi194_121512_object35974:head# 2024-04-29T14:26:37.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6828548b:::benchmark_data_smithi194_121512_object8859:head# 2024-04-29T14:26:37.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6828c9a6:::benchmark_data_smithi194_121512_object15491:head# 2024-04-29T14:26:37.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6828dca3:::benchmark_data_smithi194_121512_object71604:head# 2024-04-29T14:26:37.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68294c8c:::benchmark_data_smithi194_121512_object107655:head# 2024-04-29T14:26:37.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6829e69a:::benchmark_data_smithi194_121512_object61391:head# 2024-04-29T14:26:37.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682a472d:::benchmark_data_smithi194_121512_object88268:head# 2024-04-29T14:26:37.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682b34cb:::benchmark_data_smithi194_121512_object31676:head# 2024-04-29T14:26:37.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682b53fd:::benchmark_data_smithi194_121512_object18757:head# 2024-04-29T14:26:37.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682ba652:::benchmark_data_smithi194_121512_object125317:head# 2024-04-29T14:26:37.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682c4dda:::benchmark_data_smithi194_121512_object74439:head# 2024-04-29T14:26:37.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682c629b:::benchmark_data_smithi194_121512_object6915:head# 2024-04-29T14:26:37.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682d3391:::benchmark_data_smithi194_121512_object59460:head# 2024-04-29T14:26:37.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682d772f:::benchmark_data_smithi194_121512_object82274:head# 2024-04-29T14:26:37.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682d84c6:::benchmark_data_smithi194_121512_object12093:head# 2024-04-29T14:26:37.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682f82ab:::benchmark_data_smithi194_121512_object128950:head# 2024-04-29T14:26:37.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:682f8d27:::benchmark_data_smithi194_121512_object58004:head# 2024-04-29T14:26:37.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68300030:::benchmark_data_smithi194_121512_object118269:head# 2024-04-29T14:26:37.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6830e56a:::benchmark_data_smithi194_121512_object54942:head# 2024-04-29T14:26:37.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:37 smithi040 ceph-mon[39581]: pgmap v1248: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 38967/247944 objects degraded (15.716%); 9.9 MiB/s, 158 objects/s recovering 2024-04-29T14:26:37.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:37 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:37.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3241230645' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:37.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:37 smithi040 ceph-mon[39581]: osdmap e474: 12 total, 11 up, 7 in 2024-04-29T14:26:37.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:37.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:37.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:37 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 38967/247944 objects degraded (15.716%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68315618:::benchmark_data_smithi194_121512_object23827:head# 2024-04-29T14:26:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6831fe13:::benchmark_data_smithi194_121512_object63037:head# 2024-04-29T14:26:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68320099:::benchmark_data_smithi194_121512_object130215:head# 2024-04-29T14:26:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6832c73e:::benchmark_data_smithi194_121512_object9099:head# 2024-04-29T14:26:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6834972a:::benchmark_data_smithi194_121512_object69316:head# 2024-04-29T14:26:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6834b3d1:::benchmark_data_smithi194_121512_object108869:head# 2024-04-29T14:26:37.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6834c2a4:::benchmark_data_smithi194_121512_object94664:head# 2024-04-29T14:26:37.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6834ec17:::benchmark_data_smithi194_121512_object132290:head# 2024-04-29T14:26:37.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68354138:::benchmark_data_smithi194_121512_object50472:head# 2024-04-29T14:26:37.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6835763c:::benchmark_data_smithi194_121512_object18580:head# 2024-04-29T14:26:37.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6835b8ce:::benchmark_data_smithi194_121512_object21547:head# 2024-04-29T14:26:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68382659:::benchmark_data_smithi194_121512_object86260:head# 2024-04-29T14:26:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68383b61:::benchmark_data_smithi194_121512_object68120:head# 2024-04-29T14:26:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68386f32:::benchmark_data_smithi194_121512_object70000:head# 2024-04-29T14:26:37.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6838f62b:::benchmark_data_smithi194_121512_object124891:head# 2024-04-29T14:26:37.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683980e2:::benchmark_data_smithi194_121512_object73927:head# 2024-04-29T14:26:37.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68398e1b:::benchmark_data_smithi194_121512_object33715:head# 2024-04-29T14:26:37.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683ad75d:::benchmark_data_smithi194_121512_object97318:head# 2024-04-29T14:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683af90a:::benchmark_data_smithi194_121512_object12978:head# 2024-04-29T14:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683afb71:::benchmark_data_smithi194_121512_object3654:head# 2024-04-29T14:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683b06cc:::benchmark_data_smithi194_121512_object108734:head# 2024-04-29T14:26:37.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683b4ea3:::benchmark_data_smithi194_121512_object28933:head# 2024-04-29T14:26:37.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683b5e84:::benchmark_data_smithi194_121512_object130354:head# 2024-04-29T14:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683d045e:::benchmark_data_smithi194_121512_object40536:head# 2024-04-29T14:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683d1632:::benchmark_data_smithi194_121512_object95065:head# 2024-04-29T14:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683dbb2c:::benchmark_data_smithi194_121512_object75435:head# 2024-04-29T14:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683dd8d3:::benchmark_data_smithi194_121512_object93332:head# 2024-04-29T14:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683e2ff6:::benchmark_data_smithi194_121512_object26810:head# 2024-04-29T14:26:37.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683e8d38:::benchmark_data_smithi194_121512_object57481:head# 2024-04-29T14:26:37.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683eca43:::benchmark_data_smithi194_121512_object20067:head# 2024-04-29T14:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683ed0da:::benchmark_data_smithi194_121512_object107982:head# 2024-04-29T14:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683f43d6:::benchmark_data_smithi194_121512_object122553:head# 2024-04-29T14:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683f5f12:::benchmark_data_smithi194_121512_object99253:head# 2024-04-29T14:26:37.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:683fbfeb:::benchmark_data_smithi194_121512_object15868:head# 2024-04-29T14:26:37.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68406ea1:::benchmark_data_smithi194_121512_object95736:head# 2024-04-29T14:26:37.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6840a96e:::benchmark_data_smithi194_121512_object104327:head# 2024-04-29T14:26:37.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68411213:::benchmark_data_smithi194_121512_object114436:head# 2024-04-29T14:26:37.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68418902:::benchmark_data_smithi194_121512_object109859:head# 2024-04-29T14:26:37.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6841ef50:::benchmark_data_smithi194_121512_object19748:head# 2024-04-29T14:26:37.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6842a95f:::benchmark_data_smithi194_121512_object117873:head# 2024-04-29T14:26:37.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68434604:::benchmark_data_smithi194_121512_object77458:head# 2024-04-29T14:26:37.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6843ef94:::benchmark_data_smithi194_121512_object22242:head# 2024-04-29T14:26:37.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6844a5d8:::benchmark_data_smithi194_121512_object1449:head# 2024-04-29T14:26:37.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684625f7:::benchmark_data_smithi194_121512_object123711:head# 2024-04-29T14:26:37.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68465e0b:::benchmark_data_smithi194_121512_object109198:head# 2024-04-29T14:26:37.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684699a0:::benchmark_data_smithi194_121512_object31376:head# 2024-04-29T14:26:37.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684730e1:::benchmark_data_smithi194_121512_object114545:head# 2024-04-29T14:26:37.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68476815:::benchmark_data_smithi194_121512_object131627:head# 2024-04-29T14:26:37.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684769d7:::benchmark_data_smithi194_121512_object45195:head# 2024-04-29T14:26:37.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6847d1ff:::benchmark_data_smithi194_121512_object54153:head# 2024-04-29T14:26:37.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6847d75d:::benchmark_data_smithi194_121512_object53819:head# 2024-04-29T14:26:37.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6848178c:::benchmark_data_smithi194_121512_object98816:head# 2024-04-29T14:26:37.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68486404:::benchmark_data_smithi194_121512_object66744:head# 2024-04-29T14:26:37.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6849deac:::benchmark_data_smithi194_121512_object102617:head# 2024-04-29T14:26:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:37 smithi121 ceph-mon[44533]: pgmap v1248: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 38967/247944 objects degraded (15.716%); 9.9 MiB/s, 158 objects/s recovering 2024-04-29T14:26:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:37 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3241230645' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:37 smithi121 ceph-mon[44533]: osdmap e474: 12 total, 11 up, 7 in 2024-04-29T14:26:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:37.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:37 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 38967/247944 objects degraded (15.716%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:37.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684a474e:::benchmark_data_smithi194_121512_object40237:head# 2024-04-29T14:26:37.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684ab1f1:::benchmark_data_smithi194_121512_object14375:head# 2024-04-29T14:26:37.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684b65d8:::benchmark_data_smithi194_121512_object54802:head# 2024-04-29T14:26:37.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684b7bc3:::benchmark_data_smithi194_121512_object106084:head# 2024-04-29T14:26:37.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684ba011:::benchmark_data_smithi194_121512_object116037:head# 2024-04-29T14:26:37.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684cb553:::benchmark_data_smithi194_121512_object127998:head# 2024-04-29T14:26:37.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684cbcca:::benchmark_data_smithi194_121512_object16115:head# 2024-04-29T14:26:37.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684d9aa5:::benchmark_data_smithi194_121512_object95542:head# 2024-04-29T14:26:37.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684e15dd:::benchmark_data_smithi194_121512_object35109:head# 2024-04-29T14:26:37.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684f369b:::benchmark_data_smithi194_121512_object61616:head# 2024-04-29T14:26:37.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684fa0ea:::benchmark_data_smithi194_121512_object73086:head# 2024-04-29T14:26:37.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:684fd10d:::benchmark_data_smithi194_121512_object96778:head# 2024-04-29T14:26:37.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6852a9b1:::benchmark_data_smithi194_121512_object127332:head# 2024-04-29T14:26:37.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6852cb27:::benchmark_data_smithi194_121512_object43609:head# 2024-04-29T14:26:37.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68534de7:::benchmark_data_smithi194_121512_object83063:head# 2024-04-29T14:26:37.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685483ac:::benchmark_data_smithi194_121512_object108169:head# 2024-04-29T14:26:37.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68554fd5:::benchmark_data_smithi194_121512_object117098:head# 2024-04-29T14:26:37.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6855e81d:::benchmark_data_smithi194_121512_object69199:head# 2024-04-29T14:26:37.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68568a06:::benchmark_data_smithi194_121512_object107682:head# 2024-04-29T14:26:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68575dd8:::benchmark_data_smithi194_121512_object37126:head# 2024-04-29T14:26:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685883d7:::benchmark_data_smithi194_121512_object13464:head# 2024-04-29T14:26:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6858b2d0:::benchmark_data_smithi194_121512_object75554:head# 2024-04-29T14:26:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68592a88:::benchmark_data_smithi194_121512_object123595:head# 2024-04-29T14:26:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68593140:::benchmark_data_smithi194_121512_object88558:head# 2024-04-29T14:26:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6859562c:::benchmark_data_smithi194_121512_object11947:head# 2024-04-29T14:26:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6859ac9a:::benchmark_data_smithi194_121512_object132193:head# 2024-04-29T14:26:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685a2e4a:::benchmark_data_smithi194_121512_object12111:head# 2024-04-29T14:26:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685a73d4:::benchmark_data_smithi194_121512_object2895:head# 2024-04-29T14:26:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685b72e7:::benchmark_data_smithi194_121512_object94045:head# 2024-04-29T14:26:37.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685c280f:::benchmark_data_smithi194_121512_object286:head# 2024-04-29T14:26:37.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685c40e0:::benchmark_data_smithi194_121512_object54738:head# 2024-04-29T14:26:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685ce150:::benchmark_data_smithi194_121512_object85724:head# 2024-04-29T14:26:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685dc22e:::benchmark_data_smithi194_121512_object92583:head# 2024-04-29T14:26:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685df142:::benchmark_data_smithi194_121512_object45602:head# 2024-04-29T14:26:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685dfa65:::benchmark_data_smithi194_121512_object91556:head# 2024-04-29T14:26:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685f2ad1:::benchmark_data_smithi194_121512_object65911:head# 2024-04-29T14:26:37.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685f5d44:::benchmark_data_smithi194_121512_object112314:head# 2024-04-29T14:26:37.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685f81a5:::benchmark_data_smithi194_121512_object65235:head# 2024-04-29T14:26:37.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685f99e0:::benchmark_data_smithi194_121512_object40217:head# 2024-04-29T14:26:37.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685f9e02:::benchmark_data_smithi194_121512_object94716:head# 2024-04-29T14:26:37.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685fad57:::benchmark_data_smithi194_121512_object25144:head# 2024-04-29T14:26:37.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:685fc633:::benchmark_data_smithi194_121512_object118677:head# 2024-04-29T14:26:37.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68609608:::benchmark_data_smithi194_121512_object112729:head# 2024-04-29T14:26:37.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68625ba2:::benchmark_data_smithi194_121512_object120536:head# 2024-04-29T14:26:37.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6862b3a5:::benchmark_data_smithi194_121512_object84374:head# 2024-04-29T14:26:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68630bcb:::benchmark_data_smithi194_121512_object114344:head# 2024-04-29T14:26:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6863204e:::benchmark_data_smithi194_121512_object17884:head# 2024-04-29T14:26:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6863c011:::benchmark_data_smithi194_121512_object14810:head# 2024-04-29T14:26:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68649175:::benchmark_data_smithi194_121512_object125679:head# 2024-04-29T14:26:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6864a4d5:::benchmark_data_smithi194_121512_object114968:head# 2024-04-29T14:26:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686527c2:::benchmark_data_smithi194_121512_object23716:head# 2024-04-29T14:26:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68656883:::benchmark_data_smithi194_121512_object56660:head# 2024-04-29T14:26:37.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68674516:::benchmark_data_smithi194_121512_object46694:head# 2024-04-29T14:26:37.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68675d84:::benchmark_data_smithi194_121512_object110334:head# 2024-04-29T14:26:37.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6867974a:::benchmark_data_smithi194_121512_object131833:head# 2024-04-29T14:26:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68681d00:::benchmark_data_smithi194_121512_object54224:head# 2024-04-29T14:26:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68688503:::benchmark_data_smithi194_121512_object107816:head# 2024-04-29T14:26:37.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6868cdde:::benchmark_data_smithi194_121512_object111352:head# 2024-04-29T14:26:37.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68696782:::benchmark_data_smithi194_121512_object79745:head# 2024-04-29T14:26:37.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686a378f:::benchmark_data_smithi194_121512_object46365:head# 2024-04-29T14:26:37.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686a7527:::benchmark_data_smithi194_121512_object63698:head# 2024-04-29T14:26:37.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686aeb38:::benchmark_data_smithi194_121512_object120747:head# 2024-04-29T14:26:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686af2a5:::benchmark_data_smithi194_121512_object52818:head# 2024-04-29T14:26:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686b282c:::benchmark_data_smithi194_121512_object26418:head# 2024-04-29T14:26:37.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686ca86e:::benchmark_data_smithi194_121512_object3789:head# 2024-04-29T14:26:37.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686f7587:::benchmark_data_smithi194_121512_object22515:head# 2024-04-29T14:26:37.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:686ffbf0:::benchmark_data_smithi194_121512_object115167:head# 2024-04-29T14:26:37.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687071e3:::benchmark_data_smithi194_121512_object72545:head# 2024-04-29T14:26:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687150e2:::benchmark_data_smithi194_121512_object77178:head# 2024-04-29T14:26:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6872e698:::benchmark_data_smithi194_121512_object60760:head# 2024-04-29T14:26:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687428bc:::benchmark_data_smithi194_121512_object89468:head# 2024-04-29T14:26:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6874d4e6:::benchmark_data_smithi194_121512_object6220:head# 2024-04-29T14:26:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6875ba37:::benchmark_data_smithi194_121512_object111206:head# 2024-04-29T14:26:37.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68771396:::benchmark_data_smithi194_121512_object86438:head# 2024-04-29T14:26:37.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68778f59:::benchmark_data_smithi194_121512_object39582:head# 2024-04-29T14:26:37.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6877f3c0:::benchmark_data_smithi194_121512_object118964:head# 2024-04-29T14:26:37.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68780570:::benchmark_data_smithi194_121512_object96770:head# 2024-04-29T14:26:37.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687875e6:::benchmark_data_smithi194_121512_object3103:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687888b6:::benchmark_data_smithi194_121512_object13105:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6878de26:::benchmark_data_smithi194_121512_object18174:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6879541d:::benchmark_data_smithi194_121512_object16749:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68798992:::benchmark_data_smithi194_121512_object62163:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687a1c2c:::benchmark_data_smithi194_121512_object53117:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687b2932:::benchmark_data_smithi194_121512_object15257:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687b48ad:::benchmark_data_smithi194_121512_object54769:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687b65a8:::benchmark_data_smithi194_121512_object83700:head# 2024-04-29T14:26:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687cea5f:::benchmark_data_smithi194_121512_object5305:head# 2024-04-29T14:26:37.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687cf44d:::benchmark_data_smithi194_121512_object2492:head# 2024-04-29T14:26:37.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687e1b75:::benchmark_data_smithi194_121512_object52133:head# 2024-04-29T14:26:37.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687f818d:::benchmark_data_smithi194_121512_object109560:head# 2024-04-29T14:26:37.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687fa7cb:::benchmark_data_smithi194_121512_object5034:head# 2024-04-29T14:26:37.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:687fd90e:::benchmark_data_smithi194_121512_object132921:head# 2024-04-29T14:26:37.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68814bfe:::benchmark_data_smithi194_121512_object96846:head# 2024-04-29T14:26:37.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68818aa0:::benchmark_data_smithi194_121512_object180:head# 2024-04-29T14:26:37.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688260b5:::benchmark_data_smithi194_121512_object123710:head# 2024-04-29T14:26:37.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6882ce94:::benchmark_data_smithi194_121512_object126389:head# 2024-04-29T14:26:37.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688469b8:::benchmark_data_smithi194_121512_object118560:head# 2024-04-29T14:26:37.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6884bbb7:::benchmark_data_smithi194_121512_object89971:head# 2024-04-29T14:26:37.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6884e255:::benchmark_data_smithi194_121512_object111732:head# 2024-04-29T14:26:37.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6885299f:::benchmark_data_smithi194_121512_object116032:head# 2024-04-29T14:26:37.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6885a7c8:::benchmark_data_smithi194_121512_object77835:head# 2024-04-29T14:26:37.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688628eb:::benchmark_data_smithi194_121512_object21831:head# 2024-04-29T14:26:37.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68863498:::benchmark_data_smithi194_121512_object45501:head# 2024-04-29T14:26:37.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688645d4:::benchmark_data_smithi194_121512_object99808:head# 2024-04-29T14:26:37.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68867fea:::benchmark_data_smithi194_121512_object75022:head# 2024-04-29T14:26:37.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6886c171:::benchmark_data_smithi194_121512_object56631:head# 2024-04-29T14:26:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6886d415:::benchmark_data_smithi194_121512_object104654:head# 2024-04-29T14:26:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6887a555:::benchmark_data_smithi194_121512_object24251:head# 2024-04-29T14:26:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6887b224:::benchmark_data_smithi194_121512_object130677:head# 2024-04-29T14:26:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6887c5c7:::benchmark_data_smithi194_121512_object8894:head# 2024-04-29T14:26:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6888af23:::benchmark_data_smithi194_121512_object19639:head# 2024-04-29T14:26:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68897243:::benchmark_data_smithi194_121512_object34018:head# 2024-04-29T14:26:37.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6889a867:::benchmark_data_smithi194_121512_object66542:head# 2024-04-29T14:26:37.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688a6497:::benchmark_data_smithi194_121512_object37534:head# 2024-04-29T14:26:37.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688a8bd0:::benchmark_data_smithi194_121512_object23318:head# 2024-04-29T14:26:37.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688b6893:::benchmark_data_smithi194_121512_object37398:head# 2024-04-29T14:26:37.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688b7a75:::benchmark_data_smithi194_121512_object68258:head# 2024-04-29T14:26:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688c24f6:::benchmark_data_smithi194_121512_object115916:head# 2024-04-29T14:26:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688ca76a:::benchmark_data_smithi194_121512_object115328:head# 2024-04-29T14:26:37.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688d328d:::benchmark_data_smithi194_121512_object43652:head# 2024-04-29T14:26:37.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:688ea471:::benchmark_data_smithi194_121512_object59913:head# 2024-04-29T14:26:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68900e9d:::benchmark_data_smithi194_121512_object123628:head# 2024-04-29T14:26:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68904bd2:::benchmark_data_smithi194_121512_object26774:head# 2024-04-29T14:26:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689057d9:::benchmark_data_smithi194_121512_object83140:head# 2024-04-29T14:26:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68908321:::benchmark_data_smithi194_121512_object41746:head# 2024-04-29T14:26:37.679 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6890e4d5:::benchmark_data_smithi194_121512_object52543:head# 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6891acd8:::benchmark_data_smithi194_121512_object51133:head# 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6892a1a3:::benchmark_data_smithi194_121512_object69174:head# 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68936750:::benchmark_data_smithi194_121512_object35501:head# 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68941191:::benchmark_data_smithi194_121512_object116841:head# 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68942673:::benchmark_data_smithi194_121512_object25707:head# 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68943d57:::benchmark_data_smithi194_121512_object119313:head# 2024-04-29T14:26:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6895fa44:::benchmark_data_smithi194_121512_object46272:head# 2024-04-29T14:26:37.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689651e1:::benchmark_data_smithi194_121512_object108917:head# 2024-04-29T14:26:37.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689751c6:::benchmark_data_smithi194_121512_object40102:head# 2024-04-29T14:26:37.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68978aa3:::benchmark_data_smithi194_121512_object66416:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6897fb42:::benchmark_data_smithi194_121512_object12685:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689845dc:::benchmark_data_smithi194_121512_object87665:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68986e6b:::benchmark_data_smithi194_121512_object15229:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6899bbf5:::benchmark_data_smithi194_121512_object54439:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6899f051:::benchmark_data_smithi194_121512_object46087:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689abc66:::benchmark_data_smithi194_121512_object74746:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689b1484:::benchmark_data_smithi194_121512_object68748:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689c9c6b:::benchmark_data_smithi194_121512_object88649:head# 2024-04-29T14:26:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689e3890:::benchmark_data_smithi194_121512_object131048:head# 2024-04-29T14:26:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689ea6a3:::benchmark_data_smithi194_121512_object40371:head# 2024-04-29T14:26:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689f2618:::benchmark_data_smithi194_121512_object130144:head# 2024-04-29T14:26:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:689f2e2d:::benchmark_data_smithi194_121512_object50525:head# 2024-04-29T14:26:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a06247:::benchmark_data_smithi194_121512_object47882:head# 2024-04-29T14:26:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a16e33:::benchmark_data_smithi194_121512_object32642:head# 2024-04-29T14:26:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a17fdb:::benchmark_data_smithi194_121512_object88504:head# 2024-04-29T14:26:37.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a18110:::benchmark_data_smithi194_121512_object9346:head# 2024-04-29T14:26:37.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a1944d:::benchmark_data_smithi194_121512_object118401:head# 2024-04-29T14:26:37.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a2181b:::benchmark_data_smithi194_121512_object61520:head# 2024-04-29T14:26:37.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a3039d:::benchmark_data_smithi194_121512_object45372:head# 2024-04-29T14:26:37.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a3c6fa:::benchmark_data_smithi194_121512_object71306:head# 2024-04-29T14:26:37.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a47716:::benchmark_data_smithi194_121512_object70515:head# 2024-04-29T14:26:37.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a4b425:::benchmark_data_smithi194_121512_object131685:head# 2024-04-29T14:26:37.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a4c19d:::benchmark_data_smithi194_121512_object32762:head# 2024-04-29T14:26:37.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a4efec:::benchmark_data_smithi194_121512_object104568:head# 2024-04-29T14:26:37.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a5981d:::benchmark_data_smithi194_121512_object64050:head# 2024-04-29T14:26:37.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a65a0e:::benchmark_data_smithi194_121512_object78502:head# 2024-04-29T14:26:37.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a6be7c:::benchmark_data_smithi194_121512_object106269:head# 2024-04-29T14:26:37.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a777e3:::benchmark_data_smithi194_121512_object23244:head# 2024-04-29T14:26:37.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a7cdad:::benchmark_data_smithi194_121512_object29702:head# 2024-04-29T14:26:37.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a7fc94:::benchmark_data_smithi194_121512_object81489:head# 2024-04-29T14:26:37.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a92829:::benchmark_data_smithi194_121512_object130440:head# 2024-04-29T14:26:37.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a98bcd:::benchmark_data_smithi194_121512_object37013:head# 2024-04-29T14:26:37.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68a9b406:::benchmark_data_smithi194_121512_object80865:head# 2024-04-29T14:26:37.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ab0c42:::benchmark_data_smithi194_121512_object19683:head# 2024-04-29T14:26:37.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68abb2b2:::benchmark_data_smithi194_121512_object70083:head# 2024-04-29T14:26:37.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68abb801:::benchmark_data_smithi194_121512_object90478:head# 2024-04-29T14:26:37.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ac655f:::benchmark_data_smithi194_121512_object73417:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ad355a:::benchmark_data_smithi194_121512_object73976:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ad86c5:::benchmark_data_smithi194_121512_object81960:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ad9519:::benchmark_data_smithi194_121512_object50736:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68adb090:::benchmark_data_smithi194_121512_object122494:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ae0dec:::benchmark_data_smithi194_121512_object67550:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ae155a:::benchmark_data_smithi194_121512_object26451:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ae1c9b:::benchmark_data_smithi194_121512_object630:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ae559f:::benchmark_data_smithi194_121512_object110179:head# 2024-04-29T14:26:37.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68af2824:::benchmark_data_smithi194_121512_object132728:head# 2024-04-29T14:26:37.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68afc849:::benchmark_data_smithi194_121512_object33723:head# 2024-04-29T14:26:37.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68afea97:::benchmark_data_smithi194_121512_object46508:head# 2024-04-29T14:26:37.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b03586:::benchmark_data_smithi194_121512_object82337:head# 2024-04-29T14:26:37.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b071a6:::benchmark_data_smithi194_121512_object82958:head# 2024-04-29T14:26:37.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b20215:::benchmark_data_smithi194_121512_object15403:head# 2024-04-29T14:26:37.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b41581:::benchmark_data_smithi194_121512_object16216:head# 2024-04-29T14:26:37.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b4e304:::benchmark_data_smithi194_121512_object14971:head# 2024-04-29T14:26:37.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b59181:::benchmark_data_smithi194_121512_object62998:head# 2024-04-29T14:26:37.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b70676:::benchmark_data_smithi194_121512_object93744:head# 2024-04-29T14:26:37.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b734c4:::benchmark_data_smithi194_121512_object18070:head# 2024-04-29T14:26:37.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b77483:::benchmark_data_smithi194_121512_object13326:head# 2024-04-29T14:26:37.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b7939b:::benchmark_data_smithi194_121512_object68979:head# 2024-04-29T14:26:37.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b7a954:::benchmark_data_smithi194_121512_object53460:head# 2024-04-29T14:26:37.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b7c262:::benchmark_data_smithi194_121512_object54644:head# 2024-04-29T14:26:37.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b847c6:::benchmark_data_smithi194_121512_object52326:head# 2024-04-29T14:26:37.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b8500c:::benchmark_data_smithi194_121512_object84907:head# 2024-04-29T14:26:37.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b8d92a:::benchmark_data_smithi194_121512_object106190:head# 2024-04-29T14:26:37.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b92679:::benchmark_data_smithi194_121512_object79455:head# 2024-04-29T14:26:37.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b946fe:::benchmark_data_smithi194_121512_object7444:head# 2024-04-29T14:26:37.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b94f44:::benchmark_data_smithi194_121512_object20391:head# 2024-04-29T14:26:37.712 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:37.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b962af:::benchmark_data_smithi194_121512_object77586:head# 2024-04-29T14:26:37.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68b9fe38:::benchmark_data_smithi194_121512_object71222:head# 2024-04-29T14:26:37.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ba828d:::benchmark_data_smithi194_121512_object73915:head# 2024-04-29T14:26:37.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68baab02:::benchmark_data_smithi194_121512_object26140:head# 2024-04-29T14:26:37.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bab103:::benchmark_data_smithi194_121512_object12966:head# 2024-04-29T14:26:37.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bb2d8a:::benchmark_data_smithi194_121512_object17994:head# 2024-04-29T14:26:37.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bb9cfe:::benchmark_data_smithi194_121512_object25711:head# 2024-04-29T14:26:37.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bba058:::benchmark_data_smithi194_121512_object78924:head# 2024-04-29T14:26:37.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bc2d19:::benchmark_data_smithi194_121512_object29063:head# 2024-04-29T14:26:37.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bc3a5b:::benchmark_data_smithi194_121512_object126:head# 2024-04-29T14:26:37.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bc78d3:::benchmark_data_smithi194_121512_object129531:head# 2024-04-29T14:26:37.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bcd5d9:::benchmark_data_smithi194_121512_object30667:head# 2024-04-29T14:26:37.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bd873c:::benchmark_data_smithi194_121512_object85930:head# 2024-04-29T14:26:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bdd22c:::benchmark_data_smithi194_121512_object70270:head# 2024-04-29T14:26:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68be0923:::benchmark_data_smithi194_121512_object51666:head# 2024-04-29T14:26:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68be22d1:::benchmark_data_smithi194_121512_object99648:head# 2024-04-29T14:26:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bea684:::benchmark_data_smithi194_121512_object241:head# 2024-04-29T14:26:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bf4ca5:::benchmark_data_smithi194_121512_object88745:head# 2024-04-29T14:26:37.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68bf5caa:::benchmark_data_smithi194_121512_object47165:head# 2024-04-29T14:26:37.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c0f247:::benchmark_data_smithi194_121512_object118424:head# 2024-04-29T14:26:37.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c102f6:::benchmark_data_smithi194_121512_object90681:head# 2024-04-29T14:26:37.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c14fcf:::benchmark_data_smithi194_121512_object86989:head# 2024-04-29T14:26:37.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c1a8bd:::benchmark_data_smithi194_121512_object78445:head# 2024-04-29T14:26:37.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c2077a:::benchmark_data_smithi194_121512_object64009:head# 2024-04-29T14:26:37.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c2da86:::benchmark_data_smithi194_121512_object73364:head# 2024-04-29T14:26:37.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c30a20:::benchmark_data_smithi194_121512_object54339:head# 2024-04-29T14:26:37.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c33180:::benchmark_data_smithi194_121512_object111781:head# 2024-04-29T14:26:37.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c36d22:::benchmark_data_smithi194_121512_object30535:head# 2024-04-29T14:26:37.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c391d1:::benchmark_data_smithi194_121512_object83008:head# 2024-04-29T14:26:37.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c3ef56:::benchmark_data_smithi194_121512_object131180:head# 2024-04-29T14:26:37.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c40cd9:::benchmark_data_smithi194_121512_object65326:head# 2024-04-29T14:26:37.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c42151:::benchmark_data_smithi194_121512_object31068:head# 2024-04-29T14:26:37.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c440d1:::benchmark_data_smithi194_121512_object79408:head# 2024-04-29T14:26:37.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c466eb:::benchmark_data_smithi194_121512_object41982:head# 2024-04-29T14:26:37.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c47da3:::benchmark_data_smithi194_121512_object23507:head# 2024-04-29T14:26:37.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c520d6:::benchmark_data_smithi194_121512_object132859:head# 2024-04-29T14:26:37.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c58bc9:::benchmark_data_smithi194_121512_object74854:head# 2024-04-29T14:26:37.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c5d119:::benchmark_data_smithi194_121512_object41143:head# 2024-04-29T14:26:37.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c60a4e:::benchmark_data_smithi194_121512_object31245:head# 2024-04-29T14:26:37.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c64e21:::benchmark_data_smithi194_121512_object44697:head# 2024-04-29T14:26:37.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c6a912:::benchmark_data_smithi194_121512_object52831:head# 2024-04-29T14:26:37.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c6d627:::benchmark_data_smithi194_121512_object30706:head# 2024-04-29T14:26:37.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c76dbd:::benchmark_data_smithi194_121512_object15704:head# 2024-04-29T14:26:37.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c7e414:::benchmark_data_smithi194_121512_object30092:head# 2024-04-29T14:26:37.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c9039e:::benchmark_data_smithi194_121512_object103095:head# 2024-04-29T14:26:37.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c90fbf:::benchmark_data_smithi194_121512_object79652:head# 2024-04-29T14:26:37.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68c91934:::benchmark_data_smithi194_121512_object28249:head# 2024-04-29T14:26:37.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ca03a4:::benchmark_data_smithi194_121512_object79399:head# 2024-04-29T14:26:37.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cae501:::benchmark_data_smithi194_121512_object69076:head# 2024-04-29T14:26:37.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cb0fc2:::benchmark_data_smithi194_121512_object96599:head# 2024-04-29T14:26:37.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cb6796:::benchmark_data_smithi194_121512_object43148:head# 2024-04-29T14:26:37.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cbb125:::benchmark_data_smithi194_121512_object77157:head# 2024-04-29T14:26:37.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cbb534:::benchmark_data_smithi194_121512_object20734:head# 2024-04-29T14:26:37.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cbbeb9:::benchmark_data_smithi194_121512_object42240:head# 2024-04-29T14:26:37.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ccc22a:::benchmark_data_smithi194_121512_object64160:head# 2024-04-29T14:26:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ce2444:::benchmark_data_smithi194_121512_object66136:head# 2024-04-29T14:26:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cf0f33:::benchmark_data_smithi194_121512_object119585:head# 2024-04-29T14:26:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68cf74ee:::benchmark_data_smithi194_121512_object102150:head# 2024-04-29T14:26:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d0032f:::benchmark_data_smithi194_121512_object32974:head# 2024-04-29T14:26:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d0056d:::benchmark_data_smithi194_121512_object70886:head# 2024-04-29T14:26:37.738 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:26:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d00e59:::benchmark_data_smithi194_121512_object8849:head# 2024-04-29T14:26:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d0d26c:::benchmark_data_smithi194_121512_object33814:head# 2024-04-29T14:26:37.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d10052:::benchmark_data_smithi194_121512_object3106:head# 2024-04-29T14:26:37.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d1a180:::benchmark_data_smithi194_121512_object132019:head# 2024-04-29T14:26:37.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d1e33a:::benchmark_data_smithi194_121512_object21268:head# 2024-04-29T14:26:37.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d24ec9:::benchmark_data_smithi194_121512_object55765:head# 2024-04-29T14:26:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d2a9ba:::benchmark_data_smithi194_121512_object7359:head# 2024-04-29T14:26:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d40791:::benchmark_data_smithi194_121512_object58592:head# 2024-04-29T14:26:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d48f34:::benchmark_data_smithi194_121512_object4871:head# 2024-04-29T14:26:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d4cdca:::benchmark_data_smithi194_121512_object89596:head# 2024-04-29T14:26:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d4dda8:::benchmark_data_smithi194_121512_object95540:head# 2024-04-29T14:26:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d4e674:::benchmark_data_smithi194_121512_object61811:head# 2024-04-29T14:26:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d57314:::benchmark_data_smithi194_121512_object50407:head# 2024-04-29T14:26:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d5d294:::benchmark_data_smithi194_121512_object114646:head# 2024-04-29T14:26:37.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d617b5:::benchmark_data_smithi194_121512_object77852:head# 2024-04-29T14:26:37.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d69356:::benchmark_data_smithi194_121512_object9556:head# 2024-04-29T14:26:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:37.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:26:37.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d71c22:::benchmark_data_smithi194_121512_object80108:head# 2024-04-29T14:26:37.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d752b1:::benchmark_data_smithi194_121512_object98703:head# 2024-04-29T14:26:37.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d7a580:::benchmark_data_smithi194_121512_object17990:head# 2024-04-29T14:26:37.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d84bca:::benchmark_data_smithi194_121512_object47710:head# 2024-04-29T14:26:37.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d87fc4:::benchmark_data_smithi194_121512_object66905:head# 2024-04-29T14:26:37.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d8fb91:::benchmark_data_smithi194_121512_object34329:head# 2024-04-29T14:26:37.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d942a1:::benchmark_data_smithi194_121512_object44459:head# 2024-04-29T14:26:37.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d94b9a:::benchmark_data_smithi194_121512_object60306:head# 2024-04-29T14:26:37.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68d9c386:::benchmark_data_smithi194_121512_object62120:head# 2024-04-29T14:26:37.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68da3524:::benchmark_data_smithi194_121512_object1084:head# 2024-04-29T14:26:37.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68daa5b6:::benchmark_data_smithi194_121512_object115226:head# 2024-04-29T14:26:37.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68dbf159:::benchmark_data_smithi194_121512_object131215:head# 2024-04-29T14:26:37.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68dc5b8d:::benchmark_data_smithi194_121512_object72972:head# 2024-04-29T14:26:37.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68dc6ce9:::benchmark_data_smithi194_121512_object90426:head# 2024-04-29T14:26:37.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68dd5aeb:::benchmark_data_smithi194_121512_object82136:head# 2024-04-29T14:26:37.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68df1062:::benchmark_data_smithi194_121512_object102373:head# 2024-04-29T14:26:37.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68df28da:::benchmark_data_smithi194_121512_object6263:head# 2024-04-29T14:26:37.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68dfd07b:::benchmark_data_smithi194_121512_object39964:head# 2024-04-29T14:26:37.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e0b5fd:::benchmark_data_smithi194_121512_object129443:head# 2024-04-29T14:26:37.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e10953:::benchmark_data_smithi194_121512_object125475:head# 2024-04-29T14:26:37.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e14c87:::benchmark_data_smithi194_121512_object98092:head# 2024-04-29T14:26:37.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e1b66f:::benchmark_data_smithi194_121512_object10507:head# 2024-04-29T14:26:37.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e1be6a:::benchmark_data_smithi194_121512_object60885:head# 2024-04-29T14:26:37.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e336e5:::benchmark_data_smithi194_121512_object57104:head# 2024-04-29T14:26:37.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e435bc:::benchmark_data_smithi194_121512_object16318:head# 2024-04-29T14:26:37.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e46120:::benchmark_data_smithi194_121512_object39138:head# 2024-04-29T14:26:37.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e53b7e:::benchmark_data_smithi194_121512_object35415:head# 2024-04-29T14:26:37.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e5eecf:::benchmark_data_smithi194_121512_object68677:head# 2024-04-29T14:26:37.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e66191:::benchmark_data_smithi194_121512_object35123:head# 2024-04-29T14:26:37.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e727f3:::benchmark_data_smithi194_121512_object9395:head# 2024-04-29T14:26:37.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68e926ab:::benchmark_data_smithi194_121512_object116593:head# 2024-04-29T14:26:37.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68eb6c0f:::benchmark_data_smithi194_121512_object84137:head# 2024-04-29T14:26:37.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68eb82dd:::benchmark_data_smithi194_121512_object105378:head# 2024-04-29T14:26:37.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68eca731:::benchmark_data_smithi194_121512_object65085:head# 2024-04-29T14:26:37.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ecb671:::benchmark_data_smithi194_121512_object65712:head# 2024-04-29T14:26:37.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ecd6ae:::benchmark_data_smithi194_121512_object82062:head# 2024-04-29T14:26:37.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ed4a1c:::benchmark_data_smithi194_121512_object71206:head# 2024-04-29T14:26:37.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68edceb8:::benchmark_data_smithi194_121512_object56900:head# 2024-04-29T14:26:37.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68edfc4a:::benchmark_data_smithi194_121512_object15955:head# 2024-04-29T14:26:37.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ee94e1:::benchmark_data_smithi194_121512_object120002:head# 2024-04-29T14:26:37.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68eea2a5:::benchmark_data_smithi194_121512_object125881:head# 2024-04-29T14:26:37.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ef01bd:::benchmark_data_smithi194_121512_object69179:head# 2024-04-29T14:26:37.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68efe90a:::benchmark_data_smithi194_121512_object3137:head# 2024-04-29T14:26:37.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f053af:::benchmark_data_smithi194_121512_object112074:head# 2024-04-29T14:26:37.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f07f1a:::benchmark_data_smithi194_121512_object90603:head# 2024-04-29T14:26:37.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f0ce9a:::benchmark_data_smithi194_121512_object127167:head# 2024-04-29T14:26:37.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f18104:::benchmark_data_smithi194_121512_object36105:head# 2024-04-29T14:26:37.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f3345a:::benchmark_data_smithi194_121512_object57148:head# 2024-04-29T14:26:37.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f39a4f:::benchmark_data_smithi194_121512_object131282:head# 2024-04-29T14:26:37.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f4555f:::benchmark_data_smithi194_121512_object1975:head# 2024-04-29T14:26:37.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f4a134:::benchmark_data_smithi194_121512_object59719:head# 2024-04-29T14:26:37.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f4b261:::benchmark_data_smithi194_121512_object129567:head# 2024-04-29T14:26:37.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f53bed:::benchmark_data_smithi194_121512_object35831:head# 2024-04-29T14:26:37.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f56222:::benchmark_data_smithi194_121512_object12668:head# 2024-04-29T14:26:37.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f570c3:::benchmark_data_smithi194_121512_object111926:head# 2024-04-29T14:26:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f68c02:::benchmark_data_smithi194_121512_object129493:head# 2024-04-29T14:26:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f6c4c4:::benchmark_data_smithi194_121512_object113341:head# 2024-04-29T14:26:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68f6cfb0:::benchmark_data_smithi194_121512_object54020:head# 2024-04-29T14:26:37.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fa098c:::benchmark_data_smithi194_121512_object114955:head# 2024-04-29T14:26:37.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fa802a:::benchmark_data_smithi194_121512_object99415:head# 2024-04-29T14:26:37.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68faae84:::benchmark_data_smithi194_121512_object117756:head# 2024-04-29T14:26:37.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fb0d9c:::benchmark_data_smithi194_121512_object123741:head# 2024-04-29T14:26:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fb5a56:::benchmark_data_smithi194_121512_object104554:head# 2024-04-29T14:26:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fbd2bb:::benchmark_data_smithi194_121512_object25531:head# 2024-04-29T14:26:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fc4cb3:::benchmark_data_smithi194_121512_object714:head# 2024-04-29T14:26:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fce922:::benchmark_data_smithi194_121512_object6735:head# 2024-04-29T14:26:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fd8216:::benchmark_data_smithi194_121512_object124956:head# 2024-04-29T14:26:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68fdc44d:::benchmark_data_smithi194_121512_object36207:head# 2024-04-29T14:26:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68feea94:::benchmark_data_smithi194_121512_object28511:head# 2024-04-29T14:26:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ff3c87:::benchmark_data_smithi194_121512_object55362:head# 2024-04-29T14:26:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:68ff51ef:::benchmark_data_smithi194_121512_object112428:head# 2024-04-29T14:26:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69001a60:::benchmark_data_smithi194_121512_object38757:head# 2024-04-29T14:26:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6900dec3:::benchmark_data_smithi194_121512_object13558:head# 2024-04-29T14:26:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69017e0d:::benchmark_data_smithi194_121512_object49522:head# 2024-04-29T14:26:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690276e0:::benchmark_data_smithi194_121512_object114842:head# 2024-04-29T14:26:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690284ba:::benchmark_data_smithi194_121512_object35884:head# 2024-04-29T14:26:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69034075:::benchmark_data_smithi194_121512_object93904:head# 2024-04-29T14:26:37.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69038797:::benchmark_data_smithi194_121512_object21576:head# 2024-04-29T14:26:37.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6904140f:::benchmark_data_smithi194_121512_object20349:head# 2024-04-29T14:26:37.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6905779c:::benchmark_data_smithi194_121512_object79056:head# 2024-04-29T14:26:37.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690585b0:::benchmark_data_smithi194_121512_object78106:head# 2024-04-29T14:26:37.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6905ce43:::benchmark_data_smithi194_121512_object105096:head# 2024-04-29T14:26:37.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6905f944:::benchmark_data_smithi194_121512_object91387:head# 2024-04-29T14:26:37.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69065d2b:::benchmark_data_smithi194_121512_object110137:head# 2024-04-29T14:26:37.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6908177e:::benchmark_data_smithi194_121512_object52472:head# 2024-04-29T14:26:37.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69082eab:::benchmark_data_smithi194_121512_object50426:head# 2024-04-29T14:26:37.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690843dc:::benchmark_data_smithi194_121512_object34397:head# 2024-04-29T14:26:37.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69089519:::benchmark_data_smithi194_121512_object47440:head# 2024-04-29T14:26:37.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69094b9e:::benchmark_data_smithi194_121512_object61154:head# 2024-04-29T14:26:37.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6909ece7:::benchmark_data_smithi194_121512_object30190:head# 2024-04-29T14:26:37.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690a159d:::benchmark_data_smithi194_121512_object47807:head# 2024-04-29T14:26:37.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690a305b:::benchmark_data_smithi194_121512_object44724:head# 2024-04-29T14:26:37.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690aa3b6:::benchmark_data_smithi194_121512_object83090:head# 2024-04-29T14:26:37.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690ab7b6:::benchmark_data_smithi194_121512_object7224:head# 2024-04-29T14:26:37.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690acc20:::benchmark_data_smithi194_121512_object41833:head# 2024-04-29T14:26:37.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690b3bf3:::benchmark_data_smithi194_121512_object95495:head# 2024-04-29T14:26:37.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690bfa5a:::benchmark_data_smithi194_121512_object40396:head# 2024-04-29T14:26:37.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690caa0b:::benchmark_data_smithi194_121512_object95707:head# 2024-04-29T14:26:37.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690dc4b9:::benchmark_data_smithi194_121512_object67995:head# 2024-04-29T14:26:37.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:690ff740:::benchmark_data_smithi194_121512_object50390:head# 2024-04-29T14:26:37.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69103a5f:::benchmark_data_smithi194_121512_object8992:head# 2024-04-29T14:26:37.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69109c30:::benchmark_data_smithi194_121512_object73352:head# 2024-04-29T14:26:37.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6910b00a:::benchmark_data_smithi194_121512_object53129:head# 2024-04-29T14:26:37.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6910df4f:::benchmark_data_smithi194_121512_object74427:head# 2024-04-29T14:26:37.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69115a22:::benchmark_data_smithi194_121512_object121742:head# 2024-04-29T14:26:37.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6911b1a9:::benchmark_data_smithi194_121512_object26216:head# 2024-04-29T14:26:37.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69124e2b:::benchmark_data_smithi194_121512_object24171:head# 2024-04-29T14:26:37.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69127fdc:::benchmark_data_smithi194_121512_object56264:head# 2024-04-29T14:26:37.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69129ee4:::benchmark_data_smithi194_121512_object94552:head# 2024-04-29T14:26:37.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6912cc0e:::benchmark_data_smithi194_121512_object92931:head# 2024-04-29T14:26:37.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691403f9:::benchmark_data_smithi194_121512_object117628:head# 2024-04-29T14:26:37.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691499e4:::benchmark_data_smithi194_121512_object58394:head# 2024-04-29T14:26:37.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69152a4b:::benchmark_data_smithi194_121512_object83826:head# 2024-04-29T14:26:37.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691584cd:::benchmark_data_smithi194_121512_object124128:head# 2024-04-29T14:26:37.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6916300a:::benchmark_data_smithi194_121512_object25258:head# 2024-04-29T14:26:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691640c4:::benchmark_data_smithi194_121512_object59107:head# 2024-04-29T14:26:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6916aa2a:::benchmark_data_smithi194_121512_object8808:head# 2024-04-29T14:26:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6916d263:::benchmark_data_smithi194_121512_object107145:head# 2024-04-29T14:26:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691742c7:::benchmark_data_smithi194_121512_object10806:head# 2024-04-29T14:26:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691785d6:::benchmark_data_smithi194_121512_object7058:head# 2024-04-29T14:26:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691890ee:::benchmark_data_smithi194_121512_object64767:head# 2024-04-29T14:26:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69189131:::benchmark_data_smithi194_121512_object124738:head# 2024-04-29T14:26:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69196093:::benchmark_data_smithi194_121512_object103190:head# 2024-04-29T14:26:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69199b00:::benchmark_data_smithi194_121512_object121223:head# 2024-04-29T14:26:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691a26f5:::benchmark_data_smithi194_121512_object37570:head# 2024-04-29T14:26:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691a50ec:::benchmark_data_smithi194_121512_object81798:head# 2024-04-29T14:26:37.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691a9a8f:::benchmark_data_smithi194_121512_object100574:head# 2024-04-29T14:26:37.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691abd64:::benchmark_data_smithi194_121512_object76848:head# 2024-04-29T14:26:37.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691b35b7:::benchmark_data_smithi194_121512_object130856:head# 2024-04-29T14:26:37.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691b4dd8:::benchmark_data_smithi194_121512_object125885:head# 2024-04-29T14:26:37.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691b6e8e:::benchmark_data_smithi194_121512_object26786:head# 2024-04-29T14:26:37.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691bc88a:::benchmark_data_smithi194_121512_object63704:head# 2024-04-29T14:26:37.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691bf62f:::benchmark_data_smithi194_121512_object75437:head# 2024-04-29T14:26:37.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691e4b29:::benchmark_data_smithi194_121512_object7075:head# 2024-04-29T14:26:37.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691f4cf7:::benchmark_data_smithi194_121512_object5198:head# 2024-04-29T14:26:37.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691f8ec8:::benchmark_data_smithi194_121512_object60181:head# 2024-04-29T14:26:37.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:691fd29b:::benchmark_data_smithi194_121512_object123346:head# 2024-04-29T14:26:37.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692024cf:::benchmark_data_smithi194_121512_object34325:head# 2024-04-29T14:26:37.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69208e29:::benchmark_data_smithi194_121512_object29148:head# 2024-04-29T14:26:37.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69218ae6:::benchmark_data_smithi194_121512_object64648:head# 2024-04-29T14:26:37.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6921c0f3:::benchmark_data_smithi194_121512_object18407:head# 2024-04-29T14:26:37.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6921c5b6:::benchmark_data_smithi194_121512_object92045:head# 2024-04-29T14:26:37.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6922ce70:::benchmark_data_smithi194_121512_object55439:head# 2024-04-29T14:26:37.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6922cea0:::benchmark_data_smithi194_121512_object39520:head# 2024-04-29T14:26:37.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6923df9a:::benchmark_data_smithi194_121512_object44528:head# 2024-04-29T14:26:37.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6923e58a:::benchmark_data_smithi194_121512_object80329:head# 2024-04-29T14:26:37.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6923fecd:::benchmark_data_smithi194_121512_object124818:head# 2024-04-29T14:26:37.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69243b54:::benchmark_data_smithi194_121512_object107895:head# 2024-04-29T14:26:37.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69244228:::benchmark_data_smithi194_121512_object114982:head# 2024-04-29T14:26:37.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6924eda2:::benchmark_data_smithi194_121512_object85695:head# 2024-04-29T14:26:37.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69258369:::benchmark_data_smithi194_121512_object80213:head# 2024-04-29T14:26:37.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692592d6:::benchmark_data_smithi194_121512_object87247:head# 2024-04-29T14:26:37.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69262734:::benchmark_data_smithi194_121512_object76586:head# 2024-04-29T14:26:37.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6926f24c:::benchmark_data_smithi194_121512_object50876:head# 2024-04-29T14:26:37.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6928222d:::benchmark_data_smithi194_121512_object22195:head# 2024-04-29T14:26:37.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69292386:::benchmark_data_smithi194_121512_object104358:head# 2024-04-29T14:26:37.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6929290b:::benchmark_data_smithi194_121512_object20399:head# 2024-04-29T14:26:37.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692933d7:::benchmark_data_smithi194_121512_object35957:head# 2024-04-29T14:26:37.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69295a3d:::benchmark_data_smithi194_121512_object91037:head# 2024-04-29T14:26:37.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6929d1f3:::benchmark_data_smithi194_121512_object7572:head# 2024-04-29T14:26:37.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6929d858:::benchmark_data_smithi194_121512_object65897:head# 2024-04-29T14:26:37.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692a91ab:::benchmark_data_smithi194_121512_object58622:head# 2024-04-29T14:26:37.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692ae0f6:::benchmark_data_smithi194_121512_object90307:head# 2024-04-29T14:26:37.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692b66c7:::benchmark_data_smithi194_121512_object24078:head# 2024-04-29T14:26:37.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692c995c:::benchmark_data_smithi194_121512_object85009:head# 2024-04-29T14:26:37.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692d2934:::benchmark_data_smithi194_121512_object99869:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692d5f6d:::benchmark_data_smithi194_121512_object84066:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692dcd6c:::benchmark_data_smithi194_121512_object51887:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692de5fd:::benchmark_data_smithi194_121512_object87171:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692e9c34:::benchmark_data_smithi194_121512_object6769:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692e9e12:::benchmark_data_smithi194_121512_object100913:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692f2065:::benchmark_data_smithi194_121512_object13561:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692f7323:::benchmark_data_smithi194_121512_object74019:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:692fec75:::benchmark_data_smithi194_121512_object118920:head# 2024-04-29T14:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6931d096:::benchmark_data_smithi194_121512_object118583:head# 2024-04-29T14:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69332fb4:::benchmark_data_smithi194_121512_object108950:head# 2024-04-29T14:26:37.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693331cb:::benchmark_data_smithi194_121512_object46770:head# 2024-04-29T14:26:37.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693384cc:::benchmark_data_smithi194_121512_object64793:head# 2024-04-29T14:26:37.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693592d5:::benchmark_data_smithi194_121512_object48917:head# 2024-04-29T14:26:37.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693674c1:::benchmark_data_smithi194_121512_object61402:head# 2024-04-29T14:26:37.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69367821:::benchmark_data_smithi194_121512_object4101:head# 2024-04-29T14:26:37.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6936851f:::benchmark_data_smithi194_121512_object24700:head# 2024-04-29T14:26:37.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69373abe:::benchmark_data_smithi194_121512_object8092:head# 2024-04-29T14:26:37.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6937d423:::benchmark_data_smithi194_121512_object18397:head# 2024-04-29T14:26:37.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6939262e:::benchmark_data_smithi194_121512_object83164:head# 2024-04-29T14:26:37.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693a59ad:::benchmark_data_smithi194_121512_object6164:head# 2024-04-29T14:26:37.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693a65dd:::benchmark_data_smithi194_121512_object123141:head# 2024-04-29T14:26:37.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693a819f:::benchmark_data_smithi194_121512_object85907:head# 2024-04-29T14:26:37.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693abd3f:::benchmark_data_smithi194_121512_object26024:head# 2024-04-29T14:26:37.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693b10d9:::benchmark_data_smithi194_121512_object30959:head# 2024-04-29T14:26:37.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693b305d:::benchmark_data_smithi194_121512_object132634:head# 2024-04-29T14:26:37.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693b8044:::benchmark_data_smithi194_121512_object129914:head# 2024-04-29T14:26:37.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693bd5a4:::benchmark_data_smithi194_121512_object78469:head# 2024-04-29T14:26:37.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693c2c14:::benchmark_data_smithi194_121512_object54634:head# 2024-04-29T14:26:37.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693cfa64:::benchmark_data_smithi194_121512_object1536:head# 2024-04-29T14:26:37.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693dbec7:::benchmark_data_smithi194_121512_object56538:head# 2024-04-29T14:26:37.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693dd690:::benchmark_data_smithi194_121512_object77664:head# 2024-04-29T14:26:37.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693e3dbe:::benchmark_data_smithi194_121512_object130310:head# 2024-04-29T14:26:37.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693f2bea:::benchmark_data_smithi194_121512_object9079:head# 2024-04-29T14:26:37.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:693fc7fc:::benchmark_data_smithi194_121512_object83336:head# 2024-04-29T14:26:37.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694044a3:::benchmark_data_smithi194_121512_object58324:head# 2024-04-29T14:26:37.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69414b34:::benchmark_data_smithi194_121512_object63623:head# 2024-04-29T14:26:37.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69415cbc:::benchmark_data_smithi194_121512_object106007:head# 2024-04-29T14:26:37.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6942952f:::benchmark_data_smithi194_121512_object94150:head# 2024-04-29T14:26:37.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694331a5:::benchmark_data_smithi194_121512_object35064:head# 2024-04-29T14:26:37.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6943bf4a:::benchmark_data_smithi194_121512_object116080:head# 2024-04-29T14:26:37.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6944797d:::benchmark_data_smithi194_121512_object95218:head# 2024-04-29T14:26:37.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69448311:::benchmark_data_smithi194_121512_object36333:head# 2024-04-29T14:26:37.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6944f3b9:::benchmark_data_smithi194_121512_object54121:head# 2024-04-29T14:26:37.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6945031e:::benchmark_data_smithi194_121512_object81709:head# 2024-04-29T14:26:37.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694577db:::benchmark_data_smithi194_121512_object18216:head# 2024-04-29T14:26:37.845 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:26:37.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69457d55:::benchmark_data_smithi194_121512_object89955:head# 2024-04-29T14:26:37.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6945d7cf:::benchmark_data_smithi194_121512_object27759:head# 2024-04-29T14:26:37.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69476784:::benchmark_data_smithi194_121512_object35333:head# 2024-04-29T14:26:37.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694788c2:::benchmark_data_smithi194_121512_object65813:head# 2024-04-29T14:26:37.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69485e5e:::benchmark_data_smithi194_121512_object16282:head# 2024-04-29T14:26:37.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6949b653:::benchmark_data_smithi194_121512_object39877:head# 2024-04-29T14:26:37.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6949f60c:::benchmark_data_smithi194_121512_object126834:head# 2024-04-29T14:26:37.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6949f81d:::benchmark_data_smithi194_121512_object3618:head# 2024-04-29T14:26:37.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694ac996:::benchmark_data_smithi194_121512_object74599:head# 2024-04-29T14:26:37.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694ca331:::benchmark_data_smithi194_121512_object32532:head# 2024-04-29T14:26:37.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694ca66a:::benchmark_data_smithi194_121512_object130636:head# 2024-04-29T14:26:37.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694ca7fc:::benchmark_data_smithi194_121512_object61110:head# 2024-04-29T14:26:37.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694cca91:::benchmark_data_smithi194_121512_object51397:head# 2024-04-29T14:26:37.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694e874e:::benchmark_data_smithi194_121512_object72436:head# 2024-04-29T14:26:37.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694f71ab:::benchmark_data_smithi194_121512_object65540:head# 2024-04-29T14:26:37.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694f9913:::benchmark_data_smithi194_121512_object6975:head# 2024-04-29T14:26:37.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:694ffa4b:::benchmark_data_smithi194_121512_object60974:head# 2024-04-29T14:26:37.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69500312:::benchmark_data_smithi194_121512_object93152:head# 2024-04-29T14:26:37.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69520ec7:::benchmark_data_smithi194_121512_object113411:head# 2024-04-29T14:26:37.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69525f57:::benchmark_data_smithi194_121512_object111233:head# 2024-04-29T14:26:37.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6952928e:::benchmark_data_smithi194_121512_object73410:head# 2024-04-29T14:26:37.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69529d4f:::benchmark_data_smithi194_121512_object26402:head# 2024-04-29T14:26:37.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69531ddb:::benchmark_data_smithi194_121512_object88583:head# 2024-04-29T14:26:37.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69531ff9:::benchmark_data_smithi194_121512_object64359:head# 2024-04-29T14:26:37.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6953a3d4:::benchmark_data_smithi194_121512_object52051:head# 2024-04-29T14:26:37.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69542048:::benchmark_data_smithi194_121512_object56588:head# 2024-04-29T14:26:37.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69545da2:::benchmark_data_smithi194_121512_object109910:head# 2024-04-29T14:26:37.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69545dd2:::benchmark_data_smithi194_121512_object123150:head# 2024-04-29T14:26:37.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6954c398:::benchmark_data_smithi194_121512_object101565:head# 2024-04-29T14:26:37.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6955483b:::benchmark_data_smithi194_121512_object73367:head# 2024-04-29T14:26:37.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69556a97:::benchmark_data_smithi194_121512_object47712:head# 2024-04-29T14:26:37.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6955aad1:::benchmark_data_smithi194_121512_object32906:head# 2024-04-29T14:26:37.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6955ec0e:::benchmark_data_smithi194_121512_object116100:head# 2024-04-29T14:26:37.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695633d5:::benchmark_data_smithi194_121512_object50404:head# 2024-04-29T14:26:37.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69565fbe:::benchmark_data_smithi194_121512_object56008:head# 2024-04-29T14:26:37.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69568dfb:::benchmark_data_smithi194_121512_object125471:head# 2024-04-29T14:26:37.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6956ade1:::benchmark_data_smithi194_121512_object11661:head# 2024-04-29T14:26:37.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69571020:::benchmark_data_smithi194_121512_object69428:head# 2024-04-29T14:26:37.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69574160:::benchmark_data_smithi194_121512_object66377:head# 2024-04-29T14:26:37.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69574cf7:::benchmark_data_smithi194_121512_object38482:head# 2024-04-29T14:26:37.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6957544c:::benchmark_data_smithi194_121512_object61485:head# 2024-04-29T14:26:37.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6957c07c:::benchmark_data_smithi194_121512_object59750:head# 2024-04-29T14:26:37.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6957fbf7:::benchmark_data_smithi194_121512_object60578:head# 2024-04-29T14:26:37.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695829a3:::benchmark_data_smithi194_121512_object84288:head# 2024-04-29T14:26:37.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6958f6a1:::benchmark_data_smithi194_121512_object108541:head# 2024-04-29T14:26:37.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69595938:::benchmark_data_smithi194_121512_object36677:head# 2024-04-29T14:26:37.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69598030:::benchmark_data_smithi194_121512_object58498:head# 2024-04-29T14:26:37.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69599895:::benchmark_data_smithi194_121512_object66567:head# 2024-04-29T14:26:37.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695a4854:::benchmark_data_smithi194_121512_object124437:head# 2024-04-29T14:26:37.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695a4acb:::benchmark_data_smithi194_121512_object130248:head# 2024-04-29T14:26:37.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695aa78e:::benchmark_data_smithi194_121512_object130356:head# 2024-04-29T14:26:37.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695b12db:::benchmark_data_smithi194_121512_object73594:head# 2024-04-29T14:26:37.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695c8539:::benchmark_data_smithi194_121512_object44636:head# 2024-04-29T14:26:37.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695ccb7a:::benchmark_data_smithi194_121512_object93506:head# 2024-04-29T14:26:37.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695ede6b:::benchmark_data_smithi194_121512_object36283:head# 2024-04-29T14:26:37.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695f3ead:::benchmark_data_smithi194_121512_object112458:head# 2024-04-29T14:26:37.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695f49bf:::benchmark_data_smithi194_121512_object38264:head# 2024-04-29T14:26:37.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695f6816:::benchmark_data_smithi194_121512_object120111:head# 2024-04-29T14:26:37.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695f8308:::benchmark_data_smithi194_121512_object55379:head# 2024-04-29T14:26:37.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:695fae80:::benchmark_data_smithi194_121512_object4778:head# 2024-04-29T14:26:37.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6960f89c:::benchmark_data_smithi194_121512_object49154:head# 2024-04-29T14:26:37.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6961943b:::benchmark_data_smithi194_121512_object6599:head# 2024-04-29T14:26:37.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6961a948:::benchmark_data_smithi194_121512_object112367:head# 2024-04-29T14:26:37.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69623feb:::benchmark_data_smithi194_121512_object29466:head# 2024-04-29T14:26:37.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6962aed6:::benchmark_data_smithi194_121512_object96627:head# 2024-04-29T14:26:37.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69630bab:::benchmark_data_smithi194_121512_object127326:head# 2024-04-29T14:26:37.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69632fe6:::benchmark_data_smithi194_121512_object112759:head# 2024-04-29T14:26:37.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69633f30:::benchmark_data_smithi194_121512_object121878:head# 2024-04-29T14:26:37.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69634cfc:::benchmark_data_smithi194_121512_object124701:head# 2024-04-29T14:26:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696372d2:::benchmark_data_smithi194_121512_object25901:head# 2024-04-29T14:26:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6963a59e:::benchmark_data_smithi194_121512_object31111:head# 2024-04-29T14:26:37.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6963a765:::benchmark_data_smithi194_121512_object129266:head# 2024-04-29T14:26:37.897 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6963b926:::benchmark_data_smithi194_121512_object86993:head# 2024-04-29T14:26:37.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6963d1e3:::benchmark_data_smithi194_121512_object130075:head# 2024-04-29T14:26:37.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69641a84:::benchmark_data_smithi194_121512_object102312:head# 2024-04-29T14:26:37.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69641ffd:::benchmark_data_smithi194_121512_object122440:head# 2024-04-29T14:26:37.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6964228b:::benchmark_data_smithi194_121512_object73201:head# 2024-04-29T14:26:37.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69661f4e:::benchmark_data_smithi194_121512_object118399:head# 2024-04-29T14:26:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6966b985:::benchmark_data_smithi194_121512_object117669:head# 2024-04-29T14:26:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69670bcd:::benchmark_data_smithi194_121512_object8163:head# 2024-04-29T14:26:37.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69696407:::benchmark_data_smithi194_121512_object89606:head# 2024-04-29T14:26:37.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69697615:::benchmark_data_smithi194_121512_object2488:head# 2024-04-29T14:26:37.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69699617:::benchmark_data_smithi194_121512_object52465:head# 2024-04-29T14:26:37.902 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:37.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6969cfaf:::benchmark_data_smithi194_121512_object28307:head# 2024-04-29T14:26:37.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696a2204:::benchmark_data_smithi194_121512_object41795:head# 2024-04-29T14:26:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696a4a42:::benchmark_data_smithi194_121512_object79983:head# 2024-04-29T14:26:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696bc2f8:::benchmark_data_smithi194_121512_object105301:head# 2024-04-29T14:26:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696c2749:::benchmark_data_smithi194_121512_object92503:head# 2024-04-29T14:26:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696c82aa:::benchmark_data_smithi194_121512_object117185:head# 2024-04-29T14:26:37.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696e110f:::benchmark_data_smithi194_121512_object18116:head# 2024-04-29T14:26:37.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696e210f:::benchmark_data_smithi194_121512_object97636:head# 2024-04-29T14:26:37.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696e3055:::benchmark_data_smithi194_121512_object98115:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696e9759:::benchmark_data_smithi194_121512_object60070:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696ea061:::benchmark_data_smithi194_121512_object53850:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696ea568:::benchmark_data_smithi194_121512_object57182:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:696eee2f:::benchmark_data_smithi194_121512_object526:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697029c0:::benchmark_data_smithi194_121512_object99346:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69708bf1:::benchmark_data_smithi194_121512_object16562:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69718488:::benchmark_data_smithi194_121512_object94619:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69721609:::benchmark_data_smithi194_121512_object63011:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69737bf8:::benchmark_data_smithi194_121512_object125136:head# 2024-04-29T14:26:37.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697395d2:::benchmark_data_smithi194_121512_object75396:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6974330e:::benchmark_data_smithi194_121512_object62037:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697521e5:::benchmark_data_smithi194_121512_object122326:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6975b36f:::benchmark_data_smithi194_121512_object21367:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69767ca2:::benchmark_data_smithi194_121512_object5502:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6976be60:::benchmark_data_smithi194_121512_object42503:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6976c25a:::benchmark_data_smithi194_121512_object65618:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6976fe3d:::benchmark_data_smithi194_121512_object97672:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6976ff93:::benchmark_data_smithi194_121512_object90916:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697714c6:::benchmark_data_smithi194_121512_object23309:head# 2024-04-29T14:26:37.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69772412:::benchmark_data_smithi194_121512_object53977:head# 2024-04-29T14:26:37.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69774739:::benchmark_data_smithi194_121512_object54200:head# 2024-04-29T14:26:37.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6977921b:::benchmark_data_smithi194_121512_object67999:head# 2024-04-29T14:26:37.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69789617:::benchmark_data_smithi194_121512_object129804:head# 2024-04-29T14:26:37.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697907bc:::benchmark_data_smithi194_121512_object133073:head# 2024-04-29T14:26:37.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69793aa6:::benchmark_data_smithi194_121512_object119265:head# 2024-04-29T14:26:37.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6979ca15:::benchmark_data_smithi194_121512_object59960:head# 2024-04-29T14:26:37.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6979df24:::benchmark_data_smithi194_121512_object9325:head# 2024-04-29T14:26:37.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697a030c:::benchmark_data_smithi194_121512_object825:head# 2024-04-29T14:26:37.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697a6bcc:::benchmark_data_smithi194_121512_object110500:head# 2024-04-29T14:26:37.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697a82c8:::benchmark_data_smithi194_121512_object83985:head# 2024-04-29T14:26:37.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697b56cc:::benchmark_data_smithi194_121512_object72187:head# 2024-04-29T14:26:37.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697dac31:::benchmark_data_smithi194_121512_object30940:head# 2024-04-29T14:26:37.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697e7004:::benchmark_data_smithi194_121512_object17666:head# 2024-04-29T14:26:37.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697ea5f0:::benchmark_data_smithi194_121512_object26999:head# 2024-04-29T14:26:37.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:697ed2f6:::benchmark_data_smithi194_121512_object108018:head# 2024-04-29T14:26:37.930 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:26:37.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698022f3:::benchmark_data_smithi194_121512_object94872:head# 2024-04-29T14:26:37.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698031a5:::benchmark_data_smithi194_121512_object28225:head# 2024-04-29T14:26:37.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69805171:::benchmark_data_smithi194_121512_object33243:head# 2024-04-29T14:26:37.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6980cc12:::benchmark_data_smithi194_121512_object11583:head# 2024-04-29T14:26:37.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69812a5c:::benchmark_data_smithi194_121512_object96110:head# 2024-04-29T14:26:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:37.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:26:37.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69817ab6:::benchmark_data_smithi194_121512_object85166:head# 2024-04-29T14:26:37.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698194c2:::benchmark_data_smithi194_121512_object56412:head# 2024-04-29T14:26:37.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6982adff:::benchmark_data_smithi194_121512_object67715:head# 2024-04-29T14:26:37.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69836ff4:::benchmark_data_smithi194_121512_object76834:head# 2024-04-29T14:26:37.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698384bc:::benchmark_data_smithi194_121512_object9970:head# 2024-04-29T14:26:37.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6983fd28:::benchmark_data_smithi194_121512_object49624:head# 2024-04-29T14:26:37.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69866302:::benchmark_data_smithi194_121512_object33762:head# 2024-04-29T14:26:37.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69871372:::benchmark_data_smithi194_121512_object74028:head# 2024-04-29T14:26:37.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69873246:::benchmark_data_smithi194_121512_object105981:head# 2024-04-29T14:26:37.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6987b634:::benchmark_data_smithi194_121512_object117630:head# 2024-04-29T14:26:37.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6987c180:::benchmark_data_smithi194_121512_object75686:head# 2024-04-29T14:26:37.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6987d83e:::benchmark_data_smithi194_121512_object115932:head# 2024-04-29T14:26:37.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6987f261:::benchmark_data_smithi194_121512_object42556:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698887c3:::benchmark_data_smithi194_121512_object7721:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6988fa1a:::benchmark_data_smithi194_121512_object131721:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69890dbc:::benchmark_data_smithi194_121512_object45509:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69891577:::benchmark_data_smithi194_121512_object452:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69899eba:::benchmark_data_smithi194_121512_object50177:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698a67ea:::benchmark_data_smithi194_121512_object112014:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698aa001:::benchmark_data_smithi194_121512_object111765:head# 2024-04-29T14:26:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698acf34:::benchmark_data_smithi194_121512_object105690:head# 2024-04-29T14:26:37.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698af620:::benchmark_data_smithi194_121512_object7101:head# 2024-04-29T14:26:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698b9987:::benchmark_data_smithi194_121512_object31670:head# 2024-04-29T14:26:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698c4e47:::benchmark_data_smithi194_121512_object132958:head# 2024-04-29T14:26:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698cc5e0:::benchmark_data_smithi194_121512_object16552:head# 2024-04-29T14:26:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698d125c:::benchmark_data_smithi194_121512_object70504:head# 2024-04-29T14:26:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698d2dc4:::benchmark_data_smithi194_121512_object104776:head# 2024-04-29T14:26:37.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698d449b:::benchmark_data_smithi194_121512_object36607:head# 2024-04-29T14:26:37.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698e0216:::benchmark_data_smithi194_121512_object60700:head# 2024-04-29T14:26:37.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698e34d8:::benchmark_data_smithi194_121512_object82220:head# 2024-04-29T14:26:37.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698e6c94:::benchmark_data_smithi194_121512_object117983:head# 2024-04-29T14:26:37.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698e979c:::benchmark_data_smithi194_121512_object99790:head# 2024-04-29T14:26:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698f11ba:::benchmark_data_smithi194_121512_object86297:head# 2024-04-29T14:26:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:698f9491:::benchmark_data_smithi194_121512_object74710:head# 2024-04-29T14:26:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69906707:::benchmark_data_smithi194_121512_object65908:head# 2024-04-29T14:26:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6990b75c:::benchmark_data_smithi194_121512_object2698:head# 2024-04-29T14:26:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6990fccb:::benchmark_data_smithi194_121512_object109142:head# 2024-04-29T14:26:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699135cb:::benchmark_data_smithi194_121512_object12297:head# 2024-04-29T14:26:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69914c13:::benchmark_data_smithi194_121512_object75708:head# 2024-04-29T14:26:37.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69914f3a:::benchmark_data_smithi194_121512_object55284:head# 2024-04-29T14:26:37.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69917b1b:::benchmark_data_smithi194_121512_object109743:head# 2024-04-29T14:26:37.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69926b27:::benchmark_data_smithi194_121512_object82674:head# 2024-04-29T14:26:37.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699337ad:::benchmark_data_smithi194_121512_object6056:head# 2024-04-29T14:26:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6993a9fa:::benchmark_data_smithi194_121512_object49701:head# 2024-04-29T14:26:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6993b6ef:::benchmark_data_smithi194_121512_object111338:head# 2024-04-29T14:26:37.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6993c02e:::benchmark_data_smithi194_121512_object7002:head# 2024-04-29T14:26:37.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6993e684:::benchmark_data_smithi194_121512_object74376:head# 2024-04-29T14:26:37.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69945106:::benchmark_data_smithi194_121512_object50853:head# 2024-04-29T14:26:37.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69946aba:::benchmark_data_smithi194_121512_object49325:head# 2024-04-29T14:26:37.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6994bc93:::benchmark_data_smithi194_121512_object96656:head# 2024-04-29T14:26:37.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6994da69:::benchmark_data_smithi194_121512_object3931:head# 2024-04-29T14:26:37.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69950bb0:::benchmark_data_smithi194_121512_object122751:head# 2024-04-29T14:26:37.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6995d3ba:::benchmark_data_smithi194_121512_object51928:head# 2024-04-29T14:26:37.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6996773a:::benchmark_data_smithi194_121512_object93627:head# 2024-04-29T14:26:37.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6996a0b1:::benchmark_data_smithi194_121512_object89478:head# 2024-04-29T14:26:37.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6996af80:::benchmark_data_smithi194_121512_object14947:head# 2024-04-29T14:26:37.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6996b2c8:::benchmark_data_smithi194_121512_object123198:head# 2024-04-29T14:26:37.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699760c2:::benchmark_data_smithi194_121512_object46322:head# 2024-04-29T14:26:37.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6997f1c4:::benchmark_data_smithi194_121512_object28171:head# 2024-04-29T14:26:37.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699818f6:::benchmark_data_smithi194_121512_object89536:head# 2024-04-29T14:26:37.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699825ba:::benchmark_data_smithi194_121512_object11717:head# 2024-04-29T14:26:37.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6998e8bd:::benchmark_data_smithi194_121512_object68006:head# 2024-04-29T14:26:37.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6999d818:::benchmark_data_smithi194_121512_object117072:head# 2024-04-29T14:26:37.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699a14a8:::benchmark_data_smithi194_121512_object88581:head# 2024-04-29T14:26:37.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699adf6c:::benchmark_data_smithi194_121512_object38980:head# 2024-04-29T14:26:37.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699af20c:::benchmark_data_smithi194_121512_object61387:head# 2024-04-29T14:26:37.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699b9d9a:::benchmark_data_smithi194_121512_object32904:head# 2024-04-29T14:26:37.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699c323a:::benchmark_data_smithi194_121512_object37100:head# 2024-04-29T14:26:37.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699cc2f9:::benchmark_data_smithi194_121512_object35767:head# 2024-04-29T14:26:37.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699ce121:::benchmark_data_smithi194_121512_object93729:head# 2024-04-29T14:26:37.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699d4f0c:::benchmark_data_smithi194_121512_object23809:head# 2024-04-29T14:26:37.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699e0e94:::benchmark_data_smithi194_121512_object84656:head# 2024-04-29T14:26:37.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699e37e5:::benchmark_data_smithi194_121512_object105936:head# 2024-04-29T14:26:37.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:699f9f7f:::benchmark_data_smithi194_121512_object117973:head# 2024-04-29T14:26:37.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a0101d:::benchmark_data_smithi194_121512_object86062:head# 2024-04-29T14:26:37.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a092a8:::benchmark_data_smithi194_121512_object23520:head# 2024-04-29T14:26:37.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a168ef:::benchmark_data_smithi194_121512_object15120:head# 2024-04-29T14:26:37.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a1c52c:::benchmark_data_smithi194_121512_object37634:head# 2024-04-29T14:26:37.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a1c99b:::benchmark_data_smithi194_121512_object95557:head# 2024-04-29T14:26:37.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a2416c:::benchmark_data_smithi194_121512_object43697:head# 2024-04-29T14:26:37.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a30e56:::benchmark_data_smithi194_121512_object36621:head# 2024-04-29T14:26:37.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a3c6a2:::benchmark_data_smithi194_121512_object34922:head# 2024-04-29T14:26:37.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a4b8b4:::benchmark_data_smithi194_121512_object72913:head# 2024-04-29T14:26:37.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a6e256:::benchmark_data_smithi194_121512_object107508:head# 2024-04-29T14:26:37.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a6ed90:::benchmark_data_smithi194_121512_object131580:head# 2024-04-29T14:26:37.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a71c0d:::benchmark_data_smithi194_121512_object42619:head# 2024-04-29T14:26:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a722f7:::benchmark_data_smithi194_121512_object24530:head# 2024-04-29T14:26:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a73abe:::benchmark_data_smithi194_121512_object92779:head# 2024-04-29T14:26:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a74302:::benchmark_data_smithi194_121512_object729:head# 2024-04-29T14:26:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a88451:::benchmark_data_smithi194_121512_object40688:head# 2024-04-29T14:26:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69a9eaba:::benchmark_data_smithi194_121512_object124353:head# 2024-04-29T14:26:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ab3004:::benchmark_data_smithi194_121512_object116728:head# 2024-04-29T14:26:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69aba0a6:::benchmark_data_smithi194_121512_object98178:head# 2024-04-29T14:26:38.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ac2807:::benchmark_data_smithi194_121512_object56982:head# 2024-04-29T14:26:38.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ac2b8c:::benchmark_data_smithi194_121512_object113466:head# 2024-04-29T14:26:38.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69acb25c:::benchmark_data_smithi194_121512_object15373:head# 2024-04-29T14:26:38.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ad032c:::benchmark_data_smithi194_121512_object86077:head# 2024-04-29T14:26:38.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69adc710:::benchmark_data_smithi194_121512_object115619:head# 2024-04-29T14:26:38.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ae8aff:::benchmark_data_smithi194_121512_object51359:head# 2024-04-29T14:26:38.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69af2ee1:::benchmark_data_smithi194_121512_object94562:head# 2024-04-29T14:26:38.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69af3176:::benchmark_data_smithi194_121512_object127804:head# 2024-04-29T14:26:38.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b16514:::benchmark_data_smithi194_121512_object98339:head# 2024-04-29T14:26:38.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b178fd:::benchmark_data_smithi194_121512_object127974:head# 2024-04-29T14:26:38.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b1907f:::benchmark_data_smithi194_121512_object82740:head# 2024-04-29T14:26:38.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b1afa8:::benchmark_data_smithi194_121512_object25875:head# 2024-04-29T14:26:38.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b23880:::benchmark_data_smithi194_121512_object25362:head# 2024-04-29T14:26:38.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b2ee3d:::benchmark_data_smithi194_121512_object70875:head# 2024-04-29T14:26:38.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b346df:::benchmark_data_smithi194_121512_object36416:head# 2024-04-29T14:26:38.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b38841:::benchmark_data_smithi194_121512_object32965:head# 2024-04-29T14:26:38.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b40d0d:::benchmark_data_smithi194_121512_object77397:head# 2024-04-29T14:26:38.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b4602a:::benchmark_data_smithi194_121512_object118614:head# 2024-04-29T14:26:38.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b4af60:::benchmark_data_smithi194_121512_object438:head# 2024-04-29T14:26:38.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b4b345:::benchmark_data_smithi194_121512_object92464:head# 2024-04-29T14:26:38.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b4ca55:::benchmark_data_smithi194_121512_object45694:head# 2024-04-29T14:26:38.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b51401:::benchmark_data_smithi194_121512_object6238:head# 2024-04-29T14:26:38.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b723bc:::benchmark_data_smithi194_121512_object19056:head# 2024-04-29T14:26:38.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b7601e:::benchmark_data_smithi194_121512_object40574:head# 2024-04-29T14:26:38.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b7ac6d:::benchmark_data_smithi194_121512_object112622:head# 2024-04-29T14:26:38.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b83f92:::benchmark_data_smithi194_121512_object112026:head# 2024-04-29T14:26:38.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b855a6:::benchmark_data_smithi194_121512_object88139:head# 2024-04-29T14:26:38.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b9938f:::benchmark_data_smithi194_121512_object107950:head# 2024-04-29T14:26:38.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69b9fecc:::benchmark_data_smithi194_121512_object22211:head# 2024-04-29T14:26:38.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69baece7:::benchmark_data_smithi194_121512_object21293:head# 2024-04-29T14:26:38.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69bc8e6d:::benchmark_data_smithi194_121512_object4857:head# 2024-04-29T14:26:38.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69bcaf13:::benchmark_data_smithi194_121512_object112457:head# 2024-04-29T14:26:38.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69bcfac3:::benchmark_data_smithi194_121512_object59391:head# 2024-04-29T14:26:38.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69bd1c38:::benchmark_data_smithi194_121512_object48396:head# 2024-04-29T14:26:38.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69bd76da:::benchmark_data_smithi194_121512_object41829:head# 2024-04-29T14:26:38.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69bf8745:::benchmark_data_smithi194_121512_object112231:head# 2024-04-29T14:26:38.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c01c03:::benchmark_data_smithi194_121512_object53226:head# 2024-04-29T14:26:38.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c0230c:::benchmark_data_smithi194_121512_object86475:head# 2024-04-29T14:26:38.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c10f7f:::benchmark_data_smithi194_121512_object81881:head# 2024-04-29T14:26:38.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c1533c:::benchmark_data_smithi194_121512_object10319:head# 2024-04-29T14:26:38.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c16279:::benchmark_data_smithi194_121512_object28087:head# 2024-04-29T14:26:38.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c18374:::benchmark_data_smithi194_121512_object129781:head# 2024-04-29T14:26:38.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c29348:::benchmark_data_smithi194_121512_object51914:head# 2024-04-29T14:26:38.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c2f0d5:::benchmark_data_smithi194_121512_object79820:head# 2024-04-29T14:26:38.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c3d1a7:::benchmark_data_smithi194_121512_object43642:head# 2024-04-29T14:26:38.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c3d83f:::benchmark_data_smithi194_121512_object51268:head# 2024-04-29T14:26:38.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c40f8c:::benchmark_data_smithi194_121512_object17030:head# 2024-04-29T14:26:38.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c45bbf:::benchmark_data_smithi194_121512_object63287:head# 2024-04-29T14:26:38.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c6e6c7:::benchmark_data_smithi194_121512_object35469:head# 2024-04-29T14:26:38.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c6fd98:::benchmark_data_smithi194_121512_object97484:head# 2024-04-29T14:26:38.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c7657a:::benchmark_data_smithi194_121512_object111738:head# 2024-04-29T14:26:38.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c795e7:::benchmark_data_smithi194_121512_object57125:head# 2024-04-29T14:26:38.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c8ee1e:::benchmark_data_smithi194_121512_object127668:head# 2024-04-29T14:26:38.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c93eb1:::benchmark_data_smithi194_121512_object112783:head# 2024-04-29T14:26:38.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c9478f:::benchmark_data_smithi194_121512_object57459:head# 2024-04-29T14:26:38.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c9884e:::benchmark_data_smithi194_121512_object87022:head# 2024-04-29T14:26:38.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69c9ed4c:::benchmark_data_smithi194_121512_object53772:head# 2024-04-29T14:26:38.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ca2bd4:::benchmark_data_smithi194_121512_object73613:head# 2024-04-29T14:26:38.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:26:38.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cb86f1:::benchmark_data_smithi194_121512_object107108:head# 2024-04-29T14:26:38.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cbf7af:::benchmark_data_smithi194_121512_object13253:head# 2024-04-29T14:26:38.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cc6bcb:::benchmark_data_smithi194_121512_object67471:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cca570:::benchmark_data_smithi194_121512_object8493:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cd8e47:::benchmark_data_smithi194_121512_object49586:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cd8f97:::benchmark_data_smithi194_121512_object122498:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cde3fa:::benchmark_data_smithi194_121512_object52167:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ce3c67:::benchmark_data_smithi194_121512_object130232:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ce5d7b:::benchmark_data_smithi194_121512_object118502:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ce7180:::benchmark_data_smithi194_121512_object105488:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ce922c:::benchmark_data_smithi194_121512_object24359:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69cfe0a1:::benchmark_data_smithi194_121512_object124924:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d1a088:::benchmark_data_smithi194_121512_object107205:head# 2024-04-29T14:26:38.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d1dc70:::benchmark_data_smithi194_121512_object18654:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d1e390:::benchmark_data_smithi194_121512_object64942:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d240b0:::benchmark_data_smithi194_121512_object79824:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d24d05:::benchmark_data_smithi194_121512_object107677:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d28d90:::benchmark_data_smithi194_121512_object55685:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d308de:::benchmark_data_smithi194_121512_object101487:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d3603f:::benchmark_data_smithi194_121512_object101480:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d3d788:::benchmark_data_smithi194_121512_object42495:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d49ae7:::benchmark_data_smithi194_121512_object65934:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d4c340:::benchmark_data_smithi194_121512_object50738:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d54ea5:::benchmark_data_smithi194_121512_object25520:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d65e7d:::benchmark_data_smithi194_121512_object130968:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d6cff2:::benchmark_data_smithi194_121512_object38714:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d6d333:::benchmark_data_smithi194_121512_object61346:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d70909:::benchmark_data_smithi194_121512_object52589:head# 2024-04-29T14:26:38.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d76366:::benchmark_data_smithi194_121512_object23726:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d8a348:::benchmark_data_smithi194_121512_object16771:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69d8df73:::benchmark_data_smithi194_121512_object25507:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69da9765:::benchmark_data_smithi194_121512_object127055:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69daaedb:::benchmark_data_smithi194_121512_object24895:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69dc4bcb:::benchmark_data_smithi194_121512_object21721:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69dc6af4:::benchmark_data_smithi194_121512_object92431:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69dc7370:::benchmark_data_smithi194_121512_object76006:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69dca1d2:::benchmark_data_smithi194_121512_object89560:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69dcc360:::benchmark_data_smithi194_121512_object63396:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69dd56d0:::benchmark_data_smithi194_121512_object131770:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ded09a:::benchmark_data_smithi194_121512_object116705:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69df0409:::benchmark_data_smithi194_121512_object83366:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69dff002:::benchmark_data_smithi194_121512_object34904:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e04d9b:::benchmark_data_smithi194_121512_object39814:head# 2024-04-29T14:26:38.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e094f4:::benchmark_data_smithi194_121512_object39092:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e14d8b:::benchmark_data_smithi194_121512_object93759:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e35f17:::benchmark_data_smithi194_121512_object105677:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e3cd4d:::benchmark_data_smithi194_121512_object116371:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e48885:::benchmark_data_smithi194_121512_object130734:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e48a62:::benchmark_data_smithi194_121512_object123893:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e4c348:::benchmark_data_smithi194_121512_object131745:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e545cd:::benchmark_data_smithi194_121512_object99142:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e5556e:::benchmark_data_smithi194_121512_object107550:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e5a496:::benchmark_data_smithi194_121512_object71593:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e5e7aa:::benchmark_data_smithi194_121512_object115020:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e686fc:::benchmark_data_smithi194_121512_object56764:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e72fab:::benchmark_data_smithi194_121512_object131097:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69e84d9d:::benchmark_data_smithi194_121512_object55917:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ea7af3:::benchmark_data_smithi194_121512_object75459:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69eb1098:::benchmark_data_smithi194_121512_object84194:head# 2024-04-29T14:26:38.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69eb7157:::benchmark_data_smithi194_121512_object24543:head# 2024-04-29T14:26:38.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69edc3ad:::benchmark_data_smithi194_121512_object16725:head# 2024-04-29T14:26:38.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69eddf95:::benchmark_data_smithi194_121512_object61837:head# 2024-04-29T14:26:38.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69eead49:::benchmark_data_smithi194_121512_object66732:head# 2024-04-29T14:26:38.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69eeb9aa:::benchmark_data_smithi194_121512_object15347:head# 2024-04-29T14:26:38.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ef008c:::benchmark_data_smithi194_121512_object82752:head# 2024-04-29T14:26:38.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ef2787:::benchmark_data_smithi194_121512_object26478:head# 2024-04-29T14:26:38.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ef839b:::benchmark_data_smithi194_121512_object49806:head# 2024-04-29T14:26:38.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f00030:::benchmark_data_smithi194_121512_object51642:head# 2024-04-29T14:26:38.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f03b63:::benchmark_data_smithi194_121512_object7466:head# 2024-04-29T14:26:38.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f08745:::benchmark_data_smithi194_121512_object95139:head# 2024-04-29T14:26:38.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f0c268:::benchmark_data_smithi194_121512_object107497:head# 2024-04-29T14:26:38.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f10abf:::benchmark_data_smithi194_121512_object81929:head# 2024-04-29T14:26:38.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f12dcc:::benchmark_data_smithi194_121512_object103030:head# 2024-04-29T14:26:38.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f148fe:::benchmark_data_smithi194_121512_object120057:head# 2024-04-29T14:26:38.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f1dfc8:::benchmark_data_smithi194_121512_object54303:head# 2024-04-29T14:26:38.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f1ecaa:::benchmark_data_smithi194_121512_object97968:head# 2024-04-29T14:26:38.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f25ea6:::benchmark_data_smithi194_121512_object56818:head# 2024-04-29T14:26:38.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f27dfe:::benchmark_data_smithi194_121512_object78405:head# 2024-04-29T14:26:38.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f36716:::benchmark_data_smithi194_121512_object36878:head# 2024-04-29T14:26:38.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f3699a:::benchmark_data_smithi194_121512_object101875:head# 2024-04-29T14:26:38.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f3d60a:::benchmark_data_smithi194_121512_object117418:head# 2024-04-29T14:26:38.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f448f2:::benchmark_data_smithi194_121512_object62939:head# 2024-04-29T14:26:38.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f45889:::benchmark_data_smithi194_121512_object53557:head# 2024-04-29T14:26:38.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f4b993:::benchmark_data_smithi194_121512_object89771:head# 2024-04-29T14:26:38.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f576d1:::benchmark_data_smithi194_121512_object51381:head# 2024-04-29T14:26:38.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f5f528:::benchmark_data_smithi194_121512_object18024:head# 2024-04-29T14:26:38.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f6867e:::benchmark_data_smithi194_121512_object128743:head# 2024-04-29T14:26:38.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f6cd69:::benchmark_data_smithi194_121512_object93350:head# 2024-04-29T14:26:38.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f6f95b:::benchmark_data_smithi194_121512_object101907:head# 2024-04-29T14:26:38.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f70f09:::benchmark_data_smithi194_121512_object61751:head# 2024-04-29T14:26:38.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f72ab3:::benchmark_data_smithi194_121512_object127464:head# 2024-04-29T14:26:38.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f76e18:::benchmark_data_smithi194_121512_object26472:head# 2024-04-29T14:26:38.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69f94bb2:::benchmark_data_smithi194_121512_object52027:head# 2024-04-29T14:26:38.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fa1907:::benchmark_data_smithi194_121512_object130600:head# 2024-04-29T14:26:38.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fa74aa:::benchmark_data_smithi194_121512_object23259:head# 2024-04-29T14:26:38.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fa9ada:::benchmark_data_smithi194_121512_object25697:head# 2024-04-29T14:26:38.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fb0b61:::benchmark_data_smithi194_121512_object114617:head# 2024-04-29T14:26:38.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fb9d4b:::benchmark_data_smithi194_121512_object19858:head# 2024-04-29T14:26:38.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fc309f:::benchmark_data_smithi194_121512_object28981:head# 2024-04-29T14:26:38.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fd8923:::benchmark_data_smithi194_121512_object62967:head# 2024-04-29T14:26:38.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fda80e:::benchmark_data_smithi194_121512_object46669:head# 2024-04-29T14:26:38.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fde4e6:::benchmark_data_smithi194_121512_object91011:head# 2024-04-29T14:26:38.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fde89f:::benchmark_data_smithi194_121512_object129698:head# 2024-04-29T14:26:38.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fe1b05:::benchmark_data_smithi194_121512_object68802:head# 2024-04-29T14:26:38.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69fe537a:::benchmark_data_smithi194_121512_object43120:head# 2024-04-29T14:26:38.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ff0353:::benchmark_data_smithi194_121512_object100437:head# 2024-04-29T14:26:38.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ff62a6:::benchmark_data_smithi194_121512_object76531:head# 2024-04-29T14:26:38.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ff7467:::benchmark_data_smithi194_121512_object69083:head# 2024-04-29T14:26:38.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:69ffd8c1:::benchmark_data_smithi194_121512_object12257:head# 2024-04-29T14:26:38.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a00087a:::benchmark_data_smithi194_121512_object91401:head# 2024-04-29T14:26:38.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a01b382:::benchmark_data_smithi194_121512_object50843:head# 2024-04-29T14:26:38.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a01f559:::benchmark_data_smithi194_121512_object29987:head# 2024-04-29T14:26:38.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a021516:::benchmark_data_smithi194_121512_object13126:head# 2024-04-29T14:26:38.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a02db9f:::benchmark_data_smithi194_121512_object132206:head# 2024-04-29T14:26:38.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a02e1b9:::benchmark_data_smithi194_121512_object53542:head# 2024-04-29T14:26:38.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a031151:::benchmark_data_smithi194_121512_object23396:head# 2024-04-29T14:26:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a03819a:::benchmark_data_smithi194_121512_object96793:head# 2024-04-29T14:26:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a045cc4:::benchmark_data_smithi194_121512_object9178:head# 2024-04-29T14:26:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a045cfb:::benchmark_data_smithi194_121512_object49143:head# 2024-04-29T14:26:38.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a046fb5:::benchmark_data_smithi194_121512_object69696:head# 2024-04-29T14:26:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0496fc:::benchmark_data_smithi194_121512_object6030:head# 2024-04-29T14:26:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a062ca2:::benchmark_data_smithi194_121512_object111160:head# 2024-04-29T14:26:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a065d16:::benchmark_data_smithi194_121512_object81183:head# 2024-04-29T14:26:38.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a066111:::benchmark_data_smithi194_121512_object96144:head# 2024-04-29T14:26:38.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0718a0:::benchmark_data_smithi194_121512_object121786:head# 2024-04-29T14:26:38.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a07f663:::benchmark_data_smithi194_121512_object28621:head# 2024-04-29T14:26:38.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a08a0fb:::benchmark_data_smithi194_121512_object15274:head# 2024-04-29T14:26:38.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a093ebe:::benchmark_data_smithi194_121512_object29143:head# 2024-04-29T14:26:38.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a095379:::benchmark_data_smithi194_121512_object82812:head# 2024-04-29T14:26:38.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0ac44b:::benchmark_data_smithi194_121512_object7594:head# 2024-04-29T14:26:38.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0b1294:::benchmark_data_smithi194_121512_object75051:head# 2024-04-29T14:26:38.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0bda7a:::benchmark_data_smithi194_121512_object128465:head# 2024-04-29T14:26:38.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0bf29e:::benchmark_data_smithi194_121512_object85053:head# 2024-04-29T14:26:38.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0c7562:::benchmark_data_smithi194_121512_object43789:head# 2024-04-29T14:26:38.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0cff7b:::benchmark_data_smithi194_121512_object75502:head# 2024-04-29T14:26:38.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0d4657:::benchmark_data_smithi194_121512_object86780:head# 2024-04-29T14:26:38.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0e1d7c:::benchmark_data_smithi194_121512_object77134:head# 2024-04-29T14:26:38.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a0f459c:::benchmark_data_smithi194_121512_object117781:head# 2024-04-29T14:26:38.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a102b4c:::benchmark_data_smithi194_121512_object77316:head# 2024-04-29T14:26:38.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a10dad8:::benchmark_data_smithi194_121512_object67269:head# 2024-04-29T14:26:38.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a10f9b4:::benchmark_data_smithi194_121512_object95813:head# 2024-04-29T14:26:38.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a113c1c:::benchmark_data_smithi194_121512_object90082:head# 2024-04-29T14:26:38.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a125c2e:::benchmark_data_smithi194_121512_object123896:head# 2024-04-29T14:26:38.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a149ed3:::benchmark_data_smithi194_121512_object64233:head# 2024-04-29T14:26:38.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a14c209:::benchmark_data_smithi194_121512_object48498:head# 2024-04-29T14:26:38.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a162b47:::benchmark_data_smithi194_121512_object65171:head# 2024-04-29T14:26:38.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a179ea9:::benchmark_data_smithi194_121512_object95969:head# 2024-04-29T14:26:38.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a191f89:::benchmark_data_smithi194_121512_object40467:head# 2024-04-29T14:26:38.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a19c4a3:::benchmark_data_smithi194_121512_object83389:head# 2024-04-29T14:26:38.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a19e3b7:::benchmark_data_smithi194_121512_object57926:head# 2024-04-29T14:26:38.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a19f8a5:::benchmark_data_smithi194_121512_object72087:head# 2024-04-29T14:26:38.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a1a6f13:::benchmark_data_smithi194_121512_object124712:head# 2024-04-29T14:26:38.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a1a8970:::benchmark_data_smithi194_121512_object81372:head# 2024-04-29T14:26:38.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a1aaf45:::benchmark_data_smithi194_121512_object33194:head# 2024-04-29T14:26:38.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a1bf9c8:::benchmark_data_smithi194_121512_object34765:head# 2024-04-29T14:26:38.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a1c60bb:::benchmark_data_smithi194_121512_object13710:head# 2024-04-29T14:26:38.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a1e85c0:::benchmark_data_smithi194_121512_object85329:head# 2024-04-29T14:26:38.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a1faa06:::benchmark_data_smithi194_121512_object81225:head# 2024-04-29T14:26:38.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a202372:::benchmark_data_smithi194_121512_object119798:head# 2024-04-29T14:26:38.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a20d099:::benchmark_data_smithi194_121512_object28611:head# 2024-04-29T14:26:38.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a210b84:::benchmark_data_smithi194_121512_object37959:head# 2024-04-29T14:26:38.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a238729:::benchmark_data_smithi194_121512_object14001:head# 2024-04-29T14:26:38.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2460da:::benchmark_data_smithi194_121512_object86071:head# 2024-04-29T14:26:38.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a24756f:::benchmark_data_smithi194_121512_object18279:head# 2024-04-29T14:26:38.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a24cb15:::benchmark_data_smithi194_121512_object67098:head# 2024-04-29T14:26:38.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a25034e:::benchmark_data_smithi194_121512_object101673:head# 2024-04-29T14:26:38.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a25103c:::benchmark_data_smithi194_121512_object112978:head# 2024-04-29T14:26:38.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a252dd2:::benchmark_data_smithi194_121512_object131045:head# 2024-04-29T14:26:38.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a25e2e0:::benchmark_data_smithi194_121512_object27197:head# 2024-04-29T14:26:38.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a266206:::benchmark_data_smithi194_121512_object29062:head# 2024-04-29T14:26:38.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a26cf17:::benchmark_data_smithi194_121512_object119301:head# 2024-04-29T14:26:38.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a277437:::benchmark_data_smithi194_121512_object56320:head# 2024-04-29T14:26:38.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2788a2:::benchmark_data_smithi194_121512_object75657:head# 2024-04-29T14:26:38.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a27f025:::benchmark_data_smithi194_121512_object124568:head# 2024-04-29T14:26:38.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a281237:::benchmark_data_smithi194_121512_object40823:head# 2024-04-29T14:26:38.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a296406:::benchmark_data_smithi194_121512_object114922:head# 2024-04-29T14:26:38.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a298303:::benchmark_data_smithi194_121512_object110078:head# 2024-04-29T14:26:38.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2a936f:::benchmark_data_smithi194_121512_object100432:head# 2024-04-29T14:26:38.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2acb92:::benchmark_data_smithi194_121512_object93362:head# 2024-04-29T14:26:38.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2aeb9d:::benchmark_data_smithi194_121512_object98669:head# 2024-04-29T14:26:38.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2e33cc:::benchmark_data_smithi194_121512_object101468:head# 2024-04-29T14:26:38.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2e52b4:::benchmark_data_smithi194_121512_object70622:head# 2024-04-29T14:26:38.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2e8a1a:::benchmark_data_smithi194_121512_object1283:head# 2024-04-29T14:26:38.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2ea4d6:::benchmark_data_smithi194_121512_object87372:head# 2024-04-29T14:26:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2f00b7:::benchmark_data_smithi194_121512_object125286:head# 2024-04-29T14:26:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2f0500:::benchmark_data_smithi194_121512_object22850:head# 2024-04-29T14:26:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2f362a:::benchmark_data_smithi194_121512_object79948:head# 2024-04-29T14:26:38.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2f55b8:::benchmark_data_smithi194_121512_object112451:head# 2024-04-29T14:26:38.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a2ffed9:::benchmark_data_smithi194_121512_object91473:head# 2024-04-29T14:26:38.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a304ce9:::benchmark_data_smithi194_121512_object129603:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3130a6:::benchmark_data_smithi194_121512_object26315:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a317204:::benchmark_data_smithi194_121512_object59595:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a31d001:::benchmark_data_smithi194_121512_object113664:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a328fc0:::benchmark_data_smithi194_121512_object83348:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a33d524:::benchmark_data_smithi194_121512_object28121:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a33fe14:::benchmark_data_smithi194_121512_object34365:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a345605:::benchmark_data_smithi194_121512_object44969:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a34a455:::benchmark_data_smithi194_121512_object63109:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a35d902:::benchmark_data_smithi194_121512_object8886:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a371125:::benchmark_data_smithi194_121512_object110743:head# 2024-04-29T14:26:38.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3748d1:::benchmark_data_smithi194_121512_object54612:head# 2024-04-29T14:26:38.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a376001:::benchmark_data_smithi194_121512_object112935:head# 2024-04-29T14:26:38.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a376b42:::benchmark_data_smithi194_121512_object83542:head# 2024-04-29T14:26:38.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a37dc38:::benchmark_data_smithi194_121512_object33031:head# 2024-04-29T14:26:38.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a38ab8e:::benchmark_data_smithi194_121512_object73570:head# 2024-04-29T14:26:38.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a396037:::benchmark_data_smithi194_121512_object119111:head# 2024-04-29T14:26:38.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3a0981:::benchmark_data_smithi194_121512_object63007:head# 2024-04-29T14:26:38.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3a4017:::benchmark_data_smithi194_121512_object80527:head# 2024-04-29T14:26:38.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3a76c9:::benchmark_data_smithi194_121512_object99203:head# 2024-04-29T14:26:38.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3adc5d:::benchmark_data_smithi194_121512_object38867:head# 2024-04-29T14:26:38.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3b3283:::benchmark_data_smithi194_121512_object61092:head# 2024-04-29T14:26:38.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3b7095:::benchmark_data_smithi194_121512_object68884:head# 2024-04-29T14:26:38.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3b7c0a:::benchmark_data_smithi194_121512_object39773:head# 2024-04-29T14:26:38.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3b9220:::benchmark_data_smithi194_121512_object86776:head# 2024-04-29T14:26:38.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3c3f5c:::benchmark_data_smithi194_121512_object84941:head# 2024-04-29T14:26:38.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3c4b16:::benchmark_data_smithi194_121512_object15884:head# 2024-04-29T14:26:38.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3dfcfc:::benchmark_data_smithi194_121512_object132431:head# 2024-04-29T14:26:38.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3e9b84:::benchmark_data_smithi194_121512_object92944:head# 2024-04-29T14:26:38.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3f75e5:::benchmark_data_smithi194_121512_object50546:head# 2024-04-29T14:26:38.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3fabe7:::benchmark_data_smithi194_121512_object132256:head# 2024-04-29T14:26:38.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a3ff874:::benchmark_data_smithi194_121512_object71882:head# 2024-04-29T14:26:38.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a404987:::benchmark_data_smithi194_121512_object105834:head# 2024-04-29T14:26:38.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a405a77:::benchmark_data_smithi194_121512_object88287:head# 2024-04-29T14:26:38.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4092ee:::benchmark_data_smithi194_121512_object27676:head# 2024-04-29T14:26:38.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a41d63c:::benchmark_data_smithi194_121512_object49251:head# 2024-04-29T14:26:38.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a42892e:::benchmark_data_smithi194_121512_object22612:head# 2024-04-29T14:26:38.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a42b731:::benchmark_data_smithi194_121512_object76403:head# 2024-04-29T14:26:38.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a42e7be:::benchmark_data_smithi194_121512_object107523:head# 2024-04-29T14:26:38.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4393c3:::benchmark_data_smithi194_121512_object56261:head# 2024-04-29T14:26:38.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a446b59:::benchmark_data_smithi194_121512_object14085:head# 2024-04-29T14:26:38.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4575aa:::benchmark_data_smithi194_121512_object112992:head# 2024-04-29T14:26:38.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4584e5:::benchmark_data_smithi194_121512_object39198:head# 2024-04-29T14:26:38.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a467438:::benchmark_data_smithi194_121512_object87652:head# 2024-04-29T14:26:38.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a46b422:::benchmark_data_smithi194_121512_object30809:head# 2024-04-29T14:26:38.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a472107:::benchmark_data_smithi194_121512_object50446:head# 2024-04-29T14:26:38.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a475946:::benchmark_data_smithi194_121512_object54103:head# 2024-04-29T14:26:38.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a482edf:::benchmark_data_smithi194_121512_object72378:head# 2024-04-29T14:26:38.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a487e78:::benchmark_data_smithi194_121512_object72509:head# 2024-04-29T14:26:38.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a49a91b:::benchmark_data_smithi194_121512_object91744:head# 2024-04-29T14:26:38.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a49aeaf:::benchmark_data_smithi194_121512_object117118:head# 2024-04-29T14:26:38.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4a3aee:::benchmark_data_smithi194_121512_object102031:head# 2024-04-29T14:26:38.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4c1c4e:::benchmark_data_smithi194_121512_object108769:head# 2024-04-29T14:26:38.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4ce0ea:::benchmark_data_smithi194_121512_object116760:head# 2024-04-29T14:26:38.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4cf9f5:::benchmark_data_smithi194_121512_object38652:head# 2024-04-29T14:26:38.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4d1b20:::benchmark_data_smithi194_121512_object17586:head# 2024-04-29T14:26:38.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4d490c:::benchmark_data_smithi194_121512_object68201:head# 2024-04-29T14:26:38.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4dff7f:::benchmark_data_smithi194_121512_object10793:head# 2024-04-29T14:26:38.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4e9305:::benchmark_data_smithi194_121512_object12047:head# 2024-04-29T14:26:38.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a4ef087:::benchmark_data_smithi194_121512_object43453:head# 2024-04-29T14:26:38.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a51896d:::benchmark_data_smithi194_121512_object115072:head# 2024-04-29T14:26:38.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a528204:::benchmark_data_smithi194_121512_object106646:head# 2024-04-29T14:26:38.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a52bfec:::benchmark_data_smithi194_121512_object54487:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a53302e:::benchmark_data_smithi194_121512_object29403:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a535a77:::benchmark_data_smithi194_121512_object125804:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a535aa6:::benchmark_data_smithi194_121512_object42154:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a53a9e1:::benchmark_data_smithi194_121512_object78428:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a53b7ce:::benchmark_data_smithi194_121512_object111220:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a54678c:::benchmark_data_smithi194_121512_object32331:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a548603:::benchmark_data_smithi194_121512_object36713:head# 2024-04-29T14:26:38.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5527aa:::benchmark_data_smithi194_121512_object27411:head# 2024-04-29T14:26:38.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a55a706:::benchmark_data_smithi194_121512_object58305:head# 2024-04-29T14:26:38.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a55d5b3:::benchmark_data_smithi194_121512_object73979:head# 2024-04-29T14:26:38.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a55da94:::benchmark_data_smithi194_121512_object9248:head# 2024-04-29T14:26:38.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a55db1d:::benchmark_data_smithi194_121512_object50629:head# 2024-04-29T14:26:38.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a560494:::benchmark_data_smithi194_121512_object106908:head# 2024-04-29T14:26:38.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a587580:::benchmark_data_smithi194_121512_object109263:head# 2024-04-29T14:26:38.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a589ef0:::benchmark_data_smithi194_121512_object103135:head# 2024-04-29T14:26:38.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5a6102:::benchmark_data_smithi194_121512_object82352:head# 2024-04-29T14:26:38.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5ac6b7:::benchmark_data_smithi194_121512_object86314:head# 2024-04-29T14:26:38.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5bc10f:::benchmark_data_smithi194_121512_object43956:head# 2024-04-29T14:26:38.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5c3634:::benchmark_data_smithi194_121512_object51379:head# 2024-04-29T14:26:38.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5cd209:::benchmark_data_smithi194_121512_object16245:head# 2024-04-29T14:26:38.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5ce116:::benchmark_data_smithi194_121512_object25737:head# 2024-04-29T14:26:38.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5d78a0:::benchmark_data_smithi194_121512_object83172:head# 2024-04-29T14:26:38.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5da603:::benchmark_data_smithi194_121512_object132249:head# 2024-04-29T14:26:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5dd3b7:::benchmark_data_smithi194_121512_object47025:head# 2024-04-29T14:26:38.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5f0871:::benchmark_data_smithi194_121512_object60112:head# 2024-04-29T14:26:38.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a5f08fd:::benchmark_data_smithi194_121512_object124059:head# 2024-04-29T14:26:38.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a60eb14:::benchmark_data_smithi194_121512_object33297:head# 2024-04-29T14:26:38.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a611576:::benchmark_data_smithi194_121512_object24892:head# 2024-04-29T14:26:38.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6188e5:::benchmark_data_smithi194_121512_object69953:head# 2024-04-29T14:26:38.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a61f74f:::benchmark_data_smithi194_121512_object28340:head# 2024-04-29T14:26:38.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a621d64:::benchmark_data_smithi194_121512_object92435:head# 2024-04-29T14:26:38.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a631950:::benchmark_data_smithi194_121512_object61692:head# 2024-04-29T14:26:38.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a63550d:::benchmark_data_smithi194_121512_object20181:head# 2024-04-29T14:26:38.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6355da:::benchmark_data_smithi194_121512_object95040:head# 2024-04-29T14:26:38.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a63703f:::benchmark_data_smithi194_121512_object10195:head# 2024-04-29T14:26:38.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6372b8:::benchmark_data_smithi194_121512_object89115:head# 2024-04-29T14:26:38.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a63ad88:::benchmark_data_smithi194_121512_object69591:head# 2024-04-29T14:26:38.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a64320a:::benchmark_data_smithi194_121512_object62474:head# 2024-04-29T14:26:38.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6478ff:::benchmark_data_smithi194_121512_object31635:head# 2024-04-29T14:26:38.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a656a5f:::benchmark_data_smithi194_121512_object84767:head# 2024-04-29T14:26:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a664385:::benchmark_data_smithi194_121512_object121435:head# 2024-04-29T14:26:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a667a9c:::benchmark_data_smithi194_121512_object15208:head# 2024-04-29T14:26:38.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a66fcc8:::benchmark_data_smithi194_121512_object83075:head# 2024-04-29T14:26:38.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a67427d:::benchmark_data_smithi194_121512_object98181:head# 2024-04-29T14:26:38.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a67c9bd:::benchmark_data_smithi194_121512_object11982:head# 2024-04-29T14:26:38.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a68128d:::benchmark_data_smithi194_121512_object31560:head# 2024-04-29T14:26:38.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a684cf0:::benchmark_data_smithi194_121512_object123280:head# 2024-04-29T14:26:38.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6855c9:::benchmark_data_smithi194_121512_object108439:head# 2024-04-29T14:26:38.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a69daf1:::benchmark_data_smithi194_121512_object88232:head# 2024-04-29T14:26:38.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6b5219:::benchmark_data_smithi194_121512_object15677:head# 2024-04-29T14:26:38.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6b6757:::benchmark_data_smithi194_121512_object32248:head# 2024-04-29T14:26:38.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6c126a:::benchmark_data_smithi194_121512_object10433:head# 2024-04-29T14:26:38.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6d8722:::benchmark_data_smithi194_121512_object111183:head# 2024-04-29T14:26:38.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6eb84a:::benchmark_data_smithi194_121512_object47135:head# 2024-04-29T14:26:38.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a6fb0ad:::benchmark_data_smithi194_121512_object79949:head# 2024-04-29T14:26:38.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7028b2:::benchmark_data_smithi194_121512_object42205:head# 2024-04-29T14:26:38.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a70466c:::benchmark_data_smithi194_121512_object31126:head# 2024-04-29T14:26:38.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a706350:::benchmark_data_smithi194_121512_object43239:head# 2024-04-29T14:26:38.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a707ac5:::benchmark_data_smithi194_121512_object90784:head# 2024-04-29T14:26:38.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a710789:::benchmark_data_smithi194_121512_object26890:head# 2024-04-29T14:26:38.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7290ba:::benchmark_data_smithi194_121512_object37291:head# 2024-04-29T14:26:38.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a72c5ce:::benchmark_data_smithi194_121512_object105877:head# 2024-04-29T14:26:38.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a73500d:::benchmark_data_smithi194_121512_object25608:head# 2024-04-29T14:26:38.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7375c7:::benchmark_data_smithi194_121512_object110298:head# 2024-04-29T14:26:38.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a73a718:::benchmark_data_smithi194_121512_object43984:head# 2024-04-29T14:26:38.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a744091:::benchmark_data_smithi194_121512_object104256:head# 2024-04-29T14:26:38.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a74b253:::benchmark_data_smithi194_121512_object58550:head# 2024-04-29T14:26:38.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a756640:::benchmark_data_smithi194_121512_object23299:head# 2024-04-29T14:26:38.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a758083:::benchmark_data_smithi194_121512_object98890:head# 2024-04-29T14:26:38.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a75ff19:::benchmark_data_smithi194_121512_object5110:head# 2024-04-29T14:26:38.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7626b8:::benchmark_data_smithi194_121512_object125554:head# 2024-04-29T14:26:38.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a76419b:::benchmark_data_smithi194_121512_object25990:head# 2024-04-29T14:26:38.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7662c8:::benchmark_data_smithi194_121512_object111479:head# 2024-04-29T14:26:38.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7668c9:::benchmark_data_smithi194_121512_object117681:head# 2024-04-29T14:26:38.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a76b06e:::benchmark_data_smithi194_121512_object108002:head# 2024-04-29T14:26:38.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a770e65:::benchmark_data_smithi194_121512_object36390:head# 2024-04-29T14:26:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a77ee48:::benchmark_data_smithi194_121512_object126883:head# 2024-04-29T14:26:38.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a781d03:::benchmark_data_smithi194_121512_object6784:head# 2024-04-29T14:26:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a783140:::benchmark_data_smithi194_121512_object258:head# 2024-04-29T14:26:38.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a796406:::benchmark_data_smithi194_121512_object27692:head# 2024-04-29T14:26:38.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7972c8:::benchmark_data_smithi194_121512_object21374:head# 2024-04-29T14:26:38.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a797a93:::benchmark_data_smithi194_121512_object10344:head# 2024-04-29T14:26:38.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a79becb:::benchmark_data_smithi194_121512_object50022:head# 2024-04-29T14:26:38.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a79ee95:::benchmark_data_smithi194_121512_object121221:head# 2024-04-29T14:26:38.228 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7ac85d:::benchmark_data_smithi194_121512_object17790:head# 2024-04-29T14:26:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7aef8f:::benchmark_data_smithi194_121512_object104434:head# 2024-04-29T14:26:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7b1721:::benchmark_data_smithi194_121512_object29373:head# 2024-04-29T14:26:38.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7bb2d0:::benchmark_data_smithi194_121512_object81259:head# 2024-04-29T14:26:38.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7d14ce:::benchmark_data_smithi194_121512_object64819:head# 2024-04-29T14:26:38.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7d1578:::benchmark_data_smithi194_121512_object193:head# 2024-04-29T14:26:38.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7d5573:::benchmark_data_smithi194_121512_object13836:head# 2024-04-29T14:26:38.232 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:38.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7dcc93:::benchmark_data_smithi194_121512_object97757:head# 2024-04-29T14:26:38.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7deceb:::benchmark_data_smithi194_121512_object132510:head# 2024-04-29T14:26:38.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7ed80d:::benchmark_data_smithi194_121512_object38838:head# 2024-04-29T14:26:38.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7f0fe5:::benchmark_data_smithi194_121512_object60318:head# 2024-04-29T14:26:38.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a7f643e:::benchmark_data_smithi194_121512_object72662:head# 2024-04-29T14:26:38.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a808a1a:::benchmark_data_smithi194_121512_object119777:head# 2024-04-29T14:26:38.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a80d81c:::benchmark_data_smithi194_121512_object100427:head# 2024-04-29T14:26:38.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a80dde3:::benchmark_data_smithi194_121512_object22818:head# 2024-04-29T14:26:38.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a815b9a:::benchmark_data_smithi194_121512_object130129:head# 2024-04-29T14:26:38.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a817652:::benchmark_data_smithi194_121512_object53568:head# 2024-04-29T14:26:38.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a817e6e:::benchmark_data_smithi194_121512_object10696:head# 2024-04-29T14:26:38.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8278cb:::benchmark_data_smithi194_121512_object89954:head# 2024-04-29T14:26:38.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a830fcd:::benchmark_data_smithi194_121512_object17054:head# 2024-04-29T14:26:38.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a832c55:::benchmark_data_smithi194_121512_object20943:head# 2024-04-29T14:26:38.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a83d243:::benchmark_data_smithi194_121512_object40719:head# 2024-04-29T14:26:38.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a840493:::benchmark_data_smithi194_121512_object80973:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a84565e:::benchmark_data_smithi194_121512_object117495:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a84d1b8:::benchmark_data_smithi194_121512_object4091:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a855e68:::benchmark_data_smithi194_121512_object117771:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a860c82:::benchmark_data_smithi194_121512_object88842:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a86a7d7:::benchmark_data_smithi194_121512_object126162:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a86dfbd:::benchmark_data_smithi194_121512_object35534:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a877118:::benchmark_data_smithi194_121512_object54681:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a87ac54:::benchmark_data_smithi194_121512_object27502:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a880624:::benchmark_data_smithi194_121512_object39385:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a88668c:::benchmark_data_smithi194_121512_object46804:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a888363:::benchmark_data_smithi194_121512_object51607:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a89bb42:::benchmark_data_smithi194_121512_object57649:head# 2024-04-29T14:26:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8a38b0:::benchmark_data_smithi194_121512_object118992:head# 2024-04-29T14:26:38.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8a439b:::benchmark_data_smithi194_121512_object73408:head# 2024-04-29T14:26:38.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8a6fea:::benchmark_data_smithi194_121512_object78838:head# 2024-04-29T14:26:38.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8acedd:::benchmark_data_smithi194_121512_object69827:head# 2024-04-29T14:26:38.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8ad396:::benchmark_data_smithi194_121512_object93640:head# 2024-04-29T14:26:38.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8b1d46:::benchmark_data_smithi194_121512_object90863:head# 2024-04-29T14:26:38.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8b52eb:::benchmark_data_smithi194_121512_object4995:head# 2024-04-29T14:26:38.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8b68b4:::benchmark_data_smithi194_121512_object127772:head# 2024-04-29T14:26:38.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8bc701:::benchmark_data_smithi194_121512_object14092:head# 2024-04-29T14:26:38.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8c020f:::benchmark_data_smithi194_121512_object35400:head# 2024-04-29T14:26:38.254 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:38 smithi194 ceph-mon[44089]: osdmap e475: 12 total, 11 up, 7 in 2024-04-29T14:26:38.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8c1d5e:::benchmark_data_smithi194_121512_object33435:head# 2024-04-29T14:26:38.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8c4f91:::benchmark_data_smithi194_121512_object111359:head# 2024-04-29T14:26:38.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8c7dee:::benchmark_data_smithi194_121512_object125372:head# 2024-04-29T14:26:38.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8cd0a5:::benchmark_data_smithi194_121512_object2960:head# 2024-04-29T14:26:38.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8d0878:::benchmark_data_smithi194_121512_object55247:head# 2024-04-29T14:26:38.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8d100c:::benchmark_data_smithi194_121512_object109795:head# 2024-04-29T14:26:38.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8d1251:::benchmark_data_smithi194_121512_object133407:head# 2024-04-29T14:26:38.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8d822d:::benchmark_data_smithi194_121512_object123296:head# 2024-04-29T14:26:38.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8dee64:::benchmark_data_smithi194_121512_object126050:head# 2024-04-29T14:26:38.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8eb5fd:::benchmark_data_smithi194_121512_object24107:head# 2024-04-29T14:26:38.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8f02db:::benchmark_data_smithi194_121512_object66722:head# 2024-04-29T14:26:38.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8f0dc5:::benchmark_data_smithi194_121512_object121621:head# 2024-04-29T14:26:38.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8f2458:::benchmark_data_smithi194_121512_object129347:head# 2024-04-29T14:26:38.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a8f76ba:::benchmark_data_smithi194_121512_object56842:head# 2024-04-29T14:26:38.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a907b21:::benchmark_data_smithi194_121512_object64613:head# 2024-04-29T14:26:38.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9213f3:::benchmark_data_smithi194_121512_object43754:head# 2024-04-29T14:26:38.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a92ad14:::benchmark_data_smithi194_121512_object44811:head# 2024-04-29T14:26:38.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a92fe50:::benchmark_data_smithi194_121512_object733:head# 2024-04-29T14:26:38.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a934abb:::benchmark_data_smithi194_121512_object14241:head# 2024-04-29T14:26:38.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a93adc8:::benchmark_data_smithi194_121512_object129717:head# 2024-04-29T14:26:38.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a941fa1:::benchmark_data_smithi194_121512_object81419:head# 2024-04-29T14:26:38.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a944fcf:::benchmark_data_smithi194_121512_object112850:head# 2024-04-29T14:26:38.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a94d157:::benchmark_data_smithi194_121512_object66255:head# 2024-04-29T14:26:38.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9535dd:::benchmark_data_smithi194_121512_object74410:head# 2024-04-29T14:26:38.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a958d8f:::benchmark_data_smithi194_121512_object73459:head# 2024-04-29T14:26:38.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a95af7c:::benchmark_data_smithi194_121512_object43355:head# 2024-04-29T14:26:38.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a95d1ca:::benchmark_data_smithi194_121512_object83830:head# 2024-04-29T14:26:38.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9623d0:::benchmark_data_smithi194_121512_object84772:head# 2024-04-29T14:26:38.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a976682:::benchmark_data_smithi194_121512_object5583:head# 2024-04-29T14:26:38.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a983f76:::benchmark_data_smithi194_121512_object63692:head# 2024-04-29T14:26:38.276 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a98c55a:::benchmark_data_smithi194_121512_object15784:head# 2024-04-29T14:26:38.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a98eafa:::benchmark_data_smithi194_121512_object101699:head# 2024-04-29T14:26:38.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9977dc:::benchmark_data_smithi194_121512_object65156:head# 2024-04-29T14:26:38.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9a20dd:::benchmark_data_smithi194_121512_object57697:head# 2024-04-29T14:26:38.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9a4caa:::benchmark_data_smithi194_121512_object74195:head# 2024-04-29T14:26:38.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9a6489:::benchmark_data_smithi194_121512_object29847:head# 2024-04-29T14:26:38.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9ac548:::benchmark_data_smithi194_121512_object56452:head# 2024-04-29T14:26:38.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9acda9:::benchmark_data_smithi194_121512_object10411:head# 2024-04-29T14:26:38.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9b0fc4:::benchmark_data_smithi194_121512_object83686:head# 2024-04-29T14:26:38.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9b78e6:::benchmark_data_smithi194_121512_object104166:head# 2024-04-29T14:26:38.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9bc6f2:::benchmark_data_smithi194_121512_object90939:head# 2024-04-29T14:26:38.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9c107d:::benchmark_data_smithi194_121512_object114473:head# 2024-04-29T14:26:38.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9c5c8a:::benchmark_data_smithi194_121512_object49630:head# 2024-04-29T14:26:38.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9c9f05:::benchmark_data_smithi194_121512_object128582:head# 2024-04-29T14:26:38.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9cd198:::benchmark_data_smithi194_121512_object89152:head# 2024-04-29T14:26:38.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9d1d2c:::benchmark_data_smithi194_121512_object80032:head# 2024-04-29T14:26:38.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9d34b3:::benchmark_data_smithi194_121512_object119537:head# 2024-04-29T14:26:38.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9d5900:::benchmark_data_smithi194_121512_object13056:head# 2024-04-29T14:26:38.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9e5319:::benchmark_data_smithi194_121512_object76537:head# 2024-04-29T14:26:38.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9e73cc:::benchmark_data_smithi194_121512_object53353:head# 2024-04-29T14:26:38.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9e9242:::benchmark_data_smithi194_121512_object82702:head# 2024-04-29T14:26:38.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9edf6e:::benchmark_data_smithi194_121512_object78195:head# 2024-04-29T14:26:38.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9f3873:::benchmark_data_smithi194_121512_object68200:head# 2024-04-29T14:26:38.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6a9f4573:::benchmark_data_smithi194_121512_object101064:head# 2024-04-29T14:26:38.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa0fbb3:::benchmark_data_smithi194_121512_object89950:head# 2024-04-29T14:26:38.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa11577:::benchmark_data_smithi194_121512_object101603:head# 2024-04-29T14:26:38.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa17856:::benchmark_data_smithi194_121512_object125825:head# 2024-04-29T14:26:38.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa1e478:::benchmark_data_smithi194_121512_object83390:head# 2024-04-29T14:26:38.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa26633:::benchmark_data_smithi194_121512_object109182:head# 2024-04-29T14:26:38.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa31590:::benchmark_data_smithi194_121512_object57464:head# 2024-04-29T14:26:38.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:26:38.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa3e447:::benchmark_data_smithi194_121512_object118466:head# 2024-04-29T14:26:38.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa41180:::benchmark_data_smithi194_121512_object11511:head# 2024-04-29T14:26:38.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa4e950:::benchmark_data_smithi194_121512_object114633:head# 2024-04-29T14:26:38.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa557d1:::benchmark_data_smithi194_121512_object13444:head# 2024-04-29T14:26:38.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa599ca:::benchmark_data_smithi194_121512_object61765:head# 2024-04-29T14:26:38.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa67628:::benchmark_data_smithi194_121512_object17795:head# 2024-04-29T14:26:38.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa69cf2:::benchmark_data_smithi194_121512_object52146:head# 2024-04-29T14:26:38.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa6a1f1:::benchmark_data_smithi194_121512_object125159:head# 2024-04-29T14:26:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:38.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa7a161:::benchmark_data_smithi194_121512_object84052:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa822d2:::benchmark_data_smithi194_121512_object13296:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa8a43c:::benchmark_data_smithi194_121512_object45791:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aa8c853:::benchmark_data_smithi194_121512_object46202:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aaa012d:::benchmark_data_smithi194_121512_object49563:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aaa204a:::benchmark_data_smithi194_121512_object123324:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aaa2c55:::benchmark_data_smithi194_121512_object119284:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aabf949:::benchmark_data_smithi194_121512_object10616:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aac2a4f:::benchmark_data_smithi194_121512_object66559:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aacf616:::benchmark_data_smithi194_121512_object51196:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aad25fb:::benchmark_data_smithi194_121512_object20309:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aad2b56:::benchmark_data_smithi194_121512_object16260:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aad927d:::benchmark_data_smithi194_121512_object13730:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aae05f8:::benchmark_data_smithi194_121512_object4556:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aaea04c:::benchmark_data_smithi194_121512_object91621:head# 2024-04-29T14:26:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aaf4ac5:::benchmark_data_smithi194_121512_object116755:head# 2024-04-29T14:26:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab0071a:::benchmark_data_smithi194_121512_object96091:head# 2024-04-29T14:26:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab02887:::benchmark_data_smithi194_121512_object5392:head# 2024-04-29T14:26:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab3925b:::benchmark_data_smithi194_121512_object67586:head# 2024-04-29T14:26:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab3ea99:::benchmark_data_smithi194_121512_object130757:head# 2024-04-29T14:26:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab3f046:::benchmark_data_smithi194_121512_object9262:head# 2024-04-29T14:26:38.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:26:38.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab4d1d3:::benchmark_data_smithi194_121512_object59932:head# 2024-04-29T14:26:38.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab5b082:::benchmark_data_smithi194_121512_object3894:head# 2024-04-29T14:26:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab5bb79:::benchmark_data_smithi194_121512_object133454:head# 2024-04-29T14:26:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab67f20:::benchmark_data_smithi194_121512_object23892:head# 2024-04-29T14:26:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab70d86:::benchmark_data_smithi194_121512_object28829:head# 2024-04-29T14:26:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab7f8cc:::benchmark_data_smithi194_121512_object53957:head# 2024-04-29T14:26:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab80c89:::benchmark_data_smithi194_121512_object27579:head# 2024-04-29T14:26:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab8a23a:::benchmark_data_smithi194_121512_object55486:head# 2024-04-29T14:26:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ab95dd6:::benchmark_data_smithi194_121512_object22462:head# 2024-04-29T14:26:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abb4527:::benchmark_data_smithi194_121512_object95417:head# 2024-04-29T14:26:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abcc3d9:::benchmark_data_smithi194_121512_object31256:head# 2024-04-29T14:26:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abd5cde:::benchmark_data_smithi194_121512_object54834:head# 2024-04-29T14:26:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abdc98a:::benchmark_data_smithi194_121512_object57517:head# 2024-04-29T14:26:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abe3663:::benchmark_data_smithi194_121512_object26990:head# 2024-04-29T14:26:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abe6164:::benchmark_data_smithi194_121512_object2067:head# 2024-04-29T14:26:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abe83d9:::benchmark_data_smithi194_121512_object3318:head# 2024-04-29T14:26:38.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abe9d15:::benchmark_data_smithi194_121512_object46156:head# 2024-04-29T14:26:38.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abec926:::benchmark_data_smithi194_121512_object121431:head# 2024-04-29T14:26:38.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abf2ce7:::benchmark_data_smithi194_121512_object35895:head# 2024-04-29T14:26:38.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abf8f42:::benchmark_data_smithi194_121512_object74804:head# 2024-04-29T14:26:38.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abfc507:::benchmark_data_smithi194_121512_object122841:head# 2024-04-29T14:26:38.314 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:38 smithi040 ceph-mon[39581]: osdmap e475: 12 total, 11 up, 7 in 2024-04-29T14:26:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6abff386:::benchmark_data_smithi194_121512_object32826:head# 2024-04-29T14:26:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac01b3f:::benchmark_data_smithi194_121512_object22467:head# 2024-04-29T14:26:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac05430:::benchmark_data_smithi194_121512_object51892:head# 2024-04-29T14:26:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac0ffd9:::benchmark_data_smithi194_121512_object32398:head# 2024-04-29T14:26:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac14f8d:::benchmark_data_smithi194_121512_object46818:head# 2024-04-29T14:26:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac1d816:::benchmark_data_smithi194_121512_object58953:head# 2024-04-29T14:26:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac1daf8:::benchmark_data_smithi194_121512_object58079:head# 2024-04-29T14:26:38.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac36bb7:::benchmark_data_smithi194_121512_object125357:head# 2024-04-29T14:26:38.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac3a9a3:::benchmark_data_smithi194_121512_object125752:head# 2024-04-29T14:26:38.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac3b1b3:::benchmark_data_smithi194_121512_object24914:head# 2024-04-29T14:26:38.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac41833:::benchmark_data_smithi194_121512_object115554:head# 2024-04-29T14:26:38.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac56ae5:::benchmark_data_smithi194_121512_object111238:head# 2024-04-29T14:26:38.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac5fd58:::benchmark_data_smithi194_121512_object40179:head# 2024-04-29T14:26:38.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac605af:::benchmark_data_smithi194_121512_object37664:head# 2024-04-29T14:26:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac838e3:::benchmark_data_smithi194_121512_object70700:head# 2024-04-29T14:26:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac88a26:::benchmark_data_smithi194_121512_object71152:head# 2024-04-29T14:26:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ac9130c:::benchmark_data_smithi194_121512_object123889:head# 2024-04-29T14:26:38.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aca7da5:::benchmark_data_smithi194_121512_object111273:head# 2024-04-29T14:26:38.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acb2336:::benchmark_data_smithi194_121512_object114035:head# 2024-04-29T14:26:38.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acc5994:::benchmark_data_smithi194_121512_object133419:head# 2024-04-29T14:26:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acc78e7:::benchmark_data_smithi194_121512_object62922:head# 2024-04-29T14:26:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6accccc5:::benchmark_data_smithi194_121512_object63926:head# 2024-04-29T14:26:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acce28c:::benchmark_data_smithi194_121512_object107871:head# 2024-04-29T14:26:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acd21dc:::benchmark_data_smithi194_121512_object69839:head# 2024-04-29T14:26:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acdd1c1:::benchmark_data_smithi194_121512_object90493:head# 2024-04-29T14:26:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ace6140:::benchmark_data_smithi194_121512_object78912:head# 2024-04-29T14:26:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ace7d4c:::benchmark_data_smithi194_121512_object128900:head# 2024-04-29T14:26:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acf6ab8:::benchmark_data_smithi194_121512_object56316:head# 2024-04-29T14:26:38.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acfd57f:::benchmark_data_smithi194_121512_object21158:head# 2024-04-29T14:26:38.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acfe96b:::benchmark_data_smithi194_121512_object44164:head# 2024-04-29T14:26:38.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6acfeafb:::benchmark_data_smithi194_121512_object19170:head# 2024-04-29T14:26:38.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad0ee45:::benchmark_data_smithi194_121512_object128159:head# 2024-04-29T14:26:38.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad105de:::benchmark_data_smithi194_121512_object123148:head# 2024-04-29T14:26:38.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad2a888:::benchmark_data_smithi194_121512_object41250:head# 2024-04-29T14:26:38.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad2f946:::benchmark_data_smithi194_121512_object83515:head# 2024-04-29T14:26:38.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad388a9:::benchmark_data_smithi194_121512_object26725:head# 2024-04-29T14:26:38.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad3f0c5:::benchmark_data_smithi194_121512_object121698:head# 2024-04-29T14:26:38.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad45341:::benchmark_data_smithi194_121512_object126700:head# 2024-04-29T14:26:38.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad48326:::benchmark_data_smithi194_121512_object39919:head# 2024-04-29T14:26:38.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad525fd:::benchmark_data_smithi194_121512_object124729:head# 2024-04-29T14:26:38.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad61749:::benchmark_data_smithi194_121512_object96657:head# 2024-04-29T14:26:38.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad67302:::benchmark_data_smithi194_121512_object49394:head# 2024-04-29T14:26:38.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad67d87:::benchmark_data_smithi194_121512_object1609:head# 2024-04-29T14:26:38.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad68a4f:::benchmark_data_smithi194_121512_object38822:head# 2024-04-29T14:26:38.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad6ac3c:::benchmark_data_smithi194_121512_object8114:head# 2024-04-29T14:26:38.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad7fbcf:::benchmark_data_smithi194_121512_object111971:head# 2024-04-29T14:26:38.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad90299:::benchmark_data_smithi194_121512_object51282:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad95290:::benchmark_data_smithi194_121512_object19553:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad953d5:::benchmark_data_smithi194_121512_object90940:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad95678:::benchmark_data_smithi194_121512_object99691:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad9cbd7:::benchmark_data_smithi194_121512_object50667:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad9cd7b:::benchmark_data_smithi194_121512_object116891:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ad9fc51:::benchmark_data_smithi194_121512_object9846:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ada6561:::benchmark_data_smithi194_121512_object89833:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adac22a:::benchmark_data_smithi194_121512_object51814:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adb2e44:::benchmark_data_smithi194_121512_object57708:head# 2024-04-29T14:26:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adb583b:::benchmark_data_smithi194_121512_object10880:head# 2024-04-29T14:26:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adb619e:::benchmark_data_smithi194_121512_object61167:head# 2024-04-29T14:26:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adbd250:::benchmark_data_smithi194_121512_object126315:head# 2024-04-29T14:26:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adc96a6:::benchmark_data_smithi194_121512_object6995:head# 2024-04-29T14:26:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6addf34a:::benchmark_data_smithi194_121512_object44646:head# 2024-04-29T14:26:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ade733e:::benchmark_data_smithi194_121512_object64091:head# 2024-04-29T14:26:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adf054b:::benchmark_data_smithi194_121512_object74965:head# 2024-04-29T14:26:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6adf271e:::benchmark_data_smithi194_121512_object58716:head# 2024-04-29T14:26:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae08146:::benchmark_data_smithi194_121512_object125959:head# 2024-04-29T14:26:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae0856a:::benchmark_data_smithi194_121512_object103599:head# 2024-04-29T14:26:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae10d01:::benchmark_data_smithi194_121512_object37295:head# 2024-04-29T14:26:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae10ddb:::benchmark_data_smithi194_121512_object82577:head# 2024-04-29T14:26:38.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae2e30a:::benchmark_data_smithi194_121512_object71141:head# 2024-04-29T14:26:38.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae362c0:::benchmark_data_smithi194_121512_object92897:head# 2024-04-29T14:26:38.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae3eae3:::benchmark_data_smithi194_121512_object129389:head# 2024-04-29T14:26:38.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae4c67d:::benchmark_data_smithi194_121512_object42013:head# 2024-04-29T14:26:38.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae58a4d:::benchmark_data_smithi194_121512_object70538:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae59483:::benchmark_data_smithi194_121512_object23529:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae69f58:::benchmark_data_smithi194_121512_object38116:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae80c27:::benchmark_data_smithi194_121512_object131813:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae81821:::benchmark_data_smithi194_121512_object66126:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae8c92b:::benchmark_data_smithi194_121512_object121725:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae98cea:::benchmark_data_smithi194_121512_object23855:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ae996dc:::benchmark_data_smithi194_121512_object103431:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aebe70a:::benchmark_data_smithi194_121512_object22919:head# 2024-04-29T14:26:38.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aecd78c:::benchmark_data_smithi194_121512_object130654:head# 2024-04-29T14:26:38.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aecf32d:::benchmark_data_smithi194_121512_object115979:head# 2024-04-29T14:26:38.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aeed11e:::benchmark_data_smithi194_121512_object30183:head# 2024-04-29T14:26:38.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aef2bdd:::benchmark_data_smithi194_121512_object85213:head# 2024-04-29T14:26:38.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aef42cd:::benchmark_data_smithi194_121512_object99194:head# 2024-04-29T14:26:38.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aefe604:::benchmark_data_smithi194_121512_object94327:head# 2024-04-29T14:26:38.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af27f2b:::benchmark_data_smithi194_121512_object69918:head# 2024-04-29T14:26:38.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af2c6cc:::benchmark_data_smithi194_121512_object7255:head# 2024-04-29T14:26:38.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af40a75:::benchmark_data_smithi194_121512_object91031:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af44955:::benchmark_data_smithi194_121512_object76090:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af613aa:::benchmark_data_smithi194_121512_object8174:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af61a30:::benchmark_data_smithi194_121512_object59563:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af62a8e:::benchmark_data_smithi194_121512_object92366:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af62c41:::benchmark_data_smithi194_121512_object46050:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af663c5:::benchmark_data_smithi194_121512_object22813:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af72701:::benchmark_data_smithi194_121512_object12634:head# 2024-04-29T14:26:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6af99cf4:::benchmark_data_smithi194_121512_object39356:head# 2024-04-29T14:26:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afacb9a:::benchmark_data_smithi194_121512_object123094:head# 2024-04-29T14:26:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afb2e8c:::benchmark_data_smithi194_121512_object70313:head# 2024-04-29T14:26:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afb9146:::benchmark_data_smithi194_121512_object6446:head# 2024-04-29T14:26:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afbcf8d:::benchmark_data_smithi194_121512_object55903:head# 2024-04-29T14:26:38.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afbcff8:::benchmark_data_smithi194_121512_object35001:head# 2024-04-29T14:26:38.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afbdfa1:::benchmark_data_smithi194_121512_object23443:head# 2024-04-29T14:26:38.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afc0eb8:::benchmark_data_smithi194_121512_object84807:head# 2024-04-29T14:26:38.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afc532f:::benchmark_data_smithi194_121512_object26800:head# 2024-04-29T14:26:38.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afdf17c:::benchmark_data_smithi194_121512_object9891:head# 2024-04-29T14:26:38.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afe2525:::benchmark_data_smithi194_121512_object111345:head# 2024-04-29T14:26:38.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afe3a51:::benchmark_data_smithi194_121512_object32167:head# 2024-04-29T14:26:38.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afe8db4:::benchmark_data_smithi194_121512_object92848:head# 2024-04-29T14:26:38.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6afedf88:::benchmark_data_smithi194_121512_object84155:head# 2024-04-29T14:26:38.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aff393a:::benchmark_data_smithi194_121512_object110694:head# 2024-04-29T14:26:38.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aff71c6:::benchmark_data_smithi194_121512_object107518:head# 2024-04-29T14:26:38.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6aff85eb:::benchmark_data_smithi194_121512_object115509:head# 2024-04-29T14:26:38.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b000236:::benchmark_data_smithi194_121512_object68236:head# 2024-04-29T14:26:38.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0024cc:::benchmark_data_smithi194_121512_object29653:head# 2024-04-29T14:26:38.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b00c9a8:::benchmark_data_smithi194_121512_object92802:head# 2024-04-29T14:26:38.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b016e52:::benchmark_data_smithi194_121512_object122049:head# 2024-04-29T14:26:38.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0255be:::benchmark_data_smithi194_121512_object119196:head# 2024-04-29T14:26:38.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b026667:::benchmark_data_smithi194_121512_object6299:head# 2024-04-29T14:26:38.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b02fb50:::benchmark_data_smithi194_121512_object41644:head# 2024-04-29T14:26:38.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b032abc:::benchmark_data_smithi194_121512_object122186:head# 2024-04-29T14:26:38.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0355a8:::benchmark_data_smithi194_121512_object94006:head# 2024-04-29T14:26:38.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b03926a:::benchmark_data_smithi194_121512_object35195:head# 2024-04-29T14:26:38.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b039a01:::benchmark_data_smithi194_121512_object119118:head# 2024-04-29T14:26:38.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b03c764:::benchmark_data_smithi194_121512_object65321:head# 2024-04-29T14:26:38.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b049b70:::benchmark_data_smithi194_121512_object133085:head# 2024-04-29T14:26:38.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b053aaa:::benchmark_data_smithi194_121512_object32289:head# 2024-04-29T14:26:38.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b055f45:::benchmark_data_smithi194_121512_object67246:head# 2024-04-29T14:26:38.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b05a143:::benchmark_data_smithi194_121512_object28076:head# 2024-04-29T14:26:38.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b060ccf:::benchmark_data_smithi194_121512_object25968:head# 2024-04-29T14:26:38.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b065228:::benchmark_data_smithi194_121512_object19315:head# 2024-04-29T14:26:38.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b068a28:::benchmark_data_smithi194_121512_object120605:head# 2024-04-29T14:26:38.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b06e74e:::benchmark_data_smithi194_121512_object10424:head# 2024-04-29T14:26:38.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b06fb02:::benchmark_data_smithi194_121512_object95566:head# 2024-04-29T14:26:38.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b070545:::benchmark_data_smithi194_121512_object74909:head# 2024-04-29T14:26:38.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b078230:::benchmark_data_smithi194_121512_object45371:head# 2024-04-29T14:26:38.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b097b95:::benchmark_data_smithi194_121512_object71705:head# 2024-04-29T14:26:38.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b09bd42:::benchmark_data_smithi194_121512_object28855:head# 2024-04-29T14:26:38.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b09c802:::benchmark_data_smithi194_121512_object95626:head# 2024-04-29T14:26:38.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0a7c66:::benchmark_data_smithi194_121512_object9937:head# 2024-04-29T14:26:38.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0b6991:::benchmark_data_smithi194_121512_object122150:head# 2024-04-29T14:26:38.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0be9ea:::benchmark_data_smithi194_121512_object92796:head# 2024-04-29T14:26:38.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0ceae7:::benchmark_data_smithi194_121512_object36312:head# 2024-04-29T14:26:38.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0d39b7:::benchmark_data_smithi194_121512_object83665:head# 2024-04-29T14:26:38.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0d3cc8:::benchmark_data_smithi194_121512_object39297:head# 2024-04-29T14:26:38.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0d3e9d:::benchmark_data_smithi194_121512_object121348:head# 2024-04-29T14:26:38.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0e6409:::benchmark_data_smithi194_121512_object90163:head# 2024-04-29T14:26:38.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0e9047:::benchmark_data_smithi194_121512_object1120:head# 2024-04-29T14:26:38.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0eb5e6:::benchmark_data_smithi194_121512_object80368:head# 2024-04-29T14:26:38.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b0f8767:::benchmark_data_smithi194_121512_object28705:head# 2024-04-29T14:26:38.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b107c0e:::benchmark_data_smithi194_121512_object29655:head# 2024-04-29T14:26:38.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1097a8:::benchmark_data_smithi194_121512_object96190:head# 2024-04-29T14:26:38.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b117320:::benchmark_data_smithi194_121512_object96710:head# 2024-04-29T14:26:38.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b12b08a:::benchmark_data_smithi194_121512_object26779:head# 2024-04-29T14:26:38.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b12e0f1:::benchmark_data_smithi194_121512_object28589:head# 2024-04-29T14:26:38.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b141686:::benchmark_data_smithi194_121512_object93188:head# 2024-04-29T14:26:38.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b154962:::benchmark_data_smithi194_121512_object126920:head# 2024-04-29T14:26:38.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b155623:::benchmark_data_smithi194_121512_object116713:head# 2024-04-29T14:26:38.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b15bd7c:::benchmark_data_smithi194_121512_object51066:head# 2024-04-29T14:26:38.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1709ab:::benchmark_data_smithi194_121512_object64508:head# 2024-04-29T14:26:38.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b17103a:::benchmark_data_smithi194_121512_object106366:head# 2024-04-29T14:26:38.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1848bc:::benchmark_data_smithi194_121512_object109804:head# 2024-04-29T14:26:38.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b18ba32:::benchmark_data_smithi194_121512_object120885:head# 2024-04-29T14:26:38.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b18ca5c:::benchmark_data_smithi194_121512_object61994:head# 2024-04-29T14:26:38.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1a6895:::benchmark_data_smithi194_121512_object100791:head# 2024-04-29T14:26:38.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1b0495:::benchmark_data_smithi194_121512_object1847:head# 2024-04-29T14:26:38.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1ba0bc:::benchmark_data_smithi194_121512_object64397:head# 2024-04-29T14:26:38.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:38.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1d8e5f:::benchmark_data_smithi194_121512_object48433:head# 2024-04-29T14:26:38.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1f410d:::benchmark_data_smithi194_121512_object111071:head# 2024-04-29T14:26:38.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b1fbb41:::benchmark_data_smithi194_121512_object130131:head# 2024-04-29T14:26:38.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b20417b:::benchmark_data_smithi194_121512_object9449:head# 2024-04-29T14:26:38.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b209d91:::benchmark_data_smithi194_121512_object21281:head# 2024-04-29T14:26:38.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b2174bf:::benchmark_data_smithi194_121512_object129724:head# 2024-04-29T14:26:38.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b21f179:::benchmark_data_smithi194_121512_object122329:head# 2024-04-29T14:26:38.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b21fe5f:::benchmark_data_smithi194_121512_object37264:head# 2024-04-29T14:26:38.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b221791:::benchmark_data_smithi194_121512_object102833:head# 2024-04-29T14:26:38.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b22201f:::benchmark_data_smithi194_121512_object44916:head# 2024-04-29T14:26:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b22af07:::benchmark_data_smithi194_121512_object29513:head# 2024-04-29T14:26:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b22d0b2:::benchmark_data_smithi194_121512_object49228:head# 2024-04-29T14:26:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b239fb2:::benchmark_data_smithi194_121512_object25140:head# 2024-04-29T14:26:38.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b23ad89:::benchmark_data_smithi194_121512_object28152:head# 2024-04-29T14:26:38.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b244c9d:::benchmark_data_smithi194_121512_object121835:head# 2024-04-29T14:26:38.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b25dd0e:::benchmark_data_smithi194_121512_object81526:head# 2024-04-29T14:26:38.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b268a4a:::benchmark_data_smithi194_121512_object78523:head# 2024-04-29T14:26:38.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b26e6b6:::benchmark_data_smithi194_121512_object96092:head# 2024-04-29T14:26:38.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b2735b9:::benchmark_data_smithi194_121512_object83992:head# 2024-04-29T14:26:38.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b278634:::benchmark_data_smithi194_121512_object23169:head# 2024-04-29T14:26:38.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b288201:::benchmark_data_smithi194_121512_object51631:head# 2024-04-29T14:26:38.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b289ef5:::benchmark_data_smithi194_121512_object130774:head# 2024-04-29T14:26:38.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b28f435:::benchmark_data_smithi194_121512_object56511:head# 2024-04-29T14:26:38.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b292c29:::benchmark_data_smithi194_121512_object39793:head# 2024-04-29T14:26:38.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b294c7e:::benchmark_data_smithi194_121512_object99500:head# 2024-04-29T14:26:38.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b296840:::benchmark_data_smithi194_121512_object67453:head# 2024-04-29T14:26:38.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b298e57:::benchmark_data_smithi194_121512_object50188:head# 2024-04-29T14:26:38.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b2a6312:::benchmark_data_smithi194_121512_object96574:head# 2024-04-29T14:26:38.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b2aaeaa:::benchmark_data_smithi194_121512_object9285:head# 2024-04-29T14:26:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b2d232c:::benchmark_data_smithi194_121512_object13278:head# 2024-04-29T14:26:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b2f44f7:::benchmark_data_smithi194_121512_object99317:head# 2024-04-29T14:26:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b2ff1f4:::benchmark_data_smithi194_121512_object63598:head# 2024-04-29T14:26:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3064c6:::benchmark_data_smithi194_121512_object8124:head# 2024-04-29T14:26:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b311962:::benchmark_data_smithi194_121512_object124686:head# 2024-04-29T14:26:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b31625e:::benchmark_data_smithi194_121512_object36040:head# 2024-04-29T14:26:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b316730:::benchmark_data_smithi194_121512_object132369:head# 2024-04-29T14:26:38.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b319793:::benchmark_data_smithi194_121512_object83461:head# 2024-04-29T14:26:38.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b33690f:::benchmark_data_smithi194_121512_object21280:head# 2024-04-29T14:26:38.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b342042:::benchmark_data_smithi194_121512_object62388:head# 2024-04-29T14:26:38.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b344f3f:::benchmark_data_smithi194_121512_object84039:head# 2024-04-29T14:26:38.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b34c675:::benchmark_data_smithi194_121512_object60269:head# 2024-04-29T14:26:38.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b351abe:::benchmark_data_smithi194_121512_object9443:head# 2024-04-29T14:26:38.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b35ed43:::benchmark_data_smithi194_121512_object112894:head# 2024-04-29T14:26:38.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3635e3:::benchmark_data_smithi194_121512_object119549:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3866a9:::benchmark_data_smithi194_121512_object130274:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b389dfd:::benchmark_data_smithi194_121512_object10289:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b38a664:::benchmark_data_smithi194_121512_object54242:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b392502:::benchmark_data_smithi194_121512_object39617:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b396b90:::benchmark_data_smithi194_121512_object85870:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b39c35f:::benchmark_data_smithi194_121512_object60102:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3ba669:::benchmark_data_smithi194_121512_object120252:head# 2024-04-29T14:26:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3c052a:::benchmark_data_smithi194_121512_object1703:head# 2024-04-29T14:26:38.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3c7b68:::benchmark_data_smithi194_121512_object30388:head# 2024-04-29T14:26:38.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3cbfa9:::benchmark_data_smithi194_121512_object47314:head# 2024-04-29T14:26:38.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3d5645:::benchmark_data_smithi194_121512_object103543:head# 2024-04-29T14:26:38.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3d59d8:::benchmark_data_smithi194_121512_object35153:head# 2024-04-29T14:26:38.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3d8493:::benchmark_data_smithi194_121512_object68926:head# 2024-04-29T14:26:38.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3e7cdc:::benchmark_data_smithi194_121512_object128245:head# 2024-04-29T14:26:38.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3e931b:::benchmark_data_smithi194_121512_object130872:head# 2024-04-29T14:26:38.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3ee1af:::benchmark_data_smithi194_121512_object10637:head# 2024-04-29T14:26:38.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b3f0105:::benchmark_data_smithi194_121512_object99936:head# 2024-04-29T14:26:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b408b2e:::benchmark_data_smithi194_121512_object67769:head# 2024-04-29T14:26:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b40d350:::benchmark_data_smithi194_121512_object30532:head# 2024-04-29T14:26:38.436 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:38.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b417e11:::benchmark_data_smithi194_121512_object45395:head# 2024-04-29T14:26:38.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b423d61:::benchmark_data_smithi194_121512_object29493:head# 2024-04-29T14:26:38.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b429ef0:::benchmark_data_smithi194_121512_object1616:head# 2024-04-29T14:26:38.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b42aac0:::benchmark_data_smithi194_121512_object44110:head# 2024-04-29T14:26:38.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b42e251:::benchmark_data_smithi194_121512_object95138:head# 2024-04-29T14:26:38.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b432923:::benchmark_data_smithi194_121512_object35132:head# 2024-04-29T14:26:38.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b435579:::benchmark_data_smithi194_121512_object19527:head# 2024-04-29T14:26:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b43a02a:::benchmark_data_smithi194_121512_object1604:head# 2024-04-29T14:26:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b44dd8e:::benchmark_data_smithi194_121512_object129605:head# 2024-04-29T14:26:38.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b44ecf4:::benchmark_data_smithi194_121512_object44530:head# 2024-04-29T14:26:38.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b450e70:::benchmark_data_smithi194_121512_object71177:head# 2024-04-29T14:26:38.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b452f3a:::benchmark_data_smithi194_121512_object79950:head# 2024-04-29T14:26:38.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b45a489:::benchmark_data_smithi194_121512_object79955:head# 2024-04-29T14:26:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4622b7:::benchmark_data_smithi194_121512_object97177:head# 2024-04-29T14:26:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4625b4:::benchmark_data_smithi194_121512_object18231:head# 2024-04-29T14:26:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b46842e:::benchmark_data_smithi194_121512_object123994:head# 2024-04-29T14:26:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b471755:::benchmark_data_smithi194_121512_object61991:head# 2024-04-29T14:26:38.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b473236:::benchmark_data_smithi194_121512_object6428:head# 2024-04-29T14:26:38.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b47ce8a:::benchmark_data_smithi194_121512_object3509:head# 2024-04-29T14:26:38.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b47ed78:::benchmark_data_smithi194_121512_object118791:head# 2024-04-29T14:26:38.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b48bf92:::benchmark_data_smithi194_121512_object123738:head# 2024-04-29T14:26:38.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b498330:::benchmark_data_smithi194_121512_object13726:head# 2024-04-29T14:26:38.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b49be40:::benchmark_data_smithi194_121512_object80580:head# 2024-04-29T14:26:38.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4a29b2:::benchmark_data_smithi194_121512_object111424:head# 2024-04-29T14:26:38.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4a5892:::benchmark_data_smithi194_121512_object128106:head# 2024-04-29T14:26:38.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4bfb98:::benchmark_data_smithi194_121512_object102562:head# 2024-04-29T14:26:38.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4cf0f1:::benchmark_data_smithi194_121512_object108070:head# 2024-04-29T14:26:38.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4cf879:::benchmark_data_smithi194_121512_object66784:head# 2024-04-29T14:26:38.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4d3759:::benchmark_data_smithi194_121512_object71858:head# 2024-04-29T14:26:38.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4d4879:::benchmark_data_smithi194_121512_object103037:head# 2024-04-29T14:26:38.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4d9e6f:::benchmark_data_smithi194_121512_object84176:head# 2024-04-29T14:26:38.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4fe671:::benchmark_data_smithi194_121512_object119745:head# 2024-04-29T14:26:38.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b4fe802:::benchmark_data_smithi194_121512_object117799:head# 2024-04-29T14:26:38.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b50df77:::benchmark_data_smithi194_121512_object58754:head# 2024-04-29T14:26:38.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b522962:::benchmark_data_smithi194_121512_object13251:head# 2024-04-29T14:26:38.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b524498:::benchmark_data_smithi194_121512_object70929:head# 2024-04-29T14:26:38.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b52c54c:::benchmark_data_smithi194_121512_object100994:head# 2024-04-29T14:26:38.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b52f245:::benchmark_data_smithi194_121512_object106264:head# 2024-04-29T14:26:38.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b52f95c:::benchmark_data_smithi194_121512_object19092:head# 2024-04-29T14:26:38.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b53effd:::benchmark_data_smithi194_121512_object91986:head# 2024-04-29T14:26:38.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b54085c:::benchmark_data_smithi194_121512_object129742:head# 2024-04-29T14:26:38.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b545b45:::benchmark_data_smithi194_121512_object33333:head# 2024-04-29T14:26:38.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b550050:::benchmark_data_smithi194_121512_object74597:head# 2024-04-29T14:26:38.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b550a81:::benchmark_data_smithi194_121512_object116178:head# 2024-04-29T14:26:38.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b55eae5:::benchmark_data_smithi194_121512_object22274:head# 2024-04-29T14:26:38.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5651ea:::benchmark_data_smithi194_121512_object90760:head# 2024-04-29T14:26:38.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b568fb7:::benchmark_data_smithi194_121512_object93173:head# 2024-04-29T14:26:38.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b56fc57:::benchmark_data_smithi194_121512_object15466:head# 2024-04-29T14:26:38.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b575141:::benchmark_data_smithi194_121512_object101542:head# 2024-04-29T14:26:38.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b577cb8:::benchmark_data_smithi194_121512_object29981:head# 2024-04-29T14:26:38.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5842c7:::benchmark_data_smithi194_121512_object22282:head# 2024-04-29T14:26:38.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b584fc0:::benchmark_data_smithi194_121512_object23090:head# 2024-04-29T14:26:38.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5867ed:::benchmark_data_smithi194_121512_object63345:head# 2024-04-29T14:26:38.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b58ab2d:::benchmark_data_smithi194_121512_object96343:head# 2024-04-29T14:26:38.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b58ba4e:::benchmark_data_smithi194_121512_object125322:head# 2024-04-29T14:26:38.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5943b8:::benchmark_data_smithi194_121512_object86023:head# 2024-04-29T14:26:38.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b597ea7:::benchmark_data_smithi194_121512_object96632:head# 2024-04-29T14:26:38.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5988b6:::benchmark_data_smithi194_121512_object19366:head# 2024-04-29T14:26:38.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b59d0d4:::benchmark_data_smithi194_121512_object131658:head# 2024-04-29T14:26:38.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b59dd82:::benchmark_data_smithi194_121512_object104483:head# 2024-04-29T14:26:38.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5a004d:::benchmark_data_smithi194_121512_object27963:head# 2024-04-29T14:26:38.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5a2304:::benchmark_data_smithi194_121512_object20133:head# 2024-04-29T14:26:38.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5abdbf:::benchmark_data_smithi194_121512_object128296:head# 2024-04-29T14:26:38.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5aebb4:::benchmark_data_smithi194_121512_object27823:head# 2024-04-29T14:26:38.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5bbe27:::benchmark_data_smithi194_121512_object47228:head# 2024-04-29T14:26:38.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5be0ea:::benchmark_data_smithi194_121512_object96517:head# 2024-04-29T14:26:38.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5bf316:::benchmark_data_smithi194_121512_object36959:head# 2024-04-29T14:26:38.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5c58ad:::benchmark_data_smithi194_121512_object11502:head# 2024-04-29T14:26:38.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5c59f8:::benchmark_data_smithi194_121512_object7647:head# 2024-04-29T14:26:38.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5cb878:::benchmark_data_smithi194_121512_object119653:head# 2024-04-29T14:26:38.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5d33f7:::benchmark_data_smithi194_121512_object113057:head# 2024-04-29T14:26:38.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5d43e0:::benchmark_data_smithi194_121512_object84871:head# 2024-04-29T14:26:38.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5d73dd:::benchmark_data_smithi194_121512_object76526:head# 2024-04-29T14:26:38.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5def71:::benchmark_data_smithi194_121512_object77432:head# 2024-04-29T14:26:38.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5e6a57:::benchmark_data_smithi194_121512_object31183:head# 2024-04-29T14:26:38.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5e8efe:::benchmark_data_smithi194_121512_object82983:head# 2024-04-29T14:26:38.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5ee1d1:::benchmark_data_smithi194_121512_object27629:head# 2024-04-29T14:26:38.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5f8970:::benchmark_data_smithi194_121512_object92363:head# 2024-04-29T14:26:38.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5f9306:::benchmark_data_smithi194_121512_object74050:head# 2024-04-29T14:26:38.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b5f94b8:::benchmark_data_smithi194_121512_object16996:head# 2024-04-29T14:26:38.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b617677:::benchmark_data_smithi194_121512_object101934:head# 2024-04-29T14:26:38.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b61abc8:::benchmark_data_smithi194_121512_object58330:head# 2024-04-29T14:26:38.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b61e5c3:::benchmark_data_smithi194_121512_object72690:head# 2024-04-29T14:26:38.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b62bf4d:::benchmark_data_smithi194_121512_object115895:head# 2024-04-29T14:26:38.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6320f5:::benchmark_data_smithi194_121512_object27577:head# 2024-04-29T14:26:38.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b633ae1:::benchmark_data_smithi194_121512_object116789:head# 2024-04-29T14:26:38.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b641550:::benchmark_data_smithi194_121512_object22771:head# 2024-04-29T14:26:38.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b64283d:::benchmark_data_smithi194_121512_object79777:head# 2024-04-29T14:26:38.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b64b2ce:::benchmark_data_smithi194_121512_object36444:head# 2024-04-29T14:26:38.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b653a1b:::benchmark_data_smithi194_121512_object78975:head# 2024-04-29T14:26:38.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b65e041:::benchmark_data_smithi194_121512_object8982:head# 2024-04-29T14:26:38.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b65fdc5:::benchmark_data_smithi194_121512_object123261:head# 2024-04-29T14:26:38.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6885cd:::benchmark_data_smithi194_121512_object70520:head# 2024-04-29T14:26:38.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b690959:::benchmark_data_smithi194_121512_object58373:head# 2024-04-29T14:26:38.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b69c9c2:::benchmark_data_smithi194_121512_object35093:head# 2024-04-29T14:26:38.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6a0d89:::benchmark_data_smithi194_121512_object5884:head# 2024-04-29T14:26:38.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6a7102:::benchmark_data_smithi194_121512_object72269:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6b8646:::benchmark_data_smithi194_121512_object100878:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6caadf:::benchmark_data_smithi194_121512_object5022:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6cad8b:::benchmark_data_smithi194_121512_object59086:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6cbdfe:::benchmark_data_smithi194_121512_object26876:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6cd16f:::benchmark_data_smithi194_121512_object5733:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6d1747:::benchmark_data_smithi194_121512_object43318:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6d69e3:::benchmark_data_smithi194_121512_object92707:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6da370:::benchmark_data_smithi194_121512_object32093:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6df7c9:::benchmark_data_smithi194_121512_object99128:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6ed4a7:::benchmark_data_smithi194_121512_object69059:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b6f61cc:::benchmark_data_smithi194_121512_object87093:head# 2024-04-29T14:26:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b706344:::benchmark_data_smithi194_121512_object71180:head# 2024-04-29T14:26:38.497 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:26:38.497 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:26:38.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b70665a:::benchmark_data_smithi194_121512_object66312:head# 2024-04-29T14:26:38.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b70bb44:::benchmark_data_smithi194_121512_object113661:head# 2024-04-29T14:26:38.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7146f3:::benchmark_data_smithi194_121512_object100157:head# 2024-04-29T14:26:38.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b71739c:::benchmark_data_smithi194_121512_object49439:head# 2024-04-29T14:26:38.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b71b124:::benchmark_data_smithi194_121512_object56564:head# 2024-04-29T14:26:38.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7258be:::benchmark_data_smithi194_121512_object83481:head# 2024-04-29T14:26:38.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b72d70a:::benchmark_data_smithi194_121512_object100714:head# 2024-04-29T14:26:38.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b730a5f:::benchmark_data_smithi194_121512_object48485:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b733fde:::benchmark_data_smithi194_121512_object46597:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7394be:::benchmark_data_smithi194_121512_object32053:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b73ebba:::benchmark_data_smithi194_121512_object39415:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b745656:::benchmark_data_smithi194_121512_object80625:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b74ea5d:::benchmark_data_smithi194_121512_object39529:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b74f565:::benchmark_data_smithi194_121512_object61148:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7564b8:::benchmark_data_smithi194_121512_object24980:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b75aca5:::benchmark_data_smithi194_121512_object42572:head# 2024-04-29T14:26:38.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b75c492:::benchmark_data_smithi194_121512_object29380:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b76ec0e:::benchmark_data_smithi194_121512_object4052:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7751cf:::benchmark_data_smithi194_121512_object93440:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7837c2:::benchmark_data_smithi194_121512_object17771:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7963f2:::benchmark_data_smithi194_121512_object92788:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b79f2ca:::benchmark_data_smithi194_121512_object2835:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7a316f:::benchmark_data_smithi194_121512_object13312:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7ab99d:::benchmark_data_smithi194_121512_object44885:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7ae9d7:::benchmark_data_smithi194_121512_object107360:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7ba5fe:::benchmark_data_smithi194_121512_object50139:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7bd933:::benchmark_data_smithi194_121512_object111290:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7c9006:::benchmark_data_smithi194_121512_object90475:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7cb07c:::benchmark_data_smithi194_121512_object78095:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7cd2b3:::benchmark_data_smithi194_121512_object26436:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7cec4c:::benchmark_data_smithi194_121512_object126166:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7d02a6:::benchmark_data_smithi194_121512_object46740:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7d238f:::benchmark_data_smithi194_121512_object5299:head# 2024-04-29T14:26:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7df99e:::benchmark_data_smithi194_121512_object37051:head# 2024-04-29T14:26:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7ea31f:::benchmark_data_smithi194_121512_object9706:head# 2024-04-29T14:26:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7ecc12:::benchmark_data_smithi194_121512_object69312:head# 2024-04-29T14:26:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7eeff6:::benchmark_data_smithi194_121512_object140:head# 2024-04-29T14:26:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b7f4445:::benchmark_data_smithi194_121512_object84854:head# 2024-04-29T14:26:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b810d69:::benchmark_data_smithi194_121512_object9266:head# 2024-04-29T14:26:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b811d12:::benchmark_data_smithi194_121512_object17099:head# 2024-04-29T14:26:38.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b816819:::benchmark_data_smithi194_121512_object44288:head# 2024-04-29T14:26:38.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b828b57:::benchmark_data_smithi194_121512_object103251:head# 2024-04-29T14:26:38.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b828ff8:::benchmark_data_smithi194_121512_object14801:head# 2024-04-29T14:26:38.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b83e56f:::benchmark_data_smithi194_121512_object29171:head# 2024-04-29T14:26:38.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b83f66e:::benchmark_data_smithi194_121512_object68073:head# 2024-04-29T14:26:38.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8464c3:::benchmark_data_smithi194_121512_object36432:head# 2024-04-29T14:26:38.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b84849e:::benchmark_data_smithi194_121512_object26862:head# 2024-04-29T14:26:38.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b84d278:::benchmark_data_smithi194_121512_object27802:head# 2024-04-29T14:26:38.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b84f566:::benchmark_data_smithi194_121512_object35584:head# 2024-04-29T14:26:38.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b87336a:::benchmark_data_smithi194_121512_object133025:head# 2024-04-29T14:26:38.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8760a1:::benchmark_data_smithi194_121512_object95645:head# 2024-04-29T14:26:38.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b87a900:::benchmark_data_smithi194_121512_object98570:head# 2024-04-29T14:26:38.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b87dd16:::benchmark_data_smithi194_121512_object113315:head# 2024-04-29T14:26:38.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b886681:::benchmark_data_smithi194_121512_object56390:head# 2024-04-29T14:26:38.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8875bc:::benchmark_data_smithi194_121512_object56985:head# 2024-04-29T14:26:38.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b888587:::benchmark_data_smithi194_121512_object22672:head# 2024-04-29T14:26:38.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b893037:::benchmark_data_smithi194_121512_object118341:head# 2024-04-29T14:26:38.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b89f6d3:::benchmark_data_smithi194_121512_object112735:head# 2024-04-29T14:26:38.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b89ff6e:::benchmark_data_smithi194_121512_object79642:head# 2024-04-29T14:26:38.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8a14a0:::benchmark_data_smithi194_121512_object121905:head# 2024-04-29T14:26:38.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8a838a:::benchmark_data_smithi194_121512_object77042:head# 2024-04-29T14:26:38.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8bd383:::benchmark_data_smithi194_121512_object30879:head# 2024-04-29T14:26:38.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8c50f3:::benchmark_data_smithi194_121512_object37415:head# 2024-04-29T14:26:38.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8c55de:::benchmark_data_smithi194_121512_object40900:head# 2024-04-29T14:26:38.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8ca20e:::benchmark_data_smithi194_121512_object20208:head# 2024-04-29T14:26:38.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8ca932:::benchmark_data_smithi194_121512_object34810:head# 2024-04-29T14:26:38.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8dc108:::benchmark_data_smithi194_121512_object120880:head# 2024-04-29T14:26:38.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8de8b3:::benchmark_data_smithi194_121512_object101715:head# 2024-04-29T14:26:38.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8ec076:::benchmark_data_smithi194_121512_object59762:head# 2024-04-29T14:26:38.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8efec0:::benchmark_data_smithi194_121512_object89642:head# 2024-04-29T14:26:38.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8f8ecd:::benchmark_data_smithi194_121512_object106071:head# 2024-04-29T14:26:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8fbf72:::benchmark_data_smithi194_121512_object21792:head# 2024-04-29T14:26:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8fbffb:::benchmark_data_smithi194_121512_object120120:head# 2024-04-29T14:26:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b8fe2be:::benchmark_data_smithi194_121512_object47379:head# 2024-04-29T14:26:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9009c5:::benchmark_data_smithi194_121512_object124870:head# 2024-04-29T14:26:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9016c2:::benchmark_data_smithi194_121512_object45186:head# 2024-04-29T14:26:38.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b902dcb:::benchmark_data_smithi194_121512_object121651:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b906c2b:::benchmark_data_smithi194_121512_object57498:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b90cf5b:::benchmark_data_smithi194_121512_object55293:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b910384:::benchmark_data_smithi194_121512_object61809:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b91905e:::benchmark_data_smithi194_121512_object82245:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b91d023:::benchmark_data_smithi194_121512_object4865:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b925ac3:::benchmark_data_smithi194_121512_object36332:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9268e6:::benchmark_data_smithi194_121512_object26369:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b928cdc:::benchmark_data_smithi194_121512_object115682:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b928ebe:::benchmark_data_smithi194_121512_object73370:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b934838:::benchmark_data_smithi194_121512_object101523:head# 2024-04-29T14:26:38.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9610d5:::benchmark_data_smithi194_121512_object15288:head# 2024-04-29T14:26:38.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b961982:::benchmark_data_smithi194_121512_object113080:head# 2024-04-29T14:26:38.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b964cc9:::benchmark_data_smithi194_121512_object110046:head# 2024-04-29T14:26:38.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b97fd37:::benchmark_data_smithi194_121512_object7984:head# 2024-04-29T14:26:38.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b98b334:::benchmark_data_smithi194_121512_object13566:head# 2024-04-29T14:26:38.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b991530:::benchmark_data_smithi194_121512_object22626:head# 2024-04-29T14:26:38.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b996f64:::benchmark_data_smithi194_121512_object129379:head# 2024-04-29T14:26:38.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b99a6b4:::benchmark_data_smithi194_121512_object39685:head# 2024-04-29T14:26:38.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9a28d5:::benchmark_data_smithi194_121512_object85464:head# 2024-04-29T14:26:38.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9a7fd3:::benchmark_data_smithi194_121512_object132426:head# 2024-04-29T14:26:38.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9b2003:::benchmark_data_smithi194_121512_object51476:head# 2024-04-29T14:26:38.549 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:38 smithi121 ceph-mon[44533]: osdmap e475: 12 total, 11 up, 7 in 2024-04-29T14:26:38.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9b2cb2:::benchmark_data_smithi194_121512_object55279:head# 2024-04-29T14:26:38.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9cd780:::benchmark_data_smithi194_121512_object31032:head# 2024-04-29T14:26:38.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9d668d:::benchmark_data_smithi194_121512_object47273:head# 2024-04-29T14:26:38.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9df81c:::benchmark_data_smithi194_121512_object519:head# 2024-04-29T14:26:38.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9e008d:::benchmark_data_smithi194_121512_object51973:head# 2024-04-29T14:26:38.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9e02c2:::benchmark_data_smithi194_121512_object110173:head# 2024-04-29T14:26:38.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9e9fc3:::benchmark_data_smithi194_121512_object85360:head# 2024-04-29T14:26:38.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6b9ff100:::benchmark_data_smithi194_121512_object96172:head# 2024-04-29T14:26:38.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba08144:::benchmark_data_smithi194_121512_object123450:head# 2024-04-29T14:26:38.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba09552:::benchmark_data_smithi194_121512_object1911:head# 2024-04-29T14:26:38.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba1483d:::benchmark_data_smithi194_121512_object900:head# 2024-04-29T14:26:38.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba14dbd:::benchmark_data_smithi194_121512_object44088:head# 2024-04-29T14:26:38.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba1ba22:::benchmark_data_smithi194_121512_object119003:head# 2024-04-29T14:26:38.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba1ed8f:::benchmark_data_smithi194_121512_object49048:head# 2024-04-29T14:26:38.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba2d8f1:::benchmark_data_smithi194_121512_object55540:head# 2024-04-29T14:26:38.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba34a15:::benchmark_data_smithi194_121512_object354:head# 2024-04-29T14:26:38.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba3b4f3:::benchmark_data_smithi194_121512_object65378:head# 2024-04-29T14:26:38.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba465f9:::benchmark_data_smithi194_121512_object116083:head# 2024-04-29T14:26:38.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba5784c:::benchmark_data_smithi194_121512_object51981:head# 2024-04-29T14:26:38.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba5e11e:::benchmark_data_smithi194_121512_object46565:head# 2024-04-29T14:26:38.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba675c5:::benchmark_data_smithi194_121512_object104441:head# 2024-04-29T14:26:38.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba6f469:::benchmark_data_smithi194_121512_object88154:head# 2024-04-29T14:26:38.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba71901:::benchmark_data_smithi194_121512_object6720:head# 2024-04-29T14:26:38.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba7d4cd:::benchmark_data_smithi194_121512_object22405:head# 2024-04-29T14:26:38.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba87b69:::benchmark_data_smithi194_121512_object52366:head# 2024-04-29T14:26:38.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba89351:::benchmark_data_smithi194_121512_object113288:head# 2024-04-29T14:26:38.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba8b1f2:::benchmark_data_smithi194_121512_object32212:head# 2024-04-29T14:26:38.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba8c6e8:::benchmark_data_smithi194_121512_object131591:head# 2024-04-29T14:26:38.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba8ff98:::benchmark_data_smithi194_121512_object81531:head# 2024-04-29T14:26:38.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba923ce:::benchmark_data_smithi194_121512_object90247:head# 2024-04-29T14:26:38.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba99c1e:::benchmark_data_smithi194_121512_object8698:head# 2024-04-29T14:26:38.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ba9c6f6:::benchmark_data_smithi194_121512_object16295:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6babc12f:::benchmark_data_smithi194_121512_object41794:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bacbdf9:::benchmark_data_smithi194_121512_object97380:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bacf70d:::benchmark_data_smithi194_121512_object7769:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6badd56a:::benchmark_data_smithi194_121512_object16353:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6badfee5:::benchmark_data_smithi194_121512_object86261:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bae1c2d:::benchmark_data_smithi194_121512_object133171:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bae2b47:::benchmark_data_smithi194_121512_object238:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6baed100:::benchmark_data_smithi194_121512_object32341:head# 2024-04-29T14:26:38.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb12ac5:::benchmark_data_smithi194_121512_object26853:head# 2024-04-29T14:26:38.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb15531:::benchmark_data_smithi194_121512_object34123:head# 2024-04-29T14:26:38.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb17a73:::benchmark_data_smithi194_121512_object99135:head# 2024-04-29T14:26:38.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb38d24:::benchmark_data_smithi194_121512_object40408:head# 2024-04-29T14:26:38.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb43566:::benchmark_data_smithi194_121512_object30670:head# 2024-04-29T14:26:38.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb5e4c0:::benchmark_data_smithi194_121512_object58844:head# 2024-04-29T14:26:38.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb60a0d:::benchmark_data_smithi194_121512_object110674:head# 2024-04-29T14:26:38.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb78d95:::benchmark_data_smithi194_121512_object2205:head# 2024-04-29T14:26:38.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb89c50:::benchmark_data_smithi194_121512_object33313:head# 2024-04-29T14:26:38.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb8ac05:::benchmark_data_smithi194_121512_object22965:head# 2024-04-29T14:26:38.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb939f4:::benchmark_data_smithi194_121512_object8556:head# 2024-04-29T14:26:38.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb955b3:::benchmark_data_smithi194_121512_object119917:head# 2024-04-29T14:26:38.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb969d3:::benchmark_data_smithi194_121512_object58818:head# 2024-04-29T14:26:38.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bb9f3b8:::benchmark_data_smithi194_121512_object28241:head# 2024-04-29T14:26:38.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bba6b39:::benchmark_data_smithi194_121512_object120695:head# 2024-04-29T14:26:38.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bba9f48:::benchmark_data_smithi194_121512_object132013:head# 2024-04-29T14:26:38.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbac0ea:::benchmark_data_smithi194_121512_object112228:head# 2024-04-29T14:26:38.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbb2a7d:::benchmark_data_smithi194_121512_object24694:head# 2024-04-29T14:26:38.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbb3985:::benchmark_data_smithi194_121512_object44011:head# 2024-04-29T14:26:38.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbb8214:::benchmark_data_smithi194_121512_object43406:head# 2024-04-29T14:26:38.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbbdc57:::benchmark_data_smithi194_121512_object2912:head# 2024-04-29T14:26:38.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbca0fe:::benchmark_data_smithi194_121512_object99572:head# 2024-04-29T14:26:38.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbccaa0:::benchmark_data_smithi194_121512_object57224:head# 2024-04-29T14:26:38.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbd1dc0:::benchmark_data_smithi194_121512_object65014:head# 2024-04-29T14:26:38.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbe6fc8:::benchmark_data_smithi194_121512_object92352:head# 2024-04-29T14:26:38.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbee757:::benchmark_data_smithi194_121512_object579:head# 2024-04-29T14:26:38.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbf8c1d:::benchmark_data_smithi194_121512_object100187:head# 2024-04-29T14:26:38.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbfaa84:::benchmark_data_smithi194_121512_object61361:head# 2024-04-29T14:26:38.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbfd590:::benchmark_data_smithi194_121512_object21750:head# 2024-04-29T14:26:38.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbfe0e9:::benchmark_data_smithi194_121512_object11350:head# 2024-04-29T14:26:38.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bbff631:::benchmark_data_smithi194_121512_object76351:head# 2024-04-29T14:26:38.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc04706:::benchmark_data_smithi194_121512_object128007:head# 2024-04-29T14:26:38.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc07efd:::benchmark_data_smithi194_121512_object109571:head# 2024-04-29T14:26:38.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc11917:::benchmark_data_smithi194_121512_object9197:head# 2024-04-29T14:26:38.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc12d5f:::benchmark_data_smithi194_121512_object114012:head# 2024-04-29T14:26:38.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc1ced1:::benchmark_data_smithi194_121512_object72787:head# 2024-04-29T14:26:38.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc29663:::benchmark_data_smithi194_121512_object46630:head# 2024-04-29T14:26:38.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc353f3:::benchmark_data_smithi194_121512_object63465:head# 2024-04-29T14:26:38.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc3bf99:::benchmark_data_smithi194_121512_object71465:head# 2024-04-29T14:26:38.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc4a7bd:::benchmark_data_smithi194_121512_object18179:head# 2024-04-29T14:26:38.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc5f816:::benchmark_data_smithi194_121512_object109785:head# 2024-04-29T14:26:38.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc61ce4:::benchmark_data_smithi194_121512_object16373:head# 2024-04-29T14:26:38.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc6b204:::benchmark_data_smithi194_121512_object98054:head# 2024-04-29T14:26:38.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc7302e:::benchmark_data_smithi194_121512_object110990:head# 2024-04-29T14:26:38.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc73421:::benchmark_data_smithi194_121512_object122576:head# 2024-04-29T14:26:38.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc7630b:::benchmark_data_smithi194_121512_object68547:head# 2024-04-29T14:26:38.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc7df47:::benchmark_data_smithi194_121512_object36524:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc86748:::benchmark_data_smithi194_121512_object91662:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc8d055:::benchmark_data_smithi194_121512_object41998:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bc91ea2:::benchmark_data_smithi194_121512_object45226:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bca957e:::benchmark_data_smithi194_121512_object55061:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcac416:::benchmark_data_smithi194_121512_object36826:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcb266e:::benchmark_data_smithi194_121512_object48208:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcba431:::benchmark_data_smithi194_121512_object50104:head# 2024-04-29T14:26:38.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcce5f0:::benchmark_data_smithi194_121512_object104333:head# 2024-04-29T14:26:38.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bccf039:::benchmark_data_smithi194_121512_object105818:head# 2024-04-29T14:26:38.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcd1c19:::benchmark_data_smithi194_121512_object6103:head# 2024-04-29T14:26:38.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcd46e4:::benchmark_data_smithi194_121512_object71113:head# 2024-04-29T14:26:38.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcdabcc:::benchmark_data_smithi194_121512_object29790:head# 2024-04-29T14:26:38.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcdb0e9:::benchmark_data_smithi194_121512_object28168:head# 2024-04-29T14:26:38.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bce819f:::benchmark_data_smithi194_121512_object91260:head# 2024-04-29T14:26:38.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bce9ef7:::benchmark_data_smithi194_121512_object16686:head# 2024-04-29T14:26:38.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcf4949:::benchmark_data_smithi194_121512_object67827:head# 2024-04-29T14:26:38.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcfc289:::benchmark_data_smithi194_121512_object63977:head# 2024-04-29T14:26:38.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bcfe6e1:::benchmark_data_smithi194_121512_object33960:head# 2024-04-29T14:26:38.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd04046:::benchmark_data_smithi194_121512_object120744:head# 2024-04-29T14:26:38.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd0e79e:::benchmark_data_smithi194_121512_object59783:head# 2024-04-29T14:26:38.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd2ca51:::benchmark_data_smithi194_121512_object91220:head# 2024-04-29T14:26:38.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd316cd:::benchmark_data_smithi194_121512_object126537:head# 2024-04-29T14:26:38.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd337f3:::benchmark_data_smithi194_121512_object93064:head# 2024-04-29T14:26:38.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd489d9:::benchmark_data_smithi194_121512_object123691:head# 2024-04-29T14:26:38.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd50426:::benchmark_data_smithi194_121512_object53902:head# 2024-04-29T14:26:38.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd551e6:::benchmark_data_smithi194_121512_object79094:head# 2024-04-29T14:26:38.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd622c2:::benchmark_data_smithi194_121512_object97601:head# 2024-04-29T14:26:38.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd652ff:::benchmark_data_smithi194_121512_object107796:head# 2024-04-29T14:26:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd77b36:::benchmark_data_smithi194_121512_object71305:head# 2024-04-29T14:26:38.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd7c5ea:::benchmark_data_smithi194_121512_object62671:head# 2024-04-29T14:26:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd83520:::benchmark_data_smithi194_121512_object68589:head# 2024-04-29T14:26:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd8bb70:::benchmark_data_smithi194_121512_object80152:head# 2024-04-29T14:26:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd90261:::benchmark_data_smithi194_121512_object48623:head# 2024-04-29T14:26:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd92eab:::benchmark_data_smithi194_121512_object13597:head# 2024-04-29T14:26:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd9857d:::benchmark_data_smithi194_121512_object125768:head# 2024-04-29T14:26:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bd9db11:::benchmark_data_smithi194_121512_object39977:head# 2024-04-29T14:26:38.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bda10de:::benchmark_data_smithi194_121512_object105306:head# 2024-04-29T14:26:38.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdaf2e0:::benchmark_data_smithi194_121512_object30672:head# 2024-04-29T14:26:38.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdc30d1:::benchmark_data_smithi194_121512_object95026:head# 2024-04-29T14:26:38.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdd78bb:::benchmark_data_smithi194_121512_object51974:head# 2024-04-29T14:26:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bddf3d7:::benchmark_data_smithi194_121512_object39939:head# 2024-04-29T14:26:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bde8c89:::benchmark_data_smithi194_121512_object24286:head# 2024-04-29T14:26:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdec469:::benchmark_data_smithi194_121512_object26737:head# 2024-04-29T14:26:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdf3f69:::benchmark_data_smithi194_121512_object95352:head# 2024-04-29T14:26:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdfa6ac:::benchmark_data_smithi194_121512_object92509:head# 2024-04-29T14:26:38.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdfac0d:::benchmark_data_smithi194_121512_object57494:head# 2024-04-29T14:26:38.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdfcd89:::benchmark_data_smithi194_121512_object25014:head# 2024-04-29T14:26:38.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bdfd17d:::benchmark_data_smithi194_121512_object126149:head# 2024-04-29T14:26:38.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be018a4:::benchmark_data_smithi194_121512_object75116:head# 2024-04-29T14:26:38.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be049ae:::benchmark_data_smithi194_121512_object4825:head# 2024-04-29T14:26:38.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be106e8:::benchmark_data_smithi194_121512_object44827:head# 2024-04-29T14:26:38.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be135a5:::benchmark_data_smithi194_121512_object3874:head# 2024-04-29T14:26:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be25b20:::benchmark_data_smithi194_121512_object105601:head# 2024-04-29T14:26:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be2ced5:::benchmark_data_smithi194_121512_object99180:head# 2024-04-29T14:26:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be2f535:::benchmark_data_smithi194_121512_object53988:head# 2024-04-29T14:26:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be360fa:::benchmark_data_smithi194_121512_object131018:head# 2024-04-29T14:26:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be39264:::benchmark_data_smithi194_121512_object74045:head# 2024-04-29T14:26:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be4636b:::benchmark_data_smithi194_121512_object50175:head# 2024-04-29T14:26:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be4d346:::benchmark_data_smithi194_121512_object105524:head# 2024-04-29T14:26:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be5483a:::benchmark_data_smithi194_121512_object60103:head# 2024-04-29T14:26:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be6820a:::benchmark_data_smithi194_121512_object126757:head# 2024-04-29T14:26:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be6d09e:::benchmark_data_smithi194_121512_object13286:head# 2024-04-29T14:26:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be6f58b:::benchmark_data_smithi194_121512_object31849:head# 2024-04-29T14:26:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be77ca2:::benchmark_data_smithi194_121512_object18578:head# 2024-04-29T14:26:38.637 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:38.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be7c9c0:::benchmark_data_smithi194_121512_object59360:head# 2024-04-29T14:26:38.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be7fb7d:::benchmark_data_smithi194_121512_object109740:head# 2024-04-29T14:26:38.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be8496d:::benchmark_data_smithi194_121512_object91455:head# 2024-04-29T14:26:38.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be899f6:::benchmark_data_smithi194_121512_object123935:head# 2024-04-29T14:26:38.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be8f032:::benchmark_data_smithi194_121512_object119877:head# 2024-04-29T14:26:38.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6be955d6:::benchmark_data_smithi194_121512_object11291:head# 2024-04-29T14:26:38.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6beb01f3:::benchmark_data_smithi194_121512_object32530:head# 2024-04-29T14:26:38.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6beb926c:::benchmark_data_smithi194_121512_object48225:head# 2024-04-29T14:26:38.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bebbfb0:::benchmark_data_smithi194_121512_object21457:head# 2024-04-29T14:26:38.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bec0113:::benchmark_data_smithi194_121512_object99251:head# 2024-04-29T14:26:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:38.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:38.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bec4e2c:::benchmark_data_smithi194_121512_object119661:head# 2024-04-29T14:26:38.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bec8201:::benchmark_data_smithi194_121512_object74954:head# 2024-04-29T14:26:38.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6becb24a:::benchmark_data_smithi194_121512_object81896:head# 2024-04-29T14:26:38.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bee953b:::benchmark_data_smithi194_121512_object107631:head# 2024-04-29T14:26:38.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6befd8b8:::benchmark_data_smithi194_121512_object51288:head# 2024-04-29T14:26:38.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6befdbff:::benchmark_data_smithi194_121512_object75910:head# 2024-04-29T14:26:38.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf0ee83:::benchmark_data_smithi194_121512_object47832:head# 2024-04-29T14:26:38.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf30fc3:::benchmark_data_smithi194_121512_object30792:head# 2024-04-29T14:26:38.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf328c9:::benchmark_data_smithi194_121512_object114215:head# 2024-04-29T14:26:38.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf4697c:::benchmark_data_smithi194_121512_object85177:head# 2024-04-29T14:26:38.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf5e5d0:::benchmark_data_smithi194_121512_object61952:head# 2024-04-29T14:26:38.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf61da9:::benchmark_data_smithi194_121512_object13131:head# 2024-04-29T14:26:38.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf67977:::benchmark_data_smithi194_121512_object95753:head# 2024-04-29T14:26:38.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf6b97b:::benchmark_data_smithi194_121512_object67601:head# 2024-04-29T14:26:38.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf73b8d:::benchmark_data_smithi194_121512_object37900:head# 2024-04-29T14:26:38.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf78847:::benchmark_data_smithi194_121512_object100473:head# 2024-04-29T14:26:38.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf8275b:::benchmark_data_smithi194_121512_object2175:head# 2024-04-29T14:26:38.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf83ba3:::benchmark_data_smithi194_121512_object69519:head# 2024-04-29T14:26:38.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf86b59:::benchmark_data_smithi194_121512_object102284:head# 2024-04-29T14:26:38.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bf958c9:::benchmark_data_smithi194_121512_object90927:head# 2024-04-29T14:26:38.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfa3359:::benchmark_data_smithi194_121512_object36291:head# 2024-04-29T14:26:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfa3d14:::benchmark_data_smithi194_121512_object82522:head# 2024-04-29T14:26:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfa4801:::benchmark_data_smithi194_121512_object76460:head# 2024-04-29T14:26:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfaca35:::benchmark_data_smithi194_121512_object95688:head# 2024-04-29T14:26:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfae586:::benchmark_data_smithi194_121512_object56545:head# 2024-04-29T14:26:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfb58bf:::benchmark_data_smithi194_121512_object63136:head# 2024-04-29T14:26:38.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfb6e6a:::benchmark_data_smithi194_121512_object73466:head# 2024-04-29T14:26:38.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfba4e2:::benchmark_data_smithi194_121512_object27871:head# 2024-04-29T14:26:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfc11b9:::benchmark_data_smithi194_121512_object89705:head# 2024-04-29T14:26:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfc2e5e:::benchmark_data_smithi194_121512_object21889:head# 2024-04-29T14:26:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfd49d2:::benchmark_data_smithi194_121512_object12827:head# 2024-04-29T14:26:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfe1c91:::benchmark_data_smithi194_121512_object57605:head# 2024-04-29T14:26:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfe28e4:::benchmark_data_smithi194_121512_object32629:head# 2024-04-29T14:26:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfe3da2:::benchmark_data_smithi194_121512_object45140:head# 2024-04-29T14:26:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfe517e:::benchmark_data_smithi194_121512_object30676:head# 2024-04-29T14:26:38.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bfeabff:::benchmark_data_smithi194_121512_object74711:head# 2024-04-29T14:26:38.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bffc0b1:::benchmark_data_smithi194_121512_object39518:head# 2024-04-29T14:26:38.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bffc969:::benchmark_data_smithi194_121512_object97812:head# 2024-04-29T14:26:38.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6bffc9ca:::benchmark_data_smithi194_121512_object51773:head# 2024-04-29T14:26:38.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0010ca:::benchmark_data_smithi194_121512_object21648:head# 2024-04-29T14:26:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c014f8a:::benchmark_data_smithi194_121512_object34358:head# 2024-04-29T14:26:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c018b04:::benchmark_data_smithi194_121512_object24268:head# 2024-04-29T14:26:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c02d68e:::benchmark_data_smithi194_121512_object96071:head# 2024-04-29T14:26:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c032af6:::benchmark_data_smithi194_121512_object30323:head# 2024-04-29T14:26:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c034264:::benchmark_data_smithi194_121512_object644:head# 2024-04-29T14:26:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c03475d:::benchmark_data_smithi194_121512_object78410:head# 2024-04-29T14:26:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c037462:::benchmark_data_smithi194_121512_object19867:head# 2024-04-29T14:26:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c044a60:::benchmark_data_smithi194_121512_object87198:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c04d198:::benchmark_data_smithi194_121512_object34232:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c051845:::benchmark_data_smithi194_121512_object96087:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c055056:::benchmark_data_smithi194_121512_object76715:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c05e93a:::benchmark_data_smithi194_121512_object86456:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c068553:::benchmark_data_smithi194_121512_object8193:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c068b5f:::benchmark_data_smithi194_121512_object83068:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c069d57:::benchmark_data_smithi194_121512_object13440:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c06b8a8:::benchmark_data_smithi194_121512_object35410:head# 2024-04-29T14:26:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c06c6e5:::benchmark_data_smithi194_121512_object21215:head# 2024-04-29T14:26:38.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c06e76c:::benchmark_data_smithi194_121512_object74436:head# 2024-04-29T14:26:38.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c07cca1:::benchmark_data_smithi194_121512_object102322:head# 2024-04-29T14:26:38.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c07e400:::benchmark_data_smithi194_121512_object119921:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c07f7d2:::benchmark_data_smithi194_121512_object78430:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c082369:::benchmark_data_smithi194_121512_object112844:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0860f6:::benchmark_data_smithi194_121512_object105751:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c090358:::benchmark_data_smithi194_121512_object43327:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c09184a:::benchmark_data_smithi194_121512_object115388:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c092619:::benchmark_data_smithi194_121512_object26149:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0960cb:::benchmark_data_smithi194_121512_object30569:head# 2024-04-29T14:26:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c09b3f2:::benchmark_data_smithi194_121512_object9723:head# 2024-04-29T14:26:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0b1c35:::benchmark_data_smithi194_121512_object3250:head# 2024-04-29T14:26:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0b6eb8:::benchmark_data_smithi194_121512_object72959:head# 2024-04-29T14:26:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0bb5b6:::benchmark_data_smithi194_121512_object115221:head# 2024-04-29T14:26:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0bba60:::benchmark_data_smithi194_121512_object30932:head# 2024-04-29T14:26:38.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0bcb34:::benchmark_data_smithi194_121512_object63839:head# 2024-04-29T14:26:38.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0c3176:::benchmark_data_smithi194_121512_object21177:head# 2024-04-29T14:26:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0cf44a:::benchmark_data_smithi194_121512_object126994:head# 2024-04-29T14:26:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0d56ac:::benchmark_data_smithi194_121512_object49997:head# 2024-04-29T14:26:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0dc4cb:::benchmark_data_smithi194_121512_object10757:head# 2024-04-29T14:26:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0df3ce:::benchmark_data_smithi194_121512_object70816:head# 2024-04-29T14:26:38.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0e1d3a:::benchmark_data_smithi194_121512_object96226:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0f03d3:::benchmark_data_smithi194_121512_object61122:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0f05e5:::benchmark_data_smithi194_121512_object126224:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0f4f5c:::benchmark_data_smithi194_121512_object33731:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0f5611:::benchmark_data_smithi194_121512_object100000:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0fa1e7:::benchmark_data_smithi194_121512_object66823:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c0fcfc2:::benchmark_data_smithi194_121512_object12639:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c104af8:::benchmark_data_smithi194_121512_object116692:head# 2024-04-29T14:26:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c106437:::benchmark_data_smithi194_121512_object71953:head# 2024-04-29T14:26:38.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c111791:::benchmark_data_smithi194_121512_object103557:head# 2024-04-29T14:26:38.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c113dbf:::benchmark_data_smithi194_121512_object78597:head# 2024-04-29T14:26:38.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1180a8:::benchmark_data_smithi194_121512_object16510:head# 2024-04-29T14:26:38.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c119388:::benchmark_data_smithi194_121512_object54148:head# 2024-04-29T14:26:38.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c11cd96:::benchmark_data_smithi194_121512_object55395:head# 2024-04-29T14:26:38.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c11d970:::benchmark_data_smithi194_121512_object48584:head# 2024-04-29T14:26:38.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c11d9e3:::benchmark_data_smithi194_121512_object114014:head# 2024-04-29T14:26:38.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c11dd41:::benchmark_data_smithi194_121512_object93561:head# 2024-04-29T14:26:38.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c121276:::benchmark_data_smithi194_121512_object103142:head# 2024-04-29T14:26:38.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c122169:::benchmark_data_smithi194_121512_object80510:head# 2024-04-29T14:26:38.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1221a6:::benchmark_data_smithi194_121512_object34190:head# 2024-04-29T14:26:38.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c122835:::benchmark_data_smithi194_121512_object133461:head# 2024-04-29T14:26:38.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c12300e:::benchmark_data_smithi194_121512_object68405:head# 2024-04-29T14:26:38.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1261c2:::benchmark_data_smithi194_121512_object67883:head# 2024-04-29T14:26:38.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c12812a:::benchmark_data_smithi194_121512_object77907:head# 2024-04-29T14:26:38.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1334a9:::benchmark_data_smithi194_121512_object62059:head# 2024-04-29T14:26:38.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c13f5c9:::benchmark_data_smithi194_121512_object58935:head# 2024-04-29T14:26:38.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c14b5ba:::benchmark_data_smithi194_121512_object86191:head# 2024-04-29T14:26:38.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c14b98a:::benchmark_data_smithi194_121512_object11460:head# 2024-04-29T14:26:38.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c15d68a:::benchmark_data_smithi194_121512_object61475:head# 2024-04-29T14:26:38.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1649da:::benchmark_data_smithi194_121512_object93647:head# 2024-04-29T14:26:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1651ef:::benchmark_data_smithi194_121512_object116681:head# 2024-04-29T14:26:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c16b772:::benchmark_data_smithi194_121512_object61613:head# 2024-04-29T14:26:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c17b9a0:::benchmark_data_smithi194_121512_object128037:head# 2024-04-29T14:26:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c18eaca:::benchmark_data_smithi194_121512_object106592:head# 2024-04-29T14:26:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1958ed:::benchmark_data_smithi194_121512_object15015:head# 2024-04-29T14:26:38.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1a1f63:::benchmark_data_smithi194_121512_object118669:head# 2024-04-29T14:26:38.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1b7efa:::benchmark_data_smithi194_121512_object125329:head# 2024-04-29T14:26:38.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1c0914:::benchmark_data_smithi194_121512_object114553:head# 2024-04-29T14:26:38.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1cc97e:::benchmark_data_smithi194_121512_object12538:head# 2024-04-29T14:26:38.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1cf0cd:::benchmark_data_smithi194_121512_object102526:head# 2024-04-29T14:26:38.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1de35c:::benchmark_data_smithi194_121512_object20628:head# 2024-04-29T14:26:38.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1def8b:::benchmark_data_smithi194_121512_object3561:head# 2024-04-29T14:26:38.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1f07f4:::benchmark_data_smithi194_121512_object45124:head# 2024-04-29T14:26:38.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1fa7e2:::benchmark_data_smithi194_121512_object16812:head# 2024-04-29T14:26:38.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c1fdbaa:::benchmark_data_smithi194_121512_object75161:head# 2024-04-29T14:26:38.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c208c31:::benchmark_data_smithi194_121512_object2988:head# 2024-04-29T14:26:38.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c20ad84:::benchmark_data_smithi194_121512_object4538:head# 2024-04-29T14:26:38.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c20c223:::benchmark_data_smithi194_121512_object126855:head# 2024-04-29T14:26:38.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c20f9c5:::benchmark_data_smithi194_121512_object126930:head# 2024-04-29T14:26:38.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2266fa:::benchmark_data_smithi194_121512_object78153:head# 2024-04-29T14:26:38.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c226858:::benchmark_data_smithi194_121512_object113656:head# 2024-04-29T14:26:38.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c229354:::benchmark_data_smithi194_121512_object112360:head# 2024-04-29T14:26:38.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c22c2c6:::benchmark_data_smithi194_121512_object16786:head# 2024-04-29T14:26:38.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c24089c:::benchmark_data_smithi194_121512_object79225:head# 2024-04-29T14:26:38.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c241bfe:::benchmark_data_smithi194_121512_object113618:head# 2024-04-29T14:26:38.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c242d60:::benchmark_data_smithi194_121512_object24748:head# 2024-04-29T14:26:38.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c24a4fa:::benchmark_data_smithi194_121512_object15368:head# 2024-04-29T14:26:38.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c24ec1f:::benchmark_data_smithi194_121512_object79113:head# 2024-04-29T14:26:38.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c251a88:::benchmark_data_smithi194_121512_object125816:head# 2024-04-29T14:26:38.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2577c0:::benchmark_data_smithi194_121512_object103200:head# 2024-04-29T14:26:38.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c25be9f:::benchmark_data_smithi194_121512_object83014:head# 2024-04-29T14:26:38.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c25df7e:::benchmark_data_smithi194_121512_object83887:head# 2024-04-29T14:26:38.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c262782:::benchmark_data_smithi194_121512_object11547:head# 2024-04-29T14:26:38.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c269c72:::benchmark_data_smithi194_121512_object86753:head# 2024-04-29T14:26:38.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c271e72:::benchmark_data_smithi194_121512_object58045:head# 2024-04-29T14:26:38.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c27cc5b:::benchmark_data_smithi194_121512_object100054:head# 2024-04-29T14:26:38.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2963cb:::benchmark_data_smithi194_121512_object24261:head# 2024-04-29T14:26:38.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c29e5fc:::benchmark_data_smithi194_121512_object131188:head# 2024-04-29T14:26:38.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2b168e:::benchmark_data_smithi194_121512_object130988:head# 2024-04-29T14:26:38.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2b58d6:::benchmark_data_smithi194_121512_object68177:head# 2024-04-29T14:26:38.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2dfaf8:::benchmark_data_smithi194_121512_object109807:head# 2024-04-29T14:26:38.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2e0060:::benchmark_data_smithi194_121512_object103785:head# 2024-04-29T14:26:38.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2e0722:::benchmark_data_smithi194_121512_object100306:head# 2024-04-29T14:26:38.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c2fbab3:::benchmark_data_smithi194_121512_object48060:head# 2024-04-29T14:26:38.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3004d5:::benchmark_data_smithi194_121512_object23675:head# 2024-04-29T14:26:38.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c30388d:::benchmark_data_smithi194_121512_object71874:head# 2024-04-29T14:26:38.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c306734:::benchmark_data_smithi194_121512_object17238:head# 2024-04-29T14:26:38.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c30f918:::benchmark_data_smithi194_121512_object36669:head# 2024-04-29T14:26:38.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c31d566:::benchmark_data_smithi194_121512_object121828:head# 2024-04-29T14:26:38.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c31dad3:::benchmark_data_smithi194_121512_object45562:head# 2024-04-29T14:26:38.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c31dbfd:::benchmark_data_smithi194_121512_object91881:head# 2024-04-29T14:26:38.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c31e9bd:::benchmark_data_smithi194_121512_object89246:head# 2024-04-29T14:26:38.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c33d98d:::benchmark_data_smithi194_121512_object119805:head# 2024-04-29T14:26:38.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c347901:::benchmark_data_smithi194_121512_object99457:head# 2024-04-29T14:26:38.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c357550:::benchmark_data_smithi194_121512_object63035:head# 2024-04-29T14:26:38.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c363b0d:::benchmark_data_smithi194_121512_object127308:head# 2024-04-29T14:26:38.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c36b5f8:::benchmark_data_smithi194_121512_object31402:head# 2024-04-29T14:26:38.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c36b8bd:::benchmark_data_smithi194_121512_object65342:head# 2024-04-29T14:26:38.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c36f8ed:::benchmark_data_smithi194_121512_object22231:head# 2024-04-29T14:26:38.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c387f7d:::benchmark_data_smithi194_121512_object122913:head# 2024-04-29T14:26:38.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c39809a:::benchmark_data_smithi194_121512_object88623:head# 2024-04-29T14:26:38.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c39a7fd:::benchmark_data_smithi194_121512_object102539:head# 2024-04-29T14:26:38.742 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:26:38.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c39d49c:::benchmark_data_smithi194_121512_object125093:head# 2024-04-29T14:26:38.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3a152f:::benchmark_data_smithi194_121512_object4280:head# 2024-04-29T14:26:38.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3a5fa6:::benchmark_data_smithi194_121512_object52204:head# 2024-04-29T14:26:38.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3afb41:::benchmark_data_smithi194_121512_object2164:head# 2024-04-29T14:26:38.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3b5f32:::benchmark_data_smithi194_121512_object54425:head# 2024-04-29T14:26:38.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3b685e:::benchmark_data_smithi194_121512_object93805:head# 2024-04-29T14:26:38.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3b76f2:::benchmark_data_smithi194_121512_object81933:head# 2024-04-29T14:26:38.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3b7b05:::benchmark_data_smithi194_121512_object20592:head# 2024-04-29T14:26:38.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3bcce2:::benchmark_data_smithi194_121512_object101348:head# 2024-04-29T14:26:38.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3c0df7:::benchmark_data_smithi194_121512_object100708:head# 2024-04-29T14:26:38.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3cbaaf:::benchmark_data_smithi194_121512_object81010:head# 2024-04-29T14:26:38.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3cf2c9:::benchmark_data_smithi194_121512_object15561:head# 2024-04-29T14:26:38.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3d0d7d:::benchmark_data_smithi194_121512_object30643:head# 2024-04-29T14:26:38.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3d4a76:::benchmark_data_smithi194_121512_object52379:head# 2024-04-29T14:26:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3e5763:::benchmark_data_smithi194_121512_object27667:head# 2024-04-29T14:26:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3e98b8:::benchmark_data_smithi194_121512_object52168:head# 2024-04-29T14:26:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3ef4ec:::benchmark_data_smithi194_121512_object55181:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c3f6008:::benchmark_data_smithi194_121512_object91140:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4009f3:::benchmark_data_smithi194_121512_object118204:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c40224a:::benchmark_data_smithi194_121512_object127807:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c410003:::benchmark_data_smithi194_121512_object117109:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c41fe5e:::benchmark_data_smithi194_121512_object75613:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c426f4a:::benchmark_data_smithi194_121512_object7544:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c434998:::benchmark_data_smithi194_121512_object48684:head# 2024-04-29T14:26:38.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c43e458:::benchmark_data_smithi194_121512_object84782:head# 2024-04-29T14:26:38.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c441373:::benchmark_data_smithi194_121512_object60034:head# 2024-04-29T14:26:38.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c444c45:::benchmark_data_smithi194_121512_object101167:head# 2024-04-29T14:26:38.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c44828e:::benchmark_data_smithi194_121512_object71454:head# 2024-04-29T14:26:38.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4610e7:::benchmark_data_smithi194_121512_object46277:head# 2024-04-29T14:26:38.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c48647b:::benchmark_data_smithi194_121512_object1275:head# 2024-04-29T14:26:38.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4872d8:::benchmark_data_smithi194_121512_object95185:head# 2024-04-29T14:26:38.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4893a0:::benchmark_data_smithi194_121512_object46517:head# 2024-04-29T14:26:38.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c48f99f:::benchmark_data_smithi194_121512_object18001:head# 2024-04-29T14:26:38.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c490a73:::benchmark_data_smithi194_121512_object35587:head# 2024-04-29T14:26:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c493406:::benchmark_data_smithi194_121512_object46784:head# 2024-04-29T14:26:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c497330:::benchmark_data_smithi194_121512_object127490:head# 2024-04-29T14:26:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c49a846:::benchmark_data_smithi194_121512_object31022:head# 2024-04-29T14:26:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c49acb8:::benchmark_data_smithi194_121512_object91026:head# 2024-04-29T14:26:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4a1e83:::benchmark_data_smithi194_121512_object13681:head# 2024-04-29T14:26:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4a507d:::benchmark_data_smithi194_121512_object114591:head# 2024-04-29T14:26:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4b2c6e:::benchmark_data_smithi194_121512_object19701:head# 2024-04-29T14:26:38.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4b927b:::benchmark_data_smithi194_121512_object22724:head# 2024-04-29T14:26:38.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4bc3d5:::benchmark_data_smithi194_121512_object92472:head# 2024-04-29T14:26:38.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4c2581:::benchmark_data_smithi194_121512_object51635:head# 2024-04-29T14:26:38.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4cc48d:::benchmark_data_smithi194_121512_object86985:head# 2024-04-29T14:26:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4d0360:::benchmark_data_smithi194_121512_object120332:head# 2024-04-29T14:26:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4d4fcc:::benchmark_data_smithi194_121512_object46337:head# 2024-04-29T14:26:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4e9d4d:::benchmark_data_smithi194_121512_object108032:head# 2024-04-29T14:26:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4f8b99:::benchmark_data_smithi194_121512_object48605:head# 2024-04-29T14:26:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4fa793:::benchmark_data_smithi194_121512_object103319:head# 2024-04-29T14:26:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c4fec62:::benchmark_data_smithi194_121512_object42599:head# 2024-04-29T14:26:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c51637c:::benchmark_data_smithi194_121512_object116999:head# 2024-04-29T14:26:38.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c518e6f:::benchmark_data_smithi194_121512_object107374:head# 2024-04-29T14:26:38.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c51af4a:::benchmark_data_smithi194_121512_object17324:head# 2024-04-29T14:26:38.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c542103:::benchmark_data_smithi194_121512_object101390:head# 2024-04-29T14:26:38.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c54bc49:::benchmark_data_smithi194_121512_object81694:head# 2024-04-29T14:26:38.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c551d3f:::benchmark_data_smithi194_121512_object10041:head# 2024-04-29T14:26:38.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5588e6:::benchmark_data_smithi194_121512_object86782:head# 2024-04-29T14:26:38.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5592b9:::benchmark_data_smithi194_121512_object53908:head# 2024-04-29T14:26:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c55c0bb:::benchmark_data_smithi194_121512_object52454:head# 2024-04-29T14:26:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c55c4ec:::benchmark_data_smithi194_121512_object11153:head# 2024-04-29T14:26:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5645e7:::benchmark_data_smithi194_121512_object35556:head# 2024-04-29T14:26:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c566272:::benchmark_data_smithi194_121512_object124774:head# 2024-04-29T14:26:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c56c131:::benchmark_data_smithi194_121512_object12545:head# 2024-04-29T14:26:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c56ee64:::benchmark_data_smithi194_121512_object37241:head# 2024-04-29T14:26:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c56f95e:::benchmark_data_smithi194_121512_object88052:head# 2024-04-29T14:26:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c570327:::benchmark_data_smithi194_121512_object59981:head# 2024-04-29T14:26:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c57a729:::benchmark_data_smithi194_121512_object34086:head# 2024-04-29T14:26:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c583309:::benchmark_data_smithi194_121512_object98727:head# 2024-04-29T14:26:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c591705:::benchmark_data_smithi194_121512_object26971:head# 2024-04-29T14:26:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c595cb7:::benchmark_data_smithi194_121512_object100232:head# 2024-04-29T14:26:38.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c597153:::benchmark_data_smithi194_121512_object123975:head# 2024-04-29T14:26:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c59a8b3:::benchmark_data_smithi194_121512_object56265:head# 2024-04-29T14:26:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c59affd:::benchmark_data_smithi194_121512_object39289:head# 2024-04-29T14:26:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5a1079:::benchmark_data_smithi194_121512_object86757:head# 2024-04-29T14:26:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5acd35:::benchmark_data_smithi194_121512_object51980:head# 2024-04-29T14:26:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5c36b6:::benchmark_data_smithi194_121512_object115175:head# 2024-04-29T14:26:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5ce473:::benchmark_data_smithi194_121512_object18693:head# 2024-04-29T14:26:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c5ed43b:::benchmark_data_smithi194_121512_object90224:head# 2024-04-29T14:26:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6091a1:::benchmark_data_smithi194_121512_object128369:head# 2024-04-29T14:26:38.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6094f9:::benchmark_data_smithi194_121512_object118276:head# 2024-04-29T14:26:38.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c60c732:::benchmark_data_smithi194_121512_object27009:head# 2024-04-29T14:26:38.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c60cb4e:::benchmark_data_smithi194_121512_object45652:head# 2024-04-29T14:26:38.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c60ec77:::benchmark_data_smithi194_121512_object66003:head# 2024-04-29T14:26:38.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c61628e:::benchmark_data_smithi194_121512_object83849:head# 2024-04-29T14:26:38.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c617486:::benchmark_data_smithi194_121512_object5213:head# 2024-04-29T14:26:38.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c620f2d:::benchmark_data_smithi194_121512_object2310:head# 2024-04-29T14:26:38.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c625cf5:::benchmark_data_smithi194_121512_object59740:head# 2024-04-29T14:26:38.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c63272f:::benchmark_data_smithi194_121512_object116092:head# 2024-04-29T14:26:38.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6372e5:::benchmark_data_smithi194_121512_object129646:head# 2024-04-29T14:26:38.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6373a5:::benchmark_data_smithi194_121512_object76558:head# 2024-04-29T14:26:38.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c63ee3c:::benchmark_data_smithi194_121512_object121531:head# 2024-04-29T14:26:38.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c64e51f:::benchmark_data_smithi194_121512_object103113:head# 2024-04-29T14:26:38.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c652d44:::benchmark_data_smithi194_121512_object90250:head# 2024-04-29T14:26:38.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c66462d:::benchmark_data_smithi194_121512_object6798:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c669305:::benchmark_data_smithi194_121512_object122017:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c673f14:::benchmark_data_smithi194_121512_object13404:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c693bdc:::benchmark_data_smithi194_121512_object10430:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6952ac:::benchmark_data_smithi194_121512_object72775:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c69a6e8:::benchmark_data_smithi194_121512_object2077:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c69cd51:::benchmark_data_smithi194_121512_object64071:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6a1ca1:::benchmark_data_smithi194_121512_object4728:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6acbef:::benchmark_data_smithi194_121512_object84650:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6ba9b5:::benchmark_data_smithi194_121512_object112776:head# 2024-04-29T14:26:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6bac3f:::benchmark_data_smithi194_121512_object7686:head# 2024-04-29T14:26:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6c44d0:::benchmark_data_smithi194_121512_object11401:head# 2024-04-29T14:26:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6cf642:::benchmark_data_smithi194_121512_object7273:head# 2024-04-29T14:26:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6d981c:::benchmark_data_smithi194_121512_object118073:head# 2024-04-29T14:26:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6e00fe:::benchmark_data_smithi194_121512_object26460:head# 2024-04-29T14:26:38.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6f5e9c:::benchmark_data_smithi194_121512_object44809:head# 2024-04-29T14:26:38.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c6fb8b7:::benchmark_data_smithi194_121512_object130671:head# 2024-04-29T14:26:38.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c70b5d3:::benchmark_data_smithi194_121512_object3614:head# 2024-04-29T14:26:38.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c712696:::benchmark_data_smithi194_121512_object108246:head# 2024-04-29T14:26:38.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c714fd0:::benchmark_data_smithi194_121512_object57765:head# 2024-04-29T14:26:38.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c72e482:::benchmark_data_smithi194_121512_object77602:head# 2024-04-29T14:26:38.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7353e8:::benchmark_data_smithi194_121512_object119121:head# 2024-04-29T14:26:38.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c742708:::benchmark_data_smithi194_121512_object35461:head# 2024-04-29T14:26:38.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7555c7:::benchmark_data_smithi194_121512_object69707:head# 2024-04-29T14:26:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c765c0a:::benchmark_data_smithi194_121512_object108334:head# 2024-04-29T14:26:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c76d2d7:::benchmark_data_smithi194_121512_object9881:head# 2024-04-29T14:26:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c771d0d:::benchmark_data_smithi194_121512_object38037:head# 2024-04-29T14:26:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7753c8:::benchmark_data_smithi194_121512_object68643:head# 2024-04-29T14:26:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c77d1b6:::benchmark_data_smithi194_121512_object71167:head# 2024-04-29T14:26:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c780c57:::benchmark_data_smithi194_121512_object109241:head# 2024-04-29T14:26:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7818e8:::benchmark_data_smithi194_121512_object96262:head# 2024-04-29T14:26:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7840b5:::benchmark_data_smithi194_121512_object131369:head# 2024-04-29T14:26:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c79a102:::benchmark_data_smithi194_121512_object81504:head# 2024-04-29T14:26:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c79d12b:::benchmark_data_smithi194_121512_object20373:head# 2024-04-29T14:26:38.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7a1314:::benchmark_data_smithi194_121512_object25490:head# 2024-04-29T14:26:38.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7a589d:::benchmark_data_smithi194_121512_object74402:head# 2024-04-29T14:26:38.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7aa7ec:::benchmark_data_smithi194_121512_object105447:head# 2024-04-29T14:26:38.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7aff5e:::benchmark_data_smithi194_121512_object5546:head# 2024-04-29T14:26:38.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7bd130:::benchmark_data_smithi194_121512_object131803:head# 2024-04-29T14:26:38.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7c1144:::benchmark_data_smithi194_121512_object29372:head# 2024-04-29T14:26:38.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7c548b:::benchmark_data_smithi194_121512_object57320:head# 2024-04-29T14:26:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7c7703:::benchmark_data_smithi194_121512_object63098:head# 2024-04-29T14:26:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7d444d:::benchmark_data_smithi194_121512_object22112:head# 2024-04-29T14:26:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7e1ccc:::benchmark_data_smithi194_121512_object3599:head# 2024-04-29T14:26:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7ea0f9:::benchmark_data_smithi194_121512_object86167:head# 2024-04-29T14:26:38.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7ed76e:::benchmark_data_smithi194_121512_object9803:head# 2024-04-29T14:26:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7f9497:::benchmark_data_smithi194_121512_object97909:head# 2024-04-29T14:26:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c7fd6a8:::benchmark_data_smithi194_121512_object84612:head# 2024-04-29T14:26:38.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c803be2:::benchmark_data_smithi194_121512_object84897:head# 2024-04-29T14:26:38.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8055b6:::benchmark_data_smithi194_121512_object77068:head# 2024-04-29T14:26:38.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c805695:::benchmark_data_smithi194_121512_object16203:head# 2024-04-29T14:26:38.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c808ca7:::benchmark_data_smithi194_121512_object8880:head# 2024-04-29T14:26:38.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c82c9c4:::benchmark_data_smithi194_121512_object19054:head# 2024-04-29T14:26:38.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c844cc4:::benchmark_data_smithi194_121512_object17963:head# 2024-04-29T14:26:38.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c84df77:::benchmark_data_smithi194_121512_object133431:head# 2024-04-29T14:26:38.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8679c1:::benchmark_data_smithi194_121512_object69089:head# 2024-04-29T14:26:38.820 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:38.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c874cb7:::benchmark_data_smithi194_121512_object25972:head# 2024-04-29T14:26:38.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c87f8dc:::benchmark_data_smithi194_121512_object28327:head# 2024-04-29T14:26:38.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c88c5f7:::benchmark_data_smithi194_121512_object58533:head# 2024-04-29T14:26:38.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c88fd89:::benchmark_data_smithi194_121512_object18391:head# 2024-04-29T14:26:38.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c890804:::benchmark_data_smithi194_121512_object10313:head# 2024-04-29T14:26:38.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c893a1b:::benchmark_data_smithi194_121512_object27079:head# 2024-04-29T14:26:38.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c899d05:::benchmark_data_smithi194_121512_object38389:head# 2024-04-29T14:26:38.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c89bfc6:::benchmark_data_smithi194_121512_object120584:head# 2024-04-29T14:26:38.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c89f790:::benchmark_data_smithi194_121512_object28451:head# 2024-04-29T14:26:38.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8a0c66:::benchmark_data_smithi194_121512_object77393:head# 2024-04-29T14:26:38.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8b04f6:::benchmark_data_smithi194_121512_object45577:head# 2024-04-29T14:26:38.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8b0728:::benchmark_data_smithi194_121512_object78227:head# 2024-04-29T14:26:38.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8b1e38:::benchmark_data_smithi194_121512_object13605:head# 2024-04-29T14:26:38.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8b3889:::benchmark_data_smithi194_121512_object87118:head# 2024-04-29T14:26:38.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8b6790:::benchmark_data_smithi194_121512_object100581:head# 2024-04-29T14:26:38.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8c9400:::benchmark_data_smithi194_121512_object96492:head# 2024-04-29T14:26:38.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8ca54d:::benchmark_data_smithi194_121512_object101036:head# 2024-04-29T14:26:38.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8cef0c:::benchmark_data_smithi194_121512_object68795:head# 2024-04-29T14:26:38.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8d81ff:::benchmark_data_smithi194_121512_object70678:head# 2024-04-29T14:26:38.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8eb663:::benchmark_data_smithi194_121512_object6675:head# 2024-04-29T14:26:38.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8f5840:::benchmark_data_smithi194_121512_object106611:head# 2024-04-29T14:26:38.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c8f8be9:::benchmark_data_smithi194_121512_object79040:head# 2024-04-29T14:26:38.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c909299:::benchmark_data_smithi194_121512_object23417:head# 2024-04-29T14:26:38.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c90d149:::benchmark_data_smithi194_121512_object40817:head# 2024-04-29T14:26:38.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c916a70:::benchmark_data_smithi194_121512_object63086:head# 2024-04-29T14:26:38.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9215dc:::benchmark_data_smithi194_121512_object93622:head# 2024-04-29T14:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c923ab9:::benchmark_data_smithi194_121512_object39091:head# 2024-04-29T14:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c938c32:::benchmark_data_smithi194_121512_object77364:head# 2024-04-29T14:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c939db6:::benchmark_data_smithi194_121512_object103065:head# 2024-04-29T14:26:38.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9489db:::benchmark_data_smithi194_121512_object72125:head# 2024-04-29T14:26:38.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9499ec:::benchmark_data_smithi194_121512_object6231:head# 2024-04-29T14:26:38.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c950653:::benchmark_data_smithi194_121512_object49426:head# 2024-04-29T14:26:38.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9532a3:::benchmark_data_smithi194_121512_object118678:head# 2024-04-29T14:26:38.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c95a1bd:::benchmark_data_smithi194_121512_object100740:head# 2024-04-29T14:26:38.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c95eb06:::benchmark_data_smithi194_121512_object38531:head# 2024-04-29T14:26:38.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c95ee25:::benchmark_data_smithi194_121512_object90184:head# 2024-04-29T14:26:38.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c96203f:::benchmark_data_smithi194_121512_object108511:head# 2024-04-29T14:26:38.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c964848:::benchmark_data_smithi194_121512_object83054:head# 2024-04-29T14:26:38.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c985cd4:::benchmark_data_smithi194_121512_object56952:head# 2024-04-29T14:26:38.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c98f21c:::benchmark_data_smithi194_121512_object102305:head# 2024-04-29T14:26:38.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c996c5f:::benchmark_data_smithi194_121512_object103724:head# 2024-04-29T14:26:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9a3bf3:::benchmark_data_smithi194_121512_object125973:head# 2024-04-29T14:26:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9a4eaa:::benchmark_data_smithi194_121512_object124488:head# 2024-04-29T14:26:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9ae347:::benchmark_data_smithi194_121512_object51702:head# 2024-04-29T14:26:38.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9afc5f:::benchmark_data_smithi194_121512_object47764:head# 2024-04-29T14:26:38.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9b100e:::benchmark_data_smithi194_121512_object26915:head# 2024-04-29T14:26:38.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9b7992:::benchmark_data_smithi194_121512_object11996:head# 2024-04-29T14:26:38.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9b9600:::benchmark_data_smithi194_121512_object65922:head# 2024-04-29T14:26:38.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9bb982:::benchmark_data_smithi194_121512_object113263:head# 2024-04-29T14:26:38.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9bcdc6:::benchmark_data_smithi194_121512_object17940:head# 2024-04-29T14:26:38.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9c7b58:::benchmark_data_smithi194_121512_object120656:head# 2024-04-29T14:26:38.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9d1821:::benchmark_data_smithi194_121512_object97742:head# 2024-04-29T14:26:38.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9dcad6:::benchmark_data_smithi194_121512_object96156:head# 2024-04-29T14:26:38.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9df603:::benchmark_data_smithi194_121512_object69652:head# 2024-04-29T14:26:38.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9e4495:::benchmark_data_smithi194_121512_object124928:head# 2024-04-29T14:26:38.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9ea01e:::benchmark_data_smithi194_121512_object109554:head# 2024-04-29T14:26:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9eb5d5:::benchmark_data_smithi194_121512_object101271:head# 2024-04-29T14:26:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9f5785:::benchmark_data_smithi194_121512_object25713:head# 2024-04-29T14:26:38.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9f8e9d:::benchmark_data_smithi194_121512_object31907:head# 2024-04-29T14:26:38.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9fdae7:::benchmark_data_smithi194_121512_object98385:head# 2024-04-29T14:26:38.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6c9ff980:::benchmark_data_smithi194_121512_object121664:head# 2024-04-29T14:26:38.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca00fb7:::benchmark_data_smithi194_121512_object83439:head# 2024-04-29T14:26:38.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca06d27:::benchmark_data_smithi194_121512_object62977:head# 2024-04-29T14:26:38.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca0868f:::benchmark_data_smithi194_121512_object132851:head# 2024-04-29T14:26:38.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca087d0:::benchmark_data_smithi194_121512_object47985:head# 2024-04-29T14:26:38.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca2064b:::benchmark_data_smithi194_121512_object44787:head# 2024-04-29T14:26:38.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca3f36a:::benchmark_data_smithi194_121512_object119800:head# 2024-04-29T14:26:38.850 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:26:38.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca4ad75:::benchmark_data_smithi194_121512_object88595:head# 2024-04-29T14:26:38.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca5613a:::benchmark_data_smithi194_121512_object47139:head# 2024-04-29T14:26:38.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca60fb3:::benchmark_data_smithi194_121512_object78763:head# 2024-04-29T14:26:38.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca6bb7e:::benchmark_data_smithi194_121512_object61082:head# 2024-04-29T14:26:38.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca74e45:::benchmark_data_smithi194_121512_object68010:head# 2024-04-29T14:26:38.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca74e88:::benchmark_data_smithi194_121512_object51208:head# 2024-04-29T14:26:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:38.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:26:38.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca781ee:::benchmark_data_smithi194_121512_object94470:head# 2024-04-29T14:26:38.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca7acd9:::benchmark_data_smithi194_121512_object51018:head# 2024-04-29T14:26:38.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca8241d:::benchmark_data_smithi194_121512_object90680:head# 2024-04-29T14:26:38.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ca9cb6f:::benchmark_data_smithi194_121512_object24350:head# 2024-04-29T14:26:38.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6caa3361:::benchmark_data_smithi194_121512_object69457:head# 2024-04-29T14:26:38.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6caac4d8:::benchmark_data_smithi194_121512_object99170:head# 2024-04-29T14:26:38.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6caac4e0:::benchmark_data_smithi194_121512_object23167:head# 2024-04-29T14:26:38.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cab0f64:::benchmark_data_smithi194_121512_object129814:head# 2024-04-29T14:26:38.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cab5b86:::benchmark_data_smithi194_121512_object3004:head# 2024-04-29T14:26:38.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cab88f6:::benchmark_data_smithi194_121512_object116447:head# 2024-04-29T14:26:38.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cabdd68:::benchmark_data_smithi194_121512_object26734:head# 2024-04-29T14:26:38.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6caca3c6:::benchmark_data_smithi194_121512_object54545:head# 2024-04-29T14:26:38.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cace3d2:::benchmark_data_smithi194_121512_object78719:head# 2024-04-29T14:26:38.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cacf70f:::benchmark_data_smithi194_121512_object9312:head# 2024-04-29T14:26:38.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cacfc7b:::benchmark_data_smithi194_121512_object24693:head# 2024-04-29T14:26:38.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cadc7d6:::benchmark_data_smithi194_121512_object92470:head# 2024-04-29T14:26:38.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cafdee6:::benchmark_data_smithi194_121512_object126881:head# 2024-04-29T14:26:38.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb03fbb:::benchmark_data_smithi194_121512_object32500:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb12a09:::benchmark_data_smithi194_121512_object5231:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb149c9:::benchmark_data_smithi194_121512_object108351:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb230b8:::benchmark_data_smithi194_121512_object57405:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb31fdf:::benchmark_data_smithi194_121512_object71379:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb32cfb:::benchmark_data_smithi194_121512_object100450:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb35f81:::benchmark_data_smithi194_121512_object41006:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb3f08f:::benchmark_data_smithi194_121512_object67857:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb50c6b:::benchmark_data_smithi194_121512_object86392:head# 2024-04-29T14:26:38.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb51929:::benchmark_data_smithi194_121512_object104783:head# 2024-04-29T14:26:38.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb62a12:::benchmark_data_smithi194_121512_object11239:head# 2024-04-29T14:26:38.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb6c703:::benchmark_data_smithi194_121512_object130819:head# 2024-04-29T14:26:38.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb74ea8:::benchmark_data_smithi194_121512_object113463:head# 2024-04-29T14:26:38.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb8412c:::benchmark_data_smithi194_121512_object89474:head# 2024-04-29T14:26:38.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb84d73:::benchmark_data_smithi194_121512_object110827:head# 2024-04-29T14:26:38.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb87703:::benchmark_data_smithi194_121512_object114592:head# 2024-04-29T14:26:38.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb8c136:::benchmark_data_smithi194_121512_object126673:head# 2024-04-29T14:26:38.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb8cfef:::benchmark_data_smithi194_121512_object69875:head# 2024-04-29T14:26:38.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb9543f:::benchmark_data_smithi194_121512_object80058:head# 2024-04-29T14:26:38.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb96f1e:::benchmark_data_smithi194_121512_object124440:head# 2024-04-29T14:26:38.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cb9b49e:::benchmark_data_smithi194_121512_object10307:head# 2024-04-29T14:26:38.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cba1686:::benchmark_data_smithi194_121512_object22815:head# 2024-04-29T14:26:38.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cba3047:::benchmark_data_smithi194_121512_object73676:head# 2024-04-29T14:26:38.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cba865a:::benchmark_data_smithi194_121512_object70246:head# 2024-04-29T14:26:38.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cba9a9b:::benchmark_data_smithi194_121512_object78282:head# 2024-04-29T14:26:38.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbaf544:::benchmark_data_smithi194_121512_object125981:head# 2024-04-29T14:26:38.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbbdfb5:::benchmark_data_smithi194_121512_object5353:head# 2024-04-29T14:26:38.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbca524:::benchmark_data_smithi194_121512_object30548:head# 2024-04-29T14:26:38.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbcb5b7:::benchmark_data_smithi194_121512_object60910:head# 2024-04-29T14:26:38.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbd47fa:::benchmark_data_smithi194_121512_object84674:head# 2024-04-29T14:26:38.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbd743d:::benchmark_data_smithi194_121512_object106076:head# 2024-04-29T14:26:38.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbe233e:::benchmark_data_smithi194_121512_object60864:head# 2024-04-29T14:26:38.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbf5097:::benchmark_data_smithi194_121512_object132016:head# 2024-04-29T14:26:38.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbf72d5:::benchmark_data_smithi194_121512_object3962:head# 2024-04-29T14:26:38.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbf7667:::benchmark_data_smithi194_121512_object99919:head# 2024-04-29T14:26:38.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cbfd9b0:::benchmark_data_smithi194_121512_object43388:head# 2024-04-29T14:26:38.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc026ab:::benchmark_data_smithi194_121512_object67503:head# 2024-04-29T14:26:38.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc036a5:::benchmark_data_smithi194_121512_object14181:head# 2024-04-29T14:26:38.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc0f919:::benchmark_data_smithi194_121512_object71680:head# 2024-04-29T14:26:38.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc10fab:::benchmark_data_smithi194_121512_object54931:head# 2024-04-29T14:26:38.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc19cba:::benchmark_data_smithi194_121512_object45637:head# 2024-04-29T14:26:38.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc1ad61:::benchmark_data_smithi194_121512_object95814:head# 2024-04-29T14:26:38.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc24d41:::benchmark_data_smithi194_121512_object103512:head# 2024-04-29T14:26:38.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc34d19:::benchmark_data_smithi194_121512_object113896:head# 2024-04-29T14:26:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc387ad:::benchmark_data_smithi194_121512_object62592:head# 2024-04-29T14:26:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc3cbea:::benchmark_data_smithi194_121512_object120668:head# 2024-04-29T14:26:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc3e363:::benchmark_data_smithi194_121512_object130932:head# 2024-04-29T14:26:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc3fdfb:::benchmark_data_smithi194_121512_object32172:head# 2024-04-29T14:26:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc4cdea:::benchmark_data_smithi194_121512_object38040:head# 2024-04-29T14:26:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc5f3dc:::benchmark_data_smithi194_121512_object101510:head# 2024-04-29T14:26:38.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc695ed:::benchmark_data_smithi194_121512_object26438:head# 2024-04-29T14:26:38.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc69ce2:::benchmark_data_smithi194_121512_object57437:head# 2024-04-29T14:26:38.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc70f01:::benchmark_data_smithi194_121512_object61004:head# 2024-04-29T14:26:38.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc75a12:::benchmark_data_smithi194_121512_object47589:head# 2024-04-29T14:26:38.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc7f845:::benchmark_data_smithi194_121512_object7114:head# 2024-04-29T14:26:38.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc83154:::benchmark_data_smithi194_121512_object128318:head# 2024-04-29T14:26:38.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc88829:::benchmark_data_smithi194_121512_object72344:head# 2024-04-29T14:26:38.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc893ce:::benchmark_data_smithi194_121512_object121644:head# 2024-04-29T14:26:38.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc8a7a8:::benchmark_data_smithi194_121512_object93887:head# 2024-04-29T14:26:38.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc97218:::benchmark_data_smithi194_121512_object43284:head# 2024-04-29T14:26:38.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc9a4e8:::benchmark_data_smithi194_121512_object34458:head# 2024-04-29T14:26:38.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cc9c10a:::benchmark_data_smithi194_121512_object110120:head# 2024-04-29T14:26:38.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cca31f8:::benchmark_data_smithi194_121512_object58390:head# 2024-04-29T14:26:38.895 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cca9a24:::benchmark_data_smithi194_121512_object79826:head# 2024-04-29T14:26:38.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccac270:::benchmark_data_smithi194_121512_object21184:head# 2024-04-29T14:26:38.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccb717f:::benchmark_data_smithi194_121512_object17731:head# 2024-04-29T14:26:38.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccbac8f:::benchmark_data_smithi194_121512_object63499:head# 2024-04-29T14:26:38.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccbd4b5:::benchmark_data_smithi194_121512_object91437:head# 2024-04-29T14:26:38.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccc1d1e:::benchmark_data_smithi194_121512_object108131:head# 2024-04-29T14:26:38.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccd0392:::benchmark_data_smithi194_121512_object38268:head# 2024-04-29T14:26:38.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccd3471:::benchmark_data_smithi194_121512_object29675:head# 2024-04-29T14:26:38.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccefbcf:::benchmark_data_smithi194_121512_object85954:head# 2024-04-29T14:26:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ccf1ce4:::benchmark_data_smithi194_121512_object133026:head# 2024-04-29T14:26:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd04936:::benchmark_data_smithi194_121512_object52394:head# 2024-04-29T14:26:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd0e00d:::benchmark_data_smithi194_121512_object8912:head# 2024-04-29T14:26:38.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd182da:::benchmark_data_smithi194_121512_object18986:head# 2024-04-29T14:26:38.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd1aae2:::benchmark_data_smithi194_121512_object108532:head# 2024-04-29T14:26:38.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd1b857:::benchmark_data_smithi194_121512_object71922:head# 2024-04-29T14:26:38.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd1e8dd:::benchmark_data_smithi194_121512_object123114:head# 2024-04-29T14:26:38.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd26530:::benchmark_data_smithi194_121512_object24282:head# 2024-04-29T14:26:38.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd26c07:::benchmark_data_smithi194_121512_object40858:head# 2024-04-29T14:26:38.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd29c6f:::benchmark_data_smithi194_121512_object55797:head# 2024-04-29T14:26:38.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd32775:::benchmark_data_smithi194_121512_object84694:head# 2024-04-29T14:26:38.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd39f47:::benchmark_data_smithi194_121512_object42838:head# 2024-04-29T14:26:38.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd3f05b:::benchmark_data_smithi194_121512_object22260:head# 2024-04-29T14:26:38.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd4280c:::benchmark_data_smithi194_121512_object25974:head# 2024-04-29T14:26:38.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd4b96d:::benchmark_data_smithi194_121512_object100561:head# 2024-04-29T14:26:38.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd543df:::benchmark_data_smithi194_121512_object1957:head# 2024-04-29T14:26:38.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd5f049:::benchmark_data_smithi194_121512_object68306:head# 2024-04-29T14:26:38.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd62391:::benchmark_data_smithi194_121512_object109151:head# 2024-04-29T14:26:38.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd6b515:::benchmark_data_smithi194_121512_object126228:head# 2024-04-29T14:26:38.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd6c0a5:::benchmark_data_smithi194_121512_object55020:head# 2024-04-29T14:26:38.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd6c3db:::benchmark_data_smithi194_121512_object4619:head# 2024-04-29T14:26:38.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cd96ba8:::benchmark_data_smithi194_121512_object7025:head# 2024-04-29T14:26:38.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cda5add:::benchmark_data_smithi194_121512_object67375:head# 2024-04-29T14:26:38.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cda7b8d:::benchmark_data_smithi194_121512_object33217:head# 2024-04-29T14:26:38.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cda7b95:::benchmark_data_smithi194_121512_object60305:head# 2024-04-29T14:26:38.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cda834d:::benchmark_data_smithi194_121512_object129122:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdab632:::benchmark_data_smithi194_121512_object69783:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdb1356:::benchmark_data_smithi194_121512_object26390:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdba2d4:::benchmark_data_smithi194_121512_object59279:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdbfbdd:::benchmark_data_smithi194_121512_object10340:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdcc2b8:::benchmark_data_smithi194_121512_object133084:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdcc43a:::benchmark_data_smithi194_121512_object69055:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdd17fc:::benchmark_data_smithi194_121512_object39972:head# 2024-04-29T14:26:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cddbefd:::benchmark_data_smithi194_121512_object41964:head# 2024-04-29T14:26:38.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cddf372:::benchmark_data_smithi194_121512_object4842:head# 2024-04-29T14:26:38.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cddfdf9:::benchmark_data_smithi194_121512_object68695:head# 2024-04-29T14:26:38.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdef16c:::benchmark_data_smithi194_121512_object90066:head# 2024-04-29T14:26:38.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdf7f86:::benchmark_data_smithi194_121512_object29697:head# 2024-04-29T14:26:38.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cdfbdcb:::benchmark_data_smithi194_121512_object42115:head# 2024-04-29T14:26:38.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce00ac8:::benchmark_data_smithi194_121512_object41900:head# 2024-04-29T14:26:38.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce0c04e:::benchmark_data_smithi194_121512_object50001:head# 2024-04-29T14:26:38.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce10f24:::benchmark_data_smithi194_121512_object59261:head# 2024-04-29T14:26:38.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce28d3f:::benchmark_data_smithi194_121512_object61894:head# 2024-04-29T14:26:38.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce32ae6:::benchmark_data_smithi194_121512_object16850:head# 2024-04-29T14:26:38.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce64e1b:::benchmark_data_smithi194_121512_object59374:head# 2024-04-29T14:26:38.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce65cc5:::benchmark_data_smithi194_121512_object119899:head# 2024-04-29T14:26:38.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce6bb40:::benchmark_data_smithi194_121512_object24493:head# 2024-04-29T14:26:38.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce6d1c0:::benchmark_data_smithi194_121512_object75626:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce713d2:::benchmark_data_smithi194_121512_object62523:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce83bb0:::benchmark_data_smithi194_121512_object73617:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce8a7da:::benchmark_data_smithi194_121512_object8901:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce94d2a:::benchmark_data_smithi194_121512_object119682:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ce9eba0:::benchmark_data_smithi194_121512_object154:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cea1e56:::benchmark_data_smithi194_121512_object22984:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ceb115a:::benchmark_data_smithi194_121512_object7607:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ceb34df:::benchmark_data_smithi194_121512_object92526:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cec9cd4:::benchmark_data_smithi194_121512_object100185:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ceca475:::benchmark_data_smithi194_121512_object130377:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cee0092:::benchmark_data_smithi194_121512_object94734:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cee66f9:::benchmark_data_smithi194_121512_object74097:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cef4c6b:::benchmark_data_smithi194_121512_object95618:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cef5cc3:::benchmark_data_smithi194_121512_object18482:head# 2024-04-29T14:26:38.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cefc394:::benchmark_data_smithi194_121512_object61380:head# 2024-04-29T14:26:38.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf09d29:::benchmark_data_smithi194_121512_object15805:head# 2024-04-29T14:26:38.930 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:38.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf0b121:::benchmark_data_smithi194_121512_object13080:head# 2024-04-29T14:26:38.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf0b88a:::benchmark_data_smithi194_121512_object49006:head# 2024-04-29T14:26:38.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf1c4cb:::benchmark_data_smithi194_121512_object82822:head# 2024-04-29T14:26:38.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf1ed98:::benchmark_data_smithi194_121512_object119730:head# 2024-04-29T14:26:38.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf2556d:::benchmark_data_smithi194_121512_object28749:head# 2024-04-29T14:26:38.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf2700b:::benchmark_data_smithi194_121512_object31523:head# 2024-04-29T14:26:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf2b186:::benchmark_data_smithi194_121512_object17255:head# 2024-04-29T14:26:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf2c075:::benchmark_data_smithi194_121512_object87673:head# 2024-04-29T14:26:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf36cea:::benchmark_data_smithi194_121512_object53971:head# 2024-04-29T14:26:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf36e2c:::benchmark_data_smithi194_121512_object15505:head# 2024-04-29T14:26:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf3c341:::benchmark_data_smithi194_121512_object113420:head# 2024-04-29T14:26:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf464dc:::benchmark_data_smithi194_121512_object11261:head# 2024-04-29T14:26:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf47000:::benchmark_data_smithi194_121512_object91096:head# 2024-04-29T14:26:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf551c9:::benchmark_data_smithi194_121512_object38206:head# 2024-04-29T14:26:38.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf5983b:::benchmark_data_smithi194_121512_object8040:head# 2024-04-29T14:26:38.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf639e0:::benchmark_data_smithi194_121512_object101605:head# 2024-04-29T14:26:38.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf6c64e:::benchmark_data_smithi194_121512_object58784:head# 2024-04-29T14:26:38.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf6cd31:::benchmark_data_smithi194_121512_object94643:head# 2024-04-29T14:26:38.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf6cdea:::benchmark_data_smithi194_121512_object85270:head# 2024-04-29T14:26:38.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf70b2e:::benchmark_data_smithi194_121512_object56209:head# 2024-04-29T14:26:38.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf79643:::benchmark_data_smithi194_121512_object94269:head# 2024-04-29T14:26:38.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf80b40:::benchmark_data_smithi194_121512_object19289:head# 2024-04-29T14:26:38.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf86cee:::benchmark_data_smithi194_121512_object59329:head# 2024-04-29T14:26:38.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf8c27f:::benchmark_data_smithi194_121512_object111335:head# 2024-04-29T14:26:38.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf91467:::benchmark_data_smithi194_121512_object58164:head# 2024-04-29T14:26:38.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf9fba4:::benchmark_data_smithi194_121512_object58485:head# 2024-04-29T14:26:38.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cf9fcbe:::benchmark_data_smithi194_121512_object96911:head# 2024-04-29T14:26:38.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfb57d1:::benchmark_data_smithi194_121512_object23668:head# 2024-04-29T14:26:38.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfbd447:::benchmark_data_smithi194_121512_object88422:head# 2024-04-29T14:26:38.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfbe18c:::benchmark_data_smithi194_121512_object46127:head# 2024-04-29T14:26:38.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfc06c1:::benchmark_data_smithi194_121512_object49509:head# 2024-04-29T14:26:38.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfc19bb:::benchmark_data_smithi194_121512_object22644:head# 2024-04-29T14:26:38.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfc7ce5:::benchmark_data_smithi194_121512_object74662:head# 2024-04-29T14:26:38.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfc8add:::benchmark_data_smithi194_121512_object15596:head# 2024-04-29T14:26:38.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfcbacb:::benchmark_data_smithi194_121512_object130433:head# 2024-04-29T14:26:38.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfd0344:::benchmark_data_smithi194_121512_object99921:head# 2024-04-29T14:26:38.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfd440b:::benchmark_data_smithi194_121512_object13220:head# 2024-04-29T14:26:38.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfd4eea:::benchmark_data_smithi194_121512_object49535:head# 2024-04-29T14:26:38.950 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:38.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfe1a51:::benchmark_data_smithi194_121512_object116737:head# 2024-04-29T14:26:38.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfe43e1:::benchmark_data_smithi194_121512_object109996:head# 2024-04-29T14:26:38.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cfe6f3a:::benchmark_data_smithi194_121512_object133216:head# 2024-04-29T14:26:38.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cff125c:::benchmark_data_smithi194_121512_object84075:head# 2024-04-29T14:26:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cff4837:::benchmark_data_smithi194_121512_object52776:head# 2024-04-29T14:26:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6cff67b1:::benchmark_data_smithi194_121512_object97898:head# 2024-04-29T14:26:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d00a13f:::benchmark_data_smithi194_121512_object57154:head# 2024-04-29T14:26:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0107c7:::benchmark_data_smithi194_121512_object89425:head# 2024-04-29T14:26:38.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0134c0:::benchmark_data_smithi194_121512_object56321:head# 2024-04-29T14:26:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d01ac0f:::benchmark_data_smithi194_121512_object16723:head# 2024-04-29T14:26:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d02165e:::benchmark_data_smithi194_121512_object3899:head# 2024-04-29T14:26:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d026176:::benchmark_data_smithi194_121512_object60005:head# 2024-04-29T14:26:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d02cf0f:::benchmark_data_smithi194_121512_object44946:head# 2024-04-29T14:26:38.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0395bc:::benchmark_data_smithi194_121512_object97085:head# 2024-04-29T14:26:38.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d03c913:::benchmark_data_smithi194_121512_object59675:head# 2024-04-29T14:26:38.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d060140:::benchmark_data_smithi194_121512_object109538:head# 2024-04-29T14:26:38.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0678c1:::benchmark_data_smithi194_121512_object75727:head# 2024-04-29T14:26:38.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0829bb:::benchmark_data_smithi194_121512_object101188:head# 2024-04-29T14:26:38.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d099640:::benchmark_data_smithi194_121512_object121072:head# 2024-04-29T14:26:38.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d099f1c:::benchmark_data_smithi194_121512_object127105:head# 2024-04-29T14:26:38.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d09dd08:::benchmark_data_smithi194_121512_object53987:head# 2024-04-29T14:26:38.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0a4877:::benchmark_data_smithi194_121512_object122903:head# 2024-04-29T14:26:38.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0aee0b:::benchmark_data_smithi194_121512_object52792:head# 2024-04-29T14:26:38.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0bbc72:::benchmark_data_smithi194_121512_object93264:head# 2024-04-29T14:26:38.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0c56d4:::benchmark_data_smithi194_121512_object49222:head# 2024-04-29T14:26:38.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0c6c84:::benchmark_data_smithi194_121512_object47899:head# 2024-04-29T14:26:38.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0d05a9:::benchmark_data_smithi194_121512_object76858:head# 2024-04-29T14:26:38.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0d14a4:::benchmark_data_smithi194_121512_object15930:head# 2024-04-29T14:26:38.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0d345f:::benchmark_data_smithi194_121512_object47651:head# 2024-04-29T14:26:38.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0d8023:::benchmark_data_smithi194_121512_object99713:head# 2024-04-29T14:26:38.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0d8ab5:::benchmark_data_smithi194_121512_object102853:head# 2024-04-29T14:26:38.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0eb849:::benchmark_data_smithi194_121512_object16885:head# 2024-04-29T14:26:38.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d0fd578:::benchmark_data_smithi194_121512_object42520:head# 2024-04-29T14:26:38.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d10212e:::benchmark_data_smithi194_121512_object96727:head# 2024-04-29T14:26:38.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d104736:::benchmark_data_smithi194_121512_object65173:head# 2024-04-29T14:26:38.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d10bfaa:::benchmark_data_smithi194_121512_object61281:head# 2024-04-29T14:26:38.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d115dc8:::benchmark_data_smithi194_121512_object130471:head# 2024-04-29T14:26:38.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d117526:::benchmark_data_smithi194_121512_object17472:head# 2024-04-29T14:26:38.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d118faa:::benchmark_data_smithi194_121512_object132065:head# 2024-04-29T14:26:38.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d12387b:::benchmark_data_smithi194_121512_object95383:head# 2024-04-29T14:26:38.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d128b03:::benchmark_data_smithi194_121512_object112133:head# 2024-04-29T14:26:38.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d12e917:::benchmark_data_smithi194_121512_object102380:head# 2024-04-29T14:26:38.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1394a1:::benchmark_data_smithi194_121512_object73798:head# 2024-04-29T14:26:38.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d141aa8:::benchmark_data_smithi194_121512_object107837:head# 2024-04-29T14:26:38.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1503ac:::benchmark_data_smithi194_121512_object43832:head# 2024-04-29T14:26:38.977 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:38.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1545fa:::benchmark_data_smithi194_121512_object120171:head# 2024-04-29T14:26:38.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d154878:::benchmark_data_smithi194_121512_object38607:head# 2024-04-29T14:26:38.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d15b85b:::benchmark_data_smithi194_121512_object16798:head# 2024-04-29T14:26:38.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d16ea96:::benchmark_data_smithi194_121512_object83424:head# 2024-04-29T14:26:38.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d17589c:::benchmark_data_smithi194_121512_object129816:head# 2024-04-29T14:26:38.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d179daf:::benchmark_data_smithi194_121512_object76445:head# 2024-04-29T14:26:38.986 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:38.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d17c044:::benchmark_data_smithi194_121512_object126664:head# 2024-04-29T14:26:38.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d17f3f5:::benchmark_data_smithi194_121512_object23604:head# 2024-04-29T14:26:38.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d181377:::benchmark_data_smithi194_121512_object80926:head# 2024-04-29T14:26:38.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d197372:::benchmark_data_smithi194_121512_object89259:head# 2024-04-29T14:26:38.996 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1a68eb:::benchmark_data_smithi194_121512_object38017:head# 2024-04-29T14:26:38.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1d7ead:::benchmark_data_smithi194_121512_object95339:head# 2024-04-29T14:26:38.997 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:39.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1db0e7:::benchmark_data_smithi194_121512_object68052:head# 2024-04-29T14:26:39.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1dcbf1:::benchmark_data_smithi194_121512_object103182:head# 2024-04-29T14:26:39.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1df985:::benchmark_data_smithi194_121512_object89944:head# 2024-04-29T14:26:39.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1e3b00:::benchmark_data_smithi194_121512_object128996:head# 2024-04-29T14:26:39.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1e3f2f:::benchmark_data_smithi194_121512_object21432:head# 2024-04-29T14:26:39.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1ebd10:::benchmark_data_smithi194_121512_object126269:head# 2024-04-29T14:26:39.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1f3580:::benchmark_data_smithi194_121512_object36542:head# 2024-04-29T14:26:39.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1f5c81:::benchmark_data_smithi194_121512_object88281:head# 2024-04-29T14:26:39.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1fa1e0:::benchmark_data_smithi194_121512_object113922:head# 2024-04-29T14:26:39.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1fdab8:::benchmark_data_smithi194_121512_object53242:head# 2024-04-29T14:26:39.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1fdd40:::benchmark_data_smithi194_121512_object121315:head# 2024-04-29T14:26:39.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d1fe761:::benchmark_data_smithi194_121512_object69822:head# 2024-04-29T14:26:39.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d200c5b:::benchmark_data_smithi194_121512_object83896:head# 2024-04-29T14:26:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d20360f:::benchmark_data_smithi194_121512_object81860:head# 2024-04-29T14:26:39.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d21fa80:::benchmark_data_smithi194_121512_object88049:head# 2024-04-29T14:26:39.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d22a658:::benchmark_data_smithi194_121512_object50132:head# 2024-04-29T14:26:39.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d23bd3c:::benchmark_data_smithi194_121512_object109473:head# 2024-04-29T14:26:39.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2450c5:::benchmark_data_smithi194_121512_object106781:head# 2024-04-29T14:26:39.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d25ab07:::benchmark_data_smithi194_121512_object1650:head# 2024-04-29T14:26:39.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2705a3:::benchmark_data_smithi194_121512_object102162:head# 2024-04-29T14:26:39.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2796f7:::benchmark_data_smithi194_121512_object12295:head# 2024-04-29T14:26:39.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2901d1:::benchmark_data_smithi194_121512_object118093:head# 2024-04-29T14:26:39.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d297e33:::benchmark_data_smithi194_121512_object129818:head# 2024-04-29T14:26:39.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d29840c:::benchmark_data_smithi194_121512_object39477:head# 2024-04-29T14:26:39.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2a401c:::benchmark_data_smithi194_121512_object97836:head# 2024-04-29T14:26:39.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2a79f2:::benchmark_data_smithi194_121512_object53397:head# 2024-04-29T14:26:39.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2ab758:::benchmark_data_smithi194_121512_object106087:head# 2024-04-29T14:26:39.023 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:39.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2bd545:::benchmark_data_smithi194_121512_object122952:head# 2024-04-29T14:26:39.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2be359:::benchmark_data_smithi194_121512_object131162:head# 2024-04-29T14:26:39.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2d02e4:::benchmark_data_smithi194_121512_object85511:head# 2024-04-29T14:26:39.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2d6a25:::benchmark_data_smithi194_121512_object81724:head# 2024-04-29T14:26:39.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2e5ab1:::benchmark_data_smithi194_121512_object78582:head# 2024-04-29T14:26:39.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2eb9ac:::benchmark_data_smithi194_121512_object71845:head# 2024-04-29T14:26:39.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2ef5af:::benchmark_data_smithi194_121512_object125413:head# 2024-04-29T14:26:39.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2f8272:::benchmark_data_smithi194_121512_object70712:head# 2024-04-29T14:26:39.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d2fdabd:::benchmark_data_smithi194_121512_object31948:head# 2024-04-29T14:26:39.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d304e51:::benchmark_data_smithi194_121512_object13680:head# 2024-04-29T14:26:39.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d308367:::benchmark_data_smithi194_121512_object52461:head# 2024-04-29T14:26:39.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d309fb3:::benchmark_data_smithi194_121512_object81140:head# 2024-04-29T14:26:39.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d312953:::benchmark_data_smithi194_121512_object40047:head# 2024-04-29T14:26:39.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d313af2:::benchmark_data_smithi194_121512_object71195:head# 2024-04-29T14:26:39.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d313f12:::benchmark_data_smithi194_121512_object102804:head# 2024-04-29T14:26:39.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3151a3:::benchmark_data_smithi194_121512_object96501:head# 2024-04-29T14:26:39.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d315260:::benchmark_data_smithi194_121512_object39075:head# 2024-04-29T14:26:39.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d31da1a:::benchmark_data_smithi194_121512_object70683:head# 2024-04-29T14:26:39.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3306af:::benchmark_data_smithi194_121512_object122031:head# 2024-04-29T14:26:39.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d334c49:::benchmark_data_smithi194_121512_object86591:head# 2024-04-29T14:26:39.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d33a9ba:::benchmark_data_smithi194_121512_object64586:head# 2024-04-29T14:26:39.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d33d86a:::benchmark_data_smithi194_121512_object15740:head# 2024-04-29T14:26:39.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d34a0de:::benchmark_data_smithi194_121512_object53896:head# 2024-04-29T14:26:39.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d35abbe:::benchmark_data_smithi194_121512_object36226:head# 2024-04-29T14:26:39.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d36376d:::benchmark_data_smithi194_121512_object72382:head# 2024-04-29T14:26:39.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d36a995:::benchmark_data_smithi194_121512_object122404:head# 2024-04-29T14:26:39.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d37970d:::benchmark_data_smithi194_121512_object107333:head# 2024-04-29T14:26:39.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d37a31e:::benchmark_data_smithi194_121512_object95755:head# 2024-04-29T14:26:39.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d37c499:::benchmark_data_smithi194_121512_object54110:head# 2024-04-29T14:26:39.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d38280c:::benchmark_data_smithi194_121512_object72979:head# 2024-04-29T14:26:39.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d382de4:::benchmark_data_smithi194_121512_object47125:head# 2024-04-29T14:26:39.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d386674:::benchmark_data_smithi194_121512_object66733:head# 2024-04-29T14:26:39.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d395d23:::benchmark_data_smithi194_121512_object60204:head# 2024-04-29T14:26:39.040 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d398e16:::benchmark_data_smithi194_121512_object113435:head# 2024-04-29T14:26:39.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d39e342:::benchmark_data_smithi194_121512_object90779:head# 2024-04-29T14:26:39.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3b1e41:::benchmark_data_smithi194_121512_object116168:head# 2024-04-29T14:26:39.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3b5149:::benchmark_data_smithi194_121512_object127606:head# 2024-04-29T14:26:39.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3ba153:::benchmark_data_smithi194_121512_object100194:head# 2024-04-29T14:26:39.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3c12e0:::benchmark_data_smithi194_121512_object87585:head# 2024-04-29T14:26:39.044 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:39.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3cb9dd:::benchmark_data_smithi194_121512_object131192:head# 2024-04-29T14:26:39.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3d03bd:::benchmark_data_smithi194_121512_object18259:head# 2024-04-29T14:26:39.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3d06c5:::benchmark_data_smithi194_121512_object110916:head# 2024-04-29T14:26:39.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3d4f29:::benchmark_data_smithi194_121512_object49460:head# 2024-04-29T14:26:39.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3d9518:::benchmark_data_smithi194_121512_object62750:head# 2024-04-29T14:26:39.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3e2f97:::benchmark_data_smithi194_121512_object52330:head# 2024-04-29T14:26:39.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3e38fa:::benchmark_data_smithi194_121512_object71616:head# 2024-04-29T14:26:39.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3eb263:::benchmark_data_smithi194_121512_object6515:head# 2024-04-29T14:26:39.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3f4655:::benchmark_data_smithi194_121512_object1663:head# 2024-04-29T14:26:39.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3f9445:::benchmark_data_smithi194_121512_object4608:head# 2024-04-29T14:26:39.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3fc27d:::benchmark_data_smithi194_121512_object32890:head# 2024-04-29T14:26:39.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3fcbe5:::benchmark_data_smithi194_121512_object90752:head# 2024-04-29T14:26:39.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3fdeb6:::benchmark_data_smithi194_121512_object115488:head# 2024-04-29T14:26:39.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3fe634:::benchmark_data_smithi194_121512_object7308:head# 2024-04-29T14:26:39.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3ff01d:::benchmark_data_smithi194_121512_object133377:head# 2024-04-29T14:26:39.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d3ff7eb:::benchmark_data_smithi194_121512_object108921:head# 2024-04-29T14:26:39.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d41b55e:::benchmark_data_smithi194_121512_object8727:head# 2024-04-29T14:26:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d41eddc:::benchmark_data_smithi194_121512_object15896:head# 2024-04-29T14:26:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d423703:::benchmark_data_smithi194_121512_object49273:head# 2024-04-29T14:26:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d42a358:::benchmark_data_smithi194_121512_object62830:head# 2024-04-29T14:26:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d454329:::benchmark_data_smithi194_121512_object32312:head# 2024-04-29T14:26:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d455fdb:::benchmark_data_smithi194_121512_object42050:head# 2024-04-29T14:26:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d45ce6b:::benchmark_data_smithi194_121512_object119717:head# 2024-04-29T14:26:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4604dc:::benchmark_data_smithi194_121512_object105597:head# 2024-04-29T14:26:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d469b85:::benchmark_data_smithi194_121512_object68087:head# 2024-04-29T14:26:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d472338:::benchmark_data_smithi194_121512_object59665:head# 2024-04-29T14:26:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d482879:::benchmark_data_smithi194_121512_object108764:head# 2024-04-29T14:26:39.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4879ef:::benchmark_data_smithi194_121512_object76871:head# 2024-04-29T14:26:39.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4881bb:::benchmark_data_smithi194_121512_object88372:head# 2024-04-29T14:26:39.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d491647:::benchmark_data_smithi194_121512_object125331:head# 2024-04-29T14:26:39.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4948f7:::benchmark_data_smithi194_121512_object38382:head# 2024-04-29T14:26:39.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d496891:::benchmark_data_smithi194_121512_object121241:head# 2024-04-29T14:26:39.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4a79dc:::benchmark_data_smithi194_121512_object1988:head# 2024-04-29T14:26:39.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4b7cac:::benchmark_data_smithi194_121512_object9550:head# 2024-04-29T14:26:39.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4b7cc0:::benchmark_data_smithi194_121512_object82974:head# 2024-04-29T14:26:39.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4c6ac0:::benchmark_data_smithi194_121512_object7294:head# 2024-04-29T14:26:39.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4cab5b:::benchmark_data_smithi194_121512_object74117:head# 2024-04-29T14:26:39.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4e27fc:::benchmark_data_smithi194_121512_object125656:head# 2024-04-29T14:26:39.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4e2e10:::benchmark_data_smithi194_121512_object108242:head# 2024-04-29T14:26:39.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4e4135:::benchmark_data_smithi194_121512_object72297:head# 2024-04-29T14:26:39.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4ebebc:::benchmark_data_smithi194_121512_object68071:head# 2024-04-29T14:26:39.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d4f2502:::benchmark_data_smithi194_121512_object98520:head# 2024-04-29T14:26:39.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d50449d:::benchmark_data_smithi194_121512_object2979:head# 2024-04-29T14:26:39.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d508925:::benchmark_data_smithi194_121512_object49243:head# 2024-04-29T14:26:39.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d50ef93:::benchmark_data_smithi194_121512_object57473:head# 2024-04-29T14:26:39.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d50f713:::benchmark_data_smithi194_121512_object8301:head# 2024-04-29T14:26:39.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d51c45a:::benchmark_data_smithi194_121512_object84087:head# 2024-04-29T14:26:39.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d52c4c3:::benchmark_data_smithi194_121512_object83819:head# 2024-04-29T14:26:39.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d52d064:::benchmark_data_smithi194_121512_object37970:head# 2024-04-29T14:26:39.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5351f8:::benchmark_data_smithi194_121512_object129762:head# 2024-04-29T14:26:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d539e73:::benchmark_data_smithi194_121512_object28391:head# 2024-04-29T14:26:39.065 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d53c314:::benchmark_data_smithi194_121512_object126108:head# 2024-04-29T14:26:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d552717:::benchmark_data_smithi194_121512_object111386:head# 2024-04-29T14:26:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d555668:::benchmark_data_smithi194_121512_object39843:head# 2024-04-29T14:26:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d570d6f:::benchmark_data_smithi194_121512_object39757:head# 2024-04-29T14:26:39.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d59e27d:::benchmark_data_smithi194_121512_object102479:head# 2024-04-29T14:26:39.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d59f978:::benchmark_data_smithi194_121512_object34211:head# 2024-04-29T14:26:39.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d59fd76:::benchmark_data_smithi194_121512_object125775:head# 2024-04-29T14:26:39.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5a20a7:::benchmark_data_smithi194_121512_object42181:head# 2024-04-29T14:26:39.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5aaae3:::benchmark_data_smithi194_121512_object36568:head# 2024-04-29T14:26:39.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5b1374:::benchmark_data_smithi194_121512_object83650:head# 2024-04-29T14:26:39.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5b8679:::benchmark_data_smithi194_121512_object18290:head# 2024-04-29T14:26:39.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5c2813:::benchmark_data_smithi194_121512_object47592:head# 2024-04-29T14:26:39.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5c5bcc:::benchmark_data_smithi194_121512_object91738:head# 2024-04-29T14:26:39.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5cc059:::benchmark_data_smithi194_121512_object67896:head# 2024-04-29T14:26:39.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5e06f0:::benchmark_data_smithi194_121512_object120198:head# 2024-04-29T14:26:39.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5e138f:::benchmark_data_smithi194_121512_object31950:head# 2024-04-29T14:26:39.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d5fb686:::benchmark_data_smithi194_121512_object86578:head# 2024-04-29T14:26:39.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d608396:::benchmark_data_smithi194_121512_object89529:head# 2024-04-29T14:26:39.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d60e220:::benchmark_data_smithi194_121512_object50204:head# 2024-04-29T14:26:39.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d611a42:::benchmark_data_smithi194_121512_object82102:head# 2024-04-29T14:26:39.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d611f7e:::benchmark_data_smithi194_121512_object94738:head# 2024-04-29T14:26:39.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d61becf:::benchmark_data_smithi194_121512_object132385:head# 2024-04-29T14:26:39.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d634e6b:::benchmark_data_smithi194_121512_object20912:head# 2024-04-29T14:26:39.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d63d00e:::benchmark_data_smithi194_121512_object42570:head# 2024-04-29T14:26:39.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d648d8f:::benchmark_data_smithi194_121512_object59495:head# 2024-04-29T14:26:39.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d661d93:::benchmark_data_smithi194_121512_object120389:head# 2024-04-29T14:26:39.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d66bf60:::benchmark_data_smithi194_121512_object111726:head# 2024-04-29T14:26:39.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d688bcd:::benchmark_data_smithi194_121512_object102744:head# 2024-04-29T14:26:39.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d68ed23:::benchmark_data_smithi194_121512_object80029:head# 2024-04-29T14:26:39.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6908d8:::benchmark_data_smithi194_121512_object25330:head# 2024-04-29T14:26:39.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d690ab6:::benchmark_data_smithi194_121512_object48613:head# 2024-04-29T14:26:39.084 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d69713d:::benchmark_data_smithi194_121512_object113200:head# 2024-04-29T14:26:39.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d69a0e5:::benchmark_data_smithi194_121512_object40319:head# 2024-04-29T14:26:39.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6a47b6:::benchmark_data_smithi194_121512_object9879:head# 2024-04-29T14:26:39.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6af579:::benchmark_data_smithi194_121512_object26019:head# 2024-04-29T14:26:39.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6b92d4:::benchmark_data_smithi194_121512_object91799:head# 2024-04-29T14:26:39.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6c0593:::benchmark_data_smithi194_121512_object37116:head# 2024-04-29T14:26:39.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6c5e45:::benchmark_data_smithi194_121512_object114168:head# 2024-04-29T14:26:39.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6cce1e:::benchmark_data_smithi194_121512_object65695:head# 2024-04-29T14:26:39.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6cdd2d:::benchmark_data_smithi194_121512_object30763:head# 2024-04-29T14:26:39.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6d581e:::benchmark_data_smithi194_121512_object47828:head# 2024-04-29T14:26:39.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6d5a66:::benchmark_data_smithi194_121512_object44686:head# 2024-04-29T14:26:39.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6d6387:::benchmark_data_smithi194_121512_object95324:head# 2024-04-29T14:26:39.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6db56c:::benchmark_data_smithi194_121512_object32661:head# 2024-04-29T14:26:39.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d6e1ac0:::benchmark_data_smithi194_121512_object67245:head# 2024-04-29T14:26:39.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d701030:::benchmark_data_smithi194_121512_object68276:head# 2024-04-29T14:26:39.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d704a75:::benchmark_data_smithi194_121512_object110992:head# 2024-04-29T14:26:39.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d70e892:::benchmark_data_smithi194_121512_object105760:head# 2024-04-29T14:26:39.091 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:39.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7168e7:::benchmark_data_smithi194_121512_object114561:head# 2024-04-29T14:26:39.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d719a6e:::benchmark_data_smithi194_121512_object27987:head# 2024-04-29T14:26:39.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d720774:::benchmark_data_smithi194_121512_object115408:head# 2024-04-29T14:26:39.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d726063:::benchmark_data_smithi194_121512_object129911:head# 2024-04-29T14:26:39.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7276a8:::benchmark_data_smithi194_121512_object92549:head# 2024-04-29T14:26:39.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d730fcc:::benchmark_data_smithi194_121512_object818:head# 2024-04-29T14:26:39.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d73980c:::benchmark_data_smithi194_121512_object49254:head# 2024-04-29T14:26:39.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d73b40c:::benchmark_data_smithi194_121512_object51439:head# 2024-04-29T14:26:39.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d76744f:::benchmark_data_smithi194_121512_object53830:head# 2024-04-29T14:26:39.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d76d64b:::benchmark_data_smithi194_121512_object11415:head# 2024-04-29T14:26:39.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d77b07c:::benchmark_data_smithi194_121512_object56591:head# 2024-04-29T14:26:39.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7844a6:::benchmark_data_smithi194_121512_object120466:head# 2024-04-29T14:26:39.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7887a6:::benchmark_data_smithi194_121512_object30901:head# 2024-04-29T14:26:39.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d794636:::benchmark_data_smithi194_121512_object120035:head# 2024-04-29T14:26:39.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d79cd9a:::benchmark_data_smithi194_121512_object79093:head# 2024-04-29T14:26:39.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d79f8bf:::benchmark_data_smithi194_121512_object29640:head# 2024-04-29T14:26:39.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7a1273:::benchmark_data_smithi194_121512_object107063:head# 2024-04-29T14:26:39.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7a31ae:::benchmark_data_smithi194_121512_object128688:head# 2024-04-29T14:26:39.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7a345a:::benchmark_data_smithi194_121512_object85875:head# 2024-04-29T14:26:39.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7aed3b:::benchmark_data_smithi194_121512_object11631:head# 2024-04-29T14:26:39.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7b0909:::benchmark_data_smithi194_121512_object34278:head# 2024-04-29T14:26:39.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7b90e6:::benchmark_data_smithi194_121512_object79348:head# 2024-04-29T14:26:39.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7c5327:::benchmark_data_smithi194_121512_object116783:head# 2024-04-29T14:26:39.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7c55b1:::benchmark_data_smithi194_121512_object114874:head# 2024-04-29T14:26:39.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7d8445:::benchmark_data_smithi194_121512_object34999:head# 2024-04-29T14:26:39.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7dbaca:::benchmark_data_smithi194_121512_object18049:head# 2024-04-29T14:26:39.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7df5f1:::benchmark_data_smithi194_121512_object117836:head# 2024-04-29T14:26:39.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7e1bbf:::benchmark_data_smithi194_121512_object88978:head# 2024-04-29T14:26:39.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7f452b:::benchmark_data_smithi194_121512_object99777:head# 2024-04-29T14:26:39.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d7fd544:::benchmark_data_smithi194_121512_object40519:head# 2024-04-29T14:26:39.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d80a29f:::benchmark_data_smithi194_121512_object16594:head# 2024-04-29T14:26:39.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d80adf9:::benchmark_data_smithi194_121512_object57548:head# 2024-04-29T14:26:39.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d80f63e:::benchmark_data_smithi194_121512_object42054:head# 2024-04-29T14:26:39.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d81e6e1:::benchmark_data_smithi194_121512_object17146:head# 2024-04-29T14:26:39.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d821c44:::benchmark_data_smithi194_121512_object52890:head# 2024-04-29T14:26:39.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d824120:::benchmark_data_smithi194_121512_object60779:head# 2024-04-29T14:26:39.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d828b92:::benchmark_data_smithi194_121512_object106732:head# 2024-04-29T14:26:39.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d82ade5:::benchmark_data_smithi194_121512_object96106:head# 2024-04-29T14:26:39.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d83ac91:::benchmark_data_smithi194_121512_object93451:head# 2024-04-29T14:26:39.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d83e80f:::benchmark_data_smithi194_121512_object70213:head# 2024-04-29T14:26:39.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d848577:::benchmark_data_smithi194_121512_object9730:head# 2024-04-29T14:26:39.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d84893c:::benchmark_data_smithi194_121512_object92824:head# 2024-04-29T14:26:39.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8548e2:::benchmark_data_smithi194_121512_object1669:head# 2024-04-29T14:26:39.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d854bb8:::benchmark_data_smithi194_121512_object88626:head# 2024-04-29T14:26:39.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d858d9d:::benchmark_data_smithi194_121512_object950:head# 2024-04-29T14:26:39.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d85d701:::benchmark_data_smithi194_121512_object52886:head# 2024-04-29T14:26:39.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d85fc59:::benchmark_data_smithi194_121512_object35906:head# 2024-04-29T14:26:39.113 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d862b61:::benchmark_data_smithi194_121512_object5138:head# 2024-04-29T14:26:39.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8639d4:::benchmark_data_smithi194_121512_object27879:head# 2024-04-29T14:26:39.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8651db:::benchmark_data_smithi194_121512_object82300:head# 2024-04-29T14:26:39.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d86582a:::benchmark_data_smithi194_121512_object78072:head# 2024-04-29T14:26:39.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d86624a:::benchmark_data_smithi194_121512_object52898:head# 2024-04-29T14:26:39.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d867e19:::benchmark_data_smithi194_121512_object85665:head# 2024-04-29T14:26:39.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8698b7:::benchmark_data_smithi194_121512_object18665:head# 2024-04-29T14:26:39.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d87203c:::benchmark_data_smithi194_121512_object2330:head# 2024-04-29T14:26:39.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d87e6f9:::benchmark_data_smithi194_121512_object60592:head# 2024-04-29T14:26:39.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d87e8f4:::benchmark_data_smithi194_121512_object7197:head# 2024-04-29T14:26:39.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d88fdcf:::benchmark_data_smithi194_121512_object94838:head# 2024-04-29T14:26:39.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d89eb31:::benchmark_data_smithi194_121512_object5491:head# 2024-04-29T14:26:39.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8cd37e:::benchmark_data_smithi194_121512_object55763:head# 2024-04-29T14:26:39.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8d8c43:::benchmark_data_smithi194_121512_object72118:head# 2024-04-29T14:26:39.119 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:39.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8de6f8:::benchmark_data_smithi194_121512_object25030:head# 2024-04-29T14:26:39.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8e1f99:::benchmark_data_smithi194_121512_object109590:head# 2024-04-29T14:26:39.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8ebab8:::benchmark_data_smithi194_121512_object28722:head# 2024-04-29T14:26:39.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8f0a4d:::benchmark_data_smithi194_121512_object59564:head# 2024-04-29T14:26:39.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8f56a6:::benchmark_data_smithi194_121512_object4105:head# 2024-04-29T14:26:39.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d8f59b8:::benchmark_data_smithi194_121512_object85:head# 2024-04-29T14:26:39.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d909310:::benchmark_data_smithi194_121512_object12104:head# 2024-04-29T14:26:39.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9095d2:::benchmark_data_smithi194_121512_object83698:head# 2024-04-29T14:26:39.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d90c804:::benchmark_data_smithi194_121512_object35284:head# 2024-04-29T14:26:39.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d91ad2d:::benchmark_data_smithi194_121512_object23943:head# 2024-04-29T14:26:39.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d91cc27:::benchmark_data_smithi194_121512_object97990:head# 2024-04-29T14:26:39.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d920387:::benchmark_data_smithi194_121512_object22265:head# 2024-04-29T14:26:39.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d921910:::benchmark_data_smithi194_121512_object53147:head# 2024-04-29T14:26:39.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d924456:::benchmark_data_smithi194_121512_object12875:head# 2024-04-29T14:26:39.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d938860:::benchmark_data_smithi194_121512_object72784:head# 2024-04-29T14:26:39.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d93bffa:::benchmark_data_smithi194_121512_object4171:head# 2024-04-29T14:26:39.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d93f51f:::benchmark_data_smithi194_121512_object34459:head# 2024-04-29T14:26:39.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d942b57:::benchmark_data_smithi194_121512_object28350:head# 2024-04-29T14:26:39.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d94bdad:::benchmark_data_smithi194_121512_object71585:head# 2024-04-29T14:26:39.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d957f25:::benchmark_data_smithi194_121512_object98710:head# 2024-04-29T14:26:39.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d958903:::benchmark_data_smithi194_121512_object65166:head# 2024-04-29T14:26:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d964573:::benchmark_data_smithi194_121512_object97513:head# 2024-04-29T14:26:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d966299:::benchmark_data_smithi194_121512_object22278:head# 2024-04-29T14:26:39.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d96d9c0:::benchmark_data_smithi194_121512_object44669:head# 2024-04-29T14:26:39.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9707ed:::benchmark_data_smithi194_121512_object25478:head# 2024-04-29T14:26:39.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d975b84:::benchmark_data_smithi194_121512_object999:head# 2024-04-29T14:26:39.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9766b1:::benchmark_data_smithi194_121512_object127047:head# 2024-04-29T14:26:39.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d97e5bd:::benchmark_data_smithi194_121512_object99659:head# 2024-04-29T14:26:39.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d97f27a:::benchmark_data_smithi194_121512_object25238:head# 2024-04-29T14:26:39.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d97f68d:::benchmark_data_smithi194_121512_object23775:head# 2024-04-29T14:26:39.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d983ce8:::benchmark_data_smithi194_121512_object81342:head# 2024-04-29T14:26:39.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d98e08f:::benchmark_data_smithi194_121512_object48532:head# 2024-04-29T14:26:39.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d98ee2e:::benchmark_data_smithi194_121512_object101143:head# 2024-04-29T14:26:39.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9a5a3d:::benchmark_data_smithi194_121512_object120790:head# 2024-04-29T14:26:39.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9aacb0:::benchmark_data_smithi194_121512_object132738:head# 2024-04-29T14:26:39.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9b2855:::benchmark_data_smithi194_121512_object5730:head# 2024-04-29T14:26:39.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9b4bf2:::benchmark_data_smithi194_121512_object7590:head# 2024-04-29T14:26:39.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9b686f:::benchmark_data_smithi194_121512_object62284:head# 2024-04-29T14:26:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9b6dbd:::benchmark_data_smithi194_121512_object46361:head# 2024-04-29T14:26:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9b6fbf:::benchmark_data_smithi194_121512_object117010:head# 2024-04-29T14:26:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9c075b:::benchmark_data_smithi194_121512_object132384:head# 2024-04-29T14:26:39.141 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:39.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9c3587:::benchmark_data_smithi194_121512_object25061:head# 2024-04-29T14:26:39.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9c56b9:::benchmark_data_smithi194_121512_object79235:head# 2024-04-29T14:26:39.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9cbaa0:::benchmark_data_smithi194_121512_object80159:head# 2024-04-29T14:26:39.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9cecb8:::benchmark_data_smithi194_121512_object43071:head# 2024-04-29T14:26:39.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9de406:::benchmark_data_smithi194_121512_object88261:head# 2024-04-29T14:26:39.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9e49ac:::benchmark_data_smithi194_121512_object48180:head# 2024-04-29T14:26:39.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9e57ba:::benchmark_data_smithi194_121512_object7262:head# 2024-04-29T14:26:39.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6d9e6828:::benchmark_data_smithi194_121512_object78403:head# 2024-04-29T14:26:39.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da060d2:::benchmark_data_smithi194_121512_object119315:head# 2024-04-29T14:26:39.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da0c133:::benchmark_data_smithi194_121512_object69029:head# 2024-04-29T14:26:39.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da1392d:::benchmark_data_smithi194_121512_object111768:head# 2024-04-29T14:26:39.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da1661c:::benchmark_data_smithi194_121512_object44821:head# 2024-04-29T14:26:39.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da1c76a:::benchmark_data_smithi194_121512_object79649:head# 2024-04-29T14:26:39.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da1e4b1:::benchmark_data_smithi194_121512_object110108:head# 2024-04-29T14:26:39.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da22d35:::benchmark_data_smithi194_121512_object99376:head# 2024-04-29T14:26:39.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da3ab57:::benchmark_data_smithi194_121512_object28450:head# 2024-04-29T14:26:39.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da493c2:::benchmark_data_smithi194_121512_object51852:head# 2024-04-29T14:26:39.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da4cb68:::benchmark_data_smithi194_121512_object34679:head# 2024-04-29T14:26:39.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da4eab5:::benchmark_data_smithi194_121512_object81521:head# 2024-04-29T14:26:39.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da5f332:::benchmark_data_smithi194_121512_object44303:head# 2024-04-29T14:26:39.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da64cf2:::benchmark_data_smithi194_121512_object81757:head# 2024-04-29T14:26:39.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da6e501:::benchmark_data_smithi194_121512_object11576:head# 2024-04-29T14:26:39.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da7a51e:::benchmark_data_smithi194_121512_object107078:head# 2024-04-29T14:26:39.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da7c0ff:::benchmark_data_smithi194_121512_object23616:head# 2024-04-29T14:26:39.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da857c4:::benchmark_data_smithi194_121512_object127698:head# 2024-04-29T14:26:39.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da889d1:::benchmark_data_smithi194_121512_object33857:head# 2024-04-29T14:26:39.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6da91e70:::benchmark_data_smithi194_121512_object15579:head# 2024-04-29T14:26:39.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6daa83bb:::benchmark_data_smithi194_121512_object28835:head# 2024-04-29T14:26:39.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dab1525:::benchmark_data_smithi194_121512_object6794:head# 2024-04-29T14:26:39.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dab15e6:::benchmark_data_smithi194_121512_object72455:head# 2024-04-29T14:26:39.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dab1a4a:::benchmark_data_smithi194_121512_object41819:head# 2024-04-29T14:26:39.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dac6bf6:::benchmark_data_smithi194_121512_object78877:head# 2024-04-29T14:26:39.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dad137f:::benchmark_data_smithi194_121512_object90085:head# 2024-04-29T14:26:39.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dad418d:::benchmark_data_smithi194_121512_object131154:head# 2024-04-29T14:26:39.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dad8854:::benchmark_data_smithi194_121512_object34536:head# 2024-04-29T14:26:39.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dae2746:::benchmark_data_smithi194_121512_object54212:head# 2024-04-29T14:26:39.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dae34c3:::benchmark_data_smithi194_121512_object66809:head# 2024-04-29T14:26:39.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dae63ec:::benchmark_data_smithi194_121512_object105433:head# 2024-04-29T14:26:39.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6daea169:::benchmark_data_smithi194_121512_object79878:head# 2024-04-29T14:26:39.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6daee1ef:::benchmark_data_smithi194_121512_object24388:head# 2024-04-29T14:26:39.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db0d8c8:::benchmark_data_smithi194_121512_object12801:head# 2024-04-29T14:26:39.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db0fc02:::benchmark_data_smithi194_121512_object20139:head# 2024-04-29T14:26:39.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db1961a:::benchmark_data_smithi194_121512_object43805:head# 2024-04-29T14:26:39.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db1e24a:::benchmark_data_smithi194_121512_object83932:head# 2024-04-29T14:26:39.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db2120b:::benchmark_data_smithi194_121512_object92517:head# 2024-04-29T14:26:39.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db21a9f:::benchmark_data_smithi194_121512_object10867:head# 2024-04-29T14:26:39.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db228e1:::benchmark_data_smithi194_121512_object20016:head# 2024-04-29T14:26:39.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db2756d:::benchmark_data_smithi194_121512_object102914:head# 2024-04-29T14:26:39.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db2ac00:::benchmark_data_smithi194_121512_object102712:head# 2024-04-29T14:26:39.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db35daf:::benchmark_data_smithi194_121512_object112048:head# 2024-04-29T14:26:39.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db387e2:::benchmark_data_smithi194_121512_object1776:head# 2024-04-29T14:26:39.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db3a9b9:::benchmark_data_smithi194_121512_object71984:head# 2024-04-29T14:26:39.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db3dc1d:::benchmark_data_smithi194_121512_object88452:head# 2024-04-29T14:26:39.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db40536:::benchmark_data_smithi194_121512_object120324:head# 2024-04-29T14:26:39.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db49801:::benchmark_data_smithi194_121512_object26277:head# 2024-04-29T14:26:39.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db4e595:::benchmark_data_smithi194_121512_object125947:head# 2024-04-29T14:26:39.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db505f2:::benchmark_data_smithi194_121512_object24218:head# 2024-04-29T14:26:39.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db57925:::benchmark_data_smithi194_121512_object38173:head# 2024-04-29T14:26:39.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db58de5:::benchmark_data_smithi194_121512_object18364:head# 2024-04-29T14:26:39.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db5c01c:::benchmark_data_smithi194_121512_object55794:head# 2024-04-29T14:26:39.170 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:39.171 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:39.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db69958:::benchmark_data_smithi194_121512_object35361:head# 2024-04-29T14:26:39.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db6fd90:::benchmark_data_smithi194_121512_object15294:head# 2024-04-29T14:26:39.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db7c4e5:::benchmark_data_smithi194_121512_object728:head# 2024-04-29T14:26:39.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db7f3b5:::benchmark_data_smithi194_121512_object61153:head# 2024-04-29T14:26:39.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db8d813:::benchmark_data_smithi194_121512_object25309:head# 2024-04-29T14:26:39.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6db940a7:::benchmark_data_smithi194_121512_object58459:head# 2024-04-29T14:26:39.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dba2bd3:::benchmark_data_smithi194_121512_object126569:head# 2024-04-29T14:26:39.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbacb2e:::benchmark_data_smithi194_121512_object15600:head# 2024-04-29T14:26:39.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbb063a:::benchmark_data_smithi194_121512_object108590:head# 2024-04-29T14:26:39.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbb2d34:::benchmark_data_smithi194_121512_object11628:head# 2024-04-29T14:26:39.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbbbe64:::benchmark_data_smithi194_121512_object18458:head# 2024-04-29T14:26:39.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbc5cbc:::benchmark_data_smithi194_121512_object30730:head# 2024-04-29T14:26:39.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbcbd47:::benchmark_data_smithi194_121512_object47782:head# 2024-04-29T14:26:39.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbcdcc9:::benchmark_data_smithi194_121512_object97438:head# 2024-04-29T14:26:39.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbd2ec9:::benchmark_data_smithi194_121512_object11828:head# 2024-04-29T14:26:39.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbd2ed4:::benchmark_data_smithi194_121512_object19060:head# 2024-04-29T14:26:39.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbd7f30:::benchmark_data_smithi194_121512_object58348:head# 2024-04-29T14:26:39.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbe8f10:::benchmark_data_smithi194_121512_object68255:head# 2024-04-29T14:26:39.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbea6a0:::benchmark_data_smithi194_121512_object93458:head# 2024-04-29T14:26:39.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbea8e1:::benchmark_data_smithi194_121512_object62949:head# 2024-04-29T14:26:39.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbf0d31:::benchmark_data_smithi194_121512_object63963:head# 2024-04-29T14:26:39.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbfc81b:::benchmark_data_smithi194_121512_object114844:head# 2024-04-29T14:26:39.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dbfeb03:::benchmark_data_smithi194_121512_object54363:head# 2024-04-29T14:26:39.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc08296:::benchmark_data_smithi194_121512_object75504:head# 2024-04-29T14:26:39.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc139c5:::benchmark_data_smithi194_121512_object13357:head# 2024-04-29T14:26:39.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc18875:::benchmark_data_smithi194_121512_object49189:head# 2024-04-29T14:26:39.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc1ab86:::benchmark_data_smithi194_121512_object30906:head# 2024-04-29T14:26:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc2122d:::benchmark_data_smithi194_121512_object35492:head# 2024-04-29T14:26:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc23c57:::benchmark_data_smithi194_121512_object89987:head# 2024-04-29T14:26:39.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc2a02b:::benchmark_data_smithi194_121512_object80977:head# 2024-04-29T14:26:39.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc2c040:::benchmark_data_smithi194_121512_object39019:head# 2024-04-29T14:26:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc30c34:::benchmark_data_smithi194_121512_object35334:head# 2024-04-29T14:26:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc384c6:::benchmark_data_smithi194_121512_object46858:head# 2024-04-29T14:26:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc4ac78:::benchmark_data_smithi194_121512_object78261:head# 2024-04-29T14:26:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc4d01c:::benchmark_data_smithi194_121512_object129583:head# 2024-04-29T14:26:39.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc5dc2d:::benchmark_data_smithi194_121512_object27428:head# 2024-04-29T14:26:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc65b16:::benchmark_data_smithi194_121512_object90376:head# 2024-04-29T14:26:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc68518:::benchmark_data_smithi194_121512_object55887:head# 2024-04-29T14:26:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc6a46f:::benchmark_data_smithi194_121512_object129085:head# 2024-04-29T14:26:39.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc707e4:::benchmark_data_smithi194_121512_object87081:head# 2024-04-29T14:26:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc718af:::benchmark_data_smithi194_121512_object39830:head# 2024-04-29T14:26:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc7f76c:::benchmark_data_smithi194_121512_object54535:head# 2024-04-29T14:26:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dc87327:::benchmark_data_smithi194_121512_object42424:head# 2024-04-29T14:26:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dca02e8:::benchmark_data_smithi194_121512_object125831:head# 2024-04-29T14:26:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dca208e:::benchmark_data_smithi194_121512_object39318:head# 2024-04-29T14:26:39.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dca53fe:::benchmark_data_smithi194_121512_object107728:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dcb4f8e:::benchmark_data_smithi194_121512_object123081:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dcbb564:::benchmark_data_smithi194_121512_object123178:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dcbc72a:::benchmark_data_smithi194_121512_object119048:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dcc2248:::benchmark_data_smithi194_121512_object1729:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dcd8aea:::benchmark_data_smithi194_121512_object109928:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dced2f0:::benchmark_data_smithi194_121512_object84501:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dced967:::benchmark_data_smithi194_121512_object31557:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dcf047b:::benchmark_data_smithi194_121512_object27490:head# 2024-04-29T14:26:39.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd082a9:::benchmark_data_smithi194_121512_object17618:head# 2024-04-29T14:26:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd11ca1:::benchmark_data_smithi194_121512_object44690:head# 2024-04-29T14:26:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd17626:::benchmark_data_smithi194_121512_object44225:head# 2024-04-29T14:26:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd192d1:::benchmark_data_smithi194_121512_object127558:head# 2024-04-29T14:26:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd1ffa2:::benchmark_data_smithi194_121512_object72901:head# 2024-04-29T14:26:39.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd214c6:::benchmark_data_smithi194_121512_object80577:head# 2024-04-29T14:26:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd254f3:::benchmark_data_smithi194_121512_object1648:head# 2024-04-29T14:26:39.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd40915:::benchmark_data_smithi194_121512_object90233:head# 2024-04-29T14:26:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd41906:::benchmark_data_smithi194_121512_object116543:head# 2024-04-29T14:26:39.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd47905:::benchmark_data_smithi194_121512_object82376:head# 2024-04-29T14:26:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd532f2:::benchmark_data_smithi194_121512_object110606:head# 2024-04-29T14:26:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd56d28:::benchmark_data_smithi194_121512_object123957:head# 2024-04-29T14:26:39.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd60c06:::benchmark_data_smithi194_121512_object48118:head# 2024-04-29T14:26:39.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd6426c:::benchmark_data_smithi194_121512_object102351:head# 2024-04-29T14:26:39.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dd6b92f:::benchmark_data_smithi194_121512_object8067:head# 2024-04-29T14:26:39.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dda64f5:::benchmark_data_smithi194_121512_object114039:head# 2024-04-29T14:26:39.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dda9aa4:::benchmark_data_smithi194_121512_object59462:head# 2024-04-29T14:26:39.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dde00ec:::benchmark_data_smithi194_121512_object46491:head# 2024-04-29T14:26:39.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ddf52b0:::benchmark_data_smithi194_121512_object114742:head# 2024-04-29T14:26:39.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ddfc99c:::benchmark_data_smithi194_121512_object121898:head# 2024-04-29T14:26:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ddfd12b:::benchmark_data_smithi194_121512_object55370:head# 2024-04-29T14:26:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de04fd4:::benchmark_data_smithi194_121512_object55739:head# 2024-04-29T14:26:39.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de07f0f:::benchmark_data_smithi194_121512_object59418:head# 2024-04-29T14:26:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de1025e:::benchmark_data_smithi194_121512_object38369:head# 2024-04-29T14:26:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de111f9:::benchmark_data_smithi194_121512_object34721:head# 2024-04-29T14:26:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de23b54:::benchmark_data_smithi194_121512_object124016:head# 2024-04-29T14:26:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de2c0e8:::benchmark_data_smithi194_121512_object61638:head# 2024-04-29T14:26:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de333a7:::benchmark_data_smithi194_121512_object7657:head# 2024-04-29T14:26:39.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de3f466:::benchmark_data_smithi194_121512_object38598:head# 2024-04-29T14:26:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de455e4:::benchmark_data_smithi194_121512_object110706:head# 2024-04-29T14:26:39.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de4bb22:::benchmark_data_smithi194_121512_object126383:head# 2024-04-29T14:26:39.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de51e63:::benchmark_data_smithi194_121512_object86037:head# 2024-04-29T14:26:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de51fa1:::benchmark_data_smithi194_121512_object617:head# 2024-04-29T14:26:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de6510f:::benchmark_data_smithi194_121512_object80928:head# 2024-04-29T14:26:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de6638d:::benchmark_data_smithi194_121512_object36658:head# 2024-04-29T14:26:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de71039:::benchmark_data_smithi194_121512_object111919:head# 2024-04-29T14:26:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de72070:::benchmark_data_smithi194_121512_object40357:head# 2024-04-29T14:26:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de765b1:::benchmark_data_smithi194_121512_object93547:head# 2024-04-29T14:26:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de7b202:::benchmark_data_smithi194_121512_object114054:head# 2024-04-29T14:26:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de7e62a:::benchmark_data_smithi194_121512_object3065:head# 2024-04-29T14:26:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de865c9:::benchmark_data_smithi194_121512_object117355:head# 2024-04-29T14:26:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de92e49:::benchmark_data_smithi194_121512_object44751:head# 2024-04-29T14:26:39.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de95677:::benchmark_data_smithi194_121512_object80411:head# 2024-04-29T14:26:39.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de9e388:::benchmark_data_smithi194_121512_object96944:head# 2024-04-29T14:26:39.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6de9ef43:::benchmark_data_smithi194_121512_object44537:head# 2024-04-29T14:26:39.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dea44b2:::benchmark_data_smithi194_121512_object14676:head# 2024-04-29T14:26:39.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6deb13fd:::benchmark_data_smithi194_121512_object30000:head# 2024-04-29T14:26:39.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6debc1a4:::benchmark_data_smithi194_121512_object48925:head# 2024-04-29T14:26:39.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dec6ead:::benchmark_data_smithi194_121512_object126839:head# 2024-04-29T14:26:39.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6decd1c0:::benchmark_data_smithi194_121512_object47111:head# 2024-04-29T14:26:39.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6decf199:::benchmark_data_smithi194_121512_object67671:head# 2024-04-29T14:26:39.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ded21f3:::benchmark_data_smithi194_121512_object49706:head# 2024-04-29T14:26:39.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ded2c5d:::benchmark_data_smithi194_121512_object434:head# 2024-04-29T14:26:39.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ded87fa:::benchmark_data_smithi194_121512_object100624:head# 2024-04-29T14:26:39.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dee6093:::benchmark_data_smithi194_121512_object66303:head# 2024-04-29T14:26:39.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6defc521:::benchmark_data_smithi194_121512_object65708:head# 2024-04-29T14:26:39.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df0daaf:::benchmark_data_smithi194_121512_object90947:head# 2024-04-29T14:26:39.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df1108d:::benchmark_data_smithi194_121512_object34185:head# 2024-04-29T14:26:39.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df25b48:::benchmark_data_smithi194_121512_object94621:head# 2024-04-29T14:26:39.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df4c4b8:::benchmark_data_smithi194_121512_object123509:head# 2024-04-29T14:26:39.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df555d5:::benchmark_data_smithi194_121512_object95995:head# 2024-04-29T14:26:39.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df58b8c:::benchmark_data_smithi194_121512_object46936:head# 2024-04-29T14:26:39.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df63843:::benchmark_data_smithi194_121512_object43580:head# 2024-04-29T14:26:39.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df6e4e3:::benchmark_data_smithi194_121512_object62802:head# 2024-04-29T14:26:39.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df6f8fb:::benchmark_data_smithi194_121512_object115217:head# 2024-04-29T14:26:39.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df73cec:::benchmark_data_smithi194_121512_object64589:head# 2024-04-29T14:26:39.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df749fa:::benchmark_data_smithi194_121512_object6052:head# 2024-04-29T14:26:39.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df78ba1:::benchmark_data_smithi194_121512_object43141:head# 2024-04-29T14:26:39.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df858aa:::benchmark_data_smithi194_121512_object2017:head# 2024-04-29T14:26:39.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df87fb4:::benchmark_data_smithi194_121512_object111168:head# 2024-04-29T14:26:39.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df88bac:::benchmark_data_smithi194_121512_object48238:head# 2024-04-29T14:26:39.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6df89354:::benchmark_data_smithi194_121512_object93022:head# 2024-04-29T14:26:39.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfabda0:::benchmark_data_smithi194_121512_object80405:head# 2024-04-29T14:26:39.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfb6222:::benchmark_data_smithi194_121512_object70372:head# 2024-04-29T14:26:39.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfbcc48:::benchmark_data_smithi194_121512_object132446:head# 2024-04-29T14:26:39.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfce5c7:::benchmark_data_smithi194_121512_object115846:head# 2024-04-29T14:26:39.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfdb299:::benchmark_data_smithi194_121512_object88928:head# 2024-04-29T14:26:39.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfe283c:::benchmark_data_smithi194_121512_object54751:head# 2024-04-29T14:26:39.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfe7ca9:::benchmark_data_smithi194_121512_object59308:head# 2024-04-29T14:26:39.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfeb1bd:::benchmark_data_smithi194_121512_object66170:head# 2024-04-29T14:26:39.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dfecab7:::benchmark_data_smithi194_121512_object33397:head# 2024-04-29T14:26:39.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dff1756:::benchmark_data_smithi194_121512_object3197:head# 2024-04-29T14:26:39.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dff2a39:::benchmark_data_smithi194_121512_object103348:head# 2024-04-29T14:26:39.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6dffc25d:::benchmark_data_smithi194_121512_object3920:head# 2024-04-29T14:26:39.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e001073:::benchmark_data_smithi194_121512_object33558:head# 2024-04-29T14:26:39.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e002662:::benchmark_data_smithi194_121512_object8754:head# 2024-04-29T14:26:39.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e008f14:::benchmark_data_smithi194_121512_object20474:head# 2024-04-29T14:26:39.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0160e6:::benchmark_data_smithi194_121512_object86928:head# 2024-04-29T14:26:39.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e02364c:::benchmark_data_smithi194_121512_object49799:head# 2024-04-29T14:26:39.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e023778:::benchmark_data_smithi194_121512_object76034:head# 2024-04-29T14:26:39.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e024efa:::benchmark_data_smithi194_121512_object64499:head# 2024-04-29T14:26:39.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e03b7af:::benchmark_data_smithi194_121512_object79754:head# 2024-04-29T14:26:39.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e040de4:::benchmark_data_smithi194_121512_object19924:head# 2024-04-29T14:26:39.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e04315e:::benchmark_data_smithi194_121512_object31865:head# 2024-04-29T14:26:39.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e04b3ca:::benchmark_data_smithi194_121512_object123773:head# 2024-04-29T14:26:39.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e04b4e8:::benchmark_data_smithi194_121512_object58355:head# 2024-04-29T14:26:39.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e04f37b:::benchmark_data_smithi194_121512_object91773:head# 2024-04-29T14:26:39.256 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e053142:::benchmark_data_smithi194_121512_object5176:head# 2024-04-29T14:26:39.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0573a5:::benchmark_data_smithi194_121512_object11836:head# 2024-04-29T14:26:39.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e062dc0:::benchmark_data_smithi194_121512_object75256:head# 2024-04-29T14:26:39.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e06336e:::benchmark_data_smithi194_121512_object116504:head# 2024-04-29T14:26:39.257 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e066985:::benchmark_data_smithi194_121512_object107925:head# 2024-04-29T14:26:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e067efd:::benchmark_data_smithi194_121512_object113522:head# 2024-04-29T14:26:39.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:39 smithi121 ceph-mon[44533]: pgmap v1251: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 38584/245910 objects degraded (15.690%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:26:39.260 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1123644810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:39.261 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:39 smithi194 ceph-mon[44089]: pgmap v1251: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 38584/245910 objects degraded (15.690%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:26:39.261 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1123644810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:39.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e06ad82:::benchmark_data_smithi194_121512_object23689:head# 2024-04-29T14:26:39.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e07b421:::benchmark_data_smithi194_121512_object59266:head# 2024-04-29T14:26:39.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e07df9b:::benchmark_data_smithi194_121512_object93862:head# 2024-04-29T14:26:39.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e08c334:::benchmark_data_smithi194_121512_object29085:head# 2024-04-29T14:26:39.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0a5f73:::benchmark_data_smithi194_121512_object10509:head# 2024-04-29T14:26:39.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0a8631:::benchmark_data_smithi194_121512_object43574:head# 2024-04-29T14:26:39.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0a9ee1:::benchmark_data_smithi194_121512_object23762:head# 2024-04-29T14:26:39.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0b0c6a:::benchmark_data_smithi194_121512_object87236:head# 2024-04-29T14:26:39.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0b5500:::benchmark_data_smithi194_121512_object62872:head# 2024-04-29T14:26:39.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0b9858:::benchmark_data_smithi194_121512_object72354:head# 2024-04-29T14:26:39.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0cbd37:::benchmark_data_smithi194_121512_object94612:head# 2024-04-29T14:26:39.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0d77ad:::benchmark_data_smithi194_121512_object112847:head# 2024-04-29T14:26:39.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0e0bde:::benchmark_data_smithi194_121512_object4483:head# 2024-04-29T14:26:39.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0e8f21:::benchmark_data_smithi194_121512_object107352:head# 2024-04-29T14:26:39.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0eee64:::benchmark_data_smithi194_121512_object41629:head# 2024-04-29T14:26:39.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e0f49ac:::benchmark_data_smithi194_121512_object133131:head# 2024-04-29T14:26:39.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e105d78:::benchmark_data_smithi194_121512_object101070:head# 2024-04-29T14:26:39.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e11aa5b:::benchmark_data_smithi194_121512_object98226:head# 2024-04-29T14:26:39.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e129b43:::benchmark_data_smithi194_121512_object3527:head# 2024-04-29T14:26:39.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e147f84:::benchmark_data_smithi194_121512_object72928:head# 2024-04-29T14:26:39.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e15231b:::benchmark_data_smithi194_121512_object93989:head# 2024-04-29T14:26:39.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e157ac2:::benchmark_data_smithi194_121512_object3312:head# 2024-04-29T14:26:39.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e15a51f:::benchmark_data_smithi194_121512_object128100:head# 2024-04-29T14:26:39.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e15caae:::benchmark_data_smithi194_121512_object35677:head# 2024-04-29T14:26:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e161156:::benchmark_data_smithi194_121512_object61999:head# 2024-04-29T14:26:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e16a9e5:::benchmark_data_smithi194_121512_object98831:head# 2024-04-29T14:26:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e17030d:::benchmark_data_smithi194_121512_object110746:head# 2024-04-29T14:26:39.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e17c22e:::benchmark_data_smithi194_121512_object1612:head# 2024-04-29T14:26:39.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e17ef62:::benchmark_data_smithi194_121512_object18558:head# 2024-04-29T14:26:39.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e17f83b:::benchmark_data_smithi194_121512_object48655:head# 2024-04-29T14:26:39.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e180c0c:::benchmark_data_smithi194_121512_object47011:head# 2024-04-29T14:26:39.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e184b1f:::benchmark_data_smithi194_121512_object38462:head# 2024-04-29T14:26:39.275 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e19124c:::benchmark_data_smithi194_121512_object99480:head# 2024-04-29T14:26:39.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e192964:::benchmark_data_smithi194_121512_object66282:head# 2024-04-29T14:26:39.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e198688:::benchmark_data_smithi194_121512_object20665:head# 2024-04-29T14:26:39.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e19b67c:::benchmark_data_smithi194_121512_object52742:head# 2024-04-29T14:26:39.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e1ad206:::benchmark_data_smithi194_121512_object61973:head# 2024-04-29T14:26:39.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e1ae649:::benchmark_data_smithi194_121512_object25957:head# 2024-04-29T14:26:39.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e1b5727:::benchmark_data_smithi194_121512_object130037:head# 2024-04-29T14:26:39.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e1c3b70:::benchmark_data_smithi194_121512_object76208:head# 2024-04-29T14:26:39.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e1eac43:::benchmark_data_smithi194_121512_object8965:head# 2024-04-29T14:26:39.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e1f211a:::benchmark_data_smithi194_121512_object100063:head# 2024-04-29T14:26:39.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e1f82ea:::benchmark_data_smithi194_121512_object28961:head# 2024-04-29T14:26:39.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e213d7f:::benchmark_data_smithi194_121512_object17477:head# 2024-04-29T14:26:39.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e219ad1:::benchmark_data_smithi194_121512_object107526:head# 2024-04-29T14:26:39.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e21d284:::benchmark_data_smithi194_121512_object54195:head# 2024-04-29T14:26:39.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e21da19:::benchmark_data_smithi194_121512_object842:head# 2024-04-29T14:26:39.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2214d2:::benchmark_data_smithi194_121512_object75997:head# 2024-04-29T14:26:39.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e22a961:::benchmark_data_smithi194_121512_object131109:head# 2024-04-29T14:26:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2314b6:::benchmark_data_smithi194_121512_object49961:head# 2024-04-29T14:26:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2393df:::benchmark_data_smithi194_121512_object57726:head# 2024-04-29T14:26:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e24b62b:::benchmark_data_smithi194_121512_object42996:head# 2024-04-29T14:26:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e24f5cd:::benchmark_data_smithi194_121512_object763:head# 2024-04-29T14:26:39.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e253713:::benchmark_data_smithi194_121512_object51960:head# 2024-04-29T14:26:39.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e267265:::benchmark_data_smithi194_121512_object72861:head# 2024-04-29T14:26:39.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e271bd0:::benchmark_data_smithi194_121512_object33441:head# 2024-04-29T14:26:39.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2730a1:::benchmark_data_smithi194_121512_object32160:head# 2024-04-29T14:26:39.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2743be:::benchmark_data_smithi194_121512_object113630:head# 2024-04-29T14:26:39.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e275957:::benchmark_data_smithi194_121512_object101831:head# 2024-04-29T14:26:39.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e28da61:::benchmark_data_smithi194_121512_object27043:head# 2024-04-29T14:26:39.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e28e7a9:::benchmark_data_smithi194_121512_object96485:head# 2024-04-29T14:26:39.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e29b72c:::benchmark_data_smithi194_121512_object30451:head# 2024-04-29T14:26:39.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e29b80d:::benchmark_data_smithi194_121512_object37584:head# 2024-04-29T14:26:39.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e29c05e:::benchmark_data_smithi194_121512_object75621:head# 2024-04-29T14:26:39.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e29e619:::benchmark_data_smithi194_121512_object54924:head# 2024-04-29T14:26:39.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2a59b5:::benchmark_data_smithi194_121512_object113787:head# 2024-04-29T14:26:39.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2a9cf2:::benchmark_data_smithi194_121512_object54111:head# 2024-04-29T14:26:39.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2ae9c8:::benchmark_data_smithi194_121512_object126693:head# 2024-04-29T14:26:39.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2b6240:::benchmark_data_smithi194_121512_object79879:head# 2024-04-29T14:26:39.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2b97ad:::benchmark_data_smithi194_121512_object121547:head# 2024-04-29T14:26:39.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2c5f2b:::benchmark_data_smithi194_121512_object122636:head# 2024-04-29T14:26:39.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2dccde:::benchmark_data_smithi194_121512_object84731:head# 2024-04-29T14:26:39.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e2e1658:::benchmark_data_smithi194_121512_object116855:head# 2024-04-29T14:26:39.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e301c46:::benchmark_data_smithi194_121512_object27352:head# 2024-04-29T14:26:39.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e304063:::benchmark_data_smithi194_121512_object117397:head# 2024-04-29T14:26:39.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e307c05:::benchmark_data_smithi194_121512_object22147:head# 2024-04-29T14:26:39.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e30ba09:::benchmark_data_smithi194_121512_object22206:head# 2024-04-29T14:26:39.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e312b15:::benchmark_data_smithi194_121512_object59061:head# 2024-04-29T14:26:39.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e317415:::benchmark_data_smithi194_121512_object98129:head# 2024-04-29T14:26:39.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e31b9a3:::benchmark_data_smithi194_121512_object74874:head# 2024-04-29T14:26:39.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e328906:::benchmark_data_smithi194_121512_object33785:head# 2024-04-29T14:26:39.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e32a9a6:::benchmark_data_smithi194_121512_object5476:head# 2024-04-29T14:26:39.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e32ab00:::benchmark_data_smithi194_121512_object52047:head# 2024-04-29T14:26:39.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3396a0:::benchmark_data_smithi194_121512_object11639:head# 2024-04-29T14:26:39.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e33ef5e:::benchmark_data_smithi194_121512_object21970:head# 2024-04-29T14:26:39.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e343f62:::benchmark_data_smithi194_121512_object3656:head# 2024-04-29T14:26:39.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3476d3:::benchmark_data_smithi194_121512_object91876:head# 2024-04-29T14:26:39.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e354574:::benchmark_data_smithi194_121512_object8814:head# 2024-04-29T14:26:39.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e38cc49:::benchmark_data_smithi194_121512_object21358:head# 2024-04-29T14:26:39.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e39a324:::benchmark_data_smithi194_121512_object9145:head# 2024-04-29T14:26:39.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e39be9f:::benchmark_data_smithi194_121512_object9540:head# 2024-04-29T14:26:39.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e39fd77:::benchmark_data_smithi194_121512_object53898:head# 2024-04-29T14:26:39.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3a5a7d:::benchmark_data_smithi194_121512_object32755:head# 2024-04-29T14:26:39.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3ab3ea:::benchmark_data_smithi194_121512_object23487:head# 2024-04-29T14:26:39.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3ac765:::benchmark_data_smithi194_121512_object65110:head# 2024-04-29T14:26:39.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3c24ba:::benchmark_data_smithi194_121512_object55019:head# 2024-04-29T14:26:39.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3c59c7:::benchmark_data_smithi194_121512_object62330:head# 2024-04-29T14:26:39.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3e181a:::benchmark_data_smithi194_121512_object13543:head# 2024-04-29T14:26:39.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3e3b61:::benchmark_data_smithi194_121512_object78201:head# 2024-04-29T14:26:39.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3e98c9:::benchmark_data_smithi194_121512_object11602:head# 2024-04-29T14:26:39.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3ee857:::benchmark_data_smithi194_121512_object38298:head# 2024-04-29T14:26:39.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3f3342:::benchmark_data_smithi194_121512_object2833:head# 2024-04-29T14:26:39.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3f685f:::benchmark_data_smithi194_121512_object66326:head# 2024-04-29T14:26:39.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3f6ce7:::benchmark_data_smithi194_121512_object27845:head# 2024-04-29T14:26:39.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3faa4c:::benchmark_data_smithi194_121512_object133374:head# 2024-04-29T14:26:39.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e3fbd8b:::benchmark_data_smithi194_121512_object85808:head# 2024-04-29T14:26:39.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e41ef4a:::benchmark_data_smithi194_121512_object96537:head# 2024-04-29T14:26:39.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e41f5bf:::benchmark_data_smithi194_121512_object78989:head# 2024-04-29T14:26:39.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e428cee:::benchmark_data_smithi194_121512_object65533:head# 2024-04-29T14:26:39.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e42900d:::benchmark_data_smithi194_121512_object32810:head# 2024-04-29T14:26:39.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e42ca45:::benchmark_data_smithi194_121512_object65025:head# 2024-04-29T14:26:39.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e42d14b:::benchmark_data_smithi194_121512_object70041:head# 2024-04-29T14:26:39.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e43be69:::benchmark_data_smithi194_121512_object88163:head# 2024-04-29T14:26:39.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e43f9bc:::benchmark_data_smithi194_121512_object109753:head# 2024-04-29T14:26:39.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e442406:::benchmark_data_smithi194_121512_object59521:head# 2024-04-29T14:26:39.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e449e7d:::benchmark_data_smithi194_121512_object111854:head# 2024-04-29T14:26:39.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e45a1f8:::benchmark_data_smithi194_121512_object9903:head# 2024-04-29T14:26:39.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e46070d:::benchmark_data_smithi194_121512_object11773:head# 2024-04-29T14:26:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e472d78:::benchmark_data_smithi194_121512_object13057:head# 2024-04-29T14:26:39.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e478d26:::benchmark_data_smithi194_121512_object89387:head# 2024-04-29T14:26:39.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e479fd8:::benchmark_data_smithi194_121512_object70671:head# 2024-04-29T14:26:39.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e47ad9a:::benchmark_data_smithi194_121512_object78315:head# 2024-04-29T14:26:39.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e47d6b8:::benchmark_data_smithi194_121512_object68228:head# 2024-04-29T14:26:39.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e480800:::benchmark_data_smithi194_121512_object7796:head# 2024-04-29T14:26:39.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e481e22:::benchmark_data_smithi194_121512_object26168:head# 2024-04-29T14:26:39.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4848a5:::benchmark_data_smithi194_121512_object96098:head# 2024-04-29T14:26:39.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e48614d:::benchmark_data_smithi194_121512_object57689:head# 2024-04-29T14:26:39.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e48633d:::benchmark_data_smithi194_121512_object76539:head# 2024-04-29T14:26:39.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e48df10:::benchmark_data_smithi194_121512_object130551:head# 2024-04-29T14:26:39.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4a1b31:::benchmark_data_smithi194_121512_object9437:head# 2024-04-29T14:26:39.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4a40d4:::benchmark_data_smithi194_121512_object121737:head# 2024-04-29T14:26:39.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4a5206:::benchmark_data_smithi194_121512_object47263:head# 2024-04-29T14:26:39.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4a5459:::benchmark_data_smithi194_121512_object28772:head# 2024-04-29T14:26:39.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4aea0e:::benchmark_data_smithi194_121512_object90211:head# 2024-04-29T14:26:39.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4b6ce5:::benchmark_data_smithi194_121512_object132222:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4bfe0e:::benchmark_data_smithi194_121512_object84766:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4d5f0f:::benchmark_data_smithi194_121512_object60950:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4d900d:::benchmark_data_smithi194_121512_object130464:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4dafef:::benchmark_data_smithi194_121512_object59077:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4e0c2d:::benchmark_data_smithi194_121512_object49169:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4e2c0d:::benchmark_data_smithi194_121512_object115082:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4e3b0b:::benchmark_data_smithi194_121512_object11518:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4e6236:::benchmark_data_smithi194_121512_object13119:head# 2024-04-29T14:26:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e4ef674:::benchmark_data_smithi194_121512_object30698:head# 2024-04-29T14:26:39.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e502505:::benchmark_data_smithi194_121512_object59882:head# 2024-04-29T14:26:39.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e50a5b5:::benchmark_data_smithi194_121512_object121496:head# 2024-04-29T14:26:39.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e51afa2:::benchmark_data_smithi194_121512_object72338:head# 2024-04-29T14:26:39.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e51c92c:::benchmark_data_smithi194_121512_object118996:head# 2024-04-29T14:26:39.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e52b0eb:::benchmark_data_smithi194_121512_object122456:head# 2024-04-29T14:26:39.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e52ceaf:::benchmark_data_smithi194_121512_object112872:head# 2024-04-29T14:26:39.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e532434:::benchmark_data_smithi194_121512_object94883:head# 2024-04-29T14:26:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e533657:::benchmark_data_smithi194_121512_object110421:head# 2024-04-29T14:26:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e53f3c2:::benchmark_data_smithi194_121512_object4444:head# 2024-04-29T14:26:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e54b43e:::benchmark_data_smithi194_121512_object7537:head# 2024-04-29T14:26:39.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e54cef3:::benchmark_data_smithi194_121512_object108721:head# 2024-04-29T14:26:39.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e551fb5:::benchmark_data_smithi194_121512_object110774:head# 2024-04-29T14:26:39.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e55ed71:::benchmark_data_smithi194_121512_object62898:head# 2024-04-29T14:26:39.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e566932:::benchmark_data_smithi194_121512_object48176:head# 2024-04-29T14:26:39.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e57cc13:::benchmark_data_smithi194_121512_object13395:head# 2024-04-29T14:26:39.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e590284:::benchmark_data_smithi194_121512_object84456:head# 2024-04-29T14:26:39.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5943a6:::benchmark_data_smithi194_121512_object20966:head# 2024-04-29T14:26:39.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e59449a:::benchmark_data_smithi194_121512_object119660:head# 2024-04-29T14:26:39.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5baf76:::benchmark_data_smithi194_121512_object120572:head# 2024-04-29T14:26:39.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5bf815:::benchmark_data_smithi194_121512_object77240:head# 2024-04-29T14:26:39.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5c15fd:::benchmark_data_smithi194_121512_object42233:head# 2024-04-29T14:26:39.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5c34e2:::benchmark_data_smithi194_121512_object74623:head# 2024-04-29T14:26:39.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5cdb7f:::benchmark_data_smithi194_121512_object117817:head# 2024-04-29T14:26:39.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5d59c0:::benchmark_data_smithi194_121512_object89383:head# 2024-04-29T14:26:39.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5d6a2c:::benchmark_data_smithi194_121512_object103957:head# 2024-04-29T14:26:39.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5dec40:::benchmark_data_smithi194_121512_object105752:head# 2024-04-29T14:26:39.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5e26c6:::benchmark_data_smithi194_121512_object48054:head# 2024-04-29T14:26:39.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5f08bd:::benchmark_data_smithi194_121512_object65859:head# 2024-04-29T14:26:39.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e5fbde7:::benchmark_data_smithi194_121512_object42497:head# 2024-04-29T14:26:39.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e612d2e:::benchmark_data_smithi194_121512_object95306:head# 2024-04-29T14:26:39.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e614aac:::benchmark_data_smithi194_121512_object6165:head# 2024-04-29T14:26:39.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6200f2:::benchmark_data_smithi194_121512_object85078:head# 2024-04-29T14:26:39.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e62333a:::benchmark_data_smithi194_121512_object39890:head# 2024-04-29T14:26:39.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e624829:::benchmark_data_smithi194_121512_object119341:head# 2024-04-29T14:26:39.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e62ca00:::benchmark_data_smithi194_121512_object16330:head# 2024-04-29T14:26:39.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6403b1:::benchmark_data_smithi194_121512_object116437:head# 2024-04-29T14:26:39.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e64c65e:::benchmark_data_smithi194_121512_object59411:head# 2024-04-29T14:26:39.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6553d1:::benchmark_data_smithi194_121512_object120525:head# 2024-04-29T14:26:39.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6643b3:::benchmark_data_smithi194_121512_object23950:head# 2024-04-29T14:26:39.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e670d2c:::benchmark_data_smithi194_121512_object95358:head# 2024-04-29T14:26:39.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e67b202:::benchmark_data_smithi194_121512_object23990:head# 2024-04-29T14:26:39.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e68c3cb:::benchmark_data_smithi194_121512_object107820:head# 2024-04-29T14:26:39.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6981ea:::benchmark_data_smithi194_121512_object103896:head# 2024-04-29T14:26:39.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e69b7d8:::benchmark_data_smithi194_121512_object113248:head# 2024-04-29T14:26:39.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6a3fd1:::benchmark_data_smithi194_121512_object125562:head# 2024-04-29T14:26:39.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6a830e:::benchmark_data_smithi194_121512_object36855:head# 2024-04-29T14:26:39.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6a95f1:::benchmark_data_smithi194_121512_object28740:head# 2024-04-29T14:26:39.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6ad626:::benchmark_data_smithi194_121512_object47847:head# 2024-04-29T14:26:39.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6b3fdc:::benchmark_data_smithi194_121512_object109297:head# 2024-04-29T14:26:39.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6b651d:::benchmark_data_smithi194_121512_object123064:head# 2024-04-29T14:26:39.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6b8094:::benchmark_data_smithi194_121512_object74304:head# 2024-04-29T14:26:39.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6ba479:::benchmark_data_smithi194_121512_object63125:head# 2024-04-29T14:26:39.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6ce351:::benchmark_data_smithi194_121512_object123506:head# 2024-04-29T14:26:39.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6d4c90:::benchmark_data_smithi194_121512_object73252:head# 2024-04-29T14:26:39.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6d6580:::benchmark_data_smithi194_121512_object20590:head# 2024-04-29T14:26:39.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6e2702:::benchmark_data_smithi194_121512_object95162:head# 2024-04-29T14:26:39.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6e3e92:::benchmark_data_smithi194_121512_object67744:head# 2024-04-29T14:26:39.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6e528b:::benchmark_data_smithi194_121512_object67145:head# 2024-04-29T14:26:39.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6e7153:::benchmark_data_smithi194_121512_object104286:head# 2024-04-29T14:26:39.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6e8cb2:::benchmark_data_smithi194_121512_object31893:head# 2024-04-29T14:26:39.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6eedb1:::benchmark_data_smithi194_121512_object127128:head# 2024-04-29T14:26:39.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6eef22:::benchmark_data_smithi194_121512_object66681:head# 2024-04-29T14:26:39.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6f6c07:::benchmark_data_smithi194_121512_object25970:head# 2024-04-29T14:26:39.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6faaf3:::benchmark_data_smithi194_121512_object40176:head# 2024-04-29T14:26:39.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e6fc06a:::benchmark_data_smithi194_121512_object50121:head# 2024-04-29T14:26:39.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e70369e:::benchmark_data_smithi194_121512_object81507:head# 2024-04-29T14:26:39.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e703e03:::benchmark_data_smithi194_121512_object20736:head# 2024-04-29T14:26:39.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e709ebe:::benchmark_data_smithi194_121512_object93251:head# 2024-04-29T14:26:39.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e710091:::benchmark_data_smithi194_121512_object86116:head# 2024-04-29T14:26:39.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e713421:::benchmark_data_smithi194_121512_object34466:head# 2024-04-29T14:26:39.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e71ed70:::benchmark_data_smithi194_121512_object120458:head# 2024-04-29T14:26:39.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e72059a:::benchmark_data_smithi194_121512_object1354:head# 2024-04-29T14:26:39.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e72913c:::benchmark_data_smithi194_121512_object98523:head# 2024-04-29T14:26:39.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7322a2:::benchmark_data_smithi194_121512_object120487:head# 2024-04-29T14:26:39.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e747c99:::benchmark_data_smithi194_121512_object51703:head# 2024-04-29T14:26:39.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e756d94:::benchmark_data_smithi194_121512_object105167:head# 2024-04-29T14:26:39.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7608a4:::benchmark_data_smithi194_121512_object130976:head# 2024-04-29T14:26:39.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7694b5:::benchmark_data_smithi194_121512_object114041:head# 2024-04-29T14:26:39.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e76a7e8:::benchmark_data_smithi194_121512_object71081:head# 2024-04-29T14:26:39.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e76af8e:::benchmark_data_smithi194_121512_object47475:head# 2024-04-29T14:26:39.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7712f9:::benchmark_data_smithi194_121512_object102739:head# 2024-04-29T14:26:39.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7783fe:::benchmark_data_smithi194_121512_object78316:head# 2024-04-29T14:26:39.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e77d0e4:::benchmark_data_smithi194_121512_object65187:head# 2024-04-29T14:26:39.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e781999:::benchmark_data_smithi194_121512_object26347:head# 2024-04-29T14:26:39.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e789c73:::benchmark_data_smithi194_121512_object93676:head# 2024-04-29T14:26:39.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7969b5:::benchmark_data_smithi194_121512_object1773:head# 2024-04-29T14:26:39.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7982c3:::benchmark_data_smithi194_121512_object35128:head# 2024-04-29T14:26:39.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e79bd04:::benchmark_data_smithi194_121512_object130422:head# 2024-04-29T14:26:39.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7a2d01:::benchmark_data_smithi194_121512_object124875:head# 2024-04-29T14:26:39.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7bd955:::benchmark_data_smithi194_121512_object117529:head# 2024-04-29T14:26:39.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7c076e:::benchmark_data_smithi194_121512_object123621:head# 2024-04-29T14:26:39.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7c8176:::benchmark_data_smithi194_121512_object64402:head# 2024-04-29T14:26:39.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7c9511:::benchmark_data_smithi194_121512_object32663:head# 2024-04-29T14:26:39.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7cf29b:::benchmark_data_smithi194_121512_object123241:head# 2024-04-29T14:26:39.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7d2d36:::benchmark_data_smithi194_121512_object66169:head# 2024-04-29T14:26:39.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7d47ea:::benchmark_data_smithi194_121512_object3238:head# 2024-04-29T14:26:39.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7d4d1f:::benchmark_data_smithi194_121512_object95186:head# 2024-04-29T14:26:39.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7d870a:::benchmark_data_smithi194_121512_object60120:head# 2024-04-29T14:26:39.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7de5b9:::benchmark_data_smithi194_121512_object49722:head# 2024-04-29T14:26:39.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7e6ddb:::benchmark_data_smithi194_121512_object49144:head# 2024-04-29T14:26:39.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7ea65c:::benchmark_data_smithi194_121512_object68347:head# 2024-04-29T14:26:39.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7ecee3:::benchmark_data_smithi194_121512_object31869:head# 2024-04-29T14:26:39.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7ee4fe:::benchmark_data_smithi194_121512_object95133:head# 2024-04-29T14:26:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7f1c99:::benchmark_data_smithi194_121512_object51803:head# 2024-04-29T14:26:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e7fc07d:::benchmark_data_smithi194_121512_object47287:head# 2024-04-29T14:26:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e806a5d:::benchmark_data_smithi194_121512_object106540:head# 2024-04-29T14:26:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e80fa70:::benchmark_data_smithi194_121512_object120533:head# 2024-04-29T14:26:39.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e810d7e:::benchmark_data_smithi194_121512_object96254:head# 2024-04-29T14:26:39.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e81b89d:::benchmark_data_smithi194_121512_object101871:head# 2024-04-29T14:26:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e81e4cb:::benchmark_data_smithi194_121512_object32009:head# 2024-04-29T14:26:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8381b0:::benchmark_data_smithi194_121512_object111524:head# 2024-04-29T14:26:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e83b59a:::benchmark_data_smithi194_121512_object106645:head# 2024-04-29T14:26:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8482dc:::benchmark_data_smithi194_121512_object76860:head# 2024-04-29T14:26:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e84b774:::benchmark_data_smithi194_121512_object61016:head# 2024-04-29T14:26:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e850d7b:::benchmark_data_smithi194_121512_object73664:head# 2024-04-29T14:26:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e85a518:::benchmark_data_smithi194_121512_object62550:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e85c66a:::benchmark_data_smithi194_121512_object106419:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e85d444:::benchmark_data_smithi194_121512_object42557:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8611d6:::benchmark_data_smithi194_121512_object124179:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e868d81:::benchmark_data_smithi194_121512_object70821:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e86d544:::benchmark_data_smithi194_121512_object45736:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e87b94d:::benchmark_data_smithi194_121512_object92165:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e881fd1:::benchmark_data_smithi194_121512_object72501:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e88b4ba:::benchmark_data_smithi194_121512_object9995:head# 2024-04-29T14:26:39.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e88e24f:::benchmark_data_smithi194_121512_object67976:head# 2024-04-29T14:26:39.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e892270:::benchmark_data_smithi194_121512_object81512:head# 2024-04-29T14:26:39.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e893184:::benchmark_data_smithi194_121512_object100106:head# 2024-04-29T14:26:39.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e893f74:::benchmark_data_smithi194_121512_object54523:head# 2024-04-29T14:26:39.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8a8e71:::benchmark_data_smithi194_121512_object115541:head# 2024-04-29T14:26:39.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8b1f0e:::benchmark_data_smithi194_121512_object31195:head# 2024-04-29T14:26:39.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8b5f39:::benchmark_data_smithi194_121512_object16974:head# 2024-04-29T14:26:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8c37fe:::benchmark_data_smithi194_121512_object18878:head# 2024-04-29T14:26:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8d3e76:::benchmark_data_smithi194_121512_object49494:head# 2024-04-29T14:26:39.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8d6460:::benchmark_data_smithi194_121512_object1704:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8d9122:::benchmark_data_smithi194_121512_object30687:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8daccb:::benchmark_data_smithi194_121512_object33670:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8dd606:::benchmark_data_smithi194_121512_object115458:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8e69a9:::benchmark_data_smithi194_121512_object89989:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8ed271:::benchmark_data_smithi194_121512_object79741:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8f3579:::benchmark_data_smithi194_121512_object70797:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8f4e8c:::benchmark_data_smithi194_121512_object33745:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e8f8b1f:::benchmark_data_smithi194_121512_object94616:head# 2024-04-29T14:26:39.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e905a95:::benchmark_data_smithi194_121512_object107903:head# 2024-04-29T14:26:39.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e906110:::benchmark_data_smithi194_121512_object30983:head# 2024-04-29T14:26:39.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e91584d:::benchmark_data_smithi194_121512_object70066:head# 2024-04-29T14:26:39.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e91c5cb:::benchmark_data_smithi194_121512_object108419:head# 2024-04-29T14:26:39.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e91ddab:::benchmark_data_smithi194_121512_object42221:head# 2024-04-29T14:26:39.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e92c4e9:::benchmark_data_smithi194_121512_object74072:head# 2024-04-29T14:26:39.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e92fb4c:::benchmark_data_smithi194_121512_object87367:head# 2024-04-29T14:26:39.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e930e80:::benchmark_data_smithi194_121512_object45464:head# 2024-04-29T14:26:39.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9331eb:::benchmark_data_smithi194_121512_object33151:head# 2024-04-29T14:26:39.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e937050:::benchmark_data_smithi194_121512_object97495:head# 2024-04-29T14:26:39.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e93fac4:::benchmark_data_smithi194_121512_object123714:head# 2024-04-29T14:26:39.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9427a5:::benchmark_data_smithi194_121512_object4601:head# 2024-04-29T14:26:39.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e943836:::benchmark_data_smithi194_121512_object64329:head# 2024-04-29T14:26:39.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e94b579:::benchmark_data_smithi194_121512_object70352:head# 2024-04-29T14:26:39.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e957e61:::benchmark_data_smithi194_121512_object31747:head# 2024-04-29T14:26:39.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e957f6e:::benchmark_data_smithi194_121512_object40133:head# 2024-04-29T14:26:39.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e95cc50:::benchmark_data_smithi194_121512_object117028:head# 2024-04-29T14:26:39.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e964d33:::benchmark_data_smithi194_121512_object35295:head# 2024-04-29T14:26:39.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e969f7a:::benchmark_data_smithi194_121512_object57874:head# 2024-04-29T14:26:39.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e980c06:::benchmark_data_smithi194_121512_object119834:head# 2024-04-29T14:26:39.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e98eca4:::benchmark_data_smithi194_121512_object37883:head# 2024-04-29T14:26:39.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9a9540:::benchmark_data_smithi194_121512_object87402:head# 2024-04-29T14:26:39.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9aa733:::benchmark_data_smithi194_121512_object48408:head# 2024-04-29T14:26:39.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9c2064:::benchmark_data_smithi194_121512_object132687:head# 2024-04-29T14:26:39.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9c59a7:::benchmark_data_smithi194_121512_object2176:head# 2024-04-29T14:26:39.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9d1063:::benchmark_data_smithi194_121512_object65186:head# 2024-04-29T14:26:39.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9de2bd:::benchmark_data_smithi194_121512_object19705:head# 2024-04-29T14:26:39.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9e7948:::benchmark_data_smithi194_121512_object27791:head# 2024-04-29T14:26:39.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9ea94f:::benchmark_data_smithi194_121512_object105004:head# 2024-04-29T14:26:39.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9faa8e:::benchmark_data_smithi194_121512_object11954:head# 2024-04-29T14:26:39.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6e9fca3e:::benchmark_data_smithi194_121512_object59825:head# 2024-04-29T14:26:39.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea00a4d:::benchmark_data_smithi194_121512_object76458:head# 2024-04-29T14:26:39.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea03e7b:::benchmark_data_smithi194_121512_object55877:head# 2024-04-29T14:26:39.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea0988e:::benchmark_data_smithi194_121512_object43170:head# 2024-04-29T14:26:39.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea0dc45:::benchmark_data_smithi194_121512_object76637:head# 2024-04-29T14:26:39.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea105bc:::benchmark_data_smithi194_121512_object51361:head# 2024-04-29T14:26:39.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea12d3b:::benchmark_data_smithi194_121512_object88383:head# 2024-04-29T14:26:39.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea19c8d:::benchmark_data_smithi194_121512_object20579:head# 2024-04-29T14:26:39.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea27ed7:::benchmark_data_smithi194_121512_object70317:head# 2024-04-29T14:26:39.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea2b12e:::benchmark_data_smithi194_121512_object59482:head# 2024-04-29T14:26:39.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea45208:::benchmark_data_smithi194_121512_object129040:head# 2024-04-29T14:26:39.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea49314:::benchmark_data_smithi194_121512_object13459:head# 2024-04-29T14:26:39.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea4b702:::benchmark_data_smithi194_121512_object81124:head# 2024-04-29T14:26:39.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:26:39.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea4e83b:::benchmark_data_smithi194_121512_object99850:head# 2024-04-29T14:26:39.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea544b1:::benchmark_data_smithi194_121512_object72364:head# 2024-04-29T14:26:39.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea58aaf:::benchmark_data_smithi194_121512_object127914:head# 2024-04-29T14:26:39.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea61fc2:::benchmark_data_smithi194_121512_object39842:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea6e007:::benchmark_data_smithi194_121512_object116250:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea75012:::benchmark_data_smithi194_121512_object117126:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea7584a:::benchmark_data_smithi194_121512_object49667:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea7b1c9:::benchmark_data_smithi194_121512_object40403:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea7cf25:::benchmark_data_smithi194_121512_object85336:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea83031:::benchmark_data_smithi194_121512_object26684:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea85fd1:::benchmark_data_smithi194_121512_object113849:head# 2024-04-29T14:26:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea8686b:::benchmark_data_smithi194_121512_object22629:head# 2024-04-29T14:26:39.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea8d2e6:::benchmark_data_smithi194_121512_object113545:head# 2024-04-29T14:26:39.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea93e01:::benchmark_data_smithi194_121512_object17285:head# 2024-04-29T14:26:39.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea94f5f:::benchmark_data_smithi194_121512_object58095:head# 2024-04-29T14:26:39.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ea9c318:::benchmark_data_smithi194_121512_object28190:head# 2024-04-29T14:26:39.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eaa42fa:::benchmark_data_smithi194_121512_object72866:head# 2024-04-29T14:26:39.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eaa881d:::benchmark_data_smithi194_121512_object95175:head# 2024-04-29T14:26:39.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eab7649:::benchmark_data_smithi194_121512_object101687:head# 2024-04-29T14:26:39.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eabc38a:::benchmark_data_smithi194_121512_object64556:head# 2024-04-29T14:26:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eac0eed:::benchmark_data_smithi194_121512_object66960:head# 2024-04-29T14:26:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eac9bcb:::benchmark_data_smithi194_121512_object54814:head# 2024-04-29T14:26:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ead45a8:::benchmark_data_smithi194_121512_object13784:head# 2024-04-29T14:26:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ead892d:::benchmark_data_smithi194_121512_object67612:head# 2024-04-29T14:26:39.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eae1d6d:::benchmark_data_smithi194_121512_object96443:head# 2024-04-29T14:26:39.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eaf4209:::benchmark_data_smithi194_121512_object20539:head# 2024-04-29T14:26:39.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eafbf23:::benchmark_data_smithi194_121512_object74998:head# 2024-04-29T14:26:39.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb0245d:::benchmark_data_smithi194_121512_object86473:head# 2024-04-29T14:26:39.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb04f2a:::benchmark_data_smithi194_121512_object29458:head# 2024-04-29T14:26:39.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb0aefa:::benchmark_data_smithi194_121512_object3764:head# 2024-04-29T14:26:39.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb2e85a:::benchmark_data_smithi194_121512_object45768:head# 2024-04-29T14:26:39.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb3eec3:::benchmark_data_smithi194_121512_object94873:head# 2024-04-29T14:26:39.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb4aa81:::benchmark_data_smithi194_121512_object112368:head# 2024-04-29T14:26:39.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb55851:::benchmark_data_smithi194_121512_object66325:head# 2024-04-29T14:26:39.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb56779:::benchmark_data_smithi194_121512_object1414:head# 2024-04-29T14:26:39.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb5ae94:::benchmark_data_smithi194_121512_object44193:head# 2024-04-29T14:26:39.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb5ebe3:::benchmark_data_smithi194_121512_object112927:head# 2024-04-29T14:26:39.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb64fc1:::benchmark_data_smithi194_121512_object28616:head# 2024-04-29T14:26:39.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb66f59:::benchmark_data_smithi194_121512_object117347:head# 2024-04-29T14:26:39.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb673ac:::benchmark_data_smithi194_121512_object131531:head# 2024-04-29T14:26:39.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb720d8:::benchmark_data_smithi194_121512_object41072:head# 2024-04-29T14:26:39.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb7f423:::benchmark_data_smithi194_121512_object66:head# 2024-04-29T14:26:39.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb87b02:::benchmark_data_smithi194_121512_object15559:head# 2024-04-29T14:26:39.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb8d46b:::benchmark_data_smithi194_121512_object1852:head# 2024-04-29T14:26:39.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb9a37c:::benchmark_data_smithi194_121512_object92151:head# 2024-04-29T14:26:39.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb9d57e:::benchmark_data_smithi194_121512_object52781:head# 2024-04-29T14:26:39.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb9f182:::benchmark_data_smithi194_121512_object56031:head# 2024-04-29T14:26:39.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eb9fa8d:::benchmark_data_smithi194_121512_object30286:head# 2024-04-29T14:26:39.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebb587a:::benchmark_data_smithi194_121512_object132722:head# 2024-04-29T14:26:39.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebbc986:::benchmark_data_smithi194_121512_object53521:head# 2024-04-29T14:26:39.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:39 smithi040 ceph-mon[39581]: pgmap v1251: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 38584/245910 objects degraded (15.690%); 13 MiB/s, 206 objects/s recovering 2024-04-29T14:26:39.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1123644810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:39.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebc7f34:::benchmark_data_smithi194_121512_object24927:head# 2024-04-29T14:26:39.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebc8caa:::benchmark_data_smithi194_121512_object74966:head# 2024-04-29T14:26:39.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebc8dfc:::benchmark_data_smithi194_121512_object108953:head# 2024-04-29T14:26:39.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebcdfb8:::benchmark_data_smithi194_121512_object12781:head# 2024-04-29T14:26:39.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebd232a:::benchmark_data_smithi194_121512_object1875:head# 2024-04-29T14:26:39.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebd6576:::benchmark_data_smithi194_121512_object32691:head# 2024-04-29T14:26:39.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebd6c60:::benchmark_data_smithi194_121512_object14057:head# 2024-04-29T14:26:39.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebe23b7:::benchmark_data_smithi194_121512_object105146:head# 2024-04-29T14:26:39.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebe6f10:::benchmark_data_smithi194_121512_object62900:head# 2024-04-29T14:26:39.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebed9a7:::benchmark_data_smithi194_121512_object124109:head# 2024-04-29T14:26:39.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebee20b:::benchmark_data_smithi194_121512_object50150:head# 2024-04-29T14:26:39.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebf55f1:::benchmark_data_smithi194_121512_object113053:head# 2024-04-29T14:26:39.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ebfc56b:::benchmark_data_smithi194_121512_object129771:head# 2024-04-29T14:26:39.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec064f3:::benchmark_data_smithi194_121512_object64426:head# 2024-04-29T14:26:39.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec0810e:::benchmark_data_smithi194_121512_object97841:head# 2024-04-29T14:26:39.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec1a385:::benchmark_data_smithi194_121512_object69685:head# 2024-04-29T14:26:39.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec1f5df:::benchmark_data_smithi194_121512_object17712:head# 2024-04-29T14:26:39.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec29b9c:::benchmark_data_smithi194_121512_object37478:head# 2024-04-29T14:26:39.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec319c1:::benchmark_data_smithi194_121512_object76819:head# 2024-04-29T14:26:39.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec338e2:::benchmark_data_smithi194_121512_object105798:head# 2024-04-29T14:26:39.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec38111:::benchmark_data_smithi194_121512_object32963:head# 2024-04-29T14:26:39.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec40388:::benchmark_data_smithi194_121512_object127513:head# 2024-04-29T14:26:39.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec43fcc:::benchmark_data_smithi194_121512_object111500:head# 2024-04-29T14:26:39.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec45723:::benchmark_data_smithi194_121512_object32087:head# 2024-04-29T14:26:39.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec46fa9:::benchmark_data_smithi194_121512_object124125:head# 2024-04-29T14:26:39.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec4a018:::benchmark_data_smithi194_121512_object75002:head# 2024-04-29T14:26:39.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec59f47:::benchmark_data_smithi194_121512_object8367:head# 2024-04-29T14:26:39.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec5a4fa:::benchmark_data_smithi194_121512_object17388:head# 2024-04-29T14:26:39.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec5f644:::benchmark_data_smithi194_121512_object75803:head# 2024-04-29T14:26:39.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec63e9b:::benchmark_data_smithi194_121512_object32316:head# 2024-04-29T14:26:39.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec6bca7:::benchmark_data_smithi194_121512_object911:head# 2024-04-29T14:26:39.485 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec7db04:::benchmark_data_smithi194_121512_object116918:head# 2024-04-29T14:26:39.486 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec91f67:::benchmark_data_smithi194_121512_object29115:head# 2024-04-29T14:26:39.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ec9bf5b:::benchmark_data_smithi194_121512_object93246:head# 2024-04-29T14:26:39.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecba693:::benchmark_data_smithi194_121512_object109649:head# 2024-04-29T14:26:39.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecc1570:::benchmark_data_smithi194_121512_object37283:head# 2024-04-29T14:26:39.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecc17df:::benchmark_data_smithi194_121512_object119941:head# 2024-04-29T14:26:39.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eccfd5f:::benchmark_data_smithi194_121512_object26169:head# 2024-04-29T14:26:39.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecd3a70:::benchmark_data_smithi194_121512_object77481:head# 2024-04-29T14:26:39.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecdad81:::benchmark_data_smithi194_121512_object114106:head# 2024-04-29T14:26:39.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ece759d:::benchmark_data_smithi194_121512_object128199:head# 2024-04-29T14:26:39.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecece53:::benchmark_data_smithi194_121512_object96634:head# 2024-04-29T14:26:39.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecee595:::benchmark_data_smithi194_121512_object110333:head# 2024-04-29T14:26:39.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecfdf8a:::benchmark_data_smithi194_121512_object80547:head# 2024-04-29T14:26:39.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ecff714:::benchmark_data_smithi194_121512_object17149:head# 2024-04-29T14:26:39.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed03966:::benchmark_data_smithi194_121512_object109230:head# 2024-04-29T14:26:39.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed0777c:::benchmark_data_smithi194_121512_object101490:head# 2024-04-29T14:26:39.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed07b1f:::benchmark_data_smithi194_121512_object20758:head# 2024-04-29T14:26:39.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed0b7b5:::benchmark_data_smithi194_121512_object28233:head# 2024-04-29T14:26:39.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed1685f:::benchmark_data_smithi194_121512_object55291:head# 2024-04-29T14:26:39.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed232f7:::benchmark_data_smithi194_121512_object29369:head# 2024-04-29T14:26:39.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed269a4:::benchmark_data_smithi194_121512_object59330:head# 2024-04-29T14:26:39.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed33aa7:::benchmark_data_smithi194_121512_object52906:head# 2024-04-29T14:26:39.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed3d6ab:::benchmark_data_smithi194_121512_object69460:head# 2024-04-29T14:26:39.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed3fc49:::benchmark_data_smithi194_121512_object12151:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed439c3:::benchmark_data_smithi194_121512_object17835:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed44dc1:::benchmark_data_smithi194_121512_object36142:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed46a4b:::benchmark_data_smithi194_121512_object109581:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed477b8:::benchmark_data_smithi194_121512_object39178:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed4fb8a:::benchmark_data_smithi194_121512_object25453:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed56da9:::benchmark_data_smithi194_121512_object24665:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed59095:::benchmark_data_smithi194_121512_object8455:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed6cd71:::benchmark_data_smithi194_121512_object45839:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed72c45:::benchmark_data_smithi194_121512_object29789:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed787c7:::benchmark_data_smithi194_121512_object32206:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed7ee9f:::benchmark_data_smithi194_121512_object107635:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed82b7d:::benchmark_data_smithi194_121512_object87395:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed89cc7:::benchmark_data_smithi194_121512_object6979:head# 2024-04-29T14:26:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed8a846:::benchmark_data_smithi194_121512_object73304:head# 2024-04-29T14:26:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed93d24:::benchmark_data_smithi194_121512_object130465:head# 2024-04-29T14:26:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ed94b12:::benchmark_data_smithi194_121512_object24811:head# 2024-04-29T14:26:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eda3ba4:::benchmark_data_smithi194_121512_object51228:head# 2024-04-29T14:26:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6edb8483:::benchmark_data_smithi194_121512_object81692:head# 2024-04-29T14:26:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6edc5118:::benchmark_data_smithi194_121512_object36650:head# 2024-04-29T14:26:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6edd1f36:::benchmark_data_smithi194_121512_object45148:head# 2024-04-29T14:26:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eddd1ed:::benchmark_data_smithi194_121512_object107461:head# 2024-04-29T14:26:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ede32d3:::benchmark_data_smithi194_121512_object110553:head# 2024-04-29T14:26:39.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ede4fb2:::benchmark_data_smithi194_121512_object52435:head# 2024-04-29T14:26:39.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ede7bc4:::benchmark_data_smithi194_121512_object98604:head# 2024-04-29T14:26:39.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ede99f3:::benchmark_data_smithi194_121512_object73395:head# 2024-04-29T14:26:39.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6edea9c3:::benchmark_data_smithi194_121512_object124931:head# 2024-04-29T14:26:39.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee06b31:::benchmark_data_smithi194_121512_object76386:head# 2024-04-29T14:26:39.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee2d5b3:::benchmark_data_smithi194_121512_object64795:head# 2024-04-29T14:26:39.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee30a66:::benchmark_data_smithi194_121512_object28779:head# 2024-04-29T14:26:39.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee36eaa:::benchmark_data_smithi194_121512_object57860:head# 2024-04-29T14:26:39.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee371d8:::benchmark_data_smithi194_121512_object112570:head# 2024-04-29T14:26:39.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee37e3d:::benchmark_data_smithi194_121512_object36737:head# 2024-04-29T14:26:39.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee3cad8:::benchmark_data_smithi194_121512_object8884:head# 2024-04-29T14:26:39.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee3cf8d:::benchmark_data_smithi194_121512_object34475:head# 2024-04-29T14:26:39.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee4096c:::benchmark_data_smithi194_121512_object101698:head# 2024-04-29T14:26:39.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee61575:::benchmark_data_smithi194_121512_object15520:head# 2024-04-29T14:26:39.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee63463:::benchmark_data_smithi194_121512_object49870:head# 2024-04-29T14:26:39.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee67d33:::benchmark_data_smithi194_121512_object68358:head# 2024-04-29T14:26:39.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee6b971:::benchmark_data_smithi194_121512_object52556:head# 2024-04-29T14:26:39.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee773e6:::benchmark_data_smithi194_121512_object129374:head# 2024-04-29T14:26:39.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee7c94e:::benchmark_data_smithi194_121512_object77298:head# 2024-04-29T14:26:39.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee7f300:::benchmark_data_smithi194_121512_object58918:head# 2024-04-29T14:26:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee82290:::benchmark_data_smithi194_121512_object43595:head# 2024-04-29T14:26:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee824c7:::benchmark_data_smithi194_121512_object32825:head# 2024-04-29T14:26:39.522 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee830ed:::benchmark_data_smithi194_121512_object45204:head# 2024-04-29T14:26:39.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee8598b:::benchmark_data_smithi194_121512_object4155:head# 2024-04-29T14:26:39.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee863ab:::benchmark_data_smithi194_121512_object4169:head# 2024-04-29T14:26:39.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee89f1b:::benchmark_data_smithi194_121512_object22839:head# 2024-04-29T14:26:39.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee8c057:::benchmark_data_smithi194_121512_object35241:head# 2024-04-29T14:26:39.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee8cf93:::benchmark_data_smithi194_121512_object7990:head# 2024-04-29T14:26:39.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ee9b135:::benchmark_data_smithi194_121512_object101225:head# 2024-04-29T14:26:39.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eea7044:::benchmark_data_smithi194_121512_object67963:head# 2024-04-29T14:26:39.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eea7751:::benchmark_data_smithi194_121512_object95849:head# 2024-04-29T14:26:39.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eea7cef:::benchmark_data_smithi194_121512_object77330:head# 2024-04-29T14:26:39.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eeae9e5:::benchmark_data_smithi194_121512_object98988:head# 2024-04-29T14:26:39.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eebc626:::benchmark_data_smithi194_121512_object21851:head# 2024-04-29T14:26:39.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eec9e6d:::benchmark_data_smithi194_121512_object61676:head# 2024-04-29T14:26:39.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eec9ea8:::benchmark_data_smithi194_121512_object101735:head# 2024-04-29T14:26:39.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eed342b:::benchmark_data_smithi194_121512_object72168:head# 2024-04-29T14:26:39.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eed47f3:::benchmark_data_smithi194_121512_object117129:head# 2024-04-29T14:26:39.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eedf5de:::benchmark_data_smithi194_121512_object97740:head# 2024-04-29T14:26:39.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eee0b0b:::benchmark_data_smithi194_121512_object105537:head# 2024-04-29T14:26:39.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eee1662:::benchmark_data_smithi194_121512_object75891:head# 2024-04-29T14:26:39.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eee1df4:::benchmark_data_smithi194_121512_object13426:head# 2024-04-29T14:26:39.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eee2b92:::benchmark_data_smithi194_121512_object95613:head# 2024-04-29T14:26:39.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eee5b6c:::benchmark_data_smithi194_121512_object37682:head# 2024-04-29T14:26:39.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eeeadd3:::benchmark_data_smithi194_121512_object5042:head# 2024-04-29T14:26:39.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eeee554:::benchmark_data_smithi194_121512_object43426:head# 2024-04-29T14:26:39.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eef1fb9:::benchmark_data_smithi194_121512_object102010:head# 2024-04-29T14:26:39.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eef3a76:::benchmark_data_smithi194_121512_object72116:head# 2024-04-29T14:26:39.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eef5618:::benchmark_data_smithi194_121512_object109440:head# 2024-04-29T14:26:39.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eef7b5a:::benchmark_data_smithi194_121512_object5101:head# 2024-04-29T14:26:39.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eef8fc6:::benchmark_data_smithi194_121512_object21751:head# 2024-04-29T14:26:39.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef01423:::benchmark_data_smithi194_121512_object91920:head# 2024-04-29T14:26:39.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef03123:::benchmark_data_smithi194_121512_object109863:head# 2024-04-29T14:26:39.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef0790f:::benchmark_data_smithi194_121512_object130565:head# 2024-04-29T14:26:39.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef0e460:::benchmark_data_smithi194_121512_object14572:head# 2024-04-29T14:26:39.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef0f7d7:::benchmark_data_smithi194_121512_object7097:head# 2024-04-29T14:26:39.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef1492d:::benchmark_data_smithi194_121512_object111754:head# 2024-04-29T14:26:39.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef23c58:::benchmark_data_smithi194_121512_object63119:head# 2024-04-29T14:26:39.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef37f34:::benchmark_data_smithi194_121512_object125514:head# 2024-04-29T14:26:39.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef3a180:::benchmark_data_smithi194_121512_object40424:head# 2024-04-29T14:26:39.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef3a1fb:::benchmark_data_smithi194_121512_object80315:head# 2024-04-29T14:26:39.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef40cb4:::benchmark_data_smithi194_121512_object39214:head# 2024-04-29T14:26:39.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef448f7:::benchmark_data_smithi194_121512_object125738:head# 2024-04-29T14:26:39.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef47415:::benchmark_data_smithi194_121512_object55204:head# 2024-04-29T14:26:39.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef4a0e0:::benchmark_data_smithi194_121512_object98462:head# 2024-04-29T14:26:39.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef5c519:::benchmark_data_smithi194_121512_object80212:head# 2024-04-29T14:26:39.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef68ec4:::benchmark_data_smithi194_121512_object78624:head# 2024-04-29T14:26:39.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef717f5:::benchmark_data_smithi194_121512_object119707:head# 2024-04-29T14:26:39.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef726aa:::benchmark_data_smithi194_121512_object78590:head# 2024-04-29T14:26:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef7275c:::benchmark_data_smithi194_121512_object27515:head# 2024-04-29T14:26:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef913a6:::benchmark_data_smithi194_121512_object54004:head# 2024-04-29T14:26:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef913b2:::benchmark_data_smithi194_121512_object34767:head# 2024-04-29T14:26:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ef92388:::benchmark_data_smithi194_121512_object11298:head# 2024-04-29T14:26:39.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efa48d7:::benchmark_data_smithi194_121512_object109261:head# 2024-04-29T14:26:39.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efa4d18:::benchmark_data_smithi194_121512_object132108:head# 2024-04-29T14:26:39.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efa4ea2:::benchmark_data_smithi194_121512_object4383:head# 2024-04-29T14:26:39.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efac8b7:::benchmark_data_smithi194_121512_object52468:head# 2024-04-29T14:26:39.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efb7970:::benchmark_data_smithi194_121512_object127157:head# 2024-04-29T14:26:39.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efb8b4d:::benchmark_data_smithi194_121512_object36914:head# 2024-04-29T14:26:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efba416:::benchmark_data_smithi194_121512_object70158:head# 2024-04-29T14:26:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efc5aee:::benchmark_data_smithi194_121512_object11383:head# 2024-04-29T14:26:39.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efcaa3a:::benchmark_data_smithi194_121512_object89774:head# 2024-04-29T14:26:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efd35bc:::benchmark_data_smithi194_121512_object127424:head# 2024-04-29T14:26:39.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efe271b:::benchmark_data_smithi194_121512_object77782:head# 2024-04-29T14:26:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6efe809a:::benchmark_data_smithi194_121512_object85642:head# 2024-04-29T14:26:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eff2466:::benchmark_data_smithi194_121512_object14593:head# 2024-04-29T14:26:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eff5baf:::benchmark_data_smithi194_121512_object1759:head# 2024-04-29T14:26:39.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6eff8bda:::benchmark_data_smithi194_121512_object75332:head# 2024-04-29T14:26:39.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6effff04:::benchmark_data_smithi194_121512_object79317:head# 2024-04-29T14:26:39.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0011e2:::benchmark_data_smithi194_121512_object32579:head# 2024-04-29T14:26:39.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f00778e:::benchmark_data_smithi194_121512_object44863:head# 2024-04-29T14:26:39.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f00e2af:::benchmark_data_smithi194_121512_object96283:head# 2024-04-29T14:26:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f023948:::benchmark_data_smithi194_121512_object102652:head# 2024-04-29T14:26:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f027fa2:::benchmark_data_smithi194_121512_object87165:head# 2024-04-29T14:26:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0386f6:::benchmark_data_smithi194_121512_object132702:head# 2024-04-29T14:26:39.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f03a9ac:::benchmark_data_smithi194_121512_object116366:head# 2024-04-29T14:26:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f03c4a2:::benchmark_data_smithi194_121512_object68655:head# 2024-04-29T14:26:39.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f040070:::benchmark_data_smithi194_121512_object1128:head# 2024-04-29T14:26:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f043ee1:::benchmark_data_smithi194_121512_object45426:head# 2024-04-29T14:26:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f04930b:::benchmark_data_smithi194_121512_object36223:head# 2024-04-29T14:26:39.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f04e450:::benchmark_data_smithi194_121512_object10320:head# 2024-04-29T14:26:39.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f04e758:::benchmark_data_smithi194_121512_object43680:head# 2024-04-29T14:26:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f04e774:::benchmark_data_smithi194_121512_object54830:head# 2024-04-29T14:26:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f05b46a:::benchmark_data_smithi194_121512_object82999:head# 2024-04-29T14:26:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0600a4:::benchmark_data_smithi194_121512_object60990:head# 2024-04-29T14:26:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f06dccf:::benchmark_data_smithi194_121512_object52081:head# 2024-04-29T14:26:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f070351:::benchmark_data_smithi194_121512_object54575:head# 2024-04-29T14:26:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f072a2c:::benchmark_data_smithi194_121512_object53338:head# 2024-04-29T14:26:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f07e1d7:::benchmark_data_smithi194_121512_object31404:head# 2024-04-29T14:26:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f082f07:::benchmark_data_smithi194_121512_object110027:head# 2024-04-29T14:26:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f08b8ce:::benchmark_data_smithi194_121512_object67795:head# 2024-04-29T14:26:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f08e4ff:::benchmark_data_smithi194_121512_object40845:head# 2024-04-29T14:26:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f08ee10:::benchmark_data_smithi194_121512_object99207:head# 2024-04-29T14:26:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f096174:::benchmark_data_smithi194_121512_object63040:head# 2024-04-29T14:26:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0988b0:::benchmark_data_smithi194_121512_object111521:head# 2024-04-29T14:26:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f09dab9:::benchmark_data_smithi194_121512_object55397:head# 2024-04-29T14:26:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f09dde8:::benchmark_data_smithi194_121512_object103963:head# 2024-04-29T14:26:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f09e16a:::benchmark_data_smithi194_121512_object133393:head# 2024-04-29T14:26:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0a00bb:::benchmark_data_smithi194_121512_object106969:head# 2024-04-29T14:26:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0a1b01:::benchmark_data_smithi194_121512_object64840:head# 2024-04-29T14:26:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0a747b:::benchmark_data_smithi194_121512_object103581:head# 2024-04-29T14:26:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0a85a8:::benchmark_data_smithi194_121512_object25639:head# 2024-04-29T14:26:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0b140d:::benchmark_data_smithi194_121512_object90132:head# 2024-04-29T14:26:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0c8ea4:::benchmark_data_smithi194_121512_object25788:head# 2024-04-29T14:26:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0c9fc3:::benchmark_data_smithi194_121512_object42171:head# 2024-04-29T14:26:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0cb543:::benchmark_data_smithi194_121512_object88145:head# 2024-04-29T14:26:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0d75e0:::benchmark_data_smithi194_121512_object117254:head# 2024-04-29T14:26:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0db5ee:::benchmark_data_smithi194_121512_object34131:head# 2024-04-29T14:26:39.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0db8b2:::benchmark_data_smithi194_121512_object54360:head# 2024-04-29T14:26:39.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0dd742:::benchmark_data_smithi194_121512_object4500:head# 2024-04-29T14:26:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0e1d52:::benchmark_data_smithi194_121512_object105420:head# 2024-04-29T14:26:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0e1f51:::benchmark_data_smithi194_121512_object68420:head# 2024-04-29T14:26:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0f6e8d:::benchmark_data_smithi194_121512_object82944:head# 2024-04-29T14:26:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f0f8568:::benchmark_data_smithi194_121512_object9960:head# 2024-04-29T14:26:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1091c7:::benchmark_data_smithi194_121512_object18605:head# 2024-04-29T14:26:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1095b9:::benchmark_data_smithi194_121512_object122850:head# 2024-04-29T14:26:39.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f10bc0b:::benchmark_data_smithi194_121512_object118628:head# 2024-04-29T14:26:39.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f11540b:::benchmark_data_smithi194_121512_object128308:head# 2024-04-29T14:26:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f118662:::benchmark_data_smithi194_121512_object26930:head# 2024-04-29T14:26:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f129d22:::benchmark_data_smithi194_121512_object27211:head# 2024-04-29T14:26:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f12a2bb:::benchmark_data_smithi194_121512_object61645:head# 2024-04-29T14:26:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f13b08e:::benchmark_data_smithi194_121512_object22409:head# 2024-04-29T14:26:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f13b2ab:::benchmark_data_smithi194_121512_object11312:head# 2024-04-29T14:26:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f15e712:::benchmark_data_smithi194_121512_object132205:head# 2024-04-29T14:26:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1614ef:::benchmark_data_smithi194_121512_object13659:head# 2024-04-29T14:26:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1650cb:::benchmark_data_smithi194_121512_object117427:head# 2024-04-29T14:26:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f171b60:::benchmark_data_smithi194_121512_object17372:head# 2024-04-29T14:26:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f172425:::benchmark_data_smithi194_121512_object120655:head# 2024-04-29T14:26:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f17ff75:::benchmark_data_smithi194_121512_object24606:head# 2024-04-29T14:26:39.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f186fae:::benchmark_data_smithi194_121512_object93732:head# 2024-04-29T14:26:39.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f187daf:::benchmark_data_smithi194_121512_object26713:head# 2024-04-29T14:26:39.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1896e9:::benchmark_data_smithi194_121512_object103941:head# 2024-04-29T14:26:39.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f19bfc1:::benchmark_data_smithi194_121512_object26387:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1a0910:::benchmark_data_smithi194_121512_object64666:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1a92dc:::benchmark_data_smithi194_121512_object112634:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1ab9c8:::benchmark_data_smithi194_121512_object120731:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1b44c6:::benchmark_data_smithi194_121512_object8414:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1bb2f9:::benchmark_data_smithi194_121512_object103846:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1ca1da:::benchmark_data_smithi194_121512_object29536:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1cb19a:::benchmark_data_smithi194_121512_object130777:head# 2024-04-29T14:26:39.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1d6636:::benchmark_data_smithi194_121512_object56091:head# 2024-04-29T14:26:39.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1e749b:::benchmark_data_smithi194_121512_object20710:head# 2024-04-29T14:26:39.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f1fb473:::benchmark_data_smithi194_121512_object6710:head# 2024-04-29T14:26:39.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f203ee5:::benchmark_data_smithi194_121512_object125219:head# 2024-04-29T14:26:39.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f206807:::benchmark_data_smithi194_121512_object82280:head# 2024-04-29T14:26:39.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f208d6f:::benchmark_data_smithi194_121512_object64663:head# 2024-04-29T14:26:39.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f218e3d:::benchmark_data_smithi194_121512_object87347:head# 2024-04-29T14:26:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f21a524:::benchmark_data_smithi194_121512_object31140:head# 2024-04-29T14:26:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f21e49b:::benchmark_data_smithi194_121512_object59946:head# 2024-04-29T14:26:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f21edf0:::benchmark_data_smithi194_121512_object61289:head# 2024-04-29T14:26:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2238ca:::benchmark_data_smithi194_121512_object57947:head# 2024-04-29T14:26:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f22cb44:::benchmark_data_smithi194_121512_object18671:head# 2024-04-29T14:26:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f24c080:::benchmark_data_smithi194_121512_object24064:head# 2024-04-29T14:26:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f260a98:::benchmark_data_smithi194_121512_object131676:head# 2024-04-29T14:26:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f267db2:::benchmark_data_smithi194_121512_object121497:head# 2024-04-29T14:26:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f26ceaf:::benchmark_data_smithi194_121512_object15771:head# 2024-04-29T14:26:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f279aa1:::benchmark_data_smithi194_121512_object43212:head# 2024-04-29T14:26:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f285bce:::benchmark_data_smithi194_121512_object42197:head# 2024-04-29T14:26:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f28d062:::benchmark_data_smithi194_121512_object56571:head# 2024-04-29T14:26:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f28fafc:::benchmark_data_smithi194_121512_object79975:head# 2024-04-29T14:26:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f29323a:::benchmark_data_smithi194_121512_object106552:head# 2024-04-29T14:26:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f29e121:::benchmark_data_smithi194_121512_object106895:head# 2024-04-29T14:26:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2a2962:::benchmark_data_smithi194_121512_object93756:head# 2024-04-29T14:26:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2a4b3c:::benchmark_data_smithi194_121512_object56050:head# 2024-04-29T14:26:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2ae563:::benchmark_data_smithi194_121512_object115713:head# 2024-04-29T14:26:39.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2af4f4:::benchmark_data_smithi194_121512_object24342:head# 2024-04-29T14:26:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2b3feb:::benchmark_data_smithi194_121512_object126552:head# 2024-04-29T14:26:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2b4603:::benchmark_data_smithi194_121512_object42895:head# 2024-04-29T14:26:39.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2b7bc9:::benchmark_data_smithi194_121512_object123611:head# 2024-04-29T14:26:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2b98c7:::benchmark_data_smithi194_121512_object24514:head# 2024-04-29T14:26:39.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2bbd38:::benchmark_data_smithi194_121512_object107560:head# 2024-04-29T14:26:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2bd44e:::benchmark_data_smithi194_121512_object23581:head# 2024-04-29T14:26:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2d9afd:::benchmark_data_smithi194_121512_object117901:head# 2024-04-29T14:26:39.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f2f9578:::benchmark_data_smithi194_121512_object60973:head# 2024-04-29T14:26:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f30165f:::benchmark_data_smithi194_121512_object95071:head# 2024-04-29T14:26:39.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f308de3:::benchmark_data_smithi194_121512_object60706:head# 2024-04-29T14:26:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f30b8aa:::benchmark_data_smithi194_121512_object94452:head# 2024-04-29T14:26:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f30d05e:::benchmark_data_smithi194_121512_object21888:head# 2024-04-29T14:26:39.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f313fdb:::benchmark_data_smithi194_121512_object23703:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f31ff2b:::benchmark_data_smithi194_121512_object95314:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f335aaf:::benchmark_data_smithi194_121512_object60347:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f34014b:::benchmark_data_smithi194_121512_object51227:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f343e50:::benchmark_data_smithi194_121512_object96368:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f34b64b:::benchmark_data_smithi194_121512_object74595:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f350e34:::benchmark_data_smithi194_121512_object131209:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f35e37f:::benchmark_data_smithi194_121512_object10050:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f362379:::benchmark_data_smithi194_121512_object125371:head# 2024-04-29T14:26:39.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f362ed0:::benchmark_data_smithi194_121512_object119072:head# 2024-04-29T14:26:39.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f369b6a:::benchmark_data_smithi194_121512_object123868:head# 2024-04-29T14:26:39.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f374d28:::benchmark_data_smithi194_121512_object1050:head# 2024-04-29T14:26:39.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f376b6d:::benchmark_data_smithi194_121512_object2819:head# 2024-04-29T14:26:39.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f377d03:::benchmark_data_smithi194_121512_object110722:head# 2024-04-29T14:26:39.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f37c9c9:::benchmark_data_smithi194_121512_object17479:head# 2024-04-29T14:26:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f38c1c8:::benchmark_data_smithi194_121512_object88331:head# 2024-04-29T14:26:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f398e92:::benchmark_data_smithi194_121512_object54756:head# 2024-04-29T14:26:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3996ac:::benchmark_data_smithi194_121512_object126635:head# 2024-04-29T14:26:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3a014d:::benchmark_data_smithi194_121512_object8274:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3a5aff:::benchmark_data_smithi194_121512_object33823:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3a60da:::benchmark_data_smithi194_121512_object91001:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3aa5ab:::benchmark_data_smithi194_121512_object14534:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3b1d96:::benchmark_data_smithi194_121512_object110261:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3bc032:::benchmark_data_smithi194_121512_object26430:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3be288:::benchmark_data_smithi194_121512_object86044:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3c8704:::benchmark_data_smithi194_121512_object12874:head# 2024-04-29T14:26:39.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3cacd5:::benchmark_data_smithi194_121512_object52562:head# 2024-04-29T14:26:39.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3cca3a:::benchmark_data_smithi194_121512_object38785:head# 2024-04-29T14:26:39.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3d1438:::benchmark_data_smithi194_121512_object21444:head# 2024-04-29T14:26:39.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3d7e71:::benchmark_data_smithi194_121512_object16132:head# 2024-04-29T14:26:39.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3e6784:::benchmark_data_smithi194_121512_object116782:head# 2024-04-29T14:26:39.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3e92d9:::benchmark_data_smithi194_121512_object3594:head# 2024-04-29T14:26:39.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3f1e32:::benchmark_data_smithi194_121512_object69285:head# 2024-04-29T14:26:39.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f3f2ca0:::benchmark_data_smithi194_121512_object112355:head# 2024-04-29T14:26:39.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f412d21:::benchmark_data_smithi194_121512_object60900:head# 2024-04-29T14:26:39.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f417d1d:::benchmark_data_smithi194_121512_object56972:head# 2024-04-29T14:26:39.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f42b59c:::benchmark_data_smithi194_121512_object87942:head# 2024-04-29T14:26:39.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f438aee:::benchmark_data_smithi194_121512_object23763:head# 2024-04-29T14:26:39.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4443be:::benchmark_data_smithi194_121512_object63058:head# 2024-04-29T14:26:39.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f455a39:::benchmark_data_smithi194_121512_object118766:head# 2024-04-29T14:26:39.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f45857a:::benchmark_data_smithi194_121512_object128459:head# 2024-04-29T14:26:39.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f46479f:::benchmark_data_smithi194_121512_object110633:head# 2024-04-29T14:26:39.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f472473:::benchmark_data_smithi194_121512_object88381:head# 2024-04-29T14:26:39.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f475d9b:::benchmark_data_smithi194_121512_object90420:head# 2024-04-29T14:26:39.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f488905:::benchmark_data_smithi194_121512_object56243:head# 2024-04-29T14:26:39.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f48955d:::benchmark_data_smithi194_121512_object38265:head# 2024-04-29T14:26:39.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f495bb1:::benchmark_data_smithi194_121512_object27931:head# 2024-04-29T14:26:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f49da19:::benchmark_data_smithi194_121512_object85859:head# 2024-04-29T14:26:39.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4a167e:::benchmark_data_smithi194_121512_object61105:head# 2024-04-29T14:26:39.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4a46d0:::benchmark_data_smithi194_121512_object68590:head# 2024-04-29T14:26:39.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4a7df3:::benchmark_data_smithi194_121512_object94167:head# 2024-04-29T14:26:39.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4af965:::benchmark_data_smithi194_121512_object34636:head# 2024-04-29T14:26:39.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4c45a2:::benchmark_data_smithi194_121512_object78678:head# 2024-04-29T14:26:39.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4c4823:::benchmark_data_smithi194_121512_object27637:head# 2024-04-29T14:26:39.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4cd94e:::benchmark_data_smithi194_121512_object112474:head# 2024-04-29T14:26:39.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4d783e:::benchmark_data_smithi194_121512_object91405:head# 2024-04-29T14:26:39.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4d893b:::benchmark_data_smithi194_121512_object88183:head# 2024-04-29T14:26:39.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4d8aa8:::benchmark_data_smithi194_121512_object11791:head# 2024-04-29T14:26:39.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4e631f:::benchmark_data_smithi194_121512_object105966:head# 2024-04-29T14:26:39.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4ee2b0:::benchmark_data_smithi194_121512_object80326:head# 2024-04-29T14:26:39.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4f2aa1:::benchmark_data_smithi194_121512_object110377:head# 2024-04-29T14:26:39.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4f843e:::benchmark_data_smithi194_121512_object8238:head# 2024-04-29T14:26:39.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4fa3fe:::benchmark_data_smithi194_121512_object64225:head# 2024-04-29T14:26:39.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f4fbf3f:::benchmark_data_smithi194_121512_object50391:head# 2024-04-29T14:26:39.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5126f9:::benchmark_data_smithi194_121512_object127081:head# 2024-04-29T14:26:39.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f516e86:::benchmark_data_smithi194_121512_object131366:head# 2024-04-29T14:26:39.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f52fa4b:::benchmark_data_smithi194_121512_object61066:head# 2024-04-29T14:26:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5345c3:::benchmark_data_smithi194_121512_object121145:head# 2024-04-29T14:26:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f537640:::benchmark_data_smithi194_121512_object81361:head# 2024-04-29T14:26:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f53ca10:::benchmark_data_smithi194_121512_object112358:head# 2024-04-29T14:26:39.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f543ff4:::benchmark_data_smithi194_121512_object86440:head# 2024-04-29T14:26:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f54f142:::benchmark_data_smithi194_121512_object88094:head# 2024-04-29T14:26:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f555b37:::benchmark_data_smithi194_121512_object24793:head# 2024-04-29T14:26:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f55e268:::benchmark_data_smithi194_121512_object4766:head# 2024-04-29T14:26:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f55ff46:::benchmark_data_smithi194_121512_object117585:head# 2024-04-29T14:26:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f562e73:::benchmark_data_smithi194_121512_object63925:head# 2024-04-29T14:26:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5665bc:::benchmark_data_smithi194_121512_object97379:head# 2024-04-29T14:26:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f56a740:::benchmark_data_smithi194_121512_object101834:head# 2024-04-29T14:26:39.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f571a5f:::benchmark_data_smithi194_121512_object124102:head# 2024-04-29T14:26:39.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f57a4b0:::benchmark_data_smithi194_121512_object30867:head# 2024-04-29T14:26:39.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f57da33:::benchmark_data_smithi194_121512_object64103:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f597b7a:::benchmark_data_smithi194_121512_object29991:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f59890d:::benchmark_data_smithi194_121512_object124117:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f599334:::benchmark_data_smithi194_121512_object132038:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5a597f:::benchmark_data_smithi194_121512_object17386:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5ad36d:::benchmark_data_smithi194_121512_object23779:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5b0f3e:::benchmark_data_smithi194_121512_object88787:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5b6b0f:::benchmark_data_smithi194_121512_object91749:head# 2024-04-29T14:26:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5b8b1c:::benchmark_data_smithi194_121512_object83212:head# 2024-04-29T14:26:39.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5b8bb8:::benchmark_data_smithi194_121512_object35775:head# 2024-04-29T14:26:39.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5bb77e:::benchmark_data_smithi194_121512_object13879:head# 2024-04-29T14:26:39.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5ccdbd:::benchmark_data_smithi194_121512_object93612:head# 2024-04-29T14:26:39.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5cfa69:::benchmark_data_smithi194_121512_object86242:head# 2024-04-29T14:26:39.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5e1307:::benchmark_data_smithi194_121512_object44708:head# 2024-04-29T14:26:39.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5e4485:::benchmark_data_smithi194_121512_object12174:head# 2024-04-29T14:26:39.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5f14ad:::benchmark_data_smithi194_121512_object1501:head# 2024-04-29T14:26:39.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f5ff264:::benchmark_data_smithi194_121512_object17529:head# 2024-04-29T14:26:39.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f607172:::benchmark_data_smithi194_121512_object110507:head# 2024-04-29T14:26:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f609561:::benchmark_data_smithi194_121512_object73692:head# 2024-04-29T14:26:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f60d7b8:::benchmark_data_smithi194_121512_object101331:head# 2024-04-29T14:26:39.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6267e8:::benchmark_data_smithi194_121512_object84193:head# 2024-04-29T14:26:39.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f626af6:::benchmark_data_smithi194_121512_object25744:head# 2024-04-29T14:26:39.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f62f20e:::benchmark_data_smithi194_121512_object68818:head# 2024-04-29T14:26:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f63273f:::benchmark_data_smithi194_121512_object19204:head# 2024-04-29T14:26:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f640248:::benchmark_data_smithi194_121512_object31350:head# 2024-04-29T14:26:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f647d38:::benchmark_data_smithi194_121512_object43813:head# 2024-04-29T14:26:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f64ef15:::benchmark_data_smithi194_121512_object85840:head# 2024-04-29T14:26:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f651cb1:::benchmark_data_smithi194_121512_object42364:head# 2024-04-29T14:26:39.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f652ec6:::benchmark_data_smithi194_121512_object110638:head# 2024-04-29T14:26:39.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6660f9:::benchmark_data_smithi194_121512_object130717:head# 2024-04-29T14:26:39.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f666b3c:::benchmark_data_smithi194_121512_object118043:head# 2024-04-29T14:26:39.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f66c568:::benchmark_data_smithi194_121512_object132560:head# 2024-04-29T14:26:39.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f66ec2f:::benchmark_data_smithi194_121512_object76354:head# 2024-04-29T14:26:39.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6732de:::benchmark_data_smithi194_121512_object121863:head# 2024-04-29T14:26:39.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f67364e:::benchmark_data_smithi194_121512_object97419:head# 2024-04-29T14:26:39.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f680410:::benchmark_data_smithi194_121512_object118158:head# 2024-04-29T14:26:39.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f683a70:::benchmark_data_smithi194_121512_object112316:head# 2024-04-29T14:26:39.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f686a5b:::benchmark_data_smithi194_121512_object88699:head# 2024-04-29T14:26:39.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f68d983:::benchmark_data_smithi194_121512_object28680:head# 2024-04-29T14:26:39.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f68dd1c:::benchmark_data_smithi194_121512_object114173:head# 2024-04-29T14:26:39.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f691001:::benchmark_data_smithi194_121512_object15958:head# 2024-04-29T14:26:39.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f69c5cc:::benchmark_data_smithi194_121512_object74846:head# 2024-04-29T14:26:39.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f69d294:::benchmark_data_smithi194_121512_object51031:head# 2024-04-29T14:26:39.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6a3139:::benchmark_data_smithi194_121512_object40370:head# 2024-04-29T14:26:39.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6a3e96:::benchmark_data_smithi194_121512_object97094:head# 2024-04-29T14:26:39.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6a532b:::benchmark_data_smithi194_121512_object70676:head# 2024-04-29T14:26:39.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6a8312:::benchmark_data_smithi194_121512_object12385:head# 2024-04-29T14:26:39.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6a9612:::benchmark_data_smithi194_121512_object104658:head# 2024-04-29T14:26:39.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6aedd0:::benchmark_data_smithi194_121512_object33622:head# 2024-04-29T14:26:39.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6c0910:::benchmark_data_smithi194_121512_object60106:head# 2024-04-29T14:26:39.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6c1343:::benchmark_data_smithi194_121512_object105538:head# 2024-04-29T14:26:39.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6c405e:::benchmark_data_smithi194_121512_object124744:head# 2024-04-29T14:26:39.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6c86cc:::benchmark_data_smithi194_121512_object121239:head# 2024-04-29T14:26:39.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6c8b21:::benchmark_data_smithi194_121512_object26675:head# 2024-04-29T14:26:39.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6c991b:::benchmark_data_smithi194_121512_object21026:head# 2024-04-29T14:26:39.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6cbd08:::benchmark_data_smithi194_121512_object94539:head# 2024-04-29T14:26:39.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6d0d6a:::benchmark_data_smithi194_121512_object118736:head# 2024-04-29T14:26:39.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6d1d6c:::benchmark_data_smithi194_121512_object39832:head# 2024-04-29T14:26:39.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6dec37:::benchmark_data_smithi194_121512_object10864:head# 2024-04-29T14:26:39.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6e24cf:::benchmark_data_smithi194_121512_object128097:head# 2024-04-29T14:26:39.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6ece2a:::benchmark_data_smithi194_121512_object82272:head# 2024-04-29T14:26:39.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f6f6e50:::benchmark_data_smithi194_121512_object91800:head# 2024-04-29T14:26:39.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f70ffc6:::benchmark_data_smithi194_121512_object50681:head# 2024-04-29T14:26:39.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f714abe:::benchmark_data_smithi194_121512_object113820:head# 2024-04-29T14:26:39.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f71a6ad:::benchmark_data_smithi194_121512_object17756:head# 2024-04-29T14:26:39.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7267a0:::benchmark_data_smithi194_121512_object101076:head# 2024-04-29T14:26:39.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f72e623:::benchmark_data_smithi194_121512_object72066:head# 2024-04-29T14:26:39.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f734118:::benchmark_data_smithi194_121512_object17924:head# 2024-04-29T14:26:39.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f73a857:::benchmark_data_smithi194_121512_object36604:head# 2024-04-29T14:26:39.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f745b8c:::benchmark_data_smithi194_121512_object90345:head# 2024-04-29T14:26:39.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7487dc:::benchmark_data_smithi194_121512_object116234:head# 2024-04-29T14:26:39.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f749b89:::benchmark_data_smithi194_121512_object41692:head# 2024-04-29T14:26:39.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f74a6a7:::benchmark_data_smithi194_121512_object12176:head# 2024-04-29T14:26:39.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7592b1:::benchmark_data_smithi194_121512_object106027:head# 2024-04-29T14:26:39.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f75ceed:::benchmark_data_smithi194_121512_object66660:head# 2024-04-29T14:26:39.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f75dec7:::benchmark_data_smithi194_121512_object98807:head# 2024-04-29T14:26:39.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f761464:::benchmark_data_smithi194_121512_object51942:head# 2024-04-29T14:26:39.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f766631:::benchmark_data_smithi194_121512_object27096:head# 2024-04-29T14:26:39.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f76d311:::benchmark_data_smithi194_121512_object118160:head# 2024-04-29T14:26:39.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f76fe3c:::benchmark_data_smithi194_121512_object16073:head# 2024-04-29T14:26:39.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f78eeb8:::benchmark_data_smithi194_121512_object67636:head# 2024-04-29T14:26:39.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f79300a:::benchmark_data_smithi194_121512_object55678:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7962a0:::benchmark_data_smithi194_121512_object132735:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f799321:::benchmark_data_smithi194_121512_object61882:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7a0602:::benchmark_data_smithi194_121512_object16747:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7a5b20:::benchmark_data_smithi194_121512_object108913:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7a73e7:::benchmark_data_smithi194_121512_object71575:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7a75f7:::benchmark_data_smithi194_121512_object123499:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7ae9a6:::benchmark_data_smithi194_121512_object62459:head# 2024-04-29T14:26:39.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7bcdb2:::benchmark_data_smithi194_121512_object43541:head# 2024-04-29T14:26:39.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7bf9f2:::benchmark_data_smithi194_121512_object43025:head# 2024-04-29T14:26:39.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7c1cb3:::benchmark_data_smithi194_121512_object40362:head# 2024-04-29T14:26:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7c2beb:::benchmark_data_smithi194_121512_object54259:head# 2024-04-29T14:26:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7ca8da:::benchmark_data_smithi194_121512_object55078:head# 2024-04-29T14:26:39.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7d468d:::benchmark_data_smithi194_121512_object34837:head# 2024-04-29T14:26:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7d6526:::benchmark_data_smithi194_121512_object29741:head# 2024-04-29T14:26:39.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7d696e:::benchmark_data_smithi194_121512_object121603:head# 2024-04-29T14:26:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7dc660:::benchmark_data_smithi194_121512_object38284:head# 2024-04-29T14:26:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7e297a:::benchmark_data_smithi194_121512_object75933:head# 2024-04-29T14:26:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7e93d9:::benchmark_data_smithi194_121512_object96578:head# 2024-04-29T14:26:39.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f7f1169:::benchmark_data_smithi194_121512_object29577:head# 2024-04-29T14:26:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f803e58:::benchmark_data_smithi194_121512_object48516:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f804b34:::benchmark_data_smithi194_121512_object64021:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f807d1d:::benchmark_data_smithi194_121512_object48550:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f812d49:::benchmark_data_smithi194_121512_object49252:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f822334:::benchmark_data_smithi194_121512_object102822:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f829b96:::benchmark_data_smithi194_121512_object56884:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f837f74:::benchmark_data_smithi194_121512_object47115:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8405ec:::benchmark_data_smithi194_121512_object94815:head# 2024-04-29T14:26:39.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f842cb8:::benchmark_data_smithi194_121512_object70560:head# 2024-04-29T14:26:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f848ca9:::benchmark_data_smithi194_121512_object40153:head# 2024-04-29T14:26:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f868524:::benchmark_data_smithi194_121512_object33109:head# 2024-04-29T14:26:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f868a1f:::benchmark_data_smithi194_121512_object80501:head# 2024-04-29T14:26:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f871474:::benchmark_data_smithi194_121512_object19546:head# 2024-04-29T14:26:39.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f871b36:::benchmark_data_smithi194_121512_object10714:head# 2024-04-29T14:26:39.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f876d96:::benchmark_data_smithi194_121512_object25768:head# 2024-04-29T14:26:39.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f88456c:::benchmark_data_smithi194_121512_object107618:head# 2024-04-29T14:26:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f88e5c7:::benchmark_data_smithi194_121512_object43080:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f890141:::benchmark_data_smithi194_121512_object122744:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f89735c:::benchmark_data_smithi194_121512_object34035:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f89b7aa:::benchmark_data_smithi194_121512_object90687:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f89dd76:::benchmark_data_smithi194_121512_object113510:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f89faa0:::benchmark_data_smithi194_121512_object130200:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8a09b2:::benchmark_data_smithi194_121512_object54157:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8a1b61:::benchmark_data_smithi194_121512_object111829:head# 2024-04-29T14:26:39.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8a2a4f:::benchmark_data_smithi194_121512_object25857:head# 2024-04-29T14:26:39.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8ab53c:::benchmark_data_smithi194_121512_object87872:head# 2024-04-29T14:26:39.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8aed08:::benchmark_data_smithi194_121512_object44938:head# 2024-04-29T14:26:39.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8aeec0:::benchmark_data_smithi194_121512_object52856:head# 2024-04-29T14:26:39.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8b2c39:::benchmark_data_smithi194_121512_object41405:head# 2024-04-29T14:26:39.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8b9a81:::benchmark_data_smithi194_121512_object119464:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8bb92b:::benchmark_data_smithi194_121512_object24471:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8bbf67:::benchmark_data_smithi194_121512_object31857:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8d26fe:::benchmark_data_smithi194_121512_object60078:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8d3298:::benchmark_data_smithi194_121512_object47948:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8d74ee:::benchmark_data_smithi194_121512_object103137:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8e6a7f:::benchmark_data_smithi194_121512_object106417:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8e8615:::benchmark_data_smithi194_121512_object46578:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8ea099:::benchmark_data_smithi194_121512_object48922:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8f43f9:::benchmark_data_smithi194_121512_object49782:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8fc34a:::benchmark_data_smithi194_121512_object98634:head# 2024-04-29T14:26:39.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f8fd13b:::benchmark_data_smithi194_121512_object4369:head# 2024-04-29T14:26:39.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f908731:::benchmark_data_smithi194_121512_object31933:head# 2024-04-29T14:26:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f92b142:::benchmark_data_smithi194_121512_object6654:head# 2024-04-29T14:26:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f934700:::benchmark_data_smithi194_121512_object24816:head# 2024-04-29T14:26:39.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f93ccbb:::benchmark_data_smithi194_121512_object104608:head# 2024-04-29T14:26:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f93f763:::benchmark_data_smithi194_121512_object35836:head# 2024-04-29T14:26:39.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9418ea:::benchmark_data_smithi194_121512_object125111:head# 2024-04-29T14:26:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f942ac9:::benchmark_data_smithi194_121512_object72709:head# 2024-04-29T14:26:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f94755c:::benchmark_data_smithi194_121512_object87300:head# 2024-04-29T14:26:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f951796:::benchmark_data_smithi194_121512_object68472:head# 2024-04-29T14:26:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9570fd:::benchmark_data_smithi194_121512_object78514:head# 2024-04-29T14:26:39.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f967675:::benchmark_data_smithi194_121512_object124576:head# 2024-04-29T14:26:39.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f969f6a:::benchmark_data_smithi194_121512_object93775:head# 2024-04-29T14:26:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f96cc5e:::benchmark_data_smithi194_121512_object28042:head# 2024-04-29T14:26:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f976333:::benchmark_data_smithi194_121512_object72904:head# 2024-04-29T14:26:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f97c1fc:::benchmark_data_smithi194_121512_object91943:head# 2024-04-29T14:26:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9887bd:::benchmark_data_smithi194_121512_object81377:head# 2024-04-29T14:26:39.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f98b772:::benchmark_data_smithi194_121512_object130640:head# 2024-04-29T14:26:39.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f992c4e:::benchmark_data_smithi194_121512_object55698:head# 2024-04-29T14:26:39.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f993a22:::benchmark_data_smithi194_121512_object79710:head# 2024-04-29T14:26:39.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f997661:::benchmark_data_smithi194_121512_object13465:head# 2024-04-29T14:26:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f997c5d:::benchmark_data_smithi194_121512_object27877:head# 2024-04-29T14:26:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f998a53:::benchmark_data_smithi194_121512_object89690:head# 2024-04-29T14:26:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f99ad59:::benchmark_data_smithi194_121512_object104882:head# 2024-04-29T14:26:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9a27d7:::benchmark_data_smithi194_121512_object39475:head# 2024-04-29T14:26:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9aa1a0:::benchmark_data_smithi194_121512_object70994:head# 2024-04-29T14:26:39.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9ad5d6:::benchmark_data_smithi194_121512_object89999:head# 2024-04-29T14:26:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9b20ad:::benchmark_data_smithi194_121512_object60478:head# 2024-04-29T14:26:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9b4f63:::benchmark_data_smithi194_121512_object126468:head# 2024-04-29T14:26:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9c0a01:::benchmark_data_smithi194_121512_object51450:head# 2024-04-29T14:26:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9cfeb9:::benchmark_data_smithi194_121512_object22418:head# 2024-04-29T14:26:39.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9da2e6:::benchmark_data_smithi194_121512_object15453:head# 2024-04-29T14:26:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9e1ff5:::benchmark_data_smithi194_121512_object122483:head# 2024-04-29T14:26:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9e4e1f:::benchmark_data_smithi194_121512_object79603:head# 2024-04-29T14:26:39.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9e8c2b:::benchmark_data_smithi194_121512_object78745:head# 2024-04-29T14:26:39.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6f9e9291:::benchmark_data_smithi194_121512_object111135:head# 2024-04-29T14:26:39.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa020d4:::benchmark_data_smithi194_121512_object126607:head# 2024-04-29T14:26:39.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa1e4e9:::benchmark_data_smithi194_121512_object73685:head# 2024-04-29T14:26:39.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa1e67f:::benchmark_data_smithi194_121512_object27398:head# 2024-04-29T14:26:39.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa20f42:::benchmark_data_smithi194_121512_object29780:head# 2024-04-29T14:26:39.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa23c2f:::benchmark_data_smithi194_121512_object103181:head# 2024-04-29T14:26:39.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa2bd10:::benchmark_data_smithi194_121512_object102660:head# 2024-04-29T14:26:39.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa409a9:::benchmark_data_smithi194_121512_object52209:head# 2024-04-29T14:26:39.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa425a7:::benchmark_data_smithi194_121512_object43945:head# 2024-04-29T14:26:39.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa46a4c:::benchmark_data_smithi194_121512_object36497:head# 2024-04-29T14:26:39.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa4b7fc:::benchmark_data_smithi194_121512_object126140:head# 2024-04-29T14:26:39.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa58c04:::benchmark_data_smithi194_121512_object53795:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa5f6be:::benchmark_data_smithi194_121512_object129615:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa622fd:::benchmark_data_smithi194_121512_object46883:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa66bb5:::benchmark_data_smithi194_121512_object31744:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa6ba66:::benchmark_data_smithi194_121512_object87040:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa6d302:::benchmark_data_smithi194_121512_object41025:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa6f084:::benchmark_data_smithi194_121512_object77508:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa70809:::benchmark_data_smithi194_121512_object141:head# 2024-04-29T14:26:39.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa8bfe0:::benchmark_data_smithi194_121512_object106737:head# 2024-04-29T14:26:39.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa93096:::benchmark_data_smithi194_121512_object72234:head# 2024-04-29T14:26:39.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fa9bbbe:::benchmark_data_smithi194_121512_object12075:head# 2024-04-29T14:26:39.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6faadbcb:::benchmark_data_smithi194_121512_object53341:head# 2024-04-29T14:26:39.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fabb945:::benchmark_data_smithi194_121512_object23390:head# 2024-04-29T14:26:39.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fabc4e4:::benchmark_data_smithi194_121512_object54149:head# 2024-04-29T14:26:39.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fabd0ec:::benchmark_data_smithi194_121512_object39058:head# 2024-04-29T14:26:39.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fac127d:::benchmark_data_smithi194_121512_object119376:head# 2024-04-29T14:26:39.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fac5975:::benchmark_data_smithi194_121512_object101412:head# 2024-04-29T14:26:39.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fac6eee:::benchmark_data_smithi194_121512_object28960:head# 2024-04-29T14:26:39.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fad0b68:::benchmark_data_smithi194_121512_object100833:head# 2024-04-29T14:26:39.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fad4ea5:::benchmark_data_smithi194_121512_object54973:head# 2024-04-29T14:26:39.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fad65b2:::benchmark_data_smithi194_121512_object46134:head# 2024-04-29T14:26:39.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fae8ec0:::benchmark_data_smithi194_121512_object83307:head# 2024-04-29T14:26:39.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6faed24f:::benchmark_data_smithi194_121512_object112703:head# 2024-04-29T14:26:39.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6faee788:::benchmark_data_smithi194_121512_object75088:head# 2024-04-29T14:26:39.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb02ca3:::benchmark_data_smithi194_121512_object32186:head# 2024-04-29T14:26:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb10511:::benchmark_data_smithi194_121512_object120784:head# 2024-04-29T14:26:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb180b3:::benchmark_data_smithi194_121512_object1597:head# 2024-04-29T14:26:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb19019:::benchmark_data_smithi194_121512_object36584:head# 2024-04-29T14:26:39.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb1e060:::benchmark_data_smithi194_121512_object81187:head# 2024-04-29T14:26:39.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb2d5a2:::benchmark_data_smithi194_121512_object116529:head# 2024-04-29T14:26:39.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb49f2f:::benchmark_data_smithi194_121512_object81096:head# 2024-04-29T14:26:39.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb5c4b7:::benchmark_data_smithi194_121512_object48597:head# 2024-04-29T14:26:39.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb6408d:::benchmark_data_smithi194_121512_object91813:head# 2024-04-29T14:26:39.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb6da70:::benchmark_data_smithi194_121512_object8260:head# 2024-04-29T14:26:39.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb763de:::benchmark_data_smithi194_121512_object52176:head# 2024-04-29T14:26:39.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb8550e:::benchmark_data_smithi194_121512_object54880:head# 2024-04-29T14:26:39.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb9a453:::benchmark_data_smithi194_121512_object2025:head# 2024-04-29T14:26:39.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb9bf20:::benchmark_data_smithi194_121512_object112519:head# 2024-04-29T14:26:39.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fb9cabd:::benchmark_data_smithi194_121512_object130808:head# 2024-04-29T14:26:39.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fba5848:::benchmark_data_smithi194_121512_object35665:head# 2024-04-29T14:26:39.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbac197:::benchmark_data_smithi194_121512_object83933:head# 2024-04-29T14:26:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbad3b3:::benchmark_data_smithi194_121512_object113835:head# 2024-04-29T14:26:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbb16f9:::benchmark_data_smithi194_121512_object3385:head# 2024-04-29T14:26:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbbacb9:::benchmark_data_smithi194_121512_object74696:head# 2024-04-29T14:26:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbc1361:::benchmark_data_smithi194_121512_object79780:head# 2024-04-29T14:26:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbcb077:::benchmark_data_smithi194_121512_object112083:head# 2024-04-29T14:26:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbd228d:::benchmark_data_smithi194_121512_object47746:head# 2024-04-29T14:26:39.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbddf1c:::benchmark_data_smithi194_121512_object44188:head# 2024-04-29T14:26:39.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbe2ba7:::benchmark_data_smithi194_121512_object101636:head# 2024-04-29T14:26:39.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbe336a:::benchmark_data_smithi194_121512_object124212:head# 2024-04-29T14:26:39.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbebaf7:::benchmark_data_smithi194_121512_object108276:head# 2024-04-29T14:26:39.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbee244:::benchmark_data_smithi194_121512_object76631:head# 2024-04-29T14:26:39.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbfa497:::benchmark_data_smithi194_121512_object51200:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fbff837:::benchmark_data_smithi194_121512_object44585:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc00aea:::benchmark_data_smithi194_121512_object28265:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc14c4b:::benchmark_data_smithi194_121512_object80242:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc18fd0:::benchmark_data_smithi194_121512_object128164:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc1cf9c:::benchmark_data_smithi194_121512_object23597:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc1ffb8:::benchmark_data_smithi194_121512_object68663:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc231c8:::benchmark_data_smithi194_121512_object76990:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc36a63:::benchmark_data_smithi194_121512_object89658:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc38e3a:::benchmark_data_smithi194_121512_object63674:head# 2024-04-29T14:26:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc3b50a:::benchmark_data_smithi194_121512_object67156:head# 2024-04-29T14:26:39.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc3c431:::benchmark_data_smithi194_121512_object9961:head# 2024-04-29T14:26:39.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc4329b:::benchmark_data_smithi194_121512_object112212:head# 2024-04-29T14:26:39.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc467d5:::benchmark_data_smithi194_121512_object96280:head# 2024-04-29T14:26:39.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc56c67:::benchmark_data_smithi194_121512_object59159:head# 2024-04-29T14:26:39.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc63ca3:::benchmark_data_smithi194_121512_object16926:head# 2024-04-29T14:26:39.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc6f0d9:::benchmark_data_smithi194_121512_object84005:head# 2024-04-29T14:26:39.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc6f184:::benchmark_data_smithi194_121512_object43774:head# 2024-04-29T14:26:39.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc86209:::benchmark_data_smithi194_121512_object40173:head# 2024-04-29T14:26:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc89287:::benchmark_data_smithi194_121512_object103028:head# 2024-04-29T14:26:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc89700:::benchmark_data_smithi194_121512_object33878:head# 2024-04-29T14:26:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc90bc1:::benchmark_data_smithi194_121512_object124972:head# 2024-04-29T14:26:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc9bed8:::benchmark_data_smithi194_121512_object39081:head# 2024-04-29T14:26:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fc9e578:::benchmark_data_smithi194_121512_object10959:head# 2024-04-29T14:26:39.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fca0b84:::benchmark_data_smithi194_121512_object68684:head# 2024-04-29T14:26:39.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcb888d:::benchmark_data_smithi194_121512_object129201:head# 2024-04-29T14:26:39.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcbcbbf:::benchmark_data_smithi194_121512_object16165:head# 2024-04-29T14:26:39.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcbeb81:::benchmark_data_smithi194_121512_object57633:head# 2024-04-29T14:26:39.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcc567d:::benchmark_data_smithi194_121512_object78926:head# 2024-04-29T14:26:39.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcc6d41:::benchmark_data_smithi194_121512_object39553:head# 2024-04-29T14:26:39.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fccec78:::benchmark_data_smithi194_121512_object44165:head# 2024-04-29T14:26:39.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcd2299:::benchmark_data_smithi194_121512_object9148:head# 2024-04-29T14:26:39.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcd6d8c:::benchmark_data_smithi194_121512_object33568:head# 2024-04-29T14:26:39.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fce64b7:::benchmark_data_smithi194_121512_object70729:head# 2024-04-29T14:26:39.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcf30ba:::benchmark_data_smithi194_121512_object1518:head# 2024-04-29T14:26:39.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcf4c89:::benchmark_data_smithi194_121512_object97981:head# 2024-04-29T14:26:39.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcf8134:::benchmark_data_smithi194_121512_object102176:head# 2024-04-29T14:26:39.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fcf959c:::benchmark_data_smithi194_121512_object89308:head# 2024-04-29T14:26:39.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd01d36:::benchmark_data_smithi194_121512_object88905:head# 2024-04-29T14:26:39.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd06ea6:::benchmark_data_smithi194_121512_object110127:head# 2024-04-29T14:26:39.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd0710c:::benchmark_data_smithi194_121512_object18501:head# 2024-04-29T14:26:39.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd0973a:::benchmark_data_smithi194_121512_object52329:head# 2024-04-29T14:26:39.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd128b1:::benchmark_data_smithi194_121512_object10229:head# 2024-04-29T14:26:39.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd1bcec:::benchmark_data_smithi194_121512_object33463:head# 2024-04-29T14:26:39.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd247da:::benchmark_data_smithi194_121512_object46382:head# 2024-04-29T14:26:39.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd30609:::benchmark_data_smithi194_121512_object96507:head# 2024-04-29T14:26:39.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd3f78a:::benchmark_data_smithi194_121512_object105425:head# 2024-04-29T14:26:39.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd485b0:::benchmark_data_smithi194_121512_object44376:head# 2024-04-29T14:26:39.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd51080:::benchmark_data_smithi194_121512_object73343:head# 2024-04-29T14:26:39.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd66ea3:::benchmark_data_smithi194_121512_object4960:head# 2024-04-29T14:26:39.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd6e9d7:::benchmark_data_smithi194_121512_object61919:head# 2024-04-29T14:26:39.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd6fbee:::benchmark_data_smithi194_121512_object69911:head# 2024-04-29T14:26:39.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd71023:::benchmark_data_smithi194_121512_object4821:head# 2024-04-29T14:26:39.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd725ee:::benchmark_data_smithi194_121512_object54056:head# 2024-04-29T14:26:39.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd739b3:::benchmark_data_smithi194_121512_object129964:head# 2024-04-29T14:26:39.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd80def:::benchmark_data_smithi194_121512_object100226:head# 2024-04-29T14:26:39.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd8113c:::benchmark_data_smithi194_121512_object126709:head# 2024-04-29T14:26:39.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd88366:::benchmark_data_smithi194_121512_object30422:head# 2024-04-29T14:26:39.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd89f38:::benchmark_data_smithi194_121512_object75707:head# 2024-04-29T14:26:39.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fd9d5a1:::benchmark_data_smithi194_121512_object71635:head# 2024-04-29T14:26:39.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fda891e:::benchmark_data_smithi194_121512_object77744:head# 2024-04-29T14:26:39.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fda9b52:::benchmark_data_smithi194_121512_object43679:head# 2024-04-29T14:26:39.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdad788:::benchmark_data_smithi194_121512_object874:head# 2024-04-29T14:26:39.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdaeb9a:::benchmark_data_smithi194_121512_object27900:head# 2024-04-29T14:26:39.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdb3843:::benchmark_data_smithi194_121512_object36062:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdb68e6:::benchmark_data_smithi194_121512_object93670:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdd2a32:::benchmark_data_smithi194_121512_object51432:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdd7860:::benchmark_data_smithi194_121512_object110869:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdd8a00:::benchmark_data_smithi194_121512_object123548:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fddf732:::benchmark_data_smithi194_121512_object52157:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fde4416:::benchmark_data_smithi194_121512_object61394:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fde7bd8:::benchmark_data_smithi194_121512_object69588:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fdf312f:::benchmark_data_smithi194_121512_object74480:head# 2024-04-29T14:26:39.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe17e60:::benchmark_data_smithi194_121512_object23327:head# 2024-04-29T14:26:39.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe199ff:::benchmark_data_smithi194_121512_object69393:head# 2024-04-29T14:26:39.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe1adba:::benchmark_data_smithi194_121512_object88048:head# 2024-04-29T14:26:39.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe28dc8:::benchmark_data_smithi194_121512_object28405:head# 2024-04-29T14:26:39.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe388a3:::benchmark_data_smithi194_121512_object84175:head# 2024-04-29T14:26:39.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe39c89:::benchmark_data_smithi194_121512_object24560:head# 2024-04-29T14:26:39.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe480c8:::benchmark_data_smithi194_121512_object110386:head# 2024-04-29T14:26:39.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe512c2:::benchmark_data_smithi194_121512_object9883:head# 2024-04-29T14:26:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe5775d:::benchmark_data_smithi194_121512_object95997:head# 2024-04-29T14:26:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe5dc04:::benchmark_data_smithi194_121512_object12501:head# 2024-04-29T14:26:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe6af3b:::benchmark_data_smithi194_121512_object52646:head# 2024-04-29T14:26:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe6c3b6:::benchmark_data_smithi194_121512_object98519:head# 2024-04-29T14:26:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe767f2:::benchmark_data_smithi194_121512_object51389:head# 2024-04-29T14:26:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe845f1:::benchmark_data_smithi194_121512_object35037:head# 2024-04-29T14:26:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe8deb6:::benchmark_data_smithi194_121512_object76982:head# 2024-04-29T14:26:39.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe8f8e8:::benchmark_data_smithi194_121512_object16635:head# 2024-04-29T14:26:39.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fe9a03a:::benchmark_data_smithi194_121512_object131536:head# 2024-04-29T14:26:39.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fea62f7:::benchmark_data_smithi194_121512_object106075:head# 2024-04-29T14:26:39.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6feafe19:::benchmark_data_smithi194_121512_object126270:head# 2024-04-29T14:26:39.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6feb1f8c:::benchmark_data_smithi194_121512_object45218:head# 2024-04-29T14:26:39.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6feb9b14:::benchmark_data_smithi194_121512_object124220:head# 2024-04-29T14:26:39.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fec19ac:::benchmark_data_smithi194_121512_object92953:head# 2024-04-29T14:26:39.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fec68fb:::benchmark_data_smithi194_121512_object111318:head# 2024-04-29T14:26:39.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fecbbbc:::benchmark_data_smithi194_121512_object21911:head# 2024-04-29T14:26:39.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fed270e:::benchmark_data_smithi194_121512_object55243:head# 2024-04-29T14:26:39.843 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fed7119:::benchmark_data_smithi194_121512_object29673:head# 2024-04-29T14:26:39.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fee249a:::benchmark_data_smithi194_121512_object124641:head# 2024-04-29T14:26:39.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6feec8b7:::benchmark_data_smithi194_121512_object74118:head# 2024-04-29T14:26:39.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fef1661:::benchmark_data_smithi194_121512_object111105:head# 2024-04-29T14:26:39.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fef873b:::benchmark_data_smithi194_121512_object29127:head# 2024-04-29T14:26:39.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fefc8df:::benchmark_data_smithi194_121512_object46076:head# 2024-04-29T14:26:39.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff07b7d:::benchmark_data_smithi194_121512_object107565:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff0a69f:::benchmark_data_smithi194_121512_object130023:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff14e7d:::benchmark_data_smithi194_121512_object120991:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff1cf1a:::benchmark_data_smithi194_121512_object92297:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff1dc8c:::benchmark_data_smithi194_121512_object102735:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff22f1d:::benchmark_data_smithi194_121512_object122646:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff289da:::benchmark_data_smithi194_121512_object3788:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff34d15:::benchmark_data_smithi194_121512_object124735:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff3ad98:::benchmark_data_smithi194_121512_object130206:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff3b505:::benchmark_data_smithi194_121512_object71164:head# 2024-04-29T14:26:39.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff406a2:::benchmark_data_smithi194_121512_object118230:head# 2024-04-29T14:26:39.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff517c6:::benchmark_data_smithi194_121512_object33628:head# 2024-04-29T14:26:39.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff55a80:::benchmark_data_smithi194_121512_object111619:head# 2024-04-29T14:26:39.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff58f88:::benchmark_data_smithi194_121512_object76008:head# 2024-04-29T14:26:39.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff5fa43:::benchmark_data_smithi194_121512_object14149:head# 2024-04-29T14:26:39.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff6d387:::benchmark_data_smithi194_121512_object69109:head# 2024-04-29T14:26:39.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff789eb:::benchmark_data_smithi194_121512_object31827:head# 2024-04-29T14:26:39.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff7b840:::benchmark_data_smithi194_121512_object1377:head# 2024-04-29T14:26:39.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff83d1b:::benchmark_data_smithi194_121512_object108410:head# 2024-04-29T14:26:39.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff959b7:::benchmark_data_smithi194_121512_object47145:head# 2024-04-29T14:26:39.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ff9c679:::benchmark_data_smithi194_121512_object61046:head# 2024-04-29T14:26:39.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ffa2784:::benchmark_data_smithi194_121512_object111311:head# 2024-04-29T14:26:39.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ffa8552:::benchmark_data_smithi194_121512_object25623:head# 2024-04-29T14:26:39.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ffae441:::benchmark_data_smithi194_121512_object2582:head# 2024-04-29T14:26:39.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ffb574e:::benchmark_data_smithi194_121512_object116625:head# 2024-04-29T14:26:39.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ffc061e:::benchmark_data_smithi194_121512_object74912:head# 2024-04-29T14:26:39.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6ffebf92:::benchmark_data_smithi194_121512_object32399:head# 2024-04-29T14:26:39.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #5:6fff3a0d:::benchmark_data_smithi194_121512_object114453:head# 2024-04-29T14:26:39.861 INFO:teuthology.orchestra.run.smithi121.stderr:Export successful 2024-04-29T14:26:39.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:39.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:40.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:40.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:26:40.080 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:40.097 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:40.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:41.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:41 smithi040 ceph-mon[39581]: pgmap v1252: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 406 op/s; 38414/244860 objects degraded (15.688%); 8.4 MiB/s, 135 objects/s recovering 2024-04-29T14:26:41.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:41 smithi194 ceph-mon[44089]: pgmap v1252: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 406 op/s; 38414/244860 objects degraded (15.688%); 8.4 MiB/s, 135 objects/s recovering 2024-04-29T14:26:41.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:41 smithi121 ceph-mon[44533]: pgmap v1252: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 406 op/s; 38414/244860 objects degraded (15.688%); 8.4 MiB/s, 135 objects/s recovering 2024-04-29T14:26:42.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:26:42.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:26:42.335 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:42 smithi040 ceph-mon[39581]: pgmap v1253: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 621 op/s; 37715/240364 objects degraded (15.691%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:26:42.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:42 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 38414/244860 objects degraded (15.688%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:42 smithi194 ceph-mon[44089]: pgmap v1253: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 621 op/s; 37715/240364 objects degraded (15.691%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:26:42.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:42 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 38414/244860 objects degraded (15.688%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:42.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:42 smithi121 ceph-mon[44533]: pgmap v1253: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 621 op/s; 37715/240364 objects degraded (15.691%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:26:42.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:42 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 38414/244860 objects degraded (15.688%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:43.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:26:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:43.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:26:43.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:43.206 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:43.229 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:43.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:26:43.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:43.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/167137845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:26:43.370 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/167137845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:26:43.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/167137845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:26:44.408 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:44.428 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:44.449 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:44.472 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:44.499 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:44.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:44.541 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:44 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/167137845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:26:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:44 smithi194 ceph-mon[44089]: osdmap e476: 12 total, 11 up, 7 in 2024-04-29T14:26:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:44 smithi194 ceph-mon[44089]: pgmap v1255: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 557 op/s; 37525/239322 objects degraded (15.680%); 12 MiB/s, 186 objects/s recovering 2024-04-29T14:26:44.567 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:44.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:44 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:44.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/167137845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:26:44.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:44 smithi040 ceph-mon[39581]: osdmap e476: 12 total, 11 up, 7 in 2024-04-29T14:26:44.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:44 smithi040 ceph-mon[39581]: pgmap v1255: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 557 op/s; 37525/239322 objects degraded (15.680%); 12 MiB/s, 186 objects/s recovering 2024-04-29T14:26:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:44 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/167137845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:26:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:44 smithi121 ceph-mon[44533]: osdmap e476: 12 total, 11 up, 7 in 2024-04-29T14:26:44.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:44 smithi121 ceph-mon[44533]: pgmap v1255: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 557 op/s; 37525/239322 objects degraded (15.680%); 12 MiB/s, 186 objects/s recovering 2024-04-29T14:26:44.647 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:44.672 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:44.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:44.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:44.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:44.673 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2838322526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2838322526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:45.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2838322526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:45.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:46.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:46 smithi194 ceph-mon[44089]: pgmap v1256: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 37173/236714 objects degraded (15.704%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:26:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:46.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:46 smithi040 ceph-mon[39581]: pgmap v1256: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 37173/236714 objects degraded (15.704%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:26:46.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:46.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:46 smithi121 ceph-mon[44533]: pgmap v1256: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 37173/236714 objects degraded (15.704%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:26:46.975 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:26:47.280 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:47.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:47 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 37173/236714 objects degraded (15.704%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:47.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:47 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 37173/236714 objects degraded (15.704%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:47.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:47 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 37173/236714 objects degraded (15.704%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:47.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:26:47.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 5.6 2024-04-29T14:26:47.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:47.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:47.887 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:26:47.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:26:47.945 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:26:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:48.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:26:48.179 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:48.205 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:48.265 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:48.297 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:48.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:48.404 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:48.453 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:48.487 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:48.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:48.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:48 smithi194 ceph-mon[44089]: pgmap v1257: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 36799/234538 objects degraded (15.690%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:26:48.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:48.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:48 smithi040 ceph-mon[39581]: pgmap v1257: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 36799/234538 objects degraded (15.690%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:26:48.620 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:48 smithi121 ceph-mon[44533]: pgmap v1257: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 36799/234538 objects degraded (15.690%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:26:48.622 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:49.891 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:26:49.936 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:26:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:50.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:50.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4242291406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:50.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:50.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4242291406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:50.205 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:50.278 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:50.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:50.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4242291406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:26:50.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:50.415 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:51.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:50 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:26:51.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4242291406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:51.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:50 smithi040 ceph-mon[39581]: osdmap e477: 12 total, 11 up, 7 in 2024-04-29T14:26:51.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:50 smithi040 ceph-mon[39581]: pgmap v1259: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 432 op/s; 36640/233444 objects degraded (15.695%); 8.4 MiB/s, 134 objects/s recovering 2024-04-29T14:26:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:50 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:26:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4242291406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:50 smithi194 ceph-mon[44089]: osdmap e477: 12 total, 11 up, 7 in 2024-04-29T14:26:51.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:50 smithi194 ceph-mon[44089]: pgmap v1259: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 432 op/s; 36640/233444 objects degraded (15.695%); 8.4 MiB/s, 134 objects/s recovering 2024-04-29T14:26:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:50 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:26:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4242291406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:26:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:50 smithi121 ceph-mon[44533]: osdmap e477: 12 total, 11 up, 7 in 2024-04-29T14:26:51.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:50 smithi121 ceph-mon[44533]: pgmap v1259: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 432 op/s; 36640/233444 objects degraded (15.695%); 8.4 MiB/s, 134 objects/s recovering 2024-04-29T14:26:51.925 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:51.946 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:51.966 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:51.990 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:52.010 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:52.032 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:52.054 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:52.077 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:52.102 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:52.123 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:52.145 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:52.158 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:26:52.158 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:26:52.158 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2562114536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:52.162 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:26:52.162 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:26:52.162 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2562114536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:52.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:26:52.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:26:52.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2562114536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:52.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:52.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:52.682 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:52.717 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:53.009 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:53.067 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:53.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:53.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osdmap e478: 12 total, 11 up, 7 in 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: pgmap v1261: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 666 op/s; 35905/228660 objects degraded (15.702%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:52 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 36640/233444 objects degraded (15.695%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:53.200 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:53.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osdmap e478: 12 total, 11 up, 7 in 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: pgmap v1261: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 666 op/s; 35905/228660 objects degraded (15.702%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:52 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 36640/233444 objects degraded (15.695%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:53.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:53.352 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osdmap e478: 12 total, 11 up, 7 in 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: pgmap v1261: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 7.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 666 op/s; 35905/228660 objects degraded (15.702%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:52 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 36640/233444 objects degraded (15.695%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:53.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:53.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:26:53.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:53.509 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:53.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:53.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:53.632 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:53.706 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:53.714 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:53.766 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:53.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:53.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:53.919 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:53.973 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:53.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:53.979 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:53.979 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:54.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:54.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:54.263 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:54.263 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:54.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:54.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:54.963 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:26:54.983 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:54.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:55.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:26:55.089 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:55.142 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:55.165 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:55.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:55.230 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:54 smithi121 ceph-mon[44533]: pgmap v1262: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 567 op/s; 35726/227624 objects degraded (15.695%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:26:55.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:55.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:54 smithi040 ceph-mon[39581]: pgmap v1262: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 567 op/s; 35726/227624 objects degraded (15.695%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:26:55.302 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:55.313 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:54 smithi194 ceph-mon[44089]: pgmap v1262: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 567 op/s; 35726/227624 objects degraded (15.695%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:26:55.454 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:55.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:55.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:55.590 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:55.630 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:55.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:55.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:55.772 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:55.774 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:26:55.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:55.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:56.028 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:26:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:26:56.089 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2094010904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:56.089 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:56 smithi121 ceph-mon[44533]: from='client.26245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:56.090 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:26:56.090 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:26:56.090 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:26:56.091 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:26:56.091 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:26:56.100 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:56.149 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:26:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:56.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:26:56.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:26:56.316 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:26:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:56.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:26:56.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2094010904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:56.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:56 smithi040 ceph-mon[39581]: from='client.26245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:56.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:56.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2094010904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:56.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:56 smithi194 ceph-mon[44089]: from='client.26245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:26:56.511 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:56.543 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:56.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:56.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:26:56.684 INFO:teuthology.orchestra.run.smithi121.stdout: marking collection for removal 2024-04-29T14:26:56.685 INFO:teuthology.orchestra.run.smithi121.stdout:setting '_remove' omap key 2024-04-29T14:26:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:56.687 INFO:teuthology.orchestra.run.smithi121.stdout:finish_remove_pgs 5.6_head removing 5.6 2024-04-29T14:26:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:26:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:56.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:26:56.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:26:56.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:26:56.893 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:56.901 INFO:teuthology.orchestra.run.smithi121.stdout:Remove successful 2024-04-29T14:26:56.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:26:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:56.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:26:57.034 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:26:57.087 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:57 smithi121 ceph-mon[44533]: pgmap v1263: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 585 op/s; 35345/225162 objects degraded (15.698%); 11 MiB/s, 181 objects/s recovering 2024-04-29T14:26:57.088 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:57 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:57.088 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:57 smithi121 ceph-mon[44533]: from='client.26245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:26:57.088 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:57 smithi121 ceph-mon[44533]: osdmap e479: 12 total, 11 up, 7 in 2024-04-29T14:26:57.089 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:57.116 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:26:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:57.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:26:57.220 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:26:57.284 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:57.312 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:26:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:57.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:26:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:57 smithi040 ceph-mon[39581]: pgmap v1263: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 585 op/s; 35345/225162 objects degraded (15.698%); 11 MiB/s, 181 objects/s recovering 2024-04-29T14:26:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:57 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:57 smithi040 ceph-mon[39581]: from='client.26245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:26:57.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:57 smithi040 ceph-mon[39581]: osdmap e479: 12 total, 11 up, 7 in 2024-04-29T14:26:57.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:26:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:57.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:57 smithi194 ceph-mon[44089]: pgmap v1263: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 585 op/s; 35345/225162 objects degraded (15.698%); 11 MiB/s, 181 objects/s recovering 2024-04-29T14:26:57.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:57 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:26:57.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:57 smithi194 ceph-mon[44089]: from='client.26245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:26:57.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:57 smithi194 ceph-mon[44089]: osdmap e479: 12 total, 11 up, 7 in 2024-04-29T14:26:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:26:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:58.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:26:58.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:58.310 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:58.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:58.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:58 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 35345/225162 objects degraded (15.698%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:58.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:58 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 35345/225162 objects degraded (15.698%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:58.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:26:58.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:26:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:58.466 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:26:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:26:58.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:26:58.470 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:26:58.491 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:26:58.513 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:26:58.540 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:26:58.562 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:26:58.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:26:58.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:58 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 35345/225162 objects degraded (15.698%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:26:58.587 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:26:58.612 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:26:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:26:58.634 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:26:58.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:26:58.678 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:26:58.679 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:26:59.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: pgmap v1265: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 34992/222936 objects degraded (15.696%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2866916845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:26:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: pgmap v1265: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 34992/222936 objects degraded (15.696%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2866916845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:26:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:26:59.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:26:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: pgmap v1265: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 34992/222936 objects degraded (15.696%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:26:59.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2866916845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:26:59.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:26:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:00.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:27:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:01.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:01 smithi040 ceph-mon[39581]: pgmap v1266: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 419 op/s; 34822/221894 objects degraded (15.693%); 8.4 MiB/s, 134 objects/s recovering 2024-04-29T14:27:01.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:27:01.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:01 smithi194 ceph-mon[44089]: pgmap v1266: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 419 op/s; 34822/221894 objects degraded (15.693%); 8.4 MiB/s, 134 objects/s recovering 2024-04-29T14:27:01.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:01 smithi121 ceph-mon[44533]: pgmap v1266: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 419 op/s; 34822/221894 objects degraded (15.693%); 8.4 MiB/s, 134 objects/s recovering 2024-04-29T14:27:02.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:02 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 34822/221894 objects degraded (15.693%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:27:02.314 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:02.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.5 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.6.5 2024-04-29T14:27:02.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:02.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:02.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:02.503 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:02 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 34822/221894 objects degraded (15.693%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:02.504 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:02.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:02 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 34822/221894 objects degraded (15.693%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:02.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:02.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:02.636 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.5/config 2024-04-29T14:27:02.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:02.760 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:03.218 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:27:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:03.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:03.320 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:03 smithi040 ceph-mon[39581]: pgmap v1267: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 34074/217144 objects degraded (15.692%); 11 MiB/s, 183 objects/s recovering 2024-04-29T14:27:03.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1476911458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:03.387 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:03.442 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:03.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:03.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:03 smithi194 ceph-mon[44089]: pgmap v1267: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 34074/217144 objects degraded (15.692%); 11 MiB/s, 183 objects/s recovering 2024-04-29T14:27:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1476911458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:03.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:03 smithi121 ceph-mon[44533]: pgmap v1267: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 34074/217144 objects degraded (15.692%); 11 MiB/s, 183 objects/s recovering 2024-04-29T14:27:03.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1476911458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:03.617 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:04.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:04 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:04.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1476911458' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:04.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:04 smithi121 ceph-mon[44533]: osdmap e480: 12 total, 11 up, 7 in 2024-04-29T14:27:04.238 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:04.239 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:04.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:04.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:04.476 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:04.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:04 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:04.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1476911458' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:04.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:04 smithi194 ceph-mon[44089]: osdmap e480: 12 total, 11 up, 7 in 2024-04-29T14:27:04.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:04.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:04.571 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:04.592 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:04.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:04 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1476911458' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:04 smithi040 ceph-mon[39581]: osdmap e480: 12 total, 11 up, 7 in 2024-04-29T14:27:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:04.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:04.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:27:04.728 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:04.787 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:27:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:04.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:27:04.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:04.925 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:05.235 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: osdmap e481: 12 total, 11 up, 7 in 2024-04-29T14:27:05.235 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:27:05.235 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:27:05.235 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:27:05.235 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:27:05.235 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:27:05.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:27:05.236 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:27:05.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:05.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:05.429 INFO:teuthology.orchestra.run.smithi121.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:27:05.493 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:05.516 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:27:05.516 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:27:05.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: osdmap e481: 12 total, 11 up, 7 in 2024-04-29T14:27:05.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:27:05.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:27:05.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:27:05.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:27:05.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:27:05.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:27:05.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:27:05.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: osdmap e481: 12 total, 11 up, 7 in 2024-04-29T14:27:05.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:27:05.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:27:05.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:27:05.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:27:05.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:27:05.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:27:05.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:27:05.554 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:05.577 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:05.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:05.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:27:05.717 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:27:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:05.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:27:05.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:27:05.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:27:05.898 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:27:05.920 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:27:05.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:05.941 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:27:05.965 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:27:05.981 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:05.986 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:27:06.012 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:06.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:06.029 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:27:06.063 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:27:06.091 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:27:06.116 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:06.118 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:27:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:27:06.182 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:06.212 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:06.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:06.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3003731439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3509810291,config=1981513354,health=3817268772,kv=2362877167}) 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3714140303,logm=3765697147}) 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608431645}) 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1950738688}) 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166919106}) 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999053849}) 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786746025}) 2024-04-29T14:27:06.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178282411}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486645052}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300169039}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663479725}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2470423364}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1307168760}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1655301221}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3189184800}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3192632032}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2504420979}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2862595758}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3647935847}) 2024-04-29T14:27:06.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2592655727}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307559877}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750377993}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753898443}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1772245517}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878714438}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373384506}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682523130}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277843662}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3582137584}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3579241745}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=391340225}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197561425}) 2024-04-29T14:27:06.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672522323}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75256367}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668847110}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940697355}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949509747}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550266313}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=299834926}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1655231523}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678797118}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930059194}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417897322}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559757702}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380272772}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3653224301}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3402259655}) 2024-04-29T14:27:06.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2514939031}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998442016}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588013636}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858135431}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066550373}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1797102146}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3401019755}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809841710}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315429135}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2835496416}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=402155934}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834029641}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225777433}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=81,mdsmap=3,mgr=16} crc {logm=3714401982,mdsmap=2840952586,mgr=344556108}) 2024-04-29T14:27:06.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=78} crc {mgr=1248726048,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=4032440458,mon_config_key=4003228758}) 2024-04-29T14:27:06.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1607534414}) 2024-04-29T14:27:06.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=19805058}) 2024-04-29T14:27:06.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=82} crc {monmap=455069592,osd_metadata=137591028,osd_pg_creating=3632408415,osdmap=1335367453}) 2024-04-29T14:27:06.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=37} crc {osdmap=2264914641}) 2024-04-29T14:27:06.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3003731439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:06.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3509810291,config=1981513354,health=3817268772,kv=2362877167}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3714140303,logm=3765697147}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608431645}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1950738688}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166919106}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999053849}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786746025}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178282411}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486645052}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300169039}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663479725}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2470423364}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1307168760}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1655301221}) 2024-04-29T14:27:06.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3189184800}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3192632032}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2504420979}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2862595758}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3647935847}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2592655727}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307559877}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750377993}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753898443}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1772245517}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878714438}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373384506}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682523130}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277843662}) 2024-04-29T14:27:06.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3582137584}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3579241745}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=391340225}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197561425}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672522323}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75256367}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668847110}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940697355}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949509747}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550266313}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=299834926}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1655231523}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678797118}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930059194}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417897322}) 2024-04-29T14:27:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559757702}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380272772}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3653224301}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3402259655}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2514939031}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998442016}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588013636}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858135431}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066550373}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1797102146}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3401019755}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809841710}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315429135}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2835496416}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=402155934}) 2024-04-29T14:27:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834029641}) 2024-04-29T14:27:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225777433}) 2024-04-29T14:27:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=81,mdsmap=3,mgr=16} crc {logm=3714401982,mdsmap=2840952586,mgr=344556108}) 2024-04-29T14:27:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=78} crc {mgr=1248726048,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=4032440458,mon_config_key=4003228758}) 2024-04-29T14:27:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1607534414}) 2024-04-29T14:27:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=19805058}) 2024-04-29T14:27:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=82} crc {monmap=455069592,osd_metadata=137591028,osd_pg_creating=3632408415,osdmap=1335367453}) 2024-04-29T14:27:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=37} crc {osdmap=2264914641}) 2024-04-29T14:27:06.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3003731439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=3509810291,config=1981513354,health=3817268772,kv=2362877167}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3714140303,logm=3765697147}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608431645}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1950738688}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166919106}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999053849}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786746025}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178282411}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486645052}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300169039}) 2024-04-29T14:27:06.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663479725}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2470423364}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1307168760}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1655301221}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3189184800}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3192632032}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2504420979}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2862595758}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3647935847}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2592655727}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307559877}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750377993}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753898443}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1772245517}) 2024-04-29T14:27:06.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878714438}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373384506}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682523130}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277843662}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3582137584}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3579241745}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=391340225}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197561425}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672522323}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75256367}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668847110}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940697355}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949509747}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550266313}) 2024-04-29T14:27:06.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=299834926}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1655231523}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678797118}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930059194}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417897322}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559757702}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3380272772}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3653224301}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3402259655}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2514939031}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998442016}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588013636}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858135431}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066550373}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1797102146}) 2024-04-29T14:27:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3401019755}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809841710}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315429135}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2835496416}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=402155934}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834029641}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225777433}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=81,mdsmap=3,mgr=16} crc {logm=3714401982,mdsmap=2840952586,mgr=344556108}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=78} crc {mgr=1248726048,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=4032440458,mon_config_key=4003228758}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1607534414}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=19805058}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=82} crc {monmap=455069592,osd_metadata=137591028,osd_pg_creating=3632408415,osdmap=1335367453}) 2024-04-29T14:27:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=37} crc {osdmap=2264914641}) 2024-04-29T14:27:06.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:27:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:27:06.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:27:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: pgmap v1271: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 33553/213540 objects degraded (15.713%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:27:07.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:07 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 33553/213540 objects degraded (15.713%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: pgmap v1271: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 33553/213540 objects degraded (15.713%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:27:07.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:07 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 33553/213540 objects degraded (15.713%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: pgmap v1271: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 33553/213540 objects degraded (15.713%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:07.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:07 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 33553/213540 objects degraded (15.713%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:08.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:27:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:08.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:27:08.627 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:08.685 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:08.712 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:08.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:08.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:27:08.849 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:27:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:08.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:27:08.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:27:09.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:09.046 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:09.296 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:09.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:09 smithi194 ceph-mon[44089]: pgmap v1272: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 33174/211342 objects degraded (15.697%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:27:09.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:09 smithi040 ceph-mon[39581]: pgmap v1272: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 33174/211342 objects degraded (15.697%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:27:09.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:09 smithi121 ceph-mon[44533]: pgmap v1272: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 33174/211342 objects degraded (15.697%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:27:10.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:27:10.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:10.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:10.239 INFO:teuthology.orchestra.run.smithi121.stdout:get_pg_num_history pg_num_history pg_num_history(e457 pg_nums {1={14=1},4={323=16},5={432=16}} deleted_pools 431,4) 2024-04-29T14:27:10.239 INFO:teuthology.orchestra.run.smithi121.stdout:Importing pgid 5.6 2024-04-29T14:27:10.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60001ed0:::benchmark_data_smithi194_121512_object24470:head# 2024-04-29T14:27:10.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6000a506:::benchmark_data_smithi194_121512_object73522:head# 2024-04-29T14:27:10.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6000eb80:::benchmark_data_smithi194_121512_object30226:head# 2024-04-29T14:27:10.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6001080f:::benchmark_data_smithi194_121512_object78388:head# 2024-04-29T14:27:10.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60028613:::benchmark_data_smithi194_121512_object45583:head# 2024-04-29T14:27:10.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6002aa12:::benchmark_data_smithi194_121512_object71361:head# 2024-04-29T14:27:10.263 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6003031b:::benchmark_data_smithi194_121512_object98011:head# 2024-04-29T14:27:10.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6004218c:::benchmark_data_smithi194_121512_object14532:head# 2024-04-29T14:27:10.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60042ddd:::benchmark_data_smithi194_121512_object47941:head# 2024-04-29T14:27:10.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6004519a:::benchmark_data_smithi194_121512_object69753:head# 2024-04-29T14:27:10.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60049131:::benchmark_data_smithi194_121512_object127112:head# 2024-04-29T14:27:10.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:10.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6004f0ff:::benchmark_data_smithi194_121512_object121400:head# 2024-04-29T14:27:10.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600538b9:::benchmark_data_smithi194_121512_object4360:head# 2024-04-29T14:27:10.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60063bd2:::benchmark_data_smithi194_121512_object47110:head# 2024-04-29T14:27:10.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60064e30:::benchmark_data_smithi194_121512_object43763:head# 2024-04-29T14:27:10.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60074edf:::benchmark_data_smithi194_121512_object82297:head# 2024-04-29T14:27:10.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.272 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6007589e:::benchmark_data_smithi194_121512_object73634:head# 2024-04-29T14:27:10.272 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6007d98d:::benchmark_data_smithi194_121512_object118990:head# 2024-04-29T14:27:10.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.273 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600855e0:::benchmark_data_smithi194_121512_object83524:head# 2024-04-29T14:27:10.273 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600b1280:::benchmark_data_smithi194_121512_object105085:head# 2024-04-29T14:27:10.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600b8e01:::benchmark_data_smithi194_121512_object123610:head# 2024-04-29T14:27:10.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600dd11d:::benchmark_data_smithi194_121512_object76597:head# 2024-04-29T14:27:10.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600df137:::benchmark_data_smithi194_121512_object81861:head# 2024-04-29T14:27:10.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:27:10.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600e0973:::benchmark_data_smithi194_121512_object81877:head# 2024-04-29T14:27:10.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600e5d43:::benchmark_data_smithi194_121512_object124695:head# 2024-04-29T14:27:10.297 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.297 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600e646e:::benchmark_data_smithi194_121512_object59793:head# 2024-04-29T14:27:10.298 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.300 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:10.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600e92ef:::benchmark_data_smithi194_121512_object6904:head# 2024-04-29T14:27:10.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:600ef351:::benchmark_data_smithi194_121512_object833:head# 2024-04-29T14:27:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6010f4eb:::benchmark_data_smithi194_121512_object27410:head# 2024-04-29T14:27:10.321 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.322 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60114e11:::benchmark_data_smithi194_121512_object14919:head# 2024-04-29T14:27:10.322 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6011593d:::benchmark_data_smithi194_121512_object58807:head# 2024-04-29T14:27:10.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60116a40:::benchmark_data_smithi194_121512_object83691:head# 2024-04-29T14:27:10.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601204fe:::benchmark_data_smithi194_121512_object27226:head# 2024-04-29T14:27:10.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60122e35:::benchmark_data_smithi194_121512_object52362:head# 2024-04-29T14:27:10.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60123e1c:::benchmark_data_smithi194_121512_object9504:head# 2024-04-29T14:27:10.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60138350:::benchmark_data_smithi194_121512_object93164:head# 2024-04-29T14:27:10.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6013cc60:::benchmark_data_smithi194_121512_object32954:head# 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6013e97e:::benchmark_data_smithi194_121512_object113602:head# 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60142d6e:::benchmark_data_smithi194_121512_object97725:head# 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60143892:::benchmark_data_smithi194_121512_object12706:head# 2024-04-29T14:27:10.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601512fc:::benchmark_data_smithi194_121512_object87962:head# 2024-04-29T14:27:10.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60157b71:::benchmark_data_smithi194_121512_object132573:head# 2024-04-29T14:27:10.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6015bb54:::benchmark_data_smithi194_121512_object68143:head# 2024-04-29T14:27:10.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.347 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6016eef1:::benchmark_data_smithi194_121512_object95236:head# 2024-04-29T14:27:10.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.350 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6017228f:::benchmark_data_smithi194_121512_object27501:head# 2024-04-29T14:27:10.350 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.350 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60179ebc:::benchmark_data_smithi194_121512_object99439:head# 2024-04-29T14:27:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601820b4:::benchmark_data_smithi194_121512_object41161:head# 2024-04-29T14:27:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60197e9c:::benchmark_data_smithi194_121512_object16862:head# 2024-04-29T14:27:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601ae4af:::benchmark_data_smithi194_121512_object4313:head# 2024-04-29T14:27:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.352 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601b1e91:::benchmark_data_smithi194_121512_object118363:head# 2024-04-29T14:27:10.352 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.352 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601bb5df:::benchmark_data_smithi194_121512_object1031:head# 2024-04-29T14:27:10.352 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.353 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601cafd8:::benchmark_data_smithi194_121512_object99000:head# 2024-04-29T14:27:10.353 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.353 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601e0cc4:::benchmark_data_smithi194_121512_object90368:head# 2024-04-29T14:27:10.353 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.353 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601e259f:::benchmark_data_smithi194_121512_object26976:head# 2024-04-29T14:27:10.353 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601f1b83:::benchmark_data_smithi194_121512_object69058:head# 2024-04-29T14:27:10.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:601f600d:::benchmark_data_smithi194_121512_object119925:head# 2024-04-29T14:27:10.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6021b1c1:::benchmark_data_smithi194_121512_object86794:head# 2024-04-29T14:27:10.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6022701b:::benchmark_data_smithi194_121512_object115765:head# 2024-04-29T14:27:10.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60227c9a:::benchmark_data_smithi194_121512_object120838:head# 2024-04-29T14:27:10.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6022a1a6:::benchmark_data_smithi194_121512_object93875:head# 2024-04-29T14:27:10.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6022d939:::benchmark_data_smithi194_121512_object92603:head# 2024-04-29T14:27:10.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6023419a:::benchmark_data_smithi194_121512_object41922:head# 2024-04-29T14:27:10.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6023cc94:::benchmark_data_smithi194_121512_object30709:head# 2024-04-29T14:27:10.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6023de63:::benchmark_data_smithi194_121512_object72728:head# 2024-04-29T14:27:10.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.361 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6023ff5a:::benchmark_data_smithi194_121512_object7379:head# 2024-04-29T14:27:10.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60246553:::benchmark_data_smithi194_121512_object113303:head# 2024-04-29T14:27:10.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60252bb7:::benchmark_data_smithi194_121512_object65096:head# 2024-04-29T14:27:10.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60252e1e:::benchmark_data_smithi194_121512_object124130:head# 2024-04-29T14:27:10.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602644bc:::benchmark_data_smithi194_121512_object56286:head# 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6026b460:::benchmark_data_smithi194_121512_object118360:head# 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602719a3:::benchmark_data_smithi194_121512_object71043:head# 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60273a38:::benchmark_data_smithi194_121512_object83910:head# 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60277d2e:::benchmark_data_smithi194_121512_object123537:head# 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6027df36:::benchmark_data_smithi194_121512_object25457:head# 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6028a1a5:::benchmark_data_smithi194_121512_object81426:head# 2024-04-29T14:27:10.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6028c098:::benchmark_data_smithi194_121512_object71238:head# 2024-04-29T14:27:10.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6029f301:::benchmark_data_smithi194_121512_object10067:head# 2024-04-29T14:27:10.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2938748432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:10.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:10 smithi121 ceph-mon[44533]: from='client.36080 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:10.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602aaa54:::benchmark_data_smithi194_121512_object96596:head# 2024-04-29T14:27:10.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602aadb2:::benchmark_data_smithi194_121512_object99279:head# 2024-04-29T14:27:10.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602b19d0:::benchmark_data_smithi194_121512_object87833:head# 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602bcc05:::benchmark_data_smithi194_121512_object38365:head# 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602bd279:::benchmark_data_smithi194_121512_object71703:head# 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602c37ab:::benchmark_data_smithi194_121512_object37715:head# 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602cf058:::benchmark_data_smithi194_121512_object129962:head# 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602d1bbb:::benchmark_data_smithi194_121512_object7326:head# 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602de7ef:::benchmark_data_smithi194_121512_object6415:head# 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602ded4d:::benchmark_data_smithi194_121512_object53889:head# 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602def05:::benchmark_data_smithi194_121512_object43903:head# 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602e54e9:::benchmark_data_smithi194_121512_object116609:head# 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602e65bc:::benchmark_data_smithi194_121512_object51161:head# 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602ece5e:::benchmark_data_smithi194_121512_object48668:head# 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:602f14b9:::benchmark_data_smithi194_121512_object127704:head# 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60309085:::benchmark_data_smithi194_121512_object71950:head# 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6030e435:::benchmark_data_smithi194_121512_object123704:head# 2024-04-29T14:27:10.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60311c96:::benchmark_data_smithi194_121512_object102932:head# 2024-04-29T14:27:10.386 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60312de7:::benchmark_data_smithi194_121512_object119494:head# 2024-04-29T14:27:10.386 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60314d80:::benchmark_data_smithi194_121512_object47377:head# 2024-04-29T14:27:10.386 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6031ff13:::benchmark_data_smithi194_121512_object117718:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6032edae:::benchmark_data_smithi194_121512_object90328:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60335707:::benchmark_data_smithi194_121512_object66207:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6033591e:::benchmark_data_smithi194_121512_object56089:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60336818:::benchmark_data_smithi194_121512_object125856:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60340fbb:::benchmark_data_smithi194_121512_object10494:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60343949:::benchmark_data_smithi194_121512_object53124:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60345a56:::benchmark_data_smithi194_121512_object46420:head# 2024-04-29T14:27:10.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6034adad:::benchmark_data_smithi194_121512_object35407:head# 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6034b087:::benchmark_data_smithi194_121512_object101005:head# 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6036400b:::benchmark_data_smithi194_121512_object132962:head# 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603649c8:::benchmark_data_smithi194_121512_object96620:head# 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6036749b:::benchmark_data_smithi194_121512_object128259:head# 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6036b899:::benchmark_data_smithi194_121512_object26969:head# 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6036c528:::benchmark_data_smithi194_121512_object30200:head# 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603733d6:::benchmark_data_smithi194_121512_object118012:head# 2024-04-29T14:27:10.389 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.389 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6037b033:::benchmark_data_smithi194_121512_object101083:head# 2024-04-29T14:27:10.389 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6037f30e:::benchmark_data_smithi194_121512_object71755:head# 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6037f95a:::benchmark_data_smithi194_121512_object22610:head# 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6038637e:::benchmark_data_smithi194_121512_object94895:head# 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6038d5aa:::benchmark_data_smithi194_121512_object5815:head# 2024-04-29T14:27:10.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6038f1d6:::benchmark_data_smithi194_121512_object29760:head# 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60391419:::benchmark_data_smithi194_121512_object102126:head# 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603982f6:::benchmark_data_smithi194_121512_object22933:head# 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6039b209:::benchmark_data_smithi194_121512_object65081:head# 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6039cda0:::benchmark_data_smithi194_121512_object113982:head# 2024-04-29T14:27:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603a2355:::benchmark_data_smithi194_121512_object64898:head# 2024-04-29T14:27:10.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603abecb:::benchmark_data_smithi194_121512_object70910:head# 2024-04-29T14:27:10.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603adbb7:::benchmark_data_smithi194_121512_object73117:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603ae62c:::benchmark_data_smithi194_121512_object92507:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603bd947:::benchmark_data_smithi194_121512_object120483:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603be836:::benchmark_data_smithi194_121512_object92591:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603c73de:::benchmark_data_smithi194_121512_object108719:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603c7588:::benchmark_data_smithi194_121512_object51339:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603c81ce:::benchmark_data_smithi194_121512_object99852:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603d2050:::benchmark_data_smithi194_121512_object50688:head# 2024-04-29T14:27:10.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603dc609:::benchmark_data_smithi194_121512_object115748:head# 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603e5d39:::benchmark_data_smithi194_121512_object9451:head# 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603f1228:::benchmark_data_smithi194_121512_object78930:head# 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603f25cf:::benchmark_data_smithi194_121512_object34462:head# 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603f37a2:::benchmark_data_smithi194_121512_object9387:head# 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603f4196:::benchmark_data_smithi194_121512_object118040:head# 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:603fff34:::benchmark_data_smithi194_121512_object92380:head# 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60400a01:::benchmark_data_smithi194_121512_object9045:head# 2024-04-29T14:27:10.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60401670:::benchmark_data_smithi194_121512_object24657:head# 2024-04-29T14:27:10.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60402a21:::benchmark_data_smithi194_121512_object31025:head# 2024-04-29T14:27:10.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60406954:::benchmark_data_smithi194_121512_object72109:head# 2024-04-29T14:27:10.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60408026:::benchmark_data_smithi194_121512_object6549:head# 2024-04-29T14:27:10.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.414 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60408980:::benchmark_data_smithi194_121512_object57917:head# 2024-04-29T14:27:10.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60411417:::benchmark_data_smithi194_121512_object98457:head# 2024-04-29T14:27:10.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60415e7c:::benchmark_data_smithi194_121512_object72786:head# 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6041647f:::benchmark_data_smithi194_121512_object14167:head# 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60418c7d:::benchmark_data_smithi194_121512_object93593:head# 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6041b38f:::benchmark_data_smithi194_121512_object112745:head# 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6041b808:::benchmark_data_smithi194_121512_object54183:head# 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6041db43:::benchmark_data_smithi194_121512_object70444:head# 2024-04-29T14:27:10.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60435e15:::benchmark_data_smithi194_121512_object86469:head# 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6043c9a6:::benchmark_data_smithi194_121512_object111667:head# 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6044063c:::benchmark_data_smithi194_121512_object57702:head# 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604421ea:::benchmark_data_smithi194_121512_object5282:head# 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60447130:::benchmark_data_smithi194_121512_object26528:head# 2024-04-29T14:27:10.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6044b7ca:::benchmark_data_smithi194_121512_object20582:head# 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6044d09c:::benchmark_data_smithi194_121512_object24823:head# 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6046e968:::benchmark_data_smithi194_121512_object18595:head# 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604705a8:::benchmark_data_smithi194_121512_object83800:head# 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6047a365:::benchmark_data_smithi194_121512_object40019:head# 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6047b6af:::benchmark_data_smithi194_121512_object75783:head# 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6047e9ef:::benchmark_data_smithi194_121512_object112262:head# 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604867de:::benchmark_data_smithi194_121512_object115718:head# 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6048f64f:::benchmark_data_smithi194_121512_object97939:head# 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604945f0:::benchmark_data_smithi194_121512_object128660:head# 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604980ce:::benchmark_data_smithi194_121512_object96784:head# 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6049eb94:::benchmark_data_smithi194_121512_object53633:head# 2024-04-29T14:27:10.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604af895:::benchmark_data_smithi194_121512_object56009:head# 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604cbb59:::benchmark_data_smithi194_121512_object9484:head# 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604d3e45:::benchmark_data_smithi194_121512_object27996:head# 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604f31a9:::benchmark_data_smithi194_121512_object133282:head# 2024-04-29T14:27:10.424 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:604f860b:::benchmark_data_smithi194_121512_object50887:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6050e208:::benchmark_data_smithi194_121512_object112247:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6051590b:::benchmark_data_smithi194_121512_object96296:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60515c45:::benchmark_data_smithi194_121512_object39152:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6051a015:::benchmark_data_smithi194_121512_object59297:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6051e268:::benchmark_data_smithi194_121512_object41487:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605200b5:::benchmark_data_smithi194_121512_object6914:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60524077:::benchmark_data_smithi194_121512_object71600:head# 2024-04-29T14:27:10.425 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60544b54:::benchmark_data_smithi194_121512_object39767:head# 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605475e1:::benchmark_data_smithi194_121512_object132355:head# 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6054e205:::benchmark_data_smithi194_121512_object58999:head# 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60552f35:::benchmark_data_smithi194_121512_object34364:head# 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6055adfd:::benchmark_data_smithi194_121512_object103207:head# 2024-04-29T14:27:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6055fe87:::benchmark_data_smithi194_121512_object49371:head# 2024-04-29T14:27:10.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60561dc9:::benchmark_data_smithi194_121512_object86843:head# 2024-04-29T14:27:10.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60574b9e:::benchmark_data_smithi194_121512_object100765:head# 2024-04-29T14:27:10.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60580be6:::benchmark_data_smithi194_121512_object111539:head# 2024-04-29T14:27:10.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60582677:::benchmark_data_smithi194_121512_object8966:head# 2024-04-29T14:27:10.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60582ad6:::benchmark_data_smithi194_121512_object31878:head# 2024-04-29T14:27:10.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6059302c:::benchmark_data_smithi194_121512_object38297:head# 2024-04-29T14:27:10.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605982e2:::benchmark_data_smithi194_121512_object93765:head# 2024-04-29T14:27:10.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.433 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6059ce4d:::benchmark_data_smithi194_121512_object125868:head# 2024-04-29T14:27:10.433 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605a0e20:::benchmark_data_smithi194_121512_object132055:head# 2024-04-29T14:27:10.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605a9ea8:::benchmark_data_smithi194_121512_object88173:head# 2024-04-29T14:27:10.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605c7658:::benchmark_data_smithi194_121512_object37338:head# 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605c97ae:::benchmark_data_smithi194_121512_object64205:head# 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605e26ff:::benchmark_data_smithi194_121512_object54518:head# 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605efcff:::benchmark_data_smithi194_121512_object36828:head# 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605f2f7a:::benchmark_data_smithi194_121512_object79755:head# 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605f51ef:::benchmark_data_smithi194_121512_object80992:head# 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:605f6f8b:::benchmark_data_smithi194_121512_object9487:head# 2024-04-29T14:27:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60608483:::benchmark_data_smithi194_121512_object19577:head# 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606122a4:::benchmark_data_smithi194_121512_object74273:head# 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60621065:::benchmark_data_smithi194_121512_object121899:head# 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6062526e:::benchmark_data_smithi194_121512_object42753:head# 2024-04-29T14:27:10.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60635c3b:::benchmark_data_smithi194_121512_object45725:head# 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60637a30:::benchmark_data_smithi194_121512_object132980:head# 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6063863d:::benchmark_data_smithi194_121512_object122131:head# 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6066000d:::benchmark_data_smithi194_121512_object112168:head# 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606603e2:::benchmark_data_smithi194_121512_object34119:head# 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60660805:::benchmark_data_smithi194_121512_object22698:head# 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60663461:::benchmark_data_smithi194_121512_object111987:head# 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60666ecc:::benchmark_data_smithi194_121512_object12343:head# 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60667fe4:::benchmark_data_smithi194_121512_object11068:head# 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6066b7e0:::benchmark_data_smithi194_121512_object7445:head# 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6067681f:::benchmark_data_smithi194_121512_object34066:head# 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60689bcc:::benchmark_data_smithi194_121512_object23027:head# 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6068fb69:::benchmark_data_smithi194_121512_object886:head# 2024-04-29T14:27:10.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60696f25:::benchmark_data_smithi194_121512_object69329:head# 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606a0156:::benchmark_data_smithi194_121512_object19231:head# 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606af595:::benchmark_data_smithi194_121512_object72668:head# 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606b3657:::benchmark_data_smithi194_121512_object64989:head# 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606bb67a:::benchmark_data_smithi194_121512_object35792:head# 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606bcf39:::benchmark_data_smithi194_121512_object27123:head# 2024-04-29T14:27:10.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606c4cc0:::benchmark_data_smithi194_121512_object97670:head# 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606ca393:::benchmark_data_smithi194_121512_object111275:head# 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606e0ef5:::benchmark_data_smithi194_121512_object3281:head# 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606e61b2:::benchmark_data_smithi194_121512_object9681:head# 2024-04-29T14:27:10.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606e7d06:::benchmark_data_smithi194_121512_object13414:head# 2024-04-29T14:27:10.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606ee524:::benchmark_data_smithi194_121512_object39327:head# 2024-04-29T14:27:10.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606fc9f3:::benchmark_data_smithi194_121512_object128751:head# 2024-04-29T14:27:10.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:606ff705:::benchmark_data_smithi194_121512_object76578:head# 2024-04-29T14:27:10.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60723580:::benchmark_data_smithi194_121512_object84677:head# 2024-04-29T14:27:10.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6072d363:::benchmark_data_smithi194_121512_object101476:head# 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6073cf87:::benchmark_data_smithi194_121512_object94190:head# 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6075321a:::benchmark_data_smithi194_121512_object99956:head# 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60757b1f:::benchmark_data_smithi194_121512_object69803:head# 2024-04-29T14:27:10.447 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6075ea7f:::benchmark_data_smithi194_121512_object88628:head# 2024-04-29T14:27:10.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607635d5:::benchmark_data_smithi194_121512_object5326:head# 2024-04-29T14:27:10.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.448 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60764c21:::benchmark_data_smithi194_121512_object101391:head# 2024-04-29T14:27:10.448 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.449 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60765280:::benchmark_data_smithi194_121512_object88964:head# 2024-04-29T14:27:10.449 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6076cfa0:::benchmark_data_smithi194_121512_object88105:head# 2024-04-29T14:27:10.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6077c843:::benchmark_data_smithi194_121512_object124913:head# 2024-04-29T14:27:10.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6077d5d0:::benchmark_data_smithi194_121512_object28624:head# 2024-04-29T14:27:10.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607838a6:::benchmark_data_smithi194_121512_object73055:head# 2024-04-29T14:27:10.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60787bda:::benchmark_data_smithi194_121512_object73393:head# 2024-04-29T14:27:10.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607896e6:::benchmark_data_smithi194_121512_object95767:head# 2024-04-29T14:27:10.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60797acc:::benchmark_data_smithi194_121512_object110:head# 2024-04-29T14:27:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60799bf7:::benchmark_data_smithi194_121512_object107242:head# 2024-04-29T14:27:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6079b9cc:::benchmark_data_smithi194_121512_object65544:head# 2024-04-29T14:27:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607c077c:::benchmark_data_smithi194_121512_object21800:head# 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607c2df2:::benchmark_data_smithi194_121512_object57569:head# 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607dceb2:::benchmark_data_smithi194_121512_object10370:head# 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607ddcc2:::benchmark_data_smithi194_121512_object46122:head# 2024-04-29T14:27:10.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607e2473:::benchmark_data_smithi194_121512_object60450:head# 2024-04-29T14:27:10.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607e3c56:::benchmark_data_smithi194_121512_object89803:head# 2024-04-29T14:27:10.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607e6c20:::benchmark_data_smithi194_121512_object132589:head# 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607e86cc:::benchmark_data_smithi194_121512_object97517:head# 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607e9bd8:::benchmark_data_smithi194_121512_object76177:head# 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607ed2bb:::benchmark_data_smithi194_121512_object31519:head# 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607f19e0:::benchmark_data_smithi194_121512_object10647:head# 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:607fc49e:::benchmark_data_smithi194_121512_object113584:head# 2024-04-29T14:27:10.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60801edb:::benchmark_data_smithi194_121512_object116689:head# 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60804bf6:::benchmark_data_smithi194_121512_object73633:head# 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608104a8:::benchmark_data_smithi194_121512_object69597:head# 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60821635:::benchmark_data_smithi194_121512_object75898:head# 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60826d68:::benchmark_data_smithi194_121512_object128742:head# 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6084d5ae:::benchmark_data_smithi194_121512_object30651:head# 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6085087b:::benchmark_data_smithi194_121512_object125968:head# 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6085552e:::benchmark_data_smithi194_121512_object102648:head# 2024-04-29T14:27:10.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6085a8b8:::benchmark_data_smithi194_121512_object126029:head# 2024-04-29T14:27:10.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60867899:::benchmark_data_smithi194_121512_object36139:head# 2024-04-29T14:27:10.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6086caf8:::benchmark_data_smithi194_121512_object82748:head# 2024-04-29T14:27:10.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6087aa2f:::benchmark_data_smithi194_121512_object31751:head# 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6087f548:::benchmark_data_smithi194_121512_object41768:head# 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60883fb2:::benchmark_data_smithi194_121512_object87778:head# 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608871af:::benchmark_data_smithi194_121512_object103356:head# 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6089b311:::benchmark_data_smithi194_121512_object36147:head# 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608a2678:::benchmark_data_smithi194_121512_object19190:head# 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608a300f:::benchmark_data_smithi194_121512_object6194:head# 2024-04-29T14:27:10.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608a98d0:::benchmark_data_smithi194_121512_object128652:head# 2024-04-29T14:27:10.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608afc43:::benchmark_data_smithi194_121512_object83038:head# 2024-04-29T14:27:10.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608bb840:::benchmark_data_smithi194_121512_object84213:head# 2024-04-29T14:27:10.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608bfe14:::benchmark_data_smithi194_121512_object65284:head# 2024-04-29T14:27:10.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608d3ae3:::benchmark_data_smithi194_121512_object41449:head# 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608d7d90:::benchmark_data_smithi194_121512_object10306:head# 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608d960b:::benchmark_data_smithi194_121512_object63596:head# 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608dd361:::benchmark_data_smithi194_121512_object71515:head# 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608e434a:::benchmark_data_smithi194_121512_object80837:head# 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608e7e5b:::benchmark_data_smithi194_121512_object92306:head# 2024-04-29T14:27:10.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608e8f83:::benchmark_data_smithi194_121512_object56226:head# 2024-04-29T14:27:10.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608ec516:::benchmark_data_smithi194_121512_object69324:head# 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608ed8f9:::benchmark_data_smithi194_121512_object11387:head# 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608ee51c:::benchmark_data_smithi194_121512_object52102:head# 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608f1fe5:::benchmark_data_smithi194_121512_object65216:head# 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:608f95d0:::benchmark_data_smithi194_121512_object67478:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609043e4:::benchmark_data_smithi194_121512_object54825:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6090f9ad:::benchmark_data_smithi194_121512_object22305:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6091034b:::benchmark_data_smithi194_121512_object37519:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60919788:::benchmark_data_smithi194_121512_object102808:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60932e35:::benchmark_data_smithi194_121512_object52685:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609334f9:::benchmark_data_smithi194_121512_object106769:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609344b1:::benchmark_data_smithi194_121512_object60733:head# 2024-04-29T14:27:10.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6094b845:::benchmark_data_smithi194_121512_object91111:head# 2024-04-29T14:27:10.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609521c4:::benchmark_data_smithi194_121512_object77205:head# 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60953afe:::benchmark_data_smithi194_121512_object12218:head# 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6095b424:::benchmark_data_smithi194_121512_object15630:head# 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609635d5:::benchmark_data_smithi194_121512_object129136:head# 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6097ea13:::benchmark_data_smithi194_121512_object61313:head# 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6098099d:::benchmark_data_smithi194_121512_object47701:head# 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6098a99a:::benchmark_data_smithi194_121512_object96672:head# 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60994443:::benchmark_data_smithi194_121512_object129903:head# 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609a1198:::benchmark_data_smithi194_121512_object43937:head# 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609abed9:::benchmark_data_smithi194_121512_object115552:head# 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609b240f:::benchmark_data_smithi194_121512_object53626:head# 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609b25eb:::benchmark_data_smithi194_121512_object121345:head# 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609b948d:::benchmark_data_smithi194_121512_object17338:head# 2024-04-29T14:27:10.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609ec101:::benchmark_data_smithi194_121512_object121347:head# 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609f5013:::benchmark_data_smithi194_121512_object73837:head# 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:609fa3d3:::benchmark_data_smithi194_121512_object36024:head# 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a04623:::benchmark_data_smithi194_121512_object50852:head# 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a07e97:::benchmark_data_smithi194_121512_object93792:head# 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a11cbf:::benchmark_data_smithi194_121512_object124190:head# 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a2437a:::benchmark_data_smithi194_121512_object67561:head# 2024-04-29T14:27:10.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a268a3:::benchmark_data_smithi194_121512_object1457:head# 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a417d0:::benchmark_data_smithi194_121512_object132155:head# 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a4afbe:::benchmark_data_smithi194_121512_object37181:head# 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a4b237:::benchmark_data_smithi194_121512_object46713:head# 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a4c2e6:::benchmark_data_smithi194_121512_object96480:head# 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a4ff0b:::benchmark_data_smithi194_121512_object12190:head# 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a56505:::benchmark_data_smithi194_121512_object53657:head# 2024-04-29T14:27:10.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a569ca:::benchmark_data_smithi194_121512_object66930:head# 2024-04-29T14:27:10.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a56d3c:::benchmark_data_smithi194_121512_object115228:head# 2024-04-29T14:27:10.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a62d36:::benchmark_data_smithi194_121512_object30991:head# 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a75cfd:::benchmark_data_smithi194_121512_object66192:head# 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a8c470:::benchmark_data_smithi194_121512_object15633:head# 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a8cc48:::benchmark_data_smithi194_121512_object7702:head# 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a90615:::benchmark_data_smithi194_121512_object62903:head# 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a9643c:::benchmark_data_smithi194_121512_object116764:head# 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60a9c7f3:::benchmark_data_smithi194_121512_object129354:head# 2024-04-29T14:27:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60aa8907:::benchmark_data_smithi194_121512_object54483:head# 2024-04-29T14:27:10.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60aaa317:::benchmark_data_smithi194_121512_object50975:head# 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ab75db:::benchmark_data_smithi194_121512_object131069:head# 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ab7620:::benchmark_data_smithi194_121512_object105773:head# 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60abb941:::benchmark_data_smithi194_121512_object107828:head# 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ac0dbc:::benchmark_data_smithi194_121512_object44139:head# 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ac1069:::benchmark_data_smithi194_121512_object99299:head# 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60acf79f:::benchmark_data_smithi194_121512_object111291:head# 2024-04-29T14:27:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ad11b7:::benchmark_data_smithi194_121512_object24444:head# 2024-04-29T14:27:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ad5bbd:::benchmark_data_smithi194_121512_object63878:head# 2024-04-29T14:27:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ade9c6:::benchmark_data_smithi194_121512_object18443:head# 2024-04-29T14:27:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60adeefb:::benchmark_data_smithi194_121512_object33712:head# 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ae2146:::benchmark_data_smithi194_121512_object37411:head# 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60aec211:::benchmark_data_smithi194_121512_object50226:head# 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60aecb83:::benchmark_data_smithi194_121512_object85208:head# 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b0779a:::benchmark_data_smithi194_121512_object104865:head# 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b08c60:::benchmark_data_smithi194_121512_object19287:head# 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b09549:::benchmark_data_smithi194_121512_object690:head# 2024-04-29T14:27:10.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b18f5f:::benchmark_data_smithi194_121512_object55752:head# 2024-04-29T14:27:10.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b1be2f:::benchmark_data_smithi194_121512_object45034:head# 2024-04-29T14:27:10.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b1beb5:::benchmark_data_smithi194_121512_object64371:head# 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b249c4:::benchmark_data_smithi194_121512_object106183:head# 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b33aa1:::benchmark_data_smithi194_121512_object111566:head# 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b3c403:::benchmark_data_smithi194_121512_object23702:head# 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b422a3:::benchmark_data_smithi194_121512_object16322:head# 2024-04-29T14:27:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b4ce85:::benchmark_data_smithi194_121512_object24988:head# 2024-04-29T14:27:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b4d3ee:::benchmark_data_smithi194_121512_object31443:head# 2024-04-29T14:27:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b5c1a4:::benchmark_data_smithi194_121512_object115427:head# 2024-04-29T14:27:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b6170c:::benchmark_data_smithi194_121512_object113175:head# 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b74204:::benchmark_data_smithi194_121512_object80898:head# 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b74f8c:::benchmark_data_smithi194_121512_object124044:head# 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b75074:::benchmark_data_smithi194_121512_object89451:head# 2024-04-29T14:27:10.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b78f13:::benchmark_data_smithi194_121512_object42287:head# 2024-04-29T14:27:10.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.517 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b80cc7:::benchmark_data_smithi194_121512_object81673:head# 2024-04-29T14:27:10.517 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b81135:::benchmark_data_smithi194_121512_object62668:head# 2024-04-29T14:27:10.518 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.518 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60b86d79:::benchmark_data_smithi194_121512_object3539:head# 2024-04-29T14:27:10.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ba8428:::benchmark_data_smithi194_121512_object60966:head# 2024-04-29T14:27:10.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.519 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bb1d0d:::benchmark_data_smithi194_121512_object26789:head# 2024-04-29T14:27:10.519 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bb2084:::benchmark_data_smithi194_121512_object32061:head# 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bbc925:::benchmark_data_smithi194_121512_object61062:head# 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bd5bd8:::benchmark_data_smithi194_121512_object75508:head# 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bdcbb9:::benchmark_data_smithi194_121512_object11538:head# 2024-04-29T14:27:10.520 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bdcc77:::benchmark_data_smithi194_121512_object81359:head# 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bddcd6:::benchmark_data_smithi194_121512_object56711:head# 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bebe3d:::benchmark_data_smithi194_121512_object42791:head# 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bf0ba1:::benchmark_data_smithi194_121512_object132185:head# 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60bf8743:::benchmark_data_smithi194_121512_object78357:head# 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c0af7e:::benchmark_data_smithi194_121512_object133031:head# 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c21e3c:::benchmark_data_smithi194_121512_object80922:head# 2024-04-29T14:27:10.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c25b60:::benchmark_data_smithi194_121512_object130003:head# 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c2f22f:::benchmark_data_smithi194_121512_object110964:head# 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c33d2e:::benchmark_data_smithi194_121512_object96338:head# 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c355e8:::benchmark_data_smithi194_121512_object1331:head# 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c3d6f3:::benchmark_data_smithi194_121512_object88922:head# 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c3d764:::benchmark_data_smithi194_121512_object51047:head# 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c3e6bb:::benchmark_data_smithi194_121512_object114700:head# 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c40a42:::benchmark_data_smithi194_121512_object108428:head# 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c41f99:::benchmark_data_smithi194_121512_object74100:head# 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c4f256:::benchmark_data_smithi194_121512_object119679:head# 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c5484c:::benchmark_data_smithi194_121512_object53722:head# 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c5cc06:::benchmark_data_smithi194_121512_object94195:head# 2024-04-29T14:27:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c6017b:::benchmark_data_smithi194_121512_object24857:head# 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c6ec09:::benchmark_data_smithi194_121512_object91058:head# 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c6f62d:::benchmark_data_smithi194_121512_object62365:head# 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c73f6d:::benchmark_data_smithi194_121512_object104135:head# 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c74298:::benchmark_data_smithi194_121512_object46006:head# 2024-04-29T14:27:10.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c7bf57:::benchmark_data_smithi194_121512_object67524:head# 2024-04-29T14:27:10.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c850d5:::benchmark_data_smithi194_121512_object11139:head# 2024-04-29T14:27:10.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c8d6d9:::benchmark_data_smithi194_121512_object84664:head# 2024-04-29T14:27:10.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c90b8c:::benchmark_data_smithi194_121512_object100997:head# 2024-04-29T14:27:10.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c92c1d:::benchmark_data_smithi194_121512_object22693:head# 2024-04-29T14:27:10.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c9768b:::benchmark_data_smithi194_121512_object58976:head# 2024-04-29T14:27:10.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60c9e60f:::benchmark_data_smithi194_121512_object94953:head# 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ca6f3a:::benchmark_data_smithi194_121512_object88467:head# 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cb3b6d:::benchmark_data_smithi194_121512_object118373:head# 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cb92d8:::benchmark_data_smithi194_121512_object60850:head# 2024-04-29T14:27:10.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cbe3e0:::benchmark_data_smithi194_121512_object49558:head# 2024-04-29T14:27:10.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cc7157:::benchmark_data_smithi194_121512_object59042:head# 2024-04-29T14:27:10.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cc990a:::benchmark_data_smithi194_121512_object127827:head# 2024-04-29T14:27:10.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cdbc6b:::benchmark_data_smithi194_121512_object68:head# 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cdbf44:::benchmark_data_smithi194_121512_object105369:head# 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ce1adb:::benchmark_data_smithi194_121512_object127354:head# 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ce1e60:::benchmark_data_smithi194_121512_object13938:head# 2024-04-29T14:27:10.544 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ce9322:::benchmark_data_smithi194_121512_object48649:head# 2024-04-29T14:27:10.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ceb8de:::benchmark_data_smithi194_121512_object30072:head# 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cee45f:::benchmark_data_smithi194_121512_object126619:head# 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cfa323:::benchmark_data_smithi194_121512_object43404:head# 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60cfc8ca:::benchmark_data_smithi194_121512_object104285:head# 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d0cfed:::benchmark_data_smithi194_121512_object119499:head# 2024-04-29T14:27:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d15d14:::benchmark_data_smithi194_121512_object10613:head# 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d20aca:::benchmark_data_smithi194_121512_object39205:head# 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d28f26:::benchmark_data_smithi194_121512_object15758:head# 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d458d1:::benchmark_data_smithi194_121512_object125663:head# 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d4e082:::benchmark_data_smithi194_121512_object125347:head# 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d5147c:::benchmark_data_smithi194_121512_object80711:head# 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d54066:::benchmark_data_smithi194_121512_object11755:head# 2024-04-29T14:27:10.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d54a09:::benchmark_data_smithi194_121512_object102854:head# 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d58919:::benchmark_data_smithi194_121512_object78815:head# 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d595ab:::benchmark_data_smithi194_121512_object131982:head# 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d5e118:::benchmark_data_smithi194_121512_object110715:head# 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d62005:::benchmark_data_smithi194_121512_object132506:head# 2024-04-29T14:27:10.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d6373f:::benchmark_data_smithi194_121512_object94840:head# 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d6b2a0:::benchmark_data_smithi194_121512_object79297:head# 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d72625:::benchmark_data_smithi194_121512_object112573:head# 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d735c7:::benchmark_data_smithi194_121512_object51502:head# 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d739c8:::benchmark_data_smithi194_121512_object60381:head# 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d86fcc:::benchmark_data_smithi194_121512_object74868:head# 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d88b7c:::benchmark_data_smithi194_121512_object75637:head# 2024-04-29T14:27:10.554 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60d8aa71:::benchmark_data_smithi194_121512_object12134:head# 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60da45ad:::benchmark_data_smithi194_121512_object121500:head# 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60da52ea:::benchmark_data_smithi194_121512_object51371:head# 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60db8d5c:::benchmark_data_smithi194_121512_object56539:head# 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60dce35e:::benchmark_data_smithi194_121512_object89912:head# 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60dce6e8:::benchmark_data_smithi194_121512_object74294:head# 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60dd26e5:::benchmark_data_smithi194_121512_object113631:head# 2024-04-29T14:27:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60dddcb5:::benchmark_data_smithi194_121512_object5521:head# 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60de355a:::benchmark_data_smithi194_121512_object96589:head# 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60de79fa:::benchmark_data_smithi194_121512_object113256:head# 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60dfa9ec:::benchmark_data_smithi194_121512_object69093:head# 2024-04-29T14:27:10.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60dfb8c5:::benchmark_data_smithi194_121512_object112606:head# 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e035f2:::benchmark_data_smithi194_121512_object101635:head# 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e0a073:::benchmark_data_smithi194_121512_object8919:head# 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e0c09c:::benchmark_data_smithi194_121512_object101776:head# 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e0ee7f:::benchmark_data_smithi194_121512_object81208:head# 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e24203:::benchmark_data_smithi194_121512_object68004:head# 2024-04-29T14:27:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e29b8e:::benchmark_data_smithi194_121512_object98492:head# 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e29bba:::benchmark_data_smithi194_121512_object22849:head# 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e2f27f:::benchmark_data_smithi194_121512_object42392:head# 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e3bf9b:::benchmark_data_smithi194_121512_object117278:head# 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e3f8a8:::benchmark_data_smithi194_121512_object103729:head# 2024-04-29T14:27:10.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e437e3:::benchmark_data_smithi194_121512_object7150:head# 2024-04-29T14:27:10.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e45090:::benchmark_data_smithi194_121512_object110136:head# 2024-04-29T14:27:10.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e5349d:::benchmark_data_smithi194_121512_object52668:head# 2024-04-29T14:27:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e54b2e:::benchmark_data_smithi194_121512_object112296:head# 2024-04-29T14:27:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e649e1:::benchmark_data_smithi194_121512_object111519:head# 2024-04-29T14:27:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e656b3:::benchmark_data_smithi194_121512_object47233:head# 2024-04-29T14:27:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e66cc3:::benchmark_data_smithi194_121512_object39247:head# 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e75c5d:::benchmark_data_smithi194_121512_object89066:head# 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e8b580:::benchmark_data_smithi194_121512_object47921:head# 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e8c5b9:::benchmark_data_smithi194_121512_object131832:head# 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60e95542:::benchmark_data_smithi194_121512_object9592:head# 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ec164c:::benchmark_data_smithi194_121512_object46746:head# 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ec9bf5:::benchmark_data_smithi194_121512_object53821:head# 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ecb6d4:::benchmark_data_smithi194_121512_object48747:head# 2024-04-29T14:27:10.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60edc851:::benchmark_data_smithi194_121512_object37901:head# 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ee3549:::benchmark_data_smithi194_121512_object48201:head# 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ee5f44:::benchmark_data_smithi194_121512_object68250:head# 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60eec1c3:::benchmark_data_smithi194_121512_object62051:head# 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60efb8a3:::benchmark_data_smithi194_121512_object104566:head# 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f03709:::benchmark_data_smithi194_121512_object46596:head# 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f1be93:::benchmark_data_smithi194_121512_object112832:head# 2024-04-29T14:27:10.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f2993a:::benchmark_data_smithi194_121512_object21405:head# 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f2ffda:::benchmark_data_smithi194_121512_object37070:head# 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f3348e:::benchmark_data_smithi194_121512_object111320:head# 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f3bb70:::benchmark_data_smithi194_121512_object34623:head# 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f4a4d6:::benchmark_data_smithi194_121512_object82057:head# 2024-04-29T14:27:10.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f6d0c2:::benchmark_data_smithi194_121512_object16046:head# 2024-04-29T14:27:10.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f7ca46:::benchmark_data_smithi194_121512_object21828:head# 2024-04-29T14:27:10.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f7ec03:::benchmark_data_smithi194_121512_object95954:head# 2024-04-29T14:27:10.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.568 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f90377:::benchmark_data_smithi194_121512_object124476:head# 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f94c6c:::benchmark_data_smithi194_121512_object44998:head# 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f96a6d:::benchmark_data_smithi194_121512_object12891:head# 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f9a57c:::benchmark_data_smithi194_121512_object28598:head# 2024-04-29T14:27:10.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60f9c802:::benchmark_data_smithi194_121512_object3644:head# 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60fb3a2a:::benchmark_data_smithi194_121512_object123553:head# 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60fc4bf1:::benchmark_data_smithi194_121512_object98672:head# 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60fc5ef0:::benchmark_data_smithi194_121512_object10813:head# 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60fca358:::benchmark_data_smithi194_121512_object78787:head# 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60fd1648:::benchmark_data_smithi194_121512_object21496:head# 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ff88ad:::benchmark_data_smithi194_121512_object91468:head# 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:60ffa60e:::benchmark_data_smithi194_121512_object85399:head# 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6100353a:::benchmark_data_smithi194_121512_object16390:head# 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61007484:::benchmark_data_smithi194_121512_object92209:head# 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6100bb1f:::benchmark_data_smithi194_121512_object132914:head# 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6100eb9d:::benchmark_data_smithi194_121512_object77891:head# 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610110e1:::benchmark_data_smithi194_121512_object120957:head# 2024-04-29T14:27:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2938748432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:10.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:10 smithi194 ceph-mon[44089]: from='client.36080 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:10.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610195ce:::benchmark_data_smithi194_121512_object44718:head# 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6101b1b7:::benchmark_data_smithi194_121512_object77682:head# 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6101eddd:::benchmark_data_smithi194_121512_object101080:head# 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6102077c:::benchmark_data_smithi194_121512_object77101:head# 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610324d8:::benchmark_data_smithi194_121512_object82826:head# 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6105a913:::benchmark_data_smithi194_121512_object9072:head# 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6106cd5c:::benchmark_data_smithi194_121512_object70704:head# 2024-04-29T14:27:10.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6106d661:::benchmark_data_smithi194_121512_object35550:head# 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61075f77:::benchmark_data_smithi194_121512_object118233:head# 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61078eb7:::benchmark_data_smithi194_121512_object30054:head# 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6107bd22:::benchmark_data_smithi194_121512_object126674:head# 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6107eb09:::benchmark_data_smithi194_121512_object57356:head# 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610a590f:::benchmark_data_smithi194_121512_object132899:head# 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610a6fc5:::benchmark_data_smithi194_121512_object20101:head# 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610ab1f0:::benchmark_data_smithi194_121512_object91995:head# 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610b320c:::benchmark_data_smithi194_121512_object36732:head# 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610b6688:::benchmark_data_smithi194_121512_object64502:head# 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610b6990:::benchmark_data_smithi194_121512_object95387:head# 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610c552b:::benchmark_data_smithi194_121512_object90316:head# 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610c99ae:::benchmark_data_smithi194_121512_object41194:head# 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610ccb43:::benchmark_data_smithi194_121512_object40440:head# 2024-04-29T14:27:10.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610d408f:::benchmark_data_smithi194_121512_object129130:head# 2024-04-29T14:27:10.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610dd529:::benchmark_data_smithi194_121512_object109054:head# 2024-04-29T14:27:10.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610de88d:::benchmark_data_smithi194_121512_object7931:head# 2024-04-29T14:27:10.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610f097d:::benchmark_data_smithi194_121512_object82606:head# 2024-04-29T14:27:10.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:610ffeaf:::benchmark_data_smithi194_121512_object80399:head# 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61100468:::benchmark_data_smithi194_121512_object100117:head# 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61101782:::benchmark_data_smithi194_121512_object118683:head# 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6110cbb3:::benchmark_data_smithi194_121512_object72558:head# 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611170dc:::benchmark_data_smithi194_121512_object99137:head# 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6111fc40:::benchmark_data_smithi194_121512_object42484:head# 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6111fe56:::benchmark_data_smithi194_121512_object80844:head# 2024-04-29T14:27:10.583 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611202fa:::benchmark_data_smithi194_121512_object52623:head# 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6112790c:::benchmark_data_smithi194_121512_object14411:head# 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61136a95:::benchmark_data_smithi194_121512_object41507:head# 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61138fb3:::benchmark_data_smithi194_121512_object20504:head# 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6113d9bd:::benchmark_data_smithi194_121512_object13829:head# 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611409ce:::benchmark_data_smithi194_121512_object39926:head# 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61149ca4:::benchmark_data_smithi194_121512_object1758:head# 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6114d68c:::benchmark_data_smithi194_121512_object92409:head# 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6114fff3:::benchmark_data_smithi194_121512_object55941:head# 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6115a044:::benchmark_data_smithi194_121512_object21978:head# 2024-04-29T14:27:10.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6116aa05:::benchmark_data_smithi194_121512_object122666:head# 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61170103:::benchmark_data_smithi194_121512_object19394:head# 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61173395:::benchmark_data_smithi194_121512_object14194:head# 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61173462:::benchmark_data_smithi194_121512_object75543:head# 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61179624:::benchmark_data_smithi194_121512_object86856:head# 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611a7375:::benchmark_data_smithi194_121512_object124669:head# 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611b0a68:::benchmark_data_smithi194_121512_object70261:head# 2024-04-29T14:27:10.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611c6e74:::benchmark_data_smithi194_121512_object82342:head# 2024-04-29T14:27:10.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611d52d5:::benchmark_data_smithi194_121512_object57793:head# 2024-04-29T14:27:10.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611ddcda:::benchmark_data_smithi194_121512_object19552:head# 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611dfd24:::benchmark_data_smithi194_121512_object7663:head# 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611ea034:::benchmark_data_smithi194_121512_object111968:head# 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611f15a0:::benchmark_data_smithi194_121512_object117143:head# 2024-04-29T14:27:10.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:611ff2cc:::benchmark_data_smithi194_121512_object73612:head# 2024-04-29T14:27:10.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612016dd:::benchmark_data_smithi194_121512_object37858:head# 2024-04-29T14:27:10.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612075fe:::benchmark_data_smithi194_121512_object128493:head# 2024-04-29T14:27:10.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6120e0ef:::benchmark_data_smithi194_121512_object107949:head# 2024-04-29T14:27:10.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6121d48a:::benchmark_data_smithi194_121512_object34015:head# 2024-04-29T14:27:10.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61222f94:::benchmark_data_smithi194_121512_object82019:head# 2024-04-29T14:27:10.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6122427b:::benchmark_data_smithi194_121512_object27969:head# 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61226bb1:::benchmark_data_smithi194_121512_object47490:head# 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61232455:::benchmark_data_smithi194_121512_object50751:head# 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61232b86:::benchmark_data_smithi194_121512_object13914:head# 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6123524d:::benchmark_data_smithi194_121512_object112196:head# 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6123c6a6:::benchmark_data_smithi194_121512_object55883:head# 2024-04-29T14:27:10.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6123cf4c:::benchmark_data_smithi194_121512_object2351:head# 2024-04-29T14:27:10.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6123f33b:::benchmark_data_smithi194_121512_object43916:head# 2024-04-29T14:27:10.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61241349:::benchmark_data_smithi194_121512_object12081:head# 2024-04-29T14:27:10.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61241bb3:::benchmark_data_smithi194_121512_object32541:head# 2024-04-29T14:27:10.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6124b2a6:::benchmark_data_smithi194_121512_object15181:head# 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61254977:::benchmark_data_smithi194_121512_object67452:head# 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6125a0c6:::benchmark_data_smithi194_121512_object65286:head# 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6125dfd7:::benchmark_data_smithi194_121512_object32518:head# 2024-04-29T14:27:10.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2938748432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:10.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:10 smithi040 ceph-mon[39581]: from='client.36080 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61263995:::benchmark_data_smithi194_121512_object127156:head# 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612693b7:::benchmark_data_smithi194_121512_object121066:head# 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6127c10f:::benchmark_data_smithi194_121512_object123895:head# 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612811a2:::benchmark_data_smithi194_121512_object25739:head# 2024-04-29T14:27:10.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61281ef5:::benchmark_data_smithi194_121512_object92550:head# 2024-04-29T14:27:10.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61284199:::benchmark_data_smithi194_121512_object65826:head# 2024-04-29T14:27:10.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6129f9ea:::benchmark_data_smithi194_121512_object34371:head# 2024-04-29T14:27:10.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612a961e:::benchmark_data_smithi194_121512_object48110:head# 2024-04-29T14:27:10.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612ae5ed:::benchmark_data_smithi194_121512_object7009:head# 2024-04-29T14:27:10.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.616 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612b0401:::benchmark_data_smithi194_121512_object83304:head# 2024-04-29T14:27:10.616 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612b7e01:::benchmark_data_smithi194_121512_object9927:head# 2024-04-29T14:27:10.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612bcb78:::benchmark_data_smithi194_121512_object31282:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612bd829:::benchmark_data_smithi194_121512_object131957:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612befe2:::benchmark_data_smithi194_121512_object3514:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612c4b9b:::benchmark_data_smithi194_121512_object11853:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612c84d9:::benchmark_data_smithi194_121512_object844:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612ca6f4:::benchmark_data_smithi194_121512_object119465:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612ce852:::benchmark_data_smithi194_121512_object44768:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612d5f52:::benchmark_data_smithi194_121512_object105502:head# 2024-04-29T14:27:10.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612e1f3a:::benchmark_data_smithi194_121512_object24538:head# 2024-04-29T14:27:10.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612e5261:::benchmark_data_smithi194_121512_object100373:head# 2024-04-29T14:27:10.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612e654d:::benchmark_data_smithi194_121512_object86752:head# 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612e750c:::benchmark_data_smithi194_121512_object128105:head# 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612e9761:::benchmark_data_smithi194_121512_object8879:head# 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612f82a6:::benchmark_data_smithi194_121512_object133048:head# 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612fa1ea:::benchmark_data_smithi194_121512_object86540:head# 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612fb788:::benchmark_data_smithi194_121512_object97778:head# 2024-04-29T14:27:10.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:612fe535:::benchmark_data_smithi194_121512_object110153:head# 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61300674:::benchmark_data_smithi194_121512_object9508:head# 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61303100:::benchmark_data_smithi194_121512_object84981:head# 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61308d38:::benchmark_data_smithi194_121512_object44709:head# 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6130a034:::benchmark_data_smithi194_121512_object8432:head# 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613156da:::benchmark_data_smithi194_121512_object118254:head# 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61322fea:::benchmark_data_smithi194_121512_object124204:head# 2024-04-29T14:27:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6132818d:::benchmark_data_smithi194_121512_object124984:head# 2024-04-29T14:27:10.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61329f9f:::benchmark_data_smithi194_121512_object70993:head# 2024-04-29T14:27:10.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6132f50b:::benchmark_data_smithi194_121512_object116028:head# 2024-04-29T14:27:10.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61330b64:::benchmark_data_smithi194_121512_object129685:head# 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613312f3:::benchmark_data_smithi194_121512_object64087:head# 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61334d0c:::benchmark_data_smithi194_121512_object127356:head# 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6133b5c2:::benchmark_data_smithi194_121512_object67161:head# 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6134df02:::benchmark_data_smithi194_121512_object108138:head# 2024-04-29T14:27:10.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6134fcf4:::benchmark_data_smithi194_121512_object122722:head# 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6135216b:::benchmark_data_smithi194_121512_object86691:head# 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6135d3d3:::benchmark_data_smithi194_121512_object82163:head# 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6135dde4:::benchmark_data_smithi194_121512_object8542:head# 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6135f5ad:::benchmark_data_smithi194_121512_object84159:head# 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6137348a:::benchmark_data_smithi194_121512_object121308:head# 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6137b1c0:::benchmark_data_smithi194_121512_object35771:head# 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6137cd98:::benchmark_data_smithi194_121512_object11785:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6137ce15:::benchmark_data_smithi194_121512_object45720:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6137dd9a:::benchmark_data_smithi194_121512_object48631:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6137e51b:::benchmark_data_smithi194_121512_object79065:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6137e6b1:::benchmark_data_smithi194_121512_object89166:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613800d7:::benchmark_data_smithi194_121512_object120646:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613815a6:::benchmark_data_smithi194_121512_object120453:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61399bbc:::benchmark_data_smithi194_121512_object115633:head# 2024-04-29T14:27:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613a412e:::benchmark_data_smithi194_121512_object62053:head# 2024-04-29T14:27:10.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613a5769:::benchmark_data_smithi194_121512_object111293:head# 2024-04-29T14:27:10.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613a9244:::benchmark_data_smithi194_121512_object119628:head# 2024-04-29T14:27:10.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613a97e9:::benchmark_data_smithi194_121512_object87119:head# 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613b83e9:::benchmark_data_smithi194_121512_object118244:head# 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613b97fa:::benchmark_data_smithi194_121512_object102694:head# 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613e125f:::benchmark_data_smithi194_121512_object99708:head# 2024-04-29T14:27:10.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613e1c9f:::benchmark_data_smithi194_121512_object20623:head# 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613e8b5d:::benchmark_data_smithi194_121512_object110755:head# 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613ee82c:::benchmark_data_smithi194_121512_object40571:head# 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613f36d0:::benchmark_data_smithi194_121512_object73450:head# 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613fe3d2:::benchmark_data_smithi194_121512_object10688:head# 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:613ff546:::benchmark_data_smithi194_121512_object114261:head# 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61408e70:::benchmark_data_smithi194_121512_object124425:head# 2024-04-29T14:27:10.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61411643:::benchmark_data_smithi194_121512_object106336:head# 2024-04-29T14:27:10.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6141b428:::benchmark_data_smithi194_121512_object51375:head# 2024-04-29T14:27:10.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6141c37a:::benchmark_data_smithi194_121512_object27795:head# 2024-04-29T14:27:10.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61420f9c:::benchmark_data_smithi194_121512_object47846:head# 2024-04-29T14:27:10.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614243e1:::benchmark_data_smithi194_121512_object51426:head# 2024-04-29T14:27:10.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6142b64c:::benchmark_data_smithi194_121512_object88792:head# 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61433c6a:::benchmark_data_smithi194_121512_object18672:head# 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614368c6:::benchmark_data_smithi194_121512_object120740:head# 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6145ff8b:::benchmark_data_smithi194_121512_object1053:head# 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61460845:::benchmark_data_smithi194_121512_object209:head# 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6146a13e:::benchmark_data_smithi194_121512_object21081:head# 2024-04-29T14:27:10.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6147616f:::benchmark_data_smithi194_121512_object111795:head# 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6147b19d:::benchmark_data_smithi194_121512_object126765:head# 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61485250:::benchmark_data_smithi194_121512_object39627:head# 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6148e5ab:::benchmark_data_smithi194_121512_object57621:head# 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61490e79:::benchmark_data_smithi194_121512_object26743:head# 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61492b3c:::benchmark_data_smithi194_121512_object94487:head# 2024-04-29T14:27:10.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614a907d:::benchmark_data_smithi194_121512_object115539:head# 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614b5a19:::benchmark_data_smithi194_121512_object62249:head# 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614c7d44:::benchmark_data_smithi194_121512_object34181:head# 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614d0ce2:::benchmark_data_smithi194_121512_object105786:head# 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614d12f9:::benchmark_data_smithi194_121512_object4600:head# 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614e9fac:::benchmark_data_smithi194_121512_object68502:head# 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614ee247:::benchmark_data_smithi194_121512_object122348:head# 2024-04-29T14:27:10.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614fa7dd:::benchmark_data_smithi194_121512_object73882:head# 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614fb61f:::benchmark_data_smithi194_121512_object19224:head# 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614fc236:::benchmark_data_smithi194_121512_object8208:head# 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:614fdfb4:::benchmark_data_smithi194_121512_object123413:head# 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61500d8c:::benchmark_data_smithi194_121512_object116055:head# 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61505e41:::benchmark_data_smithi194_121512_object5195:head# 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6150c2ff:::benchmark_data_smithi194_121512_object75208:head# 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6150ca7e:::benchmark_data_smithi194_121512_object124070:head# 2024-04-29T14:27:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61513a10:::benchmark_data_smithi194_121512_object83197:head# 2024-04-29T14:27:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61519920:::benchmark_data_smithi194_121512_object36691:head# 2024-04-29T14:27:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61536135:::benchmark_data_smithi194_121512_object55074:head# 2024-04-29T14:27:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61538619:::benchmark_data_smithi194_121512_object16232:head# 2024-04-29T14:27:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61544991:::benchmark_data_smithi194_121512_object119566:head# 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61551bda:::benchmark_data_smithi194_121512_object116456:head# 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61570f3f:::benchmark_data_smithi194_121512_object123696:head# 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615733e2:::benchmark_data_smithi194_121512_object30238:head# 2024-04-29T14:27:10.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615828b3:::benchmark_data_smithi194_121512_object9007:head# 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6158893a:::benchmark_data_smithi194_121512_object20392:head# 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6158d276:::benchmark_data_smithi194_121512_object355:head# 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6158fe25:::benchmark_data_smithi194_121512_object76610:head# 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615923f7:::benchmark_data_smithi194_121512_object112313:head# 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61594d93:::benchmark_data_smithi194_121512_object85467:head# 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615957ef:::benchmark_data_smithi194_121512_object8104:head# 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6159b3a0:::benchmark_data_smithi194_121512_object96626:head# 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615a2836:::benchmark_data_smithi194_121512_object101018:head# 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615dc1d6:::benchmark_data_smithi194_121512_object23296:head# 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615e6d3b:::benchmark_data_smithi194_121512_object67192:head# 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615edfa4:::benchmark_data_smithi194_121512_object21605:head# 2024-04-29T14:27:10.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615f5528:::benchmark_data_smithi194_121512_object11135:head# 2024-04-29T14:27:10.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:615f64da:::benchmark_data_smithi194_121512_object112600:head# 2024-04-29T14:27:10.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61601215:::benchmark_data_smithi194_121512_object87932:head# 2024-04-29T14:27:10.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.650 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61602df4:::benchmark_data_smithi194_121512_object101572:head# 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6160cb7f:::benchmark_data_smithi194_121512_object26955:head# 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61616e98:::benchmark_data_smithi194_121512_object68332:head# 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616186be:::benchmark_data_smithi194_121512_object78083:head# 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61619a42:::benchmark_data_smithi194_121512_object68153:head# 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6163e29a:::benchmark_data_smithi194_121512_object130135:head# 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61649f77:::benchmark_data_smithi194_121512_object51252:head# 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6165f7e3:::benchmark_data_smithi194_121512_object125587:head# 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61663fe6:::benchmark_data_smithi194_121512_object93071:head# 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61667706:::benchmark_data_smithi194_121512_object22311:head# 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61667e92:::benchmark_data_smithi194_121512_object115930:head# 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6166b7bf:::benchmark_data_smithi194_121512_object89753:head# 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61694720:::benchmark_data_smithi194_121512_object3255:head# 2024-04-29T14:27:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616a168e:::benchmark_data_smithi194_121512_object109529:head# 2024-04-29T14:27:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616a987e:::benchmark_data_smithi194_121512_object129617:head# 2024-04-29T14:27:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:10.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616aa055:::benchmark_data_smithi194_121512_object42006:head# 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616c60e4:::benchmark_data_smithi194_121512_object127415:head# 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616cb148:::benchmark_data_smithi194_121512_object40183:head# 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616d6c7e:::benchmark_data_smithi194_121512_object39250:head# 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616dd8d6:::benchmark_data_smithi194_121512_object34593:head# 2024-04-29T14:27:10.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616ea03e:::benchmark_data_smithi194_121512_object95024:head# 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616edaeb:::benchmark_data_smithi194_121512_object68112:head# 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616f0ce3:::benchmark_data_smithi194_121512_object9950:head# 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616f22a0:::benchmark_data_smithi194_121512_object77777:head# 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:616f53a5:::benchmark_data_smithi194_121512_object53959:head# 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617039ca:::benchmark_data_smithi194_121512_object39147:head# 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6170af8b:::benchmark_data_smithi194_121512_object49153:head# 2024-04-29T14:27:10.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61712540:::benchmark_data_smithi194_121512_object131987:head# 2024-04-29T14:27:10.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6171473c:::benchmark_data_smithi194_121512_object97995:head# 2024-04-29T14:27:10.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617179d8:::benchmark_data_smithi194_121512_object53714:head# 2024-04-29T14:27:10.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6171d5fd:::benchmark_data_smithi194_121512_object125459:head# 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6172c646:::benchmark_data_smithi194_121512_object109394:head# 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61730ee2:::benchmark_data_smithi194_121512_object119391:head# 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6173d67b:::benchmark_data_smithi194_121512_object10934:head# 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61746772:::benchmark_data_smithi194_121512_object91949:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6174f532:::benchmark_data_smithi194_121512_object106290:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617505c7:::benchmark_data_smithi194_121512_object91092:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61750902:::benchmark_data_smithi194_121512_object14547:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6175337f:::benchmark_data_smithi194_121512_object124077:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61763a22:::benchmark_data_smithi194_121512_object74120:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6176846f:::benchmark_data_smithi194_121512_object46119:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6177deb2:::benchmark_data_smithi194_121512_object132518:head# 2024-04-29T14:27:10.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61782626:::benchmark_data_smithi194_121512_object46484:head# 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61782706:::benchmark_data_smithi194_121512_object130058:head# 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61783abb:::benchmark_data_smithi194_121512_object28552:head# 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61784712:::benchmark_data_smithi194_121512_object126474:head# 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61784ecb:::benchmark_data_smithi194_121512_object106307:head# 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61787fcd:::benchmark_data_smithi194_121512_object74207:head# 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6179d36b:::benchmark_data_smithi194_121512_object27374:head# 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617a4338:::benchmark_data_smithi194_121512_object92544:head# 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617ae57b:::benchmark_data_smithi194_121512_object68272:head# 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617af5b1:::benchmark_data_smithi194_121512_object89938:head# 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617b38ed:::benchmark_data_smithi194_121512_object26896:head# 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617b7488:::benchmark_data_smithi194_121512_object36028:head# 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617c6058:::benchmark_data_smithi194_121512_object7100:head# 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617c62e8:::benchmark_data_smithi194_121512_object73501:head# 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617ca6b4:::benchmark_data_smithi194_121512_object110111:head# 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617cab8b:::benchmark_data_smithi194_121512_object21596:head# 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617ce145:::benchmark_data_smithi194_121512_object110343:head# 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617d83b3:::benchmark_data_smithi194_121512_object94493:head# 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617edc91:::benchmark_data_smithi194_121512_object123700:head# 2024-04-29T14:27:10.668 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617f7654:::benchmark_data_smithi194_121512_object128507:head# 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:617fd8a1:::benchmark_data_smithi194_121512_object310:head# 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61801654:::benchmark_data_smithi194_121512_object122588:head# 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61803ae1:::benchmark_data_smithi194_121512_object57204:head# 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61812003:::benchmark_data_smithi194_121512_object114958:head# 2024-04-29T14:27:10.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61820e09:::benchmark_data_smithi194_121512_object111525:head# 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61834240:::benchmark_data_smithi194_121512_object133019:head# 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61835b98:::benchmark_data_smithi194_121512_object15523:head# 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6184e52d:::benchmark_data_smithi194_121512_object19207:head# 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6186562c:::benchmark_data_smithi194_121512_object106961:head# 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6187b5d0:::benchmark_data_smithi194_121512_object18474:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61882280:::benchmark_data_smithi194_121512_object73345:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61882dec:::benchmark_data_smithi194_121512_object99466:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61886628:::benchmark_data_smithi194_121512_object78792:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61888597:::benchmark_data_smithi194_121512_object6805:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61888691:::benchmark_data_smithi194_121512_object92562:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618887c3:::benchmark_data_smithi194_121512_object71035:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61889954:::benchmark_data_smithi194_121512_object4421:head# 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.672 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61890cd7:::benchmark_data_smithi194_121512_object25264:head# 2024-04-29T14:27:10.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61891f9f:::benchmark_data_smithi194_121512_object123861:head# 2024-04-29T14:27:10.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618961dc:::benchmark_data_smithi194_121512_object60261:head# 2024-04-29T14:27:10.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6189c5b5:::benchmark_data_smithi194_121512_object5899:head# 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618b9c49:::benchmark_data_smithi194_121512_object39110:head# 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618c25f1:::benchmark_data_smithi194_121512_object119804:head# 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618c2faa:::benchmark_data_smithi194_121512_object125232:head# 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618c619e:::benchmark_data_smithi194_121512_object111776:head# 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618c8f98:::benchmark_data_smithi194_121512_object10245:head# 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618cfd56:::benchmark_data_smithi194_121512_object8426:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618d0dcc:::benchmark_data_smithi194_121512_object51326:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618dc2d9:::benchmark_data_smithi194_121512_object32461:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618ece4d:::benchmark_data_smithi194_121512_object55603:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618efea0:::benchmark_data_smithi194_121512_object37498:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618f5baa:::benchmark_data_smithi194_121512_object20154:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618f9a51:::benchmark_data_smithi194_121512_object34743:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618fdbfa:::benchmark_data_smithi194_121512_object64683:head# 2024-04-29T14:27:10.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:618fddf4:::benchmark_data_smithi194_121512_object15658:head# 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61903cd0:::benchmark_data_smithi194_121512_object28724:head# 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6191c0e4:::benchmark_data_smithi194_121512_object79007:head# 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6191cd44:::benchmark_data_smithi194_121512_object41667:head# 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.680 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6192849c:::benchmark_data_smithi194_121512_object81221:head# 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61929874:::benchmark_data_smithi194_121512_object104225:head# 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6193005c:::benchmark_data_smithi194_121512_object61298:head# 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619336ff:::benchmark_data_smithi194_121512_object123551:head# 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6193524d:::benchmark_data_smithi194_121512_object127970:head# 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6194a92e:::benchmark_data_smithi194_121512_object29021:head# 2024-04-29T14:27:10.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61958865:::benchmark_data_smithi194_121512_object17435:head# 2024-04-29T14:27:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6195f12b:::benchmark_data_smithi194_121512_object52919:head# 2024-04-29T14:27:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619620fc:::benchmark_data_smithi194_121512_object70533:head# 2024-04-29T14:27:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61969168:::benchmark_data_smithi194_121512_object49069:head# 2024-04-29T14:27:10.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61969ee6:::benchmark_data_smithi194_121512_object118258:head# 2024-04-29T14:27:10.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61971142:::benchmark_data_smithi194_121512_object46268:head# 2024-04-29T14:27:10.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619725e9:::benchmark_data_smithi194_121512_object22174:head# 2024-04-29T14:27:10.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6197ebe9:::benchmark_data_smithi194_121512_object9546:head# 2024-04-29T14:27:10.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.690 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619861f0:::benchmark_data_smithi194_121512_object63443:head# 2024-04-29T14:27:10.690 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6199124e:::benchmark_data_smithi194_121512_object86399:head# 2024-04-29T14:27:10.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61997841:::benchmark_data_smithi194_121512_object72967:head# 2024-04-29T14:27:10.691 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.691 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61999a73:::benchmark_data_smithi194_121512_object18718:head# 2024-04-29T14:27:10.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6199d0e6:::benchmark_data_smithi194_121512_object74275:head# 2024-04-29T14:27:10.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6199d176:::benchmark_data_smithi194_121512_object115195:head# 2024-04-29T14:27:10.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619ae05f:::benchmark_data_smithi194_121512_object126114:head# 2024-04-29T14:27:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619afae2:::benchmark_data_smithi194_121512_object33864:head# 2024-04-29T14:27:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619d10ea:::benchmark_data_smithi194_121512_object127007:head# 2024-04-29T14:27:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619d29e2:::benchmark_data_smithi194_121512_object64457:head# 2024-04-29T14:27:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619d9394:::benchmark_data_smithi194_121512_object69327:head# 2024-04-29T14:27:10.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619db6ea:::benchmark_data_smithi194_121512_object61329:head# 2024-04-29T14:27:10.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619e2f56:::benchmark_data_smithi194_121512_object119260:head# 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619e71be:::benchmark_data_smithi194_121512_object69563:head# 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619eae22:::benchmark_data_smithi194_121512_object73035:head# 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619f36e6:::benchmark_data_smithi194_121512_object50247:head# 2024-04-29T14:27:10.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:619f8f27:::benchmark_data_smithi194_121512_object112753:head# 2024-04-29T14:27:10.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a0107c:::benchmark_data_smithi194_121512_object127680:head# 2024-04-29T14:27:10.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a01761:::benchmark_data_smithi194_121512_object90162:head# 2024-04-29T14:27:10.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a0dcd5:::benchmark_data_smithi194_121512_object30047:head# 2024-04-29T14:27:10.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a0df67:::benchmark_data_smithi194_121512_object28725:head# 2024-04-29T14:27:10.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a2310f:::benchmark_data_smithi194_121512_object122872:head# 2024-04-29T14:27:10.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a2a074:::benchmark_data_smithi194_121512_object50754:head# 2024-04-29T14:27:10.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a2c513:::benchmark_data_smithi194_121512_object105444:head# 2024-04-29T14:27:10.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a3a8f1:::benchmark_data_smithi194_121512_object38321:head# 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a518c4:::benchmark_data_smithi194_121512_object98959:head# 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a52999:::benchmark_data_smithi194_121512_object53516:head# 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a5f740:::benchmark_data_smithi194_121512_object103971:head# 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a695e5:::benchmark_data_smithi194_121512_object19451:head# 2024-04-29T14:27:10.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a7ef17:::benchmark_data_smithi194_121512_object83379:head# 2024-04-29T14:27:10.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a89c94:::benchmark_data_smithi194_121512_object69982:head# 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a8f58b:::benchmark_data_smithi194_121512_object76238:head# 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61a9dbff:::benchmark_data_smithi194_121512_object62017:head# 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61aa279e:::benchmark_data_smithi194_121512_object125306:head# 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61aaebae:::benchmark_data_smithi194_121512_object19267:head# 2024-04-29T14:27:10.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ab388a:::benchmark_data_smithi194_121512_object28783:head# 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ab6641:::benchmark_data_smithi194_121512_object90181:head# 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ab7999:::benchmark_data_smithi194_121512_object129920:head# 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ab8351:::benchmark_data_smithi194_121512_object67490:head# 2024-04-29T14:27:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61abb341:::benchmark_data_smithi194_121512_object610:head# 2024-04-29T14:27:10.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61abf247:::benchmark_data_smithi194_121512_object61253:head# 2024-04-29T14:27:10.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ac02a6:::benchmark_data_smithi194_121512_object124502:head# 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ac4224:::benchmark_data_smithi194_121512_object100915:head# 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ac5edd:::benchmark_data_smithi194_121512_object92589:head# 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ac983b:::benchmark_data_smithi194_121512_object5217:head# 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61acd565:::benchmark_data_smithi194_121512_object110214:head# 2024-04-29T14:27:10.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ad170a:::benchmark_data_smithi194_121512_object111440:head# 2024-04-29T14:27:10.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ad4f00:::benchmark_data_smithi194_121512_object115796:head# 2024-04-29T14:27:10.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ae7adb:::benchmark_data_smithi194_121512_object74534:head# 2024-04-29T14:27:10.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61af0dde:::benchmark_data_smithi194_121512_object119403:head# 2024-04-29T14:27:10.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61af3d36:::benchmark_data_smithi194_121512_object15136:head# 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b0415f:::benchmark_data_smithi194_121512_object9051:head# 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b0d7cd:::benchmark_data_smithi194_121512_object104268:head# 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b0e64e:::benchmark_data_smithi194_121512_object57483:head# 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b1b7f9:::benchmark_data_smithi194_121512_object57714:head# 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b1d4f0:::benchmark_data_smithi194_121512_object6140:head# 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b1d95e:::benchmark_data_smithi194_121512_object129243:head# 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b23695:::benchmark_data_smithi194_121512_object18246:head# 2024-04-29T14:27:10.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b26062:::benchmark_data_smithi194_121512_object66309:head# 2024-04-29T14:27:10.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b294d0:::benchmark_data_smithi194_121512_object16870:head# 2024-04-29T14:27:10.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b2fdc2:::benchmark_data_smithi194_121512_object79538:head# 2024-04-29T14:27:10.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b3de95:::benchmark_data_smithi194_121512_object14497:head# 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b3e032:::benchmark_data_smithi194_121512_object111791:head# 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b50d81:::benchmark_data_smithi194_121512_object44323:head# 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b5635b:::benchmark_data_smithi194_121512_object18860:head# 2024-04-29T14:27:10.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b5ff27:::benchmark_data_smithi194_121512_object98249:head# 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b61e15:::benchmark_data_smithi194_121512_object76105:head# 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b69499:::benchmark_data_smithi194_121512_object14430:head# 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b6be74:::benchmark_data_smithi194_121512_object82142:head# 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b785dd:::benchmark_data_smithi194_121512_object60016:head# 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b7b368:::benchmark_data_smithi194_121512_object120529:head# 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b81472:::benchmark_data_smithi194_121512_object9140:head# 2024-04-29T14:27:10.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b8be05:::benchmark_data_smithi194_121512_object15388:head# 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b8d94e:::benchmark_data_smithi194_121512_object52554:head# 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b924ce:::benchmark_data_smithi194_121512_object32475:head# 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61b9c10d:::benchmark_data_smithi194_121512_object94656:head# 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61baada7:::benchmark_data_smithi194_121512_object111343:head# 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bb3feb:::benchmark_data_smithi194_121512_object9732:head# 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bbf97c:::benchmark_data_smithi194_121512_object47884:head# 2024-04-29T14:27:10.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bbfb47:::benchmark_data_smithi194_121512_object4596:head# 2024-04-29T14:27:10.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bc0a8b:::benchmark_data_smithi194_121512_object132624:head# 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bc54ab:::benchmark_data_smithi194_121512_object91594:head# 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bc5dd7:::benchmark_data_smithi194_121512_object40304:head# 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bd1001:::benchmark_data_smithi194_121512_object25652:head# 2024-04-29T14:27:10.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bd187b:::benchmark_data_smithi194_121512_object29091:head# 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bf0126:::benchmark_data_smithi194_121512_object73871:head# 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61bfe14d:::benchmark_data_smithi194_121512_object7404:head# 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c0d03f:::benchmark_data_smithi194_121512_object131230:head# 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c19ae1:::benchmark_data_smithi194_121512_object73456:head# 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c23b09:::benchmark_data_smithi194_121512_object8847:head# 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c2ddb7:::benchmark_data_smithi194_121512_object55656:head# 2024-04-29T14:27:10.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c344b1:::benchmark_data_smithi194_121512_object63074:head# 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c37488:::benchmark_data_smithi194_121512_object61838:head# 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c40c61:::benchmark_data_smithi194_121512_object127775:head# 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c49ce3:::benchmark_data_smithi194_121512_object62057:head# 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c59928:::benchmark_data_smithi194_121512_object15221:head# 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c5ee4e:::benchmark_data_smithi194_121512_object100239:head# 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c67b77:::benchmark_data_smithi194_121512_object108291:head# 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c6a89b:::benchmark_data_smithi194_121512_object3578:head# 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c6ed74:::benchmark_data_smithi194_121512_object75805:head# 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c76517:::benchmark_data_smithi194_121512_object99853:head# 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c77ae1:::benchmark_data_smithi194_121512_object19303:head# 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c7f0cd:::benchmark_data_smithi194_121512_object37751:head# 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c8cfff:::benchmark_data_smithi194_121512_object72877:head# 2024-04-29T14:27:10.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c9494d:::benchmark_data_smithi194_121512_object55329:head# 2024-04-29T14:27:10.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61c9a306:::benchmark_data_smithi194_121512_object35753:head# 2024-04-29T14:27:10.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ca7313:::benchmark_data_smithi194_121512_object25778:head# 2024-04-29T14:27:10.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61cbcb72:::benchmark_data_smithi194_121512_object30749:head# 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61cc8775:::benchmark_data_smithi194_121512_object68119:head# 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61cc8fe0:::benchmark_data_smithi194_121512_object8091:head# 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61cd8c09:::benchmark_data_smithi194_121512_object72770:head# 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61cdceb3:::benchmark_data_smithi194_121512_object22652:head# 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ce6320:::benchmark_data_smithi194_121512_object100891:head# 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ce7a5b:::benchmark_data_smithi194_121512_object116840:head# 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61cf1823:::benchmark_data_smithi194_121512_object110704:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61cfc6a0:::benchmark_data_smithi194_121512_object31180:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d03d81:::benchmark_data_smithi194_121512_object90032:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d09683:::benchmark_data_smithi194_121512_object60977:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d0ae47:::benchmark_data_smithi194_121512_object72988:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d1f2ea:::benchmark_data_smithi194_121512_object104471:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d25484:::benchmark_data_smithi194_121512_object78987:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d32c3a:::benchmark_data_smithi194_121512_object91467:head# 2024-04-29T14:27:10.728 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d33567:::benchmark_data_smithi194_121512_object19160:head# 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d3809d:::benchmark_data_smithi194_121512_object4789:head# 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d3d9e3:::benchmark_data_smithi194_121512_object120761:head# 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d3daf5:::benchmark_data_smithi194_121512_object125137:head# 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d48664:::benchmark_data_smithi194_121512_object36945:head# 2024-04-29T14:27:10.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d50ee2:::benchmark_data_smithi194_121512_object37820:head# 2024-04-29T14:27:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d5133c:::benchmark_data_smithi194_121512_object8093:head# 2024-04-29T14:27:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d74bff:::benchmark_data_smithi194_121512_object22080:head# 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d74da4:::benchmark_data_smithi194_121512_object69301:head# 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d8b700:::benchmark_data_smithi194_121512_object70731:head# 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61d97c6b:::benchmark_data_smithi194_121512_object109403:head# 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61da8df0:::benchmark_data_smithi194_121512_object42991:head# 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61dacc3d:::benchmark_data_smithi194_121512_object114245:head# 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61db1d15:::benchmark_data_smithi194_121512_object342:head# 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61db95db:::benchmark_data_smithi194_121512_object31247:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61db9b0a:::benchmark_data_smithi194_121512_object53178:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61dc10b4:::benchmark_data_smithi194_121512_object5707:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61dc1aed:::benchmark_data_smithi194_121512_object35758:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61dc7203:::benchmark_data_smithi194_121512_object104253:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61dcc92c:::benchmark_data_smithi194_121512_object23132:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61dd2d2e:::benchmark_data_smithi194_121512_object76044:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61dd6cd4:::benchmark_data_smithi194_121512_object92480:head# 2024-04-29T14:27:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.735 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61df3065:::benchmark_data_smithi194_121512_object62810:head# 2024-04-29T14:27:10.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e00a09:::benchmark_data_smithi194_121512_object72226:head# 2024-04-29T14:27:10.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e05253:::benchmark_data_smithi194_121512_object48995:head# 2024-04-29T14:27:10.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e0c601:::benchmark_data_smithi194_121512_object103797:head# 2024-04-29T14:27:10.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e0edbe:::benchmark_data_smithi194_121512_object119092:head# 2024-04-29T14:27:10.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e1b371:::benchmark_data_smithi194_121512_object107362:head# 2024-04-29T14:27:10.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e1b823:::benchmark_data_smithi194_121512_object37547:head# 2024-04-29T14:27:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e34b30:::benchmark_data_smithi194_121512_object39174:head# 2024-04-29T14:27:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e357cd:::benchmark_data_smithi194_121512_object87747:head# 2024-04-29T14:27:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e57bdb:::benchmark_data_smithi194_121512_object25482:head# 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e58c3e:::benchmark_data_smithi194_121512_object9996:head# 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e61e98:::benchmark_data_smithi194_121512_object124325:head# 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e64163:::benchmark_data_smithi194_121512_object36751:head# 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e66b88:::benchmark_data_smithi194_121512_object118609:head# 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e75ea3:::benchmark_data_smithi194_121512_object62275:head# 2024-04-29T14:27:10.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e7e4c0:::benchmark_data_smithi194_121512_object32163:head# 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e813f3:::benchmark_data_smithi194_121512_object56784:head# 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e85caa:::benchmark_data_smithi194_121512_object112924:head# 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e8724f:::benchmark_data_smithi194_121512_object127161:head# 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61e96304:::benchmark_data_smithi194_121512_object70565:head# 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ea2b4f:::benchmark_data_smithi194_121512_object115689:head# 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61eaa665:::benchmark_data_smithi194_121512_object3269:head# 2024-04-29T14:27:10.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61eb7363:::benchmark_data_smithi194_121512_object122772:head# 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61eba579:::benchmark_data_smithi194_121512_object118734:head# 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ebf73e:::benchmark_data_smithi194_121512_object12626:head# 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ec860d:::benchmark_data_smithi194_121512_object92134:head# 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ecc230:::benchmark_data_smithi194_121512_object79332:head# 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ecd25d:::benchmark_data_smithi194_121512_object132666:head# 2024-04-29T14:27:10.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ecf933:::benchmark_data_smithi194_121512_object123633:head# 2024-04-29T14:27:10.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ed45ed:::benchmark_data_smithi194_121512_object45656:head# 2024-04-29T14:27:10.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61edc3b3:::benchmark_data_smithi194_121512_object66338:head# 2024-04-29T14:27:10.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61edc3df:::benchmark_data_smithi194_121512_object62304:head# 2024-04-29T14:27:10.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ee1e88:::benchmark_data_smithi194_121512_object106847:head# 2024-04-29T14:27:10.757 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:10.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ee5f81:::benchmark_data_smithi194_121512_object100308:head# 2024-04-29T14:27:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ef173a:::benchmark_data_smithi194_121512_object128661:head# 2024-04-29T14:27:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f056eb:::benchmark_data_smithi194_121512_object31620:head# 2024-04-29T14:27:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f0992b:::benchmark_data_smithi194_121512_object5666:head# 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f0b66e:::benchmark_data_smithi194_121512_object921:head# 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f11c99:::benchmark_data_smithi194_121512_object85447:head# 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f2002d:::benchmark_data_smithi194_121512_object16159:head# 2024-04-29T14:27:10.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f28821:::benchmark_data_smithi194_121512_object89913:head# 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f29700:::benchmark_data_smithi194_121512_object15734:head# 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f35749:::benchmark_data_smithi194_121512_object119183:head# 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f63f0c:::benchmark_data_smithi194_121512_object71224:head# 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f7f336:::benchmark_data_smithi194_121512_object89231:head# 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f8ad69:::benchmark_data_smithi194_121512_object5931:head# 2024-04-29T14:27:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f99928:::benchmark_data_smithi194_121512_object87505:head# 2024-04-29T14:27:10.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61f9f16a:::benchmark_data_smithi194_121512_object19304:head# 2024-04-29T14:27:10.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fa0b07:::benchmark_data_smithi194_121512_object96319:head# 2024-04-29T14:27:10.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fa62f4:::benchmark_data_smithi194_121512_object53817:head# 2024-04-29T14:27:10.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fa94a2:::benchmark_data_smithi194_121512_object103229:head# 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61faada0:::benchmark_data_smithi194_121512_object122925:head# 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fcc1b1:::benchmark_data_smithi194_121512_object81251:head# 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fd66ee:::benchmark_data_smithi194_121512_object55861:head# 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fda5b3:::benchmark_data_smithi194_121512_object86350:head# 2024-04-29T14:27:10.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fe79aa:::benchmark_data_smithi194_121512_object56630:head# 2024-04-29T14:27:10.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61fef512:::benchmark_data_smithi194_121512_object127335:head# 2024-04-29T14:27:10.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:61ffa8e1:::benchmark_data_smithi194_121512_object105339:head# 2024-04-29T14:27:10.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62004f14:::benchmark_data_smithi194_121512_object114076:head# 2024-04-29T14:27:10.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6200657a:::benchmark_data_smithi194_121512_object34915:head# 2024-04-29T14:27:10.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62006709:::benchmark_data_smithi194_121512_object48101:head# 2024-04-29T14:27:10.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6200afef:::benchmark_data_smithi194_121512_object128860:head# 2024-04-29T14:27:10.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6200ed52:::benchmark_data_smithi194_121512_object114735:head# 2024-04-29T14:27:10.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620121e2:::benchmark_data_smithi194_121512_object16025:head# 2024-04-29T14:27:10.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6201adbf:::benchmark_data_smithi194_121512_object5728:head# 2024-04-29T14:27:10.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6201d91e:::benchmark_data_smithi194_121512_object77844:head# 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6201e60e:::benchmark_data_smithi194_121512_object43043:head# 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6203055b:::benchmark_data_smithi194_121512_object90337:head# 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6203663b:::benchmark_data_smithi194_121512_object130022:head# 2024-04-29T14:27:10.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6203ba12:::benchmark_data_smithi194_121512_object88513:head# 2024-04-29T14:27:10.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6203dc5b:::benchmark_data_smithi194_121512_object41859:head# 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62049a43:::benchmark_data_smithi194_121512_object40010:head# 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6204dc61:::benchmark_data_smithi194_121512_object13644:head# 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6204fee6:::benchmark_data_smithi194_121512_object98215:head# 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62058c3e:::benchmark_data_smithi194_121512_object100422:head# 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620597c1:::benchmark_data_smithi194_121512_object53501:head# 2024-04-29T14:27:10.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6205c9d4:::benchmark_data_smithi194_121512_object117290:head# 2024-04-29T14:27:10.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6205f243:::benchmark_data_smithi194_121512_object83571:head# 2024-04-29T14:27:10.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6205f7be:::benchmark_data_smithi194_121512_object126427:head# 2024-04-29T14:27:10.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6206a0f4:::benchmark_data_smithi194_121512_object43494:head# 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6206a9e1:::benchmark_data_smithi194_121512_object53313:head# 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6206c7d4:::benchmark_data_smithi194_121512_object38303:head# 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62082e52:::benchmark_data_smithi194_121512_object95222:head# 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62085841:::benchmark_data_smithi194_121512_object96468:head# 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6208e071:::benchmark_data_smithi194_121512_object12582:head# 2024-04-29T14:27:10.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6208f1ae:::benchmark_data_smithi194_121512_object108703:head# 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62092d0e:::benchmark_data_smithi194_121512_object129740:head# 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620948b4:::benchmark_data_smithi194_121512_object19502:head# 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620972aa:::benchmark_data_smithi194_121512_object21003:head# 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620a815f:::benchmark_data_smithi194_121512_object29568:head# 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620af14a:::benchmark_data_smithi194_121512_object80200:head# 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620b1887:::benchmark_data_smithi194_121512_object43944:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620b6958:::benchmark_data_smithi194_121512_object32976:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620c3cb2:::benchmark_data_smithi194_121512_object109666:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620cba16:::benchmark_data_smithi194_121512_object91911:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620cfd90:::benchmark_data_smithi194_121512_object96032:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620dc4c6:::benchmark_data_smithi194_121512_object129273:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620dd0d6:::benchmark_data_smithi194_121512_object51055:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620df572:::benchmark_data_smithi194_121512_object7423:head# 2024-04-29T14:27:10.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620e6fd0:::benchmark_data_smithi194_121512_object1625:head# 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620eb69a:::benchmark_data_smithi194_121512_object111283:head# 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620ee9a4:::benchmark_data_smithi194_121512_object124561:head# 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620f0294:::benchmark_data_smithi194_121512_object107908:head# 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:620f3a5a:::benchmark_data_smithi194_121512_object75281:head# 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621024de:::benchmark_data_smithi194_121512_object41633:head# 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6210344a:::benchmark_data_smithi194_121512_object103390:head# 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62105ef5:::benchmark_data_smithi194_121512_object78674:head# 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62106dc8:::benchmark_data_smithi194_121512_object6062:head# 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6210a671:::benchmark_data_smithi194_121512_object73496:head# 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6211b01c:::benchmark_data_smithi194_121512_object31148:head# 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6211be32:::benchmark_data_smithi194_121512_object87299:head# 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62128d90:::benchmark_data_smithi194_121512_object22871:head# 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62129cb5:::benchmark_data_smithi194_121512_object126049:head# 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6212bfe7:::benchmark_data_smithi194_121512_object127850:head# 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62131dcc:::benchmark_data_smithi194_121512_object100279:head# 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62156223:::benchmark_data_smithi194_121512_object55839:head# 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6215a067:::benchmark_data_smithi194_121512_object7786:head# 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621619aa:::benchmark_data_smithi194_121512_object47034:head# 2024-04-29T14:27:10.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62178778:::benchmark_data_smithi194_121512_object51404:head# 2024-04-29T14:27:10.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6217db94:::benchmark_data_smithi194_121512_object18635:head# 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6217e763:::benchmark_data_smithi194_121512_object88717:head# 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6218b222:::benchmark_data_smithi194_121512_object47504:head# 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6218e457:::benchmark_data_smithi194_121512_object118322:head# 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621911b5:::benchmark_data_smithi194_121512_object117698:head# 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62196d32:::benchmark_data_smithi194_121512_object6998:head# 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621a5763:::benchmark_data_smithi194_121512_object114300:head# 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621a900e:::benchmark_data_smithi194_121512_object101994:head# 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621ac1a2:::benchmark_data_smithi194_121512_object62065:head# 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621b61cb:::benchmark_data_smithi194_121512_object105671:head# 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621c3687:::benchmark_data_smithi194_121512_object97873:head# 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621c4993:::benchmark_data_smithi194_121512_object75598:head# 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621cc2f2:::benchmark_data_smithi194_121512_object65296:head# 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621cefe9:::benchmark_data_smithi194_121512_object75480:head# 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621d0a0c:::benchmark_data_smithi194_121512_object10012:head# 2024-04-29T14:27:10.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621d77e0:::benchmark_data_smithi194_121512_object82224:head# 2024-04-29T14:27:10.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621e2c06:::benchmark_data_smithi194_121512_object1544:head# 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621ed0de:::benchmark_data_smithi194_121512_object14239:head# 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621f0290:::benchmark_data_smithi194_121512_object125332:head# 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621f673a:::benchmark_data_smithi194_121512_object86099:head# 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:621fd012:::benchmark_data_smithi194_121512_object87442:head# 2024-04-29T14:27:10.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6220e15b:::benchmark_data_smithi194_121512_object80310:head# 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6221b025:::benchmark_data_smithi194_121512_object85250:head# 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62227908:::benchmark_data_smithi194_121512_object120679:head# 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622354f4:::benchmark_data_smithi194_121512_object68184:head# 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6223baa6:::benchmark_data_smithi194_121512_object4888:head# 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6224ab57:::benchmark_data_smithi194_121512_object29375:head# 2024-04-29T14:27:10.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6224fe8e:::benchmark_data_smithi194_121512_object111455:head# 2024-04-29T14:27:10.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62262598:::benchmark_data_smithi194_121512_object55635:head# 2024-04-29T14:27:10.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62263ae9:::benchmark_data_smithi194_121512_object38950:head# 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6226cf39:::benchmark_data_smithi194_121512_object107306:head# 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62278f64:::benchmark_data_smithi194_121512_object53186:head# 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62283ae8:::benchmark_data_smithi194_121512_object125288:head# 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62287cb2:::benchmark_data_smithi194_121512_object24270:head# 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62292e31:::benchmark_data_smithi194_121512_object72801:head# 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62297135:::benchmark_data_smithi194_121512_object63604:head# 2024-04-29T14:27:10.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6229962b:::benchmark_data_smithi194_121512_object25497:head# 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6229e204:::benchmark_data_smithi194_121512_object104552:head# 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622a247b:::benchmark_data_smithi194_121512_object48965:head# 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622b87e9:::benchmark_data_smithi194_121512_object110379:head# 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622be8fb:::benchmark_data_smithi194_121512_object3621:head# 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622c0d23:::benchmark_data_smithi194_121512_object81291:head# 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622c96d1:::benchmark_data_smithi194_121512_object77209:head# 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622ddf46:::benchmark_data_smithi194_121512_object90633:head# 2024-04-29T14:27:10.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622e0fa3:::benchmark_data_smithi194_121512_object61709:head# 2024-04-29T14:27:10.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622e0fe3:::benchmark_data_smithi194_121512_object56476:head# 2024-04-29T14:27:10.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622ed649:::benchmark_data_smithi194_121512_object126863:head# 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:622f71f1:::benchmark_data_smithi194_121512_object39001:head# 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6230116d:::benchmark_data_smithi194_121512_object31352:head# 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6230b589:::benchmark_data_smithi194_121512_object15143:head# 2024-04-29T14:27:10.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6230da57:::benchmark_data_smithi194_121512_object86880:head# 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62312b42:::benchmark_data_smithi194_121512_object5078:head# 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62323f93:::benchmark_data_smithi194_121512_object35348:head# 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62348937:::benchmark_data_smithi194_121512_object33975:head# 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62349ba0:::benchmark_data_smithi194_121512_object89393:head# 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623757da:::benchmark_data_smithi194_121512_object56766:head# 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6238e420:::benchmark_data_smithi194_121512_object6625:head# 2024-04-29T14:27:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623b000a:::benchmark_data_smithi194_121512_object14827:head# 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623b251c:::benchmark_data_smithi194_121512_object130896:head# 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623d4f82:::benchmark_data_smithi194_121512_object420:head# 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623e35cb:::benchmark_data_smithi194_121512_object64941:head# 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623e5114:::benchmark_data_smithi194_121512_object129338:head# 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623eba47:::benchmark_data_smithi194_121512_object73824:head# 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623f358b:::benchmark_data_smithi194_121512_object28845:head# 2024-04-29T14:27:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623f4dfa:::benchmark_data_smithi194_121512_object83116:head# 2024-04-29T14:27:10.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:623f5cd7:::benchmark_data_smithi194_121512_object53658:head# 2024-04-29T14:27:10.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62401323:::benchmark_data_smithi194_121512_object113894:head# 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62404de0:::benchmark_data_smithi194_121512_object90161:head# 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62409d98:::benchmark_data_smithi194_121512_object17245:head# 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6240b285:::benchmark_data_smithi194_121512_object56069:head# 2024-04-29T14:27:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62415a36:::benchmark_data_smithi194_121512_object80294:head# 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62425762:::benchmark_data_smithi194_121512_object8955:head# 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6243fd1e:::benchmark_data_smithi194_121512_object129177:head# 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62449f43:::benchmark_data_smithi194_121512_object76958:head# 2024-04-29T14:27:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6244b1e5:::benchmark_data_smithi194_121512_object100093:head# 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6245efd9:::benchmark_data_smithi194_121512_object105835:head# 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624678e9:::benchmark_data_smithi194_121512_object14307:head# 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62468795:::benchmark_data_smithi194_121512_object65885:head# 2024-04-29T14:27:10.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6246fb0a:::benchmark_data_smithi194_121512_object74323:head# 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6247d51d:::benchmark_data_smithi194_121512_object86689:head# 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6249681b:::benchmark_data_smithi194_121512_object24066:head# 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6249846a:::benchmark_data_smithi194_121512_object28113:head# 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624990c4:::benchmark_data_smithi194_121512_object59878:head# 2024-04-29T14:27:10.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6249d44d:::benchmark_data_smithi194_121512_object38311:head# 2024-04-29T14:27:10.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6249fc48:::benchmark_data_smithi194_121512_object81621:head# 2024-04-29T14:27:10.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6249ff6c:::benchmark_data_smithi194_121512_object24968:head# 2024-04-29T14:27:10.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624aca0d:::benchmark_data_smithi194_121512_object16433:head# 2024-04-29T14:27:10.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624b00a9:::benchmark_data_smithi194_121512_object20629:head# 2024-04-29T14:27:10.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624b322e:::benchmark_data_smithi194_121512_object55830:head# 2024-04-29T14:27:10.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624c1a8d:::benchmark_data_smithi194_121512_object58236:head# 2024-04-29T14:27:10.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624c8f81:::benchmark_data_smithi194_121512_object107685:head# 2024-04-29T14:27:10.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624cae3f:::benchmark_data_smithi194_121512_object49202:head# 2024-04-29T14:27:10.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624fc082:::benchmark_data_smithi194_121512_object82655:head# 2024-04-29T14:27:10.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:624ffc11:::benchmark_data_smithi194_121512_object82656:head# 2024-04-29T14:27:10.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625017c1:::benchmark_data_smithi194_121512_object56277:head# 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62504f3d:::benchmark_data_smithi194_121512_object8905:head# 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6250645f:::benchmark_data_smithi194_121512_object128777:head# 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6250d709:::benchmark_data_smithi194_121512_object3679:head# 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62514b27:::benchmark_data_smithi194_121512_object80750:head# 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62516b0d:::benchmark_data_smithi194_121512_object20381:head# 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62536a0d:::benchmark_data_smithi194_121512_object72175:head# 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6253c732:::benchmark_data_smithi194_121512_object94408:head# 2024-04-29T14:27:10.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625471c1:::benchmark_data_smithi194_121512_object129115:head# 2024-04-29T14:27:10.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6254d5f6:::benchmark_data_smithi194_121512_object46668:head# 2024-04-29T14:27:10.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6254f821:::benchmark_data_smithi194_121512_object43368:head# 2024-04-29T14:27:10.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625510c6:::benchmark_data_smithi194_121512_object123880:head# 2024-04-29T14:27:10.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6255263f:::benchmark_data_smithi194_121512_object115775:head# 2024-04-29T14:27:10.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6255271f:::benchmark_data_smithi194_121512_object57454:head# 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6255648e:::benchmark_data_smithi194_121512_object23378:head# 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62562411:::benchmark_data_smithi194_121512_object105250:head# 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625640d3:::benchmark_data_smithi194_121512_object69160:head# 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62567d61:::benchmark_data_smithi194_121512_object38725:head# 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6256c777:::benchmark_data_smithi194_121512_object103539:head# 2024-04-29T14:27:10.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62580bd2:::benchmark_data_smithi194_121512_object105699:head# 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6258a831:::benchmark_data_smithi194_121512_object121236:head# 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6259446f:::benchmark_data_smithi194_121512_object60277:head# 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625a3574:::benchmark_data_smithi194_121512_object37391:head# 2024-04-29T14:27:10.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625adb4b:::benchmark_data_smithi194_121512_object111966:head# 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625b632c:::benchmark_data_smithi194_121512_object97859:head# 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625cea85:::benchmark_data_smithi194_121512_object93558:head# 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625e26d3:::benchmark_data_smithi194_121512_object101567:head# 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625e7034:::benchmark_data_smithi194_121512_object94783:head# 2024-04-29T14:27:10.842 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625e8ddc:::benchmark_data_smithi194_121512_object8610:head# 2024-04-29T14:27:10.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625eb2ef:::benchmark_data_smithi194_121512_object72585:head# 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625eff6e:::benchmark_data_smithi194_121512_object121046:head# 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:625f8b91:::benchmark_data_smithi194_121512_object16183:head# 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62607544:::benchmark_data_smithi194_121512_object49415:head# 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6260ad02:::benchmark_data_smithi194_121512_object103544:head# 2024-04-29T14:27:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62612198:::benchmark_data_smithi194_121512_object52921:head# 2024-04-29T14:27:10.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626148ee:::benchmark_data_smithi194_121512_object39260:head# 2024-04-29T14:27:10.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6261a403:::benchmark_data_smithi194_121512_object84182:head# 2024-04-29T14:27:10.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6261e4f8:::benchmark_data_smithi194_121512_object33658:head# 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6265c0a9:::benchmark_data_smithi194_121512_object123046:head# 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6265d713:::benchmark_data_smithi194_121512_object25721:head# 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6266224d:::benchmark_data_smithi194_121512_object130529:head# 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62672317:::benchmark_data_smithi194_121512_object67800:head# 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62680155:::benchmark_data_smithi194_121512_object35934:head# 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6268866f:::benchmark_data_smithi194_121512_object28669:head# 2024-04-29T14:27:10.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6268946f:::benchmark_data_smithi194_121512_object69104:head# 2024-04-29T14:27:10.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626a1ddc:::benchmark_data_smithi194_121512_object88707:head# 2024-04-29T14:27:10.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.859 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:10.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626a9f42:::benchmark_data_smithi194_121512_object41535:head# 2024-04-29T14:27:10.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626bebe6:::benchmark_data_smithi194_121512_object86756:head# 2024-04-29T14:27:10.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626c963b:::benchmark_data_smithi194_121512_object67632:head# 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626ce58e:::benchmark_data_smithi194_121512_object116595:head# 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626d6cf6:::benchmark_data_smithi194_121512_object15396:head# 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626d77f0:::benchmark_data_smithi194_121512_object110026:head# 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626d97d6:::benchmark_data_smithi194_121512_object105870:head# 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626e5bd4:::benchmark_data_smithi194_121512_object43455:head# 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626f6a43:::benchmark_data_smithi194_121512_object35038:head# 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:626f859d:::benchmark_data_smithi194_121512_object20958:head# 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62708b03:::benchmark_data_smithi194_121512_object62402:head# 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62709c26:::benchmark_data_smithi194_121512_object14973:head# 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6270baa0:::benchmark_data_smithi194_121512_object93046:head# 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62710ae8:::benchmark_data_smithi194_121512_object114112:head# 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62715f92:::benchmark_data_smithi194_121512_object54685:head# 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62721e47:::benchmark_data_smithi194_121512_object100957:head# 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62722a31:::benchmark_data_smithi194_121512_object55016:head# 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6272700b:::benchmark_data_smithi194_121512_object53603:head# 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62728307:::benchmark_data_smithi194_121512_object7586:head# 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6272c8af:::benchmark_data_smithi194_121512_object40604:head# 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6273e1a7:::benchmark_data_smithi194_121512_object51479:head# 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62740f92:::benchmark_data_smithi194_121512_object19673:head# 2024-04-29T14:27:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62742426:::benchmark_data_smithi194_121512_object21917:head# 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627463f1:::benchmark_data_smithi194_121512_object32134:head# 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62747dcb:::benchmark_data_smithi194_121512_object123112:head# 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6274a4f1:::benchmark_data_smithi194_121512_object90146:head# 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6274bbd0:::benchmark_data_smithi194_121512_object110203:head# 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62752559:::benchmark_data_smithi194_121512_object58052:head# 2024-04-29T14:27:10.866 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62753a3d:::benchmark_data_smithi194_121512_object74947:head# 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62756fa6:::benchmark_data_smithi194_121512_object42234:head# 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6276f01e:::benchmark_data_smithi194_121512_object85310:head# 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6276f7f8:::benchmark_data_smithi194_121512_object35379:head# 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627798b8:::benchmark_data_smithi194_121512_object34621:head# 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627830d3:::benchmark_data_smithi194_121512_object61322:head# 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6279bf04:::benchmark_data_smithi194_121512_object100738:head# 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6279ffce:::benchmark_data_smithi194_121512_object16530:head# 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627a360a:::benchmark_data_smithi194_121512_object1761:head# 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627b07c3:::benchmark_data_smithi194_121512_object1793:head# 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627bb2a8:::benchmark_data_smithi194_121512_object111485:head# 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627bb36b:::benchmark_data_smithi194_121512_object112465:head# 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627c2c1a:::benchmark_data_smithi194_121512_object130266:head# 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627c4869:::benchmark_data_smithi194_121512_object83099:head# 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627d85b6:::benchmark_data_smithi194_121512_object102489:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627f3e4b:::benchmark_data_smithi194_121512_object118488:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627fa4dd:::benchmark_data_smithi194_121512_object4472:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:627fdcc6:::benchmark_data_smithi194_121512_object50807:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62806c33:::benchmark_data_smithi194_121512_object110096:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6280a00f:::benchmark_data_smithi194_121512_object84010:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6280a016:::benchmark_data_smithi194_121512_object4839:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6280acb7:::benchmark_data_smithi194_121512_object13964:head# 2024-04-29T14:27:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62819c51:::benchmark_data_smithi194_121512_object114630:head# 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62824be5:::benchmark_data_smithi194_121512_object20546:head# 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62828f80:::benchmark_data_smithi194_121512_object16075:head# 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6282f6f8:::benchmark_data_smithi194_121512_object59767:head# 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62835d95:::benchmark_data_smithi194_121512_object7263:head# 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62838cc2:::benchmark_data_smithi194_121512_object40144:head# 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6283a219:::benchmark_data_smithi194_121512_object104204:head# 2024-04-29T14:27:10.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6284075f:::benchmark_data_smithi194_121512_object45959:head# 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628415f2:::benchmark_data_smithi194_121512_object130175:head# 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628440e5:::benchmark_data_smithi194_121512_object61450:head# 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62849932:::benchmark_data_smithi194_121512_object27325:head# 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62857bef:::benchmark_data_smithi194_121512_object117622:head# 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628660dc:::benchmark_data_smithi194_121512_object132083:head# 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62868720:::benchmark_data_smithi194_121512_object76522:head# 2024-04-29T14:27:10.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6286a2e9:::benchmark_data_smithi194_121512_object37433:head# 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62872dd8:::benchmark_data_smithi194_121512_object824:head# 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62880795:::benchmark_data_smithi194_121512_object84720:head# 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6288378b:::benchmark_data_smithi194_121512_object87143:head# 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6288d6a8:::benchmark_data_smithi194_121512_object86413:head# 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62892d70:::benchmark_data_smithi194_121512_object84362:head# 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62897243:::benchmark_data_smithi194_121512_object122367:head# 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62898898:::benchmark_data_smithi194_121512_object6368:head# 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628b1aa8:::benchmark_data_smithi194_121512_object121230:head# 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628c553c:::benchmark_data_smithi194_121512_object78236:head# 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628c5ae6:::benchmark_data_smithi194_121512_object21210:head# 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628d5783:::benchmark_data_smithi194_121512_object114814:head# 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628d7f46:::benchmark_data_smithi194_121512_object34657:head# 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628dc938:::benchmark_data_smithi194_121512_object114062:head# 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628e5251:::benchmark_data_smithi194_121512_object34843:head# 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628e80b1:::benchmark_data_smithi194_121512_object21642:head# 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628efd50:::benchmark_data_smithi194_121512_object83020:head# 2024-04-29T14:27:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628f3a08:::benchmark_data_smithi194_121512_object15712:head# 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628f8c1a:::benchmark_data_smithi194_121512_object69135:head# 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628fab2a:::benchmark_data_smithi194_121512_object118153:head# 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:628fdf0a:::benchmark_data_smithi194_121512_object65550:head# 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62900528:::benchmark_data_smithi194_121512_object7913:head# 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62908b4c:::benchmark_data_smithi194_121512_object78333:head# 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6290c623:::benchmark_data_smithi194_121512_object96214:head# 2024-04-29T14:27:10.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6290f320:::benchmark_data_smithi194_121512_object64287:head# 2024-04-29T14:27:10.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629181e2:::benchmark_data_smithi194_121512_object34712:head# 2024-04-29T14:27:10.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629193b1:::benchmark_data_smithi194_121512_object107400:head# 2024-04-29T14:27:10.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62921ed9:::benchmark_data_smithi194_121512_object38841:head# 2024-04-29T14:27:10.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629318a4:::benchmark_data_smithi194_121512_object109539:head# 2024-04-29T14:27:10.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629327bb:::benchmark_data_smithi194_121512_object4170:head# 2024-04-29T14:27:10.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6293521e:::benchmark_data_smithi194_121512_object116190:head# 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6293555f:::benchmark_data_smithi194_121512_object17161:head# 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6294cfda:::benchmark_data_smithi194_121512_object28165:head# 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6294e7a2:::benchmark_data_smithi194_121512_object2591:head# 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629529da:::benchmark_data_smithi194_121512_object35481:head# 2024-04-29T14:27:10.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6296006a:::benchmark_data_smithi194_121512_object56397:head# 2024-04-29T14:27:10.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62975adb:::benchmark_data_smithi194_121512_object3467:head# 2024-04-29T14:27:10.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629796ee:::benchmark_data_smithi194_121512_object21398:head# 2024-04-29T14:27:10.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6297d0b8:::benchmark_data_smithi194_121512_object17320:head# 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62980500:::benchmark_data_smithi194_121512_object69946:head# 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6299b3ac:::benchmark_data_smithi194_121512_object38754:head# 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629b1323:::benchmark_data_smithi194_121512_object36692:head# 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629b1ad3:::benchmark_data_smithi194_121512_object22023:head# 2024-04-29T14:27:10.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629b751c:::benchmark_data_smithi194_121512_object125865:head# 2024-04-29T14:27:10.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629bc224:::benchmark_data_smithi194_121512_object87010:head# 2024-04-29T14:27:10.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629bf5e0:::benchmark_data_smithi194_121512_object4599:head# 2024-04-29T14:27:10.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629c57c3:::benchmark_data_smithi194_121512_object68731:head# 2024-04-29T14:27:10.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629d847c:::benchmark_data_smithi194_121512_object71531:head# 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629ddad1:::benchmark_data_smithi194_121512_object71365:head# 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629e3ed1:::benchmark_data_smithi194_121512_object61320:head# 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629e60ec:::benchmark_data_smithi194_121512_object31772:head# 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629e65a7:::benchmark_data_smithi194_121512_object65759:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629e715d:::benchmark_data_smithi194_121512_object13235:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629fb4cb:::benchmark_data_smithi194_121512_object99993:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:629fd2b3:::benchmark_data_smithi194_121512_object131479:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a0c9b6:::benchmark_data_smithi194_121512_object116475:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a12279:::benchmark_data_smithi194_121512_object93021:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a24927:::benchmark_data_smithi194_121512_object2537:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a2525a:::benchmark_data_smithi194_121512_object86391:head# 2024-04-29T14:27:10.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a2801b:::benchmark_data_smithi194_121512_object39728:head# 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a2fbb7:::benchmark_data_smithi194_121512_object103168:head# 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a311f5:::benchmark_data_smithi194_121512_object47686:head# 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a327cb:::benchmark_data_smithi194_121512_object70961:head# 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a36d6c:::benchmark_data_smithi194_121512_object64988:head# 2024-04-29T14:27:10.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a398a6:::benchmark_data_smithi194_121512_object124383:head# 2024-04-29T14:27:10.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a4ac83:::benchmark_data_smithi194_121512_object109073:head# 2024-04-29T14:27:10.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a4afb8:::benchmark_data_smithi194_121512_object132121:head# 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a5983e:::benchmark_data_smithi194_121512_object97193:head# 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a5df38:::benchmark_data_smithi194_121512_object105875:head# 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a6aa86:::benchmark_data_smithi194_121512_object17633:head# 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a6d23b:::benchmark_data_smithi194_121512_object66470:head# 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a8342a:::benchmark_data_smithi194_121512_object25629:head# 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a8ade1:::benchmark_data_smithi194_121512_object17906:head# 2024-04-29T14:27:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a8eaf9:::benchmark_data_smithi194_121512_object74387:head# 2024-04-29T14:27:10.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a95d85:::benchmark_data_smithi194_121512_object28719:head# 2024-04-29T14:27:10.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62a98e34:::benchmark_data_smithi194_121512_object17524:head# 2024-04-29T14:27:10.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62aa9058:::benchmark_data_smithi194_121512_object69290:head# 2024-04-29T14:27:10.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ab5465:::benchmark_data_smithi194_121512_object8392:head# 2024-04-29T14:27:10.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62abacd2:::benchmark_data_smithi194_121512_object92706:head# 2024-04-29T14:27:10.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.912 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:10.913 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62abd08a:::benchmark_data_smithi194_121512_object124116:head# 2024-04-29T14:27:10.913 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ac0b32:::benchmark_data_smithi194_121512_object35826:head# 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ac6657:::benchmark_data_smithi194_121512_object34160:head# 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ad63e8:::benchmark_data_smithi194_121512_object7534:head# 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ae5178:::benchmark_data_smithi194_121512_object40755:head# 2024-04-29T14:27:10.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62af4ba3:::benchmark_data_smithi194_121512_object38944:head# 2024-04-29T14:27:10.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62af7925:::benchmark_data_smithi194_121512_object62502:head# 2024-04-29T14:27:10.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b057af:::benchmark_data_smithi194_121512_object98270:head# 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b0e23d:::benchmark_data_smithi194_121512_object117067:head# 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b197da:::benchmark_data_smithi194_121512_object117439:head# 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b33881:::benchmark_data_smithi194_121512_object91005:head# 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b46166:::benchmark_data_smithi194_121512_object96498:head# 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b48a8d:::benchmark_data_smithi194_121512_object127255:head# 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b4e42c:::benchmark_data_smithi194_121512_object97611:head# 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b4f49b:::benchmark_data_smithi194_121512_object45945:head# 2024-04-29T14:27:10.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b4fcb6:::benchmark_data_smithi194_121512_object37938:head# 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b55b07:::benchmark_data_smithi194_121512_object49924:head# 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b57a78:::benchmark_data_smithi194_121512_object30309:head# 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b5835e:::benchmark_data_smithi194_121512_object84831:head# 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b5dcc9:::benchmark_data_smithi194_121512_object10185:head# 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b68a03:::benchmark_data_smithi194_121512_object125061:head# 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b743e2:::benchmark_data_smithi194_121512_object17811:head# 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b7b638:::benchmark_data_smithi194_121512_object26286:head# 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b8644c:::benchmark_data_smithi194_121512_object94798:head# 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b88665:::benchmark_data_smithi194_121512_object121544:head# 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b8cfa1:::benchmark_data_smithi194_121512_object78495:head# 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b9356d:::benchmark_data_smithi194_121512_object5090:head# 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b94269:::benchmark_data_smithi194_121512_object19700:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62b9c628:::benchmark_data_smithi194_121512_object72499:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ba72f4:::benchmark_data_smithi194_121512_object24377:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ba898c:::benchmark_data_smithi194_121512_object24311:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62bb5f16:::benchmark_data_smithi194_121512_object19890:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62bc577f:::benchmark_data_smithi194_121512_object58127:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62bc7eea:::benchmark_data_smithi194_121512_object23686:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62bcf7b8:::benchmark_data_smithi194_121512_object17466:head# 2024-04-29T14:27:10.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62bd6cdd:::benchmark_data_smithi194_121512_object98374:head# 2024-04-29T14:27:10.925 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62bdf7ed:::benchmark_data_smithi194_121512_object115295:head# 2024-04-29T14:27:10.925 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:10.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:10.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62beb2d8:::benchmark_data_smithi194_121512_object35293:head# 2024-04-29T14:27:10.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62bedb0a:::benchmark_data_smithi194_121512_object82657:head# 2024-04-29T14:27:10.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c00cc1:::benchmark_data_smithi194_121512_object33806:head# 2024-04-29T14:27:10.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c05d70:::benchmark_data_smithi194_121512_object43975:head# 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c0a342:::benchmark_data_smithi194_121512_object127324:head# 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c11de4:::benchmark_data_smithi194_121512_object116895:head# 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c13dd1:::benchmark_data_smithi194_121512_object69606:head# 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c15a62:::benchmark_data_smithi194_121512_object1768:head# 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c18404:::benchmark_data_smithi194_121512_object131182:head# 2024-04-29T14:27:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c18f90:::benchmark_data_smithi194_121512_object128645:head# 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c1a0c1:::benchmark_data_smithi194_121512_object58685:head# 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c219cf:::benchmark_data_smithi194_121512_object132537:head# 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c233f2:::benchmark_data_smithi194_121512_object15082:head# 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c244e7:::benchmark_data_smithi194_121512_object36859:head# 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c2bc1a:::benchmark_data_smithi194_121512_object86153:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c2e3e1:::benchmark_data_smithi194_121512_object36243:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c312bd:::benchmark_data_smithi194_121512_object13590:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c3c434:::benchmark_data_smithi194_121512_object98437:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c42fd0:::benchmark_data_smithi194_121512_object132308:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c482b4:::benchmark_data_smithi194_121512_object123816:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c54a8d:::benchmark_data_smithi194_121512_object115808:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c670b1:::benchmark_data_smithi194_121512_object61985:head# 2024-04-29T14:27:10.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c899ea:::benchmark_data_smithi194_121512_object55335:head# 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62c987dc:::benchmark_data_smithi194_121512_object12660:head# 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62cc229d:::benchmark_data_smithi194_121512_object62331:head# 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62cc27b3:::benchmark_data_smithi194_121512_object94216:head# 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62cd1cec:::benchmark_data_smithi194_121512_object95963:head# 2024-04-29T14:27:10.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ce4537:::benchmark_data_smithi194_121512_object71018:head# 2024-04-29T14:27:10.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62cfca34:::benchmark_data_smithi194_121512_object117335:head# 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62cffc56:::benchmark_data_smithi194_121512_object73323:head# 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d16054:::benchmark_data_smithi194_121512_object88766:head# 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d161cd:::benchmark_data_smithi194_121512_object107492:head# 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d1ae9f:::benchmark_data_smithi194_121512_object102472:head# 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d1c4c6:::benchmark_data_smithi194_121512_object50458:head# 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d1f208:::benchmark_data_smithi194_121512_object6408:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d2051c:::benchmark_data_smithi194_121512_object62192:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d2df60:::benchmark_data_smithi194_121512_object112821:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d2e1f0:::benchmark_data_smithi194_121512_object48010:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d2ef6e:::benchmark_data_smithi194_121512_object77729:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d333e1:::benchmark_data_smithi194_121512_object64907:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d3a3ff:::benchmark_data_smithi194_121512_object84917:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d40ef4:::benchmark_data_smithi194_121512_object128598:head# 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d56781:::benchmark_data_smithi194_121512_object27945:head# 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d5a490:::benchmark_data_smithi194_121512_object8643:head# 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d5d0d4:::benchmark_data_smithi194_121512_object18232:head# 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d61ee2:::benchmark_data_smithi194_121512_object88107:head# 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d62011:::benchmark_data_smithi194_121512_object26122:head# 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d67b48:::benchmark_data_smithi194_121512_object98856:head# 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d680aa:::benchmark_data_smithi194_121512_object109056:head# 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d738ba:::benchmark_data_smithi194_121512_object2832:head# 2024-04-29T14:27:10.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d7a62f:::benchmark_data_smithi194_121512_object33035:head# 2024-04-29T14:27:10.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d7e22e:::benchmark_data_smithi194_121512_object82117:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d8204c:::benchmark_data_smithi194_121512_object55276:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d97cd9:::benchmark_data_smithi194_121512_object94356:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62d9ccd5:::benchmark_data_smithi194_121512_object7658:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62da0903:::benchmark_data_smithi194_121512_object97633:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62da318e:::benchmark_data_smithi194_121512_object102841:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62da95b8:::benchmark_data_smithi194_121512_object48348:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62db0341:::benchmark_data_smithi194_121512_object3241:head# 2024-04-29T14:27:10.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62db10f6:::benchmark_data_smithi194_121512_object123626:head# 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62dbf65a:::benchmark_data_smithi194_121512_object10363:head# 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62dc72c3:::benchmark_data_smithi194_121512_object71290:head# 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62dd94e0:::benchmark_data_smithi194_121512_object4305:head# 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ded6e0:::benchmark_data_smithi194_121512_object94675:head# 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62dfa07a:::benchmark_data_smithi194_121512_object80933:head# 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e03ef0:::benchmark_data_smithi194_121512_object25967:head# 2024-04-29T14:27:10.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e05ce0:::benchmark_data_smithi194_121512_object50842:head# 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e10e02:::benchmark_data_smithi194_121512_object48120:head# 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e1561f:::benchmark_data_smithi194_121512_object59185:head# 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e226ec:::benchmark_data_smithi194_121512_object78344:head# 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e23bde:::benchmark_data_smithi194_121512_object127445:head# 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e282a9:::benchmark_data_smithi194_121512_object117576:head# 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e28ea3:::benchmark_data_smithi194_121512_object2713:head# 2024-04-29T14:27:10.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e2b496:::benchmark_data_smithi194_121512_object60213:head# 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e60129:::benchmark_data_smithi194_121512_object55643:head# 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e6b341:::benchmark_data_smithi194_121512_object126998:head# 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e6c71b:::benchmark_data_smithi194_121512_object115484:head# 2024-04-29T14:27:10.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e6e994:::benchmark_data_smithi194_121512_object132419:head# 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e71b8d:::benchmark_data_smithi194_121512_object104921:head# 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e794bb:::benchmark_data_smithi194_121512_object128921:head# 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e8f741:::benchmark_data_smithi194_121512_object94447:head# 2024-04-29T14:27:10.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62e92cc9:::benchmark_data_smithi194_121512_object13377:head# 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ea731e:::benchmark_data_smithi194_121512_object85795:head# 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ebe374:::benchmark_data_smithi194_121512_object100910:head# 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ecb9ac:::benchmark_data_smithi194_121512_object78156:head# 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62eccf9e:::benchmark_data_smithi194_121512_object10657:head# 2024-04-29T14:27:10.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ed5adf:::benchmark_data_smithi194_121512_object81776:head# 2024-04-29T14:27:10.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62eda8b9:::benchmark_data_smithi194_121512_object22892:head# 2024-04-29T14:27:10.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:10.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62edab1d:::benchmark_data_smithi194_121512_object79186:head# 2024-04-29T14:27:10.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ee0691:::benchmark_data_smithi194_121512_object124872:head# 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ee24f4:::benchmark_data_smithi194_121512_object72165:head# 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ef0ff6:::benchmark_data_smithi194_121512_object113182:head# 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ef45b9:::benchmark_data_smithi194_121512_object41984:head# 2024-04-29T14:27:11.002 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62efa4ed:::benchmark_data_smithi194_121512_object8057:head# 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62efcc2d:::benchmark_data_smithi194_121512_object71140:head# 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f03ee4:::benchmark_data_smithi194_121512_object95212:head# 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f04358:::benchmark_data_smithi194_121512_object127378:head# 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f085ab:::benchmark_data_smithi194_121512_object80900:head# 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f0a7e9:::benchmark_data_smithi194_121512_object61208:head# 2024-04-29T14:27:11.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f177f1:::benchmark_data_smithi194_121512_object126258:head# 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f284cc:::benchmark_data_smithi194_121512_object27082:head# 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f29102:::benchmark_data_smithi194_121512_object113117:head# 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f38971:::benchmark_data_smithi194_121512_object111297:head# 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f3c172:::benchmark_data_smithi194_121512_object14295:head# 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f48810:::benchmark_data_smithi194_121512_object85950:head# 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f4b86f:::benchmark_data_smithi194_121512_object87504:head# 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f50bb7:::benchmark_data_smithi194_121512_object27557:head# 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f62d50:::benchmark_data_smithi194_121512_object40720:head# 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f6a53e:::benchmark_data_smithi194_121512_object94926:head# 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f6d0bc:::benchmark_data_smithi194_121512_object81741:head# 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f6f685:::benchmark_data_smithi194_121512_object15514:head# 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f6fde2:::benchmark_data_smithi194_121512_object132118:head# 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f70232:::benchmark_data_smithi194_121512_object129672:head# 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f7050a:::benchmark_data_smithi194_121512_object87721:head# 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f717f3:::benchmark_data_smithi194_121512_object76827:head# 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f7b255:::benchmark_data_smithi194_121512_object91674:head# 2024-04-29T14:27:11.006 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f80154:::benchmark_data_smithi194_121512_object109809:head# 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f82fba:::benchmark_data_smithi194_121512_object8833:head# 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62f85558:::benchmark_data_smithi194_121512_object15916:head# 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fa6033:::benchmark_data_smithi194_121512_object58122:head# 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fac8f9:::benchmark_data_smithi194_121512_object14103:head# 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fafda2:::benchmark_data_smithi194_121512_object26766:head# 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.007 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fb203d:::benchmark_data_smithi194_121512_object50916:head# 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fb21ed:::benchmark_data_smithi194_121512_object9230:head# 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fb3b2e:::benchmark_data_smithi194_121512_object85963:head# 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fbcace:::benchmark_data_smithi194_121512_object53852:head# 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fbea09:::benchmark_data_smithi194_121512_object100932:head# 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fc1ee3:::benchmark_data_smithi194_121512_object9039:head# 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fc8ac8:::benchmark_data_smithi194_121512_object94463:head# 2024-04-29T14:27:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fc8ce0:::benchmark_data_smithi194_121512_object43727:head# 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fcccd2:::benchmark_data_smithi194_121512_object29607:head# 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fd0c82:::benchmark_data_smithi194_121512_object71631:head# 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fd593d:::benchmark_data_smithi194_121512_object121910:head# 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fde262:::benchmark_data_smithi194_121512_object7938:head# 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fe0eaa:::benchmark_data_smithi194_121512_object104662:head# 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.009 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62fe1407:::benchmark_data_smithi194_121512_object48688:head# 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ff6177:::benchmark_data_smithi194_121512_object108433:head# 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:62ffb8f8:::benchmark_data_smithi194_121512_object75419:head# 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63000955:::benchmark_data_smithi194_121512_object51238:head# 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6300edb4:::benchmark_data_smithi194_121512_object94478:head# 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63031c39:::benchmark_data_smithi194_121512_object106998:head# 2024-04-29T14:27:11.010 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63037a0a:::benchmark_data_smithi194_121512_object44905:head# 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6304ad82:::benchmark_data_smithi194_121512_object21085:head# 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6304c67c:::benchmark_data_smithi194_121512_object57640:head# 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630527c6:::benchmark_data_smithi194_121512_object34245:head# 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63056322:::benchmark_data_smithi194_121512_object58242:head# 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6306fcea:::benchmark_data_smithi194_121512_object104281:head# 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.011 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63078ed8:::benchmark_data_smithi194_121512_object80861:head# 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63096109:::benchmark_data_smithi194_121512_object124974:head# 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6309c768:::benchmark_data_smithi194_121512_object65942:head# 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630a15c2:::benchmark_data_smithi194_121512_object37016:head# 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630a6924:::benchmark_data_smithi194_121512_object27556:head# 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630b090b:::benchmark_data_smithi194_121512_object43764:head# 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630b3b86:::benchmark_data_smithi194_121512_object5000:head# 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630bf406:::benchmark_data_smithi194_121512_object63359:head# 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630c31bd:::benchmark_data_smithi194_121512_object46161:head# 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630d00ed:::benchmark_data_smithi194_121512_object99756:head# 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630db6be:::benchmark_data_smithi194_121512_object80781:head# 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630ebc37:::benchmark_data_smithi194_121512_object68116:head# 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630f512f:::benchmark_data_smithi194_121512_object68779:head# 2024-04-29T14:27:11.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:630f74f7:::benchmark_data_smithi194_121512_object100406:head# 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6310889d:::benchmark_data_smithi194_121512_object30582:head# 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6310a3a1:::benchmark_data_smithi194_121512_object96145:head# 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6310ceb9:::benchmark_data_smithi194_121512_object32303:head# 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63110914:::benchmark_data_smithi194_121512_object15890:head# 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63112646:::benchmark_data_smithi194_121512_object128286:head# 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63122170:::benchmark_data_smithi194_121512_object53154:head# 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6313279d:::benchmark_data_smithi194_121512_object5324:head# 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6313366b:::benchmark_data_smithi194_121512_object82594:head# 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6314828e:::benchmark_data_smithi194_121512_object118178:head# 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631489af:::benchmark_data_smithi194_121512_object10888:head# 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6314c7fd:::benchmark_data_smithi194_121512_object32493:head# 2024-04-29T14:27:11.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63152da1:::benchmark_data_smithi194_121512_object58917:head# 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63152ed1:::benchmark_data_smithi194_121512_object103905:head# 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63159941:::benchmark_data_smithi194_121512_object81413:head# 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6315aa35:::benchmark_data_smithi194_121512_object11452:head# 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6316fb3b:::benchmark_data_smithi194_121512_object8636:head# 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63179126:::benchmark_data_smithi194_121512_object110020:head# 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6318ebba:::benchmark_data_smithi194_121512_object115891:head# 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631911e3:::benchmark_data_smithi194_121512_object63395:head# 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63197bfc:::benchmark_data_smithi194_121512_object110852:head# 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631a09ad:::benchmark_data_smithi194_121512_object81567:head# 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631aed62:::benchmark_data_smithi194_121512_object67238:head# 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631bc011:::benchmark_data_smithi194_121512_object129614:head# 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631cc97c:::benchmark_data_smithi194_121512_object61784:head# 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631cd7bc:::benchmark_data_smithi194_121512_object76730:head# 2024-04-29T14:27:11.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631d773e:::benchmark_data_smithi194_121512_object59451:head# 2024-04-29T14:27:11.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631dde59:::benchmark_data_smithi194_121512_object123890:head# 2024-04-29T14:27:11.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631e079e:::benchmark_data_smithi194_121512_object29334:head# 2024-04-29T14:27:11.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631e359e:::benchmark_data_smithi194_121512_object37549:head# 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:631efb0f:::benchmark_data_smithi194_121512_object73437:head# 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6320367e:::benchmark_data_smithi194_121512_object53926:head# 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6321756a:::benchmark_data_smithi194_121512_object52464:head# 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63219ec7:::benchmark_data_smithi194_121512_object132267:head# 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6321d835:::benchmark_data_smithi194_121512_object67442:head# 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632213b3:::benchmark_data_smithi194_121512_object35697:head# 2024-04-29T14:27:11.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632262b9:::benchmark_data_smithi194_121512_object126704:head# 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6322f43f:::benchmark_data_smithi194_121512_object23971:head# 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632375c8:::benchmark_data_smithi194_121512_object56252:head# 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632378e3:::benchmark_data_smithi194_121512_object110459:head# 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632380ad:::benchmark_data_smithi194_121512_object113579:head# 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6323e692:::benchmark_data_smithi194_121512_object87362:head# 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632483b7:::benchmark_data_smithi194_121512_object9759:head# 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632514f6:::benchmark_data_smithi194_121512_object43833:head# 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632618d7:::benchmark_data_smithi194_121512_object34846:head# 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63263ba2:::benchmark_data_smithi194_121512_object124608:head# 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63266e0d:::benchmark_data_smithi194_121512_object85019:head# 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63272649:::benchmark_data_smithi194_121512_object109973:head# 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63277230:::benchmark_data_smithi194_121512_object115771:head# 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63281e65:::benchmark_data_smithi194_121512_object29237:head# 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63287f13:::benchmark_data_smithi194_121512_object66979:head# 2024-04-29T14:27:11.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.029 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:11.035 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63289aff:::benchmark_data_smithi194_121512_object58282:head# 2024-04-29T14:27:11.035 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.035 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6329a665:::benchmark_data_smithi194_121512_object97915:head# 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632ab229:::benchmark_data_smithi194_121512_object40836:head# 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632b0153:::benchmark_data_smithi194_121512_object60267:head# 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632b73ed:::benchmark_data_smithi194_121512_object54540:head# 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632b9cea:::benchmark_data_smithi194_121512_object79607:head# 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632c3884:::benchmark_data_smithi194_121512_object4740:head# 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632cc1d7:::benchmark_data_smithi194_121512_object70901:head# 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632dca0f:::benchmark_data_smithi194_121512_object92427:head# 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632dd10b:::benchmark_data_smithi194_121512_object37684:head# 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632de8a4:::benchmark_data_smithi194_121512_object46862:head# 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632e6ae9:::benchmark_data_smithi194_121512_object78440:head# 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632f7df4:::benchmark_data_smithi194_121512_object97861:head# 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:632fc79c:::benchmark_data_smithi194_121512_object118025:head# 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6330b789:::benchmark_data_smithi194_121512_object58173:head# 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63310b04:::benchmark_data_smithi194_121512_object130674:head# 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63317c4f:::benchmark_data_smithi194_121512_object63591:head# 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6331f872:::benchmark_data_smithi194_121512_object93072:head# 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63321ec0:::benchmark_data_smithi194_121512_object97581:head# 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6332d8ca:::benchmark_data_smithi194_121512_object67359:head# 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6334df91:::benchmark_data_smithi194_121512_object36772:head# 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63351c75:::benchmark_data_smithi194_121512_object27753:head# 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6335a8ca:::benchmark_data_smithi194_121512_object122227:head# 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63363107:::benchmark_data_smithi194_121512_object46832:head# 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63367abc:::benchmark_data_smithi194_121512_object109025:head# 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6336c79e:::benchmark_data_smithi194_121512_object97753:head# 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63376115:::benchmark_data_smithi194_121512_object81560:head# 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63378e6b:::benchmark_data_smithi194_121512_object113645:head# 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63389b2e:::benchmark_data_smithi194_121512_object122306:head# 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6338b459:::benchmark_data_smithi194_121512_object69497:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6338d139:::benchmark_data_smithi194_121512_object100636:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6338e8ad:::benchmark_data_smithi194_121512_object99835:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633a1023:::benchmark_data_smithi194_121512_object79158:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633a8b94:::benchmark_data_smithi194_121512_object111901:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633ac809:::benchmark_data_smithi194_121512_object63639:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633af042:::benchmark_data_smithi194_121512_object95561:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633b0f99:::benchmark_data_smithi194_121512_object90875:head# 2024-04-29T14:27:11.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633b75ba:::benchmark_data_smithi194_121512_object106103:head# 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633bf2f9:::benchmark_data_smithi194_121512_object69377:head# 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633c1723:::benchmark_data_smithi194_121512_object131584:head# 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633c7a60:::benchmark_data_smithi194_121512_object92925:head# 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633cf6ba:::benchmark_data_smithi194_121512_object96807:head# 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633e7962:::benchmark_data_smithi194_121512_object105893:head# 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633f84e0:::benchmark_data_smithi194_121512_object122263:head# 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633f8a1d:::benchmark_data_smithi194_121512_object62720:head# 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633fd514:::benchmark_data_smithi194_121512_object120732:head# 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:633fef8d:::benchmark_data_smithi194_121512_object63970:head# 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6340f4b4:::benchmark_data_smithi194_121512_object66690:head# 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6341626b:::benchmark_data_smithi194_121512_object96063:head# 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6341a83a:::benchmark_data_smithi194_121512_object112951:head# 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6341b2f5:::benchmark_data_smithi194_121512_object8595:head# 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6341e494:::benchmark_data_smithi194_121512_object52223:head# 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6342856d:::benchmark_data_smithi194_121512_object86465:head# 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6343ac54:::benchmark_data_smithi194_121512_object132796:head# 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63448093:::benchmark_data_smithi194_121512_object94661:head# 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6344a292:::benchmark_data_smithi194_121512_object78427:head# 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63450954:::benchmark_data_smithi194_121512_object109565:head# 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63451a19:::benchmark_data_smithi194_121512_object54445:head# 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6345d1ca:::benchmark_data_smithi194_121512_object123709:head# 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63460aed:::benchmark_data_smithi194_121512_object77762:head# 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634621d4:::benchmark_data_smithi194_121512_object47871:head# 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63465333:::benchmark_data_smithi194_121512_object113721:head# 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6346ac81:::benchmark_data_smithi194_121512_object73644:head# 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6346b2c5:::benchmark_data_smithi194_121512_object23046:head# 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6346dcf8:::benchmark_data_smithi194_121512_object2340:head# 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63475150:::benchmark_data_smithi194_121512_object98624:head# 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634775e0:::benchmark_data_smithi194_121512_object115527:head# 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6349dbf9:::benchmark_data_smithi194_121512_object99133:head# 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634ada4a:::benchmark_data_smithi194_121512_object49629:head# 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634b3d7b:::benchmark_data_smithi194_121512_object34412:head# 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634bd299:::benchmark_data_smithi194_121512_object92295:head# 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634c3edd:::benchmark_data_smithi194_121512_object18782:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634cd29b:::benchmark_data_smithi194_121512_object131806:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634d178a:::benchmark_data_smithi194_121512_object98883:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634d81fb:::benchmark_data_smithi194_121512_object76510:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634dca65:::benchmark_data_smithi194_121512_object111176:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634dcd93:::benchmark_data_smithi194_121512_object11563:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634e422c:::benchmark_data_smithi194_121512_object72519:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634f445a:::benchmark_data_smithi194_121512_object89724:head# 2024-04-29T14:27:11.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:634f8449:::benchmark_data_smithi194_121512_object7433:head# 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6350100b:::benchmark_data_smithi194_121512_object107762:head# 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63503d37:::benchmark_data_smithi194_121512_object81028:head# 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635040c1:::benchmark_data_smithi194_121512_object40158:head# 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63510e69:::benchmark_data_smithi194_121512_object29005:head# 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635182e7:::benchmark_data_smithi194_121512_object124080:head# 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6351ca16:::benchmark_data_smithi194_121512_object74342:head# 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63521ba4:::benchmark_data_smithi194_121512_object84952:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63529baa:::benchmark_data_smithi194_121512_object17695:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6352eccc:::benchmark_data_smithi194_121512_object130395:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63532ce6:::benchmark_data_smithi194_121512_object92867:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6353dc7b:::benchmark_data_smithi194_121512_object22323:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6354a67e:::benchmark_data_smithi194_121512_object45765:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6354c9e1:::benchmark_data_smithi194_121512_object90389:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635569a2:::benchmark_data_smithi194_121512_object93122:head# 2024-04-29T14:27:11.048 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63559cbb:::benchmark_data_smithi194_121512_object121493:head# 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6355e5ae:::benchmark_data_smithi194_121512_object53594:head# 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6355f367:::benchmark_data_smithi194_121512_object43546:head# 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6356f277:::benchmark_data_smithi194_121512_object24616:head# 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63576233:::benchmark_data_smithi194_121512_object127052:head# 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63576df4:::benchmark_data_smithi194_121512_object73257:head# 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635801d7:::benchmark_data_smithi194_121512_object11149:head# 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.049 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63582201:::benchmark_data_smithi194_121512_object34401:head# 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63591efb:::benchmark_data_smithi194_121512_object83096:head# 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6359624f:::benchmark_data_smithi194_121512_object104177:head# 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6359bc55:::benchmark_data_smithi194_121512_object22197:head# 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635a6cb8:::benchmark_data_smithi194_121512_object122403:head# 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635ab7a2:::benchmark_data_smithi194_121512_object39394:head# 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635ac381:::benchmark_data_smithi194_121512_object51140:head# 2024-04-29T14:27:11.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635af05b:::benchmark_data_smithi194_121512_object103254:head# 2024-04-29T14:27:11.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635afcfb:::benchmark_data_smithi194_121512_object126117:head# 2024-04-29T14:27:11.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635bbb8f:::benchmark_data_smithi194_121512_object106933:head# 2024-04-29T14:27:11.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635c0b2e:::benchmark_data_smithi194_121512_object39583:head# 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635c28a9:::benchmark_data_smithi194_121512_object123250:head# 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635ccd9a:::benchmark_data_smithi194_121512_object47816:head# 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635d85e0:::benchmark_data_smithi194_121512_object77966:head# 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635d9ac1:::benchmark_data_smithi194_121512_object98850:head# 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635dbb4e:::benchmark_data_smithi194_121512_object61606:head# 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635dd63b:::benchmark_data_smithi194_121512_object41046:head# 2024-04-29T14:27:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635de5ce:::benchmark_data_smithi194_121512_object114153:head# 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635e0433:::benchmark_data_smithi194_121512_object5245:head# 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635e269b:::benchmark_data_smithi194_121512_object104689:head# 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635e5c83:::benchmark_data_smithi194_121512_object112382:head# 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:635f109f:::benchmark_data_smithi194_121512_object108092:head# 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636056c4:::benchmark_data_smithi194_121512_object62979:head# 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6360d63b:::benchmark_data_smithi194_121512_object74899:head# 2024-04-29T14:27:11.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636359d2:::benchmark_data_smithi194_121512_object80488:head# 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63641bcb:::benchmark_data_smithi194_121512_object29219:head# 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63647420:::benchmark_data_smithi194_121512_object52990:head# 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63647c77:::benchmark_data_smithi194_121512_object67080:head# 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63652430:::benchmark_data_smithi194_121512_object18630:head# 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63655f85:::benchmark_data_smithi194_121512_object38434:head# 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63657612:::benchmark_data_smithi194_121512_object67889:head# 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6365eb27:::benchmark_data_smithi194_121512_object77875:head# 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63670f65:::benchmark_data_smithi194_121512_object51303:head# 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6367c911:::benchmark_data_smithi194_121512_object77840:head# 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63692c4a:::benchmark_data_smithi194_121512_object82261:head# 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63694365:::benchmark_data_smithi194_121512_object53747:head# 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636963e5:::benchmark_data_smithi194_121512_object121639:head# 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636a42e7:::benchmark_data_smithi194_121512_object115107:head# 2024-04-29T14:27:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636ae4e8:::benchmark_data_smithi194_121512_object7057:head# 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636b9781:::benchmark_data_smithi194_121512_object23518:head# 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636bbeb8:::benchmark_data_smithi194_121512_object27755:head# 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636caaff:::benchmark_data_smithi194_121512_object57249:head# 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636cbc8e:::benchmark_data_smithi194_121512_object31188:head# 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636dcf90:::benchmark_data_smithi194_121512_object61355:head# 2024-04-29T14:27:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636e03e2:::benchmark_data_smithi194_121512_object14351:head# 2024-04-29T14:27:11.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636e3ab8:::benchmark_data_smithi194_121512_object101175:head# 2024-04-29T14:27:11.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636e51c6:::benchmark_data_smithi194_121512_object60594:head# 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636e5339:::benchmark_data_smithi194_121512_object81718:head# 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:636e5ee4:::benchmark_data_smithi194_121512_object33591:head# 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6370ba14:::benchmark_data_smithi194_121512_object10693:head# 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6370d797:::benchmark_data_smithi194_121512_object63131:head# 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6371621c:::benchmark_data_smithi194_121512_object96773:head# 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6372b105:::benchmark_data_smithi194_121512_object30304:head# 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63732ecd:::benchmark_data_smithi194_121512_object130390:head# 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637375f4:::benchmark_data_smithi194_121512_object62497:head# 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6373b3a9:::benchmark_data_smithi194_121512_object120680:head# 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63745c6a:::benchmark_data_smithi194_121512_object76998:head# 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6374ba71:::benchmark_data_smithi194_121512_object105879:head# 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6375d811:::benchmark_data_smithi194_121512_object4769:head# 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63766770:::benchmark_data_smithi194_121512_object71040:head# 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63767724:::benchmark_data_smithi194_121512_object66827:head# 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637717ad:::benchmark_data_smithi194_121512_object1958:head# 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637798e3:::benchmark_data_smithi194_121512_object106297:head# 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63781ac2:::benchmark_data_smithi194_121512_object113804:head# 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6378db22:::benchmark_data_smithi194_121512_object48066:head# 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637944f9:::benchmark_data_smithi194_121512_object10482:head# 2024-04-29T14:27:11.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6379924b:::benchmark_data_smithi194_121512_object107795:head# 2024-04-29T14:27:11.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.068 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6379afb7:::benchmark_data_smithi194_121512_object93241:head# 2024-04-29T14:27:11.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6379e3b7:::benchmark_data_smithi194_121512_object31830:head# 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6379e495:::benchmark_data_smithi194_121512_object71368:head# 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637a1dfb:::benchmark_data_smithi194_121512_object4412:head# 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637a3bd8:::benchmark_data_smithi194_121512_object44935:head# 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637b7411:::benchmark_data_smithi194_121512_object30936:head# 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637b9967:::benchmark_data_smithi194_121512_object31147:head# 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.069 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637c51f4:::benchmark_data_smithi194_121512_object133237:head# 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637ca89a:::benchmark_data_smithi194_121512_object80448:head# 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637da7df:::benchmark_data_smithi194_121512_object11011:head# 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637db6ff:::benchmark_data_smithi194_121512_object81013:head# 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637ed515:::benchmark_data_smithi194_121512_object27993:head# 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637f28b7:::benchmark_data_smithi194_121512_object69884:head# 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637f6e60:::benchmark_data_smithi194_121512_object60564:head# 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.070 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:637ffa0c:::benchmark_data_smithi194_121512_object102103:head# 2024-04-29T14:27:11.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6381497b:::benchmark_data_smithi194_121512_object28582:head# 2024-04-29T14:27:11.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.071 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63815c3a:::benchmark_data_smithi194_121512_object81585:head# 2024-04-29T14:27:11.071 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63816aff:::benchmark_data_smithi194_121512_object75085:head# 2024-04-29T14:27:11.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638181f2:::benchmark_data_smithi194_121512_object28946:head# 2024-04-29T14:27:11.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6382121b:::benchmark_data_smithi194_121512_object131380:head# 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63824d4f:::benchmark_data_smithi194_121512_object90619:head# 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63837725:::benchmark_data_smithi194_121512_object99928:head# 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6383c676:::benchmark_data_smithi194_121512_object24890:head# 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6383fe40:::benchmark_data_smithi194_121512_object28281:head# 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6384025d:::benchmark_data_smithi194_121512_object80022:head# 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63840837:::benchmark_data_smithi194_121512_object82495:head# 2024-04-29T14:27:11.075 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6384b154:::benchmark_data_smithi194_121512_object128547:head# 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6384b1a2:::benchmark_data_smithi194_121512_object90460:head# 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6384b9ad:::benchmark_data_smithi194_121512_object88834:head# 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6384d5cd:::benchmark_data_smithi194_121512_object14286:head# 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6384ecdb:::benchmark_data_smithi194_121512_object126189:head# 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6384f9ff:::benchmark_data_smithi194_121512_object40300:head# 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638549bc:::benchmark_data_smithi194_121512_object53266:head# 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.076 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63860a36:::benchmark_data_smithi194_121512_object21335:head# 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6386d937:::benchmark_data_smithi194_121512_object91135:head# 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63877d59:::benchmark_data_smithi194_121512_object35183:head# 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6387ece3:::benchmark_data_smithi194_121512_object92740:head# 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638862ae:::benchmark_data_smithi194_121512_object9001:head# 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63890918:::benchmark_data_smithi194_121512_object116139:head# 2024-04-29T14:27:11.077 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638976d1:::benchmark_data_smithi194_121512_object118498:head# 2024-04-29T14:27:11.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638b225b:::benchmark_data_smithi194_121512_object92842:head# 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638cd8ba:::benchmark_data_smithi194_121512_object63404:head# 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638d1b23:::benchmark_data_smithi194_121512_object54449:head# 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638ea9ea:::benchmark_data_smithi194_121512_object41000:head# 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638eae4f:::benchmark_data_smithi194_121512_object50244:head# 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:638f7f3c:::benchmark_data_smithi194_121512_object68164:head# 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63901115:::benchmark_data_smithi194_121512_object17490:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63903f4e:::benchmark_data_smithi194_121512_object89880:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639066f3:::benchmark_data_smithi194_121512_object125542:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6391c2df:::benchmark_data_smithi194_121512_object9233:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63926047:::benchmark_data_smithi194_121512_object9384:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6392fccd:::benchmark_data_smithi194_121512_object43444:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639304d5:::benchmark_data_smithi194_121512_object58711:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6393150e:::benchmark_data_smithi194_121512_object106885:head# 2024-04-29T14:27:11.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63938292:::benchmark_data_smithi194_121512_object118636:head# 2024-04-29T14:27:11.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6393e9a3:::benchmark_data_smithi194_121512_object29011:head# 2024-04-29T14:27:11.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6393eb59:::benchmark_data_smithi194_121512_object10551:head# 2024-04-29T14:27:11.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63949075:::benchmark_data_smithi194_121512_object103488:head# 2024-04-29T14:27:11.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6394a215:::benchmark_data_smithi194_121512_object91830:head# 2024-04-29T14:27:11.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6395d225:::benchmark_data_smithi194_121512_object14310:head# 2024-04-29T14:27:11.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6395f8c1:::benchmark_data_smithi194_121512_object13701:head# 2024-04-29T14:27:11.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63960695:::benchmark_data_smithi194_121512_object67668:head# 2024-04-29T14:27:11.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6397637d:::benchmark_data_smithi194_121512_object84998:head# 2024-04-29T14:27:11.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6398f24c:::benchmark_data_smithi194_121512_object20978:head# 2024-04-29T14:27:11.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6398f69f:::benchmark_data_smithi194_121512_object81603:head# 2024-04-29T14:27:11.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639907be:::benchmark_data_smithi194_121512_object94351:head# 2024-04-29T14:27:11.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6399a434:::benchmark_data_smithi194_121512_object52621:head# 2024-04-29T14:27:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639b3164:::benchmark_data_smithi194_121512_object79852:head# 2024-04-29T14:27:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639b97ad:::benchmark_data_smithi194_121512_object84810:head# 2024-04-29T14:27:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639b9c2a:::benchmark_data_smithi194_121512_object108148:head# 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639ba6ae:::benchmark_data_smithi194_121512_object57163:head# 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639bbae8:::benchmark_data_smithi194_121512_object47441:head# 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639bc07a:::benchmark_data_smithi194_121512_object101325:head# 2024-04-29T14:27:11.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639c96b8:::benchmark_data_smithi194_121512_object68512:head# 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639cf621:::benchmark_data_smithi194_121512_object35425:head# 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639d20c1:::benchmark_data_smithi194_121512_object8654:head# 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639d5891:::benchmark_data_smithi194_121512_object69857:head# 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639d5fc2:::benchmark_data_smithi194_121512_object24750:head# 2024-04-29T14:27:11.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639e2250:::benchmark_data_smithi194_121512_object111852:head# 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639ef868:::benchmark_data_smithi194_121512_object5745:head# 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639f2888:::benchmark_data_smithi194_121512_object34878:head# 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:639f51a7:::benchmark_data_smithi194_121512_object1772:head# 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a074ed:::benchmark_data_smithi194_121512_object16665:head# 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a10225:::benchmark_data_smithi194_121512_object79494:head# 2024-04-29T14:27:11.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a21831:::benchmark_data_smithi194_121512_object54211:head# 2024-04-29T14:27:11.103 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a323aa:::benchmark_data_smithi194_121512_object112946:head# 2024-04-29T14:27:11.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a3e4a0:::benchmark_data_smithi194_121512_object75663:head# 2024-04-29T14:27:11.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a46a0a:::benchmark_data_smithi194_121512_object116166:head# 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a48264:::benchmark_data_smithi194_121512_object109036:head# 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a4ba2c:::benchmark_data_smithi194_121512_object112361:head# 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a524f3:::benchmark_data_smithi194_121512_object94037:head# 2024-04-29T14:27:11.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a55f64:::benchmark_data_smithi194_121512_object45556:head# 2024-04-29T14:27:11.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a643e1:::benchmark_data_smithi194_121512_object7208:head# 2024-04-29T14:27:11.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a6d407:::benchmark_data_smithi194_121512_object43220:head# 2024-04-29T14:27:11.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a6d738:::benchmark_data_smithi194_121512_object51039:head# 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a812ba:::benchmark_data_smithi194_121512_object5774:head# 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a83114:::benchmark_data_smithi194_121512_object50179:head# 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a8ad34:::benchmark_data_smithi194_121512_object99105:head# 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a8eea2:::benchmark_data_smithi194_121512_object90139:head# 2024-04-29T14:27:11.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a8f065:::benchmark_data_smithi194_121512_object69063:head# 2024-04-29T14:27:11.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a97698:::benchmark_data_smithi194_121512_object58429:head# 2024-04-29T14:27:11.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63a9e884:::benchmark_data_smithi194_121512_object103420:head# 2024-04-29T14:27:11.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63aa0776:::benchmark_data_smithi194_121512_object19340:head# 2024-04-29T14:27:11.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ab8ccf:::benchmark_data_smithi194_121512_object109765:head# 2024-04-29T14:27:11.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63abd129:::benchmark_data_smithi194_121512_object93779:head# 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ac0bd5:::benchmark_data_smithi194_121512_object78938:head# 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ac0fcb:::benchmark_data_smithi194_121512_object127469:head# 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ac85a1:::benchmark_data_smithi194_121512_object116424:head# 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ad48cc:::benchmark_data_smithi194_121512_object39955:head# 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ad4aa0:::benchmark_data_smithi194_121512_object67206:head# 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ad63f4:::benchmark_data_smithi194_121512_object11991:head# 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63adbc26:::benchmark_data_smithi194_121512_object121932:head# 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ae3cc8:::benchmark_data_smithi194_121512_object78972:head# 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ae4e1f:::benchmark_data_smithi194_121512_object105144:head# 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63af4135:::benchmark_data_smithi194_121512_object25488:head# 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63af596b:::benchmark_data_smithi194_121512_object16744:head# 2024-04-29T14:27:11.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63afa618:::benchmark_data_smithi194_121512_object74858:head# 2024-04-29T14:27:11.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b1219b:::benchmark_data_smithi194_121512_object25690:head# 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b12948:::benchmark_data_smithi194_121512_object21464:head# 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b173f1:::benchmark_data_smithi194_121512_object27145:head# 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b1d6f4:::benchmark_data_smithi194_121512_object42133:head# 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b1f25b:::benchmark_data_smithi194_121512_object127355:head# 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b2a856:::benchmark_data_smithi194_121512_object66093:head# 2024-04-29T14:27:11.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b2be51:::benchmark_data_smithi194_121512_object47225:head# 2024-04-29T14:27:11.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b38a95:::benchmark_data_smithi194_121512_object87875:head# 2024-04-29T14:27:11.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b498de:::benchmark_data_smithi194_121512_object122421:head# 2024-04-29T14:27:11.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b4a196:::benchmark_data_smithi194_121512_object62168:head# 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b51ecf:::benchmark_data_smithi194_121512_object98162:head# 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b6ec4f:::benchmark_data_smithi194_121512_object26566:head# 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b75168:::benchmark_data_smithi194_121512_object79632:head# 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b85a23:::benchmark_data_smithi194_121512_object8140:head# 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b93957:::benchmark_data_smithi194_121512_object23556:head# 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63b9471b:::benchmark_data_smithi194_121512_object116400:head# 2024-04-29T14:27:11.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bac310:::benchmark_data_smithi194_121512_object73250:head# 2024-04-29T14:27:11.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bb0909:::benchmark_data_smithi194_121512_object21606:head# 2024-04-29T14:27:11.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bb1416:::benchmark_data_smithi194_121512_object59894:head# 2024-04-29T14:27:11.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bb3293:::benchmark_data_smithi194_121512_object130830:head# 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bb6876:::benchmark_data_smithi194_121512_object1805:head# 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bbc0a7:::benchmark_data_smithi194_121512_object77319:head# 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bbd185:::benchmark_data_smithi194_121512_object40629:head# 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bbfadb:::benchmark_data_smithi194_121512_object25834:head# 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bc87a2:::benchmark_data_smithi194_121512_object37377:head# 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bd0fed:::benchmark_data_smithi194_121512_object84787:head# 2024-04-29T14:27:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bd404e:::benchmark_data_smithi194_121512_object22401:head# 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bd748d:::benchmark_data_smithi194_121512_object117830:head# 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bdec19:::benchmark_data_smithi194_121512_object46659:head# 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63be9ebd:::benchmark_data_smithi194_121512_object96370:head# 2024-04-29T14:27:11.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63bf0f38:::benchmark_data_smithi194_121512_object97052:head# 2024-04-29T14:27:11.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c02e2b:::benchmark_data_smithi194_121512_object52628:head# 2024-04-29T14:27:11.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c070bc:::benchmark_data_smithi194_121512_object89072:head# 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c096cd:::benchmark_data_smithi194_121512_object571:head# 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c0b3c1:::benchmark_data_smithi194_121512_object36342:head# 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c23b46:::benchmark_data_smithi194_121512_object39088:head# 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c2f52a:::benchmark_data_smithi194_121512_object119935:head# 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c335b7:::benchmark_data_smithi194_121512_object8606:head# 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c36182:::benchmark_data_smithi194_121512_object22665:head# 2024-04-29T14:27:11.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c3aec8:::benchmark_data_smithi194_121512_object18457:head# 2024-04-29T14:27:11.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c3c309:::benchmark_data_smithi194_121512_object13890:head# 2024-04-29T14:27:11.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.139 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c3e9bc:::benchmark_data_smithi194_121512_object92770:head# 2024-04-29T14:27:11.139 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c494d7:::benchmark_data_smithi194_121512_object80798:head# 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c57d74:::benchmark_data_smithi194_121512_object20364:head# 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c580bb:::benchmark_data_smithi194_121512_object108140:head# 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c645bd:::benchmark_data_smithi194_121512_object21105:head# 2024-04-29T14:27:11.140 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c73512:::benchmark_data_smithi194_121512_object126168:head# 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c78c5a:::benchmark_data_smithi194_121512_object39616:head# 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c82e87:::benchmark_data_smithi194_121512_object11500:head# 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c83ac4:::benchmark_data_smithi194_121512_object92632:head# 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c8a935:::benchmark_data_smithi194_121512_object55400:head# 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c9654b:::benchmark_data_smithi194_121512_object66157:head# 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c976ff:::benchmark_data_smithi194_121512_object42963:head# 2024-04-29T14:27:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63c9bb9d:::benchmark_data_smithi194_121512_object55963:head# 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ca0ad7:::benchmark_data_smithi194_121512_object130959:head# 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ca1a41:::benchmark_data_smithi194_121512_object99027:head# 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cb08cc:::benchmark_data_smithi194_121512_object7995:head# 2024-04-29T14:27:11.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cc7510:::benchmark_data_smithi194_121512_object16921:head# 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cc7e94:::benchmark_data_smithi194_121512_object90451:head# 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cd08be:::benchmark_data_smithi194_121512_object21877:head# 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cd6f16:::benchmark_data_smithi194_121512_object90458:head# 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cd829b:::benchmark_data_smithi194_121512_object6370:head# 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ce075e:::benchmark_data_smithi194_121512_object100955:head# 2024-04-29T14:27:11.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ce31af:::benchmark_data_smithi194_121512_object55084:head# 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ceb044:::benchmark_data_smithi194_121512_object126444:head# 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cf454e:::benchmark_data_smithi194_121512_object116548:head# 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cf7cdc:::benchmark_data_smithi194_121512_object77943:head# 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63cf7d98:::benchmark_data_smithi194_121512_object47008:head# 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d06dc2:::benchmark_data_smithi194_121512_object122594:head# 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d14dbd:::benchmark_data_smithi194_121512_object127645:head# 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d27937:::benchmark_data_smithi194_121512_object84863:head# 2024-04-29T14:27:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d324d8:::benchmark_data_smithi194_121512_object61239:head# 2024-04-29T14:27:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d36376:::benchmark_data_smithi194_121512_object83433:head# 2024-04-29T14:27:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d3d001:::benchmark_data_smithi194_121512_object45655:head# 2024-04-29T14:27:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d3f815:::benchmark_data_smithi194_121512_object77931:head# 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d4043a:::benchmark_data_smithi194_121512_object96774:head# 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d536ff:::benchmark_data_smithi194_121512_object42949:head# 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d54e9a:::benchmark_data_smithi194_121512_object37558:head# 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d58539:::benchmark_data_smithi194_121512_object91696:head# 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d5d565:::benchmark_data_smithi194_121512_object60587:head# 2024-04-29T14:27:11.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d66a4b:::benchmark_data_smithi194_121512_object23805:head# 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d6f63f:::benchmark_data_smithi194_121512_object16678:head# 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d7137b:::benchmark_data_smithi194_121512_object62121:head# 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d74e8a:::benchmark_data_smithi194_121512_object115844:head# 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d78bdc:::benchmark_data_smithi194_121512_object16266:head# 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d7a232:::benchmark_data_smithi194_121512_object65914:head# 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.157 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d7df08:::benchmark_data_smithi194_121512_object1078:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d863dc:::benchmark_data_smithi194_121512_object109035:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d896ad:::benchmark_data_smithi194_121512_object74002:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d8ac30:::benchmark_data_smithi194_121512_object50221:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63d9fb45:::benchmark_data_smithi194_121512_object53862:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63da0e70:::benchmark_data_smithi194_121512_object128329:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63da51fa:::benchmark_data_smithi194_121512_object92592:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63da65ad:::benchmark_data_smithi194_121512_object126865:head# 2024-04-29T14:27:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63da8c2c:::benchmark_data_smithi194_121512_object44240:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63daa0ae:::benchmark_data_smithi194_121512_object81817:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63db7c4c:::benchmark_data_smithi194_121512_object129649:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63dbb850:::benchmark_data_smithi194_121512_object33567:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63dbd6e8:::benchmark_data_smithi194_121512_object109451:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63dc9b69:::benchmark_data_smithi194_121512_object78571:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63dcc9c3:::benchmark_data_smithi194_121512_object123127:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63dd2e05:::benchmark_data_smithi194_121512_object126730:head# 2024-04-29T14:27:11.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e16033:::benchmark_data_smithi194_121512_object109096:head# 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e2699d:::benchmark_data_smithi194_121512_object111991:head# 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e2b7e9:::benchmark_data_smithi194_121512_object73704:head# 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e3422d:::benchmark_data_smithi194_121512_object9193:head# 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e38655:::benchmark_data_smithi194_121512_object38523:head# 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e3b630:::benchmark_data_smithi194_121512_object87733:head# 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e40034:::benchmark_data_smithi194_121512_object39948:head# 2024-04-29T14:27:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.161 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:11.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e4659d:::benchmark_data_smithi194_121512_object113823:head# 2024-04-29T14:27:11.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e4ecb0:::benchmark_data_smithi194_121512_object23948:head# 2024-04-29T14:27:11.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e4f370:::benchmark_data_smithi194_121512_object97278:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e4fc23:::benchmark_data_smithi194_121512_object65661:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e502a4:::benchmark_data_smithi194_121512_object126331:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e64865:::benchmark_data_smithi194_121512_object108011:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e7c234:::benchmark_data_smithi194_121512_object22668:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e84739:::benchmark_data_smithi194_121512_object120770:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e84fde:::benchmark_data_smithi194_121512_object6462:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e85dd0:::benchmark_data_smithi194_121512_object108530:head# 2024-04-29T14:27:11.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e8d6a7:::benchmark_data_smithi194_121512_object22447:head# 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e965d2:::benchmark_data_smithi194_121512_object35844:head# 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e9a02d:::benchmark_data_smithi194_121512_object96411:head# 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63e9d2a7:::benchmark_data_smithi194_121512_object21384:head# 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ea1a0b:::benchmark_data_smithi194_121512_object32249:head# 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ea88ba:::benchmark_data_smithi194_121512_object81165:head# 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ea943c:::benchmark_data_smithi194_121512_object79290:head# 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63eaa8d2:::benchmark_data_smithi194_121512_object39590:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63eac077:::benchmark_data_smithi194_121512_object77276:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63eb7822:::benchmark_data_smithi194_121512_object100992:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ebfb2e:::benchmark_data_smithi194_121512_object125448:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ec26e6:::benchmark_data_smithi194_121512_object125446:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ef394a:::benchmark_data_smithi194_121512_object12924:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ef8cf3:::benchmark_data_smithi194_121512_object29082:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f0ab2d:::benchmark_data_smithi194_121512_object79440:head# 2024-04-29T14:27:11.167 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f0ff8e:::benchmark_data_smithi194_121512_object44607:head# 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f18198:::benchmark_data_smithi194_121512_object118212:head# 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f29cf0:::benchmark_data_smithi194_121512_object8518:head# 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f306dd:::benchmark_data_smithi194_121512_object102222:head# 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f34a3b:::benchmark_data_smithi194_121512_object19149:head# 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f382de:::benchmark_data_smithi194_121512_object64602:head# 2024-04-29T14:27:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f3d589:::benchmark_data_smithi194_121512_object124648:head# 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f46a8c:::benchmark_data_smithi194_121512_object44963:head# 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f55046:::benchmark_data_smithi194_121512_object81958:head# 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f55746:::benchmark_data_smithi194_121512_object74505:head# 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f63c8f:::benchmark_data_smithi194_121512_object40202:head# 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f7bc69:::benchmark_data_smithi194_121512_object8177:head# 2024-04-29T14:27:11.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f7ed91:::benchmark_data_smithi194_121512_object96516:head# 2024-04-29T14:27:11.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.172 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63f96547:::benchmark_data_smithi194_121512_object108730:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fa02c6:::benchmark_data_smithi194_121512_object49723:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fa35f7:::benchmark_data_smithi194_121512_object103310:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fa405f:::benchmark_data_smithi194_121512_object99419:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fa97a1:::benchmark_data_smithi194_121512_object43336:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fab3a7:::benchmark_data_smithi194_121512_object107841:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fba485:::benchmark_data_smithi194_121512_object63147:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fbb905:::benchmark_data_smithi194_121512_object92205:head# 2024-04-29T14:27:11.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.174 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fcae6e:::benchmark_data_smithi194_121512_object62293:head# 2024-04-29T14:27:11.174 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.174 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fd1dcc:::benchmark_data_smithi194_121512_object99604:head# 2024-04-29T14:27:11.174 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.178 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fd9c0e:::benchmark_data_smithi194_121512_object94670:head# 2024-04-29T14:27:11.178 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.179 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fe4889:::benchmark_data_smithi194_121512_object71715:head# 2024-04-29T14:27:11.179 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.179 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fe5b06:::benchmark_data_smithi194_121512_object62829:head# 2024-04-29T14:27:11.179 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fe85a5:::benchmark_data_smithi194_121512_object16016:head# 2024-04-29T14:27:11.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fed532:::benchmark_data_smithi194_121512_object42887:head# 2024-04-29T14:27:11.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ff2b6a:::benchmark_data_smithi194_121512_object97354:head# 2024-04-29T14:27:11.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63ff5f79:::benchmark_data_smithi194_121512_object63218:head# 2024-04-29T14:27:11.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fff8e8:::benchmark_data_smithi194_121512_object26866:head# 2024-04-29T14:27:11.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:63fffeac:::benchmark_data_smithi194_121512_object92376:head# 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64003101:::benchmark_data_smithi194_121512_object42359:head# 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640054e4:::benchmark_data_smithi194_121512_object87251:head# 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6401251f:::benchmark_data_smithi194_121512_object24800:head# 2024-04-29T14:27:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64019614:::benchmark_data_smithi194_121512_object53392:head# 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6401b09f:::benchmark_data_smithi194_121512_object102961:head# 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64021394:::benchmark_data_smithi194_121512_object91406:head# 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64022415:::benchmark_data_smithi194_121512_object10589:head# 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6402931c:::benchmark_data_smithi194_121512_object74456:head# 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64045ad3:::benchmark_data_smithi194_121512_object43391:head# 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64049d1f:::benchmark_data_smithi194_121512_object58996:head# 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6404e5f6:::benchmark_data_smithi194_121512_object101003:head# 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640562ae:::benchmark_data_smithi194_121512_object107763:head# 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640617fc:::benchmark_data_smithi194_121512_object54000:head# 2024-04-29T14:27:11.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64069e1c:::benchmark_data_smithi194_121512_object129294:head# 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6406b78a:::benchmark_data_smithi194_121512_object2659:head# 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6406c033:::benchmark_data_smithi194_121512_object87853:head# 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6407263f:::benchmark_data_smithi194_121512_object76770:head# 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640764ba:::benchmark_data_smithi194_121512_object63304:head# 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64079245:::benchmark_data_smithi194_121512_object65407:head# 2024-04-29T14:27:11.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6407b836:::benchmark_data_smithi194_121512_object27237:head# 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6408d545:::benchmark_data_smithi194_121512_object127616:head# 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6409c4bc:::benchmark_data_smithi194_121512_object60655:head# 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640a124a:::benchmark_data_smithi194_121512_object60186:head# 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640a4c23:::benchmark_data_smithi194_121512_object119251:head# 2024-04-29T14:27:11.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.193 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:11.193 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640a8f6e:::benchmark_data_smithi194_121512_object62117:head# 2024-04-29T14:27:11.193 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.193 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640ae9cd:::benchmark_data_smithi194_121512_object63034:head# 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640b1132:::benchmark_data_smithi194_121512_object86858:head# 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640b2e9b:::benchmark_data_smithi194_121512_object133448:head# 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640b74e0:::benchmark_data_smithi194_121512_object81270:head# 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640d646c:::benchmark_data_smithi194_121512_object29593:head# 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640d8b28:::benchmark_data_smithi194_121512_object129457:head# 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640dc43b:::benchmark_data_smithi194_121512_object100622:head# 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640df3f3:::benchmark_data_smithi194_121512_object108065:head# 2024-04-29T14:27:11.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640df452:::benchmark_data_smithi194_121512_object130265:head# 2024-04-29T14:27:11.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640e858e:::benchmark_data_smithi194_121512_object24420:head# 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640f3e43:::benchmark_data_smithi194_121512_object866:head# 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640fbc9b:::benchmark_data_smithi194_121512_object120452:head# 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:640fe840:::benchmark_data_smithi194_121512_object67353:head# 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6410d6de:::benchmark_data_smithi194_121512_object119103:head# 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641182c0:::benchmark_data_smithi194_121512_object56121:head# 2024-04-29T14:27:11.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641221b4:::benchmark_data_smithi194_121512_object75505:head# 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64123dfe:::benchmark_data_smithi194_121512_object139:head# 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6412fd8c:::benchmark_data_smithi194_121512_object10009:head# 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6413866a:::benchmark_data_smithi194_121512_object81485:head# 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6415492b:::benchmark_data_smithi194_121512_object73705:head# 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6415e8f6:::benchmark_data_smithi194_121512_object42726:head# 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6416074d:::benchmark_data_smithi194_121512_object26923:head# 2024-04-29T14:27:11.199 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.199 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641609c4:::benchmark_data_smithi194_121512_object132596:head# 2024-04-29T14:27:11.199 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6417914e:::benchmark_data_smithi194_121512_object20795:head# 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64179786:::benchmark_data_smithi194_121512_object79368:head# 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6418824f:::benchmark_data_smithi194_121512_object95117:head# 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6418e5ce:::benchmark_data_smithi194_121512_object38513:head# 2024-04-29T14:27:11.200 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:11.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:11.205 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64193f10:::benchmark_data_smithi194_121512_object68985:head# 2024-04-29T14:27:11.205 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6419ffe8:::benchmark_data_smithi194_121512_object67755:head# 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641a9b5c:::benchmark_data_smithi194_121512_object82310:head# 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641ab7df:::benchmark_data_smithi194_121512_object32239:head# 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641ac837:::benchmark_data_smithi194_121512_object40226:head# 2024-04-29T14:27:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641ae881:::benchmark_data_smithi194_121512_object57449:head# 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641b4d4d:::benchmark_data_smithi194_121512_object124393:head# 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641b55e7:::benchmark_data_smithi194_121512_object117209:head# 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641b9257:::benchmark_data_smithi194_121512_object267:head# 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641bb064:::benchmark_data_smithi194_121512_object64582:head# 2024-04-29T14:27:11.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641c93aa:::benchmark_data_smithi194_121512_object37156:head# 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641d2b21:::benchmark_data_smithi194_121512_object18687:head# 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641da77a:::benchmark_data_smithi194_121512_object79088:head# 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641db815:::benchmark_data_smithi194_121512_object117165:head# 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641e4175:::benchmark_data_smithi194_121512_object27513:head# 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641ee982:::benchmark_data_smithi194_121512_object19461:head# 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641eea47:::benchmark_data_smithi194_121512_object26487:head# 2024-04-29T14:27:11.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641f01be:::benchmark_data_smithi194_121512_object56172:head# 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:641fe5e1:::benchmark_data_smithi194_121512_object69651:head# 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642038d4:::benchmark_data_smithi194_121512_object74839:head# 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6420e173:::benchmark_data_smithi194_121512_object10275:head# 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64212e98:::benchmark_data_smithi194_121512_object46543:head# 2024-04-29T14:27:11.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64214391:::benchmark_data_smithi194_121512_object33710:head# 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64223601:::benchmark_data_smithi194_121512_object119246:head# 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6422ee4b:::benchmark_data_smithi194_121512_object85867:head# 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64244341:::benchmark_data_smithi194_121512_object29727:head# 2024-04-29T14:27:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6424b5b2:::benchmark_data_smithi194_121512_object50914:head# 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6424d0ed:::benchmark_data_smithi194_121512_object76063:head# 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6425195e:::benchmark_data_smithi194_121512_object11039:head# 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642583df:::benchmark_data_smithi194_121512_object52645:head# 2024-04-29T14:27:11.215 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6425d04e:::benchmark_data_smithi194_121512_object126746:head# 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642610d9:::benchmark_data_smithi194_121512_object27281:head# 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6426c8fc:::benchmark_data_smithi194_121512_object85667:head# 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6426ed1b:::benchmark_data_smithi194_121512_object86783:head# 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6427639e:::benchmark_data_smithi194_121512_object21231:head# 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6428b2a1:::benchmark_data_smithi194_121512_object19131:head# 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6429aea6:::benchmark_data_smithi194_121512_object87667:head# 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6429e3c9:::benchmark_data_smithi194_121512_object45011:head# 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642a088a:::benchmark_data_smithi194_121512_object65416:head# 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642a92d6:::benchmark_data_smithi194_121512_object127726:head# 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642ae337:::benchmark_data_smithi194_121512_object31422:head# 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642b2d43:::benchmark_data_smithi194_121512_object5443:head# 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642bee38:::benchmark_data_smithi194_121512_object49843:head# 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642c1af0:::benchmark_data_smithi194_121512_object21173:head# 2024-04-29T14:27:11.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.219 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642c2f8e:::benchmark_data_smithi194_121512_object91937:head# 2024-04-29T14:27:11.219 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642c8adc:::benchmark_data_smithi194_121512_object46652:head# 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642ced8e:::benchmark_data_smithi194_121512_object13841:head# 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642e3653:::benchmark_data_smithi194_121512_object22033:head# 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642f215d:::benchmark_data_smithi194_121512_object44915:head# 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:642fed45:::benchmark_data_smithi194_121512_object15339:head# 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64307f04:::benchmark_data_smithi194_121512_object69081:head# 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64314dda:::benchmark_data_smithi194_121512_object49007:head# 2024-04-29T14:27:11.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64318bf4:::benchmark_data_smithi194_121512_object57678:head# 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6431b266:::benchmark_data_smithi194_121512_object6113:head# 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6431d7ca:::benchmark_data_smithi194_121512_object64814:head# 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643264ed:::benchmark_data_smithi194_121512_object106658:head# 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6432dc69:::benchmark_data_smithi194_121512_object33212:head# 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6433e137:::benchmark_data_smithi194_121512_object40021:head# 2024-04-29T14:27:11.221 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.222 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64345eb5:::benchmark_data_smithi194_121512_object89329:head# 2024-04-29T14:27:11.223 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.223 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64349014:::benchmark_data_smithi194_121512_object26408:head# 2024-04-29T14:27:11.223 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.223 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6434d4ea:::benchmark_data_smithi194_121512_object49136:head# 2024-04-29T14:27:11.224 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.224 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6435a644:::benchmark_data_smithi194_121512_object68533:head# 2024-04-29T14:27:11.224 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.224 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64368bf5:::benchmark_data_smithi194_121512_object88211:head# 2024-04-29T14:27:11.224 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.224 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6436e584:::benchmark_data_smithi194_121512_object117463:head# 2024-04-29T14:27:11.225 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.225 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64372690:::benchmark_data_smithi194_121512_object36115:head# 2024-04-29T14:27:11.225 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.225 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6437312b:::benchmark_data_smithi194_121512_object63421:head# 2024-04-29T14:27:11.225 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6438caa6:::benchmark_data_smithi194_121512_object46782:head# 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6439aaae:::benchmark_data_smithi194_121512_object31567:head# 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6439e865:::benchmark_data_smithi194_121512_object129731:head# 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643bdfc3:::benchmark_data_smithi194_121512_object51560:head# 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643c41e5:::benchmark_data_smithi194_121512_object16950:head# 2024-04-29T14:27:11.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643c4bff:::benchmark_data_smithi194_121512_object113047:head# 2024-04-29T14:27:11.227 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643c50ed:::benchmark_data_smithi194_121512_object92576:head# 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643d637b:::benchmark_data_smithi194_121512_object126471:head# 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643d67c0:::benchmark_data_smithi194_121512_object81671:head# 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643db7a9:::benchmark_data_smithi194_121512_object112958:head# 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643dec51:::benchmark_data_smithi194_121512_object16969:head# 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643decd5:::benchmark_data_smithi194_121512_object96749:head# 2024-04-29T14:27:11.228 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643e10aa:::benchmark_data_smithi194_121512_object18084:head# 2024-04-29T14:27:11.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643e6a73:::benchmark_data_smithi194_121512_object129278:head# 2024-04-29T14:27:11.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643e91b7:::benchmark_data_smithi194_121512_object115410:head# 2024-04-29T14:27:11.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:643f0a79:::benchmark_data_smithi194_121512_object17834:head# 2024-04-29T14:27:11.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64403b25:::benchmark_data_smithi194_121512_object116759:head# 2024-04-29T14:27:11.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6440a1b1:::benchmark_data_smithi194_121512_object89171:head# 2024-04-29T14:27:11.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.234 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6441d828:::benchmark_data_smithi194_121512_object128788:head# 2024-04-29T14:27:11.234 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644228e0:::benchmark_data_smithi194_121512_object113610:head# 2024-04-29T14:27:11.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64425044:::benchmark_data_smithi194_121512_object92786:head# 2024-04-29T14:27:11.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6442db8f:::benchmark_data_smithi194_121512_object58311:head# 2024-04-29T14:27:11.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6442f95a:::benchmark_data_smithi194_121512_object126796:head# 2024-04-29T14:27:11.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64435c9f:::benchmark_data_smithi194_121512_object22901:head# 2024-04-29T14:27:11.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6444b3d6:::benchmark_data_smithi194_121512_object76363:head# 2024-04-29T14:27:11.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64459300:::benchmark_data_smithi194_121512_object72743:head# 2024-04-29T14:27:11.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64459632:::benchmark_data_smithi194_121512_object102389:head# 2024-04-29T14:27:11.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6445a3b8:::benchmark_data_smithi194_121512_object102734:head# 2024-04-29T14:27:11.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6445d65d:::benchmark_data_smithi194_121512_object74334:head# 2024-04-29T14:27:11.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64460515:::benchmark_data_smithi194_121512_object113114:head# 2024-04-29T14:27:11.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644689e6:::benchmark_data_smithi194_121512_object10760:head# 2024-04-29T14:27:11.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6446f8f4:::benchmark_data_smithi194_121512_object62659:head# 2024-04-29T14:27:11.245 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.245 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644720bb:::benchmark_data_smithi194_121512_object87681:head# 2024-04-29T14:27:11.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6447ac92:::benchmark_data_smithi194_121512_object40580:head# 2024-04-29T14:27:11.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644807c1:::benchmark_data_smithi194_121512_object10458:head# 2024-04-29T14:27:11.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64480acd:::benchmark_data_smithi194_121512_object46128:head# 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64481810:::benchmark_data_smithi194_121512_object13382:head# 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64495f59:::benchmark_data_smithi194_121512_object123429:head# 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64497695:::benchmark_data_smithi194_121512_object11992:head# 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644a919a:::benchmark_data_smithi194_121512_object119976:head# 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644b5dd2:::benchmark_data_smithi194_121512_object57589:head# 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644bb079:::benchmark_data_smithi194_121512_object90801:head# 2024-04-29T14:27:11.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644bc5ac:::benchmark_data_smithi194_121512_object31872:head# 2024-04-29T14:27:11.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644caefa:::benchmark_data_smithi194_121512_object34863:head# 2024-04-29T14:27:11.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644ce91a:::benchmark_data_smithi194_121512_object57208:head# 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644d114d:::benchmark_data_smithi194_121512_object98987:head# 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644de59b:::benchmark_data_smithi194_121512_object32803:head# 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644f4c7c:::benchmark_data_smithi194_121512_object107963:head# 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:644f5ae9:::benchmark_data_smithi194_121512_object127340:head# 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6450b04d:::benchmark_data_smithi194_121512_object14659:head# 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6450d14d:::benchmark_data_smithi194_121512_object46786:head# 2024-04-29T14:27:11.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6450f185:::benchmark_data_smithi194_121512_object114803:head# 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64517461:::benchmark_data_smithi194_121512_object53163:head# 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6451cf35:::benchmark_data_smithi194_121512_object94437:head# 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6452b08b:::benchmark_data_smithi194_121512_object44087:head# 2024-04-29T14:27:11.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6452efb7:::benchmark_data_smithi194_121512_object113015:head# 2024-04-29T14:27:11.258 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6452f59d:::benchmark_data_smithi194_121512_object65297:head# 2024-04-29T14:27:11.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64536074:::benchmark_data_smithi194_121512_object68881:head# 2024-04-29T14:27:11.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645360d8:::benchmark_data_smithi194_121512_object86717:head# 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6453e2fe:::benchmark_data_smithi194_121512_object67534:head# 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64540cc0:::benchmark_data_smithi194_121512_object123134:head# 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64545c46:::benchmark_data_smithi194_121512_object40303:head# 2024-04-29T14:27:11.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6454f5a0:::benchmark_data_smithi194_121512_object19623:head# 2024-04-29T14:27:11.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645525ce:::benchmark_data_smithi194_121512_object16321:head# 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645585b8:::benchmark_data_smithi194_121512_object76984:head# 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6455d3a7:::benchmark_data_smithi194_121512_object64207:head# 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645622de:::benchmark_data_smithi194_121512_object45117:head# 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6456679e:::benchmark_data_smithi194_121512_object66772:head# 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64567654:::benchmark_data_smithi194_121512_object27555:head# 2024-04-29T14:27:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6456e536:::benchmark_data_smithi194_121512_object26381:head# 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6457640a:::benchmark_data_smithi194_121512_object120169:head# 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6457ba70:::benchmark_data_smithi194_121512_object23618:head# 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64585eed:::benchmark_data_smithi194_121512_object25563:head# 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6458d5c1:::benchmark_data_smithi194_121512_object95238:head# 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645900fb:::benchmark_data_smithi194_121512_object93961:head# 2024-04-29T14:27:11.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6459554c:::benchmark_data_smithi194_121512_object26970:head# 2024-04-29T14:27:11.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645ad179:::benchmark_data_smithi194_121512_object55907:head# 2024-04-29T14:27:11.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645aedd7:::benchmark_data_smithi194_121512_object43098:head# 2024-04-29T14:27:11.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645d2210:::benchmark_data_smithi194_121512_object32517:head# 2024-04-29T14:27:11.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645d6d0c:::benchmark_data_smithi194_121512_object73678:head# 2024-04-29T14:27:11.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645d80fc:::benchmark_data_smithi194_121512_object17891:head# 2024-04-29T14:27:11.269 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645da42d:::benchmark_data_smithi194_121512_object43143:head# 2024-04-29T14:27:11.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645e9413:::benchmark_data_smithi194_121512_object117392:head# 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645eb110:::benchmark_data_smithi194_121512_object105499:head# 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645f2c5e:::benchmark_data_smithi194_121512_object28882:head# 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645f7a59:::benchmark_data_smithi194_121512_object44490:head# 2024-04-29T14:27:11.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:645feae5:::benchmark_data_smithi194_121512_object946:head# 2024-04-29T14:27:11.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6460674e:::benchmark_data_smithi194_121512_object116290:head# 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6460ac74:::benchmark_data_smithi194_121512_object45635:head# 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6460b098:::benchmark_data_smithi194_121512_object116861:head# 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646125b3:::benchmark_data_smithi194_121512_object27856:head# 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6462596e:::benchmark_data_smithi194_121512_object76552:head# 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6462bad6:::benchmark_data_smithi194_121512_object132345:head# 2024-04-29T14:27:11.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6462c42d:::benchmark_data_smithi194_121512_object71440:head# 2024-04-29T14:27:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64631085:::benchmark_data_smithi194_121512_object132544:head# 2024-04-29T14:27:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64631c5e:::benchmark_data_smithi194_121512_object104060:head# 2024-04-29T14:27:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6463922e:::benchmark_data_smithi194_121512_object34787:head# 2024-04-29T14:27:11.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6464972c:::benchmark_data_smithi194_121512_object62091:head# 2024-04-29T14:27:11.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6464b5a3:::benchmark_data_smithi194_121512_object77772:head# 2024-04-29T14:27:11.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.279 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646582a8:::benchmark_data_smithi194_121512_object95929:head# 2024-04-29T14:27:11.279 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64660868:::benchmark_data_smithi194_121512_object30494:head# 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6466e1bd:::benchmark_data_smithi194_121512_object63580:head# 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6467aad0:::benchmark_data_smithi194_121512_object58342:head# 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6467b91c:::benchmark_data_smithi194_121512_object38484:head# 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6467d992:::benchmark_data_smithi194_121512_object53198:head# 2024-04-29T14:27:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6468566d:::benchmark_data_smithi194_121512_object82608:head# 2024-04-29T14:27:11.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6468bb1a:::benchmark_data_smithi194_121512_object83563:head# 2024-04-29T14:27:11.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.281 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6469759c:::benchmark_data_smithi194_121512_object74318:head# 2024-04-29T14:27:11.281 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64699a78:::benchmark_data_smithi194_121512_object72171:head# 2024-04-29T14:27:11.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6469e99e:::benchmark_data_smithi194_121512_object118251:head# 2024-04-29T14:27:11.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646a0e15:::benchmark_data_smithi194_121512_object111505:head# 2024-04-29T14:27:11.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646a66bd:::benchmark_data_smithi194_121512_object6013:head# 2024-04-29T14:27:11.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646a8a1b:::benchmark_data_smithi194_121512_object93234:head# 2024-04-29T14:27:11.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646a90b9:::benchmark_data_smithi194_121512_object95720:head# 2024-04-29T14:27:11.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646b57cb:::benchmark_data_smithi194_121512_object10839:head# 2024-04-29T14:27:11.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646b6a07:::benchmark_data_smithi194_121512_object77479:head# 2024-04-29T14:27:11.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646bd90c:::benchmark_data_smithi194_121512_object53704:head# 2024-04-29T14:27:11.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646d1dd0:::benchmark_data_smithi194_121512_object16821:head# 2024-04-29T14:27:11.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646e1105:::benchmark_data_smithi194_121512_object20613:head# 2024-04-29T14:27:11.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646e8fe4:::benchmark_data_smithi194_121512_object19609:head# 2024-04-29T14:27:11.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646ec62d:::benchmark_data_smithi194_121512_object51314:head# 2024-04-29T14:27:11.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:646ffbe9:::benchmark_data_smithi194_121512_object55538:head# 2024-04-29T14:27:11.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6471fb8c:::benchmark_data_smithi194_121512_object2183:head# 2024-04-29T14:27:11.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6472a443:::benchmark_data_smithi194_121512_object116384:head# 2024-04-29T14:27:11.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6473395e:::benchmark_data_smithi194_121512_object27541:head# 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64733df4:::benchmark_data_smithi194_121512_object19766:head# 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64749781:::benchmark_data_smithi194_121512_object126512:head# 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6474d681:::benchmark_data_smithi194_121512_object121486:head# 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64758844:::benchmark_data_smithi194_121512_object4380:head# 2024-04-29T14:27:11.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6475f607:::benchmark_data_smithi194_121512_object4438:head# 2024-04-29T14:27:11.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64761183:::benchmark_data_smithi194_121512_object52643:head# 2024-04-29T14:27:11.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64762474:::benchmark_data_smithi194_121512_object83921:head# 2024-04-29T14:27:11.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6476684e:::benchmark_data_smithi194_121512_object13215:head# 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6476d4e9:::benchmark_data_smithi194_121512_object66753:head# 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64782421:::benchmark_data_smithi194_121512_object79100:head# 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6478428a:::benchmark_data_smithi194_121512_object14439:head# 2024-04-29T14:27:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64787d11:::benchmark_data_smithi194_121512_object130322:head# 2024-04-29T14:27:11.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6478b395:::benchmark_data_smithi194_121512_object81395:head# 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64790a6c:::benchmark_data_smithi194_121512_object61452:head# 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64795628:::benchmark_data_smithi194_121512_object45585:head# 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6479fe46:::benchmark_data_smithi194_121512_object40541:head# 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647a49b0:::benchmark_data_smithi194_121512_object7548:head# 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647b06d0:::benchmark_data_smithi194_121512_object58142:head# 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647b23e4:::benchmark_data_smithi194_121512_object836:head# 2024-04-29T14:27:11.299 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647be61b:::benchmark_data_smithi194_121512_object72649:head# 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647be748:::benchmark_data_smithi194_121512_object94710:head# 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647d283c:::benchmark_data_smithi194_121512_object93015:head# 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647dff06:::benchmark_data_smithi194_121512_object11941:head# 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647e7151:::benchmark_data_smithi194_121512_object26925:head# 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647e9043:::benchmark_data_smithi194_121512_object3369:head# 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647e9aa3:::benchmark_data_smithi194_121512_object42718:head# 2024-04-29T14:27:11.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.301 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647f4e35:::benchmark_data_smithi194_121512_object40247:head# 2024-04-29T14:27:11.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.301 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647f63f0:::benchmark_data_smithi194_121512_object38285:head# 2024-04-29T14:27:11.301 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:647ff2b7:::benchmark_data_smithi194_121512_object41513:head# 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64807e41:::benchmark_data_smithi194_121512_object123540:head# 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6480d3e5:::benchmark_data_smithi194_121512_object74772:head# 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6480fd4c:::benchmark_data_smithi194_121512_object110269:head# 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6482583d:::benchmark_data_smithi194_121512_object80769:head# 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6482f65d:::benchmark_data_smithi194_121512_object123088:head# 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.305 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6483230c:::benchmark_data_smithi194_121512_object47808:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64832b28:::benchmark_data_smithi194_121512_object74248:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6483b9c0:::benchmark_data_smithi194_121512_object35685:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6483bb14:::benchmark_data_smithi194_121512_object12041:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64848272:::benchmark_data_smithi194_121512_object115086:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64869218:::benchmark_data_smithi194_121512_object122210:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6486f806:::benchmark_data_smithi194_121512_object66425:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64871c14:::benchmark_data_smithi194_121512_object21122:head# 2024-04-29T14:27:11.306 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64877491:::benchmark_data_smithi194_121512_object35423:head# 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648778fe:::benchmark_data_smithi194_121512_object121178:head# 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64881d1c:::benchmark_data_smithi194_121512_object41478:head# 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6488398c:::benchmark_data_smithi194_121512_object93003:head# 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64894bb9:::benchmark_data_smithi194_121512_object124334:head# 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648a176c:::benchmark_data_smithi194_121512_object113189:head# 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648a530a:::benchmark_data_smithi194_121512_object11322:head# 2024-04-29T14:27:11.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:11.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648a72a5:::benchmark_data_smithi194_121512_object107397:head# 2024-04-29T14:27:11.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648b3802:::benchmark_data_smithi194_121512_object2312:head# 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648bd1b4:::benchmark_data_smithi194_121512_object117774:head# 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648c6a3e:::benchmark_data_smithi194_121512_object53620:head# 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648c9e79:::benchmark_data_smithi194_121512_object123420:head# 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648cad46:::benchmark_data_smithi194_121512_object6230:head# 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648e1812:::benchmark_data_smithi194_121512_object78525:head# 2024-04-29T14:27:11.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648e758a:::benchmark_data_smithi194_121512_object20757:head# 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648ed44e:::benchmark_data_smithi194_121512_object53320:head# 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648f357f:::benchmark_data_smithi194_121512_object108310:head# 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:648f6703:::benchmark_data_smithi194_121512_object56223:head# 2024-04-29T14:27:11.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649022e8:::benchmark_data_smithi194_121512_object12602:head# 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6491d6e2:::benchmark_data_smithi194_121512_object20011:head# 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6492a6e1:::benchmark_data_smithi194_121512_object76933:head# 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64943bce:::benchmark_data_smithi194_121512_object120853:head# 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6494e0cf:::benchmark_data_smithi194_121512_object18185:head# 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64952977:::benchmark_data_smithi194_121512_object91701:head# 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64956b18:::benchmark_data_smithi194_121512_object22739:head# 2024-04-29T14:27:11.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64956e9e:::benchmark_data_smithi194_121512_object22046:head# 2024-04-29T14:27:11.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649589f0:::benchmark_data_smithi194_121512_object32362:head# 2024-04-29T14:27:11.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6495bff7:::benchmark_data_smithi194_121512_object47295:head# 2024-04-29T14:27:11.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6495ca2e:::benchmark_data_smithi194_121512_object10565:head# 2024-04-29T14:27:11.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6496b46a:::benchmark_data_smithi194_121512_object13171:head# 2024-04-29T14:27:11.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64974975:::benchmark_data_smithi194_121512_object55570:head# 2024-04-29T14:27:11.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6497914d:::benchmark_data_smithi194_121512_object85721:head# 2024-04-29T14:27:11.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649793b2:::benchmark_data_smithi194_121512_object108726:head# 2024-04-29T14:27:11.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6497b293:::benchmark_data_smithi194_121512_object54655:head# 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6497cf2f:::benchmark_data_smithi194_121512_object115528:head# 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6497f3e3:::benchmark_data_smithi194_121512_object52719:head# 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64981070:::benchmark_data_smithi194_121512_object15651:head# 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6498d267:::benchmark_data_smithi194_121512_object86250:head# 2024-04-29T14:27:11.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6498fdcd:::benchmark_data_smithi194_121512_object129124:head# 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64991caa:::benchmark_data_smithi194_121512_object122532:head# 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64998963:::benchmark_data_smithi194_121512_object21051:head# 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6499e5c4:::benchmark_data_smithi194_121512_object89073:head# 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6499ed15:::benchmark_data_smithi194_121512_object67060:head# 2024-04-29T14:27:11.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649b3bf4:::benchmark_data_smithi194_121512_object10013:head# 2024-04-29T14:27:11.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649b984f:::benchmark_data_smithi194_121512_object123678:head# 2024-04-29T14:27:11.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.320 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649cdd97:::benchmark_data_smithi194_121512_object131372:head# 2024-04-29T14:27:11.320 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649def01:::benchmark_data_smithi194_121512_object132892:head# 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649f0891:::benchmark_data_smithi194_121512_object119318:head# 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649fb035:::benchmark_data_smithi194_121512_object83833:head# 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649fc864:::benchmark_data_smithi194_121512_object48100:head# 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:649fcd84:::benchmark_data_smithi194_121512_object52310:head# 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a03219:::benchmark_data_smithi194_121512_object7884:head# 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.323 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a0327c:::benchmark_data_smithi194_121512_object53396:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a0e5ba:::benchmark_data_smithi194_121512_object83032:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a16b91:::benchmark_data_smithi194_121512_object70811:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a1c941:::benchmark_data_smithi194_121512_object78177:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a28854:::benchmark_data_smithi194_121512_object31351:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a2a2f1:::benchmark_data_smithi194_121512_object28909:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a2eca1:::benchmark_data_smithi194_121512_object49268:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a3a84a:::benchmark_data_smithi194_121512_object62422:head# 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a3c99d:::benchmark_data_smithi194_121512_object67811:head# 2024-04-29T14:27:11.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a4b261:::benchmark_data_smithi194_121512_object65863:head# 2024-04-29T14:27:11.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a4fe2f:::benchmark_data_smithi194_121512_object23898:head# 2024-04-29T14:27:11.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a5069d:::benchmark_data_smithi194_121512_object21326:head# 2024-04-29T14:27:11.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a51f0e:::benchmark_data_smithi194_121512_object115586:head# 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a5cbb9:::benchmark_data_smithi194_121512_object58773:head# 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a5e684:::benchmark_data_smithi194_121512_object36149:head# 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a5f634:::benchmark_data_smithi194_121512_object129406:head# 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a61cb7:::benchmark_data_smithi194_121512_object94450:head# 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a658c1:::benchmark_data_smithi194_121512_object75940:head# 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a662f8:::benchmark_data_smithi194_121512_object117206:head# 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a73f8e:::benchmark_data_smithi194_121512_object63828:head# 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a76f6b:::benchmark_data_smithi194_121512_object31253:head# 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a7f4c8:::benchmark_data_smithi194_121512_object100922:head# 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a83c92:::benchmark_data_smithi194_121512_object62738:head# 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a84c29:::benchmark_data_smithi194_121512_object39298:head# 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a8bb63:::benchmark_data_smithi194_121512_object10046:head# 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a8c117:::benchmark_data_smithi194_121512_object15404:head# 2024-04-29T14:27:11.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64a945ef:::benchmark_data_smithi194_121512_object43942:head# 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64aa1297:::benchmark_data_smithi194_121512_object61895:head# 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64aa3dce:::benchmark_data_smithi194_121512_object146:head# 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64aa66a1:::benchmark_data_smithi194_121512_object131587:head# 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64aa7668:::benchmark_data_smithi194_121512_object37201:head# 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64aadc27:::benchmark_data_smithi194_121512_object118347:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ab20d0:::benchmark_data_smithi194_121512_object104114:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64abed7d:::benchmark_data_smithi194_121512_object35867:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ad50e1:::benchmark_data_smithi194_121512_object2393:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64aea7f2:::benchmark_data_smithi194_121512_object9596:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b058c8:::benchmark_data_smithi194_121512_object53102:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b2089c:::benchmark_data_smithi194_121512_object1481:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b2b60e:::benchmark_data_smithi194_121512_object59353:head# 2024-04-29T14:27:11.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.333 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b2cc8f:::benchmark_data_smithi194_121512_object119131:head# 2024-04-29T14:27:11.333 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.333 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b35f72:::benchmark_data_smithi194_121512_object107041:head# 2024-04-29T14:27:11.333 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b3e2af:::benchmark_data_smithi194_121512_object44467:head# 2024-04-29T14:27:11.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b4769d:::benchmark_data_smithi194_121512_object24376:head# 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b6799e:::benchmark_data_smithi194_121512_object59359:head# 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b74373:::benchmark_data_smithi194_121512_object32511:head# 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b7e39b:::benchmark_data_smithi194_121512_object131030:head# 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b83bbb:::benchmark_data_smithi194_121512_object63968:head# 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b88f8f:::benchmark_data_smithi194_121512_object54943:head# 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b8bf80:::benchmark_data_smithi194_121512_object39476:head# 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b8e821:::benchmark_data_smithi194_121512_object116925:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b9356c:::benchmark_data_smithi194_121512_object11682:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b9bfba:::benchmark_data_smithi194_121512_object99138:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b9e122:::benchmark_data_smithi194_121512_object94245:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64b9f49e:::benchmark_data_smithi194_121512_object32330:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ba3275:::benchmark_data_smithi194_121512_object44535:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ba57f2:::benchmark_data_smithi194_121512_object596:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bae39e:::benchmark_data_smithi194_121512_object31194:head# 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bae909:::benchmark_data_smithi194_121512_object11130:head# 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64baf579:::benchmark_data_smithi194_121512_object128473:head# 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bb776b:::benchmark_data_smithi194_121512_object5035:head# 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bc2376:::benchmark_data_smithi194_121512_object113758:head# 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bd36d5:::benchmark_data_smithi194_121512_object119317:head# 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bd7d5d:::benchmark_data_smithi194_121512_object101537:head# 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64be6281:::benchmark_data_smithi194_121512_object70818:head# 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.341 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bf0d65:::benchmark_data_smithi194_121512_object98177:head# 2024-04-29T14:27:11.342 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.342 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bf3c15:::benchmark_data_smithi194_121512_object121276:head# 2024-04-29T14:27:11.342 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.342 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bfc15a:::benchmark_data_smithi194_121512_object14978:head# 2024-04-29T14:27:11.342 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.342 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64bfd939:::benchmark_data_smithi194_121512_object19052:head# 2024-04-29T14:27:11.342 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c015e6:::benchmark_data_smithi194_121512_object100423:head# 2024-04-29T14:27:11.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c02d72:::benchmark_data_smithi194_121512_object10895:head# 2024-04-29T14:27:11.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c0a349:::benchmark_data_smithi194_121512_object78587:head# 2024-04-29T14:27:11.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c241da:::benchmark_data_smithi194_121512_object24222:head# 2024-04-29T14:27:11.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c2b000:::benchmark_data_smithi194_121512_object2009:head# 2024-04-29T14:27:11.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c4072d:::benchmark_data_smithi194_121512_object56434:head# 2024-04-29T14:27:11.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.344 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:11.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c498ba:::benchmark_data_smithi194_121512_object65304:head# 2024-04-29T14:27:11.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c4c2d9:::benchmark_data_smithi194_121512_object25509:head# 2024-04-29T14:27:11.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c4cf9d:::benchmark_data_smithi194_121512_object66316:head# 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c52022:::benchmark_data_smithi194_121512_object39801:head# 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c5bfaf:::benchmark_data_smithi194_121512_object79045:head# 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c5f00a:::benchmark_data_smithi194_121512_object110156:head# 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c6101d:::benchmark_data_smithi194_121512_object81407:head# 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c6547c:::benchmark_data_smithi194_121512_object7746:head# 2024-04-29T14:27:11.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.361 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c6bdaa:::benchmark_data_smithi194_121512_object16009:head# 2024-04-29T14:27:11.361 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.361 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c6c7e1:::benchmark_data_smithi194_121512_object111680:head# 2024-04-29T14:27:11.361 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.361 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c89f38:::benchmark_data_smithi194_121512_object28199:head# 2024-04-29T14:27:11.361 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c8cfc5:::benchmark_data_smithi194_121512_object61605:head# 2024-04-29T14:27:11.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64c9cb32:::benchmark_data_smithi194_121512_object88419:head# 2024-04-29T14:27:11.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ca331e:::benchmark_data_smithi194_121512_object105866:head# 2024-04-29T14:27:11.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64cb5bc3:::benchmark_data_smithi194_121512_object15828:head# 2024-04-29T14:27:11.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64cbc2c9:::benchmark_data_smithi194_121512_object22825:head# 2024-04-29T14:27:11.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64cc634b:::benchmark_data_smithi194_121512_object104403:head# 2024-04-29T14:27:11.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ce02c1:::benchmark_data_smithi194_121512_object49705:head# 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64cf4a5e:::benchmark_data_smithi194_121512_object123972:head# 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64cf5f13:::benchmark_data_smithi194_121512_object73102:head# 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64cf814c:::benchmark_data_smithi194_121512_object49990:head# 2024-04-29T14:27:11.364 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d0a1ac:::benchmark_data_smithi194_121512_object33398:head# 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d11621:::benchmark_data_smithi194_121512_object60889:head# 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d1249c:::benchmark_data_smithi194_121512_object48529:head# 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d188b3:::benchmark_data_smithi194_121512_object133125:head# 2024-04-29T14:27:11.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.366 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d1f119:::benchmark_data_smithi194_121512_object50388:head# 2024-04-29T14:27:11.366 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d20e5b:::benchmark_data_smithi194_121512_object81004:head# 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d2999d:::benchmark_data_smithi194_121512_object6385:head# 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d39d80:::benchmark_data_smithi194_121512_object25432:head# 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d45afb:::benchmark_data_smithi194_121512_object65678:head# 2024-04-29T14:27:11.367 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d5596e:::benchmark_data_smithi194_121512_object121720:head# 2024-04-29T14:27:11.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d5fa09:::benchmark_data_smithi194_121512_object47947:head# 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d7f71d:::benchmark_data_smithi194_121512_object11751:head# 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d8916f:::benchmark_data_smithi194_121512_object46046:head# 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d89af4:::benchmark_data_smithi194_121512_object90471:head# 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d8ed83:::benchmark_data_smithi194_121512_object42799:head# 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d966e4:::benchmark_data_smithi194_121512_object22108:head# 2024-04-29T14:27:11.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d96720:::benchmark_data_smithi194_121512_object83262:head# 2024-04-29T14:27:11.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d97c33:::benchmark_data_smithi194_121512_object6862:head# 2024-04-29T14:27:11.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64d989fc:::benchmark_data_smithi194_121512_object82098:head# 2024-04-29T14:27:11.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64da88fc:::benchmark_data_smithi194_121512_object92455:head# 2024-04-29T14:27:11.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64daa7a2:::benchmark_data_smithi194_121512_object81023:head# 2024-04-29T14:27:11.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64dad584:::benchmark_data_smithi194_121512_object56832:head# 2024-04-29T14:27:11.372 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.372 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64dae713:::benchmark_data_smithi194_121512_object106972:head# 2024-04-29T14:27:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64db6b5a:::benchmark_data_smithi194_121512_object79085:head# 2024-04-29T14:27:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64db8c23:::benchmark_data_smithi194_121512_object84468:head# 2024-04-29T14:27:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64dc773f:::benchmark_data_smithi194_121512_object2982:head# 2024-04-29T14:27:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64dda657:::benchmark_data_smithi194_121512_object27826:head# 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64dde1a4:::benchmark_data_smithi194_121512_object15963:head# 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64de026a:::benchmark_data_smithi194_121512_object54488:head# 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64dfdb5e:::benchmark_data_smithi194_121512_object478:head# 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e008e3:::benchmark_data_smithi194_121512_object67179:head# 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e0aa58:::benchmark_data_smithi194_121512_object7727:head# 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e0b334:::benchmark_data_smithi194_121512_object44764:head# 2024-04-29T14:27:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e2019d:::benchmark_data_smithi194_121512_object133307:head# 2024-04-29T14:27:11.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e22a40:::benchmark_data_smithi194_121512_object69583:head# 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e22dd3:::benchmark_data_smithi194_121512_object124650:head# 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e2bacf:::benchmark_data_smithi194_121512_object91896:head# 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e35e8e:::benchmark_data_smithi194_121512_object30003:head# 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e35f3d:::benchmark_data_smithi194_121512_object129241:head# 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e39d96:::benchmark_data_smithi194_121512_object47748:head# 2024-04-29T14:27:11.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e3d109:::benchmark_data_smithi194_121512_object63066:head# 2024-04-29T14:27:11.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e3d2fa:::benchmark_data_smithi194_121512_object8351:head# 2024-04-29T14:27:11.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e7479e:::benchmark_data_smithi194_121512_object57442:head# 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e76843:::benchmark_data_smithi194_121512_object124500:head# 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e79c08:::benchmark_data_smithi194_121512_object64972:head# 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e7b119:::benchmark_data_smithi194_121512_object37465:head# 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e82f80:::benchmark_data_smithi194_121512_object90079:head# 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e8afd4:::benchmark_data_smithi194_121512_object96325:head# 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e8c7cf:::benchmark_data_smithi194_121512_object16417:head# 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e8cb80:::benchmark_data_smithi194_121512_object5539:head# 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e8e49b:::benchmark_data_smithi194_121512_object94008:head# 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64e9d1c8:::benchmark_data_smithi194_121512_object30623:head# 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ea33f3:::benchmark_data_smithi194_121512_object3563:head# 2024-04-29T14:27:11.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ea557a:::benchmark_data_smithi194_121512_object130656:head# 2024-04-29T14:27:11.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64eaa1d9:::benchmark_data_smithi194_121512_object39882:head# 2024-04-29T14:27:11.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64eb16b1:::benchmark_data_smithi194_121512_object4852:head# 2024-04-29T14:27:11.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ec5c6b:::benchmark_data_smithi194_121512_object62765:head# 2024-04-29T14:27:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ec6aea:::benchmark_data_smithi194_121512_object92484:head# 2024-04-29T14:27:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ecd1bb:::benchmark_data_smithi194_121512_object52599:head# 2024-04-29T14:27:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ee6332:::benchmark_data_smithi194_121512_object97759:head# 2024-04-29T14:27:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64eeab54:::benchmark_data_smithi194_121512_object14747:head# 2024-04-29T14:27:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64eeaff8:::benchmark_data_smithi194_121512_object31405:head# 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64eeb3f5:::benchmark_data_smithi194_121512_object132807:head# 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f07e83:::benchmark_data_smithi194_121512_object10938:head# 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f0fd0a:::benchmark_data_smithi194_121512_object45251:head# 2024-04-29T14:27:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f1057a:::benchmark_data_smithi194_121512_object39436:head# 2024-04-29T14:27:11.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f1d4c1:::benchmark_data_smithi194_121512_object53563:head# 2024-04-29T14:27:11.395 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.395 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f20584:::benchmark_data_smithi194_121512_object44367:head# 2024-04-29T14:27:11.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f2a5ad:::benchmark_data_smithi194_121512_object79140:head# 2024-04-29T14:27:11.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f36017:::benchmark_data_smithi194_121512_object51009:head# 2024-04-29T14:27:11.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f3a7c7:::benchmark_data_smithi194_121512_object24462:head# 2024-04-29T14:27:11.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f3e36b:::benchmark_data_smithi194_121512_object37697:head# 2024-04-29T14:27:11.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.402 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f3eeb2:::benchmark_data_smithi194_121512_object105882:head# 2024-04-29T14:27:11.402 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f422f2:::benchmark_data_smithi194_121512_object67993:head# 2024-04-29T14:27:11.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f48f55:::benchmark_data_smithi194_121512_object4326:head# 2024-04-29T14:27:11.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f627b7:::benchmark_data_smithi194_121512_object95722:head# 2024-04-29T14:27:11.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f65ca4:::benchmark_data_smithi194_121512_object102446:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f6a596:::benchmark_data_smithi194_121512_object83993:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f6e811:::benchmark_data_smithi194_121512_object25667:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f7c136:::benchmark_data_smithi194_121512_object88824:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f8a44d:::benchmark_data_smithi194_121512_object32591:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f92132:::benchmark_data_smithi194_121512_object10123:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f964ca:::benchmark_data_smithi194_121512_object50907:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f999c8:::benchmark_data_smithi194_121512_object28578:head# 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64f9d478:::benchmark_data_smithi194_121512_object117055:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fa54f8:::benchmark_data_smithi194_121512_object131478:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fb4d2e:::benchmark_data_smithi194_121512_object131811:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fb9c1e:::benchmark_data_smithi194_121512_object115864:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fbbb3e:::benchmark_data_smithi194_121512_object32414:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fd2c79:::benchmark_data_smithi194_121512_object50216:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fd6797:::benchmark_data_smithi194_121512_object126605:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fdd286:::benchmark_data_smithi194_121512_object51443:head# 2024-04-29T14:27:11.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64fea2e8:::benchmark_data_smithi194_121512_object46347:head# 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:64ffda6d:::benchmark_data_smithi194_121512_object34859:head# 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650036ef:::benchmark_data_smithi194_121512_object123177:head# 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6500e0ca:::benchmark_data_smithi194_121512_object128216:head# 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6501206c:::benchmark_data_smithi194_121512_object118416:head# 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65024f1f:::benchmark_data_smithi194_121512_object38580:head# 2024-04-29T14:27:11.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650258bf:::benchmark_data_smithi194_121512_object14613:head# 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65029371:::benchmark_data_smithi194_121512_object131518:head# 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6502ece0:::benchmark_data_smithi194_121512_object16829:head# 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6502efbf:::benchmark_data_smithi194_121512_object22465:head# 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65030a9d:::benchmark_data_smithi194_121512_object124740:head# 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65034a86:::benchmark_data_smithi194_121512_object24114:head# 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650350dc:::benchmark_data_smithi194_121512_object77948:head# 2024-04-29T14:27:11.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65037fda:::benchmark_data_smithi194_121512_object100125:head# 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6504cba2:::benchmark_data_smithi194_121512_object126689:head# 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6504f24a:::benchmark_data_smithi194_121512_object41940:head# 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650545d3:::benchmark_data_smithi194_121512_object21066:head# 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65067073:::benchmark_data_smithi194_121512_object30871:head# 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65067413:::benchmark_data_smithi194_121512_object68558:head# 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6506ceff:::benchmark_data_smithi194_121512_object57188:head# 2024-04-29T14:27:11.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650713bd:::benchmark_data_smithi194_121512_object83466:head# 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650774fb:::benchmark_data_smithi194_121512_object115444:head# 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650778eb:::benchmark_data_smithi194_121512_object65379:head# 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6507e198:::benchmark_data_smithi194_121512_object70276:head# 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6507edcd:::benchmark_data_smithi194_121512_object17690:head# 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650b23cf:::benchmark_data_smithi194_121512_object131343:head# 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650b5289:::benchmark_data_smithi194_121512_object94574:head# 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650b7861:::benchmark_data_smithi194_121512_object118617:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650b976d:::benchmark_data_smithi194_121512_object42806:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650c3f0e:::benchmark_data_smithi194_121512_object127638:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650c7080:::benchmark_data_smithi194_121512_object88765:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650ce4d9:::benchmark_data_smithi194_121512_object713:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650dba8b:::benchmark_data_smithi194_121512_object3324:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650e2722:::benchmark_data_smithi194_121512_object113646:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650e378e:::benchmark_data_smithi194_121512_object99326:head# 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650e3ade:::benchmark_data_smithi194_121512_object88678:head# 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650e5160:::benchmark_data_smithi194_121512_object73556:head# 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650ea54e:::benchmark_data_smithi194_121512_object110605:head# 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650f6150:::benchmark_data_smithi194_121512_object27368:head# 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:650fbe2a:::benchmark_data_smithi194_121512_object32806:head# 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6510ead9:::benchmark_data_smithi194_121512_object98234:head# 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65118ec7:::benchmark_data_smithi194_121512_object42163:head# 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6512018d:::benchmark_data_smithi194_121512_object55205:head# 2024-04-29T14:27:11.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65123891:::benchmark_data_smithi194_121512_object129765:head# 2024-04-29T14:27:11.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6512f846:::benchmark_data_smithi194_121512_object92528:head# 2024-04-29T14:27:11.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:11.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:11 smithi040 ceph-mon[39581]: pgmap v1273: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 430 op/s; 32988/210258 objects degraded (15.689%); 8.5 MiB/s, 135 objects/s recovering 2024-04-29T14:27:11.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:11 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:11.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:11 smithi040 ceph-mon[39581]: from='client.36080 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:27:11.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:11 smithi040 ceph-mon[39581]: osdmap e482: 12 total, 11 up, 7 in 2024-04-29T14:27:11.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:11.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:11.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65133f63:::benchmark_data_smithi194_121512_object78741:head# 2024-04-29T14:27:11.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6513899b:::benchmark_data_smithi194_121512_object29889:head# 2024-04-29T14:27:11.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65145440:::benchmark_data_smithi194_121512_object19819:head# 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6515b3f2:::benchmark_data_smithi194_121512_object10658:head# 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651615ac:::benchmark_data_smithi194_121512_object123199:head# 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65166705:::benchmark_data_smithi194_121512_object86382:head# 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6516edc0:::benchmark_data_smithi194_121512_object49539:head# 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65180e58:::benchmark_data_smithi194_121512_object12510:head# 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65184205:::benchmark_data_smithi194_121512_object78493:head# 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6518d194:::benchmark_data_smithi194_121512_object69524:head# 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65195d52:::benchmark_data_smithi194_121512_object46155:head# 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6519eea9:::benchmark_data_smithi194_121512_object128834:head# 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651a04aa:::benchmark_data_smithi194_121512_object97868:head# 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651a6427:::benchmark_data_smithi194_121512_object56043:head# 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651a8937:::benchmark_data_smithi194_121512_object128729:head# 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651bb7b7:::benchmark_data_smithi194_121512_object113483:head# 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651cf157:::benchmark_data_smithi194_121512_object14909:head# 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651d0f59:::benchmark_data_smithi194_121512_object69298:head# 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651df2e1:::benchmark_data_smithi194_121512_object40050:head# 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:651f3933:::benchmark_data_smithi194_121512_object72650:head# 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652006dc:::benchmark_data_smithi194_121512_object39495:head# 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6520bc1f:::benchmark_data_smithi194_121512_object92194:head# 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6520cf21:::benchmark_data_smithi194_121512_object125951:head# 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65216d4c:::benchmark_data_smithi194_121512_object109962:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65221fe9:::benchmark_data_smithi194_121512_object92398:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65224e1a:::benchmark_data_smithi194_121512_object36990:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65226ff0:::benchmark_data_smithi194_121512_object19214:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65227b63:::benchmark_data_smithi194_121512_object65083:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6522bf4f:::benchmark_data_smithi194_121512_object70894:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65231ecd:::benchmark_data_smithi194_121512_object2107:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65237692:::benchmark_data_smithi194_121512_object119136:head# 2024-04-29T14:27:11.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6523e945:::benchmark_data_smithi194_121512_object26933:head# 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65259799:::benchmark_data_smithi194_121512_object34193:head# 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65259bd3:::benchmark_data_smithi194_121512_object113777:head# 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6525e20e:::benchmark_data_smithi194_121512_object79892:head# 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65262e81:::benchmark_data_smithi194_121512_object104761:head# 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65267957:::benchmark_data_smithi194_121512_object129998:head# 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652871ed:::benchmark_data_smithi194_121512_object83585:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65290d49:::benchmark_data_smithi194_121512_object3244:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652a59c6:::benchmark_data_smithi194_121512_object39478:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652ab123:::benchmark_data_smithi194_121512_object18140:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652ae3aa:::benchmark_data_smithi194_121512_object117982:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652b9c2b:::benchmark_data_smithi194_121512_object17128:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652bcdbe:::benchmark_data_smithi194_121512_object124672:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652bff0c:::benchmark_data_smithi194_121512_object62948:head# 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652c4df8:::benchmark_data_smithi194_121512_object52515:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652c58a4:::benchmark_data_smithi194_121512_object11584:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652cdee4:::benchmark_data_smithi194_121512_object73049:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652ceb91:::benchmark_data_smithi194_121512_object79633:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652d55ba:::benchmark_data_smithi194_121512_object121433:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652dca78:::benchmark_data_smithi194_121512_object63330:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652dd719:::benchmark_data_smithi194_121512_object19597:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652df6e5:::benchmark_data_smithi194_121512_object88832:head# 2024-04-29T14:27:11.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652dfb98:::benchmark_data_smithi194_121512_object23967:head# 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652ecf16:::benchmark_data_smithi194_121512_object102671:head# 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652f523b:::benchmark_data_smithi194_121512_object80465:head# 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:652f6393:::benchmark_data_smithi194_121512_object104834:head# 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65327c90:::benchmark_data_smithi194_121512_object40254:head# 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6532db55:::benchmark_data_smithi194_121512_object115102:head# 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6532fa09:::benchmark_data_smithi194_121512_object49905:head# 2024-04-29T14:27:11.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65333cb0:::benchmark_data_smithi194_121512_object57705:head# 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65336f54:::benchmark_data_smithi194_121512_object43467:head# 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6533cf24:::benchmark_data_smithi194_121512_object75992:head# 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653411f9:::benchmark_data_smithi194_121512_object15110:head# 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653422df:::benchmark_data_smithi194_121512_object23725:head# 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65345147:::benchmark_data_smithi194_121512_object36309:head# 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6535ddd3:::benchmark_data_smithi194_121512_object97346:head# 2024-04-29T14:27:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6535e1c4:::benchmark_data_smithi194_121512_object80835:head# 2024-04-29T14:27:11.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65363ffc:::benchmark_data_smithi194_121512_object125713:head# 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65368cf7:::benchmark_data_smithi194_121512_object26586:head# 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65373689:::benchmark_data_smithi194_121512_object60730:head# 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65379122:::benchmark_data_smithi194_121512_object98979:head# 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6538a588:::benchmark_data_smithi194_121512_object42847:head# 2024-04-29T14:27:11.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6538d508:::benchmark_data_smithi194_121512_object30844:head# 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65398a1a:::benchmark_data_smithi194_121512_object107901:head# 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65398a92:::benchmark_data_smithi194_121512_object33445:head# 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6539ab11:::benchmark_data_smithi194_121512_object23836:head# 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653a011e:::benchmark_data_smithi194_121512_object18845:head# 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653aca17:::benchmark_data_smithi194_121512_object122577:head# 2024-04-29T14:27:11.450 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653aea7b:::benchmark_data_smithi194_121512_object63210:head# 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653b1c2a:::benchmark_data_smithi194_121512_object8789:head# 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653bfc0a:::benchmark_data_smithi194_121512_object26917:head# 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653c1e45:::benchmark_data_smithi194_121512_object40216:head# 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653cde72:::benchmark_data_smithi194_121512_object20119:head# 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653d1886:::benchmark_data_smithi194_121512_object63879:head# 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653d98ca:::benchmark_data_smithi194_121512_object107843:head# 2024-04-29T14:27:11.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653ed6eb:::benchmark_data_smithi194_121512_object54665:head# 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:653ff84e:::benchmark_data_smithi194_121512_object111019:head# 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65416669:::benchmark_data_smithi194_121512_object38653:head# 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65416fd5:::benchmark_data_smithi194_121512_object118807:head# 2024-04-29T14:27:11.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65437458:::benchmark_data_smithi194_121512_object126872:head# 2024-04-29T14:27:11.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6543f45e:::benchmark_data_smithi194_121512_object113895:head# 2024-04-29T14:27:11.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65446598:::benchmark_data_smithi194_121512_object78992:head# 2024-04-29T14:27:11.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6544e878:::benchmark_data_smithi194_121512_object60952:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65458327:::benchmark_data_smithi194_121512_object128854:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65474a6d:::benchmark_data_smithi194_121512_object6481:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6547675a:::benchmark_data_smithi194_121512_object38115:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6547a3a2:::benchmark_data_smithi194_121512_object106291:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6548bb46:::benchmark_data_smithi194_121512_object76144:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6549e83a:::benchmark_data_smithi194_121512_object61537:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654a6058:::benchmark_data_smithi194_121512_object64766:head# 2024-04-29T14:27:11.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654aab2a:::benchmark_data_smithi194_121512_object84972:head# 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654b1bc8:::benchmark_data_smithi194_121512_object58430:head# 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654b5aeb:::benchmark_data_smithi194_121512_object92147:head# 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654b7149:::benchmark_data_smithi194_121512_object83289:head# 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654bb09e:::benchmark_data_smithi194_121512_object88121:head# 2024-04-29T14:27:11.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654be810:::benchmark_data_smithi194_121512_object9020:head# 2024-04-29T14:27:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654c1340:::benchmark_data_smithi194_121512_object17630:head# 2024-04-29T14:27:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654c57da:::benchmark_data_smithi194_121512_object2426:head# 2024-04-29T14:27:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654c7cf9:::benchmark_data_smithi194_121512_object63361:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654d50c5:::benchmark_data_smithi194_121512_object78041:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654de0fa:::benchmark_data_smithi194_121512_object89247:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654df28f:::benchmark_data_smithi194_121512_object70589:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654e23cb:::benchmark_data_smithi194_121512_object4120:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654e8cdd:::benchmark_data_smithi194_121512_object102127:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654f8123:::benchmark_data_smithi194_121512_object90524:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:654f996a:::benchmark_data_smithi194_121512_object129548:head# 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.460 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6550628d:::benchmark_data_smithi194_121512_object115435:head# 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6550f501:::benchmark_data_smithi194_121512_object1742:head# 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65529934:::benchmark_data_smithi194_121512_object54636:head# 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6552d827:::benchmark_data_smithi194_121512_object101402:head# 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65534070:::benchmark_data_smithi194_121512_object13897:head# 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6553827d:::benchmark_data_smithi194_121512_object89180:head# 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655411f5:::benchmark_data_smithi194_121512_object119926:head# 2024-04-29T14:27:11.461 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6554ed27:::benchmark_data_smithi194_121512_object59854:head# 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6555adcd:::benchmark_data_smithi194_121512_object93944:head# 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6555d3b0:::benchmark_data_smithi194_121512_object103839:head# 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65560ca7:::benchmark_data_smithi194_121512_object11046:head# 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6556480b:::benchmark_data_smithi194_121512_object104939:head# 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6556507c:::benchmark_data_smithi194_121512_object126905:head# 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6556fd8a:::benchmark_data_smithi194_121512_object121738:head# 2024-04-29T14:27:11.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655738b9:::benchmark_data_smithi194_121512_object130876:head# 2024-04-29T14:27:11.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65574867:::benchmark_data_smithi194_121512_object117422:head# 2024-04-29T14:27:11.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65582fbb:::benchmark_data_smithi194_121512_object122747:head# 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65588fbb:::benchmark_data_smithi194_121512_object89616:head# 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6558c803:::benchmark_data_smithi194_121512_object35067:head# 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6559ec1a:::benchmark_data_smithi194_121512_object124770:head# 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6559fd2a:::benchmark_data_smithi194_121512_object97478:head# 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655a9c45:::benchmark_data_smithi194_121512_object98510:head# 2024-04-29T14:27:11.469 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:27:11.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655acce7:::benchmark_data_smithi194_121512_object56852:head# 2024-04-29T14:27:11.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.475 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655b13c9:::benchmark_data_smithi194_121512_object109994:head# 2024-04-29T14:27:11.475 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655b4fcd:::benchmark_data_smithi194_121512_object98296:head# 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655c0b33:::benchmark_data_smithi194_121512_object109180:head# 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655c97da:::benchmark_data_smithi194_121512_object23232:head# 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655caedb:::benchmark_data_smithi194_121512_object12917:head# 2024-04-29T14:27:11.476 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655cfb28:::benchmark_data_smithi194_121512_object70050:head# 2024-04-29T14:27:11.477 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.477 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655d4d92:::benchmark_data_smithi194_121512_object114232:head# 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655d6de0:::benchmark_data_smithi194_121512_object84996:head# 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655d7cc9:::benchmark_data_smithi194_121512_object109178:head# 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655d8ac1:::benchmark_data_smithi194_121512_object105764:head# 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655f201b:::benchmark_data_smithi194_121512_object97627:head# 2024-04-29T14:27:11.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655f4dd9:::benchmark_data_smithi194_121512_object67748:head# 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:655ff704:::benchmark_data_smithi194_121512_object82555:head# 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65605f54:::benchmark_data_smithi194_121512_object9091:head# 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6560fb65:::benchmark_data_smithi194_121512_object122919:head# 2024-04-29T14:27:11.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65614d53:::benchmark_data_smithi194_121512_object124549:head# 2024-04-29T14:27:11.480 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65614fba:::benchmark_data_smithi194_121512_object85796:head# 2024-04-29T14:27:11.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6561b82e:::benchmark_data_smithi194_121512_object131225:head# 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6561e73b:::benchmark_data_smithi194_121512_object1172:head# 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65624ad6:::benchmark_data_smithi194_121512_object120012:head# 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6562928e:::benchmark_data_smithi194_121512_object69571:head# 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65629e50:::benchmark_data_smithi194_121512_object53602:head# 2024-04-29T14:27:11.482 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6563847f:::benchmark_data_smithi194_121512_object124429:head# 2024-04-29T14:27:11.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6563928b:::benchmark_data_smithi194_121512_object56792:head# 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6564049d:::benchmark_data_smithi194_121512_object37124:head# 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656470e0:::benchmark_data_smithi194_121512_object128878:head# 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6564a7df:::benchmark_data_smithi194_121512_object116578:head# 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6565575d:::benchmark_data_smithi194_121512_object51630:head# 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65657e3d:::benchmark_data_smithi194_121512_object121752:head# 2024-04-29T14:27:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6566881f:::benchmark_data_smithi194_121512_object109480:head# 2024-04-29T14:27:11.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65669d39:::benchmark_data_smithi194_121512_object23526:head# 2024-04-29T14:27:11.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6566a54b:::benchmark_data_smithi194_121512_object1581:head# 2024-04-29T14:27:11.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6566f122:::benchmark_data_smithi194_121512_object95825:head# 2024-04-29T14:27:11.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6566f5c4:::benchmark_data_smithi194_121512_object91249:head# 2024-04-29T14:27:11.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65674053:::benchmark_data_smithi194_121512_object122059:head# 2024-04-29T14:27:11.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65685227:::benchmark_data_smithi194_121512_object53398:head# 2024-04-29T14:27:11.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65689bd5:::benchmark_data_smithi194_121512_object118202:head# 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6568ea66:::benchmark_data_smithi194_121512_object61671:head# 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6568f031:::benchmark_data_smithi194_121512_object25302:head# 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65691a3c:::benchmark_data_smithi194_121512_object121038:head# 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6569a6c4:::benchmark_data_smithi194_121512_object20932:head# 2024-04-29T14:27:11.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6569d5f9:::benchmark_data_smithi194_121512_object39733:head# 2024-04-29T14:27:11.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6569f6ad:::benchmark_data_smithi194_121512_object119695:head# 2024-04-29T14:27:11.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656a6707:::benchmark_data_smithi194_121512_object7442:head# 2024-04-29T14:27:11.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656b0178:::benchmark_data_smithi194_121512_object45772:head# 2024-04-29T14:27:11.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656b0499:::benchmark_data_smithi194_121512_object56845:head# 2024-04-29T14:27:11.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656b7cc7:::benchmark_data_smithi194_121512_object64147:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656ba6c5:::benchmark_data_smithi194_121512_object40061:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656c1476:::benchmark_data_smithi194_121512_object122290:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656d13a4:::benchmark_data_smithi194_121512_object127121:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656ef0cc:::benchmark_data_smithi194_121512_object123246:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656f1bd5:::benchmark_data_smithi194_121512_object132552:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656ff8a5:::benchmark_data_smithi194_121512_object30811:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:656ff989:::benchmark_data_smithi194_121512_object100302:head# 2024-04-29T14:27:11.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6570640d:::benchmark_data_smithi194_121512_object29051:head# 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6571b642:::benchmark_data_smithi194_121512_object86381:head# 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6571ef96:::benchmark_data_smithi194_121512_object90564:head# 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6572c09b:::benchmark_data_smithi194_121512_object13923:head# 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6572dcce:::benchmark_data_smithi194_121512_object68009:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6572e137:::benchmark_data_smithi194_121512_object87647:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65731e85:::benchmark_data_smithi194_121512_object88580:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6573f562:::benchmark_data_smithi194_121512_object95296:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65748a76:::benchmark_data_smithi194_121512_object17982:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6574a58e:::benchmark_data_smithi194_121512_object66808:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6574d1ce:::benchmark_data_smithi194_121512_object17648:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6575349e:::benchmark_data_smithi194_121512_object115743:head# 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657556eb:::benchmark_data_smithi194_121512_object108385:head# 2024-04-29T14:27:11.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6575878b:::benchmark_data_smithi194_121512_object266:head# 2024-04-29T14:27:11.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65759c2a:::benchmark_data_smithi194_121512_object30264:head# 2024-04-29T14:27:11.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:27:11.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6575e843:::benchmark_data_smithi194_121512_object88113:head# 2024-04-29T14:27:11.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65760b5b:::benchmark_data_smithi194_121512_object8298:head# 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65766303:::benchmark_data_smithi194_121512_object71369:head# 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6576ccd7:::benchmark_data_smithi194_121512_object102071:head# 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6577102e:::benchmark_data_smithi194_121512_object28890:head# 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6577120e:::benchmark_data_smithi194_121512_object129049:head# 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65782c04:::benchmark_data_smithi194_121512_object133383:head# 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6578613c:::benchmark_data_smithi194_121512_object73574:head# 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6579bed7:::benchmark_data_smithi194_121512_object118027:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657a2262:::benchmark_data_smithi194_121512_object30834:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657b7e18:::benchmark_data_smithi194_121512_object97790:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657bb644:::benchmark_data_smithi194_121512_object55403:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657cae0e:::benchmark_data_smithi194_121512_object111561:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657d598a:::benchmark_data_smithi194_121512_object72057:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657e2ba3:::benchmark_data_smithi194_121512_object1020:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657ea150:::benchmark_data_smithi194_121512_object52639:head# 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657ede44:::benchmark_data_smithi194_121512_object51712:head# 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657f5b83:::benchmark_data_smithi194_121512_object35337:head# 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657f66bc:::benchmark_data_smithi194_121512_object45384:head# 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657f77e6:::benchmark_data_smithi194_121512_object105415:head# 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657f79b8:::benchmark_data_smithi194_121512_object131330:head# 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657f7d68:::benchmark_data_smithi194_121512_object75231:head# 2024-04-29T14:27:11.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:657fb8a7:::benchmark_data_smithi194_121512_object85463:head# 2024-04-29T14:27:11.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6580764e:::benchmark_data_smithi194_121512_object110574:head# 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6581f8b7:::benchmark_data_smithi194_121512_object9982:head# 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6581fe18:::benchmark_data_smithi194_121512_object117352:head# 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6582006c:::benchmark_data_smithi194_121512_object24529:head# 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65821f36:::benchmark_data_smithi194_121512_object23416:head# 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65828f44:::benchmark_data_smithi194_121512_object88400:head# 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6582b39d:::benchmark_data_smithi194_121512_object67711:head# 2024-04-29T14:27:11.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.507 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65852e04:::benchmark_data_smithi194_121512_object8382:head# 2024-04-29T14:27:11.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65859c91:::benchmark_data_smithi194_121512_object18726:head# 2024-04-29T14:27:11.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6585c7d5:::benchmark_data_smithi194_121512_object95096:head# 2024-04-29T14:27:11.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6585cf66:::benchmark_data_smithi194_121512_object82130:head# 2024-04-29T14:27:11.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65867276:::benchmark_data_smithi194_121512_object114353:head# 2024-04-29T14:27:11.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658718f6:::benchmark_data_smithi194_121512_object98284:head# 2024-04-29T14:27:11.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65877410:::benchmark_data_smithi194_121512_object35667:head# 2024-04-29T14:27:11.511 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.511 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65884dbf:::benchmark_data_smithi194_121512_object114403:head# 2024-04-29T14:27:11.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65889451:::benchmark_data_smithi194_121512_object5291:head# 2024-04-29T14:27:11.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6588c2a6:::benchmark_data_smithi194_121512_object40433:head# 2024-04-29T14:27:11.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.512 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6589b08b:::benchmark_data_smithi194_121512_object16011:head# 2024-04-29T14:27:11.512 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.513 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658a82d0:::benchmark_data_smithi194_121512_object28950:head# 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658aaaad:::benchmark_data_smithi194_121512_object81753:head# 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658aeba5:::benchmark_data_smithi194_121512_object93343:head# 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658b8b80:::benchmark_data_smithi194_121512_object94635:head# 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658baa49:::benchmark_data_smithi194_121512_object2802:head# 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658cdd98:::benchmark_data_smithi194_121512_object106484:head# 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658d07f1:::benchmark_data_smithi194_121512_object51190:head# 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.514 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658d5820:::benchmark_data_smithi194_121512_object108542:head# 2024-04-29T14:27:11.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658d6fb4:::benchmark_data_smithi194_121512_object110879:head# 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658def58:::benchmark_data_smithi194_121512_object132574:head# 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658e1f6a:::benchmark_data_smithi194_121512_object118904:head# 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:658faef1:::benchmark_data_smithi194_121512_object51390:head# 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6590026d:::benchmark_data_smithi194_121512_object43556:head# 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659058f2:::benchmark_data_smithi194_121512_object28322:head# 2024-04-29T14:27:11.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:27:11.522 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6590ac29:::benchmark_data_smithi194_121512_object58821:head# 2024-04-29T14:27:11.522 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6590d33d:::benchmark_data_smithi194_121512_object7161:head# 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6590e802:::benchmark_data_smithi194_121512_object87662:head# 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65910cd9:::benchmark_data_smithi194_121512_object53878:head# 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659144d8:::benchmark_data_smithi194_121512_object66049:head# 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65916a97:::benchmark_data_smithi194_121512_object41073:head# 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65918b71:::benchmark_data_smithi194_121512_object38121:head# 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6591a2e0:::benchmark_data_smithi194_121512_object104394:head# 2024-04-29T14:27:11.523 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6592919b:::benchmark_data_smithi194_121512_object93566:head# 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6592bde4:::benchmark_data_smithi194_121512_object43132:head# 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6592bfff:::benchmark_data_smithi194_121512_object43711:head# 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6593c150:::benchmark_data_smithi194_121512_object35701:head# 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6594674c:::benchmark_data_smithi194_121512_object121893:head# 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6594f3da:::benchmark_data_smithi194_121512_object42635:head# 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65954f9e:::benchmark_data_smithi194_121512_object100704:head# 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.524 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65958fa7:::benchmark_data_smithi194_121512_object55504:head# 2024-04-29T14:27:11.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65960032:::benchmark_data_smithi194_121512_object3448:head# 2024-04-29T14:27:11.525 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.525 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65961fa4:::benchmark_data_smithi194_121512_object27342:head# 2024-04-29T14:27:11.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.526 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659688c2:::benchmark_data_smithi194_121512_object53068:head# 2024-04-29T14:27:11.526 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65973388:::benchmark_data_smithi194_121512_object107283:head# 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6597ec07:::benchmark_data_smithi194_121512_object33490:head# 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65985c9a:::benchmark_data_smithi194_121512_object55717:head# 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659865e6:::benchmark_data_smithi194_121512_object14653:head# 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.528 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65989fd6:::benchmark_data_smithi194_121512_object117336:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659916cf:::benchmark_data_smithi194_121512_object42507:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65993b6b:::benchmark_data_smithi194_121512_object131639:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65993ef1:::benchmark_data_smithi194_121512_object63472:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6599e422:::benchmark_data_smithi194_121512_object44374:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659a1276:::benchmark_data_smithi194_121512_object24441:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659c2dad:::benchmark_data_smithi194_121512_object75395:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659c44b4:::benchmark_data_smithi194_121512_object30137:head# 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659d6015:::benchmark_data_smithi194_121512_object130688:head# 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659da3f4:::benchmark_data_smithi194_121512_object6366:head# 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659e1a4e:::benchmark_data_smithi194_121512_object53120:head# 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659e2a60:::benchmark_data_smithi194_121512_object32938:head# 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659e6fa6:::benchmark_data_smithi194_121512_object24489:head# 2024-04-29T14:27:11.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659e89c7:::benchmark_data_smithi194_121512_object72516:head# 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659f707e:::benchmark_data_smithi194_121512_object93116:head# 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:659fe6d9:::benchmark_data_smithi194_121512_object59098:head# 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a03f2f:::benchmark_data_smithi194_121512_object129019:head# 2024-04-29T14:27:11.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a0523c:::benchmark_data_smithi194_121512_object74211:head# 2024-04-29T14:27:11.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a0c8a2:::benchmark_data_smithi194_121512_object15130:head# 2024-04-29T14:27:11.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a15e43:::benchmark_data_smithi194_121512_object115898:head# 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a1661c:::benchmark_data_smithi194_121512_object21397:head# 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a18642:::benchmark_data_smithi194_121512_object41632:head# 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a1d3ca:::benchmark_data_smithi194_121512_object23842:head# 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a30667:::benchmark_data_smithi194_121512_object115901:head# 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a33653:::benchmark_data_smithi194_121512_object39382:head# 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a3d243:::benchmark_data_smithi194_121512_object97441:head# 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a3f447:::benchmark_data_smithi194_121512_object92542:head# 2024-04-29T14:27:11.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a3f8ba:::benchmark_data_smithi194_121512_object31089:head# 2024-04-29T14:27:11.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a4652c:::benchmark_data_smithi194_121512_object42276:head# 2024-04-29T14:27:11.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a56169:::benchmark_data_smithi194_121512_object11973:head# 2024-04-29T14:27:11.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.538 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a5955f:::benchmark_data_smithi194_121512_object20136:head# 2024-04-29T14:27:11.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a5b691:::benchmark_data_smithi194_121512_object97021:head# 2024-04-29T14:27:11.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a5c87f:::benchmark_data_smithi194_121512_object82840:head# 2024-04-29T14:27:11.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a5cb3e:::benchmark_data_smithi194_121512_object125079:head# 2024-04-29T14:27:11.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a67d78:::benchmark_data_smithi194_121512_object26900:head# 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a6b2c7:::benchmark_data_smithi194_121512_object90587:head# 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a6f9b3:::benchmark_data_smithi194_121512_object19963:head# 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a7c2e7:::benchmark_data_smithi194_121512_object118068:head# 2024-04-29T14:27:11.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.542 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a82c89:::benchmark_data_smithi194_121512_object127857:head# 2024-04-29T14:27:11.542 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a8709d:::benchmark_data_smithi194_121512_object93693:head# 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a8961e:::benchmark_data_smithi194_121512_object110367:head# 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65a9d9eb:::benchmark_data_smithi194_121512_object2850:head# 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65aa3a65:::benchmark_data_smithi194_121512_object17437:head# 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65aa3e48:::benchmark_data_smithi194_121512_object14510:head# 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65abf29e:::benchmark_data_smithi194_121512_object26762:head# 2024-04-29T14:27:11.543 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ae9660:::benchmark_data_smithi194_121512_object130824:head# 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ae9c2c:::benchmark_data_smithi194_121512_object98902:head# 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65af104b:::benchmark_data_smithi194_121512_object78743:head# 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65af19d3:::benchmark_data_smithi194_121512_object65237:head# 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65af7309:::benchmark_data_smithi194_121512_object14302:head# 2024-04-29T14:27:11.545 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65afa600:::benchmark_data_smithi194_121512_object41939:head# 2024-04-29T14:27:11.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65afc4df:::benchmark_data_smithi194_121512_object104355:head# 2024-04-29T14:27:11.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.546 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65afe071:::benchmark_data_smithi194_121512_object70373:head# 2024-04-29T14:27:11.546 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b01bbf:::benchmark_data_smithi194_121512_object105244:head# 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b07019:::benchmark_data_smithi194_121512_object13623:head# 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b0d033:::benchmark_data_smithi194_121512_object29469:head# 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b0de2f:::benchmark_data_smithi194_121512_object13006:head# 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b10d07:::benchmark_data_smithi194_121512_object55854:head# 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b1b1bc:::benchmark_data_smithi194_121512_object27367:head# 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b1cf7e:::benchmark_data_smithi194_121512_object12722:head# 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b28d53:::benchmark_data_smithi194_121512_object28530:head# 2024-04-29T14:27:11.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b2dba5:::benchmark_data_smithi194_121512_object95429:head# 2024-04-29T14:27:11.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b2e2ef:::benchmark_data_smithi194_121512_object105842:head# 2024-04-29T14:27:11.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b3082b:::benchmark_data_smithi194_121512_object65622:head# 2024-04-29T14:27:11.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b31a84:::benchmark_data_smithi194_121512_object102474:head# 2024-04-29T14:27:11.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b34fd0:::benchmark_data_smithi194_121512_object15982:head# 2024-04-29T14:27:11.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b3dd62:::benchmark_data_smithi194_121512_object109154:head# 2024-04-29T14:27:11.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b3eba9:::benchmark_data_smithi194_121512_object9605:head# 2024-04-29T14:27:11.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b46bc7:::benchmark_data_smithi194_121512_object21499:head# 2024-04-29T14:27:11.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b48b0e:::benchmark_data_smithi194_121512_object2328:head# 2024-04-29T14:27:11.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b4daa3:::benchmark_data_smithi194_121512_object93853:head# 2024-04-29T14:27:11.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b4daee:::benchmark_data_smithi194_121512_object93740:head# 2024-04-29T14:27:11.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b643bf:::benchmark_data_smithi194_121512_object26548:head# 2024-04-29T14:27:11.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b7cb66:::benchmark_data_smithi194_121512_object80569:head# 2024-04-29T14:27:11.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b859c2:::benchmark_data_smithi194_121512_object102546:head# 2024-04-29T14:27:11.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.560 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:11.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65b9fa86:::benchmark_data_smithi194_121512_object51289:head# 2024-04-29T14:27:11.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ba6f2c:::benchmark_data_smithi194_121512_object10330:head# 2024-04-29T14:27:11.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ba90ec:::benchmark_data_smithi194_121512_object61400:head# 2024-04-29T14:27:11.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bad191:::benchmark_data_smithi194_121512_object78891:head# 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bb6235:::benchmark_data_smithi194_121512_object83127:head# 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bb81bd:::benchmark_data_smithi194_121512_object26248:head# 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bb96a3:::benchmark_data_smithi194_121512_object35179:head# 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bba451:::benchmark_data_smithi194_121512_object78782:head# 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bbca23:::benchmark_data_smithi194_121512_object84368:head# 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bc525d:::benchmark_data_smithi194_121512_object79988:head# 2024-04-29T14:27:11.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bc6de5:::benchmark_data_smithi194_121512_object34299:head# 2024-04-29T14:27:11.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.563 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bd426c:::benchmark_data_smithi194_121512_object95776:head# 2024-04-29T14:27:11.563 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bd4f0a:::benchmark_data_smithi194_121512_object7227:head# 2024-04-29T14:27:11.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bdd3e1:::benchmark_data_smithi194_121512_object72064:head# 2024-04-29T14:27:11.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bed5c2:::benchmark_data_smithi194_121512_object101554:head# 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65bf161d:::benchmark_data_smithi194_121512_object115533:head# 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c00e38:::benchmark_data_smithi194_121512_object7414:head# 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c02a39:::benchmark_data_smithi194_121512_object36556:head# 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c0beae:::benchmark_data_smithi194_121512_object63154:head# 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c33702:::benchmark_data_smithi194_121512_object45680:head# 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c37bd0:::benchmark_data_smithi194_121512_object11918:head# 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c39a81:::benchmark_data_smithi194_121512_object80828:head# 2024-04-29T14:27:11.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:11 smithi194 ceph-mon[44089]: pgmap v1273: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 430 op/s; 32988/210258 objects degraded (15.689%); 8.5 MiB/s, 135 objects/s recovering 2024-04-29T14:27:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:11 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:11 smithi194 ceph-mon[44089]: from='client.36080 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:27:11.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:11 smithi194 ceph-mon[44089]: osdmap e482: 12 total, 11 up, 7 in 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c3ea2f:::benchmark_data_smithi194_121512_object43750:head# 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c63c6e:::benchmark_data_smithi194_121512_object33039:head# 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c6443f:::benchmark_data_smithi194_121512_object27790:head# 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c78184:::benchmark_data_smithi194_121512_object108241:head# 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c7ea3b:::benchmark_data_smithi194_121512_object10896:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c7f52a:::benchmark_data_smithi194_121512_object22135:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c88282:::benchmark_data_smithi194_121512_object20556:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65c9fcd3:::benchmark_data_smithi194_121512_object9055:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ca11bf:::benchmark_data_smithi194_121512_object27140:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ca879c:::benchmark_data_smithi194_121512_object112090:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65caaf7a:::benchmark_data_smithi194_121512_object119952:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cb0d60:::benchmark_data_smithi194_121512_object24867:head# 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ccb619:::benchmark_data_smithi194_121512_object99418:head# 2024-04-29T14:27:11.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ccd1d7:::benchmark_data_smithi194_121512_object129381:head# 2024-04-29T14:27:11.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cd05ec:::benchmark_data_smithi194_121512_object14403:head# 2024-04-29T14:27:11.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:27:11.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cd3a52:::benchmark_data_smithi194_121512_object106746:head# 2024-04-29T14:27:11.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cd4595:::benchmark_data_smithi194_121512_object89597:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cd5ca8:::benchmark_data_smithi194_121512_object69811:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cd81b3:::benchmark_data_smithi194_121512_object55421:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ce98c3:::benchmark_data_smithi194_121512_object93030:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ceaa88:::benchmark_data_smithi194_121512_object34610:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cf31e4:::benchmark_data_smithi194_121512_object21824:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cf3846:::benchmark_data_smithi194_121512_object14026:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cf5791:::benchmark_data_smithi194_121512_object97777:head# 2024-04-29T14:27:11.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cf68d0:::benchmark_data_smithi194_121512_object61123:head# 2024-04-29T14:27:11.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65cfb12b:::benchmark_data_smithi194_121512_object62081:head# 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d0a575:::benchmark_data_smithi194_121512_object45776:head# 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d0ca6a:::benchmark_data_smithi194_121512_object26705:head# 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d1e851:::benchmark_data_smithi194_121512_object32632:head# 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d22f02:::benchmark_data_smithi194_121512_object48285:head# 2024-04-29T14:27:11.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d45854:::benchmark_data_smithi194_121512_object74877:head# 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d52a61:::benchmark_data_smithi194_121512_object95717:head# 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d5e902:::benchmark_data_smithi194_121512_object93057:head# 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d72a05:::benchmark_data_smithi194_121512_object56787:head# 2024-04-29T14:27:11.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d7472c:::benchmark_data_smithi194_121512_object95336:head# 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d7edaa:::benchmark_data_smithi194_121512_object129715:head# 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d8fdfe:::benchmark_data_smithi194_121512_object8279:head# 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d97df8:::benchmark_data_smithi194_121512_object50896:head# 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d99776:::benchmark_data_smithi194_121512_object73820:head# 2024-04-29T14:27:11.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65d9b27f:::benchmark_data_smithi194_121512_object111534:head# 2024-04-29T14:27:11.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65daa285:::benchmark_data_smithi194_121512_object83013:head# 2024-04-29T14:27:11.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65dabed7:::benchmark_data_smithi194_121512_object12234:head# 2024-04-29T14:27:11.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65daf3aa:::benchmark_data_smithi194_121512_object108080:head# 2024-04-29T14:27:11.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65db5956:::benchmark_data_smithi194_121512_object119924:head# 2024-04-29T14:27:11.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65db5ba8:::benchmark_data_smithi194_121512_object47161:head# 2024-04-29T14:27:11.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65dbac1c:::benchmark_data_smithi194_121512_object25101:head# 2024-04-29T14:27:11.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65dc0ffe:::benchmark_data_smithi194_121512_object58114:head# 2024-04-29T14:27:11.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65dc732d:::benchmark_data_smithi194_121512_object130845:head# 2024-04-29T14:27:11.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65dddb34:::benchmark_data_smithi194_121512_object129565:head# 2024-04-29T14:27:11.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65dde095:::benchmark_data_smithi194_121512_object77428:head# 2024-04-29T14:27:11.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65de64d2:::benchmark_data_smithi194_121512_object114659:head# 2024-04-29T14:27:11.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e053a4:::benchmark_data_smithi194_121512_object64137:head# 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e05689:::benchmark_data_smithi194_121512_object62318:head# 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e1918f:::benchmark_data_smithi194_121512_object93326:head# 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e1cd24:::benchmark_data_smithi194_121512_object12374:head# 2024-04-29T14:27:11.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e24785:::benchmark_data_smithi194_121512_object80702:head# 2024-04-29T14:27:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e2b630:::benchmark_data_smithi194_121512_object88061:head# 2024-04-29T14:27:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e2f4f4:::benchmark_data_smithi194_121512_object100178:head# 2024-04-29T14:27:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.592 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:27:11.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e32c63:::benchmark_data_smithi194_121512_object127835:head# 2024-04-29T14:27:11.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e35ff1:::benchmark_data_smithi194_121512_object56336:head# 2024-04-29T14:27:11.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e36810:::benchmark_data_smithi194_121512_object107739:head# 2024-04-29T14:27:11.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e47781:::benchmark_data_smithi194_121512_object60705:head# 2024-04-29T14:27:11.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e4ae28:::benchmark_data_smithi194_121512_object130801:head# 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e4b2ef:::benchmark_data_smithi194_121512_object89233:head# 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e4c4ef:::benchmark_data_smithi194_121512_object73684:head# 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e4ccdd:::benchmark_data_smithi194_121512_object131762:head# 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e53e02:::benchmark_data_smithi194_121512_object124618:head# 2024-04-29T14:27:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e5534e:::benchmark_data_smithi194_121512_object26416:head# 2024-04-29T14:27:11.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e58eec:::benchmark_data_smithi194_121512_object94264:head# 2024-04-29T14:27:11.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e73d2a:::benchmark_data_smithi194_121512_object67466:head# 2024-04-29T14:27:11.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e76e35:::benchmark_data_smithi194_121512_object94646:head# 2024-04-29T14:27:11.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e87731:::benchmark_data_smithi194_121512_object77331:head# 2024-04-29T14:27:11.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e88f1c:::benchmark_data_smithi194_121512_object25854:head# 2024-04-29T14:27:11.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e8b98b:::benchmark_data_smithi194_121512_object133326:head# 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e8e39c:::benchmark_data_smithi194_121512_object13036:head# 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e949ed:::benchmark_data_smithi194_121512_object50859:head# 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e96b39:::benchmark_data_smithi194_121512_object8558:head# 2024-04-29T14:27:11.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65e9c52c:::benchmark_data_smithi194_121512_object9480:head# 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ea076a:::benchmark_data_smithi194_121512_object86966:head# 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65eb5c5c:::benchmark_data_smithi194_121512_object48488:head# 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65eb6629:::benchmark_data_smithi194_121512_object46635:head# 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ebcee0:::benchmark_data_smithi194_121512_object6929:head# 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ec0b9a:::benchmark_data_smithi194_121512_object114979:head# 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ec79ae:::benchmark_data_smithi194_121512_object95227:head# 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ecb919:::benchmark_data_smithi194_121512_object83751:head# 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ed762c:::benchmark_data_smithi194_121512_object22026:head# 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ed7942:::benchmark_data_smithi194_121512_object18597:head# 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ee11c1:::benchmark_data_smithi194_121512_object50103:head# 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ee906f:::benchmark_data_smithi194_121512_object10030:head# 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65eefed3:::benchmark_data_smithi194_121512_object124636:head# 2024-04-29T14:27:11.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ef850e:::benchmark_data_smithi194_121512_object14188:head# 2024-04-29T14:27:11.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65efb9bf:::benchmark_data_smithi194_121512_object54929:head# 2024-04-29T14:27:11.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f008df:::benchmark_data_smithi194_121512_object85414:head# 2024-04-29T14:27:11.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f043c7:::benchmark_data_smithi194_121512_object24034:head# 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f0f42a:::benchmark_data_smithi194_121512_object131191:head# 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f1c4c2:::benchmark_data_smithi194_121512_object38792:head# 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f33eec:::benchmark_data_smithi194_121512_object27664:head# 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f3ae4d:::benchmark_data_smithi194_121512_object112129:head# 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f3b442:::benchmark_data_smithi194_121512_object88853:head# 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f40861:::benchmark_data_smithi194_121512_object48279:head# 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f4b7f6:::benchmark_data_smithi194_121512_object46198:head# 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f50d9c:::benchmark_data_smithi194_121512_object43344:head# 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f5f8cc:::benchmark_data_smithi194_121512_object86595:head# 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f666f1:::benchmark_data_smithi194_121512_object19386:head# 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f7d6b7:::benchmark_data_smithi194_121512_object33959:head# 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f821fd:::benchmark_data_smithi194_121512_object66144:head# 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f84e8b:::benchmark_data_smithi194_121512_object35192:head# 2024-04-29T14:27:11.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.612 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65f89df8:::benchmark_data_smithi194_121512_object133248:head# 2024-04-29T14:27:11.612 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fa16ff:::benchmark_data_smithi194_121512_object122801:head# 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fabc95:::benchmark_data_smithi194_121512_object34625:head# 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fb1629:::benchmark_data_smithi194_121512_object85698:head# 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fb94a8:::benchmark_data_smithi194_121512_object7360:head# 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fc8d42:::benchmark_data_smithi194_121512_object71696:head# 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fd5f50:::benchmark_data_smithi194_121512_object55218:head# 2024-04-29T14:27:11.613 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.614 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fe294e:::benchmark_data_smithi194_121512_object97443:head# 2024-04-29T14:27:11.614 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fe923e:::benchmark_data_smithi194_121512_object32984:head# 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ff0992:::benchmark_data_smithi194_121512_object112843:head# 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65ffef52:::benchmark_data_smithi194_121512_object124391:head# 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:65fffb7e:::benchmark_data_smithi194_121512_object65401:head# 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66004278:::benchmark_data_smithi194_121512_object24233:head# 2024-04-29T14:27:11.615 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66005234:::benchmark_data_smithi194_121512_object95409:head# 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66011701:::benchmark_data_smithi194_121512_object58138:head# 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6601cffb:::benchmark_data_smithi194_121512_object125844:head# 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6601f0c9:::benchmark_data_smithi194_121512_object95059:head# 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660216b8:::benchmark_data_smithi194_121512_object44123:head# 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.617 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6602fa6e:::benchmark_data_smithi194_121512_object125158:head# 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660391ec:::benchmark_data_smithi194_121512_object94715:head# 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66046d99:::benchmark_data_smithi194_121512_object31847:head# 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6604f73a:::benchmark_data_smithi194_121512_object128503:head# 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6605f629:::benchmark_data_smithi194_121512_object129562:head# 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6606ef2f:::benchmark_data_smithi194_121512_object124968:head# 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66073e65:::benchmark_data_smithi194_121512_object131564:head# 2024-04-29T14:27:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660799ef:::benchmark_data_smithi194_121512_object82445:head# 2024-04-29T14:27:11.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6607d93a:::benchmark_data_smithi194_121512_object47830:head# 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660803d5:::benchmark_data_smithi194_121512_object40939:head# 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66081873:::benchmark_data_smithi194_121512_object35052:head# 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6608cfd1:::benchmark_data_smithi194_121512_object70422:head# 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6608e382:::benchmark_data_smithi194_121512_object92723:head# 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6608ed32:::benchmark_data_smithi194_121512_object81966:head# 2024-04-29T14:27:11.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6608f0ac:::benchmark_data_smithi194_121512_object64165:head# 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660995c6:::benchmark_data_smithi194_121512_object123054:head# 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660996a8:::benchmark_data_smithi194_121512_object71764:head# 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660ab413:::benchmark_data_smithi194_121512_object14832:head# 2024-04-29T14:27:11.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660b3c18:::benchmark_data_smithi194_121512_object119034:head# 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660b775b:::benchmark_data_smithi194_121512_object38982:head# 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660c121e:::benchmark_data_smithi194_121512_object84486:head# 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660c21f9:::benchmark_data_smithi194_121512_object30283:head# 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660c22e1:::benchmark_data_smithi194_121512_object101415:head# 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660c48be:::benchmark_data_smithi194_121512_object83586:head# 2024-04-29T14:27:11.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660c9045:::benchmark_data_smithi194_121512_object45862:head# 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660d15c5:::benchmark_data_smithi194_121512_object18713:head# 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660de4ca:::benchmark_data_smithi194_121512_object116994:head# 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660edbee:::benchmark_data_smithi194_121512_object100425:head# 2024-04-29T14:27:11.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:660f9f7e:::benchmark_data_smithi194_121512_object103027:head# 2024-04-29T14:27:11.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66100b9e:::benchmark_data_smithi194_121512_object55524:head# 2024-04-29T14:27:11.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6611688a:::benchmark_data_smithi194_121512_object130426:head# 2024-04-29T14:27:11.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6611835a:::benchmark_data_smithi194_121512_object21719:head# 2024-04-29T14:27:11.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6611cd9b:::benchmark_data_smithi194_121512_object131724:head# 2024-04-29T14:27:11.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:11 smithi121 ceph-mon[44533]: pgmap v1273: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 430 op/s; 32988/210258 objects degraded (15.689%); 8.5 MiB/s, 135 objects/s recovering 2024-04-29T14:27:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:11 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:11 smithi121 ceph-mon[44533]: from='client.36080 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:27:11.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:11 smithi121 ceph-mon[44533]: osdmap e482: 12 total, 11 up, 7 in 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6611f3d4:::benchmark_data_smithi194_121512_object87218:head# 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6612e537:::benchmark_data_smithi194_121512_object45761:head# 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6614e786:::benchmark_data_smithi194_121512_object42632:head# 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6614f192:::benchmark_data_smithi194_121512_object110669:head# 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66152641:::benchmark_data_smithi194_121512_object115872:head# 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66164689:::benchmark_data_smithi194_121512_object3230:head# 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66170a75:::benchmark_data_smithi194_121512_object114455:head# 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661740c1:::benchmark_data_smithi194_121512_object117271:head# 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66179036:::benchmark_data_smithi194_121512_object121097:head# 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66183b34:::benchmark_data_smithi194_121512_object23859:head# 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6618c372:::benchmark_data_smithi194_121512_object45590:head# 2024-04-29T14:27:11.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6619eaa2:::benchmark_data_smithi194_121512_object97964:head# 2024-04-29T14:27:11.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661a1bcf:::benchmark_data_smithi194_121512_object23743:head# 2024-04-29T14:27:11.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661a7173:::benchmark_data_smithi194_121512_object108098:head# 2024-04-29T14:27:11.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661b38cc:::benchmark_data_smithi194_121512_object32305:head# 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661b683b:::benchmark_data_smithi194_121512_object65007:head# 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661be344:::benchmark_data_smithi194_121512_object81139:head# 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661c1650:::benchmark_data_smithi194_121512_object79751:head# 2024-04-29T14:27:11.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661cde5b:::benchmark_data_smithi194_121512_object28767:head# 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661d5327:::benchmark_data_smithi194_121512_object102986:head# 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661efd17:::benchmark_data_smithi194_121512_object122802:head# 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661fbfd9:::benchmark_data_smithi194_121512_object123345:head# 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:661ff19d:::benchmark_data_smithi194_121512_object125255:head# 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662028f8:::benchmark_data_smithi194_121512_object128859:head# 2024-04-29T14:27:11.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66203829:::benchmark_data_smithi194_121512_object39711:head# 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6620d27d:::benchmark_data_smithi194_121512_object43386:head# 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6620fa1c:::benchmark_data_smithi194_121512_object74199:head# 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6621481f:::benchmark_data_smithi194_121512_object105125:head# 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662152ad:::benchmark_data_smithi194_121512_object19604:head# 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66221fe1:::benchmark_data_smithi194_121512_object25698:head# 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6622609b:::benchmark_data_smithi194_121512_object116078:head# 2024-04-29T14:27:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662319ef:::benchmark_data_smithi194_121512_object80474:head# 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6625dcef:::benchmark_data_smithi194_121512_object113168:head# 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6626a72c:::benchmark_data_smithi194_121512_object61416:head# 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6626bc9a:::benchmark_data_smithi194_121512_object72240:head# 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.638 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:27:11.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66270e2f:::benchmark_data_smithi194_121512_object69368:head# 2024-04-29T14:27:11.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66277e62:::benchmark_data_smithi194_121512_object16149:head# 2024-04-29T14:27:11.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66281702:::benchmark_data_smithi194_121512_object37232:head# 2024-04-29T14:27:11.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:11.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66284b82:::benchmark_data_smithi194_121512_object50743:head# 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6628a7fc:::benchmark_data_smithi194_121512_object112596:head# 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6628f71a:::benchmark_data_smithi194_121512_object59892:head# 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662aed06:::benchmark_data_smithi194_121512_object64143:head# 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662afe68:::benchmark_data_smithi194_121512_object116526:head# 2024-04-29T14:27:11.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662b49f2:::benchmark_data_smithi194_121512_object37088:head# 2024-04-29T14:27:11.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662bc5e4:::benchmark_data_smithi194_121512_object92349:head# 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662c7275:::benchmark_data_smithi194_121512_object61641:head# 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662cbf6f:::benchmark_data_smithi194_121512_object102844:head# 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662cc8be:::benchmark_data_smithi194_121512_object124121:head# 2024-04-29T14:27:11.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662de40b:::benchmark_data_smithi194_121512_object50016:head# 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662e0c58:::benchmark_data_smithi194_121512_object73237:head# 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662f294a:::benchmark_data_smithi194_121512_object36565:head# 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662fb3a5:::benchmark_data_smithi194_121512_object61362:head# 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:27:11.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:662fd33e:::benchmark_data_smithi194_121512_object107480:head# 2024-04-29T14:27:11.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66309dde:::benchmark_data_smithi194_121512_object76836:head# 2024-04-29T14:27:11.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66331ce1:::benchmark_data_smithi194_121512_object45397:head# 2024-04-29T14:27:11.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663343cb:::benchmark_data_smithi194_121512_object64717:head# 2024-04-29T14:27:11.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663527d5:::benchmark_data_smithi194_121512_object110803:head# 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6635e017:::benchmark_data_smithi194_121512_object5112:head# 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6636384d:::benchmark_data_smithi194_121512_object98013:head# 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663652ef:::benchmark_data_smithi194_121512_object25504:head# 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6636dca2:::benchmark_data_smithi194_121512_object97682:head# 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66377f42:::benchmark_data_smithi194_121512_object122203:head# 2024-04-29T14:27:11.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6637826a:::benchmark_data_smithi194_121512_object123238:head# 2024-04-29T14:27:11.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6638da1f:::benchmark_data_smithi194_121512_object22539:head# 2024-04-29T14:27:11.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663901ea:::benchmark_data_smithi194_121512_object108628:head# 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663a6cb3:::benchmark_data_smithi194_121512_object1939:head# 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663ab0e1:::benchmark_data_smithi194_121512_object79580:head# 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663ad323:::benchmark_data_smithi194_121512_object63400:head# 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663ae05f:::benchmark_data_smithi194_121512_object64104:head# 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663ae74e:::benchmark_data_smithi194_121512_object17966:head# 2024-04-29T14:27:11.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663bbb2a:::benchmark_data_smithi194_121512_object3234:head# 2024-04-29T14:27:11.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663c6025:::benchmark_data_smithi194_121512_object78954:head# 2024-04-29T14:27:11.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663cc1aa:::benchmark_data_smithi194_121512_object129088:head# 2024-04-29T14:27:11.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663e09bc:::benchmark_data_smithi194_121512_object48178:head# 2024-04-29T14:27:11.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663e5b38:::benchmark_data_smithi194_121512_object48338:head# 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663e6799:::benchmark_data_smithi194_121512_object30346:head# 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663f2168:::benchmark_data_smithi194_121512_object107323:head# 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663f31b3:::benchmark_data_smithi194_121512_object119066:head# 2024-04-29T14:27:11.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663f3384:::benchmark_data_smithi194_121512_object52897:head# 2024-04-29T14:27:11.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663f6ce4:::benchmark_data_smithi194_121512_object131252:head# 2024-04-29T14:27:11.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:663f8d6f:::benchmark_data_smithi194_121512_object28325:head# 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6640271e:::benchmark_data_smithi194_121512_object117750:head# 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66405bb8:::benchmark_data_smithi194_121512_object79700:head# 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66409054:::benchmark_data_smithi194_121512_object42016:head# 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6640959b:::benchmark_data_smithi194_121512_object77877:head# 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664127f3:::benchmark_data_smithi194_121512_object113765:head# 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.669 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66419182:::benchmark_data_smithi194_121512_object5999:head# 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6641f96b:::benchmark_data_smithi194_121512_object35396:head# 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66424af3:::benchmark_data_smithi194_121512_object35443:head# 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664306b8:::benchmark_data_smithi194_121512_object54627:head# 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664590a2:::benchmark_data_smithi194_121512_object53445:head# 2024-04-29T14:27:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6646975f:::benchmark_data_smithi194_121512_object30325:head# 2024-04-29T14:27:11.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66489e9f:::benchmark_data_smithi194_121512_object118170:head# 2024-04-29T14:27:11.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664973ee:::benchmark_data_smithi194_121512_object93354:head# 2024-04-29T14:27:11.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.672 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:27:11.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6649aa1f:::benchmark_data_smithi194_121512_object19624:head# 2024-04-29T14:27:11.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664ac707:::benchmark_data_smithi194_121512_object49482:head# 2024-04-29T14:27:11.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664b5250:::benchmark_data_smithi194_121512_object33413:head# 2024-04-29T14:27:11.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664bfc25:::benchmark_data_smithi194_121512_object127412:head# 2024-04-29T14:27:11.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664c1e9f:::benchmark_data_smithi194_121512_object44140:head# 2024-04-29T14:27:11.675 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.675 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664c860c:::benchmark_data_smithi194_121512_object6823:head# 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664cbe31:::benchmark_data_smithi194_121512_object63812:head# 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664cf879:::benchmark_data_smithi194_121512_object19925:head# 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664cfc79:::benchmark_data_smithi194_121512_object132154:head# 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664d33d9:::benchmark_data_smithi194_121512_object67115:head# 2024-04-29T14:27:11.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664d5bfe:::benchmark_data_smithi194_121512_object31959:head# 2024-04-29T14:27:11.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664d7184:::benchmark_data_smithi194_121512_object5306:head# 2024-04-29T14:27:11.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664e52f4:::benchmark_data_smithi194_121512_object18406:head# 2024-04-29T14:27:11.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:664fd313:::benchmark_data_smithi194_121512_object132379:head# 2024-04-29T14:27:11.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66515ea6:::benchmark_data_smithi194_121512_object108115:head# 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66515ea9:::benchmark_data_smithi194_121512_object132763:head# 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6651b202:::benchmark_data_smithi194_121512_object107212:head# 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6651d8aa:::benchmark_data_smithi194_121512_object26823:head# 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66543d35:::benchmark_data_smithi194_121512_object28081:head# 2024-04-29T14:27:11.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6655dc1e:::benchmark_data_smithi194_121512_object50088:head# 2024-04-29T14:27:11.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66566f24:::benchmark_data_smithi194_121512_object25886:head# 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665696cd:::benchmark_data_smithi194_121512_object51683:head# 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6656a30e:::benchmark_data_smithi194_121512_object105840:head# 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66575227:::benchmark_data_smithi194_121512_object119406:head# 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66580de1:::benchmark_data_smithi194_121512_object95501:head# 2024-04-29T14:27:11.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665815c0:::benchmark_data_smithi194_121512_object29988:head# 2024-04-29T14:27:11.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:27:11.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665909e4:::benchmark_data_smithi194_121512_object26529:head# 2024-04-29T14:27:11.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66592490:::benchmark_data_smithi194_121512_object128400:head# 2024-04-29T14:27:11.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6659bb11:::benchmark_data_smithi194_121512_object69589:head# 2024-04-29T14:27:11.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665a7625:::benchmark_data_smithi194_121512_object84934:head# 2024-04-29T14:27:11.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:27:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:27:11.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665a93f7:::benchmark_data_smithi194_121512_object6752:head# 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665b1fba:::benchmark_data_smithi194_121512_object73885:head# 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665b3216:::benchmark_data_smithi194_121512_object33384:head# 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665b33c5:::benchmark_data_smithi194_121512_object27228:head# 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665bb433:::benchmark_data_smithi194_121512_object4427:head# 2024-04-29T14:27:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665bdb3f:::benchmark_data_smithi194_121512_object70035:head# 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665bfd8d:::benchmark_data_smithi194_121512_object107547:head# 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665c870b:::benchmark_data_smithi194_121512_object86148:head# 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665ce3f9:::benchmark_data_smithi194_121512_object4774:head# 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665cf903:::benchmark_data_smithi194_121512_object48138:head# 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665d2192:::benchmark_data_smithi194_121512_object63807:head# 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665d521f:::benchmark_data_smithi194_121512_object90762:head# 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665dff5c:::benchmark_data_smithi194_121512_object87772:head# 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665e1963:::benchmark_data_smithi194_121512_object124827:head# 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665e7a11:::benchmark_data_smithi194_121512_object97458:head# 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:665edc96:::benchmark_data_smithi194_121512_object108316:head# 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666148fc:::benchmark_data_smithi194_121512_object29973:head# 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66614ff4:::benchmark_data_smithi194_121512_object22799:head# 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6661799a:::benchmark_data_smithi194_121512_object13360:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6661d1bf:::benchmark_data_smithi194_121512_object29368:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666215f1:::benchmark_data_smithi194_121512_object13445:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6662b1b9:::benchmark_data_smithi194_121512_object127370:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66653a96:::benchmark_data_smithi194_121512_object14677:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66669582:::benchmark_data_smithi194_121512_object90499:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6666c064:::benchmark_data_smithi194_121512_object8341:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6666dbe5:::benchmark_data_smithi194_121512_object19356:head# 2024-04-29T14:27:11.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6666e677:::benchmark_data_smithi194_121512_object26832:head# 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66678ba4:::benchmark_data_smithi194_121512_object125052:head# 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6667a8b7:::benchmark_data_smithi194_121512_object757:head# 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6667ec19:::benchmark_data_smithi194_121512_object1688:head# 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666807e2:::benchmark_data_smithi194_121512_object271:head# 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66687db8:::benchmark_data_smithi194_121512_object18781:head# 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6668a350:::benchmark_data_smithi194_121512_object98603:head# 2024-04-29T14:27:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6668eb79:::benchmark_data_smithi194_121512_object76179:head# 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666a0e0f:::benchmark_data_smithi194_121512_object24110:head# 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666a2f9d:::benchmark_data_smithi194_121512_object48081:head# 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666a44f1:::benchmark_data_smithi194_121512_object77965:head# 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666b60ce:::benchmark_data_smithi194_121512_object31088:head# 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666b6abc:::benchmark_data_smithi194_121512_object97274:head# 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666c2d05:::benchmark_data_smithi194_121512_object101411:head# 2024-04-29T14:27:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666c3360:::benchmark_data_smithi194_121512_object46909:head# 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666c97e6:::benchmark_data_smithi194_121512_object4196:head# 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666c9b1f:::benchmark_data_smithi194_121512_object130391:head# 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666cf06d:::benchmark_data_smithi194_121512_object120500:head# 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666d2887:::benchmark_data_smithi194_121512_object60936:head# 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666d7a5e:::benchmark_data_smithi194_121512_object81710:head# 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666f41ef:::benchmark_data_smithi194_121512_object85768:head# 2024-04-29T14:27:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:666fe42a:::benchmark_data_smithi194_121512_object63064:head# 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66704db3:::benchmark_data_smithi194_121512_object50127:head# 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6670709f:::benchmark_data_smithi194_121512_object2073:head# 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66709a3c:::benchmark_data_smithi194_121512_object61069:head# 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66725813:::benchmark_data_smithi194_121512_object20448:head# 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6672c8ec:::benchmark_data_smithi194_121512_object39623:head# 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667513ab:::benchmark_data_smithi194_121512_object46790:head# 2024-04-29T14:27:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66751b2e:::benchmark_data_smithi194_121512_object108578:head# 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667596b4:::benchmark_data_smithi194_121512_object20931:head# 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6675fe44:::benchmark_data_smithi194_121512_object86259:head# 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667620d7:::benchmark_data_smithi194_121512_object263:head# 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66765798:::benchmark_data_smithi194_121512_object31006:head# 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667699e0:::benchmark_data_smithi194_121512_object62105:head# 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6677533a:::benchmark_data_smithi194_121512_object113229:head# 2024-04-29T14:27:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667759b0:::benchmark_data_smithi194_121512_object89290:head# 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667893a3:::benchmark_data_smithi194_121512_object13546:head# 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66789b1b:::benchmark_data_smithi194_121512_object77582:head# 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6678c6f8:::benchmark_data_smithi194_121512_object79104:head# 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667967ec:::benchmark_data_smithi194_121512_object31873:head# 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6679e060:::benchmark_data_smithi194_121512_object66203:head# 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667a26b5:::benchmark_data_smithi194_121512_object100460:head# 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667abec1:::benchmark_data_smithi194_121512_object38772:head# 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667b89f6:::benchmark_data_smithi194_121512_object85243:head# 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667bd463:::benchmark_data_smithi194_121512_object83729:head# 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667d2be9:::benchmark_data_smithi194_121512_object37576:head# 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667eebd0:::benchmark_data_smithi194_121512_object132017:head# 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:667ffff6:::benchmark_data_smithi194_121512_object18972:head# 2024-04-29T14:27:11.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66817ff7:::benchmark_data_smithi194_121512_object124449:head# 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6681c380:::benchmark_data_smithi194_121512_object52182:head# 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6681dac1:::benchmark_data_smithi194_121512_object114882:head# 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6681f407:::benchmark_data_smithi194_121512_object130883:head# 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6682320c:::benchmark_data_smithi194_121512_object96612:head# 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66835ef1:::benchmark_data_smithi194_121512_object118224:head# 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6683a2c3:::benchmark_data_smithi194_121512_object115256:head# 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6683f94d:::benchmark_data_smithi194_121512_object88368:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6685334f:::benchmark_data_smithi194_121512_object82808:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6685801e:::benchmark_data_smithi194_121512_object71135:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668647fe:::benchmark_data_smithi194_121512_object111865:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668697be:::benchmark_data_smithi194_121512_object1080:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6686e960:::benchmark_data_smithi194_121512_object69372:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668704d0:::benchmark_data_smithi194_121512_object80336:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668756bf:::benchmark_data_smithi194_121512_object64762:head# 2024-04-29T14:27:11.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66878d30:::benchmark_data_smithi194_121512_object14388:head# 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668818c9:::benchmark_data_smithi194_121512_object102402:head# 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66891bca:::benchmark_data_smithi194_121512_object119256:head# 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6689fded:::benchmark_data_smithi194_121512_object42268:head# 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668a3e15:::benchmark_data_smithi194_121512_object100834:head# 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668ae6f3:::benchmark_data_smithi194_121512_object52238:head# 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668aeb57:::benchmark_data_smithi194_121512_object111663:head# 2024-04-29T14:27:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668b3d9d:::benchmark_data_smithi194_121512_object55930:head# 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668b9c9a:::benchmark_data_smithi194_121512_object69025:head# 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668bca0b:::benchmark_data_smithi194_121512_object5753:head# 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668c02e4:::benchmark_data_smithi194_121512_object86941:head# 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668c6f26:::benchmark_data_smithi194_121512_object45229:head# 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668caaed:::benchmark_data_smithi194_121512_object100959:head# 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668dd92b:::benchmark_data_smithi194_121512_object42405:head# 2024-04-29T14:27:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668df4a4:::benchmark_data_smithi194_121512_object69052:head# 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668eb091:::benchmark_data_smithi194_121512_object102868:head# 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668f046f:::benchmark_data_smithi194_121512_object36753:head# 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668fdbaa:::benchmark_data_smithi194_121512_object51262:head# 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:668ff146:::benchmark_data_smithi194_121512_object69049:head# 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6690094a:::benchmark_data_smithi194_121512_object108563:head# 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66905a91:::benchmark_data_smithi194_121512_object3342:head# 2024-04-29T14:27:11.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6690975e:::benchmark_data_smithi194_121512_object57568:head# 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66927c70:::benchmark_data_smithi194_121512_object23307:head# 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66937411:::benchmark_data_smithi194_121512_object116729:head# 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6693d544:::benchmark_data_smithi194_121512_object21832:head# 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66942503:::benchmark_data_smithi194_121512_object21279:head# 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66955616:::benchmark_data_smithi194_121512_object43800:head# 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6695ac0a:::benchmark_data_smithi194_121512_object63453:head# 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6696146f:::benchmark_data_smithi194_121512_object112157:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669632c3:::benchmark_data_smithi194_121512_object38198:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6696734a:::benchmark_data_smithi194_121512_object119028:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6696d93b:::benchmark_data_smithi194_121512_object37203:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66970887:::benchmark_data_smithi194_121512_object20413:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6697c850:::benchmark_data_smithi194_121512_object79774:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6698c2da:::benchmark_data_smithi194_121512_object97001:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669976d8:::benchmark_data_smithi194_121512_object121269:head# 2024-04-29T14:27:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669afacd:::benchmark_data_smithi194_121512_object49217:head# 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669b03e8:::benchmark_data_smithi194_121512_object15626:head# 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669b5e66:::benchmark_data_smithi194_121512_object63826:head# 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669c2e13:::benchmark_data_smithi194_121512_object3356:head# 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669c7372:::benchmark_data_smithi194_121512_object119078:head# 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669ce14d:::benchmark_data_smithi194_121512_object47986:head# 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669dcde2:::benchmark_data_smithi194_121512_object100739:head# 2024-04-29T14:27:11.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669e0b2c:::benchmark_data_smithi194_121512_object10338:head# 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669f1553:::benchmark_data_smithi194_121512_object83181:head# 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669fb6b4:::benchmark_data_smithi194_121512_object63329:head# 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:669fffdb:::benchmark_data_smithi194_121512_object6358:head# 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a01c77:::benchmark_data_smithi194_121512_object68146:head# 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a0b3b7:::benchmark_data_smithi194_121512_object52823:head# 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a1b5e8:::benchmark_data_smithi194_121512_object118708:head# 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a1c282:::benchmark_data_smithi194_121512_object41758:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a1ee68:::benchmark_data_smithi194_121512_object84557:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a505cd:::benchmark_data_smithi194_121512_object126193:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a604f8:::benchmark_data_smithi194_121512_object117933:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a68cbd:::benchmark_data_smithi194_121512_object60390:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a6e80a:::benchmark_data_smithi194_121512_object25446:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a79980:::benchmark_data_smithi194_121512_object41573:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a7cc9e:::benchmark_data_smithi194_121512_object83254:head# 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a7ed47:::benchmark_data_smithi194_121512_object49564:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a8402a:::benchmark_data_smithi194_121512_object64792:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a8efea:::benchmark_data_smithi194_121512_object82264:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66a8f720:::benchmark_data_smithi194_121512_object6517:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ab74e2:::benchmark_data_smithi194_121512_object121953:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ab75ec:::benchmark_data_smithi194_121512_object83368:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66abc9c2:::benchmark_data_smithi194_121512_object118032:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ac5c1b:::benchmark_data_smithi194_121512_object20134:head# 2024-04-29T14:27:11.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ac9d75:::benchmark_data_smithi194_121512_object25394:head# 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ace734:::benchmark_data_smithi194_121512_object47857:head# 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ad1b27:::benchmark_data_smithi194_121512_object38257:head# 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66adbd8b:::benchmark_data_smithi194_121512_object3254:head# 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ade7b6:::benchmark_data_smithi194_121512_object18437:head# 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ae21d5:::benchmark_data_smithi194_121512_object108331:head# 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ae3972:::benchmark_data_smithi194_121512_object107488:head# 2024-04-29T14:27:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ae5aa4:::benchmark_data_smithi194_121512_object98729:head# 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66af1805:::benchmark_data_smithi194_121512_object22332:head# 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66af1c5e:::benchmark_data_smithi194_121512_object8226:head# 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66afcd35:::benchmark_data_smithi194_121512_object30713:head# 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66afd7dd:::benchmark_data_smithi194_121512_object79735:head# 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b0d528:::benchmark_data_smithi194_121512_object89922:head# 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b18dc4:::benchmark_data_smithi194_121512_object97271:head# 2024-04-29T14:27:11.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b27602:::benchmark_data_smithi194_121512_object17486:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b2c0a7:::benchmark_data_smithi194_121512_object32456:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b2e653:::benchmark_data_smithi194_121512_object13517:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b32e88:::benchmark_data_smithi194_121512_object58778:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b33419:::benchmark_data_smithi194_121512_object48422:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b35901:::benchmark_data_smithi194_121512_object64868:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b421fc:::benchmark_data_smithi194_121512_object120789:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b4596a:::benchmark_data_smithi194_121512_object33942:head# 2024-04-29T14:27:11.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b53299:::benchmark_data_smithi194_121512_object66401:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b569ef:::benchmark_data_smithi194_121512_object1486:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b58500:::benchmark_data_smithi194_121512_object19523:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b6c009:::benchmark_data_smithi194_121512_object48936:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b70657:::benchmark_data_smithi194_121512_object8102:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b7100b:::benchmark_data_smithi194_121512_object100689:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b72a60:::benchmark_data_smithi194_121512_object111174:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b72b6b:::benchmark_data_smithi194_121512_object49975:head# 2024-04-29T14:27:11.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b7bdd1:::benchmark_data_smithi194_121512_object77812:head# 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b7e413:::benchmark_data_smithi194_121512_object79964:head# 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b81548:::benchmark_data_smithi194_121512_object53356:head# 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b841f1:::benchmark_data_smithi194_121512_object24490:head# 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b8c1fa:::benchmark_data_smithi194_121512_object88679:head# 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b8c688:::benchmark_data_smithi194_121512_object125810:head# 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b95fa0:::benchmark_data_smithi194_121512_object8582:head# 2024-04-29T14:27:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66b97193:::benchmark_data_smithi194_121512_object101787:head# 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bb9936:::benchmark_data_smithi194_121512_object65244:head# 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bcc2b6:::benchmark_data_smithi194_121512_object14138:head# 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bcfbb9:::benchmark_data_smithi194_121512_object13447:head# 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bd6b07:::benchmark_data_smithi194_121512_object74428:head# 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bde04d:::benchmark_data_smithi194_121512_object107010:head# 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66be5201:::benchmark_data_smithi194_121512_object128872:head# 2024-04-29T14:27:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bec755:::benchmark_data_smithi194_121512_object34356:head# 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bf1470:::benchmark_data_smithi194_121512_object56666:head# 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bf45f2:::benchmark_data_smithi194_121512_object19086:head# 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66bf8fe5:::benchmark_data_smithi194_121512_object65348:head# 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c218da:::benchmark_data_smithi194_121512_object129494:head# 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c256a7:::benchmark_data_smithi194_121512_object107710:head# 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c2c822:::benchmark_data_smithi194_121512_object97466:head# 2024-04-29T14:27:11.808 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c3c1e5:::benchmark_data_smithi194_121512_object131562:head# 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c40fce:::benchmark_data_smithi194_121512_object21170:head# 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c50a61:::benchmark_data_smithi194_121512_object95818:head# 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c5690d:::benchmark_data_smithi194_121512_object2686:head# 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c58568:::benchmark_data_smithi194_121512_object99367:head# 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c60756:::benchmark_data_smithi194_121512_object104520:head# 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c6250a:::benchmark_data_smithi194_121512_object113898:head# 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c7b6f6:::benchmark_data_smithi194_121512_object26365:head# 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c7c3d8:::benchmark_data_smithi194_121512_object20409:head# 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c8bf95:::benchmark_data_smithi194_121512_object26450:head# 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c9a068:::benchmark_data_smithi194_121512_object103996:head# 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66c9ea94:::benchmark_data_smithi194_121512_object30953:head# 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cab4bb:::benchmark_data_smithi194_121512_object36885:head# 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cacafa:::benchmark_data_smithi194_121512_object124346:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cb4201:::benchmark_data_smithi194_121512_object121678:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cc0850:::benchmark_data_smithi194_121512_object66438:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cc6b80:::benchmark_data_smithi194_121512_object132584:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cc8d30:::benchmark_data_smithi194_121512_object13098:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ccfdcf:::benchmark_data_smithi194_121512_object93340:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cd8dc6:::benchmark_data_smithi194_121512_object128697:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ce7f01:::benchmark_data_smithi194_121512_object111091:head# 2024-04-29T14:27:11.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cf08a7:::benchmark_data_smithi194_121512_object121289:head# 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cf3620:::benchmark_data_smithi194_121512_object125946:head# 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cf7861:::benchmark_data_smithi194_121512_object16913:head# 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cf95c9:::benchmark_data_smithi194_121512_object117027:head# 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66cf9b95:::benchmark_data_smithi194_121512_object99641:head# 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d063d5:::benchmark_data_smithi194_121512_object65820:head# 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d0c434:::benchmark_data_smithi194_121512_object25343:head# 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d1a736:::benchmark_data_smithi194_121512_object127546:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d1f08c:::benchmark_data_smithi194_121512_object72603:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d3684b:::benchmark_data_smithi194_121512_object86697:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d3bb91:::benchmark_data_smithi194_121512_object122671:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d623f5:::benchmark_data_smithi194_121512_object77722:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d7bee5:::benchmark_data_smithi194_121512_object1061:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d87d48:::benchmark_data_smithi194_121512_object31980:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d8d827:::benchmark_data_smithi194_121512_object18432:head# 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d8efd9:::benchmark_data_smithi194_121512_object17830:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d8fa04:::benchmark_data_smithi194_121512_object57307:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66d9f851:::benchmark_data_smithi194_121512_object98232:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66da0c86:::benchmark_data_smithi194_121512_object31093:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66da18a3:::benchmark_data_smithi194_121512_object44941:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66da3708:::benchmark_data_smithi194_121512_object24279:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66da72e8:::benchmark_data_smithi194_121512_object19062:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66daad6b:::benchmark_data_smithi194_121512_object79242:head# 2024-04-29T14:27:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66dad7cc:::benchmark_data_smithi194_121512_object97078:head# 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66dae16b:::benchmark_data_smithi194_121512_object39465:head# 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66db27e0:::benchmark_data_smithi194_121512_object47946:head# 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66db45f8:::benchmark_data_smithi194_121512_object35312:head# 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66db9925:::benchmark_data_smithi194_121512_object75747:head# 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66dc7ee7:::benchmark_data_smithi194_121512_object74039:head# 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66dce764:::benchmark_data_smithi194_121512_object59507:head# 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66dd233b:::benchmark_data_smithi194_121512_object23511:head# 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ddd0aa:::benchmark_data_smithi194_121512_object123619:head# 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ddd4ef:::benchmark_data_smithi194_121512_object21691:head# 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ddec82:::benchmark_data_smithi194_121512_object95744:head# 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66de0861:::benchmark_data_smithi194_121512_object124143:head# 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66df2790:::benchmark_data_smithi194_121512_object40990:head# 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66dfadbf:::benchmark_data_smithi194_121512_object77751:head# 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e00189:::benchmark_data_smithi194_121512_object59118:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e00f01:::benchmark_data_smithi194_121512_object102777:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e02d45:::benchmark_data_smithi194_121512_object70356:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e05613:::benchmark_data_smithi194_121512_object45447:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e095c0:::benchmark_data_smithi194_121512_object31777:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e10601:::benchmark_data_smithi194_121512_object82614:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e16b64:::benchmark_data_smithi194_121512_object89157:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e1deb8:::benchmark_data_smithi194_121512_object25177:head# 2024-04-29T14:27:11.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e34a21:::benchmark_data_smithi194_121512_object63259:head# 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e35f1d:::benchmark_data_smithi194_121512_object120663:head# 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e6356b:::benchmark_data_smithi194_121512_object115717:head# 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e64990:::benchmark_data_smithi194_121512_object23860:head# 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66e8f0ce:::benchmark_data_smithi194_121512_object23980:head# 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ea3c80:::benchmark_data_smithi194_121512_object131821:head# 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66eb5cb0:::benchmark_data_smithi194_121512_object93001:head# 2024-04-29T14:27:11.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66eb86c1:::benchmark_data_smithi194_121512_object124748:head# 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66eba125:::benchmark_data_smithi194_121512_object25143:head# 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ec0b7e:::benchmark_data_smithi194_121512_object80258:head# 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ed12a9:::benchmark_data_smithi194_121512_object28771:head# 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66ed48a3:::benchmark_data_smithi194_121512_object101217:head# 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66eddb52:::benchmark_data_smithi194_121512_object61119:head# 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66eea894:::benchmark_data_smithi194_121512_object106180:head# 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66eec5ab:::benchmark_data_smithi194_121512_object34791:head# 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66eefd2c:::benchmark_data_smithi194_121512_object76503:head# 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66efcb12:::benchmark_data_smithi194_121512_object81961:head# 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f0b47b:::benchmark_data_smithi194_121512_object69387:head# 2024-04-29T14:27:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f1ab3e:::benchmark_data_smithi194_121512_object61917:head# 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f2001d:::benchmark_data_smithi194_121512_object120435:head# 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f2de0d:::benchmark_data_smithi194_121512_object2336:head# 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f2eda1:::benchmark_data_smithi194_121512_object60087:head# 2024-04-29T14:27:11.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f354f5:::benchmark_data_smithi194_121512_object94983:head# 2024-04-29T14:27:11.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f3a6a0:::benchmark_data_smithi194_121512_object63215:head# 2024-04-29T14:27:11.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f3a85c:::benchmark_data_smithi194_121512_object114258:head# 2024-04-29T14:27:11.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f3b1c1:::benchmark_data_smithi194_121512_object84248:head# 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f43a07:::benchmark_data_smithi194_121512_object108613:head# 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f4c572:::benchmark_data_smithi194_121512_object12991:head# 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f53678:::benchmark_data_smithi194_121512_object42973:head# 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f5a71b:::benchmark_data_smithi194_121512_object72727:head# 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f5fe66:::benchmark_data_smithi194_121512_object120840:head# 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f62b91:::benchmark_data_smithi194_121512_object126524:head# 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f64ee0:::benchmark_data_smithi194_121512_object59587:head# 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f7015a:::benchmark_data_smithi194_121512_object78314:head# 2024-04-29T14:27:11.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f7410d:::benchmark_data_smithi194_121512_object80371:head# 2024-04-29T14:27:11.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f7cc82:::benchmark_data_smithi194_121512_object7505:head# 2024-04-29T14:27:11.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f97ce4:::benchmark_data_smithi194_121512_object41716:head# 2024-04-29T14:27:11.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66f9f40b:::benchmark_data_smithi194_121512_object63922:head# 2024-04-29T14:27:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fb1945:::benchmark_data_smithi194_121512_object124232:head# 2024-04-29T14:27:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fbaf4d:::benchmark_data_smithi194_121512_object82737:head# 2024-04-29T14:27:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fbc0ef:::benchmark_data_smithi194_121512_object129257:head# 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fc2216:::benchmark_data_smithi194_121512_object101688:head# 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fc5902:::benchmark_data_smithi194_121512_object93217:head# 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fc7900:::benchmark_data_smithi194_121512_object116327:head# 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fc8049:::benchmark_data_smithi194_121512_object92660:head# 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fd06db:::benchmark_data_smithi194_121512_object105260:head# 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fd71ee:::benchmark_data_smithi194_121512_object66130:head# 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fd7fc2:::benchmark_data_smithi194_121512_object16063:head# 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fe35df:::benchmark_data_smithi194_121512_object98950:head# 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:66fe69cf:::benchmark_data_smithi194_121512_object48347:head# 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670054dd:::benchmark_data_smithi194_121512_object10624:head# 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67006631:::benchmark_data_smithi194_121512_object101289:head# 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6700cc70:::benchmark_data_smithi194_121512_object108808:head# 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67021b0e:::benchmark_data_smithi194_121512_object103814:head# 2024-04-29T14:27:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67022e74:::benchmark_data_smithi194_121512_object76206:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6702b2c3:::benchmark_data_smithi194_121512_object92716:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6702e60d:::benchmark_data_smithi194_121512_object16822:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67048856:::benchmark_data_smithi194_121512_object36721:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67054fc6:::benchmark_data_smithi194_121512_object36576:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6705ecc1:::benchmark_data_smithi194_121512_object33459:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670611d2:::benchmark_data_smithi194_121512_object115128:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6706f05f:::benchmark_data_smithi194_121512_object82085:head# 2024-04-29T14:27:11.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67071b8c:::benchmark_data_smithi194_121512_object96128:head# 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670749b4:::benchmark_data_smithi194_121512_object70302:head# 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67076b12:::benchmark_data_smithi194_121512_object69557:head# 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6707b686:::benchmark_data_smithi194_121512_object6110:head# 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67080f35:::benchmark_data_smithi194_121512_object90315:head# 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67082e61:::benchmark_data_smithi194_121512_object14108:head# 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670844f5:::benchmark_data_smithi194_121512_object89992:head# 2024-04-29T14:27:11.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67088bb3:::benchmark_data_smithi194_121512_object58258:head# 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6708d240:::benchmark_data_smithi194_121512_object29700:head# 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670a3722:::benchmark_data_smithi194_121512_object23002:head# 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670a5944:::benchmark_data_smithi194_121512_object131165:head# 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670ab510:::benchmark_data_smithi194_121512_object81247:head# 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670b247a:::benchmark_data_smithi194_121512_object111490:head# 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670b535a:::benchmark_data_smithi194_121512_object112739:head# 2024-04-29T14:27:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670b5907:::benchmark_data_smithi194_121512_object68561:head# 2024-04-29T14:27:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670b9852:::benchmark_data_smithi194_121512_object91947:head# 2024-04-29T14:27:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670c14f6:::benchmark_data_smithi194_121512_object91517:head# 2024-04-29T14:27:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670c845e:::benchmark_data_smithi194_121512_object84745:head# 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670d3582:::benchmark_data_smithi194_121512_object112188:head# 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670d9552:::benchmark_data_smithi194_121512_object2084:head# 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670dd819:::benchmark_data_smithi194_121512_object91176:head# 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670f2731:::benchmark_data_smithi194_121512_object17972:head# 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:670f7121:::benchmark_data_smithi194_121512_object22067:head# 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67102393:::benchmark_data_smithi194_121512_object109934:head# 2024-04-29T14:27:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67122f95:::benchmark_data_smithi194_121512_object71805:head# 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67125c89:::benchmark_data_smithi194_121512_object120149:head# 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671265c0:::benchmark_data_smithi194_121512_object128987:head# 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6712703f:::benchmark_data_smithi194_121512_object10989:head# 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67149c09:::benchmark_data_smithi194_121512_object96398:head# 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6714ffdb:::benchmark_data_smithi194_121512_object72033:head# 2024-04-29T14:27:11.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.842 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67154bf8:::benchmark_data_smithi194_121512_object124966:head# 2024-04-29T14:27:11.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67158acd:::benchmark_data_smithi194_121512_object97631:head# 2024-04-29T14:27:11.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67158c15:::benchmark_data_smithi194_121512_object8746:head# 2024-04-29T14:27:11.843 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6715ba81:::benchmark_data_smithi194_121512_object6896:head# 2024-04-29T14:27:11.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67162b44:::benchmark_data_smithi194_121512_object82428:head# 2024-04-29T14:27:11.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6716f775:::benchmark_data_smithi194_121512_object13085:head# 2024-04-29T14:27:11.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6717386b:::benchmark_data_smithi194_121512_object86492:head# 2024-04-29T14:27:11.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6717f483:::benchmark_data_smithi194_121512_object18350:head# 2024-04-29T14:27:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67186b0e:::benchmark_data_smithi194_121512_object23062:head# 2024-04-29T14:27:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67187aa7:::benchmark_data_smithi194_121512_object77459:head# 2024-04-29T14:27:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6719c1f7:::benchmark_data_smithi194_121512_object96356:head# 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6719fc2e:::benchmark_data_smithi194_121512_object93106:head# 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671aaa03:::benchmark_data_smithi194_121512_object18758:head# 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671b8a64:::benchmark_data_smithi194_121512_object86424:head# 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671be89e:::benchmark_data_smithi194_121512_object17060:head# 2024-04-29T14:27:11.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671c5584:::benchmark_data_smithi194_121512_object107115:head# 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671cd58b:::benchmark_data_smithi194_121512_object55114:head# 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671d424e:::benchmark_data_smithi194_121512_object66950:head# 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671d8fe5:::benchmark_data_smithi194_121512_object91150:head# 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671de008:::benchmark_data_smithi194_121512_object132723:head# 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671e02d7:::benchmark_data_smithi194_121512_object70667:head# 2024-04-29T14:27:11.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671e092e:::benchmark_data_smithi194_121512_object128233:head# 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671e0a0c:::benchmark_data_smithi194_121512_object77570:head# 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671e8f8e:::benchmark_data_smithi194_121512_object14986:head# 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671ea205:::benchmark_data_smithi194_121512_object91564:head# 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671edbf8:::benchmark_data_smithi194_121512_object86812:head# 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671f7252:::benchmark_data_smithi194_121512_object29901:head# 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:671f7a1e:::benchmark_data_smithi194_121512_object80268:head# 2024-04-29T14:27:11.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67207a9a:::benchmark_data_smithi194_121512_object15450:head# 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6720ca6e:::benchmark_data_smithi194_121512_object89106:head# 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67212ebf:::benchmark_data_smithi194_121512_object54872:head# 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6721fc58:::benchmark_data_smithi194_121512_object21979:head# 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672240e6:::benchmark_data_smithi194_121512_object51868:head# 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67225e34:::benchmark_data_smithi194_121512_object68511:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67227261:::benchmark_data_smithi194_121512_object56529:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6722829d:::benchmark_data_smithi194_121512_object26550:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6722d929:::benchmark_data_smithi194_121512_object132048:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67232ed7:::benchmark_data_smithi194_121512_object74955:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6723deb0:::benchmark_data_smithi194_121512_object13626:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6723fa64:::benchmark_data_smithi194_121512_object10203:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67248295:::benchmark_data_smithi194_121512_object7112:head# 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672501d0:::benchmark_data_smithi194_121512_object101909:head# 2024-04-29T14:27:11.856 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.857 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6725e9a3:::benchmark_data_smithi194_121512_object10441:head# 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6726dd8c:::benchmark_data_smithi194_121512_object11328:head# 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67270cf5:::benchmark_data_smithi194_121512_object115679:head# 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6727570d:::benchmark_data_smithi194_121512_object90432:head# 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6727a589:::benchmark_data_smithi194_121512_object79098:head# 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67280d33:::benchmark_data_smithi194_121512_object97284:head# 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67282517:::benchmark_data_smithi194_121512_object39820:head# 2024-04-29T14:27:11.858 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67289ce9:::benchmark_data_smithi194_121512_object96736:head# 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6729059e:::benchmark_data_smithi194_121512_object57522:head# 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672a535d:::benchmark_data_smithi194_121512_object114272:head# 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672a8608:::benchmark_data_smithi194_121512_object57503:head# 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672b6fc9:::benchmark_data_smithi194_121512_object63880:head# 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672b8fb3:::benchmark_data_smithi194_121512_object16206:head# 2024-04-29T14:27:11.859 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.862 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672b9e28:::benchmark_data_smithi194_121512_object129550:head# 2024-04-29T14:27:11.862 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672ba4cb:::benchmark_data_smithi194_121512_object19988:head# 2024-04-29T14:27:11.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672bcefc:::benchmark_data_smithi194_121512_object38888:head# 2024-04-29T14:27:11.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672bf550:::benchmark_data_smithi194_121512_object54034:head# 2024-04-29T14:27:11.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672c6e45:::benchmark_data_smithi194_121512_object114299:head# 2024-04-29T14:27:11.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.864 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672c7852:::benchmark_data_smithi194_121512_object111254:head# 2024-04-29T14:27:11.864 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672c7bb2:::benchmark_data_smithi194_121512_object126385:head# 2024-04-29T14:27:11.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672d331c:::benchmark_data_smithi194_121512_object115643:head# 2024-04-29T14:27:11.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.865 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672d6475:::benchmark_data_smithi194_121512_object120223:head# 2024-04-29T14:27:11.865 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672e0e4f:::benchmark_data_smithi194_121512_object119544:head# 2024-04-29T14:27:11.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672e7852:::benchmark_data_smithi194_121512_object110350:head# 2024-04-29T14:27:11.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672ea111:::benchmark_data_smithi194_121512_object107813:head# 2024-04-29T14:27:11.867 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.867 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672efe7d:::benchmark_data_smithi194_121512_object106752:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:672f578b:::benchmark_data_smithi194_121512_object22220:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673041f8:::benchmark_data_smithi194_121512_object19843:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67308460:::benchmark_data_smithi194_121512_object79921:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6730d099:::benchmark_data_smithi194_121512_object121993:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6730e14a:::benchmark_data_smithi194_121512_object73693:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67332cef:::benchmark_data_smithi194_121512_object50143:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67335774:::benchmark_data_smithi194_121512_object111855:head# 2024-04-29T14:27:11.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.869 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67336673:::benchmark_data_smithi194_121512_object2376:head# 2024-04-29T14:27:11.869 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67338da6:::benchmark_data_smithi194_121512_object65144:head# 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67349c00:::benchmark_data_smithi194_121512_object128999:head# 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6734e6b5:::benchmark_data_smithi194_121512_object58354:head# 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673562ee:::benchmark_data_smithi194_121512_object30367:head# 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6736b731:::benchmark_data_smithi194_121512_object132697:head# 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6736bb15:::benchmark_data_smithi194_121512_object464:head# 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673732f5:::benchmark_data_smithi194_121512_object62616:head# 2024-04-29T14:27:11.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6737ff52:::benchmark_data_smithi194_121512_object54265:head# 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67383335:::benchmark_data_smithi194_121512_object56738:head# 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67390156:::benchmark_data_smithi194_121512_object46193:head# 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67391768:::benchmark_data_smithi194_121512_object33001:head# 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6739551c:::benchmark_data_smithi194_121512_object55374:head# 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67399449:::benchmark_data_smithi194_121512_object4417:head# 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6739dff2:::benchmark_data_smithi194_121512_object65327:head# 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673a0143:::benchmark_data_smithi194_121512_object19421:head# 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673a212a:::benchmark_data_smithi194_121512_object18641:head# 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673a539c:::benchmark_data_smithi194_121512_object92364:head# 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673a70e1:::benchmark_data_smithi194_121512_object35727:head# 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673ac7ef:::benchmark_data_smithi194_121512_object95386:head# 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673b3d5d:::benchmark_data_smithi194_121512_object26665:head# 2024-04-29T14:27:11.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673c0766:::benchmark_data_smithi194_121512_object79415:head# 2024-04-29T14:27:11.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673c9dc6:::benchmark_data_smithi194_121512_object102187:head# 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673d0bbb:::benchmark_data_smithi194_121512_object55733:head# 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673e65dd:::benchmark_data_smithi194_121512_object17078:head# 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:673e79c6:::benchmark_data_smithi194_121512_object32065:head# 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67406785:::benchmark_data_smithi194_121512_object131393:head# 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6740a9aa:::benchmark_data_smithi194_121512_object31509:head# 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6740ee9b:::benchmark_data_smithi194_121512_object68121:head# 2024-04-29T14:27:11.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6742e9e8:::benchmark_data_smithi194_121512_object73818:head# 2024-04-29T14:27:11.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67432774:::benchmark_data_smithi194_121512_object130179:head# 2024-04-29T14:27:11.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67450038:::benchmark_data_smithi194_121512_object45482:head# 2024-04-29T14:27:11.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6745e949:::benchmark_data_smithi194_121512_object83792:head# 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67461310:::benchmark_data_smithi194_121512_object41639:head# 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67461666:::benchmark_data_smithi194_121512_object47147:head# 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6747b99b:::benchmark_data_smithi194_121512_object98846:head# 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6749de7f:::benchmark_data_smithi194_121512_object71966:head# 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674a0da0:::benchmark_data_smithi194_121512_object52484:head# 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674acad1:::benchmark_data_smithi194_121512_object17778:head# 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674b02af:::benchmark_data_smithi194_121512_object130516:head# 2024-04-29T14:27:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674b0e65:::benchmark_data_smithi194_121512_object27738:head# 2024-04-29T14:27:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674b0f7d:::benchmark_data_smithi194_121512_object26293:head# 2024-04-29T14:27:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674b4355:::benchmark_data_smithi194_121512_object114902:head# 2024-04-29T14:27:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674b678e:::benchmark_data_smithi194_121512_object90577:head# 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674b855c:::benchmark_data_smithi194_121512_object32765:head# 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674c2cde:::benchmark_data_smithi194_121512_object22659:head# 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674c5781:::benchmark_data_smithi194_121512_object121727:head# 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674c84a2:::benchmark_data_smithi194_121512_object100453:head# 2024-04-29T14:27:11.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674d47c3:::benchmark_data_smithi194_121512_object17252:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674da98e:::benchmark_data_smithi194_121512_object61633:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674dabfd:::benchmark_data_smithi194_121512_object115886:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674dd1af:::benchmark_data_smithi194_121512_object75883:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674e8432:::benchmark_data_smithi194_121512_object24611:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674e8c8a:::benchmark_data_smithi194_121512_object86012:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674f8690:::benchmark_data_smithi194_121512_object107625:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:674fbf15:::benchmark_data_smithi194_121512_object85740:head# 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67505742:::benchmark_data_smithi194_121512_object20545:head# 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6750b203:::benchmark_data_smithi194_121512_object114369:head# 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6752273b:::benchmark_data_smithi194_121512_object13690:head# 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675247ae:::benchmark_data_smithi194_121512_object107328:head# 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675261c4:::benchmark_data_smithi194_121512_object51714:head# 2024-04-29T14:27:11.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675339f0:::benchmark_data_smithi194_121512_object20235:head# 2024-04-29T14:27:11.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67536055:::benchmark_data_smithi194_121512_object100576:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675378e0:::benchmark_data_smithi194_121512_object83837:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6753c3f8:::benchmark_data_smithi194_121512_object16269:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67548994:::benchmark_data_smithi194_121512_object88808:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675663e6:::benchmark_data_smithi194_121512_object71470:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6756c547:::benchmark_data_smithi194_121512_object96932:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675810d9:::benchmark_data_smithi194_121512_object123443:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675853c4:::benchmark_data_smithi194_121512_object46179:head# 2024-04-29T14:27:11.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67588752:::benchmark_data_smithi194_121512_object124671:head# 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67590e33:::benchmark_data_smithi194_121512_object75586:head# 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675955a8:::benchmark_data_smithi194_121512_object74555:head# 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6759929f:::benchmark_data_smithi194_121512_object48006:head# 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675abe19:::benchmark_data_smithi194_121512_object78503:head# 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675b4564:::benchmark_data_smithi194_121512_object126333:head# 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675bb8a8:::benchmark_data_smithi194_121512_object130828:head# 2024-04-29T14:27:11.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675bfddf:::benchmark_data_smithi194_121512_object129546:head# 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675c2486:::benchmark_data_smithi194_121512_object59765:head# 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675c6199:::benchmark_data_smithi194_121512_object110971:head# 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675dd97c:::benchmark_data_smithi194_121512_object127201:head# 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675df9f8:::benchmark_data_smithi194_121512_object111653:head# 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675e51c5:::benchmark_data_smithi194_121512_object60039:head# 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675e9b9a:::benchmark_data_smithi194_121512_object53949:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675ec270:::benchmark_data_smithi194_121512_object76130:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675eff1b:::benchmark_data_smithi194_121512_object56305:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:675f7dce:::benchmark_data_smithi194_121512_object38552:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6760f2ec:::benchmark_data_smithi194_121512_object18752:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67620862:::benchmark_data_smithi194_121512_object65856:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67621614:::benchmark_data_smithi194_121512_object130260:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67622aeb:::benchmark_data_smithi194_121512_object623:head# 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676267e5:::benchmark_data_smithi194_121512_object111486:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6762b62c:::benchmark_data_smithi194_121512_object45125:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6762bcd9:::benchmark_data_smithi194_121512_object119098:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67633c6a:::benchmark_data_smithi194_121512_object50321:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6764a8e2:::benchmark_data_smithi194_121512_object98937:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6764c118:::benchmark_data_smithi194_121512_object105639:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6765d039:::benchmark_data_smithi194_121512_object59956:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67669818:::benchmark_data_smithi194_121512_object71438:head# 2024-04-29T14:27:11.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67695d54:::benchmark_data_smithi194_121512_object3543:head# 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676a5df1:::benchmark_data_smithi194_121512_object61190:head# 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676a9168:::benchmark_data_smithi194_121512_object45626:head# 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676a9bde:::benchmark_data_smithi194_121512_object42834:head# 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676acb26:::benchmark_data_smithi194_121512_object40609:head# 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676acf0d:::benchmark_data_smithi194_121512_object50565:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676b2d7b:::benchmark_data_smithi194_121512_object109709:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676c5135:::benchmark_data_smithi194_121512_object2511:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676c6d6d:::benchmark_data_smithi194_121512_object117806:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676c8712:::benchmark_data_smithi194_121512_object119101:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676e51c5:::benchmark_data_smithi194_121512_object130673:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676ef593:::benchmark_data_smithi194_121512_object7697:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676f163c:::benchmark_data_smithi194_121512_object5500:head# 2024-04-29T14:27:11.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:676ff616:::benchmark_data_smithi194_121512_object118673:head# 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67702f88:::benchmark_data_smithi194_121512_object53809:head# 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6770a249:::benchmark_data_smithi194_121512_object133053:head# 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6770c7fc:::benchmark_data_smithi194_121512_object47454:head# 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6771624b:::benchmark_data_smithi194_121512_object14739:head# 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677164b0:::benchmark_data_smithi194_121512_object64791:head# 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67727f4a:::benchmark_data_smithi194_121512_object20307:head# 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.908 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6772d58f:::benchmark_data_smithi194_121512_object9827:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6773b8bd:::benchmark_data_smithi194_121512_object33633:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6773fd4c:::benchmark_data_smithi194_121512_object67650:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67750266:::benchmark_data_smithi194_121512_object111993:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67755552:::benchmark_data_smithi194_121512_object129569:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6775fefe:::benchmark_data_smithi194_121512_object101181:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677684ce:::benchmark_data_smithi194_121512_object65970:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677703f6:::benchmark_data_smithi194_121512_object73950:head# 2024-04-29T14:27:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67774ada:::benchmark_data_smithi194_121512_object1047:head# 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6777b422:::benchmark_data_smithi194_121512_object40549:head# 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6778e892:::benchmark_data_smithi194_121512_object86836:head# 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677a6764:::benchmark_data_smithi194_121512_object99773:head# 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677b5031:::benchmark_data_smithi194_121512_object24121:head# 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677bd119:::benchmark_data_smithi194_121512_object54440:head# 2024-04-29T14:27:11.910 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677d1f24:::benchmark_data_smithi194_121512_object65125:head# 2024-04-29T14:27:11.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677db6a6:::benchmark_data_smithi194_121512_object114162:head# 2024-04-29T14:27:11.914 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.914 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677dc407:::benchmark_data_smithi194_121512_object84805:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677eaf76:::benchmark_data_smithi194_121512_object85634:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677ecdb3:::benchmark_data_smithi194_121512_object22102:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:677efb14:::benchmark_data_smithi194_121512_object107878:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67800590:::benchmark_data_smithi194_121512_object65989:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6780e3de:::benchmark_data_smithi194_121512_object116862:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678103d7:::benchmark_data_smithi194_121512_object117289:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67814f96:::benchmark_data_smithi194_121512_object26949:head# 2024-04-29T14:27:11.915 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6781a6a8:::benchmark_data_smithi194_121512_object38293:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67823cf5:::benchmark_data_smithi194_121512_object113108:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678266ec:::benchmark_data_smithi194_121512_object72542:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6782a20f:::benchmark_data_smithi194_121512_object15733:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6782b1b3:::benchmark_data_smithi194_121512_object104825:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6782d5bb:::benchmark_data_smithi194_121512_object15270:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67834d17:::benchmark_data_smithi194_121512_object43823:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6783a780:::benchmark_data_smithi194_121512_object58137:head# 2024-04-29T14:27:11.916 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6783a78e:::benchmark_data_smithi194_121512_object85847:head# 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6783da59:::benchmark_data_smithi194_121512_object78292:head# 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6784547d:::benchmark_data_smithi194_121512_object133177:head# 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6784b5cd:::benchmark_data_smithi194_121512_object77862:head# 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678563ce:::benchmark_data_smithi194_121512_object48518:head# 2024-04-29T14:27:11.917 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6785ca50:::benchmark_data_smithi194_121512_object94364:head# 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6785f64e:::benchmark_data_smithi194_121512_object89127:head# 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678617f7:::benchmark_data_smithi194_121512_object22890:head# 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678633a3:::benchmark_data_smithi194_121512_object118089:head# 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67864d7a:::benchmark_data_smithi194_121512_object3407:head# 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.918 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6787348e:::benchmark_data_smithi194_121512_object106056:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6787f19c:::benchmark_data_smithi194_121512_object96619:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6787f998:::benchmark_data_smithi194_121512_object18555:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6788437a:::benchmark_data_smithi194_121512_object118574:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6788991e:::benchmark_data_smithi194_121512_object110259:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6789c7b8:::benchmark_data_smithi194_121512_object86395:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678a96af:::benchmark_data_smithi194_121512_object97294:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678aa99b:::benchmark_data_smithi194_121512_object77501:head# 2024-04-29T14:27:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678abe00:::benchmark_data_smithi194_121512_object57089:head# 2024-04-29T14:27:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678bbe97:::benchmark_data_smithi194_121512_object131004:head# 2024-04-29T14:27:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678be556:::benchmark_data_smithi194_121512_object68161:head# 2024-04-29T14:27:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.925 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678d7b22:::benchmark_data_smithi194_121512_object120446:head# 2024-04-29T14:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678dea63:::benchmark_data_smithi194_121512_object120978:head# 2024-04-29T14:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678ece68:::benchmark_data_smithi194_121512_object51622:head# 2024-04-29T14:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678f36e1:::benchmark_data_smithi194_121512_object113587:head# 2024-04-29T14:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:678f9547:::benchmark_data_smithi194_121512_object132978:head# 2024-04-29T14:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6790cfee:::benchmark_data_smithi194_121512_object75765:head# 2024-04-29T14:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6790f947:::benchmark_data_smithi194_121512_object1849:head# 2024-04-29T14:27:11.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6791b54d:::benchmark_data_smithi194_121512_object110955:head# 2024-04-29T14:27:11.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6791f7c7:::benchmark_data_smithi194_121512_object86601:head# 2024-04-29T14:27:11.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67920137:::benchmark_data_smithi194_121512_object91991:head# 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67923f7f:::benchmark_data_smithi194_121512_object35916:head# 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67926618:::benchmark_data_smithi194_121512_object42766:head# 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6792ce21:::benchmark_data_smithi194_121512_object9941:head# 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6792e69f:::benchmark_data_smithi194_121512_object38780:head# 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67938f41:::benchmark_data_smithi194_121512_object49885:head# 2024-04-29T14:27:11.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67939995:::benchmark_data_smithi194_121512_object1517:head# 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67945fe2:::benchmark_data_smithi194_121512_object76211:head# 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67949a90:::benchmark_data_smithi194_121512_object26007:head# 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67954419:::benchmark_data_smithi194_121512_object10002:head# 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67961dfe:::benchmark_data_smithi194_121512_object80398:head# 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6796e0e8:::benchmark_data_smithi194_121512_object62218:head# 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679758c8:::benchmark_data_smithi194_121512_object13542:head# 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6797777d:::benchmark_data_smithi194_121512_object69226:head# 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6797b390:::benchmark_data_smithi194_121512_object60234:head# 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6797e3f1:::benchmark_data_smithi194_121512_object7258:head# 2024-04-29T14:27:11.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6798cd25:::benchmark_data_smithi194_121512_object68620:head# 2024-04-29T14:27:11.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679965f0:::benchmark_data_smithi194_121512_object56455:head# 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6799807b:::benchmark_data_smithi194_121512_object18374:head# 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6799838c:::benchmark_data_smithi194_121512_object46020:head# 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6799ec85:::benchmark_data_smithi194_121512_object110504:head# 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679a119d:::benchmark_data_smithi194_121512_object50556:head# 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679c105e:::benchmark_data_smithi194_121512_object110527:head# 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679d47ea:::benchmark_data_smithi194_121512_object16453:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679d5942:::benchmark_data_smithi194_121512_object125160:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679d86e2:::benchmark_data_smithi194_121512_object120371:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679daa04:::benchmark_data_smithi194_121512_object43338:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679e8fb4:::benchmark_data_smithi194_121512_object3698:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679ec7c5:::benchmark_data_smithi194_121512_object88822:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679f672f:::benchmark_data_smithi194_121512_object100579:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:679fd35f:::benchmark_data_smithi194_121512_object122908:head# 2024-04-29T14:27:11.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a00f63:::benchmark_data_smithi194_121512_object72238:head# 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a031e5:::benchmark_data_smithi194_121512_object43366:head# 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a07c7e:::benchmark_data_smithi194_121512_object62403:head# 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a08206:::benchmark_data_smithi194_121512_object125436:head# 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a0baaa:::benchmark_data_smithi194_121512_object83642:head# 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a11ff1:::benchmark_data_smithi194_121512_object95453:head# 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a139d7:::benchmark_data_smithi194_121512_object4936:head# 2024-04-29T14:27:11.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a13be9:::benchmark_data_smithi194_121512_object42063:head# 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a198eb:::benchmark_data_smithi194_121512_object54666:head# 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a1d743:::benchmark_data_smithi194_121512_object52344:head# 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a1e572:::benchmark_data_smithi194_121512_object5020:head# 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a260c7:::benchmark_data_smithi194_121512_object9526:head# 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a276ac:::benchmark_data_smithi194_121512_object63943:head# 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a2e681:::benchmark_data_smithi194_121512_object92670:head# 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a315db:::benchmark_data_smithi194_121512_object28752:head# 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a36232:::benchmark_data_smithi194_121512_object93355:head# 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a37bdf:::benchmark_data_smithi194_121512_object117519:head# 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a389e7:::benchmark_data_smithi194_121512_object81865:head# 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a49b98:::benchmark_data_smithi194_121512_object107809:head# 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a52e10:::benchmark_data_smithi194_121512_object60329:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a64da4:::benchmark_data_smithi194_121512_object124588:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a6816d:::benchmark_data_smithi194_121512_object41864:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a70ed9:::benchmark_data_smithi194_121512_object86792:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a72452:::benchmark_data_smithi194_121512_object112234:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a8b115:::benchmark_data_smithi194_121512_object121741:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67a8e182:::benchmark_data_smithi194_121512_object29631:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67aa1735:::benchmark_data_smithi194_121512_object94689:head# 2024-04-29T14:27:11.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ac3936:::benchmark_data_smithi194_121512_object56098:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ad32f1:::benchmark_data_smithi194_121512_object94400:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ae695b:::benchmark_data_smithi194_121512_object45116:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ae9175:::benchmark_data_smithi194_121512_object55729:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b115a6:::benchmark_data_smithi194_121512_object102315:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b13176:::benchmark_data_smithi194_121512_object50303:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b154bb:::benchmark_data_smithi194_121512_object15251:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b16d4a:::benchmark_data_smithi194_121512_object30641:head# 2024-04-29T14:27:11.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b19523:::benchmark_data_smithi194_121512_object101482:head# 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b1d615:::benchmark_data_smithi194_121512_object10277:head# 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b20196:::benchmark_data_smithi194_121512_object93109:head# 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b23192:::benchmark_data_smithi194_121512_object106652:head# 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b356ef:::benchmark_data_smithi194_121512_object37741:head# 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b394c7:::benchmark_data_smithi194_121512_object75574:head# 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b42cad:::benchmark_data_smithi194_121512_object22602:head# 2024-04-29T14:27:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b42ce3:::benchmark_data_smithi194_121512_object92840:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b4cb6e:::benchmark_data_smithi194_121512_object57036:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b67677:::benchmark_data_smithi194_121512_object52302:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b7760a:::benchmark_data_smithi194_121512_object119616:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b83bfd:::benchmark_data_smithi194_121512_object71796:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b98e4b:::benchmark_data_smithi194_121512_object38251:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67b9ec93:::benchmark_data_smithi194_121512_object7314:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ba5fb3:::benchmark_data_smithi194_121512_object19189:head# 2024-04-29T14:27:11.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bad79b:::benchmark_data_smithi194_121512_object2810:head# 2024-04-29T14:27:11.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bb7c76:::benchmark_data_smithi194_121512_object82402:head# 2024-04-29T14:27:11.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bc09fa:::benchmark_data_smithi194_121512_object18109:head# 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bc27ec:::benchmark_data_smithi194_121512_object106302:head# 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bc40fc:::benchmark_data_smithi194_121512_object92792:head# 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bc7dca:::benchmark_data_smithi194_121512_object18456:head# 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bc9ea4:::benchmark_data_smithi194_121512_object21422:head# 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67be0cda:::benchmark_data_smithi194_121512_object124486:head# 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67be23c3:::benchmark_data_smithi194_121512_object98765:head# 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67be9fb5:::benchmark_data_smithi194_121512_object121779:head# 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bef82c:::benchmark_data_smithi194_121512_object89598:head# 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67bf75b2:::benchmark_data_smithi194_121512_object11822:head# 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c04e1f:::benchmark_data_smithi194_121512_object38374:head# 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c0c2ac:::benchmark_data_smithi194_121512_object96266:head# 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c0cd3e:::benchmark_data_smithi194_121512_object119500:head# 2024-04-29T14:27:11.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c123d2:::benchmark_data_smithi194_121512_object22827:head# 2024-04-29T14:27:11.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c1264c:::benchmark_data_smithi194_121512_object20531:head# 2024-04-29T14:27:11.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c16299:::benchmark_data_smithi194_121512_object76638:head# 2024-04-29T14:27:11.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c215d6:::benchmark_data_smithi194_121512_object26412:head# 2024-04-29T14:27:11.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c21df7:::benchmark_data_smithi194_121512_object38182:head# 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c23989:::benchmark_data_smithi194_121512_object93469:head# 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c24d73:::benchmark_data_smithi194_121512_object78246:head# 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c254a7:::benchmark_data_smithi194_121512_object108880:head# 2024-04-29T14:27:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c271a1:::benchmark_data_smithi194_121512_object39131:head# 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c3dc17:::benchmark_data_smithi194_121512_object113350:head# 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c45061:::benchmark_data_smithi194_121512_object103057:head# 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c59cd4:::benchmark_data_smithi194_121512_object25468:head# 2024-04-29T14:27:11.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c5a6c1:::benchmark_data_smithi194_121512_object105961:head# 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c62ec3:::benchmark_data_smithi194_121512_object126091:head# 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c67748:::benchmark_data_smithi194_121512_object72497:head# 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c67b20:::benchmark_data_smithi194_121512_object32794:head# 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c7b941:::benchmark_data_smithi194_121512_object93111:head# 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c88baa:::benchmark_data_smithi194_121512_object95592:head# 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c936eb:::benchmark_data_smithi194_121512_object28573:head# 2024-04-29T14:27:11.964 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67c9c510:::benchmark_data_smithi194_121512_object58672:head# 2024-04-29T14:27:11.965 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.965 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ca5d96:::benchmark_data_smithi194_121512_object17413:head# 2024-04-29T14:27:11.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ca9343:::benchmark_data_smithi194_121512_object132585:head# 2024-04-29T14:27:11.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.966 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67cb68ae:::benchmark_data_smithi194_121512_object113601:head# 2024-04-29T14:27:11.966 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67cb9e63:::benchmark_data_smithi194_121512_object113541:head# 2024-04-29T14:27:11.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67cd86bf:::benchmark_data_smithi194_121512_object78074:head# 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67cec881:::benchmark_data_smithi194_121512_object57329:head# 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67cf57d8:::benchmark_data_smithi194_121512_object94027:head# 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67cf95b4:::benchmark_data_smithi194_121512_object67177:head# 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d083d9:::benchmark_data_smithi194_121512_object20887:head# 2024-04-29T14:27:11.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d08482:::benchmark_data_smithi194_121512_object73317:head# 2024-04-29T14:27:11.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d09f59:::benchmark_data_smithi194_121512_object63433:head# 2024-04-29T14:27:11.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d0abae:::benchmark_data_smithi194_121512_object17196:head# 2024-04-29T14:27:11.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d0fb82:::benchmark_data_smithi194_121512_object106946:head# 2024-04-29T14:27:11.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d112a2:::benchmark_data_smithi194_121512_object94115:head# 2024-04-29T14:27:11.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d118fc:::benchmark_data_smithi194_121512_object62385:head# 2024-04-29T14:27:11.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d27723:::benchmark_data_smithi194_121512_object53450:head# 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d36b78:::benchmark_data_smithi194_121512_object10945:head# 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d37af0:::benchmark_data_smithi194_121512_object113007:head# 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d593b5:::benchmark_data_smithi194_121512_object12738:head# 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d67627:::benchmark_data_smithi194_121512_object46240:head# 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d72001:::benchmark_data_smithi194_121512_object61516:head# 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d75300:::benchmark_data_smithi194_121512_object131589:head# 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d80fe7:::benchmark_data_smithi194_121512_object83145:head# 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d82372:::benchmark_data_smithi194_121512_object40324:head# 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d8a764:::benchmark_data_smithi194_121512_object26023:head# 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67d9f60c:::benchmark_data_smithi194_121512_object45307:head# 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67da01d4:::benchmark_data_smithi194_121512_object75108:head# 2024-04-29T14:27:11.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67db5402:::benchmark_data_smithi194_121512_object57361:head# 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67db9ac6:::benchmark_data_smithi194_121512_object108091:head# 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67dba27d:::benchmark_data_smithi194_121512_object108837:head# 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67dbb705:::benchmark_data_smithi194_121512_object86755:head# 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67dc9cfe:::benchmark_data_smithi194_121512_object19248:head# 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67dca028:::benchmark_data_smithi194_121512_object54403:head# 2024-04-29T14:27:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67dd0d01:::benchmark_data_smithi194_121512_object31835:head# 2024-04-29T14:27:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67dd324d:::benchmark_data_smithi194_121512_object5778:head# 2024-04-29T14:27:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67dd45e9:::benchmark_data_smithi194_121512_object1998:head# 2024-04-29T14:27:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ddbec8:::benchmark_data_smithi194_121512_object124772:head# 2024-04-29T14:27:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67de3b09:::benchmark_data_smithi194_121512_object5252:head# 2024-04-29T14:27:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e0087a:::benchmark_data_smithi194_121512_object35079:head# 2024-04-29T14:27:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e0a9b2:::benchmark_data_smithi194_121512_object58256:head# 2024-04-29T14:27:11.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e12335:::benchmark_data_smithi194_121512_object56435:head# 2024-04-29T14:27:11.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e1a953:::benchmark_data_smithi194_121512_object35428:head# 2024-04-29T14:27:11.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e1b30d:::benchmark_data_smithi194_121512_object109628:head# 2024-04-29T14:27:11.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e271f6:::benchmark_data_smithi194_121512_object89791:head# 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e476f3:::benchmark_data_smithi194_121512_object116991:head# 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e494cb:::benchmark_data_smithi194_121512_object29632:head# 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e4a6a8:::benchmark_data_smithi194_121512_object86625:head# 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e6aa90:::benchmark_data_smithi194_121512_object113739:head# 2024-04-29T14:27:11.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e849d7:::benchmark_data_smithi194_121512_object108681:head# 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e87007:::benchmark_data_smithi194_121512_object93817:head# 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e8c9c3:::benchmark_data_smithi194_121512_object8768:head# 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e8e8ce:::benchmark_data_smithi194_121512_object91711:head# 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e943fe:::benchmark_data_smithi194_121512_object40349:head# 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67e9f9e0:::benchmark_data_smithi194_121512_object62415:head# 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67eab210:::benchmark_data_smithi194_121512_object89408:head# 2024-04-29T14:27:11.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67eaf1e2:::benchmark_data_smithi194_121512_object57427:head# 2024-04-29T14:27:11.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.997 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67eb2418:::benchmark_data_smithi194_121512_object78258:head# 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67eb3fbd:::benchmark_data_smithi194_121512_object28849:head# 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67eb9e06:::benchmark_data_smithi194_121512_object130237:head# 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ebcb52:::benchmark_data_smithi194_121512_object15116:head# 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ec1127:::benchmark_data_smithi194_121512_object35285:head# 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ed89c4:::benchmark_data_smithi194_121512_object8950:head# 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67efa4ed:::benchmark_data_smithi194_121512_object101282:head# 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f0cf62:::benchmark_data_smithi194_121512_object16964:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f0f862:::benchmark_data_smithi194_121512_object15695:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f23f6f:::benchmark_data_smithi194_121512_object76296:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f32a1e:::benchmark_data_smithi194_121512_object61172:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f6b2df:::benchmark_data_smithi194_121512_object52948:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f7001e:::benchmark_data_smithi194_121512_object23369:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f7efe8:::benchmark_data_smithi194_121512_object10740:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f98bea:::benchmark_data_smithi194_121512_object97487:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f99e04:::benchmark_data_smithi194_121512_object2731:head# 2024-04-29T14:27:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67f9d647:::benchmark_data_smithi194_121512_object111915:head# 2024-04-29T14:27:12.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fb1613:::benchmark_data_smithi194_121512_object129363:head# 2024-04-29T14:27:12.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fb331e:::benchmark_data_smithi194_121512_object83377:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fb8b3f:::benchmark_data_smithi194_121512_object64619:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fbd859:::benchmark_data_smithi194_121512_object118031:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fbeda2:::benchmark_data_smithi194_121512_object38972:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fc2498:::benchmark_data_smithi194_121512_object91016:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fc4ca1:::benchmark_data_smithi194_121512_object26253:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fcab31:::benchmark_data_smithi194_121512_object75076:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fd107b:::benchmark_data_smithi194_121512_object87023:head# 2024-04-29T14:27:12.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fda63b:::benchmark_data_smithi194_121512_object36343:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fde024:::benchmark_data_smithi194_121512_object113991:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fde851:::benchmark_data_smithi194_121512_object74215:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67fe0f49:::benchmark_data_smithi194_121512_object120431:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:67ffbf99:::benchmark_data_smithi194_121512_object90627:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68000ddd:::benchmark_data_smithi194_121512_object92948:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68001fb1:::benchmark_data_smithi194_121512_object96523:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6800ac4a:::benchmark_data_smithi194_121512_object83632:head# 2024-04-29T14:27:12.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6800b420:::benchmark_data_smithi194_121512_object126918:head# 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68016356:::benchmark_data_smithi194_121512_object51677:head# 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68018d97:::benchmark_data_smithi194_121512_object97962:head# 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6801af2f:::benchmark_data_smithi194_121512_object7790:head# 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6801c86a:::benchmark_data_smithi194_121512_object37044:head# 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68033226:::benchmark_data_smithi194_121512_object67568:head# 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680406c8:::benchmark_data_smithi194_121512_object127404:head# 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68041bc0:::benchmark_data_smithi194_121512_object58668:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68043135:::benchmark_data_smithi194_121512_object56460:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6804b15f:::benchmark_data_smithi194_121512_object29639:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6804f2f8:::benchmark_data_smithi194_121512_object94753:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6805118a:::benchmark_data_smithi194_121512_object48903:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68054053:::benchmark_data_smithi194_121512_object71551:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6805c978:::benchmark_data_smithi194_121512_object115242:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68061749:::benchmark_data_smithi194_121512_object79058:head# 2024-04-29T14:27:12.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68067cbc:::benchmark_data_smithi194_121512_object52958:head# 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68068c7b:::benchmark_data_smithi194_121512_object87296:head# 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68072a0d:::benchmark_data_smithi194_121512_object79561:head# 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6807f594:::benchmark_data_smithi194_121512_object77611:head# 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6808358b:::benchmark_data_smithi194_121512_object49267:head# 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68089337:::benchmark_data_smithi194_121512_object75880:head# 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680941e4:::benchmark_data_smithi194_121512_object115543:head# 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68096ee8:::benchmark_data_smithi194_121512_object45601:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6809a73e:::benchmark_data_smithi194_121512_object48192:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6809b031:::benchmark_data_smithi194_121512_object11377:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680a4699:::benchmark_data_smithi194_121512_object54140:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680ac8cf:::benchmark_data_smithi194_121512_object30098:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680bbcc0:::benchmark_data_smithi194_121512_object57581:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680c17b1:::benchmark_data_smithi194_121512_object18169:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680c60dd:::benchmark_data_smithi194_121512_object67611:head# 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680cd3fd:::benchmark_data_smithi194_121512_object103308:head# 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680cf62c:::benchmark_data_smithi194_121512_object17203:head# 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:680ecec1:::benchmark_data_smithi194_121512_object10971:head# 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68101bd0:::benchmark_data_smithi194_121512_object109892:head# 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68105f73:::benchmark_data_smithi194_121512_object83695:head# 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68106b64:::benchmark_data_smithi194_121512_object103336:head# 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6810f013:::benchmark_data_smithi194_121512_object121189:head# 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68112441:::benchmark_data_smithi194_121512_object100107:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68129cb5:::benchmark_data_smithi194_121512_object6573:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6812ab4f:::benchmark_data_smithi194_121512_object80721:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68130715:::benchmark_data_smithi194_121512_object24627:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6814029f:::benchmark_data_smithi194_121512_object30073:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681498dd:::benchmark_data_smithi194_121512_object46482:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6814c0ee:::benchmark_data_smithi194_121512_object67483:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6815a375:::benchmark_data_smithi194_121512_object54696:head# 2024-04-29T14:27:12.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68170186:::benchmark_data_smithi194_121512_object63817:head# 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68172f74:::benchmark_data_smithi194_121512_object36241:head# 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68180a3d:::benchmark_data_smithi194_121512_object82050:head# 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6818a14a:::benchmark_data_smithi194_121512_object31084:head# 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6819a2e8:::benchmark_data_smithi194_121512_object87352:head# 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6819c1be:::benchmark_data_smithi194_121512_object80881:head# 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681a3448:::benchmark_data_smithi194_121512_object72336:head# 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681bb299:::benchmark_data_smithi194_121512_object66391:head# 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681cf7c8:::benchmark_data_smithi194_121512_object48767:head# 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681d810c:::benchmark_data_smithi194_121512_object130185:head# 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681ef027:::benchmark_data_smithi194_121512_object46726:head# 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681f8e5c:::benchmark_data_smithi194_121512_object101930:head# 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681f9346:::benchmark_data_smithi194_121512_object122137:head# 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681fa203:::benchmark_data_smithi194_121512_object9631:head# 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:681fda6e:::benchmark_data_smithi194_121512_object59893:head# 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682032c5:::benchmark_data_smithi194_121512_object13021:head# 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6820dbfb:::benchmark_data_smithi194_121512_object101577:head# 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68227573:::benchmark_data_smithi194_121512_object9201:head# 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6822b73a:::benchmark_data_smithi194_121512_object29273:head# 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6822c6d7:::benchmark_data_smithi194_121512_object127956:head# 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6822fd48:::benchmark_data_smithi194_121512_object99863:head# 2024-04-29T14:27:12.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6823eeee:::benchmark_data_smithi194_121512_object31143:head# 2024-04-29T14:27:12.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6824605d:::benchmark_data_smithi194_121512_object40878:head# 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68250005:::benchmark_data_smithi194_121512_object103845:head# 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6825ba8a:::benchmark_data_smithi194_121512_object7775:head# 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682620dc:::benchmark_data_smithi194_121512_object130980:head# 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68268815:::benchmark_data_smithi194_121512_object113853:head# 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6826a518:::benchmark_data_smithi194_121512_object70774:head# 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.036 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6826c0db:::benchmark_data_smithi194_121512_object82055:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6826df23:::benchmark_data_smithi194_121512_object127988:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682747c1:::benchmark_data_smithi194_121512_object7214:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68276f4d:::benchmark_data_smithi194_121512_object35974:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6828548b:::benchmark_data_smithi194_121512_object8859:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6828c9a6:::benchmark_data_smithi194_121512_object15491:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6828dca3:::benchmark_data_smithi194_121512_object71604:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68294c8c:::benchmark_data_smithi194_121512_object107655:head# 2024-04-29T14:27:12.037 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6829e69a:::benchmark_data_smithi194_121512_object61391:head# 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682a472d:::benchmark_data_smithi194_121512_object88268:head# 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682b34cb:::benchmark_data_smithi194_121512_object31676:head# 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682b53fd:::benchmark_data_smithi194_121512_object18757:head# 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682ba652:::benchmark_data_smithi194_121512_object125317:head# 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682c4dda:::benchmark_data_smithi194_121512_object74439:head# 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682c629b:::benchmark_data_smithi194_121512_object6915:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682d3391:::benchmark_data_smithi194_121512_object59460:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682d772f:::benchmark_data_smithi194_121512_object82274:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682d84c6:::benchmark_data_smithi194_121512_object12093:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682f82ab:::benchmark_data_smithi194_121512_object128950:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:682f8d27:::benchmark_data_smithi194_121512_object58004:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68300030:::benchmark_data_smithi194_121512_object118269:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6830e56a:::benchmark_data_smithi194_121512_object54942:head# 2024-04-29T14:27:12.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68315618:::benchmark_data_smithi194_121512_object23827:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6831fe13:::benchmark_data_smithi194_121512_object63037:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68320099:::benchmark_data_smithi194_121512_object130215:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6832c73e:::benchmark_data_smithi194_121512_object9099:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6834972a:::benchmark_data_smithi194_121512_object69316:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6834b3d1:::benchmark_data_smithi194_121512_object108869:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6834c2a4:::benchmark_data_smithi194_121512_object94664:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6834ec17:::benchmark_data_smithi194_121512_object132290:head# 2024-04-29T14:27:12.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68354138:::benchmark_data_smithi194_121512_object50472:head# 2024-04-29T14:27:12.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6835763c:::benchmark_data_smithi194_121512_object18580:head# 2024-04-29T14:27:12.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6835b8ce:::benchmark_data_smithi194_121512_object21547:head# 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68382659:::benchmark_data_smithi194_121512_object86260:head# 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68383b61:::benchmark_data_smithi194_121512_object68120:head# 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68386f32:::benchmark_data_smithi194_121512_object70000:head# 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6838f62b:::benchmark_data_smithi194_121512_object124891:head# 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683980e2:::benchmark_data_smithi194_121512_object73927:head# 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68398e1b:::benchmark_data_smithi194_121512_object33715:head# 2024-04-29T14:27:12.042 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683ad75d:::benchmark_data_smithi194_121512_object97318:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683af90a:::benchmark_data_smithi194_121512_object12978:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683afb71:::benchmark_data_smithi194_121512_object3654:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683b06cc:::benchmark_data_smithi194_121512_object108734:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683b4ea3:::benchmark_data_smithi194_121512_object28933:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683b5e84:::benchmark_data_smithi194_121512_object130354:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683d045e:::benchmark_data_smithi194_121512_object40536:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683d1632:::benchmark_data_smithi194_121512_object95065:head# 2024-04-29T14:27:12.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683dbb2c:::benchmark_data_smithi194_121512_object75435:head# 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683dd8d3:::benchmark_data_smithi194_121512_object93332:head# 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683e2ff6:::benchmark_data_smithi194_121512_object26810:head# 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683e8d38:::benchmark_data_smithi194_121512_object57481:head# 2024-04-29T14:27:12.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683eca43:::benchmark_data_smithi194_121512_object20067:head# 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683ed0da:::benchmark_data_smithi194_121512_object107982:head# 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683f43d6:::benchmark_data_smithi194_121512_object122553:head# 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683f5f12:::benchmark_data_smithi194_121512_object99253:head# 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:683fbfeb:::benchmark_data_smithi194_121512_object15868:head# 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68406ea1:::benchmark_data_smithi194_121512_object95736:head# 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6840a96e:::benchmark_data_smithi194_121512_object104327:head# 2024-04-29T14:27:12.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68411213:::benchmark_data_smithi194_121512_object114436:head# 2024-04-29T14:27:12.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68418902:::benchmark_data_smithi194_121512_object109859:head# 2024-04-29T14:27:12.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6841ef50:::benchmark_data_smithi194_121512_object19748:head# 2024-04-29T14:27:12.050 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.050 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6842a95f:::benchmark_data_smithi194_121512_object117873:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68434604:::benchmark_data_smithi194_121512_object77458:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6843ef94:::benchmark_data_smithi194_121512_object22242:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6844a5d8:::benchmark_data_smithi194_121512_object1449:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684625f7:::benchmark_data_smithi194_121512_object123711:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68465e0b:::benchmark_data_smithi194_121512_object109198:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684699a0:::benchmark_data_smithi194_121512_object31376:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684730e1:::benchmark_data_smithi194_121512_object114545:head# 2024-04-29T14:27:12.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68476815:::benchmark_data_smithi194_121512_object131627:head# 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684769d7:::benchmark_data_smithi194_121512_object45195:head# 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6847d1ff:::benchmark_data_smithi194_121512_object54153:head# 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6847d75d:::benchmark_data_smithi194_121512_object53819:head# 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6848178c:::benchmark_data_smithi194_121512_object98816:head# 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68486404:::benchmark_data_smithi194_121512_object66744:head# 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6849deac:::benchmark_data_smithi194_121512_object102617:head# 2024-04-29T14:27:12.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684a474e:::benchmark_data_smithi194_121512_object40237:head# 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684ab1f1:::benchmark_data_smithi194_121512_object14375:head# 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684b65d8:::benchmark_data_smithi194_121512_object54802:head# 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684b7bc3:::benchmark_data_smithi194_121512_object106084:head# 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684ba011:::benchmark_data_smithi194_121512_object116037:head# 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684cb553:::benchmark_data_smithi194_121512_object127998:head# 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684cbcca:::benchmark_data_smithi194_121512_object16115:head# 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684d9aa5:::benchmark_data_smithi194_121512_object95542:head# 2024-04-29T14:27:12.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684e15dd:::benchmark_data_smithi194_121512_object35109:head# 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684f369b:::benchmark_data_smithi194_121512_object61616:head# 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684fa0ea:::benchmark_data_smithi194_121512_object73086:head# 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:684fd10d:::benchmark_data_smithi194_121512_object96778:head# 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6852a9b1:::benchmark_data_smithi194_121512_object127332:head# 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6852cb27:::benchmark_data_smithi194_121512_object43609:head# 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68534de7:::benchmark_data_smithi194_121512_object83063:head# 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685483ac:::benchmark_data_smithi194_121512_object108169:head# 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68554fd5:::benchmark_data_smithi194_121512_object117098:head# 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6855e81d:::benchmark_data_smithi194_121512_object69199:head# 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68568a06:::benchmark_data_smithi194_121512_object107682:head# 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68575dd8:::benchmark_data_smithi194_121512_object37126:head# 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685883d7:::benchmark_data_smithi194_121512_object13464:head# 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6858b2d0:::benchmark_data_smithi194_121512_object75554:head# 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68592a88:::benchmark_data_smithi194_121512_object123595:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68593140:::benchmark_data_smithi194_121512_object88558:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6859562c:::benchmark_data_smithi194_121512_object11947:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6859ac9a:::benchmark_data_smithi194_121512_object132193:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685a2e4a:::benchmark_data_smithi194_121512_object12111:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685a73d4:::benchmark_data_smithi194_121512_object2895:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685b72e7:::benchmark_data_smithi194_121512_object94045:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685c280f:::benchmark_data_smithi194_121512_object286:head# 2024-04-29T14:27:12.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685c40e0:::benchmark_data_smithi194_121512_object54738:head# 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685ce150:::benchmark_data_smithi194_121512_object85724:head# 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685dc22e:::benchmark_data_smithi194_121512_object92583:head# 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685df142:::benchmark_data_smithi194_121512_object45602:head# 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685dfa65:::benchmark_data_smithi194_121512_object91556:head# 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685f2ad1:::benchmark_data_smithi194_121512_object65911:head# 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685f5d44:::benchmark_data_smithi194_121512_object112314:head# 2024-04-29T14:27:12.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685f81a5:::benchmark_data_smithi194_121512_object65235:head# 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685f99e0:::benchmark_data_smithi194_121512_object40217:head# 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685f9e02:::benchmark_data_smithi194_121512_object94716:head# 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685fad57:::benchmark_data_smithi194_121512_object25144:head# 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:685fc633:::benchmark_data_smithi194_121512_object118677:head# 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68609608:::benchmark_data_smithi194_121512_object112729:head# 2024-04-29T14:27:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68625ba2:::benchmark_data_smithi194_121512_object120536:head# 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6862b3a5:::benchmark_data_smithi194_121512_object84374:head# 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68630bcb:::benchmark_data_smithi194_121512_object114344:head# 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6863204e:::benchmark_data_smithi194_121512_object17884:head# 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6863c011:::benchmark_data_smithi194_121512_object14810:head# 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68649175:::benchmark_data_smithi194_121512_object125679:head# 2024-04-29T14:27:12.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6864a4d5:::benchmark_data_smithi194_121512_object114968:head# 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686527c2:::benchmark_data_smithi194_121512_object23716:head# 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68656883:::benchmark_data_smithi194_121512_object56660:head# 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68674516:::benchmark_data_smithi194_121512_object46694:head# 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68675d84:::benchmark_data_smithi194_121512_object110334:head# 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6867974a:::benchmark_data_smithi194_121512_object131833:head# 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68681d00:::benchmark_data_smithi194_121512_object54224:head# 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68688503:::benchmark_data_smithi194_121512_object107816:head# 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6868cdde:::benchmark_data_smithi194_121512_object111352:head# 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68696782:::benchmark_data_smithi194_121512_object79745:head# 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686a378f:::benchmark_data_smithi194_121512_object46365:head# 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686a7527:::benchmark_data_smithi194_121512_object63698:head# 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686aeb38:::benchmark_data_smithi194_121512_object120747:head# 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686af2a5:::benchmark_data_smithi194_121512_object52818:head# 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686b282c:::benchmark_data_smithi194_121512_object26418:head# 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686ca86e:::benchmark_data_smithi194_121512_object3789:head# 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686f7587:::benchmark_data_smithi194_121512_object22515:head# 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:686ffbf0:::benchmark_data_smithi194_121512_object115167:head# 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687071e3:::benchmark_data_smithi194_121512_object72545:head# 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687150e2:::benchmark_data_smithi194_121512_object77178:head# 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6872e698:::benchmark_data_smithi194_121512_object60760:head# 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.082 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687428bc:::benchmark_data_smithi194_121512_object89468:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6874d4e6:::benchmark_data_smithi194_121512_object6220:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6875ba37:::benchmark_data_smithi194_121512_object111206:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68771396:::benchmark_data_smithi194_121512_object86438:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68778f59:::benchmark_data_smithi194_121512_object39582:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6877f3c0:::benchmark_data_smithi194_121512_object118964:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68780570:::benchmark_data_smithi194_121512_object96770:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687875e6:::benchmark_data_smithi194_121512_object3103:head# 2024-04-29T14:27:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687888b6:::benchmark_data_smithi194_121512_object13105:head# 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6878de26:::benchmark_data_smithi194_121512_object18174:head# 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6879541d:::benchmark_data_smithi194_121512_object16749:head# 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68798992:::benchmark_data_smithi194_121512_object62163:head# 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687a1c2c:::benchmark_data_smithi194_121512_object53117:head# 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687b2932:::benchmark_data_smithi194_121512_object15257:head# 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687b48ad:::benchmark_data_smithi194_121512_object54769:head# 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687b65a8:::benchmark_data_smithi194_121512_object83700:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687cea5f:::benchmark_data_smithi194_121512_object5305:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687cf44d:::benchmark_data_smithi194_121512_object2492:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687e1b75:::benchmark_data_smithi194_121512_object52133:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687f818d:::benchmark_data_smithi194_121512_object109560:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687fa7cb:::benchmark_data_smithi194_121512_object5034:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:687fd90e:::benchmark_data_smithi194_121512_object132921:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68814bfe:::benchmark_data_smithi194_121512_object96846:head# 2024-04-29T14:27:12.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68818aa0:::benchmark_data_smithi194_121512_object180:head# 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688260b5:::benchmark_data_smithi194_121512_object123710:head# 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6882ce94:::benchmark_data_smithi194_121512_object126389:head# 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688469b8:::benchmark_data_smithi194_121512_object118560:head# 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6884bbb7:::benchmark_data_smithi194_121512_object89971:head# 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6884e255:::benchmark_data_smithi194_121512_object111732:head# 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6885299f:::benchmark_data_smithi194_121512_object116032:head# 2024-04-29T14:27:12.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6885a7c8:::benchmark_data_smithi194_121512_object77835:head# 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688628eb:::benchmark_data_smithi194_121512_object21831:head# 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68863498:::benchmark_data_smithi194_121512_object45501:head# 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688645d4:::benchmark_data_smithi194_121512_object99808:head# 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68867fea:::benchmark_data_smithi194_121512_object75022:head# 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6886c171:::benchmark_data_smithi194_121512_object56631:head# 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6886d415:::benchmark_data_smithi194_121512_object104654:head# 2024-04-29T14:27:12.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6887a555:::benchmark_data_smithi194_121512_object24251:head# 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6887b224:::benchmark_data_smithi194_121512_object130677:head# 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6887c5c7:::benchmark_data_smithi194_121512_object8894:head# 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6888af23:::benchmark_data_smithi194_121512_object19639:head# 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68897243:::benchmark_data_smithi194_121512_object34018:head# 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6889a867:::benchmark_data_smithi194_121512_object66542:head# 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688a6497:::benchmark_data_smithi194_121512_object37534:head# 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688a8bd0:::benchmark_data_smithi194_121512_object23318:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688b6893:::benchmark_data_smithi194_121512_object37398:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688b7a75:::benchmark_data_smithi194_121512_object68258:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688c24f6:::benchmark_data_smithi194_121512_object115916:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688ca76a:::benchmark_data_smithi194_121512_object115328:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688d328d:::benchmark_data_smithi194_121512_object43652:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:688ea471:::benchmark_data_smithi194_121512_object59913:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68900e9d:::benchmark_data_smithi194_121512_object123628:head# 2024-04-29T14:27:12.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68904bd2:::benchmark_data_smithi194_121512_object26774:head# 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689057d9:::benchmark_data_smithi194_121512_object83140:head# 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68908321:::benchmark_data_smithi194_121512_object41746:head# 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6890e4d5:::benchmark_data_smithi194_121512_object52543:head# 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6891acd8:::benchmark_data_smithi194_121512_object51133:head# 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6892a1a3:::benchmark_data_smithi194_121512_object69174:head# 2024-04-29T14:27:12.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68936750:::benchmark_data_smithi194_121512_object35501:head# 2024-04-29T14:27:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68941191:::benchmark_data_smithi194_121512_object116841:head# 2024-04-29T14:27:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68942673:::benchmark_data_smithi194_121512_object25707:head# 2024-04-29T14:27:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68943d57:::benchmark_data_smithi194_121512_object119313:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6895fa44:::benchmark_data_smithi194_121512_object46272:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689651e1:::benchmark_data_smithi194_121512_object108917:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689751c6:::benchmark_data_smithi194_121512_object40102:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68978aa3:::benchmark_data_smithi194_121512_object66416:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6897fb42:::benchmark_data_smithi194_121512_object12685:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689845dc:::benchmark_data_smithi194_121512_object87665:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68986e6b:::benchmark_data_smithi194_121512_object15229:head# 2024-04-29T14:27:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6899bbf5:::benchmark_data_smithi194_121512_object54439:head# 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6899f051:::benchmark_data_smithi194_121512_object46087:head# 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689abc66:::benchmark_data_smithi194_121512_object74746:head# 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689b1484:::benchmark_data_smithi194_121512_object68748:head# 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689c9c6b:::benchmark_data_smithi194_121512_object88649:head# 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689e3890:::benchmark_data_smithi194_121512_object131048:head# 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689ea6a3:::benchmark_data_smithi194_121512_object40371:head# 2024-04-29T14:27:12.106 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689f2618:::benchmark_data_smithi194_121512_object130144:head# 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:689f2e2d:::benchmark_data_smithi194_121512_object50525:head# 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a06247:::benchmark_data_smithi194_121512_object47882:head# 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a16e33:::benchmark_data_smithi194_121512_object32642:head# 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a17fdb:::benchmark_data_smithi194_121512_object88504:head# 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a18110:::benchmark_data_smithi194_121512_object9346:head# 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a1944d:::benchmark_data_smithi194_121512_object118401:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a2181b:::benchmark_data_smithi194_121512_object61520:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a3039d:::benchmark_data_smithi194_121512_object45372:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a3c6fa:::benchmark_data_smithi194_121512_object71306:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a47716:::benchmark_data_smithi194_121512_object70515:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a4b425:::benchmark_data_smithi194_121512_object131685:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a4c19d:::benchmark_data_smithi194_121512_object32762:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a4efec:::benchmark_data_smithi194_121512_object104568:head# 2024-04-29T14:27:12.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a5981d:::benchmark_data_smithi194_121512_object64050:head# 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a65a0e:::benchmark_data_smithi194_121512_object78502:head# 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a6be7c:::benchmark_data_smithi194_121512_object106269:head# 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a777e3:::benchmark_data_smithi194_121512_object23244:head# 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a7cdad:::benchmark_data_smithi194_121512_object29702:head# 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a7fc94:::benchmark_data_smithi194_121512_object81489:head# 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a92829:::benchmark_data_smithi194_121512_object130440:head# 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a98bcd:::benchmark_data_smithi194_121512_object37013:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68a9b406:::benchmark_data_smithi194_121512_object80865:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ab0c42:::benchmark_data_smithi194_121512_object19683:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68abb2b2:::benchmark_data_smithi194_121512_object70083:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68abb801:::benchmark_data_smithi194_121512_object90478:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ac655f:::benchmark_data_smithi194_121512_object73417:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ad355a:::benchmark_data_smithi194_121512_object73976:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ad86c5:::benchmark_data_smithi194_121512_object81960:head# 2024-04-29T14:27:12.110 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ad9519:::benchmark_data_smithi194_121512_object50736:head# 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68adb090:::benchmark_data_smithi194_121512_object122494:head# 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ae0dec:::benchmark_data_smithi194_121512_object67550:head# 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ae155a:::benchmark_data_smithi194_121512_object26451:head# 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ae1c9b:::benchmark_data_smithi194_121512_object630:head# 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ae559f:::benchmark_data_smithi194_121512_object110179:head# 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68af2824:::benchmark_data_smithi194_121512_object132728:head# 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.111 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68afc849:::benchmark_data_smithi194_121512_object33723:head# 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68afea97:::benchmark_data_smithi194_121512_object46508:head# 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b03586:::benchmark_data_smithi194_121512_object82337:head# 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b071a6:::benchmark_data_smithi194_121512_object82958:head# 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b20215:::benchmark_data_smithi194_121512_object15403:head# 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b41581:::benchmark_data_smithi194_121512_object16216:head# 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b4e304:::benchmark_data_smithi194_121512_object14971:head# 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b59181:::benchmark_data_smithi194_121512_object62998:head# 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b70676:::benchmark_data_smithi194_121512_object93744:head# 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b734c4:::benchmark_data_smithi194_121512_object18070:head# 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b77483:::benchmark_data_smithi194_121512_object13326:head# 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b7939b:::benchmark_data_smithi194_121512_object68979:head# 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b7a954:::benchmark_data_smithi194_121512_object53460:head# 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b7c262:::benchmark_data_smithi194_121512_object54644:head# 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b847c6:::benchmark_data_smithi194_121512_object52326:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b8500c:::benchmark_data_smithi194_121512_object84907:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b8d92a:::benchmark_data_smithi194_121512_object106190:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b92679:::benchmark_data_smithi194_121512_object79455:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b946fe:::benchmark_data_smithi194_121512_object7444:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b94f44:::benchmark_data_smithi194_121512_object20391:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b962af:::benchmark_data_smithi194_121512_object77586:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68b9fe38:::benchmark_data_smithi194_121512_object71222:head# 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ba828d:::benchmark_data_smithi194_121512_object73915:head# 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68baab02:::benchmark_data_smithi194_121512_object26140:head# 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bab103:::benchmark_data_smithi194_121512_object12966:head# 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bb2d8a:::benchmark_data_smithi194_121512_object17994:head# 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bb9cfe:::benchmark_data_smithi194_121512_object25711:head# 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bba058:::benchmark_data_smithi194_121512_object78924:head# 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bc2d19:::benchmark_data_smithi194_121512_object29063:head# 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bc3a5b:::benchmark_data_smithi194_121512_object126:head# 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bc78d3:::benchmark_data_smithi194_121512_object129531:head# 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bcd5d9:::benchmark_data_smithi194_121512_object30667:head# 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bd873c:::benchmark_data_smithi194_121512_object85930:head# 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bdd22c:::benchmark_data_smithi194_121512_object70270:head# 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68be0923:::benchmark_data_smithi194_121512_object51666:head# 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68be22d1:::benchmark_data_smithi194_121512_object99648:head# 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bea684:::benchmark_data_smithi194_121512_object241:head# 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bf4ca5:::benchmark_data_smithi194_121512_object88745:head# 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68bf5caa:::benchmark_data_smithi194_121512_object47165:head# 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c0f247:::benchmark_data_smithi194_121512_object118424:head# 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c102f6:::benchmark_data_smithi194_121512_object90681:head# 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c14fcf:::benchmark_data_smithi194_121512_object86989:head# 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c1a8bd:::benchmark_data_smithi194_121512_object78445:head# 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c2077a:::benchmark_data_smithi194_121512_object64009:head# 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c2da86:::benchmark_data_smithi194_121512_object73364:head# 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c30a20:::benchmark_data_smithi194_121512_object54339:head# 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c33180:::benchmark_data_smithi194_121512_object111781:head# 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c36d22:::benchmark_data_smithi194_121512_object30535:head# 2024-04-29T14:27:12.118 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c391d1:::benchmark_data_smithi194_121512_object83008:head# 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c3ef56:::benchmark_data_smithi194_121512_object131180:head# 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c40cd9:::benchmark_data_smithi194_121512_object65326:head# 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c42151:::benchmark_data_smithi194_121512_object31068:head# 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c440d1:::benchmark_data_smithi194_121512_object79408:head# 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c466eb:::benchmark_data_smithi194_121512_object41982:head# 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c47da3:::benchmark_data_smithi194_121512_object23507:head# 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c520d6:::benchmark_data_smithi194_121512_object132859:head# 2024-04-29T14:27:12.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c58bc9:::benchmark_data_smithi194_121512_object74854:head# 2024-04-29T14:27:12.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c5d119:::benchmark_data_smithi194_121512_object41143:head# 2024-04-29T14:27:12.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c60a4e:::benchmark_data_smithi194_121512_object31245:head# 2024-04-29T14:27:12.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c64e21:::benchmark_data_smithi194_121512_object44697:head# 2024-04-29T14:27:12.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c6a912:::benchmark_data_smithi194_121512_object52831:head# 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c6d627:::benchmark_data_smithi194_121512_object30706:head# 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c76dbd:::benchmark_data_smithi194_121512_object15704:head# 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c7e414:::benchmark_data_smithi194_121512_object30092:head# 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c9039e:::benchmark_data_smithi194_121512_object103095:head# 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c90fbf:::benchmark_data_smithi194_121512_object79652:head# 2024-04-29T14:27:12.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68c91934:::benchmark_data_smithi194_121512_object28249:head# 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ca03a4:::benchmark_data_smithi194_121512_object79399:head# 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cae501:::benchmark_data_smithi194_121512_object69076:head# 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cb0fc2:::benchmark_data_smithi194_121512_object96599:head# 2024-04-29T14:27:12.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cb6796:::benchmark_data_smithi194_121512_object43148:head# 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cbb125:::benchmark_data_smithi194_121512_object77157:head# 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cbb534:::benchmark_data_smithi194_121512_object20734:head# 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cbbeb9:::benchmark_data_smithi194_121512_object42240:head# 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ccc22a:::benchmark_data_smithi194_121512_object64160:head# 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ce2444:::benchmark_data_smithi194_121512_object66136:head# 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cf0f33:::benchmark_data_smithi194_121512_object119585:head# 2024-04-29T14:27:12.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68cf74ee:::benchmark_data_smithi194_121512_object102150:head# 2024-04-29T14:27:12.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d0032f:::benchmark_data_smithi194_121512_object32974:head# 2024-04-29T14:27:12.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d0056d:::benchmark_data_smithi194_121512_object70886:head# 2024-04-29T14:27:12.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d00e59:::benchmark_data_smithi194_121512_object8849:head# 2024-04-29T14:27:12.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d0d26c:::benchmark_data_smithi194_121512_object33814:head# 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d10052:::benchmark_data_smithi194_121512_object3106:head# 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d1a180:::benchmark_data_smithi194_121512_object132019:head# 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d1e33a:::benchmark_data_smithi194_121512_object21268:head# 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d24ec9:::benchmark_data_smithi194_121512_object55765:head# 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d2a9ba:::benchmark_data_smithi194_121512_object7359:head# 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d40791:::benchmark_data_smithi194_121512_object58592:head# 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d48f34:::benchmark_data_smithi194_121512_object4871:head# 2024-04-29T14:27:12.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d4cdca:::benchmark_data_smithi194_121512_object89596:head# 2024-04-29T14:27:12.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d4dda8:::benchmark_data_smithi194_121512_object95540:head# 2024-04-29T14:27:12.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d4e674:::benchmark_data_smithi194_121512_object61811:head# 2024-04-29T14:27:12.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d57314:::benchmark_data_smithi194_121512_object50407:head# 2024-04-29T14:27:12.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d5d294:::benchmark_data_smithi194_121512_object114646:head# 2024-04-29T14:27:12.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d617b5:::benchmark_data_smithi194_121512_object77852:head# 2024-04-29T14:27:12.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d69356:::benchmark_data_smithi194_121512_object9556:head# 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d71c22:::benchmark_data_smithi194_121512_object80108:head# 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d752b1:::benchmark_data_smithi194_121512_object98703:head# 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d7a580:::benchmark_data_smithi194_121512_object17990:head# 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d84bca:::benchmark_data_smithi194_121512_object47710:head# 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d87fc4:::benchmark_data_smithi194_121512_object66905:head# 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d8fb91:::benchmark_data_smithi194_121512_object34329:head# 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d942a1:::benchmark_data_smithi194_121512_object44459:head# 2024-04-29T14:27:12.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d94b9a:::benchmark_data_smithi194_121512_object60306:head# 2024-04-29T14:27:12.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68d9c386:::benchmark_data_smithi194_121512_object62120:head# 2024-04-29T14:27:12.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68da3524:::benchmark_data_smithi194_121512_object1084:head# 2024-04-29T14:27:12.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68daa5b6:::benchmark_data_smithi194_121512_object115226:head# 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68dbf159:::benchmark_data_smithi194_121512_object131215:head# 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68dc5b8d:::benchmark_data_smithi194_121512_object72972:head# 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68dc6ce9:::benchmark_data_smithi194_121512_object90426:head# 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68dd5aeb:::benchmark_data_smithi194_121512_object82136:head# 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68df1062:::benchmark_data_smithi194_121512_object102373:head# 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68df28da:::benchmark_data_smithi194_121512_object6263:head# 2024-04-29T14:27:12.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68dfd07b:::benchmark_data_smithi194_121512_object39964:head# 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e0b5fd:::benchmark_data_smithi194_121512_object129443:head# 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e10953:::benchmark_data_smithi194_121512_object125475:head# 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e14c87:::benchmark_data_smithi194_121512_object98092:head# 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e1b66f:::benchmark_data_smithi194_121512_object10507:head# 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e1be6a:::benchmark_data_smithi194_121512_object60885:head# 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e336e5:::benchmark_data_smithi194_121512_object57104:head# 2024-04-29T14:27:12.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e435bc:::benchmark_data_smithi194_121512_object16318:head# 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e46120:::benchmark_data_smithi194_121512_object39138:head# 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e53b7e:::benchmark_data_smithi194_121512_object35415:head# 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e5eecf:::benchmark_data_smithi194_121512_object68677:head# 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e66191:::benchmark_data_smithi194_121512_object35123:head# 2024-04-29T14:27:12.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e727f3:::benchmark_data_smithi194_121512_object9395:head# 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68e926ab:::benchmark_data_smithi194_121512_object116593:head# 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68eb6c0f:::benchmark_data_smithi194_121512_object84137:head# 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68eb82dd:::benchmark_data_smithi194_121512_object105378:head# 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68eca731:::benchmark_data_smithi194_121512_object65085:head# 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ecb671:::benchmark_data_smithi194_121512_object65712:head# 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ecd6ae:::benchmark_data_smithi194_121512_object82062:head# 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ed4a1c:::benchmark_data_smithi194_121512_object71206:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68edceb8:::benchmark_data_smithi194_121512_object56900:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68edfc4a:::benchmark_data_smithi194_121512_object15955:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ee94e1:::benchmark_data_smithi194_121512_object120002:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68eea2a5:::benchmark_data_smithi194_121512_object125881:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ef01bd:::benchmark_data_smithi194_121512_object69179:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68efe90a:::benchmark_data_smithi194_121512_object3137:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f053af:::benchmark_data_smithi194_121512_object112074:head# 2024-04-29T14:27:12.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f07f1a:::benchmark_data_smithi194_121512_object90603:head# 2024-04-29T14:27:12.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f0ce9a:::benchmark_data_smithi194_121512_object127167:head# 2024-04-29T14:27:12.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f18104:::benchmark_data_smithi194_121512_object36105:head# 2024-04-29T14:27:12.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f3345a:::benchmark_data_smithi194_121512_object57148:head# 2024-04-29T14:27:12.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f39a4f:::benchmark_data_smithi194_121512_object131282:head# 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f4555f:::benchmark_data_smithi194_121512_object1975:head# 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f4a134:::benchmark_data_smithi194_121512_object59719:head# 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f4b261:::benchmark_data_smithi194_121512_object129567:head# 2024-04-29T14:27:12.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f53bed:::benchmark_data_smithi194_121512_object35831:head# 2024-04-29T14:27:12.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f56222:::benchmark_data_smithi194_121512_object12668:head# 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f570c3:::benchmark_data_smithi194_121512_object111926:head# 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f68c02:::benchmark_data_smithi194_121512_object129493:head# 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f6c4c4:::benchmark_data_smithi194_121512_object113341:head# 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68f6cfb0:::benchmark_data_smithi194_121512_object54020:head# 2024-04-29T14:27:12.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.154 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fa098c:::benchmark_data_smithi194_121512_object114955:head# 2024-04-29T14:27:12.154 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.154 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fa802a:::benchmark_data_smithi194_121512_object99415:head# 2024-04-29T14:27:12.154 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.154 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68faae84:::benchmark_data_smithi194_121512_object117756:head# 2024-04-29T14:27:12.154 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fb0d9c:::benchmark_data_smithi194_121512_object123741:head# 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fb5a56:::benchmark_data_smithi194_121512_object104554:head# 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fbd2bb:::benchmark_data_smithi194_121512_object25531:head# 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fc4cb3:::benchmark_data_smithi194_121512_object714:head# 2024-04-29T14:27:12.155 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fce922:::benchmark_data_smithi194_121512_object6735:head# 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fd8216:::benchmark_data_smithi194_121512_object124956:head# 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68fdc44d:::benchmark_data_smithi194_121512_object36207:head# 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68feea94:::benchmark_data_smithi194_121512_object28511:head# 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ff3c87:::benchmark_data_smithi194_121512_object55362:head# 2024-04-29T14:27:12.156 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:68ff51ef:::benchmark_data_smithi194_121512_object112428:head# 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69001a60:::benchmark_data_smithi194_121512_object38757:head# 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6900dec3:::benchmark_data_smithi194_121512_object13558:head# 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69017e0d:::benchmark_data_smithi194_121512_object49522:head# 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690276e0:::benchmark_data_smithi194_121512_object114842:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690284ba:::benchmark_data_smithi194_121512_object35884:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69034075:::benchmark_data_smithi194_121512_object93904:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69038797:::benchmark_data_smithi194_121512_object21576:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6904140f:::benchmark_data_smithi194_121512_object20349:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6905779c:::benchmark_data_smithi194_121512_object79056:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690585b0:::benchmark_data_smithi194_121512_object78106:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6905ce43:::benchmark_data_smithi194_121512_object105096:head# 2024-04-29T14:27:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6905f944:::benchmark_data_smithi194_121512_object91387:head# 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69065d2b:::benchmark_data_smithi194_121512_object110137:head# 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6908177e:::benchmark_data_smithi194_121512_object52472:head# 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69082eab:::benchmark_data_smithi194_121512_object50426:head# 2024-04-29T14:27:12.161 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690843dc:::benchmark_data_smithi194_121512_object34397:head# 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69089519:::benchmark_data_smithi194_121512_object47440:head# 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69094b9e:::benchmark_data_smithi194_121512_object61154:head# 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6909ece7:::benchmark_data_smithi194_121512_object30190:head# 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690a159d:::benchmark_data_smithi194_121512_object47807:head# 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690a305b:::benchmark_data_smithi194_121512_object44724:head# 2024-04-29T14:27:12.162 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690aa3b6:::benchmark_data_smithi194_121512_object83090:head# 2024-04-29T14:27:12.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690ab7b6:::benchmark_data_smithi194_121512_object7224:head# 2024-04-29T14:27:12.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.166 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690acc20:::benchmark_data_smithi194_121512_object41833:head# 2024-04-29T14:27:12.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690b3bf3:::benchmark_data_smithi194_121512_object95495:head# 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690bfa5a:::benchmark_data_smithi194_121512_object40396:head# 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690caa0b:::benchmark_data_smithi194_121512_object95707:head# 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690dc4b9:::benchmark_data_smithi194_121512_object67995:head# 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:690ff740:::benchmark_data_smithi194_121512_object50390:head# 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69103a5f:::benchmark_data_smithi194_121512_object8992:head# 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69109c30:::benchmark_data_smithi194_121512_object73352:head# 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6910b00a:::benchmark_data_smithi194_121512_object53129:head# 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6910df4f:::benchmark_data_smithi194_121512_object74427:head# 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69115a22:::benchmark_data_smithi194_121512_object121742:head# 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6911b1a9:::benchmark_data_smithi194_121512_object26216:head# 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69124e2b:::benchmark_data_smithi194_121512_object24171:head# 2024-04-29T14:27:12.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69127fdc:::benchmark_data_smithi194_121512_object56264:head# 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69129ee4:::benchmark_data_smithi194_121512_object94552:head# 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6912cc0e:::benchmark_data_smithi194_121512_object92931:head# 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691403f9:::benchmark_data_smithi194_121512_object117628:head# 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691499e4:::benchmark_data_smithi194_121512_object58394:head# 2024-04-29T14:27:12.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69152a4b:::benchmark_data_smithi194_121512_object83826:head# 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691584cd:::benchmark_data_smithi194_121512_object124128:head# 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6916300a:::benchmark_data_smithi194_121512_object25258:head# 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691640c4:::benchmark_data_smithi194_121512_object59107:head# 2024-04-29T14:27:12.174 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.175 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6916aa2a:::benchmark_data_smithi194_121512_object8808:head# 2024-04-29T14:27:12.175 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.175 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6916d263:::benchmark_data_smithi194_121512_object107145:head# 2024-04-29T14:27:12.175 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.175 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691742c7:::benchmark_data_smithi194_121512_object10806:head# 2024-04-29T14:27:12.175 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.175 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691785d6:::benchmark_data_smithi194_121512_object7058:head# 2024-04-29T14:27:12.176 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.176 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691890ee:::benchmark_data_smithi194_121512_object64767:head# 2024-04-29T14:27:12.176 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69189131:::benchmark_data_smithi194_121512_object124738:head# 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69196093:::benchmark_data_smithi194_121512_object103190:head# 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69199b00:::benchmark_data_smithi194_121512_object121223:head# 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691a26f5:::benchmark_data_smithi194_121512_object37570:head# 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691a50ec:::benchmark_data_smithi194_121512_object81798:head# 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691a9a8f:::benchmark_data_smithi194_121512_object100574:head# 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691abd64:::benchmark_data_smithi194_121512_object76848:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691b35b7:::benchmark_data_smithi194_121512_object130856:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691b4dd8:::benchmark_data_smithi194_121512_object125885:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691b6e8e:::benchmark_data_smithi194_121512_object26786:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691bc88a:::benchmark_data_smithi194_121512_object63704:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691bf62f:::benchmark_data_smithi194_121512_object75437:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691e4b29:::benchmark_data_smithi194_121512_object7075:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691f4cf7:::benchmark_data_smithi194_121512_object5198:head# 2024-04-29T14:27:12.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691f8ec8:::benchmark_data_smithi194_121512_object60181:head# 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:691fd29b:::benchmark_data_smithi194_121512_object123346:head# 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692024cf:::benchmark_data_smithi194_121512_object34325:head# 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69208e29:::benchmark_data_smithi194_121512_object29148:head# 2024-04-29T14:27:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69218ae6:::benchmark_data_smithi194_121512_object64648:head# 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6921c0f3:::benchmark_data_smithi194_121512_object18407:head# 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6921c5b6:::benchmark_data_smithi194_121512_object92045:head# 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6922ce70:::benchmark_data_smithi194_121512_object55439:head# 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6922cea0:::benchmark_data_smithi194_121512_object39520:head# 2024-04-29T14:27:12.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6923df9a:::benchmark_data_smithi194_121512_object44528:head# 2024-04-29T14:27:12.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.185 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6923e58a:::benchmark_data_smithi194_121512_object80329:head# 2024-04-29T14:27:12.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6923fecd:::benchmark_data_smithi194_121512_object124818:head# 2024-04-29T14:27:12.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69243b54:::benchmark_data_smithi194_121512_object107895:head# 2024-04-29T14:27:12.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69244228:::benchmark_data_smithi194_121512_object114982:head# 2024-04-29T14:27:12.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6924eda2:::benchmark_data_smithi194_121512_object85695:head# 2024-04-29T14:27:12.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.187 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69258369:::benchmark_data_smithi194_121512_object80213:head# 2024-04-29T14:27:12.187 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.190 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692592d6:::benchmark_data_smithi194_121512_object87247:head# 2024-04-29T14:27:12.191 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.191 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69262734:::benchmark_data_smithi194_121512_object76586:head# 2024-04-29T14:27:12.191 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6926f24c:::benchmark_data_smithi194_121512_object50876:head# 2024-04-29T14:27:12.193 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.193 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6928222d:::benchmark_data_smithi194_121512_object22195:head# 2024-04-29T14:27:12.193 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.193 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69292386:::benchmark_data_smithi194_121512_object104358:head# 2024-04-29T14:27:12.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6929290b:::benchmark_data_smithi194_121512_object20399:head# 2024-04-29T14:27:12.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692933d7:::benchmark_data_smithi194_121512_object35957:head# 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69295a3d:::benchmark_data_smithi194_121512_object91037:head# 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6929d1f3:::benchmark_data_smithi194_121512_object7572:head# 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6929d858:::benchmark_data_smithi194_121512_object65897:head# 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692a91ab:::benchmark_data_smithi194_121512_object58622:head# 2024-04-29T14:27:12.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.196 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692ae0f6:::benchmark_data_smithi194_121512_object90307:head# 2024-04-29T14:27:12.196 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692b66c7:::benchmark_data_smithi194_121512_object24078:head# 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692c995c:::benchmark_data_smithi194_121512_object85009:head# 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692d2934:::benchmark_data_smithi194_121512_object99869:head# 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692d5f6d:::benchmark_data_smithi194_121512_object84066:head# 2024-04-29T14:27:12.197 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.198 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692dcd6c:::benchmark_data_smithi194_121512_object51887:head# 2024-04-29T14:27:12.198 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.200 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692de5fd:::benchmark_data_smithi194_121512_object87171:head# 2024-04-29T14:27:12.200 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.201 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692e9c34:::benchmark_data_smithi194_121512_object6769:head# 2024-04-29T14:27:12.201 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.201 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692e9e12:::benchmark_data_smithi194_121512_object100913:head# 2024-04-29T14:27:12.201 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.201 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692f2065:::benchmark_data_smithi194_121512_object13561:head# 2024-04-29T14:27:12.201 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692f7323:::benchmark_data_smithi194_121512_object74019:head# 2024-04-29T14:27:12.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:692fec75:::benchmark_data_smithi194_121512_object118920:head# 2024-04-29T14:27:12.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6931d096:::benchmark_data_smithi194_121512_object118583:head# 2024-04-29T14:27:12.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69332fb4:::benchmark_data_smithi194_121512_object108950:head# 2024-04-29T14:27:12.208 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693331cb:::benchmark_data_smithi194_121512_object46770:head# 2024-04-29T14:27:12.208 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693384cc:::benchmark_data_smithi194_121512_object64793:head# 2024-04-29T14:27:12.208 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.210 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693592d5:::benchmark_data_smithi194_121512_object48917:head# 2024-04-29T14:27:12.210 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.210 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693674c1:::benchmark_data_smithi194_121512_object61402:head# 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69367821:::benchmark_data_smithi194_121512_object4101:head# 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6936851f:::benchmark_data_smithi194_121512_object24700:head# 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69373abe:::benchmark_data_smithi194_121512_object8092:head# 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6937d423:::benchmark_data_smithi194_121512_object18397:head# 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6939262e:::benchmark_data_smithi194_121512_object83164:head# 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693a59ad:::benchmark_data_smithi194_121512_object6164:head# 2024-04-29T14:27:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693a65dd:::benchmark_data_smithi194_121512_object123141:head# 2024-04-29T14:27:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693a819f:::benchmark_data_smithi194_121512_object85907:head# 2024-04-29T14:27:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693abd3f:::benchmark_data_smithi194_121512_object26024:head# 2024-04-29T14:27:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693b10d9:::benchmark_data_smithi194_121512_object30959:head# 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693b305d:::benchmark_data_smithi194_121512_object132634:head# 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693b8044:::benchmark_data_smithi194_121512_object129914:head# 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693bd5a4:::benchmark_data_smithi194_121512_object78469:head# 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693c2c14:::benchmark_data_smithi194_121512_object54634:head# 2024-04-29T14:27:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.219 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693cfa64:::benchmark_data_smithi194_121512_object1536:head# 2024-04-29T14:27:12.219 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.219 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693dbec7:::benchmark_data_smithi194_121512_object56538:head# 2024-04-29T14:27:12.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693dd690:::benchmark_data_smithi194_121512_object77664:head# 2024-04-29T14:27:12.221 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693e3dbe:::benchmark_data_smithi194_121512_object130310:head# 2024-04-29T14:27:12.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693f2bea:::benchmark_data_smithi194_121512_object9079:head# 2024-04-29T14:27:12.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:693fc7fc:::benchmark_data_smithi194_121512_object83336:head# 2024-04-29T14:27:12.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694044a3:::benchmark_data_smithi194_121512_object58324:head# 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69414b34:::benchmark_data_smithi194_121512_object63623:head# 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69415cbc:::benchmark_data_smithi194_121512_object106007:head# 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6942952f:::benchmark_data_smithi194_121512_object94150:head# 2024-04-29T14:27:12.227 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.228 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694331a5:::benchmark_data_smithi194_121512_object35064:head# 2024-04-29T14:27:12.228 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6943bf4a:::benchmark_data_smithi194_121512_object116080:head# 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6944797d:::benchmark_data_smithi194_121512_object95218:head# 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69448311:::benchmark_data_smithi194_121512_object36333:head# 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6944f3b9:::benchmark_data_smithi194_121512_object54121:head# 2024-04-29T14:27:12.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6945031e:::benchmark_data_smithi194_121512_object81709:head# 2024-04-29T14:27:12.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694577db:::benchmark_data_smithi194_121512_object18216:head# 2024-04-29T14:27:12.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69457d55:::benchmark_data_smithi194_121512_object89955:head# 2024-04-29T14:27:12.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6945d7cf:::benchmark_data_smithi194_121512_object27759:head# 2024-04-29T14:27:12.231 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.231 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69476784:::benchmark_data_smithi194_121512_object35333:head# 2024-04-29T14:27:12.231 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.231 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694788c2:::benchmark_data_smithi194_121512_object65813:head# 2024-04-29T14:27:12.231 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69485e5e:::benchmark_data_smithi194_121512_object16282:head# 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6949b653:::benchmark_data_smithi194_121512_object39877:head# 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6949f60c:::benchmark_data_smithi194_121512_object126834:head# 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6949f81d:::benchmark_data_smithi194_121512_object3618:head# 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694ac996:::benchmark_data_smithi194_121512_object74599:head# 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.232 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694ca331:::benchmark_data_smithi194_121512_object32532:head# 2024-04-29T14:27:12.233 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.234 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694ca66a:::benchmark_data_smithi194_121512_object130636:head# 2024-04-29T14:27:12.234 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694ca7fc:::benchmark_data_smithi194_121512_object61110:head# 2024-04-29T14:27:12.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694cca91:::benchmark_data_smithi194_121512_object51397:head# 2024-04-29T14:27:12.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694e874e:::benchmark_data_smithi194_121512_object72436:head# 2024-04-29T14:27:12.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694f71ab:::benchmark_data_smithi194_121512_object65540:head# 2024-04-29T14:27:12.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694f9913:::benchmark_data_smithi194_121512_object6975:head# 2024-04-29T14:27:12.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.238 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:694ffa4b:::benchmark_data_smithi194_121512_object60974:head# 2024-04-29T14:27:12.238 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69500312:::benchmark_data_smithi194_121512_object93152:head# 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69520ec7:::benchmark_data_smithi194_121512_object113411:head# 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69525f57:::benchmark_data_smithi194_121512_object111233:head# 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6952928e:::benchmark_data_smithi194_121512_object73410:head# 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69529d4f:::benchmark_data_smithi194_121512_object26402:head# 2024-04-29T14:27:12.239 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69531ddb:::benchmark_data_smithi194_121512_object88583:head# 2024-04-29T14:27:12.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69531ff9:::benchmark_data_smithi194_121512_object64359:head# 2024-04-29T14:27:12.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6953a3d4:::benchmark_data_smithi194_121512_object52051:head# 2024-04-29T14:27:12.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69542048:::benchmark_data_smithi194_121512_object56588:head# 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69545da2:::benchmark_data_smithi194_121512_object109910:head# 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69545dd2:::benchmark_data_smithi194_121512_object123150:head# 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6954c398:::benchmark_data_smithi194_121512_object101565:head# 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6955483b:::benchmark_data_smithi194_121512_object73367:head# 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69556a97:::benchmark_data_smithi194_121512_object47712:head# 2024-04-29T14:27:12.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6955aad1:::benchmark_data_smithi194_121512_object32906:head# 2024-04-29T14:27:12.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6955ec0e:::benchmark_data_smithi194_121512_object116100:head# 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695633d5:::benchmark_data_smithi194_121512_object50404:head# 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69565fbe:::benchmark_data_smithi194_121512_object56008:head# 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69568dfb:::benchmark_data_smithi194_121512_object125471:head# 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6956ade1:::benchmark_data_smithi194_121512_object11661:head# 2024-04-29T14:27:12.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.246 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69571020:::benchmark_data_smithi194_121512_object69428:head# 2024-04-29T14:27:12.246 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69574160:::benchmark_data_smithi194_121512_object66377:head# 2024-04-29T14:27:12.247 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.247 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69574cf7:::benchmark_data_smithi194_121512_object38482:head# 2024-04-29T14:27:12.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6957544c:::benchmark_data_smithi194_121512_object61485:head# 2024-04-29T14:27:12.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.248 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6957c07c:::benchmark_data_smithi194_121512_object59750:head# 2024-04-29T14:27:12.248 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6957fbf7:::benchmark_data_smithi194_121512_object60578:head# 2024-04-29T14:27:12.249 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695829a3:::benchmark_data_smithi194_121512_object84288:head# 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6958f6a1:::benchmark_data_smithi194_121512_object108541:head# 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69595938:::benchmark_data_smithi194_121512_object36677:head# 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69598030:::benchmark_data_smithi194_121512_object58498:head# 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69599895:::benchmark_data_smithi194_121512_object66567:head# 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695a4854:::benchmark_data_smithi194_121512_object124437:head# 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695a4acb:::benchmark_data_smithi194_121512_object130248:head# 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695aa78e:::benchmark_data_smithi194_121512_object130356:head# 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695b12db:::benchmark_data_smithi194_121512_object73594:head# 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695c8539:::benchmark_data_smithi194_121512_object44636:head# 2024-04-29T14:27:12.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695ccb7a:::benchmark_data_smithi194_121512_object93506:head# 2024-04-29T14:27:12.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695ede6b:::benchmark_data_smithi194_121512_object36283:head# 2024-04-29T14:27:12.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695f3ead:::benchmark_data_smithi194_121512_object112458:head# 2024-04-29T14:27:12.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695f49bf:::benchmark_data_smithi194_121512_object38264:head# 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695f6816:::benchmark_data_smithi194_121512_object120111:head# 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695f8308:::benchmark_data_smithi194_121512_object55379:head# 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:695fae80:::benchmark_data_smithi194_121512_object4778:head# 2024-04-29T14:27:12.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6960f89c:::benchmark_data_smithi194_121512_object49154:head# 2024-04-29T14:27:12.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.256 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6961943b:::benchmark_data_smithi194_121512_object6599:head# 2024-04-29T14:27:12.256 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6961a948:::benchmark_data_smithi194_121512_object112367:head# 2024-04-29T14:27:12.258 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69623feb:::benchmark_data_smithi194_121512_object29466:head# 2024-04-29T14:27:12.258 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.258 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6962aed6:::benchmark_data_smithi194_121512_object96627:head# 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69630bab:::benchmark_data_smithi194_121512_object127326:head# 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69632fe6:::benchmark_data_smithi194_121512_object112759:head# 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69633f30:::benchmark_data_smithi194_121512_object121878:head# 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69634cfc:::benchmark_data_smithi194_121512_object124701:head# 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696372d2:::benchmark_data_smithi194_121512_object25901:head# 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6963a59e:::benchmark_data_smithi194_121512_object31111:head# 2024-04-29T14:27:12.259 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6963a765:::benchmark_data_smithi194_121512_object129266:head# 2024-04-29T14:27:12.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6963b926:::benchmark_data_smithi194_121512_object86993:head# 2024-04-29T14:27:12.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6963d1e3:::benchmark_data_smithi194_121512_object130075:head# 2024-04-29T14:27:12.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69641a84:::benchmark_data_smithi194_121512_object102312:head# 2024-04-29T14:27:12.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69641ffd:::benchmark_data_smithi194_121512_object122440:head# 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6964228b:::benchmark_data_smithi194_121512_object73201:head# 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69661f4e:::benchmark_data_smithi194_121512_object118399:head# 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6966b985:::benchmark_data_smithi194_121512_object117669:head# 2024-04-29T14:27:12.265 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.266 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69670bcd:::benchmark_data_smithi194_121512_object8163:head# 2024-04-29T14:27:12.266 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69696407:::benchmark_data_smithi194_121512_object89606:head# 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69697615:::benchmark_data_smithi194_121512_object2488:head# 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69699617:::benchmark_data_smithi194_121512_object52465:head# 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6969cfaf:::benchmark_data_smithi194_121512_object28307:head# 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.267 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696a2204:::benchmark_data_smithi194_121512_object41795:head# 2024-04-29T14:27:12.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696a4a42:::benchmark_data_smithi194_121512_object79983:head# 2024-04-29T14:27:12.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.268 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696bc2f8:::benchmark_data_smithi194_121512_object105301:head# 2024-04-29T14:27:12.268 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.269 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696c2749:::benchmark_data_smithi194_121512_object92503:head# 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696c82aa:::benchmark_data_smithi194_121512_object117185:head# 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696e110f:::benchmark_data_smithi194_121512_object18116:head# 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696e210f:::benchmark_data_smithi194_121512_object97636:head# 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696e3055:::benchmark_data_smithi194_121512_object98115:head# 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696e9759:::benchmark_data_smithi194_121512_object60070:head# 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696ea061:::benchmark_data_smithi194_121512_object53850:head# 2024-04-29T14:27:12.270 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696ea568:::benchmark_data_smithi194_121512_object57182:head# 2024-04-29T14:27:12.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.271 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:696eee2f:::benchmark_data_smithi194_121512_object526:head# 2024-04-29T14:27:12.271 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697029c0:::benchmark_data_smithi194_121512_object99346:head# 2024-04-29T14:27:12.274 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69708bf1:::benchmark_data_smithi194_121512_object16562:head# 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69718488:::benchmark_data_smithi194_121512_object94619:head# 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69721609:::benchmark_data_smithi194_121512_object63011:head# 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69737bf8:::benchmark_data_smithi194_121512_object125136:head# 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697395d2:::benchmark_data_smithi194_121512_object75396:head# 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6974330e:::benchmark_data_smithi194_121512_object62037:head# 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697521e5:::benchmark_data_smithi194_121512_object122326:head# 2024-04-29T14:27:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6975b36f:::benchmark_data_smithi194_121512_object21367:head# 2024-04-29T14:27:12.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69767ca2:::benchmark_data_smithi194_121512_object5502:head# 2024-04-29T14:27:12.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6976be60:::benchmark_data_smithi194_121512_object42503:head# 2024-04-29T14:27:12.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6976c25a:::benchmark_data_smithi194_121512_object65618:head# 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6976fe3d:::benchmark_data_smithi194_121512_object97672:head# 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6976ff93:::benchmark_data_smithi194_121512_object90916:head# 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697714c6:::benchmark_data_smithi194_121512_object23309:head# 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69772412:::benchmark_data_smithi194_121512_object53977:head# 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69774739:::benchmark_data_smithi194_121512_object54200:head# 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6977921b:::benchmark_data_smithi194_121512_object67999:head# 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69789617:::benchmark_data_smithi194_121512_object129804:head# 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697907bc:::benchmark_data_smithi194_121512_object133073:head# 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69793aa6:::benchmark_data_smithi194_121512_object119265:head# 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6979ca15:::benchmark_data_smithi194_121512_object59960:head# 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6979df24:::benchmark_data_smithi194_121512_object9325:head# 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697a030c:::benchmark_data_smithi194_121512_object825:head# 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697a6bcc:::benchmark_data_smithi194_121512_object110500:head# 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697a82c8:::benchmark_data_smithi194_121512_object83985:head# 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697b56cc:::benchmark_data_smithi194_121512_object72187:head# 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697dac31:::benchmark_data_smithi194_121512_object30940:head# 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697e7004:::benchmark_data_smithi194_121512_object17666:head# 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697ea5f0:::benchmark_data_smithi194_121512_object26999:head# 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:697ed2f6:::benchmark_data_smithi194_121512_object108018:head# 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698022f3:::benchmark_data_smithi194_121512_object94872:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698031a5:::benchmark_data_smithi194_121512_object28225:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69805171:::benchmark_data_smithi194_121512_object33243:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6980cc12:::benchmark_data_smithi194_121512_object11583:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69812a5c:::benchmark_data_smithi194_121512_object96110:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69817ab6:::benchmark_data_smithi194_121512_object85166:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698194c2:::benchmark_data_smithi194_121512_object56412:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6982adff:::benchmark_data_smithi194_121512_object67715:head# 2024-04-29T14:27:12.293 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69836ff4:::benchmark_data_smithi194_121512_object76834:head# 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698384bc:::benchmark_data_smithi194_121512_object9970:head# 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6983fd28:::benchmark_data_smithi194_121512_object49624:head# 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69866302:::benchmark_data_smithi194_121512_object33762:head# 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69871372:::benchmark_data_smithi194_121512_object74028:head# 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69873246:::benchmark_data_smithi194_121512_object105981:head# 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6987b634:::benchmark_data_smithi194_121512_object117630:head# 2024-04-29T14:27:12.294 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6987c180:::benchmark_data_smithi194_121512_object75686:head# 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6987d83e:::benchmark_data_smithi194_121512_object115932:head# 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6987f261:::benchmark_data_smithi194_121512_object42556:head# 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698887c3:::benchmark_data_smithi194_121512_object7721:head# 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6988fa1a:::benchmark_data_smithi194_121512_object131721:head# 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69890dbc:::benchmark_data_smithi194_121512_object45509:head# 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69891577:::benchmark_data_smithi194_121512_object452:head# 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.295 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69899eba:::benchmark_data_smithi194_121512_object50177:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698a67ea:::benchmark_data_smithi194_121512_object112014:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698aa001:::benchmark_data_smithi194_121512_object111765:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698acf34:::benchmark_data_smithi194_121512_object105690:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698af620:::benchmark_data_smithi194_121512_object7101:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698b9987:::benchmark_data_smithi194_121512_object31670:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698c4e47:::benchmark_data_smithi194_121512_object132958:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698cc5e0:::benchmark_data_smithi194_121512_object16552:head# 2024-04-29T14:27:12.296 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.307 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698d125c:::benchmark_data_smithi194_121512_object70504:head# 2024-04-29T14:27:12.307 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698d2dc4:::benchmark_data_smithi194_121512_object104776:head# 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698d449b:::benchmark_data_smithi194_121512_object36607:head# 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698e0216:::benchmark_data_smithi194_121512_object60700:head# 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698e34d8:::benchmark_data_smithi194_121512_object82220:head# 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698e6c94:::benchmark_data_smithi194_121512_object117983:head# 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698e979c:::benchmark_data_smithi194_121512_object99790:head# 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698f11ba:::benchmark_data_smithi194_121512_object86297:head# 2024-04-29T14:27:12.308 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:698f9491:::benchmark_data_smithi194_121512_object74710:head# 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69906707:::benchmark_data_smithi194_121512_object65908:head# 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6990b75c:::benchmark_data_smithi194_121512_object2698:head# 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6990fccb:::benchmark_data_smithi194_121512_object109142:head# 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699135cb:::benchmark_data_smithi194_121512_object12297:head# 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69914c13:::benchmark_data_smithi194_121512_object75708:head# 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69914f3a:::benchmark_data_smithi194_121512_object55284:head# 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.309 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69917b1b:::benchmark_data_smithi194_121512_object109743:head# 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69926b27:::benchmark_data_smithi194_121512_object82674:head# 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699337ad:::benchmark_data_smithi194_121512_object6056:head# 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6993a9fa:::benchmark_data_smithi194_121512_object49701:head# 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6993b6ef:::benchmark_data_smithi194_121512_object111338:head# 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6993c02e:::benchmark_data_smithi194_121512_object7002:head# 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6993e684:::benchmark_data_smithi194_121512_object74376:head# 2024-04-29T14:27:12.310 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69945106:::benchmark_data_smithi194_121512_object50853:head# 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69946aba:::benchmark_data_smithi194_121512_object49325:head# 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6994bc93:::benchmark_data_smithi194_121512_object96656:head# 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6994da69:::benchmark_data_smithi194_121512_object3931:head# 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69950bb0:::benchmark_data_smithi194_121512_object122751:head# 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6995d3ba:::benchmark_data_smithi194_121512_object51928:head# 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6996773a:::benchmark_data_smithi194_121512_object93627:head# 2024-04-29T14:27:12.311 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6996a0b1:::benchmark_data_smithi194_121512_object89478:head# 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6996af80:::benchmark_data_smithi194_121512_object14947:head# 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6996b2c8:::benchmark_data_smithi194_121512_object123198:head# 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699760c2:::benchmark_data_smithi194_121512_object46322:head# 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6997f1c4:::benchmark_data_smithi194_121512_object28171:head# 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699818f6:::benchmark_data_smithi194_121512_object89536:head# 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699825ba:::benchmark_data_smithi194_121512_object11717:head# 2024-04-29T14:27:12.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6998e8bd:::benchmark_data_smithi194_121512_object68006:head# 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6999d818:::benchmark_data_smithi194_121512_object117072:head# 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699a14a8:::benchmark_data_smithi194_121512_object88581:head# 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699adf6c:::benchmark_data_smithi194_121512_object38980:head# 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699af20c:::benchmark_data_smithi194_121512_object61387:head# 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699b9d9a:::benchmark_data_smithi194_121512_object32904:head# 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699c323a:::benchmark_data_smithi194_121512_object37100:head# 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.313 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699cc2f9:::benchmark_data_smithi194_121512_object35767:head# 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699ce121:::benchmark_data_smithi194_121512_object93729:head# 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699d4f0c:::benchmark_data_smithi194_121512_object23809:head# 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699e0e94:::benchmark_data_smithi194_121512_object84656:head# 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699e37e5:::benchmark_data_smithi194_121512_object105936:head# 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:699f9f7f:::benchmark_data_smithi194_121512_object117973:head# 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a0101d:::benchmark_data_smithi194_121512_object86062:head# 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.314 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a092a8:::benchmark_data_smithi194_121512_object23520:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a168ef:::benchmark_data_smithi194_121512_object15120:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a1c52c:::benchmark_data_smithi194_121512_object37634:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a1c99b:::benchmark_data_smithi194_121512_object95557:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a2416c:::benchmark_data_smithi194_121512_object43697:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a30e56:::benchmark_data_smithi194_121512_object36621:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a3c6a2:::benchmark_data_smithi194_121512_object34922:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a4b8b4:::benchmark_data_smithi194_121512_object72913:head# 2024-04-29T14:27:12.315 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a6e256:::benchmark_data_smithi194_121512_object107508:head# 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a6ed90:::benchmark_data_smithi194_121512_object131580:head# 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a71c0d:::benchmark_data_smithi194_121512_object42619:head# 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a722f7:::benchmark_data_smithi194_121512_object24530:head# 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a73abe:::benchmark_data_smithi194_121512_object92779:head# 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a74302:::benchmark_data_smithi194_121512_object729:head# 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a88451:::benchmark_data_smithi194_121512_object40688:head# 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.316 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69a9eaba:::benchmark_data_smithi194_121512_object124353:head# 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ab3004:::benchmark_data_smithi194_121512_object116728:head# 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69aba0a6:::benchmark_data_smithi194_121512_object98178:head# 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ac2807:::benchmark_data_smithi194_121512_object56982:head# 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ac2b8c:::benchmark_data_smithi194_121512_object113466:head# 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69acb25c:::benchmark_data_smithi194_121512_object15373:head# 2024-04-29T14:27:12.317 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ad032c:::benchmark_data_smithi194_121512_object86077:head# 2024-04-29T14:27:12.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69adc710:::benchmark_data_smithi194_121512_object115619:head# 2024-04-29T14:27:12.324 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.324 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ae8aff:::benchmark_data_smithi194_121512_object51359:head# 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69af2ee1:::benchmark_data_smithi194_121512_object94562:head# 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69af3176:::benchmark_data_smithi194_121512_object127804:head# 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b16514:::benchmark_data_smithi194_121512_object98339:head# 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b178fd:::benchmark_data_smithi194_121512_object127974:head# 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b1907f:::benchmark_data_smithi194_121512_object82740:head# 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b1afa8:::benchmark_data_smithi194_121512_object25875:head# 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.325 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b23880:::benchmark_data_smithi194_121512_object25362:head# 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b2ee3d:::benchmark_data_smithi194_121512_object70875:head# 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b346df:::benchmark_data_smithi194_121512_object36416:head# 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b38841:::benchmark_data_smithi194_121512_object32965:head# 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b40d0d:::benchmark_data_smithi194_121512_object77397:head# 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b4602a:::benchmark_data_smithi194_121512_object118614:head# 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b4af60:::benchmark_data_smithi194_121512_object438:head# 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.326 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b4b345:::benchmark_data_smithi194_121512_object92464:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b4ca55:::benchmark_data_smithi194_121512_object45694:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b51401:::benchmark_data_smithi194_121512_object6238:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b723bc:::benchmark_data_smithi194_121512_object19056:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b7601e:::benchmark_data_smithi194_121512_object40574:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b7ac6d:::benchmark_data_smithi194_121512_object112622:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b83f92:::benchmark_data_smithi194_121512_object112026:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b855a6:::benchmark_data_smithi194_121512_object88139:head# 2024-04-29T14:27:12.327 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b9938f:::benchmark_data_smithi194_121512_object107950:head# 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69b9fecc:::benchmark_data_smithi194_121512_object22211:head# 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69baece7:::benchmark_data_smithi194_121512_object21293:head# 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69bc8e6d:::benchmark_data_smithi194_121512_object4857:head# 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69bcaf13:::benchmark_data_smithi194_121512_object112457:head# 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69bcfac3:::benchmark_data_smithi194_121512_object59391:head# 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69bd1c38:::benchmark_data_smithi194_121512_object48396:head# 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69bd76da:::benchmark_data_smithi194_121512_object41829:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69bf8745:::benchmark_data_smithi194_121512_object112231:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c01c03:::benchmark_data_smithi194_121512_object53226:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c0230c:::benchmark_data_smithi194_121512_object86475:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c10f7f:::benchmark_data_smithi194_121512_object81881:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c1533c:::benchmark_data_smithi194_121512_object10319:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c16279:::benchmark_data_smithi194_121512_object28087:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c18374:::benchmark_data_smithi194_121512_object129781:head# 2024-04-29T14:27:12.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c29348:::benchmark_data_smithi194_121512_object51914:head# 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c2f0d5:::benchmark_data_smithi194_121512_object79820:head# 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c3d1a7:::benchmark_data_smithi194_121512_object43642:head# 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c3d83f:::benchmark_data_smithi194_121512_object51268:head# 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c40f8c:::benchmark_data_smithi194_121512_object17030:head# 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c45bbf:::benchmark_data_smithi194_121512_object63287:head# 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c6e6c7:::benchmark_data_smithi194_121512_object35469:head# 2024-04-29T14:27:12.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c6fd98:::benchmark_data_smithi194_121512_object97484:head# 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c7657a:::benchmark_data_smithi194_121512_object111738:head# 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c795e7:::benchmark_data_smithi194_121512_object57125:head# 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c8ee1e:::benchmark_data_smithi194_121512_object127668:head# 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c93eb1:::benchmark_data_smithi194_121512_object112783:head# 2024-04-29T14:27:12.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c9478f:::benchmark_data_smithi194_121512_object57459:head# 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c9884e:::benchmark_data_smithi194_121512_object87022:head# 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69c9ed4c:::benchmark_data_smithi194_121512_object53772:head# 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ca2bd4:::benchmark_data_smithi194_121512_object73613:head# 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cb86f1:::benchmark_data_smithi194_121512_object107108:head# 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cbf7af:::benchmark_data_smithi194_121512_object13253:head# 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cc6bcb:::benchmark_data_smithi194_121512_object67471:head# 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cca570:::benchmark_data_smithi194_121512_object8493:head# 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cd8e47:::benchmark_data_smithi194_121512_object49586:head# 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cd8f97:::benchmark_data_smithi194_121512_object122498:head# 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cde3fa:::benchmark_data_smithi194_121512_object52167:head# 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ce3c67:::benchmark_data_smithi194_121512_object130232:head# 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ce5d7b:::benchmark_data_smithi194_121512_object118502:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ce7180:::benchmark_data_smithi194_121512_object105488:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ce922c:::benchmark_data_smithi194_121512_object24359:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69cfe0a1:::benchmark_data_smithi194_121512_object124924:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d1a088:::benchmark_data_smithi194_121512_object107205:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d1dc70:::benchmark_data_smithi194_121512_object18654:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d1e390:::benchmark_data_smithi194_121512_object64942:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d240b0:::benchmark_data_smithi194_121512_object79824:head# 2024-04-29T14:27:12.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d24d05:::benchmark_data_smithi194_121512_object107677:head# 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d28d90:::benchmark_data_smithi194_121512_object55685:head# 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d308de:::benchmark_data_smithi194_121512_object101487:head# 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d3603f:::benchmark_data_smithi194_121512_object101480:head# 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d3d788:::benchmark_data_smithi194_121512_object42495:head# 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d49ae7:::benchmark_data_smithi194_121512_object65934:head# 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d4c340:::benchmark_data_smithi194_121512_object50738:head# 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.338 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d54ea5:::benchmark_data_smithi194_121512_object25520:head# 2024-04-29T14:27:12.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d65e7d:::benchmark_data_smithi194_121512_object130968:head# 2024-04-29T14:27:12.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d6cff2:::benchmark_data_smithi194_121512_object38714:head# 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d6d333:::benchmark_data_smithi194_121512_object61346:head# 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d70909:::benchmark_data_smithi194_121512_object52589:head# 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d76366:::benchmark_data_smithi194_121512_object23726:head# 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d8a348:::benchmark_data_smithi194_121512_object16771:head# 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69d8df73:::benchmark_data_smithi194_121512_object25507:head# 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69da9765:::benchmark_data_smithi194_121512_object127055:head# 2024-04-29T14:27:12.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69daaedb:::benchmark_data_smithi194_121512_object24895:head# 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69dc4bcb:::benchmark_data_smithi194_121512_object21721:head# 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69dc6af4:::benchmark_data_smithi194_121512_object92431:head# 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69dc7370:::benchmark_data_smithi194_121512_object76006:head# 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69dca1d2:::benchmark_data_smithi194_121512_object89560:head# 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69dcc360:::benchmark_data_smithi194_121512_object63396:head# 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69dd56d0:::benchmark_data_smithi194_121512_object131770:head# 2024-04-29T14:27:12.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ded09a:::benchmark_data_smithi194_121512_object116705:head# 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69df0409:::benchmark_data_smithi194_121512_object83366:head# 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69dff002:::benchmark_data_smithi194_121512_object34904:head# 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e04d9b:::benchmark_data_smithi194_121512_object39814:head# 2024-04-29T14:27:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e094f4:::benchmark_data_smithi194_121512_object39092:head# 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e14d8b:::benchmark_data_smithi194_121512_object93759:head# 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e35f17:::benchmark_data_smithi194_121512_object105677:head# 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e3cd4d:::benchmark_data_smithi194_121512_object116371:head# 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e48885:::benchmark_data_smithi194_121512_object130734:head# 2024-04-29T14:27:12.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e48a62:::benchmark_data_smithi194_121512_object123893:head# 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e4c348:::benchmark_data_smithi194_121512_object131745:head# 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e545cd:::benchmark_data_smithi194_121512_object99142:head# 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e5556e:::benchmark_data_smithi194_121512_object107550:head# 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e5a496:::benchmark_data_smithi194_121512_object71593:head# 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e5e7aa:::benchmark_data_smithi194_121512_object115020:head# 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e686fc:::benchmark_data_smithi194_121512_object56764:head# 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.355 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e72fab:::benchmark_data_smithi194_121512_object131097:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69e84d9d:::benchmark_data_smithi194_121512_object55917:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ea7af3:::benchmark_data_smithi194_121512_object75459:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69eb1098:::benchmark_data_smithi194_121512_object84194:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69eb7157:::benchmark_data_smithi194_121512_object24543:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69edc3ad:::benchmark_data_smithi194_121512_object16725:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69eddf95:::benchmark_data_smithi194_121512_object61837:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69eead49:::benchmark_data_smithi194_121512_object66732:head# 2024-04-29T14:27:12.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69eeb9aa:::benchmark_data_smithi194_121512_object15347:head# 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ef008c:::benchmark_data_smithi194_121512_object82752:head# 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ef2787:::benchmark_data_smithi194_121512_object26478:head# 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ef839b:::benchmark_data_smithi194_121512_object49806:head# 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f00030:::benchmark_data_smithi194_121512_object51642:head# 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f03b63:::benchmark_data_smithi194_121512_object7466:head# 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f08745:::benchmark_data_smithi194_121512_object95139:head# 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f0c268:::benchmark_data_smithi194_121512_object107497:head# 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f10abf:::benchmark_data_smithi194_121512_object81929:head# 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f12dcc:::benchmark_data_smithi194_121512_object103030:head# 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f148fe:::benchmark_data_smithi194_121512_object120057:head# 2024-04-29T14:27:12.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3737555475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: pgmap v1275: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 654 op/s; 32234/205572 objects degraded (15.680%); 13 MiB/s, 208 objects/s recovering 2024-04-29T14:27:12.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:12 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 32988/210258 objects degraded (15.689%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:12.364 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f1dfc8:::benchmark_data_smithi194_121512_object54303:head# 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f1ecaa:::benchmark_data_smithi194_121512_object97968:head# 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f25ea6:::benchmark_data_smithi194_121512_object56818:head# 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f27dfe:::benchmark_data_smithi194_121512_object78405:head# 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f36716:::benchmark_data_smithi194_121512_object36878:head# 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f3699a:::benchmark_data_smithi194_121512_object101875:head# 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f3d60a:::benchmark_data_smithi194_121512_object117418:head# 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f448f2:::benchmark_data_smithi194_121512_object62939:head# 2024-04-29T14:27:12.366 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.367 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f45889:::benchmark_data_smithi194_121512_object53557:head# 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f4b993:::benchmark_data_smithi194_121512_object89771:head# 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f576d1:::benchmark_data_smithi194_121512_object51381:head# 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f5f528:::benchmark_data_smithi194_121512_object18024:head# 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f6867e:::benchmark_data_smithi194_121512_object128743:head# 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f6cd69:::benchmark_data_smithi194_121512_object93350:head# 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f6f95b:::benchmark_data_smithi194_121512_object101907:head# 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.368 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f70f09:::benchmark_data_smithi194_121512_object61751:head# 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f72ab3:::benchmark_data_smithi194_121512_object127464:head# 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f76e18:::benchmark_data_smithi194_121512_object26472:head# 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69f94bb2:::benchmark_data_smithi194_121512_object52027:head# 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fa1907:::benchmark_data_smithi194_121512_object130600:head# 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fa74aa:::benchmark_data_smithi194_121512_object23259:head# 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fa9ada:::benchmark_data_smithi194_121512_object25697:head# 2024-04-29T14:27:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fb0b61:::benchmark_data_smithi194_121512_object114617:head# 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fb9d4b:::benchmark_data_smithi194_121512_object19858:head# 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fc309f:::benchmark_data_smithi194_121512_object28981:head# 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fd8923:::benchmark_data_smithi194_121512_object62967:head# 2024-04-29T14:27:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fda80e:::benchmark_data_smithi194_121512_object46669:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fde4e6:::benchmark_data_smithi194_121512_object91011:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fde89f:::benchmark_data_smithi194_121512_object129698:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fe1b05:::benchmark_data_smithi194_121512_object68802:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69fe537a:::benchmark_data_smithi194_121512_object43120:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ff0353:::benchmark_data_smithi194_121512_object100437:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ff62a6:::benchmark_data_smithi194_121512_object76531:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ff7467:::benchmark_data_smithi194_121512_object69083:head# 2024-04-29T14:27:12.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:69ffd8c1:::benchmark_data_smithi194_121512_object12257:head# 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a00087a:::benchmark_data_smithi194_121512_object91401:head# 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a01b382:::benchmark_data_smithi194_121512_object50843:head# 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a01f559:::benchmark_data_smithi194_121512_object29987:head# 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a021516:::benchmark_data_smithi194_121512_object13126:head# 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a02db9f:::benchmark_data_smithi194_121512_object132206:head# 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a02e1b9:::benchmark_data_smithi194_121512_object53542:head# 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a031151:::benchmark_data_smithi194_121512_object23396:head# 2024-04-29T14:27:12.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a03819a:::benchmark_data_smithi194_121512_object96793:head# 2024-04-29T14:27:12.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.376 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a045cc4:::benchmark_data_smithi194_121512_object9178:head# 2024-04-29T14:27:12.376 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a045cfb:::benchmark_data_smithi194_121512_object49143:head# 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a046fb5:::benchmark_data_smithi194_121512_object69696:head# 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0496fc:::benchmark_data_smithi194_121512_object6030:head# 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a062ca2:::benchmark_data_smithi194_121512_object111160:head# 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a065d16:::benchmark_data_smithi194_121512_object81183:head# 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a066111:::benchmark_data_smithi194_121512_object96144:head# 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0718a0:::benchmark_data_smithi194_121512_object121786:head# 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a07f663:::benchmark_data_smithi194_121512_object28621:head# 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a08a0fb:::benchmark_data_smithi194_121512_object15274:head# 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a093ebe:::benchmark_data_smithi194_121512_object29143:head# 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a095379:::benchmark_data_smithi194_121512_object82812:head# 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0ac44b:::benchmark_data_smithi194_121512_object7594:head# 2024-04-29T14:27:12.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0b1294:::benchmark_data_smithi194_121512_object75051:head# 2024-04-29T14:27:12.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0bda7a:::benchmark_data_smithi194_121512_object128465:head# 2024-04-29T14:27:12.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0bf29e:::benchmark_data_smithi194_121512_object85053:head# 2024-04-29T14:27:12.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0c7562:::benchmark_data_smithi194_121512_object43789:head# 2024-04-29T14:27:12.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0cff7b:::benchmark_data_smithi194_121512_object75502:head# 2024-04-29T14:27:12.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0d4657:::benchmark_data_smithi194_121512_object86780:head# 2024-04-29T14:27:12.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0e1d7c:::benchmark_data_smithi194_121512_object77134:head# 2024-04-29T14:27:12.384 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.384 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a0f459c:::benchmark_data_smithi194_121512_object117781:head# 2024-04-29T14:27:12.384 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.384 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a102b4c:::benchmark_data_smithi194_121512_object77316:head# 2024-04-29T14:27:12.384 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.384 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a10dad8:::benchmark_data_smithi194_121512_object67269:head# 2024-04-29T14:27:12.384 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a10f9b4:::benchmark_data_smithi194_121512_object95813:head# 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a113c1c:::benchmark_data_smithi194_121512_object90082:head# 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a125c2e:::benchmark_data_smithi194_121512_object123896:head# 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a149ed3:::benchmark_data_smithi194_121512_object64233:head# 2024-04-29T14:27:12.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a14c209:::benchmark_data_smithi194_121512_object48498:head# 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a162b47:::benchmark_data_smithi194_121512_object65171:head# 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a179ea9:::benchmark_data_smithi194_121512_object95969:head# 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a191f89:::benchmark_data_smithi194_121512_object40467:head# 2024-04-29T14:27:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a19c4a3:::benchmark_data_smithi194_121512_object83389:head# 2024-04-29T14:27:12.387 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.387 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a19e3b7:::benchmark_data_smithi194_121512_object57926:head# 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a19f8a5:::benchmark_data_smithi194_121512_object72087:head# 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a1a6f13:::benchmark_data_smithi194_121512_object124712:head# 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a1a8970:::benchmark_data_smithi194_121512_object81372:head# 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a1aaf45:::benchmark_data_smithi194_121512_object33194:head# 2024-04-29T14:27:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a1bf9c8:::benchmark_data_smithi194_121512_object34765:head# 2024-04-29T14:27:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a1c60bb:::benchmark_data_smithi194_121512_object13710:head# 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a1e85c0:::benchmark_data_smithi194_121512_object85329:head# 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a1faa06:::benchmark_data_smithi194_121512_object81225:head# 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a202372:::benchmark_data_smithi194_121512_object119798:head# 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a20d099:::benchmark_data_smithi194_121512_object28611:head# 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a210b84:::benchmark_data_smithi194_121512_object37959:head# 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a238729:::benchmark_data_smithi194_121512_object14001:head# 2024-04-29T14:27:12.390 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2460da:::benchmark_data_smithi194_121512_object86071:head# 2024-04-29T14:27:12.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a24756f:::benchmark_data_smithi194_121512_object18279:head# 2024-04-29T14:27:12.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a24cb15:::benchmark_data_smithi194_121512_object67098:head# 2024-04-29T14:27:12.391 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.391 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a25034e:::benchmark_data_smithi194_121512_object101673:head# 2024-04-29T14:27:12.392 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a25103c:::benchmark_data_smithi194_121512_object112978:head# 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a252dd2:::benchmark_data_smithi194_121512_object131045:head# 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a25e2e0:::benchmark_data_smithi194_121512_object27197:head# 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a266206:::benchmark_data_smithi194_121512_object29062:head# 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a26cf17:::benchmark_data_smithi194_121512_object119301:head# 2024-04-29T14:27:12.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.398 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a277437:::benchmark_data_smithi194_121512_object56320:head# 2024-04-29T14:27:12.398 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2788a2:::benchmark_data_smithi194_121512_object75657:head# 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a27f025:::benchmark_data_smithi194_121512_object124568:head# 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a281237:::benchmark_data_smithi194_121512_object40823:head# 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a296406:::benchmark_data_smithi194_121512_object114922:head# 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a298303:::benchmark_data_smithi194_121512_object110078:head# 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2a936f:::benchmark_data_smithi194_121512_object100432:head# 2024-04-29T14:27:12.399 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.400 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2acb92:::benchmark_data_smithi194_121512_object93362:head# 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2aeb9d:::benchmark_data_smithi194_121512_object98669:head# 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2e33cc:::benchmark_data_smithi194_121512_object101468:head# 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2e52b4:::benchmark_data_smithi194_121512_object70622:head# 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2e8a1a:::benchmark_data_smithi194_121512_object1283:head# 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2ea4d6:::benchmark_data_smithi194_121512_object87372:head# 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2f00b7:::benchmark_data_smithi194_121512_object125286:head# 2024-04-29T14:27:12.401 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2f0500:::benchmark_data_smithi194_121512_object22850:head# 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2f362a:::benchmark_data_smithi194_121512_object79948:head# 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2f55b8:::benchmark_data_smithi194_121512_object112451:head# 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a2ffed9:::benchmark_data_smithi194_121512_object91473:head# 2024-04-29T14:27:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a304ce9:::benchmark_data_smithi194_121512_object129603:head# 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3130a6:::benchmark_data_smithi194_121512_object26315:head# 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a317204:::benchmark_data_smithi194_121512_object59595:head# 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a31d001:::benchmark_data_smithi194_121512_object113664:head# 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a328fc0:::benchmark_data_smithi194_121512_object83348:head# 2024-04-29T14:27:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a33d524:::benchmark_data_smithi194_121512_object28121:head# 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a33fe14:::benchmark_data_smithi194_121512_object34365:head# 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a345605:::benchmark_data_smithi194_121512_object44969:head# 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a34a455:::benchmark_data_smithi194_121512_object63109:head# 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a35d902:::benchmark_data_smithi194_121512_object8886:head# 2024-04-29T14:27:12.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a371125:::benchmark_data_smithi194_121512_object110743:head# 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3748d1:::benchmark_data_smithi194_121512_object54612:head# 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a376001:::benchmark_data_smithi194_121512_object112935:head# 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a376b42:::benchmark_data_smithi194_121512_object83542:head# 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a37dc38:::benchmark_data_smithi194_121512_object33031:head# 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a38ab8e:::benchmark_data_smithi194_121512_object73570:head# 2024-04-29T14:27:12.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a396037:::benchmark_data_smithi194_121512_object119111:head# 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3a0981:::benchmark_data_smithi194_121512_object63007:head# 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3a4017:::benchmark_data_smithi194_121512_object80527:head# 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3a76c9:::benchmark_data_smithi194_121512_object99203:head# 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3adc5d:::benchmark_data_smithi194_121512_object38867:head# 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3b3283:::benchmark_data_smithi194_121512_object61092:head# 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3b7095:::benchmark_data_smithi194_121512_object68884:head# 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3b7c0a:::benchmark_data_smithi194_121512_object39773:head# 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3b9220:::benchmark_data_smithi194_121512_object86776:head# 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3c3f5c:::benchmark_data_smithi194_121512_object84941:head# 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3c4b16:::benchmark_data_smithi194_121512_object15884:head# 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3dfcfc:::benchmark_data_smithi194_121512_object132431:head# 2024-04-29T14:27:12.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3e9b84:::benchmark_data_smithi194_121512_object92944:head# 2024-04-29T14:27:12.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3f75e5:::benchmark_data_smithi194_121512_object50546:head# 2024-04-29T14:27:12.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3fabe7:::benchmark_data_smithi194_121512_object132256:head# 2024-04-29T14:27:12.415 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.415 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a3ff874:::benchmark_data_smithi194_121512_object71882:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a404987:::benchmark_data_smithi194_121512_object105834:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a405a77:::benchmark_data_smithi194_121512_object88287:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4092ee:::benchmark_data_smithi194_121512_object27676:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a41d63c:::benchmark_data_smithi194_121512_object49251:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a42892e:::benchmark_data_smithi194_121512_object22612:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a42b731:::benchmark_data_smithi194_121512_object76403:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a42e7be:::benchmark_data_smithi194_121512_object107523:head# 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.416 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4393c3:::benchmark_data_smithi194_121512_object56261:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a446b59:::benchmark_data_smithi194_121512_object14085:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4575aa:::benchmark_data_smithi194_121512_object112992:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4584e5:::benchmark_data_smithi194_121512_object39198:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a467438:::benchmark_data_smithi194_121512_object87652:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a46b422:::benchmark_data_smithi194_121512_object30809:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a472107:::benchmark_data_smithi194_121512_object50446:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a475946:::benchmark_data_smithi194_121512_object54103:head# 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.417 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a482edf:::benchmark_data_smithi194_121512_object72378:head# 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a487e78:::benchmark_data_smithi194_121512_object72509:head# 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a49a91b:::benchmark_data_smithi194_121512_object91744:head# 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a49aeaf:::benchmark_data_smithi194_121512_object117118:head# 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4a3aee:::benchmark_data_smithi194_121512_object102031:head# 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4c1c4e:::benchmark_data_smithi194_121512_object108769:head# 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4ce0ea:::benchmark_data_smithi194_121512_object116760:head# 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4cf9f5:::benchmark_data_smithi194_121512_object38652:head# 2024-04-29T14:27:12.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4d1b20:::benchmark_data_smithi194_121512_object17586:head# 2024-04-29T14:27:12.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4d490c:::benchmark_data_smithi194_121512_object68201:head# 2024-04-29T14:27:12.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4dff7f:::benchmark_data_smithi194_121512_object10793:head# 2024-04-29T14:27:12.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4e9305:::benchmark_data_smithi194_121512_object12047:head# 2024-04-29T14:27:12.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a4ef087:::benchmark_data_smithi194_121512_object43453:head# 2024-04-29T14:27:12.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a51896d:::benchmark_data_smithi194_121512_object115072:head# 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a528204:::benchmark_data_smithi194_121512_object106646:head# 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a52bfec:::benchmark_data_smithi194_121512_object54487:head# 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a53302e:::benchmark_data_smithi194_121512_object29403:head# 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a535a77:::benchmark_data_smithi194_121512_object125804:head# 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a535aa6:::benchmark_data_smithi194_121512_object42154:head# 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a53a9e1:::benchmark_data_smithi194_121512_object78428:head# 2024-04-29T14:27:12.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a53b7ce:::benchmark_data_smithi194_121512_object111220:head# 2024-04-29T14:27:12.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.423 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a54678c:::benchmark_data_smithi194_121512_object32331:head# 2024-04-29T14:27:12.423 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:12.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a548603:::benchmark_data_smithi194_121512_object36713:head# 2024-04-29T14:27:12.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5527aa:::benchmark_data_smithi194_121512_object27411:head# 2024-04-29T14:27:12.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.427 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a55a706:::benchmark_data_smithi194_121512_object58305:head# 2024-04-29T14:27:12.427 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a55d5b3:::benchmark_data_smithi194_121512_object73979:head# 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a55da94:::benchmark_data_smithi194_121512_object9248:head# 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a55db1d:::benchmark_data_smithi194_121512_object50629:head# 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a560494:::benchmark_data_smithi194_121512_object106908:head# 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a587580:::benchmark_data_smithi194_121512_object109263:head# 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a589ef0:::benchmark_data_smithi194_121512_object103135:head# 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5a6102:::benchmark_data_smithi194_121512_object82352:head# 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5ac6b7:::benchmark_data_smithi194_121512_object86314:head# 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5bc10f:::benchmark_data_smithi194_121512_object43956:head# 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5c3634:::benchmark_data_smithi194_121512_object51379:head# 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5cd209:::benchmark_data_smithi194_121512_object16245:head# 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5ce116:::benchmark_data_smithi194_121512_object25737:head# 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5d78a0:::benchmark_data_smithi194_121512_object83172:head# 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5da603:::benchmark_data_smithi194_121512_object132249:head# 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5dd3b7:::benchmark_data_smithi194_121512_object47025:head# 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5f0871:::benchmark_data_smithi194_121512_object60112:head# 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a5f08fd:::benchmark_data_smithi194_121512_object124059:head# 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a60eb14:::benchmark_data_smithi194_121512_object33297:head# 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a611576:::benchmark_data_smithi194_121512_object24892:head# 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6188e5:::benchmark_data_smithi194_121512_object69953:head# 2024-04-29T14:27:12.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a61f74f:::benchmark_data_smithi194_121512_object28340:head# 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a621d64:::benchmark_data_smithi194_121512_object92435:head# 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a631950:::benchmark_data_smithi194_121512_object61692:head# 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a63550d:::benchmark_data_smithi194_121512_object20181:head# 2024-04-29T14:27:12.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:12.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6355da:::benchmark_data_smithi194_121512_object95040:head# 2024-04-29T14:27:12.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a63703f:::benchmark_data_smithi194_121512_object10195:head# 2024-04-29T14:27:12.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6372b8:::benchmark_data_smithi194_121512_object89115:head# 2024-04-29T14:27:12.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.437 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a63ad88:::benchmark_data_smithi194_121512_object69591:head# 2024-04-29T14:27:12.437 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a64320a:::benchmark_data_smithi194_121512_object62474:head# 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6478ff:::benchmark_data_smithi194_121512_object31635:head# 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a656a5f:::benchmark_data_smithi194_121512_object84767:head# 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a664385:::benchmark_data_smithi194_121512_object121435:head# 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a667a9c:::benchmark_data_smithi194_121512_object15208:head# 2024-04-29T14:27:12.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a66fcc8:::benchmark_data_smithi194_121512_object83075:head# 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a67427d:::benchmark_data_smithi194_121512_object98181:head# 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a67c9bd:::benchmark_data_smithi194_121512_object11982:head# 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a68128d:::benchmark_data_smithi194_121512_object31560:head# 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a684cf0:::benchmark_data_smithi194_121512_object123280:head# 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6855c9:::benchmark_data_smithi194_121512_object108439:head# 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a69daf1:::benchmark_data_smithi194_121512_object88232:head# 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6b5219:::benchmark_data_smithi194_121512_object15677:head# 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6b6757:::benchmark_data_smithi194_121512_object32248:head# 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6c126a:::benchmark_data_smithi194_121512_object10433:head# 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6d8722:::benchmark_data_smithi194_121512_object111183:head# 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6eb84a:::benchmark_data_smithi194_121512_object47135:head# 2024-04-29T14:27:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.443 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a6fb0ad:::benchmark_data_smithi194_121512_object79949:head# 2024-04-29T14:27:12.443 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.444 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7028b2:::benchmark_data_smithi194_121512_object42205:head# 2024-04-29T14:27:12.444 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a70466c:::benchmark_data_smithi194_121512_object31126:head# 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a706350:::benchmark_data_smithi194_121512_object43239:head# 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a707ac5:::benchmark_data_smithi194_121512_object90784:head# 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a710789:::benchmark_data_smithi194_121512_object26890:head# 2024-04-29T14:27:12.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7290ba:::benchmark_data_smithi194_121512_object37291:head# 2024-04-29T14:27:12.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a72c5ce:::benchmark_data_smithi194_121512_object105877:head# 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a73500d:::benchmark_data_smithi194_121512_object25608:head# 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7375c7:::benchmark_data_smithi194_121512_object110298:head# 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a73a718:::benchmark_data_smithi194_121512_object43984:head# 2024-04-29T14:27:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a744091:::benchmark_data_smithi194_121512_object104256:head# 2024-04-29T14:27:12.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a74b253:::benchmark_data_smithi194_121512_object58550:head# 2024-04-29T14:27:12.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a756640:::benchmark_data_smithi194_121512_object23299:head# 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a758083:::benchmark_data_smithi194_121512_object98890:head# 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a75ff19:::benchmark_data_smithi194_121512_object5110:head# 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7626b8:::benchmark_data_smithi194_121512_object125554:head# 2024-04-29T14:27:12.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a76419b:::benchmark_data_smithi194_121512_object25990:head# 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7662c8:::benchmark_data_smithi194_121512_object111479:head# 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7668c9:::benchmark_data_smithi194_121512_object117681:head# 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a76b06e:::benchmark_data_smithi194_121512_object108002:head# 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a770e65:::benchmark_data_smithi194_121512_object36390:head# 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a77ee48:::benchmark_data_smithi194_121512_object126883:head# 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a781d03:::benchmark_data_smithi194_121512_object6784:head# 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a783140:::benchmark_data_smithi194_121512_object258:head# 2024-04-29T14:27:12.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.458 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a796406:::benchmark_data_smithi194_121512_object27692:head# 2024-04-29T14:27:12.458 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7972c8:::benchmark_data_smithi194_121512_object21374:head# 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a797a93:::benchmark_data_smithi194_121512_object10344:head# 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a79becb:::benchmark_data_smithi194_121512_object50022:head# 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a79ee95:::benchmark_data_smithi194_121512_object121221:head# 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7ac85d:::benchmark_data_smithi194_121512_object17790:head# 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7aef8f:::benchmark_data_smithi194_121512_object104434:head# 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7b1721:::benchmark_data_smithi194_121512_object29373:head# 2024-04-29T14:27:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7bb2d0:::benchmark_data_smithi194_121512_object81259:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7d14ce:::benchmark_data_smithi194_121512_object64819:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7d1578:::benchmark_data_smithi194_121512_object193:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7d5573:::benchmark_data_smithi194_121512_object13836:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7dcc93:::benchmark_data_smithi194_121512_object97757:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7deceb:::benchmark_data_smithi194_121512_object132510:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7ed80d:::benchmark_data_smithi194_121512_object38838:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7f0fe5:::benchmark_data_smithi194_121512_object60318:head# 2024-04-29T14:27:12.464 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a7f643e:::benchmark_data_smithi194_121512_object72662:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a808a1a:::benchmark_data_smithi194_121512_object119777:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a80d81c:::benchmark_data_smithi194_121512_object100427:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a80dde3:::benchmark_data_smithi194_121512_object22818:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a815b9a:::benchmark_data_smithi194_121512_object130129:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a817652:::benchmark_data_smithi194_121512_object53568:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a817e6e:::benchmark_data_smithi194_121512_object10696:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8278cb:::benchmark_data_smithi194_121512_object89954:head# 2024-04-29T14:27:12.465 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a830fcd:::benchmark_data_smithi194_121512_object17054:head# 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a832c55:::benchmark_data_smithi194_121512_object20943:head# 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a83d243:::benchmark_data_smithi194_121512_object40719:head# 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a840493:::benchmark_data_smithi194_121512_object80973:head# 2024-04-29T14:27:12.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a84565e:::benchmark_data_smithi194_121512_object117495:head# 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a84d1b8:::benchmark_data_smithi194_121512_object4091:head# 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a855e68:::benchmark_data_smithi194_121512_object117771:head# 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a860c82:::benchmark_data_smithi194_121512_object88842:head# 2024-04-29T14:27:12.467 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.468 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a86a7d7:::benchmark_data_smithi194_121512_object126162:head# 2024-04-29T14:27:12.468 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a86dfbd:::benchmark_data_smithi194_121512_object35534:head# 2024-04-29T14:27:12.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.471 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a877118:::benchmark_data_smithi194_121512_object54681:head# 2024-04-29T14:27:12.471 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a87ac54:::benchmark_data_smithi194_121512_object27502:head# 2024-04-29T14:27:12.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a880624:::benchmark_data_smithi194_121512_object39385:head# 2024-04-29T14:27:12.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a88668c:::benchmark_data_smithi194_121512_object46804:head# 2024-04-29T14:27:12.472 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.472 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a888363:::benchmark_data_smithi194_121512_object51607:head# 2024-04-29T14:27:12.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a89bb42:::benchmark_data_smithi194_121512_object57649:head# 2024-04-29T14:27:12.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8a38b0:::benchmark_data_smithi194_121512_object118992:head# 2024-04-29T14:27:12.473 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.473 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8a439b:::benchmark_data_smithi194_121512_object73408:head# 2024-04-29T14:27:12.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8a6fea:::benchmark_data_smithi194_121512_object78838:head# 2024-04-29T14:27:12.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8acedd:::benchmark_data_smithi194_121512_object69827:head# 2024-04-29T14:27:12.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8ad396:::benchmark_data_smithi194_121512_object93640:head# 2024-04-29T14:27:12.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.478 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8b1d46:::benchmark_data_smithi194_121512_object90863:head# 2024-04-29T14:27:12.478 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.479 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8b52eb:::benchmark_data_smithi194_121512_object4995:head# 2024-04-29T14:27:12.479 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8b68b4:::benchmark_data_smithi194_121512_object127772:head# 2024-04-29T14:27:12.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8bc701:::benchmark_data_smithi194_121512_object14092:head# 2024-04-29T14:27:12.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.481 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8c020f:::benchmark_data_smithi194_121512_object35400:head# 2024-04-29T14:27:12.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.482 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8c1d5e:::benchmark_data_smithi194_121512_object33435:head# 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8c4f91:::benchmark_data_smithi194_121512_object111359:head# 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8c7dee:::benchmark_data_smithi194_121512_object125372:head# 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8cd0a5:::benchmark_data_smithi194_121512_object2960:head# 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8d0878:::benchmark_data_smithi194_121512_object55247:head# 2024-04-29T14:27:12.483 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.484 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8d100c:::benchmark_data_smithi194_121512_object109795:head# 2024-04-29T14:27:12.484 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8d1251:::benchmark_data_smithi194_121512_object133407:head# 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8d822d:::benchmark_data_smithi194_121512_object123296:head# 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8dee64:::benchmark_data_smithi194_121512_object126050:head# 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8eb5fd:::benchmark_data_smithi194_121512_object24107:head# 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8f02db:::benchmark_data_smithi194_121512_object66722:head# 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8f0dc5:::benchmark_data_smithi194_121512_object121621:head# 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8f2458:::benchmark_data_smithi194_121512_object129347:head# 2024-04-29T14:27:12.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a8f76ba:::benchmark_data_smithi194_121512_object56842:head# 2024-04-29T14:27:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a907b21:::benchmark_data_smithi194_121512_object64613:head# 2024-04-29T14:27:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9213f3:::benchmark_data_smithi194_121512_object43754:head# 2024-04-29T14:27:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a92ad14:::benchmark_data_smithi194_121512_object44811:head# 2024-04-29T14:27:12.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a92fe50:::benchmark_data_smithi194_121512_object733:head# 2024-04-29T14:27:12.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a934abb:::benchmark_data_smithi194_121512_object14241:head# 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a93adc8:::benchmark_data_smithi194_121512_object129717:head# 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a941fa1:::benchmark_data_smithi194_121512_object81419:head# 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a944fcf:::benchmark_data_smithi194_121512_object112850:head# 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a94d157:::benchmark_data_smithi194_121512_object66255:head# 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9535dd:::benchmark_data_smithi194_121512_object74410:head# 2024-04-29T14:27:12.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a958d8f:::benchmark_data_smithi194_121512_object73459:head# 2024-04-29T14:27:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a95af7c:::benchmark_data_smithi194_121512_object43355:head# 2024-04-29T14:27:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a95d1ca:::benchmark_data_smithi194_121512_object83830:head# 2024-04-29T14:27:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9623d0:::benchmark_data_smithi194_121512_object84772:head# 2024-04-29T14:27:12.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a976682:::benchmark_data_smithi194_121512_object5583:head# 2024-04-29T14:27:12.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a983f76:::benchmark_data_smithi194_121512_object63692:head# 2024-04-29T14:27:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a98c55a:::benchmark_data_smithi194_121512_object15784:head# 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a98eafa:::benchmark_data_smithi194_121512_object101699:head# 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9977dc:::benchmark_data_smithi194_121512_object65156:head# 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9a20dd:::benchmark_data_smithi194_121512_object57697:head# 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9a4caa:::benchmark_data_smithi194_121512_object74195:head# 2024-04-29T14:27:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.495 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9a6489:::benchmark_data_smithi194_121512_object29847:head# 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9ac548:::benchmark_data_smithi194_121512_object56452:head# 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9acda9:::benchmark_data_smithi194_121512_object10411:head# 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9b0fc4:::benchmark_data_smithi194_121512_object83686:head# 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9b78e6:::benchmark_data_smithi194_121512_object104166:head# 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9bc6f2:::benchmark_data_smithi194_121512_object90939:head# 2024-04-29T14:27:12.496 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9c107d:::benchmark_data_smithi194_121512_object114473:head# 2024-04-29T14:27:12.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9c5c8a:::benchmark_data_smithi194_121512_object49630:head# 2024-04-29T14:27:12.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9c9f05:::benchmark_data_smithi194_121512_object128582:head# 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9cd198:::benchmark_data_smithi194_121512_object89152:head# 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9d1d2c:::benchmark_data_smithi194_121512_object80032:head# 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9d34b3:::benchmark_data_smithi194_121512_object119537:head# 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9d5900:::benchmark_data_smithi194_121512_object13056:head# 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9e5319:::benchmark_data_smithi194_121512_object76537:head# 2024-04-29T14:27:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9e73cc:::benchmark_data_smithi194_121512_object53353:head# 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9e9242:::benchmark_data_smithi194_121512_object82702:head# 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9edf6e:::benchmark_data_smithi194_121512_object78195:head# 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9f3873:::benchmark_data_smithi194_121512_object68200:head# 2024-04-29T14:27:12.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6a9f4573:::benchmark_data_smithi194_121512_object101064:head# 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa0fbb3:::benchmark_data_smithi194_121512_object89950:head# 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa11577:::benchmark_data_smithi194_121512_object101603:head# 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa17856:::benchmark_data_smithi194_121512_object125825:head# 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa1e478:::benchmark_data_smithi194_121512_object83390:head# 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa26633:::benchmark_data_smithi194_121512_object109182:head# 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa31590:::benchmark_data_smithi194_121512_object57464:head# 2024-04-29T14:27:12.501 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa3e447:::benchmark_data_smithi194_121512_object118466:head# 2024-04-29T14:27:12.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa41180:::benchmark_data_smithi194_121512_object11511:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa4e950:::benchmark_data_smithi194_121512_object114633:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa557d1:::benchmark_data_smithi194_121512_object13444:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa599ca:::benchmark_data_smithi194_121512_object61765:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa67628:::benchmark_data_smithi194_121512_object17795:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa69cf2:::benchmark_data_smithi194_121512_object52146:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa6a1f1:::benchmark_data_smithi194_121512_object125159:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa7a161:::benchmark_data_smithi194_121512_object84052:head# 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa822d2:::benchmark_data_smithi194_121512_object13296:head# 2024-04-29T14:27:12.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa8a43c:::benchmark_data_smithi194_121512_object45791:head# 2024-04-29T14:27:12.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aa8c853:::benchmark_data_smithi194_121512_object46202:head# 2024-04-29T14:27:12.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aaa012d:::benchmark_data_smithi194_121512_object49563:head# 2024-04-29T14:27:12.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aaa204a:::benchmark_data_smithi194_121512_object123324:head# 2024-04-29T14:27:12.507 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aaa2c55:::benchmark_data_smithi194_121512_object119284:head# 2024-04-29T14:27:12.514 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aabf949:::benchmark_data_smithi194_121512_object10616:head# 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aac2a4f:::benchmark_data_smithi194_121512_object66559:head# 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aacf616:::benchmark_data_smithi194_121512_object51196:head# 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aad25fb:::benchmark_data_smithi194_121512_object20309:head# 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aad2b56:::benchmark_data_smithi194_121512_object16260:head# 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.515 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aad927d:::benchmark_data_smithi194_121512_object13730:head# 2024-04-29T14:27:12.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.516 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aae05f8:::benchmark_data_smithi194_121512_object4556:head# 2024-04-29T14:27:12.516 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.521 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aaea04c:::benchmark_data_smithi194_121512_object91621:head# 2024-04-29T14:27:12.521 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.527 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aaf4ac5:::benchmark_data_smithi194_121512_object116755:head# 2024-04-29T14:27:12.527 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab0071a:::benchmark_data_smithi194_121512_object96091:head# 2024-04-29T14:27:12.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab02887:::benchmark_data_smithi194_121512_object5392:head# 2024-04-29T14:27:12.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab3925b:::benchmark_data_smithi194_121512_object67586:head# 2024-04-29T14:27:12.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab3ea99:::benchmark_data_smithi194_121512_object130757:head# 2024-04-29T14:27:12.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.533 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:12.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab3f046:::benchmark_data_smithi194_121512_object9262:head# 2024-04-29T14:27:12.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab4d1d3:::benchmark_data_smithi194_121512_object59932:head# 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab5b082:::benchmark_data_smithi194_121512_object3894:head# 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab5bb79:::benchmark_data_smithi194_121512_object133454:head# 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab67f20:::benchmark_data_smithi194_121512_object23892:head# 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab70d86:::benchmark_data_smithi194_121512_object28829:head# 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab7f8cc:::benchmark_data_smithi194_121512_object53957:head# 2024-04-29T14:27:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.539 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab80c89:::benchmark_data_smithi194_121512_object27579:head# 2024-04-29T14:27:12.539 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab8a23a:::benchmark_data_smithi194_121512_object55486:head# 2024-04-29T14:27:12.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.540 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ab95dd6:::benchmark_data_smithi194_121512_object22462:head# 2024-04-29T14:27:12.540 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abb4527:::benchmark_data_smithi194_121512_object95417:head# 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abcc3d9:::benchmark_data_smithi194_121512_object31256:head# 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abd5cde:::benchmark_data_smithi194_121512_object54834:head# 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abdc98a:::benchmark_data_smithi194_121512_object57517:head# 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abe3663:::benchmark_data_smithi194_121512_object26990:head# 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abe6164:::benchmark_data_smithi194_121512_object2067:head# 2024-04-29T14:27:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abe83d9:::benchmark_data_smithi194_121512_object3318:head# 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abe9d15:::benchmark_data_smithi194_121512_object46156:head# 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abec926:::benchmark_data_smithi194_121512_object121431:head# 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abf2ce7:::benchmark_data_smithi194_121512_object35895:head# 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abf8f42:::benchmark_data_smithi194_121512_object74804:head# 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abfc507:::benchmark_data_smithi194_121512_object122841:head# 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6abff386:::benchmark_data_smithi194_121512_object32826:head# 2024-04-29T14:27:12.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac01b3f:::benchmark_data_smithi194_121512_object22467:head# 2024-04-29T14:27:12.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac05430:::benchmark_data_smithi194_121512_object51892:head# 2024-04-29T14:27:12.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.551 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac0ffd9:::benchmark_data_smithi194_121512_object32398:head# 2024-04-29T14:27:12.552 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac14f8d:::benchmark_data_smithi194_121512_object46818:head# 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac1d816:::benchmark_data_smithi194_121512_object58953:head# 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac1daf8:::benchmark_data_smithi194_121512_object58079:head# 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac36bb7:::benchmark_data_smithi194_121512_object125357:head# 2024-04-29T14:27:12.553 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.554 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac3a9a3:::benchmark_data_smithi194_121512_object125752:head# 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac3b1b3:::benchmark_data_smithi194_121512_object24914:head# 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac41833:::benchmark_data_smithi194_121512_object115554:head# 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac56ae5:::benchmark_data_smithi194_121512_object111238:head# 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac5fd58:::benchmark_data_smithi194_121512_object40179:head# 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac605af:::benchmark_data_smithi194_121512_object37664:head# 2024-04-29T14:27:12.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac838e3:::benchmark_data_smithi194_121512_object70700:head# 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac88a26:::benchmark_data_smithi194_121512_object71152:head# 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ac9130c:::benchmark_data_smithi194_121512_object123889:head# 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aca7da5:::benchmark_data_smithi194_121512_object111273:head# 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acb2336:::benchmark_data_smithi194_121512_object114035:head# 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acc5994:::benchmark_data_smithi194_121512_object133419:head# 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acc78e7:::benchmark_data_smithi194_121512_object62922:head# 2024-04-29T14:27:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6accccc5:::benchmark_data_smithi194_121512_object63926:head# 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acce28c:::benchmark_data_smithi194_121512_object107871:head# 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acd21dc:::benchmark_data_smithi194_121512_object69839:head# 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acdd1c1:::benchmark_data_smithi194_121512_object90493:head# 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ace6140:::benchmark_data_smithi194_121512_object78912:head# 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ace7d4c:::benchmark_data_smithi194_121512_object128900:head# 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acf6ab8:::benchmark_data_smithi194_121512_object56316:head# 2024-04-29T14:27:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acfd57f:::benchmark_data_smithi194_121512_object21158:head# 2024-04-29T14:27:12.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acfe96b:::benchmark_data_smithi194_121512_object44164:head# 2024-04-29T14:27:12.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6acfeafb:::benchmark_data_smithi194_121512_object19170:head# 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad0ee45:::benchmark_data_smithi194_121512_object128159:head# 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad105de:::benchmark_data_smithi194_121512_object123148:head# 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad2a888:::benchmark_data_smithi194_121512_object41250:head# 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad2f946:::benchmark_data_smithi194_121512_object83515:head# 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad388a9:::benchmark_data_smithi194_121512_object26725:head# 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad3f0c5:::benchmark_data_smithi194_121512_object121698:head# 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad45341:::benchmark_data_smithi194_121512_object126700:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad48326:::benchmark_data_smithi194_121512_object39919:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad525fd:::benchmark_data_smithi194_121512_object124729:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad61749:::benchmark_data_smithi194_121512_object96657:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad67302:::benchmark_data_smithi194_121512_object49394:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad67d87:::benchmark_data_smithi194_121512_object1609:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad68a4f:::benchmark_data_smithi194_121512_object38822:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad6ac3c:::benchmark_data_smithi194_121512_object8114:head# 2024-04-29T14:27:12.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad7fbcf:::benchmark_data_smithi194_121512_object111971:head# 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad90299:::benchmark_data_smithi194_121512_object51282:head# 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad95290:::benchmark_data_smithi194_121512_object19553:head# 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad953d5:::benchmark_data_smithi194_121512_object90940:head# 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad95678:::benchmark_data_smithi194_121512_object99691:head# 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad9cbd7:::benchmark_data_smithi194_121512_object50667:head# 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad9cd7b:::benchmark_data_smithi194_121512_object116891:head# 2024-04-29T14:27:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ad9fc51:::benchmark_data_smithi194_121512_object9846:head# 2024-04-29T14:27:12.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ada6561:::benchmark_data_smithi194_121512_object89833:head# 2024-04-29T14:27:12.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3737555475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:12.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: pgmap v1275: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 654 op/s; 32234/205572 objects degraded (15.680%); 13 MiB/s, 208 objects/s recovering 2024-04-29T14:27:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:12 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 32988/210258 objects degraded (15.689%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adac22a:::benchmark_data_smithi194_121512_object51814:head# 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adb2e44:::benchmark_data_smithi194_121512_object57708:head# 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adb583b:::benchmark_data_smithi194_121512_object10880:head# 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adb619e:::benchmark_data_smithi194_121512_object61167:head# 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adbd250:::benchmark_data_smithi194_121512_object126315:head# 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adc96a6:::benchmark_data_smithi194_121512_object6995:head# 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6addf34a:::benchmark_data_smithi194_121512_object44646:head# 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ade733e:::benchmark_data_smithi194_121512_object64091:head# 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adf054b:::benchmark_data_smithi194_121512_object74965:head# 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6adf271e:::benchmark_data_smithi194_121512_object58716:head# 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae08146:::benchmark_data_smithi194_121512_object125959:head# 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae0856a:::benchmark_data_smithi194_121512_object103599:head# 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae10d01:::benchmark_data_smithi194_121512_object37295:head# 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae10ddb:::benchmark_data_smithi194_121512_object82577:head# 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae2e30a:::benchmark_data_smithi194_121512_object71141:head# 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae362c0:::benchmark_data_smithi194_121512_object92897:head# 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae3eae3:::benchmark_data_smithi194_121512_object129389:head# 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae4c67d:::benchmark_data_smithi194_121512_object42013:head# 2024-04-29T14:27:12.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae58a4d:::benchmark_data_smithi194_121512_object70538:head# 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae59483:::benchmark_data_smithi194_121512_object23529:head# 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae69f58:::benchmark_data_smithi194_121512_object38116:head# 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae80c27:::benchmark_data_smithi194_121512_object131813:head# 2024-04-29T14:27:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae81821:::benchmark_data_smithi194_121512_object66126:head# 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae8c92b:::benchmark_data_smithi194_121512_object121725:head# 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae98cea:::benchmark_data_smithi194_121512_object23855:head# 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ae996dc:::benchmark_data_smithi194_121512_object103431:head# 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aebe70a:::benchmark_data_smithi194_121512_object22919:head# 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aecd78c:::benchmark_data_smithi194_121512_object130654:head# 2024-04-29T14:27:12.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aecf32d:::benchmark_data_smithi194_121512_object115979:head# 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aeed11e:::benchmark_data_smithi194_121512_object30183:head# 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aef2bdd:::benchmark_data_smithi194_121512_object85213:head# 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aef42cd:::benchmark_data_smithi194_121512_object99194:head# 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aefe604:::benchmark_data_smithi194_121512_object94327:head# 2024-04-29T14:27:12.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af27f2b:::benchmark_data_smithi194_121512_object69918:head# 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af2c6cc:::benchmark_data_smithi194_121512_object7255:head# 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af40a75:::benchmark_data_smithi194_121512_object91031:head# 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af44955:::benchmark_data_smithi194_121512_object76090:head# 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af613aa:::benchmark_data_smithi194_121512_object8174:head# 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af61a30:::benchmark_data_smithi194_121512_object59563:head# 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af62a8e:::benchmark_data_smithi194_121512_object92366:head# 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af62c41:::benchmark_data_smithi194_121512_object46050:head# 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af663c5:::benchmark_data_smithi194_121512_object22813:head# 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af72701:::benchmark_data_smithi194_121512_object12634:head# 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6af99cf4:::benchmark_data_smithi194_121512_object39356:head# 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afacb9a:::benchmark_data_smithi194_121512_object123094:head# 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afb2e8c:::benchmark_data_smithi194_121512_object70313:head# 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afb9146:::benchmark_data_smithi194_121512_object6446:head# 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afbcf8d:::benchmark_data_smithi194_121512_object55903:head# 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afbcff8:::benchmark_data_smithi194_121512_object35001:head# 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afbdfa1:::benchmark_data_smithi194_121512_object23443:head# 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afc0eb8:::benchmark_data_smithi194_121512_object84807:head# 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afc532f:::benchmark_data_smithi194_121512_object26800:head# 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afdf17c:::benchmark_data_smithi194_121512_object9891:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afe2525:::benchmark_data_smithi194_121512_object111345:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afe3a51:::benchmark_data_smithi194_121512_object32167:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afe8db4:::benchmark_data_smithi194_121512_object92848:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6afedf88:::benchmark_data_smithi194_121512_object84155:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aff393a:::benchmark_data_smithi194_121512_object110694:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aff71c6:::benchmark_data_smithi194_121512_object107518:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6aff85eb:::benchmark_data_smithi194_121512_object115509:head# 2024-04-29T14:27:12.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b000236:::benchmark_data_smithi194_121512_object68236:head# 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0024cc:::benchmark_data_smithi194_121512_object29653:head# 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b00c9a8:::benchmark_data_smithi194_121512_object92802:head# 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b016e52:::benchmark_data_smithi194_121512_object122049:head# 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0255be:::benchmark_data_smithi194_121512_object119196:head# 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b026667:::benchmark_data_smithi194_121512_object6299:head# 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b02fb50:::benchmark_data_smithi194_121512_object41644:head# 2024-04-29T14:27:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b032abc:::benchmark_data_smithi194_121512_object122186:head# 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0355a8:::benchmark_data_smithi194_121512_object94006:head# 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b03926a:::benchmark_data_smithi194_121512_object35195:head# 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b039a01:::benchmark_data_smithi194_121512_object119118:head# 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b03c764:::benchmark_data_smithi194_121512_object65321:head# 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b049b70:::benchmark_data_smithi194_121512_object133085:head# 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b053aaa:::benchmark_data_smithi194_121512_object32289:head# 2024-04-29T14:27:12.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b055f45:::benchmark_data_smithi194_121512_object67246:head# 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b05a143:::benchmark_data_smithi194_121512_object28076:head# 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b060ccf:::benchmark_data_smithi194_121512_object25968:head# 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b065228:::benchmark_data_smithi194_121512_object19315:head# 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b068a28:::benchmark_data_smithi194_121512_object120605:head# 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b06e74e:::benchmark_data_smithi194_121512_object10424:head# 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b06fb02:::benchmark_data_smithi194_121512_object95566:head# 2024-04-29T14:27:12.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b070545:::benchmark_data_smithi194_121512_object74909:head# 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b078230:::benchmark_data_smithi194_121512_object45371:head# 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b097b95:::benchmark_data_smithi194_121512_object71705:head# 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b09bd42:::benchmark_data_smithi194_121512_object28855:head# 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b09c802:::benchmark_data_smithi194_121512_object95626:head# 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0a7c66:::benchmark_data_smithi194_121512_object9937:head# 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0b6991:::benchmark_data_smithi194_121512_object122150:head# 2024-04-29T14:27:12.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0be9ea:::benchmark_data_smithi194_121512_object92796:head# 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0ceae7:::benchmark_data_smithi194_121512_object36312:head# 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0d39b7:::benchmark_data_smithi194_121512_object83665:head# 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0d3cc8:::benchmark_data_smithi194_121512_object39297:head# 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0d3e9d:::benchmark_data_smithi194_121512_object121348:head# 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0e6409:::benchmark_data_smithi194_121512_object90163:head# 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0e9047:::benchmark_data_smithi194_121512_object1120:head# 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0eb5e6:::benchmark_data_smithi194_121512_object80368:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b0f8767:::benchmark_data_smithi194_121512_object28705:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b107c0e:::benchmark_data_smithi194_121512_object29655:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1097a8:::benchmark_data_smithi194_121512_object96190:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b117320:::benchmark_data_smithi194_121512_object96710:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b12b08a:::benchmark_data_smithi194_121512_object26779:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b12e0f1:::benchmark_data_smithi194_121512_object28589:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b141686:::benchmark_data_smithi194_121512_object93188:head# 2024-04-29T14:27:12.631 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b154962:::benchmark_data_smithi194_121512_object126920:head# 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b155623:::benchmark_data_smithi194_121512_object116713:head# 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b15bd7c:::benchmark_data_smithi194_121512_object51066:head# 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1709ab:::benchmark_data_smithi194_121512_object64508:head# 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b17103a:::benchmark_data_smithi194_121512_object106366:head# 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1848bc:::benchmark_data_smithi194_121512_object109804:head# 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b18ba32:::benchmark_data_smithi194_121512_object120885:head# 2024-04-29T14:27:12.632 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b18ca5c:::benchmark_data_smithi194_121512_object61994:head# 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1a6895:::benchmark_data_smithi194_121512_object100791:head# 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1b0495:::benchmark_data_smithi194_121512_object1847:head# 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1ba0bc:::benchmark_data_smithi194_121512_object64397:head# 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1d8e5f:::benchmark_data_smithi194_121512_object48433:head# 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1f410d:::benchmark_data_smithi194_121512_object111071:head# 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b1fbb41:::benchmark_data_smithi194_121512_object130131:head# 2024-04-29T14:27:12.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b20417b:::benchmark_data_smithi194_121512_object9449:head# 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b209d91:::benchmark_data_smithi194_121512_object21281:head# 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b2174bf:::benchmark_data_smithi194_121512_object129724:head# 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b21f179:::benchmark_data_smithi194_121512_object122329:head# 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b21fe5f:::benchmark_data_smithi194_121512_object37264:head# 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b221791:::benchmark_data_smithi194_121512_object102833:head# 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b22201f:::benchmark_data_smithi194_121512_object44916:head# 2024-04-29T14:27:12.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b22af07:::benchmark_data_smithi194_121512_object29513:head# 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b22d0b2:::benchmark_data_smithi194_121512_object49228:head# 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b239fb2:::benchmark_data_smithi194_121512_object25140:head# 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b23ad89:::benchmark_data_smithi194_121512_object28152:head# 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b244c9d:::benchmark_data_smithi194_121512_object121835:head# 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b25dd0e:::benchmark_data_smithi194_121512_object81526:head# 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b268a4a:::benchmark_data_smithi194_121512_object78523:head# 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b26e6b6:::benchmark_data_smithi194_121512_object96092:head# 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b2735b9:::benchmark_data_smithi194_121512_object83992:head# 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b278634:::benchmark_data_smithi194_121512_object23169:head# 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b288201:::benchmark_data_smithi194_121512_object51631:head# 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b289ef5:::benchmark_data_smithi194_121512_object130774:head# 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b28f435:::benchmark_data_smithi194_121512_object56511:head# 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b292c29:::benchmark_data_smithi194_121512_object39793:head# 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.636 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b294c7e:::benchmark_data_smithi194_121512_object99500:head# 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b296840:::benchmark_data_smithi194_121512_object67453:head# 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b298e57:::benchmark_data_smithi194_121512_object50188:head# 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b2a6312:::benchmark_data_smithi194_121512_object96574:head# 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b2aaeaa:::benchmark_data_smithi194_121512_object9285:head# 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b2d232c:::benchmark_data_smithi194_121512_object13278:head# 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b2f44f7:::benchmark_data_smithi194_121512_object99317:head# 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b2ff1f4:::benchmark_data_smithi194_121512_object63598:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3064c6:::benchmark_data_smithi194_121512_object8124:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b311962:::benchmark_data_smithi194_121512_object124686:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b31625e:::benchmark_data_smithi194_121512_object36040:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b316730:::benchmark_data_smithi194_121512_object132369:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b319793:::benchmark_data_smithi194_121512_object83461:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b33690f:::benchmark_data_smithi194_121512_object21280:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b342042:::benchmark_data_smithi194_121512_object62388:head# 2024-04-29T14:27:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b344f3f:::benchmark_data_smithi194_121512_object84039:head# 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b34c675:::benchmark_data_smithi194_121512_object60269:head# 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b351abe:::benchmark_data_smithi194_121512_object9443:head# 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b35ed43:::benchmark_data_smithi194_121512_object112894:head# 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3635e3:::benchmark_data_smithi194_121512_object119549:head# 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3866a9:::benchmark_data_smithi194_121512_object130274:head# 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b389dfd:::benchmark_data_smithi194_121512_object10289:head# 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b38a664:::benchmark_data_smithi194_121512_object54242:head# 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b392502:::benchmark_data_smithi194_121512_object39617:head# 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b396b90:::benchmark_data_smithi194_121512_object85870:head# 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b39c35f:::benchmark_data_smithi194_121512_object60102:head# 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3ba669:::benchmark_data_smithi194_121512_object120252:head# 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3c052a:::benchmark_data_smithi194_121512_object1703:head# 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3c7b68:::benchmark_data_smithi194_121512_object30388:head# 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3cbfa9:::benchmark_data_smithi194_121512_object47314:head# 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3d5645:::benchmark_data_smithi194_121512_object103543:head# 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3d59d8:::benchmark_data_smithi194_121512_object35153:head# 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3d8493:::benchmark_data_smithi194_121512_object68926:head# 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3e7cdc:::benchmark_data_smithi194_121512_object128245:head# 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3e931b:::benchmark_data_smithi194_121512_object130872:head# 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3ee1af:::benchmark_data_smithi194_121512_object10637:head# 2024-04-29T14:27:12.641 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b3f0105:::benchmark_data_smithi194_121512_object99936:head# 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b408b2e:::benchmark_data_smithi194_121512_object67769:head# 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b40d350:::benchmark_data_smithi194_121512_object30532:head# 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b417e11:::benchmark_data_smithi194_121512_object45395:head# 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b423d61:::benchmark_data_smithi194_121512_object29493:head# 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b429ef0:::benchmark_data_smithi194_121512_object1616:head# 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b42aac0:::benchmark_data_smithi194_121512_object44110:head# 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b42e251:::benchmark_data_smithi194_121512_object95138:head# 2024-04-29T14:27:12.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b432923:::benchmark_data_smithi194_121512_object35132:head# 2024-04-29T14:27:12.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b435579:::benchmark_data_smithi194_121512_object19527:head# 2024-04-29T14:27:12.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b43a02a:::benchmark_data_smithi194_121512_object1604:head# 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b44dd8e:::benchmark_data_smithi194_121512_object129605:head# 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b44ecf4:::benchmark_data_smithi194_121512_object44530:head# 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b450e70:::benchmark_data_smithi194_121512_object71177:head# 2024-04-29T14:27:12.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b452f3a:::benchmark_data_smithi194_121512_object79950:head# 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b45a489:::benchmark_data_smithi194_121512_object79955:head# 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4622b7:::benchmark_data_smithi194_121512_object97177:head# 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4625b4:::benchmark_data_smithi194_121512_object18231:head# 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b46842e:::benchmark_data_smithi194_121512_object123994:head# 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b471755:::benchmark_data_smithi194_121512_object61991:head# 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b473236:::benchmark_data_smithi194_121512_object6428:head# 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b47ce8a:::benchmark_data_smithi194_121512_object3509:head# 2024-04-29T14:27:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b47ed78:::benchmark_data_smithi194_121512_object118791:head# 2024-04-29T14:27:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b48bf92:::benchmark_data_smithi194_121512_object123738:head# 2024-04-29T14:27:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b498330:::benchmark_data_smithi194_121512_object13726:head# 2024-04-29T14:27:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3737555475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:12.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: pgmap v1275: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 654 op/s; 32234/205572 objects degraded (15.680%); 13 MiB/s, 208 objects/s recovering 2024-04-29T14:27:12.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:12 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 32988/210258 objects degraded (15.689%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b49be40:::benchmark_data_smithi194_121512_object80580:head# 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4a29b2:::benchmark_data_smithi194_121512_object111424:head# 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4a5892:::benchmark_data_smithi194_121512_object128106:head# 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4bfb98:::benchmark_data_smithi194_121512_object102562:head# 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4cf0f1:::benchmark_data_smithi194_121512_object108070:head# 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4cf879:::benchmark_data_smithi194_121512_object66784:head# 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4d3759:::benchmark_data_smithi194_121512_object71858:head# 2024-04-29T14:27:12.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4d4879:::benchmark_data_smithi194_121512_object103037:head# 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4d9e6f:::benchmark_data_smithi194_121512_object84176:head# 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4fe671:::benchmark_data_smithi194_121512_object119745:head# 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b4fe802:::benchmark_data_smithi194_121512_object117799:head# 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b50df77:::benchmark_data_smithi194_121512_object58754:head# 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b522962:::benchmark_data_smithi194_121512_object13251:head# 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b524498:::benchmark_data_smithi194_121512_object70929:head# 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b52c54c:::benchmark_data_smithi194_121512_object100994:head# 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b52f245:::benchmark_data_smithi194_121512_object106264:head# 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b52f95c:::benchmark_data_smithi194_121512_object19092:head# 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b53effd:::benchmark_data_smithi194_121512_object91986:head# 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b54085c:::benchmark_data_smithi194_121512_object129742:head# 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b545b45:::benchmark_data_smithi194_121512_object33333:head# 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b550050:::benchmark_data_smithi194_121512_object74597:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b550a81:::benchmark_data_smithi194_121512_object116178:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b55eae5:::benchmark_data_smithi194_121512_object22274:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5651ea:::benchmark_data_smithi194_121512_object90760:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b568fb7:::benchmark_data_smithi194_121512_object93173:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b56fc57:::benchmark_data_smithi194_121512_object15466:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b575141:::benchmark_data_smithi194_121512_object101542:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b577cb8:::benchmark_data_smithi194_121512_object29981:head# 2024-04-29T14:27:12.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5842c7:::benchmark_data_smithi194_121512_object22282:head# 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b584fc0:::benchmark_data_smithi194_121512_object23090:head# 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5867ed:::benchmark_data_smithi194_121512_object63345:head# 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b58ab2d:::benchmark_data_smithi194_121512_object96343:head# 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b58ba4e:::benchmark_data_smithi194_121512_object125322:head# 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5943b8:::benchmark_data_smithi194_121512_object86023:head# 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b597ea7:::benchmark_data_smithi194_121512_object96632:head# 2024-04-29T14:27:12.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5988b6:::benchmark_data_smithi194_121512_object19366:head# 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b59d0d4:::benchmark_data_smithi194_121512_object131658:head# 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b59dd82:::benchmark_data_smithi194_121512_object104483:head# 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5a004d:::benchmark_data_smithi194_121512_object27963:head# 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5a2304:::benchmark_data_smithi194_121512_object20133:head# 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5abdbf:::benchmark_data_smithi194_121512_object128296:head# 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5aebb4:::benchmark_data_smithi194_121512_object27823:head# 2024-04-29T14:27:12.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5bbe27:::benchmark_data_smithi194_121512_object47228:head# 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5be0ea:::benchmark_data_smithi194_121512_object96517:head# 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5bf316:::benchmark_data_smithi194_121512_object36959:head# 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5c58ad:::benchmark_data_smithi194_121512_object11502:head# 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5c59f8:::benchmark_data_smithi194_121512_object7647:head# 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5cb878:::benchmark_data_smithi194_121512_object119653:head# 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5d33f7:::benchmark_data_smithi194_121512_object113057:head# 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5d43e0:::benchmark_data_smithi194_121512_object84871:head# 2024-04-29T14:27:12.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5d73dd:::benchmark_data_smithi194_121512_object76526:head# 2024-04-29T14:27:12.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5def71:::benchmark_data_smithi194_121512_object77432:head# 2024-04-29T14:27:12.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5e6a57:::benchmark_data_smithi194_121512_object31183:head# 2024-04-29T14:27:12.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5e8efe:::benchmark_data_smithi194_121512_object82983:head# 2024-04-29T14:27:12.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5ee1d1:::benchmark_data_smithi194_121512_object27629:head# 2024-04-29T14:27:12.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5f8970:::benchmark_data_smithi194_121512_object92363:head# 2024-04-29T14:27:12.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5f9306:::benchmark_data_smithi194_121512_object74050:head# 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b5f94b8:::benchmark_data_smithi194_121512_object16996:head# 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b617677:::benchmark_data_smithi194_121512_object101934:head# 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b61abc8:::benchmark_data_smithi194_121512_object58330:head# 2024-04-29T14:27:12.662 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b61e5c3:::benchmark_data_smithi194_121512_object72690:head# 2024-04-29T14:27:12.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b62bf4d:::benchmark_data_smithi194_121512_object115895:head# 2024-04-29T14:27:12.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6320f5:::benchmark_data_smithi194_121512_object27577:head# 2024-04-29T14:27:12.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b633ae1:::benchmark_data_smithi194_121512_object116789:head# 2024-04-29T14:27:12.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b641550:::benchmark_data_smithi194_121512_object22771:head# 2024-04-29T14:27:12.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b64283d:::benchmark_data_smithi194_121512_object79777:head# 2024-04-29T14:27:12.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b64b2ce:::benchmark_data_smithi194_121512_object36444:head# 2024-04-29T14:27:12.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b653a1b:::benchmark_data_smithi194_121512_object78975:head# 2024-04-29T14:27:12.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b65e041:::benchmark_data_smithi194_121512_object8982:head# 2024-04-29T14:27:12.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b65fdc5:::benchmark_data_smithi194_121512_object123261:head# 2024-04-29T14:27:12.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6885cd:::benchmark_data_smithi194_121512_object70520:head# 2024-04-29T14:27:12.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b690959:::benchmark_data_smithi194_121512_object58373:head# 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b69c9c2:::benchmark_data_smithi194_121512_object35093:head# 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6a0d89:::benchmark_data_smithi194_121512_object5884:head# 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6a7102:::benchmark_data_smithi194_121512_object72269:head# 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6b8646:::benchmark_data_smithi194_121512_object100878:head# 2024-04-29T14:27:12.670 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6caadf:::benchmark_data_smithi194_121512_object5022:head# 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6cad8b:::benchmark_data_smithi194_121512_object59086:head# 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6cbdfe:::benchmark_data_smithi194_121512_object26876:head# 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6cd16f:::benchmark_data_smithi194_121512_object5733:head# 2024-04-29T14:27:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6d1747:::benchmark_data_smithi194_121512_object43318:head# 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6d69e3:::benchmark_data_smithi194_121512_object92707:head# 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6da370:::benchmark_data_smithi194_121512_object32093:head# 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6df7c9:::benchmark_data_smithi194_121512_object99128:head# 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6ed4a7:::benchmark_data_smithi194_121512_object69059:head# 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.673 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b6f61cc:::benchmark_data_smithi194_121512_object87093:head# 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b706344:::benchmark_data_smithi194_121512_object71180:head# 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b70665a:::benchmark_data_smithi194_121512_object66312:head# 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b70bb44:::benchmark_data_smithi194_121512_object113661:head# 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7146f3:::benchmark_data_smithi194_121512_object100157:head# 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b71739c:::benchmark_data_smithi194_121512_object49439:head# 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b71b124:::benchmark_data_smithi194_121512_object56564:head# 2024-04-29T14:27:12.674 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7258be:::benchmark_data_smithi194_121512_object83481:head# 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b72d70a:::benchmark_data_smithi194_121512_object100714:head# 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b730a5f:::benchmark_data_smithi194_121512_object48485:head# 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b733fde:::benchmark_data_smithi194_121512_object46597:head# 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7394be:::benchmark_data_smithi194_121512_object32053:head# 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b73ebba:::benchmark_data_smithi194_121512_object39415:head# 2024-04-29T14:27:12.676 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b745656:::benchmark_data_smithi194_121512_object80625:head# 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b74ea5d:::benchmark_data_smithi194_121512_object39529:head# 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b74f565:::benchmark_data_smithi194_121512_object61148:head# 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7564b8:::benchmark_data_smithi194_121512_object24980:head# 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b75aca5:::benchmark_data_smithi194_121512_object42572:head# 2024-04-29T14:27:12.677 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b75c492:::benchmark_data_smithi194_121512_object29380:head# 2024-04-29T14:27:12.678 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.678 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b76ec0e:::benchmark_data_smithi194_121512_object4052:head# 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7751cf:::benchmark_data_smithi194_121512_object93440:head# 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7837c2:::benchmark_data_smithi194_121512_object17771:head# 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7963f2:::benchmark_data_smithi194_121512_object92788:head# 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b79f2ca:::benchmark_data_smithi194_121512_object2835:head# 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7a316f:::benchmark_data_smithi194_121512_object13312:head# 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7ab99d:::benchmark_data_smithi194_121512_object44885:head# 2024-04-29T14:27:12.679 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7ae9d7:::benchmark_data_smithi194_121512_object107360:head# 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7ba5fe:::benchmark_data_smithi194_121512_object50139:head# 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7bd933:::benchmark_data_smithi194_121512_object111290:head# 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7c9006:::benchmark_data_smithi194_121512_object90475:head# 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7cb07c:::benchmark_data_smithi194_121512_object78095:head# 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7cd2b3:::benchmark_data_smithi194_121512_object26436:head# 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7cec4c:::benchmark_data_smithi194_121512_object126166:head# 2024-04-29T14:27:12.681 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7d02a6:::benchmark_data_smithi194_121512_object46740:head# 2024-04-29T14:27:12.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7d238f:::benchmark_data_smithi194_121512_object5299:head# 2024-04-29T14:27:12.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.682 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7df99e:::benchmark_data_smithi194_121512_object37051:head# 2024-04-29T14:27:12.682 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7ea31f:::benchmark_data_smithi194_121512_object9706:head# 2024-04-29T14:27:12.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7ecc12:::benchmark_data_smithi194_121512_object69312:head# 2024-04-29T14:27:12.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7eeff6:::benchmark_data_smithi194_121512_object140:head# 2024-04-29T14:27:12.683 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b7f4445:::benchmark_data_smithi194_121512_object84854:head# 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b810d69:::benchmark_data_smithi194_121512_object9266:head# 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b811d12:::benchmark_data_smithi194_121512_object17099:head# 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b816819:::benchmark_data_smithi194_121512_object44288:head# 2024-04-29T14:27:12.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b828b57:::benchmark_data_smithi194_121512_object103251:head# 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b828ff8:::benchmark_data_smithi194_121512_object14801:head# 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b83e56f:::benchmark_data_smithi194_121512_object29171:head# 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b83f66e:::benchmark_data_smithi194_121512_object68073:head# 2024-04-29T14:27:12.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8464c3:::benchmark_data_smithi194_121512_object36432:head# 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b84849e:::benchmark_data_smithi194_121512_object26862:head# 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b84d278:::benchmark_data_smithi194_121512_object27802:head# 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b84f566:::benchmark_data_smithi194_121512_object35584:head# 2024-04-29T14:27:12.687 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b87336a:::benchmark_data_smithi194_121512_object133025:head# 2024-04-29T14:27:12.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.692 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8760a1:::benchmark_data_smithi194_121512_object95645:head# 2024-04-29T14:27:12.692 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b87a900:::benchmark_data_smithi194_121512_object98570:head# 2024-04-29T14:27:12.693 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.693 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b87dd16:::benchmark_data_smithi194_121512_object113315:head# 2024-04-29T14:27:12.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b886681:::benchmark_data_smithi194_121512_object56390:head# 2024-04-29T14:27:12.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8875bc:::benchmark_data_smithi194_121512_object56985:head# 2024-04-29T14:27:12.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.694 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b888587:::benchmark_data_smithi194_121512_object22672:head# 2024-04-29T14:27:12.694 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b893037:::benchmark_data_smithi194_121512_object118341:head# 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b89f6d3:::benchmark_data_smithi194_121512_object112735:head# 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b89ff6e:::benchmark_data_smithi194_121512_object79642:head# 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8a14a0:::benchmark_data_smithi194_121512_object121905:head# 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.695 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8a838a:::benchmark_data_smithi194_121512_object77042:head# 2024-04-29T14:27:12.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8bd383:::benchmark_data_smithi194_121512_object30879:head# 2024-04-29T14:27:12.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.696 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8c50f3:::benchmark_data_smithi194_121512_object37415:head# 2024-04-29T14:27:12.696 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8c55de:::benchmark_data_smithi194_121512_object40900:head# 2024-04-29T14:27:12.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8ca20e:::benchmark_data_smithi194_121512_object20208:head# 2024-04-29T14:27:12.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8ca932:::benchmark_data_smithi194_121512_object34810:head# 2024-04-29T14:27:12.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8dc108:::benchmark_data_smithi194_121512_object120880:head# 2024-04-29T14:27:12.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8de8b3:::benchmark_data_smithi194_121512_object101715:head# 2024-04-29T14:27:12.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8ec076:::benchmark_data_smithi194_121512_object59762:head# 2024-04-29T14:27:12.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8efec0:::benchmark_data_smithi194_121512_object89642:head# 2024-04-29T14:27:12.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8f8ecd:::benchmark_data_smithi194_121512_object106071:head# 2024-04-29T14:27:12.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8fbf72:::benchmark_data_smithi194_121512_object21792:head# 2024-04-29T14:27:12.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8fbffb:::benchmark_data_smithi194_121512_object120120:head# 2024-04-29T14:27:12.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b8fe2be:::benchmark_data_smithi194_121512_object47379:head# 2024-04-29T14:27:12.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9009c5:::benchmark_data_smithi194_121512_object124870:head# 2024-04-29T14:27:12.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9016c2:::benchmark_data_smithi194_121512_object45186:head# 2024-04-29T14:27:12.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b902dcb:::benchmark_data_smithi194_121512_object121651:head# 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b906c2b:::benchmark_data_smithi194_121512_object57498:head# 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b90cf5b:::benchmark_data_smithi194_121512_object55293:head# 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b910384:::benchmark_data_smithi194_121512_object61809:head# 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b91905e:::benchmark_data_smithi194_121512_object82245:head# 2024-04-29T14:27:12.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b91d023:::benchmark_data_smithi194_121512_object4865:head# 2024-04-29T14:27:12.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b925ac3:::benchmark_data_smithi194_121512_object36332:head# 2024-04-29T14:27:12.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9268e6:::benchmark_data_smithi194_121512_object26369:head# 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b928cdc:::benchmark_data_smithi194_121512_object115682:head# 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b928ebe:::benchmark_data_smithi194_121512_object73370:head# 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b934838:::benchmark_data_smithi194_121512_object101523:head# 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9610d5:::benchmark_data_smithi194_121512_object15288:head# 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b961982:::benchmark_data_smithi194_121512_object113080:head# 2024-04-29T14:27:12.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b964cc9:::benchmark_data_smithi194_121512_object110046:head# 2024-04-29T14:27:12.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b97fd37:::benchmark_data_smithi194_121512_object7984:head# 2024-04-29T14:27:12.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b98b334:::benchmark_data_smithi194_121512_object13566:head# 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b991530:::benchmark_data_smithi194_121512_object22626:head# 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b996f64:::benchmark_data_smithi194_121512_object129379:head# 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b99a6b4:::benchmark_data_smithi194_121512_object39685:head# 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9a28d5:::benchmark_data_smithi194_121512_object85464:head# 2024-04-29T14:27:12.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9a7fd3:::benchmark_data_smithi194_121512_object132426:head# 2024-04-29T14:27:12.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9b2003:::benchmark_data_smithi194_121512_object51476:head# 2024-04-29T14:27:12.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9b2cb2:::benchmark_data_smithi194_121512_object55279:head# 2024-04-29T14:27:12.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9cd780:::benchmark_data_smithi194_121512_object31032:head# 2024-04-29T14:27:12.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9d668d:::benchmark_data_smithi194_121512_object47273:head# 2024-04-29T14:27:12.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9df81c:::benchmark_data_smithi194_121512_object519:head# 2024-04-29T14:27:12.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9e008d:::benchmark_data_smithi194_121512_object51973:head# 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9e02c2:::benchmark_data_smithi194_121512_object110173:head# 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9e9fc3:::benchmark_data_smithi194_121512_object85360:head# 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6b9ff100:::benchmark_data_smithi194_121512_object96172:head# 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba08144:::benchmark_data_smithi194_121512_object123450:head# 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba09552:::benchmark_data_smithi194_121512_object1911:head# 2024-04-29T14:27:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba1483d:::benchmark_data_smithi194_121512_object900:head# 2024-04-29T14:27:12.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba14dbd:::benchmark_data_smithi194_121512_object44088:head# 2024-04-29T14:27:12.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba1ba22:::benchmark_data_smithi194_121512_object119003:head# 2024-04-29T14:27:12.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba1ed8f:::benchmark_data_smithi194_121512_object49048:head# 2024-04-29T14:27:12.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba2d8f1:::benchmark_data_smithi194_121512_object55540:head# 2024-04-29T14:27:12.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba34a15:::benchmark_data_smithi194_121512_object354:head# 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba3b4f3:::benchmark_data_smithi194_121512_object65378:head# 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba465f9:::benchmark_data_smithi194_121512_object116083:head# 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba5784c:::benchmark_data_smithi194_121512_object51981:head# 2024-04-29T14:27:12.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba5e11e:::benchmark_data_smithi194_121512_object46565:head# 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba675c5:::benchmark_data_smithi194_121512_object104441:head# 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba6f469:::benchmark_data_smithi194_121512_object88154:head# 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba71901:::benchmark_data_smithi194_121512_object6720:head# 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba7d4cd:::benchmark_data_smithi194_121512_object22405:head# 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba87b69:::benchmark_data_smithi194_121512_object52366:head# 2024-04-29T14:27:12.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba89351:::benchmark_data_smithi194_121512_object113288:head# 2024-04-29T14:27:12.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba8b1f2:::benchmark_data_smithi194_121512_object32212:head# 2024-04-29T14:27:12.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba8c6e8:::benchmark_data_smithi194_121512_object131591:head# 2024-04-29T14:27:12.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba8ff98:::benchmark_data_smithi194_121512_object81531:head# 2024-04-29T14:27:12.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba923ce:::benchmark_data_smithi194_121512_object90247:head# 2024-04-29T14:27:12.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba99c1e:::benchmark_data_smithi194_121512_object8698:head# 2024-04-29T14:27:12.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ba9c6f6:::benchmark_data_smithi194_121512_object16295:head# 2024-04-29T14:27:12.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6babc12f:::benchmark_data_smithi194_121512_object41794:head# 2024-04-29T14:27:12.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bacbdf9:::benchmark_data_smithi194_121512_object97380:head# 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bacf70d:::benchmark_data_smithi194_121512_object7769:head# 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6badd56a:::benchmark_data_smithi194_121512_object16353:head# 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6badfee5:::benchmark_data_smithi194_121512_object86261:head# 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bae1c2d:::benchmark_data_smithi194_121512_object133171:head# 2024-04-29T14:27:12.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bae2b47:::benchmark_data_smithi194_121512_object238:head# 2024-04-29T14:27:12.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6baed100:::benchmark_data_smithi194_121512_object32341:head# 2024-04-29T14:27:12.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb12ac5:::benchmark_data_smithi194_121512_object26853:head# 2024-04-29T14:27:12.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb15531:::benchmark_data_smithi194_121512_object34123:head# 2024-04-29T14:27:12.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb17a73:::benchmark_data_smithi194_121512_object99135:head# 2024-04-29T14:27:12.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb38d24:::benchmark_data_smithi194_121512_object40408:head# 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb43566:::benchmark_data_smithi194_121512_object30670:head# 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb5e4c0:::benchmark_data_smithi194_121512_object58844:head# 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb60a0d:::benchmark_data_smithi194_121512_object110674:head# 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb78d95:::benchmark_data_smithi194_121512_object2205:head# 2024-04-29T14:27:12.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb89c50:::benchmark_data_smithi194_121512_object33313:head# 2024-04-29T14:27:12.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb8ac05:::benchmark_data_smithi194_121512_object22965:head# 2024-04-29T14:27:12.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb939f4:::benchmark_data_smithi194_121512_object8556:head# 2024-04-29T14:27:12.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb955b3:::benchmark_data_smithi194_121512_object119917:head# 2024-04-29T14:27:12.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb969d3:::benchmark_data_smithi194_121512_object58818:head# 2024-04-29T14:27:12.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bb9f3b8:::benchmark_data_smithi194_121512_object28241:head# 2024-04-29T14:27:12.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bba6b39:::benchmark_data_smithi194_121512_object120695:head# 2024-04-29T14:27:12.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bba9f48:::benchmark_data_smithi194_121512_object132013:head# 2024-04-29T14:27:12.740 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.740 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbac0ea:::benchmark_data_smithi194_121512_object112228:head# 2024-04-29T14:27:12.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbb2a7d:::benchmark_data_smithi194_121512_object24694:head# 2024-04-29T14:27:12.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbb3985:::benchmark_data_smithi194_121512_object44011:head# 2024-04-29T14:27:12.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbb8214:::benchmark_data_smithi194_121512_object43406:head# 2024-04-29T14:27:12.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbbdc57:::benchmark_data_smithi194_121512_object2912:head# 2024-04-29T14:27:12.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbca0fe:::benchmark_data_smithi194_121512_object99572:head# 2024-04-29T14:27:12.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbccaa0:::benchmark_data_smithi194_121512_object57224:head# 2024-04-29T14:27:12.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbd1dc0:::benchmark_data_smithi194_121512_object65014:head# 2024-04-29T14:27:12.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbe6fc8:::benchmark_data_smithi194_121512_object92352:head# 2024-04-29T14:27:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbee757:::benchmark_data_smithi194_121512_object579:head# 2024-04-29T14:27:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbf8c1d:::benchmark_data_smithi194_121512_object100187:head# 2024-04-29T14:27:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbfaa84:::benchmark_data_smithi194_121512_object61361:head# 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbfd590:::benchmark_data_smithi194_121512_object21750:head# 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbfe0e9:::benchmark_data_smithi194_121512_object11350:head# 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bbff631:::benchmark_data_smithi194_121512_object76351:head# 2024-04-29T14:27:12.745 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc04706:::benchmark_data_smithi194_121512_object128007:head# 2024-04-29T14:27:12.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc07efd:::benchmark_data_smithi194_121512_object109571:head# 2024-04-29T14:27:12.746 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.746 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc11917:::benchmark_data_smithi194_121512_object9197:head# 2024-04-29T14:27:12.747 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc12d5f:::benchmark_data_smithi194_121512_object114012:head# 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc1ced1:::benchmark_data_smithi194_121512_object72787:head# 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc29663:::benchmark_data_smithi194_121512_object46630:head# 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc353f3:::benchmark_data_smithi194_121512_object63465:head# 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc3bf99:::benchmark_data_smithi194_121512_object71465:head# 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc4a7bd:::benchmark_data_smithi194_121512_object18179:head# 2024-04-29T14:27:12.749 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc5f816:::benchmark_data_smithi194_121512_object109785:head# 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc61ce4:::benchmark_data_smithi194_121512_object16373:head# 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc6b204:::benchmark_data_smithi194_121512_object98054:head# 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc7302e:::benchmark_data_smithi194_121512_object110990:head# 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.751 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc73421:::benchmark_data_smithi194_121512_object122576:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc7630b:::benchmark_data_smithi194_121512_object68547:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc7df47:::benchmark_data_smithi194_121512_object36524:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc86748:::benchmark_data_smithi194_121512_object91662:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc8d055:::benchmark_data_smithi194_121512_object41998:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bc91ea2:::benchmark_data_smithi194_121512_object45226:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bca957e:::benchmark_data_smithi194_121512_object55061:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcac416:::benchmark_data_smithi194_121512_object36826:head# 2024-04-29T14:27:12.752 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcb266e:::benchmark_data_smithi194_121512_object48208:head# 2024-04-29T14:27:12.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcba431:::benchmark_data_smithi194_121512_object50104:head# 2024-04-29T14:27:12.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcce5f0:::benchmark_data_smithi194_121512_object104333:head# 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bccf039:::benchmark_data_smithi194_121512_object105818:head# 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcd1c19:::benchmark_data_smithi194_121512_object6103:head# 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcd46e4:::benchmark_data_smithi194_121512_object71113:head# 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcdabcc:::benchmark_data_smithi194_121512_object29790:head# 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcdb0e9:::benchmark_data_smithi194_121512_object28168:head# 2024-04-29T14:27:12.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bce819f:::benchmark_data_smithi194_121512_object91260:head# 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bce9ef7:::benchmark_data_smithi194_121512_object16686:head# 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcf4949:::benchmark_data_smithi194_121512_object67827:head# 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcfc289:::benchmark_data_smithi194_121512_object63977:head# 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bcfe6e1:::benchmark_data_smithi194_121512_object33960:head# 2024-04-29T14:27:12.756 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd04046:::benchmark_data_smithi194_121512_object120744:head# 2024-04-29T14:27:12.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd0e79e:::benchmark_data_smithi194_121512_object59783:head# 2024-04-29T14:27:12.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd2ca51:::benchmark_data_smithi194_121512_object91220:head# 2024-04-29T14:27:12.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd316cd:::benchmark_data_smithi194_121512_object126537:head# 2024-04-29T14:27:12.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd337f3:::benchmark_data_smithi194_121512_object93064:head# 2024-04-29T14:27:12.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.791 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd489d9:::benchmark_data_smithi194_121512_object123691:head# 2024-04-29T14:27:12.791 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd50426:::benchmark_data_smithi194_121512_object53902:head# 2024-04-29T14:27:12.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.792 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd551e6:::benchmark_data_smithi194_121512_object79094:head# 2024-04-29T14:27:12.792 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd622c2:::benchmark_data_smithi194_121512_object97601:head# 2024-04-29T14:27:12.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd652ff:::benchmark_data_smithi194_121512_object107796:head# 2024-04-29T14:27:12.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.793 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd77b36:::benchmark_data_smithi194_121512_object71305:head# 2024-04-29T14:27:12.793 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd7c5ea:::benchmark_data_smithi194_121512_object62671:head# 2024-04-29T14:27:12.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.794 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd83520:::benchmark_data_smithi194_121512_object68589:head# 2024-04-29T14:27:12.794 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd8bb70:::benchmark_data_smithi194_121512_object80152:head# 2024-04-29T14:27:12.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd90261:::benchmark_data_smithi194_121512_object48623:head# 2024-04-29T14:27:12.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd92eab:::benchmark_data_smithi194_121512_object13597:head# 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd9857d:::benchmark_data_smithi194_121512_object125768:head# 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bd9db11:::benchmark_data_smithi194_121512_object39977:head# 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bda10de:::benchmark_data_smithi194_121512_object105306:head# 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.796 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdaf2e0:::benchmark_data_smithi194_121512_object30672:head# 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdc30d1:::benchmark_data_smithi194_121512_object95026:head# 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdd78bb:::benchmark_data_smithi194_121512_object51974:head# 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bddf3d7:::benchmark_data_smithi194_121512_object39939:head# 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bde8c89:::benchmark_data_smithi194_121512_object24286:head# 2024-04-29T14:27:12.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdec469:::benchmark_data_smithi194_121512_object26737:head# 2024-04-29T14:27:12.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdf3f69:::benchmark_data_smithi194_121512_object95352:head# 2024-04-29T14:27:12.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdfa6ac:::benchmark_data_smithi194_121512_object92509:head# 2024-04-29T14:27:12.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdfac0d:::benchmark_data_smithi194_121512_object57494:head# 2024-04-29T14:27:12.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdfcd89:::benchmark_data_smithi194_121512_object25014:head# 2024-04-29T14:27:12.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bdfd17d:::benchmark_data_smithi194_121512_object126149:head# 2024-04-29T14:27:12.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be018a4:::benchmark_data_smithi194_121512_object75116:head# 2024-04-29T14:27:12.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be049ae:::benchmark_data_smithi194_121512_object4825:head# 2024-04-29T14:27:12.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be106e8:::benchmark_data_smithi194_121512_object44827:head# 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be135a5:::benchmark_data_smithi194_121512_object3874:head# 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be25b20:::benchmark_data_smithi194_121512_object105601:head# 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be2ced5:::benchmark_data_smithi194_121512_object99180:head# 2024-04-29T14:27:12.803 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be2f535:::benchmark_data_smithi194_121512_object53988:head# 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be360fa:::benchmark_data_smithi194_121512_object131018:head# 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be39264:::benchmark_data_smithi194_121512_object74045:head# 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be4636b:::benchmark_data_smithi194_121512_object50175:head# 2024-04-29T14:27:12.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be4d346:::benchmark_data_smithi194_121512_object105524:head# 2024-04-29T14:27:12.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be5483a:::benchmark_data_smithi194_121512_object60103:head# 2024-04-29T14:27:12.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be6820a:::benchmark_data_smithi194_121512_object126757:head# 2024-04-29T14:27:12.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be6d09e:::benchmark_data_smithi194_121512_object13286:head# 2024-04-29T14:27:12.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.806 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be6f58b:::benchmark_data_smithi194_121512_object31849:head# 2024-04-29T14:27:12.806 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be77ca2:::benchmark_data_smithi194_121512_object18578:head# 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be7c9c0:::benchmark_data_smithi194_121512_object59360:head# 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be7fb7d:::benchmark_data_smithi194_121512_object109740:head# 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be8496d:::benchmark_data_smithi194_121512_object91455:head# 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be899f6:::benchmark_data_smithi194_121512_object123935:head# 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be8f032:::benchmark_data_smithi194_121512_object119877:head# 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6be955d6:::benchmark_data_smithi194_121512_object11291:head# 2024-04-29T14:27:12.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6beb01f3:::benchmark_data_smithi194_121512_object32530:head# 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6beb926c:::benchmark_data_smithi194_121512_object48225:head# 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bebbfb0:::benchmark_data_smithi194_121512_object21457:head# 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bec0113:::benchmark_data_smithi194_121512_object99251:head# 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bec4e2c:::benchmark_data_smithi194_121512_object119661:head# 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bec8201:::benchmark_data_smithi194_121512_object74954:head# 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6becb24a:::benchmark_data_smithi194_121512_object81896:head# 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bee953b:::benchmark_data_smithi194_121512_object107631:head# 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6befd8b8:::benchmark_data_smithi194_121512_object51288:head# 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6befdbff:::benchmark_data_smithi194_121512_object75910:head# 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf0ee83:::benchmark_data_smithi194_121512_object47832:head# 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf30fc3:::benchmark_data_smithi194_121512_object30792:head# 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf328c9:::benchmark_data_smithi194_121512_object114215:head# 2024-04-29T14:27:12.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf4697c:::benchmark_data_smithi194_121512_object85177:head# 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf5e5d0:::benchmark_data_smithi194_121512_object61952:head# 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf61da9:::benchmark_data_smithi194_121512_object13131:head# 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf67977:::benchmark_data_smithi194_121512_object95753:head# 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf6b97b:::benchmark_data_smithi194_121512_object67601:head# 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf73b8d:::benchmark_data_smithi194_121512_object37900:head# 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf78847:::benchmark_data_smithi194_121512_object100473:head# 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf8275b:::benchmark_data_smithi194_121512_object2175:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf83ba3:::benchmark_data_smithi194_121512_object69519:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf86b59:::benchmark_data_smithi194_121512_object102284:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bf958c9:::benchmark_data_smithi194_121512_object90927:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfa3359:::benchmark_data_smithi194_121512_object36291:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfa3d14:::benchmark_data_smithi194_121512_object82522:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfa4801:::benchmark_data_smithi194_121512_object76460:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfaca35:::benchmark_data_smithi194_121512_object95688:head# 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfae586:::benchmark_data_smithi194_121512_object56545:head# 2024-04-29T14:27:12.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfb58bf:::benchmark_data_smithi194_121512_object63136:head# 2024-04-29T14:27:12.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfb6e6a:::benchmark_data_smithi194_121512_object73466:head# 2024-04-29T14:27:12.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfba4e2:::benchmark_data_smithi194_121512_object27871:head# 2024-04-29T14:27:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfc11b9:::benchmark_data_smithi194_121512_object89705:head# 2024-04-29T14:27:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfc2e5e:::benchmark_data_smithi194_121512_object21889:head# 2024-04-29T14:27:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfd49d2:::benchmark_data_smithi194_121512_object12827:head# 2024-04-29T14:27:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfe1c91:::benchmark_data_smithi194_121512_object57605:head# 2024-04-29T14:27:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfe28e4:::benchmark_data_smithi194_121512_object32629:head# 2024-04-29T14:27:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfe3da2:::benchmark_data_smithi194_121512_object45140:head# 2024-04-29T14:27:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfe517e:::benchmark_data_smithi194_121512_object30676:head# 2024-04-29T14:27:12.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bfeabff:::benchmark_data_smithi194_121512_object74711:head# 2024-04-29T14:27:12.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bffc0b1:::benchmark_data_smithi194_121512_object39518:head# 2024-04-29T14:27:12.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bffc969:::benchmark_data_smithi194_121512_object97812:head# 2024-04-29T14:27:12.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6bffc9ca:::benchmark_data_smithi194_121512_object51773:head# 2024-04-29T14:27:12.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0010ca:::benchmark_data_smithi194_121512_object21648:head# 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c014f8a:::benchmark_data_smithi194_121512_object34358:head# 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c018b04:::benchmark_data_smithi194_121512_object24268:head# 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c02d68e:::benchmark_data_smithi194_121512_object96071:head# 2024-04-29T14:27:12.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c032af6:::benchmark_data_smithi194_121512_object30323:head# 2024-04-29T14:27:12.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c034264:::benchmark_data_smithi194_121512_object644:head# 2024-04-29T14:27:12.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c03475d:::benchmark_data_smithi194_121512_object78410:head# 2024-04-29T14:27:12.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c037462:::benchmark_data_smithi194_121512_object19867:head# 2024-04-29T14:27:12.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c044a60:::benchmark_data_smithi194_121512_object87198:head# 2024-04-29T14:27:12.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c04d198:::benchmark_data_smithi194_121512_object34232:head# 2024-04-29T14:27:12.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c051845:::benchmark_data_smithi194_121512_object96087:head# 2024-04-29T14:27:12.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c055056:::benchmark_data_smithi194_121512_object76715:head# 2024-04-29T14:27:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c05e93a:::benchmark_data_smithi194_121512_object86456:head# 2024-04-29T14:27:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c068553:::benchmark_data_smithi194_121512_object8193:head# 2024-04-29T14:27:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c068b5f:::benchmark_data_smithi194_121512_object83068:head# 2024-04-29T14:27:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c069d57:::benchmark_data_smithi194_121512_object13440:head# 2024-04-29T14:27:12.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c06b8a8:::benchmark_data_smithi194_121512_object35410:head# 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c06c6e5:::benchmark_data_smithi194_121512_object21215:head# 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c06e76c:::benchmark_data_smithi194_121512_object74436:head# 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c07cca1:::benchmark_data_smithi194_121512_object102322:head# 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c07e400:::benchmark_data_smithi194_121512_object119921:head# 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c07f7d2:::benchmark_data_smithi194_121512_object78430:head# 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c082369:::benchmark_data_smithi194_121512_object112844:head# 2024-04-29T14:27:12.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0860f6:::benchmark_data_smithi194_121512_object105751:head# 2024-04-29T14:27:12.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c090358:::benchmark_data_smithi194_121512_object43327:head# 2024-04-29T14:27:12.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c09184a:::benchmark_data_smithi194_121512_object115388:head# 2024-04-29T14:27:12.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c092619:::benchmark_data_smithi194_121512_object26149:head# 2024-04-29T14:27:12.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0960cb:::benchmark_data_smithi194_121512_object30569:head# 2024-04-29T14:27:12.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c09b3f2:::benchmark_data_smithi194_121512_object9723:head# 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0b1c35:::benchmark_data_smithi194_121512_object3250:head# 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0b6eb8:::benchmark_data_smithi194_121512_object72959:head# 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0bb5b6:::benchmark_data_smithi194_121512_object115221:head# 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0bba60:::benchmark_data_smithi194_121512_object30932:head# 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0bcb34:::benchmark_data_smithi194_121512_object63839:head# 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.839 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0c3176:::benchmark_data_smithi194_121512_object21177:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0cf44a:::benchmark_data_smithi194_121512_object126994:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0d56ac:::benchmark_data_smithi194_121512_object49997:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0dc4cb:::benchmark_data_smithi194_121512_object10757:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0df3ce:::benchmark_data_smithi194_121512_object70816:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0e1d3a:::benchmark_data_smithi194_121512_object96226:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0f03d3:::benchmark_data_smithi194_121512_object61122:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0f05e5:::benchmark_data_smithi194_121512_object126224:head# 2024-04-29T14:27:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0f4f5c:::benchmark_data_smithi194_121512_object33731:head# 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0f5611:::benchmark_data_smithi194_121512_object100000:head# 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0fa1e7:::benchmark_data_smithi194_121512_object66823:head# 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c0fcfc2:::benchmark_data_smithi194_121512_object12639:head# 2024-04-29T14:27:12.841 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.843 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c104af8:::benchmark_data_smithi194_121512_object116692:head# 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c106437:::benchmark_data_smithi194_121512_object71953:head# 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c111791:::benchmark_data_smithi194_121512_object103557:head# 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c113dbf:::benchmark_data_smithi194_121512_object78597:head# 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1180a8:::benchmark_data_smithi194_121512_object16510:head# 2024-04-29T14:27:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c119388:::benchmark_data_smithi194_121512_object54148:head# 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c11cd96:::benchmark_data_smithi194_121512_object55395:head# 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c11d970:::benchmark_data_smithi194_121512_object48584:head# 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c11d9e3:::benchmark_data_smithi194_121512_object114014:head# 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c11dd41:::benchmark_data_smithi194_121512_object93561:head# 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c121276:::benchmark_data_smithi194_121512_object103142:head# 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c122169:::benchmark_data_smithi194_121512_object80510:head# 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1221a6:::benchmark_data_smithi194_121512_object34190:head# 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c122835:::benchmark_data_smithi194_121512_object133461:head# 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c12300e:::benchmark_data_smithi194_121512_object68405:head# 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1261c2:::benchmark_data_smithi194_121512_object67883:head# 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c12812a:::benchmark_data_smithi194_121512_object77907:head# 2024-04-29T14:27:12.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1334a9:::benchmark_data_smithi194_121512_object62059:head# 2024-04-29T14:27:12.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c13f5c9:::benchmark_data_smithi194_121512_object58935:head# 2024-04-29T14:27:12.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c14b5ba:::benchmark_data_smithi194_121512_object86191:head# 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c14b98a:::benchmark_data_smithi194_121512_object11460:head# 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c15d68a:::benchmark_data_smithi194_121512_object61475:head# 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1649da:::benchmark_data_smithi194_121512_object93647:head# 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1651ef:::benchmark_data_smithi194_121512_object116681:head# 2024-04-29T14:27:12.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c16b772:::benchmark_data_smithi194_121512_object61613:head# 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c17b9a0:::benchmark_data_smithi194_121512_object128037:head# 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c18eaca:::benchmark_data_smithi194_121512_object106592:head# 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1958ed:::benchmark_data_smithi194_121512_object15015:head# 2024-04-29T14:27:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1a1f63:::benchmark_data_smithi194_121512_object118669:head# 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1b7efa:::benchmark_data_smithi194_121512_object125329:head# 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1c0914:::benchmark_data_smithi194_121512_object114553:head# 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1cc97e:::benchmark_data_smithi194_121512_object12538:head# 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1cf0cd:::benchmark_data_smithi194_121512_object102526:head# 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1de35c:::benchmark_data_smithi194_121512_object20628:head# 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1def8b:::benchmark_data_smithi194_121512_object3561:head# 2024-04-29T14:27:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1f07f4:::benchmark_data_smithi194_121512_object45124:head# 2024-04-29T14:27:12.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1fa7e2:::benchmark_data_smithi194_121512_object16812:head# 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c1fdbaa:::benchmark_data_smithi194_121512_object75161:head# 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c208c31:::benchmark_data_smithi194_121512_object2988:head# 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c20ad84:::benchmark_data_smithi194_121512_object4538:head# 2024-04-29T14:27:12.863 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.868 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c20c223:::benchmark_data_smithi194_121512_object126855:head# 2024-04-29T14:27:12.868 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c20f9c5:::benchmark_data_smithi194_121512_object126930:head# 2024-04-29T14:27:12.870 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.870 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2266fa:::benchmark_data_smithi194_121512_object78153:head# 2024-04-29T14:27:12.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c226858:::benchmark_data_smithi194_121512_object113656:head# 2024-04-29T14:27:12.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c229354:::benchmark_data_smithi194_121512_object112360:head# 2024-04-29T14:27:12.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.871 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c22c2c6:::benchmark_data_smithi194_121512_object16786:head# 2024-04-29T14:27:12.871 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c24089c:::benchmark_data_smithi194_121512_object79225:head# 2024-04-29T14:27:12.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.872 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c241bfe:::benchmark_data_smithi194_121512_object113618:head# 2024-04-29T14:27:12.872 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c242d60:::benchmark_data_smithi194_121512_object24748:head# 2024-04-29T14:27:12.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c24a4fa:::benchmark_data_smithi194_121512_object15368:head# 2024-04-29T14:27:12.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.873 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c24ec1f:::benchmark_data_smithi194_121512_object79113:head# 2024-04-29T14:27:12.873 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c251a88:::benchmark_data_smithi194_121512_object125816:head# 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2577c0:::benchmark_data_smithi194_121512_object103200:head# 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c25be9f:::benchmark_data_smithi194_121512_object83014:head# 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c25df7e:::benchmark_data_smithi194_121512_object83887:head# 2024-04-29T14:27:12.874 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c262782:::benchmark_data_smithi194_121512_object11547:head# 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c269c72:::benchmark_data_smithi194_121512_object86753:head# 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c271e72:::benchmark_data_smithi194_121512_object58045:head# 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c27cc5b:::benchmark_data_smithi194_121512_object100054:head# 2024-04-29T14:27:12.875 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2963cb:::benchmark_data_smithi194_121512_object24261:head# 2024-04-29T14:27:12.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c29e5fc:::benchmark_data_smithi194_121512_object131188:head# 2024-04-29T14:27:12.876 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.876 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2b168e:::benchmark_data_smithi194_121512_object130988:head# 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2b58d6:::benchmark_data_smithi194_121512_object68177:head# 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2dfaf8:::benchmark_data_smithi194_121512_object109807:head# 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2e0060:::benchmark_data_smithi194_121512_object103785:head# 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.877 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2e0722:::benchmark_data_smithi194_121512_object100306:head# 2024-04-29T14:27:12.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.878 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c2fbab3:::benchmark_data_smithi194_121512_object48060:head# 2024-04-29T14:27:12.878 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.879 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3004d5:::benchmark_data_smithi194_121512_object23675:head# 2024-04-29T14:27:12.879 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:27:12.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:27:12.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c30388d:::benchmark_data_smithi194_121512_object71874:head# 2024-04-29T14:27:12.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c306734:::benchmark_data_smithi194_121512_object17238:head# 2024-04-29T14:27:12.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c30f918:::benchmark_data_smithi194_121512_object36669:head# 2024-04-29T14:27:12.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c31d566:::benchmark_data_smithi194_121512_object121828:head# 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c31dad3:::benchmark_data_smithi194_121512_object45562:head# 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c31dbfd:::benchmark_data_smithi194_121512_object91881:head# 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c31e9bd:::benchmark_data_smithi194_121512_object89246:head# 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c33d98d:::benchmark_data_smithi194_121512_object119805:head# 2024-04-29T14:27:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c347901:::benchmark_data_smithi194_121512_object99457:head# 2024-04-29T14:27:12.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c357550:::benchmark_data_smithi194_121512_object63035:head# 2024-04-29T14:27:12.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c363b0d:::benchmark_data_smithi194_121512_object127308:head# 2024-04-29T14:27:12.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c36b5f8:::benchmark_data_smithi194_121512_object31402:head# 2024-04-29T14:27:12.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c36b8bd:::benchmark_data_smithi194_121512_object65342:head# 2024-04-29T14:27:12.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c36f8ed:::benchmark_data_smithi194_121512_object22231:head# 2024-04-29T14:27:12.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c387f7d:::benchmark_data_smithi194_121512_object122913:head# 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c39809a:::benchmark_data_smithi194_121512_object88623:head# 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c39a7fd:::benchmark_data_smithi194_121512_object102539:head# 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c39d49c:::benchmark_data_smithi194_121512_object125093:head# 2024-04-29T14:27:12.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3a152f:::benchmark_data_smithi194_121512_object4280:head# 2024-04-29T14:27:12.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3a5fa6:::benchmark_data_smithi194_121512_object52204:head# 2024-04-29T14:27:12.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3afb41:::benchmark_data_smithi194_121512_object2164:head# 2024-04-29T14:27:12.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3b5f32:::benchmark_data_smithi194_121512_object54425:head# 2024-04-29T14:27:12.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3b685e:::benchmark_data_smithi194_121512_object93805:head# 2024-04-29T14:27:12.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3b76f2:::benchmark_data_smithi194_121512_object81933:head# 2024-04-29T14:27:12.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3b7b05:::benchmark_data_smithi194_121512_object20592:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3bcce2:::benchmark_data_smithi194_121512_object101348:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3c0df7:::benchmark_data_smithi194_121512_object100708:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3cbaaf:::benchmark_data_smithi194_121512_object81010:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3cf2c9:::benchmark_data_smithi194_121512_object15561:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3d0d7d:::benchmark_data_smithi194_121512_object30643:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3d4a76:::benchmark_data_smithi194_121512_object52379:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3e5763:::benchmark_data_smithi194_121512_object27667:head# 2024-04-29T14:27:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3e98b8:::benchmark_data_smithi194_121512_object52168:head# 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3ef4ec:::benchmark_data_smithi194_121512_object55181:head# 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c3f6008:::benchmark_data_smithi194_121512_object91140:head# 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4009f3:::benchmark_data_smithi194_121512_object118204:head# 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c40224a:::benchmark_data_smithi194_121512_object127807:head# 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c410003:::benchmark_data_smithi194_121512_object117109:head# 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c41fe5e:::benchmark_data_smithi194_121512_object75613:head# 2024-04-29T14:27:12.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c426f4a:::benchmark_data_smithi194_121512_object7544:head# 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c434998:::benchmark_data_smithi194_121512_object48684:head# 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c43e458:::benchmark_data_smithi194_121512_object84782:head# 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c441373:::benchmark_data_smithi194_121512_object60034:head# 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c444c45:::benchmark_data_smithi194_121512_object101167:head# 2024-04-29T14:27:12.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c44828e:::benchmark_data_smithi194_121512_object71454:head# 2024-04-29T14:27:12.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4610e7:::benchmark_data_smithi194_121512_object46277:head# 2024-04-29T14:27:12.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c48647b:::benchmark_data_smithi194_121512_object1275:head# 2024-04-29T14:27:12.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4872d8:::benchmark_data_smithi194_121512_object95185:head# 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4893a0:::benchmark_data_smithi194_121512_object46517:head# 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c48f99f:::benchmark_data_smithi194_121512_object18001:head# 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c490a73:::benchmark_data_smithi194_121512_object35587:head# 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c493406:::benchmark_data_smithi194_121512_object46784:head# 2024-04-29T14:27:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c497330:::benchmark_data_smithi194_121512_object127490:head# 2024-04-29T14:27:12.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c49a846:::benchmark_data_smithi194_121512_object31022:head# 2024-04-29T14:27:12.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c49acb8:::benchmark_data_smithi194_121512_object91026:head# 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4a1e83:::benchmark_data_smithi194_121512_object13681:head# 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4a507d:::benchmark_data_smithi194_121512_object114591:head# 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4b2c6e:::benchmark_data_smithi194_121512_object19701:head# 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4b927b:::benchmark_data_smithi194_121512_object22724:head# 2024-04-29T14:27:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4bc3d5:::benchmark_data_smithi194_121512_object92472:head# 2024-04-29T14:27:12.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.919 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4c2581:::benchmark_data_smithi194_121512_object51635:head# 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4cc48d:::benchmark_data_smithi194_121512_object86985:head# 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4d0360:::benchmark_data_smithi194_121512_object120332:head# 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4d4fcc:::benchmark_data_smithi194_121512_object46337:head# 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4e9d4d:::benchmark_data_smithi194_121512_object108032:head# 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4f8b99:::benchmark_data_smithi194_121512_object48605:head# 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4fa793:::benchmark_data_smithi194_121512_object103319:head# 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c4fec62:::benchmark_data_smithi194_121512_object42599:head# 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c51637c:::benchmark_data_smithi194_121512_object116999:head# 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c518e6f:::benchmark_data_smithi194_121512_object107374:head# 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c51af4a:::benchmark_data_smithi194_121512_object17324:head# 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c542103:::benchmark_data_smithi194_121512_object101390:head# 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c54bc49:::benchmark_data_smithi194_121512_object81694:head# 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c551d3f:::benchmark_data_smithi194_121512_object10041:head# 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.921 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5588e6:::benchmark_data_smithi194_121512_object86782:head# 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5592b9:::benchmark_data_smithi194_121512_object53908:head# 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c55c0bb:::benchmark_data_smithi194_121512_object52454:head# 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c55c4ec:::benchmark_data_smithi194_121512_object11153:head# 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5645e7:::benchmark_data_smithi194_121512_object35556:head# 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c566272:::benchmark_data_smithi194_121512_object124774:head# 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c56c131:::benchmark_data_smithi194_121512_object12545:head# 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.922 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c56ee64:::benchmark_data_smithi194_121512_object37241:head# 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c56f95e:::benchmark_data_smithi194_121512_object88052:head# 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c570327:::benchmark_data_smithi194_121512_object59981:head# 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c57a729:::benchmark_data_smithi194_121512_object34086:head# 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c583309:::benchmark_data_smithi194_121512_object98727:head# 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c591705:::benchmark_data_smithi194_121512_object26971:head# 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c595cb7:::benchmark_data_smithi194_121512_object100232:head# 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.923 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c597153:::benchmark_data_smithi194_121512_object123975:head# 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c59a8b3:::benchmark_data_smithi194_121512_object56265:head# 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c59affd:::benchmark_data_smithi194_121512_object39289:head# 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5a1079:::benchmark_data_smithi194_121512_object86757:head# 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5acd35:::benchmark_data_smithi194_121512_object51980:head# 2024-04-29T14:27:12.924 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5c36b6:::benchmark_data_smithi194_121512_object115175:head# 2024-04-29T14:27:12.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5ce473:::benchmark_data_smithi194_121512_object18693:head# 2024-04-29T14:27:12.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c5ed43b:::benchmark_data_smithi194_121512_object90224:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6091a1:::benchmark_data_smithi194_121512_object128369:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6094f9:::benchmark_data_smithi194_121512_object118276:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c60c732:::benchmark_data_smithi194_121512_object27009:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c60cb4e:::benchmark_data_smithi194_121512_object45652:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c60ec77:::benchmark_data_smithi194_121512_object66003:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c61628e:::benchmark_data_smithi194_121512_object83849:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c617486:::benchmark_data_smithi194_121512_object5213:head# 2024-04-29T14:27:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c620f2d:::benchmark_data_smithi194_121512_object2310:head# 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c625cf5:::benchmark_data_smithi194_121512_object59740:head# 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c63272f:::benchmark_data_smithi194_121512_object116092:head# 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6372e5:::benchmark_data_smithi194_121512_object129646:head# 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6373a5:::benchmark_data_smithi194_121512_object76558:head# 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c63ee3c:::benchmark_data_smithi194_121512_object121531:head# 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c64e51f:::benchmark_data_smithi194_121512_object103113:head# 2024-04-29T14:27:12.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c652d44:::benchmark_data_smithi194_121512_object90250:head# 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c66462d:::benchmark_data_smithi194_121512_object6798:head# 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c669305:::benchmark_data_smithi194_121512_object122017:head# 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c673f14:::benchmark_data_smithi194_121512_object13404:head# 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c693bdc:::benchmark_data_smithi194_121512_object10430:head# 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6952ac:::benchmark_data_smithi194_121512_object72775:head# 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c69a6e8:::benchmark_data_smithi194_121512_object2077:head# 2024-04-29T14:27:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c69cd51:::benchmark_data_smithi194_121512_object64071:head# 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6a1ca1:::benchmark_data_smithi194_121512_object4728:head# 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6acbef:::benchmark_data_smithi194_121512_object84650:head# 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6ba9b5:::benchmark_data_smithi194_121512_object112776:head# 2024-04-29T14:27:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6bac3f:::benchmark_data_smithi194_121512_object7686:head# 2024-04-29T14:27:12.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6c44d0:::benchmark_data_smithi194_121512_object11401:head# 2024-04-29T14:27:12.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6cf642:::benchmark_data_smithi194_121512_object7273:head# 2024-04-29T14:27:12.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6d981c:::benchmark_data_smithi194_121512_object118073:head# 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6e00fe:::benchmark_data_smithi194_121512_object26460:head# 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6f5e9c:::benchmark_data_smithi194_121512_object44809:head# 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c6fb8b7:::benchmark_data_smithi194_121512_object130671:head# 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c70b5d3:::benchmark_data_smithi194_121512_object3614:head# 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c712696:::benchmark_data_smithi194_121512_object108246:head# 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c714fd0:::benchmark_data_smithi194_121512_object57765:head# 2024-04-29T14:27:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c72e482:::benchmark_data_smithi194_121512_object77602:head# 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7353e8:::benchmark_data_smithi194_121512_object119121:head# 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c742708:::benchmark_data_smithi194_121512_object35461:head# 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7555c7:::benchmark_data_smithi194_121512_object69707:head# 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c765c0a:::benchmark_data_smithi194_121512_object108334:head# 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c76d2d7:::benchmark_data_smithi194_121512_object9881:head# 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c771d0d:::benchmark_data_smithi194_121512_object38037:head# 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7753c8:::benchmark_data_smithi194_121512_object68643:head# 2024-04-29T14:27:12.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.952 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c77d1b6:::benchmark_data_smithi194_121512_object71167:head# 2024-04-29T14:27:12.952 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.953 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c780c57:::benchmark_data_smithi194_121512_object109241:head# 2024-04-29T14:27:12.953 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7818e8:::benchmark_data_smithi194_121512_object96262:head# 2024-04-29T14:27:12.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7840b5:::benchmark_data_smithi194_121512_object131369:head# 2024-04-29T14:27:12.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.954 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c79a102:::benchmark_data_smithi194_121512_object81504:head# 2024-04-29T14:27:12.954 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c79d12b:::benchmark_data_smithi194_121512_object20373:head# 2024-04-29T14:27:12.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7a1314:::benchmark_data_smithi194_121512_object25490:head# 2024-04-29T14:27:12.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.955 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7a589d:::benchmark_data_smithi194_121512_object74402:head# 2024-04-29T14:27:12.955 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.956 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7aa7ec:::benchmark_data_smithi194_121512_object105447:head# 2024-04-29T14:27:12.956 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7aff5e:::benchmark_data_smithi194_121512_object5546:head# 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7bd130:::benchmark_data_smithi194_121512_object131803:head# 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7c1144:::benchmark_data_smithi194_121512_object29372:head# 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7c548b:::benchmark_data_smithi194_121512_object57320:head# 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7c7703:::benchmark_data_smithi194_121512_object63098:head# 2024-04-29T14:27:12.957 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7d444d:::benchmark_data_smithi194_121512_object22112:head# 2024-04-29T14:27:12.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7e1ccc:::benchmark_data_smithi194_121512_object3599:head# 2024-04-29T14:27:12.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.958 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7ea0f9:::benchmark_data_smithi194_121512_object86167:head# 2024-04-29T14:27:12.958 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7ed76e:::benchmark_data_smithi194_121512_object9803:head# 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7f9497:::benchmark_data_smithi194_121512_object97909:head# 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c7fd6a8:::benchmark_data_smithi194_121512_object84612:head# 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c803be2:::benchmark_data_smithi194_121512_object84897:head# 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8055b6:::benchmark_data_smithi194_121512_object77068:head# 2024-04-29T14:27:12.959 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c805695:::benchmark_data_smithi194_121512_object16203:head# 2024-04-29T14:27:12.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.960 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c808ca7:::benchmark_data_smithi194_121512_object8880:head# 2024-04-29T14:27:12.960 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.961 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c82c9c4:::benchmark_data_smithi194_121512_object19054:head# 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c844cc4:::benchmark_data_smithi194_121512_object17963:head# 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c84df77:::benchmark_data_smithi194_121512_object133431:head# 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8679c1:::benchmark_data_smithi194_121512_object69089:head# 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c874cb7:::benchmark_data_smithi194_121512_object25972:head# 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c87f8dc:::benchmark_data_smithi194_121512_object28327:head# 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c88c5f7:::benchmark_data_smithi194_121512_object58533:head# 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.962 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c88fd89:::benchmark_data_smithi194_121512_object18391:head# 2024-04-29T14:27:12.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c890804:::benchmark_data_smithi194_121512_object10313:head# 2024-04-29T14:27:12.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.963 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c893a1b:::benchmark_data_smithi194_121512_object27079:head# 2024-04-29T14:27:12.963 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c899d05:::benchmark_data_smithi194_121512_object38389:head# 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c89bfc6:::benchmark_data_smithi194_121512_object120584:head# 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c89f790:::benchmark_data_smithi194_121512_object28451:head# 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8a0c66:::benchmark_data_smithi194_121512_object77393:head# 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8b04f6:::benchmark_data_smithi194_121512_object45577:head# 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.967 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8b0728:::benchmark_data_smithi194_121512_object78227:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8b1e38:::benchmark_data_smithi194_121512_object13605:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8b3889:::benchmark_data_smithi194_121512_object87118:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8b6790:::benchmark_data_smithi194_121512_object100581:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8c9400:::benchmark_data_smithi194_121512_object96492:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8ca54d:::benchmark_data_smithi194_121512_object101036:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8cef0c:::benchmark_data_smithi194_121512_object68795:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8d81ff:::benchmark_data_smithi194_121512_object70678:head# 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.968 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8eb663:::benchmark_data_smithi194_121512_object6675:head# 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8f5840:::benchmark_data_smithi194_121512_object106611:head# 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c8f8be9:::benchmark_data_smithi194_121512_object79040:head# 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c909299:::benchmark_data_smithi194_121512_object23417:head# 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c90d149:::benchmark_data_smithi194_121512_object40817:head# 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c916a70:::benchmark_data_smithi194_121512_object63086:head# 2024-04-29T14:27:12.969 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9215dc:::benchmark_data_smithi194_121512_object93622:head# 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c923ab9:::benchmark_data_smithi194_121512_object39091:head# 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c938c32:::benchmark_data_smithi194_121512_object77364:head# 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c939db6:::benchmark_data_smithi194_121512_object103065:head# 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9489db:::benchmark_data_smithi194_121512_object72125:head# 2024-04-29T14:27:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9499ec:::benchmark_data_smithi194_121512_object6231:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c950653:::benchmark_data_smithi194_121512_object49426:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9532a3:::benchmark_data_smithi194_121512_object118678:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c95a1bd:::benchmark_data_smithi194_121512_object100740:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c95eb06:::benchmark_data_smithi194_121512_object38531:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c95ee25:::benchmark_data_smithi194_121512_object90184:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c96203f:::benchmark_data_smithi194_121512_object108511:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c964848:::benchmark_data_smithi194_121512_object83054:head# 2024-04-29T14:27:12.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c985cd4:::benchmark_data_smithi194_121512_object56952:head# 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c98f21c:::benchmark_data_smithi194_121512_object102305:head# 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c996c5f:::benchmark_data_smithi194_121512_object103724:head# 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9a3bf3:::benchmark_data_smithi194_121512_object125973:head# 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9a4eaa:::benchmark_data_smithi194_121512_object124488:head# 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9ae347:::benchmark_data_smithi194_121512_object51702:head# 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9afc5f:::benchmark_data_smithi194_121512_object47764:head# 2024-04-29T14:27:12.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9b100e:::benchmark_data_smithi194_121512_object26915:head# 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9b7992:::benchmark_data_smithi194_121512_object11996:head# 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9b9600:::benchmark_data_smithi194_121512_object65922:head# 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9bb982:::benchmark_data_smithi194_121512_object113263:head# 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9bcdc6:::benchmark_data_smithi194_121512_object17940:head# 2024-04-29T14:27:12.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9c7b58:::benchmark_data_smithi194_121512_object120656:head# 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9d1821:::benchmark_data_smithi194_121512_object97742:head# 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9dcad6:::benchmark_data_smithi194_121512_object96156:head# 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9df603:::benchmark_data_smithi194_121512_object69652:head# 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9e4495:::benchmark_data_smithi194_121512_object124928:head# 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9ea01e:::benchmark_data_smithi194_121512_object109554:head# 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9eb5d5:::benchmark_data_smithi194_121512_object101271:head# 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9f5785:::benchmark_data_smithi194_121512_object25713:head# 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9f8e9d:::benchmark_data_smithi194_121512_object31907:head# 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9fdae7:::benchmark_data_smithi194_121512_object98385:head# 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6c9ff980:::benchmark_data_smithi194_121512_object121664:head# 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca00fb7:::benchmark_data_smithi194_121512_object83439:head# 2024-04-29T14:27:12.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca06d27:::benchmark_data_smithi194_121512_object62977:head# 2024-04-29T14:27:12.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca0868f:::benchmark_data_smithi194_121512_object132851:head# 2024-04-29T14:27:12.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca087d0:::benchmark_data_smithi194_121512_object47985:head# 2024-04-29T14:27:12.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca2064b:::benchmark_data_smithi194_121512_object44787:head# 2024-04-29T14:27:12.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca3f36a:::benchmark_data_smithi194_121512_object119800:head# 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca4ad75:::benchmark_data_smithi194_121512_object88595:head# 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca5613a:::benchmark_data_smithi194_121512_object47139:head# 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca60fb3:::benchmark_data_smithi194_121512_object78763:head# 2024-04-29T14:27:12.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca6bb7e:::benchmark_data_smithi194_121512_object61082:head# 2024-04-29T14:27:12.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca74e45:::benchmark_data_smithi194_121512_object68010:head# 2024-04-29T14:27:12.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca74e88:::benchmark_data_smithi194_121512_object51208:head# 2024-04-29T14:27:12.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca781ee:::benchmark_data_smithi194_121512_object94470:head# 2024-04-29T14:27:12.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca7acd9:::benchmark_data_smithi194_121512_object51018:head# 2024-04-29T14:27:12.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca8241d:::benchmark_data_smithi194_121512_object90680:head# 2024-04-29T14:27:12.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ca9cb6f:::benchmark_data_smithi194_121512_object24350:head# 2024-04-29T14:27:12.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.993 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6caa3361:::benchmark_data_smithi194_121512_object69457:head# 2024-04-29T14:27:12.993 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6caac4d8:::benchmark_data_smithi194_121512_object99170:head# 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6caac4e0:::benchmark_data_smithi194_121512_object23167:head# 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cab0f64:::benchmark_data_smithi194_121512_object129814:head# 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cab5b86:::benchmark_data_smithi194_121512_object3004:head# 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cab88f6:::benchmark_data_smithi194_121512_object116447:head# 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cabdd68:::benchmark_data_smithi194_121512_object26734:head# 2024-04-29T14:27:12.994 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6caca3c6:::benchmark_data_smithi194_121512_object54545:head# 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cace3d2:::benchmark_data_smithi194_121512_object78719:head# 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cacf70f:::benchmark_data_smithi194_121512_object9312:head# 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cacfc7b:::benchmark_data_smithi194_121512_object24693:head# 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cadc7d6:::benchmark_data_smithi194_121512_object92470:head# 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cafdee6:::benchmark_data_smithi194_121512_object126881:head# 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.996 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb03fbb:::benchmark_data_smithi194_121512_object32500:head# 2024-04-29T14:27:12.997 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.997 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb12a09:::benchmark_data_smithi194_121512_object5231:head# 2024-04-29T14:27:12.997 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb149c9:::benchmark_data_smithi194_121512_object108351:head# 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb230b8:::benchmark_data_smithi194_121512_object57405:head# 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb31fdf:::benchmark_data_smithi194_121512_object71379:head# 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb32cfb:::benchmark_data_smithi194_121512_object100450:head# 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb35f81:::benchmark_data_smithi194_121512_object41006:head# 2024-04-29T14:27:12.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb3f08f:::benchmark_data_smithi194_121512_object67857:head# 2024-04-29T14:27:12.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb50c6b:::benchmark_data_smithi194_121512_object86392:head# 2024-04-29T14:27:12.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:12.999 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb51929:::benchmark_data_smithi194_121512_object104783:head# 2024-04-29T14:27:12.999 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb62a12:::benchmark_data_smithi194_121512_object11239:head# 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb6c703:::benchmark_data_smithi194_121512_object130819:head# 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb74ea8:::benchmark_data_smithi194_121512_object113463:head# 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb8412c:::benchmark_data_smithi194_121512_object89474:head# 2024-04-29T14:27:13.000 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb84d73:::benchmark_data_smithi194_121512_object110827:head# 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb87703:::benchmark_data_smithi194_121512_object114592:head# 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb8c136:::benchmark_data_smithi194_121512_object126673:head# 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb8cfef:::benchmark_data_smithi194_121512_object69875:head# 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb9543f:::benchmark_data_smithi194_121512_object80058:head# 2024-04-29T14:27:13.003 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb96f1e:::benchmark_data_smithi194_121512_object124440:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cb9b49e:::benchmark_data_smithi194_121512_object10307:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cba1686:::benchmark_data_smithi194_121512_object22815:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cba3047:::benchmark_data_smithi194_121512_object73676:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cba865a:::benchmark_data_smithi194_121512_object70246:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cba9a9b:::benchmark_data_smithi194_121512_object78282:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbaf544:::benchmark_data_smithi194_121512_object125981:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbbdfb5:::benchmark_data_smithi194_121512_object5353:head# 2024-04-29T14:27:13.004 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbca524:::benchmark_data_smithi194_121512_object30548:head# 2024-04-29T14:27:13.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbcb5b7:::benchmark_data_smithi194_121512_object60910:head# 2024-04-29T14:27:13.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.005 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbd47fa:::benchmark_data_smithi194_121512_object84674:head# 2024-04-29T14:27:13.005 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbd743d:::benchmark_data_smithi194_121512_object106076:head# 2024-04-29T14:27:13.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbe233e:::benchmark_data_smithi194_121512_object60864:head# 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbf5097:::benchmark_data_smithi194_121512_object132016:head# 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbf72d5:::benchmark_data_smithi194_121512_object3962:head# 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbf7667:::benchmark_data_smithi194_121512_object99919:head# 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cbfd9b0:::benchmark_data_smithi194_121512_object43388:head# 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc026ab:::benchmark_data_smithi194_121512_object67503:head# 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc036a5:::benchmark_data_smithi194_121512_object14181:head# 2024-04-29T14:27:13.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc0f919:::benchmark_data_smithi194_121512_object71680:head# 2024-04-29T14:27:13.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc10fab:::benchmark_data_smithi194_121512_object54931:head# 2024-04-29T14:27:13.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc19cba:::benchmark_data_smithi194_121512_object45637:head# 2024-04-29T14:27:13.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc1ad61:::benchmark_data_smithi194_121512_object95814:head# 2024-04-29T14:27:13.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc24d41:::benchmark_data_smithi194_121512_object103512:head# 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc34d19:::benchmark_data_smithi194_121512_object113896:head# 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc387ad:::benchmark_data_smithi194_121512_object62592:head# 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc3cbea:::benchmark_data_smithi194_121512_object120668:head# 2024-04-29T14:27:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc3e363:::benchmark_data_smithi194_121512_object130932:head# 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc3fdfb:::benchmark_data_smithi194_121512_object32172:head# 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc4cdea:::benchmark_data_smithi194_121512_object38040:head# 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc5f3dc:::benchmark_data_smithi194_121512_object101510:head# 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc695ed:::benchmark_data_smithi194_121512_object26438:head# 2024-04-29T14:27:13.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc69ce2:::benchmark_data_smithi194_121512_object57437:head# 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc70f01:::benchmark_data_smithi194_121512_object61004:head# 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc75a12:::benchmark_data_smithi194_121512_object47589:head# 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc7f845:::benchmark_data_smithi194_121512_object7114:head# 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc83154:::benchmark_data_smithi194_121512_object128318:head# 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc88829:::benchmark_data_smithi194_121512_object72344:head# 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc893ce:::benchmark_data_smithi194_121512_object121644:head# 2024-04-29T14:27:13.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc8a7a8:::benchmark_data_smithi194_121512_object93887:head# 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc97218:::benchmark_data_smithi194_121512_object43284:head# 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc9a4e8:::benchmark_data_smithi194_121512_object34458:head# 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cc9c10a:::benchmark_data_smithi194_121512_object110120:head# 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cca31f8:::benchmark_data_smithi194_121512_object58390:head# 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cca9a24:::benchmark_data_smithi194_121512_object79826:head# 2024-04-29T14:27:13.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccac270:::benchmark_data_smithi194_121512_object21184:head# 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccb717f:::benchmark_data_smithi194_121512_object17731:head# 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccbac8f:::benchmark_data_smithi194_121512_object63499:head# 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccbd4b5:::benchmark_data_smithi194_121512_object91437:head# 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccc1d1e:::benchmark_data_smithi194_121512_object108131:head# 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccd0392:::benchmark_data_smithi194_121512_object38268:head# 2024-04-29T14:27:13.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccd3471:::benchmark_data_smithi194_121512_object29675:head# 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccefbcf:::benchmark_data_smithi194_121512_object85954:head# 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ccf1ce4:::benchmark_data_smithi194_121512_object133026:head# 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd04936:::benchmark_data_smithi194_121512_object52394:head# 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd0e00d:::benchmark_data_smithi194_121512_object8912:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd182da:::benchmark_data_smithi194_121512_object18986:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd1aae2:::benchmark_data_smithi194_121512_object108532:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd1b857:::benchmark_data_smithi194_121512_object71922:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd1e8dd:::benchmark_data_smithi194_121512_object123114:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd26530:::benchmark_data_smithi194_121512_object24282:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd26c07:::benchmark_data_smithi194_121512_object40858:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd29c6f:::benchmark_data_smithi194_121512_object55797:head# 2024-04-29T14:27:13.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd32775:::benchmark_data_smithi194_121512_object84694:head# 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd39f47:::benchmark_data_smithi194_121512_object42838:head# 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd3f05b:::benchmark_data_smithi194_121512_object22260:head# 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd4280c:::benchmark_data_smithi194_121512_object25974:head# 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd4b96d:::benchmark_data_smithi194_121512_object100561:head# 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd543df:::benchmark_data_smithi194_121512_object1957:head# 2024-04-29T14:27:13.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd5f049:::benchmark_data_smithi194_121512_object68306:head# 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd62391:::benchmark_data_smithi194_121512_object109151:head# 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd6b515:::benchmark_data_smithi194_121512_object126228:head# 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd6c0a5:::benchmark_data_smithi194_121512_object55020:head# 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd6c3db:::benchmark_data_smithi194_121512_object4619:head# 2024-04-29T14:27:13.038 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cd96ba8:::benchmark_data_smithi194_121512_object7025:head# 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cda5add:::benchmark_data_smithi194_121512_object67375:head# 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cda7b8d:::benchmark_data_smithi194_121512_object33217:head# 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cda7b95:::benchmark_data_smithi194_121512_object60305:head# 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cda834d:::benchmark_data_smithi194_121512_object129122:head# 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdab632:::benchmark_data_smithi194_121512_object69783:head# 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdb1356:::benchmark_data_smithi194_121512_object26390:head# 2024-04-29T14:27:13.039 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdba2d4:::benchmark_data_smithi194_121512_object59279:head# 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdbfbdd:::benchmark_data_smithi194_121512_object10340:head# 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdcc2b8:::benchmark_data_smithi194_121512_object133084:head# 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdcc43a:::benchmark_data_smithi194_121512_object69055:head# 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdd17fc:::benchmark_data_smithi194_121512_object39972:head# 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cddbefd:::benchmark_data_smithi194_121512_object41964:head# 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cddf372:::benchmark_data_smithi194_121512_object4842:head# 2024-04-29T14:27:13.040 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cddfdf9:::benchmark_data_smithi194_121512_object68695:head# 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdef16c:::benchmark_data_smithi194_121512_object90066:head# 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdf7f86:::benchmark_data_smithi194_121512_object29697:head# 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cdfbdcb:::benchmark_data_smithi194_121512_object42115:head# 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce00ac8:::benchmark_data_smithi194_121512_object41900:head# 2024-04-29T14:27:13.041 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce0c04e:::benchmark_data_smithi194_121512_object50001:head# 2024-04-29T14:27:13.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce10f24:::benchmark_data_smithi194_121512_object59261:head# 2024-04-29T14:27:13.043 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.043 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce28d3f:::benchmark_data_smithi194_121512_object61894:head# 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce32ae6:::benchmark_data_smithi194_121512_object16850:head# 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce64e1b:::benchmark_data_smithi194_121512_object59374:head# 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce65cc5:::benchmark_data_smithi194_121512_object119899:head# 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce6bb40:::benchmark_data_smithi194_121512_object24493:head# 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce6d1c0:::benchmark_data_smithi194_121512_object75626:head# 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce713d2:::benchmark_data_smithi194_121512_object62523:head# 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.044 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce83bb0:::benchmark_data_smithi194_121512_object73617:head# 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce8a7da:::benchmark_data_smithi194_121512_object8901:head# 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce94d2a:::benchmark_data_smithi194_121512_object119682:head# 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ce9eba0:::benchmark_data_smithi194_121512_object154:head# 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cea1e56:::benchmark_data_smithi194_121512_object22984:head# 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ceb115a:::benchmark_data_smithi194_121512_object7607:head# 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ceb34df:::benchmark_data_smithi194_121512_object92526:head# 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.045 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cec9cd4:::benchmark_data_smithi194_121512_object100185:head# 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ceca475:::benchmark_data_smithi194_121512_object130377:head# 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cee0092:::benchmark_data_smithi194_121512_object94734:head# 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cee66f9:::benchmark_data_smithi194_121512_object74097:head# 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cef4c6b:::benchmark_data_smithi194_121512_object95618:head# 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.046 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cef5cc3:::benchmark_data_smithi194_121512_object18482:head# 2024-04-29T14:27:13.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.047 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cefc394:::benchmark_data_smithi194_121512_object61380:head# 2024-04-29T14:27:13.047 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf09d29:::benchmark_data_smithi194_121512_object15805:head# 2024-04-29T14:27:13.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf0b121:::benchmark_data_smithi194_121512_object13080:head# 2024-04-29T14:27:13.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.074 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf0b88a:::benchmark_data_smithi194_121512_object49006:head# 2024-04-29T14:27:13.074 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf1c4cb:::benchmark_data_smithi194_121512_object82822:head# 2024-04-29T14:27:13.078 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf1ed98:::benchmark_data_smithi194_121512_object119730:head# 2024-04-29T14:27:13.078 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.078 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf2556d:::benchmark_data_smithi194_121512_object28749:head# 2024-04-29T14:27:13.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.079 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf2700b:::benchmark_data_smithi194_121512_object31523:head# 2024-04-29T14:27:13.079 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf2b186:::benchmark_data_smithi194_121512_object17255:head# 2024-04-29T14:27:13.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf2c075:::benchmark_data_smithi194_121512_object87673:head# 2024-04-29T14:27:13.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf36cea:::benchmark_data_smithi194_121512_object53971:head# 2024-04-29T14:27:13.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf36e2c:::benchmark_data_smithi194_121512_object15505:head# 2024-04-29T14:27:13.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf3c341:::benchmark_data_smithi194_121512_object113420:head# 2024-04-29T14:27:13.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf464dc:::benchmark_data_smithi194_121512_object11261:head# 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf47000:::benchmark_data_smithi194_121512_object91096:head# 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf551c9:::benchmark_data_smithi194_121512_object38206:head# 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf5983b:::benchmark_data_smithi194_121512_object8040:head# 2024-04-29T14:27:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf639e0:::benchmark_data_smithi194_121512_object101605:head# 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf6c64e:::benchmark_data_smithi194_121512_object58784:head# 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf6cd31:::benchmark_data_smithi194_121512_object94643:head# 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf6cdea:::benchmark_data_smithi194_121512_object85270:head# 2024-04-29T14:27:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf70b2e:::benchmark_data_smithi194_121512_object56209:head# 2024-04-29T14:27:13.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf79643:::benchmark_data_smithi194_121512_object94269:head# 2024-04-29T14:27:13.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf80b40:::benchmark_data_smithi194_121512_object19289:head# 2024-04-29T14:27:13.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf86cee:::benchmark_data_smithi194_121512_object59329:head# 2024-04-29T14:27:13.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf8c27f:::benchmark_data_smithi194_121512_object111335:head# 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf91467:::benchmark_data_smithi194_121512_object58164:head# 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf9fba4:::benchmark_data_smithi194_121512_object58485:head# 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cf9fcbe:::benchmark_data_smithi194_121512_object96911:head# 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfb57d1:::benchmark_data_smithi194_121512_object23668:head# 2024-04-29T14:27:13.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfbd447:::benchmark_data_smithi194_121512_object88422:head# 2024-04-29T14:27:13.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfbe18c:::benchmark_data_smithi194_121512_object46127:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfc06c1:::benchmark_data_smithi194_121512_object49509:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfc19bb:::benchmark_data_smithi194_121512_object22644:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfc7ce5:::benchmark_data_smithi194_121512_object74662:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfc8add:::benchmark_data_smithi194_121512_object15596:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfcbacb:::benchmark_data_smithi194_121512_object130433:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfd0344:::benchmark_data_smithi194_121512_object99921:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfd440b:::benchmark_data_smithi194_121512_object13220:head# 2024-04-29T14:27:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfd4eea:::benchmark_data_smithi194_121512_object49535:head# 2024-04-29T14:27:13.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfe1a51:::benchmark_data_smithi194_121512_object116737:head# 2024-04-29T14:27:13.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfe43e1:::benchmark_data_smithi194_121512_object109996:head# 2024-04-29T14:27:13.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cfe6f3a:::benchmark_data_smithi194_121512_object133216:head# 2024-04-29T14:27:13.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cff125c:::benchmark_data_smithi194_121512_object84075:head# 2024-04-29T14:27:13.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cff4837:::benchmark_data_smithi194_121512_object52776:head# 2024-04-29T14:27:13.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6cff67b1:::benchmark_data_smithi194_121512_object97898:head# 2024-04-29T14:27:13.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d00a13f:::benchmark_data_smithi194_121512_object57154:head# 2024-04-29T14:27:13.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0107c7:::benchmark_data_smithi194_121512_object89425:head# 2024-04-29T14:27:13.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0134c0:::benchmark_data_smithi194_121512_object56321:head# 2024-04-29T14:27:13.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d01ac0f:::benchmark_data_smithi194_121512_object16723:head# 2024-04-29T14:27:13.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d02165e:::benchmark_data_smithi194_121512_object3899:head# 2024-04-29T14:27:13.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d026176:::benchmark_data_smithi194_121512_object60005:head# 2024-04-29T14:27:13.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d02cf0f:::benchmark_data_smithi194_121512_object44946:head# 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0395bc:::benchmark_data_smithi194_121512_object97085:head# 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d03c913:::benchmark_data_smithi194_121512_object59675:head# 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d060140:::benchmark_data_smithi194_121512_object109538:head# 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0678c1:::benchmark_data_smithi194_121512_object75727:head# 2024-04-29T14:27:13.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0829bb:::benchmark_data_smithi194_121512_object101188:head# 2024-04-29T14:27:13.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d099640:::benchmark_data_smithi194_121512_object121072:head# 2024-04-29T14:27:13.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d099f1c:::benchmark_data_smithi194_121512_object127105:head# 2024-04-29T14:27:13.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d09dd08:::benchmark_data_smithi194_121512_object53987:head# 2024-04-29T14:27:13.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.103 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0a4877:::benchmark_data_smithi194_121512_object122903:head# 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0aee0b:::benchmark_data_smithi194_121512_object52792:head# 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0bbc72:::benchmark_data_smithi194_121512_object93264:head# 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0c56d4:::benchmark_data_smithi194_121512_object49222:head# 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0c6c84:::benchmark_data_smithi194_121512_object47899:head# 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0d05a9:::benchmark_data_smithi194_121512_object76858:head# 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0d14a4:::benchmark_data_smithi194_121512_object15930:head# 2024-04-29T14:27:13.104 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0d345f:::benchmark_data_smithi194_121512_object47651:head# 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0d8023:::benchmark_data_smithi194_121512_object99713:head# 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0d8ab5:::benchmark_data_smithi194_121512_object102853:head# 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0eb849:::benchmark_data_smithi194_121512_object16885:head# 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d0fd578:::benchmark_data_smithi194_121512_object42520:head# 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d10212e:::benchmark_data_smithi194_121512_object96727:head# 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.107 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d104736:::benchmark_data_smithi194_121512_object65173:head# 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d10bfaa:::benchmark_data_smithi194_121512_object61281:head# 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d115dc8:::benchmark_data_smithi194_121512_object130471:head# 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d117526:::benchmark_data_smithi194_121512_object17472:head# 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d118faa:::benchmark_data_smithi194_121512_object132065:head# 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d12387b:::benchmark_data_smithi194_121512_object95383:head# 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d128b03:::benchmark_data_smithi194_121512_object112133:head# 2024-04-29T14:27:13.108 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d12e917:::benchmark_data_smithi194_121512_object102380:head# 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1394a1:::benchmark_data_smithi194_121512_object73798:head# 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d141aa8:::benchmark_data_smithi194_121512_object107837:head# 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1503ac:::benchmark_data_smithi194_121512_object43832:head# 2024-04-29T14:27:13.112 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1545fa:::benchmark_data_smithi194_121512_object120171:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d154878:::benchmark_data_smithi194_121512_object38607:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d15b85b:::benchmark_data_smithi194_121512_object16798:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d16ea96:::benchmark_data_smithi194_121512_object83424:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d17589c:::benchmark_data_smithi194_121512_object129816:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d179daf:::benchmark_data_smithi194_121512_object76445:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d17c044:::benchmark_data_smithi194_121512_object126664:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d17f3f5:::benchmark_data_smithi194_121512_object23604:head# 2024-04-29T14:27:13.113 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d181377:::benchmark_data_smithi194_121512_object80926:head# 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d197372:::benchmark_data_smithi194_121512_object89259:head# 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1a68eb:::benchmark_data_smithi194_121512_object38017:head# 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1d7ead:::benchmark_data_smithi194_121512_object95339:head# 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1db0e7:::benchmark_data_smithi194_121512_object68052:head# 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1dcbf1:::benchmark_data_smithi194_121512_object103182:head# 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1df985:::benchmark_data_smithi194_121512_object89944:head# 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.114 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1e3b00:::benchmark_data_smithi194_121512_object128996:head# 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1e3f2f:::benchmark_data_smithi194_121512_object21432:head# 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1ebd10:::benchmark_data_smithi194_121512_object126269:head# 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1f3580:::benchmark_data_smithi194_121512_object36542:head# 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1f5c81:::benchmark_data_smithi194_121512_object88281:head# 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1fa1e0:::benchmark_data_smithi194_121512_object113922:head# 2024-04-29T14:27:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1fdab8:::benchmark_data_smithi194_121512_object53242:head# 2024-04-29T14:27:13.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.116 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1fdd40:::benchmark_data_smithi194_121512_object121315:head# 2024-04-29T14:27:13.116 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d1fe761:::benchmark_data_smithi194_121512_object69822:head# 2024-04-29T14:27:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d200c5b:::benchmark_data_smithi194_121512_object83896:head# 2024-04-29T14:27:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d20360f:::benchmark_data_smithi194_121512_object81860:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d21fa80:::benchmark_data_smithi194_121512_object88049:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d22a658:::benchmark_data_smithi194_121512_object50132:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d23bd3c:::benchmark_data_smithi194_121512_object109473:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2450c5:::benchmark_data_smithi194_121512_object106781:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d25ab07:::benchmark_data_smithi194_121512_object1650:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2705a3:::benchmark_data_smithi194_121512_object102162:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2796f7:::benchmark_data_smithi194_121512_object12295:head# 2024-04-29T14:27:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2901d1:::benchmark_data_smithi194_121512_object118093:head# 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d297e33:::benchmark_data_smithi194_121512_object129818:head# 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d29840c:::benchmark_data_smithi194_121512_object39477:head# 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2a401c:::benchmark_data_smithi194_121512_object97836:head# 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2a79f2:::benchmark_data_smithi194_121512_object53397:head# 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2ab758:::benchmark_data_smithi194_121512_object106087:head# 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2bd545:::benchmark_data_smithi194_121512_object122952:head# 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2be359:::benchmark_data_smithi194_121512_object131162:head# 2024-04-29T14:27:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2d02e4:::benchmark_data_smithi194_121512_object85511:head# 2024-04-29T14:27:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2d6a25:::benchmark_data_smithi194_121512_object81724:head# 2024-04-29T14:27:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2e5ab1:::benchmark_data_smithi194_121512_object78582:head# 2024-04-29T14:27:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2eb9ac:::benchmark_data_smithi194_121512_object71845:head# 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2ef5af:::benchmark_data_smithi194_121512_object125413:head# 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2f8272:::benchmark_data_smithi194_121512_object70712:head# 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d2fdabd:::benchmark_data_smithi194_121512_object31948:head# 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d304e51:::benchmark_data_smithi194_121512_object13680:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d308367:::benchmark_data_smithi194_121512_object52461:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d309fb3:::benchmark_data_smithi194_121512_object81140:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d312953:::benchmark_data_smithi194_121512_object40047:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d313af2:::benchmark_data_smithi194_121512_object71195:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d313f12:::benchmark_data_smithi194_121512_object102804:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3151a3:::benchmark_data_smithi194_121512_object96501:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d315260:::benchmark_data_smithi194_121512_object39075:head# 2024-04-29T14:27:13.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d31da1a:::benchmark_data_smithi194_121512_object70683:head# 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3306af:::benchmark_data_smithi194_121512_object122031:head# 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d334c49:::benchmark_data_smithi194_121512_object86591:head# 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d33a9ba:::benchmark_data_smithi194_121512_object64586:head# 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d33d86a:::benchmark_data_smithi194_121512_object15740:head# 2024-04-29T14:27:13.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d34a0de:::benchmark_data_smithi194_121512_object53896:head# 2024-04-29T14:27:13.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d35abbe:::benchmark_data_smithi194_121512_object36226:head# 2024-04-29T14:27:13.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d36376d:::benchmark_data_smithi194_121512_object72382:head# 2024-04-29T14:27:13.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d36a995:::benchmark_data_smithi194_121512_object122404:head# 2024-04-29T14:27:13.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d37970d:::benchmark_data_smithi194_121512_object107333:head# 2024-04-29T14:27:13.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d37a31e:::benchmark_data_smithi194_121512_object95755:head# 2024-04-29T14:27:13.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d37c499:::benchmark_data_smithi194_121512_object54110:head# 2024-04-29T14:27:13.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d38280c:::benchmark_data_smithi194_121512_object72979:head# 2024-04-29T14:27:13.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d382de4:::benchmark_data_smithi194_121512_object47125:head# 2024-04-29T14:27:13.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d386674:::benchmark_data_smithi194_121512_object66733:head# 2024-04-29T14:27:13.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d395d23:::benchmark_data_smithi194_121512_object60204:head# 2024-04-29T14:27:13.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d398e16:::benchmark_data_smithi194_121512_object113435:head# 2024-04-29T14:27:13.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d39e342:::benchmark_data_smithi194_121512_object90779:head# 2024-04-29T14:27:13.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3b1e41:::benchmark_data_smithi194_121512_object116168:head# 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3b5149:::benchmark_data_smithi194_121512_object127606:head# 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3ba153:::benchmark_data_smithi194_121512_object100194:head# 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3c12e0:::benchmark_data_smithi194_121512_object87585:head# 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3cb9dd:::benchmark_data_smithi194_121512_object131192:head# 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3d03bd:::benchmark_data_smithi194_121512_object18259:head# 2024-04-29T14:27:13.142 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3d06c5:::benchmark_data_smithi194_121512_object110916:head# 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3d4f29:::benchmark_data_smithi194_121512_object49460:head# 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3d9518:::benchmark_data_smithi194_121512_object62750:head# 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3e2f97:::benchmark_data_smithi194_121512_object52330:head# 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3e38fa:::benchmark_data_smithi194_121512_object71616:head# 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3eb263:::benchmark_data_smithi194_121512_object6515:head# 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3f4655:::benchmark_data_smithi194_121512_object1663:head# 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3f9445:::benchmark_data_smithi194_121512_object4608:head# 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3fc27d:::benchmark_data_smithi194_121512_object32890:head# 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3fcbe5:::benchmark_data_smithi194_121512_object90752:head# 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3fdeb6:::benchmark_data_smithi194_121512_object115488:head# 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3fe634:::benchmark_data_smithi194_121512_object7308:head# 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3ff01d:::benchmark_data_smithi194_121512_object133377:head# 2024-04-29T14:27:13.144 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.145 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d3ff7eb:::benchmark_data_smithi194_121512_object108921:head# 2024-04-29T14:27:13.145 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.145 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d41b55e:::benchmark_data_smithi194_121512_object8727:head# 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d41eddc:::benchmark_data_smithi194_121512_object15896:head# 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d423703:::benchmark_data_smithi194_121512_object49273:head# 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d42a358:::benchmark_data_smithi194_121512_object62830:head# 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d454329:::benchmark_data_smithi194_121512_object32312:head# 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d455fdb:::benchmark_data_smithi194_121512_object42050:head# 2024-04-29T14:27:13.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d45ce6b:::benchmark_data_smithi194_121512_object119717:head# 2024-04-29T14:27:13.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4604dc:::benchmark_data_smithi194_121512_object105597:head# 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d469b85:::benchmark_data_smithi194_121512_object68087:head# 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d472338:::benchmark_data_smithi194_121512_object59665:head# 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d482879:::benchmark_data_smithi194_121512_object108764:head# 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4879ef:::benchmark_data_smithi194_121512_object76871:head# 2024-04-29T14:27:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4881bb:::benchmark_data_smithi194_121512_object88372:head# 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d491647:::benchmark_data_smithi194_121512_object125331:head# 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4948f7:::benchmark_data_smithi194_121512_object38382:head# 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d496891:::benchmark_data_smithi194_121512_object121241:head# 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4a79dc:::benchmark_data_smithi194_121512_object1988:head# 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4b7cac:::benchmark_data_smithi194_121512_object9550:head# 2024-04-29T14:27:13.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4b7cc0:::benchmark_data_smithi194_121512_object82974:head# 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4c6ac0:::benchmark_data_smithi194_121512_object7294:head# 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4cab5b:::benchmark_data_smithi194_121512_object74117:head# 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4e27fc:::benchmark_data_smithi194_121512_object125656:head# 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4e2e10:::benchmark_data_smithi194_121512_object108242:head# 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4e4135:::benchmark_data_smithi194_121512_object72297:head# 2024-04-29T14:27:13.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4ebebc:::benchmark_data_smithi194_121512_object68071:head# 2024-04-29T14:27:13.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d4f2502:::benchmark_data_smithi194_121512_object98520:head# 2024-04-29T14:27:13.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d50449d:::benchmark_data_smithi194_121512_object2979:head# 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d508925:::benchmark_data_smithi194_121512_object49243:head# 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d50ef93:::benchmark_data_smithi194_121512_object57473:head# 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d50f713:::benchmark_data_smithi194_121512_object8301:head# 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d51c45a:::benchmark_data_smithi194_121512_object84087:head# 2024-04-29T14:27:13.154 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.154 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d52c4c3:::benchmark_data_smithi194_121512_object83819:head# 2024-04-29T14:27:13.154 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.154 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d52d064:::benchmark_data_smithi194_121512_object37970:head# 2024-04-29T14:27:13.154 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.155 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5351f8:::benchmark_data_smithi194_121512_object129762:head# 2024-04-29T14:27:13.155 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.155 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d539e73:::benchmark_data_smithi194_121512_object28391:head# 2024-04-29T14:27:13.155 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.155 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d53c314:::benchmark_data_smithi194_121512_object126108:head# 2024-04-29T14:27:13.155 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d552717:::benchmark_data_smithi194_121512_object111386:head# 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d555668:::benchmark_data_smithi194_121512_object39843:head# 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d570d6f:::benchmark_data_smithi194_121512_object39757:head# 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d59e27d:::benchmark_data_smithi194_121512_object102479:head# 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d59f978:::benchmark_data_smithi194_121512_object34211:head# 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.164 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d59fd76:::benchmark_data_smithi194_121512_object125775:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5a20a7:::benchmark_data_smithi194_121512_object42181:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5aaae3:::benchmark_data_smithi194_121512_object36568:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5b1374:::benchmark_data_smithi194_121512_object83650:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5b8679:::benchmark_data_smithi194_121512_object18290:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5c2813:::benchmark_data_smithi194_121512_object47592:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5c5bcc:::benchmark_data_smithi194_121512_object91738:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5cc059:::benchmark_data_smithi194_121512_object67896:head# 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5e06f0:::benchmark_data_smithi194_121512_object120198:head# 2024-04-29T14:27:13.166 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5e138f:::benchmark_data_smithi194_121512_object31950:head# 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d5fb686:::benchmark_data_smithi194_121512_object86578:head# 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d608396:::benchmark_data_smithi194_121512_object89529:head# 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d60e220:::benchmark_data_smithi194_121512_object50204:head# 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.169 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d611a42:::benchmark_data_smithi194_121512_object82102:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d611f7e:::benchmark_data_smithi194_121512_object94738:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d61becf:::benchmark_data_smithi194_121512_object132385:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d634e6b:::benchmark_data_smithi194_121512_object20912:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d63d00e:::benchmark_data_smithi194_121512_object42570:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d648d8f:::benchmark_data_smithi194_121512_object59495:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d661d93:::benchmark_data_smithi194_121512_object120389:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d66bf60:::benchmark_data_smithi194_121512_object111726:head# 2024-04-29T14:27:13.170 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d688bcd:::benchmark_data_smithi194_121512_object102744:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d68ed23:::benchmark_data_smithi194_121512_object80029:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6908d8:::benchmark_data_smithi194_121512_object25330:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d690ab6:::benchmark_data_smithi194_121512_object48613:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d69713d:::benchmark_data_smithi194_121512_object113200:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d69a0e5:::benchmark_data_smithi194_121512_object40319:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6a47b6:::benchmark_data_smithi194_121512_object9879:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6af579:::benchmark_data_smithi194_121512_object26019:head# 2024-04-29T14:27:13.171 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.172 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6b92d4:::benchmark_data_smithi194_121512_object91799:head# 2024-04-29T14:27:13.172 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.172 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6c0593:::benchmark_data_smithi194_121512_object37116:head# 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6c5e45:::benchmark_data_smithi194_121512_object114168:head# 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6cce1e:::benchmark_data_smithi194_121512_object65695:head# 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6cdd2d:::benchmark_data_smithi194_121512_object30763:head# 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6d581e:::benchmark_data_smithi194_121512_object47828:head# 2024-04-29T14:27:13.173 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.180 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6d5a66:::benchmark_data_smithi194_121512_object44686:head# 2024-04-29T14:27:13.180 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6d6387:::benchmark_data_smithi194_121512_object95324:head# 2024-04-29T14:27:13.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6db56c:::benchmark_data_smithi194_121512_object32661:head# 2024-04-29T14:27:13.181 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.181 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d6e1ac0:::benchmark_data_smithi194_121512_object67245:head# 2024-04-29T14:27:13.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d701030:::benchmark_data_smithi194_121512_object68276:head# 2024-04-29T14:27:13.182 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.182 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d704a75:::benchmark_data_smithi194_121512_object110992:head# 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d70e892:::benchmark_data_smithi194_121512_object105760:head# 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7168e7:::benchmark_data_smithi194_121512_object114561:head# 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d719a6e:::benchmark_data_smithi194_121512_object27987:head# 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d720774:::benchmark_data_smithi194_121512_object115408:head# 2024-04-29T14:27:13.183 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d726063:::benchmark_data_smithi194_121512_object129911:head# 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7276a8:::benchmark_data_smithi194_121512_object92549:head# 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d730fcc:::benchmark_data_smithi194_121512_object818:head# 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d73980c:::benchmark_data_smithi194_121512_object49254:head# 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d73b40c:::benchmark_data_smithi194_121512_object51439:head# 2024-04-29T14:27:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.185 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d76744f:::benchmark_data_smithi194_121512_object53830:head# 2024-04-29T14:27:13.185 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.185 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d76d64b:::benchmark_data_smithi194_121512_object11415:head# 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d77b07c:::benchmark_data_smithi194_121512_object56591:head# 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7844a6:::benchmark_data_smithi194_121512_object120466:head# 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7887a6:::benchmark_data_smithi194_121512_object30901:head# 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d794636:::benchmark_data_smithi194_121512_object120035:head# 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d79cd9a:::benchmark_data_smithi194_121512_object79093:head# 2024-04-29T14:27:13.186 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d79f8bf:::benchmark_data_smithi194_121512_object29640:head# 2024-04-29T14:27:13.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7a1273:::benchmark_data_smithi194_121512_object107063:head# 2024-04-29T14:27:13.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7a31ae:::benchmark_data_smithi194_121512_object128688:head# 2024-04-29T14:27:13.188 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.188 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7a345a:::benchmark_data_smithi194_121512_object85875:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7aed3b:::benchmark_data_smithi194_121512_object11631:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7b0909:::benchmark_data_smithi194_121512_object34278:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7b90e6:::benchmark_data_smithi194_121512_object79348:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7c5327:::benchmark_data_smithi194_121512_object116783:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7c55b1:::benchmark_data_smithi194_121512_object114874:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7d8445:::benchmark_data_smithi194_121512_object34999:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7dbaca:::benchmark_data_smithi194_121512_object18049:head# 2024-04-29T14:27:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.191 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7df5f1:::benchmark_data_smithi194_121512_object117836:head# 2024-04-29T14:27:13.191 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.191 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7e1bbf:::benchmark_data_smithi194_121512_object88978:head# 2024-04-29T14:27:13.191 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.191 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7f452b:::benchmark_data_smithi194_121512_object99777:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d7fd544:::benchmark_data_smithi194_121512_object40519:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d80a29f:::benchmark_data_smithi194_121512_object16594:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d80adf9:::benchmark_data_smithi194_121512_object57548:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d80f63e:::benchmark_data_smithi194_121512_object42054:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d81e6e1:::benchmark_data_smithi194_121512_object17146:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d821c44:::benchmark_data_smithi194_121512_object52890:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d824120:::benchmark_data_smithi194_121512_object60779:head# 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.192 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d828b92:::benchmark_data_smithi194_121512_object106732:head# 2024-04-29T14:27:13.193 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.193 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d82ade5:::benchmark_data_smithi194_121512_object96106:head# 2024-04-29T14:27:13.193 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d83ac91:::benchmark_data_smithi194_121512_object93451:head# 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d83e80f:::benchmark_data_smithi194_121512_object70213:head# 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d848577:::benchmark_data_smithi194_121512_object9730:head# 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d84893c:::benchmark_data_smithi194_121512_object92824:head# 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8548e2:::benchmark_data_smithi194_121512_object1669:head# 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.194 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d854bb8:::benchmark_data_smithi194_121512_object88626:head# 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d858d9d:::benchmark_data_smithi194_121512_object950:head# 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d85d701:::benchmark_data_smithi194_121512_object52886:head# 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d85fc59:::benchmark_data_smithi194_121512_object35906:head# 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d862b61:::benchmark_data_smithi194_121512_object5138:head# 2024-04-29T14:27:13.195 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.196 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8639d4:::benchmark_data_smithi194_121512_object27879:head# 2024-04-29T14:27:13.196 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.205 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8651db:::benchmark_data_smithi194_121512_object82300:head# 2024-04-29T14:27:13.205 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.205 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d86582a:::benchmark_data_smithi194_121512_object78072:head# 2024-04-29T14:27:13.205 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d86624a:::benchmark_data_smithi194_121512_object52898:head# 2024-04-29T14:27:13.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d867e19:::benchmark_data_smithi194_121512_object85665:head# 2024-04-29T14:27:13.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.206 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8698b7:::benchmark_data_smithi194_121512_object18665:head# 2024-04-29T14:27:13.206 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.207 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d87203c:::benchmark_data_smithi194_121512_object2330:head# 2024-04-29T14:27:13.207 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d87e6f9:::benchmark_data_smithi194_121512_object60592:head# 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d87e8f4:::benchmark_data_smithi194_121512_object7197:head# 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d88fdcf:::benchmark_data_smithi194_121512_object94838:head# 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d89eb31:::benchmark_data_smithi194_121512_object5491:head# 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.208 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8cd37e:::benchmark_data_smithi194_121512_object55763:head# 2024-04-29T14:27:13.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8d8c43:::benchmark_data_smithi194_121512_object72118:head# 2024-04-29T14:27:13.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.209 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8de6f8:::benchmark_data_smithi194_121512_object25030:head# 2024-04-29T14:27:13.209 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.210 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8e1f99:::benchmark_data_smithi194_121512_object109590:head# 2024-04-29T14:27:13.210 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.210 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8ebab8:::benchmark_data_smithi194_121512_object28722:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8f0a4d:::benchmark_data_smithi194_121512_object59564:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8f56a6:::benchmark_data_smithi194_121512_object4105:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d8f59b8:::benchmark_data_smithi194_121512_object85:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d909310:::benchmark_data_smithi194_121512_object12104:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9095d2:::benchmark_data_smithi194_121512_object83698:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d90c804:::benchmark_data_smithi194_121512_object35284:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d91ad2d:::benchmark_data_smithi194_121512_object23943:head# 2024-04-29T14:27:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d91cc27:::benchmark_data_smithi194_121512_object97990:head# 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d920387:::benchmark_data_smithi194_121512_object22265:head# 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d921910:::benchmark_data_smithi194_121512_object53147:head# 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d924456:::benchmark_data_smithi194_121512_object12875:head# 2024-04-29T14:27:13.213 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d938860:::benchmark_data_smithi194_121512_object72784:head# 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d93bffa:::benchmark_data_smithi194_121512_object4171:head# 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d93f51f:::benchmark_data_smithi194_121512_object34459:head# 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d942b57:::benchmark_data_smithi194_121512_object28350:head# 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d94bdad:::benchmark_data_smithi194_121512_object71585:head# 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d957f25:::benchmark_data_smithi194_121512_object98710:head# 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d958903:::benchmark_data_smithi194_121512_object65166:head# 2024-04-29T14:27:13.214 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.215 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d964573:::benchmark_data_smithi194_121512_object97513:head# 2024-04-29T14:27:13.215 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.215 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d966299:::benchmark_data_smithi194_121512_object22278:head# 2024-04-29T14:27:13.215 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.215 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d96d9c0:::benchmark_data_smithi194_121512_object44669:head# 2024-04-29T14:27:13.215 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9707ed:::benchmark_data_smithi194_121512_object25478:head# 2024-04-29T14:27:13.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.216 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d975b84:::benchmark_data_smithi194_121512_object999:head# 2024-04-29T14:27:13.216 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9766b1:::benchmark_data_smithi194_121512_object127047:head# 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d97e5bd:::benchmark_data_smithi194_121512_object99659:head# 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d97f27a:::benchmark_data_smithi194_121512_object25238:head# 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d97f68d:::benchmark_data_smithi194_121512_object23775:head# 2024-04-29T14:27:13.217 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d983ce8:::benchmark_data_smithi194_121512_object81342:head# 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d98e08f:::benchmark_data_smithi194_121512_object48532:head# 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d98ee2e:::benchmark_data_smithi194_121512_object101143:head# 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9a5a3d:::benchmark_data_smithi194_121512_object120790:head# 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9aacb0:::benchmark_data_smithi194_121512_object132738:head# 2024-04-29T14:27:13.218 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.219 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9b2855:::benchmark_data_smithi194_121512_object5730:head# 2024-04-29T14:27:13.219 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.219 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9b4bf2:::benchmark_data_smithi194_121512_object7590:head# 2024-04-29T14:27:13.219 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.219 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9b686f:::benchmark_data_smithi194_121512_object62284:head# 2024-04-29T14:27:13.219 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9b6dbd:::benchmark_data_smithi194_121512_object46361:head# 2024-04-29T14:27:13.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.220 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9b6fbf:::benchmark_data_smithi194_121512_object117010:head# 2024-04-29T14:27:13.220 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.224 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9c075b:::benchmark_data_smithi194_121512_object132384:head# 2024-04-29T14:27:13.224 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.225 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9c3587:::benchmark_data_smithi194_121512_object25061:head# 2024-04-29T14:27:13.225 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.225 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9c56b9:::benchmark_data_smithi194_121512_object79235:head# 2024-04-29T14:27:13.225 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.225 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9cbaa0:::benchmark_data_smithi194_121512_object80159:head# 2024-04-29T14:27:13.225 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9cecb8:::benchmark_data_smithi194_121512_object43071:head# 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9de406:::benchmark_data_smithi194_121512_object88261:head# 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9e49ac:::benchmark_data_smithi194_121512_object48180:head# 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9e57ba:::benchmark_data_smithi194_121512_object7262:head# 2024-04-29T14:27:13.226 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6d9e6828:::benchmark_data_smithi194_121512_object78403:head# 2024-04-29T14:27:13.227 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.227 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da060d2:::benchmark_data_smithi194_121512_object119315:head# 2024-04-29T14:27:13.227 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da0c133:::benchmark_data_smithi194_121512_object69029:head# 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da1392d:::benchmark_data_smithi194_121512_object111768:head# 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da1661c:::benchmark_data_smithi194_121512_object44821:head# 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da1c76a:::benchmark_data_smithi194_121512_object79649:head# 2024-04-29T14:27:13.229 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da1e4b1:::benchmark_data_smithi194_121512_object110108:head# 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da22d35:::benchmark_data_smithi194_121512_object99376:head# 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da3ab57:::benchmark_data_smithi194_121512_object28450:head# 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da493c2:::benchmark_data_smithi194_121512_object51852:head# 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da4cb68:::benchmark_data_smithi194_121512_object34679:head# 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da4eab5:::benchmark_data_smithi194_121512_object81521:head# 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da5f332:::benchmark_data_smithi194_121512_object44303:head# 2024-04-29T14:27:13.230 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.231 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da64cf2:::benchmark_data_smithi194_121512_object81757:head# 2024-04-29T14:27:13.231 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.231 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da6e501:::benchmark_data_smithi194_121512_object11576:head# 2024-04-29T14:27:13.231 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da7a51e:::benchmark_data_smithi194_121512_object107078:head# 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da7c0ff:::benchmark_data_smithi194_121512_object23616:head# 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da857c4:::benchmark_data_smithi194_121512_object127698:head# 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da889d1:::benchmark_data_smithi194_121512_object33857:head# 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.234 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6da91e70:::benchmark_data_smithi194_121512_object15579:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6daa83bb:::benchmark_data_smithi194_121512_object28835:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dab1525:::benchmark_data_smithi194_121512_object6794:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dab15e6:::benchmark_data_smithi194_121512_object72455:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dab1a4a:::benchmark_data_smithi194_121512_object41819:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dac6bf6:::benchmark_data_smithi194_121512_object78877:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dad137f:::benchmark_data_smithi194_121512_object90085:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dad418d:::benchmark_data_smithi194_121512_object131154:head# 2024-04-29T14:27:13.235 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dad8854:::benchmark_data_smithi194_121512_object34536:head# 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dae2746:::benchmark_data_smithi194_121512_object54212:head# 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dae34c3:::benchmark_data_smithi194_121512_object66809:head# 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dae63ec:::benchmark_data_smithi194_121512_object105433:head# 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6daea169:::benchmark_data_smithi194_121512_object79878:head# 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6daee1ef:::benchmark_data_smithi194_121512_object24388:head# 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.236 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db0d8c8:::benchmark_data_smithi194_121512_object12801:head# 2024-04-29T14:27:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db0fc02:::benchmark_data_smithi194_121512_object20139:head# 2024-04-29T14:27:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db1961a:::benchmark_data_smithi194_121512_object43805:head# 2024-04-29T14:27:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db1e24a:::benchmark_data_smithi194_121512_object83932:head# 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db2120b:::benchmark_data_smithi194_121512_object92517:head# 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db21a9f:::benchmark_data_smithi194_121512_object10867:head# 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db228e1:::benchmark_data_smithi194_121512_object20016:head# 2024-04-29T14:27:13.241 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db2756d:::benchmark_data_smithi194_121512_object102914:head# 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db2ac00:::benchmark_data_smithi194_121512_object102712:head# 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db35daf:::benchmark_data_smithi194_121512_object112048:head# 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db387e2:::benchmark_data_smithi194_121512_object1776:head# 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db3a9b9:::benchmark_data_smithi194_121512_object71984:head# 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db3dc1d:::benchmark_data_smithi194_121512_object88452:head# 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db40536:::benchmark_data_smithi194_121512_object120324:head# 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db49801:::benchmark_data_smithi194_121512_object26277:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db4e595:::benchmark_data_smithi194_121512_object125947:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db505f2:::benchmark_data_smithi194_121512_object24218:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db57925:::benchmark_data_smithi194_121512_object38173:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db58de5:::benchmark_data_smithi194_121512_object18364:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db5c01c:::benchmark_data_smithi194_121512_object55794:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db69958:::benchmark_data_smithi194_121512_object35361:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db6fd90:::benchmark_data_smithi194_121512_object15294:head# 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db7c4e5:::benchmark_data_smithi194_121512_object728:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db7f3b5:::benchmark_data_smithi194_121512_object61153:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db8d813:::benchmark_data_smithi194_121512_object25309:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6db940a7:::benchmark_data_smithi194_121512_object58459:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dba2bd3:::benchmark_data_smithi194_121512_object126569:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbacb2e:::benchmark_data_smithi194_121512_object15600:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbb063a:::benchmark_data_smithi194_121512_object108590:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbb2d34:::benchmark_data_smithi194_121512_object11628:head# 2024-04-29T14:27:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.249 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbbbe64:::benchmark_data_smithi194_121512_object18458:head# 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbc5cbc:::benchmark_data_smithi194_121512_object30730:head# 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbcbd47:::benchmark_data_smithi194_121512_object47782:head# 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbcdcc9:::benchmark_data_smithi194_121512_object97438:head# 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbd2ec9:::benchmark_data_smithi194_121512_object11828:head# 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbd2ed4:::benchmark_data_smithi194_121512_object19060:head# 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.250 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbd7f30:::benchmark_data_smithi194_121512_object58348:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbe8f10:::benchmark_data_smithi194_121512_object68255:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbea6a0:::benchmark_data_smithi194_121512_object93458:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbea8e1:::benchmark_data_smithi194_121512_object62949:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbf0d31:::benchmark_data_smithi194_121512_object63963:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbfc81b:::benchmark_data_smithi194_121512_object114844:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dbfeb03:::benchmark_data_smithi194_121512_object54363:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc08296:::benchmark_data_smithi194_121512_object75504:head# 2024-04-29T14:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc139c5:::benchmark_data_smithi194_121512_object13357:head# 2024-04-29T14:27:13.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc18875:::benchmark_data_smithi194_121512_object49189:head# 2024-04-29T14:27:13.252 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.252 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc1ab86:::benchmark_data_smithi194_121512_object30906:head# 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc2122d:::benchmark_data_smithi194_121512_object35492:head# 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc23c57:::benchmark_data_smithi194_121512_object89987:head# 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc2a02b:::benchmark_data_smithi194_121512_object80977:head# 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc2c040:::benchmark_data_smithi194_121512_object39019:head# 2024-04-29T14:27:13.253 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc30c34:::benchmark_data_smithi194_121512_object35334:head# 2024-04-29T14:27:13.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc384c6:::benchmark_data_smithi194_121512_object46858:head# 2024-04-29T14:27:13.254 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.254 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc4ac78:::benchmark_data_smithi194_121512_object78261:head# 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc4d01c:::benchmark_data_smithi194_121512_object129583:head# 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc5dc2d:::benchmark_data_smithi194_121512_object27428:head# 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc65b16:::benchmark_data_smithi194_121512_object90376:head# 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc68518:::benchmark_data_smithi194_121512_object55887:head# 2024-04-29T14:27:13.255 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc6a46f:::benchmark_data_smithi194_121512_object129085:head# 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc707e4:::benchmark_data_smithi194_121512_object87081:head# 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc718af:::benchmark_data_smithi194_121512_object39830:head# 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc7f76c:::benchmark_data_smithi194_121512_object54535:head# 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.260 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dc87327:::benchmark_data_smithi194_121512_object42424:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dca02e8:::benchmark_data_smithi194_121512_object125831:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dca208e:::benchmark_data_smithi194_121512_object39318:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dca53fe:::benchmark_data_smithi194_121512_object107728:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dcb4f8e:::benchmark_data_smithi194_121512_object123081:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dcbb564:::benchmark_data_smithi194_121512_object123178:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dcbc72a:::benchmark_data_smithi194_121512_object119048:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dcc2248:::benchmark_data_smithi194_121512_object1729:head# 2024-04-29T14:27:13.261 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dcd8aea:::benchmark_data_smithi194_121512_object109928:head# 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dced2f0:::benchmark_data_smithi194_121512_object84501:head# 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dced967:::benchmark_data_smithi194_121512_object31557:head# 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dcf047b:::benchmark_data_smithi194_121512_object27490:head# 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd082a9:::benchmark_data_smithi194_121512_object17618:head# 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd11ca1:::benchmark_data_smithi194_121512_object44690:head# 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd17626:::benchmark_data_smithi194_121512_object44225:head# 2024-04-29T14:27:13.262 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.263 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd192d1:::benchmark_data_smithi194_121512_object127558:head# 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd1ffa2:::benchmark_data_smithi194_121512_object72901:head# 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd214c6:::benchmark_data_smithi194_121512_object80577:head# 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd254f3:::benchmark_data_smithi194_121512_object1648:head# 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd40915:::benchmark_data_smithi194_121512_object90233:head# 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd41906:::benchmark_data_smithi194_121512_object116543:head# 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd47905:::benchmark_data_smithi194_121512_object82376:head# 2024-04-29T14:27:13.264 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.274 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd532f2:::benchmark_data_smithi194_121512_object110606:head# 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd56d28:::benchmark_data_smithi194_121512_object123957:head# 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd60c06:::benchmark_data_smithi194_121512_object48118:head# 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd6426c:::benchmark_data_smithi194_121512_object102351:head# 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dd6b92f:::benchmark_data_smithi194_121512_object8067:head# 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dda64f5:::benchmark_data_smithi194_121512_object114039:head# 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dda9aa4:::benchmark_data_smithi194_121512_object59462:head# 2024-04-29T14:27:13.275 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dde00ec:::benchmark_data_smithi194_121512_object46491:head# 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ddf52b0:::benchmark_data_smithi194_121512_object114742:head# 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ddfc99c:::benchmark_data_smithi194_121512_object121898:head# 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ddfd12b:::benchmark_data_smithi194_121512_object55370:head# 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de04fd4:::benchmark_data_smithi194_121512_object55739:head# 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de07f0f:::benchmark_data_smithi194_121512_object59418:head# 2024-04-29T14:27:13.276 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de1025e:::benchmark_data_smithi194_121512_object38369:head# 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de111f9:::benchmark_data_smithi194_121512_object34721:head# 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de23b54:::benchmark_data_smithi194_121512_object124016:head# 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de2c0e8:::benchmark_data_smithi194_121512_object61638:head# 2024-04-29T14:27:13.277 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.278 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de333a7:::benchmark_data_smithi194_121512_object7657:head# 2024-04-29T14:27:13.278 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de3f466:::benchmark_data_smithi194_121512_object38598:head# 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de455e4:::benchmark_data_smithi194_121512_object110706:head# 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de4bb22:::benchmark_data_smithi194_121512_object126383:head# 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de51e63:::benchmark_data_smithi194_121512_object86037:head# 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de51fa1:::benchmark_data_smithi194_121512_object617:head# 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de6510f:::benchmark_data_smithi194_121512_object80928:head# 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de6638d:::benchmark_data_smithi194_121512_object36658:head# 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de71039:::benchmark_data_smithi194_121512_object111919:head# 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de72070:::benchmark_data_smithi194_121512_object40357:head# 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de765b1:::benchmark_data_smithi194_121512_object93547:head# 2024-04-29T14:27:13.283 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de7b202:::benchmark_data_smithi194_121512_object114054:head# 2024-04-29T14:27:13.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.285 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de7e62a:::benchmark_data_smithi194_121512_object3065:head# 2024-04-29T14:27:13.285 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.286 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de865c9:::benchmark_data_smithi194_121512_object117355:head# 2024-04-29T14:27:13.286 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de92e49:::benchmark_data_smithi194_121512_object44751:head# 2024-04-29T14:27:13.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.287 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de95677:::benchmark_data_smithi194_121512_object80411:head# 2024-04-29T14:27:13.287 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de9e388:::benchmark_data_smithi194_121512_object96944:head# 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6de9ef43:::benchmark_data_smithi194_121512_object44537:head# 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dea44b2:::benchmark_data_smithi194_121512_object14676:head# 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6deb13fd:::benchmark_data_smithi194_121512_object30000:head# 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6debc1a4:::benchmark_data_smithi194_121512_object48925:head# 2024-04-29T14:27:13.288 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dec6ead:::benchmark_data_smithi194_121512_object126839:head# 2024-04-29T14:27:13.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6decd1c0:::benchmark_data_smithi194_121512_object47111:head# 2024-04-29T14:27:13.289 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.289 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6decf199:::benchmark_data_smithi194_121512_object67671:head# 2024-04-29T14:27:13.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ded21f3:::benchmark_data_smithi194_121512_object49706:head# 2024-04-29T14:27:13.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.290 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ded2c5d:::benchmark_data_smithi194_121512_object434:head# 2024-04-29T14:27:13.290 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ded87fa:::benchmark_data_smithi194_121512_object100624:head# 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dee6093:::benchmark_data_smithi194_121512_object66303:head# 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6defc521:::benchmark_data_smithi194_121512_object65708:head# 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df0daaf:::benchmark_data_smithi194_121512_object90947:head# 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df1108d:::benchmark_data_smithi194_121512_object34185:head# 2024-04-29T14:27:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df25b48:::benchmark_data_smithi194_121512_object94621:head# 2024-04-29T14:27:13.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.292 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df4c4b8:::benchmark_data_smithi194_121512_object123509:head# 2024-04-29T14:27:13.292 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.294 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df555d5:::benchmark_data_smithi194_121512_object95995:head# 2024-04-29T14:27:13.295 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df58b8c:::benchmark_data_smithi194_121512_object46936:head# 2024-04-29T14:27:13.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df63843:::benchmark_data_smithi194_121512_object43580:head# 2024-04-29T14:27:13.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.300 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df6e4e3:::benchmark_data_smithi194_121512_object62802:head# 2024-04-29T14:27:13.300 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df6f8fb:::benchmark_data_smithi194_121512_object115217:head# 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df73cec:::benchmark_data_smithi194_121512_object64589:head# 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df749fa:::benchmark_data_smithi194_121512_object6052:head# 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df78ba1:::benchmark_data_smithi194_121512_object43141:head# 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df858aa:::benchmark_data_smithi194_121512_object2017:head# 2024-04-29T14:27:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.303 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df87fb4:::benchmark_data_smithi194_121512_object111168:head# 2024-04-29T14:27:13.303 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.312 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df88bac:::benchmark_data_smithi194_121512_object48238:head# 2024-04-29T14:27:13.312 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.328 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6df89354:::benchmark_data_smithi194_121512_object93022:head# 2024-04-29T14:27:13.329 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.330 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfabda0:::benchmark_data_smithi194_121512_object80405:head# 2024-04-29T14:27:13.330 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfb6222:::benchmark_data_smithi194_121512_object70372:head# 2024-04-29T14:27:13.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.331 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfbcc48:::benchmark_data_smithi194_121512_object132446:head# 2024-04-29T14:27:13.331 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfce5c7:::benchmark_data_smithi194_121512_object115846:head# 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfdb299:::benchmark_data_smithi194_121512_object88928:head# 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfe283c:::benchmark_data_smithi194_121512_object54751:head# 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfe7ca9:::benchmark_data_smithi194_121512_object59308:head# 2024-04-29T14:27:13.332 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.333 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfeb1bd:::benchmark_data_smithi194_121512_object66170:head# 2024-04-29T14:27:13.333 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.333 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dfecab7:::benchmark_data_smithi194_121512_object33397:head# 2024-04-29T14:27:13.333 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.333 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dff1756:::benchmark_data_smithi194_121512_object3197:head# 2024-04-29T14:27:13.333 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.334 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dff2a39:::benchmark_data_smithi194_121512_object103348:head# 2024-04-29T14:27:13.334 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.334 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6dffc25d:::benchmark_data_smithi194_121512_object3920:head# 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e001073:::benchmark_data_smithi194_121512_object33558:head# 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e002662:::benchmark_data_smithi194_121512_object8754:head# 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e008f14:::benchmark_data_smithi194_121512_object20474:head# 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0160e6:::benchmark_data_smithi194_121512_object86928:head# 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e02364c:::benchmark_data_smithi194_121512_object49799:head# 2024-04-29T14:27:13.336 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e023778:::benchmark_data_smithi194_121512_object76034:head# 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e024efa:::benchmark_data_smithi194_121512_object64499:head# 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e03b7af:::benchmark_data_smithi194_121512_object79754:head# 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e040de4:::benchmark_data_smithi194_121512_object19924:head# 2024-04-29T14:27:13.337 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e04315e:::benchmark_data_smithi194_121512_object31865:head# 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e04b3ca:::benchmark_data_smithi194_121512_object123773:head# 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e04b4e8:::benchmark_data_smithi194_121512_object58355:head# 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e04f37b:::benchmark_data_smithi194_121512_object91773:head# 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e053142:::benchmark_data_smithi194_121512_object5176:head# 2024-04-29T14:27:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0573a5:::benchmark_data_smithi194_121512_object11836:head# 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e062dc0:::benchmark_data_smithi194_121512_object75256:head# 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e06336e:::benchmark_data_smithi194_121512_object116504:head# 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e066985:::benchmark_data_smithi194_121512_object107925:head# 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e067efd:::benchmark_data_smithi194_121512_object113522:head# 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e06ad82:::benchmark_data_smithi194_121512_object23689:head# 2024-04-29T14:27:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e07b421:::benchmark_data_smithi194_121512_object59266:head# 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e07df9b:::benchmark_data_smithi194_121512_object93862:head# 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e08c334:::benchmark_data_smithi194_121512_object29085:head# 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0a5f73:::benchmark_data_smithi194_121512_object10509:head# 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0a8631:::benchmark_data_smithi194_121512_object43574:head# 2024-04-29T14:27:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0a9ee1:::benchmark_data_smithi194_121512_object23762:head# 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0b0c6a:::benchmark_data_smithi194_121512_object87236:head# 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0b5500:::benchmark_data_smithi194_121512_object62872:head# 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0b9858:::benchmark_data_smithi194_121512_object72354:head# 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0cbd37:::benchmark_data_smithi194_121512_object94612:head# 2024-04-29T14:27:13.344 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0d77ad:::benchmark_data_smithi194_121512_object112847:head# 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0e0bde:::benchmark_data_smithi194_121512_object4483:head# 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0e8f21:::benchmark_data_smithi194_121512_object107352:head# 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0eee64:::benchmark_data_smithi194_121512_object41629:head# 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e0f49ac:::benchmark_data_smithi194_121512_object133131:head# 2024-04-29T14:27:13.345 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.347 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e105d78:::benchmark_data_smithi194_121512_object101070:head# 2024-04-29T14:27:13.347 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.347 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e11aa5b:::benchmark_data_smithi194_121512_object98226:head# 2024-04-29T14:27:13.347 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.347 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e129b43:::benchmark_data_smithi194_121512_object3527:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e147f84:::benchmark_data_smithi194_121512_object72928:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e15231b:::benchmark_data_smithi194_121512_object93989:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e157ac2:::benchmark_data_smithi194_121512_object3312:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e15a51f:::benchmark_data_smithi194_121512_object128100:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e15caae:::benchmark_data_smithi194_121512_object35677:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e161156:::benchmark_data_smithi194_121512_object61999:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e16a9e5:::benchmark_data_smithi194_121512_object98831:head# 2024-04-29T14:27:13.348 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e17030d:::benchmark_data_smithi194_121512_object110746:head# 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e17c22e:::benchmark_data_smithi194_121512_object1612:head# 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e17ef62:::benchmark_data_smithi194_121512_object18558:head# 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e17f83b:::benchmark_data_smithi194_121512_object48655:head# 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e180c0c:::benchmark_data_smithi194_121512_object47011:head# 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e184b1f:::benchmark_data_smithi194_121512_object38462:head# 2024-04-29T14:27:13.349 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.350 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e19124c:::benchmark_data_smithi194_121512_object99480:head# 2024-04-29T14:27:13.350 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.350 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e192964:::benchmark_data_smithi194_121512_object66282:head# 2024-04-29T14:27:13.350 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e198688:::benchmark_data_smithi194_121512_object20665:head# 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e19b67c:::benchmark_data_smithi194_121512_object52742:head# 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e1ad206:::benchmark_data_smithi194_121512_object61973:head# 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e1ae649:::benchmark_data_smithi194_121512_object25957:head# 2024-04-29T14:27:13.351 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.352 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e1b5727:::benchmark_data_smithi194_121512_object130037:head# 2024-04-29T14:27:13.352 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.353 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e1c3b70:::benchmark_data_smithi194_121512_object76208:head# 2024-04-29T14:27:13.353 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.353 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e1eac43:::benchmark_data_smithi194_121512_object8965:head# 2024-04-29T14:27:13.353 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e1f211a:::benchmark_data_smithi194_121512_object100063:head# 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e1f82ea:::benchmark_data_smithi194_121512_object28961:head# 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e213d7f:::benchmark_data_smithi194_121512_object17477:head# 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e219ad1:::benchmark_data_smithi194_121512_object107526:head# 2024-04-29T14:27:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e21d284:::benchmark_data_smithi194_121512_object54195:head# 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e21da19:::benchmark_data_smithi194_121512_object842:head# 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2214d2:::benchmark_data_smithi194_121512_object75997:head# 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e22a961:::benchmark_data_smithi194_121512_object131109:head# 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.356 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2314b6:::benchmark_data_smithi194_121512_object49961:head# 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2393df:::benchmark_data_smithi194_121512_object57726:head# 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e24b62b:::benchmark_data_smithi194_121512_object42996:head# 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e24f5cd:::benchmark_data_smithi194_121512_object763:head# 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e253713:::benchmark_data_smithi194_121512_object51960:head# 2024-04-29T14:27:13.357 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e267265:::benchmark_data_smithi194_121512_object72861:head# 2024-04-29T14:27:13.358 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.358 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e271bd0:::benchmark_data_smithi194_121512_object33441:head# 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2730a1:::benchmark_data_smithi194_121512_object32160:head# 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2743be:::benchmark_data_smithi194_121512_object113630:head# 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e275957:::benchmark_data_smithi194_121512_object101831:head# 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e28da61:::benchmark_data_smithi194_121512_object27043:head# 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e28e7a9:::benchmark_data_smithi194_121512_object96485:head# 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e29b72c:::benchmark_data_smithi194_121512_object30451:head# 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.359 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e29b80d:::benchmark_data_smithi194_121512_object37584:head# 2024-04-29T14:27:13.361 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e29c05e:::benchmark_data_smithi194_121512_object75621:head# 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e29e619:::benchmark_data_smithi194_121512_object54924:head# 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2a59b5:::benchmark_data_smithi194_121512_object113787:head# 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2a9cf2:::benchmark_data_smithi194_121512_object54111:head# 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2ae9c8:::benchmark_data_smithi194_121512_object126693:head# 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2b6240:::benchmark_data_smithi194_121512_object79879:head# 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2b97ad:::benchmark_data_smithi194_121512_object121547:head# 2024-04-29T14:27:13.362 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2c5f2b:::benchmark_data_smithi194_121512_object122636:head# 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2dccde:::benchmark_data_smithi194_121512_object84731:head# 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e2e1658:::benchmark_data_smithi194_121512_object116855:head# 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e301c46:::benchmark_data_smithi194_121512_object27352:head# 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e304063:::benchmark_data_smithi194_121512_object117397:head# 2024-04-29T14:27:13.363 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.364 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e307c05:::benchmark_data_smithi194_121512_object22147:head# 2024-04-29T14:27:13.364 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.364 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e30ba09:::benchmark_data_smithi194_121512_object22206:head# 2024-04-29T14:27:13.364 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.364 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e312b15:::benchmark_data_smithi194_121512_object59061:head# 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e317415:::benchmark_data_smithi194_121512_object98129:head# 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e31b9a3:::benchmark_data_smithi194_121512_object74874:head# 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e328906:::benchmark_data_smithi194_121512_object33785:head# 2024-04-29T14:27:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e32a9a6:::benchmark_data_smithi194_121512_object5476:head# 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e32ab00:::benchmark_data_smithi194_121512_object52047:head# 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3396a0:::benchmark_data_smithi194_121512_object11639:head# 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e33ef5e:::benchmark_data_smithi194_121512_object21970:head# 2024-04-29T14:27:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e343f62:::benchmark_data_smithi194_121512_object3656:head# 2024-04-29T14:27:13.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3476d3:::benchmark_data_smithi194_121512_object91876:head# 2024-04-29T14:27:13.369 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.369 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e354574:::benchmark_data_smithi194_121512_object8814:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e38cc49:::benchmark_data_smithi194_121512_object21358:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e39a324:::benchmark_data_smithi194_121512_object9145:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e39be9f:::benchmark_data_smithi194_121512_object9540:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e39fd77:::benchmark_data_smithi194_121512_object53898:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3a5a7d:::benchmark_data_smithi194_121512_object32755:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3ab3ea:::benchmark_data_smithi194_121512_object23487:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3ac765:::benchmark_data_smithi194_121512_object65110:head# 2024-04-29T14:27:13.370 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3c24ba:::benchmark_data_smithi194_121512_object55019:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3c59c7:::benchmark_data_smithi194_121512_object62330:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3e181a:::benchmark_data_smithi194_121512_object13543:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3e3b61:::benchmark_data_smithi194_121512_object78201:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3e98c9:::benchmark_data_smithi194_121512_object11602:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3ee857:::benchmark_data_smithi194_121512_object38298:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3f3342:::benchmark_data_smithi194_121512_object2833:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3f685f:::benchmark_data_smithi194_121512_object66326:head# 2024-04-29T14:27:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3f6ce7:::benchmark_data_smithi194_121512_object27845:head# 2024-04-29T14:27:13.373 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.373 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3faa4c:::benchmark_data_smithi194_121512_object133374:head# 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e3fbd8b:::benchmark_data_smithi194_121512_object85808:head# 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e41ef4a:::benchmark_data_smithi194_121512_object96537:head# 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e41f5bf:::benchmark_data_smithi194_121512_object78989:head# 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e428cee:::benchmark_data_smithi194_121512_object65533:head# 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e42900d:::benchmark_data_smithi194_121512_object32810:head# 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e42ca45:::benchmark_data_smithi194_121512_object65025:head# 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.374 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e42d14b:::benchmark_data_smithi194_121512_object70041:head# 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e43be69:::benchmark_data_smithi194_121512_object88163:head# 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e43f9bc:::benchmark_data_smithi194_121512_object109753:head# 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e442406:::benchmark_data_smithi194_121512_object59521:head# 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e449e7d:::benchmark_data_smithi194_121512_object111854:head# 2024-04-29T14:27:13.375 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e45a1f8:::benchmark_data_smithi194_121512_object9903:head# 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e46070d:::benchmark_data_smithi194_121512_object11773:head# 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e472d78:::benchmark_data_smithi194_121512_object13057:head# 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e478d26:::benchmark_data_smithi194_121512_object89387:head# 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e479fd8:::benchmark_data_smithi194_121512_object70671:head# 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e47ad9a:::benchmark_data_smithi194_121512_object78315:head# 2024-04-29T14:27:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e47d6b8:::benchmark_data_smithi194_121512_object68228:head# 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e480800:::benchmark_data_smithi194_121512_object7796:head# 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e481e22:::benchmark_data_smithi194_121512_object26168:head# 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4848a5:::benchmark_data_smithi194_121512_object96098:head# 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e48614d:::benchmark_data_smithi194_121512_object57689:head# 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e48633d:::benchmark_data_smithi194_121512_object76539:head# 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e48df10:::benchmark_data_smithi194_121512_object130551:head# 2024-04-29T14:27:13.378 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4a1b31:::benchmark_data_smithi194_121512_object9437:head# 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4a40d4:::benchmark_data_smithi194_121512_object121737:head# 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4a5206:::benchmark_data_smithi194_121512_object47263:head# 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4a5459:::benchmark_data_smithi194_121512_object28772:head# 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4aea0e:::benchmark_data_smithi194_121512_object90211:head# 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4b6ce5:::benchmark_data_smithi194_121512_object132222:head# 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4bfe0e:::benchmark_data_smithi194_121512_object84766:head# 2024-04-29T14:27:13.380 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4d5f0f:::benchmark_data_smithi194_121512_object60950:head# 2024-04-29T14:27:13.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.381 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4d900d:::benchmark_data_smithi194_121512_object130464:head# 2024-04-29T14:27:13.381 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4dafef:::benchmark_data_smithi194_121512_object59077:head# 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4e0c2d:::benchmark_data_smithi194_121512_object49169:head# 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4e2c0d:::benchmark_data_smithi194_121512_object115082:head# 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4e3b0b:::benchmark_data_smithi194_121512_object11518:head# 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4e6236:::benchmark_data_smithi194_121512_object13119:head# 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e4ef674:::benchmark_data_smithi194_121512_object30698:head# 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e502505:::benchmark_data_smithi194_121512_object59882:head# 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e50a5b5:::benchmark_data_smithi194_121512_object121496:head# 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e51afa2:::benchmark_data_smithi194_121512_object72338:head# 2024-04-29T14:27:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e51c92c:::benchmark_data_smithi194_121512_object118996:head# 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e52b0eb:::benchmark_data_smithi194_121512_object122456:head# 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e52ceaf:::benchmark_data_smithi194_121512_object112872:head# 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e532434:::benchmark_data_smithi194_121512_object94883:head# 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e533657:::benchmark_data_smithi194_121512_object110421:head# 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e53f3c2:::benchmark_data_smithi194_121512_object4444:head# 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e54b43e:::benchmark_data_smithi194_121512_object7537:head# 2024-04-29T14:27:13.385 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.392 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e54cef3:::benchmark_data_smithi194_121512_object108721:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e551fb5:::benchmark_data_smithi194_121512_object110774:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e55ed71:::benchmark_data_smithi194_121512_object62898:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e566932:::benchmark_data_smithi194_121512_object48176:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e57cc13:::benchmark_data_smithi194_121512_object13395:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e590284:::benchmark_data_smithi194_121512_object84456:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5943a6:::benchmark_data_smithi194_121512_object20966:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e59449a:::benchmark_data_smithi194_121512_object119660:head# 2024-04-29T14:27:13.393 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5baf76:::benchmark_data_smithi194_121512_object120572:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5bf815:::benchmark_data_smithi194_121512_object77240:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5c15fd:::benchmark_data_smithi194_121512_object42233:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5c34e2:::benchmark_data_smithi194_121512_object74623:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5cdb7f:::benchmark_data_smithi194_121512_object117817:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5d59c0:::benchmark_data_smithi194_121512_object89383:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5d6a2c:::benchmark_data_smithi194_121512_object103957:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5dec40:::benchmark_data_smithi194_121512_object105752:head# 2024-04-29T14:27:13.394 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5e26c6:::benchmark_data_smithi194_121512_object48054:head# 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5f08bd:::benchmark_data_smithi194_121512_object65859:head# 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e5fbde7:::benchmark_data_smithi194_121512_object42497:head# 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e612d2e:::benchmark_data_smithi194_121512_object95306:head# 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e614aac:::benchmark_data_smithi194_121512_object6165:head# 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6200f2:::benchmark_data_smithi194_121512_object85078:head# 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.396 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e62333a:::benchmark_data_smithi194_121512_object39890:head# 2024-04-29T14:27:13.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e624829:::benchmark_data_smithi194_121512_object119341:head# 2024-04-29T14:27:13.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.397 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e62ca00:::benchmark_data_smithi194_121512_object16330:head# 2024-04-29T14:27:13.397 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.400 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6403b1:::benchmark_data_smithi194_121512_object116437:head# 2024-04-29T14:27:13.400 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.402 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e64c65e:::benchmark_data_smithi194_121512_object59411:head# 2024-04-29T14:27:13.402 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.403 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6553d1:::benchmark_data_smithi194_121512_object120525:head# 2024-04-29T14:27:13.403 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.403 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6643b3:::benchmark_data_smithi194_121512_object23950:head# 2024-04-29T14:27:13.403 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.405 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e670d2c:::benchmark_data_smithi194_121512_object95358:head# 2024-04-29T14:27:13.405 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e67b202:::benchmark_data_smithi194_121512_object23990:head# 2024-04-29T14:27:13.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.406 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e68c3cb:::benchmark_data_smithi194_121512_object107820:head# 2024-04-29T14:27:13.406 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6981ea:::benchmark_data_smithi194_121512_object103896:head# 2024-04-29T14:27:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e69b7d8:::benchmark_data_smithi194_121512_object113248:head# 2024-04-29T14:27:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6a3fd1:::benchmark_data_smithi194_121512_object125562:head# 2024-04-29T14:27:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6a830e:::benchmark_data_smithi194_121512_object36855:head# 2024-04-29T14:27:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6a95f1:::benchmark_data_smithi194_121512_object28740:head# 2024-04-29T14:27:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6ad626:::benchmark_data_smithi194_121512_object47847:head# 2024-04-29T14:27:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6b3fdc:::benchmark_data_smithi194_121512_object109297:head# 2024-04-29T14:27:13.409 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6b651d:::benchmark_data_smithi194_121512_object123064:head# 2024-04-29T14:27:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6b8094:::benchmark_data_smithi194_121512_object74304:head# 2024-04-29T14:27:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6ba479:::benchmark_data_smithi194_121512_object63125:head# 2024-04-29T14:27:13.410 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.411 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6ce351:::benchmark_data_smithi194_121512_object123506:head# 2024-04-29T14:27:13.411 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6d4c90:::benchmark_data_smithi194_121512_object73252:head# 2024-04-29T14:27:13.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.412 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6d6580:::benchmark_data_smithi194_121512_object20590:head# 2024-04-29T14:27:13.412 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6e2702:::benchmark_data_smithi194_121512_object95162:head# 2024-04-29T14:27:13.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6e3e92:::benchmark_data_smithi194_121512_object67744:head# 2024-04-29T14:27:13.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6e528b:::benchmark_data_smithi194_121512_object67145:head# 2024-04-29T14:27:13.413 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.413 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6e7153:::benchmark_data_smithi194_121512_object104286:head# 2024-04-29T14:27:13.414 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6e8cb2:::benchmark_data_smithi194_121512_object31893:head# 2024-04-29T14:27:13.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6eedb1:::benchmark_data_smithi194_121512_object127128:head# 2024-04-29T14:27:13.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.418 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6eef22:::benchmark_data_smithi194_121512_object66681:head# 2024-04-29T14:27:13.418 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6f6c07:::benchmark_data_smithi194_121512_object25970:head# 2024-04-29T14:27:13.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6faaf3:::benchmark_data_smithi194_121512_object40176:head# 2024-04-29T14:27:13.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.419 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e6fc06a:::benchmark_data_smithi194_121512_object50121:head# 2024-04-29T14:27:13.419 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e70369e:::benchmark_data_smithi194_121512_object81507:head# 2024-04-29T14:27:13.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e703e03:::benchmark_data_smithi194_121512_object20736:head# 2024-04-29T14:27:13.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.420 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e709ebe:::benchmark_data_smithi194_121512_object93251:head# 2024-04-29T14:27:13.420 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e710091:::benchmark_data_smithi194_121512_object86116:head# 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e713421:::benchmark_data_smithi194_121512_object34466:head# 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e71ed70:::benchmark_data_smithi194_121512_object120458:head# 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e72059a:::benchmark_data_smithi194_121512_object1354:head# 2024-04-29T14:27:13.421 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.422 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e72913c:::benchmark_data_smithi194_121512_object98523:head# 2024-04-29T14:27:13.422 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7322a2:::benchmark_data_smithi194_121512_object120487:head# 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e747c99:::benchmark_data_smithi194_121512_object51703:head# 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e756d94:::benchmark_data_smithi194_121512_object105167:head# 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7608a4:::benchmark_data_smithi194_121512_object130976:head# 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7694b5:::benchmark_data_smithi194_121512_object114041:head# 2024-04-29T14:27:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e76a7e8:::benchmark_data_smithi194_121512_object71081:head# 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e76af8e:::benchmark_data_smithi194_121512_object47475:head# 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7712f9:::benchmark_data_smithi194_121512_object102739:head# 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7783fe:::benchmark_data_smithi194_121512_object78316:head# 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e77d0e4:::benchmark_data_smithi194_121512_object65187:head# 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e781999:::benchmark_data_smithi194_121512_object26347:head# 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e789c73:::benchmark_data_smithi194_121512_object93676:head# 2024-04-29T14:27:13.430 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7969b5:::benchmark_data_smithi194_121512_object1773:head# 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7982c3:::benchmark_data_smithi194_121512_object35128:head# 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e79bd04:::benchmark_data_smithi194_121512_object130422:head# 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7a2d01:::benchmark_data_smithi194_121512_object124875:head# 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7bd955:::benchmark_data_smithi194_121512_object117529:head# 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7c076e:::benchmark_data_smithi194_121512_object123621:head# 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7c8176:::benchmark_data_smithi194_121512_object64402:head# 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.431 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7c9511:::benchmark_data_smithi194_121512_object32663:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7cf29b:::benchmark_data_smithi194_121512_object123241:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7d2d36:::benchmark_data_smithi194_121512_object66169:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7d47ea:::benchmark_data_smithi194_121512_object3238:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7d4d1f:::benchmark_data_smithi194_121512_object95186:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7d870a:::benchmark_data_smithi194_121512_object60120:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7de5b9:::benchmark_data_smithi194_121512_object49722:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7e6ddb:::benchmark_data_smithi194_121512_object49144:head# 2024-04-29T14:27:13.432 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7ea65c:::benchmark_data_smithi194_121512_object68347:head# 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7ecee3:::benchmark_data_smithi194_121512_object31869:head# 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7ee4fe:::benchmark_data_smithi194_121512_object95133:head# 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7f1c99:::benchmark_data_smithi194_121512_object51803:head# 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.434 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e7fc07d:::benchmark_data_smithi194_121512_object47287:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e806a5d:::benchmark_data_smithi194_121512_object106540:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e80fa70:::benchmark_data_smithi194_121512_object120533:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e810d7e:::benchmark_data_smithi194_121512_object96254:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e81b89d:::benchmark_data_smithi194_121512_object101871:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e81e4cb:::benchmark_data_smithi194_121512_object32009:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8381b0:::benchmark_data_smithi194_121512_object111524:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e83b59a:::benchmark_data_smithi194_121512_object106645:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8482dc:::benchmark_data_smithi194_121512_object76860:head# 2024-04-29T14:27:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e84b774:::benchmark_data_smithi194_121512_object61016:head# 2024-04-29T14:27:13.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e850d7b:::benchmark_data_smithi194_121512_object73664:head# 2024-04-29T14:27:13.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.436 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e85a518:::benchmark_data_smithi194_121512_object62550:head# 2024-04-29T14:27:13.436 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e85c66a:::benchmark_data_smithi194_121512_object106419:head# 2024-04-29T14:27:13.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e85d444:::benchmark_data_smithi194_121512_object42557:head# 2024-04-29T14:27:13.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.438 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8611d6:::benchmark_data_smithi194_121512_object124179:head# 2024-04-29T14:27:13.438 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e868d81:::benchmark_data_smithi194_121512_object70821:head# 2024-04-29T14:27:13.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e86d544:::benchmark_data_smithi194_121512_object45736:head# 2024-04-29T14:27:13.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.439 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e87b94d:::benchmark_data_smithi194_121512_object92165:head# 2024-04-29T14:27:13.439 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.440 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e881fd1:::benchmark_data_smithi194_121512_object72501:head# 2024-04-29T14:27:13.440 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e88b4ba:::benchmark_data_smithi194_121512_object9995:head# 2024-04-29T14:27:13.445 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.445 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e88e24f:::benchmark_data_smithi194_121512_object67976:head# 2024-04-29T14:27:13.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.446 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e892270:::benchmark_data_smithi194_121512_object81512:head# 2024-04-29T14:27:13.446 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.451 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e893184:::benchmark_data_smithi194_121512_object100106:head# 2024-04-29T14:27:13.451 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e893f74:::benchmark_data_smithi194_121512_object54523:head# 2024-04-29T14:27:13.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8a8e71:::benchmark_data_smithi194_121512_object115541:head# 2024-04-29T14:27:13.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.452 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8b1f0e:::benchmark_data_smithi194_121512_object31195:head# 2024-04-29T14:27:13.452 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8b5f39:::benchmark_data_smithi194_121512_object16974:head# 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8c37fe:::benchmark_data_smithi194_121512_object18878:head# 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8d3e76:::benchmark_data_smithi194_121512_object49494:head# 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8d6460:::benchmark_data_smithi194_121512_object1704:head# 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8d9122:::benchmark_data_smithi194_121512_object30687:head# 2024-04-29T14:27:13.453 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8daccb:::benchmark_data_smithi194_121512_object33670:head# 2024-04-29T14:27:13.454 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.454 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8dd606:::benchmark_data_smithi194_121512_object115458:head# 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8e69a9:::benchmark_data_smithi194_121512_object89989:head# 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8ed271:::benchmark_data_smithi194_121512_object79741:head# 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8f3579:::benchmark_data_smithi194_121512_object70797:head# 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8f4e8c:::benchmark_data_smithi194_121512_object33745:head# 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e8f8b1f:::benchmark_data_smithi194_121512_object94616:head# 2024-04-29T14:27:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e905a95:::benchmark_data_smithi194_121512_object107903:head# 2024-04-29T14:27:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e906110:::benchmark_data_smithi194_121512_object30983:head# 2024-04-29T14:27:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e91584d:::benchmark_data_smithi194_121512_object70066:head# 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e91c5cb:::benchmark_data_smithi194_121512_object108419:head# 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e91ddab:::benchmark_data_smithi194_121512_object42221:head# 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e92c4e9:::benchmark_data_smithi194_121512_object74072:head# 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e92fb4c:::benchmark_data_smithi194_121512_object87367:head# 2024-04-29T14:27:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e930e80:::benchmark_data_smithi194_121512_object45464:head# 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9331eb:::benchmark_data_smithi194_121512_object33151:head# 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e937050:::benchmark_data_smithi194_121512_object97495:head# 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e93fac4:::benchmark_data_smithi194_121512_object123714:head# 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9427a5:::benchmark_data_smithi194_121512_object4601:head# 2024-04-29T14:27:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.470 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e943836:::benchmark_data_smithi194_121512_object64329:head# 2024-04-29T14:27:13.470 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.474 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e94b579:::benchmark_data_smithi194_121512_object70352:head# 2024-04-29T14:27:13.474 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.480 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e957e61:::benchmark_data_smithi194_121512_object31747:head# 2024-04-29T14:27:13.481 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e957f6e:::benchmark_data_smithi194_121512_object40133:head# 2024-04-29T14:27:13.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.485 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e95cc50:::benchmark_data_smithi194_121512_object117028:head# 2024-04-29T14:27:13.485 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e964d33:::benchmark_data_smithi194_121512_object35295:head# 2024-04-29T14:27:13.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e969f7a:::benchmark_data_smithi194_121512_object57874:head# 2024-04-29T14:27:13.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.486 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e980c06:::benchmark_data_smithi194_121512_object119834:head# 2024-04-29T14:27:13.486 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e98eca4:::benchmark_data_smithi194_121512_object37883:head# 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9a9540:::benchmark_data_smithi194_121512_object87402:head# 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9aa733:::benchmark_data_smithi194_121512_object48408:head# 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9c2064:::benchmark_data_smithi194_121512_object132687:head# 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9c59a7:::benchmark_data_smithi194_121512_object2176:head# 2024-04-29T14:27:13.487 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9d1063:::benchmark_data_smithi194_121512_object65186:head# 2024-04-29T14:27:13.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9de2bd:::benchmark_data_smithi194_121512_object19705:head# 2024-04-29T14:27:13.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.488 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9e7948:::benchmark_data_smithi194_121512_object27791:head# 2024-04-29T14:27:13.488 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9ea94f:::benchmark_data_smithi194_121512_object105004:head# 2024-04-29T14:27:13.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.489 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9faa8e:::benchmark_data_smithi194_121512_object11954:head# 2024-04-29T14:27:13.489 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6e9fca3e:::benchmark_data_smithi194_121512_object59825:head# 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea00a4d:::benchmark_data_smithi194_121512_object76458:head# 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea03e7b:::benchmark_data_smithi194_121512_object55877:head# 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea0988e:::benchmark_data_smithi194_121512_object43170:head# 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea0dc45:::benchmark_data_smithi194_121512_object76637:head# 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea105bc:::benchmark_data_smithi194_121512_object51361:head# 2024-04-29T14:27:13.490 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea12d3b:::benchmark_data_smithi194_121512_object88383:head# 2024-04-29T14:27:13.491 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.491 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea19c8d:::benchmark_data_smithi194_121512_object20579:head# 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea27ed7:::benchmark_data_smithi194_121512_object70317:head# 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea2b12e:::benchmark_data_smithi194_121512_object59482:head# 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea45208:::benchmark_data_smithi194_121512_object129040:head# 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea49314:::benchmark_data_smithi194_121512_object13459:head# 2024-04-29T14:27:13.492 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea4b702:::benchmark_data_smithi194_121512_object81124:head# 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea4e83b:::benchmark_data_smithi194_121512_object99850:head# 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea544b1:::benchmark_data_smithi194_121512_object72364:head# 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea58aaf:::benchmark_data_smithi194_121512_object127914:head# 2024-04-29T14:27:13.493 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.496 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea61fc2:::benchmark_data_smithi194_121512_object39842:head# 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea6e007:::benchmark_data_smithi194_121512_object116250:head# 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea75012:::benchmark_data_smithi194_121512_object117126:head# 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea7584a:::benchmark_data_smithi194_121512_object49667:head# 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea7b1c9:::benchmark_data_smithi194_121512_object40403:head# 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea7cf25:::benchmark_data_smithi194_121512_object85336:head# 2024-04-29T14:27:13.497 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.499 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea83031:::benchmark_data_smithi194_121512_object26684:head# 2024-04-29T14:27:13.499 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.500 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea85fd1:::benchmark_data_smithi194_121512_object113849:head# 2024-04-29T14:27:13.500 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.502 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea8686b:::benchmark_data_smithi194_121512_object22629:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea8d2e6:::benchmark_data_smithi194_121512_object113545:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea93e01:::benchmark_data_smithi194_121512_object17285:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea94f5f:::benchmark_data_smithi194_121512_object58095:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ea9c318:::benchmark_data_smithi194_121512_object28190:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eaa42fa:::benchmark_data_smithi194_121512_object72866:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eaa881d:::benchmark_data_smithi194_121512_object95175:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eab7649:::benchmark_data_smithi194_121512_object101687:head# 2024-04-29T14:27:13.503 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eabc38a:::benchmark_data_smithi194_121512_object64556:head# 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eac0eed:::benchmark_data_smithi194_121512_object66960:head# 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eac9bcb:::benchmark_data_smithi194_121512_object54814:head# 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ead45a8:::benchmark_data_smithi194_121512_object13784:head# 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ead892d:::benchmark_data_smithi194_121512_object67612:head# 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eae1d6d:::benchmark_data_smithi194_121512_object96443:head# 2024-04-29T14:27:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.505 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eaf4209:::benchmark_data_smithi194_121512_object20539:head# 2024-04-29T14:27:13.505 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eafbf23:::benchmark_data_smithi194_121512_object74998:head# 2024-04-29T14:27:13.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb0245d:::benchmark_data_smithi194_121512_object86473:head# 2024-04-29T14:27:13.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.506 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb04f2a:::benchmark_data_smithi194_121512_object29458:head# 2024-04-29T14:27:13.506 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb0aefa:::benchmark_data_smithi194_121512_object3764:head# 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb2e85a:::benchmark_data_smithi194_121512_object45768:head# 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb3eec3:::benchmark_data_smithi194_121512_object94873:head# 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb4aa81:::benchmark_data_smithi194_121512_object112368:head# 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb55851:::benchmark_data_smithi194_121512_object66325:head# 2024-04-29T14:27:13.508 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb56779:::benchmark_data_smithi194_121512_object1414:head# 2024-04-29T14:27:13.509 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.509 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb5ae94:::benchmark_data_smithi194_121512_object44193:head# 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb5ebe3:::benchmark_data_smithi194_121512_object112927:head# 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb64fc1:::benchmark_data_smithi194_121512_object28616:head# 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb66f59:::benchmark_data_smithi194_121512_object117347:head# 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb673ac:::benchmark_data_smithi194_121512_object131531:head# 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb720d8:::benchmark_data_smithi194_121512_object41072:head# 2024-04-29T14:27:13.510 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb7f423:::benchmark_data_smithi194_121512_object66:head# 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb87b02:::benchmark_data_smithi194_121512_object15559:head# 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb8d46b:::benchmark_data_smithi194_121512_object1852:head# 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb9a37c:::benchmark_data_smithi194_121512_object92151:head# 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb9d57e:::benchmark_data_smithi194_121512_object52781:head# 2024-04-29T14:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb9f182:::benchmark_data_smithi194_121512_object56031:head# 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eb9fa8d:::benchmark_data_smithi194_121512_object30286:head# 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebb587a:::benchmark_data_smithi194_121512_object132722:head# 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebbc986:::benchmark_data_smithi194_121512_object53521:head# 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebc7f34:::benchmark_data_smithi194_121512_object24927:head# 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebc8caa:::benchmark_data_smithi194_121512_object74966:head# 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebc8dfc:::benchmark_data_smithi194_121512_object108953:head# 2024-04-29T14:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebcdfb8:::benchmark_data_smithi194_121512_object12781:head# 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebd232a:::benchmark_data_smithi194_121512_object1875:head# 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebd6576:::benchmark_data_smithi194_121512_object32691:head# 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebd6c60:::benchmark_data_smithi194_121512_object14057:head# 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebe23b7:::benchmark_data_smithi194_121512_object105146:head# 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebe6f10:::benchmark_data_smithi194_121512_object62900:head# 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebed9a7:::benchmark_data_smithi194_121512_object124109:head# 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.531 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebee20b:::benchmark_data_smithi194_121512_object50150:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebf55f1:::benchmark_data_smithi194_121512_object113053:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ebfc56b:::benchmark_data_smithi194_121512_object129771:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec064f3:::benchmark_data_smithi194_121512_object64426:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec0810e:::benchmark_data_smithi194_121512_object97841:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec1a385:::benchmark_data_smithi194_121512_object69685:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec1f5df:::benchmark_data_smithi194_121512_object17712:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec29b9c:::benchmark_data_smithi194_121512_object37478:head# 2024-04-29T14:27:13.532 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec319c1:::benchmark_data_smithi194_121512_object76819:head# 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec338e2:::benchmark_data_smithi194_121512_object105798:head# 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec38111:::benchmark_data_smithi194_121512_object32963:head# 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec40388:::benchmark_data_smithi194_121512_object127513:head# 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec43fcc:::benchmark_data_smithi194_121512_object111500:head# 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec45723:::benchmark_data_smithi194_121512_object32087:head# 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec46fa9:::benchmark_data_smithi194_121512_object124125:head# 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec4a018:::benchmark_data_smithi194_121512_object75002:head# 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec59f47:::benchmark_data_smithi194_121512_object8367:head# 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec5a4fa:::benchmark_data_smithi194_121512_object17388:head# 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec5f644:::benchmark_data_smithi194_121512_object75803:head# 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec63e9b:::benchmark_data_smithi194_121512_object32316:head# 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec6bca7:::benchmark_data_smithi194_121512_object911:head# 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec7db04:::benchmark_data_smithi194_121512_object116918:head# 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.534 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec91f67:::benchmark_data_smithi194_121512_object29115:head# 2024-04-29T14:27:13.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ec9bf5b:::benchmark_data_smithi194_121512_object93246:head# 2024-04-29T14:27:13.535 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.535 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecba693:::benchmark_data_smithi194_121512_object109649:head# 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecc1570:::benchmark_data_smithi194_121512_object37283:head# 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecc17df:::benchmark_data_smithi194_121512_object119941:head# 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eccfd5f:::benchmark_data_smithi194_121512_object26169:head# 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecd3a70:::benchmark_data_smithi194_121512_object77481:head# 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecdad81:::benchmark_data_smithi194_121512_object114106:head# 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ece759d:::benchmark_data_smithi194_121512_object128199:head# 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecece53:::benchmark_data_smithi194_121512_object96634:head# 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecee595:::benchmark_data_smithi194_121512_object110333:head# 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecfdf8a:::benchmark_data_smithi194_121512_object80547:head# 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ecff714:::benchmark_data_smithi194_121512_object17149:head# 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed03966:::benchmark_data_smithi194_121512_object109230:head# 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed0777c:::benchmark_data_smithi194_121512_object101490:head# 2024-04-29T14:27:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.541 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed07b1f:::benchmark_data_smithi194_121512_object20758:head# 2024-04-29T14:27:13.541 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.547 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed0b7b5:::benchmark_data_smithi194_121512_object28233:head# 2024-04-29T14:27:13.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed1685f:::benchmark_data_smithi194_121512_object55291:head# 2024-04-29T14:27:13.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed232f7:::benchmark_data_smithi194_121512_object29369:head# 2024-04-29T14:27:13.548 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.548 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed269a4:::benchmark_data_smithi194_121512_object59330:head# 2024-04-29T14:27:13.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed33aa7:::benchmark_data_smithi194_121512_object52906:head# 2024-04-29T14:27:13.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.549 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed3d6ab:::benchmark_data_smithi194_121512_object69460:head# 2024-04-29T14:27:13.549 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed3fc49:::benchmark_data_smithi194_121512_object12151:head# 2024-04-29T14:27:13.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.550 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed439c3:::benchmark_data_smithi194_121512_object17835:head# 2024-04-29T14:27:13.550 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed44dc1:::benchmark_data_smithi194_121512_object36142:head# 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed46a4b:::benchmark_data_smithi194_121512_object109581:head# 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed477b8:::benchmark_data_smithi194_121512_object39178:head# 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed4fb8a:::benchmark_data_smithi194_121512_object25453:head# 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed56da9:::benchmark_data_smithi194_121512_object24665:head# 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed59095:::benchmark_data_smithi194_121512_object8455:head# 2024-04-29T14:27:13.555 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed6cd71:::benchmark_data_smithi194_121512_object45839:head# 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed72c45:::benchmark_data_smithi194_121512_object29789:head# 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed787c7:::benchmark_data_smithi194_121512_object32206:head# 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed7ee9f:::benchmark_data_smithi194_121512_object107635:head# 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed82b7d:::benchmark_data_smithi194_121512_object87395:head# 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed89cc7:::benchmark_data_smithi194_121512_object6979:head# 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed8a846:::benchmark_data_smithi194_121512_object73304:head# 2024-04-29T14:27:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed93d24:::benchmark_data_smithi194_121512_object130465:head# 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ed94b12:::benchmark_data_smithi194_121512_object24811:head# 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eda3ba4:::benchmark_data_smithi194_121512_object51228:head# 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6edb8483:::benchmark_data_smithi194_121512_object81692:head# 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6edc5118:::benchmark_data_smithi194_121512_object36650:head# 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6edd1f36:::benchmark_data_smithi194_121512_object45148:head# 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eddd1ed:::benchmark_data_smithi194_121512_object107461:head# 2024-04-29T14:27:13.557 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ede32d3:::benchmark_data_smithi194_121512_object110553:head# 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ede4fb2:::benchmark_data_smithi194_121512_object52435:head# 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ede7bc4:::benchmark_data_smithi194_121512_object98604:head# 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ede99f3:::benchmark_data_smithi194_121512_object73395:head# 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6edea9c3:::benchmark_data_smithi194_121512_object124931:head# 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee06b31:::benchmark_data_smithi194_121512_object76386:head# 2024-04-29T14:27:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee2d5b3:::benchmark_data_smithi194_121512_object64795:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee30a66:::benchmark_data_smithi194_121512_object28779:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee36eaa:::benchmark_data_smithi194_121512_object57860:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee371d8:::benchmark_data_smithi194_121512_object112570:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee37e3d:::benchmark_data_smithi194_121512_object36737:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee3cad8:::benchmark_data_smithi194_121512_object8884:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee3cf8d:::benchmark_data_smithi194_121512_object34475:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee4096c:::benchmark_data_smithi194_121512_object101698:head# 2024-04-29T14:27:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee61575:::benchmark_data_smithi194_121512_object15520:head# 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee63463:::benchmark_data_smithi194_121512_object49870:head# 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee67d33:::benchmark_data_smithi194_121512_object68358:head# 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee6b971:::benchmark_data_smithi194_121512_object52556:head# 2024-04-29T14:27:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee773e6:::benchmark_data_smithi194_121512_object129374:head# 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee7c94e:::benchmark_data_smithi194_121512_object77298:head# 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee7f300:::benchmark_data_smithi194_121512_object58918:head# 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee82290:::benchmark_data_smithi194_121512_object43595:head# 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee824c7:::benchmark_data_smithi194_121512_object32825:head# 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.564 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee830ed:::benchmark_data_smithi194_121512_object45204:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee8598b:::benchmark_data_smithi194_121512_object4155:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee863ab:::benchmark_data_smithi194_121512_object4169:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee89f1b:::benchmark_data_smithi194_121512_object22839:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee8c057:::benchmark_data_smithi194_121512_object35241:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee8cf93:::benchmark_data_smithi194_121512_object7990:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ee9b135:::benchmark_data_smithi194_121512_object101225:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eea7044:::benchmark_data_smithi194_121512_object67963:head# 2024-04-29T14:27:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.566 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eea7751:::benchmark_data_smithi194_121512_object95849:head# 2024-04-29T14:27:13.566 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eea7cef:::benchmark_data_smithi194_121512_object77330:head# 2024-04-29T14:27:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eeae9e5:::benchmark_data_smithi194_121512_object98988:head# 2024-04-29T14:27:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eebc626:::benchmark_data_smithi194_121512_object21851:head# 2024-04-29T14:27:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eec9e6d:::benchmark_data_smithi194_121512_object61676:head# 2024-04-29T14:27:13.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eec9ea8:::benchmark_data_smithi194_121512_object101735:head# 2024-04-29T14:27:13.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.568 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eed342b:::benchmark_data_smithi194_121512_object72168:head# 2024-04-29T14:27:13.568 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.569 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eed47f3:::benchmark_data_smithi194_121512_object117129:head# 2024-04-29T14:27:13.569 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eedf5de:::benchmark_data_smithi194_121512_object97740:head# 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eee0b0b:::benchmark_data_smithi194_121512_object105537:head# 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eee1662:::benchmark_data_smithi194_121512_object75891:head# 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eee1df4:::benchmark_data_smithi194_121512_object13426:head# 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eee2b92:::benchmark_data_smithi194_121512_object95613:head# 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eee5b6c:::benchmark_data_smithi194_121512_object37682:head# 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eeeadd3:::benchmark_data_smithi194_121512_object5042:head# 2024-04-29T14:27:13.570 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eeee554:::benchmark_data_smithi194_121512_object43426:head# 2024-04-29T14:27:13.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eef1fb9:::benchmark_data_smithi194_121512_object102010:head# 2024-04-29T14:27:13.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.571 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eef3a76:::benchmark_data_smithi194_121512_object72116:head# 2024-04-29T14:27:13.571 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eef5618:::benchmark_data_smithi194_121512_object109440:head# 2024-04-29T14:27:13.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eef7b5a:::benchmark_data_smithi194_121512_object5101:head# 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eef8fc6:::benchmark_data_smithi194_121512_object21751:head# 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef01423:::benchmark_data_smithi194_121512_object91920:head# 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef03123:::benchmark_data_smithi194_121512_object109863:head# 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef0790f:::benchmark_data_smithi194_121512_object130565:head# 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef0e460:::benchmark_data_smithi194_121512_object14572:head# 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef0f7d7:::benchmark_data_smithi194_121512_object7097:head# 2024-04-29T14:27:13.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef1492d:::benchmark_data_smithi194_121512_object111754:head# 2024-04-29T14:27:13.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef23c58:::benchmark_data_smithi194_121512_object63119:head# 2024-04-29T14:27:13.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef37f34:::benchmark_data_smithi194_121512_object125514:head# 2024-04-29T14:27:13.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef3a180:::benchmark_data_smithi194_121512_object40424:head# 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef3a1fb:::benchmark_data_smithi194_121512_object80315:head# 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef40cb4:::benchmark_data_smithi194_121512_object39214:head# 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef448f7:::benchmark_data_smithi194_121512_object125738:head# 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef47415:::benchmark_data_smithi194_121512_object55204:head# 2024-04-29T14:27:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef4a0e0:::benchmark_data_smithi194_121512_object98462:head# 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef5c519:::benchmark_data_smithi194_121512_object80212:head# 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef68ec4:::benchmark_data_smithi194_121512_object78624:head# 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef717f5:::benchmark_data_smithi194_121512_object119707:head# 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef726aa:::benchmark_data_smithi194_121512_object78590:head# 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef7275c:::benchmark_data_smithi194_121512_object27515:head# 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef913a6:::benchmark_data_smithi194_121512_object54004:head# 2024-04-29T14:27:13.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef913b2:::benchmark_data_smithi194_121512_object34767:head# 2024-04-29T14:27:13.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ef92388:::benchmark_data_smithi194_121512_object11298:head# 2024-04-29T14:27:13.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.578 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efa48d7:::benchmark_data_smithi194_121512_object109261:head# 2024-04-29T14:27:13.578 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efa4d18:::benchmark_data_smithi194_121512_object132108:head# 2024-04-29T14:27:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efa4ea2:::benchmark_data_smithi194_121512_object4383:head# 2024-04-29T14:27:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efac8b7:::benchmark_data_smithi194_121512_object52468:head# 2024-04-29T14:27:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efb7970:::benchmark_data_smithi194_121512_object127157:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efb8b4d:::benchmark_data_smithi194_121512_object36914:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efba416:::benchmark_data_smithi194_121512_object70158:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efc5aee:::benchmark_data_smithi194_121512_object11383:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efcaa3a:::benchmark_data_smithi194_121512_object89774:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efd35bc:::benchmark_data_smithi194_121512_object127424:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efe271b:::benchmark_data_smithi194_121512_object77782:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6efe809a:::benchmark_data_smithi194_121512_object85642:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eff2466:::benchmark_data_smithi194_121512_object14593:head# 2024-04-29T14:27:13.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eff5baf:::benchmark_data_smithi194_121512_object1759:head# 2024-04-29T14:27:13.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6eff8bda:::benchmark_data_smithi194_121512_object75332:head# 2024-04-29T14:27:13.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.582 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6effff04:::benchmark_data_smithi194_121512_object79317:head# 2024-04-29T14:27:13.582 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0011e2:::benchmark_data_smithi194_121512_object32579:head# 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f00778e:::benchmark_data_smithi194_121512_object44863:head# 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f00e2af:::benchmark_data_smithi194_121512_object96283:head# 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f023948:::benchmark_data_smithi194_121512_object102652:head# 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f027fa2:::benchmark_data_smithi194_121512_object87165:head# 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0386f6:::benchmark_data_smithi194_121512_object132702:head# 2024-04-29T14:27:13.584 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f03a9ac:::benchmark_data_smithi194_121512_object116366:head# 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f03c4a2:::benchmark_data_smithi194_121512_object68655:head# 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f040070:::benchmark_data_smithi194_121512_object1128:head# 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f043ee1:::benchmark_data_smithi194_121512_object45426:head# 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f04930b:::benchmark_data_smithi194_121512_object36223:head# 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f04e450:::benchmark_data_smithi194_121512_object10320:head# 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f04e758:::benchmark_data_smithi194_121512_object43680:head# 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.585 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f04e774:::benchmark_data_smithi194_121512_object54830:head# 2024-04-29T14:27:13.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f05b46a:::benchmark_data_smithi194_121512_object82999:head# 2024-04-29T14:27:13.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.586 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0600a4:::benchmark_data_smithi194_121512_object60990:head# 2024-04-29T14:27:13.586 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f06dccf:::benchmark_data_smithi194_121512_object52081:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f070351:::benchmark_data_smithi194_121512_object54575:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f072a2c:::benchmark_data_smithi194_121512_object53338:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f07e1d7:::benchmark_data_smithi194_121512_object31404:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f082f07:::benchmark_data_smithi194_121512_object110027:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f08b8ce:::benchmark_data_smithi194_121512_object67795:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f08e4ff:::benchmark_data_smithi194_121512_object40845:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f08ee10:::benchmark_data_smithi194_121512_object99207:head# 2024-04-29T14:27:13.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f096174:::benchmark_data_smithi194_121512_object63040:head# 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0988b0:::benchmark_data_smithi194_121512_object111521:head# 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f09dab9:::benchmark_data_smithi194_121512_object55397:head# 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f09dde8:::benchmark_data_smithi194_121512_object103963:head# 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f09e16a:::benchmark_data_smithi194_121512_object133393:head# 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0a00bb:::benchmark_data_smithi194_121512_object106969:head# 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0a1b01:::benchmark_data_smithi194_121512_object64840:head# 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0a747b:::benchmark_data_smithi194_121512_object103581:head# 2024-04-29T14:27:13.591 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0a85a8:::benchmark_data_smithi194_121512_object25639:head# 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0b140d:::benchmark_data_smithi194_121512_object90132:head# 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0c8ea4:::benchmark_data_smithi194_121512_object25788:head# 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0c9fc3:::benchmark_data_smithi194_121512_object42171:head# 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0cb543:::benchmark_data_smithi194_121512_object88145:head# 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0d75e0:::benchmark_data_smithi194_121512_object117254:head# 2024-04-29T14:27:13.593 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0db5ee:::benchmark_data_smithi194_121512_object34131:head# 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0db8b2:::benchmark_data_smithi194_121512_object54360:head# 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0dd742:::benchmark_data_smithi194_121512_object4500:head# 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0e1d52:::benchmark_data_smithi194_121512_object105420:head# 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0e1f51:::benchmark_data_smithi194_121512_object68420:head# 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0f6e8d:::benchmark_data_smithi194_121512_object82944:head# 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f0f8568:::benchmark_data_smithi194_121512_object9960:head# 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.594 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1091c7:::benchmark_data_smithi194_121512_object18605:head# 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1095b9:::benchmark_data_smithi194_121512_object122850:head# 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f10bc0b:::benchmark_data_smithi194_121512_object118628:head# 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f11540b:::benchmark_data_smithi194_121512_object128308:head# 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f118662:::benchmark_data_smithi194_121512_object26930:head# 2024-04-29T14:27:13.595 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f129d22:::benchmark_data_smithi194_121512_object27211:head# 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f12a2bb:::benchmark_data_smithi194_121512_object61645:head# 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f13b08e:::benchmark_data_smithi194_121512_object22409:head# 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f13b2ab:::benchmark_data_smithi194_121512_object11312:head# 2024-04-29T14:27:13.596 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f15e712:::benchmark_data_smithi194_121512_object132205:head# 2024-04-29T14:27:13.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1614ef:::benchmark_data_smithi194_121512_object13659:head# 2024-04-29T14:27:13.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.597 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1650cb:::benchmark_data_smithi194_121512_object117427:head# 2024-04-29T14:27:13.597 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f171b60:::benchmark_data_smithi194_121512_object17372:head# 2024-04-29T14:27:13.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.598 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f172425:::benchmark_data_smithi194_121512_object120655:head# 2024-04-29T14:27:13.598 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f17ff75:::benchmark_data_smithi194_121512_object24606:head# 2024-04-29T14:27:13.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f186fae:::benchmark_data_smithi194_121512_object93732:head# 2024-04-29T14:27:13.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f187daf:::benchmark_data_smithi194_121512_object26713:head# 2024-04-29T14:27:13.599 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1896e9:::benchmark_data_smithi194_121512_object103941:head# 2024-04-29T14:27:13.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f19bfc1:::benchmark_data_smithi194_121512_object26387:head# 2024-04-29T14:27:13.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1a0910:::benchmark_data_smithi194_121512_object64666:head# 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1a92dc:::benchmark_data_smithi194_121512_object112634:head# 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1ab9c8:::benchmark_data_smithi194_121512_object120731:head# 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1b44c6:::benchmark_data_smithi194_121512_object8414:head# 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1bb2f9:::benchmark_data_smithi194_121512_object103846:head# 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1ca1da:::benchmark_data_smithi194_121512_object29536:head# 2024-04-29T14:27:13.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1cb19a:::benchmark_data_smithi194_121512_object130777:head# 2024-04-29T14:27:13.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1d6636:::benchmark_data_smithi194_121512_object56091:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1e749b:::benchmark_data_smithi194_121512_object20710:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f1fb473:::benchmark_data_smithi194_121512_object6710:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f203ee5:::benchmark_data_smithi194_121512_object125219:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f206807:::benchmark_data_smithi194_121512_object82280:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f208d6f:::benchmark_data_smithi194_121512_object64663:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f218e3d:::benchmark_data_smithi194_121512_object87347:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f21a524:::benchmark_data_smithi194_121512_object31140:head# 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f21e49b:::benchmark_data_smithi194_121512_object59946:head# 2024-04-29T14:27:13.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f21edf0:::benchmark_data_smithi194_121512_object61289:head# 2024-04-29T14:27:13.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2238ca:::benchmark_data_smithi194_121512_object57947:head# 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f22cb44:::benchmark_data_smithi194_121512_object18671:head# 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f24c080:::benchmark_data_smithi194_121512_object24064:head# 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f260a98:::benchmark_data_smithi194_121512_object131676:head# 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f267db2:::benchmark_data_smithi194_121512_object121497:head# 2024-04-29T14:27:13.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f26ceaf:::benchmark_data_smithi194_121512_object15771:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f279aa1:::benchmark_data_smithi194_121512_object43212:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f285bce:::benchmark_data_smithi194_121512_object42197:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f28d062:::benchmark_data_smithi194_121512_object56571:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f28fafc:::benchmark_data_smithi194_121512_object79975:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f29323a:::benchmark_data_smithi194_121512_object106552:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f29e121:::benchmark_data_smithi194_121512_object106895:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2a2962:::benchmark_data_smithi194_121512_object93756:head# 2024-04-29T14:27:13.609 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2a4b3c:::benchmark_data_smithi194_121512_object56050:head# 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2ae563:::benchmark_data_smithi194_121512_object115713:head# 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2af4f4:::benchmark_data_smithi194_121512_object24342:head# 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2b3feb:::benchmark_data_smithi194_121512_object126552:head# 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2b4603:::benchmark_data_smithi194_121512_object42895:head# 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2b7bc9:::benchmark_data_smithi194_121512_object123611:head# 2024-04-29T14:27:13.610 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2b98c7:::benchmark_data_smithi194_121512_object24514:head# 2024-04-29T14:27:13.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.611 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2bbd38:::benchmark_data_smithi194_121512_object107560:head# 2024-04-29T14:27:13.611 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:13.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2bd44e:::benchmark_data_smithi194_121512_object23581:head# 2024-04-29T14:27:13.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2d9afd:::benchmark_data_smithi194_121512_object117901:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f2f9578:::benchmark_data_smithi194_121512_object60973:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f30165f:::benchmark_data_smithi194_121512_object95071:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f308de3:::benchmark_data_smithi194_121512_object60706:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f30b8aa:::benchmark_data_smithi194_121512_object94452:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f30d05e:::benchmark_data_smithi194_121512_object21888:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f313fdb:::benchmark_data_smithi194_121512_object23703:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f31ff2b:::benchmark_data_smithi194_121512_object95314:head# 2024-04-29T14:27:13.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f335aaf:::benchmark_data_smithi194_121512_object60347:head# 2024-04-29T14:27:13.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f34014b:::benchmark_data_smithi194_121512_object51227:head# 2024-04-29T14:27:13.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f343e50:::benchmark_data_smithi194_121512_object96368:head# 2024-04-29T14:27:13.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f34b64b:::benchmark_data_smithi194_121512_object74595:head# 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f350e34:::benchmark_data_smithi194_121512_object131209:head# 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f35e37f:::benchmark_data_smithi194_121512_object10050:head# 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f362379:::benchmark_data_smithi194_121512_object125371:head# 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f362ed0:::benchmark_data_smithi194_121512_object119072:head# 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f369b6a:::benchmark_data_smithi194_121512_object123868:head# 2024-04-29T14:27:13.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f374d28:::benchmark_data_smithi194_121512_object1050:head# 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f376b6d:::benchmark_data_smithi194_121512_object2819:head# 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f377d03:::benchmark_data_smithi194_121512_object110722:head# 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f37c9c9:::benchmark_data_smithi194_121512_object17479:head# 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f38c1c8:::benchmark_data_smithi194_121512_object88331:head# 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f398e92:::benchmark_data_smithi194_121512_object54756:head# 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3996ac:::benchmark_data_smithi194_121512_object126635:head# 2024-04-29T14:27:13.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3a014d:::benchmark_data_smithi194_121512_object8274:head# 2024-04-29T14:27:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3a5aff:::benchmark_data_smithi194_121512_object33823:head# 2024-04-29T14:27:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3a60da:::benchmark_data_smithi194_121512_object91001:head# 2024-04-29T14:27:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3aa5ab:::benchmark_data_smithi194_121512_object14534:head# 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3b1d96:::benchmark_data_smithi194_121512_object110261:head# 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3bc032:::benchmark_data_smithi194_121512_object26430:head# 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3be288:::benchmark_data_smithi194_121512_object86044:head# 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3c8704:::benchmark_data_smithi194_121512_object12874:head# 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3cacd5:::benchmark_data_smithi194_121512_object52562:head# 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3cca3a:::benchmark_data_smithi194_121512_object38785:head# 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.629 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3d1438:::benchmark_data_smithi194_121512_object21444:head# 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3d7e71:::benchmark_data_smithi194_121512_object16132:head# 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3e6784:::benchmark_data_smithi194_121512_object116782:head# 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3e92d9:::benchmark_data_smithi194_121512_object3594:head# 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3f1e32:::benchmark_data_smithi194_121512_object69285:head# 2024-04-29T14:27:13.630 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f3f2ca0:::benchmark_data_smithi194_121512_object112355:head# 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f412d21:::benchmark_data_smithi194_121512_object60900:head# 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f417d1d:::benchmark_data_smithi194_121512_object56972:head# 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f42b59c:::benchmark_data_smithi194_121512_object87942:head# 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f438aee:::benchmark_data_smithi194_121512_object23763:head# 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4443be:::benchmark_data_smithi194_121512_object63058:head# 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f455a39:::benchmark_data_smithi194_121512_object118766:head# 2024-04-29T14:27:13.633 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f45857a:::benchmark_data_smithi194_121512_object128459:head# 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f46479f:::benchmark_data_smithi194_121512_object110633:head# 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f472473:::benchmark_data_smithi194_121512_object88381:head# 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f475d9b:::benchmark_data_smithi194_121512_object90420:head# 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f488905:::benchmark_data_smithi194_121512_object56243:head# 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f48955d:::benchmark_data_smithi194_121512_object38265:head# 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f495bb1:::benchmark_data_smithi194_121512_object27931:head# 2024-04-29T14:27:13.634 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f49da19:::benchmark_data_smithi194_121512_object85859:head# 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4a167e:::benchmark_data_smithi194_121512_object61105:head# 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4a46d0:::benchmark_data_smithi194_121512_object68590:head# 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4a7df3:::benchmark_data_smithi194_121512_object94167:head# 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4af965:::benchmark_data_smithi194_121512_object34636:head# 2024-04-29T14:27:13.635 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.638 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4c45a2:::benchmark_data_smithi194_121512_object78678:head# 2024-04-29T14:27:13.638 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4c4823:::benchmark_data_smithi194_121512_object27637:head# 2024-04-29T14:27:13.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.639 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4cd94e:::benchmark_data_smithi194_121512_object112474:head# 2024-04-29T14:27:13.639 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.640 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4d783e:::benchmark_data_smithi194_121512_object91405:head# 2024-04-29T14:27:13.640 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4d893b:::benchmark_data_smithi194_121512_object88183:head# 2024-04-29T14:27:13.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4d8aa8:::benchmark_data_smithi194_121512_object11791:head# 2024-04-29T14:27:13.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4e631f:::benchmark_data_smithi194_121512_object105966:head# 2024-04-29T14:27:13.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4ee2b0:::benchmark_data_smithi194_121512_object80326:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4f2aa1:::benchmark_data_smithi194_121512_object110377:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4f843e:::benchmark_data_smithi194_121512_object8238:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4fa3fe:::benchmark_data_smithi194_121512_object64225:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f4fbf3f:::benchmark_data_smithi194_121512_object50391:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5126f9:::benchmark_data_smithi194_121512_object127081:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f516e86:::benchmark_data_smithi194_121512_object131366:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f52fa4b:::benchmark_data_smithi194_121512_object61066:head# 2024-04-29T14:27:13.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5345c3:::benchmark_data_smithi194_121512_object121145:head# 2024-04-29T14:27:13.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f537640:::benchmark_data_smithi194_121512_object81361:head# 2024-04-29T14:27:13.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f53ca10:::benchmark_data_smithi194_121512_object112358:head# 2024-04-29T14:27:13.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f543ff4:::benchmark_data_smithi194_121512_object86440:head# 2024-04-29T14:27:13.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f54f142:::benchmark_data_smithi194_121512_object88094:head# 2024-04-29T14:27:13.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f555b37:::benchmark_data_smithi194_121512_object24793:head# 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f55e268:::benchmark_data_smithi194_121512_object4766:head# 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f55ff46:::benchmark_data_smithi194_121512_object117585:head# 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f562e73:::benchmark_data_smithi194_121512_object63925:head# 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5665bc:::benchmark_data_smithi194_121512_object97379:head# 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f56a740:::benchmark_data_smithi194_121512_object101834:head# 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f571a5f:::benchmark_data_smithi194_121512_object124102:head# 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f57a4b0:::benchmark_data_smithi194_121512_object30867:head# 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f57da33:::benchmark_data_smithi194_121512_object64103:head# 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f597b7a:::benchmark_data_smithi194_121512_object29991:head# 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f59890d:::benchmark_data_smithi194_121512_object124117:head# 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f599334:::benchmark_data_smithi194_121512_object132038:head# 2024-04-29T14:27:13.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5a597f:::benchmark_data_smithi194_121512_object17386:head# 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5ad36d:::benchmark_data_smithi194_121512_object23779:head# 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5b0f3e:::benchmark_data_smithi194_121512_object88787:head# 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5b6b0f:::benchmark_data_smithi194_121512_object91749:head# 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5b8b1c:::benchmark_data_smithi194_121512_object83212:head# 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5b8bb8:::benchmark_data_smithi194_121512_object35775:head# 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5bb77e:::benchmark_data_smithi194_121512_object13879:head# 2024-04-29T14:27:13.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5ccdbd:::benchmark_data_smithi194_121512_object93612:head# 2024-04-29T14:27:13.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5cfa69:::benchmark_data_smithi194_121512_object86242:head# 2024-04-29T14:27:13.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5e1307:::benchmark_data_smithi194_121512_object44708:head# 2024-04-29T14:27:13.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5e4485:::benchmark_data_smithi194_121512_object12174:head# 2024-04-29T14:27:13.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5f14ad:::benchmark_data_smithi194_121512_object1501:head# 2024-04-29T14:27:13.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f5ff264:::benchmark_data_smithi194_121512_object17529:head# 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f607172:::benchmark_data_smithi194_121512_object110507:head# 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f609561:::benchmark_data_smithi194_121512_object73692:head# 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f60d7b8:::benchmark_data_smithi194_121512_object101331:head# 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6267e8:::benchmark_data_smithi194_121512_object84193:head# 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f626af6:::benchmark_data_smithi194_121512_object25744:head# 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f62f20e:::benchmark_data_smithi194_121512_object68818:head# 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f63273f:::benchmark_data_smithi194_121512_object19204:head# 2024-04-29T14:27:13.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.661 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f640248:::benchmark_data_smithi194_121512_object31350:head# 2024-04-29T14:27:13.661 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f647d38:::benchmark_data_smithi194_121512_object43813:head# 2024-04-29T14:27:13.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f64ef15:::benchmark_data_smithi194_121512_object85840:head# 2024-04-29T14:27:13.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.663 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f651cb1:::benchmark_data_smithi194_121512_object42364:head# 2024-04-29T14:27:13.663 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f652ec6:::benchmark_data_smithi194_121512_object110638:head# 2024-04-29T14:27:13.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.664 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6660f9:::benchmark_data_smithi194_121512_object130717:head# 2024-04-29T14:27:13.664 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f666b3c:::benchmark_data_smithi194_121512_object118043:head# 2024-04-29T14:27:13.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.665 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f66c568:::benchmark_data_smithi194_121512_object132560:head# 2024-04-29T14:27:13.665 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f66ec2f:::benchmark_data_smithi194_121512_object76354:head# 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6732de:::benchmark_data_smithi194_121512_object121863:head# 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f67364e:::benchmark_data_smithi194_121512_object97419:head# 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f680410:::benchmark_data_smithi194_121512_object118158:head# 2024-04-29T14:27:13.666 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f683a70:::benchmark_data_smithi194_121512_object112316:head# 2024-04-29T14:27:13.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.667 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f686a5b:::benchmark_data_smithi194_121512_object88699:head# 2024-04-29T14:27:13.667 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.683 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f68d983:::benchmark_data_smithi194_121512_object28680:head# 2024-04-29T14:27:13.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f68dd1c:::benchmark_data_smithi194_121512_object114173:head# 2024-04-29T14:27:13.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f691001:::benchmark_data_smithi194_121512_object15958:head# 2024-04-29T14:27:13.684 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.684 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f69c5cc:::benchmark_data_smithi194_121512_object74846:head# 2024-04-29T14:27:13.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f69d294:::benchmark_data_smithi194_121512_object51031:head# 2024-04-29T14:27:13.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.685 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6a3139:::benchmark_data_smithi194_121512_object40370:head# 2024-04-29T14:27:13.685 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6a3e96:::benchmark_data_smithi194_121512_object97094:head# 2024-04-29T14:27:13.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.686 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6a532b:::benchmark_data_smithi194_121512_object70676:head# 2024-04-29T14:27:13.686 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6a8312:::benchmark_data_smithi194_121512_object12385:head# 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6a9612:::benchmark_data_smithi194_121512_object104658:head# 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6aedd0:::benchmark_data_smithi194_121512_object33622:head# 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6c0910:::benchmark_data_smithi194_121512_object60106:head# 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6c1343:::benchmark_data_smithi194_121512_object105538:head# 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.688 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6c405e:::benchmark_data_smithi194_121512_object124744:head# 2024-04-29T14:27:13.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6c86cc:::benchmark_data_smithi194_121512_object121239:head# 2024-04-29T14:27:13.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6c8b21:::benchmark_data_smithi194_121512_object26675:head# 2024-04-29T14:27:13.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.689 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6c991b:::benchmark_data_smithi194_121512_object21026:head# 2024-04-29T14:27:13.689 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6cbd08:::benchmark_data_smithi194_121512_object94539:head# 2024-04-29T14:27:13.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6d0d6a:::benchmark_data_smithi194_121512_object118736:head# 2024-04-29T14:27:13.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6d1d6c:::benchmark_data_smithi194_121512_object39832:head# 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6dec37:::benchmark_data_smithi194_121512_object10864:head# 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6e24cf:::benchmark_data_smithi194_121512_object128097:head# 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6ece2a:::benchmark_data_smithi194_121512_object82272:head# 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f6f6e50:::benchmark_data_smithi194_121512_object91800:head# 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f70ffc6:::benchmark_data_smithi194_121512_object50681:head# 2024-04-29T14:27:13.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f714abe:::benchmark_data_smithi194_121512_object113820:head# 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f71a6ad:::benchmark_data_smithi194_121512_object17756:head# 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7267a0:::benchmark_data_smithi194_121512_object101076:head# 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f72e623:::benchmark_data_smithi194_121512_object72066:head# 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f734118:::benchmark_data_smithi194_121512_object17924:head# 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f73a857:::benchmark_data_smithi194_121512_object36604:head# 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f745b8c:::benchmark_data_smithi194_121512_object90345:head# 2024-04-29T14:27:13.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7487dc:::benchmark_data_smithi194_121512_object116234:head# 2024-04-29T14:27:13.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f749b89:::benchmark_data_smithi194_121512_object41692:head# 2024-04-29T14:27:13.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f74a6a7:::benchmark_data_smithi194_121512_object12176:head# 2024-04-29T14:27:13.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7592b1:::benchmark_data_smithi194_121512_object106027:head# 2024-04-29T14:27:13.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f75ceed:::benchmark_data_smithi194_121512_object66660:head# 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f75dec7:::benchmark_data_smithi194_121512_object98807:head# 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f761464:::benchmark_data_smithi194_121512_object51942:head# 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f766631:::benchmark_data_smithi194_121512_object27096:head# 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f76d311:::benchmark_data_smithi194_121512_object118160:head# 2024-04-29T14:27:13.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f76fe3c:::benchmark_data_smithi194_121512_object16073:head# 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f78eeb8:::benchmark_data_smithi194_121512_object67636:head# 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f79300a:::benchmark_data_smithi194_121512_object55678:head# 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7962a0:::benchmark_data_smithi194_121512_object132735:head# 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f799321:::benchmark_data_smithi194_121512_object61882:head# 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7a0602:::benchmark_data_smithi194_121512_object16747:head# 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7a5b20:::benchmark_data_smithi194_121512_object108913:head# 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7a73e7:::benchmark_data_smithi194_121512_object71575:head# 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7a75f7:::benchmark_data_smithi194_121512_object123499:head# 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7ae9a6:::benchmark_data_smithi194_121512_object62459:head# 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7bcdb2:::benchmark_data_smithi194_121512_object43541:head# 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7bf9f2:::benchmark_data_smithi194_121512_object43025:head# 2024-04-29T14:27:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7c1cb3:::benchmark_data_smithi194_121512_object40362:head# 2024-04-29T14:27:13.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7c2beb:::benchmark_data_smithi194_121512_object54259:head# 2024-04-29T14:27:13.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7ca8da:::benchmark_data_smithi194_121512_object55078:head# 2024-04-29T14:27:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7d468d:::benchmark_data_smithi194_121512_object34837:head# 2024-04-29T14:27:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7d6526:::benchmark_data_smithi194_121512_object29741:head# 2024-04-29T14:27:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7d696e:::benchmark_data_smithi194_121512_object121603:head# 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7dc660:::benchmark_data_smithi194_121512_object38284:head# 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7e297a:::benchmark_data_smithi194_121512_object75933:head# 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7e93d9:::benchmark_data_smithi194_121512_object96578:head# 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f7f1169:::benchmark_data_smithi194_121512_object29577:head# 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f803e58:::benchmark_data_smithi194_121512_object48516:head# 2024-04-29T14:27:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f804b34:::benchmark_data_smithi194_121512_object64021:head# 2024-04-29T14:27:13.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f807d1d:::benchmark_data_smithi194_121512_object48550:head# 2024-04-29T14:27:13.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f812d49:::benchmark_data_smithi194_121512_object49252:head# 2024-04-29T14:27:13.729 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.729 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f822334:::benchmark_data_smithi194_121512_object102822:head# 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f829b96:::benchmark_data_smithi194_121512_object56884:head# 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f837f74:::benchmark_data_smithi194_121512_object47115:head# 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8405ec:::benchmark_data_smithi194_121512_object94815:head# 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f842cb8:::benchmark_data_smithi194_121512_object70560:head# 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f848ca9:::benchmark_data_smithi194_121512_object40153:head# 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f868524:::benchmark_data_smithi194_121512_object33109:head# 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.730 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f868a1f:::benchmark_data_smithi194_121512_object80501:head# 2024-04-29T14:27:13.731 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f871474:::benchmark_data_smithi194_121512_object19546:head# 2024-04-29T14:27:13.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f871b36:::benchmark_data_smithi194_121512_object10714:head# 2024-04-29T14:27:13.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.732 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f876d96:::benchmark_data_smithi194_121512_object25768:head# 2024-04-29T14:27:13.732 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f88456c:::benchmark_data_smithi194_121512_object107618:head# 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f88e5c7:::benchmark_data_smithi194_121512_object43080:head# 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f890141:::benchmark_data_smithi194_121512_object122744:head# 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f89735c:::benchmark_data_smithi194_121512_object34035:head# 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f89b7aa:::benchmark_data_smithi194_121512_object90687:head# 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f89dd76:::benchmark_data_smithi194_121512_object113510:head# 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f89faa0:::benchmark_data_smithi194_121512_object130200:head# 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8a09b2:::benchmark_data_smithi194_121512_object54157:head# 2024-04-29T14:27:13.734 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.736 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8a1b61:::benchmark_data_smithi194_121512_object111829:head# 2024-04-29T14:27:13.736 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8a2a4f:::benchmark_data_smithi194_121512_object25857:head# 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8ab53c:::benchmark_data_smithi194_121512_object87872:head# 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8aed08:::benchmark_data_smithi194_121512_object44938:head# 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8aeec0:::benchmark_data_smithi194_121512_object52856:head# 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8b2c39:::benchmark_data_smithi194_121512_object41405:head# 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8b9a81:::benchmark_data_smithi194_121512_object119464:head# 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8bb92b:::benchmark_data_smithi194_121512_object24471:head# 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.737 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8bbf67:::benchmark_data_smithi194_121512_object31857:head# 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8d26fe:::benchmark_data_smithi194_121512_object60078:head# 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8d3298:::benchmark_data_smithi194_121512_object47948:head# 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8d74ee:::benchmark_data_smithi194_121512_object103137:head# 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8e6a7f:::benchmark_data_smithi194_121512_object106417:head# 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8e8615:::benchmark_data_smithi194_121512_object46578:head# 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8ea099:::benchmark_data_smithi194_121512_object48922:head# 2024-04-29T14:27:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8f43f9:::benchmark_data_smithi194_121512_object49782:head# 2024-04-29T14:27:13.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8fc34a:::benchmark_data_smithi194_121512_object98634:head# 2024-04-29T14:27:13.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.739 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f8fd13b:::benchmark_data_smithi194_121512_object4369:head# 2024-04-29T14:27:13.739 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f908731:::benchmark_data_smithi194_121512_object31933:head# 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f92b142:::benchmark_data_smithi194_121512_object6654:head# 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f934700:::benchmark_data_smithi194_121512_object24816:head# 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f93ccbb:::benchmark_data_smithi194_121512_object104608:head# 2024-04-29T14:27:13.741 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f93f763:::benchmark_data_smithi194_121512_object35836:head# 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9418ea:::benchmark_data_smithi194_121512_object125111:head# 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f942ac9:::benchmark_data_smithi194_121512_object72709:head# 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f94755c:::benchmark_data_smithi194_121512_object87300:head# 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f951796:::benchmark_data_smithi194_121512_object68472:head# 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9570fd:::benchmark_data_smithi194_121512_object78514:head# 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f967675:::benchmark_data_smithi194_121512_object124576:head# 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.742 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f969f6a:::benchmark_data_smithi194_121512_object93775:head# 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f96cc5e:::benchmark_data_smithi194_121512_object28042:head# 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f976333:::benchmark_data_smithi194_121512_object72904:head# 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f97c1fc:::benchmark_data_smithi194_121512_object91943:head# 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9887bd:::benchmark_data_smithi194_121512_object81377:head# 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f98b772:::benchmark_data_smithi194_121512_object130640:head# 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f992c4e:::benchmark_data_smithi194_121512_object55698:head# 2024-04-29T14:27:13.743 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f993a22:::benchmark_data_smithi194_121512_object79710:head# 2024-04-29T14:27:13.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f997661:::benchmark_data_smithi194_121512_object13465:head# 2024-04-29T14:27:13.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.744 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f997c5d:::benchmark_data_smithi194_121512_object27877:head# 2024-04-29T14:27:13.744 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f998a53:::benchmark_data_smithi194_121512_object89690:head# 2024-04-29T14:27:13.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.748 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f99ad59:::benchmark_data_smithi194_121512_object104882:head# 2024-04-29T14:27:13.748 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.750 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9a27d7:::benchmark_data_smithi194_121512_object39475:head# 2024-04-29T14:27:13.750 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9aa1a0:::benchmark_data_smithi194_121512_object70994:head# 2024-04-29T14:27:13.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9ad5d6:::benchmark_data_smithi194_121512_object89999:head# 2024-04-29T14:27:13.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.753 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9b20ad:::benchmark_data_smithi194_121512_object60478:head# 2024-04-29T14:27:13.753 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9b4f63:::benchmark_data_smithi194_121512_object126468:head# 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9c0a01:::benchmark_data_smithi194_121512_object51450:head# 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9cfeb9:::benchmark_data_smithi194_121512_object22418:head# 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9da2e6:::benchmark_data_smithi194_121512_object15453:head# 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9e1ff5:::benchmark_data_smithi194_121512_object122483:head# 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9e4e1f:::benchmark_data_smithi194_121512_object79603:head# 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9e8c2b:::benchmark_data_smithi194_121512_object78745:head# 2024-04-29T14:27:13.754 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.755 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6f9e9291:::benchmark_data_smithi194_121512_object111135:head# 2024-04-29T14:27:13.755 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa020d4:::benchmark_data_smithi194_121512_object126607:head# 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa1e4e9:::benchmark_data_smithi194_121512_object73685:head# 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa1e67f:::benchmark_data_smithi194_121512_object27398:head# 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa20f42:::benchmark_data_smithi194_121512_object29780:head# 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa23c2f:::benchmark_data_smithi194_121512_object103181:head# 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa2bd10:::benchmark_data_smithi194_121512_object102660:head# 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa409a9:::benchmark_data_smithi194_121512_object52209:head# 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa425a7:::benchmark_data_smithi194_121512_object43945:head# 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa46a4c:::benchmark_data_smithi194_121512_object36497:head# 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa4b7fc:::benchmark_data_smithi194_121512_object126140:head# 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa58c04:::benchmark_data_smithi194_121512_object53795:head# 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa5f6be:::benchmark_data_smithi194_121512_object129615:head# 2024-04-29T14:27:13.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa622fd:::benchmark_data_smithi194_121512_object46883:head# 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa66bb5:::benchmark_data_smithi194_121512_object31744:head# 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa6ba66:::benchmark_data_smithi194_121512_object87040:head# 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa6d302:::benchmark_data_smithi194_121512_object41025:head# 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa6f084:::benchmark_data_smithi194_121512_object77508:head# 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa70809:::benchmark_data_smithi194_121512_object141:head# 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa8bfe0:::benchmark_data_smithi194_121512_object106737:head# 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa93096:::benchmark_data_smithi194_121512_object72234:head# 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fa9bbbe:::benchmark_data_smithi194_121512_object12075:head# 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6faadbcb:::benchmark_data_smithi194_121512_object53341:head# 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fabb945:::benchmark_data_smithi194_121512_object23390:head# 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fabc4e4:::benchmark_data_smithi194_121512_object54149:head# 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fabd0ec:::benchmark_data_smithi194_121512_object39058:head# 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fac127d:::benchmark_data_smithi194_121512_object119376:head# 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fac5975:::benchmark_data_smithi194_121512_object101412:head# 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fac6eee:::benchmark_data_smithi194_121512_object28960:head# 2024-04-29T14:27:13.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fad0b68:::benchmark_data_smithi194_121512_object100833:head# 2024-04-29T14:27:13.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fad4ea5:::benchmark_data_smithi194_121512_object54973:head# 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fad65b2:::benchmark_data_smithi194_121512_object46134:head# 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fae8ec0:::benchmark_data_smithi194_121512_object83307:head# 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6faed24f:::benchmark_data_smithi194_121512_object112703:head# 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6faee788:::benchmark_data_smithi194_121512_object75088:head# 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb02ca3:::benchmark_data_smithi194_121512_object32186:head# 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb10511:::benchmark_data_smithi194_121512_object120784:head# 2024-04-29T14:27:13.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb180b3:::benchmark_data_smithi194_121512_object1597:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb19019:::benchmark_data_smithi194_121512_object36584:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb1e060:::benchmark_data_smithi194_121512_object81187:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb2d5a2:::benchmark_data_smithi194_121512_object116529:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb49f2f:::benchmark_data_smithi194_121512_object81096:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb5c4b7:::benchmark_data_smithi194_121512_object48597:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb6408d:::benchmark_data_smithi194_121512_object91813:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb6da70:::benchmark_data_smithi194_121512_object8260:head# 2024-04-29T14:27:13.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb763de:::benchmark_data_smithi194_121512_object52176:head# 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb8550e:::benchmark_data_smithi194_121512_object54880:head# 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb9a453:::benchmark_data_smithi194_121512_object2025:head# 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb9bf20:::benchmark_data_smithi194_121512_object112519:head# 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fb9cabd:::benchmark_data_smithi194_121512_object130808:head# 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fba5848:::benchmark_data_smithi194_121512_object35665:head# 2024-04-29T14:27:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbac197:::benchmark_data_smithi194_121512_object83933:head# 2024-04-29T14:27:13.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbad3b3:::benchmark_data_smithi194_121512_object113835:head# 2024-04-29T14:27:13.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbb16f9:::benchmark_data_smithi194_121512_object3385:head# 2024-04-29T14:27:13.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbbacb9:::benchmark_data_smithi194_121512_object74696:head# 2024-04-29T14:27:13.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbc1361:::benchmark_data_smithi194_121512_object79780:head# 2024-04-29T14:27:13.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbcb077:::benchmark_data_smithi194_121512_object112083:head# 2024-04-29T14:27:13.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbd228d:::benchmark_data_smithi194_121512_object47746:head# 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbddf1c:::benchmark_data_smithi194_121512_object44188:head# 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbe2ba7:::benchmark_data_smithi194_121512_object101636:head# 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbe336a:::benchmark_data_smithi194_121512_object124212:head# 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbebaf7:::benchmark_data_smithi194_121512_object108276:head# 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbee244:::benchmark_data_smithi194_121512_object76631:head# 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbfa497:::benchmark_data_smithi194_121512_object51200:head# 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fbff837:::benchmark_data_smithi194_121512_object44585:head# 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc00aea:::benchmark_data_smithi194_121512_object28265:head# 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc14c4b:::benchmark_data_smithi194_121512_object80242:head# 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc18fd0:::benchmark_data_smithi194_121512_object128164:head# 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc1cf9c:::benchmark_data_smithi194_121512_object23597:head# 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc1ffb8:::benchmark_data_smithi194_121512_object68663:head# 2024-04-29T14:27:13.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc231c8:::benchmark_data_smithi194_121512_object76990:head# 2024-04-29T14:27:13.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc36a63:::benchmark_data_smithi194_121512_object89658:head# 2024-04-29T14:27:13.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc38e3a:::benchmark_data_smithi194_121512_object63674:head# 2024-04-29T14:27:13.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc3b50a:::benchmark_data_smithi194_121512_object67156:head# 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc3c431:::benchmark_data_smithi194_121512_object9961:head# 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc4329b:::benchmark_data_smithi194_121512_object112212:head# 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc467d5:::benchmark_data_smithi194_121512_object96280:head# 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc56c67:::benchmark_data_smithi194_121512_object59159:head# 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc63ca3:::benchmark_data_smithi194_121512_object16926:head# 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc6f0d9:::benchmark_data_smithi194_121512_object84005:head# 2024-04-29T14:27:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc6f184:::benchmark_data_smithi194_121512_object43774:head# 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc86209:::benchmark_data_smithi194_121512_object40173:head# 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc89287:::benchmark_data_smithi194_121512_object103028:head# 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc89700:::benchmark_data_smithi194_121512_object33878:head# 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc90bc1:::benchmark_data_smithi194_121512_object124972:head# 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc9bed8:::benchmark_data_smithi194_121512_object39081:head# 2024-04-29T14:27:13.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fc9e578:::benchmark_data_smithi194_121512_object10959:head# 2024-04-29T14:27:13.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fca0b84:::benchmark_data_smithi194_121512_object68684:head# 2024-04-29T14:27:13.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcb888d:::benchmark_data_smithi194_121512_object129201:head# 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcbcbbf:::benchmark_data_smithi194_121512_object16165:head# 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcbeb81:::benchmark_data_smithi194_121512_object57633:head# 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcc567d:::benchmark_data_smithi194_121512_object78926:head# 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcc6d41:::benchmark_data_smithi194_121512_object39553:head# 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fccec78:::benchmark_data_smithi194_121512_object44165:head# 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcd2299:::benchmark_data_smithi194_121512_object9148:head# 2024-04-29T14:27:13.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.790 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcd6d8c:::benchmark_data_smithi194_121512_object33568:head# 2024-04-29T14:27:13.790 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fce64b7:::benchmark_data_smithi194_121512_object70729:head# 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcf30ba:::benchmark_data_smithi194_121512_object1518:head# 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcf4c89:::benchmark_data_smithi194_121512_object97981:head# 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcf8134:::benchmark_data_smithi194_121512_object102176:head# 2024-04-29T14:27:13.795 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fcf959c:::benchmark_data_smithi194_121512_object89308:head# 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd01d36:::benchmark_data_smithi194_121512_object88905:head# 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd06ea6:::benchmark_data_smithi194_121512_object110127:head# 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd0710c:::benchmark_data_smithi194_121512_object18501:head# 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd0973a:::benchmark_data_smithi194_121512_object52329:head# 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.797 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd128b1:::benchmark_data_smithi194_121512_object10229:head# 2024-04-29T14:27:13.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.798 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd1bcec:::benchmark_data_smithi194_121512_object33463:head# 2024-04-29T14:27:13.798 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd247da:::benchmark_data_smithi194_121512_object46382:head# 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd30609:::benchmark_data_smithi194_121512_object96507:head# 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd3f78a:::benchmark_data_smithi194_121512_object105425:head# 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd485b0:::benchmark_data_smithi194_121512_object44376:head# 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd51080:::benchmark_data_smithi194_121512_object73343:head# 2024-04-29T14:27:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd66ea3:::benchmark_data_smithi194_121512_object4960:head# 2024-04-29T14:27:13.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd6e9d7:::benchmark_data_smithi194_121512_object61919:head# 2024-04-29T14:27:13.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.800 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd6fbee:::benchmark_data_smithi194_121512_object69911:head# 2024-04-29T14:27:13.800 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd71023:::benchmark_data_smithi194_121512_object4821:head# 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd725ee:::benchmark_data_smithi194_121512_object54056:head# 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd739b3:::benchmark_data_smithi194_121512_object129964:head# 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd80def:::benchmark_data_smithi194_121512_object100226:head# 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd8113c:::benchmark_data_smithi194_121512_object126709:head# 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd88366:::benchmark_data_smithi194_121512_object30422:head# 2024-04-29T14:27:13.801 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.802 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd89f38:::benchmark_data_smithi194_121512_object75707:head# 2024-04-29T14:27:13.802 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fd9d5a1:::benchmark_data_smithi194_121512_object71635:head# 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fda891e:::benchmark_data_smithi194_121512_object77744:head# 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fda9b52:::benchmark_data_smithi194_121512_object43679:head# 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdad788:::benchmark_data_smithi194_121512_object874:head# 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdaeb9a:::benchmark_data_smithi194_121512_object27900:head# 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdb3843:::benchmark_data_smithi194_121512_object36062:head# 2024-04-29T14:27:13.804 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdb68e6:::benchmark_data_smithi194_121512_object93670:head# 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdd2a32:::benchmark_data_smithi194_121512_object51432:head# 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdd7860:::benchmark_data_smithi194_121512_object110869:head# 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdd8a00:::benchmark_data_smithi194_121512_object123548:head# 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fddf732:::benchmark_data_smithi194_121512_object52157:head# 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fde4416:::benchmark_data_smithi194_121512_object61394:head# 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fde7bd8:::benchmark_data_smithi194_121512_object69588:head# 2024-04-29T14:27:13.805 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fdf312f:::benchmark_data_smithi194_121512_object74480:head# 2024-04-29T14:27:13.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe17e60:::benchmark_data_smithi194_121512_object23327:head# 2024-04-29T14:27:13.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe199ff:::benchmark_data_smithi194_121512_object69393:head# 2024-04-29T14:27:13.809 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.809 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe1adba:::benchmark_data_smithi194_121512_object88048:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe28dc8:::benchmark_data_smithi194_121512_object28405:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe388a3:::benchmark_data_smithi194_121512_object84175:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe39c89:::benchmark_data_smithi194_121512_object24560:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe480c8:::benchmark_data_smithi194_121512_object110386:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe512c2:::benchmark_data_smithi194_121512_object9883:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe5775d:::benchmark_data_smithi194_121512_object95997:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe5dc04:::benchmark_data_smithi194_121512_object12501:head# 2024-04-29T14:27:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe6af3b:::benchmark_data_smithi194_121512_object52646:head# 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe6c3b6:::benchmark_data_smithi194_121512_object98519:head# 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe767f2:::benchmark_data_smithi194_121512_object51389:head# 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe845f1:::benchmark_data_smithi194_121512_object35037:head# 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe8deb6:::benchmark_data_smithi194_121512_object76982:head# 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe8f8e8:::benchmark_data_smithi194_121512_object16635:head# 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fe9a03a:::benchmark_data_smithi194_121512_object131536:head# 2024-04-29T14:27:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fea62f7:::benchmark_data_smithi194_121512_object106075:head# 2024-04-29T14:27:13.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6feafe19:::benchmark_data_smithi194_121512_object126270:head# 2024-04-29T14:27:13.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.812 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6feb1f8c:::benchmark_data_smithi194_121512_object45218:head# 2024-04-29T14:27:13.812 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6feb9b14:::benchmark_data_smithi194_121512_object124220:head# 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fec19ac:::benchmark_data_smithi194_121512_object92953:head# 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fec68fb:::benchmark_data_smithi194_121512_object111318:head# 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fecbbbc:::benchmark_data_smithi194_121512_object21911:head# 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fed270e:::benchmark_data_smithi194_121512_object55243:head# 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fed7119:::benchmark_data_smithi194_121512_object29673:head# 2024-04-29T14:27:13.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fee249a:::benchmark_data_smithi194_121512_object124641:head# 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6feec8b7:::benchmark_data_smithi194_121512_object74118:head# 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fef1661:::benchmark_data_smithi194_121512_object111105:head# 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fef873b:::benchmark_data_smithi194_121512_object29127:head# 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fefc8df:::benchmark_data_smithi194_121512_object46076:head# 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff07b7d:::benchmark_data_smithi194_121512_object107565:head# 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff0a69f:::benchmark_data_smithi194_121512_object130023:head# 2024-04-29T14:27:13.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff14e7d:::benchmark_data_smithi194_121512_object120991:head# 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff1cf1a:::benchmark_data_smithi194_121512_object92297:head# 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff1dc8c:::benchmark_data_smithi194_121512_object102735:head# 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff22f1d:::benchmark_data_smithi194_121512_object122646:head# 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff289da:::benchmark_data_smithi194_121512_object3788:head# 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff34d15:::benchmark_data_smithi194_121512_object124735:head# 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff3ad98:::benchmark_data_smithi194_121512_object130206:head# 2024-04-29T14:27:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff3b505:::benchmark_data_smithi194_121512_object71164:head# 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff406a2:::benchmark_data_smithi194_121512_object118230:head# 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff517c6:::benchmark_data_smithi194_121512_object33628:head# 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff55a80:::benchmark_data_smithi194_121512_object111619:head# 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff58f88:::benchmark_data_smithi194_121512_object76008:head# 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff5fa43:::benchmark_data_smithi194_121512_object14149:head# 2024-04-29T14:27:13.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff6d387:::benchmark_data_smithi194_121512_object69109:head# 2024-04-29T14:27:13.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff789eb:::benchmark_data_smithi194_121512_object31827:head# 2024-04-29T14:27:13.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff7b840:::benchmark_data_smithi194_121512_object1377:head# 2024-04-29T14:27:13.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff83d1b:::benchmark_data_smithi194_121512_object108410:head# 2024-04-29T14:27:13.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff959b7:::benchmark_data_smithi194_121512_object47145:head# 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ff9c679:::benchmark_data_smithi194_121512_object61046:head# 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ffa2784:::benchmark_data_smithi194_121512_object111311:head# 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ffa8552:::benchmark_data_smithi194_121512_object25623:head# 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ffae441:::benchmark_data_smithi194_121512_object2582:head# 2024-04-29T14:27:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ffb574e:::benchmark_data_smithi194_121512_object116625:head# 2024-04-29T14:27:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ffc061e:::benchmark_data_smithi194_121512_object74912:head# 2024-04-29T14:27:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6ffebf92:::benchmark_data_smithi194_121512_object32399:head# 2024-04-29T14:27:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #5:6fff3a0d:::benchmark_data_smithi194_121512_object114453:head# 2024-04-29T14:27:13.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-04-29T14:27:13.832 INFO:teuthology.orchestra.run.smithi121.stdout:write_pg epoch 457 info 5.6( v 457'8382 (0'0,457'8382] local-lis/les=432/433 n=8382 ec=432/432 lis/c=432/432 les/c/f=433/433/0 sis=432) 2024-04-29T14:27:13.893 INFO:teuthology.orchestra.run.smithi121.stdout:Import successful 2024-04-29T14:27:14.024 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:14.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:27:14.150 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:14.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:14.262 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:14.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:14 smithi194 ceph-mon[44089]: pgmap v1276: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 592 op/s; 32048/204514 objects degraded (15.670%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:27:14.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:14 smithi040 ceph-mon[39581]: pgmap v1276: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 592 op/s; 32048/204514 objects degraded (15.670%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:27:14.376 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:14.536 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:14 smithi121 ceph-mon[44533]: pgmap v1276: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 592 op/s; 32048/204514 objects degraded (15.670%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:27:14.553 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:14.578 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:14.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:14.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:14.732 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:14.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:27:15.412 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:27:15.412 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:27:15.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/exp.5.6.5 2024-04-29T14:27:15.696 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-29T14:27:15.696 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:27:15.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:27:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:15.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:15.859 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:15.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:16.110 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:16.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:16.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:27:16.217 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:16.271 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:16.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:16.308 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:16.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:16.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:27:16.442 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:27:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:27:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:16.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:27:16.797 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:16.845 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:16.874 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:16.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:16.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:17.012 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:27:17.044 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:17.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:17.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:17 smithi121 ceph-mon[44533]: pgmap v1277: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 31677/201950 objects degraded (15.686%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:27:17.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3978434638' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:17.150 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:17.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:17 smithi194 ceph-mon[44089]: pgmap v1277: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 31677/201950 objects degraded (15.686%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:27:17.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3978434638' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:17.206 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:17.250 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:17.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:17.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:17.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:17 smithi040 ceph-mon[39581]: pgmap v1277: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 31677/201950 objects degraded (15.686%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:27:17.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3978434638' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:17.392 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:17.417 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:17.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:17.522 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:17.553 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:18.253 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:18 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:27:18.253 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3978434638' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:18.253 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:18 smithi121 ceph-mon[44533]: osdmap e483: 12 total, 11 up, 7 in 2024-04-29T14:27:18.253 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:18 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 31677/201950 objects degraded (15.686%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:18.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:18 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:27:18.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3978434638' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:18.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:18 smithi194 ceph-mon[44089]: osdmap e483: 12 total, 11 up, 7 in 2024-04-29T14:27:18.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:18 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 31677/201950 objects degraded (15.686%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:18 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:27:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3978434638' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:18 smithi040 ceph-mon[39581]: osdmap e483: 12 total, 11 up, 7 in 2024-04-29T14:27:18.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:18 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 31677/201950 objects degraded (15.686%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:18.431 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:18.568 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:27:18.568 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:27:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:18.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:18.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:18.841 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:19.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:19 smithi194 ceph-mon[44089]: pgmap v1279: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 31313/199718 objects degraded (15.679%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:27:19.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:19 smithi121 ceph-mon[44533]: pgmap v1279: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 31313/199718 objects degraded (15.679%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:27:19.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:19 smithi040 ceph-mon[39581]: pgmap v1279: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 31313/199718 objects degraded (15.679%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:27:19.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:27:19.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:27:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:27:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:27:19.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:27:20.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1282521716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.311 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:20.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:27:20.336 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:27:20.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:20.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:20.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1282521716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:20.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:27:20.445 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:20.483 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:27:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:20.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1282521716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:20.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:20.640 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:20.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:20.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:20.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:27:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:20.844 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:27:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:20.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:27:20.959 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:21.003 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:21.034 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:21.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:21.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:21.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:21.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:21 smithi121 ceph-mon[44533]: pgmap v1280: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 31133/198536 objects degraded (15.681%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:21.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:21 smithi121 ceph-mon[44533]: osdmap e484: 12 total, 11 up, 7 in 2024-04-29T14:27:21.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:21.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:21.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:21.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:21.179 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:21.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:21.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:21.315 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:21 smithi194 ceph-mon[44089]: pgmap v1280: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 31133/198536 objects degraded (15.681%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:21 smithi194 ceph-mon[44089]: osdmap e484: 12 total, 11 up, 7 in 2024-04-29T14:27:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:21.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:21.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:21 smithi040 ceph-mon[39581]: pgmap v1280: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 6.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 31133/198536 objects degraded (15.681%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:21.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:21 smithi040 ceph-mon[39581]: osdmap e484: 12 total, 11 up, 7 in 2024-04-29T14:27:21.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:21.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:21.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:21.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:21.368 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:21.410 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:21.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:21.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:21.555 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:21.603 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:21.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:21.717 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:21.801 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:21.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:21.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:21.948 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:22.337 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:27:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:22 smithi040 ceph-mon[39581]: osdmap e485: 12 total, 11 up, 7 in 2024-04-29T14:27:22.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:22 smithi121 ceph-mon[44533]: osdmap e485: 12 total, 11 up, 7 in 2024-04-29T14:27:22.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:22 smithi194 ceph-mon[44089]: osdmap e485: 12 total, 11 up, 7 in 2024-04-29T14:27:22.736 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:22.948 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:27:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:27:23.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:23 smithi040 ceph-mon[39581]: pgmap v1283: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 30373/193466 objects degraded (15.699%); 14 MiB/s, 217 objects/s recovering 2024-04-29T14:27:23.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:23 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 30373/193466 objects degraded (15.699%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:23 smithi121 ceph-mon[44533]: pgmap v1283: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 30373/193466 objects degraded (15.699%); 14 MiB/s, 217 objects/s recovering 2024-04-29T14:27:23.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:23 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 30373/193466 objects degraded (15.699%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:23.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:23 smithi194 ceph-mon[44089]: pgmap v1283: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 30373/193466 objects degraded (15.699%); 14 MiB/s, 217 objects/s recovering 2024-04-29T14:27:23.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:23 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 30373/193466 objects degraded (15.699%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:23.841 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:23.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:23.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:24.037 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:24.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:24.096 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:24 smithi121 ceph-mon[44533]: from='client.36098 v1:172.21.15.40:0/2684978207' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:24.097 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:24.097 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:24.097 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1205744346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:24.098 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:27:24.146 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:24.209 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:24.249 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:24.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:24.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:24.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:24 smithi040 ceph-mon[39581]: from='client.36098 v1:172.21.15.40:0/2684978207' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:24.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:24.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:24.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1205744346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:24.433 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1283,"stamp":"2024-04-29T14:27:21.978807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6339887672,"num_objects":96733,"num_object_clones":0,"num_object_copies":193466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96733,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":170378,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11576,"num_bytes_recovered":760236160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160951,"ondisk_log_size":160951,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13859096,"kb_used_data":10457424,"kb_used_omap":10,"kb_used_meta":3401525,"kb_avail":642356968,"statfs":{"total":671965249536,"available":657773535232,"internally_reserved":0,"allocated":10708402176,"data_stored":10705595312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3483161694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-278003712,"num_objects":-4242,"num_object_clones":0,"num_object_copies":-8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4242,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":85458944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001014"},"pg_stats":[{"pgid":"5.f","version":"485'10900","reported_seq":15952,"reported_epoch":485,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:21.393988+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:27:21.393988+0000","last_peered":"2024-04-29T14:27:21.393988+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:27:21.393988+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:27:21.393988+0000","mapping_epoch":467,"log_start":"439'800","ondisk_log_start":"439'800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397410304,"num_objects":6064,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10900,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2423,"num_bytes_recovered":158793728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":6059},{"shards":"4,11","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"484'10731","reported_seq":10923,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-29T14:27:20.441194+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:27:20.441194+0000","last_peered":"2024-04-29T14:27:20.441194+0000","last_clean":"2024-04-29T14:27:20.441194+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:27:20.441194+0000","last_undegraded":"2024-04-29T14:27:20.441194+0000","last_fullsized":"2024-04-29T14:27:20.441194+0000","mapping_epoch":432,"log_start":"439'700","ondisk_log_start":"439'700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397344768,"num_objects":6063,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10731,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"484'10652","reported_seq":15458,"reported_epoch":484,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:27:20.974789+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:27:20.974789+0000","last_peered":"2024-04-29T14:27:20.974789+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:27:20.974789+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:27:20.974789+0000","mapping_epoch":468,"log_start":"439'600","ondisk_log_start":"439'600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395903000,"num_objects":6042,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6042,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10652,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2306,"num_bytes_recovered":151126016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":2},{"shards":"11","objects":6040}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"485'10749","reported_seq":10956,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-04-29T14:27:21.398333+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:27:21.398333+0000","last_peered":"2024-04-29T14:27:21.398333+0000","last_clean":"2024-04-29T14:27:21.398333+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:27:21.398333+0000","last_undegraded":"2024-04-29T14:27:21.398333+0000","last_fullsized":"2024-04-29T14:27:21.398333+0000","mapping_epoch":432,"log_start":"439'700","ondisk_log_start":"439'700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396034048,"num_objects":6043,"num_object_clones":0,"num_object_copies":12086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10749,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"484'10658","reported_seq":10857,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-29T14:27:20.533014+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:27:20.533014+0000","last_peered":"2024-04-29T14:27:20.533014+0000","last_clean":"2024-04-29T14:27:20.533014+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:27:20.533014+0000","last_undegraded":"2024-04-29T14:27:20.533014+0000","last_fullsized":"2024-04-29T14:27:20.533014+0000","mapping_epoch":432,"log_start":"439'600","ondisk_log_start":"439'600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393740288,"num_objects":6008,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10658,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"483'12750","reported_seq":13077,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-04-29T14:27:18.301654+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:27:18.301654+0000","last_peered":"2024-04-29T14:27:18.301654+0000","last_clean":"2024-04-29T14:27:18.301654+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:27:18.301654+0000","last_undegraded":"2024-04-29T14:27:18.301654+0000","last_fullsized":"2024-04-29T14:27:18.301654+0000","mapping_epoch":432,"log_start":"443'2700","ondisk_log_start":"443'2700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:33.665333+0000","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396558336,"num_objects":6051,"num_object_clones":0,"num_object_copies":12102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10495,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"485'10638","reported_seq":10841,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-04-29T14:27:21.418162+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:27:21.418162+0000","last_peered":"2024-04-29T14:27:21.418162+0000","last_clean":"2024-04-29T14:27:21.418162+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:27:21.418162+0000","last_undegraded":"2024-04-29T14:27:21.418162+0000","last_fullsized":"2024-04-29T14:27:21.418162+0000","mapping_epoch":432,"log_start":"439'600","ondisk_log_start":"439'600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":11888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"484'10601","reported_seq":10796,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-29T14:27:20.976935+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:27:20.976935+0000","last_peered":"2024-04-29T14:27:20.976935+0000","last_clean":"2024-04-29T14:27:20.976935+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:27:20.976935+0000","last_undegraded":"2024-04-29T14:27:20.976935+0000","last_fullsized":"2024-04-29T14:27:20.976935+0000","mapping_epoch":432,"log_start":"439'600","ondisk_log_start":"439'600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393674752,"num_objects":6007,"num_object_clones":0,"num_object_copies":12014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"484'10704","reported_seq":15515,"reported_epoch":484,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:20.560913+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:27:20.560913+0000","last_peered":"2024-04-29T14:27:20.560913+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:27:20.560913+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:27:20.560913+0000","mapping_epoch":467,"log_start":"439'700","ondisk_log_start":"439'700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399376384,"num_objects":6094,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10704,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2307,"num_bytes_recovered":151191552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":6092},{"shards":"9,10","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"485'10736","reported_seq":15657,"reported_epoch":485,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:21.486213+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:27:21.486213+0000","last_peered":"2024-04-29T14:27:21.486213+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:27:21.486213+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:27:21.486213+0000","mapping_epoch":467,"log_start":"439'700","ondisk_log_start":"439'700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395182080,"num_objects":6030,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10736,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2358,"num_bytes_recovered":154533888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":6008},{"shards":"7,10","objects":22}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"485'10690","reported_seq":10893,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-04-29T14:27:21.407651+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:27:21.407651+0000","last_peered":"2024-04-29T14:27:21.407651+0000","last_clean":"2024-04-29T14:27:21.407651+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:27:21.407651+0000","last_undegraded":"2024-04-29T14:27:21.407651+0000","last_fullsized":"2024-04-29T14:27:21.407651+0000","mapping_epoch":432,"log_start":"439'600","ondisk_log_start":"439'600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397803520,"num_objects":6070,"num_object_clones":0,"num_object_copies":12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10690,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"484'10573","reported_seq":10818,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-29T14:27:20.306396+0000","last_change":"2024-04-29T14:26:17.483610+0000","last_active":"2024-04-29T14:27:20.306396+0000","last_peered":"2024-04-29T14:27:20.306396+0000","last_clean":"2024-04-29T14:27:20.306396+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:27:20.306396+0000","last_undegraded":"2024-04-29T14:27:20.306396+0000","last_fullsized":"2024-04-29T14:27:20.306396+0000","mapping_epoch":432,"log_start":"439'500","ondisk_log_start":"439'500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'8320","last_scrub_stamp":"2024-04-29T14:26:17.483424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:17.483424+0000","objects_scrubbed":8299,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:45.122376+0000","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395640832,"num_objects":6037,"num_object_clones":0,"num_object_copies":12074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10573,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"484'10584","reported_seq":10788,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-29T14:27:20.306432+0000","last_change":"2024-04-29T14:26:19.678585+0000","last_active":"2024-04-29T14:27:20.306432+0000","last_peered":"2024-04-29T14:27:20.306432+0000","last_clean":"2024-04-29T14:27:20.306432+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:27:20.306432+0000","last_undegraded":"2024-04-29T14:27:20.306432+0000","last_fullsized":"2024-04-29T14:27:20.306432+0000","mapping_epoch":432,"log_start":"439'500","ondisk_log_start":"439'500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'8397","last_scrub_stamp":"2024-04-29T14:26:19.678439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:19.678439+0000","objects_scrubbed":8210,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:30.039863+0000","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393609216,"num_objects":6006,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"485'10471","reported_seq":10677,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-04-29T14:27:21.408489+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:27:21.408489+0000","last_peered":"2024-04-29T14:27:21.408489+0000","last_clean":"2024-04-29T14:27:21.408489+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:27:21.408489+0000","last_undegraded":"2024-04-29T14:27:21.408489+0000","last_fullsized":"2024-04-29T14:27:21.408489+0000","mapping_epoch":432,"log_start":"439'400","ondisk_log_start":"439'400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391577600,"num_objects":5975,"num_object_clones":0,"num_object_copies":11950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10471,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"483'13458","reported_seq":13832,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-04-29T14:27:18.275429+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:27:18.275429+0000","last_peered":"2024-04-29T14:27:18.275429+0000","last_clean":"2024-04-29T14:27:18.275429+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:27:18.275429+0000","last_undegraded":"2024-04-29T14:27:18.275429+0000","last_fullsized":"2024-04-29T14:27:18.275429+0000","mapping_epoch":432,"log_start":"444'3400","ondisk_log_start":"444'3400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401080320,"num_objects":6120,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10496,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"483'10519","reported_seq":14985,"reported_epoch":483,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:18.307698+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:27:18.307698+0000","last_peered":"2024-04-29T14:27:18.307698+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:27:18.307698+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:27:18.307698+0000","mapping_epoch":467,"log_start":"439'500","ondisk_log_start":"439'500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404815872,"num_objects":6177,"num_object_clones":0,"num_object_copies":12354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10519,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":3},{"shards":"10","objects":6174}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1212,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-04-29T14:27:21.093068+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:27:21.093068+0000","last_peered":"2024-04-29T14:27:21.093068+0000","last_clean":"2024-04-29T14:27:21.093068+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:27:21.093068+0000","last_undegraded":"2024-04-29T14:27:21.093068+0000","last_fullsized":"2024-04-29T14:27:21.093068+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6339297304,"num_objects":96731,"num_object_clones":0,"num_object_copies":193462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96731,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170197,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11568,"num_bytes_recovered":758120448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13420404736,"data_stored":13420396592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160814,"ondisk_log_size":160814,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122293,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1291272,"kb_used_data":776956,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92453880,"statfs":{"total":95995035648,"available":94672773120,"internally_reserved":0,"allocated":795602944,"data_stored":795194722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":526645700},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383929,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558576,"kb_used_data":1180324,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92186576,"statfs":{"total":95995035648,"available":94399053824,"internally_reserved":0,"allocated":1208651776,"data_stored":1208257413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645567,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557436,"kb_used_data":1165296,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92187716,"statfs":{"total":95995035648,"available":94400221184,"internally_reserved":0,"allocated":1193263104,"data_stored":1192862526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618888,"kb_used_data":1154556,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92126264,"statfs":{"total":95995035648,"available":94337294336,"internally_reserved":0,"allocated":1182265344,"data_stored":1181857142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475462090},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528589,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1715628,"kb_used_data":1160996,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92029524,"statfs":{"total":95995035648,"available":94238232576,"internally_reserved":0,"allocated":1188859904,"data_stored":1188465561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575142,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3125176,"kb_used_data":2702512,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":90619976,"statfs":{"total":95995035648,"available":92794855424,"internally_reserved":0,"allocated":2767372288,"data_stored":2766971710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":432798149},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2992120,"kb_used_data":2316784,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":90753032,"statfs":{"total":95995035648,"available":92931104768,"internally_reserved":0,"allocated":2372386816,"data_stored":2371986238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2764439552,"data_stored":2764439552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2364669952,"data_stored":2364669952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1181548544,"data_stored":1181548544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1176113152,"data_stored":1176109080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1190461440,"data_stored":1190461440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1203634176,"data_stored":1203634176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":789843968,"data_stored":789839896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:27:24.434 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:27:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:24 smithi194 ceph-mon[44089]: from='client.36098 v1:172.21.15.40:0/2684978207' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1205744346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:24.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:24.751 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:24.820 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:24.936 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285494 4.a 4.c1e0645a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.171799+0000", 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.755623525, 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.99503064399999996, 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285494 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171799+0000", 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171799+0000", 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171800+0000", 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4899999999999999e-06 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171806+0000", 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1430000000000003e-06 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171818+0000", 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1541e-05 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171826+0000", 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.507e-06 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171891+0000", 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4344999999999995e-05 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171903+0000", 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2147000000000001e-05 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166793+0000", 2024-04-29T14:27:24.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.99488997099999998 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166829+0000", 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-05 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285535 4.a 4.823c013a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.208797+0000", 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.718625205, 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95811517000000002, 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285535 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208797+0000", 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208797+0000", 2024-04-29T14:27:24.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208801+0000", 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4039999999999997e-06 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208810+0000", 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4889999999999995e-06 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208827+0000", 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7631000000000001e-05 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208838+0000", 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208911+0000", 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3070000000000003e-05 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208925+0000", 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3434e-05 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166904+0000", 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95797888200000003 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166912+0000", 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4530000000000002e-06 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285539 4.a 4.3cb842da (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.214823+0000", 2024-04-29T14:27:24.945 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.71259874399999, 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95213097099999999, 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285539 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214823+0000", 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214823+0000", 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214825+0000", 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.883e-06 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214832+0000", 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8469999999999998e-06 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214846+0000", 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3531e-05 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214852+0000", 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.533e-06 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214910+0000", 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8216e-05 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214924+0000", 2024-04-29T14:27:24.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3303e-05 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166948+0000", 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95202394899999998 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166954+0000", 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7089999999999996e-06 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285543 4.a 4.2fcae8fa (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.949 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.217886+0000", 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.70953569700001, 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94910000999999999, 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285543 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217886+0000", 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217886+0000", 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217888+0000", 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4479999999999999e-06 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217892+0000", 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5709999999999999e-06 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217902+0000", 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4809999999999995e-06 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217909+0000", 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8460000000000002e-06 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217963+0000", 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4664999999999999e-05 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217974+0000", 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0424e-05 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166980+0000", 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94900661399999997 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166986+0000", 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9610000000000002e-06 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.953 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285544 4.a 4.151257ca (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.709434483, 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94902677599999996, 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285544 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2399999999999997e-07 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217992+0000", 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5470000000000001e-06 2024-04-29T14:27:24.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217999+0000", 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2030000000000003e-06 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.218005+0000", 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6720000000000001e-06 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.218049+0000", 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4567999999999997e-05 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.218058+0000", 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4659999999999999e-06 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167009+0000", 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94895064699999998 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167014+0000", 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8490000000000002e-06 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285562 4.a 4.dc1a1a6a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.239841+0000", 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.68758086, 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92720931200000001, 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285562 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239841+0000", 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239841+0000", 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239843+0000", 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239849+0000", 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4829999999999999e-06 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239860+0000", 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0839000000000001e-05 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239867+0000", 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6979999999999998e-06 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.244829+0000", 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049617280000000003 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.244844+0000", 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5390999999999999e-05 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167044+0000", 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92219957900000005 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167051+0000", 2024-04-29T14:27:24.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5010000000000001e-06 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285587 4.a 4.a38f394a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.260019+0000", 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.667403339, 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.907057318, 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285587 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260019+0000", 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260019+0000", 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260020+0000", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.488e-06 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260029+0000", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2919999999999996e-06 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260037+0000", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5210000000000002e-06 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260044+0000", 2024-04-29T14:27:24.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4309999999999997e-06 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260104+0000", 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9293999999999998e-05 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260114+0000", 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.013e-05 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167073+0000", 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90695879700000004 2024-04-29T14:27:24.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167076+0000", 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.365e-06 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285588 4.a 4.2f09c4a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.263570+0000", 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.66385254799999, 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90352384399999996, 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285588 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263570+0000", 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263570+0000", 2024-04-29T14:27:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263571+0000", 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9269999999999999e-06 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263577+0000", 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8200000000000002e-06 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263587+0000", 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0164e-05 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263597+0000", 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.004e-05 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263656+0000", 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8779000000000001e-05 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263668+0000", 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2037e-05 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167090+0000", 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90342208400000001 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167093+0000", 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.993e-06 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285589 4.a 4.ec2d9ada (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.66380252299999, 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90349196200000004, 2024-04-29T14:27:24.970 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285589 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:27:24.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4099999999999998e-07 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263623+0000", 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.162e-06 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263630+0000", 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3709999999999996e-06 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263634+0000", 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8489999999999999e-06 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263692+0000", 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8046e-05 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263698+0000", 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7150000000000001e-06 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167108+0000", 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90341010799999999 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167112+0000", 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0699999999999998e-06 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285597 4.a 4.66c85c1a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.270706+0000", 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.656715945, 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.896431804, 2024-04-29T14:27:24.974 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285597 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270706+0000", 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270706+0000", 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270708+0000", 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.68e-06 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270714+0000", 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0140000000000004e-06 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270726+0000", 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1968e-05 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270734+0000", 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0209999999999999e-06 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270807+0000", 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2979999999999996e-05 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270818+0000", 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1398e-05 2024-04-29T14:27:24.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167132+0000", 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.89631423300000002 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167138+0000", 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5099999999999998e-06 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.978 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285610 4.a 4.5af2b80a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.285975+0000", 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.641446992, 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.88120892399999995, 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285610 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285975+0000", 2024-04-29T14:27:24.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285975+0000", 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285977+0000", 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.59e-06 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285982+0000", 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0250000000000002e-06 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285992+0000", 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0117e-05 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285998+0000", 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.088e-06 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.286049+0000", 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0779999999999998e-05 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.286056+0000", 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2420000000000003e-06 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167180+0000", 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.88112453099999999 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167184+0000", 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.551e-06 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285635 4.a 4.fce1415a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.320869+0000", 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.606553217, 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.84633211799999997, 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285635 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320869+0000", 2024-04-29T14:27:24.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320869+0000", 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320871+0000", 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5289999999999998e-06 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320878+0000", 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8979999999999999e-06 2024-04-29T14:27:24.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320896+0000", 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7269999999999999e-05 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320905+0000", 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5300000000000002e-06 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320978+0000", 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3125000000000002e-05 2024-04-29T14:27:24.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320988+0000", 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167198+0000", 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.84621011199999996 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167201+0000", 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2600000000000001e-06 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.986 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285640 4.a 4.7f958c0a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.327392+0000", 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.600029619, 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.83982455300000003, 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285640 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327392+0000", 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327392+0000", 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327394+0000", 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.113e-06 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327398+0000", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.941e-06 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327404+0000", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8689999999999998e-06 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327411+0000", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5100000000000004e-06 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327463+0000", 2024-04-29T14:27:24.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2351000000000001e-05 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327472+0000", 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167214+0000", 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.83974157299999996 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167217+0000", 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.083e-06 2024-04-29T14:27:24.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285645 4.a 4.74d5244a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.337231+0000", 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.59019150699999, 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.83000898400000001, 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285645 2024-04-29T14:27:24.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337231+0000", 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337231+0000", 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337234+0000", 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337243+0000", 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1239999999999994e-06 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337260+0000", 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6954000000000001e-05 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337270+0000", 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0404999999999999e-05 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337335+0000", 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4577000000000002e-05 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337348+0000", 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3803e-05 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167234+0000", 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.829885769 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167240+0000", 2024-04-29T14:27:24.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3739999999999997e-06 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285655 4.a 4.60222c3a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.360089+0000", 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.567333526, 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80718411999999995, 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285655 2024-04-29T14:27:24.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360089+0000", 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360089+0000", 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360091+0000", 2024-04-29T14:27:24.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7240000000000001e-06 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360103+0000", 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1438e-05 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360121+0000", 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7929000000000001e-05 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360131+0000", 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0832000000000001e-05 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360213+0000", 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1219999999999998e-05 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360226+0000", 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2989999999999999e-05 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167266+0000", 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80704071700000002 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167273+0000", 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2700000000000001e-06 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285667 4.a 4.2c5a08aa (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.417619+0000", 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 225.50980269199999, 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.74968378899999999, 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:24.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285667 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417619+0000", 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417619+0000", 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:25.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417623+0000", 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4120000000000001e-06 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417632+0000", 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8070000000000006e-06 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417649+0000", 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7195999999999999e-05 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417659+0000", 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0685e-05 2024-04-29T14:27:25.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417733+0000", 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3819999999999995e-05 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417747+0000", 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3725e-05 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167299+0000", 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.74955182099999995 2024-04-29T14:27:25.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167303+0000", 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3229999999999999e-06 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285640 4.a 4:5031a9fe:::benchmark_data_smithi194_103039_object128957:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194370+0000", 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 223.73305182799999, 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26445676699999998, 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:27:25.003 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285640 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194370+0000", 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194370+0000", 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194371+0000", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3500000000000005e-07 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194374+0000", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.067e-06 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194380+0000", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4580000000000001e-06 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194383+0000", 2024-04-29T14:27:25.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3720000000000001e-06 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194601+0000", 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021826699999999999 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194608+0000", 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1509999999999998e-06 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.426687+0000", 2024-04-29T14:27:25.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23207839099999999 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.426766+0000", 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9264000000000005e-05 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.426848+0000", 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1884000000000004e-05 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.436426+0000", 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095781340000000003 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.436926+0000", 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049993599999999998 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458674+0000", 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021747836 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458727+0000", 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3257999999999997e-05 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458827+0000", 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9914000000000006e-05 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285645 4.a 4:5224ab2e:::benchmark_data_smithi194_103039_object128962:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 223.73303456900001, 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.264734882, 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:25.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285645 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3500000000000002e-07 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194389+0000", 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3999999999999999e-06 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194393+0000", 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6049999999999998e-06 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194398+0000", 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6709999999999998e-06 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194644+0000", 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024664799999999997 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194652+0000", 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9789999999999993e-06 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.431280+0000", 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.236627529 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.431411+0000", 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000130877 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.431538+0000", 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127627 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.436700+0000", 2024-04-29T14:27:25.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051618590000000004 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437199+0000", 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049851799999999999 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458963+0000", 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021764034000000002 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459020+0000", 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7380000000000003e-05 2024-04-29T14:27:25.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459122+0000", 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010242 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285655 4.a 4:5c344406:::benchmark_data_smithi194_103039_object128972:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194403+0000", 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 223.733018609, 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26493427200000003, 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:25.015 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285655 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194403+0000", 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194403+0000", 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194404+0000", 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2300000000000002e-07 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194405+0000", 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5710000000000001e-06 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194409+0000", 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4960000000000001e-06 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:25.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194412+0000", 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9699999999999999e-06 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194687+0000", 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027504299999999999 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194695+0000", 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1429999999999998e-06 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437242+0000", 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242547229 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437351+0000", 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108433 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437478+0000", 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012781199999999999 2024-04-29T14:27:25.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.450522+0000", 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013043232 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.450536+0000", 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.468e-05 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459173+0000", 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086369210000000005 2024-04-29T14:27:25.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459225+0000", 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1768999999999999e-05 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459338+0000", 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011265 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285667 4.a 4:55105a34:::benchmark_data_smithi194_103039_object128984:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194445+0000", 2024-04-29T14:27:25.021 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 223.73297696500001, 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26555013399999999, 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285667 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194445+0000", 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194445+0000", 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194446+0000", 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5000000000000001e-07 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194449+0000", 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7319999999999998e-06 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:27:25.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194453+0000", 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4669999999999998e-06 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194456+0000", 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194719+0000", 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026247299999999999 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194728+0000", 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9479999999999997e-06 2024-04-29T14:27:25.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.442078+0000", 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.247350505 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.442204+0000", 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012555300000000001 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.442341+0000", 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013681000000000001 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.450848+0000", 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085075080000000004 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458636+0000", 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077882380000000003 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459833+0000", 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011965109999999999 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459892+0000", 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8785e-05 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459995+0000", 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103527 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:27:25.027 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:27:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:25.028 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:25.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:25.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:25 smithi194 ceph-mon[44089]: pgmap v1284: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 30186/192348 objects degraded (15.693%); 10 MiB/s, 162 objects/s recovering 2024-04-29T14:27:25.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:25 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:25.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1205744346' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:27:25.129 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:25.167 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:25.200 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:25.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:25.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:25.336 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:25.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:25 smithi121 ceph-mon[44533]: pgmap v1284: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 30186/192348 objects degraded (15.693%); 10 MiB/s, 162 objects/s recovering 2024-04-29T14:27:25.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:25 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:25.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1205744346' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:27:25.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:25 smithi121 ceph-mon[44533]: osdmap e486: 12 total, 11 up, 7 in 2024-04-29T14:27:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:25 smithi040 ceph-mon[39581]: pgmap v1284: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 30186/192348 objects degraded (15.693%); 10 MiB/s, 162 objects/s recovering 2024-04-29T14:27:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:25 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1205744346' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:27:25.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:25 smithi040 ceph-mon[39581]: osdmap e486: 12 total, 11 up, 7 in 2024-04-29T14:27:25.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:25 smithi194 ceph-mon[44089]: osdmap e486: 12 total, 11 up, 7 in 2024-04-29T14:27:25.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:27:26.086 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:27:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:26.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:26.210 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:26.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:26.330 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:26.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:26.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:26.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:26.500 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:26.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:26.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:26.765 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:26.899 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:27:26.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:27:26.946 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:27:26.970 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:27:26.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:26.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:26.993 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:27:27.021 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:27:27.046 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:27:27.080 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:27:27.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:27.108 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:27:27.131 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:27:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:27:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:27:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:27:27.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:27 smithi040 ceph-mon[39581]: pgmap v1286: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 29986/190514 objects degraded (15.740%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:27.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4169514957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:27 smithi121 ceph-mon[44533]: pgmap v1286: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 29986/190514 objects degraded (15.740%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:27.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4169514957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:27.486 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:27 smithi194 ceph-mon[44089]: pgmap v1286: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 29986/190514 objects degraded (15.740%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:27.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4169514957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:27.492 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:27.517 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:27.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:27.623 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:27.656 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.105 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.106 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.106 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.106 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:28 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 29986/190514 objects degraded (15.740%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:28.111 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:28.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:27:28.120 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:27:28.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:27:28.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:28.281 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:28.323 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:28.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:28 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 29986/190514 objects degraded (15.740%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:28.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:27:28.429 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:27:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:28.556 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:28.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:28.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:28 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 29986/190514 objects degraded (15.740%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:28.663 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:28.699 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:28.724 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:28.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:28.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:27:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:28.880 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:27:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:28.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:27:28.984 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:29.031 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:29.070 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:29.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:29.182 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:29.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:27:29.271 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:29.309 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:29.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:29.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:29 smithi194 ceph-mon[44089]: pgmap v1287: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 694 op/s; 29399/187548 objects degraded (15.675%); 14 MiB/s, 219 objects/s recovering 2024-04-29T14:27:29.383 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:29 smithi040 ceph-mon[39581]: pgmap v1287: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 694 op/s; 29399/187548 objects degraded (15.675%); 14 MiB/s, 219 objects/s recovering 2024-04-29T14:27:29.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:29.453 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:29.598 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:27:29.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:29 smithi121 ceph-mon[44533]: pgmap v1287: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 694 op/s; 29399/187548 objects degraded (15.675%); 14 MiB/s, 219 objects/s recovering 2024-04-29T14:27:29.807 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:29.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:29.914 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:29.951 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:29.979 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:29.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:30.083 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:30.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:27:30.147 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:30.186 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:30.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:30.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:27:30.326 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:27:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:27:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:27:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:30.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:27:30.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:27:30.829 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:27:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:31.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:27:31.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:31.135 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:31.177 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:31.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:31.219 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1287,"stamp":"2024-04-29T14:27:27.979959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6145966648,"num_objects":93774,"num_object_clones":0,"num_object_copies":187548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93774,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":173337,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12550,"num_bytes_recovered":824068224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160910,"ondisk_log_size":160910,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13542676,"kb_used_data":10135492,"kb_used_omap":10,"kb_used_meta":3407093,"kb_avail":642673388,"statfs":{"total":671965249536,"available":658097549312,"internally_reserved":0,"allocated":10378743808,"data_stored":10375843769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3488863326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,11,8,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-360054784,"num_objects":-5494,"num_object_clones":0,"num_object_copies":-10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":113639424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.909314"},"pg_stats":[{"pgid":"5.f","version":"486'11094","reported_seq":16548,"reported_epoch":486,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:26.413307+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:27:26.413307+0000","last_peered":"2024-04-29T14:27:26.413307+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:27:26.413307+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:27:26.413307+0000","mapping_epoch":467,"log_start":"440'1000","ondisk_log_start":"440'1000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384696320,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2623,"num_bytes_recovered":171900928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":5859},{"shards":"4,11","objects":11}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"486'10917","reported_seq":11113,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:25.529566+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:27:25.529566+0000","last_peered":"2024-04-29T14:27:25.529566+0000","last_clean":"2024-04-29T14:27:25.529566+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:27:25.529566+0000","last_undegraded":"2024-04-29T14:27:25.529566+0000","last_fullsized":"2024-04-29T14:27:25.529566+0000","mapping_epoch":432,"log_start":"439'900","ondisk_log_start":"439'900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385155072,"num_objects":5877,"num_object_clones":0,"num_object_copies":11754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10917,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"486'10838","reported_seq":16048,"reported_epoch":486,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:27:25.993603+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:27:25.993603+0000","last_peered":"2024-04-29T14:27:25.993603+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:27:25.993603+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:27:25.993603+0000","mapping_epoch":468,"log_start":"439'800","ondisk_log_start":"439'800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383713304,"num_objects":5856,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5841,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10838,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2505,"num_bytes_recovered":164167680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":15},{"shards":"11","objects":5841}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"486'10939","reported_seq":11148,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:26.369980+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:27:26.369980+0000","last_peered":"2024-04-29T14:27:26.369980+0000","last_clean":"2024-04-29T14:27:26.369980+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:27:26.369980+0000","last_undegraded":"2024-04-29T14:27:26.369980+0000","last_fullsized":"2024-04-29T14:27:26.369980+0000","mapping_epoch":432,"log_start":"439'900","ondisk_log_start":"439'900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383582208,"num_objects":5853,"num_object_clones":0,"num_object_copies":11706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"486'10837","reported_seq":11040,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:25.476283+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:27:25.476283+0000","last_peered":"2024-04-29T14:27:25.476283+0000","last_clean":"2024-04-29T14:27:25.476283+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:27:25.476283+0000","last_undegraded":"2024-04-29T14:27:25.476283+0000","last_fullsized":"2024-04-29T14:27:25.476283+0000","mapping_epoch":432,"log_start":"439'800","ondisk_log_start":"439'800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382009344,"num_objects":5829,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10837,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"485'12943","reported_seq":13274,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-04-29T14:27:23.342012+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:27:23.342012+0000","last_peered":"2024-04-29T14:27:23.342012+0000","last_clean":"2024-04-29T14:27:23.342012+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:27:23.342012+0000","last_undegraded":"2024-04-29T14:27:23.342012+0000","last_fullsized":"2024-04-29T14:27:23.342012+0000","mapping_epoch":432,"log_start":"443'2900","ondisk_log_start":"443'2900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:33.665333+0000","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383909888,"num_objects":5858,"num_object_clones":0,"num_object_copies":11716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"486'10819","reported_seq":11024,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:26.455811+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:27:26.455811+0000","last_peered":"2024-04-29T14:27:26.455811+0000","last_clean":"2024-04-29T14:27:26.455811+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:27:26.455811+0000","last_undegraded":"2024-04-29T14:27:26.455811+0000","last_fullsized":"2024-04-29T14:27:26.455811+0000","mapping_epoch":432,"log_start":"439'800","ondisk_log_start":"439'800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377683968,"num_objects":5763,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10819,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"486'10771","reported_seq":10971,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:25.956546+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:27:25.956546+0000","last_peered":"2024-04-29T14:27:25.956546+0000","last_clean":"2024-04-29T14:27:25.956546+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:27:25.956546+0000","last_undegraded":"2024-04-29T14:27:25.956546+0000","last_fullsized":"2024-04-29T14:27:25.956546+0000","mapping_epoch":432,"log_start":"439'700","ondisk_log_start":"439'700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382533632,"num_objects":5837,"num_object_clones":0,"num_object_copies":11674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"486'10905","reported_seq":16120,"reported_epoch":486,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:25.592156+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:27:25.592156+0000","last_peered":"2024-04-29T14:27:25.592156+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:27:25.592156+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:27:25.592156+0000","mapping_epoch":467,"log_start":"439'900","ondisk_log_start":"439'900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386203648,"num_objects":5893,"num_object_clones":0,"num_object_copies":11786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2507,"num_bytes_recovered":164298752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":5892},{"shards":"9,10","objects":1}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"486'10926","reported_seq":16225,"reported_epoch":486,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:26.517496+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:27:26.517496+0000","last_peered":"2024-04-29T14:27:26.517496+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:27:26.517496+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:27:26.517496+0000","mapping_epoch":467,"log_start":"440'900","ondisk_log_start":"440'900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382730240,"num_objects":5840,"num_object_clones":0,"num_object_copies":11680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10926,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2546,"num_bytes_recovered":166854656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":5820},{"shards":"7,10","objects":20}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"486'10897","reported_seq":11102,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:26.386191+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:27:26.386191+0000","last_peered":"2024-04-29T14:27:26.386191+0000","last_clean":"2024-04-29T14:27:26.386191+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:27:26.386191+0000","last_undegraded":"2024-04-29T14:27:26.386191+0000","last_fullsized":"2024-04-29T14:27:26.386191+0000","mapping_epoch":432,"log_start":"439'800","ondisk_log_start":"439'800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384237568,"num_objects":5863,"num_object_clones":0,"num_object_copies":11726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10897,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"486'10757","reported_seq":11006,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:25.281887+0000","last_change":"2024-04-29T14:26:17.483610+0000","last_active":"2024-04-29T14:27:25.281887+0000","last_peered":"2024-04-29T14:27:25.281887+0000","last_clean":"2024-04-29T14:27:25.281887+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:27:25.281887+0000","last_undegraded":"2024-04-29T14:27:25.281887+0000","last_fullsized":"2024-04-29T14:27:25.281887+0000","mapping_epoch":432,"log_start":"439'700","ondisk_log_start":"439'700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'8320","last_scrub_stamp":"2024-04-29T14:26:17.483424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:17.483424+0000","objects_scrubbed":8299,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:45.122376+0000","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383582208,"num_objects":5853,"num_object_clones":0,"num_object_copies":11706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10757,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"486'10751","reported_seq":10959,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:25.226843+0000","last_change":"2024-04-29T14:26:19.678585+0000","last_active":"2024-04-29T14:27:25.226843+0000","last_peered":"2024-04-29T14:27:25.226843+0000","last_clean":"2024-04-29T14:27:25.226843+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:27:25.226843+0000","last_undegraded":"2024-04-29T14:27:25.226843+0000","last_fullsized":"2024-04-29T14:27:25.226843+0000","mapping_epoch":432,"log_start":"439'700","ondisk_log_start":"439'700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'8397","last_scrub_stamp":"2024-04-29T14:26:19.678439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:19.678439+0000","objects_scrubbed":8210,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:30.039863+0000","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382664704,"num_objects":5839,"num_object_clones":0,"num_object_copies":11678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10751,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"486'10636","reported_seq":10844,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:26.271003+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:27:26.271003+0000","last_peered":"2024-04-29T14:27:26.271003+0000","last_clean":"2024-04-29T14:27:26.271003+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:27:26.271003+0000","last_undegraded":"2024-04-29T14:27:26.271003+0000","last_fullsized":"2024-04-29T14:27:26.271003+0000","mapping_epoch":432,"log_start":"439'600","ondisk_log_start":"439'600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380764160,"num_objects":5810,"num_object_clones":0,"num_object_copies":11620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10636,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"485'13651","reported_seq":14029,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-04-29T14:27:23.299211+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:27:23.299211+0000","last_peered":"2024-04-29T14:27:23.299211+0000","last_clean":"2024-04-29T14:27:23.299211+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:27:23.299211+0000","last_undegraded":"2024-04-29T14:27:23.299211+0000","last_fullsized":"2024-04-29T14:27:23.299211+0000","mapping_epoch":432,"log_start":"444'3600","ondisk_log_start":"444'3600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388431872,"num_objects":5927,"num_object_clones":0,"num_object_copies":11854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"485'10692","reported_seq":15536,"reported_epoch":485,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:23.336998+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:27:23.336998+0000","last_peered":"2024-04-29T14:27:23.336998+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:27:23.336998+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:27:23.336998+0000","mapping_epoch":467,"log_start":"439'600","ondisk_log_start":"439'600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393478144,"num_objects":6004,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2361,"num_bytes_recovered":154730496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":17},{"shards":"10","objects":5987}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1214,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:24.112429+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:27:24.112429+0000","last_peered":"2024-04-29T14:27:24.112429+0000","last_clean":"2024-04-29T14:27:24.112429+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:27:24.112429+0000","last_undegraded":"2024-04-29T14:27:24.112429+0000","last_fullsized":"2024-04-29T14:27:24.112429+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6145376280,"num_objects":93772,"num_object_clones":0,"num_object_copies":187544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93772,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173156,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12542,"num_bytes_recovered":821952512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13093642240,"data_stored":13093634096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160773,"ondisk_log_size":160773,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267872,"kb_used_data":753556,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92477280,"statfs":{"total":95995035648,"available":94696734720,"internally_reserved":0,"allocated":771641344,"data_stored":771220702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":526645700},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383930,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1531656,"kb_used_data":1145084,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":92213496,"statfs":{"total":95995035648,"available":94426619904,"internally_reserved":0,"allocated":1172566016,"data_stored":1172159233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":395835840},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645568,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521172,"kb_used_data":1129032,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92223980,"statfs":{"total":95995035648,"available":94437355520,"internally_reserved":0,"allocated":1156128768,"data_stored":1155715770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234249,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1607580,"kb_used_data":1121044,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92137572,"statfs":{"total":95995035648,"available":94348873728,"internally_reserved":0,"allocated":1147949056,"data_stored":1147528434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680340,"kb_used_data":1125768,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92064812,"statfs":{"total":95995035648,"available":94274367488,"internally_reserved":0,"allocated":1152786432,"data_stored":1152373434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3028756,"kb_used_data":2618568,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":90716396,"statfs":{"total":95995035648,"available":92893589504,"internally_reserved":0,"allocated":2681413632,"data_stored":2681000634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497101,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2905300,"kb_used_data":2242440,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":90839852,"statfs":{"total":95995035648,"available":93020008448,"internally_reserved":0,"allocated":2296258560,"data_stored":2295845562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2677604352,"data_stored":2677604352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2291007488,"data_stored":2291007488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1146683392,"data_stored":1146683392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1141641216,"data_stored":1141637144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1153171456,"data_stored":1153171456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1167654912,"data_stored":1167654912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":766185472,"data_stored":766181400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:27:31.220 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:27:31.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:31.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:31 smithi121 ceph-mon[44533]: from='client.16506 v1:172.21.15.40:0/4129320637' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:31.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:31 smithi121 ceph-mon[44533]: pgmap v1288: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 530 op/s; 29207/186360 objects degraded (15.672%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:27:31.332 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:31.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:31.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:31 smithi040 ceph-mon[39581]: from='client.16506 v1:172.21.15.40:0/4129320637' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:31.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:31 smithi040 ceph-mon[39581]: pgmap v1288: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 530 op/s; 29207/186360 objects degraded (15.672%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:27:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:31.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:31.541 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:31 smithi194 ceph-mon[44089]: from='client.16506 v1:172.21.15.40:0/4129320637' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:31 smithi194 ceph-mon[44089]: pgmap v1288: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 530 op/s; 29207/186360 objects degraded (15.672%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:27:31.606 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:31.660 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:31.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:31.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:27:31.782 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:32.002 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:27:32.002 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:27:32.071 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:27:32.072 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:32.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:27:32.177 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:27:32.234 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:32.278 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:32.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:32.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2652151206' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:32.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:32 smithi040 ceph-mon[39581]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:32.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:32 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 29207/186360 objects degraded (15.672%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2652151206' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:32 smithi121 ceph-mon[44533]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:32.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:32 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 29207/186360 objects degraded (15.672%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:32.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:27:32.439 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:32.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2652151206' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:32.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:32 smithi194 ceph-mon[44089]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:32.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:32 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 29207/186360 objects degraded (15.672%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:33.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:33 smithi040 ceph-mon[39581]: pgmap v1289: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 28453/181442 objects degraded (15.682%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:33.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:33 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:33.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:33 smithi040 ceph-mon[39581]: from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:33.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:33 smithi040 ceph-mon[39581]: osdmap e487: 12 total, 11 up, 7 in 2024-04-29T14:27:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:33 smithi194 ceph-mon[44089]: pgmap v1289: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 28453/181442 objects degraded (15.682%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:33 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:33 smithi194 ceph-mon[44089]: from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:33.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:33 smithi194 ceph-mon[44089]: osdmap e487: 12 total, 11 up, 7 in 2024-04-29T14:27:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:33 smithi121 ceph-mon[44533]: pgmap v1289: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 28453/181442 objects degraded (15.682%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:27:33.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:33 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:33.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:33 smithi121 ceph-mon[44533]: from='client.36116 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:33.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:33 smithi121 ceph-mon[44533]: osdmap e487: 12 total, 11 up, 7 in 2024-04-29T14:27:34.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:27:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:34.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:27:34.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:34.922 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:35.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:35 smithi194 ceph-mon[44089]: pgmap v1291: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 612 op/s; 28263/180254 objects degraded (15.680%); 12 MiB/s, 194 objects/s recovering 2024-04-29T14:27:35.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:35.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:35.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:35 smithi040 ceph-mon[39581]: pgmap v1291: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 612 op/s; 28263/180254 objects degraded (15.680%); 12 MiB/s, 194 objects/s recovering 2024-04-29T14:27:35.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:35.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:35 smithi121 ceph-mon[44533]: pgmap v1291: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 612 op/s; 28263/180254 objects degraded (15.680%); 12 MiB/s, 194 objects/s recovering 2024-04-29T14:27:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:35.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:36.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:36.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:36.151 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:36.171 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:36.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:36.218 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:27:36.239 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:27:36.258 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:27:36.278 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:27:36.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:36.311 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:27:36.314 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:36.336 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:27:36.357 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:27:36.384 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:27:36.407 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:27:36.416 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:36 smithi040 ceph-mon[39581]: osdmap e488: 12 total, 11 up, 7 in 2024-04-29T14:27:36.416 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:36.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:36.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3234994484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:36.417 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:36.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:36.431 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:27:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:27:36.522 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:36.555 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:36 smithi194 ceph-mon[44089]: osdmap e488: 12 total, 11 up, 7 in 2024-04-29T14:27:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:36.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3234994484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:36.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:36 smithi121 ceph-mon[44533]: osdmap e488: 12 total, 11 up, 7 in 2024-04-29T14:27:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3234994484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:36.911 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:36.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:36.954 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:27:36.956 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:27:37.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:37.047 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:37.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:27:37.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: pgmap v1293: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 584 op/s; 28066/178190 objects degraded (15.751%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:27:37.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osdmap e489: 12 total, 11 up, 7 in 2024-04-29T14:27:37.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 28066/178190 objects degraded (15.751%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:37.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:37 smithi040 ceph-mon[39581]: osdmap e490: 12 total, 11 up, 7 in 2024-04-29T14:27:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:27:37.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:27:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: pgmap v1293: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 584 op/s; 28066/178190 objects degraded (15.751%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:27:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osdmap e489: 12 total, 11 up, 7 in 2024-04-29T14:27:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 28066/178190 objects degraded (15.751%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:37 smithi194 ceph-mon[44089]: osdmap e490: 12 total, 11 up, 7 in 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: pgmap v1293: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 584 op/s; 28066/178190 objects degraded (15.751%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osdmap e489: 12 total, 11 up, 7 in 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 28066/178190 objects degraded (15.751%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:37 smithi121 ceph-mon[44533]: osdmap e490: 12 total, 11 up, 7 in 2024-04-29T14:27:38.232 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:38.273 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1294,"stamp":"2024-04-29T14:27:36.235896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5811405368,"num_objects":88669,"num_object_clones":0,"num_object_copies":177338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88669,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":178442,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14097,"num_bytes_recovered":925452416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161215,"ondisk_log_size":161215,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13065080,"kb_used_data":9573040,"kb_used_omap":10,"kb_used_meta":3491765,"kb_avail":643150984,"statfs":{"total":671965249536,"available":658586607616,"internally_reserved":0,"allocated":9802792960,"data_stored":9799836556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3575567454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-295632896,"num_objects":-4511,"num_object_clones":0,"num_object_copies":-9022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1355,"num_bytes_recovered":88801280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.255513"},"pg_stats":[{"pgid":"5.f","version":"486'11278","reported_seq":17088,"reported_epoch":486,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:31.432912+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:27:31.432912+0000","last_peered":"2024-04-29T14:27:31.432912+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:27:31.432912+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:27:31.432912+0000","mapping_epoch":467,"log_start":"440'1200","ondisk_log_start":"440'1200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372637696,"num_objects":5686,"num_object_clones":0,"num_object_copies":11372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11278,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":5681},{"shards":"4,11","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"488'11288","reported_seq":11488,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-29T14:27:35.557970+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:27:35.557970+0000","last_peered":"2024-04-29T14:27:35.557970+0000","last_clean":"2024-04-29T14:27:35.557970+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:27:35.557970+0000","last_undegraded":"2024-04-29T14:27:35.557970+0000","last_fullsized":"2024-04-29T14:27:35.557970+0000","mapping_epoch":432,"log_start":"440'1200","ondisk_log_start":"440'1200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360841216,"num_objects":5506,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"488'11255","reported_seq":17285,"reported_epoch":488,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:27:36.046407+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:27:36.046407+0000","last_peered":"2024-04-29T14:27:36.046407+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:27:36.046407+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:27:36.046407+0000","mapping_epoch":468,"log_start":"440'1200","ondisk_log_start":"440'1200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356384792,"num_objects":5439,"num_object_clones":0,"num_object_copies":10878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11255,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2913,"num_bytes_recovered":190906368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":6},{"shards":"11","objects":5433}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"486'11139","reported_seq":11348,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:31.336606+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:27:31.336606+0000","last_peered":"2024-04-29T14:27:31.336606+0000","last_clean":"2024-04-29T14:27:31.336606+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:27:31.336606+0000","last_undegraded":"2024-04-29T14:27:31.336606+0000","last_fullsized":"2024-04-29T14:27:31.336606+0000","mapping_epoch":432,"log_start":"440'1100","ondisk_log_start":"440'1100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370475008,"num_objects":5653,"num_object_clones":0,"num_object_copies":11306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"488'11226","reported_seq":11433,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-29T14:27:35.576953+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:27:35.576953+0000","last_peered":"2024-04-29T14:27:35.576953+0000","last_clean":"2024-04-29T14:27:35.576953+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:27:35.576953+0000","last_undegraded":"2024-04-29T14:27:35.576953+0000","last_fullsized":"2024-04-29T14:27:35.576953+0000","mapping_epoch":432,"log_start":"440'1200","ondisk_log_start":"440'1200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356515840,"num_objects":5440,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11226,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"487'13364","reported_seq":13699,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-29T14:27:33.374805+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:27:33.374805+0000","last_peered":"2024-04-29T14:27:33.374805+0000","last_clean":"2024-04-29T14:27:33.374805+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:27:33.374805+0000","last_undegraded":"2024-04-29T14:27:33.374805+0000","last_fullsized":"2024-04-29T14:27:33.374805+0000","mapping_epoch":432,"log_start":"444'3300","ondisk_log_start":"444'3300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:33.665333+0000","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356319232,"num_objects":5437,"num_object_clones":0,"num_object_copies":10874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11109,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"486'10993","reported_seq":11198,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:31.477160+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:27:31.477160+0000","last_peered":"2024-04-29T14:27:31.477160+0000","last_clean":"2024-04-29T14:27:31.477160+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:27:31.477160+0000","last_undegraded":"2024-04-29T14:27:31.477160+0000","last_fullsized":"2024-04-29T14:27:31.477160+0000","mapping_epoch":432,"log_start":"439'900","ondisk_log_start":"439'900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366280704,"num_objects":5589,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"488'11189","reported_seq":11393,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-29T14:27:36.059245+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:27:36.059245+0000","last_peered":"2024-04-29T14:27:36.059245+0000","last_clean":"2024-04-29T14:27:36.059245+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:27:36.059245+0000","last_undegraded":"2024-04-29T14:27:36.059245+0000","last_fullsized":"2024-04-29T14:27:36.059245+0000","mapping_epoch":432,"log_start":"440'1100","ondisk_log_start":"440'1100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355139584,"num_objects":5419,"num_object_clones":0,"num_object_copies":10838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11189,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"488'11294","reported_seq":17292,"reported_epoch":488,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:35.621916+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:27:35.621916+0000","last_peered":"2024-04-29T14:27:35.621916+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:27:35.621916+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:27:35.621916+0000","mapping_epoch":467,"log_start":"440'1200","ondisk_log_start":"440'1200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710144,"num_objects":5504,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":189792256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":5503},{"shards":"9,10","objects":1}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"486'11090","reported_seq":16769,"reported_epoch":486,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:31.525452+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:27:31.525452+0000","last_peered":"2024-04-29T14:27:31.525452+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:27:31.525452+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:27:31.525452+0000","mapping_epoch":467,"log_start":"440'1000","ondisk_log_start":"440'1000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371982336,"num_objects":5676,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11090,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2736,"num_bytes_recovered":179306496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":5630},{"shards":"7,10","objects":46}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"486'11099","reported_seq":11304,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:31.385140+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:27:31.385140+0000","last_peered":"2024-04-29T14:27:31.385140+0000","last_clean":"2024-04-29T14:27:31.385140+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:27:31.385140+0000","last_undegraded":"2024-04-29T14:27:31.385140+0000","last_fullsized":"2024-04-29T14:27:31.385140+0000","mapping_epoch":432,"log_start":"440'1000","ondisk_log_start":"440'1000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370999296,"num_objects":5661,"num_object_clones":0,"num_object_copies":11322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11099,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"488'11157","reported_seq":11410,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-29T14:27:35.252685+0000","last_change":"2024-04-29T14:26:17.483610+0000","last_active":"2024-04-29T14:27:35.252685+0000","last_peered":"2024-04-29T14:27:35.252685+0000","last_clean":"2024-04-29T14:27:35.252685+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:27:35.252685+0000","last_undegraded":"2024-04-29T14:27:35.252685+0000","last_fullsized":"2024-04-29T14:27:35.252685+0000","mapping_epoch":432,"log_start":"440'1100","ondisk_log_start":"440'1100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'8320","last_scrub_stamp":"2024-04-29T14:26:17.483424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:17.483424+0000","objects_scrubbed":8299,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:45.122376+0000","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357367808,"num_objects":5453,"num_object_clones":0,"num_object_copies":10906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11157,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"488'11149","reported_seq":11361,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-29T14:27:35.270497+0000","last_change":"2024-04-29T14:26:19.678585+0000","last_active":"2024-04-29T14:27:35.270497+0000","last_peered":"2024-04-29T14:27:35.270497+0000","last_clean":"2024-04-29T14:27:35.270497+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:27:35.270497+0000","last_undegraded":"2024-04-29T14:27:35.270497+0000","last_fullsized":"2024-04-29T14:27:35.270497+0000","mapping_epoch":432,"log_start":"440'1100","ondisk_log_start":"440'1100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'8397","last_scrub_stamp":"2024-04-29T14:26:19.678439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:19.678439+0000","objects_scrubbed":8210,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:27:35.532132+0000","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356581376,"num_objects":5441,"num_object_clones":0,"num_object_copies":10882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"486'10847","reported_seq":11055,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:31.397070+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:27:31.397070+0000","last_peered":"2024-04-29T14:27:31.397070+0000","last_clean":"2024-04-29T14:27:31.397070+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:27:31.397070+0000","last_undegraded":"2024-04-29T14:27:31.397070+0000","last_fullsized":"2024-04-29T14:27:31.397070+0000","mapping_epoch":432,"log_start":"439'800","ondisk_log_start":"439'800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366936064,"num_objects":5599,"num_object_clones":0,"num_object_copies":11198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"487'14023","reported_seq":14405,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-29T14:27:33.299092+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:27:33.299092+0000","last_peered":"2024-04-29T14:27:33.299092+0000","last_clean":"2024-04-29T14:27:33.299092+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:27:33.299092+0000","last_undegraded":"2024-04-29T14:27:33.299092+0000","last_fullsized":"2024-04-29T14:27:33.299092+0000","mapping_epoch":432,"log_start":"446'4000","ondisk_log_start":"446'4000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364052480,"num_objects":5555,"num_object_clones":0,"num_object_copies":11110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11061,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"487'11087","reported_seq":16699,"reported_epoch":487,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:33.363819+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:27:33.363819+0000","last_peered":"2024-04-29T14:27:33.363819+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:27:33.363819+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:27:33.363819+0000","mapping_epoch":467,"log_start":"440'1000","ondisk_log_start":"440'1000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367591424,"num_objects":5609,"num_object_clones":0,"num_object_copies":11218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2743,"num_bytes_recovered":179765248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":4},{"shards":"10","objects":5605}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1214,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-29T14:27:24.112429+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:27:24.112429+0000","last_peered":"2024-04-29T14:27:24.112429+0000","last_clean":"2024-04-29T14:27:24.112429+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:27:24.112429+0000","last_undegraded":"2024-04-29T14:27:24.112429+0000","last_fullsized":"2024-04-29T14:27:24.112429+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5810815000,"num_objects":88667,"num_object_clones":0,"num_object_copies":177334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178261,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14089,"num_bytes_recovered":923336704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12515090432,"data_stored":12515082288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161078,"ondisk_log_size":161078,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1219192,"kb_used_data":696556,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92525960,"statfs":{"total":95995035648,"available":94746583040,"internally_reserved":0,"allocated":713273344,"data_stored":712840182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":535165380},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383932,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1483808,"kb_used_data":1069268,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92261344,"statfs":{"total":95995035648,"available":94475616256,"internally_reserved":0,"allocated":1094930432,"data_stored":1094511129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645570,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1452908,"kb_used_data":1056608,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92292244,"statfs":{"total":95995035648,"available":94507257856,"internally_reserved":0,"allocated":1081966592,"data_stored":1081541074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234250,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1571996,"kb_used_data":1085460,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92173156,"statfs":{"total":95995035648,"available":94385311744,"internally_reserved":0,"allocated":1111511040,"data_stored":1111090418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528592,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1610784,"kb_used_data":1052052,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92134368,"statfs":{"total":95995035648,"available":94345592832,"internally_reserved":0,"allocated":1077301248,"data_stored":1076881965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":572127686},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575144,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738671,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2889832,"kb_used_data":2443552,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90855320,"statfs":{"total":95995035648,"available":93035847680,"internally_reserved":0,"allocated":2502197248,"data_stored":2501771730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986458,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2836560,"kb_used_data":2169544,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":90908592,"statfs":{"total":95995035648,"available":93090398208,"internally_reserved":0,"allocated":2221613056,"data_stored":2221200058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2497970176,"data_stored":2497970176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2212560896,"data_stored":2212560896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1069547520,"data_stored":1069547520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1104744448,"data_stored":1104740376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1078067200,"data_stored":1078067200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1090650112,"data_stored":1090650112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":711856128,"data_stored":711852056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.f is not active+clean 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.f', 'version': "486'11278", 'reported_seq': 17088, 'reported_epoch': 486, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:27:31.432912+0000', 'last_change': '2024-04-29T14:26:16.020854+0000', 'last_active': '2024-04-29T14:27:31.432912+0000', 'last_peered': '2024-04-29T14:27:31.432912+0000', 'last_clean': '2024-04-29T14:25:54.182894+0000', 'last_became_active': '2024-04-29T14:26:16.010174+0000', 'last_became_peered': '2024-04-29T14:26:16.010174+0000', 'last_unstale': '2024-04-29T14:27:31.432912+0000', 'last_undegraded': '2024-04-29T14:26:15.821893+0000', 'last_fullsized': '2024-04-29T14:27:31.432912+0000', 'mapping_epoch': 467, 'log_start': "440'1200", 'ondisk_log_start': "440'1200", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'7968", 'last_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'objects_scrubbed': 7919, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2114, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 372637696, 'num_objects': 5686, 'num_object_clones': 0, 'num_object_copies': 11372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 5681, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11278, 'num_write_kb': 542848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2801, 'num_bytes_recovered': 183566336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 11], 'acting': [4, 11], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 5681}, {'shards': '4,11', 'objects': 5}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.d is not active+clean 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.d', 'version': "488'11255", 'reported_seq': 17285, 'reported_epoch': 488, 'state': 'active+recovering+degraded+remapped', 'last_fresh': '2024-04-29T14:27:36.046407+0000', 'last_change': '2024-04-29T14:26:17.048329+0000', 'last_active': '2024-04-29T14:27:36.046407+0000', 'last_peered': '2024-04-29T14:27:36.046407+0000', 'last_clean': '2024-04-29T14:25:54.192906+0000', 'last_became_active': '2024-04-29T14:26:17.022020+0000', 'last_became_peered': '2024-04-29T14:26:17.022020+0000', 'last_unstale': '2024-04-29T14:27:36.046407+0000', 'last_undegraded': '2024-04-29T14:26:16.808487+0000', 'last_fullsized': '2024-04-29T14:27:36.046407+0000', 'mapping_epoch': 468, 'log_start': "440'1200", 'ondisk_log_start': "440'1200", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "454'7099", 'last_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'objects_scrubbed': 7053, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1409, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 356384792, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 10878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 5433, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11255, 'num_write_kb': 534145, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2913, 'num_bytes_recovered': 190906368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 11], 'acting': [11, 7], 'avail_no_missing': ['11'], 'object_location_counts': [{'shards': '7,11', 'objects': 6}, {'shards': '11', 'objects': 5433}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 11, 'purged_snaps': []} 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.7 is not active+clean 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.7', 'version': "488'11294", 'reported_seq': 17292, 'reported_epoch': 488, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:27:35.621916+0000', 'last_change': '2024-04-29T14:26:16.052811+0000', 'last_active': '2024-04-29T14:27:35.621916+0000', 'last_peered': '2024-04-29T14:27:35.621916+0000', 'last_clean': '2024-04-29T14:25:54.182628+0000', 'last_became_active': '2024-04-29T14:26:16.038648+0000', 'last_became_peered': '2024-04-29T14:26:16.038648+0000', 'last_unstale': '2024-04-29T14:27:35.621916+0000', 'last_undegraded': '2024-04-29T14:26:15.818605+0000', 'last_fullsized': '2024-04-29T14:27:35.621916+0000', 'mapping_epoch': 467, 'log_start': "440'1200", 'ondisk_log_start': "440'1200", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "455'7479", 'last_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'objects_scrubbed': 7424, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1679, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 360710144, 'num_objects': 5504, 'num_object_clones': 0, 'num_object_copies': 11008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 5503, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11294, 'num_write_kb': 537536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2896, 'num_bytes_recovered': 189792256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 10], 'acting': [9, 10], 'avail_no_missing': ['9'], 'object_location_counts': [{'shards': '9', 'objects': 5503}, {'shards': '9,10', 'objects': 1}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-29T14:27:38.274 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "486'11090", 'reported_seq': 16769, 'reported_epoch': 486, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:27:31.525452+0000', 'last_change': '2024-04-29T14:26:16.126893+0000', 'last_active': '2024-04-29T14:27:31.525452+0000', 'last_peered': '2024-04-29T14:27:31.525452+0000', 'last_clean': '2024-04-29T14:25:53.790194+0000', 'last_became_active': '2024-04-29T14:26:16.054003+0000', 'last_became_peered': '2024-04-29T14:26:16.054003+0000', 'last_unstale': '2024-04-29T14:27:31.525452+0000', 'last_undegraded': '2024-04-29T14:26:15.820366+0000', 'last_fullsized': '2024-04-29T14:27:31.525452+0000', 'mapping_epoch': 467, 'log_start': "440'1000", 'ondisk_log_start': "440'1000", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'8070", 'last_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'objects_scrubbed': 8029, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1689, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 371982336, 'num_objects': 5676, 'num_object_clones': 0, 'num_object_copies': 11352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 5630, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11090, 'num_write_kb': 536512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2736, 'num_bytes_recovered': 179306496, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 10], 'acting': [7, 10], 'avail_no_missing': ['7'], 'object_location_counts': [{'shards': '7', 'objects': 5630}, {'shards': '7,10', 'objects': 46}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-29T14:27:38.275 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-29T14:27:38.275 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "487'11087", 'reported_seq': 16699, 'reported_epoch': 487, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:27:33.363819+0000', 'last_change': '2024-04-29T14:26:15.998964+0000', 'last_active': '2024-04-29T14:27:33.363819+0000', 'last_peered': '2024-04-29T14:27:33.363819+0000', 'last_clean': '2024-04-29T14:25:59.019434+0000', 'last_became_active': '2024-04-29T14:26:15.982158+0000', 'last_became_peered': '2024-04-29T14:26:15.982158+0000', 'last_unstale': '2024-04-29T14:27:33.363819+0000', 'last_undegraded': '2024-04-29T14:26:15.829402+0000', 'last_fullsized': '2024-04-29T14:27:33.363819+0000', 'mapping_epoch': 467, 'log_start': "440'1000", 'ondisk_log_start': "440'1000", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 367591424, 'num_objects': 5609, 'num_object_clones': 0, 'num_object_copies': 11218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 5605, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11087, 'num_write_kb': 534272, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2743, 'num_bytes_recovered': 179765248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 0], 'acting': [10, 0], 'avail_no_missing': ['10'], 'object_location_counts': [{'shards': '0,10', 'objects': 4}, {'shards': '10', 'objects': 5605}], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-04-29T14:27:38.275 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:27:38.512 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:38 smithi040 ceph-mon[39581]: from='client.16524 v1:172.21.15.40:0/3496774259' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:38.512 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:38 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:38.512 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:38 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:38.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:38 smithi194 ceph-mon[44089]: from='client.16524 v1:172.21.15.40:0/3496774259' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:38.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:38 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:38.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:38 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:38.596 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:38.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:38.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:38 smithi121 ceph-mon[44533]: from='client.16524 v1:172.21.15.40:0/3496774259' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:38.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:38 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:38.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:38 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:38.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:38.730 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:39.150 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:39.398 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:39.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:39.505 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:39.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:39 smithi040 ceph-mon[39581]: pgmap v1296: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 27479/175060 objects degraded (15.697%); 11 MiB/s, 169 objects/s recovering 2024-04-29T14:27:39.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:39 smithi194 ceph-mon[44089]: pgmap v1296: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 27479/175060 objects degraded (15.697%); 11 MiB/s, 169 objects/s recovering 2024-04-29T14:27:39.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:27:39.569 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:39.612 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:39.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:39 smithi121 ceph-mon[44533]: pgmap v1296: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 27479/175060 objects degraded (15.697%); 11 MiB/s, 169 objects/s recovering 2024-04-29T14:27:39.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:39.755 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:39.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:27:40.010 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:40.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:40.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:40.162 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:40.199 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:40.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:40.240 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:27:40.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:40.342 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:40.894 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:40.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:40.911 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:40.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:41.037 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:41.375 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:41 smithi121 ceph-mon[44533]: pgmap v1297: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 27278/173874 objects degraded (15.688%); 10 MiB/s, 164 objects/s recovering 2024-04-29T14:27:41.431 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:41 smithi040 ceph-mon[39581]: pgmap v1297: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 27278/173874 objects degraded (15.688%); 10 MiB/s, 164 objects/s recovering 2024-04-29T14:27:41.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:41 smithi194 ceph-mon[44089]: pgmap v1297: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 27278/173874 objects degraded (15.688%); 10 MiB/s, 164 objects/s recovering 2024-04-29T14:27:42.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:42 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 27278/173874 objects degraded (15.688%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:42.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:42 smithi040 ceph-mon[39581]: osdmap e491: 12 total, 11 up, 7 in 2024-04-29T14:27:42.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:42 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 27278/173874 objects degraded (15.688%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:42.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:42 smithi194 ceph-mon[44089]: osdmap e491: 12 total, 11 up, 7 in 2024-04-29T14:27:42.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:42 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 27278/173874 objects degraded (15.688%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:42.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:42 smithi121 ceph-mon[44533]: osdmap e491: 12 total, 11 up, 7 in 2024-04-29T14:27:42.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:42.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:42.804 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:42.837 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:27:42.837 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26492/168698 objects degraded (15.704%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1536,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":491,"num_osds":12,"num_up_osds":11,"osd_up_since":1714400760,"num_in_osds":7,"osd_in_since":1714400774,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovering+degraded","count":4},{"state_name":"active+recovering+degraded+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":84349,"data_bytes":5528289848,"bytes_used":12914229248,"bytes_avail":659051020288,"bytes_total":671965249536,"degraded_objects":26492,"degraded_total":168698,"degraded_ratio":0.15703802060486788,"recovering_objects_per_sec":262,"recovering_bytes_per_sec":17176261,"recovering_keys_per_sec":0,"num_objects_recovered":1771,"num_bytes_recovered":116064256,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":855},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":40,"modified":"2024-04-29T14:24:06.771444+0000","services":{}},"progress_events":{}} 2024-04-29T14:27:42.854 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:42.895 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:42.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:42.999 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:43.049 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:43.099 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:43.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:43.203 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:43.253 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:43 smithi121 ceph-mon[44533]: pgmap v1298: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 26492/168698 objects degraded (15.704%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:27:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:43 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:27:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:43 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:27:43.263 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1915346461' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:27:43.301 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:43.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:43.410 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:43.451 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:43 smithi194 ceph-mon[44089]: pgmap v1298: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 26492/168698 objects degraded (15.704%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:27:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:43 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:27:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:43 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:27:43.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1915346461' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:27:43.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:43 smithi040 ceph-mon[39581]: pgmap v1298: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 26492/168698 objects degraded (15.704%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:27:43.613 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:43 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:27:43.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:43 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:27:43.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1915346461' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:27:43.774 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:27:43.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:27:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:44.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:44.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:44.140 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:44.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:44.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:27:44.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:44 smithi040 ceph-mon[39581]: osdmap e492: 12 total, 11 up, 7 in 2024-04-29T14:27:44.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1048908974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:44.279 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:44 smithi040 ceph-mon[39581]: from='client.36137 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:44.294 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:27:44.307 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:44.323 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:27:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:27:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:44.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:27:44.353 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:27:44.374 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:27:44.400 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:27:44.430 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:27:44.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:44.459 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:27:44.473 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:44.485 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:27:44.508 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:27:44.532 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:27:44.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:44 smithi194 ceph-mon[44089]: osdmap e492: 12 total, 11 up, 7 in 2024-04-29T14:27:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1048908974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:44.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:44 smithi194 ceph-mon[44089]: from='client.36137 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:44.586 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:27:44.617 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:27:44.618 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:27:44.619 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:27:44.619 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:27:44.619 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:27:44.619 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:27:44.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:44 smithi121 ceph-mon[44533]: osdmap e492: 12 total, 11 up, 7 in 2024-04-29T14:27:44.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1048908974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:44.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:44 smithi121 ceph-mon[44533]: from='client.36137 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:27:45.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: pgmap v1301: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 556 op/s; 26284/167506 objects degraded (15.691%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:27:45.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:45.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: from='client.36137 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osdmap e493: 12 total, 11 up, 7 in 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2813063635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: pgmap v1301: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 556 op/s; 26284/167506 objects degraded (15.691%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: from='client.36137 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osdmap e493: 12 total, 11 up, 7 in 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2813063635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: pgmap v1301: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 556 op/s; 26284/167506 objects degraded (15.691%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: from='client.36137 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osdmap e493: 12 total, 11 up, 7 in 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2813063635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:27:45.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:27:46.060 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:46.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:46 smithi194 ceph-mon[44089]: 5.2 scrub starts 2024-04-29T14:27:46.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:46 smithi194 ceph-mon[44089]: pgmap v1303: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 11 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 26074/165554 objects degraded (15.750%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:27:46.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:46 smithi040 ceph-mon[39581]: 5.2 scrub starts 2024-04-29T14:27:46.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:46 smithi040 ceph-mon[39581]: pgmap v1303: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 11 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 26074/165554 objects degraded (15.750%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:27:46.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:46 smithi121 ceph-mon[44533]: 5.2 scrub starts 2024-04-29T14:27:46.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:46 smithi121 ceph-mon[44533]: pgmap v1303: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 11 active+clean; 5.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 693 op/s; 26074/165554 objects degraded (15.750%); 13 MiB/s, 200 objects/s recovering 2024-04-29T14:27:46.902 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:46.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:47.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:27:47.057 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:47.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:27:47.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:47 smithi121 ceph-mon[44533]: 5.2 scrub ok 2024-04-29T14:27:47.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:47 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 26074/165554 objects degraded (15.750%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:47.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:47 smithi121 ceph-mon[44533]: osdmap e494: 12 total, 11 up, 7 in 2024-04-29T14:27:47.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:47 smithi040 ceph-mon[39581]: 5.2 scrub ok 2024-04-29T14:27:47.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:47 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 26074/165554 objects degraded (15.750%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:47.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:47 smithi040 ceph-mon[39581]: osdmap e494: 12 total, 11 up, 7 in 2024-04-29T14:27:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:47 smithi194 ceph-mon[44089]: 5.2 scrub ok 2024-04-29T14:27:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:47 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 26074/165554 objects degraded (15.750%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:47 smithi194 ceph-mon[44089]: osdmap e494: 12 total, 11 up, 7 in 2024-04-29T14:27:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:47.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:27:48.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:48 smithi040 ceph-mon[39581]: pgmap v1305: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+degraded, 11 active+clean; 5.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 545 op/s; 25494/162388 objects degraded (15.699%); 11 MiB/s, 172 objects/s recovering 2024-04-29T14:27:48.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:48 smithi194 ceph-mon[44089]: pgmap v1305: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+degraded, 11 active+clean; 5.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 545 op/s; 25494/162388 objects degraded (15.699%); 11 MiB/s, 172 objects/s recovering 2024-04-29T14:27:48.620 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:27:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:27:48.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:48 smithi121 ceph-mon[44533]: pgmap v1305: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+degraded, 11 active+clean; 5.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 545 op/s; 25494/162388 objects degraded (15.699%); 11 MiB/s, 172 objects/s recovering 2024-04-29T14:27:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:27:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:49.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:27:49.123 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:49.205 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:49.243 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:49.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:49.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:49.407 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:49.437 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:49.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:49.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:49 smithi194 ceph-mon[44089]: from='client.36146 v1:172.21.15.40:0/3449909362' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:49.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:27:49.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:49 smithi040 ceph-mon[39581]: from='client.36146 v1:172.21.15.40:0/3449909362' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:49.575 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:49.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:49 smithi121 ceph-mon[44533]: from='client.36146 v1:172.21.15.40:0/3449909362' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:49.677 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:27:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:49.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:27:49.698 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1305,"stamp":"2024-04-29T14:27:47.984055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5321523768,"num_objects":81194,"num_object_clones":0,"num_object_copies":162388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81194,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":185917,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16455,"num_bytes_recovered":1079986304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160990,"ondisk_log_size":160990,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12264088,"kb_used_data":8776324,"kb_used_omap":10,"kb_used_meta":3487605,"kb_avail":643951976,"statfs":{"total":671965249536,"available":659406823424,"internally_reserved":0,"allocated":8986955776,"data_stored":8983755807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3571307614},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-206766080,"num_objects":-3155,"num_object_clones":0,"num_object_copies":-6310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":998,"num_bytes_recovered":65404928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.780805"},"pg_stats":[{"pgid":"5.f","version":"493'11880","reported_seq":18899,"reported_epoch":493,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:46.454402+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:27:46.454402+0000","last_peered":"2024-04-29T14:27:46.454402+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:27:46.454402+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:27:46.454402+0000","mapping_epoch":467,"log_start":"441'1800","ondisk_log_start":"441'1800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333185024,"num_objects":5084,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3399,"num_bytes_recovered":222756864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":5083},{"shards":"4,11","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"493'11721","reported_seq":11931,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:45.621838+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:27:45.621838+0000","last_peered":"2024-04-29T14:27:45.621838+0000","last_clean":"2024-04-29T14:27:45.621838+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:27:45.621838+0000","last_undegraded":"2024-04-29T14:27:45.621838+0000","last_fullsized":"2024-04-29T14:27:45.621838+0000","mapping_epoch":432,"log_start":"441'1700","ondisk_log_start":"441'1700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332464128,"num_objects":5073,"num_object_clones":0,"num_object_copies":10146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11721,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"493'11636","reported_seq":18430,"reported_epoch":493,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:27:46.106342+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:27:46.106342+0000","last_peered":"2024-04-29T14:27:46.106342+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:27:46.106342+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:27:46.106342+0000","mapping_epoch":468,"log_start":"440'1600","ondisk_log_start":"440'1600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331415576,"num_objects":5058,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11636,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3290,"num_bytes_recovered":215613440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":2},{"shards":"11","objects":5056}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"493'11740","reported_seq":11962,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:46.468006+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:27:46.468006+0000","last_peered":"2024-04-29T14:27:46.468006+0000","last_clean":"2024-04-29T14:27:46.468006+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:27:46.468006+0000","last_undegraded":"2024-04-29T14:27:46.468006+0000","last_fullsized":"2024-04-29T14:27:46.468006+0000","mapping_epoch":432,"log_start":"441'1700","ondisk_log_start":"441'1700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331087872,"num_objects":5052,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11740,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"493'11616","reported_seq":11833,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:45.620932+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:27:45.620932+0000","last_peered":"2024-04-29T14:27:45.620932+0000","last_clean":"2024-04-29T14:27:45.620932+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:27:45.620932+0000","last_undegraded":"2024-04-29T14:27:45.620932+0000","last_fullsized":"2024-04-29T14:27:45.620932+0000","mapping_epoch":432,"log_start":"440'1600","ondisk_log_start":"440'1600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330956800,"num_objects":5050,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"492'13768","reported_seq":14113,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-04-29T14:27:43.373517+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:27:43.373517+0000","last_peered":"2024-04-29T14:27:43.373517+0000","last_clean":"2024-04-29T14:27:43.373517+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:27:43.373517+0000","last_undegraded":"2024-04-29T14:27:43.373517+0000","last_fullsized":"2024-04-29T14:27:43.373517+0000","mapping_epoch":432,"log_start":"444'3700","ondisk_log_start":"444'3700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329842688,"num_objects":5033,"num_object_clones":0,"num_object_copies":10066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11513,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"493'11560","reported_seq":11779,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:46.468883+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:27:46.468883+0000","last_peered":"2024-04-29T14:27:46.468883+0000","last_clean":"2024-04-29T14:27:46.468883+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:27:46.468883+0000","last_undegraded":"2024-04-29T14:27:46.468883+0000","last_fullsized":"2024-04-29T14:27:46.468883+0000","mapping_epoch":432,"log_start":"440'1500","ondisk_log_start":"440'1500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329121792,"num_objects":5022,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"493'11579","reported_seq":11793,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:46.084813+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:27:46.084813+0000","last_peered":"2024-04-29T14:27:46.084813+0000","last_clean":"2024-04-29T14:27:46.084813+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:27:46.084813+0000","last_undegraded":"2024-04-29T14:27:46.084813+0000","last_fullsized":"2024-04-29T14:27:46.084813+0000","mapping_epoch":432,"log_start":"440'1500","ondisk_log_start":"440'1500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329580544,"num_objects":5029,"num_object_clones":0,"num_object_copies":10058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"493'11698","reported_seq":18516,"reported_epoch":493,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:45.661942+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:27:45.661942+0000","last_peered":"2024-04-29T14:27:45.661942+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:27:45.661942+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:27:45.661942+0000","mapping_epoch":467,"log_start":"440'1600","ondisk_log_start":"440'1600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334233600,"num_objects":5100,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3301,"num_bytes_recovered":216334336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":5098},{"shards":"9,10","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"493'11688","reported_seq":18516,"reported_epoch":493,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:46.539580+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:27:46.539580+0000","last_peered":"2024-04-29T14:27:46.539580+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:27:46.539580+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:27:46.539580+0000","mapping_epoch":467,"log_start":"440'1600","ondisk_log_start":"440'1600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332791808,"num_objects":5078,"num_object_clones":0,"num_object_copies":10156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3305,"num_bytes_recovered":216596480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":5061},{"shards":"7,10","objects":17}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"493'11669","reported_seq":11888,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:46.445355+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:27:46.445355+0000","last_peered":"2024-04-29T14:27:46.445355+0000","last_clean":"2024-04-29T14:27:46.445355+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:27:46.445355+0000","last_undegraded":"2024-04-29T14:27:46.445355+0000","last_fullsized":"2024-04-29T14:27:46.445355+0000","mapping_epoch":432,"log_start":"440'1600","ondisk_log_start":"440'1600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333643776,"num_objects":5091,"num_object_clones":0,"num_object_copies":10182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"493'11548","reported_seq":11811,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:45.320925+0000","last_change":"2024-04-29T14:26:17.483610+0000","last_active":"2024-04-29T14:27:45.320925+0000","last_peered":"2024-04-29T14:27:45.320925+0000","last_clean":"2024-04-29T14:27:45.320925+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:27:45.320925+0000","last_undegraded":"2024-04-29T14:27:45.320925+0000","last_fullsized":"2024-04-29T14:27:45.320925+0000","mapping_epoch":432,"log_start":"440'1500","ondisk_log_start":"440'1500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'8320","last_scrub_stamp":"2024-04-29T14:26:17.483424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:17.483424+0000","objects_scrubbed":8299,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331743232,"num_objects":5062,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11548,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"493'11530","reported_seq":11758,"reported_epoch":493,"state":"active+clean+scrubbing","last_fresh":"2024-04-29T14:27:45.326751+0000","last_change":"2024-04-29T14:27:45.302099+0000","last_active":"2024-04-29T14:27:45.326751+0000","last_peered":"2024-04-29T14:27:45.326751+0000","last_clean":"2024-04-29T14:27:45.326751+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:27:45.326751+0000","last_undegraded":"2024-04-29T14:27:45.326751+0000","last_fullsized":"2024-04-29T14:27:45.326751+0000","mapping_epoch":432,"log_start":"440'1500","ondisk_log_start":"440'1500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'8397","last_scrub_stamp":"2024-04-29T14:26:19.678439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:19.678439+0000","objects_scrubbed":199,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331612160,"num_objects":5060,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11530,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"493'11425","reported_seq":11647,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:46.437998+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:27:46.437998+0000","last_peered":"2024-04-29T14:27:46.437998+0000","last_clean":"2024-04-29T14:27:46.437998+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:27:46.437998+0000","last_undegraded":"2024-04-29T14:27:46.437998+0000","last_fullsized":"2024-04-29T14:27:46.437998+0000","mapping_epoch":432,"log_start":"440'1400","ondisk_log_start":"440'1400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329056256,"num_objects":5021,"num_object_clones":0,"num_object_copies":10042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11425,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"492'14402","reported_seq":14794,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-04-29T14:27:43.398961+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:27:43.398961+0000","last_peered":"2024-04-29T14:27:43.398961+0000","last_clean":"2024-04-29T14:27:43.398961+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:27:43.398961+0000","last_undegraded":"2024-04-29T14:27:43.398961+0000","last_fullsized":"2024-04-29T14:27:43.398961+0000","mapping_epoch":432,"log_start":"447'4400","ondisk_log_start":"447'4400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339214336,"num_objects":5176,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11440,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"492'11493","reported_seq":17933,"reported_epoch":492,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:43.426866+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:27:43.426866+0000","last_peered":"2024-04-29T14:27:43.426866+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:27:43.426866+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:27:43.426866+0000","mapping_epoch":467,"log_start":"440'1400","ondisk_log_start":"440'1400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340983808,"num_objects":5203,"num_object_clones":0,"num_object_copies":10406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3152,"num_bytes_recovered":206569472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":7},{"shards":"10","objects":5196}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1229,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-29T14:27:44.291678+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:27:44.291678+0000","last_peered":"2024-04-29T14:27:44.291678+0000","last_clean":"2024-04-29T14:27:44.291678+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:27:44.291678+0000","last_undegraded":"2024-04-29T14:27:44.291678+0000","last_fullsized":"2024-04-29T14:27:44.291678+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5320933400,"num_objects":81192,"num_object_clones":0,"num_object_copies":162384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185736,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16447,"num_bytes_recovered":1077870592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11703820288,"data_stored":11703812144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122298,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1175028,"kb_used_data":648232,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92570124,"statfs":{"total":95995035648,"available":94791806976,"internally_reserved":0,"allocated":663789568,"data_stored":663325181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":539425220},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383934,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1402896,"kb_used_data":992708,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92342256,"statfs":{"total":95995035648,"available":94558470144,"internally_reserved":0,"allocated":1016532992,"data_stored":1016088749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":420018624},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645572,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1379504,"kb_used_data":978972,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92365648,"statfs":{"total":95995035648,"available":94582423552,"internally_reserved":0,"allocated":1002467328,"data_stored":1002010585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422876,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234253,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457780,"kb_used_data":975400,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92287372,"statfs":{"total":95995035648,"available":94502268928,"internally_reserved":0,"allocated":998809600,"data_stored":998345233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1536616,"kb_used_data":973724,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92208536,"statfs":{"total":95995035648,"available":94421540864,"internally_reserved":0,"allocated":997093376,"data_stored":996636633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":576387526},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2706660,"kb_used_data":2264540,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91038492,"statfs":{"total":95995035648,"available":93223415808,"internally_reserved":0,"allocated":2318888960,"data_stored":2318432217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986462,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2605604,"kb_used_data":1942748,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91139548,"statfs":{"total":95995035648,"available":93326897152,"internally_reserved":0,"allocated":1989373952,"data_stored":1988917209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2315255808,"data_stored":2315255808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1981874176,"data_stored":1981874176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":995885056,"data_stored":995885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":990777344,"data_stored":990773272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":997130240,"data_stored":997130240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1011744768,"data_stored":1011744768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":661458944,"data_stored":661454872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:27:49.729 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:49.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:27:49.815 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:49.891 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:27:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:49.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:27:49.996 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:50.064 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:50.100 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:50.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:50.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:50.243 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:50.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:50.373 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:50.443 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:50.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:50.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:50.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:50 smithi040 ceph-mon[39581]: 5.3 scrub starts 2024-04-29T14:27:50.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:50 smithi040 ceph-mon[39581]: pgmap v1306: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+degraded, 11 active+clean; 4.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 560 op/s; 25277/161172 objects degraded (15.683%); 11 MiB/s, 181 objects/s recovering 2024-04-29T14:27:50.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:50 smithi121 ceph-mon[44533]: 5.3 scrub starts 2024-04-29T14:27:50.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:50 smithi121 ceph-mon[44533]: pgmap v1306: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+degraded, 11 active+clean; 4.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 560 op/s; 25277/161172 objects degraded (15.683%); 11 MiB/s, 181 objects/s recovering 2024-04-29T14:27:50.644 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:50 smithi194 ceph-mon[44089]: 5.3 scrub starts 2024-04-29T14:27:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:50 smithi194 ceph-mon[44089]: pgmap v1306: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+degraded, 11 active+clean; 4.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 560 op/s; 25277/161172 objects degraded (15.683%); 11 MiB/s, 181 objects/s recovering 2024-04-29T14:27:50.992 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:27:50.992 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:27:51.498 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:51.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:51.603 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:51.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:51 smithi040 ceph-mon[39581]: 5.3 scrub ok 2024-04-29T14:27:51.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:51 smithi121 ceph-mon[44533]: 5.3 scrub ok 2024-04-29T14:27:51.680 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:51.682 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:51 smithi194 ceph-mon[44089]: 5.3 scrub ok 2024-04-29T14:27:51.717 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:51.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:51.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:51.855 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:52.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:52.162 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:52.207 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:27:52.223 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:52.269 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:27:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:52.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:27:52.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:27:52.415 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:52.466 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:27:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:52.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:27:52.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/661369857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:52.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:52 smithi194 ceph-mon[44089]: from='client.36149 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:52.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:52 smithi194 ceph-mon[44089]: pgmap v1307: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 730 op/s; 24450/155812 objects degraded (15.692%); 14 MiB/s, 229 objects/s recovering 2024-04-29T14:27:52.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:52 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 25277/161172 objects degraded (15.683%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:52.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:52 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:27:52.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:52 smithi194 ceph-mon[44089]: from='client.36149 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:52.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:52 smithi194 ceph-mon[44089]: osdmap e495: 12 total, 11 up, 7 in 2024-04-29T14:27:52.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:27:52.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/661369857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:52.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:52 smithi040 ceph-mon[39581]: from='client.36149 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:52.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:52 smithi040 ceph-mon[39581]: pgmap v1307: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 730 op/s; 24450/155812 objects degraded (15.692%); 14 MiB/s, 229 objects/s recovering 2024-04-29T14:27:52.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:52 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 25277/161172 objects degraded (15.683%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:52.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:52 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:27:52.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:52 smithi040 ceph-mon[39581]: from='client.36149 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:52.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:52 smithi040 ceph-mon[39581]: osdmap e495: 12 total, 11 up, 7 in 2024-04-29T14:27:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/661369857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:52 smithi121 ceph-mon[44533]: from='client.36149 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:27:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:52 smithi121 ceph-mon[44533]: pgmap v1307: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 730 op/s; 24450/155812 objects degraded (15.692%); 14 MiB/s, 229 objects/s recovering 2024-04-29T14:27:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:52 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 25277/161172 objects degraded (15.683%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:52 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:27:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:52 smithi121 ceph-mon[44533]: from='client.36149 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:27:52.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:52 smithi121 ceph-mon[44533]: osdmap e495: 12 total, 11 up, 7 in 2024-04-29T14:27:52.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:27:52.963 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:27:53.503 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:27:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:53.530 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:27:53.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:27:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:27:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:53.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:27:53.580 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:27:53.600 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:27:53.622 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:27:53.646 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:27:53.666 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:53.681 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:27:53.707 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:27:53.728 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:53.741 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:27:53.742 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:27:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:27:53.772 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:53.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:53.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2533013197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:53.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2533013197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:53.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2533013197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:27:53.877 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:53.928 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:53.984 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:53.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:54.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:27:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:54.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:54 smithi194 ceph-mon[44089]: pgmap v1309: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 24258/154626 objects degraded (15.688%); 14 MiB/s, 226 objects/s recovering 2024-04-29T14:27:54.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:54 smithi040 ceph-mon[39581]: pgmap v1309: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 24258/154626 objects degraded (15.688%); 14 MiB/s, 226 objects/s recovering 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:54.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:54 smithi121 ceph-mon[44533]: pgmap v1309: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 24258/154626 objects degraded (15.688%); 14 MiB/s, 226 objects/s recovering 2024-04-29T14:27:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:27:54.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:27:55.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:27:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:27:55.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:55.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:27:55.609 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:55.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:27:55.742 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:55.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:27:55.782 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:55.792 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:55.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:27:55.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:27:55.894 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:55.961 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:56.004 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:56.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:56.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:56.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:56.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:56.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:27:56.325 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:56.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:27:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:56.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:27:56.457 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:27:56.511 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:56.575 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:27:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:27:56.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:56 smithi194 ceph-mon[44089]: pgmap v1310: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 24058/152664 objects degraded (15.759%); 10 MiB/s, 163 objects/s recovering 2024-04-29T14:27:56.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:56 smithi040 ceph-mon[39581]: pgmap v1310: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 24058/152664 objects degraded (15.759%); 10 MiB/s, 163 objects/s recovering 2024-04-29T14:27:56.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:56 smithi121 ceph-mon[44533]: pgmap v1310: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 553 op/s; 24058/152664 objects degraded (15.759%); 10 MiB/s, 163 objects/s recovering 2024-04-29T14:27:56.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:192601 5.7 5:e1f865db:::benchmark_data_smithi194_121512_object59135:head [delete] snapc 0=[] ondisk+write+known_if_redirected e495)", 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:27:56.191350+0000", 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0016365349999999999, 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016726519999999999, 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:27:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 192601 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191350+0000", 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191350+0000", 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191353+0000", 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2940000000000001e-06 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191366+0000", 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2697000000000001e-05 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191387+0000", 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0788000000000002e-05 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:27:57.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191400+0000", 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3735e-05 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191477+0000", 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6243999999999997e-05 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191604+0000", 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012730200000000001 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:56.191780+0000", 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000176207 2024-04-29T14:27:57.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:27:57.129 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:27:57.129 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:27:57.129 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:27:57.129 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:27:57.129 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 1 2024-04-29T14:27:57.129 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:27:57.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:57 smithi194 ceph-mon[44089]: from='client.16566 v1:172.21.15.40:0/3679737156' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:57.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:57 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 24058/152664 objects degraded (15.759%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:57.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:57 smithi040 ceph-mon[39581]: from='client.16566 v1:172.21.15.40:0/3679737156' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:57.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:57 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 24058/152664 objects degraded (15.759%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:57.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:57 smithi121 ceph-mon[44533]: from='client.16566 v1:172.21.15.40:0/3679737156' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:27:57.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:57 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 24058/152664 objects degraded (15.759%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:27:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1310,"stamp":"2024-04-29T14:27:55.985723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5002887736,"num_objects":76332,"num_object_clones":0,"num_object_copies":152664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76332,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":190779,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17891,"num_bytes_recovered":1174096000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11792656,"kb_used_data":8308884,"kb_used_omap":10,"kb_used_meta":3483445,"kb_avail":644423408,"statfs":{"total":671965249536,"available":659889569792,"internally_reserved":0,"allocated":8508297216,"data_stored":8505021877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3567047774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-318636032,"num_objects":-4862,"num_object_clones":0,"num_object_copies":-9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":94109696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.787777"},"pg_stats":[{"pgid":"5.f","version":"494'12071","reported_seq":19496,"reported_epoch":494,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:51.456723+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:27:51.456723+0000","last_peered":"2024-04-29T14:27:51.456723+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:27:51.456723+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:27:51.456723+0000","mapping_epoch":467,"log_start":"441'2000","ondisk_log_start":"441'2000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320667648,"num_objects":4893,"num_object_clones":0,"num_object_copies":9786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3601,"num_bytes_recovered":235995136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":4881},{"shards":"4,11","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"495'12123","reported_seq":12337,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-04-29T14:27:55.678978+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:27:55.678978+0000","last_peered":"2024-04-29T14:27:55.678978+0000","last_clean":"2024-04-29T14:27:55.678978+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:27:55.678978+0000","last_undegraded":"2024-04-29T14:27:55.678978+0000","last_fullsized":"2024-04-29T14:27:55.678978+0000","mapping_epoch":432,"log_start":"441'2100","ondisk_log_start":"441'2100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306118656,"num_objects":4671,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12123,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"494'11841","reported_seq":19044,"reported_epoch":494,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:27:51.106979+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:27:51.106979+0000","last_peered":"2024-04-29T14:27:51.106979+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:27:51.106979+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:27:51.106979+0000","mapping_epoch":468,"log_start":"441'1800","ondisk_log_start":"441'1800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317980696,"num_objects":4853,"num_object_clones":0,"num_object_copies":9706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11841,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3494,"num_bytes_recovered":228982784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":1},{"shards":"11","objects":4852}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"494'11958","reported_seq":12183,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-29T14:27:51.423841+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:27:51.423841+0000","last_peered":"2024-04-29T14:27:51.423841+0000","last_clean":"2024-04-29T14:27:51.423841+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:27:51.423841+0000","last_undegraded":"2024-04-29T14:27:51.423841+0000","last_fullsized":"2024-04-29T14:27:51.423841+0000","mapping_epoch":432,"log_start":"441'1900","ondisk_log_start":"441'1900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316801024,"num_objects":4834,"num_object_clones":0,"num_object_copies":9668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11958,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"495'12043","reported_seq":12264,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-04-29T14:27:55.605952+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:27:55.605952+0000","last_peered":"2024-04-29T14:27:55.605952+0000","last_clean":"2024-04-29T14:27:55.605952+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:27:55.605952+0000","last_undegraded":"2024-04-29T14:27:55.605952+0000","last_fullsized":"2024-04-29T14:27:55.605952+0000","mapping_epoch":432,"log_start":"441'2000","ondisk_log_start":"441'2000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302972928,"num_objects":4623,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"495'14144","reported_seq":14495,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-04-29T14:27:53.456062+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:27:53.456062+0000","last_peered":"2024-04-29T14:27:53.456062+0000","last_clean":"2024-04-29T14:27:53.456062+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:27:53.456062+0000","last_undegraded":"2024-04-29T14:27:53.456062+0000","last_fullsized":"2024-04-29T14:27:53.456062+0000","mapping_epoch":432,"log_start":"446'4100","ondisk_log_start":"446'4100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305201152,"num_objects":4657,"num_object_clones":0,"num_object_copies":9314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"494'11757","reported_seq":11978,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-29T14:27:51.476169+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:27:51.476169+0000","last_peered":"2024-04-29T14:27:51.476169+0000","last_clean":"2024-04-29T14:27:51.476169+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:27:51.476169+0000","last_undegraded":"2024-04-29T14:27:51.476169+0000","last_fullsized":"2024-04-29T14:27:51.476169+0000","mapping_epoch":432,"log_start":"441'1700","ondisk_log_start":"441'1700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316211200,"num_objects":4825,"num_object_clones":0,"num_object_copies":9650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"494'11799","reported_seq":12015,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-29T14:27:51.072299+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:27:51.072299+0000","last_peered":"2024-04-29T14:27:51.072299+0000","last_clean":"2024-04-29T14:27:51.072299+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:27:51.072299+0000","last_undegraded":"2024-04-29T14:27:51.072299+0000","last_fullsized":"2024-04-29T14:27:51.072299+0000","mapping_epoch":432,"log_start":"441'1700","ondisk_log_start":"441'1700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315162624,"num_objects":4809,"num_object_clones":0,"num_object_copies":9618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11799,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"495'12096","reported_seq":19712,"reported_epoch":495,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:55.657044+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:27:55.657044+0000","last_peered":"2024-04-29T14:27:55.657044+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:27:55.657044+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:27:55.657044+0000","mapping_epoch":467,"log_start":"441'2000","ondisk_log_start":"441'2000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308150272,"num_objects":4702,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12096,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3698,"num_bytes_recovered":242352128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":4701},{"shards":"9,10","objects":1}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"494'11910","reported_seq":19191,"reported_epoch":494,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:51.530456+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:27:51.530456+0000","last_peered":"2024-04-29T14:27:51.530456+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:27:51.530456+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:27:51.530456+0000","mapping_epoch":467,"log_start":"441'1900","ondisk_log_start":"441'1900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318242816,"num_objects":4856,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3529,"num_bytes_recovered":231276544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":4837},{"shards":"7,10","objects":19}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"494'11875","reported_seq":12096,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-29T14:27:51.402618+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:27:51.402618+0000","last_peered":"2024-04-29T14:27:51.402618+0000","last_clean":"2024-04-29T14:27:51.402618+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:27:51.402618+0000","last_undegraded":"2024-04-29T14:27:51.402618+0000","last_fullsized":"2024-04-29T14:27:51.402618+0000","mapping_epoch":432,"log_start":"441'1800","ondisk_log_start":"441'1800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320143360,"num_objects":4885,"num_object_clones":0,"num_object_copies":9770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11875,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"495'11955","reported_seq":12278,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-04-29T14:27:55.306852+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:27:55.306852+0000","last_peered":"2024-04-29T14:27:55.306852+0000","last_clean":"2024-04-29T14:27:55.306852+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:27:55.306852+0000","last_undegraded":"2024-04-29T14:27:55.306852+0000","last_fullsized":"2024-04-29T14:27:55.306852+0000","mapping_epoch":432,"log_start":"441'1900","ondisk_log_start":"441'1900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:53.158953+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305070080,"num_objects":4655,"num_object_clones":0,"num_object_copies":9310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11955,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"495'11908","reported_seq":12193,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-04-29T14:27:55.323512+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:27:55.323512+0000","last_peered":"2024-04-29T14:27:55.323512+0000","last_clean":"2024-04-29T14:27:55.323512+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:27:55.323512+0000","last_undegraded":"2024-04-29T14:27:55.323512+0000","last_fullsized":"2024-04-29T14:27:55.323512+0000","mapping_epoch":432,"log_start":"441'1900","ondisk_log_start":"441'1900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306839552,"num_objects":4682,"num_object_clones":0,"num_object_copies":9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"494'11615","reported_seq":11839,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-29T14:27:51.402971+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:27:51.402971+0000","last_peered":"2024-04-29T14:27:51.402971+0000","last_clean":"2024-04-29T14:27:51.402971+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:27:51.402971+0000","last_undegraded":"2024-04-29T14:27:51.402971+0000","last_fullsized":"2024-04-29T14:27:51.402971+0000","mapping_epoch":432,"log_start":"441'1600","ondisk_log_start":"441'1600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316604416,"num_objects":4831,"num_object_clones":0,"num_object_copies":9662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"495'14814","reported_seq":15212,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-04-29T14:27:53.401116+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:27:53.401116+0000","last_peered":"2024-04-29T14:27:53.401116+0000","last_clean":"2024-04-29T14:27:53.401116+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:27:53.401116+0000","last_undegraded":"2024-04-29T14:27:53.401116+0000","last_fullsized":"2024-04-29T14:27:53.401116+0000","mapping_epoch":432,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312213504,"num_objects":4764,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11852,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"495'11906","reported_seq":19170,"reported_epoch":495,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:27:53.533268+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:27:53.533268+0000","last_peered":"2024-04-29T14:27:53.533268+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:27:53.533268+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:27:53.533268+0000","mapping_epoch":467,"log_start":"441'1900","ondisk_log_start":"441'1900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313917440,"num_objects":4790,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3561,"num_bytes_recovered":233373696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":3},{"shards":"10","objects":4787}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1231,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-29T14:27:47.214648+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:27:47.214648+0000","last_peered":"2024-04-29T14:27:47.214648+0000","last_clean":"2024-04-29T14:27:47.214648+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:27:47.214648+0000","last_undegraded":"2024-04-29T14:27:47.214648+0000","last_fullsized":"2024-04-29T14:27:47.214648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5002297368,"num_objects":76330,"num_object_clones":0,"num_object_copies":152660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76330,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190598,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17883,"num_bytes_recovered":1171980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11224227840,"data_stored":11224219696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135484,"kb_used_data":621172,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92609668,"statfs":{"total":95995035648,"available":94832300032,"internally_reserved":0,"allocated":636080128,"data_stored":635609456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":526645700},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383936,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335104,"kb_used_data":916468,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92410048,"statfs":{"total":95995035648,"available":94627889152,"internally_reserved":0,"allocated":938463232,"data_stored":937993899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":428538304},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645574,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1294268,"kb_used_data":902004,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92450884,"statfs":{"total":95995035648,"available":94669705216,"internally_reserved":0,"allocated":923652096,"data_stored":923182783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422878,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1421820,"kb_used_data":935284,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92323332,"statfs":{"total":95995035648,"available":94539091968,"internally_reserved":0,"allocated":957730816,"data_stored":957260164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528596,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1451260,"kb_used_data":896692,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92293892,"statfs":{"total":95995035648,"available":94508945408,"internally_reserved":0,"allocated":918212608,"data_stored":917743295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2621872,"kb_used_data":2171432,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91123280,"statfs":{"total":95995035648,"available":93310238720,"internally_reserved":0,"allocated":2223546368,"data_stored":2223083340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497106,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986463,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2532848,"kb_used_data":1865832,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91212304,"statfs":{"total":95995035648,"available":93401399296,"internally_reserved":0,"allocated":1910611968,"data_stored":1910148940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2219704320,"data_stored":2219704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1902641152,"data_stored":1902641152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":916783104,"data_stored":916783104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":951193600,"data_stored":951189528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":917176320,"data_stored":917176320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":932904960,"data_stored":932904960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":634130432,"data_stored":634126360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:27:58.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:27:58.327 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:27:58.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:58 smithi040 ceph-mon[39581]: pgmap v1311: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 652 op/s; 23416/149344 objects degraded (15.679%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:27:58.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:58 smithi121 ceph-mon[44533]: pgmap v1311: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 652 op/s; 23416/149344 objects degraded (15.679%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:27:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:58 smithi194 ceph-mon[44089]: pgmap v1311: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 652 op/s; 23416/149344 objects degraded (15.679%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:27:59.314 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:27:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:59.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:27:59.421 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:27:59.473 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:59.504 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:27:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:27:59.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:27:59.582 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1760032006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:59.582 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:27:59 smithi194 ceph-mon[44089]: from='client.36161 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:59.612 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:27:59.671 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:27:59.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1760032006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:59.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:27:59 smithi040 ceph-mon[39581]: from='client.36161 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1760032006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:27:59.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:27:59 smithi121 ceph-mon[44533]: from='client.36161 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:28:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:28:00.405 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:00.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:00 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:00.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:00 smithi040 ceph-mon[39581]: from='client.36161 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:28:00.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:00 smithi040 ceph-mon[39581]: osdmap e496: 12 total, 11 up, 7 in 2024-04-29T14:28:00.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:00 smithi040 ceph-mon[39581]: pgmap v1313: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 477 op/s; 23214/148178 objects degraded (15.666%); 9.7 MiB/s, 154 objects/s recovering 2024-04-29T14:28:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:00 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:00 smithi121 ceph-mon[44533]: from='client.36161 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:28:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:00 smithi121 ceph-mon[44533]: osdmap e496: 12 total, 11 up, 7 in 2024-04-29T14:28:00.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:00 smithi121 ceph-mon[44533]: pgmap v1313: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 477 op/s; 23214/148178 objects degraded (15.666%); 9.7 MiB/s, 154 objects/s recovering 2024-04-29T14:28:00.967 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:28:00.996 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:28:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:28:01.042 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:28:01.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:00 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:01.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:00 smithi194 ceph-mon[44089]: from='client.36161 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:28:01.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:00 smithi194 ceph-mon[44089]: osdmap e496: 12 total, 11 up, 7 in 2024-04-29T14:28:01.065 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:00 smithi194 ceph-mon[44089]: pgmap v1313: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 477 op/s; 23214/148178 objects degraded (15.666%); 9.7 MiB/s, 154 objects/s recovering 2024-04-29T14:28:01.067 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:28:01.090 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:28:01.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:28:01.142 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:28:01.170 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:28:01.196 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:28:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:28:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:28:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:28:01.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3703085892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:01.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3703085892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:28:02.009 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:28:02.064 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3703085892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: pgmap v1314: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 22398/142922 objects degraded (15.671%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:28:02.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 23214/148178 objects degraded (15.666%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:02.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:02 smithi040 ceph-mon[39581]: osdmap e497: 12 total, 11 up, 7 in 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:02.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: pgmap v1314: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 22398/142922 objects degraded (15.671%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:28:02.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 23214/148178 objects degraded (15.666%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:02.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:02 smithi121 ceph-mon[44533]: osdmap e497: 12 total, 11 up, 7 in 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:03.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: pgmap v1314: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 22398/142922 objects degraded (15.671%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:28:03.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 23214/148178 objects degraded (15.666%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:03.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:02 smithi194 ceph-mon[44089]: osdmap e497: 12 total, 11 up, 7 in 2024-04-29T14:28:03.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:28:04.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:28:04.058 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:28:04.364 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:28:04.372 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:04.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:04.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:04.510 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:04.533 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:04.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:04.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:04.670 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:04.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:04.803 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:04.884 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:04.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:05.030 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:05.056 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:05 smithi040 ceph-mon[39581]: pgmap v1316: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 684 op/s; 22171/141702 objects degraded (15.646%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:28:05.056 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:05 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:28:05.056 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:05.056 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:05.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:05 smithi194 ceph-mon[44089]: pgmap v1316: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 684 op/s; 22171/141702 objects degraded (15.646%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:28:05.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:05 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:28:05.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:05.143 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:05.143 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:05.192 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:05.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:05.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:05.372 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:28:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:05 smithi121 ceph-mon[44533]: pgmap v1316: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 684 op/s; 22171/141702 objects degraded (15.646%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:28:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:05 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:28:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:05.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:05.731 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1316,"stamp":"2024-04-29T14:28:03.987200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4643684920,"num_objects":70851,"num_object_clones":0,"num_object_copies":141702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70851,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":196260,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19778,"num_bytes_recovered":1297762432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11159536,"kb_used_data":7692196,"kb_used_omap":10,"kb_used_meta":3466805,"kb_avail":645056528,"statfs":{"total":671965249536,"available":660537884672,"internally_reserved":0,"allocated":7876808704,"data_stored":7873457945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3550008414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,6,1,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-359202816,"num_objects":-5481,"num_object_clones":0,"num_object_copies":-10962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1887,"num_bytes_recovered":123666432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001477"},"pg_stats":[{"pgid":"5.f","version":"496'12498","reported_seq":20762,"reported_epoch":496,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:01.488030+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:28:01.488030+0000","last_peered":"2024-04-29T14:28:01.488030+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:28:01.488030+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:28:01.488030+0000","mapping_epoch":467,"log_start":"443'2400","ondisk_log_start":"443'2400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292683776,"num_objects":4466,"num_object_clones":0,"num_object_copies":8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12498,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4018,"num_bytes_recovered":263323648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":4464},{"shards":"4,11","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"496'12326","reported_seq":12542,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:00.609577+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:28:00.609577+0000","last_peered":"2024-04-29T14:28:00.609577+0000","last_clean":"2024-04-29T14:28:00.609577+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:28:00.609577+0000","last_undegraded":"2024-04-29T14:28:00.609577+0000","last_fullsized":"2024-04-29T14:28:00.609577+0000","mapping_epoch":432,"log_start":"442'2300","ondisk_log_start":"442'2300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292814848,"num_objects":4468,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"496'12285","reported_seq":20385,"reported_epoch":496,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:28:01.136904+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:28:01.136904+0000","last_peered":"2024-04-29T14:28:01.136904+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:28:01.136904+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:28:01.136904+0000","mapping_epoch":468,"log_start":"441'2200","ondisk_log_start":"441'2200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288882712,"num_objects":4409,"num_object_clones":0,"num_object_copies":8818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12285,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3940,"num_bytes_recovered":258211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":3},{"shards":"11","objects":4406}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"496'12354","reported_seq":12583,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:01.499310+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:28:01.499310+0000","last_peered":"2024-04-29T14:28:01.499310+0000","last_clean":"2024-04-29T14:28:01.499310+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:28:01.499310+0000","last_undegraded":"2024-04-29T14:28:01.499310+0000","last_fullsized":"2024-04-29T14:28:01.499310+0000","mapping_epoch":432,"log_start":"442'2300","ondisk_log_start":"442'2300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290848768,"num_objects":4438,"num_object_clones":0,"num_object_copies":8876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12354,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"496'12265","reported_seq":12488,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:00.673617+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:28:00.673617+0000","last_peered":"2024-04-29T14:28:00.673617+0000","last_clean":"2024-04-29T14:28:00.673617+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:28:00.673617+0000","last_undegraded":"2024-04-29T14:28:00.673617+0000","last_fullsized":"2024-04-29T14:28:00.673617+0000","mapping_epoch":432,"log_start":"441'2200","ondisk_log_start":"441'2200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288423936,"num_objects":4401,"num_object_clones":0,"num_object_copies":8802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"497'14509","reported_seq":14864,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-29T14:28:03.586742+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:28:03.586742+0000","last_peered":"2024-04-29T14:28:03.586742+0000","last_clean":"2024-04-29T14:28:03.586742+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:28:03.586742+0000","last_undegraded":"2024-04-29T14:28:03.586742+0000","last_fullsized":"2024-04-29T14:28:03.586742+0000","mapping_epoch":432,"log_start":"447'4500","ondisk_log_start":"447'4500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281280512,"num_objects":4292,"num_object_clones":0,"num_object_copies":8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12254,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"496'12158","reported_seq":12383,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:01.460189+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:28:01.460189+0000","last_peered":"2024-04-29T14:28:01.460189+0000","last_clean":"2024-04-29T14:28:01.460189+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:28:01.460189+0000","last_undegraded":"2024-04-29T14:28:01.460189+0000","last_fullsized":"2024-04-29T14:28:01.460189+0000","mapping_epoch":432,"log_start":"441'2100","ondisk_log_start":"441'2100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289931264,"num_objects":4424,"num_object_clones":0,"num_object_copies":8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12158,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"496'12240","reported_seq":12460,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:01.072847+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:28:01.072847+0000","last_peered":"2024-04-29T14:28:01.072847+0000","last_clean":"2024-04-29T14:28:01.072847+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:28:01.072847+0000","last_undegraded":"2024-04-29T14:28:01.072847+0000","last_fullsized":"2024-04-29T14:28:01.072847+0000","mapping_epoch":432,"log_start":"441'2200","ondisk_log_start":"441'2200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286261248,"num_objects":4368,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12240,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"496'12281","reported_seq":20271,"reported_epoch":496,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:00.689593+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:28:00.689593+0000","last_peered":"2024-04-29T14:28:00.689593+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:28:00.689593+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:28:00.689593+0000","mapping_epoch":467,"log_start":"441'2200","ondisk_log_start":"441'2200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296026112,"num_objects":4517,"num_object_clones":0,"num_object_copies":9034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3884,"num_bytes_recovered":254541824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":4515},{"shards":"9,10","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"496'12314","reported_seq":20417,"reported_epoch":496,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:01.549149+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:28:01.549149+0000","last_peered":"2024-04-29T14:28:01.549149+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:28:01.549149+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:28:01.549149+0000","mapping_epoch":467,"log_start":"442'2300","ondisk_log_start":"442'2300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291766272,"num_objects":4452,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12314,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3938,"num_bytes_recovered":258080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":4428},{"shards":"7,10","objects":24}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"496'12267","reported_seq":12492,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:01.443870+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:28:01.443870+0000","last_peered":"2024-04-29T14:28:01.443870+0000","last_clean":"2024-04-29T14:28:01.443870+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:28:01.443870+0000","last_undegraded":"2024-04-29T14:28:01.443870+0000","last_fullsized":"2024-04-29T14:28:01.443870+0000","mapping_epoch":432,"log_start":"441'2200","ondisk_log_start":"441'2200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294453248,"num_objects":4493,"num_object_clones":0,"num_object_copies":8986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"496'12143","reported_seq":12468,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:00.206882+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:28:00.206882+0000","last_peered":"2024-04-29T14:28:00.206882+0000","last_clean":"2024-04-29T14:28:00.206882+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:28:00.206882+0000","last_undegraded":"2024-04-29T14:28:00.206882+0000","last_fullsized":"2024-04-29T14:28:00.206882+0000","mapping_epoch":432,"log_start":"441'2100","ondisk_log_start":"441'2100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:53.158953+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292749312,"num_objects":4467,"num_object_clones":0,"num_object_copies":8934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12143,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"496'12087","reported_seq":12374,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:00.298464+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:28:00.298464+0000","last_peered":"2024-04-29T14:28:00.298464+0000","last_clean":"2024-04-29T14:28:00.298464+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:28:00.298464+0000","last_undegraded":"2024-04-29T14:28:00.298464+0000","last_fullsized":"2024-04-29T14:28:00.298464+0000","mapping_epoch":432,"log_start":"441'2000","ondisk_log_start":"441'2000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295108608,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"496'12021","reported_seq":12249,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:28:01.445896+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:28:01.445896+0000","last_peered":"2024-04-29T14:28:01.445896+0000","last_clean":"2024-04-29T14:28:01.445896+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:28:01.445896+0000","last_undegraded":"2024-04-29T14:28:01.445896+0000","last_fullsized":"2024-04-29T14:28:01.445896+0000","mapping_epoch":432,"log_start":"441'2000","ondisk_log_start":"441'2000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289996800,"num_objects":4425,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12021,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"497'15213","reported_seq":15613,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-29T14:28:03.555038+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:28:03.555038+0000","last_peered":"2024-04-29T14:28:03.555038+0000","last_clean":"2024-04-29T14:28:03.555038+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:28:03.555038+0000","last_undegraded":"2024-04-29T14:28:03.555038+0000","last_fullsized":"2024-04-29T14:28:03.555038+0000","mapping_epoch":432,"log_start":"450'5200","ondisk_log_start":"450'5200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286064640,"num_objects":4365,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12251,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"497'12335","reported_seq":20461,"reported_epoch":497,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:03.586484+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:28:03.586484+0000","last_peered":"2024-04-29T14:28:03.586484+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:28:03.586484+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:28:03.586484+0000","mapping_epoch":467,"log_start":"443'2300","ondisk_log_start":"443'2300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285802496,"num_objects":4361,"num_object_clones":0,"num_object_copies":8722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12335,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3990,"num_bytes_recovered":261488640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":3},{"shards":"10","objects":4358}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1235,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-04-29T14:27:59.607960+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:27:59.607960+0000","last_peered":"2024-04-29T14:27:59.607960+0000","last_clean":"2024-04-29T14:27:59.607960+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:27:59.607960+0000","last_undegraded":"2024-04-29T14:27:59.607960+0000","last_fullsized":"2024-04-29T14:27:59.607960+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4643094552,"num_objects":70849,"num_object_clones":0,"num_object_copies":141698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196079,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19770,"num_bytes_recovered":1295646720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10591477760,"data_stored":10591469616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160796,"ondisk_log_size":160796,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122301,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090776,"kb_used_data":568076,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92654376,"statfs":{"total":95995035648,"available":94878081024,"internally_reserved":0,"allocated":581709824,"data_stored":581226582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":535165380},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383938,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251288,"kb_used_data":840716,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92493864,"statfs":{"total":95995035648,"available":94713716736,"internally_reserved":0,"allocated":860893184,"data_stored":860411281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":420018624},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645575,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1254028,"kb_used_data":861888,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92491124,"statfs":{"total":95995035648,"available":94710910976,"internally_reserved":0,"allocated":882573312,"data_stored":882097714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422879,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234256,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335512,"kb_used_data":857292,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92409640,"statfs":{"total":95995035648,"available":94627471360,"internally_reserved":0,"allocated":877867008,"data_stored":877383786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":489683402},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528597,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1415436,"kb_used_data":860864,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92329716,"statfs":{"total":95995035648,"available":94545629184,"internally_reserved":0,"allocated":881524736,"data_stored":881049138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575150,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2440904,"kb_used_data":1990464,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91304248,"statfs":{"total":95995035648,"available":93495549952,"internally_reserved":0,"allocated":2038235136,"data_stored":2037759538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497108,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986465,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2371592,"kb_used_data":1712896,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":91373560,"statfs":{"total":95995035648,"available":93566525440,"internally_reserved":0,"allocated":1754005504,"data_stored":1753529906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2036137984,"data_stored":2036137984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1744896000,"data_stored":1744896000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":880345088,"data_stored":880345088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":871960576,"data_stored":871956504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":877264896,"data_stored":877264896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":855900160,"data_stored":855900160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":575279104,"data_stored":575275032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.f is not active+clean 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.f', 'version': "496'12498", 'reported_seq': 20762, 'reported_epoch': 496, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:01.488030+0000', 'last_change': '2024-04-29T14:26:16.020854+0000', 'last_active': '2024-04-29T14:28:01.488030+0000', 'last_peered': '2024-04-29T14:28:01.488030+0000', 'last_clean': '2024-04-29T14:25:54.182894+0000', 'last_became_active': '2024-04-29T14:26:16.010174+0000', 'last_became_peered': '2024-04-29T14:26:16.010174+0000', 'last_unstale': '2024-04-29T14:28:01.488030+0000', 'last_undegraded': '2024-04-29T14:26:15.821893+0000', 'last_fullsized': '2024-04-29T14:28:01.488030+0000', 'mapping_epoch': 467, 'log_start': "443'2400", 'ondisk_log_start': "443'2400", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'7968", 'last_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'objects_scrubbed': 7919, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2114, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 292683776, 'num_objects': 4466, 'num_object_clones': 0, 'num_object_copies': 8932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4464, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12498, 'num_write_kb': 542848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4018, 'num_bytes_recovered': 263323648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 11], 'acting': [4, 11], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 4464}, {'shards': '4,11', 'objects': 2}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.d is not active+clean 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.d', 'version': "496'12285", 'reported_seq': 20385, 'reported_epoch': 496, 'state': 'active+recovering+degraded+remapped', 'last_fresh': '2024-04-29T14:28:01.136904+0000', 'last_change': '2024-04-29T14:26:17.048329+0000', 'last_active': '2024-04-29T14:28:01.136904+0000', 'last_peered': '2024-04-29T14:28:01.136904+0000', 'last_clean': '2024-04-29T14:25:54.192906+0000', 'last_became_active': '2024-04-29T14:26:17.022020+0000', 'last_became_peered': '2024-04-29T14:26:17.022020+0000', 'last_unstale': '2024-04-29T14:28:01.136904+0000', 'last_undegraded': '2024-04-29T14:26:16.808487+0000', 'last_fullsized': '2024-04-29T14:28:01.136904+0000', 'mapping_epoch': 468, 'log_start': "441'2200", 'ondisk_log_start': "441'2200", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "454'7099", 'last_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'objects_scrubbed': 7053, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1409, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 288882712, 'num_objects': 4409, 'num_object_clones': 0, 'num_object_copies': 8818, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4406, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4409, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12285, 'num_write_kb': 534145, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3940, 'num_bytes_recovered': 258211840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 11], 'acting': [11, 7], 'avail_no_missing': ['11'], 'object_location_counts': [{'shards': '7,11', 'objects': 3}, {'shards': '11', 'objects': 4406}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 11, 'purged_snaps': []} 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.7 is not active+clean 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.7', 'version': "496'12281", 'reported_seq': 20271, 'reported_epoch': 496, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:00.689593+0000', 'last_change': '2024-04-29T14:26:16.052811+0000', 'last_active': '2024-04-29T14:28:00.689593+0000', 'last_peered': '2024-04-29T14:28:00.689593+0000', 'last_clean': '2024-04-29T14:25:54.182628+0000', 'last_became_active': '2024-04-29T14:26:16.038648+0000', 'last_became_peered': '2024-04-29T14:26:16.038648+0000', 'last_unstale': '2024-04-29T14:28:00.689593+0000', 'last_undegraded': '2024-04-29T14:26:15.818605+0000', 'last_fullsized': '2024-04-29T14:28:00.689593+0000', 'mapping_epoch': 467, 'log_start': "441'2200", 'ondisk_log_start': "441'2200", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "455'7479", 'last_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'objects_scrubbed': 7424, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1679, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 296026112, 'num_objects': 4517, 'num_object_clones': 0, 'num_object_copies': 9034, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4515, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12281, 'num_write_kb': 537536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3884, 'num_bytes_recovered': 254541824, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 10], 'acting': [9, 10], 'avail_no_missing': ['9'], 'object_location_counts': [{'shards': '9', 'objects': 4515}, {'shards': '9,10', 'objects': 2}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "496'12314", 'reported_seq': 20417, 'reported_epoch': 496, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:01.549149+0000', 'last_change': '2024-04-29T14:26:16.126893+0000', 'last_active': '2024-04-29T14:28:01.549149+0000', 'last_peered': '2024-04-29T14:28:01.549149+0000', 'last_clean': '2024-04-29T14:25:53.790194+0000', 'last_became_active': '2024-04-29T14:26:16.054003+0000', 'last_became_peered': '2024-04-29T14:26:16.054003+0000', 'last_unstale': '2024-04-29T14:28:01.549149+0000', 'last_undegraded': '2024-04-29T14:26:15.820366+0000', 'last_fullsized': '2024-04-29T14:28:01.549149+0000', 'mapping_epoch': 467, 'log_start': "442'2300", 'ondisk_log_start': "442'2300", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'8070", 'last_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'objects_scrubbed': 8029, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1689, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 291766272, 'num_objects': 4452, 'num_object_clones': 0, 'num_object_copies': 8904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4428, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12314, 'num_write_kb': 536512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3938, 'num_bytes_recovered': 258080768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 10], 'acting': [7, 10], 'avail_no_missing': ['7'], 'object_location_counts': [{'shards': '7', 'objects': 4428}, {'shards': '7,10', 'objects': 24}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-29T14:28:05.732 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "497'12335", 'reported_seq': 20461, 'reported_epoch': 497, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:03.586484+0000', 'last_change': '2024-04-29T14:26:15.998964+0000', 'last_active': '2024-04-29T14:28:03.586484+0000', 'last_peered': '2024-04-29T14:28:03.586484+0000', 'last_clean': '2024-04-29T14:25:59.019434+0000', 'last_became_active': '2024-04-29T14:26:15.982158+0000', 'last_became_peered': '2024-04-29T14:26:15.982158+0000', 'last_unstale': '2024-04-29T14:28:03.586484+0000', 'last_undegraded': '2024-04-29T14:26:15.829402+0000', 'last_fullsized': '2024-04-29T14:28:03.586484+0000', 'mapping_epoch': 467, 'log_start': "443'2300", 'ondisk_log_start': "443'2300", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 285802496, 'num_objects': 4361, 'num_object_clones': 0, 'num_object_copies': 8722, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4358, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12335, 'num_write_kb': 534272, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3990, 'num_bytes_recovered': 261488640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 0], 'acting': [10, 0], 'avail_no_missing': ['10'], 'object_location_counts': [{'shards': '0,10', 'objects': 3}, {'shards': '10', 'objects': 4358}], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-04-29T14:28:05.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:28:06.182 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:06.208 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:06.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:06.313 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:06 smithi194 ceph-mon[44089]: from='client.16581 v1:172.21.15.40:0/2871533255' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:06.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:06 smithi194 ceph-mon[44089]: osdmap e498: 12 total, 11 up, 7 in 2024-04-29T14:28:06.339 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:06 smithi040 ceph-mon[39581]: from='client.16581 v1:172.21.15.40:0/2871533255' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:06.339 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:06 smithi040 ceph-mon[39581]: osdmap e498: 12 total, 11 up, 7 in 2024-04-29T14:28:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:06 smithi121 ceph-mon[44533]: from='client.16581 v1:172.21.15.40:0/2871533255' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:06.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:06 smithi121 ceph-mon[44533]: osdmap e498: 12 total, 11 up, 7 in 2024-04-29T14:28:06.466 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:06.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:06.571 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:07.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:07 smithi194 ceph-mon[44089]: pgmap v1318: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 21943/139648 objects degraded (15.713%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:28:07.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:07 smithi040 ceph-mon[39581]: pgmap v1318: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 21943/139648 objects degraded (15.713%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:28:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:07 smithi121 ceph-mon[44533]: pgmap v1318: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 21943/139648 objects degraded (15.713%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:28:07.426 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:07.606 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:07.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:07.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:28:07.776 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:08.221 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:28:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:08 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 21943/139648 objects degraded (15.713%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:08 smithi194 ceph-mon[44089]: osdmap e499: 12 total, 11 up, 7 in 2024-04-29T14:28:08.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/239984613' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:08 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 21943/139648 objects degraded (15.713%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:08 smithi040 ceph-mon[39581]: osdmap e499: 12 total, 11 up, 7 in 2024-04-29T14:28:08.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/239984613' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:08 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 21943/139648 objects degraded (15.713%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:08 smithi121 ceph-mon[44533]: osdmap e499: 12 total, 11 up, 7 in 2024-04-29T14:28:08.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/239984613' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:09.066 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:28:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:09.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:28:09.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:28:09.230 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:09.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:09 smithi194 ceph-mon[44089]: pgmap v1320: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 21325/136494 objects degraded (15.623%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:28:09.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:09 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:09.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/239984613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:09.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:09 smithi194 ceph-mon[44089]: osdmap e500: 12 total, 11 up, 7 in 2024-04-29T14:28:09.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:09 smithi040 ceph-mon[39581]: pgmap v1320: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 21325/136494 objects degraded (15.623%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:28:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:09 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/239984613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:09.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:09 smithi040 ceph-mon[39581]: osdmap e500: 12 total, 11 up, 7 in 2024-04-29T14:28:09.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:09 smithi121 ceph-mon[44533]: pgmap v1320: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 21325/136494 objects degraded (15.623%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:28:09.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:09 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:09.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/239984613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:09.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:09 smithi121 ceph-mon[44533]: osdmap e500: 12 total, 11 up, 7 in 2024-04-29T14:28:10.498 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:28:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:10.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:28:10.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:10.681 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:11.083 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:11.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:11.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:11.223 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:11.261 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:11.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:11.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:11 smithi194 ceph-mon[44089]: pgmap v1322: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 534 op/s; 21125/135286 objects degraded (15.615%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:28:11.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:11 smithi040 ceph-mon[39581]: pgmap v1322: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 534 op/s; 21125/135286 objects degraded (15.615%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:28:11.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:28:11.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:11 smithi121 ceph-mon[44533]: pgmap v1322: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 534 op/s; 21125/135286 objects degraded (15.615%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:28:11.406 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:11.431 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:11.432 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21125/135286 objects degraded (15.615%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1565,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":500,"num_osds":12,"num_up_osds":11,"osd_up_since":1714400760,"num_in_osds":7,"osd_in_since":1714400774,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovering+degraded","count":4},{"state_name":"active+recovering+degraded+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":67643,"data_bytes":4433445432,"bytes_used":11108704256,"bytes_avail":660856545280,"bytes_total":671965249536,"degraded_objects":21125,"degraded_total":135286,"degraded_ratio":0.15615067338822938,"recovering_objects_per_sec":174,"recovering_bytes_per_sec":11423071,"recovering_keys_per_sec":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":534},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":40,"modified":"2024-04-29T14:24:06.771444+0000","services":{}},"progress_events":{}} 2024-04-29T14:28:11.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:28:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:11.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:28:11.542 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:11.609 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:11.754 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:11.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:11.843 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:28:11.844 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:28:11.860 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:11.927 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:11.968 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:11.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:12.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:28:12.129 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:12.169 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:28:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:12.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:28:12.217 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:28:12.242 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:28:12.264 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:28:12.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:12.290 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:28:12.311 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:12.314 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:28:12.343 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:28:12.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1384039111' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:28:12.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:12.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:12.350 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:12.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1384039111' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:28:12.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:12.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:12.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:12.367 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:28:12.391 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:28:12.421 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:28:12.446 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:28:12.453 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:12.470 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:28:12.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:28:12.478 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:28:12.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1384039111' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:28:12.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:12.510 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:12.513 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:12.711 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:28:12.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: pgmap v1323: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 20253/129722 objects degraded (15.613%); 17 MiB/s, 276 objects/s recovering 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2848360410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 21125/135286 objects degraded (15.615%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osdmap e501: 12 total, 11 up, 7 in 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: pgmap v1323: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 20253/129722 objects degraded (15.613%); 17 MiB/s, 276 objects/s recovering 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2848360410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 21125/135286 objects degraded (15.615%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osdmap e501: 12 total, 11 up, 7 in 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.434 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:28:13.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:28:13.478 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: pgmap v1323: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 4.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 20253/129722 objects degraded (15.613%); 17 MiB/s, 276 objects/s recovering 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2848360410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 21125/135286 objects degraded (15.615%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osdmap e501: 12 total, 11 up, 7 in 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:14.392 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:14.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:14.498 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:14.529 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:15.080 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:15.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:15.185 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:28:15.232 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:15.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:15 smithi040 ceph-mon[39581]: pgmap v1325: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 590 op/s; 20058/128486 objects degraded (15.611%); 12 MiB/s, 186 objects/s recovering 2024-04-29T14:28:15.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:28:15.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:28:15.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:15 smithi121 ceph-mon[44533]: pgmap v1325: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 590 op/s; 20058/128486 objects degraded (15.611%); 12 MiB/s, 186 objects/s recovering 2024-04-29T14:28:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:28:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:28:15.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:15.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:28:15.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:15 smithi194 ceph-mon[44089]: pgmap v1325: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 590 op/s; 20058/128486 objects degraded (15.611%); 12 MiB/s, 186 objects/s recovering 2024-04-29T14:28:15.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:28:15.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:28:15.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:16.950 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:28:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:16.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:28:16.960 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:16.980 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:16.983 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:28:16.993 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:28:17.058 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:17.144 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:17.174 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:17.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:17.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:17.271 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:17 smithi040 ceph-mon[39581]: pgmap v1326: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 19855/126262 objects degraded (15.725%); 11 MiB/s, 183 objects/s recovering 2024-04-29T14:28:17.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:17.318 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:17.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:17 smithi194 ceph-mon[44089]: pgmap v1326: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 19855/126262 objects degraded (15.725%); 11 MiB/s, 183 objects/s recovering 2024-04-29T14:28:17.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:17.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:17.350 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:17 smithi121 ceph-mon[44533]: pgmap v1326: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 19855/126262 objects degraded (15.725%); 11 MiB/s, 183 objects/s recovering 2024-04-29T14:28:17.451 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:17.471 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:28:17.516 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:17.798 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:17.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:17.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:17.944 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:18.073 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:18.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:18.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:28:18.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:18.212 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:18 smithi040 ceph-mon[39581]: from='client.16605 v1:172.21.15.40:0/495697430' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:18.212 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/884354043' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:18.212 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:18 smithi040 ceph-mon[39581]: from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:18.212 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:18 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 19855/126262 objects degraded (15.725%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:18.215 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:18.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:18 smithi121 ceph-mon[44533]: from='client.16605 v1:172.21.15.40:0/495697430' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:18.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/884354043' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:18.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:18 smithi121 ceph-mon[44533]: from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:18.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:18 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 19855/126262 objects degraded (15.725%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:18.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:18.490 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1326,"stamp":"2024-04-29T14:28:15.989371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4137747000,"num_objects":63131,"num_object_clones":0,"num_object_copies":126262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63131,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":203980,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22094,"num_bytes_recovered":1449543808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10458500,"kb_used_data":6905612,"kb_used_omap":10,"kb_used_meta":3552821,"kb_avail":645757564,"statfs":{"total":671965249536,"available":661255745536,"internally_reserved":0,"allocated":7071346688,"data_stored":7067807729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3638088798},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-335282176,"num_objects":-5116,"num_object_clones":0,"num_object_copies":-10232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1470,"num_bytes_recovered":96337920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001428"},"pg_stats":[{"pgid":"5.f","version":"500'12946","reported_seq":22118,"reported_epoch":500,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:11.498016+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:28:11.498016+0000","last_peered":"2024-04-29T14:28:11.498016+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:28:11.498016+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:28:11.498016+0000","mapping_epoch":467,"log_start":"443'2900","ondisk_log_start":"443'2900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263323648,"num_objects":4018,"num_object_clones":0,"num_object_copies":8036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12946,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4468,"num_bytes_recovered":292814848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":4014},{"shards":"4,11","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"501'12994","reported_seq":13220,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-04-29T14:28:15.671696+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:28:15.671696+0000","last_peered":"2024-04-29T14:28:15.671696+0000","last_clean":"2024-04-29T14:28:15.671696+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:28:15.671696+0000","last_undegraded":"2024-04-29T14:28:15.671696+0000","last_fullsized":"2024-04-29T14:28:15.671696+0000","mapping_epoch":432,"log_start":"443'2900","ondisk_log_start":"443'2900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249036800,"num_objects":3800,"num_object_clones":0,"num_object_copies":7600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12994,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"500'12703","reported_seq":21645,"reported_epoch":500,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:28:11.176816+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:28:11.176816+0000","last_peered":"2024-04-29T14:28:11.176816+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:28:11.176816+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:28:11.176816+0000","mapping_epoch":468,"log_start":"443'2700","ondisk_log_start":"443'2700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261488664,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12703,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4357,"num_bytes_recovered":285540352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":2},{"shards":"11","objects":3989}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"500'12758","reported_seq":12995,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-29T14:28:11.474555+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:28:11.474555+0000","last_peered":"2024-04-29T14:28:11.474555+0000","last_clean":"2024-04-29T14:28:11.474555+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:28:11.474555+0000","last_undegraded":"2024-04-29T14:28:11.474555+0000","last_fullsized":"2024-04-29T14:28:11.474555+0000","mapping_epoch":432,"log_start":"443'2700","ondisk_log_start":"443'2700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264372224,"num_objects":4034,"num_object_clones":0,"num_object_copies":8068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12758,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"501'12884","reported_seq":13117,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-04-29T14:28:15.726820+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:28:15.726820+0000","last_peered":"2024-04-29T14:28:15.726820+0000","last_clean":"2024-04-29T14:28:15.726820+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:28:15.726820+0000","last_undegraded":"2024-04-29T14:28:15.726820+0000","last_fullsized":"2024-04-29T14:28:15.726820+0000","mapping_epoch":432,"log_start":"443'2800","ondisk_log_start":"443'2800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247857152,"num_objects":3782,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12884,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"501'14929","reported_seq":15293,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-04-29T14:28:13.548314+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:28:13.548314+0000","last_peered":"2024-04-29T14:28:13.548314+0000","last_clean":"2024-04-29T14:28:13.548314+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:28:13.548314+0000","last_undegraded":"2024-04-29T14:28:13.548314+0000","last_fullsized":"2024-04-29T14:28:13.548314+0000","mapping_epoch":432,"log_start":"449'4900","ondisk_log_start":"449'4900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253755392,"num_objects":3872,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"500'12561","reported_seq":12794,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-29T14:28:11.544666+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:28:11.544666+0000","last_peered":"2024-04-29T14:28:11.544666+0000","last_clean":"2024-04-29T14:28:11.544666+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:28:11.544666+0000","last_undegraded":"2024-04-29T14:28:11.544666+0000","last_fullsized":"2024-04-29T14:28:11.544666+0000","mapping_epoch":432,"log_start":"443'2500","ondisk_log_start":"443'2500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12561,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"500'12654","reported_seq":12881,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-29T14:28:11.155706+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:28:11.155706+0000","last_peered":"2024-04-29T14:28:11.155706+0000","last_clean":"2024-04-29T14:28:11.155706+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:28:11.155706+0000","last_undegraded":"2024-04-29T14:28:11.155706+0000","last_fullsized":"2024-04-29T14:28:11.155706+0000","mapping_epoch":432,"log_start":"443'2600","ondisk_log_start":"443'2600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259129344,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12654,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"501'12929","reported_seq":22235,"reported_epoch":501,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:15.736382+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:28:15.736382+0000","last_peered":"2024-04-29T14:28:15.736382+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:28:15.736382+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:28:15.736382+0000","mapping_epoch":467,"log_start":"443'2900","ondisk_log_start":"443'2900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253558784,"num_objects":3869,"num_object_clones":0,"num_object_copies":7738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12929,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":297336832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":3862},{"shards":"9,10","objects":7}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"500'12722","reported_seq":21635,"reported_epoch":500,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:11.556606+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:28:11.556606+0000","last_peered":"2024-04-29T14:28:11.556606+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:28:11.556606+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:28:11.556606+0000","mapping_epoch":467,"log_start":"443'2700","ondisk_log_start":"443'2700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12722,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4339,"num_bytes_recovered":284360704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":4027},{"shards":"7,10","objects":17}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"500'12674","reported_seq":12907,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-29T14:28:11.482402+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:28:11.482402+0000","last_peered":"2024-04-29T14:28:11.482402+0000","last_clean":"2024-04-29T14:28:11.482402+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:28:11.482402+0000","last_undegraded":"2024-04-29T14:28:11.482402+0000","last_fullsized":"2024-04-29T14:28:11.482402+0000","mapping_epoch":432,"log_start":"443'2600","ondisk_log_start":"443'2600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267780096,"num_objects":4086,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"501'12775","reported_seq":13110,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-04-29T14:28:15.265531+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:28:15.265531+0000","last_peered":"2024-04-29T14:28:15.265531+0000","last_clean":"2024-04-29T14:28:15.265531+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:28:15.265531+0000","last_undegraded":"2024-04-29T14:28:15.265531+0000","last_fullsized":"2024-04-29T14:28:15.265531+0000","mapping_epoch":432,"log_start":"443'2700","ondisk_log_start":"443'2700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:53.158953+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251330560,"num_objects":3835,"num_object_clones":0,"num_object_copies":7670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"501'12710","reported_seq":13007,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-04-29T14:28:15.289855+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:28:15.289855+0000","last_peered":"2024-04-29T14:28:15.289855+0000","last_clean":"2024-04-29T14:28:15.289855+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:28:15.289855+0000","last_undegraded":"2024-04-29T14:28:15.289855+0000","last_fullsized":"2024-04-29T14:28:15.289855+0000","mapping_epoch":432,"log_start":"443'2700","ondisk_log_start":"443'2700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254279680,"num_objects":3880,"num_object_clones":0,"num_object_copies":7760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12710,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"500'12427","reported_seq":12663,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-29T14:28:11.471032+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:28:11.471032+0000","last_peered":"2024-04-29T14:28:11.471032+0000","last_clean":"2024-04-29T14:28:11.471032+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:28:11.471032+0000","last_undegraded":"2024-04-29T14:28:11.471032+0000","last_fullsized":"2024-04-29T14:28:11.471032+0000","mapping_epoch":432,"log_start":"443'2400","ondisk_log_start":"443'2400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263389184,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12427,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"501'15641","reported_seq":16052,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-04-29T14:28:13.616857+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:28:13.616857+0000","last_peered":"2024-04-29T14:28:13.616857+0000","last_clean":"2024-04-29T14:28:13.616857+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:28:13.616857+0000","last_undegraded":"2024-04-29T14:28:13.616857+0000","last_fullsized":"2024-04-29T14:28:13.616857+0000","mapping_epoch":432,"log_start":"452'5600","ondisk_log_start":"452'5600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258015232,"num_objects":3937,"num_object_clones":0,"num_object_copies":7874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12679,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"501'12709","reported_seq":21633,"reported_epoch":501,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:13.616352+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:28:13.616352+0000","last_peered":"2024-04-29T14:28:13.616352+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:28:13.616352+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:28:13.616352+0000","mapping_epoch":467,"log_start":"443'2700","ondisk_log_start":"443'2700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261292032,"num_objects":3987,"num_object_clones":0,"num_object_copies":7974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12709,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4385,"num_bytes_recovered":287375360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":24},{"shards":"10","objects":3963}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1243,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-29T14:28:08.253418+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:28:08.253418+0000","last_peered":"2024-04-29T14:28:08.253418+0000","last_clean":"2024-04-29T14:28:08.253418+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:28:08.253418+0000","last_undegraded":"2024-04-29T14:28:08.253418+0000","last_fullsized":"2024-04-29T14:28:08.253418+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4137156632,"num_objects":63129,"num_object_clones":0,"num_object_copies":126258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63129,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203799,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22086,"num_bytes_recovered":1447428096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9776668672,"data_stored":9776660528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160716,"ondisk_log_size":160716,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122303,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1042372,"kb_used_data":515580,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92702780,"statfs":{"total":95995035648,"available":94927646720,"internally_reserved":0,"allocated":527953920,"data_stored":527445588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":539425220},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383940,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1186500,"kb_used_data":763836,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":92558652,"statfs":{"total":95995035648,"available":94780059648,"internally_reserved":0,"allocated":782168064,"data_stored":781661071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":432798144},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645578,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157572,"kb_used_data":739068,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92587580,"statfs":{"total":95995035648,"available":94809681920,"internally_reserved":0,"allocated":756805632,"data_stored":756298659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":428538304},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234258,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286532,"kb_used_data":781948,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92458620,"statfs":{"total":95995035648,"available":94677626880,"internally_reserved":0,"allocated":800714752,"data_stored":800206440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":516684234},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528600,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1303940,"kb_used_data":741052,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92441212,"statfs":{"total":95995035648,"available":94659801088,"internally_reserved":0,"allocated":758837248,"data_stored":758330275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":576387526},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2255800,"kb_used_data":1809520,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91489352,"statfs":{"total":95995035648,"available":93685096448,"internally_reserved":0,"allocated":1852948480,"data_stored":1852447792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497110,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986467,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2225784,"kb_used_data":1554608,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91519368,"statfs":{"total":95995035648,"available":93715832832,"internally_reserved":0,"allocated":1591918592,"data_stored":1591417904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1848967168,"data_stored":1848967168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1582891008,"data_stored":1582891008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":757530624,"data_stored":757530624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":791744512,"data_stored":791740440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":750321664,"data_stored":750321664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":774766592,"data_stored":774766592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":520753152,"data_stored":520749080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:28:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:18 smithi194 ceph-mon[44089]: from='client.16605 v1:172.21.15.40:0/495697430' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/884354043' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:18 smithi194 ceph-mon[44089]: from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:18 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 19855/126262 objects degraded (15.725%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:18.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:18.612 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:18.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:18.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:18.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:18.780 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:18.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:18.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:18.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:19.191 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:19.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:19.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:19.328 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:19.356 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:19 smithi121 ceph-mon[44533]: pgmap v1327: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 19178/122520 objects degraded (15.653%); 14 MiB/s, 219 objects/s recovering 2024-04-29T14:28:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:19 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:19 smithi121 ceph-mon[44533]: from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:28:19.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:19 smithi121 ceph-mon[44533]: osdmap e502: 12 total, 11 up, 7 in 2024-04-29T14:28:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:19.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:19 smithi040 ceph-mon[39581]: pgmap v1327: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 19178/122520 objects degraded (15.653%); 14 MiB/s, 219 objects/s recovering 2024-04-29T14:28:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:19 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:19 smithi040 ceph-mon[39581]: from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:28:19.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:19 smithi040 ceph-mon[39581]: osdmap e502: 12 total, 11 up, 7 in 2024-04-29T14:28:19.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:19.491 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:19.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:19 smithi194 ceph-mon[44089]: pgmap v1327: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 19178/122520 objects degraded (15.653%); 14 MiB/s, 219 objects/s recovering 2024-04-29T14:28:19.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:19 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:19.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:19 smithi194 ceph-mon[44089]: from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:28:19.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:19 smithi194 ceph-mon[44089]: osdmap e502: 12 total, 11 up, 7 in 2024-04-29T14:28:20.042 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:20.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:20.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:20.183 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:20.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:20.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:20.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:20.355 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:20.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:28:20.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:20.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:20.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:28:20.429 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:28:20.451 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:28:20.473 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:28:20.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:20.500 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:28:20.527 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:28:20.534 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:20.550 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:28:20.581 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:28:20.621 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:28:20.648 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:28:20.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:28:20.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:28:20.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:28:20.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:28:20.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:20.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:20.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:20.788 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:20.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:20.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:21.034 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:21.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:21.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:21.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:21.194 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:21.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:21.283 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: pgmap v1329: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 18981/121222 objects degraded (15.658%); 9.9 MiB/s, 158 objects/s recovering 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3751174460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.284 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:21.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: pgmap v1329: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 18981/121222 objects degraded (15.658%); 9.9 MiB/s, 158 objects/s recovering 2024-04-29T14:28:21.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3751174460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:21.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.355 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: pgmap v1329: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 18981/121222 objects degraded (15.658%); 9.9 MiB/s, 158 objects/s recovering 2024-04-29T14:28:21.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3751174460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:21.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:28:21.408 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:28:21.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:21.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:28:21.523 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:21.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:21.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:21.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:21.694 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:21.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:21.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:21.804 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:21.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:21.885 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:21.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:28:22.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:22.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:22.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:22 smithi194 ceph-mon[44089]: osdmap e503: 12 total, 11 up, 7 in 2024-04-29T14:28:22.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:22.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:22.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:22 smithi040 ceph-mon[39581]: osdmap e503: 12 total, 11 up, 7 in 2024-04-29T14:28:22.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:22.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:22.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:22 smithi121 ceph-mon[44533]: osdmap e503: 12 total, 11 up, 7 in 2024-04-29T14:28:22.653 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:22.754 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:22.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:22.866 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:22.942 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:23.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:28:23.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:23 smithi040 ceph-mon[39581]: pgmap v1331: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 18068/115382 objects degraded (15.659%); 16 MiB/s, 248 objects/s recovering 2024-04-29T14:28:23.438 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:23 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 18068/115382 objects degraded (15.659%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:23.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:23 smithi194 ceph-mon[44089]: pgmap v1331: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 18068/115382 objects degraded (15.659%); 16 MiB/s, 248 objects/s recovering 2024-04-29T14:28:23.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:23 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 18068/115382 objects degraded (15.659%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:23 smithi121 ceph-mon[44533]: pgmap v1331: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 18068/115382 objects degraded (15.659%); 16 MiB/s, 248 objects/s recovering 2024-04-29T14:28:23.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:23 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 18068/115382 objects degraded (15.659%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:23.825 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:28:23.825 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:28:24.108 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:24.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:24.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:24.247 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:24.372 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:24.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:24.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:24.523 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:24.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:24.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:24.620 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:28:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:24.627 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:28:24.688 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:24.737 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:24.779 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:24.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:24.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:25.086 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:28:25.179 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:25.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:25 smithi040 ceph-mon[39581]: pgmap v1332: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 770 op/s; 17828/113938 objects degraded (15.647%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:28:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:25.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:25.187 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:25.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:25.315 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:25.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:25.480 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:25 smithi194 ceph-mon[44089]: pgmap v1332: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 770 op/s; 17828/113938 objects degraded (15.647%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:28:25.573 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:28:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:28:25.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:25 smithi121 ceph-mon[44533]: pgmap v1332: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 770 op/s; 17828/113938 objects degraded (15.647%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:28:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:25.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:28:25.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:28:26.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:26 smithi040 ceph-mon[39581]: from='client.16623 v1:172.21.15.40:0/3165342328' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:26.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1248986523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:26.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:26 smithi194 ceph-mon[44089]: from='client.16623 v1:172.21.15.40:0/3165342328' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:26.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1248986523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:26.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:26 smithi121 ceph-mon[44533]: from='client.16623 v1:172.21.15.40:0/3165342328' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:26.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1248986523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:26.985 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1332,"stamp":"2024-04-29T14:28:23.991037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3733914168,"num_objects":56969,"num_object_clones":0,"num_object_copies":113938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56969,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":210142,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24121,"num_bytes_recovered":1582385280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9794916,"kb_used_data":6202920,"kb_used_omap":10,"kb_used_meta":3591733,"kb_avail":646421148,"statfs":{"total":671965249536,"available":661935255552,"internally_reserved":0,"allocated":6351790080,"data_stored":6348169516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3677934686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-403832832,"num_objects":-6162,"num_object_clones":0,"num_object_copies":-12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2027,"num_bytes_recovered":132841472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001666"},"pg_stats":[{"pgid":"5.f","version":"503'13403","reported_seq":23520,"reported_epoch":503,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:21.527972+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:28:21.527972+0000","last_peered":"2024-04-29T14:28:21.527972+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:28:21.527972+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:28:21.527972+0000","mapping_epoch":467,"log_start":"444'3400","ondisk_log_start":"444'3400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233373696,"num_objects":3561,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3545,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13403,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4937,"num_bytes_recovered":323551232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3545},{"shards":"4,11","objects":16}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"502'13203","reported_seq":13431,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-04-29T14:28:20.744880+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:28:20.744880+0000","last_peered":"2024-04-29T14:28:20.744880+0000","last_clean":"2024-04-29T14:28:20.744880+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:28:20.744880+0000","last_undegraded":"2024-04-29T14:28:20.744880+0000","last_fullsized":"2024-04-29T14:28:20.744880+0000","mapping_epoch":432,"log_start":"444'3200","ondisk_log_start":"444'3200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235339776,"num_objects":3591,"num_object_clones":0,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13203,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"503'13154","reported_seq":23011,"reported_epoch":503,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:28:21.215620+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:28:21.215620+0000","last_peered":"2024-04-29T14:28:21.215620+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:28:21.215620+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:28:21.215620+0000","mapping_epoch":468,"log_start":"443'3100","ondisk_log_start":"443'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231931928,"num_objects":3540,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13154,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4811,"num_bytes_recovered":315293696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":5},{"shards":"11","objects":3535}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"503'13211","reported_seq":13453,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:21.530307+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:28:21.530307+0000","last_peered":"2024-04-29T14:28:21.530307+0000","last_clean":"2024-04-29T14:28:21.530307+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:28:21.530307+0000","last_undegraded":"2024-04-29T14:28:21.530307+0000","last_fullsized":"2024-04-29T14:28:21.530307+0000","mapping_epoch":432,"log_start":"444'3200","ondisk_log_start":"444'3200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234684416,"num_objects":3581,"num_object_clones":0,"num_object_copies":7162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13211,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"502'13102","reported_seq":13337,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-04-29T14:28:20.731700+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:28:20.731700+0000","last_peered":"2024-04-29T14:28:20.731700+0000","last_clean":"2024-04-29T14:28:20.731700+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:28:20.731700+0000","last_undegraded":"2024-04-29T14:28:20.731700+0000","last_fullsized":"2024-04-29T14:28:20.731700+0000","mapping_epoch":432,"log_start":"443'3100","ondisk_log_start":"443'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233570304,"num_objects":3564,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13102,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"503'15375","reported_seq":15743,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:23.567206+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:28:23.567206+0000","last_peered":"2024-04-29T14:28:23.567206+0000","last_clean":"2024-04-29T14:28:23.567206+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:28:23.567206+0000","last_undegraded":"2024-04-29T14:28:23.567206+0000","last_fullsized":"2024-04-29T14:28:23.567206+0000","mapping_epoch":432,"log_start":"450'5300","ondisk_log_start":"450'5300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224526336,"num_objects":3426,"num_object_clones":0,"num_object_copies":6852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13120,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"503'13024","reported_seq":13263,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:21.520859+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:28:21.520859+0000","last_peered":"2024-04-29T14:28:21.520859+0000","last_clean":"2024-04-29T14:28:21.520859+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:28:21.520859+0000","last_undegraded":"2024-04-29T14:28:21.520859+0000","last_fullsized":"2024-04-29T14:28:21.520859+0000","mapping_epoch":432,"log_start":"443'3000","ondisk_log_start":"443'3000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233177088,"num_objects":3558,"num_object_clones":0,"num_object_copies":7116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13024,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"503'13112","reported_seq":13345,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:21.194382+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:28:21.194382+0000","last_peered":"2024-04-29T14:28:21.194382+0000","last_clean":"2024-04-29T14:28:21.194382+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:28:21.194382+0000","last_undegraded":"2024-04-29T14:28:21.194382+0000","last_fullsized":"2024-04-29T14:28:21.194382+0000","mapping_epoch":432,"log_start":"444'3100","ondisk_log_start":"444'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229113856,"num_objects":3496,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13112,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"502'13152","reported_seq":22899,"reported_epoch":502,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:20.740222+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:28:20.740222+0000","last_peered":"2024-04-29T14:28:20.740222+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:28:20.740222+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:28:20.740222+0000","mapping_epoch":467,"log_start":"443'3100","ondisk_log_start":"443'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238944256,"num_objects":3646,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13152,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4756,"num_bytes_recovered":311689216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":3643},{"shards":"9,10","objects":3}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"503'13160","reported_seq":22976,"reported_epoch":503,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:21.557969+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:28:21.557969+0000","last_peered":"2024-04-29T14:28:21.557969+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:28:21.557969+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:28:21.557969+0000","mapping_epoch":467,"log_start":"444'3100","ondisk_log_start":"444'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236322816,"num_objects":3606,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13160,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4787,"num_bytes_recovered":313720832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":3579},{"shards":"7,10","objects":27}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"503'13154","reported_seq":13393,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:21.489509+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:28:21.489509+0000","last_peered":"2024-04-29T14:28:21.489509+0000","last_clean":"2024-04-29T14:28:21.489509+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:28:21.489509+0000","last_undegraded":"2024-04-29T14:28:21.489509+0000","last_fullsized":"2024-04-29T14:28:21.489509+0000","mapping_epoch":432,"log_start":"444'3100","ondisk_log_start":"444'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236322816,"num_objects":3606,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13154,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"502'13006","reported_seq":13343,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-04-29T14:28:20.324869+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:28:20.324869+0000","last_peered":"2024-04-29T14:28:20.324869+0000","last_clean":"2024-04-29T14:28:20.324869+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:28:20.324869+0000","last_undegraded":"2024-04-29T14:28:20.324869+0000","last_fullsized":"2024-04-29T14:28:20.324869+0000","mapping_epoch":432,"log_start":"443'3000","ondisk_log_start":"443'3000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:53.158953+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236191744,"num_objects":3604,"num_object_clones":0,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13006,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"502'12950","reported_seq":13249,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-04-29T14:28:20.322080+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:28:20.322080+0000","last_peered":"2024-04-29T14:28:20.322080+0000","last_clean":"2024-04-29T14:28:20.322080+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:28:20.322080+0000","last_undegraded":"2024-04-29T14:28:20.322080+0000","last_fullsized":"2024-04-29T14:28:20.322080+0000","mapping_epoch":432,"log_start":"443'2900","ondisk_log_start":"443'2900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238551040,"num_objects":3640,"num_object_clones":0,"num_object_copies":7280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12950,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"503'12897","reported_seq":13140,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:21.478655+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:28:21.478655+0000","last_peered":"2024-04-29T14:28:21.478655+0000","last_clean":"2024-04-29T14:28:21.478655+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:28:21.478655+0000","last_undegraded":"2024-04-29T14:28:21.478655+0000","last_fullsized":"2024-04-29T14:28:21.478655+0000","mapping_epoch":432,"log_start":"443'2800","ondisk_log_start":"443'2800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232587264,"num_objects":3549,"num_object_clones":0,"num_object_copies":7098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12897,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"503'16107","reported_seq":16522,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:23.622227+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:28:23.622227+0000","last_peered":"2024-04-29T14:28:23.622227+0000","last_clean":"2024-04-29T14:28:23.622227+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:28:23.622227+0000","last_undegraded":"2024-04-29T14:28:23.622227+0000","last_fullsized":"2024-04-29T14:28:23.622227+0000","mapping_epoch":432,"log_start":"453'6100","ondisk_log_start":"453'6100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227475456,"num_objects":3471,"num_object_clones":0,"num_object_copies":6942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13145,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"503'13168","reported_seq":22970,"reported_epoch":503,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:23.651008+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:28:23.651008+0000","last_peered":"2024-04-29T14:28:23.651008+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:28:23.651008+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:28:23.651008+0000","mapping_epoch":467,"log_start":"444'3100","ondisk_log_start":"444'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231211008,"num_objects":3528,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13168,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4822,"num_bytes_recovered":316014592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":2},{"shards":"10","objects":3526}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1249,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:21.124524+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:28:21.124524+0000","last_peered":"2024-04-29T14:28:21.124524+0000","last_clean":"2024-04-29T14:28:21.124524+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:28:21.124524+0000","last_undegraded":"2024-04-29T14:28:21.124524+0000","last_fullsized":"2024-04-29T14:28:21.124524+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3733323800,"num_objects":56967,"num_object_clones":0,"num_object_copies":113934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209961,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24113,"num_bytes_recovered":1580269568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9060622336,"data_stored":9060614192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160578,"ondisk_log_size":160578,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122305,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1007916,"kb_used_data":453280,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92737236,"statfs":{"total":95995035648,"available":94962929664,"internally_reserved":0,"allocated":464158720,"data_stored":463631583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":567867844},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383942,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1111196,"kb_used_data":678804,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92633956,"statfs":{"total":95995035648,"available":94857170944,"internally_reserved":0,"allocated":695095296,"data_stored":694575783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645579,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117264,"kb_used_data":698696,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92627888,"statfs":{"total":95995035648,"available":94850957312,"internally_reserved":0,"allocated":715464704,"data_stored":714951446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":428538304},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188956,"kb_used_data":692692,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92556196,"statfs":{"total":95995035648,"available":94777544704,"internally_reserved":0,"allocated":709316608,"data_stored":708795726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":508164554},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528601,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258704,"kb_used_data":695816,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92486448,"statfs":{"total":95995035648,"available":94706122752,"internally_reserved":0,"allocated":712515584,"data_stored":712002326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":576387526},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2051792,"kb_used_data":1605512,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91693360,"statfs":{"total":95995035648,"available":93894000640,"internally_reserved":0,"allocated":1644044288,"data_stored":1643531030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497112,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182698,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986469,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2059088,"kb_used_data":1378120,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91686064,"statfs":{"total":95995035648,"available":93886529536,"internally_reserved":0,"allocated":1411194880,"data_stored":1410681622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1640431616,"data_stored":1640431616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1402667008,"data_stored":1402667008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":710934528,"data_stored":710934528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":702353408,"data_stored":702349336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":707723264,"data_stored":707723264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":684457984,"data_stored":684457984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":462360576,"data_stored":462356504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:28:26.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:28:27.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:27 smithi194 ceph-mon[44089]: pgmap v1333: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 676 op/s; 17608/111694 objects degraded (15.764%); 12 MiB/s, 196 objects/s recovering 2024-04-29T14:28:27.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:27 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:28:27.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1248986523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:27.310 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:27 smithi194 ceph-mon[44089]: osdmap e504: 12 total, 11 up, 7 in 2024-04-29T14:28:27.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:27 smithi040 ceph-mon[39581]: pgmap v1333: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 676 op/s; 17608/111694 objects degraded (15.764%); 12 MiB/s, 196 objects/s recovering 2024-04-29T14:28:27.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:27 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:28:27.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1248986523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:27.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:27 smithi040 ceph-mon[39581]: osdmap e504: 12 total, 11 up, 7 in 2024-04-29T14:28:27.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:27 smithi121 ceph-mon[44533]: pgmap v1333: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 676 op/s; 17608/111694 objects degraded (15.764%); 12 MiB/s, 196 objects/s recovering 2024-04-29T14:28:27.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:27 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:28:27.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1248986523' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:27.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:27 smithi121 ceph-mon[44533]: osdmap e504: 12 total, 11 up, 7 in 2024-04-29T14:28:27.826 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:27.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:27.933 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:27.985 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:28.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:28 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 17608/111694 objects degraded (15.764%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:28.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:28 smithi194 ceph-mon[44089]: osdmap e505: 12 total, 11 up, 7 in 2024-04-29T14:28:28.457 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:28 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 17608/111694 objects degraded (15.764%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:28.457 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:28 smithi040 ceph-mon[39581]: osdmap e505: 12 total, 11 up, 7 in 2024-04-29T14:28:28.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:28 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 17608/111694 objects degraded (15.764%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:28.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:28 smithi121 ceph-mon[44533]: osdmap e505: 12 total, 11 up, 7 in 2024-04-29T14:28:28.693 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:28:28.693 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:28:28.751 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:28.769 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:28:28.794 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:28:28.815 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:28:28.837 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:28:28.862 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:28:28.886 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:28:28.913 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:28:28.938 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:28:28.961 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:28:28.988 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:28:29.014 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:28:29.014 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:28:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: pgmap v1336: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 792 op/s; 16924/108086 objects degraded (15.658%); 17 MiB/s, 267 objects/s recovering 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3801743933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:28:29.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: pgmap v1336: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 792 op/s; 16924/108086 objects degraded (15.658%); 17 MiB/s, 267 objects/s recovering 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3801743933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:28:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:28:29.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:28:29.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: pgmap v1336: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 792 op/s; 16924/108086 objects degraded (15.658%); 17 MiB/s, 267 objects/s recovering 2024-04-29T14:28:29.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3801743933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:29.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:28:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:28:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:28:29.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:28:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:28:30.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:28:30.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:28:30.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:30.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:28:30.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:28:30.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:28:30.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:28:30.973 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:30.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:30.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:31.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:31.120 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:31.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:31.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:31.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:31 smithi040 ceph-mon[39581]: pgmap v1337: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 540 op/s; 16695/106736 objects degraded (15.641%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:28:31.391 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:31.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:31 smithi194 ceph-mon[44089]: pgmap v1337: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 540 op/s; 16695/106736 objects degraded (15.641%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:28:31.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:31 smithi121 ceph-mon[44533]: pgmap v1337: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 540 op/s; 16695/106736 objects degraded (15.641%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:28:31.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:31.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:31.799 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:31.945 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:31.981 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:31.983 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:28:32.060 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:32.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:32.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:28:32.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:32.218 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:32.258 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:32.260 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:32 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 16695/106736 objects degraded (15.641%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:32.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:32.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:32.400 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:32.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:32.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:32 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 16695/106736 objects degraded (15.641%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:32.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:32.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:32 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 16695/106736 objects degraded (15.641%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:32.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:32.670 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:32.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:32.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:32.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:32.885 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:32.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:32.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:28:33.031 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:33.060 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:28:33.061 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:28:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:33.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:28:33.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:33.199 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:33.298 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:33.305 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:33 smithi040 ceph-mon[39581]: from='client.16638 v1:172.21.15.40:0/2709183033' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:33.305 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:33 smithi040 ceph-mon[39581]: pgmap v1338: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 823 op/s; 15725/100754 objects degraded (15.607%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:28:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:33.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:33.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:33 smithi194 ceph-mon[44089]: from='client.16638 v1:172.21.15.40:0/2709183033' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:33.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:33 smithi194 ceph-mon[44089]: pgmap v1338: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 823 op/s; 15725/100754 objects degraded (15.607%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:28:33.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:33.435 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:33.464 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:33.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:33.524 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:28:33.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:28:33.598 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:33.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:33 smithi121 ceph-mon[44533]: from='client.16638 v1:172.21.15.40:0/2709183033' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:33.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:33 smithi121 ceph-mon[44533]: pgmap v1338: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 823 op/s; 15725/100754 objects degraded (15.607%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:28:33.747 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:28:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:33.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:28:33.782 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1337,"stamp":"2024-04-29T14:28:29.992255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3497919032,"num_objects":53368,"num_object_clones":0,"num_object_copies":106736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53368,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":213743,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25254,"num_bytes_recovered":1656637568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161016,"ondisk_log_size":161016,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9442460,"kb_used_data":5814548,"kb_used_omap":10,"kb_used_meta":3627829,"kb_avail":646773604,"statfs":{"total":671965249536,"available":662296170496,"internally_reserved":0,"allocated":5954097152,"data_stored":5950420273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3714896990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-283312128,"num_objects":-4323,"num_object_clones":0,"num_object_copies":-8646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4323,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1373,"num_bytes_recovered":89980928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001670"},"pg_stats":[{"pgid":"5.f","version":"504'13644","reported_seq":24215,"reported_epoch":504,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:26.536618+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:28:26.536618+0000","last_peered":"2024-04-29T14:28:26.536618+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:28:26.536618+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:28:26.536618+0000","mapping_epoch":467,"log_start":"444'3600","ondisk_log_start":"444'3600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217579520,"num_objects":3320,"num_object_clones":0,"num_object_copies":6640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13644,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5163,"num_bytes_recovered":338362368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3319},{"shards":"4,11","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"503'13413","reported_seq":13639,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:25.777931+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:28:25.777931+0000","last_peered":"2024-04-29T14:28:25.777931+0000","last_clean":"2024-04-29T14:28:25.777931+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:28:25.777931+0000","last_undegraded":"2024-04-29T14:28:25.777931+0000","last_fullsized":"2024-04-29T14:28:25.777931+0000","mapping_epoch":432,"log_start":"444'3400","ondisk_log_start":"444'3400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221577216,"num_objects":3381,"num_object_clones":0,"num_object_copies":6762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13413,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"503'13381","reported_seq":23700,"reported_epoch":504,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:28:26.216847+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:28:26.216847+0000","last_peered":"2024-04-29T14:28:26.216847+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:28:26.216847+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:28:26.216847+0000","mapping_epoch":468,"log_start":"444'3300","ondisk_log_start":"444'3300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217055256,"num_objects":3313,"num_object_clones":0,"num_object_copies":6626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13381,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5041,"num_bytes_recovered":330366976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":8},{"shards":"11","objects":3305}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"504'13425","reported_seq":13670,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-29T14:28:26.469025+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:28:26.469025+0000","last_peered":"2024-04-29T14:28:26.469025+0000","last_clean":"2024-04-29T14:28:26.469025+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:28:26.469025+0000","last_undegraded":"2024-04-29T14:28:26.469025+0000","last_fullsized":"2024-04-29T14:28:26.469025+0000","mapping_epoch":432,"log_start":"444'3400","ondisk_log_start":"444'3400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220659712,"num_objects":3367,"num_object_clones":0,"num_object_copies":6734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13425,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"503'13327","reported_seq":13564,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:25.773935+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:28:25.773935+0000","last_peered":"2024-04-29T14:28:25.773935+0000","last_clean":"2024-04-29T14:28:25.773935+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:28:25.773935+0000","last_undegraded":"2024-04-29T14:28:25.773935+0000","last_fullsized":"2024-04-29T14:28:25.773935+0000","mapping_epoch":432,"log_start":"444'3300","ondisk_log_start":"444'3300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218824704,"num_objects":3339,"num_object_clones":0,"num_object_copies":6678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13327,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"505'15597","reported_seq":15969,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-04-29T14:28:28.643899+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:28:28.643899+0000","last_peered":"2024-04-29T14:28:28.643899+0000","last_clean":"2024-04-29T14:28:28.643899+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:28:28.643899+0000","last_undegraded":"2024-04-29T14:28:28.643899+0000","last_fullsized":"2024-04-29T14:28:28.643899+0000","mapping_epoch":432,"log_start":"451'5500","ondisk_log_start":"451'5500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209977344,"num_objects":3204,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13342,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"504'13254","reported_seq":13495,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-29T14:28:26.549120+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:28:26.549120+0000","last_peered":"2024-04-29T14:28:26.549120+0000","last_clean":"2024-04-29T14:28:26.549120+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:28:26.549120+0000","last_undegraded":"2024-04-29T14:28:26.549120+0000","last_fullsized":"2024-04-29T14:28:26.549120+0000","mapping_epoch":432,"log_start":"444'3200","ondisk_log_start":"444'3200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13254,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"503'13308","reported_seq":13544,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-29T14:28:26.195118+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:28:26.195118+0000","last_peered":"2024-04-29T14:28:26.195118+0000","last_clean":"2024-04-29T14:28:26.195118+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:28:26.195118+0000","last_undegraded":"2024-04-29T14:28:26.195118+0000","last_fullsized":"2024-04-29T14:28:26.195118+0000","mapping_epoch":432,"log_start":"444'3300","ondisk_log_start":"444'3300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216268800,"num_objects":3300,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13308,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"503'13371","reported_seq":23560,"reported_epoch":503,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:25.778929+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:28:25.778929+0000","last_peered":"2024-04-29T14:28:25.778929+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:28:25.778929+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:28:25.778929+0000","mapping_epoch":467,"log_start":"444'3300","ondisk_log_start":"444'3300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224591872,"num_objects":3427,"num_object_clones":0,"num_object_copies":6854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13371,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4976,"num_bytes_recovered":326107136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":3423},{"shards":"9,10","objects":4}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"504'13397","reported_seq":23671,"reported_epoch":504,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:26.578352+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:28:26.578352+0000","last_peered":"2024-04-29T14:28:26.578352+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:28:26.578352+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:28:26.578352+0000","mapping_epoch":467,"log_start":"444'3300","ondisk_log_start":"444'3300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220790784,"num_objects":3369,"num_object_clones":0,"num_object_copies":6738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13397,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5015,"num_bytes_recovered":328663040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":3351},{"shards":"7,10","objects":18}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"504'13417","reported_seq":13658,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-29T14:28:26.536584+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:28:26.536584+0000","last_peered":"2024-04-29T14:28:26.536584+0000","last_clean":"2024-04-29T14:28:26.536584+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:28:26.536584+0000","last_undegraded":"2024-04-29T14:28:26.536584+0000","last_fullsized":"2024-04-29T14:28:26.536584+0000","mapping_epoch":432,"log_start":"444'3400","ondisk_log_start":"444'3400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219086848,"num_objects":3343,"num_object_clones":0,"num_object_copies":6686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13417,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"503'13252","reported_seq":13591,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:25.298711+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:28:25.298711+0000","last_peered":"2024-04-29T14:28:25.298711+0000","last_clean":"2024-04-29T14:28:25.298711+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:28:25.298711+0000","last_undegraded":"2024-04-29T14:28:25.298711+0000","last_fullsized":"2024-04-29T14:28:25.298711+0000","mapping_epoch":432,"log_start":"444'3200","ondisk_log_start":"444'3200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:53.158953+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220069888,"num_objects":3358,"num_object_clones":0,"num_object_copies":6716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13252,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"503'13172","reported_seq":13473,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-04-29T14:28:25.328576+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:28:25.328576+0000","last_peered":"2024-04-29T14:28:25.328576+0000","last_clean":"2024-04-29T14:28:25.328576+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:28:25.328576+0000","last_undegraded":"2024-04-29T14:28:25.328576+0000","last_fullsized":"2024-04-29T14:28:25.328576+0000","mapping_epoch":432,"log_start":"443'3100","ondisk_log_start":"443'3100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224002048,"num_objects":3418,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13172,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"504'13093","reported_seq":13338,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-29T14:28:26.503985+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:28:26.503985+0000","last_peered":"2024-04-29T14:28:26.503985+0000","last_clean":"2024-04-29T14:28:26.503985+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:28:26.503985+0000","last_undegraded":"2024-04-29T14:28:26.503985+0000","last_fullsized":"2024-04-29T14:28:26.503985+0000","mapping_epoch":432,"log_start":"444'3000","ondisk_log_start":"444'3000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":6706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13093,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"505'16333","reported_seq":16751,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-04-29T14:28:28.668240+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:28:28.668240+0000","last_peered":"2024-04-29T14:28:28.668240+0000","last_clean":"2024-04-29T14:28:28.668240+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:28:28.668240+0000","last_undegraded":"2024-04-29T14:28:28.668240+0000","last_fullsized":"2024-04-29T14:28:28.668240+0000","mapping_epoch":432,"log_start":"453'6300","ondisk_log_start":"453'6300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212664320,"num_objects":3245,"num_object_clones":0,"num_object_copies":6490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13371,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"505'13395","reported_seq":23659,"reported_epoch":505,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:28.671586+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:28:28.671586+0000","last_peered":"2024-04-29T14:28:28.671586+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:28:28.671586+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:28:28.671586+0000","mapping_epoch":467,"log_start":"444'3300","ondisk_log_start":"444'3300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216334336,"num_objects":3301,"num_object_clones":0,"num_object_copies":6602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13395,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5051,"num_bytes_recovered":331022336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":4},{"shards":"10","objects":3297}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1251,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-29T14:28:26.186970+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:28:26.186970+0000","last_peered":"2024-04-29T14:28:26.186970+0000","last_clean":"2024-04-29T14:28:26.186970+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:28:26.186970+0000","last_undegraded":"2024-04-29T14:28:26.186970+0000","last_fullsized":"2024-04-29T14:28:26.186970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3497328664,"num_objects":53366,"num_object_clones":0,"num_object_copies":106732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213562,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25246,"num_bytes_recovered":1654521856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8662163456,"data_stored":8662155312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122306,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":983608,"kb_used_data":424876,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92761544,"statfs":{"total":95995035648,"available":94987821056,"internally_reserved":0,"allocated":435073024,"data_stored":434539602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":572127684},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383943,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1089148,"kb_used_data":634540,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92656004,"statfs":{"total":95995035648,"available":94879748096,"internally_reserved":0,"allocated":649768960,"data_stored":649236877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":465500608},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645580,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061852,"kb_used_data":655828,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92683300,"statfs":{"total":95995035648,"available":94907699200,"internally_reserved":0,"allocated":671567872,"data_stored":671048379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":415758784},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1171304,"kb_used_data":652832,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92573848,"statfs":{"total":95995035648,"available":94795620352,"internally_reserved":0,"allocated":668499968,"data_stored":667972851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":530905546},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528602,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202332,"kb_used_data":651924,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92542820,"statfs":{"total":95995035648,"available":94763847680,"internally_reserved":0,"allocated":667570176,"data_stored":667050683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":563608006},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575155,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959580,"kb_used_data":1504980,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91785572,"statfs":{"total":95995035648,"available":93988425728,"internally_reserved":0,"allocated":1541099520,"data_stored":1540580027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497113,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986470,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1974636,"kb_used_data":1289568,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":91770516,"statfs":{"total":95995035648,"available":93973008384,"internally_reserved":0,"allocated":1320517632,"data_stored":1319991854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1537081344,"data_stored":1537081344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1312161792,"data_stored":1312161792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":665845760,"data_stored":665845760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":658903040,"data_stored":658898968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":664862720,"data_stored":664862720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":640221184,"data_stored":640221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":433393664,"data_stored":433389592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:28:33.783 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.f is not active+clean 2024-04-29T14:28:33.783 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.f', 'version': "504'13644", 'reported_seq': 24215, 'reported_epoch': 504, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:26.536618+0000', 'last_change': '2024-04-29T14:26:16.020854+0000', 'last_active': '2024-04-29T14:28:26.536618+0000', 'last_peered': '2024-04-29T14:28:26.536618+0000', 'last_clean': '2024-04-29T14:25:54.182894+0000', 'last_became_active': '2024-04-29T14:26:16.010174+0000', 'last_became_peered': '2024-04-29T14:26:16.010174+0000', 'last_unstale': '2024-04-29T14:28:26.536618+0000', 'last_undegraded': '2024-04-29T14:26:15.821893+0000', 'last_fullsized': '2024-04-29T14:28:26.536618+0000', 'mapping_epoch': 467, 'log_start': "444'3600", 'ondisk_log_start': "444'3600", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'7968", 'last_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'objects_scrubbed': 7919, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2114, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 217579520, 'num_objects': 3320, 'num_object_clones': 0, 'num_object_copies': 6640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13644, 'num_write_kb': 542848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5163, 'num_bytes_recovered': 338362368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 11], 'acting': [4, 11], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 3319}, {'shards': '4,11', 'objects': 1}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-29T14:28:33.783 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.d is not active+clean 2024-04-29T14:28:33.783 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.d', 'version': "503'13381", 'reported_seq': 23700, 'reported_epoch': 504, 'state': 'active+recovering+degraded+remapped', 'last_fresh': '2024-04-29T14:28:26.216847+0000', 'last_change': '2024-04-29T14:26:17.048329+0000', 'last_active': '2024-04-29T14:28:26.216847+0000', 'last_peered': '2024-04-29T14:28:26.216847+0000', 'last_clean': '2024-04-29T14:25:54.192906+0000', 'last_became_active': '2024-04-29T14:26:17.022020+0000', 'last_became_peered': '2024-04-29T14:26:17.022020+0000', 'last_unstale': '2024-04-29T14:28:26.216847+0000', 'last_undegraded': '2024-04-29T14:26:16.808487+0000', 'last_fullsized': '2024-04-29T14:28:26.216847+0000', 'mapping_epoch': 468, 'log_start': "444'3300", 'ondisk_log_start': "444'3300", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "454'7099", 'last_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'objects_scrubbed': 7053, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1409, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 217055256, 'num_objects': 3313, 'num_object_clones': 0, 'num_object_copies': 6626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3313, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13381, 'num_write_kb': 534145, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5041, 'num_bytes_recovered': 330366976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 11], 'acting': [11, 7], 'avail_no_missing': ['11'], 'object_location_counts': [{'shards': '7,11', 'objects': 8}, {'shards': '11', 'objects': 3305}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 11, 'purged_snaps': []} 2024-04-29T14:28:33.784 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.7 is not active+clean 2024-04-29T14:28:33.784 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.7', 'version': "503'13371", 'reported_seq': 23560, 'reported_epoch': 503, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:25.778929+0000', 'last_change': '2024-04-29T14:26:16.052811+0000', 'last_active': '2024-04-29T14:28:25.778929+0000', 'last_peered': '2024-04-29T14:28:25.778929+0000', 'last_clean': '2024-04-29T14:25:54.182628+0000', 'last_became_active': '2024-04-29T14:26:16.038648+0000', 'last_became_peered': '2024-04-29T14:26:16.038648+0000', 'last_unstale': '2024-04-29T14:28:25.778929+0000', 'last_undegraded': '2024-04-29T14:26:15.818605+0000', 'last_fullsized': '2024-04-29T14:28:25.778929+0000', 'mapping_epoch': 467, 'log_start': "444'3300", 'ondisk_log_start': "444'3300", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "455'7479", 'last_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'objects_scrubbed': 7424, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1679, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 224591872, 'num_objects': 3427, 'num_object_clones': 0, 'num_object_copies': 6854, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3423, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13371, 'num_write_kb': 537536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4976, 'num_bytes_recovered': 326107136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 10], 'acting': [9, 10], 'avail_no_missing': ['9'], 'object_location_counts': [{'shards': '9', 'objects': 3423}, {'shards': '9,10', 'objects': 4}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-29T14:28:33.784 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-29T14:28:33.784 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "504'13397", 'reported_seq': 23671, 'reported_epoch': 504, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:26.578352+0000', 'last_change': '2024-04-29T14:26:16.126893+0000', 'last_active': '2024-04-29T14:28:26.578352+0000', 'last_peered': '2024-04-29T14:28:26.578352+0000', 'last_clean': '2024-04-29T14:25:53.790194+0000', 'last_became_active': '2024-04-29T14:26:16.054003+0000', 'last_became_peered': '2024-04-29T14:26:16.054003+0000', 'last_unstale': '2024-04-29T14:28:26.578352+0000', 'last_undegraded': '2024-04-29T14:26:15.820366+0000', 'last_fullsized': '2024-04-29T14:28:26.578352+0000', 'mapping_epoch': 467, 'log_start': "444'3300", 'ondisk_log_start': "444'3300", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'8070", 'last_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'objects_scrubbed': 8029, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1689, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 220790784, 'num_objects': 3369, 'num_object_clones': 0, 'num_object_copies': 6738, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3351, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13397, 'num_write_kb': 536512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5015, 'num_bytes_recovered': 328663040, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 10], 'acting': [7, 10], 'avail_no_missing': ['7'], 'object_location_counts': [{'shards': '7', 'objects': 3351}, {'shards': '7,10', 'objects': 18}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-29T14:28:33.784 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-29T14:28:33.784 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "505'13395", 'reported_seq': 23659, 'reported_epoch': 505, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:28.671586+0000', 'last_change': '2024-04-29T14:26:15.998964+0000', 'last_active': '2024-04-29T14:28:28.671586+0000', 'last_peered': '2024-04-29T14:28:28.671586+0000', 'last_clean': '2024-04-29T14:25:59.019434+0000', 'last_became_active': '2024-04-29T14:26:15.982158+0000', 'last_became_peered': '2024-04-29T14:26:15.982158+0000', 'last_unstale': '2024-04-29T14:28:28.671586+0000', 'last_undegraded': '2024-04-29T14:26:15.829402+0000', 'last_fullsized': '2024-04-29T14:28:28.671586+0000', 'mapping_epoch': 467, 'log_start': "444'3300", 'ondisk_log_start': "444'3300", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 216334336, 'num_objects': 3301, 'num_object_clones': 0, 'num_object_copies': 6602, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13395, 'num_write_kb': 534272, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5051, 'num_bytes_recovered': 331022336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 0], 'acting': [10, 0], 'avail_no_missing': ['10'], 'object_location_counts': [{'shards': '0,10', 'objects': 4}, {'shards': '10', 'objects': 3297}], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-04-29T14:28:33.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:28:33.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:33.902 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:34.009 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:34.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:34.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:34.308 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:35.108 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:28:35.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:35 smithi194 ceph-mon[44089]: pgmap v1339: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 15500/99424 objects degraded (15.590%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:28:35.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:35.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:35.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1671406496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:28:35.551 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:35.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:35.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:35 smithi040 ceph-mon[39581]: pgmap v1339: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 15500/99424 objects degraded (15.590%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:28:35.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:35.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:35.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1671406496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:28:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:35 smithi121 ceph-mon[44533]: pgmap v1339: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 15500/99424 objects degraded (15.590%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:28:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:35.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1671406496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:28:35.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:35.693 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:35.733 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:35.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:35.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:35.874 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:36.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:36.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:36.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:36 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:36.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1671406496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:28:36.272 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:36 smithi194 ceph-mon[44089]: osdmap e506: 12 total, 11 up, 7 in 2024-04-29T14:28:36.326 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:36.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:36.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:36.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:36.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:36.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:36.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:36.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:36.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:36 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:36.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1671406496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:28:36.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:36 smithi040 ceph-mon[39581]: osdmap e506: 12 total, 11 up, 7 in 2024-04-29T14:28:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:36.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:36.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:36 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:36.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1671406496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:28:36.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:36 smithi121 ceph-mon[44533]: osdmap e506: 12 total, 11 up, 7 in 2024-04-29T14:28:36.646 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:36.669 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:36.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:36.777 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:36.830 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:37.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:37 smithi194 ceph-mon[44089]: pgmap v1341: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 630 op/s; 15248/97024 objects degraded (15.716%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:28:37.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:37 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 15248/97024 objects degraded (15.716%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:37.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:37 smithi194 ceph-mon[44089]: osdmap e507: 12 total, 11 up, 7 in 2024-04-29T14:28:37.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:37 smithi040 ceph-mon[39581]: pgmap v1341: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 630 op/s; 15248/97024 objects degraded (15.716%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:28:37.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:37 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 15248/97024 objects degraded (15.716%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:37.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:37 smithi040 ceph-mon[39581]: osdmap e507: 12 total, 11 up, 7 in 2024-04-29T14:28:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:37 smithi121 ceph-mon[44533]: pgmap v1341: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 3.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 630 op/s; 15248/97024 objects degraded (15.716%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:28:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:37 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 15248/97024 objects degraded (15.716%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:37 smithi121 ceph-mon[44533]: osdmap e507: 12 total, 11 up, 7 in 2024-04-29T14:28:38.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:38.607 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14542/93264 objects degraded (15.592%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1592,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":507,"num_osds":12,"num_up_osds":11,"osd_up_since":1714400760,"num_in_osds":7,"osd_in_since":1714400774,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovering+degraded","count":4},{"state_name":"active+recovering+degraded+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":46632,"data_bytes":3056468536,"bytes_used":9018138624,"bytes_avail":662947110912,"bytes_total":671965249536,"degraded_objects":14542,"degraded_total":93264,"degraded_ratio":0.15592297135014582,"recovering_objects_per_sec":269,"recovering_bytes_per_sec":17633598,"recovering_keys_per_sec":0,"num_objects_recovered":2153,"num_bytes_recovered":141099008,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":841},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":40,"modified":"2024-04-29T14:24:06.771444+0000","services":{}},"progress_events":{}} 2024-04-29T14:28:38.706 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:38.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:38.777 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:28:38.798 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:28:38.813 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:28:38.845 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:28:38.858 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:38.866 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:28:38.893 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:28:38.918 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:28:38.945 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:28:38.970 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:28:39.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:28:39.036 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:28:39.037 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:28:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:39.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:39.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:39.227 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:39 smithi040 ceph-mon[39581]: pgmap v1343: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 14542/93264 objects degraded (15.592%); 17 MiB/s, 269 objects/s recovering 2024-04-29T14:28:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/235868285' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:28:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1899136058' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.539 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:28:39.539 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:28:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:39 smithi194 ceph-mon[44089]: pgmap v1343: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 14542/93264 objects degraded (15.592%); 17 MiB/s, 269 objects/s recovering 2024-04-29T14:28:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/235868285' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:28:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1899136058' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:39 smithi121 ceph-mon[44533]: pgmap v1343: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 14542/93264 objects degraded (15.592%); 17 MiB/s, 269 objects/s recovering 2024-04-29T14:28:39.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/235868285' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:28:39.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1899136058' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:39.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.475 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:28:40.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:28:40.481 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:40.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:40.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:40.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:40.676 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:40.783 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:41.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:41 smithi040 ceph-mon[39581]: pgmap v1344: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 14282/91798 objects degraded (15.558%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:28:41.543 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:41.546 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:41 smithi194 ceph-mon[44089]: pgmap v1344: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 14282/91798 objects degraded (15.558%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:28:41.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:41.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:41.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:41 smithi121 ceph-mon[44533]: pgmap v1344: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 14282/91798 objects degraded (15.558%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:28:41.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:41.682 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:41.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:41.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:41.993 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:42.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:42.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:42.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:42 smithi040 ceph-mon[39581]: osdmap e508: 12 total, 11 up, 7 in 2024-04-29T14:28:42.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:42 smithi040 ceph-mon[39581]: pgmap v1346: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 13281/85456 objects degraded (15.541%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:28:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:42 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 14282/91798 objects degraded (15.558%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:42.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:42 smithi040 ceph-mon[39581]: osdmap e509: 12 total, 11 up, 7 in 2024-04-29T14:28:42.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:42 smithi121 ceph-mon[44533]: osdmap e508: 12 total, 11 up, 7 in 2024-04-29T14:28:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:42 smithi121 ceph-mon[44533]: pgmap v1346: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 13281/85456 objects degraded (15.541%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:28:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:42 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 14282/91798 objects degraded (15.558%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:42.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:42 smithi121 ceph-mon[44533]: osdmap e509: 12 total, 11 up, 7 in 2024-04-29T14:28:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:42 smithi194 ceph-mon[44089]: osdmap e508: 12 total, 11 up, 7 in 2024-04-29T14:28:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:42 smithi194 ceph-mon[44089]: pgmap v1346: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 13281/85456 objects degraded (15.541%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:28:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:42 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 14282/91798 objects degraded (15.558%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:42.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:42 smithi194 ceph-mon[44089]: osdmap e509: 12 total, 11 up, 7 in 2024-04-29T14:28:42.961 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:42.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:42.995 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:28:43.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:43.099 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:28:43.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:28:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:43.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:43.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:28:43.493 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:43.506 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:28:43.595 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:28:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:43.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:28:43.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:28:43.731 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:43.759 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:28:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:43.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:28:43.864 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:43.991 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:28:43.995 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:43.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:44.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:44.130 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:44.255 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:44.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:44.358 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:44.436 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:44 smithi040 ceph-mon[39581]: osdmap e510: 12 total, 11 up, 7 in 2024-04-29T14:28:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:44 smithi040 ceph-mon[39581]: from='client.16662 v1:172.21.15.40:0/1867916409' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:44.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:44 smithi040 ceph-mon[39581]: pgmap v1349: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 13034/84008 objects degraded (15.515%); 16 MiB/s, 251 objects/s recovering 2024-04-29T14:28:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:44 smithi121 ceph-mon[44533]: osdmap e510: 12 total, 11 up, 7 in 2024-04-29T14:28:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:44 smithi121 ceph-mon[44533]: from='client.16662 v1:172.21.15.40:0/1867916409' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:44 smithi121 ceph-mon[44533]: pgmap v1349: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 13034/84008 objects degraded (15.515%); 16 MiB/s, 251 objects/s recovering 2024-04-29T14:28:44.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:44.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:44.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:44.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:44.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:44 smithi194 ceph-mon[44089]: osdmap e510: 12 total, 11 up, 7 in 2024-04-29T14:28:44.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:44 smithi194 ceph-mon[44089]: from='client.16662 v1:172.21.15.40:0/1867916409' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:44.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:44 smithi194 ceph-mon[44089]: pgmap v1349: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 13034/84008 objects degraded (15.515%); 16 MiB/s, 251 objects/s recovering 2024-04-29T14:28:44.914 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:44.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:45.012 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1348,"stamp":"2024-04-29T14:28:43.327951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2800615992,"num_objects":42728,"num_object_clones":0,"num_object_copies":85456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42728,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":224383,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28668,"num_bytes_recovered":1880377472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161056,"ondisk_log_size":161056,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8412660,"kb_used_data":4648628,"kb_used_omap":10,"kb_used_meta":3763893,"kb_avail":647803404,"statfs":{"total":671965249536,"available":663350685696,"internally_reserved":0,"allocated":4760195072,"data_stored":4756368053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3854226526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-255852544,"num_objects":-3904,"num_object_clones":0,"num_object_copies":-7808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3904,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":82640896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.113100"},"pg_stats":[{"pgid":"5.f","version":"508'14376","reported_seq":26471,"reported_epoch":508,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:41.550957+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:28:41.550957+0000","last_peered":"2024-04-29T14:28:41.550957+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:28:41.550957+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:28:41.550957+0000","mapping_epoch":467,"log_start":"446'4300","ondisk_log_start":"446'4300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14376,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":388038656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":2561},{"shards":"4,11","objects":27}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"507'14133","reported_seq":14371,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-29T14:28:40.722273+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:28:40.722273+0000","last_peered":"2024-04-29T14:28:40.722273+0000","last_clean":"2024-04-29T14:28:40.722273+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:28:40.722273+0000","last_undegraded":"2024-04-29T14:28:40.722273+0000","last_fullsized":"2024-04-29T14:28:40.722273+0000","mapping_epoch":432,"log_start":"446'4100","ondisk_log_start":"446'4100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174391296,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14133,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"507'14064","reported_seq":25808,"reported_epoch":507,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:28:41.288736+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:28:41.288736+0000","last_peered":"2024-04-29T14:28:41.288736+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:28:41.288736+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:28:41.288736+0000","mapping_epoch":468,"log_start":"446'4000","ondisk_log_start":"446'4000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172294168,"num_objects":2630,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14064,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5751,"num_bytes_recovered":376897536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":35},{"shards":"11","objects":2595}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"508'14153","reported_seq":14406,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-29T14:28:41.537807+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:28:41.537807+0000","last_peered":"2024-04-29T14:28:41.537807+0000","last_clean":"2024-04-29T14:28:41.537807+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:28:41.537807+0000","last_undegraded":"2024-04-29T14:28:41.537807+0000","last_fullsized":"2024-04-29T14:28:41.537807+0000","mapping_epoch":432,"log_start":"446'4100","ondisk_log_start":"446'4100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172949504,"num_objects":2639,"num_object_clones":0,"num_object_copies":5278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14153,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"507'14054","reported_seq":14295,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-29T14:28:40.791939+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:28:40.791939+0000","last_peered":"2024-04-29T14:28:40.791939+0000","last_clean":"2024-04-29T14:28:40.791939+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:28:40.791939+0000","last_undegraded":"2024-04-29T14:28:40.791939+0000","last_fullsized":"2024-04-29T14:28:40.791939+0000","mapping_epoch":432,"log_start":"446'4000","ondisk_log_start":"446'4000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171180032,"num_objects":2612,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14054,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"507'16072","reported_seq":16448,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-29T14:28:38.761818+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:28:38.761818+0000","last_peered":"2024-04-29T14:28:38.761818+0000","last_clean":"2024-04-29T14:28:38.761818+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:28:38.761818+0000","last_undegraded":"2024-04-29T14:28:38.761818+0000","last_fullsized":"2024-04-29T14:28:38.761818+0000","mapping_epoch":432,"log_start":"453'6000","ondisk_log_start":"453'6000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178847744,"num_objects":2729,"num_object_clones":0,"num_object_copies":5458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13817,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"508'13957","reported_seq":14206,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-29T14:28:41.595586+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:28:41.595586+0000","last_peered":"2024-04-29T14:28:41.595586+0000","last_clean":"2024-04-29T14:28:41.595586+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:28:41.595586+0000","last_undegraded":"2024-04-29T14:28:41.595586+0000","last_fullsized":"2024-04-29T14:28:41.595586+0000","mapping_epoch":432,"log_start":"446'3900","ondisk_log_start":"446'3900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172032000,"num_objects":2625,"num_object_clones":0,"num_object_copies":5250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13957,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"507'14007","reported_seq":14249,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-29T14:28:41.253040+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:28:41.253040+0000","last_peered":"2024-04-29T14:28:41.253040+0000","last_clean":"2024-04-29T14:28:41.253040+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:28:41.253040+0000","last_undegraded":"2024-04-29T14:28:41.253040+0000","last_fullsized":"2024-04-29T14:28:41.253040+0000","mapping_epoch":432,"log_start":"446'4000","ondisk_log_start":"446'4000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170459136,"num_objects":2601,"num_object_clones":0,"num_object_copies":5202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"507'14134","reported_seq":25851,"reported_epoch":507,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:40.816211+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:28:40.816211+0000","last_peered":"2024-04-29T14:28:40.816211+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:28:40.816211+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:28:40.816211+0000","mapping_epoch":467,"log_start":"446'4100","ondisk_log_start":"446'4100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174587904,"num_objects":2664,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14134,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5737,"num_bytes_recovered":375980032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":2662},{"shards":"9,10","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"508'14085","reported_seq":25767,"reported_epoch":508,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:41.594603+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:28:41.594603+0000","last_peered":"2024-04-29T14:28:41.594603+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:28:41.594603+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:28:41.594603+0000","mapping_epoch":467,"log_start":"446'4000","ondisk_log_start":"446'4000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175702016,"num_objects":2681,"num_object_clones":0,"num_object_copies":5362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14085,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5715,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":2651},{"shards":"7,10","objects":30}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"508'14107","reported_seq":14357,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-29T14:28:41.562123+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:28:41.562123+0000","last_peered":"2024-04-29T14:28:41.562123+0000","last_clean":"2024-04-29T14:28:41.562123+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:28:41.562123+0000","last_undegraded":"2024-04-29T14:28:41.562123+0000","last_fullsized":"2024-04-29T14:28:41.562123+0000","mapping_epoch":432,"log_start":"446'4100","ondisk_log_start":"446'4100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":5306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14107,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"507'13973","reported_seq":14321,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-29T14:28:40.335103+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:28:40.335103+0000","last_peered":"2024-04-29T14:28:40.335103+0000","last_clean":"2024-04-29T14:28:40.335103+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:28:40.335103+0000","last_undegraded":"2024-04-29T14:28:40.335103+0000","last_fullsized":"2024-04-29T14:28:40.335103+0000","mapping_epoch":432,"log_start":"446'3900","ondisk_log_start":"446'3900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:53.158953+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172818432,"num_objects":2637,"num_object_clones":0,"num_object_copies":5274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13973,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"507'13875","reported_seq":14183,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-29T14:28:40.349866+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:28:40.349866+0000","last_peered":"2024-04-29T14:28:40.349866+0000","last_clean":"2024-04-29T14:28:40.349866+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:28:40.349866+0000","last_undegraded":"2024-04-29T14:28:40.349866+0000","last_fullsized":"2024-04-29T14:28:40.349866+0000","mapping_epoch":432,"log_start":"446'3800","ondisk_log_start":"446'3800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177930240,"num_objects":2715,"num_object_clones":0,"num_object_copies":5430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13875,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"508'13778","reported_seq":14031,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-29T14:28:41.561300+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:28:41.561300+0000","last_peered":"2024-04-29T14:28:41.561300+0000","last_clean":"2024-04-29T14:28:41.561300+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:28:41.561300+0000","last_undegraded":"2024-04-29T14:28:41.561300+0000","last_fullsized":"2024-04-29T14:28:41.561300+0000","mapping_epoch":432,"log_start":"446'3700","ondisk_log_start":"446'3700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":5336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13778,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"507'16769","reported_seq":17192,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-29T14:28:38.691461+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:28:38.691461+0000","last_peered":"2024-04-29T14:28:38.691461+0000","last_clean":"2024-04-29T14:28:38.691461+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:28:38.691461+0000","last_undegraded":"2024-04-29T14:28:38.691461+0000","last_fullsized":"2024-04-29T14:28:38.691461+0000","mapping_epoch":432,"log_start":"454'6700","ondisk_log_start":"454'6700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184090624,"num_objects":2809,"num_object_clones":0,"num_object_copies":5618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13807,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"507'13882","reported_seq":25120,"reported_epoch":507,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:38.773510+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:28:38.773510+0000","last_peered":"2024-04-29T14:28:38.773510+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:28:38.773510+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:28:38.773510+0000","mapping_epoch":467,"log_start":"446'3800","ondisk_log_start":"446'3800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13882,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5536,"num_bytes_recovered":362807296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":2},{"shards":"10","objects":2812}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1259,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-29T14:28:41.328698+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:28:41.328698+0000","last_peered":"2024-04-29T14:28:41.328698+0000","last_clean":"2024-04-29T14:28:41.328698+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:28:41.328698+0000","last_undegraded":"2024-04-29T14:28:41.328698+0000","last_fullsized":"2024-04-29T14:28:41.328698+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2800025624,"num_objects":42726,"num_object_clones":0,"num_object_copies":85452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224202,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28660,"num_bytes_recovered":1878261760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7462461440,"data_stored":7462453296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160919,"ondisk_log_size":160919,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122309,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888920,"kb_used_data":338512,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92856232,"statfs":{"total":95995035648,"available":95084781568,"internally_reserved":0,"allocated":346636288,"data_stored":346084111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":563608004},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383945,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":998800,"kb_used_data":548292,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92746352,"statfs":{"total":95995035648,"available":94972264448,"internally_reserved":0,"allocated":561451008,"data_stored":560906455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":461240768},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645583,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948048,"kb_used_data":515652,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92797104,"statfs":{"total":95995035648,"available":95024234496,"internally_reserved":0,"allocated":528027648,"data_stored":527483115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234264,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054936,"kb_used_data":522576,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92690216,"statfs":{"total":95995035648,"available":94914781184,"internally_reserved":0,"allocated":535117824,"data_stored":534565667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":545126858},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528605,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093452,"kb_used_data":516676,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92651700,"statfs":{"total":95995035648,"available":94875340800,"internally_reserved":0,"allocated":529076224,"data_stored":528531691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575158,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1721484,"kb_used_data":1184964,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92023668,"statfs":{"total":95995035648,"available":94232236032,"internally_reserved":0,"allocated":1213403136,"data_stored":1212858603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986473,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707020,"kb_used_data":1021956,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92038132,"statfs":{"total":95995035648,"available":94247047168,"internally_reserved":0,"allocated":1046482944,"data_stored":1045938411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1207500800,"data_stored":1207500800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1037500416,"data_stored":1037500416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":527826944,"data_stored":527826944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":525733888,"data_stored":525729816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":519962624,"data_stored":519962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":550436864,"data_stored":550436864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":343805952,"data_stored":343801880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:28:45.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:45.055 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:45.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:45.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:45.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:28:45.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:45.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:45.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:45.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:45.377 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:45.446 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:45.565 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:45.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:45.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:45.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:45.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:46.119 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:46.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:46.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:28:46.259 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:46.372 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:28:46.408 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:28:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:46.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:28:46.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:46.551 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:46.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:46 smithi194 ceph-mon[44089]: osdmap e511: 12 total, 11 up, 7 in 2024-04-29T14:28:46.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/363796063' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:46.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:46 smithi194 ceph-mon[44089]: from='client.36245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:46.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:46 smithi194 ceph-mon[44089]: pgmap v1351: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.5 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 723 op/s; 12789/81530 objects degraded (15.686%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:28:46.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:46.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:46 smithi040 ceph-mon[39581]: osdmap e511: 12 total, 11 up, 7 in 2024-04-29T14:28:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/363796063' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:46 smithi040 ceph-mon[39581]: from='client.36245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:46.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:46 smithi040 ceph-mon[39581]: pgmap v1351: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.5 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 723 op/s; 12789/81530 objects degraded (15.686%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:28:46.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:46 smithi121 ceph-mon[44533]: osdmap e511: 12 total, 11 up, 7 in 2024-04-29T14:28:46.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/363796063' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:46.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:46 smithi121 ceph-mon[44533]: from='client.36245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:28:46.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:46 smithi121 ceph-mon[44533]: pgmap v1351: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.5 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 723 op/s; 12789/81530 objects degraded (15.686%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:28:46.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:46.725 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:46.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:46.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:46.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:46.883 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:47.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:47.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:47.367 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:28:47.367 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:28:47.374 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:47.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:47.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:47.507 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:28:47.553 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:47.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:47 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:47.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:47 smithi040 ceph-mon[39581]: from='client.36245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:47.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:47 smithi040 ceph-mon[39581]: osdmap e512: 12 total, 11 up, 7 in 2024-04-29T14:28:47.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:47 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 12789/81530 objects degraded (15.686%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:47.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:47 smithi040 ceph-mon[39581]: osdmap e513: 12 total, 11 up, 7 in 2024-04-29T14:28:47.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:47 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:47.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:47 smithi121 ceph-mon[44533]: from='client.36245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:47.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:47 smithi121 ceph-mon[44533]: osdmap e512: 12 total, 11 up, 7 in 2024-04-29T14:28:47.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:47 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 12789/81530 objects degraded (15.686%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:47.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:47 smithi121 ceph-mon[44533]: osdmap e513: 12 total, 11 up, 7 in 2024-04-29T14:28:47.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:47 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:47.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:47 smithi194 ceph-mon[44089]: from='client.36245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:28:47.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:47 smithi194 ceph-mon[44089]: osdmap e512: 12 total, 11 up, 7 in 2024-04-29T14:28:47.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:47 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 12789/81530 objects degraded (15.686%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:47.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:47 smithi194 ceph-mon[44089]: osdmap e513: 12 total, 11 up, 7 in 2024-04-29T14:28:47.931 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:28:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:47.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:28:48.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:28:48.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:48.091 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:48.197 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:48.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:48.301 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:48.363 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:48.539 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:28:48.569 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:28:48.596 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:28:48.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:48 smithi040 ceph-mon[39581]: pgmap v1354: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.4 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 12011/77502 objects degraded (15.498%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:28:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:28:48.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:48 smithi121 ceph-mon[44533]: pgmap v1354: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.4 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 12011/77502 objects degraded (15.498%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:28:48.646 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:28:48.667 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:28:48.694 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:28:48.741 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:28:48.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:28:48.793 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:28:48.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:48 smithi194 ceph-mon[44089]: pgmap v1354: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.4 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 12011/77502 objects degraded (15.498%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:28:48.822 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:28:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:28:49.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:28:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/60413370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/60413370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:49.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/60413370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:49.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:28:50.019 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:28:50.519 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:50.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:50.560 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:50.562 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:28:50.624 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:50.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:50.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:50 smithi194 ceph-mon[44089]: pgmap v1355: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.3 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 676 op/s; 11740/75892 objects degraded (15.469%); 13 MiB/s, 215 objects/s recovering 2024-04-29T14:28:50.687 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:50.745 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:50.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:50.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:50 smithi040 ceph-mon[39581]: pgmap v1355: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.3 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 676 op/s; 11740/75892 objects degraded (15.469%); 13 MiB/s, 215 objects/s recovering 2024-04-29T14:28:50.856 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:50.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:50.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:50 smithi121 ceph-mon[44533]: pgmap v1355: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.3 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 676 op/s; 11740/75892 objects degraded (15.469%); 13 MiB/s, 215 objects/s recovering 2024-04-29T14:28:50.932 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:50.971 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:50.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:51.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:28:51.107 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:28:51.114 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:28:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:51.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:28:51.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:51.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:28:51.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:28:51.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:51 smithi194 ceph-mon[44089]: osdmap e514: 12 total, 11 up, 7 in 2024-04-29T14:28:51.724 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:51.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:28:51.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:28:51.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:51 smithi040 ceph-mon[39581]: osdmap e514: 12 total, 11 up, 7 in 2024-04-29T14:28:51.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:28:51.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:28:51.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:51 smithi121 ceph-mon[44533]: osdmap e514: 12 total, 11 up, 7 in 2024-04-29T14:28:52.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:28:52.082 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:28:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:52 smithi040 ceph-mon[39581]: pgmap v1357: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 10715/69254 objects degraded (15.472%); 22 MiB/s, 345 objects/s recovering 2024-04-29T14:28:52.766 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:52 smithi040 ceph-mon[39581]: from='client.16674 v1:172.21.15.40:0/2921140260' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:52.766 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:52 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 11740/75892 objects degraded (15.469%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:52.766 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:52 smithi040 ceph-mon[39581]: osdmap e515: 12 total, 11 up, 7 in 2024-04-29T14:28:52.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:52 smithi194 ceph-mon[44089]: pgmap v1357: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 10715/69254 objects degraded (15.472%); 22 MiB/s, 345 objects/s recovering 2024-04-29T14:28:52.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:52 smithi194 ceph-mon[44089]: from='client.16674 v1:172.21.15.40:0/2921140260' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:52.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:52 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 11740/75892 objects degraded (15.469%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:52.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:52 smithi194 ceph-mon[44089]: osdmap e515: 12 total, 11 up, 7 in 2024-04-29T14:28:52.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:52 smithi121 ceph-mon[44533]: pgmap v1357: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 10715/69254 objects degraded (15.472%); 22 MiB/s, 345 objects/s recovering 2024-04-29T14:28:52.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:52 smithi121 ceph-mon[44533]: from='client.16674 v1:172.21.15.40:0/2921140260' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:52.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:52 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 11740/75892 objects degraded (15.469%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:52.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:52 smithi121 ceph-mon[44533]: osdmap e515: 12 total, 11 up, 7 in 2024-04-29T14:28:53.158 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1357,"stamp":"2024-04-29T14:28:51.997555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2269708856,"num_objects":34627,"num_object_clones":0,"num_object_copies":69254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34627,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":232484,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31234,"num_bytes_recovered":2048542848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161057,"ondisk_log_size":161057,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7591524,"kb_used_data":3784476,"kb_used_omap":10,"kb_used_meta":3806901,"kb_avail":648624540,"statfs":{"total":671965249536,"available":664191528960,"internally_reserved":0,"allocated":3875303424,"data_stored":3871188895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3898266718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-402259968,"num_objects":-6138,"num_object_clones":0,"num_object_copies":-12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2074,"num_bytes_recovered":135921664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001110"},"pg_stats":[{"pgid":"5.f","version":"514'14868","reported_seq":27971,"reported_epoch":514,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:51.565329+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:28:51.565329+0000","last_peered":"2024-04-29T14:28:51.565329+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:28:51.565329+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:28:51.565329+0000","mapping_epoch":467,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137363456,"num_objects":2096,"num_object_clones":0,"num_object_copies":4192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2063,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14868,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6419,"num_bytes_recovered":420675584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":2063},{"shards":"4,11","objects":33}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"514'14631","reported_seq":14883,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:50.776426+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:28:50.776426+0000","last_peered":"2024-04-29T14:28:50.776426+0000","last_clean":"2024-04-29T14:28:50.776426+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:28:50.776426+0000","last_undegraded":"2024-04-29T14:28:50.776426+0000","last_fullsized":"2024-04-29T14:28:50.776426+0000","mapping_epoch":432,"log_start":"447'4600","ondisk_log_start":"447'4600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141754368,"num_objects":2163,"num_object_clones":0,"num_object_copies":4326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14631,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"514'14564","reported_seq":27368,"reported_epoch":514,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:28:51.286139+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:28:51.286139+0000","last_peered":"2024-04-29T14:28:51.286139+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:28:51.286139+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:28:51.286139+0000","mapping_epoch":468,"log_start":"447'4500","ondisk_log_start":"447'4500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139526168,"num_objects":2130,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14564,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6274,"num_bytes_recovered":411172864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":58},{"shards":"11","objects":2072}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"514'14656","reported_seq":14921,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:51.532937+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:28:51.532937+0000","last_peered":"2024-04-29T14:28:51.532937+0000","last_clean":"2024-04-29T14:28:51.532937+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:28:51.532937+0000","last_undegraded":"2024-04-29T14:28:51.532937+0000","last_fullsized":"2024-04-29T14:28:51.532937+0000","mapping_epoch":432,"log_start":"448'4600","ondisk_log_start":"448'4600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139984896,"num_objects":2136,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14656,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"514'14532","reported_seq":14791,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:50.769302+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:28:50.769302+0000","last_peered":"2024-04-29T14:28:50.769302+0000","last_clean":"2024-04-29T14:28:50.769302+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:28:50.769302+0000","last_undegraded":"2024-04-29T14:28:50.769302+0000","last_fullsized":"2024-04-29T14:28:50.769302+0000","mapping_epoch":432,"log_start":"447'4500","ondisk_log_start":"447'4500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139853824,"num_objects":2134,"num_object_clones":0,"num_object_copies":4268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14532,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"513'16568","reported_seq":16956,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-29T14:28:48.797926+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:28:48.797926+0000","last_peered":"2024-04-29T14:28:48.797926+0000","last_clean":"2024-04-29T14:28:48.797926+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:28:48.797926+0000","last_undegraded":"2024-04-29T14:28:48.797926+0000","last_fullsized":"2024-04-29T14:28:48.797926+0000","mapping_epoch":432,"log_start":"454'6500","ondisk_log_start":"454'6500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146341888,"num_objects":2233,"num_object_clones":0,"num_object_copies":4466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14313,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"514'14462","reported_seq":14723,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:51.621960+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:28:51.621960+0000","last_peered":"2024-04-29T14:28:51.621960+0000","last_clean":"2024-04-29T14:28:51.621960+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:28:51.621960+0000","last_undegraded":"2024-04-29T14:28:51.621960+0000","last_fullsized":"2024-04-29T14:28:51.621960+0000","mapping_epoch":432,"log_start":"447'4400","ondisk_log_start":"447'4400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138936320,"num_objects":2120,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14462,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"514'14546","reported_seq":14803,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:51.282251+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:28:51.282251+0000","last_peered":"2024-04-29T14:28:51.282251+0000","last_clean":"2024-04-29T14:28:51.282251+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:28:51.282251+0000","last_undegraded":"2024-04-29T14:28:51.282251+0000","last_fullsized":"2024-04-29T14:28:51.282251+0000","mapping_epoch":432,"log_start":"447'4500","ondisk_log_start":"447'4500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135135232,"num_objects":2062,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14546,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"514'14611","reported_seq":27310,"reported_epoch":514,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:50.826893+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:28:50.826893+0000","last_peered":"2024-04-29T14:28:50.826893+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:28:50.826893+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:28:50.826893+0000","mapping_epoch":467,"log_start":"448'4600","ondisk_log_start":"448'4600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143327232,"num_objects":2187,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14611,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6220,"num_bytes_recovered":407633920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":2179},{"shards":"9,10","objects":8}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"514'14628","reported_seq":27410,"reported_epoch":514,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:51.636146+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:28:51.636146+0000","last_peered":"2024-04-29T14:28:51.636146+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:28:51.636146+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:28:51.636146+0000","mapping_epoch":467,"log_start":"448'4600","ondisk_log_start":"448'4600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140115968,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6259,"num_bytes_recovered":410189824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":2107},{"shards":"7,10","objects":31}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"514'14636","reported_seq":14898,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:51.568177+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:28:51.568177+0000","last_peered":"2024-04-29T14:28:51.568177+0000","last_clean":"2024-04-29T14:28:51.568177+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:28:51.568177+0000","last_undegraded":"2024-04-29T14:28:51.568177+0000","last_fullsized":"2024-04-29T14:28:51.568177+0000","mapping_epoch":432,"log_start":"448'4600","ondisk_log_start":"448'4600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139198464,"num_objects":2124,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14636,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"513'14465","reported_seq":14825,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-29T14:28:50.329116+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:28:50.329116+0000","last_peered":"2024-04-29T14:28:50.329116+0000","last_clean":"2024-04-29T14:28:50.329116+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:28:50.329116+0000","last_undegraded":"2024-04-29T14:28:50.329116+0000","last_fullsized":"2024-04-29T14:28:50.329116+0000","mapping_epoch":432,"log_start":"447'4400","ondisk_log_start":"447'4400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:53.158953+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140574720,"num_objects":2145,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14465,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"513'14380","reported_seq":14700,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-29T14:28:50.381709+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:28:50.381709+0000","last_peered":"2024-04-29T14:28:50.381709+0000","last_clean":"2024-04-29T14:28:50.381709+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:28:50.381709+0000","last_undegraded":"2024-04-29T14:28:50.381709+0000","last_fullsized":"2024-04-29T14:28:50.381709+0000","mapping_epoch":432,"log_start":"447'4300","ondisk_log_start":"447'4300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144834560,"num_objects":2210,"num_object_clones":0,"num_object_copies":4420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14380,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"514'14289","reported_seq":14554,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:51.540090+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:28:51.540090+0000","last_peered":"2024-04-29T14:28:51.540090+0000","last_clean":"2024-04-29T14:28:51.540090+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:28:51.540090+0000","last_undegraded":"2024-04-29T14:28:51.540090+0000","last_fullsized":"2024-04-29T14:28:51.540090+0000","mapping_epoch":432,"log_start":"447'4200","ondisk_log_start":"447'4200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141361152,"num_objects":2157,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"513'17289","reported_seq":17724,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-04-29T14:28:48.809929+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:28:48.809929+0000","last_peered":"2024-04-29T14:28:48.809929+0000","last_clean":"2024-04-29T14:28:48.809929+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:28:48.809929+0000","last_undegraded":"2024-04-29T14:28:48.809929+0000","last_fullsized":"2024-04-29T14:28:48.809929+0000","mapping_epoch":432,"log_start":"454'7200","ondisk_log_start":"454'7200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150011904,"num_objects":2289,"num_object_clones":0,"num_object_copies":4578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14327,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"513'14395","reported_seq":26681,"reported_epoch":513,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:48.824379+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:28:48.824379+0000","last_peered":"2024-04-29T14:28:48.824379+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:28:48.824379+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:28:48.824379+0000","mapping_epoch":467,"log_start":"447'4300","ondisk_log_start":"447'4300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150798336,"num_objects":2301,"num_object_clones":0,"num_object_copies":4602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14395,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6054,"num_bytes_recovered":396754944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":7},{"shards":"10","objects":2294}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1271,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-04-29T14:28:50.404981+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:28:50.404981+0000","last_peered":"2024-04-29T14:28:50.404981+0000","last_clean":"2024-04-29T14:28:50.404981+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:28:50.404981+0000","last_undegraded":"2024-04-29T14:28:50.404981+0000","last_fullsized":"2024-04-29T14:28:50.404981+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2269118488,"num_objects":34625,"num_object_clones":0,"num_object_copies":69250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34625,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232303,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31226,"num_bytes_recovered":2046427136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6567370752,"data_stored":6567362608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160920,"ondisk_log_size":160920,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122311,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831920,"kb_used_data":273124,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92913232,"statfs":{"total":95995035648,"available":95143149568,"internally_reserved":0,"allocated":279678976,"data_stored":279083054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":572127684},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383947,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895064,"kb_used_data":448780,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92850088,"statfs":{"total":95995035648,"available":95078490112,"internally_reserved":0,"allocated":459550720,"data_stored":458968657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":456980928},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645585,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859540,"kb_used_data":422988,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92885612,"statfs":{"total":95995035648,"available":95114866688,"internally_reserved":0,"allocated":433139712,"data_stored":432557669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422889,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234266,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988080,"kb_used_data":425188,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92757072,"statfs":{"total":95995035648,"available":94983241728,"internally_reserved":0,"allocated":435392512,"data_stored":434796610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":576387530},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528607,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009048,"kb_used_data":428108,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92736104,"statfs":{"total":95995035648,"available":94961770496,"internally_reserved":0,"allocated":438382592,"data_stored":437800549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575160,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501920,"kb_used_data":961240,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92243232,"statfs":{"total":95995035648,"available":94457069568,"internally_reserved":0,"allocated":984309760,"data_stored":983721482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497117,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986475,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1505952,"kb_used_data":825048,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92239200,"statfs":{"total":95995035648,"available":94452940800,"internally_reserved":0,"allocated":844849152,"data_stored":844260874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":977207296,"data_stored":977207296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":836632576,"data_stored":836632576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":429064192,"data_stored":429064192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":423694336,"data_stored":423690264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":424804352,"data_stored":424804352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":449183744,"data_stored":449183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":277090304,"data_stored":277086232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:28:53.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:28:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:54.015 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:28:54.195 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:54.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:54.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:54.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2227653900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:54.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:54 smithi194 ceph-mon[44089]: from='client.36254 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:54.340 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:54.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2227653900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:54.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:54 smithi121 ceph-mon[44533]: from='client.36254 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2227653900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:54.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:54 smithi040 ceph-mon[39581]: from='client.36254 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:28:54.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:54.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:54.473 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:54.528 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:55.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:55 smithi194 ceph-mon[44089]: pgmap v1359: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 722 op/s; 10444/67714 objects degraded (15.424%); 14 MiB/s, 231 objects/s recovering 2024-04-29T14:28:55.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:55 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:55.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:55 smithi194 ceph-mon[44089]: from='client.36254 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:28:55.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:55 smithi194 ceph-mon[44089]: osdmap e516: 12 total, 11 up, 7 in 2024-04-29T14:28:55.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:55 smithi040 ceph-mon[39581]: pgmap v1359: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 722 op/s; 10444/67714 objects degraded (15.424%); 14 MiB/s, 231 objects/s recovering 2024-04-29T14:28:55.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:55 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:55.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:55 smithi040 ceph-mon[39581]: from='client.36254 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:28:55.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:55 smithi040 ceph-mon[39581]: osdmap e516: 12 total, 11 up, 7 in 2024-04-29T14:28:55.376 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:55 smithi121 ceph-mon[44533]: pgmap v1359: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 722 op/s; 10444/67714 objects degraded (15.424%); 14 MiB/s, 231 objects/s recovering 2024-04-29T14:28:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:55 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:28:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:55 smithi121 ceph-mon[44533]: from='client.36254 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:28:55.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:55 smithi121 ceph-mon[44533]: osdmap e516: 12 total, 11 up, 7 in 2024-04-29T14:28:55.712 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:55.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:55.822 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:28:55.896 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:56.016 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:28:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:56.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:28:56.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:56.159 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:56.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:56.520 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:56.550 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:237079 5.d 5:b082d880:::benchmark_data_smithi194_121512_object103613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:56.547867+0000", 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0067692500000000001, 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068012530000000002, 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:28:56.554 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 237079 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.547867+0000", 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.547867+0000", 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.547869+0000", 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.844e-06 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.547876+0000", 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5590000000000002e-06 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.547887+0000", 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1049e-05 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.547894+0000", 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4610000000000002e-06 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.547998+0000", 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103819 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548132+0000", 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013351500000000001 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548265+0000", 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013353700000000001 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(5.d 516/468 [PushReplyOp(5:b6786bc7:::benchmark_data_smithi194_121512_object104338:head)])", 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:56.548825+0000", 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0058116039999999997, 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059298210000000001, 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:28:56.558 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548825+0000", 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548825+0000", 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:28:56.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548826+0000", 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.412e-06 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548832+0000", 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4280000000000004e-06 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548840+0000", 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1969999999999996e-06 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.548846+0000", 2024-04-29T14:28:56.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8880000000000002e-06 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.553012+0000", 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041663120000000001 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:237081 5.d 5.cfb600bd (undecoded) ondisk+write+known_if_redirected e516)", 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:56.551837+0000", 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.002799382, 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029688900000000001, 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:28:56.561 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "queued for pg", 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 237081 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.551837+0000", 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:28:56.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.551837+0000", 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.551840+0000", 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9799999999999998e-06 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.551848+0000", 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9719999999999992e-06 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:28:56.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.551862+0000", 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4268e-05 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.551872+0000", 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8179999999999999e-06 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26047.0:237084 5.f e516/467 5:f190d90d:::benchmark_data_smithi194_121512_object103618:head v 516'15137, mlcod=454'6670)", 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:56.552333+0000", 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0023033319999999999, 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025256219999999999, 2024-04-29T14:28:56.564 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552333+0000", 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552333+0000", 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:28:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552335+0000", 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.412e-06 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552343+0000", 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7539999999999995e-06 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552356+0000", 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2279999999999999e-05 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552363+0000", 2024-04-29T14:28:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7810000000000005e-06 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552670+0000", 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030613199999999998 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.552695+0000", 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5137e-05 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.26047.0:237079 5.d e516/468)", 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:56.553974+0000", 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.00066231199999999997, 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00093831999999999997, 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "queued for pg", 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.553974+0000", 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:28:56.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.553974+0000", 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.553976+0000", 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2839999999999999e-06 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.553983+0000", 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3270000000000003e-06 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.553995+0000", 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.202e-05 2024-04-29T14:28:56.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:56.554002+0000", 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.764e-06 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 5 2024-04-29T14:28:56.570 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:28:56.702 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:56.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:56.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:28:56.842 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:28:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:56.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:28:57.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:28:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:57.214 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:28:57.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:57 smithi040 ceph-mon[39581]: pgmap v1361: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 10167/65078 objects degraded (15.623%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:28:57.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:57.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:28:57.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:57 smithi194 ceph-mon[44089]: pgmap v1361: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 10167/65078 objects degraded (15.623%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:28:57.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:57.369 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:57 smithi121 ceph-mon[44533]: pgmap v1361: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 10167/65078 objects degraded (15.623%); 16 MiB/s, 262 objects/s recovering 2024-04-29T14:28:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:57.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:57.493 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:28:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:28:57.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:28:57.555 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:57.585 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:28:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:57.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:28:57.619 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:28:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:28:57.674 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:28:57.699 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:57.702 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:28:57.727 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:28:57.749 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:28:57.763 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:57.777 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:28:57.805 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:28:57.837 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:28:57.863 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:28:57.889 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:28:57.890 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:28:57.890 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:28:57.890 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:28:57.890 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:28:58.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 10167/65078 objects degraded (15.623%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:58.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osdmap e517: 12 total, 11 up, 7 in 2024-04-29T14:28:58.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1931250997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:58.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.030 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.030 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.030 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.030 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.030 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.030 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 10167/65078 objects degraded (15.623%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:58.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osdmap e517: 12 total, 11 up, 7 in 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1931250997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 10167/65078 objects degraded (15.623%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osdmap e517: 12 total, 11 up, 7 in 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1931250997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:58.630 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1362,"stamp":"2024-04-29T14:28:57.220579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1999766072,"num_objects":30508,"num_object_clones":0,"num_object_copies":61016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30508,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":236603,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32566,"num_bytes_recovered":2135836800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7111528,"kb_used_data":3333532,"kb_used_omap":10,"kb_used_meta":3777781,"kb_avail":649104536,"statfs":{"total":671965249536,"available":664683044864,"internally_reserved":0,"allocated":3413536768,"data_stored":3409321829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3868447838},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,5,7,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-457179136,"num_objects":-6976,"num_object_clones":0,"num_object_copies":-13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2357,"num_bytes_recovered":154468352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.822538"},"pg_stats":[{"pgid":"5.f","version":"516'15138","reported_seq":28751,"reported_epoch":516,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:56.578479+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:28:56.578479+0000","last_peered":"2024-04-29T14:28:56.578479+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:28:56.578479+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:28:56.578479+0000","mapping_epoch":467,"log_start":"449'5100","ondisk_log_start":"449'5100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119668736,"num_objects":1826,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15138,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6672,"num_bytes_recovered":437256192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1810},{"shards":"4,11","objects":16}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"516'14901","reported_seq":15157,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:55.839841+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:28:55.839841+0000","last_peered":"2024-04-29T14:28:55.839841+0000","last_clean":"2024-04-29T14:28:55.839841+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:28:55.839841+0000","last_undegraded":"2024-04-29T14:28:55.839841+0000","last_fullsized":"2024-04-29T14:28:55.839841+0000","mapping_epoch":432,"log_start":"448'4900","ondisk_log_start":"448'4900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124059648,"num_objects":1893,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"516'14845","reported_seq":28194,"reported_epoch":516,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:28:56.292474+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:28:56.292474+0000","last_peered":"2024-04-29T14:28:56.292474+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:28:56.292474+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:28:56.292474+0000","mapping_epoch":468,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121110552,"num_objects":1849,"num_object_clones":0,"num_object_copies":3698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1802,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14845,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6544,"num_bytes_recovered":428867584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":47},{"shards":"11","objects":1802}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"516'14896","reported_seq":15165,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:56.547800+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:28:56.547800+0000","last_peered":"2024-04-29T14:28:56.547800+0000","last_clean":"2024-04-29T14:28:56.547800+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:28:56.547800+0000","last_undegraded":"2024-04-29T14:28:56.547800+0000","last_fullsized":"2024-04-29T14:28:56.547800+0000","mapping_epoch":432,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124256256,"num_objects":1896,"num_object_clones":0,"num_object_copies":3792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14896,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"516'14808","reported_seq":15071,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:55.821611+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:28:55.821611+0000","last_peered":"2024-04-29T14:28:55.821611+0000","last_clean":"2024-04-29T14:28:55.821611+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:28:55.821611+0000","last_undegraded":"2024-04-29T14:28:55.821611+0000","last_fullsized":"2024-04-29T14:28:55.821611+0000","mapping_epoch":432,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121765888,"num_objects":1858,"num_object_clones":0,"num_object_copies":3716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14808,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"515'16811","reported_seq":17202,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-04-29T14:28:53.821451+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:28:53.821451+0000","last_peered":"2024-04-29T14:28:53.821451+0000","last_clean":"2024-04-29T14:28:53.821451+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:28:53.821451+0000","last_undegraded":"2024-04-29T14:28:53.821451+0000","last_fullsized":"2024-04-29T14:28:53.821451+0000","mapping_epoch":432,"log_start":"454'6800","ondisk_log_start":"454'6800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130416640,"num_objects":1990,"num_object_clones":0,"num_object_copies":3980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14556,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"516'14709","reported_seq":14975,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:56.614690+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:28:56.614690+0000","last_peered":"2024-04-29T14:28:56.614690+0000","last_clean":"2024-04-29T14:28:56.614690+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:28:56.614690+0000","last_undegraded":"2024-04-29T14:28:56.614690+0000","last_fullsized":"2024-04-29T14:28:56.614690+0000","mapping_epoch":432,"log_start":"448'4700","ondisk_log_start":"448'4700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122748928,"num_objects":1873,"num_object_clones":0,"num_object_copies":3746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14709,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"516'14788","reported_seq":15049,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:56.262688+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:28:56.262688+0000","last_peered":"2024-04-29T14:28:56.262688+0000","last_clean":"2024-04-29T14:28:56.262688+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:28:56.262688+0000","last_undegraded":"2024-04-29T14:28:56.262688+0000","last_fullsized":"2024-04-29T14:28:56.262688+0000","mapping_epoch":432,"log_start":"448'4700","ondisk_log_start":"448'4700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119275520,"num_objects":1820,"num_object_clones":0,"num_object_copies":3640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14788,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"516'14893","reported_seq":28150,"reported_epoch":516,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:55.838810+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:28:55.838810+0000","last_peered":"2024-04-29T14:28:55.838810+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:28:55.838810+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:28:55.838810+0000","mapping_epoch":467,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124846080,"num_objects":1905,"num_object_clones":0,"num_object_copies":3810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14893,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6497,"num_bytes_recovered":425787392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":1902},{"shards":"9,10","objects":3}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"516'14874","reported_seq":28182,"reported_epoch":516,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:56.644901+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:28:56.644901+0000","last_peered":"2024-04-29T14:28:56.644901+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:28:56.644901+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:28:56.644901+0000","mapping_epoch":467,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123994112,"num_objects":1892,"num_object_clones":0,"num_object_copies":3784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14874,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6520,"num_bytes_recovered":427294720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":1846},{"shards":"7,10","objects":46}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"516'14882","reported_seq":15148,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:56.532473+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:28:56.532473+0000","last_peered":"2024-04-29T14:28:56.532473+0000","last_clean":"2024-04-29T14:28:56.532473+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:28:56.532473+0000","last_undegraded":"2024-04-29T14:28:56.532473+0000","last_fullsized":"2024-04-29T14:28:56.532473+0000","mapping_epoch":432,"log_start":"448'4800","ondisk_log_start":"448'4800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123076608,"num_objects":1878,"num_object_clones":0,"num_object_copies":3756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14882,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"516'14711","reported_seq":15077,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:55.336459+0000","last_change":"2024-04-29T14:27:50.011031+0000","last_active":"2024-04-29T14:28:55.336459+0000","last_peered":"2024-04-29T14:28:55.336459+0000","last_clean":"2024-04-29T14:28:55.336459+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:28:55.336459+0000","last_undegraded":"2024-04-29T14:28:55.336459+0000","last_fullsized":"2024-04-29T14:28:55.336459+0000","mapping_epoch":432,"log_start":"448'4700","ondisk_log_start":"448'4700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'11730","last_scrub_stamp":"2024-04-29T14:27:50.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:50.010733+0000","objects_scrubbed":4896,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:58.384523+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124452864,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14711,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"516'14624","reported_seq":14951,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:55.335819+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:28:55.335819+0000","last_peered":"2024-04-29T14:28:55.335819+0000","last_clean":"2024-04-29T14:28:55.335819+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:28:55.335819+0000","last_undegraded":"2024-04-29T14:28:55.335819+0000","last_fullsized":"2024-04-29T14:28:55.335819+0000","mapping_epoch":432,"log_start":"448'4600","ondisk_log_start":"448'4600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128843776,"num_objects":1966,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14624,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"516'14548","reported_seq":14817,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:56.528553+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:28:56.528553+0000","last_peered":"2024-04-29T14:28:56.528553+0000","last_clean":"2024-04-29T14:28:56.528553+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:28:56.528553+0000","last_undegraded":"2024-04-29T14:28:56.528553+0000","last_fullsized":"2024-04-29T14:28:56.528553+0000","mapping_epoch":432,"log_start":"448'4500","ondisk_log_start":"448'4500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124387328,"num_objects":1898,"num_object_clones":0,"num_object_copies":3796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14548,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"515'17542","reported_seq":17981,"reported_epoch":515,"state":"active+clean","last_fresh":"2024-04-29T14:28:53.782688+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:28:53.782688+0000","last_peered":"2024-04-29T14:28:53.782688+0000","last_clean":"2024-04-29T14:28:53.782688+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:28:53.782688+0000","last_undegraded":"2024-04-29T14:28:53.782688+0000","last_fullsized":"2024-04-29T14:28:53.782688+0000","mapping_epoch":432,"log_start":"455'7500","ondisk_log_start":"455'7500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133431296,"num_objects":2036,"num_object_clones":0,"num_object_copies":4072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14580,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"515'14669","reported_seq":27502,"reported_epoch":515,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:28:53.824316+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:28:53.824316+0000","last_peered":"2024-04-29T14:28:53.824316+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:28:53.824316+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:28:53.824316+0000","mapping_epoch":467,"log_start":"448'4600","ondisk_log_start":"448'4600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132841472,"num_objects":2027,"num_object_clones":0,"num_object_copies":4054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14669,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6325,"num_bytes_recovered":414515200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":4},{"shards":"10","objects":2023}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1275,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-04-29T14:28:54.014800+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:28:54.014800+0000","last_peered":"2024-04-29T14:28:54.014800+0000","last_clean":"2024-04-29T14:28:54.014800+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:28:54.014800+0000","last_undegraded":"2024-04-29T14:28:54.014800+0000","last_fullsized":"2024-04-29T14:28:54.014800+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1999175704,"num_objects":30506,"num_object_clones":0,"num_object_copies":61012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9383,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236422,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32558,"num_bytes_recovered":2133721088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6112944128,"data_stored":6112935984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122312,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801220,"kb_used_data":238332,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92943932,"statfs":{"total":95995035648,"available":95174586368,"internally_reserved":0,"allocated":244051968,"data_stored":243443476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":576387524},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383948,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855148,"kb_used_data":400548,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92890004,"statfs":{"total":95995035648,"available":95119364096,"internally_reserved":0,"allocated":410161152,"data_stored":409566569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":465500608},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645586,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794684,"kb_used_data":370480,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92950468,"statfs":{"total":95995035648,"available":95181279232,"internally_reserved":0,"allocated":379371520,"data_stored":378770672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":434239936},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422890,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234267,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925256,"kb_used_data":374844,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92819896,"statfs":{"total":95995035648,"available":95047573504,"internally_reserved":0,"allocated":383840256,"data_stored":383231784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":563608010},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528608,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940348,"kb_used_data":371888,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92804804,"statfs":{"total":95995035648,"available":95032119296,"internally_reserved":0,"allocated":380813312,"data_stored":380212464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":582089158},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575161,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382524,"kb_used_data":850160,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92362628,"statfs":{"total":95995035648,"available":94579331072,"internally_reserved":0,"allocated":870563840,"data_stored":869962992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497119,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986476,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1412348,"kb_used_data":727280,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92332804,"statfs":{"total":95995035648,"available":94548791296,"internally_reserved":0,"allocated":744734720,"data_stored":744133872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":862060544,"data_stored":862060544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":737280000,"data_stored":737280000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":379453440,"data_stored":379453440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":373886976,"data_stored":373882904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":370606080,"data_stored":370606080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":398655488,"data_stored":398655488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":241307648,"data_stored":241303576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.f is not active+clean 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.f', 'version': "516'15138", 'reported_seq': 28751, 'reported_epoch': 516, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:56.578479+0000', 'last_change': '2024-04-29T14:26:16.020854+0000', 'last_active': '2024-04-29T14:28:56.578479+0000', 'last_peered': '2024-04-29T14:28:56.578479+0000', 'last_clean': '2024-04-29T14:25:54.182894+0000', 'last_became_active': '2024-04-29T14:26:16.010174+0000', 'last_became_peered': '2024-04-29T14:26:16.010174+0000', 'last_unstale': '2024-04-29T14:28:56.578479+0000', 'last_undegraded': '2024-04-29T14:26:15.821893+0000', 'last_fullsized': '2024-04-29T14:28:56.578479+0000', 'mapping_epoch': 467, 'log_start': "449'5100", 'ondisk_log_start': "449'5100", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'7968", 'last_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:44.698252+0000', 'objects_scrubbed': 7919, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2114, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 119668736, 'num_objects': 1826, 'num_object_clones': 0, 'num_object_copies': 3652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1810, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15138, 'num_write_kb': 542848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6672, 'num_bytes_recovered': 437256192, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 11], 'acting': [4, 11], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 1810}, {'shards': '4,11', 'objects': 16}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.d is not active+clean 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.d', 'version': "516'14845", 'reported_seq': 28194, 'reported_epoch': 516, 'state': 'active+recovering+degraded+remapped', 'last_fresh': '2024-04-29T14:28:56.292474+0000', 'last_change': '2024-04-29T14:26:17.048329+0000', 'last_active': '2024-04-29T14:28:56.292474+0000', 'last_peered': '2024-04-29T14:28:56.292474+0000', 'last_clean': '2024-04-29T14:25:54.192906+0000', 'last_became_active': '2024-04-29T14:26:17.022020+0000', 'last_became_peered': '2024-04-29T14:26:17.022020+0000', 'last_unstale': '2024-04-29T14:28:56.292474+0000', 'last_undegraded': '2024-04-29T14:26:16.808487+0000', 'last_fullsized': '2024-04-29T14:28:56.292474+0000', 'mapping_epoch': 468, 'log_start': "448'4800", 'ondisk_log_start': "448'4800", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "454'7099", 'last_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:36.136513+0000', 'objects_scrubbed': 7053, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1409, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 121110552, 'num_objects': 1849, 'num_object_clones': 0, 'num_object_copies': 3698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1802, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1849, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14845, 'num_write_kb': 534145, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6544, 'num_bytes_recovered': 428867584, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 11], 'acting': [11, 7], 'avail_no_missing': ['11'], 'object_location_counts': [{'shards': '7,11', 'objects': 47}, {'shards': '11', 'objects': 1802}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 11, 'purged_snaps': []} 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.7 is not active+clean 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.7', 'version': "516'14893", 'reported_seq': 28150, 'reported_epoch': 516, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:55.838810+0000', 'last_change': '2024-04-29T14:26:16.052811+0000', 'last_active': '2024-04-29T14:28:55.838810+0000', 'last_peered': '2024-04-29T14:28:55.838810+0000', 'last_clean': '2024-04-29T14:25:54.182628+0000', 'last_became_active': '2024-04-29T14:26:16.038648+0000', 'last_became_peered': '2024-04-29T14:26:16.038648+0000', 'last_unstale': '2024-04-29T14:28:55.838810+0000', 'last_undegraded': '2024-04-29T14:26:15.818605+0000', 'last_fullsized': '2024-04-29T14:28:55.838810+0000', 'mapping_epoch': 467, 'log_start': "448'4800", 'ondisk_log_start': "448'4800", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "455'7479", 'last_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:40.322101+0000', 'objects_scrubbed': 7424, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1679, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 124846080, 'num_objects': 1905, 'num_object_clones': 0, 'num_object_copies': 3810, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1902, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14893, 'num_write_kb': 537536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6497, 'num_bytes_recovered': 425787392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 10], 'acting': [9, 10], 'avail_no_missing': ['9'], 'object_location_counts': [{'shards': '9', 'objects': 1902}, {'shards': '9,10', 'objects': 3}], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.6 is not active+clean 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.6', 'version': "516'14874", 'reported_seq': 28182, 'reported_epoch': 516, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:56.644901+0000', 'last_change': '2024-04-29T14:26:16.126893+0000', 'last_active': '2024-04-29T14:28:56.644901+0000', 'last_peered': '2024-04-29T14:28:56.644901+0000', 'last_clean': '2024-04-29T14:25:53.790194+0000', 'last_became_active': '2024-04-29T14:26:16.054003+0000', 'last_became_peered': '2024-04-29T14:26:16.054003+0000', 'last_unstale': '2024-04-29T14:28:56.644901+0000', 'last_undegraded': '2024-04-29T14:26:15.820366+0000', 'last_fullsized': '2024-04-29T14:28:56.644901+0000', 'mapping_epoch': 467, 'log_start': "448'4800", 'ondisk_log_start': "448'4800", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "456'8070", 'last_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:25:49.240028+0000', 'objects_scrubbed': 8029, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1689, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 123994112, 'num_objects': 1892, 'num_object_clones': 0, 'num_object_copies': 3784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1846, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14874, 'num_write_kb': 536512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6520, 'num_bytes_recovered': 427294720, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 10], 'acting': [7, 10], 'avail_no_missing': ['7'], 'object_location_counts': [{'shards': '7', 'objects': 1846}, {'shards': '7,10', 'objects': 46}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.4 is not active+clean 2024-04-29T14:28:58.632 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.4', 'version': "515'14669", 'reported_seq': 27502, 'reported_epoch': 515, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:28:53.824316+0000', 'last_change': '2024-04-29T14:26:15.998964+0000', 'last_active': '2024-04-29T14:28:53.824316+0000', 'last_peered': '2024-04-29T14:28:53.824316+0000', 'last_clean': '2024-04-29T14:25:59.019434+0000', 'last_became_active': '2024-04-29T14:26:15.982158+0000', 'last_became_peered': '2024-04-29T14:26:15.982158+0000', 'last_unstale': '2024-04-29T14:28:53.824316+0000', 'last_undegraded': '2024-04-29T14:26:15.829402+0000', 'last_fullsized': '2024-04-29T14:28:53.824316+0000', 'mapping_epoch': 467, 'log_start': "448'4600", 'ondisk_log_start': "448'4600", 'created': 432, 'last_epoch_clean': 433, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'last_clean_scrub_stamp': '2024-04-29T14:24:29.217284+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 132841472, 'num_objects': 2027, 'num_object_clones': 0, 'num_object_copies': 4054, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2023, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14669, 'num_write_kb': 534272, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6325, 'num_bytes_recovered': 414515200, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 0], 'acting': [10, 0], 'avail_no_missing': ['10'], 'object_location_counts': [{'shards': '0,10', 'objects': 4}, {'shards': '10', 'objects': 2023}], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-04-29T14:28:58.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:28:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:28:59.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:28:59.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:59 smithi194 ceph-mon[44089]: from='client.16683 v1:172.21.15.40:0/1001831338' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:59.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:59.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:59.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:28:59 smithi194 ceph-mon[44089]: pgmap v1363: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 9383/61016 objects degraded (15.378%); 14 MiB/s, 221 objects/s recovering 2024-04-29T14:28:59.320 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:28:59.347 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:28:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:59.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:28:59.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:59 smithi040 ceph-mon[39581]: from='client.16683 v1:172.21.15.40:0/1001831338' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:59.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:59.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:59.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:28:59 smithi040 ceph-mon[39581]: pgmap v1363: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 9383/61016 objects degraded (15.378%); 14 MiB/s, 221 objects/s recovering 2024-04-29T14:28:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:59 smithi121 ceph-mon[44533]: from='client.16683 v1:172.21.15.40:0/1001831338' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:28:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:28:59.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:28:59 smithi121 ceph-mon[44533]: pgmap v1363: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 9383/61016 objects degraded (15.378%); 14 MiB/s, 221 objects/s recovering 2024-04-29T14:28:59.453 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:28:59.538 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:59.609 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:28:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:59.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:28:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:28:59.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:28:59.745 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:28:59.748 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:28:59.774 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:28:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:28:59.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:28:59.879 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:28:59.940 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:00.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:00.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:00 smithi121 ceph-mon[44533]: 5.3 scrub starts 2024-04-29T14:29:00.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:29:00.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:00.035 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:00 smithi121 ceph-mon[44533]: 5.3 scrub ok 2024-04-29T14:29:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:00.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:00.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:00.216 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:00.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:00.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:00 smithi194 ceph-mon[44089]: 5.3 scrub starts 2024-04-29T14:29:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:29:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:00.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:00 smithi194 ceph-mon[44089]: 5.3 scrub ok 2024-04-29T14:29:00.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:00 smithi040 ceph-mon[39581]: 5.3 scrub starts 2024-04-29T14:29:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:29:00.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:00.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:00 smithi040 ceph-mon[39581]: 5.3 scrub ok 2024-04-29T14:29:00.416 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:01.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:29:01.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:01 smithi194 ceph-mon[44089]: pgmap v1364: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.8 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 9080/59342 objects degraded (15.301%); 13 MiB/s, 210 objects/s recovering 2024-04-29T14:29:01.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:01 smithi194 ceph-mon[44089]: osdmap e518: 12 total, 11 up, 7 in 2024-04-29T14:29:01.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:01 smithi040 ceph-mon[39581]: pgmap v1364: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.8 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 9080/59342 objects degraded (15.301%); 13 MiB/s, 210 objects/s recovering 2024-04-29T14:29:01.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:01 smithi040 ceph-mon[39581]: osdmap e518: 12 total, 11 up, 7 in 2024-04-29T14:29:01.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:01 smithi121 ceph-mon[44533]: pgmap v1364: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.8 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 9080/59342 objects degraded (15.301%); 13 MiB/s, 210 objects/s recovering 2024-04-29T14:29:01.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:01 smithi121 ceph-mon[44533]: osdmap e518: 12 total, 11 up, 7 in 2024-04-29T14:29:01.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:29:01.610 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:01.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:01.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:01.805 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:01.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:01.916 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:01.977 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:02.219 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:29:02.220 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7960/52368 objects degraded (15.200%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1616,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":519,"num_osds":12,"num_up_osds":11,"osd_up_since":1714400760,"num_in_osds":7,"osd_in_since":1714400774,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovering+degraded","count":4},{"state_name":"active+recovering+degraded+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":26184,"data_bytes":1716388408,"bytes_used":6794313728,"bytes_avail":665170935808,"bytes_total":671965249536,"degraded_objects":7960,"degraded_total":52368,"degraded_ratio":0.15200122212037886,"recovering_objects_per_sec":311,"recovering_bytes_per_sec":20382822,"recovering_keys_per_sec":0,"num_objects_recovered":2484,"num_bytes_recovered":162791424,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":960},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":41,"modified":"2024-04-29T14:28:47.996826+0000","services":{}},"progress_events":{}} 2024-04-29T14:29:02.434 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:02.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:02.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:02.572 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:02.673 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:02.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:02.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:02.822 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:02.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:02.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:02.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:03.023 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:03 smithi121 ceph-mon[44533]: pgmap v1366: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.6 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 7960/52368 objects degraded (15.200%); 19 MiB/s, 311 objects/s recovering 2024-04-29T14:29:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:03 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 7960/52368 objects degraded (15.200%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:03 smithi121 ceph-mon[44533]: osdmap e519: 12 total, 11 up, 7 in 2024-04-29T14:29:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1540365257' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:29:03.315 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:03 smithi194 ceph-mon[44089]: pgmap v1366: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.6 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 7960/52368 objects degraded (15.200%); 19 MiB/s, 311 objects/s recovering 2024-04-29T14:29:03.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:03 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 7960/52368 objects degraded (15.200%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:03.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:03 smithi194 ceph-mon[44089]: osdmap e519: 12 total, 11 up, 7 in 2024-04-29T14:29:03.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1540365257' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:29:03.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:03 smithi040 ceph-mon[39581]: pgmap v1366: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.6 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 7960/52368 objects degraded (15.200%); 19 MiB/s, 311 objects/s recovering 2024-04-29T14:29:03.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:03 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 7960/52368 objects degraded (15.200%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:03.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:03 smithi040 ceph-mon[39581]: osdmap e519: 12 total, 11 up, 7 in 2024-04-29T14:29:03.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1540365257' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:29:03.499 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:03.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:03.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:03.676 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:03.704 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:03.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:03.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:03.942 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:03.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:03.963 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:29:03.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:29:04.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:04.088 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:04.110 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:29:04.110 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:29:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:04.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:04.231 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:04.289 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:04.512 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:04.591 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:29:04.614 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:29:04.637 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:29:04.664 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:29:04.699 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:29:04.723 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:29:04.751 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:29:04.775 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:29:04.802 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:29:04.833 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:29:04.853 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:29:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:29:05.152 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:05.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:05.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:05.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:05.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:05.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: pgmap v1368: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 7645/50734 objects degraded (15.069%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:29:05.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3005454538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:05.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: pgmap v1368: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 7645/50734 objects degraded (15.069%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3005454538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:05.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: pgmap v1368: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.5 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 7645/50734 objects degraded (15.069%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3005454538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:05.443 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:05.491 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:29:06.258 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:29:06.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:06.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:06.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:07.057 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:29:07.063 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:07.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:07.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:07 smithi040 ceph-mon[39581]: pgmap v1369: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.5 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 7355/47958 objects degraded (15.336%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:29:07.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:07.204 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:07.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:07.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:07.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:07 smithi194 ceph-mon[44089]: pgmap v1369: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.5 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 7355/47958 objects degraded (15.336%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:29:07.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:07.376 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:07.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:07 smithi121 ceph-mon[44533]: pgmap v1369: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.5 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 7355/47958 objects degraded (15.336%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:29:07.402 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:07.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:07.502 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:29:07.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:07.570 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:07.782 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:07.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:07.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:07.919 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:08.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:08.126 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:08.153 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:08 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 7355/47958 objects degraded (15.336%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:08.217 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:08.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:29:08.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:08 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 7355/47958 objects degraded (15.336%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:08.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:08 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 7355/47958 objects degraded (15.336%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:09.093 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:29:09.221 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:29:09.236 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:29:09.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:09 smithi194 ceph-mon[44089]: pgmap v1370: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.3 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 6478/43394 objects degraded (14.928%); 20 MiB/s, 325 objects/s recovering 2024-04-29T14:29:09.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/581449242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:09.566 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:09.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:09.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:09 smithi040 ceph-mon[39581]: pgmap v1370: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.3 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 6478/43394 objects degraded (14.928%); 20 MiB/s, 325 objects/s recovering 2024-04-29T14:29:09.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/581449242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:09.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:09 smithi121 ceph-mon[44533]: pgmap v1370: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.3 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 6478/43394 objects degraded (14.928%); 20 MiB/s, 325 objects/s recovering 2024-04-29T14:29:09.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/581449242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:09.675 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:09.717 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:10.277 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:10.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:10.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:10.417 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:10.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:10 smithi194 ceph-mon[44089]: from='client.16707 v1:172.21.15.40:0/3120260464' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:10.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:10 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:29:10.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/581449242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:10.550 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:10 smithi194 ceph-mon[44089]: osdmap e520: 12 total, 11 up, 7 in 2024-04-29T14:29:10.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:10 smithi040 ceph-mon[39581]: from='client.16707 v1:172.21.15.40:0/3120260464' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:10.587 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:10 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:29:10.587 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/581449242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:10.587 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:10 smithi040 ceph-mon[39581]: osdmap e520: 12 total, 11 up, 7 in 2024-04-29T14:29:10.621 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1370,"stamp":"2024-04-29T14:29:08.000912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1422328376,"num_objects":21697,"num_object_clones":0,"num_object_copies":43394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21697,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":245414,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35471,"num_bytes_recovered":2326218880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160787,"ondisk_log_size":160787,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6184480,"kb_used_data":2389732,"kb_used_omap":10,"kb_used_meta":3794421,"kb_avail":650031584,"statfs":{"total":671965249536,"available":665632342016,"internally_reserved":0,"allocated":2447085568,"data_stored":2442733059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3885487198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-522584064,"num_objects":-7974,"num_object_clones":0,"num_object_copies":-15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2602,"num_bytes_recovered":170524672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001741"},"pg_stats":[{"pgid":"5.f","version":"519'15716","reported_seq":30495,"reported_epoch":519,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:06.572352+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:29:06.572352+0000","last_peered":"2024-04-29T14:29:06.572352+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:29:06.572352+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:29:06.572352+0000","mapping_epoch":467,"log_start":"451'5700","ondisk_log_start":"451'5700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15716,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7252,"num_bytes_recovered":475267072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1230},{"shards":"4,11","objects":18}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"519'15460","reported_seq":15723,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:05.768821+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:29:05.768821+0000","last_peered":"2024-04-29T14:29:05.768821+0000","last_clean":"2024-04-29T14:29:05.768821+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:29:05.768821+0000","last_undegraded":"2024-04-29T14:29:05.768821+0000","last_fullsized":"2024-04-29T14:29:05.768821+0000","mapping_epoch":432,"log_start":"450'5400","ondisk_log_start":"450'5400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87425024,"num_objects":1334,"num_object_clones":0,"num_object_copies":2668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15460,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"519'15349","reported_seq":29950,"reported_epoch":519,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:29:06.289815+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:29:06.289815+0000","last_peered":"2024-04-29T14:29:06.289815+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:29:06.289815+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:29:06.289815+0000","mapping_epoch":468,"log_start":"450'5300","ondisk_log_start":"450'5300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080408,"num_objects":1345,"num_object_clones":0,"num_object_copies":2690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15349,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7167,"num_bytes_recovered":469696512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":166},{"shards":"11","objects":1179}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"519'15489","reported_seq":15765,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:06.548933+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:29:06.548933+0000","last_peered":"2024-04-29T14:29:06.548933+0000","last_clean":"2024-04-29T14:29:06.548933+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:29:06.548933+0000","last_undegraded":"2024-04-29T14:29:06.548933+0000","last_fullsized":"2024-04-29T14:29:06.548933+0000","mapping_epoch":432,"log_start":"450'5400","ondisk_log_start":"450'5400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85393408,"num_objects":1303,"num_object_clones":0,"num_object_copies":2606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15489,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"519'15342","reported_seq":15612,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:05.814646+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:29:05.814646+0000","last_peered":"2024-04-29T14:29:05.814646+0000","last_clean":"2024-04-29T14:29:05.814646+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:29:05.814646+0000","last_undegraded":"2024-04-29T14:29:05.814646+0000","last_fullsized":"2024-04-29T14:29:05.814646+0000","mapping_epoch":432,"log_start":"450'5300","ondisk_log_start":"450'5300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86769664,"num_objects":1324,"num_object_clones":0,"num_object_copies":2648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15342,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"519'17311","reported_seq":17710,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:03.834946+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:29:03.834946+0000","last_peered":"2024-04-29T14:29:03.834946+0000","last_clean":"2024-04-29T14:29:03.834946+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:29:03.834946+0000","last_undegraded":"2024-04-29T14:29:03.834946+0000","last_fullsized":"2024-04-29T14:29:03.834946+0000","mapping_epoch":432,"log_start":"454'7300","ondisk_log_start":"454'7300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97648640,"num_objects":1490,"num_object_clones":0,"num_object_copies":2980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15056,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"519'15273","reported_seq":15545,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:06.655104+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:29:06.655104+0000","last_peered":"2024-04-29T14:29:06.655104+0000","last_clean":"2024-04-29T14:29:06.655104+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:29:06.655104+0000","last_undegraded":"2024-04-29T14:29:06.655104+0000","last_fullsized":"2024-04-29T14:29:06.655104+0000","mapping_epoch":432,"log_start":"450'5200","ondisk_log_start":"450'5200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85786624,"num_objects":1309,"num_object_clones":0,"num_object_copies":2618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"519'15327","reported_seq":15594,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:06.284621+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:29:06.284621+0000","last_peered":"2024-04-29T14:29:06.284621+0000","last_clean":"2024-04-29T14:29:06.284621+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:29:06.284621+0000","last_undegraded":"2024-04-29T14:29:06.284621+0000","last_fullsized":"2024-04-29T14:29:06.284621+0000","mapping_epoch":432,"log_start":"450'5300","ondisk_log_start":"450'5300","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83951616,"num_objects":1281,"num_object_clones":0,"num_object_copies":2562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15327,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"519'15432","reported_seq":29796,"reported_epoch":519,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:05.862791+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:29:05.862791+0000","last_peered":"2024-04-29T14:29:05.862791+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:29:05.862791+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:29:05.862791+0000","mapping_epoch":467,"log_start":"451'5400","ondisk_log_start":"451'5400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15432,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7048,"num_bytes_recovered":461897728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":1351},{"shards":"9,10","objects":15}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"519'15402","reported_seq":29780,"reported_epoch":519,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:06.649115+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:29:06.649115+0000","last_peered":"2024-04-29T14:29:06.649115+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:29:06.649115+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:29:06.649115+0000","mapping_epoch":467,"log_start":"451'5400","ondisk_log_start":"451'5400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15402,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":1313},{"shards":"7,10","objects":51}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"519'15423","reported_seq":15695,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:06.582649+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:29:06.582649+0000","last_peered":"2024-04-29T14:29:06.582649+0000","last_clean":"2024-04-29T14:29:06.582649+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:29:06.582649+0000","last_undegraded":"2024-04-29T14:29:06.582649+0000","last_fullsized":"2024-04-29T14:29:06.582649+0000","mapping_epoch":432,"log_start":"451'5400","ondisk_log_start":"451'5400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87621632,"num_objects":1337,"num_object_clones":0,"num_object_copies":2674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15423,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"519'15261","reported_seq":15658,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:05.353870+0000","last_change":"2024-04-29T14:28:59.693015+0000","last_active":"2024-04-29T14:29:05.353870+0000","last_peered":"2024-04-29T14:29:05.353870+0000","last_clean":"2024-04-29T14:29:05.353870+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:29:05.353870+0000","last_undegraded":"2024-04-29T14:29:05.353870+0000","last_fullsized":"2024-04-29T14:29:05.353870+0000","mapping_epoch":432,"log_start":"450'5200","ondisk_log_start":"450'5200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"517'14944","last_scrub_stamp":"2024-04-29T14:28:59.692847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:28:59.692847+0000","objects_scrubbed":1678,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:18.941948+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88408064,"num_objects":1349,"num_object_clones":0,"num_object_copies":2698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15261,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"519'15176","reported_seq":15509,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:05.318921+0000","last_change":"2024-04-29T14:27:46.065765+0000","last_active":"2024-04-29T14:29:05.318921+0000","last_peered":"2024-04-29T14:29:05.318921+0000","last_clean":"2024-04-29T14:29:05.318921+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:29:05.318921+0000","last_undegraded":"2024-04-29T14:29:05.318921+0000","last_fullsized":"2024-04-29T14:29:05.318921+0000","mapping_epoch":432,"log_start":"450'5100","ondisk_log_start":"450'5100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":5050,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:14.401450+0000","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92667904,"num_objects":1414,"num_object_clones":0,"num_object_copies":2828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15176,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"519'15124","reported_seq":15399,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:06.585055+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:29:06.585055+0000","last_peered":"2024-04-29T14:29:06.585055+0000","last_clean":"2024-04-29T14:29:06.585055+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:29:06.585055+0000","last_undegraded":"2024-04-29T14:29:06.585055+0000","last_fullsized":"2024-04-29T14:29:06.585055+0000","mapping_epoch":432,"log_start":"450'5100","ondisk_log_start":"450'5100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86638592,"num_objects":1322,"num_object_clones":0,"num_object_copies":2644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15124,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"519'18137","reported_seq":18584,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:03.852404+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:29:03.852404+0000","last_peered":"2024-04-29T14:29:03.852404+0000","last_clean":"2024-04-29T14:29:03.852404+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:29:03.852404+0000","last_undegraded":"2024-04-29T14:29:03.852404+0000","last_fullsized":"2024-04-29T14:29:03.852404+0000","mapping_epoch":432,"log_start":"456'8100","ondisk_log_start":"456'8100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94437376,"num_objects":1441,"num_object_clones":0,"num_object_copies":2882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15175,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"519'15228","reported_seq":29305,"reported_epoch":519,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:03.862798+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:29:03.862798+0000","last_peered":"2024-04-29T14:29:03.862798+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:29:03.862798+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:29:03.862798+0000","mapping_epoch":467,"log_start":"450'5200","ondisk_log_start":"450'5200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":2936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15228,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6943,"num_bytes_recovered":455016448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":63},{"shards":"10","objects":1405}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1281,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-29T14:29:02.225420+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:29:02.225420+0000","last_peered":"2024-04-29T14:29:02.225420+0000","last_clean":"2024-04-29T14:29:02.225420+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:29:02.225420+0000","last_undegraded":"2024-04-29T14:29:02.225420+0000","last_fullsized":"2024-04-29T14:29:02.225420+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1421738008,"num_objects":21695,"num_object_clones":0,"num_object_copies":43390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21695,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245233,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35463,"num_bytes_recovered":2324103168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5151137792,"data_stored":5151129648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160650,"ondisk_log_size":160650,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725992,"kb_used_data":171360,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93019160,"statfs":{"total":95995035648,"available":95251619840,"internally_reserved":0,"allocated":175472640,"data_stored":174845393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":567867844},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383950,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739936,"kb_used_data":289428,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93005216,"statfs":{"total":95995035648,"available":95237341184,"internally_reserved":0,"allocated":296374272,"data_stored":295754649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":461240768},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645588,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":698844,"kb_used_data":266388,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93046308,"statfs":{"total":95995035648,"available":95279419392,"internally_reserved":0,"allocated":272781312,"data_stored":272161709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422892,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234269,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":835880,"kb_used_data":277152,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92909272,"statfs":{"total":95995035648,"available":95139094528,"internally_reserved":0,"allocated":283803648,"data_stored":283176421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":572127690},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528610,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843676,"kb_used_data":266900,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92901476,"statfs":{"total":95995035648,"available":95131111424,"internally_reserved":0,"allocated":273305600,"data_stored":272685997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575163,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145052,"kb_used_data":608532,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92600100,"statfs":{"total":95995035648,"available":94822502400,"internally_reserved":0,"allocated":623136768,"data_stored":622517165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497121,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986478,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1195100,"kb_used_data":509972,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92550052,"statfs":{"total":95995035648,"available":94771253248,"internally_reserved":0,"allocated":522211328,"data_stored":521591725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":614793216,"data_stored":614793216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":512557056,"data_stored":512557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":271122432,"data_stored":271122432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":272306176,"data_stored":272302104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":263716864,"data_stored":263716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":294125568,"data_stored":294125568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":172822528,"data_stored":172818456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:29:10.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:10 smithi121 ceph-mon[44533]: from='client.16707 v1:172.21.15.40:0/3120260464' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:10.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:10 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:29:10.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/581449242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:10.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:10 smithi121 ceph-mon[44533]: osdmap e520: 12 total, 11 up, 7 in 2024-04-29T14:29:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:10.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:10.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:11.020 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:11.047 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:29:11.047 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:29:11.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:29:11.047 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:29:11.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.047 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58846 5.8 5:17594c5c:::benchmark_data_smithi194_121512_object58845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.047 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.576782+0000", 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.47196324800001, 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20781687600000001, 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58846 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576782+0000", 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576782+0000", 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576783+0000", 2024-04-29T14:29:11.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2219999999999999e-06 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576804+0000", 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0959e-05 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576813+0000", 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4419999999999998e-06 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576820+0000", 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6669999999999999e-06 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.577008+0000", 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000188505 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.577129+0000", 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012085 2024-04-29T14:29:11.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.577260+0000", 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000131197 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.578988+0000", 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017283579999999999 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784512+0000", 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20552357499999999 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784549+0000", 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6488e-05 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784599+0000", 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0612999999999998e-05 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.053 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58917 5.8 5:171315a4:::benchmark_data_smithi194_121512_object58916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.613671+0000", 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.435074934, 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17105858700000001, 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.054 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58917 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613671+0000", 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613671+0000", 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613673+0000", 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2989999999999999e-06 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613697+0000", 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4136999999999999e-05 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613710+0000", 2024-04-29T14:29:11.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3457e-05 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613728+0000", 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7331000000000001e-05 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613793+0000", 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5303999999999997e-05 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613942+0000", 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014877400000000001 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.614094+0000", 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000151958 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.616078+0000", 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019839499999999999 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784637+0000", 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16855897 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784675+0000", 2024-04-29T14:29:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8547999999999998e-05 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784729+0000", 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859000000000001e-05 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58929 5.8 5:1ed179d5:::benchmark_data_smithi194_121512_object58928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.620507+0000", 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.428238435, 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16437480500000001, 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58929 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.061 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620507+0000", 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620507+0000", 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620510+0000", 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3119999999999998e-06 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620533+0000", 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2243999999999999e-05 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620546+0000", 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3226e-05 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620556+0000", 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0149999999999999e-05 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620621+0000", 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4888999999999995e-05 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620769+0000", 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014842999999999999 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620887+0000", 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011804200000000001 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.622653+0000", 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017655080000000001 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784779+0000", 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16212600799999999 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784828+0000", 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8715999999999999e-05 2024-04-29T14:29:11.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784882+0000", 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4280000000000002e-05 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58948 5.8 5:15d469f1:::benchmark_data_smithi194_121512_object58947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.628765+0000", 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.41998096, 2024-04-29T14:29:11.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15817314399999999, 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58948 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628765+0000", 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628765+0000", 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628767+0000", 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6989999999999999e-06 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628790+0000", 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2337999999999999e-05 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628804+0000", 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4782e-05 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628812+0000", 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9459999999999998e-06 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628878+0000", 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5933999999999996e-05 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628998+0000", 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012002699999999999 2024-04-29T14:29:11.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.629092+0000", 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3881000000000005e-05 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.630983+0000", 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001890468 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786808+0000", 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15582528800000001 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786869+0000", 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1112999999999996e-05 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786938+0000", 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8668000000000001e-05 2024-04-29T14:29:11.073 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58957 5.8 5:19cc827b:::benchmark_data_smithi194_121512_object58956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.636223+0000", 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.41252264299999, 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15196238300000001, 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.074 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58957 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636223+0000", 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636223+0000", 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636226+0000", 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.672e-06 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636251+0000", 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5738e-05 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636269+0000", 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7452000000000002e-05 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636279+0000", 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.893e-06 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636351+0000", 2024-04-29T14:29:11.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1933000000000004e-05 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636484+0000", 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013344799999999999 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636640+0000", 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015584099999999999 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.638479+0000", 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018393649999999999 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788068+0000", 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14958922399999999 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788123+0000", 2024-04-29T14:29:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4959e-05 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788185+0000", 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1858000000000003e-05 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58972 5.8 5:10477e2b:::benchmark_data_smithi194_121512_object58971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.640882+0000", 2024-04-29T14:29:11.080 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.40786395399999, 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14746427200000001, 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58972 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640882+0000", 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640882+0000", 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640884+0000", 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.83e-06 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640937+0000", 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2292e-05 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640956+0000", 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8941e-05 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640966+0000", 2024-04-29T14:29:11.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0791000000000001e-05 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.641038+0000", 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1482000000000004e-05 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.641183+0000", 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014528100000000001 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.641316+0000", 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013237399999999999 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.643457+0000", 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002141839 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788227+0000", 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.144769433 2024-04-29T14:29:11.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788279+0000", 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1786999999999997e-05 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788346+0000", 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7222000000000001e-05 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58992 5.8 5:1acc57d2:::benchmark_data_smithi194_121512_object58991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.654227+0000", 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.39451806599999, 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13512590899999999, 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.087 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58992 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654227+0000", 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654227+0000", 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654230+0000", 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.576e-06 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654283+0000", 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3436999999999997e-05 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654301+0000", 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7632e-05 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654310+0000", 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3540000000000006e-06 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654386+0000", 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5333999999999994e-05 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654532+0000", 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000146582 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654637+0000", 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000104934 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.656491+0000", 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001853593 2024-04-29T14:29:11.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789250+0000", 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.132758986 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789301+0000", 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0924000000000002e-05 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789353+0000", 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2556999999999998e-05 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58999 5.8 5:156f7ae1:::benchmark_data_smithi194_121512_object58998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.657159+0000", 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.39158624699999, 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13463576999999999, 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.093 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58999 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657159+0000", 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657159+0000", 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657162+0000", 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5550000000000001e-06 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657206+0000", 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4434999999999997e-05 2024-04-29T14:29:11.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657218+0000", 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1651000000000001e-05 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657224+0000", 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.981e-06 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657287+0000", 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3028999999999996e-05 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657418+0000", 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000130925 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657529+0000", 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011137999999999999 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.659257+0000", 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001727288 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791653+0000", 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13239661699999999 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791685+0000", 2024-04-29T14:29:11.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1702000000000002e-05 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791795+0000", 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110207 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58948 5.8 5:15d469f1:::benchmark_data_smithi194_121512_object58947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:29:11.099 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.37368182700001, 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.111819161, 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58948 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.08e-07 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675093+0000", 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9105999999999999e-05 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675097+0000", 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.625e-06 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675100+0000", 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0259999999999999e-06 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675347+0000", 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00024665700000000001 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675410+0000", 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3100000000000002e-05 2024-04-29T14:29:11.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786883+0000", 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.111473239 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58957 5.8 5:19cc827b:::benchmark_data_smithi194_121512_object58956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.37362274899999, 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.1130111, 2024-04-29T14:29:11.104 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58957 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675160+0000", 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6313000000000003e-05 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675166+0000", 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7560000000000002e-06 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675171+0000", 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.989e-06 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675443+0000", 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00027163600000000001 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675478+0000", 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4638000000000003e-05 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788134+0000", 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.112656136 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58972 5.8 5:10477e2b:::benchmark_data_smithi194_121512_object58971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675178+0000", 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.37356723299999, 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11311273099999999, 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.109 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58972 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675178+0000", 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675178+0000", 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675179+0000", 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3500000000000002e-07 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675212+0000", 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3343000000000002e-05 2024-04-29T14:29:11.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675218+0000", 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4609999999999999e-06 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675223+0000", 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.502e-06 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675514+0000", 2024-04-29T14:29:11.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00029081499999999999 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675578+0000", 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4257000000000005e-05 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788291+0000", 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.112712918 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.113 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58992 5.8 5:1acc57d2:::benchmark_data_smithi194_121512_object58991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675230+0000", 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.37351551099999, 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.114080165, 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58992 2024-04-29T14:29:11.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675230+0000", 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675230+0000", 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675231+0000", 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2699999999999999e-07 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675273+0000", 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2676999999999998e-05 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675279+0000", 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8359999999999997e-06 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675285+0000", 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1410000000000002e-06 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675622+0000", 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00033625700000000002 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675664+0000", 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2814e-05 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789310+0000", 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.113645813 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58999 5.8 5:156f7ae1:::benchmark_data_smithi194_121512_object58998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675293+0000", 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.37345256200001, 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.116400928, 2024-04-29T14:29:11.118 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58999 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675293+0000", 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675293+0000", 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675294+0000", 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675333+0000", 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9604999999999998e-05 2024-04-29T14:29:11.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675338+0000", 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4780000000000002e-06 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675342+0000", 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0260000000000001e-06 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675701+0000", 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00035939899999999999 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675770+0000", 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9208999999999994e-05 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791694+0000", 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.115923555 2024-04-29T14:29:11.122 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59022 5.8 5:1f3c6cca:::benchmark_data_smithi194_121512_object59021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675348+0000", 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.37339724700001, 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.118538427, 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59022 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675348+0000", 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675348+0000", 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675349+0000", 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.32e-07 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675379+0000", 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9879e-05 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675382+0000", 2024-04-29T14:29:11.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8020000000000002e-06 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675385+0000", 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0970000000000002e-06 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675797+0000", 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00041133100000000002 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675906+0000", 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108934 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.676004+0000", 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8628000000000005e-05 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.677972+0000", 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019677330000000002 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793782+0000", 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11581021499999999 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793844+0000", 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1427000000000007e-05 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793887+0000", 2024-04-29T14:29:11.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2948999999999998e-05 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59033 5.8 5:1536a9bf:::benchmark_data_smithi194_121512_object59032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.679421+0000", 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.36932455300001, 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11468714200000001, 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.129 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59033 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679421+0000", 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679421+0000", 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679423+0000", 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9599999999999999e-06 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679463+0000", 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9712000000000002e-05 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679474+0000", 2024-04-29T14:29:11.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1756999999999999e-05 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679481+0000", 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3629999999999999e-06 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679529+0000", 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8418999999999998e-05 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679644+0000", 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011482 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679793+0000", 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014947100000000001 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.683670+0000", 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003876857 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793955+0000", 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.110284493 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794056+0000", 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010158 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794108+0000", 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1709999999999998e-05 2024-04-29T14:29:11.134 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59036 5.8 5:18c5f67b:::benchmark_data_smithi194_121512_object59035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.681129+0000", 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.36761625299999, 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11312299000000001, 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59036 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681129+0000", 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681129+0000", 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681131+0000", 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.497e-06 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681174+0000", 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3650999999999999e-05 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681188+0000", 2024-04-29T14:29:11.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3233e-05 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681195+0000", 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2810000000000003e-06 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681451+0000", 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000255949 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681578+0000", 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012705 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681714+0000", 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013629199999999999 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.685889+0000", 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041743509999999998 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794162+0000", 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.108273149 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794217+0000", 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5469e-05 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794252+0000", 2024-04-29T14:29:11.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5067999999999997e-05 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59040 5.8 5:1c3d728f:::benchmark_data_smithi194_121512_object59039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.682661+0000", 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 249.36608463300001, 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.111861925, 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59040 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682661+0000", 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682661+0000", 2024-04-29T14:29:11.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682663+0000", 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.795e-06 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682713+0000", 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0575000000000003e-05 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682736+0000", 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2339000000000001e-05 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682744+0000", 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5820000000000002e-06 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.683702+0000", 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00095744000000000005 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.683851+0000", 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014912899999999999 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.684007+0000", 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015646799999999999 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.685983+0000", 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019753290000000001 2024-04-29T14:29:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794426+0000", 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.108443609 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794478+0000", 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.223e-05 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794523+0000", 2024-04-29T14:29:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4428999999999998e-05 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:194006 5.d 5:b9c4042c:::benchmark_data_smithi194_121512_object60540:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e495)", 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:27:58.279362+0000", 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 72.769383243999997, 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.115618417, 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 194006 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279362+0000", 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279362+0000", 2024-04-29T14:29:11.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279364+0000", 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8449999999999999e-06 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279369+0000", 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0209999999999999e-06 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279378+0000", 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9740000000000008e-06 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279384+0000", 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1890000000000004e-06 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.352886+0000", 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.073501739999999996 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.352978+0000", 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1903000000000004e-05 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.353117+0000", 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013911699999999999 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.373664+0000", 2024-04-29T14:29:11.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020547267000000001 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.393340+0000", 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019675482000000001 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.393376+0000", 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6297000000000002e-05 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.394981+0000", 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001604582 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26047.0:235037 5.f e515/467 5:f2be22cd:::benchmark_data_smithi194_121512_object101571:head v 515'15013, mlcod=453'6538)", 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:53.999083+0000", 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.049662975, 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11755185, 2024-04-29T14:29:11.153 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999083+0000", 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999083+0000", 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999085+0000", 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1789999999999998e-06 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999093+0000", 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3690000000000007e-06 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999101+0000", 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9500000000000001e-06 2024-04-29T14:29:11.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999107+0000", 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8159999999999999e-06 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999155+0000", 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8597999999999998e-05 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999179+0000", 2024-04-29T14:29:11.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3328000000000001e-05 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.116589+0000", 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11740991100000001 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.116634+0000", 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5698999999999997e-05 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.157 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:235049 5.d 5:b25b9240:::benchmark_data_smithi194_121512_object101583:head [delete] snapc 0=[] ondisk+write+known_if_redirected e515)", 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:54.013173+0000", 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.035572124000002, 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.10991333, 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:29:11.158 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 235049 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013173+0000", 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013173+0000", 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:29:11.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013176+0000", 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9189999999999999e-06 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013195+0000", 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9032e-05 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.103738+0000", 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.090542330000000004 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.103750+0000", 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2676e-05 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.107286+0000", 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035357230000000002 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.107445+0000", 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000158715 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.107594+0000", 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014871099999999999 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.121692+0000", 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014098589999999999 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.121760+0000", 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7533999999999994e-05 2024-04-29T14:29:11.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.121816+0000", 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6462999999999998e-05 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.123087+0000", 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001270637 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:29:11.163 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:29:11.164 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:29:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:11.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:11.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:11.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:11.343 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:11 smithi194 ceph-mon[44089]: pgmap v1372: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.3 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 6113/41682 objects degraded (14.666%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:29:11.352 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:11.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:11 smithi121 ceph-mon[44533]: pgmap v1372: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.3 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 6113/41682 objects degraded (14.666%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:29:11.378 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:11.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:11.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:11.521 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:11.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:11.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:11.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:11 smithi040 ceph-mon[39581]: pgmap v1372: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.3 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 6113/41682 objects degraded (14.666%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:29:11.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:11.691 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:11.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:11.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:11.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:29:11.985 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:12.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:12.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:12.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:12.162 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:12.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:12.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:12.273 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:12 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 6113/41682 objects degraded (14.666%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:12.273 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:12 smithi194 ceph-mon[44089]: osdmap e521: 12 total, 11 up, 7 in 2024-04-29T14:29:12.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:12 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 6113/41682 objects degraded (14.666%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:12.275 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:12 smithi121 ceph-mon[44533]: osdmap e521: 12 total, 11 up, 7 in 2024-04-29T14:29:12.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:12.332 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:12.537 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:12.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:12 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 6113/41682 objects degraded (14.666%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:12.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:12 smithi040 ceph-mon[39581]: osdmap e521: 12 total, 11 up, 7 in 2024-04-29T14:29:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:12.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:12.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:12.673 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:12.704 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:12.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:12.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:29:12.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:12.846 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:12.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:12.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:12.980 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:13.045 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:13.081 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:13.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:13.180 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:13.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:13.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:13.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:13.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:13 smithi194 ceph-mon[44089]: pgmap v1373: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.0 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 4920/34100 objects degraded (14.428%); 19 MiB/s, 310 objects/s recovering 2024-04-29T14:29:13.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:13 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:29:13.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:13 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:13.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:13.404 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:13.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:13.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:13 smithi040 ceph-mon[39581]: pgmap v1373: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.0 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 4920/34100 objects degraded (14.428%); 19 MiB/s, 310 objects/s recovering 2024-04-29T14:29:13.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:13 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:29:13.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:13 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:13.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:13.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:13.584 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:13.623 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:29:13.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:13 smithi121 ceph-mon[44533]: pgmap v1373: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1.0 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 4920/34100 objects degraded (14.428%); 19 MiB/s, 310 objects/s recovering 2024-04-29T14:29:13.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:13 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:29:13.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:13 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:13.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:13.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:13.881 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:29:14.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:29:14.466 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:29:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:29:14.493 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:29:14.493 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:29:14.493 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:29:14.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:14 smithi194 ceph-mon[44089]: osdmap e522: 12 total, 11 up, 7 in 2024-04-29T14:29:14.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/378479337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:14.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:14 smithi040 ceph-mon[39581]: osdmap e522: 12 total, 11 up, 7 in 2024-04-29T14:29:14.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/378479337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:14.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:14 smithi121 ceph-mon[44533]: osdmap e522: 12 total, 11 up, 7 in 2024-04-29T14:29:14.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/378479337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: pgmap v1376: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1011 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 4516/32332 objects degraded (13.968%); 20 MiB/s, 326 objects/s recovering 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:15.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:15.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: pgmap v1376: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1011 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 4516/32332 objects degraded (13.968%); 20 MiB/s, 326 objects/s recovering 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: pgmap v1376: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 1011 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 4516/32332 objects degraded (13.968%); 20 MiB/s, 326 objects/s recovering 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:15.586 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:15.614 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:15.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:15.720 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:15.766 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:15.802 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:15.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:15.909 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:15.984 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:15.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:16.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:16.180 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:16.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:16.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:16.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:16.357 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:16.364 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:16 smithi040 ceph-mon[39581]: 5.2 deep-scrub starts 2024-04-29T14:29:16.394 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:16.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:16.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:16.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:16 smithi194 ceph-mon[44089]: 5.2 deep-scrub starts 2024-04-29T14:29:16.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:16 smithi121 ceph-mon[44533]: 5.2 deep-scrub starts 2024-04-29T14:29:16.677 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:16.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:16.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:16.833 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:16.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:16.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:16.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:29:17.024 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:29:17.213 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:17.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:17.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:17.375 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:17.430 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:17.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:17.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:17.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:17 smithi194 ceph-mon[44089]: pgmap v1377: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 11 active+clean; 922 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 901 op/s; 4128/29484 objects degraded (14.001%); 18 MiB/s, 293 objects/s recovering 2024-04-29T14:29:17.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:17 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 4128/29484 objects degraded (14.001%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:17.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:17 smithi194 ceph-mon[44089]: osdmap e523: 12 total, 11 up, 7 in 2024-04-29T14:29:17.569 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:17 smithi121 ceph-mon[44533]: pgmap v1377: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 11 active+clean; 922 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 901 op/s; 4128/29484 objects degraded (14.001%); 18 MiB/s, 293 objects/s recovering 2024-04-29T14:29:17.569 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:17 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 4128/29484 objects degraded (14.001%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:17.569 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:17 smithi121 ceph-mon[44533]: osdmap e523: 12 total, 11 up, 7 in 2024-04-29T14:29:17.572 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:17.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:17.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:17.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:17 smithi040 ceph-mon[39581]: pgmap v1377: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 11 active+clean; 922 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 901 op/s; 4128/29484 objects degraded (14.001%); 18 MiB/s, 293 objects/s recovering 2024-04-29T14:29:17.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:17 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 4128/29484 objects degraded (14.001%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:17.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:17 smithi040 ceph-mon[39581]: osdmap e523: 12 total, 11 up, 7 in 2024-04-29T14:29:17.711 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:17.750 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:17.779 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:17.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:17.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:17.936 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:17.958 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:29:17.959 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:29:17.959 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:17.970 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:17.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:18.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:18.114 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:18.241 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:18.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:18.324 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:29:18.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:18.390 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:18.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:18.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:18.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:18 smithi040 ceph-mon[39581]: pgmap v1379: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+degraded, 11 active+clean; 778 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 3034/24878 objects degraded (12.196%); 20 MiB/s, 314 objects/s recovering 2024-04-29T14:29:18.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2097314149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:29:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:18.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:18 smithi194 ceph-mon[44089]: pgmap v1379: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+degraded, 11 active+clean; 778 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 3034/24878 objects degraded (12.196%); 20 MiB/s, 314 objects/s recovering 2024-04-29T14:29:18.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2097314149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:29:18.614 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:18.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:18.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:18 smithi121 ceph-mon[44533]: pgmap v1379: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+degraded, 11 active+clean; 778 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 3034/24878 objects degraded (12.196%); 20 MiB/s, 314 objects/s recovering 2024-04-29T14:29:18.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2097314149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:29:18.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:18.749 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:19.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:29:19.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:19 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:19.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2097314149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:29:19.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:19 smithi040 ceph-mon[39581]: osdmap e524: 12 total, 11 up, 7 in 2024-04-29T14:29:19.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:19 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:19.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2097314149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:29:19.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:19 smithi194 ceph-mon[44089]: osdmap e524: 12 total, 11 up, 7 in 2024-04-29T14:29:19.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:19 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:19.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2097314149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:29:19.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:19 smithi121 ceph-mon[44533]: osdmap e524: 12 total, 11 up, 7 in 2024-04-29T14:29:19.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:29:20.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:20 smithi040 ceph-mon[39581]: pgmap v1381: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+degraded, 11 active+clean; 726 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 2491/23208 objects degraded (10.733%); 23 MiB/s, 361 objects/s recovering 2024-04-29T14:29:20.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:20 smithi121 ceph-mon[44533]: pgmap v1381: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+degraded, 11 active+clean; 726 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 2491/23208 objects degraded (10.733%); 23 MiB/s, 361 objects/s recovering 2024-04-29T14:29:20.707 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:20.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:20.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:20.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:20 smithi194 ceph-mon[44089]: pgmap v1381: 17 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+degraded, 11 active+clean; 726 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 2491/23208 objects degraded (10.733%); 23 MiB/s, 361 objects/s recovering 2024-04-29T14:29:20.844 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:21.046 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:21.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:21.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:21.198 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:21.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:21.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:21.250 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:21.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:21.549 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:21.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:21.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:21 smithi040 ceph-mon[39581]: 5.2 deep-scrub ok 2024-04-29T14:29:21.596 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:29:21.596 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:29:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:21 smithi121 ceph-mon[44533]: 5.2 deep-scrub ok 2024-04-29T14:29:21.655 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:21.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:21 smithi194 ceph-mon[44089]: 5.2 deep-scrub ok 2024-04-29T14:29:21.710 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:21.741 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:21.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:21.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:21.847 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:21.891 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:21.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:22.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:22.068 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:22.115 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:22.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:22.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:29:22.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:22.266 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:22.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:22 smithi040 ceph-mon[39581]: from='client.16725 v1:172.21.15.40:0/3433220833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:22.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:22 smithi040 ceph-mon[39581]: pgmap v1382: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 497 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 731/15890 objects degraded (4.600%); 30 MiB/s, 473 objects/s recovering 2024-04-29T14:29:22.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:22 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 2491/23208 objects degraded (10.733%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:22.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:22 smithi040 ceph-mon[39581]: osdmap e525: 12 total, 11 up, 7 in 2024-04-29T14:29:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:22.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:22.575 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:22.608 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:22 smithi194 ceph-mon[44089]: from='client.16725 v1:172.21.15.40:0/3433220833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:22.608 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:22 smithi194 ceph-mon[44089]: pgmap v1382: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 497 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 731/15890 objects degraded (4.600%); 30 MiB/s, 473 objects/s recovering 2024-04-29T14:29:22.608 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:22 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 2491/23208 objects degraded (10.733%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:22.608 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:22 smithi194 ceph-mon[44089]: osdmap e525: 12 total, 11 up, 7 in 2024-04-29T14:29:22.610 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:22.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:22 smithi121 ceph-mon[44533]: from='client.16725 v1:172.21.15.40:0/3433220833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:22.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:22 smithi121 ceph-mon[44533]: pgmap v1382: 17 pgs: 1 active+recovering+degraded+remapped, 4 active+recovering+degraded, 12 active+clean; 497 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 731/15890 objects degraded (4.600%); 30 MiB/s, 473 objects/s recovering 2024-04-29T14:29:22.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:22 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 2491/23208 objects degraded (10.733%), 5 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:22.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:22 smithi121 ceph-mon[44533]: osdmap e525: 12 total, 11 up, 7 in 2024-04-29T14:29:22.633 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:22.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:22.737 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:22.766 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:22.788 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:22.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:22.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:22.921 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:23.044 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:23.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:23.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:23.174 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:23.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:23.298 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:23.330 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:23.360 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:23.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:23.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:23.513 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:23.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:23.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:23.707 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:23.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:23.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:23.853 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:23.893 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:23.942 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:23.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:24.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:24.086 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:24.114 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:24.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:24.218 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:24.239 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:24 smithi194 ceph-mon[44089]: osdmap e526: 12 total, 11 up, 7 in 2024-04-29T14:29:24.254 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:24.285 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:24.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:24.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:24 smithi040 ceph-mon[39581]: osdmap e526: 12 total, 11 up, 7 in 2024-04-29T14:29:24.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:24.423 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:24.452 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:24.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:24.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:24.587 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1381,"stamp":"2024-04-29T14:29:20.003181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":760873528,"num_objects":11604,"num_object_clones":0,"num_object_copies":23208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11604,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":255507,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39458,"num_bytes_recovered":2587510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161216,"ondisk_log_size":161216,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5279612,"kb_used_data":1396216,"kb_used_omap":10,"kb_used_meta":3883253,"kb_avail":650936452,"statfs":{"total":671965249536,"available":666558926848,"internally_reserved":0,"allocated":1429725184,"data_stored":1425228470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3976451166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-356909056,"num_objects":-5446,"num_object_clones":0,"num_object_copies":-10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2429,"num_bytes_recovered":159186944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.726699"},"pg_stats":[{"pgid":"5.f","version":"522'16282","reported_seq":32335,"reported_epoch":522,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:16.593525+0000","last_change":"2024-04-29T14:26:16.020854+0000","last_active":"2024-04-29T14:29:16.593525+0000","last_peered":"2024-04-29T14:29:16.593525+0000","last_clean":"2024-04-29T14:25:54.182894+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:29:16.593525+0000","last_undegraded":"2024-04-29T14:26:15.821893+0000","last_fullsized":"2024-04-29T14:29:16.593525+0000","mapping_epoch":467,"log_start":"453'6200","ondisk_log_start":"453'6200","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44695552,"num_objects":682,"num_object_clones":0,"num_object_copies":1364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16282,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7886,"num_bytes_recovered":516816896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":596},{"shards":"4,11","objects":86}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"522'16041","reported_seq":16310,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:15.872329+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:29:15.872329+0000","last_peered":"2024-04-29T14:29:15.872329+0000","last_clean":"2024-04-29T14:29:15.872329+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:29:15.872329+0000","last_undegraded":"2024-04-29T14:29:15.872329+0000","last_fullsized":"2024-04-29T14:29:15.872329+0000","mapping_epoch":432,"log_start":"453'6000","ondisk_log_start":"453'6000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49348608,"num_objects":753,"num_object_clones":0,"num_object_copies":1506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16041,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"522'15935","reported_seq":31966,"reported_epoch":522,"state":"active+recovering+degraded+remapped","last_fresh":"2024-04-29T14:29:16.306022+0000","last_change":"2024-04-29T14:26:17.048329+0000","last_active":"2024-04-29T14:29:16.306022+0000","last_peered":"2024-04-29T14:29:16.306022+0000","last_clean":"2024-04-29T14:25:54.192906+0000","last_became_active":"2024-04-29T14:26:17.022020+0000","last_became_peered":"2024-04-29T14:26:17.022020+0000","last_unstale":"2024-04-29T14:29:16.306022+0000","last_undegraded":"2024-04-29T14:26:16.808487+0000","last_fullsized":"2024-04-29T14:29:16.306022+0000","mapping_epoch":468,"log_start":"453'5900","ondisk_log_start":"453'5900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676312,"num_objects":759,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15935,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7879,"num_bytes_recovered":516358144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[11,7],"avail_no_missing":["11"],"object_location_counts":[{"shards":"7,11","objects":292},{"shards":"11","objects":467}],"blocked_by":[],"up_primary":7,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"522'16089","reported_seq":16371,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:16.591315+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:29:16.591315+0000","last_peered":"2024-04-29T14:29:16.591315+0000","last_clean":"2024-04-29T14:29:16.591315+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:29:16.591315+0000","last_undegraded":"2024-04-29T14:29:16.591315+0000","last_fullsized":"2024-04-29T14:29:16.591315+0000","mapping_epoch":432,"log_start":"453'6000","ondisk_log_start":"453'6000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":703,"num_object_clones":0,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16089,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"522'15908","reported_seq":16181,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:15.845016+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:29:15.845016+0000","last_peered":"2024-04-29T14:29:15.845016+0000","last_clean":"2024-04-29T14:29:15.845016+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:29:15.845016+0000","last_undegraded":"2024-04-29T14:29:15.845016+0000","last_fullsized":"2024-04-29T14:29:15.845016+0000","mapping_epoch":432,"log_start":"453'5900","ondisk_log_start":"453'5900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":1516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15908,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"524'18198","reported_seq":18608,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-04-29T14:29:18.880877+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:29:18.880877+0000","last_peered":"2024-04-29T14:29:18.880877+0000","last_clean":"2024-04-29T14:29:18.880877+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:29:18.880877+0000","last_undegraded":"2024-04-29T14:29:18.880877+0000","last_fullsized":"2024-04-29T14:29:18.880877+0000","mapping_epoch":432,"log_start":"456'8100","ondisk_log_start":"456'8100","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15943,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"522'15842","reported_seq":16120,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:16.679984+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:29:16.679984+0000","last_peered":"2024-04-29T14:29:16.679984+0000","last_clean":"2024-04-29T14:29:16.679984+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:29:16.679984+0000","last_undegraded":"2024-04-29T14:29:16.679984+0000","last_fullsized":"2024-04-29T14:29:16.679984+0000","mapping_epoch":432,"log_start":"452'5800","ondisk_log_start":"452'5800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15842,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"522'15875","reported_seq":16148,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:16.277936+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:29:16.277936+0000","last_peered":"2024-04-29T14:29:16.277936+0000","last_clean":"2024-04-29T14:29:16.277936+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:29:16.277936+0000","last_undegraded":"2024-04-29T14:29:16.277936+0000","last_fullsized":"2024-04-29T14:29:16.277936+0000","mapping_epoch":432,"log_start":"452'5800","ondisk_log_start":"452'5800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48037888,"num_objects":733,"num_object_clones":0,"num_object_copies":1466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15875,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"522'15997","reported_seq":31724,"reported_epoch":522,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:15.880915+0000","last_change":"2024-04-29T14:26:16.052811+0000","last_active":"2024-04-29T14:29:15.880915+0000","last_peered":"2024-04-29T14:29:15.880915+0000","last_clean":"2024-04-29T14:25:54.182628+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:29:15.880915+0000","last_undegraded":"2024-04-29T14:26:15.818605+0000","last_fullsized":"2024-04-29T14:29:15.880915+0000","mapping_epoch":467,"log_start":"452'5900","ondisk_log_start":"452'5900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52494336,"num_objects":801,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15997,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7726,"num_bytes_recovered":506331136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":673},{"shards":"9,10","objects":128}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"522'15996","reported_seq":31684,"reported_epoch":522,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:16.680875+0000","last_change":"2024-04-29T14:26:16.126893+0000","last_active":"2024-04-29T14:29:16.680875+0000","last_peered":"2024-04-29T14:29:16.680875+0000","last_clean":"2024-04-29T14:25:53.790194+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:29:16.680875+0000","last_undegraded":"2024-04-29T14:26:15.820366+0000","last_fullsized":"2024-04-29T14:29:16.680875+0000","mapping_epoch":467,"log_start":"453'5900","ondisk_log_start":"453'5900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50462720,"num_objects":770,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15996,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7704,"num_bytes_recovered":504889344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":662},{"shards":"7,10","objects":108}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"522'15989","reported_seq":16267,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:16.576400+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:29:16.576400+0000","last_peered":"2024-04-29T14:29:16.576400+0000","last_clean":"2024-04-29T14:29:16.576400+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:29:16.576400+0000","last_undegraded":"2024-04-29T14:29:16.576400+0000","last_fullsized":"2024-04-29T14:29:16.576400+0000","mapping_epoch":432,"log_start":"453'5900","ondisk_log_start":"453'5900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50528256,"num_objects":771,"num_object_clones":0,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15989,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"522'15855","reported_seq":16258,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:15.266900+0000","last_change":"2024-04-29T14:28:59.693015+0000","last_active":"2024-04-29T14:29:15.266900+0000","last_peered":"2024-04-29T14:29:15.266900+0000","last_clean":"2024-04-29T14:29:15.266900+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:29:15.266900+0000","last_undegraded":"2024-04-29T14:29:15.266900+0000","last_fullsized":"2024-04-29T14:29:15.266900+0000","mapping_epoch":432,"log_start":"452'5800","ondisk_log_start":"452'5800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"517'14944","last_scrub_stamp":"2024-04-29T14:28:59.692847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:28:59.692847+0000","objects_scrubbed":1678,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:18.941948+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":1510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15855,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"522'15776","reported_seq":16108,"reported_epoch":522,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T14:29:15.388004+0000","last_change":"2024-04-29T14:29:15.143702+0000","last_active":"2024-04-29T14:29:15.388004+0000","last_peered":"2024-04-29T14:29:15.388004+0000","last_clean":"2024-04-29T14:29:15.388004+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:29:15.388004+0000","last_undegraded":"2024-04-29T14:29:15.388004+0000","last_fullsized":"2024-04-29T14:29:15.388004+0000","mapping_epoch":432,"log_start":"452'5700","ondisk_log_start":"452'5700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11565","last_scrub_stamp":"2024-04-29T14:27:46.065551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:27:46.065551+0000","objects_scrubbed":36,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55705600,"num_objects":850,"num_object_clones":0,"num_object_copies":1700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15740,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"522'15743","reported_seq":16024,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:16.573671+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:29:16.573671+0000","last_peered":"2024-04-29T14:29:16.573671+0000","last_clean":"2024-04-29T14:29:16.573671+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:29:16.573671+0000","last_undegraded":"2024-04-29T14:29:16.573671+0000","last_fullsized":"2024-04-29T14:29:16.573671+0000","mapping_epoch":432,"log_start":"453'5700","ondisk_log_start":"453'5700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":703,"num_object_clones":0,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15743,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"524'18975","reported_seq":19432,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-04-29T14:29:18.841769+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:29:18.841769+0000","last_peered":"2024-04-29T14:29:18.841769+0000","last_clean":"2024-04-29T14:29:18.841769+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:29:18.841769+0000","last_undegraded":"2024-04-29T14:29:18.841769+0000","last_fullsized":"2024-04-29T14:29:18.841769+0000","mapping_epoch":432,"log_start":"456'8900","ondisk_log_start":"456'8900","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16013,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"524'16078","reported_seq":32789,"reported_epoch":524,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:29:18.914890+0000","last_change":"2024-04-29T14:26:15.998964+0000","last_active":"2024-04-29T14:29:18.914890+0000","last_peered":"2024-04-29T14:29:18.914890+0000","last_clean":"2024-04-29T14:25:59.019434+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:29:18.914890+0000","last_undegraded":"2024-04-29T14:26:15.829402+0000","last_fullsized":"2024-04-29T14:29:18.914890+0000","mapping_epoch":467,"log_start":"453'6000","ondisk_log_start":"453'6000","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16078,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8255,"num_bytes_recovered":540999680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":["10"],"object_location_counts":[{"shards":"0,10","objects":525},{"shards":"10","objects":93}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1287,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-29T14:29:13.289358+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:29:13.289358+0000","last_peered":"2024-04-29T14:29:13.289358+0000","last_clean":"2024-04-29T14:29:13.289358+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:29:13.289358+0000","last_undegraded":"2024-04-29T14:29:13.289358+0000","last_fullsized":"2024-04-29T14:29:13.289358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":760283160,"num_objects":11602,"num_object_clones":0,"num_object_copies":23204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255326,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39450,"num_bytes_recovered":2585395200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4127399936,"data_stored":4127391792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161079,"ondisk_log_size":161079,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122316,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":669968,"kb_used_data":107076,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93075184,"statfs":{"total":95995035648,"available":95308988416,"internally_reserved":0,"allocated":109645824,"data_stored":108999772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":576387524},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383953,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638496,"kb_used_data":147792,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93106656,"statfs":{"total":95995035648,"available":95341215744,"internally_reserved":0,"allocated":151339008,"data_stored":150688010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":502462912},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593536,"kb_used_data":156984,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93151616,"statfs":{"total":95995035648,"available":95387254784,"internally_reserved":0,"allocated":160751616,"data_stored":160113208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422894,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234271,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":723980,"kb_used_data":169412,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93021172,"statfs":{"total":95995035648,"available":95253680128,"internally_reserved":0,"allocated":173477888,"data_stored":172831856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":567867850},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528612,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":757824,"kb_used_data":158840,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92987328,"statfs":{"total":95995035648,"available":95219023872,"internally_reserved":0,"allocated":162652160,"data_stored":162013752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":613349830},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575165,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929088,"kb_used_data":370360,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92816064,"statfs":{"total":95995035648,"available":95043649536,"internally_reserved":0,"allocated":379248640,"data_stored":378610232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497123,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986480,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966720,"kb_used_data":285752,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92778432,"statfs":{"total":95995035648,"available":95005114368,"internally_reserved":0,"allocated":292610048,"data_stored":291971640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368115712,"data_stored":368115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":283770880,"data_stored":283770880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":158728192,"data_stored":158728192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":164892672,"data_stored":164888600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":151519232,"data_stored":151519232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":148373504,"data_stored":148373504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":102305792,"data_stored":102301720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:29:24.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:29:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:24.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:24 smithi121 ceph-mon[44533]: osdmap e526: 12 total, 11 up, 7 in 2024-04-29T14:29:24.865 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:24.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:24.970 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:25.006 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:25.029 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:25.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:25.135 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:25.169 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:25.199 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:25.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:25.235 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:29:25.260 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:29:25.282 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:29:25.304 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:29:25.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:25.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:25 smithi194 ceph-mon[44089]: pgmap v1385: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 445 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 638/14242 objects degraded (4.480%); 25 MiB/s, 399 objects/s recovering 2024-04-29T14:29:25.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3885108966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:25.346 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:29:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:25.400 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:29:25.428 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:29:25.457 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:29:25.485 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:29:25.509 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:29:25.538 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:29:25.539 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:29:25.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:25 smithi040 ceph-mon[39581]: pgmap v1385: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 445 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 638/14242 objects degraded (4.480%); 25 MiB/s, 399 objects/s recovering 2024-04-29T14:29:25.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3885108966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:25.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:25 smithi121 ceph-mon[44533]: pgmap v1385: 17 pgs: 1 active+recovering+degraded+remapped, 3 active+recovering+degraded, 13 active+clean; 445 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 638/14242 objects degraded (4.480%); 25 MiB/s, 399 objects/s recovering 2024-04-29T14:29:25.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3885108966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:26.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:26.298 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:29:26.298 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:29:26.298 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:29:26.298 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:29:26.298 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:29:26.364 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:26.398 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:26.431 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:26.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:26.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:26.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.579 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:26.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:26.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:26.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:26.750 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:26.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:29:26.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:29:26.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:26.966 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:26.987 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:26.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:27.091 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:27.133 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:27.168 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:27.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:27.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:27.264 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:27 smithi194 ceph-mon[44089]: pgmap v1386: 17 pgs: 1 active+recovering+degraded+remapped, 2 active+recovering+degraded, 14 active+clean; 350 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 441/11188 objects degraded (3.942%); 21 MiB/s, 337 objects/s recovering 2024-04-29T14:29:27.264 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:27 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 441/11188 objects degraded (3.942%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:27.264 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:27 smithi194 ceph-mon[44089]: osdmap e527: 12 total, 11 up, 7 in 2024-04-29T14:29:27.275 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:27.310 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:27.343 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:27.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:27.450 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:27.493 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:27.526 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:27.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:27.583 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:27 smithi040 ceph-mon[39581]: pgmap v1386: 17 pgs: 1 active+recovering+degraded+remapped, 2 active+recovering+degraded, 14 active+clean; 350 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 441/11188 objects degraded (3.942%); 21 MiB/s, 337 objects/s recovering 2024-04-29T14:29:27.583 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:27 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 441/11188 objects degraded (3.942%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:27.583 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:27 smithi040 ceph-mon[39581]: osdmap e527: 12 total, 11 up, 7 in 2024-04-29T14:29:27.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:27 smithi121 ceph-mon[44533]: pgmap v1386: 17 pgs: 1 active+recovering+degraded+remapped, 2 active+recovering+degraded, 14 active+clean; 350 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 441/11188 objects degraded (3.942%); 21 MiB/s, 337 objects/s recovering 2024-04-29T14:29:27.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:27 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 441/11188 objects degraded (3.942%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:29:27.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:27 smithi121 ceph-mon[44533]: osdmap e527: 12 total, 11 up, 7 in 2024-04-29T14:29:27.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:27.673 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:27.713 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:27.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:27.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:27.874 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:27.883 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:28.139 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:28.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:28.246 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:28.280 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:28.308 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:28.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:28.320 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 133463 objects 2024-04-29T14:29:28.320 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :191.27 2024-04-29T14:29:28.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:28.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:28 smithi040 ceph-mon[39581]: pgmap v1388: 17 pgs: 17 active+clean; 194 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 807 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:29:28.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:28 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 441/11188 objects degraded (3.942%), 3 pgs degraded) 2024-04-29T14:29:28.444 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:28.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:28 smithi194 ceph-mon[44089]: pgmap v1388: 17 pgs: 17 active+clean; 194 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 807 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:29:28.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:28 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 441/11188 objects degraded (3.942%), 3 pgs degraded) 2024-04-29T14:29:28.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:28 smithi121 ceph-mon[44533]: pgmap v1388: 17 pgs: 17 active+clean; 194 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 807 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:29:28.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:28 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 441/11188 objects degraded (3.942%), 3 pgs degraded) 2024-04-29T14:29:28.693 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:28.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:28.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:29:28.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:28.828 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:28.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:28.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:29:28.914 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-04-29T14:29:28.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-04-29T14:29:28.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:28.980 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:28.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:29.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:29.102 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:29.132 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:29.153 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:29.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:29.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:29.283 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:29.301 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:29.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:29.323 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:29:29.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:29.558 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:29.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:29.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3729728331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:29.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:29 smithi194 ceph-mon[44089]: from='client.36302 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:29.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3729728331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:29.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:29 smithi040 ceph-mon[39581]: from='client.36302 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3729728331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:29.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:29 smithi121 ceph-mon[44533]: from='client.36302 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:29.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:29.690 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:29.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:29.815 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:29.827 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:29.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:29.919 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:29.939 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:29.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:30.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:30.070 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:30.088 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:30.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:30.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:30.219 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:30.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:30.341 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:30.367 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:30.385 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:30.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:30.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:30.517 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:30.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:30.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:30 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:30 smithi040 ceph-mon[39581]: from='client.36302 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:30 smithi040 ceph-mon[39581]: osdmap e528: 12 total, 11 up, 7 in 2024-04-29T14:29:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:29:30.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:30 smithi040 ceph-mon[39581]: pgmap v1390: 17 pgs: 17 active+clean; 132 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 6.7 MiB/s, 107 objects/s recovering 2024-04-29T14:29:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:30 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:30 smithi121 ceph-mon[44533]: from='client.36302 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:30 smithi121 ceph-mon[44533]: osdmap e528: 12 total, 11 up, 7 in 2024-04-29T14:29:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:29:30.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:30 smithi121 ceph-mon[44533]: pgmap v1390: 17 pgs: 17 active+clean; 132 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 6.7 MiB/s, 107 objects/s recovering 2024-04-29T14:29:30.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:30.666 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:30.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:30.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:30.769 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:30.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:30.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:30 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:30.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:30 smithi194 ceph-mon[44089]: from='client.36302 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:30.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:30 smithi194 ceph-mon[44089]: osdmap e528: 12 total, 11 up, 7 in 2024-04-29T14:29:30.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:29:30.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:30 smithi194 ceph-mon[44089]: pgmap v1390: 17 pgs: 17 active+clean; 132 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 6.7 MiB/s, 107 objects/s recovering 2024-04-29T14:29:30.816 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:31.571 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:31.573 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:29:31.573 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:29:31.573 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:29:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:31.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:31.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:29:31.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:29:31.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:29:31.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:31.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:29:31.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:29:31.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:29:31.710 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:31.732 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:31.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:31.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:31.868 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:31.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:31.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:31.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:32.023 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:32.043 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:32.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:32.103 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:29:32.105 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:29:32.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:32.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:29:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:33.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:33 smithi040 ceph-mon[39581]: pgmap v1391: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:29:33.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:33 smithi040 ceph-mon[39581]: from='client.16740 v1:172.21.15.40:0/4281576980' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:33.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:33 smithi040 ceph-mon[39581]: osdmap e529: 12 total, 11 up, 7 in 2024-04-29T14:29:33.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:33 smithi194 ceph-mon[44089]: pgmap v1391: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:29:33.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:33 smithi194 ceph-mon[44089]: from='client.16740 v1:172.21.15.40:0/4281576980' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:33.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:33 smithi194 ceph-mon[44089]: osdmap e529: 12 total, 11 up, 7 in 2024-04-29T14:29:33.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:33 smithi121 ceph-mon[44533]: pgmap v1391: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:29:33.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:33 smithi121 ceph-mon[44533]: from='client.16740 v1:172.21.15.40:0/4281576980' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:33.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:33 smithi121 ceph-mon[44533]: osdmap e529: 12 total, 11 up, 7 in 2024-04-29T14:29:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:34.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:34.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:34.996 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:35.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:35.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:35.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:35.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:35 smithi121 ceph-mon[44533]: pgmap v1393: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 457 op/s 2024-04-29T14:29:35.409 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:35 smithi040 ceph-mon[39581]: pgmap v1393: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 457 op/s 2024-04-29T14:29:35.410 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1391,"stamp":"2024-04-29T14:29:32.005284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":267109,"num_write_kb":8544571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41949,"num_bytes_recovered":2750761088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161104,"ondisk_log_size":161104,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3886428,"kb_used_data":9988,"kb_used_omap":10,"kb_used_meta":3876341,"kb_avail":652329636,"statfs":{"total":671965249536,"available":667985547264,"internally_reserved":0,"allocated":10227712,"data_stored":5493503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":3969373278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-466485272,"num_objects":-7119,"num_object_clones":0,"num_object_copies":-14238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-638,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7119,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":41811968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001380"},"pg_stats":[{"pgid":"5.f","version":"527'16964","reported_seq":34224,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442808+0000","last_change":"2024-04-29T14:29:22.949687+0000","last_active":"2024-04-29T14:29:29.442808+0000","last_peered":"2024-04-29T14:29:29.442808+0000","last_clean":"2024-04-29T14:29:29.442808+0000","last_became_active":"2024-04-29T14:26:16.010174+0000","last_became_peered":"2024-04-29T14:26:16.010174+0000","last_unstale":"2024-04-29T14:29:29.442808+0000","last_undegraded":"2024-04-29T14:29:29.442808+0000","last_fullsized":"2024-04-29T14:29:29.442808+0000","mapping_epoch":467,"log_start":"454'6900","ondisk_log_start":"454'6900","created":432,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7968","last_scrub_stamp":"2024-04-29T14:25:44.698252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.698252+0000","objects_scrubbed":7919,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:31.436223+0000","scrub_duration":2114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16964,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8482,"num_bytes_recovered":555876352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"527'16794","reported_seq":17075,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442808+0000","last_change":"2024-04-29T14:25:34.644892+0000","last_active":"2024-04-29T14:29:29.442808+0000","last_peered":"2024-04-29T14:29:29.442808+0000","last_clean":"2024-04-29T14:29:29.442808+0000","last_became_active":"2024-04-29T14:24:30.769169+0000","last_became_peered":"2024-04-29T14:24:30.769169+0000","last_unstale":"2024-04-29T14:29:29.442808+0000","last_undegraded":"2024-04-29T14:29:29.442808+0000","last_fullsized":"2024-04-29T14:29:29.442808+0000","mapping_epoch":432,"log_start":"454'6700","ondisk_log_start":"454'6700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7069","last_scrub_stamp":"2024-04-29T14:25:34.644792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:34.644792+0000","objects_scrubbed":7029,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:33.127774+0000","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16794,"num_write_kb":537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"527'16694","reported_seq":33679,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.322341+0000","last_change":"2024-04-29T14:29:23.277827+0000","last_active":"2024-04-29T14:29:29.322341+0000","last_peered":"2024-04-29T14:29:29.322341+0000","last_clean":"2024-04-29T14:29:29.322341+0000","last_became_active":"2024-04-29T14:29:23.239939+0000","last_became_peered":"2024-04-29T14:29:23.239939+0000","last_unstale":"2024-04-29T14:29:29.322341+0000","last_undegraded":"2024-04-29T14:29:29.322341+0000","last_fullsized":"2024-04-29T14:29:29.322341+0000","mapping_epoch":525,"log_start":"454'6600","ondisk_log_start":"454'6600","created":432,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'7099","last_scrub_stamp":"2024-04-29T14:25:36.136513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:36.136513+0000","objects_scrubbed":7053,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.953997+0000","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16694,"num_write_kb":534145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8346,"num_bytes_recovered":546963456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"527'16792","reported_seq":17086,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442809+0000","last_change":"2024-04-29T14:26:14.534761+0000","last_active":"2024-04-29T14:29:29.442809+0000","last_peered":"2024-04-29T14:29:29.442809+0000","last_clean":"2024-04-29T14:29:29.442809+0000","last_became_active":"2024-04-29T14:24:30.769098+0000","last_became_peered":"2024-04-29T14:24:30.769098+0000","last_unstale":"2024-04-29T14:29:29.442809+0000","last_undegraded":"2024-04-29T14:29:29.442809+0000","last_fullsized":"2024-04-29T14:29:29.442809+0000","mapping_epoch":432,"log_start":"454'6700","ondisk_log_start":"454'6700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8396","last_scrub_stamp":"2024-04-29T14:26:14.534638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.534638+0000","objects_scrubbed":8396,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:31.436292+0000","scrub_duration":671,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16792,"num_write_kb":537344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"527'16666","reported_seq":16954,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442772+0000","last_change":"2024-04-29T14:25:44.799891+0000","last_active":"2024-04-29T14:29:29.442772+0000","last_peered":"2024-04-29T14:29:29.442772+0000","last_clean":"2024-04-29T14:29:29.442772+0000","last_became_active":"2024-04-29T14:24:30.769181+0000","last_became_peered":"2024-04-29T14:24:30.769181+0000","last_unstale":"2024-04-29T14:29:29.442772+0000","last_undegraded":"2024-04-29T14:29:29.442772+0000","last_fullsized":"2024-04-29T14:29:29.442772+0000","mapping_epoch":432,"log_start":"454'6600","ondisk_log_start":"454'6600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7785","last_scrub_stamp":"2024-04-29T14:25:44.799690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:44.799690+0000","objects_scrubbed":7741,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:33.127757+0000","scrub_duration":1021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16666,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"527'18801","reported_seq":19217,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-29T14:29:28.309983+0000","last_change":"2024-04-29T14:26:14.430893+0000","last_active":"2024-04-29T14:29:28.309983+0000","last_peered":"2024-04-29T14:29:28.309983+0000","last_clean":"2024-04-29T14:29:28.309983+0000","last_became_active":"2024-04-29T14:24:30.235931+0000","last_became_peered":"2024-04-29T14:24:30.235931+0000","last_unstale":"2024-04-29T14:29:28.309983+0000","last_undegraded":"2024-04-29T14:29:28.309983+0000","last_fullsized":"2024-04-29T14:29:28.309983+0000","mapping_epoch":432,"log_start":"457'8800","ondisk_log_start":"457'8800","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'10528","last_scrub_stamp":"2024-04-29T14:26:14.430772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:14.430772+0000","objects_scrubbed":8273,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:31.726845+0000","scrub_duration":669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16546,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"527'16582","reported_seq":16872,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.322347+0000","last_change":"2024-04-29T14:25:48.636210+0000","last_active":"2024-04-29T14:29:29.322347+0000","last_peered":"2024-04-29T14:29:29.322347+0000","last_clean":"2024-04-29T14:29:29.322347+0000","last_became_active":"2024-04-29T14:24:30.235385+0000","last_became_peered":"2024-04-29T14:24:30.235385+0000","last_unstale":"2024-04-29T14:29:29.322347+0000","last_undegraded":"2024-04-29T14:29:29.322347+0000","last_fullsized":"2024-04-29T14:29:29.322347+0000","mapping_epoch":432,"log_start":"454'6500","ondisk_log_start":"454'6500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'7979","last_scrub_stamp":"2024-04-29T14:25:48.636059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.636059+0000","objects_scrubbed":7921,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.955662+0000","scrub_duration":1782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16582,"num_write_kb":530624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"527'16608","reported_seq":16893,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442921+0000","last_change":"2024-04-29T14:25:40.045105+0000","last_active":"2024-04-29T14:29:29.442921+0000","last_peered":"2024-04-29T14:29:29.442921+0000","last_clean":"2024-04-29T14:29:29.442921+0000","last_became_active":"2024-04-29T14:24:30.769099+0000","last_became_peered":"2024-04-29T14:24:30.769099+0000","last_unstale":"2024-04-29T14:29:29.442921+0000","last_undegraded":"2024-04-29T14:29:29.442921+0000","last_fullsized":"2024-04-29T14:29:29.442921+0000","mapping_epoch":432,"log_start":"454'6600","ondisk_log_start":"454'6600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7407","last_scrub_stamp":"2024-04-29T14:25:40.044959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.044959+0000","objects_scrubbed":7367,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:30.288401+0000","scrub_duration":1016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16608,"num_write_kb":531456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"527'16798","reported_seq":33886,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.443008+0000","last_change":"2024-04-29T14:29:22.872679+0000","last_active":"2024-04-29T14:29:29.443008+0000","last_peered":"2024-04-29T14:29:29.443008+0000","last_clean":"2024-04-29T14:29:29.443008+0000","last_became_active":"2024-04-29T14:26:16.038648+0000","last_became_peered":"2024-04-29T14:26:16.038648+0000","last_unstale":"2024-04-29T14:29:29.443008+0000","last_undegraded":"2024-04-29T14:29:29.443008+0000","last_fullsized":"2024-04-29T14:29:29.443008+0000","mapping_epoch":467,"log_start":"454'6700","ondisk_log_start":"454'6700","created":432,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'7479","last_scrub_stamp":"2024-04-29T14:25:40.322101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:40.322101+0000","objects_scrubbed":7424,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:33.127736+0000","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16798,"num_write_kb":537536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8399,"num_bytes_recovered":550436864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.6","version":"527'16766","reported_seq":33793,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.322340+0000","last_change":"2024-04-29T14:29:23.337119+0000","last_active":"2024-04-29T14:29:29.322340+0000","last_peered":"2024-04-29T14:29:29.322340+0000","last_clean":"2024-04-29T14:29:29.322340+0000","last_became_active":"2024-04-29T14:26:16.054003+0000","last_became_peered":"2024-04-29T14:26:16.054003+0000","last_unstale":"2024-04-29T14:29:29.322340+0000","last_undegraded":"2024-04-29T14:29:29.322340+0000","last_fullsized":"2024-04-29T14:29:29.322340+0000","mapping_epoch":467,"log_start":"454'6700","ondisk_log_start":"454'6700","created":432,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8070","last_scrub_stamp":"2024-04-29T14:25:49.240028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:49.240028+0000","objects_scrubbed":8029,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.971297+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16766,"num_write_kb":536512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8366,"num_bytes_recovered":548274176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"527'16760","reported_seq":17050,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442800+0000","last_change":"2024-04-29T14:25:48.533728+0000","last_active":"2024-04-29T14:29:29.442800+0000","last_peered":"2024-04-29T14:29:29.442800+0000","last_clean":"2024-04-29T14:29:29.442800+0000","last_became_active":"2024-04-29T14:24:30.769027+0000","last_became_peered":"2024-04-29T14:24:30.769027+0000","last_unstale":"2024-04-29T14:29:29.442800+0000","last_undegraded":"2024-04-29T14:29:29.442800+0000","last_fullsized":"2024-04-29T14:29:29.442800+0000","mapping_epoch":432,"log_start":"454'6700","ondisk_log_start":"454'6700","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'8032","last_scrub_stamp":"2024-04-29T14:25:48.533409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:25:48.533409+0000","objects_scrubbed":7979,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:31.436259+0000","scrub_duration":1691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16760,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"527'16610","reported_seq":17025,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442388+0000","last_change":"2024-04-29T14:28:59.693015+0000","last_active":"2024-04-29T14:29:29.442388+0000","last_peered":"2024-04-29T14:29:29.442388+0000","last_clean":"2024-04-29T14:29:29.442388+0000","last_became_active":"2024-04-29T14:24:30.769087+0000","last_became_peered":"2024-04-29T14:24:30.769087+0000","last_unstale":"2024-04-29T14:29:29.442388+0000","last_undegraded":"2024-04-29T14:29:29.442388+0000","last_fullsized":"2024-04-29T14:29:29.442388+0000","mapping_epoch":432,"log_start":"454'6600","ondisk_log_start":"454'6600","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"517'14944","last_scrub_stamp":"2024-04-29T14:28:59.692847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:28:59.692847+0000","objects_scrubbed":1678,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:18.941948+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16610,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"527'17313","reported_seq":17704,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442385+0000","last_change":"2024-04-29T14:29:20.195924+0000","last_active":"2024-04-29T14:29:29.442385+0000","last_peered":"2024-04-29T14:29:29.442385+0000","last_clean":"2024-04-29T14:29:29.442385+0000","last_became_active":"2024-04-29T14:24:30.769038+0000","last_became_peered":"2024-04-29T14:24:30.769038+0000","last_unstale":"2024-04-29T14:29:29.442385+0000","last_undegraded":"2024-04-29T14:29:29.442385+0000","last_fullsized":"2024-04-29T14:29:29.442385+0000","mapping_epoch":432,"log_start":"455'7301","ondisk_log_start":"455'7301","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'16725","last_scrub_stamp":"2024-04-29T14:29:20.195803+0000","last_deep_scrub":"524'16725","last_deep_scrub_stamp":"2024-04-29T14:29:20.195803+0000","last_clean_scrub_stamp":"2024-04-29T14:29:20.195803+0000","objects_scrubbed":723,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:20.939982+0000","scrub_duration":5050,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16590,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"527'16446","reported_seq":16739,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.442780+0000","last_change":"2024-04-29T14:26:11.432469+0000","last_active":"2024-04-29T14:29:29.442780+0000","last_peered":"2024-04-29T14:29:29.442780+0000","last_clean":"2024-04-29T14:29:29.442780+0000","last_became_active":"2024-04-29T14:24:30.236751+0000","last_became_peered":"2024-04-29T14:24:30.236751+0000","last_unstale":"2024-04-29T14:29:29.442780+0000","last_undegraded":"2024-04-29T14:29:29.442780+0000","last_fullsized":"2024-04-29T14:29:29.442780+0000","mapping_epoch":432,"log_start":"454'6400","ondisk_log_start":"454'6400","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'8223","last_scrub_stamp":"2024-04-29T14:26:11.432358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:11.432358+0000","objects_scrubbed":8223,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:31.457410+0000","scrub_duration":614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16446,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"527'19578","reported_seq":20041,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-29T14:29:28.301130+0000","last_change":"2024-04-29T14:26:13.431440+0000","last_active":"2024-04-29T14:29:28.301130+0000","last_peered":"2024-04-29T14:29:28.301130+0000","last_clean":"2024-04-29T14:29:28.301130+0000","last_became_active":"2024-04-29T14:24:30.235984+0000","last_became_peered":"2024-04-29T14:24:30.235984+0000","last_unstale":"2024-04-29T14:29:28.301130+0000","last_undegraded":"2024-04-29T14:29:28.301130+0000","last_fullsized":"2024-04-29T14:29:28.301130+0000","mapping_epoch":432,"log_start":"457'9500","ondisk_log_start":"457'9500","created":432,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11270","last_scrub_stamp":"2024-04-29T14:26:13.431323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:26:13.431323+0000","objects_scrubbed":8308,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:31.726826+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16616,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"527'16696","reported_seq":33602,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-29T14:29:28.321645+0000","last_change":"2024-04-29T14:29:19.616696+0000","last_active":"2024-04-29T14:29:28.321645+0000","last_peered":"2024-04-29T14:29:28.321645+0000","last_clean":"2024-04-29T14:29:28.321645+0000","last_became_active":"2024-04-29T14:26:15.982158+0000","last_became_peered":"2024-04-29T14:26:15.982158+0000","last_unstale":"2024-04-29T14:29:28.321645+0000","last_undegraded":"2024-04-29T14:29:28.321645+0000","last_fullsized":"2024-04-29T14:29:28.321645+0000","mapping_epoch":467,"log_start":"454'6600","ondisk_log_start":"454'6600","created":432,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:24:29.217284+0000","last_clean_scrub_stamp":"2024-04-29T14:24:29.217284+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:31.726794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16696,"num_write_kb":534272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8348,"num_bytes_recovered":547094528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1299,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-29T14:29:29.322442+0000","last_change":"2024-04-29T14:25:30.928018+0000","last_active":"2024-04-29T14:29:29.322442+0000","last_peered":"2024-04-29T14:29:29.322442+0000","last_clean":"2024-04-29T14:29:29.322442+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:29:29.322442+0000","last_undegraded":"2024-04-29T14:29:29.322442+0000","last_fullsized":"2024-04-29T14:29:29.322442+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:25:30.927859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:25:30.927859+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.953962+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266928,"num_write_kb":8541633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41941,"num_bytes_recovered":2748645376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122319,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578580,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93166572,"statfs":{"total":95995035648,"available":95402569728,"internally_reserved":0,"allocated":1843200,"data_stored":1163431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":590608836},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383955,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491968,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253184,"statfs":{"total":95995035648,"available":95491260416,"internally_reserved":0,"allocated":1298432,"data_stored":628529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":502462912},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645593,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":429516,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93315636,"statfs":{"total":95995035648,"available":95555211264,"internally_reserved":0,"allocated":1310720,"data_stored":634523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422897,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234274,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556372,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93188780,"statfs":{"total":95995035648,"available":95425310720,"internally_reserved":0,"allocated":1843200,"data_stored":1163451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":567867850},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528615,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596108,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149044,"statfs":{"total":95995035648,"available":95384621056,"internally_reserved":0,"allocated":1310720,"data_stored":634523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555856,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93189296,"statfs":{"total":95995035648,"available":95425839104,"internally_reserved":0,"allocated":1310720,"data_stored":634523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497125,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986483,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678028,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93067124,"statfs":{"total":95995035648,"available":95300734976,"internally_reserved":0,"allocated":1310720,"data_stored":634523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2749693952,"data_stored":2749693952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:29:35.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:29:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:35.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:35.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:35.542 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:35.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:35 smithi194 ceph-mon[44089]: pgmap v1393: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 457 op/s 2024-04-29T14:29:35.640 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:35.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:35.745 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:35.778 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:35.800 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:35.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:35.904 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:35.932 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:36.285 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:36.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:36.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:36.418 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:36.766 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:36.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:36.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:37.023 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:37.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:37.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:37.158 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:37.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:37.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:37.239 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_3' removed 2024-04-29T14:29:37.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:37.311 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:37.330 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:37.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:37.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:37 smithi040 ceph-mon[39581]: pgmap v1394: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 387 op/s 2024-04-29T14:29:37.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3025481574' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:29:37.434 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:37.463 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:37.483 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:37.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:37.531 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:29:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:29:37.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:29:37.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:29:37.532 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:29:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:37 smithi194 ceph-mon[44089]: pgmap v1394: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 387 op/s 2024-04-29T14:29:37.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3025481574' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:29:37.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:37.614 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:37 smithi121 ceph-mon[44533]: pgmap v1394: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 387 op/s 2024-04-29T14:29:37.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3025481574' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:29:37.712 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:37.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:37.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:37.842 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:37.861 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:37.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:37.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:37.993 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:38.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3025481574' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:29:38.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:38 smithi040 ceph-mon[39581]: osdmap e530: 12 total, 11 up, 7 in 2024-04-29T14:29:38.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:38 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:29:38.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:38 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:38.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3025481574' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:29:38.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:38 smithi194 ceph-mon[44089]: osdmap e530: 12 total, 11 up, 7 in 2024-04-29T14:29:38.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:38 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:29:38.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:38 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:38.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3025481574' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:29:38.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:38 smithi121 ceph-mon[44533]: osdmap e530: 12 total, 11 up, 7 in 2024-04-29T14:29:38.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:38 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:29:38.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:38 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:38.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:29:39.137 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:29:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:39 smithi040 ceph-mon[39581]: pgmap v1396: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:39 smithi040 ceph-mon[39581]: osdmap e531: 12 total, 11 up, 7 in 2024-04-29T14:29:39.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:39 smithi194 ceph-mon[44089]: pgmap v1396: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:39 smithi194 ceph-mon[44089]: osdmap e531: 12 total, 11 up, 7 in 2024-04-29T14:29:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:39 smithi121 ceph-mon[44533]: pgmap v1396: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:39 smithi121 ceph-mon[44533]: osdmap e531: 12 total, 11 up, 7 in 2024-04-29T14:29:39.908 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:29:39.932 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:29:39.951 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:29:39.971 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:29:39.991 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:29:40.014 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:29:40.035 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:29:40.055 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:29:40.076 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:29:40.096 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:29:40.117 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:29:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:29:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:29:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:29:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:29:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:29:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:29:40.118 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:29:40.118 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:29:40.118 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:29:40.118 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:29:40.118 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:29:40.118 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:29:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:29:40.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/247034302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:29:40.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:40 smithi194 ceph-mon[44089]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:29:40.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/113223326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:40.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:40.448 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/247034302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:29:40.448 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:40 smithi040 ceph-mon[39581]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:29:40.448 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/113223326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:40.448 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:40.472 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:29:40.473 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:29:40.473 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-04-29T14:29:40.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-04-29T14:29:40.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/247034302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:29:40.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:40 smithi121 ceph-mon[44533]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:29:40.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/113223326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:40.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:40.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:40.782 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:40.815 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:41.266 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:41.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:41.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:29:41.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:29:41.403 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:41.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:41.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:41.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: pgmap v1398: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:41.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:29:41.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:41 smithi040 ceph-mon[39581]: osdmap e532: 12 total, 11 up, 7 in 2024-04-29T14:29:41.564 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: pgmap v1398: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:41.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:29:41.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:41 smithi194 ceph-mon[44089]: osdmap e532: 12 total, 11 up, 7 in 2024-04-29T14:29:41.587 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:41.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: pgmap v1398: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:29:41.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:41 smithi121 ceph-mon[44533]: osdmap e532: 12 total, 11 up, 7 in 2024-04-29T14:29:41.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:41.720 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:41.868 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:29:41.868 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:29:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:42.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:42.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:42.303 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:42.435 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:42.452 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:42.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:42.527 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:29:42.556 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:42.595 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:42.614 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:42.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:42.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:42.746 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:42.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:42.791 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:29:42.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:42.903 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:42.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:42.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:43.028 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:43.058 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:43.076 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:43.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:43.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:43.209 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:43 smithi194 ceph-mon[44089]: pgmap v1400: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:43.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:43 smithi194 ceph-mon[44089]: osdmap e533: 12 total, 11 up, 7 in 2024-04-29T14:29:43.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:43 smithi040 ceph-mon[39581]: pgmap v1400: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:43.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:43 smithi040 ceph-mon[39581]: osdmap e533: 12 total, 11 up, 7 in 2024-04-29T14:29:43.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:29:43.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:43 smithi121 ceph-mon[44533]: pgmap v1400: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:43.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:43 smithi121 ceph-mon[44533]: osdmap e533: 12 total, 11 up, 7 in 2024-04-29T14:29:43.758 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:43.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:43.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:43.892 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:44.040 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:44.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:44.144 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:44.170 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:44.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:29:44.368 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:44.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:44.473 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:44.574 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:44.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:44.637 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:29:44.637 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:29:44.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:44.760 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:44.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:44.782 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:44.864 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:29:44.883 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1659,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":533,"num_osds":12,"num_up_osds":11,"osd_up_since":1714400760,"num_in_osds":7,"osd_in_since":1714400774,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1}],"num_pgs":1,"num_pools":1,"num_objects":2,"data_bytes":590368,"bytes_used":4017045504,"bytes_avail":667948204032,"bytes_total":671965249536},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":41,"modified":"2024-04-29T14:28:47.996826+0000","services":{}},"progress_events":{}} 2024-04-29T14:29:44.893 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:44.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:45.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:45.055 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:45.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:45.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:45.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_4' created 2024-04-29T14:29:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:45.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:45.325 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:29:45.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:45.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:45 smithi121 ceph-mon[44533]: pgmap v1402: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:45.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1907459555' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:29:45.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3867536441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-29T14:29:45.394 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:45.413 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:45.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:45.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:45.550 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:45.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:45 smithi194 ceph-mon[44089]: pgmap v1402: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:45.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1907459555' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:29:45.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3867536441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-29T14:29:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:45.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:45.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:45.589 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:29:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:45 smithi040 ceph-mon[39581]: pgmap v1402: 1 pgs: 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1907459555' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:29:45.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3867536441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-29T14:29:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:45.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:45.703 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:45.802 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:45.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:45.907 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:45.934 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:46.031 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:46.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:46.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-04-29T14:29:46.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:46.162 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:46.235 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:46.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:46.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3867536441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-29T14:29:46.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:46 smithi040 ceph-mon[39581]: osdmap e534: 12 total, 11 up, 7 in 2024-04-29T14:29:46.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3867536441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-29T14:29:46.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:46 smithi194 ceph-mon[44089]: osdmap e534: 12 total, 11 up, 7 in 2024-04-29T14:29:46.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:46.367 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:46.404 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:46.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:46.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:46.537 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:46.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3867536441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-29T14:29:46.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:46 smithi121 ceph-mon[44533]: osdmap e534: 12 total, 11 up, 7 in 2024-04-29T14:29:46.986 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:46.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:47.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:47.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:47.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:47.354 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:47.375 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:47.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:47.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:47 smithi040 ceph-mon[39581]: pgmap v1404: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:47.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:47 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:29:47.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:47 smithi040 ceph-mon[39581]: osdmap e535: 12 total, 11 up, 7 in 2024-04-29T14:29:47.479 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:47.504 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:29:47.504 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:29:47.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:47 smithi194 ceph-mon[44089]: pgmap v1404: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:47.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:47 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:29:47.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:47 smithi194 ceph-mon[44089]: osdmap e535: 12 total, 11 up, 7 in 2024-04-29T14:29:47.508 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:47.527 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:47.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:47.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:47 smithi121 ceph-mon[44533]: pgmap v1404: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:47.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:47 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:29:47.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:47 smithi121 ceph-mon[44533]: osdmap e535: 12 total, 11 up, 7 in 2024-04-29T14:29:47.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:47.659 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:47.687 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:47.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:47.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:47.819 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:47.968 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:47.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:47.994 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-29T14:29:47.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:29:48.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:48.101 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:48.128 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:48.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:48.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:29:48.201 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:29:48.220 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:29:48.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:48.241 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:29:48.264 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:48.273 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:29:48.296 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:29:48.317 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:29:48.338 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:29:48.358 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:29:48.379 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:29:48.399 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:29:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:29:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:29:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:29:48.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:48 smithi194 ceph-mon[44089]: osdmap e536: 12 total, 11 up, 7 in 2024-04-29T14:29:48.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3881926052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:48.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:48 smithi040 ceph-mon[39581]: osdmap e536: 12 total, 11 up, 7 in 2024-04-29T14:29:48.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3881926052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:48.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:48 smithi121 ceph-mon[44533]: osdmap e536: 12 total, 11 up, 7 in 2024-04-29T14:29:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3881926052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:48.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:48.921 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:48.950 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:48.969 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:48.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:49.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:49.100 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:49.119 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:49.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:49.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:49.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:29:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:29:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:49.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:29:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:29:49.408 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:29:49.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: pgmap v1407: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:29:49.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:49.534 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:49.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: pgmap v1407: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:29:49.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: pgmap v1407: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:29:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:49.841 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:49.845 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:49.882 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:49.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:49.990 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:50.022 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:50.043 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:50.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:50.147 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:50.177 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:50.198 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:50.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:50.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:50.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:50.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:50 smithi040 ceph-mon[39581]: osdmap e537: 12 total, 11 up, 7 in 2024-04-29T14:29:50.472 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:50.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:50.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:50.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:50 smithi194 ceph-mon[44089]: osdmap e537: 12 total, 11 up, 7 in 2024-04-29T14:29:50.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:50.606 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:50.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:50.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:50 smithi121 ceph-mon[44533]: osdmap e537: 12 total, 11 up, 7 in 2024-04-29T14:29:51.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:29:51.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:51 smithi194 ceph-mon[44089]: pgmap v1409: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:51.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:51 smithi040 ceph-mon[39581]: pgmap v1409: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:51.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:51 smithi121 ceph-mon[44533]: pgmap v1409: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:51.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:51.963 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:52.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:52.097 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:52.118 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:52.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:52.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:52.252 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:52.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:52.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:52.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:52.430 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:52 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:29:52.430 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:52 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:29:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:52.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:52.532 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:29:52.556 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:29:52.557 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:29:52.557 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:29:52.557 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:29:52.561 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:52 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:29:52.561 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:52 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:29:52.563 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:52.585 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:29:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:52.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:29:52.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:52 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:29:52.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:52 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:29:52.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:52.717 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:53.065 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:53.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:53.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:53.204 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:53.225 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:53.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:53.292 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:29:53.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:53 smithi040 ceph-mon[39581]: pgmap v1410: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:53.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3636671926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:53.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:53.361 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:53.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:53.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:53.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:53.516 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:53.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:53.538 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:53 smithi121 ceph-mon[44533]: pgmap v1410: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:53.538 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3636671926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:53.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:53.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:53 smithi194 ceph-mon[44089]: pgmap v1410: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:53.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3636671926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:29:53.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:53.667 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:53.687 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:53.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:53.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:53.824 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:53.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:29:54.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:54 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:29:54.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3636671926' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:54.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:54 smithi194 ceph-mon[44089]: osdmap e538: 12 total, 11 up, 7 in 2024-04-29T14:29:54.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:54 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:29:54.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3636671926' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:54.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:54 smithi040 ceph-mon[39581]: osdmap e538: 12 total, 11 up, 7 in 2024-04-29T14:29:54.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:54 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:29:54.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3636671926' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:29:54.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:54 smithi121 ceph-mon[44533]: osdmap e538: 12 total, 11 up, 7 in 2024-04-29T14:29:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:54.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:54.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:54.813 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:54.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:54.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:54.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:55.079 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:55.092 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:55.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:55.197 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:29:55.229 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:55.250 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:29:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:55.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:29:55.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:55.385 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:55.405 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:55.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:55.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:29:55.536 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:55 smithi194 ceph-mon[44089]: pgmap v1412: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:55.538 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:55.557 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:29:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:55.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:29:55.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:55 smithi040 ceph-mon[39581]: pgmap v1412: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:55.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:55 smithi121 ceph-mon[44533]: pgmap v1412: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:55.661 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:55.690 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:55.710 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:55.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:55.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:29:55.846 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:55.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:29:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:55.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:29:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:29:55.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:56.000 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:56.020 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:56.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:56.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:56.156 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:56.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:56.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:56.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:29:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:56.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:29:57.061 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:29:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:57.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:29:57.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:57.198 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:57.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:57.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:57.323 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:29:57.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:57 smithi040 ceph-mon[39581]: pgmap v1413: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:57.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:57 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:29:57.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:57 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:57.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:57 smithi040 ceph-mon[39581]: osdmap e539: 12 total, 11 up, 7 in 2024-04-29T14:29:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:29:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:57.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:29:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:57 smithi194 ceph-mon[44089]: pgmap v1413: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:57 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:29:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:57 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:57.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:57 smithi194 ceph-mon[44089]: osdmap e539: 12 total, 11 up, 7 in 2024-04-29T14:29:57.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:57 smithi121 ceph-mon[44533]: pgmap v1413: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:57.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:57 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:29:57.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:57 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:57.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:57 smithi121 ceph-mon[44533]: osdmap e539: 12 total, 11 up, 7 in 2024-04-29T14:29:57.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:57.687 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:57.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:57.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:57.784 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:29:57.784 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:29:57.785 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:29:57.785 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:29:57.785 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:29:57.792 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:29:57.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:29:57.813 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:29:57.833 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:29:57.844 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:57.853 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:29:57.866 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:29:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:57.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:29:57.874 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:29:57.895 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:29:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:29:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:29:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:29:57.896 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:29:57.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:29:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:29:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:58.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:29:58.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:29:58.158 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:58.205 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:29:58.207 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:29:58.304 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/918171770' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4065809989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.464 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: pgmap v1415: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:58.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:58 smithi040 ceph-mon[39581]: from='client.16800 v1:172.21.15.40:0/3318537738' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/918171770' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4065809989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: pgmap v1415: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:58 smithi194 ceph-mon[44089]: from='client.16800 v1:172.21.15.40:0/3318537738' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/918171770' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4065809989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: pgmap v1415: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:29:58.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:58 smithi121 ceph-mon[44533]: from='client.16800 v1:172.21.15.40:0/3318537738' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:29:59.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:59.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:59.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:29:59.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:29:59.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/918171770' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:29:59.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:29:59 smithi194 ceph-mon[44089]: osdmap e540: 12 total, 11 up, 7 in 2024-04-29T14:29:59.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:59.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:59.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:29:59.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:29:59.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/918171770' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:29:59.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:29:59 smithi040 ceph-mon[39581]: osdmap e540: 12 total, 11 up, 7 in 2024-04-29T14:29:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:29:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:29:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:29:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/918171770' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:29:59.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:29:59 smithi121 ceph-mon[44533]: osdmap e540: 12 total, 11 up, 7 in 2024-04-29T14:29:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:29:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:59.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:29:59.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:29:59.845 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:29:59.866 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:29:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:29:59.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:29:59.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:00.001 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:00.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:00.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:00.119 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1415,"stamp":"2024-04-29T14:29:58.010677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3923388,"kb_used_data":10856,"kb_used_omap":10,"kb_used_meta":3912437,"kb_avail":652292676,"statfs":{"total":671965249536,"available":667947700224,"internally_reserved":0,"allocated":11116544,"data_stored":5909855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":4006335582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001901"},"pg_stats":[{"pgid":"6.f","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897878+0000","last_change":"2024-04-29T14:29:46.264074+0000","last_active":"2024-04-29T14:29:53.897878+0000","last_peered":"2024-04-29T14:29:53.897878+0000","last_clean":"2024-04-29T14:29:53.897878+0000","last_became_active":"2024-04-29T14:29:46.263829+0000","last_became_peered":"2024-04-29T14:29:46.263829+0000","last_unstale":"2024-04-29T14:29:53.897878+0000","last_undegraded":"2024-04-29T14:29:53.897878+0000","last_fullsized":"2024-04-29T14:29:53.897878+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:54.574625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.898042+0000","last_change":"2024-04-29T14:29:46.267030+0000","last_active":"2024-04-29T14:29:53.898042+0000","last_peered":"2024-04-29T14:29:53.898042+0000","last_clean":"2024-04-29T14:29:53.898042+0000","last_became_active":"2024-04-29T14:29:46.266773+0000","last_became_peered":"2024-04-29T14:29:46.266773+0000","last_unstale":"2024-04-29T14:29:53.898042+0000","last_undegraded":"2024-04-29T14:29:53.898042+0000","last_fullsized":"2024-04-29T14:29:53.898042+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:46.138986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897859+0000","last_change":"2024-04-29T14:29:46.295684+0000","last_active":"2024-04-29T14:29:53.897859+0000","last_peered":"2024-04-29T14:29:53.897859+0000","last_clean":"2024-04-29T14:29:53.897859+0000","last_became_active":"2024-04-29T14:29:46.295428+0000","last_became_peered":"2024-04-29T14:29:46.295428+0000","last_unstale":"2024-04-29T14:29:53.897859+0000","last_undegraded":"2024-04-29T14:29:53.897859+0000","last_fullsized":"2024-04-29T14:29:53.897859+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:57.221222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.293904+0000","last_change":"2024-04-29T14:29:46.265824+0000","last_active":"2024-04-29T14:29:53.293904+0000","last_peered":"2024-04-29T14:29:53.293904+0000","last_clean":"2024-04-29T14:29:53.293904+0000","last_became_active":"2024-04-29T14:29:46.265543+0000","last_became_peered":"2024-04-29T14:29:46.265543+0000","last_unstale":"2024-04-29T14:29:53.293904+0000","last_undegraded":"2024-04-29T14:29:53.293904+0000","last_fullsized":"2024-04-29T14:29:53.293904+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:13.012555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897966+0000","last_change":"2024-04-29T14:29:46.267438+0000","last_active":"2024-04-29T14:29:53.897966+0000","last_peered":"2024-04-29T14:29:53.897966+0000","last_clean":"2024-04-29T14:29:53.897966+0000","last_became_active":"2024-04-29T14:29:46.267120+0000","last_became_peered":"2024-04-29T14:29:46.267120+0000","last_unstale":"2024-04-29T14:29:53.897966+0000","last_undegraded":"2024-04-29T14:29:53.897966+0000","last_fullsized":"2024-04-29T14:29:53.897966+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:52.016044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.293870+0000","last_change":"2024-04-29T14:29:46.295741+0000","last_active":"2024-04-29T14:29:53.293870+0000","last_peered":"2024-04-29T14:29:53.293870+0000","last_clean":"2024-04-29T14:29:53.293870+0000","last_became_active":"2024-04-29T14:29:46.295534+0000","last_became_peered":"2024-04-29T14:29:46.295534+0000","last_unstale":"2024-04-29T14:29:53.293870+0000","last_undegraded":"2024-04-29T14:29:53.293870+0000","last_fullsized":"2024-04-29T14:29:53.293870+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:51.932209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.397181+0000","last_change":"2024-04-29T14:29:46.295841+0000","last_active":"2024-04-29T14:29:53.397181+0000","last_peered":"2024-04-29T14:29:53.397181+0000","last_clean":"2024-04-29T14:29:53.397181+0000","last_became_active":"2024-04-29T14:29:46.295508+0000","last_became_peered":"2024-04-29T14:29:46.295508+0000","last_unstale":"2024-04-29T14:29:53.397181+0000","last_undegraded":"2024-04-29T14:29:53.397181+0000","last_fullsized":"2024-04-29T14:29:53.397181+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:57.867645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897944+0000","last_change":"2024-04-29T14:29:46.265023+0000","last_active":"2024-04-29T14:29:53.897944+0000","last_peered":"2024-04-29T14:29:53.897944+0000","last_clean":"2024-04-29T14:29:53.897944+0000","last_became_active":"2024-04-29T14:29:46.264710+0000","last_became_peered":"2024-04-29T14:29:46.264710+0000","last_unstale":"2024-04-29T14:29:53.897944+0000","last_undegraded":"2024-04-29T14:29:53.897944+0000","last_fullsized":"2024-04-29T14:29:53.897944+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:08.298591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897943+0000","last_change":"2024-04-29T14:29:46.266008+0000","last_active":"2024-04-29T14:29:53.897943+0000","last_peered":"2024-04-29T14:29:53.897943+0000","last_clean":"2024-04-29T14:29:53.897943+0000","last_became_active":"2024-04-29T14:29:46.265695+0000","last_became_peered":"2024-04-29T14:29:46.265695+0000","last_unstale":"2024-04-29T14:29:53.897943+0000","last_undegraded":"2024-04-29T14:29:53.897943+0000","last_fullsized":"2024-04-29T14:29:53.897943+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:49.279877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897981+0000","last_change":"2024-04-29T14:29:46.265287+0000","last_active":"2024-04-29T14:29:53.897981+0000","last_peered":"2024-04-29T14:29:53.897981+0000","last_clean":"2024-04-29T14:29:53.897981+0000","last_became_active":"2024-04-29T14:29:46.265022+0000","last_became_peered":"2024-04-29T14:29:46.265022+0000","last_unstale":"2024-04-29T14:29:53.897981+0000","last_undegraded":"2024-04-29T14:29:53.897981+0000","last_fullsized":"2024-04-29T14:29:53.897981+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:05.066292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.397202+0000","last_change":"2024-04-29T14:29:46.264951+0000","last_active":"2024-04-29T14:29:53.397202+0000","last_peered":"2024-04-29T14:29:53.397202+0000","last_clean":"2024-04-29T14:29:53.397202+0000","last_became_active":"2024-04-29T14:29:46.264654+0000","last_became_peered":"2024-04-29T14:29:46.264654+0000","last_unstale":"2024-04-29T14:29:53.397202+0000","last_undegraded":"2024-04-29T14:29:53.397202+0000","last_fullsized":"2024-04-29T14:29:53.397202+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:00.281669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.898154+0000","last_change":"2024-04-29T14:29:46.267153+0000","last_active":"2024-04-29T14:29:53.898154+0000","last_peered":"2024-04-29T14:29:53.898154+0000","last_clean":"2024-04-29T14:29:53.898154+0000","last_became_active":"2024-04-29T14:29:46.266681+0000","last_became_peered":"2024-04-29T14:29:46.266681+0000","last_unstale":"2024-04-29T14:29:53.898154+0000","last_undegraded":"2024-04-29T14:29:53.898154+0000","last_fullsized":"2024-04-29T14:29:53.898154+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:00.149652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897927+0000","last_change":"2024-04-29T14:29:46.266359+0000","last_active":"2024-04-29T14:29:53.897927+0000","last_peered":"2024-04-29T14:29:53.897927+0000","last_clean":"2024-04-29T14:29:53.897927+0000","last_became_active":"2024-04-29T14:29:46.266100+0000","last_became_peered":"2024-04-29T14:29:46.266100+0000","last_unstale":"2024-04-29T14:29:53.897927+0000","last_undegraded":"2024-04-29T14:29:53.897927+0000","last_fullsized":"2024-04-29T14:29:53.897927+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:52.243107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.397195+0000","last_change":"2024-04-29T14:29:46.264883+0000","last_active":"2024-04-29T14:29:53.397195+0000","last_peered":"2024-04-29T14:29:53.397195+0000","last_clean":"2024-04-29T14:29:53.397195+0000","last_became_active":"2024-04-29T14:29:46.264619+0000","last_became_peered":"2024-04-29T14:29:46.264619+0000","last_unstale":"2024-04-29T14:29:53.397195+0000","last_undegraded":"2024-04-29T14:29:53.397195+0000","last_fullsized":"2024-04-29T14:29:53.397195+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:49.978078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897854+0000","last_change":"2024-04-29T14:29:46.295774+0000","last_active":"2024-04-29T14:29:53.897854+0000","last_peered":"2024-04-29T14:29:53.897854+0000","last_clean":"2024-04-29T14:29:53.897854+0000","last_became_active":"2024-04-29T14:29:46.295491+0000","last_became_peered":"2024-04-29T14:29:46.295491+0000","last_unstale":"2024-04-29T14:29:53.897854+0000","last_undegraded":"2024-04-29T14:29:53.897854+0000","last_fullsized":"2024-04-29T14:29:53.897854+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:04.552992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":16,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.897927+0000","last_change":"2024-04-29T14:29:46.267944+0000","last_active":"2024-04-29T14:29:53.897927+0000","last_peered":"2024-04-29T14:29:53.897927+0000","last_clean":"2024-04-29T14:29:53.897927+0000","last_became_active":"2024-04-29T14:29:46.267596+0000","last_became_peered":"2024-04-29T14:29:46.267596+0000","last_unstale":"2024-04-29T14:29:53.897927+0000","last_undegraded":"2024-04-29T14:29:53.897927+0000","last_fullsized":"2024-04-29T14:29:53.897927+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.869284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1328,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-29T14:29:53.898061+0000","last_change":"2024-04-29T14:29:51.622414+0000","last_active":"2024-04-29T14:29:53.898061+0000","last_peered":"2024-04-29T14:29:53.898061+0000","last_clean":"2024-04-29T14:29:53.898061+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:29:53.898061+0000","last_undegraded":"2024-04-29T14:29:53.898061+0000","last_fullsized":"2024-04-29T14:29:53.898061+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:29:51.622280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:51.622280+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:53.114926+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122324,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596752,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93148400,"statfs":{"total":95995035648,"available":95383961600,"internally_reserved":0,"allocated":1970176,"data_stored":1222908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":609089988},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383960,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492092,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253060,"statfs":{"total":95995035648,"available":95491133440,"internally_reserved":0,"allocated":1425408,"data_stored":688019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":502462912},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":429640,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93315512,"statfs":{"total":95995035648,"available":95555084288,"internally_reserved":0,"allocated":1437696,"data_stored":694000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422902,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234279,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556496,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93188656,"statfs":{"total":95995035648,"available":95425183744,"internally_reserved":0,"allocated":1970176,"data_stored":1222928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":567867850},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528620,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596232,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93148920,"statfs":{"total":95995035648,"available":95384494080,"internally_reserved":0,"allocated":1437696,"data_stored":694000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575173,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738699,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555976,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93189176,"statfs":{"total":95995035648,"available":95425716224,"internally_reserved":0,"allocated":1437696,"data_stored":694000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986488,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696200,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93048952,"statfs":{"total":95995035648,"available":95282126848,"internally_reserved":0,"allocated":1437696,"data_stored":694000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:30:00.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:00.157 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:00.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:00.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:00.283 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:00.316 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:00.338 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:00.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:00.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:00 smithi040 ceph-mon[39581]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-29T14:30:00.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:00 smithi040 ceph-mon[39581]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-29T14:30:00.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:00 smithi040 ceph-mon[39581]: application not enabled on pool 'unique_pool_4' 2024-04-29T14:30:00.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:00 smithi040 ceph-mon[39581]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-29T14:30:00.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:00 smithi040 ceph-mon[39581]: pgmap v1417: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:30:00.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:00.476 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:00 smithi194 ceph-mon[44089]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-29T14:30:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:00 smithi194 ceph-mon[44089]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-29T14:30:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:00 smithi194 ceph-mon[44089]: application not enabled on pool 'unique_pool_4' 2024-04-29T14:30:00.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:00 smithi194 ceph-mon[44089]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-29T14:30:00.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:00 smithi194 ceph-mon[44089]: pgmap v1417: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:30:00.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:00 smithi121 ceph-mon[44533]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-29T14:30:00.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:00 smithi121 ceph-mon[44533]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-29T14:30:00.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:00 smithi121 ceph-mon[44533]: application not enabled on pool 'unique_pool_4' 2024-04-29T14:30:00.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:00 smithi121 ceph-mon[44533]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-29T14:30:00.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:00 smithi121 ceph-mon[44533]: pgmap v1417: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:30:01.026 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:01.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:01.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:01.165 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:01.186 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:01.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:01.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:01.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:01.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:30:01.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:30:01.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:01.484 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:30:01.484 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:30:01.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:01 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:30:01.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:01 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:30:01.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:01 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:30:01.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:01 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:30:01.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:01 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:30:01.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:01 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:30:01.699 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:01.732 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:01.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:01.836 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:01.870 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:01.891 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:01.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:01.996 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:02.025 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:02.045 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:02.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:02.054 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-04-29T14:30:02.091 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:30:02.091 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:30:02.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:02.180 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:02.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:02.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:02.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:30:02.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:30:02.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_134202 2024-04-29T14:30:02.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:30:02.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:30:02.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:02.355 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:02.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:02 smithi040 ceph-mon[39581]: pgmap v1418: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:30:02.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:02.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:02.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:02 smithi040 ceph-mon[39581]: osdmap e541: 12 total, 11 up, 7 in 2024-04-29T14:30:02.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:02.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:02.487 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:02.526 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:02.552 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:02.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:02.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:02 smithi121 ceph-mon[44533]: pgmap v1418: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:30:02.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:02.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:02.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:02 smithi121 ceph-mon[44533]: osdmap e541: 12 total, 11 up, 7 in 2024-04-29T14:30:02.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:02.702 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:02.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:02.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:02.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:02 smithi194 ceph-mon[44089]: pgmap v1418: 17 pgs: 17 active+clean; 577 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail 2024-04-29T14:30:02.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:02.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:02.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:02 smithi194 ceph-mon[44089]: osdmap e541: 12 total, 11 up, 7 in 2024-04-29T14:30:02.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:02.881 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:02.897 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:30:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:02.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:03.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:03.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:03.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:03.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:30:03.198 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:03.241 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:03.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 16 5199 5183 323.921 323.938 0.00347318 0.0030778 2024-04-29T14:30:03.269 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:03.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:03.374 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:03.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:30:03.404 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:03.404 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:03.407 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:03.432 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:03.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:03.538 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:03.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:03.574 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:03.599 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:03.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:03.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:03.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:03.703 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:03.741 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:03.770 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:03.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:03.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:03.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:03.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:04.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:04.095 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:04.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:04.142 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:04.227 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:04.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 16 9901 9885 308.886 293.875 0.00428283 0.00323238 2024-04-29T14:30:04.263 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:04.290 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:04.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:04.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:04.427 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:04.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:04 smithi194 ceph-mon[44089]: osdmap e542: 12 total, 11 up, 7 in 2024-04-29T14:30:04.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:04 smithi194 ceph-mon[44089]: pgmap v1421: 17 pgs: 17 active+clean; 98 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 16 MiB/s wr, 260 op/s 2024-04-29T14:30:04.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:04 smithi040 ceph-mon[39581]: osdmap e542: 12 total, 11 up, 7 in 2024-04-29T14:30:04.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:04 smithi040 ceph-mon[39581]: pgmap v1421: 17 pgs: 17 active+clean; 98 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 16 MiB/s wr, 260 op/s 2024-04-29T14:30:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:04.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:04 smithi121 ceph-mon[44533]: osdmap e542: 12 total, 11 up, 7 in 2024-04-29T14:30:04.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:04 smithi121 ceph-mon[44533]: pgmap v1421: 17 pgs: 17 active+clean; 98 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 16 MiB/s wr, 260 op/s 2024-04-29T14:30:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:04.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:04.734 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:04.772 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:04.798 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:04.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:04.904 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:04.935 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:05.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:05.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:05.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:05.250 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 16 14482 14466 301.354 286.312 0.00387376 0.00331502 2024-04-29T14:30:05.275 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:30:05.524 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:05.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2564725524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:30:05.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:05 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:05.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2564725524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:30:05.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:05 smithi040 ceph-mon[39581]: osdmap e543: 12 total, 11 up, 7 in 2024-04-29T14:30:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:05.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:05.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2564725524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:30:05.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:05 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2564725524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:30:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:05 smithi121 ceph-mon[44533]: osdmap e543: 12 total, 11 up, 7 in 2024-04-29T14:30:05.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:05.667 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:05.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:05.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:05.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2564725524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:30:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:05 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2564725524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:30:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:05 smithi194 ceph-mon[44089]: osdmap e543: 12 total, 11 up, 7 in 2024-04-29T14:30:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:05.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:05.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:05.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:06.007 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:06.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:06.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:06.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:06.207 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:06.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:06.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:06.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 16 18520 18504 289.104 252.375 0.00297629 0.00345644 2024-04-29T14:30:06.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:06.369 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:06.492 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:06.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:06.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:06.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:06 smithi040 ceph-mon[39581]: pgmap v1423: 17 pgs: 17 active+clean; 415 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:30:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:06 smithi121 ceph-mon[44533]: pgmap v1423: 17 pgs: 17 active+clean; 415 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:30:06.631 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:06 smithi194 ceph-mon[44089]: pgmap v1423: 17 pgs: 17 active+clean; 415 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:30:06.634 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:06.659 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:06.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:06.764 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:06.806 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:06.835 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:06.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:06.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:06.989 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:07.009 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:07.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:07.115 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:07.153 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:07.177 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:07.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:07.251 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 16 22296 22280 278.48 236 0.00206286 0.0035882 2024-04-29T14:30:07.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:07.324 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:07.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:07.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:07.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:07.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:08.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:08.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 16 25289 25273 263.195 187.062 0.00201181 0.00379609 2024-04-29T14:30:08.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:08.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:08.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:08 smithi040 ceph-mon[39581]: osdmap e544: 12 total, 11 up, 7 in 2024-04-29T14:30:08.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:08 smithi040 ceph-mon[39581]: pgmap v1425: 17 pgs: 17 active+clean; 1.0 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-29T14:30:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:08 smithi121 ceph-mon[44533]: osdmap e544: 12 total, 11 up, 7 in 2024-04-29T14:30:08.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:08 smithi121 ceph-mon[44533]: pgmap v1425: 17 pgs: 17 active+clean; 1.0 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-29T14:30:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:08 smithi194 ceph-mon[44089]: osdmap e544: 12 total, 11 up, 7 in 2024-04-29T14:30:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:08 smithi194 ceph-mon[44089]: pgmap v1425: 17 pgs: 17 active+clean; 1.0 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-29T14:30:09.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 16 28764 28748 256.621 217.188 0.00585045 0.00389476 2024-04-29T14:30:09.329 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:30:09.351 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:30:09.352 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2275319311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:09.372 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:30:09.394 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:30:09.428 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:30:09.452 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:30:09.479 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:30:09.502 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:30:09.526 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:30:09.535 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:09.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:09.560 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:30:09.576 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:30:09.576 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:30:09.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:181330 4.b 4:de33ed73:::benchmark_data_smithi194_103039_object24647:head [delete] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 558.04215808000004, 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039659663999999997, 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.577 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 181330 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535639+0000", 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535641+0000", 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.248e-06 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535645+0000", 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4429999999999996e-06 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535655+0000", 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0348e-05 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.535662+0000", 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0970000000000002e-06 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.537976+0000", 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023144060000000002 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.538102+0000", 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012558 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.538229+0000", 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126928 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.540630+0000", 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024012769999999998 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575087+0000", 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034457152999999997 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575253+0000", 2024-04-29T14:30:09.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000166441 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:51.575299+0000", 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5742999999999999e-05 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187965 4.b 4:d7e0b0fd:::benchmark_data_smithi194_103039_object31282:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:30:09.584 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 551.756440419, 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045912334999999999, 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187965 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821357+0000", 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821359+0000", 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5039999999999999e-06 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821363+0000", 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4939999999999997e-06 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821373+0000", 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0001e-05 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821380+0000", 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9589999999999998e-06 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821426+0000", 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5896999999999998e-05 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821500+0000", 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3790999999999999e-05 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.821568+0000", 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8112000000000001e-05 2024-04-29T14:30:09.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.823920+0000", 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023521409999999999 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867185+0000", 2024-04-29T14:30:09.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043265190000000002 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867231+0000", 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5726999999999998e-05 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867269+0000", 2024-04-29T14:30:09.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8519000000000002e-05 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187971 4.b 4:d0a42069:::benchmark_data_smithi194_103039_object31288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 551.74630931700005, 2024-04-29T14:30:09.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035888784999999999, 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187971 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831488+0000", 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831492+0000", 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3139999999999999e-06 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831502+0000", 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0713000000000001e-05 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831529+0000", 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6893000000000002e-05 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831548+0000", 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8697000000000001e-05 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831712+0000", 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016401100000000001 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831816+0000", 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104293 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831888+0000", 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1792000000000007e-05 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.836180+0000", 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004292293 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867316+0000", 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031135787000000002 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867351+0000", 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4573999999999998e-05 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867377+0000", 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6418e-05 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:187972 4.b 4:d50db589:::benchmark_data_smithi194_103039_object31289:head [delete] snapc 0=[] ondisk+write+known_if_redirected e366)", 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 551.74622340400003, 2024-04-29T14:30:09.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035895613999999999, 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 187972 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.601 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831574+0000", 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831575+0000", 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.24e-06 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831580+0000", 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1359999999999999e-06 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831589+0000", 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5740000000000002e-06 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.831602+0000", 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2585999999999999e-05 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834234+0000", 2024-04-29T14:30:09.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002631839 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834303+0000", 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9339999999999997e-05 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.834370+0000", 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.703e-05 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.838064+0000", 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036939109999999998 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867419+0000", 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029355322999999999 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867447+0000", 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8164999999999999e-05 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:20:57.867470+0000", 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.247e-05 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:195494 4.b 4:d756e1a7:::benchmark_data_smithi194_103039_object38811:head [delete] snapc 0=[] ondisk+write+known_if_redirected e368)", 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 544.79598138699998, 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039141214000000001, 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.608 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 195494 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781816+0000", 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781818+0000", 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.916e-06 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781824+0000", 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5049999999999998e-06 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781834+0000", 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0494e-05 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781842+0000", 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0260000000000007e-06 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.781908+0000", 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5474000000000004e-05 2024-04-29T14:30:09.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.782012+0000", 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104421 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.782099+0000", 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7496000000000004e-05 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.785711+0000", 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036118610000000001 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820846+0000", 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035134336000000002 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820907+0000", 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1050999999999995e-05 2024-04-29T14:30:09.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:21:04.820957+0000", 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0633999999999997e-05 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205500 4.b 4:dcf18273:::benchmark_data_smithi194_103039_object48817:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:30:09.614 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 469.21216327399998, 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038420901, 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205500 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365634+0000", 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365635+0000", 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8099999999999998e-07 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365637+0000", 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.889e-06 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365640+0000", 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.619e-06 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365644+0000", 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4359999999999998e-06 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372228+0000", 2024-04-29T14:30:09.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065845260000000003 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372328+0000", 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100299 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.372423+0000", 2024-04-29T14:30:09.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4629999999999994e-05 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.383904+0000", 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011481358000000001 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.403987+0000", 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020082472000000001 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404022+0000", 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4737999999999998e-05 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404055+0000", 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3553000000000002e-05 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205501 4.b 4:d60cc47a:::benchmark_data_smithi194_103039_object48818:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 469.21214965000001, 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038509798999999997, 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205501 2024-04-29T14:30:09.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365648+0000", 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9999999999999999e-07 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365650+0000", 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.545e-06 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365653+0000", 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.456e-06 2024-04-29T14:30:09.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.365656+0000", 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9390000000000002e-06 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.375824+0000", 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010167756 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.375966+0000", 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014242200000000001 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.376106+0000", 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140068 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.387047+0000", 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01094083 2024-04-29T14:30:09.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404096+0000", 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01704864 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404128+0000", 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2452e-05 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404158+0000", 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9391e-05 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.25522.0:205503 4.b 4:d23be51c:::benchmark_data_smithi194_103039_object48820:head [delete] snapc 0=[] ondisk+write+known_if_redirected e390)", 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 469.21059829799998, 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037041931, 2024-04-29T14:30:09.628 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.25522", 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 205503 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:30:09.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367199+0000", 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367200+0000", 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0580000000000001e-06 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367205+0000", 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6e-06 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367214+0000", 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5249999999999994e-06 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.367220+0000", 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6200000000000004e-06 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380152+0000", 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012932018 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380291+0000", 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013889600000000001 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.380429+0000", 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000138541 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.390770+0000", 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010340066 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:30:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404193+0000", 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013423403 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404221+0000", 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7872e-05 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:22:20.404241+0000", 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0332000000000001e-05 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:147726 5.4 e475/467 5:2361ef92:::benchmark_data_smithi194_121512_object14260:head v 475'9258, mlcod=439'912)", 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:26:42.095603+0000", 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 207.48219410300001, 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037681798000000002, 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.635 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.095603+0000", 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.095603+0000", 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.095604+0000", 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0800000000000004e-07 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.095611+0000", 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8940000000000004e-06 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.095619+0000", 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5700000000000004e-06 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.095625+0000", 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5189999999999998e-06 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.114987+0000", 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019361715000000002 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.115017+0000", 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9740999999999999e-05 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.133245+0000", 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018228807999999999 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.133285+0000", 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9743000000000002e-05 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:147743 5.4 e475/467 5:29c8882c:::benchmark_data_smithi194_121512_object14277:head v 475'9261, mlcod=439'916)", 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:26:42.194390+0000", 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 207.38340793200001, 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036201464000000003, 2024-04-29T14:30:09.640 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.194390+0000", 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.194390+0000", 2024-04-29T14:30:09.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.194391+0000", 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8840000000000001e-06 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.194401+0000", 2024-04-29T14:30:09.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8090000000000004e-06 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.194412+0000", 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1129e-05 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.194421+0000", 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2269999999999993e-06 2024-04-29T14:30:09.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.198944+0000", 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004523333 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.198971+0000", 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.724e-05 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.230549+0000", 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031577798999999997 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:42.230591+0000", 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2042999999999997e-05 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:155577 5.4 e478/467 5:20a39b06:::benchmark_data_smithi194_121512_object22111:head v 478'9715, mlcod=440'1371)", 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:26:55.711797+0000", 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 193.866000192, 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037024567000000001, 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.711797+0000", 2024-04-29T14:30:09.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.711797+0000", 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.711798+0000", 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0470000000000001e-06 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.711805+0000", 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4309999999999999e-06 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.711812+0000", 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5209999999999997e-06 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.711817+0000", 2024-04-29T14:30:09.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5700000000000003e-06 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.713620+0000", 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001802894 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.713644+0000", 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4185e-05 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.748766+0000", 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035121830999999999 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:26:55.748822+0000", 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6088000000000002e-05 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.650 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219428 5.4 e505/467 5:2335f3c6:::benchmark_data_smithi194_121512_object85962:head v 505'13658, mlcod=451'5314)", 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674734+0000", 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.903063467999999, 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036107325000000003, 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674734+0000", 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674734+0000", 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674735+0000", 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5899999999999996e-07 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674746+0000", 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1233000000000001e-05 2024-04-29T14:30:09.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674754+0000", 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1699999999999997e-06 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674760+0000", 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1890000000000004e-06 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.683616+0000", 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0088559199999999998 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.683641+0000", 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5245999999999999e-05 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.710802+0000", 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027160535999999999 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.710841+0000", 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9471999999999997e-05 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219437 5.4 e505/467 5:2842a7fb:::benchmark_data_smithi194_121512_object85971:head v 505'13659, mlcod=451'5314)", 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674770+0000", 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.903027393000002, 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036092447999999999, 2024-04-29T14:30:09.655 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674770+0000", 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674770+0000", 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674771+0000", 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9399999999999995e-07 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674787+0000", 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5991999999999999e-05 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674795+0000", 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9179999999999994e-06 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674800+0000", 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7440000000000004e-06 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.688752+0000", 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013951465 2024-04-29T14:30:09.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.688768+0000", 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5763000000000001e-05 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.710846+0000", 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022078159999999999 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.710863+0000", 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6912e-05 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219441 5.4 e505/467 5:26aa65fc:::benchmark_data_smithi194_121512_object85975:head v 505'13660, mlcod=451'5314)", 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674809+0000", 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.902988617000005, 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053316958999999997, 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674809+0000", 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674809+0000", 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674809+0000", 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4200000000000001e-07 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674814+0000", 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8980000000000004e-06 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674820+0000", 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6180000000000003e-06 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674825+0000", 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1209999999999998e-06 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.699483+0000", 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024658164999999999 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.699508+0000", 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4749e-05 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728084+0000", 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028576283000000001 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728126+0000", 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1683000000000001e-05 2024-04-29T14:30:09.664 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219451 5.4 e505/467 5:2e2ad772:::benchmark_data_smithi194_121512_object85985:head v 505'13661, mlcod=451'5314)", 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674832+0000", 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.902965038999994, 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053323963000000002, 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674832+0000", 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674832+0000", 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674833+0000", 2024-04-29T14:30:09.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5499999999999999e-07 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674837+0000", 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2690000000000001e-06 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674844+0000", 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5540000000000003e-06 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674849+0000", 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9609999999999996e-06 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.703786+0000", 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028937715999999999 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.703813+0000", 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7064e-05 2024-04-29T14:30:09.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728131+0000", 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024317252000000001 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728156+0000", 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5792e-05 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219473 5.4 e505/467 5:2c9f9058:::benchmark_data_smithi194_121512_object86007:head v 505'13662, mlcod=451'5314)", 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674854+0000", 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.902943105999995, 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053330800999999997, 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674854+0000", 2024-04-29T14:30:09.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674854+0000", 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674855+0000", 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3999999999999997e-07 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674863+0000", 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7419999999999997e-06 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674869+0000", 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7399999999999998e-06 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674874+0000", 2024-04-29T14:30:09.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0250000000000002e-06 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.707448+0000", 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032574095999999997 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.707465+0000", 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6765e-05 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728161+0000", 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020695425999999999 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728185+0000", 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4666999999999999e-05 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.674 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219477 5.4 e505/467 5:26f89fdf:::benchmark_data_smithi194_121512_object86011:head v 505'13663, mlcod=451'5314)", 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674881+0000", 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.902916059999995, 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053332431, 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674881+0000", 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674881+0000", 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674882+0000", 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4999999999999998e-07 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674886+0000", 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4000000000000002e-06 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674893+0000", 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5139999999999998e-06 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674898+0000", 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8670000000000002e-06 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.710894+0000", 2024-04-29T14:30:09.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035996062000000002 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.710910+0000", 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6113999999999999e-05 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728188+0000", 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017278410000000001 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728214+0000", 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5714000000000001e-05 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219484 5.4 e505/467 5:2e2035ed:::benchmark_data_smithi194_121512_object86018:head v 505'13664, mlcod=451'5314)", 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674904+0000", 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.902893156999994, 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053338744, 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674904+0000", 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674904+0000", 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674905+0000", 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.53e-07 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674912+0000", 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0129999999999996e-06 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674918+0000", 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7449999999999998e-06 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674926+0000", 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2799999999999998e-06 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.715044+0000", 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040118473000000002 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.715071+0000", 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6982e-05 2024-04-29T14:30:09.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728219+0000", 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013147417 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728243+0000", 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4481000000000002e-05 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(5.4 505/467 [PushOp(5:21de3d9b:::benchmark_data_smithi194_121512_object86029:head, version: 451'5317, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:21de3d9b:::benchmark_data_smithi194_121512_object86029:head@451'5317, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.674932+0000", 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.902865820000002, 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053152908999999998, 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674932+0000", 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674932+0000", 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674932+0000", 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5999999999999999e-07 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674969+0000", 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6677999999999997e-05 2024-04-29T14:30:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.674985+0000", 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5826999999999999e-05 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.675002+0000", 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7458000000000001e-05 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.725449+0000", 2024-04-29T14:30:09.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050446594999999997 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.725457+0000", 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2849999999999995e-06 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.728085+0000", 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026277060000000001 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26047.0:219520 5.c e505/432 5:38f18d24:::benchmark_data_smithi194_121512_object86054:head v 505'13821, mlcod=505'13819)", 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:28:34.686193+0000", 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.891604725999997, 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039255890000000002, 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.686193+0000", 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.686193+0000", 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.686194+0000", 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6440000000000001e-06 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.686204+0000", 2024-04-29T14:30:09.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1800000000000002e-06 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.686214+0000", 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0693000000000001e-05 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.686224+0000", 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7480000000000005e-06 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:30:09.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.699463+0000", 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013238982 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.699478+0000", 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5107e-05 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.725418+0000", 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025939825999999999 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:30:09.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:28:34.725449+0000", 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0710000000000002e-05 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:30:09.692 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:30:09.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:09.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2275319311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:09.693 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:30:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:30:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:30:09.726 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:09.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2275319311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:09.825 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:09.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:09.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:09.972 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:10.003 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:10.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:10.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:10.150 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:10.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:10.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:10.252 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 16 31939 31923 249.347 198.438 0.00560862 0.00400922 2024-04-29T14:30:10.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:10.350 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:30:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.360 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.361 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:10 smithi121 ceph-mon[44533]: pgmap v1426: 17 pgs: 17 active+clean; 1.2 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 190 MiB/s wr, 3.04k op/s 2024-04-29T14:30:10.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:10 smithi040 ceph-mon[39581]: pgmap v1426: 17 pgs: 17 active+clean; 1.2 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 190 MiB/s wr, 3.04k op/s 2024-04-29T14:30:10.673 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:10.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:10.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:10.809 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:10.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:10 smithi194 ceph-mon[44089]: pgmap v1426: 17 pgs: 17 active+clean; 1.2 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 190 MiB/s wr, 3.04k op/s 2024-04-29T14:30:11.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:30:11.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:30:11.238 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:11.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:11.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 15 34975 34960 242.693 189.812 0.0046289 0.0041186 2024-04-29T14:30:11.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:11.449 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:11.475 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:11.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:11.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:11 smithi040 ceph-mon[39581]: from='client.26533 v1:172.21.15.40:0/1957030854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:30:11.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:11.605 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1426,"stamp":"2024-04-29T14:30:10.012994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1333527072,"num_objects":20341,"num_object_clones":0,"num_object_copies":40682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20341,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":20520,"num_write_kb":1304634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20476,"ondisk_log_size":20476,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6245732,"kb_used_data":2270732,"kb_used_omap":10,"kb_used_meta":3974901,"kb_avail":649970332,"statfs":{"total":671965249536,"available":665569619968,"internally_reserved":0,"allocated":2325229568,"data_stored":2319805615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":4070298718},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1332936704,"num_objects":20339,"num_object_clones":0,"num_object_copies":40678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20339,"num_write_kb":1301696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.681314"},"pg_stats":[{"pgid":"6.f","version":"544'1751","reported_seq":1780,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-04-29T14:30:08.922840+0000","last_change":"2024-04-29T14:29:46.264074+0000","last_active":"2024-04-29T14:30:08.922840+0000","last_peered":"2024-04-29T14:30:08.922840+0000","last_clean":"2024-04-29T14:30:08.922840+0000","last_became_active":"2024-04-29T14:29:46.263829+0000","last_became_peered":"2024-04-29T14:29:46.263829+0000","last_unstale":"2024-04-29T14:30:08.922840+0000","last_undegraded":"2024-04-29T14:30:08.922840+0000","last_fullsized":"2024-04-29T14:30:08.922840+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1751,"log_dups_size":0,"ondisk_log_size":1751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:53.723687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114753536,"num_objects":1751,"num_object_clones":0,"num_object_copies":3502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1751,"num_write_kb":112064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.e","version":"543'1273","reported_seq":1299,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.612748+0000","last_change":"2024-04-29T14:29:46.267030+0000","last_active":"2024-04-29T14:30:06.612748+0000","last_peered":"2024-04-29T14:30:06.612748+0000","last_clean":"2024-04-29T14:30:06.612748+0000","last_became_active":"2024-04-29T14:29:46.266773+0000","last_became_peered":"2024-04-29T14:29:46.266773+0000","last_unstale":"2024-04-29T14:30:06.612748+0000","last_undegraded":"2024-04-29T14:30:06.612748+0000","last_fullsized":"2024-04-29T14:30:06.612748+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:01.096742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83427328,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1273,"num_write_kb":81472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"544'1697","reported_seq":1725,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-04-29T14:30:08.928675+0000","last_change":"2024-04-29T14:29:46.295684+0000","last_active":"2024-04-29T14:30:08.928675+0000","last_peered":"2024-04-29T14:30:08.928675+0000","last_clean":"2024-04-29T14:30:08.928675+0000","last_became_active":"2024-04-29T14:29:46.295428+0000","last_became_peered":"2024-04-29T14:29:46.295428+0000","last_unstale":"2024-04-29T14:30:08.928675+0000","last_undegraded":"2024-04-29T14:30:08.928675+0000","last_fullsized":"2024-04-29T14:30:08.928675+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1697,"log_dups_size":0,"ondisk_log_size":1697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:47.052832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":3394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1697,"num_write_kb":108608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.c","version":"543'947","reported_seq":974,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:05.389279+0000","last_change":"2024-04-29T14:29:46.265824+0000","last_active":"2024-04-29T14:30:05.389279+0000","last_peered":"2024-04-29T14:30:05.389279+0000","last_clean":"2024-04-29T14:30:05.389279+0000","last_became_active":"2024-04-29T14:29:46.265543+0000","last_became_peered":"2024-04-29T14:29:46.265543+0000","last_unstale":"2024-04-29T14:30:05.389279+0000","last_undegraded":"2024-04-29T14:30:05.389279+0000","last_fullsized":"2024-04-29T14:30:05.389279+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:54.589485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62062592,"num_objects":947,"num_object_clones":0,"num_object_copies":1894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":60608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"543'1311","reported_seq":1338,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.694562+0000","last_change":"2024-04-29T14:29:46.267438+0000","last_active":"2024-04-29T14:30:06.694562+0000","last_peered":"2024-04-29T14:30:06.694562+0000","last_clean":"2024-04-29T14:30:06.694562+0000","last_became_active":"2024-04-29T14:29:46.267120+0000","last_became_peered":"2024-04-29T14:29:46.267120+0000","last_unstale":"2024-04-29T14:30:06.694562+0000","last_undegraded":"2024-04-29T14:30:06.694562+0000","last_fullsized":"2024-04-29T14:30:06.694562+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1311,"log_dups_size":0,"ondisk_log_size":1311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:13.031163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85917696,"num_objects":1311,"num_object_clones":0,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1311,"num_write_kb":83904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"543'953","reported_seq":980,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:05.395670+0000","last_change":"2024-04-29T14:29:46.295741+0000","last_active":"2024-04-29T14:30:05.395670+0000","last_peered":"2024-04-29T14:30:05.395670+0000","last_clean":"2024-04-29T14:30:05.395670+0000","last_became_active":"2024-04-29T14:29:46.295534+0000","last_became_peered":"2024-04-29T14:29:46.295534+0000","last_unstale":"2024-04-29T14:30:05.395670+0000","last_undegraded":"2024-04-29T14:30:05.395670+0000","last_fullsized":"2024-04-29T14:30:05.395670+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:14.426093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62455808,"num_objects":953,"num_object_clones":0,"num_object_copies":1906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":953,"num_write_kb":60992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"543'1062","reported_seq":1089,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:05.900121+0000","last_change":"2024-04-29T14:29:46.295841+0000","last_active":"2024-04-29T14:30:05.900121+0000","last_peered":"2024-04-29T14:30:05.900121+0000","last_clean":"2024-04-29T14:30:05.900121+0000","last_became_active":"2024-04-29T14:29:46.295508+0000","last_became_peered":"2024-04-29T14:29:46.295508+0000","last_unstale":"2024-04-29T14:30:05.900121+0000","last_undegraded":"2024-04-29T14:30:05.900121+0000","last_fullsized":"2024-04-29T14:30:05.900121+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1062,"log_dups_size":0,"ondisk_log_size":1062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:11.220174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69599232,"num_objects":1062,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1062,"num_write_kb":67968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"543'1256","reported_seq":1283,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.684096+0000","last_change":"2024-04-29T14:29:46.265023+0000","last_active":"2024-04-29T14:30:06.684096+0000","last_peered":"2024-04-29T14:30:06.684096+0000","last_clean":"2024-04-29T14:30:06.684096+0000","last_became_active":"2024-04-29T14:29:46.264710+0000","last_became_peered":"2024-04-29T14:29:46.264710+0000","last_unstale":"2024-04-29T14:30:06.684096+0000","last_undegraded":"2024-04-29T14:30:06.684096+0000","last_fullsized":"2024-04-29T14:30:06.684096+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:06.504101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82313216,"num_objects":1256,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1256,"num_write_kb":80384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"543'1221","reported_seq":1247,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.601569+0000","last_change":"2024-04-29T14:29:46.266008+0000","last_active":"2024-04-29T14:30:06.601569+0000","last_peered":"2024-04-29T14:30:06.601569+0000","last_clean":"2024-04-29T14:30:06.601569+0000","last_became_active":"2024-04-29T14:29:46.265695+0000","last_became_peered":"2024-04-29T14:29:46.265695+0000","last_unstale":"2024-04-29T14:30:06.601569+0000","last_undegraded":"2024-04-29T14:30:06.601569+0000","last_fullsized":"2024-04-29T14:30:06.601569+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1221,"log_dups_size":0,"ondisk_log_size":1221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:08.904025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80019456,"num_objects":1221,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":78144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"543'1266","reported_seq":1292,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.699812+0000","last_change":"2024-04-29T14:29:46.265287+0000","last_active":"2024-04-29T14:30:06.699812+0000","last_peered":"2024-04-29T14:30:06.699812+0000","last_clean":"2024-04-29T14:30:06.699812+0000","last_became_active":"2024-04-29T14:29:46.265022+0000","last_became_peered":"2024-04-29T14:29:46.265022+0000","last_unstale":"2024-04-29T14:30:06.699812+0000","last_undegraded":"2024-04-29T14:30:06.699812+0000","last_fullsized":"2024-04-29T14:30:06.699812+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1266,"log_dups_size":0,"ondisk_log_size":1266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:52.635956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82968576,"num_objects":1266,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1266,"num_write_kb":81024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"543'1050","reported_seq":1077,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:05.906355+0000","last_change":"2024-04-29T14:29:46.264951+0000","last_active":"2024-04-29T14:30:05.906355+0000","last_peered":"2024-04-29T14:30:05.906355+0000","last_clean":"2024-04-29T14:30:05.906355+0000","last_became_active":"2024-04-29T14:29:46.264654+0000","last_became_peered":"2024-04-29T14:29:46.264654+0000","last_unstale":"2024-04-29T14:30:05.906355+0000","last_undegraded":"2024-04-29T14:30:05.906355+0000","last_fullsized":"2024-04-29T14:30:05.906355+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:55.963493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68812800,"num_objects":1050,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1050,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"543'1250","reported_seq":1277,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.698892+0000","last_change":"2024-04-29T14:29:46.267153+0000","last_active":"2024-04-29T14:30:06.698892+0000","last_peered":"2024-04-29T14:30:06.698892+0000","last_clean":"2024-04-29T14:30:06.698892+0000","last_became_active":"2024-04-29T14:29:46.266681+0000","last_became_peered":"2024-04-29T14:29:46.266681+0000","last_unstale":"2024-04-29T14:30:06.698892+0000","last_undegraded":"2024-04-29T14:30:06.698892+0000","last_fullsized":"2024-04-29T14:30:06.698892+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:04.029935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"543'1229","reported_seq":1256,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.612152+0000","last_change":"2024-04-29T14:29:46.266359+0000","last_active":"2024-04-29T14:30:06.612152+0000","last_peered":"2024-04-29T14:30:06.612152+0000","last_clean":"2024-04-29T14:30:06.612152+0000","last_became_active":"2024-04-29T14:29:46.266100+0000","last_became_peered":"2024-04-29T14:29:46.266100+0000","last_unstale":"2024-04-29T14:30:06.612152+0000","last_undegraded":"2024-04-29T14:30:06.612152+0000","last_fullsized":"2024-04-29T14:30:06.612152+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1229,"log_dups_size":0,"ondisk_log_size":1229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:45.543095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80543744,"num_objects":1229,"num_object_clones":0,"num_object_copies":2458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1229,"num_write_kb":78656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"543'1054","reported_seq":1081,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:05.901758+0000","last_change":"2024-04-29T14:29:46.264883+0000","last_active":"2024-04-29T14:30:05.901758+0000","last_peered":"2024-04-29T14:30:05.901758+0000","last_clean":"2024-04-29T14:30:05.901758+0000","last_became_active":"2024-04-29T14:29:46.264619+0000","last_became_peered":"2024-04-29T14:29:46.264619+0000","last_unstale":"2024-04-29T14:30:05.901758+0000","last_undegraded":"2024-04-29T14:30:05.901758+0000","last_fullsized":"2024-04-29T14:30:05.901758+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1054,"log_dups_size":0,"ondisk_log_size":1054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:14.110181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69074944,"num_objects":1054,"num_object_clones":0,"num_object_copies":2108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":67456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"544'1741","reported_seq":1770,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-04-29T14:30:08.914089+0000","last_change":"2024-04-29T14:29:46.295774+0000","last_active":"2024-04-29T14:30:08.914089+0000","last_peered":"2024-04-29T14:30:08.914089+0000","last_clean":"2024-04-29T14:30:08.914089+0000","last_became_active":"2024-04-29T14:29:46.295491+0000","last_became_peered":"2024-04-29T14:29:46.295491+0000","last_unstale":"2024-04-29T14:30:08.914089+0000","last_undegraded":"2024-04-29T14:30:08.914089+0000","last_fullsized":"2024-04-29T14:30:08.914089+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1741,"log_dups_size":0,"ondisk_log_size":1741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:31:10.882019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114098176,"num_objects":1741,"num_object_clones":0,"num_object_copies":3482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1741,"num_write_kb":111424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.7","version":"543'1278","reported_seq":1305,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:06.600035+0000","last_change":"2024-04-29T14:29:46.267944+0000","last_active":"2024-04-29T14:30:06.600035+0000","last_peered":"2024-04-29T14:30:06.600035+0000","last_clean":"2024-04-29T14:30:06.600035+0000","last_became_active":"2024-04-29T14:29:46.267596+0000","last_became_peered":"2024-04-29T14:29:46.267596+0000","last_unstale":"2024-04-29T14:30:06.600035+0000","last_undegraded":"2024-04-29T14:30:06.600035+0000","last_fullsized":"2024-04-29T14:30:06.600035+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:45.239629+0000","objects_scrubbed":0,"log_size":1278,"log_dups_size":0,"ondisk_log_size":1278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:47.451485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83755008,"num_objects":1278,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1278,"num_write_kb":81792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1338,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-29T14:30:05.341189+0000","last_change":"2024-04-29T14:29:51.622414+0000","last_active":"2024-04-29T14:30:05.341189+0000","last_peered":"2024-04-29T14:30:05.341189+0000","last_clean":"2024-04-29T14:30:05.341189+0000","last_became_active":"2024-04-29T14:07:20.632242+0000","last_became_peered":"2024-04-29T14:07:20.632242+0000","last_unstale":"2024-04-29T14:30:05.341189+0000","last_undegraded":"2024-04-29T14:30:05.341189+0000","last_fullsized":"2024-04-29T14:30:05.341189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:29:51.622280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:29:51.622280+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:53.114926+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1332936704,"num_objects":20339,"num_object_clones":0,"num_object_copies":40678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20339,"num_write_kb":1301696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2566455296,"data_stored":2566455296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20339,"ondisk_log_size":20339,"up":32,"acting":32,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122326,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":803020,"kb_used_data":212352,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92942132,"statfs":{"total":95995035648,"available":95172743168,"internally_reserved":0,"allocated":217448448,"data_stored":216670137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":604830148},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383963,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913612,"kb_used_data":413188,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92831540,"statfs":{"total":95995035648,"available":95059496960,"internally_reserved":0,"allocated":423104512,"data_stored":422323489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":512424384},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645600,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636348,"kb_used_data":212268,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93108804,"statfs":{"total":95995035648,"available":95343415296,"internally_reserved":0,"allocated":217362432,"data_stored":216594007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":434239936},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422904,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234281,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1213516,"kb_used_data":614528,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92531636,"statfs":{"total":95995035648,"available":94752395264,"internally_reserved":0,"allocated":629276672,"data_stored":628498381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528622,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870904,"kb_used_data":276076,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92874248,"statfs":{"total":95995035648,"available":95103229952,"internally_reserved":0,"allocated":282701824,"data_stored":281933399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575175,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738701,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831176,"kb_used_data":276600,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92913976,"statfs":{"total":95995035648,"available":95143911424,"internally_reserved":0,"allocated":283238400,"data_stored":282463661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497133,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986490,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":977156,"kb_used_data":265720,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":92767996,"statfs":{"total":95995035648,"available":94994427904,"internally_reserved":0,"allocated":272097280,"data_stored":271322541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303169536,"data_stored":303169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":328007680,"data_stored":328007680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":311623680,"data_stored":311623680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":667615232,"data_stored":667615232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":278200320,"data_stored":278200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":450297856,"data_stored":450297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":227540992,"data_stored":227540992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:30:11.607 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:30:11.607 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-29T14:30:11.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:11 smithi121 ceph-mon[44533]: from='client.26533 v1:172.21.15.40:0/1957030854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:30:11.631 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:11.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:11.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:11.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:11.796 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:11.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:11 smithi194 ceph-mon[44089]: from='client.26533 v1:172.21.15.40:0/1957030854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:30:11.852 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:11.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:11.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:12.004 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:12.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:12.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:12.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:12.172 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:12.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 15 37809 37794 236.136 177.125 0.00926609 0.00423237 2024-04-29T14:30:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:12.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:13.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:13.097 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:13.253 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 40616 40600 230.612 175.375 0.00534343 0.00433415 2024-04-29T14:30:13.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:13 smithi040 ceph-mon[39581]: pgmap v1427: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 251 MiB/s wr, 4.01k op/s 2024-04-29T14:30:13.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:13 smithi040 ceph-mon[39581]: osdmap e545: 12 total, 11 up, 7 in 2024-04-29T14:30:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:13 smithi194 ceph-mon[44089]: pgmap v1427: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 251 MiB/s wr, 4.01k op/s 2024-04-29T14:30:13.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:13 smithi194 ceph-mon[44089]: osdmap e545: 12 total, 11 up, 7 in 2024-04-29T14:30:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:13 smithi121 ceph-mon[44533]: pgmap v1427: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 251 MiB/s wr, 4.01k op/s 2024-04-29T14:30:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:13 smithi121 ceph-mon[44533]: osdmap e545: 12 total, 11 up, 7 in 2024-04-29T14:30:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:30:13.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:30:14.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 43278 43262 225.259 166.375 0.00351771 0.00443517 2024-04-29T14:30:14.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:14.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:14.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:14.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:14.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:14.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:15.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 16 45489 45473 218.562 138.188 0.00263889 0.00457275 2024-04-29T14:30:15.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:15 smithi194 ceph-mon[44089]: pgmap v1429: 17 pgs: 17 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 232 MiB/s wr, 3.72k op/s 2024-04-29T14:30:15.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:15 smithi194 ceph-mon[44089]: osdmap e546: 12 total, 11 up, 7 in 2024-04-29T14:30:15.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:15 smithi040 ceph-mon[39581]: pgmap v1429: 17 pgs: 17 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 232 MiB/s wr, 3.72k op/s 2024-04-29T14:30:15.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:15 smithi040 ceph-mon[39581]: osdmap e546: 12 total, 11 up, 7 in 2024-04-29T14:30:15.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:15 smithi121 ceph-mon[44533]: pgmap v1429: 17 pgs: 17 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 232 MiB/s wr, 3.72k op/s 2024-04-29T14:30:15.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:15 smithi121 ceph-mon[44533]: osdmap e546: 12 total, 11 up, 7 in 2024-04-29T14:30:15.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:30:16.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 16 48056 48040 214.41 160.438 0.00858018 0.00466147 2024-04-29T14:30:16.351 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:16.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:16.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:16.510 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:16.510 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:16.512 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:16.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:16.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:16.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:16.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:16.605 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:16.605 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:16.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:16.689 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:16.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:16.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:16.824 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:16.884 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:16.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:16.900 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:30:16.900 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:30:16.900 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:30:16.900 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:30:16.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:17.014 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:17.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:17.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:17.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:17.187 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:17.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:17.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:17.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 15 50162 50147 208.895 131.688 0.00762475 0.00478513 2024-04-29T14:30:17.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:17.354 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:17.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:17 smithi194 ceph-mon[44089]: pgmap v1431: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 186 MiB/s wr, 2.98k op/s 2024-04-29T14:30:17.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:17 smithi194 ceph-mon[44089]: osdmap e547: 12 total, 11 up, 7 in 2024-04-29T14:30:17.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:17 smithi040 ceph-mon[39581]: pgmap v1431: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 186 MiB/s wr, 2.98k op/s 2024-04-29T14:30:17.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:17 smithi040 ceph-mon[39581]: osdmap e547: 12 total, 11 up, 7 in 2024-04-29T14:30:17.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:17 smithi121 ceph-mon[44533]: pgmap v1431: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 186 MiB/s wr, 2.98k op/s 2024-04-29T14:30:17.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:17 smithi121 ceph-mon[44533]: osdmap e547: 12 total, 11 up, 7 in 2024-04-29T14:30:17.769 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:30:17.904 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:17.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:17.927 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:18.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:18.039 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:18.240 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:18.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:18.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 16 52518 52502 205.038 147.188 0.0121577 0.00487547 2024-04-29T14:30:18.279 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:30:18.300 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1358590957' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:18.344 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:18.405 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:18.430 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:18.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:18.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:18.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1358590957' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:18.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1358590957' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:18.621 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:18.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:18.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:18.756 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:19.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:19.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:19.240 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:19.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 54793 54777 201.341 142.188 0.00274043 0.00496443 2024-04-29T14:30:19.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:19 smithi040 ceph-mon[39581]: pgmap v1433: 17 pgs: 17 active+clean; 2.9 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-04-29T14:30:19.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:19 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:19.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1358590957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:19.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:19 smithi040 ceph-mon[39581]: osdmap e548: 12 total, 11 up, 7 in 2024-04-29T14:30:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:19 smithi194 ceph-mon[44089]: pgmap v1433: 17 pgs: 17 active+clean; 2.9 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-04-29T14:30:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:19 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1358590957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:19 smithi194 ceph-mon[44089]: osdmap e548: 12 total, 11 up, 7 in 2024-04-29T14:30:19.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:19 smithi121 ceph-mon[44533]: pgmap v1433: 17 pgs: 17 active+clean; 2.9 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-04-29T14:30:19.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:19 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:19.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1358590957' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:19.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:19 smithi121 ceph-mon[44533]: osdmap e548: 12 total, 11 up, 7 in 2024-04-29T14:30:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:19.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:20.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:20.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 57084 57068 198.11 143.188 0.00356863 0.0050462 2024-04-29T14:30:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:20.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:20.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:20.437 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:20.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:20.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:20.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:20.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:20.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:20 smithi040 ceph-mon[39581]: osdmap e549: 12 total, 11 up, 7 in 2024-04-29T14:30:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:20.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:20 smithi194 ceph-mon[44089]: osdmap e549: 12 total, 11 up, 7 in 2024-04-29T14:30:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:20.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:20.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:20.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:20 smithi121 ceph-mon[44533]: osdmap e549: 12 total, 11 up, 7 in 2024-04-29T14:30:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:21.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:21.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:30:21.129 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:21.171 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:21.199 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:21.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:21.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 59273 59257 194.884 136.812 0.00742674 0.00512654 2024-04-29T14:30:21.298 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:30:21.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:21.319 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:30:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:21.341 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:30:21.363 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:30:21.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:30:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:21.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:21.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:30:21.429 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:30:21.450 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:30:21.473 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:30:21.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:30:21.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:21.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:30:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:30:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:30:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:30:21.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:21 smithi040 ceph-mon[39581]: pgmap v1436: 17 pgs: 17 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-29T14:30:21.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3882835034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:21.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:21 smithi121 ceph-mon[44533]: pgmap v1436: 17 pgs: 17 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-29T14:30:21.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3882835034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:21.552 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:21.566 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:21 smithi194 ceph-mon[44089]: pgmap v1436: 17 pgs: 17 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-29T14:30:21.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3882835034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:21.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:21.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:21.734 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:21.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:21.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:21.867 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:21.910 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:21.936 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:21.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:22.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:22.144 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:22.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:22.177 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:22.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:22.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:30:22.259063 min lat: 0.000943876 max lat: 0.20915 avg lat: 0.00523214 2024-04-29T14:30:22.254 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:30:22.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 16 61168 61152 191.061 118.438 0.00946065 0.00523214 2024-04-29T14:30:22.287 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:22.318 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:22.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:22.423 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.464 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:22.488 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:22.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:22.651 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:22.681 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:22.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:22.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:22.817 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:30:23.121 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:30:23.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 63119 63103 187.77 121.938 0.0131354 0.00532317 2024-04-29T14:30:23.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:23 smithi040 ceph-mon[39581]: pgmap v1437: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 185 MiB/s wr, 2.95k op/s 2024-04-29T14:30:23.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:23 smithi194 ceph-mon[44089]: pgmap v1437: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 185 MiB/s wr, 2.95k op/s 2024-04-29T14:30:23.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:23 smithi121 ceph-mon[44533]: pgmap v1437: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 185 MiB/s wr, 2.95k op/s 2024-04-29T14:30:24.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 16 65172 65156 185.067 128.312 0.00777781 0.00540214 2024-04-29T14:30:24.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:24 smithi040 ceph-mon[39581]: pgmap v1438: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 158 MiB/s wr, 2.53k op/s 2024-04-29T14:30:24.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:24 smithi121 ceph-mon[44533]: pgmap v1438: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 158 MiB/s wr, 2.53k op/s 2024-04-29T14:30:24.706 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:30:24.706 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:30:24.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:24 smithi194 ceph-mon[44089]: pgmap v1438: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 158 MiB/s wr, 2.53k op/s 2024-04-29T14:30:25.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:30:25.125 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:25.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:25.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:25.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 67150 67134 182.395 123.625 0.00507678 0.00548074 2024-04-29T14:30:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:25.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:25.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:25.411 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:30:25.423 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:25.459 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:25.495 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:25.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:25.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:25.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1094088136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:30:25.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:25 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:25.685 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1094088136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:30:25.685 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:25 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:25.685 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:25.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:25.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:30:25.789 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:25.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1094088136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:30:25.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:25 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:25.821 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:25.849 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:25.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:25.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:25.987 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:26.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 16 69080 69064 179.822 120.625 0.0101409 0.0055593 2024-04-29T14:30:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:26.538 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:26.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:26.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:26.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1094088136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:30:26.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:26 smithi040 ceph-mon[39581]: osdmap e550: 12 total, 11 up, 7 in 2024-04-29T14:30:26.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:26 smithi040 ceph-mon[39581]: pgmap v1440: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-04-29T14:30:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:26.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:26.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1094088136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:30:26.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:26 smithi194 ceph-mon[44089]: osdmap e550: 12 total, 11 up, 7 in 2024-04-29T14:30:26.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:26 smithi194 ceph-mon[44089]: pgmap v1440: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-04-29T14:30:26.883 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1094088136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:30:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:26 smithi121 ceph-mon[44533]: osdmap e550: 12 total, 11 up, 7 in 2024-04-29T14:30:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:26 smithi121 ceph-mon[44533]: pgmap v1440: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-04-29T14:30:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:27.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 16 70999 70983 177.426 119.938 0.00360152 0.00563484 2024-04-29T14:30:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:28.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 72887 72871 175.14 118 0.00940875 0.0057077 2024-04-29T14:30:28.363 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:28.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:28.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:28.516 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:28 smithi121 ceph-mon[44533]: osdmap e551: 12 total, 11 up, 7 in 2024-04-29T14:30:28.518 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:28.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:30:28.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:28.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:28.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:28.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:28 smithi194 ceph-mon[44089]: osdmap e551: 12 total, 11 up, 7 in 2024-04-29T14:30:28.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:28 smithi040 ceph-mon[39581]: osdmap e551: 12 total, 11 up, 7 in 2024-04-29T14:30:28.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:28.726 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:28.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:28.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:28.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:28.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:28.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:29.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:29.084 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:29.106 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:29.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:29.213 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:29.242 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:29.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 16 74642 74626 172.716 109.688 0.0030438 0.00578756 2024-04-29T14:30:29.257 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:29 smithi121 ceph-mon[44533]: pgmap v1442: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-29T14:30:29.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:29 smithi040 ceph-mon[39581]: pgmap v1442: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-29T14:30:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:29.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:29.477 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:29.509 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:29 smithi194 ceph-mon[44089]: pgmap v1442: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-29T14:30:29.509 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:29.537 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:29.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:29.643 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:29.680 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:29.711 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:29.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:29.816 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:29.856 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:29.882 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:29.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:29.947 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:30:29.947 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:30:29.947 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:30:29.955 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:29.966 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:30:29.987 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:29.994 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:30:30.026 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:30:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:30.050 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:30:30.053 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:30.056 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:30:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:30:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:30:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:30:30.056 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:30:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:30.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:30.081 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:30:30.103 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:30:30.127 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:30:30.150 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:30:30.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:30:30.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:30.212 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:30.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:30.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:30.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 16 76455 76439 170.594 113.312 0.021238 0.00585982 2024-04-29T14:30:30.346 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:30.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2915547131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:30.378 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:30.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2915547131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:30.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:30.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2915547131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:30.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:30.709 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:31.160 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:31.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:30:31.188 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:30:31.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 78151 78135 168.367 106 0.0138118 0.0059373 2024-04-29T14:30:31.264 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:31.301 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:31.335 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:31.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:31.440 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:31.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: pgmap v1443: 17 pgs: 17 active+clean; 4.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:30:31.476 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:31.503 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:31.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: pgmap v1443: 17 pgs: 17 active+clean; 4.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:30:31.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:31.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: pgmap v1443: 17 pgs: 17 active+clean; 4.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:31.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:30:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:31.678 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:31.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:31.785 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:31.824 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:31.850 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:31.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:31.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:31.992 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:32.018 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:32.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:32.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:30:32.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:32.193 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:32.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 16 79879 79863 166.355 108 0.00314996 0.0060091 2024-04-29T14:30:32.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:30:32.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:30:32.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:30:32.444 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:32.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:32.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:32.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:30:32.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:30:32.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:30:32.578 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:32.608 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:32.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:30:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:30:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:30:32.714 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:32.749 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:32.773 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:32.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:32.877 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:32.910 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:32.939 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:32.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:33.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:33.071 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:33.087 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:33.112 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:33.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:33.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:30:33.217 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:30:33.247 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:33.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 16 81573 81557 164.404 105.875 0.00864313 0.00608093 2024-04-29T14:30:33.278 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:30:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:33.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:30:33.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:33 smithi040 ceph-mon[39581]: pgmap v1444: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-04-29T14:30:33.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:33.432 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:33.462 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:33.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:33.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:33 smithi194 ceph-mon[44089]: pgmap v1444: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-04-29T14:30:33.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:33.603 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:33.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:33 smithi121 ceph-mon[44533]: pgmap v1444: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 135 MiB/s wr, 2.15k op/s 2024-04-29T14:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:33.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:33.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:33.733 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:33.765 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:33.792 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:33.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:33.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:33.936 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:33.962 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:33.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:34.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:34.122 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:34.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:34.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:34.255 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 83233 83217 162.508 103.75 0.00300689 0.00615198 2024-04-29T14:30:34.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:34.294 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:34.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:34.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:34.573 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:34.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:34.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:34.707 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:34.738 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:34.765 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:34.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:34.869 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:34.870 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:34.901 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:34.928 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:34.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:35.033 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:35.066 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:35.092 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:35.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:35.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:35.228 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:35.255 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:35.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 84891 84875 160.724 103.625 0.00780445 0.00621981 2024-04-29T14:30:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:35.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:35.289 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:30:35.316 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:35 smithi194 ceph-mon[44089]: pgmap v1445: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-04-29T14:30:35.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3643062845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:35.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:35.404 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:35.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:35 smithi040 ceph-mon[39581]: pgmap v1445: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-04-29T14:30:35.418 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3643062845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:35.434 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:35.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:35.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:35.574 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:35.605 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:35.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:35.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:35 smithi121 ceph-mon[44533]: pgmap v1445: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-04-29T14:30:35.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3643062845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:35.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:35.756 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:35.788 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:35.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:35.866 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:30:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:30:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:30:35.867 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:30:35.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:35.927 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:35.953 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:35.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:36.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:36.256 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 16 86538 86522 159.024 102.938 0.00320268 0.0062865 2024-04-29T14:30:36.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:36 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:30:36.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:36 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:30:36.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3643062845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:36.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:36 smithi194 ceph-mon[44089]: osdmap e552: 12 total, 11 up, 7 in 2024-04-29T14:30:36.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:36 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:30:36.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:36 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:30:36.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3643062845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:36.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:36 smithi040 ceph-mon[39581]: osdmap e552: 12 total, 11 up, 7 in 2024-04-29T14:30:36.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:36 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:30:36.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:36 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:30:36.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3643062845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:36.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:36 smithi121 ceph-mon[44533]: osdmap e552: 12 total, 11 up, 7 in 2024-04-29T14:30:36.642 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:36.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:36.747 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:36.782 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:36.807 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:36.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:36.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:36.979 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:37.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:37.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:37.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:37.142 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:37.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 88167 88151 157.385 101.812 0.00990665 0.00635276 2024-04-29T14:30:37.265 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:37.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:37.294 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:37 smithi121 ceph-mon[44533]: pgmap v1447: 17 pgs: 17 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:30:37.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:37.409 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:37.430 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:37 smithi040 ceph-mon[39581]: pgmap v1447: 17 pgs: 17 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:30:37.430 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:37 smithi040 ceph-mon[39581]: osdmap e553: 12 total, 11 up, 7 in 2024-04-29T14:30:37.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:37.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:37.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:37 smithi194 ceph-mon[44089]: pgmap v1447: 17 pgs: 17 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-29T14:30:37.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:37 smithi194 ceph-mon[44089]: osdmap e553: 12 total, 11 up, 7 in 2024-04-29T14:30:37.586 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:37.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:37 smithi121 ceph-mon[44533]: osdmap e553: 12 total, 11 up, 7 in 2024-04-29T14:30:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:37.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:38.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:38.096 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:38.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:38.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:38.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:38.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 89772 89756 155.8 100.312 0.0170054 0.00641645 2024-04-29T14:30:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:38.512 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:30:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:38.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:38.535 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:30:38.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:30:38.578 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:30:38.608 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:30:38.623 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:38.635 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:30:38.654 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:38.664 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:30:38.675 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:30:38.688 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:30:38.715 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:30:38.757 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:30:38.779 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:30:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:30:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:30:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:30:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:30:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:30:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:30:38.780 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:30:38.780 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:30:38.780 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:30:38.780 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:30:38.780 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:30:38.780 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:30:38.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:30:39.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 91407 91391 154.351 102.188 0.0101678 0.00647686 2024-04-29T14:30:39.365 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: pgmap v1449: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1704184332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: pgmap v1449: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1704184332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: pgmap v1449: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-04-29T14:30:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1704184332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:30:39.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:30:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:39.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:39.830 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:39.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:39.937 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:40.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:40.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:40.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:40.235 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:40.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 92995 92979 152.901 99.25 0.00835825 0.00653894 2024-04-29T14:30:40.264 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:40.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:40.353 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:40 smithi040 ceph-mon[39581]: pgmap v1450: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:30:40.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:40.402 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:40.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:40 smithi194 ceph-mon[44089]: pgmap v1450: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:30:40.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:40 smithi121 ceph-mon[44533]: pgmap v1450: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:30:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:40.954 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:40.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:41.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:41.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:41.144 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:41.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:41.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:41.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 94558 94542 151.485 97.6875 0.0115869 0.00659982 2024-04-29T14:30:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:41.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:41.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:41.480 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:41.508 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:41.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:41.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:41.659 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:41.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:41.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:41.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:30:41.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:41.830 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:41.957 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:41.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:42.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:42.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:42.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:42.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:30:42.261475 min lat: 0.000943876 max lat: 0.20915 avg lat: 0.00666293 2024-04-29T14:30:42.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:30:42.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 96056 96040 150.038 93.625 0.00366274 0.00666293 2024-04-29T14:30:42.264 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:42.301 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:42.331 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:42.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:42.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:42.462 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:42.481 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:42.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:42.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:42.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:42.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:42.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:42.898 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:43.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 97351 97335 148.353 80.9375 0.00716144 0.00673889 2024-04-29T14:30:43.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:43 smithi040 ceph-mon[39581]: pgmap v1451: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-29T14:30:43.424 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:30:43.424 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:30:43.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:43 smithi194 ceph-mon[44089]: pgmap v1451: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-29T14:30:43.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:43 smithi121 ceph-mon[44533]: pgmap v1451: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-29T14:30:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:43.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:43.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:43.919 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:43.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:44.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:44.121 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:44.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:44.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 16 98860 98844 147.066 94.3125 0.0165261 0.00679788 2024-04-29T14:30:44.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:44.265 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:30:44.301 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:44.325 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:44.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:30:44.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:44.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:44.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:44.503 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:44.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2790634334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:30:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:44.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:44.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2790634334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:30:44.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2790634334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:30:44.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:44.682 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:44.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:44.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:44.857 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:45.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 16 100336 100320 145.791 92.25 0.00320289 0.00685743 2024-04-29T14:30:45.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:45 smithi040 ceph-mon[39581]: pgmap v1452: 17 pgs: 17 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-29T14:30:45.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:45 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:45.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2790634334' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:30:45.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:45 smithi040 ceph-mon[39581]: osdmap e554: 12 total, 11 up, 7 in 2024-04-29T14:30:45.407 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:45.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:45.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:45.511 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:45.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:45 smithi194 ceph-mon[44089]: pgmap v1452: 17 pgs: 17 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-29T14:30:45.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:45 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:45.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2790634334' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:30:45.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:45 smithi194 ceph-mon[44089]: osdmap e554: 12 total, 11 up, 7 in 2024-04-29T14:30:45.547 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:45.579 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:45.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:45.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:45 smithi121 ceph-mon[44533]: pgmap v1452: 17 pgs: 17 active+clean; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-29T14:30:45.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:45 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:45.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2790634334' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:30:45.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:45 smithi121 ceph-mon[44533]: osdmap e554: 12 total, 11 up, 7 in 2024-04-29T14:30:45.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:45.715 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:45.966 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:45.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:46.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:46.100 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:46.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 16 101794 101778 144.549 91.125 0.0175543 0.00691545 2024-04-29T14:30:46.533 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:46.550 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:46.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:46.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:46.701 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:46.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:46.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:46.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:46.882 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:46.913 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:46.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:47.020 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:47.053 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:47.076 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:47.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:47.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:30:47.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:47.257 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 103246 103230 143.353 90.75 0.00654495 0.00697395 2024-04-29T14:30:47.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:47 smithi194 ceph-mon[44089]: pgmap v1454: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-29T14:30:47.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:47 smithi194 ceph-mon[44089]: osdmap e555: 12 total, 11 up, 7 in 2024-04-29T14:30:47.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:47 smithi040 ceph-mon[39581]: pgmap v1454: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-29T14:30:47.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:47 smithi040 ceph-mon[39581]: osdmap e555: 12 total, 11 up, 7 in 2024-04-29T14:30:47.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:47 smithi121 ceph-mon[44533]: pgmap v1454: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-29T14:30:47.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:47 smithi121 ceph-mon[44533]: osdmap e555: 12 total, 11 up, 7 in 2024-04-29T14:30:48.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 104711 104695 142.227 91.5625 0.020994 0.00702948 2024-04-29T14:30:48.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:30:48.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:30:48.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:30:48.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:30:48.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:30:48.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:30:48.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:48 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:30:48.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:30:48.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:30:48.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:30:48.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:30:48.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:30:48.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:30:48.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:48 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:30:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:30:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:30:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:30:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:30:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:30:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:30:48.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:48 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:30:48.786 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:48.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:48.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:48.944 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:48.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:48.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:49.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:49.132 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:49.232 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:49.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:49.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 106176 106160 141.149 91.5625 0.00403834 0.00708345 2024-04-29T14:30:49.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:49.416 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:49.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:49.514 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:30:49.534 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:30:49.555 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:30:49.564 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:49.581 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:30:49.601 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:49.609 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:30:49.635 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:30:49.659 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:30:49.675 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:49.678 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:49.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:49.684 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:30:49.709 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:30:49.719 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:30:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:30:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:30:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:30:49.720 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:30:49.732 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:30:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:30:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:30:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:30:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:30:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:30:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:30:49.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:49.809 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/781618287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:49.813 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:49.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/781618287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:49.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/781618287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:50.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:50.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:50.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:30:50.177 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:30:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:50.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:30:50.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 16 107591 107575 140.051 88.4375 0.0144108 0.00713887 2024-04-29T14:30:50.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:50.356 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:50.381 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:50.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:50.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:50.529 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:50.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:50.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:50.650 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:50 smithi040 ceph-mon[39581]: pgmap v1457: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:30:50.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:50.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:50 smithi194 ceph-mon[44089]: pgmap v1457: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:50.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:50 smithi121 ceph-mon[44533]: pgmap v1457: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:30:51.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:30:51.148 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:51.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:51.252 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:51.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 108976 108960 138.959 86.5625 0.0138532 0.00719524 2024-04-29T14:30:51.283 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:51.307 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:51.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:51.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:30:51.442 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:51.463 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:30:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:51.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:30:51.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:30:51.578 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:30:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:51.607 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-04-29T14:30:51.607 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-04-29T14:30:51.608 DEBUG:teuthology.orchestra.run.smithi121:osd.5> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5.service 2024-04-29T14:30:51.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl start ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5 2024-04-29T14:30:51.670 INFO:journalctl@ceph.osd.5.smithi121.stdout:-- Logs begin at Mon 2024-04-29 13:48:48 UTC. -- 2024-04-29T14:30:51.981 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:51 smithi121 systemd[1]: Starting Ceph osd.5 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:30:52.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 110194 110178 137.703 76.125 0.0143688 0.00726083 2024-04-29T14:30:52.544 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:30:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:52.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:30:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:30:52.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:30:52.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:30:52.684 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:52.784 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:30:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:52.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:30:52.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:52.946 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:53.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:53 smithi121 ceph-mon[44533]: pgmap v1458: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-29T14:30:53.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:53.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:53.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 111503 111487 136.607 81.8125 0.0265777 0.00731868 2024-04-29T14:30:53.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:53.333 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:53.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:53 smithi040 ceph-mon[39581]: pgmap v1458: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-29T14:30:53.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:53 smithi194 ceph-mon[44089]: pgmap v1458: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-29T14:30:54.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:54.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:30:54.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:54.164 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:54.188 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:54.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:54.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 112723 112707 135.446 76.25 0.016464 0.00738119 2024-04-29T14:30:54.293 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:54.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2775551891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:54.325 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:54.325 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:54.326 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:54.349 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2775551891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:54.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:54.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:54.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:54.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2775551891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:30:54.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:54.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:54.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:54.519 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:54.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:30:54.628 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-29T14:30:54.629 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-29T14:30:54.629 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-29T14:30:54.629 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-29T14:30:54.629 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-29T14:30:54.629 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-29T14:30:54.629 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-29T14:30:54.629 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:54 smithi121 bash[136668]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-29T14:30:54.948 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:30:54.948 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:30:54.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.948 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:113397 6.6 6:6d6a4823:::benchmark_data_smithi194_134202_object113396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-29T14:30:54.948 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.815800+0000", 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0024130890000000002, 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024557440000000002, 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113397 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.815800+0000", 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.815800+0000", 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.815802+0000", 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4830000000000002e-06 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.815838+0000", 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6075000000000003e-05 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.815853+0000", 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4868000000000001e-05 2024-04-29T14:30:54.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.815863+0000", 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6209999999999999e-06 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.815929+0000", 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6848999999999996e-05 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.816086+0000", 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015693900000000001 2024-04-29T14:30:54.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.816242+0000", 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155439 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16809.0:113364 6.e e556/534)", 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.817581+0000", 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00063170900000000005, 2024-04-29T14:30:54.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000752671, 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817581+0000", 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817581+0000", 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817583+0000", 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6029999999999999e-06 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817588+0000", 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3929999999999999e-06 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817602+0000", 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4095e-05 2024-04-29T14:30:54.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817609+0000", 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4680000000000003e-06 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16809.0:113369 6.6 e556/534)", 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.817620+0000", 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00059238699999999999, 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075951999999999997, 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817620+0000", 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817620+0000", 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817621+0000", 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0699999999999997e-07 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817624+0000", 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8830000000000002e-06 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817630+0000", 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9240000000000004e-06 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817635+0000", 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8559999999999998e-06 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16809.0:113372 6.6 e556/534)", 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.817641+0000", 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.000571282, 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078200299999999997, 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817641+0000", 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817641+0000", 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817642+0000", 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6400000000000003e-07 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817655+0000", 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3008e-05 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817660+0000", 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4190000000000001e-06 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817665+0000", 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7439999999999998e-06 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.961 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16809.0:113378 6.e e556/534)", 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.817671+0000", 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00054178099999999997, 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079566800000000003, 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817671+0000", 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817671+0000", 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817671+0000", 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.63e-07 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817673+0000", 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0470000000000001e-06 2024-04-29T14:30:54.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817678+0000", 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.817682+0000", 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4449999999999998e-06 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:113364 6.e 6:7ce67093:::benchmark_data_smithi194_134202_object113363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-29T14:30:54.964 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.792425+0000", 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.025787253999999999, 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026095912999999998, 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113364 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792425+0000", 2024-04-29T14:30:54.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792425+0000", 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792427+0000", 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.739e-06 2024-04-29T14:30:54.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792481+0000", 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4141999999999998e-05 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792491+0000", 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0138e-05 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792498+0000", 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1299999999999998e-06 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792549+0000", 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1402e-05 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792638+0000", 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8692999999999999e-05 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.792716+0000", 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8690999999999996e-05 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.802093+0000", 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093765589999999996 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:113369 6.6 6:6b9bd95b:::benchmark_data_smithi194_134202_object113368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.796277+0000", 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.021935972000000001, 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022325362000000001, 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.969 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113369 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.796277+0000", 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.796277+0000", 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.796278+0000", 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9999999999999995e-07 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.796370+0000", 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2725000000000004e-05 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.796380+0000", 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6299999999999993e-06 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.796387+0000", 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6429999999999998e-06 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798219+0000", 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018324599999999999 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:30:54.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798336+0000", 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117168 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798481+0000", 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000145031 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.811732+0000", 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013250583 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:113372 6.6 6:66391b31:::benchmark_data_smithi194_134202_object113371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.798339+0000", 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.019873503000000001, 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020344580000000001, 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113372 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.974 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798339+0000", 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798339+0000", 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798341+0000", 2024-04-29T14:30:54.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.409e-06 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798395+0000", 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4440000000000001e-05 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798405+0000", 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.01e-05 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.798412+0000", 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1520000000000003e-06 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.802258+0000", 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003845425 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.802378+0000", 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120327 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:30:54.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.802506+0000", 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128035 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.811737+0000", 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092306249999999992 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:113378 6.e 6:7d6b2713:::benchmark_data_smithi194_134202_object113377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.803842+0000", 2024-04-29T14:30:54.978 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.014370717, 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014924735, 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113378 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.803842+0000", 2024-04-29T14:30:54.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.803842+0000", 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.803844+0000", 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.762e-06 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.803905+0000", 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1033999999999999e-05 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.803915+0000", 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0682999999999999e-05 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.803923+0000", 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6979999999999998e-06 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.806066+0000", 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002142676 2024-04-29T14:30:54.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.806184+0000", 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118527 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.806311+0000", 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012680999999999999 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.812824+0000", 2024-04-29T14:30:54.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065132649999999999 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:113385 6.6 6:630290c7:::benchmark_data_smithi194_134202_object113384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:30:54.807917+0000", 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.010295963999999999, 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010935823000000001, 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:30:54.983 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113385 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.807917+0000", 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.807917+0000", 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:30:54.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.807918+0000", 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8100000000000001e-07 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.807970+0000", 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2369e-05 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.807978+0000", 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7859999999999997e-06 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:30:54.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.807984+0000", 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2910000000000004e-06 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.808199+0000", 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021524900000000001 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.808337+0000", 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013762799999999999 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:30:54.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.808475+0000", 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013770899999999999 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:30:54.812831+0000", 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043561490000000001 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 10 2024-04-29T14:30:54.987 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:30:55.000 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:55.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:55 smithi121 ceph-mon[44533]: pgmap v1459: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:30:55.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:55 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:55.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2775551891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:55.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:55 smithi121 ceph-mon[44533]: osdmap e556: 12 total, 11 up, 7 in 2024-04-29T14:30:55.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 16 113984 113968 134.377 78.8125 0.00876346 0.00743987 2024-04-29T14:30:55.272 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:55 smithi040 ceph-mon[39581]: pgmap v1459: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:30:55.272 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:55 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:55.272 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2775551891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:55.272 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:55 smithi040 ceph-mon[39581]: osdmap e556: 12 total, 11 up, 7 in 2024-04-29T14:30:55.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:55 smithi194 ceph-mon[44089]: pgmap v1459: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:30:55.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:55 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:30:55.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2775551891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:30:55.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:55 smithi194 ceph-mon[44089]: osdmap e556: 12 total, 11 up, 7 in 2024-04-29T14:30:55.585 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:30:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:55.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:30:55.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:30:55.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:30:55.778 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:56.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:30:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:56.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:30:56.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:30:56.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 115194 115178 133.289 75.625 0.0262947 0.00749975 2024-04-29T14:30:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:56.284 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:30:56.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:30:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:56.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:30:56.501 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:56.532 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:56.559 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:56.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:56.664 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:56.699 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:56.721 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:56.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:56.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:30:56.855 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:56.885 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:30:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:56.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:30:56.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:30:56.992 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:30:57.009 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:30:57.027 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:57.035 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:30:57.053 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:30:57.059 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:30:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:57.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:30:57.080 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:30:57.107 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:30:57.148 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:30:57.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:30:57.177 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:30:57.198 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:30:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:57.226 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:30:57.250 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:30:57.258 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 116393 116377 132.228 74.9375 0.00408087 0.00755976 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:30:57.276 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:30:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:30:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:30:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:30:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:30:57.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:57 smithi040 ceph-mon[39581]: pgmap v1461: 17 pgs: 17 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:30:57.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4035635460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:57.303 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:30:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:57 smithi194 ceph-mon[44089]: pgmap v1461: 17 pgs: 17 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:30:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4035635460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:57.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:57 smithi121 ceph-mon[44533]: pgmap v1461: 17 pgs: 17 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:30:57.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4035635460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:30:57.880 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:30:57 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T14:30:57.457+0000 7f951b2857c0 -1 Falling back to public interface 2024-04-29T14:30:58.262 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 117672 117656 131.286 79.9375 0.0130373 0.00761521 2024-04-29T14:30:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:30:58.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:30:58.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osdmap e557: 12 total, 11 up, 7 in 2024-04-29T14:30:58.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:30:58.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:30:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osdmap e557: 12 total, 11 up, 7 in 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:30:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osdmap e557: 12 total, 11 up, 7 in 2024-04-29T14:30:58.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:30:58.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:30:58.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:30:59.262 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 16 118883 118867 130.311 75.6875 0.0143443 0.00767231 2024-04-29T14:30:59.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:30:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:30:59.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:30:59.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:30:59.562 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:30:59.707 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:59 smithi040 ceph-mon[39581]: pgmap v1463: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-04-29T14:30:59.707 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:30:59 smithi040 ceph-mon[39581]: osdmap e558: 12 total, 11 up, 7 in 2024-04-29T14:30:59.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:59 smithi194 ceph-mon[44089]: pgmap v1463: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-04-29T14:30:59.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:30:59 smithi194 ceph-mon[44089]: osdmap e558: 12 total, 11 up, 7 in 2024-04-29T14:30:59.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:59 smithi121 ceph-mon[44533]: pgmap v1463: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-04-29T14:30:59.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:30:59 smithi121 ceph-mon[44533]: osdmap e558: 12 total, 11 up, 7 in 2024-04-29T14:31:00.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:31:00.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:00.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:00.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:00.262 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 120062 120046 129.335 73.6875 0.00287097 0.00773049 2024-04-29T14:31:00.319 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:00.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:31:00.382 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:31:00 smithi121 bash[137185]: a74ec602820fc38befb102058a3a458c8d0450e05eb8f23dd06ff1ecb14fd756 2024-04-29T14:31:00.382 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:31:00 smithi121 systemd[1]: Started Ceph osd.5 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:31:00.504 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:31:00.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:00 smithi194 ceph-mon[44089]: pgmap v1465: 17 pgs: 17 active+clean; 7.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-29T14:31:00.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2436313785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:31:00.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:00 smithi040 ceph-mon[39581]: pgmap v1465: 17 pgs: 17 active+clean; 7.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-29T14:31:00.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2436313785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:31:00.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:00 smithi121 ceph-mon[44533]: pgmap v1465: 17 pgs: 17 active+clean; 7.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-29T14:31:00.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2436313785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:31:01.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:01.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:01.262 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 121051 121035 128.19 61.8125 0.00374071 0.00779907 2024-04-29T14:31:01.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:01.373 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:01.380 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:01.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:01.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:01.496 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:01 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:01.514 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:01.538 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:01.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:01.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:01 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:01.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2436313785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:31:01.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:01 smithi040 ceph-mon[39581]: osdmap e559: 12 total, 11 up, 7 in 2024-04-29T14:31:01.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:01 smithi040 ceph-mon[39581]: 6.7 scrub starts 2024-04-29T14:31:01.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:01.672 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:01.693 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:01.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:01.797 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:01.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:01 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:01.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2436313785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:31:01.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:01 smithi194 ceph-mon[44089]: osdmap e559: 12 total, 11 up, 7 in 2024-04-29T14:31:01.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:01 smithi194 ceph-mon[44089]: 6.7 scrub starts 2024-04-29T14:31:01.836 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:01.862 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:01.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:01.880 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:31:01 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T14:31:01.633+0000 7f951b2857c0 -1 osd.5 457 log_to_monitors true 2024-04-29T14:31:01.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2436313785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:31:01.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:01 smithi121 ceph-mon[44533]: osdmap e559: 12 total, 11 up, 7 in 2024-04-29T14:31:01.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:01 smithi121 ceph-mon[44533]: 6.7 scrub starts 2024-04-29T14:31:01.966 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:01.998 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:02.021 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:02.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:02.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:02.154 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:02.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:02.262 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:31:02.266465 min lat: 0.000943876 max lat: 0.222113 avg lat: 0.00786908 2024-04-29T14:31:02.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:31:02.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 122001 121985 127.043 59.375 0.018019 0.00786908 2024-04-29T14:31:02.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:02.363 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:02.387 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:02.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:02.491 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:02.518 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:02.541 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:02.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:02.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:02 smithi194 ceph-mon[44089]: 6.c scrub starts 2024-04-29T14:31:02.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:02 smithi194 ceph-mon[44089]: from='osd.5 v1:172.21.15.121:6804/2522198013' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:31:02.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:02 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:31:02.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:02 smithi194 ceph-mon[44089]: pgmap v1467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:31:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:02.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:02 smithi040 ceph-mon[39581]: 6.c scrub starts 2024-04-29T14:31:02.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:02 smithi040 ceph-mon[39581]: from='osd.5 v1:172.21.15.121:6804/2522198013' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:31:02.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:02 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:31:02.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:02 smithi040 ceph-mon[39581]: pgmap v1467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:31:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:02.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:02.878 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 14:31:02 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T14:31:02.617+0000 7f950c6dd700 -1 osd.5 560 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:31:02.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:02 smithi121 ceph-mon[44533]: 6.c scrub starts 2024-04-29T14:31:02.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:02 smithi121 ceph-mon[44533]: from='osd.5 v1:172.21.15.121:6804/2522198013' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:31:02.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:02 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-29T14:31:02.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:02 smithi121 ceph-mon[44533]: pgmap v1467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-04-29T14:31:02.925 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:02.952 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:02.975 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:02.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:03.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:03.182 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:03.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:03.262 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 122838 122822 125.818 52.3125 0.0255853 0.00794627 2024-04-29T14:31:03.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:03.303 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:31:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:03.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:03.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:03.472 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:03.481 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:03.494 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:03.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:03.504 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:03.527 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:03.548 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:03.577 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:03.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:03.604 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:03.619 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:03.626 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:03.645 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:03.658 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:03.678 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:03.700 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:03.721 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:03.742 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:03.743 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:03.743 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:03.743 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:03.743 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:03.744 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: 6.5 scrub starts 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: osdmap e560: 12 total, 11 up, 7 in 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: from='osd.5 v1:172.21.15.121:6804/2522198013' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: 6.e scrub starts 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: 6.7 scrub ok 2024-04-29T14:31:03.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2650373291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: 6.5 scrub starts 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: osdmap e560: 12 total, 11 up, 7 in 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: from='osd.5 v1:172.21.15.121:6804/2522198013' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: 6.e scrub starts 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: 6.7 scrub ok 2024-04-29T14:31:03.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2650373291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:03.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: 6.5 scrub starts 2024-04-29T14:31:03.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-29T14:31:03.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: osdmap e560: 12 total, 11 up, 7 in 2024-04-29T14:31:03.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: from='osd.5 v1:172.21.15.121:6804/2522198013' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:31:03.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-29T14:31:03.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:31:03.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: 6.e scrub starts 2024-04-29T14:31:03.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: 6.7 scrub ok 2024-04-29T14:31:03.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2650373291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:04.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 16 123735 123719 124.693 56.0625 0.0206607 0.00801787 2024-04-29T14:31:04.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:31:04.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: 6.c scrub ok 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd.5 v1:172.21.15.121:6804/2522198013 boot 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osdmap e561: 12 total, 12 up, 7 in 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:04 smithi194 ceph-mon[44089]: pgmap v1470: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:31:04.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:31:04.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: 6.c scrub ok 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd.5 v1:172.21.15.121:6804/2522198013 boot 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osdmap e561: 12 total, 12 up, 7 in 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:04 smithi040 ceph-mon[39581]: pgmap v1470: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:31:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:31:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: 6.c scrub ok 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd.5 v1:172.21.15.121:6804/2522198013 boot 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osdmap e561: 12 total, 12 up, 7 in 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:04.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:04 smithi121 ceph-mon[44533]: pgmap v1470: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:31:05.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 124759 124743 123.73 64 0.00372835 0.00807971 2024-04-29T14:31:05.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:05.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:05.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:05.433 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:05.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:05 smithi121 ceph-mon[44533]: 6.5 scrub ok 2024-04-29T14:31:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:05 smithi121 ceph-mon[44533]: osdmap e562: 12 total, 12 up, 7 in 2024-04-29T14:31:05.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:05.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:05.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:05.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:05.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:05 smithi040 ceph-mon[39581]: 6.5 scrub ok 2024-04-29T14:31:05.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:05 smithi040 ceph-mon[39581]: osdmap e562: 12 total, 12 up, 7 in 2024-04-29T14:31:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:05.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:05 smithi194 ceph-mon[44089]: 6.5 scrub ok 2024-04-29T14:31:05.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:05 smithi194 ceph-mon[44089]: osdmap e562: 12 total, 12 up, 7 in 2024-04-29T14:31:06.177 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:06.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:06.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 16 125743 125727 122.758 61.5 0.00574028 0.00814441 2024-04-29T14:31:06.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:06.311 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:06.333 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:06.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:06.437 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:06.466 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:06.487 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:06.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:06.529 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:31:06.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:06.628 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:06.648 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:06.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:06.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:06.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:06 smithi194 ceph-mon[44089]: 6.3 scrub starts 2024-04-29T14:31:06.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:06 smithi194 ceph-mon[44089]: 6.e scrub ok 2024-04-29T14:31:06.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:06 smithi194 ceph-mon[44089]: pgmap v1472: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:31:06.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1619677455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:06.818 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:06 smithi121 ceph-mon[44533]: 6.3 scrub starts 2024-04-29T14:31:06.818 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:06 smithi121 ceph-mon[44533]: 6.e scrub ok 2024-04-29T14:31:06.818 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:06 smithi121 ceph-mon[44533]: pgmap v1472: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:31:06.818 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1619677455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:06.822 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:06.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:06 smithi040 ceph-mon[39581]: 6.3 scrub starts 2024-04-29T14:31:06.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:06 smithi040 ceph-mon[39581]: 6.e scrub ok 2024-04-29T14:31:06.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:06 smithi040 ceph-mon[39581]: pgmap v1472: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-29T14:31:06.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1619677455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:07.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 126716 126700 121.805 60.8125 0.022794 0.0082081 2024-04-29T14:31:07.695 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: 6.d scrub starts 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1619677455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: osdmap e563: 12 total, 12 up, 7 in 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:07 smithi040 ceph-mon[39581]: 6.2 scrub starts 2024-04-29T14:31:07.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: 6.d scrub starts 2024-04-29T14:31:07.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:31:07.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:31:07.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1619677455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:07.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: osdmap e563: 12 total, 12 up, 7 in 2024-04-29T14:31:07.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:07.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:07.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:07 smithi194 ceph-mon[44089]: 6.2 scrub starts 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: 6.d scrub starts 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1619677455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: osdmap e563: 12 total, 12 up, 7 in 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:07.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:07 smithi121 ceph-mon[44533]: 6.2 scrub starts 2024-04-29T14:31:07.993 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:08.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:08.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:08.121 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:31:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:31:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:31:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:31:08.122 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:31:08.133 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:08.261 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:08.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 16 127764 127748 120.952 65.5 0.00531238 0.00826581 2024-04-29T14:31:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:08.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:08.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:08.398 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:08 smithi194 ceph-mon[44089]: 6.d scrub ok 2024-04-29T14:31:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:08 smithi194 ceph-mon[44089]: osdmap e564: 12 total, 12 up, 7 in 2024-04-29T14:31:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:08 smithi194 ceph-mon[44089]: 6.3 scrub ok 2024-04-29T14:31:08.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:08 smithi194 ceph-mon[44089]: pgmap v1475: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-29T14:31:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:08.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:08 smithi040 ceph-mon[39581]: 6.d scrub ok 2024-04-29T14:31:08.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:08 smithi040 ceph-mon[39581]: osdmap e564: 12 total, 12 up, 7 in 2024-04-29T14:31:08.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:08 smithi040 ceph-mon[39581]: 6.3 scrub ok 2024-04-29T14:31:08.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:08 smithi040 ceph-mon[39581]: pgmap v1475: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-29T14:31:08.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:08.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:08.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:08 smithi121 ceph-mon[44533]: 6.d scrub ok 2024-04-29T14:31:08.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:08 smithi121 ceph-mon[44533]: osdmap e564: 12 total, 12 up, 7 in 2024-04-29T14:31:08.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:08 smithi121 ceph-mon[44533]: 6.3 scrub ok 2024-04-29T14:31:08.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:08 smithi121 ceph-mon[44533]: pgmap v1475: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-29T14:31:08.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:08.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:09.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 16 128780 128764 120.094 63.5 0.00520608 0.00832534 2024-04-29T14:31:09.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:31:09.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:31:09.571 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:09 smithi121 ceph-mon[44533]: 6.2 scrub ok 2024-04-29T14:31:09.571 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:09 smithi121 ceph-mon[44533]: 6.f scrub starts 2024-04-29T14:31:09.572 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:09 smithi121 ceph-mon[44533]: osdmap e565: 12 total, 12 up, 7 in 2024-04-29T14:31:09.652 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:09.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:09.757 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:09.798 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:09.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:09 smithi194 ceph-mon[44089]: 6.2 scrub ok 2024-04-29T14:31:09.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:09 smithi194 ceph-mon[44089]: 6.f scrub starts 2024-04-29T14:31:09.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:09 smithi194 ceph-mon[44089]: osdmap e565: 12 total, 12 up, 7 in 2024-04-29T14:31:09.824 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:09.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:09.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:09 smithi040 ceph-mon[39581]: 6.2 scrub ok 2024-04-29T14:31:09.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:09 smithi040 ceph-mon[39581]: 6.f scrub starts 2024-04-29T14:31:09.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:09 smithi040 ceph-mon[39581]: osdmap e565: 12 total, 12 up, 7 in 2024-04-29T14:31:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:09.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:10.009 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:10.023 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:31:10.023 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:31:10.023 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:31:10.024 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:31:10.042 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:10.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:10.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:10.180 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:10.212 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:10.231 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:10.251 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:10.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 129800 129784 119.266 63.75 0.0142843 0.00838295 2024-04-29T14:31:10.272 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:10.301 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:10.334 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:10.359 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:10.396 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:10.419 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:10.441 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:10.465 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:10.487 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:10.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:10.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:10.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:10.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:10.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:10.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:10.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:10.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:10.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:10.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:10.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:10.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:10.533 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:10.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:10.564 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:10 smithi121 ceph-mon[44533]: 6.f scrub ok 2024-04-29T14:31:10.564 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:10 smithi121 ceph-mon[44533]: 6.6 scrub starts 2024-04-29T14:31:10.564 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:10 smithi121 ceph-mon[44533]: pgmap v1477: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-04-29T14:31:10.564 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3122559996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:10.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:10.699 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:10 smithi194 ceph-mon[44089]: 6.f scrub ok 2024-04-29T14:31:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:10 smithi194 ceph-mon[44089]: 6.6 scrub starts 2024-04-29T14:31:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:10 smithi194 ceph-mon[44089]: pgmap v1477: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-04-29T14:31:10.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3122559996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:10.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:10 smithi040 ceph-mon[39581]: 6.f scrub ok 2024-04-29T14:31:10.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:10 smithi040 ceph-mon[39581]: 6.6 scrub starts 2024-04-29T14:31:10.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:10 smithi040 ceph-mon[39581]: pgmap v1477: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-04-29T14:31:10.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3122559996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:11.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 130799 130783 118.442 62.4375 0.0154401 0.00844074 2024-04-29T14:31:11.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: 6.4 scrub starts 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:11 smithi040 ceph-mon[39581]: osdmap e566: 12 total, 12 up, 7 in 2024-04-29T14:31:11.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: 6.4 scrub starts 2024-04-29T14:31:11.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:11 smithi194 ceph-mon[44089]: osdmap e566: 12 total, 12 up, 7 in 2024-04-29T14:31:11.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: 6.4 scrub starts 2024-04-29T14:31:11.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:11 smithi121 ceph-mon[44533]: osdmap e566: 12 total, 12 up, 7 in 2024-04-29T14:31:12.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 131657 131641 117.516 53.625 0.00395342 0.00850456 2024-04-29T14:31:12.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:12 smithi040 ceph-mon[39581]: 6.8 scrub starts 2024-04-29T14:31:12.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:12 smithi040 ceph-mon[39581]: 6.4 scrub ok 2024-04-29T14:31:12.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:12 smithi040 ceph-mon[39581]: pgmap v1479: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-29T14:31:12.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:12 smithi040 ceph-mon[39581]: osdmap e567: 12 total, 12 up, 7 in 2024-04-29T14:31:12.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:12 smithi121 ceph-mon[44533]: 6.8 scrub starts 2024-04-29T14:31:12.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:12 smithi121 ceph-mon[44533]: 6.4 scrub ok 2024-04-29T14:31:12.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:12 smithi121 ceph-mon[44533]: pgmap v1479: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-29T14:31:12.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:12 smithi121 ceph-mon[44533]: osdmap e567: 12 total, 12 up, 7 in 2024-04-29T14:31:13.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:12 smithi194 ceph-mon[44089]: 6.8 scrub starts 2024-04-29T14:31:13.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:12 smithi194 ceph-mon[44089]: 6.4 scrub ok 2024-04-29T14:31:13.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:12 smithi194 ceph-mon[44089]: pgmap v1479: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-29T14:31:13.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:12 smithi194 ceph-mon[44089]: osdmap e567: 12 total, 12 up, 7 in 2024-04-29T14:31:13.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 16 132671 132655 116.754 63.375 0.0126343 0.00856287 2024-04-29T14:31:13.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:13.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:13.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:13.586 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:31:13.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:31:13.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:13 smithi040 ceph-mon[39581]: 6.6 scrub ok 2024-04-29T14:31:13.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3285379963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:31:13.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:13 smithi121 ceph-mon[44533]: 6.6 scrub ok 2024-04-29T14:31:13.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3285379963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:31:14.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:13 smithi194 ceph-mon[44089]: 6.6 scrub ok 2024-04-29T14:31:14.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3285379963' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:31:14.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 133802 133786 116.114 70.6875 0.00544395 0.00861068 2024-04-29T14:31:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:14.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:14 smithi040 ceph-mon[39581]: 6.8 scrub ok 2024-04-29T14:31:14.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:14 smithi040 ceph-mon[39581]: 6.0 scrub starts 2024-04-29T14:31:14.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:14 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:14.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3285379963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:31:14.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:14 smithi040 ceph-mon[39581]: osdmap e568: 12 total, 12 up, 7 in 2024-04-29T14:31:14.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:14 smithi040 ceph-mon[39581]: pgmap v1482: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:31:14.878 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:14 smithi121 ceph-mon[44533]: 6.8 scrub ok 2024-04-29T14:31:14.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:14 smithi121 ceph-mon[44533]: 6.0 scrub starts 2024-04-29T14:31:14.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:14 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:14.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3285379963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:31:14.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:14 smithi121 ceph-mon[44533]: osdmap e568: 12 total, 12 up, 7 in 2024-04-29T14:31:14.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:14 smithi121 ceph-mon[44533]: pgmap v1482: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:31:15.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:14 smithi194 ceph-mon[44089]: 6.8 scrub ok 2024-04-29T14:31:15.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:14 smithi194 ceph-mon[44089]: 6.0 scrub starts 2024-04-29T14:31:15.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:14 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:15.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3285379963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:31:15.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:14 smithi194 ceph-mon[44089]: osdmap e568: 12 total, 12 up, 7 in 2024-04-29T14:31:15.068 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:14 smithi194 ceph-mon[44089]: pgmap v1482: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:31:15.263 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 134991 134975 115.541 74.3125 0.02506 0.0086534 2024-04-29T14:31:15.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:15.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:15.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:15.476 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:15.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:15.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:15.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:15.899 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:15.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:15.923 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:31:15.923 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-29T14:31:15.923 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:31:15.923 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:31:15.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-04-29T14:31:16.003 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:16.040 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:16.064 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:16.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:16.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:16.250 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:16.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 136063 136047 114.885 67 0.00393778 0.00870199 2024-04-29T14:31:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:16.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:16.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:16.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:31:16.489 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:16.498 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:16.510 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:16.534 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:16.558 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:16.583 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:16.584 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:16.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:16.605 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:16.626 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:16.652 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:16.676 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:16.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:16.700 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:16.727 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:16.727 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:16.730 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:16.749 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:16.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:16.761 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:16.855 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:16.865 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:16.890 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:16.913 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:16.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:17.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:17.052 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:17.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 16 137153 137137 114.262 68.125 0.00405899 0.00874996 2024-04-29T14:31:17.315 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:31:17.315 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:31:17.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:147906 5.f 5:f0d11147:::benchmark_data_smithi194_121512_object14440:head [delete] snapc 0=[] ondisk+write+known_if_redirected e475)", 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:26:42.375558+0000", 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 274.94037537000003, 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.245455862, 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:17.316 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 147906 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375558+0000", 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375558+0000", 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375560+0000", 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2079999999999999e-06 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375566+0000", 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3539999999999996e-06 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375580+0000", 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3319000000000001e-05 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375588+0000", 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6999999999999997e-06 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.388800+0000", 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013211268 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.388918+0000", 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118381 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.389062+0000", 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014409399999999999 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416268+0000", 2024-04-29T14:31:17.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027205521999999999 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.618705+0000", 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202437123 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.618784+0000", 2024-04-29T14:31:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9177000000000004e-05 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.621014+0000", 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022297160000000001 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.323 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:147909 5.f 5:fc13d4e9:::benchmark_data_smithi194_121512_object14443:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e475)", 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:26:42.416417+0000", 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 274.89951660600002, 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27121119900000001, 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 147909 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416417+0000", 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416417+0000", 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416418+0000", 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8599999999999996e-07 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416421+0000", 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4529999999999998e-06 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416429+0000", 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5750000000000003e-06 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416434+0000", 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8989999999999998e-06 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.443012+0000", 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026577536999999998 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.443105+0000", 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3059000000000005e-05 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.443231+0000", 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125485 2024-04-29T14:31:17.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.449050+0000", 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058199380000000002 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.686242+0000", 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.237191072 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.686303+0000", 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1875e-05 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.687628+0000", 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013243199999999999 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:194025 5.f 5:fa12a96e:::benchmark_data_smithi194_121512_object60559:head [delete] snapc 0=[] ondisk+write+known_if_redirected e495)", 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:27:58.274041+0000", 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 199.04189215, 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.11923998700000001, 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:17.331 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 194025 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.274041+0000", 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.274041+0000", 2024-04-29T14:31:17.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.274042+0000", 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4600000000000003e-07 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.274046+0000", 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2010000000000001e-06 2024-04-29T14:31:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.274053+0000", 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9229999999999996e-06 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.274059+0000", 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1330000000000004e-06 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.286997+0000", 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012937854 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.287086+0000", 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9284999999999996e-05 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.287174+0000", 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7467999999999997e-05 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.297918+0000", 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010743819 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.391251+0000", 2024-04-29T14:31:17.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093333532999999996 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.391293+0000", 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2058000000000003e-05 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:27:58.393281+0000", 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019877670000000001 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.337 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:235037 5.f 5:f2be22cd:::benchmark_data_smithi194_121512_object101571:head [delete] snapc 0=[] ondisk+write+known_if_redirected e515)", 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:28:53.997102+0000", 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.31883145500001, 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.120824156, 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 235037 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.997102+0000", 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.997102+0000", 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.997103+0000", 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9999999999999996e-07 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.997106+0000", 2024-04-29T14:31:17.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3689999999999998e-06 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.997114+0000", 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6890000000000004e-06 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.997119+0000", 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0350000000000001e-06 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.998707+0000", 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015879889999999999 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.998811+0000", 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104219 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:53.998948+0000", 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000137288 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:54.006119+0000", 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071709900000000004 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:54.116757+0000", 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.11063748 2024-04-29T14:31:17.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:54.116829+0000", 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1945000000000003e-05 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:54.117926+0000", 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010972519999999999 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127897 6.e 6:75170b92:::benchmark_data_smithi194_134202_object127896:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.373844+0000", 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9420889399999997, 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21297782900000001, 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127897 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373844+0000", 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373844+0000", 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373847+0000", 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.339e-06 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373876+0000", 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9181999999999999e-05 2024-04-29T14:31:17.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373889+0000", 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3719e-05 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373899+0000", 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4299999999999995e-06 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373972+0000", 2024-04-29T14:31:17.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3341000000000001e-05 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374111+0000", 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013890300000000001 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374238+0000", 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012690999999999999 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.383381+0000", 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091430639999999994 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586704+0000", 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20332249799999999 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586763+0000", 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9762000000000001e-05 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586822+0000", 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8681000000000003e-05 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.351 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127898 6.7 6:e4fbbafd:::benchmark_data_smithi194_134202_object127897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9420124530000002, 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.212923687, 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127898 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8599999999999998e-07 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373955+0000", 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3615999999999998e-05 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373962+0000", 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3699999999999997e-06 2024-04-29T14:31:17.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373969+0000", 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374042+0000", 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3023999999999997e-05 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374140+0000", 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8722999999999997e-05 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374223+0000", 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2781000000000006e-05 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.383397+0000", 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091737160000000002 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586732+0000", 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20333527300000001 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586799+0000", 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6882000000000001e-05 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586844+0000", 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5367000000000001e-05 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127917 6.7 6:ecf9ff1a:::benchmark_data_smithi194_134202_object127916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.387947+0000", 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9279859209999994, 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.223737357, 2024-04-29T14:31:17.358 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127917 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387947+0000", 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387947+0000", 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387950+0000", 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7140000000000002e-06 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387997+0000", 2024-04-29T14:31:17.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6962000000000002e-05 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388009+0000", 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759999999999999e-05 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388016+0000", 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0700000000000001e-06 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388071+0000", 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5438e-05 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388233+0000", 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016152000000000001 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388391+0000", 2024-04-29T14:31:17.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015797399999999999 2024-04-29T14:31:17.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.392441+0000", 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040504249999999999 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611560+0000", 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.219118484 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611616+0000", 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6659000000000001e-05 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611685+0000", 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8350999999999997e-05 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127918 6.6 6:682fa8a0:::benchmark_data_smithi194_134202_object127917:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.388032+0000", 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.927901189, 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20950328200000001, 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127918 2024-04-29T14:31:17.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388032+0000", 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388032+0000", 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388033+0000", 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4300000000000003e-07 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388098+0000", 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5759999999999994e-05 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388108+0000", 2024-04-29T14:31:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.781e-06 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388112+0000", 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9659999999999998e-06 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388160+0000", 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7688999999999997e-05 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388291+0000", 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000130913 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388420+0000", 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129305 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.396987+0000", 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085665970000000004 2024-04-29T14:31:17.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.597426+0000", 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20043913399999999 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.597484+0000", 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8397999999999999e-05 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.597535+0000", 2024-04-29T14:31:17.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1096e-05 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127921 6.6 6:667e9e78:::benchmark_data_smithi194_134202_object127920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.390238+0000", 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9256955839999996, 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21192386599999999, 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.373 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127921 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.390238+0000", 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.390238+0000", 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.390239+0000", 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7689999999999999e-06 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.390310+0000", 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0582999999999998e-05 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.390321+0000", 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1270000000000001e-05 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.390329+0000", 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.43e-06 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.392812+0000", 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024836120000000001 2024-04-29T14:31:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.392964+0000", 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015185700000000001 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.393125+0000", 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000160734 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.397183+0000", 2024-04-29T14:31:17.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004058636 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602058+0000", 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204874688 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602113+0000", 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5220999999999999e-05 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602161+0000", 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8065999999999997e-05 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127931 6.7 6:ef914c6b:::benchmark_data_smithi194_134202_object127930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.399418+0000", 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9165153569999998, 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21240136800000001, 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.379 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127931 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399418+0000", 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399418+0000", 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399420+0000", 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0480000000000001e-06 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399487+0000", 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6824999999999998e-05 2024-04-29T14:31:17.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399500+0000", 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3033999999999999e-05 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399508+0000", 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4069999999999993e-06 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399564+0000", 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6212000000000003e-05 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399691+0000", 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012656500000000001 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.399831+0000", 2024-04-29T14:31:17.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013975500000000001 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.407995+0000", 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081639739999999992 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611727+0000", 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20373254900000001 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611776+0000", 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8344000000000003e-05 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611819+0000", 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127933 6.e 6:7f448987:::benchmark_data_smithi194_134202_object127932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.385 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.401233+0000", 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9146999119999997, 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19637354000000001, 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127933 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.401233+0000", 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.401233+0000", 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.401235+0000", 2024-04-29T14:31:17.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.708e-06 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.401282+0000", 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7499e-05 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.401295+0000", 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2081e-05 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.401302+0000", 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.131e-06 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.401353+0000", 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1597000000000001e-05 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.402319+0000", 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00096546499999999999 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.402452+0000", 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133366 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.408211+0000", 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005758712 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.597517+0000", 2024-04-29T14:31:17.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.189306367 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.597568+0000", 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1221999999999998e-05 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.597607+0000", 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8392000000000001e-05 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127941 6.7 6:e5244f40:::benchmark_data_smithi194_134202_object127940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.411095+0000", 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9048381540000001, 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20088029199999999, 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127941 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.392 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411095+0000", 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411095+0000", 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411098+0000", 2024-04-29T14:31:17.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6769999999999999e-06 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411172+0000", 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4436000000000004e-05 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411187+0000", 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5034e-05 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411195+0000", 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2600000000000005e-06 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411255+0000", 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9685999999999997e-05 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411402+0000", 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000146861 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411538+0000", 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013562700000000001 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.415700+0000", 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004162313 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611875+0000", 2024-04-29T14:31:17.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19617462899999999 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611924+0000", 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9459999999999997e-05 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611975+0000", 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1309e-05 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127943 6.6 6:6c7a9dbc:::benchmark_data_smithi194_134202_object127942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.411221+0000", 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9047121889999996, 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19122408199999999, 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127943 2024-04-29T14:31:17.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411221+0000", 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411221+0000", 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411222+0000", 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7399999999999991e-07 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411281+0000", 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9412e-05 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411289+0000", 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9030000000000004e-06 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411295+0000", 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0870000000000003e-06 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411350+0000", 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5013000000000003e-05 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411475+0000", 2024-04-29T14:31:17.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012498899999999999 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.411617+0000", 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014126699999999999 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.415788+0000", 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041710319999999999 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602316+0000", 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18652822499999999 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602408+0000", 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1706999999999995e-05 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602445+0000", 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7472999999999998e-05 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.403 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127951 6.7 6:e3c04ed6:::benchmark_data_smithi194_134202_object127950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.419794+0000", 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.8961392509999992, 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.192316877, 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127951 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.419794+0000", 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.419794+0000", 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.419797+0000", 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4240000000000002e-06 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.419856+0000", 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8776000000000001e-05 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.419873+0000", 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6657999999999999e-05 2024-04-29T14:31:17.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.419881+0000", 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8830000000000006e-06 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.419930+0000", 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9611000000000003e-05 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.420032+0000", 2024-04-29T14:31:17.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010177400000000001 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.420123+0000", 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0650999999999997e-05 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427237+0000", 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071139640000000004 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612023+0000", 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.184786328 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612068+0000", 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4830000000000003e-05 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612111+0000", 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2978000000000001e-05 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127957 6.7 6:edc29919:::benchmark_data_smithi194_134202_object127956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.423092+0000", 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.8928409150000007, 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18913737, 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127957 2024-04-29T14:31:17.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423092+0000", 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423092+0000", 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423094+0000", 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0150000000000002e-06 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423156+0000", 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1809000000000004e-05 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423168+0000", 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1844999999999999e-05 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423174+0000", 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423218+0000", 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4339999999999999e-05 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423345+0000", 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012655899999999999 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.423489+0000", 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000144253 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427418+0000", 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039285020000000004 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612146+0000", 2024-04-29T14:31:17.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.184728531 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612189+0000", 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2979999999999998e-05 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612230+0000", 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0287000000000002e-05 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127960 6.e 6:770b8299:::benchmark_data_smithi194_134202_object127959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.427451+0000", 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.8884816799999999, 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17483077399999999, 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127960 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427451+0000", 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427451+0000", 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427453+0000", 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6199999999999999e-06 2024-04-29T14:31:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427517+0000", 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4208000000000006e-05 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427525+0000", 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7819999999999994e-06 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427531+0000", 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.139e-06 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427571+0000", 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9418000000000001e-05 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427668+0000", 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7773000000000001e-05 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.427752+0000", 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3434000000000001e-05 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.431028+0000", 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032758560000000002 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602202+0000", 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17117415699999999 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602248+0000", 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5720999999999998e-05 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.602282+0000", 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4666000000000003e-05 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127964 6.6 6:65513ed8:::benchmark_data_smithi194_134202_object127963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.435438+0000", 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.880494916, 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.171904999, 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127964 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435438+0000", 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435438+0000", 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435440+0000", 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.745e-06 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435486+0000", 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5674999999999999e-05 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435496+0000", 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0125e-05 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435502+0000", 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4999999999999996e-06 2024-04-29T14:31:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435546+0000", 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.354e-05 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435657+0000", 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111135 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.435764+0000", 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107432 2024-04-29T14:31:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.439005+0000", 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003240167 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.607259+0000", 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16825425399999999 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.607306+0000", 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7216999999999999e-05 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.607343+0000", 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7209000000000002e-05 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127969 6.7 6:ef6b030d:::benchmark_data_smithi194_134202_object127968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.440810+0000", 2024-04-29T14:31:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.875123189, 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17153336899999999, 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127969 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440810+0000", 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440810+0000", 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440811+0000", 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5599999999999995e-07 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440849+0000", 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8022000000000003e-05 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440853+0000", 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7829999999999998e-06 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440857+0000", 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.019e-06 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440895+0000", 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7194000000000003e-05 2024-04-29T14:31:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.440982+0000", 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7274999999999994e-05 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.441070+0000", 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8442000000000005e-05 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.444121+0000", 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030503639999999999 2024-04-29T14:31:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612263+0000", 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16814264900000001 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612304+0000", 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0549000000000001e-05 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612343+0000", 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9515999999999999e-05 2024-04-29T14:31:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127986 6.e 6:70d37946:::benchmark_data_smithi194_134202_object127985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.478493+0000", 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.8374403780000002, 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12884419899999999, 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127986 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478493+0000", 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478493+0000", 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478494+0000", 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2589999999999999e-06 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478549+0000", 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5111999999999997e-05 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: pgmap v1483: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/458440821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.436 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: pgmap v1483: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/458440821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.437 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478559+0000", 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5389999999999996e-06 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478564+0000", 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7350000000000001e-06 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478613+0000", 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8899000000000001e-05 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478750+0000", 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013626999999999999 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.478899+0000", 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014977999999999999 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.482157+0000", 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032572410000000001 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.607245+0000", 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12508877199999999 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.607296+0000", 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0538999999999998e-05 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.607337+0000", 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1053000000000001e-05 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127987 6.7 6:e9790f65:::benchmark_data_smithi194_134202_object127986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.479218+0000", 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.8367153599999995, 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13324159799999999, 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127987 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479218+0000", 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479218+0000", 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479219+0000", 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.015e-06 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479287+0000", 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8515000000000004e-05 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479294+0000", 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0140000000000001e-06 2024-04-29T14:31:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479299+0000", 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9960000000000001e-06 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479343+0000", 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3784e-05 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479436+0000", 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3147999999999997e-05 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.479538+0000", 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101258 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.482414+0000", 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002876651 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612378+0000", 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.129964044 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612419+0000", 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1248000000000002e-05 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.612459+0000", 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9925000000000001e-05 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:17.446 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: pgmap v1483: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/458440821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:17.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:18.082 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:18.082 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:18 smithi121 ceph-mon[44533]: osdmap e569: 12 total, 12 up, 7 in 2024-04-29T14:31:18.082 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:18.082 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:18.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 138292 138276 113.695 71.1875 0.010552 0.00879456 2024-04-29T14:31:18.305 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:18.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:18.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:18.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:18 smithi040 ceph-mon[39581]: osdmap e569: 12 total, 12 up, 7 in 2024-04-29T14:31:18.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:18.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:18.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:18.496 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:18.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:18.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:18.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:18.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:18 smithi194 ceph-mon[44089]: osdmap e569: 12 total, 12 up, 7 in 2024-04-29T14:31:18.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:18.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:18.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:18.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:19.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:31:19.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:19.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 139380 139364 113.101 68 0.0301262 0.00883916 2024-04-29T14:31:19.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:19 smithi194 ceph-mon[44089]: pgmap v1485: 17 pgs: 17 active+clean; 8.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-04-29T14:31:19.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:19.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:19.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:19 smithi194 ceph-mon[44089]: osdmap e570: 12 total, 12 up, 7 in 2024-04-29T14:31:19.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:19 smithi040 ceph-mon[39581]: pgmap v1485: 17 pgs: 17 active+clean; 8.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-04-29T14:31:19.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:19.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:19.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:19 smithi040 ceph-mon[39581]: osdmap e570: 12 total, 12 up, 7 in 2024-04-29T14:31:19.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:19 smithi121 ceph-mon[44533]: pgmap v1485: 17 pgs: 17 active+clean; 8.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-04-29T14:31:19.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:19.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:19.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:19 smithi121 ceph-mon[44533]: osdmap e570: 12 total, 12 up, 7 in 2024-04-29T14:31:19.652 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:31:19.652 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "" 2024-04-29T14:31:19.652 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:31:19.672 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:20.091 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:20.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:20.197 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:20.250 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:20.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:20.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:31:20.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 140404 140388 112.472 64 0.00398519 0.00888923 2024-04-29T14:31:20.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:20.414 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:20.580 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:21.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 141518 141502 111.929 69.625 0.0207289 0.00893263 2024-04-29T14:31:21.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:21.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:21.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:21 smithi121 ceph-mon[44533]: pgmap v1487: 17 pgs: 17 active+clean; 8.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 982 op/s 2024-04-29T14:31:21.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:21 smithi121 ceph-mon[44533]: osdmap e571: 12 total, 12 up, 7 in 2024-04-29T14:31:21.292 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/737157620' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:31:21.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:21.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:21.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:21 smithi194 ceph-mon[44089]: pgmap v1487: 17 pgs: 17 active+clean; 8.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 982 op/s 2024-04-29T14:31:21.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:21 smithi194 ceph-mon[44089]: osdmap e571: 12 total, 12 up, 7 in 2024-04-29T14:31:21.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/737157620' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:21.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:21.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:21.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:21.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:21.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:21.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:21 smithi040 ceph-mon[39581]: pgmap v1487: 17 pgs: 17 active+clean; 8.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 982 op/s 2024-04-29T14:31:21.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:21 smithi040 ceph-mon[39581]: osdmap e571: 12 total, 12 up, 7 in 2024-04-29T14:31:21.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/737157620' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:21.690 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:21.723 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:21.760 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:21.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:21.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:21.902 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:22.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:22.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:22.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:22.262 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:22.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:31:22.268045 min lat: 0.000943876 max lat: 0.224116 avg lat: 0.00897845 2024-04-29T14:31:22.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:31:22.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 142576 142560 111.357 66.125 0.0106505 0.00897845 2024-04-29T14:31:22.387 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:22.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:22.493 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:22.532 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:22.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:22 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:22.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/737157620' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:22.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:22 smithi121 ceph-mon[44533]: osdmap e572: 12 total, 12 up, 7 in 2024-04-29T14:31:22.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:22 smithi121 ceph-mon[44533]: osdmap e573: 12 total, 12 up, 7 in 2024-04-29T14:31:22.557 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:22 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/737157620' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:22 smithi194 ceph-mon[44089]: osdmap e572: 12 total, 12 up, 7 in 2024-04-29T14:31:22.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:22 smithi194 ceph-mon[44089]: osdmap e573: 12 total, 12 up, 7 in 2024-04-29T14:31:22.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:22.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:22 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:22.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/737157620' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:22.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:22 smithi040 ceph-mon[39581]: osdmap e572: 12 total, 12 up, 7 in 2024-04-29T14:31:22.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:22 smithi040 ceph-mon[39581]: osdmap e573: 12 total, 12 up, 7 in 2024-04-29T14:31:22.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:22.693 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:22.827 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:22.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:143212 6.c 6:31680502:::benchmark_data_smithi194_134202_object143211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e573)", 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:22.895162+0000", 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.014315427, 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014348127, 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 143212 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.895162+0000", 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.895162+0000", 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.895162+0000", 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4499999999999996e-07 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.895204+0000", 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1288000000000002e-05 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.895211+0000", 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2350000000000002e-06 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.895217+0000", 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.897872+0000", 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002654974 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.898020+0000", 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000148209 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.898180+0000", 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015948500000000001 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908097+0000", 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099168180000000009 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16809.0:143225 6.3 e573/534 6:c036eefc:::benchmark_data_smithi194_134202_object143224:head v 573'9042, mlcod=573'9040)", 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:22.905935+0000", 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.003542459, 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036627399999999998, 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.905935+0000", 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.905935+0000", 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:22.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.905935+0000", 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0399999999999996e-07 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.905958+0000", 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2396999999999999e-05 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.905964+0000", 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0630000000000002e-06 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.905968+0000", 2024-04-29T14:31:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.906027+0000", 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.906048+0000", 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0231e-05 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:143231 6.c 6:36355f7d:::benchmark_data_smithi194_134202_object143230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e573)", 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:22.908041+0000", 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0014356880000000001, 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001620401, 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 143231 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908041+0000", 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908041+0000", 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908043+0000", 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7060000000000001e-06 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908076+0000", 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3151000000000002e-05 2024-04-29T14:31:22.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908084+0000", 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1119999999999996e-06 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908091+0000", 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2559999999999999e-06 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908128+0000", 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7023000000000001e-05 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908320+0000", 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000192552 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908478+0000", 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000157443 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:143235 6.a 6:57346528:::benchmark_data_smithi194_134202_object143234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e573)", 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:22.908530+0000", 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00094674400000000001, 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012053319999999999, 2024-04-29T14:31:22.936 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 143235 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908530+0000", 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908530+0000", 2024-04-29T14:31:22.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908532+0000", 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.331e-06 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908561+0000", 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9204e-05 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908571+0000", 2024-04-29T14:31:22.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0028e-05 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908578+0000", 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5510000000000002e-06 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908622+0000", 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3896000000000001e-05 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908747+0000", 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012472399999999999 2024-04-29T14:31:22.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:22.908873+0000", 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012584799999999999 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 4 2024-04-29T14:31:22.940 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:31:22.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:23.000 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:23.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:23.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:23.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 143629 143613 110.795 65.8125 0.0125884 0.00902415 2024-04-29T14:31:23.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:23.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:23 smithi121 ceph-mon[44533]: pgmap v1490: 17 pgs: 17 active+clean; 8.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-29T14:31:23.296 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:23.322 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:23.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:23.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:23.459 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:23 smithi040 ceph-mon[39581]: pgmap v1490: 17 pgs: 17 active+clean; 8.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-29T14:31:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:23.472 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:23.483 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:23.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:23.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:23.516 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:23.536 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:23.565 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:23.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:23 smithi194 ceph-mon[44089]: pgmap v1490: 17 pgs: 17 active+clean; 8.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-29T14:31:23.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:23.589 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:23.612 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:23.618 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:23.639 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:23.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:23.642 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:23.665 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:23.689 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:23.712 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:23.732 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:23.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:31:23.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:23.817 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:23.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:23.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:23.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:24.033 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:24.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:24.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:24.166 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:24.203 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:24.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:31:24.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 144631 144615 110.207 62.625 0.0208856 0.00907213 2024-04-29T14:31:24.305 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3037072210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:24.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3037072210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:24.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.547 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:24.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:24.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3037072210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:24.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:24.597 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:24.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:24.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:24.987 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:25.021 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:25.049 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:25.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:25.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:25.187 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:25.209 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:25.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:25.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 16 145702 145686 109.686 66.9375 0.00771857 0.00911545 2024-04-29T14:31:25.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:25.345 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:25.420 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:25.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:25 smithi040 ceph-mon[39581]: pgmap v1492: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:31:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:25.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:25.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:25.555 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:25 smithi121 ceph-mon[44533]: pgmap v1492: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:31:25.558 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:25.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:25 smithi194 ceph-mon[44089]: pgmap v1492: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:31:25.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:25.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:25.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:25.921 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:25.994 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:25.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:26.066 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:31:26.066 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:31:26.066 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:31:26.067 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:31:26.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:26.166 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:26.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:26.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 146749 146733 109.159 65.4375 0.0124976 0.00915968 2024-04-29T14:31:26.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:26.339 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:26.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:26 smithi121 ceph-mon[44533]: pgmap v1493: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:31:26.367 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:26.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:26.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:26.544 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:26.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:31:26.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:26.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:26.600 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:26 smithi040 ceph-mon[39581]: pgmap v1493: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:31:26.605 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:26.652 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:31:26.652 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:31:26.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:26.724 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:26 smithi194 ceph-mon[44089]: pgmap v1493: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-29T14:31:26.726 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:26.760 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:26.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:26.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:26.926 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:27.264 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 16 147778 147762 108.631 64.3125 0.00423621 0.00920356 2024-04-29T14:31:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:31:27.396 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:27.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1561090230' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:31:27.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1561090230' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:31:27.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1561090230' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:31:28.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 148883 148867 108.171 69.0625 0.00785932 0.00924285 2024-04-29T14:31:28.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:28 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:28.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1561090230' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:31:28.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:28 smithi040 ceph-mon[39581]: osdmap e574: 12 total, 12 up, 7 in 2024-04-29T14:31:28.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:28 smithi040 ceph-mon[39581]: pgmap v1495: 17 pgs: 17 active+clean; 8.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:31:28.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:28 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:28.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1561090230' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:31:28.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:28 smithi121 ceph-mon[44533]: osdmap e574: 12 total, 12 up, 7 in 2024-04-29T14:31:28.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:28 smithi121 ceph-mon[44533]: pgmap v1495: 17 pgs: 17 active+clean; 8.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:31:28.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:28 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:28.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1561090230' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:31:28.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:28 smithi194 ceph-mon[44089]: osdmap e574: 12 total, 12 up, 7 in 2024-04-29T14:31:28.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:28 smithi194 ceph-mon[44089]: pgmap v1495: 17 pgs: 17 active+clean; 8.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-29T14:31:29.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 149945 149929 107.691 66.375 0.00545662 0.00928414 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 574, 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:31:27.356856+0000", 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:31:03.499853+0000", 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:26:14.761287+0000", 2024-04-29T14:31:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5804032, 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "nodeep-scrub", 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 30, 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:31:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 6, 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:31:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:31:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:31:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:31:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:31:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 10, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 10, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.20000000298023224, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:31:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 6, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_4", 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:29:45.238640+0000", 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:31:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:31:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "540", 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:31:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:31:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:31:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.5, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 2.5, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.69999998807907104, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 1.75, 2024-04-29T14:31:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 1.75, 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:31:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 323, 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.299 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.300 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:31:29.301 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 152, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:31:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:31:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:31:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:31:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 534, 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:31:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:31:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:31:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 388, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 457, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 561, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 432, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 459, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:31:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/2522198013", 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/2522198013", 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/2522198013", 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/2522198013", 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:31:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 534, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 534, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:31:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:31:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:31:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 534, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:31:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 534, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:31:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 524, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:31:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:31:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:31:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:30:53.510888+0000", 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:30:57.315762+0000", 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:30:18.882428+0000", 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:31:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:31:10.369253+0000", 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:31:06.678981+0000", 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:31:02.692181+0000", 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 457 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:30:13.342698+0000", 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:31:17.372888+0000", 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:31:08.049175+0000", 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:31:18.080551+0000", 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:30:07.094019+0000", 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:31:19.976118+0000", 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:31:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:31:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:31:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:31:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:31:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:31:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:31:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:31:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:31:29.357 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:31:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:29 smithi040 ceph-mon[39581]: 6.0 scrub starts 2024-04-29T14:31:29.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3825638062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:31:29.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:29 smithi121 ceph-mon[44533]: 6.0 scrub starts 2024-04-29T14:31:29.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3825638062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:31:29.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:29 smithi194 ceph-mon[44089]: 6.0 scrub starts 2024-04-29T14:31:29.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3825638062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:31:30.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:30.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:30.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:31:30.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 150940 150924 107.174 62.1875 0.0132224 0.00932927 2024-04-29T14:31:30.316 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:30.345 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:30.379 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:30.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:30.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:30.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:30 smithi121 ceph-mon[44533]: 6.0 scrub ok 2024-04-29T14:31:30.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:30 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:30.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:30 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:30.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:30 smithi121 ceph-mon[44533]: pgmap v1496: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 49 MiB/s wr, 776 op/s 2024-04-29T14:31:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:30.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:30 smithi040 ceph-mon[39581]: 6.0 scrub ok 2024-04-29T14:31:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:30 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:30 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:30.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:30 smithi040 ceph-mon[39581]: pgmap v1496: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 49 MiB/s wr, 776 op/s 2024-04-29T14:31:30.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:30.708 INFO:tasks.thrashosds.thrasher:j is {'epoch': 574, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:31:27.356856+0000', 'last_up_change': '2024-04-29T14:31:03.499853+0000', 'last_in_change': '2024-04-29T14:26:14.761287+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-04-29T14:29:45.238640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '540', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.699999988079071, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 152, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 388, 'last_clean_end': 457, 'up_from': 561, 'up_thru': 432, 'down_at': 459, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 2522198013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2522198013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2522198013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2522198013}]}, 'public_addr': '172.21.15.121:6804/2522198013', 'cluster_addr': '172.21.15.121:6805/2522198013', 'heartbeat_back_addr': '172.21.15.121:6807/2522198013', 'heartbeat_front_addr': '172.21.15.121:6806/2522198013', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 206, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 524, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:30:53.510888+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:30:57.315762+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:30:18.882428+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:31:10.369253+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:31:06.678981+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:31:02.692181+0000', 'dead_epoch': 457}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:30:13.342698+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:31:17.372888+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:31:08.049175+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:31:18.080551+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:30:07.094019+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:31:19.976118+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:31:30.708 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-29T14:31:30.719 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:30.730 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:30 smithi194 ceph-mon[44089]: 6.0 scrub ok 2024-04-29T14:31:30.730 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:30 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:30.730 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:30 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:30.730 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:30 smithi194 ceph-mon[44089]: pgmap v1496: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 49 MiB/s wr, 776 op/s 2024-04-29T14:31:30.733 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:30.761 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:30.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:30.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:31.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 151912 151896 106.652 60.75 0.00826987 0.00937462 2024-04-29T14:31:31.350 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:31.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:31.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:31.485 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:31.526 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:31.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:31.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:31 smithi040 ceph-mon[39581]: osdmap e575: 12 total, 12 up, 7 in 2024-04-29T14:31:31.630 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:31.657 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:31 smithi194 ceph-mon[44089]: osdmap e575: 12 total, 12 up, 7 in 2024-04-29T14:31:31.659 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:31.670 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:31.688 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:31.690 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:31.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:31.709 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:31.730 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:31.754 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:31.774 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:31.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:31.806 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:31.826 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:31.836 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:31.850 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:31.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:31.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:31.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:31 smithi121 ceph-mon[44533]: osdmap e575: 12 total, 12 up, 7 in 2024-04-29T14:31:31.880 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:31.943 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:31.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:31.967 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:31.968 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:31.968 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:32.022 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:32.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 90.0063 2024-04-29T14:31:32.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 152850 2024-04-29T14:31:32.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:31:32.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:31:32.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 106.138 2024-04-29T14:31:32.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 56.2835 2024-04-29T14:31:32.265 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 323.938 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 52.3125 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1698 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 900.536 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 5183 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 837 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.00942096 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.00884947 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 0.22573 2024-04-29T14:31:32.266 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.000943876 2024-04-29T14:31:32.273 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:31:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:32.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:32.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:32.541 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:32.569 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:32.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:32.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: 6.b scrub starts 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/66882542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: 6.a scrub starts 2024-04-29T14:31:32.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:32 smithi040 ceph-mon[39581]: pgmap v1498: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: 6.b scrub starts 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/66882542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: 6.a scrub starts 2024-04-29T14:31:32.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:32 smithi121 ceph-mon[44533]: pgmap v1498: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:31:32.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:32.732 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:32.774 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:32.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:32.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: 6.b scrub starts 2024-04-29T14:31:32.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/66882542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: 6.a scrub starts 2024-04-29T14:31:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:32 smithi194 ceph-mon[44089]: pgmap v1498: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:31:32.883 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:32.911 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:32.932 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:32.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:33.036 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:33.072 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:33.098 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:33.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:33.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:31:33.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:33.281 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:33.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:33.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:33.415 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:33.415 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:33.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:33.446 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:33.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:33.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:33.462 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:33.535 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:31:33.535 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:31:33.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:31:33.535 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:31:33.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285494 4.a 4.c1e0645a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.171799+0000", 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.36044734900003, 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.99503064399999996, 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285494 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171799+0000", 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171799+0000", 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171800+0000", 2024-04-29T14:31:33.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4899999999999999e-06 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171806+0000", 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1430000000000003e-06 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171818+0000", 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1541e-05 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171826+0000", 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.507e-06 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171891+0000", 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4344999999999995e-05 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.171903+0000", 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2147000000000001e-05 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166793+0000", 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.99488997099999998 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166829+0000", 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-05 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.540 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285535 4.a 4.823c013a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.208797+0000", 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.32344902900002, 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95811517000000002, 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285535 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208797+0000", 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208797+0000", 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208801+0000", 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4039999999999997e-06 2024-04-29T14:31:33.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208810+0000", 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4889999999999995e-06 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208827+0000", 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7631000000000001e-05 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208838+0000", 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-29T14:31:33.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208911+0000", 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3070000000000003e-05 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.208925+0000", 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3434e-05 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166904+0000", 2024-04-29T14:31:33.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95797888200000003 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166912+0000", 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4530000000000002e-06 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285539 4.a 4.3cb842da (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.214823+0000", 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.31742256799998, 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95213097099999999, 2024-04-29T14:31:33.545 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285539 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214823+0000", 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214823+0000", 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214825+0000", 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.883e-06 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214832+0000", 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8469999999999998e-06 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214846+0000", 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3531e-05 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214852+0000", 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.533e-06 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214910+0000", 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8216e-05 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.214924+0000", 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3303e-05 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166948+0000", 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.95202394899999998 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166954+0000", 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7089999999999996e-06 2024-04-29T14:31:33.549 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285543 4.a 4.2fcae8fa (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.217886+0000", 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.31435952099997, 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94910000999999999, 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.550 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285543 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217886+0000", 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217886+0000", 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217888+0000", 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4479999999999999e-06 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217892+0000", 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5709999999999999e-06 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217902+0000", 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4809999999999995e-06 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217909+0000", 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8460000000000002e-06 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217963+0000", 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4664999999999999e-05 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217974+0000", 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0424e-05 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166980+0000", 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94900661399999997 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.166986+0000", 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9610000000000002e-06 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285544 4.a 4.151257ca (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.554 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.31425830699999, 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94902677599999996, 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285544 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217988+0000", 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2399999999999997e-07 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217992+0000", 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5470000000000001e-06 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.217999+0000", 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2030000000000003e-06 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.218005+0000", 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6720000000000001e-06 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.218049+0000", 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4567999999999997e-05 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.218058+0000", 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4659999999999999e-06 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167009+0000", 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94895064699999998 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167014+0000", 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8490000000000002e-06 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285562 4.a 4.dc1a1a6a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.239841+0000", 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.29240468400002, 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92720931200000001, 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.559 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285562 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239841+0000", 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239841+0000", 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239843+0000", 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239849+0000", 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4829999999999999e-06 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239860+0000", 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0839000000000001e-05 2024-04-29T14:31:33.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.239867+0000", 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6979999999999998e-06 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.244829+0000", 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049617280000000003 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.244844+0000", 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5390999999999999e-05 2024-04-29T14:31:33.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167044+0000", 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92219957900000005 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167051+0000", 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5010000000000001e-06 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.563 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285587 4.a 4.a38f394a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.260019+0000", 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.27222716300003, 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.907057318, 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285587 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260019+0000", 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260019+0000", 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260020+0000", 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.488e-06 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260029+0000", 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2919999999999996e-06 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260037+0000", 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5210000000000002e-06 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260044+0000", 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4309999999999997e-06 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260104+0000", 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9293999999999998e-05 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.260114+0000", 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.013e-05 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167073+0000", 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90695879700000004 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167076+0000", 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.365e-06 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285588 4.a 4.2f09c4a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.263570+0000", 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.26867637200002, 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90352384399999996, 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285588 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263570+0000", 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263570+0000", 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263571+0000", 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9269999999999999e-06 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263577+0000", 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8200000000000002e-06 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263587+0000", 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0164e-05 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263597+0000", 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.004e-05 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263656+0000", 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8779000000000001e-05 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263668+0000", 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2037e-05 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167090+0000", 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90342208400000001 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167093+0000", 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.993e-06 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285589 4.a 4.ec2d9ada (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.26862634700001, 2024-04-29T14:31:33.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90349196200000004, 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285589 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263620+0000", 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4099999999999998e-07 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263623+0000", 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.162e-06 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263630+0000", 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3709999999999996e-06 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263634+0000", 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8489999999999999e-06 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263692+0000", 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8046e-05 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.263698+0000", 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7150000000000001e-06 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167108+0000", 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.90341010799999999 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167112+0000", 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0699999999999998e-06 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285597 4.a 4.66c85c1a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.270706+0000", 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.26153976900002, 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.896431804, 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285597 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270706+0000", 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270706+0000", 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270708+0000", 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.68e-06 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270714+0000", 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0140000000000004e-06 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270726+0000", 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1968e-05 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270734+0000", 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0209999999999999e-06 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270807+0000", 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2979999999999996e-05 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.270818+0000", 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1398e-05 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167132+0000", 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.89631423300000002 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167138+0000", 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5099999999999998e-06 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285610 4.a 4.5af2b80a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.285975+0000", 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.24627081599999, 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.88120892399999995, 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.581 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285610 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285975+0000", 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285975+0000", 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285977+0000", 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.59e-06 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285982+0000", 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0250000000000002e-06 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285992+0000", 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0117e-05 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.285998+0000", 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.088e-06 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.286049+0000", 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0779999999999998e-05 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.286056+0000", 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2420000000000003e-06 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167180+0000", 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.88112453099999999 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167184+0000", 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.551e-06 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285635 4.a 4.fce1415a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.320869+0000", 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.21137704099999, 2024-04-29T14:31:33.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.84633211799999997, 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285635 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320869+0000", 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320869+0000", 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320871+0000", 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5289999999999998e-06 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320878+0000", 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8979999999999999e-06 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320896+0000", 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7269999999999999e-05 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320905+0000", 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5300000000000002e-06 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320978+0000", 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3125000000000002e-05 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.320988+0000", 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167198+0000", 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.84621011199999996 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167201+0000", 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2600000000000001e-06 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285640 4.a 4.7f958c0a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.327392+0000", 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.20485344299999, 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.83982455300000003, 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285640 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327392+0000", 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327392+0000", 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327394+0000", 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.113e-06 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327398+0000", 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.941e-06 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327404+0000", 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8689999999999998e-06 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327411+0000", 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5100000000000004e-06 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327463+0000", 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2351000000000001e-05 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.327472+0000", 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167214+0000", 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.83974157299999996 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167217+0000", 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.083e-06 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285645 4.a 4.74d5244a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.337231+0000", 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.19501533099998, 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.83000898400000001, 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285645 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337231+0000", 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337231+0000", 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337234+0000", 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337243+0000", 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1239999999999994e-06 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337260+0000", 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6954000000000001e-05 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337270+0000", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0404999999999999e-05 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337335+0000", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4577000000000002e-05 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.337348+0000", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3803e-05 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167234+0000", 2024-04-29T14:31:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.829885769 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167240+0000", 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3739999999999997e-06 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285655 4.a 4.60222c3a (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.360089+0000", 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.17215735000002, 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80718411999999995, 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285655 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360089+0000", 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360089+0000", 2024-04-29T14:31:33.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360091+0000", 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7240000000000001e-06 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360103+0000", 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1438e-05 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360121+0000", 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7929000000000001e-05 2024-04-29T14:31:33.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360131+0000", 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0832000000000001e-05 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360213+0000", 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1219999999999998e-05 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.360226+0000", 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2989999999999999e-05 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167266+0000", 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80704071700000002 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167273+0000", 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2700000000000001e-06 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285667 4.a 4.2c5a08aa (undecoded) ondisk+write+known_if_redirected e414)", 2024-04-29T14:31:33.602 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:39.417619+0000", 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 474.11462651599999, 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.74968378899999999, 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285667 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417619+0000", 2024-04-29T14:31:33.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417619+0000", 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417623+0000", 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4120000000000001e-06 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417632+0000", 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8070000000000006e-06 2024-04-29T14:31:33.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417649+0000", 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7195999999999999e-05 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417659+0000", 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0685e-05 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417733+0000", 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3819999999999995e-05 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:39.417747+0000", 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3725e-05 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167299+0000", 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.74955182099999995 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:40.167303+0000", 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3229999999999999e-06 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285640 4.a 4:5031a9fe:::benchmark_data_smithi194_103039_object128957:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194370+0000", 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.33787565199998, 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26445676699999998, 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285640 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194370+0000", 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194370+0000", 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194371+0000", 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3500000000000005e-07 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194374+0000", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.067e-06 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194380+0000", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4580000000000001e-06 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194383+0000", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3720000000000001e-06 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194601+0000", 2024-04-29T14:31:33.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021826699999999999 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194608+0000", 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1509999999999998e-06 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.426687+0000", 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23207839099999999 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.426766+0000", 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9264000000000005e-05 2024-04-29T14:31:33.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.426848+0000", 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1884000000000004e-05 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.436426+0000", 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095781340000000003 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.436926+0000", 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049993599999999998 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458674+0000", 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021747836 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458727+0000", 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3257999999999997e-05 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458827+0000", 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9914000000000006e-05 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285645 4.a 4:5224ab2e:::benchmark_data_smithi194_103039_object128962:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.33785839299998, 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.264734882, 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285645 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.613 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194388+0000", 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3500000000000002e-07 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194389+0000", 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3999999999999999e-06 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194393+0000", 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6049999999999998e-06 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194398+0000", 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6709999999999998e-06 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194644+0000", 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024664799999999997 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194652+0000", 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9789999999999993e-06 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.431280+0000", 2024-04-29T14:31:33.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.236627529 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.431411+0000", 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000130877 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.431538+0000", 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127627 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.436700+0000", 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051618590000000004 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437199+0000", 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049851799999999999 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458963+0000", 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021764034000000002 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459020+0000", 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7380000000000003e-05 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459122+0000", 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010242 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285655 4.a 4:5c344406:::benchmark_data_smithi194_103039_object128972:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194403+0000", 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.33784243299999, 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26493427200000003, 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285655 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194403+0000", 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194403+0000", 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194404+0000", 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2300000000000002e-07 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194405+0000", 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5710000000000001e-06 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194409+0000", 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4960000000000001e-06 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194412+0000", 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9699999999999999e-06 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194687+0000", 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027504299999999999 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194695+0000", 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1429999999999998e-06 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437242+0000", 2024-04-29T14:31:33.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242547229 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437351+0000", 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108433 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.437478+0000", 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012781199999999999 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.450522+0000", 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013043232 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.450536+0000", 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.468e-05 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459173+0000", 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086369210000000005 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:33.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459225+0000", 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1768999999999999e-05 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459338+0000", 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011265 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25522.0:285667 4.a 4:55105a34:::benchmark_data_smithi194_103039_object128984:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e415)", 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:23:41.194445+0000", 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.33780078900003, 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26555013399999999, 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:33.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25522", 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3706143395", 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 285667 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194445+0000", 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194445+0000", 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194446+0000", 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5000000000000001e-07 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194449+0000", 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7319999999999998e-06 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:33.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194453+0000", 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4669999999999998e-06 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194456+0000", 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194719+0000", 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026247299999999999 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.194728+0000", 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9479999999999997e-06 2024-04-29T14:31:33.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.442078+0000", 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.247350505 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.442204+0000", 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012555300000000001 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10,11", 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.442341+0000", 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013681000000000001 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.450848+0000", 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085075080000000004 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.458636+0000", 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077882380000000003 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459833+0000", 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011965109999999999 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:33.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459892+0000", 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8785e-05 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:23:41.459995+0000", 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103527 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:33.631 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:31:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:33.713 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:33.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:33.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:33.873 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:34.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:34.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:34.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:31:34.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:34.620 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:34.652 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:34.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:34.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:31:34.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:34.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:34 smithi040 ceph-mon[39581]: 6.9 scrub starts 2024-04-29T14:31:34.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:34.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:34.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:34 smithi040 ceph-mon[39581]: 6.a scrub ok 2024-04-29T14:31:34.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:34 smithi040 ceph-mon[39581]: 6.b scrub ok 2024-04-29T14:31:34.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:34 smithi040 ceph-mon[39581]: pgmap v1499: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 66 MiB/s wr, 1.09k op/s 2024-04-29T14:31:34.811 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:34.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:34 smithi194 ceph-mon[44089]: 6.9 scrub starts 2024-04-29T14:31:34.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:34.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:34.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:34 smithi194 ceph-mon[44089]: 6.a scrub ok 2024-04-29T14:31:34.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:34 smithi194 ceph-mon[44089]: 6.b scrub ok 2024-04-29T14:31:34.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:34 smithi194 ceph-mon[44089]: pgmap v1499: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 66 MiB/s wr, 1.09k op/s 2024-04-29T14:31:34.837 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:34.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:34.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:34 smithi121 ceph-mon[44533]: 6.9 scrub starts 2024-04-29T14:31:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:34 smithi121 ceph-mon[44533]: 6.a scrub ok 2024-04-29T14:31:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:34 smithi121 ceph-mon[44533]: 6.b scrub ok 2024-04-29T14:31:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:34 smithi121 ceph-mon[44533]: pgmap v1499: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 66 MiB/s wr, 1.09k op/s 2024-04-29T14:31:34.930 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:34.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:34.968 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:35.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:35.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:31:35.308 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:31:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:35.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:31:35.440 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:35.516 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:35.562 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:35.577 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:35 smithi121 ceph-mon[44533]: 6.1 scrub starts 2024-04-29T14:31:35.577 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:35 smithi121 ceph-mon[44533]: osdmap e576: 12 total, 12 up, 7 in 2024-04-29T14:31:35.577 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:35 smithi121 ceph-mon[44533]: 6.9 scrub ok 2024-04-29T14:31:35.601 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:35.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:35.669 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:35 smithi040 ceph-mon[39581]: 6.1 scrub starts 2024-04-29T14:31:35.669 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:35 smithi040 ceph-mon[39581]: osdmap e576: 12 total, 12 up, 7 in 2024-04-29T14:31:35.669 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:35 smithi040 ceph-mon[39581]: 6.9 scrub ok 2024-04-29T14:31:35.709 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:31:35.710 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:31:35.710 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-29T14:31:35.710 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-04-29T14:31:35.711 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:35.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:35 smithi194 ceph-mon[44089]: 6.1 scrub starts 2024-04-29T14:31:35.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:35 smithi194 ceph-mon[44089]: osdmap e576: 12 total, 12 up, 7 in 2024-04-29T14:31:35.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:35 smithi194 ceph-mon[44089]: 6.9 scrub ok 2024-04-29T14:31:35.749 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:35.776 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:35.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:35.883 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:31:35.911 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:36.263 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:31:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:36.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:31:36.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:36.396 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:36.446 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:31:36.449 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:36.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:36.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:36.587 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/154521959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:36.587 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:36 smithi040 ceph-mon[39581]: pgmap v1501: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 49 MiB/s wr, 944 op/s 2024-04-29T14:31:36.589 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:36.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/154521959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:36.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:36 smithi194 ceph-mon[44089]: pgmap v1501: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 49 MiB/s wr, 944 op/s 2024-04-29T14:31:36.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/154521959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:36.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:36 smithi121 ceph-mon[44533]: pgmap v1501: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 49 MiB/s wr, 944 op/s 2024-04-29T14:31:37.121 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:37.340 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:37.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:37.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:37.481 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:37.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:37 smithi040 ceph-mon[39581]: 6.1 scrub ok 2024-04-29T14:31:37.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:37 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:31:37.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:37 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:31:37.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/154521959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:37.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:37 smithi040 ceph-mon[39581]: osdmap e577: 12 total, 12 up, 7 in 2024-04-29T14:31:37.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:37 smithi194 ceph-mon[44089]: 6.1 scrub ok 2024-04-29T14:31:37.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:37 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:31:37.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:37 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:31:37.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/154521959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:37.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:37 smithi194 ceph-mon[44089]: osdmap e577: 12 total, 12 up, 7 in 2024-04-29T14:31:37.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:37 smithi121 ceph-mon[44533]: 6.1 scrub ok 2024-04-29T14:31:37.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:37 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:31:37.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:37 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:31:37.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/154521959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:37.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:37 smithi121 ceph-mon[44533]: osdmap e577: 12 total, 12 up, 7 in 2024-04-29T14:31:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:37.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:38.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:38.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:38.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:38.445 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:38.446 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:38.446 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:38.446 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:38.446 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3446982455' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.580 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:38 smithi040 ceph-mon[39581]: pgmap v1503: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 134 B/s rd, 45 MiB/s wr, 1.17k op/s 2024-04-29T14:31:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3446982455' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:38 smithi194 ceph-mon[44089]: pgmap v1503: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 134 B/s rd, 45 MiB/s wr, 1.17k op/s 2024-04-29T14:31:38.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3446982455' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:38.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:38 smithi121 ceph-mon[44533]: pgmap v1503: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 134 B/s rd, 45 MiB/s wr, 1.17k op/s 2024-04-29T14:31:39.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:39.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:39.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:39.263 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:39.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:39.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:39.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:39.543 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:39.570 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:39.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:39.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.678 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:39.706 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.706 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.706 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.706 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.709 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:39.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:39.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:39.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:39.884 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:31:39.884 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:31:39.917 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:39.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:39.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:40.058 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:40.092 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:40.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:31:40.116 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:40.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:40.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:40.257 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:40.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:40.496 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:40.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:40.597 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:40 smithi121 ceph-mon[44533]: pgmap v1504: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 10 MiB/s wr, 711 op/s 2024-04-29T14:31:40.599 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:40.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:31:40.666 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:40.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:40.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:40 smithi040 ceph-mon[39581]: pgmap v1504: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 10 MiB/s wr, 711 op/s 2024-04-29T14:31:40.769 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:40.797 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:40 smithi194 ceph-mon[44089]: pgmap v1504: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 10 MiB/s wr, 711 op/s 2024-04-29T14:31:40.799 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:40.823 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:40.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:40.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:40.959 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:41.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:41.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:41.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:41.163 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:41.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:31:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:41.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:41.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:moved 0 / 34 (0%) 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:avg 4.85714 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.72615 -> 1.72615 (expected baseline 2.04041) 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.0 with 4 -> 4 pgs (0.823529 -> 0.823529 * mean) 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.7 with 9 -> 9 pgs (1.85294 -> 1.85294 * mean) 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:oload 1000 2024-04-29T14:31:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 1 2024-04-29T14:31:41.596 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:31:41.596 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0358 2024-04-29T14:31:41.596 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.3582 2024-04-29T14:31:41.601 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:31:41.623 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:42.541 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-29T14:31:42.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-04-29T14:31:42.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:42.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:42.824 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:42.856 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:42.890 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:42.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:42.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:43.056 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:43.084 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:43.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:43.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:43.310 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:43.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:43 smithi194 ceph-mon[44089]: from='client.16977 v1:172.21.15.40:0/2831361614' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:43.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:43 smithi194 ceph-mon[44089]: pgmap v1505: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.2 MiB/s wr, 1.11k op/s 2024-04-29T14:31:43.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:43 smithi040 ceph-mon[39581]: from='client.16977 v1:172.21.15.40:0/2831361614' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:43.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:43 smithi040 ceph-mon[39581]: pgmap v1505: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.2 MiB/s wr, 1.11k op/s 2024-04-29T14:31:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:43 smithi121 ceph-mon[44533]: from='client.16977 v1:172.21.15.40:0/2831361614' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:43 smithi121 ceph-mon[44533]: pgmap v1505: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5.2 MiB/s wr, 1.11k op/s 2024-04-29T14:31:43.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:43.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:43.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:31:43.928 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:44.134 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:31:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:44.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:31:44.211 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:44.211 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:44.211 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:44 smithi040 ceph-mon[39581]: osdmap e578: 12 total, 12 up, 7 in 2024-04-29T14:31:44.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:44.271 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:44.304 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:44.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:44.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:44.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:44.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:44 smithi121 ceph-mon[44533]: osdmap e578: 12 total, 12 up, 7 in 2024-04-29T14:31:44.409 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:44.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:44.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:44.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:44 smithi194 ceph-mon[44089]: osdmap e578: 12 total, 12 up, 7 in 2024-04-29T14:31:44.446 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:44.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:44.572 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:31:44.572 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:31:44.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:44.620 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:44.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:44.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:44.820 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:45.152 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:31:45.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:45 smithi040 ceph-mon[39581]: pgmap v1507: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.5 MiB/s wr, 1.04k op/s 2024-04-29T14:31:45.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2655667149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:31:45.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:45 smithi121 ceph-mon[44533]: pgmap v1507: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.5 MiB/s wr, 1.04k op/s 2024-04-29T14:31:45.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2655667149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:31:45.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:45 smithi194 ceph-mon[44089]: pgmap v1507: 17 pgs: 17 active+clean; 8.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.5 MiB/s wr, 1.04k op/s 2024-04-29T14:31:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2655667149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:31:45.916 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:45.936 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:45.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:45.975 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:45.995 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:46.015 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:46.038 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:46.067 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:46.088 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:46.108 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:46.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:46 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:46.110 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2655667149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:31:46.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:46 smithi121 ceph-mon[44533]: osdmap e579: 12 total, 12 up, 7 in 2024-04-29T14:31:46.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1321520204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:46.111 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:46.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:46.151 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:46.152 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:46.152 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:46.152 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:46.152 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:46.152 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:46.153 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:46.153 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:46.153 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:46.153 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:46.153 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:46.153 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:46.153 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:46.289 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:46.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:46.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:31:46.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:46 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:46.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2655667149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:31:46.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:46 smithi040 ceph-mon[39581]: osdmap e579: 12 total, 12 up, 7 in 2024-04-29T14:31:46.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1321520204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:46.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:46.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:46.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:46 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:46.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2655667149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:31:46.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:46 smithi194 ceph-mon[44089]: osdmap e579: 12 total, 12 up, 7 in 2024-04-29T14:31:46.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1321520204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:46.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: pgmap v1509: 17 pgs: 17 active+clean; 8.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: pgmap v1509: 17 pgs: 17 active+clean; 8.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-04-29T14:31:47.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: pgmap v1509: 17 pgs: 17 active+clean; 8.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:48.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:48.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:48.518 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:48.541 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:48.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:48.645 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:48.677 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:48.721 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:48.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:48.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:48.858 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:48.878 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:48.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:48.982 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:49.027 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:49.051 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:49.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:49.121 INFO:teuthology.orchestra.run.smithi040.stdout:moved 0 / 34 (0%) 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:avg 4.85714 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.72615 -> 1.72615 (expected baseline 2.04041) 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.0 with 4 -> 4 pgs (0.823529 -> 0.823529 * mean) 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.7 with 9 -> 9 pgs (1.85294 -> 1.85294 * mean) 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 0.05 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0340 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0374 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 1.0000 -> 0.9500 2024-04-29T14:31:49.122 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 weight 1.0000 -> 0.9500 2024-04-29T14:31:49.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:49.182 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:49.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:49 smithi194 ceph-mon[44089]: pgmap v1510: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:49.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:49 smithi194 ceph-mon[44089]: from='client.16995 v1:172.21.15.40:0/476689125' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:49.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260'}"}]: dispatch 2024-04-29T14:31:49.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:31:49.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:49 smithi040 ceph-mon[39581]: pgmap v1510: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:49.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:49 smithi040 ceph-mon[39581]: from='client.16995 v1:172.21.15.40:0/476689125' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:49.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260'}"}]: dispatch 2024-04-29T14:31:49.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:31:49.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:49 smithi121 ceph-mon[44533]: pgmap v1510: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:49.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:49 smithi121 ceph-mon[44533]: from='client.16995 v1:172.21.15.40:0/476689125' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:49.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260'}"}]: dispatch 2024-04-29T14:31:49.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:31:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:49.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:31:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:49.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:49.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:49.564 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:49.588 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:49.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:49.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:49.722 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:50.069 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-29T14:31:50.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-04-29T14:31:50.074 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:50.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:50.168 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:31:50.168 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:31:50.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:81344 5.9 5:971ead96:::benchmark_data_smithi194_121512_object81343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:14.189696+0000", 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 395.97736662400001, 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104765369, 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.169 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81344 2024-04-29T14:31:50.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.170 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189696+0000", 2024-04-29T14:31:50.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189696+0000", 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189698+0000", 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9690000000000001e-06 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189727+0000", 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8183000000000001e-05 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189741+0000", 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4250000000000001e-05 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189748+0000", 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9500000000000004e-06 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189807+0000", 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9463999999999999e-05 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.189937+0000", 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129791 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.190037+0000", 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9660999999999993e-05 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.196209+0000", 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061724980000000002 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.294364+0000", 2024-04-29T14:31:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098154929000000002 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.294422+0000", 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.791e-05 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.294462+0000", 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9764000000000001e-05 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:81351 5.9 5:90cc511c:::benchmark_data_smithi194_121512_object81350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:14.191802+0000", 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 395.97526072699998, 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.116692612, 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81351 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.191802+0000", 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.191802+0000", 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.191804+0000", 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.852e-06 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.191834+0000", 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9896e-05 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.191847+0000", 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2928000000000001e-05 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.191854+0000", 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1239999999999999e-06 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.193817+0000", 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019625269999999999 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.193940+0000", 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012291099999999999 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.194071+0000", 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013114800000000001 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.202535+0000", 2024-04-29T14:31:50.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084643800000000005 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.308389+0000", 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105854199 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.308446+0000", 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6895999999999999e-05 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.308495+0000", 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8751000000000001e-05 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.180 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:81359 5.9 5:91711517:::benchmark_data_smithi194_121512_object81358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:14.197296+0000", 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 395.96976683000003, 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.116895942, 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81359 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.181 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.197296+0000", 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.197296+0000", 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.197298+0000", 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0279999999999999e-06 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.197360+0000", 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1612999999999994e-05 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.197370+0000", 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0305e-05 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.197378+0000", 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3479999999999998e-06 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.199790+0000", 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002412483 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.199931+0000", 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014097099999999999 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.200067+0000", 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000135828 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.202741+0000", 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026738119999999998 2024-04-29T14:31:50.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314093+0000", 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.11135244399999999 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314149+0000", 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5554999999999999e-05 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314192+0000", 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3554999999999999e-05 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.185 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:81370 5.9 5:9a8ff0f9:::benchmark_data_smithi194_121512_object81369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:14.203396+0000", 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 395.96366677999998, 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.11091812099999999, 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81370 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.186 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203396+0000", 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203396+0000", 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203398+0000", 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8190000000000001e-06 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203441+0000", 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3349000000000001e-05 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203453+0000", 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1066999999999999e-05 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203459+0000", 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5320000000000003e-06 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203508+0000", 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9412000000000001e-05 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203635+0000", 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012646700000000001 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.203794+0000", 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000158701 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.208470+0000", 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046764340000000001 2024-04-29T14:31:50.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314237+0000", 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105766531 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314283+0000", 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5997e-05 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314314+0000", 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1812e-05 2024-04-29T14:31:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:81375 5.9 5:94e972bb:::benchmark_data_smithi194_121512_object81374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:25:14.205791+0000", 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 395.96127241400001, 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.108665898, 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81375 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.205791+0000", 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.205791+0000", 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.205792+0000", 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.432e-06 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.205858+0000", 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5764000000000003e-05 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.205867+0000", 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9439999999999994e-06 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.205872+0000", 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9110000000000001e-06 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.206083+0000", 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021148 2024-04-29T14:31:50.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.206211+0000", 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012747999999999999 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.206314+0000", 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103683 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.212197+0000", 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058828739999999997 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314378+0000", 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102180986 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314424+0000", 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6097000000000003e-05 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:25:14.314457+0000", 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2246999999999998e-05 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:215568 5.6 5:6d611a42:::benchmark_data_smithi194_121512_object82102:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e505)", 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:28:29.336020+0000", 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 200.83104335300001, 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106592981, 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 215568 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.336020+0000", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.336020+0000", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.336021+0000", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4079999999999999e-06 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.336025+0000", 2024-04-29T14:31:50.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0319999999999997e-06 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.336034+0000", 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5099999999999998e-06 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.336039+0000", 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6849999999999998e-06 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.342766+0000", 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067266549999999998 2024-04-29T14:31:50.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.342883+0000", 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011723899999999999 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.343048+0000", 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016428800000000001 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.354525+0000", 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011477193 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.440543+0000", 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.086018489000000004 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.440601+0000", 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7544000000000003e-05 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.442613+0000", 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002011938 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:215583 5.6 5:62d7e22e:::benchmark_data_smithi194_121512_object82117:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e505)", 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:28:29.337675+0000", 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 200.82938770000001, 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106833728, 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 215583 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.337675+0000", 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.337675+0000", 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.337677+0000", 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.316e-06 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.337681+0000", 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0130000000000004e-06 2024-04-29T14:31:50.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.337691+0000", 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0000000000000001e-05 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.337697+0000", 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0689999999999998e-06 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.347564+0000", 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098675949999999998 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.347662+0000", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7726000000000006e-05 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.347764+0000", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010155299999999999 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.354535+0000", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067710560000000001 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.442668+0000", 2024-04-29T14:31:50.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.088133058 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.442721+0000", 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2825000000000003e-05 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:29.444509+0000", 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017885170000000001 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.205 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:236160 5.6 5:613b97fa:::benchmark_data_smithi194_121512_object102694:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:28:55.347222+0000", 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 174.81984083899999, 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13574907999999999, 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 236160 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.347222+0000", 2024-04-29T14:31:50.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.347222+0000", 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.347225+0000", 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4530000000000001e-06 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.347231+0000", 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5980000000000002e-06 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.347241+0000", 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5799999999999998e-06 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.347249+0000", 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9219999999999997e-06 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.348076+0000", 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000827121 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.348175+0000", 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9177000000000003e-05 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.348323+0000", 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000148047 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.357657+0000", 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093342630000000006 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.481650+0000", 2024-04-29T14:31:50.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12399241699999999 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.481714+0000", 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.423e-05 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.482971+0000", 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001257272 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:236178 5.6 5:6db2ac00:::benchmark_data_smithi194_121512_object102712:head [delete] snapc 0=[] ondisk+write+known_if_redirected e516)", 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:28:55.365810+0000", 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 174.801252906, 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.117971992, 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 236178 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365810+0000", 2024-04-29T14:31:50.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365810+0000", 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365812+0000", 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.99e-06 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365817+0000", 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8709999999999996e-06 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365828+0000", 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0468e-05 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365834+0000", 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.545e-06 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365896+0000", 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1894000000000001e-05 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.365973+0000", 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6997999999999997e-05 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.366061+0000", 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8011999999999997e-05 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.370548+0000", 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044868299999999998 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.483025+0000", 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.112477019 2024-04-29T14:31:50.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.483081+0000", 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6641000000000002e-05 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:28:55.483782+0000", 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070072400000000001 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:123348 6.8 6:1d0655ee:::benchmark_data_smithi194_134202_object123347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.803484+0000", 2024-04-29T14:31:50.215 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.363579334999997, 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.096323571999999996, 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 123348 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.803484+0000", 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.803484+0000", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.803486+0000", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8479999999999999e-06 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.803539+0000", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3060999999999999e-05 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.803548+0000", 2024-04-29T14:31:50.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8710000000000003e-06 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.803554+0000", 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7139999999999996e-06 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806033+0000", 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002478732 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806170+0000", 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000137477 2024-04-29T14:31:50.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806298+0000", 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012750399999999999 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899649+0000", 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093351181000000005 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899696+0000", 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6891999999999998e-05 2024-04-29T14:31:50.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899751+0000", 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4806999999999999e-05 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899807+0000", 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6484999999999999e-05 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.220 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:123351 6.8 6:169f4a3f:::benchmark_data_smithi194_134202_object123350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.806395+0000", 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.360667951000003, 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.094226708000000006, 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 123351 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806395+0000", 2024-04-29T14:31:50.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806395+0000", 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806397+0000", 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.968e-06 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806467+0000", 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9622999999999994e-05 2024-04-29T14:31:50.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806476+0000", 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.799e-06 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.806483+0000", 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.25e-06 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810043+0000", 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035605200000000002 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810156+0000", 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112769 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810292+0000", 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013630700000000001 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899821+0000", 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089528653 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.900511+0000", 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00068973000000000005 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.900579+0000", 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8442000000000006e-05 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.900622+0000", 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2647e-05 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16809.0:123343 6.e e561/534 6:7b3eb05d:::benchmark_data_smithi194_134202_object123342:head v 561'7643, mlcod=561'7641)", 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.807017+0000", 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.360046498999999, 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.092637919999999999, 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807017+0000", 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807017+0000", 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807018+0000", 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.37e-06 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807039+0000", 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1058999999999999e-05 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807048+0000", 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0010000000000007e-06 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807054+0000", 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.46e-06 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807581+0000", 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052675599999999997 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807602+0000", 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0369000000000001e-05 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899608+0000", 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.092006129000000006 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899655+0000", 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6776000000000001e-05 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16809.0:123346 6.6 e561/534 6:61b9ae63:::benchmark_data_smithi194_134202_object123345:head v 561'7745, mlcod=561'7743)", 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.807073+0000", 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.359990318000001, 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.092614816000000003, 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807073+0000", 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807073+0000", 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807073+0000", 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7800000000000001e-07 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807093+0000", 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9440999999999999e-05 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807098+0000", 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4489999999999998e-06 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807103+0000", 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7489999999999998e-06 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.809812+0000", 2024-04-29T14:31:50.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027085880000000001 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.809831+0000", 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8960000000000001e-05 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899659+0000", 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089828696999999999 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899688+0000", 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8354e-05 2024-04-29T14:31:50.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16809.0:123345 6.e e561/534 6:78d5875d:::benchmark_data_smithi194_134202_object123344:head v 561'7644, mlcod=561'7641)", 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.807214+0000", 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.359848861000003, 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.092499682999999999, 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807214+0000", 2024-04-29T14:31:50.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807214+0000", 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807215+0000", 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5799999999999998e-07 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807239+0000", 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.385e-05 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807245+0000", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1700000000000002e-06 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.807250+0000", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0409999999999997e-06 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811398+0000", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041483420000000002 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811418+0000", 2024-04-29T14:31:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9967999999999999e-05 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899690+0000", 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.088271042999999993 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899714+0000", 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4411000000000001e-05 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:123354 6.3 6:cef2f438:::benchmark_data_smithi194_134202_object123353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-04-29T14:31:50.237 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.810580+0000", 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.356483044000001, 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.094065466, 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 123354 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810580+0000", 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810580+0000", 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810582+0000", 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2340000000000001e-06 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810626+0000", 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4042000000000003e-05 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810635+0000", 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.687e-06 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810642+0000", 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7100000000000001e-06 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811818+0000", 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176697 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811936+0000", 2024-04-29T14:31:50.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117891 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.812053+0000", 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117173 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.900172+0000", 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.088118428999999998 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.904551+0000", 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043794539999999996 2024-04-29T14:31:50.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.904599+0000", 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7621999999999999e-05 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.904646+0000", 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6526999999999997e-05 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:123359 6.8 6:14a0a413:::benchmark_data_smithi194_134202_object123358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-04-29T14:31:50.242 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.810894+0000", 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.356169061999999, 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090615349999999997, 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 123359 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810894+0000", 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810894+0000", 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.810896+0000", 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4869999999999999e-06 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811026+0000", 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000130696 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811036+0000", 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3540000000000006e-06 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811043+0000", 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.311e-06 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.814120+0000", 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030772690000000001 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.814251+0000", 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013074199999999999 2024-04-29T14:31:50.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.814382+0000", 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013095399999999999 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899827+0000", 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085444929000000003 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.901413+0000", 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015859419999999999 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.901464+0000", 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1187000000000003e-05 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.901509+0000", 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5479000000000003e-05 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16809.0:123353 6.e e561/534 6:75a08b4f:::benchmark_data_smithi194_134202_object123352:head v 561'7645, mlcod=561'7641)", 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.811413+0000", 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.355650117000003, 2024-04-29T14:31:50.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10357068, 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811413+0000", 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811413+0000", 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811414+0000", 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2389999999999999e-06 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811437+0000", 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2297e-05 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811444+0000", 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1999999999999997e-06 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.811450+0000", 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.891e-06 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899744+0000", 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.088294361000000002 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899763+0000", 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8830000000000001e-05 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.914941+0000", 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015177783 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.914984+0000", 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3078999999999999e-05 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16809.0:123360 6.c e561/534 6:3cdca07a:::benchmark_data_smithi194_134202_object123359:head v 561'7744, mlcod=561'7742)", 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:03.813861+0000", 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.353202240000002, 2024-04-29T14:31:50.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.095364841000000006, 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.813861+0000", 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.813861+0000", 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.813861+0000", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7100000000000002e-07 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.813886+0000", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5111000000000001e-05 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.813892+0000", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8499999999999999e-06 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.813897+0000", 2024-04-29T14:31:50.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0159999999999999e-06 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899632+0000", 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085734315000000005 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.899655+0000", 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3838000000000001e-05 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.909182+0000", 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095266650000000001 2024-04-29T14:31:50.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:03.909226+0000", 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3575000000000002e-05 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:125433 6.5 6:a6959291:::benchmark_data_smithi194_134202_object125432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e562)", 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:05.935338+0000", 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.231724989, 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089817183999999994, 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.255 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 125433 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935338+0000", 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935338+0000", 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935339+0000", 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.16e-07 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935376+0000", 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7302999999999998e-05 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935389+0000", 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3021e-05 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935400+0000", 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0285000000000001e-05 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935460+0000", 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.071e-05 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935584+0000", 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123869 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.935708+0000", 2024-04-29T14:31:50.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012425000000000001 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.947127+0000", 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011419054 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.025049+0000", 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077921615 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.025105+0000", 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5918999999999998e-05 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.025155+0000", 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0342e-05 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:125438 6.8 6:17d6f52c:::benchmark_data_smithi194_134202_object125437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e562)", 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:05.936567+0000", 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.230496193, 2024-04-29T14:31:50.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097844650000000005, 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 125438 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.936567+0000", 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:31:50.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.936567+0000", 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.936568+0000", 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3120000000000001e-06 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.936866+0000", 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000297874 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:31:50.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.936876+0000", 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7699999999999996e-06 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:05.936883+0000", 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0550000000000002e-06 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.020695+0000", 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.083811659999999996 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.020820+0000", 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012499599999999999 2024-04-29T14:31:50.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.020942+0000", 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012240800000000001 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.025312+0000", 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043701479999999999 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.034300+0000", 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089882090000000005 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.034352+0000", 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1885000000000002e-05 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:06.034412+0000", 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9333000000000003e-05 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:31:50.265 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:31:50.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:50.266 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260'}"}]': finished 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: osdmap e580: 12 total, 12 up, 7 in 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:31:50.293 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.295 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:50.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:31:50.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260'}"}]': finished 2024-04-29T14:31:50.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: osdmap e580: 12 total, 12 up, 7 in 2024-04-29T14:31:50.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:31:50.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:31:50.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.567 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260'}"}]': finished 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: osdmap e580: 12 total, 12 up, 7 in 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:31:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:31:51.147 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:51.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:51.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:51.279 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:51.352 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:51.354 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:51 smithi040 ceph-mon[39581]: pgmap v1512: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-04-29T14:31:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:51.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:51.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:31:51.457 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:51 smithi121 ceph-mon[44533]: pgmap v1512: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-04-29T14:31:51.488 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:51.499 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:31:51.536 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:51 smithi194 ceph-mon[44089]: pgmap v1512: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-04-29T14:31:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:31:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:51.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:31:51.715 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:51.715 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:51.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:51.844 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:51.855 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:31:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:51.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:51.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:52.050 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:52.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:52.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:52.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:52.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:31:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:52.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:31:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:52.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:31:52.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:31:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:52.620 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:31:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:52.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:31:52.724 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:52.763 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:52.891 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:52.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:53.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:31:53.045 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:31:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:53.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:31:53.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:31:53.232 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:53.286 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:31:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:53.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:31:53.317 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:53 smithi194 ceph-mon[44089]: pgmap v1513: 17 pgs: 17 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:53.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:53 smithi121 ceph-mon[44533]: pgmap v1513: 17 pgs: 17 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:53.391 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:53.421 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:53.529 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:53.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:53.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:53 smithi040 ceph-mon[39581]: pgmap v1513: 17 pgs: 17 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:53.636 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:31:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:53.789 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:31:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:53.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:31:53.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:54.046 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:54.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:54.149 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:54.196 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:54.228 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:54.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:54.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:54.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:54.508 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:31:54.508 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:31:54.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.508 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.16809.0:174865 6.f e580/534 6:f8fb6ecf:::benchmark_data_smithi194_134202_object22012:head v 580'10920, mlcod=580'10918)", 2024-04-29T14:31:54.508 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:31:54.503674+0000", 2024-04-29T14:31:54.508 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0071110160000000004, 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00714243, 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503674+0000", 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503674+0000", 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503676+0000", 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5659999999999999e-06 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503683+0000", 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6899999999999992e-06 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503696+0000", 2024-04-29T14:31:54.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2762000000000001e-05 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503704+0000", 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5569999999999998e-06 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503787+0000", 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2849999999999995e-05 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.503809+0000", 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2750000000000001e-05 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.16809.0:174866 6.f e580/534 6:f10cc184:::benchmark_data_smithi194_134202_object22013:head v 580'10921, mlcod=580'10918)", 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:31:54.507356+0000", 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0034291059999999999, 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035277009999999998, 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:31:54.512 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507356+0000", 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507356+0000", 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:31:54.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507358+0000", 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.621e-06 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507367+0000", 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9910000000000008e-06 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507380+0000", 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3689000000000001e-05 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507388+0000", 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7239999999999992e-06 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.508670+0000", 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001282032 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.508693+0000", 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.302e-05 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:31:54.515 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:174870 6.1 6:8d2b86c0:::benchmark_data_smithi194_134202_object22017:head [delete] snapc 0=[] ondisk+write+known_if_redirected e580)", 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:31:54.507530+0000", 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0032548080000000001, 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003417584, 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174870 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:31:54.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507530+0000", 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507530+0000", 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507532+0000", 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0279999999999999e-06 2024-04-29T14:31:54.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507539+0000", 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.212e-06 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507552+0000", 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3525e-05 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507560+0000", 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4599999999999997e-06 2024-04-29T14:31:54.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507618+0000", 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8489999999999997e-05 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507734+0000", 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116247 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:31:54.507839+0000", 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000104942 2024-04-29T14:31:54.519 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:31:54.520 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:31:54.520 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:31:54.520 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:31:54.520 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:31:54.520 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 3 2024-04-29T14:31:54.520 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:31:54.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:54.562 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:55.291 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:31:55.313 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:55.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:55.418 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:55.455 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:55.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:55 smithi194 ceph-mon[44089]: pgmap v1514: 17 pgs: 17 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:55.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/565199380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:55.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:55 smithi040 ceph-mon[39581]: pgmap v1514: 17 pgs: 17 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:55.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/565199380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:55.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:55 smithi121 ceph-mon[44533]: pgmap v1514: 17 pgs: 17 active+clean; 8.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:31:55.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/565199380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:31:55.677 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:55.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:55.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:31:55.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:55.826 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:56.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:56.113 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:31:56.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:56.159 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:56.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:56 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:56.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/565199380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:56.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:56 smithi194 ceph-mon[44089]: osdmap e581: 12 total, 12 up, 7 in 2024-04-29T14:31:56.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:56 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:31:56.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:56 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:31:56.582 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:31:56.582 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:31:56.582 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:31:56.582 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:31:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:moved 0 / 34 (0%) 2024-04-29T14:31:56.590 INFO:teuthology.orchestra.run.smithi040.stdout:avg 4.85714 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.72615 -> 1.72615 (expected baseline 2.04041) 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.0 with 4 -> 4 pgs (0.823529 -> 0.823529 * mean) 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.7 with 9 -> 9 pgs (1.85294 -> 1.85294 * mean) 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:oload 1000 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 1 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0326 2024-04-29T14:31:56.591 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.3256 2024-04-29T14:31:56.592 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:31:56.594 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:31:56.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:56 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:56.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/565199380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:56.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:56 smithi040 ceph-mon[39581]: osdmap e581: 12 total, 12 up, 7 in 2024-04-29T14:31:56.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:56 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:31:56.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:56 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:31:56.622 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:31:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:56 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:31:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/565199380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:31:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:56 smithi121 ceph-mon[44533]: osdmap e581: 12 total, 12 up, 7 in 2024-04-29T14:31:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:56 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:31:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:56 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:31:56.648 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:31:56.690 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:31:56.719 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:31:56.745 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:31:56.766 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:31:56.790 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:31:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:31:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:31:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:31:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:57.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:57.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:57.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:57.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: pgmap v1516: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osdmap e582: 12 total, 12 up, 7 in 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/979873872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.351 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:57.392 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:57.426 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:57.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:57.534 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:57.584 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: pgmap v1516: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osdmap e582: 12 total, 12 up, 7 in 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/979873872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: pgmap v1516: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-04-29T14:31:57.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osdmap e582: 12 total, 12 up, 7 in 2024-04-29T14:31:57.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/979873872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:31:57.749 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:57.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:57.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:57.854 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:31:57.891 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:57.941 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:31:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:57.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:31:58.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:58.078 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:58.107 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:58.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:58.155 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:31:58.156 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:31:58.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:31:58.268 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:58.329 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:31:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:58.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:31:58.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:31:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:58.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:58 smithi194 ceph-mon[44089]: from='client.17007 v1:172.21.15.40:0/1436179080' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:58.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:58 smithi040 ceph-mon[39581]: from='client.17007 v1:172.21.15.40:0/1436179080' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:31:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:58.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:31:58.605 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-29T14:31:58.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-04-29T14:31:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:58 smithi121 ceph-mon[44533]: from='client.17007 v1:172.21.15.40:0/1436179080' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:31:58.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:31:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:58.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:31:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:58.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:31:58.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:58.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:31:58.938 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:31:58.980 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:58.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:59.042 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:31:59.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:59.138 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:59.177 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:31:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:59.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:31:59.284 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:31:59.325 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:31:59.327 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:59.353 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:31:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:59.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:31:59.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:31:59 smithi040 ceph-mon[39581]: pgmap v1518: 17 pgs: 17 active+clean; 7.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-29T14:31:59.459 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:31:59.496 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:31:59 smithi194 ceph-mon[44089]: pgmap v1518: 17 pgs: 17 active+clean; 7.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-29T14:31:59.498 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:31:59.521 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:31:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:31:59.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:31:59.629 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:31:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:31:59 smithi121 ceph-mon[44533]: pgmap v1518: 17 pgs: 17 active+clean; 7.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-29T14:31:59.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:00.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:32:00.226 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:00.525 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:00 smithi040 ceph-mon[39581]: pgmap v1519: 17 pgs: 17 active+clean; 7.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 775 op/s 2024-04-29T14:32:00.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:00 smithi121 ceph-mon[44533]: pgmap v1519: 17 pgs: 17 active+clean; 7.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 775 op/s 2024-04-29T14:32:00.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:00 smithi194 ceph-mon[44089]: pgmap v1519: 17 pgs: 17 active+clean; 7.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 775 op/s 2024-04-29T14:32:01.616 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:01 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:01.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:01 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:01.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:01 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:01.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:01 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:01.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:01 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:01.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:01 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:02.126 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:02.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:02.233 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:02.262 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:02.268 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:02.301 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:02.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:moved 3 / 34 (8.82353%) 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:avg 4.85714 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.72615 -> 1.12486 (expected baseline 2.04041) 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.0 with 4 -> 4 pgs (0.823529 -> 0.823529 * mean) 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.7 with 9 -> 6 pgs (1.85294 -> 1.23529 * mean) 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0316 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0348 2024-04-29T14:32:02.336 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 0.9500 -> 0.5831 2024-04-29T14:32:02.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:02.435 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:02.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:02 smithi040 ceph-mon[39581]: osdmap e583: 12 total, 12 up, 7 in 2024-04-29T14:32:02.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:02 smithi040 ceph-mon[39581]: from='client.26701 v1:172.21.15.40:0/694381209' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:32:02.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:02 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'38217'}"}]: dispatch 2024-04-29T14:32:02.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:02 smithi040 ceph-mon[39581]: pgmap v1521: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-29T14:32:02.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:02 smithi121 ceph-mon[44533]: osdmap e583: 12 total, 12 up, 7 in 2024-04-29T14:32:02.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:02 smithi121 ceph-mon[44533]: from='client.26701 v1:172.21.15.40:0/694381209' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:32:02.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:02 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'38217'}"}]: dispatch 2024-04-29T14:32:02.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:02 smithi121 ceph-mon[44533]: pgmap v1521: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-29T14:32:02.664 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:02 smithi194 ceph-mon[44089]: osdmap e583: 12 total, 12 up, 7 in 2024-04-29T14:32:02.664 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:02 smithi194 ceph-mon[44089]: from='client.26701 v1:172.21.15.40:0/694381209' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:32:02.664 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:02 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'7':'38217'}"}]: dispatch 2024-04-29T14:32:02.664 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:02 smithi194 ceph-mon[44089]: pgmap v1521: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.41k op/s 2024-04-29T14:32:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:02.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:02.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:03.018 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:03.041 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:03.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:03.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:03.183 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:32:03.448 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'38217'}"}]': finished 2024-04-29T14:32:03.448 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:03 smithi040 ceph-mon[39581]: osdmap e584: 12 total, 12 up, 7 in 2024-04-29T14:32:03.448 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3483773401' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:32:03.472 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-29T14:32:03.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-04-29T14:32:03.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'38217'}"}]': finished 2024-04-29T14:32:03.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:03 smithi194 ceph-mon[44089]: osdmap e584: 12 total, 12 up, 7 in 2024-04-29T14:32:03.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3483773401' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:32:03.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'7':'38217'}"}]': finished 2024-04-29T14:32:03.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:03 smithi121 ceph-mon[44533]: osdmap e584: 12 total, 12 up, 7 in 2024-04-29T14:32:03.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3483773401' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:32:03.995 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:32:03.995 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:32:04.133 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:04.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:04.236 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:04.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:04.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:04 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:04.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3483773401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:32:04.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:04 smithi040 ceph-mon[39581]: osdmap e585: 12 total, 12 up, 7 in 2024-04-29T14:32:04.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:04 smithi040 ceph-mon[39581]: pgmap v1524: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1k op/s 2024-04-29T14:32:04.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:04.427 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:04.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:04.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:04.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:04.585 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:04.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:04.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:04 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:04.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3483773401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:32:04.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:04 smithi121 ceph-mon[44533]: osdmap e585: 12 total, 12 up, 7 in 2024-04-29T14:32:04.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:04 smithi121 ceph-mon[44533]: pgmap v1524: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1k op/s 2024-04-29T14:32:04.726 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:04.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:04 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:04.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3483773401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:32:04.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:04 smithi194 ceph-mon[44089]: osdmap e585: 12 total, 12 up, 7 in 2024-04-29T14:32:04.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:04 smithi194 ceph-mon[44089]: pgmap v1524: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1k op/s 2024-04-29T14:32:04.763 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:05.240 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:05.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:05 smithi040 ceph-mon[39581]: osdmap e586: 12 total, 12 up, 7 in 2024-04-29T14:32:05.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:05 smithi040 ceph-mon[39581]: 6.8 starting backfill to osd.6 from (0'0,0'0] MAX to 583'11510 2024-04-29T14:32:05.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:05 smithi040 ceph-mon[39581]: 6.b starting backfill to osd.10 from (0'0,0'0] MAX to 583'11566 2024-04-29T14:32:05.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:05 smithi121 ceph-mon[44533]: osdmap e586: 12 total, 12 up, 7 in 2024-04-29T14:32:05.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:05 smithi121 ceph-mon[44533]: 6.8 starting backfill to osd.6 from (0'0,0'0] MAX to 583'11510 2024-04-29T14:32:05.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:05 smithi121 ceph-mon[44533]: 6.b starting backfill to osd.10 from (0'0,0'0] MAX to 583'11566 2024-04-29T14:32:05.686 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:05 smithi194 ceph-mon[44089]: osdmap e586: 12 total, 12 up, 7 in 2024-04-29T14:32:05.686 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:05 smithi194 ceph-mon[44089]: 6.8 starting backfill to osd.6 from (0'0,0'0] MAX to 583'11510 2024-04-29T14:32:05.686 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:05 smithi194 ceph-mon[44089]: 6.b starting backfill to osd.10 from (0'0,0'0] MAX to 583'11566 2024-04-29T14:32:05.717 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:05.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:05.726 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:32:05.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:05.899 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:05.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:06.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:06.066 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:06.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:06.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:06.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:32:06.155 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:06.194 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:06.224 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:06.279 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:06.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:06.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: osdmap e587: 12 total, 12 up, 7 in 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: pgmap v1527: 17 pgs: 1 active+remapped+backfilling, 1 peering, 15 active+clean; 7.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 7499/244574 objects misplaced (3.066%); 1.9 MiB/s, 28 objects/s recovering 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=2543142174,config=1981513354,health=1997216375,kv=1765154590}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=922487200,logm=4251823315}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2810114863}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686552495}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1515282198}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1852520498}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994523954}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299021641}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903140785}) 2024-04-29T14:32:06.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2729215579}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343031670}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827283320}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2065720000}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2203548925}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488117310}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2403267985}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603146744}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440867325}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477013805}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528017702}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769924536}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1628678745}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976088586}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1732784061}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3861233568}) 2024-04-29T14:32:06.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656245915}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1552156934}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2087144462}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503185934}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707525264}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2183553525}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060336865}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1316082246}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3849666162}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175698253}) 2024-04-29T14:32:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=51829615}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987707212}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028031856}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3931088709}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093779403}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901682753}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962385739}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2911383412}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=645566984}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2365799495}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926929227}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522180241}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399241385}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=469112098}) 2024-04-29T14:32:06.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3356676680}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042566286}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2507583422}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021154841}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1668070417}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889787965}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177617823}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312586013}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851666502}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=306977434}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2811429942}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101528359}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186584258}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730822055}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784809595}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4102935603}) 2024-04-29T14:32:06.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2470380374}) 2024-04-29T14:32:06.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455378423}) 2024-04-29T14:32:06.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559983904}) 2024-04-29T14:32:06.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=460701283}) 2024-04-29T14:32:06.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128800136}) 2024-04-29T14:32:06.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733643390}) 2024-04-29T14:32:06.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=1557093311,mdsmap=2840952586,mgr=2006437882}) 2024-04-29T14:32:06.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=70} crc {mgr=2571415008,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3271244804,mon_config_key=3610619021}) 2024-04-29T14:32:06.388 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:06.429 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:06.516 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:32:06.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:32:06.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:32:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: osdmap e587: 12 total, 12 up, 7 in 2024-04-29T14:32:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: pgmap v1527: 17 pgs: 1 active+remapped+backfilling, 1 peering, 15 active+clean; 7.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 7499/244574 objects misplaced (3.066%); 1.9 MiB/s, 28 objects/s recovering 2024-04-29T14:32:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=2543142174,config=1981513354,health=1997216375,kv=1765154590}) 2024-04-29T14:32:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=922487200,logm=4251823315}) 2024-04-29T14:32:06.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2810114863}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686552495}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1515282198}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1852520498}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994523954}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299021641}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903140785}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2729215579}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343031670}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827283320}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2065720000}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2203548925}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488117310}) 2024-04-29T14:32:06.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2403267985}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603146744}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440867325}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477013805}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528017702}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769924536}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1628678745}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976088586}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1732784061}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3861233568}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656245915}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1552156934}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2087144462}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503185934}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707525264}) 2024-04-29T14:32:06.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2183553525}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060336865}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1316082246}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3849666162}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175698253}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=51829615}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987707212}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028031856}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3931088709}) 2024-04-29T14:32:06.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093779403}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901682753}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962385739}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2911383412}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=645566984}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2365799495}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926929227}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522180241}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399241385}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=469112098}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3356676680}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042566286}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2507583422}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021154841}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1668070417}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889787965}) 2024-04-29T14:32:06.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177617823}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312586013}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851666502}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=306977434}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2811429942}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101528359}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186584258}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730822055}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784809595}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4102935603}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2470380374}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455378423}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559983904}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=460701283}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128800136}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733643390}) 2024-04-29T14:32:06.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=1557093311,mdsmap=2840952586,mgr=2006437882}) 2024-04-29T14:32:06.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=70} crc {mgr=2571415008,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3271244804,mon_config_key=3610619021}) 2024-04-29T14:32:06.576 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:32:06.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:32:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: osdmap e587: 12 total, 12 up, 7 in 2024-04-29T14:32:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: pgmap v1527: 17 pgs: 1 active+remapped+backfilling, 1 peering, 15 active+clean; 7.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 7499/244574 objects misplaced (3.066%); 1.9 MiB/s, 28 objects/s recovering 2024-04-29T14:32:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=2543142174,config=1981513354,health=1997216375,kv=1765154590}) 2024-04-29T14:32:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=922487200,logm=4251823315}) 2024-04-29T14:32:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2810114863}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686552495}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1515282198}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1852520498}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994523954}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299021641}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903140785}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2729215579}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343031670}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827283320}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2065720000}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2203548925}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488117310}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2403267985}) 2024-04-29T14:32:06.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603146744}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440867325}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477013805}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528017702}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769924536}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1628678745}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2976088586}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1732784061}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3861233568}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656245915}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1552156934}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2087144462}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503185934}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707525264}) 2024-04-29T14:32:06.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2183553525}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060336865}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1316082246}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3849666162}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175698253}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=51829615}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987707212}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028031856}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3931088709}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2093779403}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901682753}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962385739}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2911383412}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=645566984}) 2024-04-29T14:32:06.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2365799495}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926929227}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522180241}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399241385}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=469112098}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3356676680}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042566286}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2507583422}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021154841}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1668070417}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889787965}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177617823}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312586013}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851666502}) 2024-04-29T14:32:06.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=306977434}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2811429942}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101528359}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186584258}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730822055}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784809595}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4102935603}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2470380374}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455378423}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559983904}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=460701283}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128800136}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733643390}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=1557093311,mdsmap=2840952586,mgr=2006437882}) 2024-04-29T14:32:06.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=70} crc {mgr=2571415008,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3271244804,mon_config_key=3610619021}) 2024-04-29T14:32:06.623 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:32:06.643 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:32:06.666 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:32:06.682 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:06.686 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:32:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:06.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:06.713 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:32:06.736 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:32:06.757 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:32:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:32:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:32:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:32:06.758 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:32:06.759 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:32:06.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:06.840 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:06.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:06.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:06.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:07.031 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:07.068 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:07.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:07.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:07.204 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:07.337 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:07.353 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:07.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:07.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:07.487 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:07.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2620795658}) 2024-04-29T14:32:07.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1521947411}) 2024-04-29T14:32:07.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=74} crc {mon_config_key=1585855425,monmap=455069592,osd_metadata=809037704,osd_pg_creating=2190768430,osdmap=792955341}) 2024-04-29T14:32:07.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osdmap e588: 12 total, 12 up, 7 in 2024-04-29T14:32:07.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=1824382758}) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2358012711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.689 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:07.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2620795658}) 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1521947411}) 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=74} crc {mon_config_key=1585855425,monmap=455069592,osd_metadata=809037704,osd_pg_creating=2190768430,osdmap=792955341}) 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osdmap e588: 12 total, 12 up, 7 in 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=1824382758}) 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2358012711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.774 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.794 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:07.840 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2620795658}) 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1521947411}) 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=74} crc {mon_config_key=1585855425,monmap=455069592,osd_metadata=809037704,osd_pg_creating=2190768430,osdmap=792955341}) 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osdmap e588: 12 total, 12 up, 7 in 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=1824382758}) 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2358012711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:07.883 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:07.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:07.985 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:08.016 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:08.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:08.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:08.414 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:08.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:08.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:08.525 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:08 smithi040 ceph-mon[39581]: pgmap v1529: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 622 op/s; 14851/242224 objects misplaced (6.131%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:32:08.561 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:08.617 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:08.655 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:08.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:08 smithi194 ceph-mon[44089]: pgmap v1529: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 622 op/s; 14851/242224 objects misplaced (6.131%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:32:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:08.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.16809.0:187097 6.f e588/534)", 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:08.663540+0000", 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0012944339999999999, 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0013185580000000001, 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:08.663 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "queued for pg", 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663540+0000", 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663540+0000", 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663541+0000", 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5039999999999999e-06 2024-04-29T14:32:08.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663548+0000", 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.365e-06 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663560+0000", 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2566e-05 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663568+0000", 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0399999999999993e-06 2024-04-29T14:32:08.665 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:187097 6.f 6:f2be7437:::benchmark_data_smithi194_134202_object34244:head [delete] snapc 0=[] ondisk+write+known_if_redirected e588)", 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:08.655638+0000", 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0091961560000000005, 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0092834030000000008, 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:32:08.666 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 187097 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655638+0000", 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655638+0000", 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655639+0000", 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655642+0000", 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.909e-06 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655648+0000", 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:08.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655654+0000", 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0449999999999997e-06 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655706+0000", 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2008000000000001e-05 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.655874+0000", 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000168543 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:32:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.656015+0000", 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014114100000000001 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:187101 6.f 6:f816140d:::benchmark_data_smithi194_134202_object34248:head [delete] snapc 0=[] ondisk+write+known_if_redirected e588)", 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:08.659016+0000", 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0058181600000000002, 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059561350000000004, 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 187101 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659016+0000", 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659016+0000", 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659019+0000", 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9000000000000002e-06 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659029+0000", 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9010000000000006e-06 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659042+0000", 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2825e-05 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659052+0000", 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0281e-05 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663323+0000", 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042711429999999998 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.663444+0000", 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000121186 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.664117+0000", 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00067282799999999999 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.16809.0:187102 6.1 e588/534 6:83701aee:::benchmark_data_smithi194_134202_object34249:head v 588'11679, mlcod=588'11677)", 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:08.659387+0000", 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0054475979999999997, 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056288229999999998, 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:08.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659387+0000", 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659387+0000", 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659388+0000", 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.432e-06 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659398+0000", 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6299999999999993e-06 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659407+0000", 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8579999999999995e-06 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659415+0000", 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0779999999999998e-06 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659481+0000", 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6178000000000002e-05 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.659502+0000", 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1064e-05 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:187103 6.f 6.bb0741f (undecoded) ondisk+write+known_if_redirected e588)", 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:08.661557+0000", 2024-04-29T14:32:08.677 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0032768710000000002, 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034930460000000001, 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "queued for pg", 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 187103 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.661557+0000", 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.661557+0000", 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.661559+0000", 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0990000000000001e-06 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.661565+0000", 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9050000000000002e-06 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.661579+0000", 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4069e-05 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:08.661589+0000", 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 5 2024-04-29T14:32:08.680 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:32:08.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:08.805 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:08.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:08 smithi121 ceph-mon[44533]: pgmap v1529: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 7.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 622 op/s; 14851/242224 objects misplaced (6.131%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:32:08.917 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:08.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:08.998 INFO:teuthology.orchestra.run.smithi040.stdout:moved 0 / 34 (0%) 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:avg 4.85714 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.12486 -> 1.12486 (expected baseline 2.04041) 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.0 with 4 -> 4 pgs (0.823529 -> 0.823529 * mean) 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.6 with 7 -> 7 pgs (1.44118 -> 1.44118 * mean) 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:oload 1000 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0305 2024-04-29T14:32:08.999 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.3052 2024-04-29T14:32:09.002 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:32:09.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:09.089 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:09.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:09.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:09.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:09.260 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:09.458 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:32:09.468 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:09.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:09.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:09 smithi194 ceph-mon[44089]: from='client.17034 v1:172.21.15.40:0/1004303604' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:32:09.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:09 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:09.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:09 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:09.575 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:09.612 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:09.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:09 smithi040 ceph-mon[39581]: from='client.17034 v1:172.21.15.40:0/1004303604' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:32:09.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:09 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:09.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:09 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:09.645 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:09.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:09.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:09.783 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:09.839 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:09.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:09 smithi121 ceph-mon[44533]: from='client.17034 v1:172.21.15.40:0/1004303604' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:32:09.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:09 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:09.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:09 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:10.033 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:10.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:10.137 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:10.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:32:10.616 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:10.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:10.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:10.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:10 smithi121 ceph-mon[44533]: 6.e scrub starts 2024-04-29T14:32:10.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:10 smithi121 ceph-mon[44533]: osdmap e589: 12 total, 12 up, 7 in 2024-04-29T14:32:10.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:10 smithi121 ceph-mon[44533]: pgmap v1531: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 7.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 518 op/s; 14851/240680 objects misplaced (6.170%); 4.0 MiB/s, 62 objects/s recovering 2024-04-29T14:32:10.788 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:10.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:10 smithi194 ceph-mon[44089]: 6.e scrub starts 2024-04-29T14:32:10.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:10 smithi194 ceph-mon[44089]: osdmap e589: 12 total, 12 up, 7 in 2024-04-29T14:32:10.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:10 smithi194 ceph-mon[44089]: pgmap v1531: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 7.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 518 op/s; 14851/240680 objects misplaced (6.170%); 4.0 MiB/s, 62 objects/s recovering 2024-04-29T14:32:10.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:10.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:10.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:10 smithi040 ceph-mon[39581]: 6.e scrub starts 2024-04-29T14:32:10.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:10 smithi040 ceph-mon[39581]: osdmap e589: 12 total, 12 up, 7 in 2024-04-29T14:32:10.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:10 smithi040 ceph-mon[39581]: pgmap v1531: 17 pgs: 2 active+remapped+backfilling, 1 peering, 14 active+clean; 7.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 518 op/s; 14851/240680 objects misplaced (6.170%); 4.0 MiB/s, 62 objects/s recovering 2024-04-29T14:32:10.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:10.996 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:11.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:11.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:11.157 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:11.193 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:11.248 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:11.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:32:11.567 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:11.601 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:11.637 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:11.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:11.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:11.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2800337129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:11.774 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:11.801 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:32:11.801 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:32:11.801 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:32:11.801 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:32:11.801 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:32:11.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2800337129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:11.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2800337129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:12.026 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:12.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:12.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:12.188 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:12.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:12.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:12.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:12.377 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:12.418 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:12.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:12.520 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:12 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:32:12.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:12 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:32:12.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:12 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:32:12.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2800337129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:12.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:12 smithi121 ceph-mon[44533]: osdmap e590: 12 total, 12 up, 7 in 2024-04-29T14:32:12.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:12 smithi121 ceph-mon[44533]: 6.e scrub ok 2024-04-29T14:32:12.523 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:12 smithi121 ceph-mon[44533]: pgmap v1533: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 13170/233092 objects misplaced (5.650%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:32:12.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:12.585 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:12.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:12.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:32:12.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:12.753 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:12.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:12.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:12.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:12 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:32:12.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2800337129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:12.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:12 smithi194 ceph-mon[44089]: osdmap e590: 12 total, 12 up, 7 in 2024-04-29T14:32:12.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:12 smithi194 ceph-mon[44089]: 6.e scrub ok 2024-04-29T14:32:12.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:12 smithi194 ceph-mon[44089]: pgmap v1533: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 13170/233092 objects misplaced (5.650%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:32:12.866 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:12 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:32:12.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:12 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:32:12.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2800337129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:12.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:12 smithi040 ceph-mon[39581]: osdmap e590: 12 total, 12 up, 7 in 2024-04-29T14:32:12.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:12 smithi040 ceph-mon[39581]: 6.e scrub ok 2024-04-29T14:32:12.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:12 smithi040 ceph-mon[39581]: pgmap v1533: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 13170/233092 objects misplaced (5.650%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:32:12.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:12.953 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:12.974 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:13.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:13.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:13.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:13.191 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:13.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:13.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:13.337 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:13.378 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:13.416 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:13.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:13.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:13.599 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:13.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:13.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:13.782 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:13.881 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:13.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:13.986 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:14.018 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:32:14.026 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:14.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:32:14.064 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:32:14.086 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:32:14.107 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:32:14.135 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:32:14.164 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:32:14.185 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:32:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:32:14.249 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1359239826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:14.249 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:32:14.278 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:32:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:32:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:32:14.280 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:14.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:14.307 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1359239826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:14.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1359239826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:14.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:14.419 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:14.542 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:14.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:14.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:14.696 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:14.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:14.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:14.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:14.839 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:14.849 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:32:14.849 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:32:14.868 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:15.058 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:32:15.058 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:32:15.059 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-29T14:32:15.059 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:32:15.059 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:32:15.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-29T14:32:15.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: pgmap v1534: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 13170/231524 objects misplaced (5.688%); 11 MiB/s, 182 objects/s recovering 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.118 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:15.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:15.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:15.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: pgmap v1534: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 13170/231524 objects misplaced (5.688%); 11 MiB/s, 182 objects/s recovering 2024-04-29T14:32:15.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.300 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.301 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:15.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: pgmap v1534: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 13170/231524 objects misplaced (5.688%); 11 MiB/s, 182 objects/s recovering 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:15.427 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:15.459 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:15.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:32:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:32:15.660 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:15.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:15.706 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:32:15.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:15.798 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:15.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:15.927 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:15.964 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:15.999 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:16.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:16.074 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:16.104 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:16.114 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:16 smithi121 ceph-mon[44533]: 6.7 scrub starts 2024-04-29T14:32:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:16.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:16 smithi194 ceph-mon[44089]: 6.7 scrub starts 2024-04-29T14:32:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:16 smithi040 ceph-mon[39581]: 6.7 scrub starts 2024-04-29T14:32:16.395 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:16.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:16.505 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:16.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:16.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:17.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:32:17.244 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:17.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:17.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:17.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:17 smithi040 ceph-mon[39581]: pgmap v1535: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 12269/228128 objects misplaced (5.378%); 14 MiB/s, 227 objects/s recovering 2024-04-29T14:32:17.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:17 smithi040 ceph-mon[39581]: 6.7 scrub ok 2024-04-29T14:32:17.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:17 smithi121 ceph-mon[44533]: pgmap v1535: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 12269/228128 objects misplaced (5.378%); 14 MiB/s, 227 objects/s recovering 2024-04-29T14:32:17.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:17 smithi121 ceph-mon[44533]: 6.7 scrub ok 2024-04-29T14:32:17.419 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:17.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:17 smithi194 ceph-mon[44089]: pgmap v1535: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 7.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 12269/228128 objects misplaced (5.378%); 14 MiB/s, 227 objects/s recovering 2024-04-29T14:32:17.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:17 smithi194 ceph-mon[44089]: 6.7 scrub ok 2024-04-29T14:32:17.787 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:18.299 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:32:18.323 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:32:18.323 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:32:18.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2415170122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:32:18.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2415170122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:32:18.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2415170122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:32:19.118 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:32:19.506 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:19 smithi194 ceph-mon[44089]: pgmap v1536: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 11624/224168 objects misplaced (5.185%); 16 MiB/s, 258 objects/s recovering 2024-04-29T14:32:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:19 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2415170122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:32:19.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:19 smithi194 ceph-mon[44089]: osdmap e591: 12 total, 12 up, 7 in 2024-04-29T14:32:19.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:19 smithi040 ceph-mon[39581]: pgmap v1536: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 11624/224168 objects misplaced (5.185%); 16 MiB/s, 258 objects/s recovering 2024-04-29T14:32:19.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:19 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:19.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2415170122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:32:19.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:19 smithi040 ceph-mon[39581]: osdmap e591: 12 total, 12 up, 7 in 2024-04-29T14:32:19.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:19 smithi121 ceph-mon[44533]: pgmap v1536: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 11624/224168 objects misplaced (5.185%); 16 MiB/s, 258 objects/s recovering 2024-04-29T14:32:19.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:19 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:19.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2415170122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:32:19.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:19 smithi121 ceph-mon[44533]: osdmap e591: 12 total, 12 up, 7 in 2024-04-29T14:32:19.892 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:19.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:19.999 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:20.130 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:20.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:20.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:20.299 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:20.456 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:32:20.476 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:32:20.500 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:32:20.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:32:20.555 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:32:20.580 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:32:20.609 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:32:20.638 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:32:20.659 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:32:20.681 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:32:20.705 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:32:20.727 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:32:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:32:21.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:32:21.413 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: pgmap v1538: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 11624/222496 objects misplaced (5.224%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:32:21.413 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1668686810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:21.413 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.413 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.414 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: pgmap v1538: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 11624/222496 objects misplaced (5.224%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:32:21.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1668686810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:21.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.604 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:32:21.604 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:32:21.604 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:32:21.604 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:32:21.604 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:32:21.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: pgmap v1538: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 11624/222496 objects misplaced (5.224%); 11 MiB/s, 180 objects/s recovering 2024-04-29T14:32:21.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1668686810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:21.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:21.683 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:22.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:22.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:22.317 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:22.344 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:22.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:22.388 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:32:22.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:22.488 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:22.543 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:22.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:22.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:22.676 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:22.785 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:22.929 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:32:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-29T14:32:22.929 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:32:23.131 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:23.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:23.236 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:23.304 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:23.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:23.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:23 smithi121 ceph-mon[44533]: pgmap v1539: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10831/216908 objects misplaced (4.993%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:32:23.329 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:23 smithi194 ceph-mon[44089]: pgmap v1539: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10831/216908 objects misplaced (4.993%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:32:23.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:23 smithi040 ceph-mon[39581]: pgmap v1539: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10831/216908 objects misplaced (4.993%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:32:23.410 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:23.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:23.567 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:23.600 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:23.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:32:23.634 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:23.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:23.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:23.799 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:23.922 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:24.318 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:32:24.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2553685450' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:24.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:24 smithi040 ceph-mon[39581]: pgmap v1540: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 10104/215136 objects misplaced (4.697%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:32:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2553685450' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:24 smithi121 ceph-mon[44533]: pgmap v1540: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 10104/215136 objects misplaced (4.697%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:32:24.736 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58846 5.8 5:17594c5c:::benchmark_data_smithi194_121512_object58845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.576782+0000", 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.16045965199999, 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20781687600000001, 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.737 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58846 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576782+0000", 2024-04-29T14:32:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576782+0000", 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576783+0000", 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2219999999999999e-06 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576804+0000", 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0959e-05 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576813+0000", 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4419999999999998e-06 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.576820+0000", 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6669999999999999e-06 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.577008+0000", 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000188505 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.577129+0000", 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012085 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.577260+0000", 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000131197 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.578988+0000", 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017283579999999999 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784512+0000", 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20552357499999999 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784549+0000", 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6488e-05 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784599+0000", 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0612999999999998e-05 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.745 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58917 5.8 5:171315a4:::benchmark_data_smithi194_121512_object58916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.613671+0000", 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.12357133799998, 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17105858700000001, 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.746 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58917 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613671+0000", 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613671+0000", 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613673+0000", 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2989999999999999e-06 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613697+0000", 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4136999999999999e-05 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613710+0000", 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3457e-05 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613728+0000", 2024-04-29T14:32:24.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7331000000000001e-05 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613793+0000", 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5303999999999997e-05 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.613942+0000", 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014877400000000001 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.614094+0000", 2024-04-29T14:32:24.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000151958 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.616078+0000", 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019839499999999999 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784637+0000", 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16855897 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784675+0000", 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8547999999999998e-05 2024-04-29T14:32:24.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784729+0000", 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859000000000001e-05 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58929 5.8 5:1ed179d5:::benchmark_data_smithi194_121512_object58928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.620507+0000", 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.116734839, 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16437480500000001, 2024-04-29T14:32:24.751 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58929 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620507+0000", 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620507+0000", 2024-04-29T14:32:24.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620510+0000", 2024-04-29T14:32:24.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3119999999999998e-06 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620533+0000", 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2243999999999999e-05 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620546+0000", 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3226e-05 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620556+0000", 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0149999999999999e-05 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620621+0000", 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4888999999999995e-05 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620769+0000", 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014842999999999999 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.620887+0000", 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011804200000000001 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.622653+0000", 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017655080000000001 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784779+0000", 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16212600799999999 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784828+0000", 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8715999999999999e-05 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.784882+0000", 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4280000000000002e-05 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58948 5.8 5:15d469f1:::benchmark_data_smithi194_121512_object58947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.628765+0000", 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.10847736400001, 2024-04-29T14:32:24.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15817314399999999, 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58948 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628765+0000", 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628765+0000", 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628767+0000", 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6989999999999999e-06 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628790+0000", 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2337999999999999e-05 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628804+0000", 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4782e-05 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628812+0000", 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9459999999999998e-06 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628878+0000", 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5933999999999996e-05 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.628998+0000", 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012002699999999999 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.629092+0000", 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3881000000000005e-05 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.630983+0000", 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001890468 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786808+0000", 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15582528800000001 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786869+0000", 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1112999999999996e-05 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786938+0000", 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8668000000000001e-05 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.762 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58957 5.8 5:19cc827b:::benchmark_data_smithi194_121512_object58956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.636223+0000", 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.10101904700002, 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15196238300000001, 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58957 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.763 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636223+0000", 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636223+0000", 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636226+0000", 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.672e-06 2024-04-29T14:32:24.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636251+0000", 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5738e-05 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636269+0000", 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7452000000000002e-05 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636279+0000", 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.893e-06 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636351+0000", 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1933000000000004e-05 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636484+0000", 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013344799999999999 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.636640+0000", 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015584099999999999 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.638479+0000", 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018393649999999999 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788068+0000", 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14958922399999999 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788123+0000", 2024-04-29T14:32:24.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4959e-05 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788185+0000", 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1858000000000003e-05 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58972 5.8 5:10477e2b:::benchmark_data_smithi194_121512_object58971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.640882+0000", 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.09636035800003, 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14746427200000001, 2024-04-29T14:32:24.768 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58972 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640882+0000", 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640882+0000", 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640884+0000", 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.83e-06 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640937+0000", 2024-04-29T14:32:24.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2292e-05 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640956+0000", 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8941e-05 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.640966+0000", 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0791000000000001e-05 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.641038+0000", 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1482000000000004e-05 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.641183+0000", 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014528100000000001 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.641316+0000", 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013237399999999999 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.643457+0000", 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002141839 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788227+0000", 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.144769433 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788279+0000", 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1786999999999997e-05 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788346+0000", 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7222000000000001e-05 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58992 5.8 5:1acc57d2:::benchmark_data_smithi194_121512_object58991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.654227+0000", 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.08301447000002, 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13512590899999999, 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.774 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58992 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654227+0000", 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654227+0000", 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654230+0000", 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.576e-06 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654283+0000", 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3436999999999997e-05 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654301+0000", 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7632e-05 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654310+0000", 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3540000000000006e-06 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654386+0000", 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5333999999999994e-05 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654532+0000", 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000146582 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.654637+0000", 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000104934 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.656491+0000", 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001853593 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789250+0000", 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.132758986 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789301+0000", 2024-04-29T14:32:24.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0924000000000002e-05 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789353+0000", 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2556999999999998e-05 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58999 5.8 5:156f7ae1:::benchmark_data_smithi194_121512_object58998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.657159+0000", 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.080082651, 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13463576999999999, 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.779 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58999 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657159+0000", 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657159+0000", 2024-04-29T14:32:24.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657162+0000", 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5550000000000001e-06 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657206+0000", 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4434999999999997e-05 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657218+0000", 2024-04-29T14:32:24.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1651000000000001e-05 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657224+0000", 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.981e-06 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657287+0000", 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3028999999999996e-05 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657418+0000", 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000130925 2024-04-29T14:32:24.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.657529+0000", 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011137999999999999 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.659257+0000", 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001727288 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791653+0000", 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13239661699999999 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791685+0000", 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1702000000000002e-05 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791795+0000", 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110207 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58948 5.8 5:15d469f1:::benchmark_data_smithi194_121512_object58947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.06217823100002, 2024-04-29T14:32:24.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.111819161, 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58948 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675064+0000", 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.08e-07 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675093+0000", 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9105999999999999e-05 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675097+0000", 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.625e-06 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675100+0000", 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0259999999999999e-06 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675347+0000", 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00024665700000000001 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:32:24.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675410+0000", 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3100000000000002e-05 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.786883+0000", 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.111473239 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58957 5.8 5:19cc827b:::benchmark_data_smithi194_121512_object58956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.06211915300003, 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.1130111, 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58957 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:32:24.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675123+0000", 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675160+0000", 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6313000000000003e-05 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675166+0000", 2024-04-29T14:32:24.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7560000000000002e-06 2024-04-29T14:32:24.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675171+0000", 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.989e-06 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675443+0000", 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00027163600000000001 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675478+0000", 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4638000000000003e-05 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788134+0000", 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.112656136 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58972 5.8 5:10477e2b:::benchmark_data_smithi194_121512_object58971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675178+0000", 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.06206363699999, 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11311273099999999, 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.793 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58972 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675178+0000", 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675178+0000", 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675179+0000", 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3500000000000002e-07 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675212+0000", 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3343000000000002e-05 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675218+0000", 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4609999999999999e-06 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675223+0000", 2024-04-29T14:32:24.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.502e-06 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675514+0000", 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00029081499999999999 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675578+0000", 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4257000000000005e-05 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.788291+0000", 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.112712918 2024-04-29T14:32:24.796 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58992 5.8 5:1acc57d2:::benchmark_data_smithi194_121512_object58991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675230+0000", 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.06201191500003, 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.114080165, 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58992 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.797 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675230+0000", 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675230+0000", 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675231+0000", 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2699999999999999e-07 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675273+0000", 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2676999999999998e-05 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675279+0000", 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8359999999999997e-06 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675285+0000", 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1410000000000002e-06 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.799 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675622+0000", 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00033625700000000002 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675664+0000", 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2814e-05 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.789310+0000", 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.113645813 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.800 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:58999 5.8 5:156f7ae1:::benchmark_data_smithi194_121512_object58998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675293+0000", 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.06194896599999, 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.116400928, 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for ondisk", 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 58999 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675293+0000", 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675293+0000", 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675294+0000", 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675333+0000", 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9604999999999998e-05 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675338+0000", 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4780000000000002e-06 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675342+0000", 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0260000000000001e-06 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675701+0000", 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00035939899999999999 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for ondisk", 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675770+0000", 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9208999999999994e-05 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.791694+0000", 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.115923555 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59022 5.8 5:1f3c6cca:::benchmark_data_smithi194_121512_object59021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e444)", 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.675348+0000", 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.06189365099999, 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.118538427, 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59022 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675348+0000", 2024-04-29T14:32:24.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675348+0000", 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675349+0000", 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.32e-07 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675379+0000", 2024-04-29T14:32:24.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9879e-05 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675382+0000", 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8020000000000002e-06 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675385+0000", 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0970000000000002e-06 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675797+0000", 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00041133100000000002 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.675906+0000", 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108934 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.676004+0000", 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8628000000000005e-05 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.677972+0000", 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019677330000000002 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793782+0000", 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11581021499999999 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793844+0000", 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1427000000000007e-05 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793887+0000", 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2948999999999998e-05 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59033 5.8 5:1536a9bf:::benchmark_data_smithi194_121512_object59032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.679421+0000", 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.05782095699999, 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11468714200000001, 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59033 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.810 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679421+0000", 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679421+0000", 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679423+0000", 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9599999999999999e-06 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679463+0000", 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9712000000000002e-05 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679474+0000", 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1756999999999999e-05 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679481+0000", 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3629999999999999e-06 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679529+0000", 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8418999999999998e-05 2024-04-29T14:32:24.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679644+0000", 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011482 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.679793+0000", 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014947100000000001 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.683670+0000", 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003876857 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.793955+0000", 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.110284493 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794056+0000", 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010158 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794108+0000", 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1709999999999998e-05 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.814 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59036 5.8 5:18c5f67b:::benchmark_data_smithi194_121512_object59035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.681129+0000", 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.05611265700003, 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11312299000000001, 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59036 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.815 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681129+0000", 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681129+0000", 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681131+0000", 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.497e-06 2024-04-29T14:32:24.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681174+0000", 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3650999999999999e-05 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681188+0000", 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3233e-05 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681195+0000", 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2810000000000003e-06 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681451+0000", 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000255949 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681578+0000", 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012705 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.681714+0000", 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013629199999999999 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.685889+0000", 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041743509999999998 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794162+0000", 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.108273149 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794217+0000", 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5469e-05 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794252+0000", 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5067999999999997e-05 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:59040 5.8 5:1c3d728f:::benchmark_data_smithi194_121512_object59039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e444)", 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:25:01.682661+0000", 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 443.05458103699999, 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.111861925, 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.820 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 59040 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682661+0000", 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682661+0000", 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682663+0000", 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.795e-06 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682713+0000", 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0575000000000003e-05 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682736+0000", 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2339000000000001e-05 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.682744+0000", 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5820000000000002e-06 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.683702+0000", 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00095744000000000005 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.683851+0000", 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014912899999999999 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:32:24.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.684007+0000", 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015646799999999999 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.685983+0000", 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019753290000000001 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794426+0000", 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.108443609 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794478+0000", 2024-04-29T14:32:24.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.223e-05 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:25:01.794523+0000", 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4428999999999998e-05 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:194006 5.d 5:b9c4042c:::benchmark_data_smithi194_121512_object60540:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e495)", 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:27:58.279362+0000", 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 266.45787964800002, 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.115618417, 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.825 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 194006 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279362+0000", 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279362+0000", 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279364+0000", 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8449999999999999e-06 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279369+0000", 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0209999999999999e-06 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279378+0000", 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9740000000000008e-06 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.279384+0000", 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1890000000000004e-06 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.352886+0000", 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.073501739999999996 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.352978+0000", 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1903000000000004e-05 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:32:24.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.353117+0000", 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013911699999999999 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.373664+0000", 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020547267000000001 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.393340+0000", 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019675482000000001 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.393376+0000", 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6297000000000002e-05 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:27:58.394981+0000", 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001604582 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26047.0:235037 5.f e515/467 5:f2be22cd:::benchmark_data_smithi194_121512_object101571:head v 515'15013, mlcod=453'6538)", 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:53.999083+0000", 2024-04-29T14:32:24.830 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 210.738159379, 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11755185, 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999083+0000", 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999083+0000", 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999085+0000", 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1789999999999998e-06 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999093+0000", 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3690000000000007e-06 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999101+0000", 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9500000000000001e-06 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999107+0000", 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8159999999999999e-06 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999155+0000", 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8597999999999998e-05 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:53.999179+0000", 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3328000000000001e-05 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.116589+0000", 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.11740991100000001 2024-04-29T14:32:24.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.116634+0000", 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5698999999999997e-05 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26047.0:235049 5.d 5:b25b9240:::benchmark_data_smithi194_121512_object101583:head [delete] snapc 0=[] ondisk+write+known_if_redirected e515)", 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:28:54.013173+0000", 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 210.724068528, 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.10991333, 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:32:24.834 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26047", 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 235049 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013173+0000", 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013173+0000", 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:32:24.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013176+0000", 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9189999999999999e-06 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.013195+0000", 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9032e-05 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.103738+0000", 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.090542330000000004 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.103750+0000", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2676e-05 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.107286+0000", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035357230000000002 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.107445+0000", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000158715 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 7", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.107594+0000", 2024-04-29T14:32:24.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014871099999999999 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.121692+0000", 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014098589999999999 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.121760+0000", 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7533999999999994e-05 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.121816+0000", 2024-04-29T14:32:24.838 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6462999999999998e-05 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:28:54.123087+0000", 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001270637 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:32:24.839 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:32:24.839 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2553685450' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:24.839 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:24 smithi194 ceph-mon[44089]: pgmap v1540: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 10104/215136 objects misplaced (4.697%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:32:25.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:25 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:25.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2553685450' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:25.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:25 smithi194 ceph-mon[44089]: osdmap e592: 12 total, 12 up, 7 in 2024-04-29T14:32:25.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:32:25.508 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:25 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:25.508 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2553685450' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:25.508 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:25 smithi040 ceph-mon[39581]: osdmap e592: 12 total, 12 up, 7 in 2024-04-29T14:32:25.582 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:25.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:25.594 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:25 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:25.594 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2553685450' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:25.594 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:25 smithi121 ceph-mon[44533]: osdmap e592: 12 total, 12 up, 7 in 2024-04-29T14:32:25.695 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:26.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:26.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:26.351 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:26.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:26 smithi040 ceph-mon[39581]: pgmap v1542: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 9375/209872 objects misplaced (4.467%); 13 MiB/s, 203 objects/s recovering 2024-04-29T14:32:26.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:26 smithi121 ceph-mon[44533]: pgmap v1542: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 9375/209872 objects misplaced (4.467%); 13 MiB/s, 203 objects/s recovering 2024-04-29T14:32:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:26.722 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:32:26.744 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:32:26.764 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:32:26.789 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:32:26.817 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:32:26.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:26 smithi194 ceph-mon[44089]: pgmap v1542: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 893 op/s; 9375/209872 objects misplaced (4.467%); 13 MiB/s, 203 objects/s recovering 2024-04-29T14:32:26.849 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:32:26.872 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:32:26.895 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:32:26.916 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:32:26.937 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:32:26.965 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:32:26.993 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:32:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:32:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:32:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:32:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:32:27.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:27.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:27.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:27.309 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:27.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:27.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osdmap e593: 12 total, 12 up, 7 in 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2189418858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osdmap e593: 12 total, 12 up, 7 in 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2189418858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.483 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:27.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:27.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:27.741 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:27.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osdmap e593: 12 total, 12 up, 7 in 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2189418858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:27.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:27.885 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:27.906 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:27.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:28.013 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:28.048 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:28.073 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:28.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:28.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:32:28.178 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:28.212 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:28.240 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:28.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:28.348 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:28.379 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:28.398 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:28.399 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.399 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.399 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.399 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:28 smithi040 ceph-mon[39581]: pgmap v1544: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 8675/206026 objects misplaced (4.211%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:32:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:28.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:28.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:28.544 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.544 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.544 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.544 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:28 smithi194 ceph-mon[44089]: pgmap v1544: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 8675/206026 objects misplaced (4.211%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:32:28.547 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:28.568 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:28.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:28.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:28 smithi121 ceph-mon[44533]: pgmap v1544: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 8675/206026 objects misplaced (4.211%); 16 MiB/s, 263 objects/s recovering 2024-04-29T14:32:28.671 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:28.717 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:28.748 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:28.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:28.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:28.922 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:28.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:28.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:29.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:29.110 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:29.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:29.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:29.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:29.286 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:29.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:29.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:29.390 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:32:29.390 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:32:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:29.434 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:29.461 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:29.488 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:29.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:29.594 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:29.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:29.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:29.623 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:29.624 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:29.624 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:29.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:29.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:29.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:29.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:29.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:29.798 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:29.827 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:29.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:29.936 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:29.966 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:29.990 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:29.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:30.095 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:30.122 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:30.149 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:30.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:30.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:32:30.254 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:30.297 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:30.321 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:30.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:30.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:30.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:30 smithi040 ceph-mon[39581]: osdmap e594: 12 total, 12 up, 7 in 2024-04-29T14:32:30.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:30 smithi040 ceph-mon[39581]: pgmap v1546: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 8675/204268 objects misplaced (4.247%); 11 MiB/s, 169 objects/s recovering 2024-04-29T14:32:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:30.537 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:30.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:30.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:30 smithi194 ceph-mon[44089]: osdmap e594: 12 total, 12 up, 7 in 2024-04-29T14:32:30.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:30 smithi194 ceph-mon[44089]: pgmap v1546: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 8675/204268 objects misplaced (4.247%); 11 MiB/s, 169 objects/s recovering 2024-04-29T14:32:30.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:30 smithi121 ceph-mon[44533]: osdmap e594: 12 total, 12 up, 7 in 2024-04-29T14:32:30.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:30 smithi121 ceph-mon[44533]: pgmap v1546: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 8675/204268 objects misplaced (4.247%); 11 MiB/s, 169 objects/s recovering 2024-04-29T14:32:30.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:30.684 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:30.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:30.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:30.736 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:32:30.836 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:30.870 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:30.888 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:30.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:30.994 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:31.031 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:31.054 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:31.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:31.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:31.163 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:31.189 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:31.215 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:31.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:31.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:31.393 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:31.418 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:31.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:31.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:31.595 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:31.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:31.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:31.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:31.976 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:31.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:32.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:32.108 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:32.147 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:32:32.147 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:32:32.147 INFO:tasks.thrashosds.thrasher:Reweighting osd 11 to 0.42977336161757407 2024-04-29T14:32:32.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 11 0.42977336161757407 2024-04-29T14:32:32.412 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:32.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:32.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:32.628 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:32.662 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:32.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:32.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:32.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:32.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:32.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:33.015 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:33.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:33.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:33.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:33.225 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:33.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:33.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:33.372 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:33.407 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:33.433 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:33.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:33.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:33.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:33 smithi194 ceph-mon[44089]: pgmap v1547: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 974 op/s; 7855/198386 objects misplaced (3.959%); 13 MiB/s, 215 objects/s recovering 2024-04-29T14:32:33.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:33 smithi040 ceph-mon[39581]: pgmap v1547: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 974 op/s; 7855/198386 objects misplaced (3.959%); 13 MiB/s, 215 objects/s recovering 2024-04-29T14:32:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:33.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:33 smithi121 ceph-mon[44533]: pgmap v1547: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 974 op/s; 7855/198386 objects misplaced (3.959%); 13 MiB/s, 215 objects/s recovering 2024-04-29T14:32:33.820 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:33.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:33.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:33.974 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:34.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:34.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:34.125 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:34.167 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:34.193 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:34.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:34.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:34.332 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:34.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:34.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:34.496 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:34.523 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:34.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:34.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:34.735 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:34.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:34.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:34.887 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:34.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:34.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:35.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:35.073 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:35.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:35 smithi121 ceph-mon[44533]: pgmap v1548: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 7170/196558 objects misplaced (3.648%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:32:35.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2740835991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:32:35.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1969972790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:35.354 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:32:35.354 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:32:35.354 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:32:35.355 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:32:35.355 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:32:35.355 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:32:35.355 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:32:35.385 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:32:35.406 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:32:35.437 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:32:35.462 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:32:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:32:35.486 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:32:35.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:32:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:32:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:35 smithi194 ceph-mon[44089]: pgmap v1548: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 7170/196558 objects misplaced (3.648%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:32:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2740835991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:32:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1969972790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:35.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:35 smithi040 ceph-mon[39581]: pgmap v1548: 17 pgs: 2 active+remapped+backfilling, 15 active+clean; 6.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 7170/196558 objects misplaced (3.648%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:32:35.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2740835991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:32:35.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1969972790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:36.039 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:32:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:32:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:32:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:32:36.040 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:32:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:36.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:36.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:36.406 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:36.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:36.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:36.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:36.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2740835991' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osdmap e595: 12 total, 12 up, 7 in 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: 6.4 scrub starts 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:36.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2740835991' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osdmap e595: 12 total, 12 up, 7 in 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: 6.4 scrub starts 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.610 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:36.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:36.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2740835991' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:32:36.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osdmap e595: 12 total, 12 up, 7 in 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: 6.4 scrub starts 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:36.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:36.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:36.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:36.801 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:36.836 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:36.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:36.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:37.003 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:37.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:37.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:37.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:37.172 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:37.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:37.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:37.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:37 smithi121 ceph-mon[44533]: pgmap v1550: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 6373/191104 objects misplaced (3.335%); 14 MiB/s, 218 objects/s recovering 2024-04-29T14:32:37.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:37 smithi121 ceph-mon[44533]: 6.4 scrub ok 2024-04-29T14:32:37.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:37 smithi121 ceph-mon[44533]: 6.2 scrub starts 2024-04-29T14:32:37.351 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:37.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:37 smithi040 ceph-mon[39581]: pgmap v1550: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 6373/191104 objects misplaced (3.335%); 14 MiB/s, 218 objects/s recovering 2024-04-29T14:32:37.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:37 smithi040 ceph-mon[39581]: 6.4 scrub ok 2024-04-29T14:32:37.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:37 smithi040 ceph-mon[39581]: 6.2 scrub starts 2024-04-29T14:32:37.702 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:37.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:37.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:37.848 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:37 smithi194 ceph-mon[44089]: pgmap v1550: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 6373/191104 objects misplaced (3.335%); 14 MiB/s, 218 objects/s recovering 2024-04-29T14:32:37.848 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:37 smithi194 ceph-mon[44089]: 6.4 scrub ok 2024-04-29T14:32:37.848 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:37 smithi194 ceph-mon[44089]: 6.2 scrub starts 2024-04-29T14:32:37.879 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:37.908 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:37.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:38.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:38.044 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:38.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:38.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:38.258 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:38.301 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:38.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:38.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:38.434 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:38.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:38 smithi040 ceph-mon[39581]: 6.2 scrub ok 2024-04-29T14:32:38.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:38 smithi040 ceph-mon[39581]: pgmap v1551: 17 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 13 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 997 op/s; 5722/186918 objects misplaced (3.061%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:32:38.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:38 smithi121 ceph-mon[44533]: 6.2 scrub ok 2024-04-29T14:32:38.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:38 smithi121 ceph-mon[44533]: pgmap v1551: 17 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 13 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 997 op/s; 5722/186918 objects misplaced (3.061%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:32:38.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:32:38.784 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:38.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:38.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:32:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:38 smithi194 ceph-mon[44089]: 6.2 scrub ok 2024-04-29T14:32:38.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:38 smithi194 ceph-mon[44089]: pgmap v1551: 17 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 13 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 997 op/s; 5722/186918 objects misplaced (3.061%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:32:38.888 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:38.921 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:38.946 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:38.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:39.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:39.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:39.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:39.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:39.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:39.346 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:39.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:39.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:39.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:39.528 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:39.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:32:39.726 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:39.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:39.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:39.860 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:40.613 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:40.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:40.648 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:40.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:40.750 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:40.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:40.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:41.095 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.11 to 0.429773 (6e05) 2024-04-29T14:32:41.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:41 smithi040 ceph-mon[39581]: 6.6 scrub starts 2024-04-29T14:32:41.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:41 smithi040 ceph-mon[39581]: pgmap v1552: 17 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 13 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 5722/185208 objects misplaced (3.089%); 14 MiB/s, 222 objects/s recovering 2024-04-29T14:32:41.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2473392152' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 0.42977336161757407}]: dispatch 2024-04-29T14:32:41.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:41 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:41.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:41 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:41.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:41 smithi121 ceph-mon[44533]: 6.6 scrub starts 2024-04-29T14:32:41.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:41 smithi121 ceph-mon[44533]: pgmap v1552: 17 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 13 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 5722/185208 objects misplaced (3.089%); 14 MiB/s, 222 objects/s recovering 2024-04-29T14:32:41.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2473392152' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 0.42977336161757407}]: dispatch 2024-04-29T14:32:41.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:41 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:41.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:41 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:41.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:41 smithi194 ceph-mon[44089]: 6.6 scrub starts 2024-04-29T14:32:41.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:41 smithi194 ceph-mon[44089]: pgmap v1552: 17 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfilling, 13 active+clean; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 5722/185208 objects misplaced (3.089%); 14 MiB/s, 222 objects/s recovering 2024-04-29T14:32:41.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2473392152' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 0.42977336161757407}]: dispatch 2024-04-29T14:32:41.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:41 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:41.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:41 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:42.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:42.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:42.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2473392152' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 0.42977336161757407}]': finished 2024-04-29T14:32:42.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:42 smithi040 ceph-mon[39581]: osdmap e596: 12 total, 12 up, 7 in 2024-04-29T14:32:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:42.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:42.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:42.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:42.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2473392152' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 0.42977336161757407}]': finished 2024-04-29T14:32:42.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:42 smithi121 ceph-mon[44533]: osdmap e596: 12 total, 12 up, 7 in 2024-04-29T14:32:42.471 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:42.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:42.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:42.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2473392152' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 0.42977336161757407}]': finished 2024-04-29T14:32:42.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:42 smithi194 ceph-mon[44089]: osdmap e596: 12 total, 12 up, 7 in 2024-04-29T14:32:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:42.543 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:42.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:42.647 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:42.674 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:42.693 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:42.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:42.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:42.831 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:42.886 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:42.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:42.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:43.019 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:43.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:43 smithi040 ceph-mon[39581]: 6.6 scrub ok 2024-04-29T14:32:43.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:43 smithi040 ceph-mon[39581]: pgmap v1554: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 5.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 4970/179824 objects misplaced (2.764%); 14 MiB/s, 220 objects/s recovering 2024-04-29T14:32:43.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:43 smithi040 ceph-mon[39581]: osdmap e597: 12 total, 12 up, 7 in 2024-04-29T14:32:43.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:43 smithi121 ceph-mon[44533]: 6.6 scrub ok 2024-04-29T14:32:43.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:43 smithi121 ceph-mon[44533]: pgmap v1554: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 5.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 4970/179824 objects misplaced (2.764%); 14 MiB/s, 220 objects/s recovering 2024-04-29T14:32:43.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:43 smithi121 ceph-mon[44533]: osdmap e597: 12 total, 12 up, 7 in 2024-04-29T14:32:43.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:43 smithi194 ceph-mon[44089]: 6.6 scrub ok 2024-04-29T14:32:43.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:43 smithi194 ceph-mon[44089]: pgmap v1554: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 5.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 4970/179824 objects misplaced (2.764%); 14 MiB/s, 220 objects/s recovering 2024-04-29T14:32:43.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:43 smithi194 ceph-mon[44089]: osdmap e597: 12 total, 12 up, 7 in 2024-04-29T14:32:44.131 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:32:44.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:44 smithi040 ceph-mon[39581]: osdmap e598: 12 total, 12 up, 7 in 2024-04-29T14:32:44.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:44 smithi040 ceph-mon[39581]: 6.2 starting backfill to osd.9 from (0'0,0'0] MAX to 595'13617 2024-04-29T14:32:44.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:44 smithi040 ceph-mon[39581]: 6.4 starting backfill to osd.6 from (0'0,0'0] MAX to 595'13493 2024-04-29T14:32:44.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1888936039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:44.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:44 smithi121 ceph-mon[44533]: osdmap e598: 12 total, 12 up, 7 in 2024-04-29T14:32:44.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:44 smithi121 ceph-mon[44533]: 6.2 starting backfill to osd.9 from (0'0,0'0] MAX to 595'13617 2024-04-29T14:32:44.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:44 smithi121 ceph-mon[44533]: 6.4 starting backfill to osd.6 from (0'0,0'0] MAX to 595'13493 2024-04-29T14:32:44.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1888936039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:44.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:44 smithi194 ceph-mon[44089]: osdmap e598: 12 total, 12 up, 7 in 2024-04-29T14:32:44.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:44 smithi194 ceph-mon[44089]: 6.2 starting backfill to osd.9 from (0'0,0'0] MAX to 595'13617 2024-04-29T14:32:44.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:44 smithi194 ceph-mon[44089]: 6.4 starting backfill to osd.6 from (0'0,0'0] MAX to 595'13493 2024-04-29T14:32:44.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1888936039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:32:44.870 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:44.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:44.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:45.007 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:45.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:45 smithi040 ceph-mon[39581]: pgmap v1557: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 3957/178326 objects misplaced (2.219%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:32:45.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:45 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:32:45.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:45 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:32:45.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1888936039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:45.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:45 smithi040 ceph-mon[39581]: osdmap e599: 12 total, 12 up, 7 in 2024-04-29T14:32:45.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:45 smithi121 ceph-mon[44533]: pgmap v1557: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 3957/178326 objects misplaced (2.219%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:32:45.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:45 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:32:45.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:45 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:32:45.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1888936039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:45.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:45 smithi121 ceph-mon[44533]: osdmap e599: 12 total, 12 up, 7 in 2024-04-29T14:32:45.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:45 smithi194 ceph-mon[44089]: pgmap v1557: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 3957/178326 objects misplaced (2.219%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:32:45.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:45 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:32:45.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:45 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:32:45.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1888936039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:32:45.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:45 smithi194 ceph-mon[44089]: osdmap e599: 12 total, 12 up, 7 in 2024-04-29T14:32:45.857 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:45.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:45.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:46.018 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:46.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:46.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:46.176 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:46.205 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:46.231 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:46.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:46.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:46.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:46.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:46.374 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:46.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:46.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:46.476 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:46.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:46.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:46.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:46.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:46.615 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:32:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:32:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:32:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T14:32:46.615 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:32:46.640 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:46.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:46.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:46.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:46.855 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:46.873 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:32:46.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:46.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:46.894 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:32:46.914 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:32:46.935 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:32:46.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:32:46.977 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:32:46.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:47.014 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:32:47.038 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:32:47.060 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:32:47.065 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:47.083 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:32:47.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:47.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:47.104 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:32:47.125 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:32:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:32:47.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:32:47.156 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:47 smithi121 ceph-mon[44533]: pgmap v1559: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 7783/175318 objects misplaced (4.439%); 30 MiB/s, 481 objects/s recovering 2024-04-29T14:32:47.156 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:47 smithi121 ceph-mon[44533]: osdmap e600: 12 total, 12 up, 7 in 2024-04-29T14:32:47.156 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3286443485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:47.156 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:47.156 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:47.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:47.260 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:47.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:47.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:47.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:47 smithi040 ceph-mon[39581]: pgmap v1559: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 7783/175318 objects misplaced (4.439%); 30 MiB/s, 481 objects/s recovering 2024-04-29T14:32:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:47 smithi040 ceph-mon[39581]: osdmap e600: 12 total, 12 up, 7 in 2024-04-29T14:32:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3286443485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:47.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:47 smithi194 ceph-mon[44089]: pgmap v1559: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 7783/175318 objects misplaced (4.439%); 30 MiB/s, 481 objects/s recovering 2024-04-29T14:32:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:47 smithi194 ceph-mon[44089]: osdmap e600: 12 total, 12 up, 7 in 2024-04-29T14:32:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3286443485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:47.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:47.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:47.665 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:32:47.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:47.807 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:47.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:47.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:47.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:32:47.943 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:47.973 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:48.129 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.463 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.464 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:48.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:48.601 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:48.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:48.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:48.692 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:48.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:48.846 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:48.876 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:48.959 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:48.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:49.063 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:49.101 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:49.232 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:49.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:49.340 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:49.376 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:49.397 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:49.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:49.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:49.558 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:49 smithi121 ceph-mon[44533]: pgmap v1561: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 498 op/s; 11535/172404 objects misplaced (6.691%); 32 MiB/s, 507 objects/s recovering 2024-04-29T14:32:49.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:49 smithi040 ceph-mon[39581]: pgmap v1561: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 498 op/s; 11535/172404 objects misplaced (6.691%); 32 MiB/s, 507 objects/s recovering 2024-04-29T14:32:49.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:49 smithi194 ceph-mon[44089]: pgmap v1561: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 498 op/s; 11535/172404 objects misplaced (6.691%); 32 MiB/s, 507 objects/s recovering 2024-04-29T14:32:49.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:49.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:49.695 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:32:49.725 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:49.746 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:32:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:49.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:32:49.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:32:49.854 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:32:49.854 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:32:49.854 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.5530548560514451 2024-04-29T14:32:49.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 4 0.5530548560514451 2024-04-29T14:32:49.887 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:49.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:32:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:49.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:32:50.015 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:50.050 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:50.081 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:50.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:50.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:50.218 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:50.326 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:50.326 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:50.326 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: osdmap e601: 12 total, 12 up, 7 in 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: 6.f scrub starts 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:32:50.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:50 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.421 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:50.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:50.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:50.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: osdmap e601: 12 total, 12 up, 7 in 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: 6.f scrub starts 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:32:50.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:50 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:50.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:50.466 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:50.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:32:50.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:50.583 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:50.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:50.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:50.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:50.736 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:32:50.736 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:32:50.748 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:50.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:50.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:50.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: osdmap e601: 12 total, 12 up, 7 in 2024-04-29T14:32:50.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:50.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:50.767 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: 6.f scrub starts 2024-04-29T14:32:50.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:50.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:32:50.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:32:50.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:32:50.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:50 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:50.787 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:50.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:50.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:50.926 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:51.031 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:51.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:51.136 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:32:51.164 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:32:51.362 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:32:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:51.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:32:51.370 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:51.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:51.498 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:51.541 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:32:51.541 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:32:51.541 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: 6.f scrub ok 2024-04-29T14:32:51.541 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: osdmap e602: 12 total, 12 up, 7 in 2024-04-29T14:32:51.541 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.541 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4268844333' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:51.542 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.575 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:32:51.596 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:51.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: 6.f scrub ok 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: osdmap e602: 12 total, 12 up, 7 in 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4268844333' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:51.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:51.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:51.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:51.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:51.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:32:51.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:32:51.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: 6.f scrub ok 2024-04-29T14:32:51.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: osdmap e602: 12 total, 12 up, 7 in 2024-04-29T14:32:51.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4268844333' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:51.701 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:51.741 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:51.777 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:51.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:51.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:51.923 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:51.953 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:51.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:51.961 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:52.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:52.090 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:52.188 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:52.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:52.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:52.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:52 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:52.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4268844333' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:32:52.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:52 smithi121 ceph-mon[44533]: osdmap e603: 12 total, 12 up, 7 in 2024-04-29T14:32:52.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:52 smithi121 ceph-mon[44533]: pgmap v1566: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 5.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 9020/166246 objects misplaced (5.426%); 38 MiB/s, 612 objects/s recovering 2024-04-29T14:32:52.359 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:52.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:52 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:52.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4268844333' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:32:52.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:52 smithi040 ceph-mon[39581]: osdmap e603: 12 total, 12 up, 7 in 2024-04-29T14:32:52.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:52 smithi040 ceph-mon[39581]: pgmap v1566: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 5.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 9020/166246 objects misplaced (5.426%); 38 MiB/s, 612 objects/s recovering 2024-04-29T14:32:52.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:52.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:52.504 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:52.531 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:52 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:32:52.531 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4268844333' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:32:52.531 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:52 smithi194 ceph-mon[44089]: osdmap e603: 12 total, 12 up, 7 in 2024-04-29T14:32:52.531 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:52 smithi194 ceph-mon[44089]: pgmap v1566: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 5.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 9020/166246 objects misplaced (5.426%); 38 MiB/s, 612 objects/s recovering 2024-04-29T14:32:52.544 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:52.904 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:52.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:53.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:53.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:54.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:54.123 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:54.132 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:32:54.132 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:32:54.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:32:54.986 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:32:55.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:55 smithi040 ceph-mon[39581]: pgmap v1567: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 8200/164816 objects misplaced (4.975%); 30 MiB/s, 481 objects/s recovering 2024-04-29T14:32:55.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:55 smithi194 ceph-mon[44089]: pgmap v1567: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 8200/164816 objects misplaced (4.975%); 30 MiB/s, 481 objects/s recovering 2024-04-29T14:32:55.348 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:32:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:55.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:55.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:55 smithi121 ceph-mon[44533]: pgmap v1567: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 8200/164816 objects misplaced (4.975%); 30 MiB/s, 481 objects/s recovering 2024-04-29T14:32:55.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:32:55.505 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:55.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:32:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:55.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:32:55.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:55.688 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:55.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:55.792 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:55.823 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:32:55.823 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:55.842 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:32:55.862 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:32:55.881 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:32:55.901 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:32:55.923 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:32:55.945 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:32:55.948 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:32:55.966 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:32:55.990 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:32:56.010 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:32:56.030 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:32:56.050 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:32:56.052 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:32:56.123 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.4 to 0.553055 (8d95) 2024-04-29T14:32:56.174 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:56.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:56.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:56.345 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:56.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:56.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:56 smithi040 ceph-mon[39581]: osdmap e604: 12 total, 12 up, 7 in 2024-04-29T14:32:56.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2539623204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:56.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/582403810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5530548560514451}]: dispatch 2024-04-29T14:32:56.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:56 smithi121 ceph-mon[44533]: osdmap e604: 12 total, 12 up, 7 in 2024-04-29T14:32:56.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2539623204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:56.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/582403810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5530548560514451}]: dispatch 2024-04-29T14:32:56.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.454 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:56.478 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:56 smithi194 ceph-mon[44089]: osdmap e604: 12 total, 12 up, 7 in 2024-04-29T14:32:56.479 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2539623204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:32:56.479 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.479 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/582403810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.5530548560514451}]: dispatch 2024-04-29T14:32:56.479 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.479 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:56.481 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:56.729 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:56.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:56.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:56.861 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:57.060 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:57.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:57.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:57.197 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:57.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:57.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: pgmap v1569: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 926 op/s; 6855/159722 objects misplaced (4.292%); 42 MiB/s, 668 objects/s recovering 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/582403810' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.5530548560514451}]': finished 2024-04-29T14:32:57.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:57 smithi194 ceph-mon[44089]: osdmap e605: 12 total, 12 up, 7 in 2024-04-29T14:32:57.342 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:32:57.343 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:32:57.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:32:57.343 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [] 2024-04-29T14:32:57.343 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:32:57.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: pgmap v1569: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 926 op/s; 6855/159722 objects misplaced (4.292%); 42 MiB/s, 668 objects/s recovering 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/582403810' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.5530548560514451}]': finished 2024-04-29T14:32:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:57 smithi040 ceph-mon[39581]: osdmap e605: 12 total, 12 up, 7 in 2024-04-29T14:32:57.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:57.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: pgmap v1569: 17 pgs: 1 peering, 2 active+remapped+backfilling, 14 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 926 op/s; 6855/159722 objects misplaced (4.292%); 42 MiB/s, 668 objects/s recovering 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/582403810' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.5530548560514451}]': finished 2024-04-29T14:32:57.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:57 smithi121 ceph-mon[44533]: osdmap e605: 12 total, 12 up, 7 in 2024-04-29T14:32:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:57.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:57.708 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:32:57.739 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:57.769 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:32:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:57.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:32:57.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:58.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:32:58.258 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:58.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:58.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:32:58.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:58 smithi121 ceph-mon[44533]: 6.2 continuing backfill to osd.9 from (549'4200,604'14231] 6:47a476dc:::benchmark_data_smithi194_134202_object150245:head to 604'14231 2024-04-29T14:32:58.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:58 smithi121 ceph-mon[44533]: 6.2 starting backfill to osd.10 from (0'0,0'0] MAX to 604'14231 2024-04-29T14:32:58.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:58 smithi121 ceph-mon[44533]: osdmap e606: 12 total, 12 up, 7 in 2024-04-29T14:32:58.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:58 smithi121 ceph-mon[44533]: osdmap e607: 12 total, 12 up, 7 in 2024-04-29T14:32:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:58.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:32:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:58.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:32:58.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:58 smithi040 ceph-mon[39581]: 6.2 continuing backfill to osd.9 from (549'4200,604'14231] 6:47a476dc:::benchmark_data_smithi194_134202_object150245:head to 604'14231 2024-04-29T14:32:58.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:58 smithi040 ceph-mon[39581]: 6.2 starting backfill to osd.10 from (0'0,0'0] MAX to 604'14231 2024-04-29T14:32:58.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:58 smithi040 ceph-mon[39581]: osdmap e606: 12 total, 12 up, 7 in 2024-04-29T14:32:58.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:58 smithi040 ceph-mon[39581]: osdmap e607: 12 total, 12 up, 7 in 2024-04-29T14:32:58.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:58 smithi194 ceph-mon[44089]: 6.2 continuing backfill to osd.9 from (549'4200,604'14231] 6:47a476dc:::benchmark_data_smithi194_134202_object150245:head to 604'14231 2024-04-29T14:32:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:58 smithi194 ceph-mon[44089]: 6.2 starting backfill to osd.10 from (0'0,0'0] MAX to 604'14231 2024-04-29T14:32:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:58 smithi194 ceph-mon[44089]: osdmap e606: 12 total, 12 up, 7 in 2024-04-29T14:32:58.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:58 smithi194 ceph-mon[44089]: osdmap e607: 12 total, 12 up, 7 in 2024-04-29T14:32:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:58.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:58.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:58.656 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:32:58.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:32:58.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:58.838 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:32:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:58.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:32:58.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:32:59.013 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:59.053 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:32:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:59.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:32:59.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:32:59.206 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:59.237 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:32:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:59.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:32:59.344 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:32:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 604'14076 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: 6.6 starting backfill to osd.9 from (0'0,0'0] MAX to 604'14301 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:59.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:59.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:32:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 604'14076 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: 6.6 starting backfill to osd.9 from (0'0,0'0] MAX to 604'14301 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:59.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:32:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:59.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:32:59.499 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:59.528 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:59.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:59.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 604'14076 2024-04-29T14:32:59.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: 6.6 starting backfill to osd.9 from (0'0,0'0] MAX to 604'14301 2024-04-29T14:32:59.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:32:59.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:32:59.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:32:59.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:32:59.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:32:59.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:32:59.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:32:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:32:59.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:32:59.677 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:32:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:59.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:32:59.802 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:32:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:32:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:32:59.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:32:59.900 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:32:59.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:32:59.984 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:00.243 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2506571125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:00.244 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:00 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:00.244 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2506571125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:00.244 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:00 smithi040 ceph-mon[39581]: osdmap e608: 12 total, 12 up, 7 in 2024-04-29T14:33:00.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2506571125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:00.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:00 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:00.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2506571125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:00.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:00 smithi194 ceph-mon[44089]: osdmap e608: 12 total, 12 up, 7 in 2024-04-29T14:33:00.597 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2506571125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:00.597 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:00 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:00.597 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2506571125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:00.597 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:00 smithi121 ceph-mon[44533]: osdmap e608: 12 total, 12 up, 7 in 2024-04-29T14:33:00.733 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:00.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:00.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:00.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:00.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:01.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:01.066 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:01.087 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:01.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:01.137 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:33:01.137 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:33:01.137 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:33:01.137 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:33:01.149 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:01 smithi121 ceph-mon[44533]: pgmap v1575: 17 pgs: 1 remapped+peering, 1 peering, 2 unknown, 1 active+remapped+backfilling, 12 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 372 op/s; 3169/156738 objects misplaced (2.022%); 15 MiB/s, 245 objects/s recovering 2024-04-29T14:33:01.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:33:01.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:01.227 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:01.263 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:01.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:01.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:01 smithi040 ceph-mon[39581]: pgmap v1575: 17 pgs: 1 remapped+peering, 1 peering, 2 unknown, 1 active+remapped+backfilling, 12 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 372 op/s; 3169/156738 objects misplaced (2.022%); 15 MiB/s, 245 objects/s recovering 2024-04-29T14:33:01.369 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:01.402 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:01.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:01.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:01.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:01 smithi194 ceph-mon[44089]: pgmap v1575: 17 pgs: 1 remapped+peering, 1 peering, 2 unknown, 1 active+remapped+backfilling, 12 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 372 op/s; 3169/156738 objects misplaced (2.022%); 15 MiB/s, 245 objects/s recovering 2024-04-29T14:33:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:01.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:01.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:01.874 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:01.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:33:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:01.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:02.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:02.110 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:02.140 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:33:02.140 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:33:02.141 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-04-29T14:33:02.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:33:02.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:02.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:02.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:02.284 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:02.307 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:02.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:02.392 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:33:02.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:02.477 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:02.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:02.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:02.620 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:02.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:33:02.970 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:02.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:03.009 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:03.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:03.104 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:03.237 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:03.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:03.313 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:03 smithi121 ceph-mon[44533]: pgmap v1576: 17 pgs: 2 unknown, 2 active+remapped+backfilling, 13 active+clean; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 479 op/s; 7924/154048 objects misplaced (5.144%); 35 MiB/s, 557 objects/s recovering 2024-04-29T14:33:03.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:03.413 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:03.500 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:03 smithi040 ceph-mon[39581]: pgmap v1576: 17 pgs: 2 unknown, 2 active+remapped+backfilling, 13 active+clean; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 479 op/s; 7924/154048 objects misplaced (5.144%); 35 MiB/s, 557 objects/s recovering 2024-04-29T14:33:03.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:03 smithi194 ceph-mon[44089]: pgmap v1576: 17 pgs: 2 unknown, 2 active+remapped+backfilling, 13 active+clean; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 479 op/s; 7924/154048 objects misplaced (5.144%); 35 MiB/s, 557 objects/s recovering 2024-04-29T14:33:03.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:03.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:03.897 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:03.944 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:03.971 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:03.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:04.076 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:04.136 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:04.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:33:04.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:33:04.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:33:04.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:33:04.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:33:04.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:33:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:33:04.159 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:33:04.159 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:33:04.159 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:33:04.159 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:33:04.163 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:04.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:04.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:04.306 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:04.333 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:04.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:04.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:04.503 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:33:04.503 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:33:04.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/13198092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:04.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/13198092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:04.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/13198092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:04.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:04.922 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:04.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:05.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:05.055 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:05.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:33:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:05.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:05.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:05.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: pgmap v1577: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 16746/152350 objects misplaced (10.992%); 26 MiB/s, 423 objects/s recovering 2024-04-29T14:33:05.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:05.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:05.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:05 smithi040 ceph-mon[39581]: osdmap e609: 12 total, 12 up, 7 in 2024-04-29T14:33:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: pgmap v1577: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 16746/152350 objects misplaced (10.992%); 26 MiB/s, 423 objects/s recovering 2024-04-29T14:33:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:05 smithi121 ceph-mon[44533]: osdmap e609: 12 total, 12 up, 7 in 2024-04-29T14:33:05.699 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:33:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: pgmap v1577: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 16746/152350 objects misplaced (10.992%); 26 MiB/s, 423 objects/s recovering 2024-04-29T14:33:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:05.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:05.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:05 smithi194 ceph-mon[44089]: osdmap e609: 12 total, 12 up, 7 in 2024-04-29T14:33:06.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:06 smithi040 ceph-mon[39581]: pgmap v1579: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 674 op/s; 16216/146688 objects misplaced (11.055%); 27 MiB/s, 428 objects/s recovering 2024-04-29T14:33:06.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:06 smithi121 ceph-mon[44533]: pgmap v1579: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 674 op/s; 16216/146688 objects misplaced (11.055%); 27 MiB/s, 428 objects/s recovering 2024-04-29T14:33:06.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:06 smithi194 ceph-mon[44089]: pgmap v1579: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 674 op/s; 16216/146688 objects misplaced (11.055%); 27 MiB/s, 428 objects/s recovering 2024-04-29T14:33:06.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:33:06.941 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:06.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:06.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:07.045 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:07.077 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:07.114 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:07.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:07.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:07.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:07.515 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:33:07.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1888071847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:07.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:07 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:07.580 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:07.623 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:07.649 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:07.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:07.728 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:33:07.728 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:33:07.754 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1888071847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:07.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:07 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:07.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1888071847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:07.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:07 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:07.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:07.826 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:07.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:07.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:08.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:08.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:33:08.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:08.543 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:33:08.544 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:33:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:08.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:08.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:08.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1888071847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:33:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:08 smithi121 ceph-mon[44533]: osdmap e610: 12 total, 12 up, 7 in 2024-04-29T14:33:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:08 smithi121 ceph-mon[44533]: from='client.17133 v1:172.21.15.40:0/958890124' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:33:08.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:08 smithi121 ceph-mon[44533]: pgmap v1581: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 839 op/s; 13792/143302 objects misplaced (9.624%); 39 MiB/s, 617 objects/s recovering 2024-04-29T14:33:08.767 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:08.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:08.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1888071847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:33:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:08 smithi194 ceph-mon[44089]: osdmap e610: 12 total, 12 up, 7 in 2024-04-29T14:33:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:08 smithi194 ceph-mon[44089]: from='client.17133 v1:172.21.15.40:0/958890124' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:33:08.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:08 smithi194 ceph-mon[44089]: pgmap v1581: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 839 op/s; 13792/143302 objects misplaced (9.624%); 39 MiB/s, 617 objects/s recovering 2024-04-29T14:33:08.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1888071847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:33:08.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:08 smithi040 ceph-mon[39581]: osdmap e610: 12 total, 12 up, 7 in 2024-04-29T14:33:08.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:08 smithi040 ceph-mon[39581]: from='client.17133 v1:172.21.15.40:0/958890124' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:33:08.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:08 smithi040 ceph-mon[39581]: pgmap v1581: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 839 op/s; 13792/143302 objects misplaced (9.624%); 39 MiB/s, 617 objects/s recovering 2024-04-29T14:33:08.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:08.937 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:09.230 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:09.268 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2024-04-29T14:33:07.514832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4696113720,"num_objects":71651,"num_object_clones":0,"num_object_copies":143302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13792,"num_objects_unfound":0,"num_objects_dirty":71651,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":234234,"num_write_kb":9785339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21398,"num_bytes_recovered":1404726464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160890,"ondisk_log_size":160890,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14417324,"kb_used_data":9704680,"kb_used_omap":10,"kb_used_meta":4712117,"kb_avail":641798740,"statfs":{"total":671965249536,"available":657201909760,"internally_reserved":0,"allocated":9937592320,"data_stored":9934943935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11170,"internal_metadata":4825207902},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,2,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-440270848,"num_objects":-6718,"num_object_clones":0,"num_object_copies":-13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10623,"num_objects_unfound":0,"num_objects_dirty":-6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4945,"num_bytes_recovered":324075520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.616255"},"pg_stats":[{"pgid":"6.f","version":"608'14515","reported_seq":14820,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-29T14:33:04.059444+0000","last_change":"2024-04-29T14:32:50.326594+0000","last_active":"2024-04-29T14:33:04.059444+0000","last_peered":"2024-04-29T14:33:04.059444+0000","last_clean":"2024-04-29T14:33:04.059444+0000","last_became_active":"2024-04-29T14:29:46.263829+0000","last_became_peered":"2024-04-29T14:29:46.263829+0000","last_unstale":"2024-04-29T14:33:04.059444+0000","last_undegraded":"2024-04-29T14:33:04.059444+0000","last_fullsized":"2024-04-29T14:33:04.059444+0000","mapping_epoch":534,"log_start":"551'4500","ondisk_log_start":"551'4500","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'13920","last_scrub_stamp":"2024-04-29T14:32:50.326447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:32:50.326447+0000","objects_scrubbed":5150,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:33:56.147337+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298516480,"num_objects":4555,"num_object_clones":0,"num_object_copies":9110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14515,"num_write_kb":610240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.e","version":"609'14535","reported_seq":16338,"reported_epoch":609,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:33:07.099011+0000","last_change":"2024-04-29T14:32:58.550179+0000","last_active":"2024-04-29T14:33:07.099011+0000","last_peered":"2024-04-29T14:33:07.099011+0000","last_clean":"2024-04-29T14:32:55.359011+0000","last_became_active":"2024-04-29T14:32:58.480739+0000","last_became_peered":"2024-04-29T14:32:58.480739+0000","last_unstale":"2024-04-29T14:33:07.099011+0000","last_undegraded":"2024-04-29T14:33:07.099011+0000","last_fullsized":"2024-04-29T14:33:07.099011+0000","mapping_epoch":606,"log_start":"551'4500","ondisk_log_start":"551'4500","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'11822","last_scrub_stamp":"2024-04-29T14:32:11.711130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:32:11.711130+0000","objects_scrubbed":7231,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291569664,"num_objects":4449,"num_object_clones":0,"num_object_copies":8898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3740,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14535,"num_write_kb":607488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":737,"num_bytes_recovered":48300032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":4449}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"608'14561","reported_seq":14808,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-29T14:33:04.025352+0000","last_change":"2024-04-29T14:31:07.366696+0000","last_active":"2024-04-29T14:33:04.025352+0000","last_peered":"2024-04-29T14:33:04.025352+0000","last_clean":"2024-04-29T14:33:04.025352+0000","last_became_active":"2024-04-29T14:29:46.295428+0000","last_became_peered":"2024-04-29T14:29:46.295428+0000","last_unstale":"2024-04-29T14:33:04.025352+0000","last_undegraded":"2024-04-29T14:33:04.025352+0000","last_fullsized":"2024-04-29T14:33:04.025352+0000","mapping_epoch":534,"log_start":"551'4500","ondisk_log_start":"551'4500","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'7946","last_scrub_stamp":"2024-04-29T14:31:07.366598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:07.366598+0000","objects_scrubbed":7922,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306446360,"num_objects":4677,"num_object_clones":0,"num_object_copies":9354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14561,"num_write_kb":615553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.c","version":"609'14642","reported_seq":14885,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:05.516858+0000","last_change":"2024-04-29T14:31:03.410345+0000","last_active":"2024-04-29T14:33:05.516858+0000","last_peered":"2024-04-29T14:33:05.516858+0000","last_clean":"2024-04-29T14:33:05.516858+0000","last_became_active":"2024-04-29T14:29:46.265543+0000","last_became_peered":"2024-04-29T14:29:46.265543+0000","last_unstale":"2024-04-29T14:33:05.516858+0000","last_undegraded":"2024-04-29T14:33:05.516858+0000","last_fullsized":"2024-04-29T14:33:05.516858+0000","mapping_epoch":534,"log_start":"551'4600","ondisk_log_start":"551'4600","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"560'7725","last_scrub_stamp":"2024-04-29T14:31:03.410174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:03.410174+0000","objects_scrubbed":7647,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297402368,"num_objects":4538,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14642,"num_write_kb":613760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"609'14669","reported_seq":27479,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:05.512233+0000","last_change":"2024-04-29T14:32:50.361398+0000","last_active":"2024-04-29T14:33:05.512233+0000","last_peered":"2024-04-29T14:33:05.512233+0000","last_clean":"2024-04-29T14:33:05.512233+0000","last_became_active":"2024-04-29T14:32:50.330100+0000","last_became_peered":"2024-04-29T14:32:50.330100+0000","last_unstale":"2024-04-29T14:33:05.512233+0000","last_undegraded":"2024-04-29T14:33:05.512233+0000","last_fullsized":"2024-04-29T14:33:05.512233+0000","mapping_epoch":601,"log_start":"551'4600","ondisk_log_start":"551'4600","created":534,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'9724","last_scrub_stamp":"2024-04-29T14:31:33.993982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:33.993982+0000","objects_scrubbed":9576,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299302912,"num_objects":4567,"num_object_clones":0,"num_object_copies":9134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14669,"num_write_kb":615552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6268,"num_bytes_recovered":410779648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"609'14850","reported_seq":15114,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:05.547729+0000","last_change":"2024-04-29T14:31:33.868461+0000","last_active":"2024-04-29T14:33:05.547729+0000","last_peered":"2024-04-29T14:33:05.547729+0000","last_clean":"2024-04-29T14:33:05.547729+0000","last_became_active":"2024-04-29T14:29:46.295534+0000","last_became_peered":"2024-04-29T14:29:46.295534+0000","last_unstale":"2024-04-29T14:33:05.547729+0000","last_undegraded":"2024-04-29T14:33:05.547729+0000","last_fullsized":"2024-04-29T14:33:05.547729+0000","mapping_epoch":534,"log_start":"551'4800","ondisk_log_start":"551'4800","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'9792","last_scrub_stamp":"2024-04-29T14:31:33.868257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:33.868257+0000","objects_scrubbed":9668,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299499520,"num_objects":4570,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14850,"num_write_kb":621440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"609'14356","reported_seq":14616,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:05.980563+0000","last_change":"2024-04-29T14:31:34.796445+0000","last_active":"2024-04-29T14:33:05.980563+0000","last_peered":"2024-04-29T14:33:05.980563+0000","last_clean":"2024-04-29T14:33:05.980563+0000","last_became_active":"2024-04-29T14:29:46.295508+0000","last_became_peered":"2024-04-29T14:29:46.295508+0000","last_unstale":"2024-04-29T14:33:05.980563+0000","last_undegraded":"2024-04-29T14:33:05.980563+0000","last_fullsized":"2024-04-29T14:33:05.980563+0000","mapping_epoch":534,"log_start":"550'4300","ondisk_log_start":"550'4300","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9481","last_scrub_stamp":"2024-04-29T14:31:34.796246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:34.796246+0000","objects_scrubbed":9253,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283639808,"num_objects":4328,"num_object_clones":0,"num_object_copies":8656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14356,"num_write_kb":597888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"609'14756","reported_seq":26995,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:05.510368+0000","last_change":"2024-04-29T14:32:56.166753+0000","last_active":"2024-04-29T14:33:05.510368+0000","last_peered":"2024-04-29T14:33:05.510368+0000","last_clean":"2024-04-29T14:33:05.510368+0000","last_became_active":"2024-04-29T14:32:56.129025+0000","last_became_peered":"2024-04-29T14:32:56.129025+0000","last_unstale":"2024-04-29T14:33:05.510368+0000","last_undegraded":"2024-04-29T14:33:05.510368+0000","last_fullsized":"2024-04-29T14:33:05.510368+0000","mapping_epoch":604,"log_start":"551'4700","ondisk_log_start":"551'4700","created":534,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'8413","last_scrub_stamp":"2024-04-29T14:31:13.363787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:13.363787+0000","objects_scrubbed":8333,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295305216,"num_objects":4506,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14756,"num_write_kb":616384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5988,"num_bytes_recovered":392429568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"609'14811","reported_seq":16681,"reported_epoch":609,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:33:07.094028+0000","last_change":"2024-04-29T14:32:58.332001+0000","last_active":"2024-04-29T14:33:07.094028+0000","last_peered":"2024-04-29T14:33:07.094028+0000","last_clean":"2024-04-29T14:32:55.370269+0000","last_became_active":"2024-04-29T14:32:58.274160+0000","last_became_peered":"2024-04-29T14:32:58.274160+0000","last_unstale":"2024-04-29T14:33:07.094028+0000","last_undegraded":"2024-04-29T14:33:07.094028+0000","last_fullsized":"2024-04-29T14:33:07.094028+0000","mapping_epoch":606,"log_start":"551'4800","ondisk_log_start":"551'4800","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'13672","last_scrub_stamp":"2024-04-29T14:32:41.231848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:32:41.231848+0000","objects_scrubbed":5603,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1510,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290258944,"num_objects":4429,"num_object_clones":0,"num_object_copies":8858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3701,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14811,"num_write_kb":615680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":50855936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":4429}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"609'14733","reported_seq":14980,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:07.080036+0000","last_change":"2024-04-29T14:31:04.370256+0000","last_active":"2024-04-29T14:33:07.080036+0000","last_peered":"2024-04-29T14:33:07.080036+0000","last_clean":"2024-04-29T14:33:07.080036+0000","last_became_active":"2024-04-29T14:29:46.265022+0000","last_became_peered":"2024-04-29T14:29:46.265022+0000","last_unstale":"2024-04-29T14:33:07.080036+0000","last_undegraded":"2024-04-29T14:33:07.080036+0000","last_fullsized":"2024-04-29T14:33:07.080036+0000","mapping_epoch":534,"log_start":"551'4700","ondisk_log_start":"551'4700","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'7749","last_scrub_stamp":"2024-04-29T14:31:04.370113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:04.370113+0000","objects_scrubbed":7682,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287506432,"num_objects":4387,"num_object_clones":0,"num_object_copies":8774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14733,"num_write_kb":611840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"609'14611","reported_seq":22583,"reported_epoch":609,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:33:06.010706+0000","last_change":"2024-04-29T14:32:43.296870+0000","last_active":"2024-04-29T14:33:06.010706+0000","last_peered":"2024-04-29T14:33:06.010706+0000","last_clean":"2024-04-29T14:32:41.018713+0000","last_became_active":"2024-04-29T14:32:43.248184+0000","last_became_peered":"2024-04-29T14:32:43.248184+0000","last_unstale":"2024-04-29T14:33:06.010706+0000","last_undegraded":"2024-04-29T14:33:06.010706+0000","last_fullsized":"2024-04-29T14:33:06.010706+0000","mapping_epoch":597,"log_start":"551'4600","ondisk_log_start":"551'4600","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13245","last_scrub_stamp":"2024-04-29T14:32:36.397911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:32:36.397911+0000","objects_scrubbed":5794,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287768576,"num_objects":4391,"num_object_clones":0,"num_object_copies":8782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":975,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14611,"num_write_kb":608064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3826,"num_bytes_recovered":250740736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,11],"avail_no_missing":["9","11"],"object_location_counts":[{"shards":"9,11","objects":4391}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"609'14796","reported_seq":15042,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:07.070751+0000","last_change":"2024-04-29T14:31:07.543044+0000","last_active":"2024-04-29T14:33:07.070751+0000","last_peered":"2024-04-29T14:33:07.070751+0000","last_clean":"2024-04-29T14:33:07.070751+0000","last_became_active":"2024-04-29T14:29:46.266681+0000","last_became_peered":"2024-04-29T14:29:46.266681+0000","last_unstale":"2024-04-29T14:33:07.070751+0000","last_undegraded":"2024-04-29T14:33:07.070751+0000","last_fullsized":"2024-04-29T14:33:07.070751+0000","mapping_epoch":534,"log_start":"551'4700","ondisk_log_start":"551'4700","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'7960","last_scrub_stamp":"2024-04-29T14:31:07.542797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:07.542797+0000","objects_scrubbed":7890,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291241984,"num_objects":4444,"num_object_clones":0,"num_object_copies":8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14796,"num_write_kb":615680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"609'14738","reported_seq":22647,"reported_epoch":609,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:33:06.885653+0000","last_change":"2024-04-29T14:32:57.877364+0000","last_active":"2024-04-29T14:33:06.885653+0000","last_peered":"2024-04-29T14:33:06.885653+0000","last_clean":"2024-04-29T14:32:41.094457+0000","last_became_active":"2024-04-29T14:32:57.834037+0000","last_became_peered":"2024-04-29T14:32:57.834037+0000","last_unstale":"2024-04-29T14:33:06.885653+0000","last_undegraded":"2024-04-29T14:33:06.885653+0000","last_fullsized":"2024-04-29T14:33:06.885653+0000","mapping_epoch":605,"log_start":"551'4700","ondisk_log_start":"551'4700","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'13406","last_scrub_stamp":"2024-04-29T14:32:37.780422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:32:37.780422+0000","objects_scrubbed":5729,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286261248,"num_objects":4368,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5376,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14738,"num_write_kb":611392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3791,"num_bytes_recovered":248446976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[4,11],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"4,11","objects":4368}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"609'14678","reported_seq":14940,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:05.980159+0000","last_change":"2024-04-29T14:31:36.055861+0000","last_active":"2024-04-29T14:33:05.980159+0000","last_peered":"2024-04-29T14:33:05.980159+0000","last_clean":"2024-04-29T14:33:05.980159+0000","last_became_active":"2024-04-29T14:29:46.264619+0000","last_became_peered":"2024-04-29T14:29:46.264619+0000","last_unstale":"2024-04-29T14:33:05.980159+0000","last_undegraded":"2024-04-29T14:33:05.980159+0000","last_fullsized":"2024-04-29T14:33:05.980159+0000","mapping_epoch":534,"log_start":"551'4600","ondisk_log_start":"551'4600","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9820","last_scrub_stamp":"2024-04-29T14:31:36.055637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:36.055637+0000","objects_scrubbed":9425,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1738,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295698432,"num_objects":4512,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":614080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10],"acting":[9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"608'14254","reported_seq":14528,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-29T14:33:04.058964+0000","last_change":"2024-04-29T14:31:29.585723+0000","last_active":"2024-04-29T14:33:04.058964+0000","last_peered":"2024-04-29T14:33:04.058964+0000","last_clean":"2024-04-29T14:33:04.058964+0000","last_became_active":"2024-04-29T14:29:46.295491+0000","last_became_peered":"2024-04-29T14:29:46.295491+0000","last_unstale":"2024-04-29T14:33:04.058964+0000","last_undegraded":"2024-04-29T14:33:04.058964+0000","last_fullsized":"2024-04-29T14:33:04.058964+0000","mapping_epoch":534,"log_start":"550'4200","ondisk_log_start":"550'4200","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'9235","last_scrub_stamp":"2024-04-29T14:31:29.585566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:29.585566+0000","objects_scrubbed":9210,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":4544,"num_object_clones":0,"num_object_copies":9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14254,"num_write_kb":601536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.7","version":"609'14548","reported_seq":14870,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:06.857143+0000","last_change":"2024-04-29T14:32:16.497154+0000","last_active":"2024-04-29T14:33:06.857143+0000","last_peered":"2024-04-29T14:33:06.857143+0000","last_clean":"2024-04-29T14:33:06.857143+0000","last_became_active":"2024-04-29T14:29:46.267596+0000","last_became_peered":"2024-04-29T14:29:46.267596+0000","last_unstale":"2024-04-29T14:33:06.857143+0000","last_undegraded":"2024-04-29T14:33:06.857143+0000","last_fullsized":"2024-04-29T14:33:06.857143+0000","mapping_epoch":534,"log_start":"551'4500","ondisk_log_start":"551'4500","created":534,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'12080","last_scrub_stamp":"2024-04-29T14:32:16.497024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:29:45.239629+0000","last_clean_scrub_stamp":"2024-04-29T14:32:16.497024+0000","objects_scrubbed":6905,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:33:42.706094+0000","scrub_duration":1998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287309824,"num_objects":4384,"num_object_clones":0,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14548,"num_write_kb":605824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1493,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-29T14:33:04.333685+0000","last_change":"2024-04-29T14:32:42.120228+0000","last_active":"2024-04-29T14:33:04.333685+0000","last_peered":"2024-04-29T14:33:04.333685+0000","last_clean":"2024-04-29T14:33:04.333685+0000","last_became_active":"2024-04-29T14:32:42.114595+0000","last_became_peered":"2024-04-29T14:32:42.114595+0000","last_unstale":"2024-04-29T14:33:04.333685+0000","last_undegraded":"2024-04-29T14:33:04.333685+0000","last_fullsized":"2024-04-29T14:33:04.333685+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:31:03.264355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:31:03.264355+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4695523352,"num_objects":71649,"num_object_clones":0,"num_object_copies":143298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13792,"num_objects_unfound":0,"num_objects_dirty":71649,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234053,"num_write_kb":9782401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21386,"num_bytes_recovered":1401552896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9891487744,"data_stored":9891479600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":32,"acting":32,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122362,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1537548,"kb_used_data":892740,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92207604,"statfs":{"total":95995035648,"available":94420586496,"internally_reserved":0,"allocated":914165760,"data_stored":913788705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":660273604},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597383998,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2202308,"kb_used_data":1543420,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":91542844,"statfs":{"total":95995035648,"available":93739872256,"internally_reserved":0,"allocated":1580462080,"data_stored":1580087101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":674494912},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645636,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1829460,"kb_used_data":1306828,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91915692,"statfs":{"total":95995035648,"available":94121668608,"internally_reserved":0,"allocated":1338191872,"data_stored":1337811285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":535165376},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422940,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234317,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2424012,"kb_used_data":1711172,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91321140,"statfs":{"total":95995035648,"available":93512847360,"internally_reserved":0,"allocated":1752240128,"data_stored":1751863093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":729938378},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528659,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2696216,"kb_used_data":1965068,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":91048936,"statfs":{"total":95995035648,"available":93234110464,"internally_reserved":0,"allocated":2012229632,"data_stored":2011849045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575211,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738737,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1778452,"kb_used_data":1151688,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91966700,"statfs":{"total":95995035648,"available":94173900800,"internally_reserved":0,"allocated":1179328512,"data_stored":1178947405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497170,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986526,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949328,"kb_used_data":1133764,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91795824,"statfs":{"total":95995035648,"available":93998923776,"internally_reserved":0,"allocated":1160974336,"data_stored":1160597301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":0,"seq":2409476653081,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1170739200,"data_stored":1170735128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1157103616,"data_stored":1157103616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2004418560,"data_stored":2004418560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1739063296,"data_stored":1739063296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1336934400,"data_stored":1336934400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1574703104,"data_stored":1574699032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":908525568,"data_stored":908525568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:33:09.269 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-29T14:33:09.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-04-29T14:33:09.588 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:09.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:09.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:09.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:09.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:09.845 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:09.883 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:09.909 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:09.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:10.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:10.067 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:10.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:10.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:10.201 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:10.243 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:10.269 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:10.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:10.375 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:10.419 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:10.437 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:33:10.443 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:10.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:10.554 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:10.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:10.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:10.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:10 smithi040 ceph-mon[39581]: osdmap e611: 12 total, 12 up, 7 in 2024-04-29T14:33:10.603 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:10 smithi040 ceph-mon[39581]: pgmap v1583: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 13792/141552 objects misplaced (9.743%); 21 MiB/s, 335 objects/s recovering 2024-04-29T14:33:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:10 smithi194 ceph-mon[44089]: osdmap e611: 12 total, 12 up, 7 in 2024-04-29T14:33:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:10 smithi194 ceph-mon[44089]: pgmap v1583: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 13792/141552 objects misplaced (9.743%); 21 MiB/s, 335 objects/s recovering 2024-04-29T14:33:10.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:10.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:10.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:10 smithi121 ceph-mon[44533]: osdmap e611: 12 total, 12 up, 7 in 2024-04-29T14:33:10.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:10 smithi121 ceph-mon[44533]: pgmap v1583: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 13792/141552 objects misplaced (9.743%); 21 MiB/s, 335 objects/s recovering 2024-04-29T14:33:11.242 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:11.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:11.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:11.400 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:11.432 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:11.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:11.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:11.606 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:11.634 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:11.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:11.741 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:11.780 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:11.810 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:11.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:11.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:11.975 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:12.003 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:12.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:12.108 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:12.152 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:12.192 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:12.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:12.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:12.345 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:12.370 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:12.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:12.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:13.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:13.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:13.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:13.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:13.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:13.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:13 smithi040 ceph-mon[39581]: pgmap v1584: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.2 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 12279/136390 objects misplaced (9.003%); 25 MiB/s, 405 objects/s recovering 2024-04-29T14:33:13.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:13 smithi121 ceph-mon[44533]: pgmap v1584: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.2 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 12279/136390 objects misplaced (9.003%); 25 MiB/s, 405 objects/s recovering 2024-04-29T14:33:13.404 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:13.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:13 smithi194 ceph-mon[44089]: pgmap v1584: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.2 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 12279/136390 objects misplaced (9.003%); 25 MiB/s, 405 objects/s recovering 2024-04-29T14:33:13.436 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:13.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:13.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:13.576 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:13.597 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:13.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:13.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:13.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:13.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:13.894 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:13.911 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:33:13.911 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:33:13.924 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:13.964 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:13.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:14.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:14.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:14.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:14.308 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:14.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:14.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:14.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:14.499 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:14.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:14.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:14.649 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:14.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:14.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:33:15.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:15.081 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:15.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:15.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:15.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:15 smithi040 ceph-mon[39581]: pgmap v1585: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 10822/134096 objects misplaced (8.070%); 29 MiB/s, 460 objects/s recovering 2024-04-29T14:33:15.222 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:15.230 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:15.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:15 smithi121 ceph-mon[44533]: pgmap v1585: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 10822/134096 objects misplaced (8.070%); 29 MiB/s, 460 objects/s recovering 2024-04-29T14:33:15.441 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:15.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:15.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:15.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:15 smithi194 ceph-mon[44089]: pgmap v1585: 17 pgs: 4 active+remapped+backfilling, 13 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 10822/134096 objects misplaced (8.070%); 29 MiB/s, 460 objects/s recovering 2024-04-29T14:33:15.586 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:15.609 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:15.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:15.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:15.743 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:15.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:15.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:15.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:33:15.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:33:15.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:33:15.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:33:15.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:15.871 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:33:15.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:33:15.900 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:15.915 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:33:15.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:15.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:15.935 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:33:15.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:33:15.976 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:33:15.997 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:33:16.016 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:33:16.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:33:16.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 6 pg_num to 26 2024-04-29T14:33:16.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:16 smithi040 ceph-mon[39581]: osdmap e612: 12 total, 12 up, 7 in 2024-04-29T14:33:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/153231669' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:33:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2862464291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:16 smithi121 ceph-mon[44533]: osdmap e612: 12 total, 12 up, 7 in 2024-04-29T14:33:16.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/153231669' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:33:16.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2862464291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:16.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:16 smithi194 ceph-mon[44089]: osdmap e612: 12 total, 12 up, 7 in 2024-04-29T14:33:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/153231669' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-04-29T14:33:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2862464291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:16.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:16.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:16.914 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:16.959 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:16.986 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:16.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:17.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:17.149 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:17.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:17.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:17.288 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:17.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: pgmap v1587: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 10403/128550 objects misplaced (8.093%); 20 MiB/s, 317 objects/s recovering 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/153231669' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-04-29T14:33:17.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:17 smithi194 ceph-mon[44089]: osdmap e613: 12 total, 12 up, 7 in 2024-04-29T14:33:17.349 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:17.375 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:17.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:17.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: pgmap v1587: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 10403/128550 objects misplaced (8.093%); 20 MiB/s, 317 objects/s recovering 2024-04-29T14:33:17.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/153231669' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-04-29T14:33:17.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:17 smithi040 ceph-mon[39581]: osdmap e613: 12 total, 12 up, 7 in 2024-04-29T14:33:17.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:17.541 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:17.545 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: pgmap v1587: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 921 op/s; 10403/128550 objects misplaced (8.093%); 20 MiB/s, 317 objects/s recovering 2024-04-29T14:33:17.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/153231669' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-04-29T14:33:17.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:17 smithi121 ceph-mon[44533]: osdmap e613: 12 total, 12 up, 7 in 2024-04-29T14:33:17.579 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:17.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:17.687 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:17.738 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:17.763 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:17.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:17.777 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:17.868 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:17.941 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:17.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:18.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:18.112 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:18.137 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:18.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:18.249 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:18.303 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:18.335 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:18.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:18.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:18 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:33:18.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:18 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:33:18.435 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:18.478 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:18.499 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:18.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:18 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:33:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:18 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:33:18.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:18 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:33:18.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:18 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:33:18.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:18.667 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:18.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:18.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:18.862 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:18.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:18.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:18.995 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:19.042 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:19.075 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:19.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:19.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:19.241 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:19.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:19.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:19.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:19.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:19 smithi194 ceph-mon[44089]: pgmap v1589: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 998 op/s; 6843/125570 objects misplaced (5.450%); 39 MiB/s, 619 objects/s recovering 2024-04-29T14:33:19.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:19.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:33:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:19.487 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:19.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:19.592 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:19.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:19 smithi040 ceph-mon[39581]: pgmap v1589: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 998 op/s; 6843/125570 objects misplaced (5.450%); 39 MiB/s, 619 objects/s recovering 2024-04-29T14:33:19.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:19 smithi121 ceph-mon[44533]: pgmap v1589: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 998 op/s; 6843/125570 objects misplaced (5.450%); 39 MiB/s, 619 objects/s recovering 2024-04-29T14:33:19.645 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:19.679 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:19.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:19.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:19.817 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:20.068 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:20.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:20.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:20.204 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:20.236 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:20.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:20.332 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:33:20.341 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:20.382 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:20.410 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:20.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:20.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:20.545 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:20.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1359127933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:20.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:20 smithi194 ceph-mon[44089]: pgmap v1590: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 6843/124042 objects misplaced (5.517%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:33:20.600 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:20.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1359127933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:20.602 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:20 smithi040 ceph-mon[39581]: pgmap v1590: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 6843/124042 objects misplaced (5.517%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:33:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:20.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:20.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1359127933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:20.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:20 smithi121 ceph-mon[44533]: pgmap v1590: 17 pgs: 1 peering, 3 active+remapped+backfilling, 13 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 6843/124042 objects misplaced (5.517%); 30 MiB/s, 480 objects/s recovering 2024-04-29T14:33:20.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:21.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:21.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:21 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:21 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1359127933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:21.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:21 smithi040 ceph-mon[39581]: osdmap e614: 12 total, 12 up, 7 in 2024-04-29T14:33:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:21.435 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:33:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:33:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:33:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:33:21.435 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:33:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:21.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:21.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:21.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:21 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:33:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:21 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:33:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1359127933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:21 smithi121 ceph-mon[44533]: osdmap e614: 12 total, 12 up, 7 in 2024-04-29T14:33:21.705 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:21.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:21.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:21.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:21 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:33:21.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:21 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:33:21.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1359127933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:21.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:21 smithi194 ceph-mon[44089]: osdmap e614: 12 total, 12 up, 7 in 2024-04-29T14:33:21.840 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:21.916 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:21.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:22.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:22.072 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:22.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:22.236 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:22.273 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:22.299 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:22.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:22.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:22.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:22 smithi194 ceph-mon[44089]: 6.c scrub starts 2024-04-29T14:33:22.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:22 smithi194 ceph-mon[44089]: 6.c scrub ok 2024-04-29T14:33:22.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:22 smithi194 ceph-mon[44089]: pgmap v1592: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 957 op/s; 5627/119104 objects misplaced (4.724%); 30 MiB/s, 483 objects/s recovering 2024-04-29T14:33:22.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:33:22.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:22 smithi040 ceph-mon[39581]: 6.c scrub starts 2024-04-29T14:33:22.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:22 smithi040 ceph-mon[39581]: 6.c scrub ok 2024-04-29T14:33:22.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:22 smithi040 ceph-mon[39581]: pgmap v1592: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 957 op/s; 5627/119104 objects misplaced (4.724%); 30 MiB/s, 483 objects/s recovering 2024-04-29T14:33:22.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:33:22.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:22 smithi121 ceph-mon[44533]: 6.c scrub starts 2024-04-29T14:33:22.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:22 smithi121 ceph-mon[44533]: 6.c scrub ok 2024-04-29T14:33:22.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:22 smithi121 ceph-mon[44533]: pgmap v1592: 17 pgs: 3 active+remapped+backfilling, 14 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 957 op/s; 5627/119104 objects misplaced (4.724%); 30 MiB/s, 483 objects/s recovering 2024-04-29T14:33:22.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-29T14:33:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:22.665 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:22.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:22.774 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:22.799 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:22.819 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:22.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:22.901 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:33:22.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:22.973 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:23.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:33:23.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:23 smithi040 ceph-mon[39581]: 6.8 scrub starts 2024-04-29T14:33:23.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-29T14:33:23.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:23 smithi040 ceph-mon[39581]: osdmap e615: 12 total, 12 up, 7 in 2024-04-29T14:33:23.522 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:23.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:23.565 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:23.624 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6] out_osds: [1, 2, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:33:23.624 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:33:23.624 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-29T14:33:23.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 2 2024-04-29T14:33:23.626 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:23.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:23 smithi121 ceph-mon[44533]: 6.8 scrub starts 2024-04-29T14:33:23.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-29T14:33:23.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:23 smithi121 ceph-mon[44533]: osdmap e615: 12 total, 12 up, 7 in 2024-04-29T14:33:23.653 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:23.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:23.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:23 smithi194 ceph-mon[44089]: 6.8 scrub starts 2024-04-29T14:33:23.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-29T14:33:23.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:23 smithi194 ceph-mon[44089]: osdmap e615: 12 total, 12 up, 7 in 2024-04-29T14:33:23.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:23.933 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:24.265 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:33:24.284 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:33:24.303 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:33:24.323 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:33:24.344 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:33:24.366 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:33:24.388 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:33:24.409 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:33:24.429 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:33:24.448 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:33:24.468 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:33:24.488 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:33:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:33:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:33:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:33:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:33:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:33:24.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:24 smithi040 ceph-mon[39581]: 6.6 starting backfill to osd.9 from (553'5600,614'15667] MIN to 614'15669 2024-04-29T14:33:24.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:24 smithi040 ceph-mon[39581]: 6.2 continuing backfill to osd.9 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:24.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:24 smithi040 ceph-mon[39581]: 6.2 continuing backfill to osd.10 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:24.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:24 smithi040 ceph-mon[39581]: osdmap e616: 12 total, 12 up, 7 in 2024-04-29T14:33:24.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:24 smithi040 ceph-mon[39581]: pgmap v1595: 27 pgs: 10 unknown, 3 active+remapped+backfilling, 14 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 4435/116810 objects misplaced (3.797%); 15 MiB/s, 239 objects/s recovering 2024-04-29T14:33:24.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:33:24.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4054248448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:24.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:24 smithi121 ceph-mon[44533]: 6.6 starting backfill to osd.9 from (553'5600,614'15667] MIN to 614'15669 2024-04-29T14:33:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:24 smithi121 ceph-mon[44533]: 6.2 continuing backfill to osd.9 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:24 smithi121 ceph-mon[44533]: 6.2 continuing backfill to osd.10 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:24 smithi121 ceph-mon[44533]: osdmap e616: 12 total, 12 up, 7 in 2024-04-29T14:33:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:24 smithi121 ceph-mon[44533]: pgmap v1595: 27 pgs: 10 unknown, 3 active+remapped+backfilling, 14 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 4435/116810 objects misplaced (3.797%); 15 MiB/s, 239 objects/s recovering 2024-04-29T14:33:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:33:24.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4054248448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:24.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:24 smithi194 ceph-mon[44089]: 6.6 starting backfill to osd.9 from (553'5600,614'15667] MIN to 614'15669 2024-04-29T14:33:24.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:24 smithi194 ceph-mon[44089]: 6.2 continuing backfill to osd.9 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:24.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:24 smithi194 ceph-mon[44089]: 6.2 continuing backfill to osd.10 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:24.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:24 smithi194 ceph-mon[44089]: osdmap e616: 12 total, 12 up, 7 in 2024-04-29T14:33:24.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:24 smithi194 ceph-mon[44089]: pgmap v1595: 27 pgs: 10 unknown, 3 active+remapped+backfilling, 14 active+clean; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 729 op/s; 4435/116810 objects misplaced (3.797%); 15 MiB/s, 239 objects/s recovering 2024-04-29T14:33:24.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-29T14:33:24.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4054248448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:25.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:25.487 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:25.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: 6.8 deep-scrub starts 2024-04-29T14:33:25.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osdmap e617: 12 total, 12 up, 7 in 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: 6.7 scrub starts 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: 6.7 scrub ok 2024-04-29T14:33:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: 6.0 scrub starts 2024-04-29T14:33:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:25 smithi040 ceph-mon[39581]: 6.0 scrub ok 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: 6.8 deep-scrub starts 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osdmap e617: 12 total, 12 up, 7 in 2024-04-29T14:33:25.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: 6.7 scrub starts 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: 6.7 scrub ok 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: 6.0 scrub starts 2024-04-29T14:33:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:25 smithi121 ceph-mon[44533]: 6.0 scrub ok 2024-04-29T14:33:25.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: 6.8 deep-scrub starts 2024-04-29T14:33:25.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osdmap e617: 12 total, 12 up, 7 in 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: 6.7 scrub starts 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: 6.7 scrub ok 2024-04-29T14:33:25.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: 6.0 scrub starts 2024-04-29T14:33:25.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:25 smithi194 ceph-mon[44089]: 6.0 scrub ok 2024-04-29T14:33:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:26.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:26.415 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:26.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:26.534 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:26.585 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:26.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:26.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:26.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:26 smithi040 ceph-mon[39581]: 6.1 scrub starts 2024-04-29T14:33:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:26 smithi040 ceph-mon[39581]: 6.1 scrub ok 2024-04-29T14:33:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:26 smithi040 ceph-mon[39581]: 6.12 continuing backfill to osd.9 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:26 smithi040 ceph-mon[39581]: osdmap e618: 12 total, 12 up, 7 in 2024-04-29T14:33:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:26 smithi040 ceph-mon[39581]: 6.6 scrub starts 2024-04-29T14:33:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:26 smithi040 ceph-mon[39581]: 6.6 scrub ok 2024-04-29T14:33:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:26 smithi040 ceph-mon[39581]: pgmap v1598: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 4 unknown, 2 active+remapped+backfilling, 14 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2779/114176 objects misplaced (2.434%) 2024-04-29T14:33:26.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:26.740 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:26 smithi121 ceph-mon[44533]: 6.1 scrub starts 2024-04-29T14:33:26.740 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:26 smithi121 ceph-mon[44533]: 6.1 scrub ok 2024-04-29T14:33:26.741 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:26 smithi121 ceph-mon[44533]: 6.12 continuing backfill to osd.9 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:26.741 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:26 smithi121 ceph-mon[44533]: osdmap e618: 12 total, 12 up, 7 in 2024-04-29T14:33:26.741 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:26 smithi121 ceph-mon[44533]: 6.6 scrub starts 2024-04-29T14:33:26.741 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:26 smithi121 ceph-mon[44533]: 6.6 scrub ok 2024-04-29T14:33:26.741 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:26 smithi121 ceph-mon[44533]: pgmap v1598: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 4 unknown, 2 active+remapped+backfilling, 14 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2779/114176 objects misplaced (2.434%) 2024-04-29T14:33:26.743 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:26.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:26.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:26.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:26 smithi194 ceph-mon[44089]: 6.1 scrub starts 2024-04-29T14:33:26.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:26 smithi194 ceph-mon[44089]: 6.1 scrub ok 2024-04-29T14:33:26.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:26 smithi194 ceph-mon[44089]: 6.12 continuing backfill to osd.9 from (553'5600,614'15611] MIN to 614'15611 2024-04-29T14:33:26.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:26 smithi194 ceph-mon[44089]: osdmap e618: 12 total, 12 up, 7 in 2024-04-29T14:33:26.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:26 smithi194 ceph-mon[44089]: 6.6 scrub starts 2024-04-29T14:33:26.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:26 smithi194 ceph-mon[44089]: 6.6 scrub ok 2024-04-29T14:33:26.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:26 smithi194 ceph-mon[44089]: pgmap v1598: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 4 unknown, 2 active+remapped+backfilling, 14 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2779/114176 objects misplaced (2.434%) 2024-04-29T14:33:26.886 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:26.930 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:26.961 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:26.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:27.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.13 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15592 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: osdmap e619: 12 total, 12 up, 7 in 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.11 starting backfill to osd.0 from (0'0,0'0] MAX to 616'15597 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.17 starting backfill to osd.10 from (0'0,0'0] MAX to 616'15385 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.16 starting backfill to osd.0 from (0'0,0'0] MAX to 614'15669 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.19 starting backfill to osd.10 from (0'0,0'0] MAX to 616'15219 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 616'15506 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.10 starting backfill to osd.6 from (0'0,0'0] MAX to 616'15330 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.11 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15597 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.14 starting backfill to osd.7 from (0'0,0'0] MAX to 616'15506 2024-04-29T14:33:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.17 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15385 2024-04-29T14:33:27.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.18 starting backfill to osd.0 from (0'0,0'0] MAX to 616'15686 2024-04-29T14:33:27.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:27 smithi194 ceph-mon[44089]: 6.18 starting backfill to osd.7 from (0'0,0'0] MAX to 616'15686 2024-04-29T14:33:27.863 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:33:27.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.13 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15592 2024-04-29T14:33:27.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: osdmap e619: 12 total, 12 up, 7 in 2024-04-29T14:33:27.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.11 starting backfill to osd.0 from (0'0,0'0] MAX to 616'15597 2024-04-29T14:33:27.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.17 starting backfill to osd.10 from (0'0,0'0] MAX to 616'15385 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.16 starting backfill to osd.0 from (0'0,0'0] MAX to 614'15669 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.19 starting backfill to osd.10 from (0'0,0'0] MAX to 616'15219 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 616'15506 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.10 starting backfill to osd.6 from (0'0,0'0] MAX to 616'15330 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.11 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15597 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.14 starting backfill to osd.7 from (0'0,0'0] MAX to 616'15506 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.17 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15385 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.18 starting backfill to osd.0 from (0'0,0'0] MAX to 616'15686 2024-04-29T14:33:27.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:27 smithi040 ceph-mon[39581]: 6.18 starting backfill to osd.7 from (0'0,0'0] MAX to 616'15686 2024-04-29T14:33:27.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.13 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15592 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: osdmap e619: 12 total, 12 up, 7 in 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.11 starting backfill to osd.0 from (0'0,0'0] MAX to 616'15597 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.17 starting backfill to osd.10 from (0'0,0'0] MAX to 616'15385 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.16 starting backfill to osd.0 from (0'0,0'0] MAX to 614'15669 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.19 starting backfill to osd.10 from (0'0,0'0] MAX to 616'15219 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 616'15506 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.10 starting backfill to osd.6 from (0'0,0'0] MAX to 616'15330 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.11 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15597 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.14 starting backfill to osd.7 from (0'0,0'0] MAX to 616'15506 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.17 starting backfill to osd.11 from (0'0,0'0] MAX to 616'15385 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.18 starting backfill to osd.0 from (0'0,0'0] MAX to 616'15686 2024-04-29T14:33:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:27 smithi121 ceph-mon[44533]: 6.18 starting backfill to osd.7 from (0'0,0'0] MAX to 616'15686 2024-04-29T14:33:28.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:28 smithi040 ceph-mon[39581]: pgmap v1600: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 17 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5142/113488 objects misplaced (4.531%); 8.8 MiB/s, 141 objects/s recovering 2024-04-29T14:33:28.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:28 smithi194 ceph-mon[44089]: pgmap v1600: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 17 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5142/113488 objects misplaced (4.531%); 8.8 MiB/s, 141 objects/s recovering 2024-04-29T14:33:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:28.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:28.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:28 smithi121 ceph-mon[44533]: pgmap v1600: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 17 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5142/113488 objects misplaced (4.531%); 8.8 MiB/s, 141 objects/s recovering 2024-04-29T14:33:28.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:28.994 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:29.095 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:29.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:29.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:29.240 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:29.271 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:29.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:29.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:29.424 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:29.453 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:29.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:29.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:29.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:29.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:29.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:29 smithi194 ceph-mon[44089]: osdmap e620: 12 total, 12 up, 7 in 2024-04-29T14:33:29.596 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:29 smithi040 ceph-mon[39581]: osdmap e620: 12 total, 12 up, 7 in 2024-04-29T14:33:29.626 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:29.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:29.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:29.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:29.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:29.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:29 smithi121 ceph-mon[44533]: osdmap e620: 12 total, 12 up, 7 in 2024-04-29T14:33:29.788 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:29.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:29.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:29.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:29.977 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:33:29.977 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:33:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:29.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:30.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:30.133 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:30.269 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:30.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:30.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:30.413 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:33:30.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:30.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:30.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:30.595 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:30.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:30 smithi040 ceph-mon[39581]: osdmap e621: 12 total, 12 up, 7 in 2024-04-29T14:33:30.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/218047922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:33:30.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:30 smithi040 ceph-mon[39581]: 6.8 deep-scrub ok 2024-04-29T14:33:30.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:30 smithi040 ceph-mon[39581]: pgmap v1603: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 17 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5142/112694 objects misplaced (4.563%); 8.9 MiB/s, 141 objects/s recovering 2024-04-29T14:33:30.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:30.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:30.732 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:30.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:30 smithi194 ceph-mon[44089]: osdmap e621: 12 total, 12 up, 7 in 2024-04-29T14:33:30.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/218047922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:33:30.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:30 smithi194 ceph-mon[44089]: 6.8 deep-scrub ok 2024-04-29T14:33:30.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:30 smithi194 ceph-mon[44089]: pgmap v1603: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 17 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5142/112694 objects misplaced (4.563%); 8.9 MiB/s, 141 objects/s recovering 2024-04-29T14:33:30.763 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:30.789 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:30.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:30 smithi121 ceph-mon[44533]: osdmap e621: 12 total, 12 up, 7 in 2024-04-29T14:33:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/218047922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:33:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:30 smithi121 ceph-mon[44533]: 6.8 deep-scrub ok 2024-04-29T14:33:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:30 smithi121 ceph-mon[44533]: pgmap v1603: 27 pgs: 6 remapped+peering, 1 active+clean+scrubbing+deep, 3 active+remapped+backfilling, 17 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5142/112694 objects misplaced (4.563%); 8.9 MiB/s, 141 objects/s recovering 2024-04-29T14:33:30.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:30.925 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:31.280 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:31.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:31.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:33:31.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:31.467 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:31.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:31.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:31.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:31 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:31.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/218047922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:33:31.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:31 smithi040 ceph-mon[39581]: osdmap e622: 12 total, 12 up, 7 in 2024-04-29T14:33:31.606 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:31.608 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:31.647 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:31.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:31.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:31.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:31 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:31.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/218047922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:33:31.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:31 smithi194 ceph-mon[44089]: osdmap e622: 12 total, 12 up, 7 in 2024-04-29T14:33:31.811 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:31.846 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:31.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:31.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:31 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:31.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/218047922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:33:31.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:31 smithi121 ceph-mon[44533]: osdmap e622: 12 total, 12 up, 7 in 2024-04-29T14:33:31.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:32.012 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:32.052 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:32.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:32.139 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:31 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:33:31.940+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e622 definitely_dead 0 2024-04-29T14:33:32.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:32.218 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:32.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:32.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:32.322 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.2. 2024-04-29T14:33:32.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:32.409 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1227280918' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T14:33:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:32 smithi121 ceph-mon[44533]: from='client.26848 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T14:33:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:32 smithi121 ceph-mon[44533]: pgmap v1605: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 3.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 10576/108258 objects misplaced (9.769%); 83 MiB/s, 1.33k objects/s recovering 2024-04-29T14:33:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:32 smithi121 ceph-mon[44533]: from='client.26848 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-29T14:33:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:32 smithi121 ceph-mon[44533]: osdmap e623: 12 total, 12 up, 8 in 2024-04-29T14:33:32.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1227280918' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T14:33:32.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:32 smithi194 ceph-mon[44089]: from='client.26848 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T14:33:32.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:32 smithi194 ceph-mon[44089]: pgmap v1605: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 3.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 10576/108258 objects misplaced (9.769%); 83 MiB/s, 1.33k objects/s recovering 2024-04-29T14:33:32.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:32 smithi194 ceph-mon[44089]: from='client.26848 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-29T14:33:32.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:32 smithi194 ceph-mon[44089]: osdmap e623: 12 total, 12 up, 8 in 2024-04-29T14:33:32.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1227280918' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T14:33:32.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:32 smithi040 ceph-mon[39581]: from='client.26848 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T14:33:32.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:32 smithi040 ceph-mon[39581]: pgmap v1605: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 3.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 10576/108258 objects misplaced (9.769%); 83 MiB/s, 1.33k objects/s recovering 2024-04-29T14:33:32.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:32 smithi040 ceph-mon[39581]: from='client.26848 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-29T14:33:32.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:32 smithi040 ceph-mon[39581]: osdmap e623: 12 total, 12 up, 8 in 2024-04-29T14:33:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:32.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:33.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:33.103 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:33.253 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:33.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:33.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:33.393 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:33.593 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:33.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:33.697 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:33.724 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:33.743 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:33.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:33.847 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:33.877 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:34.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:34.332 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:34.387 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:34.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:34 smithi194 ceph-mon[44089]: 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15663 2024-04-29T14:33:34.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:34 smithi194 ceph-mon[44089]: osdmap e624: 12 total, 12 up, 8 in 2024-04-29T14:33:34.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:34 smithi194 ceph-mon[44089]: 6.17 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15527 2024-04-29T14:33:34.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:34 smithi194 ceph-mon[44089]: 6.14 continuing backfill to osd.4 from (552'5400,622'15663] 6:2b721ce3:::benchmark_data_smithi194_134202_object136531:head to 622'15663 2024-04-29T14:33:34.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:34 smithi194 ceph-mon[44089]: 6.17 continuing backfill to osd.10 from (552'5300,622'15527] 6:ec3e7ed8:::benchmark_data_smithi194_134202_object124712:head to 622'15527 2024-04-29T14:33:34.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:34.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:34.583 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:33:34.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:34 smithi040 ceph-mon[39581]: 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15663 2024-04-29T14:33:34.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:34 smithi040 ceph-mon[39581]: osdmap e624: 12 total, 12 up, 8 in 2024-04-29T14:33:34.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:34 smithi040 ceph-mon[39581]: 6.17 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15527 2024-04-29T14:33:34.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:34 smithi040 ceph-mon[39581]: 6.14 continuing backfill to osd.4 from (552'5400,622'15663] 6:2b721ce3:::benchmark_data_smithi194_134202_object136531:head to 622'15663 2024-04-29T14:33:34.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:34 smithi040 ceph-mon[39581]: 6.17 continuing backfill to osd.10 from (552'5300,622'15527] 6:ec3e7ed8:::benchmark_data_smithi194_134202_object124712:head to 622'15527 2024-04-29T14:33:34.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:34.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:34.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:34 smithi121 ceph-mon[44533]: 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15663 2024-04-29T14:33:34.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:34 smithi121 ceph-mon[44533]: osdmap e624: 12 total, 12 up, 8 in 2024-04-29T14:33:34.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:34 smithi121 ceph-mon[44533]: 6.17 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15527 2024-04-29T14:33:34.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:34 smithi121 ceph-mon[44533]: 6.14 continuing backfill to osd.4 from (552'5400,622'15663] 6:2b721ce3:::benchmark_data_smithi194_134202_object136531:head to 622'15663 2024-04-29T14:33:34.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:34 smithi121 ceph-mon[44533]: 6.17 continuing backfill to osd.10 from (552'5300,622'15527] 6:ec3e7ed8:::benchmark_data_smithi194_134202_object124712:head to 622'15527 2024-04-29T14:33:34.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:34.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:34.738 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-29T14:33:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:34.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:34.965 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:34.997 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:35.018 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:35.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:35.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:35.163 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:35.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:35.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:35.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:35.330 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:35.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:35.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:35.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:35 smithi040 ceph-mon[39581]: pgmap v1608: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 3.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 10576/106630 objects misplaced (9.918%); 55 MiB/s, 884 objects/s recovering 2024-04-29T14:33:35.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:35 smithi040 ceph-mon[39581]: 6.19 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15385 2024-04-29T14:33:35.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:35 smithi040 ceph-mon[39581]: 6.f starting backfill to osd.2 from (0'0,0'0] MAX to 622'15809 2024-04-29T14:33:35.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:35 smithi040 ceph-mon[39581]: 6.2 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15759 2024-04-29T14:33:35.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:35 smithi040 ceph-mon[39581]: 6.e starting backfill to osd.2 from (0'0,0'0] MAX to 622'15719 2024-04-29T14:33:35.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:35 smithi040 ceph-mon[39581]: 6.0 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15470 2024-04-29T14:33:35.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:35 smithi040 ceph-mon[39581]: osdmap e625: 12 total, 12 up, 8 in 2024-04-29T14:33:35.498 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:35.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:35 smithi121 ceph-mon[44533]: pgmap v1608: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 3.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 10576/106630 objects misplaced (9.918%); 55 MiB/s, 884 objects/s recovering 2024-04-29T14:33:35.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:35 smithi121 ceph-mon[44533]: 6.19 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15385 2024-04-29T14:33:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:35 smithi121 ceph-mon[44533]: 6.f starting backfill to osd.2 from (0'0,0'0] MAX to 622'15809 2024-04-29T14:33:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:35 smithi121 ceph-mon[44533]: 6.2 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15759 2024-04-29T14:33:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:35 smithi121 ceph-mon[44533]: 6.e starting backfill to osd.2 from (0'0,0'0] MAX to 622'15719 2024-04-29T14:33:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:35 smithi121 ceph-mon[44533]: 6.0 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15470 2024-04-29T14:33:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:35 smithi121 ceph-mon[44533]: osdmap e625: 12 total, 12 up, 8 in 2024-04-29T14:33:35.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:35 smithi194 ceph-mon[44089]: pgmap v1608: 27 pgs: 8 active+remapped+backfilling, 19 active+clean; 3.3 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 10576/106630 objects misplaced (9.918%); 55 MiB/s, 884 objects/s recovering 2024-04-29T14:33:35.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:35 smithi194 ceph-mon[44089]: 6.19 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15385 2024-04-29T14:33:35.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:35 smithi194 ceph-mon[44089]: 6.f starting backfill to osd.2 from (0'0,0'0] MAX to 622'15809 2024-04-29T14:33:35.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:35 smithi194 ceph-mon[44089]: 6.2 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15759 2024-04-29T14:33:35.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:35 smithi194 ceph-mon[44089]: 6.e starting backfill to osd.2 from (0'0,0'0] MAX to 622'15719 2024-04-29T14:33:35.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:35 smithi194 ceph-mon[44089]: 6.0 starting backfill to osd.2 from (0'0,0'0] MAX to 622'15470 2024-04-29T14:33:35.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:35 smithi194 ceph-mon[44089]: osdmap e625: 12 total, 12 up, 8 in 2024-04-29T14:33:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:36.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:36.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:36.298 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:36.321 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:36.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:36.399 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:36 smithi040 ceph-mon[39581]: osdmap e626: 12 total, 12 up, 8 in 2024-04-29T14:33:36.399 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:36 smithi040 ceph-mon[39581]: pgmap v1611: 27 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 remapped+peering, 6 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 8813/103090 objects misplaced (8.549%); 100 MiB/s, 1.59k objects/s recovering 2024-04-29T14:33:36.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:36.473 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:36.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:36 smithi194 ceph-mon[44089]: osdmap e626: 12 total, 12 up, 8 in 2024-04-29T14:33:36.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:36 smithi194 ceph-mon[44089]: pgmap v1611: 27 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 remapped+peering, 6 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 8813/103090 objects misplaced (8.549%); 100 MiB/s, 1.59k objects/s recovering 2024-04-29T14:33:36.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:36 smithi121 ceph-mon[44533]: osdmap e626: 12 total, 12 up, 8 in 2024-04-29T14:33:36.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:36 smithi121 ceph-mon[44533]: pgmap v1611: 27 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 remapped+peering, 6 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 8813/103090 objects misplaced (8.549%); 100 MiB/s, 1.59k objects/s recovering 2024-04-29T14:33:36.679 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:36.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:36.724 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:33:36.724 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:33:36.785 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:36.815 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:36.837 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:36.840 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:36.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:36.982 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:37.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:37.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:37.149 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:37.156 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:33:37.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:33:37.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:33:37.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:33:37.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:33:37.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:33:37.165 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:37.170 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:33:37.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:37.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:37.205 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:33:37.237 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:33:37.268 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:33:37.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:37.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:33:37.310 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:37.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:33:37.333 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:33:37.334 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:37.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:37.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:37.480 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:37.514 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:37.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:37.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:37 smithi040 ceph-mon[39581]: osdmap e627: 12 total, 12 up, 8 in 2024-04-29T14:33:37.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2675374463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:37.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:37.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:37.619 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:37 smithi121 ceph-mon[44533]: osdmap e627: 12 total, 12 up, 8 in 2024-04-29T14:33:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2675374463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:37.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:37.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:37.660 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:37 smithi194 ceph-mon[44089]: osdmap e627: 12 total, 12 up, 8 in 2024-04-29T14:33:37.660 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2675374463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:37.660 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:37.660 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:37.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:37.692 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:37.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:37.798 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:37.841 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:37.870 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:37.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:37.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:38.021 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:38.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:38.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:38.204 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:38.232 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:38.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:38.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:38.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: osdmap e628: 12 total, 12 up, 8 in 2024-04-29T14:33:38.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:38 smithi121 ceph-mon[44533]: pgmap v1614: 27 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 remapped+peering, 5 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 10132/102040 objects misplaced (9.929%); 133 MiB/s, 2.13k objects/s recovering 2024-04-29T14:33:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:38.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: osdmap e628: 12 total, 12 up, 8 in 2024-04-29T14:33:38.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:38 smithi194 ceph-mon[44089]: pgmap v1614: 27 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 remapped+peering, 5 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 10132/102040 objects misplaced (9.929%); 133 MiB/s, 2.13k objects/s recovering 2024-04-29T14:33:38.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: osdmap e628: 12 total, 12 up, 8 in 2024-04-29T14:33:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:38 smithi040 ceph-mon[39581]: pgmap v1614: 27 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 remapped+peering, 5 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 10132/102040 objects misplaced (9.929%); 133 MiB/s, 2.13k objects/s recovering 2024-04-29T14:33:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:39.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:39.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:33:39.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:39.370 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:39.389 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:39.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:39.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:39.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:39 smithi040 ceph-mon[39581]: osdmap e629: 12 total, 12 up, 8 in 2024-04-29T14:33:39.524 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:39.602 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:39.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:39.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:39 smithi121 ceph-mon[44533]: osdmap e629: 12 total, 12 up, 8 in 2024-04-29T14:33:39.679 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:39.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:39.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:39 smithi194 ceph-mon[44089]: osdmap e629: 12 total, 12 up, 8 in 2024-04-29T14:33:39.737 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:39.738 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2] out_osds: [1, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:33:39.739 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:33:39.739 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-29T14:33:39.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-04-29T14:33:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:39.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:39.862 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:39.890 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:39.910 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:39.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:40.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:40.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:40.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:40.192 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:40.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:40.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:40.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:40.353 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:40.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:40.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:40.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:40 smithi040 ceph-mon[39581]: osdmap e630: 12 total, 12 up, 8 in 2024-04-29T14:33:40.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:40 smithi040 ceph-mon[39581]: pgmap v1617: 27 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 10132/101434 objects misplaced (9.989%); 57 MiB/s, 908 objects/s recovering 2024-04-29T14:33:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:40 smithi121 ceph-mon[44533]: osdmap e630: 12 total, 12 up, 8 in 2024-04-29T14:33:40.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:40 smithi121 ceph-mon[44533]: pgmap v1617: 27 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 10132/101434 objects misplaced (9.989%); 57 MiB/s, 908 objects/s recovering 2024-04-29T14:33:40.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:33:40.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:40.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:40.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:40 smithi194 ceph-mon[44089]: osdmap e630: 12 total, 12 up, 8 in 2024-04-29T14:33:40.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:40 smithi194 ceph-mon[44089]: pgmap v1617: 27 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 18 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 206 op/s; 10132/101434 objects misplaced (9.989%); 57 MiB/s, 908 objects/s recovering 2024-04-29T14:33:41.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:41 smithi040 ceph-mon[39581]: osdmap e631: 12 total, 12 up, 8 in 2024-04-29T14:33:41.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/810565524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:41.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:41 smithi121 ceph-mon[44533]: osdmap e631: 12 total, 12 up, 8 in 2024-04-29T14:33:41.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/810565524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:41.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:41 smithi194 ceph-mon[44089]: osdmap e631: 12 total, 12 up, 8 in 2024-04-29T14:33:41.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/810565524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:33:42.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:42 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:42.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/810565524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:42.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:42 smithi121 ceph-mon[44533]: osdmap e632: 12 total, 12 up, 8 in 2024-04-29T14:33:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:42 smithi121 ceph-mon[44533]: pgmap v1620: 27 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 19 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 515 op/s; 8926/97914 objects misplaced (9.116%); 39 MiB/s, 630 objects/s recovering 2024-04-29T14:33:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:42 smithi121 ceph-mon[44533]: osdmap e633: 12 total, 12 up, 8 in 2024-04-29T14:33:42.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:42 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:42.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/810565524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:42.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:42 smithi194 ceph-mon[44089]: osdmap e632: 12 total, 12 up, 8 in 2024-04-29T14:33:42.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:42.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:42.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:42 smithi194 ceph-mon[44089]: pgmap v1620: 27 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 19 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 515 op/s; 8926/97914 objects misplaced (9.116%); 39 MiB/s, 630 objects/s recovering 2024-04-29T14:33:42.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:42 smithi194 ceph-mon[44089]: osdmap e633: 12 total, 12 up, 8 in 2024-04-29T14:33:42.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:42 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:42.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/810565524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:33:42.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:42 smithi040 ceph-mon[39581]: osdmap e632: 12 total, 12 up, 8 in 2024-04-29T14:33:42.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:42.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:42.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:42 smithi040 ceph-mon[39581]: pgmap v1620: 27 pgs: 1 peering, 1 active+clean+remapped, 6 active+remapped+backfilling, 19 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 515 op/s; 8926/97914 objects misplaced (9.116%); 39 MiB/s, 630 objects/s recovering 2024-04-29T14:33:42.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:42 smithi040 ceph-mon[39581]: osdmap e633: 12 total, 12 up, 8 in 2024-04-29T14:33:44.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:44.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:44.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:44 smithi194 ceph-mon[44089]: osdmap e634: 12 total, 12 up, 8 in 2024-04-29T14:33:44.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:44 smithi194 ceph-mon[44089]: pgmap v1623: 27 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 23 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 3660/94942 objects misplaced (3.855%); 84 MiB/s, 1.34k objects/s recovering 2024-04-29T14:33:44.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:44.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:44.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:44 smithi040 ceph-mon[39581]: osdmap e634: 12 total, 12 up, 8 in 2024-04-29T14:33:44.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:44 smithi040 ceph-mon[39581]: pgmap v1623: 27 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 23 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 3660/94942 objects misplaced (3.855%); 84 MiB/s, 1.34k objects/s recovering 2024-04-29T14:33:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:44 smithi121 ceph-mon[44533]: osdmap e634: 12 total, 12 up, 8 in 2024-04-29T14:33:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:44 smithi121 ceph-mon[44533]: pgmap v1623: 27 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 23 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 3660/94942 objects misplaced (3.855%); 84 MiB/s, 1.34k objects/s recovering 2024-04-29T14:33:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:45.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:45.221 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:45.256 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:45.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:45.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:45.424 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:45.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:45.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:45.548 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:46.239 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:46.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:46.344 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:33:46.370 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:46.390 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:46.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:46.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:46.494 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:46.523 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:46.561 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:46.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:46.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:46.695 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:47.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:47.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:47.188 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:47.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:47.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:47.318 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:47 smithi194 ceph-mon[44089]: osdmap e635: 12 total, 12 up, 8 in 2024-04-29T14:33:47.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:47 smithi194 ceph-mon[44089]: pgmap v1625: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2694/88972 objects misplaced (3.028%); 64 MiB/s, 1.03k objects/s recovering 2024-04-29T14:33:47.322 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:47.347 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:47 smithi040 ceph-mon[39581]: osdmap e635: 12 total, 12 up, 8 in 2024-04-29T14:33:47.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:47 smithi040 ceph-mon[39581]: pgmap v1625: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2694/88972 objects misplaced (3.028%); 64 MiB/s, 1.03k objects/s recovering 2024-04-29T14:33:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:47.375 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:47.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:47.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:47 smithi121 ceph-mon[44533]: osdmap e635: 12 total, 12 up, 8 in 2024-04-29T14:33:47.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:47 smithi121 ceph-mon[44533]: pgmap v1625: 27 pgs: 2 active+remapped+backfilling, 25 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2694/88972 objects misplaced (3.028%); 64 MiB/s, 1.03k objects/s recovering 2024-04-29T14:33:47.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:47.513 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:47.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:47.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:47.641 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:47.667 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:47.686 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:47.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:47.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:47.817 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:47.823 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:33:47.823 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:33:47.823 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:33:47.823 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:33:47.823 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:33:47.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:47.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:47.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:47.962 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:47.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:47.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:47.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:48.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:48.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:48 smithi121 ceph-mon[44533]: osdmap e636: 12 total, 12 up, 8 in 2024-04-29T14:33:48.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:48 smithi121 ceph-mon[44533]: osdmap e637: 12 total, 12 up, 8 in 2024-04-29T14:33:48.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:48 smithi194 ceph-mon[44089]: osdmap e636: 12 total, 12 up, 8 in 2024-04-29T14:33:48.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:48 smithi194 ceph-mon[44089]: osdmap e637: 12 total, 12 up, 8 in 2024-04-29T14:33:48.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:48 smithi040 ceph-mon[39581]: osdmap e636: 12 total, 12 up, 8 in 2024-04-29T14:33:48.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:48 smithi040 ceph-mon[39581]: osdmap e637: 12 total, 12 up, 8 in 2024-04-29T14:33:48.371 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:48.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:48.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:48.533 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:48.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:48.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:48.688 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:48.718 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:48.742 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:48.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:48.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:48.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:48.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:49.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:49.075 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:33:49.092 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:49.325 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:49 smithi121 ceph-mon[44533]: pgmap v1628: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 623/81832 objects misplaced (0.761%); 32 MiB/s, 508 objects/s recovering 2024-04-29T14:33:49.326 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:49 smithi121 ceph-mon[44533]: osdmap e638: 12 total, 12 up, 8 in 2024-04-29T14:33:49.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:33:49.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:49 smithi040 ceph-mon[39581]: pgmap v1628: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 623/81832 objects misplaced (0.761%); 32 MiB/s, 508 objects/s recovering 2024-04-29T14:33:49.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:49 smithi040 ceph-mon[39581]: osdmap e638: 12 total, 12 up, 8 in 2024-04-29T14:33:49.818 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:49 smithi194 ceph-mon[44089]: pgmap v1628: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 623/81832 objects misplaced (0.761%); 32 MiB/s, 508 objects/s recovering 2024-04-29T14:33:49.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:49 smithi194 ceph-mon[44089]: osdmap e638: 12 total, 12 up, 8 in 2024-04-29T14:33:49.850 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:49.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:49.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:50.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:50.145 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:50.174 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:50.200 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:50.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:50.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:33:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:50.373 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:50 smithi121 ceph-mon[44533]: pgmap v1630: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 623/80548 objects misplaced (0.773%); 28 MiB/s, 450 objects/s recovering 2024-04-29T14:33:50.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:50 smithi040 ceph-mon[39581]: pgmap v1630: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 623/80548 objects misplaced (0.773%); 28 MiB/s, 450 objects/s recovering 2024-04-29T14:33:50.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:50 smithi194 ceph-mon[44089]: pgmap v1630: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 623/80548 objects misplaced (0.773%); 28 MiB/s, 450 objects/s recovering 2024-04-29T14:33:50.653 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:51.912 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:33:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:51.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:33:52.017 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:52.049 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:52.075 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:52.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:52.181 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:52.213 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:52.236 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:52.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:52.340 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:52.373 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:52.400 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:52.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:52.507 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:33:52.541 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:52.564 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:33:52.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:52.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:33:52.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:52.704 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:53.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:53 smithi040 ceph-mon[39581]: pgmap v1631: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 623/75142 objects misplaced (0.829%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:33:53.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:53 smithi121 ceph-mon[44533]: pgmap v1631: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 623/75142 objects misplaced (0.829%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:33:53.458 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:53.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:53.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-04-29T14:33:53.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:53.568 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:53 smithi194 ceph-mon[44089]: pgmap v1631: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 623/75142 objects misplaced (0.829%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:33:53.602 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:53.635 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:53.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:53.743 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:53.783 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:53.809 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:53.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:53.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:33:54.016 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:54.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:54.023 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:33:54.054 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:33:54.081 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:33:54.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:33:54.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:54.152 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:33:54.174 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:54.182 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:33:54.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:54.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:54.225 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:33:54.266 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:33:54.300 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:33:54.311 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:54.324 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:33:54.343 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:54.363 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:33:54.364 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:54.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:54.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2545681574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:54.370 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:33:54.370 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:33:54.370 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:33:54.370 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:33:54.370 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:33:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:33:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:33:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:33:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:33:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:33:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:33:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:33:54.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2545681574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:54.469 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:54.505 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2545681574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:33:54.507 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:54.531 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:54.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:54.638 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:54.671 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:54.697 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:54.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:54.804 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:33:54.834 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:54.859 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:33:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:54.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:33:54.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:55.010 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:55.043 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:55.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:55.133 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:33:55.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:55.188 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:55.231 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:55.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:55.336 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:55.372 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:55.389 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: pgmap v1632: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 11 MiB/s, 181 objects/s recovering 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.390 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1525216769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:55.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:55 smithi121 ceph-mon[44533]: from='client.26875 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:55.394 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:55.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: pgmap v1632: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 11 MiB/s, 181 objects/s recovering 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.399 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.399 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.399 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.399 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.399 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1525216769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:55.399 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:55 smithi194 ceph-mon[44089]: from='client.26875 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:55.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:55.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: pgmap v1632: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 11 MiB/s, 181 objects/s recovering 2024-04-29T14:33:55.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1525216769' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:55.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:55 smithi040 ceph-mon[39581]: from='client.26875 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:33:55.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:55.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:55.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:33:55.674 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:33:55.707 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:56.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:56 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:56.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:56 smithi040 ceph-mon[39581]: from='client.26875 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:33:56.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:56 smithi040 ceph-mon[39581]: osdmap e639: 12 total, 12 up, 8 in 2024-04-29T14:33:56.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:56 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:56.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:56 smithi194 ceph-mon[44089]: from='client.26875 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:33:56.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:56 smithi194 ceph-mon[44089]: osdmap e639: 12 total, 12 up, 8 in 2024-04-29T14:33:56.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:56 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:33:56.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:56 smithi121 ceph-mon[44533]: from='client.26875 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:33:56.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:56 smithi121 ceph-mon[44533]: osdmap e639: 12 total, 12 up, 8 in 2024-04-29T14:33:56.860 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:33:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:56.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:33:56.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:57.016 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:57.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:57.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:57.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:57.207 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:57.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:57.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:57.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:57.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: 6.16 scrub starts 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: 6.16 scrub ok 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: 6.2 scrub starts 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: pgmap v1634: 27 pgs: 27 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: 6.2 scrub ok 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: osdmap e640: 12 total, 12 up, 8 in 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: 6.11 scrub starts 2024-04-29T14:33:57.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:57 smithi121 ceph-mon[44533]: 6.11 scrub ok 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: 6.16 scrub starts 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: 6.16 scrub ok 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: 6.2 scrub starts 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: pgmap v1634: 27 pgs: 27 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: 6.2 scrub ok 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: osdmap e640: 12 total, 12 up, 8 in 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: 6.11 scrub starts 2024-04-29T14:33:57.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:57 smithi040 ceph-mon[39581]: 6.11 scrub ok 2024-04-29T14:33:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:57.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:57.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:57.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:33:57.569 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: 6.16 scrub starts 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: 6.16 scrub ok 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: 6.2 scrub starts 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: pgmap v1634: 27 pgs: 27 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: 6.2 scrub ok 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: osdmap e640: 12 total, 12 up, 8 in 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: 6.11 scrub starts 2024-04-29T14:33:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:57 smithi194 ceph-mon[44089]: 6.11 scrub ok 2024-04-29T14:33:57.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:33:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:57.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:33:57.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:57.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:57.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:33:57.999 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:58.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:33:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:33:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:58.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:33:58.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:58.155 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:33:58.155 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:33:58.165 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:58.188 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:58.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:58.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:33:58.336 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:58 smithi040 ceph-mon[39581]: 6.17 scrub starts 2024-04-29T14:33:58.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:58 smithi040 ceph-mon[39581]: 6.17 scrub ok 2024-04-29T14:33:58.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:58 smithi040 ceph-mon[39581]: 6.15 scrub starts 2024-04-29T14:33:58.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:58 smithi040 ceph-mon[39581]: 6.15 scrub ok 2024-04-29T14:33:58.343 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:58.456 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:33:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:58.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:33:58.562 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:33:58.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:58 smithi194 ceph-mon[44089]: 6.17 scrub starts 2024-04-29T14:33:58.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:58 smithi194 ceph-mon[44089]: 6.17 scrub ok 2024-04-29T14:33:58.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:58 smithi194 ceph-mon[44089]: 6.15 scrub starts 2024-04-29T14:33:58.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:58 smithi194 ceph-mon[44089]: 6.15 scrub ok 2024-04-29T14:33:58.592 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:58.616 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:33:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:58.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:33:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:58 smithi121 ceph-mon[44533]: 6.17 scrub starts 2024-04-29T14:33:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:58 smithi121 ceph-mon[44533]: 6.17 scrub ok 2024-04-29T14:33:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:58 smithi121 ceph-mon[44533]: 6.15 scrub starts 2024-04-29T14:33:58.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:58 smithi121 ceph-mon[44533]: 6.15 scrub ok 2024-04-29T14:33:58.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:33:58.758 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:58.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:33:59.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:33:59.410 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:33:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:59.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:33:59.474 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:33:59.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:33:59.560 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:59.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:33:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:59.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:33:59.606 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.f scrub starts 2024-04-29T14:33:59.606 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.12 scrub starts 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: pgmap v1636: 27 pgs: 27 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.12 scrub ok 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.f scrub ok 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.19 scrub starts 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.5 scrub starts 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.19 scrub ok 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.5 scrub ok 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: 6.13 scrub starts 2024-04-29T14:33:59.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:33:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:33:59.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.f scrub starts 2024-04-29T14:33:59.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.12 scrub starts 2024-04-29T14:33:59.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: pgmap v1636: 27 pgs: 27 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:33:59.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.12 scrub ok 2024-04-29T14:33:59.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.f scrub ok 2024-04-29T14:33:59.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.19 scrub starts 2024-04-29T14:33:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:33:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:33:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.5 scrub starts 2024-04-29T14:33:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.19 scrub ok 2024-04-29T14:33:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.5 scrub ok 2024-04-29T14:33:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: 6.13 scrub starts 2024-04-29T14:33:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:33:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:33:59.707 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:33:59.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.f scrub starts 2024-04-29T14:33:59.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.12 scrub starts 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: pgmap v1636: 27 pgs: 27 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.12 scrub ok 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.f scrub ok 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.19 scrub starts 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.5 scrub starts 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.19 scrub ok 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.5 scrub ok 2024-04-29T14:33:59.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: 6.13 scrub starts 2024-04-29T14:33:59.737 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:33:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:33:59.737 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:59.760 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:33:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:59.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:33:59.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:33:59.913 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:33:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:33:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:33:59.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:00.050 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:00.082 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:00.111 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:00.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:00.217 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:34:00.252 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:00.275 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:34:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:00.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:34:00.344 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:00.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:00.424 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:00.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:00.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:00.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: 6.4 scrub starts 2024-04-29T14:34:00.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: 6.13 scrub ok 2024-04-29T14:34:00.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: 6.4 scrub ok 2024-04-29T14:34:00.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: osdmap e641: 12 total, 12 up, 8 in 2024-04-29T14:34:00.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: 6.3 scrub starts 2024-04-29T14:34:00.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: 6.3 scrub ok 2024-04-29T14:34:00.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: 6.10 scrub starts 2024-04-29T14:34:00.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: 6.10 scrub ok 2024-04-29T14:34:00.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:00 smithi040 ceph-mon[39581]: pgmap v1638: 27 pgs: 27 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:34:00.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:00.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: 6.4 scrub starts 2024-04-29T14:34:00.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: 6.13 scrub ok 2024-04-29T14:34:00.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: 6.4 scrub ok 2024-04-29T14:34:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: osdmap e641: 12 total, 12 up, 8 in 2024-04-29T14:34:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: 6.3 scrub starts 2024-04-29T14:34:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: 6.3 scrub ok 2024-04-29T14:34:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: 6.10 scrub starts 2024-04-29T14:34:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: 6.10 scrub ok 2024-04-29T14:34:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:00 smithi121 ceph-mon[44533]: pgmap v1638: 27 pgs: 27 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: 6.4 scrub starts 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: 6.13 scrub ok 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: 6.4 scrub ok 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: osdmap e641: 12 total, 12 up, 8 in 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: 6.3 scrub starts 2024-04-29T14:34:00.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: 6.3 scrub ok 2024-04-29T14:34:00.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: 6.10 scrub starts 2024-04-29T14:34:00.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: 6.10 scrub ok 2024-04-29T14:34:00.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:00 smithi194 ceph-mon[44089]: pgmap v1638: 27 pgs: 27 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:34:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:01.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:01 smithi040 ceph-mon[39581]: 6.9 scrub starts 2024-04-29T14:34:01.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:01 smithi040 ceph-mon[39581]: 6.9 scrub ok 2024-04-29T14:34:01.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:01 smithi040 ceph-mon[39581]: 6.18 scrub starts 2024-04-29T14:34:01.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:01 smithi040 ceph-mon[39581]: 6.18 scrub ok 2024-04-29T14:34:01.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:01 smithi121 ceph-mon[44533]: 6.9 scrub starts 2024-04-29T14:34:01.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:01 smithi121 ceph-mon[44533]: 6.9 scrub ok 2024-04-29T14:34:01.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:01 smithi121 ceph-mon[44533]: 6.18 scrub starts 2024-04-29T14:34:01.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:01 smithi121 ceph-mon[44533]: 6.18 scrub ok 2024-04-29T14:34:01.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:01 smithi194 ceph-mon[44089]: 6.9 scrub starts 2024-04-29T14:34:01.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:01 smithi194 ceph-mon[44089]: 6.9 scrub ok 2024-04-29T14:34:01.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:01 smithi194 ceph-mon[44089]: 6.18 scrub starts 2024-04-29T14:34:01.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:01 smithi194 ceph-mon[44089]: 6.18 scrub ok 2024-04-29T14:34:01.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-04-29T14:34:02.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:02 smithi040 ceph-mon[39581]: 6.14 scrub starts 2024-04-29T14:34:02.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:02 smithi040 ceph-mon[39581]: 6.14 scrub ok 2024-04-29T14:34:02.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:02.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:02.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:02 smithi040 ceph-mon[39581]: 6.b scrub starts 2024-04-29T14:34:02.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:02 smithi040 ceph-mon[39581]: pgmap v1639: 27 pgs: 27 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:34:02.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:02 smithi040 ceph-mon[39581]: osdmap e642: 12 total, 12 up, 8 in 2024-04-29T14:34:02.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:02 smithi121 ceph-mon[44533]: 6.14 scrub starts 2024-04-29T14:34:02.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:02 smithi121 ceph-mon[44533]: 6.14 scrub ok 2024-04-29T14:34:02.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:02.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:02.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:02 smithi121 ceph-mon[44533]: 6.b scrub starts 2024-04-29T14:34:02.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:02 smithi121 ceph-mon[44533]: pgmap v1639: 27 pgs: 27 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:34:02.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:02 smithi121 ceph-mon[44533]: osdmap e642: 12 total, 12 up, 8 in 2024-04-29T14:34:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:02 smithi194 ceph-mon[44089]: 6.14 scrub starts 2024-04-29T14:34:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:02 smithi194 ceph-mon[44089]: 6.14 scrub ok 2024-04-29T14:34:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:02 smithi194 ceph-mon[44089]: 6.b scrub starts 2024-04-29T14:34:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:02 smithi194 ceph-mon[44089]: pgmap v1639: 27 pgs: 27 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail 2024-04-29T14:34:02.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:02 smithi194 ceph-mon[44089]: osdmap e642: 12 total, 12 up, 8 in 2024-04-29T14:34:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:03.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:03.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:03.347 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:03.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:03.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:03.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:03.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:03 smithi040 ceph-mon[39581]: 6.b scrub ok 2024-04-29T14:34:03.522 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:03.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:03 smithi121 ceph-mon[44533]: 6.b scrub ok 2024-04-29T14:34:03.652 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:03.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:03.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:03.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:03 smithi194 ceph-mon[44089]: 6.b scrub ok 2024-04-29T14:34:03.792 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:03.825 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:03.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:03.933 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:03.966 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:03.988 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:03.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:04.095 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:34:04.135 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:34:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:04.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:34:04.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:04.318 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:04.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:04.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:04.459 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:04.494 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:04 smithi194 ceph-mon[44089]: pgmap v1641: 27 pgs: 27 active+clean; 1.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-04-29T14:34:04.495 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:04.526 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:04.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:04.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:04.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:04 smithi121 ceph-mon[44533]: pgmap v1641: 27 pgs: 27 active+clean; 1.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-04-29T14:34:04.686 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:04.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:04.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:04.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:04 smithi040 ceph-mon[39581]: pgmap v1641: 27 pgs: 27 active+clean; 1.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail 2024-04-29T14:34:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:04.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:04.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:05.024 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:05.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:05.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:05.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:05 smithi040 ceph-mon[39581]: osdmap e643: 12 total, 12 up, 8 in 2024-04-29T14:34:05.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:05 smithi040 ceph-mon[39581]: 6.d scrub starts 2024-04-29T14:34:05.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:05 smithi040 ceph-mon[39581]: 6.d scrub ok 2024-04-29T14:34:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:05.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:05 smithi121 ceph-mon[44533]: osdmap e643: 12 total, 12 up, 8 in 2024-04-29T14:34:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:05 smithi121 ceph-mon[44533]: 6.d scrub starts 2024-04-29T14:34:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:05 smithi121 ceph-mon[44533]: 6.d scrub ok 2024-04-29T14:34:05.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:05.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:05.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:05 smithi194 ceph-mon[44089]: osdmap e643: 12 total, 12 up, 8 in 2024-04-29T14:34:05.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:05 smithi194 ceph-mon[44089]: 6.d scrub starts 2024-04-29T14:34:05.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:05 smithi194 ceph-mon[44089]: 6.d scrub ok 2024-04-29T14:34:06.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:06.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:06.163 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:06.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:06.204 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:06.387 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:34:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:06 smithi121 ceph-mon[44533]: 6.a scrub starts 2024-04-29T14:34:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:06 smithi121 ceph-mon[44533]: 6.a scrub ok 2024-04-29T14:34:06.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:06 smithi121 ceph-mon[44533]: pgmap v1643: 27 pgs: 27 active+clean; 1.0 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-04-29T14:34:06.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/114616372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:06.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:06 smithi121 ceph-mon[44533]: from='client.36638 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:06.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:06 smithi040 ceph-mon[39581]: 6.a scrub starts 2024-04-29T14:34:06.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:06 smithi040 ceph-mon[39581]: 6.a scrub ok 2024-04-29T14:34:06.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:06 smithi040 ceph-mon[39581]: pgmap v1643: 27 pgs: 27 active+clean; 1.0 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-04-29T14:34:06.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/114616372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:06.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:06 smithi040 ceph-mon[39581]: from='client.36638 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:06.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:06 smithi194 ceph-mon[44089]: 6.a scrub starts 2024-04-29T14:34:06.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:06 smithi194 ceph-mon[44089]: 6.a scrub ok 2024-04-29T14:34:06.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:06 smithi194 ceph-mon[44089]: pgmap v1643: 27 pgs: 27 active+clean; 1.0 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-04-29T14:34:06.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/114616372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:06.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:06 smithi194 ceph-mon[44089]: from='client.36638 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:07.556 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:07.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:07.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:07 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:34:07.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:07 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:34:07.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:07 smithi121 ceph-mon[44533]: from='client.36638 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:07.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:07 smithi121 ceph-mon[44533]: osdmap e644: 12 total, 12 up, 8 in 2024-04-29T14:34:07.661 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:07.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:07 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:34:07.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:07 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:34:07.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:07 smithi194 ceph-mon[44089]: from='client.36638 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:07.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:07 smithi194 ceph-mon[44089]: osdmap e644: 12 total, 12 up, 8 in 2024-04-29T14:34:07.693 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:07.715 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:07.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:07.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:07.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:07 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:34:07.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:07 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:34:07.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:07 smithi040 ceph-mon[39581]: from='client.36638 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:07.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:07 smithi040 ceph-mon[39581]: osdmap e644: 12 total, 12 up, 8 in 2024-04-29T14:34:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:07.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:07.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:08.008 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:08.029 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:08.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:08.134 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:08.163 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:08.184 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:08.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:08.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:08.320 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:34:08.320 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:34:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:08.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:08 smithi040 ceph-mon[39581]: pgmap v1645: 27 pgs: 27 active+clean; 853 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-04-29T14:34:08.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:08 smithi194 ceph-mon[44089]: pgmap v1645: 27 pgs: 27 active+clean; 853 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-04-29T14:34:08.871 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:08.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:08.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:08 smithi121 ceph-mon[44533]: pgmap v1645: 27 pgs: 27 active+clean; 853 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-04-29T14:34:08.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:34:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:09.020 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:34:09.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:09.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:09.043 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:34:09.063 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:34:09.084 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:34:09.104 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:34:09.124 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:34:09.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:09.144 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:34:09.164 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:09.164 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:34:09.186 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:34:09.186 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:09.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:34:09.226 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:34:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:34:09.290 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:09.321 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:09.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4036256128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:09.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:09.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:09.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4036256128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:09.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:09.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:09.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4036256128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:09.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:09.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:09.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:34:10.504 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:10.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:10.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:34:10.608 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:10 smithi194 ceph-mon[44089]: pgmap v1646: 27 pgs: 27 active+clean; 838 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-04-29T14:34:10.822 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.822 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.822 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:10 smithi040 ceph-mon[39581]: pgmap v1646: 27 pgs: 27 active+clean; 838 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-04-29T14:34:10.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:10.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:10 smithi121 ceph-mon[44533]: pgmap v1646: 27 pgs: 27 active+clean; 838 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-04-29T14:34:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:11.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:11.292 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:11.320 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:11.634 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:11.669 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:11.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:11.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:11.808 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:11.829 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:11.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:11.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:11.965 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:11.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:11.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:11.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:12.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:12.123 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:12.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:12.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:12.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:12.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:12.162 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:12.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:12.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:12.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:12.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:12.890 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:12.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:12.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:12.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:13.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:13.045 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:13.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:13.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:13.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:13.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:34:13.201 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:13.324 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:13.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:13 smithi040 ceph-mon[39581]: pgmap v1647: 27 pgs: 27 active+clean; 838 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 801 op/s 2024-04-29T14:34:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:13.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:13.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:13.457 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:13.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:13 smithi194 ceph-mon[44089]: pgmap v1647: 27 pgs: 27 active+clean; 838 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 801 op/s 2024-04-29T14:34:13.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:13 smithi121 ceph-mon[44533]: pgmap v1647: 27 pgs: 27 active+clean; 838 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 801 op/s 2024-04-29T14:34:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:14.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:14.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:34:14.744 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:14.766 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:34:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:14.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:34:14.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:14.903 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:14.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:14.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:15.029 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:15.061 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:15.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-04-29T14:34:15.082 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:15.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:15.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:15.218 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:15.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:15.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:15.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:15.377 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:15.398 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:15.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:15.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:15.533 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:15 smithi121 ceph-mon[44533]: pgmap v1648: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 562 op/s 2024-04-29T14:34:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:15.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:15.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:15 smithi040 ceph-mon[39581]: pgmap v1648: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 562 op/s 2024-04-29T14:34:15.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:15.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:15.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:15.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:15 smithi194 ceph-mon[44089]: pgmap v1648: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 562 op/s 2024-04-29T14:34:15.828 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:34:15.828 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:34:15.828 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:34:15.828 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:34:15.828 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:34:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:15.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:15.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:16.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:16.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:16.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:16.263 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:16.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:16.302 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:16.324 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:16.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:34:16.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:16.453 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/404029800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:34:16.455 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:16.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:16.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:16.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/404029800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:34:16.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:16.604 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:16.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/404029800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:34:16.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:16.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:16.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:16.674 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:34:16.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:16.757 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:17.061 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:17.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:17.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:17.193 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:17.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:17.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:17.318 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:17.351 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:17.372 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:17.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:17.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:17.501 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:17 smithi121 ceph-mon[44533]: pgmap v1649: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 316 op/s 2024-04-29T14:34:17.501 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:17 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:17.501 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/404029800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:34:17.501 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:17 smithi121 ceph-mon[44533]: osdmap e645: 12 total, 12 up, 8 in 2024-04-29T14:34:17.502 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:17 smithi121 ceph-mon[44533]: osdmap e646: 12 total, 12 up, 8 in 2024-04-29T14:34:17.503 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:17.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:17.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:17.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:17 smithi040 ceph-mon[39581]: pgmap v1649: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 316 op/s 2024-04-29T14:34:17.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:17 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:17.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/404029800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:34:17.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:17 smithi040 ceph-mon[39581]: osdmap e645: 12 total, 12 up, 8 in 2024-04-29T14:34:17.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:17 smithi040 ceph-mon[39581]: osdmap e646: 12 total, 12 up, 8 in 2024-04-29T14:34:17.627 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:34:17.654 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:17 smithi194 ceph-mon[44089]: pgmap v1649: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 316 op/s 2024-04-29T14:34:17.654 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:17 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:17.654 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/404029800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:34:17.654 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:17 smithi194 ceph-mon[44089]: osdmap e645: 12 total, 12 up, 8 in 2024-04-29T14:34:17.654 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:17 smithi194 ceph-mon[44089]: osdmap e646: 12 total, 12 up, 8 in 2024-04-29T14:34:17.656 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:17.677 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:34:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:17.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:34:17.740 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:17.781 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:17.808 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:17.829 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:17.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:17.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:17.965 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:18.669 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:19.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:19 smithi040 ceph-mon[39581]: pgmap v1652: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-04-29T14:34:19.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:19 smithi121 ceph-mon[44533]: pgmap v1652: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-04-29T14:34:19.716 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:19.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:19.777 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:34:19.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:19 smithi194 ceph-mon[44089]: pgmap v1652: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-04-29T14:34:19.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:19.854 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:19.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:19.980 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:20.012 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:20.034 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:20.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:20.046 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:34:20.046 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:34:20.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:34:20.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:34:20.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:34:20.052 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:34:20.077 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:34:20.099 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:34:20.119 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:34:20.138 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:20.139 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:34:20.158 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:34:20.167 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:20.178 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:34:20.179 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:34:20.180 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:20.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:20.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:20.343 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:20.372 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:20.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:20.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:20.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/431085942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:20.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:20.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:20.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/431085942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:20.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.703 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:20.731 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/431085942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:20.732 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.732 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.732 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:20.734 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:20.761 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:20.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:20.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:20.915 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:20.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:20.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:21.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:21.111 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:21.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:21.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: pgmap v1653: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:21 smithi040 ceph-mon[39581]: 6.e scrub starts 2024-04-29T14:34:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: pgmap v1653: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-04-29T14:34:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:21 smithi121 ceph-mon[44533]: 6.e scrub starts 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: pgmap v1653: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 837 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:21.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:21 smithi194 ceph-mon[44089]: 6.e scrub starts 2024-04-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:22.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:22.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:22.480 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:22.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:22.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:22.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:22 smithi040 ceph-mon[39581]: pgmap v1654: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 801 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 72 op/s 2024-04-29T14:34:22.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:22.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:22 smithi121 ceph-mon[44533]: pgmap v1654: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 801 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 72 op/s 2024-04-29T14:34:22.664 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:22.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:22.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:22 smithi194 ceph-mon[44089]: pgmap v1654: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 801 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 72 op/s 2024-04-29T14:34:22.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:22.943 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:22.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:22.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:23.070 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:23.083 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:23.114 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:23.144 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:23.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:23.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:23.286 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:23.318 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:23.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:23.426 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:23.463 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:23.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:23.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:23.644 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:23.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:23.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:34:23.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:23.825 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:23.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:23.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:24.016 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:24.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:24.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:24.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:24.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-04-29T14:34:25.052 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:25.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:25.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:25.180 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:34:25.180 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:34:25.209 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:25.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:25.356 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: pgmap v1655: 27 pgs: 27 active+clean; 726 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 223 op/s 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:25.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:25 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: pgmap v1655: 27 pgs: 27 active+clean; 726 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 223 op/s 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:25 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:34:25.392 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: pgmap v1655: 27 pgs: 27 active+clean; 726 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 223 op/s 2024-04-29T14:34:25.392 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:34:25.392 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:34:25.392 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:34:25.392 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:34:25.392 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:25.393 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:25.393 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:25 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:34:25.395 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:25.421 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:25.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:25.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:25.561 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:25.913 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:25.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:25.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:34:26.020 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:34:26.054 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:26.078 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:34:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:26.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:34:26.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:26.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:26.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:26.395 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:26.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:26.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:26.613 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:27.153 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:34:27.169 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:27 smithi040 ceph-mon[39581]: pgmap v1656: 27 pgs: 27 active+clean; 535 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 498 op/s 2024-04-29T14:34:27.169 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2332458908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:27.169 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:27 smithi040 ceph-mon[39581]: from='client.26923 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:27.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:27 smithi121 ceph-mon[44533]: pgmap v1656: 27 pgs: 27 active+clean; 535 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 498 op/s 2024-04-29T14:34:27.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2332458908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:27.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:27 smithi121 ceph-mon[44533]: from='client.26923 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:27.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:27 smithi194 ceph-mon[44089]: pgmap v1656: 27 pgs: 27 active+clean; 535 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 498 op/s 2024-04-29T14:34:27.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2332458908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:27.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:27 smithi194 ceph-mon[44089]: from='client.26923 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:28.141 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:28.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:28.246 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:28.278 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:28.308 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:28.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:28.417 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:28.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:28 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:28.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:28 smithi194 ceph-mon[44089]: from='client.26923 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:28.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:28 smithi194 ceph-mon[44089]: osdmap e647: 12 total, 12 up, 8 in 2024-04-29T14:34:28.443 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:28 smithi194 ceph-mon[44089]: osdmap e648: 12 total, 12 up, 8 in 2024-04-29T14:34:28.445 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:28.465 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:28.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:28.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:28.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:28 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:28.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:28 smithi040 ceph-mon[39581]: from='client.26923 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:28.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:28 smithi040 ceph-mon[39581]: osdmap e647: 12 total, 12 up, 8 in 2024-04-29T14:34:28.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:28 smithi040 ceph-mon[39581]: osdmap e648: 12 total, 12 up, 8 in 2024-04-29T14:34:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:28.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:28 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:28.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:28 smithi121 ceph-mon[44533]: from='client.26923 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:28.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:28 smithi121 ceph-mon[44533]: osdmap e647: 12 total, 12 up, 8 in 2024-04-29T14:34:28.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:28 smithi121 ceph-mon[44533]: osdmap e648: 12 total, 12 up, 8 in 2024-04-29T14:34:28.654 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:28.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:28.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:28.810 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:28.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:28.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:28.862 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:34:28.862 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:34:28.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:28.988 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:29.012 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:29.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:29.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:29.408 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 152850 objects 2024-04-29T14:34:29.408 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :177.133 2024-04-29T14:34:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:29 smithi040 ceph-mon[39581]: pgmap v1659: 27 pgs: 27 active+clean; 278 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:34:29.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:29 smithi121 ceph-mon[44533]: pgmap v1659: 27 pgs: 27 active+clean; 278 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:34:29.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:34:29.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:29 smithi194 ceph-mon[44089]: pgmap v1659: 27 pgs: 27 active+clean; 278 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:34:29.942 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:30.298 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:34:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:34:30.338 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:34:30.358 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:34:30.377 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:34:30.397 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:34:30.417 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:34:30.437 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:34:30.439 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/787023426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:30.457 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:34:30.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:34:30.497 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:34:30.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:34:30.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:34:30.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:34:30.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:34:30.573 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-04-29T14:34:30.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-04-29T14:34:30.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/787023426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:30.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/787023426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:30.823 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:30.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:30.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:30.976 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:30.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:31.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:31.116 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:31.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:31.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:31.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:31.272 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:31.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: pgmap v1660: 27 pgs: 27 active+clean; 222 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:31.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:31.500 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:31.533 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:31.554 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:31.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:31.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: pgmap v1660: 27 pgs: 27 active+clean; 222 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:34:31.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:31.703 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:34:31.703 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:34:31.703 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:34:31.703 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:34:31.704 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:34:31.716 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:31.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:31.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:34:31.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: pgmap v1660: 27 pgs: 27 active+clean; 222 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:34:31.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:31.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:31.850 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:31.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:31.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:32.052 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:32.085 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:32.107 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:32.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:32.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:32.241 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:32.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:32.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:32.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:34:32.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:32.499 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:32.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:32 smithi121 ceph-mon[44533]: pgmap v1661: 27 pgs: 27 active+clean; 74 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:34:32.684 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:32.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:32 smithi040 ceph-mon[39581]: pgmap v1661: 27 pgs: 27 active+clean; 74 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:34:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:32.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:32.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:32 smithi194 ceph-mon[44089]: pgmap v1661: 27 pgs: 27 active+clean; 74 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-29T14:34:32.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:32.880 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:32.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:32.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:33.022 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:33.052 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:33.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:34:34.376 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:34:34.376 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:34:34.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:34:34.376 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:34:34.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.376 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:147906 5.f 5:f0d11147:::benchmark_data_smithi194_121512_object14440:head [delete] snapc 0=[] ondisk+write+known_if_redirected e475)", 2024-04-29T14:34:34.376 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:26:42.375558+0000", 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 471.99873951799998, 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.245455862, 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 147906 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375558+0000", 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375558+0000", 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375560+0000", 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2079999999999999e-06 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375566+0000", 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3539999999999996e-06 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375580+0000", 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3319000000000001e-05 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.375588+0000", 2024-04-29T14:34:34.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6999999999999997e-06 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.388800+0000", 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013211268 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.388918+0000", 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118381 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:34:34.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.389062+0000", 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014409399999999999 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416268+0000", 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027205521999999999 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.618705+0000", 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202437123 2024-04-29T14:34:34.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.618784+0000", 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9177000000000004e-05 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.621014+0000", 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022297160000000001 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26047.0:147909 5.f 5:fc13d4e9:::benchmark_data_smithi194_121512_object14443:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e475)", 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:26:42.416417+0000", 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 471.95788075399997, 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27121119900000001, 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26047", 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3161972568", 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 147909 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416417+0000", 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416417+0000", 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416418+0000", 2024-04-29T14:34:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8599999999999996e-07 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416421+0000", 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4529999999999998e-06 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416429+0000", 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5750000000000003e-06 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.416434+0000", 2024-04-29T14:34:34.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8989999999999998e-06 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.443012+0000", 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026577536999999998 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.443105+0000", 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3059000000000005e-05 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.443231+0000", 2024-04-29T14:34:34.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125485 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.449050+0000", 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058199380000000002 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.686242+0000", 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.237191072 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.686303+0000", 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1875e-05 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:26:42.687628+0000", 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013243199999999999 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127897 6.e 6:75170b92:::benchmark_data_smithi194_134202_object127896:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.373844+0000", 2024-04-29T14:34:34.388 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 206.000453088, 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21297782900000001, 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127897 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373844+0000", 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373844+0000", 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373847+0000", 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.339e-06 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373876+0000", 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9181999999999999e-05 2024-04-29T14:34:34.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373889+0000", 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3719e-05 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373899+0000", 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4299999999999995e-06 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373972+0000", 2024-04-29T14:34:34.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3341000000000001e-05 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374111+0000", 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013890300000000001 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374238+0000", 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012690999999999999 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.383381+0000", 2024-04-29T14:34:34.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091430639999999994 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586704+0000", 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20332249799999999 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586763+0000", 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9762000000000001e-05 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586822+0000", 2024-04-29T14:34:34.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8681000000000003e-05 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127898 6.7 6:e4fbbafd:::benchmark_data_smithi194_134202_object127897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 206.000376601, 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.212923687, 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.394 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127898 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:34:34.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373921+0000", 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8599999999999998e-07 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373955+0000", 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3615999999999998e-05 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373962+0000", 2024-04-29T14:34:34.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3699999999999997e-06 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.373969+0000", 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374042+0000", 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3023999999999997e-05 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374140+0000", 2024-04-29T14:34:34.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8722999999999997e-05 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.374223+0000", 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2781000000000006e-05 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.383397+0000", 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091737160000000002 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586732+0000", 2024-04-29T14:34:34.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20333527300000001 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586799+0000", 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6882000000000001e-05 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.586844+0000", 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5367000000000001e-05 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:127917 6.7 6:ecf9ff1a:::benchmark_data_smithi194_134202_object127916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e564)", 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:31:08.387947+0000", 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 205.986350069, 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.223737357, 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 127917 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387947+0000", 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387947+0000", 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387950+0000", 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7140000000000002e-06 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.387997+0000", 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6962000000000002e-05 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388009+0000", 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759999999999999e-05 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388016+0000", 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0700000000000001e-06 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388071+0000", 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5438e-05 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388233+0000", 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016152000000000001 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.388391+0000", 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015797399999999999 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.392441+0000", 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040504249999999999 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611560+0000", 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.219118484 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611616+0000", 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6659000000000001e-05 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:31:08.611685+0000", 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8350999999999997e-05 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:218730 6.2 6:45826de4:::benchmark_data_smithi194_134202_object65877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:45.137800+0000", 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 109.23649687299999, 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21727151, 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.405 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 218730 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.137800+0000", 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.137800+0000", 2024-04-29T14:34:34.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.137803+0000", 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2869999999999999e-06 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.137810+0000", 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0520000000000004e-06 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.137822+0000", 2024-04-29T14:34:34.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2659e-05 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.137832+0000", 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.285e-06 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.137922+0000", 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9893000000000001e-05 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.138008+0000", 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6499e-05 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.138088+0000", 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9988999999999996e-05 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.146834+0000", 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087463239999999998 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.148330+0000", 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001495447 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.354902+0000", 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20657235600000001 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.354956+0000", 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3517000000000003e-05 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:45.355072+0000", 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116202 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227246 6.2 6:44d3a8bb:::benchmark_data_smithi194_134202_object74393:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e606)", 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141071+0000", 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233226701000007, 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.731695655, 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.411 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227246 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141071+0000", 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141071+0000", 2024-04-29T14:34:34.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141073+0000", 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0219999999999999e-06 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141078+0000", 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2639999999999999e-06 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141093+0000", 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5387e-05 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141101+0000", 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6190000000000002e-06 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141163+0000", 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2552999999999996e-05 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141175+0000", 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1518000000000001e-05 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.859421+0000", 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.71824634300000001 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.859522+0000", 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100667 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.859611+0000", 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9271999999999995e-05 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.863400+0000", 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037891959999999999 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.863450+0000", 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9385999999999998e-05 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.866898+0000", 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034477729999999999 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872639+0000", 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057416660000000003 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872681+0000", 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1474999999999998e-05 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872766+0000", 2024-04-29T14:34:34.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5513999999999995e-05 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227259 6.2 6:4bcae690:::benchmark_data_smithi194_134202_object74406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141124+0000", 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233172955000001, 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.736456942, 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227259 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141124+0000", 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141124+0000", 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141125+0000", 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2399999999999997e-07 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141128+0000", 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.101e-06 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141135+0000", 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6590000000000001e-06 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141146+0000", 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1035e-05 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141221+0000", 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4745000000000003e-05 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141232+0000", 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1392e-05 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.863475+0000", 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72224302799999995 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.863568+0000", 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2856999999999995e-05 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.863668+0000", 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9877999999999995e-05 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.867046+0000", 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033779080000000002 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872510+0000", 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054645029999999999 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872598+0000", 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8174e-05 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.877479+0000", 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048808610000000002 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.877516+0000", 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6921000000000001e-05 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.877581+0000", 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5055999999999995e-05 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.424 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227275 6.2 6:4fdd8cb3:::benchmark_data_smithi194_134202_object74422:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141156+0000", 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233141297000003, 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.74488699800000002, 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.425 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227275 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141156+0000", 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141156+0000", 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141157+0000", 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6799999999999999e-07 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141160+0000", 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.678e-06 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141165+0000", 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9930000000000003e-06 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141171+0000", 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3790000000000003e-06 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141275+0000", 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104206 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141284+0000", 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629999999999995e-06 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.867084+0000", 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72579969600000005 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.867241+0000", 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015707399999999999 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.867329+0000", 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8383999999999999e-05 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872607+0000", 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052778570000000004 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872810+0000", 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000203174 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.877441+0000", 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004630286 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.885902+0000", 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084617189999999995 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.885963+0000", 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0948e-05 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.886043+0000", 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9672999999999994e-05 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.431 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227281 6.2 6:43ff8db8:::benchmark_data_smithi194_134202_object74428:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141176+0000", 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233121206000007, 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.75017836999999998, 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227281 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.432 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141176+0000", 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141176+0000", 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141176+0000", 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8000000000000001e-07 2024-04-29T14:34:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141178+0000", 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.536e-06 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141188+0000", 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5999999999999996e-06 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141194+0000", 2024-04-29T14:34:34.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6669999999999999e-06 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141328+0000", 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134115 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141334+0000", 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5260000000000001e-06 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.869916+0000", 2024-04-29T14:34:34.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.72858199400000001 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.869990+0000", 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3957999999999999e-05 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.870067+0000", 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7035999999999999e-05 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872819+0000", 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027523090000000001 2024-04-29T14:34:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.877307+0000", 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004488194 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.877623+0000", 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031580999999999997 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891153+0000", 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013529694 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891217+0000", 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4338000000000006e-05 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891354+0000", 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013721299999999999 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227283 6.2 6:4c8c3c19:::benchmark_data_smithi194_134202_object74430:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-29T14:34:34.438 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141200+0000", 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233097633, 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.75042536999999998, 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227283 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141200+0000", 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141200+0000", 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141200+0000", 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.06e-07 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141211+0000", 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0477e-05 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141218+0000", 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0740000000000004e-06 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141223+0000", 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2170000000000002e-06 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141403+0000", 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018030800000000001 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141407+0000", 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.76e-06 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872843+0000", 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.73143635799999995 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.872937+0000", 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3745000000000005e-05 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.873021+0000", 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3473000000000005e-05 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.877590+0000", 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045690499999999998 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.885753+0000", 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081628929999999992 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.885811+0000", 2024-04-29T14:34:34.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8446999999999998e-05 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891517+0000", 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057063169999999998 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891552+0000", 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5055000000000002e-05 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891625+0000", 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2689999999999997e-05 2024-04-29T14:34:34.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227285 6.2 6:47f439cd:::benchmark_data_smithi194_134202_object74432:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141235+0000", 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233062040999997, 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.75497624600000002, 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227285 2024-04-29T14:34:34.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141235+0000", 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141235+0000", 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141236+0000", 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1399999999999997e-07 2024-04-29T14:34:34.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141239+0000", 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.847e-06 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141245+0000", 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9309999999999996e-06 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141251+0000", 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8860000000000001e-06 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141440+0000", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018987799999999999 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141448+0000", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.525e-06 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.875180+0000", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.73373253800000005 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.875248+0000", 2024-04-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7329999999999994e-05 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.875323+0000", 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5549000000000005e-05 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.882326+0000", 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070027350000000004 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.885852+0000", 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035262570000000001 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.886079+0000", 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022639399999999999 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.896026+0000", 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099471330000000004 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.896090+0000", 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3843000000000005e-05 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.896212+0000", 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121786 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227286 6.2 6:4cb2ee20:::benchmark_data_smithi194_134202_object74433:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141264+0000", 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233033659, 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.75519082999999998, 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227286 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141264+0000", 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141264+0000", 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141264+0000", 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.46e-07 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141266+0000", 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.691e-06 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141270+0000", 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1289999999999998e-06 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141273+0000", 2024-04-29T14:34:34.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9639999999999999e-06 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141472+0000", 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019944799999999999 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141476+0000", 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4290000000000001e-06 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.882367+0000", 2024-04-29T14:34:34.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.74089163400000002 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.882488+0000", 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121062 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.882572+0000", 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3552000000000002e-05 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.886087+0000", 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003515415 2024-04-29T14:34:34.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.886119+0000", 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1371999999999997e-05 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891101+0000", 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049822490000000002 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.896318+0000", 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052173109999999997 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.896358+0000", 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9317000000000003e-05 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.896454+0000", 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6910999999999994e-05 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:227297 6.2 6:4fd7abca:::benchmark_data_smithi194_134202_object74444:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-29T14:34:34.457 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:32:57.141276+0000", 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.233021172999997, 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.75776030299999997, 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227297 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141276+0000", 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141276+0000", 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141276+0000", 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8099999999999999e-07 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141278+0000", 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.359e-06 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141282+0000", 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9689999999999996e-06 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141285+0000", 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.287e-06 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141491+0000", 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000205515 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.141495+0000", 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.048e-06 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.886196+0000", 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.74470134300000002 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.886270+0000", 2024-04-29T14:34:34.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3924000000000006e-05 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.886350+0000", 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9878999999999999e-05 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891368+0000", 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050187349999999999 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891422+0000", 2024-04-29T14:34:34.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3201000000000001e-05 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.891481+0000", 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8860999999999998e-05 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.898888+0000", 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074079389999999997 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.898950+0000", 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1866999999999996e-05 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:32:57.899036+0000", 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6094999999999995e-05 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:248601 6.7 6:edd24c11:::benchmark_data_smithi194_134202_object95748:head [delete] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.346318+0000", 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.027979729999998, 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.248223895, 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 248601 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346318+0000", 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346318+0000", 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346320+0000", 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.419e-06 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346329+0000", 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5949999999999999e-06 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346343+0000", 2024-04-29T14:34:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4158999999999999e-05 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346351+0000", 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3040000000000005e-06 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346443+0000", 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1914000000000001e-05 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346561+0000", 2024-04-29T14:34:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117877 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.346666+0000", 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010542 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.355958+0000", 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092921070000000008 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.594526+0000", 2024-04-29T14:34:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23856788100000001 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.594541+0000", 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5218999999999999e-05 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:248596 6.2 6.718df42 (undecoded) ondisk+retry+write+known_if_redirected e615)", 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.356249+0000", 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.018047906000007, 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.226904149, 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 248596 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356249+0000", 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356249+0000", 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356251+0000", 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.638e-06 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356257+0000", 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9920000000000004e-06 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356266+0000", 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7369999999999995e-06 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356272+0000", 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2280000000000003e-06 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356318+0000", 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5785000000000003e-05 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-29T14:34:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356325+0000", 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5970000000000003e-06 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.583024+0000", 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22669858300000001 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.583154+0000", 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129589 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:248601 6.17 6:edd24c11:::benchmark_data_smithi194_134202_object95748:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e615)", 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.356285+0000", 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.018012767000002, 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777752940000001, 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 248601 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356285+0000", 2024-04-29T14:34:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356285+0000", 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356285+0000", 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4000000000000001e-07 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356288+0000", 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5390000000000002e-06 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356293+0000", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6420000000000004e-06 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356299+0000", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6640000000000003e-06 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.595233+0000", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23893415300000001 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.595246+0000", 2024-04-29T14:34:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2510999999999999e-05 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.433984+0000", 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.83873813699999999 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.434060+0000", 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6007999999999995e-05 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:248611 6.2 6.947d69c2 (undecoded) ondisk+retry+write+known_if_redirected e615)", 2024-04-29T14:34:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.356312+0000", 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.017985753000005, 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22754587200000001, 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 248611 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356312+0000", 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356312+0000", 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356312+0000", 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1999999999999999e-07 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356315+0000", 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5320000000000001e-06 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356320+0000", 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4149999999999998e-06 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356326+0000", 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5960000000000003e-06 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356369+0000", 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2852999999999998e-05 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.356375+0000", 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1990000000000003e-06 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.583845+0000", 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22747063200000001 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.583857+0000", 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2024999999999999e-05 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(6.2 614/605 [PushReplyOp(6:4c6185aa:::benchmark_data_smithi194_134202_object101263:head)])", 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.361514+0000", 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.012783165000002, 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.222724806, 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.361514+0000", 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.361514+0000", 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.361515+0000", 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6100000000000001e-07 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.361519+0000", 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9779999999999999e-06 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.361525+0000", 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.083e-06 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.361533+0000", 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9789999999999993e-06 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.584224+0000", 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.222690998 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.584239+0000", 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5207e-05 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:248646 6.2 6:43b3c4a2:::benchmark_data_smithi194_134202_object95793:head [delete] snapc 0=[] ondisk+write+known_if_redirected e616)", 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:23.447213+0000", 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 70.927084242000006, 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25651044000000001, 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 248646 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.447213+0000", 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.447213+0000", 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.447216+0000", 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0060000000000001e-06 2024-04-29T14:34:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.447224+0000", 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6480000000000003e-06 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.447239+0000", 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5384999999999999e-05 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.447249+0000", 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6010000000000001e-06 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.698882+0000", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25163302300000001 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.699026+0000", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014379599999999999 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10,11", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.699184+0000", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000158076 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.702967+0000", 2024-04-29T14:34:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003783135 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.702985+0000", 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7825e-05 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.703042+0000", 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7037999999999998e-05 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.703510+0000", 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000468844 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.703581+0000", 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0109000000000002e-05 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:23.703723+0000", 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014295399999999999 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:34:34.490 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: osdmap e649: 12 total, 12 up, 8 in 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:34.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:34.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:34.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:34 smithi121 ceph-mon[44533]: pgmap v1663: 27 pgs: 27 active+clean; 4.7 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-29T14:34:34.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:34.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:34.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: osdmap e649: 12 total, 12 up, 8 in 2024-04-29T14:34:34.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:34:34.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:34.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:34 smithi194 ceph-mon[44089]: pgmap v1663: 27 pgs: 27 active+clean; 4.7 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: osdmap e649: 12 total, 12 up, 8 in 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:34:34.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:34:34.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:34:34.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:34.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:34.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:34 smithi040 ceph-mon[39581]: pgmap v1663: 27 pgs: 27 active+clean; 4.7 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-29T14:34:34.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:34:35.229 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:35.717 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:35 smithi121 ceph-mon[44533]: osdmap e650: 12 total, 12 up, 8 in 2024-04-29T14:34:35.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:35 smithi194 ceph-mon[44089]: osdmap e650: 12 total, 12 up, 8 in 2024-04-29T14:34:35.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:35 smithi040 ceph-mon[39581]: osdmap e650: 12 total, 12 up, 8 in 2024-04-29T14:34:36.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:36 smithi121 ceph-mon[44533]: pgmap v1665: 27 pgs: 27 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 555 op/s 2024-04-29T14:34:36.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:36 smithi194 ceph-mon[44089]: pgmap v1665: 27 pgs: 27 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 555 op/s 2024-04-29T14:34:36.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:36 smithi040 ceph-mon[39581]: pgmap v1665: 27 pgs: 27 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 555 op/s 2024-04-29T14:34:36.915 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:34:36.915 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:34:36.915 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:34:36.915 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:34:37.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-04-29T14:34:37.241 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:37.284 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:37.504 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:37.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:37.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:34:37.608 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:37.641 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:37.662 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:37.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:37.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:37.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:37.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:37.829 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:37.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:37.948 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:38.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:38.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:38.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:38.207 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:38.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:38.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:38.469 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:38.491 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:38.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:38.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:39.336 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_4' removed 2024-04-29T14:34:39.580 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:34:39.581 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:34:39.581 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:34:39.581 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:34:39.581 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:34:39.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:39 smithi121 ceph-mon[44533]: pgmap v1666: 27 pgs: 27 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 443 op/s 2024-04-29T14:34:39.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/818167607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:34:39.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:39 smithi040 ceph-mon[39581]: pgmap v1666: 27 pgs: 27 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 443 op/s 2024-04-29T14:34:39.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/818167607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:34:39.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:39 smithi194 ceph-mon[44089]: pgmap v1666: 27 pgs: 27 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 443 op/s 2024-04-29T14:34:39.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/818167607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:34:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:40.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:40.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:34:40.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:40.309 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:40.407 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:40.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:40.505 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:40.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:40.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/818167607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:34:40.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:40 smithi040 ceph-mon[39581]: osdmap e651: 12 total, 12 up, 8 in 2024-04-29T14:34:40.539 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/818167607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:34:40.539 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:40 smithi121 ceph-mon[44533]: osdmap e651: 12 total, 12 up, 8 in 2024-04-29T14:34:40.541 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:40.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:40.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:40.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/818167607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:34:40.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:40 smithi194 ceph-mon[44089]: osdmap e651: 12 total, 12 up, 8 in 2024-04-29T14:34:40.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:40.870 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:40.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:40.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:40.980 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:40.981 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:34:40.982 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:34:40.982 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-04-29T14:34:40.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-04-29T14:34:41.071 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:41.103 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:41.124 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:41.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:41.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:41.259 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:41.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:41 smithi194 ceph-mon[44089]: pgmap v1668: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:41.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:41 smithi040 ceph-mon[39581]: pgmap v1668: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:41.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:41 smithi121 ceph-mon[44533]: pgmap v1668: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:41.721 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:34:41.721 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:34:41.730 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:34:41.751 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:34:41.772 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:34:41.792 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:34:41.812 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:34:41.832 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:34:41.870 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:34:41.895 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:34:41.915 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:34:41.935 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:34:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:34:42.152 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:34:42.153 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:34:42.354 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3294085217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1795730867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:34:42.580 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:42 smithi121 ceph-mon[44533]: from='client.36704 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:34:42.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3294085217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:42.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1795730867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:34:42.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:42 smithi040 ceph-mon[39581]: from='client.36704 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:34:42.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3294085217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1795730867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:34:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:42 smithi194 ceph-mon[44089]: from='client.36704 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:34:42.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:34:43.124 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:43.209 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:43.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:43.316 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:43.348 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:43.370 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:43.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:43.474 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:43.504 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:43.524 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:43.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:43.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:43.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:43.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:43 smithi040 ceph-mon[39581]: pgmap v1669: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:43.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:43 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:43.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:43 smithi040 ceph-mon[39581]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:34:43.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:43 smithi040 ceph-mon[39581]: osdmap e652: 12 total, 12 up, 8 in 2024-04-29T14:34:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:43 smithi121 ceph-mon[44533]: pgmap v1669: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:43 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:43 smithi121 ceph-mon[44533]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:34:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:43 smithi121 ceph-mon[44533]: osdmap e652: 12 total, 12 up, 8 in 2024-04-29T14:34:43.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:43.657 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:43.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:43.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:43.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:43 smithi194 ceph-mon[44089]: pgmap v1669: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:43.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:43 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:34:43.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:43 smithi194 ceph-mon[44089]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:34:43.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:43 smithi194 ceph-mon[44089]: osdmap e652: 12 total, 12 up, 8 in 2024-04-29T14:34:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:43.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:44.009 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:34:44.041 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:34:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:44.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:34:44.118 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:44.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:44.194 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:44.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:44.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:44.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:44 smithi121 ceph-mon[44533]: pgmap v1671: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:44.449 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:44.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:44.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:44.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:44.604 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:44.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:44 smithi040 ceph-mon[39581]: pgmap v1671: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:44.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:44.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:44.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:34:44.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:44 smithi194 ceph-mon[44089]: pgmap v1671: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:44.827 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:34:44.828 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:34:44.836 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:44.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:34:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:44.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:34:44.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:34:44.959 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:44.991 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:45.013 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:45.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:45.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:45.146 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:45.166 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:45.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:45.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:45.300 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:45.353 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:45.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:45.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:45.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:45.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:45 smithi121 ceph-mon[44533]: osdmap e653: 12 total, 12 up, 8 in 2024-04-29T14:34:45.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:45.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:34:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:45.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:45.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:45.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:45 smithi040 ceph-mon[39581]: osdmap e653: 12 total, 12 up, 8 in 2024-04-29T14:34:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:45.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:45.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:34:45.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:45.692 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:45.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:45.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:45.769 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:45.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:45.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:45.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:45.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:45 smithi194 ceph-mon[44089]: osdmap e653: 12 total, 12 up, 8 in 2024-04-29T14:34:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:45.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:45.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:46.017 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:46.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:46.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:46.142 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:46.174 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:46.196 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:46.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:46.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:46.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:46.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:46.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:46 smithi194 ceph-mon[44089]: pgmap v1673: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:46.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:46.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:46.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:46 smithi121 ceph-mon[44533]: pgmap v1673: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:46.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:46.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:46.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:46 smithi040 ceph-mon[39581]: pgmap v1673: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:47.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-04-29T14:34:47.462 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:34:47.462 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:34:47.483 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:47 smithi121 ceph-mon[44533]: osdmap e654: 12 total, 12 up, 8 in 2024-04-29T14:34:47.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:47 smithi040 ceph-mon[39581]: osdmap e654: 12 total, 12 up, 8 in 2024-04-29T14:34:47.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:47 smithi194 ceph-mon[44089]: osdmap e654: 12 total, 12 up, 8 in 2024-04-29T14:34:48.009 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:48.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:48.113 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:48.142 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:48.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:34:48.163 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:48.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:48.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:48.375 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_5' created 2024-04-29T14:34:48.424 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:34:48.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/688710100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-29T14:34:48.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:48 smithi194 ceph-mon[44089]: from='client.36716 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-29T14:34:48.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:48 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:34:48.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:48 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:34:48.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:48 smithi194 ceph-mon[44089]: pgmap v1675: 1 pgs: 1 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/688710100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-29T14:34:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:48 smithi040 ceph-mon[39581]: from='client.36716 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-29T14:34:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:48 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:34:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:48 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:34:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:48 smithi040 ceph-mon[39581]: pgmap v1675: 1 pgs: 1 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:48.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/688710100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-29T14:34:48.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:48 smithi121 ceph-mon[44533]: from='client.36716 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-29T14:34:48.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:48 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:34:48.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:48 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:34:48.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:48 smithi121 ceph-mon[44533]: pgmap v1675: 1 pgs: 1 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:49.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-04-29T14:34:49.346 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:49.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:49.353 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:49.451 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:49.483 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:49.504 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:49.559 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:49 smithi121 ceph-mon[44533]: from='client.36716 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-29T14:34:49.559 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:49 smithi121 ceph-mon[44533]: osdmap e655: 12 total, 12 up, 8 in 2024-04-29T14:34:49.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:49.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:49 smithi040 ceph-mon[39581]: from='client.36716 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-29T14:34:49.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:49 smithi040 ceph-mon[39581]: osdmap e655: 12 total, 12 up, 8 in 2024-04-29T14:34:49.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:49.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:49 smithi194 ceph-mon[44089]: from='client.36716 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-29T14:34:49.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:49 smithi194 ceph-mon[44089]: osdmap e655: 12 total, 12 up, 8 in 2024-04-29T14:34:49.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:49.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:49.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:49.970 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:49.988 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:49.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:50.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:50.113 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:34:50.113 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:34:50.122 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:50.372 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:50.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:50.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:50 smithi121 ceph-mon[44533]: osdmap e656: 12 total, 12 up, 8 in 2024-04-29T14:34:50.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:50.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:50.469 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:50 smithi121 ceph-mon[44533]: pgmap v1678: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:50.476 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:50.508 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:50.528 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:50.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:50.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:50 smithi040 ceph-mon[39581]: osdmap e656: 12 total, 12 up, 8 in 2024-04-29T14:34:50.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:50.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:50.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:50 smithi040 ceph-mon[39581]: pgmap v1678: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:50.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:50.665 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:50.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:34:50.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:50 smithi194 ceph-mon[44089]: osdmap e656: 12 total, 12 up, 8 in 2024-04-29T14:34:50.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:50.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:50.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:50 smithi194 ceph-mon[44089]: pgmap v1678: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:50.843 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:50.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:50.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:34:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:51.097 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:34:51.164 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:34:51.164 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:34:51.164 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:34:51.179 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:34:51.180 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:34:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:51.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:34:51.201 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:34:51.220 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:34:51.240 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:34:51.260 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:34:51.279 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:34:51.284 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:51.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:34:51.310 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:51.318 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:34:51.338 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:34:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:34:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:34:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:34:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:34:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:34:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:34:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:34:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:34:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:34:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:34:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:34:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:34:51.384 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:51.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:51.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:51.517 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:51.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:51 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:34:51.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:51 smithi194 ceph-mon[44089]: osdmap e657: 12 total, 12 up, 8 in 2024-04-29T14:34:51.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/65616240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:51.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:51.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:51 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:34:51.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:51 smithi040 ceph-mon[39581]: osdmap e657: 12 total, 12 up, 8 in 2024-04-29T14:34:51.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/65616240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:51.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:51.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:51 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:34:51.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:51 smithi121 ceph-mon[44533]: osdmap e657: 12 total, 12 up, 8 in 2024-04-29T14:34:51.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/65616240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:34:51.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:52.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:52.471 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:52.504 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:52.525 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:52.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:52.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:52 smithi040 ceph-mon[39581]: pgmap v1680: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:52.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:52.807 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:52.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:52 smithi194 ceph-mon[44089]: pgmap v1680: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:52.836 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:52.857 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:52.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:52.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:34:52.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:52 smithi121 ceph-mon[44533]: pgmap v1680: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:52.961 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:52.991 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:53.183 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:34:53.184 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:34:53.184 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:34:53.184 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:34:53.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:53.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:53.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:34:53.321 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:53.520 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:34:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:53.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:34:53.624 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:53.653 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:53.846 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:34:53.853 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:53.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:53.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:53.982 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:54.001 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:54.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:54.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:34:54.121 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:34:54.141 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:54.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:34:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:54.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:34:54.267 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:54.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:54.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:54.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:54.556 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:55.131 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:34:55.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:34:55.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:55 smithi040 ceph-mon[39581]: pgmap v1681: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:55.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1294989702' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:55.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:55 smithi194 ceph-mon[44089]: pgmap v1681: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:55.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1294989702' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:55.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:55 smithi121 ceph-mon[44533]: pgmap v1681: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:55.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1294989702' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:34:55.706 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:55.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:55.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:34:55.838 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:55.862 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:34:55.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:55.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:34:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:34:55.908 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:34:55.908 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:34:55.908 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:34:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:34:55.965 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:34:55.994 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:56.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:56 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:34:56.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1294989702' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:56.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:56 smithi194 ceph-mon[44089]: osdmap e658: 12 total, 12 up, 8 in 2024-04-29T14:34:56.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:56 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:34:56.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1294989702' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:56.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:56 smithi121 ceph-mon[44533]: osdmap e658: 12 total, 12 up, 8 in 2024-04-29T14:34:56.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:56 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:34:56.440 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1294989702' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:34:56.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:56 smithi040 ceph-mon[39581]: osdmap e658: 12 total, 12 up, 8 in 2024-04-29T14:34:56.638 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:34:56.644 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:34:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:56.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:34:56.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:56.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:56.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-04-29T14:34:56.902 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:34:56.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:34:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:56.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:34:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:56.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:34:57.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:34:57.100 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-04-29T14:34:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:57 smithi040 ceph-mon[39581]: pgmap v1683: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:57.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2284178580' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:34:57.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:57 smithi121 ceph-mon[44533]: pgmap v1683: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:57.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2284178580' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:34:57.448 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:34:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:57.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:34:57.458 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:57 smithi194 ceph-mon[44089]: pgmap v1683: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:57.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2284178580' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:34:57.553 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:57.581 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:58.232 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:58.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:58.337 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:58.365 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:58.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2284178580' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:34:58.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:58 smithi040 ceph-mon[39581]: osdmap e659: 12 total, 12 up, 8 in 2024-04-29T14:34:58.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2284178580' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:34:58.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:58 smithi194 ceph-mon[44089]: osdmap e659: 12 total, 12 up, 8 in 2024-04-29T14:34:58.463 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:58.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:58.567 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:34:58.595 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:58.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2284178580' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:34:58.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:58 smithi121 ceph-mon[44533]: osdmap e659: 12 total, 12 up, 8 in 2024-04-29T14:34:58.743 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:34:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:58.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217008 6.4 6.189c5b14 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.155698+0000", 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.595235852, 2024-04-29T14:34:58.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.95438274499999998, 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217008 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155698+0000", 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155698+0000", 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155699+0000", 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0950000000000001e-06 2024-04-29T14:34:58.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155703+0000", 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9029999999999997e-06 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155712+0000", 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8030000000000003e-06 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155723+0000", 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0987e-05 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155783+0000", 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9419999999999997e-05 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155795+0000", 2024-04-29T14:34:58.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2836e-05 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110048+0000", 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.95425232699999996 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110081+0000", 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3374000000000003e-05 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217019 6.4 6.5a8858b4 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.169404+0000", 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.581530276, 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.94071997500000004, 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217019 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169404+0000", 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169404+0000", 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169405+0000", 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3209999999999999e-06 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169409+0000", 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8890000000000004e-06 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169418+0000", 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3289999999999994e-06 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169422+0000", 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3930000000000001e-06 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169474+0000", 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1799999999999999e-05 2024-04-29T14:34:58.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169482+0000", 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3529999999999995e-06 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110115+0000", 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.94063343600000004 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110124+0000", 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4540000000000007e-06 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217027 6.4 6.eb4e04f4 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.172245+0000", 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.57868892799999, 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.93791217500000001, 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.763 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217027 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172245+0000", 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172245+0000", 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172247+0000", 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.668e-06 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172252+0000", 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9110000000000001e-06 2024-04-29T14:34:58.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172261+0000", 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7369999999999995e-06 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172269+0000", 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9000000000000006e-06 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172320+0000", 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0986999999999998e-05 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172327+0000", 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6040000000000004e-06 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110151+0000", 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.93782421299999996 2024-04-29T14:34:58.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110157+0000", 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217028 6.4 6.2544a704 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.174731+0000", 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.576203405, 2024-04-29T14:34:58.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.935446954, 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217028 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174731+0000", 2024-04-29T14:34:58.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174731+0000", 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174732+0000", 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5990000000000001e-06 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174738+0000", 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1970000000000004e-06 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174749+0000", 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1712999999999999e-05 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174757+0000", 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1040000000000007e-06 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174815+0000", 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7386000000000002e-05 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174824+0000", 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5270000000000004e-06 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110174+0000", 2024-04-29T14:34:58.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.93534993600000005 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110178+0000", 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4920000000000002e-06 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217047 6.4 6.c0a94fb4 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.259271+0000", 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.49166290900001, 2024-04-29T14:34:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.85093163900000002, 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217047 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259271+0000", 2024-04-29T14:34:58.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259271+0000", 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259273+0000", 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.595e-06 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259278+0000", 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.118e-06 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259287+0000", 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0680000000000003e-06 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259293+0000", 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.716e-06 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259349+0000", 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5931999999999999e-05 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259356+0000", 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2400000000000001e-06 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110197+0000", 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.85084111500000004 2024-04-29T14:34:58.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110203+0000", 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8549999999999998e-06 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217050 6.4 6.a7a30a54 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.778 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.264752+0000", 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.48618227200001, 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84547245999999998, 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217050 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264752+0000", 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264752+0000", 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264753+0000", 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.288e-06 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264757+0000", 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1849999999999997e-06 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264769+0000", 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2048000000000001e-05 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264777+0000", 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9200000000000004e-06 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264833+0000", 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5821e-05 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264843+0000", 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6679999999999996e-06 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110221+0000", 2024-04-29T14:34:58.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84537816200000004 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110224+0000", 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3680000000000002e-06 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217051 6.4 6.23a71304 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.265194+0000", 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.48574039499999, 2024-04-29T14:34:58.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84505052999999997, 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217051 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265194+0000", 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265194+0000", 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265195+0000", 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.347e-06 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265200+0000", 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8960000000000002e-06 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265211+0000", 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1216000000000001e-05 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265219+0000", 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1270000000000003e-06 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265274+0000", 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4738999999999999e-05 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265283+0000", 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5380000000000002e-06 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110239+0000", 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84495622999999997 2024-04-29T14:34:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110244+0000", 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4369999999999998e-06 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217076 6.4 6.3aebfd54 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.362559+0000", 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.38837533899999, 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.74770942699999998, 2024-04-29T14:34:58.788 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217076 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362559+0000", 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362559+0000", 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362560+0000", 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6000000000000002e-07 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362563+0000", 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6449999999999998e-06 2024-04-29T14:34:58.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362573+0000", 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0086e-05 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362579+0000", 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5740000000000003e-06 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362625+0000", 2024-04-29T14:34:58.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6011999999999999e-05 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362633+0000", 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5410000000000003e-06 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110264+0000", 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.74763127500000004 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110268+0000", 2024-04-29T14:34:58.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4340000000000002e-06 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248607 6.4 6:24aa9aec:::benchmark_data_smithi194_134202_object95754:head [delete] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:22.345858+0000", 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.405076140000006, 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27482016399999998, 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.793 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248607 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345858+0000", 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345858+0000", 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345860+0000", 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8610000000000001e-06 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345866+0000", 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5050000000000004e-06 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345882+0000", 2024-04-29T14:34:58.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5281000000000002e-05 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345890+0000", 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2130000000000008e-06 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345945+0000", 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5426000000000001e-05 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:34:58.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.346075+0000", 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012936999999999999 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 6", 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.346208+0000", 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013304699999999999 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.357761+0000", 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011553686000000001 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.620668+0000", 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26290639100000002 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.620678+0000", 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0383999999999999e-05 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248624 6.19 6:9f476182:::benchmark_data_smithi194_134202_object95771:head [delete] snapc 0=[] ondisk+write+known_if_redirected e615)", 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:22.679411+0000", 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 96.071522879, 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.76381612300000001, 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248624 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.799 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679411+0000", 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679411+0000", 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679414+0000", 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3580000000000001e-06 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679420+0000", 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.212e-06 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679432+0000", 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2427e-05 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679439+0000", 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1199999999999996e-06 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679484+0000", 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4459000000000002e-05 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679491+0000", 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4120000000000004e-06 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.438321+0000", 2024-04-29T14:34:58.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.75882953600000003 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.438420+0000", 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9006999999999996e-05 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.438505+0000", 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5372999999999997e-05 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:34:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.440373+0000", 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018676960000000001 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.443131+0000", 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027580460000000001 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.443182+0000", 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0949000000000003e-05 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.443227+0000", 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5528000000000002e-05 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248694 6.12 6.ae9c9b12 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.497991+0000", 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.252942848000004, 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86975254599999996, 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.805 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248694 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497991+0000", 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497991+0000", 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497993+0000", 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.305e-06 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497997+0000", 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0840000000000002e-06 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498005+0000", 2024-04-29T14:34:58.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3550000000000005e-06 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498010+0000", 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0340000000000004e-06 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498052+0000", 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2240000000000002e-05 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498059+0000", 2024-04-29T14:34:58.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5660000000000003e-06 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367735+0000", 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86967563699999995 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367744+0000", 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3249999999999997e-06 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248695 6.12 6.660180b2 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.498079+0000", 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.252854854000006, 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86969207100000001, 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248695 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.810 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498079+0000", 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498079+0000", 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498080+0000", 2024-04-29T14:34:58.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2500000000000005e-07 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498083+0000", 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.514e-06 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498089+0000", 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1500000000000004e-06 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498093+0000", 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1799999999999998e-06 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498138+0000", 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4645999999999999e-05 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498147+0000", 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5049999999999996e-06 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367764+0000", 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86961689200000003 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367771+0000", 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4590000000000001e-06 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248696 6.12 6.c9a57c72 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.250731740999996, 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86759332, 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248696 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.815 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.68e-07 2024-04-29T14:34:58.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500206+0000", 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4990000000000001e-06 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500212+0000", 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9619999999999998e-06 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500216+0000", 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9829999999999998e-06 2024-04-29T14:34:58.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500257+0000", 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1254999999999997e-05 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500267+0000", 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7270000000000002e-06 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367788+0000", 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86752159399999995 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367796+0000", 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6320000000000008e-06 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248716 6.12 6.fb26d272 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.525801+0000", 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.225133579000001, 2024-04-29T14:34:58.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84202116199999999, 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248716 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525801+0000", 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525801+0000", 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525802+0000", 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2699999999999999e-06 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525806+0000", 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.386e-06 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525815+0000", 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4530000000000002e-06 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525820+0000", 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3460000000000001e-06 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525870+0000", 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0068000000000002e-05 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525880+0000", 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0210999999999999e-05 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367813+0000", 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84193279600000004 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367822+0000", 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6319999999999997e-06 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.823 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248694 6.12 6.ae9c9b12 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569493+0000", 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.181441030000002, 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79837771000000002, 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248694 2024-04-29T14:34:58.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569493+0000", 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569493+0000", 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569494+0000", 2024-04-29T14:34:58.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1629999999999999e-06 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569498+0000", 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.269e-06 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569505+0000", 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9770000000000003e-06 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569510+0000", 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4739999999999996e-06 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569559+0000", 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9249999999999998e-05 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569565+0000", 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0689999999999998e-06 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367862+0000", 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79829612599999999 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367871+0000", 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3819999999999993e-06 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.828 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248695 6.12 6.660180b2 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.181414437000001, 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79838047300000003, 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248695 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8899999999999999e-07 2024-04-29T14:34:58.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569527+0000", 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0380000000000002e-06 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569533+0000", 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0610000000000001e-06 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569539+0000", 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1370000000000001e-06 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569600+0000", 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1320000000000002e-05 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569608+0000", 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9719999999999992e-06 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367893+0000", 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79828478000000003 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367900+0000", 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.576e-06 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248696 6.12 6.c9a57c72 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.181389324999998, 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79838399400000004, 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.833 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248696 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0900000000000002e-07 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569548+0000", 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2390000000000001e-06 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569553+0000", 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0359999999999997e-06 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569558+0000", 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2410000000000003e-06 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569633+0000", 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5220000000000001e-05 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569637+0000", 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4209999999999999e-06 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367922+0000", 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79828494800000005 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367929+0000", 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3799999999999996e-06 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248716 6.12 6.fb26d272 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.181371347999999, 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.798394677, 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248716 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.838 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.72e-07 2024-04-29T14:34:58.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569565+0000", 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4279999999999999e-06 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569568+0000", 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569573+0000", 2024-04-29T14:34:58.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1000000000000003e-06 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569651+0000", 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8524999999999998e-05 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569655+0000", 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7100000000000001e-06 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.841 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367949+0000", 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79829424800000004 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367958+0000", 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.174e-06 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248749 6.12 6.6f65012 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.716256+0000", 2024-04-29T14:34:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.034678643999996, 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65172211199999996, 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248749 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716256+0000", 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.843 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716256+0000", 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716257+0000", 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2950000000000001e-06 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716261+0000", 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1760000000000003e-06 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.844 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716270+0000", 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6640000000000004e-06 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716277+0000", 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0990000000000001e-06 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716315+0000", 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7965e-05 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716321+0000", 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1600000000000003e-06 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367973+0000", 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65165252900000004 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367978+0000", 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2239999999999997e-06 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.846 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248767 6.12 6.14c2c972 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.743688+0000", 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.007246077999994, 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62430483599999997, 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248767 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.847 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743688+0000", 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743688+0000", 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743689+0000", 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0100000000000001e-06 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.848 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743693+0000", 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9890000000000003e-06 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743701+0000", 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1850000000000004e-06 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743707+0000", 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3260000000000003e-06 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.849 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743759+0000", 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2441999999999997e-05 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743765+0000", 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5999999999999997e-06 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367989+0000", 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62422426099999995 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367993+0000", 2024-04-29T14:34:58.850 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0230000000000003e-06 2024-04-29T14:34:58.851 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.851 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.851 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.851 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:34:58.851 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:34:58.851 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:34:58.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:34:58.883 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:58.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:34:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:58.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:34:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:34:59.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:34:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:59.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:59 smithi040 ceph-mon[39581]: pgmap v1685: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:59.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:59 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:34:59.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:59 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:34:59.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:34:59.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:34:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:34:59.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:59 smithi194 ceph-mon[44089]: pgmap v1685: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:59.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:59 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:34:59.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:59 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:34:59.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:34:59.459 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:34:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:34:59.478 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:34:59.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:34:59.595 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-04-29T14:34:59.598 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:34:59.598 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:34:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:59 smithi121 ceph-mon[44533]: pgmap v1685: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:34:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:59 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:34:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:59 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:34:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:34:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:34:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:34:59.689 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:34:59.689 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:34:59.689 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_147200 2024-04-29T14:34:59.690 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:34:59.690 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:34:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:34:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:59.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:34:59.803 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:34:59.850 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:34:59.873 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:34:59.879 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:34:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:34:59.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:34:59.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:00.025 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:00.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:00.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:00.325 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:00.347 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:00.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:00.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:00.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:00 smithi194 ceph-mon[44089]: osdmap e660: 12 total, 12 up, 8 in 2024-04-29T14:35:00.491 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:00.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:00.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:00.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:00 smithi040 ceph-mon[39581]: osdmap e660: 12 total, 12 up, 8 in 2024-04-29T14:35:00.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:00 smithi121 ceph-mon[44533]: osdmap e660: 12 total, 12 up, 8 in 2024-04-29T14:35:00.645 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:00.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:00.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:00.689 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 15 3880 3865 241.562 241.562 0.00631629 0.00412727 2024-04-29T14:35:00.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:00.809 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:35:00.813 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:00.834 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:35:00.865 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:35:00.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:35:00.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:35:00.953 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:35:00.982 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:35:01.012 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:35:01.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:35:01.081 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:35:01.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:35:01.148 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:35:01.150 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:35:01.150 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:35:01.150 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:35:01.150 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:35:01.150 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:35:01.150 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:35:01.150 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:35:01.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:35:01.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:35:01.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:35:01.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:35:01.151 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:35:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:01.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:01.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:01 smithi040 ceph-mon[39581]: pgmap v1687: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:35:01.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1507658591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:01.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.479 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.579 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:01.589 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:01.624 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:01 smithi194 ceph-mon[44089]: pgmap v1687: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:35:01.624 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1507658591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:01.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:01 smithi121 ceph-mon[44533]: pgmap v1687: 17 pgs: 17 active+clean; 577 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-04-29T14:35:01.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1507658591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:01.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:01.633 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:01.669 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:01.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:01.689 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 16 8349 8333 260.378 279.25 0.00368204 0.00383266 2024-04-29T14:35:01.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:01.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:01.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:01.955 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:01.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:01.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:02.078 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:02.127 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:02.335 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:02.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:02.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:02.474 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:02.497 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:02.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:02.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:02.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:02.629 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:02.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:02.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:02.690 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 16 11934 11918 248.207 224.062 0.00335568 0.00402547 2024-04-29T14:35:02.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:02.783 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:02.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:02.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:02.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:03.017 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:03.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:03.066 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:35:03.066 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:35:03.122 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:03.164 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:03.192 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:03.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:03.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:03.361 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:03 smithi194 ceph-mon[44089]: pgmap v1688: 17 pgs: 17 active+clean; 163 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s wr, 373 op/s 2024-04-29T14:35:03.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:03 smithi040 ceph-mon[39581]: pgmap v1688: 17 pgs: 17 active+clean; 163 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s wr, 373 op/s 2024-04-29T14:35:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:03.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:03.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:03 smithi121 ceph-mon[44533]: pgmap v1688: 17 pgs: 17 active+clean; 163 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s wr, 373 op/s 2024-04-29T14:35:03.690 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 15 16055 16040 250.555 257.625 0.00315867 0.00398784 2024-04-29T14:35:03.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:03.739 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:03.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:03.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:03.864 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:04.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:35:04.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:04 smithi040 ceph-mon[39581]: pgmap v1689: 17 pgs: 17 active+clean; 278 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 35 MiB/s wr, 555 op/s 2024-04-29T14:35:04.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:04 smithi121 ceph-mon[44533]: pgmap v1689: 17 pgs: 17 active+clean; 278 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 35 MiB/s wr, 555 op/s 2024-04-29T14:35:04.691 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 16 19589 19573 244.604 220.812 0.00316163 0.00408529 2024-04-29T14:35:04.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:04 smithi194 ceph-mon[44089]: pgmap v1689: 17 pgs: 17 active+clean; 278 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 35 MiB/s wr, 555 op/s 2024-04-29T14:35:05.260 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:05.639 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:35:05.690 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 16 22846 22830 237.762 203.562 0.00698509 0.00420321 2024-04-29T14:35:05.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1190971117' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:05.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:05 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:05.972 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:05.977 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1190971117' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:05.977 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:05 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:05.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:06.079 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:06.120 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1190971117' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:06.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:05 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:06.653 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:06.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:06.691 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 15 26114 26099 232.981 204.312 0.0104166 0.00428907 2024-04-29T14:35:06.703 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:35:06.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:06.790 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:06.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:06.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:06.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:06.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1190971117' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:35:06.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:06 smithi121 ceph-mon[44533]: osdmap e661: 12 total, 12 up, 8 in 2024-04-29T14:35:06.940 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:06 smithi121 ceph-mon[44533]: pgmap v1691: 17 pgs: 17 active+clean; 839 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-29T14:35:06.942 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:06.962 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:06.963 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1190971117' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:35:06.963 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:06 smithi040 ceph-mon[39581]: osdmap e661: 12 total, 12 up, 8 in 2024-04-29T14:35:06.963 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:06 smithi040 ceph-mon[39581]: pgmap v1691: 17 pgs: 17 active+clean; 839 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-29T14:35:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:06.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:07.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:07.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1190971117' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:35:07.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:06 smithi194 ceph-mon[44089]: osdmap e661: 12 total, 12 up, 8 in 2024-04-29T14:35:07.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:06 smithi194 ceph-mon[44089]: pgmap v1691: 17 pgs: 17 active+clean; 839 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-29T14:35:07.094 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:07.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:07.163 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:35:07.219 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:07.262 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:07.691 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 16 29151 29135 227.573 189.75 0.00584587 0.00439185 2024-04-29T14:35:08.691 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 16 32080 32064 222.626 183.062 0.0040798 0.00448961 2024-04-29T14:35:08.833 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:08.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:08.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:08.968 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:09.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:09.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:09.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: pgmap v1692: 17 pgs: 17 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 162 MiB/s wr, 2.58k op/s 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osdmap e662: 12 total, 12 up, 8 in 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/368372912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:35:09.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:35:09.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:35:09.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:35:09.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:35:09.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:35:09.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:35:09.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:35:09.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:35:09.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:35:09.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: pgmap v1692: 17 pgs: 17 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 162 MiB/s wr, 2.58k op/s 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osdmap e662: 12 total, 12 up, 8 in 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/368372912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.531 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: pgmap v1692: 17 pgs: 17 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 162 MiB/s wr, 2.58k op/s 2024-04-29T14:35:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osdmap e662: 12 total, 12 up, 8 in 2024-04-29T14:35:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/368372912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:09.692 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 16 34542 34526 215.723 153.875 0.00302126 0.00463337 2024-04-29T14:35:09.697 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2] out_osds: [1, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:35:09.697 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:35:09.697 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:35:09.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:35:10.694 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 37195 37179 211.162 165.812 0.00936708 0.0047338 2024-04-29T14:35:11.114 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:11.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:11.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:11.250 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:11.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:11.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:11.348 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:11 smithi194 ceph-mon[44089]: pgmap v1694: 17 pgs: 17 active+clean; 1.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 194 MiB/s wr, 3.11k op/s 2024-04-29T14:35:11.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:11 smithi040 ceph-mon[39581]: pgmap v1694: 17 pgs: 17 active+clean; 1.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 194 MiB/s wr, 3.11k op/s 2024-04-29T14:35:11.377 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:11.600 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:11.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:11 smithi121 ceph-mon[44533]: pgmap v1694: 17 pgs: 17 active+clean; 1.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 194 MiB/s wr, 3.11k op/s 2024-04-29T14:35:11.694 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 39664 39648 206.425 154.312 0.00597124 0.00484208 2024-04-29T14:35:11.792 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:11.796 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:12.260 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:12.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:12.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:12.396 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:12.417 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:12.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:12.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:12.547 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:35:12.547 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:35:12.547 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:35:12.547 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:35:12.547 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:35:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:12.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:12 smithi194 ceph-mon[44089]: pgmap v1695: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 236 MiB/s wr, 3.77k op/s 2024-04-29T14:35:12.571 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:12.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:12.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:12 smithi040 ceph-mon[39581]: pgmap v1695: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 236 MiB/s wr, 3.77k op/s 2024-04-29T14:35:12.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:12 smithi121 ceph-mon[44533]: pgmap v1695: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 236 MiB/s wr, 3.77k op/s 2024-04-29T14:35:12.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:12.694 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 16 42274 42258 203.094 163.125 0.00282231 0.00492135 2024-04-29T14:35:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:12.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:12.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:12.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:13.394 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:13.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:13.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:13.549 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:13.694 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 16 44724 44708 199.525 153.125 0.00776031 0.00501018 2024-04-29T14:35:13.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:35:13.905 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:13.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:14.010 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:14.052 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:14.263 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:14.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:14.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:35:14.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:14.397 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:14.418 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:14.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:14.523 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:14.535 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:14.568 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:14.600 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:14.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:14.695 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 16 46760 46744 194.707 127.25 0.0116839 0.00513345 2024-04-29T14:35:14.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:14.751 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:35:14.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:14.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:14.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:14 smithi194 ceph-mon[44089]: osdmap e663: 12 total, 12 up, 8 in 2024-04-29T14:35:14.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:14 smithi194 ceph-mon[44089]: pgmap v1697: 17 pgs: 17 active+clean; 2.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 185 MiB/s wr, 2.96k op/s 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 663, 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:35:13.638814+0000", 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:31:03.499853+0000", 2024-04-29T14:35:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:33:32.300692+0000", 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5801984, 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "noscrub", 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 30, 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 7, 2024-04-29T14:35:14.883 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "jewel", 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:35:14.884 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:35:14.885 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:35:14.886 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:35:14.887 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:35:14.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 10, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 10, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.20000000298023224, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 7, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_5", 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:34:47.804723+0000", 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:35:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "659", 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:35:14.891 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:35:14.892 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:35:14.893 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 3.130000114440918, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 3.130000114440918, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.80000001192092896, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2.5, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2.5, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:35:14.894 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:35:14.895 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 629, 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:35:14.896 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 160, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.901 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:35:14.902 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 655, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.904 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.905 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.906 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:35:14.907 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.908 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:35:14.909 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.5530548095703125, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 629, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.912 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:35:14.913 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 388, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 457, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 561, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 432, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 459, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.914 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:35:14.915 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/2522198013", 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/2522198013", 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/2522198013", 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/2522198013", 2024-04-29T14:35:14.916 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.95001220703125, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 655, 2024-04-29T14:35:14.917 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.920 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.5831451416015625, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 634, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:35:14.921 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:35:14.923 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:35:14.924 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 206, 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.927 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 655, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:35:14.928 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:35:14.930 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.931 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 655, 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.4297637939453125, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 524, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:35:14.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.937 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.938 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:33.588005+0000", 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 51693, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:32.977917+0000", 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.939 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:33:55.763817+0000", 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:35:13.586878+0000", 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:35:08.274496+0000", 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.941 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:49.404679+0000", 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 457 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:04.257760+0000", 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:35:14.942 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:45.541348+0000", 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:58.852312+0000", 2024-04-29T14:35:14.943 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:44.042872+0000", 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.944 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:34:01.650789+0000", 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:33:59.028493+0000", 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:35:14.945 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [], 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:35:14.946 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:35:14.947 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:35:14.948 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:35:14.948 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:35:14.948 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:35:14.948 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:35:14.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:14.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:14.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:14 smithi040 ceph-mon[39581]: osdmap e663: 12 total, 12 up, 8 in 2024-04-29T14:35:14.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:14 smithi040 ceph-mon[39581]: pgmap v1697: 17 pgs: 17 active+clean; 2.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 185 MiB/s wr, 2.96k op/s 2024-04-29T14:35:15.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:15.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:15.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:14 smithi121 ceph-mon[44533]: osdmap e663: 12 total, 12 up, 8 in 2024-04-29T14:35:15.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:14 smithi121 ceph-mon[44533]: pgmap v1697: 17 pgs: 17 active+clean; 2.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 185 MiB/s wr, 2.96k op/s 2024-04-29T14:35:15.694 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 16 49191 49175 192.034 151.938 0.00287252 0.0052059 2024-04-29T14:35:15.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1937154695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:35:16.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1937154695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:35:16.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1937154695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:35:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:16.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:16.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:16.694 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 51575 51559 189.502 149 0.00548025 0.0052753 2024-04-29T14:35:16.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:16.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:16.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:16.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:16.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:16.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:16 smithi040 ceph-mon[39581]: pgmap v1698: 17 pgs: 17 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-04-29T14:35:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:17.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:17.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:17.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:16 smithi194 ceph-mon[44089]: pgmap v1698: 17 pgs: 17 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-04-29T14:35:17.109 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:17.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:17.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:17.130 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:17.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:16 smithi121 ceph-mon[44533]: pgmap v1698: 17 pgs: 17 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-04-29T14:35:17.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:17.268 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:35:17.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:35:17.680 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:35:17.695 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 53645 53629 186.149 129.375 0.0110772 0.00537022 2024-04-29T14:35:17.711 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:35:17.729 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:17.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:17.738 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:35:17.766 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:35:17.794 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:35:17.822 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:35:17.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:17.854 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:35:17.885 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:17.892 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:35:17.910 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:17.913 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:17 smithi194 ceph-mon[44089]: osdmap e664: 12 total, 12 up, 8 in 2024-04-29T14:35:17.913 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3195056338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:17.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:17.930 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:35:17.965 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:35:17.997 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:17 smithi040 ceph-mon[39581]: osdmap e664: 12 total, 12 up, 8 in 2024-04-29T14:35:17.997 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3195056338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:18.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:18.070 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:18.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:17 smithi121 ceph-mon[44533]: osdmap e664: 12 total, 12 up, 8 in 2024-04-29T14:35:18.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3195056338' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:18.201 INFO:tasks.thrashosds.thrasher:j is {'epoch': 663, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:35:13.638814+0000', 'last_up_change': '2024-04-29T14:31:03.499853+0000', 'last_in_change': '2024-04-29T14:33:32.300692+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-04-29T14:34:47.804723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '659', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.130000114440918, 'score_stable': 3.130000114440918, 'optimal_score': 0.800000011920929, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 655, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 0.5530548095703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 388, 'last_clean_end': 457, 'up_from': 561, 'up_thru': 432, 'down_at': 459, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 2522198013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2522198013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2522198013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2522198013}]}, 'public_addr': '172.21.15.121:6804/2522198013', 'cluster_addr': '172.21.15.121:6805/2522198013', 'heartbeat_back_addr': '172.21.15.121:6807/2522198013', 'heartbeat_front_addr': '172.21.15.121:6806/2522198013', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 655, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 0.5831451416015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 634, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 206, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 655, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 655, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 0.4297637939453125, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 524, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:34:33.588005+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 51693, 'last_purged_snaps_scrub': '2024-04-29T14:34:32.977917+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:33:55.763817+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:35:13.586878+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:35:08.274496+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:34:49.404679+0000', 'dead_epoch': 457}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:34:04.257760+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:34:45.541348+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:34:58.852312+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:34:44.042872+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:34:01.650789+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:33:59.028493+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:35:18.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:35:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:35:18.696 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 55794 55778 183.42 134.312 0.00775455 0.00544948 2024-04-29T14:35:18.738 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:35:18.738 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:35:18.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3394206647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: from='client.36785 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:18.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:18 smithi040 ceph-mon[39581]: pgmap v1700: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-04-29T14:35:19.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3394206647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: from='client.36785 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:18 smithi194 ceph-mon[44089]: pgmap v1700: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-04-29T14:35:19.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3394206647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:19.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: from='client.36785 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:19.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:18 smithi121 ceph-mon[44533]: pgmap v1700: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-04-29T14:35:19.698 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:35:19.699402 min lat: 0.0010054 max lat: 0.207846 avg lat: 0.00552717 2024-04-29T14:35:19.698 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:35:19.698 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 16 57916 57900 180.863 132.625 0.00495563 0.00552717 2024-04-29T14:35:19.729 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:19.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:19.762 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:19.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:19.878 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:19.917 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:19.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:19.922 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:19 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:19.923 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:19 smithi194 ceph-mon[44089]: from='client.36785 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:35:19.923 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:19 smithi194 ceph-mon[44089]: osdmap e665: 12 total, 12 up, 8 in 2024-04-29T14:35:19.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:19.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:20.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:20.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:20.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:19 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:20.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:19 smithi040 ceph-mon[39581]: from='client.36785 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:35:20.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:19 smithi040 ceph-mon[39581]: osdmap e665: 12 total, 12 up, 8 in 2024-04-29T14:35:20.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:20.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:20.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:20.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:19 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:20.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:19 smithi121 ceph-mon[44533]: from='client.36785 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:35:20.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:19 smithi121 ceph-mon[44533]: osdmap e665: 12 total, 12 up, 8 in 2024-04-29T14:35:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:20.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:20.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:20.208 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:20.227 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:20.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:20.362 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:20.412 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:20.443 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:20.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:20.488 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:35:20.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:20.604 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:20.645 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:20.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:20.698 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 59905 59889 178.17 124.312 0.00654491 0.00561011 2024-04-29T14:35:20.751 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:20.790 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:20.821 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:20.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:20.924 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:35:20.927 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:20.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:20 smithi194 ceph-mon[44089]: pgmap v1702: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-29T14:35:20.969 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:20.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:35:21.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:20 smithi040 ceph-mon[39581]: pgmap v1702: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-29T14:35:21.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:20 smithi121 ceph-mon[44533]: pgmap v1702: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-29T14:35:21.238 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:21.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:21.344 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:21.381 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:21.403 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:21.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:21.700 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 15 61834 61819 175.539 120.625 0.00873993 0.00569537 2024-04-29T14:35:21.719 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:21.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:35:21.769 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:21.937 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:21.937 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:21.937 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:21 smithi040 ceph-mon[39581]: osdmap e666: 12 total, 12 up, 8 in 2024-04-29T14:35:22.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:22.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:22.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:21 smithi194 ceph-mon[44089]: osdmap e666: 12 total, 12 up, 8 in 2024-04-29T14:35:22.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:22.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:22.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:21 smithi121 ceph-mon[44533]: osdmap e666: 12 total, 12 up, 8 in 2024-04-29T14:35:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:35:22.670 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:35:22.700 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 63811 63795 173.277 123.5 0.00354717 0.00576928 2024-04-29T14:35:22.741 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:22.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:22.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:22.892 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:23.497 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:23.497 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:23.497 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:23 smithi194 ceph-mon[44089]: pgmap v1704: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-04-29T14:35:23.497 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:23 smithi194 ceph-mon[44089]: osdmap e667: 12 total, 12 up, 8 in 2024-04-29T14:35:23.562 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:23.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:23.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:23.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:23.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:23 smithi040 ceph-mon[39581]: pgmap v1704: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-04-29T14:35:23.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:23 smithi040 ceph-mon[39581]: osdmap e667: 12 total, 12 up, 8 in 2024-04-29T14:35:23.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:23.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:23.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:23 smithi121 ceph-mon[44533]: pgmap v1704: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-04-29T14:35:23.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:23 smithi121 ceph-mon[44533]: osdmap e667: 12 total, 12 up, 8 in 2024-04-29T14:35:23.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:23.700 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 16 65835 65819 171.328 126.5 0.0133477 0.00583512 2024-04-29T14:35:23.713 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:23.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:23.962 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1705,"stamp":"2024-04-29T14:35:22.340230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3911909920,"num_objects":59684,"num_object_clones":0,"num_object_copies":119368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59684,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":59863,"num_write_kb":3822586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59819,"ondisk_log_size":59819,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13114216,"kb_used_data":7645012,"kb_used_omap":15,"kb_used_meta":5469104,"kb_avail":736847000,"statfs":{"total":767960285184,"available":754531328000,"internally_reserved":0,"allocated":7828492288,"data_stored":7824988768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15582,"internal_metadata":5600363298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":839254016,"num_objects":12806,"num_object_clones":0,"num_object_copies":25612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":819584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.679565"},"pg_stats":[{"pgid":"7.f","version":"666'3823","reported_seq":3854,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:21.076996+0000","last_change":"2024-04-29T14:34:49.390069+0000","last_active":"2024-04-29T14:35:21.076996+0000","last_peered":"2024-04-29T14:35:21.076996+0000","last_clean":"2024-04-29T14:35:21.076996+0000","last_became_active":"2024-04-29T14:34:49.389713+0000","last_became_peered":"2024-04-29T14:34:49.389713+0000","last_unstale":"2024-04-29T14:35:21.076996+0000","last_undegraded":"2024-04-29T14:35:21.076996+0000","last_fullsized":"2024-04-29T14:35:21.076996+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:10.173910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250544128,"num_objects":3823,"num_object_clones":0,"num_object_copies":7646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":244672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"666'4013","reported_seq":4044,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:22.276950+0000","last_change":"2024-04-29T14:34:49.387393+0000","last_active":"2024-04-29T14:35:22.276950+0000","last_peered":"2024-04-29T14:35:22.276950+0000","last_clean":"2024-04-29T14:35:22.276950+0000","last_became_active":"2024-04-29T14:34:49.387078+0000","last_became_peered":"2024-04-29T14:34:49.387078+0000","last_unstale":"2024-04-29T14:35:22.276950+0000","last_undegraded":"2024-04-29T14:35:22.276950+0000","last_fullsized":"2024-04-29T14:35:22.276950+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:50.804757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262995968,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":256832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"666'3709","reported_seq":3740,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:20.700757+0000","last_change":"2024-04-29T14:34:49.389726+0000","last_active":"2024-04-29T14:35:20.700757+0000","last_peered":"2024-04-29T14:35:20.700757+0000","last_clean":"2024-04-29T14:35:20.700757+0000","last_became_active":"2024-04-29T14:34:49.389551+0000","last_became_peered":"2024-04-29T14:34:49.389551+0000","last_unstale":"2024-04-29T14:35:20.700757+0000","last_undegraded":"2024-04-29T14:35:20.700757+0000","last_fullsized":"2024-04-29T14:35:20.700757+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:59.519121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243073024,"num_objects":3709,"num_object_clones":0,"num_object_copies":7418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3709,"num_write_kb":237376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"666'3756","reported_seq":3787,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:20.685735+0000","last_change":"2024-04-29T14:34:49.386907+0000","last_active":"2024-04-29T14:35:20.685735+0000","last_peered":"2024-04-29T14:35:20.685735+0000","last_clean":"2024-04-29T14:35:20.685735+0000","last_became_active":"2024-04-29T14:34:49.386724+0000","last_became_peered":"2024-04-29T14:34:49.386724+0000","last_unstale":"2024-04-29T14:35:20.685735+0000","last_undegraded":"2024-04-29T14:35:20.685735+0000","last_fullsized":"2024-04-29T14:35:20.685735+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:16.401985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246153216,"num_objects":3756,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":240384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"665'3541","reported_seq":3570,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-29T14:35:19.092665+0000","last_change":"2024-04-29T14:34:49.387810+0000","last_active":"2024-04-29T14:35:19.092665+0000","last_peered":"2024-04-29T14:35:19.092665+0000","last_clean":"2024-04-29T14:35:19.092665+0000","last_became_active":"2024-04-29T14:34:49.387508+0000","last_became_peered":"2024-04-29T14:34:49.387508+0000","last_unstale":"2024-04-29T14:35:19.092665+0000","last_undegraded":"2024-04-29T14:35:19.092665+0000","last_fullsized":"2024-04-29T14:35:19.092665+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:03.136994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232062976,"num_objects":3541,"num_object_clones":0,"num_object_copies":7082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3541,"num_write_kb":226624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.a","version":"666'3894","reported_seq":3925,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:21.089316+0000","last_change":"2024-04-29T14:34:49.387962+0000","last_active":"2024-04-29T14:35:21.089316+0000","last_peered":"2024-04-29T14:35:21.089316+0000","last_clean":"2024-04-29T14:35:21.089316+0000","last_became_active":"2024-04-29T14:34:49.387647+0000","last_became_peered":"2024-04-29T14:34:49.387647+0000","last_unstale":"2024-04-29T14:35:21.089316+0000","last_undegraded":"2024-04-29T14:35:21.089316+0000","last_fullsized":"2024-04-29T14:35:21.089316+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:49.714282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255197184,"num_objects":3894,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":249216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"665'3553","reported_seq":3582,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-29T14:35:19.113275+0000","last_change":"2024-04-29T14:34:49.390147+0000","last_active":"2024-04-29T14:35:19.113275+0000","last_peered":"2024-04-29T14:35:19.113275+0000","last_clean":"2024-04-29T14:35:19.113275+0000","last_became_active":"2024-04-29T14:34:49.389823+0000","last_became_peered":"2024-04-29T14:34:49.389823+0000","last_unstale":"2024-04-29T14:35:19.113275+0000","last_undegraded":"2024-04-29T14:35:19.113275+0000","last_fullsized":"2024-04-29T14:35:19.113275+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:57.498260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":7106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":227392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.8","version":"666'3914","reported_seq":3943,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:22.271592+0000","last_change":"2024-04-29T14:34:49.515093+0000","last_active":"2024-04-29T14:35:22.271592+0000","last_peered":"2024-04-29T14:35:22.271592+0000","last_clean":"2024-04-29T14:35:22.271592+0000","last_became_active":"2024-04-29T14:34:49.514880+0000","last_became_peered":"2024-04-29T14:34:49.514880+0000","last_unstale":"2024-04-29T14:35:22.271592+0000","last_undegraded":"2024-04-29T14:35:22.271592+0000","last_fullsized":"2024-04-29T14:35:22.271592+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:15.887128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256507904,"num_objects":3914,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":250496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"666'3810","reported_seq":3839,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:20.710812+0000","last_change":"2024-04-29T14:34:49.388599+0000","last_active":"2024-04-29T14:35:20.710812+0000","last_peered":"2024-04-29T14:35:20.710812+0000","last_clean":"2024-04-29T14:35:20.710812+0000","last_became_active":"2024-04-29T14:34:49.388403+0000","last_became_peered":"2024-04-29T14:34:49.388403+0000","last_unstale":"2024-04-29T14:35:20.710812+0000","last_undegraded":"2024-04-29T14:35:20.710812+0000","last_fullsized":"2024-04-29T14:35:20.710812+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3810,"log_dups_size":0,"ondisk_log_size":3810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:17.801058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249692160,"num_objects":3810,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3810,"num_write_kb":243840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"666'3845","reported_seq":3876,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:21.097752+0000","last_change":"2024-04-29T14:34:49.388873+0000","last_active":"2024-04-29T14:35:21.097752+0000","last_peered":"2024-04-29T14:35:21.097752+0000","last_clean":"2024-04-29T14:35:21.097752+0000","last_became_active":"2024-04-29T14:34:49.388394+0000","last_became_peered":"2024-04-29T14:34:49.388394+0000","last_unstale":"2024-04-29T14:35:21.097752+0000","last_undegraded":"2024-04-29T14:35:21.097752+0000","last_fullsized":"2024-04-29T14:35:21.097752+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:05.171204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251985920,"num_objects":3845,"num_object_clones":0,"num_object_copies":7690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3845,"num_write_kb":246080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"666'3848","reported_seq":3879,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:21.100571+0000","last_change":"2024-04-29T14:34:49.390551+0000","last_active":"2024-04-29T14:35:21.100571+0000","last_peered":"2024-04-29T14:35:21.100571+0000","last_clean":"2024-04-29T14:35:21.100571+0000","last_became_active":"2024-04-29T14:34:49.390276+0000","last_became_peered":"2024-04-29T14:34:49.390276+0000","last_unstale":"2024-04-29T14:35:21.100571+0000","last_undegraded":"2024-04-29T14:35:21.100571+0000","last_fullsized":"2024-04-29T14:35:21.100571+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:13.163642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252182528,"num_objects":3848,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":246272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.3","version":"666'3824","reported_seq":3855,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:22.271429+0000","last_change":"2024-04-29T14:34:49.592437+0000","last_active":"2024-04-29T14:35:22.271429+0000","last_peered":"2024-04-29T14:35:22.271429+0000","last_clean":"2024-04-29T14:35:22.271429+0000","last_became_active":"2024-04-29T14:34:49.592105+0000","last_became_peered":"2024-04-29T14:34:49.592105+0000","last_unstale":"2024-04-29T14:35:22.271429+0000","last_undegraded":"2024-04-29T14:35:22.271429+0000","last_fullsized":"2024-04-29T14:35:22.271429+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:16.575881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250609664,"num_objects":3824,"num_object_clones":0,"num_object_copies":7648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":244736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"665'3481","reported_seq":3510,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-29T14:35:19.105196+0000","last_change":"2024-04-29T14:34:49.387331+0000","last_active":"2024-04-29T14:35:19.105196+0000","last_peered":"2024-04-29T14:35:19.105196+0000","last_clean":"2024-04-29T14:35:19.105196+0000","last_became_active":"2024-04-29T14:34:49.387077+0000","last_became_peered":"2024-04-29T14:34:49.387077+0000","last_unstale":"2024-04-29T14:35:19.105196+0000","last_undegraded":"2024-04-29T14:35:19.105196+0000","last_fullsized":"2024-04-29T14:35:19.105196+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:07.645531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228130816,"num_objects":3481,"num_object_clones":0,"num_object_copies":6962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3481,"num_write_kb":222784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"665'3442","reported_seq":3471,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-29T14:35:19.122847+0000","last_change":"2024-04-29T14:34:49.390028+0000","last_active":"2024-04-29T14:35:19.122847+0000","last_peered":"2024-04-29T14:35:19.122847+0000","last_clean":"2024-04-29T14:35:19.122847+0000","last_became_active":"2024-04-29T14:34:49.389739+0000","last_became_peered":"2024-04-29T14:34:49.389739+0000","last_unstale":"2024-04-29T14:35:19.122847+0000","last_undegraded":"2024-04-29T14:35:19.122847+0000","last_fullsized":"2024-04-29T14:35:19.122847+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3442,"log_dups_size":0,"ondisk_log_size":3442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:54.124232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225574912,"num_objects":3442,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":220288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"666'3760","reported_seq":3791,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:21.090553+0000","last_change":"2024-04-29T14:34:49.388617+0000","last_active":"2024-04-29T14:35:21.090553+0000","last_peered":"2024-04-29T14:35:21.090553+0000","last_clean":"2024-04-29T14:35:21.090553+0000","last_became_active":"2024-04-29T14:34:49.388328+0000","last_became_peered":"2024-04-29T14:34:49.388328+0000","last_unstale":"2024-04-29T14:35:21.090553+0000","last_undegraded":"2024-04-29T14:35:21.090553+0000","last_fullsized":"2024-04-29T14:35:21.090553+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:48.983426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246415360,"num_objects":3760,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3760,"num_write_kb":240640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"665'3469","reported_seq":3498,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-04-29T14:35:19.119975+0000","last_change":"2024-04-29T14:34:49.389977+0000","last_active":"2024-04-29T14:35:19.119975+0000","last_peered":"2024-04-29T14:35:19.119975+0000","last_clean":"2024-04-29T14:35:19.119975+0000","last_became_active":"2024-04-29T14:34:49.389669+0000","last_became_peered":"2024-04-29T14:34:49.389669+0000","last_unstale":"2024-04-29T14:35:19.119975+0000","last_undegraded":"2024-04-29T14:35:19.119975+0000","last_fullsized":"2024-04-29T14:35:19.119975+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:16.629361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227344384,"num_objects":3469,"num_object_clones":0,"num_object_copies":6938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3469,"num_write_kb":222016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1624,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-04-29T14:35:20.685468+0000","last_change":"2024-04-29T14:34:48.012622+0000","last_active":"2024-04-29T14:35:20.685468+0000","last_peered":"2024-04-29T14:35:20.685468+0000","last_clean":"2024-04-29T14:35:20.685468+0000","last_became_active":"2024-04-29T14:32:42.114595+0000","last_became_peered":"2024-04-29T14:32:42.114595+0000","last_unstale":"2024-04-29T14:35:20.685468+0000","last_undegraded":"2024-04-29T14:35:20.685468+0000","last_fullsized":"2024-04-29T14:35:20.685468+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:34:48.012447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.012447+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:04.370874+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3911319552,"num_objects":59682,"num_object_clones":0,"num_object_copies":119364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59682,"num_write_kb":3819648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7923367936,"data_stored":7923367936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59682,"ondisk_log_size":59682,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1176700,"kb_used_data":477748,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92568452,"statfs":{"total":95995035648,"available":94790094848,"internally_reserved":0,"allocated":489213952,"data_stored":488779234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":715717060},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384025,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2090368,"kb_used_data":1328948,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":91654784,"statfs":{"total":95995035648,"available":93854498816,"internally_reserved":0,"allocated":1360842752,"data_stored":1360408034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":779680192},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645663,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2089096,"kb_used_data":1444284,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91656056,"statfs":{"total":95995035648,"available":93855801344,"internally_reserved":0,"allocated":1478946816,"data_stored":1478508566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":0,"seq":1520418422967,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053832,"kb_used_data":250752,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92691320,"statfs":{"total":95995035648,"available":94915911680,"internally_reserved":0,"allocated":256770048,"data_stored":256329086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":822344138},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528686,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1549320,"kb_used_data":705980,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92195832,"statfs":{"total":95995035648,"available":94408531968,"internally_reserved":0,"allocated":722923520,"data_stored":722485270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863566278},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738764,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2133372,"kb_used_data":1426100,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91611780,"statfs":{"total":95995035648,"available":93810462720,"internally_reserved":0,"allocated":1460326400,"data_stored":1459891702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497197,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1895308,"kb_used_data":1736640,"kb_used_omap":4,"kb_used_meta":158651,"kb_avail":91849844,"statfs":{"total":95995035648,"available":94054240256,"internally_reserved":0,"allocated":1778319360,"data_stored":1777878398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":162459332},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182781,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1126220,"kb_used_data":274560,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92618932,"statfs":{"total":95995035648,"available":94841786368,"internally_reserved":0,"allocated":281149440,"data_stored":280708478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":2409476653108,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1479081984,"data_stored":1479081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1804402688,"data_stored":1804402688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":247857152,"data_stored":247857152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":739115008,"data_stored":739115008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":256180224,"data_stored":256180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1508835328,"data_stored":1508835328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1383268352,"data_stored":1383268352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":504627200,"data_stored":504627200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:35:23.964 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [9, 2, 8, 11] 2024-04-29T14:35:23.964 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.0', '9', '2', '8', '11'] 2024-04-29T14:35:23.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 1.0 9 2 8 11 2024-04-29T14:35:23.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:24.009 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:24.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:24.135 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:24.188 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:24.220 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:24.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:24.326 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:24.377 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:24.415 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:24.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:24.465 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:24 smithi194 ceph-mon[44089]: from='client.17349 v1:172.21.15.40:0/2998403792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:35:24.486 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:35:24.486 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:35:24.527 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:24.570 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:24.588 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:35:24.603 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:24.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:24.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:24 smithi040 ceph-mon[39581]: from='client.17349 v1:172.21.15.40:0/2998403792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:35:24.621 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:35:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:24.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:24 smithi121 ceph-mon[44533]: from='client.17349 v1:172.21.15.40:0/2998403792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:35:24.667 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:35:24.694 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:35:24.700 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 16 67505 67489 168.651 104.375 0.00344378 0.00592676 2024-04-29T14:35:24.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:24.714 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:35:24.746 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:35:24.755 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:24.773 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:35:24.777 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:24.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:24.804 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:35:24.842 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:35:24.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:35:24.881 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:35:24.927 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:24.936 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:35:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:35:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:35:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:35:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:35:24.938 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:35:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:35:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:35:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:35:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:35:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:35:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:35:24.939 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:35:25.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: pgmap v1706: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1701964853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: pgmap v1706: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1701964853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.700 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 69371 69355 166.65 116.625 0.0048715 0.00599846 2024-04-29T14:35:25.814 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:25.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: pgmap v1706: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-29T14:35:25.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1701964853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:25.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.815 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.816 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:25.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:25.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:25.958 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:26.209 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:26.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:26.260 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:35:26.316 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:26.369 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:26.400 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:26.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:26.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:26.534 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:26.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:26.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:26.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:26.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:26.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:26.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:26 smithi194 ceph-mon[44089]: osdmap e668: 12 total, 12 up, 8 in 2024-04-29T14:35:26.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:26 smithi194 ceph-mon[44089]: pgmap v1708: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-29T14:35:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:26.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:26 smithi040 ceph-mon[39581]: osdmap e668: 12 total, 12 up, 8 in 2024-04-29T14:35:26.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:26 smithi040 ceph-mon[39581]: pgmap v1708: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-29T14:35:26.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:26.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:26.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:26.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:26 smithi121 ceph-mon[44533]: osdmap e668: 12 total, 12 up, 8 in 2024-04-29T14:35:26.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:26 smithi121 ceph-mon[44533]: pgmap v1708: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-29T14:35:26.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:26.687 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:26.692 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:35:26.700 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 16 71080 71064 164.434 106.812 0.0121649 0.0060796 2024-04-29T14:35:26.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:26.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:26.811 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:26.850 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:26.942 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:26.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:27.046 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:27.080 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:27.701 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 16 72734 72718 162.25 103.375 0.00380967 0.00616115 2024-04-29T14:35:28.045 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:28.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:28.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:28.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:28.181 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:28.356 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:35:28.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:28 smithi040 ceph-mon[39581]: osdmap e669: 12 total, 12 up, 8 in 2024-04-29T14:35:28.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3250870522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:35:28.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:28 smithi040 ceph-mon[39581]: from='client.36803 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:35:28.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:28 smithi121 ceph-mon[44533]: osdmap e669: 12 total, 12 up, 8 in 2024-04-29T14:35:28.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3250870522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:35:28.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:28 smithi121 ceph-mon[44533]: from='client.36803 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:35:28.701 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 74164 74148 159.738 89.375 0.00282612 0.00624374 2024-04-29T14:35:28.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:35:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:28 smithi194 ceph-mon[44089]: osdmap e669: 12 total, 12 up, 8 in 2024-04-29T14:35:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3250870522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:35:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:28 smithi194 ceph-mon[44089]: from='client.36803 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:35:29.134 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:29.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:29.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:29.278 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:29.313 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:29.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:29.423 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:29.476 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:29 smithi194 ceph-mon[44089]: pgmap v1710: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-29T14:35:29.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:29 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:29.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:29 smithi194 ceph-mon[44089]: from='client.36803 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:35:29.477 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:29 smithi194 ceph-mon[44089]: osdmap e670: 12 total, 12 up, 8 in 2024-04-29T14:35:29.480 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:29.592 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:29.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:29.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:29.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:29 smithi040 ceph-mon[39581]: pgmap v1710: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-29T14:35:29.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:29 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:29.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:29 smithi040 ceph-mon[39581]: from='client.36803 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:35:29.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:29 smithi040 ceph-mon[39581]: osdmap e670: 12 total, 12 up, 8 in 2024-04-29T14:35:29.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:29 smithi121 ceph-mon[44533]: pgmap v1710: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-29T14:35:29.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:29 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:29.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:29 smithi121 ceph-mon[44533]: from='client.36803 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:35:29.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:29 smithi121 ceph-mon[44533]: osdmap e670: 12 total, 12 up, 8 in 2024-04-29T14:35:29.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:29.701 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 16 75823 75807 157.87 103.688 0.0115513 0.00633261 2024-04-29T14:35:29.724 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:29.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:29.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:29.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:29.904 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:29.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:30.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:30.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:30.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:30.155 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:35:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:35:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:35:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:35:30.156 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:35:30.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:30.216 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:30.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:30.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:30.343 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:30.354 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/901813551' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]: dispatch 2024-04-29T14:35:30.354 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:30 smithi121 ceph-mon[44533]: from='client.36809 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]: dispatch 2024-04-29T14:35:30.361 INFO:teuthology.orchestra.run.smithi040.stderr:set 1.0 pg_upmap_items mapping to [9->2,8->11] 2024-04-29T14:35:30.391 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:30.420 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:30.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:30.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/901813551' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]: dispatch 2024-04-29T14:35:30.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:30 smithi040 ceph-mon[39581]: from='client.36809 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]: dispatch 2024-04-29T14:35:30.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:30.556 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:30.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/901813551' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]: dispatch 2024-04-29T14:35:30.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:30 smithi194 ceph-mon[44089]: from='client.36809 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]: dispatch 2024-04-29T14:35:30.703 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 16 77544 77528 156.237 107.562 0.00486933 0.00639896 2024-04-29T14:35:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:31 smithi194 ceph-mon[44089]: pgmap v1712: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-29T14:35:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:31 smithi194 ceph-mon[44089]: from='client.36809 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]': finished 2024-04-29T14:35:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:31 smithi194 ceph-mon[44089]: osdmap e671: 12 total, 12 up, 8 in 2024-04-29T14:35:31.612 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:35:31.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:31 smithi040 ceph-mon[39581]: pgmap v1712: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-29T14:35:31.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:31 smithi040 ceph-mon[39581]: from='client.36809 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]': finished 2024-04-29T14:35:31.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:31 smithi040 ceph-mon[39581]: osdmap e671: 12 total, 12 up, 8 in 2024-04-29T14:35:31.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:31 smithi121 ceph-mon[44533]: pgmap v1712: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-29T14:35:31.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:31 smithi121 ceph-mon[44533]: from='client.36809 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [9, 2, 8, 11]}]': finished 2024-04-29T14:35:31.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:31 smithi121 ceph-mon[44533]: osdmap e671: 12 total, 12 up, 8 in 2024-04-29T14:35:31.703 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 79284 79268 154.754 108.75 0.00593591 0.00645968 2024-04-29T14:35:31.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:35:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:31.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:31.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:31.949 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:31.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:31.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:32.019 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:35:32.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:32.102 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:32.124 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:32.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:32.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:32.278 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:32.309 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:32.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:32.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:32 smithi040 ceph-mon[39581]: osdmap e672: 12 total, 12 up, 8 in 2024-04-29T14:35:32.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:32 smithi040 ceph-mon[39581]: pgmap v1715: 17 pgs: 1 peering, 16 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-29T14:35:32.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:32 smithi040 ceph-mon[39581]: osdmap e673: 12 total, 12 up, 8 in 2024-04-29T14:35:32.416 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:32.454 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:32.503 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:32.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:32.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:32 smithi194 ceph-mon[44089]: osdmap e672: 12 total, 12 up, 8 in 2024-04-29T14:35:32.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:32 smithi194 ceph-mon[44089]: pgmap v1715: 17 pgs: 1 peering, 16 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-29T14:35:32.551 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:32 smithi194 ceph-mon[44089]: osdmap e673: 12 total, 12 up, 8 in 2024-04-29T14:35:32.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:32 smithi121 ceph-mon[44533]: osdmap e672: 12 total, 12 up, 8 in 2024-04-29T14:35:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:32 smithi121 ceph-mon[44533]: pgmap v1715: 17 pgs: 1 peering, 16 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-29T14:35:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:32 smithi121 ceph-mon[44533]: osdmap e673: 12 total, 12 up, 8 in 2024-04-29T14:35:32.660 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:32.684 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:32.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:32.703 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 80816 80800 152.966 95.75 0.00484391 0.00653565 2024-04-29T14:35:32.792 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:32.841 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:32.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:32.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:33.028 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:33.055 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:35:33.066 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:33.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:33.078 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:33.090 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:35:33.127 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:35:33.159 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:35:33.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:33.186 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:35:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:33.226 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:35:33.247 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:33.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:33.258 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:35:33.283 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:35:33.313 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:35:33.347 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:35:33.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:33.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:35:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:33.417 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:35:33.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:35:33.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:35:33.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:35:33.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:35:33.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:35:33.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:35:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:35:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:35:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:35:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:35:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:35:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:35:33.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:33.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:33.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/285417201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:33.585 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:33.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/285417201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:33.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:33.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:33.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:33.703 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 16 82428 82412 151.43 100.75 0.00689692 0.006602 2024-04-29T14:35:33.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:33.838 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:33.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/285417201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:33.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:33.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:35:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:34 smithi040 ceph-mon[39581]: pgmap v1717: 17 pgs: 1 peering, 16 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-29T14:35:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:34.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 83990 83974 149.893 97.625 0.0114479 0.00666911 2024-04-29T14:35:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:34.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:34.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:35:34.788 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:34 smithi194 ceph-mon[44089]: pgmap v1717: 17 pgs: 1 peering, 16 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-29T14:35:34.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:34.838 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:35:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:34 smithi121 ceph-mon[44533]: pgmap v1717: 17 pgs: 1 peering, 16 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-29T14:35:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:35.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:35.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:35.334 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:35.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:35.428 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:35:35.428 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:35:35.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:35.500 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:35.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 85614 85598 148.549 101.5 0.00378216 0.0067301 2024-04-29T14:35:35.867 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:35.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:35.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:36.004 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:36.026 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:36.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:36.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:36.159 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:36.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:36.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:36.284 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:36.329 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:36.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 87237 87221 147.276 101.438 0.00812511 0.00678861 2024-04-29T14:35:36.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:35:37.033 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:37.085 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:37.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:37.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:35:37.169 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:35:37.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:37.232 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:37.256 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:37.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:37.360 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:37.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:37 smithi194 ceph-mon[44089]: pgmap v1718: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s; 86 KiB/s, 0 objects/s recovering 2024-04-29T14:35:37.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2657034621' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:37.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:37 smithi194 ceph-mon[44089]: from='client.27055 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:37.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:37.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:37.403 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:37.435 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:37.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:37.506 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2] out_osds: [1, 3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:35:37.506 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:35:37.506 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-04-29T14:35:37.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 1 2024-04-29T14:35:37.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:37 smithi040 ceph-mon[39581]: pgmap v1718: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s; 86 KiB/s, 0 objects/s recovering 2024-04-29T14:35:37.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2657034621' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:37.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:37 smithi040 ceph-mon[39581]: from='client.27055 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:37.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:37.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:37.541 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:37.577 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:37.603 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:37.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:37 smithi121 ceph-mon[44533]: pgmap v1718: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s; 86 KiB/s, 0 objects/s recovering 2024-04-29T14:35:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2657034621' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:37 smithi121 ceph-mon[44533]: from='client.27055 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:37.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 88384 88368 145.288 71.6875 0.0187243 0.00688082 2024-04-29T14:35:37.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:37.735 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:37.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:37.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:37.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:37.891 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:37.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:37.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:38.017 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:38.059 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:38.097 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:38.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:38.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:38.241 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:38.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:35:38.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:35:38.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: from='client.27055 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:35:38.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: osdmap e674: 12 total, 12 up, 8 in 2024-04-29T14:35:38.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:35:38.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:35:38.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:38.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:38 smithi194 ceph-mon[44089]: osdmap e675: 12 total, 12 up, 8 in 2024-04-29T14:35:38.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:35:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:35:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: from='client.27055 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:35:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: osdmap e674: 12 total, 12 up, 8 in 2024-04-29T14:35:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:35:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:35:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:38 smithi040 ceph-mon[39581]: osdmap e675: 12 total, 12 up, 8 in 2024-04-29T14:35:38.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:35:38.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:35:38.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: from='client.27055 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:35:38.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: osdmap e674: 12 total, 12 up, 8 in 2024-04-29T14:35:38.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:35:38.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:35:38.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:35:38.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:38 smithi121 ceph-mon[44533]: osdmap e675: 12 total, 12 up, 8 in 2024-04-29T14:35:38.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 90037 90021 144.211 103.312 0.00624884 0.00693262 2024-04-29T14:35:39.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:39 smithi194 ceph-mon[44089]: pgmap v1721: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.59k op/s; 86 KiB/s, 0 objects/s recovering 2024-04-29T14:35:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:39 smithi040 ceph-mon[39581]: pgmap v1721: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.59k op/s; 86 KiB/s, 0 objects/s recovering 2024-04-29T14:35:39.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:39 smithi121 ceph-mon[44533]: pgmap v1721: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.59k op/s; 86 KiB/s, 0 objects/s recovering 2024-04-29T14:35:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:39.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:35:39.705601 min lat: 0.0010054 max lat: 0.211929 avg lat: 0.00698652 2024-04-29T14:35:39.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:35:39.705 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 91634 91618 143.101 99.8125 0.00944453 0.00698652 2024-04-29T14:35:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:39.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:39.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:39.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:39.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:39.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:39.993 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:40.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:40.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:40.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:40.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:40.146 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:40.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:40.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:40.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:40.323 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:40.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 93284 93268 142.126 103.125 0.0121976 0.00703464 2024-04-29T14:35:41.114 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:41.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:41.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:41.264 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:41.301 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:41.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:41.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:41 smithi040 ceph-mon[39581]: pgmap v1722: 17 pgs: 17 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 67 KiB/s, 0 objects/s recovering 2024-04-29T14:35:41.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:41.443 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:41.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:41 smithi194 ceph-mon[44089]: pgmap v1722: 17 pgs: 17 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 67 KiB/s, 0 objects/s recovering 2024-04-29T14:35:41.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:41 smithi121 ceph-mon[44533]: pgmap v1722: 17 pgs: 17 active+clean; 5.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 67 KiB/s, 0 objects/s recovering 2024-04-29T14:35:41.704 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 16 94930 94914 141.192 102.875 0.0102737 0.00708082 2024-04-29T14:35:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:41.795 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:42.708 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 16 96509 96493 140.191 98.6875 0.00565098 0.00713126 2024-04-29T14:35:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:42.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:42.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:42.964 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:43.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: pgmap v1723: 17 pgs: 17 active+clean; 5.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/153658973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: pgmap v1723: 17 pgs: 17 active+clean; 5.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/153658973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: pgmap v1723: 17 pgs: 17 active+clean; 5.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s; 65 KiB/s, 0 objects/s recovering 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/153658973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:43.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:43.708 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 16 97979 97963 139.093 91.875 0.00434398 0.00717131 2024-04-29T14:35:43.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:43.767 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:35:43.767 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:35:43.767 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:35:43.767 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:35:43.767 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:35:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:35:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:35:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:35:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:35:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:35:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:35:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:35:43.785 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:43.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:43.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:43.911 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:43.949 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:43.982 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:43.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:44.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:44.135 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:44.319 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:44.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:44.498 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:44.535 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:44.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:44.708 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 99517 99501 138.138 96.125 0.00358191 0.00723782 2024-04-29T14:35:45.191 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:45.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:45.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:45 smithi040 ceph-mon[39581]: pgmap v1724: 17 pgs: 17 active+clean; 5.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-29T14:35:45.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:45.343 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:45.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:45 smithi194 ceph-mon[44089]: pgmap v1724: 17 pgs: 17 active+clean; 5.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-29T14:35:45.610 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:45.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:45.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:45 smithi121 ceph-mon[44533]: pgmap v1724: 17 pgs: 17 active+clean; 5.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-29T14:35:45.708 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 101064 101048 137.237 96.6875 0.00706378 0.0072853 2024-04-29T14:35:45.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:45.775 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:45.806 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:45.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:45.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:45 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:35:45.843+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e675 definitely_dead 0 2024-04-29T14:35:45.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:46.064 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:46.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:46.076 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:46.177 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.1. 2024-04-29T14:35:46.213 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:46.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/202214154' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-29T14:35:46.233 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:46.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:46.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:46.369 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:46.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/202214154' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-29T14:35:46.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/202214154' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-29T14:35:46.709 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 101557 101541 134.973 30.8125 0.00312514 0.00732949 2024-04-29T14:35:47.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:47.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:47.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:35:47.125 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:47.157 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:47.178 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:47.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:47.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:47.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:47.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:47.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:47 smithi040 ceph-mon[39581]: pgmap v1725: 17 pgs: 17 active+clean; 5.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:35:47.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/202214154' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-29T14:35:47.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:47 smithi040 ceph-mon[39581]: osdmap e676: 12 total, 12 up, 9 in 2024-04-29T14:35:47.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:47.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:47 smithi194 ceph-mon[44089]: pgmap v1725: 17 pgs: 17 active+clean; 5.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:35:47.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/202214154' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-29T14:35:47.460 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:47 smithi194 ceph-mon[44089]: osdmap e676: 12 total, 12 up, 9 in 2024-04-29T14:35:47.463 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:47.484 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:47.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:47.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:47.618 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:47.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:47 smithi121 ceph-mon[44533]: pgmap v1725: 17 pgs: 17 active+clean; 5.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-29T14:35:47.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/202214154' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-29T14:35:47.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:47 smithi121 ceph-mon[44533]: osdmap e676: 12 total, 12 up, 9 in 2024-04-29T14:35:47.637 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:47.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:47.709 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 16 101599 101583 132.217 2.625 0.00354626 0.00742377 2024-04-29T14:35:47.741 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:47.768 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:47.786 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:47.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:47.890 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:47.916 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:47.935 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:47.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:48.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:48.066 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:48.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:48.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:48.193 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:48.220 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:48.240 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:48.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:48.297 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:35:48.297 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:35:48.297 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:35:48.297 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:35:48.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:48 smithi040 ceph-mon[39581]: osdmap e677: 12 total, 12 up, 9 in 2024-04-29T14:35:48.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:48.370 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:48 smithi121 ceph-mon[44533]: osdmap e677: 12 total, 12 up, 9 in 2024-04-29T14:35:48.372 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:48.391 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:48.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:48.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:48.526 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:48.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:48.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:48.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:48 smithi194 ceph-mon[44089]: osdmap e677: 12 total, 12 up, 9 in 2024-04-29T14:35:48.654 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:48.709 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 101801 101785 129.777 12.625 0.031053 0.00770109 2024-04-29T14:35:48.762 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:49.136 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-04-29T14:35:49.136 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:49.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:49.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:49.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:49.247 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:35:49.271 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:49.297 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:49.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:49.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:49 smithi040 ceph-mon[39581]: pgmap v1728: 17 pgs: 2 unknown, 15 active+clean; 5.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 920 KiB/s wr, 14 op/s 2024-04-29T14:35:49.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:49 smithi040 ceph-mon[39581]: osdmap e678: 12 total, 12 up, 9 in 2024-04-29T14:35:49.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:49.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:49.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1495812821' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:49.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:49 smithi040 ceph-mon[39581]: from='client.27067 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:49.403 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:49.438 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:49.467 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:49.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:49 smithi194 ceph-mon[44089]: pgmap v1728: 17 pgs: 2 unknown, 15 active+clean; 5.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 920 KiB/s wr, 14 op/s 2024-04-29T14:35:49.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:49 smithi194 ceph-mon[44089]: osdmap e678: 12 total, 12 up, 9 in 2024-04-29T14:35:49.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:49.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:49.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1495812821' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:49.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:49 smithi194 ceph-mon[44089]: from='client.27067 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:49.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:49.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:49 smithi121 ceph-mon[44533]: pgmap v1728: 17 pgs: 2 unknown, 15 active+clean; 5.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 920 KiB/s wr, 14 op/s 2024-04-29T14:35:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:49 smithi121 ceph-mon[44533]: osdmap e678: 12 total, 12 up, 9 in 2024-04-29T14:35:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:49.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1495812821' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:49.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:49 smithi121 ceph-mon[44533]: from='client.27067 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:35:49.602 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:49.626 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:49.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:49.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:35:49.709 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 102719 102703 128.329 57.375 0.0551023 0.00779048 2024-04-29T14:35:49.731 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:49.765 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:49.788 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:49.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:49.885 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:49.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:49.919 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:49.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:49.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:50.045 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:50.074 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:50.104 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:50.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:50.213 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:50.277 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:50 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:50.277 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:50 smithi040 ceph-mon[39581]: from='client.27067 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:35:50.277 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:50 smithi040 ceph-mon[39581]: osdmap e679: 12 total, 12 up, 9 in 2024-04-29T14:35:50.281 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:50.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:50.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:35:50.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:50 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:50.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:50 smithi194 ceph-mon[44089]: from='client.27067 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:35:50.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:50 smithi194 ceph-mon[44089]: osdmap e679: 12 total, 12 up, 9 in 2024-04-29T14:35:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:50.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:50 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:35:50.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:50 smithi121 ceph-mon[44533]: from='client.27067 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:35:50.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:50 smithi121 ceph-mon[44533]: osdmap e679: 12 total, 12 up, 9 in 2024-04-29T14:35:50.711 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 103625 103609 126.917 56.625 0.0176791 0.00787671 2024-04-29T14:35:50.806 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:35:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:50.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:35:50.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:51.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:51 smithi040 ceph-mon[39581]: pgmap v1731: 17 pgs: 1 active+recovering+remapped, 1 unknown, 15 active+clean; 5.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 6303/190454 objects misplaced (3.309%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:35:51.439 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:51.455 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:51.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:51.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:51.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:51 smithi194 ceph-mon[44089]: pgmap v1731: 17 pgs: 1 active+recovering+remapped, 1 unknown, 15 active+clean; 5.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 6303/190454 objects misplaced (3.309%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:35:51.612 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:51 smithi121 ceph-mon[44533]: pgmap v1731: 17 pgs: 1 active+recovering+remapped, 1 unknown, 15 active+clean; 5.8 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 6303/190454 objects misplaced (3.309%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:35:51.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 104515 104499 125.547 55.625 0.0042269 0.0079604 2024-04-29T14:35:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:52.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:52.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:35:52.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:35:52.458 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:52.481 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:35:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:52.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:35:52.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:52.613 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:52.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:52.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:52.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 16 105420 105404 124.245 56.5625 0.00826989 0.00804433 2024-04-29T14:35:52.743 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:35:52.778 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:52.798 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:35:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:52.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:35:52.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:52.930 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:52.962 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:52.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:53.069 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:53.105 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:53.132 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:53.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:53.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:53.264 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:53.286 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:53.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:53.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:35:53.420 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:53.430 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:35:53.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:35:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:53.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:35:53.486 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:35:53.528 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:35:53.547 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:53.564 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:35:53.579 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:53.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:35:53.607 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:53.611 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:53 smithi194 ceph-mon[44089]: pgmap v1732: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 36 MiB/s wr, 581 op/s; 25158/205742 objects misplaced (12.228%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:35:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:53.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:53 smithi040 ceph-mon[39581]: pgmap v1732: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 36 MiB/s wr, 581 op/s; 25158/205742 objects misplaced (12.228%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:35:53.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:53 smithi121 ceph-mon[44533]: pgmap v1732: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 36 MiB/s wr, 581 op/s; 25158/205742 objects misplaced (12.228%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:35:53.646 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:35:53.686 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:35:53.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 106303 106287 122.967 55.1875 0.00525624 0.00812895 2024-04-29T14:35:53.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:53.724 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:35:53.741 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:53.765 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:53.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:53.776 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:35:53.827 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:35:53.848 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:53.872 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:53.884 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:35:53.906 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:53.930 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:35:53.933 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:53.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:35:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:35:54.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:54.137 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1] out_osds: [3, 8, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:35:54.137 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:35:54.137 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-29T14:35:54.137 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 5 2024-04-29T14:35:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:54.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:54.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:54.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:54.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:54.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:54.587 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3473581210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osdmap e680: 12 total, 12 up, 9 in 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.630 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:54.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:54.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:54.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3473581210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:54.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osdmap e680: 12 total, 12 up, 9 in 2024-04-29T14:35:54.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:54.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:54.634 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3473581210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osdmap e680: 12 total, 12 up, 9 in 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.635 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.636 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:54.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 107104 107088 121.642 50.0625 0.00946333 0.00821694 2024-04-29T14:35:54.735 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:54.773 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:54.797 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:54.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:54.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:35:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:35:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:54.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:35:55.057 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:35:55.089 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:35:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:55.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:35:55.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:55.270 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:55.306 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:55.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:55.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:55 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:55.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:55 smithi194 ceph-mon[44089]: pgmap v1734: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 162 MiB/s wr, 2.59k op/s; 25158/207154 objects misplaced (12.145%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:35:55.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:35:55.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:35:55.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:55 smithi194 ceph-mon[44089]: osdmap e681: 12 total, 12 up, 9 in 2024-04-29T14:35:55.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:55.468 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:35:55.469 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:35:55.474 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:55 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:55 smithi040 ceph-mon[39581]: pgmap v1734: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 162 MiB/s wr, 2.59k op/s; 25158/207154 objects misplaced (12.145%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:35:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:35:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:35:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:55 smithi040 ceph-mon[39581]: osdmap e681: 12 total, 12 up, 9 in 2024-04-29T14:35:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:55 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:35:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:55 smithi121 ceph-mon[44533]: pgmap v1734: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 162 MiB/s wr, 2.59k op/s; 25158/207154 objects misplaced (12.145%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:35:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:35:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:35:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:55 smithi121 ceph-mon[44533]: osdmap e681: 12 total, 12 up, 9 in 2024-04-29T14:35:55.710 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:55.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 108002 107986 120.472 56.125 0.0262925 0.00829812 2024-04-29T14:35:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:55.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:55.818 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:55.849 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:55.875 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:55.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:55.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:56.008 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:56.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:56.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:56.135 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:35:56.167 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:56.190 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:35:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:56.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:35:56.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:35:56.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:56.323 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:56.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:56.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:56.398 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:56 smithi040 ceph-mon[39581]: pgmap v1736: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 24603/212084 objects misplaced (11.601%); 7.9 MiB/s, 126 objects/s recovering 2024-04-29T14:35:56.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:56.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:56 smithi194 ceph-mon[44089]: pgmap v1736: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 24603/212084 objects misplaced (11.601%); 7.9 MiB/s, 126 objects/s recovering 2024-04-29T14:35:56.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:56 smithi121 ceph-mon[44533]: pgmap v1736: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s; 24603/212084 objects misplaced (11.601%); 7.9 MiB/s, 126 objects/s recovering 2024-04-29T14:35:56.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 16 108910 108894 119.354 56.75 0.0269723 0.00837606 2024-04-29T14:35:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:57.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:57.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:57.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:57.398 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:35:57.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:35:57.446 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:57.472 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:35:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:57.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:35:57.577 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:35:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:57.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 109755 109739 118.207 52.8125 0.0192649 0.00845724 2024-04-29T14:35:58.036 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:35:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:58.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:35:58.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:58.265 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:58.376 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:35:58.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:58.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:35:58.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:35:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:58.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 110618 110602 117.118 53.9375 0.0344656 0.00853601 2024-04-29T14:35:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:35:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:59.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:35:59.065 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:35:59.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:35:59.160 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:59.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:35:59.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:35:59.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:35:59.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:35:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:35:59.413 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:35:59.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:59 smithi040 ceph-mon[39581]: pgmap v1737: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 24603/216026 objects misplaced (11.389%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:35:59.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:35:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1767815122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:59 smithi121 ceph-mon[44533]: pgmap v1737: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 24603/216026 objects misplaced (11.389%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:35:59.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:35:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1767815122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:35:59.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:35:59.713863 min lat: 0.0010054 max lat: 2.27798 avg lat: 0.00861997 2024-04-29T14:35:59.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:35:59.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 111404 111388 115.985 49.125 0.0132736 0.00861997 2024-04-29T14:35:59.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:59 smithi194 ceph-mon[44089]: pgmap v1737: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 100 MiB/s wr, 1.60k op/s; 24603/216026 objects misplaced (11.389%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:35:59.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:35:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1767815122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:36:00.235 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:36:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:00.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:36:00.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:36:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:36:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:00.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:36:00.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:00 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:36:00.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1767815122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:36:00.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:00 smithi040 ceph-mon[39581]: osdmap e682: 12 total, 12 up, 9 in 2024-04-29T14:36:00.490 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:00 smithi040 ceph-mon[39581]: pgmap v1739: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 54 MiB/s wr, 868 op/s; 24461/218742 objects misplaced (11.183%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:36:00.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:36:00.628 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:00.712 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 112312 112296 115.014 56.75 0.0207655 0.00869204 2024-04-29T14:36:00.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:00 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:36:00.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1767815122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:36:00.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:00 smithi194 ceph-mon[44089]: osdmap e682: 12 total, 12 up, 9 in 2024-04-29T14:36:00.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:00 smithi194 ceph-mon[44089]: pgmap v1739: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 54 MiB/s wr, 868 op/s; 24461/218742 objects misplaced (11.183%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:36:00.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:00 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:36:00.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1767815122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:36:00.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:00 smithi121 ceph-mon[44533]: osdmap e682: 12 total, 12 up, 9 in 2024-04-29T14:36:00.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:00 smithi121 ceph-mon[44533]: pgmap v1739: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 54 MiB/s wr, 868 op/s; 24461/218742 objects misplaced (11.183%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:36:01.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 16 113270 113254 114.125 59.875 0.0267418 0.00875975 2024-04-29T14:36:01.727 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:36:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:01.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:36:01.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:36:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:02.597 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:36:02.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:36:02.692 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:36:02.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 114110 114094 113.147 52.5 0.025158 0.00883583 2024-04-29T14:36:02.754 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:36:02.801 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:36:02.834 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:36:02.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:36:02.935 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:36:02.976 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:36:03.013 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:36:03.046 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:36:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:36:03.088 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:36:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:03.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:36:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:36:03.195 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:36:03.225 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:03.249 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:36:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:03.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:36:03.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:36:03.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:02 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:36:02.906+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e683 definitely_dead 0 2024-04-29T14:36:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:03.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:36:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:03.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:36:03.421 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.5. 2024-04-29T14:36:03.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:36:03.543 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:03.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:36:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:03.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: pgmap v1740: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 992 op/s; 24037/223036 objects misplaced (10.777%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osdmap e683: 12 total, 12 up, 9 in 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/428055138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2506804284' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-29T14:36:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: from='client.27082 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-29T14:36:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: pgmap v1740: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 992 op/s; 24037/223036 objects misplaced (10.777%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osdmap e683: 12 total, 12 up, 9 in 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/428055138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2506804284' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: from='client.27082 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.703 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:03.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 16 114745 114729 112 39.6875 0.00443371 0.00889206 2024-04-29T14:36:03.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: pgmap v1740: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 992 op/s; 24037/223036 objects misplaced (10.777%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:36:03.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osdmap e683: 12 total, 12 up, 9 in 2024-04-29T14:36:03.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/428055138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:36:03.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2506804284' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-29T14:36:03.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: from='client.27082 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-29T14:36:03.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:03.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.155 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:36:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:04.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:36:04.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:04.335 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:04.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:04.447 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:36:04.477 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:04.498 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:36:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:04.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:36:04.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:36:04.630 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:04.688 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:36:04.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:04 smithi194 ceph-mon[44089]: from='client.27082 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-29T14:36:04.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:04 smithi194 ceph-mon[44089]: osdmap e684: 12 total, 12 up, 10 in 2024-04-29T14:36:04.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:04 smithi194 ceph-mon[44089]: pgmap v1743: 17 pgs: 1 unknown, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 24037/210738 objects misplaced (11.406%); 5.9 MiB/s, 94 objects/s recovering 2024-04-29T14:36:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:04.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:36:04.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 114765 114749 110.296 1.25 0.00437306 0.00894345 2024-04-29T14:36:04.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:04 smithi040 ceph-mon[39581]: from='client.27082 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-29T14:36:04.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:04 smithi040 ceph-mon[39581]: osdmap e684: 12 total, 12 up, 10 in 2024-04-29T14:36:04.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:04 smithi040 ceph-mon[39581]: pgmap v1743: 17 pgs: 1 unknown, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 24037/210738 objects misplaced (11.406%); 5.9 MiB/s, 94 objects/s recovering 2024-04-29T14:36:04.756 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-29T14:36:04.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:36:04.822 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:04 smithi121 ceph-mon[44533]: from='client.27082 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-29T14:36:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:04 smithi121 ceph-mon[44533]: osdmap e684: 12 total, 12 up, 10 in 2024-04-29T14:36:04.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:04 smithi121 ceph-mon[44533]: pgmap v1743: 17 pgs: 1 unknown, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 24037/210738 objects misplaced (11.406%); 5.9 MiB/s, 94 objects/s recovering 2024-04-29T14:36:04.948 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:36:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:04.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:36:05.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:36:05.085 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:05.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:36:05.544 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:36:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:05.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:36:05.584 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:36:05.584 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:36:05.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:36:05.584 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:36:05.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.16809.0:265504 6.e 6:768da67e:::benchmark_data_smithi194_134202_object112651:head [delete] snapc 0=[] ondisk+write+known_if_redirected e636)", 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:33:47.023982+0000", 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 138.55730534400001, 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15135131199999999, 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.16809", 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 265504 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.023982+0000", 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.023982+0000", 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.023983+0000", 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2300000000000001e-07 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.023985+0000", 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.379e-06 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.023991+0000", 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4160000000000001e-06 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.023997+0000", 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3299999999999998e-06 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024144+0000", 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014691800000000001 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024150+0000", 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8719999999999996e-06 2024-04-29T14:36:05.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.125086+0000", 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100935782 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.125229+0000", 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000143017 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.125380+0000", 2024-04-29T14:36:05.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015084000000000001 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.129989+0000", 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046085559999999998 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.175237+0000", 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045248838999999999 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.175290+0000", 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2479999999999999e-05 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.175333+0000", 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3359999999999998e-05 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.16809.0:265509 6.e 6:78ecafd4:::benchmark_data_smithi194_134202_object112656:head [delete] snapc 0=[] ondisk+write+known_if_redirected e636)", 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:33:47.024002+0000", 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 138.557284874, 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.151457226, 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.16809", 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:36:05.592 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 265509 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024002+0000", 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024002+0000", 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024003+0000", 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0500000000000004e-07 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024005+0000", 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3499999999999999e-06 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024010+0000", 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9250000000000003e-06 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024015+0000", 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1789999999999999e-06 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024177+0000", 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000161337 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.024184+0000", 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.221e-06 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.127499+0000", 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103315027 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.127601+0000", 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101594 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.127705+0000", 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104912 2024-04-29T14:36:05.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.130674+0000", 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002968273 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.175377+0000", 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.044703251999999999 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.175419+0000", 2024-04-29T14:36:05.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2045000000000001e-05 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:33:47.175460+0000", 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0605999999999997e-05 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3562 7.e 7:75f8ba1a:::benchmark_data_smithi194_147200_object3561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.460204+0000", 2024-04-29T14:36:05.598 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.121082959000006, 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20432719399999999, 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3562 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460204+0000", 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460204+0000", 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460206+0000", 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3230000000000001e-06 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460243+0000", 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6918999999999997e-05 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460254+0000", 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1799e-05 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460263+0000", 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.081e-06 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460629+0000", 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036599499999999998 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460754+0000", 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012410899999999999 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.460885+0000", 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013178800000000001 2024-04-29T14:36:05.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.461453+0000", 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00056713 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.664411+0000", 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20295881800000001 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.664478+0000", 2024-04-29T14:36:05.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6673000000000004e-05 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.664532+0000", 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3559e-05 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3576 7.e 7:7d2297df:::benchmark_data_smithi194_147200_object3575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.462356+0000", 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.118930961000004, 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203306336, 2024-04-29T14:36:05.604 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3576 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462356+0000", 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462356+0000", 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462357+0000", 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0079999999999999e-06 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462421+0000", 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3406999999999998e-05 2024-04-29T14:36:05.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462430+0000", 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8289999999999997e-06 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462436+0000", 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8539999999999999e-06 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462607+0000", 2024-04-29T14:36:05.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017051800000000001 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462725+0000", 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118149 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.462885+0000", 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0001595 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.463424+0000", 2024-04-29T14:36:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053893799999999998 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.665558+0000", 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20213393900000001 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.665617+0000", 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9654999999999997e-05 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.665663+0000", 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5538999999999999e-05 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3597 7.e 7:7bbb6010:::benchmark_data_smithi194_147200_object3596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.465815+0000", 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.115472166000004, 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.200013473, 2024-04-29T14:36:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3597 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.465815+0000", 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.465815+0000", 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.465817+0000", 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.001e-06 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.465939+0000", 2024-04-29T14:36:05.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012200400000000001 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.465950+0000", 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0847e-05 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.465962+0000", 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1498000000000001e-05 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466023+0000", 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1037999999999994e-05 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466166+0000", 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014347000000000001 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466373+0000", 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000206868 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467395+0000", 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001021861 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.665712+0000", 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19831676600000001 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.665786+0000", 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4249e-05 2024-04-29T14:36:05.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.665829+0000", 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2870999999999997e-05 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3599 7.e 7:71acd8c1:::benchmark_data_smithi194_147200_object3598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.466298+0000", 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.114989671000004, 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.199801064, 2024-04-29T14:36:05.616 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3599 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466298+0000", 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466298+0000", 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466299+0000", 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4529999999999999e-06 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466353+0000", 2024-04-29T14:36:05.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3998e-05 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466365+0000", 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1592e-05 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466373+0000", 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7069999999999998e-06 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466739+0000", 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036535499999999999 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.466882+0000", 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000143628 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467053+0000", 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000170225 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467851+0000", 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00079883700000000005 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.665992+0000", 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19814080000000001 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.666049+0000", 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.715e-05 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.666099+0000", 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9319e-05 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3603 7.e 7:7def8bdf:::benchmark_data_smithi194_147200_object3602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.467177+0000", 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.114109921999997, 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19902231400000001, 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3603 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467177+0000", 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467177+0000", 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467179+0000", 2024-04-29T14:36:05.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2470000000000001e-06 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467242+0000", 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3003999999999995e-05 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467254+0000", 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2228999999999999e-05 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467263+0000", 2024-04-29T14:36:05.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3239999999999992e-06 2024-04-29T14:36:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467447+0000", 2024-04-29T14:36:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018393899999999999 2024-04-29T14:36:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467574+0000", 2024-04-29T14:36:05.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126706 2024-04-29T14:36:05.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.467650+0000", 2024-04-29T14:36:05.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6459999999999996e-05 2024-04-29T14:36:05.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.468001+0000", 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035077300000000002 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.666144+0000", 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.198142662 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.666176+0000", 2024-04-29T14:36:05.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2097000000000001e-05 2024-04-29T14:36:05.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.666200+0000", 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3873e-05 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3611 7.e 7:7a0245d8:::benchmark_data_smithi194_147200_object3610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.469485+0000", 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.111801911000001, 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19781117500000001, 2024-04-29T14:36:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3611 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469485+0000", 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469485+0000", 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469487+0000", 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8989999999999999e-06 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469541+0000", 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3989000000000001e-05 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469552+0000", 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0538e-05 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469559+0000", 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3390000000000004e-06 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469601+0000", 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1789000000000003e-05 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469688+0000", 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7224000000000004e-05 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.469839+0000", 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015119499999999999 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.470374+0000", 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053433099999999998 2024-04-29T14:36:05.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667199+0000", 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.196825053 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667251+0000", 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2232999999999999e-05 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667297+0000", 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5584999999999998e-05 2024-04-29T14:36:05.636 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3635 7.e 7:797ebfa9:::benchmark_data_smithi194_147200_object3634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.475072+0000", 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.106215736999999, 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19232860299999999, 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.637 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3635 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475072+0000", 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475072+0000", 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475073+0000", 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.424e-06 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475143+0000", 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9842e-05 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475152+0000", 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2450000000000004e-06 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475160+0000", 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6680000000000001e-06 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475207+0000", 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7518000000000001e-05 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475304+0000", 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7152999999999996e-05 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475386+0000", 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2039000000000005e-05 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.475789+0000", 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040299200000000002 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667338+0000", 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.191548835 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667367+0000", 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8966e-05 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667400+0000", 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2920999999999999e-05 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3647 7.e 7:7aec96b1:::benchmark_data_smithi194_147200_object3646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.478206+0000", 2024-04-29T14:36:05.642 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.103080869999999, 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.189280743, 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3647 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478206+0000", 2024-04-29T14:36:05.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478206+0000", 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478208+0000", 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.888e-06 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478268+0000", 2024-04-29T14:36:05.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9840000000000003e-05 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478281+0000", 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3220999999999999e-05 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478289+0000", 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6120000000000001e-06 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478342+0000", 2024-04-29T14:36:05.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3183999999999998e-05 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478456+0000", 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000113589 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478547+0000", 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1580000000000001e-05 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.479582+0000", 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001034732 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667427+0000", 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18784461 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667456+0000", 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9303e-05 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667487+0000", 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1183999999999998e-05 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:3650 7.e 7:7ad8fdee:::benchmark_data_smithi194_147200_object3649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e660)", 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:00.478353+0000", 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 65.102934664000003, 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18933651500000001, 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.648 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 3650 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478353+0000", 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478353+0000", 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478354+0000", 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.59e-06 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478432+0000", 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7642e-05 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478442+0000", 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8360000000000004e-06 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478448+0000", 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9390000000000002e-06 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.478866+0000", 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00041880999999999998 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.479005+0000", 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013830400000000001 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.479234+0000", 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022918799999999999 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.479728+0000", 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00049382500000000004 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667529+0000", 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18780086300000001 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667620+0000", 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0960999999999999e-05 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:00.667689+0000", 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9556999999999998e-05 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33772 7.8 7:1df5f09a:::benchmark_data_smithi194_147200_object33771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.271951+0000", 2024-04-29T14:36:05.653 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.309336555000002, 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.207541112, 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33772 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.271951+0000", 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.271951+0000", 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.271952+0000", 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.271e-06 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272034+0000", 2024-04-29T14:36:05.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1984000000000007e-05 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272045+0000", 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1209000000000001e-05 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272053+0000", 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2949999999999996e-06 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272097+0000", 2024-04-29T14:36:05.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4566e-05 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272193+0000", 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5659e-05 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272275+0000", 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1841000000000005e-05 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274639+0000", 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002364891 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.479407+0000", 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20476723999999999 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.479446+0000", 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9582000000000002e-05 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.479492+0000", 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5574000000000001e-05 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33775 7.8 7:1e591255:::benchmark_data_smithi194_147200_object33774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.272069+0000", 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.309218708000003, 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.207517851, 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.659 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33775 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272069+0000", 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272069+0000", 2024-04-29T14:36:05.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272069+0000", 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1999999999999998e-07 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272148+0000", 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8627000000000005e-05 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272162+0000", 2024-04-29T14:36:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.346e-05 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.272168+0000", 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.319e-06 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.273393+0000", 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012254169999999999 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.273524+0000", 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000130252 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.273675+0000", 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015186 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.275857+0000", 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021813230000000002 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.479527+0000", 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203669988 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.479561+0000", 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4079999999999999e-05 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.479586+0000", 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5704999999999999e-05 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33782 7.8 7:1b1e5924:::benchmark_data_smithi194_147200_object33781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.274354+0000", 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.306933770000001, 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206162236, 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33782 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274354+0000", 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274354+0000", 2024-04-29T14:36:05.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274354+0000", 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4600000000000003e-07 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274400+0000", 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5677999999999999e-05 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274425+0000", 2024-04-29T14:36:05.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4674000000000001e-05 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274432+0000", 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.951e-06 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274666+0000", 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000234546 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274766+0000", 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9742999999999997e-05 2024-04-29T14:36:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.274892+0000", 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125964 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.276281+0000", 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0013892150000000001 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480435+0000", 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20415401799999999 2024-04-29T14:36:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480490+0000", 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4806999999999999e-05 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480516+0000", 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5694000000000002e-05 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.670 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33823 7.8 7:1baf4091:::benchmark_data_smithi194_147200_object33822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.290797+0000", 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.290490001000002, 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18993591600000001, 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33823 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.290797+0000", 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.290797+0000", 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.290798+0000", 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7999999999999995e-07 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.290881+0000", 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2874000000000006e-05 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.290890+0000", 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8310000000000007e-06 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.290896+0000", 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9429999999999997e-06 2024-04-29T14:36:05.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.290937+0000", 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1431999999999999e-05 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.291024+0000", 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6722000000000001e-05 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.291123+0000", 2024-04-29T14:36:05.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.925e-05 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.292260+0000", 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011364610000000001 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480637+0000", 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18837722900000001 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480693+0000", 2024-04-29T14:36:05.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6184000000000003e-05 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480733+0000", 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0309999999999999e-05 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33833 7.8 7:1ff33937:::benchmark_data_smithi194_147200_object33832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.293432+0000", 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.287855387999997, 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18740602100000001, 2024-04-29T14:36:05.676 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33833 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293432+0000", 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293432+0000", 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293433+0000", 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0690000000000001e-06 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293537+0000", 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010395000000000001 2024-04-29T14:36:05.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293546+0000", 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293553+0000", 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3209999999999999e-06 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293613+0000", 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0069999999999999e-05 2024-04-29T14:36:05.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293737+0000", 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000123569 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.293889+0000", 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015220500000000001 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.295009+0000", 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011200260000000001 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480785+0000", 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18577579699999999 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480815+0000", 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0329e-05 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480838+0000", 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2535e-05 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.681 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33865 7.8 7:11971510:::benchmark_data_smithi194_147200_object33864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.304249+0000", 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.277038611999998, 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.176715333, 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33865 2024-04-29T14:36:05.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304249+0000", 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304249+0000", 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304250+0000", 2024-04-29T14:36:05.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.16e-07 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304322+0000", 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2247000000000001e-05 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304328+0000", 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5019999999999998e-06 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304333+0000", 2024-04-29T14:36:05.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6489999999999999e-06 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304379+0000", 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6108e-05 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304494+0000", 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011482800000000001 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.304654+0000", 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016025499999999999 2024-04-29T14:36:05.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.305917+0000", 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001262921 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480877+0000", 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.174959794 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480929+0000", 2024-04-29T14:36:05.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2154999999999998e-05 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.480964+0000", 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4958e-05 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33880 7.8 7:1d3b4410:::benchmark_data_smithi194_147200_object33879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.308732+0000", 2024-04-29T14:36:05.687 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.272555457000003, 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.17292310799999999, 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33880 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.308732+0000", 2024-04-29T14:36:05.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.308732+0000", 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.308734+0000", 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.658e-06 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.308835+0000", 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010135499999999999 2024-04-29T14:36:05.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.308845+0000", 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0455000000000001e-05 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.308851+0000", 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7389999999999996e-06 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.308903+0000", 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.219e-05 2024-04-29T14:36:05.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.309037+0000", 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013362700000000001 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.309163+0000", 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126131 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.310250+0000", 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010870459999999999 2024-04-29T14:36:05.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.481547+0000", 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.17129702199999999 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.481615+0000", 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7577999999999996e-05 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.481655+0000", 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0306999999999999e-05 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33889 7.8 7:19dea543:::benchmark_data_smithi194_147200_object33888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.311736+0000", 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.269550895000002, 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.17002842700000001, 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.693 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33889 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311736+0000", 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311736+0000", 2024-04-29T14:36:05.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311738+0000", 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1999999999999999e-06 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311791+0000", 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3732e-05 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311800+0000", 2024-04-29T14:36:05.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9469999999999992e-06 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311806+0000", 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2360000000000004e-06 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311852+0000", 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6199000000000003e-05 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.311968+0000", 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116236 2024-04-29T14:36:05.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.312123+0000", 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000155301 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.313144+0000", 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010206080000000001 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.481689+0000", 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.1685451 2024-04-29T14:36:05.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.481720+0000", 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1368000000000002e-05 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.481765+0000", 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4499999999999997e-05 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:33896 7.8 7:153cdc6a:::benchmark_data_smithi194_147200_object33895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:09.316622+0000", 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.264664998999997, 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.166032819, 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 33896 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316622+0000", 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316622+0000", 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316623+0000", 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1739999999999999e-06 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316672+0000", 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8347999999999999e-05 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316681+0000", 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9479999999999997e-06 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316686+0000", 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4469999999999997e-06 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:36:05.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316725+0000", 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9243999999999999e-05 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316816+0000", 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0915999999999995e-05 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 7", 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.316905+0000", 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8632000000000001e-05 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:36:05.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.317759+0000", 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00085374599999999995 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.482565+0000", 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.164806604 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.482618+0000", 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2703e-05 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:36:05.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:09.482655+0000", 2024-04-29T14:36:05.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7057000000000001e-05 2024-04-29T14:36:05.704 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.704 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.704 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.704 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:36:05.704 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:36:05.704 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:36:05.704 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:36:05.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:05 smithi040 ceph-mon[39581]: osdmap e685: 12 total, 12 up, 10 in 2024-04-29T14:36:05.706 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:36:05.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 16 114804 114788 108.662 2.4375 0.0134143 0.00894297 2024-04-29T14:36:05.729 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:05 smithi194 ceph-mon[44089]: osdmap e685: 12 total, 12 up, 10 in 2024-04-29T14:36:05.731 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:05.757 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:36:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:05.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:36:05.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:36:05.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:05 smithi121 ceph-mon[44533]: osdmap e685: 12 total, 12 up, 10 in 2024-04-29T14:36:05.889 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:36:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:05.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:36:06.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:36:06.042 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:06.068 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:36:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:06.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:36:06.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:36:06.198 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:06.219 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:36:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:06.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:36:06.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:36:06.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:06 smithi194 ceph-mon[44089]: osdmap e686: 12 total, 12 up, 10 in 2024-04-29T14:36:06.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:06.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:06.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:06.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:06.455 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:06 smithi194 ceph-mon[44089]: pgmap v1746: 17 pgs: 1 remapped+peering, 1 peering, 1 active+recovering+remapped, 1 unknown, 2 active+recovering, 11 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 17353/214174 objects misplaced (8.102%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:36:06.457 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:06.476 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:36:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:06.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:36:06.580 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:36:06.606 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:06.626 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:36:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:06.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:36:06.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 16 114908 114892 107.138 6.5 0.00509886 0.0090666 2024-04-29T14:36:06.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:36:06.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:06 smithi040 ceph-mon[39581]: osdmap e686: 12 total, 12 up, 10 in 2024-04-29T14:36:06.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:06.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:06.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:06.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:06.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:06 smithi040 ceph-mon[39581]: pgmap v1746: 17 pgs: 1 remapped+peering, 1 peering, 1 active+recovering+remapped, 1 unknown, 2 active+recovering, 11 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 17353/214174 objects misplaced (8.102%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:36:06.767 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:06.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:06 smithi121 ceph-mon[44533]: osdmap e686: 12 total, 12 up, 10 in 2024-04-29T14:36:06.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:06.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:06.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:06.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:06.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:06 smithi121 ceph-mon[44533]: pgmap v1746: 17 pgs: 1 remapped+peering, 1 peering, 1 active+recovering+remapped, 1 unknown, 2 active+recovering, 11 active+clean; 6.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 17353/214174 objects misplaced (8.102%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:36:07.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:36:07.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 114908 114892 105.563 0 - 0.0090666 2024-04-29T14:36:07.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:07 smithi194 ceph-mon[44089]: osdmap e687: 12 total, 12 up, 10 in 2024-04-29T14:36:07.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:07 smithi040 ceph-mon[39581]: osdmap e687: 12 total, 12 up, 10 in 2024-04-29T14:36:07.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:07 smithi121 ceph-mon[44533]: osdmap e687: 12 total, 12 up, 10 in 2024-04-29T14:36:08.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 114908 114892 104.033 0 - 0.0090666 2024-04-29T14:36:08.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:08 smithi194 ceph-mon[44089]: pgmap v1748: 17 pgs: 1 remapped+peering, 1 peering, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 33 MiB/s wr, 530 op/s; 24379/229670 objects misplaced (10.615%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:36:08.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:08 smithi040 ceph-mon[39581]: pgmap v1748: 17 pgs: 1 remapped+peering, 1 peering, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 33 MiB/s wr, 530 op/s; 24379/229670 objects misplaced (10.615%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:36:08.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:08 smithi121 ceph-mon[44533]: pgmap v1748: 17 pgs: 1 remapped+peering, 1 peering, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 33 MiB/s wr, 530 op/s; 24379/229670 objects misplaced (10.615%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:36:09.713 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 114908 114892 102.547 0 - 0.0090666 2024-04-29T14:36:09.757 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5] out_osds: [3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:36:09.757 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:36:09.757 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-29T14:36:09.757 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:36:09.757 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:36:09.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-29T14:36:10.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 16 114908 114892 101.103 0 - 0.0090666 2024-04-29T14:36:11.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:11 smithi194 ceph-mon[44089]: pgmap v1749: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s; 31207/229748 objects misplaced (13.583%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:36:11.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:11 smithi040 ceph-mon[39581]: pgmap v1749: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s; 31207/229748 objects misplaced (13.583%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:36:11.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:11 smithi121 ceph-mon[44533]: pgmap v1749: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s; 31207/229748 objects misplaced (13.583%); 12 MiB/s, 187 objects/s recovering 2024-04-29T14:36:11.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 114908 114892 99.6995 0 - 0.0090666 2024-04-29T14:36:12.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 114908 114892 98.334 0 - 0.0090666 2024-04-29T14:36:13.420 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:13 smithi040 ceph-mon[39581]: pgmap v1750: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 30271/229798 objects misplaced (13.173%); 16 MiB/s, 251 objects/s recovering 2024-04-29T14:36:13.430 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:36:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:13.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:36:13.462 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:36:13.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:13 smithi194 ceph-mon[44089]: pgmap v1750: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 30271/229798 objects misplaced (13.173%); 16 MiB/s, 251 objects/s recovering 2024-04-29T14:36:13.535 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:36:13.560 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:13.583 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:36:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:13.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:36:13.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:13 smithi121 ceph-mon[44533]: pgmap v1750: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 30271/229798 objects misplaced (13.173%); 16 MiB/s, 251 objects/s recovering 2024-04-29T14:36:13.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:13.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 114908 114892 97.0054 0 - 0.0090666 2024-04-29T14:36:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:13.781 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:36:14.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 16 114908 114892 95.7123 0 - 0.0090666 2024-04-29T14:36:15.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:15 smithi194 ceph-mon[44089]: pgmap v1751: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 61 MiB/s wr, 970 op/s; 29970/229798 objects misplaced (13.042%); 16 MiB/s, 257 objects/s recovering 2024-04-29T14:36:15.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:15 smithi040 ceph-mon[39581]: pgmap v1751: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 61 MiB/s wr, 970 op/s; 29970/229798 objects misplaced (13.042%); 16 MiB/s, 257 objects/s recovering 2024-04-29T14:36:15.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:15 smithi121 ceph-mon[44533]: pgmap v1751: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 61 MiB/s wr, 970 op/s; 29970/229798 objects misplaced (13.042%); 16 MiB/s, 257 objects/s recovering 2024-04-29T14:36:15.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 114908 114892 94.4532 0 - 0.0090666 2024-04-29T14:36:16.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 114908 114892 93.2268 0 - 0.0090666 2024-04-29T14:36:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:16.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:17.085 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:36:17.115 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:17.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:17 smithi040 ceph-mon[39581]: pgmap v1752: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 49 MiB/s wr, 781 op/s; 28676/229798 objects misplaced (12.479%); 18 MiB/s, 286 objects/s recovering 2024-04-29T14:36:17.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:17 smithi194 ceph-mon[44089]: pgmap v1752: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 49 MiB/s wr, 781 op/s; 28676/229798 objects misplaced (12.479%); 18 MiB/s, 286 objects/s recovering 2024-04-29T14:36:17.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:17 smithi121 ceph-mon[44533]: pgmap v1752: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 49 MiB/s wr, 781 op/s; 28676/229798 objects misplaced (12.479%); 18 MiB/s, 286 objects/s recovering 2024-04-29T14:36:17.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 114908 114892 92.0319 0 - 0.0090666 2024-04-29T14:36:18.366 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:36:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:18.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:36:18.470 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:36:18.499 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:18.647 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:36:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:18.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:36:18.714 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 114908 114892 90.8672 0 - 0.0090666 2024-04-29T14:36:18.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:36:18.788 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:18.821 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:36:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:18.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:36:18.834 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:36:18.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:18.970 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:19.006 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:19.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:19.113 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:36:19.141 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:19.161 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:36:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:19.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:36:19.254 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:36:19.254 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:36:19.255 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:36:19.255 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:36:19.266 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:36:19.301 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:36:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:19.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:36:19.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:36:19.490 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:19.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:36:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:19.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:36:19.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:19 smithi194 ceph-mon[44089]: pgmap v1753: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 40 MiB/s wr, 633 op/s; 28354/229798 objects misplaced (12.339%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:36:19.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:36:19.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:19 smithi040 ceph-mon[39581]: pgmap v1753: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 40 MiB/s wr, 633 op/s; 28354/229798 objects misplaced (12.339%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:36:19.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:19 smithi121 ceph-mon[44533]: pgmap v1753: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 40 MiB/s wr, 633 op/s; 28354/229798 objects misplaced (12.339%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:36:19.652 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:19.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:36:19.715940 min lat: 0.0010054 max lat: 2.36606 avg lat: 0.0090666 2024-04-29T14:36:19.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:36:19.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 114908 114892 89.7316 0 - 0.0090666 2024-04-29T14:36:20.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:20.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:20.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:20 smithi040 ceph-mon[39581]: osdmap e688: 12 total, 12 up, 10 in 2024-04-29T14:36:20.572 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:20 smithi040 ceph-mon[39581]: pgmap v1755: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 160 KiB/s wr, 2 op/s; 28046/229798 objects misplaced (12.205%); 20 MiB/s, 316 objects/s recovering 2024-04-29T14:36:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:36:20.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 114908 114892 88.624 0 - 0.0090666 2024-04-29T14:36:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:20 smithi194 ceph-mon[44089]: osdmap e688: 12 total, 12 up, 10 in 2024-04-29T14:36:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:20 smithi194 ceph-mon[44089]: pgmap v1755: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 160 KiB/s wr, 2 op/s; 28046/229798 objects misplaced (12.205%); 20 MiB/s, 316 objects/s recovering 2024-04-29T14:36:20.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:20.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:20.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:20 smithi121 ceph-mon[44533]: osdmap e688: 12 total, 12 up, 10 in 2024-04-29T14:36:20.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:20 smithi121 ceph-mon[44533]: pgmap v1755: 17 pgs: 1 remapped+peering, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 160 KiB/s wr, 2 op/s; 28046/229798 objects misplaced (12.205%); 20 MiB/s, 316 objects/s recovering 2024-04-29T14:36:21.347 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:36:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:36:21.517 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:36:21.517 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "" 2024-04-29T14:36:21.517 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:36:21.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3709585548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:36:21.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:21 smithi194 ceph-mon[44089]: from='client.27094 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:36:21.626 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:36:21.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 114908 114892 87.5434 0 - 0.0090666 2024-04-29T14:36:21.845 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:36:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3709585548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:36:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:21 smithi040 ceph-mon[39581]: from='client.27094 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:36:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:21.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:36:21.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3709585548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:36:21.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:21 smithi121 ceph-mon[44533]: from='client.27094 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:36:21.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:36:21.984 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:22.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:36:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:22.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:36:22.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:36:22.140 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:22.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:36:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:22.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:36:22.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:36:22.297 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:22.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:36:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:22.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:36:22.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:22.463 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:22.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:22 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:36:22.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:22 smithi121 ceph-mon[44533]: from='client.27094 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:36:22.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:22 smithi121 ceph-mon[44533]: osdmap e689: 12 total, 12 up, 10 in 2024-04-29T14:36:22.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:22 smithi121 ceph-mon[44533]: pgmap v1757: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 27020/229798 objects misplaced (11.758%); 23 MiB/s, 368 objects/s recovering 2024-04-29T14:36:22.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:22 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:36:22.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:22 smithi040 ceph-mon[39581]: from='client.27094 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:36:22.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:22 smithi040 ceph-mon[39581]: osdmap e689: 12 total, 12 up, 10 in 2024-04-29T14:36:22.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:22 smithi040 ceph-mon[39581]: pgmap v1757: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 27020/229798 objects misplaced (11.758%); 23 MiB/s, 368 objects/s recovering 2024-04-29T14:36:22.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 16 114908 114892 86.4889 0 - 0.0090666 2024-04-29T14:36:22.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:22 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:36:22.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:22 smithi194 ceph-mon[44089]: from='client.27094 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:36:22.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:22 smithi194 ceph-mon[44089]: osdmap e689: 12 total, 12 up, 10 in 2024-04-29T14:36:22.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:22 smithi194 ceph-mon[44089]: pgmap v1757: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 27020/229798 objects misplaced (11.758%); 23 MiB/s, 368 objects/s recovering 2024-04-29T14:36:23.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:23 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 13018/229798 objects degraded (5.665%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:36:23.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 114908 114892 85.4594 0 - 0.0090666 2024-04-29T14:36:23.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:23 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 13018/229798 objects degraded (5.665%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:36:23.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:23 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 13018/229798 objects degraded (5.665%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:36:24.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:24 smithi194 ceph-mon[44089]: pgmap v1758: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 26699/229798 objects misplaced (11.618%); 15 MiB/s, 247 objects/s recovering 2024-04-29T14:36:24.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 16 114908 114892 84.4542 0 - 0.0090666 2024-04-29T14:36:24.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:24 smithi040 ceph-mon[39581]: pgmap v1758: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 26699/229798 objects misplaced (11.618%); 15 MiB/s, 247 objects/s recovering 2024-04-29T14:36:24.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:24 smithi121 ceph-mon[44533]: pgmap v1758: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 26699/229798 objects misplaced (11.618%); 15 MiB/s, 247 objects/s recovering 2024-04-29T14:36:25.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 114908 114892 83.4724 0 - 0.0090666 2024-04-29T14:36:26.562 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:36:26.562 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:36:26.562 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:36:26.562 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:36:26.562 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:36:26.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 114908 114892 82.5131 0 - 0.0090666 2024-04-29T14:36:27.327 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:27.327 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:26 smithi194 ceph-mon[44089]: osdmap e690: 12 total, 12 up, 10 in 2024-04-29T14:36:27.327 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:26 smithi194 ceph-mon[44089]: pgmap v1760: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 25305/229798 objects misplaced (11.012%); 26 MiB/s, 408 objects/s recovering 2024-04-29T14:36:27.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:27.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:26 smithi040 ceph-mon[39581]: osdmap e690: 12 total, 12 up, 10 in 2024-04-29T14:36:27.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:26 smithi040 ceph-mon[39581]: pgmap v1760: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 25305/229798 objects misplaced (11.012%); 26 MiB/s, 408 objects/s recovering 2024-04-29T14:36:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:26 smithi121 ceph-mon[44533]: osdmap e690: 12 total, 12 up, 10 in 2024-04-29T14:36:27.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:26 smithi121 ceph-mon[44533]: pgmap v1760: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 25305/229798 objects misplaced (11.012%); 26 MiB/s, 408 objects/s recovering 2024-04-29T14:36:27.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 114908 114892 81.5757 0 - 0.0090666 2024-04-29T14:36:28.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5] out_osds: [3, 8] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:36:28.089 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:36:28.090 INFO:tasks.thrashosds.thrasher:Adding osd 8 2024-04-29T14:36:28.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 8 2024-04-29T14:36:28.209 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:36:28.496 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:36:28.715 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 114908 114892 80.6593 0 - 0.0090666 2024-04-29T14:36:29.354 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:29.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:29.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:36:29.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:29 smithi121 ceph-mon[44533]: pgmap v1761: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 24966/229798 objects misplaced (10.864%); 24 MiB/s, 384 objects/s recovering 2024-04-29T14:36:29.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:29 smithi040 ceph-mon[39581]: pgmap v1761: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 24966/229798 objects misplaced (10.864%); 24 MiB/s, 384 objects/s recovering 2024-04-29T14:36:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:29.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 90 16 114908 114892 79.7632 0 - 0.0090666 2024-04-29T14:36:29.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:29 smithi194 ceph-mon[44089]: pgmap v1761: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 24966/229798 objects misplaced (10.864%); 24 MiB/s, 384 objects/s recovering 2024-04-29T14:36:30.282 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:36:30.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:36:30.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/951093997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:36:30.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:36:30.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/951093997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:36:30.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:30.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:36:30.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 91 16 114908 114892 78.8868 0 - 0.0090666 2024-04-29T14:36:30.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:30.776 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:30.800 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:30.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:30.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/951093997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:36:30.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:30.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:36:30.933 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:30.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:36:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:30.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:36:31.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:36:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:31.121 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:36:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:31.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:36:31.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:31.257 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:31.284 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:31.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:31.389 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:36:31.421 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:31.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:31 smithi121 ceph-mon[44533]: pgmap v1762: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 24627/229798 objects misplaced (10.717%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:36:31.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 92 16 114908 114892 78.0295 0 - 0.0090666 2024-04-29T14:36:31.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:31 smithi194 ceph-mon[44089]: pgmap v1762: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 24627/229798 objects misplaced (10.717%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:36:31.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:31 smithi040 ceph-mon[39581]: pgmap v1762: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 24627/229798 objects misplaced (10.717%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:36:31.871 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:36:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:31.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:36:31.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:36:32.003 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:36:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:32.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:36:32.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:36:32.164 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:32.189 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:36:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:32.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:36:32.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:36:32.320 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:32.327 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:36:32.327 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:36:32.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:36:32.327 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [] 2024-04-29T14:36:32.327 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:36:32.341 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:36:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:32.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:36:32.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:36:32.475 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:36:32.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:32.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:36:32.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:32.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:32.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:32 smithi040 ceph-mon[39581]: pgmap v1763: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 23476/229798 objects misplaced (10.216%); 22 MiB/s, 354 objects/s recovering 2024-04-29T14:36:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:32.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:32 smithi121 ceph-mon[44533]: pgmap v1763: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 23476/229798 objects misplaced (10.216%); 22 MiB/s, 354 objects/s recovering 2024-04-29T14:36:32.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:36:32.682 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:32.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:36:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:32.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:36:32.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 93 16 114908 114892 77.1907 0 - 0.0090666 2024-04-29T14:36:32.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:32.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:32.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:32 smithi194 ceph-mon[44089]: pgmap v1763: 17 pgs: 1 activating+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 23476/229798 objects misplaced (10.216%); 22 MiB/s, 354 objects/s recovering 2024-04-29T14:36:32.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:36:32.840 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:36:32.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:32.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:36:32.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:36:32.996 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:33.019 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:36:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:33.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:36:33.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:36:33.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:36:33.171 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:33.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:36:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:33.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:36:33.236 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:36:33.260 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:36:33.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:36:33.303 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:36:33.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:36:33.334 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:36:33.338 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:33.366 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:36:33.371 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:36:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:33.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:36:33.393 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:36:33.418 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:36:33.425 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:36:33.444 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:36:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:36:33.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:36:33.504 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:36:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:36:33.507 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:36:33.507 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:36:33.507 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:36:33.507 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:36:33.512 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:33.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:36:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:33.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:36:33.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:33 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:33.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:33 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:33.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:33 smithi040 ceph-mon[39581]: osdmap e691: 12 total, 12 up, 10 in 2024-04-29T14:36:33.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:36:33.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 94 16 114908 114892 76.3696 0 - 0.0090666 2024-04-29T14:36:33.738 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:33 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:33.738 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:33 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:33.738 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:33 smithi121 ceph-mon[44533]: osdmap e691: 12 total, 12 up, 10 in 2024-04-29T14:36:33.740 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:33.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:33 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:33.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:33 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:33.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:33 smithi194 ceph-mon[44089]: osdmap e691: 12 total, 12 up, 10 in 2024-04-29T14:36:34.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 95 16 114909 114893 75.5665 0.00223214 30.5253 0.00933221 2024-04-29T14:36:34.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:34.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:34.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osdmap e692: 12 total, 12 up, 10 in 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:34 smithi194 ceph-mon[44089]: pgmap v1766: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 23127/229798 objects misplaced (10.064%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:36:34.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osdmap e692: 12 total, 12 up, 10 in 2024-04-29T14:36:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:34 smithi121 ceph-mon[44533]: pgmap v1766: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 23127/229798 objects misplaced (10.064%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.902 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osdmap e692: 12 total, 12 up, 10 in 2024-04-29T14:36:34.903 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.903 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.903 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.903 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.903 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:36:34.903 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:34 smithi040 ceph-mon[39581]: pgmap v1766: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 23127/229798 objects misplaced (10.064%); 17 MiB/s, 272 objects/s recovering 2024-04-29T14:36:35.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:36:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:35.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:36:35.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:35.716 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 96 12 114909 114897 74.7821 0.25 29.3309 0.0104032 2024-04-29T14:36:36.027 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:36.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:36.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:36:36.686 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:36.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 96.2111 2024-04-29T14:36:36.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 114909 2024-04-29T14:36:36.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:36:36.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:36:36.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 74.6464 2024-04-29T14:36:36.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 71.4078 2024-04-29T14:36:36.717 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 279.25 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 0 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1194 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 1142.53 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 4468 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 0 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.0133806 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.365777 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 32.5119 2024-04-29T14:36:36.718 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.0010054 2024-04-29T14:36:36.730 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:36:37.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:37 smithi121 ceph-mon[44533]: pgmap v1767: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 21955/229798 objects misplaced (9.554%); 24 MiB/s, 376 objects/s recovering 2024-04-29T14:36:37.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:37 smithi194 ceph-mon[44089]: pgmap v1767: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 21955/229798 objects misplaced (9.554%); 24 MiB/s, 376 objects/s recovering 2024-04-29T14:36:37.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:37 smithi040 ceph-mon[39581]: pgmap v1767: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 13018/229798 objects degraded (5.665%); 21955/229798 objects misplaced (9.554%); 24 MiB/s, 376 objects/s recovering 2024-04-29T14:36:38.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:36:38.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:36:38.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:36:39.508 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:36:39.508 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:36:39.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:39 smithi121 ceph-mon[44533]: pgmap v1768: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 8.0 KiB/s wr, 10 op/s; 13018/229638 objects degraded (5.669%); 21223/229638 objects misplaced (9.242%); 27 MiB/s, 425 objects/s recovering 2024-04-29T14:36:39.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:39 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 13018/229638 objects degraded (5.669%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:36:39.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:39 smithi194 ceph-mon[44089]: pgmap v1768: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 8.0 KiB/s wr, 10 op/s; 13018/229638 objects degraded (5.669%); 21223/229638 objects misplaced (9.242%); 27 MiB/s, 425 objects/s recovering 2024-04-29T14:36:39.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:39 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 13018/229638 objects degraded (5.669%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:36:39.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:39 smithi040 ceph-mon[39581]: pgmap v1768: 17 pgs: 1 active+degraded, 2 active+recovering+remapped, 3 active+recovering, 11 active+clean; 7.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 8.0 KiB/s wr, 10 op/s; 13018/229638 objects degraded (5.669%); 21223/229638 objects misplaced (9.242%); 27 MiB/s, 425 objects/s recovering 2024-04-29T14:36:39.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:39 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 13018/229638 objects degraded (5.669%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:36:40.373 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:36:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:40.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:36:40.433 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:36:40.478 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:36:40.487 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:36:40.507 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:40.529 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:36:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:40.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:36:40.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:40 smithi040 ceph-mon[39581]: pgmap v1769: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 96 KiB/s wr, 24 op/s; 33856/229450 objects misplaced (14.755%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:36:40.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:40 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13018/229638 objects degraded (5.669%), 1 pg degraded) 2024-04-29T14:36:40.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:36:40.668 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:40.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:40 smithi121 ceph-mon[44533]: pgmap v1769: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 96 KiB/s wr, 24 op/s; 33856/229450 objects misplaced (14.755%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:36:40.744 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:40 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13018/229638 objects degraded (5.669%), 1 pg degraded) 2024-04-29T14:36:40.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:40 smithi194 ceph-mon[44089]: pgmap v1769: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 96 KiB/s wr, 24 op/s; 33856/229450 objects misplaced (14.755%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:36:40.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:40 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13018/229638 objects degraded (5.669%), 1 pg degraded) 2024-04-29T14:36:41.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:36:41.615 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:36:42.366 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:36:42.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:36:42.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:36:42.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:42 smithi040 ceph-mon[39581]: pgmap v1770: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 105 B/s rd, 79 KiB/s wr, 46 op/s; 32659/228944 objects misplaced (14.265%); 22 MiB/s, 357 objects/s recovering 2024-04-29T14:36:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:36:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:36:42.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:42 smithi194 ceph-mon[44089]: pgmap v1770: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 105 B/s rd, 79 KiB/s wr, 46 op/s; 32659/228944 objects misplaced (14.265%); 22 MiB/s, 357 objects/s recovering 2024-04-29T14:36:42.827 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:36:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:42.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:36:42.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:36:42.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:36:42.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:42 smithi121 ceph-mon[44533]: pgmap v1770: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 105 B/s rd, 79 KiB/s wr, 46 op/s; 32659/228944 objects misplaced (14.265%); 22 MiB/s, 357 objects/s recovering 2024-04-29T14:36:42.934 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:36:42.973 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:45.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:45 smithi040 ceph-mon[39581]: pgmap v1771: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 95 B/s rd, 72 KiB/s wr, 49 op/s; 32281/228786 objects misplaced (14.110%); 22 MiB/s, 359 objects/s recovering 2024-04-29T14:36:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:45 smithi194 ceph-mon[44089]: pgmap v1771: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 95 B/s rd, 72 KiB/s wr, 49 op/s; 32281/228786 objects misplaced (14.110%); 22 MiB/s, 359 objects/s recovering 2024-04-29T14:36:45.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:45 smithi121 ceph-mon[44533]: pgmap v1771: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 95 B/s rd, 72 KiB/s wr, 49 op/s; 32281/228786 objects misplaced (14.110%); 22 MiB/s, 359 objects/s recovering 2024-04-29T14:36:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:36:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:46.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:36:46.573 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:36:46.601 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:46.638 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:36:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:46.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:36:46.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:36:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:47.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:47 smithi194 ceph-mon[44089]: pgmap v1772: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 64 KiB/s wr, 54 op/s; 30963/228550 objects misplaced (13.548%); 27 MiB/s, 430 objects/s recovering 2024-04-29T14:36:47.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:47 smithi040 ceph-mon[39581]: pgmap v1772: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 64 KiB/s wr, 54 op/s; 30963/228550 objects misplaced (13.548%); 27 MiB/s, 430 objects/s recovering 2024-04-29T14:36:47.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:47 smithi121 ceph-mon[44533]: pgmap v1772: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 64 KiB/s wr, 54 op/s; 30963/228550 objects misplaced (13.548%); 27 MiB/s, 430 objects/s recovering 2024-04-29T14:36:49.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:49 smithi121 ceph-mon[44533]: pgmap v1773: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 64 KiB/s wr, 63 op/s; 30142/228326 objects misplaced (13.201%); 25 MiB/s, 401 objects/s recovering 2024-04-29T14:36:49.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:49 smithi194 ceph-mon[44089]: pgmap v1773: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 64 KiB/s wr, 63 op/s; 30142/228326 objects misplaced (13.201%); 25 MiB/s, 401 objects/s recovering 2024-04-29T14:36:49.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:49 smithi040 ceph-mon[39581]: pgmap v1773: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 64 KiB/s wr, 63 op/s; 30142/228326 objects misplaced (13.201%); 25 MiB/s, 401 objects/s recovering 2024-04-29T14:36:50.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:50.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:50.602 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:36:50.602 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:36:50.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:50.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:50.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:50.630 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:51.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:51 smithi194 ceph-mon[44089]: pgmap v1774: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 59 KiB/s wr, 62 op/s; 29659/228176 objects misplaced (12.998%); 24 MiB/s, 380 objects/s recovering 2024-04-29T14:36:51.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:51 smithi194 ceph-mon[44089]: osdmap e693: 12 total, 12 up, 10 in 2024-04-29T14:36:51.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:51 smithi040 ceph-mon[39581]: pgmap v1774: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 59 KiB/s wr, 62 op/s; 29659/228176 objects misplaced (12.998%); 24 MiB/s, 380 objects/s recovering 2024-04-29T14:36:51.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:51 smithi040 ceph-mon[39581]: osdmap e693: 12 total, 12 up, 10 in 2024-04-29T14:36:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:51 smithi121 ceph-mon[44533]: pgmap v1774: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 85 B/s rd, 59 KiB/s wr, 62 op/s; 29659/228176 objects misplaced (12.998%); 24 MiB/s, 380 objects/s recovering 2024-04-29T14:36:51.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:51 smithi121 ceph-mon[44533]: osdmap e693: 12 total, 12 up, 10 in 2024-04-29T14:36:53.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:53 smithi194 ceph-mon[44089]: pgmap v1776: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 49 op/s; 28130/227944 objects misplaced (12.341%); 28 MiB/s, 452 objects/s recovering 2024-04-29T14:36:53.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:53 smithi040 ceph-mon[39581]: pgmap v1776: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 49 op/s; 28130/227944 objects misplaced (12.341%); 28 MiB/s, 452 objects/s recovering 2024-04-29T14:36:53.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:53 smithi121 ceph-mon[44533]: pgmap v1776: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 49 op/s; 28130/227944 objects misplaced (12.341%); 28 MiB/s, 452 objects/s recovering 2024-04-29T14:36:55.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:55 smithi194 ceph-mon[44089]: pgmap v1777: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 49 op/s; 27683/227804 objects misplaced (12.152%); 29 MiB/s, 459 objects/s recovering 2024-04-29T14:36:55.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:36:55.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:36:55.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:55 smithi040 ceph-mon[39581]: pgmap v1777: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 49 op/s; 27683/227804 objects misplaced (12.152%); 29 MiB/s, 459 objects/s recovering 2024-04-29T14:36:55.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:36:55.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:36:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:55 smithi121 ceph-mon[44533]: pgmap v1777: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 7.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 49 op/s; 27683/227804 objects misplaced (12.152%); 29 MiB/s, 459 objects/s recovering 2024-04-29T14:36:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:36:55.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:36:56.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:56 smithi194 ceph-mon[44089]: osdmap e694: 12 total, 12 up, 10 in 2024-04-29T14:36:56.570 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:56 smithi121 ceph-mon[44533]: osdmap e694: 12 total, 12 up, 10 in 2024-04-29T14:36:56.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:56 smithi040 ceph-mon[39581]: osdmap e694: 12 total, 12 up, 10 in 2024-04-29T14:36:57.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:57 smithi194 ceph-mon[44089]: pgmap v1779: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 39 op/s; 25878/227688 objects misplaced (11.366%); 33 MiB/s, 532 objects/s recovering 2024-04-29T14:36:57.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:57 smithi040 ceph-mon[39581]: pgmap v1779: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 39 op/s; 25878/227688 objects misplaced (11.366%); 33 MiB/s, 532 objects/s recovering 2024-04-29T14:36:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:57 smithi121 ceph-mon[44533]: pgmap v1779: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 39 op/s; 25878/227688 objects misplaced (11.366%); 33 MiB/s, 532 objects/s recovering 2024-04-29T14:36:57.728 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:36:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:57.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:36:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:36:57.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:36:57.862 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:57.922 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:36:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:57.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:36:58.026 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:36:58.054 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:58.073 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:36:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:36:58.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:36:58.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:36:58.213 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:36:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:36:58 smithi121 ceph-mon[44533]: pgmap v1780: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 40 op/s; 24894/227526 objects misplaced (10.941%); 37 MiB/s, 595 objects/s recovering 2024-04-29T14:36:58.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:36:58 smithi194 ceph-mon[44089]: pgmap v1780: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 40 op/s; 24894/227526 objects misplaced (10.941%); 37 MiB/s, 595 objects/s recovering 2024-04-29T14:36:58.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:36:58 smithi040 ceph-mon[39581]: pgmap v1780: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 40 op/s; 24894/227526 objects misplaced (10.941%); 37 MiB/s, 595 objects/s recovering 2024-04-29T14:37:00.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:37:01.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:01 smithi121 ceph-mon[44533]: pgmap v1781: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 37 op/s; 24263/227426 objects misplaced (10.669%); 34 MiB/s, 543 objects/s recovering 2024-04-29T14:37:01.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:01 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:01.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:01 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:01.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:01 smithi194 ceph-mon[44089]: pgmap v1781: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 37 op/s; 24263/227426 objects misplaced (10.669%); 34 MiB/s, 543 objects/s recovering 2024-04-29T14:37:01.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:01 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:01.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:01 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:01.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:01 smithi040 ceph-mon[39581]: pgmap v1781: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 37 op/s; 24263/227426 objects misplaced (10.669%); 34 MiB/s, 543 objects/s recovering 2024-04-29T14:37:01.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:01 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:01 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:02.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:02.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:02.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:02 smithi194 ceph-mon[44089]: osdmap e695: 12 total, 12 up, 10 in 2024-04-29T14:37:02.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:02.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:02.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:02 smithi040 ceph-mon[39581]: osdmap e695: 12 total, 12 up, 10 in 2024-04-29T14:37:02.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:02.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:02.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:02 smithi121 ceph-mon[44533]: osdmap e695: 12 total, 12 up, 10 in 2024-04-29T14:37:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:37:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:02.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:37:03.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:37:03.055 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:37:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:03.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:37:03.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:37:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:37:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:03.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:37:03.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:37:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:03.384 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:37:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:03.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:37:03.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:37:03.514 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:03 smithi121 ceph-mon[44533]: pgmap v1783: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 45 op/s; 22159/227074 objects misplaced (9.758%); 43 MiB/s, 690 objects/s recovering 2024-04-29T14:37:03.517 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:03.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:37:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:03.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:37:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:03 smithi194 ceph-mon[44089]: pgmap v1783: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 45 op/s; 22159/227074 objects misplaced (9.758%); 43 MiB/s, 690 objects/s recovering 2024-04-29T14:37:03.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:03 smithi040 ceph-mon[39581]: pgmap v1783: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 45 op/s; 22159/227074 objects misplaced (9.758%); 43 MiB/s, 690 objects/s recovering 2024-04-29T14:37:03.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:03.682 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:04.214 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:37:05.513 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:05 smithi040 ceph-mon[39581]: pgmap v1784: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 47 op/s; 21676/226928 objects misplaced (9.552%); 38 MiB/s, 612 objects/s recovering 2024-04-29T14:37:05.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:05 smithi194 ceph-mon[44089]: pgmap v1784: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 47 op/s; 21676/226928 objects misplaced (9.552%); 38 MiB/s, 612 objects/s recovering 2024-04-29T14:37:05.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:05 smithi121 ceph-mon[44533]: pgmap v1784: 17 pgs: 2 active+recovering+remapped, 4 active+recovering, 11 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 47 op/s; 21676/226928 objects misplaced (9.552%); 38 MiB/s, 612 objects/s recovering 2024-04-29T14:37:05.739 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:05.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:05.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:37:05.885 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:06.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:37:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:06.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:37:06.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:37:06.170 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:06 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:37:06.212+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e695 definitely_dead 0 2024-04-29T14:37:06.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:37:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:06.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:37:06.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:37:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:06.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:37:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:06.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:37:06.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:37:06.866 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:06.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:37:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:07.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:37:07.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:07.104 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:37:07.104 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:37:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:07.279 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.8. 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: pgmap v1785: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 19703/226634 objects misplaced (8.694%); 39 MiB/s, 617 objects/s recovering 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1909098623' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: from='client.27106 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3504287465,config=1981513354,health=1997216375,kv=3587628367,logm=2791582795}) 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2758840786}) 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2566533506}) 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045775072}) 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4244239798}) 2024-04-29T14:37:07.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1290894676}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2080820262}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3739984852}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1341390953}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169810067}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1404600860}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1054580183}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2497883049}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189504328}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458557902}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751830849}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940146611}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=205255836}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109312757}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054667040}) 2024-04-29T14:37:07.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067704013}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3008183899}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=71945777}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=964193710}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2698923588}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1850882131}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549006975}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3926810175}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703956735}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237642439}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319916937}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829189350}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906792827}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3062075840}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833644476}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561973763}) 2024-04-29T14:37:07.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=270056445}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899587253}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2061204661}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=761024094}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1355720642}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3134770263}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823708373}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560782956}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383518980}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209050387}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878445250}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1164656640}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938750038}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3625369729}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=283996303}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495645685}) 2024-04-29T14:37:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656396611}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3995908841}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2821305917}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167768903}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3489288338}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370165542}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1748618896}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2620447658}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605212387}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=157581040}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962086252}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1501880853}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2258821311}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2768662843}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2833730522}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537974397}) 2024-04-29T14:37:07.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2842425175}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3720462764}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1075077402}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90753233}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586751883}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316825504}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901848934}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245744304}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782679733}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3304233444}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=39509541}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2336040095}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4174866504}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1774836575}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=19,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=44} crc {logm=3670224388,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1055032321,mon_config_key=819999892}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=97761960}) 2024-04-29T14:37:07.387 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3994931407}) 2024-04-29T14:37:07.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=34,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=48} crc {mon_config_key=577581734,monmap=455069592,osd_metadata=809037704,osd_pg_creating=894803502,osdmap=3521434337}) 2024-04-29T14:37:07.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=73} crc {osdmap=1450303630}) 2024-04-29T14:37:07.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:37:07.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:37:07.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: pgmap v1785: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 19703/226634 objects misplaced (8.694%); 39 MiB/s, 617 objects/s recovering 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1909098623' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: from='client.27106 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3504287465,config=1981513354,health=1997216375,kv=3587628367,logm=2791582795}) 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2758840786}) 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2566533506}) 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045775072}) 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4244239798}) 2024-04-29T14:37:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1290894676}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2080820262}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3739984852}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1341390953}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169810067}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1404600860}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1054580183}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2497883049}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189504328}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458557902}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751830849}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940146611}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=205255836}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109312757}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054667040}) 2024-04-29T14:37:07.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067704013}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3008183899}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=71945777}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=964193710}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2698923588}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1850882131}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549006975}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3926810175}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703956735}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237642439}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319916937}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829189350}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906792827}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3062075840}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833644476}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561973763}) 2024-04-29T14:37:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=270056445}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899587253}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2061204661}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=761024094}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1355720642}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3134770263}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823708373}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560782956}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383518980}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209050387}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878445250}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1164656640}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938750038}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3625369729}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=283996303}) 2024-04-29T14:37:07.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495645685}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656396611}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3995908841}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2821305917}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167768903}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3489288338}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370165542}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1748618896}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2620447658}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605212387}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=157581040}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962086252}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1501880853}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2258821311}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2768662843}) 2024-04-29T14:37:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2833730522}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537974397}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2842425175}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3720462764}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1075077402}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90753233}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586751883}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316825504}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901848934}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245744304}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782679733}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3304233444}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=39509541}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2336040095}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4174866504}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1774836575}) 2024-04-29T14:37:07.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=19,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=44} crc {logm=3670224388,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1055032321,mon_config_key=819999892}) 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=97761960}) 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3994931407}) 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=34,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=48} crc {mon_config_key=577581734,monmap=455069592,osd_metadata=809037704,osd_pg_creating=894803502,osdmap=3521434337}) 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=73} crc {osdmap=1450303630}) 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:37:07.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: pgmap v1785: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 19703/226634 objects misplaced (8.694%); 39 MiB/s, 617 objects/s recovering 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1909098623' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: from='client.27106 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3504287465,config=1981513354,health=1997216375,kv=3587628367,logm=2791582795}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2758840786}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2566533506}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045775072}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4244239798}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1290894676}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2080820262}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3739984852}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1341390953}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169810067}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1404600860}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1054580183}) 2024-04-29T14:37:07.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2497883049}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1189504328}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458557902}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751830849}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940146611}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=205255836}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109312757}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054667040}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067704013}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3008183899}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=71945777}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=964193710}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2698923588}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1850882131}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549006975}) 2024-04-29T14:37:07.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3926810175}) 2024-04-29T14:37:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703956735}) 2024-04-29T14:37:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237642439}) 2024-04-29T14:37:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319916937}) 2024-04-29T14:37:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829189350}) 2024-04-29T14:37:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906792827}) 2024-04-29T14:37:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3062075840}) 2024-04-29T14:37:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833644476}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561973763}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=270056445}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899587253}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2061204661}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=761024094}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1355720642}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3134770263}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823708373}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560782956}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383518980}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=209050387}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878445250}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1164656640}) 2024-04-29T14:37:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938750038}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3625369729}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=283996303}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1495645685}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656396611}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3995908841}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2821305917}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167768903}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3489288338}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370165542}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1748618896}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2620447658}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605212387}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=157581040}) 2024-04-29T14:37:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962086252}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1501880853}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2258821311}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2768662843}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2833730522}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1537974397}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2842425175}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3720462764}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1075077402}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90753233}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586751883}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=316825504}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901848934}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245744304}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2782679733}) 2024-04-29T14:37:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3304233444}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=39509541}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2336040095}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4174866504}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1774836575}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=19,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=44} crc {logm=3670224388,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1055032321,mon_config_key=819999892}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=97761960}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3994931407}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=34,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=48} crc {mon_config_key=577581734,monmap=455069592,osd_metadata=809037704,osd_pg_creating=894803502,osdmap=3521434337}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=73} crc {osdmap=1450303630}) 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:37:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:37:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:37:07.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:37:08.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:08 smithi040 ceph-mon[39581]: from='client.27106 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-04-29T14:37:08.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:08 smithi040 ceph-mon[39581]: osdmap e696: 12 total, 12 up, 11 in 2024-04-29T14:37:08.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:08 smithi040 ceph-mon[39581]: osdmap e697: 12 total, 12 up, 11 in 2024-04-29T14:37:08.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1180851765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:37:08.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:08 smithi040 ceph-mon[39581]: from='client.27112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:37:08.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:08 smithi194 ceph-mon[44089]: from='client.27106 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-04-29T14:37:08.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:08 smithi194 ceph-mon[44089]: osdmap e696: 12 total, 12 up, 11 in 2024-04-29T14:37:08.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:08 smithi194 ceph-mon[44089]: osdmap e697: 12 total, 12 up, 11 in 2024-04-29T14:37:08.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1180851765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:37:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:08 smithi194 ceph-mon[44089]: from='client.27112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:37:08.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:08 smithi121 ceph-mon[44533]: from='client.27106 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-04-29T14:37:08.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:08 smithi121 ceph-mon[44533]: osdmap e696: 12 total, 12 up, 11 in 2024-04-29T14:37:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:08 smithi121 ceph-mon[44533]: osdmap e697: 12 total, 12 up, 11 in 2024-04-29T14:37:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1180851765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:37:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:08 smithi121 ceph-mon[44533]: from='client.27112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:37:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:09 smithi194 ceph-mon[44089]: pgmap v1788: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 46 op/s; 18376/226424 objects misplaced (8.116%); 34 MiB/s, 544 objects/s recovering 2024-04-29T14:37:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:09 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:37:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:09 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:37:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:09 smithi194 ceph-mon[44089]: from='client.27112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:37:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:09 smithi194 ceph-mon[44089]: osdmap e698: 12 total, 12 up, 11 in 2024-04-29T14:37:09.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:09 smithi040 ceph-mon[39581]: pgmap v1788: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 46 op/s; 18376/226424 objects misplaced (8.116%); 34 MiB/s, 544 objects/s recovering 2024-04-29T14:37:09.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:09 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:37:09.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:09 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:37:09.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:09 smithi040 ceph-mon[39581]: from='client.27112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:37:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:09 smithi040 ceph-mon[39581]: osdmap e698: 12 total, 12 up, 11 in 2024-04-29T14:37:09.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:09 smithi121 ceph-mon[44533]: pgmap v1788: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 46 op/s; 18376/226424 objects misplaced (8.116%); 34 MiB/s, 544 objects/s recovering 2024-04-29T14:37:09.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:09 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:37:09.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:09 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:37:09.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:09 smithi121 ceph-mon[44533]: from='client.27112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:37:09.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:09 smithi121 ceph-mon[44533]: osdmap e698: 12 total, 12 up, 11 in 2024-04-29T14:37:10.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:10.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:10.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:10.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:10.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:10.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:37:11.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:11 smithi194 ceph-mon[44089]: pgmap v1790: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 17459/226292 objects misplaced (7.715%); 44 MiB/s, 702 objects/s recovering 2024-04-29T14:37:11.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:11 smithi194 ceph-mon[44089]: osdmap e699: 12 total, 12 up, 11 in 2024-04-29T14:37:11.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:11 smithi040 ceph-mon[39581]: pgmap v1790: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 17459/226292 objects misplaced (7.715%); 44 MiB/s, 702 objects/s recovering 2024-04-29T14:37:11.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:11 smithi040 ceph-mon[39581]: osdmap e699: 12 total, 12 up, 11 in 2024-04-29T14:37:11.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:11 smithi121 ceph-mon[44533]: pgmap v1790: 17 pgs: 2 active+recovering+remapped, 3 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 17459/226292 objects misplaced (7.715%); 44 MiB/s, 702 objects/s recovering 2024-04-29T14:37:11.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:11 smithi121 ceph-mon[44533]: osdmap e699: 12 total, 12 up, 11 in 2024-04-29T14:37:12.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:12 smithi194 ceph-mon[44089]: osdmap e700: 12 total, 12 up, 11 in 2024-04-29T14:37:12.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:12 smithi040 ceph-mon[39581]: osdmap e700: 12 total, 12 up, 11 in 2024-04-29T14:37:12.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:12 smithi121 ceph-mon[44533]: osdmap e700: 12 total, 12 up, 11 in 2024-04-29T14:37:13.325 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:13.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:13.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:37:13.461 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:13.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:37:13.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:13 smithi040 ceph-mon[39581]: pgmap v1793: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 38162/226132 objects misplaced (16.876%); 28 MiB/s, 453 objects/s recovering 2024-04-29T14:37:13.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:13 smithi121 ceph-mon[44533]: pgmap v1793: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 38162/226132 objects misplaced (16.876%); 28 MiB/s, 453 objects/s recovering 2024-04-29T14:37:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:13.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:37:13.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:13 smithi194 ceph-mon[44089]: pgmap v1793: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 38162/226132 objects misplaced (16.876%); 28 MiB/s, 453 objects/s recovering 2024-04-29T14:37:13.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:37:13.927 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:14.411 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:37:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:14.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:37:14.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:14.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:14 smithi040 ceph-mon[39581]: pgmap v1794: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 37533/226048 objects misplaced (16.604%); 23 MiB/s, 369 objects/s recovering 2024-04-29T14:37:14.630 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:14.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:14 smithi194 ceph-mon[44089]: pgmap v1794: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 37533/226048 objects misplaced (16.604%); 23 MiB/s, 369 objects/s recovering 2024-04-29T14:37:14.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:14 smithi121 ceph-mon[44533]: pgmap v1794: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 37533/226048 objects misplaced (16.604%); 23 MiB/s, 369 objects/s recovering 2024-04-29T14:37:15.584 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:15.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:15.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:37:15.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:15.813 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:37:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:15.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:37:15.918 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:37:15.944 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:15.963 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:37:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:15.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:37:16.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:37:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:16.116 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:37:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:16.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:37:16.221 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:37:16.248 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:37:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:16.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:37:16.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:37:16.397 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:16.417 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:37:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:16.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:37:16.523 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:37:16.550 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:16.569 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:37:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:16.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:37:16.673 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:37:16.787 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:17.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:17 smithi194 ceph-mon[44089]: pgmap v1795: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 44 op/s; 35361/225738 objects misplaced (15.665%); 36 MiB/s, 568 objects/s recovering 2024-04-29T14:37:17.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:17 smithi040 ceph-mon[39581]: pgmap v1795: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 44 op/s; 35361/225738 objects misplaced (15.665%); 36 MiB/s, 568 objects/s recovering 2024-04-29T14:37:17.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:17 smithi121 ceph-mon[44533]: pgmap v1795: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 44 op/s; 35361/225738 objects misplaced (15.665%); 36 MiB/s, 568 objects/s recovering 2024-04-29T14:37:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:18 smithi194 ceph-mon[44089]: osdmap e701: 12 total, 12 up, 11 in 2024-04-29T14:37:18.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:18 smithi194 ceph-mon[44089]: 7.1 scrub starts 2024-04-29T14:37:18.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:18.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:18.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:18 smithi040 ceph-mon[39581]: osdmap e701: 12 total, 12 up, 11 in 2024-04-29T14:37:18.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:18 smithi040 ceph-mon[39581]: 7.1 scrub starts 2024-04-29T14:37:18.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:18.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:18.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:18 smithi121 ceph-mon[44533]: osdmap e701: 12 total, 12 up, 11 in 2024-04-29T14:37:18.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:18 smithi121 ceph-mon[44533]: 7.1 scrub starts 2024-04-29T14:37:19.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:19 smithi121 ceph-mon[44533]: pgmap v1797: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 34077/225382 objects misplaced (15.120%); 38 MiB/s, 601 objects/s recovering 2024-04-29T14:37:19.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:19 smithi121 ceph-mon[44533]: 7.1 scrub ok 2024-04-29T14:37:19.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:19 smithi194 ceph-mon[44089]: pgmap v1797: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 34077/225382 objects misplaced (15.120%); 38 MiB/s, 601 objects/s recovering 2024-04-29T14:37:19.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:19 smithi194 ceph-mon[44089]: 7.1 scrub ok 2024-04-29T14:37:19.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:19 smithi040 ceph-mon[39581]: pgmap v1797: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 34077/225382 objects misplaced (15.120%); 38 MiB/s, 601 objects/s recovering 2024-04-29T14:37:19.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:19 smithi040 ceph-mon[39581]: 7.1 scrub ok 2024-04-29T14:37:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:20 smithi194 ceph-mon[44089]: osdmap e702: 12 total, 12 up, 11 in 2024-04-29T14:37:20.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:20 smithi194 ceph-mon[44089]: pgmap v1799: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 56 op/s; 34051/225220 objects misplaced (15.119%); 32 MiB/s, 513 objects/s recovering 2024-04-29T14:37:20.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:20.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:20.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:20 smithi040 ceph-mon[39581]: osdmap e702: 12 total, 12 up, 11 in 2024-04-29T14:37:20.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:20 smithi040 ceph-mon[39581]: pgmap v1799: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 56 op/s; 34051/225220 objects misplaced (15.119%); 32 MiB/s, 513 objects/s recovering 2024-04-29T14:37:20.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:20.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:20.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:20 smithi121 ceph-mon[44533]: osdmap e702: 12 total, 12 up, 11 in 2024-04-29T14:37:20.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:20 smithi121 ceph-mon[44533]: pgmap v1799: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 56 op/s; 34051/225220 objects misplaced (15.119%); 32 MiB/s, 513 objects/s recovering 2024-04-29T14:37:21.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:21 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:37:21.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:21 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:37:21.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:21 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:37:22.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:22 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:37:22.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:22 smithi194 ceph-mon[44089]: pgmap v1800: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 33306/224782 objects misplaced (14.817%); 33 MiB/s, 528 objects/s recovering 2024-04-29T14:37:22.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:22 smithi194 ceph-mon[44089]: osdmap e703: 12 total, 12 up, 11 in 2024-04-29T14:37:22.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:22 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:37:22.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:22 smithi040 ceph-mon[39581]: pgmap v1800: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 33306/224782 objects misplaced (14.817%); 33 MiB/s, 528 objects/s recovering 2024-04-29T14:37:22.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:22 smithi040 ceph-mon[39581]: osdmap e703: 12 total, 12 up, 11 in 2024-04-29T14:37:22.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:22 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:37:22.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:22 smithi121 ceph-mon[44533]: pgmap v1800: 17 pgs: 3 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 33306/224782 objects misplaced (14.817%); 33 MiB/s, 528 objects/s recovering 2024-04-29T14:37:22.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:22 smithi121 ceph-mon[44533]: osdmap e703: 12 total, 12 up, 11 in 2024-04-29T14:37:23.155 INFO:tasks.thrashosds.thrasher:Added osd 8 2024-04-29T14:37:23.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:23 smithi194 ceph-mon[44089]: 7.e deep-scrub starts 2024-04-29T14:37:23.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:23 smithi194 ceph-mon[44089]: 7.a deep-scrub starts 2024-04-29T14:37:23.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:23 smithi194 ceph-mon[44089]: osdmap e704: 12 total, 12 up, 11 in 2024-04-29T14:37:23.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:23 smithi040 ceph-mon[39581]: 7.e deep-scrub starts 2024-04-29T14:37:23.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:23 smithi040 ceph-mon[39581]: 7.a deep-scrub starts 2024-04-29T14:37:23.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:23 smithi040 ceph-mon[39581]: osdmap e704: 12 total, 12 up, 11 in 2024-04-29T14:37:23.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:23 smithi121 ceph-mon[44533]: 7.e deep-scrub starts 2024-04-29T14:37:23.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:23 smithi121 ceph-mon[44533]: 7.a deep-scrub starts 2024-04-29T14:37:23.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:23 smithi121 ceph-mon[44533]: osdmap e704: 12 total, 12 up, 11 in 2024-04-29T14:37:24.125 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:37:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:24.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:37:24.230 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:37:24.260 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:24.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:37:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:24.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:37:24.384 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:37:24.413 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:24.434 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:37:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:24.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:37:24.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:37:24.565 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:24.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:24 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:37:24.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:24 smithi194 ceph-mon[44089]: pgmap v1803: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 69 op/s; 32349/224544 objects misplaced (14.407%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:37:24.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:37:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:24.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:37:24.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:37:24.714 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:24 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:37:24.714 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:24 smithi121 ceph-mon[44533]: pgmap v1803: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 69 op/s; 32349/224544 objects misplaced (14.407%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:37:24.716 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:24.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:37:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:24.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:37:24.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:37:24.867 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:24.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:24 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:37:24.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:24 smithi040 ceph-mon[39581]: pgmap v1803: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.9 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 69 op/s; 32349/224544 objects misplaced (14.407%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:37:24.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:37:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:24.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:37:24.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:37:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:25.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:25 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:37:25.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:25 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:37:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:25.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:37:25.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:25 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:37:25.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:25 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:37:25.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:25 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:37:25.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:25 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:37:25.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:37:25.986 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:26.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:37:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:26.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:37:26.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:37:26.137 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:26.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:37:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:26.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:37:26.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:37:26.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:37:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:26.446 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:37:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:26.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:37:26.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:37:26.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:37:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:26.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:37:26.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:37:26.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:26 smithi121 ceph-mon[44533]: 7.3 scrub starts 2024-04-29T14:37:26.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:26 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:37:26.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:26 smithi121 ceph-mon[44533]: pgmap v1804: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 32053/224220 objects misplaced (14.295%); 19 MiB/s, 300 objects/s recovering 2024-04-29T14:37:26.778 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:26.798 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:37:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:26.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:37:26.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:26 smithi194 ceph-mon[44089]: 7.3 scrub starts 2024-04-29T14:37:26.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:26 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:37:26.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:26 smithi194 ceph-mon[44089]: pgmap v1804: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 32053/224220 objects misplaced (14.295%); 19 MiB/s, 300 objects/s recovering 2024-04-29T14:37:26.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:26 smithi040 ceph-mon[39581]: 7.3 scrub starts 2024-04-29T14:37:26.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:26 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:37:26.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:26 smithi040 ceph-mon[39581]: pgmap v1804: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 32053/224220 objects misplaced (14.295%); 19 MiB/s, 300 objects/s recovering 2024-04-29T14:37:26.901 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:37:26.929 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:26.950 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:37:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:26.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:37:27.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:37:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:27.230 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:37:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:27.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:37:27.335 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:37:27.353 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:37:27.371 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:27.404 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:37:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:27.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:37:27.423 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:37:27.461 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:37:27.492 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:37:27.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:37:27.545 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:37:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:27.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:37:27.600 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:37:27.643 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:37:27.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:37:27.686 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:37:27.732 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:27 smithi040 ceph-mon[39581]: 7.3 scrub ok 2024-04-29T14:37:27.732 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2844287322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:37:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:27.745 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:37:27.816 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:37:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:27 smithi194 ceph-mon[44089]: 7.3 scrub ok 2024-04-29T14:37:27.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2844287322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:37:27.860 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:37:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:27 smithi121 ceph-mon[44533]: 7.3 scrub ok 2024-04-29T14:37:27.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2844287322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:37:27.886 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:37:27.916 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:37:27.954 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:37:27.994 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:37:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:37:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:37:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:37:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:37:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:37:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:37:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:37:28.014 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:37:28.014 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:37:28.014 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:37:28.014 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:37:28.014 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:37:28.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8] out_osds: [3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:37:28.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:37:28.156 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-29T14:37:28.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:37:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:28 smithi194 ceph-mon[44089]: pgmap v1805: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 10 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 31590/223912 objects misplaced (14.108%); 19 MiB/s, 307 objects/s recovering 2024-04-29T14:37:28.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:37:28.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:28 smithi040 ceph-mon[39581]: pgmap v1805: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 10 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 31590/223912 objects misplaced (14.108%); 19 MiB/s, 307 objects/s recovering 2024-04-29T14:37:28.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:37:28.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:37:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:28 smithi121 ceph-mon[44533]: pgmap v1805: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 10 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 31590/223912 objects misplaced (14.108%); 19 MiB/s, 307 objects/s recovering 2024-04-29T14:37:29.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:29.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:29.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:37:30.304 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:37:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:30.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:37:30.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:37:30.436 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:30.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:37:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:30.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:37:30.559 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:37:30.588 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:30.610 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:37:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:30.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:37:30.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:30 smithi040 ceph-mon[39581]: pgmap v1806: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 63 op/s; 31590/223758 objects misplaced (14.118%); 13 MiB/s, 214 objects/s recovering 2024-04-29T14:37:30.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:37:30.738 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:30 smithi121 ceph-mon[44533]: pgmap v1806: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 63 op/s; 31590/223758 objects misplaced (14.118%); 13 MiB/s, 214 objects/s recovering 2024-04-29T14:37:30.740 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:30.764 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:37:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:30.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:37:30.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:30 smithi194 ceph-mon[44089]: pgmap v1806: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 63 op/s; 31590/223758 objects misplaced (14.118%); 13 MiB/s, 214 objects/s recovering 2024-04-29T14:37:30.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:37:30.896 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:30.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:37:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:30.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:37:31.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:37:31.046 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:31.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:37:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:31.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:37:31.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:37:31.201 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:31.221 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:37:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:31.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:37:31.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:37:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:32.349 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:37:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:32.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:37:32.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:37:32.488 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:32.511 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:37:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:32.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:37:32.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:37:32.654 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:33.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:33 smithi194 ceph-mon[44089]: pgmap v1807: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 30810/223316 objects misplaced (13.797%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:37:33.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:33 smithi040 ceph-mon[39581]: pgmap v1807: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 30810/223316 objects misplaced (13.797%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:37:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:33 smithi121 ceph-mon[44533]: pgmap v1807: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 30810/223316 objects misplaced (13.797%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:37:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:37:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:34.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:37:34.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:37:34.760 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:34.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:37:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:34.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:37:34.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:37:34.916 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:34.937 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:37:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:34.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:37:35.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:37:35.068 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:35.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:37:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:35.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:37:35.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:35.240 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:35.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:35 smithi121 ceph-mon[44533]: pgmap v1808: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 69 op/s; 30810/223118 objects misplaced (13.809%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:37:35.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:35.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:35.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:35 smithi194 ceph-mon[44089]: pgmap v1808: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 69 op/s; 30810/223118 objects misplaced (13.809%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:37:35.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:35.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:35.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:35 smithi040 ceph-mon[39581]: pgmap v1808: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 69 op/s; 30810/223118 objects misplaced (13.809%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:37:35.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:35.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:36.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:37:36.509 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:36 smithi040 ceph-mon[39581]: osdmap e705: 12 total, 12 up, 11 in 2024-04-29T14:37:36.568 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:36.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:36 smithi194 ceph-mon[44089]: osdmap e705: 12 total, 12 up, 11 in 2024-04-29T14:37:36.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:36 smithi121 ceph-mon[44533]: osdmap e705: 12 total, 12 up, 11 in 2024-04-29T14:37:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:36.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:36.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:37:36.800 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:36.822 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:37:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:36.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:37:36.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:37:36.964 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:37.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:37 smithi040 ceph-mon[39581]: pgmap v1810: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 30485/222644 objects misplaced (13.692%); 9.8 MiB/s, 156 objects/s recovering 2024-04-29T14:37:37.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:37 smithi194 ceph-mon[44089]: pgmap v1810: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 30485/222644 objects misplaced (13.692%); 9.8 MiB/s, 156 objects/s recovering 2024-04-29T14:37:37.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:37 smithi121 ceph-mon[44533]: pgmap v1810: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 30485/222644 objects misplaced (13.692%); 9.8 MiB/s, 156 objects/s recovering 2024-04-29T14:37:39.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:39 smithi194 ceph-mon[44089]: pgmap v1811: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 29978/222270 objects misplaced (13.487%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:37:39.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:39 smithi040 ceph-mon[39581]: pgmap v1811: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 29978/222270 objects misplaced (13.487%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:37:39.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:39 smithi121 ceph-mon[44533]: pgmap v1811: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 29978/222270 objects misplaced (13.487%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:37:40.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:40 smithi194 ceph-mon[44089]: pgmap v1812: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 29978/222110 objects misplaced (13.497%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:37:40.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:40 smithi040 ceph-mon[39581]: pgmap v1812: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 29978/222110 objects misplaced (13.497%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:37:40.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:40 smithi121 ceph-mon[44533]: pgmap v1812: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 29978/222110 objects misplaced (13.497%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:37:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:37:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:42.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:37:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:37:42.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:42.902 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:43.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:43.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:43.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:42 smithi040 ceph-mon[39581]: osdmap e706: 12 total, 12 up, 11 in 2024-04-29T14:37:43.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:42 smithi040 ceph-mon[39581]: pgmap v1814: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 29092/221810 objects misplaced (13.116%); 13 MiB/s, 214 objects/s recovering 2024-04-29T14:37:43.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:43.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:43.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:42 smithi194 ceph-mon[44089]: osdmap e706: 12 total, 12 up, 11 in 2024-04-29T14:37:43.320 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:42 smithi194 ceph-mon[44089]: pgmap v1814: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 29092/221810 objects misplaced (13.116%); 13 MiB/s, 214 objects/s recovering 2024-04-29T14:37:43.381 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:43.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:43.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:42 smithi121 ceph-mon[44533]: osdmap e706: 12 total, 12 up, 11 in 2024-04-29T14:37:43.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:42 smithi121 ceph-mon[44533]: pgmap v1814: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 29092/221810 objects misplaced (13.116%); 13 MiB/s, 214 objects/s recovering 2024-04-29T14:37:44.885 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:44.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:44.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:37:45.020 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:37:45.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:45.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:37:45.144 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:37:45.172 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:45.194 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:37:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:45.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:37:45.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:37:45.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:45 smithi121 ceph-mon[44533]: pgmap v1815: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 67 op/s; 29092/221686 objects misplaced (13.123%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:37:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:37:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:45.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:37:45.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:37:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:45.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:37:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:45.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:37:45.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:45 smithi194 ceph-mon[44089]: pgmap v1815: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 67 op/s; 29092/221686 objects misplaced (13.123%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:37:45.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:37:45.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:45 smithi040 ceph-mon[39581]: pgmap v1815: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 67 op/s; 29092/221686 objects misplaced (13.123%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:37:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:46.156 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:37:46.156 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:37:46.157 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:37:46.157 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:37:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:37:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:47.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:37:47.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:47.162 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:47.210 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:37:47.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:47 smithi040 ceph-mon[39581]: pgmap v1816: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 28705/221552 objects misplaced (12.956%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:37:47.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3980530780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:37:47.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:47 smithi040 ceph-mon[39581]: from='client.36878 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:37:47.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:47 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:47.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:47 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:47.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:47 smithi194 ceph-mon[44089]: pgmap v1816: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 28705/221552 objects misplaced (12.956%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:37:47.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3980530780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:37:47.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:47 smithi194 ceph-mon[44089]: from='client.36878 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:37:47.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:47 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:47.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:47 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:47 smithi121 ceph-mon[44533]: pgmap v1816: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 28705/221552 objects misplaced (12.956%); 11 MiB/s, 177 objects/s recovering 2024-04-29T14:37:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3980530780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:37:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:47 smithi121 ceph-mon[44533]: from='client.36878 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:37:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:47 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:47 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:47.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:48.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:37:48.081 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:48.342 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:37:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:48.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:37:48.454 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:37:48.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:48 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:37:48.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:48 smithi040 ceph-mon[39581]: from='client.36878 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:37:48.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:48 smithi040 ceph-mon[39581]: osdmap e707: 12 total, 12 up, 11 in 2024-04-29T14:37:48.483 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:48 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:37:48.483 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:48 smithi194 ceph-mon[44089]: from='client.36878 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:37:48.483 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:48 smithi194 ceph-mon[44089]: osdmap e707: 12 total, 12 up, 11 in 2024-04-29T14:37:48.486 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:48.505 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:37:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:48.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:37:48.610 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:48.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:48 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:37:48.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:48 smithi121 ceph-mon[44533]: from='client.36878 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:37:48.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:48 smithi121 ceph-mon[44533]: osdmap e707: 12 total, 12 up, 11 in 2024-04-29T14:37:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:49.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:49 smithi194 ceph-mon[44089]: pgmap v1818: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 28152/221388 objects misplaced (12.716%); 14 MiB/s, 228 objects/s recovering 2024-04-29T14:37:49.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:49 smithi040 ceph-mon[39581]: pgmap v1818: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 28152/221388 objects misplaced (12.716%); 14 MiB/s, 228 objects/s recovering 2024-04-29T14:37:49.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:49 smithi121 ceph-mon[44533]: pgmap v1818: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 28152/221388 objects misplaced (12.716%); 14 MiB/s, 228 objects/s recovering 2024-04-29T14:37:51.141 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:37:51.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:51 smithi194 ceph-mon[44089]: pgmap v1819: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 33 op/s; 28152/221266 objects misplaced (12.723%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:37:51.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:51 smithi040 ceph-mon[39581]: pgmap v1819: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 33 op/s; 28152/221266 objects misplaced (12.723%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:37:51.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:51 smithi121 ceph-mon[44533]: pgmap v1819: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 33 op/s; 28152/221266 objects misplaced (12.723%); 7.2 MiB/s, 115 objects/s recovering 2024-04-29T14:37:52.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:52.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:52.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:52.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:52.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:52.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:53.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:53 smithi040 ceph-mon[39581]: pgmap v1820: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 27177/221004 objects misplaced (12.297%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:37:53.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:53 smithi040 ceph-mon[39581]: osdmap e708: 12 total, 12 up, 11 in 2024-04-29T14:37:53.476 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:53.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:53 smithi194 ceph-mon[44089]: pgmap v1820: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 27177/221004 objects misplaced (12.297%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:37:53.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:53 smithi194 ceph-mon[44089]: osdmap e708: 12 total, 12 up, 11 in 2024-04-29T14:37:53.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:53 smithi121 ceph-mon[44533]: pgmap v1820: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 27177/221004 objects misplaced (12.297%); 12 MiB/s, 191 objects/s recovering 2024-04-29T14:37:53.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:53 smithi121 ceph-mon[44533]: osdmap e708: 12 total, 12 up, 11 in 2024-04-29T14:37:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:54.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:54.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:54.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:54.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:54.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:54.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:37:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:54.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:37:54.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:37:54.982 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:55.003 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:37:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:37:55.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:37:55.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:37:55.145 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:37:55.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:55 smithi194 ceph-mon[44089]: pgmap v1822: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 27177/220898 objects misplaced (12.303%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:37:55.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:55 smithi194 ceph-mon[44089]: osdmap e709: 12 total, 12 up, 11 in 2024-04-29T14:37:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:55 smithi040 ceph-mon[39581]: pgmap v1822: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 27177/220898 objects misplaced (12.303%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:37:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:55 smithi040 ceph-mon[39581]: osdmap e709: 12 total, 12 up, 11 in 2024-04-29T14:37:55.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:55 smithi121 ceph-mon[44533]: pgmap v1822: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 27177/220898 objects misplaced (12.303%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:37:55.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:55 smithi121 ceph-mon[44533]: osdmap e709: 12 total, 12 up, 11 in 2024-04-29T14:37:56.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:37:57.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:57 smithi194 ceph-mon[44089]: pgmap v1824: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 41 op/s; 26778/220716 objects misplaced (12.132%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:37:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:57 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:37:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:57 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:37:57.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:57 smithi040 ceph-mon[39581]: pgmap v1824: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 41 op/s; 26778/220716 objects misplaced (12.132%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:37:57.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:57 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:37:57.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:57 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:37:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:57 smithi121 ceph-mon[44533]: pgmap v1824: 17 pgs: 2 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 11 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 41 op/s; 26778/220716 objects misplaced (12.132%); 11 MiB/s, 171 objects/s recovering 2024-04-29T14:37:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:57 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:37:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:57 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:37:58.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:58 smithi194 ceph-mon[44089]: 7.e deep-scrub ok 2024-04-29T14:37:58.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:58 smithi194 ceph-mon[44089]: osdmap e710: 12 total, 12 up, 11 in 2024-04-29T14:37:58.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:58 smithi040 ceph-mon[39581]: 7.e deep-scrub ok 2024-04-29T14:37:58.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:58 smithi040 ceph-mon[39581]: osdmap e710: 12 total, 12 up, 11 in 2024-04-29T14:37:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:58 smithi121 ceph-mon[44533]: 7.e deep-scrub ok 2024-04-29T14:37:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:58 smithi121 ceph-mon[44533]: osdmap e710: 12 total, 12 up, 11 in 2024-04-29T14:37:59.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:59 smithi194 ceph-mon[44089]: pgmap v1826: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 26175/220316 objects misplaced (11.881%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:37:59.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:37:59 smithi194 ceph-mon[44089]: 7.a deep-scrub ok 2024-04-29T14:37:59.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:59 smithi040 ceph-mon[39581]: pgmap v1826: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 26175/220316 objects misplaced (11.881%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:37:59.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:37:59 smithi040 ceph-mon[39581]: 7.a deep-scrub ok 2024-04-29T14:37:59.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:59 smithi121 ceph-mon[44533]: pgmap v1826: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 26175/220316 objects misplaced (11.881%); 10 MiB/s, 166 objects/s recovering 2024-04-29T14:37:59.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:37:59 smithi121 ceph-mon[44533]: 7.a deep-scrub ok 2024-04-29T14:38:00.115 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:00.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:00.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:38:00.253 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:00.273 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:38:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:00.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:38:00.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:38:00.402 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:01.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:01 smithi194 ceph-mon[44089]: pgmap v1827: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 53 op/s; 26175/220156 objects misplaced (11.889%); 7.9 MiB/s, 127 objects/s recovering 2024-04-29T14:38:01.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:01 smithi040 ceph-mon[39581]: pgmap v1827: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 53 op/s; 26175/220156 objects misplaced (11.889%); 7.9 MiB/s, 127 objects/s recovering 2024-04-29T14:38:01.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:01 smithi121 ceph-mon[44533]: pgmap v1827: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+remapped, 2 active+recovering, 12 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 53 op/s; 26175/220156 objects misplaced (11.889%); 7.9 MiB/s, 127 objects/s recovering 2024-04-29T14:38:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:01.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:01.880 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 710, 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:37:57.263030+0000", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:31:03.499853+0000", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:37:07.184321+0000", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5801984, 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "noscrub", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:38:01.881 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 30, 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 7, 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "luminous", 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.882 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:38:01.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:38:01.884 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:38:01.885 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:38:01.886 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:38:01.887 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 10, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 10, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.20000000298023224, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 7, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_5", 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:34:47.804723+0000", 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:38:01.888 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:38:01.889 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "659", 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:38:01.890 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:38:01.891 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:38:01.892 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.4800000190734863, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 1.8700000047683716, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 1.1000000238418579, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2.7300000190734863, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2.059999942779541, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946, 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: "error_message": "pool 7 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.893 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 698, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.894 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:38:01.895 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.896 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.897 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.7887725830078125, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 699, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.899 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.901 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 702, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:38:01.903 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:38:01.904 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:38:01.905 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.906 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.907 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.5530548095703125, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 629, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.910 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 388, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 457, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 561, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 703, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 459, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:38:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.914 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/2522198013", 2024-04-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/2522198013", 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/2522198013", 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/2522198013", 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.95001220703125, 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 696, 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:38:01.919 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.5831451416015625, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 634, 2024-04-29T14:38:01.920 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.921 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 696, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.924 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:38:01.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.927 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 655, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.928 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:38:01.930 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.931 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 684, 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.933 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.934 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.4297637939453125, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 524, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:38:01.935 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.936 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:38:01.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:38:01.938 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:01.026449+0000", 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:00.577774+0000", 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:41.901856+0000", 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:38:01.942 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:36:31.461697+0000", 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:54.024639+0000", 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.943 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:18.768794+0000", 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 457 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:56.655099+0000", 2024-04-29T14:38:01.944 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:16.393755+0000", 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.945 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:35.096976+0000", 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.946 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:09.820836+0000", 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:46.866094+0000", 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:38:01.947 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:51.469138+0000", 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [], 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [ 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "1.0", 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "mappings": [ 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi040.stdout: "from": 9, 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: "to": 2 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [ 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "7.9", 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: 8 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "7.f", 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: 8 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:38:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:38:01.952 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:02.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:38:02.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:38:02.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2609423195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:38:02.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:38:02.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:38:02.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2609423195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:38:02.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:38:02.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:38:02.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2609423195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:38:03.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:03 smithi040 ceph-mon[39581]: pgmap v1828: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 25048/219644 objects misplaced (11.404%); 17 MiB/s, 265 objects/s recovering 2024-04-29T14:38:03.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:03 smithi040 ceph-mon[39581]: osdmap e711: 12 total, 12 up, 11 in 2024-04-29T14:38:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:03 smithi194 ceph-mon[44089]: pgmap v1828: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 25048/219644 objects misplaced (11.404%); 17 MiB/s, 265 objects/s recovering 2024-04-29T14:38:03.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:03 smithi194 ceph-mon[44089]: osdmap e711: 12 total, 12 up, 11 in 2024-04-29T14:38:03.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:03 smithi121 ceph-mon[44533]: pgmap v1828: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 25048/219644 objects misplaced (11.404%); 17 MiB/s, 265 objects/s recovering 2024-04-29T14:38:03.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:03 smithi121 ceph-mon[44533]: osdmap e711: 12 total, 12 up, 11 in 2024-04-29T14:38:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:03.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:03.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:38:03.967 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:03.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:38:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:03.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:38:04.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:04.146 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:04.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:04.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:38:04.299 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:04.318 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:38:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:04.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:38:04.421 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:04.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:04.482 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:05.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:05.017 INFO:tasks.thrashosds.thrasher:j is {'epoch': 710, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:37:57.263030+0000', 'last_up_change': '2024-04-29T14:31:03.499853+0000', 'last_in_change': '2024-04-29T14:37:07.184321+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-04-29T14:34:47.804723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '659', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 1.8700000047683716, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995, 'error_message': 'pool 7 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 698, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 1, 'weight': 0.7887725830078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 699, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 702, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 0.5530548095703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 388, 'last_clean_end': 457, 'up_from': 561, 'up_thru': 703, 'down_at': 459, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 2522198013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2522198013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2522198013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2522198013}]}, 'public_addr': '172.21.15.121:6804/2522198013', 'cluster_addr': '172.21.15.121:6805/2522198013', 'heartbeat_back_addr': '172.21.15.121:6807/2522198013', 'heartbeat_front_addr': '172.21.15.121:6806/2522198013', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 696, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 0.5831451416015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 634, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 696, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 655, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 0.4297637939453125, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 524, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:01.026449+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:00.577774+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:41.901856+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:36:31.461697+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:54.024639+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:18.768794+0000', 'dead_epoch': 457}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:56.655099+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:16.393755+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:35.096976+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:09.820836+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:46.866094+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:51.469138+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 9, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.9', 'osds': [0, 8]}, {'pgid': '7.f', 'osds': [0, 8]}], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:38:05.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:38:05.021 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:05.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:38:05.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:05.147 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:38:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:05.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:38:05.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:05.286 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:05.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:05.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:05 smithi121 ceph-mon[44533]: pgmap v1830: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 25048/219406 objects misplaced (11.416%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:38:05.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:05.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:05 smithi040 ceph-mon[39581]: pgmap v1830: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 25048/219406 objects misplaced (11.416%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:38:05.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:05 smithi194 ceph-mon[44089]: pgmap v1830: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 25048/219406 objects misplaced (11.416%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:38:05.649 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:05.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:05.755 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:05.782 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:05.802 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:05.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:05.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:38:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:05.954 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:38:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:05.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:38:06.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:38:06.085 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:06.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:38:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:06.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:38:06.209 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:06.235 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:06.255 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:06.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:06.359 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:38:06.387 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:06.409 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:38:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:06.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:38:06.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:38:06.542 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:06.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:38:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:06.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:38:06.665 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:06.693 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:06.713 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:06.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:06.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:06.861 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:07.098 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:38:07.118 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:07.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:07.131 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:07.165 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:38:07.194 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:38:07.216 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:38:07.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:38:07.249 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:38:07.253 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:07.273 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:38:07.276 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:38:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:07.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:38:07.295 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:38:07.322 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:38:07.347 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:38:07.372 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:38:07.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:07.405 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:38:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:38:07.408 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:38:07.408 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:38:07.408 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:38:07.415 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:07.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:07 smithi040 ceph-mon[39581]: pgmap v1831: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 24562/218908 objects misplaced (11.220%); 11 MiB/s, 182 objects/s recovering 2024-04-29T14:38:07.415 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:07 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:38:07.416 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3937252722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:07.416 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:07 smithi194 ceph-mon[44089]: pgmap v1831: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 24562/218908 objects misplaced (11.220%); 11 MiB/s, 182 objects/s recovering 2024-04-29T14:38:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:07 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:38:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3937252722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:07 smithi121 ceph-mon[44533]: pgmap v1831: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 24562/218908 objects misplaced (11.220%); 11 MiB/s, 182 objects/s recovering 2024-04-29T14:38:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:07 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:38:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3937252722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:08.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:09.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:09 smithi040 ceph-mon[39581]: pgmap v1832: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 23804/218424 objects misplaced (10.898%); 15 MiB/s, 237 objects/s recovering 2024-04-29T14:38:09.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:09 smithi194 ceph-mon[44089]: pgmap v1832: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 23804/218424 objects misplaced (10.898%); 15 MiB/s, 237 objects/s recovering 2024-04-29T14:38:09.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:09 smithi121 ceph-mon[44533]: pgmap v1832: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 23804/218424 objects misplaced (10.898%); 15 MiB/s, 237 objects/s recovering 2024-04-29T14:38:09.673 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:09.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:09.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:09.814 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:11.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:11 smithi194 ceph-mon[44089]: pgmap v1833: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 23804/218174 objects misplaced (10.911%); 15 MiB/s, 237 objects/s recovering 2024-04-29T14:38:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:11 smithi040 ceph-mon[39581]: pgmap v1833: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 23804/218174 objects misplaced (10.911%); 15 MiB/s, 237 objects/s recovering 2024-04-29T14:38:11.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:11 smithi121 ceph-mon[44533]: pgmap v1833: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 23804/218174 objects misplaced (10.911%); 15 MiB/s, 237 objects/s recovering 2024-04-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:12.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:12.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:38:12.605 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:12.628 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:38:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:12.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:38:12.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:12.766 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:13.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:13 smithi194 ceph-mon[44089]: pgmap v1834: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 97 op/s; 22495/217684 objects misplaced (10.334%); 16 MiB/s, 255 objects/s recovering 2024-04-29T14:38:13.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:38:13.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:38:13.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:13 smithi040 ceph-mon[39581]: pgmap v1834: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 97 op/s; 22495/217684 objects misplaced (10.334%); 16 MiB/s, 255 objects/s recovering 2024-04-29T14:38:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:38:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:38:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:13 smithi121 ceph-mon[44533]: pgmap v1834: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 97 op/s; 22495/217684 objects misplaced (10.334%); 16 MiB/s, 255 objects/s recovering 2024-04-29T14:38:13.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:13.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:38:13.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:38:13.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:38:15.489 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:15 smithi040 ceph-mon[39581]: pgmap v1835: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 22467/217388 objects misplaced (10.335%); 14 MiB/s, 218 objects/s recovering 2024-04-29T14:38:15.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:15 smithi194 ceph-mon[44089]: pgmap v1835: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 22467/217388 objects misplaced (10.335%); 14 MiB/s, 218 objects/s recovering 2024-04-29T14:38:15.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:15 smithi121 ceph-mon[44533]: pgmap v1835: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 22467/217388 objects misplaced (10.335%); 14 MiB/s, 218 objects/s recovering 2024-04-29T14:38:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:15.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:15.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:15.919 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:15.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:16.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:38:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:16.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:38:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:16.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:38:16.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:16.254 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:17.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:38:17.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:17 smithi040 ceph-mon[39581]: pgmap v1836: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 21806/216944 objects misplaced (10.051%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:38:17.570 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:17 smithi194 ceph-mon[44089]: pgmap v1836: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 21806/216944 objects misplaced (10.051%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:38:17.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:17 smithi121 ceph-mon[44533]: pgmap v1836: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 21806/216944 objects misplaced (10.051%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:38:18.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:18 smithi040 ceph-mon[39581]: pgmap v1837: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 20935/216614 objects misplaced (9.665%); 19 MiB/s, 302 objects/s recovering 2024-04-29T14:38:18.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:18 smithi194 ceph-mon[44089]: pgmap v1837: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 20935/216614 objects misplaced (9.665%); 19 MiB/s, 302 objects/s recovering 2024-04-29T14:38:18.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:18 smithi121 ceph-mon[44533]: pgmap v1837: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 20935/216614 objects misplaced (9.665%); 19 MiB/s, 302 objects/s recovering 2024-04-29T14:38:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:38:19.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3760651375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:19.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:19 smithi194 ceph-mon[44089]: from='client.36893 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:19.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3760651375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:19.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:19 smithi040 ceph-mon[39581]: from='client.36893 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:19.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3760651375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:19.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:19 smithi121 ceph-mon[44533]: from='client.36893 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:20.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:20 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:38:20.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:20 smithi194 ceph-mon[44089]: from='client.36893 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:38:20.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:20 smithi194 ceph-mon[44089]: osdmap e712: 12 total, 12 up, 11 in 2024-04-29T14:38:20.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:20 smithi194 ceph-mon[44089]: pgmap v1839: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 90 op/s; 20935/216366 objects misplaced (9.676%); 18 MiB/s, 286 objects/s recovering 2024-04-29T14:38:20.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:20 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:38:20.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:20 smithi040 ceph-mon[39581]: from='client.36893 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:38:20.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:20 smithi040 ceph-mon[39581]: osdmap e712: 12 total, 12 up, 11 in 2024-04-29T14:38:20.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:20 smithi040 ceph-mon[39581]: pgmap v1839: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 90 op/s; 20935/216366 objects misplaced (9.676%); 18 MiB/s, 286 objects/s recovering 2024-04-29T14:38:20.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:20 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:38:20.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:20 smithi121 ceph-mon[44533]: from='client.36893 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:38:20.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:20 smithi121 ceph-mon[44533]: osdmap e712: 12 total, 12 up, 11 in 2024-04-29T14:38:20.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:20 smithi121 ceph-mon[44533]: pgmap v1839: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 90 op/s; 20935/216366 objects misplaced (9.676%); 18 MiB/s, 286 objects/s recovering 2024-04-29T14:38:21.912 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:21.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:22.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:23.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:23 smithi194 ceph-mon[44089]: pgmap v1840: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 19191/215972 objects misplaced (8.886%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:38:23.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:23 smithi040 ceph-mon[39581]: pgmap v1840: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 19191/215972 objects misplaced (8.886%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:38:23.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:23 smithi121 ceph-mon[44533]: pgmap v1840: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 19191/215972 objects misplaced (8.886%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:38:24.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:38:24.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:38:24.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:24 smithi194 ceph-mon[44089]: osdmap e713: 12 total, 12 up, 11 in 2024-04-29T14:38:24.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:38:24.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:38:24.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:38:24.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:38:24.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:24 smithi040 ceph-mon[39581]: osdmap e713: 12 total, 12 up, 11 in 2024-04-29T14:38:24.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:38:24.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:38:24.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:38:24.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:38:24.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:24 smithi121 ceph-mon[44533]: osdmap e713: 12 total, 12 up, 11 in 2024-04-29T14:38:24.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:38:24.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:38:24.928 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:24.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:24.954 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:25.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:38:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:25.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:38:25.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:25.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:38:25.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:25.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:38:25.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:38:25.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:25 smithi121 ceph-mon[44533]: pgmap v1842: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 19161/215654 objects misplaced (8.885%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:38:25.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:25 smithi121 ceph-mon[44533]: osdmap e714: 12 total, 12 up, 11 in 2024-04-29T14:38:25.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:38:25.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:38:25.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:38:25.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:38:25.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:25 smithi040 ceph-mon[39581]: pgmap v1842: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 19161/215654 objects misplaced (8.885%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:38:25.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:25 smithi040 ceph-mon[39581]: osdmap e714: 12 total, 12 up, 11 in 2024-04-29T14:38:25.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:38:25.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:38:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:38:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:38:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:25 smithi194 ceph-mon[44089]: pgmap v1842: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 19161/215654 objects misplaced (8.885%); 21 MiB/s, 330 objects/s recovering 2024-04-29T14:38:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:25 smithi194 ceph-mon[44089]: osdmap e714: 12 total, 12 up, 11 in 2024-04-29T14:38:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:38:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:38:26.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:26 smithi194 ceph-mon[44089]: osdmap e715: 12 total, 12 up, 11 in 2024-04-29T14:38:26.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:26 smithi040 ceph-mon[39581]: osdmap e715: 12 total, 12 up, 11 in 2024-04-29T14:38:26.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:26 smithi121 ceph-mon[44533]: osdmap e715: 12 total, 12 up, 11 in 2024-04-29T14:38:26.937 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:38:27.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:27 smithi040 ceph-mon[39581]: pgmap v1845: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 18040/215146 objects misplaced (8.385%); 30 MiB/s, 482 objects/s recovering 2024-04-29T14:38:27.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:27 smithi194 ceph-mon[44089]: pgmap v1845: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 18040/215146 objects misplaced (8.385%); 30 MiB/s, 482 objects/s recovering 2024-04-29T14:38:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:27 smithi121 ceph-mon[44533]: pgmap v1845: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 18040/215146 objects misplaced (8.385%); 30 MiB/s, 482 objects/s recovering 2024-04-29T14:38:27.910 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:38:27.910 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:38:28.196 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:28.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:28.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:38:28.333 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:28.372 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:38:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:28.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:38:28.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:28 smithi040 ceph-mon[39581]: osdmap e716: 12 total, 12 up, 11 in 2024-04-29T14:38:28.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:28.517 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:28.670 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:28.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:28.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:28.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:28 smithi194 ceph-mon[44089]: osdmap e716: 12 total, 12 up, 11 in 2024-04-29T14:38:28.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:28 smithi121 ceph-mon[44533]: osdmap e716: 12 total, 12 up, 11 in 2024-04-29T14:38:29.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:29 smithi194 ceph-mon[44089]: pgmap v1847: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 16968/214756 objects misplaced (7.901%); 28 MiB/s, 440 objects/s recovering 2024-04-29T14:38:29.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:29 smithi040 ceph-mon[39581]: pgmap v1847: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 16968/214756 objects misplaced (7.901%); 28 MiB/s, 440 objects/s recovering 2024-04-29T14:38:29.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:29 smithi121 ceph-mon[44533]: pgmap v1847: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 16968/214756 objects misplaced (7.901%); 28 MiB/s, 440 objects/s recovering 2024-04-29T14:38:30.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:30 smithi194 ceph-mon[44089]: pgmap v1848: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 16968/214514 objects misplaced (7.910%); 23 MiB/s, 362 objects/s recovering 2024-04-29T14:38:30.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:30 smithi040 ceph-mon[39581]: pgmap v1848: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 16968/214514 objects misplaced (7.910%); 23 MiB/s, 362 objects/s recovering 2024-04-29T14:38:30.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:30 smithi121 ceph-mon[44533]: pgmap v1848: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 16968/214514 objects misplaced (7.910%); 23 MiB/s, 362 objects/s recovering 2024-04-29T14:38:31.381 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:31.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:31.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:31.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:31.527 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:31.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:38:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:31.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:31.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:38:31.977 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:31.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:38:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:31.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:38:32.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:38:32.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:32.147 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:38:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:32.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:38:32.250 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:32.280 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:32.301 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:32.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:32.406 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:32.437 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:32.460 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:32.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:32.566 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:38:32.593 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:32.613 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:38:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:32.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:38:32.718 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:38:32.745 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:32.758 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:38:32.765 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:38:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:32.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:38:32.826 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:38:32.852 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:38:32.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:38:32.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:38:32.896 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:32.905 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:38:32.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:38:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:32.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:38:32.930 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:38:32.958 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:38:32.985 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:38:33.017 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:38:33.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:33.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:38:33.057 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:33.080 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:38:33.105 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:38:33.107 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:38:33.107 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:38:33.108 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:38:33.369 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: pgmap v1849: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 91 op/s; 15537/214208 objects misplaced (7.253%); 28 MiB/s, 470 objects/s recovering 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osdmap e717: 12 total, 12 up, 11 in 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/947081883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: pgmap v1849: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 91 op/s; 15537/214208 objects misplaced (7.253%); 28 MiB/s, 470 objects/s recovering 2024-04-29T14:38:33.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osdmap e717: 12 total, 12 up, 11 in 2024-04-29T14:38:33.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/947081883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:33.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: pgmap v1849: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 91 op/s; 15537/214208 objects misplaced (7.253%); 28 MiB/s, 470 objects/s recovering 2024-04-29T14:38:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osdmap e717: 12 total, 12 up, 11 in 2024-04-29T14:38:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/947081883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:33.742 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:38:33.743 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:38:34.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:34 smithi194 ceph-mon[44089]: from='client.27145 v1:172.21.15.40:0/2177629870' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:38:34.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:34 smithi040 ceph-mon[39581]: from='client.27145 v1:172.21.15.40:0/2177629870' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:38:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:34 smithi121 ceph-mon[44533]: from='client.27145 v1:172.21.15.40:0/2177629870' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:38:34.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:34.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:35.529 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:35.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:35 smithi194 ceph-mon[44089]: pgmap v1851: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 15489/213848 objects misplaced (7.243%); 20 MiB/s, 329 objects/s recovering 2024-04-29T14:38:35.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:35 smithi040 ceph-mon[39581]: pgmap v1851: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 15489/213848 objects misplaced (7.243%); 20 MiB/s, 329 objects/s recovering 2024-04-29T14:38:35.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:35 smithi121 ceph-mon[44533]: pgmap v1851: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 15489/213848 objects misplaced (7.243%); 20 MiB/s, 329 objects/s recovering 2024-04-29T14:38:36.335 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1850,"stamp":"2024-04-29T14:38:32.401478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7019561528,"num_objects":107104,"num_object_clones":0,"num_object_copies":214208,"num_objects_missing_on_primary":5858,"num_objects_missing":5858,"num_objects_degraded":0,"num_objects_misplaced":15537,"num_objects_unfound":0,"num_objects_dirty":107104,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":122899,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52758,"num_bytes_recovered":3452206816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127385,"ondisk_log_size":127385,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":22802268,"kb_used_data":16498048,"kb_used_omap":672,"kb_used_meta":6303455,"kb_avail":1008394404,"statfs":{"total":1055945392128,"available":1032595869696,"internally_reserved":0,"allocated":16894001152,"data_stored":16889123800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":688465,"internal_metadata":6454738607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-41811968,"num_objects":-638,"num_object_clones":0,"num_object_copies":-1276,"num_objects_missing_on_primary":-16,"num_objects_missing":-16,"num_objects_degraded":0,"num_objects_misplaced":-3624,"num_objects_unfound":0,"num_objects_dirty":-638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3732,"num_bytes_recovered":236453888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.184585"},"pg_stats":[{"pgid":"7.f","version":"716'7757","reported_seq":19124,"reported_epoch":716,"state":"active+recovering+remapped","last_fresh":"2024-04-29T14:38:31.407956+0000","last_change":"2024-04-29T14:37:08.687150+0000","last_active":"2024-04-29T14:38:31.407956+0000","last_peered":"2024-04-29T14:38:31.407956+0000","last_clean":"2024-04-29T14:37:05.246705+0000","last_became_active":"2024-04-29T14:37:08.629213+0000","last_became_peered":"2024-04-29T14:37:08.629213+0000","last_unstale":"2024-04-29T14:38:31.407956+0000","last_undegraded":"2024-04-29T14:38:31.407956+0000","last_fullsized":"2024-04-29T14:38:31.407956+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438239232,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1476,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7757,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5617,"num_bytes_recovered":368115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[0,8],"avail_no_missing":["0","9"],"object_location_counts":[{"shards":"0,8","objects":5211},{"shards":"0,9","objects":1476}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"716'14798","reported_seq":15223,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:27.722403+0000","last_change":"2024-04-29T14:37:57.239255+0000","last_active":"2024-04-29T14:38:27.722403+0000","last_peered":"2024-04-29T14:38:27.722403+0000","last_clean":"2024-04-29T14:38:27.722403+0000","last_became_active":"2024-04-29T14:34:49.387078+0000","last_became_peered":"2024-04-29T14:34:49.387078+0000","last_unstale":"2024-04-29T14:38:27.722403+0000","last_undegraded":"2024-04-29T14:38:27.722403+0000","last_fullsized":"2024-04-29T14:38:27.722403+0000","mapping_epoch":655,"log_start":"670'4700","ondisk_log_start":"670'4700","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"709'14617","last_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:37:57.239106+0000","objects_scrubbed":7056,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":36,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:19.725665+0000","scrub_duration":35381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446169088,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7742,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"716'7685","reported_seq":20403,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:30.114620+0000","last_change":"2024-04-29T14:37:02.322228+0000","last_active":"2024-04-29T14:38:30.114620+0000","last_peered":"2024-04-29T14:38:30.114620+0000","last_clean":"2024-04-29T14:38:30.114620+0000","last_became_active":"2024-04-29T14:35:47.383003+0000","last_became_peered":"2024-04-29T14:35:47.383003+0000","last_unstale":"2024-04-29T14:38:30.114620+0000","last_undegraded":"2024-04-29T14:38:30.114620+0000","last_fullsized":"2024-04-29T14:38:30.114620+0000","mapping_epoch":676,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7685,"log_dups_size":0,"ondisk_log_size":7685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:33.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435159064,"num_objects":6641,"num_object_clones":0,"num_object_copies":13282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7685,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6289,"num_bytes_recovered":412155904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"716'7668","reported_seq":7800,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:30.380530+0000","last_change":"2024-04-29T14:34:49.386907+0000","last_active":"2024-04-29T14:38:30.380530+0000","last_peered":"2024-04-29T14:38:30.380530+0000","last_clean":"2024-04-29T14:38:30.380530+0000","last_became_active":"2024-04-29T14:34:49.386724+0000","last_became_peered":"2024-04-29T14:34:49.386724+0000","last_unstale":"2024-04-29T14:38:30.380530+0000","last_undegraded":"2024-04-29T14:38:30.380530+0000","last_fullsized":"2024-04-29T14:38:30.380530+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7668,"log_dups_size":0,"ondisk_log_size":7668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:33.150136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437780480,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7668,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"715'7593","reported_seq":9537,"reported_epoch":715,"state":"active+recovering","last_fresh":"2024-04-29T14:38:27.387235+0000","last_change":"2024-04-29T14:36:34.055204+0000","last_active":"2024-04-29T14:38:27.387235+0000","last_peered":"2024-04-29T14:38:27.387235+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:36:33.135845+0000","last_became_peered":"2024-04-29T14:36:33.135845+0000","last_unstale":"2024-04-29T14:38:27.387235+0000","last_undegraded":"2024-04-29T14:38:27.387235+0000","last_fullsized":"2024-04-29T14:38:27.387235+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7593,"log_dups_size":0,"ondisk_log_size":7593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438370304,"num_objects":6689,"num_object_clones":0,"num_object_copies":13378,"num_objects_missing_on_primary":5858,"num_objects_missing":5858,"num_objects_degraded":0,"num_objects_misplaced":12531,"num_objects_unfound":0,"num_objects_dirty":6689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":893,"num_bytes_recovered":58523648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["10","11"],"object_location_counts":[{"shards":"1,5","objects":16},{"shards":"1,10,11","objects":815},{"shards":"10,11","objects":5858}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"716'14916","reported_seq":15341,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:30.196288+0000","last_change":"2024-04-29T14:37:59.002976+0000","last_active":"2024-04-29T14:38:30.196288+0000","last_peered":"2024-04-29T14:38:30.196288+0000","last_clean":"2024-04-29T14:38:30.196288+0000","last_became_active":"2024-04-29T14:34:49.387647+0000","last_became_peered":"2024-04-29T14:34:49.387647+0000","last_unstale":"2024-04-29T14:38:30.196288+0000","last_undegraded":"2024-04-29T14:38:30.196288+0000","last_fullsized":"2024-04-29T14:38:30.196288+0000","mapping_epoch":655,"log_start":"670'4900","ondisk_log_start":"670'4900","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'14736","last_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:37:59.002803+0000","objects_scrubbed":7074,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":36,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:14.256577+0000","scrub_duration":35998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447741952,"num_objects":6832,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7842,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"716'7697","reported_seq":18891,"reported_epoch":716,"state":"active+recovering+remapped","last_fresh":"2024-04-29T14:38:31.409005+0000","last_change":"2024-04-29T14:37:08.680862+0000","last_active":"2024-04-29T14:38:31.409005+0000","last_peered":"2024-04-29T14:38:31.409005+0000","last_clean":"2024-04-29T14:37:05.258984+0000","last_became_active":"2024-04-29T14:37:08.620633+0000","last_became_peered":"2024-04-29T14:37:08.620633+0000","last_unstale":"2024-04-29T14:38:31.409005+0000","last_undegraded":"2024-04-29T14:38:31.409005+0000","last_fullsized":"2024-04-29T14:38:31.409005+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435355648,"num_objects":6643,"num_object_clones":0,"num_object_copies":13286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1530,"num_objects_unfound":0,"num_objects_dirty":6643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5530,"num_bytes_recovered":362414080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[0,8],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,8","objects":5113},{"shards":"0,10","objects":1530}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"716'7596","reported_seq":7805,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:27.584394+0000","last_change":"2024-04-29T14:37:25.026135+0000","last_active":"2024-04-29T14:38:27.584394+0000","last_peered":"2024-04-29T14:38:27.584394+0000","last_clean":"2024-04-29T14:38:27.584394+0000","last_became_active":"2024-04-29T14:34:49.514880+0000","last_became_peered":"2024-04-29T14:34:49.514880+0000","last_unstale":"2024-04-29T14:38:27.584394+0000","last_undegraded":"2024-04-29T14:38:27.584394+0000","last_fullsized":"2024-04-29T14:38:27.584394+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'7313","last_scrub_stamp":"2024-04-29T14:37:25.025880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:37:25.025880+0000","objects_scrubbed":6980,"log_size":7596,"log_dups_size":0,"ondisk_log_size":7596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:36.200619+0000","scrub_duration":1104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438697984,"num_objects":6694,"num_object_clones":0,"num_object_copies":13388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7596,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"716'7887","reported_seq":35712,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:30.408565+0000","last_change":"2024-04-29T14:38:26.774529+0000","last_active":"2024-04-29T14:38:30.408565+0000","last_peered":"2024-04-29T14:38:30.408565+0000","last_clean":"2024-04-29T14:38:30.408565+0000","last_became_active":"2024-04-29T14:37:07.575390+0000","last_became_peered":"2024-04-29T14:37:07.575390+0000","last_unstale":"2024-04-29T14:38:30.408565+0000","last_undegraded":"2024-04-29T14:38:30.408565+0000","last_fullsized":"2024-04-29T14:38:30.408565+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7887,"log_dups_size":0,"ondisk_log_size":7887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:33.150101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448331776,"num_objects":6841,"num_object_clones":0,"num_object_copies":13682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7887,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13841,"num_bytes_recovered":898957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"716'7654","reported_seq":7863,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:30.074904+0000","last_change":"2024-04-29T14:37:27.778579+0000","last_active":"2024-04-29T14:38:30.074904+0000","last_peered":"2024-04-29T14:38:30.074904+0000","last_clean":"2024-04-29T14:38:30.074904+0000","last_became_active":"2024-04-29T14:34:49.388394+0000","last_became_peered":"2024-04-29T14:34:49.388394+0000","last_unstale":"2024-04-29T14:38:30.074904+0000","last_undegraded":"2024-04-29T14:38:30.074904+0000","last_fullsized":"2024-04-29T14:38:30.074904+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'7325","last_scrub_stamp":"2024-04-29T14:37:27.778410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:37:27.778410+0000","objects_scrubbed":6950,"log_size":7654,"log_dups_size":0,"ondisk_log_size":7654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":730,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433848320,"num_objects":6620,"num_object_clones":0,"num_object_copies":13240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7654,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"715'7702","reported_seq":20619,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-04-29T14:38:25.464669+0000","last_change":"2024-04-29T14:37:11.232594+0000","last_active":"2024-04-29T14:38:25.464669+0000","last_peered":"2024-04-29T14:38:25.464669+0000","last_clean":"2024-04-29T14:38:25.464669+0000","last_became_active":"2024-04-29T14:37:11.225189+0000","last_became_peered":"2024-04-29T14:37:11.225189+0000","last_unstale":"2024-04-29T14:38:25.464669+0000","last_undegraded":"2024-04-29T14:38:25.464669+0000","last_fullsized":"2024-04-29T14:38:25.464669+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7702,"log_dups_size":0,"ondisk_log_size":7702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445775872,"num_objects":6802,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7702,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6389,"num_bytes_recovered":418709504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"716'7682","reported_seq":7892,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:27.405046+0000","last_change":"2024-04-29T14:37:25.989267+0000","last_active":"2024-04-29T14:38:27.405046+0000","last_peered":"2024-04-29T14:38:27.405046+0000","last_clean":"2024-04-29T14:38:27.405046+0000","last_became_active":"2024-04-29T14:34:49.592105+0000","last_became_peered":"2024-04-29T14:34:49.592105+0000","last_unstale":"2024-04-29T14:38:27.405046+0000","last_undegraded":"2024-04-29T14:38:27.405046+0000","last_fullsized":"2024-04-29T14:38:27.405046+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'7405","last_scrub_stamp":"2024-04-29T14:37:25.989163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:37:25.989163+0000","objects_scrubbed":7011,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:40.668079+0000","scrub_duration":1063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440795136,"num_objects":6726,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"716'7544","reported_seq":7753,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:29.157946+0000","last_change":"2024-04-29T14:37:21.610409+0000","last_active":"2024-04-29T14:38:29.157946+0000","last_peered":"2024-04-29T14:38:29.157946+0000","last_clean":"2024-04-29T14:38:29.157946+0000","last_became_active":"2024-04-29T14:34:49.387077+0000","last_became_peered":"2024-04-29T14:34:49.387077+0000","last_unstale":"2024-04-29T14:38:29.157946+0000","last_undegraded":"2024-04-29T14:38:29.157946+0000","last_fullsized":"2024-04-29T14:38:29.157946+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'7231","last_scrub_stamp":"2024-04-29T14:37:21.610286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:37:21.610286+0000","objects_scrubbed":6961,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:31.177866+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435552256,"num_objects":6646,"num_object_clones":0,"num_object_copies":13292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7544,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"716'7575","reported_seq":21944,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:29.198665+0000","last_change":"2024-04-29T14:37:18.552775+0000","last_active":"2024-04-29T14:38:29.198665+0000","last_peered":"2024-04-29T14:38:29.198665+0000","last_clean":"2024-04-29T14:38:29.198665+0000","last_became_active":"2024-04-29T14:36:04.621648+0000","last_became_peered":"2024-04-29T14:36:04.621648+0000","last_unstale":"2024-04-29T14:38:29.198665+0000","last_undegraded":"2024-04-29T14:38:29.198665+0000","last_fullsized":"2024-04-29T14:38:29.198665+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"701'7237","last_scrub_stamp":"2024-04-29T14:37:18.552663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:37:18.552663+0000","objects_scrubbed":6944,"log_size":7575,"log_dups_size":0,"ondisk_log_size":7575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:39.921502+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432734208,"num_objects":6603,"num_object_clones":0,"num_object_copies":13206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7575,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7077,"num_bytes_recovered":463798272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"716'7597","reported_seq":21952,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:27.965894+0000","last_change":"2024-04-29T14:37:23.391838+0000","last_active":"2024-04-29T14:38:27.965894+0000","last_peered":"2024-04-29T14:38:27.965894+0000","last_clean":"2024-04-29T14:38:27.965894+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:38:27.965894+0000","last_undegraded":"2024-04-29T14:38:27.965894+0000","last_fullsized":"2024-04-29T14:38:27.965894+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435617792,"num_objects":6647,"num_object_clones":0,"num_object_copies":13294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7597,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"716'7497","reported_seq":7706,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:28.767407+0000","last_change":"2024-04-29T14:37:24.914899+0000","last_active":"2024-04-29T14:38:28.767407+0000","last_peered":"2024-04-29T14:38:28.767407+0000","last_clean":"2024-04-29T14:38:28.767407+0000","last_became_active":"2024-04-29T14:34:49.389669+0000","last_became_peered":"2024-04-29T14:34:49.389669+0000","last_unstale":"2024-04-29T14:38:28.767407+0000","last_undegraded":"2024-04-29T14:38:28.767407+0000","last_fullsized":"2024-04-29T14:38:28.767407+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'7179","last_scrub_stamp":"2024-04-29T14:37:24.914732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:37:24.914732+0000","objects_scrubbed":6861,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:46.795809+0000","scrub_duration":1000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428802048,"num_objects":6543,"num_object_clones":0,"num_object_copies":13086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1737,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-04-29T14:38:27.402644+0000","last_change":"2024-04-29T14:35:31.378616+0000","last_active":"2024-04-29T14:38:27.402644+0000","last_peered":"2024-04-29T14:38:27.402644+0000","last_clean":"2024-04-29T14:38:27.402644+0000","last_became_active":"2024-04-29T14:35:31.373888+0000","last_became_peered":"2024-04-29T14:35:31.373888+0000","last_unstale":"2024-04-29T14:38:27.402644+0000","last_undegraded":"2024-04-29T14:38:27.402644+0000","last_fullsized":"2024-04-29T14:38:27.402644+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:34:48.012447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.012447+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7018971160,"num_objects":107102,"num_object_clones":0,"num_object_copies":214204,"num_objects_missing_on_primary":5858,"num_objects_missing":5858,"num_objects_degraded":0,"num_objects_misplaced":15537,"num_objects_unfound":0,"num_objects_dirty":107102,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122718,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52744,"num_bytes_recovered":3448504320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16887783424,"data_stored":16887775280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127248,"ondisk_log_size":127248,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1611272,"kb_used_data":880384,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92133880,"statfs":{"total":95995035648,"available":94345093120,"internally_reserved":0,"allocated":901513216,"data_stored":901070058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":748419524},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384065,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3400520,"kb_used_data":2611328,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":90344632,"statfs":{"total":95995035648,"available":92512903168,"internally_reserved":0,"allocated":2673999872,"data_stored":2673556714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":808122816},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645701,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3085448,"kb_used_data":2404544,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":90659704,"statfs":{"total":95995035648,"available":92835536896,"internally_reserved":0,"allocated":2462253056,"data_stored":2461809918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":697235904},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423006,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181512,"kb_used_data":1068480,"kb_used_omap":4,"kb_used_meta":113019,"kb_avail":92563640,"statfs":{"total":95995035648,"available":94785167360,"internally_reserved":0,"allocated":1094123520,"data_stored":1093680382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":115732160},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267464,"kb_used_data":428288,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92477688,"statfs":{"total":95995035648,"available":94697152512,"internally_reserved":0,"allocated":438566912,"data_stored":438123774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":859306442},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528724,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2154004,"kb_used_data":1292620,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":91591148,"statfs":{"total":95995035648,"available":93789335552,"internally_reserved":0,"allocated":1323642880,"data_stored":1323192118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":882047430},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575270,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1673536,"kb_used_data":1375800,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92071616,"statfs":{"total":95995035648,"available":94281334784,"internally_reserved":0,"allocated":1408819200,"data_stored":1408378217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738802,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3018824,"kb_used_data":2243520,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":90726328,"statfs":{"total":95995035648,"available":92903759872,"internally_reserved":0,"allocated":2297364480,"data_stored":2296921342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3133124,"kb_used_data":2911996,"kb_used_omap":4,"kb_used_meta":221115,"kb_avail":90612028,"statfs":{"total":95995035648,"available":92786716672,"internally_reserved":0,"allocated":2981883904,"data_stored":2981443441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":226422468},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299912,"kb_used_data":430208,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92445240,"statfs":{"total":95995035648,"available":94663925760,"internally_reserved":0,"allocated":440532992,"data_stored":440089854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653147,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976652,"kb_used_data":850880,"kb_used_omap":651,"kb_used_meta":125108,"kb_avail":92768500,"statfs":{"total":95995035648,"available":94994944000,"internally_reserved":0,"allocated":871301120,"data_stored":870857982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":128111368},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2295070720,"data_stored":2295070720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1407586304,"data_stored":1407582232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2977169408,"data_stored":2977169408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":439222272,"data_stored":439222272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":870055936,"data_stored":870055936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1321275392,"data_stored":1321271320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":437518336,"data_stored":437518336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1105657856,"data_stored":1105657856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2460745728,"data_stored":2460745728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2673016832,"data_stored":2673016832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":900464640,"data_stored":900464640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:38:36.336 INFO:tasks.thrashosds.thrasher:Setting 7.9 to [1, 8] 2024-04-29T14:38:36.336 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.9', '1', '8'] 2024-04-29T14:38:36.337 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 7.9 1 8 2024-04-29T14:38:36.509 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:36.616 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:36.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:36.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:36.755 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:37.404 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:37 smithi040 ceph-mon[39581]: pgmap v1852: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 15489/213378 objects misplaced (7.259%); 10 MiB/s, 181 objects/s recovering 2024-04-29T14:38:37.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:37 smithi194 ceph-mon[44089]: pgmap v1852: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 15489/213378 objects misplaced (7.259%); 10 MiB/s, 181 objects/s recovering 2024-04-29T14:38:37.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:37 smithi121 ceph-mon[44533]: pgmap v1852: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 15489/213378 objects misplaced (7.259%); 10 MiB/s, 181 objects/s recovering 2024-04-29T14:38:38.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:38 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:38:38.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:38 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:38:38.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:38 smithi194 ceph-mon[44089]: osdmap e718: 12 total, 12 up, 11 in 2024-04-29T14:38:38.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:38 smithi194 ceph-mon[44089]: osdmap e719: 12 total, 12 up, 11 in 2024-04-29T14:38:38.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:38 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:38:38.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:38 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:38:38.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:38 smithi040 ceph-mon[39581]: osdmap e718: 12 total, 12 up, 11 in 2024-04-29T14:38:38.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:38 smithi040 ceph-mon[39581]: osdmap e719: 12 total, 12 up, 11 in 2024-04-29T14:38:38.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:38 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:38:38.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:38 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:38:38.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:38 smithi121 ceph-mon[44533]: osdmap e718: 12 total, 12 up, 11 in 2024-04-29T14:38:38.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:38 smithi121 ceph-mon[44533]: osdmap e719: 12 total, 12 up, 11 in 2024-04-29T14:38:38.741 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:38.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:38.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:38:38.879 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:38.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:39.008 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:39.036 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:39.058 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:39.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:39.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:38:39.195 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:39.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:38:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:39.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:38:39.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:39.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:39 smithi194 ceph-mon[44089]: pgmap v1855: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 13758/213024 objects misplaced (6.458%); 18 MiB/s, 288 objects/s recovering 2024-04-29T14:38:39.350 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:39.389 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:39.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:39.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:38:39.500 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:39 smithi040 ceph-mon[39581]: pgmap v1855: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 13758/213024 objects misplaced (6.458%); 18 MiB/s, 288 objects/s recovering 2024-04-29T14:38:39.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:39 smithi121 ceph-mon[44533]: pgmap v1855: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 13758/213024 objects misplaced (6.458%); 18 MiB/s, 288 objects/s recovering 2024-04-29T14:38:39.526 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:39.545 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:38:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:39.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:38:39.650 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:38:39.677 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:39.696 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:38:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:39.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:38:39.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:38:39.828 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:39.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:38:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:39.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:38:39.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:38:39.952 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:38:39.978 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:39.998 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:38:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:40.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:38:40.102 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:38:40.128 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:40.145 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:38:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:38:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:38:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101557 7.4 7:2877201c:::benchmark_data_smithi194_147200_object101556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e677)", 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.225065+0000", 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 171.91605759699999, 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21204304199999999, 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101557 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.225065+0000", 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.225065+0000", 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.225068+0000", 2024-04-29T14:38:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.506e-06 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227037+0000", 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019693929999999998 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227059+0000", 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1464999999999999e-05 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227073+0000", 2024-04-29T14:38:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.396e-05 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227565+0000", 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00049246999999999995 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227578+0000", 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2788e-05 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.419206+0000", 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19162751 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.421041+0000", 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018351419999999999 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.421160+0000", 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119332 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.434204+0000", 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01304367 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.434238+0000", 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4638000000000003e-05 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.434280+0000", 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1931999999999998e-05 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437108+0000", 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0028272359999999999 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101561 7.4 7:2ce6de55:::benchmark_data_smithi194_147200_object101560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227092+0000", 2024-04-29T14:38:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 171.91403026899999, 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.226205358, 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101561 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227092+0000", 2024-04-29T14:38:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227092+0000", 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227094+0000", 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0040000000000002e-06 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227138+0000", 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3936000000000001e-05 2024-04-29T14:38:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227148+0000", 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0246e-05 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227156+0000", 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5349999999999999e-06 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227619+0000", 2024-04-29T14:38:40.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046307199999999997 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227624+0000", 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7459999999999997e-06 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.427370+0000", 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.199745069 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.427490+0000", 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120734 2024-04-29T14:38:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.427614+0000", 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012385799999999999 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437125+0000", 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0095105610000000007 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.449227+0000", 2024-04-29T14:38:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012102768 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.449307+0000", 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9357999999999995e-05 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.453297+0000", 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039904709999999998 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101578 7.4 7:215cf8ee:::benchmark_data_smithi194_147200_object101577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227165+0000", 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 171.91395717, 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.228836451, 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101578 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227165+0000", 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227165+0000", 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227166+0000", 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0599999999999999e-07 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227204+0000", 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8099999999999998e-05 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227216+0000", 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2306e-05 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227224+0000", 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3610000000000003e-06 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227670+0000", 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044646300000000002 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227680+0000", 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7580000000000004e-06 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.430921+0000", 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20324070999999999 2024-04-29T14:38:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.431061+0000", 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140128 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.431203+0000", 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000142497 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440344+0000", 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091407250000000006 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.453374+0000", 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013029981 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.453443+0000", 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9066999999999994e-05 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.456001+0000", 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025585489999999998 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101598 7.4 7:2ba08c3a:::benchmark_data_smithi194_147200_object101597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227232+0000", 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 171.91389036000001, 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23998618499999999, 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101598 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227232+0000", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227232+0000", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227233+0000", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3399999999999998e-07 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227272+0000", 2024-04-29T14:38:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9266e-05 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227279+0000", 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3880000000000002e-06 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227286+0000", 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1160000000000001e-06 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227714+0000", 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00042732099999999998 2024-04-29T14:38:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227719+0000", 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7940000000000002e-06 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437248+0000", 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20952917200000001 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437389+0000", 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014079199999999999 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437502+0000", 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000113903 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.446652+0000", 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091495410000000006 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.464557+0000", 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017904908000000001 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.464613+0000", 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6239999999999997e-05 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.467218+0000", 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026049100000000002 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101599 7.4 7:2fafdeca:::benchmark_data_smithi194_147200_object101598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227294+0000", 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 171.91382795800001, 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.245211925, 2024-04-29T14:38:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101599 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227294+0000", 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227294+0000", 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227295+0000", 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8899999999999999e-07 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227331+0000", 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6353000000000002e-05 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227338+0000", 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299999999999997e-06 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227345+0000", 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3579999999999999e-06 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227775+0000", 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00043071100000000002 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227784+0000", 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.217e-06 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440383+0000", 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21259881999999999 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440576+0000", 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019309500000000001 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440790+0000", 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000213375 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.446676+0000", 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058865360000000004 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.469672+0000", 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022995439999999999 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.469716+0000", 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4489e-05 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.472506+0000", 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0027899119999999999 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103965 7.4 7:28b010ad:::benchmark_data_smithi194_147200_object103964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.057521+0000", 2024-04-29T14:38:40.178 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 169.08360080099999, 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.083140487999999999, 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103965 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.057521+0000", 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.057521+0000", 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.057522+0000", 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8000000000000004e-07 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.057580+0000", 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7380999999999998e-05 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.057591+0000", 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1589e-05 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.057598+0000", 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1010000000000003e-06 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.080471+0000", 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022872422 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.080667+0000", 2024-04-29T14:38:40.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019625900000000001 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.080883+0000", 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021631600000000001 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.097379+0000", 2024-04-29T14:38:40.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016495803999999999 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.137381+0000", 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040001414999999999 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.137436+0000", 2024-04-29T14:38:40.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5458000000000003e-05 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.140662+0000", 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032258629999999998 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103969 7.4 7:276d9935:::benchmark_data_smithi194_147200_object103968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.063102+0000", 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 169.078020358, 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.081083337000000005, 2024-04-29T14:38:40.185 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103969 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.063102+0000", 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.063102+0000", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.063104+0000", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.733e-06 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.063170+0000", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6276e-05 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.063182+0000", 2024-04-29T14:38:40.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1915e-05 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.063189+0000", 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8589999999999999e-06 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086906+0000", 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023717782 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.087091+0000", 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018429899999999999 2024-04-29T14:38:40.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.087281+0000", 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000190008 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.103354+0000", 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016073227999999998 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.140723+0000", 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037369225999999998 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.140797+0000", 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4239000000000005e-05 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.144185+0000", 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0033877719999999998 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103989 7.4 7:2fdc10c1:::benchmark_data_smithi194_147200_object103988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.085580+0000", 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 169.05554199599999, 2024-04-29T14:38:40.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.089585465000000003, 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103989 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085580+0000", 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085580+0000", 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085583+0000", 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7860000000000001e-06 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085646+0000", 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2711000000000003e-05 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085666+0000", 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9857999999999999e-05 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085675+0000", 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8770000000000005e-06 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.103410+0000", 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017734507 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.103530+0000", 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120442 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.103699+0000", 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016831699999999999 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113090+0000", 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0093912230000000006 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.172174+0000", 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.059084018000000002 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.172242+0000", 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7991999999999995e-05 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.175166+0000", 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029237339999999999 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.195 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103991 7.4 7:24c10593:::benchmark_data_smithi194_147200_object103990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.086386+0000", 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 169.05473598200001, 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091505929999999999, 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103991 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086386+0000", 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086386+0000", 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086388+0000", 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8780000000000001e-06 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086453+0000", 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4550999999999999e-05 2024-04-29T14:38:40.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086464+0000", 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1590000000000001e-05 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086472+0000", 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6059999999999997e-06 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.109466+0000", 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022994297 2024-04-29T14:38:40.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.109617+0000", 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000150507 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.109735+0000", 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118713 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.124089+0000", 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014353922 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.175225+0000", 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051135271000000003 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.175267+0000", 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.282e-05 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.177892+0000", 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026247750000000002 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103995 7.4 7:24fcae4a:::benchmark_data_smithi194_147200_object103994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.088764+0000", 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 169.052358314, 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091943251000000004, 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103995 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088764+0000", 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088764+0000", 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088766+0000", 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1129999999999999e-06 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088828+0000", 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2187000000000006e-05 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088844+0000", 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5364000000000001e-05 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088853+0000", 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3000000000000007e-06 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113122+0000", 2024-04-29T14:38:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024269625999999999 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113287+0000", 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000164814 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113458+0000", 2024-04-29T14:38:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017102 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.124098+0000", 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010639696000000001 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.177962+0000", 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053863992999999999 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.178033+0000", 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.059e-05 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.180707+0000", 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026745480000000001 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103996 7.4 7:2d132abf:::benchmark_data_smithi194_147200_object103995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.091595+0000", 2024-04-29T14:38:40.206 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 169.04952752299999, 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.092339152999999993, 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103996 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091595+0000", 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091595+0000", 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091597+0000", 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3290000000000001e-06 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091670+0000", 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2698000000000001e-05 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091682+0000", 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2441e-05 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091691+0000", 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3820000000000004e-06 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.116708+0000", 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025017158000000001 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.116849+0000", 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140982 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.116957+0000", 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108034 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.137328+0000", 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020371520000000001 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.180800+0000", 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043471744 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.180846+0000", 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5765e-05 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.183934+0000", 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030880999999999999 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:104001 7.4 7:24312fb7:::benchmark_data_smithi194_147200_object104000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.098323+0000", 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 169.04279920900001, 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.092250929999999995, 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 104001 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.213 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098323+0000", 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098323+0000", 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098326+0000", 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7020000000000001e-06 2024-04-29T14:38:40.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098377+0000", 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1025e-05 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098392+0000", 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5248999999999999e-05 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098400+0000", 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0800000000000006e-06 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.120267+0000", 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021867119000000001 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.120459+0000", 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.120638+0000", 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017919400000000001 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.137335+0000", 2024-04-29T14:38:40.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016696414999999999 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.187832+0000", 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050497487000000001 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.187883+0000", 2024-04-29T14:38:40.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1165000000000002e-05 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.190574+0000", 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026904939999999999 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:104054 7.4 7:21dd03d3:::benchmark_data_smithi194_147200_object104053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.181102+0000", 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.96002007300001, 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.079526024000000001, 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 104054 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.181102+0000", 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.181102+0000", 2024-04-29T14:38:40.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.181104+0000", 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.669e-06 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.181171+0000", 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7645999999999996e-05 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.181182+0000", 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0835e-05 2024-04-29T14:38:40.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.181190+0000", 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9650000000000008e-06 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.211944+0000", 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030754027999999999 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.212062+0000", 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117596 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.212162+0000", 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100608 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.232504+0000", 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020341472999999999 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.257327+0000", 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024823332 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.257389+0000", 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1804999999999995e-05 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.260628+0000", 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003239067 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105142 7.4 7:208edaec:::benchmark_data_smithi194_147200_object105141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.403482+0000", 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.73764027600001, 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.079485074000000003, 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105142 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.225 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.403482+0000", 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.403482+0000", 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.403484+0000", 2024-04-29T14:38:40.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.64e-06 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.403542+0000", 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8879999999999999e-05 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.403554+0000", 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1990999999999999e-05 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.403563+0000", 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1110000000000008e-06 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.425505+0000", 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021942920000000001 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.425675+0000", 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016968699999999999 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.425852+0000", 2024-04-29T14:38:40.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017647299999999999 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.438645+0000", 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012793023000000001 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.479289+0000", 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040644395999999999 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.479364+0000", 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5209000000000004e-05 2024-04-29T14:38:40.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.482967+0000", 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036027440000000002 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105351 7.4 7:2a103c1d:::benchmark_data_smithi194_147200_object105350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.634257+0000", 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.50686514099999, 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.082515811999999994, 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.230 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105351 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634257+0000", 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634257+0000", 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634259+0000", 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9309999999999998e-06 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634306+0000", 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7160999999999998e-05 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634323+0000", 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6484e-05 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634331+0000", 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5930000000000006e-06 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.640107+0000", 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057753090000000002 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.640327+0000", 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022013800000000001 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.640594+0000", 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026735000000000002 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.670105+0000", 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029510628000000001 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.713270+0000", 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043165716999999999 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.713326+0000", 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5760000000000001e-05 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.716773+0000", 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0034467410000000001 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105352 7.4 7:27b86f5f:::benchmark_data_smithi194_147200_object105351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.634352+0000", 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.50677052699999, 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.085763735999999993, 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.235 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105352 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634352+0000", 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634352+0000", 2024-04-29T14:38:40.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634353+0000", 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0300000000000001e-06 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634390+0000", 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7567000000000001e-05 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634401+0000", 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370999999999999e-05 2024-04-29T14:38:40.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.634413+0000", 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2553e-05 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.644146+0000", 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0097329510000000001 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.644360+0000", 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021369600000000001 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.644571+0000", 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021161500000000001 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.673484+0000", 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028912304999999999 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.716832+0000", 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043348486999999998 2024-04-29T14:38:40.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.716888+0000", 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5322999999999999e-05 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.720115+0000", 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032278379999999998 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105442 7.4 7:2acf0821:::benchmark_data_smithi194_147200_object105441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.742407+0000", 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.39871557500001, 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.092712567999999995, 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.241 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105442 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742407+0000", 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742407+0000", 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742409+0000", 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0829999999999998e-06 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742454+0000", 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5633999999999997e-05 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742465+0000", 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0669000000000001e-05 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742473+0000", 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9489999999999996e-06 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.765614+0000", 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023141199000000001 2024-04-29T14:38:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.765795+0000", 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018102299999999999 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.765958+0000", 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016311900000000001 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804981+0000", 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039023059999999998 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.832270+0000", 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027288501999999999 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.832336+0000", 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6462000000000002e-05 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.835119+0000", 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002782868 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105444 7.4 7:26a44d0b:::benchmark_data_smithi194_147200_object105443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.742770+0000", 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.39835193100001, 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.095570181000000004, 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.248 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105444 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742770+0000", 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742770+0000", 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742772+0000", 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2589999999999999e-06 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742835+0000", 2024-04-29T14:38:40.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3095000000000005e-05 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742843+0000", 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4810000000000006e-06 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742850+0000", 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.596e-06 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.769038+0000", 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026188526 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.769160+0000", 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012136 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.769258+0000", 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8357999999999996e-05 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804988+0000", 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035730383999999997 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.835202+0000", 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030213256000000001 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.835256+0000", 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3967e-05 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.838340+0000", 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030848989999999999 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.253 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105451 7.4 7:2191b480:::benchmark_data_smithi194_147200_object105450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.755824+0000", 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.385297788, 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.088790426000000006, 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105451 2024-04-29T14:38:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755824+0000", 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755824+0000", 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755828+0000", 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5470000000000001e-06 2024-04-29T14:38:40.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755880+0000", 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2132999999999997e-05 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755898+0000", 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7810000000000001e-05 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755908+0000", 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0540000000000001e-05 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.772946+0000", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01703795 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.773108+0000", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016127000000000001 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.773277+0000", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016983699999999999 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804992+0000", 2024-04-29T14:38:40.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031714786000000002 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.841925+0000", 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036932765999999999 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.841971+0000", 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6010000000000002e-05 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.844615+0000", 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026437769999999999 2024-04-29T14:38:40.258 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105455 7.4 7:252782f5:::benchmark_data_smithi194_147200_object105454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.761216+0000", 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.37990668099999, 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.085881415000000003, 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105455 2024-04-29T14:38:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761216+0000", 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761216+0000", 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761218+0000", 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.429e-06 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761256+0000", 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7985999999999998e-05 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761268+0000", 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2187999999999999e-05 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761277+0000", 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4980000000000006e-06 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.776722+0000", 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015445468 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.776880+0000", 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000158247 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.777022+0000", 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014190700000000001 2024-04-29T14:38:40.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804996+0000", 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027973591999999999 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.844665+0000", 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039669641999999998 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.844706+0000", 2024-04-29T14:38:40.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0064000000000001e-05 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.847097+0000", 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023913939999999998 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:38:40.264 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:38:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:40.265 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:38:40.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:38:40.366 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:40.392 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:40.411 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:40.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:40.423 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:38:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:40.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:38:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:38:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:40.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:38:40.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:38:40.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:40 smithi121 ceph-mon[44533]: osdmap e720: 12 total, 12 up, 11 in 2024-04-29T14:38:40.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:40 smithi121 ceph-mon[44533]: pgmap v1857: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 13758/212800 objects misplaced (6.465%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:38:40.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2054201511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:38:40.693 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:40.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:38:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:40.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:38:40.739 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:40.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:40.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:40 smithi194 ceph-mon[44089]: osdmap e720: 12 total, 12 up, 11 in 2024-04-29T14:38:40.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:40 smithi194 ceph-mon[44089]: pgmap v1857: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 13758/212800 objects misplaced (6.465%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:38:40.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2054201511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:38:40.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:40 smithi040 ceph-mon[39581]: osdmap e720: 12 total, 12 up, 11 in 2024-04-29T14:38:40.842 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:40 smithi040 ceph-mon[39581]: pgmap v1857: 17 pgs: 2 active+recovering+remapped, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 13758/212800 objects misplaced (6.465%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:38:40.842 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2054201511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:38:40.845 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:41.044 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:41.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:41.148 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:41.174 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:41.193 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:41.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:41.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:38:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:41.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:38:41.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:38:41.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:41 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:38:41.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2054201511' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:38:41.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:41 smithi194 ceph-mon[44089]: osdmap e721: 12 total, 12 up, 11 in 2024-04-29T14:38:41.472 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:41.492 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:38:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:41.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:38:41.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:41.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:41 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:38:41.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2054201511' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:38:41.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:41 smithi040 ceph-mon[39581]: osdmap e721: 12 total, 12 up, 11 in 2024-04-29T14:38:41.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:41 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:38:41.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2054201511' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:38:41.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:41 smithi121 ceph-mon[44533]: osdmap e721: 12 total, 12 up, 11 in 2024-04-29T14:38:41.635 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:42.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:42.196 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:38:42.222 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:42.241 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:38:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:42.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:38:42.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:38:42.374 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:42.393 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:38:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:42.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:38:42.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:42.529 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:42.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:42 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:38:42.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:42 smithi194 ceph-mon[44089]: 7.3 scrub starts 2024-04-29T14:38:42.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:42 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:38:42.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:42 smithi194 ceph-mon[44089]: 7.9 scrub starts 2024-04-29T14:38:42.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:42 smithi194 ceph-mon[44089]: 7.9 scrub ok 2024-04-29T14:38:42.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:42 smithi194 ceph-mon[44089]: pgmap v1859: 17 pgs: 2 peering, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 12433/212514 objects misplaced (5.850%); 16 MiB/s, 289 objects/s recovering 2024-04-29T14:38:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:42 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:38:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:42 smithi121 ceph-mon[44533]: 7.3 scrub starts 2024-04-29T14:38:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:42 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:38:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:42 smithi121 ceph-mon[44533]: 7.9 scrub starts 2024-04-29T14:38:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:42 smithi121 ceph-mon[44533]: 7.9 scrub ok 2024-04-29T14:38:42.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:42 smithi121 ceph-mon[44533]: pgmap v1859: 17 pgs: 2 peering, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 12433/212514 objects misplaced (5.850%); 16 MiB/s, 289 objects/s recovering 2024-04-29T14:38:42.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:42 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:38:42.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:42 smithi040 ceph-mon[39581]: 7.3 scrub starts 2024-04-29T14:38:42.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:42 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:38:42.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:42 smithi040 ceph-mon[39581]: 7.9 scrub starts 2024-04-29T14:38:42.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:42 smithi040 ceph-mon[39581]: 7.9 scrub ok 2024-04-29T14:38:42.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:42 smithi040 ceph-mon[39581]: pgmap v1859: 17 pgs: 2 peering, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 12433/212514 objects misplaced (5.850%); 16 MiB/s, 289 objects/s recovering 2024-04-29T14:38:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:42.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:42.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:38:42.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:43.020 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:43.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:43 smithi040 ceph-mon[39581]: 7.3 scrub ok 2024-04-29T14:38:43.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:43 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:38:43.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:43 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:38:43.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:38:43.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:43 smithi194 ceph-mon[44089]: 7.3 scrub ok 2024-04-29T14:38:43.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:43 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:38:43.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:43 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:38:43.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:43 smithi121 ceph-mon[44533]: 7.3 scrub ok 2024-04-29T14:38:43.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:43 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:38:43.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:43 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:38:44.179 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:38:44.203 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:38:44.241 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:38:44.262 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:38:44.284 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:38:44.306 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:38:44.332 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:38:44.364 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:38:44.392 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:38:44.417 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:38:44.443 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:38:44.467 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:38:44.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:38:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:38:44.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:44 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:38:44.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:44 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:38:44.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:44 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:38:44.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:44 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:38:44.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:44 smithi040 ceph-mon[39581]: pgmap v1860: 17 pgs: 2 peering, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 12377/212216 objects misplaced (5.832%); 12 MiB/s, 216 objects/s recovering 2024-04-29T14:38:44.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4147011548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:44.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:44 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:38:44.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:44 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:38:44.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:44 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:38:44.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:44 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:38:44.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:44 smithi194 ceph-mon[44089]: pgmap v1860: 17 pgs: 2 peering, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 12377/212216 objects misplaced (5.832%); 12 MiB/s, 216 objects/s recovering 2024-04-29T14:38:44.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4147011548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:44.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:44.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:44 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:38:44.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:44 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:38:44.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:44 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:38:44.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:44 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:38:44.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:44 smithi121 ceph-mon[44533]: pgmap v1860: 17 pgs: 2 peering, 1 active+recovering, 14 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 12377/212216 objects misplaced (5.832%); 12 MiB/s, 216 objects/s recovering 2024-04-29T14:38:44.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4147011548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:44.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:45.013 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:45.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.438 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: 7.1 scrub starts 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.439 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: 7.1 scrub starts 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.440 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:45 smithi194 ceph-mon[44089]: 7.1 scrub ok 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: 7.1 scrub starts 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:45 smithi040 ceph-mon[39581]: 7.1 scrub ok 2024-04-29T14:38:45.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:45.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:45 smithi121 ceph-mon[44533]: 7.1 scrub ok 2024-04-29T14:38:46.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:46 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:38:46.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:46 smithi040 ceph-mon[39581]: 7.c scrub starts 2024-04-29T14:38:46.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:46 smithi040 ceph-mon[39581]: 7.c scrub ok 2024-04-29T14:38:46.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:46 smithi040 ceph-mon[39581]: pgmap v1861: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 12377/211598 objects misplaced (5.849%); 10 MiB/s, 179 objects/s recovering 2024-04-29T14:38:46.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:46 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:38:46.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:46 smithi194 ceph-mon[44089]: 7.c scrub starts 2024-04-29T14:38:46.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:46 smithi194 ceph-mon[44089]: 7.c scrub ok 2024-04-29T14:38:46.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:46 smithi194 ceph-mon[44089]: pgmap v1861: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 12377/211598 objects misplaced (5.849%); 10 MiB/s, 179 objects/s recovering 2024-04-29T14:38:46.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:46 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:38:46.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:46 smithi121 ceph-mon[44533]: 7.c scrub starts 2024-04-29T14:38:46.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:46 smithi121 ceph-mon[44533]: 7.c scrub ok 2024-04-29T14:38:46.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:46 smithi121 ceph-mon[44533]: pgmap v1861: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 12377/211598 objects misplaced (5.849%); 10 MiB/s, 179 objects/s recovering 2024-04-29T14:38:48.268 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:48.271 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:48 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:38:48.271 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:48 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:38:48.271 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:48 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:38:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:48.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:48.373 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:38:48.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:48 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:38:48.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:48 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:38:48.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:48 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:38:48.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:48 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:38:48.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:48 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:38:48.398 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:48 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:38:48.400 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:48.424 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:38:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:48.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:38:48.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:38:48.560 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:48.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:38:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:48.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:38:48.684 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:38:48.711 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:48.732 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:38:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:48.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:38:48.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:38:48.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:38:48.863 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:38:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:48.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:38:48.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:38:49.020 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:49.089 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:38:49.089 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:38:49.106 INFO:teuthology.orchestra.run.smithi040.stderr:set 7.9 pg_upmap mapping to [1,8] 2024-04-29T14:38:49.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:49 smithi121 ceph-mon[44533]: pgmap v1862: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 12377/211366 objects misplaced (5.856%); 9.3 MiB/s, 164 objects/s recovering 2024-04-29T14:38:49.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4149422548' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]: dispatch 2024-04-29T14:38:49.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:49 smithi121 ceph-mon[44533]: from='client.27169 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]: dispatch 2024-04-29T14:38:49.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:49 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:38:49.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:49 smithi040 ceph-mon[39581]: pgmap v1862: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 12377/211366 objects misplaced (5.856%); 9.3 MiB/s, 164 objects/s recovering 2024-04-29T14:38:49.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4149422548' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]: dispatch 2024-04-29T14:38:49.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:49 smithi040 ceph-mon[39581]: from='client.27169 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]: dispatch 2024-04-29T14:38:49.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:49 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:38:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:49 smithi194 ceph-mon[44089]: pgmap v1862: 17 pgs: 1 active+recovering, 16 active+clean; 6.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 12377/211366 objects misplaced (5.856%); 9.3 MiB/s, 164 objects/s recovering 2024-04-29T14:38:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4149422548' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]: dispatch 2024-04-29T14:38:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:49 smithi194 ceph-mon[44089]: from='client.27169 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]: dispatch 2024-04-29T14:38:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:49 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:38:50.081 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:50.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:38:50.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:50.213 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:50.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:50 smithi040 ceph-mon[39581]: from='client.27169 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]': finished 2024-04-29T14:38:50.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:50 smithi040 ceph-mon[39581]: osdmap e722: 12 total, 12 up, 11 in 2024-04-29T14:38:50.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2951424970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:50.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:50 smithi121 ceph-mon[44533]: from='client.27169 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]': finished 2024-04-29T14:38:50.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:50 smithi121 ceph-mon[44533]: osdmap e722: 12 total, 12 up, 11 in 2024-04-29T14:38:50.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2951424970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:50.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:50 smithi194 ceph-mon[44089]: from='client.27169 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.9", "id": [1, 8]}]': finished 2024-04-29T14:38:50.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:50 smithi194 ceph-mon[44089]: osdmap e722: 12 total, 12 up, 11 in 2024-04-29T14:38:50.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2951424970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:38:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:51 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:38:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:51 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:38:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2951424970' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:38:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:51 smithi040 ceph-mon[39581]: osdmap e723: 12 total, 12 up, 11 in 2024-04-29T14:38:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:51 smithi040 ceph-mon[39581]: pgmap v1865: 17 pgs: 1 active+recovering, 16 active+clean; 6.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 12319/210868 objects misplaced (5.842%); 456 KiB/s, 7 objects/s recovering 2024-04-29T14:38:51.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:51 smithi040 ceph-mon[39581]: osdmap e724: 12 total, 12 up, 11 in 2024-04-29T14:38:51.264 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:51.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:51.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:38:51.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:38:51.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:51 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:38:51.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:51 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:38:51.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2951424970' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:38:51.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:51 smithi121 ceph-mon[44533]: osdmap e723: 12 total, 12 up, 11 in 2024-04-29T14:38:51.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:51 smithi121 ceph-mon[44533]: pgmap v1865: 17 pgs: 1 active+recovering, 16 active+clean; 6.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 12319/210868 objects misplaced (5.842%); 456 KiB/s, 7 objects/s recovering 2024-04-29T14:38:51.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:51 smithi121 ceph-mon[44533]: osdmap e724: 12 total, 12 up, 11 in 2024-04-29T14:38:51.427 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:51.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:51 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:38:51.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:51 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:38:51.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2951424970' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:38:51.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:51 smithi194 ceph-mon[44089]: osdmap e723: 12 total, 12 up, 11 in 2024-04-29T14:38:51.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:51 smithi194 ceph-mon[44089]: pgmap v1865: 17 pgs: 1 active+recovering, 16 active+clean; 6.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 12319/210868 objects misplaced (5.842%); 456 KiB/s, 7 objects/s recovering 2024-04-29T14:38:51.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:51 smithi194 ceph-mon[44089]: osdmap e724: 12 total, 12 up, 11 in 2024-04-29T14:38:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:51.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:52.023 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:52.049 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:52.069 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:52.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:52.168 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:52 smithi040 ceph-mon[39581]: 7.d scrub starts 2024-04-29T14:38:52.168 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:52 smithi040 ceph-mon[39581]: 7.d scrub ok 2024-04-29T14:38:52.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:38:52.199 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:52.218 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:38:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:52.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:38:52.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:52 smithi194 ceph-mon[44089]: 7.d scrub starts 2024-04-29T14:38:52.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:52 smithi194 ceph-mon[44089]: 7.d scrub ok 2024-04-29T14:38:52.322 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:38:52.350 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:38:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:52.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:38:52.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:52 smithi121 ceph-mon[44533]: 7.d scrub starts 2024-04-29T14:38:52.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:52 smithi121 ceph-mon[44533]: 7.d scrub ok 2024-04-29T14:38:52.472 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:38:52.497 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:52.516 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:38:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:52.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:38:52.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:52.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:52.765 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:38:52.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:38:52.861 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:52.880 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:38:52.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:52.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:38:52.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:53.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:53.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:38:53.175 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:53.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:38:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:53.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:38:53.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:38:53.326 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:53.345 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:38:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:53.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:38:53.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:53 smithi040 ceph-mon[39581]: pgmap v1867: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 18872/210576 objects misplaced (8.962%); 331 KiB/s, 5 objects/s recovering 2024-04-29T14:38:53.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:53 smithi040 ceph-mon[39581]: osdmap e725: 12 total, 12 up, 11 in 2024-04-29T14:38:53.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:53 smithi121 ceph-mon[44533]: pgmap v1867: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 18872/210576 objects misplaced (8.962%); 331 KiB/s, 5 objects/s recovering 2024-04-29T14:38:53.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:53 smithi121 ceph-mon[44533]: osdmap e725: 12 total, 12 up, 11 in 2024-04-29T14:38:53.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:38:53.476 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:53.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:38:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:53.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:38:53.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:53 smithi194 ceph-mon[44089]: pgmap v1867: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 18872/210576 objects misplaced (8.962%); 331 KiB/s, 5 objects/s recovering 2024-04-29T14:38:53.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:53 smithi194 ceph-mon[44089]: osdmap e725: 12 total, 12 up, 11 in 2024-04-29T14:38:53.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:38:53.626 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:53.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:38:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:53.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:38:53.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:38:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:53.802 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:38:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:53.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:38:53.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:38:53.937 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:53.961 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:38:54.891 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:38:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:54.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:38:55.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:38:55.033 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:55.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:38:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:55.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:38:55.082 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8] out_osds: [3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:38:55.082 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:38:55.082 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:38:55.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:38:55.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:38:55.191 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:55.252 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:55.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:55.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:38:55.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:55 smithi040 ceph-mon[39581]: pgmap v1869: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 18819/210346 objects misplaced (8.947%); 368 KiB/s, 5 objects/s recovering 2024-04-29T14:38:55.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:55 smithi121 ceph-mon[44533]: pgmap v1869: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 18819/210346 objects misplaced (8.947%); 368 KiB/s, 5 objects/s recovering 2024-04-29T14:38:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:55.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:55.553 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:38:55.554 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:38:55.554 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:38:55.554 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:38:55.554 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:38:55.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:55 smithi194 ceph-mon[44089]: pgmap v1869: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 18819/210346 objects misplaced (8.947%); 368 KiB/s, 5 objects/s recovering 2024-04-29T14:38:55.633 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:38:55.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:55.684 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:38:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:55.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:38:55.696 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:38:55.696 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:38:55.696 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:38:55.697 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:38:55.713 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:38:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:38:55.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:38:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:56.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1138168548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:56.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:56 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1138168548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:56.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:56 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1138168548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:38:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:56.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:56 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:57.472 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:38:57.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:57.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:57.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:57 smithi040 ceph-mon[39581]: pgmap v1870: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 18819/209914 objects misplaced (8.965%); 307 KiB/s, 4 objects/s recovering 2024-04-29T14:38:57.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:57 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:38:57.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:57 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:38:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:57.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:38:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:57 smithi194 ceph-mon[44089]: pgmap v1870: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 18819/209914 objects misplaced (8.965%); 307 KiB/s, 4 objects/s recovering 2024-04-29T14:38:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:57 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:38:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:57 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:38:57.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:38:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:38:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:38:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:57 smithi121 ceph-mon[44533]: pgmap v1870: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 18819/209914 objects misplaced (8.965%); 307 KiB/s, 4 objects/s recovering 2024-04-29T14:38:57.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:57 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:38:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:57 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:38:58.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:58 smithi194 ceph-mon[44089]: osdmap e726: 12 total, 12 up, 11 in 2024-04-29T14:38:58.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:58 smithi194 ceph-mon[44089]: osdmap e727: 12 total, 12 up, 11 in 2024-04-29T14:38:58.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:58 smithi040 ceph-mon[39581]: osdmap e726: 12 total, 12 up, 11 in 2024-04-29T14:38:58.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:58 smithi040 ceph-mon[39581]: osdmap e727: 12 total, 12 up, 11 in 2024-04-29T14:38:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:58 smithi121 ceph-mon[44533]: osdmap e726: 12 total, 12 up, 11 in 2024-04-29T14:38:58.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:58 smithi121 ceph-mon[44533]: osdmap e727: 12 total, 12 up, 11 in 2024-04-29T14:38:59.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:38:59 smithi194 ceph-mon[44089]: pgmap v1873: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 18598/209668 objects misplaced (8.870%); 2.6 MiB/s, 41 objects/s recovering 2024-04-29T14:38:59.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:38:59 smithi040 ceph-mon[39581]: pgmap v1873: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 18598/209668 objects misplaced (8.870%); 2.6 MiB/s, 41 objects/s recovering 2024-04-29T14:38:59.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:38:59 smithi121 ceph-mon[44533]: pgmap v1873: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 18598/209668 objects misplaced (8.870%); 2.6 MiB/s, 41 objects/s recovering 2024-04-29T14:38:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:38:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:38:59.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:38:59.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:00.014 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:00.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:00.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:00.168 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:01.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:01 smithi194 ceph-mon[44089]: pgmap v1874: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 18549/209338 objects misplaced (8.861%); 2.2 MiB/s, 35 objects/s recovering 2024-04-29T14:39:01.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:01 smithi040 ceph-mon[39581]: pgmap v1874: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 18549/209338 objects misplaced (8.861%); 2.2 MiB/s, 35 objects/s recovering 2024-04-29T14:39:01.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:01 smithi121 ceph-mon[44533]: pgmap v1874: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 18549/209338 objects misplaced (8.861%); 2.2 MiB/s, 35 objects/s recovering 2024-04-29T14:39:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:03 smithi194 ceph-mon[44089]: pgmap v1875: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 18299/209030 objects misplaced (8.754%); 3.9 MiB/s, 61 objects/s recovering 2024-04-29T14:39:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:03 smithi194 ceph-mon[44089]: osdmap e728: 12 total, 12 up, 11 in 2024-04-29T14:39:03.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:03 smithi194 ceph-mon[44089]: osdmap e729: 12 total, 12 up, 11 in 2024-04-29T14:39:03.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:03.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:03.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:03 smithi040 ceph-mon[39581]: pgmap v1875: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 18299/209030 objects misplaced (8.754%); 3.9 MiB/s, 61 objects/s recovering 2024-04-29T14:39:03.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:03 smithi040 ceph-mon[39581]: osdmap e728: 12 total, 12 up, 11 in 2024-04-29T14:39:03.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:03 smithi040 ceph-mon[39581]: osdmap e729: 12 total, 12 up, 11 in 2024-04-29T14:39:03.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:03.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:03.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:03 smithi121 ceph-mon[44533]: pgmap v1875: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 18299/209030 objects misplaced (8.754%); 3.9 MiB/s, 61 objects/s recovering 2024-04-29T14:39:03.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:03 smithi121 ceph-mon[44533]: osdmap e728: 12 total, 12 up, 11 in 2024-04-29T14:39:03.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:03 smithi121 ceph-mon[44533]: osdmap e729: 12 total, 12 up, 11 in 2024-04-29T14:39:04.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:04.805 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:04.805 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:04 smithi040 ceph-mon[39581]: osdmap e730: 12 total, 12 up, 11 in 2024-04-29T14:39:04.805 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:04 smithi040 ceph-mon[39581]: pgmap v1879: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 18245/208740 objects misplaced (8.741%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:39:04.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:04.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:04.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:04 smithi194 ceph-mon[44089]: osdmap e730: 12 total, 12 up, 11 in 2024-04-29T14:39:04.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:04 smithi194 ceph-mon[44089]: pgmap v1879: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 18245/208740 objects misplaced (8.741%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:39:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:04 smithi121 ceph-mon[44533]: osdmap e730: 12 total, 12 up, 11 in 2024-04-29T14:39:04.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:04 smithi121 ceph-mon[44533]: pgmap v1879: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 18245/208740 objects misplaced (8.741%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:39:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:04.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:04.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:39:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:05.005 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:05.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:05.084 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:05.290 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:05.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:05.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:05.425 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:05.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:05.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:05.768 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:05.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:05.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:05.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:05.923 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:05.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:06.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:06.078 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:07.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:39:07.131 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:07.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:07.211 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:07.239 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:07.269 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:07.288 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:07.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:07.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:07 smithi040 ceph-mon[39581]: pgmap v1880: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 18245/208272 objects misplaced (8.760%); 2.9 MiB/s, 45 objects/s recovering 2024-04-29T14:39:07.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:07 smithi194 ceph-mon[44089]: pgmap v1880: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 18245/208272 objects misplaced (8.760%); 2.9 MiB/s, 45 objects/s recovering 2024-04-29T14:39:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:07 smithi121 ceph-mon[44533]: pgmap v1880: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 18245/208272 objects misplaced (8.760%); 2.9 MiB/s, 45 objects/s recovering 2024-04-29T14:39:08.813 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:08.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:08 smithi040 ceph-mon[39581]: osdmap e731: 12 total, 12 up, 11 in 2024-04-29T14:39:08.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:08 smithi040 ceph-mon[39581]: pgmap v1882: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 17999/208122 objects misplaced (8.648%); 2.9 MiB/s, 45 objects/s recovering 2024-04-29T14:39:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:08.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:08.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:08 smithi194 ceph-mon[44089]: osdmap e731: 12 total, 12 up, 11 in 2024-04-29T14:39:08.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:08 smithi194 ceph-mon[44089]: pgmap v1882: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 17999/208122 objects misplaced (8.648%); 2.9 MiB/s, 45 objects/s recovering 2024-04-29T14:39:08.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:08 smithi121 ceph-mon[44533]: osdmap e731: 12 total, 12 up, 11 in 2024-04-29T14:39:08.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:08 smithi121 ceph-mon[44533]: pgmap v1882: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 17999/208122 objects misplaced (8.648%); 2.9 MiB/s, 45 objects/s recovering 2024-04-29T14:39:08.918 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:08.946 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:08.966 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:08.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:09.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:39:09.102 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:09.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:39:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:09.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:39:09.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:09.262 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:09 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:09 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:09 smithi040 ceph-mon[39581]: osdmap e732: 12 total, 12 up, 11 in 2024-04-29T14:39:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:09.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:09 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:09.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:09 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:09.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:09 smithi194 ceph-mon[44089]: osdmap e732: 12 total, 12 up, 11 in 2024-04-29T14:39:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:09.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:09.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:09 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:09.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:09 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:09.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:09 smithi121 ceph-mon[44533]: osdmap e732: 12 total, 12 up, 11 in 2024-04-29T14:39:09.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:09.958 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:10.470 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 732, 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:39:08.414369+0000", 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:31:03.499853+0000", 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:37:07.184321+0000", 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:39:10.471 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5799936, 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 30, 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:39:10.472 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 7, 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "luminous", 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.473 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:39:10.474 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:39:10.475 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.476 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "16", 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:39:10.477 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:39:10.478 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:39:10.479 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:39:10.480 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:39:10.481 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 10, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 10, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.20000000298023224, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 7, 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_5", 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:34:47.804723+0000", 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:39:10.483 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:39:10.484 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "659", 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:39:10.485 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:39:10.486 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:39:10.487 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:39:10.488 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.4800000190734863, 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 1.8700000047683716, 2024-04-29T14:39:10.489 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 1.1000000238418579, 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2.7300000190734863, 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2.059999942779541, 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946, 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: "error_message": "pool 7 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:39:10.490 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:39:10.491 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 723, 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.493 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.494 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.495 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.7887725830078125, 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.496 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 722, 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:39:10.497 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:39:10.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:39:10.500 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:39:10.501 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 702, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.502 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.503 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:39:10.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.505 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:39:10.506 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 276, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.507 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:39:10.509 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.510 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.5530548095703125, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 629, 2024-04-29T14:39:10.512 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.513 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.515 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.516 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 388, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 457, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 561, 2024-04-29T14:39:10.517 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 703, 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 459, 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:39:10.520 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/2522198013", 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/2522198013", 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/2522198013", 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/2522198013", 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.95001220703125, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:39:10.522 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 696, 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.523 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.525 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.526 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.5831451416015625, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 634, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.527 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.528 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:39:10.529 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.530 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.531 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 720, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.532 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.533 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.535 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:39:10.536 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 655, 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:39:10.538 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:39:10.540 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 684, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:39:10.542 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.543 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.544 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.4297637939453125, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:39:10.545 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 524, 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.548 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:38:24.244354+0000", 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.550 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:38:23.412079+0000", 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:39:03.024179+0000", 2024-04-29T14:39:10.551 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:39:08.380155+0000", 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:54.024639+0000", 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.553 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:38:23.966415+0000", 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 457 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:56.655099+0000", 2024-04-29T14:39:10.554 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:38:22.413195+0000", 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:39:10.555 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:38:56.257380+0000", 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.556 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:38:37.095230+0000", 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:39:02.117851+0000", 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:37:51.469138+0000", 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [ 2024-04-29T14:39:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "7.9", 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: 8 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [ 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "1.0", 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "mappings": [ 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "from": 9, 2024-04-29T14:39:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "to": 2 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [ 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "7.9", 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: 8 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.560 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:39:10.561 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:39:10.562 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:39:10.563 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:39:10.563 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:39:10.563 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:39:10.563 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:39:10.563 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:10.563 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:39:10.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:10 smithi194 ceph-mon[44089]: pgmap v1884: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 71 op/s; 17951/207912 objects misplaced (8.634%); 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:39:10.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:10 smithi040 ceph-mon[39581]: pgmap v1884: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 71 op/s; 17951/207912 objects misplaced (8.634%); 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:39:10.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:10 smithi121 ceph-mon[44533]: pgmap v1884: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 71 op/s; 17951/207912 objects misplaced (8.634%); 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:39:11.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1271607115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:39:11.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1271607115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:39:11.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1271607115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:39:12.265 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:12.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:12.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:39:12.400 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:12.420 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:39:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:12.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:39:12.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:12.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:12.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:12 smithi040 ceph-mon[39581]: pgmap v1885: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 17692/207714 objects misplaced (8.517%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:12.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:12 smithi040 ceph-mon[39581]: osdmap e733: 12 total, 12 up, 11 in 2024-04-29T14:39:12.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:12 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:39:12.672 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:12.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:12 smithi194 ceph-mon[44089]: pgmap v1885: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 17692/207714 objects misplaced (8.517%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:12.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:12 smithi194 ceph-mon[44089]: osdmap e733: 12 total, 12 up, 11 in 2024-04-29T14:39:12.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:12 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:39:12.699 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:12.720 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:12.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:12.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:12.859 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:12.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:12 smithi121 ceph-mon[44533]: pgmap v1885: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 17692/207714 objects misplaced (8.517%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:12.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:12 smithi121 ceph-mon[44533]: osdmap e733: 12 total, 12 up, 11 in 2024-04-29T14:39:12.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:12 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:39:13.222 INFO:tasks.thrashosds.thrasher:j is {'epoch': 732, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:39:08.414369+0000', 'last_up_change': '2024-04-29T14:31:03.499853+0000', 'last_in_change': '2024-04-29T14:37:07.184321+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-04-29T14:34:47.804723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '659', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 1.8700000047683716, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995, 'error_message': 'pool 7 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 723, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 1, 'weight': 0.7887725830078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 722, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 702, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 276, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 0.5530548095703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 388, 'last_clean_end': 457, 'up_from': 561, 'up_thru': 703, 'down_at': 459, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 2522198013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2522198013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2522198013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2522198013}]}, 'public_addr': '172.21.15.121:6804/2522198013', 'cluster_addr': '172.21.15.121:6805/2522198013', 'heartbeat_back_addr': '172.21.15.121:6807/2522198013', 'heartbeat_front_addr': '172.21.15.121:6806/2522198013', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 696, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 0.5831451416015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 634, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 720, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 655, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 1, 'weight': 0.4297637939453125, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 524, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:38:24.244354+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:38:23.412079+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:39:03.024179+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:39:08.380155+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:54.024639+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:38:23.966415+0000', 'dead_epoch': 457}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:56.655099+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:38:22.413195+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:38:56.257380+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:38:37.095230+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:39:02.117851+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:37:51.469138+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '7.9', 'osds': [1, 8]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 9, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.9', 'osds': [0, 8]}], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:39:13.222 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:39:13.678 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:13.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:13.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:13.819 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:14.432 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:14.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:14.539 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:14.566 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:14.586 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:14.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:14.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:14.724 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:14.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:14.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:14.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:14.912 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:14.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:15.027 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:15.055 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:15.075 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:15.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:15.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:39:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:39:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:15.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:39:15.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:15 smithi194 ceph-mon[44089]: pgmap v1887: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 60 op/s; 17641/207412 objects misplaced (8.505%); 2.8 MiB/s, 45 objects/s recovering 2024-04-29T14:39:15.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:15.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:15.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:39:15.356 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:15.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:39:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:15.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:39:15.394 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:15.401 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:15.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:15.502 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:15 smithi121 ceph-mon[44533]: pgmap v1887: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 60 op/s; 17641/207412 objects misplaced (8.505%); 2.8 MiB/s, 45 objects/s recovering 2024-04-29T14:39:15.502 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:15.502 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:15.504 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:15.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:15.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:15 smithi040 ceph-mon[39581]: pgmap v1887: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 60 op/s; 17641/207412 objects misplaced (8.505%); 2.8 MiB/s, 45 objects/s recovering 2024-04-29T14:39:15.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:15.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:15.627 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:15.655 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:15.674 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:15.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:15.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:15.811 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:16.321 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:39:16.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3612044726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:39:16.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:16 smithi194 ceph-mon[44089]: from='client.27193 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:39:16.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:39:16.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:39:16.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:16.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3612044726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:39:16.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:16 smithi040 ceph-mon[39581]: from='client.27193 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:39:16.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:39:16.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:39:16.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:16.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3612044726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:39:16.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:16 smithi121 ceph-mon[44533]: from='client.27193 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:39:16.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:39:16.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:39:16.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:39:17.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:17 smithi194 ceph-mon[44089]: pgmap v1888: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 73 op/s; 17641/206946 objects misplaced (8.524%); 2.4 MiB/s, 38 objects/s recovering 2024-04-29T14:39:17.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:17 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:17.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:17 smithi194 ceph-mon[44089]: from='client.27193 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:39:17.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:17 smithi194 ceph-mon[44089]: osdmap e734: 12 total, 12 up, 11 in 2024-04-29T14:39:17.592 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:17 smithi040 ceph-mon[39581]: pgmap v1888: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 73 op/s; 17641/206946 objects misplaced (8.524%); 2.4 MiB/s, 38 objects/s recovering 2024-04-29T14:39:17.592 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:17 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:17.592 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:17 smithi040 ceph-mon[39581]: from='client.27193 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:39:17.592 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:17 smithi040 ceph-mon[39581]: osdmap e734: 12 total, 12 up, 11 in 2024-04-29T14:39:17.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:17 smithi121 ceph-mon[44533]: pgmap v1888: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 73 op/s; 17641/206946 objects misplaced (8.524%); 2.4 MiB/s, 38 objects/s recovering 2024-04-29T14:39:17.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:17 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:17.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:17 smithi121 ceph-mon[44533]: from='client.27193 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:39:17.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:17 smithi121 ceph-mon[44533]: osdmap e734: 12 total, 12 up, 11 in 2024-04-29T14:39:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:18.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:18.183 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:39:18.210 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:39:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:18.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:39:18.333 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:18.361 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:18.381 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:18.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:18.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:18.515 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:18.535 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:18.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:18.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:18 smithi194 ceph-mon[44089]: osdmap e735: 12 total, 12 up, 11 in 2024-04-29T14:39:18.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:18 smithi194 ceph-mon[44089]: pgmap v1891: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 17385/206732 objects misplaced (8.409%); 2.9 MiB/s, 46 objects/s recovering 2024-04-29T14:39:18.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:18.664 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:18 smithi121 ceph-mon[44533]: osdmap e735: 12 total, 12 up, 11 in 2024-04-29T14:39:18.664 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:18 smithi121 ceph-mon[44533]: pgmap v1891: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 17385/206732 objects misplaced (8.409%); 2.9 MiB/s, 46 objects/s recovering 2024-04-29T14:39:18.666 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:18.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:18.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:18.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:18.816 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:18.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:18.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:18.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:18 smithi040 ceph-mon[39581]: osdmap e735: 12 total, 12 up, 11 in 2024-04-29T14:39:18.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:18 smithi040 ceph-mon[39581]: pgmap v1891: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 17385/206732 objects misplaced (8.409%); 2.9 MiB/s, 46 objects/s recovering 2024-04-29T14:39:18.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:18.976 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:19.241 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:19.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:19.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:19.376 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:19.394 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:19.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:19.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:19.527 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:19.547 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:19.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:19.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:19.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:19.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:19.828 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:19.847 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:19.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:19.951 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:39:19.978 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:19.996 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:39:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:20.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:39:20.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:20.147 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:20.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:20.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:20.277 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:20.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:20.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:20.399 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:20.426 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:20.446 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:20.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:20.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:20.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:20.577 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:20.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:20.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:20.700 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:20.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:20.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:21.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:21 smithi040 ceph-mon[39581]: pgmap v1892: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 86 op/s; 17337/206380 objects misplaced (8.401%); 2.4 MiB/s, 39 objects/s recovering 2024-04-29T14:39:21.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:21.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:21.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:21 smithi040 ceph-mon[39581]: osdmap e736: 12 total, 12 up, 11 in 2024-04-29T14:39:21.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:21 smithi194 ceph-mon[44089]: pgmap v1892: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 86 op/s; 17337/206380 objects misplaced (8.401%); 2.4 MiB/s, 39 objects/s recovering 2024-04-29T14:39:21.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:21.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:21.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:21 smithi194 ceph-mon[44089]: osdmap e736: 12 total, 12 up, 11 in 2024-04-29T14:39:21.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:21 smithi121 ceph-mon[44533]: pgmap v1892: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 86 op/s; 17337/206380 objects misplaced (8.401%); 2.4 MiB/s, 39 objects/s recovering 2024-04-29T14:39:21.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:21.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:21.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:21 smithi121 ceph-mon[44533]: osdmap e736: 12 total, 12 up, 11 in 2024-04-29T14:39:22.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:22.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:22.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:22 smithi194 ceph-mon[44089]: osdmap e737: 12 total, 12 up, 11 in 2024-04-29T14:39:22.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:22.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:22.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:22 smithi040 ceph-mon[39581]: osdmap e737: 12 total, 12 up, 11 in 2024-04-29T14:39:22.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:22.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:22.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:22 smithi121 ceph-mon[44533]: osdmap e737: 12 total, 12 up, 11 in 2024-04-29T14:39:23.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:23 smithi194 ceph-mon[44089]: pgmap v1894: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 17069/205944 objects misplaced (8.288%); 5.7 MiB/s, 91 objects/s recovering 2024-04-29T14:39:23.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:23 smithi040 ceph-mon[39581]: pgmap v1894: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 17069/205944 objects misplaced (8.288%); 5.7 MiB/s, 91 objects/s recovering 2024-04-29T14:39:23.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:23 smithi121 ceph-mon[44533]: pgmap v1894: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 17069/205944 objects misplaced (8.288%); 5.7 MiB/s, 91 objects/s recovering 2024-04-29T14:39:23.837 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:23.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:23.949 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:24.028 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:39:24.028 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:117990 7.b 7:de64f127:::benchmark_data_smithi194_147200_object3078:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e704)", 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:29.105154+0000", 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 111.54391156299999, 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.267556355, 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 117990 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.029 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105154+0000", 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105154+0000", 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105156+0000", 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4499999999999998e-06 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105176+0000", 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0128000000000001e-05 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105192+0000", 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.575e-05 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105203+0000", 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.13e-05 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:30.951863+0000", 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8466592209999999 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:30.963084+0000", 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011221139999999999 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:30.991007+0000", 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027923228000000001 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.158557+0000", 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.167550486 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.313582+0000", 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.155024103 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.326972+0000", 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013390465000000001 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.372710+0000", 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045738083999999998 2024-04-29T14:39:24.033 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:118814 7.b 7:d91c083d:::benchmark_data_smithi194_147200_object3902:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e705)", 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:39.999367+0000", 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.64969848, 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2752686199999999, 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.034 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 118814 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999367+0000", 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999367+0000", 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999369+0000", 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5670000000000002e-06 2024-04-29T14:39:24.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999376+0000", 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4350000000000002e-06 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999388+0000", 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1773999999999999e-05 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999398+0000", 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9809999999999998e-06 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.081094+0000", 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0816960959999999 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.095508+0000", 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014414089 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.113227+0000", 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017718854999999999 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.230362+0000", 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117135268 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.235821+0000", 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054593080000000004 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.242421+0000", 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065999220000000003 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.274635+0000", 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032214325000000002 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.b 706/685 [PushOp(7:db144375:::benchmark_data_smithi194_147200_object7387:head, version: 660'447, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:db144375:::benchmark_data_smithi194_147200_object7387:head@660'447, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:42.076484+0000", 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 98.572581229999997, 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0814666800000001, 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.076484+0000", 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.076484+0000", 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.076485+0000", 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0589999999999999e-06 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.078894+0000", 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024089049999999998 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.078919+0000", 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5150000000000001e-05 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.078936+0000", 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6759e-05 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.092967+0000", 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0140306480000003 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.094795+0000", 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001828742 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.157951+0000", 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.063155417000000005 2024-04-29T14:39:24.042 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:118994 7.b 7:d9c86171:::benchmark_data_smithi194_147200_object4082:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e706)", 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:44.016228+0000", 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 96.632837043999999, 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.26731229, 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.043 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 118994 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016228+0000", 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016228+0000", 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016230+0000", 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1569999999999998e-06 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016236+0000", 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9460000000000003e-06 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016250+0000", 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3439e-05 2024-04-29T14:39:24.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016260+0000", 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0132000000000001e-05 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.144175+0000", 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1279146870000001 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.163477+0000", 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019302875000000001 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.201695+0000", 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038217802000000002 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.231766+0000", 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030071123000000002 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.236716+0000", 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049493640000000004 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.249555+0000", 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012838887 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.283541+0000", 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033985877999999997 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119001 7.b 7:dabe6e1c:::benchmark_data_smithi194_147200_object4089:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e706)", 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:44.278109+0000", 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 96.370956276000001, 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3018859850000002, 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119001 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.049 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278109+0000", 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278109+0000", 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278112+0000", 2024-04-29T14:39:24.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8700000000000001e-06 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278117+0000", 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9749999999999998e-06 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278129+0000", 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2351000000000001e-05 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278138+0000", 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6130000000000004e-06 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.298009+0000", 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.019871331 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.315737+0000", 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017727837 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.347200+0000", 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031463158999999997 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.446480+0000", 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099279661000000005 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.537923+0000", 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091443128999999998 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.546321+0000", 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0083982280000000006 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.579995+0000", 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033673831000000001 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119012 7.b 7:d129b0f8:::benchmark_data_smithi194_147200_object4100:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e706)", 2024-04-29T14:39:24.054 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:44.364840+0000", 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 96.284224940000001, 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.272131312, 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119012 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364840+0000", 2024-04-29T14:39:24.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364840+0000", 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364843+0000", 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4870000000000001e-06 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364849+0000", 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.409e-06 2024-04-29T14:39:24.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364864+0000", 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4358999999999999e-05 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364874+0000", 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0329999999999999e-05 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.367558+0000", 2024-04-29T14:39:24.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0026840539999999 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.389974+0000", 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022416044 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.423904+0000", 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033930035999999997 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.454036+0000", 2024-04-29T14:39:24.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030132246000000001 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.592166+0000", 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.138129746 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.601343+0000", 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091768779999999994 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.636972+0000", 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035628723000000001 2024-04-29T14:39:24.059 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.b 706/685 [PushOp(7:df05b564:::benchmark_data_smithi194_147200_object7394:head, version: 660'448, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:df05b564:::benchmark_data_smithi194_147200_object7394:head@660'448, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:47.217991+0000", 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.431074021000001, 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5504673069999999, 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.217991+0000", 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.217991+0000", 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.217993+0000", 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9039999999999999e-06 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.218037+0000", 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3708000000000002e-05 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.218068+0000", 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1226999999999998e-05 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.218087+0000", 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9157999999999999e-05 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.695381+0000", 2024-04-29T14:39:24.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4772933449999996 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.696681+0000", 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0013008150000000001 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.768459+0000", 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.071777149999999998 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119255 7.b 7:d1e0d635:::benchmark_data_smithi194_147200_object4343:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:49.935746+0000", 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 90.713319530000007, 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3167341920000002, 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119255 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935746+0000", 2024-04-29T14:39:24.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935746+0000", 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935748+0000", 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2840000000000001e-06 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935754+0000", 2024-04-29T14:39:24.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.268e-06 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935767+0000", 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2537999999999999e-05 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935780+0000", 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2989e-05 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.985549+0000", 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.049769038 2024-04-29T14:39:24.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.002198+0000", 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016649489999999999 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.027095+0000", 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024896965 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.057472+0000", 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030376844 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.199007+0000", 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.14153477 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.210572+0000", 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011565351999999999 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.252480+0000", 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041907654000000003 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_backoff(7.b ack-block id 13 [7:de377bca:::benchmark_data_smithi194_147200_object4372:head,7:de377bca:::benchmark_data_smithi194_147200_object4372:head) e707)", 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.173628+0000", 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 90.475437317000001, 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.313792522, 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 0 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173628+0000", 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173628+0000", 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173630+0000", 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0339999999999999e-06 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173636+0000", 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9989999999999997e-06 2024-04-29T14:39:24.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173648+0000", 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1668000000000001e-05 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173657+0000", 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3030000000000005e-06 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.483091+0000", 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3094342719999998 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.487420+0000", 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043292460000000001 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_backoff(7.b ack-block id 14 [7:d521d79c:::benchmark_data_smithi194_147200_object4393:head,7:d521d79c:::benchmark_data_smithi194_147200_object4393:head) e707)", 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.226908+0000", 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 90.422157694999996, 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2674437890000001, 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.072 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 0 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226908+0000", 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226908+0000", 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226911+0000", 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1209999999999998e-06 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226919+0000", 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7329999999999992e-06 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226936+0000", 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6331e-05 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226948+0000", 2024-04-29T14:39:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2583e-05 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.489526+0000", 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2625775940000001 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.494351+0000", 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048254270000000002 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119284 7.b 7:de377bca:::benchmark_data_smithi194_147200_object4372:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:39:24.075 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.366292+0000", 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 90.282773602999995, 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3351182779999999, 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119284 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366292+0000", 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366292+0000", 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366294+0000", 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.526e-06 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366300+0000", 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2179999999999996e-06 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366314+0000", 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.362e-05 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366327+0000", 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2935999999999999e-05 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.496304+0000", 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1299769460000002 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.515416+0000", 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019111624000000001 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.547443+0000", 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032027003999999998 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.630161+0000", 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.082718749999999994 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.637612+0000", 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0074507410000000003 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.652514+0000", 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014901456 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.701410+0000", 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.048896456999999997 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119305 7.b 7:d521d79c:::benchmark_data_smithi194_147200_object4393:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.452694+0000", 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 90.196371169000003, 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3006495139999998, 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119305 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452694+0000", 2024-04-29T14:39:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452694+0000", 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452697+0000", 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2550000000000002e-06 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452707+0000", 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2979999999999997e-06 2024-04-29T14:39:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452721+0000", 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4333e-05 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452733+0000", 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2201999999999999e-05 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.564745+0000", 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1120117120000002 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.580509+0000", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015763762000000001 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.611657+0000", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031148216999999999 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.706129+0000", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.094471609999999998 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.708457+0000", 2024-04-29T14:39:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023288789999999998 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.718029+0000", 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0095712799999999997 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.753344+0000", 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035314966000000003 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119327 7.b 7:d9b2aa38:::benchmark_data_smithi194_147200_object4415:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:39:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.720028+0000", 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 89.929037213000001, 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4320200239999998, 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119327 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720028+0000", 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720028+0000", 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720030+0000", 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3470000000000001e-06 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720042+0000", 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1753000000000001e-05 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720056+0000", 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3743e-05 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720066+0000", 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0532e-05 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.812713+0000", 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.092646647 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.829361+0000", 2024-04-29T14:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016648185999999999 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.856838+0000", 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027476295000000001 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.013113+0000", 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.156275108 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.098238+0000", 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.085125485000000001 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.110242+0000", 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012003443000000001 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.152048+0000", 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041806484999999997 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_backoff(7.b ack-block id 16 [7:d0fb8c6f:::benchmark_data_smithi194_147200_object4426:head,7:d0fb8c6f:::benchmark_data_smithi194_147200_object4426:head) e707)", 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.871369+0000", 2024-04-29T14:39:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 89.777696359999993, 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.285846764, 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 0 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871369+0000", 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871369+0000", 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871371+0000", 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2500000000000001e-06 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871378+0000", 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5880000000000003e-06 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871389+0000", 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1100999999999999e-05 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871398+0000", 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4520000000000003e-06 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.154010+0000", 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2826120670000001 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.157216+0000", 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032053060000000002 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119369 7.b 7:dbd47a03:::benchmark_data_smithi194_147200_object4457:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:53.599267+0000", 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 87.049798151000005, 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2839892009999998, 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.094 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119369 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599267+0000", 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599267+0000", 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599270+0000", 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599280+0000", 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0003000000000001e-05 2024-04-29T14:39:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599295+0000", 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5602000000000001e-05 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599307+0000", 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1816e-05 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.630256+0000", 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.030948413 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.647832+0000", 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017576590999999999 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.672860+0000", 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025027997 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.774087+0000", 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10122699 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.845211+0000", 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.071124110000000004 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.850670+0000", 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054586119999999998 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.883256+0000", 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032586400000000001 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119376 7.b 7:d6a7ee34:::benchmark_data_smithi194_147200_object4464:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:53.726473+0000", 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 86.922592559999998, 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3746222100000001, 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119376 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726473+0000", 2024-04-29T14:39:24.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726473+0000", 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726475+0000", 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4990000000000001e-06 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726483+0000", 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5310000000000004e-06 2024-04-29T14:39:24.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726496+0000", 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2986999999999999e-05 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726506+0000", 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0224000000000001e-05 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.778681+0000", 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0521747239999999 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.794207+0000", 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015526280999999999 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.819389+0000", 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025181756999999999 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.888522+0000", 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.069132790999999999 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.051202+0000", 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.162680882 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.059671+0000", 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084690540000000002 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.101095+0000", 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041423479999999999 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119384 7.b 7:dadd1e30:::benchmark_data_smithi194_147200_object4472:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:53.858936+0000", 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 86.790129027999996, 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3006659549999999, 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119384 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858936+0000", 2024-04-29T14:39:24.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858936+0000", 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858940+0000", 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.557e-06 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858949+0000", 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3670000000000003e-06 2024-04-29T14:39:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858967+0000", 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.747e-05 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858980+0000", 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3413e-05 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.892216+0000", 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.033235763 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.910300+0000", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018083825000000001 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.942081+0000", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031780924000000002 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.976254+0000", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034173583 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.109755+0000", 2024-04-29T14:39:24.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.133500392 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.120572+0000", 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010817149999999999 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.159602+0000", 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039030510999999997 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119426 7.b 7:d9880d86:::benchmark_data_smithi194_147200_object4514:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:39:24.109 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:54.130484+0000", 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 86.518580892000003, 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.285222273, 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119426 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130484+0000", 2024-04-29T14:39:24.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130484+0000", 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130488+0000", 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1250000000000001e-06 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130500+0000", 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2599000000000001e-05 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130514+0000", 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3791000000000001e-05 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130529+0000", 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4832e-05 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.231644+0000", 2024-04-29T14:39:24.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.101114919 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.254569+0000", 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022925634 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.286229+0000", 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031659364000000002 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.368671+0000", 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.082442721999999996 2024-04-29T14:39:24.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.372801+0000", 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041300160000000002 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.385050+0000", 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012248785999999999 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.415707+0000", 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030656485000000001 2024-04-29T14:39:24.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119436 7.b 7:dbb5cb9b:::benchmark_data_smithi194_147200_object4524:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:54.225054+0000", 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 86.424011160999996, 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2775057159999998, 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119436 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.115 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225054+0000", 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225054+0000", 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225057+0000", 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3239999999999999e-06 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225070+0000", 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2310000000000001e-05 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225087+0000", 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6838e-05 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225101+0000", 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4218e-05 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.305907+0000", 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0808057230000001 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.319013+0000", 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013106922999999999 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.349486+0000", 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030472803999999999 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.421863+0000", 2024-04-29T14:39:24.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.072376531999999993 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.446339+0000", 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024475820999999998 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.458057+0000", 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011718833 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.502560+0000", 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.044502390000000003 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPushReply(7.b 711/685 [PushReplyOp(7:db144375:::benchmark_data_smithi194_147200_object7387:head)])", 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:38:02.687791+0000", 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.961274525999997, 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0677553089999998, 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:39:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687791+0000", 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687791+0000", 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687793+0000", 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9470000000000002e-06 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687801+0000", 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0439999999999996e-06 2024-04-29T14:39:24.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687814+0000", 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3577999999999999e-05 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687825+0000", 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0341e-05 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:05.709764+0000", 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0219390709999998 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:39:24.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:05.755546+0000", 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045782327999999997 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:39:24.123 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:39:24.124 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:39:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:39:25.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:39:25.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:25 smithi040 ceph-mon[39581]: from='client.17493 v1:172.21.15.40:0/2355072986' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:39:25.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:25 smithi040 ceph-mon[39581]: pgmap v1896: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 17015/205538 objects misplaced (8.278%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:39:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:25 smithi194 ceph-mon[44089]: from='client.17493 v1:172.21.15.40:0/2355072986' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:39:25.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:25 smithi194 ceph-mon[44089]: pgmap v1896: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 17015/205538 objects misplaced (8.278%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:39:25.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:25 smithi121 ceph-mon[44533]: from='client.17493 v1:172.21.15.40:0/2355072986' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:39:25.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:25 smithi121 ceph-mon[44533]: pgmap v1896: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 17015/205538 objects misplaced (8.278%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:39:26.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:39:26.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:39:26.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:39:26.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:39:26.116 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:39:26.117 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:39:26.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:26.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osdmap e738: 12 total, 12 up, 11 in 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2096462986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:26.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:26.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:26.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osdmap e738: 12 total, 12 up, 11 in 2024-04-29T14:39:26.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2096462986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:26.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osdmap e738: 12 total, 12 up, 11 in 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2096462986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:26.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:26.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:26.779 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:26.798 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:26.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:26.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:27.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:27.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:27.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:27.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:27 smithi040 ceph-mon[39581]: pgmap v1898: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 116 op/s; 17015/204980 objects misplaced (8.301%); 3.1 MiB/s, 49 objects/s recovering 2024-04-29T14:39:27.290 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:27 smithi040 ceph-mon[39581]: osdmap e739: 12 total, 12 up, 11 in 2024-04-29T14:39:27.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:27.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:27.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:27.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:27 smithi194 ceph-mon[44089]: pgmap v1898: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 116 op/s; 17015/204980 objects misplaced (8.301%); 3.1 MiB/s, 49 objects/s recovering 2024-04-29T14:39:27.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:27 smithi194 ceph-mon[44089]: osdmap e739: 12 total, 12 up, 11 in 2024-04-29T14:39:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:27.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:27.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:27.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:27 smithi121 ceph-mon[44533]: pgmap v1898: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 116 op/s; 17015/204980 objects misplaced (8.301%); 3.1 MiB/s, 49 objects/s recovering 2024-04-29T14:39:27.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:27 smithi121 ceph-mon[44533]: osdmap e739: 12 total, 12 up, 11 in 2024-04-29T14:39:28.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:39:28.493 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:28.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:28.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:28.626 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:28.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:28.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:28.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:28.774 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:28.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:28.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:28.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:28.922 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:28.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:28.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:29.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:39:29.070 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:29.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:39:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:29.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:39:29.173 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1895,"stamp":"2024-04-29T14:39:22.228731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6748766776,"num_objects":102972,"num_object_clones":0,"num_object_copies":205944,"num_objects_missing_on_primary":5599,"num_objects_missing":5599,"num_objects_degraded":0,"num_objects_misplaced":17069,"num_objects_unfound":0,"num_objects_dirty":102972,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":127031,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57602,"num_bytes_recovered":3761209056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131017,"ondisk_log_size":131017,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":21934976,"kb_used_data":15634888,"kb_used_omap":672,"kb_used_meta":6299295,"kb_avail":1009261696,"statfs":{"total":1055945392128,"available":1033483976704,"internally_reserved":0,"allocated":16010125312,"data_stored":16003918189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":688465,"internal_metadata":6450478767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-32833536,"num_objects":-501,"num_object_clones":0,"num_object_copies":-1002,"num_objects_missing_on_primary":-24,"num_objects_missing":-24,"num_objects_degraded":0,"num_objects_misplaced":-572,"num_objects_unfound":0,"num_objects_dirty":-501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":501,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":548,"num_bytes_recovered":35913728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.018901"},"pg_stats":[{"pgid":"7.f","version":"735'8007","reported_seq":22706,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-29T14:39:20.219607+0000","last_change":"2024-04-29T14:38:44.032328+0000","last_active":"2024-04-29T14:39:20.219607+0000","last_peered":"2024-04-29T14:39:20.219607+0000","last_clean":"2024-04-29T14:39:20.219607+0000","last_became_active":"2024-04-29T14:38:40.422563+0000","last_became_peered":"2024-04-29T14:38:40.422563+0000","last_unstale":"2024-04-29T14:39:20.219607+0000","last_undegraded":"2024-04-29T14:39:20.219607+0000","last_fullsized":"2024-04-29T14:39:20.219607+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7824","last_scrub_stamp":"2024-04-29T14:38:44.032227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:44.032227+0000","objects_scrubbed":6620,"log_size":8007,"log_dups_size":0,"ondisk_log_size":8007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:54.066706+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421855232,"num_objects":6437,"num_object_clones":0,"num_object_copies":12874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8007,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7222,"num_bytes_recovered":464846848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"735'15076","reported_seq":15538,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:17.610538+0000","last_change":"2024-04-29T14:37:57.239255+0000","last_active":"2024-04-29T14:39:17.610538+0000","last_peered":"2024-04-29T14:39:17.610538+0000","last_clean":"2024-04-29T14:39:17.610538+0000","last_became_active":"2024-04-29T14:34:49.387078+0000","last_became_peered":"2024-04-29T14:34:49.387078+0000","last_unstale":"2024-04-29T14:39:17.610538+0000","last_undegraded":"2024-04-29T14:39:17.610538+0000","last_fullsized":"2024-04-29T14:39:17.610538+0000","mapping_epoch":655,"log_start":"671'5000","ondisk_log_start":"671'5000","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"709'14617","last_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:37:57.239106+0000","objects_scrubbed":7056,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":36,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:19.725665+0000","scrub_duration":35381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427950080,"num_objects":6530,"num_object_clones":0,"num_object_copies":13060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"736'7913","reported_seq":20745,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-29T14:39:20.705518+0000","last_change":"2024-04-29T14:38:50.737150+0000","last_active":"2024-04-29T14:39:20.705518+0000","last_peered":"2024-04-29T14:39:20.705518+0000","last_clean":"2024-04-29T14:39:20.705518+0000","last_became_active":"2024-04-29T14:35:47.383003+0000","last_became_peered":"2024-04-29T14:35:47.383003+0000","last_unstale":"2024-04-29T14:39:20.705518+0000","last_undegraded":"2024-04-29T14:39:20.705518+0000","last_fullsized":"2024-04-29T14:39:20.705518+0000","mapping_epoch":676,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'7777","last_scrub_stamp":"2024-04-29T14:38:50.737012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:50.737012+0000","objects_scrubbed":6549,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:40:05.629384+0000","scrub_duration":513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420216856,"num_objects":6413,"num_object_clones":0,"num_object_copies":12826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6413,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7913,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6289,"num_bytes_recovered":412155904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"736'7933","reported_seq":8179,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-29T14:39:20.434651+0000","last_change":"2024-04-29T14:38:45.775613+0000","last_active":"2024-04-29T14:39:20.434651+0000","last_peered":"2024-04-29T14:39:20.434651+0000","last_clean":"2024-04-29T14:39:20.434651+0000","last_became_active":"2024-04-29T14:34:49.386724+0000","last_became_peered":"2024-04-29T14:34:49.386724+0000","last_unstale":"2024-04-29T14:39:20.434651+0000","last_undegraded":"2024-04-29T14:39:20.434651+0000","last_fullsized":"2024-04-29T14:39:20.434651+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7767","last_scrub_stamp":"2024-04-29T14:38:45.775459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:45.775459+0000","objects_scrubbed":6583,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:54.299055+0000","scrub_duration":556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420413440,"num_objects":6415,"num_object_clones":0,"num_object_copies":12830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"735'7849","reported_seq":10603,"reported_epoch":735,"state":"active+recovering","last_fresh":"2024-04-29T14:39:18.681424+0000","last_change":"2024-04-29T14:36:34.055204+0000","last_active":"2024-04-29T14:39:18.681424+0000","last_peered":"2024-04-29T14:39:18.681424+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:36:33.135845+0000","last_became_peered":"2024-04-29T14:36:33.135845+0000","last_unstale":"2024-04-29T14:39:18.681424+0000","last_undegraded":"2024-04-29T14:39:18.681424+0000","last_fullsized":"2024-04-29T14:39:18.681424+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421593088,"num_objects":6433,"num_object_clones":0,"num_object_copies":12866,"num_objects_missing_on_primary":5599,"num_objects_missing":5599,"num_objects_degraded":0,"num_objects_misplaced":12016,"num_objects_unfound":0,"num_objects_dirty":6433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1149,"num_bytes_recovered":75300864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["10","11"],"object_location_counts":[{"shards":"1,5","objects":16},{"shards":"1,10,11","objects":818},{"shards":"10,11","objects":5599}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"735'15182","reported_seq":15645,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:20.102647+0000","last_change":"2024-04-29T14:37:59.002976+0000","last_active":"2024-04-29T14:39:20.102647+0000","last_peered":"2024-04-29T14:39:20.102647+0000","last_clean":"2024-04-29T14:39:20.102647+0000","last_became_active":"2024-04-29T14:34:49.387647+0000","last_became_peered":"2024-04-29T14:34:49.387647+0000","last_unstale":"2024-04-29T14:39:20.102647+0000","last_undegraded":"2024-04-29T14:39:20.102647+0000","last_fullsized":"2024-04-29T14:39:20.102647+0000","mapping_epoch":655,"log_start":"672'5100","ondisk_log_start":"672'5100","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'14736","last_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:37:59.002803+0000","objects_scrubbed":7074,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":36,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:22.937516+0000","scrub_duration":35998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430309376,"num_objects":6566,"num_object_clones":0,"num_object_copies":13132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8108,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"736'7951","reported_seq":25231,"reported_epoch":736,"state":"active+recovering+remapped","last_fresh":"2024-04-29T14:39:21.442831+0000","last_change":"2024-04-29T14:38:51.363941+0000","last_active":"2024-04-29T14:39:21.442831+0000","last_peered":"2024-04-29T14:39:21.442831+0000","last_clean":"2024-04-29T14:38:48.778809+0000","last_became_active":"2024-04-29T14:38:51.314146+0000","last_became_peered":"2024-04-29T14:38:51.314146+0000","last_unstale":"2024-04-29T14:39:21.442831+0000","last_undegraded":"2024-04-29T14:39:21.442831+0000","last_fullsized":"2024-04-29T14:39:21.442831+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7744","last_scrub_stamp":"2024-04-29T14:38:41.906760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.906760+0000","objects_scrubbed":6597,"log_size":7951,"log_dups_size":0,"ondisk_log_size":7951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418709504,"num_objects":6389,"num_object_clones":0,"num_object_copies":12778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5053,"num_objects_unfound":0,"num_objects_dirty":6389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7951,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8513,"num_bytes_recovered":557907968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[0,8],"avail_no_missing":["0","8","10"],"object_location_counts":[{"shards":"0,8","objects":1336}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"735'7847","reported_seq":8168,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:17.491337+0000","last_change":"2024-04-29T14:38:44.968069+0000","last_active":"2024-04-29T14:39:17.491337+0000","last_peered":"2024-04-29T14:39:17.491337+0000","last_clean":"2024-04-29T14:39:17.491337+0000","last_became_active":"2024-04-29T14:34:49.514880+0000","last_became_peered":"2024-04-29T14:34:49.514880+0000","last_unstale":"2024-04-29T14:39:17.491337+0000","last_undegraded":"2024-04-29T14:39:17.491337+0000","last_fullsized":"2024-04-29T14:39:17.491337+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7685","last_scrub_stamp":"2024-04-29T14:38:44.967869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:44.967869+0000","objects_scrubbed":6611,"log_size":7847,"log_dups_size":0,"ondisk_log_size":7847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:40:08.630622+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422248448,"num_objects":6443,"num_object_clones":0,"num_object_copies":12886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7847,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"736'8146","reported_seq":36087,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-29T14:39:20.219476+0000","last_change":"2024-04-29T14:38:42.724701+0000","last_active":"2024-04-29T14:39:20.219476+0000","last_peered":"2024-04-29T14:39:20.219476+0000","last_clean":"2024-04-29T14:39:20.219476+0000","last_became_active":"2024-04-29T14:37:07.575390+0000","last_became_peered":"2024-04-29T14:37:07.575390+0000","last_unstale":"2024-04-29T14:39:20.219476+0000","last_undegraded":"2024-04-29T14:39:20.219476+0000","last_fullsized":"2024-04-29T14:39:20.219476+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7943","last_scrub_stamp":"2024-04-29T14:38:42.724491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:42.724491+0000","objects_scrubbed":6787,"log_size":8146,"log_dups_size":0,"ondisk_log_size":8146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:55.119106+0000","scrub_duration":548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431357952,"num_objects":6582,"num_object_clones":0,"num_object_copies":13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8146,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13841,"num_bytes_recovered":898957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"735'7911","reported_seq":8232,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:20.053229+0000","last_change":"2024-04-29T14:38:48.535629+0000","last_active":"2024-04-29T14:39:20.053229+0000","last_peered":"2024-04-29T14:39:20.053229+0000","last_clean":"2024-04-29T14:39:20.053229+0000","last_became_active":"2024-04-29T14:34:49.388394+0000","last_became_peered":"2024-04-29T14:34:49.388394+0000","last_unstale":"2024-04-29T14:39:20.053229+0000","last_undegraded":"2024-04-29T14:39:20.053229+0000","last_fullsized":"2024-04-29T14:39:20.053229+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7774","last_scrub_stamp":"2024-04-29T14:38:48.535530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:48.535530+0000","objects_scrubbed":6500,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:40:09.827204+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417005568,"num_objects":6363,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"735'7986","reported_seq":20942,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:18.697579+0000","last_change":"2024-04-29T14:37:11.232594+0000","last_active":"2024-04-29T14:39:18.697579+0000","last_peered":"2024-04-29T14:39:18.697579+0000","last_clean":"2024-04-29T14:39:18.697579+0000","last_became_active":"2024-04-29T14:37:11.225189+0000","last_became_peered":"2024-04-29T14:37:11.225189+0000","last_unstale":"2024-04-29T14:39:18.697579+0000","last_undegraded":"2024-04-29T14:39:18.697579+0000","last_fullsized":"2024-04-29T14:39:18.697579+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7986,"log_dups_size":0,"ondisk_log_size":7986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427163648,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6389,"num_bytes_recovered":418709504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"735'7922","reported_seq":8244,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:17.645457+0000","last_change":"2024-04-29T14:38:41.886934+0000","last_active":"2024-04-29T14:39:17.645457+0000","last_peered":"2024-04-29T14:39:17.645457+0000","last_clean":"2024-04-29T14:39:17.645457+0000","last_became_active":"2024-04-29T14:34:49.592105+0000","last_became_peered":"2024-04-29T14:34:49.592105+0000","last_unstale":"2024-04-29T14:39:17.645457+0000","last_undegraded":"2024-04-29T14:39:17.645457+0000","last_fullsized":"2024-04-29T14:39:17.645457+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7748","last_scrub_stamp":"2024-04-29T14:38:41.886707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.886707+0000","objects_scrubbed":6661,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:59.864964+0000","scrub_duration":623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425066496,"num_objects":6486,"num_object_clones":0,"num_object_copies":12972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"735'7819","reported_seq":8139,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:19.128761+0000","last_change":"2024-04-29T14:38:43.888778+0000","last_active":"2024-04-29T14:39:19.128761+0000","last_peered":"2024-04-29T14:39:19.128761+0000","last_clean":"2024-04-29T14:39:19.128761+0000","last_became_active":"2024-04-29T14:34:49.387077+0000","last_became_peered":"2024-04-29T14:34:49.387077+0000","last_unstale":"2024-04-29T14:39:19.128761+0000","last_undegraded":"2024-04-29T14:39:19.128761+0000","last_fullsized":"2024-04-29T14:39:19.128761+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7626","last_scrub_stamp":"2024-04-29T14:38:43.888536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:43.888536+0000","objects_scrubbed":6565,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:40:10.867671+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417529856,"num_objects":6371,"num_object_clones":0,"num_object_copies":12742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7819,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"735'7851","reported_seq":22331,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:19.061581+0000","last_change":"2024-04-29T14:38:44.844669+0000","last_active":"2024-04-29T14:39:19.061581+0000","last_peered":"2024-04-29T14:39:19.061581+0000","last_clean":"2024-04-29T14:39:19.061581+0000","last_became_active":"2024-04-29T14:36:04.621648+0000","last_became_peered":"2024-04-29T14:36:04.621648+0000","last_unstale":"2024-04-29T14:39:19.061581+0000","last_undegraded":"2024-04-29T14:39:19.061581+0000","last_fullsized":"2024-04-29T14:39:19.061581+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7659","last_scrub_stamp":"2024-04-29T14:38:44.844574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:44.844574+0000","objects_scrubbed":6522,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:40:03.189206+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414646272,"num_objects":6327,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7077,"num_bytes_recovered":463798272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"735'7839","reported_seq":22231,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:18.504350+0000","last_change":"2024-04-29T14:37:23.391838+0000","last_active":"2024-04-29T14:39:18.504350+0000","last_peered":"2024-04-29T14:39:18.504350+0000","last_clean":"2024-04-29T14:39:18.504350+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:39:18.504350+0000","last_undegraded":"2024-04-29T14:39:18.504350+0000","last_fullsized":"2024-04-29T14:39:18.504350+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":7839,"log_dups_size":0,"ondisk_log_size":7839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419758080,"num_objects":6405,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7839,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"735'7748","reported_seq":8066,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-29T14:39:19.098995+0000","last_change":"2024-04-29T14:38:47.788822+0000","last_active":"2024-04-29T14:39:19.098995+0000","last_peered":"2024-04-29T14:39:19.098995+0000","last_clean":"2024-04-29T14:39:19.098995+0000","last_became_active":"2024-04-29T14:34:49.389669+0000","last_became_peered":"2024-04-29T14:34:49.389669+0000","last_unstale":"2024-04-29T14:39:19.098995+0000","last_undegraded":"2024-04-29T14:39:19.098995+0000","last_fullsized":"2024-04-29T14:39:19.098995+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7605","last_scrub_stamp":"2024-04-29T14:38:47.788696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:47.788696+0000","objects_scrubbed":6437,"log_size":7748,"log_dups_size":0,"ondisk_log_size":7748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:53.296944+0000","scrub_duration":550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412352512,"num_objects":6292,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1785,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-29T14:39:20.219410+0000","last_change":"2024-04-29T14:38:41.263213+0000","last_active":"2024-04-29T14:39:20.219410+0000","last_peered":"2024-04-29T14:39:20.219410+0000","last_clean":"2024-04-29T14:39:20.219410+0000","last_became_active":"2024-04-29T14:35:31.373888+0000","last_became_peered":"2024-04-29T14:35:31.373888+0000","last_unstale":"2024-04-29T14:39:20.219410+0000","last_undegraded":"2024-04-29T14:39:20.219410+0000","last_fullsized":"2024-04-29T14:39:20.219410+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:38:41.263110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.263110+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:39:52.971546+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6748176408,"num_objects":102970,"num_object_clones":0,"num_object_copies":205940,"num_objects_missing_on_primary":5599,"num_objects_missing":5599,"num_objects_degraded":0,"num_objects_misplaced":17069,"num_objects_unfound":0,"num_objects_dirty":102970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126850,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57588,"num_bytes_recovered":3757506560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15984107520,"data_stored":15984099376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130880,"ondisk_log_size":130880,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122437,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1595132,"kb_used_data":864240,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92150020,"statfs":{"total":95995035648,"available":94361620480,"internally_reserved":0,"allocated":884981760,"data_stored":884414349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":748419524},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384076,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3316592,"kb_used_data":2523236,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":90428560,"statfs":{"total":95995035648,"available":92598845440,"internally_reserved":0,"allocated":2583793664,"data_stored":2583232430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":812382656},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645711,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2770028,"kb_used_data":2084964,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":90975124,"statfs":{"total":95995035648,"available":93158526976,"internally_reserved":0,"allocated":2135003136,"data_stored":2134441922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":701495744},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423016,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1345712,"kb_used_data":1245156,"kb_used_omap":4,"kb_used_meta":100539,"kb_avail":92399440,"statfs":{"total":95995035648,"available":94617026560,"internally_reserved":0,"allocated":1275039744,"data_stored":1274478530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":102952640},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234392,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251116,"kb_used_data":411940,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92494036,"statfs":{"total":95995035648,"available":94713892864,"internally_reserved":0,"allocated":421826560,"data_stored":421265346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":859306442},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528734,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2109640,"kb_used_data":1244092,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91635512,"statfs":{"total":95995035648,"available":93834764288,"internally_reserved":0,"allocated":1273950208,"data_stored":1273375193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":886307270},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1627828,"kb_used_data":1321768,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92117324,"statfs":{"total":95995035648,"available":94328139776,"internally_reserved":0,"allocated":1353490432,"data_stored":1352925146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738812,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2769464,"kb_used_data":1990000,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":90975688,"statfs":{"total":95995035648,"available":93159104512,"internally_reserved":0,"allocated":2037760000,"data_stored":2037192609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497244,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2930028,"kb_used_data":2717216,"kb_used_omap":4,"kb_used_meta":212795,"kb_avail":90815124,"statfs":{"total":95995035648,"available":92994686976,"internally_reserved":0,"allocated":2782429184,"data_stored":2781870645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":217902788},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986601,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271480,"kb_used_data":414256,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92473672,"statfs":{"total":95995035648,"available":94693040128,"internally_reserved":0,"allocated":424198144,"data_stored":423630753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653157,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947956,"kb_used_data":818020,"kb_used_omap":651,"kb_used_meta":129268,"kb_avail":92797196,"statfs":{"total":95995035648,"available":95024328704,"internally_reserved":0,"allocated":837652480,"data_stored":837091266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":132371208},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2030305280,"data_stored":2030305280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1353977856,"data_stored":1353973784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2776432640,"data_stored":2776432640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":423100416,"data_stored":423100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":835715072,"data_stored":835715072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1271992320,"data_stored":1271988248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":420347904,"data_stored":420347904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1272905728,"data_stored":1272905728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2133524480,"data_stored":2133524480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2582249472,"data_stored":2582249472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":883556352,"data_stored":883556352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:39:29.175 INFO:tasks.thrashosds.thrasher:Setting 7.5 to [2, 9, 1, 4] 2024-04-29T14:39:29.175 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.5', '2', '9', '1', '4'] 2024-04-29T14:39:29.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 7.5 2 9 1 4 2024-04-29T14:39:29.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:29.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:29.339 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:29.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:29.367 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:29.385 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:29.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:29.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:29.517 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:29 smithi040 ceph-mon[39581]: pgmap v1900: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 16728/204728 objects misplaced (8.171%); 3.3 MiB/s, 52 objects/s recovering 2024-04-29T14:39:29.519 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:29.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:29 smithi194 ceph-mon[44089]: pgmap v1900: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 16728/204728 objects misplaced (8.171%); 3.3 MiB/s, 52 objects/s recovering 2024-04-29T14:39:29.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:29 smithi121 ceph-mon[44533]: pgmap v1900: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 16728/204728 objects misplaced (8.171%); 3.3 MiB/s, 52 objects/s recovering 2024-04-29T14:39:29.647 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:29.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:29.753 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:29.780 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:29.799 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:29.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:29.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:29.930 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:29.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:29.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:30.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:30.087 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:30.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:39:31.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:31 smithi194 ceph-mon[44089]: pgmap v1901: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 16676/204360 objects misplaced (8.160%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:39:31.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:31 smithi040 ceph-mon[39581]: pgmap v1901: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 16676/204360 objects misplaced (8.160%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:39:31.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:31 smithi121 ceph-mon[44533]: pgmap v1901: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 16676/204360 objects misplaced (8.160%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:39:31.731 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:31.742 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:31.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:31.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:31.882 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:32.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:32.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:32.144 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:32.163 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:32.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:32.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:32.298 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:32.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:32.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:32.423 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:32.449 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:32.468 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:32.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:32.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:32.604 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:32.681 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:32.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:32.786 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:32.812 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:32.831 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:32.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:32.934 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:39:32.961 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:32.979 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:39:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:32.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:39:33.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:33.114 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:33.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:33.199 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:33.249 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:33.276 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:33.295 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:33.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:33.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:33 smithi040 ceph-mon[39581]: pgmap v1902: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 16374/203922 objects misplaced (8.030%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:39:33.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:33.429 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:33.463 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:33.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:33.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:33 smithi194 ceph-mon[44089]: pgmap v1902: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 16374/203922 objects misplaced (8.030%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:39:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:33.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:33 smithi121 ceph-mon[44533]: pgmap v1902: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 16374/203922 objects misplaced (8.030%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:39:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:34.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:34.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:35.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:35 smithi194 ceph-mon[44089]: pgmap v1903: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 16374/203732 objects misplaced (8.037%); 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:39:35.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:35 smithi040 ceph-mon[39581]: pgmap v1903: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 16374/203732 objects misplaced (8.037%); 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:39:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:35 smithi121 ceph-mon[44533]: pgmap v1903: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 16374/203732 objects misplaced (8.037%); 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:39:37.090 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:37.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:37.200 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:37.229 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:37.248 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:37.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:37.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:37.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:37 smithi040 ceph-mon[39581]: pgmap v1904: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 93 op/s; 16321/203114 objects misplaced (8.035%); 4.0 MiB/s, 64 objects/s recovering 2024-04-29T14:39:37.387 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:37.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:37 smithi194 ceph-mon[44089]: pgmap v1904: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 93 op/s; 16321/203114 objects misplaced (8.035%); 4.0 MiB/s, 64 objects/s recovering 2024-04-29T14:39:37.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:37 smithi121 ceph-mon[44533]: pgmap v1904: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 93 op/s; 16321/203114 objects misplaced (8.035%); 4.0 MiB/s, 64 objects/s recovering 2024-04-29T14:39:38.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:38 smithi194 ceph-mon[44089]: pgmap v1905: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 16008/202866 objects misplaced (7.891%); 5.0 MiB/s, 80 objects/s recovering 2024-04-29T14:39:38.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:38 smithi040 ceph-mon[39581]: pgmap v1905: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 16008/202866 objects misplaced (7.891%); 5.0 MiB/s, 80 objects/s recovering 2024-04-29T14:39:38.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:38 smithi121 ceph-mon[44533]: pgmap v1905: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 16008/202866 objects misplaced (7.891%); 5.0 MiB/s, 80 objects/s recovering 2024-04-29T14:39:39.594 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:39:39.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1965909627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:39:39.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:39 smithi040 ceph-mon[39581]: from='client.27211 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:39:39.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1965909627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:39:39.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:39 smithi121 ceph-mon[44533]: from='client.27211 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:39:40.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1965909627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:39:40.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:39 smithi194 ceph-mon[44089]: from='client.27211 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:39:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:40.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:40.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:40.578 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:40.600 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:40.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:40.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:40 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:40.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:40 smithi040 ceph-mon[39581]: from='client.27211 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:39:40.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:40 smithi040 ceph-mon[39581]: osdmap e740: 12 total, 12 up, 11 in 2024-04-29T14:39:40.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:40 smithi040 ceph-mon[39581]: pgmap v1907: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 91 op/s; 15954/202534 objects misplaced (7.877%); 4.2 MiB/s, 66 objects/s recovering 2024-04-29T14:39:40.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:40.737 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:40.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:40.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:40.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:40.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:40 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:40.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:40 smithi121 ceph-mon[44533]: from='client.27211 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:39:40.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:40 smithi121 ceph-mon[44533]: osdmap e740: 12 total, 12 up, 11 in 2024-04-29T14:39:40.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:40 smithi121 ceph-mon[44533]: pgmap v1907: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 91 op/s; 15954/202534 objects misplaced (7.877%); 4.2 MiB/s, 66 objects/s recovering 2024-04-29T14:39:40.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:40 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:40.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:40 smithi194 ceph-mon[44089]: from='client.27211 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:39:40.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:40 smithi194 ceph-mon[44089]: osdmap e740: 12 total, 12 up, 11 in 2024-04-29T14:39:40.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:40 smithi194 ceph-mon[44089]: pgmap v1907: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 91 op/s; 15954/202534 objects misplaced (7.877%); 4.2 MiB/s, 66 objects/s recovering 2024-04-29T14:39:40.894 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:40.914 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:40.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:41.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:41.052 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:41.603 INFO:teuthology.orchestra.run.smithi040.stderr:set 7.5 pg_upmap_items mapping to [2->9,1->4] 2024-04-29T14:39:41.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3672721615' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [2, 9, 1, 4]}]: dispatch 2024-04-29T14:39:41.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3672721615' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [2, 9, 1, 4]}]: dispatch 2024-04-29T14:39:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:42.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:42.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3672721615' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [2, 9, 1, 4]}]: dispatch 2024-04-29T14:39:42.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:42.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3672721615' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [2, 9, 1, 4]}]': finished 2024-04-29T14:39:42.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:42 smithi040 ceph-mon[39581]: osdmap e741: 12 total, 12 up, 11 in 2024-04-29T14:39:42.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:42 smithi040 ceph-mon[39581]: pgmap v1909: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 93 op/s; 15610/202236 objects misplaced (7.719%); 5.6 MiB/s, 88 objects/s recovering 2024-04-29T14:39:42.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3672721615' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [2, 9, 1, 4]}]': finished 2024-04-29T14:39:42.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:42 smithi121 ceph-mon[44533]: osdmap e741: 12 total, 12 up, 11 in 2024-04-29T14:39:42.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:42 smithi121 ceph-mon[44533]: pgmap v1909: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 93 op/s; 15610/202236 objects misplaced (7.719%); 5.6 MiB/s, 88 objects/s recovering 2024-04-29T14:39:43.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3672721615' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [2, 9, 1, 4]}]': finished 2024-04-29T14:39:43.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:42 smithi194 ceph-mon[44089]: osdmap e741: 12 total, 12 up, 11 in 2024-04-29T14:39:43.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:42 smithi194 ceph-mon[44089]: pgmap v1909: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 93 op/s; 15610/202236 objects misplaced (7.719%); 5.6 MiB/s, 88 objects/s recovering 2024-04-29T14:39:43.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:39:43.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:39:43.142 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:39:43.163 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:39:43.185 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:39:43.199 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:43.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:43.209 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:39:43.233 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:39:43.255 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:39:43.278 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:39:43.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:39:43.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:43.322 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:39:43.344 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:43.349 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:39:43.351 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:39:43.352 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:39:43.352 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:39:43.352 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:39:43.352 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:39:43.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2821354025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:39:43.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:43.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:43.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2821354025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:39:43.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:43.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:43.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:43.949 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:39:43.949 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:39:43.949 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:39:43.949 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:39:44.006 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:44.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2821354025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:39:44.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.036 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:44.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:39:44.056 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:44.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:44.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:44.187 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:44.206 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:44.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:44.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:44.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:44 smithi040 ceph-mon[39581]: pgmap v1910: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 15610/202086 objects misplaced (7.724%); 5.3 MiB/s, 85 objects/s recovering 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:44.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:44 smithi121 ceph-mon[44533]: pgmap v1910: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 15610/202086 objects misplaced (7.724%); 5.3 MiB/s, 85 objects/s recovering 2024-04-29T14:39:45.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:39:45.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:44 smithi194 ceph-mon[44089]: pgmap v1910: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 15610/202086 objects misplaced (7.724%); 5.3 MiB/s, 85 objects/s recovering 2024-04-29T14:39:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:45.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:45 smithi040 ceph-mon[39581]: osdmap e742: 12 total, 12 up, 11 in 2024-04-29T14:39:45.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:45.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:45.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:45 smithi121 ceph-mon[44533]: osdmap e742: 12 total, 12 up, 11 in 2024-04-29T14:39:46.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:46.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:46.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:45 smithi194 ceph-mon[44089]: osdmap e742: 12 total, 12 up, 11 in 2024-04-29T14:39:46.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:46.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:46.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:46 smithi040 ceph-mon[39581]: osdmap e743: 12 total, 12 up, 11 in 2024-04-29T14:39:46.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:46 smithi040 ceph-mon[39581]: pgmap v1913: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 15557/201626 objects misplaced (7.716%); 3.9 MiB/s, 61 objects/s recovering 2024-04-29T14:39:46.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:46.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:46.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:46 smithi121 ceph-mon[44533]: osdmap e743: 12 total, 12 up, 11 in 2024-04-29T14:39:46.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:46 smithi121 ceph-mon[44533]: pgmap v1913: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 15557/201626 objects misplaced (7.716%); 3.9 MiB/s, 61 objects/s recovering 2024-04-29T14:39:47.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:47.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:47.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:46 smithi194 ceph-mon[44089]: osdmap e743: 12 total, 12 up, 11 in 2024-04-29T14:39:47.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:46 smithi194 ceph-mon[44089]: pgmap v1913: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 15557/201626 objects misplaced (7.716%); 3.9 MiB/s, 61 objects/s recovering 2024-04-29T14:39:47.178 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:47.227 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:39:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:47.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:39:47.335 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:47.361 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:47.379 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:47.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:47.484 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:47.510 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:47.529 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:47.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:47.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:47.663 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:47.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:47.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:47.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:47.814 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:47.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:47.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:47.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:47.965 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:47.985 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:47.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:48.072 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8] out_osds: [3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:39:48.072 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:39:48.072 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-04-29T14:39:48.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 3 2024-04-29T14:39:48.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:48.116 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:48.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:48.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:48.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:48.265 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:48.284 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:48.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:48.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:39:48.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:48.621 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:48.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:48.679 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:48.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:48.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:48.913 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:48.940 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:48.959 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:48.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:49.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:39:49.092 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:49.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:39:49.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:49 smithi040 ceph-mon[39581]: pgmap v1914: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 58 op/s; 15193/201472 objects misplaced (7.541%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:39:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:49 smithi194 ceph-mon[44089]: pgmap v1914: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 58 op/s; 15193/201472 objects misplaced (7.541%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:39:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:49 smithi121 ceph-mon[44533]: pgmap v1914: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 58 op/s; 15193/201472 objects misplaced (7.541%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:39:51.146 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:39:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:51.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:39:51.250 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:51.279 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:51.298 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:51.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:51.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:51 smithi040 ceph-mon[39581]: pgmap v1915: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 15140/201186 objects misplaced (7.525%); 3.3 MiB/s, 52 objects/s recovering 2024-04-29T14:39:51.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:39:51.434 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:51.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:39:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:51.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:39:51.559 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:51.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:51 smithi194 ceph-mon[44089]: pgmap v1915: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 15140/201186 objects misplaced (7.525%); 3.3 MiB/s, 52 objects/s recovering 2024-04-29T14:39:51.600 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:51.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:51 smithi121 ceph-mon[44533]: pgmap v1915: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 15140/201186 objects misplaced (7.525%); 3.3 MiB/s, 52 objects/s recovering 2024-04-29T14:39:51.638 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:51.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:51.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:51.773 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:51.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:51.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:51.897 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:51.942 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:51.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:52.045 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:52.071 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:52.091 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:52.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:39:52.194 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:52.227 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:52.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:39:52.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:39:52.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:52 smithi194 ceph-mon[44089]: osdmap e744: 12 total, 12 up, 11 in 2024-04-29T14:39:52.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:39:52.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:39:52.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:52 smithi040 ceph-mon[39581]: osdmap e744: 12 total, 12 up, 11 in 2024-04-29T14:39:52.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:39:52.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:39:52.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:52 smithi121 ceph-mon[44533]: osdmap e744: 12 total, 12 up, 11 in 2024-04-29T14:39:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:39:53.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:53 smithi194 ceph-mon[44089]: pgmap v1917: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 14769/200750 objects misplaced (7.357%); 6.3 MiB/s, 100 objects/s recovering 2024-04-29T14:39:53.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:53 smithi194 ceph-mon[44089]: osdmap e745: 12 total, 12 up, 11 in 2024-04-29T14:39:53.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3323968179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:39:53.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:53 smithi194 ceph-mon[44089]: from='client.27229 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:39:53.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:53 smithi040 ceph-mon[39581]: pgmap v1917: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 14769/200750 objects misplaced (7.357%); 6.3 MiB/s, 100 objects/s recovering 2024-04-29T14:39:53.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:53 smithi040 ceph-mon[39581]: osdmap e745: 12 total, 12 up, 11 in 2024-04-29T14:39:53.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3323968179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:39:53.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:53 smithi040 ceph-mon[39581]: from='client.27229 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:39:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:53 smithi121 ceph-mon[44533]: pgmap v1917: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 75 op/s; 14769/200750 objects misplaced (7.357%); 6.3 MiB/s, 100 objects/s recovering 2024-04-29T14:39:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:53 smithi121 ceph-mon[44533]: osdmap e745: 12 total, 12 up, 11 in 2024-04-29T14:39:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3323968179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:39:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:53 smithi121 ceph-mon[44533]: from='client.27229 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:39:54.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:54 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:54.336 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:54 smithi040 ceph-mon[39581]: from='client.27229 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:39:54.336 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:54 smithi040 ceph-mon[39581]: osdmap e746: 12 total, 12 up, 11 in 2024-04-29T14:39:54.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:54 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:54.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:54 smithi194 ceph-mon[44089]: from='client.27229 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:39:54.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:54 smithi194 ceph-mon[44089]: osdmap e746: 12 total, 12 up, 11 in 2024-04-29T14:39:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:54 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:39:54.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:54 smithi121 ceph-mon[44533]: from='client.27229 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:39:54.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:54 smithi121 ceph-mon[44533]: osdmap e746: 12 total, 12 up, 11 in 2024-04-29T14:39:55.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:55 smithi194 ceph-mon[44089]: pgmap v1920: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 14769/200482 objects misplaced (7.367%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:55.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:55 smithi194 ceph-mon[44089]: 7.c scrub starts 2024-04-29T14:39:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:55 smithi040 ceph-mon[39581]: pgmap v1920: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 14769/200482 objects misplaced (7.367%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:55.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:55 smithi040 ceph-mon[39581]: 7.c scrub starts 2024-04-29T14:39:55.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:55 smithi121 ceph-mon[44533]: pgmap v1920: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 14769/200482 objects misplaced (7.367%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:55.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:55 smithi121 ceph-mon[44533]: 7.c scrub starts 2024-04-29T14:39:56.184 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:56.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:56.292 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:39:56.318 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:56.338 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:39:56.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:56.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:39:56.443 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:39:56.469 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:56.492 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:39:56.494 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:56 smithi194 ceph-mon[44089]: 7.c scrub ok 2024-04-29T14:39:56.494 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:56 smithi194 ceph-mon[44089]: 7.a scrub starts 2024-04-29T14:39:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:56.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:39:56.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:56.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:56 smithi040 ceph-mon[39581]: 7.c scrub ok 2024-04-29T14:39:56.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:56 smithi040 ceph-mon[39581]: 7.a scrub starts 2024-04-29T14:39:56.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:56 smithi121 ceph-mon[44533]: 7.c scrub ok 2024-04-29T14:39:56.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:56 smithi121 ceph-mon[44533]: 7.a scrub starts 2024-04-29T14:39:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:56.665 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:39:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:56.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:39:56.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:39:56.798 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:56.817 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:39:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:56.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:39:56.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:39:56.950 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:56.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:39:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:56.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:39:57.023 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:39:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:39:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:39:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:39:57.023 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:39:57.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:39:57.099 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:57.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:39:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:57.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:39:57.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:39:57.247 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:57.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:39:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:57.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:39:57.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:57 smithi040 ceph-mon[39581]: pgmap v1921: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 14715/199806 objects misplaced (7.365%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:57.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:57 smithi040 ceph-mon[39581]: 7.a scrub ok 2024-04-29T14:39:57.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:39:57.397 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:57.418 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:39:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:57.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:39:57.522 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:39:57.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:57 smithi194 ceph-mon[44089]: pgmap v1921: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 14715/199806 objects misplaced (7.365%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:57.548 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:57 smithi194 ceph-mon[44089]: 7.a scrub ok 2024-04-29T14:39:57.549 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:57.569 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:39:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:57.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:39:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:57 smithi121 ceph-mon[44533]: pgmap v1921: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 14715/199806 objects misplaced (7.365%); 4.1 MiB/s, 66 objects/s recovering 2024-04-29T14:39:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:57 smithi121 ceph-mon[44533]: 7.a scrub ok 2024-04-29T14:39:57.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:39:57.710 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:57.816 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:39:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:57.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:39:57.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:39:57.961 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:57.981 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:39:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:39:57.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:39:58.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:39:58.118 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:39:58.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:58 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:39:58.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:58 smithi194 ceph-mon[44089]: osdmap e747: 12 total, 12 up, 11 in 2024-04-29T14:39:58.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:58 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:39:58.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:39:58 smithi194 ceph-mon[44089]: pgmap v1923: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 14298/199546 objects misplaced (7.165%); 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:39:58.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:58 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:39:58.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:58 smithi040 ceph-mon[39581]: osdmap e747: 12 total, 12 up, 11 in 2024-04-29T14:39:58.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:58 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:39:58.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:39:58 smithi040 ceph-mon[39581]: pgmap v1923: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 14298/199546 objects misplaced (7.165%); 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:39:58.882 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:58 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:39:58.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:58 smithi121 ceph-mon[44533]: osdmap e747: 12 total, 12 up, 11 in 2024-04-29T14:39:58.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:58 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:39:58.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:39:58 smithi121 ceph-mon[44533]: pgmap v1923: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 100 op/s; 14298/199546 objects misplaced (7.165%); 4.6 MiB/s, 73 objects/s recovering 2024-04-29T14:40:00.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:00 smithi194 ceph-mon[44089]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T14:40:00.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:00 smithi194 ceph-mon[44089]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-29T14:40:00.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:00 smithi040 ceph-mon[39581]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T14:40:00.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:00 smithi040 ceph-mon[39581]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-29T14:40:00.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:00 smithi121 ceph-mon[44533]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T14:40:00.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:00 smithi121 ceph-mon[44533]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-29T14:40:01.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:00 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:40:00.821+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e747 definitely_dead 0 2024-04-29T14:40:01.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:01 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:40:01.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:01 smithi040 ceph-mon[39581]: pgmap v1924: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 107 op/s; 14242/199088 objects misplaced (7.154%); 3.8 MiB/s, 61 objects/s recovering 2024-04-29T14:40:01.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1265233444' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:40:01.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:01 smithi040 ceph-mon[39581]: from='client.36989 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:40:01.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:01 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:40:01.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:01 smithi194 ceph-mon[44089]: pgmap v1924: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 107 op/s; 14242/199088 objects misplaced (7.154%); 3.8 MiB/s, 61 objects/s recovering 2024-04-29T14:40:01.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1265233444' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:40:01.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:01 smithi194 ceph-mon[44089]: from='client.36989 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:40:01.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:01 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:40:01.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:01 smithi121 ceph-mon[44533]: pgmap v1924: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 107 op/s; 14242/199088 objects misplaced (7.154%); 3.8 MiB/s, 61 objects/s recovering 2024-04-29T14:40:01.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1265233444' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:40:01.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:01 smithi121 ceph-mon[44533]: from='client.36989 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:40:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:40:01.789 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:01.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:01.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:01.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:40:01.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:01.924 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:01.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:01.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:02.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:40:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:02.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:40:02.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:02 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:40:02.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:02 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:40:02.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:02 smithi194 ceph-mon[44089]: from='client.36989 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-29T14:40:02.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:02 smithi194 ceph-mon[44089]: osdmap e748: 12 total, 12 up, 12 in 2024-04-29T14:40:02.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:02 smithi194 ceph-mon[44089]: 7.e scrub starts 2024-04-29T14:40:02.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:02 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:40:02.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:02 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:40:02.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:02 smithi040 ceph-mon[39581]: from='client.36989 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-29T14:40:02.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:02 smithi040 ceph-mon[39581]: osdmap e748: 12 total, 12 up, 12 in 2024-04-29T14:40:02.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:02 smithi040 ceph-mon[39581]: 7.e scrub starts 2024-04-29T14:40:02.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:02 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:40:02.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:02 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:40:02.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:02 smithi121 ceph-mon[44533]: from='client.36989 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-29T14:40:02.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:02 smithi121 ceph-mon[44533]: osdmap e748: 12 total, 12 up, 12 in 2024-04-29T14:40:02.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:02 smithi121 ceph-mon[44533]: 7.e scrub starts 2024-04-29T14:40:02.397 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.3. 2024-04-29T14:40:03.321 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:03 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:40:03.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:03 smithi194 ceph-mon[44089]: osdmap e749: 12 total, 12 up, 12 in 2024-04-29T14:40:03.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:03 smithi194 ceph-mon[44089]: pgmap v1927: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 105 op/s; 13794/198540 objects misplaced (6.948%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:40:03.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:03 smithi194 ceph-mon[44089]: 7.e scrub ok 2024-04-29T14:40:03.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:03 smithi194 ceph-mon[44089]: osdmap e750: 12 total, 12 up, 12 in 2024-04-29T14:40:03.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:03 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:40:03.378 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:03 smithi040 ceph-mon[39581]: osdmap e749: 12 total, 12 up, 12 in 2024-04-29T14:40:03.378 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:03 smithi040 ceph-mon[39581]: pgmap v1927: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 105 op/s; 13794/198540 objects misplaced (6.948%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:40:03.378 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:03 smithi040 ceph-mon[39581]: 7.e scrub ok 2024-04-29T14:40:03.378 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:03 smithi040 ceph-mon[39581]: osdmap e750: 12 total, 12 up, 12 in 2024-04-29T14:40:03.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:03 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:40:03.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:03 smithi121 ceph-mon[44533]: osdmap e749: 12 total, 12 up, 12 in 2024-04-29T14:40:03.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:03 smithi121 ceph-mon[44533]: pgmap v1927: 17 pgs: 1 active+recovering+remapped, 1 active+recovering, 15 active+clean; 6.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 105 op/s; 13794/198540 objects misplaced (6.948%); 9.3 MiB/s, 148 objects/s recovering 2024-04-29T14:40:03.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:03 smithi121 ceph-mon[44533]: 7.e scrub ok 2024-04-29T14:40:03.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:03 smithi121 ceph-mon[44533]: osdmap e750: 12 total, 12 up, 12 in 2024-04-29T14:40:03.490 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:40:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:03.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:40:03.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:40:03.686 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:04.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:04.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:40:04.297 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:04.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:40:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:04.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:40:04.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:04.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:04.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:04.382 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:04.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:40:04.452 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:04.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:40:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:04.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:40:04.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:04.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:04.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:40:04.603 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:04.624 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:40:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:04.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:40:04.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:04.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:40:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:04.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:40:04.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:40:04.927 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:05.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:05 smithi040 ceph-mon[39581]: osdmap e751: 12 total, 12 up, 12 in 2024-04-29T14:40:05.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:05 smithi040 ceph-mon[39581]: pgmap v1930: 17 pgs: 2 activating+remapped, 1 active+recovering+remapped, 1 active+recovering, 13 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 94 op/s; 7204/198330 objects degraded (3.632%); 20889/198330 objects misplaced (10.532%); 7.0 MiB/s, 111 objects/s recovering 2024-04-29T14:40:05.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:05 smithi121 ceph-mon[44533]: osdmap e751: 12 total, 12 up, 12 in 2024-04-29T14:40:05.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:05 smithi121 ceph-mon[44533]: pgmap v1930: 17 pgs: 2 activating+remapped, 1 active+recovering+remapped, 1 active+recovering, 13 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 94 op/s; 7204/198330 objects degraded (3.632%); 20889/198330 objects misplaced (10.532%); 7.0 MiB/s, 111 objects/s recovering 2024-04-29T14:40:05.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:05 smithi194 ceph-mon[44089]: osdmap e751: 12 total, 12 up, 12 in 2024-04-29T14:40:05.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:05 smithi194 ceph-mon[44089]: pgmap v1930: 17 pgs: 2 activating+remapped, 1 active+recovering+remapped, 1 active+recovering, 13 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 94 op/s; 7204/198330 objects degraded (3.632%); 20889/198330 objects misplaced (10.532%); 7.0 MiB/s, 111 objects/s recovering 2024-04-29T14:40:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:07 smithi194 ceph-mon[44089]: pgmap v1931: 17 pgs: 2 activating+remapped, 1 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 79 op/s; 7204/197818 objects degraded (3.642%); 32867/197818 objects misplaced (16.615%); 8.9 MiB/s, 142 objects/s recovering 2024-04-29T14:40:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:07 smithi040 ceph-mon[39581]: pgmap v1931: 17 pgs: 2 activating+remapped, 1 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 79 op/s; 7204/197818 objects degraded (3.642%); 32867/197818 objects misplaced (16.615%); 8.9 MiB/s, 142 objects/s recovering 2024-04-29T14:40:07.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:07 smithi121 ceph-mon[44533]: pgmap v1931: 17 pgs: 2 activating+remapped, 1 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 79 op/s; 7204/197818 objects degraded (3.642%); 32867/197818 objects misplaced (16.615%); 8.9 MiB/s, 142 objects/s recovering 2024-04-29T14:40:09.407 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:09.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:09.428 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-04-29T14:40:09.520 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:40:09.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:09 smithi040 ceph-mon[39581]: pgmap v1932: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s; 43345/197516 objects misplaced (21.945%); 12 MiB/s, 184 objects/s recovering 2024-04-29T14:40:09.570 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:09 smithi194 ceph-mon[44089]: pgmap v1932: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s; 43345/197516 objects misplaced (21.945%); 12 MiB/s, 184 objects/s recovering 2024-04-29T14:40:09.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:09 smithi121 ceph-mon[44533]: pgmap v1932: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s; 43345/197516 objects misplaced (21.945%); 12 MiB/s, 184 objects/s recovering 2024-04-29T14:40:10.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:10 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:40:10.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:10 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:40:10.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:10 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:40:11.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:11 smithi194 ceph-mon[44089]: pgmap v1933: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 67 op/s; 43070/197466 objects misplaced (21.811%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:40:11.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:11 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:40:11.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:11 smithi040 ceph-mon[39581]: pgmap v1933: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 67 op/s; 43070/197466 objects misplaced (21.811%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:40:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:11 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:40:11.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:11 smithi121 ceph-mon[44533]: pgmap v1933: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 67 op/s; 43070/197466 objects misplaced (21.811%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:40:11.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:11 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:40:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:40:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:12.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:40:12.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:40:12.986 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:13.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:40:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:13.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:40:13.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:13.140 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:13.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:13.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:13.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:13.267 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:13 smithi121 ceph-mon[44533]: pgmap v1934: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+remapped, 3 active+recovering, 9 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s; 42154/197288 objects misplaced (21.367%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:40:13.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:13.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:13.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:13.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:40:13.442 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:13.461 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:40:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:13.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:40:13.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:40:13.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:13 smithi194 ceph-mon[44089]: pgmap v1934: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+remapped, 3 active+recovering, 9 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s; 42154/197288 objects misplaced (21.367%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:40:13.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:13 smithi040 ceph-mon[39581]: pgmap v1934: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+remapped, 3 active+recovering, 9 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s; 42154/197288 objects misplaced (21.367%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:40:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:13.913 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:13.926 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:40:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:13.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:13.961 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:40:14.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:40:14.054 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:14.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:40:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:14.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:40:14.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:40:14.226 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:14.429 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:40:14.430 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:40:14.430 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8, 3] 2024-04-29T14:40:14.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 3 2024-04-29T14:40:15.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:15 smithi194 ceph-mon[44089]: pgmap v1935: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+remapped, 3 active+recovering, 9 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 56 op/s; 41717/197200 objects misplaced (21.155%); 17 MiB/s, 271 objects/s recovering 2024-04-29T14:40:15.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:15 smithi040 ceph-mon[39581]: pgmap v1935: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+remapped, 3 active+recovering, 9 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 56 op/s; 41717/197200 objects misplaced (21.155%); 17 MiB/s, 271 objects/s recovering 2024-04-29T14:40:15.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:15 smithi121 ceph-mon[44533]: pgmap v1935: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+remapped, 3 active+recovering, 9 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 56 op/s; 41717/197200 objects misplaced (21.155%); 17 MiB/s, 271 objects/s recovering 2024-04-29T14:40:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:16.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:16.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:16.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:16.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:40:16.297 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:16.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:40:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:16.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:40:16.346 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:40:16.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:40:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:16.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:40:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:40:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:16.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:40:16.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:16.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:40:16.834 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:16.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:16.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:16.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:40:16.986 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:17.007 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:40:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:17.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:40:17.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:40:17.155 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:17.271 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:17.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:17.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:17 smithi040 ceph-mon[39581]: pgmap v1936: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 56 op/s; 41158/196982 objects misplaced (20.894%); 17 MiB/s, 273 objects/s recovering 2024-04-29T14:40:17.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:40:17.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:40:17.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:40:17.380 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:40:17.407 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:17.429 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:40:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:17.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:40:17.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:40:17.593 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:17.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:17 smithi121 ceph-mon[44533]: pgmap v1936: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 56 op/s; 41158/196982 objects misplaced (20.894%); 17 MiB/s, 273 objects/s recovering 2024-04-29T14:40:17.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:40:17.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:40:17.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:40:17.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:17 smithi194 ceph-mon[44089]: pgmap v1936: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 56 op/s; 41158/196982 objects misplaced (20.894%); 17 MiB/s, 273 objects/s recovering 2024-04-29T14:40:17.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:40:17.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:40:17.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:40:18.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:18 smithi040 ceph-mon[39581]: pgmap v1937: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 40535/196920 objects misplaced (20.585%); 19 MiB/s, 302 objects/s recovering 2024-04-29T14:40:18.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:18 smithi194 ceph-mon[44089]: pgmap v1937: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 40535/196920 objects misplaced (20.585%); 19 MiB/s, 302 objects/s recovering 2024-04-29T14:40:18.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:18 smithi121 ceph-mon[44533]: pgmap v1937: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 40535/196920 objects misplaced (20.585%); 19 MiB/s, 302 objects/s recovering 2024-04-29T14:40:19.205 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:40:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:19.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:40:19.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:40:19.339 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:19.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:40:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:19.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:40:19.462 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:40:19.491 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:19.511 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:40:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:19.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:40:19.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:40:19.667 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:21.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:21 smithi194 ceph-mon[44089]: pgmap v1938: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 33 op/s; 39869/196706 objects misplaced (20.268%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:40:21.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:21 smithi040 ceph-mon[39581]: pgmap v1938: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 33 op/s; 39869/196706 objects misplaced (20.268%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:40:21.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:21 smithi121 ceph-mon[44533]: pgmap v1938: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 33 op/s; 39869/196706 objects misplaced (20.268%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:40:22.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:22 smithi194 ceph-mon[44089]: osdmap e752: 12 total, 12 up, 12 in 2024-04-29T14:40:22.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:22.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:22.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:22 smithi040 ceph-mon[39581]: osdmap e752: 12 total, 12 up, 12 in 2024-04-29T14:40:22.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:22.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:22.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:22 smithi121 ceph-mon[44533]: osdmap e752: 12 total, 12 up, 12 in 2024-04-29T14:40:23.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:23 smithi194 ceph-mon[44089]: pgmap v1940: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 38865/196384 objects misplaced (19.790%); 20 MiB/s, 326 objects/s recovering 2024-04-29T14:40:23.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:23 smithi194 ceph-mon[44089]: osdmap e753: 12 total, 12 up, 12 in 2024-04-29T14:40:23.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:23 smithi040 ceph-mon[39581]: pgmap v1940: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 38865/196384 objects misplaced (19.790%); 20 MiB/s, 326 objects/s recovering 2024-04-29T14:40:23.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:23 smithi040 ceph-mon[39581]: osdmap e753: 12 total, 12 up, 12 in 2024-04-29T14:40:23.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:23 smithi121 ceph-mon[44533]: pgmap v1940: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 38865/196384 objects misplaced (19.790%); 20 MiB/s, 326 objects/s recovering 2024-04-29T14:40:23.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:23 smithi121 ceph-mon[44533]: osdmap e753: 12 total, 12 up, 12 in 2024-04-29T14:40:24.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:24 smithi040 ceph-mon[39581]: pgmap v1942: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 38409/196254 objects misplaced (19.571%); 21 MiB/s, 341 objects/s recovering 2024-04-29T14:40:24.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:24 smithi194 ceph-mon[44089]: pgmap v1942: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 38409/196254 objects misplaced (19.571%); 21 MiB/s, 341 objects/s recovering 2024-04-29T14:40:24.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:24 smithi121 ceph-mon[44533]: pgmap v1942: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 38409/196254 objects misplaced (19.571%); 21 MiB/s, 341 objects/s recovering 2024-04-29T14:40:26.757 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:26.757 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:26.757 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:26 smithi040 ceph-mon[39581]: osdmap e754: 12 total, 12 up, 12 in 2024-04-29T14:40:26.757 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:26.757 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:26.757 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:26 smithi040 ceph-mon[39581]: pgmap v1944: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 57 op/s; 37790/196012 objects misplaced (19.279%); 21 MiB/s, 342 objects/s recovering 2024-04-29T14:40:26.757 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4201553191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:40:26.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:26 smithi194 ceph-mon[44089]: osdmap e754: 12 total, 12 up, 12 in 2024-04-29T14:40:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:26 smithi194 ceph-mon[44089]: pgmap v1944: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 57 op/s; 37790/196012 objects misplaced (19.279%); 21 MiB/s, 342 objects/s recovering 2024-04-29T14:40:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4201553191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:40:26.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:26.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:26.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:26 smithi121 ceph-mon[44533]: osdmap e754: 12 total, 12 up, 12 in 2024-04-29T14:40:26.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:26.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:26.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:26 smithi121 ceph-mon[44533]: pgmap v1944: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 57 op/s; 37790/196012 objects misplaced (19.279%); 21 MiB/s, 342 objects/s recovering 2024-04-29T14:40:26.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4201553191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:40:27.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:40:27.641 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:40:27.642 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:40:27.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osdmap e755: 12 total, 12 up, 12 in 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osdmap e755: 12 total, 12 up, 12 in 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osdmap e755: 12 total, 12 up, 12 in 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:27.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:40:28.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:28 smithi194 ceph-mon[44089]: pgmap v1946: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 33 op/s; 37151/195978 objects misplaced (18.957%); 18 MiB/s, 323 objects/s recovering 2024-04-29T14:40:28.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:28 smithi040 ceph-mon[39581]: pgmap v1946: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 33 op/s; 37151/195978 objects misplaced (18.957%); 18 MiB/s, 323 objects/s recovering 2024-04-29T14:40:28.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:28 smithi121 ceph-mon[44533]: pgmap v1946: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 33 op/s; 37151/195978 objects misplaced (18.957%); 18 MiB/s, 323 objects/s recovering 2024-04-29T14:40:30.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:30 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:40:30.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:30 smithi194 ceph-mon[44089]: osdmap e756: 12 total, 12 up, 12 in 2024-04-29T14:40:30.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:30 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:40:30.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:30 smithi194 ceph-mon[44089]: pgmap v1948: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 31 op/s; 36365/195874 objects misplaced (18.566%); 21 MiB/s, 378 objects/s recovering 2024-04-29T14:40:30.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:30 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:40:30.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:30 smithi040 ceph-mon[39581]: osdmap e756: 12 total, 12 up, 12 in 2024-04-29T14:40:30.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:30 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:40:30.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:30 smithi040 ceph-mon[39581]: pgmap v1948: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 31 op/s; 36365/195874 objects misplaced (18.566%); 21 MiB/s, 378 objects/s recovering 2024-04-29T14:40:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:30 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:40:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:30 smithi121 ceph-mon[44533]: osdmap e756: 12 total, 12 up, 12 in 2024-04-29T14:40:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:30 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:40:30.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:30 smithi121 ceph-mon[44533]: pgmap v1948: 17 pgs: 4 active+recovering+remapped, 3 active+recovering, 10 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 31 op/s; 36365/195874 objects misplaced (18.566%); 21 MiB/s, 378 objects/s recovering 2024-04-29T14:40:31.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:31 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:31.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:31 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:31.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:31 smithi194 ceph-mon[44089]: osdmap e757: 12 total, 12 up, 12 in 2024-04-29T14:40:31.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:31 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:31.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:31 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:31.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:31 smithi040 ceph-mon[39581]: osdmap e757: 12 total, 12 up, 12 in 2024-04-29T14:40:31.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:31 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:31.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:31 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:31.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:31 smithi121 ceph-mon[44533]: osdmap e757: 12 total, 12 up, 12 in 2024-04-29T14:40:32.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:32 smithi194 ceph-mon[44089]: pgmap v1950: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 24 op/s; 35799/195722 objects misplaced (18.291%); 21 MiB/s, 431 objects/s recovering 2024-04-29T14:40:32.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:32 smithi040 ceph-mon[39581]: pgmap v1950: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 24 op/s; 35799/195722 objects misplaced (18.291%); 21 MiB/s, 431 objects/s recovering 2024-04-29T14:40:32.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:32 smithi121 ceph-mon[44533]: pgmap v1950: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 24 op/s; 35799/195722 objects misplaced (18.291%); 21 MiB/s, 431 objects/s recovering 2024-04-29T14:40:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:35.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:35.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:35.272 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:35.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:35.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:35.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:40:35.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:35 smithi040 ceph-mon[39581]: pgmap v1951: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s; 35273/195578 objects misplaced (18.035%); 20 MiB/s, 407 objects/s recovering 2024-04-29T14:40:35.438 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:35.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:35 smithi194 ceph-mon[44089]: pgmap v1951: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s; 35273/195578 objects misplaced (18.035%); 20 MiB/s, 407 objects/s recovering 2024-04-29T14:40:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:35 smithi121 ceph-mon[44533]: pgmap v1951: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s; 35273/195578 objects misplaced (18.035%); 20 MiB/s, 407 objects/s recovering 2024-04-29T14:40:37.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:37 smithi194 ceph-mon[44089]: pgmap v1952: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 34557/195252 objects misplaced (17.699%); 20 MiB/s, 367 objects/s recovering 2024-04-29T14:40:37.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:37 smithi040 ceph-mon[39581]: pgmap v1952: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 34557/195252 objects misplaced (17.699%); 20 MiB/s, 367 objects/s recovering 2024-04-29T14:40:37.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:37 smithi121 ceph-mon[44533]: pgmap v1952: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 34557/195252 objects misplaced (17.699%); 20 MiB/s, 367 objects/s recovering 2024-04-29T14:40:38.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:38 smithi194 ceph-mon[44089]: pgmap v1953: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 34327/195234 objects misplaced (17.582%); 15 MiB/s, 276 objects/s recovering 2024-04-29T14:40:38.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:38 smithi040 ceph-mon[39581]: pgmap v1953: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 34327/195234 objects misplaced (17.582%); 15 MiB/s, 276 objects/s recovering 2024-04-29T14:40:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:38 smithi121 ceph-mon[44533]: pgmap v1953: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 34327/195234 objects misplaced (17.582%); 15 MiB/s, 276 objects/s recovering 2024-04-29T14:40:39.914 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:40:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:39.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:40:40.020 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:40:40.048 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:40.069 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:40:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:40.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:40:40.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:40:40.216 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:40.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:40:41.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:41 smithi194 ceph-mon[44089]: pgmap v1954: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 43 op/s; 33427/195010 objects misplaced (17.141%); 18 MiB/s, 328 objects/s recovering 2024-04-29T14:40:41.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:41 smithi040 ceph-mon[39581]: pgmap v1954: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 43 op/s; 33427/195010 objects misplaced (17.141%); 18 MiB/s, 328 objects/s recovering 2024-04-29T14:40:41.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:41 smithi121 ceph-mon[44533]: pgmap v1954: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 6.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 43 op/s; 33427/195010 objects misplaced (17.141%); 18 MiB/s, 328 objects/s recovering 2024-04-29T14:40:41.963 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:40:42.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:40:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:42.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:40:42.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:40:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:42.439 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:40:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:42.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:40:42.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:40:42.578 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:42.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:40:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:42.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:40:42.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:40:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:42.748 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:40:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:42.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:40:42.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:40:42.894 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:42.961 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:40:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:42.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:40:43.078 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:40:43.139 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:43.189 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:40:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:43.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:40:43.300 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:40:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:43.348 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:40:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:43.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:40:43.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:43.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:43.500 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:43 smithi121 ceph-mon[44533]: pgmap v1955: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 50 op/s; 32804/194666 objects misplaced (16.851%); 19 MiB/s, 334 objects/s recovering 2024-04-29T14:40:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:43.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:43.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:43 smithi194 ceph-mon[44089]: pgmap v1955: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 50 op/s; 32804/194666 objects misplaced (16.851%); 19 MiB/s, 334 objects/s recovering 2024-04-29T14:40:43.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:40:43.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:43 smithi040 ceph-mon[39581]: pgmap v1955: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 50 op/s; 32804/194666 objects misplaced (16.851%); 19 MiB/s, 334 objects/s recovering 2024-04-29T14:40:43.641 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:43.740 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:40:43.741 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:40:44.208 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:40:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:44.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:40:44.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:40:44.350 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:44.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:40:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:44.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:40:44.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:40:44.510 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:45.438 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:40:45.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:45.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:40:45.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:40:45.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:45 smithi194 ceph-mon[44089]: pgmap v1956: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 32252/194564 objects misplaced (16.577%); 18 MiB/s, 292 objects/s recovering 2024-04-29T14:40:45.582 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:45 smithi040 ceph-mon[39581]: pgmap v1956: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 32252/194564 objects misplaced (16.577%); 18 MiB/s, 292 objects/s recovering 2024-04-29T14:40:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:45.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:45 smithi121 ceph-mon[44533]: pgmap v1956: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 32252/194564 objects misplaced (16.577%); 18 MiB/s, 292 objects/s recovering 2024-04-29T14:40:46.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:46.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:46.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:46.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:47.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:47 smithi194 ceph-mon[44089]: pgmap v1957: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 49 op/s; 31470/194380 objects misplaced (16.190%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:40:47.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:47 smithi194 ceph-mon[44089]: osdmap e758: 12 total, 12 up, 12 in 2024-04-29T14:40:47.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:47 smithi040 ceph-mon[39581]: pgmap v1957: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 49 op/s; 31470/194380 objects misplaced (16.190%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:40:47.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:47 smithi040 ceph-mon[39581]: osdmap e758: 12 total, 12 up, 12 in 2024-04-29T14:40:47.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:47 smithi121 ceph-mon[44533]: pgmap v1957: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 49 op/s; 31470/194380 objects misplaced (16.190%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:40:47.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:47 smithi121 ceph-mon[44533]: osdmap e758: 12 total, 12 up, 12 in 2024-04-29T14:40:48.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:48 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:48.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:48 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:48.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:48 smithi194 ceph-mon[44089]: osdmap e759: 12 total, 12 up, 12 in 2024-04-29T14:40:48.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:48 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:48.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:48 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:48.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:48 smithi040 ceph-mon[39581]: osdmap e759: 12 total, 12 up, 12 in 2024-04-29T14:40:48.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:48 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:48.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:48 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:48.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:48 smithi121 ceph-mon[44533]: osdmap e759: 12 total, 12 up, 12 in 2024-04-29T14:40:49.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:49 smithi040 ceph-mon[39581]: pgmap v1960: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 40 op/s; 31235/194364 objects misplaced (16.070%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:40:49.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3649315741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:40:49.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:49 smithi040 ceph-mon[39581]: from='client.37001 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:40:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:49 smithi194 ceph-mon[44089]: pgmap v1960: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 40 op/s; 31235/194364 objects misplaced (16.070%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:40:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3649315741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:40:49.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:49 smithi194 ceph-mon[44089]: from='client.37001 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:40:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:49 smithi121 ceph-mon[44533]: pgmap v1960: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 40 op/s; 31235/194364 objects misplaced (16.070%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:40:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3649315741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:40:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:49 smithi121 ceph-mon[44533]: from='client.37001 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:40:50.156 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:40:50.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:50 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:40:50.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:50 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:40:50.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:50 smithi194 ceph-mon[44089]: from='client.37001 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:40:50.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:50 smithi194 ceph-mon[44089]: osdmap e760: 12 total, 12 up, 12 in 2024-04-29T14:40:50.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:50 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:40:50.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:50 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:40:50.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:50 smithi040 ceph-mon[39581]: from='client.37001 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:40:50.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:50 smithi040 ceph-mon[39581]: osdmap e760: 12 total, 12 up, 12 in 2024-04-29T14:40:50.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:50 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:40:50.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:50 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:40:50.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:50 smithi121 ceph-mon[44533]: from='client.37001 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:40:50.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:50 smithi121 ceph-mon[44533]: osdmap e760: 12 total, 12 up, 12 in 2024-04-29T14:40:51.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:51.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:51.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:51 smithi194 ceph-mon[44089]: pgmap v1962: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 27 op/s; 30223/194234 objects misplaced (15.560%); 21 MiB/s, 335 objects/s recovering 2024-04-29T14:40:51.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:51.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:51.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:51 smithi040 ceph-mon[39581]: pgmap v1962: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 27 op/s; 30223/194234 objects misplaced (15.560%); 21 MiB/s, 335 objects/s recovering 2024-04-29T14:40:51.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:51.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:51.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:51 smithi121 ceph-mon[44533]: pgmap v1962: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 27 op/s; 30223/194234 objects misplaced (15.560%); 21 MiB/s, 335 objects/s recovering 2024-04-29T14:40:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:40:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:40:52.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:40:52.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:40:52.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:40:52.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:40:53.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:53 smithi194 ceph-mon[44089]: pgmap v1963: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 29 op/s; 29523/194022 objects misplaced (15.216%); 20 MiB/s, 322 objects/s recovering 2024-04-29T14:40:53.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:53 smithi194 ceph-mon[44089]: osdmap e761: 12 total, 12 up, 12 in 2024-04-29T14:40:53.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:53 smithi040 ceph-mon[39581]: pgmap v1963: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 29 op/s; 29523/194022 objects misplaced (15.216%); 20 MiB/s, 322 objects/s recovering 2024-04-29T14:40:53.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:53 smithi040 ceph-mon[39581]: osdmap e761: 12 total, 12 up, 12 in 2024-04-29T14:40:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:53 smithi121 ceph-mon[44533]: pgmap v1963: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 29 op/s; 29523/194022 objects misplaced (15.216%); 20 MiB/s, 322 objects/s recovering 2024-04-29T14:40:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:53 smithi121 ceph-mon[44533]: osdmap e761: 12 total, 12 up, 12 in 2024-04-29T14:40:55.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:55 smithi194 ceph-mon[44089]: pgmap v1965: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 34 op/s; 28914/193886 objects misplaced (14.913%); 21 MiB/s, 333 objects/s recovering 2024-04-29T14:40:55.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:55 smithi040 ceph-mon[39581]: pgmap v1965: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 34 op/s; 28914/193886 objects misplaced (14.913%); 21 MiB/s, 333 objects/s recovering 2024-04-29T14:40:55.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:55 smithi121 ceph-mon[44533]: pgmap v1965: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 34 op/s; 28914/193886 objects misplaced (14.913%); 21 MiB/s, 333 objects/s recovering 2024-04-29T14:40:57.423 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:40:57.441 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:40:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:40:57.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:40:57.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:40:57.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:57 smithi194 ceph-mon[44089]: pgmap v1966: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 41 op/s; 28006/193698 objects misplaced (14.459%); 25 MiB/s, 401 objects/s recovering 2024-04-29T14:40:57.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:57 smithi040 ceph-mon[39581]: pgmap v1966: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 41 op/s; 28006/193698 objects misplaced (14.459%); 25 MiB/s, 401 objects/s recovering 2024-04-29T14:40:57.602 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:40:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:57 smithi121 ceph-mon[44533]: pgmap v1966: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 41 op/s; 28006/193698 objects misplaced (14.459%); 25 MiB/s, 401 objects/s recovering 2024-04-29T14:40:58.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:40:58.807 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:40:58.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:40:58 smithi194 ceph-mon[44089]: pgmap v1967: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 27718/193576 objects misplaced (14.319%); 18 MiB/s, 283 objects/s recovering 2024-04-29T14:40:58.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:40:58 smithi040 ceph-mon[39581]: pgmap v1967: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 27718/193576 objects misplaced (14.319%); 18 MiB/s, 283 objects/s recovering 2024-04-29T14:40:58.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:40:58 smithi121 ceph-mon[44533]: pgmap v1967: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 37 op/s; 27718/193576 objects misplaced (14.319%); 18 MiB/s, 283 objects/s recovering 2024-04-29T14:41:01.019 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:01.019 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:01.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:01 smithi121 ceph-mon[44533]: pgmap v1968: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 44 op/s; 26609/193352 objects misplaced (13.762%); 22 MiB/s, 358 objects/s recovering 2024-04-29T14:41:01.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:01 smithi194 ceph-mon[44089]: pgmap v1968: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 44 op/s; 26609/193352 objects misplaced (13.762%); 22 MiB/s, 358 objects/s recovering 2024-04-29T14:41:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:01 smithi040 ceph-mon[39581]: pgmap v1968: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 44 op/s; 26609/193352 objects misplaced (13.762%); 22 MiB/s, 358 objects/s recovering 2024-04-29T14:41:03.766 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:03 smithi040 ceph-mon[39581]: pgmap v1969: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 50 op/s; 25829/193020 objects misplaced (13.382%); 23 MiB/s, 366 objects/s recovering 2024-04-29T14:41:03.803 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:03.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:03.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:03.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:03 smithi194 ceph-mon[44089]: pgmap v1969: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 50 op/s; 25829/193020 objects misplaced (13.382%); 23 MiB/s, 366 objects/s recovering 2024-04-29T14:41:03.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:03 smithi121 ceph-mon[44533]: pgmap v1969: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 50 op/s; 25829/193020 objects misplaced (13.382%); 23 MiB/s, 366 objects/s recovering 2024-04-29T14:41:03.913 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:03.940 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:03.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:04.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:04.089 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:04.111 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:04.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:04.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:04.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:04 smithi121 ceph-mon[44533]: pgmap v1970: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 25168/192866 objects misplaced (13.049%); 23 MiB/s, 363 objects/s recovering 2024-04-29T14:41:04.821 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:04 smithi194 ceph-mon[44089]: pgmap v1970: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 25168/192866 objects misplaced (13.049%); 23 MiB/s, 363 objects/s recovering 2024-04-29T14:41:04.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:04 smithi040 ceph-mon[39581]: pgmap v1970: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 25168/192866 objects misplaced (13.049%); 23 MiB/s, 363 objects/s recovering 2024-04-29T14:41:05.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:41:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:05.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:05.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:05.772 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:05.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:05.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:05.811 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:41:05.896 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:05.923 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:05.943 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:05.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:06.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:41:06.074 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:06.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:41:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:06.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:41:06.198 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:06.225 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:06.245 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:06.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:06.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:06.375 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:07.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:07 smithi040 ceph-mon[39581]: pgmap v1971: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 24051/192722 objects misplaced (12.480%); 25 MiB/s, 402 objects/s recovering 2024-04-29T14:41:07.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:07 smithi194 ceph-mon[44089]: pgmap v1971: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 24051/192722 objects misplaced (12.480%); 25 MiB/s, 402 objects/s recovering 2024-04-29T14:41:07.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:07 smithi121 ceph-mon[44533]: pgmap v1971: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 48 op/s; 24051/192722 objects misplaced (12.480%); 25 MiB/s, 402 objects/s recovering 2024-04-29T14:41:08.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:08.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:08.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:41:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:09.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:09 smithi121 ceph-mon[44533]: pgmap v1972: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 23740/192610 objects misplaced (12.325%); 22 MiB/s, 351 objects/s recovering 2024-04-29T14:41:09.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:09 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:09.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:09 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:09 smithi194 ceph-mon[44089]: pgmap v1972: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 23740/192610 objects misplaced (12.325%); 22 MiB/s, 351 objects/s recovering 2024-04-29T14:41:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:09 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:09.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:09 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:09 smithi040 ceph-mon[39581]: pgmap v1972: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 45 op/s; 23740/192610 objects misplaced (12.325%); 22 MiB/s, 351 objects/s recovering 2024-04-29T14:41:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:09 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:09 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:09.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:09 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:41:09.470+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e762 definitely_dead 0 2024-04-29T14:41:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:10 smithi194 ceph-mon[44089]: osdmap e762: 12 total, 12 up, 12 in 2024-04-29T14:41:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3630311962' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:41:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:10 smithi194 ceph-mon[44089]: from='client.37004 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:41:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:10 smithi194 ceph-mon[44089]: Client client.admin marked osd.3 out, while it was still marked up 2024-04-29T14:41:10.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:10 smithi040 ceph-mon[39581]: osdmap e762: 12 total, 12 up, 12 in 2024-04-29T14:41:10.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:10.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:10.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3630311962' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:41:10.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:10 smithi040 ceph-mon[39581]: from='client.37004 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:41:10.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:10 smithi040 ceph-mon[39581]: Client client.admin marked osd.3 out, while it was still marked up 2024-04-29T14:41:10.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:10 smithi121 ceph-mon[44533]: osdmap e762: 12 total, 12 up, 12 in 2024-04-29T14:41:10.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:10.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:10.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3630311962' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:41:10.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:10 smithi121 ceph-mon[44533]: from='client.37004 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-29T14:41:10.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:10 smithi121 ceph-mon[44533]: Client client.admin marked osd.3 out, while it was still marked up 2024-04-29T14:41:10.853 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:10.853 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.3. 2024-04-29T14:41:11.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:11 smithi121 ceph-mon[44533]: pgmap v1974: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 42 op/s; 22444/192496 objects misplaced (11.659%); 26 MiB/s, 413 objects/s recovering 2024-04-29T14:41:11.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:11 smithi121 ceph-mon[44533]: from='client.37004 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-29T14:41:11.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:11 smithi121 ceph-mon[44533]: osdmap e763: 12 total, 12 up, 11 in 2024-04-29T14:41:11.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:11 smithi121 ceph-mon[44533]: 7.d scrub starts 2024-04-29T14:41:11.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:11 smithi194 ceph-mon[44089]: pgmap v1974: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 42 op/s; 22444/192496 objects misplaced (11.659%); 26 MiB/s, 413 objects/s recovering 2024-04-29T14:41:11.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:11 smithi194 ceph-mon[44089]: from='client.37004 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-29T14:41:11.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:11 smithi194 ceph-mon[44089]: osdmap e763: 12 total, 12 up, 11 in 2024-04-29T14:41:11.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:11 smithi194 ceph-mon[44089]: 7.d scrub starts 2024-04-29T14:41:11.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:41:11.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:11 smithi040 ceph-mon[39581]: pgmap v1974: 17 pgs: 3 active+recovering+remapped, 3 active+recovering, 11 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 42 op/s; 22444/192496 objects misplaced (11.659%); 26 MiB/s, 413 objects/s recovering 2024-04-29T14:41:11.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:11 smithi040 ceph-mon[39581]: from='client.37004 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-29T14:41:11.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:11 smithi040 ceph-mon[39581]: osdmap e763: 12 total, 12 up, 11 in 2024-04-29T14:41:11.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:11 smithi040 ceph-mon[39581]: 7.d scrub starts 2024-04-29T14:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:12.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:12 smithi121 ceph-mon[44533]: osdmap e764: 12 total, 12 up, 11 in 2024-04-29T14:41:12.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:12 smithi121 ceph-mon[44533]: 7.d scrub ok 2024-04-29T14:41:12.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:12 smithi194 ceph-mon[44089]: osdmap e764: 12 total, 12 up, 11 in 2024-04-29T14:41:12.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:12 smithi194 ceph-mon[44089]: 7.d scrub ok 2024-04-29T14:41:12.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:12 smithi040 ceph-mon[39581]: osdmap e764: 12 total, 12 up, 11 in 2024-04-29T14:41:12.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:12 smithi040 ceph-mon[39581]: 7.d scrub ok 2024-04-29T14:41:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:41:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:13.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:41:13.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:13.346 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:13.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:13 smithi194 ceph-mon[44089]: pgmap v1977: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 1 peering, 2 active+recovering+remapped, 2 active+recovering, 10 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 43 op/s; 17592/192194 objects misplaced (9.153%); 24 MiB/s, 384 objects/s recovering 2024-04-29T14:41:13.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:13 smithi194 ceph-mon[44089]: osdmap e765: 12 total, 12 up, 11 in 2024-04-29T14:41:13.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:13 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:41:13.586 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:13.586 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:13.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:13 smithi121 ceph-mon[44533]: pgmap v1977: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 1 peering, 2 active+recovering+remapped, 2 active+recovering, 10 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 43 op/s; 17592/192194 objects misplaced (9.153%); 24 MiB/s, 384 objects/s recovering 2024-04-29T14:41:13.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:13 smithi121 ceph-mon[44533]: osdmap e765: 12 total, 12 up, 11 in 2024-04-29T14:41:13.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:13 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:41:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:13 smithi040 ceph-mon[39581]: pgmap v1977: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 1 peering, 2 active+recovering+remapped, 2 active+recovering, 10 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 43 op/s; 17592/192194 objects misplaced (9.153%); 24 MiB/s, 384 objects/s recovering 2024-04-29T14:41:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:13 smithi040 ceph-mon[39581]: osdmap e765: 12 total, 12 up, 11 in 2024-04-29T14:41:13.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:13 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:41:14.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:41:14.550 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:14.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:14 smithi194 ceph-mon[44089]: osdmap e766: 12 total, 12 up, 11 in 2024-04-29T14:41:14.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:14 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:41:14.571 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:14 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:41:14.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:14 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:41:14.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:14 smithi040 ceph-mon[39581]: osdmap e766: 12 total, 12 up, 11 in 2024-04-29T14:41:14.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:14 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:41:14.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:14 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:41:14.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:14 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:41:14.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:14 smithi121 ceph-mon[44533]: osdmap e766: 12 total, 12 up, 11 in 2024-04-29T14:41:14.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:14 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:41:14.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:14 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:41:14.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:14 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:41:15.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:15 smithi121 ceph-mon[44533]: pgmap v1980: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 1 peering, 1 active+recovering+remapped, 3 active+recovering, 10 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 55 op/s; 14592/192050 objects misplaced (7.598%); 13 MiB/s, 245 objects/s recovering 2024-04-29T14:41:15.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:15 smithi121 ceph-mon[44533]: osdmap e767: 12 total, 12 up, 11 in 2024-04-29T14:41:15.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:15 smithi121 ceph-mon[44533]: 7.1 scrub starts 2024-04-29T14:41:15.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:15 smithi121 ceph-mon[44533]: 7.1 scrub ok 2024-04-29T14:41:15.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:15 smithi194 ceph-mon[44089]: pgmap v1980: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 1 peering, 1 active+recovering+remapped, 3 active+recovering, 10 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 55 op/s; 14592/192050 objects misplaced (7.598%); 13 MiB/s, 245 objects/s recovering 2024-04-29T14:41:15.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:15 smithi194 ceph-mon[44089]: osdmap e767: 12 total, 12 up, 11 in 2024-04-29T14:41:15.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:15 smithi194 ceph-mon[44089]: 7.1 scrub starts 2024-04-29T14:41:15.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:15 smithi194 ceph-mon[44089]: 7.1 scrub ok 2024-04-29T14:41:15.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:15 smithi040 ceph-mon[39581]: pgmap v1980: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 1 peering, 1 active+recovering+remapped, 3 active+recovering, 10 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 55 op/s; 14592/192050 objects misplaced (7.598%); 13 MiB/s, 245 objects/s recovering 2024-04-29T14:41:15.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:15 smithi040 ceph-mon[39581]: osdmap e767: 12 total, 12 up, 11 in 2024-04-29T14:41:15.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:15 smithi040 ceph-mon[39581]: 7.1 scrub starts 2024-04-29T14:41:15.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:15 smithi040 ceph-mon[39581]: 7.1 scrub ok 2024-04-29T14:41:16.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:16 smithi121 ceph-mon[44533]: 7.c scrub starts 2024-04-29T14:41:16.335 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:16.335 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:16.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:16 smithi194 ceph-mon[44089]: 7.c scrub starts 2024-04-29T14:41:16.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:16 smithi040 ceph-mon[39581]: 7.c scrub starts 2024-04-29T14:41:17.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:41:17.360 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:17.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: pgmap v1982: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 73 op/s; 11169/191604 objects misplaced (5.829%); 6.8 MiB/s, 221 objects/s recovering 2024-04-29T14:41:17.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: osdmap e768: 12 total, 12 up, 11 in 2024-04-29T14:41:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: 7.c scrub ok 2024-04-29T14:41:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:41:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: pgmap v1982: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 73 op/s; 11169/191604 objects misplaced (5.829%); 6.8 MiB/s, 221 objects/s recovering 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: osdmap e768: 12 total, 12 up, 11 in 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: 7.c scrub ok 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:41:17.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: pgmap v1982: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 73 op/s; 11169/191604 objects misplaced (5.829%); 6.8 MiB/s, 221 objects/s recovering 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: osdmap e768: 12 total, 12 up, 11 in 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: 7.c scrub ok 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:41:17.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:41:18.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:18 smithi194 ceph-mon[44089]: osdmap e769: 12 total, 12 up, 11 in 2024-04-29T14:41:18.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:18 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:41:18.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:18 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:41:18.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:18 smithi121 ceph-mon[44533]: osdmap e769: 12 total, 12 up, 11 in 2024-04-29T14:41:18.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:18 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:41:18.617 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:18 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:41:18.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:18 smithi040 ceph-mon[39581]: osdmap e769: 12 total, 12 up, 11 in 2024-04-29T14:41:18.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:18 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:41:18.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:18 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:41:19.189 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:19.189 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:19.576 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:19 smithi121 ceph-mon[44533]: pgmap v1985: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 11100/191386 objects misplaced (5.800%); 4.2 MiB/s, 181 objects/s recovering 2024-04-29T14:41:19.576 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:19 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:41:19.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:19 smithi194 ceph-mon[44089]: pgmap v1985: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 11100/191386 objects misplaced (5.800%); 4.2 MiB/s, 181 objects/s recovering 2024-04-29T14:41:19.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:19 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:41:19.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:41:19.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:19 smithi040 ceph-mon[39581]: pgmap v1985: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 11100/191386 objects misplaced (5.800%); 4.2 MiB/s, 181 objects/s recovering 2024-04-29T14:41:19.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:19 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:41:20.155 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:20.668 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:20 smithi121 ceph-mon[44533]: 7.3 deep-scrub starts 2024-04-29T14:41:20.816 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:20.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:20 smithi194 ceph-mon[44089]: 7.3 deep-scrub starts 2024-04-29T14:41:20.824 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:20 smithi040 ceph-mon[39581]: 7.3 deep-scrub starts 2024-04-29T14:41:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:20.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:20.925 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:20.953 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:20.972 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:20.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:21.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:21.106 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:21.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:21.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:21.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:21.282 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:21.301 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:21.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:21.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:41:21.447 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:21.549 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:41:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:21.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:41:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:21 smithi194 ceph-mon[44089]: pgmap v1986: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 11100/190904 objects misplaced (5.814%); 2.1 MiB/s, 116 objects/s recovering 2024-04-29T14:41:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:41:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:41:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:21 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:41:21.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:21 smithi040 ceph-mon[39581]: pgmap v1986: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 11100/190904 objects misplaced (5.814%); 2.1 MiB/s, 116 objects/s recovering 2024-04-29T14:41:21.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:41:21.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:41:21.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:21 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:41:21.656 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:21.683 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:21.710 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:21.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:21.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:21 smithi121 ceph-mon[44533]: pgmap v1986: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 11100/190904 objects misplaced (5.814%); 2.1 MiB/s, 116 objects/s recovering 2024-04-29T14:41:21.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:41:21.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:41:21.798 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:21 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:41:21.814 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:21.841 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:21.866 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:21.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:21.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:22.000 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:22.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:22.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:22.124 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:41:22.124 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:41:22.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:41:22.124 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:41:22.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.124 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16809.0:248599 6.3 e614/534 6:c047ea77:::benchmark_data_smithi194_134202_object95746:head v 614'15582, mlcod=614'15580)", 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.352801+0000", 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 479.76726946500003, 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34023397100000002, 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.352801+0000", 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.352801+0000", 2024-04-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.352803+0000", 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9e-06 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.352813+0000", 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6320000000000003e-06 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.352826+0000", 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3256e-05 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.352835+0000", 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7139999999999999e-06 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.353817+0000", 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00098240400000000009 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.353844+0000", 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7078999999999999e-05 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.693035+0000", 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33919098600000003 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "rep_scrubmap(6.8 e604 from shard 11)", 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.359022+0000", 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 479.761048303, 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31928887500000003, 2024-04-29T14:41:22.128 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.359022+0000", 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.359022+0000", 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.359024+0000", 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2389999999999999e-06 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.359053+0000", 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9286999999999999e-05 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.359063+0000", 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0159e-05 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.359070+0000", 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7170000000000002e-06 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.678286+0000", 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.319216369 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.678311+0000", 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5103999999999999e-05 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:248616 6.c 6:3b959721:::benchmark_data_smithi194_134202_object95763:head [delete] snapc 0=[] ondisk+write+known_if_redirected e615)", 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:22.372924+0000", 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 479.74714662000002, 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30541310399999999, 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 248616 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.372924+0000", 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.372924+0000", 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.372928+0000", 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.805e-06 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.372941+0000", 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3173000000000001e-05 2024-04-29T14:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.372956+0000", 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4842e-05 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.372965+0000", 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4089999999999992e-06 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.373027+0000", 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2248000000000004e-05 2024-04-29T14:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.373166+0000", 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000138491 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.373285+0000", 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011942699999999999 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.407000+0000", 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033714355000000001 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.678241+0000", 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27124151899999999 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.678291+0000", 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9277999999999998e-05 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:22.678337+0000", 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6557000000000002e-05 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:259860 6.c 6:3699b273:::benchmark_data_smithi194_134202_object107007:head [delete] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:42.318605+0000", 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 459.80146606, 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22541877099999999, 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 259860 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318605+0000", 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318605+0000", 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318606+0000", 2024-04-29T14:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.666e-06 2024-04-29T14:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318614+0000", 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9219999999999997e-06 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318623+0000", 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0960000000000007e-06 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318630+0000", 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8050000000000001e-06 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318673+0000", 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2536000000000001e-05 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318761+0000", 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8387999999999995e-05 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.318857+0000", 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5613999999999996e-05 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330436+0000", 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011579478000000001 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.543929+0000", 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21349290800000001 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.543982+0000", 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3192000000000002e-05 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.544023+0000", 2024-04-29T14:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1165999999999999e-05 2024-04-29T14:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:259867 6.c 6:33ecb1a2:::benchmark_data_smithi194_134202_object107014:head [delete] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:42.322366+0000", 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 459.79770409700001, 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.221773939, 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 259867 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322366+0000", 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322366+0000", 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322368+0000", 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5889999999999999e-06 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322373+0000", 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8559999999999998e-06 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322383+0000", 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0176999999999999e-05 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322390+0000", 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2389999999999996e-06 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322647+0000", 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000256204 2024-04-29T14:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322734+0000", 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7639999999999994e-05 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322813+0000", 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8651000000000003e-05 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.333286+0000", 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010473174 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.544063+0000", 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21077744100000001 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.544108+0000", 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4641000000000002e-05 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.544140+0000", 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2326999999999997e-05 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:259868 6.c 6:306d1d76:::benchmark_data_smithi194_134202_object107015:head [delete] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:42.322809+0000", 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 459.797262004, 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22144398800000001, 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 259868 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322809+0000", 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322809+0000", 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322810+0000", 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3179999999999999e-06 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322814+0000", 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5889999999999996e-06 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322823+0000", 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7120000000000006e-06 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.322831+0000", 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1780000000000003e-06 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.326378+0000", 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003547077 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.326490+0000", 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111919 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.326593+0000", 2024-04-29T14:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102224 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.333291+0000", 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066984269999999999 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.544180+0000", 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21088944100000001 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.544222+0000", 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1050000000000002e-05 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.544253+0000", 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1053000000000002e-05 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:259874 6.c 6:3f00a307:::benchmark_data_smithi194_134202_object107021:head [delete] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:42.330353+0000", 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 459.78971749599998, 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.222029316, 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 259874 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330353+0000", 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330353+0000", 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330355+0000", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4449999999999999e-06 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330359+0000", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0670000000000002e-06 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330366+0000", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2899999999999997e-06 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330371+0000", 2024-04-29T14:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.468e-06 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330452+0000", 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0461000000000001e-05 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330526+0000", 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4101000000000001e-05 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.330594+0000", 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7615999999999998e-05 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.333445+0000", 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002851344 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.552297+0000", 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21885164400000001 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.552343+0000", 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6940000000000001e-05 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:42.552382+0000", 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8940000000000003e-05 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:265596 6.c 6:369eb719:::benchmark_data_smithi194_134202_object112743:head [delete] snapc 0=[] ondisk+write+known_if_redirected e636)", 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:47.182611+0000", 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 454.9374593, 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21277606299999999, 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 265596 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182611+0000", 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182611+0000", 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182614+0000", 2024-04-29T14:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2699999999999999e-06 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182621+0000", 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.956e-06 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182640+0000", 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9632000000000001e-05 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182649+0000", 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1859999999999998e-06 2024-04-29T14:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182698+0000", 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9054000000000002e-05 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182835+0000", 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013664800000000001 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.182960+0000", 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012499599999999999 2024-04-29T14:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.198259+0000", 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015299424000000001 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395264+0000", 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.197004342 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395318+0000", 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4527000000000002e-05 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395387+0000", 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9028000000000004e-05 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:265607 6.c 6:38cc025c:::benchmark_data_smithi194_134202_object112754:head [delete] snapc 0=[] ondisk+write+known_if_redirected e636)", 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:47.186877+0000", 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 454.93319389200002, 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.208640097, 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 265607 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.186877+0000", 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.186877+0000", 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.186879+0000", 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.858e-06 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.186885+0000", 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9710000000000001e-06 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.186896+0000", 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1386000000000001e-05 2024-04-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.186905+0000", 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8890000000000008e-06 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.187984+0000", 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010795220000000001 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.188086+0000", 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101798 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.188166+0000", 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9832000000000004e-05 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.203385+0000", 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015219264 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395432+0000", 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19204721899999999 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395476+0000", 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3764999999999999e-05 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395517+0000", 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0593000000000003e-05 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:265614 6.c 6:38044331:::benchmark_data_smithi194_134202_object112761:head [delete] snapc 0=[] ondisk+write+known_if_redirected e636)", 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:47.192087+0000", 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 454.92798405399998, 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20352044499999999, 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 265614 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.192087+0000", 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.192087+0000", 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.192089+0000", 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2249999999999999e-06 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.192096+0000", 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.922e-06 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.192109+0000", 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2876e-05 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.192118+0000", 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0659999999999993e-06 2024-04-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.193133+0000", 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010156970000000001 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.193235+0000", 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101915 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.193337+0000", 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101964 2024-04-29T14:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.203395+0000", 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010057956999999999 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395552+0000", 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19215670600000001 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395578+0000", 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5780000000000001e-05 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.395607+0000", 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9337000000000001e-05 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16809.0:265620 6.c 6:389323a7:::benchmark_data_smithi194_134202_object112767:head [delete] snapc 0=[] ondisk+write+known_if_redirected e636)", 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:33:47.196496+0000", 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 454.92357486899999, 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.222995424, 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16809", 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 265620 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.196496+0000", 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.196496+0000", 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.196497+0000", 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.142e-06 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.196502+0000", 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8879999999999996e-06 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.196510+0000", 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0020000000000006e-06 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.196517+0000", 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3259999999999998e-06 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.198809+0000", 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002292021 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.198921+0000", 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112012 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.199027+0000", 2024-04-29T14:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106246 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.203491+0000", 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044638769999999998 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.419396+0000", 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.215904811 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.419441+0000", 2024-04-29T14:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5277e-05 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:33:47.419491+0000", 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9821999999999998e-05 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:69592 7.d 7:bdeaf1f0:::benchmark_data_smithi194_147200_object69591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e668)", 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:25.803783+0000", 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 356.31628752400002, 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20736255000000001, 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69592 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803783+0000", 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803783+0000", 2024-04-29T14:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803784+0000", 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.78e-07 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803845+0000", 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0943999999999998e-05 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803854+0000", 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0899999999999994e-06 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803860+0000", 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6470000000000003e-06 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803900+0000", 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9833000000000003e-05 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.803991+0000", 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1914000000000001e-05 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-29T14:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.804075+0000", 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3030999999999998e-05 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.807216+0000", 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031415940000000002 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:26.011084+0000", 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20386817199999999 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:26.011118+0000", 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3469000000000001e-05 2024-04-29T14:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:26.011146+0000", 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7878e-05 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:69596 7.d 7:b8131e15:::benchmark_data_smithi194_147200_object69595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e668)", 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:25.804923+0000", 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 356.315147728, 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20834947400000001, 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69596 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.804923+0000", 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.804923+0000", 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.804923+0000", 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8999999999999996e-07 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.805001+0000", 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7061e-05 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.805007+0000", 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1680000000000001e-06 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.805011+0000", 2024-04-29T14:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9920000000000001e-06 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.805681+0000", 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066989399999999998 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.805767+0000", 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6225999999999997e-05 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.805870+0000", 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010324599999999999 2024-04-29T14:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:25.807348+0000", 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001477625 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:26.013133+0000", 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20578485199999999 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:26.013228+0000", 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5691000000000003e-05 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:26.013272+0000", 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4128999999999997e-05 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:73408 7.c 7:36bd074e:::benchmark_data_smithi194_147200_object73407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e669)", 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:28.050842+0000", 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 354.06922880899998, 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.207153222, 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 73408 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.050842+0000", 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.050842+0000", 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.050843+0000", 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.102e-06 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.050892+0000", 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9345999999999998e-05 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.050900+0000", 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7999999999999999e-06 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.050923+0000", 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3010000000000002e-05 2024-04-29T14:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.050969+0000", 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6261000000000003e-05 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051059+0000", 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9429e-05 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051142+0000", 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3400999999999997e-05 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.056219+0000", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050766780000000003 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.257856+0000", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201637283 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.257914+0000", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7853000000000003e-05 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.257995+0000", 2024-04-29T14:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1058999999999998e-05 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:73412 7.c 7:3421f91d:::benchmark_data_smithi194_147200_object73411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e669)", 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:28.051517+0000", 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 354.06855388000002, 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20664407100000001, 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 73412 2024-04-29T14:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051517+0000", 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051517+0000", 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051522+0000", 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7690000000000002e-06 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051581+0000", 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8845999999999999e-05 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051590+0000", 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8740000000000001e-06 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051596+0000", 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.105e-06 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.052763+0000", 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011668959999999999 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.052895+0000", 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013138200000000001 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.052978+0000", 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3615999999999994e-05 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.056224+0000", 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032454099999999998 2024-04-29T14:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.258050+0000", 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20182628799999999 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.258105+0000", 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5169e-05 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.258161+0000", 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5716e-05 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:73418 7.c 7:3ddba8cf:::benchmark_data_smithi194_147200_object73417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e669)", 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:28.051941+0000", 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 354.06812926399999, 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.211661183, 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 73418 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051941+0000", 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051941+0000", 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.051942+0000", 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9199999999999999e-07 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.052119+0000", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017716800000000001 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.052125+0000", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8810000000000001e-06 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.052130+0000", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1459999999999998e-06 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.054761+0000", 2024-04-29T14:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00263101 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.054862+0000", 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101437 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.054968+0000", 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105802 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.056316+0000", 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001348721 2024-04-29T14:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.263531+0000", 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20721496 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.263565+0000", 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3754000000000002e-05 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.263603+0000", 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7311999999999998e-05 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:73429 7.c 7:30dc6f74:::benchmark_data_smithi194_147200_object73428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e669)", 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:28.060654+0000", 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 354.05941670700003, 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204651584, 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 73429 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060654+0000", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060654+0000", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060655+0000", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.279e-06 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060681+0000", 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5944000000000001e-05 2024-04-29T14:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060691+0000", 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0137e-05 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060698+0000", 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8990000000000004e-06 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060736+0000", 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8129e-05 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060826+0000", 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9306000000000002e-05 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.060936+0000", 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110134 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.062506+0000", 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001570667 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.265242+0000", 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20273598100000001 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.265270+0000", 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7487999999999998e-05 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:28.265305+0000", 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5620000000000001e-05 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:97746 7.d 7:b69787ab:::benchmark_data_smithi194_147200_object97745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e675)", 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:43.512756+0000", 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 338.60731418199998, 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20806743699999999, 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 97746 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.512756+0000", 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.512756+0000", 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.512760+0000", 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0900000000000001e-06 2024-04-29T14:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.512874+0000", 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114492 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.512896+0000", 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2005999999999999e-05 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.512922+0000", 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6016999999999998e-05 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.512967+0000", 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4842000000000002e-05 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.513109+0000", 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014171299999999999 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.513254+0000", 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000145401 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.515348+0000", 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020938799999999998 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.720755+0000", 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.205407226 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.720796+0000", 2024-04-29T14:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0812000000000002e-05 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.720824+0000", 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7957999999999999e-05 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:97771 7.d 7:b0e7c19c:::benchmark_data_smithi194_147200_object97770:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e675)", 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:43.527006+0000", 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 338.59306472399999, 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20612461000000001, 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 97771 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527006+0000", 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527006+0000", 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527007+0000", 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.308e-06 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527048+0000", 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0738999999999997e-05 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527058+0000", 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5869999999999999e-06 2024-04-29T14:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527081+0000", 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3247e-05 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527134+0000", 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3174000000000003e-05 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527265+0000", 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013076900000000001 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-29T14:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.527415+0000", 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014984899999999999 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528823+0000", 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014085479999999999 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.733070+0000", 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20424695300000001 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.733103+0000", 2024-04-29T14:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3096000000000003e-05 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.733130+0000", 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7339999999999999e-05 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:97774 7.d 7:b9ca750e:::benchmark_data_smithi194_147200_object97773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e675)", 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:35:43.528782+0000", 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 338.591288453, 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20444785700000001, 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 97774 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528782+0000", 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528782+0000", 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528783+0000", 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.341e-06 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528828+0000", 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.498e-05 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528837+0000", 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8810000000000002e-06 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528843+0000", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7100000000000004e-06 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528882+0000", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9101999999999999e-05 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.528981+0000", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8610000000000006e-05 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.529119+0000", 2024-04-29T14:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000138567 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.530630+0000", 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015106 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.733169+0000", 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20253871600000001 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.733203+0000", 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4499999999999998e-05 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:35:43.733230+0000", 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6849999999999999e-05 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:41:22.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:22.270 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:41:22.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:41:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:41:22.311 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:41:22.311 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:41:22.311 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:41:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:22.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:41:22.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: 7.e scrub starts 2024-04-29T14:41:22.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2394084064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:22.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:41:22.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:22 smithi040 ceph-mon[39581]: pgmap v1987: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 11100/190496 objects misplaced (5.827%); 1.6 MiB/s, 88 objects/s recovering 2024-04-29T14:41:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:22.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:22.635 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:22.682 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:22.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:22.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:41:22.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: 7.e scrub starts 2024-04-29T14:41:22.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2394084064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:22.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:41:22.774 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:22 smithi121 ceph-mon[44533]: pgmap v1987: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 11100/190496 objects misplaced (5.827%); 1.6 MiB/s, 88 objects/s recovering 2024-04-29T14:41:22.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:22.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:41:22.817 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:22.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:41:22.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: 7.e scrub starts 2024-04-29T14:41:22.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2394084064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:22.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:41:22.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:22.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:22 smithi194 ceph-mon[44089]: pgmap v1987: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 11100/190496 objects misplaced (5.827%); 1.6 MiB/s, 88 objects/s recovering 2024-04-29T14:41:23.131 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:23.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:23.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:41:23.282 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:23.309 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:41:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:23.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:41:23.423 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:23.450 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:23.468 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:23.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:23.573 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:23.599 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:23.618 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:23.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:23.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:23 smithi121 ceph-mon[44533]: 7.e scrub ok 2024-04-29T14:41:23.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:23.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.748 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:23 smithi194 ceph-mon[44089]: 7.e scrub ok 2024-04-29T14:41:23.749 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:23.767 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:23.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:23.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:23 smithi040 ceph-mon[39581]: 7.e scrub ok 2024-04-29T14:41:23.872 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:23.898 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:23.917 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:23.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:24.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:41:24.055 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:24.081 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:41:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:24.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:41:24.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:41:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:24.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:41:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:24.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:41:24.673 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:24.700 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:24.718 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:24.720 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:24 smithi194 ceph-mon[44089]: pgmap v1988: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 11036/190094 objects misplaced (5.806%); 496 KiB/s, 16 objects/s recovering 2024-04-29T14:41:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:24.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:24.791 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:24 smithi121 ceph-mon[44533]: pgmap v1988: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 11036/190094 objects misplaced (5.806%); 496 KiB/s, 16 objects/s recovering 2024-04-29T14:41:24.822 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:24.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:24 smithi040 ceph-mon[39581]: pgmap v1988: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 94 op/s; 11036/190094 objects misplaced (5.806%); 496 KiB/s, 16 objects/s recovering 2024-04-29T14:41:24.855 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:24.878 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:24.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:24.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:25.036 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:25.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:25.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:25.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:41:25.171 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:25.192 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:25.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:25.206 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:41:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:41:25.207 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:41:25.207 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:41:25.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:25.328 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:25.351 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:25.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:25.454 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:25.485 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:25.487 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:41:25.499 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:25.505 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:25.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:25.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:25.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1377026827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:25.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:25.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1377026827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:25.871 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:25.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1377026827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:25.903 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:25.924 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:25.929 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:25.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:41:26.030 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:26.057 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:26.091 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8] out_osds: [3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:41:26.092 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:41:26.092 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-29T14:41:26.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:41:26.165 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:26.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:26.270 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:26.284 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:26.297 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:26.315 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:26.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:26.420 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:26.447 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:26.466 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:26.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:26.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:26.597 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:26.617 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:26.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:26.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:26.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:26 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:26.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1377026827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:41:26.750 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:26 smithi040 ceph-mon[39581]: osdmap e770: 12 total, 12 up, 11 in 2024-04-29T14:41:26.750 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:26 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:41:26.750 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:26 smithi040 ceph-mon[39581]: pgmap v1990: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 97 op/s; 11036/189656 objects misplaced (5.819%); 230 KiB/s, 11 objects/s recovering 2024-04-29T14:41:26.753 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:26 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1377026827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:41:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:26 smithi194 ceph-mon[44089]: osdmap e770: 12 total, 12 up, 11 in 2024-04-29T14:41:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:26 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:41:26.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:26 smithi194 ceph-mon[44089]: pgmap v1990: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 97 op/s; 11036/189656 objects misplaced (5.819%); 230 KiB/s, 11 objects/s recovering 2024-04-29T14:41:26.825 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:26 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:26.825 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1377026827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:41:26.825 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:26 smithi121 ceph-mon[44533]: osdmap e770: 12 total, 12 up, 11 in 2024-04-29T14:41:26.825 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:26 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:41:26.825 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:26 smithi121 ceph-mon[44533]: pgmap v1990: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 97 op/s; 11036/189656 objects misplaced (5.819%); 230 KiB/s, 11 objects/s recovering 2024-04-29T14:41:27.433 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:27.703 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:27.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:27.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:27.842 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:27.863 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:27.867 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:27.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:28.279 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:41:28.279 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:41:28.280 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:41:28.280 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:41:28.280 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:41:28.461 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:28.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:28.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:28.597 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:28.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:28.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:28.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:28 smithi121 ceph-mon[44533]: osdmap e771: 12 total, 12 up, 11 in 2024-04-29T14:41:28.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:28.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:28.731 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:28.760 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:28 smithi194 ceph-mon[44089]: osdmap e771: 12 total, 12 up, 11 in 2024-04-29T14:41:28.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:28.761 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:28.763 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:28.781 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:41:28.784 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:28.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:28.804 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:41:28.826 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:41:28.855 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:41:28.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:28 smithi040 ceph-mon[39581]: osdmap e771: 12 total, 12 up, 11 in 2024-04-29T14:41:28.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:28.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:28.876 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:41:28.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:28.898 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:41:28.914 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:28.931 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:41:28.933 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:28.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:28.961 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:41:28.986 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:41:29.007 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:41:29.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:41:29.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:41:29.033 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:41:29.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:29.064 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:41:29.066 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:41:29.067 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:41:29.068 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:41:29.068 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:41:29.068 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:41:29.075 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:29.286 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:29.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:41:29.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:29.422 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:29.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:29.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: pgmap v1992: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 10979/189492 objects misplaced (5.794%); 496 KiB/s, 7 objects/s recovering 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osdmap e772: 12 total, 12 up, 11 in 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2848198186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.654 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:29.685 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:29.705 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:29.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: pgmap v1992: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 10979/189492 objects misplaced (5.794%); 496 KiB/s, 7 objects/s recovering 2024-04-29T14:41:29.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osdmap e772: 12 total, 12 up, 11 in 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2848198186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.708 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:29.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:29.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: pgmap v1992: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 10979/189492 objects misplaced (5.794%); 496 KiB/s, 7 objects/s recovering 2024-04-29T14:41:29.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osdmap e772: 12 total, 12 up, 11 in 2024-04-29T14:41:29.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2848198186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:29.794 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.795 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:29.811 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:29.841 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:29.861 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:29.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:29.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:29.999 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:30.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:30.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:30 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:30.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:30.764 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:30.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:30.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:30 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:30.793 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:30.813 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:30.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:30.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:30.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:30 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:30.918 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:41:30.924 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:41:30.958 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:31.135 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:31.135 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:31.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:31.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:41:31.351 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:31.393 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:41:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:31.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:41:31.497 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:31.524 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:31.544 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:31.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:31.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:31 smithi040 ceph-mon[39581]: pgmap v1994: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 10979/189162 objects misplaced (5.804%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:31.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:31 smithi040 ceph-mon[39581]: from='client.17568 v1:172.21.15.40:0/2919674693' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:41:31.648 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:31.674 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:31.694 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:31.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:31.710 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":1994,"stamp":"2024-04-29T14:41:30.192356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6198854200,"num_objects":94581,"num_object_clones":0,"num_object_copies":189162,"num_objects_missing_on_primary":5083,"num_objects_missing":5083,"num_objects_degraded":0,"num_objects_misplaced":10979,"num_objects_unfound":0,"num_objects_dirty":94581,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":135422,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84878,"num_bytes_recovered":5466783456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139857,"ondisk_log_size":139857,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19804368,"kb_used_data":13369636,"kb_used_omap":672,"kb_used_meta":6433951,"kb_avail":1011392304,"statfs":{"total":1055945392128,"available":1035665719296,"internally_reserved":0,"allocated":13690507264,"data_stored":13681955929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":688465,"internal_metadata":6588366511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,3,2,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-30539776,"num_objects":-466,"num_object_clones":0,"num_object_copies":-932,"num_objects_missing_on_primary":-27,"num_objects_missing":-27,"num_objects_degraded":0,"num_objects_misplaced":-57,"num_objects_unfound":0,"num_objects_dirty":-466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":466,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001113"},"pg_stats":[{"pgid":"7.f","version":"769'8518","reported_seq":23353,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-29T14:41:25.220705+0000","last_change":"2024-04-29T14:40:29.956877+0000","last_active":"2024-04-29T14:41:25.220705+0000","last_peered":"2024-04-29T14:41:25.220705+0000","last_clean":"2024-04-29T14:41:25.220705+0000","last_became_active":"2024-04-29T14:38:40.422563+0000","last_became_peered":"2024-04-29T14:38:40.422563+0000","last_unstale":"2024-04-29T14:41:25.220705+0000","last_undegraded":"2024-04-29T14:41:25.220705+0000","last_fullsized":"2024-04-29T14:41:25.220705+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"755'8324","last_scrub_stamp":"2024-04-29T14:40:29.956766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:40:29.956766+0000","objects_scrubbed":6120,"log_size":8518,"log_dups_size":0,"ondisk_log_size":8518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:41:41.598767+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388366336,"num_objects":5926,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8518,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7222,"num_bytes_recovered":464846848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"771'15634","reported_seq":16307,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-04-29T14:41:28.092511+0000","last_change":"2024-04-29T14:41:22.402971+0000","last_active":"2024-04-29T14:41:28.092511+0000","last_peered":"2024-04-29T14:41:28.092511+0000","last_clean":"2024-04-29T14:41:28.092511+0000","last_became_active":"2024-04-29T14:34:49.387078+0000","last_became_peered":"2024-04-29T14:34:49.387078+0000","last_unstale":"2024-04-29T14:41:28.092511+0000","last_undegraded":"2024-04-29T14:41:28.092511+0000","last_fullsized":"2024-04-29T14:41:28.092511+0000","mapping_epoch":655,"log_start":"675'5600","ondisk_log_start":"675'5600","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'15606","last_scrub_stamp":"2024-04-29T14:41:22.402886+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:41:22.402886+0000","objects_scrubbed":6003,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:32.816133+0000","scrub_duration":639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391380992,"num_objects":5972,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"770'8447","reported_seq":21415,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-29T14:41:26.007121+0000","last_change":"2024-04-29T14:41:11.594141+0000","last_active":"2024-04-29T14:41:26.007121+0000","last_peered":"2024-04-29T14:41:26.007121+0000","last_clean":"2024-04-29T14:41:26.007121+0000","last_became_active":"2024-04-29T14:35:47.383003+0000","last_became_peered":"2024-04-29T14:35:47.383003+0000","last_unstale":"2024-04-29T14:41:26.007121+0000","last_undegraded":"2024-04-29T14:41:26.007121+0000","last_fullsized":"2024-04-29T14:41:26.007121+0000","mapping_epoch":676,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"764'8355","last_scrub_stamp":"2024-04-29T14:41:11.593851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:11.593851+0000","objects_scrubbed":5974,"log_size":8447,"log_dups_size":0,"ondisk_log_size":8447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:14.633277+0000","scrub_duration":637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385220632,"num_objects":5879,"num_object_clones":0,"num_object_copies":11758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8447,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6289,"num_bytes_recovered":412155904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"770'8437","reported_seq":19431,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-29T14:41:25.893614+0000","last_change":"2024-04-29T14:41:16.362353+0000","last_active":"2024-04-29T14:41:25.893614+0000","last_peered":"2024-04-29T14:41:25.893614+0000","last_clean":"2024-04-29T14:41:25.893614+0000","last_became_active":"2024-04-29T14:41:11.301869+0000","last_became_peered":"2024-04-29T14:41:11.301869+0000","last_unstale":"2024-04-29T14:41:25.893614+0000","last_undegraded":"2024-04-29T14:41:25.893614+0000","last_fullsized":"2024-04-29T14:41:25.893614+0000","mapping_epoch":763,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'8385","last_scrub_stamp":"2024-04-29T14:41:16.362214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:16.362214+0000","objects_scrubbed":5963,"log_size":8437,"log_dups_size":0,"ondisk_log_size":8437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:43.231672+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387383296,"num_objects":5911,"num_object_clones":0,"num_object_copies":11822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8437,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5265,"num_bytes_recovered":332988416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"770'8369","reported_seq":12752,"reported_epoch":770,"state":"active+recovering","last_fresh":"2024-04-29T14:41:27.287621+0000","last_change":"2024-04-29T14:36:34.055204+0000","last_active":"2024-04-29T14:41:27.287621+0000","last_peered":"2024-04-29T14:41:27.287621+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:36:33.135845+0000","last_became_peered":"2024-04-29T14:36:33.135845+0000","last_unstale":"2024-04-29T14:41:27.287621+0000","last_undegraded":"2024-04-29T14:41:27.287621+0000","last_fullsized":"2024-04-29T14:41:27.287621+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387514368,"num_objects":5913,"num_object_clones":0,"num_object_copies":11826,"num_objects_missing_on_primary":5083,"num_objects_missing":5083,"num_objects_degraded":0,"num_objects_misplaced":10979,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8369,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1670,"num_bytes_recovered":109445120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["10","11"],"object_location_counts":[{"shards":"1,5","objects":17},{"shards":"1,10,11","objects":813},{"shards":"10,11","objects":5083}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"769'15697","reported_seq":16300,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-29T14:41:24.268384+0000","last_change":"2024-04-29T14:39:56.554141+0000","last_active":"2024-04-29T14:41:24.268384+0000","last_peered":"2024-04-29T14:41:24.268384+0000","last_clean":"2024-04-29T14:41:24.268384+0000","last_became_active":"2024-04-29T14:34:49.387647+0000","last_became_peered":"2024-04-29T14:34:49.387647+0000","last_unstale":"2024-04-29T14:41:24.268384+0000","last_undegraded":"2024-04-29T14:41:24.268384+0000","last_fullsized":"2024-04-29T14:41:24.268384+0000","mapping_epoch":655,"log_start":"673'5600","ondisk_log_start":"673'5600","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"746'15389","last_scrub_stamp":"2024-04-29T14:39:56.554009+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:39:56.554009+0000","objects_scrubbed":6360,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396558336,"num_objects":6051,"num_object_clones":0,"num_object_copies":12102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8623,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"770'8476","reported_seq":37169,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-29T14:41:27.138155+0000","last_change":"2024-04-29T14:40:30.568318+0000","last_active":"2024-04-29T14:41:27.138155+0000","last_peered":"2024-04-29T14:41:27.138155+0000","last_clean":"2024-04-29T14:41:27.138155+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:41:27.138155+0000","last_undegraded":"2024-04-29T14:41:27.138155+0000","last_fullsized":"2024-04-29T14:41:27.138155+0000","mapping_epoch":756,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7744","last_scrub_stamp":"2024-04-29T14:38:41.906760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.906760+0000","objects_scrubbed":6597,"log_size":8476,"log_dups_size":0,"ondisk_log_size":8476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384303104,"num_objects":5864,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8476,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"771'8396","reported_seq":8857,"reported_epoch":771,"state":"active+clean","last_fresh":"2024-04-29T14:41:27.562592+0000","last_change":"2024-04-29T14:41:18.379928+0000","last_active":"2024-04-29T14:41:27.562592+0000","last_peered":"2024-04-29T14:41:27.562592+0000","last_clean":"2024-04-29T14:41:27.562592+0000","last_became_active":"2024-04-29T14:34:49.514880+0000","last_became_peered":"2024-04-29T14:34:49.514880+0000","last_unstale":"2024-04-29T14:41:27.562592+0000","last_undegraded":"2024-04-29T14:41:27.562592+0000","last_fullsized":"2024-04-29T14:41:27.562592+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8343","last_scrub_stamp":"2024-04-29T14:41:18.379788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:18.379788+0000","objects_scrubbed":5950,"log_size":8396,"log_dups_size":0,"ondisk_log_size":8396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:46.004560+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386269184,"num_objects":5894,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"770'8663","reported_seq":36813,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-29T14:41:25.979818+0000","last_change":"2024-04-29T14:41:13.470305+0000","last_active":"2024-04-29T14:41:25.979818+0000","last_peered":"2024-04-29T14:41:25.979818+0000","last_clean":"2024-04-29T14:41:25.979818+0000","last_became_active":"2024-04-29T14:37:07.575390+0000","last_became_peered":"2024-04-29T14:37:07.575390+0000","last_unstale":"2024-04-29T14:41:25.979818+0000","last_undegraded":"2024-04-29T14:41:25.979818+0000","last_fullsized":"2024-04-29T14:41:25.979818+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"765'8579","last_scrub_stamp":"2024-04-29T14:41:13.470148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:13.470148+0000","objects_scrubbed":6149,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:34.791755+0000","scrub_duration":585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397475840,"num_objects":6065,"num_object_clones":0,"num_object_copies":12130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8663,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13841,"num_bytes_recovered":898957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"769'8434","reported_seq":8958,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-29T14:41:24.761984+0000","last_change":"2024-04-29T14:41:17.390879+0000","last_active":"2024-04-29T14:41:24.761984+0000","last_peered":"2024-04-29T14:41:24.761984+0000","last_clean":"2024-04-29T14:41:24.761984+0000","last_became_active":"2024-04-29T14:34:49.388394+0000","last_became_peered":"2024-04-29T14:34:49.388394+0000","last_unstale":"2024-04-29T14:41:24.761984+0000","last_undegraded":"2024-04-29T14:41:24.761984+0000","last_fullsized":"2024-04-29T14:41:24.761984+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'8380","last_scrub_stamp":"2024-04-29T14:41:17.390714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:17.390714+0000","objects_scrubbed":5895,"log_size":8434,"log_dups_size":0,"ondisk_log_size":8434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:21.374882+0000","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382730240,"num_objects":5840,"num_object_clones":0,"num_object_copies":11680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8434,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"770'8532","reported_seq":21559,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-29T14:41:26.696625+0000","last_change":"2024-04-29T14:37:11.232594+0000","last_active":"2024-04-29T14:41:26.696625+0000","last_peered":"2024-04-29T14:41:26.696625+0000","last_clean":"2024-04-29T14:41:26.696625+0000","last_became_active":"2024-04-29T14:37:11.225189+0000","last_became_peered":"2024-04-29T14:37:11.225189+0000","last_unstale":"2024-04-29T14:41:26.696625+0000","last_undegraded":"2024-04-29T14:41:26.696625+0000","last_fullsized":"2024-04-29T14:41:26.696625+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8532,"log_dups_size":0,"ondisk_log_size":8532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391380992,"num_objects":5972,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8532,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6389,"num_bytes_recovered":418709504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"771'10618","reported_seq":18232,"reported_epoch":771,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T14:41:28.268219+0000","last_change":"2024-04-29T14:41:19.822651+0000","last_active":"2024-04-29T14:41:28.268219+0000","last_peered":"2024-04-29T14:41:28.268219+0000","last_clean":"2024-04-29T14:41:28.268219+0000","last_became_active":"2024-04-29T14:41:12.845251+0000","last_became_peered":"2024-04-29T14:41:12.845251+0000","last_unstale":"2024-04-29T14:41:28.268219+0000","last_undegraded":"2024-04-29T14:41:28.268219+0000","last_fullsized":"2024-04-29T14:41:28.268219+0000","mapping_epoch":764,"log_start":"660'600","ondisk_log_start":"660'600","created":655,"last_epoch_clean":766,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7748","last_scrub_stamp":"2024-04-29T14:38:41.886707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.886707+0000","objects_scrubbed":2149,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389218304,"num_objects":5939,"num_object_clones":0,"num_object_copies":11878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3572,"num_bytes_recovered":220332032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"772'8368","reported_seq":16299,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-04-29T14:41:29.006451+0000","last_change":"2024-04-29T14:41:13.968136+0000","last_active":"2024-04-29T14:41:29.006451+0000","last_peered":"2024-04-29T14:41:29.006451+0000","last_clean":"2024-04-29T14:41:29.006451+0000","last_became_active":"2024-04-29T14:41:12.277628+0000","last_became_peered":"2024-04-29T14:41:12.277628+0000","last_unstale":"2024-04-29T14:41:29.006451+0000","last_undegraded":"2024-04-29T14:41:29.006451+0000","last_fullsized":"2024-04-29T14:41:29.006451+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'8272","last_scrub_stamp":"2024-04-29T14:41:13.968013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:13.968013+0000","objects_scrubbed":5918,"log_size":8368,"log_dups_size":0,"ondisk_log_size":8368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:19.226763+0000","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381550592,"num_objects":5822,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8368,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3733,"num_bytes_recovered":244645888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"772'8342","reported_seq":33837,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-04-29T14:41:29.023492+0000","last_change":"2024-04-29T14:41:14.897161+0000","last_active":"2024-04-29T14:41:29.023492+0000","last_peered":"2024-04-29T14:41:29.023492+0000","last_clean":"2024-04-29T14:41:29.023492+0000","last_became_active":"2024-04-29T14:41:11.310817+0000","last_became_peered":"2024-04-29T14:41:11.310817+0000","last_unstale":"2024-04-29T14:41:29.023492+0000","last_undegraded":"2024-04-29T14:41:29.023492+0000","last_fullsized":"2024-04-29T14:41:29.023492+0000","mapping_epoch":763,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'8279","last_scrub_stamp":"2024-04-29T14:41:14.897016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:14.897016+0000","objects_scrubbed":5899,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:43.578624+0000","scrub_duration":484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382468096,"num_objects":5836,"num_object_clones":0,"num_object_copies":11672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12508,"num_bytes_recovered":807534592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"772'8331","reported_seq":22798,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-04-29T14:41:28.462472+0000","last_change":"2024-04-29T14:37:23.391838+0000","last_active":"2024-04-29T14:41:28.462472+0000","last_peered":"2024-04-29T14:41:28.462472+0000","last_clean":"2024-04-29T14:41:28.462472+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:41:28.462472+0000","last_undegraded":"2024-04-29T14:41:28.462472+0000","last_fullsized":"2024-04-29T14:41:28.462472+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387514368,"num_objects":5913,"num_object_clones":0,"num_object_copies":11826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"772'8258","reported_seq":14967,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-04-29T14:41:29.019808+0000","last_change":"2024-04-29T14:41:21.882125+0000","last_active":"2024-04-29T14:41:29.019808+0000","last_peered":"2024-04-29T14:41:29.019808+0000","last_clean":"2024-04-29T14:41:29.019808+0000","last_became_active":"2024-04-29T14:41:14.292270+0000","last_became_peered":"2024-04-29T14:41:14.292270+0000","last_unstale":"2024-04-29T14:41:29.019808+0000","last_undegraded":"2024-04-29T14:41:29.019808+0000","last_fullsized":"2024-04-29T14:41:29.019808+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8222","last_scrub_stamp":"2024-04-29T14:41:21.882016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:21.882016+0000","objects_scrubbed":5821,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:41.833704+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378929152,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3086,"num_bytes_recovered":198836224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1865,"reported_epoch":770,"state":"active+clean+scrubbing","last_fresh":"2024-04-29T14:41:25.804673+0000","last_change":"2024-04-29T14:41:25.735974+0000","last_active":"2024-04-29T14:41:25.804673+0000","last_peered":"2024-04-29T14:41:25.804673+0000","last_clean":"2024-04-29T14:41:25.804673+0000","last_became_active":"2024-04-29T14:35:31.373888+0000","last_became_peered":"2024-04-29T14:35:31.373888+0000","last_unstale":"2024-04-29T14:41:25.804673+0000","last_undegraded":"2024-04-29T14:41:25.804673+0000","last_fullsized":"2024-04-29T14:41:25.804673+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:40:00.132523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:40:00.132523+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6198263832,"num_objects":94579,"num_object_clones":0,"num_object_copies":189158,"num_objects_missing_on_primary":5083,"num_objects_missing":5083,"num_objects_degraded":0,"num_objects_misplaced":10979,"num_objects_unfound":0,"num_objects_dirty":94579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135241,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84864,"num_bytes_recovered":5463080960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14790434816,"data_stored":14790426672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139720,"ondisk_log_size":139720,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122462,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1566420,"kb_used_data":831368,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92178732,"statfs":{"total":95995035648,"available":94391021568,"internally_reserved":0,"allocated":851320832,"data_stored":850544239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":752679364},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384102,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3085728,"kb_used_data":2278484,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":90659424,"statfs":{"total":95995035648,"available":92835250176,"internally_reserved":0,"allocated":2333167616,"data_stored":2332384767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":826603968},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645736,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2026052,"kb_used_data":1349308,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":91719100,"statfs":{"total":95995035648,"available":93920358400,"internally_reserved":0,"allocated":1381691392,"data_stored":1380921075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":692976064},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423041,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1249988,"kb_used_data":1145276,"kb_used_omap":4,"kb_used_meta":104699,"kb_avail":92495164,"statfs":{"total":95995035648,"available":94715047936,"internally_reserved":0,"allocated":1172762624,"data_stored":1171992307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":107212480},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234417,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1223108,"kb_used_data":379772,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92522044,"statfs":{"total":95995035648,"available":94742573056,"internally_reserved":0,"allocated":388886528,"data_stored":388116211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":863566282},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528759,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2015312,"kb_used_data":1145608,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":91729840,"statfs":{"total":95995035648,"available":93931356160,"internally_reserved":0,"allocated":1173102592,"data_stored":1172324651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":890567110},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575305,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1703828,"kb_used_data":1303372,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92041324,"statfs":{"total":95995035648,"available":94250315776,"internally_reserved":0,"allocated":1334652928,"data_stored":1333872283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738837,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2290388,"kb_used_data":1519240,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":91454764,"statfs":{"total":95995035648,"available":93649678336,"internally_reserved":0,"allocated":1555701760,"data_stored":1554925187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497270,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2518888,"kb_used_data":2279708,"kb_used_omap":4,"kb_used_meta":239163,"kb_avail":91226264,"statfs":{"total":95995035648,"available":93415694336,"internally_reserved":0,"allocated":2334420992,"data_stored":2333634611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":244903620},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986626,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1247700,"kb_used_data":382152,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92497452,"statfs":{"total":95995035648,"available":94717390848,"internally_reserved":0,"allocated":391323648,"data_stored":390547075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653183,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876956,"kb_used_data":755348,"kb_used_omap":651,"kb_used_meta":120948,"kb_avail":92868196,"statfs":{"total":95995035648,"available":95097032704,"internally_reserved":0,"allocated":773476352,"data_stored":772693523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":123851528},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1553858560,"data_stored":1553858560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1332350976,"data_stored":1332346904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2330656768,"data_stored":2330656768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1125711872,"data_stored":1125711872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":389808128,"data_stored":389808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":771489792,"data_stored":771489792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170083840,"data_stored":1170079768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":387055616,"data_stored":387055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1171062784,"data_stored":1171062784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1380122624,"data_stored":1380122624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2328559616,"data_stored":2328559616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":849674240,"data_stored":849674240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:41:31.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-04-29T14:41:31.798 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:31.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:31 smithi194 ceph-mon[44089]: pgmap v1994: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 10979/189162 objects misplaced (5.804%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:31.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:31 smithi194 ceph-mon[44089]: from='client.17568 v1:172.21.15.40:0/2919674693' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:41:31.826 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:31.847 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:31.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:31.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:31 smithi121 ceph-mon[44533]: pgmap v1994: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 10979/189162 objects misplaced (5.804%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:31.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:31 smithi121 ceph-mon[44533]: from='client.17568 v1:172.21.15.40:0/2919674693' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:41:31.904 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:41:31.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:31.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:41:31.979 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:32.000 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:32.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:32.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:32.134 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:32.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:32.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:32.177 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:32.258 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:32.285 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:32.484 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:32.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:32.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:32.619 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:32.715 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:32.721 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:32.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:32.828 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:32.857 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:33 smithi194 ceph-mon[44089]: pgmap v1995: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 72 op/s; 10979/188842 objects misplaced (5.814%); 286 KiB/s, 4 objects/s recovering 2024-04-29T14:41:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:33 smithi194 ceph-mon[44089]: osdmap e773: 12 total, 12 up, 11 in 2024-04-29T14:41:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:33 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:33 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:33.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:33 smithi040 ceph-mon[39581]: pgmap v1995: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 72 op/s; 10979/188842 objects misplaced (5.814%); 286 KiB/s, 4 objects/s recovering 2024-04-29T14:41:33.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:33 smithi040 ceph-mon[39581]: osdmap e773: 12 total, 12 up, 11 in 2024-04-29T14:41:33.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:33 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:33.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:33 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:33.707 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:33.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:33.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:33.840 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:33.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:33 smithi121 ceph-mon[44533]: pgmap v1995: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 72 op/s; 10979/188842 objects misplaced (5.814%); 286 KiB/s, 4 objects/s recovering 2024-04-29T14:41:33.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:33 smithi121 ceph-mon[44533]: osdmap e773: 12 total, 12 up, 11 in 2024-04-29T14:41:33.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:33 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:33.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:33 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:34.039 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:34.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:34.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:34.173 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:34.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:34.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:34.305 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:34.334 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:34.355 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:34.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:34.427 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:34.428 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:34.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:34.474 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:41:34.488 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:34.737 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:34.739 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:34 smithi194 ceph-mon[44089]: osdmap e774: 12 total, 12 up, 11 in 2024-04-29T14:41:34.739 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/241426933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:41:34.739 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:41:34.739 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:41:34.739 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:41:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:34.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:34.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:34.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:34 smithi040 ceph-mon[39581]: osdmap e774: 12 total, 12 up, 11 in 2024-04-29T14:41:34.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/241426933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:41:34.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:41:34.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:41:34.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:41:34.876 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:34 smithi121 ceph-mon[44533]: osdmap e774: 12 total, 12 up, 11 in 2024-04-29T14:41:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/241426933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:41:34.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:41:34.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:41:34.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:41:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:35.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:35.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:35.140 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:35.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:35.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:41:35.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:35.293 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:35.505 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:35.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:35 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:35.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:41:35.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/241426933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:41:35.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:35 smithi040 ceph-mon[39581]: osdmap e775: 12 total, 12 up, 11 in 2024-04-29T14:41:35.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:41:35.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:41:35.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:35 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:41:35.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:35 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:35.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:41:35.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/241426933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:41:35.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:35 smithi194 ceph-mon[44089]: osdmap e775: 12 total, 12 up, 11 in 2024-04-29T14:41:35.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:41:35.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:41:35.759 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:35 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:41:35.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:35 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:35.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:41:35.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/241426933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:41:35.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:35 smithi121 ceph-mon[44533]: osdmap e775: 12 total, 12 up, 11 in 2024-04-29T14:41:35.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:41:35.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:41:35.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:35 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:41:36.345 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:41:36.345 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:36.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:36.366 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:41:36.387 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:41:36.408 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:41:36.429 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:41:36.449 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:41:36.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:41:36.471 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:41:36.479 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:36.497 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:41:36.498 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:41:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:36.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:41:36.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:41:36.539 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:41:36.559 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:41:36.580 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:41:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:41:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:41:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:41:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:41:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:41:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:41:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:41:36.582 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:41:36.582 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:41:36.582 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:41:36.582 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:41:36.582 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:41:36.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:36.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:36 smithi040 ceph-mon[39581]: pgmap v2000: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 10922/187918 objects misplaced (5.812%); 299 KiB/s, 4 objects/s recovering 2024-04-29T14:41:36.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2656334194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:36.632 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:36.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:36.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:36 smithi194 ceph-mon[44089]: pgmap v2000: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 10922/187918 objects misplaced (5.812%); 299 KiB/s, 4 objects/s recovering 2024-04-29T14:41:36.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2656334194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:36.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:36 smithi121 ceph-mon[44533]: pgmap v2000: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 10922/187918 objects misplaced (5.812%); 299 KiB/s, 4 objects/s recovering 2024-04-29T14:41:36.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2656334194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:36.886 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:36.915 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:37.346 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:41:37.346 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:41:37.346 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:41:37.346 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:41:37.346 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:41:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:37.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:37.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:37.602 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:37.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.697 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.698 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.702 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:37.703 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:37.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:37.789 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:41:37.807 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:37.838 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:37.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:37.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:38.038 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:38.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:38.060 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:41:38.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:38.144 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:38.176 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:38.362 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:38.429 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:38.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:38.499 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 1 pgp_num to 1 2024-04-29T14:41:38.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:38.569 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:38.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:38 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:38.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:38 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:38.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:38 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:38.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:38 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:38.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:38 smithi194 ceph-mon[44089]: osdmap e776: 12 total, 12 up, 11 in 2024-04-29T14:41:38.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:38 smithi194 ceph-mon[44089]: pgmap v2002: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 10861/187750 objects misplaced (5.785%); 334 KiB/s, 5 objects/s recovering 2024-04-29T14:41:38.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3391225985' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:41:38.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:38 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:38.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:38 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:38.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:38 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:38.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:38 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:38.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:38 smithi040 ceph-mon[39581]: osdmap e776: 12 total, 12 up, 11 in 2024-04-29T14:41:38.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:38 smithi040 ceph-mon[39581]: pgmap v2002: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 10861/187750 objects misplaced (5.785%); 334 KiB/s, 5 objects/s recovering 2024-04-29T14:41:38.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3391225985' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:41:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:38 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:38 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:38.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:38 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:38.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:38 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:38.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:38 smithi121 ceph-mon[44533]: osdmap e776: 12 total, 12 up, 11 in 2024-04-29T14:41:38.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:38 smithi121 ceph-mon[44533]: pgmap v2002: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 10861/187750 objects misplaced (5.785%); 334 KiB/s, 5 objects/s recovering 2024-04-29T14:41:38.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3391225985' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:41:39.494 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:41:39.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3391225985' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-29T14:41:39.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:39 smithi040 ceph-mon[39581]: osdmap e777: 12 total, 12 up, 11 in 2024-04-29T14:41:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:39.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:39.643 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3391225985' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-29T14:41:39.643 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:39 smithi194 ceph-mon[44089]: osdmap e777: 12 total, 12 up, 11 in 2024-04-29T14:41:39.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:39.752 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3391225985' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-29T14:41:39.752 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:39 smithi121 ceph-mon[44533]: osdmap e777: 12 total, 12 up, 11 in 2024-04-29T14:41:39.754 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:39.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:39.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:39.879 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:39.908 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:39.982 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:39.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:40.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:40.113 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:40.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:40.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:40.236 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:40.266 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217008 6.4 6.189c5b14 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.155698+0000", 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 539.118126244, 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.95438274499999998, 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.277 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217008 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155698+0000", 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155698+0000", 2024-04-29T14:41:40.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155699+0000", 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0950000000000001e-06 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155703+0000", 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9029999999999997e-06 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155712+0000", 2024-04-29T14:41:40.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8030000000000003e-06 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155723+0000", 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0987e-05 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155783+0000", 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9419999999999997e-05 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.155795+0000", 2024-04-29T14:41:40.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2836e-05 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110048+0000", 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.95425232699999996 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110081+0000", 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3374000000000003e-05 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217019 6.4 6.5a8858b4 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.169404+0000", 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 539.10442066799999, 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.94071997500000004, 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217019 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169404+0000", 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169404+0000", 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169405+0000", 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3209999999999999e-06 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169409+0000", 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8890000000000004e-06 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169418+0000", 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3289999999999994e-06 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169422+0000", 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3930000000000001e-06 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169474+0000", 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1799999999999999e-05 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.169482+0000", 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3529999999999995e-06 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110115+0000", 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.94063343600000004 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110124+0000", 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4540000000000007e-06 2024-04-29T14:41:40.285 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217027 6.4 6.eb4e04f4 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.172245+0000", 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 539.10157932000004, 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.93791217500000001, 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217027 2024-04-29T14:41:40.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172245+0000", 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172245+0000", 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172247+0000", 2024-04-29T14:41:40.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.668e-06 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172252+0000", 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9110000000000001e-06 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172261+0000", 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7369999999999995e-06 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172269+0000", 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9000000000000006e-06 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172320+0000", 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0986999999999998e-05 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.172327+0000", 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6040000000000004e-06 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110151+0000", 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.93782421299999996 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110157+0000", 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217028 6.4 6.2544a704 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.174731+0000", 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 539.09909379700002, 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.935446954, 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.290 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217028 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174731+0000", 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174731+0000", 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174732+0000", 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5990000000000001e-06 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174738+0000", 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1970000000000004e-06 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174749+0000", 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1712999999999999e-05 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174757+0000", 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1040000000000007e-06 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174815+0000", 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7386000000000002e-05 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.174824+0000", 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5270000000000004e-06 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110174+0000", 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.93534993600000005 2024-04-29T14:41:40.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110178+0000", 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4920000000000002e-06 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217047 6.4 6.c0a94fb4 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.259271+0000", 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 539.01455330099998, 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.85093163900000002, 2024-04-29T14:41:40.294 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217047 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259271+0000", 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259271+0000", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259273+0000", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.595e-06 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259278+0000", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.118e-06 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259287+0000", 2024-04-29T14:41:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0680000000000003e-06 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259293+0000", 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.716e-06 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259349+0000", 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5931999999999999e-05 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.259356+0000", 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2400000000000001e-06 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110197+0000", 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.85084111500000004 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110203+0000", 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8549999999999998e-06 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217050 6.4 6.a7a30a54 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.264752+0000", 2024-04-29T14:41:40.298 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 539.00907266399997, 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84547245999999998, 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217050 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264752+0000", 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264752+0000", 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264753+0000", 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.288e-06 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264757+0000", 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1849999999999997e-06 2024-04-29T14:41:40.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264769+0000", 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2048000000000001e-05 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264777+0000", 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9200000000000004e-06 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264833+0000", 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5821e-05 2024-04-29T14:41:40.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.264843+0000", 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6679999999999996e-06 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110221+0000", 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84537816200000004 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110224+0000", 2024-04-29T14:41:40.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3680000000000002e-06 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217051 6.4 6.23a71304 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.265194+0000", 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 539.00863078700002, 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84505052999999997, 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217051 2024-04-29T14:41:40.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265194+0000", 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265194+0000", 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265195+0000", 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.347e-06 2024-04-29T14:41:40.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265200+0000", 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8960000000000002e-06 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265211+0000", 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1216000000000001e-05 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265219+0000", 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1270000000000003e-06 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265274+0000", 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4738999999999999e-05 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.265283+0000", 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5380000000000002e-06 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110239+0000", 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84495622999999997 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110244+0000", 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4369999999999998e-06 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:217076 6.4 6.3aebfd54 (undecoded) ondisk+write+known_if_redirected e596)", 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:32:41.362559+0000", 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 538.91126573099996, 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.74770942699999998, 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.307 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 217076 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362559+0000", 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362559+0000", 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362560+0000", 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6000000000000002e-07 2024-04-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362563+0000", 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6449999999999998e-06 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362573+0000", 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0086e-05 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362579+0000", 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5740000000000003e-06 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362625+0000", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6011999999999999e-05 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:41.362633+0000", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5410000000000003e-06 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110264+0000", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.74763127500000004 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:32:42.110268+0000", 2024-04-29T14:41:40.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4340000000000002e-06 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248607 6.4 6:24aa9aec:::benchmark_data_smithi194_134202_object95754:head [delete] snapc 0=[] ondisk+write+known_if_redirected e614)", 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:22.345858+0000", 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 497.92796653200003, 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27482016399999998, 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.311 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248607 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345858+0000", 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345858+0000", 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345860+0000", 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8610000000000001e-06 2024-04-29T14:41:40.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345866+0000", 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5050000000000004e-06 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345882+0000", 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5281000000000002e-05 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345890+0000", 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2130000000000008e-06 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.345945+0000", 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5426000000000001e-05 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.346075+0000", 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012936999999999999 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 6", 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.346208+0000", 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013304699999999999 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.357761+0000", 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011553686000000001 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.620668+0000", 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26290639100000002 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.620678+0000", 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0383999999999999e-05 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248624 6.19 6:9f476182:::benchmark_data_smithi194_134202_object95771:head [delete] snapc 0=[] ondisk+write+known_if_redirected e615)", 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:22.679411+0000", 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 497.59441327100001, 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.76381612300000001, 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248624 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679411+0000", 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679411+0000", 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679414+0000", 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3580000000000001e-06 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679420+0000", 2024-04-29T14:41:40.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.212e-06 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679432+0000", 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2427e-05 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679439+0000", 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1199999999999996e-06 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679484+0000", 2024-04-29T14:41:40.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4459000000000002e-05 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:22.679491+0000", 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4120000000000004e-06 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.438321+0000", 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.75882953600000003 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.438420+0000", 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9006999999999996e-05 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.438505+0000", 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5372999999999997e-05 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.440373+0000", 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018676960000000001 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.443131+0000", 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027580460000000001 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.443182+0000", 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0949000000000003e-05 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.443227+0000", 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5528000000000002e-05 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248694 6.12 6.ae9c9b12 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.497991+0000", 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.77583324, 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86975254599999996, 2024-04-29T14:41:40.321 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248694 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497991+0000", 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497991+0000", 2024-04-29T14:41:40.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497993+0000", 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.305e-06 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.497997+0000", 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0840000000000002e-06 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498005+0000", 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3550000000000005e-06 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498010+0000", 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0340000000000004e-06 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498052+0000", 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2240000000000002e-05 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498059+0000", 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5660000000000003e-06 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367735+0000", 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86967563699999995 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367744+0000", 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3249999999999997e-06 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248695 6.12 6.660180b2 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.498079+0000", 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.77574524599999, 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86969207100000001, 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.325 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248695 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498079+0000", 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498079+0000", 2024-04-29T14:41:40.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498080+0000", 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2500000000000005e-07 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498083+0000", 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.514e-06 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498089+0000", 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1500000000000004e-06 2024-04-29T14:41:40.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498093+0000", 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1799999999999998e-06 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498138+0000", 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4645999999999999e-05 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.498147+0000", 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5049999999999996e-06 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367764+0000", 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86961689200000003 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367771+0000", 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4590000000000001e-06 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248696 6.12 6.c9a57c72 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.773622133, 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86759332, 2024-04-29T14:41:40.329 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248696 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500203+0000", 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.68e-07 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500206+0000", 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4990000000000001e-06 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500212+0000", 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9619999999999998e-06 2024-04-29T14:41:40.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500216+0000", 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9829999999999998e-06 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500257+0000", 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1254999999999997e-05 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.500267+0000", 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7270000000000002e-06 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367788+0000", 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.86752159399999995 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367796+0000", 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6320000000000008e-06 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248716 6.12 6.fb26d272 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.525801+0000", 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.74802397100001, 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84202116199999999, 2024-04-29T14:41:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248716 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525801+0000", 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525801+0000", 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525802+0000", 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2699999999999999e-06 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525806+0000", 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.386e-06 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525815+0000", 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4530000000000002e-06 2024-04-29T14:41:40.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525820+0000", 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3460000000000001e-06 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525870+0000", 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0068000000000002e-05 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.525880+0000", 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0210999999999999e-05 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367813+0000", 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.84193279600000004 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367822+0000", 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6319999999999997e-06 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248694 6.12 6.ae9c9b12 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569493+0000", 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.704331422, 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79837771000000002, 2024-04-29T14:41:40.337 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248694 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569493+0000", 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569493+0000", 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569494+0000", 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1629999999999999e-06 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569498+0000", 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.269e-06 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569505+0000", 2024-04-29T14:41:40.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9770000000000003e-06 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569510+0000", 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4739999999999996e-06 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569559+0000", 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9249999999999998e-05 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569565+0000", 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0689999999999998e-06 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367862+0000", 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79829612599999999 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367871+0000", 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3819999999999993e-06 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248695 6.12 6.660180b2 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.70430482900002, 2024-04-29T14:41:40.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79838047300000003, 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248695 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569520+0000", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8899999999999999e-07 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569527+0000", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0380000000000002e-06 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569533+0000", 2024-04-29T14:41:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0610000000000001e-06 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569539+0000", 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1370000000000001e-06 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569600+0000", 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1320000000000002e-05 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569608+0000", 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9719999999999992e-06 2024-04-29T14:41:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367893+0000", 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79828478000000003 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367900+0000", 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.576e-06 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248696 6.12 6.c9a57c72 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:41:40.345 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.70427971700002, 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79838399400000004, 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248696 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569545+0000", 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0900000000000002e-07 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569548+0000", 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2390000000000001e-06 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569553+0000", 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0359999999999997e-06 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569558+0000", 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2410000000000003e-06 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569633+0000", 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5220000000000001e-05 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569637+0000", 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4209999999999999e-06 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367922+0000", 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79828494800000005 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367929+0000", 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3799999999999996e-06 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248716 6.12 6.fb26d272 (undecoded) ondisk+retry+write+known_if_redirected e616)", 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.70426173999999, 2024-04-29T14:41:40.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.798394677, 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248716 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569563+0000", 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.72e-07 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569565+0000", 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4279999999999999e-06 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569568+0000", 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569573+0000", 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1000000000000003e-06 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569651+0000", 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8524999999999998e-05 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.569655+0000", 2024-04-29T14:41:40.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7100000000000001e-06 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367949+0000", 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.79829424800000004 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367958+0000", 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.174e-06 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248749 6.12 6.6f65012 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:41:40.353 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.716256+0000", 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.55756903600002, 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65172211199999996, 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248749 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716256+0000", 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716256+0000", 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716257+0000", 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2950000000000001e-06 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716261+0000", 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1760000000000003e-06 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716270+0000", 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6640000000000004e-06 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716277+0000", 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0990000000000001e-06 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716315+0000", 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7965e-05 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.716321+0000", 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1600000000000003e-06 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367973+0000", 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65165252900000004 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367978+0000", 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2239999999999997e-06 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.357 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.16809.0:248767 6.12 6.14c2c972 (undecoded) ondisk+write+known_if_redirected e616)", 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:33:23.743688+0000", 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.53013647, 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62430483599999997, 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.16809", 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1337115678", 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 248767 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743688+0000", 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743688+0000", 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743689+0000", 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0100000000000001e-06 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743693+0000", 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9890000000000003e-06 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743701+0000", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1850000000000004e-06 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743707+0000", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3260000000000003e-06 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743759+0000", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2441999999999997e-05 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:23.743765+0000", 2024-04-29T14:41:40.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5999999999999997e-06 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367989+0000", 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62422426099999995 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:33:24.367993+0000", 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0230000000000003e-06 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:40.361 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:41:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:40.362 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:40.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:40.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:40.495 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:40.524 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:40.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:40.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:40 smithi194 ceph-mon[44089]: pgmap v2004: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 10861/187346 objects misplaced (5.797%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:40.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:40 smithi040 ceph-mon[39581]: pgmap v2004: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 10861/187346 objects misplaced (5.797%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:40.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:40.668 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:40.678 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:40.696 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:40.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:40.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:40.831 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:40 smithi121 ceph-mon[44533]: pgmap v2004: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 89 op/s; 10861/187346 objects misplaced (5.797%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:40.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:40.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:40.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:41:40.958 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:41.014 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:41.034 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:41.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:41.138 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:41.165 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:41.190 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:41.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:41.226 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:41.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:41.325 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:41.515 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:41:41.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2467997191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:41:41.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:41 smithi040 ceph-mon[39581]: from='client.27298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:41:41.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2467997191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:41:41.820 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:41 smithi194 ceph-mon[44089]: from='client.27298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:41:41.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2467997191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:41:41.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:41 smithi121 ceph-mon[44533]: from='client.27298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:41:42.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:42 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:42.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:42 smithi194 ceph-mon[44089]: from='client.27298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:41:42.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:42 smithi194 ceph-mon[44089]: osdmap e778: 12 total, 12 up, 11 in 2024-04-29T14:41:42.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:42 smithi194 ceph-mon[44089]: pgmap v2006: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 10861/186890 objects misplaced (5.811%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:42.675 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:42 smithi194 ceph-mon[44089]: osdmap e779: 12 total, 12 up, 11 in 2024-04-29T14:41:42.779 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:42.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:42 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:42.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:42 smithi040 ceph-mon[39581]: from='client.27298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:41:42.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:42 smithi040 ceph-mon[39581]: osdmap e778: 12 total, 12 up, 11 in 2024-04-29T14:41:42.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:42 smithi040 ceph-mon[39581]: pgmap v2006: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 10861/186890 objects misplaced (5.811%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:42.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:42 smithi040 ceph-mon[39581]: osdmap e779: 12 total, 12 up, 11 in 2024-04-29T14:41:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:42.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:42.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:42 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:42.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:42 smithi121 ceph-mon[44533]: from='client.27298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:41:42.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:42 smithi121 ceph-mon[44533]: osdmap e778: 12 total, 12 up, 11 in 2024-04-29T14:41:42.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:42 smithi121 ceph-mon[44533]: pgmap v2006: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 10861/186890 objects misplaced (5.811%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:41:42.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:42 smithi121 ceph-mon[44533]: osdmap e779: 12 total, 12 up, 11 in 2024-04-29T14:41:42.885 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:42.912 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:42.985 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:42.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:43.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:43.119 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:43.141 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:43.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:43.245 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:43.272 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:43.280 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:41:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:41:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:41:43.280 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:41:43.292 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:43.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:43.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:43.422 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:43.441 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:43.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:43.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:43.571 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:43.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:43.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:43.725 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:43.826 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:43.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:43.869 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:41:43.901 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:41:43.929 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:41:43.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:43.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:41:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:43.975 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:43.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:43.995 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:41:44.030 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:41:44.049 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2890366093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:44.050 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:41:44.070 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:41:44.075 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:41:44.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:44.093 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:41:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:44.117 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:41:44.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2890366093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:44.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:44.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:44.139 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:41:44.140 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:41:44.140 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:41:44.140 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:41:44.140 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:41:44.140 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:41:44.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:41:44.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:41:44.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:41:44.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:41:44.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:41:44.141 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:41:44.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:44.253 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2890366093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:44.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:44.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:44.405 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:44.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:44.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:44.435 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:44.446 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:41:44.529 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:44.560 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:44.715 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:44.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:44.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:44.846 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:44.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:44.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:44.955 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:44.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:45.002 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:41:45.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:44 smithi040 ceph-mon[39581]: pgmap v2008: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 116 op/s; 10797/186352 objects misplaced (5.794%); 331 KiB/s, 5 objects/s recovering 2024-04-29T14:41:45.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:41:45.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:44 smithi121 ceph-mon[44533]: pgmap v2008: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 116 op/s; 10797/186352 objects misplaced (5.794%); 331 KiB/s, 5 objects/s recovering 2024-04-29T14:41:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:45.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:41:45.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:44 smithi194 ceph-mon[44089]: pgmap v2008: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 116 op/s; 10797/186352 objects misplaced (5.794%); 331 KiB/s, 5 objects/s recovering 2024-04-29T14:41:45.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:45.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8] out_osds: [3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:41:45.368 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:41:45.368 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.4615037064750993 2024-04-29T14:41:45.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 7 0.4615037064750993 2024-04-29T14:41:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:45.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:45.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:45.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:45.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:45.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:45.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:45.678 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:45.697 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:45.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:45.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:45.831 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:45.859 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:45.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:45.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:45.988 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.988 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.988 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: 7.a scrub starts 2024-04-29T14:41:45.988 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.988 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.989 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:45.989 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:41:45.989 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:45 smithi121 ceph-mon[44533]: 7.a scrub ok 2024-04-29T14:41:45.990 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:46.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:46.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:46.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: 7.a scrub starts 2024-04-29T14:41:46.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:41:46.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:45 smithi194 ceph-mon[44089]: 7.a scrub ok 2024-04-29T14:41:46.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:46.140 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:46.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: 7.a scrub starts 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:41:46.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:45 smithi040 ceph-mon[39581]: 7.a scrub ok 2024-04-29T14:41:46.263 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:46.301 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:46.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:46.339 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:46.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:46.451 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:46.451 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:46.469 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:46.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:41:46.917 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:46 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:41:46.917 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:46 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:41:46.917 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:46 smithi040 ceph-mon[39581]: pgmap v2009: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 123 op/s; 10797/185706 objects misplaced (5.814%); 258 KiB/s, 4 objects/s recovering 2024-04-29T14:41:47.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:46 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:41:47.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:46 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:41:47.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:46 smithi194 ceph-mon[44089]: pgmap v2009: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 123 op/s; 10797/185706 objects misplaced (5.814%); 258 KiB/s, 4 objects/s recovering 2024-04-29T14:41:47.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:41:47.219 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:47.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:47.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:47.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:46 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:41:47.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:46 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:41:47.348 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:46 smithi121 ceph-mon[44533]: pgmap v2009: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 123 op/s; 10797/185706 objects misplaced (5.814%); 258 KiB/s, 4 objects/s recovering 2024-04-29T14:41:47.350 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:47.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:47.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:47.492 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:47.926 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:41:48.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:47 smithi194 ceph-mon[44089]: 7.3 scrub starts 2024-04-29T14:41:48.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1669217483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:41:48.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:47 smithi194 ceph-mon[44089]: 7.3 scrub ok 2024-04-29T14:41:48.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:47 smithi040 ceph-mon[39581]: 7.3 scrub starts 2024-04-29T14:41:48.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1669217483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:41:48.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:47 smithi040 ceph-mon[39581]: 7.3 scrub ok 2024-04-29T14:41:48.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:47 smithi121 ceph-mon[44533]: 7.3 scrub starts 2024-04-29T14:41:48.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1669217483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:41:48.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:47 smithi121 ceph-mon[44533]: 7.3 scrub ok 2024-04-29T14:41:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:48.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:48.559 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:48.587 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:48.814 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:48.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:48.919 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:48.947 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:49.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:48 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:41:49.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:48 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:41:49.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1669217483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:41:49.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:48 smithi194 ceph-mon[44089]: osdmap e780: 12 total, 12 up, 11 in 2024-04-29T14:41:49.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:48 smithi194 ceph-mon[44089]: pgmap v2011: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 106 op/s; 10731/185460 objects misplaced (5.786%); 622 KiB/s, 9 objects/s recovering 2024-04-29T14:41:49.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:48 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:41:49.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:48 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:41:49.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1669217483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:41:49.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:48 smithi040 ceph-mon[39581]: osdmap e780: 12 total, 12 up, 11 in 2024-04-29T14:41:49.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:48 smithi040 ceph-mon[39581]: pgmap v2011: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 106 op/s; 10731/185460 objects misplaced (5.786%); 622 KiB/s, 9 objects/s recovering 2024-04-29T14:41:49.145 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:49.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:49.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:49.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:48 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:41:49.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:48 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:41:49.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1669217483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:41:49.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:48 smithi121 ceph-mon[44533]: osdmap e780: 12 total, 12 up, 11 in 2024-04-29T14:41:49.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:48 smithi121 ceph-mon[44533]: pgmap v2011: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 106 op/s; 10731/185460 objects misplaced (5.786%); 622 KiB/s, 9 objects/s recovering 2024-04-29T14:41:49.454 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:49.454 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:50.193 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:41:50.462 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:50.525 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:50.539 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:50.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:50.643 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:50.672 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:50.695 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:50.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:50.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:50.828 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:50.846 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:50.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:50.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:51.082 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:51.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:51.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:51.216 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:51.289 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:51.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:51.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:51 smithi040 ceph-mon[39581]: pgmap v2012: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 120 op/s; 10731/184958 objects misplaced (5.802%); 520 KiB/s, 8 objects/s recovering 2024-04-29T14:41:51.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:51.421 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:51.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:51.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:51.544 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:51.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:51 smithi194 ceph-mon[44089]: pgmap v2012: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 120 op/s; 10731/184958 objects misplaced (5.802%); 520 KiB/s, 8 objects/s recovering 2024-04-29T14:41:51.572 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:51.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:51 smithi121 ceph-mon[44533]: pgmap v2012: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 120 op/s; 10731/184958 objects misplaced (5.802%); 520 KiB/s, 8 objects/s recovering 2024-04-29T14:41:51.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:41:51.821 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:51.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:51.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:51.954 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:51.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:51.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:52.076 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:52.103 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:52.124 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:52.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:52.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:52.256 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:52.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:52.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:52.370 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:52.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:52.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:52 smithi040 ceph-mon[39581]: osdmap e781: 12 total, 12 up, 11 in 2024-04-29T14:41:52.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:52.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:52.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:52.406 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:52 smithi194 ceph-mon[44089]: osdmap e781: 12 total, 12 up, 11 in 2024-04-29T14:41:52.409 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:52.429 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:52.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:52.435 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:41:52.436 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:41:52.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:52.558 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:52.558 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:52.558 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:52 smithi121 ceph-mon[44533]: osdmap e781: 12 total, 12 up, 11 in 2024-04-29T14:41:52.560 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:52.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:52.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:52.682 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:52.710 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:52.858 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:52.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:52.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:53.051 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:41:53.075 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:41:53.095 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:41:53.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:41:53.136 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:41:53.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:41:53.177 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:41:53.197 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:53.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:41:53.200 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:41:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:53.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:53.220 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:41:53.256 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:41:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.7 to 0.461504 (7625) 2024-04-29T14:41:53.280 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:41:53.302 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:41:53.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:41:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:41:53.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:41:53.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:41:53.305 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:41:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:53.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:41:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:53.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:41:53.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:53.485 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:53.505 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:53.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:53 smithi194 ceph-mon[44089]: pgmap v2014: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 10731/184516 objects misplaced (5.816%); 272 KiB/s, 4 objects/s recovering 2024-04-29T14:41:53.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2293412836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.4615037064750993}]: dispatch 2024-04-29T14:41:53.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:53.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:53.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/980981697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:53.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:53 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:53.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:53 smithi040 ceph-mon[39581]: pgmap v2014: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 10731/184516 objects misplaced (5.816%); 272 KiB/s, 4 objects/s recovering 2024-04-29T14:41:53.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2293412836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.4615037064750993}]: dispatch 2024-04-29T14:41:53.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:53.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:53.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/980981697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:53.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:53 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:53 smithi121 ceph-mon[44533]: pgmap v2014: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 10731/184516 objects misplaced (5.816%); 272 KiB/s, 4 objects/s recovering 2024-04-29T14:41:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2293412836' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.4615037064750993}]: dispatch 2024-04-29T14:41:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/980981697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:41:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:53 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:54.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:54.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:54.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:41:54.472 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:54.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:41:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:54.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2293412836' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.4615037064750993}]': finished 2024-04-29T14:41:54.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osdmap e782: 12 total, 12 up, 11 in 2024-04-29T14:41:54.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:54 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2293412836' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.4615037064750993}]': finished 2024-04-29T14:41:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osdmap e782: 12 total, 12 up, 11 in 2024-04-29T14:41:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:54 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.625 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:54.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2293412836' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.4615037064750993}]': finished 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osdmap e782: 12 total, 12 up, 11 in 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:54 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:41:54.649 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:54.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:54.754 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:54.782 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:54.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:41:55.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:55.116 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:55.299 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:41:55.300 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:41:55.300 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:41:55.300 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:41:55.300 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:55.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:55.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:41:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:55.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:41:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:55.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:41:55.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:55 smithi194 ceph-mon[44089]: pgmap v2016: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10661/184084 objects misplaced (5.791%); 367 KiB/s, 5 objects/s recovering 2024-04-29T14:41:55.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:55 smithi040 ceph-mon[39581]: pgmap v2016: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10661/184084 objects misplaced (5.791%); 367 KiB/s, 5 objects/s recovering 2024-04-29T14:41:55.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:41:55.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:55 smithi121 ceph-mon[44533]: pgmap v2016: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10661/184084 objects misplaced (5.791%); 367 KiB/s, 5 objects/s recovering 2024-04-29T14:41:55.648 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:55.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:41:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:55.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:41:55.771 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:41:55.801 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:56.000 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:41:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:56.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:41:56.007 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:41:56.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:56.136 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:56.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:56.312 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:41:56.318 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:56.350 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:56.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:56.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:56.531 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:56.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:56 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:41:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:56 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:41:56.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:56 smithi194 ceph-mon[44089]: osdmap e783: 12 total, 12 up, 11 in 2024-04-29T14:41:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:56 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:41:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:56 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:41:56.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:56 smithi040 ceph-mon[39581]: osdmap e783: 12 total, 12 up, 11 in 2024-04-29T14:41:56.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:56 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:41:56.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:56 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:41:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:56 smithi121 ceph-mon[44533]: osdmap e783: 12 total, 12 up, 11 in 2024-04-29T14:41:56.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:56.666 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:56.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:56.791 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:56.818 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:41:57.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:57 smithi194 ceph-mon[44089]: pgmap v2018: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10661/183648 objects misplaced (5.805%); 384 KiB/s, 5 objects/s recovering 2024-04-29T14:41:57.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3532707232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:57.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:57 smithi194 ceph-mon[44089]: from='client.27325 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:57.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:57 smithi040 ceph-mon[39581]: pgmap v2018: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10661/183648 objects misplaced (5.805%); 384 KiB/s, 5 objects/s recovering 2024-04-29T14:41:57.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3532707232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:57.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:57 smithi040 ceph-mon[39581]: from='client.27325 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:57 smithi121 ceph-mon[44533]: pgmap v2018: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10661/183648 objects misplaced (5.805%); 384 KiB/s, 5 objects/s recovering 2024-04-29T14:41:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3532707232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:57.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:57 smithi121 ceph-mon[44533]: from='client.27325 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:41:57.768 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:57.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:57.872 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:57.900 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:57.973 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:57.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:58.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:41:58.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:58.109 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:58.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:58.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:58.265 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:58.291 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:58.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:58.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:58 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:58.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:58 smithi040 ceph-mon[39581]: from='client.27325 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:41:58.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:58 smithi040 ceph-mon[39581]: osdmap e784: 12 total, 12 up, 11 in 2024-04-29T14:41:58.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:58 smithi040 ceph-mon[39581]: osdmap e785: 12 total, 12 up, 11 in 2024-04-29T14:41:58.375 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:41:58.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:58 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:58.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:58 smithi121 ceph-mon[44533]: from='client.27325 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:41:58.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:58 smithi121 ceph-mon[44533]: osdmap e784: 12 total, 12 up, 11 in 2024-04-29T14:41:58.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:58 smithi121 ceph-mon[44533]: osdmap e785: 12 total, 12 up, 11 in 2024-04-29T14:41:58.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:41:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:58.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:58 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:41:58.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:58 smithi194 ceph-mon[44089]: from='client.27325 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:41:58.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:58 smithi194 ceph-mon[44089]: osdmap e784: 12 total, 12 up, 11 in 2024-04-29T14:41:58.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:58 smithi194 ceph-mon[44089]: osdmap e785: 12 total, 12 up, 11 in 2024-04-29T14:41:58.624 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:41:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:58.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:41:58.728 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:41:58.755 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:58.854 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:41:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:58.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:41:58.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:41:58.990 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:41:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:59.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:41:59.113 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:41:59.141 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:59.160 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:41:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:59.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:41:59.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:41:59.294 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:59.368 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:41:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:59.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:41:59.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:41:59.499 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:41:59 smithi040 ceph-mon[39581]: pgmap v2021: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 10596/183508 objects misplaced (5.774%); 389 KiB/s, 6 objects/s recovering 2024-04-29T14:41:59.503 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:59.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:41:59 smithi194 ceph-mon[44089]: pgmap v2021: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 10596/183508 objects misplaced (5.774%); 389 KiB/s, 6 objects/s recovering 2024-04-29T14:41:59.610 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:41:59.610 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:41:59.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:41:59.610 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:71736 7.b e668/655 7:d3abf962:::benchmark_data_smithi194_147200_object71735:head v 668'4460, mlcod=668'4458)", 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:27.078874+0000", 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 392.527931185, 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015953355999999998, 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.078874+0000", 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.078874+0000", 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.078877+0000", 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.756e-06 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.078906+0000", 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9303e-05 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.078925+0000", 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8626000000000001e-05 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.078935+0000", 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8539999999999992e-06 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.079050+0000", 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115493 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.079072+0000", 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2062999999999999e-05 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.094786+0000", 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015714345000000001 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:27.094827+0000", 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0915999999999999e-05 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.616 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:78439 7.b e671/655 7:d40e9a50:::benchmark_data_smithi194_147200_object78438:head v 671'4881, mlcod=671'4879)", 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:31.207175+0000", 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 388.39963050300003, 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013795066999999999, 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207175+0000", 2024-04-29T14:41:59.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207175+0000", 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207177+0000", 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3269999999999999e-06 2024-04-29T14:41:59.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207202+0000", 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.457e-05 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207215+0000", 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.366e-05 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207223+0000", 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207281+0000", 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8499999999999999e-05 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207307+0000", 2024-04-29T14:41:59.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5633999999999999e-05 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.220926+0000", 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013618892 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.220970+0000", 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3962999999999999e-05 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.621 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:78446 7.b e671/655 7:d4c9f9f9:::benchmark_data_smithi194_147200_object78445:head v 671'4882, mlcod=671'4879)", 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:31.207239+0000", 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 388.39956578300001, 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013759165, 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.622 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207239+0000", 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207239+0000", 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207240+0000", 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2399999999999998e-07 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207262+0000", 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1786000000000001e-05 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207267+0000", 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7030000000000003e-06 2024-04-29T14:41:59.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.207272+0000", 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9150000000000004e-06 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.213848+0000", 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065753340000000004 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.213873+0000", 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.493e-05 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.220974+0000", 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0071016999999999999 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.220999+0000", 2024-04-29T14:41:59.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4273e-05 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:79056 7.b e672/655 7:d02d8ac5:::benchmark_data_smithi194_147200_object79055:head v 672'4925, mlcod=672'4923)", 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:31.559791+0000", 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 388.04701421099998, 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015843149000000001, 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.627 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559791+0000", 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559791+0000", 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559793+0000", 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0020000000000001e-06 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559820+0000", 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7154000000000001e-05 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559837+0000", 2024-04-29T14:41:59.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7071e-05 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559846+0000", 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7479999999999999e-06 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559906+0000", 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9698999999999999e-05 2024-04-29T14:41:59.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559932+0000", 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6322e-05 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.575582+0000", 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015650108999999999 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.575634+0000", 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2043999999999998e-05 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:79058 7.b e672/655 7:dac59064:::benchmark_data_smithi194_147200_object79057:head v 672'4926, mlcod=672'4923)", 2024-04-29T14:41:59.632 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:31.559869+0000", 2024-04-29T14:41:59.633 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 388.04693584300003, 2024-04-29T14:41:59.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015793709999999999, 2024-04-29T14:41:59.633 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.633 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.633 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.633 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559869+0000", 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559869+0000", 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559870+0000", 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0999999999999998e-07 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559892+0000", 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2495999999999999e-05 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559899+0000", 2024-04-29T14:41:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.703e-06 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.559904+0000", 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3539999999999999e-06 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.564730+0000", 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048250849999999998 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.564751+0000", 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1271e-05 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.575639+0000", 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010888258 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:31.575663+0000", 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3932999999999999e-05 2024-04-29T14:41:59.637 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:80545 7.b e673/655 7:d678a1d7:::benchmark_data_smithi194_147200_object80544:head v 673'5016, mlcod=673'5014)", 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:32.482281+0000", 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 387.12452374200001, 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013973685, 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.638 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482281+0000", 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482281+0000", 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482284+0000", 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4930000000000001e-06 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482310+0000", 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6279999999999999e-05 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482327+0000", 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6464999999999999e-05 2024-04-29T14:41:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482336+0000", 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1640000000000007e-06 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482395+0000", 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9137e-05 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.482421+0000", 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5749999999999999e-05 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.496208+0000", 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013787526 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:32.496255+0000", 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6869999999999997e-05 2024-04-29T14:41:59.642 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:98097 7.b e675/655 7:d1777840:::benchmark_data_smithi194_147200_object98096:head v 675'6085, mlcod=675'6081)", 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:43.802963+0000", 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 375.803842471, 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016524467000000001, 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.802963+0000", 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.802963+0000", 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.802964+0000", 2024-04-29T14:41:59.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.186e-06 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.802987+0000", 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3342000000000001e-05 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.802998+0000", 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1102e-05 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.803005+0000", 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8109999999999997e-06 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.806068+0000", 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030632490000000001 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.806092+0000", 2024-04-29T14:41:59.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4028000000000001e-05 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.819437+0000", 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013344643 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.819487+0000", 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0105999999999998e-05 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.647 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:98129 7.b e675/655 7:d0e78b8f:::benchmark_data_smithi194_147200_object98128:head v 675'6091, mlcod=675'6088)", 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:43.826359+0000", 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 375.78044617, 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016462158000000001, 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.826359+0000", 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.826359+0000", 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.826360+0000", 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8200000000000008e-07 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.826382+0000", 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1759999999999998e-05 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.826389+0000", 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2799999999999998e-06 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.826395+0000", 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3369999999999996e-06 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.829298+0000", 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029026759999999999 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.829320+0000", 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2399999999999999e-05 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.842770+0000", 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01344978 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.842821+0000", 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0943000000000003e-05 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.652 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:98132 7.b e675/655 7:d4a6cff0:::benchmark_data_smithi194_147200_object98131:head v 675'6092, mlcod=675'6088)", 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:43.834223+0000", 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 375.77258198499999, 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013752637, 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834223+0000", 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834223+0000", 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834225+0000", 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.823e-06 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834249+0000", 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4420999999999999e-05 2024-04-29T14:41:59.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834263+0000", 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3869000000000001e-05 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834271+0000", 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1499999999999999e-06 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834329+0000", 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7451999999999998e-05 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834350+0000", 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.139e-05 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.847947+0000", 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013596749999999999 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.847976+0000", 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8782e-05 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:98133 7.b e675/655 7:db9e7b9e:::benchmark_data_smithi194_147200_object98132:head v 675'6093, mlcod=675'6088)", 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:43.834292+0000", 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 375.77251319800001, 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013723997, 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.657 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834292+0000", 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834292+0000", 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834293+0000", 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4000000000000002e-07 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834313+0000", 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0324000000000001e-05 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834318+0000", 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4709999999999998e-06 2024-04-29T14:41:59.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834323+0000", 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.583e-06 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.837100+0000", 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027772700000000001 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.837122+0000", 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1838e-05 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.847979+0000", 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010856911 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.848016+0000", 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7060000000000001e-05 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.661 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:98139 7.b e675/655 7:d1007324:::benchmark_data_smithi194_147200_object98138:head v 675'6094, mlcod=675'6088)", 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:43.834512+0000", 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 375.77229271099998, 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013532788, 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834512+0000", 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834512+0000", 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834514+0000", 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.424e-06 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834539+0000", 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.497e-05 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834551+0000", 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2469999999999999e-05 2024-04-29T14:41:59.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.834558+0000", 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7649999999999996e-06 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.840000+0000", 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054414399999999996 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.840028+0000", 2024-04-29T14:41:59.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8085999999999998e-05 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.848020+0000", 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079928220000000001 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:43.848045+0000", 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4811e-05 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.666 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:98803 7.b e675/655 7:d1ae6164:::benchmark_data_smithi194_147200_object98802:head v 675'6135, mlcod=675'6131)", 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:44.263881+0000", 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 375.34292435399999, 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013869362999999999, 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.263881+0000", 2024-04-29T14:41:59.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.263881+0000", 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.263882+0000", 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0999999999999997e-07 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.263901+0000", 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9093000000000002e-05 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.263908+0000", 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6190000000000002e-06 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.263914+0000", 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3909999999999997e-06 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.266544+0000", 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002630425 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.266560+0000", 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5554e-05 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.277693+0000", 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011133692000000001 2024-04-29T14:41:59.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.277750+0000", 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6779e-05 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:99867 7.5 e675/655 7:a0789d94:::benchmark_data_smithi194_147200_object99866:head v 675'6229, mlcod=675'6223)", 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:44.931729+0000", 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 374.67507587, 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014070361, 2024-04-29T14:41:59.671 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931729+0000", 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931729+0000", 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931731+0000", 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6190000000000001e-06 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931757+0000", 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5982999999999998e-05 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931765+0000", 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6909999999999997e-06 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931770+0000", 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.096e-06 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931815+0000", 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5637999999999999e-05 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.931836+0000", 2024-04-29T14:41:59.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0492999999999998e-05 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.945750+0000", 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013914615999999999 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:44.945800+0000", 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9224999999999997e-05 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:100437 7.5 e675/655 7:aa39f6da:::benchmark_data_smithi194_147200_object100436:head v 675'6260, mlcod=675'6257)", 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:45.287196+0000", 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 374.31960909399999, 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014738502000000001, 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287196+0000", 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287196+0000", 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287198+0000", 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9250000000000002e-06 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287227+0000", 2024-04-29T14:41:59.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.87e-05 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287240+0000", 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3325e-05 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287248+0000", 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4489999999999999e-06 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287422+0000", 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000173418 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287441+0000", 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.863e-05 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.301879+0000", 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014438789 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.301935+0000", 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5266000000000002e-05 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:100435 7.b e675/655 7:d71bdaa5:::benchmark_data_smithi194_147200_object100434:head v 675'6249, mlcod=675'6247)", 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:45.287408+0000", 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 374.31939736200002, 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014534533000000001, 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287408+0000", 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287408+0000", 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287409+0000", 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4810000000000001e-06 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287432+0000", 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2748999999999999e-05 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287441+0000", 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.397e-06 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287448+0000", 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6069999999999996e-06 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287497+0000", 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8622999999999998e-05 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.287513+0000", 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6509999999999999e-05 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.301913+0000", 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014399650999999999 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.301942+0000", 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9515000000000001e-05 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:100660 7.b e675/655 7:d1bcdeb7:::benchmark_data_smithi194_147200_object100659:head v 675'6266, mlcod=675'6261)", 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:45.449091+0000", 2024-04-29T14:41:59.684 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 374.15771383700002, 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014935014999999999, 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.449091+0000", 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.449091+0000", 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.449092+0000", 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1200000000000001e-06 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.449114+0000", 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1370999999999999e-05 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.449125+0000", 2024-04-29T14:41:59.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0927000000000001e-05 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.449131+0000", 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2899999999999999e-06 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.451589+0000", 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024583980000000001 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.451610+0000", 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0256999999999999e-05 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.463967+0000", 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012357028000000001 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:45.464026+0000", 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9623999999999997e-05 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:103368 7.5 e679/655 7:a7661f47:::benchmark_data_smithi194_147200_object103367:head v 679'6423, mlcod=679'6421)", 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:35:50.449738+0000", 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 369.15706692700002, 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014908655, 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449738+0000", 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449738+0000", 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449741+0000", 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3309999999999998e-06 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449765+0000", 2024-04-29T14:41:59.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4278e-05 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449779+0000", 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3961999999999999e-05 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449786+0000", 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3409999999999997e-06 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449845+0000", 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8424000000000002e-05 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.449862+0000", 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7703999999999999e-05 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.464604+0000", 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014741249 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:35:50.464647+0000", 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3365999999999998e-05 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:130964 7.5 e750/655 7:a06d4686:::benchmark_data_smithi194_147200_object16052:head v 750'8178, mlcod=750'8175)", 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:40:04.067221+0000", 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 115.539584619, 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077772500000000003, 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.693 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067221+0000", 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067221+0000", 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067222+0000", 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.829e-06 2024-04-29T14:41:59.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067231+0000", 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7790000000000001e-06 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067243+0000", 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2197e-05 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067251+0000", 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6000000000000001e-06 2024-04-29T14:41:59.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067313+0000", 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2241000000000003e-05 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067329+0000", 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5772000000000001e-05 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.074942+0000", 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076125719999999997 2024-04-29T14:41:59.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.074998+0000", 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.626e-05 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:136420 7.5 e777/655 7:add952f2:::benchmark_data_smithi194_147200_object21508:head v 777'8520, mlcod=777'8518)", 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.815659142000001, 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074850630000000001, 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.697 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791149+0000", 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6699999999999998e-06 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791158+0000", 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4490000000000005e-06 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791172+0000", 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3655999999999999e-05 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791181+0000", 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7700000000000007e-06 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791243+0000", 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2904999999999995e-05 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791266+0000", 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2382999999999999e-05 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.798602+0000", 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073362590000000004 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.798631+0000", 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8971000000000001e-05 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:136948 7.5 e779/655 7:ae800e08:::benchmark_data_smithi194_147200_object22036:head v 779'8554, mlcod=779'8551)", 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:41:59.701 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.493685937, 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076202170000000003, 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:41:59.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113121+0000", 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8780000000000001e-06 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113130+0000", 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1679999999999993e-06 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113143+0000", 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2628e-05 2024-04-29T14:41:59.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113151+0000", 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5299999999999996e-06 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.116770+0000", 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036181640000000001 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.116798+0000", 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8748999999999999e-05 2024-04-29T14:41:59.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.120682+0000", 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038833090000000002 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.120739+0000", 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7791000000000003e-05 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:41:59.705 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:41:59.706 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:41:59 smithi121 ceph-mon[44533]: pgmap v2021: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 10596/183508 objects misplaced (5.774%); 389 KiB/s, 6 objects/s recovering 2024-04-29T14:41:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:41:59.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:41:59.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:41:59.839 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:41:59.964 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:41:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:41:59.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:00.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:00.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:42:00.100 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:00.121 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:00.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:00.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:00.271 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:42:00.278 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:00.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:00.292 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:42:00.332 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:42:00.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:42:00.364 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:42:00.381 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8] out_osds: [3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:42:00.382 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:42:00.382 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 4, 7, 9, 10, 11, 6, 2, 1, 5, 8] 2024-04-29T14:42:00.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 11 2024-04-29T14:42:00.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3418130289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:00.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3418130289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:00.394 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:42:00.422 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:42:00.429 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:00.457 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:42:00.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:00.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:00.497 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:42:00.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:42:00.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:00.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3418130289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:00.577 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:42:00.594 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:00.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:42:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:00.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:00.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:42:00.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:42:00.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:42:00.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:42:00.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:42:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:42:00.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:00.748 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:01.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: pgmap v2022: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 10596/183160 objects misplaced (5.785%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:42:01.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: pgmap v2022: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 10596/183160 objects misplaced (5.785%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: pgmap v2022: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 10596/183160 objects misplaced (5.785%); 320 KiB/s, 4 objects/s recovering 2024-04-29T14:42:01.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:02.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:02.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:02.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:03.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:03 smithi040 ceph-mon[39581]: pgmap v2023: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 10596/182872 objects misplaced (5.794%); 278 KiB/s, 4 objects/s recovering 2024-04-29T14:42:03.800 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:03.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:03.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:03 smithi194 ceph-mon[44089]: pgmap v2023: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 10596/182872 objects misplaced (5.794%); 278 KiB/s, 4 objects/s recovering 2024-04-29T14:42:03.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:03 smithi121 ceph-mon[44533]: pgmap v2023: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 10596/182872 objects misplaced (5.794%); 278 KiB/s, 4 objects/s recovering 2024-04-29T14:42:03.910 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:03.940 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:03.960 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:03.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:04.064 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:04.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:04 smithi194 ceph-mon[44089]: pgmap v2024: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 10531/182374 objects misplaced (5.774%); 512 KiB/s, 7 objects/s recovering 2024-04-29T14:42:04.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:04 smithi040 ceph-mon[39581]: pgmap v2024: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 10531/182374 objects misplaced (5.774%); 512 KiB/s, 7 objects/s recovering 2024-04-29T14:42:04.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:04 smithi121 ceph-mon[44533]: pgmap v2024: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 10531/182374 objects misplaced (5.774%); 512 KiB/s, 7 objects/s recovering 2024-04-29T14:42:05.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:42:05.479 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:42:05.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2473919418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:05.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2473919418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:05.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2473919418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:06.257 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:06.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:06.362 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:06.391 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:06.410 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:06.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:06.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:06.541 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:06.560 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:06.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:06.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:06 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:06.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2473919418' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:06.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:06 smithi040 ceph-mon[39581]: osdmap e786: 12 total, 12 up, 11 in 2024-04-29T14:42:06.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:06 smithi040 ceph-mon[39581]: pgmap v2026: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10531/181734 objects misplaced (5.795%); 468 KiB/s, 7 objects/s recovering 2024-04-29T14:42:06.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:06.691 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:06.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:06.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:06.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:06.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:06 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:06.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2473919418' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:06.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:06 smithi194 ceph-mon[44089]: osdmap e786: 12 total, 12 up, 11 in 2024-04-29T14:42:06.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:06 smithi194 ceph-mon[44089]: pgmap v2026: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10531/181734 objects misplaced (5.795%); 468 KiB/s, 7 objects/s recovering 2024-04-29T14:42:06.844 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:06.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:06 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:06.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2473919418' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:06.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:06 smithi121 ceph-mon[44533]: osdmap e786: 12 total, 12 up, 11 in 2024-04-29T14:42:06.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:06 smithi121 ceph-mon[44533]: pgmap v2026: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 109 op/s; 10531/181734 objects misplaced (5.795%); 468 KiB/s, 7 objects/s recovering 2024-04-29T14:42:07.498 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:07.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:07.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:07.630 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:07.648 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:07.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:07.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:07.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=55,logm=4} crc {auth=3759675549,config=1981513354,health=179246142,kv=667443298,logm=3114491029}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1779325575}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=172389122}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1199565817}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680173241}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900866017}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370884317}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=271048914}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3734858010}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1785039115}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474150647}) 2024-04-29T14:42:07.778 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258768418}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462666321}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791513578}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3184180196}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1741077759}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=126710050}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990906082}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1877643117}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3849412512}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1284811512}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3457761369}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746630131}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577965365}) 2024-04-29T14:42:07.779 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986256133}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219246025}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1986303783}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440004477}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1340319446}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=863990349}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2839615416}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114889448}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261566306}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1499366758}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069215578}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3286419193}) 2024-04-29T14:42:07.780 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2814679179}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321278054}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042986983}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1389651372}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103940638}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729108971}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1112914933}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630128154}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913188165}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916176995}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4275020819}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637114757}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3803691726}) 2024-04-29T14:42:07.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=301280499}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366678148}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592575924}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=171039527}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012984589}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999535079}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984977391}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138027375}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3799597832}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919068718}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2617233442}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3084282521}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1328524519}) 2024-04-29T14:42:07.782 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2392953579}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780480591}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503039276}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3196332902}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758780240}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=679357892}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4159387037}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263817140}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1752579563}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183692491}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858898549}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1343876253}) 2024-04-29T14:42:07.783 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516036764}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646140722}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3639339812}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132909606}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2272697054}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309901696}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230173757}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1658269823}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3011253276}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637493742}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1828633681}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3734032053}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3171828591}) 2024-04-29T14:42:07.784 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1759380410}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1018377703}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2937014224}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=25,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=38} crc {logm=1627395386,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=2771689837,mon_config_key=1075190414}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=832651731}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3745633335}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=40,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=42} crc {mon_config_key=1467650458,monmap=455069592,osd_metadata=809037704,osd_pg_creating=4000849499,osdmap=2201614836}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2751653003}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=94} crc {osdmap=2463231537}) 2024-04-29T14:42:07.785 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:07 smithi121 ceph-mon[44533]: osdmap e787: 12 total, 12 up, 11 in 2024-04-29T14:42:07.786 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:07.797 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:07.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:07.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=55,logm=4} crc {auth=3759675549,config=1981513354,health=179246142,kv=667443298,logm=3114491029}) 2024-04-29T14:42:07.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1779325575}) 2024-04-29T14:42:07.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=172389122}) 2024-04-29T14:42:07.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1199565817}) 2024-04-29T14:42:07.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680173241}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900866017}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370884317}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=271048914}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3734858010}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1785039115}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474150647}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258768418}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462666321}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791513578}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3184180196}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1741077759}) 2024-04-29T14:42:07.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=126710050}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990906082}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1877643117}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3849412512}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1284811512}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3457761369}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746630131}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577965365}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986256133}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219246025}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1986303783}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440004477}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1340319446}) 2024-04-29T14:42:07.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=863990349}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2839615416}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114889448}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261566306}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1499366758}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069215578}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3286419193}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2814679179}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321278054}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042986983}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1389651372}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103940638}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729108971}) 2024-04-29T14:42:07.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1112914933}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630128154}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913188165}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916176995}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4275020819}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637114757}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3803691726}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=301280499}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366678148}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592575924}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=171039527}) 2024-04-29T14:42:07.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012984589}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999535079}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984977391}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138027375}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3799597832}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919068718}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2617233442}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3084282521}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1328524519}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2392953579}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780480591}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503039276}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3196332902}) 2024-04-29T14:42:07.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758780240}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=679357892}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4159387037}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263817140}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1752579563}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183692491}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858898549}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1343876253}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516036764}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646140722}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3639339812}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132909606}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2272697054}) 2024-04-29T14:42:07.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309901696}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230173757}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1658269823}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3011253276}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637493742}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1828633681}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3734032053}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3171828591}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1759380410}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1018377703}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2937014224}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=25,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=38} crc {logm=1627395386,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=2771689837,mon_config_key=1075190414}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=832651731}) 2024-04-29T14:42:07.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3745633335}) 2024-04-29T14:42:07.818 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=40,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=42} crc {mon_config_key=1467650458,monmap=455069592,osd_metadata=809037704,osd_pg_creating=4000849499,osdmap=2201614836}) 2024-04-29T14:42:07.818 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2751653003}) 2024-04-29T14:42:07.818 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=94} crc {osdmap=2463231537}) 2024-04-29T14:42:07.818 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:07 smithi040 ceph-mon[39581]: osdmap e787: 12 total, 12 up, 11 in 2024-04-29T14:42:07.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=55,logm=4} crc {auth=3759675549,config=1981513354,health=179246142,kv=667443298,logm=3114491029}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1779325575}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=172389122}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1199565817}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2680173241}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900866017}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370884317}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=271048914}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3734858010}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1785039115}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474150647}) 2024-04-29T14:42:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258768418}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462666321}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791513578}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3184180196}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1741077759}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=126710050}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990906082}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1877643117}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3849412512}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1284811512}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3457761369}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746630131}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577965365}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986256133}) 2024-04-29T14:42:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219246025}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1986303783}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440004477}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1340319446}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=863990349}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2839615416}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114889448}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261566306}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1499366758}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069215578}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3286419193}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2814679179}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321278054}) 2024-04-29T14:42:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3042986983}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1389651372}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103940638}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729108971}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1112914933}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630128154}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913188165}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916176995}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4275020819}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637114757}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3803691726}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=301280499}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366678148}) 2024-04-29T14:42:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592575924}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=171039527}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012984589}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999535079}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984977391}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138027375}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3799597832}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919068718}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2617233442}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3084282521}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1328524519}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2392953579}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780480591}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503039276}) 2024-04-29T14:42:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3196332902}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758780240}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=679357892}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4159387037}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263817140}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1752579563}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183692491}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2858898549}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1343876253}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1516036764}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646140722}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3639339812}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132909606}) 2024-04-29T14:42:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2272697054}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309901696}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230173757}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1658269823}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3011253276}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637493742}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1828633681}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3734032053}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3171828591}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1759380410}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1018377703}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2937014224}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=25,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=38} crc {logm=1627395386,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=2771689837,mon_config_key=1075190414}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=832651731}) 2024-04-29T14:42:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3745633335}) 2024-04-29T14:42:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=40,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=42} crc {mon_config_key=1467650458,monmap=455069592,osd_metadata=809037704,osd_pg_creating=4000849499,osdmap=2201614836}) 2024-04-29T14:42:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2751653003}) 2024-04-29T14:42:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=94} crc {osdmap=2463231537}) 2024-04-29T14:42:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:07 smithi194 ceph-mon[44089]: osdmap e787: 12 total, 12 up, 11 in 2024-04-29T14:42:07.901 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:07.936 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:08.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:08.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:08.194 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:08.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:08.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:08.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:08.343 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:08.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:08.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:08.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:08.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:08.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:08.589 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:42:08.589 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:42:08.595 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:08.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:08.651 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:08.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:08 smithi194 ceph-mon[44089]: pgmap v2028: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 10461/181518 objects misplaced (5.763%); 544 KiB/s, 8 objects/s recovering 2024-04-29T14:42:08.852 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:08.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:08 smithi040 ceph-mon[39581]: pgmap v2028: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 10461/181518 objects misplaced (5.763%); 544 KiB/s, 8 objects/s recovering 2024-04-29T14:42:08.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:08 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:42:08.600+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e787 definitely_dead 0 2024-04-29T14:42:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:08.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:08.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:08 smithi121 ceph-mon[44533]: pgmap v2028: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 10461/181518 objects misplaced (5.763%); 544 KiB/s, 8 objects/s recovering 2024-04-29T14:42:08.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:08.983 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:09.003 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:09.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:09.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:09.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:42:09.493 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.11. 2024-04-29T14:42:09.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3675373243' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-29T14:42:09.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:09 smithi194 ceph-mon[44089]: from='client.37097 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-29T14:42:09.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:09 smithi194 ceph-mon[44089]: Client client.admin marked osd.11 out, while it was still marked up 2024-04-29T14:42:09.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3675373243' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-29T14:42:09.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:09 smithi040 ceph-mon[39581]: from='client.37097 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-29T14:42:09.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:09 smithi040 ceph-mon[39581]: Client client.admin marked osd.11 out, while it was still marked up 2024-04-29T14:42:09.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3675373243' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-29T14:42:09.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:09 smithi121 ceph-mon[44533]: from='client.37097 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-29T14:42:09.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:09 smithi121 ceph-mon[44533]: Client client.admin marked osd.11 out, while it was still marked up 2024-04-29T14:42:10.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:10 smithi194 ceph-mon[44089]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-04-29T14:42:10.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:10 smithi194 ceph-mon[44089]: osdmap e788: 12 total, 12 up, 10 in 2024-04-29T14:42:10.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:10 smithi194 ceph-mon[44089]: pgmap v2030: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 10461/181088 objects misplaced (5.777%); 363 KiB/s, 5 objects/s recovering 2024-04-29T14:42:10.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:10 smithi040 ceph-mon[39581]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-04-29T14:42:10.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:10 smithi040 ceph-mon[39581]: osdmap e788: 12 total, 12 up, 10 in 2024-04-29T14:42:10.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:10 smithi040 ceph-mon[39581]: pgmap v2030: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 10461/181088 objects misplaced (5.777%); 363 KiB/s, 5 objects/s recovering 2024-04-29T14:42:10.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:10 smithi121 ceph-mon[44533]: from='client.37097 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-04-29T14:42:10.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:10 smithi121 ceph-mon[44533]: osdmap e788: 12 total, 12 up, 10 in 2024-04-29T14:42:10.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:10 smithi121 ceph-mon[44533]: pgmap v2030: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 10461/181088 objects misplaced (5.777%); 363 KiB/s, 5 objects/s recovering 2024-04-29T14:42:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:11.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:11.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:11.260 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:11.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:42:11.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:11 smithi194 ceph-mon[44089]: osdmap e789: 12 total, 12 up, 10 in 2024-04-29T14:42:11.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:11 smithi040 ceph-mon[39581]: osdmap e789: 12 total, 12 up, 10 in 2024-04-29T14:42:11.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:11 smithi121 ceph-mon[44533]: osdmap e789: 12 total, 12 up, 10 in 2024-04-29T14:42:12.095 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:12.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:12.219 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:12.246 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:12.266 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:12.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:12.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:12.399 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:12.420 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:12.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:12.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:12.526 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:42:12.554 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:12.577 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:42:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:12.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:42:12.650 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:12 smithi040 ceph-mon[39581]: pgmap v2032: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 10461/180800 objects misplaced (5.786%); 363 KiB/s, 5 objects/s recovering 2024-04-29T14:42:12.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:12.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:12.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:12.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:12.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:12 smithi194 ceph-mon[44089]: pgmap v2032: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 10461/180800 objects misplaced (5.786%); 363 KiB/s, 5 objects/s recovering 2024-04-29T14:42:12.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:12.863 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:12 smithi121 ceph-mon[44533]: pgmap v2032: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 10461/180800 objects misplaced (5.786%); 363 KiB/s, 5 objects/s recovering 2024-04-29T14:42:12.865 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:12.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:12.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:12.989 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:13.016 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:13.036 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:13.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:13.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:13.168 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:13.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:13.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:13.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:42:13.324 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:13.604 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:42:13.626 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:42:13.649 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:42:13.677 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:42:13.706 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:42:13.728 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:42:13.749 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:42:13.775 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:42:13.806 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:42:13.826 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:42:13.847 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:42:13.867 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:42:13.868 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:42:13.869 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:42:13.892 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2432932028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:14.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2432932028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:14.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2432932028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:14.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:42:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:14.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:42:14.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:14.713 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:14.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:14.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:14.837 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:14.865 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:14 smithi040 ceph-mon[39581]: pgmap v2033: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 10393/180360 objects misplaced (5.762%); 645 KiB/s, 10 objects/s recovering 2024-04-29T14:42:14.888 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:14.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:14.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:14 smithi194 ceph-mon[44089]: pgmap v2033: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 10393/180360 objects misplaced (5.762%); 645 KiB/s, 10 objects/s recovering 2024-04-29T14:42:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:14.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:14.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:15.050 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:15.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:15.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:14 smithi121 ceph-mon[44533]: pgmap v2033: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 10393/180360 objects misplaced (5.762%); 645 KiB/s, 10 objects/s recovering 2024-04-29T14:42:15.155 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:15.183 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:15.204 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:15.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:15.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:15.379 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:15.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:15.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:42:15.516 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:15.546 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:42:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:15.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:42:15.654 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:15.680 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:15.704 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:15.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:15.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:15.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:15 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:15.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:15 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:15.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:15 smithi040 ceph-mon[39581]: osdmap e790: 12 total, 12 up, 10 in 2024-04-29T14:42:16.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:15 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:16.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:15 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:16.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:15 smithi194 ceph-mon[44089]: osdmap e790: 12 total, 12 up, 10 in 2024-04-29T14:42:16.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:15 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:16.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:15 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:16.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:15 smithi121 ceph-mon[44533]: osdmap e790: 12 total, 12 up, 10 in 2024-04-29T14:42:16.153 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] out_osds: [3, 11] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:42:16.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:42:16.154 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-29T14:42:16.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:42:16.294 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:16.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:16.398 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:42:16.426 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:16.446 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:42:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:16.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:42:16.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:42:16.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2106844544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:16.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:16 smithi194 ceph-mon[44089]: from='client.37109 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:16.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:16 smithi194 ceph-mon[44089]: pgmap v2035: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 15703/179812 objects misplaced (8.733%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:42:16.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2106844544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:16.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:16 smithi040 ceph-mon[39581]: from='client.37109 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:16.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:16 smithi040 ceph-mon[39581]: pgmap v2035: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 15703/179812 objects misplaced (8.733%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:42:17.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:42:17.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2106844544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:17.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:16 smithi121 ceph-mon[44533]: from='client.37109 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:17.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:16 smithi121 ceph-mon[44533]: pgmap v2035: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 15703/179812 objects misplaced (8.733%); 2.7 MiB/s, 43 objects/s recovering 2024-04-29T14:42:17.134 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:17.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:17.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:17.279 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:17.616 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:42:17.616 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:42:17.616 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:42:17.616 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:42:17.617 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:42:17.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:17 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:17.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:17 smithi040 ceph-mon[39581]: from='client.37109 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:42:17.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:17 smithi040 ceph-mon[39581]: osdmap e791: 12 total, 12 up, 10 in 2024-04-29T14:42:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:17.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:18.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:17 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:18.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:17 smithi194 ceph-mon[44089]: from='client.37109 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:42:18.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:17 smithi194 ceph-mon[44089]: osdmap e791: 12 total, 12 up, 10 in 2024-04-29T14:42:18.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:18.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:17 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:18.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:17 smithi121 ceph-mon[44533]: from='client.37109 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:42:18.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:17 smithi121 ceph-mon[44533]: osdmap e791: 12 total, 12 up, 10 in 2024-04-29T14:42:18.120 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:18.139 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:18.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:18.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:18.277 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:18.689 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:18.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:42:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:18.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:18.835 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:42:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:42:19.001 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:18 smithi040 ceph-mon[39581]: pgmap v2037: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 15703/179812 objects misplaced (8.733%); 2.4 MiB/s, 38 objects/s recovering 2024-04-29T14:42:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:19.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:42:19.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:18 smithi194 ceph-mon[44089]: pgmap v2037: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 15703/179812 objects misplaced (8.733%); 2.4 MiB/s, 38 objects/s recovering 2024-04-29T14:42:19.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:19.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:18 smithi121 ceph-mon[44533]: pgmap v2037: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 15703/179812 objects misplaced (8.733%); 2.4 MiB/s, 38 objects/s recovering 2024-04-29T14:42:19.134 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:19.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:42:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:20.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:20.797 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:21.410 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:21 smithi040 ceph-mon[39581]: pgmap v2038: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 15638/179084 objects misplaced (8.732%); 2.6 MiB/s, 40 objects/s recovering 2024-04-29T14:42:21.410 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:21 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:42:21.410 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:21.410 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:21 smithi194 ceph-mon[44089]: pgmap v2038: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 15638/179084 objects misplaced (8.732%); 2.6 MiB/s, 40 objects/s recovering 2024-04-29T14:42:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:21 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:42:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:21.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:21.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:21 smithi121 ceph-mon[44533]: pgmap v2038: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 15638/179084 objects misplaced (8.732%); 2.6 MiB/s, 40 objects/s recovering 2024-04-29T14:42:21.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:21 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:42:21.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:21.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:21.900 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:21.989 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:42:22.025 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:42:22.053 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:42:22.077 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:42:22.110 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:42:22.134 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:42:22.164 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:42:22.184 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:42:22.219 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:42:22.245 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:42:22.270 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:42:22.295 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:42:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:42:22.298 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:42:22.298 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:42:22.408 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:42:22.410 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:42:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: osdmap e792: 12 total, 12 up, 10 in 2024-04-29T14:42:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1415106203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:22 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:22.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:22.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:22.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: osdmap e792: 12 total, 12 up, 10 in 2024-04-29T14:42:22.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1415106203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:22.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:22.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:22.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:22.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:22.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:22 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:22.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: osdmap e792: 12 total, 12 up, 10 in 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1415106203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:22 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:23.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:23.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:23.144 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:23.165 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:23.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:23.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:23.296 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:23.315 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:23.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:23.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:42:23.455 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:23.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: pgmap v2040: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 94 op/s; 15347/178730 objects misplaced (8.587%); 4.8 MiB/s, 77 objects/s recovering 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: from='client.37121 v1:172.21.15.40:0/2531904829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:42:23.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: osdmap e793: 12 total, 12 up, 10 in 2024-04-29T14:42:23.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:23 smithi040 ceph-mon[39581]: 7.d scrub starts 2024-04-29T14:42:23.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: pgmap v2040: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 94 op/s; 15347/178730 objects misplaced (8.587%); 4.8 MiB/s, 77 objects/s recovering 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: from='client.37121 v1:172.21.15.40:0/2531904829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:42:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: osdmap e793: 12 total, 12 up, 10 in 2024-04-29T14:42:23.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:23 smithi194 ceph-mon[44089]: 7.d scrub starts 2024-04-29T14:42:23.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: pgmap v2040: 17 pgs: 2 active+recovering, 15 active+clean; 5.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 94 op/s; 15347/178730 objects misplaced (8.587%); 4.8 MiB/s, 77 objects/s recovering 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: from='client.37121 v1:172.21.15.40:0/2531904829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:42:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: osdmap e793: 12 total, 12 up, 10 in 2024-04-29T14:42:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:23 smithi121 ceph-mon[44533]: 7.d scrub starts 2024-04-29T14:42:24.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:24 smithi194 ceph-mon[44089]: 7.d scrub ok 2024-04-29T14:42:24.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:24 smithi040 ceph-mon[39581]: 7.d scrub ok 2024-04-29T14:42:24.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:24 smithi121 ceph-mon[44533]: 7.d scrub ok 2024-04-29T14:42:24.914 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2040,"stamp":"2024-04-29T14:42:22.201963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5857018424,"num_objects":89365,"num_object_clones":0,"num_object_copies":178730,"num_objects_missing_on_primary":4757,"num_objects_missing":4757,"num_objects_degraded":0,"num_objects_misplaced":15347,"num_objects_unfound":0,"num_objects_dirty":89365,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":140638,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85753,"num_bytes_recovered":5524127456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144023,"ondisk_log_size":144023,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":17103600,"kb_used_data":11390028,"kb_used_omap":670,"kb_used_meta":5712801,"kb_avail":920347920,"statfs":{"total":959950356480,"available":942436270080,"internally_reserved":0,"allocated":11663388672,"data_stored":11654296408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686869,"internal_metadata":5849908459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-46989312,"num_objects":-717,"num_object_clones":0,"num_object_copies":-1434,"num_objects_missing_on_primary":-31,"num_objects_missing":-31,"num_objects_degraded":0,"num_objects_misplaced":4954,"num_objects_unfound":0,"num_objects_dirty":-717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":717,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":38273024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.563394"},"pg_stats":[{"pgid":"7.f","version":"791'8854","reported_seq":23799,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:19.710801+0000","last_change":"2024-04-29T14:41:44.740584+0000","last_active":"2024-04-29T14:42:19.710801+0000","last_peered":"2024-04-29T14:42:19.710801+0000","last_clean":"2024-04-29T14:42:19.710801+0000","last_became_active":"2024-04-29T14:38:40.422563+0000","last_became_peered":"2024-04-29T14:38:40.422563+0000","last_unstale":"2024-04-29T14:42:19.710801+0000","last_undegraded":"2024-04-29T14:42:19.710801+0000","last_fullsized":"2024-04-29T14:42:19.710801+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'8640","last_scrub_stamp":"2024-04-29T14:41:44.740429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:44.740429+0000","objects_scrubbed":5807,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:51.553043+0000","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366346240,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7222,"num_bytes_recovered":464846848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"791'16011","reported_seq":16724,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:18.198517+0000","last_change":"2024-04-29T14:41:22.402971+0000","last_active":"2024-04-29T14:42:18.198517+0000","last_peered":"2024-04-29T14:42:18.198517+0000","last_clean":"2024-04-29T14:42:18.198517+0000","last_became_active":"2024-04-29T14:34:49.387078+0000","last_became_peered":"2024-04-29T14:34:49.387078+0000","last_unstale":"2024-04-29T14:42:18.198517+0000","last_undegraded":"2024-04-29T14:42:18.198517+0000","last_fullsized":"2024-04-29T14:42:18.198517+0000","mapping_epoch":655,"log_start":"675'6000","ondisk_log_start":"675'6000","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'15606","last_scrub_stamp":"2024-04-29T14:41:22.402886+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:41:22.402886+0000","objects_scrubbed":6003,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:32.816133+0000","scrub_duration":639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366673920,"num_objects":5595,"num_object_clones":0,"num_object_copies":11190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"791'8770","reported_seq":21780,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:20.138637+0000","last_change":"2024-04-29T14:41:11.594141+0000","last_active":"2024-04-29T14:42:20.138637+0000","last_peered":"2024-04-29T14:42:20.138637+0000","last_clean":"2024-04-29T14:42:20.138637+0000","last_became_active":"2024-04-29T14:35:47.383003+0000","last_became_peered":"2024-04-29T14:35:47.383003+0000","last_unstale":"2024-04-29T14:42:20.138637+0000","last_undegraded":"2024-04-29T14:42:20.138637+0000","last_fullsized":"2024-04-29T14:42:20.138637+0000","mapping_epoch":676,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"764'8355","last_scrub_stamp":"2024-04-29T14:41:11.593851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:11.593851+0000","objects_scrubbed":5974,"log_size":8770,"log_dups_size":0,"ondisk_log_size":8770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:20.950090+0000","scrub_duration":637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364052504,"num_objects":5556,"num_object_clones":0,"num_object_copies":11112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8770,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6289,"num_bytes_recovered":412155904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"791'8757","reported_seq":19793,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:20.164601+0000","last_change":"2024-04-29T14:41:16.362353+0000","last_active":"2024-04-29T14:42:20.164601+0000","last_peered":"2024-04-29T14:42:20.164601+0000","last_clean":"2024-04-29T14:42:20.164601+0000","last_became_active":"2024-04-29T14:41:11.301869+0000","last_became_peered":"2024-04-29T14:41:11.301869+0000","last_unstale":"2024-04-29T14:42:20.164601+0000","last_undegraded":"2024-04-29T14:42:20.164601+0000","last_fullsized":"2024-04-29T14:42:20.164601+0000","mapping_epoch":763,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'8385","last_scrub_stamp":"2024-04-29T14:41:16.362214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:16.362214+0000","objects_scrubbed":5963,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:43.231672+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366411776,"num_objects":5591,"num_object_clones":0,"num_object_copies":11182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8757,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5265,"num_bytes_recovered":332988416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"791'8693","reported_seq":14093,"reported_epoch":791,"state":"active+recovering","last_fresh":"2024-04-29T14:42:18.185457+0000","last_change":"2024-04-29T14:36:34.055204+0000","last_active":"2024-04-29T14:42:18.185457+0000","last_peered":"2024-04-29T14:42:18.185457+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:36:33.135845+0000","last_became_peered":"2024-04-29T14:36:33.135845+0000","last_unstale":"2024-04-29T14:42:18.185457+0000","last_undegraded":"2024-04-29T14:42:18.185457+0000","last_fullsized":"2024-04-29T14:42:18.185457+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8693,"log_dups_size":0,"ondisk_log_size":8693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366280704,"num_objects":5589,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":4757,"num_objects_missing":4757,"num_objects_degraded":0,"num_objects_misplaced":10328,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1995,"num_bytes_recovered":130744320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["10","11"],"object_location_counts":[{"shards":"1,5","objects":18},{"shards":"1,10,11","objects":814},{"shards":"10,11","objects":4757}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"791'16024","reported_seq":16739,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:20.153694+0000","last_change":"2024-04-29T14:41:44.773423+0000","last_active":"2024-04-29T14:42:20.153694+0000","last_peered":"2024-04-29T14:42:20.153694+0000","last_clean":"2024-04-29T14:42:20.153694+0000","last_became_active":"2024-04-29T14:34:49.387647+0000","last_became_peered":"2024-04-29T14:34:49.387647+0000","last_unstale":"2024-04-29T14:42:20.153694+0000","last_undegraded":"2024-04-29T14:42:20.153694+0000","last_fullsized":"2024-04-29T14:42:20.153694+0000","mapping_epoch":655,"log_start":"675'6000","ondisk_log_start":"675'6000","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'15825","last_scrub_stamp":"2024-04-29T14:41:44.773306+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:41:44.773306+0000","objects_scrubbed":5932,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:43:14.336019+0000","scrub_duration":700,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375128064,"num_objects":5724,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8950,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"791'8806","reported_seq":37541,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:18.114973+0000","last_change":"2024-04-29T14:40:30.568318+0000","last_active":"2024-04-29T14:42:18.114973+0000","last_peered":"2024-04-29T14:42:18.114973+0000","last_clean":"2024-04-29T14:42:18.114973+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:42:18.114973+0000","last_undegraded":"2024-04-29T14:42:18.114973+0000","last_fullsized":"2024-04-29T14:42:18.114973+0000","mapping_epoch":756,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7744","last_scrub_stamp":"2024-04-29T14:38:41.906760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.906760+0000","objects_scrubbed":6597,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362676224,"num_objects":5534,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8806,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"791'8692","reported_seq":9193,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:18.263511+0000","last_change":"2024-04-29T14:41:18.379928+0000","last_active":"2024-04-29T14:42:18.263511+0000","last_peered":"2024-04-29T14:42:18.263511+0000","last_clean":"2024-04-29T14:42:18.263511+0000","last_became_active":"2024-04-29T14:34:49.514880+0000","last_became_peered":"2024-04-29T14:34:49.514880+0000","last_unstale":"2024-04-29T14:42:18.263511+0000","last_undegraded":"2024-04-29T14:42:18.263511+0000","last_fullsized":"2024-04-29T14:42:18.263511+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8343","last_scrub_stamp":"2024-04-29T14:41:18.379788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:18.379788+0000","objects_scrubbed":5950,"log_size":8692,"log_dups_size":0,"ondisk_log_size":8692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:46.004560+0000","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366870528,"num_objects":5598,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"791'9013","reported_seq":37205,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:20.475663+0000","last_change":"2024-04-29T14:41:13.470305+0000","last_active":"2024-04-29T14:42:20.475663+0000","last_peered":"2024-04-29T14:42:20.475663+0000","last_clean":"2024-04-29T14:42:20.475663+0000","last_became_active":"2024-04-29T14:37:07.575390+0000","last_became_peered":"2024-04-29T14:37:07.575390+0000","last_unstale":"2024-04-29T14:42:20.475663+0000","last_undegraded":"2024-04-29T14:42:20.475663+0000","last_fullsized":"2024-04-29T14:42:20.475663+0000","mapping_epoch":696,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"765'8579","last_scrub_stamp":"2024-04-29T14:41:13.470148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:13.470148+0000","objects_scrubbed":6149,"log_size":9013,"log_dups_size":0,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:34.791755+0000","scrub_duration":585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374538240,"num_objects":5715,"num_object_clones":0,"num_object_copies":11430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13841,"num_bytes_recovered":898957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"791'8786","reported_seq":10459,"reported_epoch":791,"state":"active+recovering","last_fresh":"2024-04-29T14:42:20.250344+0000","last_change":"2024-04-29T14:42:10.763062+0000","last_active":"2024-04-29T14:42:20.250344+0000","last_peered":"2024-04-29T14:42:20.250344+0000","last_clean":"2024-04-29T14:42:09.244532+0000","last_became_active":"2024-04-29T14:42:10.727580+0000","last_became_peered":"2024-04-29T14:42:10.727580+0000","last_unstale":"2024-04-29T14:42:20.250344+0000","last_undegraded":"2024-04-29T14:42:20.250344+0000","last_fullsized":"2024-04-29T14:42:20.250344+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'8380","last_scrub_stamp":"2024-04-29T14:41:17.390714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:17.390714+0000","objects_scrubbed":5895,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359661568,"num_objects":5488,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5019,"num_objects_unfound":0,"num_objects_dirty":5488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8786,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":550,"num_bytes_recovered":36044800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":["9","11"],"object_location_counts":[{"shards":"0,9","objects":469},{"shards":"9,11","objects":5019}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"791'8874","reported_seq":21944,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:17.970385+0000","last_change":"2024-04-29T14:37:11.232594+0000","last_active":"2024-04-29T14:42:17.970385+0000","last_peered":"2024-04-29T14:42:17.970385+0000","last_clean":"2024-04-29T14:42:17.970385+0000","last_became_active":"2024-04-29T14:37:11.225189+0000","last_became_peered":"2024-04-29T14:37:11.225189+0000","last_unstale":"2024-04-29T14:42:17.970385+0000","last_undegraded":"2024-04-29T14:42:17.970385+0000","last_fullsized":"2024-04-29T14:42:17.970385+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368967680,"num_objects":5630,"num_object_clones":0,"num_object_copies":11260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8874,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6389,"num_bytes_recovered":418709504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"791'12370","reported_seq":20172,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:18.234142+0000","last_change":"2024-04-29T14:41:47.347334+0000","last_active":"2024-04-29T14:42:18.234142+0000","last_peered":"2024-04-29T14:42:18.234142+0000","last_clean":"2024-04-29T14:42:18.234142+0000","last_became_active":"2024-04-29T14:41:12.845251+0000","last_became_peered":"2024-04-29T14:41:12.845251+0000","last_unstale":"2024-04-29T14:42:18.234142+0000","last_undegraded":"2024-04-29T14:42:18.234142+0000","last_fullsized":"2024-04-29T14:42:18.234142+0000","mapping_epoch":764,"log_start":"662'2300","ondisk_log_start":"662'2300","created":655,"last_epoch_clean":766,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'12181","last_scrub_stamp":"2024-04-29T14:41:47.347210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:47.347210+0000","objects_scrubbed":5826,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:52.407792+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369426432,"num_objects":5637,"num_object_clones":0,"num_object_copies":11274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8771,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3572,"num_bytes_recovered":220332032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"791'8686","reported_seq":16655,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:19.162940+0000","last_change":"2024-04-29T14:41:13.968136+0000","last_active":"2024-04-29T14:42:19.162940+0000","last_peered":"2024-04-29T14:42:19.162940+0000","last_clean":"2024-04-29T14:42:19.162940+0000","last_became_active":"2024-04-29T14:41:12.277628+0000","last_became_peered":"2024-04-29T14:41:12.277628+0000","last_unstale":"2024-04-29T14:42:19.162940+0000","last_undegraded":"2024-04-29T14:42:19.162940+0000","last_fullsized":"2024-04-29T14:42:19.162940+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'8272","last_scrub_stamp":"2024-04-29T14:41:13.968013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:13.968013+0000","objects_scrubbed":5918,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710144,"num_objects":5504,"num_object_clones":0,"num_object_copies":11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8686,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3733,"num_bytes_recovered":244645888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"791'8652","reported_seq":34185,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:19.175127+0000","last_change":"2024-04-29T14:41:14.897161+0000","last_active":"2024-04-29T14:42:19.175127+0000","last_peered":"2024-04-29T14:42:19.175127+0000","last_clean":"2024-04-29T14:42:19.175127+0000","last_became_active":"2024-04-29T14:41:11.310817+0000","last_became_peered":"2024-04-29T14:41:11.310817+0000","last_unstale":"2024-04-29T14:42:19.175127+0000","last_undegraded":"2024-04-29T14:42:19.175127+0000","last_fullsized":"2024-04-29T14:42:19.175127+0000","mapping_epoch":763,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'8279","last_scrub_stamp":"2024-04-29T14:41:14.897016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:14.897016+0000","objects_scrubbed":5899,"log_size":8652,"log_dups_size":0,"ondisk_log_size":8652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:43.578624+0000","scrub_duration":484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362151936,"num_objects":5526,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8652,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12508,"num_bytes_recovered":807534592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"791'8625","reported_seq":23131,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:18.276951+0000","last_change":"2024-04-29T14:37:23.391838+0000","last_active":"2024-04-29T14:42:18.276951+0000","last_peered":"2024-04-29T14:42:18.276951+0000","last_clean":"2024-04-29T14:42:18.276951+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:42:18.276951+0000","last_undegraded":"2024-04-29T14:42:18.276951+0000","last_fullsized":"2024-04-29T14:42:18.276951+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8625,"log_dups_size":0,"ondisk_log_size":8625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368246784,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8625,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"791'8573","reported_seq":15321,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:19.211781+0000","last_change":"2024-04-29T14:41:21.882125+0000","last_active":"2024-04-29T14:42:19.211781+0000","last_peered":"2024-04-29T14:42:19.211781+0000","last_clean":"2024-04-29T14:42:19.211781+0000","last_became_active":"2024-04-29T14:41:14.292270+0000","last_became_peered":"2024-04-29T14:41:14.292270+0000","last_unstale":"2024-04-29T14:42:19.211781+0000","last_undegraded":"2024-04-29T14:42:19.211781+0000","last_fullsized":"2024-04-29T14:42:19.211781+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8222","last_scrub_stamp":"2024-04-29T14:41:21.882016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:21.882016+0000","objects_scrubbed":5821,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:41.833704+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358285312,"num_objects":5467,"num_object_clones":0,"num_object_copies":10934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3086,"num_bytes_recovered":198836224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1917,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-04-29T14:42:16.702882+0000","last_change":"2024-04-29T14:41:45.923167+0000","last_active":"2024-04-29T14:42:16.702882+0000","last_peered":"2024-04-29T14:42:16.702882+0000","last_clean":"2024-04-29T14:42:16.702882+0000","last_became_active":"2024-04-29T14:35:31.373888+0000","last_became_peered":"2024-04-29T14:35:31.373888+0000","last_unstale":"2024-04-29T14:42:16.702882+0000","last_undegraded":"2024-04-29T14:42:16.702882+0000","last_fullsized":"2024-04-29T14:42:16.702882+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:41:45.923057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:41:45.923057+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:42:47.339815+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5856428056,"num_objects":89363,"num_object_clones":0,"num_object_copies":178726,"num_objects_missing_on_primary":4757,"num_objects_missing":4757,"num_objects_degraded":0,"num_objects_misplaced":15347,"num_objects_unfound":0,"num_objects_dirty":89363,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140457,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85739,"num_bytes_recovered":5520424960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13103538176,"data_stored":13103530032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143886,"ondisk_log_size":143886,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122473,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384112,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2909644,"kb_used_data":2098244,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":90835508,"statfs":{"total":95995035648,"available":93015560192,"internally_reserved":0,"allocated":2148601856,"data_stored":2147694752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":830863808},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645747,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1789516,"kb_used_data":1108612,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91955636,"statfs":{"total":95995035648,"available":94162571264,"internally_reserved":0,"allocated":1135218688,"data_stored":1134311604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":697235904},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423052,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188492,"kb_used_data":1079620,"kb_used_omap":4,"kb_used_meta":108859,"kb_avail":92556660,"statfs":{"total":95995035648,"available":94778019840,"internally_reserved":0,"allocated":1105530880,"data_stored":1104623796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":111472320},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234428,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1203020,"kb_used_data":359684,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92542132,"statfs":{"total":95995035648,"available":94763143168,"internally_reserved":0,"allocated":368316416,"data_stored":367409332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":863566282},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528770,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1939288,"kb_used_data":1082064,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":91805864,"statfs":{"total":95995035648,"available":94009204736,"internally_reserved":0,"allocated":1108033536,"data_stored":1107118828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":877787590},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575315,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1538200,"kb_used_data":1146056,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92206952,"statfs":{"total":95995035648,"available":94419918848,"internally_reserved":0,"allocated":1173561344,"data_stored":1172650188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2074316,"kb_used_data":1276804,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":91670836,"statfs":{"total":95995035648,"available":93870936064,"internally_reserved":0,"allocated":1307447296,"data_stored":1306540212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497280,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2396184,"kb_used_data":2161164,"kb_used_omap":4,"kb_used_meta":235003,"kb_avail":91348968,"statfs":{"total":95995035648,"available":93541343232,"internally_reserved":0,"allocated":2213031936,"data_stored":2212121300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":240643780},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182867,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986637,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218460,"kb_used_data":361232,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92526692,"statfs":{"total":95995035648,"available":94747332608,"internally_reserved":0,"allocated":369901568,"data_stored":368988328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653193,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846480,"kb_used_data":716548,"kb_used_omap":651,"kb_used_meta":129268,"kb_avail":92898672,"statfs":{"total":95995035648,"available":95128240128,"internally_reserved":0,"allocated":733745152,"data_stored":732838068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":132371208},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1307836416,"data_stored":1307836416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1168510976,"data_stored":1168506904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2207383552,"data_stored":2207383552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":636157952,"data_stored":636157952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":367984640,"data_stored":367984640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":731709440,"data_stored":731709440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1104941056,"data_stored":1104936984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":366542848,"data_stored":366542848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1103495168,"data_stored":1103495168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1133117440,"data_stored":1133117440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2143748096,"data_stored":2143748096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":832110592,"data_stored":832110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:42:24.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-04-29T14:42:25.287 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:42:25.308 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:42:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:25.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:42:25.413 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:25.439 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:25.460 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:25.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:25.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:25.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:25 smithi194 ceph-mon[44089]: pgmap v2042: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 15285/178370 objects misplaced (8.569%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:42:25.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3001837506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:25.593 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:25 smithi121 ceph-mon[44533]: pgmap v2042: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 15285/178370 objects misplaced (8.569%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:42:25.593 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3001837506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:25.614 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:25.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:25.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:25 smithi040 ceph-mon[39581]: pgmap v2042: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 15285/178370 objects misplaced (8.569%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:42:25.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3001837506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:25.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:25.744 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:25.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:25.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:25.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:25.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:25.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:42:25.894 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:25.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:25.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:25.982 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:26.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:26.172 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:26.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:26.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:26.280 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:26.308 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:26.332 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:26.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:26.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:26.466 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:26.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:42:26.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:42:26.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3001837506' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:26.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: osdmap e794: 12 total, 12 up, 10 in 2024-04-29T14:42:26.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:26.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:26.484 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:26.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:26.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:26.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:26.485 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:26 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:26.485 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:26.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:42:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:42:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3001837506' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: osdmap e794: 12 total, 12 up, 10 in 2024-04-29T14:42:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:26.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:26 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:26.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:26.613 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:42:26.613 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:42:26.613 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3001837506' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:26.613 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: osdmap e794: 12 total, 12 up, 10 in 2024-04-29T14:42:26.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:42:26.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:42:26.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:42:26.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:42:26.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:42:26.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:42:26.614 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:26 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:42:26.617 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:26.633 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:26.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:26.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:26.764 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:26.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:26.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:26.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:26.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:26.914 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:26.934 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:26.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:26.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:27.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:27.066 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:27.088 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:27.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:27.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:27.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:27.220 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:27.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:27.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:27.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:27.380 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:27.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:27 smithi040 ceph-mon[39581]: pgmap v2044: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 14980/177828 objects misplaced (8.424%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:42:27.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:27 smithi121 ceph-mon[44533]: pgmap v2044: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 14980/177828 objects misplaced (8.424%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:42:27.634 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:27.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:27.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:27.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:27 smithi194 ceph-mon[44089]: pgmap v2044: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 14980/177828 objects misplaced (8.424%); 6.5 MiB/s, 104 objects/s recovering 2024-04-29T14:42:28.121 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:28.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:28.144 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:28.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:28.415 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:28.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:28.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:42:28.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:28.551 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:28.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:28.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:28.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:28 smithi040 ceph-mon[39581]: osdmap e795: 12 total, 12 up, 10 in 2024-04-29T14:42:28.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:28 smithi040 ceph-mon[39581]: pgmap v2046: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 14980/177828 objects misplaced (8.424%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:42:28.702 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:28.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:28.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:28 smithi121 ceph-mon[44533]: osdmap e795: 12 total, 12 up, 10 in 2024-04-29T14:42:28.722 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:28 smithi121 ceph-mon[44533]: pgmap v2046: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 14980/177828 objects misplaced (8.424%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:42:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:28.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:28 smithi194 ceph-mon[44089]: osdmap e795: 12 total, 12 up, 10 in 2024-04-29T14:42:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:28 smithi194 ceph-mon[44089]: pgmap v2046: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 14980/177828 objects misplaced (8.424%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:42:28.824 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:28.850 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:28.871 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:28.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:28.899 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:42:28.899 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:42:28.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:29.007 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:29.560 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:29.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:29.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:29.696 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:29.715 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:29.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:29.822 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:29.856 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:29.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:30.041 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:30.068 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:30.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:30.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:30.219 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:30.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:30.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:30.340 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:30.367 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:30.387 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:30.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:30.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:30.523 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:30.608 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:30.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:30.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:30.755 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:30.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:30.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:30.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:31.273 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 1 pgp_num to 1 2024-04-29T14:42:31.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:31 smithi040 ceph-mon[39581]: pgmap v2047: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 14915/177148 objects misplaced (8.420%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:42:31.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3869143542' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:42:31.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:31 smithi040 ceph-mon[39581]: from='client.27376 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:42:31.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:31 smithi194 ceph-mon[44089]: pgmap v2047: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 14915/177148 objects misplaced (8.420%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:42:31.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3869143542' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:42:31.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:31 smithi194 ceph-mon[44089]: from='client.27376 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:42:31.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:31 smithi121 ceph-mon[44533]: pgmap v2047: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 14915/177148 objects misplaced (8.420%); 3.0 MiB/s, 47 objects/s recovering 2024-04-29T14:42:31.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3869143542' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:42:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:31 smithi121 ceph-mon[44533]: from='client.27376 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-29T14:42:31.850 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:42:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:42:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:42:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:42:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:42:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:42:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:42:32.062 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:32.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:32.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:32.195 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:32.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:32.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:32.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:32.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: from='client.27376 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-29T14:42:32.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osdmap e796: 12 total, 12 up, 10 in 2024-04-29T14:42:32.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2290693300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:32.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.350 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.351 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.353 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:32.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:32.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:42:32.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:32.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: from='client.27376 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-29T14:42:32.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osdmap e796: 12 total, 12 up, 10 in 2024-04-29T14:42:32.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2290693300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:32.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: from='client.27376 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-29T14:42:32.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osdmap e796: 12 total, 12 up, 10 in 2024-04-29T14:42:32.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2290693300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:32.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.584 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:32.586 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:32.606 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:32.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:32.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:32.740 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:32.761 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:32.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:32.809 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:42:32.866 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:32.894 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:32.977 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:32.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:33.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:33.109 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:33.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:33.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:33.258 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:33.279 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:33.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:33.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:33.383 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:33 smithi121 ceph-mon[44533]: pgmap v2049: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 14597/176664 objects misplaced (8.263%); 3.2 MiB/s, 50 objects/s recovering 2024-04-29T14:42:33.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:33 smithi121 ceph-mon[44533]: osdmap e797: 12 total, 12 up, 10 in 2024-04-29T14:42:33.385 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:33.415 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:33.436 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:33.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:33.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:33.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:33.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:33 smithi040 ceph-mon[39581]: pgmap v2049: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 14597/176664 objects misplaced (8.263%); 3.2 MiB/s, 50 objects/s recovering 2024-04-29T14:42:33.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:33 smithi040 ceph-mon[39581]: osdmap e797: 12 total, 12 up, 10 in 2024-04-29T14:42:33.567 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:33 smithi194 ceph-mon[44089]: pgmap v2049: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 14597/176664 objects misplaced (8.263%); 3.2 MiB/s, 50 objects/s recovering 2024-04-29T14:42:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:33 smithi194 ceph-mon[44089]: osdmap e797: 12 total, 12 up, 10 in 2024-04-29T14:42:33.585 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:33.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:33.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:33.716 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:33.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:33.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:33.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:34.119 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:34.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:34.224 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:34.254 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:34.605 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:34.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:34.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:34.737 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:34.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:34.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:34.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:34.887 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:34.906 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:34.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:34.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:35.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:42:35.044 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:35.283 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:42:35.295 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:42:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:35.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:42:35.389 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:42:35.390 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:42:35.390 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:42:35.390 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:42:35.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:35.429 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:35.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:35.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:35.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:35 smithi194 ceph-mon[44089]: pgmap v2051: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 14533/176156 objects misplaced (8.250%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:42:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/991660384' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:42:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:35 smithi194 ceph-mon[44089]: from='client.37145 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:42:35.576 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:35 smithi121 ceph-mon[44533]: pgmap v2051: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 14533/176156 objects misplaced (8.250%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:42:35.576 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/991660384' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:42:35.576 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:35 smithi121 ceph-mon[44533]: from='client.37145 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:42:35.578 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:35.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:35.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:35.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:35 smithi040 ceph-mon[39581]: pgmap v2051: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 14533/176156 objects misplaced (8.250%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:42:35.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/991660384' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:42:35.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:35 smithi040 ceph-mon[39581]: from='client.37145 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:42:35.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:35.732 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:36.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:42:36.182 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:36.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:36.217 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:42:36.287 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:36.329 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:36.349 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:36.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:36.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:36 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:36.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:36 smithi040 ceph-mon[39581]: from='client.37145 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:42:36.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:36 smithi040 ceph-mon[39581]: osdmap e798: 12 total, 12 up, 10 in 2024-04-29T14:42:36.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:36.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:36.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:36.471 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:36.479 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:36.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:36.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:36.555 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:42:36.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:36 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:36.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:36 smithi194 ceph-mon[44089]: from='client.37145 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:42:36.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:36 smithi194 ceph-mon[44089]: osdmap e798: 12 total, 12 up, 10 in 2024-04-29T14:42:36.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:36.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:36.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:36.629 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:36.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:36 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:36.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:36 smithi121 ceph-mon[44533]: from='client.37145 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:42:36.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:36 smithi121 ceph-mon[44533]: osdmap e798: 12 total, 12 up, 10 in 2024-04-29T14:42:36.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:36.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:36.827 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:36.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:36.931 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:42:36.960 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:37.058 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:42:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:37.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:42:37.162 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:37.192 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:37.443 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:37.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:37.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:37.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:37 smithi194 ceph-mon[44089]: pgmap v2053: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 14207/175574 objects misplaced (8.092%); 7.0 MiB/s, 112 objects/s recovering 2024-04-29T14:42:37.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:37 smithi194 ceph-mon[44089]: osdmap e799: 12 total, 12 up, 10 in 2024-04-29T14:42:37.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:37.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:37.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:37 smithi121 ceph-mon[44533]: pgmap v2053: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 14207/175574 objects misplaced (8.092%); 7.0 MiB/s, 112 objects/s recovering 2024-04-29T14:42:37.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:37 smithi121 ceph-mon[44533]: osdmap e799: 12 total, 12 up, 10 in 2024-04-29T14:42:37.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:37.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:37.576 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:37.597 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:37.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:37.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:37 smithi040 ceph-mon[39581]: pgmap v2053: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 14207/175574 objects misplaced (8.092%); 7.0 MiB/s, 112 objects/s recovering 2024-04-29T14:42:37.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:37 smithi040 ceph-mon[39581]: osdmap e799: 12 total, 12 up, 10 in 2024-04-29T14:42:37.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:37.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:37.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:37.733 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:37.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:37.828 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] out_osds: [3, 11] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:42:37.828 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:42:37.829 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.4193411677631792 2024-04-29T14:42:37.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 6 0.4193411677631792 2024-04-29T14:42:37.893 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:37.920 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:38.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:42:38.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:38 smithi040 ceph-mon[39581]: osdmap e800: 12 total, 12 up, 10 in 2024-04-29T14:42:38.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:38 smithi040 ceph-mon[39581]: osdmap e801: 12 total, 12 up, 10 in 2024-04-29T14:42:38.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:42:38.407 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:42:38.409 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:38.443 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:42:38.467 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:42:38.493 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:42:38.514 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:42:38.536 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:42:38.559 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:42:38.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:38 smithi194 ceph-mon[44089]: osdmap e800: 12 total, 12 up, 10 in 2024-04-29T14:42:38.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:38 smithi194 ceph-mon[44089]: osdmap e801: 12 total, 12 up, 10 in 2024-04-29T14:42:38.583 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:42:38.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:42:38.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:42:38.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:38 smithi121 ceph-mon[44533]: osdmap e800: 12 total, 12 up, 10 in 2024-04-29T14:42:38.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:38 smithi121 ceph-mon[44533]: osdmap e801: 12 total, 12 up, 10 in 2024-04-29T14:42:38.646 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:42:38.647 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:42:38.647 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:42:38.647 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:42:38.647 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:42:38.648 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:42:38.772 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:38.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:38.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:39.297 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: pgmap v2057: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 14207/175574 objects misplaced (8.092%); 5.1 MiB/s, 81 objects/s recovering 2024-04-29T14:42:39.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4108771323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:39.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.299 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:39 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: pgmap v2057: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 14207/175574 objects misplaced (8.092%); 5.1 MiB/s, 81 objects/s recovering 2024-04-29T14:42:39.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4108771323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:39 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: pgmap v2057: 17 pgs: 2 active+recovering, 15 active+clean; 5.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 14207/175574 objects misplaced (8.092%); 5.1 MiB/s, 81 objects/s recovering 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4108771323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:39 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:39.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:39.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:39.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:39.901 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:40.552 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:40.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:40.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:40.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:40.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:40.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:40.694 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:40.896 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:40.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:41.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:41.037 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:41.068 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:41.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:41.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:41.202 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:41.279 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:41.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:41.318 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.6 to 0.419341 (6b59) 2024-04-29T14:42:41.382 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:41.412 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:41.431 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:42:41.432 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:42:41.435 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:41.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:41.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:41.456 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:41 smithi194 ceph-mon[44089]: pgmap v2058: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 14140/175096 objects misplaced (8.076%); 4.6 MiB/s, 72 objects/s recovering 2024-04-29T14:42:41.457 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1447290514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.4193411677631792}]: dispatch 2024-04-29T14:42:41.539 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:41.568 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:41.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:41 smithi040 ceph-mon[39581]: pgmap v2058: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 14140/175096 objects misplaced (8.076%); 4.6 MiB/s, 72 objects/s recovering 2024-04-29T14:42:41.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1447290514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.4193411677631792}]: dispatch 2024-04-29T14:42:41.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:41 smithi121 ceph-mon[44533]: pgmap v2058: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 14140/175096 objects misplaced (8.076%); 4.6 MiB/s, 72 objects/s recovering 2024-04-29T14:42:41.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1447290514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.4193411677631792}]: dispatch 2024-04-29T14:42:41.817 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:41.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:41.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:42:41.953 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:42.319 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1447290514' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.4193411677631792}]': finished 2024-04-29T14:42:42.319 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:42 smithi040 ceph-mon[39581]: osdmap e802: 12 total, 12 up, 10 in 2024-04-29T14:42:42.319 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:42 smithi040 ceph-mon[39581]: osdmap e803: 12 total, 12 up, 10 in 2024-04-29T14:42:42.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:42:42.404 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:42:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:42.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:42:42.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:42.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1447290514' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.4193411677631792}]': finished 2024-04-29T14:42:42.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:42 smithi194 ceph-mon[44089]: osdmap e802: 12 total, 12 up, 10 in 2024-04-29T14:42:42.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:42 smithi194 ceph-mon[44089]: osdmap e803: 12 total, 12 up, 10 in 2024-04-29T14:42:42.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1447290514' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.4193411677631792}]': finished 2024-04-29T14:42:42.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:42 smithi121 ceph-mon[44533]: osdmap e802: 12 total, 12 up, 10 in 2024-04-29T14:42:42.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:42 smithi121 ceph-mon[44533]: osdmap e803: 12 total, 12 up, 10 in 2024-04-29T14:42:43.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:43 smithi194 ceph-mon[44089]: pgmap v2060: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 64 op/s; 13792/174814 objects misplaced (7.890%); 4.0 MiB/s, 64 objects/s recovering 2024-04-29T14:42:43.573 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:43 smithi040 ceph-mon[39581]: pgmap v2060: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 64 op/s; 13792/174814 objects misplaced (7.890%); 4.0 MiB/s, 64 objects/s recovering 2024-04-29T14:42:43.592 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:43.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:43.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:43 smithi121 ceph-mon[44533]: pgmap v2060: 17 pgs: 2 active+recovering, 15 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 64 op/s; 13792/174814 objects misplaced (7.890%); 4.0 MiB/s, 64 objects/s recovering 2024-04-29T14:42:43.697 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:43.726 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:44.143 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:44.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:44.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:44.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:44.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:44.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:44.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:45.346 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:42:45.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:45 smithi040 ceph-mon[39581]: pgmap v2062: 17 pgs: 4 active+recovering, 13 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 24570/174342 objects misplaced (14.093%); 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:42:45.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:45 smithi040 ceph-mon[39581]: osdmap e804: 12 total, 12 up, 10 in 2024-04-29T14:42:45.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2716540047' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:45.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:45 smithi040 ceph-mon[39581]: from='client.27406 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:45.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:45 smithi121 ceph-mon[44533]: pgmap v2062: 17 pgs: 4 active+recovering, 13 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 24570/174342 objects misplaced (14.093%); 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:42:45.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:45 smithi121 ceph-mon[44533]: osdmap e804: 12 total, 12 up, 10 in 2024-04-29T14:42:45.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2716540047' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:45.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:45 smithi121 ceph-mon[44533]: from='client.27406 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:45.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:45 smithi194 ceph-mon[44089]: pgmap v2062: 17 pgs: 4 active+recovering, 13 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 24570/174342 objects misplaced (14.093%); 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:42:45.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:45 smithi194 ceph-mon[44089]: osdmap e804: 12 total, 12 up, 10 in 2024-04-29T14:42:45.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2716540047' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:45.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:45 smithi194 ceph-mon[44089]: from='client.27406 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:42:46.604 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:46 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:46 smithi194 ceph-mon[44089]: from='client.27406 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:46.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:46 smithi194 ceph-mon[44089]: osdmap e805: 12 total, 12 up, 10 in 2024-04-29T14:42:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:46.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:46.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:46 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:46.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:46 smithi040 ceph-mon[39581]: from='client.27406 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:46.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:46 smithi040 ceph-mon[39581]: osdmap e805: 12 total, 12 up, 10 in 2024-04-29T14:42:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:46 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:46 smithi121 ceph-mon[44533]: from='client.27406 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:42:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:46 smithi121 ceph-mon[44533]: osdmap e805: 12 total, 12 up, 10 in 2024-04-29T14:42:46.644 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:42:46.709 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:46.755 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:46.782 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:46.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:46.888 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:46.917 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:46.942 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:46.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:47.039 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:42:47.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:47.063 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:42:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:42:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:47.119 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:42:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:47.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:47.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:42:47.165 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:42:47.188 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:42:47.191 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:42:47.209 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:42:47.228 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:47.234 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:42:47.256 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:42:47.257 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:47.280 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:42:47.296 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:47.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:47.310 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:42:47.311 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:42:47.311 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:42:47.311 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:42:47.312 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:42:47.313 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:42:47.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:47.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:47.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:47.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:47.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:47 smithi194 ceph-mon[44089]: pgmap v2065: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 29687/173940 objects misplaced (17.067%); 6.9 MiB/s, 111 objects/s recovering 2024-04-29T14:42:47.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3034798491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:47.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:47.586 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:47.605 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:47.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:47 smithi121 ceph-mon[44533]: pgmap v2065: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 29687/173940 objects misplaced (17.067%); 6.9 MiB/s, 111 objects/s recovering 2024-04-29T14:42:47.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3034798491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:47.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:47.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:47.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:47 smithi040 ceph-mon[39581]: pgmap v2065: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 29687/173940 objects misplaced (17.067%); 6.9 MiB/s, 111 objects/s recovering 2024-04-29T14:42:47.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3034798491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:47.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:47.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:47.739 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:48.603 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] out_osds: [3, 11] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:42:48.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:42:48.603 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-29T14:42:48.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-29T14:42:48.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.766 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:42:48.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:49.390 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:49.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:49.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:49.524 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:49.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:49.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:49.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:49 smithi040 ceph-mon[39581]: pgmap v2066: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 29687/173940 objects misplaced (17.067%); 5.7 MiB/s, 90 objects/s recovering 2024-04-29T14:42:49.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:49 smithi040 ceph-mon[39581]: osdmap e806: 12 total, 12 up, 10 in 2024-04-29T14:42:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:49.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:49 smithi121 ceph-mon[44533]: pgmap v2066: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 29687/173940 objects misplaced (17.067%); 5.7 MiB/s, 90 objects/s recovering 2024-04-29T14:42:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:49 smithi121 ceph-mon[44533]: osdmap e806: 12 total, 12 up, 10 in 2024-04-29T14:42:49.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:42:49.684 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:49.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:49.773 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:49.773 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:49 smithi194 ceph-mon[44089]: pgmap v2066: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 29687/173940 objects misplaced (17.067%); 5.7 MiB/s, 90 objects/s recovering 2024-04-29T14:42:49.773 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:49 smithi194 ceph-mon[44089]: osdmap e806: 12 total, 12 up, 10 in 2024-04-29T14:42:49.889 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:42:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:49.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:42:49.904 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:49.997 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:50.027 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:50.050 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:50.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:50.155 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:50.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:42:50.185 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:50.938 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:50.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:51.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:51.075 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:51.457 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:51.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:51.562 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:42:51.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:51 smithi194 ceph-mon[44089]: pgmap v2068: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 29365/173420 objects misplaced (16.933%); 7.8 MiB/s, 125 objects/s recovering 2024-04-29T14:42:51.595 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:51.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:51 smithi040 ceph-mon[39581]: pgmap v2068: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 29365/173420 objects misplaced (16.933%); 7.8 MiB/s, 125 objects/s recovering 2024-04-29T14:42:51.617 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:42:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:51.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:42:51.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:51 smithi121 ceph-mon[44533]: pgmap v2068: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 29365/173420 objects misplaced (16.933%); 7.8 MiB/s, 125 objects/s recovering 2024-04-29T14:42:51.721 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:42:51.751 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:51.774 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:42:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:51.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:42:51.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:51.914 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:51.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:51.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:52.040 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:42:52.068 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:52.094 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:42:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:52.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:42:52.185 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:42:52.185 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:42:52.185 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 806, 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "998f2e22-0630-11ef-bc93-c7b262605968", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-29T14:00:01.568926+0000", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-29T14:42:48.468363+0000", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "last_up_change": "2024-04-29T14:31:03.499853+0000", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "last_in_change": "2024-04-29T14:42:09.481114+0000", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_num": 5806080, 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_set": [ 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "nodeep-scrub", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "noscrub", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "pglog_hardlimit", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "purged_snapdirs", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_deletes", 2024-04-29T14:42:52.186 INFO:teuthology.orchestra.run.smithi040.stdout: "sortbitwise" 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_version": 30, 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "full_ratio": 0.94999998807907104, 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_snapshot": "", 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_max": 7, 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd": 12, 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "require_min_compat_client": "luminous", 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "min_compat_client": "luminous", 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "require_osd_release": "squid", 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_crimson": false, 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "pools": [ 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 1, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": ".mgr", 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:03:05.389591+0000", 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:42:52.188 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 1, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 1, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 1, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 1, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 1, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "796", 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:42:52.189 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:42:52.190 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:42:52.191 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "options": { 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_max": 32, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_min": 1 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": {} 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 10, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 10, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.20000000298023224, 2024-04-29T14:42:52.192 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "pool": 7, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": "unique_pool_5", 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "create_time": "2024-04-29T14:34:47.804723+0000", 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flags_names": "hashpspool", 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "type": 1, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 2, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": 2, 2024-04-29T14:42:52.193 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_rule": 0, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_count": 0, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_target": 0, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_barrier": 0, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "object_hash": 2, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscale_mode": "off", 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": 16, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num": 16, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_placement_num_target": 16, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_target": 16, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num_pending": 16, 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "last_pg_merge_meta": { 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "source_pgid": "0.0", 2024-04-29T14:42:52.194 INFO:teuthology.orchestra.run.smithi040.stdout: "ready_epoch": 0, 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_started": 0, 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "last_epoch_clean": 0, 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "source_version": "0'0", 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "target_version": "0'0" 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "last_change": "659", 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend": "0", 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "last_force_op_resend_preluminous": "0", 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "auid": 0, 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_mode": "selfmanaged", 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_seq": 0, 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "snap_epoch": 0, 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_snaps": [], 2024-04-29T14:42:52.195 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps": "[]", 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_bytes": 0, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "quota_max_objects": 0, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "tiers": [], 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "tier_of": -1, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "read_tier": -1, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "write_tier": -1, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_mode": "none", 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_bytes": 0, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_objects": 0, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_flush_age": 0, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "cache_min_evict_age": 0, 2024-04-29T14:42:52.196 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profile": "", 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_params": { 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "none" 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_period": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_count": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "use_gmt_hitset": true, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "min_read_recency_for_promote": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "min_write_recency_for_promote": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_grade_decay_rate": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "hit_set_search_last_n": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "grade_table": [], 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "stripe_width": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "expected_num_objects": 0, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "fast_read": false, 2024-04-29T14:42:52.197 INFO:teuthology.orchestra.run.smithi040.stdout: "options": {}, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "application_metadata": { 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rados": {} 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "read_balance": { 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "score_type": "Fair distribution", 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "score_acting": 2.5, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "score_stable": 2.5, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "optimal_score": 0.89999997615814209, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_acting": 2.25, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "raw_score_stable": 2.25, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity": 1, 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.198 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "967eec9c-202c-4f98-a22a-a67db85bd462", 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 8, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 765, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6802", 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.200 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6804", 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6803", 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4255137508 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.201 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6801/4255137508", 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6802/4255137508", 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6804/4255137508", 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6803/4255137508", 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "7f52729c-855b-465c-b134-8326f99f1422", 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.202 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.7887725830078125, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 13, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 802, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6805", 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6806", 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6808", 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.204 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6807", 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4178456629 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6805/4178456629", 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6806/4178456629", 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6808/4178456629", 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6807/4178456629", 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.205 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "372baf31-26e1-4865-a272-898e7ff0c2f3", 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 20, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 230, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 260, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 802, 2024-04-29T14:42:52.206 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 231, 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6809", 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6810", 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:42:52.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6812", 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.208 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6811", 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3416690575 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6809/3416690575", 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6810/3416690575", 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6812/3416690575", 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6811/3416690575", 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:42:52.209 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "975c2fbc-115b-4950-88ef-b0b2ad778765", 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 25, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 748, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6813", 2024-04-29T14:42:52.210 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6814", 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.211 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6816", 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6815", 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 4012292460 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6813/4012292460", 2024-04-29T14:42:52.212 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.40:6814/4012292460", 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.40:6816/4012292460", 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.40:6815/4012292460", 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6aa13ecc-0959-4915-a788-03c4739da269", 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.5530548095703125, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.213 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 31, 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 629, 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6800", 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.214 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6801", 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6803", 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.215 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6802", 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3987960551 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6800/3987960551", 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6801/3987960551", 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6803/3987960551", 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6802/3987960551", 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6", 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 388, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 457, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 561, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 703, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 459, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6804", 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6805", 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6807", 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6806", 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2522198013 2024-04-29T14:42:52.219 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6804/2522198013", 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6805/2522198013", 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6807/2522198013", 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6806/2522198013", 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "40ecb14f-7b61-45d2-b48c-0a374912edcf", 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.220 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.4193267822265625, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 45, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 763, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6808", 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.221 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6809", 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6811", 2024-04-29T14:42:52.222 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6810", 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1755659098 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6808/1755659098", 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6809/1755659098", 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6811/1755659098", 2024-04-29T14:42:52.223 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6810/1755659098", 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "38e15653-4a3d-4543-8240-e17d95bdfe27", 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0.4615020751953125, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 54, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 634, 2024-04-29T14:42:52.224 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6812", 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6813", 2024-04-29T14:42:52.225 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6815", 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.226 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.121:6814", 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 82319362 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.121:6812/82319362", 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.121:6813/82319362", 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.121:6815/82319362", 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.121:6814/82319362", 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:42:52.227 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "d93f2b45-6c82-4ead-9a66-e86f9e83e0de", 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 64, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 320, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 354, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 802, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 321, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6800", 2024-04-29T14:42:52.228 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6801", 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.229 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6803", 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6802", 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 165633203 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6800/165633203", 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6801/165633203", 2024-04-29T14:42:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6803/165633203", 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6802/165633203", 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "a720d016-7cf7-4755-a25b-4d081c1f13a0", 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 72, 2024-04-29T14:42:52.231 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 788, 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6804", 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6805", 2024-04-29T14:42:52.232 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6807", 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6806", 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 453882699 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6804/453882699", 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6805/453882699", 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6807/453882699", 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6806/453882699", 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.234 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "87c864f5-90a1-48e9-bca8-65cf404bc2a0", 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 1, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 1, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 1, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 80, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 766, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.235 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6808", 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6809", 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6811", 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6810", 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1320459116 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6808/1320459116", 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6809/1320459116", 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6811/1320459116", 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6810/1320459116", 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "uuid": "5c85066b-5780-4393-be30-0033b9d2dd1f", 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "up": 1, 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "in": 0, 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-29T14:42:52.238 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_affinity": 0, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_begin": 0, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "last_clean_end": 0, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "up_from": 88, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "up_thru": 524, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "down_at": 0, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "lost_at": 0, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6812", 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addrs": { 2024-04-29T14:42:52.239 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6813", 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addrs": { 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6815", 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addrs": { 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.194:6814", 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 20977082 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.194:6812/20977082", 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "172.21.15.194:6813/20977082", 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_back_addr": "172.21.15.194:6815/20977082", 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_front_addr": "172.21.15.194:6814/20977082", 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "state": [ 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "exists", 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: "up" 2024-04-29T14:42:52.241 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_xinfo": [ 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 0, 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:42:21.214364+0000", 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 1, 2024-04-29T14:42:52.242 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:42:20.766885+0000", 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 2, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:41:37.424509+0000", 2024-04-29T14:42:52.243 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 230 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 3, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 65536, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:42:48.174082+0000", 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 4, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.244 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:41:27.805653+0000", 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 5, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:41:52.745970+0000", 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 457 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 6, 2024-04-29T14:42:52.245 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:42:14.483436+0000", 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 7, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:41:54.990137+0000", 2024-04-29T14:42:52.246 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 8, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:42:43.661764+0000", 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 320 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 9, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.247 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:42:35.300540+0000", 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 10, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 0, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:42:36.294708+0000", 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.248 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": 11, 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "down_stamp": "0.000000", 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_probability": 0, 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "laggy_interval": 0, 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "features": 4540701547738038271, 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "old_weight": 28165, 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "last_purged_snaps_scrub": "2024-04-29T14:41:51.042976+0000", 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "dead_epoch": 0 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap": [ 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "7.9", 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: "osds": [ 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: 8 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.249 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_items": [ 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "pgid": "1.0", 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "mappings": [ 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "from": 9, 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "to": 2 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_upmap_primaries": [], 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_temp": [], 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "primary_temp": [], 2024-04-29T14:42:52.250 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklist": { 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3316779772": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2003707927": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/2722668908": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/2004115894": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3681930020": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:6800/3860863692": "2024-04-30T14:01:01.302559+0000", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/3976042573": "2024-04-30T14:00:29.409543+0000", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "172.21.15.40:0/1925168828": "2024-04-30T14:01:01.302559+0000" 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "range_blocklist": {}, 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_profiles": { 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "default": { 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-failure-domain": "osd", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "k": "2", 2024-04-29T14:42:52.251 INFO:teuthology.orchestra.run.smithi040.stdout: "m": "1", 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin": "jerasure", 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "technique": "reed_sol_van" 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_snaps_queue": [], 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "new_removed_snaps": [], 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "new_purged_snaps": [], 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_node_flags": {}, 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "device_class_flags": {}, 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": { 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_enabled": false, 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_bucket_count": 0, 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "degraded_stretch_mode": 0, 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "recovering_stretch_mode": 0, 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode_bucket": 0 2024-04-29T14:42:52.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:42:52.253 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:42:52.253 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:52.280 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:52.304 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:52.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:52.354 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2376089503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:42:52.407 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:42:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:52.493 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:42:52.493 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:42:52.493 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:42:52.493 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:42:52.493 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.493 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123594 7.f 7:f88676ca:::benchmark_data_smithi194_147200_object8682:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e721)", 2024-04-29T14:42:52.493 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418053+0000", 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.80314031699999, 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022452723000000001, 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123594 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.494 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418053+0000", 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418053+0000", 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418055+0000", 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0150000000000002e-06 2024-04-29T14:42:52.495 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418063+0000", 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6070000000000008e-06 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418076+0000", 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2479e-05 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.496 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418088+0000", 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1691e-05 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418131+0000", 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3395e-05 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418138+0000", 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6699999999999997e-06 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.497 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.429115+0000", 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010977356000000001 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.429205+0000", 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0369000000000002e-05 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.429293+0000", 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7244999999999996e-05 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.498 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436060+0000", 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067675629999999999 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.440444+0000", 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043834549999999996 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.440478+0000", 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4047000000000001e-05 2024-04-29T14:42:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.440506+0000", 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7830999999999998e-05 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123596 7.f 7:f67ab9f8:::benchmark_data_smithi194_147200_object8684:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418106+0000", 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.80308682699999, 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022484299999999999, 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.500 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123596 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418106+0000", 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.501 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418106+0000", 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418107+0000", 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.85e-07 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418110+0000", 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9830000000000001e-06 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.502 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418117+0000", 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.883e-06 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418123+0000", 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0769999999999996e-06 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418162+0000", 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9155e-05 2024-04-29T14:42:52.503 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418171+0000", 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3739999999999998e-06 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.431076+0000", 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012904964999999999 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.431152+0000", 2024-04-29T14:42:52.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6197000000000002e-05 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.431253+0000", 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010157699999999999 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436109+0000", 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048559409999999999 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.440535+0000", 2024-04-29T14:42:52.505 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044255939999999997 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.440565+0000", 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0473000000000001e-05 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.440591+0000", 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5296e-05 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123598 7.f 7:f9438043:::benchmark_data_smithi194_147200_object8686:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418131+0000", 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.803062047, 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025363633999999999, 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123598 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.507 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418131+0000", 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418131+0000", 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418132+0000", 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0100000000000005e-07 2024-04-29T14:42:52.508 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418134+0000", 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0870000000000001e-06 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418137+0000", 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.393e-06 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418140+0000", 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0910000000000001e-06 2024-04-29T14:42:52.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418205+0000", 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5304999999999999e-05 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418212+0000", 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4989999999999999e-06 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.432790+0000", 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014578367 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.432893+0000", 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103038 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.432995+0000", 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101888 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436112+0000", 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031165659999999999 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.443403+0000", 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072915719999999996 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.443455+0000", 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1647999999999997e-05 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.443495+0000", 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9678999999999997e-05 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123646 7.f 7:fb3f94fb:::benchmark_data_smithi194_147200_object8734:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418158+0000", 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.803035435, 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025489161, 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.513 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123646 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418158+0000", 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418158+0000", 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418158+0000", 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.51e-07 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418160+0000", 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.984e-06 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418163+0000", 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.05e-06 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418167+0000", 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.031e-06 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418236+0000", 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8926999999999999e-05 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418243+0000", 2024-04-29T14:42:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5629999999999997e-06 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.434547+0000", 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016304279000000001 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.434614+0000", 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6585e-05 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.434677+0000", 2024-04-29T14:42:52.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3792000000000002e-05 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436207+0000", 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015299879999999999 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.443571+0000", 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073633479999999996 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.443603+0000", 2024-04-29T14:42:52.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2314999999999998e-05 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.443647+0000", 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3847999999999998e-05 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123595 7.9 7:9dc07aec:::benchmark_data_smithi194_147200_object8683:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e721)", 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418173+0000", 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.80302059499999, 2024-04-29T14:42:52.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023377430000000001, 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123595 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418173+0000", 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418173+0000", 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418173+0000", 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3599999999999999e-07 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418175+0000", 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9099999999999999e-06 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418181+0000", 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9920000000000004e-06 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418186+0000", 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0459999999999996e-06 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418227+0000", 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1245000000000002e-05 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418233+0000", 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9540000000000001e-06 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.431241+0000", 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013008126 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.431333+0000", 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1248000000000005e-05 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.431397+0000", 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4869000000000005e-05 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438277+0000", 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00687924 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.441453+0000", 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031766020000000002 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.441508+0000", 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5046000000000001e-05 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.441550+0000", 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1715999999999998e-05 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123609 7.9 7:904f1bc6:::benchmark_data_smithi194_147200_object8697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418202+0000", 2024-04-29T14:42:52.525 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.80299122, 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031149342999999999, 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123609 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418202+0000", 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418202+0000", 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418203+0000", 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5400000000000001e-07 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418205+0000", 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4609999999999998e-06 2024-04-29T14:42:52.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418210+0000", 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4049999999999999e-06 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418215+0000", 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0250000000000002e-06 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418256+0000", 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0725000000000001e-05 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418264+0000", 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7020000000000001e-06 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.433343+0000", 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015079441000000001 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.433454+0000", 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110587 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.433564+0000", 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109885 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438324+0000", 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047597919999999997 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449133+0000", 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010809272999999999 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449293+0000", 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016066299999999999 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449351+0000", 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.783e-05 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123615 7.9 7:9af3026f:::benchmark_data_smithi194_147200_object8703:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:42:52.531 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418222+0000", 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.80297074000001, 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031260789999999997, 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123615 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418222+0000", 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418222+0000", 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418223+0000", 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4199999999999996e-07 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418234+0000", 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.093e-05 2024-04-29T14:42:52.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418239+0000", 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7419999999999997e-06 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418245+0000", 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3339999999999998e-06 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418282+0000", 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6724000000000003e-05 2024-04-29T14:42:52.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418285+0000", 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4290000000000001e-06 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.435211+0000", 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016926225 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.435315+0000", 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103501 2024-04-29T14:42:52.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.435420+0000", 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105408 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438328+0000", 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029076879999999999 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449399+0000", 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011071263 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449446+0000", 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7166999999999998e-05 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449483+0000", 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6837e-05 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123619 7.9 7:98d7b41f:::benchmark_data_smithi194_147200_object8707:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418252+0000", 2024-04-29T14:42:52.537 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.802940936, 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031387258000000001, 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123619 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418252+0000", 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418252+0000", 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418253+0000", 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5400000000000001e-07 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418259+0000", 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4960000000000001e-06 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418264+0000", 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0039999999999999e-06 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418269+0000", 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1610000000000002e-06 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418312+0000", 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2868999999999999e-05 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418317+0000", 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2549999999999999e-06 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436914+0000", 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018597783 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436984+0000", 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9375999999999994e-05 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.437046+0000", 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2255999999999995e-05 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.439736+0000", 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026896279999999999 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449541+0000", 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0098051990000000006 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449598+0000", 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7167999999999999e-05 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449639+0000", 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1508999999999998e-05 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123632 7.9 7:90a1475f:::benchmark_data_smithi194_147200_object8720:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 251.80291043700001, 2024-04-29T14:42:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031508860999999999, 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123632 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418286+0000", 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.424e-06 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418291+0000", 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9230000000000001e-06 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418296+0000", 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2259999999999996e-06 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418337+0000", 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1319999999999997e-05 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:42:52.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418344+0000", 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4790000000000001e-06 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438358+0000", 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020013986000000001 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438463+0000", 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010518000000000001 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438567+0000", 2024-04-29T14:42:52.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010400600000000001 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.439853+0000", 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001286424 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449691+0000", 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0098375330000000007 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449748+0000", 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7336999999999997e-05 2024-04-29T14:42:52.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449792+0000", 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3399000000000002e-05 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143096 7.7 7:edde144f:::benchmark_data_smithi194_147200_object28184:head [delete] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:47.477922+0000", 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.7432706920000003, 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019058129, 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.549 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143096 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.477922+0000", 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.477922+0000", 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.477925+0000", 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6769999999999999e-06 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.477934+0000", 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4849999999999992e-06 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.477949+0000", 2024-04-29T14:42:52.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5279999999999999e-05 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.477959+0000", 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5980000000000003e-06 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.481967+0000", 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040086169999999999 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.482130+0000", 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016242700000000001 2024-04-29T14:42:52.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.482280+0000", 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015039299999999999 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.487069+0000", 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004788952 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.495670+0000", 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086008660000000004 2024-04-29T14:42:52.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.495730+0000", 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0192999999999998e-05 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:47.496981+0000", 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001250641 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.554 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143148 7.7 7:eeeadc73:::benchmark_data_smithi194_147200_object28236:head [delete] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:48.304922+0000", 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.916270704, 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01866172, 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143148 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.304922+0000", 2024-04-29T14:42:52.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.304922+0000", 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.304925+0000", 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0110000000000001e-06 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.304934+0000", 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6249999999999996e-06 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.304951+0000", 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7166000000000001e-05 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.304963+0000", 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1428e-05 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.306901+0000", 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001937848 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.307038+0000", 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013738699999999999 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.307165+0000", 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012736299999999999 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.310208+0000", 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030423360000000001 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.322094+0000", 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011885886 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.322162+0000", 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8177999999999997e-05 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:48.323584+0000", 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001422492 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143231 7.7 7:ee9c6570:::benchmark_data_smithi194_147200_object28319:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:49.742553+0000", 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.4786405380000001, 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025382690999999999, 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143231 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.560 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.742553+0000", 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.742553+0000", 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.742556+0000", 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8909999999999999e-06 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.561 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.742565+0000", 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1209999999999996e-06 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.742579+0000", 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4557e-05 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.742589+0000", 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7359999999999996e-06 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.752491+0000", 2024-04-29T14:42:52.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0099024620000000008 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.752593+0000", 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101481 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.752680+0000", 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6734e-05 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.754851+0000", 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021710039999999998 2024-04-29T14:42:52.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.766852+0000", 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012001615 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.766896+0000", 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3417999999999997e-05 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:49.767935+0000", 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010396719999999999 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143272 7.7 7:e84743d7:::benchmark_data_smithi194_147200_object28360:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:50.241795+0000", 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.9793984409999998, 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019048414999999999, 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143272 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.241795+0000", 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.241795+0000", 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.241797+0000", 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1519999999999999e-06 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.241804+0000", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2069999999999998e-06 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.241814+0000", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407e-05 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.241821+0000", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5300000000000002e-06 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.247474+0000", 2024-04-29T14:42:52.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005652804 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.247662+0000", 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018830299999999999 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.247814+0000", 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015219100000000001 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.250252+0000", 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024372230000000001 2024-04-29T14:42:52.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.258880+0000", 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086283169999999999 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.258935+0000", 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5451000000000002e-05 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.260843+0000", 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019078299999999999 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.569 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143295 7.7 7:ec578a99:::benchmark_data_smithi194_147200_object28383:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:50.356801+0000", 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.8643925079999999, 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.024636559999999998, 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143295 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.356801+0000", 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.356801+0000", 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.356803+0000", 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4030000000000001e-06 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.356811+0000", 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8660000000000006e-06 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.356825+0000", 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4508000000000001e-05 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.356833+0000", 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9680000000000006e-06 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.364756+0000", 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079228389999999992 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.364857+0000", 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000100826 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.364940+0000", 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2924999999999996e-05 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.367203+0000", 2024-04-29T14:42:52.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002262998 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.379633+0000", 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012430363999999999 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.379680+0000", 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6196999999999998e-05 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.381437+0000", 2024-04-29T14:42:52.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001757666 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143318 7.7 7:e82afb50:::benchmark_data_smithi194_147200_object28406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:50.721436+0000", 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.499756804, 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020705832, 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143318 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.721436+0000", 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.721436+0000", 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.721440+0000", 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6500000000000002e-06 2024-04-29T14:42:52.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.721449+0000", 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2269999999999999e-06 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.721463+0000", 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3461e-05 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.721472+0000", 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3859999999999996e-06 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.726220+0000", 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004747472 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.726346+0000", 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012655300000000001 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.726465+0000", 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118972 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.728728+0000", 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00226304 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.741097+0000", 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012368997 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.741142+0000", 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4694999999999998e-05 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.742142+0000", 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001000379 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143341 7.7 7:ea9c491e:::benchmark_data_smithi194_147200_object28429:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:50.846227+0000", 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.3749665179999999, 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.024658124999999999, 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.580 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143341 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.846227+0000", 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.846227+0000", 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.846229+0000", 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6220000000000001e-06 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.846237+0000", 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9829999999999996e-06 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.846250+0000", 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2663999999999999e-05 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.846257+0000", 2024-04-29T14:42:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2049999999999996e-06 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.853519+0000", 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072616169999999997 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.853614+0000", 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5304999999999996e-05 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.853684+0000", 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0112999999999998e-05 2024-04-29T14:42:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.856170+0000", 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024858950000000001 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.869476+0000", 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013305681 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.869522+0000", 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5911999999999997e-05 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:50.870885+0000", 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0013631279999999999 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143393 7.7 7:ebf8afa1:::benchmark_data_smithi194_147200_object28481:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:51.867308+0000", 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.35388467400000001, 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019471522000000002, 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.585 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143393 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.867308+0000", 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.867308+0000", 2024-04-29T14:42:52.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.867312+0000", 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4029999999999999e-06 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.867321+0000", 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1330000000000005e-06 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.867336+0000", 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5074e-05 2024-04-29T14:42:52.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.867345+0000", 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5949999999999999e-06 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.870517+0000", 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031721050000000002 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.870605+0000", 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8319000000000006e-05 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.870684+0000", 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9184000000000006e-05 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.873035+0000", 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023504899999999998 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.885709+0000", 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012674148999999999 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.885762+0000", 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2939000000000003e-05 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.886780+0000", 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001018131 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143394 7.7 7:e1461f13:::benchmark_data_smithi194_147200_object28482:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:51.886000+0000", 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.33519314300000003, 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.024981731, 2024-04-29T14:42:52.590 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143394 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.886000+0000", 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.886000+0000", 2024-04-29T14:42:52.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.886003+0000", 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7470000000000001e-06 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.886010+0000", 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3429999999999998e-06 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.886022+0000", 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1979e-05 2024-04-29T14:42:52.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.886031+0000", 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2900000000000008e-06 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.895551+0000", 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095195030000000003 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.895697+0000", 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000146157 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.895845+0000", 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000147477 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.899522+0000", 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036779299999999998 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.909196+0000", 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096740400000000001 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.594 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.909245+0000", 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8358e-05 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.910982+0000", 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001736907 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143401 7.7 7:e4c64e5e:::benchmark_data_smithi194_147200_object28489:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:51.951647+0000", 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.26954602100000002, 2024-04-29T14:42:52.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021114911, 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143401 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.951647+0000", 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.951647+0000", 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.951650+0000", 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2440000000000002e-06 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.951660+0000", 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1619999999999997e-06 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.951675+0000", 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5265999999999999e-05 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.951686+0000", 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0967e-05 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.956359+0000", 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046736620000000003 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.956457+0000", 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7782999999999995e-05 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.956540+0000", 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3014000000000002e-05 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.958902+0000", 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023617070000000002 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.971760+0000", 2024-04-29T14:42:52.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012858313999999999 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.971805+0000", 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4294e-05 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:51.972762+0000", 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00095749799999999999 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:143415 7.7 7:e352d263:::benchmark_data_smithi194_147200_object28503:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-29T14:42:52.600 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:42:52.060113+0000", 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.161079998, 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021220523000000002, 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 143415 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.060113+0000", 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.060113+0000", 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.060117+0000", 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2100000000000003e-06 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.060131+0000", 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3604999999999999e-05 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.060151+0000", 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9854999999999999e-05 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.060164+0000", 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3079e-05 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.063440+0000", 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032759489999999998 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:42:52.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.063560+0000", 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000119684 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.063670+0000", 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011009 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.066092+0000", 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002422113 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.079262+0000", 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013170619999999999 2024-04-29T14:42:52.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.079332+0000", 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0000999999999996e-05 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:42:52.081334+0000", 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020013169999999999 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:42:52.605 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:42:52.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2376089503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:42:52.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2376089503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-29T14:42:52.885 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:52.907 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:42:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:52.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:42:53.013 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:42:53.043 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:53.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:53 smithi040 ceph-mon[39581]: pgmap v2069: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 56 op/s; 28740/173270 objects misplaced (16.587%); 11 MiB/s, 175 objects/s recovering 2024-04-29T14:42:53.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:53 smithi040 ceph-mon[39581]: osdmap e807: 12 total, 12 up, 10 in 2024-04-29T14:42:53.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4096859695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:53.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:53 smithi040 ceph-mon[39581]: from='client.27421 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:53.489 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:42:53.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:53 smithi121 ceph-mon[44533]: pgmap v2069: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 56 op/s; 28740/173270 objects misplaced (16.587%); 11 MiB/s, 175 objects/s recovering 2024-04-29T14:42:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:53 smithi121 ceph-mon[44533]: osdmap e807: 12 total, 12 up, 10 in 2024-04-29T14:42:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4096859695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:53 smithi121 ceph-mon[44533]: from='client.27421 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:53.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:53 smithi194 ceph-mon[44089]: pgmap v2069: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 56 op/s; 28740/173270 objects misplaced (16.587%); 11 MiB/s, 175 objects/s recovering 2024-04-29T14:42:53.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:53 smithi194 ceph-mon[44089]: osdmap e807: 12 total, 12 up, 10 in 2024-04-29T14:42:53.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4096859695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:53.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:53 smithi194 ceph-mon[44089]: from='client.27421 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:42:54.108 INFO:tasks.thrashosds.thrasher:j is {'epoch': 806, 'fsid': '998f2e22-0630-11ef-bc93-c7b262605968', 'created': '2024-04-29T14:00:01.568926+0000', 'modified': '2024-04-29T14:42:48.468363+0000', 'last_up_change': '2024-04-29T14:31:03.499853+0000', 'last_in_change': '2024-04-29T14:42:09.481114+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-29T14:03:05.389591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '796', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-04-29T14:34:47.804723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '659', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.8999999761581421, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '967eec9c-202c-4f98-a22a-a67db85bd462', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 765, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 4255137508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6802', 'nonce': 4255137508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6804', 'nonce': 4255137508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 4255137508}]}, 'public_addr': '172.21.15.40:6801/4255137508', 'cluster_addr': '172.21.15.40:6802/4255137508', 'heartbeat_back_addr': '172.21.15.40:6804/4255137508', 'heartbeat_front_addr': '172.21.15.40:6803/4255137508', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7f52729c-855b-465c-b134-8326f99f1422', 'up': 1, 'in': 1, 'weight': 0.7887725830078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 802, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 4178456629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6806', 'nonce': 4178456629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6808', 'nonce': 4178456629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 4178456629}]}, 'public_addr': '172.21.15.40:6805/4178456629', 'cluster_addr': '172.21.15.40:6806/4178456629', 'heartbeat_back_addr': '172.21.15.40:6808/4178456629', 'heartbeat_front_addr': '172.21.15.40:6807/4178456629', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '372baf31-26e1-4865-a272-898e7ff0c2f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 230, 'up_from': 260, 'up_thru': 802, 'down_at': 231, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 3416690575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6810', 'nonce': 3416690575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6812', 'nonce': 3416690575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 3416690575}]}, 'public_addr': '172.21.15.40:6809/3416690575', 'cluster_addr': '172.21.15.40:6810/3416690575', 'heartbeat_back_addr': '172.21.15.40:6812/3416690575', 'heartbeat_front_addr': '172.21.15.40:6811/3416690575', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '975c2fbc-115b-4950-88ef-b0b2ad778765', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 748, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6813', 'nonce': 4012292460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6814', 'nonce': 4012292460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 4012292460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 4012292460}]}, 'public_addr': '172.21.15.40:6813/4012292460', 'cluster_addr': '172.21.15.40:6814/4012292460', 'heartbeat_back_addr': '172.21.15.40:6816/4012292460', 'heartbeat_front_addr': '172.21.15.40:6815/4012292460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6aa13ecc-0959-4915-a788-03c4739da269', 'up': 1, 'in': 1, 'weight': 0.5530548095703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 3987960551}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3987960551}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3987960551}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 3987960551}]}, 'public_addr': '172.21.15.121:6800/3987960551', 'cluster_addr': '172.21.15.121:6801/3987960551', 'heartbeat_back_addr': '172.21.15.121:6803/3987960551', 'heartbeat_front_addr': '172.21.15.121:6802/3987960551', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 388, 'last_clean_end': 457, 'up_from': 561, 'up_thru': 703, 'down_at': 459, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 2522198013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2522198013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2522198013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2522198013}]}, 'public_addr': '172.21.15.121:6804/2522198013', 'cluster_addr': '172.21.15.121:6805/2522198013', 'heartbeat_back_addr': '172.21.15.121:6807/2522198013', 'heartbeat_front_addr': '172.21.15.121:6806/2522198013', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '40ecb14f-7b61-45d2-b48c-0a374912edcf', 'up': 1, 'in': 1, 'weight': 0.4193267822265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 763, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 1755659098}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 1755659098}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 1755659098}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 1755659098}]}, 'public_addr': '172.21.15.121:6808/1755659098', 'cluster_addr': '172.21.15.121:6809/1755659098', 'heartbeat_back_addr': '172.21.15.121:6811/1755659098', 'heartbeat_front_addr': '172.21.15.121:6810/1755659098', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '38e15653-4a3d-4543-8240-e17d95bdfe27', 'up': 1, 'in': 1, 'weight': 0.4615020751953125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 634, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 82319362}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 82319362}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 82319362}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 82319362}]}, 'public_addr': '172.21.15.121:6812/82319362', 'cluster_addr': '172.21.15.121:6813/82319362', 'heartbeat_back_addr': '172.21.15.121:6815/82319362', 'heartbeat_front_addr': '172.21.15.121:6814/82319362', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'd93f2b45-6c82-4ead-9a66-e86f9e83e0de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 64, 'last_clean_end': 320, 'up_from': 354, 'up_thru': 802, 'down_at': 321, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6800', 'nonce': 165633203}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 165633203}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 165633203}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6802', 'nonce': 165633203}]}, 'public_addr': '172.21.15.194:6800/165633203', 'cluster_addr': '172.21.15.194:6801/165633203', 'heartbeat_back_addr': '172.21.15.194:6803/165633203', 'heartbeat_front_addr': '172.21.15.194:6802/165633203', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'a720d016-7cf7-4755-a25b-4d081c1f13a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 72, 'up_thru': 788, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6804', 'nonce': 453882699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 453882699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 453882699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6806', 'nonce': 453882699}]}, 'public_addr': '172.21.15.194:6804/453882699', 'cluster_addr': '172.21.15.194:6805/453882699', 'heartbeat_back_addr': '172.21.15.194:6807/453882699', 'heartbeat_front_addr': '172.21.15.194:6806/453882699', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '87c864f5-90a1-48e9-bca8-65cf404bc2a0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 766, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6808', 'nonce': 1320459116}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 1320459116}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 1320459116}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6810', 'nonce': 1320459116}]}, 'public_addr': '172.21.15.194:6808/1320459116', 'cluster_addr': '172.21.15.194:6809/1320459116', 'heartbeat_back_addr': '172.21.15.194:6811/1320459116', 'heartbeat_front_addr': '172.21.15.194:6810/1320459116', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '5c85066b-5780-4393-be30-0033b9d2dd1f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 88, 'up_thru': 524, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6812', 'nonce': 20977082}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 20977082}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 20977082}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.194:6814', 'nonce': 20977082}]}, 'public_addr': '172.21.15.194:6812/20977082', 'cluster_addr': '172.21.15.194:6813/20977082', 'heartbeat_back_addr': '172.21.15.194:6815/20977082', 'heartbeat_front_addr': '172.21.15.194:6814/20977082', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:42:21.214364+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:42:20.766885+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:41:37.424509+0000', 'dead_epoch': 230}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-29T14:42:48.174082+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:41:27.805653+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:41:52.745970+0000', 'dead_epoch': 457}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:42:14.483436+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:41:54.990137+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:42:43.661764+0000', 'dead_epoch': 320}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:42:35.300540+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-29T14:42:36.294708+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 28165, 'last_purged_snaps_scrub': '2024-04-29T14:41:51.042976+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '7.9', 'osds': [1, 8]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 9, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.40:0/3316779772': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/2003707927': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/2722668908': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:6800/2004115894': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/3681930020': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:6800/3860863692': '2024-04-30T14:01:01.302559+0000', '172.21.15.40:0/3976042573': '2024-04-30T14:00:29.409543+0000', '172.21.15.40:0/1925168828': '2024-04-30T14:01:01.302559+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-29T14:42:54.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:42:54.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:54 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:54.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:54 smithi040 ceph-mon[39581]: from='client.27421 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:42:54.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:54 smithi040 ceph-mon[39581]: osdmap e808: 12 total, 12 up, 10 in 2024-04-29T14:42:54.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:54 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:54 smithi121 ceph-mon[44533]: from='client.27421 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:42:54.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:54 smithi121 ceph-mon[44533]: osdmap e808: 12 total, 12 up, 10 in 2024-04-29T14:42:54.694 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:42:54.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:54 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:42:54.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:54 smithi194 ceph-mon[44089]: from='client.27421 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:42:54.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:54 smithi194 ceph-mon[44089]: osdmap e808: 12 total, 12 up, 10 in 2024-04-29T14:42:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:54.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:42:54.798 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:54.828 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:54.849 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:54.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:54.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:54.986 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:55.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:55.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:55.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:55.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:55.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:55.223 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:55.263 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:55.302 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:55.502 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:55.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:55.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:55 smithi040 ceph-mon[39581]: pgmap v2072: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 28417/172816 objects misplaced (16.444%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:42:55.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:42:55.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:42:55.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:55.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:55 smithi121 ceph-mon[44533]: pgmap v2072: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 28417/172816 objects misplaced (16.444%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:42:55.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:42:55.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:42:55.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:55 smithi194 ceph-mon[44089]: pgmap v2072: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 28417/172816 objects misplaced (16.444%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:42:55.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:42:55.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:42:55.640 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:55.793 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:56.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:56.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:56.147 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:56.226 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:56.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:56.332 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:42:56.367 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:56.445 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:42:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:56.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:42:56.480 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:42:56.508 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:42:56.533 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:42:56.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:56.556 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:42:56.578 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:56.584 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:42:56.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:56.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:56.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:42:56.637 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:42:56.658 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:42:56.683 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:42:56.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:56.713 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:42:56.730 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:56.736 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:42:56.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:56.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:42:56.752 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:56 smithi121 ceph-mon[44533]: osdmap e809: 12 total, 12 up, 10 in 2024-04-29T14:42:56.752 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:56 smithi121 ceph-mon[44533]: pgmap v2074: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 27778/172442 objects misplaced (16.109%); 16 MiB/s, 259 objects/s recovering 2024-04-29T14:42:56.752 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/742651445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:56.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:56.759 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:42:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:42:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:42:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:42:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:42:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:42:56.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:56 smithi040 ceph-mon[39581]: osdmap e809: 12 total, 12 up, 10 in 2024-04-29T14:42:56.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:56 smithi040 ceph-mon[39581]: pgmap v2074: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 27778/172442 objects misplaced (16.109%); 16 MiB/s, 259 objects/s recovering 2024-04-29T14:42:56.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/742651445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:56.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:56 smithi194 ceph-mon[44089]: osdmap e809: 12 total, 12 up, 10 in 2024-04-29T14:42:56.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:56 smithi194 ceph-mon[44089]: pgmap v2074: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 27778/172442 objects misplaced (16.109%); 16 MiB/s, 259 objects/s recovering 2024-04-29T14:42:56.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/742651445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:42:56.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:42:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:56.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:42:57.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:42:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:42:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:57.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:42:57.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:42:57.297 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:57.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:42:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:57.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:42:57.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:42:57.452 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:57.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:42:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:57.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:42:57.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:42:57.603 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.603 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.605 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:42:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:57.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:42:57.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:42:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:42:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:57.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:42:57.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:42:57.879 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:57.910 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:42:58.480 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:42:58.482 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:42:58.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:58 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:42:58.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:58 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:42:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:58 smithi194 ceph-mon[44089]: pgmap v2075: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 68 op/s; 27778/172442 objects misplaced (16.109%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:42:58.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:58 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:42:58.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:58 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:42:58.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:58 smithi040 ceph-mon[39581]: pgmap v2075: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 68 op/s; 27778/172442 objects misplaced (16.109%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:42:58.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:58 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:42:58.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:58 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:42:58.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:58 smithi121 ceph-mon[44533]: pgmap v2075: 17 pgs: 5 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 68 op/s; 27778/172442 objects misplaced (16.109%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:42:59.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:42:59.577 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:59.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:59.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:42:59.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:42:59.709 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:59.729 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:42:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:42:59.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:42:59.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:42:59 smithi194 ceph-mon[44089]: from='client.17727 v1:172.21.15.40:0/1124749334' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:42:59.831 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:42:59.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:42:59.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:42:59 smithi040 ceph-mon[39581]: from='client.17727 v1:172.21.15.40:0/1124749334' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:42:59.868 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:42:59.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:42:59 smithi121 ceph-mon[44533]: from='client.17727 v1:172.21.15.40:0/1124749334' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:43:00.078 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:00.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:00.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:00.213 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:00.233 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:00.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:00.336 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:00.370 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:00.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:00.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:00.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:00.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:00 smithi121 ceph-mon[44533]: pgmap v2076: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 27442/171794 objects misplaced (15.974%); 10 MiB/s, 160 objects/s recovering 2024-04-29T14:43:00.667 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2075,"stamp":"2024-04-29T14:42:58.214805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5650973240,"num_objects":86221,"num_object_clones":0,"num_object_copies":172442,"num_objects_missing_on_primary":4542,"num_objects_missing":4542,"num_objects_degraded":0,"num_objects_misplaced":27778,"num_objects_unfound":0,"num_objects_dirty":86221,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":143782,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89577,"num_bytes_recovered":5774737120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146567,"ondisk_log_size":146567,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16818572,"kb_used_data":11053636,"kb_used_omap":670,"kb_used_meta":5764129,"kb_avail":920632948,"statfs":{"total":959950356480,"available":942728138752,"internally_reserved":0,"allocated":11318923264,"data_stored":11308775230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":686869,"internal_metadata":5902468331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27131904,"num_objects":-414,"num_object_clones":0,"num_object_copies":-828,"num_objects_missing_on_primary":-28,"num_objects_missing":-28,"num_objects_degraded":0,"num_objects_misplaced":-962,"num_objects_unfound":0,"num_objects_dirty":-414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":414,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":61210624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004515"},"pg_stats":[{"pgid":"7.f","version":"808'9036","reported_seq":24015,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:54.327053+0000","last_change":"2024-04-29T14:41:44.740584+0000","last_active":"2024-04-29T14:42:54.327053+0000","last_peered":"2024-04-29T14:42:54.327053+0000","last_clean":"2024-04-29T14:42:54.327053+0000","last_became_active":"2024-04-29T14:38:40.422563+0000","last_became_peered":"2024-04-29T14:38:40.422563+0000","last_unstale":"2024-04-29T14:42:54.327053+0000","last_undegraded":"2024-04-29T14:42:54.327053+0000","last_fullsized":"2024-04-29T14:42:54.327053+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'8640","last_scrub_stamp":"2024-04-29T14:41:44.740429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:44.740429+0000","objects_scrubbed":5807,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354418688,"num_objects":5408,"num_object_clones":0,"num_object_copies":10816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9036,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7222,"num_bytes_recovered":464846848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"807'16206","reported_seq":16951,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-04-29T14:42:53.436852+0000","last_change":"2024-04-29T14:41:22.402971+0000","last_active":"2024-04-29T14:42:53.436852+0000","last_peered":"2024-04-29T14:42:53.436852+0000","last_clean":"2024-04-29T14:42:53.436852+0000","last_became_active":"2024-04-29T14:34:49.387078+0000","last_became_peered":"2024-04-29T14:34:49.387078+0000","last_unstale":"2024-04-29T14:42:53.436852+0000","last_undegraded":"2024-04-29T14:42:53.436852+0000","last_fullsized":"2024-04-29T14:42:53.436852+0000","mapping_epoch":655,"log_start":"675'6200","ondisk_log_start":"675'6200","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'15606","last_scrub_stamp":"2024-04-29T14:41:22.402886+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:41:22.402886+0000","objects_scrubbed":6003,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353894400,"num_objects":5400,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9150,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"808'8957","reported_seq":22207,"reported_epoch":808,"state":"active+recovering","last_fresh":"2024-04-29T14:42:53.904636+0000","last_change":"2024-04-29T14:42:42.691148+0000","last_active":"2024-04-29T14:42:53.904636+0000","last_peered":"2024-04-29T14:42:53.904636+0000","last_clean":"2024-04-29T14:42:40.867507+0000","last_became_active":"2024-04-29T14:42:42.638015+0000","last_became_peered":"2024-04-29T14:42:42.638015+0000","last_unstale":"2024-04-29T14:42:53.904636+0000","last_undegraded":"2024-04-29T14:42:53.904636+0000","last_fullsized":"2024-04-29T14:42:53.904636+0000","mapping_epoch":802,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"793'8773","last_scrub_stamp":"2024-04-29T14:42:23.541331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:42:23.541331+0000","objects_scrubbed":5553,"log_size":8957,"log_dups_size":0,"ondisk_log_size":8957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351797272,"num_objects":5369,"num_object_clones":0,"num_object_copies":10738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5368,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8957,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6359,"num_bytes_recovered":416743424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"1,6","objects":5368}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"807'8952","reported_seq":21063,"reported_epoch":807,"state":"active+recovering","last_fresh":"2024-04-29T14:42:53.472391+0000","last_change":"2024-04-29T14:42:42.705561+0000","last_active":"2024-04-29T14:42:53.472391+0000","last_peered":"2024-04-29T14:42:53.472391+0000","last_clean":"2024-04-29T14:42:41.142540+0000","last_became_active":"2024-04-29T14:42:42.655986+0000","last_became_peered":"2024-04-29T14:42:42.655986+0000","last_unstale":"2024-04-29T14:42:53.472391+0000","last_undegraded":"2024-04-29T14:42:53.472391+0000","last_fullsized":"2024-04-29T14:42:53.472391+0000","mapping_epoch":802,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'8385","last_scrub_stamp":"2024-04-29T14:41:16.362214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:16.362214+0000","objects_scrubbed":5963,"log_size":8952,"log_dups_size":0,"ondisk_log_size":8952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353632256,"num_objects":5396,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4936,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5785,"num_bytes_recovered":367067136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"2,6","objects":4936},{"shards":"2,10","objects":460}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"808'8908","reported_seq":14987,"reported_epoch":808,"state":"active+recovering","last_fresh":"2024-04-29T14:42:53.861918+0000","last_change":"2024-04-29T14:36:34.055204+0000","last_active":"2024-04-29T14:42:53.861918+0000","last_peered":"2024-04-29T14:42:53.861918+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:36:33.135845+0000","last_became_peered":"2024-04-29T14:36:33.135845+0000","last_unstale":"2024-04-29T14:42:53.861918+0000","last_undegraded":"2024-04-29T14:42:53.861918+0000","last_fullsized":"2024-04-29T14:42:53.861918+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352190464,"num_objects":5374,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":4542,"num_objects_missing":4542,"num_objects_degraded":0,"num_objects_misplaced":9899,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2209,"num_bytes_recovered":144769024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["10","11"],"object_location_counts":[{"shards":"1,5","objects":17},{"shards":"1,10,11","objects":815},{"shards":"10,11","objects":4542}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"808'16250","reported_seq":16999,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:54.550410+0000","last_change":"2024-04-29T14:41:44.773423+0000","last_active":"2024-04-29T14:42:54.550410+0000","last_peered":"2024-04-29T14:42:54.550410+0000","last_clean":"2024-04-29T14:42:54.550410+0000","last_became_active":"2024-04-29T14:34:49.387647+0000","last_became_peered":"2024-04-29T14:34:49.387647+0000","last_unstale":"2024-04-29T14:42:54.550410+0000","last_undegraded":"2024-04-29T14:42:54.550410+0000","last_fullsized":"2024-04-29T14:42:54.550410+0000","mapping_epoch":655,"log_start":"675'6200","ondisk_log_start":"675'6200","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'15825","last_scrub_stamp":"2024-04-29T14:41:44.773306+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:41:44.773306+0000","objects_scrubbed":5932,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:43:14.336019+0000","scrub_duration":700,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9176,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"808'9005","reported_seq":37775,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:53.509462+0000","last_change":"2024-04-29T14:40:30.568318+0000","last_active":"2024-04-29T14:42:53.509462+0000","last_peered":"2024-04-29T14:42:53.509462+0000","last_clean":"2024-04-29T14:42:53.509462+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:42:53.509462+0000","last_undegraded":"2024-04-29T14:42:53.509462+0000","last_fullsized":"2024-04-29T14:42:53.509462+0000","mapping_epoch":756,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7744","last_scrub_stamp":"2024-04-29T14:38:41.906760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.906760+0000","objects_scrubbed":6597,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349634560,"num_objects":5335,"num_object_clones":0,"num_object_copies":10670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9005,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"807'8886","reported_seq":9418,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-04-29T14:42:53.442830+0000","last_change":"2024-04-29T14:41:18.379928+0000","last_active":"2024-04-29T14:42:53.442830+0000","last_peered":"2024-04-29T14:42:53.442830+0000","last_clean":"2024-04-29T14:42:53.442830+0000","last_became_active":"2024-04-29T14:34:49.514880+0000","last_became_peered":"2024-04-29T14:34:49.514880+0000","last_unstale":"2024-04-29T14:42:53.442830+0000","last_undegraded":"2024-04-29T14:42:53.442830+0000","last_fullsized":"2024-04-29T14:42:53.442830+0000","mapping_epoch":655,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8343","last_scrub_stamp":"2024-04-29T14:41:18.379788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:18.379788+0000","objects_scrubbed":5950,"log_size":8886,"log_dups_size":0,"ondisk_log_size":8886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354156544,"num_objects":5404,"num_object_clones":0,"num_object_copies":10808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8886,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"808'9222","reported_seq":38905,"reported_epoch":808,"state":"active+recovering","last_fresh":"2024-04-29T14:42:55.269329+0000","last_change":"2024-04-29T14:42:42.720992+0000","last_active":"2024-04-29T14:42:55.269329+0000","last_peered":"2024-04-29T14:42:55.269329+0000","last_clean":"2024-04-29T14:42:40.862877+0000","last_became_active":"2024-04-29T14:42:42.680416+0000","last_became_peered":"2024-04-29T14:42:42.680416+0000","last_unstale":"2024-04-29T14:42:55.269329+0000","last_undegraded":"2024-04-29T14:42:55.269329+0000","last_fullsized":"2024-04-29T14:42:55.269329+0000","mapping_epoch":802,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":715,"parent":"0.0","parent_split_bits":0,"last_scrub":"765'8579","last_scrub_stamp":"2024-04-29T14:41:13.470148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:13.470148+0000","objects_scrubbed":6149,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360841216,"num_objects":5506,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4849,"num_objects_unfound":0,"num_objects_dirty":5506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14568,"num_bytes_recovered":946601984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":["8","6"],"object_location_counts":[{"shards":"1,8","objects":657},{"shards":"6,8","objects":4849}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"808'8958","reported_seq":15251,"reported_epoch":808,"state":"active+recovering","last_fresh":"2024-04-29T14:42:55.286999+0000","last_change":"2024-04-29T14:42:10.763062+0000","last_active":"2024-04-29T14:42:55.286999+0000","last_peered":"2024-04-29T14:42:55.286999+0000","last_clean":"2024-04-29T14:42:09.244532+0000","last_became_active":"2024-04-29T14:42:10.727580+0000","last_became_peered":"2024-04-29T14:42:10.727580+0000","last_unstale":"2024-04-29T14:42:55.286999+0000","last_undegraded":"2024-04-29T14:42:55.286999+0000","last_fullsized":"2024-04-29T14:42:55.286999+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"768'8380","last_scrub_stamp":"2024-04-29T14:41:17.390714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:17.390714+0000","objects_scrubbed":5895,"log_size":8958,"log_dups_size":0,"ondisk_log_size":8958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348389376,"num_objects":5316,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2726,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2843,"num_bytes_recovered":186318848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":["9","11"],"object_location_counts":[{"shards":"0,9","objects":2590},{"shards":"9,11","objects":2726}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"808'9066","reported_seq":22170,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:53.491113+0000","last_change":"2024-04-29T14:37:11.232594+0000","last_active":"2024-04-29T14:42:53.491113+0000","last_peered":"2024-04-29T14:42:53.491113+0000","last_clean":"2024-04-29T14:42:53.491113+0000","last_became_active":"2024-04-29T14:37:11.225189+0000","last_became_peered":"2024-04-29T14:37:11.225189+0000","last_unstale":"2024-04-29T14:42:53.491113+0000","last_undegraded":"2024-04-29T14:42:53.491113+0000","last_fullsized":"2024-04-29T14:42:53.491113+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":9066,"log_dups_size":0,"ondisk_log_size":9066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356384768,"num_objects":5438,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6389,"num_bytes_recovered":418709504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"807'12569","reported_seq":20402,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-04-29T14:42:53.058683+0000","last_change":"2024-04-29T14:41:47.347334+0000","last_active":"2024-04-29T14:42:53.058683+0000","last_peered":"2024-04-29T14:42:53.058683+0000","last_clean":"2024-04-29T14:42:53.058683+0000","last_became_active":"2024-04-29T14:41:12.845251+0000","last_became_peered":"2024-04-29T14:41:12.845251+0000","last_unstale":"2024-04-29T14:42:53.058683+0000","last_undegraded":"2024-04-29T14:42:53.058683+0000","last_fullsized":"2024-04-29T14:42:53.058683+0000","mapping_epoch":764,"log_start":"662'2500","ondisk_log_start":"662'2500","created":655,"last_epoch_clean":766,"parent":"0.0","parent_split_bits":0,"last_scrub":"779'12181","last_scrub_stamp":"2024-04-29T14:41:47.347210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:47.347210+0000","objects_scrubbed":5826,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356384768,"num_objects":5438,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3572,"num_bytes_recovered":220332032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"808'8854","reported_seq":16919,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:53.866955+0000","last_change":"2024-04-29T14:42:21.281487+0000","last_active":"2024-04-29T14:42:53.866955+0000","last_peered":"2024-04-29T14:42:53.866955+0000","last_clean":"2024-04-29T14:42:53.866955+0000","last_became_active":"2024-04-29T14:41:12.277628+0000","last_became_peered":"2024-04-29T14:41:12.277628+0000","last_unstale":"2024-04-29T14:42:53.866955+0000","last_undegraded":"2024-04-29T14:42:53.866955+0000","last_fullsized":"2024-04-29T14:42:53.866955+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"791'8696","last_scrub_stamp":"2024-04-29T14:42:21.281227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:42:21.281227+0000","objects_scrubbed":5496,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:43:33.248984+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349700096,"num_objects":5336,"num_object_clones":0,"num_object_copies":10672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3733,"num_bytes_recovered":244645888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"808'8858","reported_seq":34425,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:54.172749+0000","last_change":"2024-04-29T14:41:14.897161+0000","last_active":"2024-04-29T14:42:54.172749+0000","last_peered":"2024-04-29T14:42:54.172749+0000","last_clean":"2024-04-29T14:42:54.172749+0000","last_became_active":"2024-04-29T14:41:11.310817+0000","last_became_peered":"2024-04-29T14:41:11.310817+0000","last_unstale":"2024-04-29T14:42:54.172749+0000","last_undegraded":"2024-04-29T14:42:54.172749+0000","last_fullsized":"2024-04-29T14:42:54.172749+0000","mapping_epoch":763,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'8279","last_scrub_stamp":"2024-04-29T14:41:14.897016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:14.897016+0000","objects_scrubbed":5899,"log_size":8858,"log_dups_size":0,"ondisk_log_size":8858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348651520,"num_objects":5320,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12508,"num_bytes_recovered":807534592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"808'8826","reported_seq":23366,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:53.518413+0000","last_change":"2024-04-29T14:37:23.391838+0000","last_active":"2024-04-29T14:42:53.518413+0000","last_peered":"2024-04-29T14:42:53.518413+0000","last_clean":"2024-04-29T14:42:53.518413+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:42:53.518413+0000","last_undegraded":"2024-04-29T14:42:53.518413+0000","last_fullsized":"2024-04-29T14:42:53.518413+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":8826,"log_dups_size":0,"ondisk_log_size":8826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355074048,"num_objects":5418,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8826,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"808'8777","reported_seq":15559,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-04-29T14:42:54.177518+0000","last_change":"2024-04-29T14:41:21.882125+0000","last_active":"2024-04-29T14:42:54.177518+0000","last_peered":"2024-04-29T14:42:54.177518+0000","last_clean":"2024-04-29T14:42:54.177518+0000","last_became_active":"2024-04-29T14:41:14.292270+0000","last_became_peered":"2024-04-29T14:41:14.292270+0000","last_unstale":"2024-04-29T14:42:54.177518+0000","last_undegraded":"2024-04-29T14:42:54.177518+0000","last_fullsized":"2024-04-29T14:42:54.177518+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":655,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'8222","last_scrub_stamp":"2024-04-29T14:41:21.882016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:41:21.882016+0000","objects_scrubbed":5821,"log_size":8777,"log_dups_size":0,"ondisk_log_size":8777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344915968,"num_objects":5263,"num_object_clones":0,"num_object_copies":10526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3086,"num_bytes_recovered":198836224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1953,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-04-29T14:42:55.818657+0000","last_change":"2024-04-29T14:41:45.923167+0000","last_active":"2024-04-29T14:42:55.818657+0000","last_peered":"2024-04-29T14:42:55.818657+0000","last_clean":"2024-04-29T14:42:55.818657+0000","last_became_active":"2024-04-29T14:35:31.373888+0000","last_became_peered":"2024-04-29T14:35:31.373888+0000","last_unstale":"2024-04-29T14:42:55.818657+0000","last_undegraded":"2024-04-29T14:42:55.818657+0000","last_fullsized":"2024-04-29T14:42:55.818657+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:41:45.923057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:41:45.923057+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5650382872,"num_objects":86219,"num_object_clones":0,"num_object_copies":172438,"num_objects_missing_on_primary":4542,"num_objects_missing":4542,"num_objects_degraded":0,"num_objects_misplaced":27778,"num_objects_unfound":0,"num_objects_dirty":86219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143601,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89563,"num_bytes_recovered":5771034624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12456566784,"data_stored":12456558640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146430,"ondisk_log_size":146430,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122480,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384119,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2848412,"kb_used_data":2010640,"kb_used_omap":1,"kb_used_meta":837758,"kb_avail":90896740,"statfs":{"total":95995035648,"available":93078261760,"internally_reserved":0,"allocated":2058895360,"data_stored":2057882669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":857864640},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645754,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725340,"kb_used_data":1040272,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92019812,"statfs":{"total":95995035648,"available":94228287488,"internally_reserved":0,"allocated":1065238528,"data_stored":1064225857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":701495744},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423059,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1154460,"kb_used_data":1041424,"kb_used_omap":4,"kb_used_meta":113019,"kb_avail":92590692,"statfs":{"total":95995035648,"available":94812868608,"internally_reserved":0,"allocated":1066418176,"data_stored":1065405505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":115732160},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234435,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1190896,"kb_used_data":347548,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92554256,"statfs":{"total":95995035648,"available":94775558144,"internally_reserved":0,"allocated":355889152,"data_stored":354870275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":863566282},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528777,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917884,"kb_used_data":1056488,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":91827268,"statfs":{"total":95995035648,"available":94031122432,"internally_reserved":0,"allocated":1081843712,"data_stored":1080817211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":882047430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575322,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1524308,"kb_used_data":1123848,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92220844,"statfs":{"total":95995035648,"available":94434144256,"internally_reserved":0,"allocated":1150820352,"data_stored":1149809865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2103272,"kb_used_data":1309916,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":91641880,"statfs":{"total":95995035648,"available":93841285120,"internally_reserved":0,"allocated":1341353984,"data_stored":1340335107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497287,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322584,"kb_used_data":2083404,"kb_used_omap":4,"kb_used_meta":239163,"kb_avail":91422568,"statfs":{"total":95995035648,"available":93616709632,"internally_reserved":0,"allocated":2133405696,"data_stored":2132395729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":244903620},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986644,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1210664,"kb_used_data":349276,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92534488,"statfs":{"total":95995035648,"available":94755315712,"internally_reserved":0,"allocated":357658624,"data_stored":356639747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653200,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820752,"kb_used_data":690820,"kb_used_omap":651,"kb_used_meta":129268,"kb_avail":92924400,"statfs":{"total":95995035648,"available":95154585600,"internally_reserved":0,"allocated":707399680,"data_stored":706393265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":132371208},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1341521920,"data_stored":1341521920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1150619648,"data_stored":1150615576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2129199104,"data_stored":2129199104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324796416,"data_stored":324796416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":355663872,"data_stored":355663872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":705101824,"data_stored":705101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1079316480,"data_stored":1079312408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":353697792,"data_stored":353697792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1064435712,"data_stored":1064435712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1063256064,"data_stored":1063256064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2056847360,"data_stored":2056847360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":832110592,"data_stored":832110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:43:00.669 INFO:tasks.thrashosds.thrasher:Setting 7.a to [1, 9, 3, 10] 2024-04-29T14:43:00.669 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.a', '1', '9', '3', '10'] 2024-04-29T14:43:00.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 7.a 1 9 3 10 2024-04-29T14:43:00.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:43:00.757 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:00.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:43:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:00.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:43:00.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:00.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:00.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:00 smithi194 ceph-mon[44089]: pgmap v2076: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 27442/171794 objects misplaced (15.974%); 10 MiB/s, 160 objects/s recovering 2024-04-29T14:43:00.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:00.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:00.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:00 smithi040 ceph-mon[39581]: pgmap v2076: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 27442/171794 objects misplaced (15.974%); 10 MiB/s, 160 objects/s recovering 2024-04-29T14:43:00.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:43:00.915 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:00.939 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:43:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:00.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:43:01.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:43:01.080 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:43:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:01.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:43:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:01.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:01.439 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:01.526 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:01 smithi121 ceph-mon[44533]: osdmap e810: 12 total, 12 up, 10 in 2024-04-29T14:43:01.608 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:01.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:01 smithi040 ceph-mon[39581]: osdmap e810: 12 total, 12 up, 10 in 2024-04-29T14:43:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:01.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:01.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:43:01.740 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:01.764 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:43:01.766 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:01 smithi194 ceph-mon[44089]: osdmap e810: 12 total, 12 up, 10 in 2024-04-29T14:43:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:01.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:43:01.868 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:43:01.898 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:01.923 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:43:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:01.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:43:02.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:02.056 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:43:02.056 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:43:02.069 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:02.097 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:02.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:02.203 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:43:02.232 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:02.258 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:43:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:02.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:43:02.365 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:02.396 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:02.420 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:02.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:02.480 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:43:02.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:02.560 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:02.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1967039625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:02.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:02 smithi121 ceph-mon[44533]: pgmap v2078: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 78 op/s; 26778/171556 objects misplaced (15.609%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:43:02.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:02 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:43:02.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:02 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:43:02.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1967039625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:43:02.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:02 smithi121 ceph-mon[44533]: osdmap e811: 12 total, 12 up, 10 in 2024-04-29T14:43:02.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1967039625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:02.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:02 smithi194 ceph-mon[44089]: pgmap v2078: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 78 op/s; 26778/171556 objects misplaced (15.609%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:43:02.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:02 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:43:02.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:02 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:43:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1967039625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:43:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:02 smithi194 ceph-mon[44089]: osdmap e811: 12 total, 12 up, 10 in 2024-04-29T14:43:02.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:43:02.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1967039625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:02.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:02 smithi040 ceph-mon[39581]: pgmap v2078: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 78 op/s; 26778/171556 objects misplaced (15.609%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:43:02.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:02 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:43:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:02 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:43:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1967039625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:43:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:02 smithi040 ceph-mon[39581]: osdmap e811: 12 total, 12 up, 10 in 2024-04-29T14:43:03.090 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:43:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:03.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:03.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:03.763 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:03.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:03.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:03.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:03.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:04.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:04.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:04.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:04 smithi121 ceph-mon[44533]: pgmap v2080: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 26437/171142 objects misplaced (15.447%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:43:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:04.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:04.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:04.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:04.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:04 smithi194 ceph-mon[44089]: pgmap v2080: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 26437/171142 objects misplaced (15.447%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:43:04.824 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:43:04.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:04.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:04.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:04 smithi040 ceph-mon[39581]: pgmap v2080: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 26437/171142 objects misplaced (15.447%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:43:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:04.894 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:43:04.894 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:43:05.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:05 smithi121 ceph-mon[44533]: osdmap e812: 12 total, 12 up, 10 in 2024-04-29T14:43:05.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:43:05.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:05 smithi194 ceph-mon[44089]: osdmap e812: 12 total, 12 up, 10 in 2024-04-29T14:43:05.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:05 smithi040 ceph-mon[39581]: osdmap e812: 12 total, 12 up, 10 in 2024-04-29T14:43:05.881 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:43:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:06 smithi121 ceph-mon[44533]: pgmap v2082: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 78 op/s; 25705/170856 objects misplaced (15.045%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:43:06.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:43:06.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:06 smithi194 ceph-mon[44089]: pgmap v2082: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 78 op/s; 25705/170856 objects misplaced (15.045%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:43:06.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:06 smithi040 ceph-mon[39581]: pgmap v2082: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 78 op/s; 25705/170856 objects misplaced (15.045%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:43:07.641 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:43:07.641 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:43:08.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:43:08.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:08 smithi121 ceph-mon[44533]: osdmap e813: 12 total, 12 up, 10 in 2024-04-29T14:43:08.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:43:08.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:43:08.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:43:08.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:08 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:43:08.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:08 smithi194 ceph-mon[44089]: osdmap e813: 12 total, 12 up, 10 in 2024-04-29T14:43:08.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:43:08.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:43:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:43:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:08 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:43:08.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:08 smithi040 ceph-mon[39581]: osdmap e813: 12 total, 12 up, 10 in 2024-04-29T14:43:08.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:43:08.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:43:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:43:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:08 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:43:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:43:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:09.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:43:09.706 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:09.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:09.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:09 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:09.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:09 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:09.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:09 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:09.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:09 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:09.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:09 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:09.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:09 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:09.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:09 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:09.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:09 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:10.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:09 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:10.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:09 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:10.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:09 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:10.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:09 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:10 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:10 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:43:11.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:43:11.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:10 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:10 smithi194 ceph-mon[44089]: pgmap v2085: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 25335/170108 objects misplaced (14.893%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:43:11.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:10 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:10 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:43:11.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:43:11.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:10 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:10 smithi040 ceph-mon[39581]: pgmap v2085: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 25335/170108 objects misplaced (14.893%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:43:11.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:10 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:10 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:43:11.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:43:11.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:10 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:43:11.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:10 smithi121 ceph-mon[44533]: pgmap v2085: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 25335/170108 objects misplaced (14.893%); 11 MiB/s, 179 objects/s recovering 2024-04-29T14:43:11.695 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:11.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:11.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:11.837 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:12.989 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:12.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:13.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:43:13.140 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:13.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: pgmap v2086: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 24564/169826 objects misplaced (14.464%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:43:13.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/194890119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:13.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.275 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.487 INFO:teuthology.orchestra.run.smithi040.stderr:set 7.a pg_upmap_items mapping to [1->9,3->10] 2024-04-29T14:43:13.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: pgmap v2086: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 24564/169826 objects misplaced (14.464%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:43:13.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/194890119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:13.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2223236651' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]: dispatch 2024-04-29T14:43:13.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:13 smithi194 ceph-mon[44089]: from='client.27451 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]: dispatch 2024-04-29T14:43:13.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2223236651' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]: dispatch 2024-04-29T14:43:13.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:13 smithi040 ceph-mon[39581]: from='client.27451 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]: dispatch 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: pgmap v2086: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 24564/169826 objects misplaced (14.464%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/194890119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:13.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2223236651' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]: dispatch 2024-04-29T14:43:13.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:13 smithi121 ceph-mon[44533]: from='client.27451 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]: dispatch 2024-04-29T14:43:13.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:43:13.955 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:43:13.956 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:43:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:43:13.958 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:43:13.958 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:43:14.436 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:14.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:14 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:14.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:14 smithi194 ceph-mon[44089]: from='client.27451 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]': finished 2024-04-29T14:43:14.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:14 smithi194 ceph-mon[44089]: osdmap e814: 12 total, 12 up, 10 in 2024-04-29T14:43:14.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:14 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:14.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:14 smithi040 ceph-mon[39581]: from='client.27451 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]': finished 2024-04-29T14:43:14.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:14 smithi040 ceph-mon[39581]: osdmap e814: 12 total, 12 up, 10 in 2024-04-29T14:43:14.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:14 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:14.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:14 smithi121 ceph-mon[44533]: from='client.27451 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.a", "id": [1, 9, 3, 10]}]': finished 2024-04-29T14:43:14.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:14 smithi121 ceph-mon[44533]: osdmap e814: 12 total, 12 up, 10 in 2024-04-29T14:43:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:43:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:14.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:43:14.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:14.943 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:14.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:14.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:15.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:15.099 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:15.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:15.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:15.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:15.252 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:15.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:15.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:15 smithi040 ceph-mon[39581]: pgmap v2088: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 83 op/s; 24289/169526 objects misplaced (14.328%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:43:15.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:15.402 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:15.420 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:15.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:15.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:15.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:15 smithi194 ceph-mon[44089]: pgmap v2088: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 83 op/s; 24289/169526 objects misplaced (14.328%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:43:15.580 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:15.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:15 smithi121 ceph-mon[44533]: pgmap v2088: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 83 op/s; 24289/169526 objects misplaced (14.328%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:43:16.498 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:43:16.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:16.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:16.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:16 smithi194 ceph-mon[44089]: osdmap e815: 12 total, 12 up, 10 in 2024-04-29T14:43:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/234068878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:43:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:16 smithi194 ceph-mon[44089]: pgmap v2090: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 119 op/s; 23375/168942 objects misplaced (13.836%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:43:16.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:16.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:16.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:16 smithi040 ceph-mon[39581]: osdmap e815: 12 total, 12 up, 10 in 2024-04-29T14:43:16.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/234068878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:43:16.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:16 smithi040 ceph-mon[39581]: pgmap v2090: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 119 op/s; 23375/168942 objects misplaced (13.836%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:43:16.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:16.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:16.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:16 smithi121 ceph-mon[44533]: osdmap e815: 12 total, 12 up, 10 in 2024-04-29T14:43:16.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/234068878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:43:16.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:16 smithi121 ceph-mon[44533]: pgmap v2090: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 119 op/s; 23375/168942 objects misplaced (13.836%); 18 MiB/s, 284 objects/s recovering 2024-04-29T14:43:17.494 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:17 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:43:17.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:17 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:43:17.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/234068878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:43:17.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:17 smithi194 ceph-mon[44089]: osdmap e816: 12 total, 12 up, 10 in 2024-04-29T14:43:17.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:17 smithi194 ceph-mon[44089]: osdmap e817: 12 total, 12 up, 10 in 2024-04-29T14:43:17.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/234068878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:43:17.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:17 smithi040 ceph-mon[39581]: osdmap e816: 12 total, 12 up, 10 in 2024-04-29T14:43:17.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:17 smithi040 ceph-mon[39581]: osdmap e817: 12 total, 12 up, 10 in 2024-04-29T14:43:17.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:17 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:43:17.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/234068878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:43:17.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:17 smithi121 ceph-mon[44533]: osdmap e816: 12 total, 12 up, 10 in 2024-04-29T14:43:17.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:17 smithi121 ceph-mon[44533]: osdmap e817: 12 total, 12 up, 10 in 2024-04-29T14:43:18.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:43:18.177 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:18.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:18.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:43:18.316 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:18.359 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:43:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:18.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:43:18.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:43:18.500 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:18.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:43:18.523 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.5 817/788 [PushOp(7:a8cc89f3:::benchmark_data_smithi194_147200_object104192:head, version: 679'6478, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a8cc89f3:::benchmark_data_smithi194_147200_object104192:head@679'6478, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:43:18.521044+0000", 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.000254431, 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031160600000000002, 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:43:18.524 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521044+0000", 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521044+0000", 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:18.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521048+0000", 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3289999999999998e-06 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521104+0000", 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6904000000000003e-05 2024-04-29T14:43:18.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521138+0000", 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3059000000000003e-05 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521160+0000", 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2679000000000001e-05 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:18.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521263+0000", 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000103167 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:43:18.521278+0000", 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.456e-05 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:18.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:18.529 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:43:18.529 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-04-29T14:43:18.529 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:43:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:18.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:43:18.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:18.663 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:18.683 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:18.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:18.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:43:18.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:18.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:18.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:18 smithi040 ceph-mon[39581]: pgmap v2093: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 23375/168942 objects misplaced (13.836%); 15 MiB/s, 245 objects/s recovering 2024-04-29T14:43:18.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:18.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:18.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:18 smithi194 ceph-mon[44089]: pgmap v2093: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 23375/168942 objects misplaced (13.836%); 15 MiB/s, 245 objects/s recovering 2024-04-29T14:43:18.823 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:18 smithi121 ceph-mon[44533]: pgmap v2093: 17 pgs: 5 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 93 op/s; 23375/168942 objects misplaced (13.836%); 15 MiB/s, 245 objects/s recovering 2024-04-29T14:43:19.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:43:19.267 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:19.278 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:43:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:19.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:43:19.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:43:19.418 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:19.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:43:19.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:19.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:43:19.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:19.605 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:19.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:19.709 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:43:19.738 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:19.762 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:43:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:19.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:43:19.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:19.898 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:19.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:19.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:19.961 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:43:20.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:20.052 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:20.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:20.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:20.212 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:20.272 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:20.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:20.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:43:20.406 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:20.428 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:43:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:20.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:43:20.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:20.567 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:20.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:20.750 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:20.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:20.857 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:20.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:20.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:20.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:21.011 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:21.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:21.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:21.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:21.175 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:21.288 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:21 smithi040 ceph-mon[39581]: pgmap v2094: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 23005/168360 objects misplaced (13.664%); 13 MiB/s, 204 objects/s recovering 2024-04-29T14:43:21.481 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] out_osds: [3, 11] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:43:21.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:43:21.482 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-29T14:43:21.482 INFO:tasks.thrashosds.thrasher:Setting osd 8 primary_affinity to 1.000000 2024-04-29T14:43:21.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 8 1 2024-04-29T14:43:21.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:21 smithi194 ceph-mon[44089]: pgmap v2094: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 23005/168360 objects misplaced (13.664%); 13 MiB/s, 204 objects/s recovering 2024-04-29T14:43:21.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:21 smithi121 ceph-mon[44533]: pgmap v2094: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 23005/168360 objects misplaced (13.664%); 13 MiB/s, 204 objects/s recovering 2024-04-29T14:43:22.131 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:22.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:22.153 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:43:22.180 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:43:22.209 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:43:22.237 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:43:22.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:43:22.269 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:43:22.270 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:22.295 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:43:22.295 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:43:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:22.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:43:22.324 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:43:22.354 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:43:22.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:43:22.399 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:43:22.416 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:43:22.428 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:22.440 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:43:22.450 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:43:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:22.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:43:22.466 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:43:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:43:22.498 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/124002653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:22.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:43:22.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/124002653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:22.592 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:22.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/124002653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:22.900 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:43:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:22.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:43:22.910 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:23.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:23.046 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:23.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: pgmap v2095: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 59 op/s; 22072/168142 objects misplaced (13.127%); 12 MiB/s, 189 objects/s recovering 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: pgmap v2095: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 59 op/s; 22072/168142 objects misplaced (13.127%); 12 MiB/s, 189 objects/s recovering 2024-04-29T14:43:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: pgmap v2095: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 59 op/s; 22072/168142 objects misplaced (13.127%); 12 MiB/s, 189 objects/s recovering 2024-04-29T14:43:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:24.290 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:24.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:24.619 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:43:24.619 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:43:24.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:25.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:43:25.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:43:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:25.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:25.521 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:25 smithi040 ceph-mon[39581]: pgmap v2096: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 67 op/s; 21770/167854 objects misplaced (12.970%); 12 MiB/s, 197 objects/s recovering 2024-04-29T14:43:25.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:25 smithi194 ceph-mon[44089]: pgmap v2096: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 67 op/s; 21770/167854 objects misplaced (12.970%); 12 MiB/s, 197 objects/s recovering 2024-04-29T14:43:25.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:43:25.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:25 smithi121 ceph-mon[44533]: pgmap v2096: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 67 op/s; 21770/167854 objects misplaced (12.970%); 12 MiB/s, 197 objects/s recovering 2024-04-29T14:43:25.651 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:25.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:43:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:25.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:43:25.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:25.803 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:25.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:25.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:25.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:25.962 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:26.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:26.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:26.161 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:26.605 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:26.615 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:26.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:26.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:26.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:26.751 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:26.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:26.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:43:27.017 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:27.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:43:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:27.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:43:27.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:27.173 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:27.200 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:27.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:27.306 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:43:27.336 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:27.358 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:43:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:27.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:43:27.464 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:43:27.492 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:27.512 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:43:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:27.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:43:27.523 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:27.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:27 smithi194 ceph-mon[44089]: pgmap v2097: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 20855/167342 objects misplaced (12.463%); 16 MiB/s, 277 objects/s recovering 2024-04-29T14:43:27.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:43:27.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:27 smithi040 ceph-mon[39581]: pgmap v2097: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 20855/167342 objects misplaced (12.463%); 16 MiB/s, 277 objects/s recovering 2024-04-29T14:43:27.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:27 smithi121 ceph-mon[44533]: pgmap v2097: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 20855/167342 objects misplaced (12.463%); 16 MiB/s, 277 objects/s recovering 2024-04-29T14:43:27.646 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:27.665 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:43:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:27.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:43:27.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:43:27.798 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:43:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:27.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:43:27.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:43:27.949 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:27.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:43:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:27.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:43:28.071 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:43:28.098 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:28.121 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:43:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:28.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:43:28.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:43:28.274 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:43:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:28.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:43:28.843 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:43:28.871 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:28.894 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:43:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:28.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:43:28.998 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:43:29.027 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:29.050 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:43:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:29.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:43:29.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:43:29.182 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:29.202 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:43:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:29.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:43:29.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:43:29.335 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:29.356 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:43:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:29.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:43:29.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:29.502 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:29.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:29 smithi194 ceph-mon[44089]: pgmap v2098: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 20855/167342 objects misplaced (12.463%); 14 MiB/s, 251 objects/s recovering 2024-04-29T14:43:29.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:29 smithi040 ceph-mon[39581]: pgmap v2098: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 20855/167342 objects misplaced (12.463%); 14 MiB/s, 251 objects/s recovering 2024-04-29T14:43:29.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:29 smithi121 ceph-mon[44533]: pgmap v2098: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 20855/167342 objects misplaced (12.463%); 14 MiB/s, 251 objects/s recovering 2024-04-29T14:43:29.806 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:29.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:29.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:29.948 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:30.316 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:43:30.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3237406185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:30.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:30 smithi194 ceph-mon[44089]: from='client.27466 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:30.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3237406185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:30.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:30 smithi040 ceph-mon[39581]: from='client.27466 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:30.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3237406185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:30.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:30 smithi121 ceph-mon[44533]: from='client.27466 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:43:31.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:31 smithi194 ceph-mon[44089]: pgmap v2099: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 20441/166644 objects misplaced (12.266%); 15 MiB/s, 257 objects/s recovering 2024-04-29T14:43:31.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:31 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:43:31.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:31 smithi194 ceph-mon[44089]: from='client.27466 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:43:31.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:31 smithi194 ceph-mon[44089]: osdmap e818: 12 total, 12 up, 10 in 2024-04-29T14:43:31.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/846168761' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:43:31.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:31 smithi194 ceph-mon[44089]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:43:31.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:31 smithi040 ceph-mon[39581]: pgmap v2099: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 20441/166644 objects misplaced (12.266%); 15 MiB/s, 257 objects/s recovering 2024-04-29T14:43:31.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:31 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:43:31.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:31 smithi040 ceph-mon[39581]: from='client.27466 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:43:31.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:31 smithi040 ceph-mon[39581]: osdmap e818: 12 total, 12 up, 10 in 2024-04-29T14:43:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/846168761' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:43:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:31 smithi040 ceph-mon[39581]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:43:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:31 smithi121 ceph-mon[44533]: pgmap v2099: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 20441/166644 objects misplaced (12.266%); 15 MiB/s, 257 objects/s recovering 2024-04-29T14:43:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:31 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:43:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:31 smithi121 ceph-mon[44533]: from='client.27466 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:43:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:31 smithi121 ceph-mon[44533]: osdmap e818: 12 total, 12 up, 10 in 2024-04-29T14:43:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/846168761' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:43:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:31 smithi121 ceph-mon[44533]: from='client.37226 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:43:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:31.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:32.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:32.041 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:32.062 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:32.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:32.166 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:43:32.193 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:32.214 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:43:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:32.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:43:32.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:32.353 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:32.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:32 smithi040 ceph-mon[39581]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:43:32.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:32 smithi040 ceph-mon[39581]: osdmap e819: 12 total, 12 up, 10 in 2024-04-29T14:43:32.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:32 smithi194 ceph-mon[44089]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:43:32.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:32 smithi194 ceph-mon[44089]: osdmap e819: 12 total, 12 up, 10 in 2024-04-29T14:43:32.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:32 smithi121 ceph-mon[44533]: from='client.37226 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:43:32.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:32 smithi121 ceph-mon[44533]: osdmap e819: 12 total, 12 up, 10 in 2024-04-29T14:43:33.324 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:33 smithi040 ceph-mon[39581]: pgmap v2102: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 20042/166382 objects misplaced (12.046%); 13 MiB/s, 345 objects/s recovering 2024-04-29T14:43:33.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:33 smithi194 ceph-mon[44089]: pgmap v2102: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 20042/166382 objects misplaced (12.046%); 13 MiB/s, 345 objects/s recovering 2024-04-29T14:43:33.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:33 smithi121 ceph-mon[44533]: pgmap v2102: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 20042/166382 objects misplaced (12.046%); 13 MiB/s, 345 objects/s recovering 2024-04-29T14:43:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:35 smithi194 ceph-mon[44089]: pgmap v2103: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 19691/166140 objects misplaced (11.852%); 8.9 MiB/s, 248 objects/s recovering 2024-04-29T14:43:35.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:35 smithi040 ceph-mon[39581]: pgmap v2103: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 19691/166140 objects misplaced (11.852%); 8.9 MiB/s, 248 objects/s recovering 2024-04-29T14:43:35.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:35 smithi121 ceph-mon[44533]: pgmap v2103: 17 pgs: 5 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 19691/166140 objects misplaced (11.852%); 8.9 MiB/s, 248 objects/s recovering 2024-04-29T14:43:37.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:37 smithi040 ceph-mon[39581]: pgmap v2104: 17 pgs: 4 active+recovering, 13 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 19131/165512 objects misplaced (11.559%); 13 MiB/s, 375 objects/s recovering 2024-04-29T14:43:37.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:37 smithi121 ceph-mon[44533]: pgmap v2104: 17 pgs: 4 active+recovering, 13 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 19131/165512 objects misplaced (11.559%); 13 MiB/s, 375 objects/s recovering 2024-04-29T14:43:37.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:37 smithi194 ceph-mon[44089]: pgmap v2104: 17 pgs: 4 active+recovering, 13 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 19131/165512 objects misplaced (11.559%); 13 MiB/s, 375 objects/s recovering 2024-04-29T14:43:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:39 smithi040 ceph-mon[39581]: pgmap v2105: 17 pgs: 4 active+recovering, 13 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 70 op/s; 19131/165512 objects misplaced (11.559%); 10 MiB/s, 327 objects/s recovering 2024-04-29T14:43:39.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:39 smithi121 ceph-mon[44533]: pgmap v2105: 17 pgs: 4 active+recovering, 13 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 70 op/s; 19131/165512 objects misplaced (11.559%); 10 MiB/s, 327 objects/s recovering 2024-04-29T14:43:39.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:39 smithi194 ceph-mon[44089]: pgmap v2105: 17 pgs: 4 active+recovering, 13 active+clean; 5.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 70 op/s; 19131/165512 objects misplaced (11.559%); 10 MiB/s, 327 objects/s recovering 2024-04-29T14:43:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:39.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:39.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:39.969 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:39.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:39.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:40.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:40.124 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:40.144 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:40.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:40.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:43:40.277 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:40.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:43:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:40.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:43:40.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:40.426 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:40.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:40.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:40.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:43:40.578 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:40.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:43:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:40.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:43:40.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:40.739 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:41.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:41 smithi040 ceph-mon[39581]: pgmap v2106: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 18630/164546 objects misplaced (11.322%); 11 MiB/s, 310 objects/s recovering 2024-04-29T14:43:41.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:41 smithi121 ceph-mon[44533]: pgmap v2106: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 18630/164546 objects misplaced (11.322%); 11 MiB/s, 310 objects/s recovering 2024-04-29T14:43:41.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:41 smithi194 ceph-mon[44089]: pgmap v2106: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 18630/164546 objects misplaced (11.322%); 11 MiB/s, 310 objects/s recovering 2024-04-29T14:43:42.779 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-29T14:43:43.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:43 smithi040 ceph-mon[39581]: pgmap v2107: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 18124/164268 objects misplaced (11.033%); 11 MiB/s, 214 objects/s recovering 2024-04-29T14:43:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:43 smithi121 ceph-mon[44533]: pgmap v2107: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 18124/164268 objects misplaced (11.033%); 11 MiB/s, 214 objects/s recovering 2024-04-29T14:43:43.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:43 smithi194 ceph-mon[44089]: pgmap v2107: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 18124/164268 objects misplaced (11.033%); 11 MiB/s, 214 objects/s recovering 2024-04-29T14:43:45.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:45 smithi040 ceph-mon[39581]: pgmap v2108: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 17685/163924 objects misplaced (10.789%); 12 MiB/s, 231 objects/s recovering 2024-04-29T14:43:45.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:45 smithi121 ceph-mon[44533]: pgmap v2108: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 17685/163924 objects misplaced (10.789%); 12 MiB/s, 231 objects/s recovering 2024-04-29T14:43:45.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:45 smithi194 ceph-mon[44089]: pgmap v2108: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 17685/163924 objects misplaced (10.789%); 12 MiB/s, 231 objects/s recovering 2024-04-29T14:43:46.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:46.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:46 smithi040 ceph-mon[39581]: osdmap e820: 12 total, 12 up, 10 in 2024-04-29T14:43:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:43:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:43:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:46 smithi121 ceph-mon[44533]: osdmap e820: 12 total, 12 up, 10 in 2024-04-29T14:43:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:43:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:43:46.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:46.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:46.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:46 smithi194 ceph-mon[44089]: osdmap e820: 12 total, 12 up, 10 in 2024-04-29T14:43:46.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:46.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:46.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:43:46.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:43:47.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:47 smithi040 ceph-mon[39581]: pgmap v2110: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 17013/163230 objects misplaced (10.423%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:43:47.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:47 smithi040 ceph-mon[39581]: osdmap e821: 12 total, 12 up, 10 in 2024-04-29T14:43:47.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:47 smithi121 ceph-mon[44533]: pgmap v2110: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 17013/163230 objects misplaced (10.423%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:43:47.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:47 smithi121 ceph-mon[44533]: osdmap e821: 12 total, 12 up, 10 in 2024-04-29T14:43:47.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:47 smithi194 ceph-mon[44089]: pgmap v2110: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 17013/163230 objects misplaced (10.423%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:43:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:47 smithi194 ceph-mon[44089]: osdmap e821: 12 total, 12 up, 10 in 2024-04-29T14:43:49.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:49 smithi040 ceph-mon[39581]: pgmap v2112: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 17013/163230 objects misplaced (10.423%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:43:49.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:49 smithi121 ceph-mon[44533]: pgmap v2112: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 17013/163230 objects misplaced (10.423%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:43:49.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:49 smithi194 ceph-mon[44089]: pgmap v2112: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 17013/163230 objects misplaced (10.423%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:43:50.555 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:50.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:50.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:50.701 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:50.726 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:50.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:50.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:43:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:43:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:50.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:43:50.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:51.014 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:43:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:51.023 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:43:51.023 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:43:51.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:43:51.023 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:43:51.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.023 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101557 7.4 7:2877201c:::benchmark_data_smithi194_147200_object101556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e677)", 2024-04-29T14:43:51.023 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.225065+0000", 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 482.79308242000002, 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21204304199999999, 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101557 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.225065+0000", 2024-04-29T14:43:51.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.225065+0000", 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.225068+0000", 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.506e-06 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227037+0000", 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019693929999999998 2024-04-29T14:43:51.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227059+0000", 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1464999999999999e-05 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227073+0000", 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.396e-05 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227565+0000", 2024-04-29T14:43:51.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00049246999999999995 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227578+0000", 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2788e-05 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.419206+0000", 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19162751 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.421041+0000", 2024-04-29T14:43:51.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018351419999999999 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.421160+0000", 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119332 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.434204+0000", 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01304367 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.434238+0000", 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4638000000000003e-05 2024-04-29T14:43:51.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.434280+0000", 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1931999999999998e-05 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437108+0000", 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0028272359999999999 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101561 7.4 7:2ce6de55:::benchmark_data_smithi194_147200_object101560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:43:51.029 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227092+0000", 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 482.79105509200002, 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.226205358, 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101561 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227092+0000", 2024-04-29T14:43:51.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227092+0000", 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227094+0000", 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0040000000000002e-06 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227138+0000", 2024-04-29T14:43:51.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3936000000000001e-05 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227148+0000", 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0246e-05 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227156+0000", 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5349999999999999e-06 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227619+0000", 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046307199999999997 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227624+0000", 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7459999999999997e-06 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.427370+0000", 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.199745069 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.427490+0000", 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120734 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.427614+0000", 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012385799999999999 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437125+0000", 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0095105610000000007 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.449227+0000", 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012102768 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.449307+0000", 2024-04-29T14:43:51.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9357999999999995e-05 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.453297+0000", 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039904709999999998 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101578 7.4 7:215cf8ee:::benchmark_data_smithi194_147200_object101577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227165+0000", 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 482.790981993, 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.228836451, 2024-04-29T14:43:51.035 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101578 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227165+0000", 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227165+0000", 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227166+0000", 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0599999999999999e-07 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227204+0000", 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8099999999999998e-05 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227216+0000", 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2306e-05 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227224+0000", 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3610000000000003e-06 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227670+0000", 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044646300000000002 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227680+0000", 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7580000000000004e-06 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.430921+0000", 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20324070999999999 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.431061+0000", 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140128 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.431203+0000", 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000142497 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440344+0000", 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091407250000000006 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.453374+0000", 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013029981 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.453443+0000", 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9066999999999994e-05 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.456001+0000", 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025585489999999998 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101598 7.4 7:2ba08c3a:::benchmark_data_smithi194_147200_object101597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227232+0000", 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 482.79091518299998, 2024-04-29T14:43:51.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23998618499999999, 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101598 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227232+0000", 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227232+0000", 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227233+0000", 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3399999999999998e-07 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227272+0000", 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9266e-05 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227279+0000", 2024-04-29T14:43:51.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3880000000000002e-06 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227286+0000", 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1160000000000001e-06 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227714+0000", 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00042732099999999998 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227719+0000", 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7940000000000002e-06 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437248+0000", 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20952917200000001 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437389+0000", 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014079199999999999 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.437502+0000", 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000113903 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.446652+0000", 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091495410000000006 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.464557+0000", 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017904908000000001 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.464613+0000", 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6239999999999997e-05 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.467218+0000", 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026049100000000002 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:101599 7.4 7:2fafdeca:::benchmark_data_smithi194_147200_object101598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e677)", 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:48.227294+0000", 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 482.79085278100001, 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.245211925, 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.047 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 101599 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227294+0000", 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227294+0000", 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227295+0000", 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8899999999999999e-07 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227331+0000", 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6353000000000002e-05 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227338+0000", 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299999999999997e-06 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227345+0000", 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3579999999999999e-06 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227775+0000", 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00043071100000000002 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.227784+0000", 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.217e-06 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440383+0000", 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21259881999999999 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440576+0000", 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019309500000000001 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.440790+0000", 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000213375 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.446676+0000", 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058865360000000004 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.469672+0000", 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022995439999999999 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.469716+0000", 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4489e-05 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:48.472506+0000", 2024-04-29T14:43:51.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0027899119999999999 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103989 7.4 7:2fdc10c1:::benchmark_data_smithi194_147200_object103988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.085580+0000", 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 479.93256681899999, 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.089585465000000003, 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103989 2024-04-29T14:43:51.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085580+0000", 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085580+0000", 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085583+0000", 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7860000000000001e-06 2024-04-29T14:43:51.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085646+0000", 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2711000000000003e-05 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085666+0000", 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9857999999999999e-05 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.085675+0000", 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8770000000000005e-06 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.103410+0000", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017734507 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.103530+0000", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120442 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.103699+0000", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016831699999999999 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113090+0000", 2024-04-29T14:43:51.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0093912230000000006 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.172174+0000", 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.059084018000000002 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.172242+0000", 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7991999999999995e-05 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.175166+0000", 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029237339999999999 2024-04-29T14:43:51.057 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103991 7.4 7:24c10593:::benchmark_data_smithi194_147200_object103990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.086386+0000", 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 479.93176080500001, 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091505929999999999, 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103991 2024-04-29T14:43:51.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086386+0000", 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086386+0000", 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086388+0000", 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8780000000000001e-06 2024-04-29T14:43:51.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086453+0000", 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4550999999999999e-05 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086464+0000", 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1590000000000001e-05 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.086472+0000", 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6059999999999997e-06 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.109466+0000", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022994297 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.109617+0000", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000150507 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.109735+0000", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118713 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.124089+0000", 2024-04-29T14:43:51.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014353922 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.175225+0000", 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051135271000000003 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.175267+0000", 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.282e-05 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.177892+0000", 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026247750000000002 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.062 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103995 7.4 7:24fcae4a:::benchmark_data_smithi194_147200_object103994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.088764+0000", 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 479.929383137, 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091943251000000004, 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103995 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.063 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088764+0000", 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088764+0000", 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088766+0000", 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1129999999999999e-06 2024-04-29T14:43:51.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088828+0000", 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2187000000000006e-05 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088844+0000", 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5364000000000001e-05 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.088853+0000", 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3000000000000007e-06 2024-04-29T14:43:51.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113122+0000", 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024269625999999999 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113287+0000", 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000164814 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.113458+0000", 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017102 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.124098+0000", 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010639696000000001 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.177962+0000", 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053863992999999999 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.178033+0000", 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.059e-05 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.180707+0000", 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026745480000000001 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:103996 7.4 7:2d132abf:::benchmark_data_smithi194_147200_object103995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.091595+0000", 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 479.92655234599999, 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.092339152999999993, 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.068 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 103996 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091595+0000", 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091595+0000", 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091597+0000", 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3290000000000001e-06 2024-04-29T14:43:51.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091670+0000", 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2698000000000001e-05 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091682+0000", 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2441e-05 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.091691+0000", 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3820000000000004e-06 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.116708+0000", 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025017158000000001 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.116849+0000", 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140982 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.116957+0000", 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108034 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.137328+0000", 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020371520000000001 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.180800+0000", 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043471744 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.180846+0000", 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5765e-05 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.183934+0000", 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030880999999999999 2024-04-29T14:43:51.072 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:104001 7.4 7:24312fb7:::benchmark_data_smithi194_147200_object104000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:51.098323+0000", 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 479.91982403200001, 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.092250929999999995, 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 104001 2024-04-29T14:43:51.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098323+0000", 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098323+0000", 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098326+0000", 2024-04-29T14:43:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7020000000000001e-06 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098377+0000", 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1025e-05 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098392+0000", 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5248999999999999e-05 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.098400+0000", 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0800000000000006e-06 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.120267+0000", 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021867119000000001 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.120459+0000", 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.120638+0000", 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017919400000000001 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.137335+0000", 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016696414999999999 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.187832+0000", 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050497487000000001 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.187883+0000", 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1165000000000002e-05 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:51.190574+0000", 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026904939999999999 2024-04-29T14:43:51.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105442 7.4 7:2acf0821:::benchmark_data_smithi194_147200_object105441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.742407+0000", 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 478.27574039799998, 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.092712567999999995, 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105442 2024-04-29T14:43:51.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742407+0000", 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742407+0000", 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742409+0000", 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0829999999999998e-06 2024-04-29T14:43:51.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742454+0000", 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5633999999999997e-05 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742465+0000", 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0669000000000001e-05 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742473+0000", 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9489999999999996e-06 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.765614+0000", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023141199000000001 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.765795+0000", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018102299999999999 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.765958+0000", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016311900000000001 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804981+0000", 2024-04-29T14:43:51.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039023059999999998 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.832270+0000", 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027288501999999999 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.832336+0000", 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6462000000000002e-05 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.835119+0000", 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002782868 2024-04-29T14:43:51.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105444 7.4 7:26a44d0b:::benchmark_data_smithi194_147200_object105443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.742770+0000", 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 478.27537675399998, 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.095570181000000004, 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105444 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.083 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742770+0000", 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742770+0000", 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742772+0000", 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2589999999999999e-06 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742835+0000", 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3095000000000005e-05 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742843+0000", 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4810000000000006e-06 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.742850+0000", 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.596e-06 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.769038+0000", 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026188526 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.769160+0000", 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012136 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.769258+0000", 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8357999999999996e-05 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804988+0000", 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035730383999999997 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.835202+0000", 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030213256000000001 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.835256+0000", 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3967e-05 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.838340+0000", 2024-04-29T14:43:51.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030848989999999999 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105451 7.4 7:2191b480:::benchmark_data_smithi194_147200_object105450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.755824+0000", 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 478.262322611, 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.088790426000000006, 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105451 2024-04-29T14:43:51.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755824+0000", 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755824+0000", 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755828+0000", 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5470000000000001e-06 2024-04-29T14:43:51.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755880+0000", 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2132999999999997e-05 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755898+0000", 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7810000000000001e-05 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.755908+0000", 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0540000000000001e-05 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.772946+0000", 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01703795 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.773108+0000", 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016127000000000001 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.773277+0000", 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016983699999999999 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804992+0000", 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031714786000000002 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.841925+0000", 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036932765999999999 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.841971+0000", 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6010000000000002e-05 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.844615+0000", 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026437769999999999 2024-04-29T14:43:51.092 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:105455 7.4 7:252782f5:::benchmark_data_smithi194_147200_object105454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:35:52.761216+0000", 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 478.25693150400002, 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.085881415000000003, 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 105455 2024-04-29T14:43:51.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761216+0000", 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761216+0000", 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761218+0000", 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.429e-06 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761256+0000", 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7985999999999998e-05 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761268+0000", 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2187999999999999e-05 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.761277+0000", 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4980000000000006e-06 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.776722+0000", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015445468 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.776880+0000", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000158247 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.777022+0000", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014190700000000001 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.804996+0000", 2024-04-29T14:43:51.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027973591999999999 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.844665+0000", 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039669641999999998 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.844706+0000", 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0064000000000001e-05 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:35:52.847097+0000", 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023913939999999998 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.097 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130707 7.6 7:61bf31d8:::benchmark_data_smithi194_147200_object15795:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e749)", 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.118762+0000", 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 227.899384704, 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.319193809, 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130707 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.098 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118762+0000", 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118762+0000", 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118765+0000", 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5660000000000001e-06 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118773+0000", 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8320000000000006e-06 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118785+0000", 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2357e-05 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118795+0000", 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4409999999999999e-06 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118866+0000", 2024-04-29T14:43:51.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1235000000000005e-05 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118878+0000", 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2648e-05 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.437152+0000", 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.318273168 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.437956+0000", 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00080456199999999999 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.101 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130722 7.6 7:62be359e:::benchmark_data_smithi194_147200_object15810:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e749)", 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.118815+0000", 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 227.89933172299999, 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32446416700000003, 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130722 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.102 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118815+0000", 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118815+0000", 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118816+0000", 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.09e-07 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118820+0000", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6100000000000002e-06 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118827+0000", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9380000000000003e-06 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118833+0000", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1879999999999999e-06 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118908+0000", 2024-04-29T14:43:51.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4907000000000006e-05 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118915+0000", 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5939999999999996e-06 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.442593+0000", 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32367770099999998 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.443279+0000", 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00068641999999999998 2024-04-29T14:43:51.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130723 7.6 7:6066e8ed:::benchmark_data_smithi194_147200_object15811:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e749)", 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.118841+0000", 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 227.8993064, 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32510310599999998, 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130723 2024-04-29T14:43:51.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118841+0000", 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118841+0000", 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118841+0000", 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9500000000000002e-07 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118844+0000", 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7800000000000001e-06 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118850+0000", 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6250000000000004e-06 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118855+0000", 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8910000000000003e-06 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118939+0000", 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4571000000000001e-05 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118946+0000", 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1489999999999997e-06 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.443311+0000", 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.324365087 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.443944+0000", 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063240799999999999 2024-04-29T14:43:51.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130750 7.6 7:65bbffe6:::benchmark_data_smithi194_147200_object15838:head [delete] snapc 0=[] ondisk+write+known_if_redirected e749)", 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.118861+0000", 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 227.89928612700001, 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.325733577, 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130750 2024-04-29T14:43:51.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118861+0000", 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118861+0000", 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118861+0000", 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8400000000000004e-07 2024-04-29T14:43:51.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118870+0000", 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4479999999999994e-06 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118876+0000", 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1469999999999998e-06 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.118882+0000", 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.733e-06 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.119855+0000", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00097331999999999996 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.119866+0000", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0414999999999999e-05 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.443979+0000", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32411312599999997 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.444594+0000", 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061580399999999996 2024-04-29T14:43:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130750 7.6 7:65bbffe6:::benchmark_data_smithi194_147200_object15838:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e750)", 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.499837+0000", 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 227.51831039499999, 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.086192671999999998, 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.114 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130750 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.499837+0000", 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.499837+0000", 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.499839+0000", 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.858e-06 2024-04-29T14:43:51.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.499848+0000", 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3720000000000005e-06 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.499859+0000", 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1070000000000001e-05 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.499869+0000", 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6690000000000001e-06 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.528063+0000", 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028193967 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.528168+0000", 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000105367 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.528287+0000", 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011924100000000001 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.535608+0000", 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0073204489999999997 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.581601+0000", 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045993182000000001 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.581661+0000", 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0485000000000001e-05 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.586029+0000", 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043680120000000001 2024-04-29T14:43:51.118 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130966 7.6 7:6db0c3f3:::benchmark_data_smithi194_147200_object16054:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e751)", 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.134453+0000", 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 226.88369358, 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.086892543000000003, 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130966 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.119 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.134453+0000", 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.134453+0000", 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.134455+0000", 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9250000000000002e-06 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.134461+0000", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9499999999999998e-06 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.134472+0000", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0893999999999999e-05 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.134481+0000", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3250000000000008e-06 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.168110+0000", 2024-04-29T14:43:51.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033629210999999999 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.168219+0000", 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109598 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3", 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.168474+0000", 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025501499999999998 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.180643+0000", 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012168294 2024-04-29T14:43:51.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.216631+0000", 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035988036000000001 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.216690+0000", 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9043999999999999e-05 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.221346+0000", 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046562510000000001 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.123 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.124 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:43:51.124 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:43:51.124 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:43:51.124 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:43:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:51.125 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:51.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:51.125 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:43:51.126 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:43:51.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:43:51.149 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:43:51.171 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:43:51.209 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:43:51.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:51.244 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:43:51.269 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:51.283 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:43:51.307 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:43:51.328 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:51.331 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:43:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:51.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:43:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:43:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:43:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:43:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:43:51.434 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:43:51.462 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:51.483 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:43:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:51.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:43:51.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:43:51.612 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:51 smithi194 ceph-mon[44089]: pgmap v2113: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 16400/162574 objects misplaced (10.088%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:43:51.612 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2268408851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:51.612 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:51.615 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:51.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:51 smithi040 ceph-mon[39581]: pgmap v2113: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 16400/162574 objects misplaced (10.088%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:43:51.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2268408851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:51.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:51.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:51 smithi121 ceph-mon[44533]: pgmap v2113: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 16400/162574 objects misplaced (10.088%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:43:51.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2268408851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:43:51.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:51.634 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:43:51.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:51.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:43:51.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:43:51.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:43:51.764 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:51.783 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:43:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:51.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:43:51.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:43:51.914 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:51.935 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:43:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:51.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:43:52.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:43:52.066 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:52.086 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:43:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:52.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:43:52.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:43:52.217 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:52.236 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:43:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:52.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:43:52.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:43:52.373 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:52.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:43:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:52.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:43:52.498 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:43:52.530 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:52.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:43:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:53 smithi040 ceph-mon[39581]: pgmap v2114: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 15730/162292 objects misplaced (9.692%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:43:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:53 smithi121 ceph-mon[44533]: pgmap v2114: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 15730/162292 objects misplaced (9.692%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:43:53.773 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] out_osds: [3, 11] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:43:53.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:43:53.787 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.7140804141184572 2024-04-29T14:43:53.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 4 0.7140804141184572 2024-04-29T14:43:53.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:53 smithi194 ceph-mon[44089]: pgmap v2114: 17 pgs: 4 active+recovering, 13 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 15730/162292 objects misplaced (9.692%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:43:54.365 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:43:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:54.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:43:54.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:43:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:55.016 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:43:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:55.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:43:55.120 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:43:55.149 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:55.169 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:43:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:55.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:43:55.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:43:55.236 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:55.245 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:55.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:55.569 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:43:55.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:55 smithi040 ceph-mon[39581]: pgmap v2115: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 15156/161940 objects misplaced (9.359%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:43:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:55.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:43:55.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:55 smithi194 ceph-mon[44089]: pgmap v2115: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 15156/161940 objects misplaced (9.359%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:43:55.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:55 smithi121 ceph-mon[44533]: pgmap v2115: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 15156/161940 objects misplaced (9.359%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:43:55.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:43:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:55.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:43:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:55.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:43:55.832 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:43:55.862 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:43:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:43:55.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:43:55.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:43:56.022 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:43:56.833 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:43:57.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:43:57.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:57 smithi040 ceph-mon[39581]: pgmap v2116: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 14880/161250 objects misplaced (9.228%); 13 MiB/s, 275 objects/s recovering 2024-04-29T14:43:57.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:57 smithi121 ceph-mon[44533]: pgmap v2116: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 14880/161250 objects misplaced (9.228%); 13 MiB/s, 275 objects/s recovering 2024-04-29T14:43:57.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:57 smithi194 ceph-mon[44089]: pgmap v2116: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 14880/161250 objects misplaced (9.228%); 13 MiB/s, 275 objects/s recovering 2024-04-29T14:43:59.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:43:59 smithi040 ceph-mon[39581]: pgmap v2117: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 83 op/s; 14880/161250 objects misplaced (9.228%); 11 MiB/s, 231 objects/s recovering 2024-04-29T14:43:59.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:43:59 smithi121 ceph-mon[44533]: pgmap v2117: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 83 op/s; 14880/161250 objects misplaced (9.228%); 11 MiB/s, 231 objects/s recovering 2024-04-29T14:43:59.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:43:59 smithi194 ceph-mon[44089]: pgmap v2117: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 83 op/s; 14880/161250 objects misplaced (9.228%); 11 MiB/s, 231 objects/s recovering 2024-04-29T14:44:00.772 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:44:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:00.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:44:00.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:00.911 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:00.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:00.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:01.036 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:01.063 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:01.085 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:01.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:01.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:01.216 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:01.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:01.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:01.416 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:01.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:01.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:01 smithi040 ceph-mon[39581]: pgmap v2118: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 117 op/s; 14129/160410 objects misplaced (8.808%); 15 MiB/s, 295 objects/s recovering 2024-04-29T14:44:01.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:01 smithi121 ceph-mon[44533]: pgmap v2118: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 117 op/s; 14129/160410 objects misplaced (8.808%); 15 MiB/s, 295 objects/s recovering 2024-04-29T14:44:01.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:01 smithi194 ceph-mon[44089]: pgmap v2118: 17 pgs: 4 active+recovering, 13 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 117 op/s; 14129/160410 objects misplaced (8.808%); 15 MiB/s, 295 objects/s recovering 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:150191 7.b 7.7d62e29b (undecoded) ondisk+write+known_if_redirected e821)", 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:02.432396+0000", 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.099947089000000003, 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099976406000000004, 2024-04-29T14:44:02.535 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 150191 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.432396+0000", 2024-04-29T14:44:02.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.432396+0000", 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.432399+0000", 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0000000000000001e-06 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.432407+0000", 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1119999999999996e-06 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.432420+0000", 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3264e-05 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:02.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.432431+0000", 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0637e-05 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:150166 7.b 7.b3a72bab (undecoded) ondisk+retry+write+known_if_redirected e821)", 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:02.438672+0000", 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.093671414999999994, 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.093785902000000004, 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:02.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 150166 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.438672+0000", 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.438672+0000", 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.438674+0000", 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9309999999999998e-06 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.438679+0000", 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4349999999999996e-06 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.438689+0000", 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9580000000000002e-06 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:02.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.438696+0000", 2024-04-29T14:44:02.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9450000000000004e-06 2024-04-29T14:44:02.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.543 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:02.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.544 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_backoff(7.b ack-block id 49 [7:db546f45:::benchmark_data_smithi194_147200_object35234:head,7:db546f45:::benchmark_data_smithi194_147200_object35234:head) e821)", 2024-04-29T14:44:02.544 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:02.507445+0000", 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.024897774000000001, 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025066990000000001, 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:02.545 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 0 2024-04-29T14:44:02.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.546 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:02.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:02.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.507445+0000", 2024-04-29T14:44:02.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:02.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.507445+0000", 2024-04-29T14:44:02.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:02.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.507447+0000", 2024-04-29T14:44:02.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8780000000000001e-06 2024-04-29T14:44:02.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:02.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.507452+0000", 2024-04-29T14:44:02.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8849999999999998e-06 2024-04-29T14:44:02.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:02.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.507467+0000", 2024-04-29T14:44:02.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5259000000000001e-05 2024-04-29T14:44:02.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:02.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.507480+0000", 2024-04-29T14:44:02.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2938e-05 2024-04-29T14:44:02.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.552 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:02.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.553 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:149879 7.b 7:db803c87:::benchmark_data_smithi194_147200_object34967:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e821)", 2024-04-29T14:44:02.553 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:02.325064+0000", 2024-04-29T14:44:02.553 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.20727879099999999, 2024-04-29T14:44:02.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20751209900000001, 2024-04-29T14:44:02.553 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:02.554 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:02.554 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:02.554 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:02.554 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:02.555 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:02.555 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 149879 2024-04-29T14:44:02.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.555 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:02.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:02.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.325064+0000", 2024-04-29T14:44:02.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:02.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.325064+0000", 2024-04-29T14:44:02.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:02.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.325066+0000", 2024-04-29T14:44:02.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.934e-06 2024-04-29T14:44:02.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:02.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.325072+0000", 2024-04-29T14:44:02.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1360000000000002e-06 2024-04-29T14:44:02.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:02.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.325086+0000", 2024-04-29T14:44:02.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.305e-05 2024-04-29T14:44:02.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:02.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.325096+0000", 2024-04-29T14:44:02.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0026e-05 2024-04-29T14:44:02.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:02.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.446289+0000", 2024-04-29T14:44:02.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.12119358 2024-04-29T14:44:02.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:02.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.452308+0000", 2024-04-29T14:44:02.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.006018855 2024-04-29T14:44:02.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:02.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.462151+0000", 2024-04-29T14:44:02.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098427270000000008 2024-04-29T14:44:02.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:02.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.497576+0000", 2024-04-29T14:44:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035424799 2024-04-29T14:44:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.512540+0000", 2024-04-29T14:44:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014964565000000001 2024-04-29T14:44:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.517644+0000", 2024-04-29T14:44:02.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0051036650000000003 2024-04-29T14:44:02.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.568 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:02.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.569 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:150146 7.b 7:db546f45:::benchmark_data_smithi194_147200_object35234:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e821)", 2024-04-29T14:44:02.569 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:02.363241+0000", 2024-04-29T14:44:02.569 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.16910198700000001, 2024-04-29T14:44:02.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.169442286, 2024-04-29T14:44:02.570 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:02.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:02.570 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:44:02.570 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:02.571 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:02.571 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:02.571 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 150146 2024-04-29T14:44:02.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.571 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:02.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:02.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.363241+0000", 2024-04-29T14:44:02.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:02.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.363241+0000", 2024-04-29T14:44:02.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:02.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.363243+0000", 2024-04-29T14:44:02.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0040000000000002e-06 2024-04-29T14:44:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.363250+0000", 2024-04-29T14:44:02.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4110000000000001e-06 2024-04-29T14:44:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.363262+0000", 2024-04-29T14:44:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2279999999999999e-05 2024-04-29T14:44:02.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.363270+0000", 2024-04-29T14:44:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3690000000000007e-06 2024-04-29T14:44:02.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:02.501616+0000", 2024-04-29T14:44:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13834544600000001 2024-04-29T14:44:02.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:44:02.579 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 5 2024-04-29T14:44:02.580 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:44:03.018 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:03.021 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:44:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:03.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:44:03.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:03.157 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:03.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:03.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:03.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:44:03.312 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:03.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:03 smithi040 ceph-mon[39581]: pgmap v2119: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 14129/160114 objects misplaced (8.824%); 12 MiB/s, 339 objects/s recovering 2024-04-29T14:44:03.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2768841606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:44:03.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:03 smithi040 ceph-mon[39581]: from='client.27484 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:44:03.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:44:03.516 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:44:03.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:03 smithi121 ceph-mon[44533]: pgmap v2119: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 14129/160114 objects misplaced (8.824%); 12 MiB/s, 339 objects/s recovering 2024-04-29T14:44:03.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2768841606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:44:03.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:03 smithi121 ceph-mon[44533]: from='client.27484 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:44:03.763 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:44:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:03.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:44:03.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:03 smithi194 ceph-mon[44089]: pgmap v2119: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 14129/160114 objects misplaced (8.824%); 12 MiB/s, 339 objects/s recovering 2024-04-29T14:44:03.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2768841606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:44:03.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:03 smithi194 ceph-mon[44089]: from='client.27484 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:44:03.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:04.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:04 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:44:04.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:04 smithi121 ceph-mon[44533]: from='client.27484 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:44:04.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:04 smithi121 ceph-mon[44533]: osdmap e822: 12 total, 12 up, 10 in 2024-04-29T14:44:04.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:04 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:44:04.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:04 smithi194 ceph-mon[44089]: from='client.27484 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:44:04.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:04 smithi194 ceph-mon[44089]: osdmap e822: 12 total, 12 up, 10 in 2024-04-29T14:44:04.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:04 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:44:04.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:04 smithi040 ceph-mon[39581]: from='client.27484 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:44:04.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:04 smithi040 ceph-mon[39581]: osdmap e822: 12 total, 12 up, 10 in 2024-04-29T14:44:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:04.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:04.963 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:04.992 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:05.014 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:05.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:05.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:05.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:05 smithi040 ceph-mon[39581]: pgmap v2121: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 115 op/s; 14129/159636 objects misplaced (8.851%); 6.0 MiB/s, 402 objects/s recovering 2024-04-29T14:44:05.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:05.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:05.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:05 smithi040 ceph-mon[39581]: 7.c scrub starts 2024-04-29T14:44:05.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:05 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:44:05.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:05 smithi121 ceph-mon[44533]: pgmap v2121: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 115 op/s; 14129/159636 objects misplaced (8.851%); 6.0 MiB/s, 402 objects/s recovering 2024-04-29T14:44:05.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:05.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:05.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:05 smithi121 ceph-mon[44533]: 7.c scrub starts 2024-04-29T14:44:05.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:05 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:44:05.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:05 smithi194 ceph-mon[44089]: pgmap v2121: 17 pgs: 3 active+recovering, 14 active+clean; 4.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 115 op/s; 14129/159636 objects misplaced (8.851%); 6.0 MiB/s, 402 objects/s recovering 2024-04-29T14:44:05.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:05.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:05.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:05 smithi194 ceph-mon[44089]: 7.c scrub starts 2024-04-29T14:44:05.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:05 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:44:05.912 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:05.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:06.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:44:06.050 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:06.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:44:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:06.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:44:06.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:06.207 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:06.227 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:06.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:06.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:06.359 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:06.367 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:06.374 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:44:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:06.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:06.397 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:44:06.426 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:44:06.447 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:44:06.468 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:44:06.484 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:06.488 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:44:06.508 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:44:06.510 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:06.514 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.4 to 0.71408 (b6cd) 2024-04-29T14:44:06.530 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:44:06.531 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:06.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:06.553 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:44:06.573 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:44:06.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:44:06.620 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:44:06.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:44:06.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:44:06.622 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:44:06.623 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:44:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:06 smithi121 ceph-mon[44533]: 7.c scrub ok 2024-04-29T14:44:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:06 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:44:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:06 smithi121 ceph-mon[44533]: osdmap e823: 12 total, 12 up, 10 in 2024-04-29T14:44:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4220779481' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7140804141184572}]: dispatch 2024-04-29T14:44:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:06 smithi121 ceph-mon[44533]: 7.e scrub starts 2024-04-29T14:44:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:06 smithi121 ceph-mon[44533]: pgmap v2123: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 159 op/s; 14015/158692 objects misplaced (8.832%); 6.3 MiB/s, 396 objects/s recovering 2024-04-29T14:44:06.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3697329672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:06.635 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:06.662 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:06.680 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:06.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:06.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:06.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:06 smithi194 ceph-mon[44089]: 7.c scrub ok 2024-04-29T14:44:06.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:06 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:44:06.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:06 smithi194 ceph-mon[44089]: osdmap e823: 12 total, 12 up, 10 in 2024-04-29T14:44:06.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4220779481' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7140804141184572}]: dispatch 2024-04-29T14:44:06.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:06 smithi194 ceph-mon[44089]: 7.e scrub starts 2024-04-29T14:44:06.809 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:06 smithi194 ceph-mon[44089]: pgmap v2123: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 159 op/s; 14015/158692 objects misplaced (8.832%); 6.3 MiB/s, 396 objects/s recovering 2024-04-29T14:44:06.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3697329672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:06.811 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:06.829 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:06.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:06.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:06 smithi040 ceph-mon[39581]: 7.c scrub ok 2024-04-29T14:44:06.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:06 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:44:06.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:06 smithi040 ceph-mon[39581]: osdmap e823: 12 total, 12 up, 10 in 2024-04-29T14:44:06.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4220779481' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7140804141184572}]: dispatch 2024-04-29T14:44:06.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:06 smithi040 ceph-mon[39581]: 7.e scrub starts 2024-04-29T14:44:06.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:06 smithi040 ceph-mon[39581]: pgmap v2123: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 159 op/s; 14015/158692 objects misplaced (8.832%); 6.3 MiB/s, 396 objects/s recovering 2024-04-29T14:44:06.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3697329672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:06.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:06.962 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:06.981 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:06.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:07.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:44:07.122 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: 7.e scrub ok 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4220779481' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.7140804141184572}]': finished 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osdmap e824: 12 total, 12 up, 10 in 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:44:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:44:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:07 smithi194 ceph-mon[44089]: osdmap e825: 12 total, 12 up, 10 in 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: 7.e scrub ok 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4220779481' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.7140804141184572}]': finished 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osdmap e824: 12 total, 12 up, 10 in 2024-04-29T14:44:07.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:44:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:44:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:07 smithi040 ceph-mon[39581]: osdmap e825: 12 total, 12 up, 10 in 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: 7.e scrub ok 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4220779481' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.7140804141184572}]': finished 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osdmap e824: 12 total, 12 up, 10 in 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:44:07.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:07 smithi121 ceph-mon[44533]: osdmap e825: 12 total, 12 up, 10 in 2024-04-29T14:44:07.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:44:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:44:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:07.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:44:08.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:08.362 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:08.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:08.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:44:08.500 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:08.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:08 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:44:08.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:08 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:44:08.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:08 smithi040 ceph-mon[39581]: pgmap v2126: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 150 op/s; 14015/158692 objects misplaced (8.832%); 1.1 MiB/s, 269 objects/s recovering 2024-04-29T14:44:08.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:08 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:44:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:08 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:44:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:08 smithi194 ceph-mon[44089]: pgmap v2126: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 150 op/s; 14015/158692 objects misplaced (8.832%); 1.1 MiB/s, 269 objects/s recovering 2024-04-29T14:44:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:08 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:44:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:08 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:44:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:08 smithi121 ceph-mon[44533]: pgmap v2126: 17 pgs: 3 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 150 op/s; 14015/158692 objects misplaced (8.832%); 1.1 MiB/s, 269 objects/s recovering 2024-04-29T14:44:09.154 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:44:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:09.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:44:09.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:09.287 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:09.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:09.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:09.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:09.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:09.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:09.598 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:09 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:44:09.598 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:09 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:44:09.598 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:09 smithi121 ceph-mon[44533]: 7.1 scrub starts 2024-04-29T14:44:09.598 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:09 smithi121 ceph-mon[44533]: 7.a scrub starts 2024-04-29T14:44:09.599 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:09.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:09.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:09.722 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:09.748 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:09.749 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:09.768 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:09.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:09.785 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:44:09.785 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:44:09.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:09 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:44:09.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:09 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:44:09.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:09 smithi040 ceph-mon[39581]: 7.1 scrub starts 2024-04-29T14:44:09.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:09 smithi040 ceph-mon[39581]: 7.a scrub starts 2024-04-29T14:44:09.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:09 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:44:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:09 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:44:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:09 smithi194 ceph-mon[44089]: 7.1 scrub starts 2024-04-29T14:44:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:09 smithi194 ceph-mon[44089]: 7.a scrub starts 2024-04-29T14:44:09.871 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:09.898 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:09.918 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:09.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:09.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:44:10.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:10.047 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:10.066 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:10.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:10.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:44:10.216 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:10.387 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:44:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:10.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:44:10.492 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:10.519 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:10.540 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:10.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:10.644 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:10.671 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:10.690 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:10.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:10.794 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:10 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:44:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:10 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:44:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:10 smithi194 ceph-mon[44089]: 7.1 scrub ok 2024-04-29T14:44:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:10 smithi194 ceph-mon[44089]: 7.a scrub ok 2024-04-29T14:44:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:10 smithi194 ceph-mon[44089]: 7.3 scrub starts 2024-04-29T14:44:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:10 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:44:10.819 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:10 smithi194 ceph-mon[44089]: pgmap v2127: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 14015/158174 objects misplaced (8.860%); 864 KiB/s, 88 objects/s recovering 2024-04-29T14:44:10.821 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:10.840 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:10.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:10.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:10 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:44:10.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:10 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:44:10.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:10 smithi040 ceph-mon[39581]: 7.1 scrub ok 2024-04-29T14:44:10.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:10 smithi040 ceph-mon[39581]: 7.a scrub ok 2024-04-29T14:44:10.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:10 smithi040 ceph-mon[39581]: 7.3 scrub starts 2024-04-29T14:44:10.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:10 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:44:10.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:10 smithi040 ceph-mon[39581]: pgmap v2127: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 14015/158174 objects misplaced (8.860%); 864 KiB/s, 88 objects/s recovering 2024-04-29T14:44:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:10 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:44:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:10 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:44:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:10 smithi121 ceph-mon[44533]: 7.1 scrub ok 2024-04-29T14:44:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:10 smithi121 ceph-mon[44533]: 7.a scrub ok 2024-04-29T14:44:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:10 smithi121 ceph-mon[44533]: 7.3 scrub starts 2024-04-29T14:44:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:10 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:44:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:10 smithi121 ceph-mon[44533]: pgmap v2127: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 14015/158174 objects misplaced (8.860%); 864 KiB/s, 88 objects/s recovering 2024-04-29T14:44:10.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:10.973 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:10.994 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:10.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:11.099 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:11.125 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:11.144 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:11.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:11.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:44:11.275 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:11.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:44:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:11.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:44:11.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:44:11.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:11.424 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:11.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:11.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:11.532 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:11 smithi040 ceph-mon[39581]: 7.3 scrub ok 2024-04-29T14:44:11.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:44:11.533 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:44:11.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:44:11.572 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:11.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:44:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:11.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:44:11.695 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:11.722 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:11.740 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:11.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:11.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:11 smithi194 ceph-mon[44089]: 7.3 scrub ok 2024-04-29T14:44:11.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:44:11.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:44:11.823 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:11.844 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:11.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:11 smithi121 ceph-mon[44533]: 7.3 scrub ok 2024-04-29T14:44:11.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:44:11.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:44:12.082 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:12.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:12.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:12.215 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:12.238 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:12.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:12.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:12.377 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:12 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:44:12.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:12 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:44:12.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:12 smithi040 ceph-mon[39581]: pgmap v2128: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 71 op/s; 13920/157736 objects misplaced (8.825%); 562 KiB/s, 75 objects/s recovering 2024-04-29T14:44:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:12 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:44:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:12 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:44:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:12 smithi194 ceph-mon[44089]: pgmap v2128: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 71 op/s; 13920/157736 objects misplaced (8.825%); 562 KiB/s, 75 objects/s recovering 2024-04-29T14:44:12.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:12 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:44:12.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:12 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:44:12.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:12 smithi121 ceph-mon[44533]: pgmap v2128: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 71 op/s; 13920/157736 objects misplaced (8.825%); 562 KiB/s, 75 objects/s recovering 2024-04-29T14:44:13.127 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:44:13.129 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:13.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:44:13.236 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:13.262 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:13.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:13.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:13.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:13.418 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:13.442 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:44:13.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:13.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:44:13.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:13.554 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:44:13.573 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:13.591 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:13.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:13.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:44:13.721 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:13.740 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:44:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:13.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:44:13.752 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:13 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:44:13.752 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3747105778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:44:13.752 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:13 smithi040 ceph-mon[39581]: from='client.27502 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:44:13.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:13 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:44:13.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3747105778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:44:13.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:13 smithi194 ceph-mon[44089]: from='client.27502 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:44:13.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:13.850 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] out_osds: [3, 11] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:44:13.850 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:44:13.850 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-29T14:44:13.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-04-29T14:44:13.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:13 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:44:13.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3747105778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:44:13.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:13 smithi121 ceph-mon[44533]: from='client.27502 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:44:13.876 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:13.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:13.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:14.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:14.031 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:14.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:14.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:14.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:14.190 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:14.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:14 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:44:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:14 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:44:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:14 smithi194 ceph-mon[44089]: from='client.27502 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:44:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:14 smithi194 ceph-mon[44089]: osdmap e826: 12 total, 12 up, 10 in 2024-04-29T14:44:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:14 smithi194 ceph-mon[44089]: pgmap v2130: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 13920/157416 objects misplaced (8.843%); 489 KiB/s, 66 objects/s recovering 2024-04-29T14:44:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:14 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:44:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:14 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:44:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:14 smithi040 ceph-mon[39581]: from='client.27502 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:44:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:14 smithi040 ceph-mon[39581]: osdmap e826: 12 total, 12 up, 10 in 2024-04-29T14:44:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:14 smithi040 ceph-mon[39581]: pgmap v2130: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 13920/157416 objects misplaced (8.843%); 489 KiB/s, 66 objects/s recovering 2024-04-29T14:44:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:14 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:44:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:14 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:44:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:14 smithi121 ceph-mon[44533]: from='client.27502 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:44:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:14 smithi121 ceph-mon[44533]: osdmap e826: 12 total, 12 up, 10 in 2024-04-29T14:44:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:14 smithi121 ceph-mon[44533]: pgmap v2130: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 82 op/s; 13920/157416 objects misplaced (8.843%); 489 KiB/s, 66 objects/s recovering 2024-04-29T14:44:15.442 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:15.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:15.536 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:44:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:44:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:44:15.547 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:15.573 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:15.591 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:15.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:15.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:44:15.724 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:15.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:44:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:15.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:44:15.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1710984506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:15.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.847 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:15.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1710984506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:15.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.873 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:15.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1710984506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:15.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:15.893 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:15.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:16.000 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:16.026 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:16.045 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:16.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:16.149 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:16.175 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:16.193 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:16.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:16.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:16.326 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:16.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:16.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:16.450 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:16.477 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:16.496 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:16.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:16.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:44:16.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:16 smithi040 ceph-mon[39581]: pgmap v2131: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 13814/156618 objects misplaced (8.820%); 937 KiB/s, 66 objects/s recovering 2024-04-29T14:44:16.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:16 smithi121 ceph-mon[44533]: pgmap v2131: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 13814/156618 objects misplaced (8.820%); 937 KiB/s, 66 objects/s recovering 2024-04-29T14:44:16.627 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:16.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:44:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:16.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:44:16.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:16.780 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:16.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:16 smithi194 ceph-mon[44089]: pgmap v2131: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 13814/156618 objects misplaced (8.820%); 937 KiB/s, 66 objects/s recovering 2024-04-29T14:44:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:16.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:16.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:16.987 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:17.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:17.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:17.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:17.136 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:17.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:17.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:17.259 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:17.286 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:17.304 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:17.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:17.408 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:17.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:44:17.435 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:17.454 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:17.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:17.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:17.585 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:17.604 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:17.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:17.708 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:17.741 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:17.761 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:44:17.761 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:44:17.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:44:17.761 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:44:17.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:117990 7.b 7:de64f127:::benchmark_data_smithi194_147200_object3078:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e704)", 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:29.105154+0000", 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 408.65102002600003, 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.267556355, 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.762 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 117990 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105154+0000", 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105154+0000", 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105156+0000", 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4499999999999998e-06 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105176+0000", 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0128000000000001e-05 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105192+0000", 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.575e-05 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:29.105203+0000", 2024-04-29T14:44:17.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.13e-05 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:30.951863+0000", 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8466592209999999 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:30.963084+0000", 2024-04-29T14:44:17.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011221139999999999 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:30.991007+0000", 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027923228000000001 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.158557+0000", 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.167550486 2024-04-29T14:44:17.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.313582+0000", 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.155024103 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.326972+0000", 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013390465000000001 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:31.372710+0000", 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045738083999999998 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:118814 7.b 7:d91c083d:::benchmark_data_smithi194_147200_object3902:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e705)", 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:39.999367+0000", 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 397.75680694300002, 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2752686199999999, 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.769 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 118814 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999367+0000", 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999367+0000", 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999369+0000", 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5670000000000002e-06 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999376+0000", 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4350000000000002e-06 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999388+0000", 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1773999999999999e-05 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:39.999398+0000", 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9809999999999998e-06 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.081094+0000", 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0816960959999999 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.095508+0000", 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014414089 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.113227+0000", 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017718854999999999 2024-04-29T14:44:17.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.230362+0000", 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117135268 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.235821+0000", 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054593080000000004 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.242421+0000", 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065999220000000003 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.274635+0000", 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032214325000000002 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.b 706/685 [PushOp(7:db144375:::benchmark_data_smithi194_147200_object7387:head, version: 660'447, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:db144375:::benchmark_data_smithi194_147200_object7387:head@660'447, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:42.076484+0000", 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 395.679689693, 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0814666800000001, 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.076484+0000", 2024-04-29T14:44:17.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.076484+0000", 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.076485+0000", 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0589999999999999e-06 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.078894+0000", 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024089049999999998 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.078919+0000", 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5150000000000001e-05 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:42.078936+0000", 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6759e-05 2024-04-29T14:44:17.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.092967+0000", 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0140306480000003 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.094795+0000", 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001828742 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.157951+0000", 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.063155417000000005 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:118994 7.b 7:d9c86171:::benchmark_data_smithi194_147200_object4082:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e706)", 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:44.016228+0000", 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 393.73994550700002, 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.26731229, 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 118994 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016228+0000", 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016228+0000", 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016230+0000", 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1569999999999998e-06 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016236+0000", 2024-04-29T14:44:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9460000000000003e-06 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016250+0000", 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3439e-05 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.016260+0000", 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0132000000000001e-05 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.144175+0000", 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1279146870000001 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.163477+0000", 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019302875000000001 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.201695+0000", 2024-04-29T14:44:17.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038217802000000002 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.231766+0000", 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030071123000000002 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.236716+0000", 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049493640000000004 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.249555+0000", 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012838887 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.283541+0000", 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033985877999999997 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.786 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119001 7.b 7:dabe6e1c:::benchmark_data_smithi194_147200_object4089:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e706)", 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:44.278109+0000", 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 393.47806473899999, 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3018859850000002, 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119001 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278109+0000", 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278109+0000", 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278112+0000", 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8700000000000001e-06 2024-04-29T14:44:17.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278117+0000", 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9749999999999998e-06 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278129+0000", 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2351000000000001e-05 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.278138+0000", 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6130000000000004e-06 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.298009+0000", 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.019871331 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.315737+0000", 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017727837 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.347200+0000", 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031463158999999997 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.446480+0000", 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099279661000000005 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.537923+0000", 2024-04-29T14:44:17.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091443128999999998 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.546321+0000", 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0083982280000000006 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.579995+0000", 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033673831000000001 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119012 7.b 7:d129b0f8:::benchmark_data_smithi194_147200_object4100:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e706)", 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:44.364840+0000", 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 393.39133340299998, 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.272131312, 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119012 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.793 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364840+0000", 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364840+0000", 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364843+0000", 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4870000000000001e-06 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364849+0000", 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.409e-06 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364864+0000", 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4358999999999999e-05 2024-04-29T14:44:17.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:44.364874+0000", 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0329999999999999e-05 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.367558+0000", 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0026840539999999 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.389974+0000", 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022416044 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.423904+0000", 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033930035999999997 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.454036+0000", 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030132246000000001 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.592166+0000", 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.138129746 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.601343+0000", 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091768779999999994 2024-04-29T14:44:17.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:46.636972+0000", 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035628723000000001 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.b 706/685 [PushOp(7:df05b564:::benchmark_data_smithi194_147200_object7394:head, version: 660'448, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:df05b564:::benchmark_data_smithi194_147200_object7394:head@660'448, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:47.217991+0000", 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 390.538182484, 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5504673069999999, 2024-04-29T14:44:17.799 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.217991+0000", 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.217991+0000", 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.217993+0000", 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9039999999999999e-06 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.218037+0000", 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3708000000000002e-05 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.218068+0000", 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1226999999999998e-05 2024-04-29T14:44:17.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:47.218087+0000", 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9157999999999999e-05 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.695381+0000", 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4772933449999996 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.696681+0000", 2024-04-29T14:44:17.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0013008150000000001 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.768459+0000", 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.071777149999999998 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119255 7.b 7:d1e0d635:::benchmark_data_smithi194_147200_object4343:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:49.935746+0000", 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 387.82042799300001, 2024-04-29T14:44:17.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3167341920000002, 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119255 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935746+0000", 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935746+0000", 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935748+0000", 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2840000000000001e-06 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935754+0000", 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.268e-06 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935767+0000", 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2537999999999999e-05 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:49.935780+0000", 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2989e-05 2024-04-29T14:44:17.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:51.985549+0000", 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.049769038 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.002198+0000", 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016649489999999999 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.027095+0000", 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024896965 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.057472+0000", 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030376844 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.199007+0000", 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.14153477 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.210572+0000", 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011565351999999999 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.252480+0000", 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041907654000000003 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_backoff(7.b ack-block id 13 [7:de377bca:::benchmark_data_smithi194_147200_object4372:head,7:de377bca:::benchmark_data_smithi194_147200_object4372:head) e707)", 2024-04-29T14:44:17.809 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.173628+0000", 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 387.58254577999998, 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.313792522, 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 0 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173628+0000", 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173628+0000", 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173630+0000", 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0339999999999999e-06 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173636+0000", 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9989999999999997e-06 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173648+0000", 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1668000000000001e-05 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.173657+0000", 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3030000000000005e-06 2024-04-29T14:44:17.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.483091+0000", 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3094342719999998 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.487420+0000", 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043292460000000001 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_backoff(7.b ack-block id 14 [7:d521d79c:::benchmark_data_smithi194_147200_object4393:head,7:d521d79c:::benchmark_data_smithi194_147200_object4393:head) e707)", 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.226908+0000", 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 387.52926615799998, 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2674437890000001, 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 0 2024-04-29T14:44:17.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226908+0000", 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226908+0000", 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226911+0000", 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1209999999999998e-06 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226919+0000", 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7329999999999992e-06 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226936+0000", 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6331e-05 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.226948+0000", 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2583e-05 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.489526+0000", 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2625775940000001 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.494351+0000", 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048254270000000002 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119284 7.b 7:de377bca:::benchmark_data_smithi194_147200_object4372:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.366292+0000", 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 387.38988206599998, 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3351182779999999, 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119284 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.818 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366292+0000", 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366292+0000", 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366294+0000", 2024-04-29T14:44:17.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.526e-06 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366300+0000", 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2179999999999996e-06 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366314+0000", 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.362e-05 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.366327+0000", 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2935999999999999e-05 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.496304+0000", 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1299769460000002 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.515416+0000", 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019111624000000001 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.547443+0000", 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032027003999999998 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.630161+0000", 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.082718749999999994 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.637612+0000", 2024-04-29T14:44:17.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0074507410000000003 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.652514+0000", 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014901456 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.701410+0000", 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.048896456999999997 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119305 7.b 7:d521d79c:::benchmark_data_smithi194_147200_object4393:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.452694+0000", 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 387.30347963200001, 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3006495139999998, 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119305 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.824 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452694+0000", 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452694+0000", 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452697+0000", 2024-04-29T14:44:17.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2550000000000002e-06 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452707+0000", 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2979999999999997e-06 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452721+0000", 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4333e-05 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.452733+0000", 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2201999999999999e-05 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.564745+0000", 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1120117120000002 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.580509+0000", 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015763762000000001 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.611657+0000", 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031148216999999999 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.706129+0000", 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.094471609999999998 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.708457+0000", 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023288789999999998 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.718029+0000", 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0095712799999999997 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.753344+0000", 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035314966000000003 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119327 7.b 7:d9b2aa38:::benchmark_data_smithi194_147200_object4415:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e707)", 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.720028+0000", 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 387.03614567599999, 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4320200239999998, 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119327 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.830 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720028+0000", 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720028+0000", 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720030+0000", 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3470000000000001e-06 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720042+0000", 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1753000000000001e-05 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720056+0000", 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3743e-05 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.720066+0000", 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0532e-05 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.812713+0000", 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.092646647 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.829361+0000", 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016648185999999999 2024-04-29T14:44:17.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:52.856838+0000", 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027476295000000001 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.013113+0000", 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.156275108 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.098238+0000", 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.085125485000000001 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.110242+0000", 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012003443000000001 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.152048+0000", 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041806484999999997 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.835 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_backoff(7.b ack-block id 16 [7:d0fb8c6f:::benchmark_data_smithi194_147200_object4426:head,7:d0fb8c6f:::benchmark_data_smithi194_147200_object4426:head) e707)", 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:50.871369+0000", 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 386.88480482300002, 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.285846764, 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.836 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 0 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871369+0000", 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871369+0000", 2024-04-29T14:44:17.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871371+0000", 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2500000000000001e-06 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871378+0000", 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5880000000000003e-06 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871389+0000", 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1100999999999999e-05 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:50.871398+0000", 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4520000000000003e-06 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.154010+0000", 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2826120670000001 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.157216+0000", 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032053060000000002 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119369 7.b 7:dbd47a03:::benchmark_data_smithi194_147200_object4457:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:53.599267+0000", 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 384.15690661399998, 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2839892009999998, 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119369 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599267+0000", 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599267+0000", 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599270+0000", 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599280+0000", 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0003000000000001e-05 2024-04-29T14:44:17.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599295+0000", 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5602000000000001e-05 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.599307+0000", 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1816e-05 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.630256+0000", 2024-04-29T14:44:17.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.030948413 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.647832+0000", 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017576590999999999 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.672860+0000", 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025027997 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.774087+0000", 2024-04-29T14:44:17.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10122699 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.845211+0000", 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.071124110000000004 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.850670+0000", 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054586119999999998 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.883256+0000", 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032586400000000001 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119376 7.b 7:d6a7ee34:::benchmark_data_smithi194_147200_object4464:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:53.726473+0000", 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 384.02970102299997, 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3746222100000001, 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.846 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119376 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726473+0000", 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726473+0000", 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726475+0000", 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4990000000000001e-06 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726483+0000", 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5310000000000004e-06 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726496+0000", 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2986999999999999e-05 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.726506+0000", 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0224000000000001e-05 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.778681+0000", 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0521747239999999 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.794207+0000", 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015526280999999999 2024-04-29T14:44:17.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.819389+0000", 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025181756999999999 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.888522+0000", 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.069132790999999999 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.051202+0000", 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.162680882 2024-04-29T14:44:17.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.059671+0000", 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084690540000000002 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.101095+0000", 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041423479999999999 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.851 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119384 7.b 7:dadd1e30:::benchmark_data_smithi194_147200_object4472:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:53.858936+0000", 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 383.897237491, 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3006659549999999, 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119384 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858936+0000", 2024-04-29T14:44:17.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858936+0000", 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858940+0000", 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.557e-06 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858949+0000", 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3670000000000003e-06 2024-04-29T14:44:17.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858967+0000", 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.747e-05 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:53.858980+0000", 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3413e-05 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.892216+0000", 2024-04-29T14:44:17.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.033235763 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.910300+0000", 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018083825000000001 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.942081+0000", 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031780924000000002 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:55.976254+0000", 2024-04-29T14:44:17.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034173583 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.109755+0000", 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.133500392 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.120572+0000", 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010817149999999999 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.159602+0000", 2024-04-29T14:44:17.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039030510999999997 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119426 7.b 7:d9880d86:::benchmark_data_smithi194_147200_object4514:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:54.130484+0000", 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 383.62568935500002, 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.285222273, 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.857 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119426 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130484+0000", 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130484+0000", 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130488+0000", 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1250000000000001e-06 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130500+0000", 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2599000000000001e-05 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130514+0000", 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3791000000000001e-05 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.130529+0000", 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4832e-05 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.231644+0000", 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.101114919 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.254569+0000", 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022925634 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.286229+0000", 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031659364000000002 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.368671+0000", 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.082442721999999996 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.372801+0000", 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041300160000000002 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.385050+0000", 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012248785999999999 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.415707+0000", 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030656485000000001 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119436 7.b 7:dbb5cb9b:::benchmark_data_smithi194_147200_object4524:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e708)", 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:54.225054+0000", 2024-04-29T14:44:17.862 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 383.53111962399998, 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2775057159999998, 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119436 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225054+0000", 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225054+0000", 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225057+0000", 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3239999999999999e-06 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225070+0000", 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2310000000000001e-05 2024-04-29T14:44:17.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225087+0000", 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6838e-05 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:54.225101+0000", 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4218e-05 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.305907+0000", 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0808057230000001 2024-04-29T14:44:17.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.319013+0000", 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013106922999999999 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.349486+0000", 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030472803999999999 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.421863+0000", 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.072376531999999993 2024-04-29T14:44:17.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.446339+0000", 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024475820999999998 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.458057+0000", 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011718833 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:56.502560+0000", 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.044502390000000003 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPushReply(7.b 711/685 [PushReplyOp(7:db144375:::benchmark_data_smithi194_147200_object7387:head)])", 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:38:02.687791+0000", 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 375.06838298899999, 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0677553089999998, 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687791+0000", 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687791+0000", 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687793+0000", 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9470000000000002e-06 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687801+0000", 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0439999999999996e-06 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687814+0000", 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3577999999999999e-05 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:02.687825+0000", 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0341e-05 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:05.709764+0000", 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0219390709999998 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:44:17.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:38:05.755546+0000", 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.045782327999999997 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:44:17.871 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:44:18.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:44:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:18 smithi194 ceph-mon[44089]: osdmap e827: 12 total, 12 up, 10 in 2024-04-29T14:44:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:18 smithi194 ceph-mon[44089]: pgmap v2133: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 13814/156618 objects misplaced (8.820%); 1024 KiB/s, 15 objects/s recovering 2024-04-29T14:44:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:18 smithi040 ceph-mon[39581]: osdmap e827: 12 total, 12 up, 10 in 2024-04-29T14:44:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:18 smithi040 ceph-mon[39581]: pgmap v2133: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 13814/156618 objects misplaced (8.820%); 1024 KiB/s, 15 objects/s recovering 2024-04-29T14:44:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:18 smithi121 ceph-mon[44533]: osdmap e827: 12 total, 12 up, 10 in 2024-04-29T14:44:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:18 smithi121 ceph-mon[44533]: pgmap v2133: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 13814/156618 objects misplaced (8.820%); 1024 KiB/s, 15 objects/s recovering 2024-04-29T14:44:19.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:44:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:19.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:19 smithi040 ceph-mon[39581]: osdmap e828: 12 total, 12 up, 10 in 2024-04-29T14:44:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:19.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:19.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:19.732 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:19.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:19.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:19.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:19 smithi194 ceph-mon[44089]: osdmap e828: 12 total, 12 up, 10 in 2024-04-29T14:44:19.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:19.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:19 smithi121 ceph-mon[44533]: osdmap e828: 12 total, 12 up, 10 in 2024-04-29T14:44:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:19.913 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:44:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:19.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:44:19.965 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:19.975 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:20.018 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:20.045 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:20.063 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:20.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:20.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:20.198 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:20.272 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:44:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:20.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:44:20.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:20.408 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:moved 2 / 34 (5.88235%) 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:avg 3.4 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.95959 -> 1.8 (expected baseline 1.74929) 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.4 with 1 -> 3 pgs (0.294118 -> 0.882353 * mean) 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.2 with 7 -> 7 pgs (2.05882 -> 2.05882 * mean) 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:oload 1000 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0175 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.1751 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 0.4615 -> 0.6520 2024-04-29T14:44:20.521 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 weight 0.7141 -> 0.9927 2024-04-29T14:44:20.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:20 smithi194 ceph-mon[44089]: from='client.37268 v1:172.21.15.40:0/1477284099' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:44:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:20 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]: dispatch 2024-04-29T14:44:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:20 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]: dispatch 2024-04-29T14:44:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:20 smithi194 ceph-mon[44089]: pgmap v2135: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 13814/156070 objects misplaced (8.851%); 660 KiB/s, 10 objects/s recovering 2024-04-29T14:44:20.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:20 smithi040 ceph-mon[39581]: from='client.37268 v1:172.21.15.40:0/1477284099' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:44:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:20 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]: dispatch 2024-04-29T14:44:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:20 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]: dispatch 2024-04-29T14:44:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:20 smithi040 ceph-mon[39581]: pgmap v2135: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 13814/156070 objects misplaced (8.851%); 660 KiB/s, 10 objects/s recovering 2024-04-29T14:44:20.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:20 smithi121 ceph-mon[44533]: from='client.37268 v1:172.21.15.40:0/1477284099' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:44:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:20 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]: dispatch 2024-04-29T14:44:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:20 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]: dispatch 2024-04-29T14:44:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:20 smithi121 ceph-mon[44533]: pgmap v2135: 17 pgs: 2 active+recovering, 15 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 13814/156070 objects misplaced (8.851%); 660 KiB/s, 10 objects/s recovering 2024-04-29T14:44:21.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:21 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]': finished 2024-04-29T14:44:21.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:21 smithi040 ceph-mon[39581]: osdmap e829: 12 total, 12 up, 10 in 2024-04-29T14:44:21.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:21 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]': finished 2024-04-29T14:44:21.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:21 smithi194 ceph-mon[44089]: osdmap e829: 12 total, 12 up, 10 in 2024-04-29T14:44:21.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:21 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'65059','7':'42731'}"}]': finished 2024-04-29T14:44:21.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:21 smithi121 ceph-mon[44533]: osdmap e829: 12 total, 12 up, 10 in 2024-04-29T14:44:22.155 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-29T14:44:22.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-04-29T14:44:22.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:22 smithi194 ceph-mon[44089]: osdmap e830: 12 total, 12 up, 10 in 2024-04-29T14:44:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:22 smithi194 ceph-mon[44089]: pgmap v2138: 17 pgs: 1 remapped+peering, 1 unknown, 2 active+recovering, 13 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 13708/145914 objects misplaced (9.395%) 2024-04-29T14:44:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:22 smithi194 ceph-mon[44089]: osdmap e831: 12 total, 12 up, 10 in 2024-04-29T14:44:22.859 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:22.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:22.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:22 smithi040 ceph-mon[39581]: osdmap e830: 12 total, 12 up, 10 in 2024-04-29T14:44:22.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:22 smithi040 ceph-mon[39581]: pgmap v2138: 17 pgs: 1 remapped+peering, 1 unknown, 2 active+recovering, 13 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 13708/145914 objects misplaced (9.395%) 2024-04-29T14:44:22.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:22 smithi040 ceph-mon[39581]: osdmap e831: 12 total, 12 up, 10 in 2024-04-29T14:44:22.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:22 smithi121 ceph-mon[44533]: osdmap e830: 12 total, 12 up, 10 in 2024-04-29T14:44:22.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:22 smithi121 ceph-mon[44533]: pgmap v2138: 17 pgs: 1 remapped+peering, 1 unknown, 2 active+recovering, 13 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 13708/145914 objects misplaced (9.395%) 2024-04-29T14:44:22.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:22 smithi121 ceph-mon[44533]: osdmap e831: 12 total, 12 up, 10 in 2024-04-29T14:44:22.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:23.266 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:23.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:23.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:23.399 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:23.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:23.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:23.699 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:23.725 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:23.743 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:23.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:23.847 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:24.132 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:24.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:24.179 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:24.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:24.263 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:24.284 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:24.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:24.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:24.389 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:24.416 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:24.436 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:24.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:24.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:24.571 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:24.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:24.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:24.722 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:24.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:24.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:24.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:24.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:24.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:24 smithi194 ceph-mon[44089]: osdmap e832: 12 total, 12 up, 10 in 2024-04-29T14:44:24.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:24 smithi194 ceph-mon[44089]: pgmap v2141: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 18474/155158 objects misplaced (11.907%); 1.8 MiB/s, 29 objects/s recovering 2024-04-29T14:44:24.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:24.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:24.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:24.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:24 smithi040 ceph-mon[39581]: osdmap e832: 12 total, 12 up, 10 in 2024-04-29T14:44:24.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:24 smithi040 ceph-mon[39581]: pgmap v2141: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 18474/155158 objects misplaced (11.907%); 1.8 MiB/s, 29 objects/s recovering 2024-04-29T14:44:24.879 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:24.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:24.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:24.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:24 smithi121 ceph-mon[44533]: osdmap e832: 12 total, 12 up, 10 in 2024-04-29T14:44:24.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:24 smithi121 ceph-mon[44533]: pgmap v2141: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 18474/155158 objects misplaced (11.907%); 1.8 MiB/s, 29 objects/s recovering 2024-04-29T14:44:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:26.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:27.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:44:27.039 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:27.058 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:44:27.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:27.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:27.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:26 smithi194 ceph-mon[44089]: osdmap e833: 12 total, 12 up, 10 in 2024-04-29T14:44:27.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:26 smithi194 ceph-mon[44089]: pgmap v2143: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 190 MiB/s wr, 4.13k op/s; 18297/154574 objects misplaced (11.837%); 250 MiB/s, 4.05k objects/s recovering 2024-04-29T14:44:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:27.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:44:27.064 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:27.064 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:27.064 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:26 smithi040 ceph-mon[39581]: osdmap e833: 12 total, 12 up, 10 in 2024-04-29T14:44:27.064 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:26 smithi040 ceph-mon[39581]: pgmap v2143: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 190 MiB/s wr, 4.13k op/s; 18297/154574 objects misplaced (11.837%); 250 MiB/s, 4.05k objects/s recovering 2024-04-29T14:44:27.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:27.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:27.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:26 smithi121 ceph-mon[44533]: osdmap e833: 12 total, 12 up, 10 in 2024-04-29T14:44:27.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:26 smithi121 ceph-mon[44533]: pgmap v2143: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 190 MiB/s wr, 4.13k op/s; 18297/154574 objects misplaced (11.837%); 250 MiB/s, 4.05k objects/s recovering 2024-04-29T14:44:27.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:27.195 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:28.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:28 smithi194 ceph-mon[44089]: pgmap v2144: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 1.65k op/s; 22975/154514 objects misplaced (14.869%); 132 MiB/s, 2.15k objects/s recovering 2024-04-29T14:44:28.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:28 smithi040 ceph-mon[39581]: pgmap v2144: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 1.65k op/s; 22975/154514 objects misplaced (14.869%); 132 MiB/s, 2.15k objects/s recovering 2024-04-29T14:44:28.883 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:28 smithi121 ceph-mon[44533]: pgmap v2144: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 1.65k op/s; 22975/154514 objects misplaced (14.869%); 132 MiB/s, 2.15k objects/s recovering 2024-04-29T14:44:30.612 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:44:30.634 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:44:30.657 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:44:30.674 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:44:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:44:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:44:30.675 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:44:30.684 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:44:30.728 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:44:30.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:44:30.786 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:44:30.811 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:44:30.839 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:44:30.863 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:44:30.886 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:44:30.913 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:44:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:44:30.918 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:44:31.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:31.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: osdmap e834: 12 total, 12 up, 10 in 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: pgmap v2146: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 66 MiB/s wr, 1.51k op/s; 22624/154026 objects misplaced (14.688%); 122 MiB/s, 1.97k objects/s recovering 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1676668620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.354 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:31.357 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: osdmap e834: 12 total, 12 up, 10 in 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: pgmap v2146: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 66 MiB/s wr, 1.51k op/s; 22624/154026 objects misplaced (14.688%); 122 MiB/s, 1.97k objects/s recovering 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1676668620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.358 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:31.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: osdmap e834: 12 total, 12 up, 10 in 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: pgmap v2146: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 66 MiB/s wr, 1.51k op/s; 22624/154026 objects misplaced (14.688%); 122 MiB/s, 1.97k objects/s recovering 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1676668620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:31.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:31.487 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:31.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:31.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:31.612 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:31.647 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:32.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:44:32.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:32 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:44:32.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:44:32.322 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:32 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:32.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:32 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:44:33.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:33 smithi194 ceph-mon[44089]: pgmap v2147: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 22302/153544 objects misplaced (14.525%); 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:44:33.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:33 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:33.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:33 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:33.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:33 smithi194 ceph-mon[44089]: osdmap e835: 12 total, 12 up, 10 in 2024-04-29T14:44:33.371 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:33 smithi040 ceph-mon[39581]: pgmap v2147: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 22302/153544 objects misplaced (14.525%); 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:44:33.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:33 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:33.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:33 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:33.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:33 smithi040 ceph-mon[39581]: osdmap e835: 12 total, 12 up, 10 in 2024-04-29T14:44:33.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:33 smithi121 ceph-mon[44533]: pgmap v2147: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 22302/153544 objects misplaced (14.525%); 7.6 MiB/s, 121 objects/s recovering 2024-04-29T14:44:33.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:33 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:33.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:33 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:33.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:33 smithi121 ceph-mon[44533]: osdmap e835: 12 total, 12 up, 10 in 2024-04-29T14:44:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:35 smithi194 ceph-mon[44089]: pgmap v2149: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 21927/153110 objects misplaced (14.321%); 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:44:35.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:35 smithi040 ceph-mon[39581]: pgmap v2149: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 21927/153110 objects misplaced (14.321%); 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:44:35.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:35 smithi121 ceph-mon[44533]: pgmap v2149: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 21927/153110 objects misplaced (14.321%); 9.4 MiB/s, 150 objects/s recovering 2024-04-29T14:44:37.301 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:37.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:37.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:37.437 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:37.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:37.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:37.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:37 smithi194 ceph-mon[44089]: pgmap v2150: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 21851/152496 objects misplaced (14.329%); 8.2 MiB/s, 131 objects/s recovering 2024-04-29T14:44:37.591 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:37 smithi040 ceph-mon[39581]: pgmap v2150: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 21851/152496 objects misplaced (14.329%); 8.2 MiB/s, 131 objects/s recovering 2024-04-29T14:44:37.595 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:37.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:37 smithi121 ceph-mon[44533]: pgmap v2150: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 21851/152496 objects misplaced (14.329%); 8.2 MiB/s, 131 objects/s recovering 2024-04-29T14:44:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:44:38.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/768769940' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:44:38.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:38 smithi194 ceph-mon[44089]: from='client.37280 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:44:38.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/768769940' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:44:38.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:38 smithi040 ceph-mon[39581]: from='client.37280 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:44:38.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/768769940' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:44:38.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:38 smithi121 ceph-mon[44533]: from='client.37280 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:44:38.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:44:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:39 smithi194 ceph-mon[44089]: pgmap v2151: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 21588/152432 objects misplaced (14.162%); 7.3 MiB/s, 117 objects/s recovering 2024-04-29T14:44:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:39 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:44:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:39 smithi194 ceph-mon[44089]: from='client.37280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:44:39.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:39 smithi194 ceph-mon[44089]: osdmap e836: 12 total, 12 up, 10 in 2024-04-29T14:44:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:39 smithi040 ceph-mon[39581]: pgmap v2151: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 21588/152432 objects misplaced (14.162%); 7.3 MiB/s, 117 objects/s recovering 2024-04-29T14:44:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:39 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:44:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:39 smithi040 ceph-mon[39581]: from='client.37280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:44:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:39 smithi040 ceph-mon[39581]: osdmap e836: 12 total, 12 up, 10 in 2024-04-29T14:44:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:39 smithi121 ceph-mon[44533]: pgmap v2151: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 21588/152432 objects misplaced (14.162%); 7.3 MiB/s, 117 objects/s recovering 2024-04-29T14:44:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:39 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:44:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:39 smithi121 ceph-mon[44533]: from='client.37280 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:44:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:39 smithi121 ceph-mon[44533]: osdmap e836: 12 total, 12 up, 10 in 2024-04-29T14:44:39.649 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:39.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:39.753 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:39.781 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:39.800 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:39.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:39.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:39.932 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:39.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:39.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:40.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:44:40.089 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:40.109 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:44:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:40.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:44:40.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:40.239 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:40.259 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:40.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:40.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:40.502 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:41.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:41 smithi194 ceph-mon[44089]: pgmap v2153: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 21205/152076 objects misplaced (13.944%); 8.3 MiB/s, 132 objects/s recovering 2024-04-29T14:44:41.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:41 smithi040 ceph-mon[39581]: pgmap v2153: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 21205/152076 objects misplaced (13.944%); 8.3 MiB/s, 132 objects/s recovering 2024-04-29T14:44:41.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:41 smithi121 ceph-mon[44533]: pgmap v2153: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 91 op/s; 21205/152076 objects misplaced (13.944%); 8.3 MiB/s, 132 objects/s recovering 2024-04-29T14:44:42.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:44:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:44:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:42.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:44:42.573 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:42.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:42.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:42.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:42.759 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:42.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:42.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:42.839 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:42.840 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:42.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:44:42.914 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:42.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:44:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:42.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:44:43.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:43.077 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:43.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:43 smithi040 ceph-mon[39581]: pgmap v2154: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 20842/151590 objects misplaced (13.749%); 8.9 MiB/s, 142 objects/s recovering 2024-04-29T14:44:43.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:43 smithi040 ceph-mon[39581]: osdmap e837: 12 total, 12 up, 10 in 2024-04-29T14:44:43.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:43 smithi194 ceph-mon[44089]: pgmap v2154: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 20842/151590 objects misplaced (13.749%); 8.9 MiB/s, 142 objects/s recovering 2024-04-29T14:44:43.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:43 smithi194 ceph-mon[44089]: osdmap e837: 12 total, 12 up, 10 in 2024-04-29T14:44:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:43 smithi121 ceph-mon[44533]: pgmap v2154: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 20842/151590 objects misplaced (13.749%); 8.9 MiB/s, 142 objects/s recovering 2024-04-29T14:44:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:43 smithi121 ceph-mon[44533]: osdmap e837: 12 total, 12 up, 10 in 2024-04-29T14:44:44.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:moved 7 / 34 (20.5882%) 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:avg 3.4 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.8 -> 1.56205 (expected baseline 1.74929) 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.6 with 1 -> 1 pgs (0.294118 -> 0.294118 * mean) 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.2 with 7 -> 6 pgs (2.05882 -> 1.76471 * mean) 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:44:45.313 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0173 2024-04-29T14:44:45.314 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0190 2024-04-29T14:44:45.314 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 weight 1.0000 -> 0.5800 2024-04-29T14:44:45.314 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 0.6520 -> 0.9188 2024-04-29T14:44:45.314 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 weight 1.0000 -> 0.7866 2024-04-29T14:44:45.314 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 weight 1.0000 -> 0.7945 2024-04-29T14:44:45.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:45 smithi194 ceph-mon[44089]: pgmap v2156: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 20424/151254 objects misplaced (13.503%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:44:45.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:45 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]: dispatch 2024-04-29T14:44:45.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:45 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]: dispatch 2024-04-29T14:44:45.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:45 smithi040 ceph-mon[39581]: pgmap v2156: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 20424/151254 objects misplaced (13.503%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:44:45.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:45 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]: dispatch 2024-04-29T14:44:45.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:45 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]: dispatch 2024-04-29T14:44:45.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:45 smithi121 ceph-mon[44533]: pgmap v2156: 17 pgs: 2 active+recovering+remapped, 2 active+recovering, 13 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 20424/151254 objects misplaced (13.503%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:44:45.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:45 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]: dispatch 2024-04-29T14:44:45.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:45 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]: dispatch 2024-04-29T14:44:46.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:46 smithi194 ceph-mon[44089]: from='client.27532 v1:172.21.15.40:0/1709578597' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:44:46.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:46 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]': finished 2024-04-29T14:44:46.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:46 smithi194 ceph-mon[44089]: osdmap e838: 12 total, 12 up, 10 in 2024-04-29T14:44:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:46 smithi040 ceph-mon[39581]: from='client.27532 v1:172.21.15.40:0/1709578597' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:44:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:46 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]': finished 2024-04-29T14:44:46.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:46 smithi040 ceph-mon[39581]: osdmap e838: 12 total, 12 up, 10 in 2024-04-29T14:44:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:46 smithi121 ceph-mon[44533]: from='client.27532 v1:172.21.15.40:0/1709578597' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:44:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:46 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'52068','2':'51551','7':'60214','10':'38009'}"}]': finished 2024-04-29T14:44:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:46 smithi121 ceph-mon[44533]: osdmap e838: 12 total, 12 up, 10 in 2024-04-29T14:44:47.038 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:47.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:47.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:47.177 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:47.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:47.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:47.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:47.330 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:47.355 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:47.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:47.459 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:44:47.488 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:44:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:47.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:44:47.618 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:47.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:47 smithi040 ceph-mon[39581]: pgmap v2158: 17 pgs: 2 peering, 2 unknown, 2 active+recovering+remapped, 2 active+recovering, 9 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 20424/141662 objects misplaced (14.417%) 2024-04-29T14:44:47.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:47 smithi040 ceph-mon[39581]: osdmap e839: 12 total, 12 up, 10 in 2024-04-29T14:44:47.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:47 smithi121 ceph-mon[44533]: pgmap v2158: 17 pgs: 2 peering, 2 unknown, 2 active+recovering+remapped, 2 active+recovering, 9 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 20424/141662 objects misplaced (14.417%) 2024-04-29T14:44:47.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:47 smithi121 ceph-mon[44533]: osdmap e839: 12 total, 12 up, 10 in 2024-04-29T14:44:47.643 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:47 smithi194 ceph-mon[44089]: pgmap v2158: 17 pgs: 2 peering, 2 unknown, 2 active+recovering+remapped, 2 active+recovering, 9 active+clean; 4.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 20424/141662 objects misplaced (14.417%) 2024-04-29T14:44:47.643 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:47 smithi194 ceph-mon[44089]: osdmap e839: 12 total, 12 up, 10 in 2024-04-29T14:44:47.645 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:47.665 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:47.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:47.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:47.799 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:47.827 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:47.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:47.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:47.969 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:47.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:48.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:48.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:44:48.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:48.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:44:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:48.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:44:48.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:44:48.301 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:48.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:48 smithi121 ceph-mon[44533]: osdmap e840: 12 total, 12 up, 10 in 2024-04-29T14:44:48.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:48 smithi121 ceph-mon[44533]: osdmap e841: 12 total, 12 up, 10 in 2024-04-29T14:44:48.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:48 smithi040 ceph-mon[39581]: osdmap e840: 12 total, 12 up, 10 in 2024-04-29T14:44:48.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:48 smithi040 ceph-mon[39581]: osdmap e841: 12 total, 12 up, 10 in 2024-04-29T14:44:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:44:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:48.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:44:48.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:48.692 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:48.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:48.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:48.822 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:48 smithi194 ceph-mon[44089]: osdmap e840: 12 total, 12 up, 10 in 2024-04-29T14:44:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:48 smithi194 ceph-mon[44089]: osdmap e841: 12 total, 12 up, 10 in 2024-04-29T14:44:48.851 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:48.875 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:48.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:48.980 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:49.007 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:49.027 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:49.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:49.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:49.158 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:49.178 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:49.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:49.286 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:44:49.315 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:49.338 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:44:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:49.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:44:49.443 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:49.479 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:49.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:49 smithi194 ceph-mon[44089]: pgmap v2162: 17 pgs: 1 remapped+peering, 4 peering, 1 unknown, 2 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 35 op/s; 13156/150604 objects misplaced (8.735%); 6.0 MiB/s, 105 objects/s recovering 2024-04-29T14:44:49.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:49 smithi194 ceph-mon[44089]: osdmap e842: 12 total, 12 up, 10 in 2024-04-29T14:44:49.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:49 smithi040 ceph-mon[39581]: pgmap v2162: 17 pgs: 1 remapped+peering, 4 peering, 1 unknown, 2 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 35 op/s; 13156/150604 objects misplaced (8.735%); 6.0 MiB/s, 105 objects/s recovering 2024-04-29T14:44:49.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:49 smithi040 ceph-mon[39581]: osdmap e842: 12 total, 12 up, 10 in 2024-04-29T14:44:49.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:49 smithi121 ceph-mon[44533]: pgmap v2162: 17 pgs: 1 remapped+peering, 4 peering, 1 unknown, 2 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 35 op/s; 13156/150604 objects misplaced (8.735%); 6.0 MiB/s, 105 objects/s recovering 2024-04-29T14:44:49.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:49 smithi121 ceph-mon[44533]: osdmap e842: 12 total, 12 up, 10 in 2024-04-29T14:44:50.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:50 smithi194 ceph-mon[44089]: osdmap e843: 12 total, 12 up, 10 in 2024-04-29T14:44:50.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:50 smithi194 ceph-mon[44089]: pgmap v2165: 17 pgs: 1 remapped+peering, 4 peering, 1 unknown, 3 active+recovering, 8 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 112 MiB/s wr, 2.45k op/s; 17770/150410 objects misplaced (11.814%); 55 MiB/s, 897 objects/s recovering 2024-04-29T14:44:50.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:50 smithi040 ceph-mon[39581]: osdmap e843: 12 total, 12 up, 10 in 2024-04-29T14:44:50.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:50 smithi040 ceph-mon[39581]: pgmap v2165: 17 pgs: 1 remapped+peering, 4 peering, 1 unknown, 3 active+recovering, 8 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 112 MiB/s wr, 2.45k op/s; 17770/150410 objects misplaced (11.814%); 55 MiB/s, 897 objects/s recovering 2024-04-29T14:44:50.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:50 smithi121 ceph-mon[44533]: osdmap e843: 12 total, 12 up, 10 in 2024-04-29T14:44:50.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:50 smithi121 ceph-mon[44533]: pgmap v2165: 17 pgs: 1 remapped+peering, 4 peering, 1 unknown, 3 active+recovering, 8 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 112 MiB/s wr, 2.45k op/s; 17770/150410 objects misplaced (11.814%); 55 MiB/s, 897 objects/s recovering 2024-04-29T14:44:50.939 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:50.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:51.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:44:51.072 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:51.096 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:44:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:51.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:44:51.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:51.237 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:51.348 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:44:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:51.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:44:51.390 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-29T14:44:51.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-04-29T14:44:51.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:44:51.484 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:51.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:44:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:51.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:44:51.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:44:51.638 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:51.660 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:44:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:51.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:44:51.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:44:51.804 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:53.232 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:44:53.232 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:44:53.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:53 smithi040 ceph-mon[39581]: pgmap v2166: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 22 KiB/s rd, 599 KiB/s wr, 97 op/s; 30835/150262 objects misplaced (20.521%); 13 MiB/s, 192 objects/s recovering 2024-04-29T14:44:53.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:53 smithi194 ceph-mon[44089]: pgmap v2166: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 22 KiB/s rd, 599 KiB/s wr, 97 op/s; 30835/150262 objects misplaced (20.521%); 13 MiB/s, 192 objects/s recovering 2024-04-29T14:44:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:53 smithi121 ceph-mon[44533]: pgmap v2166: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 22 KiB/s rd, 599 KiB/s wr, 97 op/s; 30835/150262 objects misplaced (20.521%); 13 MiB/s, 192 objects/s recovering 2024-04-29T14:44:54.928 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:44:55.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:55 smithi194 ceph-mon[44089]: pgmap v2167: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 16 KiB/s rd, 436 KiB/s wr, 71 op/s; 30607/150258 objects misplaced (20.370%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:44:55.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:55 smithi040 ceph-mon[39581]: pgmap v2167: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 16 KiB/s rd, 436 KiB/s wr, 71 op/s; 30607/150258 objects misplaced (20.370%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:44:55.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:55 smithi121 ceph-mon[44533]: pgmap v2167: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 16 KiB/s rd, 436 KiB/s wr, 71 op/s; 30607/150258 objects misplaced (20.370%); 11 MiB/s, 173 objects/s recovering 2024-04-29T14:44:55.971 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:44:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:44:55.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:44:56.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:44:56.117 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:44:57.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:57 smithi194 ceph-mon[44089]: pgmap v2168: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 14 KiB/s rd, 367 KiB/s wr, 62 op/s; 30056/150228 objects misplaced (20.007%); 14 MiB/s, 215 objects/s recovering 2024-04-29T14:44:57.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:57 smithi040 ceph-mon[39581]: pgmap v2168: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 14 KiB/s rd, 367 KiB/s wr, 62 op/s; 30056/150228 objects misplaced (20.007%); 14 MiB/s, 215 objects/s recovering 2024-04-29T14:44:57.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:57 smithi121 ceph-mon[44533]: pgmap v2168: 17 pgs: 1 active+recovering+remapped, 2 peering, 5 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 14 KiB/s rd, 367 KiB/s wr, 62 op/s; 30056/150228 objects misplaced (20.007%); 14 MiB/s, 215 objects/s recovering 2024-04-29T14:44:58.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:58 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 12465/150134 objects degraded (8.303%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:44:58.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:58 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 12465/150134 objects degraded (8.303%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:44:58.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:58 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 12465/150134 objects degraded (8.303%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:44:59.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:59 smithi040 ceph-mon[39581]: pgmap v2169: 17 pgs: 1 active+recovering+remapped, 1 activating+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 11 KiB/s rd, 302 KiB/s wr, 55 op/s; 12465/150134 objects degraded (8.303%); 33147/150134 objects misplaced (22.078%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:44:59.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:44:59.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:44:59.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2026893659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:59.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:44:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:59.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:59 smithi121 ceph-mon[44533]: pgmap v2169: 17 pgs: 1 active+recovering+remapped, 1 activating+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 11 KiB/s rd, 302 KiB/s wr, 55 op/s; 12465/150134 objects degraded (8.303%); 33147/150134 objects misplaced (22.078%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:44:59.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:44:59.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:44:59.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2026893659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:59.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:44:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:44:59.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:59 smithi194 ceph-mon[44089]: pgmap v2169: 17 pgs: 1 active+recovering+remapped, 1 activating+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 11 KiB/s rd, 302 KiB/s wr, 55 op/s; 12465/150134 objects degraded (8.303%); 33147/150134 objects misplaced (22.078%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:44:59.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:44:59.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:44:59.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2026893659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:44:59.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:44:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osdmap e844: 12 total, 12 up, 10 in 2024-04-29T14:45:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osdmap e844: 12 total, 12 up, 10 in 2024-04-29T14:45:00.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osdmap e844: 12 total, 12 up, 10 in 2024-04-29T14:45:00.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:00.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:45:01.083 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:45:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:01.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:45:01.190 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:45:01.218 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:01.242 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:45:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:01.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:45:01.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:45:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:01.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:45:01.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:45:01.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:45:01.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:45:01.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:45:01.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:45:01.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:45:01.478 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:45:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:45:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:45:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:01.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:45:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:01 smithi194 ceph-mon[44089]: pgmap v2171: 17 pgs: 1 active+recovering+remapped, 1 activating+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 11 KiB/s rd, 294 KiB/s wr, 50 op/s; 12465/150024 objects degraded (8.309%); 32911/150024 objects misplaced (21.937%); 16 MiB/s, 244 objects/s recovering 2024-04-29T14:45:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:01 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:01 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:01.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:45:01.615 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:01 smithi121 ceph-mon[44533]: pgmap v2171: 17 pgs: 1 active+recovering+remapped, 1 activating+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 11 KiB/s rd, 294 KiB/s wr, 50 op/s; 12465/150024 objects degraded (8.309%); 32911/150024 objects misplaced (21.937%); 16 MiB/s, 244 objects/s recovering 2024-04-29T14:45:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:01 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:01 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:01.637 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:45:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:01.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:45:01.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:45:01.773 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:01.797 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:45:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:01.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:45:01.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:01 smithi040 ceph-mon[39581]: pgmap v2171: 17 pgs: 1 active+recovering+remapped, 1 activating+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 11 KiB/s rd, 294 KiB/s wr, 50 op/s; 12465/150024 objects degraded (8.309%); 32911/150024 objects misplaced (21.937%); 16 MiB/s, 244 objects/s recovering 2024-04-29T14:45:01.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:01 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:01.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:01 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:01.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:45:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:01.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:45:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:01.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:45:02.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:45:02.082 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:45:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:02.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:45:02.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:45:02.249 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:02.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:02 smithi121 ceph-mon[44533]: osdmap e845: 12 total, 12 up, 10 in 2024-04-29T14:45:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:02 smithi194 ceph-mon[44089]: osdmap e845: 12 total, 12 up, 10 in 2024-04-29T14:45:02.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:02 smithi040 ceph-mon[39581]: osdmap e845: 12 total, 12 up, 10 in 2024-04-29T14:45:03.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:03 smithi194 ceph-mon[44089]: pgmap v2173: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 17 op/s; 12465/149980 objects degraded (8.311%); 31590/149980 objects misplaced (21.063%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:45:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:03 smithi040 ceph-mon[39581]: pgmap v2173: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 17 op/s; 12465/149980 objects degraded (8.311%); 31590/149980 objects misplaced (21.063%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:45:03.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:03 smithi121 ceph-mon[44533]: pgmap v2173: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 17 op/s; 12465/149980 objects degraded (8.311%); 31590/149980 objects misplaced (21.063%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:45:04.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:04 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 12465/149890 objects degraded (8.316%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:45:04.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:04 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 12465/149890 objects degraded (8.316%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:45:04.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:04 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 12465/149890 objects degraded (8.316%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:45:05.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:05 smithi194 ceph-mon[44089]: pgmap v2174: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 21 op/s; 12465/149890 objects degraded (8.316%); 31352/149890 objects misplaced (20.917%); 18 MiB/s, 289 objects/s recovering 2024-04-29T14:45:05.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:05 smithi040 ceph-mon[39581]: pgmap v2174: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 21 op/s; 12465/149890 objects degraded (8.316%); 31352/149890 objects misplaced (20.917%); 18 MiB/s, 289 objects/s recovering 2024-04-29T14:45:05.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:05 smithi121 ceph-mon[44533]: pgmap v2174: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 21 op/s; 12465/149890 objects degraded (8.316%); 31352/149890 objects misplaced (20.917%); 18 MiB/s, 289 objects/s recovering 2024-04-29T14:45:07.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:07 smithi194 ceph-mon[44089]: pgmap v2175: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 12465/149610 objects degraded (8.332%); 30798/149610 objects misplaced (20.586%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:45:07.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:07 smithi040 ceph-mon[39581]: pgmap v2175: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 12465/149610 objects degraded (8.332%); 30798/149610 objects misplaced (20.586%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:45:07.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:07 smithi121 ceph-mon[44533]: pgmap v2175: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 6 active+recovering, 9 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 12465/149610 objects degraded (8.332%); 30798/149610 objects misplaced (20.586%); 18 MiB/s, 287 objects/s recovering 2024-04-29T14:45:08.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:45:08.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:08 smithi040 ceph-mon[39581]: pgmap v2176: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 53 op/s; 37897/149078 objects misplaced (25.421%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:45:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:08 smithi194 ceph-mon[44089]: pgmap v2176: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 53 op/s; 37897/149078 objects misplaced (25.421%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:45:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:08 smithi121 ceph-mon[44533]: pgmap v2176: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 53 op/s; 37897/149078 objects misplaced (25.421%); 19 MiB/s, 303 objects/s recovering 2024-04-29T14:45:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:09 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12465/149610 objects degraded (8.332%), 1 pg degraded) 2024-04-29T14:45:09.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:09 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12465/149610 objects degraded (8.332%), 1 pg degraded) 2024-04-29T14:45:09.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:09 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12465/149610 objects degraded (8.332%), 1 pg degraded) 2024-04-29T14:45:10.261 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:45:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:10.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:45:10.368 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:45:10.396 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:10.421 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:45:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:10.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:45:10.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:45:10.554 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:10.573 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:45:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:10.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:45:10.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:45:10.715 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:10.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:10 smithi194 ceph-mon[44089]: pgmap v2177: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 37674/148494 objects misplaced (25.371%); 18 MiB/s, 291 objects/s recovering 2024-04-29T14:45:10.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:10 smithi040 ceph-mon[39581]: pgmap v2177: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 37674/148494 objects misplaced (25.371%); 18 MiB/s, 291 objects/s recovering 2024-04-29T14:45:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:10 smithi121 ceph-mon[44533]: pgmap v2177: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 37674/148494 objects misplaced (25.371%); 18 MiB/s, 291 objects/s recovering 2024-04-29T14:45:11.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:45:12.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:11 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:45:12.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:11 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:45:12.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:11 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:45:13.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:12 smithi194 ceph-mon[44089]: pgmap v2178: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 36421/147538 objects misplaced (24.686%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:45:13.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:12 smithi040 ceph-mon[39581]: pgmap v2178: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 36421/147538 objects misplaced (24.686%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:45:13.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:12 smithi121 ceph-mon[44533]: pgmap v2178: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 36421/147538 objects misplaced (24.686%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:45:14.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:13 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:14.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:13 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:14.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:13 smithi194 ceph-mon[44089]: osdmap e846: 12 total, 12 up, 10 in 2024-04-29T14:45:14.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:45:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:45:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:13 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:45:14.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:13 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:14.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:13 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:14.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:13 smithi040 ceph-mon[39581]: osdmap e846: 12 total, 12 up, 10 in 2024-04-29T14:45:14.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:45:14.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:45:14.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:13 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:45:14.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:13 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:14.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:13 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:14.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:13 smithi121 ceph-mon[44533]: osdmap e846: 12 total, 12 up, 10 in 2024-04-29T14:45:14.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:45:14.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:45:14.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:13 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:45:14.696 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:45:14.736 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:45:14.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:14.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:45:14.841 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:45:14.870 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:14.891 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:45:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:14.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:45:14.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:45:15.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:45:15.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:45:15.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:45:15.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: pgmap v2180: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 36304/147266 objects misplaced (24.652%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:45:15.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:45:15.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:45:15.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:45:15.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:14 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:45:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:45:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:45:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:45:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: pgmap v2180: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 36304/147266 objects misplaced (24.652%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:45:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:45:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:45:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:45:15.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:14 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:45:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:45:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:45:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:45:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: pgmap v2180: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 36304/147266 objects misplaced (24.652%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:45:15.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:45:15.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:45:15.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:45:15.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:14 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:45:17.067 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:45:17.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:17 smithi040 ceph-mon[39581]: pgmap v2181: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 136 op/s; 35731/146876 objects misplaced (24.327%); 17 MiB/s, 277 objects/s recovering 2024-04-29T14:45:17.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:17 smithi194 ceph-mon[44089]: pgmap v2181: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 136 op/s; 35731/146876 objects misplaced (24.327%); 17 MiB/s, 277 objects/s recovering 2024-04-29T14:45:17.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:17 smithi121 ceph-mon[44533]: pgmap v2181: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 136 op/s; 35731/146876 objects misplaced (24.327%); 17 MiB/s, 277 objects/s recovering 2024-04-29T14:45:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:18 smithi194 ceph-mon[44089]: osdmap e847: 12 total, 12 up, 10 in 2024-04-29T14:45:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:18 smithi194 ceph-mon[44089]: pgmap v2183: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 150 op/s; 34798/146084 objects misplaced (23.821%); 22 MiB/s, 354 objects/s recovering 2024-04-29T14:45:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:18 smithi040 ceph-mon[39581]: osdmap e847: 12 total, 12 up, 10 in 2024-04-29T14:45:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:18 smithi040 ceph-mon[39581]: pgmap v2183: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 150 op/s; 34798/146084 objects misplaced (23.821%); 22 MiB/s, 354 objects/s recovering 2024-04-29T14:45:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:18 smithi121 ceph-mon[44533]: osdmap e847: 12 total, 12 up, 10 in 2024-04-29T14:45:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:18 smithi121 ceph-mon[44533]: pgmap v2183: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 150 op/s; 34798/146084 objects misplaced (23.821%); 22 MiB/s, 354 objects/s recovering 2024-04-29T14:45:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:45:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:45:21.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:21.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:21.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:20 smithi194 ceph-mon[44089]: osdmap e848: 12 total, 12 up, 10 in 2024-04-29T14:45:21.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:20 smithi194 ceph-mon[44089]: pgmap v2185: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 34798/145912 objects misplaced (23.849%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:45:21.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2260758446' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:45:21.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:21.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:21.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:20 smithi040 ceph-mon[39581]: osdmap e848: 12 total, 12 up, 10 in 2024-04-29T14:45:21.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:20 smithi040 ceph-mon[39581]: pgmap v2185: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 34798/145912 objects misplaced (23.849%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:45:21.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2260758446' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:45:21.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:21.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:21.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:20 smithi121 ceph-mon[44533]: osdmap e848: 12 total, 12 up, 10 in 2024-04-29T14:45:21.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:20 smithi121 ceph-mon[44533]: pgmap v2185: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 34798/145912 objects misplaced (23.849%); 12 MiB/s, 199 objects/s recovering 2024-04-29T14:45:21.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2260758446' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:45:22.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:21 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:45:22.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2260758446' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:45:22.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:21 smithi194 ceph-mon[44089]: osdmap e849: 12 total, 12 up, 10 in 2024-04-29T14:45:22.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:21 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:45:22.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2260758446' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:45:22.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:21 smithi040 ceph-mon[39581]: osdmap e849: 12 total, 12 up, 10 in 2024-04-29T14:45:22.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:21 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:45:22.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2260758446' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:45:22.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:21 smithi121 ceph-mon[44533]: osdmap e849: 12 total, 12 up, 10 in 2024-04-29T14:45:23.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:22 smithi194 ceph-mon[44089]: pgmap v2187: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 181 op/s; 33410/144692 objects misplaced (23.090%); 24 MiB/s, 382 objects/s recovering 2024-04-29T14:45:23.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:22 smithi040 ceph-mon[39581]: pgmap v2187: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 181 op/s; 33410/144692 objects misplaced (23.090%); 24 MiB/s, 382 objects/s recovering 2024-04-29T14:45:23.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:22 smithi121 ceph-mon[44533]: pgmap v2187: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 181 op/s; 33410/144692 objects misplaced (23.090%); 24 MiB/s, 382 objects/s recovering 2024-04-29T14:45:24.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:24.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:24.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:24.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:24.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:23 smithi194 ceph-mon[44089]: osdmap e850: 12 total, 12 up, 10 in 2024-04-29T14:45:24.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:24.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:24.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:24.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:24.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:23 smithi040 ceph-mon[39581]: osdmap e850: 12 total, 12 up, 10 in 2024-04-29T14:45:24.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:24.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:24.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:24.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:24.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:23 smithi121 ceph-mon[44533]: osdmap e850: 12 total, 12 up, 10 in 2024-04-29T14:45:25.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:24 smithi194 ceph-mon[44089]: pgmap v2189: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 33312/144400 objects misplaced (23.069%); 15 MiB/s, 244 objects/s recovering 2024-04-29T14:45:25.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:24 smithi040 ceph-mon[39581]: pgmap v2189: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 33312/144400 objects misplaced (23.069%); 15 MiB/s, 244 objects/s recovering 2024-04-29T14:45:25.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:24 smithi121 ceph-mon[44533]: pgmap v2189: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 33312/144400 objects misplaced (23.069%); 15 MiB/s, 244 objects/s recovering 2024-04-29T14:45:27.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:27 smithi194 ceph-mon[44089]: pgmap v2190: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 32643/143994 objects misplaced (22.670%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:45:27.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:27 smithi040 ceph-mon[39581]: pgmap v2190: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 32643/143994 objects misplaced (22.670%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:45:27.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:27 smithi121 ceph-mon[44533]: pgmap v2190: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 32643/143994 objects misplaced (22.670%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:45:28.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:28 smithi194 ceph-mon[44089]: osdmap e851: 12 total, 12 up, 10 in 2024-04-29T14:45:28.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:28 smithi194 ceph-mon[44089]: pgmap v2192: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 31639/143384 objects misplaced (22.066%); 21 MiB/s, 331 objects/s recovering 2024-04-29T14:45:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:28 smithi040 ceph-mon[39581]: osdmap e851: 12 total, 12 up, 10 in 2024-04-29T14:45:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:28 smithi040 ceph-mon[39581]: pgmap v2192: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 31639/143384 objects misplaced (22.066%); 21 MiB/s, 331 objects/s recovering 2024-04-29T14:45:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:28 smithi121 ceph-mon[44533]: osdmap e851: 12 total, 12 up, 10 in 2024-04-29T14:45:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:28 smithi121 ceph-mon[44533]: pgmap v2192: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 31639/143384 objects misplaced (22.066%); 21 MiB/s, 331 objects/s recovering 2024-04-29T14:45:31.017 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:45:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:31.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:45:31.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:45:31.153 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:31.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:45:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:31.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:45:31.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:45:31.466 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:31.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:31 smithi194 ceph-mon[44089]: pgmap v2193: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 31639/143258 objects misplaced (22.085%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:45:31.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:31 smithi040 ceph-mon[39581]: pgmap v2193: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 31639/143258 objects misplaced (22.085%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:45:31.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:31 smithi121 ceph-mon[44533]: pgmap v2193: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 31639/143258 objects misplaced (22.085%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:45:31.928 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:45:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:31.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:45:32.036 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:45:32.066 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:32.087 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:45:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:32.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:45:32.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:45:32.230 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:32.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:32.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:32.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:32.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:32.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:32.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:33.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:33 smithi194 ceph-mon[44089]: pgmap v2194: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 30072/142610 objects misplaced (21.087%); 21 MiB/s, 332 objects/s recovering 2024-04-29T14:45:33.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:33 smithi194 ceph-mon[44089]: osdmap e852: 12 total, 12 up, 10 in 2024-04-29T14:45:33.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:33 smithi194 ceph-mon[44089]: osdmap e853: 12 total, 12 up, 10 in 2024-04-29T14:45:33.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:33 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:33.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:33 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:33.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:33 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:33.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:33 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:33 smithi040 ceph-mon[39581]: pgmap v2194: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 30072/142610 objects misplaced (21.087%); 21 MiB/s, 332 objects/s recovering 2024-04-29T14:45:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:33 smithi040 ceph-mon[39581]: osdmap e852: 12 total, 12 up, 10 in 2024-04-29T14:45:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:33 smithi040 ceph-mon[39581]: osdmap e853: 12 total, 12 up, 10 in 2024-04-29T14:45:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:33 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:33 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:33 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:33 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:33.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:33 smithi121 ceph-mon[44533]: pgmap v2194: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 30072/142610 objects misplaced (21.087%); 21 MiB/s, 332 objects/s recovering 2024-04-29T14:45:33.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:33 smithi121 ceph-mon[44533]: osdmap e852: 12 total, 12 up, 10 in 2024-04-29T14:45:33.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:33 smithi121 ceph-mon[44533]: osdmap e853: 12 total, 12 up, 10 in 2024-04-29T14:45:33.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:33 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:33.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:33 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:33.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:33 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:33 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:34.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:34 smithi194 ceph-mon[44089]: osdmap e854: 12 total, 12 up, 10 in 2024-04-29T14:45:34.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:34 smithi194 ceph-mon[44089]: pgmap v2198: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 80 op/s; 29882/142422 objects misplaced (20.981%); 18 MiB/s, 281 objects/s recovering 2024-04-29T14:45:34.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:34 smithi040 ceph-mon[39581]: osdmap e854: 12 total, 12 up, 10 in 2024-04-29T14:45:34.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:34 smithi040 ceph-mon[39581]: pgmap v2198: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 80 op/s; 29882/142422 objects misplaced (20.981%); 18 MiB/s, 281 objects/s recovering 2024-04-29T14:45:34.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:34 smithi121 ceph-mon[44533]: osdmap e854: 12 total, 12 up, 10 in 2024-04-29T14:45:34.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:34 smithi121 ceph-mon[44533]: pgmap v2198: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 80 op/s; 29882/142422 objects misplaced (20.981%); 18 MiB/s, 281 objects/s recovering 2024-04-29T14:45:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:45:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:37.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:45:37.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:45:37.315 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:37.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:37 smithi040 ceph-mon[39581]: pgmap v2199: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 28885/142096 objects misplaced (20.328%); 28 MiB/s, 447 objects/s recovering 2024-04-29T14:45:37.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:37 smithi194 ceph-mon[44089]: pgmap v2199: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 28885/142096 objects misplaced (20.328%); 28 MiB/s, 447 objects/s recovering 2024-04-29T14:45:37.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:37 smithi121 ceph-mon[44533]: pgmap v2199: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 96 op/s; 28885/142096 objects misplaced (20.328%); 28 MiB/s, 447 objects/s recovering 2024-04-29T14:45:38.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:45:38.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:38 smithi194 ceph-mon[44089]: osdmap e855: 12 total, 12 up, 10 in 2024-04-29T14:45:38.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:38 smithi194 ceph-mon[44089]: pgmap v2201: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 27972/141278 objects misplaced (19.799%); 21 MiB/s, 338 objects/s recovering 2024-04-29T14:45:38.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:38 smithi040 ceph-mon[39581]: osdmap e855: 12 total, 12 up, 10 in 2024-04-29T14:45:38.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:38 smithi040 ceph-mon[39581]: pgmap v2201: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 27972/141278 objects misplaced (19.799%); 21 MiB/s, 338 objects/s recovering 2024-04-29T14:45:38.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:38 smithi121 ceph-mon[44533]: osdmap e855: 12 total, 12 up, 10 in 2024-04-29T14:45:38.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:38 smithi121 ceph-mon[44533]: pgmap v2201: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 27972/141278 objects misplaced (19.799%); 21 MiB/s, 338 objects/s recovering 2024-04-29T14:45:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:39 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:39 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:39 smithi040 ceph-mon[39581]: osdmap e856: 12 total, 12 up, 10 in 2024-04-29T14:45:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:39 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:39 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:39 smithi194 ceph-mon[44089]: osdmap e856: 12 total, 12 up, 10 in 2024-04-29T14:45:39.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:39 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:39.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:39 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:39.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:39 smithi121 ceph-mon[44533]: osdmap e856: 12 total, 12 up, 10 in 2024-04-29T14:45:40.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:40 smithi194 ceph-mon[44089]: pgmap v2203: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 27972/141096 objects misplaced (19.825%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:45:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:45:40.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:40 smithi040 ceph-mon[39581]: pgmap v2203: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 27972/141096 objects misplaced (19.825%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:45:40.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:40 smithi121 ceph-mon[44533]: pgmap v2203: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 27972/141096 objects misplaced (19.825%); 17 MiB/s, 275 objects/s recovering 2024-04-29T14:45:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:41.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:45:41.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:45:41.204 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:43.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:43 smithi194 ceph-mon[44089]: pgmap v2204: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 127 op/s; 26160/140384 objects misplaced (18.635%); 28 MiB/s, 454 objects/s recovering 2024-04-29T14:45:43.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:43 smithi194 ceph-mon[44089]: osdmap e857: 12 total, 12 up, 10 in 2024-04-29T14:45:43.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:43 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:43.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:43 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:43 smithi040 ceph-mon[39581]: pgmap v2204: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 127 op/s; 26160/140384 objects misplaced (18.635%); 28 MiB/s, 454 objects/s recovering 2024-04-29T14:45:43.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:43 smithi040 ceph-mon[39581]: osdmap e857: 12 total, 12 up, 10 in 2024-04-29T14:45:43.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:43 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:43.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:43 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:43 smithi121 ceph-mon[44533]: pgmap v2204: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 127 op/s; 26160/140384 objects misplaced (18.635%); 28 MiB/s, 454 objects/s recovering 2024-04-29T14:45:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:43 smithi121 ceph-mon[44533]: osdmap e857: 12 total, 12 up, 10 in 2024-04-29T14:45:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:43 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:43 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:44.674 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:45:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:44.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:45:44.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:45:44.807 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:44 smithi121 ceph-mon[44533]: osdmap e858: 12 total, 12 up, 10 in 2024-04-29T14:45:44.807 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:44 smithi121 ceph-mon[44533]: pgmap v2207: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 90 op/s; 25965/140186 objects misplaced (18.522%); 20 MiB/s, 324 objects/s recovering 2024-04-29T14:45:44.809 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:44.822 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:44 smithi194 ceph-mon[44089]: osdmap e858: 12 total, 12 up, 10 in 2024-04-29T14:45:44.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:44 smithi194 ceph-mon[44089]: pgmap v2207: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 90 op/s; 25965/140186 objects misplaced (18.522%); 20 MiB/s, 324 objects/s recovering 2024-04-29T14:45:44.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:45:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:44.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:45:44.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:44 smithi040 ceph-mon[39581]: osdmap e858: 12 total, 12 up, 10 in 2024-04-29T14:45:44.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:44 smithi040 ceph-mon[39581]: pgmap v2207: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 90 op/s; 25965/140186 objects misplaced (18.522%); 20 MiB/s, 324 objects/s recovering 2024-04-29T14:45:44.935 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:45:44.962 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:44.985 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:45:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:44.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:45:45.092 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:45:45.120 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:45.142 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:45:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:45.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:45:45.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:45:45.274 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:45.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:45:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:45.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:45:45.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:45:45.433 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:45.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:45:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:45.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:45:45.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:45:45.593 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:45.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:45:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:45.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:45:45.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:45:45.752 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:45.775 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:45:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:45.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:45:45.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:45:45.908 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:45.931 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:45:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:45.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:45:46.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:45:46.078 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:46.445 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:45:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:46.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:45:46.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:45:46.582 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:46.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:45:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:46.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:45:46.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:45:46.738 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:46.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:45:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:46.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:45:46.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:45:46.892 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:46.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:45:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:46.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:45:47.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:45:47.049 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:47.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:45:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:47.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:45:47.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:45:47.208 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:47.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:45:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:47.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:45:47.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:45:47.384 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:47.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:47 smithi194 ceph-mon[44089]: pgmap v2208: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 88 op/s; 24912/139908 objects misplaced (17.806%); 24 MiB/s, 388 objects/s recovering 2024-04-29T14:45:47.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:47 smithi040 ceph-mon[39581]: pgmap v2208: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 88 op/s; 24912/139908 objects misplaced (17.806%); 24 MiB/s, 388 objects/s recovering 2024-04-29T14:45:47.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:47 smithi121 ceph-mon[44533]: pgmap v2208: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 88 op/s; 24912/139908 objects misplaced (17.806%); 24 MiB/s, 388 objects/s recovering 2024-04-29T14:45:48.184 INFO:teuthology.orchestra.run.smithi040.stdout:moved 1 / 34 (2.94118%) 2024-04-29T14:45:48.184 INFO:teuthology.orchestra.run.smithi040.stdout:avg 3.4 2024-04-29T14:45:48.184 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.56205 -> 1.74356 (expected baseline 1.74929) 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.6 with 1 -> 1 pgs (0.294118 -> 0.294118 * mean) 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.1 with 6 -> 6 pgs (1.76471 -> 1.76471 * mean) 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 1 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0178 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0196 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 weight 0.5800 -> 0.3544 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 weight 0.7945 -> 0.5708 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 0.9188 -> 1.0000 2024-04-29T14:45:48.185 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 weight 0.7866 -> 0.6861 2024-04-29T14:45:48.187 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:45:48.520 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:45:48 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:45:48.179+0000 7f4cb5abe700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-04-29T14:45:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:48 smithi194 ceph-mon[44089]: osdmap e859: 12 total, 12 up, 10 in 2024-04-29T14:45:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:48 smithi194 ceph-mon[44089]: from='client.27550 v1:172.21.15.40:0/3739299028' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:45:48.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:48 smithi194 ceph-mon[44089]: pgmap v2210: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 90 op/s; 23756/139298 objects misplaced (17.054%); 24 MiB/s, 386 objects/s recovering 2024-04-29T14:45:48.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:48 smithi040 ceph-mon[39581]: osdmap e859: 12 total, 12 up, 10 in 2024-04-29T14:45:48.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:48 smithi040 ceph-mon[39581]: from='client.27550 v1:172.21.15.40:0/3739299028' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:45:48.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:48 smithi040 ceph-mon[39581]: pgmap v2210: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 90 op/s; 23756/139298 objects misplaced (17.054%); 24 MiB/s, 386 objects/s recovering 2024-04-29T14:45:48.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:48 smithi121 ceph-mon[44533]: osdmap e859: 12 total, 12 up, 10 in 2024-04-29T14:45:48.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:48 smithi121 ceph-mon[44533]: from='client.27550 v1:172.21.15.40:0/3739299028' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:45:48.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:48 smithi121 ceph-mon[44533]: pgmap v2210: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 90 op/s; 23756/139298 objects misplaced (17.054%); 24 MiB/s, 386 objects/s recovering 2024-04-29T14:45:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:45:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:49.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:45:49.252 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:45:49.280 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:49.304 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:45:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:49.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:45:49.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:45:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:50.265 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:45:50.265 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130718 7.2 7.96bde682 (undecoded) ondisk+write+known_if_redirected e748)", 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:01.317107+0000", 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 348.94227753299998, 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.747173533, 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:45:50.266 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130718 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317107+0000", 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317107+0000", 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317109+0000", 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8600000000000001e-06 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317117+0000", 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9389999999999997e-06 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317134+0000", 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6679000000000001e-05 2024-04-29T14:45:50.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317145+0000", 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0941000000000001e-05 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317214+0000", 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9047999999999993e-05 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.317223+0000", 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3510000000000008e-06 2024-04-29T14:45:50.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:02.064270+0000", 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.74704659699999998 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:02.064280+0000", 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0118e-05 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.270 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130726 7.2 7.c750ee52 (undecoded) ondisk+write+known_if_redirected e748)", 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:01.882812+0000", 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 348.37657218800001, 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18149499599999999, 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130726 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882812+0000", 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882812+0000", 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882816+0000", 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4190000000000004e-06 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882829+0000", 2024-04-29T14:45:50.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2639e-05 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882854+0000", 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.472e-05 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882868+0000", 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3896999999999999e-05 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882950+0000", 2024-04-29T14:45:50.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2520000000000003e-05 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:01.882963+0000", 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3078000000000001e-05 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:02.064301+0000", 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.18133745400000001 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:02.064307+0000", 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2689999999999996e-06 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130726 7.2 e750/749 7:4a770ae3:::benchmark_data_smithi194_147200_object15814:head v 750'8067, mlcod=0'0)", 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.436729+0000", 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.82265524100001, 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038995646000000002, 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436729+0000", 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436729+0000", 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436732+0000", 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.41e-06 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.276 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436764+0000", 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1826000000000003e-05 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436777+0000", 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3254e-05 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436788+0000", 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1063000000000001e-05 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.441072+0000", 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004283472 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.441100+0000", 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8174000000000002e-05 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.475669+0000", 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034569415999999999 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.475724+0000", 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5031000000000002e-05 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130761 7.2 e750/749 7:4e175b1d:::benchmark_data_smithi194_147200_object15849:head v 750'8068, mlcod=0'0)", 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.446653+0000", 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.81273154100001, 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029108794, 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.279 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446653+0000", 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446653+0000", 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446653+0000", 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.23e-07 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446661+0000", 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0930000000000003e-06 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446671+0000", 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2029999999999998e-06 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446677+0000", 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5869999999999998e-06 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.456076+0000", 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0093984499999999992 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.456103+0000", 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.692e-05 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.475728+0000", 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019625462999999999 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.475761+0000", 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3254999999999999e-05 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130764 7.2 e750/749 7:4494c112:::benchmark_data_smithi194_147200_object15852:head v 750'8069, mlcod=0'0)", 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.446691+0000", 2024-04-29T14:45:50.283 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.81269324599998, 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029097964, 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446691+0000", 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446691+0000", 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446691+0000", 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1399999999999997e-07 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.446696+0000", 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6129999999999997e-06 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.447414+0000", 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071786400000000002 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.447421+0000", 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.627e-06 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.466112+0000", 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018690695 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.466140+0000", 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7778000000000001e-05 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.475767+0000", 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0096267520000000006 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.475789+0000", 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2121e-05 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130722 7.6 e750/749 7:62be359e:::benchmark_data_smithi194_147200_object15810:head v 750'7984, mlcod=0'0)", 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.520369+0000", 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.73901518700001, 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030859850000000001, 2024-04-29T14:45:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.520369+0000", 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.520369+0000", 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.520370+0000", 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.139e-06 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.520378+0000", 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2020000000000004e-06 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.520386+0000", 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4649999999999997e-06 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.520391+0000", 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7810000000000002e-06 2024-04-29T14:45:50.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.523433+0000", 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00304186 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.523463+0000", 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0153000000000001e-05 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.551183+0000", 2024-04-29T14:45:50.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027719384 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.551229+0000", 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5865999999999998e-05 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130750 7.6 e750/749 7:65bbffe6:::benchmark_data_smithi194_147200_object15838:head v 750'7986, mlcod=0'0)", 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.534844+0000", 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.72453976399999, 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033816483000000001, 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.534844+0000", 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.534844+0000", 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.534846+0000", 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.407e-06 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.534856+0000", 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9650000000000003e-06 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.534866+0000", 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0121e-05 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.534873+0000", 2024-04-29T14:45:50.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3660000000000003e-06 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.551290+0000", 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016416429999999999 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.551312+0000", 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2262000000000001e-05 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.568617+0000", 2024-04-29T14:45:50.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017305589999999999 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.568661+0000", 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3341999999999999e-05 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130826 7.3 e750/749 7:c87a878e:::benchmark_data_smithi194_147200_object15914:head v 750'8177, mlcod=660'972)", 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.640275+0000", 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.61910875699999, 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025402932999999999, 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640275+0000", 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640275+0000", 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640277+0000", 2024-04-29T14:45:50.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1710000000000001e-06 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640287+0000", 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9150000000000008e-06 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640303+0000", 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5481e-05 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640311+0000", 2024-04-29T14:45:50.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7910000000000005e-06 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640366+0000", 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5621000000000002e-05 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.640386+0000", 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9258000000000001e-05 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.665638+0000", 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025252 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.665678+0000", 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0695999999999998e-05 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130833 7.1 e750/748 7:842b1558:::benchmark_data_smithi194_147200_object15921:head v 750'8087, mlcod=660'1052)", 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.680537+0000", 2024-04-29T14:45:50.299 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.57884680199999, 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029840047000000001, 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680537+0000", 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680537+0000", 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680540+0000", 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7460000000000001e-06 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680552+0000", 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1664e-05 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680562+0000", 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0686000000000001e-05 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680570+0000", 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6729999999999992e-06 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680714+0000", 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014365999999999999 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.680745+0000", 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1018e-05 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.710354+0000", 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029609404999999998 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.710377+0000", 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3195000000000001e-05 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130852 7.3 e750/749 7:cf435e8a:::benchmark_data_smithi194_147200_object15940:head v 750'8179, mlcod=660'975)", 2024-04-29T14:45:50.303 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.748006+0000", 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.51137803699999, 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025221638000000001, 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.748006+0000", 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.748006+0000", 2024-04-29T14:45:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.748008+0000", 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2989999999999999e-06 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.748020+0000", 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1877e-05 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.748038+0000", 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8136999999999999e-05 2024-04-29T14:45:50.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.748048+0000", 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0023e-05 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.751881+0000", 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0038331149999999998 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.751906+0000", 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4360000000000001e-05 2024-04-29T14:45:50.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.773175+0000", 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021268865000000001 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.773228+0000", 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2961999999999999e-05 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130961 7.6 e751/749 7:6147ead0:::benchmark_data_smithi194_147200_object16049:head v 751'8000, mlcod=660'979)", 2024-04-29T14:45:50.307 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.165945+0000", 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.09343946400003, 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033121713999999997, 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.165945+0000", 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.165945+0000", 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.165946+0000", 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.545e-06 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.165956+0000", 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8810000000000008e-06 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.165966+0000", 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0074999999999999e-05 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.165974+0000", 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4449999999999999e-06 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.166082+0000", 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108482 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.166107+0000", 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5072e-05 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.199005+0000", 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.03289798 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.199066+0000", 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1234000000000004e-05 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130987 7.1 e751/748 7:8983fc2b:::benchmark_data_smithi194_147200_object16075:head v 751'8097, mlcod=660'1067)", 2024-04-29T14:45:50.311 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.217892+0000", 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.04149161300001, 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024263541999999999, 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.217892+0000", 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.217892+0000", 2024-04-29T14:45:50.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.217893+0000", 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0999999999999998e-07 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.217900+0000", 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4390000000000003e-06 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.217907+0000", 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0800000000000002e-06 2024-04-29T14:45:50.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.217911+0000", 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8969999999999999e-06 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.221466+0000", 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003554565 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.221490+0000", 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3774999999999999e-05 2024-04-29T14:45:50.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.242118+0000", 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020627870999999999 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.242156+0000", 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8305e-05 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130989 7.1 e751/748 7:885905a7:::benchmark_data_smithi194_147200_object16077:head v 751'8098, mlcod=660'1067)", 2024-04-29T14:45:50.315 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.218443+0000", 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 346.04094109300001, 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032345350000000002, 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.218443+0000", 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.218443+0000", 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.218444+0000", 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2099999999999995e-07 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.218451+0000", 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.294e-06 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.220506+0000", 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002055065 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.220523+0000", 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7095000000000001e-05 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.232110+0000", 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011586472 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.232140+0000", 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0386e-05 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.250727+0000", 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018586565999999999 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.250788+0000", 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1650999999999996e-05 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130997 7.3 e751/749 7:c290aa8e:::benchmark_data_smithi194_147200_object16085:head v 751'8187, mlcod=660'989)", 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.276067+0000", 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.983317355, 2024-04-29T14:45:50.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029589761999999999, 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276067+0000", 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276067+0000", 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276070+0000", 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8439999999999998e-06 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276082+0000", 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2784e-05 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276093+0000", 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.111e-05 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276102+0000", 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8200000000000003e-06 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276171+0000", 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8365e-05 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276199+0000", 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8050000000000001e-05 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.305597+0000", 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029397975999999999 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.305656+0000", 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9812999999999998e-05 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:130998 7.3 e751/749 7:c5c56e97:::benchmark_data_smithi194_147200_object16086:head v 751'8188, mlcod=660'989)", 2024-04-29T14:45:50.323 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.276198+0000", 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.98318619600002, 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029487458000000001, 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276198+0000", 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276198+0000", 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276198+0000", 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8799999999999998e-07 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276202+0000", 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6349999999999999e-06 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276206+0000", 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.765e-06 2024-04-29T14:45:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276209+0000", 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0450000000000001e-06 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.285483+0000", 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0092744179999999996 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.285509+0000", 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5565e-05 2024-04-29T14:45:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.305659+0000", 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020150543999999999 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.305685+0000", 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6098e-05 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.3 751/749 [PushOp(7:c131d302:::benchmark_data_smithi194_147200_object16175:head, version: 660'990, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:c131d302:::benchmark_data_smithi194_147200_object16175:head@660'990, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:45:50.327 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.276213+0000", 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.983170917, 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029376143, 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276213+0000", 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276213+0000", 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276213+0000", 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2399999999999999e-07 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276240+0000", 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6757000000000001e-05 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276247+0000", 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2899999999999997e-06 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.276255+0000", 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6229999999999997e-06 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.302387+0000", 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026132275 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.302396+0000", 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3590000000000008e-06 2024-04-29T14:45:50.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.305589+0000", 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0031936149999999999 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:131027 7.c e751/748 7:350e4ef7:::benchmark_data_smithi194_147200_object16115:head v 751'8183, mlcod=660'1073)", 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.380771+0000", 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.87861308999999, 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040063867000000003, 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380771+0000", 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380771+0000", 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380774+0000", 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7470000000000001e-06 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380790+0000", 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5962999999999999e-05 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380808+0000", 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8661999999999999e-05 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380819+0000", 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0876000000000001e-05 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380900+0000", 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0957000000000004e-05 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.380930+0000", 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9773000000000001e-05 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.420802+0000", 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039872512999999998 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.420835+0000", 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2376000000000003e-05 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.c 751/748 [PushOp(7:35af752c:::benchmark_data_smithi194_147200_object17330:head, version: 660'1088, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:35af752c:::benchmark_data_smithi194_147200_object17330:head@660'1088, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.390002+0000", 2024-04-29T14:45:50.335 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.86938186399999, 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030835141, 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390002+0000", 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390002+0000", 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390003+0000", 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8999999999999996e-07 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390048+0000", 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5167000000000003e-05 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390061+0000", 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3166999999999999e-05 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390074+0000", 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2452999999999999e-05 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.416727+0000", 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026653364999999998 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.416736+0000", 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2429999999999994e-06 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.420837+0000", 2024-04-29T14:45:50.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041010559999999996 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:131031 7.c e751/748 7:32ab85b9:::benchmark_data_smithi194_147200_object16119:head v 751'8184, mlcod=660'1073)", 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.390091+0000", 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.86929316700002, 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053533889000000001, 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390091+0000", 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390091+0000", 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390091+0000", 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5999999999999999e-07 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390097+0000", 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2610000000000001e-06 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390102+0000", 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1660000000000002e-06 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390106+0000", 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.561e-06 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.420860+0000", 2024-04-29T14:45:50.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030753775000000001 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.420880+0000", 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0109e-05 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.443602+0000", 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022721743999999999 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.443625+0000", 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2813e-05 2024-04-29T14:45:50.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:131032 7.c e751/748 7:3e15d407:::benchmark_data_smithi194_147200_object16120:head v 751'8185, mlcod=660'1073)", 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:04.390281+0000", 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.86910277800001, 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.053557682000000002, 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390281+0000", 2024-04-29T14:45:50.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390281+0000", 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390282+0000", 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8899999999999999e-07 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390288+0000", 2024-04-29T14:45:50.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9179999999999999e-06 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390299+0000", 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1133e-05 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.390305+0000", 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0480000000000004e-06 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.429088+0000", 2024-04-29T14:45:50.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038783020000000001 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.429110+0000", 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1729000000000002e-05 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.443805+0000", 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014695311000000001 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:04.443839+0000", 2024-04-29T14:45:50.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3834000000000002e-05 2024-04-29T14:45:50.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.347 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:45:50.347 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:45:50.347 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:45:51.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:51 smithi194 ceph-mon[44089]: pgmap v2211: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 23756/139180 objects misplaced (17.069%); 19 MiB/s, 299 objects/s recovering 2024-04-29T14:45:51.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:51 smithi040 ceph-mon[39581]: pgmap v2211: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 23756/139180 objects misplaced (17.069%); 19 MiB/s, 299 objects/s recovering 2024-04-29T14:45:51.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:51 smithi121 ceph-mon[44533]: pgmap v2211: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 77 op/s; 23756/139180 objects misplaced (17.069%); 19 MiB/s, 299 objects/s recovering 2024-04-29T14:45:53.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:53 smithi194 ceph-mon[44089]: pgmap v2212: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 22287/138740 objects misplaced (16.064%); 26 MiB/s, 486 objects/s recovering 2024-04-29T14:45:53.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:53 smithi040 ceph-mon[39581]: pgmap v2212: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 22287/138740 objects misplaced (16.064%); 26 MiB/s, 486 objects/s recovering 2024-04-29T14:45:53.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:53 smithi121 ceph-mon[44533]: pgmap v2212: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 22287/138740 objects misplaced (16.064%); 26 MiB/s, 486 objects/s recovering 2024-04-29T14:45:55.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:55 smithi194 ceph-mon[44089]: pgmap v2213: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 21994/138396 objects misplaced (15.892%); 24 MiB/s, 447 objects/s recovering 2024-04-29T14:45:55.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:55 smithi040 ceph-mon[39581]: pgmap v2213: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 21994/138396 objects misplaced (15.892%); 24 MiB/s, 447 objects/s recovering 2024-04-29T14:45:55.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:55 smithi121 ceph-mon[44533]: pgmap v2213: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 21994/138396 objects misplaced (15.892%); 24 MiB/s, 447 objects/s recovering 2024-04-29T14:45:56.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:56 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:45:56.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:56 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:45:56.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:56 smithi040 ceph-mon[39581]: osdmap e860: 12 total, 12 up, 10 in 2024-04-29T14:45:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:56 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:45:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:56 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:45:56.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:56 smithi121 ceph-mon[44533]: osdmap e860: 12 total, 12 up, 10 in 2024-04-29T14:45:56.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:56 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:45:56.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:56 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:45:56.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:56 smithi194 ceph-mon[44089]: osdmap e860: 12 total, 12 up, 10 in 2024-04-29T14:45:57.559 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-29T14:45:57.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-04-29T14:45:57.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:57 smithi040 ceph-mon[39581]: pgmap v2215: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 21596/138108 objects misplaced (15.637%); 16 MiB/s, 461 objects/s recovering 2024-04-29T14:45:57.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:57 smithi121 ceph-mon[44533]: pgmap v2215: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 21596/138108 objects misplaced (15.637%); 16 MiB/s, 461 objects/s recovering 2024-04-29T14:45:57.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:57 smithi194 ceph-mon[44089]: pgmap v2215: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 21596/138108 objects misplaced (15.637%); 16 MiB/s, 461 objects/s recovering 2024-04-29T14:45:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:58 smithi194 ceph-mon[44089]: osdmap e861: 12 total, 12 up, 10 in 2024-04-29T14:45:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:45:58 smithi194 ceph-mon[44089]: pgmap v2217: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 20564/137458 objects misplaced (14.960%); 24 MiB/s, 607 objects/s recovering 2024-04-29T14:45:58.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:58 smithi040 ceph-mon[39581]: osdmap e861: 12 total, 12 up, 10 in 2024-04-29T14:45:58.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:45:58 smithi040 ceph-mon[39581]: pgmap v2217: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 20564/137458 objects misplaced (14.960%); 24 MiB/s, 607 objects/s recovering 2024-04-29T14:45:58.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:58 smithi121 ceph-mon[44533]: osdmap e861: 12 total, 12 up, 10 in 2024-04-29T14:45:58.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:45:58 smithi121 ceph-mon[44533]: pgmap v2217: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 20564/137458 objects misplaced (14.960%); 24 MiB/s, 607 objects/s recovering 2024-04-29T14:45:59.149 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:45:59.159 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:45:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:59.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:45:59.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:45:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:59.314 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:45:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:59.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:45:59.419 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:45:59.445 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:59.466 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:45:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:59.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:45:59.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:45:59.602 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:59.636 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:45:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:59.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:45:59.740 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:45:59.768 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:59.789 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:45:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:59.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:45:59.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:45:59.919 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:45:59.941 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:45:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:45:59.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:00.046 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:00.073 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:00.092 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:00.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:00.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:00.230 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:01.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:01 smithi194 ceph-mon[44089]: pgmap v2218: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 20337/137208 objects misplaced (14.822%); 14 MiB/s, 368 objects/s recovering 2024-04-29T14:46:01.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:01 smithi040 ceph-mon[39581]: pgmap v2218: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 20337/137208 objects misplaced (14.822%); 14 MiB/s, 368 objects/s recovering 2024-04-29T14:46:01.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:01 smithi121 ceph-mon[44533]: pgmap v2218: 17 pgs: 1 active+recovering+remapped, 7 active+recovering, 9 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 20337/137208 objects misplaced (14.822%); 14 MiB/s, 368 objects/s recovering 2024-04-29T14:46:02.294 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:02.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:02.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:02.438 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osdmap e862: 12 total, 12 up, 10 in 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1340546637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:02.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:02.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osdmap e862: 12 total, 12 up, 10 in 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1340546637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osdmap e862: 12 total, 12 up, 10 in 2024-04-29T14:46:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1340546637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:03.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:46:03.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:46:03.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:46:03.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:46:03.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:46:03.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:46:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:46:03.130 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:46:03.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:03.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:03.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:03 smithi040 ceph-mon[39581]: pgmap v2220: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 19655/136614 objects misplaced (14.387%); 20 MiB/s, 825 objects/s recovering 2024-04-29T14:46:03.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:03 smithi040 ceph-mon[39581]: osdmap e863: 12 total, 12 up, 10 in 2024-04-29T14:46:03.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:03.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:03.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:03 smithi121 ceph-mon[44533]: pgmap v2220: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 19655/136614 objects misplaced (14.387%); 20 MiB/s, 825 objects/s recovering 2024-04-29T14:46:03.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:03 smithi121 ceph-mon[44533]: osdmap e863: 12 total, 12 up, 10 in 2024-04-29T14:46:03.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:03.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:03.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:03 smithi194 ceph-mon[44089]: pgmap v2220: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 19655/136614 objects misplaced (14.387%); 20 MiB/s, 825 objects/s recovering 2024-04-29T14:46:03.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:03 smithi194 ceph-mon[44089]: osdmap e863: 12 total, 12 up, 10 in 2024-04-29T14:46:05.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:05 smithi040 ceph-mon[39581]: pgmap v2222: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 92 op/s; 19381/136208 objects misplaced (14.229%); 10 MiB/s, 506 objects/s recovering 2024-04-29T14:46:05.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:05 smithi121 ceph-mon[44533]: pgmap v2222: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 92 op/s; 19381/136208 objects misplaced (14.229%); 10 MiB/s, 506 objects/s recovering 2024-04-29T14:46:05.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:05 smithi194 ceph-mon[44089]: pgmap v2222: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 92 op/s; 19381/136208 objects misplaced (14.229%); 10 MiB/s, 506 objects/s recovering 2024-04-29T14:46:05.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:46:06.313 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:07.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:07.200 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:07.226 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:07.245 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:07.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:07.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:07.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:07 smithi040 ceph-mon[39581]: pgmap v2223: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 19381/135810 objects misplaced (14.271%); 8.5 MiB/s, 558 objects/s recovering 2024-04-29T14:46:07.384 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:07.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:07 smithi194 ceph-mon[44089]: pgmap v2223: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 19381/135810 objects misplaced (14.271%); 8.5 MiB/s, 558 objects/s recovering 2024-04-29T14:46:07.589 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:07.592 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:07 smithi121 ceph-mon[44533]: pgmap v2223: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 19381/135810 objects misplaced (14.271%); 8.5 MiB/s, 558 objects/s recovering 2024-04-29T14:46:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:07.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:07.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:07.725 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:08.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:08.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:08 smithi121 ceph-mon[44533]: pgmap v2224: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 120 op/s; 19261/135274 objects misplaced (14.239%); 7.7 MiB/s, 719 objects/s recovering 2024-04-29T14:46:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:08.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:08 smithi194 ceph-mon[44089]: pgmap v2224: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 120 op/s; 19261/135274 objects misplaced (14.239%); 7.7 MiB/s, 719 objects/s recovering 2024-04-29T14:46:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:08 smithi040 ceph-mon[39581]: pgmap v2224: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 120 op/s; 19261/135274 objects misplaced (14.239%); 7.7 MiB/s, 719 objects/s recovering 2024-04-29T14:46:08.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:08.924 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:09.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:09.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:09.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:09.508 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:09.508 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:10.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:46:10.696 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:11.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:11.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:11.517 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:11.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:11.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:11 smithi194 ceph-mon[44089]: pgmap v2225: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 19055/134984 objects misplaced (14.116%); 8.2 MiB/s, 460 objects/s recovering 2024-04-29T14:46:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:11 smithi040 ceph-mon[39581]: pgmap v2225: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 19055/134984 objects misplaced (14.116%); 8.2 MiB/s, 460 objects/s recovering 2024-04-29T14:46:11.622 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:11.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:11 smithi121 ceph-mon[44533]: pgmap v2225: 17 pgs: 1 active+recovering+remapped, 6 active+recovering, 10 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 19055/134984 objects misplaced (14.116%); 8.2 MiB/s, 460 objects/s recovering 2024-04-29T14:46:11.648 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:11.667 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:11.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:11.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:11.804 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:11.883 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:12.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:12.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:12.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:12.296 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:12.447 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:12.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:12.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:12.596 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:12.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:12.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:12.671 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:12.744 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:12.770 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:12.789 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:12.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:12.893 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:12.919 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:12.938 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:12.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:13.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:13.073 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:13.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:13 smithi121 ceph-mon[44533]: pgmap v2226: 17 pgs: 1 active+recovering+remapped, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 19055/134238 objects misplaced (14.195%); 3.0 MiB/s, 496 objects/s recovering 2024-04-29T14:46:13.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:46:13.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:13 smithi194 ceph-mon[44089]: pgmap v2226: 17 pgs: 1 active+recovering+remapped, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 19055/134238 objects misplaced (14.195%); 3.0 MiB/s, 496 objects/s recovering 2024-04-29T14:46:13.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:13 smithi040 ceph-mon[39581]: pgmap v2226: 17 pgs: 1 active+recovering+remapped, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 19055/134238 objects misplaced (14.195%); 3.0 MiB/s, 496 objects/s recovering 2024-04-29T14:46:13.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:13.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:13.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:46:13.839 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:13.865 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:13.885 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:13.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:13.959 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:13.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:14.028 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:14.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:14.141 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:14.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:14.212 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:14.289 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:14.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:14.329 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:46:14.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:14.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:14.460 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1711807617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:46:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:14.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:14.563 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:14.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1711807617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:46:14.588 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:14.606 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:14.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:14.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1711807617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:46:14.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:14.743 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:14.995 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:14.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:15.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:15.132 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:15.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:15 smithi121 ceph-mon[44533]: pgmap v2227: 17 pgs: 1 active+recovering+remapped, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 18738/133814 objects misplaced (14.003%); 3.7 MiB/s, 441 objects/s recovering 2024-04-29T14:46:15.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:15 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:15.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1711807617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:46:15.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:15 smithi121 ceph-mon[44533]: osdmap e864: 12 total, 12 up, 10 in 2024-04-29T14:46:15.316 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:15 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:46:15.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:15 smithi040 ceph-mon[39581]: pgmap v2227: 17 pgs: 1 active+recovering+remapped, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 18738/133814 objects misplaced (14.003%); 3.7 MiB/s, 441 objects/s recovering 2024-04-29T14:46:15.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:15 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:15.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1711807617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:46:15.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:15 smithi040 ceph-mon[39581]: osdmap e864: 12 total, 12 up, 10 in 2024-04-29T14:46:15.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:15 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:46:15.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:15 smithi194 ceph-mon[44089]: pgmap v2227: 17 pgs: 1 active+recovering+remapped, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 18738/133814 objects misplaced (14.003%); 3.7 MiB/s, 441 objects/s recovering 2024-04-29T14:46:15.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:15 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:15.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1711807617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:46:15.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:15 smithi194 ceph-mon[44089]: osdmap e864: 12 total, 12 up, 10 in 2024-04-29T14:46:15.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:15 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:46:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:15.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:15.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:15.854 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:15.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:15.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:moved 1 / 34 (2.94118%) 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:avg 3.4 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.56205 -> 1.74356 (expected baseline 1.74929) 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.6 with 1 -> 1 pgs (0.294118 -> 0.294118 * mean) 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.1 with 6 -> 6 pgs (1.76471 -> 1.76471 * mean) 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 3 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0177 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0195 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 weight 0.5800 -> 0.3558 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 weight 0.7945 -> 0.5541 2024-04-29T14:46:16.105 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 0.9188 -> 1.0000 2024-04-29T14:46:16.106 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 weight 0.7866 -> 0.7054 2024-04-29T14:46:16.108 INFO:teuthology.orchestra.run.smithi040.stderr:no change 2024-04-29T14:46:16.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:16.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:16.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:16.372 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 14:46:16 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T14:46:16.101+0000 7f4cb5abe700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-04-29T14:46:16.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:16 smithi040 ceph-mon[39581]: osdmap e865: 12 total, 12 up, 10 in 2024-04-29T14:46:16.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:16.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:16.376 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:16.394 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:16.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:16.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:16.498 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:16.534 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:16.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:16 smithi121 ceph-mon[44533]: osdmap e865: 12 total, 12 up, 10 in 2024-04-29T14:46:16.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:16.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:16 smithi194 ceph-mon[44089]: osdmap e865: 12 total, 12 up, 10 in 2024-04-29T14:46:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:16.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:16.958 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-29T14:46:16.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-04-29T14:46:16.991 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:16.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:17.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:46:17.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:17.132 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:17.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:46:17.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:17 smithi121 ceph-mon[44533]: from='client.37319 v1:172.21.15.40:0/1672487410' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:46:17.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:17 smithi121 ceph-mon[44533]: pgmap v2230: 17 pgs: 1 peering, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 18738/133484 objects misplaced (14.038%); 3.4 MiB/s, 375 objects/s recovering 2024-04-29T14:46:17.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:17 smithi121 ceph-mon[44533]: osdmap e866: 12 total, 12 up, 10 in 2024-04-29T14:46:17.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:17 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:46:17.327 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:17 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:46:17.435 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:17.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:17 smithi040 ceph-mon[39581]: from='client.37319 v1:172.21.15.40:0/1672487410' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:46:17.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:17 smithi040 ceph-mon[39581]: pgmap v2230: 17 pgs: 1 peering, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 18738/133484 objects misplaced (14.038%); 3.4 MiB/s, 375 objects/s recovering 2024-04-29T14:46:17.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:17 smithi040 ceph-mon[39581]: osdmap e866: 12 total, 12 up, 10 in 2024-04-29T14:46:17.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:17 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:46:17.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:17 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:46:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:17.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:17.581 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:17.593 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:17.596 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:17.621 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:17 smithi194 ceph-mon[44089]: from='client.37319 v1:172.21.15.40:0/1672487410' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:46:17.621 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:17 smithi194 ceph-mon[44089]: pgmap v2230: 17 pgs: 1 peering, 5 active+recovering, 11 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 18738/133484 objects misplaced (14.038%); 3.4 MiB/s, 375 objects/s recovering 2024-04-29T14:46:17.621 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:17 smithi194 ceph-mon[44089]: osdmap e866: 12 total, 12 up, 10 in 2024-04-29T14:46:17.621 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:17 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:46:17.621 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:17 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:46:17.623 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:17.643 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:17.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:17.747 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:17.773 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:17.792 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:17.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:17.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:17.926 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:17.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:17.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:18.054 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:18.080 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:18.098 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:18.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:18.202 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:18.247 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:18.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:18.351 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:18.389 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:18.412 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:46:18.412 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:46:18.412 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:46:18.412 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:46:18.414 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:18.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:18.432 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:46:18.453 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:46:18.479 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:46:18.506 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:46:18.519 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:18.532 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:18 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:46:18.532 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:18 smithi194 ceph-mon[44089]: osdmap e867: 12 total, 12 up, 10 in 2024-04-29T14:46:18.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:18 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:46:18.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:18 smithi194 ceph-mon[44089]: pgmap v2233: 17 pgs: 1 peering, 4 active+recovering, 12 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 18584/132966 objects misplaced (13.977%); 1.8 MiB/s, 362 objects/s recovering 2024-04-29T14:46:18.533 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1931830053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:18.534 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:46:18.548 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:46:18.568 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:18.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:18.588 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:46:18.613 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:46:18.614 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:46:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:46:18.671 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:18.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:18 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:46:18.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:18 smithi121 ceph-mon[44533]: osdmap e867: 12 total, 12 up, 10 in 2024-04-29T14:46:18.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:18 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:46:18.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:18 smithi121 ceph-mon[44533]: pgmap v2233: 17 pgs: 1 peering, 4 active+recovering, 12 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 18584/132966 objects misplaced (13.977%); 1.8 MiB/s, 362 objects/s recovering 2024-04-29T14:46:18.842 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1931830053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:18 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:46:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:18 smithi040 ceph-mon[39581]: osdmap e867: 12 total, 12 up, 10 in 2024-04-29T14:46:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:18 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:46:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:18 smithi040 ceph-mon[39581]: pgmap v2233: 17 pgs: 1 peering, 4 active+recovering, 12 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 18584/132966 objects misplaced (13.977%); 1.8 MiB/s, 362 objects/s recovering 2024-04-29T14:46:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1931830053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:18.988 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:18.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:19.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:19.329 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:19.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:19.434 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:19.461 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:19.479 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:19.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:19.497 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:46:19.497 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:46:19.497 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:46:19.497 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:46:19.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:19.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:19.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:19.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.844 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.844 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.845 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:19.851 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:19.877 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:19.895 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:19.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:19.999 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:20.025 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:20.043 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:20.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:20.147 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:20.172 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:20.190 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:20.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:20.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:46:20.294 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:20.320 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:20.338 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:20.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:20.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:20.475 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:20.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:20.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:20.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:20 smithi040 ceph-mon[39581]: pgmap v2234: 17 pgs: 1 peering, 4 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 94 op/s; 18355/132690 objects misplaced (13.833%); 3.4 MiB/s, 278 objects/s recovering 2024-04-29T14:46:20.607 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:20.610 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:20.636 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:20.655 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:20.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:20.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:20.786 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:20 smithi121 ceph-mon[44533]: pgmap v2234: 17 pgs: 1 peering, 4 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 94 op/s; 18355/132690 objects misplaced (13.833%); 3.4 MiB/s, 278 objects/s recovering 2024-04-29T14:46:20.788 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:20 smithi194 ceph-mon[44089]: pgmap v2234: 17 pgs: 1 peering, 4 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 94 op/s; 18355/132690 objects misplaced (13.833%); 3.4 MiB/s, 278 objects/s recovering 2024-04-29T14:46:20.847 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:20.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:20.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:20.987 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:21.441 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:21.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:21.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:21.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:21.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:21.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:21.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:21.738 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:22.593 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:22.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:22.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:46:22.637 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:46:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:46:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:46:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:46:22.637 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:46:22.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:22.740 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:22.895 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:22.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:23.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:23.216 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:23.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:23.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:46:23.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:23.556 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:23 smithi194 ceph-mon[44089]: pgmap v2235: 17 pgs: 4 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 18355/131986 objects misplaced (13.907%); 2.9 MiB/s, 238 objects/s recovering 2024-04-29T14:46:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:23 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:23 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]: dispatch 2024-04-29T14:46:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:23 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]: dispatch 2024-04-29T14:46:23.581 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:23.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:23 smithi121 ceph-mon[44533]: pgmap v2235: 17 pgs: 4 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 18355/131986 objects misplaced (13.907%); 2.9 MiB/s, 238 objects/s recovering 2024-04-29T14:46:23.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:23 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:23.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:23 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:23.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]: dispatch 2024-04-29T14:46:23.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:23 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]: dispatch 2024-04-29T14:46:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:23.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:23 smithi040 ceph-mon[39581]: pgmap v2235: 17 pgs: 4 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 18355/131986 objects misplaced (13.907%); 2.9 MiB/s, 238 objects/s recovering 2024-04-29T14:46:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:23 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:23 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]: dispatch 2024-04-29T14:46:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:23 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]: dispatch 2024-04-29T14:46:23.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:23.697 INFO:teuthology.orchestra.run.smithi040.stdout:moved 3 / 34 (8.82353%) 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:avg 3.4 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:stddev 1.56205 -> 1.68523 (expected baseline 1.74929) 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:min osd.6 with 1 -> 1 pgs (0.294118 -> 0.294118 * mean) 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:max osd.1 with 6 -> 4 pgs (1.76471 -> 1.17647 * mean) 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:oload 110 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:max_change 1 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:max_change_osds 4 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:average_utilization 0.0175 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:overload_utilization 0.0193 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 weight 0.5800 -> 0.3542 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 weight 0.7945 -> 0.5603 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 weight 0.9188 -> 1.0000 2024-04-29T14:46:23.698 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 weight 0.7888 -> 0.7063 2024-04-29T14:46:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:46:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:23.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:23.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:23.808 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:23.846 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:23.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:23.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:24.024 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:24.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:24.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:24.257 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:24.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:24.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:24.459 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:24.488 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:24.508 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:24.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='client.17862 v1:172.21.15.40:0/1933082648' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/474243874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='client.27586 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]': finished 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: from='client.27586 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:46:24.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:24 smithi194 ceph-mon[44089]: osdmap e868: 12 total, 12 up, 10 in 2024-04-29T14:46:24.612 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='client.17862 v1:172.21.15.40:0/1933082648' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/474243874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='client.27586 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:46:24.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]': finished 2024-04-29T14:46:24.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: from='client.27586 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:46:24.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:24 smithi040 ceph-mon[39581]: osdmap e868: 12 total, 12 up, 10 in 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='client.17862 v1:172.21.15.40:0/1933082648' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/474243874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='client.27586 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:46:24.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'36723','1':'46291','7':'65536','10':'23216'}"}]': finished 2024-04-29T14:46:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: from='client.27586 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:46:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:24 smithi121 ceph-mon[44533]: osdmap e868: 12 total, 12 up, 10 in 2024-04-29T14:46:24.640 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:24.659 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:24.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:24.763 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:24.790 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:24.809 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:24.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:24.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:24.942 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:25.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:25.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:25.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:25.330 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:25.467 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:25 smithi121 ceph-mon[44533]: pgmap v2237: 17 pgs: 1 peering, 3 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 122 op/s; 16041/131536 objects misplaced (12.195%); 5.7 MiB/s, 188 objects/s recovering 2024-04-29T14:46:25.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:25 smithi121 ceph-mon[44533]: osdmap e869: 12 total, 12 up, 10 in 2024-04-29T14:46:25.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:25 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:46:25.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:25 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:46:25.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:25 smithi194 ceph-mon[44089]: pgmap v2237: 17 pgs: 1 peering, 3 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 122 op/s; 16041/131536 objects misplaced (12.195%); 5.7 MiB/s, 188 objects/s recovering 2024-04-29T14:46:25.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:25 smithi194 ceph-mon[44089]: osdmap e869: 12 total, 12 up, 10 in 2024-04-29T14:46:25.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:25 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:46:25.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:25 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:46:25.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:25 smithi040 ceph-mon[39581]: pgmap v2237: 17 pgs: 1 peering, 3 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 122 op/s; 16041/131536 objects misplaced (12.195%); 5.7 MiB/s, 188 objects/s recovering 2024-04-29T14:46:25.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:25 smithi040 ceph-mon[39581]: osdmap e869: 12 total, 12 up, 10 in 2024-04-29T14:46:25.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:25 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:46:25.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:25 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDScrubReserve(1.0,GRANT,e:834,from:2,reservation_nonce:0)", 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:30.770941+0000", 2024-04-29T14:46:25.785 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.011633096, 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021065799999999999, 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.770941+0000", 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.770941+0000", 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.770943+0000", 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6360000000000001e-06 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.770948+0000", 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4049999999999999e-06 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.770960+0000", 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1548e-05 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.770970+0000", 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7030000000000001e-06 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.771022+0000", 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2232999999999999e-05 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.771036+0000", 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4348999999999999e-05 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:25.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:30.771152+0000", 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115784 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:25.789 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:46:26.181 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:26.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:26.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:26.317 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:26.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:46:26.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:26.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:26.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:26.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: 7.0 scrub starts 2024-04-29T14:46:26.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: 7.c scrub starts 2024-04-29T14:46:26.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: 7.0 scrub ok 2024-04-29T14:46:26.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: 7.c scrub ok 2024-04-29T14:46:26.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: osdmap e870: 12 total, 12 up, 10 in 2024-04-29T14:46:26.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: 7.b starting backfill to osd.4 from (0'0,0'0] MAX to 867'10221 2024-04-29T14:46:26.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:26.794 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:46:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:26.825 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:46:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:26.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:26.856 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:46:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: 7.0 scrub starts 2024-04-29T14:46:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: 7.c scrub starts 2024-04-29T14:46:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: 7.0 scrub ok 2024-04-29T14:46:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: 7.c scrub ok 2024-04-29T14:46:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: osdmap e870: 12 total, 12 up, 10 in 2024-04-29T14:46:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: 7.b starting backfill to osd.4 from (0'0,0'0] MAX to 867'10221 2024-04-29T14:46:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: 7.4 starting backfill to osd.9 from (0'0,0'0] MAX to 867'10384 2024-04-29T14:46:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: 7.e starting backfill to osd.9 from (0'0,0'0] MAX to 867'17482 2024-04-29T14:46:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:26 smithi121 ceph-mon[44533]: pgmap v2240: 17 pgs: 1 peering, 3 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 16041/131134 objects misplaced (12.233%); 4.2 MiB/s, 66 objects/s recovering 2024-04-29T14:46:26.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:46:26.924 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:46:26.953 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:46:26.956 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:26.969 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:26.981 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:46:26.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: 7.4 starting backfill to osd.9 from (0'0,0'0] MAX to 867'10384 2024-04-29T14:46:26.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: 7.e starting backfill to osd.9 from (0'0,0'0] MAX to 867'17482 2024-04-29T14:46:26.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:26 smithi040 ceph-mon[39581]: pgmap v2240: 17 pgs: 1 peering, 3 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 16041/131134 objects misplaced (12.233%); 4.2 MiB/s, 66 objects/s recovering 2024-04-29T14:46:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:27.009 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:46:27.023 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:27.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:27.036 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:46:27.072 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: 7.0 scrub starts 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: 7.c scrub starts 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: 7.0 scrub ok 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: 7.c scrub ok 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: osdmap e870: 12 total, 12 up, 10 in 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: 7.b starting backfill to osd.4 from (0'0,0'0] MAX to 867'10221 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: 7.4 starting backfill to osd.9 from (0'0,0'0] MAX to 867'10384 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: 7.e starting backfill to osd.9 from (0'0,0'0] MAX to 867'17482 2024-04-29T14:46:27.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:26 smithi194 ceph-mon[44089]: pgmap v2240: 17 pgs: 1 peering, 3 active+recovering, 13 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 16041/131134 objects misplaced (12.233%); 4.2 MiB/s, 66 objects/s recovering 2024-04-29T14:46:27.096 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:46:27.128 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:27.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:46:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:46:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:46:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:46:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:46:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:46:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:46:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:46:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:46:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:46:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:46:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:46:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:46:27.154 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:27.174 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:27.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:27.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:27.416 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:27.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:27.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:27.551 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:27.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:46:27.752 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:27.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:27.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/42298900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:27 smithi040 ceph-mon[39581]: osdmap e871: 12 total, 12 up, 10 in 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/42298900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:27.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:27 smithi121 ceph-mon[44533]: osdmap e871: 12 total, 12 up, 10 in 2024-04-29T14:46:27.883 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:27.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:27.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:28.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/42298900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:27 smithi194 ceph-mon[44089]: osdmap e871: 12 total, 12 up, 10 in 2024-04-29T14:46:28.109 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:28.140 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:28.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:28.148 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:28.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:28.280 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:28.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:28.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:28.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:28.680 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:28.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:28.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:28.986 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:28.986 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:29.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.012 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.012 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:28 smithi121 ceph-mon[44533]: pgmap v2242: 17 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 19424/131042 objects misplaced (14.823%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:46:29.023 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:29.025 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.026 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:28 smithi040 ceph-mon[39581]: pgmap v2242: 17 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 19424/131042 objects misplaced (14.823%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:46:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:29.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:29.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:29.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:28 smithi194 ceph-mon[44089]: pgmap v2242: 17 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 72 op/s; 19424/131042 objects misplaced (14.823%); 13 MiB/s, 207 objects/s recovering 2024-04-29T14:46:29.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:29.173 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:29.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:29.277 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:29.303 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:29.323 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:29.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:29.427 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:29.453 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:29.472 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:29.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:29.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:29.604 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:29.623 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:29.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:29.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:29.761 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:29.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:46:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:29.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:29.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] out_osds: [3, 11] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:46:29.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:46:29.864 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 4, 7, 9, 10, 6, 2, 1, 5, 8] 2024-04-29T14:46:29.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 2 2024-04-29T14:46:29.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:29.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:29.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:29 smithi040 ceph-mon[39581]: osdmap e872: 12 total, 12 up, 10 in 2024-04-29T14:46:29.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:29.971 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:29.971 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:29.971 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:29 smithi194 ceph-mon[44089]: osdmap e872: 12 total, 12 up, 10 in 2024-04-29T14:46:29.973 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:29.992 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:29.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:30.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:30.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:46:30.126 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:30.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:30.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:30.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:29 smithi121 ceph-mon[44533]: osdmap e872: 12 total, 12 up, 10 in 2024-04-29T14:46:30.138 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:30.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:30.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:30.425 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:30.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:30.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:30.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:30.671 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:30.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:30.734 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:46:30.776 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:30.802 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:30.822 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:30.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:30.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:30 smithi040 ceph-mon[39581]: pgmap v2244: 17 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 39 op/s; 19424/131030 objects misplaced (14.824%); 8.3 MiB/s, 133 objects/s recovering 2024-04-29T14:46:30.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2192506593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:46:30.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:30 smithi121 ceph-mon[44533]: pgmap v2244: 17 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 39 op/s; 19424/131030 objects misplaced (14.824%); 8.3 MiB/s, 133 objects/s recovering 2024-04-29T14:46:30.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2192506593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:46:30.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:30.961 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:31.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:30 smithi194 ceph-mon[44089]: pgmap v2244: 17 pgs: 1 active+remapped+backfilling, 1 peering, 3 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 39 op/s; 19424/131030 objects misplaced (14.824%); 8.3 MiB/s, 133 objects/s recovering 2024-04-29T14:46:31.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2192506593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:46:31.118 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:31.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:31.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:31.260 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:31.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:31.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:31.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:31.606 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:31.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:31.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:31.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:31.766 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:31.785 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:31.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:31.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:31 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:31.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2192506593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:46:31.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:31 smithi040 ceph-mon[39581]: osdmap e873: 12 total, 12 up, 10 in 2024-04-29T14:46:31.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:31 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:31.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2192506593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:46:31.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:31 smithi121 ceph-mon[44533]: osdmap e873: 12 total, 12 up, 10 in 2024-04-29T14:46:31.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:31.970 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:31.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:32.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:31 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:32.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2192506593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:46:32.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:31 smithi194 ceph-mon[44089]: osdmap e873: 12 total, 12 up, 10 in 2024-04-29T14:46:32.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:32.111 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:32.137 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:32.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:32.171 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:32.171 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:32.241 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:32.268 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:32.287 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:32.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:32.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:32.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:32.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:32.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:32.697 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:32.809 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:32.813 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:32 smithi040 ceph-mon[39581]: pgmap v2246: 17 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 2 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 15 op/s; 11535/130944 objects degraded (8.809%); 13433/130944 objects misplaced (10.259%); 3231/65472 objects unfound (4.935%); 44 MiB/s, 711 objects/s recovering 2024-04-29T14:46:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:32.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:32.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:32.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:46:32.944 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:32.966 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:32.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:33.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:33.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:32 smithi194 ceph-mon[44089]: pgmap v2246: 17 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 2 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 15 op/s; 11535/130944 objects degraded (8.809%); 13433/130944 objects misplaced (10.259%); 3231/65472 objects unfound (4.935%); 44 MiB/s, 711 objects/s recovering 2024-04-29T14:46:33.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:32 smithi121 ceph-mon[44533]: pgmap v2246: 17 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 2 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 15 op/s; 11535/130944 objects degraded (8.809%); 13433/130944 objects misplaced (10.259%); 3231/65472 objects unfound (4.935%); 44 MiB/s, 711 objects/s recovering 2024-04-29T14:46:33.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:33.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:33.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:33.231 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:33.258 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:33.265 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:46:33.277 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:33.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:33.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:33.415 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:33.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:33 smithi121 ceph-mon[44533]: Health check failed: 3231/65472 objects unfound (4.935%) (OBJECT_UNFOUND) 2024-04-29T14:46:33.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:33 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 11535/130944 objects degraded (8.809%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:46:33.968 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:33.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:34.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:46:34.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:33 smithi040 ceph-mon[39581]: Health check failed: 3231/65472 objects unfound (4.935%) (OBJECT_UNFOUND) 2024-04-29T14:46:34.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:33 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 11535/130944 objects degraded (8.809%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:46:34.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:33 smithi194 ceph-mon[44089]: Health check failed: 3231/65472 objects unfound (4.935%) (OBJECT_UNFOUND) 2024-04-29T14:46:34.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:33 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 11535/130944 objects degraded (8.809%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:46:34.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:34.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:34.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:34.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:34.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:34.263 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:34.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:34.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:34.822 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:34.870 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:34.904 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:34.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:34.916 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:46:34.934 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:34.946 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:46:34.973 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:46:34.997 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:46:34.999 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:34 smithi121 ceph-mon[44533]: osdmap e874: 12 total, 12 up, 10 in 2024-04-29T14:46:34.999 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:34 smithi121 ceph-mon[44533]: pgmap v2248: 17 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 2 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 8 op/s; 11535/130934 objects degraded (8.810%); 13433/130934 objects misplaced (10.259%); 3231/65467 objects unfound (4.935%); 37 MiB/s, 587 objects/s recovering 2024-04-29T14:46:35.011 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:35.021 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:46:35.041 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:34 smithi194 ceph-mon[44089]: osdmap e874: 12 total, 12 up, 10 in 2024-04-29T14:46:35.041 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:34 smithi194 ceph-mon[44089]: pgmap v2248: 17 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 2 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 8 op/s; 11535/130934 objects degraded (8.810%); 13433/130934 objects misplaced (10.259%); 3231/65467 objects unfound (4.935%); 37 MiB/s, 587 objects/s recovering 2024-04-29T14:46:35.043 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:35.046 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:46:35.064 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:35.066 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:46:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:35.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:35.093 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:46:35.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:34 smithi040 ceph-mon[39581]: osdmap e874: 12 total, 12 up, 10 in 2024-04-29T14:46:35.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:34 smithi040 ceph-mon[39581]: pgmap v2248: 17 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 2 active+recovering, 12 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 8 op/s; 11535/130934 objects degraded (8.810%); 13433/130934 objects misplaced (10.259%); 3231/65467 objects unfound (4.935%); 37 MiB/s, 587 objects/s recovering 2024-04-29T14:46:35.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:34 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:46:34.948+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e875 definitely_dead 0 2024-04-29T14:46:35.123 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:46:35.144 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:46:35.165 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:46:35.168 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:35.186 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:46:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:46:35.197 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:35.216 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:35.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:35.314 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:46:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:46:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:46:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:154966 7.6 e845/839 7:6592a997:::benchmark_data_smithi194_147200_object40054:head v 845'9460, mlcod=664'3188)", 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:03.519171+0000", 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.790848030000006, 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021034672000000001, 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.519171+0000", 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.519171+0000", 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.519172+0000", 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8699999999999995e-07 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.519178+0000", 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0390000000000001e-06 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.519182+0000", 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0890000000000002e-06 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.519185+0000", 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1810000000000001e-06 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.523795+0000", 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046097739999999996 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.523824+0000", 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9549000000000001e-05 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.540144+0000", 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016319917999999999 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:03.540206+0000", 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1535e-05 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:155121 7.6 e845/839 7:6f8de4e8:::benchmark_data_smithi194_147200_object40209:head v 845'9472, mlcod=664'3211)", 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:04.096875+0000", 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.213143666999997, 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018746505, 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.096875+0000", 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.096875+0000", 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.096877+0000", 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.189e-06 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.096884+0000", 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5979999999999999e-06 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.096890+0000", 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9009999999999999e-06 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.096894+0000", 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1189999999999999e-06 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.101247+0000", 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043530110000000004 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.101264+0000", 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6881e-05 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.115579+0000", 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014314687 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.115622+0000", 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3118999999999998e-05 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:155133 7.6 e845/839 7:6e08bd3a:::benchmark_data_smithi194_147200_object40221:head v 845'9474, mlcod=664'3214)", 2024-04-29T14:46:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:04.160469+0000", 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.149549929000003, 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019169036, 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160469+0000", 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160469+0000", 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160471+0000", 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1660000000000001e-06 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160484+0000", 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2379e-05 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160496+0000", 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2223e-05 2024-04-29T14:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160505+0000", 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6799999999999999e-06 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160694+0000", 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018931699999999999 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.160725+0000", 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0849999999999998e-05 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.179604+0000", 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018878823999999999 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.179638+0000", 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4597000000000001e-05 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:155191 7.6 e845/839 7:6abf96f3:::benchmark_data_smithi194_147200_object40279:head v 845'9481, mlcod=664'3239)", 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:04.691576+0000", 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.618443438, 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019040459999999999, 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.691576+0000", 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.691576+0000", 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.691577+0000", 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0979999999999999e-06 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.691585+0000", 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9740000000000002e-06 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.691593+0000", 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0190000000000006e-06 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.691597+0000", 2024-04-29T14:46:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0570000000000003e-06 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.695963+0000", 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004365964 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.695995+0000", 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1973e-05 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.710592+0000", 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014597066000000001 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.710616+0000", 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4309000000000001e-05 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:155575 7.6 e845/839 7:6a7a072b:::benchmark_data_smithi194_147200_object40663:head v 845'9509, mlcod=664'3315)", 2024-04-29T14:46:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:07.187013+0000", 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 88.123005926999994, 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020642172, 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.187013+0000", 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.187013+0000", 2024-04-29T14:46:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.187015+0000", 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4470000000000001e-06 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.187023+0000", 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8049999999999996e-06 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.187032+0000", 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9740000000000008e-06 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.187039+0000", 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2879999999999998e-06 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.190331+0000", 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003292522 2024-04-29T14:46:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.190364+0000", 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2666999999999997e-05 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.207609+0000", 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017245449999999999 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:07.207655+0000", 2024-04-29T14:46:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6019000000000001e-05 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:156150 7.6 e845/839 7:69c3206c:::benchmark_data_smithi194_147200_object41238:head v 845'9548, mlcod=665'3546)", 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:11.045485+0000", 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.264534424000004, 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019005221999999999, 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.045485+0000", 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.045485+0000", 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.045485+0000", 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7499999999999999e-07 2024-04-29T14:46:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.045494+0000", 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5860000000000005e-06 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.045500+0000", 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2280000000000003e-06 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.045506+0000", 2024-04-29T14:46:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4840000000000003e-06 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.049034+0000", 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003528298 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.049066+0000", 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1727999999999998e-05 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.064455+0000", 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015389045000000001 2024-04-29T14:46:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.064490+0000", 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4978000000000003e-05 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:156207 7.6 e845/839 7:6020a1a7:::benchmark_data_smithi194_147200_object41295:head v 845'9551, mlcod=665'3558)", 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:11.332012+0000", 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.978006798999999, 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022181932000000001, 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.332012+0000", 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.332012+0000", 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.332014+0000", 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.404e-06 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.332022+0000", 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6999999999999997e-06 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.332031+0000", 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7409999999999998e-06 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.332037+0000", 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0090000000000004e-06 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.335427+0000", 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033896439999999998 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.335459+0000", 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2008000000000003e-05 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.354165+0000", 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018706538000000002 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:11.354194+0000", 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8887999999999999e-05 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:156302 7.6 e845/839 7:660a1b70:::benchmark_data_smithi194_147200_object41390:head v 845'9555, mlcod=665'3579)", 2024-04-29T14:46:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:12.192781+0000", 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.117238248999996, 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022894670999999998, 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.192781+0000", 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.192781+0000", 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.192782+0000", 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6139999999999999e-06 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.192792+0000", 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4499999999999993e-06 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.192802+0000", 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0036e-05 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.192810+0000", 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2819999999999996e-06 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.196321+0000", 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035112670000000002 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.196355+0000", 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3144e-05 2024-04-29T14:46:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.215613+0000", 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019258087 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:12.215675+0000", 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2791000000000002e-05 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:156591 7.6 e846/839 7:6b7fdff7:::benchmark_data_smithi194_147200_object41679:head v 846'9573, mlcod=666'3720)", 2024-04-29T14:46:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:14.535989+0000", 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 80.774029784000007, 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019710192000000001, 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.535989+0000", 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.535989+0000", 2024-04-29T14:46:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.535990+0000", 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8700000000000005e-07 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.535997+0000", 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.481e-06 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.536004+0000", 2024-04-29T14:46:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4489999999999996e-06 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.536008+0000", 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3560000000000003e-06 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.539270+0000", 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032618339999999999 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.539302+0000", 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1649000000000001e-05 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.555633+0000", 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016331286 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.555699+0000", 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6450000000000002e-05 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:156678 7.6 e846/839 7:6267b8a4:::benchmark_data_smithi194_147200_object41766:head v 846'9579, mlcod=666'3761)", 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:15.341883+0000", 2024-04-29T14:46:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.968135567000004, 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020432670999999999, 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.341883+0000", 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.341883+0000", 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.341885+0000", 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2979999999999999e-06 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.341894+0000", 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0780000000000002e-06 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.341901+0000", 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1600000000000001e-06 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.341907+0000", 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7849999999999997e-06 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.345247+0000", 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033405380000000001 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.345279+0000", 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1866999999999998e-05 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.362248+0000", 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016969080000000001 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.362316+0000", 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7865000000000002e-05 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:156697 7.6 e846/839 7:68907e99:::benchmark_data_smithi194_147200_object41785:head v 846'9581, mlcod=666'3762)", 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:15.425986+0000", 2024-04-29T14:46:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.884032524999995, 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019480082999999999, 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.425986+0000", 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.425986+0000", 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.425988+0000", 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.266e-06 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.426000+0000", 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2027e-05 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.426010+0000", 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0030999999999999e-05 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.426017+0000", 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3849999999999996e-06 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.429399+0000", 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033819739999999998 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.429430+0000", 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1189000000000002e-05 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.445441+0000", 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016010679 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:15.445467+0000", 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5531999999999999e-05 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:157291 7.6 e847/839 7:6f4a4a31:::benchmark_data_smithi194_147200_object42379:head v 847'9625, mlcod=667'3962)", 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:19.301820+0000", 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.008199185999999, 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019084909000000001, 2024-04-29T14:46:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.301820+0000", 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.301820+0000", 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.301821+0000", 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0499999999999999e-06 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.301830+0000", 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0410000000000003e-06 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.301837+0000", 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7769999999999997e-06 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.301841+0000", 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2010000000000001e-06 2024-04-29T14:46:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.305156+0000", 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033148930000000002 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.305181+0000", 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4994000000000002e-05 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.320848+0000", 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015667443999999999 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.320905+0000", 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6508999999999997e-05 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:157454 7.6 e848/839 7:6fbc1258:::benchmark_data_smithi194_147200_object42542:head v 848'9637, mlcod=667'4023)", 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:20.484185+0000", 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.825834119999996, 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017696585000000001, 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484185+0000", 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484185+0000", 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484188+0000", 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8619999999999999e-06 2024-04-29T14:46:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484205+0000", 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7286999999999999e-05 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484219+0000", 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3984e-05 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484228+0000", 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7430000000000008e-06 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484280+0000", 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2710999999999997e-05 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.484303+0000", 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2744000000000001e-05 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.501837+0000", 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017533884 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.501881+0000", 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4369999999999997e-05 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:157458 7.6 e848/839 7:6af4d563:::benchmark_data_smithi194_147200_object42546:head v 848'9638, mlcod=667'4023)", 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:20.487960+0000", 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.822058923, 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022898903000000002, 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.487960+0000", 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.487960+0000", 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.487963+0000", 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.441e-06 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.487978+0000", 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5444999999999999e-05 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.487991+0000", 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.289e-05 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.488000+0000", 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1120000000000002e-06 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.494994+0000", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069937549999999999 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.495020+0000", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6749000000000001e-05 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.510822+0000", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015801671 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.510859+0000", 2024-04-29T14:46:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.684e-05 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:157531 7.6 e849/839 7:64619234:::benchmark_data_smithi194_147200_object42619:head v 849'9648, mlcod=667'4035)", 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:20.832251+0000", 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.477768014999995, 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019725071, 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.832251+0000", 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.832251+0000", 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.832252+0000", 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.850000000000001e-07 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.832263+0000", 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0604999999999999e-05 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.832272+0000", 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.764e-06 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.832278+0000", 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0510000000000002e-06 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.835590+0000", 2024-04-29T14:46:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003311802 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.835622+0000", 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1961000000000001e-05 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.851911+0000", 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016288552000000001 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.851976+0000", 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5351000000000005e-05 2024-04-29T14:46:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:157664 7.6 e849/839 7:694d307a:::benchmark_data_smithi194_147200_object42752:head v 849'9661, mlcod=667'4083)", 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:21.746052+0000", 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.563967486999999, 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018945968000000001, 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.746052+0000", 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.746052+0000", 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.746052+0000", 2024-04-29T14:46:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7100000000000001e-07 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.746058+0000", 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5999999999999997e-06 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.746063+0000", 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2270000000000001e-06 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.746068+0000", 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9280000000000001e-06 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.750753+0000", 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046849639999999998 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.750791+0000", 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7805000000000001e-05 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.764956+0000", 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014164859 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.764998+0000", 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1913999999999999e-05 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:157827 7.6 e850/839 7:6baa5bab:::benchmark_data_smithi194_147200_object42915:head v 850'9670, mlcod=667'4174)", 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:23.256765+0000", 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.053253964999996, 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020030922999999999, 2024-04-29T14:46:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.256765+0000", 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.256765+0000", 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.256766+0000", 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.311e-06 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.256779+0000", 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2378000000000001e-05 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.256785+0000", 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4699999999999999e-06 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.256790+0000", 2024-04-29T14:46:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4390000000000002e-06 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.261255+0000", 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044653829999999999 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.261286+0000", 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0734000000000001e-05 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.276722+0000", 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015435877000000001 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.276796+0000", 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4331000000000004e-05 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:158788 7.6 e853/839 7:600bc5ee:::benchmark_data_smithi194_147200_object43876:head v 853'9719, mlcod=671'4801)", 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:33.301681+0000", 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.008338314, 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019012930000000001, 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.301681+0000", 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.301681+0000", 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.301682+0000", 2024-04-29T14:46:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3969999999999999e-06 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.301691+0000", 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4579999999999993e-06 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.301704+0000", 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2955e-05 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.301710+0000", 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5479999999999998e-06 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.306055+0000", 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043448929999999998 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.306086+0000", 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1214999999999998e-05 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.320648+0000", 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014561746 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:33.320694+0000", 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5717999999999998e-05 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26998.0:159088 7.6 e854/839 7:6bbdb760:::benchmark_data_smithi194_147200_object44176:head v 854'9740, mlcod=673'4976)", 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:35.814841+0000", 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 59.495177658000003, 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024610891999999999, 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.814841+0000", 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.814841+0000", 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.814842+0000", 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.018e-06 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.814850+0000", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7009999999999996e-06 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.814858+0000", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5320000000000001e-06 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.814862+0000", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7400000000000004e-06 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.818173+0000", 2024-04-29T14:46:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033108999999999999 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.818203+0000", 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9734e-05 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.839414+0000", 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021210781000000001 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:35.839452+0000", 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8485999999999997e-05 2024-04-29T14:46:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:164230 7.6 7:6a44f9ff:::benchmark_data_smithi194_147200_object49318:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:46:22.937586+0000", 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.372433260999999, 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017791787, 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 164230 2024-04-29T14:46:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937586+0000", 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937586+0000", 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937589+0000", 2024-04-29T14:46:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1549999999999999e-06 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937599+0000", 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0261999999999999e-05 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937616+0000", 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6807e-05 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937645+0000", 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8807999999999999e-05 2024-04-29T14:46:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937740+0000", 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5428999999999997e-05 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937844+0000", 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010354899999999999 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.937923+0000", 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8839999999999997e-05 2024-04-29T14:46:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.941588+0000", 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003665289 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.955303+0000", 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013714703999999999 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.955341+0000", 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8503e-05 2024-04-29T14:46:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:22.955378+0000", 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6440999999999999e-05 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:46:35.398 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:46:35.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:35.430 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.2. 2024-04-29T14:46:35.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osdmap e875: 12 total, 12 up, 10 in 2024-04-29T14:46:35.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:35.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:35.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1986830777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:35.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/300305746' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: from='client.37349 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osdmap e875: 12 total, 12 up, 10 in 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1986830777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/300305746' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: from='client.37349 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:46:35.980 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-29T14:46:35.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:35.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:36.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:36.014 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osdmap e875: 12 total, 12 up, 10 in 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1986830777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/300305746' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: from='client.37349 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-29T14:46:36.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:36.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:36.136 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:36.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:36.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:36.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:36.289 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:46:36.309 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:36.762 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:36.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:36.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:37.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: from='client.37349 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: osdmap e876: 12 total, 12 up, 9 in 2024-04-29T14:46:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:36 smithi194 ceph-mon[44089]: pgmap v2251: 17 pgs: 1 activating+degraded+remapped, 1 unknown, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 11 active+clean; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 4 op/s; 7277/130912 objects degraded (5.559%); 12515/130912 objects misplaced (9.560%); 3231/65456 objects unfound (4.936%); 35 MiB/s, 553 objects/s recovering 2024-04-29T14:46:37.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:37.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:37.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: from='client.37349 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-29T14:46:37.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: osdmap e876: 12 total, 12 up, 9 in 2024-04-29T14:46:37.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:36 smithi040 ceph-mon[39581]: pgmap v2251: 17 pgs: 1 activating+degraded+remapped, 1 unknown, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 11 active+clean; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 4 op/s; 7277/130912 objects degraded (5.559%); 12515/130912 objects misplaced (9.560%); 3231/65456 objects unfound (4.936%); 35 MiB/s, 553 objects/s recovering 2024-04-29T14:46:37.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: from='client.37349 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: osdmap e876: 12 total, 12 up, 9 in 2024-04-29T14:46:37.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:36 smithi121 ceph-mon[44533]: pgmap v2251: 17 pgs: 1 activating+degraded+remapped, 1 unknown, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 11 active+clean; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 4 op/s; 7277/130912 objects degraded (5.559%); 12515/130912 objects misplaced (9.560%); 3231/65456 objects unfound (4.936%); 35 MiB/s, 553 objects/s recovering 2024-04-29T14:46:38.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:37 smithi194 ceph-mon[44089]: osdmap e877: 12 total, 12 up, 9 in 2024-04-29T14:46:38.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:37 smithi194 ceph-mon[44089]: osdmap e878: 12 total, 12 up, 9 in 2024-04-29T14:46:38.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:37 smithi040 ceph-mon[39581]: osdmap e877: 12 total, 12 up, 9 in 2024-04-29T14:46:38.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:37 smithi040 ceph-mon[39581]: osdmap e878: 12 total, 12 up, 9 in 2024-04-29T14:46:38.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:37 smithi121 ceph-mon[44533]: osdmap e877: 12 total, 12 up, 9 in 2024-04-29T14:46:38.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:37 smithi121 ceph-mon[44533]: osdmap e878: 12 total, 12 up, 9 in 2024-04-29T14:46:38.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:46:39.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 872'13800 2024-04-29T14:46:39.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: 7.c starting backfill to osd.9 from (0'0,0'0] MAX to 873'10258 2024-04-29T14:46:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 868'10231 2024-04-29T14:46:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:46:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:46:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:46:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: 7.a starting backfill to osd.5 from (0'0,0'0] MAX to 872'17600 2024-04-29T14:46:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: 7.e starting backfill to osd.5 from (0'0,0'0] MAX to 870'17484 2024-04-29T14:46:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: 7.2 starting backfill to osd.6 from (0'0,0'0] MAX to 868'10139 2024-04-29T14:46:39.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:46:39.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:46:39.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:46:39.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:39.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:38 smithi194 ceph-mon[44089]: pgmap v2254: 17 pgs: 1 active+degraded+remapped, 1 remapped+peering, 10 active+clean, 2 active+recovering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 unknown; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 7277/130906 objects degraded (5.559%); 11470/130906 objects misplaced (8.762%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:46:39.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 872'13800 2024-04-29T14:46:39.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: 7.c starting backfill to osd.9 from (0'0,0'0] MAX to 873'10258 2024-04-29T14:46:39.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 868'10231 2024-04-29T14:46:39.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:46:39.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: 7.a starting backfill to osd.5 from (0'0,0'0] MAX to 872'17600 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: 7.e starting backfill to osd.5 from (0'0,0'0] MAX to 870'17484 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: 7.2 starting backfill to osd.6 from (0'0,0'0] MAX to 868'10139 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:38 smithi040 ceph-mon[39581]: pgmap v2254: 17 pgs: 1 active+degraded+remapped, 1 remapped+peering, 10 active+clean, 2 active+recovering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 unknown; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 7277/130906 objects degraded (5.559%); 11470/130906 objects misplaced (8.762%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:46:39.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: 7.3 starting backfill to osd.6 from (0'0,0'0] MAX to 872'13800 2024-04-29T14:46:39.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: 7.c starting backfill to osd.9 from (0'0,0'0] MAX to 873'10258 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 868'10231 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: 7.a starting backfill to osd.5 from (0'0,0'0] MAX to 872'17600 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: 7.e starting backfill to osd.5 from (0'0,0'0] MAX to 870'17484 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: 7.2 starting backfill to osd.6 from (0'0,0'0] MAX to 868'10139 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:46:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:46:39.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:46:39.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:38 smithi121 ceph-mon[44533]: pgmap v2254: 17 pgs: 1 active+degraded+remapped, 1 remapped+peering, 10 active+clean, 2 active+recovering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 unknown; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 7277/130906 objects degraded (5.559%); 11470/130906 objects misplaced (8.762%); 21 MiB/s, 328 objects/s recovering 2024-04-29T14:46:40.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:39 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 7277/130906 objects degraded (5.559%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:46:40.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:39 smithi194 ceph-mon[44089]: Health check cleared: OBJECT_UNFOUND (was: 3231/65456 objects unfound (4.936%)) 2024-04-29T14:46:40.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:39 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 7277/130906 objects degraded (5.559%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:46:40.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:39 smithi040 ceph-mon[39581]: Health check cleared: OBJECT_UNFOUND (was: 3231/65456 objects unfound (4.936%)) 2024-04-29T14:46:40.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:39 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 7277/130906 objects degraded (5.559%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:46:40.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:39 smithi121 ceph-mon[44533]: Health check cleared: OBJECT_UNFOUND (was: 3231/65456 objects unfound (4.936%)) 2024-04-29T14:46:40.703 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:46:40.703 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:46:40.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.703 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:164587 7.f e878/838 7:f38374ad:::benchmark_data_smithi194_147200_object49675:head v 878'10339, mlcod=878'10337)", 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:40.696559+0000", 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0026081099999999999, 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002640983, 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:40.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696559+0000", 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696559+0000", 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696560+0000", 2024-04-29T14:46:40.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.066e-06 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696566+0000", 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4960000000000001e-06 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696578+0000", 2024-04-29T14:46:40.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1620999999999999e-05 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696584+0000", 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1369999999999999e-06 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696640+0000", 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5794000000000001e-05 2024-04-29T14:46:40.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.696660+0000", 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0630999999999999e-05 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(7.e 878/877 [PushReplyOp(7:73ad2abe:::benchmark_data_smithi194_147200_object62138:head)])", 2024-04-29T14:46:40.708 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:40.697151+0000", 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0020157679999999998, 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021190010000000001, 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "queued for pg", 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.697151+0000", 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:40.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.697151+0000", 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.697151+0000", 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9200000000000002e-07 2024-04-29T14:46:40.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.697155+0000", 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.151e-06 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.697161+0000", 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2809999999999997e-06 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:40.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:40.697168+0000", 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4159999999999998e-06 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 2 2024-04-29T14:46:40.712 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:46:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:40 smithi194 ceph-mon[44089]: pgmap v2255: 17 pgs: 1 active+degraded+remapped, 1 remapped+peering, 9 active+clean, 2 active+recovering, 2 active+remapped+backfilling, 1 active+recovering+remapped, 1 unknown; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 7277/130898 objects degraded (5.559%); 15090/130898 objects misplaced (11.528%); 20 MiB/s, 316 objects/s recovering 2024-04-29T14:46:41.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:40 smithi040 ceph-mon[39581]: pgmap v2255: 17 pgs: 1 active+degraded+remapped, 1 remapped+peering, 9 active+clean, 2 active+recovering, 2 active+remapped+backfilling, 1 active+recovering+remapped, 1 unknown; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 7277/130898 objects degraded (5.559%); 15090/130898 objects misplaced (11.528%); 20 MiB/s, 316 objects/s recovering 2024-04-29T14:46:41.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:40 smithi121 ceph-mon[44533]: pgmap v2255: 17 pgs: 1 active+degraded+remapped, 1 remapped+peering, 9 active+clean, 2 active+recovering, 2 active+remapped+backfilling, 1 active+recovering+remapped, 1 unknown; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 7277/130898 objects degraded (5.559%); 15090/130898 objects misplaced (11.528%); 20 MiB/s, 316 objects/s recovering 2024-04-29T14:46:41.580 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:46:41.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:41 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:41.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:41 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:41.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:41 smithi194 ceph-mon[44089]: osdmap e879: 12 total, 12 up, 9 in 2024-04-29T14:46:41.957 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:46:42.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:41 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:42.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:41 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:42.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:41 smithi040 ceph-mon[39581]: osdmap e879: 12 total, 12 up, 9 in 2024-04-29T14:46:42.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:41 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:42.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:41 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:42.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:41 smithi121 ceph-mon[44533]: osdmap e879: 12 total, 12 up, 9 in 2024-04-29T14:46:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:42.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:42.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:42.704 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:43.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:42 smithi194 ceph-mon[44089]: pgmap v2257: 17 pgs: 1 active+recovering+remapped, 1 active+degraded+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 25 op/s; 7277/130608 objects degraded (5.572%); 29285/130608 objects misplaced (22.422%); 79 MiB/s, 1.27k objects/s recovering 2024-04-29T14:46:43.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:42 smithi040 ceph-mon[39581]: pgmap v2257: 17 pgs: 1 active+recovering+remapped, 1 active+degraded+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 25 op/s; 7277/130608 objects degraded (5.572%); 29285/130608 objects misplaced (22.422%); 79 MiB/s, 1.27k objects/s recovering 2024-04-29T14:46:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:42 smithi121 ceph-mon[44533]: pgmap v2257: 17 pgs: 1 active+recovering+remapped, 1 active+degraded+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 25 op/s; 7277/130608 objects degraded (5.572%); 29285/130608 objects misplaced (22.422%); 79 MiB/s, 1.27k objects/s recovering 2024-04-29T14:46:43.554 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:43.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:43.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:43.696 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:43.825 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:43.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:43.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:43 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:43.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:43 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:43.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:43.965 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:44.023 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:46:44.023 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:46:44.023 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:46:44.023 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:46:44.023 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:46:44.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:43 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:44.047 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:43 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:44.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:43 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:44.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:43 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:44.780 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:46:45.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:44 smithi194 ceph-mon[44089]: osdmap e880: 12 total, 12 up, 9 in 2024-04-29T14:46:45.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:44 smithi194 ceph-mon[44089]: pgmap v2259: 17 pgs: 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 42 op/s; 36207/130356 objects misplaced (27.775%); 76 MiB/s, 1.22k objects/s recovering 2024-04-29T14:46:45.119 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:46:45.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:44 smithi040 ceph-mon[39581]: osdmap e880: 12 total, 12 up, 9 in 2024-04-29T14:46:45.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:44 smithi040 ceph-mon[39581]: pgmap v2259: 17 pgs: 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 42 op/s; 36207/130356 objects misplaced (27.775%); 76 MiB/s, 1.22k objects/s recovering 2024-04-29T14:46:45.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:44 smithi121 ceph-mon[44533]: osdmap e880: 12 total, 12 up, 9 in 2024-04-29T14:46:45.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:44 smithi121 ceph-mon[44533]: pgmap v2259: 17 pgs: 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 42 op/s; 36207/130356 objects misplaced (27.775%); 76 MiB/s, 1.22k objects/s recovering 2024-04-29T14:46:45.820 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:45.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:45.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:45.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:45.960 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:45 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7277/130608 objects degraded (5.572%), 1 pg degraded) 2024-04-29T14:46:45.964 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:46.067 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:46.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:46.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:45 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7277/130608 objects degraded (5.572%), 1 pg degraded) 2024-04-29T14:46:46.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:45 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7277/130608 objects degraded (5.572%), 1 pg degraded) 2024-04-29T14:46:46.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:46.204 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:46.555 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:46.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:46.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:46.659 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:46.691 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:46.892 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:46.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:46.996 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:47.033 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:47.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:46 smithi194 ceph-mon[44089]: pgmap v2260: 17 pgs: 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 89 op/s; 26869/129474 objects misplaced (20.752%); 124 MiB/s, 1.99k objects/s recovering 2024-04-29T14:46:47.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:46 smithi040 ceph-mon[39581]: pgmap v2260: 17 pgs: 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 89 op/s; 26869/129474 objects misplaced (20.752%); 124 MiB/s, 1.99k objects/s recovering 2024-04-29T14:46:47.132 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:47.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:46 smithi121 ceph-mon[44533]: pgmap v2260: 17 pgs: 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 6 active+remapped+backfilling; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 89 op/s; 26869/129474 objects misplaced (20.752%); 124 MiB/s, 1.99k objects/s recovering 2024-04-29T14:46:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:47.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:154928 7.5 7:a08a3035:::benchmark_data_smithi194_147200_object40016:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:03.375607+0000", 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 103.793320885, 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.045787477, 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.173 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 154928 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.375607+0000", 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.375607+0000", 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.375609+0000", 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5210000000000001e-06 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.375617+0000", 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5959999999999998e-06 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.375629+0000", 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1708e-05 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.375637+0000", 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9950000000000005e-06 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.375685+0000", 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8946999999999997e-05 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.376413+0000", 2024-04-29T14:46:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000727083 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.376575+0000", 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000162282 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.381745+0000", 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051701259999999997 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.418808+0000", 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.037062603999999999 2024-04-29T14:46:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.418887+0000", 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9214000000000004e-05 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.421394+0000", 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025074009999999998 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.178 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:155316 7.5 7:a0fcf1c0:::benchmark_data_smithi194_147200_object40404:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:05.434190+0000", 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 101.73473798099999, 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.034294682999999999, 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 155316 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434190+0000", 2024-04-29T14:46:47.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434190+0000", 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434191+0000", 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.093e-06 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434195+0000", 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9500000000000003e-06 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434203+0000", 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1790000000000008e-06 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434208+0000", 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9710000000000003e-06 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434254+0000", 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5701000000000002e-05 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434342+0000", 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8115000000000006e-05 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.434410+0000", 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8238999999999995e-05 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.438058+0000", 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036478639999999998 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.465479+0000", 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.027421008 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.465520+0000", 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0930999999999998e-05 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:05.468484+0000", 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002964632 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:155906 7.5 7:a57ee16a:::benchmark_data_smithi194_147200_object40994:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:09.308682+0000", 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 97.860245793999994, 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035905419000000001, 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 155906 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.308682+0000", 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.308682+0000", 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.308684+0000", 2024-04-29T14:46:47.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.08e-06 2024-04-29T14:46:47.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.308690+0000", 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3300000000000004e-06 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.308700+0000", 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0197000000000001e-05 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.308708+0000", 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2309999999999999e-06 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.315456+0000", 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006748034 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.315594+0000", 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013837300000000001 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.315726+0000", 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013166600000000001 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.323227+0000", 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075013850000000002 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.343374+0000", 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020146404 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.343415+0000", 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1289999999999999e-05 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.344587+0000", 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001172429 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:155907 7.5 7:ab784584:::benchmark_data_smithi194_147200_object40995:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:09.314179+0000", 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 97.854748670999996, 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031654795999999999, 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.190 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 155907 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.314179+0000", 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.314179+0000", 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.314181+0000", 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.396e-06 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.314189+0000", 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6369999999999999e-06 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.314203+0000", 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4435e-05 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.314212+0000", 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8510000000000005e-06 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.320447+0000", 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006234369 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.320531+0000", 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4056999999999998e-05 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.320601+0000", 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0579000000000003e-05 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.323350+0000", 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027489889999999999 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.344635+0000", 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021285068000000001 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.344667+0000", 2024-04-29T14:46:47.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1356999999999998e-05 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:09.345834+0000", 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011670579999999999 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:156270 7.5 7:a1d865f3:::benchmark_data_smithi194_147200_object41358:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:11.927345+0000", 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 95.241582523999995, 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031589588000000002, 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.195 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 156270 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927345+0000", 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927345+0000", 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927348+0000", 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6390000000000001e-06 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927355+0000", 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1829999999999997e-06 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927367+0000", 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1590000000000001e-05 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927378+0000", 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1573000000000001e-05 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927438+0000", 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9444000000000002e-05 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927540+0000", 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010270800000000001 2024-04-29T14:46:47.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.927624+0000", 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4238000000000002e-05 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.931671+0000", 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040464710000000003 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.957671+0000", 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.026000559999999999 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.957722+0000", 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0513999999999998e-05 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:11.958935+0000", 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012126680000000001 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:156599 7.5 7:a2720515:::benchmark_data_smithi194_147200_object41687:head [delete] snapc 0=[] ondisk+write+known_if_redirected e846)", 2024-04-29T14:46:47.200 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:14.599960+0000", 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 92.568967698999998, 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.034937164999999999, 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 156599 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.599960+0000", 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.599960+0000", 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.599963+0000", 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.943e-06 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.599972+0000", 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1020000000000003e-06 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.599985+0000", 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3237e-05 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.599993+0000", 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7500000000000003e-06 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.604286+0000", 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042926429999999996 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.604437+0000", 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000151428 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.604593+0000", 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015596100000000001 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.608583+0000", 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039903830000000001 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.632792+0000", 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.024208785 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.632839+0000", 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6728999999999999e-05 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.634897+0000", 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020582040000000001 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.205 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:156611 7.5 7:ac2964a1:::benchmark_data_smithi194_147200_object41699:head [delete] snapc 0=[] ondisk+write+known_if_redirected e846)", 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:14.674525+0000", 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 92.494402966999999, 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.038918126999999997, 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 156611 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.674525+0000", 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.674525+0000", 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.674526+0000", 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.621e-06 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.674531+0000", 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9509999999999997e-06 2024-04-29T14:46:47.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.674540+0000", 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0780000000000002e-06 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.674545+0000", 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1309999999999997e-06 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.678248+0000", 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037027980000000002 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.678350+0000", 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010186199999999999 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.678435+0000", 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4685999999999995e-05 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.690518+0000", 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012083445 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.711179+0000", 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020660333999999999 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.711260+0000", 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1094999999999995e-05 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.713443+0000", 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021831260000000001 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:156616 7.5 7:afdffb2d:::benchmark_data_smithi194_147200_object41704:head [delete] snapc 0=[] ondisk+write+known_if_redirected e846)", 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:14.757239+0000", 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 92.411688268999995, 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035161452000000003, 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 156616 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.757239+0000", 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.757239+0000", 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.757242+0000", 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3080000000000002e-06 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.757248+0000", 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3169999999999998e-06 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.757261+0000", 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3093e-05 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.757269+0000", 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.858e-06 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.764397+0000", 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0071282810000000002 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.764589+0000", 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019162699999999999 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.764750+0000", 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016117700000000001 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.768690+0000", 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003940313 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.790225+0000", 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021534899999999999 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.790275+0000", 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9481999999999998e-05 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.792401+0000", 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021260960000000001 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:157270 7.5 7:a93d9254:::benchmark_data_smithi194_147200_object42358:head [delete] snapc 0=[] ondisk+write+known_if_redirected e847)", 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:19.212355+0000", 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 87.956572463000001, 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.037799857999999999, 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.216 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 157270 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.212355+0000", 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.212355+0000", 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.212357+0000", 2024-04-29T14:46:47.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9649999999999998e-06 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.212363+0000", 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.212374+0000", 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1259e-05 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.212383+0000", 2024-04-29T14:46:47.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4009999999999997e-06 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.224608+0000", 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012225026 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.224752+0000", 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000143984 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.224884+0000", 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000132593 2024-04-29T14:46:47.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.229476+0000", 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045919070000000001 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.248787+0000", 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019310559000000001 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.248857+0000", 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0099999999999996e-05 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.250155+0000", 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012981539999999999 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:157927 7.5 7:a89e4af6:::benchmark_data_smithi194_147200_object43015:head [delete] snapc 0=[] ondisk+write+known_if_redirected e850)", 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:23.875767+0000", 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 83.293161015999999, 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031379816999999997, 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.221 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 157927 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.875767+0000", 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.875767+0000", 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.875769+0000", 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9069999999999999e-06 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.875775+0000", 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.584e-06 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.875786+0000", 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1243999999999999e-05 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.875793+0000", 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2949999999999999e-06 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.881143+0000", 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053501859999999998 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.881276+0000", 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000133115 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.881369+0000", 2024-04-29T14:46:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2907999999999999e-05 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.887159+0000", 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057898919999999996 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.904827+0000", 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017668651 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.904880+0000", 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2401999999999997e-05 2024-04-29T14:46:47.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.907146+0000", 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022666330000000001 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:157941 7.5 7:a1b810e3:::benchmark_data_smithi194_147200_object43029:head [delete] snapc 0=[] ondisk+write+known_if_redirected e850)", 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:23.921474+0000", 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 83.247453574000005, 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.032253651000000001, 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.226 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 157941 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.921474+0000", 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.921474+0000", 2024-04-29T14:46:47.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.921476+0000", 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1009999999999999e-06 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.921483+0000", 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0090000000000001e-06 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.921495+0000", 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1934e-05 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.921504+0000", 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3840000000000003e-06 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.926035+0000", 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045306469999999996 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.926137+0000", 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010155899999999999 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.926220+0000", 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3343999999999994e-05 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.934277+0000", 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0080573959999999997 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.951740+0000", 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017462617 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.951784+0000", 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4422000000000003e-05 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:23.953728+0000", 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019432379999999999 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(7.f 854/838 [PushOp(7:f87996ed:::benchmark_data_smithi194_147200_object98293:head, version: 675'6231, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:f87996ed:::benchmark_data_smithi194_147200_object98293:head@675'6231, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.064341+0000", 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.104586398999999, 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21117528599999999, 2024-04-29T14:46:47.231 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064341+0000", 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064341+0000", 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064343+0000", 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.939e-06 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064372+0000", 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8362000000000001e-05 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064386+0000", 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4302e-05 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064398+0000", 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1934e-05 2024-04-29T14:46:47.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064458+0000", 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0507999999999997e-05 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064466+0000", 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2309999999999996e-06 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275517+0000", 2024-04-29T14:46:47.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21105001000000001 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:159332 7.a 7:54b0231b:::benchmark_data_smithi194_147200_object44420:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.065002+0000", 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.103925513999997, 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21070470599999999, 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.235 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 159332 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065002+0000", 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065002+0000", 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065004+0000", 2024-04-29T14:46:47.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2330000000000001e-06 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065010+0000", 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6280000000000002e-06 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065020+0000", 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9830000000000008e-06 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065026+0000", 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.891e-06 2024-04-29T14:46:47.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065070+0000", 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3835999999999998e-05 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065156+0000", 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6051000000000007e-05 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 2", 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065231+0000", 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5686999999999995e-05 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275516+0000", 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21028424200000001 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275616+0000", 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000100148 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275665+0000", 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9237000000000003e-05 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275707+0000", 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1770000000000002e-05 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(7.5 854/838 [PushReplyOp(7:aa6deea7:::benchmark_data_smithi194_147200_object86433:head)])", 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.103433073999994, 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20996125199999999, 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.75e-07 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065499+0000", 2024-04-29T14:46:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4309999999999998e-06 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065505+0000", 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.066e-06 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065509+0000", 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3699999999999997e-06 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065547+0000", 2024-04-29T14:46:47.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.799e-05 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275456+0000", 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20990882 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.26998.0:159332 7.a e854/655 ondisk, result = 0)", 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.069827+0000", 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 70.099101081000001, 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.205886185, 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.243 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069827+0000", 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069827+0000", 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069830+0000", 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7299999999999999e-06 2024-04-29T14:46:47.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069844+0000", 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4127e-05 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275400+0000", 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20555575000000001 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275424+0000", 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3536999999999998e-05 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275582+0000", 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015807000000000001 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275611+0000", 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8892000000000001e-05 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275713+0000", 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000102079 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.246 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:159339 7.5 7:a0a8928e:::benchmark_data_smithi194_147200_object44427:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.248782+0000", 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 69.920145581, 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.046886219999999999, 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 159339 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.248782+0000", 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.248782+0000", 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.248785+0000", 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6139999999999999e-06 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.248792+0000", 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7349999999999996e-06 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275374+0000", 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.026581925999999999 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275399+0000", 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4598999999999999e-05 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.281995+0000", 2024-04-29T14:46:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065962929999999996 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.282107+0000", 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000111538 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 1", 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.282199+0000", 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2337999999999996e-05 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.286144+0000", 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039448979999999996 2024-04-29T14:46:47.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.294827+0000", 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086826549999999992 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.294894+0000", 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7401000000000001e-05 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.295668+0000", 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00077422299999999999 2024-04-29T14:46:47.251 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(7.a 878/877 [PushReplyOp(7:50de555b:::benchmark_data_smithi194_147200_object95615:head)])", 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6571319679999998, 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060137652999999999, 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:46:47.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1e-07 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511798+0000", 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2010000000000002e-06 2024-04-29T14:46:47.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511802+0000", 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3960000000000002e-06 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511807+0000", 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2070000000000003e-06 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511848+0000", 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1576000000000003e-05 2024-04-29T14:46:47.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.571933+0000", 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060084962999999998 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(7.c 878/877 [PushOp(7:30f3e7b0:::benchmark_data_smithi194_147200_object109670:head, version: 681'6869, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:30f3e7b0:::benchmark_data_smithi194_147200_object109670:head@681'6869, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:38.526886+0000", 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6420419580000001, 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.045385613999999998, 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.255 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.526886+0000", 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.526886+0000", 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.526889+0000", 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1870000000000001e-06 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.526954+0000", 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5485000000000001e-05 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.571939+0000", 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.044984435000000003 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.571961+0000", 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2433999999999999e-05 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.572049+0000", 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7825999999999996e-05 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.572065+0000", 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6427999999999999e-05 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.572271+0000", 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00020581899999999999 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.258 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(7.a 878/877 [PushReplyOp(7:51a99d8f:::benchmark_data_smithi194_147200_object50259:head)])", 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:39.167086+0000", 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.0018412730000001, 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.058698567, 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167086+0000", 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167086+0000", 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167087+0000", 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2500000000000001e-07 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167089+0000", 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5220000000000002e-06 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167093+0000", 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2069999999999997e-06 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167099+0000", 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4020000000000001e-06 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167127+0000", 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8045999999999999e-05 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.225785+0000", 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.058658065000000002 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:164443 7.a 7:54ce021b:::benchmark_data_smithi194_147200_object49531:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:39.179844+0000", 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 7.9890837660000003, 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.051124180999999998, 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:46:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 164443 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.179844+0000", 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.179844+0000", 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.179846+0000", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1600000000000001e-06 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.179852+0000", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1140000000000002e-06 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.225536+0000", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.045684238000000002 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.225551+0000", 2024-04-29T14:46:47.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4337e-05 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.225817+0000", 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00026657100000000001 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.225968+0000", 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015060299999999999 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 2,5", 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.226071+0000", 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103306 2024-04-29T14:46:47.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.230738+0000", 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046667870000000004 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.230776+0000", 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8475e-05 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.230794+0000", 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8031999999999999e-05 2024-04-29T14:46:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.230855+0000", 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0111000000000001e-05 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.230968+0000", 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000113447 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:46:47.267 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:46:47.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:47.301 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:47.377 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:47.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:47.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:47.529 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:47.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:47.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:47.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:47.731 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:47.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:47.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:46:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:47 smithi194 ceph-mon[44089]: osdmap e881: 12 total, 12 up, 9 in 2024-04-29T14:46:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1956548053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:47 smithi194 ceph-mon[44089]: from='client.27607 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:47.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:47.913 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:46:47.916 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:47.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:47.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:48.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:48.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:47 smithi121 ceph-mon[44533]: osdmap e881: 12 total, 12 up, 9 in 2024-04-29T14:46:48.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1956548053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:48.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:47 smithi121 ceph-mon[44533]: from='client.27607 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:48.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:48.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:47 smithi040 ceph-mon[39581]: osdmap e881: 12 total, 12 up, 9 in 2024-04-29T14:46:48.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1956548053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:48.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:47 smithi040 ceph-mon[39581]: from='client.27607 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:46:48.128 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:48.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:48.234 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:48.270 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:48.289 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:46:48.293 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:48.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:48.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:48.436 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:48.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:48.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:48.603 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 1, 5, 8] out_osds: [3, 11, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:46:48.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:46:48.603 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-04-29T14:46:48.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 3 2024-04-29T14:46:48.631 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:48.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:48.756 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:46:48.788 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:46:48.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:48.827 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:46:48.861 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:46:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:48.902 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:46:48.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:48.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:48.934 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:46:48.968 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:46:48.992 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:46:49.022 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:46:49.024 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:49.048 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:46:49.055 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:49.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:48 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:49.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:48 smithi194 ceph-mon[44089]: from='client.27607 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:46:49.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:48 smithi194 ceph-mon[44089]: osdmap e882: 12 total, 12 up, 9 in 2024-04-29T14:46:49.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:48 smithi194 ceph-mon[44089]: pgmap v2263: 17 pgs: 1 active+clean+remapped, 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 5 active+remapped+backfilling; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 118 op/s; 24210/129186 objects misplaced (18.740%); 125 MiB/s, 1.99k objects/s recovering 2024-04-29T14:46:49.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/513159117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:49.077 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:46:49.100 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:46:49.102 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:46:49.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:48 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:49.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:48 smithi040 ceph-mon[39581]: from='client.27607 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:46:49.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:48 smithi040 ceph-mon[39581]: osdmap e882: 12 total, 12 up, 9 in 2024-04-29T14:46:49.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:48 smithi040 ceph-mon[39581]: pgmap v2263: 17 pgs: 1 active+clean+remapped, 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 5 active+remapped+backfilling; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 118 op/s; 24210/129186 objects misplaced (18.740%); 125 MiB/s, 1.99k objects/s recovering 2024-04-29T14:46:49.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/513159117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:49.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:48 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:49.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:48 smithi121 ceph-mon[44533]: from='client.27607 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:46:49.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:48 smithi121 ceph-mon[44533]: osdmap e882: 12 total, 12 up, 9 in 2024-04-29T14:46:49.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:48 smithi121 ceph-mon[44533]: pgmap v2263: 17 pgs: 1 active+clean+remapped, 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 5 active+remapped+backfilling; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 118 op/s; 24210/129186 objects misplaced (18.740%); 125 MiB/s, 1.99k objects/s recovering 2024-04-29T14:46:49.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/513159117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:49.155 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:49.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:49.259 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:49.285 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:49.734 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:49.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:49.840 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:49.899 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:49.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:50.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:50.009 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:50.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:50.046 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:50.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:50.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:50.198 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:50.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:50.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:50.329 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:50.361 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:50.372 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:50.372 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:50.383 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:50.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:50.487 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:50.515 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:50.612 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:50.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:50.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:50.745 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:50.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:50.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:50.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:50.913 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:50.940 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:50.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:51.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:51.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:51.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:50 smithi194 ceph-mon[44089]: osdmap e883: 12 total, 12 up, 9 in 2024-04-29T14:46:51.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:50 smithi194 ceph-mon[44089]: pgmap v2265: 17 pgs: 2 active+clean+remapped, 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 4 active+remapped+backfilling; 3.9 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 22457/129032 objects misplaced (17.404%); 140 MiB/s, 2.23k objects/s recovering 2024-04-29T14:46:51.077 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:51.096 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:51.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:51.104 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:46:51.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:51.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:50 smithi040 ceph-mon[39581]: osdmap e883: 12 total, 12 up, 9 in 2024-04-29T14:46:51.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:50 smithi040 ceph-mon[39581]: pgmap v2265: 17 pgs: 2 active+clean+remapped, 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 4 active+remapped+backfilling; 3.9 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 22457/129032 objects misplaced (17.404%); 140 MiB/s, 2.23k objects/s recovering 2024-04-29T14:46:51.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:51.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:50 smithi121 ceph-mon[44533]: osdmap e883: 12 total, 12 up, 9 in 2024-04-29T14:46:51.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:50 smithi121 ceph-mon[44533]: pgmap v2265: 17 pgs: 2 active+clean+remapped, 2 active+recovering+remapped, 7 active+clean, 2 active+recovering, 4 active+remapped+backfilling; 3.9 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 22457/129032 objects misplaced (17.404%); 140 MiB/s, 2.23k objects/s recovering 2024-04-29T14:46:51.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:46:51.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:51.254 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:51.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:51.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:51.415 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:46:51.497 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:51.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:51.593 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:51.605 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:51.632 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:51.651 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:51.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:51.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:51.788 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:51.892 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:51.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:51.914 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:51 smithi194 ceph-mon[44089]: osdmap e884: 12 total, 12 up, 9 in 2024-04-29T14:46:51.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:46:51.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:52.025 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:52.046 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:52.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:52.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:51 smithi040 ceph-mon[39581]: osdmap e884: 12 total, 12 up, 9 in 2024-04-29T14:46:52.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:51 smithi121 ceph-mon[44533]: osdmap e884: 12 total, 12 up, 9 in 2024-04-29T14:46:52.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:52.191 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:52.219 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:52.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:52.327 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:52.356 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:52.431 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:52.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:52.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:52.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:52.569 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:52.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:52.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:52.695 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:52.721 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:52.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:52 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:46:52.537+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e885 definitely_dead 0 2024-04-29T14:46:52.862 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.3. 2024-04-29T14:46:52.920 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:52.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:53.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:53.053 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:52 smithi040 ceph-mon[39581]: osdmap e885: 12 total, 12 up, 9 in 2024-04-29T14:46:53.053 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:52 smithi040 ceph-mon[39581]: pgmap v2268: 17 pgs: 2 peering, 2 active+clean+remapped, 2 active+recovering+remapped, 9 active+clean, 2 active+recovering; 3.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 132 op/s; 16205/128218 objects misplaced (12.639%); 114 MiB/s, 1.82k objects/s recovering 2024-04-29T14:46:53.053 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/990888833' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:46:53.053 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:52 smithi040 ceph-mon[39581]: from='client.27616 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:52 smithi194 ceph-mon[44089]: osdmap e885: 12 total, 12 up, 9 in 2024-04-29T14:46:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:52 smithi194 ceph-mon[44089]: pgmap v2268: 17 pgs: 2 peering, 2 active+clean+remapped, 2 active+recovering+remapped, 9 active+clean, 2 active+recovering; 3.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 132 op/s; 16205/128218 objects misplaced (12.639%); 114 MiB/s, 1.82k objects/s recovering 2024-04-29T14:46:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/990888833' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:46:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:52 smithi194 ceph-mon[44089]: from='client.27616 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:46:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:53.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:53.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:52 smithi121 ceph-mon[44533]: osdmap e885: 12 total, 12 up, 9 in 2024-04-29T14:46:53.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:52 smithi121 ceph-mon[44533]: pgmap v2268: 17 pgs: 2 peering, 2 active+clean+remapped, 2 active+recovering+remapped, 9 active+clean, 2 active+recovering; 3.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 132 op/s; 16205/128218 objects misplaced (12.639%); 114 MiB/s, 1.82k objects/s recovering 2024-04-29T14:46:53.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/990888833' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:46:53.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:52 smithi121 ceph-mon[44533]: from='client.27616 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-29T14:46:53.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:46:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:53.500 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:53.501 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:46:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:53.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:46:53.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:53.812 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:53.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:53.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:53.888 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:53.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:46:53.964 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:53.986 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:46:53.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:53 smithi040 ceph-mon[39581]: from='client.27616 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-29T14:46:53.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:53 smithi040 ceph-mon[39581]: osdmap e886: 12 total, 12 up, 10 in 2024-04-29T14:46:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:53.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:46:54.091 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:46:54.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:53 smithi194 ceph-mon[44089]: from='client.27616 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-29T14:46:54.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:53 smithi194 ceph-mon[44089]: osdmap e886: 12 total, 12 up, 10 in 2024-04-29T14:46:54.121 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:54.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:53 smithi121 ceph-mon[44533]: from='client.27616 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-29T14:46:54.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:53 smithi121 ceph-mon[44533]: osdmap e886: 12 total, 12 up, 10 in 2024-04-29T14:46:54.220 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:46:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:54.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:46:54.245 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:46:54.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:54.348 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-04-29T14:46:54.361 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:54.545 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:46:54.562 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:54.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:54.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:54.696 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:54.847 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:54.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:54.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:46:54.976 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:55.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:46:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:55.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:46:55.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:54 smithi194 ceph-mon[44089]: osdmap e887: 12 total, 12 up, 10 in 2024-04-29T14:46:55.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:54 smithi194 ceph-mon[44089]: pgmap v2271: 17 pgs: 3 unknown, 2 peering, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 16125/115766 objects misplaced (13.929%) 2024-04-29T14:46:55.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:54 smithi194 ceph-mon[44089]: osdmap e888: 12 total, 12 up, 10 in 2024-04-29T14:46:55.105 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:46:55.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:54 smithi040 ceph-mon[39581]: osdmap e887: 12 total, 12 up, 10 in 2024-04-29T14:46:55.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:54 smithi040 ceph-mon[39581]: pgmap v2271: 17 pgs: 3 unknown, 2 peering, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 16125/115766 objects misplaced (13.929%) 2024-04-29T14:46:55.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:54 smithi040 ceph-mon[39581]: osdmap e888: 12 total, 12 up, 10 in 2024-04-29T14:46:55.134 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:55.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:54 smithi121 ceph-mon[44533]: osdmap e887: 12 total, 12 up, 10 in 2024-04-29T14:46:55.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:54 smithi121 ceph-mon[44533]: pgmap v2271: 17 pgs: 3 unknown, 2 peering, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 16125/115766 objects misplaced (13.929%) 2024-04-29T14:46:55.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:54 smithi121 ceph-mon[44533]: osdmap e888: 12 total, 12 up, 10 in 2024-04-29T14:46:55.483 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:46:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:55.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:46:55.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:55.686 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:56.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:55 smithi040 ceph-mon[39581]: 7.3 starting backfill to osd.3 from (0'0,0'0] MAX to 885'13925 2024-04-29T14:46:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:55 smithi040 ceph-mon[39581]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10189 2024-04-29T14:46:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:55 smithi040 ceph-mon[39581]: 7.a starting backfill to osd.3 from (0'0,0'0] MAX to 885'17724 2024-04-29T14:46:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:55 smithi040 ceph-mon[39581]: 7.c starting backfill to osd.3 from (0'0,761'8356] MIN to 885'10382 2024-04-29T14:46:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:55 smithi040 ceph-mon[39581]: 7.2 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10244 2024-04-29T14:46:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:55 smithi040 ceph-mon[39581]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10668 2024-04-29T14:46:56.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:55 smithi121 ceph-mon[44533]: 7.3 starting backfill to osd.3 from (0'0,0'0] MAX to 885'13925 2024-04-29T14:46:56.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:55 smithi121 ceph-mon[44533]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10189 2024-04-29T14:46:56.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:55 smithi121 ceph-mon[44533]: 7.a starting backfill to osd.3 from (0'0,0'0] MAX to 885'17724 2024-04-29T14:46:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:55 smithi121 ceph-mon[44533]: 7.c starting backfill to osd.3 from (0'0,761'8356] MIN to 885'10382 2024-04-29T14:46:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:55 smithi121 ceph-mon[44533]: 7.2 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10244 2024-04-29T14:46:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:55 smithi121 ceph-mon[44533]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10668 2024-04-29T14:46:56.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:55 smithi194 ceph-mon[44089]: 7.3 starting backfill to osd.3 from (0'0,0'0] MAX to 885'13925 2024-04-29T14:46:56.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:55 smithi194 ceph-mon[44089]: 7.1 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10189 2024-04-29T14:46:56.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:55 smithi194 ceph-mon[44089]: 7.a starting backfill to osd.3 from (0'0,0'0] MAX to 885'17724 2024-04-29T14:46:56.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:55 smithi194 ceph-mon[44089]: 7.c starting backfill to osd.3 from (0'0,761'8356] MIN to 885'10382 2024-04-29T14:46:56.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:55 smithi194 ceph-mon[44089]: 7.2 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10244 2024-04-29T14:46:56.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:55 smithi194 ceph-mon[44089]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 885'10668 2024-04-29T14:46:56.453 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:56.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:56.559 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:56.592 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:56.671 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:56.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:56.707 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:46:56.707 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:46:56.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:56.810 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:56.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:56.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:56.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:46:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:57.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:56 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:46:57.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:56 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:46:57.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:56 smithi194 ceph-mon[44089]: osdmap e889: 12 total, 12 up, 10 in 2024-04-29T14:46:57.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:56 smithi194 ceph-mon[44089]: pgmap v2274: 17 pgs: 3 active+remapped+backfilling, 2 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 28094/123258 objects misplaced (22.793%) 2024-04-29T14:46:57.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:56 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:46:57.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:56 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:46:57.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:56 smithi040 ceph-mon[39581]: osdmap e889: 12 total, 12 up, 10 in 2024-04-29T14:46:57.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:56 smithi040 ceph-mon[39581]: pgmap v2274: 17 pgs: 3 active+remapped+backfilling, 2 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 28094/123258 objects misplaced (22.793%) 2024-04-29T14:46:57.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:56 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:46:57.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:56 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:46:57.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:56 smithi121 ceph-mon[44533]: osdmap e889: 12 total, 12 up, 10 in 2024-04-29T14:46:57.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:56 smithi121 ceph-mon[44533]: pgmap v2274: 17 pgs: 3 active+remapped+backfilling, 2 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 28094/123258 objects misplaced (22.793%) 2024-04-29T14:46:57.526 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:46:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:46:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:57.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:46:57.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:46:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:57.857 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:46:58.057 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:46:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:58.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:46:58.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:46:58.194 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:58.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:46:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:58.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:46:58.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:46:58.358 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:58.382 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:46:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:58.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:46:58.393 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:46:58.423 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:46:58.455 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:46:58.486 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:46:58.487 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:46:58.518 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:58.559 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:46:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/729087227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:46:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:58 smithi194 ceph-mon[44089]: pgmap v2275: 17 pgs: 4 active+remapped+backfilling, 1 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 31586/119054 objects misplaced (26.531%) 2024-04-29T14:46:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/845767096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:58.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/729087227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:46:58.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:58 smithi040 ceph-mon[39581]: pgmap v2275: 17 pgs: 4 active+remapped+backfilling, 1 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 31586/119054 objects misplaced (26.531%) 2024-04-29T14:46:58.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/845767096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/729087227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:46:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:58 smithi121 ceph-mon[44533]: pgmap v2275: 17 pgs: 4 active+remapped+backfilling, 1 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 31586/119054 objects misplaced (26.531%) 2024-04-29T14:46:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/845767096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:46:59.348 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 1, 5, 8, 3] out_osds: [11, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:46:59.349 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:46:59.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:46:59.469 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:46:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:59.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:46:59.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:46:59.613 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:59.639 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:46:59.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:59 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:59.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/729087227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:46:59.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:46:59 smithi040 ceph-mon[39581]: osdmap e890: 12 total, 12 up, 10 in 2024-04-29T14:46:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:59.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:46:59.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.720 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.720 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:59 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:59.720 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/729087227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:46:59.720 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:46:59 smithi194 ceph-mon[44089]: osdmap e890: 12 total, 12 up, 10 in 2024-04-29T14:46:59.744 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:46:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:46:59.776 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:46:59.877 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:46:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:46:59.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:46:59.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:46:59.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:59 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:46:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/729087227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:46:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:46:59 smithi121 ceph-mon[44533]: osdmap e890: 12 total, 12 up, 10 in 2024-04-29T14:46:59.980 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:00.012 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:00.031 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:47:00.031 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:47:00.033 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:00.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:00.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:00.170 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:00.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:00.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:00.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:00.342 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:00.556 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:00.556 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:00.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:00 smithi040 ceph-mon[39581]: pgmap v2277: 17 pgs: 4 active+remapped+backfilling, 1 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 2.09k op/s; 31586/119004 objects misplaced (26.542%); 235 MiB/s, 4.64k objects/s recovering 2024-04-29T14:47:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:00 smithi194 ceph-mon[44089]: pgmap v2277: 17 pgs: 4 active+remapped+backfilling, 1 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 2.09k op/s; 31586/119004 objects misplaced (26.542%); 235 MiB/s, 4.64k objects/s recovering 2024-04-29T14:47:00.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:00 smithi121 ceph-mon[44533]: pgmap v2277: 17 pgs: 4 active+remapped+backfilling, 1 unknown, 2 active+recovering+remapped, 8 active+clean, 2 active+recovering; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 2.09k op/s; 31586/119004 objects misplaced (26.542%); 235 MiB/s, 4.64k objects/s recovering 2024-04-29T14:47:01.052 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:47:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:01.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:01.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:01.349 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:01.355 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:47:01.435 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:47:01.448 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:47:01.468 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:47:01.498 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:47:01.533 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:47:01.561 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:01.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:01.567 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:47:01.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:47:01.639 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:47:01.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:01.679 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:47:01.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:47:01.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:47:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:47:01.717 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:01.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:01 smithi194 ceph-mon[44089]: osdmap e891: 12 total, 12 up, 10 in 2024-04-29T14:47:01.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:01 smithi040 ceph-mon[39581]: osdmap e891: 12 total, 12 up, 10 in 2024-04-29T14:47:01.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:01 smithi121 ceph-mon[44533]: osdmap e891: 12 total, 12 up, 10 in 2024-04-29T14:47:02.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:02.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:02.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.754 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osdmap e892: 12 total, 12 up, 10 in 2024-04-29T14:47:02.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: pgmap v2280: 17 pgs: 1 activating+degraded+remapped, 5 active+remapped+backfilling, 2 active+recovering+remapped, 8 active+clean, 1 active+recovering; 3.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 75 MiB/s wr, 2.10k op/s; 8756/126854 objects degraded (6.902%); 18211/126854 objects misplaced (14.356%); 2484/63427 objects unfound (3.916%); 347 MiB/s, 6.47k objects/s recovering 2024-04-29T14:47:02.755 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:02 smithi194 ceph-mon[44089]: osdmap e893: 12 total, 12 up, 10 in 2024-04-29T14:47:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osdmap e892: 12 total, 12 up, 10 in 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: pgmap v2280: 17 pgs: 1 activating+degraded+remapped, 5 active+remapped+backfilling, 2 active+recovering+remapped, 8 active+clean, 1 active+recovering; 3.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 75 MiB/s wr, 2.10k op/s; 8756/126854 objects degraded (6.902%); 18211/126854 objects misplaced (14.356%); 2484/63427 objects unfound (3.916%); 347 MiB/s, 6.47k objects/s recovering 2024-04-29T14:47:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:02 smithi040 ceph-mon[39581]: osdmap e893: 12 total, 12 up, 10 in 2024-04-29T14:47:02.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osdmap e892: 12 total, 12 up, 10 in 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: pgmap v2280: 17 pgs: 1 activating+degraded+remapped, 5 active+remapped+backfilling, 2 active+recovering+remapped, 8 active+clean, 1 active+recovering; 3.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 75 MiB/s wr, 2.10k op/s; 8756/126854 objects degraded (6.902%); 18211/126854 objects misplaced (14.356%); 2484/63427 objects unfound (3.916%); 347 MiB/s, 6.47k objects/s recovering 2024-04-29T14:47:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:02 smithi121 ceph-mon[44533]: osdmap e893: 12 total, 12 up, 10 in 2024-04-29T14:47:02.981 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:02.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:03.007 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:47:03.010 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:47:03.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:03.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:03.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:03.260 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:03.294 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:03.353 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:03.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:03.382 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:47:03.382 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:47:03.382 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:47:03.382 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:47:03.382 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:47:03.458 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:03.488 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:03.641 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:03 smithi194 ceph-mon[44089]: Health check failed: 2484/63427 objects unfound (3.916%) (OBJECT_UNFOUND) 2024-04-29T14:47:03.641 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:03 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 8756/126854 objects degraded (6.902%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:47:03.641 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:03 smithi194 ceph-mon[44089]: from='client.27631 v1:172.21.15.40:0/217638177' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:47:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:03.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:03.656 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:03.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:03.790 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:03.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:03.824 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:03 smithi121 ceph-mon[44533]: Health check failed: 2484/63427 objects unfound (3.916%) (OBJECT_UNFOUND) 2024-04-29T14:47:03.824 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:03 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 8756/126854 objects degraded (6.902%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:47:03.824 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:03 smithi121 ceph-mon[44533]: from='client.27631 v1:172.21.15.40:0/217638177' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:47:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:03.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:03.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:03 smithi040 ceph-mon[39581]: Health check failed: 2484/63427 objects unfound (3.916%) (OBJECT_UNFOUND) 2024-04-29T14:47:03.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:03 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 8756/126854 objects degraded (6.902%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:47:03.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:03 smithi040 ceph-mon[39581]: from='client.27631 v1:172.21.15.40:0/217638177' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:47:03.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:03.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:03.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:04.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:04.143 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:04.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:04.202 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:47:04.274 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:04.305 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:04.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:04.434 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:04.473 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:04.499 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:47:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:04.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:04.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:04.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:04.663 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:04.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:04 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:47:04.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:04 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:47:04.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:04 smithi194 ceph-mon[44089]: pgmap v2282: 17 pgs: 1 activating+degraded+remapped, 5 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 79 MiB/s wr, 1.84k op/s; 8756/126854 objects degraded (6.902%); 18107/126854 objects misplaced (14.274%); 2484/63427 objects unfound (3.916%); 203 MiB/s, 3.33k objects/s recovering 2024-04-29T14:47:04.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:04 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:47:04.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:04 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:47:04.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:04 smithi040 ceph-mon[39581]: pgmap v2282: 17 pgs: 1 activating+degraded+remapped, 5 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 79 MiB/s wr, 1.84k op/s; 8756/126854 objects degraded (6.902%); 18107/126854 objects misplaced (14.274%); 2484/63427 objects unfound (3.916%); 203 MiB/s, 3.33k objects/s recovering 2024-04-29T14:47:04.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:04 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:47:04.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:04 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:47:04.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:04 smithi121 ceph-mon[44533]: pgmap v2282: 17 pgs: 1 activating+degraded+remapped, 5 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 79 MiB/s wr, 1.84k op/s; 8756/126854 objects degraded (6.902%); 18107/126854 objects misplaced (14.274%); 2484/63427 objects unfound (3.916%); 203 MiB/s, 3.33k objects/s recovering 2024-04-29T14:47:05.177 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2281,"stamp":"2024-04-29T14:47:02.536971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4157145656,"num_objects":63427,"num_object_clones":0,"num_object_copies":126854,"num_objects_missing_on_primary":10883,"num_objects_missing":10883,"num_objects_degraded":8756,"num_objects_misplaced":18211,"num_objects_unfound":2484,"num_objects_dirty":63427,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":166576,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190719,"num_bytes_recovered":11322294008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161079,"ondisk_log_size":161079,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":17067060,"kb_used_data":10607740,"kb_used_omap":668,"kb_used_meta":6458531,"kb_avail":920384460,"statfs":{"total":959950356480,"available":942473687040,"internally_reserved":0,"allocated":10862325760,"data_stored":10847952166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":6613536751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":255590400,"num_objects":3900,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":-542,"num_objects_missing":-542,"num_objects_degraded":8756,"num_objects_misplaced":-13375,"num_objects_unfound":2484,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10508,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18740,"num_bytes_recovered":1214382080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.273302"},"pg_stats":[{"pgid":"7.f","version":"889'10460","reported_seq":40162,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-04-29T14:46:58.547857+0000","last_change":"2024-04-29T14:46:16.794326+0000","last_active":"2024-04-29T14:46:58.547857+0000","last_peered":"2024-04-29T14:46:58.547857+0000","last_clean":"2024-04-29T14:46:58.547857+0000","last_became_active":"2024-04-29T14:44:46.574882+0000","last_became_peered":"2024-04-29T14:44:46.574882+0000","last_unstale":"2024-04-29T14:46:58.547857+0000","last_undegraded":"2024-04-29T14:46:58.547857+0000","last_fullsized":"2024-04-29T14:46:58.547857+0000","mapping_epoch":838,"log_start":"660'400","ondisk_log_start":"660'400","created":655,"last_epoch_clean":861,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'10283","last_scrub_stamp":"2024-04-29T14:46:16.794166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:46:16.794166+0000","objects_scrubbed":4165,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:47:40.754345+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261095424,"num_objects":3984,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10460,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14444,"num_bytes_recovered":773914624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"889'17604","reported_seq":35008,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-04-29T14:46:58.554017+0000","last_change":"2024-04-29T14:46:50.885684+0000","last_active":"2024-04-29T14:46:58.554017+0000","last_peered":"2024-04-29T14:46:58.554017+0000","last_clean":"2024-04-29T14:46:58.554017+0000","last_became_active":"2024-04-29T14:46:50.844970+0000","last_became_peered":"2024-04-29T14:46:50.844970+0000","last_unstale":"2024-04-29T14:46:58.554017+0000","last_undegraded":"2024-04-29T14:46:58.554017+0000","last_fullsized":"2024-04-29T14:46:58.554017+0000","mapping_epoch":883,"log_start":"703'7582","ondisk_log_start":"703'7582","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'16641","last_scrub_stamp":"2024-04-29T14:44:06.182304+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:44:06.182304+0000","objects_scrubbed":4965,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262275072,"num_objects":4002,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8209,"num_bytes_recovered":537985024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"889'10370","reported_seq":37541,"reported_epoch":889,"state":"active+recovering","last_fresh":"2024-04-29T14:46:58.109010+0000","last_change":"2024-04-29T14:42:42.691148+0000","last_active":"2024-04-29T14:46:58.109010+0000","last_peered":"2024-04-29T14:46:58.109010+0000","last_clean":"2024-04-29T14:42:40.867507+0000","last_became_active":"2024-04-29T14:42:42.638015+0000","last_became_peered":"2024-04-29T14:42:42.638015+0000","last_unstale":"2024-04-29T14:46:58.109010+0000","last_undegraded":"2024-04-29T14:46:58.109010+0000","last_fullsized":"2024-04-29T14:46:58.109010+0000","mapping_epoch":802,"log_start":"660'300","ondisk_log_start":"660'300","created":655,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"793'8773","last_scrub_stamp":"2024-04-29T14:42:23.541331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:42:23.541331+0000","objects_scrubbed":5553,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259194904,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10370,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13238,"num_bytes_recovered":768475160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"0,1","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"889'10392","reported_seq":62870,"reported_epoch":892,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:47:01.790935+0000","last_change":"2024-04-29T14:46:55.337835+0000","last_active":"2024-04-29T14:47:01.790935+0000","last_peered":"2024-04-29T14:47:01.790935+0000","last_clean":"2024-04-29T14:46:52.221330+0000","last_became_active":"2024-04-29T14:46:55.289520+0000","last_became_peered":"2024-04-29T14:46:55.289520+0000","last_unstale":"2024-04-29T14:47:01.790935+0000","last_undegraded":"2024-04-29T14:47:01.790935+0000","last_fullsized":"2024-04-29T14:47:01.790935+0000","mapping_epoch":887,"log_start":"660'300","ondisk_log_start":"660'300","created":655,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'10252","last_scrub_stamp":"2024-04-29T14:46:25.559263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:46:25.559263+0000","objects_scrubbed":4096,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259260416,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1816,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25820,"num_bytes_recovered":1405157376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[0,9],"avail_no_missing":["0","2","6","9","10"],"object_location_counts":[{"shards":"0,9","objects":3956}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"889'10353","reported_seq":20945,"reported_epoch":889,"state":"active+recovering+remapped","last_fresh":"2024-04-29T14:46:58.019935+0000","last_change":"2024-04-29T14:46:38.096595+0000","last_active":"2024-04-29T14:46:58.019935+0000","last_peered":"2024-04-29T14:46:58.019935+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:46:35.719521+0000","last_became_peered":"2024-04-29T14:46:35.719521+0000","last_unstale":"2024-04-29T14:46:58.019935+0000","last_undegraded":"2024-04-29T14:46:58.019935+0000","last_fullsized":"2024-04-29T14:46:58.019935+0000","mapping_epoch":869,"log_start":"660'300","ondisk_log_start":"660'300","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257490944,"num_objects":3929,"num_object_clones":0,"num_object_copies":7858,"num_objects_missing_on_primary":3101,"num_objects_missing":3101,"num_objects_degraded":0,"num_objects_misplaced":7808,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10353,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3663,"num_bytes_recovered":239992832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[1,5],"avail_no_missing":["10","11"],"object_location_counts":[{"shards":"1,5","objects":25},{"shards":"1,10,11","objects":803},{"shards":"10,11","objects":3101}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"888'17738","reported_seq":31069,"reported_epoch":890,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:47:00.375030+0000","last_change":"2024-04-29T14:46:55.316418+0000","last_active":"2024-04-29T14:47:00.375030+0000","last_peered":"2024-04-29T14:47:00.375030+0000","last_clean":"2024-04-29T14:46:52.792937+0000","last_became_active":"2024-04-29T14:46:55.289678+0000","last_became_peered":"2024-04-29T14:46:55.289678+0000","last_unstale":"2024-04-29T14:47:00.375030+0000","last_undegraded":"2024-04-29T14:47:00.375030+0000","last_fullsized":"2024-04-29T14:47:00.375030+0000","mapping_epoch":887,"log_start":"704'7700","ondisk_log_start":"704'7700","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'16714","last_scrub_stamp":"2024-04-29T14:44:09.518462+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:44:09.518462+0000","objects_scrubbed":5035,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":545,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262799360,"num_objects":4010,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1937,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10664,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6170,"num_bytes_recovered":404357120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,5],"avail_no_missing":["9","2","5"],"object_location_counts":[{"shards":"5,9","objects":4010}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"889'10409","reported_seq":39342,"reported_epoch":889,"state":"active+clean","last_fresh":"2024-04-29T14:46:56.879398+0000","last_change":"2024-04-29T14:40:30.568318+0000","last_active":"2024-04-29T14:46:56.879398+0000","last_peered":"2024-04-29T14:46:56.879398+0000","last_clean":"2024-04-29T14:46:56.879398+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:46:56.879398+0000","last_undegraded":"2024-04-29T14:46:56.879398+0000","last_fullsized":"2024-04-29T14:46:56.879398+0000","mapping_epoch":756,"log_start":"660'400","ondisk_log_start":"660'400","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'7744","last_scrub_stamp":"2024-04-29T14:38:41.906760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:38:41.906760+0000","objects_scrubbed":6597,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257622016,"num_objects":3931,"num_object_clones":0,"num_object_copies":7862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10409,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"889'10376","reported_seq":19160,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-04-29T14:47:00.906867+0000","last_change":"2024-04-29T14:46:50.882072+0000","last_active":"2024-04-29T14:47:00.906867+0000","last_peered":"2024-04-29T14:47:00.906867+0000","last_clean":"2024-04-29T14:47:00.906867+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:47:00.906867+0000","last_undegraded":"2024-04-29T14:47:00.906867+0000","last_fullsized":"2024-04-29T14:47:00.906867+0000","mapping_epoch":883,"log_start":"660'300","ondisk_log_start":"660'300","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'9359","last_scrub_stamp":"2024-04-29T14:44:09.099317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:09.099317+0000","objects_scrubbed":4931,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256507904,"num_objects":3914,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"889'10679","reported_seq":58193,"reported_epoch":890,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:47:00.377244+0000","last_change":"2024-04-29T14:46:55.325997+0000","last_active":"2024-04-29T14:47:00.377244+0000","last_peered":"2024-04-29T14:47:00.377244+0000","last_clean":"2024-04-29T14:46:52.370767+0000","last_became_active":"2024-04-29T14:46:55.302442+0000","last_became_peered":"2024-04-29T14:46:55.302442+0000","last_unstale":"2024-04-29T14:47:00.377244+0000","last_undegraded":"2024-04-29T14:47:00.377244+0000","last_fullsized":"2024-04-29T14:47:00.377244+0000","mapping_epoch":887,"log_start":"660'600","ondisk_log_start":"660'600","created":655,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"866'10501","last_scrub_stamp":"2024-04-29T14:46:17.756623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:46:17.756623+0000","objects_scrubbed":4227,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265355264,"num_objects":4049,"num_object_clones":0,"num_object_copies":8098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1913,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23343,"num_bytes_recovered":1404502016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,1],"avail_no_missing":["8","1","6"],"object_location_counts":[{"shards":"1,8","objects":4049}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"889'10374","reported_seq":39809,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-04-29T14:46:58.553902+0000","last_change":"2024-04-29T14:46:25.228786+0000","last_active":"2024-04-29T14:46:58.553902+0000","last_peered":"2024-04-29T14:46:58.553902+0000","last_clean":"2024-04-29T14:46:58.553902+0000","last_became_active":"2024-04-29T14:44:46.662215+0000","last_became_peered":"2024-04-29T14:44:46.662215+0000","last_unstale":"2024-04-29T14:46:58.553902+0000","last_undegraded":"2024-04-29T14:46:58.553902+0000","last_fullsized":"2024-04-29T14:46:58.553902+0000","mapping_epoch":838,"log_start":"660'300","ondisk_log_start":"660'300","created":655,"last_epoch_clean":863,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'10230","last_scrub_stamp":"2024-04-29T14:46:25.228620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:46:25.228620+0000","objects_scrubbed":4044,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:47:33.419657+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255590400,"num_objects":3900,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14274,"num_bytes_recovered":668729344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"889'10513","reported_seq":32264,"reported_epoch":891,"state":"active+recovering+remapped","last_fresh":"2024-04-29T14:47:01.389031+0000","last_change":"2024-04-29T14:46:30.529924+0000","last_active":"2024-04-29T14:47:01.389031+0000","last_peered":"2024-04-29T14:47:01.389031+0000","last_clean":"2024-04-29T14:44:19.702641+0000","last_became_active":"2024-04-29T14:46:30.195894+0000","last_became_peered":"2024-04-29T14:46:30.195894+0000","last_unstale":"2024-04-29T14:47:01.389031+0000","last_undegraded":"2024-04-29T14:47:01.389031+0000","last_fullsized":"2024-04-29T14:47:01.389031+0000","mapping_epoch":869,"log_start":"660'500","ondisk_log_start":"660'500","created":655,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261554176,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":3015,"num_objects_missing":3015,"num_objects_degraded":0,"num_objects_misplaced":1270,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10626,"num_bytes_recovered":696385536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,1],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0,1","objects":635},{"shards":"1,4","objects":3356}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"889'13942","reported_seq":35356,"reported_epoch":891,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:47:01.395147+0000","last_change":"2024-04-29T14:46:55.319357+0000","last_active":"2024-04-29T14:47:01.395147+0000","last_peered":"2024-04-29T14:47:01.395147+0000","last_clean":"2024-04-29T14:46:52.774827+0000","last_became_active":"2024-04-29T14:46:55.289693+0000","last_became_peered":"2024-04-29T14:46:55.289693+0000","last_unstale":"2024-04-29T14:47:01.395147+0000","last_undegraded":"2024-04-29T14:47:01.395147+0000","last_fullsized":"2024-04-29T14:47:01.395147+0000","mapping_epoch":887,"log_start":"667'3900","ondisk_log_start":"667'3900","created":655,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'13009","last_scrub_stamp":"2024-04-29T14:44:10.280508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:10.280508+0000","objects_scrubbed":5001,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":8130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1553,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10343,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10243,"num_bytes_recovered":657522688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[4,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"4,6","objects":4065}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"889'10260","reported_seq":30633,"reported_epoch":890,"state":"active+remapped+backfilling","last_fresh":"2024-04-29T14:47:00.466188+0000","last_change":"2024-04-29T14:46:55.313265+0000","last_active":"2024-04-29T14:47:00.466188+0000","last_peered":"2024-04-29T14:47:00.466188+0000","last_clean":"2024-04-29T14:46:52.181774+0000","last_became_active":"2024-04-29T14:46:55.255313+0000","last_became_peered":"2024-04-29T14:46:55.255313+0000","last_unstale":"2024-04-29T14:47:00.466188+0000","last_undegraded":"2024-04-29T14:47:00.466188+0000","last_fullsized":"2024-04-29T14:47:00.466188+0000","mapping_epoch":887,"log_start":"660'200","ondisk_log_start":"660'200","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'9329","last_scrub_stamp":"2024-04-29T14:44:12.414177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:12.414177+0000","objects_scrubbed":4862,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257556480,"num_objects":3930,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1914,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9767,"num_bytes_recovered":640090112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[6,10],"avail_no_missing":["6","2","10"],"object_location_counts":[{"shards":"6,10","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"885'10189","reported_seq":42575,"reported_epoch":891,"state":"activating+degraded+remapped","last_fresh":"2024-04-29T14:47:01.376234+0000","last_change":"2024-04-29T14:47:01.343774+0000","last_active":"2024-04-29T14:46:52.901521+0000","last_peered":"2024-04-29T14:46:52.863620+0000","last_clean":"2024-04-29T14:44:20.165555+0000","last_became_active":"2024-04-29T14:45:01.859387+0000","last_became_peered":"2024-04-29T14:45:01.859387+0000","last_unstale":"2024-04-29T14:47:01.376234+0000","last_undegraded":"2024-04-29T14:47:01.332521+0000","last_fullsized":"2024-04-29T14:47:01.376234+0000","mapping_epoch":887,"log_start":"660'100","ondisk_log_start":"660'100","created":655,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'9290","last_scrub_stamp":"2024-04-29T14:44:09.405827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:09.405827+0000","objects_scrubbed":4890,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261423104,"num_objects":3989,"num_object_clones":0,"num_object_copies":7978,"num_objects_missing_on_primary":4767,"num_objects_missing":4767,"num_objects_degraded":8756,"num_objects_misplaced":0,"num_objects_unfound":2484,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10189,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15500,"num_bytes_recovered":1003552768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,7],"avail_no_missing":["5","10"],"object_location_counts":[{"shards":"4","objects":896},{"shards":"4,7","objects":3093}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"889'10286","reported_seq":25040,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-04-29T14:46:58.547662+0000","last_change":"2024-04-29T14:46:25.294743+0000","last_active":"2024-04-29T14:46:58.547662+0000","last_peered":"2024-04-29T14:46:58.547662+0000","last_clean":"2024-04-29T14:46:58.547662+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:46:58.547662+0000","last_undegraded":"2024-04-29T14:46:58.547662+0000","last_fullsized":"2024-04-29T14:46:58.547662+0000","mapping_epoch":703,"log_start":"660'200","ondisk_log_start":"660'200","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"868'10144","last_scrub_stamp":"2024-04-29T14:46:25.294622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:46:25.294622+0000","objects_scrubbed":4100,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:47:32.365834+0000","scrub_duration":372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10286,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"889'10179","reported_seq":31234,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-04-29T14:47:00.906821+0000","last_change":"2024-04-29T14:46:16.383766+0000","last_active":"2024-04-29T14:47:00.906821+0000","last_peered":"2024-04-29T14:47:00.906821+0000","last_clean":"2024-04-29T14:47:00.906821+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:47:00.906821+0000","last_undegraded":"2024-04-29T14:47:00.906821+0000","last_fullsized":"2024-04-29T14:47:00.906821+0000","mapping_epoch":865,"log_start":"660'100","ondisk_log_start":"660'100","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"822'9191","last_scrub_stamp":"2024-04-29T14:44:05.441763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:05.441763+0000","objects_scrubbed":4850,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253034496,"num_objects":3861,"num_object_clones":0,"num_object_copies":7722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10179,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1743,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-04-29T14:47:00.906819+0000","last_change":"2024-04-29T14:44:49.524542+0000","last_active":"2024-04-29T14:47:00.906819+0000","last_peered":"2024-04-29T14:47:00.906819+0000","last_clean":"2024-04-29T14:47:00.906819+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:47:00.906819+0000","last_undegraded":"2024-04-29T14:47:00.906819+0000","last_fullsized":"2024-04-29T14:47:00.906819+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:44:30.924280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:44:30.924280+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4156555288,"num_objects":63425,"num_object_clones":0,"num_object_copies":126850,"num_objects_missing_on_primary":10883,"num_objects_missing":10883,"num_objects_degraded":8756,"num_objects_misplaced":18211,"num_objects_unfound":2484,"num_objects_dirty":63425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166395,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190705,"num_bytes_recovered":11318591512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12888322048,"data_stored":12888309832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122530,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384169,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2535960,"kb_used_data":1710668,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":91209192,"statfs":{"total":95995035648,"available":93398212608,"internally_reserved":0,"allocated":1751724032,"data_stored":1750287366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":845085120},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645804,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2293592,"kb_used_data":1526604,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":91451560,"statfs":{"total":95995035648,"available":93646397440,"internally_reserved":0,"allocated":1563242496,"data_stored":1561805850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":785381824},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423108,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899544,"kb_used_data":768460,"kb_used_omap":4,"kb_used_meta":131067,"kb_avail":92845608,"statfs":{"total":95995035648,"available":95073902592,"internally_reserved":0,"allocated":786903040,"data_stored":785466394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":134213312},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234485,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1409568,"kb_used_data":539860,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92335584,"statfs":{"total":95995035648,"available":94551638016,"internally_reserved":0,"allocated":552816640,"data_stored":551376442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":890567114},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2048548,"kb_used_data":1106648,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":91696604,"statfs":{"total":95995035648,"available":93897322496,"internally_reserved":0,"allocated":1133207552,"data_stored":1131763282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":964491718},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575369,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1783756,"kb_used_data":1333316,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91961396,"statfs":{"total":95995035648,"available":94168469504,"internally_reserved":0,"allocated":1365315584,"data_stored":1363880919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738904,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2305016,"kb_used_data":1431144,"kb_used_omap":1,"kb_used_meta":873854,"kb_avail":91440136,"statfs":{"total":95995035648,"available":93634699264,"internally_reserved":0,"allocated":1465491456,"data_stored":1464038732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":894826949},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497337,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182923,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773960,"kb_used_data":313792,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92971192,"statfs":{"total":95995035648,"available":95202500608,"internally_reserved":0,"allocated":321323008,"data_stored":319892415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":471202248},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986690,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1742288,"kb_used_data":808704,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92002864,"statfs":{"total":95995035648,"available":94210932736,"internally_reserved":0,"allocated":828112896,"data_stored":826682303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653249,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1274828,"kb_used_data":1068544,"kb_used_omap":651,"kb_used_meta":205620,"kb_avail":92470324,"statfs":{"total":95995035648,"available":94689611776,"internally_reserved":0,"allocated":1094189056,"data_stored":1092758463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":210555656},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1462636544,"data_stored":1462632472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1362694144,"data_stored":1362690072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1516371968,"data_stored":1516371968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":375521280,"data_stored":375521280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":841482240,"data_stored":841482240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1091764224,"data_stored":1091764224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1127878656,"data_stored":1127874584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":549453824,"data_stored":549453824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":784072704,"data_stored":784072704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1560412160,"data_stored":1560412160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1748893696,"data_stored":1748893696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:47:05.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:47:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:05.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:05.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:05.649 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:05.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:05.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:05.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:05.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:05 smithi040 ceph-mon[39581]: osdmap e894: 12 total, 12 up, 10 in 2024-04-29T14:47:05.812 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:05.855 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:05.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:05.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:05 smithi121 ceph-mon[44533]: osdmap e894: 12 total, 12 up, 10 in 2024-04-29T14:47:05.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:05 smithi194 ceph-mon[44089]: osdmap e894: 12 total, 12 up, 10 in 2024-04-29T14:47:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:06.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:06.145 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:06.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:06.253 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:06.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:06.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:06.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:06.404 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:06.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:06.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:06.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:06.560 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:06.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:06.594 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:47:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:06.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:06.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: osdmap e895: 12 total, 12 up, 10 in 2024-04-29T14:47:06.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: pgmap v2285: 17 pgs: 1 peering, 2 active+clean+remapped, 1 activating+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 8756/126854 objects degraded (6.902%); 12343/126854 objects misplaced (9.730%); 2484/63427 objects unfound (3.916%); 101 MiB/s, 1.66k objects/s recovering 2024-04-29T14:47:06.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1598125270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:06.681 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=3845765076,config=1981513354,health=3600748397,kv=4276078421}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3887192911,logm=1827639993}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290424916}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4198476085}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=473273601}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2336568907}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009108250}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449948806}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=894624788}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2265496886}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125718995}) 2024-04-29T14:47:06.682 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=433877393}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363435229}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980218543}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791228274}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864370957}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780453182}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3418753055}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1574998611}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2582646983}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3905334882}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=861235909}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2066988232}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205586618}) 2024-04-29T14:47:06.683 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644091401}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185824002}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342353693}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923209120}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011452793}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2475005921}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1096389609}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4205738758}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22073190}) 2024-04-29T14:47:06.684 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:06 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28725020}) 2024-04-29T14:47:06.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:06.729 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:06.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:06.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:06.857 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: osdmap e895: 12 total, 12 up, 10 in 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: pgmap v2285: 17 pgs: 1 peering, 2 active+clean+remapped, 1 activating+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 8756/126854 objects degraded (6.902%); 12343/126854 objects misplaced (9.730%); 2484/63427 objects unfound (3.916%); 101 MiB/s, 1.66k objects/s recovering 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1598125270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=3845765076,config=1981513354,health=3600748397,kv=4276078421}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3887192911,logm=1827639993}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290424916}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4198476085}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=473273601}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2336568907}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009108250}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449948806}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=894624788}) 2024-04-29T14:47:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2265496886}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125718995}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=433877393}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363435229}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980218543}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791228274}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864370957}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780453182}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3418753055}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1574998611}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2582646983}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3905334882}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=861235909}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2066988232}) 2024-04-29T14:47:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205586618}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644091401}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185824002}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342353693}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923209120}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011452793}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2475005921}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1096389609}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4205738758}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22073190}) 2024-04-29T14:47:06.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:06 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28725020}) 2024-04-29T14:47:06.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: osdmap e895: 12 total, 12 up, 10 in 2024-04-29T14:47:06.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: pgmap v2285: 17 pgs: 1 peering, 2 active+clean+remapped, 1 activating+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 8756/126854 objects degraded (6.902%); 12343/126854 objects misplaced (9.730%); 2484/63427 objects unfound (3.916%); 101 MiB/s, 1.66k objects/s recovering 2024-04-29T14:47:06.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1598125270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:06.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=3845765076,config=1981513354,health=3600748397,kv=4276078421}) 2024-04-29T14:47:06.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3887192911,logm=1827639993}) 2024-04-29T14:47:06.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290424916}) 2024-04-29T14:47:06.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4198476085}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=473273601}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2336568907}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2009108250}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449948806}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=894624788}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2265496886}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3125718995}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=433877393}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363435229}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=980218543}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791228274}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864370957}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1780453182}) 2024-04-29T14:47:06.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3418753055}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1574998611}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2582646983}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3905334882}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=861235909}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2066988232}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205586618}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644091401}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185824002}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342353693}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923209120}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011452793}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2475005921}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1096389609}) 2024-04-29T14:47:06.891 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4205738758}) 2024-04-29T14:47:06.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22073190}) 2024-04-29T14:47:06.892 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:06 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28725020}) 2024-04-29T14:47:06.892 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:07.744 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:07.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:07.849 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181097807}) 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403438755}) 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2252143780}) 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1598125270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: osdmap e896: 12 total, 12 up, 10 in 2024-04-29T14:47:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1289801447}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=738490999}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1576883098}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4244977548}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751488399}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765488467}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4070204791}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639299903}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653227362}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=35856480}) 2024-04-29T14:47:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555429420}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314248214}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427341584}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2842691970}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317725157}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2983559153}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977304396}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3917588323}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2743597645}) 2024-04-29T14:47:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428267339}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2952133657}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417416804}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937657711}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1916934527}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=881428515}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1029940550}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412270058}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1564223166}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4070349067}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332176034}) 2024-04-29T14:47:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3060741603}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182806362}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124913203}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916674951}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583672926}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2931211398}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169824691}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045125037}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997787011}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211442529}) 2024-04-29T14:47:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833963242}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3128011968}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1950012516}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1739312526}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152664391}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803217301}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212178694}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2166073139}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1806101983}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977658835}) 2024-04-29T14:47:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169913296}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882211027}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177242249}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944248262}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1407928202}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334682337}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4173267943}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113820191}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2465442778}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1592827257}) 2024-04-29T14:47:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803274069}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3219636103}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200389073}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=1,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=57} crc {logm=3527902583,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=2055497628,mon_config_key=1912826235}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2893141549}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2535878829}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=21,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=61} crc {mon_config_key=3217410368,monmap=455069592,osd_metadata=809037704,osd_pg_creating=2038009952,osdmap=2893532246}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3365414338}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2661720845}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2236741971}) 2024-04-29T14:47:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=21} crc {osdmap=2868144649}) 2024-04-29T14:47:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:47:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:07 smithi040 ceph-mon[39581]: osdmap e897: 12 total, 12 up, 10 in 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181097807}) 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403438755}) 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2252143780}) 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1598125270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: osdmap e896: 12 total, 12 up, 10 in 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1289801447}) 2024-04-29T14:47:07.882 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=738490999}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1576883098}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4244977548}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751488399}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765488467}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4070204791}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639299903}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653227362}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=35856480}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555429420}) 2024-04-29T14:47:07.883 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314248214}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427341584}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2842691970}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317725157}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2983559153}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977304396}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3917588323}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2743597645}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428267339}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2952133657}) 2024-04-29T14:47:07.884 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417416804}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937657711}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1916934527}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=881428515}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1029940550}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412270058}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1564223166}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4070349067}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332176034}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3060741603}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182806362}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124913203}) 2024-04-29T14:47:07.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916674951}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583672926}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2931211398}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169824691}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045125037}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997787011}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211442529}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833963242}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3128011968}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1950012516}) 2024-04-29T14:47:07.886 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1739312526}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152664391}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803217301}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212178694}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2166073139}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1806101983}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977658835}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169913296}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882211027}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177242249}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944248262}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1407928202}) 2024-04-29T14:47:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334682337}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4173267943}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113820191}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2465442778}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1592827257}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803274069}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3219636103}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200389073}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=1,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=57} crc {logm=3527902583,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=2055497628,mon_config_key=1912826235}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2893141549}) 2024-04-29T14:47:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2535878829}) 2024-04-29T14:47:07.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=21,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=61} crc {mon_config_key=3217410368,monmap=455069592,osd_metadata=809037704,osd_pg_creating=2038009952,osdmap=2893532246}) 2024-04-29T14:47:07.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3365414338}) 2024-04-29T14:47:07.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2661720845}) 2024-04-29T14:47:07.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2236741971}) 2024-04-29T14:47:07.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=21} crc {osdmap=2868144649}) 2024-04-29T14:47:07.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:47:07.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:07 smithi194 ceph-mon[44089]: osdmap e897: 12 total, 12 up, 10 in 2024-04-29T14:47:07.889 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:47:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181097807}) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403438755}) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2252143780}) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1598125270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: osdmap e896: 12 total, 12 up, 10 in 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1289801447}) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=738490999}) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1576883098}) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4244977548}) 2024-04-29T14:47:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751488399}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765488467}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4070204791}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639299903}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653227362}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=35856480}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555429420}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314248214}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427341584}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2842691970}) 2024-04-29T14:47:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317725157}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2983559153}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977304396}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3917588323}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2743597645}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428267339}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2952133657}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417416804}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937657711}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1916934527}) 2024-04-29T14:47:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=881428515}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1029940550}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3412270058}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1564223166}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4070349067}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332176034}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3060741603}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182806362}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124913203}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=916674951}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583672926}) 2024-04-29T14:47:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2931211398}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169824691}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045125037}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997787011}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211442529}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833963242}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3128011968}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1950012516}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1739312526}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152664391}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803217301}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212178694}) 2024-04-29T14:47:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2166073139}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1806101983}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977658835}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2169913296}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882211027}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177242249}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944248262}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1407928202}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334682337}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4173267943}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113820191}) 2024-04-29T14:47:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2465442778}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1592827257}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2803274069}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3219636103}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200389073}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=1,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=57} crc {logm=3527902583,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=2055497628,mon_config_key=1912826235}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2893141549}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2535878829}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=21,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=61} crc {mon_config_key=3217410368,monmap=455069592,osd_metadata=809037704,osd_pg_creating=2038009952,osdmap=2893532246}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3365414338}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2661720845}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2236741971}) 2024-04-29T14:47:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=21} crc {osdmap=2868144649}) 2024-04-29T14:47:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:47:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:07 smithi121 ceph-mon[44533]: osdmap e897: 12 total, 12 up, 10 in 2024-04-29T14:47:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:08 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:47:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:08 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:47:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:08 smithi040 ceph-mon[39581]: pgmap v2288: 17 pgs: 1 peering, 2 active+clean+remapped, 1 activating+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 8756/126854 objects degraded (6.902%); 12211/126854 objects misplaced (9.626%); 2484/63427 objects unfound (3.916%); 91 MiB/s, 1.46k objects/s recovering 2024-04-29T14:47:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:08 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:47:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:08 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:47:08.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:08 smithi121 ceph-mon[44533]: pgmap v2288: 17 pgs: 1 peering, 2 active+clean+remapped, 1 activating+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 8756/126854 objects degraded (6.902%); 12211/126854 objects misplaced (9.626%); 2484/63427 objects unfound (3.916%); 91 MiB/s, 1.46k objects/s recovering 2024-04-29T14:47:09.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:08 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:47:09.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:08 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:47:09.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:08 smithi194 ceph-mon[44089]: pgmap v2288: 17 pgs: 1 peering, 2 active+clean+remapped, 1 activating+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering+remapped, 9 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 8756/126854 objects degraded (6.902%); 12211/126854 objects misplaced (9.626%); 2484/63427 objects unfound (3.916%); 91 MiB/s, 1.46k objects/s recovering 2024-04-29T14:47:09.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:47:10.179 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 1, 5, 8, 3] out_osds: [11, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:47:10.179 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:47:10.179 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-29T14:47:10.179 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-04-29T14:47:10.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-29T14:47:10.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:10 smithi040 ceph-mon[39581]: Health check cleared: OBJECT_UNFOUND (was: 2484/63427 objects unfound (3.916%)) 2024-04-29T14:47:10.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:10 smithi121 ceph-mon[44533]: Health check cleared: OBJECT_UNFOUND (was: 2484/63427 objects unfound (3.916%)) 2024-04-29T14:47:10.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:10 smithi194 ceph-mon[44089]: Health check cleared: OBJECT_UNFOUND (was: 2484/63427 objects unfound (3.916%)) 2024-04-29T14:47:11.633 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:11 smithi194 ceph-mon[44089]: pgmap v2289: 17 pgs: 1 peering, 1 active+degraded+remapped, 1 active+clean+remapped, 2 active+recovering+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 8756/126854 objects degraded (6.902%); 7674/126854 objects misplaced (6.049%); 107 MiB/s, 1.71k objects/s recovering 2024-04-29T14:47:11.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:11 smithi121 ceph-mon[44533]: pgmap v2289: 17 pgs: 1 peering, 1 active+degraded+remapped, 1 active+clean+remapped, 2 active+recovering+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 8756/126854 objects degraded (6.902%); 7674/126854 objects misplaced (6.049%); 107 MiB/s, 1.71k objects/s recovering 2024-04-29T14:47:11.635 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:11.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:11.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:11.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:11 smithi040 ceph-mon[39581]: pgmap v2289: 17 pgs: 1 peering, 1 active+degraded+remapped, 1 active+clean+remapped, 2 active+recovering+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 8756/126854 objects degraded (6.902%); 7674/126854 objects misplaced (6.049%); 107 MiB/s, 1.71k objects/s recovering 2024-04-29T14:47:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:12 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 8756/126588 objects degraded (6.917%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:47:12.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:12 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 8756/126588 objects degraded (6.917%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:47:12.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:12 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 8756/126588 objects degraded (6.917%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:47:13.273 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:13.283 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:13.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:13.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:13.428 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:13.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:13.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:13.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:13.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:13 smithi040 ceph-mon[39581]: pgmap v2290: 17 pgs: 1 active+degraded+remapped, 2 active+recovering+remapped, 14 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 19 op/s; 8756/126588 objects degraded (6.917%); 7674/126588 objects misplaced (6.062%); 37 MiB/s, 589 objects/s recovering 2024-04-29T14:47:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:13.814 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:13 smithi194 ceph-mon[44089]: pgmap v2290: 17 pgs: 1 active+degraded+remapped, 2 active+recovering+remapped, 14 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 19 op/s; 8756/126588 objects degraded (6.917%); 7674/126588 objects misplaced (6.062%); 37 MiB/s, 589 objects/s recovering 2024-04-29T14:47:13.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:13 smithi121 ceph-mon[44533]: pgmap v2290: 17 pgs: 1 active+degraded+remapped, 2 active+recovering+remapped, 14 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 19 op/s; 8756/126588 objects degraded (6.917%); 7674/126588 objects misplaced (6.062%); 37 MiB/s, 589 objects/s recovering 2024-04-29T14:47:14.330 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:47:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:47:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:47:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:130964 7.5 e750/655 7:a06d4686:::benchmark_data_smithi194_147200_object16052:head v 750'8178, mlcod=750'8175)", 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:40:04.067221+0000", 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 430.251283898, 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077772500000000003, 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067221+0000", 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067221+0000", 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067222+0000", 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.829e-06 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067231+0000", 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7790000000000001e-06 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067243+0000", 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2197e-05 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067251+0000", 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6000000000000001e-06 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067313+0000", 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2241000000000003e-05 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.067329+0000", 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5772000000000001e-05 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.074942+0000", 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076125719999999997 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:40:04.074998+0000", 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.626e-05 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:136420 7.5 e777/655 7:add952f2:::benchmark_data_smithi194_147200_object21508:head v 777'8520, mlcod=777'8518)", 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 334.52735842099997, 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074850630000000001, 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791149+0000", 2024-04-29T14:47:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6699999999999998e-06 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791158+0000", 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4490000000000005e-06 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791172+0000", 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3655999999999999e-05 2024-04-29T14:47:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791181+0000", 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7700000000000007e-06 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791243+0000", 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2904999999999995e-05 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791266+0000", 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2382999999999999e-05 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.798602+0000", 2024-04-29T14:47:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073362590000000004 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.798631+0000", 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8971000000000001e-05 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:136948 7.5 e779/655 7:ae800e08:::benchmark_data_smithi194_147200_object22036:head v 779'8554, mlcod=779'8551)", 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 330.20538521600002, 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076202170000000003, 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113121+0000", 2024-04-29T14:47:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8780000000000001e-06 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113130+0000", 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1679999999999993e-06 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113143+0000", 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2628e-05 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113151+0000", 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5299999999999996e-06 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.116770+0000", 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036181640000000001 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.116798+0000", 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8748999999999999e-05 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.120682+0000", 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038833090000000002 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.120739+0000", 2024-04-29T14:47:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7791000000000003e-05 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e851/685 cost 1049576)", 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:29.618262+0000", 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 104.70024228699999, 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00088199800000000005, 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.618262+0000", 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.618262+0000", 2024-04-29T14:47:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.618266+0000", 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4580000000000002e-06 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.618283+0000", 2024-04-29T14:47:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7677e-05 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.618298+0000", 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5136e-05 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.618325+0000", 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7008999999999998e-05 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.618404+0000", 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8060999999999997e-05 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:29.619144+0000", 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00074065700000000003 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e858/685 cost 1049576)", 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:46.170812+0000", 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 88.147692219000007, 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029017560000000001, 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170812+0000", 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170812+0000", 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170815+0000", 2024-04-29T14:47:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.101e-06 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170846+0000", 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0626000000000001e-05 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170860+0000", 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4449e-05 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170870+0000", 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1940000000000004e-06 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170928+0000", 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8112000000000002e-05 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.173714+0000", 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002786274 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e858/685 cost 1049576)", 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:47.413345+0000", 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 86.905159440000006, 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00090962900000000001, 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413345+0000", 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413345+0000", 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413349+0000", 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2069999999999997e-06 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413374+0000", 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4756000000000002e-05 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413402+0000", 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7883000000000001e-05 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413417+0000", 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5310000000000001e-05 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413565+0000", 2024-04-29T14:47:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014771399999999999 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.414255+0000", 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00068975899999999997 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e867/685 cost 1049576)", 2024-04-29T14:47:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:22.147095+0000", 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 52.171409388000001, 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00086429699999999998, 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147095+0000", 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147095+0000", 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147099+0000", 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4659999999999999e-06 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147121+0000", 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2289e-05 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147143+0000", 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2274000000000001e-05 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147157+0000", 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3811999999999999e-05 2024-04-29T14:47:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147306+0000", 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014879100000000001 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.147959+0000", 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000653665 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e867/685 cost 1049576)", 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:22.291697+0000", 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 52.026807226999999, 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078105299999999998, 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.291697+0000", 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.291697+0000", 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.291701+0000", 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6169999999999998e-06 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.291742+0000", 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0695000000000003e-05 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.291768+0000", 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6466999999999999e-05 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.291782+0000", 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4256e-05 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.291929+0000", 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014655399999999999 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.292478+0000", 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00054946400000000005 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e867/685 cost 1049576)", 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:22.703050+0000", 2024-04-29T14:47:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.615454679000003, 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002770083, 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703050+0000", 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703050+0000", 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703052+0000", 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.001e-06 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703064+0000", 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2330999999999999e-05 2024-04-29T14:47:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703077+0000", 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3073e-05 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703085+0000", 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3209999999999999e-06 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703137+0000", 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2392000000000003e-05 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.705820+0000", 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026829649999999998 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e878/869 cost 1049576)", 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:39.378041+0000", 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.940463506999997, 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016862870000000001, 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378041+0000", 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378041+0000", 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378043+0000", 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3489999999999999e-06 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378055+0000", 2024-04-29T14:47:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2062000000000001e-05 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378069+0000", 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3222e-05 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378076+0000", 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1720000000000001e-06 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378133+0000", 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7169000000000001e-05 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.379727+0000", 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001594313 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e878/869 cost 1049576)", 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:39.387071+0000", 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.931433632999997, 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010839949999999999, 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387071+0000", 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387071+0000", 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387072+0000", 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6330000000000001e-06 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387083+0000", 2024-04-29T14:47:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0575e-05 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387094+0000", 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1409999999999999e-05 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387102+0000", 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9699999999999999e-06 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387159+0000", 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6434999999999998e-05 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.388155+0000", 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099597199999999992 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e880/869 cost 1049576)", 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:44.198461+0000", 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.12004383, 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00091009299999999995, 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198461+0000", 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198461+0000", 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198463+0000", 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4049999999999998e-06 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198479+0000", 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6161999999999999e-05 2024-04-29T14:47:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198495+0000", 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5449999999999999e-05 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198503+0000", 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4130000000000006e-06 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198570+0000", 2024-04-29T14:47:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6685000000000003e-05 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.199371+0000", 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00080097800000000004 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e880/869 cost 1049576)", 2024-04-29T14:47:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:45.394211+0000", 2024-04-29T14:47:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 28.924293838000001, 2024-04-29T14:47:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019670479999999999, 2024-04-29T14:47:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394211+0000", 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394211+0000", 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394213+0000", 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3460000000000001e-06 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394243+0000", 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0195000000000001e-05 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394260+0000", 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7048e-05 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394270+0000", 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.431e-06 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394331+0000", 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0885999999999999e-05 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.396178+0000", 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001847142 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e894/869 cost 1049576)", 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:05.304324+0000", 2024-04-29T14:47:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.0141808500000007, 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075424200000000002, 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.304324+0000", 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.304324+0000", 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.304326+0000", 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7970000000000001e-06 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.304346+0000", 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9956e-05 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.304369+0000", 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3119999999999999e-05 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.304383+0000", 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.359e-05 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.304490+0000", 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106554 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.305078+0000", 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058822499999999995 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e895/869 cost 1049576)", 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:05.908307+0000", 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.4101971169999992, 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075522199999999995, 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.908307+0000", 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.908307+0000", 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.908310+0000", 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.033e-06 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.908331+0000", 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0890000000000002e-05 2024-04-29T14:47:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.908354+0000", 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2849000000000001e-05 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.908368+0000", 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3499999999999999e-05 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.908484+0000", 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116871 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:05.909063+0000", 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00057807900000000001 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e895/869 cost 1049576)", 2024-04-29T14:47:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:06.519787+0000", 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 7.798717312, 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00077428600000000001, 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.519787+0000", 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.519787+0000", 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.519805+0000", 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7521999999999999e-05 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.519816+0000", 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.115e-05 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.519828+0000", 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1768e-05 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.519853+0000", 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5075e-05 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.519931+0000", 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8325999999999995e-05 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:06.520561+0000", 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063044499999999999 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e897/869 cost 1049576)", 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:08.638518+0000", 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.6799865120000002, 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00083648200000000005, 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.638518+0000", 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.638518+0000", 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.638522+0000", 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7799999999999998e-06 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.638548+0000", 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6418999999999999e-05 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.638573+0000", 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4493000000000001e-05 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.638589+0000", 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6382999999999999e-05 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.638751+0000", 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016206599999999999 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.639354+0000", 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060334099999999997 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e897/869 cost 1049576)", 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:08.761526+0000", 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.5569781459999996, 2024-04-29T14:47:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011081419999999999, 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761526+0000", 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761526+0000", 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761530+0000", 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8360000000000002e-06 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761550+0000", 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0030999999999998e-05 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761567+0000", 2024-04-29T14:47:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6342e-05 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761595+0000", 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8308000000000001e-05 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761672+0000", 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7114999999999996e-05 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.762634+0000", 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096250999999999997 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e897/869 cost 1049576)", 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:11.958509+0000", 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.3599953710000001, 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00080541300000000005, 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.958509+0000", 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.958509+0000", 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.412 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.958511+0000", 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2800000000000002e-06 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.958526+0000", 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4548e-05 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.958541+0000", 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5255e-05 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.413 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.958550+0000", 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1169999999999993e-06 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.958603+0000", 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2966000000000001e-05 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:11.959314+0000", 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071124700000000003 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.414 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e897/869 cost 1049576)", 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:14.168222+0000", 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.150282737, 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075372299999999998, 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:14.415 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168222+0000", 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168222+0000", 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168225+0000", 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3739999999999998e-06 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:14.416 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168248+0000", 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3281999999999998e-05 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168271+0000", 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2997e-05 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168293+0000", 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1219000000000001e-05 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168454+0000", 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000161607 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:14.168975+0000", 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052124400000000003 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:14.418 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:47:14.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:14 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8756/126588 objects degraded (6.917%), 1 pg degraded) 2024-04-29T14:47:14.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:14 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:47:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:14 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8756/126588 objects degraded (6.917%), 1 pg degraded) 2024-04-29T14:47:14.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:14 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:47:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:14 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8756/126588 objects degraded (6.917%), 1 pg degraded) 2024-04-29T14:47:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:14 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:47:15.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:47:15.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:15 smithi194 ceph-mon[44089]: pgmap v2291: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 13285/125662 objects misplaced (10.572%); 32 MiB/s, 613 objects/s recovering 2024-04-29T14:47:15.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:15 smithi040 ceph-mon[39581]: pgmap v2291: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 13285/125662 objects misplaced (10.572%); 32 MiB/s, 613 objects/s recovering 2024-04-29T14:47:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:15.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:15.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:15 smithi121 ceph-mon[44533]: pgmap v2291: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 13285/125662 objects misplaced (10.572%); 32 MiB/s, 613 objects/s recovering 2024-04-29T14:47:15.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:15.972 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:15.991 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:15.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:16.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:16.126 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:17.154 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:47:17.176 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:47:17.177 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:17.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:17.199 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:47:17.230 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:47:17.250 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:47:17.271 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:47:17.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:17.292 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:47:17.309 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:17.312 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:47:17.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:17.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:17.333 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:47:17.364 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:47:17.384 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:47:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:47:17.405 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:47:17.405 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:47:17.405 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:47:17.406 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:47:17.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:17.460 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:17.479 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:17.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:17.583 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:17.610 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:17.629 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:17.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:17.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:17 smithi121 ceph-mon[44533]: pgmap v2292: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 13285/124962 objects misplaced (10.631%); 26 MiB/s, 507 objects/s recovering 2024-04-29T14:47:17.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/761208831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:17.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:17.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:17.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:17.763 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:17.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:17 smithi040 ceph-mon[39581]: pgmap v2292: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 13285/124962 objects misplaced (10.631%); 26 MiB/s, 507 objects/s recovering 2024-04-29T14:47:17.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/761208831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:17.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:17.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:17.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:17.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:17.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:17 smithi194 ceph-mon[44089]: pgmap v2292: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 97 op/s; 13285/124962 objects misplaced (10.631%); 26 MiB/s, 507 objects/s recovering 2024-04-29T14:47:17.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/761208831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:17.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:17.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:17.888 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:17.933 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:17.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:18.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:18.068 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:18.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:18.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:18.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:18.305 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:18.433 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:18.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:18.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:18.573 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:18.927 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:18.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:18.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:19.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:19.062 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:19.492 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:47:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:19 smithi194 ceph-mon[44089]: pgmap v2293: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 88 op/s; 13285/124962 objects misplaced (10.631%); 24 MiB/s, 457 objects/s recovering 2024-04-29T14:47:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/228800809' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:47:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:19 smithi194 ceph-mon[44089]: from='client.27649 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:47:19.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:19 smithi040 ceph-mon[39581]: pgmap v2293: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 88 op/s; 13285/124962 objects misplaced (10.631%); 24 MiB/s, 457 objects/s recovering 2024-04-29T14:47:19.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/228800809' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:47:19.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:19 smithi040 ceph-mon[39581]: from='client.27649 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:47:19.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:19 smithi121 ceph-mon[44533]: pgmap v2293: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 88 op/s; 13285/124962 objects misplaced (10.631%); 24 MiB/s, 457 objects/s recovering 2024-04-29T14:47:19.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/228800809' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:47:19.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:19 smithi121 ceph-mon[44533]: from='client.27649 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:47:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:20 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:20 smithi194 ceph-mon[44089]: from='client.27649 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:47:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:20 smithi194 ceph-mon[44089]: osdmap e898: 12 total, 12 up, 10 in 2024-04-29T14:47:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:20 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:20 smithi040 ceph-mon[39581]: from='client.27649 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:47:20.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:20 smithi040 ceph-mon[39581]: osdmap e898: 12 total, 12 up, 10 in 2024-04-29T14:47:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:20 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:20 smithi121 ceph-mon[44533]: from='client.27649 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:47:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:20 smithi121 ceph-mon[44533]: osdmap e898: 12 total, 12 up, 10 in 2024-04-29T14:47:21.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:21 smithi194 ceph-mon[44089]: pgmap v2295: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 138 op/s; 13149/124074 objects misplaced (10.598%); 1.6 MiB/s, 159 objects/s recovering 2024-04-29T14:47:21.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:21 smithi040 ceph-mon[39581]: pgmap v2295: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 138 op/s; 13149/124074 objects misplaced (10.598%); 1.6 MiB/s, 159 objects/s recovering 2024-04-29T14:47:21.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:21 smithi121 ceph-mon[44533]: pgmap v2295: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 138 op/s; 13149/124074 objects misplaced (10.598%); 1.6 MiB/s, 159 objects/s recovering 2024-04-29T14:47:23.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:23 smithi194 ceph-mon[44089]: pgmap v2296: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 13149/123276 objects misplaced (10.666%); 1.6 MiB/s, 159 objects/s recovering 2024-04-29T14:47:23.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:23 smithi194 ceph-mon[44089]: osdmap e899: 12 total, 12 up, 10 in 2024-04-29T14:47:23.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:23 smithi040 ceph-mon[39581]: pgmap v2296: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 13149/123276 objects misplaced (10.666%); 1.6 MiB/s, 159 objects/s recovering 2024-04-29T14:47:23.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:23 smithi040 ceph-mon[39581]: osdmap e899: 12 total, 12 up, 10 in 2024-04-29T14:47:23.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:23 smithi121 ceph-mon[44533]: pgmap v2296: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 13149/123276 objects misplaced (10.666%); 1.6 MiB/s, 159 objects/s recovering 2024-04-29T14:47:23.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:23 smithi121 ceph-mon[44533]: osdmap e899: 12 total, 12 up, 10 in 2024-04-29T14:47:25.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:25 smithi194 ceph-mon[44089]: pgmap v2298: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 181 op/s; 13026/122064 objects misplaced (10.671%); 1.4 MiB/s, 89 objects/s recovering 2024-04-29T14:47:25.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:25.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:25 smithi040 ceph-mon[39581]: pgmap v2298: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 181 op/s; 13026/122064 objects misplaced (10.671%); 1.4 MiB/s, 89 objects/s recovering 2024-04-29T14:47:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:25.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:25 smithi121 ceph-mon[44533]: pgmap v2298: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 181 op/s; 13026/122064 objects misplaced (10.671%); 1.4 MiB/s, 89 objects/s recovering 2024-04-29T14:47:25.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:25.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:25.915 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:25.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:26.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:26.051 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:26.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:26.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:26.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:26.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:26.339 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:26.366 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:26.385 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:26.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:26.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:26.528 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:26.559 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:26.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:26.664 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:26.688 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:26 smithi194 ceph-mon[44089]: osdmap e900: 12 total, 12 up, 10 in 2024-04-29T14:47:26.688 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:26 smithi194 ceph-mon[44089]: pgmap v2300: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 208 op/s; 13026/121238 objects misplaced (10.744%); 849 KiB/s, 13 objects/s recovering 2024-04-29T14:47:26.690 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:26.708 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:26.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:26.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:26.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:26 smithi040 ceph-mon[39581]: osdmap e900: 12 total, 12 up, 10 in 2024-04-29T14:47:26.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:26 smithi040 ceph-mon[39581]: pgmap v2300: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 208 op/s; 13026/121238 objects misplaced (10.744%); 849 KiB/s, 13 objects/s recovering 2024-04-29T14:47:26.838 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:26.858 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:26.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:26 smithi121 ceph-mon[44533]: osdmap e900: 12 total, 12 up, 10 in 2024-04-29T14:47:26.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:26 smithi121 ceph-mon[44533]: pgmap v2300: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 208 op/s; 13026/121238 objects misplaced (10.744%); 849 KiB/s, 13 objects/s recovering 2024-04-29T14:47:26.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:26.989 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:27.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:27.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:27.147 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:27.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:47:27.528 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-29T14:47:27.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2229056508' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:47:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2229056508' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:47:27.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2229056508' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:47:27.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:47:28.100 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:28.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:28.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:28.235 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:28.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:28.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:28.390 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:28.391 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:47:28.391 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:47:28.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:28.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:28.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:28.554 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:28.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2229056508' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:47:28.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:28 smithi194 ceph-mon[44089]: osdmap e901: 12 total, 12 up, 10 in 2024-04-29T14:47:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:28 smithi194 ceph-mon[44089]: pgmap v2302: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 169 op/s; 13026/121238 objects misplaced (10.744%); 960 KiB/s, 15 objects/s recovering 2024-04-29T14:47:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2229056508' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:47:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:28 smithi040 ceph-mon[39581]: osdmap e901: 12 total, 12 up, 10 in 2024-04-29T14:47:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:28 smithi040 ceph-mon[39581]: pgmap v2302: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 169 op/s; 13026/121238 objects misplaced (10.744%); 960 KiB/s, 15 objects/s recovering 2024-04-29T14:47:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2229056508' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:47:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:28 smithi121 ceph-mon[44533]: osdmap e901: 12 total, 12 up, 10 in 2024-04-29T14:47:28.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:28 smithi121 ceph-mon[44533]: pgmap v2302: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 169 op/s; 13026/121238 objects misplaced (10.744%); 960 KiB/s, 15 objects/s recovering 2024-04-29T14:47:29.165 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:29.211 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:29.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:29.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:29.349 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:29.371 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:29.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:29.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:29.505 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:29.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:29.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:29.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:29.659 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:29.681 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:29.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:29.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:29.818 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:29.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:29.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:29.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:29.974 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:29.993 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:29.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:30.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:30.130 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:30.500 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:30.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:47:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:31 smithi040 ceph-mon[39581]: pgmap v2303: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 197 op/s; 12882/120228 objects misplaced (10.715%); 1.5 MiB/s, 153 objects/s recovering 2024-04-29T14:47:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:31 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:31 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:31 smithi121 ceph-mon[44533]: pgmap v2303: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 197 op/s; 12882/120228 objects misplaced (10.715%); 1.5 MiB/s, 153 objects/s recovering 2024-04-29T14:47:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:31 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:31 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:31.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:31 smithi194 ceph-mon[44089]: pgmap v2303: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 197 op/s; 12882/120228 objects misplaced (10.715%); 1.5 MiB/s, 153 objects/s recovering 2024-04-29T14:47:31.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:31 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:31.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:31 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:31.883 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:31.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:31.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:31.987 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:32.016 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:32.036 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:32.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:32.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:32.168 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:32.192 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:32.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:32.297 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:32.329 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:32.457 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:32.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:32.564 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:32.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:32 smithi040 ceph-mon[39581]: osdmap e902: 12 total, 12 up, 10 in 2024-04-29T14:47:32.596 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:32.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:32 smithi121 ceph-mon[44533]: osdmap e902: 12 total, 12 up, 10 in 2024-04-29T14:47:32.697 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:32.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:32.801 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:32.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:32 smithi194 ceph-mon[44089]: osdmap e902: 12 total, 12 up, 10 in 2024-04-29T14:47:32.831 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:32.854 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:32.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:32.958 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:32.985 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:33.006 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:33.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:33.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:33.152 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:33.194 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:33.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:33.252 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:47:33.273 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:47:33.293 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:47:33.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:33.314 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:47:33.330 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:33.336 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:47:33.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:33.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:33.359 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:47:33.382 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:47:33.410 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:47:33.433 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:47:33.454 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:47:33.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:33.471 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:33.485 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:47:33.486 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:33.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:33.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:33.524 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:47:33.526 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:47:33.527 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:47:33.527 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:47:33.527 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:47:33.615 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:33 smithi040 ceph-mon[39581]: pgmap v2305: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 12882/119336 objects misplaced (10.795%); 947 KiB/s, 161 objects/s recovering 2024-04-29T14:47:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:33 smithi040 ceph-mon[39581]: osdmap e903: 12 total, 12 up, 10 in 2024-04-29T14:47:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3652489132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:33.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:33 smithi121 ceph-mon[44533]: pgmap v2305: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 12882/119336 objects misplaced (10.795%); 947 KiB/s, 161 objects/s recovering 2024-04-29T14:47:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:33 smithi121 ceph-mon[44533]: osdmap e903: 12 total, 12 up, 10 in 2024-04-29T14:47:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3652489132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:33.641 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:33 smithi194 ceph-mon[44089]: pgmap v2305: 17 pgs: 3 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 12882/119336 objects misplaced (10.795%); 947 KiB/s, 161 objects/s recovering 2024-04-29T14:47:33.641 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:33 smithi194 ceph-mon[44089]: osdmap e903: 12 total, 12 up, 10 in 2024-04-29T14:47:33.641 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3652489132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:33.643 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:33.663 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:33.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:33.768 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:33.794 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:33.814 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:33.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:33.918 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:33.991 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:34.013 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:34.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:34.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:34.150 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:34.166 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 1, 5, 8, 3] out_osds: [11, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:47:34.166 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:47:34.166 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-04-29T14:47:34.167 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T14:47:34.167 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T14:47:34.167 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-04-29T14:47:34.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:34.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:34.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:34.308 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:47:34.458 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:47:34.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/431495138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:34.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/431495138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:34.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/431495138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:35.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:35 smithi194 ceph-mon[44089]: pgmap v2307: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 13939/118218 objects misplaced (11.791%); 25 MiB/s, 578 objects/s recovering 2024-04-29T14:47:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:35 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/431495138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:35 smithi194 ceph-mon[44089]: osdmap e904: 12 total, 12 up, 10 in 2024-04-29T14:47:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:35 smithi040 ceph-mon[39581]: pgmap v2307: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 13939/118218 objects misplaced (11.791%); 25 MiB/s, 578 objects/s recovering 2024-04-29T14:47:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:35 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/431495138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:35 smithi040 ceph-mon[39581]: osdmap e904: 12 total, 12 up, 10 in 2024-04-29T14:47:35.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:35.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:35 smithi121 ceph-mon[44533]: pgmap v2307: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 13939/118218 objects misplaced (11.791%); 25 MiB/s, 578 objects/s recovering 2024-04-29T14:47:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:35 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/431495138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:35 smithi121 ceph-mon[44533]: osdmap e904: 12 total, 12 up, 10 in 2024-04-29T14:47:35.761 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:35.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:35.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:35.896 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:35.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:35.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:36.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:36.049 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:36.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:36.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:36.179 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:36.208 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:36.307 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:36.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:36.332 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:47:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:47:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:47:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:47:36.333 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:47:36.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:36.343 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:36.411 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:36.439 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:36.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:47:36.492 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:36.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:36.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:36.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:36.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:36.622 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:47:36.622 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-29T14:47:36.623 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:47:36.628 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:36.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:36.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:36.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:36.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:36.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:36.789 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:36.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:36.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:36.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:36.943 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:37.143 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:37.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:37.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:47:37.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:37.280 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:37.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:37 smithi040 ceph-mon[39581]: pgmap v2309: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 235 op/s; 13939/117396 objects misplaced (11.873%); 27 MiB/s, 468 objects/s recovering 2024-04-29T14:47:37.589 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:37 smithi040 ceph-mon[39581]: osdmap e905: 12 total, 12 up, 10 in 2024-04-29T14:47:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:37.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:37 smithi121 ceph-mon[44533]: pgmap v2309: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 235 op/s; 13939/117396 objects misplaced (11.873%); 27 MiB/s, 468 objects/s recovering 2024-04-29T14:47:37.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:37 smithi121 ceph-mon[44533]: osdmap e905: 12 total, 12 up, 10 in 2024-04-29T14:47:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:37.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:37.653 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:37.737 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:37.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:37 smithi194 ceph-mon[44089]: pgmap v2309: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 235 op/s; 13939/117396 objects misplaced (11.873%); 27 MiB/s, 468 objects/s recovering 2024-04-29T14:47:37.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:37 smithi194 ceph-mon[44089]: osdmap e905: 12 total, 12 up, 10 in 2024-04-29T14:47:37.969 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:37.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:38.074 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:38.124 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:38.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:38.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:38.350 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:38 smithi194 ceph-mon[44089]: osdmap e906: 12 total, 12 up, 10 in 2024-04-29T14:47:38.350 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:38 smithi194 ceph-mon[44089]: osdmap e907: 12 total, 12 up, 10 in 2024-04-29T14:47:38.350 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:47:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:38.526 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:38.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:38 smithi040 ceph-mon[39581]: osdmap e906: 12 total, 12 up, 10 in 2024-04-29T14:47:38.529 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:38 smithi040 ceph-mon[39581]: osdmap e907: 12 total, 12 up, 10 in 2024-04-29T14:47:38.530 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:47:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:38.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:38.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:38.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:38 smithi121 ceph-mon[44533]: osdmap e906: 12 total, 12 up, 10 in 2024-04-29T14:47:38.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:38 smithi121 ceph-mon[44533]: osdmap e907: 12 total, 12 up, 10 in 2024-04-29T14:47:38.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:47:38.665 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:39.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:39 smithi040 ceph-mon[39581]: pgmap v2313: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 13939/117396 objects misplaced (11.873%) 2024-04-29T14:47:39.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:47:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:39.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:39.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:39 smithi121 ceph-mon[44533]: pgmap v2313: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 13939/117396 objects misplaced (11.873%) 2024-04-29T14:47:39.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:39.759 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:39.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:39.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:39 smithi194 ceph-mon[44089]: pgmap v2313: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+remapped, 14 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 13939/117396 objects misplaced (11.873%) 2024-04-29T14:47:39.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:39.922 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:40.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:119065 7.e 7:79acd9df:::benchmark_data_smithi194_147200_object4153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e706)", 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:37:43.384309+0000", 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 596.65598539500002, 2024-04-29T14:47:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23172073000000001, 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 119065 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.384309+0000", 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.384309+0000", 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.384311+0000", 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1679999999999998e-06 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.384316+0000", 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9819999999999999e-06 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.384324+0000", 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5700000000000004e-06 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.384329+0000", 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3430000000000003e-06 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.493186+0000", 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108857099 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.493351+0000", 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016500400000000001 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 10", 2024-04-29T14:47:40.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.493515+0000", 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016449699999999999 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.498229+0000", 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0047134689999999996 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.615941+0000", 2024-04-29T14:47:40.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11771164000000001 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.615982+0000", 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1155000000000002e-05 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:37:43.616030+0000", 2024-04-29T14:47:40.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7803000000000003e-05 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130753 7.3 7:ce35a2e7:::benchmark_data_smithi194_147200_object15841:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e749)", 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.116102+0000", 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 456.92419259799999, 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.295163866, 2024-04-29T14:47:40.055 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130753 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116102+0000", 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116102+0000", 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116105+0000", 2024-04-29T14:47:40.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0630000000000002e-06 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116113+0000", 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7630000000000006e-06 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116130+0000", 2024-04-29T14:47:40.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6937999999999999e-05 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116143+0000", 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2928999999999999e-05 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116319+0000", 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017568 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116330+0000", 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1449e-05 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.410559+0000", 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.294228975 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.411265+0000", 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00070606900000000003 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130757 7.3 7:c024f7e4:::benchmark_data_smithi194_147200_object15845:head [delete] snapc 0=[] ondisk+write+known_if_redirected e749)", 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.116158+0000", 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 456.92413577100001, 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30017344800000001, 2024-04-29T14:47:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130757 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116158+0000", 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116158+0000", 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116159+0000", 2024-04-29T14:47:40.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5899999999999995e-07 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116162+0000", 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.309e-06 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116169+0000", 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3990000000000001e-06 2024-04-29T14:47:40.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116175+0000", 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7649999999999999e-06 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116403+0000", 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022800500000000001 2024-04-29T14:47:40.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.116410+0000", 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.951e-06 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.415554+0000", 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.29914476299999998 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.416332+0000", 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00077749700000000004 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130718 7.2 7:4167bd69:::benchmark_data_smithi194_147200_object15806:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e750)", 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.124187+0000", 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 456.91610681200001, 2024-04-29T14:47:40.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.327846052, 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130718 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124187+0000", 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124187+0000", 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124189+0000", 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.04e-06 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124195+0000", 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4469999999999997e-06 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124204+0000", 2024-04-29T14:47:40.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6320000000000003e-06 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124214+0000", 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9739999999999997e-06 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.126111+0000", 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001896939 2024-04-29T14:47:40.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.126123+0000", 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1855000000000001e-05 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.427110+0000", 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30098713700000002 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.429671+0000", 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025610049999999999 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3,10", 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.429819+0000", 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014806800000000001 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.445611+0000", 2024-04-29T14:47:40.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01579146 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.445646+0000", 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4740000000000003e-05 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.449545+0000", 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0038997680000000001 2024-04-29T14:47:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.449604+0000", 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8440000000000003e-05 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.452033+0000", 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024295469999999998 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130726 7.2 7:4a770ae3:::benchmark_data_smithi194_147200_object15814:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e750)", 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.124227+0000", 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 456.91606762700002, 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.35389889699999999, 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.076 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130726 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124227+0000", 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124227+0000", 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124227+0000", 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4899999999999995e-07 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124230+0000", 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5749999999999999e-06 2024-04-29T14:47:40.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124234+0000", 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.668e-06 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.124239+0000", 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5859999999999998e-06 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.126146+0000", 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001907211 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.126153+0000", 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4489999999999996e-06 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436231+0000", 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.310078666 2024-04-29T14:47:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436405+0000", 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017357200000000001 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3,10", 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.436570+0000", 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000165394 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.445654+0000", 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090840090000000005 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.449503+0000", 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0038484819999999999 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.475954+0000", 2024-04-29T14:47:40.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026451684999999999 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.476001+0000", 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6110999999999999e-05 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.478125+0000", 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021249400000000001 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:130761 7.2 7:4e175b1d:::benchmark_data_smithi194_147200_object15849:head [delete] snapc 0=[] ondisk+write+known_if_redirected e750)", 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:40:03.137342+0000", 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 456.90295187499999, 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.34238187599999997, 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.084 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 130761 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.137342+0000", 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.137342+0000", 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.137345+0000", 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7650000000000002e-06 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.137353+0000", 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2309999999999996e-06 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.137366+0000", 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3091000000000001e-05 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.137378+0000", 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1693e-05 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.141828+0000", 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044497850000000004 2024-04-29T14:47:40.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.141854+0000", 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6539000000000001e-05 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.441537+0000", 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.299682381 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.441691+0000", 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015443499999999999 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 3,10", 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.441861+0000", 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016945299999999999 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.452108+0000", 2024-04-29T14:47:40.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010247707999999999 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.452284+0000", 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000175417 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.478187+0000", 2024-04-29T14:47:40.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025903664999999999 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.478248+0000", 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0517999999999999e-05 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:40:03.479724+0000", 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001476195 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.091 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:154954 7.c 7:3543cd94:::benchmark_data_smithi194_147200_object40042:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 156.56324491999999, 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041657131999999999, 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.092 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154954 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:47:40.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477051+0000", 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.807e-06 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477056+0000", 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1760000000000001e-06 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477067+0000", 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0828e-05 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477074+0000", 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0990000000000001e-06 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489561+0000", 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012487242000000001 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489704+0000", 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014225699999999999 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489831+0000", 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012778299999999999 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.497870+0000", 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080388730000000002 2024-04-29T14:47:40.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.515675+0000", 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017804868000000001 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.515755+0000", 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9532999999999999e-05 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518706+0000", 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029516659999999999 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:154957 7.c 7:39c117cf:::benchmark_data_smithi194_147200_object40045:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 156.557998879, 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039817678000000002, 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.098 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154957 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482296+0000", 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.054e-06 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482304+0000", 2024-04-29T14:47:40.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5410000000000003e-06 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482329+0000", 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4757999999999999e-05 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482336+0000", 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6459999999999993e-06 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.493836+0000", 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011499568 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.493962+0000", 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012657 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.494045+0000", 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3016000000000006e-05 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.498011+0000", 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039652560000000003 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518780+0000", 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020769699999999999 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518839+0000", 2024-04-29T14:47:40.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8284e-05 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.522113+0000", 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003274285 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155044 7.c 7:3e86b522:::benchmark_data_smithi194_147200_object40132:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:40.104 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 156.282739327, 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038996208999999997, 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155044 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757557+0000", 2024-04-29T14:47:40.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.447e-06 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757565+0000", 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2880000000000004e-06 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757578+0000", 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3828999999999999e-05 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757590+0000", 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1832e-05 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766688+0000", 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090977279999999994 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766861+0000", 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017302100000000001 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766993+0000", 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000131521 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.787943+0000", 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020950180999999998 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.793252+0000", 2024-04-29T14:47:40.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053089900000000004 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.793316+0000", 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4738000000000003e-05 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.796551+0000", 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032346340000000001 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155579 7.c 7:3ca30ab7:::benchmark_data_smithi194_147200_object40667:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.84097951199999, 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040246949999999997, 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.111 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155579 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199317+0000", 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0700000000000001e-06 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199322+0000", 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.604e-06 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199332+0000", 2024-04-29T14:47:40.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9869999999999994e-06 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199341+0000", 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1009999999999998e-06 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211121+0000", 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011779092 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211232+0000", 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111634 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211319+0000", 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7120999999999995e-05 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.216006+0000", 2024-04-29T14:47:40.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046872049999999998 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.236426+0000", 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020419567 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.236500+0000", 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4419999999999996e-05 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.239562+0000", 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003061149 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155964 7.c 7:342ad242:::benchmark_data_smithi194_147200_object41052:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 150.24830060100001, 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038617988999999998, 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.117 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155964 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791996+0000", 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5220000000000002e-06 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792002+0000", 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0700000000000003e-06 2024-04-29T14:47:40.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792012+0000", 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0268000000000001e-05 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792017+0000", 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.994e-06 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799209+0000", 2024-04-29T14:47:40.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0071913070000000001 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799312+0000", 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010362899999999999 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799433+0000", 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120448 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.804509+0000", 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050758950000000004 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.827370+0000", 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022861757 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.827437+0000", 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6568000000000004e-05 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.830612+0000", 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003174531 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:156496 7.c 7:335b5fcf:::benchmark_data_smithi194_147200_object41584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e846)", 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 146.048611335, 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040110869, 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 156496 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991686+0000", 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.613e-06 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991699+0000", 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2316e-05 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991715+0000", 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6147999999999999e-05 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991730+0000", 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4970000000000001e-05 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001539+0000", 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098087069999999998 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001670+0000", 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013161299999999999 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001804+0000", 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013356300000000001 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.014002+0000", 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012198525 2024-04-29T14:47:40.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.028772+0000", 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014770086 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.028824+0000", 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1232e-05 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.031794+0000", 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029700960000000002 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:157905 7.c 7:3da9029e:::benchmark_data_smithi194_147200_object42993:head [delete] snapc 0=[] ondisk+write+known_if_redirected e850)", 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 136.23907930999999, 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041407046000000003, 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.129 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 157905 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801218+0000", 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7939999999999998e-06 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801226+0000", 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4090000000000003e-06 2024-04-29T14:47:40.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801238+0000", 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2238999999999999e-05 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801247+0000", 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8240000000000006e-06 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812668+0000", 2024-04-29T14:47:40.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011420397000000001 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812852+0000", 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018423699999999999 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812973+0000", 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121585 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.824943+0000", 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011969484000000001 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.840398+0000", 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01545529 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.840441+0000", 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2951000000000003e-05 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.842622+0000", 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021808359999999998 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164264 7.c 7:3a985e73:::benchmark_data_smithi194_147200_object49352:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 76.869216249999994, 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010643537999999999, 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164264 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171082+0000", 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7680000000000002e-06 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171093+0000", 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0937e-05 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171114+0000", 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1294999999999999e-05 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171127+0000", 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3363999999999999e-05 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171212+0000", 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4429000000000001e-05 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171412+0000", 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020048099999999999 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171548+0000", 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013562900000000001 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181610+0000", 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010061746 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181626+0000", 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5973000000000001e-05 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181674+0000", 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8105000000000001e-05 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181721+0000", 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7811e-05 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164265 7.c 7:3e4dad70:::benchmark_data_smithi194_147200_object49353:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 76.865763641000001, 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019274217, 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.141 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164265 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174534+0000", 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4429999999999999e-06 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174545+0000", 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0548e-05 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174563+0000", 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8488999999999999e-05 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174581+0000", 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7569000000000001e-05 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176564+0000", 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019836049999999998 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176739+0000", 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017455199999999999 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176878+0000", 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013903 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181767+0000", 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048891000000000004 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193663+0000", 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011896105000000001 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193730+0000", 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7065999999999998e-05 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193805+0000", 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4709999999999995e-05 2024-04-29T14:47:40.146 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164330 7.e 7:7a9f05b1:::benchmark_data_smithi194_147200_object49418:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e869)", 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 74.319756048000002, 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15689136400000001, 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.147 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164330 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720541+0000", 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7659999999999999e-06 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720547+0000", 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2770000000000002e-06 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720561+0000", 2024-04-29T14:47:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3909e-05 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720571+0000", 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0066e-05 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720646+0000", 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4630999999999997e-05 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720659+0000", 2024-04-29T14:47:40.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3281e-05 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872536+0000", 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15187671799999999 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872672+0000", 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013665500000000001 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9,10", 2024-04-29T14:47:40.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872793+0000", 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012021 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877184+0000", 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043910989999999999 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877252+0000", 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8238000000000006e-05 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877270+0000", 2024-04-29T14:47:40.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7705000000000001e-05 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877326+0000", 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6563e-05 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877430+0000", 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010324599999999999 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164350 7.e 7:7e0a46a8:::benchmark_data_smithi194_147200_object49438:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e869)", 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 74.319705534999997, 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.16736200300000001, 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164350 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:47:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8400000000000003e-07 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720593+0000", 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2389999999999998e-06 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720599+0000", 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3350000000000003e-06 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720604+0000", 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2959999999999998e-06 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720698+0000", 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3924000000000004e-05 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720704+0000", 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1480000000000003e-06 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877463+0000", 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.156758704 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877555+0000", 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1974999999999999e-05 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9,10", 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877640+0000", 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5377999999999995e-05 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.881807+0000", 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041667179999999998 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.881835+0000", 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7815999999999999e-05 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887756+0000", 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059212279999999997 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887822+0000", 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5743999999999999e-05 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887951+0000", 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012871399999999999 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:164988 7.c e879/877 7:333ea8eb:::benchmark_data_smithi194_147200_object50076:head v 879'10294, mlcod=879'10291)", 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 56.327960191000003, 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012683506000000001, 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.162 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712335+0000", 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4799999999999998e-07 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712340+0000", 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9910000000000002e-06 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712355+0000", 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5591e-05 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712360+0000", 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1139999999999997e-06 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.716296+0000", 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039360840000000003 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.716327+0000", 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0960999999999997e-05 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.724966+0000", 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086382809999999994 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.725018+0000", 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1935999999999999e-05 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:165418 7.a e882/877 7:5a70a2d7:::benchmark_data_smithi194_147200_object50506:head v 882'17678, mlcod=881'17676)", 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 52.177774833000001, 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011664525, 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.166 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862520+0000", 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1699999999999999e-07 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862527+0000", 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0779999999999998e-06 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862543+0000", 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5698000000000001e-05 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862553+0000", 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0301e-05 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862624+0000", 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.148e-05 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862657+0000", 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2861000000000003e-05 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.874126+0000", 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011468580000000001 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.874184+0000", 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8109999999999998e-05 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:165421 7.2 e882/877 7:4462835d:::benchmark_data_smithi194_147200_object50509:head v 882'10201, mlcod=881'10199)", 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 52.171154727999998, 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013165216, 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:47:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869140+0000", 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8799999999999998e-07 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869147+0000", 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1590000000000004e-06 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869167+0000", 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.014e-05 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869177+0000", 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0088999999999999e-05 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:47:40.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.873690+0000", 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045127489999999999 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.873721+0000", 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1281000000000001e-05 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.882230+0000", 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0085086999999999992 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:47:40.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.882305+0000", 2024-04-29T14:47:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4709999999999995e-05 2024-04-29T14:47:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:47:40.174 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:47:40.174 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:47:40.174 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:40.221 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:40.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:40.325 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:40.352 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:40.371 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:40.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:40.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:40.500 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:40.520 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:40.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:40.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:40.651 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:40.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:40.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:40.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:40.800 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:40.818 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:40.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:40.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:40.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:40 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:40.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:40 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:40.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:40 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:47:40.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:40 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:47:40.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:40 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:40.950 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:40 smithi121 ceph-mon[44533]: pgmap v2314: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 145 op/s; 12605/116486 objects misplaced (10.821%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:47:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:40.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:40.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:40 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:40 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:40 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:47:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:40 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:47:41.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:40 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:41.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:40 smithi194 ceph-mon[44089]: pgmap v2314: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 145 op/s; 12605/116486 objects misplaced (10.821%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:47:41.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:41.107 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:40 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:41.107 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:40 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:41.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:47:41.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:40 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:47:41.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:40 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:47:41.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:40 smithi040 ceph-mon[39581]: pgmap v2314: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 145 op/s; 12605/116486 objects misplaced (10.821%); 14 MiB/s, 216 objects/s recovering 2024-04-29T14:47:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:41.131 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:41.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:41.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:41.266 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:41.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:41.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:41.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:41.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:41.418 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:41.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:41.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:41.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:41.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:41.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:41.688 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:47:41.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:41.718 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:41.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:41.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:41.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:41.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:41.960 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:47:41.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:42.183 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:42.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:42.243 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:47:42.265 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:47:42.285 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:47:42.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:42.306 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:47:42.320 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:42.329 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:47:42.352 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:47:42.388 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:47:42.411 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:47:42.431 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:47:42.455 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:47:42.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1852305663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:42.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:42.489 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:47:42.515 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:47:42.517 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:47:42.518 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:47:42.538 INFO:tasks.thrashosds.thrasher:in_osds: [0, 4, 7, 9, 10, 6, 1, 5, 8, 3] out_osds: [11, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:47:42.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-29T14:47:42.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1852305663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:42.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:42.601 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1852305663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:42.603 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:42.629 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:42.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:42.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:42.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:42.766 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:43.316 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:43.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:43.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:43.452 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:43.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: pgmap v2315: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 138 op/s; 12605/115736 objects misplaced (10.891%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:47:43.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:43 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: pgmap v2315: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 138 op/s; 12605/115736 objects misplaced (10.891%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:47:43.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:43 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: pgmap v2315: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 138 op/s; 12605/115736 objects misplaced (10.891%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:43 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:43.808 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:43.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:43.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:44.008 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:47:44.008 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:47:44.294 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:44.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:44.307 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:44.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.401 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:44.432 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:44.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:44.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:47:44.782 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:44.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:44.886 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:44.935 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:44.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:45.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:45.067 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:47:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:45.204 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:45.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:45.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:45.335 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:45.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:47:45.367 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:45 smithi040 ceph-mon[39581]: pgmap v2316: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 163 op/s; 12446/114806 objects misplaced (10.841%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:47:45.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:45 smithi194 ceph-mon[44089]: pgmap v2316: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 163 op/s; 12446/114806 objects misplaced (10.841%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:47:45.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:45 smithi121 ceph-mon[44533]: pgmap v2316: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 163 op/s; 12446/114806 objects misplaced (10.841%); 11 MiB/s, 176 objects/s recovering 2024-04-29T14:47:46.170 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:47:46.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:46.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:46.442 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3365689950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:47:46.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:46.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:46.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3365689950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:47:46.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:46.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:46.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3365689950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:47:47.184 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:47.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:47.290 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:47.318 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:47.338 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:47.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:47.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:47 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:47.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3365689950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:47:47.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:47 smithi040 ceph-mon[39581]: osdmap e908: 12 total, 12 up, 10 in 2024-04-29T14:47:47.425 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:47 smithi040 ceph-mon[39581]: pgmap v2318: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 187 op/s; 12446/114126 objects misplaced (10.905%); 10 MiB/s, 160 objects/s recovering 2024-04-29T14:47:47.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:47.469 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:47.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:47.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:47.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:47 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:47.511 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3365689950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:47:47.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:47 smithi194 ceph-mon[44089]: osdmap e908: 12 total, 12 up, 10 in 2024-04-29T14:47:47.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:47 smithi194 ceph-mon[44089]: pgmap v2318: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 187 op/s; 12446/114126 objects misplaced (10.905%); 10 MiB/s, 160 objects/s recovering 2024-04-29T14:47:47.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:47.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:47 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:47:47.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3365689950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:47:47.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:47 smithi121 ceph-mon[44533]: osdmap e908: 12 total, 12 up, 10 in 2024-04-29T14:47:47.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:47 smithi121 ceph-mon[44533]: pgmap v2318: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 187 op/s; 12446/114126 objects misplaced (10.905%); 10 MiB/s, 160 objects/s recovering 2024-04-29T14:47:47.620 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:47.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:47.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:47.746 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:47.774 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:47.848 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:47.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:47.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:47.981 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:48.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:48.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:48.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:48.134 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:48.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:48.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:48.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:48.320 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:48.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:48.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:48.458 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:48.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:48 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:48.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:48 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:48.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:48 smithi194 ceph-mon[44089]: osdmap e909: 12 total, 12 up, 10 in 2024-04-29T14:47:48.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:48 smithi194 ceph-mon[44089]: pgmap v2320: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 12446/114126 objects misplaced (10.905%); 904 KiB/s, 14 objects/s recovering 2024-04-29T14:47:48.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:48 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:48.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:48 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:48.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:48 smithi040 ceph-mon[39581]: osdmap e909: 12 total, 12 up, 10 in 2024-04-29T14:47:48.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:48 smithi040 ceph-mon[39581]: pgmap v2320: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 12446/114126 objects misplaced (10.905%); 904 KiB/s, 14 objects/s recovering 2024-04-29T14:47:48.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:48 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:48.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:48 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:48.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:48 smithi121 ceph-mon[44533]: osdmap e909: 12 total, 12 up, 10 in 2024-04-29T14:47:48.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:48 smithi121 ceph-mon[44533]: pgmap v2320: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 12446/114126 objects misplaced (10.905%); 904 KiB/s, 14 objects/s recovering 2024-04-29T14:47:49.008 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:49.010 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:47:49.010 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:47:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:49.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:49.113 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:47:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:49.154 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:47:49.175 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:47:49.178 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:47:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:49.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:47:49.200 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:47:49.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:47:49.222 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:47:49.245 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:47:49.266 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:47:49.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:49.287 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:47:49.306 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:49.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:47:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:49.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:49.331 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:47:49.353 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:47:49.375 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:47:49.416 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:47:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:47:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:47:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:47:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:47:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:47:49.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:49.458 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:49.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:49.562 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-04-29T14:47:49.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:49.609 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:49.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:49.706 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:49 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:47:49.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:49 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:47:49.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1905502029' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:47:49.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1515216355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:49.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:49.707 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:49.732 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:49.734 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:47:49.803 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:49.826 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:49.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:49.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:49 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:47:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:49 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:47:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1905502029' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:47:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1515216355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:49.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:49 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:47:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:49 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:47:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1905502029' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T14:47:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1515216355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:49.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:49.957 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:49.975 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:49.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:49.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:50.047 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:47:50.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:50.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1905502029' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:47:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osdmap e910: 12 total, 12 up, 10 in 2024-04-29T14:47:50.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: pgmap v2322: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 12287/113112 objects misplaced (10.863%); 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:47:50.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:50 smithi194 ceph-mon[44089]: osdmap e911: 12 total, 12 up, 10 in 2024-04-29T14:47:50.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.828 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1905502029' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osdmap e910: 12 total, 12 up, 10 in 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: pgmap v2322: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 12287/113112 objects misplaced (10.863%); 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:47:50.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:50 smithi040 ceph-mon[39581]: osdmap e911: 12 total, 12 up, 10 in 2024-04-29T14:47:50.860 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:50.862 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:50.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:50.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-29T14:47:50.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1905502029' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osdmap e910: 12 total, 12 up, 10 in 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: pgmap v2322: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 12287/113112 objects misplaced (10.863%); 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:47:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:50 smithi121 ceph-mon[44533]: osdmap e911: 12 total, 12 up, 10 in 2024-04-29T14:47:50.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:50.991 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:51.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:51.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:51.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:51.142 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:51.392 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:51.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:51.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:51.524 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:51.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:51.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:51.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:51.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:51 smithi040 ceph-mon[39581]: osdmap e912: 12 total, 12 up, 10 in 2024-04-29T14:47:51.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:47:51.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:47:51.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:51 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:47:51.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:51.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:51.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:51.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:51 smithi194 ceph-mon[44089]: osdmap e912: 12 total, 12 up, 10 in 2024-04-29T14:47:51.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:47:51.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:47:51.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:51 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:47:51.698 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:51.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:51.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:51 smithi121 ceph-mon[44533]: osdmap e912: 12 total, 12 up, 10 in 2024-04-29T14:47:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:47:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:47:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:51 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:47:52.002 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:47:52.002 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:47:52.002 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:47:52.002 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123609 7.9 7:904f1bc6:::benchmark_data_smithi194_147200_object8697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418202+0000", 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 551.57877019600005, 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031149342999999999, 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123609 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.003 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418202+0000", 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418202+0000", 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418203+0000", 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5400000000000001e-07 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.004 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418205+0000", 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4609999999999998e-06 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418210+0000", 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4049999999999999e-06 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418215+0000", 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0250000000000002e-06 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.005 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418256+0000", 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0725000000000001e-05 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418264+0000", 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7020000000000001e-06 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.433343+0000", 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015079441000000001 2024-04-29T14:47:52.006 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.433454+0000", 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000110587 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.433564+0000", 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109885 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438324+0000", 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047597919999999997 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.007 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449133+0000", 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010809272999999999 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449293+0000", 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016066299999999999 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449351+0000", 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.783e-05 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.008 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123615 7.9 7:9af3026f:::benchmark_data_smithi194_147200_object8703:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418222+0000", 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 551.57874971599995, 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031260789999999997, 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123615 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.009 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418222+0000", 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418222+0000", 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418223+0000", 2024-04-29T14:47:52.010 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4199999999999996e-07 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418234+0000", 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.093e-05 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418239+0000", 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7419999999999997e-06 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418245+0000", 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3339999999999998e-06 2024-04-29T14:47:52.011 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418282+0000", 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6724000000000003e-05 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418285+0000", 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4290000000000001e-06 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.435211+0000", 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016926225 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.435315+0000", 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103501 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.435420+0000", 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105408 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438328+0000", 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029076879999999999 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.013 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449399+0000", 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011071263 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449446+0000", 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7166999999999998e-05 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449483+0000", 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6837e-05 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123619 7.9 7:98d7b41f:::benchmark_data_smithi194_147200_object8707:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418252+0000", 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 551.578719912, 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031387258000000001, 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123619 2024-04-29T14:47:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418252+0000", 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418252+0000", 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418253+0000", 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5400000000000001e-07 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418259+0000", 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4960000000000001e-06 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418264+0000", 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0039999999999999e-06 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418269+0000", 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1610000000000002e-06 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418312+0000", 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2868999999999999e-05 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418317+0000", 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2549999999999999e-06 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436914+0000", 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018597783 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.436984+0000", 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9375999999999994e-05 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.437046+0000", 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2255999999999995e-05 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.439736+0000", 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026896279999999999 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449541+0000", 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0098051990000000006 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449598+0000", 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7167999999999999e-05 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449639+0000", 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1508999999999998e-05 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:123632 7.9 7:90a1475f:::benchmark_data_smithi194_147200_object8720:head [delete] snapc 0=[] ondisk+write+known_if_redirected e721)", 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 551.57868941300001, 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031508860999999999, 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 123632 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418283+0000", 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418286+0000", 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.424e-06 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418291+0000", 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9230000000000001e-06 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418296+0000", 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2259999999999996e-06 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418337+0000", 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1319999999999997e-05 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.418344+0000", 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4790000000000001e-06 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438358+0000", 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020013986000000001 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438463+0000", 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010518000000000001 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0", 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.438567+0000", 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010400600000000001 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.439853+0000", 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001286424 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449691+0000", 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0098375330000000007 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449748+0000", 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7336999999999997e-05 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:38:40.449792+0000", 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3399000000000002e-05 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:154866 7.f 7:fadc60f2:::benchmark_data_smithi194_147200_object39954:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:03.057525+0000", 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.93944672800001, 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031965947000000001, 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.027 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 154866 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.057525+0000", 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.057525+0000", 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.057528+0000", 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7310000000000002e-06 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.057536+0000", 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.011e-06 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.057550+0000", 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3351e-05 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.057559+0000", 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5650000000000007e-06 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.063036+0000", 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054769479999999997 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.063122+0000", 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5680999999999995e-05 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.063226+0000", 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103853 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.069362+0000", 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061368869999999997 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.087128+0000", 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017766046000000001 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.087187+0000", 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8646000000000001e-05 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.089491+0000", 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023042280000000002 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:154867 7.f 7:f5639864:::benchmark_data_smithi194_147200_object39955:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:03.059773+0000", 2024-04-29T14:47:52.032 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.937198763, 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031495876999999999, 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 154867 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.059773+0000", 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.059773+0000", 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.059775+0000", 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2330000000000001e-06 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.059780+0000", 2024-04-29T14:47:52.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3499999999999996e-06 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.059788+0000", 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1589999999999993e-06 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.059795+0000", 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0690000000000004e-06 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.065593+0000", 2024-04-29T14:47:52.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057975839999999997 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.065699+0000", 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106091 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.065793+0000", 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3790999999999997e-05 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.069492+0000", 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036991120000000001 2024-04-29T14:47:52.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.089539+0000", 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020047560999999998 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.089576+0000", 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6307000000000003e-05 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.091269+0000", 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016936200000000001 2024-04-29T14:47:52.037 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:155065 7.f 7:f9a4d578:::benchmark_data_smithi194_147200_object40153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:03.846628+0000", 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 168.15034450499999, 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035149000999999999, 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.038 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 155065 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.846628+0000", 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.846628+0000", 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.846629+0000", 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4099999999999997e-07 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.846633+0000", 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.437e-06 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.846639+0000", 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2829999999999998e-06 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.846644+0000", 2024-04-29T14:47:52.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2350000000000001e-06 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.850237+0000", 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00359333 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.850358+0000", 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012129399999999999 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.850458+0000", 2024-04-29T14:47:52.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000100295 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.852902+0000", 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024434539999999999 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.879311+0000", 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.026409526999999999 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.879376+0000", 2024-04-29T14:47:52.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4904000000000001e-05 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:03.881777+0000", 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024003010000000001 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:155544 7.f 7:fe07b288:::benchmark_data_smithi194_147200_object40632:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:06.798394+0000", 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 165.198577723, 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.036355187999999997, 2024-04-29T14:47:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 155544 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.798394+0000", 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.798394+0000", 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.798398+0000", 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3359999999999999e-06 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.798409+0000", 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0981e-05 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.798422+0000", 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3301e-05 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.798438+0000", 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6350000000000001e-05 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.806267+0000", 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078287500000000006 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.806422+0000", 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015493500000000001 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.806561+0000", 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013855900000000001 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.810181+0000", 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036199610000000001 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.832396+0000", 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022215269999999999 2024-04-29T14:47:52.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.832465+0000", 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9318000000000003e-05 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:06.834750+0000", 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022844269999999999 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:155591 7.f 7:fb9d99ec:::benchmark_data_smithi194_147200_object40679:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:07.256916+0000", 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 164.74005582699999, 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031726322000000001, 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 155591 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.050 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.256916+0000", 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.256916+0000", 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.256917+0000", 2024-04-29T14:47:52.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.256919+0000", 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0959999999999999e-06 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.256922+0000", 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.517e-06 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.256926+0000", 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4410000000000002e-06 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.262368+0000", 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054423919999999999 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.262513+0000", 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014491100000000001 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.262597+0000", 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3897e-05 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.268746+0000", 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061486900000000001 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.287185+0000", 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018438744999999999 2024-04-29T14:47:52.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.287218+0000", 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3893000000000002e-05 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:07.288643+0000", 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014241810000000001 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:155830 7.f 7:f62da4e6:::benchmark_data_smithi194_147200_object40918:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:08.622516+0000", 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 163.37445617700001, 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.036908335, 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 155830 2024-04-29T14:47:52.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.622516+0000", 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.622516+0000", 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.622519+0000", 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1980000000000001e-06 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.622527+0000", 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1100000000000003e-06 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.622539+0000", 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1538e-05 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.622546+0000", 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3390000000000004e-06 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.630822+0000", 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082755099999999998 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.630914+0000", 2024-04-29T14:47:52.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2591999999999997e-05 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.630991+0000", 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6364000000000003e-05 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.636249+0000", 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052585469999999997 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.657032+0000", 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020783131 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.657096+0000", 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4022999999999993e-05 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:08.659424+0000", 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002327983 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.061 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:156577 7.f 7:f9441740:::benchmark_data_smithi194_147200_object41665:head [delete] snapc 0=[] ondisk+write+known_if_redirected e846)", 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:14.460884+0000", 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 157.536088621, 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.031630332999999997, 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.062 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 156577 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.460884+0000", 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.460884+0000", 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.460886+0000", 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.311e-06 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.460892+0000", 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.460902+0000", 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.058e-05 2024-04-29T14:47:52.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.460909+0000", 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9609999999999999e-06 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.468691+0000", 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077817379999999998 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.468853+0000", 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016199699999999999 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.468996+0000", 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014310899999999999 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.471650+0000", 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002653674 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.490420+0000", 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01876978 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.490458+0000", 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8661999999999997e-05 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:14.492514+0000", 2024-04-29T14:47:52.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020556110000000002 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:157297 7.f 7:f5604e9c:::benchmark_data_smithi194_147200_object42385:head [delete] snapc 0=[] ondisk+write+known_if_redirected e847)", 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:19.327210+0000", 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 152.66976226, 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.034743951000000002, 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.068 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 157297 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.327210+0000", 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.327210+0000", 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.327213+0000", 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.892e-06 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.327221+0000", 2024-04-29T14:47:52.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7489999999999998e-06 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.327233+0000", 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2330000000000001e-05 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.327240+0000", 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0949999999999998e-06 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.333737+0000", 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064971969999999997 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.333835+0000", 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8325000000000005e-05 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.333931+0000", 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5787999999999998e-05 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.337020+0000", 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030888080000000002 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.359743+0000", 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022722510000000001 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.359788+0000", 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5176999999999998e-05 2024-04-29T14:47:52.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.361954+0000", 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021660799999999999 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:157340 7.f 7:f863e6d3:::benchmark_data_smithi194_147200_object42428:head [delete] snapc 0=[] ondisk+write+known_if_redirected e847)", 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:19.587677+0000", 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 152.40929552399999, 2024-04-29T14:47:52.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.030607807000000001, 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 157340 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.587677+0000", 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.587677+0000", 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.587678+0000", 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.759e-06 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.587684+0000", 2024-04-29T14:47:52.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4469999999999997e-06 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.587694+0000", 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8810000000000008e-06 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.587700+0000", 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8320000000000002e-06 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.594496+0000", 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067963219999999996 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.594626+0000", 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000130013 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 9", 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.594748+0000", 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000122218 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.599981+0000", 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052325749999999997 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.616417+0000", 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016436168000000001 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.616487+0000", 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0062999999999996e-05 2024-04-29T14:47:52.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:19.618284+0000", 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001797529 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173677 7.f 7.5de73dcf (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:49.669567+0000", 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.3274049479999999, 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.88943558899999997, 2024-04-29T14:47:52.080 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173677 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669567+0000", 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669567+0000", 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669569+0000", 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9259999999999999e-06 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669574+0000", 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.904e-06 2024-04-29T14:47:52.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669585+0000", 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1327e-05 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669592+0000", 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4799999999999998e-06 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669643+0000", 2024-04-29T14:47:52.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1320999999999999e-05 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669653+0000", 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3570000000000004e-06 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.558989+0000", 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.88933647800000004 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559003+0000", 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3796e-05 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173677 7.f 7.5de73dcf (undecoded) ondisk+retry+write+known_if_redirected e910)", 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:49.933187+0000", 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.0637850549999999, 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62587116700000001, 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.085 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173677 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933187+0000", 2024-04-29T14:47:52.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933187+0000", 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933189+0000", 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0889999999999998e-06 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933194+0000", 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4170000000000002e-06 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933202+0000", 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0250000000000002e-06 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933212+0000", 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0618000000000001e-05 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933265+0000", 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2488000000000003e-05 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933273+0000", 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5019999999999992e-06 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559030+0000", 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62575718300000005 2024-04-29T14:47:52.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559058+0000", 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7844999999999999e-05 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173696 7.f 7.435a06af (undecoded) ondisk+retry+write+known_if_redirected e910)", 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:49.933229+0000", 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.0637431589999999, 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62588960400000004, 2024-04-29T14:47:52.090 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173696 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933229+0000", 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933229+0000", 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933230+0000", 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4799999999999997e-07 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933233+0000", 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2449999999999998e-06 2024-04-29T14:47:52.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933239+0000", 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8080000000000001e-06 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933245+0000", 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0430000000000004e-06 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933296+0000", 2024-04-29T14:47:52.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1267000000000002e-05 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933304+0000", 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.481e-06 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559107+0000", 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62580390200000002 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559119+0000", 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.111e-05 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173702 7.f 7.a3ceca0f (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.068392+0000", 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.928579904, 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49078070699999998, 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.095 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173702 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068392+0000", 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068392+0000", 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068396+0000", 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.473e-06 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068408+0000", 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2313000000000001e-05 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068430+0000", 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1793999999999998e-05 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068441+0000", 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1674e-05 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068511+0000", 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9958999999999999e-05 2024-04-29T14:47:52.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068521+0000", 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5100000000000004e-06 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559161+0000", 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49064043299999999 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559173+0000", 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1551e-05 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173717 7.f 7.d56abeef (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.379996+0000", 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.616976194, 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17923861099999999, 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:52.100 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173717 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.379996+0000", 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.379996+0000", 2024-04-29T14:47:52.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.379999+0000", 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.875e-06 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380007+0000", 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1850000000000004e-06 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380023+0000", 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5628e-05 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380030+0000", 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8569999999999995e-06 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380098+0000", 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7161000000000003e-05 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380111+0000", 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3012e-05 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559225+0000", 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.179114256 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559235+0000", 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6369999999999994e-06 2024-04-29T14:47:52.104 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173718 7.f 7.3bfc8dbf (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.383765+0000", 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.6132069260000002, 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17552379800000001, 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.105 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173718 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383765+0000", 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383765+0000", 2024-04-29T14:47:52.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383767+0000", 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3489999999999999e-06 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383771+0000", 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9289999999999997e-06 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383782+0000", 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0034000000000001e-05 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383787+0000", 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8139999999999997e-06 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383833+0000", 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5291999999999999e-05 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383840+0000", 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4970000000000004e-06 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559278+0000", 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17543802999999999 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559289+0000", 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0852999999999999e-05 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.109 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173727 7.f 7.fc70d24f (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.492353+0000", 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.504619355, 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.066978419999999997, 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:47:52.110 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173727 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492353+0000", 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492353+0000", 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:47:52.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492355+0000", 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2469999999999998e-06 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492361+0000", 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2079999999999997e-06 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492374+0000", 2024-04-29T14:47:52.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2605e-05 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492381+0000", 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2960000000000001e-06 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492465+0000", 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3616999999999996e-05 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:47:52.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.492479+0000", 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3746e-05 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559325+0000", 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.066846560999999999 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559331+0000", 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1399999999999997e-06 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.114 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.115 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:47:52.115 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:47:52.115 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:47:52.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: 7.f starting backfill to osd.0 from (0'0,837'9728] MIN to 909'10902 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: 7.4 starting backfill to osd.0 from (0'0,837'9771] MIN to 909'10974 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: 7.5 starting backfill to osd.0 from (0'0,837'9639] MIN to 909'10847 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: pgmap v2325: 17 pgs: 2 peering, 2 active+recovering+remapped, 13 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 174 op/s; 12287/112480 objects misplaced (10.924%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:47:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:52 smithi194 ceph-mon[44089]: osdmap e913: 12 total, 12 up, 10 in 2024-04-29T14:47:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: 7.f starting backfill to osd.0 from (0'0,837'9728] MIN to 909'10902 2024-04-29T14:47:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: 7.4 starting backfill to osd.0 from (0'0,837'9771] MIN to 909'10974 2024-04-29T14:47:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: 7.5 starting backfill to osd.0 from (0'0,837'9639] MIN to 909'10847 2024-04-29T14:47:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:47:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:47:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:47:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:47:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: pgmap v2325: 17 pgs: 2 peering, 2 active+recovering+remapped, 13 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 174 op/s; 12287/112480 objects misplaced (10.924%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:47:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:52 smithi040 ceph-mon[39581]: osdmap e913: 12 total, 12 up, 10 in 2024-04-29T14:47:52.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: 7.f starting backfill to osd.0 from (0'0,837'9728] MIN to 909'10902 2024-04-29T14:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: 7.4 starting backfill to osd.0 from (0'0,837'9771] MIN to 909'10974 2024-04-29T14:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: 7.5 starting backfill to osd.0 from (0'0,837'9639] MIN to 909'10847 2024-04-29T14:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: pgmap v2325: 17 pgs: 2 peering, 2 active+recovering+remapped, 13 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 174 op/s; 12287/112480 objects misplaced (10.924%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:52 smithi121 ceph-mon[44533]: osdmap e913: 12 total, 12 up, 10 in 2024-04-29T14:47:53.189 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:47:53.257 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:53.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:53.273 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:53.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:53.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:47:53.394 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:53.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:53.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:53.552 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:47:53.765 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:47:53.796 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:53.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:53.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:47:53.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:47:53.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:53.938 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:54.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:54.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:54.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:47:54.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:47:54.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:54.196 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:54.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:54.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:54.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:54.423 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:54.445 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:54.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:54.549 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:54.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:54 smithi194 ceph-mon[44089]: osdmap e914: 12 total, 12 up, 10 in 2024-04-29T14:47:54.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:54 smithi194 ceph-mon[44089]: pgmap v2328: 17 pgs: 1 active+remapped+backfilling, 2 peering, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 158 op/s; 12994/111844 objects misplaced (11.618%); 7.0 MiB/s, 112 objects/s recovering 2024-04-29T14:47:54.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:54 smithi040 ceph-mon[39581]: osdmap e914: 12 total, 12 up, 10 in 2024-04-29T14:47:54.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:54 smithi040 ceph-mon[39581]: pgmap v2328: 17 pgs: 1 active+remapped+backfilling, 2 peering, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 158 op/s; 12994/111844 objects misplaced (11.618%); 7.0 MiB/s, 112 objects/s recovering 2024-04-29T14:47:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:54.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:54.961 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:47:54.978 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:54 smithi121 ceph-mon[44533]: osdmap e914: 12 total, 12 up, 10 in 2024-04-29T14:47:54.978 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:54 smithi121 ceph-mon[44533]: pgmap v2328: 17 pgs: 1 active+remapped+backfilling, 2 peering, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 158 op/s; 12994/111844 objects misplaced (11.618%); 7.0 MiB/s, 112 objects/s recovering 2024-04-29T14:47:55.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:55.072 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:55.281 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:47:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:47:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:47:55.281 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:47:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:47:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:55.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:47:55.630 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:47:55.662 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:55.684 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:47:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:55.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:47:55.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:55.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:55 smithi040 ceph-mon[39581]: osdmap e915: 12 total, 12 up, 10 in 2024-04-29T14:47:55.823 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:55.845 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:55.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:55.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:55 smithi121 ceph-mon[44533]: osdmap e915: 12 total, 12 up, 10 in 2024-04-29T14:47:55.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:47:55.984 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:55.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:47:56.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:47:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:56.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:47:56.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:55 smithi194 ceph-mon[44089]: osdmap e915: 12 total, 12 up, 10 in 2024-04-29T14:47:56.110 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:47:56.139 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:56.163 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:47:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:56.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:47:56.267 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:47:56.296 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:56.319 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:56.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:56.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:56.450 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:56.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:56.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:56.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:56.657 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:56.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:56.732 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:56 smithi121 ceph-mon[44533]: osdmap e916: 12 total, 12 up, 10 in 2024-04-29T14:47:56.732 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:56 smithi121 ceph-mon[44533]: pgmap v2331: 17 pgs: 3 active+remapped+backfilling, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 16996/111236 objects misplaced (15.279%); 52 MiB/s, 834 objects/s recovering 2024-04-29T14:47:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:56.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:56.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:56.863 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:57.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:57.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:57.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:56 smithi194 ceph-mon[44089]: osdmap e916: 12 total, 12 up, 10 in 2024-04-29T14:47:57.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:56 smithi194 ceph-mon[44089]: pgmap v2331: 17 pgs: 3 active+remapped+backfilling, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 16996/111236 objects misplaced (15.279%); 52 MiB/s, 834 objects/s recovering 2024-04-29T14:47:57.095 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:56 smithi040 ceph-mon[39581]: osdmap e916: 12 total, 12 up, 10 in 2024-04-29T14:47:57.095 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:56 smithi040 ceph-mon[39581]: pgmap v2331: 17 pgs: 3 active+remapped+backfilling, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 16996/111236 objects misplaced (15.279%); 52 MiB/s, 834 objects/s recovering 2024-04-29T14:47:57.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:47:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:57.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:47:57.449 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:47:57.477 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:57.499 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:47:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:57.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:47:57.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:47:57.633 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:58.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:47:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:58.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:47:58.216 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:47:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:47:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:47:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:47:58.217 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:47:58.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:47:58.323 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:58.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:47:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:58.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:47:58.569 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:47:58.571 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:47:58.573 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.4 to 1 (10000) 2024-04-29T14:47:58.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:47:58.592 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:47:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:58.615 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:47:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:47:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:58.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:47:58.640 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:47:58.664 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:47:58.685 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:47:58.705 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:47:58.724 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:47:58.741 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:47:58.745 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:47:58.767 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:58.780 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:47:58.786 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:47:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:58.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:47:58.801 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:47:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:58 smithi194 ceph-mon[44089]: osdmap e917: 12 total, 12 up, 10 in 2024-04-29T14:47:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3313471768' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:58 smithi194 ceph-mon[44089]: from='client.27694 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:58.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3275959558' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:47:58.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:58 smithi194 ceph-mon[44089]: from='client.37448 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:47:58.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:58 smithi194 ceph-mon[44089]: pgmap v2333: 17 pgs: 3 active+remapped+backfilling, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 16996/111236 objects misplaced (15.279%); 45 MiB/s, 717 objects/s recovering 2024-04-29T14:47:58.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2586589269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:58.824 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:47:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:47:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:47:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:47:58.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:58 smithi040 ceph-mon[39581]: osdmap e917: 12 total, 12 up, 10 in 2024-04-29T14:47:58.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3313471768' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:58.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:58 smithi040 ceph-mon[39581]: from='client.27694 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:58.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3275959558' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:47:58.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:58 smithi040 ceph-mon[39581]: from='client.37448 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:47:58.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:58 smithi040 ceph-mon[39581]: pgmap v2333: 17 pgs: 3 active+remapped+backfilling, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 16996/111236 objects misplaced (15.279%); 45 MiB/s, 717 objects/s recovering 2024-04-29T14:47:58.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2586589269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:58.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:58 smithi121 ceph-mon[44533]: osdmap e917: 12 total, 12 up, 10 in 2024-04-29T14:47:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3313471768' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:58 smithi121 ceph-mon[44533]: from='client.27694 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:47:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3275959558' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:47:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:58 smithi121 ceph-mon[44533]: from='client.37448 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:47:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:58 smithi121 ceph-mon[44533]: pgmap v2333: 17 pgs: 3 active+remapped+backfilling, 2 active+recovering+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 16996/111236 objects misplaced (15.279%); 45 MiB/s, 717 objects/s recovering 2024-04-29T14:47:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2586589269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:47:58.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:47:58.925 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:58.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:47:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:47:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:47:59.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:47:59.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:47:59.217 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:47:59.270 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:47:59.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: from='client.27694 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: from='client.37448 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osdmap e918: 12 total, 12 up, 10 in 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:47:59 smithi194 ceph-mon[44089]: 7.e scrub starts 2024-04-29T14:47:59.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:47:59.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: from='client.27694 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: from='client.37448 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osdmap e918: 12 total, 12 up, 10 in 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:47:59 smithi040 ceph-mon[39581]: 7.e scrub starts 2024-04-29T14:47:59.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-29T14:47:59.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: from='client.27694 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: from='client.37448 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osdmap e918: 12 total, 12 up, 10 in 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:47:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:47:59 smithi121 ceph-mon[44533]: 7.e scrub starts 2024-04-29T14:48:00.268 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:00.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:00.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:00.402 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:00.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:00.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:00.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:00.688 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:00.745 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:00.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:00.778 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:00 smithi194 ceph-mon[44089]: osdmap e919: 12 total, 12 up, 10 in 2024-04-29T14:48:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:00 smithi194 ceph-mon[44089]: 7.e scrub ok 2024-04-29T14:48:00.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:00 smithi194 ceph-mon[44089]: pgmap v2336: 17 pgs: 2 active+remapped+backfilling, 2 active+recovering+remapped, 13 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 15892/110394 objects misplaced (14.396%); 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:48:00.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:00.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:00 smithi121 ceph-mon[44533]: osdmap e919: 12 total, 12 up, 10 in 2024-04-29T14:48:00.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:00 smithi121 ceph-mon[44533]: 7.e scrub ok 2024-04-29T14:48:00.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:00 smithi121 ceph-mon[44533]: pgmap v2336: 17 pgs: 2 active+remapped+backfilling, 2 active+recovering+remapped, 13 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 15892/110394 objects misplaced (14.396%); 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:48:00.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:00 smithi040 ceph-mon[39581]: osdmap e919: 12 total, 12 up, 10 in 2024-04-29T14:48:00.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:00 smithi040 ceph-mon[39581]: 7.e scrub ok 2024-04-29T14:48:00.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:00 smithi040 ceph-mon[39581]: pgmap v2336: 17 pgs: 2 active+remapped+backfilling, 2 active+recovering+remapped, 13 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 15892/110394 objects misplaced (14.396%); 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:48:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:01.079 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:01.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:01.170 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:48:01.170 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:48:01.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:01.418 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:01.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:01.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:01.555 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:01.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:01 smithi040 ceph-mon[39581]: 7.a scrub starts 2024-04-29T14:48:01.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:01 smithi194 ceph-mon[44089]: 7.a scrub starts 2024-04-29T14:48:01.846 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:01 smithi121 ceph-mon[44533]: 7.a scrub starts 2024-04-29T14:48:01.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:48:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:01.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:02.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:02.039 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:02.058 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:02.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:02.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:02.192 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:02.223 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:02.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:02.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:02.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:48:02.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:02.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:02.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:02.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:02 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:48:02.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:02 smithi040 ceph-mon[39581]: 7.a scrub ok 2024-04-29T14:48:02.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:02 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:48:02.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:02 smithi040 ceph-mon[39581]: pgmap v2337: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 135 op/s; 11890/109606 objects misplaced (10.848%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:48:02.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:48:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:02 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:48:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:02 smithi121 ceph-mon[44533]: 7.a scrub ok 2024-04-29T14:48:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:02 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:48:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:02 smithi121 ceph-mon[44533]: pgmap v2337: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 135 op/s; 11890/109606 objects misplaced (10.848%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:48:03.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:02 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:48:03.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:02 smithi194 ceph-mon[44089]: 7.a scrub ok 2024-04-29T14:48:03.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:02 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:48:03.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:02 smithi194 ceph-mon[44089]: pgmap v2337: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 135 op/s; 11890/109606 objects misplaced (10.848%); 21 MiB/s, 329 objects/s recovering 2024-04-29T14:48:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:03.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:03.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:03.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:03.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:03.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:03 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:48:03.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:03 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:48:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:03.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:03.973 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:03.986 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:04.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:04.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:03 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:48:04.080 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:04.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:04.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.7 to 1 (10000) 2024-04-29T14:48:04.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:04 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:48:04.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4001722721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:48:04.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:04 smithi040 ceph-mon[39581]: from='client.37460 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:48:04.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:04.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:04.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:04 smithi040 ceph-mon[39581]: pgmap v2338: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 182 op/s; 11653/108774 objects misplaced (10.713%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:48:04.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:04 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:48:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4001722721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:48:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:04 smithi121 ceph-mon[44533]: from='client.37460 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:48:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:04 smithi121 ceph-mon[44533]: pgmap v2338: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 182 op/s; 11653/108774 objects misplaced (10.713%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:48:05.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:04 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:48:05.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4001722721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:48:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:04 smithi194 ceph-mon[44089]: from='client.37460 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:48:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:04 smithi194 ceph-mon[44089]: pgmap v2338: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 182 op/s; 11653/108774 objects misplaced (10.713%); 20 MiB/s, 313 objects/s recovering 2024-04-29T14:48:05.303 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:05.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:05.313 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:48:05.313 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:48:05.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:05.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:05 smithi040 ceph-mon[39581]: from='client.37460 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:48:05.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:05 smithi040 ceph-mon[39581]: osdmap e920: 12 total, 12 up, 10 in 2024-04-29T14:48:05.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:05 smithi121 ceph-mon[44533]: from='client.37460 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:48:05.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:05 smithi121 ceph-mon[44533]: osdmap e920: 12 total, 12 up, 10 in 2024-04-29T14:48:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:05 smithi194 ceph-mon[44089]: from='client.37460 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:48:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:05 smithi194 ceph-mon[44089]: osdmap e920: 12 total, 12 up, 10 in 2024-04-29T14:48:06.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:06 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:06.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:06 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:06.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:06 smithi040 ceph-mon[39581]: osdmap e921: 12 total, 12 up, 10 in 2024-04-29T14:48:06.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:06 smithi040 ceph-mon[39581]: pgmap v2341: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 11653/108240 objects misplaced (10.766%); 19 MiB/s, 297 objects/s recovering 2024-04-29T14:48:06.846 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:48:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:06 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:06 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:06 smithi121 ceph-mon[44533]: osdmap e921: 12 total, 12 up, 10 in 2024-04-29T14:48:06.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:06 smithi121 ceph-mon[44533]: pgmap v2341: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 11653/108240 objects misplaced (10.766%); 19 MiB/s, 297 objects/s recovering 2024-04-29T14:48:07.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:06 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:07.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:06 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:07.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:06 smithi194 ceph-mon[44089]: osdmap e921: 12 total, 12 up, 10 in 2024-04-29T14:48:07.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:06 smithi194 ceph-mon[44089]: pgmap v2341: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 11653/108240 objects misplaced (10.766%); 19 MiB/s, 297 objects/s recovering 2024-04-29T14:48:07.792 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-29T14:48:08.002 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:08.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:08.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:48:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:08 smithi194 ceph-mon[44089]: pgmap v2342: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 11611/108108 objects misplaced (10.740%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:48:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/452563168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:08 smithi194 ceph-mon[44089]: from='client.27718 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:08 smithi040 ceph-mon[39581]: pgmap v2342: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 11611/108108 objects misplaced (10.740%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:48:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/452563168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:08 smithi040 ceph-mon[39581]: from='client.27718 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:08 smithi121 ceph-mon[44533]: pgmap v2342: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 11611/108108 objects misplaced (10.740%); 16 MiB/s, 252 objects/s recovering 2024-04-29T14:48:08.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/452563168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:08.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:08 smithi121 ceph-mon[44533]: from='client.27718 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:09.481 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:09.483 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:09.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:09.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:09.618 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:09.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:09.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:09.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:09 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:09.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:09 smithi040 ceph-mon[39581]: from='client.27718 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:48:09.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:09 smithi040 ceph-mon[39581]: osdmap e922: 12 total, 12 up, 10 in 2024-04-29T14:48:09.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:09.775 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:09.793 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:09.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:09 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:09 smithi194 ceph-mon[44089]: from='client.27718 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:48:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:09 smithi194 ceph-mon[44089]: osdmap e922: 12 total, 12 up, 10 in 2024-04-29T14:48:09.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:09 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:09.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:09 smithi121 ceph-mon[44533]: from='client.27718 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:48:09.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:09 smithi121 ceph-mon[44533]: osdmap e922: 12 total, 12 up, 10 in 2024-04-29T14:48:09.897 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:09.924 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:09.945 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:09.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:10.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:10.076 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:10.094 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:10.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:10.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:10.226 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:10.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:10.351 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:10.384 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:10.534 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:10.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:10.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:10.661 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/980507297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:10.662 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:10 smithi040 ceph-mon[39581]: pgmap v2344: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 11474/107558 objects misplaced (10.668%); 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:48:10.671 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:48:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:48:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:48:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:48:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:48:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:48:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:48:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:48:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:48:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:48:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:48:10.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/980507297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:10.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:10 smithi194 ceph-mon[44089]: pgmap v2344: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 11474/107558 objects misplaced (10.668%); 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:48:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/980507297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:10.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:10 smithi121 ceph-mon[44533]: pgmap v2344: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 101 op/s; 11474/107558 objects misplaced (10.668%); 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:48:11.322 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:11.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:11.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:11.461 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:11.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:11.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:11.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:48:11.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:11.614 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:11.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:11.743 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:11.770 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:11.788 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:11.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:11.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.829 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:11.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:11.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:11.920 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:11.939 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:11.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:12.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:12.075 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:12.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:12.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:12.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:12.234 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:12.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:12.359 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:12.386 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:12.408 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:12.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:12.513 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:12.540 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:12.559 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:12.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:12.570 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:12 smithi121 ceph-mon[44533]: pgmap v2345: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 11376/106740 objects misplaced (10.658%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:48:12.570 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:12 smithi121 ceph-mon[44533]: osdmap e923: 12 total, 12 up, 10 in 2024-04-29T14:48:12.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:12.689 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:12.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:12.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:12 smithi194 ceph-mon[44089]: pgmap v2345: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 11376/106740 objects misplaced (10.658%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:48:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:12 smithi194 ceph-mon[44089]: osdmap e923: 12 total, 12 up, 10 in 2024-04-29T14:48:12.839 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:12.858 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:12.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:12.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:12 smithi040 ceph-mon[39581]: pgmap v2345: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 11376/106740 objects misplaced (10.658%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:48:12.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:12 smithi040 ceph-mon[39581]: osdmap e923: 12 total, 12 up, 10 in 2024-04-29T14:48:12.961 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:12.988 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:13.007 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:13.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:13.013 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:13.111 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:13.137 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:13.156 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:13.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:13.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:13.287 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:13.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:13.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:13.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:13.443 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:13.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:13.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:13.568 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:13.582 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.9 to 1 (10000) 2024-04-29T14:48:13.609 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:13.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:13 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:13 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/365197276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:48:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:13 smithi194 ceph-mon[44089]: from='client.37478 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:48:13.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:48:13.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:13 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:13.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:13 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/365197276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:48:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:13 smithi040 ceph-mon[39581]: from='client.37478 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:48:13.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:13 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:13.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:13 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:13.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/365197276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:48:13.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:13 smithi121 ceph-mon[44533]: from='client.37478 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T14:48:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:14 smithi040 ceph-mon[39581]: from='client.37478 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:48:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:14 smithi040 ceph-mon[39581]: osdmap e924: 12 total, 12 up, 10 in 2024-04-29T14:48:14.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:14 smithi040 ceph-mon[39581]: pgmap v2348: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 11376/106420 objects misplaced (10.690%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:48:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:14 smithi121 ceph-mon[44533]: from='client.37478 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:48:14.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:14 smithi121 ceph-mon[44533]: osdmap e924: 12 total, 12 up, 10 in 2024-04-29T14:48:14.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:14 smithi121 ceph-mon[44533]: pgmap v2348: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 11376/106420 objects misplaced (10.690%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:48:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:14 smithi194 ceph-mon[44089]: from='client.37478 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T14:48:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:14 smithi194 ceph-mon[44089]: osdmap e924: 12 total, 12 up, 10 in 2024-04-29T14:48:15.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:14 smithi194 ceph-mon[44089]: pgmap v2348: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 11376/106420 objects misplaced (10.690%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:48:15.638 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:15.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:15.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:15.804 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:15.961 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1387985175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:15.961 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:15 smithi040 ceph-mon[39581]: from='client.27733 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:15.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:15 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:48:16.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-29T14:48:16.256 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:16.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:16.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1387985175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:16.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:15 smithi194 ceph-mon[44089]: from='client.27733 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:16.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:15 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:16.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:16.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1387985175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:16.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:15 smithi121 ceph-mon[44533]: from='client.27733 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:16.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:15 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:16.387 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:16.406 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:16.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:16.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:17.066 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:16 smithi040 ceph-mon[39581]: from='client.27733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:48:17.066 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:16 smithi040 ceph-mon[39581]: osdmap e925: 12 total, 12 up, 10 in 2024-04-29T14:48:17.066 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:16 smithi040 ceph-mon[39581]: pgmap v2350: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 172 op/s; 11246/105490 objects misplaced (10.661%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:48:17.092 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:17.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:17.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:17.223 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:16 smithi194 ceph-mon[44089]: from='client.27733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:48:17.223 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:16 smithi194 ceph-mon[44089]: osdmap e925: 12 total, 12 up, 10 in 2024-04-29T14:48:17.223 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:16 smithi194 ceph-mon[44089]: pgmap v2350: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 172 op/s; 11246/105490 objects misplaced (10.661%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:48:17.225 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:17.244 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:17.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:17.347 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:17.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:17.372 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:48:17.372 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:48:17.372 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:48:17.372 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:48:17.372 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:48:17.373 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:16 smithi121 ceph-mon[44533]: from='client.27733 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:48:17.373 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:16 smithi121 ceph-mon[44533]: osdmap e925: 12 total, 12 up, 10 in 2024-04-29T14:48:17.373 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:16 smithi121 ceph-mon[44533]: pgmap v2350: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 172 op/s; 11246/105490 objects misplaced (10.661%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:48:17.375 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:17.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:17.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:17.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:48:17.527 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:17.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:48:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:17.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:48:17.652 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:48:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:17.699 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:48:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:17.702 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:48:17.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:17.834 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:17.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:18.094 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:18.122 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:18.142 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:18.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:18.246 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:18.272 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:18.290 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:18.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:18.395 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:48:18.421 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:48:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:18.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:48:18.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:18.578 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:18.832 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:48:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:19.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:19.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:19.564 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:19.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:19.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:19 smithi194 ceph-mon[44089]: pgmap v2351: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 11158/105200 objects misplaced (10.606%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:48:19.565 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:19 smithi194 ceph-mon[44089]: osdmap e926: 12 total, 12 up, 10 in 2024-04-29T14:48:19.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:19.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:19.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:19 smithi040 ceph-mon[39581]: pgmap v2351: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 11158/105200 objects misplaced (10.606%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:48:19.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:19 smithi040 ceph-mon[39581]: osdmap e926: 12 total, 12 up, 10 in 2024-04-29T14:48:19.567 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:19.584 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:19.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:19.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:19.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:19.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:19 smithi121 ceph-mon[44533]: pgmap v2351: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 11158/105200 objects misplaced (10.606%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:48:19.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:19 smithi121 ceph-mon[44533]: osdmap e926: 12 total, 12 up, 10 in 2024-04-29T14:48:19.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:19.723 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:19.797 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:19.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:19.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:19.933 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:20.305 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.10 to 1 (10000) 2024-04-29T14:48:20.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:20 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2058286802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:48:20.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:20 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2058286802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:48:20.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:20 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2058286802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:48:20.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:48:21.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:21.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:21.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:21.218 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:21.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:21.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:21.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:21 smithi194 ceph-mon[44089]: pgmap v2353: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 11032/104572 objects misplaced (10.550%); 2.1 MiB/s, 33 objects/s recovering 2024-04-29T14:48:21.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2058286802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:48:21.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:21 smithi194 ceph-mon[44089]: osdmap e927: 12 total, 12 up, 10 in 2024-04-29T14:48:21.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:21 smithi040 ceph-mon[39581]: pgmap v2353: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 11032/104572 objects misplaced (10.550%); 2.1 MiB/s, 33 objects/s recovering 2024-04-29T14:48:21.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2058286802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:48:21.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:21 smithi040 ceph-mon[39581]: osdmap e927: 12 total, 12 up, 10 in 2024-04-29T14:48:21.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:21 smithi121 ceph-mon[44533]: pgmap v2353: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 11032/104572 objects misplaced (10.550%); 2.1 MiB/s, 33 objects/s recovering 2024-04-29T14:48:21.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2058286802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:48:21.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:21 smithi121 ceph-mon[44533]: osdmap e927: 12 total, 12 up, 10 in 2024-04-29T14:48:21.829 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:21.833 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:21.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-29T14:48:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:21.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:21.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:22.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:22.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:22.305 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:22.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:22.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:22.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:22.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:22 smithi040 ceph-mon[39581]: osdmap e928: 12 total, 12 up, 10 in 2024-04-29T14:48:22.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:22.443 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:48:22.444 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:22.469 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:22.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:22.573 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:22.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:22.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:22.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:22 smithi194 ceph-mon[44089]: osdmap e928: 12 total, 12 up, 10 in 2024-04-29T14:48:22.605 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:22.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:22 smithi121 ceph-mon[44533]: osdmap e928: 12 total, 12 up, 10 in 2024-04-29T14:48:23.188 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:23.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:23.295 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:23.323 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:23.342 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:23.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:23.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:23 smithi040 ceph-mon[39581]: pgmap v2356: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 122 op/s; 11032/104014 objects misplaced (10.606%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:48:23.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:23 smithi040 ceph-mon[39581]: 7.c starting backfill to osd.10 from (0'0,837'9651] MIN to 926'11150 2024-04-29T14:48:23.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:23 smithi040 ceph-mon[39581]: osdmap e929: 12 total, 12 up, 10 in 2024-04-29T14:48:23.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:23 smithi040 ceph-mon[39581]: 7.e starting backfill to osd.10 from (700'7400,870'17484] MIN to 926'18358 2024-04-29T14:48:23.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/938128744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:23.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:23.447 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:23.474 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:23.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:23 smithi194 ceph-mon[44089]: pgmap v2356: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 122 op/s; 11032/104014 objects misplaced (10.606%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:48:23.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:23 smithi194 ceph-mon[44089]: 7.c starting backfill to osd.10 from (0'0,837'9651] MIN to 926'11150 2024-04-29T14:48:23.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:23 smithi194 ceph-mon[44089]: osdmap e929: 12 total, 12 up, 10 in 2024-04-29T14:48:23.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:23 smithi194 ceph-mon[44089]: 7.e starting backfill to osd.10 from (700'7400,870'17484] MIN to 926'18358 2024-04-29T14:48:23.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/938128744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:23.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:23.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:23.625 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:23 smithi121 ceph-mon[44533]: pgmap v2356: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 122 op/s; 11032/104014 objects misplaced (10.606%); 1.4 MiB/s, 22 objects/s recovering 2024-04-29T14:48:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:23 smithi121 ceph-mon[44533]: 7.c starting backfill to osd.10 from (0'0,837'9651] MIN to 926'11150 2024-04-29T14:48:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:23 smithi121 ceph-mon[44533]: osdmap e929: 12 total, 12 up, 10 in 2024-04-29T14:48:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:23 smithi121 ceph-mon[44533]: 7.e starting backfill to osd.10 from (700'7400,870'17484] MIN to 926'18358 2024-04-29T14:48:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/938128744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:23.645 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:23.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:23.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:25.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:25 smithi194 ceph-mon[44089]: pgmap v2358: 17 pgs: 2 active+remapped+backfilling, 2 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 16499/103748 objects misplaced (15.903%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:48:25.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:25 smithi040 ceph-mon[39581]: pgmap v2358: 17 pgs: 2 active+remapped+backfilling, 2 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 16499/103748 objects misplaced (15.903%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:48:25.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:25 smithi121 ceph-mon[44533]: pgmap v2358: 17 pgs: 2 active+remapped+backfilling, 2 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 16499/103748 objects misplaced (15.903%); 11 MiB/s, 178 objects/s recovering 2024-04-29T14:48:26.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:26 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 5907/103688 objects degraded (5.697%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:26.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:26 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 5907/103688 objects degraded (5.697%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:26.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:26 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 5907/103688 objects degraded (5.697%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:27.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:27 smithi040 ceph-mon[39581]: pgmap v2359: 17 pgs: 1 activating+degraded, 2 active+remapped+backfilling, 1 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 73 op/s; 5907/103688 objects degraded (5.697%); 11949/103688 objects misplaced (11.524%); 10 MiB/s, 165 objects/s recovering 2024-04-29T14:48:27.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:27 smithi121 ceph-mon[44533]: pgmap v2359: 17 pgs: 1 activating+degraded, 2 active+remapped+backfilling, 1 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 73 op/s; 5907/103688 objects degraded (5.697%); 11949/103688 objects misplaced (11.524%); 10 MiB/s, 165 objects/s recovering 2024-04-29T14:48:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:27 smithi194 ceph-mon[44089]: pgmap v2359: 17 pgs: 1 activating+degraded, 2 active+remapped+backfilling, 1 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 73 op/s; 5907/103688 objects degraded (5.697%); 11949/103688 objects misplaced (11.524%); 10 MiB/s, 165 objects/s recovering 2024-04-29T14:48:28.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:28 smithi040 ceph-mon[39581]: osdmap e930: 12 total, 12 up, 10 in 2024-04-29T14:48:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:28 smithi040 ceph-mon[39581]: osdmap e931: 12 total, 12 up, 10 in 2024-04-29T14:48:28.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:28 smithi121 ceph-mon[44533]: osdmap e930: 12 total, 12 up, 10 in 2024-04-29T14:48:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:28 smithi121 ceph-mon[44533]: osdmap e931: 12 total, 12 up, 10 in 2024-04-29T14:48:28.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:28 smithi194 ceph-mon[44089]: osdmap e930: 12 total, 12 up, 10 in 2024-04-29T14:48:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:28 smithi194 ceph-mon[44089]: osdmap e931: 12 total, 12 up, 10 in 2024-04-29T14:48:28.983 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:28.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:29.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:29.135 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:29.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:29.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:29.270 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:29.291 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:29.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:29.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:29.421 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:29.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:29.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:29.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:29 smithi194 ceph-mon[44089]: pgmap v2362: 17 pgs: 1 activating+degraded, 2 active+remapped+backfilling, 1 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5907/103688 objects degraded (5.697%); 11949/103688 objects misplaced (11.524%); 10 MiB/s, 165 objects/s recovering 2024-04-29T14:48:29.574 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:29.628 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:29.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:29 smithi040 ceph-mon[39581]: pgmap v2362: 17 pgs: 1 activating+degraded, 2 active+remapped+backfilling, 1 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5907/103688 objects degraded (5.697%); 11949/103688 objects misplaced (11.524%); 10 MiB/s, 165 objects/s recovering 2024-04-29T14:48:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:29.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:29.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:29 smithi121 ceph-mon[44533]: pgmap v2362: 17 pgs: 1 activating+degraded, 2 active+remapped+backfilling, 1 active+recovering+remapped, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5907/103688 objects degraded (5.697%); 11949/103688 objects misplaced (11.524%); 10 MiB/s, 165 objects/s recovering 2024-04-29T14:48:29.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:29.758 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:29.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:29.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:29.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:29.910 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:30.159 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:30.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:30.183 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:48:30.209 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:48:30.229 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:48:30.263 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:30.295 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:30.317 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:30.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:30.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2204952425' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:48:30.417 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:30 smithi040 ceph-mon[39581]: from='client.37502 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:48:30.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:48:30.422 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.6 to 1 (10000) 2024-04-29T14:48:30.447 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:30.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:48:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:30.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:48:30.569 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:30.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2204952425' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:48:30.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:30 smithi194 ceph-mon[44089]: from='client.37502 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:48:30.601 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:30.622 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:30.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:30.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2204952425' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:48:30.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:30 smithi121 ceph-mon[44533]: from='client.37502 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:48:30.726 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:48:30.756 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:30.776 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:48:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:30.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:48:30.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:30.909 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:30.930 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:30.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:31.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:31.314 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:31.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:31.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:31.448 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:31.493 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:48:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:31 smithi040 ceph-mon[39581]: pgmap v2363: 17 pgs: 1 activating+degraded+remapped, 1 activating+degraded, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 35 op/s; 11509/103440 objects degraded (11.126%); 1534/103440 objects misplaced (1.483%); 34 MiB/s, 536 objects/s recovering 2024-04-29T14:48:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:31 smithi040 ceph-mon[39581]: from='client.37502 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:48:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:31 smithi040 ceph-mon[39581]: osdmap e932: 12 total, 12 up, 10 in 2024-04-29T14:48:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3965629706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:48:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:31 smithi121 ceph-mon[44533]: pgmap v2363: 17 pgs: 1 activating+degraded+remapped, 1 activating+degraded, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 35 op/s; 11509/103440 objects degraded (11.126%); 1534/103440 objects misplaced (1.483%); 34 MiB/s, 536 objects/s recovering 2024-04-29T14:48:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:31 smithi121 ceph-mon[44533]: from='client.37502 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:48:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:31 smithi121 ceph-mon[44533]: osdmap e932: 12 total, 12 up, 10 in 2024-04-29T14:48:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3965629706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:48:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:31.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:31.803 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:31.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:31 smithi194 ceph-mon[44089]: pgmap v2363: 17 pgs: 1 activating+degraded+remapped, 1 activating+degraded, 15 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 35 op/s; 11509/103440 objects degraded (11.126%); 1534/103440 objects misplaced (1.483%); 34 MiB/s, 536 objects/s recovering 2024-04-29T14:48:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:31 smithi194 ceph-mon[44089]: from='client.37502 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:48:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:31 smithi194 ceph-mon[44089]: osdmap e932: 12 total, 12 up, 10 in 2024-04-29T14:48:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3965629706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:48:31.838 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:31.860 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:31.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:31.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:31.992 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:32.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:32.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:32.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:32.140 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:32.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:32.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:32.263 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:32.294 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:32.315 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:32.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:32.419 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:32.447 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:32.467 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:32.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:32.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:32.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:32 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:32.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3965629706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:48:32.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:32 smithi194 ceph-mon[44089]: osdmap e933: 12 total, 12 up, 10 in 2024-04-29T14:48:32.600 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:32 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3965629706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:48:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:32 smithi040 ceph-mon[39581]: osdmap e933: 12 total, 12 up, 10 in 2024-04-29T14:48:32.629 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:32.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:32.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:32 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:32.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3965629706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:48:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:32 smithi121 ceph-mon[44533]: osdmap e933: 12 total, 12 up, 10 in 2024-04-29T14:48:32.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:32.775 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:32.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-29T14:48:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:32.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:32.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:32.949 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:33.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:33.312 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:33.366 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:33.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:33.470 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:33.500 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:33.520 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:33.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:33 smithi040 ceph-mon[39581]: pgmap v2366: 17 pgs: 3 unknown, 1 activating+degraded+remapped, 1 activating+degraded, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11509/105640 objects degraded (10.895%); 1534/105640 objects misplaced (1.452%) 2024-04-29T14:48:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:33 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 11509/105640 objects degraded (10.895%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:48:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:33 smithi040 ceph-mon[39581]: 7.c starting backfill to osd.6 from (0'0,801'8892] MIN to 926'11150 2024-04-29T14:48:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:33 smithi040 ceph-mon[39581]: 7.7 starting backfill to osd.6 from (0'0,0'0] MAX to 926'11376 2024-04-29T14:48:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:33 smithi040 ceph-mon[39581]: osdmap e934: 12 total, 12 up, 10 in 2024-04-29T14:48:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:33 smithi040 ceph-mon[39581]: 7.d starting backfill to osd.6 from (0'0,801'8888] MIN to 926'11082 2024-04-29T14:48:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:33 smithi040 ceph-mon[39581]: osdmap e935: 12 total, 12 up, 10 in 2024-04-29T14:48:33.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:33 smithi121 ceph-mon[44533]: pgmap v2366: 17 pgs: 3 unknown, 1 activating+degraded+remapped, 1 activating+degraded, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11509/105640 objects degraded (10.895%); 1534/105640 objects misplaced (1.452%) 2024-04-29T14:48:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:33 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 11509/105640 objects degraded (10.895%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:48:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:33 smithi121 ceph-mon[44533]: 7.c starting backfill to osd.6 from (0'0,801'8892] MIN to 926'11150 2024-04-29T14:48:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:33 smithi121 ceph-mon[44533]: 7.7 starting backfill to osd.6 from (0'0,0'0] MAX to 926'11376 2024-04-29T14:48:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:33 smithi121 ceph-mon[44533]: osdmap e934: 12 total, 12 up, 10 in 2024-04-29T14:48:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:33 smithi121 ceph-mon[44533]: 7.d starting backfill to osd.6 from (0'0,801'8888] MIN to 926'11082 2024-04-29T14:48:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:33 smithi121 ceph-mon[44533]: osdmap e935: 12 total, 12 up, 10 in 2024-04-29T14:48:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:33.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:33.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:33.782 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:33.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:33 smithi194 ceph-mon[44089]: pgmap v2366: 17 pgs: 3 unknown, 1 activating+degraded+remapped, 1 activating+degraded, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11509/105640 objects degraded (10.895%); 1534/105640 objects misplaced (1.452%) 2024-04-29T14:48:33.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:33 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 11509/105640 objects degraded (10.895%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:48:33.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:33 smithi194 ceph-mon[44089]: 7.c starting backfill to osd.6 from (0'0,801'8892] MIN to 926'11150 2024-04-29T14:48:33.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:33 smithi194 ceph-mon[44089]: 7.7 starting backfill to osd.6 from (0'0,0'0] MAX to 926'11376 2024-04-29T14:48:33.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:33 smithi194 ceph-mon[44089]: osdmap e934: 12 total, 12 up, 10 in 2024-04-29T14:48:33.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:33 smithi194 ceph-mon[44089]: 7.d starting backfill to osd.6 from (0'0,801'8888] MIN to 926'11082 2024-04-29T14:48:33.813 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:33 smithi194 ceph-mon[44089]: osdmap e935: 12 total, 12 up, 10 in 2024-04-29T14:48:33.814 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:33.836 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:33.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:33.940 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:48:33.971 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:33.991 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:48:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:33.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:48:34.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:34.125 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:34.189 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:48:34.217 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:48:34.254 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:34.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:34.257 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:48:34.290 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:48:34.320 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:48:34.350 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:48:34.357 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:34.380 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:48:34.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:34.414 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:48:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:48:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:48:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:48:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:48:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:34.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.236 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:48:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: pgmap v2369: 17 pgs: 2 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 unknown, 1 activating+degraded+remapped, 12 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11509/107828 objects degraded (10.673%); 7524/107828 objects misplaced (6.978%) 2024-04-29T14:48:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: pgmap v2369: 17 pgs: 2 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 unknown, 1 activating+degraded+remapped, 12 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11509/107828 objects degraded (10.673%); 7524/107828 objects misplaced (6.978%) 2024-04-29T14:48:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: pgmap v2369: 17 pgs: 2 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 unknown, 1 activating+degraded+remapped, 12 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 11509/107828 objects degraded (10.673%); 7524/107828 objects misplaced (6.978%) 2024-04-29T14:48:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:36.192 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:36.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:36.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:48:36.386 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:36.408 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:48:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:36.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:48:36.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:48:36.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:36.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:36.553 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:36.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:36.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:36.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:36.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:37.015 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:48:37.015 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:48:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:37 smithi194 ceph-mon[44089]: 7.0 scrub starts 2024-04-29T14:48:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:37 smithi194 ceph-mon[44089]: 7.9 scrub starts 2024-04-29T14:48:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:37 smithi194 ceph-mon[44089]: pgmap v2370: 17 pgs: 3 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 3 op/s; 11509/103404 objects degraded (11.130%); 9438/103404 objects misplaced (9.127%); 20 MiB/s, 319 objects/s recovering 2024-04-29T14:48:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:37 smithi194 ceph-mon[44089]: osdmap e936: 12 total, 12 up, 10 in 2024-04-29T14:48:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:37 smithi194 ceph-mon[44089]: 7.0 scrub ok 2024-04-29T14:48:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:37 smithi194 ceph-mon[44089]: 7.9 scrub ok 2024-04-29T14:48:37.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:37 smithi040 ceph-mon[39581]: 7.0 scrub starts 2024-04-29T14:48:37.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:37 smithi040 ceph-mon[39581]: 7.9 scrub starts 2024-04-29T14:48:37.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:37 smithi040 ceph-mon[39581]: pgmap v2370: 17 pgs: 3 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 3 op/s; 11509/103404 objects degraded (11.130%); 9438/103404 objects misplaced (9.127%); 20 MiB/s, 319 objects/s recovering 2024-04-29T14:48:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:37 smithi040 ceph-mon[39581]: osdmap e936: 12 total, 12 up, 10 in 2024-04-29T14:48:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:37 smithi040 ceph-mon[39581]: 7.0 scrub ok 2024-04-29T14:48:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:37 smithi040 ceph-mon[39581]: 7.9 scrub ok 2024-04-29T14:48:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:37 smithi121 ceph-mon[44533]: 7.0 scrub starts 2024-04-29T14:48:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:37 smithi121 ceph-mon[44533]: 7.9 scrub starts 2024-04-29T14:48:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:37 smithi121 ceph-mon[44533]: pgmap v2370: 17 pgs: 3 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 3 op/s; 11509/103404 objects degraded (11.130%); 9438/103404 objects misplaced (9.127%); 20 MiB/s, 319 objects/s recovering 2024-04-29T14:48:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:37 smithi121 ceph-mon[44533]: osdmap e936: 12 total, 12 up, 10 in 2024-04-29T14:48:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:37 smithi121 ceph-mon[44533]: 7.0 scrub ok 2024-04-29T14:48:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:37 smithi121 ceph-mon[44533]: 7.9 scrub ok 2024-04-29T14:48:38.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:38 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 11509/103404 objects degraded (11.130%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:48:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:38 smithi194 ceph-mon[44089]: osdmap e937: 12 total, 12 up, 10 in 2024-04-29T14:48:38.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:38 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 11509/103404 objects degraded (11.130%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:48:38.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:38 smithi040 ceph-mon[39581]: osdmap e937: 12 total, 12 up, 10 in 2024-04-29T14:48:38.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:38 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 11509/103404 objects degraded (11.130%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:48:38.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:38 smithi121 ceph-mon[44533]: osdmap e937: 12 total, 12 up, 10 in 2024-04-29T14:48:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:39 smithi194 ceph-mon[44089]: pgmap v2373: 17 pgs: 3 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 78 MiB/s wr, 2.69k op/s; 11509/103404 objects degraded (11.130%); 9438/103404 objects misplaced (9.127%); 539 MiB/s, 9.95k objects/s recovering 2024-04-29T14:48:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:39 smithi194 ceph-mon[44089]: osdmap e938: 12 total, 12 up, 10 in 2024-04-29T14:48:39.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:39 smithi040 ceph-mon[39581]: pgmap v2373: 17 pgs: 3 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 78 MiB/s wr, 2.69k op/s; 11509/103404 objects degraded (11.130%); 9438/103404 objects misplaced (9.127%); 539 MiB/s, 9.95k objects/s recovering 2024-04-29T14:48:39.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:39 smithi040 ceph-mon[39581]: osdmap e938: 12 total, 12 up, 10 in 2024-04-29T14:48:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:39 smithi121 ceph-mon[44533]: pgmap v2373: 17 pgs: 3 active+remapped+backfilling, 1 active+recovery_wait+degraded, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 78 MiB/s wr, 2.69k op/s; 11509/103404 objects degraded (11.130%); 9438/103404 objects misplaced (9.127%); 539 MiB/s, 9.95k objects/s recovering 2024-04-29T14:48:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:39 smithi121 ceph-mon[44533]: osdmap e938: 12 total, 12 up, 10 in 2024-04-29T14:48:40.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:40 smithi040 ceph-mon[39581]: osdmap e939: 12 total, 12 up, 10 in 2024-04-29T14:48:40.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:40 smithi040 ceph-mon[39581]: pgmap v2376: 17 pgs: 1 active+recovering, 3 active+remapped+backfilling, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 4 op/s; 5602/103364 objects degraded (5.420%); 6814/103364 objects misplaced (6.592%); 75 MiB/s, 1.20k objects/s recovering 2024-04-29T14:48:40.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:40 smithi121 ceph-mon[44533]: osdmap e939: 12 total, 12 up, 10 in 2024-04-29T14:48:40.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:40 smithi121 ceph-mon[44533]: pgmap v2376: 17 pgs: 1 active+recovering, 3 active+remapped+backfilling, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 4 op/s; 5602/103364 objects degraded (5.420%); 6814/103364 objects misplaced (6.592%); 75 MiB/s, 1.20k objects/s recovering 2024-04-29T14:48:41.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:40 smithi194 ceph-mon[44089]: osdmap e939: 12 total, 12 up, 10 in 2024-04-29T14:48:41.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:40 smithi194 ceph-mon[44089]: pgmap v2376: 17 pgs: 1 active+recovering, 3 active+remapped+backfilling, 1 activating+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 4 op/s; 5602/103364 objects degraded (5.420%); 6814/103364 objects misplaced (6.592%); 75 MiB/s, 1.20k objects/s recovering 2024-04-29T14:48:41.245 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:41.296 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:41.512 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:48:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:41.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:48:41.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:41.646 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:41.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:41.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:41.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:41.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:41.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:41 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:48:41.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:41 smithi040 ceph-mon[39581]: osdmap e940: 12 total, 12 up, 10 in 2024-04-29T14:48:41.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:41 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:48:41.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:41 smithi040 ceph-mon[39581]: 7.3 scrub starts 2024-04-29T14:48:41.805 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:41.826 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:41.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:41 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:48:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:41 smithi121 ceph-mon[44533]: osdmap e940: 12 total, 12 up, 10 in 2024-04-29T14:48:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:41 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:48:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:41 smithi121 ceph-mon[44533]: 7.3 scrub starts 2024-04-29T14:48:41.931 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:41.958 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:41 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:48:41.958 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:41 smithi194 ceph-mon[44089]: osdmap e940: 12 total, 12 up, 10 in 2024-04-29T14:48:41.959 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:41 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:48:41.959 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:41 smithi194 ceph-mon[44089]: 7.3 scrub starts 2024-04-29T14:48:41.960 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:41.981 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:41.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:42.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:42.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:42.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:42.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:42.265 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:42.284 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:42.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:42.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:42.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:42 smithi040 ceph-mon[39581]: 7.3 scrub ok 2024-04-29T14:48:42.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:42 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:48:42.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:42 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:48:42.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:42 smithi040 ceph-mon[39581]: pgmap v2378: 17 pgs: 1 active+degraded+remapped, 2 peering, 1 active+recovering, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 4 op/s; 5602/103362 objects degraded (5.420%); 3580/103362 objects misplaced (3.464%); 77 MiB/s, 1.23k objects/s recovering 2024-04-29T14:48:42.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:42 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 5602/103364 objects degraded (5.420%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:42.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:42 smithi040 ceph-mon[39581]: osdmap e941: 12 total, 12 up, 10 in 2024-04-29T14:48:42.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:48:42.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:42 smithi121 ceph-mon[44533]: 7.3 scrub ok 2024-04-29T14:48:42.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:42 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:48:42.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:42 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:48:42.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:42 smithi121 ceph-mon[44533]: pgmap v2378: 17 pgs: 1 active+degraded+remapped, 2 peering, 1 active+recovering, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 4 op/s; 5602/103362 objects degraded (5.420%); 3580/103362 objects misplaced (3.464%); 77 MiB/s, 1.23k objects/s recovering 2024-04-29T14:48:42.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:42 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 5602/103364 objects degraded (5.420%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:42.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:42 smithi121 ceph-mon[44533]: osdmap e941: 12 total, 12 up, 10 in 2024-04-29T14:48:43.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:42 smithi194 ceph-mon[44089]: 7.3 scrub ok 2024-04-29T14:48:43.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:42 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:48:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:42 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:48:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:42 smithi194 ceph-mon[44089]: pgmap v2378: 17 pgs: 1 active+degraded+remapped, 2 peering, 1 active+recovering, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 4 op/s; 5602/103362 objects degraded (5.420%); 3580/103362 objects misplaced (3.464%); 77 MiB/s, 1.23k objects/s recovering 2024-04-29T14:48:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:42 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 5602/103364 objects degraded (5.420%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:42 smithi194 ceph-mon[44089]: osdmap e941: 12 total, 12 up, 10 in 2024-04-29T14:48:43.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:48:43.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:43 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:43 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:43 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:48:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:43 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:48:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:43 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:43 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:43 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:48:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:43 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:48:44.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:43 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:44.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:43 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:44.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:43 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:48:44.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:43 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:48:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:44.673 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:44.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:44.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:44 smithi040 ceph-mon[39581]: pgmap v2380: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 2 peering, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 14 op/s; 5602/103216 objects degraded (5.427%); 3426/103216 objects misplaced (3.319%); 14 MiB/s, 228 objects/s recovering 2024-04-29T14:48:44.781 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:44.807 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:44.826 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:44.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:44.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:44 smithi121 ceph-mon[44533]: pgmap v2380: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 2 peering, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 14 op/s; 5602/103216 objects degraded (5.427%); 3426/103216 objects misplaced (3.319%); 14 MiB/s, 228 objects/s recovering 2024-04-29T14:48:44.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:44.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:44.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:45.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:44 smithi194 ceph-mon[44089]: pgmap v2380: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 2 peering, 13 active+clean; 3.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 14 op/s; 5602/103216 objects degraded (5.427%); 3426/103216 objects misplaced (3.319%); 14 MiB/s, 228 objects/s recovering 2024-04-29T14:48:45.080 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:45.106 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:45.124 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:45.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:45.228 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:48:45.254 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:45.273 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:48:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:45.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:48:45.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:45.404 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:45.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:45.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:45.567 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:45.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:45 smithi040 ceph-mon[39581]: 7.d scrub starts 2024-04-29T14:48:45.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:45 smithi040 ceph-mon[39581]: 7.d scrub ok 2024-04-29T14:48:45.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/747340187' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:48:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:45 smithi040 ceph-mon[39581]: from='client.37514 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:48:45.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:45 smithi121 ceph-mon[44533]: 7.d scrub starts 2024-04-29T14:48:45.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:45 smithi121 ceph-mon[44533]: 7.d scrub ok 2024-04-29T14:48:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/747340187' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:48:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:45 smithi121 ceph-mon[44533]: from='client.37514 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:48:46.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:45 smithi194 ceph-mon[44089]: 7.d scrub starts 2024-04-29T14:48:46.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:45 smithi194 ceph-mon[44089]: 7.d scrub ok 2024-04-29T14:48:46.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/747340187' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:48:46.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:45 smithi194 ceph-mon[44089]: from='client.37514 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:48:46.168 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:46.497 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-04-29T14:48:46.623 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:48:46.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:46 smithi040 ceph-mon[39581]: from='client.37514 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:48:46.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:46 smithi040 ceph-mon[39581]: osdmap e942: 12 total, 12 up, 10 in 2024-04-29T14:48:46.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:46 smithi040 ceph-mon[39581]: pgmap v2382: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 43 op/s; 5602/102840 objects degraded (5.447%); 3426/102840 objects misplaced (3.331%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:48:46.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2891352480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:46 smithi121 ceph-mon[44533]: from='client.37514 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:48:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:46 smithi121 ceph-mon[44533]: osdmap e942: 12 total, 12 up, 10 in 2024-04-29T14:48:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:46 smithi121 ceph-mon[44533]: pgmap v2382: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 43 op/s; 5602/102840 objects degraded (5.447%); 3426/102840 objects misplaced (3.331%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:48:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2891352480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:47.023 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:48:47.024 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:48:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:46 smithi194 ceph-mon[44089]: from='client.37514 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:48:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:46 smithi194 ceph-mon[44089]: osdmap e942: 12 total, 12 up, 10 in 2024-04-29T14:48:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:46 smithi194 ceph-mon[44089]: pgmap v2382: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 43 op/s; 5602/102840 objects degraded (5.447%); 3426/102840 objects misplaced (3.331%); 12 MiB/s, 192 objects/s recovering 2024-04-29T14:48:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2891352480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:48:47.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-29T14:48:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:47.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:47.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:47 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:48:47.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2891352480' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:48:47.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:47 smithi040 ceph-mon[39581]: osdmap e943: 12 total, 12 up, 10 in 2024-04-29T14:48:47.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:47 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 5602/102840 objects degraded (5.447%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:47.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:47 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:48:47.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2891352480' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:48:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:47 smithi121 ceph-mon[44533]: osdmap e943: 12 total, 12 up, 10 in 2024-04-29T14:48:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:47 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 5602/102840 objects degraded (5.447%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:47.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:48:47.953 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:48:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:47.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:48:48.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:47 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:48:48.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2891352480' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:48:48.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:47 smithi194 ceph-mon[44089]: osdmap e943: 12 total, 12 up, 10 in 2024-04-29T14:48:48.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:47 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 5602/102840 objects degraded (5.447%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:48.075 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:48.108 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:48.132 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:48.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:48.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:48.265 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:48.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:48 smithi040 ceph-mon[39581]: pgmap v2384: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 43 op/s; 5602/102840 objects degraded (5.447%); 3426/102840 objects misplaced (3.331%); 1.6 MiB/s, 25 objects/s recovering 2024-04-29T14:48:48.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:48 smithi121 ceph-mon[44533]: pgmap v2384: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 43 op/s; 5602/102840 objects degraded (5.447%); 3426/102840 objects misplaced (3.331%); 1.6 MiB/s, 25 objects/s recovering 2024-04-29T14:48:49.014 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:49.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:49.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:48:49.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:48 smithi194 ceph-mon[44089]: pgmap v2384: 17 pgs: 1 active+recovering, 1 active+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 43 op/s; 5602/102840 objects degraded (5.447%); 3426/102840 objects misplaced (3.331%); 1.6 MiB/s, 25 objects/s recovering 2024-04-29T14:48:49.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:49.181 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:49.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:49.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:49.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:49.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:49.723 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:49.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:49 smithi040 ceph-mon[39581]: 7.c deep-scrub starts 2024-04-29T14:48:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:49 smithi121 ceph-mon[44533]: 7.c deep-scrub starts 2024-04-29T14:48:50.001 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:50.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:50.031 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:49 smithi194 ceph-mon[44089]: 7.c deep-scrub starts 2024-04-29T14:48:50.076 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:50.106 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:50.138 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:50.161 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:50.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:50.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:50.297 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:50.352 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:50.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:50.459 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:50.491 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:50.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:48:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:50 smithi121 ceph-mon[44533]: pgmap v2385: 17 pgs: 1 active+recovering, 1 active+degraded, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 54 op/s; 5543/102522 objects degraded (5.407%); 1710/102522 objects misplaced (1.668%); 2364/51261 objects unfound (4.612%); 2.9 MiB/s, 47 objects/s recovering 2024-04-29T14:48:51.021 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:48:51.042 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:48:51.062 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:48:51.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:50 smithi194 ceph-mon[44089]: pgmap v2385: 17 pgs: 1 active+recovering, 1 active+degraded, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 54 op/s; 5543/102522 objects degraded (5.407%); 1710/102522 objects misplaced (1.668%); 2364/51261 objects unfound (4.612%); 2.9 MiB/s, 47 objects/s recovering 2024-04-29T14:48:51.090 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:48:51.117 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:48:51.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:50 smithi040 ceph-mon[39581]: pgmap v2385: 17 pgs: 1 active+recovering, 1 active+degraded, 15 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 54 op/s; 5543/102522 objects degraded (5.407%); 1710/102522 objects misplaced (1.668%); 2364/51261 objects unfound (4.612%); 2.9 MiB/s, 47 objects/s recovering 2024-04-29T14:48:51.138 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:48:51.160 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:48:51.181 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:48:51.201 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:48:51.223 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:48:51.245 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:48:51.266 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:48:51.267 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:48:51.267 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:48:51.268 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: Health check failed: 2364/51261 objects unfound (4.612%) (OBJECT_UNFOUND) 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2337834215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: Health check failed: 2364/51261 objects unfound (4.612%) (OBJECT_UNFOUND) 2024-04-29T14:48:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2337834215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: Health check failed: 2364/51261 objects unfound (4.612%) (OBJECT_UNFOUND) 2024-04-29T14:48:52.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2337834215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:48:52.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:52.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:52.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:48:52.411 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:52.432 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:48:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:52.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:48:52.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:52.567 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:48:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: pgmap v2386: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 1 active+degraded, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 51 op/s; 5543/102396 objects degraded (5.413%); 1710/102396 objects misplaced (1.670%); 2364/51198 objects unfound (4.617%); 1.6 MiB/s, 26 objects/s recovering 2024-04-29T14:48:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:48:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:48:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:48:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:52 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 5543/102522 objects degraded (5.407%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: pgmap v2386: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 1 active+degraded, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 51 op/s; 5543/102396 objects degraded (5.413%); 1710/102396 objects misplaced (1.670%); 2364/51198 objects unfound (4.617%); 1.6 MiB/s, 26 objects/s recovering 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:48:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:52 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 5543/102522 objects degraded (5.407%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: pgmap v2386: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 1 active+degraded, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 51 op/s; 5543/102396 objects degraded (5.413%); 1710/102396 objects misplaced (1.670%); 2364/51198 objects unfound (4.617%); 1.6 MiB/s, 26 objects/s recovering 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:48:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:52 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 5543/102522 objects degraded (5.407%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:53.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:53.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:53.551 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:53.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:53.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:53.674 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:48:53.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:53 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:48:53.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:48:53.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:48:53.707 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:53.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:53 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:48:53.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:48:53.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:48:54.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:53 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:48:54.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:48:54.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:48:54.814 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:54.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:54 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:48:54.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:54 smithi040 ceph-mon[39581]: osdmap e944: 12 total, 12 up, 10 in 2024-04-29T14:48:54.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:54 smithi040 ceph-mon[39581]: pgmap v2388: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 1 active+degraded, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5543/102396 objects degraded (5.413%); 1516/102396 objects misplaced (1.481%); 2364/51198 objects unfound (4.617%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:48:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:54 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:48:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:54 smithi194 ceph-mon[44089]: osdmap e944: 12 total, 12 up, 10 in 2024-04-29T14:48:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:54 smithi194 ceph-mon[44089]: pgmap v2388: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 1 active+degraded, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5543/102396 objects degraded (5.413%); 1516/102396 objects misplaced (1.481%); 2364/51198 objects unfound (4.617%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:48:55.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:54 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:48:55.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:54 smithi121 ceph-mon[44533]: osdmap e944: 12 total, 12 up, 10 in 2024-04-29T14:48:55.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:54 smithi121 ceph-mon[44533]: pgmap v2388: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 1 active+degraded, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5543/102396 objects degraded (5.413%); 1516/102396 objects misplaced (1.481%); 2364/51198 objects unfound (4.617%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:48:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.5 to 1 (10000) 2024-04-29T14:48:55.659 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:48:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:55.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:48:55.767 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:48:55.798 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:55.819 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:48:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:55.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:48:55.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:48:55.950 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1794311418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:48:55.952 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:55.972 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:48:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:55.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:48:56.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:48:56.082 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1794311418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:48:56.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:48:56.102 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1794311418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:48:56.104 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:56.124 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:48:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:56.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:48:56.230 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:48:56.256 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:56.275 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:48:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:56.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:48:56.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:56.411 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:56.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-29T14:48:56.658 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1794311418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:48:56.658 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:56 smithi040 ceph-mon[39581]: osdmap e945: 12 total, 12 up, 10 in 2024-04-29T14:48:56.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:56 smithi040 ceph-mon[39581]: pgmap v2390: 17 pgs: 1 active+recovery_wait+degraded, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5543/102394 objects degraded (5.413%); 1516/102394 objects misplaced (1.481%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:48:57.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1794311418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:48:57.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:56 smithi194 ceph-mon[44089]: osdmap e945: 12 total, 12 up, 10 in 2024-04-29T14:48:57.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:56 smithi194 ceph-mon[44089]: pgmap v2390: 17 pgs: 1 active+recovery_wait+degraded, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5543/102394 objects degraded (5.413%); 1516/102394 objects misplaced (1.481%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:48:57.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1794311418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:48:57.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:56 smithi121 ceph-mon[44533]: osdmap e945: 12 total, 12 up, 10 in 2024-04-29T14:48:57.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:56 smithi121 ceph-mon[44533]: pgmap v2390: 17 pgs: 1 active+recovery_wait+degraded, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 27 op/s; 5543/102394 objects degraded (5.413%); 1516/102394 objects misplaced (1.481%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:48:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:57.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:57 smithi040 ceph-mon[39581]: Health check cleared: OBJECT_UNFOUND (was: 2364/51198 objects unfound (4.617%)) 2024-04-29T14:48:57.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:57 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 5543/102394 objects degraded (5.413%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:58.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:57 smithi194 ceph-mon[44089]: Health check cleared: OBJECT_UNFOUND (was: 2364/51198 objects unfound (4.617%)) 2024-04-29T14:48:58.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:57 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 5543/102394 objects degraded (5.413%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:58.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:57 smithi121 ceph-mon[44533]: Health check cleared: OBJECT_UNFOUND (was: 2364/51198 objects unfound (4.617%)) 2024-04-29T14:48:58.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:57 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 5543/102394 objects degraded (5.413%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:48:58.665 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:48:58.985 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:58 smithi040 ceph-mon[39581]: pgmap v2391: 17 pgs: 1 active+recovery_wait+degraded, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 7 op/s; 5543/102394 objects degraded (5.413%); 1516/102394 objects misplaced (1.481%); 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:48:58.986 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3681510332' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:58.986 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:58 smithi040 ceph-mon[39581]: from='client.27787 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:48:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:59.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:48:59.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:58 smithi194 ceph-mon[44089]: pgmap v2391: 17 pgs: 1 active+recovery_wait+degraded, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 7 op/s; 5543/102394 objects degraded (5.413%); 1516/102394 objects misplaced (1.481%); 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:48:59.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3681510332' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:59.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:58 smithi194 ceph-mon[44089]: from='client.27787 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:59.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:59.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:58 smithi121 ceph-mon[44533]: pgmap v2391: 17 pgs: 1 active+recovery_wait+degraded, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 7 op/s; 5543/102394 objects degraded (5.413%); 1516/102394 objects misplaced (1.481%); 1.5 MiB/s, 24 objects/s recovering 2024-04-29T14:48:59.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3681510332' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:59.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:58 smithi121 ceph-mon[44533]: from='client.27787 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:48:59.140 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:59.488 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:48:59.593 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:59.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:59.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:48:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:48:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:48:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:48:59.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:48:59.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:59 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:48:59.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:59 smithi040 ceph-mon[39581]: from='client.27787 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:48:59.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:59 smithi040 ceph-mon[39581]: osdmap e946: 12 total, 12 up, 10 in 2024-04-29T14:48:59.848 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:48:59 smithi040 ceph-mon[39581]: 7.c deep-scrub ok 2024-04-29T14:48:59.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:48:59.952 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:48:59.953 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:48:59.953 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:48:59.953 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:48:59.953 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:48:59.959 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:00.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:59 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:00.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:59 smithi194 ceph-mon[44089]: from='client.27787 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:49:00.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:59 smithi194 ceph-mon[44089]: osdmap e946: 12 total, 12 up, 10 in 2024-04-29T14:49:00.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:48:59 smithi194 ceph-mon[44089]: 7.c deep-scrub ok 2024-04-29T14:49:00.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:59 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:00.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:59 smithi121 ceph-mon[44533]: from='client.27787 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:49:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:59 smithi121 ceph-mon[44533]: osdmap e946: 12 total, 12 up, 10 in 2024-04-29T14:49:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:48:59 smithi121 ceph-mon[44533]: 7.c deep-scrub ok 2024-04-29T14:49:01.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:00 smithi194 ceph-mon[44089]: pgmap v2393: 17 pgs: 1 active+recovery_wait+degraded, 1 active+recovering, 1 active+clean+scrubbing+deep, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 5543/101964 objects degraded (5.436%); 1337/101964 objects misplaced (1.311%); 3.5 MiB/s, 55 objects/s recovering 2024-04-29T14:49:01.084 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:00 smithi040 ceph-mon[39581]: pgmap v2393: 17 pgs: 1 active+recovery_wait+degraded, 1 active+recovering, 1 active+clean+scrubbing+deep, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 5543/101964 objects degraded (5.436%); 1337/101964 objects misplaced (1.311%); 3.5 MiB/s, 55 objects/s recovering 2024-04-29T14:49:01.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:00 smithi121 ceph-mon[44533]: pgmap v2393: 17 pgs: 1 active+recovery_wait+degraded, 1 active+recovering, 1 active+clean+scrubbing+deep, 14 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 32 op/s; 5543/101964 objects degraded (5.436%); 1337/101964 objects misplaced (1.311%); 3.5 MiB/s, 55 objects/s recovering 2024-04-29T14:49:01.440 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:01.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:01.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:01.572 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:01.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:01.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:01.671 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.8 to 1 (10000) 2024-04-29T14:49:01.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:01.728 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:01.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:01.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:01.852 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:01.878 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:01.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3579211629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:49:01.898 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:01.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:02.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:02.032 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:02.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:02.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:02.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3579211629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:49:02.093 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3579211629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:49:02.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3579211629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:49:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:02 smithi040 ceph-mon[39581]: osdmap e947: 12 total, 12 up, 10 in 2024-04-29T14:49:02.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:02 smithi040 ceph-mon[39581]: pgmap v2395: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 6782/100962 objects misplaced (6.717%); 2.1 MiB/s, 34 objects/s recovering 2024-04-29T14:49:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:02 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 5543/101964 objects degraded (5.436%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:49:02.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:49:02.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:49:03.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3579211629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:49:03.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:02 smithi194 ceph-mon[44089]: osdmap e947: 12 total, 12 up, 10 in 2024-04-29T14:49:03.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:02 smithi194 ceph-mon[44089]: pgmap v2395: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 6782/100962 objects misplaced (6.717%); 2.1 MiB/s, 34 objects/s recovering 2024-04-29T14:49:03.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:02 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 5543/101964 objects degraded (5.436%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:49:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3579211629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:49:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:02 smithi121 ceph-mon[44533]: osdmap e947: 12 total, 12 up, 10 in 2024-04-29T14:49:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:02 smithi121 ceph-mon[44533]: pgmap v2395: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 6782/100962 objects misplaced (6.717%); 2.1 MiB/s, 34 objects/s recovering 2024-04-29T14:49:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:02 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 5543/101964 objects degraded (5.436%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:49:03.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-29T14:49:03.440 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:03.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:03.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:03.570 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:03.589 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:03.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:03.693 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:03.720 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:03.740 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:03.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:03.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:03.872 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:03.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:03.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:03.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:03 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5543/101964 objects degraded (5.436%), 1 pg degraded) 2024-04-29T14:49:03.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:03 smithi121 ceph-mon[44533]: osdmap e948: 12 total, 12 up, 10 in 2024-04-29T14:49:03.891 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:03.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:03.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:04.014 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:04.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:04.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:04.045 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:03 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5543/101964 objects degraded (5.436%), 1 pg degraded) 2024-04-29T14:49:04.045 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:03 smithi040 ceph-mon[39581]: osdmap e948: 12 total, 12 up, 10 in 2024-04-29T14:49:04.055 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:04.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:04.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:03 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5543/101964 objects degraded (5.436%), 1 pg degraded) 2024-04-29T14:49:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:03 smithi194 ceph-mon[44089]: osdmap e948: 12 total, 12 up, 10 in 2024-04-29T14:49:04.077 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:04.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:04.090 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:49:04.121 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:49:04.144 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:49:04.177 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:49:04.186 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:04.198 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:49:04.212 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:04.230 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:04.231 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:49:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:04.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:04.255 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:49:04.289 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:49:04.313 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:49:04.334 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:04.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:49:04.361 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:04.376 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:49:04.381 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:04.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:04.402 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:49:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:49:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:49:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:49:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:49:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:49:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:49:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:49:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:49:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:49:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:49:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:49:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:49:04.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:04.512 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:04.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:04.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:04.636 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:04.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:04.682 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:04.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:04.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:04.816 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:04.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1551331692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:04 smithi121 ceph-mon[44533]: pgmap v2397: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 167 op/s; 6603/100382 objects misplaced (6.578%); 4.2 MiB/s, 67 objects/s recovering 2024-04-29T14:49:04.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1551331692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:04 smithi040 ceph-mon[39581]: pgmap v2397: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 167 op/s; 6603/100382 objects misplaced (6.578%); 4.2 MiB/s, 67 objects/s recovering 2024-04-29T14:49:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:05.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1551331692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:05.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:04 smithi194 ceph-mon[44089]: pgmap v2397: 17 pgs: 2 active+recovering, 15 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 167 op/s; 6603/100382 objects misplaced (6.578%); 4.2 MiB/s, 67 objects/s recovering 2024-04-29T14:49:05.521 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:05.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:05.624 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:05.655 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:05.694 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:05.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:05.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:05.825 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.847 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:05.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:05.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:05.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.976 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.977 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:05.978 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:06.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:06.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:06.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:06.135 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:06.159 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:06.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:06.263 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:06.290 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:07.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:06 smithi194 ceph-mon[44089]: pgmap v2398: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 192 op/s; 6498/99316 objects misplaced (6.543%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:49:07.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:06 smithi040 ceph-mon[39581]: pgmap v2398: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 192 op/s; 6498/99316 objects misplaced (6.543%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:49:07.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:06 smithi121 ceph-mon[44533]: pgmap v2398: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 192 op/s; 6498/99316 objects misplaced (6.543%); 3.7 MiB/s, 59 objects/s recovering 2024-04-29T14:49:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:08.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:08.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:08.277 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:08.627 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:08.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:08.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:08.731 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:08.757 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:08.776 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:08.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:08 smithi040 ceph-mon[39581]: osdmap e949: 12 total, 12 up, 10 in 2024-04-29T14:49:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:08 smithi040 ceph-mon[39581]: pgmap v2400: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 6498/99316 objects misplaced (6.543%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:49:08.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:08 smithi121 ceph-mon[44533]: osdmap e949: 12 total, 12 up, 10 in 2024-04-29T14:49:08.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:08 smithi121 ceph-mon[44533]: pgmap v2400: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 6498/99316 objects misplaced (6.543%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:49:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:09.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:08 smithi194 ceph-mon[44089]: osdmap e949: 12 total, 12 up, 10 in 2024-04-29T14:49:09.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:08 smithi194 ceph-mon[44089]: pgmap v2400: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 6498/99316 objects misplaced (6.543%); 2.2 MiB/s, 34 objects/s recovering 2024-04-29T14:49:09.263 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:09.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:09.368 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:09.395 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:09.414 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:09.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:09.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:09.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:09.545 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:09.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:09.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:09.703 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:09.910 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:09.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:49:09.953 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/609336404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:09.953 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:09 smithi040 ceph-mon[39581]: from='client.27805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:10.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:10.047 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:10.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/609336404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:10.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:09 smithi194 ceph-mon[44089]: from='client.27805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:10.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/609336404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:10.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:09 smithi121 ceph-mon[44533]: from='client.27805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:10.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:10.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:10.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:10.528 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:10.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:10.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:10.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:10.680 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:10.699 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:10.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:10.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:10.836 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:10.861 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:10.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:10.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:11.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:49:11.075 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:11.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:11.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:10 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:10 smithi121 ceph-mon[44533]: from='client.27805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:10 smithi121 ceph-mon[44533]: osdmap e950: 12 total, 12 up, 10 in 2024-04-29T14:49:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:10 smithi121 ceph-mon[44533]: pgmap v2402: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 146 op/s; 6290/98734 objects misplaced (6.371%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:49:11.181 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:11.205 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:10 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:11.205 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:10 smithi194 ceph-mon[44089]: from='client.27805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:11.205 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:10 smithi194 ceph-mon[44089]: osdmap e950: 12 total, 12 up, 10 in 2024-04-29T14:49:11.205 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:11.205 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:11.205 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:10 smithi194 ceph-mon[44089]: pgmap v2402: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 146 op/s; 6290/98734 objects misplaced (6.371%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:49:11.207 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:11.225 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:11.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:11.267 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:10 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:11.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:10 smithi040 ceph-mon[39581]: from='client.27805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:11.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:10 smithi040 ceph-mon[39581]: osdmap e950: 12 total, 12 up, 10 in 2024-04-29T14:49:11.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:11.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:11.268 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:10 smithi040 ceph-mon[39581]: pgmap v2402: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 146 op/s; 6290/98734 objects misplaced (6.371%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:49:11.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:11.361 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:11.413 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:49:11.413 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:49:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:11.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:11.930 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:11.936 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-04-29T14:49:11.957 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:11.976 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:11.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:12.080 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:12.106 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:12.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:11 smithi040 ceph-mon[39581]: osdmap e951: 12 total, 12 up, 10 in 2024-04-29T14:49:12.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:11 smithi040 ceph-mon[39581]: from='client.37562 v1:172.21.15.40:0/2263175264' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:49:12.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:11 smithi040 ceph-mon[39581]: from='client.37562 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:49:12.125 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:12.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:12.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:12.259 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:12.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:11 smithi194 ceph-mon[44089]: osdmap e951: 12 total, 12 up, 10 in 2024-04-29T14:49:12.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:11 smithi194 ceph-mon[44089]: from='client.37562 v1:172.21.15.40:0/2263175264' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:49:12.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:11 smithi194 ceph-mon[44089]: from='client.37562 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:49:12.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:11 smithi121 ceph-mon[44533]: osdmap e951: 12 total, 12 up, 10 in 2024-04-29T14:49:12.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:11 smithi121 ceph-mon[44533]: from='client.37562 v1:172.21.15.40:0/2263175264' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:49:12.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:11 smithi121 ceph-mon[44533]: from='client.37562 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:49:12.387 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:12.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:12.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:49:12.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:12.522 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:12.650 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:12.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:12.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:12.783 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:12.801 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:12.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:12.906 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:12.935 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:12.993 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:12.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:13.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:13.122 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:13.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:13.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:12 smithi121 ceph-mon[44533]: from='client.37562 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T14:49:13.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:12 smithi121 ceph-mon[44533]: osdmap e952: 12 total, 12 up, 10 in 2024-04-29T14:49:13.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:12 smithi121 ceph-mon[44533]: pgmap v2405: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 170 op/s; 6177/97706 objects misplaced (6.322%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:49:13.123 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:12 smithi121 ceph-mon[44533]: osdmap e953: 12 total, 12 up, 10 in 2024-04-29T14:49:13.124 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:13.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:13.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:13.209 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:13.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:13.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:12 smithi040 ceph-mon[39581]: from='client.37562 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T14:49:13.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:12 smithi040 ceph-mon[39581]: osdmap e952: 12 total, 12 up, 10 in 2024-04-29T14:49:13.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:12 smithi040 ceph-mon[39581]: pgmap v2405: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 170 op/s; 6177/97706 objects misplaced (6.322%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:49:13.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:12 smithi040 ceph-mon[39581]: osdmap e953: 12 total, 12 up, 10 in 2024-04-29T14:49:13.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:13.280 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:13.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:13.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:13.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:13.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:13.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:12 smithi194 ceph-mon[44089]: from='client.37562 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T14:49:13.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:12 smithi194 ceph-mon[44089]: osdmap e952: 12 total, 12 up, 10 in 2024-04-29T14:49:13.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:12 smithi194 ceph-mon[44089]: pgmap v2405: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 170 op/s; 6177/97706 objects misplaced (6.322%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:49:13.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:12 smithi194 ceph-mon[44089]: osdmap e953: 12 total, 12 up, 10 in 2024-04-29T14:49:13.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:13.433 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:13.530 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:13.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:13.550 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-29T14:49:13.551 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-29T14:49:13.551 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-29T14:49:13.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T14:49:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:13.637 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:13.665 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:13.683 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:13.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:13.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:13.813 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:13.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:13.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:13.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:13.963 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:13.992 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:13.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:14.099 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:14.126 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:14.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:13 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:14.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:13 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:14.146 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:14.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:14.211 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:13 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:14.211 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:13 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:14.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:14.275 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:14.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:14.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:14.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:13 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:14.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:13 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:14.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:14.431 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:15.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:49:15.149 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:15.264 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:15.264 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:15.264 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:14 smithi040 ceph-mon[39581]: pgmap v2407: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 166 op/s; 5929/97274 objects misplaced (6.095%); 4.4 MiB/s, 70 objects/s recovering 2024-04-29T14:49:15.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:15.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:15.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:14 smithi194 ceph-mon[44089]: pgmap v2407: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 166 op/s; 5929/97274 objects misplaced (6.095%); 4.4 MiB/s, 70 objects/s recovering 2024-04-29T14:49:15.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:15.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:15.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:14 smithi121 ceph-mon[44533]: pgmap v2407: 17 pgs: 2 active+recovering, 15 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 166 op/s; 5929/97274 objects misplaced (6.095%); 4.4 MiB/s, 70 objects/s recovering 2024-04-29T14:49:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:15.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:15.788 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:15.815 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:15.836 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:15.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:15.940 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:15.967 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:15.987 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:15.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:16.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:15 smithi194 ceph-mon[44089]: osdmap e954: 12 total, 12 up, 10 in 2024-04-29T14:49:16.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:16.117 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:16.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:16.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:16.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:16.271 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:16.273 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:15 smithi121 ceph-mon[44533]: osdmap e954: 12 total, 12 up, 10 in 2024-04-29T14:49:16.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:16.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:16.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:15 smithi040 ceph-mon[39581]: osdmap e954: 12 total, 12 up, 10 in 2024-04-29T14:49:16.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:16.421 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:16.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:16.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:16.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:16.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:16.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:16.730 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:17.169 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:49:17.190 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:49:17.212 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:49:17.235 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:49:17.266 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:49:17.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:49:17.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:16 smithi194 ceph-mon[44089]: pgmap v2409: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 134 op/s; 5820/96472 objects misplaced (6.033%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:49:17.327 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:49:17.353 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:49:17.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:16 smithi040 ceph-mon[39581]: pgmap v2409: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 134 op/s; 5820/96472 objects misplaced (6.033%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:49:17.382 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:17.383 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:49:17.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:16 smithi121 ceph-mon[44533]: pgmap v2409: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 134 op/s; 5820/96472 objects misplaced (6.033%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:49:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:17.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:49:17.414 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:49:17.415 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:49:17.415 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:49:17.415 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:49:17.434 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:49:17.455 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:49:17.456 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:49:17.456 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:49:17.457 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:49:17.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:17.512 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:17.533 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:17.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:17.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:17.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:17.664 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:17.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:17.788 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:17.815 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:17.834 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:17.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:17.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:17.970 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:18.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/911556351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:18.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:17 smithi194 ceph-mon[44089]: osdmap e955: 12 total, 12 up, 10 in 2024-04-29T14:49:18.326 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:18.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:18.332 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/911556351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:18.332 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.333 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.333 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.333 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:17 smithi040 ceph-mon[39581]: osdmap e955: 12 total, 12 up, 10 in 2024-04-29T14:49:18.384 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/911556351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:18.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:18.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:17 smithi121 ceph-mon[44533]: osdmap e955: 12 total, 12 up, 10 in 2024-04-29T14:49:18.433 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:18.459 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:18.478 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:18.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:18.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:18.609 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:18.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:18.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:18.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:18.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:18.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:18.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:18.914 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:18.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:18.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:18.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:49:19.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:19.084 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:19.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:19.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:19.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:18 smithi040 ceph-mon[39581]: pgmap v2411: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 5820/96472 objects misplaced (6.033%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:49:19.222 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:19.301 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:19.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:19.319 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:19.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:18 smithi194 ceph-mon[44089]: pgmap v2411: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 5820/96472 objects misplaced (6.033%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:18 smithi121 ceph-mon[44533]: pgmap v2411: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 102 op/s; 5820/96472 objects misplaced (6.033%); 3.1 MiB/s, 50 objects/s recovering 2024-04-29T14:49:19.406 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:19.432 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:19.450 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:19.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:19.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:19.580 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:19.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:19.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:19.730 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:19.759 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:19.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:19.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:20.032 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:20.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:20.033 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:19 smithi040 ceph-mon[39581]: osdmap e956: 12 total, 12 up, 10 in 2024-04-29T14:49:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:20.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:20.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:20.183 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:20.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:20.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:20.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:20.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:19 smithi194 ceph-mon[44089]: osdmap e956: 12 total, 12 up, 10 in 2024-04-29T14:49:20.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:20.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:20.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:19 smithi121 ceph-mon[44533]: osdmap e956: 12 total, 12 up, 10 in 2024-04-29T14:49:20.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:20.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:20.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:20.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:20.570 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:20.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:20.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:20.698 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:20.748 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:20.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:20.755 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:20.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:20.881 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:21.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:20 smithi040 ceph-mon[39581]: pgmap v2413: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 5549/95750 objects misplaced (5.795%); 3.4 MiB/s, 53 objects/s recovering 2024-04-29T14:49:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:21.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:21.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:21.168 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:21.235 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:49:21.235 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":956,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T14:49:18.948004+0000","last_up_change":"2024-04-29T14:31:03.499853+0000","last_in_change":"2024-04-29T14:46:52.846324+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"796","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":10,"score_stable":10,"optimal_score":0.20000000298023224,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}},{"pool":7,"pool_name":"unique_pool_5","create_time":"2024-04-29T14:34:47.804723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"659","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":937,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":942,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":4178456629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":4178456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":4178456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":4178456629}]},"public_addr":"172.21.15.40:6805/4178456629","cluster_addr":"172.21.15.40:6806/4178456629","heartbeat_back_addr":"172.21.15.40:6808/4178456629","heartbeat_front_addr":"172.21.15.40:6807/4178456629","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":230,"up_from":260,"up_thru":874,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3416690575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3416690575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3416690575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3416690575}]},"public_addr":"172.21.15.40:6809/3416690575","cluster_addr":"172.21.15.40:6810/3416690575","heartbeat_back_addr":"172.21.15.40:6812/3416690575","heartbeat_front_addr":"172.21.15.40:6811/3416690575","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":938,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":928,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":388,"last_clean_end":457,"up_from":561,"up_thru":929,"down_at":459,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2522198013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2522198013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2522198013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2522198013}]},"public_addr":"172.21.15.121:6804/2522198013","cluster_addr":"172.21.15.121:6805/2522198013","heartbeat_back_addr":"172.21.15.121:6807/2522198013","heartbeat_front_addr":"172.21.15.121:6806/2522198013","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":883,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":64,"last_clean_end":320,"up_from":354,"up_thru":918,"down_at":321,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":165633203}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":165633203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":165633203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":165633203}]},"public_addr":"172.21.15.194:6800/165633203","cluster_addr":"172.21.15.194:6801/165633203","heartbeat_back_addr":"172.21.15.194:6803/165633203","heartbeat_front_addr":"172.21.15.194:6802/165633203","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":918,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":453882699}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":453882699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":453882699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":453882699}]},"public_addr":"172.21.15.194:6804/453882699","cluster_addr":"172.21.15.194:6805/453882699","heartbeat_back_addr":"172.21.15.194:6807/453882699","heartbeat_front_addr":"172.21.15.194:6806/453882699","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":930,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":0,"weight":0,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":524,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:49:02.211866+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:48:42.000051+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":51551,"last_purged_snaps_scrub":"2024-04-29T14:48:05.014203+0000","dead_epoch":230},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:49:18.852716+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:49:10.127364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:48:20.950673+0000","dead_epoch":457},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:49:12.446262+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:48:35.774306+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:49:14.137474+0000","dead_epoch":320},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:48:53.217532+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:49:11.450657+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":28165,"last_purged_snaps_scrub":"2024-04-29T14:48:18.238615+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"7.9","osds":[1,8]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T14:49:21.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:20 smithi194 ceph-mon[44089]: pgmap v2413: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 5549/95750 objects misplaced (5.795%); 3.4 MiB/s, 53 objects/s recovering 2024-04-29T14:49:21.334 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:49:21.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:20 smithi121 ceph-mon[44533]: pgmap v2413: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 5549/95750 objects misplaced (5.795%); 3.4 MiB/s, 53 objects/s recovering 2024-04-29T14:49:21.624 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:21.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:21.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:21.757 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:22.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4060646641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:49:22.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4060646641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:49:22.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4060646641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:49:22.506 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:22.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:22.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:22.640 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:22.662 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:22.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:22.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:22.804 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:22.878 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:22.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:22.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:23.025 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:23.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:22 smithi040 ceph-mon[39581]: pgmap v2414: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 5438/94458 objects misplaced (5.757%); 3.2 MiB/s, 51 objects/s recovering 2024-04-29T14:49:23.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:22 smithi040 ceph-mon[39581]: osdmap e957: 12 total, 12 up, 10 in 2024-04-29T14:49:23.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:22 smithi194 ceph-mon[44089]: pgmap v2414: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 5438/94458 objects misplaced (5.757%); 3.2 MiB/s, 51 objects/s recovering 2024-04-29T14:49:23.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:22 smithi194 ceph-mon[44089]: osdmap e957: 12 total, 12 up, 10 in 2024-04-29T14:49:23.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:22 smithi121 ceph-mon[44533]: pgmap v2414: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 5438/94458 objects misplaced (5.757%); 3.2 MiB/s, 51 objects/s recovering 2024-04-29T14:49:23.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:22 smithi121 ceph-mon[44533]: osdmap e957: 12 total, 12 up, 10 in 2024-04-29T14:49:23.575 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:23.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:23.588 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-29T14:49:23.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:23.710 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:23.736 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:23.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:23.851 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:23.897 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:23.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:49:24.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:24.029 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:24.278 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:24.281 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/528824842' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:49:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:24.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:24.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/528824842' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:49:24.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:24.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/528824842' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:49:24.413 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:24.712 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:24.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:25.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:25.101 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:25.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:25.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:25.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:25.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:24 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:25.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/528824842' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:49:25.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:24 smithi040 ceph-mon[39581]: osdmap e958: 12 total, 12 up, 10 in 2024-04-29T14:49:25.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:24 smithi040 ceph-mon[39581]: pgmap v2417: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 208 op/s; 5049/93972 objects misplaced (5.373%); 7.4 MiB/s, 119 objects/s recovering 2024-04-29T14:49:25.259 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:25.286 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:25.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:25.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:24 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:25.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/528824842' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:49:25.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:24 smithi194 ceph-mon[44089]: osdmap e958: 12 total, 12 up, 10 in 2024-04-29T14:49:25.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:24 smithi194 ceph-mon[44089]: pgmap v2417: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 208 op/s; 5049/93972 objects misplaced (5.373%); 7.4 MiB/s, 119 objects/s recovering 2024-04-29T14:49:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:24 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/528824842' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:49:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:24 smithi121 ceph-mon[44533]: osdmap e958: 12 total, 12 up, 10 in 2024-04-29T14:49:25.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:24 smithi121 ceph-mon[44533]: pgmap v2417: 17 pgs: 2 active+recovering, 15 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 208 op/s; 5049/93972 objects misplaced (5.373%); 7.4 MiB/s, 119 objects/s recovering 2024-04-29T14:49:25.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:25.418 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:25.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:25.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:25.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:25.581 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:25.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:25.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:25.969 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:49:25.970 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:49:25.970 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:49:25.970 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:49:25.970 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:49:25.972 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:26.222 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:26.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:26.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:26.363 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:26.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:49:26.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:49:27.214 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:27.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:27.224 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:27.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:27.345 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:27.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:27.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:27.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:27 smithi040 ceph-mon[39581]: pgmap v2418: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 231 op/s; 4935/92756 objects misplaced (5.320%); 6.6 MiB/s, 105 objects/s recovering 2024-04-29T14:49:27.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:27 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:49:27.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:27 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:49:27.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:27.497 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:27.520 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:27.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:27.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:27.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:27.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:27 smithi121 ceph-mon[44533]: pgmap v2418: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 231 op/s; 4935/92756 objects misplaced (5.320%); 6.6 MiB/s, 105 objects/s recovering 2024-04-29T14:49:27.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:27 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:49:27.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:27 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:49:27.651 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:27.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:27.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:27.780 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:27.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:27 smithi194 ceph-mon[44089]: pgmap v2418: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 231 op/s; 4935/92756 objects misplaced (5.320%); 6.6 MiB/s, 105 objects/s recovering 2024-04-29T14:49:27.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:27 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:49:27.804 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:27 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:49:27.806 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:27.826 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:27.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:27.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:27.944 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:49:27.956 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:27.964 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:49:27.975 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:27.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:49:28.007 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:49:28.032 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:49:28.063 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:49:28.079 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:28.092 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:49:28.105 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:28.116 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:49:28.124 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:28.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:28.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:49:28.166 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:49:28.189 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:49:28.212 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:49:28.214 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:49:28.214 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:49:28.214 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:49:28.214 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:49:28.214 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:49:28.215 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:49:28.215 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:49:28.215 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:49:28.215 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:49:28.215 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:49:28.215 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:49:28.215 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:49:28.227 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:28.254 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:28.277 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:28.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:28.358 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osdmap e959: 12 total, 12 up, 10 in 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3861368804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.359 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:28 smithi121 ceph-mon[44533]: 7.a scrub starts 2024-04-29T14:49:28.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:28.415 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:28.587 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:28.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:28.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osdmap e959: 12 total, 12 up, 10 in 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3861368804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:28 smithi040 ceph-mon[39581]: 7.a scrub starts 2024-04-29T14:49:28.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osdmap e959: 12 total, 12 up, 10 in 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3861368804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:28 smithi194 ceph-mon[44089]: 7.a scrub starts 2024-04-29T14:49:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:29 smithi040 ceph-mon[39581]: pgmap v2420: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 4935/92756 objects misplaced (5.320%); 4.6 MiB/s, 74 objects/s recovering 2024-04-29T14:49:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:29 smithi040 ceph-mon[39581]: 7.a scrub ok 2024-04-29T14:49:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:29 smithi121 ceph-mon[44533]: pgmap v2420: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 4935/92756 objects misplaced (5.320%); 4.6 MiB/s, 74 objects/s recovering 2024-04-29T14:49:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:29 smithi121 ceph-mon[44533]: 7.a scrub ok 2024-04-29T14:49:29.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:29 smithi194 ceph-mon[44089]: pgmap v2420: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 4935/92756 objects misplaced (5.320%); 4.6 MiB/s, 74 objects/s recovering 2024-04-29T14:49:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:29 smithi194 ceph-mon[44089]: 7.a scrub ok 2024-04-29T14:49:29.867 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:29.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:29.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:29.999 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:30.018 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:30.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:30.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:30.153 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:30.441 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:30 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:49:30.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:30 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:49:30.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:30 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:49:31.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:49:31.106 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:31.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:31.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:31.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:31.240 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:31.250 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:49:31.251 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:49:31.498 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:31.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:31.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:31 smithi040 ceph-mon[39581]: pgmap v2421: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 161 op/s; 4893/91958 objects misplaced (5.321%); 4.0 MiB/s, 147 objects/s recovering 2024-04-29T14:49:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:31 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:49:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:31 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:49:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:31 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:49:31.627 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:31 smithi194 ceph-mon[44089]: pgmap v2421: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 161 op/s; 4893/91958 objects misplaced (5.321%); 4.0 MiB/s, 147 objects/s recovering 2024-04-29T14:49:31.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:31 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:49:31.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:31 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:49:31.628 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:31 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:49:31.629 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:31 smithi121 ceph-mon[44533]: pgmap v2421: 17 pgs: 2 active+recovering, 15 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 161 op/s; 4893/91958 objects misplaced (5.321%); 4.0 MiB/s, 147 objects/s recovering 2024-04-29T14:49:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:31 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:49:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:31 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:49:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:31 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:49:31.648 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:31.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:31.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:31.787 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:32.381 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:49:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:32.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:49:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/625112946' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:32 smithi040 ceph-mon[39581]: from='client.37592 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/625112946' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:32 smithi121 ceph-mon[44533]: from='client.37592 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:32.800 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:32.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/625112946' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:32 smithi194 ceph-mon[44089]: from='client.37592 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:33.351 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:33.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:33.457 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:33.484 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:33.504 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:33.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:33.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:33 smithi040 ceph-mon[39581]: pgmap v2422: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 205 op/s; 4776/90736 objects misplaced (5.264%); 4.1 MiB/s, 143 objects/s recovering 2024-04-29T14:49:33.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:33 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:49:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:33 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:49:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:33 smithi040 ceph-mon[39581]: from='client.37592 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:33 smithi040 ceph-mon[39581]: osdmap e960: 12 total, 12 up, 10 in 2024-04-29T14:49:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:33 smithi040 ceph-mon[39581]: osdmap e961: 12 total, 12 up, 10 in 2024-04-29T14:49:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:33 smithi121 ceph-mon[44533]: pgmap v2422: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 205 op/s; 4776/90736 objects misplaced (5.264%); 4.1 MiB/s, 143 objects/s recovering 2024-04-29T14:49:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:33 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:49:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:33 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:49:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:33 smithi121 ceph-mon[44533]: from='client.37592 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:33 smithi121 ceph-mon[44533]: osdmap e960: 12 total, 12 up, 10 in 2024-04-29T14:49:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:33 smithi121 ceph-mon[44533]: osdmap e961: 12 total, 12 up, 10 in 2024-04-29T14:49:33.641 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:33 smithi194 ceph-mon[44089]: pgmap v2422: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 205 op/s; 4776/90736 objects misplaced (5.264%); 4.1 MiB/s, 143 objects/s recovering 2024-04-29T14:49:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:33 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:49:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:33 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:49:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:33 smithi194 ceph-mon[44089]: from='client.37592 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:33 smithi194 ceph-mon[44089]: osdmap e960: 12 total, 12 up, 10 in 2024-04-29T14:49:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:33 smithi194 ceph-mon[44089]: osdmap e961: 12 total, 12 up, 10 in 2024-04-29T14:49:34.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:34.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:34.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:34.226 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:34.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:34.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:34.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:34.562 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:34.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:34.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:34.720 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:49:34.720 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:49:34.795 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:34.821 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:34.839 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:34.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:34.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:34.969 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:34.989 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:34.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:35.030 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:49:35.053 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:49:35.073 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:49:35.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:35.094 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:49:35.115 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:49:35.122 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:35.136 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:49:35.157 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:49:35.181 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:49:35.201 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:49:35.220 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:49:35.240 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:49:35.260 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:49:35.262 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:49:35.263 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:49:35.263 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:49:35.263 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:49:35.263 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:49:35.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:49:35.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:49:35.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:35.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:35.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:35 smithi040 ceph-mon[39581]: pgmap v2425: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 188 op/s; 4776/90136 objects misplaced (5.299%); 921 KiB/s, 231 objects/s recovering 2024-04-29T14:49:35.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2263216221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:35 smithi121 ceph-mon[44533]: pgmap v2425: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 188 op/s; 4776/90136 objects misplaced (5.299%); 921 KiB/s, 231 objects/s recovering 2024-04-29T14:49:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2263216221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:35.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:35.736 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:35.770 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:35 smithi194 ceph-mon[44089]: pgmap v2425: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering, 14 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 188 op/s; 4776/90136 objects misplaced (5.299%); 921 KiB/s, 231 objects/s recovering 2024-04-29T14:49:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2263216221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:36.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:36.118 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:36.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:36.178 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:36.253 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:36.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:36.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:36.386 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:36.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.542 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:36.543 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:36 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:36.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:36.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:36.648 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.648 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.648 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.648 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.648 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:36.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:36 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:36 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:36.924 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:36.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:37.029 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:37.060 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:37.512 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:37.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:37.570 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:49:37.571 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:49:37.571 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:49:37.571 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:49:37.571 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:49:37.593 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:37 smithi121 ceph-mon[44533]: osdmap e962: 12 total, 12 up, 10 in 2024-04-29T14:49:37.594 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:49:37.594 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:49:37.594 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:49:37.594 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:37 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:49:37.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:37.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:37 smithi194 ceph-mon[44089]: osdmap e962: 12 total, 12 up, 10 in 2024-04-29T14:49:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:49:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:49:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:49:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:37 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:49:37.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:37 smithi040 ceph-mon[39581]: osdmap e962: 12 total, 12 up, 10 in 2024-04-29T14:49:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:49:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:49:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:49:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:37 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:49:38.058 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:49:38.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:49:38.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:38.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:38.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:38.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:38.447 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:38.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:38.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:38.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:38.601 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:38.613 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:38.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:38 smithi040 ceph-mon[39581]: osdmap e963: 12 total, 12 up, 10 in 2024-04-29T14:49:38.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:38 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:38.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:38 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:38.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:49:38.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:49:38.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:38 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:38.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:38 smithi040 ceph-mon[39581]: pgmap v2429: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 162 op/s; 4652/88806 objects misplaced (5.238%); 680 KiB/s, 156 objects/s recovering 2024-04-29T14:49:38.800 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:38.801 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:38.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:38.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:38 smithi121 ceph-mon[44533]: osdmap e963: 12 total, 12 up, 10 in 2024-04-29T14:49:38.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:38 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:38.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:38 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:38.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:49:38.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:49:38.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:38 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:38.839 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:38 smithi121 ceph-mon[44533]: pgmap v2429: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 162 op/s; 4652/88806 objects misplaced (5.238%); 680 KiB/s, 156 objects/s recovering 2024-04-29T14:49:38.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:38.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:39.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:39.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:38 smithi194 ceph-mon[44089]: osdmap e963: 12 total, 12 up, 10 in 2024-04-29T14:49:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:38 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:38 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:49:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:49:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:38 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:49:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:38 smithi194 ceph-mon[44089]: pgmap v2429: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 162 op/s; 4652/88806 objects misplaced (5.238%); 680 KiB/s, 156 objects/s recovering 2024-04-29T14:49:39.089 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:49:39.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:39.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:39.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:39.971 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:39.992 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:39.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:40.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2813820041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:49:40.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:39 smithi194 ceph-mon[44089]: from='client.27853 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:49:40.074 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2813820041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:49:40.075 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:39 smithi121 ceph-mon[44533]: from='client.27853 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:49:40.096 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:40.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2813820041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:49:40.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:39 smithi040 ceph-mon[39581]: from='client.27853 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:49:40.124 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:40.143 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:40.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:40.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:40.281 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:40.396 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:49:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 845/839 [PushOp(7:8290c88e:::benchmark_data_smithi194_147200_object65392:head, version: 667'3998, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8290c88e:::benchmark_data_smithi194_147200_object65392:head@667'3998, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:04.851897+0000", 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 275.53903996600002, 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7809149309999999, 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851897+0000", 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851897+0000", 2024-04-29T14:49:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851899+0000", 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.08e-06 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851927+0000", 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8034999999999998e-05 2024-04-29T14:49:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851944+0000", 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6884e-05 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851959+0000", 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4819e-05 2024-04-29T14:49:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:08.598632+0000", 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7466729330000001 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:08.599865+0000", 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012335250000000001 2024-04-29T14:49:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:08.632812+0000", 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032946654999999998 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 846/839 [PushReplyOp(7:8724884e:::benchmark_data_smithi194_147200_object65431:head)])", 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:14.097209+0000", 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 266.29372858599999, 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0735914299999996, 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097209+0000", 2024-04-29T14:49:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097209+0000", 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097211+0000", 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1919999999999999e-06 2024-04-29T14:49:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097218+0000", 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4479999999999997e-06 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097229+0000", 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1282e-05 2024-04-29T14:49:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097238+0000", 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.636e-06 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.144192+0000", 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0469534520000003 2024-04-29T14:49:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.170800+0000", 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026608420000000001 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 846/839 [PushOp(7:815c4b83:::benchmark_data_smithi194_147200_object65435:head, version: 667'4004, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:815c4b83:::benchmark_data_smithi194_147200_object65435:head@667'4004, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:14.277472+0000", 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 266.11346559100002, 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2569320919999996, 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277472+0000", 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277472+0000", 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277474+0000", 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.979e-06 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277606+0000", 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013254900000000001 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277627+0000", 2024-04-29T14:49:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0869999999999998e-05 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277646+0000", 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8822e-05 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.507156+0000", 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2295102309999999 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.508162+0000", 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001005932 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.534404+0000", 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026241708999999998 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 846/839 [PushOp(7:85b7373c:::benchmark_data_smithi194_147200_object65464:head, version: 667'4006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:85b7373c:::benchmark_data_smithi194_147200_object65464:head@667'4006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:16.510207+0000", 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.88072980300001, 2024-04-29T14:49:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.184098536, 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510207+0000", 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510207+0000", 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510209+0000", 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.564e-06 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510236+0000", 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6840000000000001e-05 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510253+0000", 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7036000000000001e-05 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510266+0000", 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2848e-05 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.644482+0000", 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1342165949999998 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.645497+0000", 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010150859999999999 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.694306+0000", 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.048808566999999997 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 847/839 [PushOp(7:803426e6:::benchmark_data_smithi194_147200_object65471:head, version: 667'4008, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:803426e6:::benchmark_data_smithi194_147200_object65471:head@667'4008, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:17.823241+0000", 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 262.567695722, 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2738754650000002, 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823241+0000", 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823241+0000", 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823243+0000", 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.95e-06 2024-04-29T14:49:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823278+0000", 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4175999999999999e-05 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823298+0000", 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0913000000000001e-05 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823318+0000", 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9752e-05 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.072385+0000", 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.249067041 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.073175+0000", 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078929100000000002 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.097117+0000", 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023942341999999998 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 847/839 [PushOp(7:8f822c3f:::benchmark_data_smithi194_147200_object65480:head, version: 667'4009, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8f822c3f:::benchmark_data_smithi194_147200_object65480:head@667'4009, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:17.996936+0000", 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 262.39400129900002, 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0732292760000002, 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996936+0000", 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996936+0000", 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996938+0000", 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8700000000000001e-06 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996966+0000", 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8141999999999999e-05 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996984+0000", 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7659000000000002e-05 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996997+0000", 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3732e-05 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.046353+0000", 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0493562150000004 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.047002+0000", 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064897399999999997 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.070165+0000", 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023162683999999999 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 847/839 [PushOp(7:8be67170:::benchmark_data_smithi194_147200_object65493:head, version: 667'4010, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8be67170:::benchmark_data_smithi194_147200_object65493:head@667'4010, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:18.099004+0000", 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 262.29193343700001, 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4893369410000004, 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099004+0000", 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099004+0000", 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099005+0000", 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5939999999999999e-06 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099033+0000", 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7946999999999999e-05 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099050+0000", 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6524e-05 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099062+0000", 2024-04-29T14:49:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2170999999999999e-05 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.566261+0000", 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4671990209999999 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.566909+0000", 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064791899999999995 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.588341+0000", 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021431764999999998 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 848/839 [PushOp(7:81c58ee4:::benchmark_data_smithi194_147200_object65517:head, version: 667'4012, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:81c58ee4:::benchmark_data_smithi194_147200_object65517:head@667'4012, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:20.562673+0000", 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 259.82826376700001, 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0525536600000001, 2024-04-29T14:49:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562673+0000", 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562673+0000", 2024-04-29T14:49:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562674+0000", 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.08e-06 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562735+0000", 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0764999999999998e-05 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562752+0000", 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6895e-05 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562767+0000", 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4763000000000001e-05 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.574454+0000", 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0116869529999999 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.575114+0000", 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065992799999999995 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.615227+0000", 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.040113276000000003 2024-04-29T14:49:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 849/839 [PushReplyOp(7:8a2e7b36:::benchmark_data_smithi194_147200_object65508:head)])", 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:21.946060+0000", 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 258.444877163, 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3263969549999999, 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946060+0000", 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946060+0000", 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946063+0000", 2024-04-29T14:49:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5550000000000001e-06 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946070+0000", 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7889999999999995e-06 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946082+0000", 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1673e-05 2024-04-29T14:49:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946092+0000", 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0397999999999999e-05 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:25.246776+0000", 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3006832780000002 2024-04-29T14:49:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:25.272457+0000", 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025681262 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 897/887 [PushOp(7:83cce57a:::benchmark_data_smithi194_147200_object74932:head, version: 670'4606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:83cce57a:::benchmark_data_smithi194_147200_object74932:head@670'4606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:09.877870+0000", 2024-04-29T14:49:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 150.51306728700001, 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.833420362, 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877870+0000", 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877870+0000", 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877872+0000", 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.826e-06 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877903+0000", 2024-04-29T14:49:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1426e-05 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877922+0000", 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8706999999999999e-05 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877940+0000", 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7745999999999999e-05 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:13.690096+0000", 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.812156436 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:13.690656+0000", 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056038400000000003 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:13.711290+0000", 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020633836999999999 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 897/887 [PushOp(7:80dd5181:::benchmark_data_smithi194_147200_object74941:head, version: 670'4607, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:80dd5181:::benchmark_data_smithi194_147200_object74941:head@670'4607, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:10.642330+0000", 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.748607314, 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7214084810000001, 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642330+0000", 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642330+0000", 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642332+0000", 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8729999999999999e-06 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642464+0000", 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013224499999999999 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642482+0000", 2024-04-29T14:49:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7906000000000001e-05 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642495+0000", 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2952999999999999e-05 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:16.344214+0000", 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7017192489999999 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:16.344718+0000", 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050386199999999999 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:16.363738+0000", 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019020393 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 903/887 [PushOp(7:803b1c22:::benchmark_data_smithi194_147200_object75163:head, version: 670'4624, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:803b1c22:::benchmark_data_smithi194_147200_object75163:head@670'4624, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:33.530104+0000", 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.860833601, 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0443010020000001, 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530104+0000", 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530104+0000", 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530105+0000", 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.088e-06 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530149+0000", 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4774999999999997e-05 2024-04-29T14:49:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530162+0000", 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3055e-05 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530180+0000", 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7371999999999999e-05 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.555736+0000", 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0255566470000002 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.556553+0000", 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000816152 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.574405+0000", 2024-04-29T14:49:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017851913000000001 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 903/887 [PushOp(7:8893102c:::benchmark_data_smithi194_147200_object75183:head, version: 670'4625, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8893102c:::benchmark_data_smithi194_147200_object75183:head@670'4625, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:33.796243+0000", 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.59469388700001, 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4972395779999998, 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796243+0000", 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796243+0000", 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796245+0000", 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.347e-06 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796385+0000", 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014044900000000001 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796400+0000", 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5208000000000001e-05 2024-04-29T14:49:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796411+0000", 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0491e-05 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.276226+0000", 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4798150310000002 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.276698+0000", 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047268000000000001 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.293483+0000", 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016784371999999999 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 904/887 [PushOp(7:8f1fbb9a:::benchmark_data_smithi194_147200_object75217:head, version: 670'4626, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8f1fbb9a:::benchmark_data_smithi194_147200_object75217:head@670'4626, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:35.303811+0000", 2024-04-29T14:49:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.08712611599999, 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2934006399999998, 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303811+0000", 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303811+0000", 2024-04-29T14:49:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303812+0000", 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.229e-06 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303857+0000", 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5207000000000003e-05 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303871+0000", 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3895e-05 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303882+0000", 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0249e-05 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.578667+0000", 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2747856049999999 2024-04-29T14:49:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.579144+0000", 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047689999999999999 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.597212+0000", 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018067554999999999 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 904/887 [PushReplyOp(7:81acef38:::benchmark_data_smithi194_147200_object75157:head)])", 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:35.677217+0000", 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 124.71372015599999, 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3540995119999999, 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677217+0000", 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677217+0000", 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677218+0000", 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.471e-06 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677225+0000", 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1970000000000001e-06 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677234+0000", 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7680000000000003e-06 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677241+0000", 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4849999999999997e-06 2024-04-29T14:49:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:39.011718+0000", 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3344772320000002 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:39.031317+0000", 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019598358999999999 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 905/887 [PushOp(7:88af4bef:::benchmark_data_smithi194_147200_object75234:head, version: 670'4627, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:88af4bef:::benchmark_data_smithi194_147200_object75234:head@670'4627, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.178519+0000", 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.212418216, 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8373110759999998, 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178519+0000", 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178519+0000", 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178520+0000", 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.497e-06 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178570+0000", 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9801e-05 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178586+0000", 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6022e-05 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178600+0000", 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3458000000000001e-05 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.998635+0000", 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8200351159999997 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.999118+0000", 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048342699999999998 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:42.015830+0000", 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016711754999999998 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 906/887 [PushOp(7:885c0f18:::benchmark_data_smithi194_147200_object75244:head, version: 670'4628, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:885c0f18:::benchmark_data_smithi194_147200_object75244:head@670'4628, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.539025+0000", 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 122.851912067, 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5707472290000002, 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539025+0000", 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539025+0000", 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539026+0000", 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28e-06 2024-04-29T14:49:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539138+0000", 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011203499999999999 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539154+0000", 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.556e-05 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539165+0000", 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1306e-05 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.092638+0000", 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.553472298 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.093242+0000", 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060396899999999999 2024-04-29T14:49:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.109772+0000", 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016530781000000001 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 907/887 [PushReplyOp(7:8893102c:::benchmark_data_smithi194_147200_object75183:head)])", 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.943579+0000", 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 122.447358091, 2024-04-29T14:49:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8411319420000001, 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943579+0000", 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943579+0000", 2024-04-29T14:49:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943580+0000", 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943586+0000", 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4049999999999999e-06 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943596+0000", 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0144e-05 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943604+0000", 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-06 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.766160+0000", 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.822556804 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.784711+0000", 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01855064 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 907/887 [PushOp(7:81e45716:::benchmark_data_smithi194_147200_object75252:head, version: 670'4629, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:81e45716:::benchmark_data_smithi194_147200_object75252:head@670'4629, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.991562+0000", 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 122.39937505899999, 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8482403650000001, 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991562+0000", 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991562+0000", 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991564+0000", 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991699+0000", 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013455800000000001 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991715+0000", 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6444999999999999e-05 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991733+0000", 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.823e-05 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.822135+0000", 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8304019990000002 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.822635+0000", 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050026699999999999 2024-04-29T14:49:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.839802+0000", 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017166976 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_backoff(7.1 ack-block id 132 [7:87f01edb:::benchmark_data_smithi194_147200_object63188:head,7:87f01edb:::benchmark_data_smithi194_147200_object63188:head) e926)", 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:20.238054+0000", 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 80.152883625000001, 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4887491290000003, 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 0 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238054+0000", 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238054+0000", 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238055+0000", 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.358e-06 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238059+0000", 2024-04-29T14:49:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5120000000000002e-06 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238070+0000", 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0566e-05 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238077+0000", 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5409999999999997e-06 2024-04-29T14:49:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.323801+0000", 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085724754 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.323810+0000", 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5199999999999997e-06 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:21.306116+0000", 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.98230652600000001 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:21.306132+0000", 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5196e-05 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:25.726052+0000", 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4199200200000002 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:25.726803+0000", 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075113600000000001 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:49:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:40.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:40.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:40.666 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:40.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:40.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:40.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:40.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:40 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:40.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:40 smithi040 ceph-mon[39581]: from='client.27853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:49:40.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:40 smithi040 ceph-mon[39581]: osdmap e964: 12 total, 12 up, 10 in 2024-04-29T14:49:40.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:40 smithi040 ceph-mon[39581]: pgmap v2431: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 4652/87992 objects misplaced (5.287%); 672 KiB/s, 174 objects/s recovering 2024-04-29T14:49:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:40.910 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:40.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:41.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:41.045 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:40 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:40 smithi194 ceph-mon[44089]: from='client.27853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:49:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:40 smithi194 ceph-mon[44089]: osdmap e964: 12 total, 12 up, 10 in 2024-04-29T14:49:41.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:40 smithi194 ceph-mon[44089]: pgmap v2431: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 4652/87992 objects misplaced (5.287%); 672 KiB/s, 174 objects/s recovering 2024-04-29T14:49:41.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:40 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:41.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:40 smithi121 ceph-mon[44533]: from='client.27853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:49:41.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:40 smithi121 ceph-mon[44533]: osdmap e964: 12 total, 12 up, 10 in 2024-04-29T14:49:41.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:40 smithi121 ceph-mon[44533]: pgmap v2431: 17 pgs: 2 active+recovering, 15 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 4652/87992 objects misplaced (5.287%); 672 KiB/s, 174 objects/s recovering 2024-04-29T14:49:41.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:49:41.244 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:41.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:41.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:41.381 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:41.404 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:41.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:41.490 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:41.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:41.552 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:49:41.561 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:41.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:41.575 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:49:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:49:41.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:49:41.639 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:49:41.663 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:49:41.667 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:41.684 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:49:41.694 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:41.706 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:49:41.714 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:41.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:41.728 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:49:41.750 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:49:41.772 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:49:41.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:49:41.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:49:41.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:49:41.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:49:41.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:49:41.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:49:41.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:49:41.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:49:41.798 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:49:41.798 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:49:41.798 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:49:41.798 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:49:41.798 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:49:41.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:41.827 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4053638208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:41.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:41.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:42.001 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4053638208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:42.004 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:42.016 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4053638208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:42.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:42.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:42.608 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:42.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:42.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:42.743 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:42.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:42.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:42.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:42.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: pgmap v2432: 17 pgs: 2 active+recovering, 15 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 4534/86572 objects misplaced (5.237%); 629 KiB/s, 173 objects/s recovering 2024-04-29T14:49:42.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:42 smithi040 ceph-mon[39581]: osdmap e965: 12 total, 12 up, 10 in 2024-04-29T14:49:42.902 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:43.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: pgmap v2432: 17 pgs: 2 active+recovering, 15 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 4534/86572 objects misplaced (5.237%); 629 KiB/s, 173 objects/s recovering 2024-04-29T14:49:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:42 smithi194 ceph-mon[44089]: osdmap e965: 12 total, 12 up, 10 in 2024-04-29T14:49:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: pgmap v2432: 17 pgs: 2 active+recovering, 15 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 4534/86572 objects misplaced (5.237%); 629 KiB/s, 173 objects/s recovering 2024-04-29T14:49:43.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:42 smithi121 ceph-mon[44533]: osdmap e965: 12 total, 12 up, 10 in 2024-04-29T14:49:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:43.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:43.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:49:43.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:43.285 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:43.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:43.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:43.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:43.457 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:49:43.457 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:49:43.486 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:43.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:43.549 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:43.588 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:49:43.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:49:43.592 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:43.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:43.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:43.848 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:43.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:43.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:43.985 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:44.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:49:44.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:44.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:44.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:44.320 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:44.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:44.457 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:44.465 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:44.550 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:44.564 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:49:44.576 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:44.595 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:44.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:44.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:44.731 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:45.295 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:45.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:45.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:45.430 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:45 smithi040 ceph-mon[39581]: pgmap v2434: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 211 op/s; 4534/85970 objects misplaced (5.274%); 562 KiB/s, 335 objects/s recovering 2024-04-29T14:49:45.433 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:45.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:45 smithi121 ceph-mon[44533]: pgmap v2434: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 211 op/s; 4534/85970 objects misplaced (5.274%); 562 KiB/s, 335 objects/s recovering 2024-04-29T14:49:45.683 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:45.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:45.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:49:45.818 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:45.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:45 smithi194 ceph-mon[44089]: pgmap v2434: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 211 op/s; 4534/85970 objects misplaced (5.274%); 562 KiB/s, 335 objects/s recovering 2024-04-29T14:49:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:49:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:45.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:49:46.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:46.075 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:46.095 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:46.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:46.198 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:46.228 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:46.247 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:49:46.247 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:49:46.592 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:46.592 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:46.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:46.615 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:46.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:46.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:46.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:49:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:46.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:47.100 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:47.129 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:47.148 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:47.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:47.205 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:47.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:47.368 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:49:47.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:47 smithi194 ceph-mon[44089]: pgmap v2435: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 258 op/s; 4407/84672 objects misplaced (5.205%); 992 KiB/s, 290 objects/s recovering 2024-04-29T14:49:47.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:47 smithi194 ceph-mon[44089]: osdmap e966: 12 total, 12 up, 10 in 2024-04-29T14:49:47.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3707521396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:47.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:47 smithi040 ceph-mon[39581]: pgmap v2435: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 258 op/s; 4407/84672 objects misplaced (5.205%); 992 KiB/s, 290 objects/s recovering 2024-04-29T14:49:47.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:47 smithi040 ceph-mon[39581]: osdmap e966: 12 total, 12 up, 10 in 2024-04-29T14:49:47.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3707521396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:47.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:47 smithi121 ceph-mon[44533]: pgmap v2435: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 258 op/s; 4407/84672 objects misplaced (5.205%); 992 KiB/s, 290 objects/s recovering 2024-04-29T14:49:47.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:47 smithi121 ceph-mon[44533]: osdmap e966: 12 total, 12 up, 10 in 2024-04-29T14:49:47.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3707521396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:49:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:48.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:48.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:49:48.261 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:49:48.262 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:49:48.275 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:48.475 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:49:48.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:48 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:48.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3707521396' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:48.477 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:48 smithi040 ceph-mon[39581]: osdmap e967: 12 total, 12 up, 10 in 2024-04-29T14:49:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:48.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:49:48.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:48.613 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:48.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:48 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:48.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3707521396' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:48.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:48 smithi121 ceph-mon[44533]: osdmap e967: 12 total, 12 up, 10 in 2024-04-29T14:49:48.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:48.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:48.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:48.789 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:48 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3707521396' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:49:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:48 smithi194 ceph-mon[44089]: osdmap e967: 12 total, 12 up, 10 in 2024-04-29T14:49:49.083 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2437,"stamp":"2024-04-29T14:49:47.364556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2774925880,"num_objects":42336,"num_object_clones":0,"num_object_copies":84672,"num_objects_missing_on_primary":1797,"num_objects_missing":1797,"num_objects_degraded":0,"num_objects_misplaced":4407,"num_objects_unfound":0,"num_objects_dirty":42336,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":187667,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231956,"num_bytes_recovered":13602488056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":13200924,"kb_used_data":6361208,"kb_used_omap":668,"kb_used_meta":6838947,"kb_avail":924250596,"statfs":{"total":959950356480,"available":946432610304,"internally_reserved":0,"allocated":6513876992,"data_stored":6494852304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7003082735},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-108789760,"num_objects":-1660,"num_object_clones":0,"num_object_copies":-3320,"num_objects_missing_on_primary":-121,"num_objects_missing":-121,"num_objects_degraded":0,"num_objects_misplaced":-245,"num_objects_unfound":0,"num_objects_dirty":-1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1660,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":8126464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.067180"},"pg_stats":[{"pgid":"7.f","version":"965'11823","reported_seq":46744,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:45.027851+0000","last_change":"2024-04-29T14:49:26.622778+0000","last_active":"2024-04-29T14:49:45.027851+0000","last_peered":"2024-04-29T14:49:45.027851+0000","last_clean":"2024-04-29T14:49:45.027851+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:49:45.027851+0000","last_undegraded":"2024-04-29T14:49:45.027851+0000","last_fullsized":"2024-04-29T14:49:45.027851+0000","mapping_epoch":918,"log_start":"661'1800","ondisk_log_start":"661'1800","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'11583","last_scrub_stamp":"2024-04-29T14:49:26.622601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:26.622601+0000","objects_scrubbed":2863,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:35.366659+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171769856,"num_objects":2621,"num_object_clones":0,"num_object_copies":5242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11823,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"965'18947","reported_seq":40902,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:44.236047+0000","last_change":"2024-04-29T14:48:27.585228+0000","last_active":"2024-04-29T14:49:44.236047+0000","last_peered":"2024-04-29T14:49:44.236047+0000","last_clean":"2024-04-29T14:49:44.236047+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:49:44.236047+0000","last_undegraded":"2024-04-29T14:49:44.236047+0000","last_fullsized":"2024-04-29T14:49:44.236047+0000","mapping_epoch":930,"log_start":"704'8858","ondisk_log_start":"704'8858","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'18172","last_scrub_stamp":"2024-04-29T14:47:59.574965+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:47:59.574965+0000","objects_scrubbed":3434,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174260224,"num_objects":2659,"num_object_clones":0,"num_object_copies":5318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"965'11684","reported_seq":42791,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:45.430288+0000","last_change":"2024-04-29T14:48:44.919582+0000","last_active":"2024-04-29T14:49:45.430288+0000","last_peered":"2024-04-29T14:49:45.430288+0000","last_clean":"2024-04-29T14:49:45.430288+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:49:45.430288+0000","last_undegraded":"2024-04-29T14:49:45.430288+0000","last_fullsized":"2024-04-29T14:49:45.430288+0000","mapping_epoch":938,"log_start":"661'1682","ondisk_log_start":"661'1682","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11104","last_scrub_stamp":"2024-04-29T14:48:44.919467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:44.919467+0000","objects_scrubbed":3222,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:55.396438+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173080600,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11684,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"965'14896","reported_seq":80366,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:45.544108+0000","last_change":"2024-04-29T14:48:59.043912+0000","last_active":"2024-04-29T14:49:45.544108+0000","last_peered":"2024-04-29T14:49:45.544108+0000","last_clean":"2024-04-29T14:49:45.544108+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:49:45.544108+0000","last_undegraded":"2024-04-29T14:49:45.544108+0000","last_fullsized":"2024-04-29T14:49:45.544108+0000","mapping_epoch":939,"log_start":"671'4850","ondisk_log_start":"671'4850","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"946'14374","last_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:48:59.043790+0000","objects_scrubbed":3160,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:22.421108+0000","scrub_duration":10465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171180032,"num_objects":2612,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11736,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"965'11652","reported_seq":26294,"reported_epoch":965,"state":"active+recovering","last_fresh":"2024-04-29T14:49:45.790434+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:49:45.790434+0000","last_peered":"2024-04-29T14:49:45.790434+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:49:45.790434+0000","last_undegraded":"2024-04-29T14:49:45.790434+0000","last_fullsized":"2024-04-29T14:49:45.790434+0000","mapping_epoch":942,"log_start":"661'1600","ondisk_log_start":"661'1600","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172359680,"num_objects":2630,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":1797,"num_objects_missing":1797,"num_objects_degraded":0,"num_objects_misplaced":4407,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4958,"num_bytes_recovered":324796416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":20},{"shards":"1,10,11","objects":813},{"shards":"10,11","objects":1797}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"965'19084","reported_seq":36523,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:45.020638+0000","last_change":"2024-04-29T14:49:28.483892+0000","last_active":"2024-04-29T14:49:45.020638+0000","last_peered":"2024-04-29T14:49:45.020638+0000","last_clean":"2024-04-29T14:49:45.020638+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:49:45.020638+0000","last_undegraded":"2024-04-29T14:49:45.020638+0000","last_fullsized":"2024-04-29T14:49:45.020638+0000","mapping_epoch":895,"log_start":"704'9000","ondisk_log_start":"704'9000","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'18861","last_scrub_stamp":"2024-04-29T14:49:28.483744+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:49:28.483744+0000","objects_scrubbed":2889,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:43.869641+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174587904,"num_objects":2664,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12010,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"965'11732","reported_seq":40859,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:45.717785+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:49:45.717785+0000","last_peered":"2024-04-29T14:49:45.717785+0000","last_clean":"2024-04-29T14:49:45.717785+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:49:45.717785+0000","last_undegraded":"2024-04-29T14:49:45.717785+0000","last_fullsized":"2024-04-29T14:49:45.717785+0000","mapping_epoch":756,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:56.779052+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170917888,"num_objects":2608,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"965'11624","reported_seq":20643,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:46.148281+0000","last_change":"2024-04-29T14:48:41.799593+0000","last_active":"2024-04-29T14:49:46.148281+0000","last_peered":"2024-04-29T14:49:46.148281+0000","last_clean":"2024-04-29T14:49:46.148281+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:49:46.148281+0000","last_undegraded":"2024-04-29T14:49:46.148281+0000","last_fullsized":"2024-04-29T14:49:46.148281+0000","mapping_epoch":883,"log_start":"661'1600","ondisk_log_start":"661'1600","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11076","last_scrub_stamp":"2024-04-29T14:48:41.799459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.799459+0000","objects_scrubbed":3214,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:11.502178+0000","scrub_duration":286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174718976,"num_objects":2666,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"965'11953","reported_seq":70231,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:45.470061+0000","last_change":"2024-04-29T14:49:30.882066+0000","last_active":"2024-04-29T14:49:45.470061+0000","last_peered":"2024-04-29T14:49:45.470061+0000","last_clean":"2024-04-29T14:49:45.470061+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:49:45.470061+0000","last_undegraded":"2024-04-29T14:49:45.470061+0000","last_fullsized":"2024-04-29T14:49:45.470061+0000","mapping_epoch":939,"log_start":"661'1876","ondisk_log_start":"661'1876","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11756","last_scrub_stamp":"2024-04-29T14:49:30.881939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.881939+0000","objects_scrubbed":2972,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:39.012112+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181862400,"num_objects":2775,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11953,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"965'11704","reported_seq":45922,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:45.006706+0000","last_change":"2024-04-29T14:49:30.401449+0000","last_active":"2024-04-29T14:49:45.006706+0000","last_peered":"2024-04-29T14:49:45.006706+0000","last_clean":"2024-04-29T14:49:45.006706+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:49:45.006706+0000","last_undegraded":"2024-04-29T14:49:45.006706+0000","last_fullsized":"2024-04-29T14:49:45.006706+0000","mapping_epoch":918,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11508","last_scrub_stamp":"2024-04-29T14:49:30.401266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.401266+0000","objects_scrubbed":2766,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:32.430363+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168427520,"num_objects":2570,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"965'11836","reported_seq":48146,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:44.258377+0000","last_change":"2024-04-29T14:47:55.674071+0000","last_active":"2024-04-29T14:49:44.258377+0000","last_peered":"2024-04-29T14:49:44.258377+0000","last_clean":"2024-04-29T14:49:44.258377+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:49:44.258377+0000","last_undegraded":"2024-04-29T14:49:44.258377+0000","last_fullsized":"2024-04-29T14:49:44.258377+0000","mapping_epoch":915,"log_start":"661'1800","ondisk_log_start":"661'1800","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:49.168576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":5336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"965'15237","reported_seq":39951,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:43.246350+0000","last_change":"2024-04-29T14:48:41.280889+0000","last_active":"2024-04-29T14:49:43.246350+0000","last_peered":"2024-04-29T14:49:43.246350+0000","last_clean":"2024-04-29T14:49:43.246350+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:49:43.246350+0000","last_undegraded":"2024-04-29T14:49:43.246350+0000","last_fullsized":"2024-04-29T14:49:43.246350+0000","mapping_epoch":895,"log_start":"673'5225","ondisk_log_start":"673'5225","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'14643","last_scrub_stamp":"2024-04-29T14:48:41.280598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.280598+0000","objects_scrubbed":3364,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:03.594726+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181534720,"num_objects":2770,"num_object_clones":0,"num_object_copies":5540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11638,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"965'11547","reported_seq":35942,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:43.226224+0000","last_change":"2024-04-29T14:48:53.235223+0000","last_active":"2024-04-29T14:49:43.226224+0000","last_peered":"2024-04-29T14:49:43.226224+0000","last_clean":"2024-04-29T14:49:43.226224+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:49:43.226224+0000","last_undegraded":"2024-04-29T14:49:43.226224+0000","last_fullsized":"2024-04-29T14:49:43.226224+0000","mapping_epoch":895,"log_start":"661'1544","ondisk_log_start":"661'1544","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'11030","last_scrub_stamp":"2024-04-29T14:48:53.235099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:53.235099+0000","objects_scrubbed":3160,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:53.820864+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173211648,"num_objects":2643,"num_object_clones":0,"num_object_copies":5286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"965'11537","reported_seq":58263,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:43.771003+0000","last_change":"2024-04-29T14:49:42.611751+0000","last_active":"2024-04-29T14:49:43.771003+0000","last_peered":"2024-04-29T14:49:43.771003+0000","last_clean":"2024-04-29T14:49:43.771003+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:49:43.771003+0000","last_undegraded":"2024-04-29T14:49:43.771003+0000","last_fullsized":"2024-04-29T14:49:43.771003+0000","mapping_epoch":928,"log_start":"661'1500","ondisk_log_start":"661'1500","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'9290","last_scrub_stamp":"2024-04-29T14:44:09.405827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:09.405827+0000","objects_scrubbed":4890,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:49.168597+0000","scrub_duration":526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173080576,"num_objects":2641,"num_object_clones":0,"num_object_copies":5282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"965'11636","reported_seq":26580,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:43.396432+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:49:43.396432+0000","last_peered":"2024-04-29T14:49:43.396432+0000","last_clean":"2024-04-29T14:49:43.396432+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:49:43.396432+0000","last_undegraded":"2024-04-29T14:49:43.396432+0000","last_fullsized":"2024-04-29T14:49:43.396432+0000","mapping_epoch":703,"log_start":"661'1600","ondisk_log_start":"661'1600","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:47.174767+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170917888,"num_objects":2608,"num_object_clones":0,"num_object_copies":5216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"965'11483","reported_seq":32771,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:46.062149+0000","last_change":"2024-04-29T14:48:40.829926+0000","last_active":"2024-04-29T14:49:46.062149+0000","last_peered":"2024-04-29T14:49:46.062149+0000","last_clean":"2024-04-29T14:49:46.062149+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:49:46.062149+0000","last_undegraded":"2024-04-29T14:49:46.062149+0000","last_fullsized":"2024-04-29T14:49:46.062149+0000","mapping_epoch":865,"log_start":"661'1400","ondisk_log_start":"661'1400","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'10915","last_scrub_stamp":"2024-04-29T14:48:40.829814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:40.829814+0000","objects_scrubbed":3125,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:48.225913+0000","scrub_duration":292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167575552,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11483,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1907,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-29T14:49:42.614718+0000","last_change":"2024-04-29T14:48:42.724622+0000","last_active":"2024-04-29T14:49:42.614718+0000","last_peered":"2024-04-29T14:49:42.614718+0000","last_clean":"2024-04-29T14:49:42.614718+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:49:42.614718+0000","last_undegraded":"2024-04-29T14:49:42.614718+0000","last_fullsized":"2024-04-29T14:49:42.614718+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:48:42.724519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:48:42.724519+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:07.483218+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2774335512,"num_objects":42334,"num_object_clones":0,"num_object_copies":84668,"num_objects_missing_on_primary":1797,"num_objects_missing":1797,"num_objects_degraded":0,"num_objects_misplaced":4407,"num_objects_unfound":0,"num_objects_dirty":42334,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187486,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231942,"num_bytes_recovered":13598785560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6926180352,"data_stored":6926172208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160640,"ondisk_log_size":160640,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122563,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384202,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1932124,"kb_used_data":1048528,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91813028,"statfs":{"total":95995035648,"available":94016540672,"internally_reserved":0,"allocated":1073692672,"data_stored":1071791710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":904788416},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645837,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1516316,"kb_used_data":704912,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92228836,"statfs":{"total":95995035648,"available":94442328064,"internally_reserved":0,"allocated":721829888,"data_stored":719928946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":830863808},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423141,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":667228,"kb_used_data":518096,"kb_used_omap":4,"kb_used_meta":149115,"kb_avail":93077924,"statfs":{"total":95995035648,"available":95311794176,"internally_reserved":0,"allocated":530530304,"data_stored":528629362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":152694464},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1210144,"kb_used_data":340440,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92535008,"statfs":{"total":95995035648,"available":94755848192,"internally_reserved":0,"allocated":348610560,"data_stored":346706066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":890567114},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528859,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506148,"kb_used_data":518428,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92239004,"statfs":{"total":95995035648,"available":94452740096,"internally_reserved":0,"allocated":530870272,"data_stored":528961706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575399,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":886044,"kb_used_data":403668,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92859108,"statfs":{"total":95995035648,"available":95087726592,"internally_reserved":0,"allocated":413356032,"data_stored":411451018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738937,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736856,"kb_used_data":817168,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92008296,"statfs":{"total":95995035648,"available":94216495104,"internally_reserved":0,"allocated":836780032,"data_stored":834879090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497371,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182956,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1363548,"kb_used_data":786768,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92381604,"statfs":{"total":95995035648,"available":94598762496,"internally_reserved":0,"allocated":805650432,"data_stored":803749490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":590608840},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986721,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1658328,"kb_used_data":692816,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":92086824,"statfs":{"total":95995035648,"available":94296907776,"internally_reserved":0,"allocated":709443584,"data_stored":707542642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":988674501},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653282,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":724188,"kb_used_data":530384,"kb_used_omap":651,"kb_used_meta":193140,"kb_avail":93020964,"statfs":{"total":95995035648,"available":95253467136,"internally_reserved":0,"allocated":543113216,"data_stored":541212274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":197776136},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":832634880,"data_stored":832634880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":406327296,"data_stored":406323224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":799014912,"data_stored":799014912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":705363968,"data_stored":705363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":535101440,"data_stored":535101440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":526127104,"data_stored":526123032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342294528,"data_stored":342294528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":525139968,"data_stored":525139968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":718012416,"data_stored":718012416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1069023232,"data_stored":1069023232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:49:49.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:49:49.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:49 smithi121 ceph-mon[44533]: from='client.18150 v1:172.21.15.40:0/4034569441' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:49:49.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:49 smithi121 ceph-mon[44533]: pgmap v2438: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 158 op/s; 4407/84672 objects misplaced (5.205%); 693 KiB/s, 212 objects/s recovering 2024-04-29T14:49:49.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:49:49.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:49:49.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:49 smithi040 ceph-mon[39581]: from='client.18150 v1:172.21.15.40:0/4034569441' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:49:49.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:49 smithi040 ceph-mon[39581]: pgmap v2438: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 158 op/s; 4407/84672 objects misplaced (5.205%); 693 KiB/s, 212 objects/s recovering 2024-04-29T14:49:49.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:49:49.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:49:49.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:49 smithi194 ceph-mon[44089]: from='client.18150 v1:172.21.15.40:0/4034569441' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:49:49.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:49 smithi194 ceph-mon[44089]: pgmap v2438: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 158 op/s; 4407/84672 objects misplaced (5.205%); 693 KiB/s, 212 objects/s recovering 2024-04-29T14:49:49.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:49:49.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:49:49.939 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:49.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:50.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:50.077 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:50.566 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:50.609 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:49:50.629 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:49:50.648 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:49:50.668 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:49:50.689 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:49:50.710 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:49:50.730 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:49:50.750 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:49:50.769 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:49:50.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:49:50.811 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:49:50.834 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:49:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:49:50.836 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:49:50.836 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:49:50.836 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:49:50.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:50 smithi040 ceph-mon[39581]: osdmap e968: 12 total, 12 up, 10 in 2024-04-29T14:49:50.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:50 smithi040 ceph-mon[39581]: pgmap v2440: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 176 op/s; 4407/83850 objects misplaced (5.256%); 693 KiB/s, 10 objects/s recovering 2024-04-29T14:49:50.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1934934776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:50 smithi121 ceph-mon[44533]: osdmap e968: 12 total, 12 up, 10 in 2024-04-29T14:49:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:50 smithi121 ceph-mon[44533]: pgmap v2440: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 176 op/s; 4407/83850 objects misplaced (5.256%); 693 KiB/s, 10 objects/s recovering 2024-04-29T14:49:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1934934776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:51.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:50 smithi194 ceph-mon[44089]: osdmap e968: 12 total, 12 up, 10 in 2024-04-29T14:49:51.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:50 smithi194 ceph-mon[44089]: pgmap v2440: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 176 op/s; 4407/83850 objects misplaced (5.256%); 693 KiB/s, 10 objects/s recovering 2024-04-29T14:49:51.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1934934776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:49:51.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:51.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:51.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:51.657 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:51.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.758 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:51.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:51.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:51.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:51.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:51.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:52.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:52.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:49:52.054 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:49:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:49:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:49:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:49:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:230993 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.231367544999998, 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076782375, 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817203+0000", 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.203e-06 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817208+0000", 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999996e-06 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817218+0000", 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5929999999999995e-06 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817224+0000", 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7530000000000004e-06 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893964+0000", 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076739872000000001 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893972+0000", 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0450000000000001e-06 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893985+0000", 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2196e-05 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231043 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.043768166, 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.084551465000000006, 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:49:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004804+0000", 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2709999999999999e-06 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004816+0000", 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1864e-05 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004832+0000", 2024-04-29T14:49:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6670000000000001e-05 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004845+0000", 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2401e-05 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089323+0000", 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.084478445999999999 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089334+0000", 2024-04-29T14:49:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1250000000000001e-05 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089353+0000", 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8563000000000002e-05 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231044 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.039452165999997, 2024-04-29T14:49:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080603953000000006, 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009119+0000", 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.866e-06 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009125+0000", 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.015e-06 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009136+0000", 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0185e-05 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009143+0000", 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1029999999999996e-06 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089391+0000", 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080247936000000006 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089398+0000", 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089721+0000", 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032323900000000001 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231045 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.0350246, 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076311253999999995, 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013547+0000", 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.624e-06 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013553+0000", 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9710000000000001e-06 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013562+0000", 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7529999999999996e-06 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013570+0000", 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2590000000000003e-06 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089749+0000", 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076179350000000007 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089757+0000", 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8290000000000008e-06 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089856+0000", 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9468000000000004e-05 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231093 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 59.854508547000002, 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082382412000000002, 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:49:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194063+0000", 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4720000000000001e-06 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194068+0000", 2024-04-29T14:49:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.558e-06 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194078+0000", 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9629999999999993e-06 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194085+0000", 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3579999999999999e-06 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276425+0000", 2024-04-29T14:49:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082340872999999995 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276432+0000", 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5309999999999998e-06 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276444+0000", 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1657e-05 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231094 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 59.851102464, 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079185309999999995, 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197469+0000", 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5579999999999999e-06 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197475+0000", 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.241e-06 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197487+0000", 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1905e-05 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197495+0000", 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5099999999999998e-06 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276469+0000", 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078973186000000001 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276474+0000", 2024-04-29T14:49:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7660000000000004e-06 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276653+0000", 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017814400000000001 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231095 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 59.848199061000003, 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076347655, 2024-04-29T14:49:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200372+0000", 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.606e-06 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200378+0000", 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0020000000000003e-06 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200390+0000", 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1772e-05 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200399+0000", 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7050000000000005e-06 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276664+0000", 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076265239999999998 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276668+0000", 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9620000000000004e-06 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276718+0000", 2024-04-29T14:49:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0368000000000003e-05 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231143 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 59.667627142999997, 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076904044000000005, 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380944+0000", 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2550000000000001e-06 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380949+0000", 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8910000000000003e-06 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380958+0000", 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5729999999999997e-06 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380966+0000", 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6469999999999998e-06 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457786+0000", 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076820090999999993 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457792+0000", 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0270000000000001e-06 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457847+0000", 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4559999999999999e-05 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231168 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 59.574861089000002, 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078192824999999994, 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473711+0000", 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4590000000000001e-06 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473719+0000", 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0930000000000003e-06 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473730+0000", 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0621e-05 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473738+0000", 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5310000000000001e-06 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551873+0000", 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078134561000000005 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551883+0000", 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0236e-05 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551901+0000", 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8323999999999999e-05 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232295 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:49:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.006575173000002, 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076427287999999996, 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041996+0000", 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8369999999999999e-06 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042004+0000", 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6930000000000007e-06 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042015+0000", 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1385e-05 2024-04-29T14:49:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042024+0000", 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1200000000000002e-06 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118396+0000", 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076372851000000005 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118406+0000", 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8090000000000004e-06 2024-04-29T14:49:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118422+0000", 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5593000000000001e-05 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232296 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.006421494999998, 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076313084000000003, 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042149+0000", 2024-04-29T14:49:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5799999999999999e-07 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042152+0000", 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.078e-06 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042159+0000", 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3309999999999998e-06 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042164+0000", 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0719999999999999e-06 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118442+0000", 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076277242999999995 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118448+0000", 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5119999999999997e-06 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118461+0000", 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3190000000000001e-05 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178706 7.c 7:34fb7533:::benchmark_data_smithi194_147200_object63794:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.727037473000003, 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093309815000000004, 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:49:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178706 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321534+0000", 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0490000000000002e-06 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321540+0000", 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9490000000000001e-06 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321552+0000", 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1925e-05 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321574+0000", 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2342999999999999e-05 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.332747+0000", 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011172127 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.332889+0000", 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014268700000000001 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.333033+0000", 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000144041 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.336903+0000", 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038697530000000001 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414719+0000", 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077815881000000003 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:49:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414779+0000", 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0061e-05 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414842+0000", 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2998999999999998e-05 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178709 7.c 7:30b10f7c:::benchmark_data_smithi194_147200_object63797:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.718924758, 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085420788999999997, 2024-04-29T14:49:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178709 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329648+0000", 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6699999999999998e-06 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329655+0000", 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0949999999999998e-06 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329672+0000", 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7569000000000001e-05 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329682+0000", 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8660000000000001e-06 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.336948+0000", 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007265886 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.337066+0000", 2024-04-29T14:49:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011782600000000001 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.337188+0000", 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012226199999999999 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414891+0000", 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077703373000000006 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414960+0000", 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8461999999999996e-05 2024-04-29T14:49:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.415006+0000", 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6026999999999998e-05 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.415066+0000", 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9753000000000002e-05 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178745 7.c 7:37c53f67:::benchmark_data_smithi194_147200_object63833:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.622759846000001, 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078737684000000002, 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178745 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425811+0000", 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4419999999999999e-06 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425816+0000", 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5249999999999999e-06 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425827+0000", 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0712e-05 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425833+0000", 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2750000000000001e-06 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432006+0000", 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061731959999999997 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432085+0000", 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8881000000000006e-05 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432164+0000", 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9328999999999999e-05 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:49:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504359+0000", 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.072195230999999999 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504427+0000", 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7816000000000003e-05 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504482+0000", 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4471000000000001e-05 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504548+0000", 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5806e-05 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178796 7.c 7:31d4dc8d:::benchmark_data_smithi194_147200_object63884:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.432816365000001, 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.081790044000000006, 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:49:52.112 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178796 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615755+0000", 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5829999999999999e-06 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615760+0000", 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000001e-06 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615774+0000", 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3879999999999999e-05 2024-04-29T14:49:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615780+0000", 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.105e-06 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619804+0000", 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004023357 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619881+0000", 2024-04-29T14:49:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7272000000000001e-05 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619956+0000", 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5086000000000006e-05 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.623354+0000", 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003398252 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697462+0000", 2024-04-29T14:49:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.074107585000000004 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697500+0000", 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8003999999999997e-05 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697543+0000", 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3529999999999998e-05 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232772 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.487065018999999, 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077453872000000007, 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561506+0000", 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.782e-06 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561515+0000", 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1289999999999996e-06 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561529+0000", 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4644000000000001e-05 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561539+0000", 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5380000000000008e-06 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638939+0000", 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077400374999999993 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638947+0000", 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8569999999999995e-06 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638959+0000", 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1547e-05 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178971 7.c 7:3606b2fb:::benchmark_data_smithi194_147200_object64059:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.221019669, 2024-04-29T14:49:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090073625000000004, 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178971 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827552+0000", 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5999999999999999e-06 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827556+0000", 2024-04-29T14:49:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7849999999999999e-06 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827565+0000", 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7970000000000006e-06 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827571+0000", 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9120000000000003e-06 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.833976+0000", 2024-04-29T14:49:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064046939999999998 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.834060+0000", 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4618999999999997e-05 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.834144+0000", 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3987999999999996e-05 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917478+0000", 2024-04-29T14:49:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.083333661000000003 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917526+0000", 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7812000000000002e-05 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917563+0000", 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6936e-05 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917624+0000", 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0820999999999999e-05 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178972 7.c 7:34cd9e3a:::benchmark_data_smithi194_147200_object64060:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.220221944999999, 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089382420000000004, 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:49:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178972 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828349+0000", 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.257e-06 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828353+0000", 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3440000000000003e-06 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828363+0000", 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0026e-05 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828370+0000", 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9700000000000002e-06 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837577+0000", 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092070030000000001 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837702+0000", 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012509000000000001 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837849+0000", 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014664800000000001 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917659+0000", 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079810324000000002 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917667+0000", 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7069999999999992e-06 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917696+0000", 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8530999999999998e-05 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917730+0000", 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26998.0:178971 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:49:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.211013295999997, 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080073583000000004, 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837557+0000", 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.066e-06 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837562+0000", 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8040000000000001e-06 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837572+0000", 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5019999999999998e-06 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837579+0000", 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1620000000000002e-06 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917515+0000", 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079936129999999994 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917522+0000", 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6059999999999999e-06 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917630+0000", 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108313 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26998.0:178972 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.207784167, 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077106775000000002, 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840787+0000", 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3820000000000001e-06 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840791+0000", 2024-04-29T14:49:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9169999999999999e-06 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840800+0000", 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1640000000000007e-06 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840806+0000", 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.648e-06 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917651+0000", 2024-04-29T14:49:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076845573 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917656+0000", 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4549999999999997e-06 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917892+0000", 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236636 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:49:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:49:52.141 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:49:52.141 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.141 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.142 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:49:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:52.143 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:52.143 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:52.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:52.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:52.271 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:52.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:52.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:52.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:52.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:52.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:52.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:49:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:52 smithi040 ceph-mon[39581]: pgmap v2441: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 184 op/s; 4284/82462 objects misplaced (5.195%); 651 KiB/s, 10 objects/s recovering 2024-04-29T14:49:52.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:52 smithi040 ceph-mon[39581]: osdmap e969: 12 total, 12 up, 10 in 2024-04-29T14:49:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:52 smithi121 ceph-mon[44533]: pgmap v2441: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 184 op/s; 4284/82462 objects misplaced (5.195%); 651 KiB/s, 10 objects/s recovering 2024-04-29T14:49:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:52 smithi121 ceph-mon[44533]: osdmap e969: 12 total, 12 up, 10 in 2024-04-29T14:49:52.935 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:52.936 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:52.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:53.038 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:53.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:49:53.068 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:52 smithi194 ceph-mon[44089]: pgmap v2441: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 184 op/s; 4284/82462 objects misplaced (5.195%); 651 KiB/s, 10 objects/s recovering 2024-04-29T14:49:53.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:52 smithi194 ceph-mon[44089]: osdmap e969: 12 total, 12 up, 10 in 2024-04-29T14:49:53.162 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:49:53.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:49:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:53.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:49:53.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:53.431 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:53.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:53.535 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:53.561 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:53.579 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:53.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:53.683 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:53.710 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:53.729 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:53.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:53.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:53 smithi194 ceph-mon[44089]: from='client.18162 v1:172.21.15.40:0/3636041211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:49:53.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:53.860 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:53.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:53 smithi040 ceph-mon[39581]: from='client.18162 v1:172.21.15.40:0/3636041211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:49:53.879 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:53.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:53.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:53 smithi121 ceph-mon[44533]: from='client.18162 v1:172.21.15.40:0/3636041211' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:49:53.983 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:54.011 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:54.030 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:54.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:54.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:49:54.167 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:54.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:49:54.712 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:54 smithi040 ceph-mon[39581]: pgmap v2443: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 4284/81848 objects misplaced (5.234%); 563 KiB/s, 8 objects/s recovering 2024-04-29T14:49:54.839 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:49:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:49:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:49:54.840 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:49:54.858 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:54 smithi121 ceph-mon[44533]: pgmap v2443: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 4284/81848 objects misplaced (5.234%); 563 KiB/s, 8 objects/s recovering 2024-04-29T14:49:55.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:54 smithi194 ceph-mon[44089]: pgmap v2443: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 4284/81848 objects misplaced (5.234%); 563 KiB/s, 8 objects/s recovering 2024-04-29T14:49:55.095 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2442,"stamp":"2024-04-29T14:49:52.585792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2702508600,"num_objects":41231,"num_object_clones":0,"num_object_copies":82462,"num_objects_missing_on_primary":1735,"num_objects_missing":1735,"num_objects_degraded":0,"num_objects_misplaced":4284,"num_objects_unfound":0,"num_objects_dirty":41231,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":188772,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232017,"num_bytes_recovered":13606485752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":12990292,"kb_used_data":6138084,"kb_used_omap":668,"kb_used_meta":6851427,"kb_avail":924461228,"statfs":{"total":959950356480,"available":946648297472,"internally_reserved":0,"allocated":6285398016,"data_stored":6266216953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7015862255},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-72417280,"num_objects":-1105,"num_object_clones":0,"num_object_copies":-2210,"num_objects_missing_on_primary":-62,"num_objects_missing":-62,"num_objects_degraded":0,"num_objects_misplaced":-123,"num_objects_unfound":0,"num_objects_dirty":-1105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1105,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":3997696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.228307"},"pg_stats":[{"pgid":"7.f","version":"968'11902","reported_seq":46829,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.364498+0000","last_change":"2024-04-29T14:49:26.622778+0000","last_active":"2024-04-29T14:49:50.364498+0000","last_peered":"2024-04-29T14:49:50.364498+0000","last_clean":"2024-04-29T14:49:50.364498+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:49:50.364498+0000","last_undegraded":"2024-04-29T14:49:50.364498+0000","last_fullsized":"2024-04-29T14:49:50.364498+0000","mapping_epoch":918,"log_start":"661'1900","ondisk_log_start":"661'1900","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'11583","last_scrub_stamp":"2024-04-29T14:49:26.622601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:26.622601+0000","objects_scrubbed":2863,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:35.366659+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166592512,"num_objects":2542,"num_object_clones":0,"num_object_copies":5084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"967'19000","reported_seq":40959,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-04-29T14:49:48.961408+0000","last_change":"2024-04-29T14:48:27.585228+0000","last_active":"2024-04-29T14:49:48.961408+0000","last_peered":"2024-04-29T14:49:48.961408+0000","last_clean":"2024-04-29T14:49:48.961408+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:49:48.961408+0000","last_undegraded":"2024-04-29T14:49:48.961408+0000","last_fullsized":"2024-04-29T14:49:48.961408+0000","mapping_epoch":930,"log_start":"704'8958","ondisk_log_start":"704'8958","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'18172","last_scrub_stamp":"2024-04-29T14:47:59.574965+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:47:59.574965+0000","objects_scrubbed":3434,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:53.005360+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"968'11750","reported_seq":42863,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.439464+0000","last_change":"2024-04-29T14:48:44.919582+0000","last_active":"2024-04-29T14:49:50.439464+0000","last_peered":"2024-04-29T14:49:50.439464+0000","last_clean":"2024-04-29T14:49:50.439464+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:49:50.439464+0000","last_undegraded":"2024-04-29T14:49:50.439464+0000","last_fullsized":"2024-04-29T14:49:50.439464+0000","mapping_epoch":938,"log_start":"661'1682","ondisk_log_start":"661'1682","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11104","last_scrub_stamp":"2024-04-29T14:48:44.919467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:44.919467+0000","objects_scrubbed":3222,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:55.396438+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168755224,"num_objects":2576,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11750,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"968'14961","reported_seq":80437,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.429665+0000","last_change":"2024-04-29T14:48:59.043912+0000","last_active":"2024-04-29T14:49:50.429665+0000","last_peered":"2024-04-29T14:49:50.429665+0000","last_clean":"2024-04-29T14:49:50.429665+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:49:50.429665+0000","last_undegraded":"2024-04-29T14:49:50.429665+0000","last_fullsized":"2024-04-29T14:49:50.429665+0000","mapping_epoch":939,"log_start":"672'4950","ondisk_log_start":"672'4950","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"946'14374","last_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:48:59.043790+0000","objects_scrubbed":3160,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:22.421108+0000","scrub_duration":10465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11801,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"968'11717","reported_seq":26549,"reported_epoch":968,"state":"active+recovering","last_fresh":"2024-04-29T14:49:50.848691+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:49:50.848691+0000","last_peered":"2024-04-29T14:49:50.848691+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:49:50.848691+0000","last_undegraded":"2024-04-29T14:49:50.848691+0000","last_fullsized":"2024-04-29T14:49:50.848691+0000","mapping_epoch":942,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168099840,"num_objects":2565,"num_object_clones":0,"num_object_copies":5130,"num_objects_missing_on_primary":1735,"num_objects_missing":1735,"num_objects_degraded":0,"num_objects_misplaced":4284,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11717,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5019,"num_bytes_recovered":328794112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":16},{"shards":"1,10,11","objects":814},{"shards":"10,11","objects":1735}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"968'19158","reported_seq":36603,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.346908+0000","last_change":"2024-04-29T14:49:28.483892+0000","last_active":"2024-04-29T14:49:50.346908+0000","last_peered":"2024-04-29T14:49:50.346908+0000","last_clean":"2024-04-29T14:49:50.346908+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:49:50.346908+0000","last_undegraded":"2024-04-29T14:49:50.346908+0000","last_fullsized":"2024-04-29T14:49:50.346908+0000","mapping_epoch":895,"log_start":"704'9100","ondisk_log_start":"704'9100","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'18861","last_scrub_stamp":"2024-04-29T14:49:28.483744+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:49:28.483744+0000","objects_scrubbed":2889,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:43.869641+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":5180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"968'11791","reported_seq":40924,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.374319+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:49:50.374319+0000","last_peered":"2024-04-29T14:49:50.374319+0000","last_clean":"2024-04-29T14:49:50.374319+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:49:50.374319+0000","last_undegraded":"2024-04-29T14:49:50.374319+0000","last_fullsized":"2024-04-29T14:49:50.374319+0000","mapping_epoch":756,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:56.779052+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167051264,"num_objects":2549,"num_object_clones":0,"num_object_copies":5098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11791,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"968'11693","reported_seq":20718,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.999835+0000","last_change":"2024-04-29T14:48:41.799593+0000","last_active":"2024-04-29T14:49:50.999835+0000","last_peered":"2024-04-29T14:49:50.999835+0000","last_clean":"2024-04-29T14:49:50.999835+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:49:50.999835+0000","last_undegraded":"2024-04-29T14:49:50.999835+0000","last_fullsized":"2024-04-29T14:49:50.999835+0000","mapping_epoch":883,"log_start":"661'1600","ondisk_log_start":"661'1600","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11076","last_scrub_stamp":"2024-04-29T14:48:41.799459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.799459+0000","objects_scrubbed":3214,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:11.502178+0000","scrub_duration":286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170196992,"num_objects":2597,"num_object_clones":0,"num_object_copies":5194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11693,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"968'12038","reported_seq":70322,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.486814+0000","last_change":"2024-04-29T14:49:30.882066+0000","last_active":"2024-04-29T14:49:50.486814+0000","last_peered":"2024-04-29T14:49:50.486814+0000","last_clean":"2024-04-29T14:49:50.486814+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:49:50.486814+0000","last_undegraded":"2024-04-29T14:49:50.486814+0000","last_fullsized":"2024-04-29T14:49:50.486814+0000","mapping_epoch":939,"log_start":"661'1976","ondisk_log_start":"661'1976","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11756","last_scrub_stamp":"2024-04-29T14:49:30.881939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.881939+0000","objects_scrubbed":2972,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:39.012112+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176291840,"num_objects":2690,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12038,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"968'11771","reported_seq":45995,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.281159+0000","last_change":"2024-04-29T14:49:30.401449+0000","last_active":"2024-04-29T14:49:50.281159+0000","last_peered":"2024-04-29T14:49:50.281159+0000","last_clean":"2024-04-29T14:49:50.281159+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:49:50.281159+0000","last_undegraded":"2024-04-29T14:49:50.281159+0000","last_fullsized":"2024-04-29T14:49:50.281159+0000","mapping_epoch":918,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11508","last_scrub_stamp":"2024-04-29T14:49:30.401266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.401266+0000","objects_scrubbed":2766,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:32.430363+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164036608,"num_objects":2503,"num_object_clones":0,"num_object_copies":5006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"967'11914","reported_seq":48228,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-04-29T14:49:48.910933+0000","last_change":"2024-04-29T14:47:55.674071+0000","last_active":"2024-04-29T14:49:48.910933+0000","last_peered":"2024-04-29T14:49:48.910933+0000","last_clean":"2024-04-29T14:49:48.910933+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:49:48.910933+0000","last_undegraded":"2024-04-29T14:49:48.910933+0000","last_fullsized":"2024-04-29T14:49:48.910933+0000","mapping_epoch":915,"log_start":"661'1900","ondisk_log_start":"661'1900","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:49.168576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":5180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"967'15323","reported_seq":40041,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-04-29T14:49:48.059342+0000","last_change":"2024-04-29T14:48:41.280889+0000","last_active":"2024-04-29T14:49:48.059342+0000","last_peered":"2024-04-29T14:49:48.059342+0000","last_clean":"2024-04-29T14:49:48.059342+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:49:48.059342+0000","last_undegraded":"2024-04-29T14:49:48.059342+0000","last_fullsized":"2024-04-29T14:49:48.059342+0000","mapping_epoch":895,"log_start":"673'5225","ondisk_log_start":"673'5225","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'14643","last_scrub_stamp":"2024-04-29T14:48:41.280598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.280598+0000","objects_scrubbed":3364,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:03.594726+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175898624,"num_objects":2684,"num_object_clones":0,"num_object_copies":5368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11724,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"967'11602","reported_seq":36001,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-04-29T14:49:48.077611+0000","last_change":"2024-04-29T14:48:53.235223+0000","last_active":"2024-04-29T14:49:48.077611+0000","last_peered":"2024-04-29T14:49:48.077611+0000","last_clean":"2024-04-29T14:49:48.077611+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:49:48.077611+0000","last_undegraded":"2024-04-29T14:49:48.077611+0000","last_fullsized":"2024-04-29T14:49:48.077611+0000","mapping_epoch":895,"log_start":"661'1544","ondisk_log_start":"661'1544","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'11030","last_scrub_stamp":"2024-04-29T14:48:53.235099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:53.235099+0000","objects_scrubbed":3160,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:53.820864+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"967'11611","reported_seq":58341,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-04-29T14:49:48.800987+0000","last_change":"2024-04-29T14:49:42.611751+0000","last_active":"2024-04-29T14:49:48.800987+0000","last_peered":"2024-04-29T14:49:48.800987+0000","last_clean":"2024-04-29T14:49:48.800987+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:49:48.800987+0000","last_undegraded":"2024-04-29T14:49:48.800987+0000","last_fullsized":"2024-04-29T14:49:48.800987+0000","mapping_epoch":928,"log_start":"661'1600","ondisk_log_start":"661'1600","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'9290","last_scrub_stamp":"2024-04-29T14:44:09.405827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:09.405827+0000","objects_scrubbed":4890,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:49.168597+0000","scrub_duration":526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":5134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11611,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"967'11701","reported_seq":26649,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-04-29T14:49:48.644231+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:49:48.644231+0000","last_peered":"2024-04-29T14:49:48.644231+0000","last_clean":"2024-04-29T14:49:48.644231+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:49:48.644231+0000","last_undegraded":"2024-04-29T14:49:48.644231+0000","last_fullsized":"2024-04-29T14:49:48.644231+0000","mapping_epoch":703,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166658048,"num_objects":2543,"num_object_clones":0,"num_object_copies":5086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"968'11548","reported_seq":32842,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:50.921615+0000","last_change":"2024-04-29T14:48:40.829926+0000","last_active":"2024-04-29T14:49:50.921615+0000","last_peered":"2024-04-29T14:49:50.921615+0000","last_clean":"2024-04-29T14:49:50.921615+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:49:50.921615+0000","last_undegraded":"2024-04-29T14:49:50.921615+0000","last_fullsized":"2024-04-29T14:49:50.921615+0000","mapping_epoch":865,"log_start":"661'1500","ondisk_log_start":"661'1500","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'10915","last_scrub_stamp":"2024-04-29T14:48:40.829814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:40.829814+0000","objects_scrubbed":3125,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163315712,"num_objects":2492,"num_object_clones":0,"num_object_copies":4984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11548,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1913,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-04-29T14:49:49.684706+0000","last_change":"2024-04-29T14:48:42.724622+0000","last_active":"2024-04-29T14:49:49.684706+0000","last_peered":"2024-04-29T14:49:49.684706+0000","last_clean":"2024-04-29T14:49:49.684706+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:49:49.684706+0000","last_undegraded":"2024-04-29T14:49:49.684706+0000","last_fullsized":"2024-04-29T14:49:49.684706+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:48:42.724519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:48:42.724519+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:07.483218+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2701918232,"num_objects":41229,"num_object_clones":0,"num_object_copies":82458,"num_objects_missing_on_primary":1735,"num_objects_missing":1735,"num_objects_degraded":0,"num_objects_misplaced":4284,"num_objects_unfound":0,"num_objects_dirty":41229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188591,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232003,"num_bytes_recovered":13602783256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6701391872,"data_stored":6701383728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122564,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384203,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1909936,"kb_used_data":1026344,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91835216,"statfs":{"total":95995035648,"available":94039261184,"internally_reserved":0,"allocated":1050976256,"data_stored":1049062774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":904788416},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645838,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1484160,"kb_used_data":668596,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92260992,"statfs":{"total":95995035648,"available":94475255808,"internally_reserved":0,"allocated":684642304,"data_stored":682722607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423142,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653104,"kb_used_data":503976,"kb_used_omap":4,"kb_used_meta":149115,"kb_avail":93092048,"statfs":{"total":95995035648,"available":95326257152,"internally_reserved":0,"allocated":516071424,"data_stored":514157962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":152694464},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234519,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1199620,"kb_used_data":329916,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92545532,"statfs":{"total":95995035648,"available":94766624768,"internally_reserved":0,"allocated":337833984,"data_stored":335910735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":890567114},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496908,"kb_used_data":505024,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92248244,"statfs":{"total":95995035648,"available":94462201856,"internally_reserved":0,"allocated":517144576,"data_stored":515217255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1015675334},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575400,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":868420,"kb_used_data":386040,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92876732,"statfs":{"total":95995035648,"available":95105773568,"internally_reserved":0,"allocated":395304960,"data_stored":393381191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738938,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1699516,"kb_used_data":779828,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92045636,"statfs":{"total":95995035648,"available":94254731264,"internally_reserved":0,"allocated":798543872,"data_stored":796624175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497372,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182957,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335800,"kb_used_data":759016,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92409352,"statfs":{"total":95995035648,"available":94627176448,"internally_reserved":0,"allocated":777232384,"data_stored":775318922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":590608840},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986722,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1637816,"kb_used_data":672296,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":92107336,"statfs":{"total":95995035648,"available":94317912064,"internally_reserved":0,"allocated":688431104,"data_stored":686517642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":988674501},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653283,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":705012,"kb_used_data":507048,"kb_used_omap":651,"kb_used_meta":197300,"kb_avail":93040140,"statfs":{"total":95995035648,"available":95273103360,"internally_reserved":0,"allocated":519217152,"data_stored":517303690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":202035976},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":794558464,"data_stored":794558464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":390008832,"data_stored":390004760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":771096576,"data_stored":771096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":684654592,"data_stored":684654592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":511311872,"data_stored":511311872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":511971328,"data_stored":511967256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":333512704,"data_stored":333512704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":510459904,"data_stored":510459904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":679804928,"data_stored":679804928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1046872064,"data_stored":1046872064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:49:55.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:49:55.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:49:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:49:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:55.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:49:55.650 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:49:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:55.677 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:55.697 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:49:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:55.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:49:55.715 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:49:55.757 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:55.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:55.830 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:55.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:56.028 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:49:56.058 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:56.084 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:56.104 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:56.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:56.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2669405500' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:49:56.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:56 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:56.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:49:56.236 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:56.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:49:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:56.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:49:56.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2669405500' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:49:56.280 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:56 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:56.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2669405500' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:49:56.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:56 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:49:56.359 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:56.385 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:56.406 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:56.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:56.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:49:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:56.557 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:49:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:56.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:49:56.661 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:56.687 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:56.706 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:56.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:56.810 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:56.837 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:56.858 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:56.862 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:56.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:57.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2669405500' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:49:57.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:57 smithi194 ceph-mon[44089]: osdmap e970: 12 total, 12 up, 10 in 2024-04-29T14:49:57.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:57 smithi194 ceph-mon[44089]: pgmap v2445: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 254 op/s; 4154/80440 objects misplaced (5.164%); 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:49:57.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2669405500' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:49:57.340 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:57 smithi121 ceph-mon[44533]: osdmap e970: 12 total, 12 up, 10 in 2024-04-29T14:49:57.341 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:57 smithi121 ceph-mon[44533]: pgmap v2445: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 254 op/s; 4154/80440 objects misplaced (5.164%); 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:49:57.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2669405500' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:49:57.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:57 smithi040 ceph-mon[39581]: osdmap e970: 12 total, 12 up, 10 in 2024-04-29T14:49:57.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:57 smithi040 ceph-mon[39581]: pgmap v2445: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 254 op/s; 4154/80440 objects misplaced (5.164%); 1.2 MiB/s, 18 objects/s recovering 2024-04-29T14:49:57.636 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:49:57.636 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:49:57.636 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:49:57.636 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:49:57.636 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:49:58.182 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:49:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:58.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:49:58.288 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:58.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:58 smithi121 ceph-mon[44533]: 7.e scrub starts 2024-04-29T14:49:58.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:58 smithi121 ceph-mon[44533]: osdmap e971: 12 total, 12 up, 10 in 2024-04-29T14:49:58.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:58 smithi194 ceph-mon[44089]: 7.e scrub starts 2024-04-29T14:49:58.312 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:58 smithi194 ceph-mon[44089]: osdmap e971: 12 total, 12 up, 10 in 2024-04-29T14:49:58.314 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:58.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:49:58.333 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:58.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:58.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:58.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:58 smithi040 ceph-mon[39581]: 7.e scrub starts 2024-04-29T14:49:58.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:58 smithi040 ceph-mon[39581]: osdmap e971: 12 total, 12 up, 10 in 2024-04-29T14:49:58.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:58.462 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:58.480 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:58.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:58.584 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:49:58.610 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:58.614 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:49:58.629 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:49:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:58.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:49:58.732 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:49:58.758 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:58.777 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:49:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:58.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:49:58.881 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:49:58.907 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:58.928 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:49:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:58.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:49:59.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:49:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:49:59.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:59 smithi121 ceph-mon[44533]: 7.e scrub ok 2024-04-29T14:49:59.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:49:59 smithi121 ceph-mon[44533]: pgmap v2447: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 4154/80440 objects misplaced (5.164%); 693 KiB/s, 10 objects/s recovering 2024-04-29T14:49:59.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:59 smithi194 ceph-mon[44089]: 7.e scrub ok 2024-04-29T14:49:59.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:49:59 smithi194 ceph-mon[44089]: pgmap v2447: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 4154/80440 objects misplaced (5.164%); 693 KiB/s, 10 objects/s recovering 2024-04-29T14:49:59.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:59 smithi040 ceph-mon[39581]: 7.e scrub ok 2024-04-29T14:49:59.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:49:59 smithi040 ceph-mon[39581]: pgmap v2447: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 4154/80440 objects misplaced (5.164%); 693 KiB/s, 10 objects/s recovering 2024-04-29T14:49:59.536 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:49:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:49:59.671 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:49:59.695 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:49:59.722 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:49:59.757 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:49:59.780 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:49:59.806 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:49:59.813 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:49:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:49:59.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:49:59.827 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:49:59.849 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:49:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:49:59.893 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:49:59.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:49:59.919 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:49:59.945 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:49:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:49:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:49:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:49:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:49:59.959 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: 7.d scrub starts 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: 7.d scrub ok 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1893146551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:00 smithi121 ceph-mon[44533]: overall HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T14:50:00.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: 7.d scrub starts 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: 7.d scrub ok 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1893146551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:00 smithi194 ceph-mon[44089]: overall HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: 7.d scrub starts 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: 7.d scrub ok 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1893146551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:00.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:00 smithi040 ceph-mon[39581]: overall HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T14:50:00.451 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:00.451 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:00.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:00.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2447,"stamp":"2024-04-29T14:49:58.300671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2636251704,"num_objects":40220,"num_object_clones":0,"num_object_copies":80440,"num_objects_missing_on_primary":1670,"num_objects_missing":1670,"num_objects_degraded":0,"num_objects_misplaced":4154,"num_objects_unfound":0,"num_objects_dirty":40220,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":189783,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232082,"num_bytes_recovered":13610745592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160993,"ondisk_log_size":160993,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":12817908,"kb_used_data":5953240,"kb_used_omap":668,"kb_used_meta":6863907,"kb_avail":924633612,"statfs":{"total":959950356480,"available":946824818688,"internally_reserved":0,"allocated":6096117760,"data_stored":6076842672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7028641775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-66256896,"num_objects":-1011,"num_object_clones":0,"num_object_copies":-2022,"num_objects_missing_on_primary":-65,"num_objects_missing":-65,"num_objects_degraded":0,"num_objects_misplaced":-130,"num_objects_unfound":0,"num_objects_dirty":-1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001118"},"pg_stats":[{"pgid":"7.f","version":"969'11976","reported_seq":46906,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:55.145726+0000","last_change":"2024-04-29T14:49:26.622778+0000","last_active":"2024-04-29T14:49:55.145726+0000","last_peered":"2024-04-29T14:49:55.145726+0000","last_clean":"2024-04-29T14:49:55.145726+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:49:55.145726+0000","last_undegraded":"2024-04-29T14:49:55.145726+0000","last_fullsized":"2024-04-29T14:49:55.145726+0000","mapping_epoch":918,"log_start":"661'1900","ondisk_log_start":"661'1900","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'11583","last_scrub_stamp":"2024-04-29T14:49:26.622601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:26.622601+0000","objects_scrubbed":2863,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:35.366659+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":4936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"969'19068","reported_seq":41031,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:54.123743+0000","last_change":"2024-04-29T14:48:27.585228+0000","last_active":"2024-04-29T14:49:54.123743+0000","last_peered":"2024-04-29T14:49:54.123743+0000","last_clean":"2024-04-29T14:49:54.123743+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:49:54.123743+0000","last_undegraded":"2024-04-29T14:49:54.123743+0000","last_fullsized":"2024-04-29T14:49:54.123743+0000","mapping_epoch":930,"log_start":"704'9058","ondisk_log_start":"704'9058","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'18172","last_scrub_stamp":"2024-04-29T14:47:59.574965+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:47:59.574965+0000","objects_scrubbed":3434,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330368,"num_objects":2538,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"969'11810","reported_seq":42925,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:55.186917+0000","last_change":"2024-04-29T14:48:44.919582+0000","last_active":"2024-04-29T14:49:55.186917+0000","last_peered":"2024-04-29T14:49:55.186917+0000","last_clean":"2024-04-29T14:49:55.186917+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:49:55.186917+0000","last_undegraded":"2024-04-29T14:49:55.186917+0000","last_fullsized":"2024-04-29T14:49:55.186917+0000","mapping_epoch":938,"log_start":"661'1782","ondisk_log_start":"661'1782","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11104","last_scrub_stamp":"2024-04-29T14:48:44.919467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:44.919467+0000","objects_scrubbed":3222,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:55.396438+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164823064,"num_objects":2516,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11810,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"969'15026","reported_seq":80504,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:55.206637+0000","last_change":"2024-04-29T14:48:59.043912+0000","last_active":"2024-04-29T14:49:55.206637+0000","last_peered":"2024-04-29T14:49:55.206637+0000","last_clean":"2024-04-29T14:49:55.206637+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:49:55.206637+0000","last_undegraded":"2024-04-29T14:49:55.206637+0000","last_fullsized":"2024-04-29T14:49:55.206637+0000","mapping_epoch":939,"log_start":"672'4950","ondisk_log_start":"672'4950","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"946'14374","last_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:48:59.043790+0000","objects_scrubbed":3160,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:22.421108+0000","scrub_duration":10465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162660352,"num_objects":2482,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11866,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"969'11780","reported_seq":26809,"reported_epoch":969,"state":"active+recovering","last_fresh":"2024-04-29T14:49:55.902820+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:49:55.902820+0000","last_peered":"2024-04-29T14:49:55.902820+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:49:55.902820+0000","last_undegraded":"2024-04-29T14:49:55.902820+0000","last_fullsized":"2024-04-29T14:49:55.902820+0000","mapping_epoch":942,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163971072,"num_objects":2502,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":1670,"num_objects_missing":1670,"num_objects_degraded":0,"num_objects_misplaced":4154,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11780,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5084,"num_bytes_recovered":333053952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":18},{"shards":"1,10,11","objects":814},{"shards":"10,11","objects":1670}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"969'19226","reported_seq":36673,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:55.159232+0000","last_change":"2024-04-29T14:49:28.483892+0000","last_active":"2024-04-29T14:49:55.159232+0000","last_peered":"2024-04-29T14:49:55.159232+0000","last_clean":"2024-04-29T14:49:55.159232+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:49:55.159232+0000","last_undegraded":"2024-04-29T14:49:55.159232+0000","last_fullsized":"2024-04-29T14:49:55.159232+0000","mapping_epoch":895,"log_start":"704'9200","ondisk_log_start":"704'9200","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'18861","last_scrub_stamp":"2024-04-29T14:49:28.483744+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:49:28.483744+0000","objects_scrubbed":2889,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:43.869641+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165281792,"num_objects":2522,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12152,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"969'11855","reported_seq":40990,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:55.587493+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:49:55.587493+0000","last_peered":"2024-04-29T14:49:55.587493+0000","last_clean":"2024-04-29T14:49:55.587493+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:49:55.587493+0000","last_undegraded":"2024-04-29T14:49:55.587493+0000","last_fullsized":"2024-04-29T14:49:55.587493+0000","mapping_epoch":756,"log_start":"661'1800","ondisk_log_start":"661'1800","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:56.779052+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162856960,"num_objects":2485,"num_object_clones":0,"num_object_copies":4970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11855,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"970'11757","reported_seq":20786,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-29T14:49:56.144850+0000","last_change":"2024-04-29T14:48:41.799593+0000","last_active":"2024-04-29T14:49:56.144850+0000","last_peered":"2024-04-29T14:49:56.144850+0000","last_clean":"2024-04-29T14:49:56.144850+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:49:56.144850+0000","last_undegraded":"2024-04-29T14:49:56.144850+0000","last_fullsized":"2024-04-29T14:49:56.144850+0000","mapping_epoch":883,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11076","last_scrub_stamp":"2024-04-29T14:48:41.799459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.799459+0000","objects_scrubbed":3214,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:11.502178+0000","scrub_duration":286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":5066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"969'12104","reported_seq":70390,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:55.174627+0000","last_change":"2024-04-29T14:49:30.882066+0000","last_active":"2024-04-29T14:49:55.174627+0000","last_peered":"2024-04-29T14:49:55.174627+0000","last_clean":"2024-04-29T14:49:55.174627+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:49:55.174627+0000","last_undegraded":"2024-04-29T14:49:55.174627+0000","last_fullsized":"2024-04-29T14:49:55.174627+0000","mapping_epoch":939,"log_start":"662'2076","ondisk_log_start":"662'2076","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11756","last_scrub_stamp":"2024-04-29T14:49:30.881939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.881939+0000","objects_scrubbed":2972,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:39.012112+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":2624,"num_object_clones":0,"num_object_copies":5248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"969'11836","reported_seq":46062,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:55.088912+0000","last_change":"2024-04-29T14:49:30.401449+0000","last_active":"2024-04-29T14:49:55.088912+0000","last_peered":"2024-04-29T14:49:55.088912+0000","last_clean":"2024-04-29T14:49:55.088912+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:49:55.088912+0000","last_undegraded":"2024-04-29T14:49:55.088912+0000","last_fullsized":"2024-04-29T14:49:55.088912+0000","mapping_epoch":918,"log_start":"661'1800","ondisk_log_start":"661'1800","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11508","last_scrub_stamp":"2024-04-29T14:49:30.401266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.401266+0000","objects_scrubbed":2766,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:32.430363+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159776768,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"969'11984","reported_seq":48302,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:54.204101+0000","last_change":"2024-04-29T14:47:55.674071+0000","last_active":"2024-04-29T14:49:54.204101+0000","last_peered":"2024-04-29T14:49:54.204101+0000","last_clean":"2024-04-29T14:49:54.204101+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:49:54.204101+0000","last_undegraded":"2024-04-29T14:49:54.204101+0000","last_fullsized":"2024-04-29T14:49:54.204101+0000","mapping_epoch":915,"log_start":"661'1900","ondisk_log_start":"661'1900","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"969'15388","reported_seq":40110,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:53.264054+0000","last_change":"2024-04-29T14:48:41.280889+0000","last_active":"2024-04-29T14:49:53.264054+0000","last_peered":"2024-04-29T14:49:53.264054+0000","last_clean":"2024-04-29T14:49:53.264054+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:49:53.264054+0000","last_undegraded":"2024-04-29T14:49:53.264054+0000","last_fullsized":"2024-04-29T14:49:53.264054+0000","mapping_epoch":895,"log_start":"673'5325","ondisk_log_start":"673'5325","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'14643","last_scrub_stamp":"2024-04-29T14:48:41.280598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.280598+0000","objects_scrubbed":3364,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:03.594726+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171638784,"num_objects":2619,"num_object_clones":0,"num_object_copies":5238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11789,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"969'11656","reported_seq":36059,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:53.309777+0000","last_change":"2024-04-29T14:48:53.235223+0000","last_active":"2024-04-29T14:49:53.309777+0000","last_peered":"2024-04-29T14:49:53.309777+0000","last_clean":"2024-04-29T14:49:53.309777+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:49:53.309777+0000","last_undegraded":"2024-04-29T14:49:53.309777+0000","last_fullsized":"2024-04-29T14:49:53.309777+0000","mapping_epoch":895,"log_start":"661'1644","ondisk_log_start":"661'1644","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'11030","last_scrub_stamp":"2024-04-29T14:48:53.235099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:53.235099+0000","objects_scrubbed":3160,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:49:53.820864+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166068224,"num_objects":2534,"num_object_clones":0,"num_object_copies":5068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11656,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"969'11665","reported_seq":58399,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:54.182837+0000","last_change":"2024-04-29T14:49:42.611751+0000","last_active":"2024-04-29T14:49:54.182837+0000","last_peered":"2024-04-29T14:49:54.182837+0000","last_clean":"2024-04-29T14:49:54.182837+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:49:54.182837+0000","last_undegraded":"2024-04-29T14:49:54.182837+0000","last_fullsized":"2024-04-29T14:49:54.182837+0000","mapping_epoch":928,"log_start":"661'1600","ondisk_log_start":"661'1600","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'9290","last_scrub_stamp":"2024-04-29T14:44:09.405827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:44:09.405827+0000","objects_scrubbed":4890,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11665,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"969'11765","reported_seq":26717,"reported_epoch":969,"state":"active+clean","last_fresh":"2024-04-29T14:49:53.480921+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:49:53.480921+0000","last_peered":"2024-04-29T14:49:53.480921+0000","last_clean":"2024-04-29T14:49:53.480921+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:49:53.480921+0000","last_undegraded":"2024-04-29T14:49:53.480921+0000","last_fullsized":"2024-04-29T14:49:53.480921+0000","mapping_epoch":703,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162463744,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11765,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"970'11595","reported_seq":32893,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-29T14:49:56.077258+0000","last_change":"2024-04-29T14:48:40.829926+0000","last_active":"2024-04-29T14:49:56.077258+0000","last_peered":"2024-04-29T14:49:56.077258+0000","last_clean":"2024-04-29T14:49:56.077258+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:49:56.077258+0000","last_undegraded":"2024-04-29T14:49:56.077258+0000","last_fullsized":"2024-04-29T14:49:56.077258+0000","mapping_epoch":865,"log_start":"661'1500","ondisk_log_start":"661'1500","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'10915","last_scrub_stamp":"2024-04-29T14:48:40.829814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:40.829814+0000","objects_scrubbed":3125,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:01.021335+0000","scrub_duration":292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160235520,"num_objects":2445,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1917,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-29T14:49:56.077217+0000","last_change":"2024-04-29T14:48:42.724622+0000","last_active":"2024-04-29T14:49:56.077217+0000","last_peered":"2024-04-29T14:49:56.077217+0000","last_clean":"2024-04-29T14:49:56.077217+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:49:56.077217+0000","last_undegraded":"2024-04-29T14:49:56.077217+0000","last_fullsized":"2024-04-29T14:49:56.077217+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:48:42.724519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:48:42.724519+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:07.483218+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2635661336,"num_objects":40218,"num_object_clones":0,"num_object_copies":80436,"num_objects_missing_on_primary":1670,"num_objects_missing":1670,"num_objects_degraded":0,"num_objects_misplaced":4154,"num_objects_unfound":0,"num_objects_dirty":40218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189602,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232068,"num_bytes_recovered":13607043096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6507732992,"data_stored":6507724848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122565,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384204,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1894220,"kb_used_data":1006464,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91850932,"statfs":{"total":95995035648,"available":94055354368,"internally_reserved":0,"allocated":1030619136,"data_stored":1028693134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":909048256},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645839,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455176,"kb_used_data":639616,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92289976,"statfs":{"total":95995035648,"available":94504935424,"internally_reserved":0,"allocated":654966784,"data_stored":653040802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423143,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":644616,"kb_used_data":491328,"kb_used_omap":4,"kb_used_meta":153275,"kb_avail":93100536,"statfs":{"total":95995035648,"available":95334948864,"internally_reserved":0,"allocated":503119872,"data_stored":501193890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":156954304},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234520,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192784,"kb_used_data":323080,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92552368,"statfs":{"total":95995035648,"available":94773624832,"internally_reserved":0,"allocated":330833920,"data_stored":328904386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":890567114},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1483860,"kb_used_data":491980,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92261292,"statfs":{"total":95995035648,"available":94475563008,"internally_reserved":0,"allocated":503787520,"data_stored":501853914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1015675334},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575401,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860044,"kb_used_data":377668,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92885108,"statfs":{"total":95995035648,"available":95114350592,"internally_reserved":0,"allocated":386732032,"data_stored":384801978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1669000,"kb_used_data":745152,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92076152,"statfs":{"total":95995035648,"available":94285979648,"internally_reserved":0,"allocated":763035648,"data_stored":761109666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497373,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182958,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1311820,"kb_used_data":735040,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92433332,"statfs":{"total":95995035648,"available":94651731968,"internally_reserved":0,"allocated":752680960,"data_stored":750754978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":590608840},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986723,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1624716,"kb_used_data":659200,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":92120436,"statfs":{"total":95995035648,"available":94331326464,"internally_reserved":0,"allocated":675020800,"data_stored":673094818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":988674501},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653284,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681672,"kb_used_data":483712,"kb_used_omap":651,"kb_used_meta":197300,"kb_avail":93063480,"statfs":{"total":95995035648,"available":95297003520,"internally_reserved":0,"allocated":495321088,"data_stored":493395106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":202035976},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":758841344,"data_stored":758841344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":382013440,"data_stored":382009368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":745472000,"data_stored":745472000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":668336128,"data_stored":668336128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":487194624,"data_stored":487194624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":499453952,"data_stored":499449880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":326172672,"data_stored":326172672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":496762880,"data_stored":496762880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":651165696,"data_stored":651165696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1025179648,"data_stored":1025179648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:50:00.668 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.b is not active+clean 2024-04-29T14:50:00.668 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.b', 'version': "969'11780", 'reported_seq': 26809, 'reported_epoch': 969, 'state': 'active+recovering', 'last_fresh': '2024-04-29T14:49:55.902820+0000', 'last_change': '2024-04-29T14:48:55.372453+0000', 'last_active': '2024-04-29T14:49:55.902820+0000', 'last_peered': '2024-04-29T14:49:55.902820+0000', 'last_clean': '2024-04-29T14:35:46.185312+0000', 'last_became_active': '2024-04-29T14:48:48.169551+0000', 'last_became_peered': '2024-04-29T14:48:48.169551+0000', 'last_unstale': '2024-04-29T14:49:55.902820+0000', 'last_undegraded': '2024-04-29T14:49:55.902820+0000', 'last_fullsized': '2024-04-29T14:49:55.902820+0000', 'mapping_epoch': 942, 'log_start': "661'1700", 'ondisk_log_start': "661'1700", 'created': 655, 'last_epoch_clean': 656, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_clean_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 163971072, 'num_objects': 2502, 'num_object_clones': 0, 'num_object_copies': 5004, 'num_objects_missing_on_primary': 1670, 'num_objects_missing': 1670, 'num_objects_degraded': 0, 'num_objects_misplaced': 4154, 'num_objects_unfound': 0, 'num_objects_dirty': 2502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11780, 'num_write_kb': 457024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5084, 'num_bytes_recovered': 333053952, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': ['4', '11'], 'object_location_counts': [{'shards': '1,5', 'objects': 18}, {'shards': '1,10,11', 'objects': 814}, {'shards': '10,11', 'objects': 1670}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-29T14:50:00.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:50:00.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:00.778 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:01.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:50:01.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:01.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: from='client.18174 v1:172.21.15.40:0/2511040586' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:01 smithi194 ceph-mon[44089]: pgmap v2448: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 184 op/s; 4154/79608 objects misplaced (5.218%); 539 KiB/s, 8 objects/s recovering 2024-04-29T14:50:01.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:01.360 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:01.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: from='client.18174 v1:172.21.15.40:0/2511040586' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:01.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:01 smithi040 ceph-mon[39581]: pgmap v2448: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 184 op/s; 4154/79608 objects misplaced (5.218%); 539 KiB/s, 8 objects/s recovering 2024-04-29T14:50:01.380 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:01.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: from='client.18174 v1:172.21.15.40:0/2511040586' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:01 smithi121 ceph-mon[44533]: pgmap v2448: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 184 op/s; 4154/79608 objects misplaced (5.218%); 539 KiB/s, 8 objects/s recovering 2024-04-29T14:50:01.484 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:01.491 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:01.511 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:01.762 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:01.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:01.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:01.893 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:01.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:01.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:02.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:02.045 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:02.104 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:02.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:02.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:02.236 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:02.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:02.358 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:02.386 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:02.406 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:02.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:02.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:02.540 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:02.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:03.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:03.122 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:03.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:03.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:03.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:03.154 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:03.154 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:03.154 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:03 smithi194 ceph-mon[44089]: osdmap e972: 12 total, 12 up, 10 in 2024-04-29T14:50:03.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:03 smithi194 ceph-mon[44089]: 7.4 scrub starts 2024-04-29T14:50:03.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:03 smithi194 ceph-mon[44089]: pgmap v2450: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 196 op/s; 4025/78192 objects misplaced (5.148%); 652 KiB/s, 10 objects/s recovering 2024-04-29T14:50:03.155 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:03 smithi194 ceph-mon[44089]: osdmap e973: 12 total, 12 up, 10 in 2024-04-29T14:50:03.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:50:03.245 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:03.274 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:03.326 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:03.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:03.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:03.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:03.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:03 smithi040 ceph-mon[39581]: osdmap e972: 12 total, 12 up, 10 in 2024-04-29T14:50:03.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:03 smithi040 ceph-mon[39581]: 7.4 scrub starts 2024-04-29T14:50:03.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:03 smithi040 ceph-mon[39581]: pgmap v2450: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 196 op/s; 4025/78192 objects misplaced (5.148%); 652 KiB/s, 10 objects/s recovering 2024-04-29T14:50:03.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:03 smithi040 ceph-mon[39581]: osdmap e973: 12 total, 12 up, 10 in 2024-04-29T14:50:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:03 smithi121 ceph-mon[44533]: osdmap e972: 12 total, 12 up, 10 in 2024-04-29T14:50:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:03 smithi121 ceph-mon[44533]: 7.4 scrub starts 2024-04-29T14:50:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:03 smithi121 ceph-mon[44533]: pgmap v2450: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 196 op/s; 4025/78192 objects misplaced (5.148%); 652 KiB/s, 10 objects/s recovering 2024-04-29T14:50:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:03 smithi121 ceph-mon[44533]: osdmap e973: 12 total, 12 up, 10 in 2024-04-29T14:50:03.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:03.456 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:03.471 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:03.471 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:03.479 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:03.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:03.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:03.609 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:03.629 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:03.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:03.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:04.005 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:04.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:04.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:04.149 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:04.238 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:04 smithi194 ceph-mon[44089]: 7.4 scrub ok 2024-04-29T14:50:04.238 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:04.238 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:04.264 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:50:04.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:04 smithi040 ceph-mon[39581]: 7.4 scrub ok 2024-04-29T14:50:04.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:04.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:04.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:04 smithi121 ceph-mon[44533]: 7.4 scrub ok 2024-04-29T14:50:04.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:04.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:04.486 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:50:04.486 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2878,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":974,"num_osds":12,"num_up_osds":12,"osd_up_since":1714401063,"num_in_osds":10,"osd_in_since":1714402012,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":39096,"data_bytes":2562589240,"bytes_used":12922036224,"bytes_avail":947028320256,"bytes_total":959950356480,"misplaced_objects":4025,"misplaced_total":78192,"misplaced_ratio":0.051475854307346018,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":667829,"recovering_keys_per_sec":0,"num_objects_recovered":64,"num_bytes_recovered":4194304,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":72,"modified":"2024-04-29T14:50:02.301403+0000","services":{}},"progress_events":{}} 2024-04-29T14:50:04.501 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:04.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:04.532 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:04.609 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:04.636 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:04.835 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:04.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:04.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:04.976 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:05.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:05 smithi194 ceph-mon[44089]: osdmap e974: 12 total, 12 up, 10 in 2024-04-29T14:50:05.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:05 smithi194 ceph-mon[44089]: pgmap v2453: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 242 op/s; 4025/77526 objects misplaced (5.192%); 683 KiB/s, 10 objects/s recovering 2024-04-29T14:50:05.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/533529432' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:50:05.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:05 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:50:05.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:05 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:50:05.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:05 smithi040 ceph-mon[39581]: osdmap e974: 12 total, 12 up, 10 in 2024-04-29T14:50:05.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:05 smithi040 ceph-mon[39581]: pgmap v2453: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 242 op/s; 4025/77526 objects misplaced (5.192%); 683 KiB/s, 10 objects/s recovering 2024-04-29T14:50:05.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/533529432' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:50:05.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:05 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:50:05.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:05 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:50:05.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:05 smithi121 ceph-mon[44533]: osdmap e974: 12 total, 12 up, 10 in 2024-04-29T14:50:05.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:05 smithi121 ceph-mon[44533]: pgmap v2453: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 242 op/s; 4025/77526 objects misplaced (5.192%); 683 KiB/s, 10 objects/s recovering 2024-04-29T14:50:05.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/533529432' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:50:05.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:05 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:50:05.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:05 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:50:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:05.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:05.531 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:05.559 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:05.586 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:05.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:05.696 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:05.723 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:05.921 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:05.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:05.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:06.030 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:06.057 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:06.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:06 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:50:06.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:06 smithi194 ceph-mon[44089]: 7.1 scrub starts 2024-04-29T14:50:06.288 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:06 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:50:06.305 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:06.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:06.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:06 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:50:06.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:06 smithi040 ceph-mon[39581]: 7.1 scrub starts 2024-04-29T14:50:06.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:06 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:50:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:06 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:50:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:06 smithi121 ceph-mon[44533]: 7.1 scrub starts 2024-04-29T14:50:06.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:06 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:50:06.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:06.482 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:06.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:06.568 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:06.568 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:06.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:06.618 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:06.907 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:50:06.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:50:07.085 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:50:07.206 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:07 smithi040 ceph-mon[39581]: 7.1 scrub ok 2024-04-29T14:50:07.206 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:07 smithi040 ceph-mon[39581]: pgmap v2454: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 286 op/s; 3893/76174 objects misplaced (5.111%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:50:07.206 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1032375670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:07.206 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:07 smithi040 ceph-mon[39581]: from='client.37664 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:07.230 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:07 smithi194 ceph-mon[44089]: 7.1 scrub ok 2024-04-29T14:50:07.230 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:07 smithi194 ceph-mon[44089]: pgmap v2454: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 286 op/s; 3893/76174 objects misplaced (5.111%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:50:07.230 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1032375670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:07.230 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:07 smithi194 ceph-mon[44089]: from='client.37664 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:07.255 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:50:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:07 smithi121 ceph-mon[44533]: 7.1 scrub ok 2024-04-29T14:50:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:07 smithi121 ceph-mon[44533]: pgmap v2454: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 286 op/s; 3893/76174 objects misplaced (5.111%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:50:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1032375670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:07 smithi121 ceph-mon[44533]: from='client.37664 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:07.521 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:07.670 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:07.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:07.708 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:07.723 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:50:07.750 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:50:07.772 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:50:07.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:07.797 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:50:07.807 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:07.818 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:50:07.833 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:07.841 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:50:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:07.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:07.864 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:50:07.885 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:50:07.907 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:50:07.927 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:50:07.945 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:07.949 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:50:07.972 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:07.980 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:50:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:50:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:50:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:50:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:50:07.992 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:07.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:08.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: from='client.37664 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: osdmap e975: 12 total, 12 up, 10 in 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1044413281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.259 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:50:08.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:50:08.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: from='client.37664 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:08.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: osdmap e975: 12 total, 12 up, 10 in 2024-04-29T14:50:08.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1044413281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:08.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.474 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:08.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: from='client.37664 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: osdmap e975: 12 total, 12 up, 10 in 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1044413281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:08.579 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:08.606 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:08.955 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:08.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:09.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:09.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.082 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.082 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.082 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:50:09.082 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: pgmap v2456: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 3893/76174 objects misplaced (5.111%); 715 KiB/s, 11 objects/s recovering 2024-04-29T14:50:09.082 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:09 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:50:09.092 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:09.216 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.216 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.216 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.216 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.216 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.216 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.217 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.217 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:50:09.217 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: pgmap v2456: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 3893/76174 objects misplaced (5.111%); 715 KiB/s, 11 objects/s recovering 2024-04-29T14:50:09.217 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:09 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: pgmap v2456: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 3893/76174 objects misplaced (5.111%); 715 KiB/s, 11 objects/s recovering 2024-04-29T14:50:09.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:09 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(7.f 854/838 [PushOp(7:f87996ed:::benchmark_data_smithi194_147200_object98293:head, version: 675'6231, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:f87996ed:::benchmark_data_smithi194_147200_object98293:head@675'6231, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.064341+0000", 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 272.50032386300001, 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21117528599999999, 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.570 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064341+0000", 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064341+0000", 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064343+0000", 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.939e-06 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064372+0000", 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8362000000000001e-05 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064386+0000", 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4302e-05 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064398+0000", 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1934e-05 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064458+0000", 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0507999999999997e-05 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.064466+0000", 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2309999999999996e-06 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275517+0000", 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21105001000000001 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:159332 7.a 7:54b0231b:::benchmark_data_smithi194_147200_object44420:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.065002+0000", 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 272.499662978, 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21070470599999999, 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.574 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 159332 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065002+0000", 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065002+0000", 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065004+0000", 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2330000000000001e-06 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065010+0000", 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6280000000000002e-06 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065020+0000", 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9830000000000008e-06 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065026+0000", 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.891e-06 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065070+0000", 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3835999999999998e-05 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065156+0000", 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6051000000000007e-05 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 2", 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065231+0000", 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5686999999999995e-05 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275516+0000", 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21028424200000001 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275616+0000", 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000100148 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275665+0000", 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9237000000000003e-05 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275707+0000", 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1770000000000002e-05 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(7.5 854/838 [PushReplyOp(7:aa6deea7:::benchmark_data_smithi194_147200_object86433:head)])", 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 272.49917053799999, 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20996125199999999, 2024-04-29T14:50:09.579 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.580 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065495+0000", 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.75e-07 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065499+0000", 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4309999999999998e-06 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065505+0000", 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.066e-06 2024-04-29T14:50:09.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065509+0000", 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3699999999999997e-06 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.065547+0000", 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.799e-05 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275456+0000", 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20990882 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.26998.0:159332 7.a e854/655 ondisk, result = 0)", 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:37.069827+0000", 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 272.49483854499999, 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.205886185, 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069827+0000", 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069827+0000", 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069830+0000", 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7299999999999999e-06 2024-04-29T14:50:09.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.069844+0000", 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4127e-05 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275400+0000", 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20555575000000001 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275424+0000", 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3536999999999998e-05 2024-04-29T14:50:09.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275582+0000", 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015807000000000001 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275611+0000", 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8892000000000001e-05 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:37.275713+0000", 2024-04-29T14:50:09.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000102079 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(7.a 878/877 [PushReplyOp(7:50de555b:::benchmark_data_smithi194_147200_object95615:head)])", 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 211.05286943199999, 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060137652999999999, 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511796+0000", 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1e-07 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511798+0000", 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2010000000000002e-06 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511802+0000", 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3960000000000002e-06 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511807+0000", 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2070000000000003e-06 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.511848+0000", 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1576000000000003e-05 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:38.571933+0000", 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060084962999999998 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPushReply(7.a 878/877 [PushReplyOp(7:51a99d8f:::benchmark_data_smithi194_147200_object50259:head)])", 2024-04-29T14:50:09.590 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:39.167086+0000", 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 210.397578737, 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.058698567, 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167086+0000", 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167086+0000", 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167087+0000", 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2500000000000001e-07 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167089+0000", 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5220000000000002e-06 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167093+0000", 2024-04-29T14:50:09.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2069999999999997e-06 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167099+0000", 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4020000000000001e-06 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.167127+0000", 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8045999999999999e-05 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.225785+0000", 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.058658065000000002 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173693 7.5 7.b0c6acd5 (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:49.867097+0000", 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 139.69756768799999, 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.69354318699999995, 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:09.594 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173693 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867097+0000", 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867097+0000", 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867099+0000", 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5239999999999999e-06 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867104+0000", 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0749999999999997e-06 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867114+0000", 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6539999999999994e-06 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867121+0000", 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0559999999999999e-06 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867167+0000", 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6641000000000003e-05 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.867174+0000", 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7379999999999997e-06 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.560610+0000", 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.69343556799999995 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.560641+0000", 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0930999999999999e-05 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.598 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173708 7.5 7.4eb6f6b5 (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.314364+0000", 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 139.25030126499999, 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24635837799999999, 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173708 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314364+0000", 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314364+0000", 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314367+0000", 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6960000000000001e-06 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314373+0000", 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8800000000000002e-06 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314386+0000", 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2393e-05 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314392+0000", 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.567e-06 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314450+0000", 2024-04-29T14:50:09.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7287999999999998e-05 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.314459+0000", 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.098e-06 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.560705+0000", 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24624658899999999 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.560722+0000", 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6867e-05 2024-04-29T14:50:09.602 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173693 7.5 7:ab35630d:::benchmark_data_smithi194_147200_object58781:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e912)", 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.582442+0000", 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.98222344800001, 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47723301699999998, 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173693 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.603 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582442+0000", 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582442+0000", 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582444+0000", 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5459999999999998e-06 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582450+0000", 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5620000000000003e-06 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582460+0000", 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0198e-05 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582469+0000", 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0650000000000005e-06 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582527+0000", 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7812000000000001e-05 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582534+0000", 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.81e-06 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.045009+0000", 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.462475056 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.045121+0000", 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011233199999999999 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.045261+0000", 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000139863 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.049626+0000", 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043647520000000004 2024-04-29T14:50:09.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059238+0000", 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096119480000000004 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059496+0000", 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00025855099999999998 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059561+0000", 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5024000000000006e-05 2024-04-29T14:50:09.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059675+0000", 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000113498 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173708 7.5 7:ad6f6d72:::benchmark_data_smithi194_147200_object58796:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e912)", 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.582488+0000", 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.98217685399999, 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47751283300000003, 2024-04-29T14:50:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173708 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582488+0000", 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582488+0000", 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582489+0000", 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3499999999999995e-07 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582492+0000", 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.004e-06 2024-04-29T14:50:09.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582498+0000", 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1070000000000001e-06 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582504+0000", 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5979999999999996e-06 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582559+0000", 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5804999999999998e-05 2024-04-29T14:50:09.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582565+0000", 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.366e-06 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.049675+0000", 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.46711033099999999 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.049831+0000", 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000155678 2024-04-29T14:50:09.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.049988+0000", 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015763100000000001 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059441+0000", 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094523890000000003 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059690+0000", 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000249225 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059825+0000", 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013531599999999999 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059889+0000", 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3731000000000004e-05 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060001+0000", 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011191700000000001 2024-04-29T14:50:09.615 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173739 7.5 7:a957b7eb:::benchmark_data_smithi194_147200_object58827:head [delete] snapc 0=[] ondisk+write+known_if_redirected e912)", 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.582511+0000", 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.98215368800001, 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47788343, 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.616 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173739 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582511+0000", 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582511+0000", 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582512+0000", 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2500000000000001e-07 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582514+0000", 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.221e-06 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582518+0000", 2024-04-29T14:50:09.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5470000000000001e-06 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582521+0000", 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0029999999999999e-06 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582587+0000", 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6031999999999994e-05 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582590+0000", 2024-04-29T14:50:09.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6839999999999998e-06 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.055037+0000", 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47244699000000001 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.055184+0000", 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000146847 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.055337+0000", 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015242700000000001 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.059757+0000", 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044207960000000003 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060015+0000", 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00025778599999999997 2024-04-29T14:50:09.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060234+0000", 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021838399999999999 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060293+0000", 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9953e-05 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060395+0000", 2024-04-29T14:50:09.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101335 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173742 7.5 7:a48b4613:::benchmark_data_smithi194_147200_object58830:head [delete] snapc 0=[] ondisk+write+known_if_redirected e912)", 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.582524+0000", 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 137.98214077099999, 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.48779032, 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173742 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582524+0000", 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582524+0000", 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582525+0000", 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9700000000000003e-07 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582526+0000", 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3349999999999999e-06 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582538+0000", 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1854e-05 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582541+0000", 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2729999999999998e-06 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582604+0000", 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2985999999999996e-05 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.582608+0000", 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4829999999999999e-06 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060446+0000", 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47783848899999998 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060622+0000", 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017569300000000001 2024-04-29T14:50:09.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.060783+0000", 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016156499999999999 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.070138+0000", 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093546879999999999 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.070156+0000", 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8168e-05 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.070201+0000", 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4845999999999997e-05 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.070241+0000", 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9719999999999999e-05 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.070315+0000", 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3923000000000004e-05 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.629 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:174594 7.5 7:ae31a2a5:::benchmark_data_smithi194_147200_object59682:head [delete] snapc 0=[] ondisk+write+known_if_redirected e916)", 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.377667+0000", 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.18699760800001, 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27275365899999998, 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174594 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.630 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.377667+0000", 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.377667+0000", 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.377669+0000", 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3659999999999999e-06 2024-04-29T14:50:09.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.377674+0000", 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7729999999999999e-06 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.377683+0000", 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.064e-06 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.377688+0000", 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4480000000000002e-06 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.504598+0000", 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12691022699999999 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.504687+0000", 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8333999999999998e-05 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.504789+0000", 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000102741 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.649950+0000", 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.145160286 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650003+0000", 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3712999999999999e-05 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650320+0000", 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000316352 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650360+0000", 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0210999999999998e-05 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650421+0000", 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1143999999999996e-05 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:174603 7.5 7:a5df9a12:::benchmark_data_smithi194_147200_object59691:head [delete] snapc 0=[] ondisk+write+known_if_redirected e916)", 2024-04-29T14:50:09.635 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.413533+0000", 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.15113160300001, 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23706266400000001, 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174603 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.413533+0000", 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.413533+0000", 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.413535+0000", 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.497e-06 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.413540+0000", 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0239999999999997e-06 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.413549+0000", 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.918e-06 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.413554+0000", 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5589999999999996e-06 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.508904+0000", 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095350022000000006 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.509012+0000", 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107286 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.509104+0000", 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2697999999999999e-05 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650226+0000", 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.14112181900000001 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650285+0000", 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8387999999999997e-05 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650461+0000", 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017662400000000001 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650498+0000", 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7109e-05 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650596+0000", 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7720000000000006e-05 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.26998.0:174594 7.5 e916/911 ondisk, result = 0)", 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.512438+0000", 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.05222668900001, 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13757185299999999, 2024-04-29T14:50:09.641 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512438+0000", 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512438+0000", 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512440+0000", 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2410000000000001e-06 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512447+0000", 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.956e-06 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512456+0000", 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7100000000000002e-06 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512463+0000", 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4300000000000003e-06 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.649992+0000", 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13752952500000001 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.649999+0000", 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9809999999999997e-06 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650010+0000", 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.101e-05 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.26998.0:174603 7.5 e916/911 ondisk, result = 0)", 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.512547+0000", 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.05211825000001, 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13774310400000001, 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.645 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512547+0000", 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512547+0000", 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512547+0000", 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.89e-07 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512549+0000", 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0710000000000002e-06 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512555+0000", 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8490000000000002e-06 2024-04-29T14:50:09.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.512559+0000", 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5899999999999999e-06 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650275+0000", 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13771589000000001 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650281+0000", 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2639999999999997e-06 2024-04-29T14:50:09.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650290+0000", 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9509999999999995e-06 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.26998.0:174594 7.5 e916/911 ondisk, result = 0)", 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.514726+0000", 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.049939462, 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13570085400000001, 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.649 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514726+0000", 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514726+0000", 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514727+0000", 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9800000000000002e-07 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514730+0000", 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5939999999999998e-06 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514739+0000", 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2959999999999999e-06 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514743+0000", 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3819999999999997e-06 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650312+0000", 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13556923100000001 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650317+0000", 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4100000000000001e-06 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650426+0000", 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109943 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop_reply(client.26998.0:174603 7.5 e916/911 ondisk, result = 0)", 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.514801+0000", 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.049864007, 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13580228599999999, 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514801+0000", 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514801+0000", 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514802+0000", 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.06e-07 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514804+0000", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3779999999999999e-06 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514808+0000", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.245e-06 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.514812+0000", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3830000000000001e-06 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650454+0000", 2024-04-29T14:50:09.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13564209999999999 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650458+0000", 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3209999999999998e-06 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650603+0000", 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000145253 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:174609 7.5 7:ad5aa30d:::benchmark_data_smithi194_147200_object59697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e916)", 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.524360+0000", 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.040305383, 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.13650556999999999, 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174609 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.524360+0000", 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.524360+0000", 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.524362+0000", 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0200000000000001e-06 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.524368+0000", 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8690000000000001e-06 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.524379+0000", 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1342000000000001e-05 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.524386+0000", 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8650000000000003e-06 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650734+0000", 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12634847699999999 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650844+0000", 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109436 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.650971+0000", 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000127534 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.659242+0000", 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082706399999999992 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.659261+0000", 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9011000000000001e-05 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.660676+0000", 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014150219999999999 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.660748+0000", 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2569000000000003e-05 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.660865+0000", 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116785 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:174610 7.5 7:af6b2a57:::benchmark_data_smithi194_147200_object59698:head [delete] snapc 0=[] ondisk+write+known_if_redirected e916)", 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.546710+0000", 2024-04-29T14:50:09.662 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 132.017955495, 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.119222964, 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174610 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.546710+0000", 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.546710+0000", 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.546713+0000", 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.01e-06 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.546726+0000", 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2916e-05 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.546741+0000", 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5401e-05 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.546751+0000", 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7599999999999997e-06 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.654791+0000", 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.108040499 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.654903+0000", 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011170100000000001 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,1", 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.654986+0000", 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3326999999999997e-05 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.659306+0000", 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043197180000000002 2024-04-29T14:50:09.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.659324+0000", 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8048000000000001e-05 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.665844+0000", 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065197459999999999 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.665877+0000", 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3272999999999998e-05 2024-04-29T14:50:09.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.665933+0000", 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5565000000000001e-05 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:09.668 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:50:10.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:50:10.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:50:10.577 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:11.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:11 smithi194 ceph-mon[44089]: pgmap v2457: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 187 op/s; 3893/75296 objects misplaced (5.170%); 556 KiB/s, 8 objects/s recovering 2024-04-29T14:50:11.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:11 smithi194 ceph-mon[44089]: 7.3 scrub starts 2024-04-29T14:50:11.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:11 smithi194 ceph-mon[44089]: 7.3 scrub ok 2024-04-29T14:50:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:11 smithi040 ceph-mon[39581]: pgmap v2457: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 187 op/s; 3893/75296 objects misplaced (5.170%); 556 KiB/s, 8 objects/s recovering 2024-04-29T14:50:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:11 smithi040 ceph-mon[39581]: 7.3 scrub starts 2024-04-29T14:50:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:11 smithi040 ceph-mon[39581]: 7.3 scrub ok 2024-04-29T14:50:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:11 smithi121 ceph-mon[44533]: pgmap v2457: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 187 op/s; 3893/75296 objects misplaced (5.170%); 556 KiB/s, 8 objects/s recovering 2024-04-29T14:50:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:11 smithi121 ceph-mon[44533]: 7.3 scrub starts 2024-04-29T14:50:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:11 smithi121 ceph-mon[44533]: 7.3 scrub ok 2024-04-29T14:50:11.746 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:11.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:11.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:12.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:12.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:12.371 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:12.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:12.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:12.562 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:12.571 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:12.571 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:50:12.571 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:50:12.571 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:50:12.571 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:50:12.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:12.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:12.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:12.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:12.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:12.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:12.712 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:13.065 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:13.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:13.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:13.196 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:13.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:13.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:13.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:13.347 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:13.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:13.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:13.424 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:50:13.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:50:13.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:13 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:50:13.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:13 smithi121 ceph-mon[44533]: pgmap v2458: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 227 op/s; 3755/74040 objects misplaced (5.072%); 1.0 MiB/s, 16 objects/s recovering 2024-04-29T14:50:13.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:13 smithi121 ceph-mon[44533]: osdmap e976: 12 total, 12 up, 10 in 2024-04-29T14:50:13.498 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:13 smithi121 ceph-mon[44533]: osdmap e977: 12 total, 12 up, 10 in 2024-04-29T14:50:13.500 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:13.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:50:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:13.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:50:13.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:13.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:13 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:50:13.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:13 smithi040 ceph-mon[39581]: pgmap v2458: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 227 op/s; 3755/74040 objects misplaced (5.072%); 1.0 MiB/s, 16 objects/s recovering 2024-04-29T14:50:13.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:13 smithi040 ceph-mon[39581]: osdmap e976: 12 total, 12 up, 10 in 2024-04-29T14:50:13.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:13 smithi040 ceph-mon[39581]: osdmap e977: 12 total, 12 up, 10 in 2024-04-29T14:50:13.650 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:13.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:13.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:13.692 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:13.766 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:50:13.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:13.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:13 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:50:13.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:13 smithi194 ceph-mon[44089]: pgmap v2458: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 227 op/s; 3755/74040 objects misplaced (5.072%); 1.0 MiB/s, 16 objects/s recovering 2024-04-29T14:50:13.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:13 smithi194 ceph-mon[44089]: osdmap e976: 12 total, 12 up, 10 in 2024-04-29T14:50:13.799 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:13 smithi194 ceph-mon[44089]: osdmap e977: 12 total, 12 up, 10 in 2024-04-29T14:50:13.802 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:50:13.858 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:50:13.951 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:13.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:14.057 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:14.088 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:14.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:14 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:50:14.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:14 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:50:14.784 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2460,"stamp":"2024-04-29T14:50:12.589671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2426536504,"num_objects":37020,"num_object_clones":0,"num_object_copies":74040,"num_objects_missing_on_primary":1471,"num_objects_missing":1471,"num_objects_degraded":0,"num_objects_misplaced":3755,"num_objects_unfound":0,"num_objects_dirty":37020,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":192983,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232282,"num_bytes_recovered":13623852792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161093,"ondisk_log_size":161093,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":12250304,"kb_used_data":5398120,"kb_used_omap":668,"kb_used_meta":6851427,"kb_avail":925201216,"statfs":{"total":959950356480,"available":947406045184,"internally_reserved":0,"allocated":5527674880,"data_stored":5508023692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7015862255},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-69926912,"num_objects":-1067,"num_object_clones":0,"num_object_copies":-2134,"num_objects_missing_on_primary":-69,"num_objects_missing":-69,"num_objects_degraded":0,"num_objects_misplaced":-138,"num_objects_unfound":0,"num_objects_dirty":-1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.287651"},"pg_stats":[{"pgid":"7.f","version":"975'12175","reported_seq":47117,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:10.243324+0000","last_change":"2024-04-29T14:49:26.622778+0000","last_active":"2024-04-29T14:50:10.243324+0000","last_peered":"2024-04-29T14:50:10.243324+0000","last_clean":"2024-04-29T14:50:10.243324+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:50:10.243324+0000","last_undegraded":"2024-04-29T14:50:10.243324+0000","last_fullsized":"2024-04-29T14:50:10.243324+0000","mapping_epoch":918,"log_start":"662'2100","ondisk_log_start":"662'2100","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'11583","last_scrub_stamp":"2024-04-29T14:49:26.622601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:26.622601+0000","objects_scrubbed":2863,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:35.366659+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148701184,"num_objects":2269,"num_object_clones":0,"num_object_copies":4538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12175,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"975'19280","reported_seq":41287,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:09.214007+0000","last_change":"2024-04-29T14:49:57.142920+0000","last_active":"2024-04-29T14:50:09.214007+0000","last_peered":"2024-04-29T14:50:09.214007+0000","last_clean":"2024-04-29T14:50:09.214007+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:50:09.214007+0000","last_undegraded":"2024-04-29T14:50:09.214007+0000","last_fullsized":"2024-04-29T14:50:09.214007+0000","mapping_epoch":930,"log_start":"704'9258","ondisk_log_start":"704'9258","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'19114","last_scrub_stamp":"2024-04-29T14:49:57.142771+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:49:57.142771+0000","objects_scrubbed":2497,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:23.319557+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152436736,"num_objects":2326,"num_object_clones":0,"num_object_copies":4652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"975'12011","reported_seq":43170,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:09.930670+0000","last_change":"2024-04-29T14:49:58.542110+0000","last_active":"2024-04-29T14:50:09.930670+0000","last_peered":"2024-04-29T14:50:09.930670+0000","last_clean":"2024-04-29T14:50:09.930670+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:50:09.930670+0000","last_undegraded":"2024-04-29T14:50:09.930670+0000","last_fullsized":"2024-04-29T14:50:09.930670+0000","mapping_epoch":938,"log_start":"662'1982","ondisk_log_start":"662'1982","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'11857","last_scrub_stamp":"2024-04-29T14:49:58.541853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:58.541853+0000","objects_scrubbed":2470,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:16.374459+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151650328,"num_objects":2315,"num_object_clones":0,"num_object_copies":4630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12011,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"975'15222","reported_seq":80712,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:09.587589+0000","last_change":"2024-04-29T14:48:59.043912+0000","last_active":"2024-04-29T14:50:09.587589+0000","last_peered":"2024-04-29T14:50:09.587589+0000","last_clean":"2024-04-29T14:50:09.587589+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:50:09.587589+0000","last_undegraded":"2024-04-29T14:50:09.587589+0000","last_fullsized":"2024-04-29T14:50:09.587589+0000","mapping_epoch":939,"log_start":"673'5150","ondisk_log_start":"673'5150","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"946'14374","last_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:48:59.043790+0000","objects_scrubbed":3160,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:22.421108+0000","scrub_duration":10465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149815296,"num_objects":2286,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12062,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"975'11981","reported_seq":27620,"reported_epoch":975,"state":"active+recovering","last_fresh":"2024-04-29T14:50:10.995708+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:50:10.995708+0000","last_peered":"2024-04-29T14:50:10.995708+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:50:10.995708+0000","last_undegraded":"2024-04-29T14:50:10.995708+0000","last_fullsized":"2024-04-29T14:50:10.995708+0000","mapping_epoch":942,"log_start":"661'1900","ondisk_log_start":"661'1900","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150798336,"num_objects":2301,"num_object_clones":0,"num_object_copies":4602,"num_objects_missing_on_primary":1471,"num_objects_missing":1471,"num_objects_degraded":0,"num_objects_misplaced":3755,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5284,"num_bytes_recovered":346161152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":17},{"shards":"1,10,11","objects":813},{"shards":"10,11","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"975'19405","reported_seq":36864,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:10.270564+0000","last_change":"2024-04-29T14:49:28.483892+0000","last_active":"2024-04-29T14:50:10.270564+0000","last_peered":"2024-04-29T14:50:10.270564+0000","last_clean":"2024-04-29T14:50:10.270564+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:50:10.270564+0000","last_undegraded":"2024-04-29T14:50:10.270564+0000","last_fullsized":"2024-04-29T14:50:10.270564+0000","mapping_epoch":895,"log_start":"704'9400","ondisk_log_start":"704'9400","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'18861","last_scrub_stamp":"2024-04-29T14:49:28.483744+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:49:28.483744+0000","objects_scrubbed":2889,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:43.869641+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153550848,"num_objects":2343,"num_object_clones":0,"num_object_copies":4686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12331,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"975'12057","reported_seq":41204,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:10.998157+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:50:10.998157+0000","last_peered":"2024-04-29T14:50:10.998157+0000","last_clean":"2024-04-29T14:50:10.998157+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:50:10.998157+0000","last_undegraded":"2024-04-29T14:50:10.998157+0000","last_fullsized":"2024-04-29T14:50:10.998157+0000","mapping_epoch":756,"log_start":"662'2000","ondisk_log_start":"662'2000","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149618688,"num_objects":2283,"num_object_clones":0,"num_object_copies":4566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12057,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"975'11946","reported_seq":20985,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:11.095830+0000","last_change":"2024-04-29T14:48:41.799593+0000","last_active":"2024-04-29T14:50:11.095830+0000","last_peered":"2024-04-29T14:50:11.095830+0000","last_clean":"2024-04-29T14:50:11.095830+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:50:11.095830+0000","last_undegraded":"2024-04-29T14:50:11.095830+0000","last_fullsized":"2024-04-29T14:50:11.095830+0000","mapping_epoch":883,"log_start":"661'1900","ondisk_log_start":"661'1900","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11076","last_scrub_stamp":"2024-04-29T14:48:41.799459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.799459+0000","objects_scrubbed":3214,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:11.502178+0000","scrub_duration":286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153616384,"num_objects":2344,"num_object_clones":0,"num_object_copies":4688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11946,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"975'12324","reported_seq":70622,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:10.395122+0000","last_change":"2024-04-29T14:49:30.882066+0000","last_active":"2024-04-29T14:50:10.395122+0000","last_peered":"2024-04-29T14:50:10.395122+0000","last_clean":"2024-04-29T14:50:10.395122+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:50:10.395122+0000","last_undegraded":"2024-04-29T14:50:10.395122+0000","last_fullsized":"2024-04-29T14:50:10.395122+0000","mapping_epoch":939,"log_start":"662'2276","ondisk_log_start":"662'2276","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11756","last_scrub_stamp":"2024-04-29T14:49:30.881939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.881939+0000","objects_scrubbed":2972,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:39.012112+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157548544,"num_objects":2404,"num_object_clones":0,"num_object_copies":4808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12324,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"975'11996","reported_seq":46234,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:09.658911+0000","last_change":"2024-04-29T14:49:30.401449+0000","last_active":"2024-04-29T14:50:09.658911+0000","last_peered":"2024-04-29T14:50:09.658911+0000","last_clean":"2024-04-29T14:50:09.658911+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:50:09.658911+0000","last_undegraded":"2024-04-29T14:50:09.658911+0000","last_fullsized":"2024-04-29T14:50:09.658911+0000","mapping_epoch":918,"log_start":"661'1900","ondisk_log_start":"661'1900","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11508","last_scrub_stamp":"2024-04-29T14:49:30.401266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.401266+0000","objects_scrubbed":2766,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:32.430363+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149291008,"num_objects":2278,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11996,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"975'12186","reported_seq":48548,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:09.086033+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:50:09.086033+0000","last_peered":"2024-04-29T14:50:09.086033+0000","last_clean":"2024-04-29T14:50:09.086033+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:50:09.086033+0000","last_undegraded":"2024-04-29T14:50:09.086033+0000","last_fullsized":"2024-04-29T14:50:09.086033+0000","mapping_epoch":915,"log_start":"662'2100","ondisk_log_start":"662'2100","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151912448,"num_objects":2318,"num_object_clones":0,"num_object_copies":4636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12186,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"975'15599","reported_seq":40333,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:08.498028+0000","last_change":"2024-04-29T14:48:41.280889+0000","last_active":"2024-04-29T14:50:08.498028+0000","last_peered":"2024-04-29T14:50:08.498028+0000","last_clean":"2024-04-29T14:50:08.498028+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:50:08.498028+0000","last_undegraded":"2024-04-29T14:50:08.498028+0000","last_fullsized":"2024-04-29T14:50:08.498028+0000","mapping_epoch":895,"log_start":"675'5525","ondisk_log_start":"675'5525","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'14643","last_scrub_stamp":"2024-04-29T14:48:41.280598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:41.280598+0000","objects_scrubbed":3364,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157810688,"num_objects":2408,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"975'11864","reported_seq":36310,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:08.457822+0000","last_change":"2024-04-29T14:50:04.932139+0000","last_active":"2024-04-29T14:50:08.457822+0000","last_peered":"2024-04-29T14:50:08.457822+0000","last_clean":"2024-04-29T14:50:08.457822+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:50:08.457822+0000","last_undegraded":"2024-04-29T14:50:08.457822+0000","last_fullsized":"2024-04-29T14:50:08.457822+0000","mapping_epoch":895,"log_start":"661'1844","ondisk_log_start":"661'1844","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11810","last_scrub_stamp":"2024-04-29T14:50:04.932001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:04.932001+0000","objects_scrubbed":2381,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:25.466879+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152436736,"num_objects":2326,"num_object_clones":0,"num_object_copies":4652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11864,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"975'11896","reported_seq":58673,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:09.015580+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:50:09.015580+0000","last_peered":"2024-04-29T14:50:09.015580+0000","last_clean":"2024-04-29T14:50:09.015580+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:50:09.015580+0000","last_undegraded":"2024-04-29T14:50:09.015580+0000","last_fullsized":"2024-04-29T14:50:09.015580+0000","mapping_epoch":928,"log_start":"661'1800","ondisk_log_start":"661'1800","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149553152,"num_objects":2282,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11896,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"975'11954","reported_seq":26919,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:08.431036+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:50:08.431036+0000","last_peered":"2024-04-29T14:50:08.431036+0000","last_clean":"2024-04-29T14:50:08.431036+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:50:08.431036+0000","last_undegraded":"2024-04-29T14:50:08.431036+0000","last_fullsized":"2024-04-29T14:50:08.431036+0000","mapping_epoch":703,"log_start":"662'1900","ondisk_log_start":"662'1900","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150077440,"num_objects":2290,"num_object_clones":0,"num_object_copies":4580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"975'11795","reported_seq":33134,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:11.133188+0000","last_change":"2024-04-29T14:50:05.324932+0000","last_active":"2024-04-29T14:50:11.133188+0000","last_peered":"2024-04-29T14:50:11.133188+0000","last_clean":"2024-04-29T14:50:11.133188+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:50:11.133188+0000","last_undegraded":"2024-04-29T14:50:11.133188+0000","last_fullsized":"2024-04-29T14:50:11.133188+0000","mapping_epoch":865,"log_start":"661'1700","ondisk_log_start":"661'1700","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11707","last_scrub_stamp":"2024-04-29T14:50:05.324827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.324827+0000","objects_scrubbed":2336,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:05.726927+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147128320,"num_objects":2245,"num_object_clones":0,"num_object_copies":4490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1935,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-04-29T14:50:08.327288+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:50:08.327288+0000","last_peered":"2024-04-29T14:50:08.327288+0000","last_clean":"2024-04-29T14:50:08.327288+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:50:08.327288+0000","last_undegraded":"2024-04-29T14:50:08.327288+0000","last_fullsized":"2024-04-29T14:50:08.327288+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2425946136,"num_objects":37018,"num_object_clones":0,"num_object_copies":74036,"num_objects_missing_on_primary":1471,"num_objects_missing":1471,"num_objects_degraded":0,"num_objects_misplaced":3755,"num_objects_unfound":0,"num_objects_dirty":37018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192802,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232268,"num_bytes_recovered":13620150296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5939863552,"data_stored":5939855408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122568,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384207,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1824724,"kb_used_data":936968,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91920428,"statfs":{"total":95995035648,"available":94126518272,"internally_reserved":0,"allocated":959455232,"data_stored":957491620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":909048256},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645842,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414096,"kb_used_data":594376,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92331056,"statfs":{"total":95995035648,"available":94547001344,"internally_reserved":0,"allocated":608641024,"data_stored":606677432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":839383488},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423146,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587216,"kb_used_data":450568,"kb_used_omap":4,"kb_used_meta":136635,"kb_avail":93157936,"statfs":{"total":95995035648,"available":95393726464,"internally_reserved":0,"allocated":461381632,"data_stored":459418040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":139914944},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234523,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1155288,"kb_used_data":298064,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92589864,"statfs":{"total":95995035648,"available":94812020736,"internally_reserved":0,"allocated":305217536,"data_stored":303250392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":877787594},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528864,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1444828,"kb_used_data":452948,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92300324,"statfs":{"total":95995035648,"available":94515531776,"internally_reserved":0,"allocated":463818752,"data_stored":461847536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1015675334},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575404,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":838484,"kb_used_data":351948,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92906668,"statfs":{"total":95995035648,"available":95136428032,"internally_reserved":0,"allocated":360394752,"data_stored":358427088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738942,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1563728,"kb_used_data":639880,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92181424,"statfs":{"total":95995035648,"available":94393778176,"internally_reserved":0,"allocated":655237120,"data_stored":653273528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497376,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182961,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1236624,"kb_used_data":655688,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92508528,"statfs":{"total":95995035648,"available":94728732672,"internally_reserved":0,"allocated":671424512,"data_stored":669460920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":594868680},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986726,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573140,"kb_used_data":603464,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92172012,"statfs":{"total":95995035648,"available":94384140288,"internally_reserved":0,"allocated":617947136,"data_stored":615983544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653287,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":612176,"kb_used_data":414216,"kb_used_omap":651,"kb_used_meta":197300,"kb_avail":93132976,"statfs":{"total":95995035648,"available":95368167424,"internally_reserved":0,"allocated":424157184,"data_stored":422193592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":202035976},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":650575872,"data_stored":650575872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":354947072,"data_stored":354943000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":662175744,"data_stored":662175744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":613679104,"data_stored":613679104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":417792000,"data_stored":417792000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":459018240,"data_stored":459014168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":300744704,"data_stored":300744704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":456589312,"data_stored":456589312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":603389952,"data_stored":603389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":953810944,"data_stored":953810944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:50:14.788 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:14.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:14 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:15.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:15.145 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:15.174 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:15.374 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:15.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:15.479 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:15.507 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:15.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:15 smithi040 ceph-mon[39581]: from='client.18204 v1:172.21.15.40:0/3981180856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:15.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:15 smithi040 ceph-mon[39581]: pgmap v2461: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 3755/73482 objects misplaced (5.110%); 611 KiB/s, 9 objects/s recovering 2024-04-29T14:50:15.576 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:15 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:15.576 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:15 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:15.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:15 smithi121 ceph-mon[44533]: from='client.18204 v1:172.21.15.40:0/3981180856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:15.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:15 smithi121 ceph-mon[44533]: pgmap v2461: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 3755/73482 objects misplaced (5.110%); 611 KiB/s, 9 objects/s recovering 2024-04-29T14:50:15.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:15 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:15.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:15 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:15.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:15 smithi194 ceph-mon[44089]: from='client.18204 v1:172.21.15.40:0/3981180856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:15.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:15 smithi194 ceph-mon[44089]: pgmap v2461: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 186 op/s; 3755/73482 objects misplaced (5.110%); 611 KiB/s, 9 objects/s recovering 2024-04-29T14:50:15.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:15 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:15.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:15 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:15.649 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:15.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:15.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:15.781 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:15.802 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:15.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:15.906 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:15.936 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:15.981 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:50:15.981 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:50:15.981 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:50:15.981 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:50:15.981 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:50:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:15.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:16.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:16.122 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:16.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:16.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:16.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:16.283 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:16.394 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:50:16.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:16 smithi194 ceph-mon[44089]: osdmap e978: 12 total, 12 up, 10 in 2024-04-29T14:50:16.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1176825413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:16.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:16 smithi194 ceph-mon[44089]: from='client.27931 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:16.789 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:50:16.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:16 smithi040 ceph-mon[39581]: osdmap e978: 12 total, 12 up, 10 in 2024-04-29T14:50:16.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1176825413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:16.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:16 smithi040 ceph-mon[39581]: from='client.27931 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:16.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:16 smithi121 ceph-mon[44533]: osdmap e978: 12 total, 12 up, 10 in 2024-04-29T14:50:16.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1176825413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:16.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:16 smithi121 ceph-mon[44533]: from='client.27931 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:17.059 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:17.144 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:50:17.163 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:50:17.182 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:50:17.202 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:50:17.223 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:50:17.242 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:50:17.262 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:50:17.282 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:50:17.302 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:50:17.323 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:50:17.346 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:50:17.370 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:50:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:50:17.372 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:50:17.544 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:17 smithi194 ceph-mon[44089]: pgmap v2463: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 255 op/s; 3611/72226 objects misplaced (5.000%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:50:17.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:17 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:17.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:17 smithi194 ceph-mon[44089]: from='client.27931 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:50:17.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:17 smithi194 ceph-mon[44089]: osdmap e979: 12 total, 12 up, 10 in 2024-04-29T14:50:17.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2707983668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:17.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:17 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:17.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:17 smithi040 ceph-mon[39581]: pgmap v2463: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 255 op/s; 3611/72226 objects misplaced (5.000%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:50:17.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:17 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:17.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:17 smithi040 ceph-mon[39581]: from='client.27931 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:50:17.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:17 smithi040 ceph-mon[39581]: osdmap e979: 12 total, 12 up, 10 in 2024-04-29T14:50:17.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2707983668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:17.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:17 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:17.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:50:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:17 smithi121 ceph-mon[44533]: pgmap v2463: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 255 op/s; 3611/72226 objects misplaced (5.000%); 1.4 MiB/s, 23 objects/s recovering 2024-04-29T14:50:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:17 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:17 smithi121 ceph-mon[44533]: from='client.27931 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:50:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:17 smithi121 ceph-mon[44533]: osdmap e979: 12 total, 12 up, 10 in 2024-04-29T14:50:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2707983668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:17.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:17 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:18.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:18.293 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:18.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:18.370 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:18.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:18.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:18.494 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:18.522 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:18.645 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:18.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:18.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:18.967 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:18.967 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:19.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:19 smithi040 ceph-mon[39581]: pgmap v2465: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 3611/72226 objects misplaced (5.000%); 754 KiB/s, 11 objects/s recovering 2024-04-29T14:50:19.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:19.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:19.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:19.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:19 smithi194 ceph-mon[44089]: pgmap v2465: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 3611/72226 objects misplaced (5.000%); 754 KiB/s, 11 objects/s recovering 2024-04-29T14:50:19.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:19.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:19.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:50:19.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:19.778 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:19.833 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:50:19.881 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:19.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:19 smithi121 ceph-mon[44533]: pgmap v2465: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 3611/72226 objects misplaced (5.000%); 754 KiB/s, 11 objects/s recovering 2024-04-29T14:50:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:19.986 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:19.997 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:20.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:50:20.254 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:20.263 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:20.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:20.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:20.395 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:20.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:20.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:20.451 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:50:20.453 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:50:20.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:20.551 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:20 smithi194 ceph-mon[44089]: osdmap e980: 12 total, 12 up, 10 in 2024-04-29T14:50:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:20 smithi194 ceph-mon[44089]: pgmap v2467: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 170 op/s; 3611/71432 objects misplaced (5.055%); 747 KiB/s, 11 objects/s recovering 2024-04-29T14:50:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:20 smithi194 ceph-mon[44089]: from='client.27943 v1:172.21.15.40:0/513780696' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:20.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:20 smithi040 ceph-mon[39581]: osdmap e980: 12 total, 12 up, 10 in 2024-04-29T14:50:20.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:20 smithi040 ceph-mon[39581]: pgmap v2467: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 170 op/s; 3611/71432 objects misplaced (5.055%); 747 KiB/s, 11 objects/s recovering 2024-04-29T14:50:20.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:20 smithi040 ceph-mon[39581]: from='client.27943 v1:172.21.15.40:0/513780696' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:20.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:20 smithi121 ceph-mon[44533]: osdmap e980: 12 total, 12 up, 10 in 2024-04-29T14:50:20.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:20 smithi121 ceph-mon[44533]: pgmap v2467: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 170 op/s; 3611/71432 objects misplaced (5.055%); 747 KiB/s, 11 objects/s recovering 2024-04-29T14:50:20.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:20 smithi121 ceph-mon[44533]: from='client.27943 v1:172.21.15.40:0/513780696' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:21.416 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2467,"stamp":"2024-04-29T14:50:20.304550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2341077560,"num_objects":35716,"num_object_clones":0,"num_object_copies":71432,"num_objects_missing_on_primary":1397,"num_objects_missing":1397,"num_objects_degraded":0,"num_objects_misplaced":3611,"num_objects_unfound":0,"num_objects_dirty":35716,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":194287,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232352,"num_bytes_recovered":13628440312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":12004256,"kb_used_data":5152068,"kb_used_omap":668,"kb_used_meta":6851427,"kb_avail":925447264,"statfs":{"total":959950356480,"available":947657998336,"internally_reserved":0,"allocated":5275717632,"data_stored":5255884929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7015862255},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67174400,"num_objects":-1025,"num_object_clones":0,"num_object_copies":-2050,"num_objects_missing_on_primary":-74,"num_objects_missing":-74,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":-1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":4587520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001124"},"pg_stats":[{"pgid":"7.f","version":"977'12234","reported_seq":47182,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.379540+0000","last_change":"2024-04-29T14:49:26.622778+0000","last_active":"2024-04-29T14:50:15.379540+0000","last_peered":"2024-04-29T14:50:15.379540+0000","last_clean":"2024-04-29T14:50:15.379540+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:50:15.379540+0000","last_undegraded":"2024-04-29T14:50:15.379540+0000","last_fullsized":"2024-04-29T14:50:15.379540+0000","mapping_epoch":918,"log_start":"662'2200","ondisk_log_start":"662'2200","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'11583","last_scrub_stamp":"2024-04-29T14:49:26.622601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:26.622601+0000","objects_scrubbed":2863,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:35.366659+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144834560,"num_objects":2210,"num_object_clones":0,"num_object_copies":4420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"979'19390","reported_seq":41405,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-04-29T14:50:19.308495+0000","last_change":"2024-04-29T14:49:57.142920+0000","last_active":"2024-04-29T14:50:19.308495+0000","last_peered":"2024-04-29T14:50:19.308495+0000","last_clean":"2024-04-29T14:50:19.308495+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:50:19.308495+0000","last_undegraded":"2024-04-29T14:50:19.308495+0000","last_fullsized":"2024-04-29T14:50:19.308495+0000","mapping_epoch":930,"log_start":"704'9358","ondisk_log_start":"704'9358","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'19114","last_scrub_stamp":"2024-04-29T14:49:57.142771+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:49:57.142771+0000","objects_scrubbed":2497,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:23.319557+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145227776,"num_objects":2216,"num_object_clones":0,"num_object_copies":4432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12334,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"978'12072","reported_seq":43237,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.389980+0000","last_change":"2024-04-29T14:49:58.542110+0000","last_active":"2024-04-29T14:50:15.389980+0000","last_peered":"2024-04-29T14:50:15.389980+0000","last_clean":"2024-04-29T14:50:15.389980+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:50:15.389980+0000","last_undegraded":"2024-04-29T14:50:15.389980+0000","last_fullsized":"2024-04-29T14:50:15.389980+0000","mapping_epoch":938,"log_start":"662'1982","ondisk_log_start":"662'1982","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'11857","last_scrub_stamp":"2024-04-29T14:49:58.541853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:58.541853+0000","objects_scrubbed":2470,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:16.374459+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652632,"num_objects":2254,"num_object_clones":0,"num_object_copies":4508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12072,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"978'15267","reported_seq":80763,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.523102+0000","last_change":"2024-04-29T14:48:59.043912+0000","last_active":"2024-04-29T14:50:15.523102+0000","last_peered":"2024-04-29T14:50:15.523102+0000","last_clean":"2024-04-29T14:50:15.523102+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:50:15.523102+0000","last_undegraded":"2024-04-29T14:50:15.523102+0000","last_fullsized":"2024-04-29T14:50:15.523102+0000","mapping_epoch":939,"log_start":"673'5250","ondisk_log_start":"673'5250","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"946'14374","last_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:48:59.043790+0000","objects_scrubbed":3160,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:22.421108+0000","scrub_duration":10465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146866176,"num_objects":2241,"num_object_clones":0,"num_object_copies":4482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"978'12052","reported_seq":27912,"reported_epoch":978,"state":"active+recovering","last_fresh":"2024-04-29T14:50:16.112200+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:50:16.112200+0000","last_peered":"2024-04-29T14:50:16.112200+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:50:16.112200+0000","last_undegraded":"2024-04-29T14:50:16.112200+0000","last_fullsized":"2024-04-29T14:50:16.112200+0000","mapping_epoch":942,"log_start":"662'2000","ondisk_log_start":"662'2000","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146145280,"num_objects":2230,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":1397,"num_objects_missing":1397,"num_objects_degraded":0,"num_objects_misplaced":3611,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12052,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5354,"num_bytes_recovered":350748672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":16},{"shards":"1,10,11","objects":817},{"shards":"10,11","objects":1397}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"978'19450","reported_seq":36915,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.397215+0000","last_change":"2024-04-29T14:49:28.483892+0000","last_active":"2024-04-29T14:50:15.397215+0000","last_peered":"2024-04-29T14:50:15.397215+0000","last_clean":"2024-04-29T14:50:15.397215+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:50:15.397215+0000","last_undegraded":"2024-04-29T14:50:15.397215+0000","last_fullsized":"2024-04-29T14:50:15.397215+0000","mapping_epoch":895,"log_start":"704'9400","ondisk_log_start":"704'9400","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'18861","last_scrub_stamp":"2024-04-29T14:49:28.483744+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:49:28.483744+0000","objects_scrubbed":2889,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:43.869641+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150601728,"num_objects":2298,"num_object_clones":0,"num_object_copies":4596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"978'12116","reported_seq":41269,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:16.030823+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:50:16.030823+0000","last_peered":"2024-04-29T14:50:16.030823+0000","last_clean":"2024-04-29T14:50:16.030823+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:50:16.030823+0000","last_undegraded":"2024-04-29T14:50:16.030823+0000","last_fullsized":"2024-04-29T14:50:16.030823+0000","mapping_epoch":756,"log_start":"662'2100","ondisk_log_start":"662'2100","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145752064,"num_objects":2224,"num_object_clones":0,"num_object_copies":4448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12116,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"978'12008","reported_seq":21084,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.975063+0000","last_change":"2024-04-29T14:50:12.400042+0000","last_active":"2024-04-29T14:50:15.975063+0000","last_peered":"2024-04-29T14:50:15.975063+0000","last_clean":"2024-04-29T14:50:15.975063+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:50:15.975063+0000","last_undegraded":"2024-04-29T14:50:15.975063+0000","last_fullsized":"2024-04-29T14:50:15.975063+0000","mapping_epoch":883,"log_start":"662'2000","ondisk_log_start":"662'2000","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'11960","last_scrub_stamp":"2024-04-29T14:50:12.399923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:12.399923+0000","objects_scrubbed":2330,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:26.243671+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149553152,"num_objects":2282,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12008,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"978'12377","reported_seq":70681,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.389958+0000","last_change":"2024-04-29T14:49:30.882066+0000","last_active":"2024-04-29T14:50:15.389958+0000","last_peered":"2024-04-29T14:50:15.389958+0000","last_clean":"2024-04-29T14:50:15.389958+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:50:15.389958+0000","last_undegraded":"2024-04-29T14:50:15.389958+0000","last_fullsized":"2024-04-29T14:50:15.389958+0000","mapping_epoch":939,"log_start":"662'2376","ondisk_log_start":"662'2376","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11756","last_scrub_stamp":"2024-04-29T14:49:30.881939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.881939+0000","objects_scrubbed":2972,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:39.012112+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154075136,"num_objects":2351,"num_object_clones":0,"num_object_copies":4702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12377,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"977'12055","reported_seq":46299,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.397168+0000","last_change":"2024-04-29T14:49:30.401449+0000","last_active":"2024-04-29T14:50:15.397168+0000","last_peered":"2024-04-29T14:50:15.397168+0000","last_clean":"2024-04-29T14:50:15.397168+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:50:15.397168+0000","last_undegraded":"2024-04-29T14:50:15.397168+0000","last_fullsized":"2024-04-29T14:50:15.397168+0000","mapping_epoch":918,"log_start":"662'2000","ondisk_log_start":"662'2000","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11508","last_scrub_stamp":"2024-04-29T14:49:30.401266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.401266+0000","objects_scrubbed":2766,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:32.430363+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145424384,"num_objects":2219,"num_object_clones":0,"num_object_copies":4438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12055,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"979'12318","reported_seq":48688,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-04-29T14:50:19.212321+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:50:19.212321+0000","last_peered":"2024-04-29T14:50:19.212321+0000","last_clean":"2024-04-29T14:50:19.212321+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:50:19.212321+0000","last_undegraded":"2024-04-29T14:50:19.212321+0000","last_fullsized":"2024-04-29T14:50:19.212321+0000","mapping_epoch":915,"log_start":"662'2300","ondisk_log_start":"662'2300","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143261696,"num_objects":2186,"num_object_clones":0,"num_object_copies":4372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"979'15720","reported_seq":40493,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-04-29T14:50:18.274763+0000","last_change":"2024-04-29T14:50:11.012445+0000","last_active":"2024-04-29T14:50:18.274763+0000","last_peered":"2024-04-29T14:50:18.274763+0000","last_clean":"2024-04-29T14:50:18.274763+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:50:18.274763+0000","last_undegraded":"2024-04-29T14:50:18.274763+0000","last_fullsized":"2024-04-29T14:50:18.274763+0000","mapping_epoch":895,"log_start":"675'5625","ondisk_log_start":"675'5625","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'15627","last_scrub_stamp":"2024-04-29T14:50:11.012229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:11.012229+0000","objects_scrubbed":2385,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.431973+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149880832,"num_objects":2287,"num_object_clones":0,"num_object_copies":4574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12121,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"979'11976","reported_seq":36430,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-04-29T14:50:18.396415+0000","last_change":"2024-04-29T14:50:04.932139+0000","last_active":"2024-04-29T14:50:18.396415+0000","last_peered":"2024-04-29T14:50:18.396415+0000","last_clean":"2024-04-29T14:50:18.396415+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:50:18.396415+0000","last_undegraded":"2024-04-29T14:50:18.396415+0000","last_fullsized":"2024-04-29T14:50:18.396415+0000","mapping_epoch":895,"log_start":"662'1944","ondisk_log_start":"662'1944","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11810","last_scrub_stamp":"2024-04-29T14:50:04.932001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:04.932001+0000","objects_scrubbed":2381,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:25.466879+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145096704,"num_objects":2214,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11976,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"979'12027","reported_seq":58812,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-04-29T14:50:19.169926+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:50:19.169926+0000","last_peered":"2024-04-29T14:50:19.169926+0000","last_clean":"2024-04-29T14:50:19.169926+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:50:19.169926+0000","last_undegraded":"2024-04-29T14:50:19.169926+0000","last_fullsized":"2024-04-29T14:50:19.169926+0000","mapping_epoch":928,"log_start":"662'2000","ondisk_log_start":"662'2000","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140967936,"num_objects":2151,"num_object_clones":0,"num_object_copies":4302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"979'12079","reported_seq":27052,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-04-29T14:50:18.308994+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:50:18.308994+0000","last_peered":"2024-04-29T14:50:18.308994+0000","last_clean":"2024-04-29T14:50:18.308994+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:50:18.308994+0000","last_undegraded":"2024-04-29T14:50:18.308994+0000","last_fullsized":"2024-04-29T14:50:18.308994+0000","mapping_epoch":703,"log_start":"662'2000","ondisk_log_start":"662'2000","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141885440,"num_objects":2165,"num_object_clones":0,"num_object_copies":4330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12079,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"978'11854","reported_seq":33199,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:16.015156+0000","last_change":"2024-04-29T14:50:05.324932+0000","last_active":"2024-04-29T14:50:16.015156+0000","last_peered":"2024-04-29T14:50:16.015156+0000","last_clean":"2024-04-29T14:50:16.015156+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:50:16.015156+0000","last_undegraded":"2024-04-29T14:50:16.015156+0000","last_fullsized":"2024-04-29T14:50:16.015156+0000","mapping_epoch":865,"log_start":"661'1800","ondisk_log_start":"661'1800","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11707","last_scrub_stamp":"2024-04-29T14:50:05.324827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.324827+0000","objects_scrubbed":2336,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:05.726927+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143261696,"num_objects":2186,"num_object_clones":0,"num_object_copies":4372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11854,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1941,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-04-29T14:50:15.433640+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:50:15.433640+0000","last_peered":"2024-04-29T14:50:15.433640+0000","last_clean":"2024-04-29T14:50:15.433640+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:50:15.433640+0000","last_undegraded":"2024-04-29T14:50:15.433640+0000","last_fullsized":"2024-04-29T14:50:15.433640+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2340487192,"num_objects":35714,"num_object_clones":0,"num_object_copies":71428,"num_objects_missing_on_primary":1397,"num_objects_missing":1397,"num_objects_degraded":0,"num_objects_misplaced":3611,"num_objects_unfound":0,"num_objects_dirty":35714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194106,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232338,"num_bytes_recovered":13624737816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5685125120,"data_stored":5685116976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122569,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384209,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1787200,"kb_used_data":895288,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":91957952,"statfs":{"total":95995035648,"available":94164942848,"internally_reserved":0,"allocated":916774912,"data_stored":914786260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":913308096},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645843,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1381736,"kb_used_data":578656,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92363416,"statfs":{"total":95995035648,"available":94580137984,"internally_reserved":0,"allocated":592543744,"data_stored":590567632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":822344128},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423147,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577776,"kb_used_data":441120,"kb_used_omap":4,"kb_used_meta":136635,"kb_avail":93167376,"statfs":{"total":95995035648,"available":95403393024,"internally_reserved":0,"allocated":451706880,"data_stored":449730768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":139914944},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234524,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1149744,"kb_used_data":292520,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92595408,"statfs":{"total":95995035648,"available":94817697792,"internally_reserved":0,"allocated":299540480,"data_stored":297560816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":877787594},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528865,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434548,"kb_used_data":442668,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92310604,"statfs":{"total":95995035648,"available":94526058496,"internally_reserved":0,"allocated":453292032,"data_stored":451308296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1015675334},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575405,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830956,"kb_used_data":344420,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92914196,"statfs":{"total":95995035648,"available":95144136704,"internally_reserved":0,"allocated":352686080,"data_stored":350705896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738943,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1534264,"kb_used_data":606252,"kb_used_omap":1,"kb_used_meta":927998,"kb_avail":92210888,"statfs":{"total":95995035648,"available":94423949312,"internally_reserved":0,"allocated":620802048,"data_stored":618819701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":950270405},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497377,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182963,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1187584,"kb_used_data":606648,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92557568,"statfs":{"total":95995035648,"available":94778949632,"internally_reserved":0,"allocated":621207552,"data_stored":619218920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":594868680},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986728,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1545472,"kb_used_data":571640,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92199680,"statfs":{"total":95995035648,"available":94412472320,"internally_reserved":0,"allocated":585359360,"data_stored":583370728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653289,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":574976,"kb_used_data":372856,"kb_used_omap":651,"kb_used_meta":201460,"kb_avail":93170176,"statfs":{"total":95995035648,"available":95406260224,"internally_reserved":0,"allocated":381804544,"data_stored":379815912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":206295816},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":614072320,"data_stored":614072320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":347607040,"data_stored":347602968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":615186432,"data_stored":615186432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":580255744,"data_stored":580255744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":375717888,"data_stored":375717888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":448532480,"data_stored":448528408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":292814848,"data_stored":292814848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":445579264,"data_stored":445579264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":588447744,"data_stored":588447744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":909770752,"data_stored":909770752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:50:21.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:50:21.706 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:21.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:21.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:21.837 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:21.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:21.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:21.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:22.013 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:22.013 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:22.221 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:22.221 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:22.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:22.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:22.382 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:22.804 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:50:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:23.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:50:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:23 smithi040 ceph-mon[39581]: pgmap v2468: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 240 op/s; 3468/70040 objects misplaced (4.951%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:50:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:23 smithi040 ceph-mon[39581]: osdmap e981: 12 total, 12 up, 10 in 2024-04-29T14:50:23.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3468108412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:23.629 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:23 smithi194 ceph-mon[44089]: pgmap v2468: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 240 op/s; 3468/70040 objects misplaced (4.951%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:50:23.629 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:23 smithi194 ceph-mon[44089]: osdmap e981: 12 total, 12 up, 10 in 2024-04-29T14:50:23.629 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3468108412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:23 smithi121 ceph-mon[44533]: pgmap v2468: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 240 op/s; 3468/70040 objects misplaced (4.951%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:50:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:23 smithi121 ceph-mon[44533]: osdmap e981: 12 total, 12 up, 10 in 2024-04-29T14:50:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3468108412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:23.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:23.666 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:24.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:24.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:24 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:24.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3468108412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:24.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:24 smithi040 ceph-mon[39581]: osdmap e982: 12 total, 12 up, 10 in 2024-04-29T14:50:24.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:50:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:24 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3468108412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:24 smithi121 ceph-mon[44533]: osdmap e982: 12 total, 12 up, 10 in 2024-04-29T14:50:24.651 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:24.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:50:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:24.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:50:24.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:24 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:24.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3468108412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:24.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:24 smithi194 ceph-mon[44089]: osdmap e982: 12 total, 12 up, 10 in 2024-04-29T14:50:24.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:24.806 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:24.826 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:24.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:24.931 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:24.961 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:25.295 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:25.296 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:25.310 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:25.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:25.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:25.450 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:25.555 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:25.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:25.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:25 smithi194 ceph-mon[44089]: pgmap v2471: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 249 op/s; 3468/69230 objects misplaced (5.009%); 768 KiB/s, 11 objects/s recovering 2024-04-29T14:50:25.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:25.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:25.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:25 smithi040 ceph-mon[39581]: pgmap v2471: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 249 op/s; 3468/69230 objects misplaced (5.009%); 768 KiB/s, 11 objects/s recovering 2024-04-29T14:50:25.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:25.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:25.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:25 smithi121 ceph-mon[44533]: pgmap v2471: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 249 op/s; 3468/69230 objects misplaced (5.009%); 768 KiB/s, 11 objects/s recovering 2024-04-29T14:50:25.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:25.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:25.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:25.689 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:25.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:25.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:25.852 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:25.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:25.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:25.960 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:50:25.973 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:25.995 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:50:26.003 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:26.017 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:50:26.043 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:50:26.063 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:50:26.064 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:50:26.084 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:50:26.105 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:50:26.125 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:50:26.150 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:50:26.174 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:50:26.195 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:50:26.216 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:50:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:50:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:50:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:50:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:50:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:50:26.218 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:50:26.218 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:50:26.218 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:50:26.218 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:50:26.218 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:50:26.218 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:50:26.218 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:50:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:26.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:26.341 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:26.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:26.387 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:26.405 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:26.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:26.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:26.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:26.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:26.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:26.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osdmap e983: 12 total, 12 up, 10 in 2024-04-29T14:50:26.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1048033487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:26.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:26 smithi121 ceph-mon[44533]: pgmap v2473: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 3326/67446 objects misplaced (4.931%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:50:26.664 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:26.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osdmap e983: 12 total, 12 up, 10 in 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1048033487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:26 smithi040 ceph-mon[39581]: pgmap v2473: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 3326/67446 objects misplaced (4.931%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osdmap e983: 12 total, 12 up, 10 in 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1048033487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:26.885 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:26 smithi194 ceph-mon[44089]: pgmap v2473: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 332 op/s; 3326/67446 objects misplaced (4.931%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:50:27.345 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:50:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:27.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:27.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:27.664 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:27.680 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:27.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:27.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:27.748 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:50:27.749 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:50:27.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:27.831 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:27.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:27.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: osdmap e984: 12 total, 12 up, 10 in 2024-04-29T14:50:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:27 smithi040 ceph-mon[39581]: osdmap e985: 12 total, 12 up, 10 in 2024-04-29T14:50:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:27.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:27.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: osdmap e984: 12 total, 12 up, 10 in 2024-04-29T14:50:27.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:27 smithi121 ceph-mon[44533]: osdmap e985: 12 total, 12 up, 10 in 2024-04-29T14:50:27.955 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: osdmap e984: 12 total, 12 up, 10 in 2024-04-29T14:50:27.965 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:27 smithi194 ceph-mon[44089]: osdmap e985: 12 total, 12 up, 10 in 2024-04-29T14:50:27.987 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:28.010 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:28.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:28.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:28.141 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:28.162 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:28.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:28.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:28.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:28.271 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:50:28.271 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:136420 7.5 e777/655 7:add952f2:::benchmark_data_smithi194_147200_object21508:head v 777'8520, mlcod=777'8518)", 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 528.47413816999995, 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074850630000000001, 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.272 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791146+0000", 2024-04-29T14:50:28.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791149+0000", 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6699999999999998e-06 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791158+0000", 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4490000000000005e-06 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791172+0000", 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3655999999999999e-05 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791181+0000", 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7700000000000007e-06 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791243+0000", 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2904999999999995e-05 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:28.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.791266+0000", 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2382999999999999e-05 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.798602+0000", 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073362590000000004 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:39.798631+0000", 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8971000000000001e-05 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.26998.0:136948 7.5 e779/655 7:ae800e08:::benchmark_data_smithi194_147200_object22036:head v 779'8554, mlcod=779'8551)", 2024-04-29T14:50:28.278 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 524.152164965, 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076202170000000003, 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113119+0000", 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113121+0000", 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8780000000000001e-06 2024-04-29T14:50:28.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113130+0000", 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1679999999999993e-06 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113143+0000", 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2628e-05 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.113151+0000", 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5299999999999996e-06 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.116770+0000", 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036181640000000001 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.116798+0000", 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8748999999999999e-05 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.120682+0000", 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038833090000000002 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:41:44.120739+0000", 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7791000000000003e-05 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e858/685 cost 1049576)", 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:46.170812+0000", 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 282.09447196799999, 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029017560000000001, 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.284 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170812+0000", 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170812+0000", 2024-04-29T14:50:28.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170815+0000", 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.101e-06 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170846+0000", 2024-04-29T14:50:28.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0626000000000001e-05 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170860+0000", 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4449e-05 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170870+0000", 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1940000000000004e-06 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.170928+0000", 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8112000000000002e-05 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:46.173714+0000", 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002786274 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e858/685 cost 1049576)", 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:45:47.413345+0000", 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 280.85193918900001, 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00090962900000000001, 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413345+0000", 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413345+0000", 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413349+0000", 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2069999999999997e-06 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413374+0000", 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4756000000000002e-05 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413402+0000", 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7883000000000001e-05 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413417+0000", 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5310000000000001e-05 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.413565+0000", 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014771399999999999 2024-04-29T14:50:28.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:45:47.414255+0000", 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00068975899999999997 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e867/685 cost 1049576)", 2024-04-29T14:50:28.293 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:22.703050+0000", 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 245.56223442800001, 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002770083, 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703050+0000", 2024-04-29T14:50:28.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703050+0000", 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703052+0000", 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.001e-06 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703064+0000", 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2330999999999999e-05 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703077+0000", 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3073e-05 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703085+0000", 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3209999999999999e-06 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.703137+0000", 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2392000000000003e-05 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:22.705820+0000", 2024-04-29T14:50:28.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026829649999999998 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e878/869 cost 1049576)", 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:39.378041+0000", 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 228.887243256, 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016862870000000001, 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.298 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378041+0000", 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378041+0000", 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378043+0000", 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3489999999999999e-06 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378055+0000", 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2062000000000001e-05 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378069+0000", 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3222e-05 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378076+0000", 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1720000000000001e-06 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.378133+0000", 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7169000000000001e-05 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.379727+0000", 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001594313 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e878/869 cost 1049576)", 2024-04-29T14:50:28.302 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:39.387071+0000", 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 228.87821338200001, 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010839949999999999, 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387071+0000", 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387071+0000", 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387072+0000", 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6330000000000001e-06 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387083+0000", 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0575e-05 2024-04-29T14:50:28.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387094+0000", 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1409999999999999e-05 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387102+0000", 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9699999999999999e-06 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.387159+0000", 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6434999999999998e-05 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:39.388155+0000", 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099597199999999992 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e880/869 cost 1049576)", 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:44.198461+0000", 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 224.06682357899999, 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00091009299999999995, 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198461+0000", 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198461+0000", 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198463+0000", 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4049999999999998e-06 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198479+0000", 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6161999999999999e-05 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198495+0000", 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5449999999999999e-05 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198503+0000", 2024-04-29T14:50:28.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4130000000000006e-06 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.198570+0000", 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6685000000000003e-05 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:44.199371+0000", 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00080097800000000004 2024-04-29T14:50:28.310 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e880/869 cost 1049576)", 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:46:45.394211+0000", 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 222.87107358700001, 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019670479999999999, 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394211+0000", 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394211+0000", 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394213+0000", 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3460000000000001e-06 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394243+0000", 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0195000000000001e-05 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394260+0000", 2024-04-29T14:50:28.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7048e-05 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394270+0000", 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.431e-06 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.394331+0000", 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0885999999999999e-05 2024-04-29T14:50:28.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:46:45.396178+0000", 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001847142 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e897/869 cost 1049576)", 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:08.761526+0000", 2024-04-29T14:50:28.315 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 199.50375789500001, 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011081419999999999, 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761526+0000", 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761526+0000", 2024-04-29T14:50:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761530+0000", 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8360000000000002e-06 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761550+0000", 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0030999999999998e-05 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761567+0000", 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6342e-05 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761595+0000", 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8308000000000001e-05 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.761672+0000", 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7114999999999996e-05 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:08.762634+0000", 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096250999999999997 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e907/869 cost 1049576)", 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:41.431280+0000", 2024-04-29T14:50:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 166.83400423800001, 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00092466599999999999, 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.431280+0000", 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.431280+0000", 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.431284+0000", 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5599999999999998e-06 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.431307+0000", 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3030000000000001e-05 2024-04-29T14:50:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.431331+0000", 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4116000000000001e-05 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.431345+0000", 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4063000000000001e-05 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.431502+0000", 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015738699999999999 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:41.432205+0000", 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00070251000000000005 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.323 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e915/869 cost 1049576)", 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:55.570391+0000", 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 152.694893307, 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012289550000000001, 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.570391+0000", 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.570391+0000", 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.570393+0000", 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.238e-06 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.570408+0000", 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4861000000000001e-05 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.570424+0000", 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6211000000000001e-05 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.570434+0000", 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7049999999999994e-06 2024-04-29T14:50:28.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.570496+0000", 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1637999999999995e-05 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:55.571620+0000", 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001124302 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e924/869 cost 1049576)", 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:48:14.613015+0000", 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 133.65226889600001, 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030559430000000002, 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.613015+0000", 2024-04-29T14:50:28.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.613015+0000", 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.613017+0000", 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.683e-06 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.613030+0000", 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2539e-05 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.613041+0000", 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1945e-05 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.613050+0000", 2024-04-29T14:50:28.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9700000000000005e-06 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.613127+0000", 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6132999999999998e-05 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:14.616071+0000", 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029446730000000001 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.331 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e945/942 cost 1049576)", 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:48:56.658215+0000", 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 91.607069666000001, 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020857079999999999, 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.658215+0000", 2024-04-29T14:50:28.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.658215+0000", 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.658217+0000", 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.249e-06 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.658230+0000", 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3529999999999999e-05 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.658245+0000", 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4932999999999999e-05 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.658253+0000", 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2790000000000001e-06 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.658311+0000", 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8388999999999999e-05 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:48:56.660300+0000", 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019893279999999998 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.335 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e946/942 cost 1049576)", 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:49:00.025713+0000", 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 88.239570901999997, 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00091846899999999999, 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.025713+0000", 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.025713+0000", 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.025733+0000", 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0122000000000001e-05 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.025763+0000", 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.991e-05 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.025789+0000", 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6146e-05 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.025805+0000", 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5075999999999999e-05 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.025964+0000", 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000159541 2024-04-29T14:50:28.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:00.026632+0000", 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066767400000000005 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e962/942 cost 1049576)", 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:49:36.855774+0000", 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.409510609000002, 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00092899400000000004, 2024-04-29T14:50:28.339 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.855774+0000", 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.855774+0000", 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.855778+0000", 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7160000000000002e-06 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.855808+0000", 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9482999999999999e-05 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.855839+0000", 2024-04-29T14:50:28.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1599e-05 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.855862+0000", 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2872999999999999e-05 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.856028+0000", 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000166224 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:36.856703+0000", 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00067409900000000005 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e963/942 cost 1049576)", 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:49:37.751052+0000", 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 50.514232651999997, 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012456559999999999, 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.751052+0000", 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.751052+0000", 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.751054+0000", 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.159e-06 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.751069+0000", 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5079000000000001e-05 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.751083+0000", 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4005000000000001e-05 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.751107+0000", 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4258000000000001e-05 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.751155+0000", 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7960999999999997e-05 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:37.752297+0000", 2024-04-29T14:50:28.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001142194 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e964/942 cost 1049576)", 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:49:40.496562+0000", 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 47.768722656000001, 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001288523, 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.347 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.496562+0000", 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.496562+0000", 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.496563+0000", 2024-04-29T14:50:28.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.511e-06 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.496573+0000", 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4739999999999994e-06 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.496581+0000", 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.721e-06 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.496587+0000", 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7429999999999999e-06 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.496633+0000", 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6394000000000003e-05 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:40.497850+0000", 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012166799999999999 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.350 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e969/942 cost 1049576)", 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:49:53.933292+0000", 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.331992366000001, 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00094971200000000002, 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.933292+0000", 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.933292+0000", 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.933296+0000", 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.783e-06 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.933319+0000", 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3560000000000001e-05 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.933344+0000", 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4899999999999999e-05 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.933359+0000", 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4653e-05 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.933494+0000", 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013572700000000001 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:49:53.934242+0000", 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00074708900000000004 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e984/942 cost 1049576)", 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:50:26.770605+0000", 2024-04-29T14:50:28.354 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.4946788689999999, 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000833704, 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.770605+0000", 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.770605+0000", 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:50:28.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.770607+0000", 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0600000000000002e-06 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.770647+0000", 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9189e-05 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.770664+0000", 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7782999999999999e-05 2024-04-29T14:50:28.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.770675+0000", 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0786e-05 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.770793+0000", 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117398 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:50:26.771439+0000", 2024-04-29T14:50:28.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064648799999999997 2024-04-29T14:50:28.358 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.358 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.358 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.358 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:50:28.358 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:50:28.358 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:50:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:28.359 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:28.359 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:28.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:28.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:28.489 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:28.510 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:28.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:28.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:28.640 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:28.661 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:28.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:28.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:28 smithi040 ceph-mon[39581]: from='client.37712 v1:172.21.15.40:0/2168752851' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:28.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:28 smithi040 ceph-mon[39581]: pgmap v2476: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 237 op/s; 3326/67446 objects misplaced (4.931%); 979 KiB/s, 15 objects/s recovering 2024-04-29T14:50:28.777 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:28.825 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2475,"stamp":"2024-04-29T14:50:27.594216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2210464312,"num_objects":33723,"num_object_clones":0,"num_object_copies":67446,"num_objects_missing_on_primary":1256,"num_objects_missing":1256,"num_objects_degraded":0,"num_objects_misplaced":3326,"num_objects_unfound":0,"num_objects_dirty":33723,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":196280,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232496,"num_bytes_recovered":13637877496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161190,"ondisk_log_size":161190,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":11681364,"kb_used_data":4862436,"kb_used_omap":668,"kb_used_meta":6818147,"kb_avail":925770156,"statfs":{"total":959950356480,"available":947988639744,"internally_reserved":0,"allocated":4979134464,"data_stored":4959051161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":6981783535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-85000192,"num_objects":-1297,"num_object_clones":0,"num_object_copies":-2594,"num_objects_missing_on_primary":-70,"num_objects_missing":-70,"num_objects_degraded":0,"num_objects_misplaced":-142,"num_objects_unfound":0,"num_objects_dirty":-1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1297,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4718592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.998404"},"pg_stats":[{"pgid":"7.f","version":"982'12383","reported_seq":47339,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.281573+0000","last_change":"2024-04-29T14:49:26.622778+0000","last_active":"2024-04-29T14:50:25.281573+0000","last_peered":"2024-04-29T14:50:25.281573+0000","last_clean":"2024-04-29T14:50:25.281573+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:50:25.281573+0000","last_undegraded":"2024-04-29T14:50:25.281573+0000","last_fullsized":"2024-04-29T14:50:25.281573+0000","mapping_epoch":918,"log_start":"662'2300","ondisk_log_start":"662'2300","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'11583","last_scrub_stamp":"2024-04-29T14:49:26.622601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:26.622601+0000","objects_scrubbed":2863,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:35.366659+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135069696,"num_objects":2061,"num_object_clones":0,"num_object_copies":4122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12383,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"982'19469","reported_seq":41490,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:24.111766+0000","last_change":"2024-04-29T14:49:57.142920+0000","last_active":"2024-04-29T14:50:24.111766+0000","last_peered":"2024-04-29T14:50:24.111766+0000","last_clean":"2024-04-29T14:50:24.111766+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:50:24.111766+0000","last_undegraded":"2024-04-29T14:50:24.111766+0000","last_fullsized":"2024-04-29T14:50:24.111766+0000","mapping_epoch":930,"log_start":"704'9458","ondisk_log_start":"704'9458","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'19114","last_scrub_stamp":"2024-04-29T14:49:57.142771+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:49:57.142771+0000","objects_scrubbed":2497,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:23.319557+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140050432,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12413,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"982'12213","reported_seq":43386,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.504856+0000","last_change":"2024-04-29T14:49:58.542110+0000","last_active":"2024-04-29T14:50:25.504856+0000","last_peered":"2024-04-29T14:50:25.504856+0000","last_clean":"2024-04-29T14:50:25.504856+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:50:25.504856+0000","last_undegraded":"2024-04-29T14:50:25.504856+0000","last_fullsized":"2024-04-29T14:50:25.504856+0000","mapping_epoch":938,"log_start":"662'2182","ondisk_log_start":"662'2182","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'11857","last_scrub_stamp":"2024-04-29T14:49:58.541853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:58.541853+0000","objects_scrubbed":2470,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:16.374459+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412056,"num_objects":2113,"num_object_clones":0,"num_object_copies":4226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12213,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"982'15427","reported_seq":80931,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.553794+0000","last_change":"2024-04-29T14:48:59.043912+0000","last_active":"2024-04-29T14:50:25.553794+0000","last_peered":"2024-04-29T14:50:25.553794+0000","last_clean":"2024-04-29T14:50:25.553794+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:50:25.553794+0000","last_undegraded":"2024-04-29T14:50:25.553794+0000","last_fullsized":"2024-04-29T14:50:25.553794+0000","mapping_epoch":939,"log_start":"673'5350","ondisk_log_start":"673'5350","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"946'14374","last_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:48:59.043790+0000","objects_scrubbed":3160,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"queued for scrub","scrub_duration":10465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136380416,"num_objects":2081,"num_object_clones":0,"num_object_copies":4162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"983'12196","reported_seq":28495,"reported_epoch":983,"state":"active+recovering","last_fresh":"2024-04-29T14:50:26.155197+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:50:26.155197+0000","last_peered":"2024-04-29T14:50:26.155197+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:50:26.155197+0000","last_undegraded":"2024-04-29T14:50:26.155197+0000","last_fullsized":"2024-04-29T14:50:26.155197+0000","mapping_epoch":942,"log_start":"662'2100","ondisk_log_start":"662'2100","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136708096,"num_objects":2086,"num_object_clones":0,"num_object_copies":4172,"num_objects_missing_on_primary":1256,"num_objects_missing":1256,"num_objects_degraded":0,"num_objects_misplaced":3326,"num_objects_unfound":0,"num_objects_dirty":2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5498,"num_bytes_recovered":360185856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":16},{"shards":"1,10,11","objects":814},{"shards":"10,11","objects":1256}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"982'19598","reported_seq":37071,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.377802+0000","last_change":"2024-04-29T14:49:28.483892+0000","last_active":"2024-04-29T14:50:25.377802+0000","last_peered":"2024-04-29T14:50:25.377802+0000","last_clean":"2024-04-29T14:50:25.377802+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:50:25.377802+0000","last_undegraded":"2024-04-29T14:50:25.377802+0000","last_fullsized":"2024-04-29T14:50:25.377802+0000","mapping_epoch":895,"log_start":"704'9500","ondisk_log_start":"704'9500","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'18861","last_scrub_stamp":"2024-04-29T14:49:28.483744+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:49:28.483744+0000","objects_scrubbed":2889,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:43.869641+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"983'12274","reported_seq":41437,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-04-29T14:50:26.164310+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:50:26.164310+0000","last_peered":"2024-04-29T14:50:26.164310+0000","last_clean":"2024-04-29T14:50:26.164310+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:50:26.164310+0000","last_undegraded":"2024-04-29T14:50:26.164310+0000","last_fullsized":"2024-04-29T14:50:26.164310+0000","mapping_epoch":756,"log_start":"662'2200","ondisk_log_start":"662'2200","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135397376,"num_objects":2066,"num_object_clones":0,"num_object_copies":4132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12274,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"983'12179","reported_seq":21265,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-04-29T14:50:26.024661+0000","last_change":"2024-04-29T14:50:12.400042+0000","last_active":"2024-04-29T14:50:26.024661+0000","last_peered":"2024-04-29T14:50:26.024661+0000","last_clean":"2024-04-29T14:50:26.024661+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:50:26.024661+0000","last_undegraded":"2024-04-29T14:50:26.024661+0000","last_fullsized":"2024-04-29T14:50:26.024661+0000","mapping_epoch":883,"log_start":"662'2100","ondisk_log_start":"662'2100","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'11960","last_scrub_stamp":"2024-04-29T14:50:12.399923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:12.399923+0000","objects_scrubbed":2330,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:26.243671+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138346496,"num_objects":2111,"num_object_clones":0,"num_object_copies":4222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12179,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"982'12526","reported_seq":70838,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.542669+0000","last_change":"2024-04-29T14:49:30.882066+0000","last_active":"2024-04-29T14:50:25.542669+0000","last_peered":"2024-04-29T14:50:25.542669+0000","last_clean":"2024-04-29T14:50:25.542669+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:50:25.542669+0000","last_undegraded":"2024-04-29T14:50:25.542669+0000","last_fullsized":"2024-04-29T14:50:25.542669+0000","mapping_epoch":939,"log_start":"662'2476","ondisk_log_start":"662'2476","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11756","last_scrub_stamp":"2024-04-29T14:49:30.881939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.881939+0000","objects_scrubbed":2972,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:39.012112+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144310272,"num_objects":2202,"num_object_clones":0,"num_object_copies":4404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"982'12209","reported_seq":46461,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.400435+0000","last_change":"2024-04-29T14:49:30.401449+0000","last_active":"2024-04-29T14:50:25.400435+0000","last_peered":"2024-04-29T14:50:25.400435+0000","last_clean":"2024-04-29T14:50:25.400435+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:50:25.400435+0000","last_undegraded":"2024-04-29T14:50:25.400435+0000","last_fullsized":"2024-04-29T14:50:25.400435+0000","mapping_epoch":918,"log_start":"662'2200","ondisk_log_start":"662'2200","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'11508","last_scrub_stamp":"2024-04-29T14:49:30.401266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:30.401266+0000","objects_scrubbed":2766,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:32.430363+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135331840,"num_objects":2065,"num_object_clones":0,"num_object_copies":4130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12209,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"982'12389","reported_seq":48765,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:24.215036+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:50:24.215036+0000","last_peered":"2024-04-29T14:50:24.215036+0000","last_clean":"2024-04-29T14:50:24.215036+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:50:24.215036+0000","last_undegraded":"2024-04-29T14:50:24.215036+0000","last_fullsized":"2024-04-29T14:50:24.215036+0000","mapping_epoch":915,"log_start":"662'2300","ondisk_log_start":"662'2300","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138608640,"num_objects":2115,"num_object_clones":0,"num_object_copies":4230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12389,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"981'15812","reported_seq":40589,"reported_epoch":981,"state":"active+clean","last_fresh":"2024-04-29T14:50:23.528883+0000","last_change":"2024-04-29T14:50:11.012445+0000","last_active":"2024-04-29T14:50:23.528883+0000","last_peered":"2024-04-29T14:50:23.528883+0000","last_clean":"2024-04-29T14:50:23.528883+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:50:23.528883+0000","last_undegraded":"2024-04-29T14:50:23.528883+0000","last_fullsized":"2024-04-29T14:50:23.528883+0000","mapping_epoch":895,"log_start":"675'5725","ondisk_log_start":"675'5725","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'15627","last_scrub_stamp":"2024-04-29T14:50:11.012229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:11.012229+0000","objects_scrubbed":2385,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.431973+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143851520,"num_objects":2195,"num_object_clones":0,"num_object_copies":4390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12213,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"981'12065","reported_seq":36523,"reported_epoch":981,"state":"active+clean","last_fresh":"2024-04-29T14:50:23.410705+0000","last_change":"2024-04-29T14:50:04.932139+0000","last_active":"2024-04-29T14:50:23.410705+0000","last_peered":"2024-04-29T14:50:23.410705+0000","last_clean":"2024-04-29T14:50:23.410705+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:50:23.410705+0000","last_undegraded":"2024-04-29T14:50:23.410705+0000","last_fullsized":"2024-04-29T14:50:23.410705+0000","mapping_epoch":895,"log_start":"662'2044","ondisk_log_start":"662'2044","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11810","last_scrub_stamp":"2024-04-29T14:50:04.932001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:04.932001+0000","objects_scrubbed":2381,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:25.466879+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139264000,"num_objects":2125,"num_object_clones":0,"num_object_copies":4250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12065,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"982'12090","reported_seq":58881,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:24.153006+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:50:24.153006+0000","last_peered":"2024-04-29T14:50:24.153006+0000","last_clean":"2024-04-29T14:50:24.153006+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:50:24.153006+0000","last_undegraded":"2024-04-29T14:50:24.153006+0000","last_fullsized":"2024-04-29T14:50:24.153006+0000","mapping_epoch":928,"log_start":"662'2000","ondisk_log_start":"662'2000","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136839168,"num_objects":2088,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12090,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"981'12169","reported_seq":27148,"reported_epoch":982,"state":"active+clean","last_fresh":"2024-04-29T14:50:23.595971+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:50:23.595971+0000","last_peered":"2024-04-29T14:50:23.595971+0000","last_clean":"2024-04-29T14:50:23.595971+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:50:23.595971+0000","last_undegraded":"2024-04-29T14:50:23.595971+0000","last_fullsized":"2024-04-29T14:50:23.595971+0000","mapping_epoch":703,"log_start":"662'2100","ondisk_log_start":"662'2100","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135987200,"num_objects":2075,"num_object_clones":0,"num_object_copies":4150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12169,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"983'11989","reported_seq":33344,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.830821+0000","last_change":"2024-04-29T14:50:05.324932+0000","last_active":"2024-04-29T14:50:25.830821+0000","last_peered":"2024-04-29T14:50:25.830821+0000","last_clean":"2024-04-29T14:50:25.830821+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:50:25.830821+0000","last_undegraded":"2024-04-29T14:50:25.830821+0000","last_fullsized":"2024-04-29T14:50:25.830821+0000","mapping_epoch":865,"log_start":"662'1900","ondisk_log_start":"662'1900","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11707","last_scrub_stamp":"2024-04-29T14:50:05.324827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.324827+0000","objects_scrubbed":2336,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:05.726927+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134414336,"num_objects":2051,"num_object_clones":0,"num_object_copies":4102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11989,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1951,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-04-29T14:50:25.626473+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:50:25.626473+0000","last_peered":"2024-04-29T14:50:25.626473+0000","last_clean":"2024-04-29T14:50:25.626473+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:50:25.626473+0000","last_undegraded":"2024-04-29T14:50:25.626473+0000","last_fullsized":"2024-04-29T14:50:25.626473+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2209873944,"num_objects":33721,"num_object_clones":0,"num_object_copies":67442,"num_objects_missing_on_primary":1256,"num_objects_missing":1256,"num_objects_degraded":0,"num_objects_misplaced":3326,"num_objects_unfound":0,"num_objects_dirty":33721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196099,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232482,"num_bytes_recovered":13634175000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5380513792,"data_stored":5380505648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161053,"ondisk_log_size":161053,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122571,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384210,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1746156,"kb_used_data":870876,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":91998996,"statfs":{"total":95995035648,"available":94206971904,"internally_reserved":0,"allocated":891777024,"data_stored":889769567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":896268736},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645845,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1348708,"kb_used_data":545628,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92396444,"statfs":{"total":95995035648,"available":94613958656,"internally_reserved":0,"allocated":558723072,"data_stored":556715635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":822344128},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552036,"kb_used_data":411228,"kb_used_omap":4,"kb_used_meta":140795,"kb_avail":93193116,"statfs":{"total":95995035648,"available":95429750784,"internally_reserved":0,"allocated":421097472,"data_stored":419090035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":144174784},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1130480,"kb_used_data":273252,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92614672,"statfs":{"total":95995035648,"available":94837424128,"internally_reserved":0,"allocated":279810048,"data_stored":277799059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":877787594},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1411952,"kb_used_data":415912,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92333200,"statfs":{"total":95995035648,"available":94549196800,"internally_reserved":0,"allocated":425893888,"data_stored":423878827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1019935174},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575407,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816552,"kb_used_data":325856,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92928600,"statfs":{"total":95995035648,"available":95158886400,"internally_reserved":0,"allocated":333676544,"data_stored":331665035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738945,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1445496,"kb_used_data":534120,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92299656,"statfs":{"total":95995035648,"available":94514847744,"internally_reserved":0,"allocated":546938880,"data_stored":544925208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1144348,"kb_used_data":575888,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92600804,"statfs":{"total":95995035648,"available":94823223296,"internally_reserved":0,"allocated":589709312,"data_stored":587708160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986729,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525676,"kb_used_data":551836,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92219476,"statfs":{"total":95995035648,"available":94432743424,"internally_reserved":0,"allocated":565080064,"data_stored":563072627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653290,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":559960,"kb_used_data":357840,"kb_used_omap":651,"kb_used_meta":201460,"kb_avail":93185192,"statfs":{"total":95995035648,"available":95421636608,"internally_reserved":0,"allocated":366428160,"data_stored":364427008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":206295816},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":540803072,"data_stored":540803072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327553024,"data_stored":327548952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":582418432,"data_stored":582418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":560660480,"data_stored":560660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360054784,"data_stored":360054784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":419041280,"data_stored":419037208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":272760832,"data_stored":272760832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":415956992,"data_stored":415956992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":548470784,"data_stored":548470784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":885653504,"data_stored":885653504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:50:28.826 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.b is not active+clean 2024-04-29T14:50:28.826 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.b', 'version': "983'12196", 'reported_seq': 28495, 'reported_epoch': 983, 'state': 'active+recovering', 'last_fresh': '2024-04-29T14:50:26.155197+0000', 'last_change': '2024-04-29T14:48:55.372453+0000', 'last_active': '2024-04-29T14:50:26.155197+0000', 'last_peered': '2024-04-29T14:50:26.155197+0000', 'last_clean': '2024-04-29T14:35:46.185312+0000', 'last_became_active': '2024-04-29T14:48:48.169551+0000', 'last_became_peered': '2024-04-29T14:48:48.169551+0000', 'last_unstale': '2024-04-29T14:50:26.155197+0000', 'last_undegraded': '2024-04-29T14:50:26.155197+0000', 'last_fullsized': '2024-04-29T14:50:26.155197+0000', 'mapping_epoch': 942, 'log_start': "662'2100", 'ondisk_log_start': "662'2100", 'created': 655, 'last_epoch_clean': 656, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_clean_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 136708096, 'num_objects': 2086, 'num_object_clones': 0, 'num_object_copies': 4172, 'num_objects_missing_on_primary': 1256, 'num_objects_missing': 1256, 'num_objects_degraded': 0, 'num_objects_misplaced': 3326, 'num_objects_unfound': 0, 'num_objects_dirty': 2086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12196, 'num_write_kb': 457024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5498, 'num_bytes_recovered': 360185856, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': ['4', '11'], 'object_location_counts': [{'shards': '1,5', 'objects': 16}, {'shards': '1,10,11', 'objects': 814}, {'shards': '10,11', 'objects': 1256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-29T14:50:28.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:50:28.836 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:28.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:28 smithi121 ceph-mon[44533]: from='client.37712 v1:172.21.15.40:0/2168752851' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:28 smithi121 ceph-mon[44533]: pgmap v2476: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 237 op/s; 3326/67446 objects misplaced (4.931%); 979 KiB/s, 15 objects/s recovering 2024-04-29T14:50:28.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:28.970 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:28.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:50:28.995 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:28 smithi194 ceph-mon[44089]: from='client.37712 v1:172.21.15.40:0/2168752851' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:28.995 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:28 smithi194 ceph-mon[44089]: pgmap v2476: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 237 op/s; 3326/67446 objects misplaced (4.931%); 979 KiB/s, 15 objects/s recovering 2024-04-29T14:50:29.022 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:50:29.177 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:29.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:29.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:29.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:29.724 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:29.728 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:29.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:29.807 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:30.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:30.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:30.780 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:30.885 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:30.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:30.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:31.022 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:31 smithi040 ceph-mon[39581]: pgmap v2477: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 220 op/s; 3326/66588 objects misplaced (4.995%); 768 KiB/s, 11 objects/s recovering 2024-04-29T14:50:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2762907572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:50:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:31 smithi040 ceph-mon[39581]: from='client.37718 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:50:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:31 smithi121 ceph-mon[44533]: pgmap v2477: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 220 op/s; 3326/66588 objects misplaced (4.995%); 768 KiB/s, 11 objects/s recovering 2024-04-29T14:50:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2762907572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:50:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:31 smithi121 ceph-mon[44533]: from='client.37718 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:50:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:31.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:31.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:31.809 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:31 smithi194 ceph-mon[44089]: pgmap v2477: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 220 op/s; 3326/66588 objects misplaced (4.995%); 768 KiB/s, 11 objects/s recovering 2024-04-29T14:50:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2762907572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:50:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:31 smithi194 ceph-mon[44089]: from='client.37718 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:50:31.960 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:31.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:32.067 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:32.087 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:50:32.098 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:32.393 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:32 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:32 smithi040 ceph-mon[39581]: from='client.37718 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:50:32.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:32 smithi040 ceph-mon[39581]: osdmap e986: 12 total, 12 up, 10 in 2024-04-29T14:50:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:32 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:32 smithi121 ceph-mon[44533]: from='client.37718 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:50:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:32 smithi121 ceph-mon[44533]: osdmap e986: 12 total, 12 up, 10 in 2024-04-29T14:50:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:32 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:32 smithi194 ceph-mon[44089]: from='client.37718 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:50:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:32 smithi194 ceph-mon[44089]: osdmap e986: 12 total, 12 up, 10 in 2024-04-29T14:50:33.553 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:33.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:33.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:33 smithi040 ceph-mon[39581]: pgmap v2479: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 3174/65460 objects misplaced (4.849%); 789 KiB/s, 12 objects/s recovering 2024-04-29T14:50:33.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:33 smithi040 ceph-mon[39581]: osdmap e987: 12 total, 12 up, 10 in 2024-04-29T14:50:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:33 smithi121 ceph-mon[44533]: pgmap v2479: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 3174/65460 objects misplaced (4.849%); 789 KiB/s, 12 objects/s recovering 2024-04-29T14:50:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:33 smithi121 ceph-mon[44533]: osdmap e987: 12 total, 12 up, 10 in 2024-04-29T14:50:33.656 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:33.685 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:33 smithi194 ceph-mon[44089]: pgmap v2479: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 3174/65460 objects misplaced (4.849%); 789 KiB/s, 12 objects/s recovering 2024-04-29T14:50:33.685 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:33 smithi194 ceph-mon[44089]: osdmap e987: 12 total, 12 up, 10 in 2024-04-29T14:50:33.688 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:33.708 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:33.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:33.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:33.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:33.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:33.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:33.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:33.989 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:34.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:34.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:34.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:34.150 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:34.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/446590653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:34.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:34 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/446590653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:34 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:34.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/446590653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:34 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:34.982 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:50:34.983 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:50:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:50:34.985 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:50:35.329 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:35.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:35.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:50:35.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:35.474 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:35.550 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:35.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: pgmap v2481: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 209 op/s; 3174/64634 objects misplaced (4.911%); 706 KiB/s, 11 objects/s recovering 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: 7.c scrub starts 2024-04-29T14:50:35.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:35 smithi040 ceph-mon[39581]: 7.c scrub ok 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: pgmap v2481: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 209 op/s; 3174/64634 objects misplaced (4.911%); 706 KiB/s, 11 objects/s recovering 2024-04-29T14:50:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: 7.c scrub starts 2024-04-29T14:50:35.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:35 smithi121 ceph-mon[44533]: 7.c scrub ok 2024-04-29T14:50:35.655 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:35.680 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.680 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.680 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.680 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.680 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: pgmap v2481: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 209 op/s; 3174/64634 objects misplaced (4.911%); 706 KiB/s, 11 objects/s recovering 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: 7.c scrub starts 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:35.681 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:35 smithi194 ceph-mon[44089]: 7.c scrub ok 2024-04-29T14:50:35.683 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:35.701 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:35.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:35.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:35.832 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:35.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:35.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:35.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:35.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:35.982 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:36.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:36.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:36.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:36.133 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:36.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:36.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:36.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:50:36.285 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:36.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:50:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:36.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:50:36.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:36.435 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:36.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:36.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:36.469 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:36.470 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:36.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:36.593 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:36.947 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:36.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:37.033 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:50:37.033 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2911,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":987,"num_osds":12,"num_up_osds":12,"osd_up_since":1714401063,"num_in_osds":10,"osd_in_since":1714402012,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":31375,"data_bytes":2056585784,"bytes_used":11639824384,"bytes_avail":948310532096,"bytes_total":959950356480,"misplaced_objects":3014,"misplaced_total":62750,"misplaced_ratio":0.048031872509960161,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":1285928,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":10289152,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":293},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":72,"modified":"2024-04-29T14:50:02.301403+0000","services":{}},"progress_events":{}} 2024-04-29T14:50:37.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:37.081 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:37.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:37.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:37.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:37.240 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:37.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:37 smithi040 ceph-mon[39581]: pgmap v2482: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 293 op/s; 3014/62750 objects misplaced (4.803%); 1.2 MiB/s, 19 objects/s recovering 2024-04-29T14:50:37.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:37 smithi040 ceph-mon[39581]: 7.5 scrub starts 2024-04-29T14:50:37.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:37 smithi040 ceph-mon[39581]: 7.5 scrub ok 2024-04-29T14:50:37.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:37.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:37.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1792411010' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:50:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:37 smithi194 ceph-mon[44089]: pgmap v2482: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 293 op/s; 3014/62750 objects misplaced (4.803%); 1.2 MiB/s, 19 objects/s recovering 2024-04-29T14:50:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:37 smithi194 ceph-mon[44089]: 7.5 scrub starts 2024-04-29T14:50:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:37 smithi194 ceph-mon[44089]: 7.5 scrub ok 2024-04-29T14:50:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1792411010' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:50:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:37 smithi121 ceph-mon[44533]: pgmap v2482: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 293 op/s; 3014/62750 objects misplaced (4.803%); 1.2 MiB/s, 19 objects/s recovering 2024-04-29T14:50:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:37 smithi121 ceph-mon[44533]: 7.5 scrub starts 2024-04-29T14:50:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:37 smithi121 ceph-mon[44533]: 7.5 scrub ok 2024-04-29T14:50:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1792411010' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:50:37.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:50:37.999 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:38.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:38.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:38.139 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:38.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:38.334 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:38.362 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:38.381 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:38.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:38.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:38.513 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:38.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:50:38.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:38 smithi040 ceph-mon[39581]: osdmap e988: 12 total, 12 up, 10 in 2024-04-29T14:50:38.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:38 smithi040 ceph-mon[39581]: osdmap e989: 12 total, 12 up, 10 in 2024-04-29T14:50:38.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:38 smithi040 ceph-mon[39581]: 7.f scrub starts 2024-04-29T14:50:38.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:50:38.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:38 smithi040 ceph-mon[39581]: 7.f scrub ok 2024-04-29T14:50:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:38 smithi194 ceph-mon[44089]: osdmap e988: 12 total, 12 up, 10 in 2024-04-29T14:50:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:38 smithi194 ceph-mon[44089]: osdmap e989: 12 total, 12 up, 10 in 2024-04-29T14:50:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:38 smithi194 ceph-mon[44089]: 7.f scrub starts 2024-04-29T14:50:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:50:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:38 smithi194 ceph-mon[44089]: 7.f scrub ok 2024-04-29T14:50:38.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:38 smithi121 ceph-mon[44533]: osdmap e988: 12 total, 12 up, 10 in 2024-04-29T14:50:38.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:38 smithi121 ceph-mon[44533]: osdmap e989: 12 total, 12 up, 10 in 2024-04-29T14:50:38.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:38 smithi121 ceph-mon[44533]: 7.f scrub starts 2024-04-29T14:50:38.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:50:38.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:38 smithi121 ceph-mon[44533]: 7.f scrub ok 2024-04-29T14:50:39.372 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:39.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:39.397 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:50:39.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:50:39.479 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:39.508 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:39.529 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:39.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:39 smithi040 ceph-mon[39581]: pgmap v2485: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 225 op/s; 3014/62750 objects misplaced (4.803%); 885 KiB/s, 13 objects/s recovering 2024-04-29T14:50:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:50:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:39 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:50:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:39 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:50:39.632 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:39.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:39 smithi194 ceph-mon[44089]: pgmap v2485: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 225 op/s; 3014/62750 objects misplaced (4.803%); 885 KiB/s, 13 objects/s recovering 2024-04-29T14:50:39.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:50:39.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:39 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:50:39.658 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:39 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:50:39.660 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:39.678 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:39.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:39.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:39.815 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:39.870 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:39.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:39 smithi121 ceph-mon[44533]: pgmap v2485: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 225 op/s; 3014/62750 objects misplaced (4.803%); 885 KiB/s, 13 objects/s recovering 2024-04-29T14:50:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:50:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:39 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:50:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:39 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:50:39.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:39.993 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:40.007 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:40.016 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:40.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:40.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:40.049 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:40.129 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:40.156 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:40.176 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:40.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:40.282 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:40.308 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:40.329 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:40.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:40.409 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:50:40.433 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:40.460 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:40.481 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:40.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:40.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:40 smithi040 ceph-mon[39581]: 7.7 scrub starts 2024-04-29T14:50:40.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:40 smithi040 ceph-mon[39581]: 7.7 scrub ok 2024-04-29T14:50:40.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:40.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:40.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/685218796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:40.527 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:40 smithi040 ceph-mon[39581]: from='client.37736 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:40.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:40.611 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:40.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:40.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:40.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:40.763 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:40 smithi194 ceph-mon[44089]: 7.7 scrub starts 2024-04-29T14:50:40.763 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:40 smithi194 ceph-mon[44089]: 7.7 scrub ok 2024-04-29T14:50:40.763 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:40.763 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:40.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/685218796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:40.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:40 smithi194 ceph-mon[44089]: from='client.37736 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:40.765 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:40.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:40.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:40 smithi121 ceph-mon[44533]: 7.7 scrub starts 2024-04-29T14:50:40.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:40 smithi121 ceph-mon[44533]: 7.7 scrub ok 2024-04-29T14:50:40.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:40.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:40.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/685218796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:40 smithi121 ceph-mon[44533]: from='client.37736 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:50:40.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:40.916 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:40.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:40.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:41.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:41.074 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:41.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:41 smithi040 ceph-mon[39581]: pgmap v2486: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 229 op/s; 3014/61922 objects misplaced (4.867%); 689 KiB/s, 10 objects/s recovering 2024-04-29T14:50:41.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:41 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:50:41.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:41 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:50:41.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:41 smithi040 ceph-mon[39581]: from='client.37736 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:41.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:41 smithi040 ceph-mon[39581]: osdmap e990: 12 total, 12 up, 10 in 2024-04-29T14:50:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:41 smithi194 ceph-mon[44089]: pgmap v2486: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 229 op/s; 3014/61922 objects misplaced (4.867%); 689 KiB/s, 10 objects/s recovering 2024-04-29T14:50:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:41 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:50:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:41 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:50:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:41 smithi194 ceph-mon[44089]: from='client.37736 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:41 smithi194 ceph-mon[44089]: osdmap e990: 12 total, 12 up, 10 in 2024-04-29T14:50:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:41 smithi121 ceph-mon[44533]: pgmap v2486: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 229 op/s; 3014/61922 objects misplaced (4.867%); 689 KiB/s, 10 objects/s recovering 2024-04-29T14:50:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:41 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:50:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:41 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:50:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:41 smithi121 ceph-mon[44533]: from='client.37736 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:50:41.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:41 smithi121 ceph-mon[44533]: osdmap e990: 12 total, 12 up, 10 in 2024-04-29T14:50:43.731 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:43 smithi040 ceph-mon[39581]: pgmap v2488: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 193 op/s; 2854/60428 objects misplaced (4.723%); 843 KiB/s, 13 objects/s recovering 2024-04-29T14:50:43.731 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:43 smithi040 ceph-mon[39581]: osdmap e991: 12 total, 12 up, 10 in 2024-04-29T14:50:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:43 smithi194 ceph-mon[44089]: pgmap v2488: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 193 op/s; 2854/60428 objects misplaced (4.723%); 843 KiB/s, 13 objects/s recovering 2024-04-29T14:50:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:43 smithi194 ceph-mon[44089]: osdmap e991: 12 total, 12 up, 10 in 2024-04-29T14:50:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:43 smithi121 ceph-mon[44533]: pgmap v2488: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 193 op/s; 2854/60428 objects misplaced (4.723%); 843 KiB/s, 13 objects/s recovering 2024-04-29T14:50:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:43 smithi121 ceph-mon[44533]: osdmap e991: 12 total, 12 up, 10 in 2024-04-29T14:50:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:44.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:44.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:44.892 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:44.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:45.015 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:45.041 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:45.060 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:45.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:45.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:45.577 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:50:45.577 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:50:45.606 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:45 smithi040 ceph-mon[39581]: pgmap v2490: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 241 op/s; 2854/59512 objects misplaced (4.796%); 753 KiB/s, 11 objects/s recovering 2024-04-29T14:50:45.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:50:45.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:45 smithi194 ceph-mon[44089]: pgmap v2490: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 241 op/s; 2854/59512 objects misplaced (4.796%); 753 KiB/s, 11 objects/s recovering 2024-04-29T14:50:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:45 smithi121 ceph-mon[44533]: pgmap v2490: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 241 op/s; 2854/59512 objects misplaced (4.796%); 753 KiB/s, 11 objects/s recovering 2024-04-29T14:50:46.477 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:50:46.478 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:50:46.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:46 smithi040 ceph-mon[39581]: 7.a scrub starts 2024-04-29T14:50:46.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:46 smithi040 ceph-mon[39581]: 7.a scrub ok 2024-04-29T14:50:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:46 smithi194 ceph-mon[44089]: 7.a scrub starts 2024-04-29T14:50:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:46 smithi194 ceph-mon[44089]: 7.a scrub ok 2024-04-29T14:50:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:46 smithi121 ceph-mon[44533]: 7.a scrub starts 2024-04-29T14:50:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:46 smithi121 ceph-mon[44533]: 7.a scrub ok 2024-04-29T14:50:47.132 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2490,"stamp":"2024-04-29T14:50:44.308803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1950483000,"num_objects":29756,"num_object_clones":0,"num_object_copies":59512,"num_objects_missing_on_primary":1020,"num_objects_missing":1020,"num_objects_degraded":0,"num_objects_misplaced":2854,"num_objects_unfound":0,"num_objects_dirty":29756,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":200247,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232732,"num_bytes_recovered":13653343992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160957,"ondisk_log_size":160957,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":11173904,"kb_used_data":4317540,"kb_used_omap":668,"kb_used_meta":6855587,"kb_avail":926277616,"statfs":{"total":959950356480,"available":948508278784,"internally_reserved":0,"allocated":4421160960,"data_stored":4400576607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7020122095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-106102784,"num_objects":-1619,"num_object_clones":0,"num_object_copies":-3238,"num_objects_missing_on_primary":-81,"num_objects_missing":-81,"num_objects_degraded":0,"num_objects_misplaced":-160,"num_objects_unfound":0,"num_objects_dirty":-1619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1619,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":5177344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.712159"},"pg_stats":[{"pgid":"7.f","version":"989'12610","reported_seq":47608,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.402558+0000","last_change":"2024-04-29T14:50:38.109612+0000","last_active":"2024-04-29T14:50:40.402558+0000","last_peered":"2024-04-29T14:50:40.402558+0000","last_clean":"2024-04-29T14:50:40.402558+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:50:40.402558+0000","last_undegraded":"2024-04-29T14:50:40.402558+0000","last_fullsized":"2024-04-29T14:50:40.402558+0000","mapping_epoch":918,"log_start":"662'2600","ondisk_log_start":"662'2600","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12570","last_scrub_stamp":"2024-04-29T14:50:38.109478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:38.109478+0000","objects_scrubbed":1874,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:03.361035+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120193024,"num_objects":1834,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12610,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"989'19678","reported_seq":41713,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-04-29T14:50:39.335334+0000","last_change":"2024-04-29T14:49:57.142920+0000","last_active":"2024-04-29T14:50:39.335334+0000","last_peered":"2024-04-29T14:50:39.335334+0000","last_clean":"2024-04-29T14:50:39.335334+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:50:39.335334+0000","last_undegraded":"2024-04-29T14:50:39.335334+0000","last_fullsized":"2024-04-29T14:50:39.335334+0000","mapping_epoch":930,"log_start":"704'9658","ondisk_log_start":"704'9658","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'19114","last_scrub_stamp":"2024-04-29T14:49:57.142771+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:49:57.142771+0000","objects_scrubbed":2497,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:23.319557+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126353408,"num_objects":1928,"num_object_clones":0,"num_object_copies":3856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12622,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"990'12408","reported_seq":43597,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.537023+0000","last_change":"2024-04-29T14:49:58.542110+0000","last_active":"2024-04-29T14:50:40.537023+0000","last_peered":"2024-04-29T14:50:40.537023+0000","last_clean":"2024-04-29T14:50:40.537023+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:50:40.537023+0000","last_undegraded":"2024-04-29T14:50:40.537023+0000","last_fullsized":"2024-04-29T14:50:40.537023+0000","mapping_epoch":938,"log_start":"662'2382","ondisk_log_start":"662'2382","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'11857","last_scrub_stamp":"2024-04-29T14:49:58.541853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:58.541853+0000","objects_scrubbed":2470,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:16.374459+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125632536,"num_objects":1918,"num_object_clones":0,"num_object_copies":3836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12408,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"990'15645","reported_seq":81192,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.498269+0000","last_change":"2024-04-29T14:50:34.579123+0000","last_active":"2024-04-29T14:50:40.498269+0000","last_peered":"2024-04-29T14:50:40.498269+0000","last_clean":"2024-04-29T14:50:40.498269+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:50:40.498269+0000","last_undegraded":"2024-04-29T14:50:40.498269+0000","last_fullsized":"2024-04-29T14:50:40.498269+0000","mapping_epoch":939,"log_start":"675'5550","ondisk_log_start":"675'5550","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'15542","last_scrub_stamp":"2024-04-29T14:50:34.578998+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:50:34.578998+0000","objects_scrubbed":1966,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:38.386277+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122093568,"num_objects":1863,"num_object_clones":0,"num_object_copies":3726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"990'12432","reported_seq":29453,"reported_epoch":990,"state":"active+recovering","last_fresh":"2024-04-29T14:50:41.264731+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:50:41.264731+0000","last_peered":"2024-04-29T14:50:41.264731+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:50:41.264731+0000","last_undegraded":"2024-04-29T14:50:41.264731+0000","last_fullsized":"2024-04-29T14:50:41.264731+0000","mapping_epoch":942,"log_start":"662'2400","ondisk_log_start":"662'2400","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":1020,"num_objects_missing":1020,"num_objects_degraded":0,"num_objects_misplaced":2854,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12432,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5734,"num_bytes_recovered":375652352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":16},{"shards":"1,10,11","objects":814},{"shards":"10,11","objects":1020}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"990'19836","reported_seq":37325,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.407745+0000","last_change":"2024-04-29T14:49:28.483892+0000","last_active":"2024-04-29T14:50:40.407745+0000","last_peered":"2024-04-29T14:50:40.407745+0000","last_clean":"2024-04-29T14:50:40.407745+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:50:40.407745+0000","last_undegraded":"2024-04-29T14:50:40.407745+0000","last_fullsized":"2024-04-29T14:50:40.407745+0000","mapping_epoch":895,"log_start":"704'9800","ondisk_log_start":"704'9800","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'18861","last_scrub_stamp":"2024-04-29T14:49:28.483744+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:49:28.483744+0000","objects_scrubbed":2889,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:50:43.869641+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":3824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12762,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"990'12497","reported_seq":41674,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.955305+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:50:40.955305+0000","last_peered":"2024-04-29T14:50:40.955305+0000","last_clean":"2024-04-29T14:50:40.955305+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:50:40.955305+0000","last_undegraded":"2024-04-29T14:50:40.955305+0000","last_fullsized":"2024-04-29T14:50:40.955305+0000","mapping_epoch":756,"log_start":"662'2400","ondisk_log_start":"662'2400","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120782848,"num_objects":1843,"num_object_clones":0,"num_object_copies":3686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12497,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"990'12393","reported_seq":21493,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.920720+0000","last_change":"2024-04-29T14:50:12.400042+0000","last_active":"2024-04-29T14:50:40.920720+0000","last_peered":"2024-04-29T14:50:40.920720+0000","last_clean":"2024-04-29T14:50:40.920720+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:50:40.920720+0000","last_undegraded":"2024-04-29T14:50:40.920720+0000","last_fullsized":"2024-04-29T14:50:40.920720+0000","mapping_epoch":883,"log_start":"662'2300","ondisk_log_start":"662'2300","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'11960","last_scrub_stamp":"2024-04-29T14:50:12.399923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:12.399923+0000","objects_scrubbed":2330,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:26.243671+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124321792,"num_objects":1897,"num_object_clones":0,"num_object_copies":3794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12393,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"990'12737","reported_seq":71094,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.417611+0000","last_change":"2024-04-29T14:50:39.638903+0000","last_active":"2024-04-29T14:50:40.417611+0000","last_peered":"2024-04-29T14:50:40.417611+0000","last_clean":"2024-04-29T14:50:40.417611+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:50:40.417611+0000","last_undegraded":"2024-04-29T14:50:40.417611+0000","last_fullsized":"2024-04-29T14:50:40.417611+0000","mapping_epoch":939,"log_start":"662'2676","ondisk_log_start":"662'2676","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12718","last_scrub_stamp":"2024-04-29T14:50:39.638817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:39.638817+0000","objects_scrubbed":2012,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:07.740501+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130482176,"num_objects":1991,"num_object_clones":0,"num_object_copies":3982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12737,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"989'12431","reported_seq":46725,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.407709+0000","last_change":"2024-04-29T14:50:36.645439+0000","last_active":"2024-04-29T14:50:40.407709+0000","last_peered":"2024-04-29T14:50:40.407709+0000","last_clean":"2024-04-29T14:50:40.407709+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:50:40.407709+0000","last_undegraded":"2024-04-29T14:50:40.407709+0000","last_fullsized":"2024-04-29T14:50:40.407709+0000","mapping_epoch":918,"log_start":"662'2400","ondisk_log_start":"662'2400","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12376","last_scrub_stamp":"2024-04-29T14:50:36.645270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:36.645270+0000","objects_scrubbed":1898,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:54.914264+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120782848,"num_objects":1843,"num_object_clones":0,"num_object_copies":3686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12431,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"991'12715","reported_seq":49109,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-04-29T14:50:44.225307+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:50:44.225307+0000","last_peered":"2024-04-29T14:50:44.225307+0000","last_clean":"2024-04-29T14:50:44.225307+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:50:44.225307+0000","last_undegraded":"2024-04-29T14:50:44.225307+0000","last_fullsized":"2024-04-29T14:50:44.225307+0000","mapping_epoch":915,"log_start":"662'2700","ondisk_log_start":"662'2700","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117243904,"num_objects":1789,"num_object_clones":0,"num_object_copies":3578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12715,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"991'16117","reported_seq":40914,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-04-29T14:50:43.498289+0000","last_change":"2024-04-29T14:50:11.012445+0000","last_active":"2024-04-29T14:50:43.498289+0000","last_peered":"2024-04-29T14:50:43.498289+0000","last_clean":"2024-04-29T14:50:43.498289+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:50:43.498289+0000","last_undegraded":"2024-04-29T14:50:43.498289+0000","last_fullsized":"2024-04-29T14:50:43.498289+0000","mapping_epoch":895,"log_start":"675'6025","ondisk_log_start":"675'6025","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'15627","last_scrub_stamp":"2024-04-29T14:50:11.012229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:11.012229+0000","objects_scrubbed":2385,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.431973+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123863040,"num_objects":1890,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"991'12376","reported_seq":36854,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-04-29T14:50:43.552581+0000","last_change":"2024-04-29T14:50:04.932139+0000","last_active":"2024-04-29T14:50:43.552581+0000","last_peered":"2024-04-29T14:50:43.552581+0000","last_clean":"2024-04-29T14:50:43.552581+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:50:43.552581+0000","last_undegraded":"2024-04-29T14:50:43.552581+0000","last_fullsized":"2024-04-29T14:50:43.552581+0000","mapping_epoch":895,"log_start":"662'2344","ondisk_log_start":"662'2344","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11810","last_scrub_stamp":"2024-04-29T14:50:04.932001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:04.932001+0000","objects_scrubbed":2381,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:25.466879+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118882304,"num_objects":1814,"num_object_clones":0,"num_object_copies":3628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"991'12406","reported_seq":59216,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-04-29T14:50:44.239572+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:50:44.239572+0000","last_peered":"2024-04-29T14:50:44.239572+0000","last_clean":"2024-04-29T14:50:44.239572+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:50:44.239572+0000","last_undegraded":"2024-04-29T14:50:44.239572+0000","last_fullsized":"2024-04-29T14:50:44.239572+0000","mapping_epoch":928,"log_start":"662'2400","ondisk_log_start":"662'2400","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"991'12489","reported_seq":27486,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-04-29T14:50:43.589637+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:50:43.589637+0000","last_peered":"2024-04-29T14:50:43.589637+0000","last_clean":"2024-04-29T14:50:43.589637+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:50:43.589637+0000","last_undegraded":"2024-04-29T14:50:43.589637+0000","last_fullsized":"2024-04-29T14:50:43.589637+0000","mapping_epoch":703,"log_start":"662'2400","ondisk_log_start":"662'2400","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115015680,"num_objects":1755,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12489,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"990'12185","reported_seq":33554,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.964808+0000","last_change":"2024-04-29T14:50:05.324932+0000","last_active":"2024-04-29T14:50:40.964808+0000","last_peered":"2024-04-29T14:50:40.964808+0000","last_clean":"2024-04-29T14:50:40.964808+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:50:40.964808+0000","last_undegraded":"2024-04-29T14:50:40.964808+0000","last_fullsized":"2024-04-29T14:50:40.964808+0000","mapping_epoch":865,"log_start":"662'2100","ondisk_log_start":"662'2100","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11707","last_scrub_stamp":"2024-04-29T14:50:05.324827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.324827+0000","objects_scrubbed":2336,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:05.726927+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121569280,"num_objects":1855,"num_object_clones":0,"num_object_copies":3710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12185,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1965,"reported_epoch":990,"state":"active+clean","last_fresh":"2024-04-29T14:50:40.401224+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:50:40.401224+0000","last_peered":"2024-04-29T14:50:40.401224+0000","last_clean":"2024-04-29T14:50:40.401224+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:50:40.401224+0000","last_undegraded":"2024-04-29T14:50:40.401224+0000","last_fullsized":"2024-04-29T14:50:40.401224+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1949892632,"num_objects":29754,"num_object_clones":0,"num_object_copies":59508,"num_objects_missing_on_primary":1020,"num_objects_missing":1020,"num_objects_degraded":0,"num_objects_misplaced":2854,"num_objects_unfound":0,"num_objects_dirty":29754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200066,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232718,"num_bytes_recovered":13649641496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4832632832,"data_stored":4832624688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122574,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384213,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1699712,"kb_used_data":824432,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92045440,"statfs":{"total":95995035648,"available":94254530560,"internally_reserved":0,"allocated":844218368,"data_stored":842167066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":896268736},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645848,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1294904,"kb_used_data":487664,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92450248,"statfs":{"total":95995035648,"available":94669053952,"internally_reserved":0,"allocated":499367936,"data_stored":497316654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":826603968},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423152,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":513784,"kb_used_data":368816,"kb_used_omap":4,"kb_used_meta":144955,"kb_avail":93231368,"statfs":{"total":95995035648,"available":95468920832,"internally_reserved":0,"allocated":377667584,"data_stored":375616302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":148434624},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234529,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106000,"kb_used_data":244612,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92639152,"statfs":{"total":95995035648,"available":94862491648,"internally_reserved":0,"allocated":250482688,"data_stored":248421619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":882047434},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1374404,"kb_used_data":374204,"kb_used_omap":1,"kb_used_meta":1000190,"kb_avail":92370748,"statfs":{"total":95995035648,"available":94587645952,"internally_reserved":0,"allocated":383184896,"data_stored":381125990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1024195014},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575410,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":792332,"kb_used_data":297472,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":92952820,"statfs":{"total":95995035648,"available":95183687680,"internally_reserved":0,"allocated":304611328,"data_stored":302549739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738948,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390536,"kb_used_data":475004,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92354616,"statfs":{"total":95995035648,"available":94571126784,"internally_reserved":0,"allocated":486404096,"data_stored":484346579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497382,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182968,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049236,"kb_used_data":476616,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92695916,"statfs":{"total":95995035648,"available":94920617984,"internally_reserved":0,"allocated":488054784,"data_stored":485990982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":586349000},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986733,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1447824,"kb_used_data":469832,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92297328,"statfs":{"total":95995035648,"available":94512463872,"internally_reserved":0,"allocated":481107968,"data_stored":479044166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653294,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505172,"kb_used_data":298888,"kb_used_omap":651,"kb_used_meta":205620,"kb_avail":93239980,"statfs":{"total":95995035648,"available":95477739520,"internally_reserved":0,"allocated":306061312,"data_stored":303997510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":210555656},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":481361920,"data_stored":481361920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":300290048,"data_stored":300285976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":482869248,"data_stored":482869248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":476184576,"data_stored":476184576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300089344,"data_stored":300089344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":378212352,"data_stored":378208280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245891072,"data_stored":245891072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":372244480,"data_stored":372244480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":490733568,"data_stored":490733568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":837615616,"data_stored":837615616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:50:47.584 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:47.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:47.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:47.703 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:47 smithi040 ceph-mon[39581]: from='client.37742 v1:172.21.15.40:0/3226069569' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:47.703 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:47 smithi040 ceph-mon[39581]: pgmap v2491: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 304 op/s; 2854/57874 objects misplaced (4.931%); 632 KiB/s, 9 objects/s recovering 2024-04-29T14:50:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:47 smithi194 ceph-mon[44089]: from='client.37742 v1:172.21.15.40:0/3226069569' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:47 smithi194 ceph-mon[44089]: pgmap v2491: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 304 op/s; 2854/57874 objects misplaced (4.931%); 632 KiB/s, 9 objects/s recovering 2024-04-29T14:50:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:47 smithi121 ceph-mon[44533]: from='client.37742 v1:172.21.15.40:0/3226069569' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:50:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:47 smithi121 ceph-mon[44533]: pgmap v2491: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 304 op/s; 2854/57874 objects misplaced (4.931%); 632 KiB/s, 9 objects/s recovering 2024-04-29T14:50:48.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:50:48.368 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:48.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:48.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:48.508 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:48.530 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:48.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:48.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:48 smithi040 ceph-mon[39581]: pgmap v2492: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 2679/57508 objects misplaced (4.658%); 1.3 MiB/s, 21 objects/s recovering 2024-04-29T14:50:48.616 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:48.636 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:48.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:48.677 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:48.682 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:48.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:48.694 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:50:48.716 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:50:48.736 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:50:48.757 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:50:48.779 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:50:48.786 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:48.801 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:50:48.812 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:48.821 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:50:48.830 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:48.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:48.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:50:48.871 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:50:48.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:48 smithi121 ceph-mon[44533]: pgmap v2492: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 2679/57508 objects misplaced (4.658%); 1.3 MiB/s, 21 objects/s recovering 2024-04-29T14:50:48.896 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:50:48.919 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:50:48.934 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:48.945 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:50:48.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:50:48.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:50:48.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:50:48.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:50:48.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:50:48.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:50:48.948 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:50:48.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:50:48.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:50:48.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:50:48.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:50:48.949 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:50:48.977 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:49.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:48 smithi194 ceph-mon[44089]: pgmap v2492: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 2679/57508 objects misplaced (4.658%); 1.3 MiB/s, 21 objects/s recovering 2024-04-29T14:50:49.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2075125049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:49.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2075125049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:49.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2075125049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:50.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:50:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:50.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:50.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:50.596 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:50.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:50.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:50.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:50.755 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:50 smithi040 ceph-mon[39581]: osdmap e992: 12 total, 12 up, 10 in 2024-04-29T14:50:50.755 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:50 smithi040 ceph-mon[39581]: pgmap v2494: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 262 op/s; 2679/56220 objects misplaced (4.765%); 720 KiB/s, 11 objects/s recovering 2024-04-29T14:50:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:50 smithi121 ceph-mon[44533]: osdmap e992: 12 total, 12 up, 10 in 2024-04-29T14:50:50.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:50 smithi121 ceph-mon[44533]: pgmap v2494: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 262 op/s; 2679/56220 objects misplaced (4.765%); 720 KiB/s, 11 objects/s recovering 2024-04-29T14:50:51.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:50 smithi194 ceph-mon[44089]: osdmap e992: 12 total, 12 up, 10 in 2024-04-29T14:50:51.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:50 smithi194 ceph-mon[44089]: pgmap v2494: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 262 op/s; 2679/56220 objects misplaced (4.765%); 720 KiB/s, 11 objects/s recovering 2024-04-29T14:50:52.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:50:52.818 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:52.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:52.896 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:50:52.896 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:154954 7.c 7:3543cd94:::benchmark_data_smithi194_147200_object40042:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 349.40728912399999, 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041657131999999999, 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.897 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154954 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477051+0000", 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.807e-06 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477056+0000", 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1760000000000001e-06 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477067+0000", 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0828e-05 2024-04-29T14:50:52.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477074+0000", 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0990000000000001e-06 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489561+0000", 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012487242000000001 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489704+0000", 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014225699999999999 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489831+0000", 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012778299999999999 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.497870+0000", 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080388730000000002 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.515675+0000", 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017804868000000001 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.515755+0000", 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9532999999999999e-05 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518706+0000", 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029516659999999999 2024-04-29T14:50:52.902 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:154957 7.c 7:39c117cf:::benchmark_data_smithi194_147200_object40045:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 349.40204308300002, 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039817678000000002, 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.903 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154957 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482296+0000", 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.054e-06 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482304+0000", 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5410000000000003e-06 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482329+0000", 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4757999999999999e-05 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482336+0000", 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6459999999999993e-06 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.493836+0000", 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011499568 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.493962+0000", 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012657 2024-04-29T14:50:52.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.494045+0000", 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3016000000000006e-05 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.498011+0000", 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039652560000000003 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518780+0000", 2024-04-29T14:50:52.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020769699999999999 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518839+0000", 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8284e-05 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.522113+0000", 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003274285 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155044 7.c 7:3e86b522:::benchmark_data_smithi194_147200_object40132:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 349.126783531, 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038996208999999997, 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155044 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757557+0000", 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.447e-06 2024-04-29T14:50:52.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757565+0000", 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2880000000000004e-06 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757578+0000", 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3828999999999999e-05 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757590+0000", 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1832e-05 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766688+0000", 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090977279999999994 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766861+0000", 2024-04-29T14:50:52.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017302100000000001 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766993+0000", 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000131521 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.787943+0000", 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020950180999999998 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.793252+0000", 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053089900000000004 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.793316+0000", 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4738000000000003e-05 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.796551+0000", 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032346340000000001 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.914 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155579 7.c 7:3ca30ab7:::benchmark_data_smithi194_147200_object40667:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 345.68502371599999, 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040246949999999997, 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155579 2024-04-29T14:50:52.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199317+0000", 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0700000000000001e-06 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199322+0000", 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.604e-06 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199332+0000", 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9869999999999994e-06 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199341+0000", 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1009999999999998e-06 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211121+0000", 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011779092 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211232+0000", 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111634 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211319+0000", 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7120999999999995e-05 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.216006+0000", 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046872049999999998 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.236426+0000", 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020419567 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.236500+0000", 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4419999999999996e-05 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.239562+0000", 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003061149 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155964 7.c 7:342ad242:::benchmark_data_smithi194_147200_object41052:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:50:52.920 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 343.09234480499998, 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038617988999999998, 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155964 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.921 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791996+0000", 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5220000000000002e-06 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792002+0000", 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0700000000000003e-06 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792012+0000", 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0268000000000001e-05 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792017+0000", 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.994e-06 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799209+0000", 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0071913070000000001 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799312+0000", 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010362899999999999 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799433+0000", 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120448 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.804509+0000", 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050758950000000004 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.827370+0000", 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022861757 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.827437+0000", 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6568000000000004e-05 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.830612+0000", 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003174531 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:156496 7.c 7:335b5fcf:::benchmark_data_smithi194_147200_object41584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e846)", 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 338.89265553899997, 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040110869, 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 156496 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991686+0000", 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.613e-06 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991699+0000", 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2316e-05 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991715+0000", 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6147999999999999e-05 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991730+0000", 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4970000000000001e-05 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001539+0000", 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098087069999999998 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001670+0000", 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013161299999999999 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001804+0000", 2024-04-29T14:50:52.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013356300000000001 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.014002+0000", 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012198525 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.028772+0000", 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014770086 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.028824+0000", 2024-04-29T14:50:52.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1232e-05 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.031794+0000", 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029700960000000002 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:157905 7.c 7:3da9029e:::benchmark_data_smithi194_147200_object42993:head [delete] snapc 0=[] ondisk+write+known_if_redirected e850)", 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 329.08312351400002, 2024-04-29T14:50:52.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041407046000000003, 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 157905 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801218+0000", 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7939999999999998e-06 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801226+0000", 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4090000000000003e-06 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801238+0000", 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2238999999999999e-05 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801247+0000", 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8240000000000006e-06 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812668+0000", 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011420397000000001 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812852+0000", 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018423699999999999 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812973+0000", 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121585 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.824943+0000", 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011969484000000001 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.840398+0000", 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01545529 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.840441+0000", 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2951000000000003e-05 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.842622+0000", 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021808359999999998 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164264 7.c 7:3a985e73:::benchmark_data_smithi194_147200_object49352:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 269.71326045400002, 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010643537999999999, 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164264 2024-04-29T14:50:52.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171082+0000", 2024-04-29T14:50:52.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7680000000000002e-06 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171093+0000", 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0937e-05 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171114+0000", 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1294999999999999e-05 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171127+0000", 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3363999999999999e-05 2024-04-29T14:50:52.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171212+0000", 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4429000000000001e-05 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171412+0000", 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020048099999999999 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171548+0000", 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013562900000000001 2024-04-29T14:50:52.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181610+0000", 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010061746 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181626+0000", 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5973000000000001e-05 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181674+0000", 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8105000000000001e-05 2024-04-29T14:50:52.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181721+0000", 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7811e-05 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164265 7.c 7:3e4dad70:::benchmark_data_smithi194_147200_object49353:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 269.709807845, 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019274217, 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164265 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:50:52.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174534+0000", 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4429999999999999e-06 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174545+0000", 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0548e-05 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174563+0000", 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8488999999999999e-05 2024-04-29T14:50:52.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174581+0000", 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7569000000000001e-05 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176564+0000", 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019836049999999998 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176739+0000", 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017455199999999999 2024-04-29T14:50:52.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176878+0000", 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013903 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181767+0000", 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048891000000000004 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193663+0000", 2024-04-29T14:50:52.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011896105000000001 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193730+0000", 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7065999999999998e-05 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193805+0000", 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4709999999999995e-05 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.948 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164330 7.e 7:7a9f05b1:::benchmark_data_smithi194_147200_object49418:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e869)", 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 267.16380025199999, 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15689136400000001, 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164330 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:50:52.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720541+0000", 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7659999999999999e-06 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720547+0000", 2024-04-29T14:50:52.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2770000000000002e-06 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720561+0000", 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3909e-05 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720571+0000", 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0066e-05 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720646+0000", 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4630999999999997e-05 2024-04-29T14:50:52.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720659+0000", 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3281e-05 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872536+0000", 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15187671799999999 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872672+0000", 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013665500000000001 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9,10", 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872793+0000", 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012021 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877184+0000", 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043910989999999999 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877252+0000", 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8238000000000006e-05 2024-04-29T14:50:52.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877270+0000", 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7705000000000001e-05 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877326+0000", 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6563e-05 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877430+0000", 2024-04-29T14:50:52.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010324599999999999 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164350 7.e 7:7e0a46a8:::benchmark_data_smithi194_147200_object49438:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e869)", 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 267.16374973900002, 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.16736200300000001, 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:50:52.955 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164350 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:50:52.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8400000000000003e-07 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720593+0000", 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2389999999999998e-06 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720599+0000", 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3350000000000003e-06 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720604+0000", 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2959999999999998e-06 2024-04-29T14:50:52.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720698+0000", 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3924000000000004e-05 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720704+0000", 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1480000000000003e-06 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877463+0000", 2024-04-29T14:50:52.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.156758704 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877555+0000", 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1974999999999999e-05 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9,10", 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877640+0000", 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5377999999999995e-05 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.881807+0000", 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041667179999999998 2024-04-29T14:50:52.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.881835+0000", 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7815999999999999e-05 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887756+0000", 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059212279999999997 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887822+0000", 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5743999999999999e-05 2024-04-29T14:50:52.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887951+0000", 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012871399999999999 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:164988 7.c e879/877 7:333ea8eb:::benchmark_data_smithi194_147200_object50076:head v 879'10294, mlcod=879'10291)", 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 249.17200439499999, 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012683506000000001, 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.961 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712335+0000", 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4799999999999998e-07 2024-04-29T14:50:52.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712340+0000", 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9910000000000002e-06 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712355+0000", 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5591e-05 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712360+0000", 2024-04-29T14:50:52.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1139999999999997e-06 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.716296+0000", 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039360840000000003 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.716327+0000", 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0960999999999997e-05 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.724966+0000", 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086382809999999994 2024-04-29T14:50:52.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.725018+0000", 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1935999999999999e-05 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:165418 7.a e882/877 7:5a70a2d7:::benchmark_data_smithi194_147200_object50506:head v 882'17678, mlcod=881'17676)", 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 245.021819037, 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011664525, 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.965 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862520+0000", 2024-04-29T14:50:52.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1699999999999999e-07 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862527+0000", 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0779999999999998e-06 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862543+0000", 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5698000000000001e-05 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862553+0000", 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0301e-05 2024-04-29T14:50:52.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862624+0000", 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.148e-05 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862657+0000", 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2861000000000003e-05 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.874126+0000", 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011468580000000001 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.874184+0000", 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8109999999999998e-05 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:165421 7.2 e882/877 7:4462835d:::benchmark_data_smithi194_147200_object50509:head v 882'10201, mlcod=881'10199)", 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 245.015198932, 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013165216, 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:50:52.969 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869140+0000", 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8799999999999998e-07 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869147+0000", 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1590000000000004e-06 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869167+0000", 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.014e-05 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869177+0000", 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0088999999999999e-05 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.873690+0000", 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045127489999999999 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.873721+0000", 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1281000000000001e-05 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.882230+0000", 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0085086999999999992 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:50:52.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.882305+0000", 2024-04-29T14:50:52.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4709999999999995e-05 2024-04-29T14:50:52.973 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.973 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.973 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.973 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:50:52.973 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:50:52.973 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:50:52.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:52.974 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:52.974 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:53.000 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:53.019 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:53.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:53.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:53.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:53.181 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:53.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:53.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:50:53.319 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:50:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:53 smithi040 ceph-mon[39581]: pgmap v2495: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 326 op/s; 2506/54090 objects misplaced (4.633%); 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:50:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:53 smithi040 ceph-mon[39581]: osdmap e993: 12 total, 12 up, 10 in 2024-04-29T14:50:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/516842869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:53.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:53 smithi121 ceph-mon[44533]: pgmap v2495: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 326 op/s; 2506/54090 objects misplaced (4.633%); 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:50:53.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:53 smithi121 ceph-mon[44533]: osdmap e993: 12 total, 12 up, 10 in 2024-04-29T14:50:53.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/516842869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:53.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:50:53.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:53 smithi194 ceph-mon[44089]: pgmap v2495: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 326 op/s; 2506/54090 objects misplaced (4.633%); 1.1 MiB/s, 17 objects/s recovering 2024-04-29T14:50:53.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:53 smithi194 ceph-mon[44089]: osdmap e993: 12 total, 12 up, 10 in 2024-04-29T14:50:53.684 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/516842869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:50:53.926 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:50:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:53.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:50:54.076 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:54.102 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:54.175 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:54.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:54.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:54.307 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:54.458 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:54.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:54.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:54.563 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:54.590 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:54 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:54.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/516842869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:50:54.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:54 smithi040 ceph-mon[39581]: osdmap e994: 12 total, 12 up, 10 in 2024-04-29T14:50:54.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:54 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:54.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/516842869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:50:54.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:54 smithi121 ceph-mon[44533]: osdmap e994: 12 total, 12 up, 10 in 2024-04-29T14:50:54.788 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:54.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:54 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:50:54.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/516842869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:50:54.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:54 smithi194 ceph-mon[44089]: osdmap e994: 12 total, 12 up, 10 in 2024-04-29T14:50:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:54.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:54.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:50:54.918 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:55.015 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:50:55.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:55.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:50:55.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:55.150 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:55.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:55.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:55.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:55.298 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:55.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:55.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:55.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:50:55.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:55 smithi121 ceph-mon[44533]: pgmap v2498: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 2506/53018 objects misplaced (4.727%); 896 KiB/s, 13 objects/s recovering 2024-04-29T14:50:55.449 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:55.469 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:50:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:55.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:50:55.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:55 smithi194 ceph-mon[44089]: pgmap v2498: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 2506/53018 objects misplaced (4.727%); 896 KiB/s, 13 objects/s recovering 2024-04-29T14:50:55.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:55.601 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:55.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:55.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:55 smithi040 ceph-mon[39581]: pgmap v2498: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 374 op/s; 2506/53018 objects misplaced (4.727%); 896 KiB/s, 13 objects/s recovering 2024-04-29T14:50:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:55.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:55.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:55.749 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:55.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:55.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:55.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:55.903 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:56.854 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:56.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:56.960 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:56.989 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:57.042 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:57.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:57.058 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:50:57.083 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:50:57.118 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:50:57.142 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:50:57.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:50:57.170 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:50:57.172 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:57.190 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:50:57.190 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:50:57.190 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:50:57.190 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:50:57.190 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:50:57.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:50:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:57.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:50:57.210 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:50:57.230 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:50:57.259 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:50:57.287 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:50:57.295 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:50:57.318 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:50:57.323 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:57.346 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:50:57.348 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:50:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:50:57.482 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:50:57.484 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:57 smithi194 ceph-mon[44089]: pgmap v2499: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 370 op/s; 2506/51270 objects misplaced (4.888%); 804 KiB/s, 12 objects/s recovering 2024-04-29T14:50:57.485 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/562850958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:57.485 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:57.485 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:57 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:57.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:50:57.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:57 smithi040 ceph-mon[39581]: pgmap v2499: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 370 op/s; 2506/51270 objects misplaced (4.888%); 804 KiB/s, 12 objects/s recovering 2024-04-29T14:50:57.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/562850958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:57.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:57.487 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:57 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:57.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:50:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:57.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:57 smithi121 ceph-mon[44533]: pgmap v2499: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 370 op/s; 2506/51270 objects misplaced (4.888%); 804 KiB/s, 12 objects/s recovering 2024-04-29T14:50:57.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/562850958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:50:57.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:57.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:57 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:57.702 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:50:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:57.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:50:57.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:57.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:50:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:57.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:50:57.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:50:57.969 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:50:57.984 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:57.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:50:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:50:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:58.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:50:58.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:58.132 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:58.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:58.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:50:58.235 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:50:58.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:58.301 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:58.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:58.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: osdmap e995: 12 total, 12 up, 10 in 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:50:58.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:50:58.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:50:58.432 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:58.450 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:50:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:58.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:50:58.553 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:50:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:50:58.582 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:58.686 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: osdmap e995: 12 total, 12 up, 10 in 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:50:58.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:50:58.709 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:58.711 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.711 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.711 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.711 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.711 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.711 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:50:58.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: osdmap e995: 12 total, 12 up, 10 in 2024-04-29T14:50:58.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:50:58.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:50:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:58.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:58.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:50:58.842 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:58.919 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:50:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:58.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:50:59.023 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:50:59.050 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:59.072 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:50:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:59.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:50:59.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:59.207 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:59.406 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2501,"stamp":"2024-04-29T14:50:58.311359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1667826232,"num_objects":25443,"num_object_clones":0,"num_object_copies":50886,"num_objects_missing_on_primary":755,"num_objects_missing":755,"num_objects_degraded":0,"num_objects_misplaced":2321,"num_objects_unfound":0,"num_objects_dirty":25443,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":204560,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233000,"num_bytes_recovered":13670907640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161070,"ondisk_log_size":161070,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":10608160,"kb_used_data":3772600,"kb_used_omap":668,"kb_used_meta":6834787,"kb_avail":926843360,"statfs":{"total":959950356480,"available":949087600640,"internally_reserved":0,"allocated":3863142400,"data_stored":3842313632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":6998822895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,1,1,6,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-104988672,"num_objects":-1602,"num_object_clones":0,"num_object_copies":-3204,"num_objects_missing_on_primary":-91,"num_objects_missing":-91,"num_objects_degraded":0,"num_objects_misplaced":-185,"num_objects_unfound":0,"num_objects_dirty":-1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1602,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":6160384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001061"},"pg_stats":[{"pgid":"7.f","version":"994'12923","reported_seq":47929,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:55.270590+0000","last_change":"2024-04-29T14:50:38.109612+0000","last_active":"2024-04-29T14:50:55.270590+0000","last_peered":"2024-04-29T14:50:55.270590+0000","last_clean":"2024-04-29T14:50:55.270590+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:50:55.270590+0000","last_undegraded":"2024-04-29T14:50:55.270590+0000","last_fullsized":"2024-04-29T14:50:55.270590+0000","mapping_epoch":918,"log_start":"663'2900","ondisk_log_start":"663'2900","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12570","last_scrub_stamp":"2024-04-29T14:50:38.109478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:38.109478+0000","objects_scrubbed":1874,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:03.361035+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99680256,"num_objects":1521,"num_object_clones":0,"num_object_copies":3042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12923,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"994'19956","reported_seq":42002,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:54.220880+0000","last_change":"2024-04-29T14:49:57.142920+0000","last_active":"2024-04-29T14:50:54.220880+0000","last_peered":"2024-04-29T14:50:54.220880+0000","last_clean":"2024-04-29T14:50:54.220880+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:50:54.220880+0000","last_undegraded":"2024-04-29T14:50:54.220880+0000","last_fullsized":"2024-04-29T14:50:54.220880+0000","mapping_epoch":930,"log_start":"704'9858","ondisk_log_start":"704'9858","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'19114","last_scrub_stamp":"2024-04-29T14:49:57.142771+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:49:57.142771+0000","objects_scrubbed":2497,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:23.319557+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108134400,"num_objects":1650,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"994'12681","reported_seq":43879,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:55.403268+0000","last_change":"2024-04-29T14:49:58.542110+0000","last_active":"2024-04-29T14:50:55.403268+0000","last_peered":"2024-04-29T14:50:55.403268+0000","last_clean":"2024-04-29T14:50:55.403268+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:50:55.403268+0000","last_undegraded":"2024-04-29T14:50:55.403268+0000","last_fullsized":"2024-04-29T14:50:55.403268+0000","mapping_epoch":938,"log_start":"662'2582","ondisk_log_start":"662'2582","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'11857","last_scrub_stamp":"2024-04-29T14:49:58.541853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:58.541853+0000","objects_scrubbed":2470,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:16.374459+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107741208,"num_objects":1645,"num_object_clones":0,"num_object_copies":3290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12681,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"994'15924","reported_seq":81479,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:55.352089+0000","last_change":"2024-04-29T14:50:34.579123+0000","last_active":"2024-04-29T14:50:55.352089+0000","last_peered":"2024-04-29T14:50:55.352089+0000","last_clean":"2024-04-29T14:50:55.352089+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:50:55.352089+0000","last_undegraded":"2024-04-29T14:50:55.352089+0000","last_fullsized":"2024-04-29T14:50:55.352089+0000","mapping_epoch":939,"log_start":"675'5850","ondisk_log_start":"675'5850","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'15542","last_scrub_stamp":"2024-04-29T14:50:34.578998+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:50:34.578998+0000","objects_scrubbed":1966,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:38.386277+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"994'12698","reported_seq":30528,"reported_epoch":994,"state":"active+recovering","last_fresh":"2024-04-29T14:50:56.348650+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:50:56.348650+0000","last_peered":"2024-04-29T14:50:56.348650+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:50:56.348650+0000","last_undegraded":"2024-04-29T14:50:56.348650+0000","last_fullsized":"2024-04-29T14:50:56.348650+0000","mapping_epoch":942,"log_start":"662'2600","ondisk_log_start":"662'2600","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":755,"num_objects_missing":755,"num_objects_degraded":0,"num_objects_misplaced":2321,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6002,"num_bytes_recovered":393216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":18},{"shards":"1,10,11","objects":811},{"shards":"10,11","objects":755}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"994'20140","reported_seq":37663,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:55.071812+0000","last_change":"2024-04-29T14:50:45.604038+0000","last_active":"2024-04-29T14:50:55.071812+0000","last_peered":"2024-04-29T14:50:55.071812+0000","last_clean":"2024-04-29T14:50:55.071812+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:50:55.071812+0000","last_undegraded":"2024-04-29T14:50:55.071812+0000","last_fullsized":"2024-04-29T14:50:55.071812+0000","mapping_epoch":895,"log_start":"705'10100","ondisk_log_start":"705'10100","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"991'19938","last_scrub_stamp":"2024-04-29T14:50:45.603880+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:50:45.603880+0000","objects_scrubbed":1811,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:01.560720+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105381888,"num_objects":1608,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13066,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"994'12816","reported_seq":42001,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:56.325079+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:50:56.325079+0000","last_peered":"2024-04-29T14:50:56.325079+0000","last_clean":"2024-04-29T14:50:56.325079+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:50:56.325079+0000","last_undegraded":"2024-04-29T14:50:56.325079+0000","last_fullsized":"2024-04-29T14:50:56.325079+0000","mapping_epoch":756,"log_start":"662'2800","ondisk_log_start":"662'2800","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99876864,"num_objects":1524,"num_object_clones":0,"num_object_copies":3048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12816,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"994'12684","reported_seq":21793,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:56.174968+0000","last_change":"2024-04-29T14:50:12.400042+0000","last_active":"2024-04-29T14:50:56.174968+0000","last_peered":"2024-04-29T14:50:56.174968+0000","last_clean":"2024-04-29T14:50:56.174968+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:50:56.174968+0000","last_undegraded":"2024-04-29T14:50:56.174968+0000","last_fullsized":"2024-04-29T14:50:56.174968+0000","mapping_epoch":883,"log_start":"662'2600","ondisk_log_start":"662'2600","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'11960","last_scrub_stamp":"2024-04-29T14:50:12.399923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:12.399923+0000","objects_scrubbed":2330,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:26.243671+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105250816,"num_objects":1606,"num_object_clones":0,"num_object_copies":3212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12684,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"994'13064","reported_seq":71429,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:55.411313+0000","last_change":"2024-04-29T14:50:39.638903+0000","last_active":"2024-04-29T14:50:55.411313+0000","last_peered":"2024-04-29T14:50:55.411313+0000","last_clean":"2024-04-29T14:50:55.411313+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:50:55.411313+0000","last_undegraded":"2024-04-29T14:50:55.411313+0000","last_fullsized":"2024-04-29T14:50:55.411313+0000","mapping_epoch":939,"log_start":"663'2976","ondisk_log_start":"663'2976","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12718","last_scrub_stamp":"2024-04-29T14:50:39.638817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:39.638817+0000","objects_scrubbed":2012,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:07.740501+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":1664,"num_object_clones":0,"num_object_copies":3328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13064,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"994'12706","reported_seq":47008,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:55.332756+0000","last_change":"2024-04-29T14:50:36.645439+0000","last_active":"2024-04-29T14:50:55.332756+0000","last_peered":"2024-04-29T14:50:55.332756+0000","last_clean":"2024-04-29T14:50:55.332756+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:50:55.332756+0000","last_undegraded":"2024-04-29T14:50:55.332756+0000","last_fullsized":"2024-04-29T14:50:55.332756+0000","mapping_epoch":918,"log_start":"662'2700","ondisk_log_start":"662'2700","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12376","last_scrub_stamp":"2024-04-29T14:50:36.645270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:36.645270+0000","objects_scrubbed":1898,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:54.914264+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102760448,"num_objects":1568,"num_object_clones":0,"num_object_copies":3136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12706,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"994'12917","reported_seq":49318,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:53.986402+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:50:53.986402+0000","last_peered":"2024-04-29T14:50:53.986402+0000","last_clean":"2024-04-29T14:50:53.986402+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:50:53.986402+0000","last_undegraded":"2024-04-29T14:50:53.986402+0000","last_fullsized":"2024-04-29T14:50:53.986402+0000","mapping_epoch":915,"log_start":"663'2900","ondisk_log_start":"663'2900","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104005632,"num_objects":1587,"num_object_clones":0,"num_object_copies":3174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"993'16352","reported_seq":41153,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-04-29T14:50:53.242397+0000","last_change":"2024-04-29T14:50:11.012445+0000","last_active":"2024-04-29T14:50:53.242397+0000","last_peered":"2024-04-29T14:50:53.242397+0000","last_clean":"2024-04-29T14:50:53.242397+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:50:53.242397+0000","last_undegraded":"2024-04-29T14:50:53.242397+0000","last_fullsized":"2024-04-29T14:50:53.242397+0000","mapping_epoch":895,"log_start":"675'6325","ondisk_log_start":"675'6325","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'15627","last_scrub_stamp":"2024-04-29T14:50:11.012229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:11.012229+0000","objects_scrubbed":2385,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.431973+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":3310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12753,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"993'12600","reported_seq":37082,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-04-29T14:50:53.254310+0000","last_change":"2024-04-29T14:50:04.932139+0000","last_active":"2024-04-29T14:50:53.254310+0000","last_peered":"2024-04-29T14:50:53.254310+0000","last_clean":"2024-04-29T14:50:53.254310+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:50:53.254310+0000","last_undegraded":"2024-04-29T14:50:53.254310+0000","last_fullsized":"2024-04-29T14:50:53.254310+0000","mapping_epoch":895,"log_start":"662'2544","ondisk_log_start":"662'2544","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11810","last_scrub_stamp":"2024-04-29T14:50:04.932001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:04.932001+0000","objects_scrubbed":2381,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:25.466879+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104202240,"num_objects":1590,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12600,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"994'12614","reported_seq":59430,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:54.014466+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:50:54.014466+0000","last_peered":"2024-04-29T14:50:54.014466+0000","last_clean":"2024-04-29T14:50:54.014466+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:50:54.014466+0000","last_undegraded":"2024-04-29T14:50:54.014466+0000","last_fullsized":"2024-04-29T14:50:54.014466+0000","mapping_epoch":928,"log_start":"662'2600","ondisk_log_start":"662'2600","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12614,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"993'12694","reported_seq":27697,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:53.605088+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:50:53.605088+0000","last_peered":"2024-04-29T14:50:53.605088+0000","last_clean":"2024-04-29T14:50:53.605088+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:50:53.605088+0000","last_undegraded":"2024-04-29T14:50:53.605088+0000","last_fullsized":"2024-04-29T14:50:53.605088+0000","mapping_epoch":703,"log_start":"662'2600","ondisk_log_start":"662'2600","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101580800,"num_objects":1550,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12694,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"994'12499","reported_seq":33876,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:55.970945+0000","last_change":"2024-04-29T14:50:05.324932+0000","last_active":"2024-04-29T14:50:55.970945+0000","last_peered":"2024-04-29T14:50:55.970945+0000","last_clean":"2024-04-29T14:50:55.970945+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:50:55.970945+0000","last_undegraded":"2024-04-29T14:50:55.970945+0000","last_fullsized":"2024-04-29T14:50:55.970945+0000","mapping_epoch":865,"log_start":"662'2400","ondisk_log_start":"662'2400","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11707","last_scrub_stamp":"2024-04-29T14:50:05.324827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.324827+0000","objects_scrubbed":2336,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:05.726927+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100990976,"num_objects":1541,"num_object_clones":0,"num_object_copies":3082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1974,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-04-29T14:50:53.612648+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:50:53.612648+0000","last_peered":"2024-04-29T14:50:53.612648+0000","last_clean":"2024-04-29T14:50:53.612648+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:50:53.612648+0000","last_undegraded":"2024-04-29T14:50:53.612648+0000","last_fullsized":"2024-04-29T14:50:53.612648+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1667235864,"num_objects":25441,"num_object_clones":0,"num_object_copies":50882,"num_objects_missing_on_primary":755,"num_objects_missing":755,"num_objects_degraded":0,"num_objects_misplaced":2321,"num_objects_unfound":0,"num_objects_dirty":25441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204379,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232986,"num_bytes_recovered":13667205144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4280229888,"data_stored":4280221744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122577,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384216,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1645500,"kb_used_data":766060,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92099652,"statfs":{"total":95995035648,"available":94310043648,"internally_reserved":0,"allocated":784445440,"data_stored":782362813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":900528576},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645851,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1224052,"kb_used_data":412652,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92521100,"statfs":{"total":95995035648,"available":94741606400,"internally_reserved":0,"allocated":422555648,"data_stored":420473041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":830863808},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423155,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":453620,"kb_used_data":308652,"kb_used_omap":4,"kb_used_meta":144955,"kb_avail":93291532,"statfs":{"total":95995035648,"available":95530528768,"internally_reserved":0,"allocated":316059648,"data_stored":313977041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":148434624},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234532,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067452,"kb_used_data":206068,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92677700,"statfs":{"total":95995035648,"available":94901964800,"internally_reserved":0,"allocated":211013632,"data_stored":208927473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":882047434},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528873,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301312,"kb_used_data":317752,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":92443840,"statfs":{"total":95995035648,"available":94662492160,"internally_reserved":0,"allocated":325378048,"data_stored":323287817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1007155654},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575413,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742648,"kb_used_data":260272,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93002504,"statfs":{"total":95995035648,"available":95234564096,"internally_reserved":0,"allocated":266518528,"data_stored":264431849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,6,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738951,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1318520,"kb_used_data":398828,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92426632,"statfs":{"total":95995035648,"available":94644871168,"internally_reserved":0,"allocated":408399872,"data_stored":406317265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497385,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182970,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":996780,"kb_used_data":420000,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92748372,"statfs":{"total":95995035648,"available":94974332928,"internally_reserved":0,"allocated":430080000,"data_stored":428003678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":590608840},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986735,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1381628,"kb_used_data":416108,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":92363524,"statfs":{"total":95995035648,"available":94580248576,"internally_reserved":0,"allocated":426094592,"data_stored":424011985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":988674501},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653296,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":476648,"kb_used_data":266208,"kb_used_omap":651,"kb_used_meta":209780,"kb_avail":93268504,"statfs":{"total":95995035648,"available":95506948096,"internally_reserved":0,"allocated":272596992,"data_stored":270520670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":214815496},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":403963904,"data_stored":403963904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260575232,"data_stored":260571160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":425656320,"data_stored":425656320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":421855232,"data_stored":421855232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":268500992,"data_stored":268500992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":320606208,"data_stored":320602136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":206176256,"data_stored":206176256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":310312960,"data_stored":310312960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":415498240,"data_stored":415498240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":779943936,"data_stored":779943936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:50:59.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:50:59.409 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:59.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:59.439 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:50:59.513 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:50:59.541 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:59.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:59 smithi040 ceph-mon[39581]: pgmap v2501: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 266 op/s; 2321/50886 objects misplaced (4.561%); 1002 KiB/s, 15 objects/s recovering 2024-04-29T14:50:59.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:50:59 smithi040 ceph-mon[39581]: osdmap e996: 12 total, 12 up, 10 in 2024-04-29T14:50:59.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:59 smithi121 ceph-mon[44533]: pgmap v2501: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 266 op/s; 2321/50886 objects misplaced (4.561%); 1002 KiB/s, 15 objects/s recovering 2024-04-29T14:50:59.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:50:59 smithi121 ceph-mon[44533]: osdmap e996: 12 total, 12 up, 10 in 2024-04-29T14:50:59.636 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:59 smithi194 ceph-mon[44089]: pgmap v2501: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 266 op/s; 2321/50886 objects misplaced (4.561%); 1002 KiB/s, 15 objects/s recovering 2024-04-29T14:50:59.636 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:50:59 smithi194 ceph-mon[44089]: osdmap e996: 12 total, 12 up, 10 in 2024-04-29T14:50:59.640 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:50:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:59.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:50:59.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:50:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:50:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:59.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:50:59.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:50:59.935 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:50:59.953 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:50:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:50:59.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:00.057 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:00.090 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:00.289 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:51:00.289 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:51:00.291 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:00.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:00.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:00.422 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:00.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:00.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:00.547 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:00.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:00 smithi194 ceph-mon[44089]: from='client.18288 v1:172.21.15.40:0/3148664209' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:00.600 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:00.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:00.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:00 smithi040 ceph-mon[39581]: from='client.18288 v1:172.21.15.40:0/3148664209' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:00.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:00 smithi121 ceph-mon[44533]: from='client.18288 v1:172.21.15.40:0/3148664209' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:00.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:00.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:51:00.731 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:00.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:00.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:00.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:01.088 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:01.115 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:01.136 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:01.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:01.152 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:51:01.241 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:01.460 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:51:01.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:01 smithi121 ceph-mon[44533]: pgmap v2503: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 276 op/s; 2321/49902 objects misplaced (4.651%); 897 KiB/s, 14 objects/s recovering 2024-04-29T14:51:01.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:01 smithi194 ceph-mon[44089]: pgmap v2503: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 276 op/s; 2321/49902 objects misplaced (4.651%); 897 KiB/s, 14 objects/s recovering 2024-04-29T14:51:01.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:01 smithi040 ceph-mon[39581]: pgmap v2503: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 276 op/s; 2321/49902 objects misplaced (4.651%); 897 KiB/s, 14 objects/s recovering 2024-04-29T14:51:02.041 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:02.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:02.148 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:02.176 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:02.195 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:02.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:02.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:02.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:02.389 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:51:02.482 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:02.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:02.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3176042853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:02.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:02.615 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:02.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3176042853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:02.740 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:02.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:02.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:02.877 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3176042853' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:02.979 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:02.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:03.084 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:03.114 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:03.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:03.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:03.316 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:51:03.317 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:51:03.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:03.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:03.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:03.518 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:03.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:03.626 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:03.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:03 smithi121 ceph-mon[44533]: pgmap v2504: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 310 op/s; 2131/48056 objects misplaced (4.434%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:03.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:03.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:03.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:03 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:03.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3176042853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:03.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:03 smithi121 ceph-mon[44533]: osdmap e997: 12 total, 12 up, 10 in 2024-04-29T14:51:03.652 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:03 smithi194 ceph-mon[44089]: pgmap v2504: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 310 op/s; 2131/48056 objects misplaced (4.434%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:03.652 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:03.652 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:03.652 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:03 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:03.652 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3176042853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:03.652 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:03 smithi194 ceph-mon[44089]: osdmap e997: 12 total, 12 up, 10 in 2024-04-29T14:51:03.655 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:03 smithi040 ceph-mon[39581]: pgmap v2504: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 310 op/s; 2131/48056 objects misplaced (4.434%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:03 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3176042853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:03 smithi040 ceph-mon[39581]: osdmap e997: 12 total, 12 up, 10 in 2024-04-29T14:51:04.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:51:04.072 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:51:04.072 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:51:04.105 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:04.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:04.210 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:04.236 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:04.254 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:04.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:04.340 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:51:04.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:04.386 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:04.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:04.719 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:04.746 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:04.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:04.811 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2505,"stamp":"2024-04-29T14:51:02.382939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1575092792,"num_objects":24028,"num_object_clones":0,"num_object_copies":48056,"num_objects_missing_on_primary":659,"num_objects_missing":659,"num_objects_degraded":0,"num_objects_misplaced":2131,"num_objects_unfound":0,"num_objects_dirty":24028,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":205975,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233094,"num_bytes_recovered":13677068024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161185,"ondisk_log_size":161185,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":10441576,"kb_used_data":3597672,"kb_used_omap":668,"kb_used_meta":6843107,"kb_avail":927009944,"statfs":{"total":959950356480,"available":949258182656,"internally_reserved":0,"allocated":3684016128,"data_stored":3663062060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7007342575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-105316352,"num_objects":-1607,"num_object_clones":0,"num_object_copies":-3214,"num_objects_missing_on_primary":-187,"num_objects_missing":-187,"num_objects_degraded":0,"num_objects_misplaced":-375,"num_objects_unfound":0,"num_objects_dirty":-1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1607,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":12320768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.071963"},"pg_stats":[{"pgid":"7.f","version":"996'12994","reported_seq":48004,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:00.157537+0000","last_change":"2024-04-29T14:50:38.109612+0000","last_active":"2024-04-29T14:51:00.157537+0000","last_peered":"2024-04-29T14:51:00.157537+0000","last_clean":"2024-04-29T14:51:00.157537+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:51:00.157537+0000","last_undegraded":"2024-04-29T14:51:00.157537+0000","last_fullsized":"2024-04-29T14:51:00.157537+0000","mapping_epoch":918,"log_start":"663'2900","ondisk_log_start":"663'2900","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12570","last_scrub_stamp":"2024-04-29T14:50:38.109478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:38.109478+0000","objects_scrubbed":1874,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:03.361035+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95027200,"num_objects":1450,"num_object_clones":0,"num_object_copies":2900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12994,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"996'20035","reported_seq":42085,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:50:59.260004+0000","last_change":"2024-04-29T14:49:57.142920+0000","last_active":"2024-04-29T14:50:59.260004+0000","last_peered":"2024-04-29T14:50:59.260004+0000","last_clean":"2024-04-29T14:50:59.260004+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:50:59.260004+0000","last_undegraded":"2024-04-29T14:50:59.260004+0000","last_fullsized":"2024-04-29T14:50:59.260004+0000","mapping_epoch":930,"log_start":"704'9958","ondisk_log_start":"704'9958","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'19114","last_scrub_stamp":"2024-04-29T14:49:57.142771+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:49:57.142771+0000","objects_scrubbed":2497,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:23.319557+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12979,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"996'12763","reported_seq":43965,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:00.538518+0000","last_change":"2024-04-29T14:49:58.542110+0000","last_active":"2024-04-29T14:51:00.538518+0000","last_peered":"2024-04-29T14:51:00.538518+0000","last_clean":"2024-04-29T14:51:00.538518+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:51:00.538518+0000","last_undegraded":"2024-04-29T14:51:00.538518+0000","last_fullsized":"2024-04-29T14:51:00.538518+0000","mapping_epoch":938,"log_start":"662'2682","ondisk_log_start":"662'2682","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'11857","last_scrub_stamp":"2024-04-29T14:49:58.541853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:58.541853+0000","objects_scrubbed":2470,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:16.374459+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367256,"num_objects":1563,"num_object_clones":0,"num_object_copies":3126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12763,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"996'16013","reported_seq":81572,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:00.541595+0000","last_change":"2024-04-29T14:50:34.579123+0000","last_active":"2024-04-29T14:51:00.541595+0000","last_peered":"2024-04-29T14:51:00.541595+0000","last_clean":"2024-04-29T14:51:00.541595+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:51:00.541595+0000","last_undegraded":"2024-04-29T14:51:00.541595+0000","last_fullsized":"2024-04-29T14:51:00.541595+0000","mapping_epoch":939,"log_start":"675'5950","ondisk_log_start":"675'5950","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'15542","last_scrub_stamp":"2024-04-29T14:50:34.578998+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:50:34.578998+0000","objects_scrubbed":1966,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:38.386277+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97976320,"num_objects":1495,"num_object_clones":0,"num_object_copies":2990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12853,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"996'12794","reported_seq":30911,"reported_epoch":996,"state":"active+recovering","last_fresh":"2024-04-29T14:51:01.369267+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:51:01.369267+0000","last_peered":"2024-04-29T14:51:01.369267+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:51:01.369267+0000","last_undegraded":"2024-04-29T14:51:01.369267+0000","last_fullsized":"2024-04-29T14:51:01.369267+0000","mapping_epoch":942,"log_start":"662'2700","ondisk_log_start":"662'2700","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97517568,"num_objects":1488,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":659,"num_objects_missing":659,"num_objects_degraded":0,"num_objects_misplaced":2131,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12794,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6096,"num_bytes_recovered":399376384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":16},{"shards":"1,10,11","objects":813},{"shards":"10,11","objects":659}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"996'20227","reported_seq":37755,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:00.417759+0000","last_change":"2024-04-29T14:50:45.604038+0000","last_active":"2024-04-29T14:51:00.417759+0000","last_peered":"2024-04-29T14:51:00.417759+0000","last_clean":"2024-04-29T14:51:00.417759+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:51:00.417759+0000","last_undegraded":"2024-04-29T14:51:00.417759+0000","last_fullsized":"2024-04-29T14:51:00.417759+0000","mapping_epoch":895,"log_start":"705'10200","ondisk_log_start":"705'10200","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"991'19938","last_scrub_stamp":"2024-04-29T14:50:45.603880+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:50:45.603880+0000","objects_scrubbed":1811,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:01.560720+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99680256,"num_objects":1521,"num_object_clones":0,"num_object_copies":3042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"996'12917","reported_seq":42106,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:01.067870+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:51:01.067870+0000","last_peered":"2024-04-29T14:51:01.067870+0000","last_clean":"2024-04-29T14:51:01.067870+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:51:01.067870+0000","last_undegraded":"2024-04-29T14:51:01.067870+0000","last_fullsized":"2024-04-29T14:51:01.067870+0000","mapping_epoch":756,"log_start":"663'2900","ondisk_log_start":"663'2900","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93257728,"num_objects":1423,"num_object_clones":0,"num_object_copies":2846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"996'12783","reported_seq":21896,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:00.995913+0000","last_change":"2024-04-29T14:50:12.400042+0000","last_active":"2024-04-29T14:51:00.995913+0000","last_peered":"2024-04-29T14:51:00.995913+0000","last_clean":"2024-04-29T14:51:00.995913+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:51:00.995913+0000","last_undegraded":"2024-04-29T14:51:00.995913+0000","last_fullsized":"2024-04-29T14:51:00.995913+0000","mapping_epoch":883,"log_start":"662'2700","ondisk_log_start":"662'2700","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'11960","last_scrub_stamp":"2024-04-29T14:50:12.399923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:12.399923+0000","objects_scrubbed":2330,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:26.243671+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98762752,"num_objects":1507,"num_object_clones":0,"num_object_copies":3014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12783,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"996'13172","reported_seq":71541,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:00.476997+0000","last_change":"2024-04-29T14:50:39.638903+0000","last_active":"2024-04-29T14:51:00.476997+0000","last_peered":"2024-04-29T14:51:00.476997+0000","last_clean":"2024-04-29T14:51:00.476997+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:51:00.476997+0000","last_undegraded":"2024-04-29T14:51:00.476997+0000","last_fullsized":"2024-04-29T14:51:00.476997+0000","mapping_epoch":939,"log_start":"663'3076","ondisk_log_start":"663'3076","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12718","last_scrub_stamp":"2024-04-29T14:50:39.638817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:39.638817+0000","objects_scrubbed":2012,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:07.740501+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101974016,"num_objects":1556,"num_object_clones":0,"num_object_copies":3112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13172,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"996'12815","reported_seq":47121,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:00.383541+0000","last_change":"2024-04-29T14:50:36.645439+0000","last_active":"2024-04-29T14:51:00.383541+0000","last_peered":"2024-04-29T14:51:00.383541+0000","last_clean":"2024-04-29T14:51:00.383541+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:51:00.383541+0000","last_undegraded":"2024-04-29T14:51:00.383541+0000","last_fullsized":"2024-04-29T14:51:00.383541+0000","mapping_epoch":918,"log_start":"663'2800","ondisk_log_start":"663'2800","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12376","last_scrub_stamp":"2024-04-29T14:50:36.645270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:36.645270+0000","objects_scrubbed":1898,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:54.914264+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95617024,"num_objects":1459,"num_object_clones":0,"num_object_copies":2918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12815,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"996'12993","reported_seq":49398,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:50:58.983697+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:50:58.983697+0000","last_peered":"2024-04-29T14:50:58.983697+0000","last_clean":"2024-04-29T14:50:58.983697+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:50:58.983697+0000","last_undegraded":"2024-04-29T14:50:58.983697+0000","last_fullsized":"2024-04-29T14:50:58.983697+0000","mapping_epoch":915,"log_start":"663'2900","ondisk_log_start":"663'2900","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99024896,"num_objects":1511,"num_object_clones":0,"num_object_copies":3022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12993,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"995'16430","reported_seq":41235,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-04-29T14:50:58.323097+0000","last_change":"2024-04-29T14:50:11.012445+0000","last_active":"2024-04-29T14:50:58.323097+0000","last_peered":"2024-04-29T14:50:58.323097+0000","last_clean":"2024-04-29T14:50:58.323097+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:50:58.323097+0000","last_undegraded":"2024-04-29T14:50:58.323097+0000","last_fullsized":"2024-04-29T14:50:58.323097+0000","mapping_epoch":895,"log_start":"679'6425","ondisk_log_start":"679'6425","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'15627","last_scrub_stamp":"2024-04-29T14:50:11.012229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:11.012229+0000","objects_scrubbed":2385,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.431973+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":3154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12831,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"995'12685","reported_seq":37171,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-04-29T14:50:58.300403+0000","last_change":"2024-04-29T14:50:04.932139+0000","last_active":"2024-04-29T14:50:58.300403+0000","last_peered":"2024-04-29T14:50:58.300403+0000","last_clean":"2024-04-29T14:50:58.300403+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:50:58.300403+0000","last_undegraded":"2024-04-29T14:50:58.300403+0000","last_fullsized":"2024-04-29T14:50:58.300403+0000","mapping_epoch":895,"log_start":"662'2644","ondisk_log_start":"662'2644","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11810","last_scrub_stamp":"2024-04-29T14:50:04.932001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:04.932001+0000","objects_scrubbed":2381,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:25.466879+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98631680,"num_objects":1505,"num_object_clones":0,"num_object_copies":3010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12685,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"996'12696","reported_seq":59516,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:50:59.220515+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:50:59.220515+0000","last_peered":"2024-04-29T14:50:59.220515+0000","last_clean":"2024-04-29T14:50:59.220515+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:50:59.220515+0000","last_undegraded":"2024-04-29T14:50:59.220515+0000","last_fullsized":"2024-04-29T14:50:59.220515+0000","mapping_epoch":928,"log_start":"662'2600","ondisk_log_start":"662'2600","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97124352,"num_objects":1482,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"996'12786","reported_seq":27793,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:50:58.613285+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:50:58.613285+0000","last_peered":"2024-04-29T14:50:58.613285+0000","last_clean":"2024-04-29T14:50:58.613285+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:50:58.613285+0000","last_undegraded":"2024-04-29T14:50:58.613285+0000","last_fullsized":"2024-04-29T14:50:58.613285+0000","mapping_epoch":703,"log_start":"662'2700","ondisk_log_start":"662'2700","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"996'12580","reported_seq":33961,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:51:01.088863+0000","last_change":"2024-04-29T14:50:05.324932+0000","last_active":"2024-04-29T14:51:01.088863+0000","last_peered":"2024-04-29T14:51:01.088863+0000","last_clean":"2024-04-29T14:51:01.088863+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:51:01.088863+0000","last_undegraded":"2024-04-29T14:51:01.088863+0000","last_fullsized":"2024-04-29T14:51:01.088863+0000","mapping_epoch":865,"log_start":"662'2500","ondisk_log_start":"662'2500","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11707","last_scrub_stamp":"2024-04-29T14:50:05.324827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.324827+0000","objects_scrubbed":2336,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:05.726927+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95682560,"num_objects":1460,"num_object_clones":0,"num_object_copies":2920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1978,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-04-29T14:50:58.615494+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:50:58.615494+0000","last_peered":"2024-04-29T14:50:58.615494+0000","last_clean":"2024-04-29T14:50:58.615494+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:50:58.615494+0000","last_undegraded":"2024-04-29T14:50:58.615494+0000","last_fullsized":"2024-04-29T14:50:58.615494+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1574502424,"num_objects":24026,"num_object_clones":0,"num_object_copies":48052,"num_objects_missing_on_primary":659,"num_objects_missing":659,"num_objects_degraded":0,"num_objects_misplaced":2131,"num_objects_unfound":0,"num_objects_dirty":24026,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205794,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233080,"num_bytes_recovered":13673365528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096532480,"data_stored":4096524336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161048,"ondisk_log_size":161048,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122578,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384217,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1633620,"kb_used_data":750020,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92111532,"statfs":{"total":95995035648,"available":94322208768,"internally_reserved":0,"allocated":768020480,"data_stored":765925333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":904788416},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1205332,"kb_used_data":389764,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92539820,"statfs":{"total":95995035648,"available":94760775680,"internally_reserved":0,"allocated":399118336,"data_stored":397023209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423156,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440340,"kb_used_data":291204,"kb_used_omap":4,"kb_used_meta":149115,"kb_avail":93304812,"statfs":{"total":95995035648,"available":95544127488,"internally_reserved":0,"allocated":298192896,"data_stored":296097769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":152694464},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234533,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055828,"kb_used_data":194444,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92689324,"statfs":{"total":95995035648,"available":94913867776,"internally_reserved":0,"allocated":199110656,"data_stored":197011977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":882047434},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289312,"kb_used_data":301584,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92455840,"statfs":{"total":95995035648,"available":94674780160,"internally_reserved":0,"allocated":308822016,"data_stored":306719265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575414,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":730068,"kb_used_data":247688,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93015084,"statfs":{"total":95995035648,"available":95247446016,"internally_reserved":0,"allocated":253632512,"data_stored":251533313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738952,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301644,"kb_used_data":377796,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92443508,"statfs":{"total":95995035648,"available":94662152192,"internally_reserved":0,"allocated":386863104,"data_stored":384767977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497386,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182971,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976132,"kb_used_data":399352,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92769020,"statfs":{"total":95995035648,"available":94995476480,"internally_reserved":0,"allocated":408936448,"data_stored":406847556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":590608840},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986736,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1364304,"kb_used_data":394628,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92380848,"statfs":{"total":95995035648,"available":94597988352,"internally_reserved":0,"allocated":404099072,"data_stored":402003945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":444996,"kb_used_data":251192,"kb_used_omap":651,"kb_used_meta":193140,"kb_avail":93300156,"statfs":{"total":95995035648,"available":95539359744,"internally_reserved":0,"allocated":257220608,"data_stored":255131716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":197776136},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":382140416,"data_stored":382140416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248713216,"data_stored":248709144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":404553728,"data_stored":404553728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":399310848,"data_stored":399310848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":252510208,"data_stored":252510208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":302190592,"data_stored":302186520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":194445312,"data_stored":194445312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":292028416,"data_stored":292028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":390397952,"data_stored":390397952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":763101184,"data_stored":763101184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:51:04.814 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.b is not active+clean 2024-04-29T14:51:04.814 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.b', 'version': "996'12794", 'reported_seq': 30911, 'reported_epoch': 996, 'state': 'active+recovering', 'last_fresh': '2024-04-29T14:51:01.369267+0000', 'last_change': '2024-04-29T14:48:55.372453+0000', 'last_active': '2024-04-29T14:51:01.369267+0000', 'last_peered': '2024-04-29T14:51:01.369267+0000', 'last_clean': '2024-04-29T14:35:46.185312+0000', 'last_became_active': '2024-04-29T14:48:48.169551+0000', 'last_became_peered': '2024-04-29T14:48:48.169551+0000', 'last_unstale': '2024-04-29T14:51:01.369267+0000', 'last_undegraded': '2024-04-29T14:51:01.369267+0000', 'last_fullsized': '2024-04-29T14:51:01.369267+0000', 'mapping_epoch': 942, 'log_start': "662'2700", 'ondisk_log_start': "662'2700", 'created': 655, 'last_epoch_clean': 656, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_clean_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 97517568, 'num_objects': 1488, 'num_object_clones': 0, 'num_object_copies': 2976, 'num_objects_missing_on_primary': 659, 'num_objects_missing': 659, 'num_objects_degraded': 0, 'num_objects_misplaced': 2131, 'num_objects_unfound': 0, 'num_objects_dirty': 1488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12794, 'num_write_kb': 457024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6096, 'num_bytes_recovered': 399376384, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': ['4', '11'], 'object_location_counts': [{'shards': '1,5', 'objects': 16}, {'shards': '1,10,11', 'objects': 813}, {'shards': '10,11', 'objects': 659}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-29T14:51:04.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:51:04.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:04.899 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:04.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:05.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:05.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:05.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:05.545 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:05.547 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:51:05.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:05 smithi121 ceph-mon[44533]: from='client.28024 v1:172.21.15.40:0/1855337780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:05.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:05 smithi121 ceph-mon[44533]: pgmap v2506: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 281 op/s; 2131/47110 objects misplaced (4.523%); 897 KiB/s, 14 objects/s recovering 2024-04-29T14:51:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:05 smithi194 ceph-mon[44089]: from='client.28024 v1:172.21.15.40:0/1855337780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:05 smithi194 ceph-mon[44089]: pgmap v2506: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 281 op/s; 2131/47110 objects misplaced (4.523%); 897 KiB/s, 14 objects/s recovering 2024-04-29T14:51:05.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:05 smithi040 ceph-mon[39581]: from='client.28024 v1:172.21.15.40:0/1855337780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:05.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:05 smithi040 ceph-mon[39581]: pgmap v2506: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 281 op/s; 2131/47110 objects misplaced (4.523%); 897 KiB/s, 14 objects/s recovering 2024-04-29T14:51:05.880 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:05.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:05.911 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:05.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:06.019 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:06.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:06.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:06.158 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:51:06.158 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:51:06.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:06.255 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:06.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:06.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:06.509 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:06.535 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:06.554 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:06.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:06.658 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:06.685 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:06.705 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:06.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:06.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:06.843 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:06.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:51:07.164 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:51:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:07 smithi040 ceph-mon[39581]: pgmap v2507: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 338 op/s; 2131/45468 objects misplaced (4.687%); 752 KiB/s, 11 objects/s recovering 2024-04-29T14:51:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1741678939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.680 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:07 smithi121 ceph-mon[44533]: pgmap v2507: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 338 op/s; 2131/45468 objects misplaced (4.687%); 752 KiB/s, 11 objects/s recovering 2024-04-29T14:51:07.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1741678939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:07.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:51:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:51:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:51:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:07 smithi194 ceph-mon[44089]: pgmap v2507: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 338 op/s; 2131/45468 objects misplaced (4.687%); 752 KiB/s, 11 objects/s recovering 2024-04-29T14:51:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1741678939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:08 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:08 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:08 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:08.995 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:09.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:09.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:09.134 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:09.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:09.215 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:51:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:51:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:51:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:51:09.215 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:51:09.264 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:09.291 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:09.312 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:09.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:09.416 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:09.441 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:09.460 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:09.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:09.563 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:09.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:09 smithi194 ceph-mon[44089]: pgmap v2508: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 281 op/s; 1926/45090 objects misplaced (4.271%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:51:09.581 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:09 smithi121 ceph-mon[44533]: pgmap v2508: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 281 op/s; 1926/45090 objects misplaced (4.271%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:51:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:09.608 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:09.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:09.670 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:09 smithi040 ceph-mon[39581]: pgmap v2508: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 281 op/s; 1926/45090 objects misplaced (4.271%); 1.3 MiB/s, 20 objects/s recovering 2024-04-29T14:51:09.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:09.738 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:09.758 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:09.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:09.862 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:09.888 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:09.906 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:09.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:09.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:10.010 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:10.039 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:10.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:51:10.060 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:10.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:10.164 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:10.191 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:10.210 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:10.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:10.314 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:10.331 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:51:10.341 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:10.360 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:10.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:10.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:10.492 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:10.591 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:10.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:10.613 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:51:10.613 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2944,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":997,"num_osds":12,"num_up_osds":12,"osd_up_since":1714401063,"num_in_osds":10,"osd_in_since":1714402012,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":21968,"data_bytes":1440088632,"bytes_used":10425081856,"bytes_avail":949525274624,"bytes_total":959950356480,"misplaced_objects":1926,"misplaced_total":43936,"misplaced_ratio":0.043836489439184258,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":1290837,"recovering_keys_per_sec":0,"num_objects_recovered":197,"num_bytes_recovered":12910592,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":298},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":72,"modified":"2024-04-29T14:50:02.301403+0000","services":{}},"progress_events":{}} 2024-04-29T14:51:10.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:10.722 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:10.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:10.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:10.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:10.953 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:10.973 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:10.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:10.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:51:11.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:11.104 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:11.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:11.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:11.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:11.266 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:11.404 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:51:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:11 smithi121 ceph-mon[44533]: pgmap v2509: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 298 op/s; 1926/43936 objects misplaced (4.384%); 1.2 MiB/s, 19 objects/s recovering 2024-04-29T14:51:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1713314879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:51:11.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/298373293' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:51:11.637 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:51:11.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:51:11.669 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:11 smithi040 ceph-mon[39581]: pgmap v2509: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 298 op/s; 1926/43936 objects misplaced (4.384%); 1.2 MiB/s, 19 objects/s recovering 2024-04-29T14:51:11.669 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1713314879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:51:11.669 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/298373293' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:51:11.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:11.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:11 smithi194 ceph-mon[44089]: pgmap v2509: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 298 op/s; 1926/43936 objects misplaced (4.384%); 1.2 MiB/s, 19 objects/s recovering 2024-04-29T14:51:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1713314879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:51:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/298373293' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:51:11.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:12.005 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:12.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:12.110 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:12.137 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:12.157 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:12.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:12.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:12.291 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:12.312 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:12.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152143 7.1 7:87f920be:::benchmark_data_smithi194_147200_object37231:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e831)", 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.531945+0000", 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.81378295299999, 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27172652000000003, 2024-04-29T14:51:12.353 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152143 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.531945+0000", 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.531945+0000", 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.531948+0000", 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1149999999999998e-06 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.531957+0000", 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6549999999999993e-06 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.531978+0000", 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0706000000000001e-05 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.531990+0000", 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2325e-05 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.533944+0000", 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019537090000000001 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.533957+0000", 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2901999999999999e-05 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.777232+0000", 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24327511800000001 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.777344+0000", 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112744 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.777443+0000", 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8474000000000006e-05 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.786727+0000", 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092845080000000003 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.786838+0000", 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110587 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.802024+0000", 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015186320999999999 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.802069+0000", 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4777000000000002e-05 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.803672+0000", 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001602579 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152150 7.1 7:8ce0d202:::benchmark_data_smithi194_147200_object37238:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e831)", 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.532016+0000", 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.81371212300002, 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27776988899999999, 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152150 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532016+0000", 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532016+0000", 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532017+0000", 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2500000000000005e-07 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532019+0000", 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6359999999999998e-06 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532024+0000", 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.135e-06 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532029+0000", 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0790000000000002e-06 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.533988+0000", 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019598850000000002 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:51:12.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.533996+0000", 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3409999999999997e-06 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.781532+0000", 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24753607399999999 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.781644+0000", 2024-04-29T14:51:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112032 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.781741+0000", 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6964000000000002e-05 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.786847+0000", 2024-04-29T14:51:12.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051060510000000003 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.786899+0000", 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1693000000000001e-05 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.808157+0000", 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021258691999999999 2024-04-29T14:51:12.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.808201+0000", 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3911e-05 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.809786+0000", 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001584671 2024-04-29T14:51:12.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152172 7.1 7:81346e8c:::benchmark_data_smithi194_147200_object37260:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e831)", 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.532033+0000", 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.81369534200002, 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27905529899999998, 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.372 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152172 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532033+0000", 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532033+0000", 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532033+0000", 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4999999999999998e-07 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532035+0000", 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7710000000000001e-06 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532038+0000", 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0469999999999998e-06 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532041+0000", 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8760000000000001e-06 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534029+0000", 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019877459999999999 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:51:12.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534036+0000", 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8210000000000002e-06 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.784093+0000", 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25005680200000002 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.784239+0000", 2024-04-29T14:51:12.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014578700000000001 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.784398+0000", 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015867100000000001 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.786970+0000", 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025721889999999999 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.786987+0000", 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.694e-05 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.809833+0000", 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022845876000000001 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.809873+0000", 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0102000000000003e-05 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.811088+0000", 2024-04-29T14:51:12.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012153209999999999 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152245 7.1 7:80015675:::benchmark_data_smithi194_147200_object37333:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.532043+0000", 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.81368463799998, 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.291737886, 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.381 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152245 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532043+0000", 2024-04-29T14:51:12.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532043+0000", 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532044+0000", 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.34e-07 2024-04-29T14:51:12.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532046+0000", 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532049+0000", 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.96e-06 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532052+0000", 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9909999999999998e-06 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534062+0000", 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002010085 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534066+0000", 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.946e-06 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.787011+0000", 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25294511600000003 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.787168+0000", 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000157114 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.787320+0000", 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015262500000000001 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796214+0000", 2024-04-29T14:51:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088936980000000002 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796234+0000", 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9752e-05 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822131+0000", 2024-04-29T14:51:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025897612 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822173+0000", 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1884000000000001e-05 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.823781+0000", 2024-04-29T14:51:12.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016079639999999999 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152250 7.1 7:8216457e:::benchmark_data_smithi194_147200_object37338:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.532055+0000", 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.81367325999997, 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30803025499999998, 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.390 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152250 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532055+0000", 2024-04-29T14:51:12.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532055+0000", 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532055+0000", 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9900000000000002e-07 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532057+0000", 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.742e-06 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532059+0000", 2024-04-29T14:51:12.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5790000000000002e-06 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532063+0000", 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5499999999999999e-06 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534097+0000", 2024-04-29T14:51:12.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002033923 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534104+0000", 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2450000000000001e-06 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.789952+0000", 2024-04-29T14:51:12.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25584754799999998 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.790127+0000", 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017497300000000001 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.790222+0000", 2024-04-29T14:51:12.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4889000000000006e-05 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796276+0000", 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060543569999999998 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796357+0000", 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1018000000000003e-05 2024-04-29T14:51:12.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.838528+0000", 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.042171475999999999 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.838559+0000", 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0128999999999999e-05 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840085+0000", 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001526527 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152251 7.1 7:8a37d68d:::benchmark_data_smithi194_147200_object37339:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.532068+0000", 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.81366050299999, 2024-04-29T14:51:12.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30913171499999997, 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152251 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532068+0000", 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532068+0000", 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532068+0000", 2024-04-29T14:51:12.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.58e-07 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532070+0000", 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8250000000000001e-06 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532073+0000", 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7999999999999999e-06 2024-04-29T14:51:12.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.532076+0000", 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0750000000000002e-06 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534126+0000", 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020503589999999999 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.534134+0000", 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2369999999999992e-06 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.792504+0000", 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25837019900000002 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.792661+0000", 2024-04-29T14:51:12.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000156384 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.792814+0000", 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015332799999999999 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796339+0000", 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035250580000000002 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.799906+0000", 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035666259999999998 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840120+0000", 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.040214634999999999 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840155+0000", 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4536000000000003e-05 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.841199+0000", 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001044295 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152269 7.1 7:850b7764:::benchmark_data_smithi194_147200_object37357:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.620424+0000", 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.725304295, 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22182671300000001, 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152269 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.408 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620424+0000", 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620424+0000", 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620427+0000", 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3720000000000001e-06 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620438+0000", 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1317e-05 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620457+0000", 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8873999999999999e-05 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620467+0000", 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0073e-05 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620543+0000", 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5617999999999993e-05 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:51:12.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.620557+0000", 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3569e-05 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796380+0000", 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.175823009 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796544+0000", 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016415700000000001 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.796702+0000", 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015775000000000001 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.799949+0000", 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032472360000000001 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.800009+0000", 2024-04-29T14:51:12.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0374000000000002e-05 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.841245+0000", 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.041236092000000002 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.841293+0000", 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7398000000000003e-05 2024-04-29T14:51:12.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.842250+0000", 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00095787400000000005 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152285 7.1 7:86e469da:::benchmark_data_smithi194_147200_object37373:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.822437+0000", 2024-04-29T14:51:12.415 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.52329142600001, 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020984796999999999, 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152285 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822437+0000", 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822437+0000", 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822438+0000", 2024-04-29T14:51:12.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4169999999999999e-06 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822443+0000", 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5519999999999998e-06 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822451+0000", 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6060000000000003e-06 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.822457+0000", 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9499999999999998e-06 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.825341+0000", 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002883456 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.825459+0000", 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118022 2024-04-29T14:51:12.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.825576+0000", 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117379 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.838472+0000", 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012896026999999999 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.838494+0000", 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1854000000000001e-05 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.842319+0000", 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038250379999999998 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.842380+0000", 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0677999999999997e-05 2024-04-29T14:51:12.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.843421+0000", 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010418179999999999 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152296 7.1 7:8e43607b:::benchmark_data_smithi194_147200_object37384:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:12.422 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:22.840345+0000", 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.50538336199998, 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022080407999999999, 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152296 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840345+0000", 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840345+0000", 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840347+0000", 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.807e-06 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840352+0000", 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9399999999999999e-06 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840361+0000", 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8060000000000001e-06 2024-04-29T14:51:12.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.840368+0000", 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6540000000000002e-06 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.843490+0000", 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031224640000000001 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.843575+0000", 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4221999999999994e-05 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.845695+0000", 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021208249999999998 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.845823+0000", 2024-04-29T14:51:12.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127544 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.845975+0000", 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015242200000000001 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.860612+0000", 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014637089000000001 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.860649+0000", 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6124000000000002e-05 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.860658+0000", 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1770000000000004e-06 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.860709+0000", 2024-04-29T14:51:12.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0782999999999998e-05 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:22.862425+0000", 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001716551 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152492 7.1 7:8e68686c:::benchmark_data_smithi194_147200_object37580:head [delete] snapc 0=[] ondisk+write+known_if_redirected e832)", 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:24.213486+0000", 2024-04-29T14:51:12.430 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 408.132241658, 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.02340062, 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152492 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.213486+0000", 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.213486+0000", 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.213489+0000", 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3590000000000002e-06 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.213496+0000", 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8249999999999999e-06 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.213508+0000", 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2564e-05 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.213517+0000", 2024-04-29T14:51:12.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8030000000000003e-06 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.222504+0000", 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089875049999999998 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.222593+0000", 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8846999999999998e-05 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.222676+0000", 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2348999999999994e-05 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.224979+0000", 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002303659 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.226720+0000", 2024-04-29T14:51:12.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001740475 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.235832+0000", 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009112317 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.235879+0000", 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6443999999999998e-05 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:24.236887+0000", 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001008473 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152637 7.1 7:8081eb67:::benchmark_data_smithi194_147200_object37725:head [delete] snapc 0=[] ondisk+write+known_if_redirected e832)", 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:25.242922+0000", 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 407.10280572400001, 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020811678, 2024-04-29T14:51:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152637 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.242922+0000", 2024-04-29T14:51:12.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.242922+0000", 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.242925+0000", 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.667e-06 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.242933+0000", 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3350000000000007e-06 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.242946+0000", 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.309e-05 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.242954+0000", 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.661e-06 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.248555+0000", 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056010030000000002 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.248640+0000", 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5227000000000002e-05 2024-04-29T14:51:12.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.248760+0000", 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119614 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.253390+0000", 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046298049999999999 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.254956+0000", 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015662950000000001 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.262404+0000", 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074478119999999998 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.262496+0000", 2024-04-29T14:51:12.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2343999999999995e-05 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.263734+0000", 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012378249999999999 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152638 7.1 7:8ae8c56d:::benchmark_data_smithi194_147200_object37726:head [delete] snapc 0=[] ondisk+write+known_if_redirected e832)", 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:25.245322+0000", 2024-04-29T14:51:12.444 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 407.10040653999999, 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029082640999999999, 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152638 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.245322+0000", 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.245322+0000", 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.245323+0000", 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.516e-06 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.245328+0000", 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9189999999999998e-06 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.245356+0000", 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7818999999999999e-05 2024-04-29T14:51:12.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.245362+0000", 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8939999999999998e-06 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.251029+0000", 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056676180000000001 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.251124+0000", 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4324e-05 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.251212+0000", 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7942999999999995e-05 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.253484+0000", 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022720430000000001 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.255002+0000", 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015184949999999999 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.272796+0000", 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017794009 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.272863+0000", 2024-04-29T14:51:12.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7168999999999993e-05 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.274404+0000", 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015408920000000001 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152664 7.1 7:8fbf3b8a:::benchmark_data_smithi194_147200_object37752:head [delete] snapc 0=[] ondisk+write+known_if_redirected e832)", 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:25.378111+0000", 2024-04-29T14:51:12.451 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 406.96761682800002, 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020489202000000001, 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152664 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.378111+0000", 2024-04-29T14:51:12.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.378111+0000", 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.378113+0000", 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.165e-06 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.378120+0000", 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5509999999999996e-06 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.378131+0000", 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1379999999999999e-05 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.378138+0000", 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4960000000000001e-06 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.384841+0000", 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067032150000000002 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.384955+0000", 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113569 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.385048+0000", 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3154999999999998e-05 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.387397+0000", 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023489299999999999 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.388926+0000", 2024-04-29T14:51:12.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00152915 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.397775+0000", 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088488209999999998 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.397818+0000", 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2923999999999998e-05 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:25.398600+0000", 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078284600000000002 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:152974 7.1 7:8156151a:::benchmark_data_smithi194_147200_object38062:head [delete] snapc 0=[] ondisk+write+known_if_redirected e833)", 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:44:28.758131+0000", 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 403.58759672100001, 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022297309000000001, 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 152974 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.758131+0000", 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.758131+0000", 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.758134+0000", 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.345e-06 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.758142+0000", 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8820000000000001e-06 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.758154+0000", 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2601e-05 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.758162+0000", 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6869999999999994e-06 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.761942+0000", 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037799740000000002 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.762100+0000", 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015833899999999999 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.762187+0000", 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6724999999999994e-05 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.764645+0000", 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024582359999999999 2024-04-29T14:51:12.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.766009+0000", 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001363918 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.779423+0000", 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013414262999999999 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.779470+0000", 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6551000000000002e-05 2024-04-29T14:51:12.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:44:28.780429+0000", 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00095878800000000002 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:159335 7.0 7:07ebd630:::benchmark_data_smithi194_147200_object44423:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:37.227447+0000", 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 335.11828080800001, 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.052559055, 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.464 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 159335 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227447+0000", 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227447+0000", 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227450+0000", 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.145e-06 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227460+0000", 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7860000000000008e-06 2024-04-29T14:51:12.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227478+0000", 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7572999999999999e-05 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227488+0000", 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8199999999999992e-06 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227545+0000", 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7479000000000003e-05 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227643+0000", 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7462999999999998e-05 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.227716+0000", 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3936000000000005e-05 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.230225+0000", 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025080390000000001 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.279894+0000", 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.049669110000000002 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.279954+0000", 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0399999999999998e-05 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:37.280006+0000", 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2303999999999999e-05 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.a 880/877 [PushOp(7:5b15c8f7:::benchmark_data_smithi194_147200_object50527:head, version: 707'12525, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:5b15c8f7:::benchmark_data_smithi194_147200_object50527:head@707'12525, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:46:45.975463+0000", 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 266.37026470400002, 2024-04-29T14:51:12.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.046230937, 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:45.975463+0000", 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:45.975463+0000", 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:45.975464+0000", 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-07 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:45.975524+0000", 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9487999999999997e-05 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:46.021315+0000", 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.045791459 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:46.021349+0000", 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3578999999999998e-05 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:46.021446+0000", 2024-04-29T14:51:12.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6961999999999998e-05 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:46.021460+0000", 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4847e-05 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:46:46.021694+0000", 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023386300000000001 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.b 998/942 [PushOp(7:dc05f691:::benchmark_data_smithi194_147200_object94700:head, version: 675'5888, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:dc05f691:::benchmark_data_smithi194_147200_object94700:head@675'5888, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:51:12.274576+0000", 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.071152312999999995, 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028204660999999999, 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.274576+0000", 2024-04-29T14:51:12.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.274576+0000", 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.274577+0000", 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.567e-06 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.274641+0000", 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4029999999999995e-05 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.274660+0000", 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.891e-05 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.274677+0000", 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6463000000000001e-05 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.287340+0000", 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012662823 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.287349+0000", 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8709999999999992e-06 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.302780+0000", 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015430997 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.478 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.b 998/942 [PushOp(7:dc8160a5:::benchmark_data_smithi194_147200_object94732:head, version: 675'5890, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:dc8160a5:::benchmark_data_smithi194_147200_object94732:head@675'5890, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:51:12.277676+0000", 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.068052328999999995, 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020384935999999999, 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.277676+0000", 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.277676+0000", 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.277677+0000", 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5820000000000001e-06 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.277741+0000", 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4147999999999996e-05 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.277764+0000", 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2099999999999998e-05 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.277788+0000", 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4017000000000001e-05 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.294882+0000", 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017094042 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.294896+0000", 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4633e-05 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.298061+0000", 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031644139999999999 2024-04-29T14:51:12.482 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.b 998/942 [PushOp(7:d0c0647a:::benchmark_data_smithi194_147200_object94749:head, version: 675'5891, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:d0c0647a:::benchmark_data_smithi194_147200_object94749:head@675'5891, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:51:12.279210+0000", 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.066518321000000005, 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023538868000000001, 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.279210+0000", 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.279210+0000", 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.279211+0000", 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2899999999999999e-06 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.279292+0000", 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0530000000000003e-05 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.279311+0000", 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9343000000000001e-05 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.279328+0000", 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6758000000000001e-05 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.298147+0000", 2024-04-29T14:51:12.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018819016000000001 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.298162+0000", 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5038999999999999e-05 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.302749+0000", 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045868920000000004 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.b 998/942 [PushOp(7:de5f4307:::benchmark_data_smithi194_147200_object94763:head, version: 675'5892, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:de5f4307:::benchmark_data_smithi194_147200_object94763:head@675'5892, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:51:12.280805+0000", 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.064923417999999997, 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035337374999999997, 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.280805+0000", 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.280805+0000", 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.280806+0000", 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.105e-06 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.280853+0000", 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7700000000000001e-05 2024-04-29T14:51:12.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.280874+0000", 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0701000000000001e-05 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.280893+0000", 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8629000000000001e-05 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.313021+0000", 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032128072000000001 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.313036+0000", 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5284000000000001e-05 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:51:12.316142+0000", 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031058840000000002 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:51:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:51:12.491 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:51:12.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:12.524 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:12.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:12 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1713314879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:51:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:12 smithi040 ceph-mon[39581]: osdmap e998: 12 total, 12 up, 10 in 2024-04-29T14:51:12.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:12.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:12 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:12.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1713314879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:51:12.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:12 smithi121 ceph-mon[44533]: osdmap e998: 12 total, 12 up, 10 in 2024-04-29T14:51:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:12.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:12.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:12.754 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:12.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:12 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:12.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1713314879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:51:12.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:12 smithi194 ceph-mon[44089]: osdmap e998: 12 total, 12 up, 10 in 2024-04-29T14:51:13.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:51:13.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:13.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:13.257 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:13.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:13.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:13.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:13.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:13.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:13 smithi121 ceph-mon[44533]: pgmap v2511: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 306 op/s; 1702/41976 objects misplaced (4.055%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:51:13.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:13 smithi121 ceph-mon[44533]: osdmap e999: 12 total, 12 up, 10 in 2024-04-29T14:51:13.484 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:13.504 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:13.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:13.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:13 smithi040 ceph-mon[39581]: pgmap v2511: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 306 op/s; 1702/41976 objects misplaced (4.055%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:51:13.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:13 smithi040 ceph-mon[39581]: osdmap e999: 12 total, 12 up, 10 in 2024-04-29T14:51:13.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:13.637 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:13.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:13.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:13.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:13.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:13 smithi194 ceph-mon[44089]: pgmap v2511: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 306 op/s; 1702/41976 objects misplaced (4.055%); 1.4 MiB/s, 21 objects/s recovering 2024-04-29T14:51:13.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:13 smithi194 ceph-mon[44089]: osdmap e999: 12 total, 12 up, 10 in 2024-04-29T14:51:14.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:51:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:15 smithi194 ceph-mon[44089]: pgmap v2513: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 294 op/s; 1702/40748 objects misplaced (4.177%); 1.7 MiB/s, 27 objects/s recovering 2024-04-29T14:51:15.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:15 smithi040 ceph-mon[39581]: pgmap v2513: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 294 op/s; 1702/40748 objects misplaced (4.177%); 1.7 MiB/s, 27 objects/s recovering 2024-04-29T14:51:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:15 smithi121 ceph-mon[44533]: pgmap v2513: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 9.5 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 294 op/s; 1702/40748 objects misplaced (4.177%); 1.7 MiB/s, 27 objects/s recovering 2024-04-29T14:51:17.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:17 smithi040 ceph-mon[39581]: pgmap v2514: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 1702/38038 objects misplaced (4.474%); 920 KiB/s, 14 objects/s recovering 2024-04-29T14:51:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:17 smithi194 ceph-mon[44089]: pgmap v2514: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 1702/38038 objects misplaced (4.474%); 920 KiB/s, 14 objects/s recovering 2024-04-29T14:51:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:17 smithi121 ceph-mon[44533]: pgmap v2514: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 440 op/s; 1702/38038 objects misplaced (4.474%); 920 KiB/s, 14 objects/s recovering 2024-04-29T14:51:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:51:18.007 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:51:18.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:51:18.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:51:18.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:51:18.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:51:18.605 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:51:18.632 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:51:18.646 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:18.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:18.655 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:51:18.693 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:51:18.720 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:51:18.742 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:51:18.753 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:18.766 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:51:18.781 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:18.787 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:51:18.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:51:18.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:51:18.788 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:51:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:51:18.799 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:18.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:18 smithi040 ceph-mon[39581]: from='client.28048 v1:172.21.15.40:0/1617461342' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:18 smithi040 ceph-mon[39581]: 7.6 scrub starts 2024-04-29T14:51:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:18 smithi040 ceph-mon[39581]: 7.6 scrub ok 2024-04-29T14:51:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:18 smithi040 ceph-mon[39581]: pgmap v2515: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 1477/37532 objects misplaced (3.935%); 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:51:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3642135118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:18 smithi121 ceph-mon[44533]: from='client.28048 v1:172.21.15.40:0/1617461342' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:18 smithi121 ceph-mon[44533]: 7.6 scrub starts 2024-04-29T14:51:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:18 smithi121 ceph-mon[44533]: 7.6 scrub ok 2024-04-29T14:51:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:18 smithi121 ceph-mon[44533]: pgmap v2515: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 1477/37532 objects misplaced (3.935%); 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:51:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3642135118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:18.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:18.930 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:18.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:18.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:19.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:18 smithi194 ceph-mon[44089]: from='client.28048 v1:172.21.15.40:0/1617461342' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:18 smithi194 ceph-mon[44089]: 7.6 scrub starts 2024-04-29T14:51:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:18 smithi194 ceph-mon[44089]: 7.6 scrub ok 2024-04-29T14:51:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:18 smithi194 ceph-mon[44089]: pgmap v2515: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 400 op/s; 1477/37532 objects misplaced (3.935%); 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:51:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3642135118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:19.093 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:20.377 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:20.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:20.683 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:20.839 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:20.842 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:20 smithi040 ceph-mon[39581]: pgmap v2516: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 1477/36190 objects misplaced (4.081%); 1.6 MiB/s, 25 objects/s recovering 2024-04-29T14:51:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:20.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:20.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:20 smithi121 ceph-mon[44533]: pgmap v2516: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 1477/36190 objects misplaced (4.081%); 1.6 MiB/s, 25 objects/s recovering 2024-04-29T14:51:20.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:20.972 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:21.027 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:21.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:21.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:20 smithi194 ceph-mon[44089]: pgmap v2516: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 346 op/s; 1477/36190 objects misplaced (4.081%); 1.6 MiB/s, 25 objects/s recovering 2024-04-29T14:51:21.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:21.159 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:21.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:21.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:21.274 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2514,"stamp":"2024-04-29T14:51:16.314391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1246822968,"num_objects":19019,"num_object_clones":0,"num_object_copies":38038,"num_objects_missing_on_primary":448,"num_objects_missing":448,"num_objects_degraded":0,"num_objects_misplaced":1702,"num_objects_unfound":0,"num_objects_dirty":19019,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":210984,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233312,"num_bytes_recovered":13691354872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":9804736,"kb_used_data":2969168,"kb_used_omap":668,"kb_used_meta":6834787,"kb_avail":927646784,"statfs":{"total":959950356480,"available":949910306816,"internally_reserved":0,"allocated":3040428032,"data_stored":3019286014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":6998822895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-231079936,"num_objects":-3526,"num_object_clones":0,"num_object_copies":-7052,"num_objects_missing_on_primary":-109,"num_objects_missing":-109,"num_objects_degraded":0,"num_objects_misplaced":-224,"num_objects_unfound":0,"num_objects_dirty":-3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3526,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":7536640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001257"},"pg_stats":[{"pgid":"7.f","version":"999'13337","reported_seq":48353,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:15.378674+0000","last_change":"2024-04-29T14:50:38.109612+0000","last_active":"2024-04-29T14:51:15.378674+0000","last_peered":"2024-04-29T14:51:15.378674+0000","last_clean":"2024-04-29T14:51:15.378674+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:51:15.378674+0000","last_undegraded":"2024-04-29T14:51:15.378674+0000","last_fullsized":"2024-04-29T14:51:15.378674+0000","mapping_epoch":918,"log_start":"664'3300","ondisk_log_start":"664'3300","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12570","last_scrub_stamp":"2024-04-29T14:50:38.109478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:38.109478+0000","objects_scrubbed":1874,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:03.361035+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72548352,"num_objects":1107,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13337,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"999'20349","reported_seq":42405,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:14.204263+0000","last_change":"2024-04-29T14:49:57.142920+0000","last_active":"2024-04-29T14:51:14.204263+0000","last_peered":"2024-04-29T14:51:14.204263+0000","last_clean":"2024-04-29T14:51:14.204263+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:51:14.204263+0000","last_undegraded":"2024-04-29T14:51:14.204263+0000","last_fullsized":"2024-04-29T14:51:14.204263+0000","mapping_epoch":930,"log_start":"704'10258","ondisk_log_start":"704'10258","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'19114","last_scrub_stamp":"2024-04-29T14:49:57.142771+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:49:57.142771+0000","objects_scrubbed":2497,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:23.319557+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13293,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"999'13102","reported_seq":44310,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:15.554668+0000","last_change":"2024-04-29T14:49:58.542110+0000","last_active":"2024-04-29T14:51:15.554668+0000","last_peered":"2024-04-29T14:51:15.554668+0000","last_clean":"2024-04-29T14:51:15.554668+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:51:15.554668+0000","last_undegraded":"2024-04-29T14:51:15.554668+0000","last_fullsized":"2024-04-29T14:51:15.554668+0000","mapping_epoch":938,"log_start":"663'3082","ondisk_log_start":"663'3082","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'11857","last_scrub_stamp":"2024-04-29T14:49:58.541853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:49:58.541853+0000","objects_scrubbed":2470,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:16.374459+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150552,"num_objects":1224,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13102,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"999'16356","reported_seq":81922,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:15.512660+0000","last_change":"2024-04-29T14:50:34.579123+0000","last_active":"2024-04-29T14:51:15.512660+0000","last_peered":"2024-04-29T14:51:15.512660+0000","last_clean":"2024-04-29T14:51:15.512660+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:51:15.512660+0000","last_undegraded":"2024-04-29T14:51:15.512660+0000","last_fullsized":"2024-04-29T14:51:15.512660+0000","mapping_epoch":939,"log_start":"675'6350","ondisk_log_start":"675'6350","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'15542","last_scrub_stamp":"2024-04-29T14:50:34.578998+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:50:34.578998+0000","objects_scrubbed":1966,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:38.386277+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":1152,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13196,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"997'13002","reported_seq":31769,"reported_epoch":997,"state":"active+recovering","last_fresh":"2024-04-29T14:51:11.441038+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:51:11.441038+0000","last_peered":"2024-04-29T14:51:11.441038+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:51:11.441038+0000","last_undegraded":"2024-04-29T14:51:11.441038+0000","last_fullsized":"2024-04-29T14:51:11.441038+0000","mapping_epoch":942,"log_start":"663'3000","ondisk_log_start":"663'3000","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":1280,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":448,"num_objects_missing":448,"num_objects_degraded":0,"num_objects_misplaced":1702,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13002,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6314,"num_bytes_recovered":413663232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":26},{"shards":"1,10,11","objects":806},{"shards":"10,11","objects":448}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"999'20579","reported_seq":38113,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:15.414227+0000","last_change":"2024-04-29T14:50:45.604038+0000","last_active":"2024-04-29T14:51:15.414227+0000","last_peered":"2024-04-29T14:51:15.414227+0000","last_clean":"2024-04-29T14:51:15.414227+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:51:15.414227+0000","last_undegraded":"2024-04-29T14:51:15.414227+0000","last_fullsized":"2024-04-29T14:51:15.414227+0000","mapping_epoch":895,"log_start":"705'10500","ondisk_log_start":"705'10500","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"991'19938","last_scrub_stamp":"2024-04-29T14:50:45.603880+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:50:45.603880+0000","objects_scrubbed":1811,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:01.560720+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76611584,"num_objects":1169,"num_object_clones":0,"num_object_copies":2338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13505,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"997'13097","reported_seq":42288,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-04-29T14:51:11.328607+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:51:11.328607+0000","last_peered":"2024-04-29T14:51:11.328607+0000","last_clean":"2024-04-29T14:51:11.328607+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:51:11.328607+0000","last_undegraded":"2024-04-29T14:51:11.328607+0000","last_fullsized":"2024-04-29T14:51:11.328607+0000","mapping_epoch":756,"log_start":"663'3000","ondisk_log_start":"663'3000","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81461248,"num_objects":1243,"num_object_clones":0,"num_object_copies":2486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13097,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"999'13114","reported_seq":22233,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:16.089864+0000","last_change":"2024-04-29T14:50:12.400042+0000","last_active":"2024-04-29T14:51:16.089864+0000","last_peered":"2024-04-29T14:51:16.089864+0000","last_clean":"2024-04-29T14:51:16.089864+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:51:16.089864+0000","last_undegraded":"2024-04-29T14:51:16.089864+0000","last_fullsized":"2024-04-29T14:51:16.089864+0000","mapping_epoch":883,"log_start":"663'3100","ondisk_log_start":"663'3100","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'11960","last_scrub_stamp":"2024-04-29T14:50:12.399923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:12.399923+0000","objects_scrubbed":2330,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:26.243671+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77070336,"num_objects":1176,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13114,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"999'13540","reported_seq":71915,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:15.529036+0000","last_change":"2024-04-29T14:50:39.638903+0000","last_active":"2024-04-29T14:51:15.529036+0000","last_peered":"2024-04-29T14:51:15.529036+0000","last_clean":"2024-04-29T14:51:15.529036+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:51:15.529036+0000","last_undegraded":"2024-04-29T14:51:15.529036+0000","last_fullsized":"2024-04-29T14:51:15.529036+0000","mapping_epoch":939,"log_start":"664'3476","ondisk_log_start":"664'3476","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12718","last_scrub_stamp":"2024-04-29T14:50:39.638817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:39.638817+0000","objects_scrubbed":2012,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:07.740501+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77856768,"num_objects":1188,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13540,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"999'13133","reported_seq":47445,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:15.345400+0000","last_change":"2024-04-29T14:50:36.645439+0000","last_active":"2024-04-29T14:51:15.345400+0000","last_peered":"2024-04-29T14:51:15.345400+0000","last_clean":"2024-04-29T14:51:15.345400+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:51:15.345400+0000","last_undegraded":"2024-04-29T14:51:15.345400+0000","last_fullsized":"2024-04-29T14:51:15.345400+0000","mapping_epoch":918,"log_start":"663'3100","ondisk_log_start":"663'3100","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12376","last_scrub_stamp":"2024-04-29T14:50:36.645270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:36.645270+0000","objects_scrubbed":1898,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:54.914264+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74776576,"num_objects":1141,"num_object_clones":0,"num_object_copies":2282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13133,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"999'13319","reported_seq":49730,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:14.218338+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:51:14.218338+0000","last_peered":"2024-04-29T14:51:14.218338+0000","last_clean":"2024-04-29T14:51:14.218338+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:51:14.218338+0000","last_undegraded":"2024-04-29T14:51:14.218338+0000","last_fullsized":"2024-04-29T14:51:14.218338+0000","mapping_epoch":915,"log_start":"664'3300","ondisk_log_start":"664'3300","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77660160,"num_objects":1185,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13319,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"999'16750","reported_seq":41564,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:13.444861+0000","last_change":"2024-04-29T14:50:11.012445+0000","last_active":"2024-04-29T14:51:13.444861+0000","last_peered":"2024-04-29T14:51:13.444861+0000","last_clean":"2024-04-29T14:51:13.444861+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:51:13.444861+0000","last_undegraded":"2024-04-29T14:51:13.444861+0000","last_fullsized":"2024-04-29T14:51:13.444861+0000","mapping_epoch":895,"log_start":"681'6725","ondisk_log_start":"681'6725","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'15627","last_scrub_stamp":"2024-04-29T14:50:11.012229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:11.012229+0000","objects_scrubbed":2385,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.431973+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13151,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"999'12989","reported_seq":37484,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:13.506284+0000","last_change":"2024-04-29T14:50:04.932139+0000","last_active":"2024-04-29T14:51:13.506284+0000","last_peered":"2024-04-29T14:51:13.506284+0000","last_clean":"2024-04-29T14:51:13.506284+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:51:13.506284+0000","last_undegraded":"2024-04-29T14:51:13.506284+0000","last_fullsized":"2024-04-29T14:51:13.506284+0000","mapping_epoch":895,"log_start":"663'2944","ondisk_log_start":"663'2944","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11810","last_scrub_stamp":"2024-04-29T14:50:04.932001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:04.932001+0000","objects_scrubbed":2381,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:25.466879+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78708736,"num_objects":1201,"num_object_clones":0,"num_object_copies":2402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12989,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"999'13005","reported_seq":59831,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:14.162003+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:51:14.162003+0000","last_peered":"2024-04-29T14:51:14.162003+0000","last_clean":"2024-04-29T14:51:14.162003+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:51:14.162003+0000","last_undegraded":"2024-04-29T14:51:14.162003+0000","last_fullsized":"2024-04-29T14:51:14.162003+0000","mapping_epoch":928,"log_start":"663'3000","ondisk_log_start":"663'3000","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76873728,"num_objects":1173,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13005,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"999'13086","reported_seq":28099,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:13.630458+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:51:13.630458+0000","last_peered":"2024-04-29T14:51:13.630458+0000","last_clean":"2024-04-29T14:51:13.630458+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:51:13.630458+0000","last_undegraded":"2024-04-29T14:51:13.630458+0000","last_fullsized":"2024-04-29T14:51:13.630458+0000","mapping_epoch":703,"log_start":"663'3000","ondisk_log_start":"663'3000","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"999'12934","reported_seq":34322,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:16.170236+0000","last_change":"2024-04-29T14:50:05.324932+0000","last_active":"2024-04-29T14:51:16.170236+0000","last_peered":"2024-04-29T14:51:16.170236+0000","last_clean":"2024-04-29T14:51:16.170236+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:51:16.170236+0000","last_undegraded":"2024-04-29T14:51:16.170236+0000","last_fullsized":"2024-04-29T14:51:16.170236+0000","mapping_epoch":865,"log_start":"663'2900","ondisk_log_start":"663'2900","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11707","last_scrub_stamp":"2024-04-29T14:50:05.324827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.324827+0000","objects_scrubbed":2336,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72482816,"num_objects":1106,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12934,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1984,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-04-29T14:51:12.624245+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:51:12.624245+0000","last_peered":"2024-04-29T14:51:12.624245+0000","last_clean":"2024-04-29T14:51:12.624245+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:51:12.624245+0000","last_undegraded":"2024-04-29T14:51:12.624245+0000","last_fullsized":"2024-04-29T14:51:12.624245+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1246232600,"num_objects":19017,"num_object_clones":0,"num_object_copies":38034,"num_objects_missing_on_primary":448,"num_objects_missing":448,"num_objects_degraded":0,"num_objects_misplaced":1702,"num_objects_unfound":0,"num_objects_dirty":19017,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210803,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233298,"num_bytes_recovered":13687652376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3440582656,"data_stored":3440574512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160657,"ondisk_log_size":160657,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122580,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384220,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1577716,"kb_used_data":689960,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92167436,"statfs":{"total":95995035648,"available":94379454464,"internally_reserved":0,"allocated":706519040,"data_stored":704405088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":909048256},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1108212,"kb_used_data":305128,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92636940,"statfs":{"total":95995035648,"available":94860226560,"internally_reserved":0,"allocated":312451072,"data_stored":310337140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":822344128},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423159,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364660,"kb_used_data":228008,"kb_used_omap":4,"kb_used_meta":136635,"kb_avail":93380492,"statfs":{"total":95995035648,"available":95621623808,"internally_reserved":0,"allocated":233480192,"data_stored":231366260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":139914944},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234536,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016504,"kb_used_data":150960,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92728648,"statfs":{"total":95995035648,"available":94954135552,"internally_reserved":0,"allocated":154583040,"data_stored":152465556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":886307274},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528877,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226304,"kb_used_data":234420,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92518848,"statfs":{"total":95995035648,"available":94739300352,"internally_reserved":0,"allocated":240046080,"data_stored":237924524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1015675334},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575416,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710364,"kb_used_data":223828,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93034788,"statfs":{"total":95995035648,"available":95267622912,"internally_reserved":0,"allocated":229199872,"data_stored":227094438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738955,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1209840,"kb_used_data":298472,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92535312,"statfs":{"total":95995035648,"available":94756159488,"internally_reserved":0,"allocated":305635328,"data_stored":303521396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182974,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899640,"kb_used_data":318696,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92845512,"statfs":{"total":95995035648,"available":95073804288,"internally_reserved":0,"allocated":326344704,"data_stored":324230772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":594868680},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986739,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1290480,"kb_used_data":316648,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92454672,"statfs":{"total":95995035648,"available":94673584128,"internally_reserved":0,"allocated":324247552,"data_stored":322133620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653300,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401016,"kb_used_data":203048,"kb_used_omap":651,"kb_used_meta":197300,"kb_avail":93344136,"statfs":{"total":95995035648,"available":95584395264,"internally_reserved":0,"allocated":207921152,"data_stored":205807220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":202035976},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293732352,"data_stored":293732352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224399360,"data_stored":224395288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":321847296,"data_stored":321847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":316014592,"data_stored":316014592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":203423744,"data_stored":203423744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":233508864,"data_stored":233504792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":149553152,"data_stored":149553152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":224460800,"data_stored":224460800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":304349184,"data_stored":304349184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":702152704,"data_stored":702152704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:51:21.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:21.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:21.634 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:21.662 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:21.682 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:21.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:21.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:21.813 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:21.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:21.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:21.940 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:21.967 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:21.990 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:21.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:22.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:22.139 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:22.228 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:51:22.229 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:51:22.245 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:22.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:22.350 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:22.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:22 smithi040 ceph-mon[39581]: 7.d scrub starts 2024-04-29T14:51:22.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:22 smithi040 ceph-mon[39581]: 7.d scrub ok 2024-04-29T14:51:22.378 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:22.399 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:22.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:22.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:22.541 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:22.567 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:22.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:22 smithi194 ceph-mon[44089]: 7.d scrub starts 2024-04-29T14:51:22.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:22 smithi194 ceph-mon[44089]: 7.d scrub ok 2024-04-29T14:51:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:22.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:22 smithi121 ceph-mon[44533]: 7.d scrub starts 2024-04-29T14:51:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:22 smithi121 ceph-mon[44533]: 7.d scrub ok 2024-04-29T14:51:22.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:22.702 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:22.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:22.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:22.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:22.859 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:22.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:22.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:22.985 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:23.012 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:23.031 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:23.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:23.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:23.161 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:23.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:23.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:23.266 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:23 smithi194 ceph-mon[44089]: pgmap v2517: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 403 op/s; 1224/33902 objects misplaced (3.610%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:23.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:23.310 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:23.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:23.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:23.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:23 smithi040 ceph-mon[39581]: pgmap v2517: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 403 op/s; 1224/33902 objects misplaced (3.610%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:23.435 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:23.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:23 smithi121 ceph-mon[44533]: pgmap v2517: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 403 op/s; 1224/33902 objects misplaced (3.610%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:24.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:51:24.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:51:24.416 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:24.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:24.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:24.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:24.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:24 smithi040 ceph-mon[39581]: osdmap e1000: 12 total, 12 up, 10 in 2024-04-29T14:51:24.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:24 smithi040 ceph-mon[39581]: 7.e scrub starts 2024-04-29T14:51:24.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:24 smithi040 ceph-mon[39581]: 7.e scrub ok 2024-04-29T14:51:24.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:51:24.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:24.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:24.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:24.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:24 smithi194 ceph-mon[44089]: osdmap e1000: 12 total, 12 up, 10 in 2024-04-29T14:51:24.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:24 smithi194 ceph-mon[44089]: 7.e scrub starts 2024-04-29T14:51:24.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:24 smithi194 ceph-mon[44089]: 7.e scrub ok 2024-04-29T14:51:24.589 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:24 smithi121 ceph-mon[44533]: osdmap e1000: 12 total, 12 up, 10 in 2024-04-29T14:51:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:24 smithi121 ceph-mon[44533]: 7.e scrub starts 2024-04-29T14:51:24.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:24 smithi121 ceph-mon[44533]: 7.e scrub ok 2024-04-29T14:51:24.727 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:24.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:24.833 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:24.864 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:24.889 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:24.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:24.993 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:25.019 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:25.037 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:25.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:25.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:25.192 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:25.281 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:25.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:25.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:25.415 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:25.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:25.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:25.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:25.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:25 smithi040 ceph-mon[39581]: pgmap v2519: 17 pgs: 1 active+recovering, 16 active+clean; 1023 MiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 401 op/s; 1224/32712 objects misplaced (3.742%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:25.569 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:25 smithi194 ceph-mon[44089]: pgmap v2519: 17 pgs: 1 active+recovering, 16 active+clean; 1023 MiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 401 op/s; 1224/32712 objects misplaced (3.742%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:25.571 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:25.592 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:25.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:25.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:25 smithi121 ceph-mon[44533]: pgmap v2519: 17 pgs: 1 active+recovering, 16 active+clean; 1023 MiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 401 op/s; 1224/32712 objects misplaced (3.742%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:25.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:26.160 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:26.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:26.265 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:26.291 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:51:26.296 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:26.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:26.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2696581276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:26.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:26 smithi040 ceph-mon[39581]: from='client.37808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:26.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:26 smithi040 ceph-mon[39581]: 7.2 scrub starts 2024-04-29T14:51:26.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:26 smithi040 ceph-mon[39581]: 7.2 scrub ok 2024-04-29T14:51:26.424 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:26.452 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:26.474 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:26.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:26.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2696581276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:26.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:26 smithi194 ceph-mon[44089]: from='client.37808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:26.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:26 smithi194 ceph-mon[44089]: 7.2 scrub starts 2024-04-29T14:51:26.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:26 smithi194 ceph-mon[44089]: 7.2 scrub ok 2024-04-29T14:51:26.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:26.611 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:26.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2696581276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:26.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:26 smithi121 ceph-mon[44533]: from='client.37808 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:26.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:26 smithi121 ceph-mon[44533]: 7.2 scrub starts 2024-04-29T14:51:26.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:26 smithi121 ceph-mon[44533]: 7.2 scrub ok 2024-04-29T14:51:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:26.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:26.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:26.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:26.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:26.947 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:27.047 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:27.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:27.153 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:27.180 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:27.199 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:27.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:27.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:27.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:51:27.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:51:27.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='client.37808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: osdmap e1001: 12 total, 12 up, 10 in 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: pgmap v2521: 17 pgs: 1 active+recovering, 16 active+clean; 953 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 1224/30472 objects misplaced (4.017%); 1000 KiB/s, 15 objects/s recovering 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:27.317 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:27.334 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:27.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:27.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:27.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='client.37808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: osdmap e1001: 12 total, 12 up, 10 in 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:51:27.482 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:51:27.482 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: pgmap v2521: 17 pgs: 1 active+recovering, 16 active+clean; 953 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 1224/30472 objects misplaced (4.017%); 1000 KiB/s, 15 objects/s recovering 2024-04-29T14:51:27.482 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:27.482 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:27.501 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:27.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:27.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='client.37808 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: osdmap e1001: 12 total, 12 up, 10 in 2024-04-29T14:51:27.631 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:51:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:51:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:51:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:51:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: pgmap v2521: 17 pgs: 1 active+recovering, 16 active+clean; 953 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 441 op/s; 1224/30472 objects misplaced (4.017%); 1000 KiB/s, 15 objects/s recovering 2024-04-29T14:51:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:27.632 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:27.633 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:27.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:27.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:27.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:27.793 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:28.051 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:28.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:28.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:28.183 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:28.204 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:28.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:28.308 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:28.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:28.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:28.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:28.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:28.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:28 smithi040 ceph-mon[39581]: osdmap e1002: 12 total, 12 up, 10 in 2024-04-29T14:51:28.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:28 smithi040 ceph-mon[39581]: osdmap e1003: 12 total, 12 up, 10 in 2024-04-29T14:51:28.335 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:28.356 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:28.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:28.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:28.487 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:28.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:28.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:28.573 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:28.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:28.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:28.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:28.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:28 smithi194 ceph-mon[44089]: osdmap e1002: 12 total, 12 up, 10 in 2024-04-29T14:51:28.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:28 smithi194 ceph-mon[44089]: osdmap e1003: 12 total, 12 up, 10 in 2024-04-29T14:51:28.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:28.634 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:28 smithi121 ceph-mon[44533]: osdmap e1002: 12 total, 12 up, 10 in 2024-04-29T14:51:28.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:28 smithi121 ceph-mon[44533]: osdmap e1003: 12 total, 12 up, 10 in 2024-04-29T14:51:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:28.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:28.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:28.798 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:29.019 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:29.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:29.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:29.166 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:29.233 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:29.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:29.326 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:29 smithi040 ceph-mon[39581]: pgmap v2524: 17 pgs: 1 active+recovering, 16 active+clean; 935 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 946/29910 objects misplaced (3.163%); 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:51:29.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:29.382 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:29.410 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:29.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:29.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:29.547 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:29.570 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:29.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:29 smithi121 ceph-mon[44533]: pgmap v2524: 17 pgs: 1 active+recovering, 16 active+clean; 935 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 946/29910 objects misplaced (3.163%); 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:51:29.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:29.740 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:29.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:29.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:29.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:29 smithi194 ceph-mon[44089]: pgmap v2524: 17 pgs: 1 active+recovering, 16 active+clean; 935 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 394 op/s; 946/29910 objects misplaced (3.163%); 1.8 MiB/s, 28 objects/s recovering 2024-04-29T14:51:29.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:29.932 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:29.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:30.040 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:30.071 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:30.093 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:30.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:30.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:30.249 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:30.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:30.396 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:30.427 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:30.454 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:30.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:30.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:30.601 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:30.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:30.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:30.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:30.795 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:31.292 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:51:31.292 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:51:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:51:31.508 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:51:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:31 smithi040 ceph-mon[39581]: 7.8 scrub starts 2024-04-29T14:51:31.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:31 smithi040 ceph-mon[39581]: 7.8 scrub ok 2024-04-29T14:51:31.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:31 smithi040 ceph-mon[39581]: pgmap v2525: 17 pgs: 1 active+recovering, 16 active+clean; 881 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 376 op/s; 946/28190 objects misplaced (3.356%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:31.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:31 smithi040 ceph-mon[39581]: 7.3 deep-scrub starts 2024-04-29T14:51:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:31 smithi121 ceph-mon[44533]: 7.8 scrub starts 2024-04-29T14:51:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:31 smithi121 ceph-mon[44533]: 7.8 scrub ok 2024-04-29T14:51:31.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:31 smithi121 ceph-mon[44533]: pgmap v2525: 17 pgs: 1 active+recovering, 16 active+clean; 881 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 376 op/s; 946/28190 objects misplaced (3.356%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:31.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:31 smithi121 ceph-mon[44533]: 7.3 deep-scrub starts 2024-04-29T14:51:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:31 smithi194 ceph-mon[44089]: 7.8 scrub starts 2024-04-29T14:51:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:31 smithi194 ceph-mon[44089]: 7.8 scrub ok 2024-04-29T14:51:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:31 smithi194 ceph-mon[44089]: pgmap v2525: 17 pgs: 1 active+recovering, 16 active+clean; 881 MiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 376 op/s; 946/28190 objects misplaced (3.356%); 1.5 MiB/s, 23 objects/s recovering 2024-04-29T14:51:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:31 smithi194 ceph-mon[44089]: 7.3 deep-scrub starts 2024-04-29T14:51:32.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:51:32.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:32 smithi040 ceph-mon[39581]: 1.0 deep-scrub starts 2024-04-29T14:51:32.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:32 smithi040 ceph-mon[39581]: 1.0 deep-scrub ok 2024-04-29T14:51:32.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:32 smithi040 ceph-mon[39581]: from='client.28063 v1:172.21.15.40:0/844786884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:32 smithi121 ceph-mon[44533]: 1.0 deep-scrub starts 2024-04-29T14:51:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:32 smithi121 ceph-mon[44533]: 1.0 deep-scrub ok 2024-04-29T14:51:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:32 smithi121 ceph-mon[44533]: from='client.28063 v1:172.21.15.40:0/844786884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:32.757 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:32.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:32 smithi194 ceph-mon[44089]: 1.0 deep-scrub starts 2024-04-29T14:51:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:32 smithi194 ceph-mon[44089]: 1.0 deep-scrub ok 2024-04-29T14:51:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:32 smithi194 ceph-mon[44089]: from='client.28063 v1:172.21.15.40:0/844786884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:32.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:32.917 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:32.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:32.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:33.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:33.112 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:33.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:33.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:33.254 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:33.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:33.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:33.345 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:33 smithi040 ceph-mon[39581]: pgmap v2526: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 15 active+clean; 722 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 609 op/s; 555/23106 objects misplaced (2.402%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:51:33.346 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:33 smithi040 ceph-mon[39581]: 7.3 deep-scrub ok 2024-04-29T14:51:33.352 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:33 smithi194 ceph-mon[44089]: pgmap v2526: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 15 active+clean; 722 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 609 op/s; 555/23106 objects misplaced (2.402%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:51:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:33 smithi194 ceph-mon[44089]: 7.3 deep-scrub ok 2024-04-29T14:51:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:33 smithi194 ceph-mon[44089]: 7.4 scrub starts 2024-04-29T14:51:33.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:33 smithi194 ceph-mon[44089]: 7.4 scrub ok 2024-04-29T14:51:33.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:33.429 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:33.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:33.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:33.573 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:33.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:33 smithi040 ceph-mon[39581]: 7.4 scrub starts 2024-04-29T14:51:33.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:33 smithi040 ceph-mon[39581]: 7.4 scrub ok 2024-04-29T14:51:33.631 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:33 smithi121 ceph-mon[44533]: pgmap v2526: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 15 active+clean; 722 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 609 op/s; 555/23106 objects misplaced (2.402%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:51:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:33 smithi121 ceph-mon[44533]: 7.3 deep-scrub ok 2024-04-29T14:51:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:33 smithi121 ceph-mon[44533]: 7.4 scrub starts 2024-04-29T14:51:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:33 smithi121 ceph-mon[44533]: 7.4 scrub ok 2024-04-29T14:51:34.568 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:51:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:34.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:34 smithi040 ceph-mon[39581]: 7.1 scrub starts 2024-04-29T14:51:34.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:34 smithi040 ceph-mon[39581]: 7.1 scrub ok 2024-04-29T14:51:34.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:34 smithi121 ceph-mon[44533]: 7.1 scrub starts 2024-04-29T14:51:34.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:34 smithi121 ceph-mon[44533]: 7.1 scrub ok 2024-04-29T14:51:34.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:34.732 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:34.792 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:34.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:34 smithi194 ceph-mon[44089]: 7.1 scrub starts 2024-04-29T14:51:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:34 smithi194 ceph-mon[44089]: 7.1 scrub ok 2024-04-29T14:51:34.839 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2525,"stamp":"2024-04-29T14:51:30.318109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":924123704,"num_objects":14095,"num_object_clones":0,"num_object_copies":28190,"num_objects_missing_on_primary":70,"num_objects_missing":70,"num_objects_degraded":0,"num_objects_misplaced":946,"num_objects_unfound":0,"num_objects_dirty":14095,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":215908,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233690,"num_bytes_recovered":13716127480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161018,"ondisk_log_size":161018,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":9182048,"kb_used_data":2346288,"kb_used_omap":668,"kb_used_meta":6834787,"kb_avail":928269472,"statfs":{"total":959950356480,"available":950547939328,"internally_reserved":0,"allocated":2402598912,"data_stored":2381307154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":6998822895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-148176896,"num_objects":-2261,"num_object_clones":0,"num_object_copies":-4522,"num_objects_missing_on_primary":-135,"num_objects_missing":-135,"num_objects_degraded":0,"num_objects_misplaced":-278,"num_objects_unfound":0,"num_objects_dirty":-2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":9371648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002222"},"pg_stats":[{"pgid":"7.f","version":"1000'13565","reported_seq":48583,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:25.349863+0000","last_change":"2024-04-29T14:50:38.109612+0000","last_active":"2024-04-29T14:51:25.349863+0000","last_peered":"2024-04-29T14:51:25.349863+0000","last_clean":"2024-04-29T14:51:25.349863+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:51:25.349863+0000","last_undegraded":"2024-04-29T14:51:25.349863+0000","last_fullsized":"2024-04-29T14:51:25.349863+0000","mapping_epoch":918,"log_start":"665'3500","ondisk_log_start":"665'3500","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12570","last_scrub_stamp":"2024-04-29T14:50:38.109478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:38.109478+0000","objects_scrubbed":1874,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:03.361035+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57606144,"num_objects":879,"num_object_clones":0,"num_object_copies":1758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13565,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"1003'20776","reported_seq":42858,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-04-29T14:51:29.378668+0000","last_change":"2024-04-29T14:51:23.433502+0000","last_active":"2024-04-29T14:51:29.378668+0000","last_peered":"2024-04-29T14:51:29.378668+0000","last_clean":"2024-04-29T14:51:29.378668+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:51:29.378668+0000","last_undegraded":"2024-04-29T14:51:29.378668+0000","last_fullsized":"2024-04-29T14:51:29.378668+0000","mapping_epoch":930,"log_start":"705'10758","ondisk_log_start":"705'10758","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'20581","last_scrub_stamp":"2024-04-29T14:51:23.433402+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:51:23.433402+0000","objects_scrubbed":1025,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:48.261279+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54394880,"num_objects":830,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13720,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"1000'13361","reported_seq":44588,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:25.568870+0000","last_change":"2024-04-29T14:51:21.470456+0000","last_active":"2024-04-29T14:51:25.568870+0000","last_peered":"2024-04-29T14:51:25.568870+0000","last_clean":"2024-04-29T14:51:25.568870+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:51:25.568870+0000","last_undegraded":"2024-04-29T14:51:25.568870+0000","last_fullsized":"2024-04-29T14:51:25.568870+0000","mapping_epoch":938,"log_start":"664'3282","ondisk_log_start":"664'3282","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"999'13234","last_scrub_stamp":"2024-04-29T14:51:21.470297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:21.470297+0000","objects_scrubbed":1093,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:34.115923+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63176728,"num_objects":965,"num_object_clones":0,"num_object_copies":1930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13361,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"1000'16601","reported_seq":82169,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:25.534374+0000","last_change":"2024-04-29T14:50:34.579123+0000","last_active":"2024-04-29T14:51:25.534374+0000","last_peered":"2024-04-29T14:51:25.534374+0000","last_clean":"2024-04-29T14:51:25.534374+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:51:25.534374+0000","last_undegraded":"2024-04-29T14:51:25.534374+0000","last_fullsized":"2024-04-29T14:51:25.534374+0000","mapping_epoch":939,"log_start":"679'6550","ondisk_log_start":"679'6550","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'15542","last_scrub_stamp":"2024-04-29T14:50:34.578998+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:50:34.578998+0000","objects_scrubbed":1966,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:38.386277+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59441152,"num_objects":907,"num_object_clones":0,"num_object_copies":1814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13441,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"1001'13384","reported_seq":33294,"reported_epoch":1001,"state":"active+recovering","last_fresh":"2024-04-29T14:51:26.511877+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:51:26.511877+0000","last_peered":"2024-04-29T14:51:26.511877+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:51:26.511877+0000","last_undegraded":"2024-04-29T14:51:26.511877+0000","last_fullsized":"2024-04-29T14:51:26.511877+0000","mapping_epoch":942,"log_start":"664'3300","ondisk_log_start":"664'3300","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58851328,"num_objects":898,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":70,"num_objects_missing":70,"num_objects_degraded":0,"num_objects_misplaced":946,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13384,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6692,"num_bytes_recovered":438435840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"1,5","objects":22},{"shards":"1,10,11","objects":806},{"shards":"10,11","objects":70}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"1000'20806","reported_seq":38342,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:25.252743+0000","last_change":"2024-04-29T14:50:45.604038+0000","last_active":"2024-04-29T14:51:25.252743+0000","last_peered":"2024-04-29T14:51:25.252743+0000","last_clean":"2024-04-29T14:51:25.252743+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:51:25.252743+0000","last_undegraded":"2024-04-29T14:51:25.252743+0000","last_fullsized":"2024-04-29T14:51:25.252743+0000","mapping_epoch":895,"log_start":"705'10800","ondisk_log_start":"705'10800","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"991'19938","last_scrub_stamp":"2024-04-29T14:50:45.603880+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:50:45.603880+0000","objects_scrubbed":1811,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:01.560720+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61734912,"num_objects":942,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13732,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"1001'13491","reported_seq":42690,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-04-29T14:51:26.328286+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:51:26.328286+0000","last_peered":"2024-04-29T14:51:26.328286+0000","last_clean":"2024-04-29T14:51:26.328286+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:51:26.328286+0000","last_undegraded":"2024-04-29T14:51:26.328286+0000","last_fullsized":"2024-04-29T14:51:26.328286+0000","mapping_epoch":756,"log_start":"664'3400","ondisk_log_start":"664'3400","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55640064,"num_objects":849,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13491,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"1000'13349","reported_seq":22470,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:26.152891+0000","last_change":"2024-04-29T14:50:12.400042+0000","last_active":"2024-04-29T14:51:26.152891+0000","last_peered":"2024-04-29T14:51:26.152891+0000","last_clean":"2024-04-29T14:51:26.152891+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:51:26.152891+0000","last_undegraded":"2024-04-29T14:51:26.152891+0000","last_fullsized":"2024-04-29T14:51:26.152891+0000","mapping_epoch":883,"log_start":"664'3300","ondisk_log_start":"664'3300","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'11960","last_scrub_stamp":"2024-04-29T14:50:12.399923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:12.399923+0000","objects_scrubbed":2330,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:26.243671+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61669376,"num_objects":941,"num_object_clones":0,"num_object_copies":1882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13349,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"1000'13766","reported_seq":72143,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:25.537932+0000","last_change":"2024-04-29T14:50:39.638903+0000","last_active":"2024-04-29T14:51:25.537932+0000","last_peered":"2024-04-29T14:51:25.537932+0000","last_clean":"2024-04-29T14:51:25.537932+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:51:25.537932+0000","last_undegraded":"2024-04-29T14:51:25.537932+0000","last_fullsized":"2024-04-29T14:51:25.537932+0000","mapping_epoch":939,"log_start":"665'3676","ondisk_log_start":"665'3676","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12718","last_scrub_stamp":"2024-04-29T14:50:39.638817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:39.638817+0000","objects_scrubbed":2012,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:07.740501+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13766,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"1000'13373","reported_seq":47687,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:25.234158+0000","last_change":"2024-04-29T14:50:36.645439+0000","last_active":"2024-04-29T14:51:25.234158+0000","last_peered":"2024-04-29T14:51:25.234158+0000","last_clean":"2024-04-29T14:51:25.234158+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:51:25.234158+0000","last_undegraded":"2024-04-29T14:51:25.234158+0000","last_fullsized":"2024-04-29T14:51:25.234158+0000","mapping_epoch":918,"log_start":"664'3300","ondisk_log_start":"664'3300","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12376","last_scrub_stamp":"2024-04-29T14:50:36.645270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:36.645270+0000","objects_scrubbed":1898,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:54.914264+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59047936,"num_objects":901,"num_object_clones":0,"num_object_copies":1802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13373,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"1003'13687","reported_seq":50106,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-04-29T14:51:29.263005+0000","last_change":"2024-04-29T14:50:02.448003+0000","last_active":"2024-04-29T14:51:29.263005+0000","last_peered":"2024-04-29T14:51:29.263005+0000","last_clean":"2024-04-29T14:51:29.263005+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:51:29.263005+0000","last_undegraded":"2024-04-29T14:51:29.263005+0000","last_fullsized":"2024-04-29T14:51:29.263005+0000","mapping_epoch":915,"log_start":"665'3600","ondisk_log_start":"665'3600","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"972'12083","last_scrub_stamp":"2024-04-29T14:50:02.447903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:02.447903+0000","objects_scrubbed":2421,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.823789+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53542912,"num_objects":817,"num_object_clones":0,"num_object_copies":1634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13687,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1003'17126","reported_seq":41948,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-04-29T14:51:28.534066+0000","last_change":"2024-04-29T14:50:11.012445+0000","last_active":"2024-04-29T14:51:28.534066+0000","last_peered":"2024-04-29T14:51:28.534066+0000","last_clean":"2024-04-29T14:51:28.534066+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:51:28.534066+0000","last_undegraded":"2024-04-29T14:51:28.534066+0000","last_fullsized":"2024-04-29T14:51:28.534066+0000","mapping_epoch":895,"log_start":"682'7125","ondisk_log_start":"682'7125","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'15627","last_scrub_stamp":"2024-04-29T14:50:11.012229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:11.012229+0000","objects_scrubbed":2385,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:29.431973+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57737216,"num_objects":881,"num_object_clones":0,"num_object_copies":1762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13527,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"1003'13375","reported_seq":37895,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-04-29T14:51:28.562925+0000","last_change":"2024-04-29T14:51:25.671619+0000","last_active":"2024-04-29T14:51:28.562925+0000","last_peered":"2024-04-29T14:51:28.562925+0000","last_clean":"2024-04-29T14:51:28.562925+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:51:28.562925+0000","last_undegraded":"2024-04-29T14:51:28.562925+0000","last_fullsized":"2024-04-29T14:51:28.562925+0000","mapping_epoch":895,"log_start":"664'3344","ondisk_log_start":"664'3344","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'13293","last_scrub_stamp":"2024-04-29T14:51:25.671430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:25.671430+0000","objects_scrubbed":900,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:33.187005+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":1630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13375,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"1003'13381","reported_seq":60215,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-04-29T14:51:29.264649+0000","last_change":"2024-04-29T14:50:05.362239+0000","last_active":"2024-04-29T14:51:29.264649+0000","last_peered":"2024-04-29T14:51:29.264649+0000","last_clean":"2024-04-29T14:51:29.264649+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:51:29.264649+0000","last_undegraded":"2024-04-29T14:51:29.264649+0000","last_fullsized":"2024-04-29T14:51:29.264649+0000","mapping_epoch":928,"log_start":"664'3300","ondisk_log_start":"664'3300","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'11841","last_scrub_stamp":"2024-04-29T14:50:05.362126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:05.362126+0000","objects_scrubbed":2337,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:31.635364+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52232192,"num_objects":797,"num_object_clones":0,"num_object_copies":1594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13381,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1003'13405","reported_seq":28426,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-04-29T14:51:28.596656+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:51:28.596656+0000","last_peered":"2024-04-29T14:51:28.596656+0000","last_clean":"2024-04-29T14:51:28.596656+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:51:28.596656+0000","last_undegraded":"2024-04-29T14:51:28.596656+0000","last_fullsized":"2024-04-29T14:51:28.596656+0000","mapping_epoch":703,"log_start":"664'3400","ondisk_log_start":"664'3400","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54984704,"num_objects":839,"num_object_clones":0,"num_object_copies":1678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13405,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'13170","reported_seq":34578,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:26.115952+0000","last_change":"2024-04-29T14:51:18.131718+0000","last_active":"2024-04-29T14:51:26.115952+0000","last_peered":"2024-04-29T14:51:26.115952+0000","last_clean":"2024-04-29T14:51:26.115952+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:51:26.115952+0000","last_undegraded":"2024-04-29T14:51:26.115952+0000","last_fullsized":"2024-04-29T14:51:26.115952+0000","mapping_epoch":865,"log_start":"663'3100","ondisk_log_start":"663'3100","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"999'12972","last_scrub_stamp":"2024-04-29T14:51:18.131573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:18.131573+0000","objects_scrubbed":1069,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:42.308743+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57016320,"num_objects":870,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13170,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1986,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-04-29T14:51:23.266618+0000","last_change":"2024-04-29T14:50:08.327336+0000","last_active":"2024-04-29T14:51:23.266618+0000","last_peered":"2024-04-29T14:51:23.266618+0000","last_clean":"2024-04-29T14:51:23.266618+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:51:23.266618+0000","last_undegraded":"2024-04-29T14:51:23.266618+0000","last_fullsized":"2024-04-29T14:51:23.266618+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-29T14:50:08.327193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:03:06.330629+0000","last_clean_scrub_stamp":"2024-04-29T14:50:08.327193+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:27.423834+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":923533336,"num_objects":14093,"num_object_clones":0,"num_object_copies":28186,"num_objects_missing_on_primary":70,"num_objects_missing":70,"num_objects_degraded":0,"num_objects_misplaced":946,"num_objects_unfound":0,"num_objects_dirty":14093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215727,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233676,"num_bytes_recovered":13712424984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2809208832,"data_stored":2809200688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160881,"ondisk_log_size":160881,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122583,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384223,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1485544,"kb_used_data":610200,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92259608,"statfs":{"total":95995035648,"available":94473838592,"internally_reserved":0,"allocated":624844800,"data_stored":622705908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":896268736},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645857,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053692,"kb_used_data":246452,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92691460,"statfs":{"total":95995035648,"available":94916055040,"internally_reserved":0,"allocated":252366848,"data_stored":250246681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":826603968},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423162,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317500,"kb_used_data":180852,"kb_used_omap":4,"kb_used_meta":136635,"kb_avail":93427652,"statfs":{"total":95995035648,"available":95669915648,"internally_reserved":0,"allocated":185192448,"data_stored":183072281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":139914944},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":986372,"kb_used_data":120828,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92758780,"statfs":{"total":95995035648,"available":94984990720,"internally_reserved":0,"allocated":123727872,"data_stored":121604153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":886307274},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528879,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1178696,"kb_used_data":186816,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92566456,"statfs":{"total":95995035648,"available":94788050944,"internally_reserved":0,"allocated":191299584,"data_stored":189171793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1015675334},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575419,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663168,"kb_used_data":172472,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93081984,"statfs":{"total":95995035648,"available":95315951616,"internally_reserved":0,"allocated":176611328,"data_stored":174487089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738957,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1152828,"kb_used_data":237300,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92592324,"statfs":{"total":95995035648,"available":94814539776,"internally_reserved":0,"allocated":242995200,"data_stored":240875033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497391,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790956,"kb_used_data":218328,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92954196,"statfs":{"total":95995035648,"available":95185096704,"internally_reserved":0,"allocated":223567872,"data_stored":221429000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":586349000},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197216,"kb_used_data":219096,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92547936,"statfs":{"total":95995035648,"available":94769086464,"internally_reserved":0,"allocated":224354304,"data_stored":222215432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":356076,"kb_used_data":153944,"kb_used_omap":651,"kb_used_meta":201460,"kb_avail":93389076,"statfs":{"total":95995035648,"available":95630413824,"internally_reserved":0,"allocated":157638656,"data_stored":155499784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":206295816},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233439232,"data_stored":233439232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171511808,"data_stored":171507736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":219086848,"data_stored":219086848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":214630400,"data_stored":214630400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":153485312,"data_stored":153485312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":185602048,"data_stored":185597976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":118685696,"data_stored":118685696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":178585600,"data_stored":178585600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":246480896,"data_stored":246480896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":620560384,"data_stored":620560384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:51:34.840 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:51:34.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:35.024 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:35.118 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:35.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:35.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:35.258 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:35.284 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:35.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:35.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:35 smithi040 ceph-mon[39581]: pgmap v2527: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 15 active+clean; 605 MiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 695 op/s; 555/19342 objects misplaced (2.869%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:51:35.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:35.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:35.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:35.413 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:35.415 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:35.471 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:35.546 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:51:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:35 smithi194 ceph-mon[44089]: pgmap v2527: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 15 active+clean; 605 MiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 695 op/s; 555/19342 objects misplaced (2.869%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:51:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:35.588 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:51:35.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:51:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:35 smithi121 ceph-mon[44533]: pgmap v2527: 17 pgs: 1 active+recovering, 1 active+clean+scrubbing+deep, 15 active+clean; 605 MiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 695 op/s; 555/19342 objects misplaced (2.869%); 3.6 MiB/s, 57 objects/s recovering 2024-04-29T14:51:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:35.649 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:51:35.681 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:51:35.686 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:35.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:35.702 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:35.725 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:51:35.755 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:51:35.786 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:51:35.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:35.817 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:51:35.839 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:35.852 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:51:35.864 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:35.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:35.882 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:51:35.919 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:51:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:51:35.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:51:35.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:51:35.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:51:35.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:51:35.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:36.008 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:36.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osdmap e1004: 12 total, 12 up, 10 in 2024-04-29T14:51:36.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/489880934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:36.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.375 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osdmap e1004: 12 total, 12 up, 10 in 2024-04-29T14:51:36.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/489880934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:36.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osdmap e1004: 12 total, 12 up, 10 in 2024-04-29T14:51:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/489880934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:37 smithi194 ceph-mon[44089]: pgmap v2529: 17 pgs: 1 active+recovering, 16 active+clean; 390 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 555/12470 objects misplaced (4.451%); 2.3 MiB/s, 36 objects/s recovering 2024-04-29T14:51:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:37 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:37 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:37 smithi040 ceph-mon[39581]: pgmap v2529: 17 pgs: 1 active+recovering, 16 active+clean; 390 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 555/12470 objects misplaced (4.451%); 2.3 MiB/s, 36 objects/s recovering 2024-04-29T14:51:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:37 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:37 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:37 smithi121 ceph-mon[44533]: pgmap v2529: 17 pgs: 1 active+recovering, 16 active+clean; 390 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 555/12470 objects misplaced (4.451%); 2.3 MiB/s, 36 objects/s recovering 2024-04-29T14:51:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:37 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:37 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:37.980 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:37.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:38.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:38.154 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:38.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:38.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:38.597 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:38 smithi040 ceph-mon[39581]: osdmap e1005: 12 total, 12 up, 10 in 2024-04-29T14:51:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:38 smithi194 ceph-mon[44089]: osdmap e1005: 12 total, 12 up, 10 in 2024-04-29T14:51:38.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:38 smithi121 ceph-mon[44533]: osdmap e1005: 12 total, 12 up, 10 in 2024-04-29T14:51:39.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 114909 objects 2024-04-29T14:51:39.299 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :902.564 2024-04-29T14:51:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:39 smithi040 ceph-mon[39581]: pgmap v2531: 17 pgs: 1 active+recovering, 16 active+clean; 344 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 102/11006 objects misplaced (0.927%); 6.0 MiB/s, 96 objects/s recovering 2024-04-29T14:51:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:39 smithi194 ceph-mon[44089]: pgmap v2531: 17 pgs: 1 active+recovering, 16 active+clean; 344 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 102/11006 objects misplaced (0.927%); 6.0 MiB/s, 96 objects/s recovering 2024-04-29T14:51:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:39 smithi121 ceph-mon[44533]: pgmap v2531: 17 pgs: 1 active+recovering, 16 active+clean; 344 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 102/11006 objects misplaced (0.927%); 6.0 MiB/s, 96 objects/s recovering 2024-04-29T14:51:39.972 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:39.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:40.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:40.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:40.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:40.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:40.229 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-04-29T14:51:40.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-04-29T14:51:40.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:40.291 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:40.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:40.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:40.425 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:40.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:40.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:40.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:40.580 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:40.600 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:40.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:40.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:40.734 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:40.772 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:40.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:40.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:40.907 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:40.927 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:40.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:41.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:41.064 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:41.085 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:41.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:41.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:41.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:41.222 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:41.243 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:41.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:41.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:41.377 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:41.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:41.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:41.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:41.528 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:41.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:41.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:41 smithi194 ceph-mon[44089]: pgmap v2532: 17 pgs: 1 active+recovering, 16 active+clean; 189 MiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 102/6022 objects misplaced (1.694%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:51:41.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:41 smithi040 ceph-mon[39581]: pgmap v2532: 17 pgs: 1 active+recovering, 16 active+clean; 189 MiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 102/6022 objects misplaced (1.694%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:51:41.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:41 smithi121 ceph-mon[44533]: pgmap v2532: 17 pgs: 1 active+recovering, 16 active+clean; 189 MiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 102/6022 objects misplaced (1.694%); 3.5 MiB/s, 56 objects/s recovering 2024-04-29T14:51:41.651 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:41.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:51:41.683 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:41.705 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:41.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:41.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:41.839 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:41.859 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:41.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:41.963 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:41.991 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:42.011 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:42.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:42.116 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:42.144 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:42.163 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:42.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:42.241 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:51:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:51:42.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:42.293 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:42.312 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:42.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:42.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:42.409 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:51:42.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:42.445 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:42.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:42.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:42.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1586106888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:51:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:42 smithi194 ceph-mon[44089]: from='client.28075 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:51:42.598 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:42.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:42.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:42.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1586106888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:51:42.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:42 smithi040 ceph-mon[39581]: from='client.28075 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:51:42.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:42.747 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1586106888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:51:42.748 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:42 smithi121 ceph-mon[44533]: from='client.28075 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:51:42.750 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:42.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:42.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:42.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:43.212 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2532,"stamp":"2024-04-29T14:51:40.320164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":197722680,"num_objects":3011,"num_object_clones":0,"num_object_copies":6022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":102,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":226992,"num_write_kb":7357115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234464,"num_bytes_recovered":13766852344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":7889848,"kb_used_data":1026444,"kb_used_omap":668,"kb_used_meta":6862563,"kb_avail":929561672,"statfs":{"total":959950356480,"available":951871152128,"internally_reserved":0,"allocated":1051078656,"data_stored":1029606083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7027265519},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-559808512,"num_objects":-8542,"num_object_clones":0,"num_object_copies":-17084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-453,"num_objects_unfound":0,"num_objects_dirty":-8542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":29687808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001642"},"pg_stats":[{"pgid":"7.f","version":"1004'14113","reported_seq":49139,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:35.402153+0000","last_change":"2024-04-29T14:50:38.109612+0000","last_active":"2024-04-29T14:51:35.402153+0000","last_peered":"2024-04-29T14:51:35.402153+0000","last_clean":"2024-04-29T14:51:35.402153+0000","last_became_active":"2024-04-29T14:47:59.564525+0000","last_became_peered":"2024-04-29T14:47:59.564525+0000","last_unstale":"2024-04-29T14:51:35.402153+0000","last_undegraded":"2024-04-29T14:51:35.402153+0000","last_fullsized":"2024-04-29T14:51:35.402153+0000","mapping_epoch":918,"log_start":"667'4100","ondisk_log_start":"667'4100","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12570","last_scrub_stamp":"2024-04-29T14:50:38.109478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:38.109478+0000","objects_scrubbed":1874,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:03.361035+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14113,"num_write_kb":462208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16923,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"1005'21606","reported_seq":43692,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-04-29T14:51:39.280244+0000","last_change":"2024-04-29T14:51:23.433502+0000","last_active":"2024-04-29T14:51:39.280244+0000","last_peered":"2024-04-29T14:51:39.280244+0000","last_clean":"2024-04-29T14:51:39.280244+0000","last_became_active":"2024-04-29T14:48:27.563523+0000","last_became_peered":"2024-04-29T14:48:27.563523+0000","last_unstale":"2024-04-29T14:51:39.280244+0000","last_undegraded":"2024-04-29T14:51:39.280244+0000","last_fullsized":"2024-04-29T14:51:39.280244+0000","mapping_epoch":930,"log_start":"705'11558","ondisk_log_start":"705'11558","created":655,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'20581","last_scrub_stamp":"2024-04-29T14:51:23.433402+0000","last_deep_scrub":"709'14617","last_deep_scrub_stamp":"2024-04-29T14:37:57.239106+0000","last_clean_scrub_stamp":"2024-04-29T14:51:23.433402+0000","objects_scrubbed":1025,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:48.261279+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14550,"num_write_kb":465600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10383,"num_bytes_recovered":680460288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.d","version":"1004'14000","reported_seq":45232,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:35.565888+0000","last_change":"2024-04-29T14:51:21.470456+0000","last_active":"2024-04-29T14:51:35.565888+0000","last_peered":"2024-04-29T14:51:35.565888+0000","last_clean":"2024-04-29T14:51:35.565888+0000","last_became_active":"2024-04-29T14:48:39.585286+0000","last_became_peered":"2024-04-29T14:48:39.585286+0000","last_unstale":"2024-04-29T14:51:35.565888+0000","last_undegraded":"2024-04-29T14:51:35.565888+0000","last_fullsized":"2024-04-29T14:51:35.565888+0000","mapping_epoch":938,"log_start":"667'3982","ondisk_log_start":"667'3982","created":655,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"999'13234","last_scrub_stamp":"2024-04-29T14:51:21.470297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:21.470297+0000","objects_scrubbed":1093,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:34.115923+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299224,"num_objects":326,"num_object_clones":0,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14000,"num_write_kb":458369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15108,"num_bytes_recovered":877002776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"1004'17219","reported_seq":82795,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:35.518533+0000","last_change":"2024-04-29T14:50:34.579123+0000","last_active":"2024-04-29T14:51:35.518533+0000","last_peered":"2024-04-29T14:51:35.518533+0000","last_clean":"2024-04-29T14:51:35.518533+0000","last_became_active":"2024-04-29T14:48:40.597062+0000","last_became_peered":"2024-04-29T14:48:40.597062+0000","last_unstale":"2024-04-29T14:51:35.518533+0000","last_undegraded":"2024-04-29T14:51:35.518533+0000","last_fullsized":"2024-04-29T14:51:35.518533+0000","mapping_epoch":939,"log_start":"683'7150","ondisk_log_start":"683'7150","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'15542","last_scrub_stamp":"2024-04-29T14:50:34.578998+0000","last_deep_scrub":"946'14374","last_deep_scrub_stamp":"2024-04-29T14:48:59.043790+0000","last_clean_scrub_stamp":"2024-04-29T14:50:34.578998+0000","objects_scrubbed":1966,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:38.386277+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18939904,"num_objects":289,"num_object_clones":0,"num_object_copies":578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14059,"num_write_kb":459136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32161,"num_bytes_recovered":1820721152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"1004'14065","reported_seq":35599,"reported_epoch":1004,"state":"active+recovering","last_fresh":"2024-04-29T14:51:36.541312+0000","last_change":"2024-04-29T14:48:55.372453+0000","last_active":"2024-04-29T14:51:36.541312+0000","last_peered":"2024-04-29T14:51:36.541312+0000","last_clean":"2024-04-29T14:35:46.185312+0000","last_became_active":"2024-04-29T14:48:48.169551+0000","last_became_peered":"2024-04-29T14:48:48.169551+0000","last_unstale":"2024-04-29T14:51:36.541312+0000","last_undegraded":"2024-04-29T14:51:36.541312+0000","last_fullsized":"2024-04-29T14:51:36.541312+0000","mapping_epoch":942,"log_start":"667'4000","ondisk_log_start":"667'4000","created":655,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:34:48.362150+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":102,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14065,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7466,"num_bytes_recovered":489160704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","4","11"],"object_location_counts":[{"shards":"1,5","objects":115},{"shards":"1,10,11","objects":102}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"1004'21440","reported_seq":38984,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:35.364169+0000","last_change":"2024-04-29T14:50:45.604038+0000","last_active":"2024-04-29T14:51:35.364169+0000","last_peered":"2024-04-29T14:51:35.364169+0000","last_clean":"2024-04-29T14:51:35.364169+0000","last_became_active":"2024-04-29T14:47:06.596290+0000","last_became_peered":"2024-04-29T14:47:06.596290+0000","last_unstale":"2024-04-29T14:51:35.364169+0000","last_undegraded":"2024-04-29T14:51:35.364169+0000","last_fullsized":"2024-04-29T14:51:35.364169+0000","mapping_epoch":895,"log_start":"705'11400","ondisk_log_start":"705'11400","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"991'19938","last_scrub_stamp":"2024-04-29T14:50:45.603880+0000","last_deep_scrub":"710'14736","last_deep_scrub_stamp":"2024-04-29T14:37:59.002803+0000","last_clean_scrub_stamp":"2024-04-29T14:50:45.603880+0000","objects_scrubbed":1811,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:01.560720+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14366,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8106,"num_bytes_recovered":531234816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"1004'14116","reported_seq":43321,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:36.526867+0000","last_change":"2024-04-29T14:48:36.519131+0000","last_active":"2024-04-29T14:51:36.526867+0000","last_peered":"2024-04-29T14:51:36.526867+0000","last_clean":"2024-04-29T14:51:36.526867+0000","last_became_active":"2024-04-29T14:40:30.550586+0000","last_became_peered":"2024-04-29T14:40:30.550586+0000","last_unstale":"2024-04-29T14:51:36.526867+0000","last_undegraded":"2024-04-29T14:51:36.526867+0000","last_fullsized":"2024-04-29T14:51:36.526867+0000","mapping_epoch":756,"log_start":"667'4100","ondisk_log_start":"667'4100","created":655,"last_epoch_clean":757,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11127","last_scrub_stamp":"2024-04-29T14:48:36.518685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.518685+0000","objects_scrubbed":3213,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14116,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14181,"num_bytes_recovered":888799232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"1004'14032","reported_seq":23176,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:36.237088+0000","last_change":"2024-04-29T14:51:30.157450+0000","last_active":"2024-04-29T14:51:36.237088+0000","last_peered":"2024-04-29T14:51:36.237088+0000","last_clean":"2024-04-29T14:51:36.237088+0000","last_became_active":"2024-04-29T14:46:50.856952+0000","last_became_peered":"2024-04-29T14:46:50.856952+0000","last_unstale":"2024-04-29T14:51:36.237088+0000","last_undegraded":"2024-04-29T14:51:36.237088+0000","last_fullsized":"2024-04-29T14:51:36.237088+0000","mapping_epoch":883,"log_start":"667'4000","ondisk_log_start":"667'4000","created":655,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'13538","last_scrub_stamp":"2024-04-29T14:51:30.157234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:30.157234+0000","objects_scrubbed":753,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:54.235618+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14032,"num_write_kb":457280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"1004'14389","reported_seq":72774,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:35.567608+0000","last_change":"2024-04-29T14:50:39.638903+0000","last_active":"2024-04-29T14:51:35.567608+0000","last_peered":"2024-04-29T14:51:35.567608+0000","last_clean":"2024-04-29T14:51:35.567608+0000","last_became_active":"2024-04-29T14:48:40.591982+0000","last_became_peered":"2024-04-29T14:48:40.591982+0000","last_unstale":"2024-04-29T14:51:35.567608+0000","last_undegraded":"2024-04-29T14:51:35.567608+0000","last_fullsized":"2024-04-29T14:51:35.567608+0000","mapping_epoch":939,"log_start":"668'4376","ondisk_log_start":"668'4376","created":655,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'12718","last_scrub_stamp":"2024-04-29T14:50:39.638817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:39.638817+0000","objects_scrubbed":2012,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:07.740501+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14389,"num_write_kb":471296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28605,"num_bytes_recovered":1749352448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"1004'13954","reported_seq":48275,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:35.397628+0000","last_change":"2024-04-29T14:50:36.645439+0000","last_active":"2024-04-29T14:51:35.397628+0000","last_peered":"2024-04-29T14:51:35.397628+0000","last_clean":"2024-04-29T14:51:35.397628+0000","last_became_active":"2024-04-29T14:47:59.564728+0000","last_became_peered":"2024-04-29T14:47:59.564728+0000","last_unstale":"2024-04-29T14:51:35.397628+0000","last_undegraded":"2024-04-29T14:51:35.397628+0000","last_fullsized":"2024-04-29T14:51:35.397628+0000","mapping_epoch":918,"log_start":"666'3900","ondisk_log_start":"666'3900","created":655,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"987'12376","last_scrub_stamp":"2024-04-29T14:50:36.645270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:50:36.645270+0000","objects_scrubbed":1898,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:51:54.914264+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16546,"num_bytes_recovered":817627136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"1005'14504","reported_seq":50941,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-04-29T14:51:39.251689+0000","last_change":"2024-04-29T14:51:33.203608+0000","last_active":"2024-04-29T14:51:39.251689+0000","last_peered":"2024-04-29T14:51:39.251689+0000","last_clean":"2024-04-29T14:51:39.251689+0000","last_became_active":"2024-04-29T14:47:55.641039+0000","last_became_peered":"2024-04-29T14:47:55.641039+0000","last_unstale":"2024-04-29T14:51:39.251689+0000","last_undegraded":"2024-04-29T14:51:39.251689+0000","last_fullsized":"2024-04-29T14:51:39.251689+0000","mapping_epoch":915,"log_start":"668'4500","ondisk_log_start":"668'4500","created":655,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'13996","last_scrub_stamp":"2024-04-29T14:51:33.203299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:33.203299+0000","objects_scrubbed":518,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:36.678557+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14504,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1002700800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1005'18177","reported_seq":43040,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-04-29T14:51:38.585857+0000","last_change":"2024-04-29T14:51:33.017965+0000","last_active":"2024-04-29T14:51:38.585857+0000","last_peered":"2024-04-29T14:51:38.585857+0000","last_clean":"2024-04-29T14:51:38.585857+0000","last_became_active":"2024-04-29T14:47:06.596160+0000","last_became_peered":"2024-04-29T14:47:06.596160+0000","last_unstale":"2024-04-29T14:51:38.585857+0000","last_undegraded":"2024-04-29T14:51:38.585857+0000","last_fullsized":"2024-04-29T14:51:38.585857+0000","mapping_epoch":895,"log_start":"746'8125","ondisk_log_start":"746'8125","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'17713","last_scrub_stamp":"2024-04-29T14:51:33.017756+0000","last_deep_scrub":"1003'17713","last_deep_scrub_stamp":"2024-04-29T14:51:33.017756+0000","last_clean_scrub_stamp":"2024-04-29T14:51:33.017756+0000","objects_scrubbed":656,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:55.187562+0000","scrub_duration":2467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4259840,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14343,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11795,"num_bytes_recovered":759234560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"1005'14133","reported_seq":38657,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-04-29T14:51:38.560650+0000","last_change":"2024-04-29T14:51:25.671619+0000","last_active":"2024-04-29T14:51:38.560650+0000","last_peered":"2024-04-29T14:51:38.560650+0000","last_clean":"2024-04-29T14:51:38.560650+0000","last_became_active":"2024-04-29T14:47:06.596717+0000","last_became_peered":"2024-04-29T14:47:06.596717+0000","last_unstale":"2024-04-29T14:51:38.560650+0000","last_undegraded":"2024-04-29T14:51:38.560650+0000","last_fullsized":"2024-04-29T14:51:38.560650+0000","mapping_epoch":895,"log_start":"667'4044","ondisk_log_start":"667'4044","created":655,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'13293","last_scrub_stamp":"2024-04-29T14:51:25.671430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:25.671430+0000","objects_scrubbed":900,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:33.187005+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14133,"num_write_kb":454080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11680,"num_bytes_recovered":765460480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"1005'14178","reported_seq":61029,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-04-29T14:51:39.283871+0000","last_change":"2024-04-29T14:51:34.213037+0000","last_active":"2024-04-29T14:51:39.283871+0000","last_peered":"2024-04-29T14:51:39.283871+0000","last_clean":"2024-04-29T14:51:39.283871+0000","last_became_active":"2024-04-29T14:48:25.086674+0000","last_became_peered":"2024-04-29T14:48:25.086674+0000","last_unstale":"2024-04-29T14:51:39.283871+0000","last_undegraded":"2024-04-29T14:51:39.283871+0000","last_fullsized":"2024-04-29T14:51:39.283871+0000","mapping_epoch":928,"log_start":"667'4100","ondisk_log_start":"667'4100","created":655,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'13739","last_scrub_stamp":"2024-04-29T14:51:34.212840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:34.212840+0000","objects_scrubbed":448,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:58.233128+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14178,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22450,"num_bytes_recovered":1215889408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"1005'14190","reported_seq":29215,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-04-29T14:51:38.689734+0000","last_change":"2024-04-29T14:48:36.431232+0000","last_active":"2024-04-29T14:51:38.689734+0000","last_peered":"2024-04-29T14:51:38.689734+0000","last_clean":"2024-04-29T14:51:38.689734+0000","last_became_active":"2024-04-29T14:37:23.385340+0000","last_became_peered":"2024-04-29T14:37:23.385340+0000","last_unstale":"2024-04-29T14:51:38.689734+0000","last_undegraded":"2024-04-29T14:51:38.689734+0000","last_fullsized":"2024-04-29T14:51:38.689734+0000","mapping_epoch":703,"log_start":"667'4100","ondisk_log_start":"667'4100","created":655,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":0,"last_scrub":"935'11048","last_scrub_stamp":"2024-04-29T14:48:36.431016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:48:36.431016+0000","objects_scrubbed":3196,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14190,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7108,"num_bytes_recovered":465829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1004'13819","reported_seq":35235,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:36.192849+0000","last_change":"2024-04-29T14:51:18.131718+0000","last_active":"2024-04-29T14:51:36.192849+0000","last_peered":"2024-04-29T14:51:36.192849+0000","last_clean":"2024-04-29T14:51:36.192849+0000","last_became_active":"2024-04-29T14:46:16.333414+0000","last_became_peered":"2024-04-29T14:46:16.333414+0000","last_unstale":"2024-04-29T14:51:36.192849+0000","last_undegraded":"2024-04-29T14:51:36.192849+0000","last_fullsized":"2024-04-29T14:51:36.192849+0000","mapping_epoch":865,"log_start":"666'3800","ondisk_log_start":"666'3800","created":655,"last_epoch_clean":866,"parent":"0.0","parent_split_bits":0,"last_scrub":"999'12972","last_scrub_stamp":"2024-04-29T14:51:18.131573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:34:48.362150+0000","last_clean_scrub_stamp":"2024-04-29T14:51:18.131573+0000","objects_scrubbed":1069,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:42.308743+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":221,"num_object_clones":0,"num_object_copies":442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13819,"num_write_kb":449280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10106,"num_bytes_recovered":500301824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2005,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-04-29T14:51:35.365289+0000","last_change":"2024-04-29T14:51:31.274002+0000","last_active":"2024-04-29T14:51:35.365289+0000","last_peered":"2024-04-29T14:51:35.365289+0000","last_clean":"2024-04-29T14:51:35.365289+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:51:35.365289+0000","last_undegraded":"2024-04-29T14:51:35.365289+0000","last_fullsized":"2024-04-29T14:51:35.365289+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:51:31.273868+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:59.621883+0000","scrub_duration":260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":197132312,"num_objects":3009,"num_object_clones":0,"num_object_copies":6018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":102,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226811,"num_write_kb":7354177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234450,"num_bytes_recovered":13763149848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1388781568,"data_stored":1388773424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160700,"ondisk_log_size":160700,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122585,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384225,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1355132,"kb_used_data":471536,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92390020,"statfs":{"total":95995035648,"available":94607380480,"internally_reserved":0,"allocated":482852864,"data_stored":480701502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":904788416},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645859,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906784,"kb_used_data":91224,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92838368,"statfs":{"total":95995035648,"available":95066488832,"internally_reserved":0,"allocated":93413376,"data_stored":91274504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423164,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218144,"kb_used_data":73176,"kb_used_omap":4,"kb_used_meta":144955,"kb_avail":93527008,"statfs":{"total":95995035648,"available":95771656192,"internally_reserved":0,"allocated":74932224,"data_stored":72793352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":148434624},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234540,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896184,"kb_used_data":38956,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92848968,"statfs":{"total":95995035648,"available":95077343232,"internally_reserved":0,"allocated":39890944,"data_stored":37742285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":877787594},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1050236,"kb_used_data":66672,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":92694916,"statfs":{"total":95995035648,"available":94919593984,"internally_reserved":0,"allocated":68272128,"data_stored":66119397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1007155654},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575421,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535476,"kb_used_data":53096,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93209676,"statfs":{"total":95995035648,"available":95446708224,"internally_reserved":0,"allocated":54370304,"data_stored":52221125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738959,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015840,"kb_used_data":91928,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92729312,"statfs":{"total":95995035648,"available":94954815488,"internally_reserved":0,"allocated":94134272,"data_stored":91995400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497393,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182979,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":608448,"kb_used_data":39984,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93136704,"statfs":{"total":95995035648,"available":95371984896,"internally_reserved":0,"allocated":40943616,"data_stored":38792274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986744,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":984956,"kb_used_data":11120,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92760196,"statfs":{"total":95995035648,"available":94986440704,"internally_reserved":0,"allocated":11386880,"data_stored":9235538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653305,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318648,"kb_used_data":88752,"kb_used_omap":651,"kb_used_meta":229236,"kb_avail":93426504,"statfs":{"total":95995035648,"available":95668740096,"internally_reserved":0,"allocated":90882048,"data_stored":88730706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":234738440},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71696384,"data_stored":71696384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45420544,"data_stored":45416472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15597568,"data_stored":15597568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":84279296,"data_stored":84279296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":62459904,"data_stored":62455832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31326208,"data_stored":31326208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":64880640,"data_stored":64880640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":84344832,"data_stored":84344832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":461570048,"data_stored":461570048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":467140608,"data_stored":467140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:51:43.213 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.b is not active+clean 2024-04-29T14:51:43.213 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.b', 'version': "1004'14065", 'reported_seq': 35599, 'reported_epoch': 1004, 'state': 'active+recovering', 'last_fresh': '2024-04-29T14:51:36.541312+0000', 'last_change': '2024-04-29T14:48:55.372453+0000', 'last_active': '2024-04-29T14:51:36.541312+0000', 'last_peered': '2024-04-29T14:51:36.541312+0000', 'last_clean': '2024-04-29T14:35:46.185312+0000', 'last_became_active': '2024-04-29T14:48:48.169551+0000', 'last_became_peered': '2024-04-29T14:48:48.169551+0000', 'last_unstale': '2024-04-29T14:51:36.541312+0000', 'last_undegraded': '2024-04-29T14:51:36.541312+0000', 'last_fullsized': '2024-04-29T14:51:36.541312+0000', 'mapping_epoch': 942, 'log_start': "667'4000", 'ondisk_log_start': "667'4000", 'created': 655, 'last_epoch_clean': 656, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'last_clean_scrub_stamp': '2024-04-29T14:34:48.362150+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14221312, 'num_objects': 217, 'num_object_clones': 0, 'num_object_copies': 434, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 102, 'num_objects_unfound': 0, 'num_objects_dirty': 217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14065, 'num_write_kb': 457024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7466, 'num_bytes_recovered': 489160704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': ['1', '4', '11'], 'object_location_counts': [{'shards': '1,5', 'objects': 115}, {'shards': '1,10,11', 'objects': 102}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-29T14:51:43.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:51:43.620 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: from='client.18351 v1:172.21.15.40:0/3307125322' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: pgmap v2533: 17 pgs: 17 active+clean; 12 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:51:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: from='client.28075 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:51:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: osdmap e1006: 12 total, 12 up, 10 in 2024-04-29T14:51:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: 7.c deep-scrub starts 2024-04-29T14:51:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: 7.c deep-scrub ok 2024-04-29T14:51:43.621 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:43 smithi040 ceph-mon[39581]: osdmap e1007: 12 total, 12 up, 10 in 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:152121 7.4 7:229fee58:::benchmark_data_smithi194_147200_object37209:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e831)", 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:22.537824+0000", 2024-04-29T14:51:43.784 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 441.05502213300002, 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23480100400000001, 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 152121 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.785 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537824+0000", 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537824+0000", 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537827+0000", 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.497e-06 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537834+0000", 2024-04-29T14:51:43.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5320000000000001e-06 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537847+0000", 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2944e-05 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537859+0000", 2024-04-29T14:51:43.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.117e-05 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539767+0000", 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001908423 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539779+0000", 2024-04-29T14:51:43.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2086000000000001e-05 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.771928+0000", 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23214864700000001 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.772625+0000", 2024-04-29T14:51:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00069770499999999996 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:152168 7.4 7:26cbbbcb:::benchmark_data_smithi194_147200_object37256:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e831)", 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:22.537875+0000", 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 441.054971094, 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23824916400000001, 2024-04-29T14:51:43.791 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 152168 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537875+0000", 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537875+0000", 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537876+0000", 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.32e-07 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537878+0000", 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-06 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537883+0000", 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.622e-06 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537886+0000", 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3919999999999999e-06 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539824+0000", 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00193808 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539832+0000", 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7489999999999998e-06 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.775497+0000", 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23566506300000001 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.776125+0000", 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062757000000000004 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:152176 7.4 7:2e82d51c:::benchmark_data_smithi194_147200_object37264:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e831)", 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:22.537890+0000", 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 441.05495631000002, 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23883427700000001, 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.798 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 152176 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537890+0000", 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537890+0000", 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537891+0000", 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6399999999999998e-07 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537892+0000", 2024-04-29T14:51:43.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4810000000000001e-06 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537896+0000", 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0690000000000003e-06 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537900+0000", 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6100000000000002e-06 2024-04-29T14:51:43.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539854+0000", 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00195414 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539860+0000", 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.93e-06 2024-04-29T14:51:43.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.776170+0000", 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23631000899999999 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.776725+0000", 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00055467400000000001 2024-04-29T14:51:43.803 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:152200 7.4 7:22636903:::benchmark_data_smithi194_147200_object37288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:22.537903+0000", 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 441.054943337, 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23956914300000001, 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 152200 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537903+0000", 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537903+0000", 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537904+0000", 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7300000000000002e-07 2024-04-29T14:51:43.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537905+0000", 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5039999999999999e-06 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537910+0000", 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5730000000000001e-06 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537913+0000", 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0000000000000001e-06 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539879+0000", 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019666779999999999 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539885+0000", 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1920000000000002e-06 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.776787+0000", 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23690150700000001 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.777472+0000", 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000685416 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:152220 7.4 7:2c88868b:::benchmark_data_smithi194_147200_object37308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:22.537919+0000", 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 441.05492787499998, 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24018151500000001, 2024-04-29T14:51:43.810 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 152220 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537919+0000", 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537919+0000", 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537919+0000", 2024-04-29T14:51:43.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4400000000000001e-07 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537921+0000", 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6500000000000001e-06 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537925+0000", 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8580000000000002e-06 2024-04-29T14:51:43.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.537927+0000", 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7860000000000001e-06 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539904+0000", 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00197649 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.539909+0000", 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5899999999999998e-06 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.777553+0000", 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23764317500000001 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.778100+0000", 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000547622 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:152268 7.4 7:2c0d0d20:::benchmark_data_smithi194_147200_object37356:head [delete] snapc 0=[] ondisk+write+known_if_redirected e831)", 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:44:22.615357+0000", 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 440.977489338, 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.163380363, 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.816 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 152268 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.615357+0000", 2024-04-29T14:51:43.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.615357+0000", 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.615360+0000", 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0000000000000001e-06 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.615373+0000", 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2813000000000001e-05 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.615386+0000", 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2794e-05 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.615395+0000", 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9099999999999994e-06 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.630657+0000", 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015262636 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.630672+0000", 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4395e-05 2024-04-29T14:51:43.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.778145+0000", 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.14747327099999999 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:44:22.778738+0000", 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059254399999999997 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPushReply(7.c 878/877 [PushReplyOp(7:30f2c703:::benchmark_data_smithi194_147200_object87587:head)])", 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:38.433954+0000", 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 305.15889244300001, 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091479740000000004, 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.822 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.433954+0000", 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.433954+0000", 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.433955+0000", 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3799999999999998e-07 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.433958+0000", 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.439e-06 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.433964+0000", 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1800000000000001e-06 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.433971+0000", 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1570000000000003e-06 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525152+0000", 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.091180824999999993 2024-04-29T14:51:43.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525434+0000", 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028170099999999998 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.d 878/802 [PushOp(7:b6474a22:::benchmark_data_smithi194_147200_object107162:head, version: 681'6656, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:b6474a22:::benchmark_data_smithi194_147200_object107162:head@681'6656, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:43.826 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:38.435202+0000", 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 305.15764461200001, 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.092907308999999993, 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.435202+0000", 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.435202+0000", 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.435203+0000", 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0599999999999999e-07 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.435228+0000", 2024-04-29T14:51:43.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4807000000000002e-05 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525124+0000", 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.089896023000000005 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525149+0000", 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4890999999999999e-05 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525234+0000", 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5394000000000003e-05 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525247+0000", 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2948e-05 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.528109+0000", 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0028623400000000001 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPull(7.4 e878/869 cost 1049576)", 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:38.441753+0000", 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 305.151094065, 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.083853194000000006, 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.831 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.441753+0000", 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.441753+0000", 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.441754+0000", 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4470000000000001e-06 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.441765+0000", 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1311999999999999e-05 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525062+0000", 2024-04-29T14:51:43.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.083296591000000003 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525077+0000", 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5569e-05 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525132+0000", 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5052000000000001e-05 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.525606+0000", 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00047322299999999999 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPushReply(7.c 878/877 [PushReplyOp(7:31cb6247:::benchmark_data_smithi194_147200_object103824:head)])", 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:38.954379+0000", 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 304.63846748999998, 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.065888275999999996, 2024-04-29T14:51:43.835 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.954379+0000", 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.954379+0000", 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.954379+0000", 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9299999999999999e-07 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.954382+0000", 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3709999999999998e-06 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.954386+0000", 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8079999999999998e-06 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.954390+0000", 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2610000000000004e-06 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.954442+0000", 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2076000000000001e-05 2024-04-29T14:51:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020267+0000", 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.065825466999999999 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.d 878/802 [PushOp(7:b21fa59f:::benchmark_data_smithi194_147200_object107956:head, version: 681'6707, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:b21fa59f:::benchmark_data_smithi194_147200_object107956:head@681'6707, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:38.955473+0000", 2024-04-29T14:51:43.839 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 304.63737349399997, 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.068959338999999995, 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.955473+0000", 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.840 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.955473+0000", 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.955474+0000", 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1429999999999999e-06 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.955498+0000", 2024-04-29T14:51:43.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3586000000000001e-05 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020325+0000", 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.064827001999999995 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020352+0000", 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7389000000000002e-05 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020455+0000", 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000102787 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020469+0000", 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4151000000000001e-05 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.024432+0000", 2024-04-29T14:51:43.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039632809999999999 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPull(7.4 e878/869 cost 1049576)", 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:38.972297+0000", 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 304.62054985399999, 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.048675825999999998, 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.844 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.972297+0000", 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.972297+0000", 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.972299+0000", 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4049999999999998e-06 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:38.972313+0000", 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.428e-05 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020238+0000", 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.047924903999999997 2024-04-29T14:51:43.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020252+0000", 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3422e-05 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020627+0000", 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037541700000000001 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.020973+0000", 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034539800000000002 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164446 7.c 7:3faa0234:::benchmark_data_smithi194_147200_object49534:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:39.193673+0000", 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 304.39917380600002, 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040139507999999997, 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.848 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164446 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193673+0000", 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193673+0000", 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193674+0000", 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6759999999999999e-06 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193680+0000", 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3530000000000002e-06 2024-04-29T14:51:43.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193689+0000", 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4220000000000006e-06 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193694+0000", 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1580000000000004e-06 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193761+0000", 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6606000000000006e-05 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193889+0000", 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012843500000000001 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 2,9", 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.193979+0000", 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9820000000000004e-05 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.199359+0000", 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053799540000000002 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.199991+0000", 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000631348 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:43.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.233645+0000", 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033654475000000003 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.233702+0000", 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6848999999999997e-05 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.233812+0000", 2024-04-29T14:51:43.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000110412 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPush(7.d 878/802 [PushOp(7:bbeba6f3:::benchmark_data_smithi194_147200_object108933:head, version: 681'6769, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:bbeba6f3:::benchmark_data_smithi194_147200_object108933:head@681'6769, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:39.512301+0000", 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 304.08054607100001, 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051812097000000001, 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512301+0000", 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512301+0000", 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512302+0000", 2024-04-29T14:51:43.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4869999999999999e-06 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512333+0000", 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1263000000000002e-05 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512350+0000", 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.666e-05 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512365+0000", 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5009e-05 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512432+0000", 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6775999999999999e-05 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512440+0000", 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9140000000000008e-06 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.564113+0000", 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051672988000000003 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPushReply(7.c 878/877 [PushReplyOp(7:32be0c2d:::benchmark_data_smithi194_147200_object65761:head)])", 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:39.512781+0000", 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 304.08006607499999, 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.05151816, 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.859 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512781+0000", 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512781+0000", 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512781+0000", 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9800000000000004e-07 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512784+0000", 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1099999999999999e-06 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512790+0000", 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9020000000000004e-06 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512796+0000", 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1909999999999997e-06 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.512838+0000", 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.138e-05 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.564299+0000", 2024-04-29T14:51:43.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.051461079 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164494 7.c 7:3852af12:::benchmark_data_smithi194_147200_object49582:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:39.519628+0000", 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 304.073218408, 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.050301285000000001, 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:51:43.863 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164494 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.519628+0000", 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.519628+0000", 2024-04-29T14:51:43.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.519630+0000", 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2809999999999998e-06 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.519637+0000", 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5520000000000001e-06 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.564148+0000", 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.044510987000000002 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.564159+0000", 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1314999999999999e-05 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.564354+0000", 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019481899999999999 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.564497+0000", 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014260400000000001 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 2,9", 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.564636+0000", 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013929799999999999 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.569617+0000", 2024-04-29T14:51:43.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049809930000000004 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.569639+0000", 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1471000000000002e-05 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.569682+0000", 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3930000000000001e-05 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.569762+0000", 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9158000000000003e-05 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:39.569929+0000", 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016787700000000001 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:228880 7.4 e1005/915 7:2776da2d:::benchmark_data_smithi194_147200_object113968:head v 1005'14433, mlcod=1005'14431)", 2024-04-29T14:51:43.869 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:51:38.633946+0000", 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.958900388, 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032861884000000001, 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633946+0000", 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633946+0000", 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633948+0000", 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6530000000000001e-06 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633956+0000", 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3359999999999995e-06 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633969+0000", 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2979999999999999e-05 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633977+0000", 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7430000000000002e-06 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634030+0000", 2024-04-29T14:51:43.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3025999999999997e-05 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634044+0000", 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4439e-05 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666775+0000", 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032730599999999999 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666808+0000", 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3107e-05 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:228882 7.4 e1005/915 7:22381866:::benchmark_data_smithi194_147200_object113970:head v 1005'14434, mlcod=1005'14431)", 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:51:38.633994+0000", 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.9588526399999999, 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032840358, 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.874 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633994+0000", 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633994+0000", 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633994+0000", 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3799999999999997e-07 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.633999+0000", 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5340000000000001e-06 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634012+0000", 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3001e-05 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634017+0000", 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0159999999999999e-06 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.637753+0000", 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003736283 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.637772+0000", 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.84e-05 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666813+0000", 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029041582999999999 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666834+0000", 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1002999999999999e-05 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:228883 7.4 e1005/915 7:26f27236:::benchmark_data_smithi194_147200_object113971:head v 1005'14435, mlcod=1005'14431)", 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:51:38.634031+0000", 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.9588151499999995, 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032822495, 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634031+0000", 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634031+0000", 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634032+0000", 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5999999999999999e-07 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634039+0000", 2024-04-29T14:51:43.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5789999999999998e-06 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634045+0000", 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3900000000000001e-06 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.634049+0000", 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.279e-06 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.642486+0000", 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084370929999999997 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.642510+0000", 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3419000000000001e-05 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666837+0000", 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024326977999999999 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666854+0000", 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7297000000000001e-05 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:228894 7.4 e1005/915 7:2aaed4ef:::benchmark_data_smithi194_147200_object113982:head v 1005'14436, mlcod=1005'14431)", 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:51:38.636201+0000", 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.956645816, 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030673892000000001, 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:51:43.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.636201+0000", 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.636201+0000", 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:51:43.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.636203+0000", 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3429999999999998e-06 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.636214+0000", 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0798999999999999e-05 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.636230+0000", 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6028e-05 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:51:43.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.636238+0000", 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2840000000000006e-06 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.647215+0000", 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010977252 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.647241+0000", 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5579e-05 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666857+0000", 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019615648999999999 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:51:38.666875+0000", 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7958e-05 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:51:43.887 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:51:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: from='client.18351 v1:172.21.15.40:0/3307125322' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: pgmap v2533: 17 pgs: 17 active+clean; 12 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:51:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: from='client.28075 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:51:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: osdmap e1006: 12 total, 12 up, 10 in 2024-04-29T14:51:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: 7.c deep-scrub starts 2024-04-29T14:51:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: 7.c deep-scrub ok 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:43 smithi121 ceph-mon[44533]: osdmap e1007: 12 total, 12 up, 10 in 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: from='client.18351 v1:172.21.15.40:0/3307125322' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: pgmap v2533: 17 pgs: 17 active+clean; 12 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: from='client.28075 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: osdmap e1006: 12 total, 12 up, 10 in 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: 7.c deep-scrub starts 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: 7.c deep-scrub ok 2024-04-29T14:51:43.889 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:43 smithi194 ceph-mon[44089]: osdmap e1007: 12 total, 12 up, 10 in 2024-04-29T14:51:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:43.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:44.060 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:44.084 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:44.092 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:44.113 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:44.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:44.131 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:44.218 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:44.247 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:44.373 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:44.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:44.478 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:44.510 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:44.532 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:44.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:44.636 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:44.666 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:44.719 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:44.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:44.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:44.857 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:44.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:44.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:45.011 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:45.460 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:45.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:45.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:45.597 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:45.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:45.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:45.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:45.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:45.753 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:45 smithi194 ceph-mon[44089]: pgmap v2536: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 792 op/s; 940 KiB/s, 14 objects/s recovering 2024-04-29T14:51:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:45.776 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:45.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:45 smithi040 ceph-mon[39581]: pgmap v2536: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 792 op/s; 940 KiB/s, 14 objects/s recovering 2024-04-29T14:51:45.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:51:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:45 smithi121 ceph-mon[44533]: pgmap v2536: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 792 op/s; 940 KiB/s, 14 objects/s recovering 2024-04-29T14:51:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:45.913 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:45.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:51:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:45.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:51:46.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:46.074 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:46.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:51:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:46.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:46.258 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:46.290 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:46.311 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:46.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:46.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:46.444 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:46.465 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:46.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:46.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:46.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:46.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:46.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:46 smithi194 ceph-mon[44089]: osdmap e1008: 12 total, 12 up, 10 in 2024-04-29T14:51:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:46 smithi040 ceph-mon[39581]: osdmap e1008: 12 total, 12 up, 10 in 2024-04-29T14:51:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:46 smithi121 ceph-mon[44533]: osdmap e1008: 12 total, 12 up, 10 in 2024-04-29T14:51:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_5' removed 2024-04-29T14:51:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:47.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:47 smithi040 ceph-mon[39581]: pgmap v2538: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 501 op/s; 1.1 MiB/s, 16 objects/s recovering 2024-04-29T14:51:47.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:47 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:47.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:47 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:47.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4141015503' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:51:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:47.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:47.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:47.684 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:47.783 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:47.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:47 smithi194 ceph-mon[44089]: pgmap v2538: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 501 op/s; 1.1 MiB/s, 16 objects/s recovering 2024-04-29T14:51:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:47 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:47 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4141015503' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:51:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:47 smithi121 ceph-mon[44533]: pgmap v2538: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 501 op/s; 1.1 MiB/s, 16 objects/s recovering 2024-04-29T14:51:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:47 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:47 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:47.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4141015503' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:51:47.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:48.403 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:51:48.404 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:51:48.404 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-04-29T14:51:48.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-04-29T14:51:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4141015503' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:51:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:48 smithi194 ceph-mon[44089]: osdmap e1009: 12 total, 12 up, 10 in 2024-04-29T14:51:48.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4141015503' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:51:48.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:48 smithi040 ceph-mon[39581]: osdmap e1009: 12 total, 12 up, 10 in 2024-04-29T14:51:48.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4141015503' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:51:48.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:48 smithi121 ceph-mon[44533]: osdmap e1009: 12 total, 12 up, 10 in 2024-04-29T14:51:48.924 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:51:49.565 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:49.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:49.675 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:49.701 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:49 smithi194 ceph-mon[44089]: pgmap v2540: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:49.703 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:49.709 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:49 smithi040 ceph-mon[39581]: pgmap v2540: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:49.723 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:49.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:49.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:49.857 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:49 smithi121 ceph-mon[44533]: pgmap v2540: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:49.990 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:49.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:50.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:50.125 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:50.474 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:50.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:50.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:50.628 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:50.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:50.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:50.732 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:50.764 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:50.786 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:50.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:50.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:50.918 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:50.940 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:50.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:51.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:51:51.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:51:51.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:51:51.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:51:51.041 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:51:51.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:51:51.049 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:51.053 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:51.053 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:51.068 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:51:51.075 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:51.096 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:51:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:51.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:51:51.103 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:51:51.124 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:51:51.144 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:51:51.164 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:51:51.184 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:51:51.200 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:51.204 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:51:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:51:51.228 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:51.249 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:51.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:51.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:51:51.262 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2985,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1009,"num_osds":12,"num_up_osds":12,"osd_up_since":1714401063,"num_in_osds":10,"osd_in_since":1714402012,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1}],"num_pgs":1,"num_pools":1,"num_objects":2,"data_bytes":590368,"bytes_used":7602036736,"bytes_avail":952348319744,"bytes_total":959950356480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":72,"modified":"2024-04-29T14:50:02.301403+0000","services":{}},"progress_events":{}} 2024-04-29T14:51:51.353 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:51.381 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:51.401 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:51.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:51.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: pgmap v2541: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:51.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1680779575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:51.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4026558329' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:51:51.505 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:51.535 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: pgmap v2541: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1680779575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4026558329' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:51:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: pgmap v2541: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:51.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1680779575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4026558329' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:51:51.887 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:51.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:51.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:52.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:52.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:52.261 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:52.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:52.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:52.385 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:51:52.414 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:52.434 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:51:52.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:52.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:51:52.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:51:52.567 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:52.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.768 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:51:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:52.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:51:52.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:51:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:53.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:53 smithi194 ceph-mon[44089]: pgmap v2542: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:53.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:53 smithi040 ceph-mon[39581]: pgmap v2542: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:53.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:53 smithi121 ceph-mon[44533]: pgmap v2542: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:54.551 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:54.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:54.657 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:54.675 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:51:54.689 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:54.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:54.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:54.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:51:54.704 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:51:54.710 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:54.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:54.712 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:54.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:51:54.713 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:51:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:54.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:54.814 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:54.844 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:54.864 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:54.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:54.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:54.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:54.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:51:54.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:51:54.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:54.997 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:55.309 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-29T14:51:55.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:51:55.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:55 smithi040 ceph-mon[39581]: pgmap v2543: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:55.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:55 smithi040 ceph-mon[39581]: osdmap e1010: 12 total, 12 up, 10 in 2024-04-29T14:51:55.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:55 smithi194 ceph-mon[44089]: pgmap v2543: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:55.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:55 smithi194 ceph-mon[44089]: osdmap e1010: 12 total, 12 up, 10 in 2024-04-29T14:51:55.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:55 smithi121 ceph-mon[44533]: pgmap v2543: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:55.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:55 smithi121 ceph-mon[44533]: osdmap e1010: 12 total, 12 up, 10 in 2024-04-29T14:51:55.911 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:51:55.911 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:51:56.347 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:56.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:56.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:56.484 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:56.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:56.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:56.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:56.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:56.641 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:56.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:56.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:56.765 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:51:56.798 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:56.819 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:51:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:56.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:51:56.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:51:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:51:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:56.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:51:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:51:57.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:51:57.117 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:57.138 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:51:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:57.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:51:57.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:57.326 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:57.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:57.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_6' created 2024-04-29T14:51:57.487 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:57.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:51:57.591 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:51:57.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:57 smithi040 ceph-mon[39581]: pgmap v2545: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:57.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4219179424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-29T14:51:57.624 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:57.646 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:51:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:57.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:51:57.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:51:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:57 smithi194 ceph-mon[44089]: pgmap v2545: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4219179424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-29T14:51:57.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:57 smithi121 ceph-mon[44533]: pgmap v2545: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:57.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4219179424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-29T14:51:58.469 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:51:58.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:51:58.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4219179424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-29T14:51:58.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:58 smithi040 ceph-mon[39581]: osdmap e1011: 12 total, 12 up, 10 in 2024-04-29T14:51:58.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3231146097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:58.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4219179424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-29T14:51:58.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:58 smithi194 ceph-mon[44089]: osdmap e1011: 12 total, 12 up, 10 in 2024-04-29T14:51:58.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3231146097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4219179424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-29T14:51:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:58 smithi121 ceph-mon[44533]: osdmap e1011: 12 total, 12 up, 10 in 2024-04-29T14:51:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3231146097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:51:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:51:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:58.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:51:59.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:51:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:59.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:51:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:59.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:51:59.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:51:59.225 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:59.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:59 smithi040 ceph-mon[39581]: pgmap v2547: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:59.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:59 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:51:59.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:59 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:59.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3231146097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:59.617 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:51:59 smithi040 ceph-mon[39581]: osdmap e1012: 12 total, 12 up, 10 in 2024-04-29T14:51:59.775 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:51:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:59.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:51:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:59 smithi194 ceph-mon[44089]: pgmap v2547: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:59 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:51:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:59 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3231146097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:51:59 smithi194 ceph-mon[44089]: osdmap e1012: 12 total, 12 up, 10 in 2024-04-29T14:51:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:59 smithi121 ceph-mon[44533]: pgmap v2547: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:51:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:59 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:51:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:59 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:51:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3231146097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:51:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:51:59 smithi121 ceph-mon[44533]: osdmap e1012: 12 total, 12 up, 10 in 2024-04-29T14:51:59.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:51:59.921 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:51:59.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:51:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:51:59.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:00.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:00.052 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:00.079 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:00.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-04-29T14:52:00.101 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:00.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:00.206 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:00.233 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:00.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:00 smithi040 ceph-mon[39581]: osdmap e1013: 12 total, 12 up, 10 in 2024-04-29T14:52:00.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:00 smithi194 ceph-mon[44089]: osdmap e1013: 12 total, 12 up, 10 in 2024-04-29T14:52:00.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:00 smithi121 ceph-mon[44533]: osdmap e1013: 12 total, 12 up, 10 in 2024-04-29T14:52:00.986 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:00.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:01.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:01.123 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:01.144 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:01.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:01.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:01.278 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:01.298 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:01.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:01.402 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:01.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:01 smithi040 ceph-mon[39581]: pgmap v2550: 17 pgs: 8 unknown, 9 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:01.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1533450507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:52:01.795 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:52:01.796 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:52:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:52:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:52:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:52:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:52:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:52:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:52:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:52:01.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:01 smithi194 ceph-mon[44089]: pgmap v2550: 17 pgs: 8 unknown, 9 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:01.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1533450507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:01.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:01 smithi121 ceph-mon[44533]: pgmap v2550: 17 pgs: 8 unknown, 9 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:01.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1533450507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:02.486 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:02.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:02.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:52:02.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:02.620 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.852 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:02.870 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:02.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:02.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:02.894 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:02.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:03.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:03.139 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:03.171 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:03.192 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:03.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:03.297 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:03.326 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:03.346 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:03.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:03.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:03.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:03.483 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:03.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:03.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:03.610 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:03.659 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:03.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:03.763 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:03.789 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:03 smithi194 ceph-mon[44089]: pgmap v2551: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:03.791 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:03.811 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:03.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:03 smithi040 ceph-mon[39581]: pgmap v2551: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:03.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:03 smithi121 ceph-mon[44533]: pgmap v2551: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:03.915 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:03.942 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:03.961 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:03.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:04.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:04.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:04.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:04.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:04.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:04.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:04.406 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:04.424 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:04.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:04.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:52:04.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:04.530 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:52:04.530 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:52:04.561 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:04.582 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:04.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:04.686 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:04.751 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:04.753 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:04.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:04.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:04.888 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:04.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:04.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:05.013 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:05.042 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:05.060 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:05.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:05.164 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:05.186 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2551,"stamp":"2024-04-29T14:52:02.324852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":7423648,"kb_used_data":488808,"kb_used_omap":668,"kb_used_meta":6930595,"kb_avail":930027872,"statfs":{"total":959950356480,"available":952348540928,"internally_reserved":0,"allocated":500539392,"data_stored":481679780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7096930287},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001209"},"pg_stats":[{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.647321+0000","last_change":"2024-04-29T14:51:58.472775+0000","last_active":"2024-04-29T14:51:59.647321+0000","last_peered":"2024-04-29T14:51:59.647321+0000","last_clean":"2024-04-29T14:51:59.647321+0000","last_became_active":"2024-04-29T14:51:58.471518+0000","last_became_peered":"2024-04-29T14:51:58.471518+0000","last_unstale":"2024-04-29T14:51:59.647321+0000","last_undegraded":"2024-04-29T14:51:59.647321+0000","last_fullsized":"2024-04-29T14:51:59.647321+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:26.447399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.470661+0000","last_change":"2024-04-29T14:51:58.469053+0000","last_active":"2024-04-29T14:51:58.470661+0000","last_peered":"2024-04-29T14:51:58.470661+0000","last_clean":"2024-04-29T14:51:58.470661+0000","last_became_active":"2024-04-29T14:51:58.468700+0000","last_became_peered":"2024-04-29T14:51:58.468700+0000","last_unstale":"2024-04-29T14:51:58.470661+0000","last_undegraded":"2024-04-29T14:51:58.470661+0000","last_fullsized":"2024-04-29T14:51:58.470661+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:08.680758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.647361+0000","last_change":"2024-04-29T14:51:58.469796+0000","last_active":"2024-04-29T14:51:59.647361+0000","last_peered":"2024-04-29T14:51:59.647361+0000","last_clean":"2024-04-29T14:51:59.647361+0000","last_became_active":"2024-04-29T14:51:58.469439+0000","last_became_peered":"2024-04-29T14:51:58.469439+0000","last_unstale":"2024-04-29T14:51:59.647361+0000","last_undegraded":"2024-04-29T14:51:59.647361+0000","last_fullsized":"2024-04-29T14:51:59.647361+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:11.520533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.647800+0000","last_change":"2024-04-29T14:51:58.471505+0000","last_active":"2024-04-29T14:51:59.647800+0000","last_peered":"2024-04-29T14:51:59.647800+0000","last_clean":"2024-04-29T14:51:59.647800+0000","last_became_active":"2024-04-29T14:51:58.471228+0000","last_became_peered":"2024-04-29T14:51:58.471228+0000","last_unstale":"2024-04-29T14:51:59.647800+0000","last_undegraded":"2024-04-29T14:51:59.647800+0000","last_fullsized":"2024-04-29T14:51:59.647800+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:03.242294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.469884+0000","last_change":"2024-04-29T14:51:58.469525+0000","last_active":"2024-04-29T14:51:59.469884+0000","last_peered":"2024-04-29T14:51:59.469884+0000","last_clean":"2024-04-29T14:51:59.469884+0000","last_became_active":"2024-04-29T14:51:58.469253+0000","last_became_peered":"2024-04-29T14:51:58.469253+0000","last_unstale":"2024-04-29T14:51:59.469884+0000","last_undegraded":"2024-04-29T14:51:59.469884+0000","last_fullsized":"2024-04-29T14:51:59.469884+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:01.655678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.469892+0000","last_change":"2024-04-29T14:51:58.470084+0000","last_active":"2024-04-29T14:51:59.469892+0000","last_peered":"2024-04-29T14:51:59.469892+0000","last_clean":"2024-04-29T14:51:59.469892+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:51:59.469892+0000","last_undegraded":"2024-04-29T14:51:59.469892+0000","last_fullsized":"2024-04-29T14:51:59.469892+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:11.159805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.471537+0000","last_change":"2024-04-29T14:51:58.470103+0000","last_active":"2024-04-29T14:51:58.471537+0000","last_peered":"2024-04-29T14:51:58.471537+0000","last_clean":"2024-04-29T14:51:58.471537+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:51:58.471537+0000","last_undegraded":"2024-04-29T14:51:58.471537+0000","last_fullsized":"2024-04-29T14:51:58.471537+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:20.104331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.647493+0000","last_change":"2024-04-29T14:51:58.473278+0000","last_active":"2024-04-29T14:51:59.647493+0000","last_peered":"2024-04-29T14:51:59.647493+0000","last_clean":"2024-04-29T14:51:59.647493+0000","last_became_active":"2024-04-29T14:51:58.473066+0000","last_became_peered":"2024-04-29T14:51:58.473066+0000","last_unstale":"2024-04-29T14:51:59.647493+0000","last_undegraded":"2024-04-29T14:51:59.647493+0000","last_fullsized":"2024-04-29T14:51:59.647493+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:02.184503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.473375+0000","last_change":"2024-04-29T14:51:58.472827+0000","last_active":"2024-04-29T14:51:58.473375+0000","last_peered":"2024-04-29T14:51:58.473375+0000","last_clean":"2024-04-29T14:51:58.473375+0000","last_became_active":"2024-04-29T14:51:58.472520+0000","last_became_peered":"2024-04-29T14:51:58.472520+0000","last_unstale":"2024-04-29T14:51:58.473375+0000","last_undegraded":"2024-04-29T14:51:58.473375+0000","last_fullsized":"2024-04-29T14:51:58.473375+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:16.607430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.473467+0000","last_change":"2024-04-29T14:51:58.473051+0000","last_active":"2024-04-29T14:51:58.473467+0000","last_peered":"2024-04-29T14:51:58.473467+0000","last_clean":"2024-04-29T14:51:58.473467+0000","last_became_active":"2024-04-29T14:51:58.472757+0000","last_became_peered":"2024-04-29T14:51:58.472757+0000","last_unstale":"2024-04-29T14:51:58.473467+0000","last_undegraded":"2024-04-29T14:51:58.473467+0000","last_fullsized":"2024-04-29T14:51:58.473467+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:25.588633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.470610+0000","last_change":"2024-04-29T14:51:58.469765+0000","last_active":"2024-04-29T14:51:58.470610+0000","last_peered":"2024-04-29T14:51:58.470610+0000","last_clean":"2024-04-29T14:51:58.470610+0000","last_became_active":"2024-04-29T14:51:58.469461+0000","last_became_peered":"2024-04-29T14:51:58.469461+0000","last_unstale":"2024-04-29T14:51:58.470610+0000","last_undegraded":"2024-04-29T14:51:58.470610+0000","last_fullsized":"2024-04-29T14:51:58.470610+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:58.389080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.647361+0000","last_change":"2024-04-29T14:51:58.469033+0000","last_active":"2024-04-29T14:51:59.647361+0000","last_peered":"2024-04-29T14:51:59.647361+0000","last_clean":"2024-04-29T14:51:59.647361+0000","last_became_active":"2024-04-29T14:51:58.468800+0000","last_became_peered":"2024-04-29T14:51:58.468800+0000","last_unstale":"2024-04-29T14:51:59.647361+0000","last_undegraded":"2024-04-29T14:51:59.647361+0000","last_fullsized":"2024-04-29T14:51:59.647361+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:06.715721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.474220+0000","last_change":"2024-04-29T14:51:58.473947+0000","last_active":"2024-04-29T14:51:58.474220+0000","last_peered":"2024-04-29T14:51:58.474220+0000","last_clean":"2024-04-29T14:51:58.474220+0000","last_became_active":"2024-04-29T14:51:58.473704+0000","last_became_peered":"2024-04-29T14:51:58.473704+0000","last_unstale":"2024-04-29T14:51:58.474220+0000","last_undegraded":"2024-04-29T14:51:58.474220+0000","last_fullsized":"2024-04-29T14:51:58.474220+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:16.968561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.647336+0000","last_change":"2024-04-29T14:51:58.471146+0000","last_active":"2024-04-29T14:51:59.647336+0000","last_peered":"2024-04-29T14:51:59.647336+0000","last_clean":"2024-04-29T14:51:59.647336+0000","last_became_active":"2024-04-29T14:51:58.470887+0000","last_became_peered":"2024-04-29T14:51:58.470887+0000","last_unstale":"2024-04-29T14:51:59.647336+0000","last_undegraded":"2024-04-29T14:51:59.647336+0000","last_fullsized":"2024-04-29T14:51:59.647336+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:03.093955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.472542+0000","last_change":"2024-04-29T14:51:58.472256+0000","last_active":"2024-04-29T14:51:58.472542+0000","last_peered":"2024-04-29T14:51:58.472542+0000","last_clean":"2024-04-29T14:51:58.472542+0000","last_became_active":"2024-04-29T14:51:58.471940+0000","last_became_peered":"2024-04-29T14:51:58.471940+0000","last_unstale":"2024-04-29T14:51:58.472542+0000","last_undegraded":"2024-04-29T14:51:58.472542+0000","last_fullsized":"2024-04-29T14:51:58.472542+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:12.296421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-04-29T14:51:58.473146+0000","last_change":"2024-04-29T14:51:58.472764+0000","last_active":"2024-04-29T14:51:58.473146+0000","last_peered":"2024-04-29T14:51:58.473146+0000","last_clean":"2024-04-29T14:51:58.473146+0000","last_became_active":"2024-04-29T14:51:58.472489+0000","last_became_peered":"2024-04-29T14:51:58.472489+0000","last_unstale":"2024-04-29T14:51:58.473146+0000","last_undegraded":"2024-04-29T14:51:58.473146+0000","last_fullsized":"2024-04-29T14:51:58.473146+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:19.827912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2023,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-04-29T14:51:59.647226+0000","last_change":"2024-04-29T14:51:31.274002+0000","last_active":"2024-04-29T14:51:59.647226+0000","last_peered":"2024-04-29T14:51:59.647226+0000","last_clean":"2024-04-29T14:51:59.647226+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:51:59.647226+0000","last_undegraded":"2024-04-29T14:51:59.647226+0000","last_fullsized":"2024-04-29T14:51:59.647226+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:51:31.273868+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:59.621883+0000","scrub_duration":260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122590,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384229,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1300068,"kb_used_data":416108,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92445084,"statfs":{"total":95995035648,"available":94663766016,"internally_reserved":0,"allocated":426094592,"data_stored":424174342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":904788416},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645864,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":811316,"kb_used_data":3524,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92933836,"statfs":{"total":95995035648,"available":95164248064,"internally_reserved":0,"allocated":3608576,"data_stored":1775359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":826603968},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423169,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153204,"kb_used_data":3524,"kb_used_omap":4,"kb_used_meta":149115,"kb_avail":93591948,"statfs":{"total":95995035648,"available":95838154752,"internally_reserved":0,"allocated":3608576,"data_stored":1775359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":152694464},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897916,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92847236,"statfs":{"total":95995035648,"available":95075569664,"internally_reserved":0,"allocated":4141056,"data_stored":2304287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":914749898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":992132,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92753020,"statfs":{"total":95995035648,"available":94979092480,"internally_reserved":0,"allocated":4141056,"data_stored":2304287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575426,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508476,"kb_used_data":3524,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93236676,"statfs":{"total":95995035648,"available":95474356224,"internally_reserved":0,"allocated":3608576,"data_stored":1775359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738964,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":915076,"kb_used_data":3524,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92830076,"statfs":{"total":95995035648,"available":95057997824,"internally_reserved":0,"allocated":3608576,"data_stored":1775359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497398,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182983,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":572500,"kb_used_data":3860,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93172652,"statfs":{"total":95995035648,"available":95408795648,"internally_reserved":0,"allocated":3952640,"data_stored":1946277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999916,"kb_used_data":3692,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92745236,"statfs":{"total":95995035648,"available":94971121664,"internally_reserved":0,"allocated":3780608,"data_stored":1860378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1019935173},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653309,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":273044,"kb_used_data":42964,"kb_used_omap":651,"kb_used_meta":229236,"kb_avail":93472108,"statfs":{"total":95995035648,"available":95715438592,"internally_reserved":0,"allocated":43995136,"data_stored":41988773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":234738440},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:52:05.192 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:05.211 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:05.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:05.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:05.344 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:05.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:05.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:52:05.554 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:05.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:52:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:05.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:52:05.680 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:05.708 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:05.727 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:05.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:05.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:05 smithi040 ceph-mon[39581]: from='client.18393 v1:172.21.15.40:0/4204584223' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:52:05.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:05 smithi040 ceph-mon[39581]: pgmap v2552: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:05 smithi194 ceph-mon[44089]: from='client.18393 v1:172.21.15.40:0/4204584223' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:52:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:05 smithi194 ceph-mon[44089]: pgmap v2552: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:05.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:05.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:05 smithi121 ceph-mon[44533]: from='client.18393 v1:172.21.15.40:0/4204584223' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:52:05.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:05 smithi121 ceph-mon[44533]: pgmap v2552: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:05.862 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:05.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:05.988 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:06.015 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:06.033 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:06.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:06.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:06.168 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:06.207 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:06.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:06.312 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:06.332 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:52:06.332 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:52:06.333 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:52:06.333 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:52:06.333 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:52:06.339 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:06.357 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:06.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:06.462 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:06.508 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:06.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:06.612 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:07.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: pgmap v2553: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:07.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=2814030303,config=1981513354,health=1989772851,kv=2214219203}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=3610712756,logm=2634272920}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=554093492}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209287513}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697555991}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132286849}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1450182813}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950468761}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539755239}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776364308}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460100659}) 2024-04-29T14:52:07.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419840721}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4090106845}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567956863}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219923055}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178331337}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007707767}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833277447}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=141091887}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1393189648}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243856756}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2989724415}) 2024-04-29T14:52:07.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183798436}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2404253098}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2057451114}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1195820920}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2847990802}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719219859}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478371726}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505898529}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843993977}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142280465}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888401753}) 2024-04-29T14:52:07.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204966049}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=427416781}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700830777}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026824570}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2234174131}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417539276}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366739870}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919433943}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968849330}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3754371686}) 2024-04-29T14:52:07.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974462646}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2912115425}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605845435}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1892830351}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329475196}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5509725}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4202113804}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448765500}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2465241994}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089922867}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269788031}) 2024-04-29T14:52:07.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362126024}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=495114027}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3950516630}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1702581307}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1153622300}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692024620}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3059986715}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1966202831}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076318957}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4176993300}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773469292}) 2024-04-29T14:52:07.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2246295082}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984022116}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523917892}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028957779}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502035610}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1266015139}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1699174746}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279571752}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167090385}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944293702}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1915357854}) 2024-04-29T14:52:07.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=346850248}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2331559221}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353032243}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651168400}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046182620}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339603224}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1973921027}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382745970}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333479405}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281627226}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1324458119}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3949875453}) 2024-04-29T14:52:07.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=610907845}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129841522}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592168004}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3660703303}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3882190228}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143788959}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986903637}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576796969}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110226501}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=710492339}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=440948909}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740292869}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2691303727}) 2024-04-29T14:52:07.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3693702491}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4229448564}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773127928}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2018044263}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972005025}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479900770}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800260726}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768968777}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771895656}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=927709256,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3635461207}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=12,mon_config_key=88} crc {mgrstat=3063860905,mon_config_key=1719629542}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1618032954}) 2024-04-29T14:52:07.749 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=90,monmap=5,osd_metadata=5} crc {mon_config_key=720394030,monmap=455069592,osd_metadata=3679688102}) 2024-04-29T14:52:07.750 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1121827151,osd_pg_creating=1910137163,osdmap=457034466}) 2024-04-29T14:52:07.750 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=21} crc {osdmap=1927946286}) 2024-04-29T14:52:07.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:52:07.779 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:07.791 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:07.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: pgmap v2553: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=2814030303,config=1981513354,health=1989772851,kv=2214219203}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=3610712756,logm=2634272920}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=554093492}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209287513}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697555991}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132286849}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1450182813}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950468761}) 2024-04-29T14:52:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539755239}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776364308}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460100659}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419840721}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4090106845}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567956863}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219923055}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178331337}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007707767}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833277447}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=141091887}) 2024-04-29T14:52:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1393189648}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243856756}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2989724415}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183798436}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2404253098}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2057451114}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1195820920}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2847990802}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719219859}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478371726}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505898529}) 2024-04-29T14:52:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843993977}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142280465}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888401753}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204966049}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=427416781}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700830777}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026824570}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2234174131}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417539276}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366739870}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919433943}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968849330}) 2024-04-29T14:52:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3754371686}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974462646}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2912115425}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605845435}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1892830351}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329475196}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5509725}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4202113804}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448765500}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2465241994}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089922867}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269788031}) 2024-04-29T14:52:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362126024}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=495114027}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3950516630}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1702581307}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1153622300}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692024620}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3059986715}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1966202831}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076318957}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4176993300}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773469292}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2246295082}) 2024-04-29T14:52:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984022116}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523917892}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028957779}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502035610}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1266015139}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1699174746}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279571752}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167090385}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944293702}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1915357854}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=346850248}) 2024-04-29T14:52:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2331559221}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353032243}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651168400}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046182620}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339603224}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1973921027}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382745970}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333479405}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281627226}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1324458119}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3949875453}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=610907845}) 2024-04-29T14:52:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129841522}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592168004}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3660703303}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3882190228}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143788959}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986903637}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576796969}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110226501}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=710492339}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=440948909}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740292869}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2691303727}) 2024-04-29T14:52:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3693702491}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4229448564}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773127928}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2018044263}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972005025}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479900770}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800260726}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768968777}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771895656}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=927709256,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3635461207}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=12,mon_config_key=88} crc {mgrstat=3063860905,mon_config_key=1719629542}) 2024-04-29T14:52:07.834 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1618032954}) 2024-04-29T14:52:07.835 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=90,monmap=5,osd_metadata=5} crc {mon_config_key=720394030,monmap=455069592,osd_metadata=3679688102}) 2024-04-29T14:52:07.835 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1121827151,osd_pg_creating=1910137163,osdmap=457034466}) 2024-04-29T14:52:07.835 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=21} crc {osdmap=1927946286}) 2024-04-29T14:52:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: pgmap v2553: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=2814030303,config=1981513354,health=1989772851,kv=2214219203}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=3610712756,logm=2634272920}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=554093492}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209287513}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697555991}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132286849}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1450182813}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950468761}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539755239}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776364308}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460100659}) 2024-04-29T14:52:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419840721}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4090106845}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567956863}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219923055}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178331337}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007707767}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833277447}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=141091887}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1393189648}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243856756}) 2024-04-29T14:52:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2989724415}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183798436}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2404253098}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2057451114}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1195820920}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2847990802}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719219859}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478371726}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505898529}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843993977}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=142280465}) 2024-04-29T14:52:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888401753}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204966049}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=427416781}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700830777}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026824570}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2234174131}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417539276}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366739870}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919433943}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968849330}) 2024-04-29T14:52:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3754371686}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974462646}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2912115425}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605845435}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1892830351}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329475196}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5509725}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4202113804}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448765500}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2465241994}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089922867}) 2024-04-29T14:52:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269788031}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362126024}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=495114027}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3950516630}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1702581307}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1153622300}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692024620}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3059986715}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1966202831}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076318957}) 2024-04-29T14:52:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4176993300}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773469292}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2246295082}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984022116}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523917892}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028957779}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502035610}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1266015139}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1699174746}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2279571752}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167090385}) 2024-04-29T14:52:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944293702}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1915357854}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=346850248}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2331559221}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353032243}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651168400}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4046182620}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339603224}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1973921027}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382745970}) 2024-04-29T14:52:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333479405}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281627226}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1324458119}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3949875453}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=610907845}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129841522}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592168004}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3660703303}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3882190228}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2143788959}) 2024-04-29T14:52:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986903637}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576796969}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110226501}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=710492339}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=440948909}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740292869}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2691303727}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3693702491}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4229448564}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773127928}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2018044263}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972005025}) 2024-04-29T14:52:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479900770}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800260726}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768968777}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771895656}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=927709256,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3635461207}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=12,mon_config_key=88} crc {mgrstat=3063860905,mon_config_key=1719629542}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1618032954}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=90,monmap=5,osd_metadata=5} crc {mon_config_key=720394030,monmap=455069592,osd_metadata=3679688102}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1121827151,osd_pg_creating=1910137163,osdmap=457034466}) 2024-04-29T14:52:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=21} crc {osdmap=1927946286}) 2024-04-29T14:52:07.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:07.926 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:08.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:52:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:52:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:52:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:08 smithi194 ceph-mon[44089]: osdmap e1014: 12 total, 12 up, 10 in 2024-04-29T14:52:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:52:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:52:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:08 smithi040 ceph-mon[39581]: osdmap e1014: 12 total, 12 up, 10 in 2024-04-29T14:52:08.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:52:08.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:52:08.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:08 smithi121 ceph-mon[44533]: osdmap e1014: 12 total, 12 up, 10 in 2024-04-29T14:52:09.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:09 smithi040 ceph-mon[39581]: pgmap v2555: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:09.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3715504899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:52:09.636 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-04-29T14:52:09.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:09 smithi194 ceph-mon[44089]: pgmap v2555: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:09.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3715504899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:52:09.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:09 smithi121 ceph-mon[44533]: pgmap v2555: 17 pgs: 17 active+clean; 577 KiB data, 7.1 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:09.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3715504899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:52:10.077 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:10.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:10.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:10.219 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:10.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3715504899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:52:10.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:10 smithi040 ceph-mon[39581]: osdmap e1015: 12 total, 12 up, 10 in 2024-04-29T14:52:10.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:10 smithi040 ceph-mon[39581]: pgmap v2557: 17 pgs: 17 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:10.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:11.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3715504899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:52:11.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:10 smithi194 ceph-mon[44089]: osdmap e1015: 12 total, 12 up, 10 in 2024-04-29T14:52:11.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:10 smithi194 ceph-mon[44089]: pgmap v2557: 17 pgs: 17 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:11.077 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:11.109 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:11.131 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:11.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3715504899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:52:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:10 smithi121 ceph-mon[44533]: osdmap e1015: 12 total, 12 up, 10 in 2024-04-29T14:52:11.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:10 smithi121 ceph-mon[44533]: pgmap v2557: 17 pgs: 17 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:11.235 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:11.266 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:11.287 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:11.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:11.364 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:11.366 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:11.391 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:11.420 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:11.440 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:11.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:11.543 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:11.572 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:11.592 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:11.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:11.644 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:52:11.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:11.727 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:11.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:11.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:11 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:52:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2573422995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:52:11.852 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:11.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:11 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:52:11.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2573422995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:52:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:11.900 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:11.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:12.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:12.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:11 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:52:12.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2573422995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:52:12.036 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:12.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:12.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:12.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:12.190 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:12.209 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:12.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:12.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:12.328 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:52:12.345 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:12.348 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:52:12.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:12.367 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:52:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:12.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:12.387 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:52:12.404 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-04-29T14:52:12.408 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:52:12.408 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:52:12.409 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:52:12.428 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:52:12.449 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:52:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:52:12.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:12.490 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:52:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:12.510 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:52:12.532 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:52:12.552 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:52:12.553 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:52:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:52:12.555 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:52:12.555 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:52:12.591 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:52:12.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:52:12.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_186548 2024-04-29T14:52:12.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:52:12.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:52:12.754 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:12.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:12.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:12 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:12.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2573422995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:52:12.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:12 smithi040 ceph-mon[39581]: osdmap e1016: 12 total, 12 up, 10 in 2024-04-29T14:52:12.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3408784142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:12.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:12 smithi040 ceph-mon[39581]: pgmap v2559: 17 pgs: 17 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:12.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:12.900 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:12.926 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:12.928 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:12 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:12.928 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2573422995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:52:12.928 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:12 smithi121 ceph-mon[44533]: osdmap e1016: 12 total, 12 up, 10 in 2024-04-29T14:52:12.928 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3408784142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:12.928 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:12 smithi121 ceph-mon[44533]: pgmap v2559: 17 pgs: 17 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:12.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:13.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:13.072 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:13.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:12 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:13.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2573422995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:52:13.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:12 smithi194 ceph-mon[44089]: osdmap e1016: 12 total, 12 up, 10 in 2024-04-29T14:52:13.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3408784142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:13.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:12 smithi194 ceph-mon[44089]: pgmap v2559: 17 pgs: 17 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T14:52:13.228 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:13.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:13.333 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:13.372 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:13.401 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:13.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:13.507 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:13.547 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:13.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 16 5652 5636 352.231 352.25 0.00124167 0.00272069 2024-04-29T14:52:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:13.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:13.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:13.768 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:14.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 16 9472 9456 295.417 238.75 0.00343472 0.00338109 2024-04-29T14:52:14.789 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:14 smithi040 ceph-mon[39581]: pgmap v2560: 17 pgs: 17 active+clean; 98 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 14 MiB/s wr, 227 op/s 2024-04-29T14:52:14.927 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:14.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:15.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:15.070 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:14 smithi121 ceph-mon[44533]: pgmap v2560: 17 pgs: 17 active+clean; 98 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 14 MiB/s wr, 227 op/s 2024-04-29T14:52:15.073 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:15.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:14 smithi194 ceph-mon[44089]: pgmap v2560: 17 pgs: 17 active+clean; 98 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 14 MiB/s wr, 227 op/s 2024-04-29T14:52:15.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:15.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:15.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:15.247 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:15.275 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:15.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:15.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:15.419 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:15.478 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:15.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:15.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:52:15.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 16 14478 14462 301.227 312.875 0.00218273 0.00331675 2024-04-29T14:52:15.622 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:15.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:52:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:15.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:52:15.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:15.791 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:15.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:15 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:52:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:15 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:52:15.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:52:15.960 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:15.986 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:52:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:15.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:52:16.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:15 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:52:16.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:15 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:52:16.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:16.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:52:16.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:15 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:52:16.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:15 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:52:16.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:16.138 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:52:16.138 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:52:16.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:16.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:16.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:16.305 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:16.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 15 18842 18827 294.117 272.812 0.00429404 0.00339755 2024-04-29T14:52:16.674 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:52:16.677 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:52:16.955 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:52:16.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:16 smithi040 ceph-mon[39581]: osdmap e1017: 12 total, 12 up, 10 in 2024-04-29T14:52:16.987 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:16 smithi040 ceph-mon[39581]: pgmap v2562: 17 pgs: 17 active+clean; 454 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:52:17.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:16 smithi194 ceph-mon[44089]: osdmap e1017: 12 total, 12 up, 10 in 2024-04-29T14:52:17.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:16 smithi194 ceph-mon[44089]: pgmap v2562: 17 pgs: 17 active+clean; 454 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:52:17.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:16 smithi121 ceph-mon[44533]: osdmap e1017: 12 total, 12 up, 10 in 2024-04-29T14:52:17.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:16 smithi121 ceph-mon[44533]: pgmap v2562: 17 pgs: 17 active+clean; 454 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:52:17.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 15 23021 23006 287.527 261.188 0.00340556 0.00347505 2024-04-29T14:52:18.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:17 smithi194 ceph-mon[44089]: from='client.28147 v1:172.21.15.40:0/1201509229' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:52:18.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:17 smithi040 ceph-mon[39581]: from='client.28147 v1:172.21.15.40:0/1201509229' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:52:18.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:17 smithi121 ceph-mon[44533]: from='client.28147 v1:172.21.15.40:0/1201509229' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:52:18.592 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 15 27008 26993 281.134 249.188 0.00257286 0.00355496 2024-04-29T14:52:18.966 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:18.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:18 smithi040 ceph-mon[39581]: pgmap v2563: 17 pgs: 17 active+clean; 748 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:52:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:18.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:19.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:18 smithi194 ceph-mon[44089]: pgmap v2563: 17 pgs: 17 active+clean; 748 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:52:19.102 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:18 smithi121 ceph-mon[44533]: pgmap v2563: 17 pgs: 17 active+clean; 748 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-29T14:52:19.104 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:19.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:19.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:19.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:19.260 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:19.284 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:19.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:19.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:19.594 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 16 30216 30200 269.561 200.438 0.00318201 0.0037079 2024-04-29T14:52:20.594 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 16 33746 33730 263.443 220.625 0.00432119 0.0037942 2024-04-29T14:52:20.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:52:21.416 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:21.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:21.465 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2562,"stamp":"2024-04-29T14:52:16.327763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476512800,"num_objects":7264,"num_object_clones":0,"num_object_copies":14528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7264,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":7443,"num_write_kb":467706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7401,"ondisk_log_size":7401,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8356432,"kb_used_data":1383176,"kb_used_omap":668,"kb_used_meta":6970851,"kb_avail":929095088,"statfs":{"total":959950356480,"available":951393370112,"internally_reserved":0,"allocated":1416372224,"data_stored":1400622375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":684049,"internal_metadata":7138152431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":475922432,"num_objects":7262,"num_object_clones":0,"num_object_copies":14524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7262,"num_write_kb":464768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.697099"},"pg_stats":[{"pgid":"8.f","version":"0'0","reported_seq":17,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-04-29T14:52:09.630513+0000","last_change":"2024-04-29T14:51:58.472775+0000","last_active":"2024-04-29T14:52:09.630513+0000","last_peered":"2024-04-29T14:52:09.630513+0000","last_clean":"2024-04-29T14:52:09.630513+0000","last_became_active":"2024-04-29T14:51:58.471518+0000","last_became_peered":"2024-04-29T14:51:58.471518+0000","last_unstale":"2024-04-29T14:52:09.630513+0000","last_undegraded":"2024-04-29T14:52:09.630513+0000","last_fullsized":"2024-04-29T14:52:09.630513+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:25.652676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"1016'481","reported_seq":500,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:14.285224+0000","last_change":"2024-04-29T14:51:58.469053+0000","last_active":"2024-04-29T14:52:14.285224+0000","last_peered":"2024-04-29T14:52:14.285224+0000","last_clean":"2024-04-29T14:52:14.285224+0000","last_became_active":"2024-04-29T14:51:58.468700+0000","last_became_peered":"2024-04-29T14:51:58.468700+0000","last_unstale":"2024-04-29T14:52:14.285224+0000","last_undegraded":"2024-04-29T14:52:14.285224+0000","last_fullsized":"2024-04-29T14:52:14.285224+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:26.495667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31522816,"num_objects":481,"num_object_clones":0,"num_object_copies":962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":481,"num_write_kb":30784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"1017'1107","reported_seq":1128,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-04-29T14:52:16.230144+0000","last_change":"2024-04-29T14:51:58.469796+0000","last_active":"2024-04-29T14:52:16.230144+0000","last_peered":"2024-04-29T14:52:16.230144+0000","last_clean":"2024-04-29T14:52:16.230144+0000","last_became_active":"2024-04-29T14:51:58.469439+0000","last_became_peered":"2024-04-29T14:51:58.469439+0000","last_unstale":"2024-04-29T14:52:16.230144+0000","last_undegraded":"2024-04-29T14:52:16.230144+0000","last_fullsized":"2024-04-29T14:52:16.230144+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":1107,"log_dups_size":0,"ondisk_log_size":1107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:25.062862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72548352,"num_objects":1107,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1107,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"1016'863","reported_seq":882,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:15.422186+0000","last_change":"2024-04-29T14:51:58.471505+0000","last_active":"2024-04-29T14:52:15.422186+0000","last_peered":"2024-04-29T14:52:15.422186+0000","last_clean":"2024-04-29T14:52:15.422186+0000","last_became_active":"2024-04-29T14:51:58.471228+0000","last_became_peered":"2024-04-29T14:51:58.471228+0000","last_unstale":"2024-04-29T14:52:15.422186+0000","last_undegraded":"2024-04-29T14:52:15.422186+0000","last_fullsized":"2024-04-29T14:52:15.422186+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":863,"log_dups_size":0,"ondisk_log_size":863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:24.887869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56557568,"num_objects":863,"num_object_clones":0,"num_object_copies":1726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":863,"num_write_kb":55232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"1016'797","reported_seq":816,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:15.401165+0000","last_change":"2024-04-29T14:51:58.469525+0000","last_active":"2024-04-29T14:52:15.401165+0000","last_peered":"2024-04-29T14:52:15.401165+0000","last_clean":"2024-04-29T14:52:15.401165+0000","last_became_active":"2024-04-29T14:51:58.469253+0000","last_became_peered":"2024-04-29T14:51:58.469253+0000","last_unstale":"2024-04-29T14:52:15.401165+0000","last_undegraded":"2024-04-29T14:52:15.401165+0000","last_fullsized":"2024-04-29T14:52:15.401165+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:10.589281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52232192,"num_objects":797,"num_object_clones":0,"num_object_copies":1594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":797,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"1016'832","reported_seq":851,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:15.408250+0000","last_change":"2024-04-29T14:51:58.470084+0000","last_active":"2024-04-29T14:52:15.408250+0000","last_peered":"2024-04-29T14:52:15.408250+0000","last_clean":"2024-04-29T14:52:15.408250+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:52:15.408250+0000","last_undegraded":"2024-04-29T14:52:15.408250+0000","last_fullsized":"2024-04-29T14:52:15.408250+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:09.863537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":832,"num_object_clones":0,"num_object_copies":1664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":832,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"1016'377","reported_seq":396,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:13.561088+0000","last_change":"2024-04-29T14:51:58.470103+0000","last_active":"2024-04-29T14:52:13.561088+0000","last_peered":"2024-04-29T14:52:13.561088+0000","last_clean":"2024-04-29T14:52:13.561088+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:52:13.561088+0000","last_undegraded":"2024-04-29T14:52:13.561088+0000","last_fullsized":"2024-04-29T14:52:13.561088+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:21.298554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":377,"num_write_kb":24128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":17,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-04-29T14:52:09.630573+0000","last_change":"2024-04-29T14:51:58.473278+0000","last_active":"2024-04-29T14:52:09.630573+0000","last_peered":"2024-04-29T14:52:09.630573+0000","last_clean":"2024-04-29T14:52:09.630573+0000","last_became_active":"2024-04-29T14:51:58.473066+0000","last_became_peered":"2024-04-29T14:51:58.473066+0000","last_unstale":"2024-04-29T14:52:09.630573+0000","last_undegraded":"2024-04-29T14:52:09.630573+0000","last_fullsized":"2024-04-29T14:52:09.630573+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:05.001209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"1016'555","reported_seq":574,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:14.402946+0000","last_change":"2024-04-29T14:51:58.472827+0000","last_active":"2024-04-29T14:52:14.402946+0000","last_peered":"2024-04-29T14:52:14.402946+0000","last_clean":"2024-04-29T14:52:14.402946+0000","last_became_active":"2024-04-29T14:51:58.472520+0000","last_became_peered":"2024-04-29T14:51:58.472520+0000","last_unstale":"2024-04-29T14:52:14.402946+0000","last_undegraded":"2024-04-29T14:52:14.402946+0000","last_fullsized":"2024-04-29T14:52:14.402946+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:58.861569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36372480,"num_objects":555,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":35520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"1016'534","reported_seq":553,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:14.400528+0000","last_change":"2024-04-29T14:51:58.473051+0000","last_active":"2024-04-29T14:52:14.400528+0000","last_peered":"2024-04-29T14:52:14.400528+0000","last_clean":"2024-04-29T14:52:14.400528+0000","last_became_active":"2024-04-29T14:51:58.472757+0000","last_became_peered":"2024-04-29T14:51:58.472757+0000","last_unstale":"2024-04-29T14:52:14.400528+0000","last_undegraded":"2024-04-29T14:52:14.400528+0000","last_fullsized":"2024-04-29T14:52:14.400528+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:16.083407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":534,"num_write_kb":34176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"1016'347","reported_seq":366,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:13.508697+0000","last_change":"2024-04-29T14:51:58.469765+0000","last_active":"2024-04-29T14:52:13.508697+0000","last_peered":"2024-04-29T14:52:13.508697+0000","last_clean":"2024-04-29T14:52:13.508697+0000","last_became_active":"2024-04-29T14:51:58.469461+0000","last_became_peered":"2024-04-29T14:51:58.469461+0000","last_unstale":"2024-04-29T14:52:13.508697+0000","last_undegraded":"2024-04-29T14:52:13.508697+0000","last_fullsized":"2024-04-29T14:52:13.508697+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:10.774140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":17,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-04-29T14:52:09.630505+0000","last_change":"2024-04-29T14:51:58.469033+0000","last_active":"2024-04-29T14:52:09.630505+0000","last_peered":"2024-04-29T14:52:09.630505+0000","last_clean":"2024-04-29T14:52:09.630505+0000","last_became_active":"2024-04-29T14:51:58.468800+0000","last_became_peered":"2024-04-29T14:51:58.468800+0000","last_unstale":"2024-04-29T14:52:09.630505+0000","last_undegraded":"2024-04-29T14:52:09.630505+0000","last_fullsized":"2024-04-29T14:52:09.630505+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:19.235549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"1016'520","reported_seq":539,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:14.400497+0000","last_change":"2024-04-29T14:51:58.473947+0000","last_active":"2024-04-29T14:52:14.400497+0000","last_peered":"2024-04-29T14:52:14.400497+0000","last_clean":"2024-04-29T14:52:14.400497+0000","last_became_active":"2024-04-29T14:51:58.473704+0000","last_became_peered":"2024-04-29T14:51:58.473704+0000","last_unstale":"2024-04-29T14:52:14.400497+0000","last_undegraded":"2024-04-29T14:52:14.400497+0000","last_fullsized":"2024-04-29T14:52:14.400497+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:11.292464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34078720,"num_objects":520,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":520,"num_write_kb":33280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":17,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-04-29T14:52:09.630511+0000","last_change":"2024-04-29T14:51:58.471146+0000","last_active":"2024-04-29T14:52:09.630511+0000","last_peered":"2024-04-29T14:52:09.630511+0000","last_clean":"2024-04-29T14:52:09.630511+0000","last_became_active":"2024-04-29T14:51:58.470887+0000","last_became_peered":"2024-04-29T14:51:58.470887+0000","last_unstale":"2024-04-29T14:52:09.630511+0000","last_undegraded":"2024-04-29T14:52:09.630511+0000","last_fullsized":"2024-04-29T14:52:09.630511+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:25.422272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"1016'346","reported_seq":365,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:13.567055+0000","last_change":"2024-04-29T14:51:58.472256+0000","last_active":"2024-04-29T14:52:13.567055+0000","last_peered":"2024-04-29T14:52:13.567055+0000","last_clean":"2024-04-29T14:52:13.567055+0000","last_became_active":"2024-04-29T14:51:58.471940+0000","last_became_peered":"2024-04-29T14:51:58.471940+0000","last_unstale":"2024-04-29T14:52:13.567055+0000","last_undegraded":"2024-04-29T14:52:13.567055+0000","last_fullsized":"2024-04-29T14:52:13.567055+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:04.201741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":346,"num_object_clones":0,"num_object_copies":692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":346,"num_write_kb":22144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"1016'503","reported_seq":522,"reported_epoch":1016,"state":"active+clean","last_fresh":"2024-04-29T14:52:14.406672+0000","last_change":"2024-04-29T14:51:58.472764+0000","last_active":"2024-04-29T14:52:14.406672+0000","last_peered":"2024-04-29T14:52:14.406672+0000","last_clean":"2024-04-29T14:52:14.406672+0000","last_became_active":"2024-04-29T14:51:58.472489+0000","last_became_peered":"2024-04-29T14:51:58.472489+0000","last_unstale":"2024-04-29T14:52:14.406672+0000","last_undegraded":"2024-04-29T14:52:14.406672+0000","last_fullsized":"2024-04-29T14:52:14.406672+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:20.436185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32964608,"num_objects":503,"num_object_clones":0,"num_object_copies":1006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":503,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2031,"reported_epoch":1017,"state":"active+clean","last_fresh":"2024-04-29T14:52:15.670392+0000","last_change":"2024-04-29T14:51:31.274002+0000","last_active":"2024-04-29T14:52:15.670392+0000","last_peered":"2024-04-29T14:52:15.670392+0000","last_clean":"2024-04-29T14:52:15.670392+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:52:15.670392+0000","last_undegraded":"2024-04-29T14:52:15.670392+0000","last_fullsized":"2024-04-29T14:52:15.670392+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:51:31.273868+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:52:59.621883+0000","scrub_duration":260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":475922432,"num_objects":7262,"num_object_clones":0,"num_object_copies":14524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7262,"num_write_kb":464768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1156907008,"data_stored":1156907008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7262,"ondisk_log_size":7262,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122592,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384232,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1425868,"kb_used_data":550412,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92319284,"statfs":{"total":95995035648,"available":94534946816,"internally_reserved":0,"allocated":563621888,"data_stored":562047483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":896268736},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":72,"seq":309237645867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024660,"kb_used_data":209100,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92720492,"statfs":{"total":95995035648,"available":94945783808,"internally_reserved":0,"allocated":214118400,"data_stored":212544015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423172,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284684,"kb_used_data":143692,"kb_used_omap":4,"kb_used_meta":140795,"kb_avail":93460468,"statfs":{"total":95995035648,"available":95703519232,"internally_reserved":0,"allocated":147140608,"data_stored":145566223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":144174784},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234548,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1026348,"kb_used_data":124524,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92718804,"statfs":{"total":95995035648,"available":94944055296,"internally_reserved":0,"allocated":127512576,"data_stored":126021437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":923269578},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991444,"kb_used_data":3540,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92753708,"statfs":{"total":95995035648,"available":94979796992,"internally_reserved":0,"allocated":3624960,"data_stored":2047023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575428,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508140,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93237012,"statfs":{"total":95995035648,"available":95474700288,"internally_reserved":0,"allocated":3264512,"data_stored":1604454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738967,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053716,"kb_used_data":115980,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92691436,"statfs":{"total":95995035648,"available":94916030464,"internally_reserved":0,"allocated":118763520,"data_stored":117189135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497400,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182986,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596484,"kb_used_data":9612,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93148668,"statfs":{"total":95995035648,"available":95384236032,"internally_reserved":0,"allocated":9842688,"data_stored":8268303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":600570312},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986751,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093972,"kb_used_data":110412,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":92651180,"statfs":{"total":95995035648,"available":94874808320,"internally_reserved":0,"allocated":113061888,"data_stored":111487503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1007155653},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653312,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351116,"kb_used_data":112716,"kb_used_omap":651,"kb_used_meta":237556,"kb_avail":93394036,"statfs":{"total":95995035648,"available":95635492864,"internally_reserved":0,"allocated":115421184,"data_stored":113846799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":243258120},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135593984,"data_stored":135593984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47382528,"data_stored":47382528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":125894656,"data_stored":125894656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":139198464,"data_stored":139198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":143392768,"data_stored":143392768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":163119104,"data_stored":163119104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":227868672,"data_stored":227868672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":174456832,"data_stored":174456832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:52:21.466 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:52:21.471 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:21.476 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:21.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:21.565 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:21.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:21.594 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 16 37155 37139 257.844 213.062 0.00434069 0.00387645 2024-04-29T14:52:21.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:21.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:21 smithi040 ceph-mon[39581]: pgmap v2564: 17 pgs: 17 active+clean; 1.4 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-29T14:52:21.727 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:21.756 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:21.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:21 smithi194 ceph-mon[44089]: pgmap v2564: 17 pgs: 17 active+clean; 1.4 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-29T14:52:21.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:21.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:21 smithi121 ceph-mon[44533]: pgmap v2564: 17 pgs: 17 active+clean; 1.4 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-29T14:52:21.898 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:21.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:22.029 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:22.074 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:22.101 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:22.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:22.205 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:22.245 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:22.269 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:22.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:22.374 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:22.416 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:22.443 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:22.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:22.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:22.594 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:22.594 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 16 40309 40293 251.759 197.125 0.00308719 0.00396911 2024-04-29T14:52:22.621 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:22.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:22.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:22.767 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:23.594 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 43440 43424 246.661 195.688 0.00295832 0.00405208 2024-04-29T14:52:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:23 smithi194 ceph-mon[44089]: pgmap v2565: 17 pgs: 17 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 200 MiB/s wr, 3.20k op/s 2024-04-29T14:52:23.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:23 smithi040 ceph-mon[39581]: pgmap v2565: 17 pgs: 17 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 200 MiB/s wr, 3.20k op/s 2024-04-29T14:52:23.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:23 smithi121 ceph-mon[44533]: pgmap v2565: 17 pgs: 17 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 200 MiB/s wr, 3.20k op/s 2024-04-29T14:52:24.023 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:24.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:24.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:24.171 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:24.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:24.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:24.335 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:24.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:24.471 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:24.513 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:24.543 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:24.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:24.596 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 46374 46358 241.353 183.375 0.00792038 0.00414148 2024-04-29T14:52:24.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:24.691 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:25.315 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:25.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 15 49237 49222 236.557 179 0.0102643 0.0042258 2024-04-29T14:52:25.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:25 smithi194 ceph-mon[44089]: pgmap v2566: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 217 MiB/s wr, 3.46k op/s 2024-04-29T14:52:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:25 smithi040 ceph-mon[39581]: pgmap v2566: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 217 MiB/s wr, 3.46k op/s 2024-04-29T14:52:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:25 smithi121 ceph-mon[44533]: pgmap v2566: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 217 MiB/s wr, 3.46k op/s 2024-04-29T14:52:26.596 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 16 51694 51678 230.625 153.5 0.00427238 0.00433437 2024-04-29T14:52:27.154 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:27.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:27.215 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:52:27.245 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:52:27.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:27.288 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:52:27.324 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:27.331 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:52:27.371 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:52:27.406 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:52:27.451 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:52:27.485 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:52:27.581 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:52:27.581 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:52:27.581 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:52:27.582 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:52:27.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:52:27.585 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:52:27.585 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:52:27.585 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:52:27.585 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:52:27.596 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 16 53951 53935 224.655 141.062 0.00267536 0.00444456 2024-04-29T14:52:27.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:27 smithi040 ceph-mon[39581]: pgmap v2567: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-04-29T14:52:27.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:27 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:52:27.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2255954469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:27 smithi194 ceph-mon[44089]: pgmap v2567: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-04-29T14:52:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:27 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:52:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2255954469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:27 smithi121 ceph-mon[44533]: pgmap v2567: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-04-29T14:52:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:27 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:52:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2255954469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:28.525 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:52:28.596 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 16 55923 55907 218.318 123.25 0.00230879 0.00455653 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:52:28.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:52:28.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1503126233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:52:28.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1503126233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:52:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1503126233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:28.930 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:28.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:29.035 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:29.062 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:29.080 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:29.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:29.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:29.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:29.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:29.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:29.386 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:29.418 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:29.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:29.522 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:29.566 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:29.589 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:29.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:29.596 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 57812 57796 212.421 118.062 0.00635449 0.00470634 2024-04-29T14:52:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:29 smithi040 ceph-mon[39581]: pgmap v2568: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 210 MiB/s wr, 3.37k op/s 2024-04-29T14:52:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:29 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:52:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:29 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:52:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1503126233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:52:29.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:29 smithi040 ceph-mon[39581]: osdmap e1018: 12 total, 12 up, 10 in 2024-04-29T14:52:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:29 smithi121 ceph-mon[44533]: pgmap v2568: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 210 MiB/s wr, 3.37k op/s 2024-04-29T14:52:29.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:29 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:52:29.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:29 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:52:29.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1503126233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:52:29.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:29 smithi121 ceph-mon[44533]: osdmap e1018: 12 total, 12 up, 10 in 2024-04-29T14:52:29.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:29.734 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:29 smithi194 ceph-mon[44089]: pgmap v2568: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 210 MiB/s wr, 3.37k op/s 2024-04-29T14:52:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:29 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:52:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:29 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:52:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1503126233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:52:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:29 smithi194 ceph-mon[44089]: osdmap e1018: 12 total, 12 up, 10 in 2024-04-29T14:52:30.167 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:30.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:30.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:30.273 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:30.316 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:30.345 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:30.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:30.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:30.489 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:30.532 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:30.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:30.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:30 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:52:30.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:30 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:52:30.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 60348 60332 209.425 158.5 0.00568242 0.00477363 2024-04-29T14:52:30.641 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:30.675 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:30.704 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:30.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:30.810 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:30.857 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:52:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28144.0:60896 8.0 8:0002ea02:::benchmark_data_smithi194_186548_object60895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:52:30.813764+0000", 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.040023019999999999, 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040050819000000001, 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28144", 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 60896 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.813764+0000", 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.813764+0000", 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.813766+0000", 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1100000000000001e-06 2024-04-29T14:52:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.813823+0000", 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7123000000000002e-05 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.813836+0000", 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2699e-05 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.813843+0000", 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.813887+0000", 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4020000000000002e-05 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.814020+0000", 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013285200000000001 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.814153+0000", 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000132569 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816203+0000", 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020496770000000002 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28144.0:60900 8.0 8:02113e1e:::benchmark_data_smithi194_186548_object60899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:52:30.816656+0000", 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.037130901000000001, 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.03721667, 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28144", 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 60900 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816656+0000", 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816656+0000", 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816658+0000", 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.902e-06 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816747+0000", 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9211999999999999e-05 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816760+0000", 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2687999999999999e-05 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816768+0000", 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0129999999999993e-06 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816824+0000", 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6283999999999998e-05 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816954+0000", 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013028000000000001 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:52:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.817101+0000", 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000147069 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.822501+0000", 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054001250000000004 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28144.0:60901 8.0 8:068bfb54:::benchmark_data_smithi194_186548_object60900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:52:30.816785+0000", 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.037001922, 2024-04-29T14:52:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037139085000000002, 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28144", 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 60901 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816785+0000", 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:52:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816785+0000", 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816786+0000", 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1900000000000001e-07 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816823+0000", 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6891999999999998e-05 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816830+0000", 2024-04-29T14:52:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.968e-06 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.816836+0000", 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0000000000000002e-06 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818705+0000", 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018689830000000001 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818803+0000", 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8262000000000002e-05 2024-04-29T14:52:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818905+0000", 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010192300000000001 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.822510+0000", 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036048849999999999 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28144.0:60906 8.0 8:04cdecda:::benchmark_data_smithi194_186548_object60905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:52:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:52:30.818805+0000", 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.034981902000000002, 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035168924999999997, 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28144", 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 60906 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818805+0000", 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818805+0000", 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818806+0000", 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5910000000000001e-06 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818860+0000", 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3977000000000002e-05 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818872+0000", 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1292e-05 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.818879+0000", 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7009999999999996e-06 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.819958+0000", 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010780570000000001 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.820055+0000", 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7665999999999996e-05 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.820142+0000", 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6509999999999997e-05 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:52:30.823540+0000", 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0033983339999999998 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:52:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:52:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 4 2024-04-29T14:52:30.875 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:52:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:30.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:30 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:52:30.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:30 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:52:30.876 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:30.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:30.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:30 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:52:30.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:30 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:52:30.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:52:31.020 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:31.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:52:31.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:31.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:52:31.149 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:31.188 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:31.214 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:31.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:31.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:31.355 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:31.514 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:31.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:31.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:31 smithi194 ceph-mon[44089]: pgmap v2570: 17 pgs: 17 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 194 MiB/s wr, 3.10k op/s 2024-04-29T14:52:31.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 62671 62655 206.044 145.188 0.00682644 0.00485116 2024-04-29T14:52:31.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:31.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:31 smithi040 ceph-mon[39581]: pgmap v2570: 17 pgs: 17 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 194 MiB/s wr, 3.10k op/s 2024-04-29T14:52:31.653 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:31.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:52:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:31.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:31 smithi121 ceph-mon[44533]: pgmap v2570: 17 pgs: 17 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 194 MiB/s wr, 3.10k op/s 2024-04-29T14:52:31.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:31.983 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:31.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:32.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:32.126 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:32.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:32.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:32.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:32.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:52:32.594295 min lat: 0.00105393 max lat: 0.239967 avg lat: 0.00491908 2024-04-29T14:52:32.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:52:32.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 16 65068 65052 203.232 149.812 0.00338116 0.00491908 2024-04-29T14:52:33.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:52:33.252 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:33.252 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:33.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:33.358 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:33.398 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:33.426 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:33.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:33.433 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:33 smithi040 ceph-mon[39581]: pgmap v2571: 17 pgs: 17 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-04-29T14:52:33.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:33.571 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:33.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 67209 67193 199.927 133.812 0.00580522 0.00500008 2024-04-29T14:52:33.601 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:33.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:33.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:33.743 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:33 smithi121 ceph-mon[44533]: pgmap v2571: 17 pgs: 17 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-04-29T14:52:33.744 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:33.773 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:33.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:33.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:33 smithi194 ceph-mon[44089]: pgmap v2571: 17 pgs: 17 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 170 MiB/s wr, 2.72k op/s 2024-04-29T14:52:33.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:52:33.877 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:33.919 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:33.943 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:33.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:34.050 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:34.095 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:34.125 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:34.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:34.231 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:34.270 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:34.300 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:34.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:34.404 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:34.441 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:34.470 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:34.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:34.488 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:34.575 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:34.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 16 69375 69359 196.989 135.375 0.0128208 0.00507506 2024-04-29T14:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:35.597 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 71528 71512 194.275 134.562 0.00283613 0.00512184 2024-04-29T14:52:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:35 smithi040 ceph-mon[39581]: pgmap v2572: 17 pgs: 17 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-29T14:52:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:35 smithi194 ceph-mon[44089]: pgmap v2572: 17 pgs: 17 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-29T14:52:35.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:35 smithi121 ceph-mon[44533]: pgmap v2572: 17 pgs: 17 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-29T14:52:36.587 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:36.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 15 73703 73688 191.838 136 0.00385891 0.00521096 2024-04-29T14:52:37.446 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:52:37.472 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:37.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:37.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:37.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:37 smithi040 ceph-mon[39581]: pgmap v2573: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-29T14:52:37.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/466077775' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:52:37.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:37 smithi040 ceph-mon[39581]: from='client.37913 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:52:37.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:37.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 16 75874 75858 189.59 135.625 0.00827225 0.00527291 2024-04-29T14:52:37.623 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:37.705 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:37.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:37.811 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:37 smithi194 ceph-mon[44089]: pgmap v2573: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-29T14:52:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/466077775' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:52:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:37 smithi194 ceph-mon[44089]: from='client.37913 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:52:37.848 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:37.877 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:37.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:37 smithi121 ceph-mon[44533]: pgmap v2573: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-29T14:52:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/466077775' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:52:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:37 smithi121 ceph-mon[44533]: from='client.37913 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:52:37.984 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:52:38.026 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:38.053 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:52:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:38.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:52:38.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:38.204 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:38.229 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:38.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:38.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:38.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 77982 77966 187.365 131.75 0.00779544 0.00533584 2024-04-29T14:52:38.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:38 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:38.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:38 smithi040 ceph-mon[39581]: from='client.37913 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:52:38.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:38 smithi040 ceph-mon[39581]: osdmap e1019: 12 total, 12 up, 10 in 2024-04-29T14:52:38.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:38 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:38.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:38 smithi194 ceph-mon[44089]: from='client.37913 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:52:38.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:38 smithi194 ceph-mon[44089]: osdmap e1019: 12 total, 12 up, 10 in 2024-04-29T14:52:38.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:38 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:38.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:38 smithi121 ceph-mon[44533]: from='client.37913 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:52:38.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:38 smithi121 ceph-mon[44533]: osdmap e1019: 12 total, 12 up, 10 in 2024-04-29T14:52:39.232 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:39.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:39.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:39.374 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:39.409 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:39.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:39.513 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:39.540 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:39.544 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:52:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:52:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:52:39.545 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:52:39.545 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:52:39.558 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:39.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:39.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 16 79809 79793 184.655 114.188 0.0165498 0.00541405 2024-04-29T14:52:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:39 smithi040 ceph-mon[39581]: pgmap v2575: 17 pgs: 17 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-04-29T14:52:39.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:39.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:39 smithi194 ceph-mon[44089]: pgmap v2575: 17 pgs: 17 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-04-29T14:52:39.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:39 smithi121 ceph-mon[44533]: pgmap v2575: 17 pgs: 17 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-04-29T14:52:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:39.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:40.059 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:40.101 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:40.126 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:40.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:40.235 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:40.273 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:40.306 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:40.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:40.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:40.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 16 81847 81831 182.609 127.375 0.00351532 0.005475 2024-04-29T14:52:40.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:52:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:40.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:40.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:52:40.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:40.948 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:40.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:40.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:41.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:41.133 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:41.166 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:41.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:41.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:41.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:41.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:41.467 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-29T14:52:41.467 INFO:tasks.thrashosds.thrasher:Killing osd 9, live_osds are [0, 1, 3, 4, 6, 7, 9, 10, 11, 2, 8, 5] 2024-04-29T14:52:41.467 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-29T14:52:41.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.9 2024-04-29T14:52:41.481 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:41.528 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:41.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:41.581 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:41.581 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:41.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 83734 83718 180.379 117.938 0.0151886 0.00554153 2024-04-29T14:52:41.632 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:41.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:41 smithi121 ceph-mon[44533]: pgmap v2576: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-29T14:52:41.663 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:52:41.690 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:52:41.691 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:41.714 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:52:41.740 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:52:41.774 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:52:41.806 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:52:41.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:52:41 smithi194 systemd[1]: Stopping Ceph osd.9 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:52:41.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:41 smithi194 ceph-mon[44089]: pgmap v2576: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-29T14:52:41.832 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:41.833 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:52:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:41.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:41.865 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:52:41.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:41 smithi040 ceph-mon[39581]: pgmap v2576: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-29T14:52:41.897 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:52:41.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:52:41.939 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:41.949 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:52:41.975 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:52:41.979 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:41.979 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:52:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:52:42.010 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:42.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:42.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:42.155 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:42.181 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:42.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:42.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:42.285 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:42.322 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:52:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[57740]: 2024-04-29T14:52:41.836+0000 7f2d44806700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T14:52:42.322 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:52:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[57740]: 2024-04-29T14:52:41.836+0000 7f2d44806700 -1 osd.9 1019 *** Got signal Terminated *** 2024-04-29T14:52:42.322 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:52:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[57740]: 2024-04-29T14:52:41.836+0000 7f2d44806700 -1 osd.9 1019 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T14:52:42.325 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:42.365 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:42.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:42.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3150133416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd.9 marked itself down and dead 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.481 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.482 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:42 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.506 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3150133416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd.9 marked itself down and dead 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:42 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 16 85380 85364 177.796 102.875 0.00319236 0.00560365 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3150133416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd.9 marked itself down and dead 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:42 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:43.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 16 85380 85364 172.061 0 - 0.00560365 2024-04-29T14:52:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:43 smithi194 ceph-mon[44089]: pgmap v2577: 17 pgs: 17 active+clean; 4.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-29T14:52:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:43 smithi194 ceph-mon[44089]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:52:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:43 smithi194 ceph-mon[44089]: osdmap e1020: 12 total, 11 up, 10 in 2024-04-29T14:52:43.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:52:43 smithi194 bash[187620]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9 2024-04-29T14:52:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:43 smithi040 ceph-mon[39581]: pgmap v2577: 17 pgs: 17 active+clean; 4.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-29T14:52:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:43 smithi040 ceph-mon[39581]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:52:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:43 smithi040 ceph-mon[39581]: osdmap e1020: 12 total, 11 up, 10 in 2024-04-29T14:52:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:43 smithi121 ceph-mon[44533]: pgmap v2577: 17 pgs: 17 active+clean; 4.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-29T14:52:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:43 smithi121 ceph-mon[44533]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:52:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:43 smithi121 ceph-mon[44533]: osdmap e1020: 12 total, 11 up, 10 in 2024-04-29T14:52:44.600 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 85380 85364 166.685 0 - 0.00560365 2024-04-29T14:52:44.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:44 smithi040 ceph-mon[39581]: osdmap e1021: 12 total, 11 up, 10 in 2024-04-29T14:52:44.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:44 smithi194 ceph-mon[44089]: osdmap e1021: 12 total, 11 up, 10 in 2024-04-29T14:52:44.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:44 smithi121 ceph-mon[44533]: osdmap e1021: 12 total, 11 up, 10 in 2024-04-29T14:52:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:44.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:45.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:45.094 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:45.599 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 85380 85364 161.635 0 - 0.00560365 2024-04-29T14:52:45.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:45 smithi040 ceph-mon[39581]: pgmap v2580: 17 pgs: 1 undersized+degraded+peered, 1 stale+active+clean, 15 active+clean; 5.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s; 5452/165336 objects degraded (3.298%) 2024-04-29T14:52:45.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:45 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 5452/165336 objects degraded (3.298%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:52:45.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:52:45.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:52:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:45 smithi194 ceph-mon[44089]: pgmap v2580: 17 pgs: 1 undersized+degraded+peered, 1 stale+active+clean, 15 active+clean; 5.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s; 5452/165336 objects degraded (3.298%) 2024-04-29T14:52:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:45 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 5452/165336 objects degraded (3.298%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:52:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:52:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:52:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:45 smithi121 ceph-mon[44533]: pgmap v2580: 17 pgs: 1 undersized+degraded+peered, 1 stale+active+clean, 15 active+clean; 5.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 110 MiB/s wr, 1.76k op/s; 5452/165336 objects degraded (3.298%) 2024-04-29T14:52:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:45 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 5452/165336 objects degraded (3.298%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:52:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:52:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:52:45.929 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.9.service' 2024-04-29T14:52:45.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T14:52:45.982 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-29T14:52:45.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd down 9 2024-04-29T14:52:46.600 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 16 85380 85364 156.881 0 - 0.00560365 2024-04-29T14:52:46.646 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:46.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:46.750 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:46.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:52:46.780 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:46 smithi194 ceph-mon[44089]: osdmap e1022: 12 total, 11 up, 10 in 2024-04-29T14:52:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:46.790 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:46 smithi040 ceph-mon[39581]: osdmap e1022: 12 total, 11 up, 10 in 2024-04-29T14:52:46.810 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:46.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:46 smithi121 ceph-mon[44533]: osdmap e1022: 12 total, 11 up, 10 in 2024-04-29T14:52:46.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:46.942 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:46.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:47.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:47.116 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:47.267 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:52:47.267 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:52:47.465 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:47.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:47.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:47.600 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 85380 85364 152.399 0 - 0.00560365 2024-04-29T14:52:47.602 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:47.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:47.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:47.728 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:47.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:47 smithi194 ceph-mon[44089]: pgmap v2582: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 95 MiB/s wr, 1.53k op/s; 16137/169912 objects degraded (9.497%) 2024-04-29T14:52:47.761 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:47.782 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:47.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:47.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:47 smithi040 ceph-mon[39581]: pgmap v2582: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 95 MiB/s wr, 1.53k op/s; 16137/169912 objects degraded (9.497%) 2024-04-29T14:52:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:47 smithi121 ceph-mon[44533]: pgmap v2582: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 95 MiB/s wr, 1.53k op/s; 16137/169912 objects degraded (9.497%) 2024-04-29T14:52:47.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:47.916 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:47.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:47.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:47.986 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:48.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:52:48.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:48.069 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:48.218 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:48.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:48.287 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:52:48.322 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:48.354 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:48.375 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:48.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:48.479 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:48.510 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:48.530 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:48.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:48.600 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 85380 85364 148.166 0 - 0.00560365 2024-04-29T14:52:48.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:48.717 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:48.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:48.750 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:48 smithi121 ceph-mon[44533]: pgmap v2583: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 47 MiB/s wr, 748 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:48.822 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:48.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:48 smithi040 ceph-mon[39581]: pgmap v2583: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 47 MiB/s wr, 748 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:48.851 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:48.904 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:48.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:49.008 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:49.038 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:48 smithi194 ceph-mon[44089]: pgmap v2583: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 47 MiB/s wr, 748 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:49.040 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:49.062 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:49.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:49.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:49.511 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:49.600 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 85380 85364 144.162 0 - 0.00560365 2024-04-29T14:52:49.658 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:52:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/583987748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:52:49.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/583987748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:52:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/583987748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:52:50.120 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:52:50.120 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:52:50.243 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:50.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:50.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:50.377 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:50.501 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:50.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:50.601 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 85380 85364 140.369 0 - 0.00560365 2024-04-29T14:52:50.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:50.634 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:50.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:50.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:52:50.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:52:50.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:50 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:50.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/583987748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:52:50.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:50 smithi121 ceph-mon[44533]: osdmap e1023: 12 total, 11 up, 10 in 2024-04-29T14:52:50.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:50 smithi121 ceph-mon[44533]: pgmap v2585: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 41 MiB/s wr, 654 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:50.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:52:50.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:52:50.791 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:50.797 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:52:50.818 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:52:50.838 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:52:50.857 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:52:50.877 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:52:50.900 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:52:50.920 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:52:50.940 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:52:50.961 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:52:50.982 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:52:50.990 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:52:50.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:52:50.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:52:50.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:50 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:50.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/583987748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:52:50.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:50 smithi040 ceph-mon[39581]: osdmap e1023: 12 total, 11 up, 10 in 2024-04-29T14:52:50.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:50 smithi040 ceph-mon[39581]: pgmap v2585: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 41 MiB/s wr, 654 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:50.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:52:51.006 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:52:51.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:52:51.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:52:51.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:52:51.006 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:52:51.007 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:52:51.038 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:52:51.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:52:51.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:52:51.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:50 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:52:51.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/583987748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:52:51.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:50 smithi194 ceph-mon[44089]: osdmap e1023: 12 total, 11 up, 10 in 2024-04-29T14:52:51.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:50 smithi194 ceph-mon[44089]: pgmap v2585: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 41 MiB/s wr, 654 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:51.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:51.122 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:51.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:51.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:51.380 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:51.601 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 85380 85364 136.768 0 - 0.00560365 2024-04-29T14:52:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2599649975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:51.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2599649975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2599649975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:52:52.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:52:52.229 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:52.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:52.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:52:52.362 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:52.510 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:52:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:52.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:52:52.601 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:52:52.598387 min lat: 0.00105393 max lat: 0.239967 avg lat: 0.00560365 2024-04-29T14:52:52.601 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:52:52.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 85380 85364 133.35 0 - 0.00560365 2024-04-29T14:52:52.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:52.642 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:52.777 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:52:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:52:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:52:52.778 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:52:52.778 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:52:52.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:52.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:52.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:52 smithi121 ceph-mon[44533]: pgmap v2586: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21 MiB/s wr, 337 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:52.793 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:52 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 21522/170740 objects degraded (12.605%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:52:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:52.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:52.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:52:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:53.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:52:53.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:53.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:52:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:52 smithi194 ceph-mon[44089]: pgmap v2586: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21 MiB/s wr, 337 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:52 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 21522/170740 objects degraded (12.605%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:52:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:52 smithi040 ceph-mon[39581]: pgmap v2586: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21 MiB/s wr, 337 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:52 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 21522/170740 objects degraded (12.605%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:52:53.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:53.159 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:53.180 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:53.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:53.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:52:53.311 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:53.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:52:53.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:52:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:53.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:52:53.601 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 85380 85364 130.097 0 - 0.00560365 2024-04-29T14:52:53.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:52:53.689 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:53.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:52:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:53.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:52:53.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:53.825 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:52:53.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:52:53.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:52:53.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:53 smithi040 ceph-mon[39581]: osdmap e1024: 12 total, 11 up, 10 in 2024-04-29T14:52:53.843 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:53.871 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:53.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:52:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:52:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:53 smithi121 ceph-mon[44533]: osdmap e1024: 12 total, 11 up, 10 in 2024-04-29T14:52:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9 is already down. 2024-04-29T14:52:53.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:52:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:54.025 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:52:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:54.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:52:54.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:52:54.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:52:54.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:53 smithi194 ceph-mon[44089]: osdmap e1024: 12 total, 11 up, 10 in 2024-04-29T14:52:54.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:53 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:52:53.965+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1024 definitely_dead 0 2024-04-29T14:52:54.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:54.158 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:54.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:54.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:54.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:52:54.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:52:54.307 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:54.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 16 85380 85364 127 0 - 0.00560365 2024-04-29T14:52:54.793 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [0, 4, 7, 9, 10, 6, 1, 5, 8, 3] 2024-04-29T14:52:54.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 9 2024-04-29T14:52:54.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:52:54.829 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:55.000 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4134335326' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["9"]}]: dispatch 2024-04-29T14:52:55.000 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:54 smithi121 ceph-mon[44533]: pgmap v2588: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 3.2 MiB/s wr, 51 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:55.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:52:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:55.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:52:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4134335326' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["9"]}]: dispatch 2024-04-29T14:52:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:54 smithi194 ceph-mon[44089]: pgmap v2588: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 3.2 MiB/s wr, 51 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:55.089 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4134335326' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["9"]}]: dispatch 2024-04-29T14:52:55.089 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:54 smithi040 ceph-mon[39581]: pgmap v2588: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 3.2 MiB/s wr, 51 op/s; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:55.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:55.192 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:55.213 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:55.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:55.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:55.348 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:55.567 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:52:55.567 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:52:55.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:52:55.567 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:52:55.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.567 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 845/839 [PushOp(7:8290c88e:::benchmark_data_smithi194_147200_object65392:head, version: 667'3998, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8290c88e:::benchmark_data_smithi194_147200_object65392:head@667'3998, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:04.851897+0000", 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 470.70892469500001, 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7809149309999999, 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851897+0000", 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851897+0000", 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851899+0000", 2024-04-29T14:52:55.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.08e-06 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851927+0000", 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8034999999999998e-05 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851944+0000", 2024-04-29T14:52:55.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6884e-05 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:04.851959+0000", 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4819e-05 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:08.598632+0000", 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7466729330000001 2024-04-29T14:52:55.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:08.599865+0000", 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012335250000000001 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:08.632812+0000", 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032946654999999998 2024-04-29T14:52:55.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 846/839 [PushReplyOp(7:8724884e:::benchmark_data_smithi194_147200_object65431:head)])", 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:14.097209+0000", 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 461.46361331499998, 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0735914299999996, 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097209+0000", 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097209+0000", 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097211+0000", 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1919999999999999e-06 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097218+0000", 2024-04-29T14:52:55.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4479999999999997e-06 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097229+0000", 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1282e-05 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.097238+0000", 2024-04-29T14:52:55.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.636e-06 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.144192+0000", 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0469534520000003 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.170800+0000", 2024-04-29T14:52:55.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026608420000000001 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 846/839 [PushOp(7:815c4b83:::benchmark_data_smithi194_147200_object65435:head, version: 667'4004, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:815c4b83:::benchmark_data_smithi194_147200_object65435:head@667'4004, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:14.277472+0000", 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 461.28335032000001, 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2569320919999996, 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277472+0000", 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277472+0000", 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277474+0000", 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.979e-06 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277606+0000", 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013254900000000001 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277627+0000", 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0869999999999998e-05 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:14.277646+0000", 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8822e-05 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.507156+0000", 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2295102309999999 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.508162+0000", 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001005932 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.534404+0000", 2024-04-29T14:52:55.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026241708999999998 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 846/839 [PushOp(7:85b7373c:::benchmark_data_smithi194_147200_object65464:head, version: 667'4006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:85b7373c:::benchmark_data_smithi194_147200_object65464:head@667'4006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:16.510207+0000", 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 459.050614532, 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.184098536, 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.583 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510207+0000", 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510207+0000", 2024-04-29T14:52:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510209+0000", 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.564e-06 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510236+0000", 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6840000000000001e-05 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510253+0000", 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7036000000000001e-05 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:16.510266+0000", 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2848e-05 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.644482+0000", 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1342165949999998 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.645497+0000", 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010150859999999999 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:19.694306+0000", 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.048808566999999997 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 847/839 [PushOp(7:803426e6:::benchmark_data_smithi194_147200_object65471:head, version: 667'4008, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:803426e6:::benchmark_data_smithi194_147200_object65471:head@667'4008, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:17.823241+0000", 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 457.73758045099999, 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2738754650000002, 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.588 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823241+0000", 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823241+0000", 2024-04-29T14:52:55.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823243+0000", 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.95e-06 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823278+0000", 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4175999999999999e-05 2024-04-29T14:52:55.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823298+0000", 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0913000000000001e-05 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.823318+0000", 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9752e-05 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.072385+0000", 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.249067041 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.073175+0000", 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078929100000000002 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.097117+0000", 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023942341999999998 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 847/839 [PushOp(7:8f822c3f:::benchmark_data_smithi194_147200_object65480:head, version: 667'4009, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8f822c3f:::benchmark_data_smithi194_147200_object65480:head@667'4009, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:17.996936+0000", 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 457.56388602800001, 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0732292760000002, 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.593 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996936+0000", 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996936+0000", 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996938+0000", 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8700000000000001e-06 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996966+0000", 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8141999999999999e-05 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996984+0000", 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7659000000000002e-05 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:17.996997+0000", 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3732e-05 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.046353+0000", 2024-04-29T14:52:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0493562150000004 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.047002+0000", 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064897399999999997 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.070165+0000", 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023162683999999999 2024-04-29T14:52:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 847/839 [PushOp(7:8be67170:::benchmark_data_smithi194_147200_object65493:head, version: 667'4010, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8be67170:::benchmark_data_smithi194_147200_object65493:head@667'4010, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:18.099004+0000", 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 457.461818166, 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4893369410000004, 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.598 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099004+0000", 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099004+0000", 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099005+0000", 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5939999999999999e-06 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099033+0000", 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7946999999999999e-05 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099050+0000", 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6524e-05 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:18.099062+0000", 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2170999999999999e-05 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.566261+0000", 2024-04-29T14:52:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4671990209999999 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.566909+0000", 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064791899999999995 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:22.588341+0000", 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021431764999999998 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 848/839 [PushOp(7:81c58ee4:::benchmark_data_smithi194_147200_object65517:head, version: 667'4012, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:81c58ee4:::benchmark_data_smithi194_147200_object65517:head@667'4012, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:20.562673+0000", 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 454.998148496, 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0525536600000001, 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562673+0000", 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562673+0000", 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562674+0000", 2024-04-29T14:52:55.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.08e-06 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562735+0000", 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0764999999999998e-05 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562752+0000", 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6895e-05 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:20.562767+0000", 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4763000000000001e-05 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.574454+0000", 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0116869529999999 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.575114+0000", 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065992799999999995 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:23.615227+0000", 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.040113276000000003 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 849/839 [PushReplyOp(7:8a2e7b36:::benchmark_data_smithi194_147200_object65508:head)])", 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:45:21.946060+0000", 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 453.61476189199999, 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3263969549999999, 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946060+0000", 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946060+0000", 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946063+0000", 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5550000000000001e-06 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946070+0000", 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7889999999999995e-06 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946082+0000", 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1673e-05 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:21.946092+0000", 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0397999999999999e-05 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:25.246776+0000", 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3006832780000002 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:45:25.272457+0000", 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025681262 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 897/887 [PushOp(7:83cce57a:::benchmark_data_smithi194_147200_object74932:head, version: 670'4606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:83cce57a:::benchmark_data_smithi194_147200_object74932:head@670'4606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:09.877870+0000", 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 345.682952016, 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.833420362, 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877870+0000", 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877870+0000", 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877872+0000", 2024-04-29T14:52:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.826e-06 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877903+0000", 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1426e-05 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877922+0000", 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8706999999999999e-05 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:09.877940+0000", 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7745999999999999e-05 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:13.690096+0000", 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.812156436 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:13.690656+0000", 2024-04-29T14:52:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056038400000000003 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:13.711290+0000", 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020633836999999999 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 897/887 [PushOp(7:80dd5181:::benchmark_data_smithi194_147200_object74941:head, version: 670'4607, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:80dd5181:::benchmark_data_smithi194_147200_object74941:head@670'4607, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:10.642330+0000", 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 344.91849204300001, 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7214084810000001, 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642330+0000", 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642330+0000", 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642332+0000", 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8729999999999999e-06 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642464+0000", 2024-04-29T14:52:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013224499999999999 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642482+0000", 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7906000000000001e-05 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:10.642495+0000", 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2952999999999999e-05 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:16.344214+0000", 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7017192489999999 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:16.344718+0000", 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050386199999999999 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:16.363738+0000", 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019020393 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 903/887 [PushOp(7:803b1c22:::benchmark_data_smithi194_147200_object75163:head, version: 670'4624, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:803b1c22:::benchmark_data_smithi194_147200_object75163:head@670'4624, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:33.530104+0000", 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.03071833000001, 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0443010020000001, 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530104+0000", 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530104+0000", 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530105+0000", 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.088e-06 2024-04-29T14:52:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530149+0000", 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4774999999999997e-05 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530162+0000", 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3055e-05 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.530180+0000", 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7371999999999999e-05 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.555736+0000", 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0255566470000002 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.556553+0000", 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000816152 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.574405+0000", 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017851913000000001 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 903/887 [PushOp(7:8893102c:::benchmark_data_smithi194_147200_object75183:head, version: 670'4625, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8893102c:::benchmark_data_smithi194_147200_object75183:head@670'4625, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:33.796243+0000", 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 321.76457861599999, 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4972395779999998, 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796243+0000", 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796243+0000", 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796245+0000", 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.347e-06 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796385+0000", 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014044900000000001 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796400+0000", 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5208000000000001e-05 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:33.796411+0000", 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0491e-05 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.276226+0000", 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4798150310000002 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.276698+0000", 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047268000000000001 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.293483+0000", 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016784371999999999 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 904/887 [PushOp(7:8f1fbb9a:::benchmark_data_smithi194_147200_object75217:head, version: 670'4626, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8f1fbb9a:::benchmark_data_smithi194_147200_object75217:head@670'4626, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:35.303811+0000", 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 320.25701084500002, 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2934006399999998, 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303811+0000", 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303811+0000", 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303812+0000", 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.229e-06 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303857+0000", 2024-04-29T14:52:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5207000000000003e-05 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303871+0000", 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3895e-05 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.303882+0000", 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0249e-05 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.578667+0000", 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2747856049999999 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.579144+0000", 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047689999999999999 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.597212+0000", 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018067554999999999 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 904/887 [PushReplyOp(7:81acef38:::benchmark_data_smithi194_147200_object75157:head)])", 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:35.677217+0000", 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 319.88360488500001, 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3540995119999999, 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677217+0000", 2024-04-29T14:52:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677217+0000", 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677218+0000", 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.471e-06 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677225+0000", 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1970000000000001e-06 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677234+0000", 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7680000000000003e-06 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:35.677241+0000", 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4849999999999997e-06 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:39.011718+0000", 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3344772320000002 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:39.031317+0000", 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019598358999999999 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 905/887 [PushOp(7:88af4bef:::benchmark_data_smithi194_147200_object75234:head, version: 670'4627, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:88af4bef:::benchmark_data_smithi194_147200_object75234:head@670'4627, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.178519+0000", 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 318.38230294499999, 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8373110759999998, 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178519+0000", 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178519+0000", 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178520+0000", 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.497e-06 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178570+0000", 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9801e-05 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178586+0000", 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6022e-05 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.178600+0000", 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3458000000000001e-05 2024-04-29T14:52:55.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.998635+0000", 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8200351159999997 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.999118+0000", 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048342699999999998 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:42.015830+0000", 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016711754999999998 2024-04-29T14:52:55.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 906/887 [PushOp(7:885c0f18:::benchmark_data_smithi194_147200_object75244:head, version: 670'4628, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:885c0f18:::benchmark_data_smithi194_147200_object75244:head@670'4628, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.539025+0000", 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 318.02179679599999, 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5707472290000002, 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539025+0000", 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539025+0000", 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539026+0000", 2024-04-29T14:52:55.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28e-06 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539138+0000", 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011203499999999999 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539154+0000", 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.556e-05 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.539165+0000", 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1306e-05 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.092638+0000", 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.553472298 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.093242+0000", 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060396899999999999 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.109772+0000", 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016530781000000001 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(7.1 907/887 [PushReplyOp(7:8893102c:::benchmark_data_smithi194_147200_object75183:head)])", 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.943579+0000", 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 317.61724282, 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8411319420000001, 2024-04-29T14:52:55.646 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943579+0000", 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943579+0000", 2024-04-29T14:52:55.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943580+0000", 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943586+0000", 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4049999999999999e-06 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943596+0000", 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0144e-05 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.943604+0000", 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-06 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.766160+0000", 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.822556804 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.784711+0000", 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01855064 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(7.1 907/887 [PushOp(7:81e45716:::benchmark_data_smithi194_147200_object75252:head, version: 670'4629, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:81e45716:::benchmark_data_smithi194_147200_object75252:head@670'4629, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:47:37.991562+0000", 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 317.56925978800001, 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8482403650000001, 2024-04-29T14:52:55.650 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991562+0000", 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991562+0000", 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991564+0000", 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991699+0000", 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013455800000000001 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991715+0000", 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6444999999999999e-05 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:37.991733+0000", 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.823e-05 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.822135+0000", 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8304019990000002 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.822635+0000", 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050026699999999999 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:47:41.839802+0000", 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017166976 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_backoff(7.1 ack-block id 132 [7:87f01edb:::benchmark_data_smithi194_147200_object63188:head,7:87f01edb:::benchmark_data_smithi194_147200_object63188:head) e926)", 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:20.238054+0000", 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 275.322768354, 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4887491290000003, 2024-04-29T14:52:55.654 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 0 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238054+0000", 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238054+0000", 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238055+0000", 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.358e-06 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238059+0000", 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5120000000000002e-06 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238070+0000", 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0566e-05 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.238077+0000", 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5409999999999997e-06 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.323801+0000", 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085724754 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:20.323810+0000", 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5199999999999997e-06 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:21.306116+0000", 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.98230652600000001 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:21.306132+0000", 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5196e-05 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:25.726052+0000", 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4199200200000002 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:25.726803+0000", 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075113600000000001 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:52:55.659 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:52:55.660 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 16 85380 85364 124.047 0 - 0.00560365 2024-04-29T14:52:55.696 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:55.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:55.701 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:52:55.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:52:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:56.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:52:56.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 16 85380 85364 121.228 0 - 0.00560365 2024-04-29T14:52:56.614 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:52:57.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 85380 85364 118.534 0 - 0.00560365 2024-04-29T14:52:57.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:57 smithi040 ceph-mon[39581]: pgmap v2589: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:57.740 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:57 smithi121 ceph-mon[44533]: pgmap v2589: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:57 smithi194 ceph-mon[44089]: pgmap v2589: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:52:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:57.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:52:57.985 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:58.017 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:58.039 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:58.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:52:58.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:58.174 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:58.197 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:58.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:52:58.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:52:58.331 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:58.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:58 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:52:58.225+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1025 definitely_dead 0 2024-04-29T14:52:58.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 85380 85364 115.957 0 - 0.00560365 2024-04-29T14:52:58.654 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.9. 2024-04-29T14:52:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:58 smithi040 ceph-mon[39581]: osdmap e1025: 12 total, 11 up, 10 in 2024-04-29T14:52:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4075473054' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-04-29T14:52:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:58 smithi040 ceph-mon[39581]: Client client.admin marked osd.9 out, after it was down for 15 seconds 2024-04-29T14:52:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:58 smithi040 ceph-mon[39581]: pgmap v2591: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:58 smithi121 ceph-mon[44533]: osdmap e1025: 12 total, 11 up, 10 in 2024-04-29T14:52:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4075473054' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-04-29T14:52:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:58 smithi121 ceph-mon[44533]: Client client.admin marked osd.9 out, after it was down for 15 seconds 2024-04-29T14:52:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:58 smithi121 ceph-mon[44533]: pgmap v2591: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:52:58.947 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:52:58.967 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:52:58.987 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:52:59.007 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:52:59.026 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:52:59.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:52:59.047 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:52:59.066 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:52:59.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:58 smithi194 ceph-mon[44089]: osdmap e1025: 12 total, 11 up, 10 in 2024-04-29T14:52:59.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4075473054' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-04-29T14:52:59.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:58 smithi194 ceph-mon[44089]: Client client.admin marked osd.9 out, after it was down for 15 seconds 2024-04-29T14:52:59.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:58 smithi194 ceph-mon[44089]: pgmap v2591: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:52:59.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:52:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:59.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:52:59.086 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:52:59.107 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:52:59.128 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:52:59.185 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:59.217 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:59.238 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:59.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:59.342 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:52:59.343 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:52:59.374 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:59.394 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:52:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:59.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:52:59.499 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:52:59.529 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:59.549 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:52:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:59.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:52:59.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 85380 85364 113.49 0 - 0.00560365 2024-04-29T14:52:59.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:52:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:52:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:59.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4075473054' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osdmap e1026: 12 total, 11 up, 9 in 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2408953762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:52:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:52:59.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:52:59.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4075473054' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-04-29T14:52:59.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osdmap e1026: 12 total, 11 up, 9 in 2024-04-29T14:52:59.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2408953762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:52:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:52:59.912 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:52:59.956 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:52:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:52:59.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:00.065 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:00.122 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4075473054' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osdmap e1026: 12 total, 11 up, 9 in 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2408953762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:52:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.151 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:00.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:00.257 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:00.294 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:00.325 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:00.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:00.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:00.479 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:00.524 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:00.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:00.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 16 86239 86223 112.245 2.98264 0.00457239 0.00890624 2024-04-29T14:53:00.610 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.9 2024-04-29T14:53:00.610 INFO:tasks.thrashosds.thrasher:remote for osd 9 is ubuntu@smithi194.front.sepia.ceph.com 2024-04-29T14:53:00.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.9 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-9 --journal-path /var/lib/ceph/osd/ceph-9/journal --op list-pgs 2024-04-29T14:53:00.637 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:00.705 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:00.740 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:00.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:00.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:00 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:00.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:00.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:00.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:00 smithi040 ceph-mon[39581]: osdmap e1027: 12 total, 11 up, 9 in 2024-04-29T14:53:00.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:00 smithi040 ceph-mon[39581]: pgmap v2594: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:53:00.846 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:53:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:00.939 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.9/config 2024-04-29T14:53:00.993 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:53:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:01.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:53:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:00 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:00 smithi194 ceph-mon[44089]: osdmap e1027: 12 total, 11 up, 9 in 2024-04-29T14:53:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:00 smithi194 ceph-mon[44089]: pgmap v2594: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:53:01.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:01.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:01.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:00 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:01.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:01.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:00 smithi121 ceph-mon[44533]: osdmap e1027: 12 total, 11 up, 9 in 2024-04-29T14:53:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:00 smithi121 ceph-mon[44533]: pgmap v2594: 17 pgs: 4 undersized+degraded+peered, 13 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 21522/170740 objects degraded (12.605%) 2024-04-29T14:53:01.142 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:01.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 87370 87354 111.396 70.6875 0.0205849 0.00897517 2024-04-29T14:53:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:53:01.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:53:02.170 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:53:02.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4138088705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:02.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:02 smithi040 ceph-mon[39581]: from='client.37955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:02.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:02 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:02.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4138088705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:02.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:02 smithi194 ceph-mon[44089]: from='client.37955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:02.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:02 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:02.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 88476 88460 110.551 69.125 0.0328195 0.00904372 2024-04-29T14:53:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4138088705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:02 smithi121 ceph-mon[44533]: from='client.37955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:02.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:02 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:03.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:03 smithi040 ceph-mon[39581]: from='client.37955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:53:03.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:03 smithi040 ceph-mon[39581]: osdmap e1028: 12 total, 11 up, 9 in 2024-04-29T14:53:03.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:03 smithi040 ceph-mon[39581]: pgmap v2596: 17 pgs: 3 active+recovering+degraded, 1 undersized+degraded+peered, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 9.4 MiB/s wr, 150 op/s; 21421/172148 objects degraded (12.443%); 1.3 MiB/s, 21 objects/s recovering 2024-04-29T14:53:03.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:03 smithi040 ceph-mon[39581]: osdmap e1029: 12 total, 11 up, 9 in 2024-04-29T14:53:03.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:03 smithi194 ceph-mon[44089]: from='client.37955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:53:03.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:03 smithi194 ceph-mon[44089]: osdmap e1028: 12 total, 11 up, 9 in 2024-04-29T14:53:03.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:03 smithi194 ceph-mon[44089]: pgmap v2596: 17 pgs: 3 active+recovering+degraded, 1 undersized+degraded+peered, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 9.4 MiB/s wr, 150 op/s; 21421/172148 objects degraded (12.443%); 1.3 MiB/s, 21 objects/s recovering 2024-04-29T14:53:03.397 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:03 smithi194 ceph-mon[44089]: osdmap e1029: 12 total, 11 up, 9 in 2024-04-29T14:53:03.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 89562 89546 109.714 67.875 0.00529414 0.0091128 2024-04-29T14:53:03.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:03 smithi121 ceph-mon[44533]: from='client.37955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:53:03.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:03 smithi121 ceph-mon[44533]: osdmap e1028: 12 total, 11 up, 9 in 2024-04-29T14:53:03.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:03 smithi121 ceph-mon[44533]: pgmap v2596: 17 pgs: 3 active+recovering+degraded, 1 undersized+degraded+peered, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 9.4 MiB/s wr, 150 op/s; 21421/172148 objects degraded (12.443%); 1.3 MiB/s, 21 objects/s recovering 2024-04-29T14:53:03.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:03 smithi121 ceph-mon[44533]: osdmap e1029: 12 total, 11 up, 9 in 2024-04-29T14:53:03.770 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:53:04.151 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:04.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:04 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 21421/172148 objects degraded (12.443%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:04.426 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:04 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 21421/172148 objects degraded (12.443%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:04.562 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:04 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 21421/172148 objects degraded (12.443%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:04.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:04.602 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 90698 90682 108.97 71 0.00721614 0.00917524 2024-04-29T14:53:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:04.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:04.708 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:53:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:53:05.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:53:05.191 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:05 smithi121 ceph-mon[44533]: pgmap v2598: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 20 MiB/s wr, 319 op/s; 21292/174376 objects degraded (12.210%); 2.5 MiB/s, 40 objects/s recovering 2024-04-29T14:53:05.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:05 smithi194 ceph-mon[44089]: pgmap v2598: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 20 MiB/s wr, 319 op/s; 21292/174376 objects degraded (12.210%); 2.5 MiB/s, 40 objects/s recovering 2024-04-29T14:53:05.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:05 smithi040 ceph-mon[39581]: pgmap v2598: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 20 MiB/s wr, 319 op/s; 21292/174376 objects degraded (12.210%); 2.5 MiB/s, 40 objects/s recovering 2024-04-29T14:53:05.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 15 91822 91807 108.24 70.3125 0.00513929 0.00923647 2024-04-29T14:53:05.689 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:53:05.689 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:53:06.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:06 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:53:06.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:06 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:53:06.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 92793 92777 107.359 60.625 0.0195158 0.0093121 2024-04-29T14:53:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:06 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:53:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:06 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:53:06.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:06 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:53:06.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:06 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:53:07.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:53:07.201 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:53:07.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:07.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:53:07.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:07.345 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:07.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:07 smithi121 ceph-mon[44533]: pgmap v2599: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 42 MiB/s wr, 679 op/s; 20929/179818 objects degraded (11.639%); 5.5 MiB/s, 88 objects/s recovering 2024-04-29T14:53:07.489 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:53:07.496 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:07 smithi040 ceph-mon[39581]: pgmap v2599: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 42 MiB/s wr, 679 op/s; 20929/179818 objects degraded (11.639%); 5.5 MiB/s, 88 objects/s recovering 2024-04-29T14:53:07.496 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:07.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:07.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:07 smithi194 ceph-mon[44089]: pgmap v2599: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 42 MiB/s wr, 679 op/s; 20929/179818 objects degraded (11.639%); 5.5 MiB/s, 88 objects/s recovering 2024-04-29T14:53:07.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 93859 93843 106.618 66.625 0.0106894 0.00937701 2024-04-29T14:53:07.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:07.995 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:08.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:08.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:08.148 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:08.294 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:08.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:08.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:08.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:08.482 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:08.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 94892 94876 105.867 64.5625 0.00610567 0.00944339 2024-04-29T14:53:08.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:08.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:08 smithi040 ceph-mon[39581]: osdmap e1030: 12 total, 11 up, 9 in 2024-04-29T14:53:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:08 smithi040 ceph-mon[39581]: pgmap v2601: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 52 MiB/s wr, 834 op/s; 20764/182452 objects degraded (11.381%); 6.6 MiB/s, 106 objects/s recovering 2024-04-29T14:53:08.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:09.009 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:08 smithi194 ceph-mon[44089]: osdmap e1030: 12 total, 11 up, 9 in 2024-04-29T14:53:09.009 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:08 smithi194 ceph-mon[44089]: pgmap v2601: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 52 MiB/s wr, 834 op/s; 20764/182452 objects degraded (11.381%); 6.6 MiB/s, 106 objects/s recovering 2024-04-29T14:53:09.011 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:09.057 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:09.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:09.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:08 smithi121 ceph-mon[44533]: osdmap e1030: 12 total, 11 up, 9 in 2024-04-29T14:53:09.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:08 smithi121 ceph-mon[44533]: pgmap v2601: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 52 MiB/s wr, 834 op/s; 20764/182452 objects degraded (11.381%); 6.6 MiB/s, 106 objects/s recovering 2024-04-29T14:53:09.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:09.208 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:09.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:09.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:09.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:53:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:09.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 16 95855 95839 105.066 60.1875 0.0190568 0.00951467 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:230993 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.90791808, 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076782375, 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817203+0000", 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.203e-06 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817208+0000", 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999996e-06 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817218+0000", 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5929999999999995e-06 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817224+0000", 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7530000000000004e-06 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893964+0000", 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076739872000000001 2024-04-29T14:53:09.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893972+0000", 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0450000000000001e-06 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893985+0000", 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2196e-05 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231043 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.720318701, 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.084551465000000006, 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:53:09.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004804+0000", 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2709999999999999e-06 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004816+0000", 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1864e-05 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004832+0000", 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6670000000000001e-05 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004845+0000", 2024-04-29T14:53:09.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2401e-05 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089323+0000", 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.084478445999999999 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089334+0000", 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1250000000000001e-05 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089353+0000", 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8563000000000002e-05 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231044 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.71600270099998, 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080603953000000006, 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.741 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009119+0000", 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.866e-06 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009125+0000", 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.015e-06 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009136+0000", 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0185e-05 2024-04-29T14:53:09.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009143+0000", 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1029999999999996e-06 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089391+0000", 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080247936000000006 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089398+0000", 2024-04-29T14:53:09.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089721+0000", 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032323900000000001 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231045 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.71157513499998, 2024-04-29T14:53:09.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076311253999999995, 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013547+0000", 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.624e-06 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013553+0000", 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9710000000000001e-06 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013562+0000", 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7529999999999996e-06 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013570+0000", 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2590000000000003e-06 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089749+0000", 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076179350000000007 2024-04-29T14:53:09.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089757+0000", 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8290000000000008e-06 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089856+0000", 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9468000000000004e-05 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231093 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.53105908200001, 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082382412000000002, 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:53:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194063+0000", 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4720000000000001e-06 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194068+0000", 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.558e-06 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194078+0000", 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9629999999999993e-06 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194085+0000", 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3579999999999999e-06 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276425+0000", 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082340872999999995 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276432+0000", 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5309999999999998e-06 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276444+0000", 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1657e-05 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231094 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.527652999, 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079185309999999995, 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197469+0000", 2024-04-29T14:53:09.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5579999999999999e-06 2024-04-29T14:53:09.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197475+0000", 2024-04-29T14:53:09.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.241e-06 2024-04-29T14:53:09.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197487+0000", 2024-04-29T14:53:09.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1905e-05 2024-04-29T14:53:09.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197495+0000", 2024-04-29T14:53:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5099999999999998e-06 2024-04-29T14:53:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276469+0000", 2024-04-29T14:53:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078973186000000001 2024-04-29T14:53:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276474+0000", 2024-04-29T14:53:09.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7660000000000004e-06 2024-04-29T14:53:09.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276653+0000", 2024-04-29T14:53:09.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017814400000000001 2024-04-29T14:53:09.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.762 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231095 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.762 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:53:09.762 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.52474959599999, 2024-04-29T14:53:09.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076347655, 2024-04-29T14:53:09.762 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.763 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.763 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.763 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:53:09.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:53:09.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200372+0000", 2024-04-29T14:53:09.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.606e-06 2024-04-29T14:53:09.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200378+0000", 2024-04-29T14:53:09.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0020000000000003e-06 2024-04-29T14:53:09.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200390+0000", 2024-04-29T14:53:09.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1772e-05 2024-04-29T14:53:09.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200399+0000", 2024-04-29T14:53:09.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7050000000000005e-06 2024-04-29T14:53:09.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276664+0000", 2024-04-29T14:53:09.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076265239999999998 2024-04-29T14:53:09.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276668+0000", 2024-04-29T14:53:09.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9620000000000004e-06 2024-04-29T14:53:09.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276718+0000", 2024-04-29T14:53:09.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0368000000000003e-05 2024-04-29T14:53:09.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.771 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231143 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.771 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:53:09.771 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.34417767799999, 2024-04-29T14:53:09.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076904044000000005, 2024-04-29T14:53:09.771 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:53:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:53:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380944+0000", 2024-04-29T14:53:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2550000000000001e-06 2024-04-29T14:53:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380949+0000", 2024-04-29T14:53:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8910000000000003e-06 2024-04-29T14:53:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380958+0000", 2024-04-29T14:53:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5729999999999997e-06 2024-04-29T14:53:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380966+0000", 2024-04-29T14:53:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6469999999999998e-06 2024-04-29T14:53:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457786+0000", 2024-04-29T14:53:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076820090999999993 2024-04-29T14:53:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457792+0000", 2024-04-29T14:53:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0270000000000001e-06 2024-04-29T14:53:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457847+0000", 2024-04-29T14:53:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4559999999999999e-05 2024-04-29T14:53:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231168 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:53:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:53:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 257.25141162400001, 2024-04-29T14:53:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078192824999999994, 2024-04-29T14:53:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:53:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:53:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473711+0000", 2024-04-29T14:53:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4590000000000001e-06 2024-04-29T14:53:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473719+0000", 2024-04-29T14:53:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0930000000000003e-06 2024-04-29T14:53:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473730+0000", 2024-04-29T14:53:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0621e-05 2024-04-29T14:53:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473738+0000", 2024-04-29T14:53:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5310000000000001e-06 2024-04-29T14:53:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551873+0000", 2024-04-29T14:53:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078134561000000005 2024-04-29T14:53:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551883+0000", 2024-04-29T14:53:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0236e-05 2024-04-29T14:53:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551901+0000", 2024-04-29T14:53:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8323999999999999e-05 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232295 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:53:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 253.68312570800001, 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076427287999999996, 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:53:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041996+0000", 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8369999999999999e-06 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042004+0000", 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6930000000000007e-06 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042015+0000", 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1385e-05 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042024+0000", 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1200000000000002e-06 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118396+0000", 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076372851000000005 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118406+0000", 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8090000000000004e-06 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118422+0000", 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5593000000000001e-05 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232296 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:53:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 253.68297203, 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076313084000000003, 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042149+0000", 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5799999999999999e-07 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042152+0000", 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.078e-06 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042159+0000", 2024-04-29T14:53:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3309999999999998e-06 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042164+0000", 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0719999999999999e-06 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118442+0000", 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076277242999999995 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118448+0000", 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5119999999999997e-06 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118461+0000", 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3190000000000001e-05 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178706 7.c 7:34fb7533:::benchmark_data_smithi194_147200_object63794:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 253.40358800800001, 2024-04-29T14:53:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093309815000000004, 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178706 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321534+0000", 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0490000000000002e-06 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321540+0000", 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9490000000000001e-06 2024-04-29T14:53:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321552+0000", 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1925e-05 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321574+0000", 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2342999999999999e-05 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.332747+0000", 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011172127 2024-04-29T14:53:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.332889+0000", 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014268700000000001 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.333033+0000", 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000144041 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.336903+0000", 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038697530000000001 2024-04-29T14:53:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414719+0000", 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077815881000000003 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414779+0000", 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0061e-05 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414842+0000", 2024-04-29T14:53:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2998999999999998e-05 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178709 7.c 7:30b10f7c:::benchmark_data_smithi194_147200_object63797:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 253.395475293, 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085420788999999997, 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:53:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178709 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329648+0000", 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6699999999999998e-06 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329655+0000", 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0949999999999998e-06 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329672+0000", 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7569000000000001e-05 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329682+0000", 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8660000000000001e-06 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.336948+0000", 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007265886 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.337066+0000", 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011782600000000001 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.337188+0000", 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012226199999999999 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414891+0000", 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077703373000000006 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414960+0000", 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8461999999999996e-05 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.415006+0000", 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6026999999999998e-05 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.415066+0000", 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9753000000000002e-05 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178745 7.c 7:37c53f67:::benchmark_data_smithi194_147200_object63833:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:53:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 253.299310381, 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078737684000000002, 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178745 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:53:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425811+0000", 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4419999999999999e-06 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425816+0000", 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5249999999999999e-06 2024-04-29T14:53:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425827+0000", 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0712e-05 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425833+0000", 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2750000000000001e-06 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432006+0000", 2024-04-29T14:53:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061731959999999997 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432085+0000", 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8881000000000006e-05 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432164+0000", 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9328999999999999e-05 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504359+0000", 2024-04-29T14:53:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.072195230999999999 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504427+0000", 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7816000000000003e-05 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504482+0000", 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4471000000000001e-05 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504548+0000", 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5806e-05 2024-04-29T14:53:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178796 7.c 7:31d4dc8d:::benchmark_data_smithi194_147200_object63884:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 253.1093669, 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.081790044000000006, 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:53:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178796 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615755+0000", 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5829999999999999e-06 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615760+0000", 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000001e-06 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615774+0000", 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3879999999999999e-05 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615780+0000", 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.105e-06 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619804+0000", 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004023357 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619881+0000", 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7272000000000001e-05 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:53:09.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619956+0000", 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5086000000000006e-05 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.623354+0000", 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003398252 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697462+0000", 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.074107585000000004 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697500+0000", 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8003999999999997e-05 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697543+0000", 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3529999999999998e-05 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232772 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:53:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 252.16361555399999, 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077453872000000007, 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561506+0000", 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.782e-06 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561515+0000", 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1289999999999996e-06 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561529+0000", 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4644000000000001e-05 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561539+0000", 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5380000000000008e-06 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638939+0000", 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077400374999999993 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638947+0000", 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8569999999999995e-06 2024-04-29T14:53:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638959+0000", 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1547e-05 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178971 7.c 7:3606b2fb:::benchmark_data_smithi194_147200_object64059:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 251.897570204, 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090073625000000004, 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178971 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827552+0000", 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5999999999999999e-06 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827556+0000", 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7849999999999999e-06 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827565+0000", 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7970000000000006e-06 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827571+0000", 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9120000000000003e-06 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.833976+0000", 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064046939999999998 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.834060+0000", 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4618999999999997e-05 2024-04-29T14:53:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.834144+0000", 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3987999999999996e-05 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917478+0000", 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.083333661000000003 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917526+0000", 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7812000000000002e-05 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917563+0000", 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6936e-05 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917624+0000", 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0820999999999999e-05 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178972 7.c 7:34cd9e3a:::benchmark_data_smithi194_147200_object64060:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:53:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 251.89677248000001, 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089382420000000004, 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178972 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828349+0000", 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.257e-06 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828353+0000", 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3440000000000003e-06 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828363+0000", 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0026e-05 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828370+0000", 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9700000000000002e-06 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837577+0000", 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092070030000000001 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837702+0000", 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012509000000000001 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837849+0000", 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014664800000000001 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917659+0000", 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079810324000000002 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917667+0000", 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7069999999999992e-06 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917696+0000", 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8530999999999998e-05 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917730+0000", 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26998.0:178971 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 251.88756383099999, 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080073583000000004, 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837557+0000", 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.066e-06 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837562+0000", 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8040000000000001e-06 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837572+0000", 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5019999999999998e-06 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837579+0000", 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1620000000000002e-06 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917515+0000", 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079936129999999994 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917522+0000", 2024-04-29T14:53:09.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6059999999999999e-06 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917630+0000", 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108313 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26998.0:178972 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 251.88433470199999, 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077106775000000002, 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:53:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840787+0000", 2024-04-29T14:53:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3820000000000001e-06 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840791+0000", 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9169999999999999e-06 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840800+0000", 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1640000000000007e-06 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840806+0000", 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.648e-06 2024-04-29T14:53:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917651+0000", 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076845573 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917656+0000", 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4549999999999997e-06 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917892+0000", 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236636 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:53:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:53:09.841 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:53:09.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:09 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 20764/182452 objects degraded (11.381%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:09.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2400597152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:10.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:09 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 20764/182452 objects degraded (11.381%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:10.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2400597152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:10.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:09 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 20764/182452 objects degraded (11.381%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:10.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2400597152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:53:10.598 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:53:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:53:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:53:10.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 96884 96868 104.363 64.3125 0.003808 0.00957958 2024-04-29T14:53:11.031 INFO:teuthology.orchestra.run.smithi194.stdout:8.c 2024-04-29T14:53:11.031 INFO:teuthology.orchestra.run.smithi194.stdout:8.a 2024-04-29T14:53:11.032 INFO:teuthology.orchestra.run.smithi194.stdout:8.1 2024-04-29T14:53:11.032 INFO:teuthology.orchestra.run.smithi194.stdout:8.4 2024-04-29T14:53:11.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:11.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:11.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osdmap e1031: 12 total, 11 up, 9 in 2024-04-29T14:53:11.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:11.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:11.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:10 smithi194 ceph-mon[44089]: pgmap v2603: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 20578/187664 objects degraded (10.965%); 6.8 MiB/s, 109 objects/s recovering 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osdmap e1031: 12 total, 11 up, 9 in 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:11.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:10 smithi040 ceph-mon[39581]: pgmap v2603: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 20578/187664 objects degraded (10.965%); 6.8 MiB/s, 109 objects/s recovering 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osdmap e1031: 12 total, 11 up, 9 in 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:11.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:11.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:11.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:10 smithi121 ceph-mon[44533]: pgmap v2603: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 63 MiB/s wr, 1.01k op/s; 20578/187664 objects degraded (10.965%); 6.8 MiB/s, 109 objects/s recovering 2024-04-29T14:53:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:53:11.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:53:11.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 97960 97944 103.734 67.25 0.010308 0.00963774 2024-04-29T14:53:11.926 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:53:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:11.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:53:12.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:12.067 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:11 smithi121 ceph-mon[44533]: osdmap e1032: 12 total, 11 up, 9 in 2024-04-29T14:53:12.070 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:12.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:11 smithi194 ceph-mon[44089]: osdmap e1032: 12 total, 11 up, 9 in 2024-04-29T14:53:12.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:11 smithi040 ceph-mon[39581]: osdmap e1032: 12 total, 11 up, 9 in 2024-04-29T14:53:12.239 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.9 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-9 --journal-path /var/lib/ceph/osd/ceph-9/journal --op export --pgid 8.c --file /var/log/ceph/exp.8.c.9 2024-04-29T14:53:12.559 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.9/config 2024-04-29T14:53:12.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:53:12.600566 min lat: 0.00105393 max lat: 17.3914 avg lat: 0.00968828 2024-04-29T14:53:12.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:53:12.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 99100 99084 103.192 71.25 0.00755532 0.00968828 2024-04-29T14:53:12.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:12.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:12.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:53:12.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:12.890 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:12.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:13.029 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:13.069 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:12 smithi194 ceph-mon[44089]: pgmap v2605: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 20053/192824 objects degraded (10.400%); 9.1 MiB/s, 145 objects/s recovering 2024-04-29T14:53:13.071 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:13.108 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:13.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:13.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:12 smithi040 ceph-mon[39581]: pgmap v2605: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 20053/192824 objects degraded (10.400%); 9.1 MiB/s, 145 objects/s recovering 2024-04-29T14:53:13.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:12 smithi121 ceph-mon[44533]: pgmap v2605: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 20053/192824 objects degraded (10.400%); 9.1 MiB/s, 145 objects/s recovering 2024-04-29T14:53:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:53:13.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:13.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:13.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:13.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:53:13.427 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:13.468 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:13.551 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:13.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:13.603 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 100168 100152 102.595 66.75 0.0479282 0.00974462 2024-04-29T14:53:13.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:13.697 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:13.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:13.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:13.751 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:53:13.823 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:13 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:13 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2413705483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:13 smithi194 ceph-mon[44089]: from='client.28204 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:13.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:13.880 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:14.077 INFO:teuthology.orchestra.run.smithi194.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:53:14.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:13 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:14.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:13 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:14.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2413705483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:14.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:13 smithi040 ceph-mon[39581]: from='client.28204 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:13 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:13 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2413705483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:13 smithi121 ceph-mon[44533]: from='client.28204 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:14.604 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 16 101305 101289 102.086 71.0625 0.0155307 0.00979347 2024-04-29T14:53:14.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:14 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:53:14.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:14 smithi194 ceph-mon[44089]: from='client.28204 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:14.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:14 smithi194 ceph-mon[44089]: osdmap e1033: 12 total, 11 up, 9 in 2024-04-29T14:53:14.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:14 smithi194 ceph-mon[44089]: pgmap v2607: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 19855/195464 objects degraded (10.158%); 9.5 MiB/s, 151 objects/s recovering 2024-04-29T14:53:15.072 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:15.077 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:14 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:53:15.077 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:14 smithi040 ceph-mon[39581]: from='client.28204 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:15.077 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:14 smithi040 ceph-mon[39581]: osdmap e1033: 12 total, 11 up, 9 in 2024-04-29T14:53:15.078 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:14 smithi040 ceph-mon[39581]: pgmap v2607: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 19855/195464 objects degraded (10.158%); 9.5 MiB/s, 151 objects/s recovering 2024-04-29T14:53:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:15.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:15.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:14 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:53:15.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:14 smithi121 ceph-mon[44533]: from='client.28204 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:15.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:14 smithi121 ceph-mon[44533]: osdmap e1033: 12 total, 11 up, 9 in 2024-04-29T14:53:15.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:14 smithi121 ceph-mon[44533]: pgmap v2607: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 68 MiB/s wr, 1.08k op/s; 19855/195464 objects degraded (10.158%); 9.5 MiB/s, 151 objects/s recovering 2024-04-29T14:53:15.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:15.214 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:15.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:15.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:15.604 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 102395 102379 101.546 68.125 0.00582597 0.00984456 2024-04-29T14:53:15.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:15.721 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:15.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:15.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:15.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:15.881 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:15 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 19855/195464 objects degraded (10.158%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:15.884 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:16.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:15 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 19855/195464 objects degraded (10.158%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:16.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:16.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:16.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:15 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 19855/195464 objects degraded (10.158%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:16.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:16.586 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:53:16.586 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:53:16.587 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:53:16.587 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:53:16.605 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 15 103351 103336 100.894 59.8125 0.00974833 0.00990945 2024-04-29T14:53:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:53:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:53:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:53:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:53:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:16.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:16.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: pgmap v2608: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 62 MiB/s wr, 999 op/s; 19491/200956 objects degraded (9.699%); 10 MiB/s, 163 objects/s recovering 2024-04-29T14:53:16.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:16 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: pgmap v2608: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 62 MiB/s wr, 999 op/s; 19491/200956 objects degraded (9.699%); 10 MiB/s, 163 objects/s recovering 2024-04-29T14:53:17.007 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:16 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: pgmap v2608: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.1 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 62 MiB/s wr, 999 op/s; 19491/200956 objects degraded (9.699%); 10 MiB/s, 163 objects/s recovering 2024-04-29T14:53:17.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:16 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:17.201 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:17.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:17.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:17.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:17.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:17.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:53:17.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:17.530 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:17.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:17.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:17.604 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 104373 104357 100.323 63.8125 0.00495921 0.009965 2024-04-29T14:53:17.664 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:17.718 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:17.766 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:17.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:17.780 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:53:17.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:18.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:18.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:18.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:18.231 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:18.279 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:18.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:18.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:18 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:18 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:53:18.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:18 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:53:18.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:18 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:18 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:18 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:53:18.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:18 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:53:18.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:18 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:18.428 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:18.455 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:18.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:18.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:18.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:18.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:18 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:18 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:53:18.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:18 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:53:18.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:18 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:53:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:18.606 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 16 105249 105233 99.63 54.75 0.0395316 0.0100344 2024-04-29T14:53:18.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:18.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:18.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:53:18.753 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:53:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:18.784 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:53:18.807 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:53:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:18.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:53:18.819 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:53:18.845 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:53:18.872 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:53:18.897 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:53:18.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:18.920 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:53:18.948 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:18.949 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:53:18.975 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:18.976 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:53:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:18.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:19.002 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:53:19.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:53:19.027 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:53:19.027 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:53:19.028 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:53:19.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:19.107 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:19.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:19.240 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:19.293 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:19.334 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:19.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:19.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:19 smithi040 ceph-mon[39581]: pgmap v2609: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 63 MiB/s wr, 1k op/s; 19338/203674 objects degraded (9.495%); 9.7 MiB/s, 154 objects/s recovering 2024-04-29T14:53:19.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1014363722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:19.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:19 smithi121 ceph-mon[44533]: pgmap v2609: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 63 MiB/s wr, 1k op/s; 19338/203674 objects degraded (9.495%); 9.7 MiB/s, 154 objects/s recovering 2024-04-29T14:53:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1014363722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:19.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.441 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:19.486 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:19.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:19 smithi194 ceph-mon[44089]: pgmap v2609: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 63 MiB/s wr, 1k op/s; 19338/203674 objects degraded (9.495%); 9.7 MiB/s, 154 objects/s recovering 2024-04-29T14:53:19.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1014363722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:19.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:19.520 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:19.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:19.528 INFO:teuthology.orchestra.run.smithi194.stderr:Exporting 8.c info 8.c( v 1019'5334 (0'0,1019'5334] local-lis/les=1011/1012 n=5334 ec=1011/1011 lis/c=1011/1011 les/c/f=1012/1012/0 sis=1011) 2024-04-29T14:53:19.606 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3001a9c1:::benchmark_data_smithi194_186548_object20385:head# 2024-04-29T14:53:19.607 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 16 105875 105859 98.727 39.125 0.0201566 0.0101271 2024-04-29T14:53:19.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3002e01d:::benchmark_data_smithi194_186548_object43396:head# 2024-04-29T14:53:19.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30031bd7:::benchmark_data_smithi194_186548_object85221:head# 2024-04-29T14:53:19.612 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30032756:::benchmark_data_smithi194_186548_object75488:head# 2024-04-29T14:53:19.612 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3004c823:::benchmark_data_smithi194_186548_object84981:head# 2024-04-29T14:53:19.612 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30054e5e:::benchmark_data_smithi194_186548_object50697:head# 2024-04-29T14:53:19.612 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3005da6d:::benchmark_data_smithi194_186548_object5106:head# 2024-04-29T14:53:19.613 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3008989b:::benchmark_data_smithi194_186548_object67505:head# 2024-04-29T14:53:19.613 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3008fa74:::benchmark_data_smithi194_186548_object26989:head# 2024-04-29T14:53:19.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:300a4e0e:::benchmark_data_smithi194_186548_object73691:head# 2024-04-29T14:53:19.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:300baae3:::benchmark_data_smithi194_186548_object54938:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:300d8b52:::benchmark_data_smithi194_186548_object27233:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:300e002e:::benchmark_data_smithi194_186548_object59932:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301062c2:::benchmark_data_smithi194_186548_object40164:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3010b94c:::benchmark_data_smithi194_186548_object15048:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3011bd7f:::benchmark_data_smithi194_186548_object44525:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3011db83:::benchmark_data_smithi194_186548_object43878:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3013568f:::benchmark_data_smithi194_186548_object65175:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30146ccb:::benchmark_data_smithi194_186548_object42660:head# 2024-04-29T14:53:19.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30155c3b:::benchmark_data_smithi194_186548_object56698:head# 2024-04-29T14:53:19.621 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301710c9:::benchmark_data_smithi194_186548_object56908:head# 2024-04-29T14:53:19.622 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3017170e:::benchmark_data_smithi194_186548_object33389:head# 2024-04-29T14:53:19.622 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3017633f:::benchmark_data_smithi194_186548_object75288:head# 2024-04-29T14:53:19.623 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30179c5f:::benchmark_data_smithi194_186548_object73705:head# 2024-04-29T14:53:19.626 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301a1f35:::benchmark_data_smithi194_186548_object73186:head# 2024-04-29T14:53:19.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301b54cb:::benchmark_data_smithi194_186548_object17569:head# 2024-04-29T14:53:19.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301bc4f8:::benchmark_data_smithi194_186548_object22895:head# 2024-04-29T14:53:19.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:19.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301cf017:::benchmark_data_smithi194_186548_object8327:head# 2024-04-29T14:53:19.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301d3a0c:::benchmark_data_smithi194_186548_object14001:head# 2024-04-29T14:53:19.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301e04df:::benchmark_data_smithi194_186548_object21756:head# 2024-04-29T14:53:19.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:301e3f1a:::benchmark_data_smithi194_186548_object59275:head# 2024-04-29T14:53:19.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3020e805:::benchmark_data_smithi194_186548_object73480:head# 2024-04-29T14:53:19.638 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3023a4ef:::benchmark_data_smithi194_186548_object62445:head# 2024-04-29T14:53:19.638 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:302442f4:::benchmark_data_smithi194_186548_object13781:head# 2024-04-29T14:53:19.639 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30244e36:::benchmark_data_smithi194_186548_object20975:head# 2024-04-29T14:53:19.639 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30249d96:::benchmark_data_smithi194_186548_object39106:head# 2024-04-29T14:53:19.639 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3024d82a:::benchmark_data_smithi194_186548_object26942:head# 2024-04-29T14:53:19.640 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:302510f2:::benchmark_data_smithi194_186548_object46162:head# 2024-04-29T14:53:19.640 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3025c475:::benchmark_data_smithi194_186548_object64863:head# 2024-04-29T14:53:19.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3025e44b:::benchmark_data_smithi194_186548_object22759:head# 2024-04-29T14:53:19.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30275198:::benchmark_data_smithi194_186548_object25641:head# 2024-04-29T14:53:19.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3027c1b4:::benchmark_data_smithi194_186548_object45871:head# 2024-04-29T14:53:19.644 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3028142a:::benchmark_data_smithi194_186548_object56958:head# 2024-04-29T14:53:19.644 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30295264:::benchmark_data_smithi194_186548_object76261:head# 2024-04-29T14:53:19.645 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:302aa7c5:::benchmark_data_smithi194_186548_object15931:head# 2024-04-29T14:53:19.645 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:302ffd6a:::benchmark_data_smithi194_186548_object74282:head# 2024-04-29T14:53:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30305a80:::benchmark_data_smithi194_186548_object69495:head# 2024-04-29T14:53:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30326251:::benchmark_data_smithi194_186548_object67737:head# 2024-04-29T14:53:19.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3032a4c5:::benchmark_data_smithi194_186548_object30090:head# 2024-04-29T14:53:19.647 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3032aa50:::benchmark_data_smithi194_186548_object1674:head# 2024-04-29T14:53:19.647 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30342c6d:::benchmark_data_smithi194_186548_object23929:head# 2024-04-29T14:53:19.648 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303474ff:::benchmark_data_smithi194_186548_object51278:head# 2024-04-29T14:53:19.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3034e2f5:::benchmark_data_smithi194_186548_object81398:head# 2024-04-29T14:53:19.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30352c6d:::benchmark_data_smithi194_186548_object21805:head# 2024-04-29T14:53:19.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30367b68:::benchmark_data_smithi194_186548_object39511:head# 2024-04-29T14:53:19.651 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30377b10:::benchmark_data_smithi194_186548_object62663:head# 2024-04-29T14:53:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30391911:::benchmark_data_smithi194_186548_object85068:head# 2024-04-29T14:53:19.652 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3039768e:::benchmark_data_smithi194_186548_object11088:head# 2024-04-29T14:53:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3039e15d:::benchmark_data_smithi194_186548_object55976:head# 2024-04-29T14:53:19.654 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303a265b:::benchmark_data_smithi194_186548_object31794:head# 2024-04-29T14:53:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303a53a8:::benchmark_data_smithi194_186548_object44519:head# 2024-04-29T14:53:19.655 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303d26eb:::benchmark_data_smithi194_186548_object2589:head# 2024-04-29T14:53:19.656 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303daf2f:::benchmark_data_smithi194_186548_object79914:head# 2024-04-29T14:53:19.658 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303dff45:::benchmark_data_smithi194_186548_object64595:head# 2024-04-29T14:53:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303e6956:::benchmark_data_smithi194_186548_object46926:head# 2024-04-29T14:53:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303e80b7:::benchmark_data_smithi194_186548_object75469:head# 2024-04-29T14:53:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303ede86:::benchmark_data_smithi194_186548_object51208:head# 2024-04-29T14:53:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303f4f23:::benchmark_data_smithi194_186548_object59160:head# 2024-04-29T14:53:19.660 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:303f881d:::benchmark_data_smithi194_186548_object85185:head# 2024-04-29T14:53:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30402b19:::benchmark_data_smithi194_186548_object24881:head# 2024-04-29T14:53:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30403988:::benchmark_data_smithi194_186548_object58815:head# 2024-04-29T14:53:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30407e5e:::benchmark_data_smithi194_186548_object2513:head# 2024-04-29T14:53:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3040dfca:::benchmark_data_smithi194_186548_object69051:head# 2024-04-29T14:53:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3042d061:::benchmark_data_smithi194_186548_object9235:head# 2024-04-29T14:53:19.663 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30437437:::benchmark_data_smithi194_186548_object16095:head# 2024-04-29T14:53:19.663 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30449994:::benchmark_data_smithi194_186548_object17740:head# 2024-04-29T14:53:19.664 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304526c8:::benchmark_data_smithi194_186548_object33960:head# 2024-04-29T14:53:19.665 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30456894:::benchmark_data_smithi194_186548_object36112:head# 2024-04-29T14:53:19.666 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3045a224:::benchmark_data_smithi194_186548_object18352:head# 2024-04-29T14:53:19.667 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30469a19:::benchmark_data_smithi194_186548_object35151:head# 2024-04-29T14:53:19.667 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:19.668 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3047160d:::benchmark_data_smithi194_186548_object66445:head# 2024-04-29T14:53:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30471f20:::benchmark_data_smithi194_186548_object28449:head# 2024-04-29T14:53:19.670 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3047f5a9:::benchmark_data_smithi194_186548_object36059:head# 2024-04-29T14:53:19.673 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30484c63:::benchmark_data_smithi194_186548_object81763:head# 2024-04-29T14:53:19.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30498a61:::benchmark_data_smithi194_186548_object55960:head# 2024-04-29T14:53:19.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304a6762:::benchmark_data_smithi194_186548_object22480:head# 2024-04-29T14:53:19.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304a7e04:::benchmark_data_smithi194_186548_object391:head# 2024-04-29T14:53:19.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304b9bef:::benchmark_data_smithi194_186548_object25484:head# 2024-04-29T14:53:19.678 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304c467f:::benchmark_data_smithi194_186548_object13432:head# 2024-04-29T14:53:19.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304d6b8e:::benchmark_data_smithi194_186548_object32154:head# 2024-04-29T14:53:19.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304e0ea5:::benchmark_data_smithi194_186548_object72429:head# 2024-04-29T14:53:19.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304e2542:::benchmark_data_smithi194_186548_object990:head# 2024-04-29T14:53:19.680 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304e3839:::benchmark_data_smithi194_186548_object26899:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304e8910:::benchmark_data_smithi194_186548_object71799:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:304ef438:::benchmark_data_smithi194_186548_object71039:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30511dd9:::benchmark_data_smithi194_186548_object26466:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30512e5d:::benchmark_data_smithi194_186548_object12434:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30514bcc:::benchmark_data_smithi194_186548_object30234:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3051be27:::benchmark_data_smithi194_186548_object46428:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3051be55:::benchmark_data_smithi194_186548_object23422:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3051e746:::benchmark_data_smithi194_186548_object1228:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3052db28:::benchmark_data_smithi194_186548_object70557:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3052de21:::benchmark_data_smithi194_186548_object54004:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30530e7e:::benchmark_data_smithi194_186548_object84549:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3054f6ff:::benchmark_data_smithi194_186548_object74284:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3056d7f4:::benchmark_data_smithi194_186548_object26271:head# 2024-04-29T14:53:19.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3057394d:::benchmark_data_smithi194_186548_object4514:head# 2024-04-29T14:53:19.688 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3057b059:::benchmark_data_smithi194_186548_object25715:head# 2024-04-29T14:53:19.688 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3057b26a:::benchmark_data_smithi194_186548_object47053:head# 2024-04-29T14:53:19.691 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3057dc75:::benchmark_data_smithi194_186548_object53431:head# 2024-04-29T14:53:19.691 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30587ea8:::benchmark_data_smithi194_186548_object72361:head# 2024-04-29T14:53:19.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3058923e:::benchmark_data_smithi194_186548_object29313:head# 2024-04-29T14:53:19.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3058cedc:::benchmark_data_smithi194_186548_object69339:head# 2024-04-29T14:53:19.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30592d46:::benchmark_data_smithi194_186548_object73652:head# 2024-04-29T14:53:19.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3059466e:::benchmark_data_smithi194_186548_object34792:head# 2024-04-29T14:53:19.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305979c2:::benchmark_data_smithi194_186548_object18266:head# 2024-04-29T14:53:19.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305ae4d0:::benchmark_data_smithi194_186548_object85181:head# 2024-04-29T14:53:19.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305cb0d5:::benchmark_data_smithi194_186548_object67209:head# 2024-04-29T14:53:19.693 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305cdec0:::benchmark_data_smithi194_186548_object39874:head# 2024-04-29T14:53:19.693 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305ce0c8:::benchmark_data_smithi194_186548_object73414:head# 2024-04-29T14:53:19.694 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305d049b:::benchmark_data_smithi194_186548_object48101:head# 2024-04-29T14:53:19.694 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305d22e8:::benchmark_data_smithi194_186548_object85127:head# 2024-04-29T14:53:19.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305dff14:::benchmark_data_smithi194_186548_object62631:head# 2024-04-29T14:53:19.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305f20a6:::benchmark_data_smithi194_186548_object49971:head# 2024-04-29T14:53:19.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305fc996:::benchmark_data_smithi194_186548_object10805:head# 2024-04-29T14:53:19.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:305fd176:::benchmark_data_smithi194_186548_object26417:head# 2024-04-29T14:53:19.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30614a41:::benchmark_data_smithi194_186548_object84380:head# 2024-04-29T14:53:19.697 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3061cdf5:::benchmark_data_smithi194_186548_object48414:head# 2024-04-29T14:53:19.697 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30625ac7:::benchmark_data_smithi194_186548_object78366:head# 2024-04-29T14:53:19.698 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3062786f:::benchmark_data_smithi194_186548_object81898:head# 2024-04-29T14:53:19.698 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3062d661:::benchmark_data_smithi194_186548_object27412:head# 2024-04-29T14:53:19.699 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3063bdc2:::benchmark_data_smithi194_186548_object18551:head# 2024-04-29T14:53:19.699 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3063fe7c:::benchmark_data_smithi194_186548_object78852:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30644165:::benchmark_data_smithi194_186548_object11059:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3066647e:::benchmark_data_smithi194_186548_object76367:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30688284:::benchmark_data_smithi194_186548_object69192:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3069a2a5:::benchmark_data_smithi194_186548_object38536:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306a88ad:::benchmark_data_smithi194_186548_object16137:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306afbfa:::benchmark_data_smithi194_186548_object20477:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306c3499:::benchmark_data_smithi194_186548_object62492:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306c9284:::benchmark_data_smithi194_186548_object5952:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306d0866:::benchmark_data_smithi194_186548_object55840:head# 2024-04-29T14:53:19.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306d2551:::benchmark_data_smithi194_186548_object74390:head# 2024-04-29T14:53:19.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306dd155:::benchmark_data_smithi194_186548_object40960:head# 2024-04-29T14:53:19.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306e1177:::benchmark_data_smithi194_186548_object57987:head# 2024-04-29T14:53:19.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306ea25e:::benchmark_data_smithi194_186548_object67115:head# 2024-04-29T14:53:19.706 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:306ecde3:::benchmark_data_smithi194_186548_object15150:head# 2024-04-29T14:53:19.706 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307046fa:::benchmark_data_smithi194_186548_object23489:head# 2024-04-29T14:53:19.707 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3070d785:::benchmark_data_smithi194_186548_object33366:head# 2024-04-29T14:53:19.708 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30712378:::benchmark_data_smithi194_186548_object13926:head# 2024-04-29T14:53:19.708 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30714ca0:::benchmark_data_smithi194_186548_object20644:head# 2024-04-29T14:53:19.708 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3071f99f:::benchmark_data_smithi194_186548_object11598:head# 2024-04-29T14:53:19.709 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30723556:::benchmark_data_smithi194_186548_object16766:head# 2024-04-29T14:53:19.709 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307437ac:::benchmark_data_smithi194_186548_object42403:head# 2024-04-29T14:53:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30745fd2:::benchmark_data_smithi194_186548_object48788:head# 2024-04-29T14:53:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3074b1fe:::benchmark_data_smithi194_186548_object62927:head# 2024-04-29T14:53:19.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3074dc86:::benchmark_data_smithi194_186548_object36424:head# 2024-04-29T14:53:19.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3075841e:::benchmark_data_smithi194_186548_object19513:head# 2024-04-29T14:53:19.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307649f1:::benchmark_data_smithi194_186548_object34991:head# 2024-04-29T14:53:19.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3077c474:::benchmark_data_smithi194_186548_object27691:head# 2024-04-29T14:53:19.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30781d92:::benchmark_data_smithi194_186548_object67315:head# 2024-04-29T14:53:19.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30798b1b:::benchmark_data_smithi194_186548_object14731:head# 2024-04-29T14:53:19.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3079b4eb:::benchmark_data_smithi194_186548_object17933:head# 2024-04-29T14:53:19.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307a7d1b:::benchmark_data_smithi194_186548_object1074:head# 2024-04-29T14:53:19.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307d1089:::benchmark_data_smithi194_186548_object64782:head# 2024-04-29T14:53:19.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307d5f14:::benchmark_data_smithi194_186548_object43747:head# 2024-04-29T14:53:19.719 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307de8bc:::benchmark_data_smithi194_186548_object54153:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307ea710:::benchmark_data_smithi194_186548_object42483:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:307f9703:::benchmark_data_smithi194_186548_object61490:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3080d7c0:::benchmark_data_smithi194_186548_object41593:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30827c5f:::benchmark_data_smithi194_186548_object25384:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30830cfe:::benchmark_data_smithi194_186548_object57150:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30832345:::benchmark_data_smithi194_186548_object40274:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3083e638:::benchmark_data_smithi194_186548_object16414:head# 2024-04-29T14:53:19.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30882003:::benchmark_data_smithi194_186548_object73298:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3088be82:::benchmark_data_smithi194_186548_object2596:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3089f44f:::benchmark_data_smithi194_186548_object64479:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3089f4a9:::benchmark_data_smithi194_186548_object72219:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308a2567:::benchmark_data_smithi194_186548_object54183:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308ab0bf:::benchmark_data_smithi194_186548_object43780:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308af0cd:::benchmark_data_smithi194_186548_object14191:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308b55a9:::benchmark_data_smithi194_186548_object31607:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308ba660:::benchmark_data_smithi194_186548_object80416:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308c2691:::benchmark_data_smithi194_186548_object26145:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308cfc41:::benchmark_data_smithi194_186548_object44516:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308e73f5:::benchmark_data_smithi194_186548_object25364:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308eea60:::benchmark_data_smithi194_186548_object70055:head# 2024-04-29T14:53:19.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:308f339b:::benchmark_data_smithi194_186548_object32972:head# 2024-04-29T14:53:19.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309037ab:::benchmark_data_smithi194_186548_object72681:head# 2024-04-29T14:53:19.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30905b2e:::benchmark_data_smithi194_186548_object14092:head# 2024-04-29T14:53:19.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3090d2bc:::benchmark_data_smithi194_186548_object30698:head# 2024-04-29T14:53:19.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30910a93:::benchmark_data_smithi194_186548_object20874:head# 2024-04-29T14:53:19.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309180c5:::benchmark_data_smithi194_186548_object80577:head# 2024-04-29T14:53:19.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30939f35:::benchmark_data_smithi194_186548_object52591:head# 2024-04-29T14:53:19.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30941345:::benchmark_data_smithi194_186548_object9034:head# 2024-04-29T14:53:19.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30941bf2:::benchmark_data_smithi194_186548_object35681:head# 2024-04-29T14:53:19.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3094c438:::benchmark_data_smithi194_186548_object64102:head# 2024-04-29T14:53:19.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30963baa:::benchmark_data_smithi194_186548_object8127:head# 2024-04-29T14:53:19.736 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30972536:::benchmark_data_smithi194_186548_object78226:head# 2024-04-29T14:53:19.736 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30979c74:::benchmark_data_smithi194_186548_object69294:head# 2024-04-29T14:53:19.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30992ffe:::benchmark_data_smithi194_186548_object55342:head# 2024-04-29T14:53:19.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3099726c:::benchmark_data_smithi194_186548_object73183:head# 2024-04-29T14:53:19.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309bc560:::benchmark_data_smithi194_186548_object69165:head# 2024-04-29T14:53:19.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309cb543:::benchmark_data_smithi194_186548_object32337:head# 2024-04-29T14:53:19.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309d217e:::benchmark_data_smithi194_186548_object68272:head# 2024-04-29T14:53:19.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309e1957:::benchmark_data_smithi194_186548_object37375:head# 2024-04-29T14:53:19.739 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309e4468:::benchmark_data_smithi194_186548_object46713:head# 2024-04-29T14:53:19.740 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:309ec3c2:::benchmark_data_smithi194_186548_object48088:head# 2024-04-29T14:53:19.740 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a08174:::benchmark_data_smithi194_186548_object56324:head# 2024-04-29T14:53:19.740 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a167da:::benchmark_data_smithi194_186548_object32674:head# 2024-04-29T14:53:19.740 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a19cc6:::benchmark_data_smithi194_186548_object9346:head# 2024-04-29T14:53:19.740 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a260c9:::benchmark_data_smithi194_186548_object51903:head# 2024-04-29T14:53:19.741 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a291aa:::benchmark_data_smithi194_186548_object24235:head# 2024-04-29T14:53:19.741 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a295e0:::benchmark_data_smithi194_186548_object11675:head# 2024-04-29T14:53:19.741 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a4cc15:::benchmark_data_smithi194_186548_object23812:head# 2024-04-29T14:53:19.742 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a640cf:::benchmark_data_smithi194_186548_object60187:head# 2024-04-29T14:53:19.742 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a741fa:::benchmark_data_smithi194_186548_object35545:head# 2024-04-29T14:53:19.743 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30a9eac8:::benchmark_data_smithi194_186548_object16126:head# 2024-04-29T14:53:19.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ac2a03:::benchmark_data_smithi194_186548_object62468:head# 2024-04-29T14:53:19.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30aceb2a:::benchmark_data_smithi194_186548_object67720:head# 2024-04-29T14:53:19.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ada11b:::benchmark_data_smithi194_186548_object12941:head# 2024-04-29T14:53:19.745 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b0ec13:::benchmark_data_smithi194_186548_object58780:head# 2024-04-29T14:53:19.745 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b2f89a:::benchmark_data_smithi194_186548_object43181:head# 2024-04-29T14:53:19.745 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b38020:::benchmark_data_smithi194_186548_object31835:head# 2024-04-29T14:53:19.746 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b3d758:::benchmark_data_smithi194_186548_object34325:head# 2024-04-29T14:53:19.746 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b59cbf:::benchmark_data_smithi194_186548_object77207:head# 2024-04-29T14:53:19.747 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b5bdd2:::benchmark_data_smithi194_186548_object26475:head# 2024-04-29T14:53:19.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b63798:::benchmark_data_smithi194_186548_object12924:head# 2024-04-29T14:53:19.750 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b7da79:::benchmark_data_smithi194_186548_object25151:head# 2024-04-29T14:53:19.750 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b84adb:::benchmark_data_smithi194_186548_object55705:head# 2024-04-29T14:53:19.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b8701e:::benchmark_data_smithi194_186548_object51927:head# 2024-04-29T14:53:19.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30b8742b:::benchmark_data_smithi194_186548_object71588:head# 2024-04-29T14:53:19.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30bc4b09:::benchmark_data_smithi194_186548_object2600:head# 2024-04-29T14:53:19.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30bf9293:::benchmark_data_smithi194_186548_object1623:head# 2024-04-29T14:53:19.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c0951f:::benchmark_data_smithi194_186548_object14275:head# 2024-04-29T14:53:19.753 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c16a07:::benchmark_data_smithi194_186548_object47432:head# 2024-04-29T14:53:19.754 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c1ce1c:::benchmark_data_smithi194_186548_object6281:head# 2024-04-29T14:53:19.755 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c20a1f:::benchmark_data_smithi194_186548_object38170:head# 2024-04-29T14:53:19.755 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c2af27:::benchmark_data_smithi194_186548_object2137:head# 2024-04-29T14:53:19.756 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c336bb:::benchmark_data_smithi194_186548_object34986:head# 2024-04-29T14:53:19.756 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c46dde:::benchmark_data_smithi194_186548_object48427:head# 2024-04-29T14:53:19.756 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c4adb7:::benchmark_data_smithi194_186548_object66866:head# 2024-04-29T14:53:19.757 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c4dbe9:::benchmark_data_smithi194_186548_object67211:head# 2024-04-29T14:53:19.757 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c4f8c6:::benchmark_data_smithi194_186548_object62882:head# 2024-04-29T14:53:19.758 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c4fae3:::benchmark_data_smithi194_186548_object24654:head# 2024-04-29T14:53:19.761 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c5bf18:::benchmark_data_smithi194_186548_object56848:head# 2024-04-29T14:53:19.762 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c67be5:::benchmark_data_smithi194_186548_object41230:head# 2024-04-29T14:53:19.762 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c6b3b3:::benchmark_data_smithi194_186548_object14814:head# 2024-04-29T14:53:19.762 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c7c77b:::benchmark_data_smithi194_186548_object73962:head# 2024-04-29T14:53:19.763 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c860f2:::benchmark_data_smithi194_186548_object48891:head# 2024-04-29T14:53:19.763 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c91c15:::benchmark_data_smithi194_186548_object70956:head# 2024-04-29T14:53:19.764 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30c9a345:::benchmark_data_smithi194_186548_object77637:head# 2024-04-29T14:53:19.766 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30cb31fb:::benchmark_data_smithi194_186548_object72311:head# 2024-04-29T14:53:19.767 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30cbb6fe:::benchmark_data_smithi194_186548_object56565:head# 2024-04-29T14:53:19.767 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30cd3b63:::benchmark_data_smithi194_186548_object54713:head# 2024-04-29T14:53:19.768 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ce1b5f:::benchmark_data_smithi194_186548_object48784:head# 2024-04-29T14:53:19.768 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d0677d:::benchmark_data_smithi194_186548_object6460:head# 2024-04-29T14:53:19.773 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d15856:::benchmark_data_smithi194_186548_object69434:head# 2024-04-29T14:53:19.773 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d16dfa:::benchmark_data_smithi194_186548_object19927:head# 2024-04-29T14:53:19.773 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d1c749:::benchmark_data_smithi194_186548_object17477:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d278de:::benchmark_data_smithi194_186548_object36130:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d2b9a5:::benchmark_data_smithi194_186548_object49711:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d3cb41:::benchmark_data_smithi194_186548_object53153:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d41d99:::benchmark_data_smithi194_186548_object50083:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d432d1:::benchmark_data_smithi194_186548_object59745:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d53d3a:::benchmark_data_smithi194_186548_object68018:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d61216:::benchmark_data_smithi194_186548_object60862:head# 2024-04-29T14:53:19.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d6277b:::benchmark_data_smithi194_186548_object22250:head# 2024-04-29T14:53:19.776 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d660d9:::benchmark_data_smithi194_186548_object44528:head# 2024-04-29T14:53:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d6ddf8:::benchmark_data_smithi194_186548_object47739:head# 2024-04-29T14:53:19.781 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d71749:::benchmark_data_smithi194_186548_object44532:head# 2024-04-29T14:53:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d7efb4:::benchmark_data_smithi194_186548_object22947:head# 2024-04-29T14:53:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d83133:::benchmark_data_smithi194_186548_object13875:head# 2024-04-29T14:53:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d8efb5:::benchmark_data_smithi194_186548_object50827:head# 2024-04-29T14:53:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d92b7b:::benchmark_data_smithi194_186548_object97:head# 2024-04-29T14:53:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d9a7e5:::benchmark_data_smithi194_186548_object31816:head# 2024-04-29T14:53:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30d9d8d3:::benchmark_data_smithi194_186548_object29137:head# 2024-04-29T14:53:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30dae327:::benchmark_data_smithi194_186548_object7598:head# 2024-04-29T14:53:19.784 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30db51de:::benchmark_data_smithi194_186548_object46186:head# 2024-04-29T14:53:19.785 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ddf9bc:::benchmark_data_smithi194_186548_object80386:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30df37b0:::benchmark_data_smithi194_186548_object60022:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30df761a:::benchmark_data_smithi194_186548_object20455:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30dfd8f2:::benchmark_data_smithi194_186548_object37746:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e02bd1:::benchmark_data_smithi194_186548_object6122:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e17f8d:::benchmark_data_smithi194_186548_object27566:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e1c61c:::benchmark_data_smithi194_186548_object35017:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e1cc69:::benchmark_data_smithi194_186548_object12986:head# 2024-04-29T14:53:19.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e21bd6:::benchmark_data_smithi194_186548_object12639:head# 2024-04-29T14:53:19.799 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e3bae8:::benchmark_data_smithi194_186548_object33255:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e3f64d:::benchmark_data_smithi194_186548_object3038:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e48e39:::benchmark_data_smithi194_186548_object51670:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e4c4bc:::benchmark_data_smithi194_186548_object3385:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e4d2cc:::benchmark_data_smithi194_186548_object46840:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e5d44b:::benchmark_data_smithi194_186548_object69729:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e65c29:::benchmark_data_smithi194_186548_object80427:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e729f8:::benchmark_data_smithi194_186548_object56965:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e74ef0:::benchmark_data_smithi194_186548_object76361:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e7d6e9:::benchmark_data_smithi194_186548_object49796:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e8cc48:::benchmark_data_smithi194_186548_object61044:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e92e6d:::benchmark_data_smithi194_186548_object54380:head# 2024-04-29T14:53:19.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30e9ec20:::benchmark_data_smithi194_186548_object34109:head# 2024-04-29T14:53:19.801 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30eb0dd7:::benchmark_data_smithi194_186548_object11369:head# 2024-04-29T14:53:19.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ed169d:::benchmark_data_smithi194_186548_object8547:head# 2024-04-29T14:53:19.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ed37bd:::benchmark_data_smithi194_186548_object8507:head# 2024-04-29T14:53:19.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ed547c:::benchmark_data_smithi194_186548_object9416:head# 2024-04-29T14:53:19.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30edf67a:::benchmark_data_smithi194_186548_object50312:head# 2024-04-29T14:53:19.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ee4146:::benchmark_data_smithi194_186548_object67236:head# 2024-04-29T14:53:19.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ee825f:::benchmark_data_smithi194_186548_object75330:head# 2024-04-29T14:53:19.807 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f02ed9:::benchmark_data_smithi194_186548_object56626:head# 2024-04-29T14:53:19.807 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f0df21:::benchmark_data_smithi194_186548_object76742:head# 2024-04-29T14:53:19.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f1299f:::benchmark_data_smithi194_186548_object25344:head# 2024-04-29T14:53:19.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f1ed91:::benchmark_data_smithi194_186548_object76027:head# 2024-04-29T14:53:19.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f25f36:::benchmark_data_smithi194_186548_object84977:head# 2024-04-29T14:53:19.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f2f1e8:::benchmark_data_smithi194_186548_object54034:head# 2024-04-29T14:53:19.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f42afa:::benchmark_data_smithi194_186548_object51970:head# 2024-04-29T14:53:19.810 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f48408:::benchmark_data_smithi194_186548_object52025:head# 2024-04-29T14:53:19.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f4d5d4:::benchmark_data_smithi194_186548_object58337:head# 2024-04-29T14:53:19.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f73fed:::benchmark_data_smithi194_186548_object36473:head# 2024-04-29T14:53:19.812 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f7aaa3:::benchmark_data_smithi194_186548_object12017:head# 2024-04-29T14:53:19.812 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f84ea5:::benchmark_data_smithi194_186548_object24033:head# 2024-04-29T14:53:19.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30f897fc:::benchmark_data_smithi194_186548_object61209:head# 2024-04-29T14:53:19.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30fa708d:::benchmark_data_smithi194_186548_object27301:head# 2024-04-29T14:53:19.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30fbc677:::benchmark_data_smithi194_186548_object9031:head# 2024-04-29T14:53:19.814 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30fcb96d:::benchmark_data_smithi194_186548_object2689:head# 2024-04-29T14:53:19.814 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30fd263f:::benchmark_data_smithi194_186548_object26595:head# 2024-04-29T14:53:19.815 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30fd3e01:::benchmark_data_smithi194_186548_object32932:head# 2024-04-29T14:53:19.817 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30fe41b1:::benchmark_data_smithi194_186548_object48636:head# 2024-04-29T14:53:19.818 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ff364d:::benchmark_data_smithi194_186548_object44492:head# 2024-04-29T14:53:19.821 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ff4010:::benchmark_data_smithi194_186548_object66194:head# 2024-04-29T14:53:19.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:30ff8e98:::benchmark_data_smithi194_186548_object71856:head# 2024-04-29T14:53:19.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31009bfe:::benchmark_data_smithi194_186548_object73274:head# 2024-04-29T14:53:19.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3100c41f:::benchmark_data_smithi194_186548_object64441:head# 2024-04-29T14:53:19.825 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3101a80e:::benchmark_data_smithi194_186548_object23938:head# 2024-04-29T14:53:19.825 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31029d07:::benchmark_data_smithi194_186548_object30287:head# 2024-04-29T14:53:19.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31035183:::benchmark_data_smithi194_186548_object27809:head# 2024-04-29T14:53:19.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31046d23:::benchmark_data_smithi194_186548_object84108:head# 2024-04-29T14:53:19.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3104b244:::benchmark_data_smithi194_186548_object51226:head# 2024-04-29T14:53:19.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31052056:::benchmark_data_smithi194_186548_object53397:head# 2024-04-29T14:53:19.833 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31056ccd:::benchmark_data_smithi194_186548_object48429:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31071b8c:::benchmark_data_smithi194_186548_object59600:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31072902:::benchmark_data_smithi194_186548_object27597:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31075fff:::benchmark_data_smithi194_186548_object72692:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3107a595:::benchmark_data_smithi194_186548_object19204:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3107ead5:::benchmark_data_smithi194_186548_object85331:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3108716a:::benchmark_data_smithi194_186548_object19734:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31088222:::benchmark_data_smithi194_186548_object8898:head# 2024-04-29T14:53:19.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3108d688:::benchmark_data_smithi194_186548_object61065:head# 2024-04-29T14:53:19.838 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31090591:::benchmark_data_smithi194_186548_object57836:head# 2024-04-29T14:53:19.840 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3109f285:::benchmark_data_smithi194_186548_object80654:head# 2024-04-29T14:53:19.840 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310a93b6:::benchmark_data_smithi194_186548_object24574:head# 2024-04-29T14:53:19.840 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310b0541:::benchmark_data_smithi194_186548_object16864:head# 2024-04-29T14:53:19.840 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310b07aa:::benchmark_data_smithi194_186548_object77083:head# 2024-04-29T14:53:19.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310b3413:::benchmark_data_smithi194_186548_object47902:head# 2024-04-29T14:53:19.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310d8780:::benchmark_data_smithi194_186548_object16672:head# 2024-04-29T14:53:19.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310e3a4b:::benchmark_data_smithi194_186548_object52503:head# 2024-04-29T14:53:19.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310f0d1d:::benchmark_data_smithi194_186548_object7177:head# 2024-04-29T14:53:19.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:310fe780:::benchmark_data_smithi194_186548_object77920:head# 2024-04-29T14:53:19.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311189fd:::benchmark_data_smithi194_186548_object65760:head# 2024-04-29T14:53:19.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3111d7ac:::benchmark_data_smithi194_186548_object43945:head# 2024-04-29T14:53:19.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31124791:::benchmark_data_smithi194_186548_object23424:head# 2024-04-29T14:53:19.844 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31128d7a:::benchmark_data_smithi194_186548_object35532:head# 2024-04-29T14:53:19.844 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311449e2:::benchmark_data_smithi194_186548_object37036:head# 2024-04-29T14:53:19.844 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311455eb:::benchmark_data_smithi194_186548_object26587:head# 2024-04-29T14:53:19.845 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3114f61b:::benchmark_data_smithi194_186548_object42950:head# 2024-04-29T14:53:19.850 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31165e35:::benchmark_data_smithi194_186548_object42600:head# 2024-04-29T14:53:19.850 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31169049:::benchmark_data_smithi194_186548_object25088:head# 2024-04-29T14:53:19.850 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31172c86:::benchmark_data_smithi194_186548_object44305:head# 2024-04-29T14:53:19.850 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31174f8e:::benchmark_data_smithi194_186548_object31217:head# 2024-04-29T14:53:19.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3118c2f8:::benchmark_data_smithi194_186548_object25901:head# 2024-04-29T14:53:19.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31190c06:::benchmark_data_smithi194_186548_object38663:head# 2024-04-29T14:53:19.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31199d44:::benchmark_data_smithi194_186548_object8183:head# 2024-04-29T14:53:19.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3119c8fc:::benchmark_data_smithi194_186548_object65014:head# 2024-04-29T14:53:19.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311a774d:::benchmark_data_smithi194_186548_object13415:head# 2024-04-29T14:53:19.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311ac079:::benchmark_data_smithi194_186548_object11218:head# 2024-04-29T14:53:19.853 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311b1e07:::benchmark_data_smithi194_186548_object38199:head# 2024-04-29T14:53:19.853 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311c4138:::benchmark_data_smithi194_186548_object36350:head# 2024-04-29T14:53:19.853 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311dd055:::benchmark_data_smithi194_186548_object50270:head# 2024-04-29T14:53:19.855 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311ddda7:::benchmark_data_smithi194_186548_object62206:head# 2024-04-29T14:53:19.855 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311e1544:::benchmark_data_smithi194_186548_object51723:head# 2024-04-29T14:53:19.855 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311e75b0:::benchmark_data_smithi194_186548_object64878:head# 2024-04-29T14:53:19.858 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311ea181:::benchmark_data_smithi194_186548_object35290:head# 2024-04-29T14:53:19.860 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311ea6fb:::benchmark_data_smithi194_186548_object76733:head# 2024-04-29T14:53:19.860 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311f4951:::benchmark_data_smithi194_186548_object83875:head# 2024-04-29T14:53:19.860 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:311f92c5:::benchmark_data_smithi194_186548_object19478:head# 2024-04-29T14:53:19.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31203251:::benchmark_data_smithi194_186548_object15804:head# 2024-04-29T14:53:19.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3121b0ed:::benchmark_data_smithi194_186548_object19468:head# 2024-04-29T14:53:19.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3121fcae:::benchmark_data_smithi194_186548_object57539:head# 2024-04-29T14:53:19.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312235d0:::benchmark_data_smithi194_186548_object67347:head# 2024-04-29T14:53:19.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312237c2:::benchmark_data_smithi194_186548_object56937:head# 2024-04-29T14:53:19.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312285b8:::benchmark_data_smithi194_186548_object63821:head# 2024-04-29T14:53:19.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31230706:::benchmark_data_smithi194_186548_object67586:head# 2024-04-29T14:53:19.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3123d9c5:::benchmark_data_smithi194_186548_object26556:head# 2024-04-29T14:53:19.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3125e007:::benchmark_data_smithi194_186548_object33600:head# 2024-04-29T14:53:19.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31262d88:::benchmark_data_smithi194_186548_object49659:head# 2024-04-29T14:53:19.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312821bc:::benchmark_data_smithi194_186548_object42718:head# 2024-04-29T14:53:19.867 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3128f8e1:::benchmark_data_smithi194_186548_object77141:head# 2024-04-29T14:53:19.867 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3128fe96:::benchmark_data_smithi194_186548_object40477:head# 2024-04-29T14:53:19.868 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312b84cb:::benchmark_data_smithi194_186548_object57579:head# 2024-04-29T14:53:19.868 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312d1acf:::benchmark_data_smithi194_186548_object10347:head# 2024-04-29T14:53:19.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312d1da4:::benchmark_data_smithi194_186548_object10956:head# 2024-04-29T14:53:19.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312e05b8:::benchmark_data_smithi194_186548_object48172:head# 2024-04-29T14:53:19.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312e2580:::benchmark_data_smithi194_186548_object22854:head# 2024-04-29T14:53:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:312eb1bf:::benchmark_data_smithi194_186548_object29495:head# 2024-04-29T14:53:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31309eaf:::benchmark_data_smithi194_186548_object49001:head# 2024-04-29T14:53:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3131f853:::benchmark_data_smithi194_186548_object33676:head# 2024-04-29T14:53:19.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3132558d:::benchmark_data_smithi194_186548_object80441:head# 2024-04-29T14:53:19.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313268ab:::benchmark_data_smithi194_186548_object80173:head# 2024-04-29T14:53:19.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313279cb:::benchmark_data_smithi194_186548_object2386:head# 2024-04-29T14:53:19.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313396e0:::benchmark_data_smithi194_186548_object24085:head# 2024-04-29T14:53:19.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31343ae4:::benchmark_data_smithi194_186548_object62459:head# 2024-04-29T14:53:19.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31346cc5:::benchmark_data_smithi194_186548_object13324:head# 2024-04-29T14:53:19.889 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31351c3a:::benchmark_data_smithi194_186548_object28320:head# 2024-04-29T14:53:19.890 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313524b5:::benchmark_data_smithi194_186548_object53066:head# 2024-04-29T14:53:19.890 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31353322:::benchmark_data_smithi194_186548_object32119:head# 2024-04-29T14:53:19.891 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313644ed:::benchmark_data_smithi194_186548_object74986:head# 2024-04-29T14:53:19.891 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313686a0:::benchmark_data_smithi194_186548_object68712:head# 2024-04-29T14:53:19.892 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3136fd17:::benchmark_data_smithi194_186548_object85028:head# 2024-04-29T14:53:19.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3138af48:::benchmark_data_smithi194_186548_object44338:head# 2024-04-29T14:53:19.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3138e6fd:::benchmark_data_smithi194_186548_object33912:head# 2024-04-29T14:53:19.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3139353a:::benchmark_data_smithi194_186548_object32661:head# 2024-04-29T14:53:19.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31397ecd:::benchmark_data_smithi194_186548_object83088:head# 2024-04-29T14:53:19.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3139a1fb:::benchmark_data_smithi194_186548_object76933:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313b2008:::benchmark_data_smithi194_186548_object48710:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313b952c:::benchmark_data_smithi194_186548_object23985:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313bccbe:::benchmark_data_smithi194_186548_object15597:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313bf906:::benchmark_data_smithi194_186548_object45895:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313c0f78:::benchmark_data_smithi194_186548_object75393:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313c7cb1:::benchmark_data_smithi194_186548_object81403:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313e30ae:::benchmark_data_smithi194_186548_object25687:head# 2024-04-29T14:53:19.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313e4de4:::benchmark_data_smithi194_186548_object64792:head# 2024-04-29T14:53:19.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313e5698:::benchmark_data_smithi194_186548_object6560:head# 2024-04-29T14:53:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313e67e8:::benchmark_data_smithi194_186548_object35513:head# 2024-04-29T14:53:19.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313ecde2:::benchmark_data_smithi194_186548_object33117:head# 2024-04-29T14:53:19.904 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:53:19.904 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313fd959:::benchmark_data_smithi194_186548_object7096:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:313fff53:::benchmark_data_smithi194_186548_object82871:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314028b9:::benchmark_data_smithi194_186548_object43300:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3140ff9d:::benchmark_data_smithi194_186548_object50415:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31414e57:::benchmark_data_smithi194_186548_object78778:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3141cde5:::benchmark_data_smithi194_186548_object15082:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314301ef:::benchmark_data_smithi194_186548_object46226:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3143f36d:::benchmark_data_smithi194_186548_object59034:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3145370a:::benchmark_data_smithi194_186548_object19709:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3145747f:::benchmark_data_smithi194_186548_object49833:head# 2024-04-29T14:53:19.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3145b48a:::benchmark_data_smithi194_186548_object18884:head# 2024-04-29T14:53:19.909 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31466651:::benchmark_data_smithi194_186548_object70618:head# 2024-04-29T14:53:19.909 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3146df33:::benchmark_data_smithi194_186548_object77743:head# 2024-04-29T14:53:19.909 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3147b845:::benchmark_data_smithi194_186548_object59646:head# 2024-04-29T14:53:19.909 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3147d96e:::benchmark_data_smithi194_186548_object43210:head# 2024-04-29T14:53:19.909 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3148eba7:::benchmark_data_smithi194_186548_object53687:head# 2024-04-29T14:53:19.910 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3148f816:::benchmark_data_smithi194_186548_object85271:head# 2024-04-29T14:53:19.910 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3149bb4b:::benchmark_data_smithi194_186548_object27402:head# 2024-04-29T14:53:19.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314b6c7f:::benchmark_data_smithi194_186548_object85070:head# 2024-04-29T14:53:19.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314b7885:::benchmark_data_smithi194_186548_object50819:head# 2024-04-29T14:53:19.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314ba73b:::benchmark_data_smithi194_186548_object26432:head# 2024-04-29T14:53:19.912 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314bf6fa:::benchmark_data_smithi194_186548_object16249:head# 2024-04-29T14:53:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314cb7ef:::benchmark_data_smithi194_186548_object12260:head# 2024-04-29T14:53:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314d971c:::benchmark_data_smithi194_186548_object73389:head# 2024-04-29T14:53:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314eabbc:::benchmark_data_smithi194_186548_object33501:head# 2024-04-29T14:53:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:314f28d6:::benchmark_data_smithi194_186548_object56314:head# 2024-04-29T14:53:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:315201e0:::benchmark_data_smithi194_186548_object10891:head# 2024-04-29T14:53:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:315361a5:::benchmark_data_smithi194_186548_object15774:head# 2024-04-29T14:53:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31536f17:::benchmark_data_smithi194_186548_object12722:head# 2024-04-29T14:53:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31547a26:::benchmark_data_smithi194_186548_object69741:head# 2024-04-29T14:53:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31554576:::benchmark_data_smithi194_186548_object17473:head# 2024-04-29T14:53:19.917 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3155bd15:::benchmark_data_smithi194_186548_object67671:head# 2024-04-29T14:53:19.917 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3156173e:::benchmark_data_smithi194_186548_object51373:head# 2024-04-29T14:53:19.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31567221:::benchmark_data_smithi194_186548_object57778:head# 2024-04-29T14:53:19.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31570137:::benchmark_data_smithi194_186548_object26986:head# 2024-04-29T14:53:19.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3157d8f0:::benchmark_data_smithi194_186548_object84713:head# 2024-04-29T14:53:19.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3158e383:::benchmark_data_smithi194_186548_object75613:head# 2024-04-29T14:53:19.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3158f80f:::benchmark_data_smithi194_186548_object24621:head# 2024-04-29T14:53:19.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31591f2b:::benchmark_data_smithi194_186548_object82540:head# 2024-04-29T14:53:19.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31596746:::benchmark_data_smithi194_186548_object71863:head# 2024-04-29T14:53:19.929 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31598846:::benchmark_data_smithi194_186548_object40205:head# 2024-04-29T14:53:19.929 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:315a9576:::benchmark_data_smithi194_186548_object26863:head# 2024-04-29T14:53:19.931 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:315b6755:::benchmark_data_smithi194_186548_object39898:head# 2024-04-29T14:53:19.932 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:315d84ed:::benchmark_data_smithi194_186548_object50934:head# 2024-04-29T14:53:19.932 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316048d0:::benchmark_data_smithi194_186548_object81002:head# 2024-04-29T14:53:19.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31606f7e:::benchmark_data_smithi194_186548_object56659:head# 2024-04-29T14:53:19.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31622739:::benchmark_data_smithi194_186548_object24578:head# 2024-04-29T14:53:19.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316307f2:::benchmark_data_smithi194_186548_object84820:head# 2024-04-29T14:53:19.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31630b26:::benchmark_data_smithi194_186548_object14321:head# 2024-04-29T14:53:19.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316529fb:::benchmark_data_smithi194_186548_object3796:head# 2024-04-29T14:53:19.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316657f5:::benchmark_data_smithi194_186548_object50749:head# 2024-04-29T14:53:19.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31669b20:::benchmark_data_smithi194_186548_object63905:head# 2024-04-29T14:53:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3166becf:::benchmark_data_smithi194_186548_object314:head# 2024-04-29T14:53:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3167f91f:::benchmark_data_smithi194_186548_object28507:head# 2024-04-29T14:53:19.940 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3168a048:::benchmark_data_smithi194_186548_object17271:head# 2024-04-29T14:53:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3168f877:::benchmark_data_smithi194_186548_object37585:head# 2024-04-29T14:53:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3169880c:::benchmark_data_smithi194_186548_object61554:head# 2024-04-29T14:53:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31699a2e:::benchmark_data_smithi194_186548_object73949:head# 2024-04-29T14:53:19.942 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316d620b:::benchmark_data_smithi194_186548_object17585:head# 2024-04-29T14:53:19.942 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316d7ef9:::benchmark_data_smithi194_186548_object78802:head# 2024-04-29T14:53:19.942 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316f71dd:::benchmark_data_smithi194_186548_object1101:head# 2024-04-29T14:53:19.946 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316f9236:::benchmark_data_smithi194_186548_object4638:head# 2024-04-29T14:53:19.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:316fb2fb:::benchmark_data_smithi194_186548_object64111:head# 2024-04-29T14:53:19.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317015a2:::benchmark_data_smithi194_186548_object35081:head# 2024-04-29T14:53:19.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3170e6db:::benchmark_data_smithi194_186548_object16604:head# 2024-04-29T14:53:19.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3172c53d:::benchmark_data_smithi194_186548_object62400:head# 2024-04-29T14:53:19.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3172d8ec:::benchmark_data_smithi194_186548_object19741:head# 2024-04-29T14:53:19.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3173504a:::benchmark_data_smithi194_186548_object69221:head# 2024-04-29T14:53:19.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31737999:::benchmark_data_smithi194_186548_object1199:head# 2024-04-29T14:53:19.951 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3173fc24:::benchmark_data_smithi194_186548_object51253:head# 2024-04-29T14:53:19.951 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31745eab:::benchmark_data_smithi194_186548_object64862:head# 2024-04-29T14:53:19.952 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3174af1e:::benchmark_data_smithi194_186548_object27273:head# 2024-04-29T14:53:19.952 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31750b75:::benchmark_data_smithi194_186548_object54588:head# 2024-04-29T14:53:19.952 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317604cc:::benchmark_data_smithi194_186548_object16027:head# 2024-04-29T14:53:19.953 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31764866:::benchmark_data_smithi194_186548_object2212:head# 2024-04-29T14:53:19.954 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31776391:::benchmark_data_smithi194_186548_object69894:head# 2024-04-29T14:53:19.955 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31778e39:::benchmark_data_smithi194_186548_object702:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3178174a:::benchmark_data_smithi194_186548_object33860:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31781f9e:::benchmark_data_smithi194_186548_object84915:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317826cb:::benchmark_data_smithi194_186548_object56505:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317b53fe:::benchmark_data_smithi194_186548_object16192:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317bb952:::benchmark_data_smithi194_186548_object74617:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317bd106:::benchmark_data_smithi194_186548_object35701:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317cdfc4:::benchmark_data_smithi194_186548_object48368:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317f8719:::benchmark_data_smithi194_186548_object39530:head# 2024-04-29T14:53:19.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:317fe2fa:::benchmark_data_smithi194_186548_object60876:head# 2024-04-29T14:53:19.960 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31800e09:::benchmark_data_smithi194_186548_object52437:head# 2024-04-29T14:53:19.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3180f8a5:::benchmark_data_smithi194_186548_object13397:head# 2024-04-29T14:53:19.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318151ee:::benchmark_data_smithi194_186548_object61349:head# 2024-04-29T14:53:19.965 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31815a11:::benchmark_data_smithi194_186548_object66615:head# 2024-04-29T14:53:19.967 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3181daed:::benchmark_data_smithi194_186548_object9820:head# 2024-04-29T14:53:19.967 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318236c4:::benchmark_data_smithi194_186548_object56140:head# 2024-04-29T14:53:19.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3182cd79:::benchmark_data_smithi194_186548_object77680:head# 2024-04-29T14:53:19.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31832af7:::benchmark_data_smithi194_186548_object9504:head# 2024-04-29T14:53:19.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3183cbd1:::benchmark_data_smithi194_186548_object1710:head# 2024-04-29T14:53:19.969 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3183dbdd:::benchmark_data_smithi194_186548_object12521:head# 2024-04-29T14:53:19.969 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31843183:::benchmark_data_smithi194_186548_object35865:head# 2024-04-29T14:53:19.971 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31849477:::benchmark_data_smithi194_186548_object15961:head# 2024-04-29T14:53:19.974 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3184aa8b:::benchmark_data_smithi194_186548_object60435:head# 2024-04-29T14:53:19.974 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31859b1a:::benchmark_data_smithi194_186548_object40491:head# 2024-04-29T14:53:19.975 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3185af81:::benchmark_data_smithi194_186548_object54079:head# 2024-04-29T14:53:19.978 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3186dba0:::benchmark_data_smithi194_186548_object14819:head# 2024-04-29T14:53:19.978 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31886db0:::benchmark_data_smithi194_186548_object30267:head# 2024-04-29T14:53:19.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31893905:::benchmark_data_smithi194_186548_object9250:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3189a5d8:::benchmark_data_smithi194_186548_object17112:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3189fbaf:::benchmark_data_smithi194_186548_object21392:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318a4c14:::benchmark_data_smithi194_186548_object4746:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318c9962:::benchmark_data_smithi194_186548_object58572:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318d8a96:::benchmark_data_smithi194_186548_object12390:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318ed984:::benchmark_data_smithi194_186548_object75942:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318f67bf:::benchmark_data_smithi194_186548_object27866:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:318f79be:::benchmark_data_smithi194_186548_object19392:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3190ef32:::benchmark_data_smithi194_186548_object54674:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31915615:::benchmark_data_smithi194_186548_object22614:head# 2024-04-29T14:53:19.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31926c6c:::benchmark_data_smithi194_186548_object39719:head# 2024-04-29T14:53:19.985 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3192d26c:::benchmark_data_smithi194_186548_object64492:head# 2024-04-29T14:53:19.985 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31931294:::benchmark_data_smithi194_186548_object33302:head# 2024-04-29T14:53:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3193914a:::benchmark_data_smithi194_186548_object53369:head# 2024-04-29T14:53:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31952a16:::benchmark_data_smithi194_186548_object51437:head# 2024-04-29T14:53:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3195a51b:::benchmark_data_smithi194_186548_object52586:head# 2024-04-29T14:53:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3195b4b8:::benchmark_data_smithi194_186548_object37903:head# 2024-04-29T14:53:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3195b8c3:::benchmark_data_smithi194_186548_object1447:head# 2024-04-29T14:53:19.990 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31967861:::benchmark_data_smithi194_186548_object4723:head# 2024-04-29T14:53:19.990 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3198748e:::benchmark_data_smithi194_186548_object60924:head# 2024-04-29T14:53:19.990 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31989de9:::benchmark_data_smithi194_186548_object16004:head# 2024-04-29T14:53:19.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3199493a:::benchmark_data_smithi194_186548_object38817:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3199e61b:::benchmark_data_smithi194_186548_object54781:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:319a0694:::benchmark_data_smithi194_186548_object59186:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:319abe2e:::benchmark_data_smithi194_186548_object21309:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:319bc8ea:::benchmark_data_smithi194_186548_object13852:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:319e7f09:::benchmark_data_smithi194_186548_object73096:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:319ed150:::benchmark_data_smithi194_186548_object53047:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:319f5ca0:::benchmark_data_smithi194_186548_object67356:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:319f784a:::benchmark_data_smithi194_186548_object63083:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a04e80:::benchmark_data_smithi194_186548_object36777:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a25c80:::benchmark_data_smithi194_186548_object51917:head# 2024-04-29T14:53:19.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a29a44:::benchmark_data_smithi194_186548_object44396:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a2ad23:::benchmark_data_smithi194_186548_object20322:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a2eefb:::benchmark_data_smithi194_186548_object35418:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a41ec8:::benchmark_data_smithi194_186548_object17313:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a69792:::benchmark_data_smithi194_186548_object54394:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a6d2f9:::benchmark_data_smithi194_186548_object47679:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a7b970:::benchmark_data_smithi194_186548_object2135:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a9344e:::benchmark_data_smithi194_186548_object22865:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a9c68c:::benchmark_data_smithi194_186548_object8644:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31a9fbdd:::benchmark_data_smithi194_186548_object55751:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31aa5bf8:::benchmark_data_smithi194_186548_object49091:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31aa72be:::benchmark_data_smithi194_186548_object20134:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ab09cc:::benchmark_data_smithi194_186548_object54151:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31abc427:::benchmark_data_smithi194_186548_object10959:head# 2024-04-29T14:53:20.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ae1cf8:::benchmark_data_smithi194_186548_object58842:head# 2024-04-29T14:53:20.003 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31aed153:::benchmark_data_smithi194_186548_object61665:head# 2024-04-29T14:53:20.003 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31af6aa3:::benchmark_data_smithi194_186548_object3559:head# 2024-04-29T14:53:20.003 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31afa861:::benchmark_data_smithi194_186548_object31654:head# 2024-04-29T14:53:20.003 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b0c370:::benchmark_data_smithi194_186548_object11898:head# 2024-04-29T14:53:20.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b0e772:::benchmark_data_smithi194_186548_object58290:head# 2024-04-29T14:53:20.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b274f2:::benchmark_data_smithi194_186548_object28311:head# 2024-04-29T14:53:20.005 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b374f4:::benchmark_data_smithi194_186548_object13373:head# 2024-04-29T14:53:20.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b3c1e3:::benchmark_data_smithi194_186548_object48621:head# 2024-04-29T14:53:20.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b40d9d:::benchmark_data_smithi194_186548_object25367:head# 2024-04-29T14:53:20.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b46325:::benchmark_data_smithi194_186548_object11109:head# 2024-04-29T14:53:20.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b4638b:::benchmark_data_smithi194_186548_object7950:head# 2024-04-29T14:53:20.010 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b4a810:::benchmark_data_smithi194_186548_object57357:head# 2024-04-29T14:53:20.010 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b4b4b2:::benchmark_data_smithi194_186548_object36672:head# 2024-04-29T14:53:20.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b4c166:::benchmark_data_smithi194_186548_object39776:head# 2024-04-29T14:53:20.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b52f7a:::benchmark_data_smithi194_186548_object68946:head# 2024-04-29T14:53:20.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b575b4:::benchmark_data_smithi194_186548_object46309:head# 2024-04-29T14:53:20.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b697c7:::benchmark_data_smithi194_186548_object78601:head# 2024-04-29T14:53:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b6e2ef:::benchmark_data_smithi194_186548_object84795:head# 2024-04-29T14:53:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b6f5e7:::benchmark_data_smithi194_186548_object108:head# 2024-04-29T14:53:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b7128c:::benchmark_data_smithi194_186548_object12361:head# 2024-04-29T14:53:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b73e1f:::benchmark_data_smithi194_186548_object70427:head# 2024-04-29T14:53:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b74f06:::benchmark_data_smithi194_186548_object47692:head# 2024-04-29T14:53:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b8a434:::benchmark_data_smithi194_186548_object6633:head# 2024-04-29T14:53:20.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b8c182:::benchmark_data_smithi194_186548_object42859:head# 2024-04-29T14:53:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b90ccf:::benchmark_data_smithi194_186548_object43604:head# 2024-04-29T14:53:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31b9ffa3:::benchmark_data_smithi194_186548_object33377:head# 2024-04-29T14:53:20.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ba3202:::benchmark_data_smithi194_186548_object18:head# 2024-04-29T14:53:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bb4f5d:::benchmark_data_smithi194_186548_object252:head# 2024-04-29T14:53:20.017 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bb50ba:::benchmark_data_smithi194_186548_object83682:head# 2024-04-29T14:53:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bb9303:::benchmark_data_smithi194_186548_object29081:head# 2024-04-29T14:53:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bbb8bf:::benchmark_data_smithi194_186548_object52085:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bbbd29:::benchmark_data_smithi194_186548_object71395:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bc213c:::benchmark_data_smithi194_186548_object41348:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bc32ec:::benchmark_data_smithi194_186548_object66257:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bc9ebf:::benchmark_data_smithi194_186548_object29663:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bd14ff:::benchmark_data_smithi194_186548_object30043:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bdba95:::benchmark_data_smithi194_186548_object41049:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31beda00:::benchmark_data_smithi194_186548_object5745:head# 2024-04-29T14:53:20.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bfc24b:::benchmark_data_smithi194_186548_object813:head# 2024-04-29T14:53:20.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31bfdd27:::benchmark_data_smithi194_186548_object79099:head# 2024-04-29T14:53:20.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31c0da67:::benchmark_data_smithi194_186548_object8438:head# 2024-04-29T14:53:20.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31c24bc0:::benchmark_data_smithi194_186548_object49321:head# 2024-04-29T14:53:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31c2f1c6:::benchmark_data_smithi194_186548_object2206:head# 2024-04-29T14:53:20.026 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31c3f14f:::benchmark_data_smithi194_186548_object77389:head# 2024-04-29T14:53:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31c56bc2:::benchmark_data_smithi194_186548_object68651:head# 2024-04-29T14:53:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31c66334:::benchmark_data_smithi194_186548_object13374:head# 2024-04-29T14:53:20.027 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31c8a899:::benchmark_data_smithi194_186548_object45223:head# 2024-04-29T14:53:20.028 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ca3c7e:::benchmark_data_smithi194_186548_object27234:head# 2024-04-29T14:53:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31cbb57d:::benchmark_data_smithi194_186548_object79711:head# 2024-04-29T14:53:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31cc6e7e:::benchmark_data_smithi194_186548_object26951:head# 2024-04-29T14:53:20.031 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ce02c0:::benchmark_data_smithi194_186548_object68276:head# 2024-04-29T14:53:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ce0400:::benchmark_data_smithi194_186548_object42304:head# 2024-04-29T14:53:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ceb25d:::benchmark_data_smithi194_186548_object46348:head# 2024-04-29T14:53:20.040 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31cf0dcc:::benchmark_data_smithi194_186548_object23528:head# 2024-04-29T14:53:20.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31cf81e2:::benchmark_data_smithi194_186548_object37755:head# 2024-04-29T14:53:20.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d18746:::benchmark_data_smithi194_186548_object20439:head# 2024-04-29T14:53:20.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d22cf8:::benchmark_data_smithi194_186548_object25487:head# 2024-04-29T14:53:20.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d368ce:::benchmark_data_smithi194_186548_object45707:head# 2024-04-29T14:53:20.042 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d37a42:::benchmark_data_smithi194_186548_object11905:head# 2024-04-29T14:53:20.042 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d3a24e:::benchmark_data_smithi194_186548_object54552:head# 2024-04-29T14:53:20.043 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d66c99:::benchmark_data_smithi194_186548_object7629:head# 2024-04-29T14:53:20.043 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d78787:::benchmark_data_smithi194_186548_object9551:head# 2024-04-29T14:53:20.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d8e80d:::benchmark_data_smithi194_186548_object29294:head# 2024-04-29T14:53:20.049 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31d8fadb:::benchmark_data_smithi194_186548_object84701:head# 2024-04-29T14:53:20.049 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31da180e:::benchmark_data_smithi194_186548_object80480:head# 2024-04-29T14:53:20.050 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31da18bf:::benchmark_data_smithi194_186548_object21718:head# 2024-04-29T14:53:20.053 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31db3503:::benchmark_data_smithi194_186548_object48775:head# 2024-04-29T14:53:20.053 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31db8de6:::benchmark_data_smithi194_186548_object57210:head# 2024-04-29T14:53:20.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31dc8028:::benchmark_data_smithi194_186548_object72016:head# 2024-04-29T14:53:20.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31dd3c16:::benchmark_data_smithi194_186548_object79773:head# 2024-04-29T14:53:20.055 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31df7bc3:::benchmark_data_smithi194_186548_object688:head# 2024-04-29T14:53:20.055 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31dfc311:::benchmark_data_smithi194_186548_object39027:head# 2024-04-29T14:53:20.058 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31dfe46f:::benchmark_data_smithi194_186548_object9386:head# 2024-04-29T14:53:20.059 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31dfeb0d:::benchmark_data_smithi194_186548_object19551:head# 2024-04-29T14:53:20.059 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e0c7ef:::benchmark_data_smithi194_186548_object20232:head# 2024-04-29T14:53:20.059 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e1a2d5:::benchmark_data_smithi194_186548_object61116:head# 2024-04-29T14:53:20.059 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e291d0:::benchmark_data_smithi194_186548_object61356:head# 2024-04-29T14:53:20.060 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e531b5:::benchmark_data_smithi194_186548_object23060:head# 2024-04-29T14:53:20.060 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e69d70:::benchmark_data_smithi194_186548_object17831:head# 2024-04-29T14:53:20.060 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e6d39c:::benchmark_data_smithi194_186548_object42554:head# 2024-04-29T14:53:20.062 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e6f228:::benchmark_data_smithi194_186548_object74675:head# 2024-04-29T14:53:20.063 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e74480:::benchmark_data_smithi194_186548_object83695:head# 2024-04-29T14:53:20.063 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e7a00b:::benchmark_data_smithi194_186548_object27912:head# 2024-04-29T14:53:20.066 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31e866db:::benchmark_data_smithi194_186548_object59670:head# 2024-04-29T14:53:20.067 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ea44ca:::benchmark_data_smithi194_186548_object65644:head# 2024-04-29T14:53:20.067 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31eb8fba:::benchmark_data_smithi194_186548_object9339:head# 2024-04-29T14:53:20.067 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ec2743:::benchmark_data_smithi194_186548_object5279:head# 2024-04-29T14:53:20.067 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ec3b28:::benchmark_data_smithi194_186548_object13617:head# 2024-04-29T14:53:20.067 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31ed9c7e:::benchmark_data_smithi194_186548_object30618:head# 2024-04-29T14:53:20.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31eec82c:::benchmark_data_smithi194_186548_object62597:head# 2024-04-29T14:53:20.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31efed0d:::benchmark_data_smithi194_186548_object79327:head# 2024-04-29T14:53:20.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f12101:::benchmark_data_smithi194_186548_object50202:head# 2024-04-29T14:53:20.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f39b31:::benchmark_data_smithi194_186548_object61101:head# 2024-04-29T14:53:20.069 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f43b79:::benchmark_data_smithi194_186548_object85194:head# 2024-04-29T14:53:20.069 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f49b4c:::benchmark_data_smithi194_186548_object70684:head# 2024-04-29T14:53:20.070 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f510e7:::benchmark_data_smithi194_186548_object18348:head# 2024-04-29T14:53:20.072 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f51e66:::benchmark_data_smithi194_186548_object39481:head# 2024-04-29T14:53:20.073 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f55c02:::benchmark_data_smithi194_186548_object58885:head# 2024-04-29T14:53:20.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f6670b:::benchmark_data_smithi194_186548_object46174:head# 2024-04-29T14:53:20.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f85332:::benchmark_data_smithi194_186548_object57264:head# 2024-04-29T14:53:20.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f946a4:::benchmark_data_smithi194_186548_object55208:head# 2024-04-29T14:53:20.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f96d64:::benchmark_data_smithi194_186548_object23454:head# 2024-04-29T14:53:20.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31f9cfd8:::benchmark_data_smithi194_186548_object49853:head# 2024-04-29T14:53:20.077 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fa1563:::benchmark_data_smithi194_186548_object73615:head# 2024-04-29T14:53:20.077 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fa8a6c:::benchmark_data_smithi194_186548_object47382:head# 2024-04-29T14:53:20.077 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fb22c2:::benchmark_data_smithi194_186548_object11476:head# 2024-04-29T14:53:20.082 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fb891f:::benchmark_data_smithi194_186548_object4741:head# 2024-04-29T14:53:20.082 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fc3175:::benchmark_data_smithi194_186548_object6189:head# 2024-04-29T14:53:20.082 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fcb5f1:::benchmark_data_smithi194_186548_object45183:head# 2024-04-29T14:53:20.084 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fdec56:::benchmark_data_smithi194_186548_object56318:head# 2024-04-29T14:53:20.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:31fe818b:::benchmark_data_smithi194_186548_object11805:head# 2024-04-29T14:53:20.086 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32017fcf:::benchmark_data_smithi194_186548_object67277:head# 2024-04-29T14:53:20.086 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32019cf7:::benchmark_data_smithi194_186548_object1658:head# 2024-04-29T14:53:20.087 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3203d4cb:::benchmark_data_smithi194_186548_object11909:head# 2024-04-29T14:53:20.087 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3205fd42:::benchmark_data_smithi194_186548_object33388:head# 2024-04-29T14:53:20.087 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3206a1ba:::benchmark_data_smithi194_186548_object77155:head# 2024-04-29T14:53:20.088 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3206add7:::benchmark_data_smithi194_186548_object1243:head# 2024-04-29T14:53:20.088 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3207023e:::benchmark_data_smithi194_186548_object67361:head# 2024-04-29T14:53:20.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32077440:::benchmark_data_smithi194_186548_object19405:head# 2024-04-29T14:53:20.093 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32088c31:::benchmark_data_smithi194_186548_object47195:head# 2024-04-29T14:53:20.093 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3208d036:::benchmark_data_smithi194_186548_object69646:head# 2024-04-29T14:53:20.094 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:320c44fb:::benchmark_data_smithi194_186548_object44591:head# 2024-04-29T14:53:20.094 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:320c5fe9:::benchmark_data_smithi194_186548_object280:head# 2024-04-29T14:53:20.094 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:320f1127:::benchmark_data_smithi194_186548_object7445:head# 2024-04-29T14:53:20.094 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32123498:::benchmark_data_smithi194_186548_object62008:head# 2024-04-29T14:53:20.094 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321348e1:::benchmark_data_smithi194_186548_object2171:head# 2024-04-29T14:53:20.094 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321358d7:::benchmark_data_smithi194_186548_object15839:head# 2024-04-29T14:53:20.094 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3213bc3c:::benchmark_data_smithi194_186548_object15784:head# 2024-04-29T14:53:20.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32148a5a:::benchmark_data_smithi194_186548_object72191:head# 2024-04-29T14:53:20.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32149622:::benchmark_data_smithi194_186548_object81345:head# 2024-04-29T14:53:20.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32158c5f:::benchmark_data_smithi194_186548_object3089:head# 2024-04-29T14:53:20.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3215b250:::benchmark_data_smithi194_186548_object23755:head# 2024-04-29T14:53:20.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321677bb:::benchmark_data_smithi194_186548_object73765:head# 2024-04-29T14:53:20.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321702a5:::benchmark_data_smithi194_186548_object78321:head# 2024-04-29T14:53:20.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32180906:::benchmark_data_smithi194_186548_object6053:head# 2024-04-29T14:53:20.097 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3218b174:::benchmark_data_smithi194_186548_object63886:head# 2024-04-29T14:53:20.097 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3218d4da:::benchmark_data_smithi194_186548_object30536:head# 2024-04-29T14:53:20.098 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32195192:::benchmark_data_smithi194_186548_object34901:head# 2024-04-29T14:53:20.098 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321ad3f5:::benchmark_data_smithi194_186548_object49942:head# 2024-04-29T14:53:20.098 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321be706:::benchmark_data_smithi194_186548_object26155:head# 2024-04-29T14:53:20.099 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321c5af9:::benchmark_data_smithi194_186548_object18911:head# 2024-04-29T14:53:20.100 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321c7aeb:::benchmark_data_smithi194_186548_object81964:head# 2024-04-29T14:53:20.100 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321e4d84:::benchmark_data_smithi194_186548_object27130:head# 2024-04-29T14:53:20.100 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:321f6ba0:::benchmark_data_smithi194_186548_object76812:head# 2024-04-29T14:53:20.101 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32203703:::benchmark_data_smithi194_186548_object7772:head# 2024-04-29T14:53:20.101 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32204cee:::benchmark_data_smithi194_186548_object17316:head# 2024-04-29T14:53:20.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3220a1a4:::benchmark_data_smithi194_186548_object82828:head# 2024-04-29T14:53:20.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3220a54f:::benchmark_data_smithi194_186548_object38412:head# 2024-04-29T14:53:20.105 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32214f12:::benchmark_data_smithi194_186548_object83126:head# 2024-04-29T14:53:20.105 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32219561:::benchmark_data_smithi194_186548_object61310:head# 2024-04-29T14:53:20.105 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3221faa2:::benchmark_data_smithi194_186548_object40868:head# 2024-04-29T14:53:20.105 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3222090d:::benchmark_data_smithi194_186548_object22655:head# 2024-04-29T14:53:20.105 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322534a9:::benchmark_data_smithi194_186548_object56015:head# 2024-04-29T14:53:20.105 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32258c39:::benchmark_data_smithi194_186548_object26746:head# 2024-04-29T14:53:20.106 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3225a784:::benchmark_data_smithi194_186548_object20809:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3225f4d2:::benchmark_data_smithi194_186548_object50382:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32274b45:::benchmark_data_smithi194_186548_object84907:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3227d933:::benchmark_data_smithi194_186548_object34086:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3227df34:::benchmark_data_smithi194_186548_object62416:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322853dd:::benchmark_data_smithi194_186548_object31029:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322887b3:::benchmark_data_smithi194_186548_object13099:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322b7df2:::benchmark_data_smithi194_186548_object6583:head# 2024-04-29T14:53:20.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322b7ec4:::benchmark_data_smithi194_186548_object7667:head# 2024-04-29T14:53:20.112 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322be446:::benchmark_data_smithi194_186548_object10671:head# 2024-04-29T14:53:20.112 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322d31d7:::benchmark_data_smithi194_186548_object12269:head# 2024-04-29T14:53:20.112 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322d3f17:::benchmark_data_smithi194_186548_object43497:head# 2024-04-29T14:53:20.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322d6738:::benchmark_data_smithi194_186548_object27733:head# 2024-04-29T14:53:20.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322e41f4:::benchmark_data_smithi194_186548_object10553:head# 2024-04-29T14:53:20.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322e857b:::benchmark_data_smithi194_186548_object49717:head# 2024-04-29T14:53:20.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:322ee828:::benchmark_data_smithi194_186548_object50388:head# 2024-04-29T14:53:20.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32308b96:::benchmark_data_smithi194_186548_object41851:head# 2024-04-29T14:53:20.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3231083c:::benchmark_data_smithi194_186548_object43687:head# 2024-04-29T14:53:20.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3231272f:::benchmark_data_smithi194_186548_object29422:head# 2024-04-29T14:53:20.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32317e9d:::benchmark_data_smithi194_186548_object66116:head# 2024-04-29T14:53:20.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32328041:::benchmark_data_smithi194_186548_object36840:head# 2024-04-29T14:53:20.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3232c2ad:::benchmark_data_smithi194_186548_object271:head# 2024-04-29T14:53:20.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323328a0:::benchmark_data_smithi194_186548_object12799:head# 2024-04-29T14:53:20.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323356d8:::benchmark_data_smithi194_186548_object10389:head# 2024-04-29T14:53:20.117 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3233bd6c:::benchmark_data_smithi194_186548_object75138:head# 2024-04-29T14:53:20.117 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32345ae9:::benchmark_data_smithi194_186548_object17649:head# 2024-04-29T14:53:20.117 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32346947:::benchmark_data_smithi194_186548_object2887:head# 2024-04-29T14:53:20.118 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323479b6:::benchmark_data_smithi194_186548_object62059:head# 2024-04-29T14:53:20.119 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3234c1e3:::benchmark_data_smithi194_186548_object37622:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3236675e:::benchmark_data_smithi194_186548_object11634:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3236bb80:::benchmark_data_smithi194_186548_object53946:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32397dea:::benchmark_data_smithi194_186548_object18101:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323a744a:::benchmark_data_smithi194_186548_object19813:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323b9c3f:::benchmark_data_smithi194_186548_object27118:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323bd106:::benchmark_data_smithi194_186548_object44393:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323ca97e:::benchmark_data_smithi194_186548_object74861:head# 2024-04-29T14:53:20.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323ceb4f:::benchmark_data_smithi194_186548_object14517:head# 2024-04-29T14:53:20.124 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323d7665:::benchmark_data_smithi194_186548_object10803:head# 2024-04-29T14:53:20.124 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323efe60:::benchmark_data_smithi194_186548_object82818:head# 2024-04-29T14:53:20.124 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:323f845a:::benchmark_data_smithi194_186548_object45165:head# 2024-04-29T14:53:20.124 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:324070d7:::benchmark_data_smithi194_186548_object17951:head# 2024-04-29T14:53:20.125 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3241b70f:::benchmark_data_smithi194_186548_object66399:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3241e757:::benchmark_data_smithi194_186548_object65254:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32430610:::benchmark_data_smithi194_186548_object25426:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32433473:::benchmark_data_smithi194_186548_object78183:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32447614:::benchmark_data_smithi194_186548_object56926:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3244debc:::benchmark_data_smithi194_186548_object65892:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32472c52:::benchmark_data_smithi194_186548_object15131:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32477eab:::benchmark_data_smithi194_186548_object41762:head# 2024-04-29T14:53:20.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:324793c2:::benchmark_data_smithi194_186548_object78120:head# 2024-04-29T14:53:20.130 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3247b82d:::benchmark_data_smithi194_186548_object32002:head# 2024-04-29T14:53:20.133 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3248e8f6:::benchmark_data_smithi194_186548_object60317:head# 2024-04-29T14:53:20.143 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3249be59:::benchmark_data_smithi194_186548_object65347:head# 2024-04-29T14:53:20.145 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3249d072:::benchmark_data_smithi194_186548_object11891:head# 2024-04-29T14:53:20.146 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3249e863:::benchmark_data_smithi194_186548_object41316:head# 2024-04-29T14:53:20.146 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:324bb2af:::benchmark_data_smithi194_186548_object83364:head# 2024-04-29T14:53:20.147 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:324cdde5:::benchmark_data_smithi194_186548_object42778:head# 2024-04-29T14:53:20.150 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:324e327c:::benchmark_data_smithi194_186548_object39392:head# 2024-04-29T14:53:20.150 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:324f5b25:::benchmark_data_smithi194_186548_object26:head# 2024-04-29T14:53:20.150 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:324fa25d:::benchmark_data_smithi194_186548_object58042:head# 2024-04-29T14:53:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32500dec:::benchmark_data_smithi194_186548_object84125:head# 2024-04-29T14:53:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32510592:::benchmark_data_smithi194_186548_object47853:head# 2024-04-29T14:53:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32513571:::benchmark_data_smithi194_186548_object69321:head# 2024-04-29T14:53:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32518ebb:::benchmark_data_smithi194_186548_object19665:head# 2024-04-29T14:53:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3251c0a5:::benchmark_data_smithi194_186548_object13901:head# 2024-04-29T14:53:20.156 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32522b98:::benchmark_data_smithi194_186548_object16702:head# 2024-04-29T14:53:20.156 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32525b2a:::benchmark_data_smithi194_186548_object33861:head# 2024-04-29T14:53:20.156 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3254c1b4:::benchmark_data_smithi194_186548_object80866:head# 2024-04-29T14:53:20.156 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3255792d:::benchmark_data_smithi194_186548_object44990:head# 2024-04-29T14:53:20.156 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32562144:::benchmark_data_smithi194_186548_object63523:head# 2024-04-29T14:53:20.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32573830:::benchmark_data_smithi194_186548_object71761:head# 2024-04-29T14:53:20.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325743ed:::benchmark_data_smithi194_186548_object60364:head# 2024-04-29T14:53:20.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3257d712:::benchmark_data_smithi194_186548_object25493:head# 2024-04-29T14:53:20.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325aa194:::benchmark_data_smithi194_186548_object36753:head# 2024-04-29T14:53:20.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325b2ae0:::benchmark_data_smithi194_186548_object51048:head# 2024-04-29T14:53:20.159 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325b700f:::benchmark_data_smithi194_186548_object27228:head# 2024-04-29T14:53:20.159 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325c4e6d:::benchmark_data_smithi194_186548_object53007:head# 2024-04-29T14:53:20.159 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325ce492:::benchmark_data_smithi194_186548_object22376:head# 2024-04-29T14:53:20.160 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325d6763:::benchmark_data_smithi194_186548_object2812:head# 2024-04-29T14:53:20.160 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:325f0065:::benchmark_data_smithi194_186548_object19839:head# 2024-04-29T14:53:20.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32612b5b:::benchmark_data_smithi194_186548_object26142:head# 2024-04-29T14:53:20.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3261ffd1:::benchmark_data_smithi194_186548_object16061:head# 2024-04-29T14:53:20.162 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3262364c:::benchmark_data_smithi194_186548_object66550:head# 2024-04-29T14:53:20.162 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3262ba5e:::benchmark_data_smithi194_186548_object65800:head# 2024-04-29T14:53:20.163 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32641201:::benchmark_data_smithi194_186548_object21990:head# 2024-04-29T14:53:20.163 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32645ac2:::benchmark_data_smithi194_186548_object74581:head# 2024-04-29T14:53:20.164 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32651e25:::benchmark_data_smithi194_186548_object73944:head# 2024-04-29T14:53:20.164 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3265d7fd:::benchmark_data_smithi194_186548_object54894:head# 2024-04-29T14:53:20.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3265e6a3:::benchmark_data_smithi194_186548_object77719:head# 2024-04-29T14:53:20.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326607e9:::benchmark_data_smithi194_186548_object55017:head# 2024-04-29T14:53:20.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32666961:::benchmark_data_smithi194_186548_object15944:head# 2024-04-29T14:53:20.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32681861:::benchmark_data_smithi194_186548_object9457:head# 2024-04-29T14:53:20.172 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326d08ca:::benchmark_data_smithi194_186548_object36697:head# 2024-04-29T14:53:20.178 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326ea9fd:::benchmark_data_smithi194_186548_object5613:head# 2024-04-29T14:53:20.178 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326ec0b4:::benchmark_data_smithi194_186548_object44261:head# 2024-04-29T14:53:20.178 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326ecba9:::benchmark_data_smithi194_186548_object52352:head# 2024-04-29T14:53:20.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326f1554:::benchmark_data_smithi194_186548_object79484:head# 2024-04-29T14:53:20.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326fbea6:::benchmark_data_smithi194_186548_object11686:head# 2024-04-29T14:53:20.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:326fc0e7:::benchmark_data_smithi194_186548_object59071:head# 2024-04-29T14:53:20.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3270384f:::benchmark_data_smithi194_186548_object29191:head# 2024-04-29T14:53:20.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3270832f:::benchmark_data_smithi194_186548_object66384:head# 2024-04-29T14:53:20.185 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:327275a4:::benchmark_data_smithi194_186548_object25673:head# 2024-04-29T14:53:20.187 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32737c39:::benchmark_data_smithi194_186548_object82543:head# 2024-04-29T14:53:20.188 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3277109a:::benchmark_data_smithi194_186548_object30492:head# 2024-04-29T14:53:20.188 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3278332f:::benchmark_data_smithi194_186548_object33376:head# 2024-04-29T14:53:20.188 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32789967:::benchmark_data_smithi194_186548_object68476:head# 2024-04-29T14:53:20.189 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3279c8c5:::benchmark_data_smithi194_186548_object3419:head# 2024-04-29T14:53:20.189 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:327a39b8:::benchmark_data_smithi194_186548_object31506:head# 2024-04-29T14:53:20.193 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:327ab457:::benchmark_data_smithi194_186548_object74984:head# 2024-04-29T14:53:20.193 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:327c2918:::benchmark_data_smithi194_186548_object30988:head# 2024-04-29T14:53:20.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:327c5565:::benchmark_data_smithi194_186548_object23927:head# 2024-04-29T14:53:20.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:327fb560:::benchmark_data_smithi194_186548_object18086:head# 2024-04-29T14:53:20.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32800e12:::benchmark_data_smithi194_186548_object83498:head# 2024-04-29T14:53:20.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3280d627:::benchmark_data_smithi194_186548_object62135:head# 2024-04-29T14:53:20.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32815ea4:::benchmark_data_smithi194_186548_object38476:head# 2024-04-29T14:53:20.197 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3281b5ff:::benchmark_data_smithi194_186548_object2667:head# 2024-04-29T14:53:20.197 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3281c0f5:::benchmark_data_smithi194_186548_object65808:head# 2024-04-29T14:53:20.207 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32824d56:::benchmark_data_smithi194_186548_object18559:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32828f9b:::benchmark_data_smithi194_186548_object84070:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3282d730:::benchmark_data_smithi194_186548_object10625:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3283fa21:::benchmark_data_smithi194_186548_object49140:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3285707b:::benchmark_data_smithi194_186548_object55649:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3285deb5:::benchmark_data_smithi194_186548_object58081:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32877148:::benchmark_data_smithi194_186548_object3209:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328794ce:::benchmark_data_smithi194_186548_object44066:head# 2024-04-29T14:53:20.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328881fd:::benchmark_data_smithi194_186548_object5348:head# 2024-04-29T14:53:20.209 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3289234f:::benchmark_data_smithi194_186548_object35311:head# 2024-04-29T14:53:20.209 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32895935:::benchmark_data_smithi194_186548_object65217:head# 2024-04-29T14:53:20.209 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328a0797:::benchmark_data_smithi194_186548_object51593:head# 2024-04-29T14:53:20.210 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328a7532:::benchmark_data_smithi194_186548_object24707:head# 2024-04-29T14:53:20.210 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328b9524:::benchmark_data_smithi194_186548_object67707:head# 2024-04-29T14:53:20.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328be3c2:::benchmark_data_smithi194_186548_object19484:head# 2024-04-29T14:53:20.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328c08b7:::benchmark_data_smithi194_186548_object44976:head# 2024-04-29T14:53:20.212 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328cdba3:::benchmark_data_smithi194_186548_object74800:head# 2024-04-29T14:53:20.213 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328e3d1e:::benchmark_data_smithi194_186548_object53041:head# 2024-04-29T14:53:20.213 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328e405b:::benchmark_data_smithi194_186548_object33241:head# 2024-04-29T14:53:20.214 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:328eb39f:::benchmark_data_smithi194_186548_object46784:head# 2024-04-29T14:53:20.216 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329080d9:::benchmark_data_smithi194_186548_object36665:head# 2024-04-29T14:53:20.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3291cc7e:::benchmark_data_smithi194_186548_object19819:head# 2024-04-29T14:53:20.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329258ff:::benchmark_data_smithi194_186548_object31838:head# 2024-04-29T14:53:20.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3292e3c0:::benchmark_data_smithi194_186548_object14223:head# 2024-04-29T14:53:20.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32932128:::benchmark_data_smithi194_186548_object64918:head# 2024-04-29T14:53:20.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32949ccd:::benchmark_data_smithi194_186548_object35462:head# 2024-04-29T14:53:20.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3294ce70:::benchmark_data_smithi194_186548_object43171:head# 2024-04-29T14:53:20.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32954d4c:::benchmark_data_smithi194_186548_object72610:head# 2024-04-29T14:53:20.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3296494b:::benchmark_data_smithi194_186548_object68940:head# 2024-04-29T14:53:20.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32969d83:::benchmark_data_smithi194_186548_object51751:head# 2024-04-29T14:53:20.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3296a31e:::benchmark_data_smithi194_186548_object57024:head# 2024-04-29T14:53:20.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329754a9:::benchmark_data_smithi194_186548_object13168:head# 2024-04-29T14:53:20.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3297aaab:::benchmark_data_smithi194_186548_object65936:head# 2024-04-29T14:53:20.224 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3297ee6d:::benchmark_data_smithi194_186548_object71206:head# 2024-04-29T14:53:20.224 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32989443:::benchmark_data_smithi194_186548_object60211:head# 2024-04-29T14:53:20.225 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3299808f:::benchmark_data_smithi194_186548_object38236:head# 2024-04-29T14:53:20.225 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3299e726:::benchmark_data_smithi194_186548_object70653:head# 2024-04-29T14:53:20.227 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329a0f31:::benchmark_data_smithi194_186548_object69902:head# 2024-04-29T14:53:20.233 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329bd8c0:::benchmark_data_smithi194_186548_object17405:head# 2024-04-29T14:53:20.234 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329c3fe2:::benchmark_data_smithi194_186548_object72608:head# 2024-04-29T14:53:20.234 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329d341b:::benchmark_data_smithi194_186548_object65311:head# 2024-04-29T14:53:20.234 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329d4d4d:::benchmark_data_smithi194_186548_object24392:head# 2024-04-29T14:53:20.235 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329ddd76:::benchmark_data_smithi194_186548_object85281:head# 2024-04-29T14:53:20.235 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:329fe143:::benchmark_data_smithi194_186548_object11781:head# 2024-04-29T14:53:20.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a09fbf:::benchmark_data_smithi194_186548_object1003:head# 2024-04-29T14:53:20.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a0b69f:::benchmark_data_smithi194_186548_object26222:head# 2024-04-29T14:53:20.238 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a0cd85:::benchmark_data_smithi194_186548_object74737:head# 2024-04-29T14:53:20.239 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a15f43:::benchmark_data_smithi194_186548_object7636:head# 2024-04-29T14:53:20.239 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a1cb1a:::benchmark_data_smithi194_186548_object1050:head# 2024-04-29T14:53:20.239 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a36cdc:::benchmark_data_smithi194_186548_object78099:head# 2024-04-29T14:53:20.240 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a3dd55:::benchmark_data_smithi194_186548_object41920:head# 2024-04-29T14:53:20.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a4ca70:::benchmark_data_smithi194_186548_object14898:head# 2024-04-29T14:53:20.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a5d1e5:::benchmark_data_smithi194_186548_object35199:head# 2024-04-29T14:53:20.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a5e8d2:::benchmark_data_smithi194_186548_object50402:head# 2024-04-29T14:53:20.242 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a6054b:::benchmark_data_smithi194_186548_object52730:head# 2024-04-29T14:53:20.244 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a7cc01:::benchmark_data_smithi194_186548_object65380:head# 2024-04-29T14:53:20.244 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a825da:::benchmark_data_smithi194_186548_object67617:head# 2024-04-29T14:53:20.244 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a8dfde:::benchmark_data_smithi194_186548_object4009:head# 2024-04-29T14:53:20.244 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32a9dfa0:::benchmark_data_smithi194_186548_object39044:head# 2024-04-29T14:53:20.245 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32aa6670:::benchmark_data_smithi194_186548_object32465:head# 2024-04-29T14:53:20.245 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32aabbba:::benchmark_data_smithi194_186548_object11779:head# 2024-04-29T14:53:20.246 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ab9a73:::benchmark_data_smithi194_186548_object78700:head# 2024-04-29T14:53:20.246 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32abe0bf:::benchmark_data_smithi194_186548_object10910:head# 2024-04-29T14:53:20.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ac03b6:::benchmark_data_smithi194_186548_object54299:head# 2024-04-29T14:53:20.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b102a3:::benchmark_data_smithi194_186548_object55941:head# 2024-04-29T14:53:20.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b1689a:::benchmark_data_smithi194_186548_object37136:head# 2024-04-29T14:53:20.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b20c62:::benchmark_data_smithi194_186548_object20573:head# 2024-04-29T14:53:20.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b24677:::benchmark_data_smithi194_186548_object42912:head# 2024-04-29T14:53:20.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b27151:::benchmark_data_smithi194_186548_object7970:head# 2024-04-29T14:53:20.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b4f5c5:::benchmark_data_smithi194_186548_object84556:head# 2024-04-29T14:53:20.250 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b4f603:::benchmark_data_smithi194_186548_object80061:head# 2024-04-29T14:53:20.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b64eb2:::benchmark_data_smithi194_186548_object9014:head# 2024-04-29T14:53:20.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b6a713:::benchmark_data_smithi194_186548_object25835:head# 2024-04-29T14:53:20.252 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b6d3f0:::benchmark_data_smithi194_186548_object8731:head# 2024-04-29T14:53:20.253 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b70f17:::benchmark_data_smithi194_186548_object1388:head# 2024-04-29T14:53:20.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b71e28:::benchmark_data_smithi194_186548_object7614:head# 2024-04-29T14:53:20.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b73d58:::benchmark_data_smithi194_186548_object69462:head# 2024-04-29T14:53:20.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b78398:::benchmark_data_smithi194_186548_object59117:head# 2024-04-29T14:53:20.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b82791:::benchmark_data_smithi194_186548_object63354:head# 2024-04-29T14:53:20.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b830a9:::benchmark_data_smithi194_186548_object21418:head# 2024-04-29T14:53:20.256 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b925de:::benchmark_data_smithi194_186548_object13359:head# 2024-04-29T14:53:20.256 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b95388:::benchmark_data_smithi194_186548_object81150:head# 2024-04-29T14:53:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32b96c78:::benchmark_data_smithi194_186548_object76352:head# 2024-04-29T14:53:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32bb1584:::benchmark_data_smithi194_186548_object1517:head# 2024-04-29T14:53:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32bcb7e3:::benchmark_data_smithi194_186548_object48790:head# 2024-04-29T14:53:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32bce8ce:::benchmark_data_smithi194_186548_object18650:head# 2024-04-29T14:53:20.262 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32bd3b0e:::benchmark_data_smithi194_186548_object46:head# 2024-04-29T14:53:20.262 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32be00e4:::benchmark_data_smithi194_186548_object22335:head# 2024-04-29T14:53:20.263 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32bf3fe3:::benchmark_data_smithi194_186548_object77470:head# 2024-04-29T14:53:20.263 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c2fb2c:::benchmark_data_smithi194_186548_object17127:head# 2024-04-29T14:53:20.264 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c31223:::benchmark_data_smithi194_186548_object54567:head# 2024-04-29T14:53:20.264 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c379fe:::benchmark_data_smithi194_186548_object59749:head# 2024-04-29T14:53:20.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c3b8ea:::benchmark_data_smithi194_186548_object65434:head# 2024-04-29T14:53:20.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c4d676:::benchmark_data_smithi194_186548_object40671:head# 2024-04-29T14:53:20.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c743de:::benchmark_data_smithi194_186548_object29260:head# 2024-04-29T14:53:20.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c8c2c4:::benchmark_data_smithi194_186548_object59756:head# 2024-04-29T14:53:20.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32c8ed1a:::benchmark_data_smithi194_186548_object57249:head# 2024-04-29T14:53:20.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ca8e9c:::benchmark_data_smithi194_186548_object63871:head# 2024-04-29T14:53:20.268 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32cb0a9d:::benchmark_data_smithi194_186548_object79821:head# 2024-04-29T14:53:20.268 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32cb32f8:::benchmark_data_smithi194_186548_object42817:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32cbadb0:::benchmark_data_smithi194_186548_object145:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32cd10e8:::benchmark_data_smithi194_186548_object68810:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32cd40cb:::benchmark_data_smithi194_186548_object10879:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32cd876a:::benchmark_data_smithi194_186548_object43820:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ceb591:::benchmark_data_smithi194_186548_object980:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32cf0a75:::benchmark_data_smithi194_186548_object46752:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d062f5:::benchmark_data_smithi194_186548_object29738:head# 2024-04-29T14:53:20.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d0edfe:::benchmark_data_smithi194_186548_object28027:head# 2024-04-29T14:53:20.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d12a04:::benchmark_data_smithi194_186548_object60095:head# 2024-04-29T14:53:20.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d1f0cc:::benchmark_data_smithi194_186548_object18195:head# 2024-04-29T14:53:20.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d33817:::benchmark_data_smithi194_186548_object4584:head# 2024-04-29T14:53:20.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d37b7f:::benchmark_data_smithi194_186548_object21266:head# 2024-04-29T14:53:20.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d47934:::benchmark_data_smithi194_186548_object34050:head# 2024-04-29T14:53:20.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d49f84:::benchmark_data_smithi194_186548_object50042:head# 2024-04-29T14:53:20.279 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d4a291:::benchmark_data_smithi194_186548_object4806:head# 2024-04-29T14:53:20.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d683bc:::benchmark_data_smithi194_186548_object80149:head# 2024-04-29T14:53:20.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d6df57:::benchmark_data_smithi194_186548_object41215:head# 2024-04-29T14:53:20.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d79a62:::benchmark_data_smithi194_186548_object460:head# 2024-04-29T14:53:20.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d8826e:::benchmark_data_smithi194_186548_object73537:head# 2024-04-29T14:53:20.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d893c3:::benchmark_data_smithi194_186548_object47433:head# 2024-04-29T14:53:20.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d90d27:::benchmark_data_smithi194_186548_object23294:head# 2024-04-29T14:53:20.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d9124e:::benchmark_data_smithi194_186548_object67258:head# 2024-04-29T14:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32d94791:::benchmark_data_smithi194_186548_object28981:head# 2024-04-29T14:53:20.286 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32da4131:::benchmark_data_smithi194_186548_object46155:head# 2024-04-29T14:53:20.287 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32da6cc7:::benchmark_data_smithi194_186548_object61368:head# 2024-04-29T14:53:20.288 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32de1328:::benchmark_data_smithi194_186548_object63023:head# 2024-04-29T14:53:20.288 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32def3a8:::benchmark_data_smithi194_186548_object65089:head# 2024-04-29T14:53:20.288 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32defd84:::benchmark_data_smithi194_186548_object52715:head# 2024-04-29T14:53:20.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e04f2c:::benchmark_data_smithi194_186548_object3685:head# 2024-04-29T14:53:20.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e0deb3:::benchmark_data_smithi194_186548_object29886:head# 2024-04-29T14:53:20.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e107ea:::benchmark_data_smithi194_186548_object41788:head# 2024-04-29T14:53:20.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e1bc8f:::benchmark_data_smithi194_186548_object61873:head# 2024-04-29T14:53:20.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e1f19a:::benchmark_data_smithi194_186548_object52070:head# 2024-04-29T14:53:20.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e20bc4:::benchmark_data_smithi194_186548_object56049:head# 2024-04-29T14:53:20.292 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e3354f:::benchmark_data_smithi194_186548_object11770:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e3ece2:::benchmark_data_smithi194_186548_object68108:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e45c5b:::benchmark_data_smithi194_186548_object78214:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e56849:::benchmark_data_smithi194_186548_object63379:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e66ed9:::benchmark_data_smithi194_186548_object84803:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e69263:::benchmark_data_smithi194_186548_object82771:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e6991f:::benchmark_data_smithi194_186548_object48305:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e6d850:::benchmark_data_smithi194_186548_object23289:head# 2024-04-29T14:53:20.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e86981:::benchmark_data_smithi194_186548_object53414:head# 2024-04-29T14:53:20.297 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32e9b46b:::benchmark_data_smithi194_186548_object66786:head# 2024-04-29T14:53:20.300 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ed9176:::benchmark_data_smithi194_186548_object78153:head# 2024-04-29T14:53:20.300 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ee069c:::benchmark_data_smithi194_186548_object57716:head# 2024-04-29T14:53:20.301 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ef3f16:::benchmark_data_smithi194_186548_object67119:head# 2024-04-29T14:53:20.301 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ef660e:::benchmark_data_smithi194_186548_object78956:head# 2024-04-29T14:53:20.301 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ef8747:::benchmark_data_smithi194_186548_object52002:head# 2024-04-29T14:53:20.301 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32efbb1d:::benchmark_data_smithi194_186548_object54361:head# 2024-04-29T14:53:20.301 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f01f95:::benchmark_data_smithi194_186548_object42766:head# 2024-04-29T14:53:20.302 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f2708e:::benchmark_data_smithi194_186548_object33412:head# 2024-04-29T14:53:20.302 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f30a26:::benchmark_data_smithi194_186548_object33390:head# 2024-04-29T14:53:20.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f33921:::benchmark_data_smithi194_186548_object84604:head# 2024-04-29T14:53:20.304 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f35099:::benchmark_data_smithi194_186548_object47591:head# 2024-04-29T14:53:20.304 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f37a79:::benchmark_data_smithi194_186548_object4358:head# 2024-04-29T14:53:20.304 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f45f19:::benchmark_data_smithi194_186548_object45023:head# 2024-04-29T14:53:20.304 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f462b1:::benchmark_data_smithi194_186548_object63068:head# 2024-04-29T14:53:20.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f4ab36:::benchmark_data_smithi194_186548_object78484:head# 2024-04-29T14:53:20.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f5cded:::benchmark_data_smithi194_186548_object62493:head# 2024-04-29T14:53:20.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f5fb24:::benchmark_data_smithi194_186548_object55190:head# 2024-04-29T14:53:20.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f625e6:::benchmark_data_smithi194_186548_object84586:head# 2024-04-29T14:53:20.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f704fc:::benchmark_data_smithi194_186548_object49372:head# 2024-04-29T14:53:20.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32f9b213:::benchmark_data_smithi194_186548_object73280:head# 2024-04-29T14:53:20.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32fdb283:::benchmark_data_smithi194_186548_object30845:head# 2024-04-29T14:53:20.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32fe636a:::benchmark_data_smithi194_186548_object11295:head# 2024-04-29T14:53:20.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ff5208:::benchmark_data_smithi194_186548_object60251:head# 2024-04-29T14:53:20.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:32ff925b:::benchmark_data_smithi194_186548_object6358:head# 2024-04-29T14:53:20.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3300d73b:::benchmark_data_smithi194_186548_object10811:head# 2024-04-29T14:53:20.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33018341:::benchmark_data_smithi194_186548_object56992:head# 2024-04-29T14:53:20.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33027b1d:::benchmark_data_smithi194_186548_object60464:head# 2024-04-29T14:53:20.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33033513:::benchmark_data_smithi194_186548_object64381:head# 2024-04-29T14:53:20.312 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33036836:::benchmark_data_smithi194_186548_object82114:head# 2024-04-29T14:53:20.313 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330396cf:::benchmark_data_smithi194_186548_object8932:head# 2024-04-29T14:53:20.313 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3303edb4:::benchmark_data_smithi194_186548_object80047:head# 2024-04-29T14:53:20.314 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3304c944:::benchmark_data_smithi194_186548_object26579:head# 2024-04-29T14:53:20.314 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3304cde9:::benchmark_data_smithi194_186548_object69048:head# 2024-04-29T14:53:20.314 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33056ff4:::benchmark_data_smithi194_186548_object62637:head# 2024-04-29T14:53:20.315 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3306d4ee:::benchmark_data_smithi194_186548_object66507:head# 2024-04-29T14:53:20.315 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330a8f51:::benchmark_data_smithi194_186548_object39699:head# 2024-04-29T14:53:20.318 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330ab92f:::benchmark_data_smithi194_186548_object76447:head# 2024-04-29T14:53:20.318 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330ac7ab:::benchmark_data_smithi194_186548_object65559:head# 2024-04-29T14:53:20.318 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330d519d:::benchmark_data_smithi194_186548_object60884:head# 2024-04-29T14:53:20.320 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330d9d22:::benchmark_data_smithi194_186548_object10170:head# 2024-04-29T14:53:20.320 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330ea4c0:::benchmark_data_smithi194_186548_object80961:head# 2024-04-29T14:53:20.320 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330ed23b:::benchmark_data_smithi194_186548_object41499:head# 2024-04-29T14:53:20.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330f2522:::benchmark_data_smithi194_186548_object81383:head# 2024-04-29T14:53:20.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330f5099:::benchmark_data_smithi194_186548_object64779:head# 2024-04-29T14:53:20.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330f632c:::benchmark_data_smithi194_186548_object13921:head# 2024-04-29T14:53:20.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:330ff981:::benchmark_data_smithi194_186548_object54939:head# 2024-04-29T14:53:20.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33104af4:::benchmark_data_smithi194_186548_object5463:head# 2024-04-29T14:53:20.322 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331449d1:::benchmark_data_smithi194_186548_object51720:head# 2024-04-29T14:53:20.322 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331593f9:::benchmark_data_smithi194_186548_object19280:head# 2024-04-29T14:53:20.323 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3315b058:::benchmark_data_smithi194_186548_object32599:head# 2024-04-29T14:53:20.323 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3315d335:::benchmark_data_smithi194_186548_object16627:head# 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:20.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:20.324 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3315d5cd:::benchmark_data_smithi194_186548_object5458:head# 2024-04-29T14:53:20.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33199299:::benchmark_data_smithi194_186548_object25179:head# 2024-04-29T14:53:20.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33199745:::benchmark_data_smithi194_186548_object43903:head# 2024-04-29T14:53:20.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331b768a:::benchmark_data_smithi194_186548_object1594:head# 2024-04-29T14:53:20.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331c26a2:::benchmark_data_smithi194_186548_object48407:head# 2024-04-29T14:53:20.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331c384a:::benchmark_data_smithi194_186548_object5887:head# 2024-04-29T14:53:20.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331c5905:::benchmark_data_smithi194_186548_object20808:head# 2024-04-29T14:53:20.327 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331d8833:::benchmark_data_smithi194_186548_object78136:head# 2024-04-29T14:53:20.327 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331dd745:::benchmark_data_smithi194_186548_object43737:head# 2024-04-29T14:53:20.328 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331e0bec:::benchmark_data_smithi194_186548_object43279:head# 2024-04-29T14:53:20.328 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331fb178:::benchmark_data_smithi194_186548_object35729:head# 2024-04-29T14:53:20.328 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:331fe400:::benchmark_data_smithi194_186548_object20619:head# 2024-04-29T14:53:20.329 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3320967d:::benchmark_data_smithi194_186548_object36759:head# 2024-04-29T14:53:20.329 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3321ca7d:::benchmark_data_smithi194_186548_object76402:head# 2024-04-29T14:53:20.330 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3322098c:::benchmark_data_smithi194_186548_object49732:head# 2024-04-29T14:53:20.330 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332213cb:::benchmark_data_smithi194_186548_object16468:head# 2024-04-29T14:53:20.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3322a6fb:::benchmark_data_smithi194_186548_object3589:head# 2024-04-29T14:53:20.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3322b3ed:::benchmark_data_smithi194_186548_object30058:head# 2024-04-29T14:53:20.337 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3322dc1b:::benchmark_data_smithi194_186548_object72510:head# 2024-04-29T14:53:20.337 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3323082e:::benchmark_data_smithi194_186548_object82458:head# 2024-04-29T14:53:20.337 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33230cd0:::benchmark_data_smithi194_186548_object46905:head# 2024-04-29T14:53:20.338 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3324145c:::benchmark_data_smithi194_186548_object29096:head# 2024-04-29T14:53:20.338 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3324382a:::benchmark_data_smithi194_186548_object51191:head# 2024-04-29T14:53:20.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3324bab3:::benchmark_data_smithi194_186548_object20233:head# 2024-04-29T14:53:20.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33255508:::benchmark_data_smithi194_186548_object36751:head# 2024-04-29T14:53:20.341 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332610a6:::benchmark_data_smithi194_186548_object33401:head# 2024-04-29T14:53:20.341 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33271b66:::benchmark_data_smithi194_186548_object70122:head# 2024-04-29T14:53:20.342 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33287821:::benchmark_data_smithi194_186548_object36260:head# 2024-04-29T14:53:20.342 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332a29eb:::benchmark_data_smithi194_186548_object79:head# 2024-04-29T14:53:20.342 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332a8743:::benchmark_data_smithi194_186548_object37126:head# 2024-04-29T14:53:20.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332a96ac:::benchmark_data_smithi194_186548_object33306:head# 2024-04-29T14:53:20.345 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332c9d8b:::benchmark_data_smithi194_186548_object2971:head# 2024-04-29T14:53:20.345 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332ca350:::benchmark_data_smithi194_186548_object28785:head# 2024-04-29T14:53:20.346 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332cafcd:::benchmark_data_smithi194_186548_object11461:head# 2024-04-29T14:53:20.351 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332dcd32:::benchmark_data_smithi194_186548_object81961:head# 2024-04-29T14:53:20.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332ded8c:::benchmark_data_smithi194_186548_object11291:head# 2024-04-29T14:53:20.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:332fc83e:::benchmark_data_smithi194_186548_object40697:head# 2024-04-29T14:53:20.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3330a6a9:::benchmark_data_smithi194_186548_object69591:head# 2024-04-29T14:53:20.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3332707e:::benchmark_data_smithi194_186548_object34228:head# 2024-04-29T14:53:20.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3332c401:::benchmark_data_smithi194_186548_object25524:head# 2024-04-29T14:53:20.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3332ed61:::benchmark_data_smithi194_186548_object51900:head# 2024-04-29T14:53:20.359 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33340135:::benchmark_data_smithi194_186548_object65403:head# 2024-04-29T14:53:20.359 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333485ea:::benchmark_data_smithi194_186548_object76182:head# 2024-04-29T14:53:20.359 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33349a16:::benchmark_data_smithi194_186548_object36564:head# 2024-04-29T14:53:20.360 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3334c319:::benchmark_data_smithi194_186548_object23611:head# 2024-04-29T14:53:20.360 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3335c105:::benchmark_data_smithi194_186548_object21394:head# 2024-04-29T14:53:20.361 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33366410:::benchmark_data_smithi194_186548_object11085:head# 2024-04-29T14:53:20.361 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333773ed:::benchmark_data_smithi194_186548_object53447:head# 2024-04-29T14:53:20.362 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3337f47c:::benchmark_data_smithi194_186548_object66056:head# 2024-04-29T14:53:20.363 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3337fa4a:::benchmark_data_smithi194_186548_object72096:head# 2024-04-29T14:53:20.366 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3338a6bf:::benchmark_data_smithi194_186548_object50460:head# 2024-04-29T14:53:20.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33394df0:::benchmark_data_smithi194_186548_object85034:head# 2024-04-29T14:53:20.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333a02b8:::benchmark_data_smithi194_186548_object50056:head# 2024-04-29T14:53:20.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333a517e:::benchmark_data_smithi194_186548_object48903:head# 2024-04-29T14:53:20.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333a7789:::benchmark_data_smithi194_186548_object35494:head# 2024-04-29T14:53:20.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333a7d39:::benchmark_data_smithi194_186548_object26884:head# 2024-04-29T14:53:20.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333aaea4:::benchmark_data_smithi194_186548_object50642:head# 2024-04-29T14:53:20.373 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333abda6:::benchmark_data_smithi194_186548_object82349:head# 2024-04-29T14:53:20.373 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333ba326:::benchmark_data_smithi194_186548_object20348:head# 2024-04-29T14:53:20.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:20.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:20.375 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333bd8ba:::benchmark_data_smithi194_186548_object41302:head# 2024-04-29T14:53:20.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:333fd17e:::benchmark_data_smithi194_186548_object53601:head# 2024-04-29T14:53:20.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3340387a:::benchmark_data_smithi194_186548_object62857:head# 2024-04-29T14:53:20.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33405925:::benchmark_data_smithi194_186548_object22839:head# 2024-04-29T14:53:20.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3340ebb1:::benchmark_data_smithi194_186548_object66714:head# 2024-04-29T14:53:20.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3341fe63:::benchmark_data_smithi194_186548_object2613:head# 2024-04-29T14:53:20.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33425253:::benchmark_data_smithi194_186548_object54407:head# 2024-04-29T14:53:20.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33426e41:::benchmark_data_smithi194_186548_object5516:head# 2024-04-29T14:53:20.377 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3342ccde:::benchmark_data_smithi194_186548_object48344:head# 2024-04-29T14:53:20.380 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33435c84:::benchmark_data_smithi194_186548_object9757:head# 2024-04-29T14:53:20.380 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33443407:::benchmark_data_smithi194_186548_object17158:head# 2024-04-29T14:53:20.380 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3344d115:::benchmark_data_smithi194_186548_object6578:head# 2024-04-29T14:53:20.381 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334516be:::benchmark_data_smithi194_186548_object17919:head# 2024-04-29T14:53:20.381 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334572db:::benchmark_data_smithi194_186548_object19579:head# 2024-04-29T14:53:20.382 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3345cc9a:::benchmark_data_smithi194_186548_object4589:head# 2024-04-29T14:53:20.382 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3346787b:::benchmark_data_smithi194_186548_object1413:head# 2024-04-29T14:53:20.383 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3346bfb8:::benchmark_data_smithi194_186548_object22257:head# 2024-04-29T14:53:20.383 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33473e68:::benchmark_data_smithi194_186548_object82272:head# 2024-04-29T14:53:20.384 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3347c440:::benchmark_data_smithi194_186548_object14375:head# 2024-04-29T14:53:20.385 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334a4d36:::benchmark_data_smithi194_186548_object61827:head# 2024-04-29T14:53:20.385 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334a7439:::benchmark_data_smithi194_186548_object22233:head# 2024-04-29T14:53:20.385 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334a7b9c:::benchmark_data_smithi194_186548_object26865:head# 2024-04-29T14:53:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334b2ec4:::benchmark_data_smithi194_186548_object69348:head# 2024-04-29T14:53:20.386 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334c3bb8:::benchmark_data_smithi194_186548_object80498:head# 2024-04-29T14:53:20.387 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334f2828:::benchmark_data_smithi194_186548_object24629:head# 2024-04-29T14:53:20.387 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:334fe63a:::benchmark_data_smithi194_186548_object32898:head# 2024-04-29T14:53:20.387 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33512911:::benchmark_data_smithi194_186548_object7534:head# 2024-04-29T14:53:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33530f84:::benchmark_data_smithi194_186548_object25731:head# 2024-04-29T14:53:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33545d10:::benchmark_data_smithi194_186548_object80007:head# 2024-04-29T14:53:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3354ad36:::benchmark_data_smithi194_186548_object75052:head# 2024-04-29T14:53:20.392 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3354ed29:::benchmark_data_smithi194_186548_object64142:head# 2024-04-29T14:53:20.393 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33550a02:::benchmark_data_smithi194_186548_object33139:head# 2024-04-29T14:53:20.393 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335559bf:::benchmark_data_smithi194_186548_object40602:head# 2024-04-29T14:53:20.393 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33557804:::benchmark_data_smithi194_186548_object9268:head# 2024-04-29T14:53:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3358e9ab:::benchmark_data_smithi194_186548_object8672:head# 2024-04-29T14:53:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335968a2:::benchmark_data_smithi194_186548_object11147:head# 2024-04-29T14:53:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335975fe:::benchmark_data_smithi194_186548_object73400:head# 2024-04-29T14:53:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33597a5c:::benchmark_data_smithi194_186548_object3821:head# 2024-04-29T14:53:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3359c314:::benchmark_data_smithi194_186548_object30337:head# 2024-04-29T14:53:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335b2815:::benchmark_data_smithi194_186548_object81892:head# 2024-04-29T14:53:20.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335d791a:::benchmark_data_smithi194_186548_object37544:head# 2024-04-29T14:53:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335df52b:::benchmark_data_smithi194_186548_object28783:head# 2024-04-29T14:53:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335e390e:::benchmark_data_smithi194_186548_object25037:head# 2024-04-29T14:53:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335e7f44:::benchmark_data_smithi194_186548_object38031:head# 2024-04-29T14:53:20.398 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335e9669:::benchmark_data_smithi194_186548_object78970:head# 2024-04-29T14:53:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335ea17d:::benchmark_data_smithi194_186548_object81729:head# 2024-04-29T14:53:20.399 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335f56a9:::benchmark_data_smithi194_186548_object30011:head# 2024-04-29T14:53:20.407 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:335fe782:::benchmark_data_smithi194_186548_object10535:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3360f0fa:::benchmark_data_smithi194_186548_object44743:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33611842:::benchmark_data_smithi194_186548_object3329:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3361a846:::benchmark_data_smithi194_186548_object62807:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33625ff3:::benchmark_data_smithi194_186548_object68915:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33628686:::benchmark_data_smithi194_186548_object20188:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3362c12d:::benchmark_data_smithi194_186548_object70027:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33632c74:::benchmark_data_smithi194_186548_object20617:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33644258:::benchmark_data_smithi194_186548_object46676:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33652b86:::benchmark_data_smithi194_186548_object23412:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3365cec2:::benchmark_data_smithi194_186548_object19503:head# 2024-04-29T14:53:20.408 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3365f855:::benchmark_data_smithi194_186548_object53145:head# 2024-04-29T14:53:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3366cc51:::benchmark_data_smithi194_186548_object15187:head# 2024-04-29T14:53:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3368b2da:::benchmark_data_smithi194_186548_object80488:head# 2024-04-29T14:53:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336a116e:::benchmark_data_smithi194_186548_object54360:head# 2024-04-29T14:53:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336b098b:::benchmark_data_smithi194_186548_object14531:head# 2024-04-29T14:53:20.409 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336b9e79:::benchmark_data_smithi194_186548_object55219:head# 2024-04-29T14:53:20.410 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336c9afa:::benchmark_data_smithi194_186548_object57890:head# 2024-04-29T14:53:20.410 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336d16eb:::benchmark_data_smithi194_186548_object83203:head# 2024-04-29T14:53:20.410 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336ef86e:::benchmark_data_smithi194_186548_object22159:head# 2024-04-29T14:53:20.410 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336f21e0:::benchmark_data_smithi194_186548_object74021:head# 2024-04-29T14:53:20.410 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:336f8a57:::benchmark_data_smithi194_186548_object61829:head# 2024-04-29T14:53:20.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33707264:::benchmark_data_smithi194_186548_object18433:head# 2024-04-29T14:53:20.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33709d46:::benchmark_data_smithi194_186548_object57025:head# 2024-04-29T14:53:20.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33710478:::benchmark_data_smithi194_186548_object76997:head# 2024-04-29T14:53:20.412 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3373cdfe:::benchmark_data_smithi194_186548_object39610:head# 2024-04-29T14:53:20.413 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3373e53c:::benchmark_data_smithi194_186548_object21190:head# 2024-04-29T14:53:20.413 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3373f99e:::benchmark_data_smithi194_186548_object67914:head# 2024-04-29T14:53:20.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33745047:::benchmark_data_smithi194_186548_object69834:head# 2024-04-29T14:53:20.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3374dbc6:::benchmark_data_smithi194_186548_object40499:head# 2024-04-29T14:53:20.415 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33752c92:::benchmark_data_smithi194_186548_object61811:head# 2024-04-29T14:53:20.415 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3375b14b:::benchmark_data_smithi194_186548_object9501:head# 2024-04-29T14:53:20.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:337614e1:::benchmark_data_smithi194_186548_object742:head# 2024-04-29T14:53:20.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33766235:::benchmark_data_smithi194_186548_object70451:head# 2024-04-29T14:53:20.419 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3377e227:::benchmark_data_smithi194_186548_object7369:head# 2024-04-29T14:53:20.419 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33782dee:::benchmark_data_smithi194_186548_object23209:head# 2024-04-29T14:53:20.420 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33785366:::benchmark_data_smithi194_186548_object10295:head# 2024-04-29T14:53:20.420 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3378b55e:::benchmark_data_smithi194_186548_object7172:head# 2024-04-29T14:53:20.420 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33792375:::benchmark_data_smithi194_186548_object70317:head# 2024-04-29T14:53:20.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33798c4c:::benchmark_data_smithi194_186548_object21783:head# 2024-04-29T14:53:20.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:337aab06:::benchmark_data_smithi194_186548_object46933:head# 2024-04-29T14:53:20.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:337afe07:::benchmark_data_smithi194_186548_object18552:head# 2024-04-29T14:53:20.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:337db06e:::benchmark_data_smithi194_186548_object17440:head# 2024-04-29T14:53:20.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:337e024a:::benchmark_data_smithi194_186548_object12422:head# 2024-04-29T14:53:20.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:337e20ac:::benchmark_data_smithi194_186548_object82746:head# 2024-04-29T14:53:20.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:337f49fe:::benchmark_data_smithi194_186548_object62090:head# 2024-04-29T14:53:20.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3380b890:::benchmark_data_smithi194_186548_object24846:head# 2024-04-29T14:53:20.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3381218e:::benchmark_data_smithi194_186548_object66591:head# 2024-04-29T14:53:20.428 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3381bd02:::benchmark_data_smithi194_186548_object59385:head# 2024-04-29T14:53:20.436 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33829805:::benchmark_data_smithi194_186548_object21581:head# 2024-04-29T14:53:20.436 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3382a02a:::benchmark_data_smithi194_186548_object19550:head# 2024-04-29T14:53:20.436 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3384bce3:::benchmark_data_smithi194_186548_object82979:head# 2024-04-29T14:53:20.437 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3384daf0:::benchmark_data_smithi194_186548_object29936:head# 2024-04-29T14:53:20.437 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33850762:::benchmark_data_smithi194_186548_object41992:head# 2024-04-29T14:53:20.437 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33858ecb:::benchmark_data_smithi194_186548_object36614:head# 2024-04-29T14:53:20.438 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3385aede:::benchmark_data_smithi194_186548_object49299:head# 2024-04-29T14:53:20.438 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3385cbde:::benchmark_data_smithi194_186548_object68372:head# 2024-04-29T14:53:20.439 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3387e92e:::benchmark_data_smithi194_186548_object54533:head# 2024-04-29T14:53:20.440 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33889029:::benchmark_data_smithi194_186548_object22172:head# 2024-04-29T14:53:20.440 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3388d0fe:::benchmark_data_smithi194_186548_object43509:head# 2024-04-29T14:53:20.441 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3388d29a:::benchmark_data_smithi194_186548_object79837:head# 2024-04-29T14:53:20.442 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3389d33d:::benchmark_data_smithi194_186548_object11123:head# 2024-04-29T14:53:20.442 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338a2f33:::benchmark_data_smithi194_186548_object11026:head# 2024-04-29T14:53:20.443 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338a5724:::benchmark_data_smithi194_186548_object10716:head# 2024-04-29T14:53:20.443 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338a7022:::benchmark_data_smithi194_186548_object25193:head# 2024-04-29T14:53:20.443 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338af070:::benchmark_data_smithi194_186548_object62987:head# 2024-04-29T14:53:20.446 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338b25a0:::benchmark_data_smithi194_186548_object5025:head# 2024-04-29T14:53:20.447 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338c598c:::benchmark_data_smithi194_186548_object2073:head# 2024-04-29T14:53:20.449 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338d5142:::benchmark_data_smithi194_186548_object32998:head# 2024-04-29T14:53:20.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338d7a34:::benchmark_data_smithi194_186548_object53692:head# 2024-04-29T14:53:20.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338e2abd:::benchmark_data_smithi194_186548_object44142:head# 2024-04-29T14:53:20.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338e4ce0:::benchmark_data_smithi194_186548_object60256:head# 2024-04-29T14:53:20.455 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:338fb119:::benchmark_data_smithi194_186548_object17239:head# 2024-04-29T14:53:20.455 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33908c03:::benchmark_data_smithi194_186548_object32310:head# 2024-04-29T14:53:20.455 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33908f5a:::benchmark_data_smithi194_186548_object71838:head# 2024-04-29T14:53:20.457 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3390dcbe:::benchmark_data_smithi194_186548_object72232:head# 2024-04-29T14:53:20.458 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33921013:::benchmark_data_smithi194_186548_object55743:head# 2024-04-29T14:53:20.458 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3392c9f3:::benchmark_data_smithi194_186548_object18967:head# 2024-04-29T14:53:20.458 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33931ad6:::benchmark_data_smithi194_186548_object73546:head# 2024-04-29T14:53:20.459 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33933555:::benchmark_data_smithi194_186548_object15170:head# 2024-04-29T14:53:20.459 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:339420c3:::benchmark_data_smithi194_186548_object53379:head# 2024-04-29T14:53:20.460 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3394bba8:::benchmark_data_smithi194_186548_object73242:head# 2024-04-29T14:53:20.462 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3396c2ea:::benchmark_data_smithi194_186548_object14060:head# 2024-04-29T14:53:20.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3396ebfd:::benchmark_data_smithi194_186548_object83375:head# 2024-04-29T14:53:20.464 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3397f4e6:::benchmark_data_smithi194_186548_object64109:head# 2024-04-29T14:53:20.464 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:339b5634:::benchmark_data_smithi194_186548_object30460:head# 2024-04-29T14:53:20.465 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:339b871d:::benchmark_data_smithi194_186548_object52338:head# 2024-04-29T14:53:20.465 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:339c6d5f:::benchmark_data_smithi194_186548_object4608:head# 2024-04-29T14:53:20.465 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:339d7007:::benchmark_data_smithi194_186548_object58835:head# 2024-04-29T14:53:20.466 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:339e0532:::benchmark_data_smithi194_186548_object24642:head# 2024-04-29T14:53:20.469 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:339fbf8a:::benchmark_data_smithi194_186548_object28348:head# 2024-04-29T14:53:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a11124:::benchmark_data_smithi194_186548_object15485:head# 2024-04-29T14:53:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a16e5c:::benchmark_data_smithi194_186548_object26565:head# 2024-04-29T14:53:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a31d2e:::benchmark_data_smithi194_186548_object79251:head# 2024-04-29T14:53:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a37ad6:::benchmark_data_smithi194_186548_object34735:head# 2024-04-29T14:53:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a4466c:::benchmark_data_smithi194_186548_object50665:head# 2024-04-29T14:53:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a4be74:::benchmark_data_smithi194_186548_object7421:head# 2024-04-29T14:53:20.471 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a5d2c2:::benchmark_data_smithi194_186548_object8377:head# 2024-04-29T14:53:20.471 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a810df:::benchmark_data_smithi194_186548_object55729:head# 2024-04-29T14:53:20.471 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a832a2:::benchmark_data_smithi194_186548_object77774:head# 2024-04-29T14:53:20.474 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a91c51:::benchmark_data_smithi194_186548_object52343:head# 2024-04-29T14:53:20.474 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a95b38:::benchmark_data_smithi194_186548_object48365:head# 2024-04-29T14:53:20.474 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a996cd:::benchmark_data_smithi194_186548_object74960:head# 2024-04-29T14:53:20.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33a9a6f9:::benchmark_data_smithi194_186548_object16854:head# 2024-04-29T14:53:20.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33aa1996:::benchmark_data_smithi194_186548_object53559:head# 2024-04-29T14:53:20.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33aa2ba7:::benchmark_data_smithi194_186548_object5001:head# 2024-04-29T14:53:20.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33aa7c4f:::benchmark_data_smithi194_186548_object7523:head# 2024-04-29T14:53:20.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33aa935f:::benchmark_data_smithi194_186548_object21147:head# 2024-04-29T14:53:20.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ac9088:::benchmark_data_smithi194_186548_object54480:head# 2024-04-29T14:53:20.481 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ad0a70:::benchmark_data_smithi194_186548_object25028:head# 2024-04-29T14:53:20.486 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ad4a8b:::benchmark_data_smithi194_186548_object72293:head# 2024-04-29T14:53:20.486 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33adeffa:::benchmark_data_smithi194_186548_object67037:head# 2024-04-29T14:53:20.486 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ae2e0a:::benchmark_data_smithi194_186548_object10817:head# 2024-04-29T14:53:20.487 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33aee628:::benchmark_data_smithi194_186548_object71330:head# 2024-04-29T14:53:20.487 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33af3399:::benchmark_data_smithi194_186548_object21188:head# 2024-04-29T14:53:20.488 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33af5caa:::benchmark_data_smithi194_186548_object1713:head# 2024-04-29T14:53:20.488 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b002a6:::benchmark_data_smithi194_186548_object41612:head# 2024-04-29T14:53:20.489 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b0ff53:::benchmark_data_smithi194_186548_object69607:head# 2024-04-29T14:53:20.489 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b14c33:::benchmark_data_smithi194_186548_object47593:head# 2024-04-29T14:53:20.490 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b1be02:::benchmark_data_smithi194_186548_object53585:head# 2024-04-29T14:53:20.491 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b1d1a1:::benchmark_data_smithi194_186548_object75539:head# 2024-04-29T14:53:20.491 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b1dc56:::benchmark_data_smithi194_186548_object84956:head# 2024-04-29T14:53:20.492 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b2ccc8:::benchmark_data_smithi194_186548_object56532:head# 2024-04-29T14:53:20.493 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b5507a:::benchmark_data_smithi194_186548_object6371:head# 2024-04-29T14:53:20.493 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b6ff15:::benchmark_data_smithi194_186548_object31521:head# 2024-04-29T14:53:20.495 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b85608:::benchmark_data_smithi194_186548_object45575:head# 2024-04-29T14:53:20.495 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b8f497:::benchmark_data_smithi194_186548_object83627:head# 2024-04-29T14:53:20.496 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33b9ed15:::benchmark_data_smithi194_186548_object71714:head# 2024-04-29T14:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ba244b:::benchmark_data_smithi194_186548_object7221:head# 2024-04-29T14:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33baa2fb:::benchmark_data_smithi194_186548_object35894:head# 2024-04-29T14:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33bac3b9:::benchmark_data_smithi194_186548_object45773:head# 2024-04-29T14:53:20.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33baca0c:::benchmark_data_smithi194_186548_object55415:head# 2024-04-29T14:53:20.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33bc6f03:::benchmark_data_smithi194_186548_object55050:head# 2024-04-29T14:53:20.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33bc9cfd:::benchmark_data_smithi194_186548_object14748:head# 2024-04-29T14:53:20.502 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33bcb10d:::benchmark_data_smithi194_186548_object17457:head# 2024-04-29T14:53:20.502 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33bf2d89:::benchmark_data_smithi194_186548_object75213:head# 2024-04-29T14:53:20.503 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33bf9015:::benchmark_data_smithi194_186548_object373:head# 2024-04-29T14:53:20.503 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33bfd36b:::benchmark_data_smithi194_186548_object7082:head# 2024-04-29T14:53:20.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c12066:::benchmark_data_smithi194_186548_object28013:head# 2024-04-29T14:53:20.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c1f5c0:::benchmark_data_smithi194_186548_object70816:head# 2024-04-29T14:53:20.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c23996:::benchmark_data_smithi194_186548_object23107:head# 2024-04-29T14:53:20.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c2e3f5:::benchmark_data_smithi194_186548_object64154:head# 2024-04-29T14:53:20.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c2fed1:::benchmark_data_smithi194_186548_object54234:head# 2024-04-29T14:53:20.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c3c314:::benchmark_data_smithi194_186548_object71503:head# 2024-04-29T14:53:20.511 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c4b413:::benchmark_data_smithi194_186548_object32533:head# 2024-04-29T14:53:20.511 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c506c7:::benchmark_data_smithi194_186548_object69171:head# 2024-04-29T14:53:20.511 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c58c3d:::benchmark_data_smithi194_186548_object40276:head# 2024-04-29T14:53:20.512 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c8a10e:::benchmark_data_smithi194_186548_object47681:head# 2024-04-29T14:53:20.512 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c96a1a:::benchmark_data_smithi194_186548_object32009:head# 2024-04-29T14:53:20.513 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c97031:::benchmark_data_smithi194_186548_object37073:head# 2024-04-29T14:53:20.513 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c9ac13:::benchmark_data_smithi194_186548_object72958:head# 2024-04-29T14:53:20.514 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33c9e220:::benchmark_data_smithi194_186548_object31285:head# 2024-04-29T14:53:20.514 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33caec0d:::benchmark_data_smithi194_186548_object53771:head# 2024-04-29T14:53:20.514 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33caf705:::benchmark_data_smithi194_186548_object56730:head# 2024-04-29T14:53:20.515 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ccab80:::benchmark_data_smithi194_186548_object1163:head# 2024-04-29T14:53:20.515 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d11c04:::benchmark_data_smithi194_186548_object33544:head# 2024-04-29T14:53:20.516 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d1298d:::benchmark_data_smithi194_186548_object66670:head# 2024-04-29T14:53:20.516 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d1cfc8:::benchmark_data_smithi194_186548_object39231:head# 2024-04-29T14:53:20.517 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d2cea7:::benchmark_data_smithi194_186548_object11343:head# 2024-04-29T14:53:20.517 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d463b5:::benchmark_data_smithi194_186548_object41897:head# 2024-04-29T14:53:20.518 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d541bf:::benchmark_data_smithi194_186548_object59127:head# 2024-04-29T14:53:20.518 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d5db30:::benchmark_data_smithi194_186548_object33762:head# 2024-04-29T14:53:20.519 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d62026:::benchmark_data_smithi194_186548_object66148:head# 2024-04-29T14:53:20.519 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d63ae2:::benchmark_data_smithi194_186548_object39644:head# 2024-04-29T14:53:20.520 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d6433f:::benchmark_data_smithi194_186548_object56271:head# 2024-04-29T14:53:20.520 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d6b3c8:::benchmark_data_smithi194_186548_object75207:head# 2024-04-29T14:53:20.521 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d6e53a:::benchmark_data_smithi194_186548_object41666:head# 2024-04-29T14:53:20.521 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d7a864:::benchmark_data_smithi194_186548_object19781:head# 2024-04-29T14:53:20.521 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d9004e:::benchmark_data_smithi194_186548_object37693:head# 2024-04-29T14:53:20.522 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33d9b15e:::benchmark_data_smithi194_186548_object39004:head# 2024-04-29T14:53:20.522 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33da8e22:::benchmark_data_smithi194_186548_object47662:head# 2024-04-29T14:53:20.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33db055b:::benchmark_data_smithi194_186548_object56972:head# 2024-04-29T14:53:20.524 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:20.524 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33db7f5a:::benchmark_data_smithi194_186548_object66003:head# 2024-04-29T14:53:20.524 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33dc108e:::benchmark_data_smithi194_186548_object37150:head# 2024-04-29T14:53:20.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33dd4c5b:::benchmark_data_smithi194_186548_object28960:head# 2024-04-29T14:53:20.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33de4966:::benchmark_data_smithi194_186548_object69342:head# 2024-04-29T14:53:20.526 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33df9beb:::benchmark_data_smithi194_186548_object62718:head# 2024-04-29T14:53:20.526 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33dfefdf:::benchmark_data_smithi194_186548_object70492:head# 2024-04-29T14:53:20.526 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33dff53d:::benchmark_data_smithi194_186548_object77763:head# 2024-04-29T14:53:20.527 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e06feb:::benchmark_data_smithi194_186548_object5768:head# 2024-04-29T14:53:20.527 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e0813c:::benchmark_data_smithi194_186548_object50946:head# 2024-04-29T14:53:20.527 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e10e9f:::benchmark_data_smithi194_186548_object58476:head# 2024-04-29T14:53:20.528 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e14ab9:::benchmark_data_smithi194_186548_object2157:head# 2024-04-29T14:53:20.528 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e1f38e:::benchmark_data_smithi194_186548_object53864:head# 2024-04-29T14:53:20.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e25b9c:::benchmark_data_smithi194_186548_object48167:head# 2024-04-29T14:53:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:20.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:20.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e314d2:::benchmark_data_smithi194_186548_object34081:head# 2024-04-29T14:53:20.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e3e11c:::benchmark_data_smithi194_186548_object81444:head# 2024-04-29T14:53:20.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e59543:::benchmark_data_smithi194_186548_object41154:head# 2024-04-29T14:53:20.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e75153:::benchmark_data_smithi194_186548_object80553:head# 2024-04-29T14:53:20.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e7a917:::benchmark_data_smithi194_186548_object18160:head# 2024-04-29T14:53:20.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e7b0d5:::benchmark_data_smithi194_186548_object55507:head# 2024-04-29T14:53:20.535 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e7e78e:::benchmark_data_smithi194_186548_object43611:head# 2024-04-29T14:53:20.535 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e865cf:::benchmark_data_smithi194_186548_object54557:head# 2024-04-29T14:53:20.536 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e8ad1d:::benchmark_data_smithi194_186548_object64747:head# 2024-04-29T14:53:20.537 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e8d801:::benchmark_data_smithi194_186548_object80882:head# 2024-04-29T14:53:20.537 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33e8e312:::benchmark_data_smithi194_186548_object18082:head# 2024-04-29T14:53:20.537 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ea52e5:::benchmark_data_smithi194_186548_object44207:head# 2024-04-29T14:53:20.538 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33eb48a1:::benchmark_data_smithi194_186548_object50399:head# 2024-04-29T14:53:20.538 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ed4b22:::benchmark_data_smithi194_186548_object47942:head# 2024-04-29T14:53:20.539 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ed59ac:::benchmark_data_smithi194_186548_object78809:head# 2024-04-29T14:53:20.539 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ef6db8:::benchmark_data_smithi194_186548_object3108:head# 2024-04-29T14:53:20.540 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ef7aa8:::benchmark_data_smithi194_186548_object55173:head# 2024-04-29T14:53:20.540 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33eff112:::benchmark_data_smithi194_186548_object36579:head# 2024-04-29T14:53:20.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33eff1b6:::benchmark_data_smithi194_186548_object55886:head# 2024-04-29T14:53:20.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f017c4:::benchmark_data_smithi194_186548_object81412:head# 2024-04-29T14:53:20.542 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f257dd:::benchmark_data_smithi194_186548_object38182:head# 2024-04-29T14:53:20.546 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f2a61b:::benchmark_data_smithi194_186548_object81753:head# 2024-04-29T14:53:20.546 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f2f9d7:::benchmark_data_smithi194_186548_object69565:head# 2024-04-29T14:53:20.546 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f5e074:::benchmark_data_smithi194_186548_object10420:head# 2024-04-29T14:53:20.546 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f69c2a:::benchmark_data_smithi194_186548_object64123:head# 2024-04-29T14:53:20.546 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f7093f:::benchmark_data_smithi194_186548_object1881:head# 2024-04-29T14:53:20.547 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f8bf5c:::benchmark_data_smithi194_186548_object70611:head# 2024-04-29T14:53:20.547 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f8dd80:::benchmark_data_smithi194_186548_object21242:head# 2024-04-29T14:53:20.547 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f9452d:::benchmark_data_smithi194_186548_object70849:head# 2024-04-29T14:53:20.547 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33f9fc88:::benchmark_data_smithi194_186548_object61457:head# 2024-04-29T14:53:20.548 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33fa6cc7:::benchmark_data_smithi194_186548_object83099:head# 2024-04-29T14:53:20.548 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33faaddc:::benchmark_data_smithi194_186548_object26463:head# 2024-04-29T14:53:20.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33fcde1b:::benchmark_data_smithi194_186548_object66509:head# 2024-04-29T14:53:20.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33fd24a2:::benchmark_data_smithi194_186548_object17274:head# 2024-04-29T14:53:20.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33fd8779:::benchmark_data_smithi194_186548_object51155:head# 2024-04-29T14:53:20.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33fd8a49:::benchmark_data_smithi194_186548_object1849:head# 2024-04-29T14:53:20.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33fdc713:::benchmark_data_smithi194_186548_object10752:head# 2024-04-29T14:53:20.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33fe5613:::benchmark_data_smithi194_186548_object26328:head# 2024-04-29T14:53:20.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ff94f1:::benchmark_data_smithi194_186548_object37517:head# 2024-04-29T14:53:20.553 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ff9e79:::benchmark_data_smithi194_186548_object13558:head# 2024-04-29T14:53:20.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:33ffe149:::benchmark_data_smithi194_186548_object36750:head# 2024-04-29T14:53:20.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3400ffe9:::benchmark_data_smithi194_186548_object38763:head# 2024-04-29T14:53:20.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34033664:::benchmark_data_smithi194_186548_object7927:head# 2024-04-29T14:53:20.559 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34042c26:::benchmark_data_smithi194_186548_object16955:head# 2024-04-29T14:53:20.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34063975:::benchmark_data_smithi194_186548_object23368:head# 2024-04-29T14:53:20.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340649d0:::benchmark_data_smithi194_186548_object41918:head# 2024-04-29T14:53:20.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3406e191:::benchmark_data_smithi194_186548_object12346:head# 2024-04-29T14:53:20.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340717cc:::benchmark_data_smithi194_186548_object19557:head# 2024-04-29T14:53:20.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34071947:::benchmark_data_smithi194_186548_object48134:head# 2024-04-29T14:53:20.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3408ba50:::benchmark_data_smithi194_186548_object64914:head# 2024-04-29T14:53:20.563 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3408e252:::benchmark_data_smithi194_186548_object21538:head# 2024-04-29T14:53:20.564 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340a9774:::benchmark_data_smithi194_186548_object84876:head# 2024-04-29T14:53:20.566 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340b6970:::benchmark_data_smithi194_186548_object79620:head# 2024-04-29T14:53:20.567 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340c7a7f:::benchmark_data_smithi194_186548_object84502:head# 2024-04-29T14:53:20.568 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340d2caa:::benchmark_data_smithi194_186548_object11034:head# 2024-04-29T14:53:20.568 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340d3c7d:::benchmark_data_smithi194_186548_object76477:head# 2024-04-29T14:53:20.568 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340d4ebb:::benchmark_data_smithi194_186548_object41985:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340ea11e:::benchmark_data_smithi194_186548_object3450:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:340ff022:::benchmark_data_smithi194_186548_object58708:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34119443:::benchmark_data_smithi194_186548_object6652:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34134734:::benchmark_data_smithi194_186548_object10620:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34134fbd:::benchmark_data_smithi194_186548_object39325:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3413b465:::benchmark_data_smithi194_186548_object19908:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3413bfe4:::benchmark_data_smithi194_186548_object79583:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3413fd34:::benchmark_data_smithi194_186548_object56033:head# 2024-04-29T14:53:20.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34142a67:::benchmark_data_smithi194_186548_object74980:head# 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:20.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:20.577 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34154c15:::benchmark_data_smithi194_186548_object12076:head# 2024-04-29T14:53:20.578 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34157347:::benchmark_data_smithi194_186548_object71536:head# 2024-04-29T14:53:20.579 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3416613d:::benchmark_data_smithi194_186548_object30965:head# 2024-04-29T14:53:20.585 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:341735cf:::benchmark_data_smithi194_186548_object8762:head# 2024-04-29T14:53:20.585 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3418cf2a:::benchmark_data_smithi194_186548_object27936:head# 2024-04-29T14:53:20.585 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34191c85:::benchmark_data_smithi194_186548_object85074:head# 2024-04-29T14:53:20.586 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3419a2ff:::benchmark_data_smithi194_186548_object76196:head# 2024-04-29T14:53:20.586 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:341ac497:::benchmark_data_smithi194_186548_object67461:head# 2024-04-29T14:53:20.586 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:341b9e38:::benchmark_data_smithi194_186548_object82549:head# 2024-04-29T14:53:20.586 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:341ceaf7:::benchmark_data_smithi194_186548_object21894:head# 2024-04-29T14:53:20.591 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:341da6bc:::benchmark_data_smithi194_186548_object29113:head# 2024-04-29T14:53:20.591 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:341dea9d:::benchmark_data_smithi194_186548_object37331:head# 2024-04-29T14:53:20.592 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:341ea21d:::benchmark_data_smithi194_186548_object21838:head# 2024-04-29T14:53:20.592 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34202e16:::benchmark_data_smithi194_186548_object45723:head# 2024-04-29T14:53:20.592 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342063f8:::benchmark_data_smithi194_186548_object36688:head# 2024-04-29T14:53:20.592 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3420900c:::benchmark_data_smithi194_186548_object38029:head# 2024-04-29T14:53:20.598 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3421e577:::benchmark_data_smithi194_186548_object13472:head# 2024-04-29T14:53:20.598 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34220ef4:::benchmark_data_smithi194_186548_object30720:head# 2024-04-29T14:53:20.598 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34231007:::benchmark_data_smithi194_186548_object69386:head# 2024-04-29T14:53:20.598 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342443b6:::benchmark_data_smithi194_186548_object38004:head# 2024-04-29T14:53:20.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34253d77:::benchmark_data_smithi194_186548_object80652:head# 2024-04-29T14:53:20.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34253e0b:::benchmark_data_smithi194_186548_object18529:head# 2024-04-29T14:53:20.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342562b6:::benchmark_data_smithi194_186548_object55657:head# 2024-04-29T14:53:20.600 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3426ba6d:::benchmark_data_smithi194_186548_object82211:head# 2024-04-29T14:53:20.600 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3428497e:::benchmark_data_smithi194_186548_object76626:head# 2024-04-29T14:53:20.603 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342879d3:::benchmark_data_smithi194_186548_object6706:head# 2024-04-29T14:53:20.604 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3428b54e:::benchmark_data_smithi194_186548_object59522:head# 2024-04-29T14:53:20.605 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3429f700:::benchmark_data_smithi194_186548_object43590:head# 2024-04-29T14:53:20.606 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 106948 106932 98.2614 67.0625 0.0067553 0.0101741 2024-04-29T14:53:20.607 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342ab245:::benchmark_data_smithi194_186548_object72752:head# 2024-04-29T14:53:20.608 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342c98d2:::benchmark_data_smithi194_186548_object57184:head# 2024-04-29T14:53:20.608 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342cf760:::benchmark_data_smithi194_186548_object12874:head# 2024-04-29T14:53:20.609 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342d2394:::benchmark_data_smithi194_186548_object57985:head# 2024-04-29T14:53:20.609 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342d5690:::benchmark_data_smithi194_186548_object25923:head# 2024-04-29T14:53:20.609 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342d9f65:::benchmark_data_smithi194_186548_object48911:head# 2024-04-29T14:53:20.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:342e41b9:::benchmark_data_smithi194_186548_object34800:head# 2024-04-29T14:53:20.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3430d18d:::benchmark_data_smithi194_186548_object36605:head# 2024-04-29T14:53:20.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34313111:::benchmark_data_smithi194_186548_object53328:head# 2024-04-29T14:53:20.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34317a15:::benchmark_data_smithi194_186548_object38358:head# 2024-04-29T14:53:20.612 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34322c3a:::benchmark_data_smithi194_186548_object13607:head# 2024-04-29T14:53:20.615 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3433364c:::benchmark_data_smithi194_186548_object4078:head# 2024-04-29T14:53:20.615 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34334ca0:::benchmark_data_smithi194_186548_object10453:head# 2024-04-29T14:53:20.615 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34335d98:::benchmark_data_smithi194_186548_object28113:head# 2024-04-29T14:53:20.615 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3435844c:::benchmark_data_smithi194_186548_object66197:head# 2024-04-29T14:53:20.616 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3435c94d:::benchmark_data_smithi194_186548_object56316:head# 2024-04-29T14:53:20.617 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3436176e:::benchmark_data_smithi194_186548_object61216:head# 2024-04-29T14:53:20.617 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:343962d5:::benchmark_data_smithi194_186548_object25335:head# 2024-04-29T14:53:20.617 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34398a39:::benchmark_data_smithi194_186548_object16258:head# 2024-04-29T14:53:20.617 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:343a8d43:::benchmark_data_smithi194_186548_object58397:head# 2024-04-29T14:53:20.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:343aea4e:::benchmark_data_smithi194_186548_object51672:head# 2024-04-29T14:53:20.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:343c9421:::benchmark_data_smithi194_186548_object15776:head# 2024-04-29T14:53:20.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:343ce4da:::benchmark_data_smithi194_186548_object37541:head# 2024-04-29T14:53:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:343d101a:::benchmark_data_smithi194_186548_object39474:head# 2024-04-29T14:53:20.623 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:343f537f:::benchmark_data_smithi194_186548_object32764:head# 2024-04-29T14:53:20.624 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3442b331:::benchmark_data_smithi194_186548_object33059:head# 2024-04-29T14:53:20.624 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34430185:::benchmark_data_smithi194_186548_object27330:head# 2024-04-29T14:53:20.625 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34432c02:::benchmark_data_smithi194_186548_object58008:head# 2024-04-29T14:53:20.625 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3443d87b:::benchmark_data_smithi194_186548_object60392:head# 2024-04-29T14:53:20.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34442c0d:::benchmark_data_smithi194_186548_object49058:head# 2024-04-29T14:53:20.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344523e5:::benchmark_data_smithi194_186548_object80651:head# 2024-04-29T14:53:20.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34476834:::benchmark_data_smithi194_186548_object14011:head# 2024-04-29T14:53:20.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3448fb2f:::benchmark_data_smithi194_186548_object58070:head# 2024-04-29T14:53:20.629 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344960c4:::benchmark_data_smithi194_186548_object52256:head# 2024-04-29T14:53:20.630 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:20.636 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344a0125:::benchmark_data_smithi194_186548_object25909:head# 2024-04-29T14:53:20.637 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344a321a:::benchmark_data_smithi194_186548_object69585:head# 2024-04-29T14:53:20.637 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344b1f7e:::benchmark_data_smithi194_186548_object40184:head# 2024-04-29T14:53:20.638 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344bc275:::benchmark_data_smithi194_186548_object73984:head# 2024-04-29T14:53:20.641 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344beaa3:::benchmark_data_smithi194_186548_object37098:head# 2024-04-29T14:53:20.641 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344d6e5d:::benchmark_data_smithi194_186548_object52141:head# 2024-04-29T14:53:20.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344d99cb:::benchmark_data_smithi194_186548_object71302:head# 2024-04-29T14:53:20.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344f5b6e:::benchmark_data_smithi194_186548_object72501:head# 2024-04-29T14:53:20.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:344fbce6:::benchmark_data_smithi194_186548_object73027:head# 2024-04-29T14:53:20.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345021a6:::benchmark_data_smithi194_186548_object77876:head# 2024-04-29T14:53:20.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3450bc93:::benchmark_data_smithi194_186548_object17618:head# 2024-04-29T14:53:20.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34521ddd:::benchmark_data_smithi194_186548_object7548:head# 2024-04-29T14:53:20.645 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34526fe9:::benchmark_data_smithi194_186548_object15108:head# 2024-04-29T14:53:20.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34536286:::benchmark_data_smithi194_186548_object75989:head# 2024-04-29T14:53:20.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34540608:::benchmark_data_smithi194_186548_object63669:head# 2024-04-29T14:53:20.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345522cb:::benchmark_data_smithi194_186548_object52400:head# 2024-04-29T14:53:20.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34563bcf:::benchmark_data_smithi194_186548_object73583:head# 2024-04-29T14:53:20.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345810d2:::benchmark_data_smithi194_186548_object20910:head# 2024-04-29T14:53:20.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34595865:::benchmark_data_smithi194_186548_object64474:head# 2024-04-29T14:53:20.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34598399:::benchmark_data_smithi194_186548_object26458:head# 2024-04-29T14:53:20.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3459fe63:::benchmark_data_smithi194_186548_object82738:head# 2024-04-29T14:53:20.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345b3d4b:::benchmark_data_smithi194_186548_object44898:head# 2024-04-29T14:53:20.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345c0240:::benchmark_data_smithi194_186548_object83055:head# 2024-04-29T14:53:20.653 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345cdd46:::benchmark_data_smithi194_186548_object79334:head# 2024-04-29T14:53:20.657 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345ced10:::benchmark_data_smithi194_186548_object66328:head# 2024-04-29T14:53:20.657 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345d1ea6:::benchmark_data_smithi194_186548_object2604:head# 2024-04-29T14:53:20.658 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345e9ead:::benchmark_data_smithi194_186548_object51376:head# 2024-04-29T14:53:20.658 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:345fb9f7:::benchmark_data_smithi194_186548_object48966:head# 2024-04-29T14:53:20.658 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34608bbf:::benchmark_data_smithi194_186548_object39871:head# 2024-04-29T14:53:20.659 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3460c6e2:::benchmark_data_smithi194_186548_object23616:head# 2024-04-29T14:53:20.660 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3461255f:::benchmark_data_smithi194_186548_object12778:head# 2024-04-29T14:53:20.662 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34614c44:::benchmark_data_smithi194_186548_object27494:head# 2024-04-29T14:53:20.662 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346215dd:::benchmark_data_smithi194_186548_object42505:head# 2024-04-29T14:53:20.663 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34626cab:::benchmark_data_smithi194_186548_object20482:head# 2024-04-29T14:53:20.664 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34627e0e:::benchmark_data_smithi194_186548_object42242:head# 2024-04-29T14:53:20.665 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3465ba09:::benchmark_data_smithi194_186548_object33349:head# 2024-04-29T14:53:20.665 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34663fea:::benchmark_data_smithi194_186548_object56561:head# 2024-04-29T14:53:20.666 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3466c853:::benchmark_data_smithi194_186548_object33099:head# 2024-04-29T14:53:20.666 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34684950:::benchmark_data_smithi194_186548_object50225:head# 2024-04-29T14:53:20.666 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34695e06:::benchmark_data_smithi194_186548_object10296:head# 2024-04-29T14:53:20.667 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346a1b04:::benchmark_data_smithi194_186548_object43230:head# 2024-04-29T14:53:20.667 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346a8a95:::benchmark_data_smithi194_186548_object6757:head# 2024-04-29T14:53:20.670 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346bad46:::benchmark_data_smithi194_186548_object83769:head# 2024-04-29T14:53:20.671 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346bd38e:::benchmark_data_smithi194_186548_object79149:head# 2024-04-29T14:53:20.672 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346bef76:::benchmark_data_smithi194_186548_object43710:head# 2024-04-29T14:53:20.672 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346c2177:::benchmark_data_smithi194_186548_object49529:head# 2024-04-29T14:53:20.673 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346c62ce:::benchmark_data_smithi194_186548_object20842:head# 2024-04-29T14:53:20.673 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346d4d5b:::benchmark_data_smithi194_186548_object70158:head# 2024-04-29T14:53:20.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346dcfb0:::benchmark_data_smithi194_186548_object47944:head# 2024-04-29T14:53:20.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:346e234d:::benchmark_data_smithi194_186548_object9028:head# 2024-04-29T14:53:20.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34710df7:::benchmark_data_smithi194_186548_object25159:head# 2024-04-29T14:53:20.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3472835d:::benchmark_data_smithi194_186548_object556:head# 2024-04-29T14:53:20.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3472cd3b:::benchmark_data_smithi194_186548_object11340:head# 2024-04-29T14:53:20.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3474a5cf:::benchmark_data_smithi194_186548_object41272:head# 2024-04-29T14:53:20.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3475133b:::benchmark_data_smithi194_186548_object83554:head# 2024-04-29T14:53:20.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3475b1c2:::benchmark_data_smithi194_186548_object35687:head# 2024-04-29T14:53:20.679 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347662fa:::benchmark_data_smithi194_186548_object54280:head# 2024-04-29T14:53:20.679 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:20.680 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347a5bfd:::benchmark_data_smithi194_186548_object1381:head# 2024-04-29T14:53:20.683 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347af2fa:::benchmark_data_smithi194_186548_object23697:head# 2024-04-29T14:53:20.683 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347be683:::benchmark_data_smithi194_186548_object21289:head# 2024-04-29T14:53:20.684 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347d2007:::benchmark_data_smithi194_186548_object32131:head# 2024-04-29T14:53:20.684 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347e9d67:::benchmark_data_smithi194_186548_object43060:head# 2024-04-29T14:53:20.685 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347f5ba4:::benchmark_data_smithi194_186548_object18231:head# 2024-04-29T14:53:20.685 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347faddb:::benchmark_data_smithi194_186548_object26446:head# 2024-04-29T14:53:20.686 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:347feca8:::benchmark_data_smithi194_186548_object77099:head# 2024-04-29T14:53:20.686 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348058c5:::benchmark_data_smithi194_186548_object84797:head# 2024-04-29T14:53:20.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3480b11c:::benchmark_data_smithi194_186548_object56218:head# 2024-04-29T14:53:20.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34810b29:::benchmark_data_smithi194_186548_object80631:head# 2024-04-29T14:53:20.688 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34817e04:::benchmark_data_smithi194_186548_object49291:head# 2024-04-29T14:53:20.688 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3481dfaa:::benchmark_data_smithi194_186548_object69763:head# 2024-04-29T14:53:20.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3482e456:::benchmark_data_smithi194_186548_object77007:head# 2024-04-29T14:53:20.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34838e8c:::benchmark_data_smithi194_186548_object49323:head# 2024-04-29T14:53:20.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34852540:::benchmark_data_smithi194_186548_object1631:head# 2024-04-29T14:53:20.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34859f1e:::benchmark_data_smithi194_186548_object74253:head# 2024-04-29T14:53:20.694 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3485b59b:::benchmark_data_smithi194_186548_object84231:head# 2024-04-29T14:53:20.694 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3486c9b8:::benchmark_data_smithi194_186548_object15765:head# 2024-04-29T14:53:20.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3488197c:::benchmark_data_smithi194_186548_object35492:head# 2024-04-29T14:53:20.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348bdef2:::benchmark_data_smithi194_186548_object57109:head# 2024-04-29T14:53:20.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348cdd73:::benchmark_data_smithi194_186548_object7976:head# 2024-04-29T14:53:20.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348d6474:::benchmark_data_smithi194_186548_object20561:head# 2024-04-29T14:53:20.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348de1c0:::benchmark_data_smithi194_186548_object3061:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348e4ad7:::benchmark_data_smithi194_186548_object5394:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348f7e22:::benchmark_data_smithi194_186548_object60858:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:348f97fe:::benchmark_data_smithi194_186548_object21983:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34900e11:::benchmark_data_smithi194_186548_object56494:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34908fc3:::benchmark_data_smithi194_186548_object39616:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3490f64e:::benchmark_data_smithi194_186548_object56187:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3491c415:::benchmark_data_smithi194_186548_object32726:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3491ea6d:::benchmark_data_smithi194_186548_object52375:head# 2024-04-29T14:53:20.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34924ba1:::benchmark_data_smithi194_186548_object72547:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3492cd04:::benchmark_data_smithi194_186548_object21988:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3492dc1c:::benchmark_data_smithi194_186548_object34843:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3492e174:::benchmark_data_smithi194_186548_object18242:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34931f53:::benchmark_data_smithi194_186548_object41698:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:349360ef:::benchmark_data_smithi194_186548_object22674:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34936d5d:::benchmark_data_smithi194_186548_object50600:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3493babc:::benchmark_data_smithi194_186548_object41490:head# 2024-04-29T14:53:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34952d11:::benchmark_data_smithi194_186548_object84201:head# 2024-04-29T14:53:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3495cc9e:::benchmark_data_smithi194_186548_object40114:head# 2024-04-29T14:53:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3495cd65:::benchmark_data_smithi194_186548_object63779:head# 2024-04-29T14:53:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3496a421:::benchmark_data_smithi194_186548_object75064:head# 2024-04-29T14:53:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3498220f:::benchmark_data_smithi194_186548_object23895:head# 2024-04-29T14:53:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:349834be:::benchmark_data_smithi194_186548_object60827:head# 2024-04-29T14:53:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34984a6a:::benchmark_data_smithi194_186548_object15433:head# 2024-04-29T14:53:20.707 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3499096f:::benchmark_data_smithi194_186548_object1550:head# 2024-04-29T14:53:20.709 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:20.710 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34990995:::benchmark_data_smithi194_186548_object10850:head# 2024-04-29T14:53:20.711 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:349a4980:::benchmark_data_smithi194_186548_object16220:head# 2024-04-29T14:53:20.711 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:349b3c07:::benchmark_data_smithi194_186548_object40242:head# 2024-04-29T14:53:20.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:349d7f8e:::benchmark_data_smithi194_186548_object33584:head# 2024-04-29T14:53:20.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:349f22ad:::benchmark_data_smithi194_186548_object10069:head# 2024-04-29T14:53:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:20.714 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:349f959a:::benchmark_data_smithi194_186548_object33368:head# 2024-04-29T14:53:20.714 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34a27a43:::benchmark_data_smithi194_186548_object68429:head# 2024-04-29T14:53:20.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:20.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34a7a753:::benchmark_data_smithi194_186548_object67251:head# 2024-04-29T14:53:20.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34a821c8:::benchmark_data_smithi194_186548_object4887:head# 2024-04-29T14:53:20.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34a8d7cb:::benchmark_data_smithi194_186548_object5663:head# 2024-04-29T14:53:20.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34a9360f:::benchmark_data_smithi194_186548_object41945:head# 2024-04-29T14:53:20.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34a9696c:::benchmark_data_smithi194_186548_object40195:head# 2024-04-29T14:53:20.724 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34a96c20:::benchmark_data_smithi194_186548_object80235:head# 2024-04-29T14:53:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34aa5ad1:::benchmark_data_smithi194_186548_object54284:head# 2024-04-29T14:53:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34aa6e1e:::benchmark_data_smithi194_186548_object60259:head# 2024-04-29T14:53:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34aad643:::benchmark_data_smithi194_186548_object17609:head# 2024-04-29T14:53:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ab6419:::benchmark_data_smithi194_186548_object66932:head# 2024-04-29T14:53:20.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ac13fe:::benchmark_data_smithi194_186548_object55791:head# 2024-04-29T14:53:20.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ac9af0:::benchmark_data_smithi194_186548_object62172:head# 2024-04-29T14:53:20.730 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34acd06c:::benchmark_data_smithi194_186548_object5356:head# 2024-04-29T14:53:20.730 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ad45b5:::benchmark_data_smithi194_186548_object30076:head# 2024-04-29T14:53:20.731 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ad6d3e:::benchmark_data_smithi194_186548_object11222:head# 2024-04-29T14:53:20.731 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34adaaef:::benchmark_data_smithi194_186548_object70608:head# 2024-04-29T14:53:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ade4ce:::benchmark_data_smithi194_186548_object8096:head# 2024-04-29T14:53:20.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ae7912:::benchmark_data_smithi194_186548_object37648:head# 2024-04-29T14:53:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34aea253:::benchmark_data_smithi194_186548_object52673:head# 2024-04-29T14:53:20.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34aea88a:::benchmark_data_smithi194_186548_object24279:head# 2024-04-29T14:53:20.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34afb9ad:::benchmark_data_smithi194_186548_object47045:head# 2024-04-29T14:53:20.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b00841:::benchmark_data_smithi194_186548_object23:head# 2024-04-29T14:53:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b010e2:::benchmark_data_smithi194_186548_object59400:head# 2024-04-29T14:53:20.735 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b0bb94:::benchmark_data_smithi194_186548_object85059:head# 2024-04-29T14:53:20.736 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b2226e:::benchmark_data_smithi194_186548_object77699:head# 2024-04-29T14:53:20.736 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b29b10:::benchmark_data_smithi194_186548_object84845:head# 2024-04-29T14:53:20.736 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b2be52:::benchmark_data_smithi194_186548_object59085:head# 2024-04-29T14:53:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b3d629:::benchmark_data_smithi194_186548_object22556:head# 2024-04-29T14:53:20.738 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b4c427:::benchmark_data_smithi194_186548_object67830:head# 2024-04-29T14:53:20.739 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b4d775:::benchmark_data_smithi194_186548_object12627:head# 2024-04-29T14:53:20.740 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b4f5f2:::benchmark_data_smithi194_186548_object30171:head# 2024-04-29T14:53:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b5a03b:::benchmark_data_smithi194_186548_object38628:head# 2024-04-29T14:53:20.741 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b5bde0:::benchmark_data_smithi194_186548_object80093:head# 2024-04-29T14:53:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b74272:::benchmark_data_smithi194_186548_object62774:head# 2024-04-29T14:53:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b7427e:::benchmark_data_smithi194_186548_object6816:head# 2024-04-29T14:53:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34b9007a:::benchmark_data_smithi194_186548_object63589:head# 2024-04-29T14:53:20.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34bc8a50:::benchmark_data_smithi194_186548_object49859:head# 2024-04-29T14:53:20.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34bede5e:::benchmark_data_smithi194_186548_object47859:head# 2024-04-29T14:53:20.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34bfc26a:::benchmark_data_smithi194_186548_object9691:head# 2024-04-29T14:53:20.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c097cb:::benchmark_data_smithi194_186548_object4247:head# 2024-04-29T14:53:20.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c111d9:::benchmark_data_smithi194_186548_object75517:head# 2024-04-29T14:53:20.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c23c7e:::benchmark_data_smithi194_186548_object60714:head# 2024-04-29T14:53:20.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c31618:::benchmark_data_smithi194_186548_object65167:head# 2024-04-29T14:53:20.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c3f27d:::benchmark_data_smithi194_186548_object48866:head# 2024-04-29T14:53:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c47f27:::benchmark_data_smithi194_186548_object80525:head# 2024-04-29T14:53:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c49bd3:::benchmark_data_smithi194_186548_object17439:head# 2024-04-29T14:53:20.749 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c60714:::benchmark_data_smithi194_186548_object2128:head# 2024-04-29T14:53:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c64213:::benchmark_data_smithi194_186548_object68775:head# 2024-04-29T14:53:20.750 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34c7cbce:::benchmark_data_smithi194_186548_object14097:head# 2024-04-29T14:53:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34cb197c:::benchmark_data_smithi194_186548_object37882:head# 2024-04-29T14:53:20.753 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34cbba8c:::benchmark_data_smithi194_186548_object46931:head# 2024-04-29T14:53:20.754 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34cdadb4:::benchmark_data_smithi194_186548_object31781:head# 2024-04-29T14:53:20.755 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34cdc2bd:::benchmark_data_smithi194_186548_object32426:head# 2024-04-29T14:53:20.755 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ce84db:::benchmark_data_smithi194_186548_object65015:head# 2024-04-29T14:53:20.756 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34cf0495:::benchmark_data_smithi194_186548_object61557:head# 2024-04-29T14:53:20.756 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34cf56c6:::benchmark_data_smithi194_186548_object82463:head# 2024-04-29T14:53:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d02c56:::benchmark_data_smithi194_186548_object32207:head# 2024-04-29T14:53:20.758 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d041de:::benchmark_data_smithi194_186548_object51037:head# 2024-04-29T14:53:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d0bd32:::benchmark_data_smithi194_186548_object10723:head# 2024-04-29T14:53:20.759 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d0ee6c:::benchmark_data_smithi194_186548_object2095:head# 2024-04-29T14:53:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d23c8b:::benchmark_data_smithi194_186548_object78890:head# 2024-04-29T14:53:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d3ab24:::benchmark_data_smithi194_186548_object3432:head# 2024-04-29T14:53:20.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d3d3a8:::benchmark_data_smithi194_186548_object59355:head# 2024-04-29T14:53:20.762 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d448f5:::benchmark_data_smithi194_186548_object80225:head# 2024-04-29T14:53:20.765 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d459a1:::benchmark_data_smithi194_186548_object70630:head# 2024-04-29T14:53:20.766 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d47546:::benchmark_data_smithi194_186548_object21359:head# 2024-04-29T14:53:20.766 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d592aa:::benchmark_data_smithi194_186548_object46099:head# 2024-04-29T14:53:20.767 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d5c6bd:::benchmark_data_smithi194_186548_object18703:head# 2024-04-29T14:53:20.770 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d5f8aa:::benchmark_data_smithi194_186548_object40877:head# 2024-04-29T14:53:20.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d66949:::benchmark_data_smithi194_186548_object54406:head# 2024-04-29T14:53:20.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d66c9b:::benchmark_data_smithi194_186548_object83940:head# 2024-04-29T14:53:20.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d6fb7b:::benchmark_data_smithi194_186548_object70754:head# 2024-04-29T14:53:20.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d77726:::benchmark_data_smithi194_186548_object9643:head# 2024-04-29T14:53:20.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d88de0:::benchmark_data_smithi194_186548_object13584:head# 2024-04-29T14:53:20.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d8abeb:::benchmark_data_smithi194_186548_object13618:head# 2024-04-29T14:53:20.776 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d9c40b:::benchmark_data_smithi194_186548_object54547:head# 2024-04-29T14:53:20.779 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34d9f105:::benchmark_data_smithi194_186548_object45973:head# 2024-04-29T14:53:20.779 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34db46cd:::benchmark_data_smithi194_186548_object1865:head# 2024-04-29T14:53:20.780 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34dccf3a:::benchmark_data_smithi194_186548_object56105:head# 2024-04-29T14:53:20.780 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34dcf768:::benchmark_data_smithi194_186548_object22344:head# 2024-04-29T14:53:20.781 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34dd0237:::benchmark_data_smithi194_186548_object20042:head# 2024-04-29T14:53:20.781 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34dd1594:::benchmark_data_smithi194_186548_object23651:head# 2024-04-29T14:53:20.781 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ddcb60:::benchmark_data_smithi194_186548_object11863:head# 2024-04-29T14:53:20.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34de02ac:::benchmark_data_smithi194_186548_object41602:head# 2024-04-29T14:53:20.786 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34de3009:::benchmark_data_smithi194_186548_object18492:head# 2024-04-29T14:53:20.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34dec983:::benchmark_data_smithi194_186548_object74029:head# 2024-04-29T14:53:20.790 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34dfb726:::benchmark_data_smithi194_186548_object15528:head# 2024-04-29T14:53:20.790 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34dfd4d6:::benchmark_data_smithi194_186548_object31332:head# 2024-04-29T14:53:20.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e0636c:::benchmark_data_smithi194_186548_object67174:head# 2024-04-29T14:53:20.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e1514c:::benchmark_data_smithi194_186548_object9690:head# 2024-04-29T14:53:20.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e225e8:::benchmark_data_smithi194_186548_object19180:head# 2024-04-29T14:53:20.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e395cb:::benchmark_data_smithi194_186548_object63425:head# 2024-04-29T14:53:20.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e3f6ab:::benchmark_data_smithi194_186548_object10159:head# 2024-04-29T14:53:20.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e4ad41:::benchmark_data_smithi194_186548_object55994:head# 2024-04-29T14:53:20.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e4fe2f:::benchmark_data_smithi194_186548_object44603:head# 2024-04-29T14:53:20.796 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e7ce56:::benchmark_data_smithi194_186548_object65853:head# 2024-04-29T14:53:20.796 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e8aab3:::benchmark_data_smithi194_186548_object51455:head# 2024-04-29T14:53:20.797 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e8e8bc:::benchmark_data_smithi194_186548_object71789:head# 2024-04-29T14:53:20.797 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e8f474:::benchmark_data_smithi194_186548_object73856:head# 2024-04-29T14:53:20.798 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34e9d9d1:::benchmark_data_smithi194_186548_object21762:head# 2024-04-29T14:53:20.798 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ea321e:::benchmark_data_smithi194_186548_object36488:head# 2024-04-29T14:53:20.799 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34eb5460:::benchmark_data_smithi194_186548_object1820:head# 2024-04-29T14:53:20.799 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34eb7ed9:::benchmark_data_smithi194_186548_object31205:head# 2024-04-29T14:53:20.802 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ecd4ec:::benchmark_data_smithi194_186548_object5346:head# 2024-04-29T14:53:20.803 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ed05cf:::benchmark_data_smithi194_186548_object35079:head# 2024-04-29T14:53:20.803 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ed1255:::benchmark_data_smithi194_186548_object20861:head# 2024-04-29T14:53:20.804 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34ed5e61:::benchmark_data_smithi194_186548_object10054:head# 2024-04-29T14:53:20.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f1f90d:::benchmark_data_smithi194_186548_object69910:head# 2024-04-29T14:53:20.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f22d76:::benchmark_data_smithi194_186548_object9835:head# 2024-04-29T14:53:20.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f2e041:::benchmark_data_smithi194_186548_object33095:head# 2024-04-29T14:53:20.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f3223d:::benchmark_data_smithi194_186548_object5376:head# 2024-04-29T14:53:20.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f5501b:::benchmark_data_smithi194_186548_object65535:head# 2024-04-29T14:53:20.810 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f59c79:::benchmark_data_smithi194_186548_object72699:head# 2024-04-29T14:53:20.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f7cdd7:::benchmark_data_smithi194_186548_object52643:head# 2024-04-29T14:53:20.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f7e0b5:::benchmark_data_smithi194_186548_object7028:head# 2024-04-29T14:53:20.812 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f7f546:::benchmark_data_smithi194_186548_object21114:head# 2024-04-29T14:53:20.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f81e87:::benchmark_data_smithi194_186548_object70483:head# 2024-04-29T14:53:20.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f94bc1:::benchmark_data_smithi194_186548_object68474:head# 2024-04-29T14:53:20.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f9911f:::benchmark_data_smithi194_186548_object32841:head# 2024-04-29T14:53:20.814 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34f9f791:::benchmark_data_smithi194_186548_object44485:head# 2024-04-29T14:53:20.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:53:20.818 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fa2692:::benchmark_data_smithi194_186548_object34751:head# 2024-04-29T14:53:20.818 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fa936b:::benchmark_data_smithi194_186548_object56200:head# 2024-04-29T14:53:20.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fac20a:::benchmark_data_smithi194_186548_object76060:head# 2024-04-29T14:53:20.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fafe95:::benchmark_data_smithi194_186548_object20862:head# 2024-04-29T14:53:20.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fc6bc7:::benchmark_data_smithi194_186548_object63505:head# 2024-04-29T14:53:20.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fc903e:::benchmark_data_smithi194_186548_object76431:head# 2024-04-29T14:53:20.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fc9d83:::benchmark_data_smithi194_186548_object44861:head# 2024-04-29T14:53:20.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fdeb22:::benchmark_data_smithi194_186548_object37124:head# 2024-04-29T14:53:20.823 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:34fe1dc6:::benchmark_data_smithi194_186548_object46737:head# 2024-04-29T14:53:20.823 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:350177b7:::benchmark_data_smithi194_186548_object45074:head# 2024-04-29T14:53:20.823 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3501e14f:::benchmark_data_smithi194_186548_object51924:head# 2024-04-29T14:53:20.823 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3501ebc6:::benchmark_data_smithi194_186548_object5604:head# 2024-04-29T14:53:20.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3502c071:::benchmark_data_smithi194_186548_object10090:head# 2024-04-29T14:53:20.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35034672:::benchmark_data_smithi194_186548_object18076:head# 2024-04-29T14:53:20.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3503edcb:::benchmark_data_smithi194_186548_object15714:head# 2024-04-29T14:53:20.825 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3504f766:::benchmark_data_smithi194_186548_object16167:head# 2024-04-29T14:53:20.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35059d78:::benchmark_data_smithi194_186548_object39124:head# 2024-04-29T14:53:20.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3505f37d:::benchmark_data_smithi194_186548_object54181:head# 2024-04-29T14:53:20.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35071646:::benchmark_data_smithi194_186548_object58114:head# 2024-04-29T14:53:20.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3508e921:::benchmark_data_smithi194_186548_object81953:head# 2024-04-29T14:53:20.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3509b99e:::benchmark_data_smithi194_186548_object59669:head# 2024-04-29T14:53:20.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:350dcfb8:::benchmark_data_smithi194_186548_object64974:head# 2024-04-29T14:53:20.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:350e8629:::benchmark_data_smithi194_186548_object51813:head# 2024-04-29T14:53:20.829 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:350ead35:::benchmark_data_smithi194_186548_object8393:head# 2024-04-29T14:53:20.829 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:350fa287:::benchmark_data_smithi194_186548_object80743:head# 2024-04-29T14:53:20.829 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35103203:::benchmark_data_smithi194_186548_object84772:head# 2024-04-29T14:53:20.830 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35117913:::benchmark_data_smithi194_186548_object62966:head# 2024-04-29T14:53:20.830 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35119a52:::benchmark_data_smithi194_186548_object2405:head# 2024-04-29T14:53:20.830 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35128576:::benchmark_data_smithi194_186548_object45014:head# 2024-04-29T14:53:20.831 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35130532:::benchmark_data_smithi194_186548_object12672:head# 2024-04-29T14:53:20.831 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3514777c:::benchmark_data_smithi194_186548_object51044:head# 2024-04-29T14:53:20.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3517f31a:::benchmark_data_smithi194_186548_object53660:head# 2024-04-29T14:53:20.833 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35187ccc:::benchmark_data_smithi194_186548_object42201:head# 2024-04-29T14:53:20.833 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3519020d:::benchmark_data_smithi194_186548_object34739:head# 2024-04-29T14:53:20.834 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3519d13e:::benchmark_data_smithi194_186548_object81957:head# 2024-04-29T14:53:20.834 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:351bf5f8:::benchmark_data_smithi194_186548_object20918:head# 2024-04-29T14:53:20.835 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:351cd285:::benchmark_data_smithi194_186548_object5422:head# 2024-04-29T14:53:20.835 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:351ce444:::benchmark_data_smithi194_186548_object77350:head# 2024-04-29T14:53:20.835 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:351d051d:::benchmark_data_smithi194_186548_object28575:head# 2024-04-29T14:53:20.836 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:351d2dbf:::benchmark_data_smithi194_186548_object80122:head# 2024-04-29T14:53:20.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3520dad7:::benchmark_data_smithi194_186548_object12172:head# 2024-04-29T14:53:20.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352203f6:::benchmark_data_smithi194_186548_object62302:head# 2024-04-29T14:53:20.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35247437:::benchmark_data_smithi194_186548_object84235:head# 2024-04-29T14:53:20.840 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35248717:::benchmark_data_smithi194_186548_object6600:head# 2024-04-29T14:53:20.841 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3527638b:::benchmark_data_smithi194_186548_object14723:head# 2024-04-29T14:53:20.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3529822a:::benchmark_data_smithi194_186548_object10025:head# 2024-04-29T14:53:20.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35299155:::benchmark_data_smithi194_186548_object83461:head# 2024-04-29T14:53:20.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352991fa:::benchmark_data_smithi194_186548_object23891:head# 2024-04-29T14:53:20.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352a7a0e:::benchmark_data_smithi194_186548_object4752:head# 2024-04-29T14:53:20.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352bdee3:::benchmark_data_smithi194_186548_object16995:head# 2024-04-29T14:53:20.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352c1421:::benchmark_data_smithi194_186548_object43842:head# 2024-04-29T14:53:20.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352c31fc:::benchmark_data_smithi194_186548_object69149:head# 2024-04-29T14:53:20.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352c5fc5:::benchmark_data_smithi194_186548_object31886:head# 2024-04-29T14:53:20.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352ece50:::benchmark_data_smithi194_186548_object41389:head# 2024-04-29T14:53:20.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352f345a:::benchmark_data_smithi194_186548_object62245:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:352f6711:::benchmark_data_smithi194_186548_object33805:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35303124:::benchmark_data_smithi194_186548_object50655:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35304f89:::benchmark_data_smithi194_186548_object72883:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3530aca2:::benchmark_data_smithi194_186548_object6409:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35315e2c:::benchmark_data_smithi194_186548_object83733:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35322852:::benchmark_data_smithi194_186548_object28527:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35335966:::benchmark_data_smithi194_186548_object57809:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3535b6de:::benchmark_data_smithi194_186548_object45291:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35370420:::benchmark_data_smithi194_186548_object9101:head# 2024-04-29T14:53:20.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35398ccd:::benchmark_data_smithi194_186548_object4960:head# 2024-04-29T14:53:20.849 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353a1acd:::benchmark_data_smithi194_186548_object24518:head# 2024-04-29T14:53:20.849 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353b16f4:::benchmark_data_smithi194_186548_object48342:head# 2024-04-29T14:53:20.849 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353b52a4:::benchmark_data_smithi194_186548_object53076:head# 2024-04-29T14:53:20.850 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353bd328:::benchmark_data_smithi194_186548_object49053:head# 2024-04-29T14:53:20.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353cdce0:::benchmark_data_smithi194_186548_object50596:head# 2024-04-29T14:53:20.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353d176b:::benchmark_data_smithi194_186548_object67468:head# 2024-04-29T14:53:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353f2cb4:::benchmark_data_smithi194_186548_object33830:head# 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353f9ca4:::benchmark_data_smithi194_186548_object18233:head# 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:353ff0b7:::benchmark_data_smithi194_186548_object53101:head# 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35405880:::benchmark_data_smithi194_186548_object57233:head# 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3541832b:::benchmark_data_smithi194_186548_object54242:head# 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3541cc44:::benchmark_data_smithi194_186548_object7373:head# 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354371b1:::benchmark_data_smithi194_186548_object34528:head# 2024-04-29T14:53:20.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3544402e:::benchmark_data_smithi194_186548_object58527:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3545c949:::benchmark_data_smithi194_186548_object77079:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354626f5:::benchmark_data_smithi194_186548_object53819:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3546aeff:::benchmark_data_smithi194_186548_object59898:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354792ac:::benchmark_data_smithi194_186548_object15231:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3548a88b:::benchmark_data_smithi194_186548_object32104:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35491606:::benchmark_data_smithi194_186548_object15832:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3549438d:::benchmark_data_smithi194_186548_object27736:head# 2024-04-29T14:53:20.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35495bb7:::benchmark_data_smithi194_186548_object38938:head# 2024-04-29T14:53:20.862 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354981b3:::benchmark_data_smithi194_186548_object80923:head# 2024-04-29T14:53:20.862 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354b3ebb:::benchmark_data_smithi194_186548_object75957:head# 2024-04-29T14:53:20.862 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354ba1d6:::benchmark_data_smithi194_186548_object52424:head# 2024-04-29T14:53:20.863 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354bc69b:::benchmark_data_smithi194_186548_object16114:head# 2024-04-29T14:53:20.863 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354be95f:::benchmark_data_smithi194_186548_object27463:head# 2024-04-29T14:53:20.864 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354d7d72:::benchmark_data_smithi194_186548_object77715:head# 2024-04-29T14:53:20.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354e1661:::benchmark_data_smithi194_186548_object1443:head# 2024-04-29T14:53:20.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:354f0947:::benchmark_data_smithi194_186548_object40701:head# 2024-04-29T14:53:20.865 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35519777:::benchmark_data_smithi194_186548_object37915:head# 2024-04-29T14:53:20.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3552d2b5:::benchmark_data_smithi194_186548_object19425:head# 2024-04-29T14:53:20.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35540870:::benchmark_data_smithi194_186548_object73420:head# 2024-04-29T14:53:20.867 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35548d22:::benchmark_data_smithi194_186548_object13817:head# 2024-04-29T14:53:20.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3554bbed:::benchmark_data_smithi194_186548_object72122:head# 2024-04-29T14:53:20.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3554e9ec:::benchmark_data_smithi194_186548_object44584:head# 2024-04-29T14:53:20.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35555356:::benchmark_data_smithi194_186548_object14212:head# 2024-04-29T14:53:20.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35555eea:::benchmark_data_smithi194_186548_object46256:head# 2024-04-29T14:53:20.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35569927:::benchmark_data_smithi194_186548_object42745:head# 2024-04-29T14:53:20.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35591ac5:::benchmark_data_smithi194_186548_object55894:head# 2024-04-29T14:53:20.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35593a15:::benchmark_data_smithi194_186548_object52322:head# 2024-04-29T14:53:20.871 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3559edd5:::benchmark_data_smithi194_186548_object10869:head# 2024-04-29T14:53:20.872 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:355a035e:::benchmark_data_smithi194_186548_object4151:head# 2024-04-29T14:53:20.872 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:355ad35e:::benchmark_data_smithi194_186548_object11820:head# 2024-04-29T14:53:20.873 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:355b334d:::benchmark_data_smithi194_186548_object82323:head# 2024-04-29T14:53:20.873 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:355db0e3:::benchmark_data_smithi194_186548_object69099:head# 2024-04-29T14:53:20.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:355fb14f:::benchmark_data_smithi194_186548_object10252:head# 2024-04-29T14:53:20.875 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:355fecc0:::benchmark_data_smithi194_186548_object24389:head# 2024-04-29T14:53:20.875 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35602b56:::benchmark_data_smithi194_186548_object57133:head# 2024-04-29T14:53:20.876 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356103ba:::benchmark_data_smithi194_186548_object7211:head# 2024-04-29T14:53:20.876 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35628420:::benchmark_data_smithi194_186548_object18155:head# 2024-04-29T14:53:20.876 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3562c2fc:::benchmark_data_smithi194_186548_object71622:head# 2024-04-29T14:53:20.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35669f15:::benchmark_data_smithi194_186548_object24585:head# 2024-04-29T14:53:20.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356804bd:::benchmark_data_smithi194_186548_object31457:head# 2024-04-29T14:53:20.878 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35683ca6:::benchmark_data_smithi194_186548_object60116:head# 2024-04-29T14:53:20.878 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3568472a:::benchmark_data_smithi194_186548_object68520:head# 2024-04-29T14:53:20.879 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3569475d:::benchmark_data_smithi194_186548_object63654:head# 2024-04-29T14:53:20.879 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356a83bf:::benchmark_data_smithi194_186548_object70576:head# 2024-04-29T14:53:20.880 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356aa8b2:::benchmark_data_smithi194_186548_object35760:head# 2024-04-29T14:53:20.880 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356c8dc0:::benchmark_data_smithi194_186548_object52800:head# 2024-04-29T14:53:20.880 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356d4948:::benchmark_data_smithi194_186548_object59779:head# 2024-04-29T14:53:20.881 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356d5081:::benchmark_data_smithi194_186548_object42080:head# 2024-04-29T14:53:20.883 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356ee0d6:::benchmark_data_smithi194_186548_object37534:head# 2024-04-29T14:53:20.884 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:356f4a90:::benchmark_data_smithi194_186548_object40414:head# 2024-04-29T14:53:20.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35702f61:::benchmark_data_smithi194_186548_object32915:head# 2024-04-29T14:53:20.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357069fb:::benchmark_data_smithi194_186548_object55966:head# 2024-04-29T14:53:20.894 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35710f98:::benchmark_data_smithi194_186548_object75898:head# 2024-04-29T14:53:20.894 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35714785:::benchmark_data_smithi194_186548_object68221:head# 2024-04-29T14:53:20.894 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357191db:::benchmark_data_smithi194_186548_object58822:head# 2024-04-29T14:53:20.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3571a5f8:::benchmark_data_smithi194_186548_object6725:head# 2024-04-29T14:53:20.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3572350b:::benchmark_data_smithi194_186548_object13097:head# 2024-04-29T14:53:20.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3572b3be:::benchmark_data_smithi194_186548_object7190:head# 2024-04-29T14:53:20.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35731a07:::benchmark_data_smithi194_186548_object76246:head# 2024-04-29T14:53:20.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3573753b:::benchmark_data_smithi194_186548_object73273:head# 2024-04-29T14:53:20.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357450a8:::benchmark_data_smithi194_186548_object82035:head# 2024-04-29T14:53:20.900 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3574537a:::benchmark_data_smithi194_186548_object14286:head# 2024-04-29T14:53:20.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35749dd0:::benchmark_data_smithi194_186548_object27888:head# 2024-04-29T14:53:20.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3574e7a5:::benchmark_data_smithi194_186548_object5040:head# 2024-04-29T14:53:20.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357638bf:::benchmark_data_smithi194_186548_object62294:head# 2024-04-29T14:53:20.902 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3576c773:::benchmark_data_smithi194_186548_object45103:head# 2024-04-29T14:53:20.904 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3577b5bd:::benchmark_data_smithi194_186548_object45090:head# 2024-04-29T14:53:20.904 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3579084e:::benchmark_data_smithi194_186548_object50284:head# 2024-04-29T14:53:20.904 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357b612c:::benchmark_data_smithi194_186548_object71586:head# 2024-04-29T14:53:20.905 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357bdb62:::benchmark_data_smithi194_186548_object26823:head# 2024-04-29T14:53:20.905 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357cc084:::benchmark_data_smithi194_186548_object74212:head# 2024-04-29T14:53:20.905 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357e41da:::benchmark_data_smithi194_186548_object37970:head# 2024-04-29T14:53:20.906 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357e91ce:::benchmark_data_smithi194_186548_object67943:head# 2024-04-29T14:53:20.907 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357ec402:::benchmark_data_smithi194_186548_object24767:head# 2024-04-29T14:53:20.907 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357ee94d:::benchmark_data_smithi194_186548_object27327:head# 2024-04-29T14:53:20.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:357f6171:::benchmark_data_smithi194_186548_object31961:head# 2024-04-29T14:53:20.908 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35805770:::benchmark_data_smithi194_186548_object42732:head# 2024-04-29T14:53:20.909 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35812c15:::benchmark_data_smithi194_186548_object48951:head# 2024-04-29T14:53:20.909 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35821d70:::benchmark_data_smithi194_186548_object6751:head# 2024-04-29T14:53:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35825863:::benchmark_data_smithi194_186548_object16172:head# 2024-04-29T14:53:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3583a327:::benchmark_data_smithi194_186548_object39208:head# 2024-04-29T14:53:20.918 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3583d0d0:::benchmark_data_smithi194_186548_object13572:head# 2024-04-29T14:53:20.918 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358649bd:::benchmark_data_smithi194_186548_object75477:head# 2024-04-29T14:53:20.918 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35865036:::benchmark_data_smithi194_186548_object45539:head# 2024-04-29T14:53:20.919 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3586be29:::benchmark_data_smithi194_186548_object81222:head# 2024-04-29T14:53:20.919 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35876bdb:::benchmark_data_smithi194_186548_object41185:head# 2024-04-29T14:53:20.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35884ad0:::benchmark_data_smithi194_186548_object32769:head# 2024-04-29T14:53:20.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35885f30:::benchmark_data_smithi194_186548_object20830:head# 2024-04-29T14:53:20.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358942ab:::benchmark_data_smithi194_186548_object35635:head# 2024-04-29T14:53:20.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35897c75:::benchmark_data_smithi194_186548_object17624:head# 2024-04-29T14:53:20.927 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3589b12d:::benchmark_data_smithi194_186548_object42244:head# 2024-04-29T14:53:20.932 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358b5cc6:::benchmark_data_smithi194_186548_object17390:head# 2024-04-29T14:53:20.933 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358b6b53:::benchmark_data_smithi194_186548_object8071:head# 2024-04-29T14:53:20.933 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358b9c3e:::benchmark_data_smithi194_186548_object77554:head# 2024-04-29T14:53:20.933 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358c427f:::benchmark_data_smithi194_186548_object38198:head# 2024-04-29T14:53:20.934 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358d5f11:::benchmark_data_smithi194_186548_object30855:head# 2024-04-29T14:53:20.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358f1413:::benchmark_data_smithi194_186548_object60919:head# 2024-04-29T14:53:20.937 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:358fe012:::benchmark_data_smithi194_186548_object61463:head# 2024-04-29T14:53:20.937 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35913331:::benchmark_data_smithi194_186548_object61125:head# 2024-04-29T14:53:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3592f9e3:::benchmark_data_smithi194_186548_object48421:head# 2024-04-29T14:53:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3593aa25:::benchmark_data_smithi194_186548_object68349:head# 2024-04-29T14:53:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35942633:::benchmark_data_smithi194_186548_object65801:head# 2024-04-29T14:53:20.941 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35975bc7:::benchmark_data_smithi194_186548_object51655:head# 2024-04-29T14:53:20.941 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3597b3c7:::benchmark_data_smithi194_186548_object46329:head# 2024-04-29T14:53:20.942 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3597d379:::benchmark_data_smithi194_186548_object1606:head# 2024-04-29T14:53:20.945 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35989517:::benchmark_data_smithi194_186548_object26480:head# 2024-04-29T14:53:20.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359a0979:::benchmark_data_smithi194_186548_object5825:head# 2024-04-29T14:53:20.957 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359a2f48:::benchmark_data_smithi194_186548_object6967:head# 2024-04-29T14:53:20.957 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359b0b0e:::benchmark_data_smithi194_186548_object3190:head# 2024-04-29T14:53:20.957 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359b2074:::benchmark_data_smithi194_186548_object11180:head# 2024-04-29T14:53:20.957 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359be5de:::benchmark_data_smithi194_186548_object63195:head# 2024-04-29T14:53:20.957 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359d5748:::benchmark_data_smithi194_186548_object4468:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359e1b17:::benchmark_data_smithi194_186548_object4924:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359e4125:::benchmark_data_smithi194_186548_object26541:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:359e4a43:::benchmark_data_smithi194_186548_object31965:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a0c446:::benchmark_data_smithi194_186548_object48032:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a1b24b:::benchmark_data_smithi194_186548_object40259:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a1cfba:::benchmark_data_smithi194_186548_object12974:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a21b0d:::benchmark_data_smithi194_186548_object34757:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a23f1e:::benchmark_data_smithi194_186548_object75461:head# 2024-04-29T14:53:20.958 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a35b09:::benchmark_data_smithi194_186548_object59550:head# 2024-04-29T14:53:20.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a38e17:::benchmark_data_smithi194_186548_object25666:head# 2024-04-29T14:53:20.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a3d844:::benchmark_data_smithi194_186548_object2987:head# 2024-04-29T14:53:20.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a60f95:::benchmark_data_smithi194_186548_object23016:head# 2024-04-29T14:53:20.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a61bf2:::benchmark_data_smithi194_186548_object13195:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a76fba:::benchmark_data_smithi194_186548_object43479:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a79fb5:::benchmark_data_smithi194_186548_object55137:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a7bc77:::benchmark_data_smithi194_186548_object41537:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35a8b441:::benchmark_data_smithi194_186548_object78195:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35aa7880:::benchmark_data_smithi194_186548_object20084:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35aa99ac:::benchmark_data_smithi194_186548_object11484:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35abfb18:::benchmark_data_smithi194_186548_object38887:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35aca256:::benchmark_data_smithi194_186548_object70273:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35acdd73:::benchmark_data_smithi194_186548_object21959:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35af1d4a:::benchmark_data_smithi194_186548_object12503:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35af9580:::benchmark_data_smithi194_186548_object56520:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b02064:::benchmark_data_smithi194_186548_object16194:head# 2024-04-29T14:53:20.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b1843c:::benchmark_data_smithi194_186548_object83016:head# 2024-04-29T14:53:20.967 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b212a2:::benchmark_data_smithi194_186548_object73597:head# 2024-04-29T14:53:20.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b222b8:::benchmark_data_smithi194_186548_object68085:head# 2024-04-29T14:53:20.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b284fa:::benchmark_data_smithi194_186548_object81316:head# 2024-04-29T14:53:20.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b2b910:::benchmark_data_smithi194_186548_object20863:head# 2024-04-29T14:53:20.969 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b2dcc0:::benchmark_data_smithi194_186548_object17821:head# 2024-04-29T14:53:20.969 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b30374:::benchmark_data_smithi194_186548_object10610:head# 2024-04-29T14:53:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b39fcd:::benchmark_data_smithi194_186548_object32549:head# 2024-04-29T14:53:20.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b3a54e:::benchmark_data_smithi194_186548_object83539:head# 2024-04-29T14:53:20.971 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b47992:::benchmark_data_smithi194_186548_object68545:head# 2024-04-29T14:53:20.971 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b69109:::benchmark_data_smithi194_186548_object47225:head# 2024-04-29T14:53:20.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b78e1e:::benchmark_data_smithi194_186548_object64379:head# 2024-04-29T14:53:20.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b7d918:::benchmark_data_smithi194_186548_object71154:head# 2024-04-29T14:53:20.973 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35b940f2:::benchmark_data_smithi194_186548_object14542:head# 2024-04-29T14:53:20.973 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35bb3153:::benchmark_data_smithi194_186548_object78729:head# 2024-04-29T14:53:20.974 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35bbe57d:::benchmark_data_smithi194_186548_object62448:head# 2024-04-29T14:53:20.974 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35bd65d6:::benchmark_data_smithi194_186548_object10866:head# 2024-04-29T14:53:20.975 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c1a7ac:::benchmark_data_smithi194_186548_object36139:head# 2024-04-29T14:53:20.975 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c266c6:::benchmark_data_smithi194_186548_object39291:head# 2024-04-29T14:53:20.976 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c288cc:::benchmark_data_smithi194_186548_object80132:head# 2024-04-29T14:53:20.976 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c2fcf9:::benchmark_data_smithi194_186548_object47516:head# 2024-04-29T14:53:20.977 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c3b8cd:::benchmark_data_smithi194_186548_object47402:head# 2024-04-29T14:53:20.977 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c3c176:::benchmark_data_smithi194_186548_object2355:head# 2024-04-29T14:53:20.978 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c4406e:::benchmark_data_smithi194_186548_object26685:head# 2024-04-29T14:53:20.978 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c4ff96:::benchmark_data_smithi194_186548_object28832:head# 2024-04-29T14:53:20.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c5c013:::benchmark_data_smithi194_186548_object10131:head# 2024-04-29T14:53:20.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c6b393:::benchmark_data_smithi194_186548_object30907:head# 2024-04-29T14:53:20.980 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c7539a:::benchmark_data_smithi194_186548_object29292:head# 2024-04-29T14:53:20.980 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c75821:::benchmark_data_smithi194_186548_object1878:head# 2024-04-29T14:53:20.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c7d00b:::benchmark_data_smithi194_186548_object50671:head# 2024-04-29T14:53:20.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c80e39:::benchmark_data_smithi194_186548_object19701:head# 2024-04-29T14:53:20.982 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c8313e:::benchmark_data_smithi194_186548_object75389:head# 2024-04-29T14:53:20.982 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35c9050e:::benchmark_data_smithi194_186548_object24609:head# 2024-04-29T14:53:20.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35cd4287:::benchmark_data_smithi194_186548_object6482:head# 2024-04-29T14:53:20.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35cded13:::benchmark_data_smithi194_186548_object10404:head# 2024-04-29T14:53:20.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35ce7e50:::benchmark_data_smithi194_186548_object66814:head# 2024-04-29T14:53:20.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35cf7d8d:::benchmark_data_smithi194_186548_object48451:head# 2024-04-29T14:53:20.985 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d09ab5:::benchmark_data_smithi194_186548_object20881:head# 2024-04-29T14:53:20.985 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d0d9ea:::benchmark_data_smithi194_186548_object26264:head# 2024-04-29T14:53:20.986 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d28ef5:::benchmark_data_smithi194_186548_object70689:head# 2024-04-29T14:53:20.986 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d36c60:::benchmark_data_smithi194_186548_object40615:head# 2024-04-29T14:53:20.986 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d43d2f:::benchmark_data_smithi194_186548_object57421:head# 2024-04-29T14:53:20.987 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d443fd:::benchmark_data_smithi194_186548_object54573:head# 2024-04-29T14:53:20.987 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d48d11:::benchmark_data_smithi194_186548_object62594:head# 2024-04-29T14:53:20.988 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d5c95a:::benchmark_data_smithi194_186548_object57141:head# 2024-04-29T14:53:20.988 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d65760:::benchmark_data_smithi194_186548_object51848:head# 2024-04-29T14:53:20.989 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d6824d:::benchmark_data_smithi194_186548_object80044:head# 2024-04-29T14:53:20.989 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d7cc42:::benchmark_data_smithi194_186548_object75792:head# 2024-04-29T14:53:20.990 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d8269a:::benchmark_data_smithi194_186548_object19758:head# 2024-04-29T14:53:20.990 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d8324b:::benchmark_data_smithi194_186548_object66425:head# 2024-04-29T14:53:20.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d85e78:::benchmark_data_smithi194_186548_object34410:head# 2024-04-29T14:53:20.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d8b8ef:::benchmark_data_smithi194_186548_object31571:head# 2024-04-29T14:53:20.992 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d9999a:::benchmark_data_smithi194_186548_object32358:head# 2024-04-29T14:53:20.992 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d9ef00:::benchmark_data_smithi194_186548_object16843:head# 2024-04-29T14:53:20.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35d9f0c2:::benchmark_data_smithi194_186548_object11643:head# 2024-04-29T14:53:20.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35dc224a:::benchmark_data_smithi194_186548_object59010:head# 2024-04-29T14:53:20.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35dc8ac7:::benchmark_data_smithi194_186548_object25868:head# 2024-04-29T14:53:20.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35dcdff8:::benchmark_data_smithi194_186548_object66330:head# 2024-04-29T14:53:20.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e068c2:::benchmark_data_smithi194_186548_object11275:head# 2024-04-29T14:53:20.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e09951:::benchmark_data_smithi194_186548_object55534:head# 2024-04-29T14:53:20.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e13309:::benchmark_data_smithi194_186548_object51967:head# 2024-04-29T14:53:20.996 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e1a692:::benchmark_data_smithi194_186548_object8739:head# 2024-04-29T14:53:20.996 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e25772:::benchmark_data_smithi194_186548_object65534:head# 2024-04-29T14:53:20.997 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e44125:::benchmark_data_smithi194_186548_object47139:head# 2024-04-29T14:53:20.997 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e47ac2:::benchmark_data_smithi194_186548_object31485:head# 2024-04-29T14:53:20.998 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e4a063:::benchmark_data_smithi194_186548_object80170:head# 2024-04-29T14:53:20.998 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e4cc4f:::benchmark_data_smithi194_186548_object20025:head# 2024-04-29T14:53:20.999 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e51e90:::benchmark_data_smithi194_186548_object79560:head# 2024-04-29T14:53:20.999 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e6f04a:::benchmark_data_smithi194_186548_object52102:head# 2024-04-29T14:53:20.999 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e7cc44:::benchmark_data_smithi194_186548_object64795:head# 2024-04-29T14:53:21.000 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e7eb53:::benchmark_data_smithi194_186548_object33601:head# 2024-04-29T14:53:21.000 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e8641f:::benchmark_data_smithi194_186548_object79227:head# 2024-04-29T14:53:21.001 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35e9127a:::benchmark_data_smithi194_186548_object27304:head# 2024-04-29T14:53:21.001 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35eb3701:::benchmark_data_smithi194_186548_object82963:head# 2024-04-29T14:53:21.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35ecf1e4:::benchmark_data_smithi194_186548_object24252:head# 2024-04-29T14:53:21.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35ede4b8:::benchmark_data_smithi194_186548_object24655:head# 2024-04-29T14:53:21.002 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35ee9fdd:::benchmark_data_smithi194_186548_object47229:head# 2024-04-29T14:53:21.003 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35efdaaf:::benchmark_data_smithi194_186548_object49504:head# 2024-04-29T14:53:21.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f0106a:::benchmark_data_smithi194_186548_object40067:head# 2024-04-29T14:53:21.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f12f37:::benchmark_data_smithi194_186548_object62571:head# 2024-04-29T14:53:21.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f1e0a7:::benchmark_data_smithi194_186548_object14919:head# 2024-04-29T14:53:21.005 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f24423:::benchmark_data_smithi194_186548_object50987:head# 2024-04-29T14:53:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f25b13:::benchmark_data_smithi194_186548_object50345:head# 2024-04-29T14:53:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f2d130:::benchmark_data_smithi194_186548_object13649:head# 2024-04-29T14:53:21.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f2f1b7:::benchmark_data_smithi194_186548_object11277:head# 2024-04-29T14:53:21.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f4801d:::benchmark_data_smithi194_186548_object27857:head# 2024-04-29T14:53:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f52acc:::benchmark_data_smithi194_186548_object50269:head# 2024-04-29T14:53:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f5782e:::benchmark_data_smithi194_186548_object52511:head# 2024-04-29T14:53:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f5f80f:::benchmark_data_smithi194_186548_object39572:head# 2024-04-29T14:53:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f5fce0:::benchmark_data_smithi194_186548_object53884:head# 2024-04-29T14:53:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f75f38:::benchmark_data_smithi194_186548_object71398:head# 2024-04-29T14:53:21.010 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35f97b67:::benchmark_data_smithi194_186548_object38012:head# 2024-04-29T14:53:21.010 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fa5ff8:::benchmark_data_smithi194_186548_object54669:head# 2024-04-29T14:53:21.011 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fa62f0:::benchmark_data_smithi194_186548_object52945:head# 2024-04-29T14:53:21.011 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fb6fe9:::benchmark_data_smithi194_186548_object26093:head# 2024-04-29T14:53:21.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fba9a5:::benchmark_data_smithi194_186548_object73289:head# 2024-04-29T14:53:21.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fc18fc:::benchmark_data_smithi194_186548_object3829:head# 2024-04-29T14:53:21.013 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fc3f97:::benchmark_data_smithi194_186548_object47173:head# 2024-04-29T14:53:21.013 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fcc97f:::benchmark_data_smithi194_186548_object10696:head# 2024-04-29T14:53:21.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fd5085:::benchmark_data_smithi194_186548_object24317:head# 2024-04-29T14:53:21.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fe0a05:::benchmark_data_smithi194_186548_object60342:head# 2024-04-29T14:53:21.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35fe7a9c:::benchmark_data_smithi194_186548_object68077:head# 2024-04-29T14:53:21.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35ff5e86:::benchmark_data_smithi194_186548_object80920:head# 2024-04-29T14:53:21.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:35ff77bf:::benchmark_data_smithi194_186548_object20130:head# 2024-04-29T14:53:21.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36010499:::benchmark_data_smithi194_186548_object17783:head# 2024-04-29T14:53:21.017 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36014c15:::benchmark_data_smithi194_186548_object62877:head# 2024-04-29T14:53:21.017 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3601624c:::benchmark_data_smithi194_186548_object41405:head# 2024-04-29T14:53:21.017 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3604757d:::benchmark_data_smithi194_186548_object35470:head# 2024-04-29T14:53:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3605087c:::benchmark_data_smithi194_186548_object47960:head# 2024-04-29T14:53:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36054b86:::benchmark_data_smithi194_186548_object64431:head# 2024-04-29T14:53:21.019 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3605c431:::benchmark_data_smithi194_186548_object9126:head# 2024-04-29T14:53:21.019 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3608dc5a:::benchmark_data_smithi194_186548_object47970:head# 2024-04-29T14:53:21.020 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3608df98:::benchmark_data_smithi194_186548_object82883:head# 2024-04-29T14:53:21.020 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360a429b:::benchmark_data_smithi194_186548_object18092:head# 2024-04-29T14:53:21.020 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360aa57a:::benchmark_data_smithi194_186548_object76404:head# 2024-04-29T14:53:21.021 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360ad5d9:::benchmark_data_smithi194_186548_object53805:head# 2024-04-29T14:53:21.021 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360b2193:::benchmark_data_smithi194_186548_object30145:head# 2024-04-29T14:53:21.022 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360b5f5e:::benchmark_data_smithi194_186548_object57781:head# 2024-04-29T14:53:21.022 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360d6018:::benchmark_data_smithi194_186548_object39648:head# 2024-04-29T14:53:21.022 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360e39e5:::benchmark_data_smithi194_186548_object50789:head# 2024-04-29T14:53:21.023 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360ef44f:::benchmark_data_smithi194_186548_object9825:head# 2024-04-29T14:53:21.023 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360f0518:::benchmark_data_smithi194_186548_object19526:head# 2024-04-29T14:53:21.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360f1d57:::benchmark_data_smithi194_186548_object54417:head# 2024-04-29T14:53:21.034 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360f654e:::benchmark_data_smithi194_186548_object5411:head# 2024-04-29T14:53:21.035 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:360faa2b:::benchmark_data_smithi194_186548_object4981:head# 2024-04-29T14:53:21.038 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3610198e:::benchmark_data_smithi194_186548_object55715:head# 2024-04-29T14:53:21.039 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36105191:::benchmark_data_smithi194_186548_object18810:head# 2024-04-29T14:53:21.040 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36114bcb:::benchmark_data_smithi194_186548_object40485:head# 2024-04-29T14:53:21.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3611a355:::benchmark_data_smithi194_186548_object80062:head# 2024-04-29T14:53:21.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3611b2eb:::benchmark_data_smithi194_186548_object18523:head# 2024-04-29T14:53:21.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3611c105:::benchmark_data_smithi194_186548_object48274:head# 2024-04-29T14:53:21.042 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36146331:::benchmark_data_smithi194_186548_object17605:head# 2024-04-29T14:53:21.042 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36147486:::benchmark_data_smithi194_186548_object39296:head# 2024-04-29T14:53:21.043 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3615868b:::benchmark_data_smithi194_186548_object38692:head# 2024-04-29T14:53:21.044 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36162ae3:::benchmark_data_smithi194_186548_object84764:head# 2024-04-29T14:53:21.044 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36174e05:::benchmark_data_smithi194_186548_object48064:head# 2024-04-29T14:53:21.044 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3619eee6:::benchmark_data_smithi194_186548_object59564:head# 2024-04-29T14:53:21.045 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:361ab0fe:::benchmark_data_smithi194_186548_object27503:head# 2024-04-29T14:53:21.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:361b2ef1:::benchmark_data_smithi194_186548_object59950:head# 2024-04-29T14:53:21.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:361da581:::benchmark_data_smithi194_186548_object28506:head# 2024-04-29T14:53:21.047 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:361e47f2:::benchmark_data_smithi194_186548_object74864:head# 2024-04-29T14:53:21.047 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:361f0758:::benchmark_data_smithi194_186548_object4827:head# 2024-04-29T14:53:21.048 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:361fef65:::benchmark_data_smithi194_186548_object63197:head# 2024-04-29T14:53:21.048 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36207e2a:::benchmark_data_smithi194_186548_object55383:head# 2024-04-29T14:53:21.048 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36217520:::benchmark_data_smithi194_186548_object58599:head# 2024-04-29T14:53:21.050 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3622446a:::benchmark_data_smithi194_186548_object51717:head# 2024-04-29T14:53:21.053 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3622fa6e:::benchmark_data_smithi194_186548_object79189:head# 2024-04-29T14:53:21.053 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3623064c:::benchmark_data_smithi194_186548_object2040:head# 2024-04-29T14:53:21.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36237562:::benchmark_data_smithi194_186548_object16965:head# 2024-04-29T14:53:21.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36237b00:::benchmark_data_smithi194_186548_object53895:head# 2024-04-29T14:53:21.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3624abd0:::benchmark_data_smithi194_186548_object24225:head# 2024-04-29T14:53:21.055 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3624c0f2:::benchmark_data_smithi194_186548_object68414:head# 2024-04-29T14:53:21.056 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36254633:::benchmark_data_smithi194_186548_object45626:head# 2024-04-29T14:53:21.059 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3625c9f5:::benchmark_data_smithi194_186548_object56158:head# 2024-04-29T14:53:21.059 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36260819:::benchmark_data_smithi194_186548_object9770:head# 2024-04-29T14:53:21.059 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:362649a4:::benchmark_data_smithi194_186548_object35118:head# 2024-04-29T14:53:21.060 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36265473:::benchmark_data_smithi194_186548_object54863:head# 2024-04-29T14:53:21.060 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36286f9e:::benchmark_data_smithi194_186548_object39277:head# 2024-04-29T14:53:21.064 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3629f1f0:::benchmark_data_smithi194_186548_object77707:head# 2024-04-29T14:53:21.064 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:362b519e:::benchmark_data_smithi194_186548_object54065:head# 2024-04-29T14:53:21.064 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:362b9929:::benchmark_data_smithi194_186548_object5141:head# 2024-04-29T14:53:21.064 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:362c2edc:::benchmark_data_smithi194_186548_object43807:head# 2024-04-29T14:53:21.064 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:362d22b0:::benchmark_data_smithi194_186548_object28118:head# 2024-04-29T14:53:21.064 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:362f417f:::benchmark_data_smithi194_186548_object83855:head# 2024-04-29T14:53:21.065 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:363090ec:::benchmark_data_smithi194_186548_object46514:head# 2024-04-29T14:53:21.065 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3630e698:::benchmark_data_smithi194_186548_object78037:head# 2024-04-29T14:53:21.065 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:363122c9:::benchmark_data_smithi194_186548_object39872:head# 2024-04-29T14:53:21.065 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3631e15a:::benchmark_data_smithi194_186548_object75453:head# 2024-04-29T14:53:21.065 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36334b84:::benchmark_data_smithi194_186548_object53292:head# 2024-04-29T14:53:21.073 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3633a2fd:::benchmark_data_smithi194_186548_object45589:head# 2024-04-29T14:53:21.073 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36349d7d:::benchmark_data_smithi194_186548_object50702:head# 2024-04-29T14:53:21.073 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3635008b:::benchmark_data_smithi194_186548_object25852:head# 2024-04-29T14:53:21.074 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3635afeb:::benchmark_data_smithi194_186548_object11837:head# 2024-04-29T14:53:21.074 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:363604be:::benchmark_data_smithi194_186548_object17743:head# 2024-04-29T14:53:21.074 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3636b3d9:::benchmark_data_smithi194_186548_object57166:head# 2024-04-29T14:53:21.074 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3636ca7c:::benchmark_data_smithi194_186548_object54593:head# 2024-04-29T14:53:21.075 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3637ddf5:::benchmark_data_smithi194_186548_object14763:head# 2024-04-29T14:53:21.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36388dae:::benchmark_data_smithi194_186548_object56628:head# 2024-04-29T14:53:21.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:363cb3ec:::benchmark_data_smithi194_186548_object34114:head# 2024-04-29T14:53:21.080 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:363e20b5:::benchmark_data_smithi194_186548_object82237:head# 2024-04-29T14:53:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:363e6c13:::benchmark_data_smithi194_186548_object43538:head# 2024-04-29T14:53:21.081 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3640816b:::benchmark_data_smithi194_186548_object8047:head# 2024-04-29T14:53:21.082 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3640e715:::benchmark_data_smithi194_186548_object68375:head# 2024-04-29T14:53:21.083 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3642fa3c:::benchmark_data_smithi194_186548_object35042:head# 2024-04-29T14:53:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3642fb77:::benchmark_data_smithi194_186548_object52714:head# 2024-04-29T14:53:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36434c05:::benchmark_data_smithi194_186548_object13872:head# 2024-04-29T14:53:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3644ccf3:::benchmark_data_smithi194_186548_object26987:head# 2024-04-29T14:53:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3645892d:::benchmark_data_smithi194_186548_object48745:head# 2024-04-29T14:53:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3647f0b3:::benchmark_data_smithi194_186548_object70764:head# 2024-04-29T14:53:21.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3648cb69:::benchmark_data_smithi194_186548_object77157:head# 2024-04-29T14:53:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3649647e:::benchmark_data_smithi194_186548_object55601:head# 2024-04-29T14:53:21.086 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364995a5:::benchmark_data_smithi194_186548_object35571:head# 2024-04-29T14:53:21.087 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364a7a7d:::benchmark_data_smithi194_186548_object3371:head# 2024-04-29T14:53:21.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364bb14b:::benchmark_data_smithi194_186548_object17386:head# 2024-04-29T14:53:21.090 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364c0025:::benchmark_data_smithi194_186548_object71145:head# 2024-04-29T14:53:21.091 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364cd5f9:::benchmark_data_smithi194_186548_object23188:head# 2024-04-29T14:53:21.091 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364d569c:::benchmark_data_smithi194_186548_object61464:head# 2024-04-29T14:53:21.092 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364ed858:::benchmark_data_smithi194_186548_object3815:head# 2024-04-29T14:53:21.092 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364ee803:::benchmark_data_smithi194_186548_object2686:head# 2024-04-29T14:53:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:364f9d42:::benchmark_data_smithi194_186548_object31422:head# 2024-04-29T14:53:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:365034dd:::benchmark_data_smithi194_186548_object23491:head# 2024-04-29T14:53:21.096 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36506f10:::benchmark_data_smithi194_186548_object30911:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3650a5c8:::benchmark_data_smithi194_186548_object13506:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3650b7a0:::benchmark_data_smithi194_186548_object68297:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:365170f5:::benchmark_data_smithi194_186548_object32593:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36520594:::benchmark_data_smithi194_186548_object13660:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36521bd8:::benchmark_data_smithi194_186548_object78954:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:365292cc:::benchmark_data_smithi194_186548_object6901:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36542533:::benchmark_data_smithi194_186548_object28859:head# 2024-04-29T14:53:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36562782:::benchmark_data_smithi194_186548_object28525:head# 2024-04-29T14:53:21.103 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3656716a:::benchmark_data_smithi194_186548_object42397:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3658308e:::benchmark_data_smithi194_186548_object85237:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:365aebf3:::benchmark_data_smithi194_186548_object40190:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:365dd4a8:::benchmark_data_smithi194_186548_object13224:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:365e00f2:::benchmark_data_smithi194_186548_object30854:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:365fefd8:::benchmark_data_smithi194_186548_object71275:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36602012:::benchmark_data_smithi194_186548_object81147:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36608f22:::benchmark_data_smithi194_186548_object75928:head# 2024-04-29T14:53:21.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3661dab0:::benchmark_data_smithi194_186548_object32329:head# 2024-04-29T14:53:21.106 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36621a7a:::benchmark_data_smithi194_186548_object11255:head# 2024-04-29T14:53:21.106 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3663d2b0:::benchmark_data_smithi194_186548_object21237:head# 2024-04-29T14:53:21.108 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36647764:::benchmark_data_smithi194_186548_object28133:head# 2024-04-29T14:53:21.108 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:366486ea:::benchmark_data_smithi194_186548_object26698:head# 2024-04-29T14:53:21.110 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36649d76:::benchmark_data_smithi194_186548_object19140:head# 2024-04-29T14:53:21.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36654b23:::benchmark_data_smithi194_186548_object68865:head# 2024-04-29T14:53:21.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3666cca2:::benchmark_data_smithi194_186548_object42931:head# 2024-04-29T14:53:21.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36677946:::benchmark_data_smithi194_186548_object12961:head# 2024-04-29T14:53:21.119 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3668b42d:::benchmark_data_smithi194_186548_object59004:head# 2024-04-29T14:53:21.119 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36692b52:::benchmark_data_smithi194_186548_object81536:head# 2024-04-29T14:53:21.120 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3669d47f:::benchmark_data_smithi194_186548_object67117:head# 2024-04-29T14:53:21.120 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:366d04d5:::benchmark_data_smithi194_186548_object45925:head# 2024-04-29T14:53:21.120 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:366ec008:::benchmark_data_smithi194_186548_object2673:head# 2024-04-29T14:53:21.127 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367034db:::benchmark_data_smithi194_186548_object902:head# 2024-04-29T14:53:21.127 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36704373:::benchmark_data_smithi194_186548_object61652:head# 2024-04-29T14:53:21.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3670ab61:::benchmark_data_smithi194_186548_object75013:head# 2024-04-29T14:53:21.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3670ad54:::benchmark_data_smithi194_186548_object24997:head# 2024-04-29T14:53:21.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3670fa37:::benchmark_data_smithi194_186548_object81678:head# 2024-04-29T14:53:21.130 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36713960:::benchmark_data_smithi194_186548_object72969:head# 2024-04-29T14:53:21.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367280b3:::benchmark_data_smithi194_186548_object74913:head# 2024-04-29T14:53:21.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3672c588:::benchmark_data_smithi194_186548_object16237:head# 2024-04-29T14:53:21.132 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3673ac72:::benchmark_data_smithi194_186548_object27388:head# 2024-04-29T14:53:21.132 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3674096c:::benchmark_data_smithi194_186548_object8829:head# 2024-04-29T14:53:21.133 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3675425e:::benchmark_data_smithi194_186548_object17348:head# 2024-04-29T14:53:21.133 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36756ee1:::benchmark_data_smithi194_186548_object56761:head# 2024-04-29T14:53:21.134 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367705f7:::benchmark_data_smithi194_186548_object24358:head# 2024-04-29T14:53:21.134 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3677537f:::benchmark_data_smithi194_186548_object44513:head# 2024-04-29T14:53:21.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36792428:::benchmark_data_smithi194_186548_object18421:head# 2024-04-29T14:53:21.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367b4c3c:::benchmark_data_smithi194_186548_object81989:head# 2024-04-29T14:53:21.136 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367ca754:::benchmark_data_smithi194_186548_object6673:head# 2024-04-29T14:53:21.136 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367cd5fb:::benchmark_data_smithi194_186548_object19714:head# 2024-04-29T14:53:21.137 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367cf949:::benchmark_data_smithi194_186548_object1128:head# 2024-04-29T14:53:21.137 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367d0d5d:::benchmark_data_smithi194_186548_object31823:head# 2024-04-29T14:53:21.138 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367d3157:::benchmark_data_smithi194_186548_object42976:head# 2024-04-29T14:53:21.138 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367e3e4d:::benchmark_data_smithi194_186548_object40207:head# 2024-04-29T14:53:21.138 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367f329c:::benchmark_data_smithi194_186548_object74205:head# 2024-04-29T14:53:21.139 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:367f81c1:::benchmark_data_smithi194_186548_object82626:head# 2024-04-29T14:53:21.139 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3680b24a:::benchmark_data_smithi194_186548_object23672:head# 2024-04-29T14:53:21.140 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36823814:::benchmark_data_smithi194_186548_object59787:head# 2024-04-29T14:53:21.140 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36825744:::benchmark_data_smithi194_186548_object70581:head# 2024-04-29T14:53:21.141 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36844a79:::benchmark_data_smithi194_186548_object42749:head# 2024-04-29T14:53:21.141 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3684cbbb:::benchmark_data_smithi194_186548_object38248:head# 2024-04-29T14:53:21.142 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3685c651:::benchmark_data_smithi194_186548_object4390:head# 2024-04-29T14:53:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36862a6c:::benchmark_data_smithi194_186548_object15688:head# 2024-04-29T14:53:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36868c78:::benchmark_data_smithi194_186548_object20899:head# 2024-04-29T14:53:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3686e21d:::benchmark_data_smithi194_186548_object3292:head# 2024-04-29T14:53:21.146 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36888d68:::benchmark_data_smithi194_186548_object67721:head# 2024-04-29T14:53:21.146 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3688f5e2:::benchmark_data_smithi194_186548_object14457:head# 2024-04-29T14:53:21.147 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36891ed0:::benchmark_data_smithi194_186548_object81751:head# 2024-04-29T14:53:21.148 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:368a8916:::benchmark_data_smithi194_186548_object75861:head# 2024-04-29T14:53:21.151 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:368b7d32:::benchmark_data_smithi194_186548_object1735:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:368de671:::benchmark_data_smithi194_186548_object78207:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:368df1cf:::benchmark_data_smithi194_186548_object13521:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:368ed2e5:::benchmark_data_smithi194_186548_object15011:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:368f540f:::benchmark_data_smithi194_186548_object15547:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:368faab5:::benchmark_data_smithi194_186548_object53761:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3690010a:::benchmark_data_smithi194_186548_object424:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3692e119:::benchmark_data_smithi194_186548_object84091:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36944868:::benchmark_data_smithi194_186548_object16383:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3694aa65:::benchmark_data_smithi194_186548_object52320:head# 2024-04-29T14:53:21.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3694d962:::benchmark_data_smithi194_186548_object63036:head# 2024-04-29T14:53:21.162 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3694dee7:::benchmark_data_smithi194_186548_object28969:head# 2024-04-29T14:53:21.162 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36959425:::benchmark_data_smithi194_186548_object3806:head# 2024-04-29T14:53:21.162 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3695c1ae:::benchmark_data_smithi194_186548_object8262:head# 2024-04-29T14:53:21.163 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369707aa:::benchmark_data_smithi194_186548_object13635:head# 2024-04-29T14:53:21.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369777f4:::benchmark_data_smithi194_186548_object32169:head# 2024-04-29T14:53:21.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3697aab6:::benchmark_data_smithi194_186548_object66311:head# 2024-04-29T14:53:21.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3697d6ee:::benchmark_data_smithi194_186548_object75336:head# 2024-04-29T14:53:21.170 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36989cc3:::benchmark_data_smithi194_186548_object27642:head# 2024-04-29T14:53:21.170 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369a117e:::benchmark_data_smithi194_186548_object42798:head# 2024-04-29T14:53:21.170 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369a9ac0:::benchmark_data_smithi194_186548_object16842:head# 2024-04-29T14:53:21.170 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369c2512:::benchmark_data_smithi194_186548_object15045:head# 2024-04-29T14:53:21.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369d1f3c:::benchmark_data_smithi194_186548_object6455:head# 2024-04-29T14:53:21.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369d4d2b:::benchmark_data_smithi194_186548_object74062:head# 2024-04-29T14:53:21.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369dc62e:::benchmark_data_smithi194_186548_object81905:head# 2024-04-29T14:53:21.172 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369e764b:::benchmark_data_smithi194_186548_object51225:head# 2024-04-29T14:53:21.173 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369ea310:::benchmark_data_smithi194_186548_object5026:head# 2024-04-29T14:53:21.173 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:369ff66d:::benchmark_data_smithi194_186548_object46727:head# 2024-04-29T14:53:21.175 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a0395b:::benchmark_data_smithi194_186548_object15856:head# 2024-04-29T14:53:21.175 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a0b260:::benchmark_data_smithi194_186548_object23708:head# 2024-04-29T14:53:21.176 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a11b13:::benchmark_data_smithi194_186548_object83374:head# 2024-04-29T14:53:21.176 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a16abd:::benchmark_data_smithi194_186548_object53426:head# 2024-04-29T14:53:21.177 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a2afdc:::benchmark_data_smithi194_186548_object2567:head# 2024-04-29T14:53:21.179 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a2d5ee:::benchmark_data_smithi194_186548_object59992:head# 2024-04-29T14:53:21.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a2ff86:::benchmark_data_smithi194_186548_object3560:head# 2024-04-29T14:53:21.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a34771:::benchmark_data_smithi194_186548_object16756:head# 2024-04-29T14:53:21.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a41aff:::benchmark_data_smithi194_186548_object38551:head# 2024-04-29T14:53:21.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a42784:::benchmark_data_smithi194_186548_object15621:head# 2024-04-29T14:53:21.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a4eac4:::benchmark_data_smithi194_186548_object57720:head# 2024-04-29T14:53:21.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a62ef6:::benchmark_data_smithi194_186548_object74162:head# 2024-04-29T14:53:21.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a92a46:::benchmark_data_smithi194_186548_object7183:head# 2024-04-29T14:53:21.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a967dd:::benchmark_data_smithi194_186548_object76519:head# 2024-04-29T14:53:21.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a99168:::benchmark_data_smithi194_186548_object6021:head# 2024-04-29T14:53:21.187 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36a9b74c:::benchmark_data_smithi194_186548_object69046:head# 2024-04-29T14:53:21.187 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36aa066a:::benchmark_data_smithi194_186548_object30893:head# 2024-04-29T14:53:21.187 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ab218e:::benchmark_data_smithi194_186548_object77224:head# 2024-04-29T14:53:21.189 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ad67a2:::benchmark_data_smithi194_186548_object50076:head# 2024-04-29T14:53:21.190 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36adcbfd:::benchmark_data_smithi194_186548_object15402:head# 2024-04-29T14:53:21.190 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36aeb29b:::benchmark_data_smithi194_186548_object5377:head# 2024-04-29T14:53:21.190 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36aeed86:::benchmark_data_smithi194_186548_object61423:head# 2024-04-29T14:53:21.190 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b00066:::benchmark_data_smithi194_186548_object52903:head# 2024-04-29T14:53:21.191 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b09947:::benchmark_data_smithi194_186548_object73926:head# 2024-04-29T14:53:21.191 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b0ce4f:::benchmark_data_smithi194_186548_object65169:head# 2024-04-29T14:53:21.192 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b10175:::benchmark_data_smithi194_186548_object75296:head# 2024-04-29T14:53:21.192 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b10b84:::benchmark_data_smithi194_186548_object53157:head# 2024-04-29T14:53:21.192 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b3b65b:::benchmark_data_smithi194_186548_object32030:head# 2024-04-29T14:53:21.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b60258:::benchmark_data_smithi194_186548_object31966:head# 2024-04-29T14:53:21.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b63cbb:::benchmark_data_smithi194_186548_object37657:head# 2024-04-29T14:53:21.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b78bcd:::benchmark_data_smithi194_186548_object13131:head# 2024-04-29T14:53:21.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b7c5b6:::benchmark_data_smithi194_186548_object3401:head# 2024-04-29T14:53:21.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b7f64e:::benchmark_data_smithi194_186548_object24205:head# 2024-04-29T14:53:21.199 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36b81eac:::benchmark_data_smithi194_186548_object70078:head# 2024-04-29T14:53:21.202 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ba1fcf:::benchmark_data_smithi194_186548_object14727:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36bac591:::benchmark_data_smithi194_186548_object5038:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36bb3d2a:::benchmark_data_smithi194_186548_object55013:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36bc0d7d:::benchmark_data_smithi194_186548_object68182:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36bdb1a0:::benchmark_data_smithi194_186548_object10801:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36be4999:::benchmark_data_smithi194_186548_object61892:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36be6d16:::benchmark_data_smithi194_186548_object173:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36bedbe0:::benchmark_data_smithi194_186548_object47392:head# 2024-04-29T14:53:21.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36bf08f9:::benchmark_data_smithi194_186548_object76205:head# 2024-04-29T14:53:21.204 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c2908e:::benchmark_data_smithi194_186548_object69175:head# 2024-04-29T14:53:21.204 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c2faa4:::benchmark_data_smithi194_186548_object20514:head# 2024-04-29T14:53:21.206 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c3367f:::benchmark_data_smithi194_186548_object7535:head# 2024-04-29T14:53:21.206 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c49f4b:::benchmark_data_smithi194_186548_object26524:head# 2024-04-29T14:53:21.206 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c60e9f:::benchmark_data_smithi194_186548_object33744:head# 2024-04-29T14:53:21.207 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c64dc2:::benchmark_data_smithi194_186548_object4049:head# 2024-04-29T14:53:21.207 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c65425:::benchmark_data_smithi194_186548_object5188:head# 2024-04-29T14:53:21.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c7a5bd:::benchmark_data_smithi194_186548_object56674:head# 2024-04-29T14:53:21.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c7de3d:::benchmark_data_smithi194_186548_object13923:head# 2024-04-29T14:53:21.209 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c85fcc:::benchmark_data_smithi194_186548_object40957:head# 2024-04-29T14:53:21.209 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c8c9c1:::benchmark_data_smithi194_186548_object26202:head# 2024-04-29T14:53:21.209 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c90569:::benchmark_data_smithi194_186548_object18343:head# 2024-04-29T14:53:21.210 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c98ca3:::benchmark_data_smithi194_186548_object81053:head# 2024-04-29T14:53:21.210 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36c9f40f:::benchmark_data_smithi194_186548_object47103:head# 2024-04-29T14:53:21.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36cad22d:::benchmark_data_smithi194_186548_object84128:head# 2024-04-29T14:53:21.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36cc285e:::benchmark_data_smithi194_186548_object55163:head# 2024-04-29T14:53:21.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ccab9e:::benchmark_data_smithi194_186548_object32594:head# 2024-04-29T14:53:21.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36cdf361:::benchmark_data_smithi194_186548_object1837:head# 2024-04-29T14:53:21.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36cef757:::benchmark_data_smithi194_186548_object58015:head# 2024-04-29T14:53:21.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36cf6838:::benchmark_data_smithi194_186548_object37671:head# 2024-04-29T14:53:21.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d084af:::benchmark_data_smithi194_186548_object2439:head# 2024-04-29T14:53:21.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d1520e:::benchmark_data_smithi194_186548_object11693:head# 2024-04-29T14:53:21.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d1536a:::benchmark_data_smithi194_186548_object58888:head# 2024-04-29T14:53:21.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d1e5b2:::benchmark_data_smithi194_186548_object69638:head# 2024-04-29T14:53:21.216 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d235eb:::benchmark_data_smithi194_186548_object14619:head# 2024-04-29T14:53:21.216 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d36edb:::benchmark_data_smithi194_186548_object68556:head# 2024-04-29T14:53:21.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d3c0c4:::benchmark_data_smithi194_186548_object73702:head# 2024-04-29T14:53:21.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d5d57a:::benchmark_data_smithi194_186548_object62505:head# 2024-04-29T14:53:21.219 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d7ca47:::benchmark_data_smithi194_186548_object84598:head# 2024-04-29T14:53:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36d97f45:::benchmark_data_smithi194_186548_object10189:head# 2024-04-29T14:53:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36daf6e5:::benchmark_data_smithi194_186548_object78106:head# 2024-04-29T14:53:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36db0979:::benchmark_data_smithi194_186548_object45289:head# 2024-04-29T14:53:21.225 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36db1653:::benchmark_data_smithi194_186548_object71456:head# 2024-04-29T14:53:21.225 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36db329c:::benchmark_data_smithi194_186548_object1629:head# 2024-04-29T14:53:21.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36dc834b:::benchmark_data_smithi194_186548_object77605:head# 2024-04-29T14:53:21.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36dd15f1:::benchmark_data_smithi194_186548_object72244:head# 2024-04-29T14:53:21.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36dfb363:::benchmark_data_smithi194_186548_object48150:head# 2024-04-29T14:53:21.229 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36dfcd61:::benchmark_data_smithi194_186548_object21600:head# 2024-04-29T14:53:21.229 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e1f967:::benchmark_data_smithi194_186548_object16338:head# 2024-04-29T14:53:21.229 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e207b5:::benchmark_data_smithi194_186548_object67422:head# 2024-04-29T14:53:21.236 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e3de5e:::benchmark_data_smithi194_186548_object6386:head# 2024-04-29T14:53:21.236 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e412c4:::benchmark_data_smithi194_186548_object1874:head# 2024-04-29T14:53:21.236 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e46443:::benchmark_data_smithi194_186548_object64394:head# 2024-04-29T14:53:21.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e4d449:::benchmark_data_smithi194_186548_object5565:head# 2024-04-29T14:53:21.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e4f5ca:::benchmark_data_smithi194_186548_object7273:head# 2024-04-29T14:53:21.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e53469:::benchmark_data_smithi194_186548_object62867:head# 2024-04-29T14:53:21.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e5a2c9:::benchmark_data_smithi194_186548_object62677:head# 2024-04-29T14:53:21.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e76767:::benchmark_data_smithi194_186548_object36171:head# 2024-04-29T14:53:21.237 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e7c333:::benchmark_data_smithi194_186548_object72551:head# 2024-04-29T14:53:21.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e7f1b5:::benchmark_data_smithi194_186548_object60173:head# 2024-04-29T14:53:21.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e87f26:::benchmark_data_smithi194_186548_object31933:head# 2024-04-29T14:53:21.245 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36e88d2b:::benchmark_data_smithi194_186548_object2617:head# 2024-04-29T14:53:21.245 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ea4b46:::benchmark_data_smithi194_186548_object50052:head# 2024-04-29T14:53:21.246 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36eafe19:::benchmark_data_smithi194_186548_object28445:head# 2024-04-29T14:53:21.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36eafebc:::benchmark_data_smithi194_186548_object72278:head# 2024-04-29T14:53:21.248 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ebf64c:::benchmark_data_smithi194_186548_object37649:head# 2024-04-29T14:53:21.248 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ec6416:::benchmark_data_smithi194_186548_object77727:head# 2024-04-29T14:53:21.249 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36edb0c1:::benchmark_data_smithi194_186548_object64273:head# 2024-04-29T14:53:21.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ef847f:::benchmark_data_smithi194_186548_object70711:head# 2024-04-29T14:53:21.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ef9ebe:::benchmark_data_smithi194_186548_object71532:head# 2024-04-29T14:53:21.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f1baca:::benchmark_data_smithi194_186548_object24517:head# 2024-04-29T14:53:21.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f22d29:::benchmark_data_smithi194_186548_object8562:head# 2024-04-29T14:53:21.253 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f28a0c:::benchmark_data_smithi194_186548_object23169:head# 2024-04-29T14:53:21.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f2c12e:::benchmark_data_smithi194_186548_object62176:head# 2024-04-29T14:53:21.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f30941:::benchmark_data_smithi194_186548_object26577:head# 2024-04-29T14:53:21.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f34dbc:::benchmark_data_smithi194_186548_object22685:head# 2024-04-29T14:53:21.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f35cb8:::benchmark_data_smithi194_186548_object30126:head# 2024-04-29T14:53:21.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f608d5:::benchmark_data_smithi194_186548_object42188:head# 2024-04-29T14:53:21.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f7b75e:::benchmark_data_smithi194_186548_object62094:head# 2024-04-29T14:53:21.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36f815b5:::benchmark_data_smithi194_186548_object12585:head# 2024-04-29T14:53:21.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36fabda1:::benchmark_data_smithi194_186548_object51609:head# 2024-04-29T14:53:21.256 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36fc2353:::benchmark_data_smithi194_186548_object438:head# 2024-04-29T14:53:21.256 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36fc826a:::benchmark_data_smithi194_186548_object5408:head# 2024-04-29T14:53:21.261 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36fc8958:::benchmark_data_smithi194_186548_object46819:head# 2024-04-29T14:53:21.261 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36fcbfd1:::benchmark_data_smithi194_186548_object12678:head# 2024-04-29T14:53:21.264 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36fd507a:::benchmark_data_smithi194_186548_object2873:head# 2024-04-29T14:53:21.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:36ff8a46:::benchmark_data_smithi194_186548_object22116:head# 2024-04-29T14:53:21.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3700427d:::benchmark_data_smithi194_186548_object58446:head# 2024-04-29T14:53:21.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3700cd7b:::benchmark_data_smithi194_186548_object80832:head# 2024-04-29T14:53:21.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3701b562:::benchmark_data_smithi194_186548_object42333:head# 2024-04-29T14:53:21.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3701cb33:::benchmark_data_smithi194_186548_object25401:head# 2024-04-29T14:53:21.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3702a9d5:::benchmark_data_smithi194_186548_object68803:head# 2024-04-29T14:53:21.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37031b70:::benchmark_data_smithi194_186548_object7727:head# 2024-04-29T14:53:21.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:370337ae:::benchmark_data_smithi194_186548_object24008:head# 2024-04-29T14:53:21.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37038236:::benchmark_data_smithi194_186548_object7374:head# 2024-04-29T14:53:21.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:370461ea:::benchmark_data_smithi194_186548_object34675:head# 2024-04-29T14:53:21.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3706517c:::benchmark_data_smithi194_186548_object56555:head# 2024-04-29T14:53:21.267 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3707b9bd:::benchmark_data_smithi194_186548_object72751:head# 2024-04-29T14:53:21.268 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3709fb54:::benchmark_data_smithi194_186548_object16127:head# 2024-04-29T14:53:21.268 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:370aa10d:::benchmark_data_smithi194_186548_object73770:head# 2024-04-29T14:53:21.268 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:370acb6b:::benchmark_data_smithi194_186548_object55832:head# 2024-04-29T14:53:21.269 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:370b9187:::benchmark_data_smithi194_186548_object59446:head# 2024-04-29T14:53:21.269 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:370c0339:::benchmark_data_smithi194_186548_object55113:head# 2024-04-29T14:53:21.270 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:370f6f69:::benchmark_data_smithi194_186548_object74328:head# 2024-04-29T14:53:21.271 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37101bcd:::benchmark_data_smithi194_186548_object17972:head# 2024-04-29T14:53:21.271 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3710ac1f:::benchmark_data_smithi194_186548_object24829:head# 2024-04-29T14:53:21.271 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37114ff4:::benchmark_data_smithi194_186548_object61757:head# 2024-04-29T14:53:21.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37116624:::benchmark_data_smithi194_186548_object45127:head# 2024-04-29T14:53:21.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3714d8a5:::benchmark_data_smithi194_186548_object34791:head# 2024-04-29T14:53:21.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3714f7cd:::benchmark_data_smithi194_186548_object22806:head# 2024-04-29T14:53:21.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3715d394:::benchmark_data_smithi194_186548_object65775:head# 2024-04-29T14:53:21.274 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3716d4de:::benchmark_data_smithi194_186548_object74614:head# 2024-04-29T14:53:21.274 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3718b896:::benchmark_data_smithi194_186548_object75492:head# 2024-04-29T14:53:21.275 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371954d2:::benchmark_data_smithi194_186548_object53558:head# 2024-04-29T14:53:21.275 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37197769:::benchmark_data_smithi194_186548_object31245:head# 2024-04-29T14:53:21.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371af259:::benchmark_data_smithi194_186548_object61448:head# 2024-04-29T14:53:21.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371afc76:::benchmark_data_smithi194_186548_object50532:head# 2024-04-29T14:53:21.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371bab2a:::benchmark_data_smithi194_186548_object42418:head# 2024-04-29T14:53:21.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371c26d5:::benchmark_data_smithi194_186548_object68312:head# 2024-04-29T14:53:21.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371d5cf0:::benchmark_data_smithi194_186548_object44582:head# 2024-04-29T14:53:21.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371e08dd:::benchmark_data_smithi194_186548_object72866:head# 2024-04-29T14:53:21.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371e80d1:::benchmark_data_smithi194_186548_object2306:head# 2024-04-29T14:53:21.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:371f5ed6:::benchmark_data_smithi194_186548_object29636:head# 2024-04-29T14:53:21.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3720c167:::benchmark_data_smithi194_186548_object21755:head# 2024-04-29T14:53:21.281 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3720cf88:::benchmark_data_smithi194_186548_object56260:head# 2024-04-29T14:53:21.281 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3720ee70:::benchmark_data_smithi194_186548_object55624:head# 2024-04-29T14:53:21.283 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3720f62b:::benchmark_data_smithi194_186548_object49856:head# 2024-04-29T14:53:21.284 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3721cf38:::benchmark_data_smithi194_186548_object83703:head# 2024-04-29T14:53:21.284 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3723e102:::benchmark_data_smithi194_186548_object31801:head# 2024-04-29T14:53:21.284 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37243bc9:::benchmark_data_smithi194_186548_object14724:head# 2024-04-29T14:53:21.284 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3724c830:::benchmark_data_smithi194_186548_object19640:head# 2024-04-29T14:53:21.284 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3726223d:::benchmark_data_smithi194_186548_object14774:head# 2024-04-29T14:53:21.285 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3726250a:::benchmark_data_smithi194_186548_object55687:head# 2024-04-29T14:53:21.285 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372629ea:::benchmark_data_smithi194_186548_object15879:head# 2024-04-29T14:53:21.286 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37267e4b:::benchmark_data_smithi194_186548_object6846:head# 2024-04-29T14:53:21.286 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37283069:::benchmark_data_smithi194_186548_object80497:head# 2024-04-29T14:53:21.287 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372892a3:::benchmark_data_smithi194_186548_object22202:head# 2024-04-29T14:53:21.287 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372993fa:::benchmark_data_smithi194_186548_object71098:head# 2024-04-29T14:53:21.288 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372a0beb:::benchmark_data_smithi194_186548_object7308:head# 2024-04-29T14:53:21.288 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372a89c2:::benchmark_data_smithi194_186548_object19945:head# 2024-04-29T14:53:21.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372a9bdc:::benchmark_data_smithi194_186548_object34054:head# 2024-04-29T14:53:21.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372be935:::benchmark_data_smithi194_186548_object60706:head# 2024-04-29T14:53:21.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372d43b7:::benchmark_data_smithi194_186548_object73556:head# 2024-04-29T14:53:21.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372f1ee8:::benchmark_data_smithi194_186548_object49892:head# 2024-04-29T14:53:21.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:372fa598:::benchmark_data_smithi194_186548_object46356:head# 2024-04-29T14:53:21.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3731e477:::benchmark_data_smithi194_186548_object8185:head# 2024-04-29T14:53:21.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37343e8b:::benchmark_data_smithi194_186548_object75231:head# 2024-04-29T14:53:21.292 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3734eed6:::benchmark_data_smithi194_186548_object41347:head# 2024-04-29T14:53:21.292 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3734f6ae:::benchmark_data_smithi194_186548_object63470:head# 2024-04-29T14:53:21.293 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3737385e:::benchmark_data_smithi194_186548_object76430:head# 2024-04-29T14:53:21.293 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37375fb2:::benchmark_data_smithi194_186548_object75977:head# 2024-04-29T14:53:21.294 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37384da8:::benchmark_data_smithi194_186548_object60788:head# 2024-04-29T14:53:21.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3738dd1a:::benchmark_data_smithi194_186548_object20304:head# 2024-04-29T14:53:21.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37397acd:::benchmark_data_smithi194_186548_object55059:head# 2024-04-29T14:53:21.297 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:373a080a:::benchmark_data_smithi194_186548_object2967:head# 2024-04-29T14:53:21.297 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:373a13dc:::benchmark_data_smithi194_186548_object84414:head# 2024-04-29T14:53:21.298 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:373a23ab:::benchmark_data_smithi194_186548_object65135:head# 2024-04-29T14:53:21.299 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:373a9837:::benchmark_data_smithi194_186548_object30044:head# 2024-04-29T14:53:21.299 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:373c2b94:::benchmark_data_smithi194_186548_object46142:head# 2024-04-29T14:53:21.299 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:373d897a:::benchmark_data_smithi194_186548_object5178:head# 2024-04-29T14:53:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:373e0cbc:::benchmark_data_smithi194_186548_object45117:head# 2024-04-29T14:53:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37404871:::benchmark_data_smithi194_186548_object27056:head# 2024-04-29T14:53:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37409e63:::benchmark_data_smithi194_186548_object46068:head# 2024-04-29T14:53:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3740c94c:::benchmark_data_smithi194_186548_object73143:head# 2024-04-29T14:53:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3740f139:::benchmark_data_smithi194_186548_object25948:head# 2024-04-29T14:53:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37417842:::benchmark_data_smithi194_186548_object54128:head# 2024-04-29T14:53:21.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37436a1c:::benchmark_data_smithi194_186548_object60471:head# 2024-04-29T14:53:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3743b277:::benchmark_data_smithi194_186548_object8439:head# 2024-04-29T14:53:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3743d7d2:::benchmark_data_smithi194_186548_object60635:head# 2024-04-29T14:53:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374438d9:::benchmark_data_smithi194_186548_object32656:head# 2024-04-29T14:53:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37444ea0:::benchmark_data_smithi194_186548_object57561:head# 2024-04-29T14:53:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37445b35:::benchmark_data_smithi194_186548_object36386:head# 2024-04-29T14:53:21.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37447420:::benchmark_data_smithi194_186548_object964:head# 2024-04-29T14:53:21.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37456aa3:::benchmark_data_smithi194_186548_object28225:head# 2024-04-29T14:53:21.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3745ee14:::benchmark_data_smithi194_186548_object68044:head# 2024-04-29T14:53:21.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374632b1:::benchmark_data_smithi194_186548_object2126:head# 2024-04-29T14:53:21.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37466f77:::benchmark_data_smithi194_186548_object51076:head# 2024-04-29T14:53:21.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374699fc:::benchmark_data_smithi194_186548_object78758:head# 2024-04-29T14:53:21.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37473fa5:::benchmark_data_smithi194_186548_object44592:head# 2024-04-29T14:53:21.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37486580:::benchmark_data_smithi194_186548_object40255:head# 2024-04-29T14:53:21.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3748815d:::benchmark_data_smithi194_186548_object40346:head# 2024-04-29T14:53:21.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3748a310:::benchmark_data_smithi194_186548_object10852:head# 2024-04-29T14:53:21.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3748b771:::benchmark_data_smithi194_186548_object56815:head# 2024-04-29T14:53:21.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3749f69f:::benchmark_data_smithi194_186548_object2434:head# 2024-04-29T14:53:21.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374b37f7:::benchmark_data_smithi194_186548_object80459:head# 2024-04-29T14:53:21.312 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374b5981:::benchmark_data_smithi194_186548_object55007:head# 2024-04-29T14:53:21.313 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374bacc8:::benchmark_data_smithi194_186548_object46616:head# 2024-04-29T14:53:21.313 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374cd659:::benchmark_data_smithi194_186548_object2510:head# 2024-04-29T14:53:21.314 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:374cf03a:::benchmark_data_smithi194_186548_object78049:head# 2024-04-29T14:53:21.314 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37509070:::benchmark_data_smithi194_186548_object26037:head# 2024-04-29T14:53:21.315 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37530ddd:::benchmark_data_smithi194_186548_object29732:head# 2024-04-29T14:53:21.315 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:375318d5:::benchmark_data_smithi194_186548_object28390:head# 2024-04-29T14:53:21.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3755ae13:::benchmark_data_smithi194_186548_object1106:head# 2024-04-29T14:53:21.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3756fe14:::benchmark_data_smithi194_186548_object53853:head# 2024-04-29T14:53:21.317 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3757ecea:::benchmark_data_smithi194_186548_object45478:head# 2024-04-29T14:53:21.318 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:375c0d4d:::benchmark_data_smithi194_186548_object33537:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:375c1fa8:::benchmark_data_smithi194_186548_object65965:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:375d33a7:::benchmark_data_smithi194_186548_object81138:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:375d7028:::benchmark_data_smithi194_186548_object63434:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:375ea99c:::benchmark_data_smithi194_186548_object84853:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3760b9f6:::benchmark_data_smithi194_186548_object57246:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3760c02b:::benchmark_data_smithi194_186548_object80910:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37621531:::benchmark_data_smithi194_186548_object72305:head# 2024-04-29T14:53:21.321 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376249a3:::benchmark_data_smithi194_186548_object83995:head# 2024-04-29T14:53:21.322 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37631a35:::benchmark_data_smithi194_186548_object65357:head# 2024-04-29T14:53:21.322 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3764cb99:::benchmark_data_smithi194_186548_object40599:head# 2024-04-29T14:53:21.322 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37654900:::benchmark_data_smithi194_186548_object77671:head# 2024-04-29T14:53:21.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3766ef3a:::benchmark_data_smithi194_186548_object85304:head# 2024-04-29T14:53:21.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376858c1:::benchmark_data_smithi194_186548_object84874:head# 2024-04-29T14:53:21.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3769b9f0:::benchmark_data_smithi194_186548_object10826:head# 2024-04-29T14:53:21.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376af6f0:::benchmark_data_smithi194_186548_object18654:head# 2024-04-29T14:53:21.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376cba4c:::benchmark_data_smithi194_186548_object19739:head# 2024-04-29T14:53:21.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376d4d4d:::benchmark_data_smithi194_186548_object33833:head# 2024-04-29T14:53:21.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376e676a:::benchmark_data_smithi194_186548_object67426:head# 2024-04-29T14:53:21.328 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376f715d:::benchmark_data_smithi194_186548_object53131:head# 2024-04-29T14:53:21.329 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:376fa38a:::benchmark_data_smithi194_186548_object60873:head# 2024-04-29T14:53:21.329 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3771e346:::benchmark_data_smithi194_186548_object65721:head# 2024-04-29T14:53:21.330 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3772c199:::benchmark_data_smithi194_186548_object47474:head# 2024-04-29T14:53:21.330 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3773e0ec:::benchmark_data_smithi194_186548_object42566:head# 2024-04-29T14:53:21.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3774cc15:::benchmark_data_smithi194_186548_object84397:head# 2024-04-29T14:53:21.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37758d2a:::benchmark_data_smithi194_186548_object60221:head# 2024-04-29T14:53:21.332 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37769a53:::benchmark_data_smithi194_186548_object41438:head# 2024-04-29T14:53:21.332 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3776f411:::benchmark_data_smithi194_186548_object62261:head# 2024-04-29T14:53:21.333 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3779a97d:::benchmark_data_smithi194_186548_object79499:head# 2024-04-29T14:53:21.333 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3779b651:::benchmark_data_smithi194_186548_object31186:head# 2024-04-29T14:53:21.333 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3779fb22:::benchmark_data_smithi194_186548_object80410:head# 2024-04-29T14:53:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:377a8246:::benchmark_data_smithi194_186548_object59832:head# 2024-04-29T14:53:21.334 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:377b297c:::benchmark_data_smithi194_186548_object10499:head# 2024-04-29T14:53:21.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:377ba0a8:::benchmark_data_smithi194_186548_object11260:head# 2024-04-29T14:53:21.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:377c2982:::benchmark_data_smithi194_186548_object73295:head# 2024-04-29T14:53:21.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:377c4d66:::benchmark_data_smithi194_186548_object50720:head# 2024-04-29T14:53:21.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:377c788c:::benchmark_data_smithi194_186548_object55716:head# 2024-04-29T14:53:21.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:377fab59:::benchmark_data_smithi194_186548_object29176:head# 2024-04-29T14:53:21.340 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3780246b:::benchmark_data_smithi194_186548_object54899:head# 2024-04-29T14:53:21.341 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3781fff7:::benchmark_data_smithi194_186548_object63020:head# 2024-04-29T14:53:21.343 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37821131:::benchmark_data_smithi194_186548_object72611:head# 2024-04-29T14:53:21.343 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3782d923:::benchmark_data_smithi194_186548_object58326:head# 2024-04-29T14:53:21.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37831918:::benchmark_data_smithi194_186548_object18883:head# 2024-04-29T14:53:21.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37831d12:::benchmark_data_smithi194_186548_object19784:head# 2024-04-29T14:53:21.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3783ba41:::benchmark_data_smithi194_186548_object83792:head# 2024-04-29T14:53:21.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3784f1d0:::benchmark_data_smithi194_186548_object16270:head# 2024-04-29T14:53:21.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:378671cf:::benchmark_data_smithi194_186548_object11377:head# 2024-04-29T14:53:21.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3786e98b:::benchmark_data_smithi194_186548_object81508:head# 2024-04-29T14:53:21.344 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37878d14:::benchmark_data_smithi194_186548_object17305:head# 2024-04-29T14:53:21.345 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37891189:::benchmark_data_smithi194_186548_object34961:head# 2024-04-29T14:53:21.345 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:378add1a:::benchmark_data_smithi194_186548_object66077:head# 2024-04-29T14:53:21.346 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:378b4f50:::benchmark_data_smithi194_186548_object48129:head# 2024-04-29T14:53:21.346 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:378cb668:::benchmark_data_smithi194_186548_object4014:head# 2024-04-29T14:53:21.347 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:378df738:::benchmark_data_smithi194_186548_object30807:head# 2024-04-29T14:53:21.347 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:378fb840:::benchmark_data_smithi194_186548_object74063:head# 2024-04-29T14:53:21.348 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:378fdce3:::benchmark_data_smithi194_186548_object36997:head# 2024-04-29T14:53:21.348 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37902068:::benchmark_data_smithi194_186548_object19830:head# 2024-04-29T14:53:21.349 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3790d36b:::benchmark_data_smithi194_186548_object13601:head# 2024-04-29T14:53:21.349 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3790ddc8:::benchmark_data_smithi194_186548_object65242:head# 2024-04-29T14:53:21.350 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37916e40:::benchmark_data_smithi194_186548_object30062:head# 2024-04-29T14:53:21.350 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3791c22e:::benchmark_data_smithi194_186548_object61124:head# 2024-04-29T14:53:21.351 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3791ee3d:::benchmark_data_smithi194_186548_object53898:head# 2024-04-29T14:53:21.351 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3792e83c:::benchmark_data_smithi194_186548_object26493:head# 2024-04-29T14:53:21.355 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37937b47:::benchmark_data_smithi194_186548_object26572:head# 2024-04-29T14:53:21.355 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3793d1ac:::benchmark_data_smithi194_186548_object72942:head# 2024-04-29T14:53:21.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3793e378:::benchmark_data_smithi194_186548_object43215:head# 2024-04-29T14:53:21.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37956146:::benchmark_data_smithi194_186548_object60645:head# 2024-04-29T14:53:21.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3795b032:::benchmark_data_smithi194_186548_object27980:head# 2024-04-29T14:53:21.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37976ac3:::benchmark_data_smithi194_186548_object61917:head# 2024-04-29T14:53:21.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37985543:::benchmark_data_smithi194_186548_object25390:head# 2024-04-29T14:53:21.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37986fed:::benchmark_data_smithi194_186548_object65739:head# 2024-04-29T14:53:21.357 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3798b20d:::benchmark_data_smithi194_186548_object25836:head# 2024-04-29T14:53:21.357 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37993f91:::benchmark_data_smithi194_186548_object67745:head# 2024-04-29T14:53:21.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:379cdfff:::benchmark_data_smithi194_186548_object21478:head# 2024-04-29T14:53:21.359 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:379d5ed4:::benchmark_data_smithi194_186548_object36183:head# 2024-04-29T14:53:21.359 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:379e778a:::benchmark_data_smithi194_186548_object66710:head# 2024-04-29T14:53:21.359 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:379f2be2:::benchmark_data_smithi194_186548_object49635:head# 2024-04-29T14:53:21.360 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:379f40a5:::benchmark_data_smithi194_186548_object35839:head# 2024-04-29T14:53:21.360 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a01220:::benchmark_data_smithi194_186548_object19165:head# 2024-04-29T14:53:21.365 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a0af6c:::benchmark_data_smithi194_186548_object43312:head# 2024-04-29T14:53:21.365 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a15073:::benchmark_data_smithi194_186548_object24921:head# 2024-04-29T14:53:21.365 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a19691:::benchmark_data_smithi194_186548_object25365:head# 2024-04-29T14:53:21.366 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a31cf5:::benchmark_data_smithi194_186548_object79044:head# 2024-04-29T14:53:21.366 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a40bc3:::benchmark_data_smithi194_186548_object29445:head# 2024-04-29T14:53:21.370 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a65213:::benchmark_data_smithi194_186548_object54997:head# 2024-04-29T14:53:21.370 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a66d6c:::benchmark_data_smithi194_186548_object75295:head# 2024-04-29T14:53:21.371 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a70667:::benchmark_data_smithi194_186548_object83085:head# 2024-04-29T14:53:21.371 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a70abd:::benchmark_data_smithi194_186548_object5615:head# 2024-04-29T14:53:21.371 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a758ce:::benchmark_data_smithi194_186548_object44640:head# 2024-04-29T14:53:21.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a821ce:::benchmark_data_smithi194_186548_object62467:head# 2024-04-29T14:53:21.374 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a8458f:::benchmark_data_smithi194_186548_object65134:head# 2024-04-29T14:53:21.374 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a8c181:::benchmark_data_smithi194_186548_object80157:head# 2024-04-29T14:53:21.375 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a941fd:::benchmark_data_smithi194_186548_object76465:head# 2024-04-29T14:53:21.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37a9cd2e:::benchmark_data_smithi194_186548_object25029:head# 2024-04-29T14:53:21.377 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37aa7d83:::benchmark_data_smithi194_186548_object69849:head# 2024-04-29T14:53:21.377 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ac00fc:::benchmark_data_smithi194_186548_object33811:head# 2024-04-29T14:53:21.378 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ac9efd:::benchmark_data_smithi194_186548_object51885:head# 2024-04-29T14:53:21.380 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ace45e:::benchmark_data_smithi194_186548_object31484:head# 2024-04-29T14:53:21.380 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ae6173:::benchmark_data_smithi194_186548_object59411:head# 2024-04-29T14:53:21.381 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37afba2a:::benchmark_data_smithi194_186548_object5971:head# 2024-04-29T14:53:21.381 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37afd0b9:::benchmark_data_smithi194_186548_object75197:head# 2024-04-29T14:53:21.382 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b19e5a:::benchmark_data_smithi194_186548_object60718:head# 2024-04-29T14:53:21.382 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b222fe:::benchmark_data_smithi194_186548_object58586:head# 2024-04-29T14:53:21.382 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b24f73:::benchmark_data_smithi194_186548_object76262:head# 2024-04-29T14:53:21.383 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b2b951:::benchmark_data_smithi194_186548_object50203:head# 2024-04-29T14:53:21.391 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b39ced:::benchmark_data_smithi194_186548_object84988:head# 2024-04-29T14:53:21.391 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b429ba:::benchmark_data_smithi194_186548_object27564:head# 2024-04-29T14:53:21.392 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b42a28:::benchmark_data_smithi194_186548_object71530:head# 2024-04-29T14:53:21.392 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b441f3:::benchmark_data_smithi194_186548_object41001:head# 2024-04-29T14:53:21.393 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b4e3bc:::benchmark_data_smithi194_186548_object4730:head# 2024-04-29T14:53:21.393 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b50737:::benchmark_data_smithi194_186548_object68517:head# 2024-04-29T14:53:21.394 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b50965:::benchmark_data_smithi194_186548_object31870:head# 2024-04-29T14:53:21.394 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b59e57:::benchmark_data_smithi194_186548_object56381:head# 2024-04-29T14:53:21.395 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b5d369:::benchmark_data_smithi194_186548_object15009:head# 2024-04-29T14:53:21.395 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b60e84:::benchmark_data_smithi194_186548_object84590:head# 2024-04-29T14:53:21.396 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b64bd3:::benchmark_data_smithi194_186548_object60942:head# 2024-04-29T14:53:21.396 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b6569d:::benchmark_data_smithi194_186548_object39343:head# 2024-04-29T14:53:21.401 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b6f440:::benchmark_data_smithi194_186548_object74263:head# 2024-04-29T14:53:21.401 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b6f5d4:::benchmark_data_smithi194_186548_object5785:head# 2024-04-29T14:53:21.402 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b88ac3:::benchmark_data_smithi194_186548_object50567:head# 2024-04-29T14:53:21.402 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b97eb6:::benchmark_data_smithi194_186548_object35028:head# 2024-04-29T14:53:21.403 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37b9f0d1:::benchmark_data_smithi194_186548_object68481:head# 2024-04-29T14:53:21.403 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37bd3a77:::benchmark_data_smithi194_186548_object24554:head# 2024-04-29T14:53:21.404 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37bd4f3c:::benchmark_data_smithi194_186548_object85111:head# 2024-04-29T14:53:21.404 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37be7adf:::benchmark_data_smithi194_186548_object21447:head# 2024-04-29T14:53:21.405 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37bf002b:::benchmark_data_smithi194_186548_object2940:head# 2024-04-29T14:53:21.405 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c07937:::benchmark_data_smithi194_186548_object40262:head# 2024-04-29T14:53:21.409 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c244fa:::benchmark_data_smithi194_186548_object64630:head# 2024-04-29T14:53:21.413 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c2c310:::benchmark_data_smithi194_186548_object53195:head# 2024-04-29T14:53:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c2dfe7:::benchmark_data_smithi194_186548_object4580:head# 2024-04-29T14:53:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c30d75:::benchmark_data_smithi194_186548_object7178:head# 2024-04-29T14:53:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c48f12:::benchmark_data_smithi194_186548_object49719:head# 2024-04-29T14:53:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c54f66:::benchmark_data_smithi194_186548_object9264:head# 2024-04-29T14:53:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c5ef35:::benchmark_data_smithi194_186548_object7479:head# 2024-04-29T14:53:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c5f2ab:::benchmark_data_smithi194_186548_object70931:head# 2024-04-29T14:53:21.418 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c61fb2:::benchmark_data_smithi194_186548_object29916:head# 2024-04-29T14:53:21.418 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c6820f:::benchmark_data_smithi194_186548_object3346:head# 2024-04-29T14:53:21.419 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37c9aa9f:::benchmark_data_smithi194_186548_object81216:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ca7d2c:::benchmark_data_smithi194_186548_object4431:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37cb10f9:::benchmark_data_smithi194_186548_object75565:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37cb32d7:::benchmark_data_smithi194_186548_object54270:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37cdd385:::benchmark_data_smithi194_186548_object9378:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ced86f:::benchmark_data_smithi194_186548_object51475:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d008b4:::benchmark_data_smithi194_186548_object50895:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d024e1:::benchmark_data_smithi194_186548_object4469:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d118a8:::benchmark_data_smithi194_186548_object81820:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d18bcd:::benchmark_data_smithi194_186548_object26196:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d1f8a0:::benchmark_data_smithi194_186548_object42312:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d24a08:::benchmark_data_smithi194_186548_object46255:head# 2024-04-29T14:53:21.426 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d29e68:::benchmark_data_smithi194_186548_object47892:head# 2024-04-29T14:53:21.427 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d309bd:::benchmark_data_smithi194_186548_object67885:head# 2024-04-29T14:53:21.427 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d34333:::benchmark_data_smithi194_186548_object63137:head# 2024-04-29T14:53:21.427 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d4ab1f:::benchmark_data_smithi194_186548_object24450:head# 2024-04-29T14:53:21.427 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d4fbf3:::benchmark_data_smithi194_186548_object50884:head# 2024-04-29T14:53:21.427 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d5730c:::benchmark_data_smithi194_186548_object42198:head# 2024-04-29T14:53:21.428 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d57699:::benchmark_data_smithi194_186548_object56321:head# 2024-04-29T14:53:21.428 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d60963:::benchmark_data_smithi194_186548_object23432:head# 2024-04-29T14:53:21.428 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d662ad:::benchmark_data_smithi194_186548_object52382:head# 2024-04-29T14:53:21.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d6a4d7:::benchmark_data_smithi194_186548_object61861:head# 2024-04-29T14:53:21.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d6f19d:::benchmark_data_smithi194_186548_object49026:head# 2024-04-29T14:53:21.430 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d74a1b:::benchmark_data_smithi194_186548_object37973:head# 2024-04-29T14:53:21.430 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d8a354:::benchmark_data_smithi194_186548_object69337:head# 2024-04-29T14:53:21.431 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d8c73a:::benchmark_data_smithi194_186548_object29432:head# 2024-04-29T14:53:21.431 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d92aa0:::benchmark_data_smithi194_186548_object45647:head# 2024-04-29T14:53:21.434 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d9bd11:::benchmark_data_smithi194_186548_object40843:head# 2024-04-29T14:53:21.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37d9cb5f:::benchmark_data_smithi194_186548_object61397:head# 2024-04-29T14:53:21.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37da7a7f:::benchmark_data_smithi194_186548_object41565:head# 2024-04-29T14:53:21.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37dcaa9c:::benchmark_data_smithi194_186548_object27047:head# 2024-04-29T14:53:21.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37dd5810:::benchmark_data_smithi194_186548_object20046:head# 2024-04-29T14:53:21.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ddf112:::benchmark_data_smithi194_186548_object13834:head# 2024-04-29T14:53:21.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37de41ab:::benchmark_data_smithi194_186548_object26452:head# 2024-04-29T14:53:21.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37de6713:::benchmark_data_smithi194_186548_object29498:head# 2024-04-29T14:53:21.436 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37dfc68d:::benchmark_data_smithi194_186548_object38511:head# 2024-04-29T14:53:21.436 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e02b17:::benchmark_data_smithi194_186548_object70594:head# 2024-04-29T14:53:21.437 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e03feb:::benchmark_data_smithi194_186548_object53608:head# 2024-04-29T14:53:21.437 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e07063:::benchmark_data_smithi194_186548_object62032:head# 2024-04-29T14:53:21.438 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e0f4e3:::benchmark_data_smithi194_186548_object39688:head# 2024-04-29T14:53:21.438 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e11387:::benchmark_data_smithi194_186548_object35697:head# 2024-04-29T14:53:21.439 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e143f3:::benchmark_data_smithi194_186548_object9354:head# 2024-04-29T14:53:21.442 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e1526d:::benchmark_data_smithi194_186548_object32262:head# 2024-04-29T14:53:21.442 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e23800:::benchmark_data_smithi194_186548_object33949:head# 2024-04-29T14:53:21.443 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e2d94f:::benchmark_data_smithi194_186548_object46755:head# 2024-04-29T14:53:21.443 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e301cd:::benchmark_data_smithi194_186548_object63250:head# 2024-04-29T14:53:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e483d3:::benchmark_data_smithi194_186548_object367:head# 2024-04-29T14:53:21.444 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e49474:::benchmark_data_smithi194_186548_object10053:head# 2024-04-29T14:53:21.445 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e5291f:::benchmark_data_smithi194_186548_object21011:head# 2024-04-29T14:53:21.445 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e61fa4:::benchmark_data_smithi194_186548_object8194:head# 2024-04-29T14:53:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e63263:::benchmark_data_smithi194_186548_object30363:head# 2024-04-29T14:53:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e690ba:::benchmark_data_smithi194_186548_object69519:head# 2024-04-29T14:53:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e69936:::benchmark_data_smithi194_186548_object35702:head# 2024-04-29T14:53:21.449 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e70e65:::benchmark_data_smithi194_186548_object138:head# 2024-04-29T14:53:21.450 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e71753:::benchmark_data_smithi194_186548_object35381:head# 2024-04-29T14:53:21.451 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e85513:::benchmark_data_smithi194_186548_object6723:head# 2024-04-29T14:53:21.451 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e8e4a8:::benchmark_data_smithi194_186548_object64737:head# 2024-04-29T14:53:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e9a396:::benchmark_data_smithi194_186548_object49063:head# 2024-04-29T14:53:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e9dfc2:::benchmark_data_smithi194_186548_object38438:head# 2024-04-29T14:53:21.453 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37e9ee9f:::benchmark_data_smithi194_186548_object38506:head# 2024-04-29T14:53:21.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ea0bdb:::benchmark_data_smithi194_186548_object58066:head# 2024-04-29T14:53:21.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ea6991:::benchmark_data_smithi194_186548_object52018:head# 2024-04-29T14:53:21.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37eb447b:::benchmark_data_smithi194_186548_object50778:head# 2024-04-29T14:53:21.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ebbc31:::benchmark_data_smithi194_186548_object48728:head# 2024-04-29T14:53:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ec8fc9:::benchmark_data_smithi194_186548_object32279:head# 2024-04-29T14:53:21.459 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ecd9a0:::benchmark_data_smithi194_186548_object76136:head# 2024-04-29T14:53:21.460 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ee1d43:::benchmark_data_smithi194_186548_object5220:head# 2024-04-29T14:53:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ee6b41:::benchmark_data_smithi194_186548_object2062:head# 2024-04-29T14:53:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37efccd5:::benchmark_data_smithi194_186548_object70432:head# 2024-04-29T14:53:21.461 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f48ff1:::benchmark_data_smithi194_186548_object58559:head# 2024-04-29T14:53:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f5731c:::benchmark_data_smithi194_186548_object47926:head# 2024-04-29T14:53:21.462 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f58574:::benchmark_data_smithi194_186548_object66204:head# 2024-04-29T14:53:21.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f67c1b:::benchmark_data_smithi194_186548_object29971:head# 2024-04-29T14:53:21.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f81190:::benchmark_data_smithi194_186548_object6965:head# 2024-04-29T14:53:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f8525b:::benchmark_data_smithi194_186548_object38228:head# 2024-04-29T14:53:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f910a3:::benchmark_data_smithi194_186548_object21628:head# 2024-04-29T14:53:21.464 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37f93eb1:::benchmark_data_smithi194_186548_object14020:head# 2024-04-29T14:53:21.465 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37fb7d08:::benchmark_data_smithi194_186548_object30365:head# 2024-04-29T14:53:21.465 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37fc0ca8:::benchmark_data_smithi194_186548_object21051:head# 2024-04-29T14:53:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37fd65b3:::benchmark_data_smithi194_186548_object41628:head# 2024-04-29T14:53:21.466 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ff1669:::benchmark_data_smithi194_186548_object66501:head# 2024-04-29T14:53:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:37ffb427:::benchmark_data_smithi194_186548_object83304:head# 2024-04-29T14:53:21.467 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3800b524:::benchmark_data_smithi194_186548_object11041:head# 2024-04-29T14:53:21.468 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3801e432:::benchmark_data_smithi194_186548_object20640:head# 2024-04-29T14:53:21.468 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38026532:::benchmark_data_smithi194_186548_object16334:head# 2024-04-29T14:53:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3802ca8b:::benchmark_data_smithi194_186548_object29228:head# 2024-04-29T14:53:21.469 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3802d0d3:::benchmark_data_smithi194_186548_object77547:head# 2024-04-29T14:53:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380437c2:::benchmark_data_smithi194_186548_object39197:head# 2024-04-29T14:53:21.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38045bf3:::benchmark_data_smithi194_186548_object22830:head# 2024-04-29T14:53:21.472 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3804db81:::benchmark_data_smithi194_186548_object21789:head# 2024-04-29T14:53:21.472 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3804edad:::benchmark_data_smithi194_186548_object33677:head# 2024-04-29T14:53:21.473 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38066868:::benchmark_data_smithi194_186548_object7648:head# 2024-04-29T14:53:21.473 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380769d1:::benchmark_data_smithi194_186548_object51920:head# 2024-04-29T14:53:21.473 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3807a381:::benchmark_data_smithi194_186548_object45715:head# 2024-04-29T14:53:21.474 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38097342:::benchmark_data_smithi194_186548_object8112:head# 2024-04-29T14:53:21.477 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38099925:::benchmark_data_smithi194_186548_object3355:head# 2024-04-29T14:53:21.477 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380be4bf:::benchmark_data_smithi194_186548_object3392:head# 2024-04-29T14:53:21.477 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380c2679:::benchmark_data_smithi194_186548_object82607:head# 2024-04-29T14:53:21.478 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380cb78f:::benchmark_data_smithi194_186548_object8610:head# 2024-04-29T14:53:21.478 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380dd467:::benchmark_data_smithi194_186548_object33983:head# 2024-04-29T14:53:21.478 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380e1125:::benchmark_data_smithi194_186548_object80069:head# 2024-04-29T14:53:21.478 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380f515e:::benchmark_data_smithi194_186548_object4110:head# 2024-04-29T14:53:21.478 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:380ff313:::benchmark_data_smithi194_186548_object49792:head# 2024-04-29T14:53:21.478 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38112fce:::benchmark_data_smithi194_186548_object16133:head# 2024-04-29T14:53:21.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3813674f:::benchmark_data_smithi194_186548_object23150:head# 2024-04-29T14:53:21.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381430bb:::benchmark_data_smithi194_186548_object65551:head# 2024-04-29T14:53:21.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38143788:::benchmark_data_smithi194_186548_object61777:head# 2024-04-29T14:53:21.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381447e4:::benchmark_data_smithi194_186548_object8373:head# 2024-04-29T14:53:21.484 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38149225:::benchmark_data_smithi194_186548_object23440:head# 2024-04-29T14:53:21.484 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3814a51c:::benchmark_data_smithi194_186548_object3083:head# 2024-04-29T14:53:21.484 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3814ce59:::benchmark_data_smithi194_186548_object72785:head# 2024-04-29T14:53:21.485 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38154f8c:::benchmark_data_smithi194_186548_object56391:head# 2024-04-29T14:53:21.485 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3815bc08:::benchmark_data_smithi194_186548_object70707:head# 2024-04-29T14:53:21.487 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38195597:::benchmark_data_smithi194_186548_object34033:head# 2024-04-29T14:53:21.487 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3819a509:::benchmark_data_smithi194_186548_object13414:head# 2024-04-29T14:53:21.488 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381b596b:::benchmark_data_smithi194_186548_object13512:head# 2024-04-29T14:53:21.488 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381b6803:::benchmark_data_smithi194_186548_object24558:head# 2024-04-29T14:53:21.488 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381bbe90:::benchmark_data_smithi194_186548_object69011:head# 2024-04-29T14:53:21.489 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381c32d6:::benchmark_data_smithi194_186548_object79950:head# 2024-04-29T14:53:21.491 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381c626b:::benchmark_data_smithi194_186548_object296:head# 2024-04-29T14:53:21.492 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381d275f:::benchmark_data_smithi194_186548_object52330:head# 2024-04-29T14:53:21.492 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381d693d:::benchmark_data_smithi194_186548_object14482:head# 2024-04-29T14:53:21.492 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:381dff4f:::benchmark_data_smithi194_186548_object44468:head# 2024-04-29T14:53:21.494 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38203092:::benchmark_data_smithi194_186548_object60007:head# 2024-04-29T14:53:21.494 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3820b9a8:::benchmark_data_smithi194_186548_object40564:head# 2024-04-29T14:53:21.494 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3820fc5c:::benchmark_data_smithi194_186548_object4583:head# 2024-04-29T14:53:21.495 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38219dd1:::benchmark_data_smithi194_186548_object30016:head# 2024-04-29T14:53:21.495 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38252fb1:::benchmark_data_smithi194_186548_object69941:head# 2024-04-29T14:53:21.499 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38279128:::benchmark_data_smithi194_186548_object47272:head# 2024-04-29T14:53:21.499 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38281d39:::benchmark_data_smithi194_186548_object39245:head# 2024-04-29T14:53:21.499 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38289cff:::benchmark_data_smithi194_186548_object28049:head# 2024-04-29T14:53:21.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3828d227:::benchmark_data_smithi194_186548_object14969:head# 2024-04-29T14:53:21.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3829417a:::benchmark_data_smithi194_186548_object10540:head# 2024-04-29T14:53:21.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382977c2:::benchmark_data_smithi194_186548_object39762:head# 2024-04-29T14:53:21.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382a26df:::benchmark_data_smithi194_186548_object63356:head# 2024-04-29T14:53:21.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382a8ab1:::benchmark_data_smithi194_186548_object18663:head# 2024-04-29T14:53:21.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382c8a94:::benchmark_data_smithi194_186548_object37637:head# 2024-04-29T14:53:21.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382cb0b1:::benchmark_data_smithi194_186548_object73591:head# 2024-04-29T14:53:21.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382cd1a5:::benchmark_data_smithi194_186548_object23682:head# 2024-04-29T14:53:21.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382cf3b1:::benchmark_data_smithi194_186548_object29324:head# 2024-04-29T14:53:21.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382db19e:::benchmark_data_smithi194_186548_object72130:head# 2024-04-29T14:53:21.503 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382e8337:::benchmark_data_smithi194_186548_object22040:head# 2024-04-29T14:53:21.503 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:382f7a2f:::benchmark_data_smithi194_186548_object80849:head# 2024-04-29T14:53:21.503 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383042bf:::benchmark_data_smithi194_186548_object26712:head# 2024-04-29T14:53:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38309cbf:::benchmark_data_smithi194_186548_object77201:head# 2024-04-29T14:53:21.504 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3830b6ae:::benchmark_data_smithi194_186548_object52537:head# 2024-04-29T14:53:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383104d2:::benchmark_data_smithi194_186548_object20767:head# 2024-04-29T14:53:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3831a64f:::benchmark_data_smithi194_186548_object20966:head# 2024-04-29T14:53:21.505 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3831b26b:::benchmark_data_smithi194_186548_object17052:head# 2024-04-29T14:53:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38323b74:::benchmark_data_smithi194_186548_object49688:head# 2024-04-29T14:53:21.506 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3833553d:::benchmark_data_smithi194_186548_object30021:head# 2024-04-29T14:53:21.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38347464:::benchmark_data_smithi194_186548_object34064:head# 2024-04-29T14:53:21.513 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3835b00f:::benchmark_data_smithi194_186548_object19401:head# 2024-04-29T14:53:21.519 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3838417b:::benchmark_data_smithi194_186548_object50003:head# 2024-04-29T14:53:21.520 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383893ae:::benchmark_data_smithi194_186548_object13766:head# 2024-04-29T14:53:21.520 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38399435:::benchmark_data_smithi194_186548_object62244:head# 2024-04-29T14:53:21.522 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383a4ccb:::benchmark_data_smithi194_186548_object4277:head# 2024-04-29T14:53:21.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383a8edf:::benchmark_data_smithi194_186548_object29694:head# 2024-04-29T14:53:21.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383b8b6b:::benchmark_data_smithi194_186548_object85017:head# 2024-04-29T14:53:21.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383d03d8:::benchmark_data_smithi194_186548_object50814:head# 2024-04-29T14:53:21.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383dd688:::benchmark_data_smithi194_186548_object66146:head# 2024-04-29T14:53:21.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383df05a:::benchmark_data_smithi194_186548_object623:head# 2024-04-29T14:53:21.525 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383e46b0:::benchmark_data_smithi194_186548_object63214:head# 2024-04-29T14:53:21.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:383e7436:::benchmark_data_smithi194_186548_object61049:head# 2024-04-29T14:53:21.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38406819:::benchmark_data_smithi194_186548_object39925:head# 2024-04-29T14:53:21.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38409ad8:::benchmark_data_smithi194_186548_object63834:head# 2024-04-29T14:53:21.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384131fe:::benchmark_data_smithi194_186548_object33725:head# 2024-04-29T14:53:21.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38423652:::benchmark_data_smithi194_186548_object84558:head# 2024-04-29T14:53:21.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38425f7c:::benchmark_data_smithi194_186548_object25859:head# 2024-04-29T14:53:21.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3843075b:::benchmark_data_smithi194_186548_object52668:head# 2024-04-29T14:53:21.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38433ddf:::benchmark_data_smithi194_186548_object50876:head# 2024-04-29T14:53:21.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3843e310:::benchmark_data_smithi194_186548_object78713:head# 2024-04-29T14:53:21.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384583fb:::benchmark_data_smithi194_186548_object42143:head# 2024-04-29T14:53:21.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3846020a:::benchmark_data_smithi194_186548_object21370:head# 2024-04-29T14:53:21.532 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3846303e:::benchmark_data_smithi194_186548_object36366:head# 2024-04-29T14:53:21.532 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38465472:::benchmark_data_smithi194_186548_object6221:head# 2024-04-29T14:53:21.534 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3846a4d6:::benchmark_data_smithi194_186548_object27810:head# 2024-04-29T14:53:21.534 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3846e144:::benchmark_data_smithi194_186548_object12282:head# 2024-04-29T14:53:21.534 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38488f11:::benchmark_data_smithi194_186548_object34773:head# 2024-04-29T14:53:21.536 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3848b1db:::benchmark_data_smithi194_186548_object58728:head# 2024-04-29T14:53:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384a0539:::benchmark_data_smithi194_186548_object61750:head# 2024-04-29T14:53:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384a6a9b:::benchmark_data_smithi194_186548_object12366:head# 2024-04-29T14:53:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384aa113:::benchmark_data_smithi194_186548_object32867:head# 2024-04-29T14:53:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384b3e69:::benchmark_data_smithi194_186548_object23771:head# 2024-04-29T14:53:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384c6dcf:::benchmark_data_smithi194_186548_object65718:head# 2024-04-29T14:53:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384cf9a6:::benchmark_data_smithi194_186548_object12584:head# 2024-04-29T14:53:21.541 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384d0850:::benchmark_data_smithi194_186548_object20675:head# 2024-04-29T14:53:21.542 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384d24a8:::benchmark_data_smithi194_186548_object75181:head# 2024-04-29T14:53:21.542 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:384ef537:::benchmark_data_smithi194_186548_object81029:head# 2024-04-29T14:53:21.542 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38513615:::benchmark_data_smithi194_186548_object14338:head# 2024-04-29T14:53:21.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38521c38:::benchmark_data_smithi194_186548_object22178:head# 2024-04-29T14:53:21.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38526846:::benchmark_data_smithi194_186548_object28409:head# 2024-04-29T14:53:21.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38535e46:::benchmark_data_smithi194_186548_object17018:head# 2024-04-29T14:53:21.544 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38549bf2:::benchmark_data_smithi194_186548_object55438:head# 2024-04-29T14:53:21.544 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38549d0c:::benchmark_data_smithi194_186548_object83035:head# 2024-04-29T14:53:21.545 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3854fd51:::benchmark_data_smithi194_186548_object9421:head# 2024-04-29T14:53:21.545 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38558bc8:::benchmark_data_smithi194_186548_object36517:head# 2024-04-29T14:53:21.546 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38584ed8:::benchmark_data_smithi194_186548_object70631:head# 2024-04-29T14:53:21.546 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3858e05c:::benchmark_data_smithi194_186548_object69770:head# 2024-04-29T14:53:21.547 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3859982e:::benchmark_data_smithi194_186548_object70680:head# 2024-04-29T14:53:21.549 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385aca15:::benchmark_data_smithi194_186548_object6293:head# 2024-04-29T14:53:21.549 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385bd47b:::benchmark_data_smithi194_186548_object25108:head# 2024-04-29T14:53:21.550 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385c0b51:::benchmark_data_smithi194_186548_object75564:head# 2024-04-29T14:53:21.550 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385c5e76:::benchmark_data_smithi194_186548_object83540:head# 2024-04-29T14:53:21.551 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385c630a:::benchmark_data_smithi194_186548_object22083:head# 2024-04-29T14:53:21.551 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385c8f2b:::benchmark_data_smithi194_186548_object14551:head# 2024-04-29T14:53:21.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385c99fe:::benchmark_data_smithi194_186548_object78824:head# 2024-04-29T14:53:21.552 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385e746f:::benchmark_data_smithi194_186548_object13899:head# 2024-04-29T14:53:21.553 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385ec4ca:::benchmark_data_smithi194_186548_object83844:head# 2024-04-29T14:53:21.553 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385eeb3c:::benchmark_data_smithi194_186548_object2237:head# 2024-04-29T14:53:21.553 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385f947a:::benchmark_data_smithi194_186548_object75493:head# 2024-04-29T14:53:21.554 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385faebb:::benchmark_data_smithi194_186548_object59130:head# 2024-04-29T14:53:21.554 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:385fed9d:::benchmark_data_smithi194_186548_object77248:head# 2024-04-29T14:53:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386048ee:::benchmark_data_smithi194_186548_object74976:head# 2024-04-29T14:53:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386055ea:::benchmark_data_smithi194_186548_object76445:head# 2024-04-29T14:53:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3860a792:::benchmark_data_smithi194_186548_object6950:head# 2024-04-29T14:53:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38624f44:::benchmark_data_smithi194_186548_object52316:head# 2024-04-29T14:53:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3862cbc5:::benchmark_data_smithi194_186548_object29037:head# 2024-04-29T14:53:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38636931:::benchmark_data_smithi194_186548_object59957:head# 2024-04-29T14:53:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3864a8e7:::benchmark_data_smithi194_186548_object54598:head# 2024-04-29T14:53:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3865c7fd:::benchmark_data_smithi194_186548_object12560:head# 2024-04-29T14:53:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38663d9c:::benchmark_data_smithi194_186548_object26919:head# 2024-04-29T14:53:21.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3867d7ac:::benchmark_data_smithi194_186548_object37866:head# 2024-04-29T14:53:21.563 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3868799c:::benchmark_data_smithi194_186548_object80639:head# 2024-04-29T14:53:21.563 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386982db:::benchmark_data_smithi194_186548_object56673:head# 2024-04-29T14:53:21.563 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386ae022:::benchmark_data_smithi194_186548_object20716:head# 2024-04-29T14:53:21.564 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386af950:::benchmark_data_smithi194_186548_object84700:head# 2024-04-29T14:53:21.565 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386c98b6:::benchmark_data_smithi194_186548_object83097:head# 2024-04-29T14:53:21.566 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386ca29e:::benchmark_data_smithi194_186548_object32151:head# 2024-04-29T14:53:21.566 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386ccde1:::benchmark_data_smithi194_186548_object43530:head# 2024-04-29T14:53:21.567 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386d1e83:::benchmark_data_smithi194_186548_object23094:head# 2024-04-29T14:53:21.567 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386e00f2:::benchmark_data_smithi194_186548_object32789:head# 2024-04-29T14:53:21.568 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386e5269:::benchmark_data_smithi194_186548_object48495:head# 2024-04-29T14:53:21.568 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386e67f2:::benchmark_data_smithi194_186548_object72387:head# 2024-04-29T14:53:21.569 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386e7893:::benchmark_data_smithi194_186548_object51360:head# 2024-04-29T14:53:21.571 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:386fe11e:::benchmark_data_smithi194_186548_object5414:head# 2024-04-29T14:53:21.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38701e4d:::benchmark_data_smithi194_186548_object55718:head# 2024-04-29T14:53:21.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38704695:::benchmark_data_smithi194_186548_object41908:head# 2024-04-29T14:53:21.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:387185dd:::benchmark_data_smithi194_186548_object9409:head# 2024-04-29T14:53:21.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3871b2a2:::benchmark_data_smithi194_186548_object70221:head# 2024-04-29T14:53:21.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38733fa7:::benchmark_data_smithi194_186548_object26940:head# 2024-04-29T14:53:21.573 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38751c39:::benchmark_data_smithi194_186548_object72153:head# 2024-04-29T14:53:21.573 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:387727d9:::benchmark_data_smithi194_186548_object80196:head# 2024-04-29T14:53:21.573 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3879acd8:::benchmark_data_smithi194_186548_object39745:head# 2024-04-29T14:53:21.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:21 smithi194 ceph-mon[44089]: osdmap e1034: 12 total, 11 up, 9 in 2024-04-29T14:53:21.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:21 smithi194 ceph-mon[44089]: pgmap v2611: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 61 MiB/s wr, 980 op/s; 19136/208518 objects degraded (9.177%); 7.2 MiB/s, 114 objects/s recovering 2024-04-29T14:53:21.575 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:387cea6c:::benchmark_data_smithi194_186548_object18560:head# 2024-04-29T14:53:21.575 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:387e60a5:::benchmark_data_smithi194_186548_object80582:head# 2024-04-29T14:53:21.575 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:387f6022:::benchmark_data_smithi194_186548_object29411:head# 2024-04-29T14:53:21.576 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:387f6e71:::benchmark_data_smithi194_186548_object78286:head# 2024-04-29T14:53:21.576 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:387f866e:::benchmark_data_smithi194_186548_object45039:head# 2024-04-29T14:53:21.576 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3880d432:::benchmark_data_smithi194_186548_object8702:head# 2024-04-29T14:53:21.576 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3880f1da:::benchmark_data_smithi194_186548_object66947:head# 2024-04-29T14:53:21.577 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38818b11:::benchmark_data_smithi194_186548_object80848:head# 2024-04-29T14:53:21.578 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3883eeeb:::benchmark_data_smithi194_186548_object9484:head# 2024-04-29T14:53:21.578 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38858cfe:::benchmark_data_smithi194_186548_object10201:head# 2024-04-29T14:53:21.579 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388755b7:::benchmark_data_smithi194_186548_object48783:head# 2024-04-29T14:53:21.579 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3887b7dd:::benchmark_data_smithi194_186548_object54214:head# 2024-04-29T14:53:21.580 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3887e3fd:::benchmark_data_smithi194_186548_object32346:head# 2024-04-29T14:53:21.580 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3888b4f1:::benchmark_data_smithi194_186548_object29468:head# 2024-04-29T14:53:21.580 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3889e333:::benchmark_data_smithi194_186548_object82001:head# 2024-04-29T14:53:21.581 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388ba6b1:::benchmark_data_smithi194_186548_object73803:head# 2024-04-29T14:53:21.581 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388bc22b:::benchmark_data_smithi194_186548_object59982:head# 2024-04-29T14:53:21.582 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388c184c:::benchmark_data_smithi194_186548_object25691:head# 2024-04-29T14:53:21.582 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388cee1e:::benchmark_data_smithi194_186548_object33685:head# 2024-04-29T14:53:21.583 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388d12ca:::benchmark_data_smithi194_186548_object3058:head# 2024-04-29T14:53:21.583 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388d27d8:::benchmark_data_smithi194_186548_object37490:head# 2024-04-29T14:53:21.584 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388d93c0:::benchmark_data_smithi194_186548_object19948:head# 2024-04-29T14:53:21.584 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388dc980:::benchmark_data_smithi194_186548_object80165:head# 2024-04-29T14:53:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388e7e17:::benchmark_data_smithi194_186548_object48589:head# 2024-04-29T14:53:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:388f0e09:::benchmark_data_smithi194_186548_object59610:head# 2024-04-29T14:53:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38911f4c:::benchmark_data_smithi194_186548_object65627:head# 2024-04-29T14:53:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389143d2:::benchmark_data_smithi194_186548_object403:head# 2024-04-29T14:53:21.589 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3891b1ce:::benchmark_data_smithi194_186548_object35919:head# 2024-04-29T14:53:21.589 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3891f06f:::benchmark_data_smithi194_186548_object57488:head# 2024-04-29T14:53:21.589 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38926362:::benchmark_data_smithi194_186548_object61322:head# 2024-04-29T14:53:21.589 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389282db:::benchmark_data_smithi194_186548_object74929:head# 2024-04-29T14:53:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38948fa5:::benchmark_data_smithi194_186548_object70249:head# 2024-04-29T14:53:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3894e19f:::benchmark_data_smithi194_186548_object45832:head# 2024-04-29T14:53:21.596 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3894fd96:::benchmark_data_smithi194_186548_object65561:head# 2024-04-29T14:53:21.596 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38957468:::benchmark_data_smithi194_186548_object11379:head# 2024-04-29T14:53:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38963041:::benchmark_data_smithi194_186548_object41533:head# 2024-04-29T14:53:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38963623:::benchmark_data_smithi194_186548_object41373:head# 2024-04-29T14:53:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38963880:::benchmark_data_smithi194_186548_object63569:head# 2024-04-29T14:53:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3896717f:::benchmark_data_smithi194_186548_object47572:head# 2024-04-29T14:53:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3897e619:::benchmark_data_smithi194_186548_object74111:head# 2024-04-29T14:53:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389987d6:::benchmark_data_smithi194_186548_object70548:head# 2024-04-29T14:53:21.597 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3899dd36:::benchmark_data_smithi194_186548_object13542:head# 2024-04-29T14:53:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389a7fda:::benchmark_data_smithi194_186548_object25051:head# 2024-04-29T14:53:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389b8ab2:::benchmark_data_smithi194_186548_object70408:head# 2024-04-29T14:53:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389c7b85:::benchmark_data_smithi194_186548_object62659:head# 2024-04-29T14:53:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389cb91f:::benchmark_data_smithi194_186548_object42562:head# 2024-04-29T14:53:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389db280:::benchmark_data_smithi194_186548_object51075:head# 2024-04-29T14:53:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389f84f1:::benchmark_data_smithi194_186548_object13795:head# 2024-04-29T14:53:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389ff57e:::benchmark_data_smithi194_186548_object78971:head# 2024-04-29T14:53:21.602 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:389ffdbd:::benchmark_data_smithi194_186548_object52612:head# 2024-04-29T14:53:21.602 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a0759b:::benchmark_data_smithi194_186548_object15453:head# 2024-04-29T14:53:21.602 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a10c27:::benchmark_data_smithi194_186548_object78048:head# 2024-04-29T14:53:21.603 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a1e765:::benchmark_data_smithi194_186548_object42730:head# 2024-04-29T14:53:21.604 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a2183b:::benchmark_data_smithi194_186548_object6157:head# 2024-04-29T14:53:21.607 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 108014 107998 97.8028 66.625 0.0209026 0.010223 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a2f653:::benchmark_data_smithi194_186548_object3881:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a46fb8:::benchmark_data_smithi194_186548_object26883:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a59098:::benchmark_data_smithi194_186548_object25140:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a71146:::benchmark_data_smithi194_186548_object56920:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a87452:::benchmark_data_smithi194_186548_object4698:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a99b86:::benchmark_data_smithi194_186548_object82795:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38a9cd6b:::benchmark_data_smithi194_186548_object31198:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38aa053e:::benchmark_data_smithi194_186548_object60056:head# 2024-04-29T14:53:21.610 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ad7bb7:::benchmark_data_smithi194_186548_object68492:head# 2024-04-29T14:53:21.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ae297e:::benchmark_data_smithi194_186548_object61883:head# 2024-04-29T14:53:21.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38af3399:::benchmark_data_smithi194_186548_object58799:head# 2024-04-29T14:53:21.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b135c0:::benchmark_data_smithi194_186548_object30235:head# 2024-04-29T14:53:21.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b1803f:::benchmark_data_smithi194_186548_object68855:head# 2024-04-29T14:53:21.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b32baf:::benchmark_data_smithi194_186548_object80291:head# 2024-04-29T14:53:21.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b343c1:::benchmark_data_smithi194_186548_object58459:head# 2024-04-29T14:53:21.611 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b4d317:::benchmark_data_smithi194_186548_object10438:head# 2024-04-29T14:53:21.612 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b523df:::benchmark_data_smithi194_186548_object85041:head# 2024-04-29T14:53:21.617 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b63c45:::benchmark_data_smithi194_186548_object52962:head# 2024-04-29T14:53:21.617 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b730a7:::benchmark_data_smithi194_186548_object42117:head# 2024-04-29T14:53:21.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b743ad:::benchmark_data_smithi194_186548_object79485:head# 2024-04-29T14:53:21.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b7861f:::benchmark_data_smithi194_186548_object43644:head# 2024-04-29T14:53:21.618 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b8aaf5:::benchmark_data_smithi194_186548_object74304:head# 2024-04-29T14:53:21.619 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38b98ab1:::benchmark_data_smithi194_186548_object58136:head# 2024-04-29T14:53:21.620 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bafb28:::benchmark_data_smithi194_186548_object70865:head# 2024-04-29T14:53:21.620 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bba71c:::benchmark_data_smithi194_186548_object63071:head# 2024-04-29T14:53:21.621 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bcc2b2:::benchmark_data_smithi194_186548_object28588:head# 2024-04-29T14:53:21.621 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bd8978:::benchmark_data_smithi194_186548_object13917:head# 2024-04-29T14:53:21.622 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bd90f0:::benchmark_data_smithi194_186548_object77132:head# 2024-04-29T14:53:21.622 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bea73f:::benchmark_data_smithi194_186548_object49921:head# 2024-04-29T14:53:21.622 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bf3e13:::benchmark_data_smithi194_186548_object18364:head# 2024-04-29T14:53:21.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:21 smithi040 ceph-mon[39581]: osdmap e1034: 12 total, 11 up, 9 in 2024-04-29T14:53:21.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:21 smithi040 ceph-mon[39581]: pgmap v2611: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 61 MiB/s wr, 980 op/s; 19136/208518 objects degraded (9.177%); 7.2 MiB/s, 114 objects/s recovering 2024-04-29T14:53:21.623 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bfb883:::benchmark_data_smithi194_186548_object63103:head# 2024-04-29T14:53:21.623 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38bfd72c:::benchmark_data_smithi194_186548_object83892:head# 2024-04-29T14:53:21.624 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c1939f:::benchmark_data_smithi194_186548_object19330:head# 2024-04-29T14:53:21.624 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c1a5d6:::benchmark_data_smithi194_186548_object19470:head# 2024-04-29T14:53:21.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c1b7a8:::benchmark_data_smithi194_186548_object10363:head# 2024-04-29T14:53:21.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c2e443:::benchmark_data_smithi194_186548_object6194:head# 2024-04-29T14:53:21.630 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c302dc:::benchmark_data_smithi194_186548_object33151:head# 2024-04-29T14:53:21.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c38c4d:::benchmark_data_smithi194_186548_object74158:head# 2024-04-29T14:53:21.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c3b830:::benchmark_data_smithi194_186548_object53095:head# 2024-04-29T14:53:21.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c3cc96:::benchmark_data_smithi194_186548_object67774:head# 2024-04-29T14:53:21.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c3d08f:::benchmark_data_smithi194_186548_object66211:head# 2024-04-29T14:53:21.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c3df87:::benchmark_data_smithi194_186548_object11671:head# 2024-04-29T14:53:21.632 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c3e19f:::benchmark_data_smithi194_186548_object84683:head# 2024-04-29T14:53:21.632 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c3ea11:::benchmark_data_smithi194_186548_object21837:head# 2024-04-29T14:53:21.632 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c4c1f0:::benchmark_data_smithi194_186548_object78546:head# 2024-04-29T14:53:21.633 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c6198e:::benchmark_data_smithi194_186548_object49546:head# 2024-04-29T14:53:21.633 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c62d75:::benchmark_data_smithi194_186548_object27571:head# 2024-04-29T14:53:21.634 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38c80629:::benchmark_data_smithi194_186548_object57707:head# 2024-04-29T14:53:21.634 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ca2bab:::benchmark_data_smithi194_186548_object25101:head# 2024-04-29T14:53:21.634 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ca4852:::benchmark_data_smithi194_186548_object72879:head# 2024-04-29T14:53:21.635 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ca5a16:::benchmark_data_smithi194_186548_object2679:head# 2024-04-29T14:53:21.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:21 smithi121 ceph-mon[44533]: osdmap e1034: 12 total, 11 up, 9 in 2024-04-29T14:53:21.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:21 smithi121 ceph-mon[44533]: pgmap v2611: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 61 MiB/s wr, 980 op/s; 19136/208518 objects degraded (9.177%); 7.2 MiB/s, 114 objects/s recovering 2024-04-29T14:53:21.636 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38caeddb:::benchmark_data_smithi194_186548_object12962:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cb405a:::benchmark_data_smithi194_186548_object55333:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cb6586:::benchmark_data_smithi194_186548_object58604:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cba04c:::benchmark_data_smithi194_186548_object73089:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cdf40b:::benchmark_data_smithi194_186548_object24988:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cdf98d:::benchmark_data_smithi194_186548_object37824:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cf179c:::benchmark_data_smithi194_186548_object28141:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cf2848:::benchmark_data_smithi194_186548_object40707:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cf5401:::benchmark_data_smithi194_186548_object3394:head# 2024-04-29T14:53:21.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cf5e44:::benchmark_data_smithi194_186548_object34824:head# 2024-04-29T14:53:21.644 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cf7181:::benchmark_data_smithi194_186548_object38205:head# 2024-04-29T14:53:21.644 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38cf73d7:::benchmark_data_smithi194_186548_object76770:head# 2024-04-29T14:53:21.644 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d0a18d:::benchmark_data_smithi194_186548_object38157:head# 2024-04-29T14:53:21.645 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d1abd1:::benchmark_data_smithi194_186548_object20914:head# 2024-04-29T14:53:21.645 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d1b4d3:::benchmark_data_smithi194_186548_object40913:head# 2024-04-29T14:53:21.645 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d2a4c5:::benchmark_data_smithi194_186548_object41419:head# 2024-04-29T14:53:21.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d38507:::benchmark_data_smithi194_186548_object74682:head# 2024-04-29T14:53:21.647 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d3a4d7:::benchmark_data_smithi194_186548_object19767:head# 2024-04-29T14:53:21.651 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d45f7a:::benchmark_data_smithi194_186548_object6039:head# 2024-04-29T14:53:21.651 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d50778:::benchmark_data_smithi194_186548_object84858:head# 2024-04-29T14:53:21.651 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d5995b:::benchmark_data_smithi194_186548_object55462:head# 2024-04-29T14:53:21.652 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d5ddfe:::benchmark_data_smithi194_186548_object84480:head# 2024-04-29T14:53:21.655 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d5ed99:::benchmark_data_smithi194_186548_object63021:head# 2024-04-29T14:53:21.655 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d66279:::benchmark_data_smithi194_186548_object31596:head# 2024-04-29T14:53:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d67486:::benchmark_data_smithi194_186548_object64722:head# 2024-04-29T14:53:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d6f3ae:::benchmark_data_smithi194_186548_object73076:head# 2024-04-29T14:53:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d70b93:::benchmark_data_smithi194_186548_object31418:head# 2024-04-29T14:53:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d8b32b:::benchmark_data_smithi194_186548_object70229:head# 2024-04-29T14:53:21.658 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38d9c912:::benchmark_data_smithi194_186548_object40150:head# 2024-04-29T14:53:21.661 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38dfce11:::benchmark_data_smithi194_186548_object23028:head# 2024-04-29T14:53:21.662 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e01efc:::benchmark_data_smithi194_186548_object21426:head# 2024-04-29T14:53:21.663 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e075c5:::benchmark_data_smithi194_186548_object21103:head# 2024-04-29T14:53:21.663 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e0bd43:::benchmark_data_smithi194_186548_object59816:head# 2024-04-29T14:53:21.673 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e1527e:::benchmark_data_smithi194_186548_object56467:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e28acd:::benchmark_data_smithi194_186548_object48117:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e2ee71:::benchmark_data_smithi194_186548_object78529:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e4384c:::benchmark_data_smithi194_186548_object5552:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e52e61:::benchmark_data_smithi194_186548_object59367:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e55a74:::benchmark_data_smithi194_186548_object13728:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e59ae4:::benchmark_data_smithi194_186548_object63544:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e687d4:::benchmark_data_smithi194_186548_object15425:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e6ffb4:::benchmark_data_smithi194_186548_object36485:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38e8883a:::benchmark_data_smithi194_186548_object2678:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ea7f28:::benchmark_data_smithi194_186548_object73023:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38eab5a9:::benchmark_data_smithi194_186548_object63879:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38eace89:::benchmark_data_smithi194_186548_object62862:head# 2024-04-29T14:53:21.674 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38eb795c:::benchmark_data_smithi194_186548_object49380:head# 2024-04-29T14:53:21.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ec6a27:::benchmark_data_smithi194_186548_object28441:head# 2024-04-29T14:53:21.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38eccec2:::benchmark_data_smithi194_186548_object36240:head# 2024-04-29T14:53:21.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38eced4b:::benchmark_data_smithi194_186548_object33975:head# 2024-04-29T14:53:21.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ed952d:::benchmark_data_smithi194_186548_object51699:head# 2024-04-29T14:53:21.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38edcabd:::benchmark_data_smithi194_186548_object24336:head# 2024-04-29T14:53:21.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38edfd18:::benchmark_data_smithi194_186548_object3201:head# 2024-04-29T14:53:21.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ee449c:::benchmark_data_smithi194_186548_object60669:head# 2024-04-29T14:53:21.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ef4e22:::benchmark_data_smithi194_186548_object22109:head# 2024-04-29T14:53:21.682 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f011cb:::benchmark_data_smithi194_186548_object78127:head# 2024-04-29T14:53:21.682 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f191de:::benchmark_data_smithi194_186548_object33056:head# 2024-04-29T14:53:21.682 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f26617:::benchmark_data_smithi194_186548_object81930:head# 2024-04-29T14:53:21.682 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f4a79d:::benchmark_data_smithi194_186548_object64270:head# 2024-04-29T14:53:21.683 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f5066a:::benchmark_data_smithi194_186548_object17844:head# 2024-04-29T14:53:21.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f53fef:::benchmark_data_smithi194_186548_object16279:head# 2024-04-29T14:53:21.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f657e4:::benchmark_data_smithi194_186548_object85032:head# 2024-04-29T14:53:21.687 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f6ed7c:::benchmark_data_smithi194_186548_object25680:head# 2024-04-29T14:53:21.688 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f72b2d:::benchmark_data_smithi194_186548_object16434:head# 2024-04-29T14:53:21.688 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f8015b:::benchmark_data_smithi194_186548_object52639:head# 2024-04-29T14:53:21.688 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f96737:::benchmark_data_smithi194_186548_object26529:head# 2024-04-29T14:53:21.689 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38f9fefb:::benchmark_data_smithi194_186548_object12236:head# 2024-04-29T14:53:21.689 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fa2219:::benchmark_data_smithi194_186548_object43886:head# 2024-04-29T14:53:21.689 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38faae0b:::benchmark_data_smithi194_186548_object14726:head# 2024-04-29T14:53:21.689 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fb0466:::benchmark_data_smithi194_186548_object68356:head# 2024-04-29T14:53:21.689 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fb35cb:::benchmark_data_smithi194_186548_object57895:head# 2024-04-29T14:53:21.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fb7b2d:::benchmark_data_smithi194_186548_object23754:head# 2024-04-29T14:53:21.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fb83d0:::benchmark_data_smithi194_186548_object48229:head# 2024-04-29T14:53:21.691 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fc4acb:::benchmark_data_smithi194_186548_object77516:head# 2024-04-29T14:53:21.691 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fd1a27:::benchmark_data_smithi194_186548_object10215:head# 2024-04-29T14:53:21.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fda7ff:::benchmark_data_smithi194_186548_object57837:head# 2024-04-29T14:53:21.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fdd4a4:::benchmark_data_smithi194_186548_object59366:head# 2024-04-29T14:53:21.693 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fe389e:::benchmark_data_smithi194_186548_object83245:head# 2024-04-29T14:53:21.693 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38fe538d:::benchmark_data_smithi194_186548_object61822:head# 2024-04-29T14:53:21.693 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ff3b71:::benchmark_data_smithi194_186548_object26388:head# 2024-04-29T14:53:21.694 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ffc25d:::benchmark_data_smithi194_186548_object994:head# 2024-04-29T14:53:21.694 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:38ffe69e:::benchmark_data_smithi194_186548_object40571:head# 2024-04-29T14:53:21.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39011f27:::benchmark_data_smithi194_186548_object76245:head# 2024-04-29T14:53:21.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39012527:::benchmark_data_smithi194_186548_object29409:head# 2024-04-29T14:53:21.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3902003a:::benchmark_data_smithi194_186548_object73405:head# 2024-04-29T14:53:21.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39028290:::benchmark_data_smithi194_186548_object40194:head# 2024-04-29T14:53:21.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3902a9bd:::benchmark_data_smithi194_186548_object73601:head# 2024-04-29T14:53:21.697 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3902d12c:::benchmark_data_smithi194_186548_object68749:head# 2024-04-29T14:53:21.698 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3904a24c:::benchmark_data_smithi194_186548_object9303:head# 2024-04-29T14:53:21.698 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3904f71d:::benchmark_data_smithi194_186548_object5207:head# 2024-04-29T14:53:21.698 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390518b3:::benchmark_data_smithi194_186548_object20502:head# 2024-04-29T14:53:21.698 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390552b3:::benchmark_data_smithi194_186548_object67153:head# 2024-04-29T14:53:21.699 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3905551b:::benchmark_data_smithi194_186548_object68457:head# 2024-04-29T14:53:21.699 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39088bcd:::benchmark_data_smithi194_186548_object71732:head# 2024-04-29T14:53:21.700 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390932c7:::benchmark_data_smithi194_186548_object55228:head# 2024-04-29T14:53:21.700 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390992c5:::benchmark_data_smithi194_186548_object9152:head# 2024-04-29T14:53:21.701 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390b2b21:::benchmark_data_smithi194_186548_object75191:head# 2024-04-29T14:53:21.701 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390b8c72:::benchmark_data_smithi194_186548_object38608:head# 2024-04-29T14:53:21.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390c4e1d:::benchmark_data_smithi194_186548_object13857:head# 2024-04-29T14:53:21.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390d7542:::benchmark_data_smithi194_186548_object53496:head# 2024-04-29T14:53:21.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:390f2520:::benchmark_data_smithi194_186548_object55337:head# 2024-04-29T14:53:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39107268:::benchmark_data_smithi194_186548_object61919:head# 2024-04-29T14:53:21.711 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3910b976:::benchmark_data_smithi194_186548_object20936:head# 2024-04-29T14:53:21.711 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39124137:::benchmark_data_smithi194_186548_object72622:head# 2024-04-29T14:53:21.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39126096:::benchmark_data_smithi194_186548_object9345:head# 2024-04-29T14:53:21.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39128b58:::benchmark_data_smithi194_186548_object70978:head# 2024-04-29T14:53:21.713 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39157b44:::benchmark_data_smithi194_186548_object57305:head# 2024-04-29T14:53:21.713 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391a5403:::benchmark_data_smithi194_186548_object73551:head# 2024-04-29T14:53:21.714 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391ac410:::benchmark_data_smithi194_186548_object57244:head# 2024-04-29T14:53:21.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391b9978:::benchmark_data_smithi194_186548_object12821:head# 2024-04-29T14:53:21.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391baf69:::benchmark_data_smithi194_186548_object64416:head# 2024-04-29T14:53:21.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391d17f5:::benchmark_data_smithi194_186548_object74042:head# 2024-04-29T14:53:21.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391d50e9:::benchmark_data_smithi194_186548_object58239:head# 2024-04-29T14:53:21.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391e452c:::benchmark_data_smithi194_186548_object70305:head# 2024-04-29T14:53:21.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391e5a7d:::benchmark_data_smithi194_186548_object45151:head# 2024-04-29T14:53:21.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391e73fe:::benchmark_data_smithi194_186548_object69617:head# 2024-04-29T14:53:21.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:391f61a8:::benchmark_data_smithi194_186548_object13225:head# 2024-04-29T14:53:21.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39204f76:::benchmark_data_smithi194_186548_object40441:head# 2024-04-29T14:53:21.718 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3920a189:::benchmark_data_smithi194_186548_object9050:head# 2024-04-29T14:53:21.719 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3922e6bd:::benchmark_data_smithi194_186548_object41168:head# 2024-04-29T14:53:21.720 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3924814d:::benchmark_data_smithi194_186548_object61978:head# 2024-04-29T14:53:21.720 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39264078:::benchmark_data_smithi194_186548_object13076:head# 2024-04-29T14:53:21.721 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39285f1b:::benchmark_data_smithi194_186548_object51289:head# 2024-04-29T14:53:21.721 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392893ca:::benchmark_data_smithi194_186548_object66538:head# 2024-04-29T14:53:21.722 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39290cea:::benchmark_data_smithi194_186548_object44850:head# 2024-04-29T14:53:21.722 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39295ff7:::benchmark_data_smithi194_186548_object1170:head# 2024-04-29T14:53:21.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3929fe55:::benchmark_data_smithi194_186548_object15491:head# 2024-04-29T14:53:21.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392a1243:::benchmark_data_smithi194_186548_object75374:head# 2024-04-29T14:53:21.724 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392baf7d:::benchmark_data_smithi194_186548_object53111:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392d07b1:::benchmark_data_smithi194_186548_object52139:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392d08a4:::benchmark_data_smithi194_186548_object79064:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392d1c2a:::benchmark_data_smithi194_186548_object66976:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392e3757:::benchmark_data_smithi194_186548_object16515:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392e9b8b:::benchmark_data_smithi194_186548_object44188:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392f0860:::benchmark_data_smithi194_186548_object67208:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:392f7bb6:::benchmark_data_smithi194_186548_object29185:head# 2024-04-29T14:53:21.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3930d89a:::benchmark_data_smithi194_186548_object19976:head# 2024-04-29T14:53:21.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3930f6e0:::benchmark_data_smithi194_186548_object11155:head# 2024-04-29T14:53:21.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3931e018:::benchmark_data_smithi194_186548_object85114:head# 2024-04-29T14:53:21.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39324982:::benchmark_data_smithi194_186548_object16600:head# 2024-04-29T14:53:21.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393337fd:::benchmark_data_smithi194_186548_object63959:head# 2024-04-29T14:53:21.730 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39341eb2:::benchmark_data_smithi194_186548_object18705:head# 2024-04-29T14:53:21.730 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3934b02a:::benchmark_data_smithi194_186548_object35565:head# 2024-04-29T14:53:21.731 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39350339:::benchmark_data_smithi194_186548_object10769:head# 2024-04-29T14:53:21.731 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3936c173:::benchmark_data_smithi194_186548_object67385:head# 2024-04-29T14:53:21.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39375dc8:::benchmark_data_smithi194_186548_object27170:head# 2024-04-29T14:53:21.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3938c676:::benchmark_data_smithi194_186548_object4192:head# 2024-04-29T14:53:21.732 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39392b76:::benchmark_data_smithi194_186548_object4335:head# 2024-04-29T14:53:21.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39396ab4:::benchmark_data_smithi194_186548_object52416:head# 2024-04-29T14:53:21.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3939ca39:::benchmark_data_smithi194_186548_object75130:head# 2024-04-29T14:53:21.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393ad217:::benchmark_data_smithi194_186548_object31244:head# 2024-04-29T14:53:21.736 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393aef3d:::benchmark_data_smithi194_186548_object78294:head# 2024-04-29T14:53:21.743 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393b02a3:::benchmark_data_smithi194_186548_object14483:head# 2024-04-29T14:53:21.747 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393b453f:::benchmark_data_smithi194_186548_object20363:head# 2024-04-29T14:53:21.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393bb3a3:::benchmark_data_smithi194_186548_object65170:head# 2024-04-29T14:53:21.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393c1822:::benchmark_data_smithi194_186548_object21397:head# 2024-04-29T14:53:21.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393c183e:::benchmark_data_smithi194_186548_object11455:head# 2024-04-29T14:53:21.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393c7352:::benchmark_data_smithi194_186548_object24418:head# 2024-04-29T14:53:21.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393c7ef4:::benchmark_data_smithi194_186548_object50835:head# 2024-04-29T14:53:21.751 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393e4462:::benchmark_data_smithi194_186548_object10282:head# 2024-04-29T14:53:21.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:393f541a:::benchmark_data_smithi194_186548_object67060:head# 2024-04-29T14:53:21.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3940e462:::benchmark_data_smithi194_186548_object20169:head# 2024-04-29T14:53:21.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3940f666:::benchmark_data_smithi194_186548_object66074:head# 2024-04-29T14:53:21.755 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39410690:::benchmark_data_smithi194_186548_object9703:head# 2024-04-29T14:53:21.756 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3941150c:::benchmark_data_smithi194_186548_object50577:head# 2024-04-29T14:53:21.756 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3941568a:::benchmark_data_smithi194_186548_object63389:head# 2024-04-29T14:53:21.757 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3941821f:::benchmark_data_smithi194_186548_object61840:head# 2024-04-29T14:53:21.764 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394344a2:::benchmark_data_smithi194_186548_object24188:head# 2024-04-29T14:53:21.764 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39436a2e:::benchmark_data_smithi194_186548_object36223:head# 2024-04-29T14:53:21.765 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394486f1:::benchmark_data_smithi194_186548_object38595:head# 2024-04-29T14:53:21.765 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3944c0c4:::benchmark_data_smithi194_186548_object80253:head# 2024-04-29T14:53:21.765 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3944c79f:::benchmark_data_smithi194_186548_object23599:head# 2024-04-29T14:53:21.765 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3945088d:::benchmark_data_smithi194_186548_object20225:head# 2024-04-29T14:53:21.770 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3945c028:::benchmark_data_smithi194_186548_object28612:head# 2024-04-29T14:53:21.770 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3945cbd7:::benchmark_data_smithi194_186548_object85335:head# 2024-04-29T14:53:21.771 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394603a4:::benchmark_data_smithi194_186548_object78654:head# 2024-04-29T14:53:21.771 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3947afe4:::benchmark_data_smithi194_186548_object23217:head# 2024-04-29T14:53:21.771 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3947da9e:::benchmark_data_smithi194_186548_object3698:head# 2024-04-29T14:53:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3948567e:::benchmark_data_smithi194_186548_object54401:head# 2024-04-29T14:53:21.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3948a90f:::benchmark_data_smithi194_186548_object43415:head# 2024-04-29T14:53:21.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3948da92:::benchmark_data_smithi194_186548_object40454:head# 2024-04-29T14:53:21.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394a590d:::benchmark_data_smithi194_186548_object49599:head# 2024-04-29T14:53:21.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394a8993:::benchmark_data_smithi194_186548_object81695:head# 2024-04-29T14:53:21.776 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394bd1f1:::benchmark_data_smithi194_186548_object57513:head# 2024-04-29T14:53:21.777 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394c9303:::benchmark_data_smithi194_186548_object62471:head# 2024-04-29T14:53:21.778 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394cf157:::benchmark_data_smithi194_186548_object36108:head# 2024-04-29T14:53:21.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394ddab4:::benchmark_data_smithi194_186548_object43241:head# 2024-04-29T14:53:21.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394e56ee:::benchmark_data_smithi194_186548_object53466:head# 2024-04-29T14:53:21.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394e6a91:::benchmark_data_smithi194_186548_object22921:head# 2024-04-29T14:53:21.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394e84f9:::benchmark_data_smithi194_186548_object57521:head# 2024-04-29T14:53:21.788 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394ea91d:::benchmark_data_smithi194_186548_object18871:head# 2024-04-29T14:53:21.788 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394f3db7:::benchmark_data_smithi194_186548_object66781:head# 2024-04-29T14:53:21.788 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:394f9626:::benchmark_data_smithi194_186548_object28056:head# 2024-04-29T14:53:21.788 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3951d457:::benchmark_data_smithi194_186548_object49575:head# 2024-04-29T14:53:21.790 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:395319ca:::benchmark_data_smithi194_186548_object55223:head# 2024-04-29T14:53:21.792 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:395400cb:::benchmark_data_smithi194_186548_object64512:head# 2024-04-29T14:53:21.792 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39542c69:::benchmark_data_smithi194_186548_object78991:head# 2024-04-29T14:53:21.795 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39544f26:::benchmark_data_smithi194_186548_object27683:head# 2024-04-29T14:53:21.796 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:395531ca:::benchmark_data_smithi194_186548_object57843:head# 2024-04-29T14:53:21.797 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3955a046:::benchmark_data_smithi194_186548_object21888:head# 2024-04-29T14:53:21.798 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3955ee48:::benchmark_data_smithi194_186548_object20850:head# 2024-04-29T14:53:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39563c0c:::benchmark_data_smithi194_186548_object9267:head# 2024-04-29T14:53:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39564d68:::benchmark_data_smithi194_186548_object46735:head# 2024-04-29T14:53:21.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3956e8c2:::benchmark_data_smithi194_186548_object51023:head# 2024-04-29T14:53:21.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3956f0f7:::benchmark_data_smithi194_186548_object47212:head# 2024-04-29T14:53:21.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3958a58a:::benchmark_data_smithi194_186548_object51285:head# 2024-04-29T14:53:21.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3958d362:::benchmark_data_smithi194_186548_object6758:head# 2024-04-29T14:53:21.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39590b3b:::benchmark_data_smithi194_186548_object56517:head# 2024-04-29T14:53:21.807 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:395a6b51:::benchmark_data_smithi194_186548_object22096:head# 2024-04-29T14:53:21.807 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:395ab122:::benchmark_data_smithi194_186548_object69871:head# 2024-04-29T14:53:21.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:395d761c:::benchmark_data_smithi194_186548_object23870:head# 2024-04-29T14:53:21.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:395efc1e:::benchmark_data_smithi194_186548_object51472:head# 2024-04-29T14:53:21.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39606d92:::benchmark_data_smithi194_186548_object48017:head# 2024-04-29T14:53:21.809 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39606fb1:::benchmark_data_smithi194_186548_object60727:head# 2024-04-29T14:53:21.809 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39610924:::benchmark_data_smithi194_186548_object81107:head# 2024-04-29T14:53:21.810 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396177b4:::benchmark_data_smithi194_186548_object35231:head# 2024-04-29T14:53:21.810 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3961cc82:::benchmark_data_smithi194_186548_object55483:head# 2024-04-29T14:53:21.810 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39622044:::benchmark_data_smithi194_186548_object81526:head# 2024-04-29T14:53:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39628b13:::benchmark_data_smithi194_186548_object79793:head# 2024-04-29T14:53:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3962ab5f:::benchmark_data_smithi194_186548_object60421:head# 2024-04-29T14:53:21.812 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39630212:::benchmark_data_smithi194_186548_object24466:head# 2024-04-29T14:53:21.812 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39630aa0:::benchmark_data_smithi194_186548_object32245:head# 2024-04-29T14:53:21.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3963eab4:::benchmark_data_smithi194_186548_object11490:head# 2024-04-29T14:53:21.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396443db:::benchmark_data_smithi194_186548_object11666:head# 2024-04-29T14:53:21.814 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39655323:::benchmark_data_smithi194_186548_object76164:head# 2024-04-29T14:53:21.814 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39659e4b:::benchmark_data_smithi194_186548_object43763:head# 2024-04-29T14:53:21.814 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3965f337:::benchmark_data_smithi194_186548_object68618:head# 2024-04-29T14:53:21.815 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396736c0:::benchmark_data_smithi194_186548_object24475:head# 2024-04-29T14:53:21.817 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39675759:::benchmark_data_smithi194_186548_object4349:head# 2024-04-29T14:53:21.817 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39681af9:::benchmark_data_smithi194_186548_object18308:head# 2024-04-29T14:53:21.818 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3968c666:::benchmark_data_smithi194_186548_object22518:head# 2024-04-29T14:53:21.818 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39693899:::benchmark_data_smithi194_186548_object3412:head# 2024-04-29T14:53:21.819 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3969db90:::benchmark_data_smithi194_186548_object77843:head# 2024-04-29T14:53:21.820 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396aa8ce:::benchmark_data_smithi194_186548_object31849:head# 2024-04-29T14:53:21.820 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396ac0b7:::benchmark_data_smithi194_186548_object77104:head# 2024-04-29T14:53:21.820 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396b5584:::benchmark_data_smithi194_186548_object49928:head# 2024-04-29T14:53:21.821 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396ba0a2:::benchmark_data_smithi194_186548_object54995:head# 2024-04-29T14:53:21.821 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396d2501:::benchmark_data_smithi194_186548_object67998:head# 2024-04-29T14:53:21.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396d7ac8:::benchmark_data_smithi194_186548_object51397:head# 2024-04-29T14:53:21.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396dba54:::benchmark_data_smithi194_186548_object5876:head# 2024-04-29T14:53:21.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396dd58a:::benchmark_data_smithi194_186548_object17513:head# 2024-04-29T14:53:21.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396df3e2:::benchmark_data_smithi194_186548_object60070:head# 2024-04-29T14:53:21.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396ebc1f:::benchmark_data_smithi194_186548_object24924:head# 2024-04-29T14:53:21.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396ee3c4:::benchmark_data_smithi194_186548_object78728:head# 2024-04-29T14:53:21.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396f1cf9:::benchmark_data_smithi194_186548_object2449:head# 2024-04-29T14:53:21.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396fc3b5:::benchmark_data_smithi194_186548_object14187:head# 2024-04-29T14:53:21.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:396fe504:::benchmark_data_smithi194_186548_object81581:head# 2024-04-29T14:53:21.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39706c4b:::benchmark_data_smithi194_186548_object14365:head# 2024-04-29T14:53:21.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39718b86:::benchmark_data_smithi194_186548_object60667:head# 2024-04-29T14:53:21.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:397221e3:::benchmark_data_smithi194_186548_object61025:head# 2024-04-29T14:53:21.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39723231:::benchmark_data_smithi194_186548_object66657:head# 2024-04-29T14:53:21.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3972c3ea:::benchmark_data_smithi194_186548_object56361:head# 2024-04-29T14:53:21.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3972d4c9:::benchmark_data_smithi194_186548_object32487:head# 2024-04-29T14:53:21.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3972f06d:::benchmark_data_smithi194_186548_object1252:head# 2024-04-29T14:53:21.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39735473:::benchmark_data_smithi194_186548_object59417:head# 2024-04-29T14:53:21.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3973d94c:::benchmark_data_smithi194_186548_object52026:head# 2024-04-29T14:53:21.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3973dd14:::benchmark_data_smithi194_186548_object37309:head# 2024-04-29T14:53:21.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39763576:::benchmark_data_smithi194_186548_object80829:head# 2024-04-29T14:53:21.833 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39786771:::benchmark_data_smithi194_186548_object22651:head# 2024-04-29T14:53:21.833 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:397b1e1f:::benchmark_data_smithi194_186548_object67435:head# 2024-04-29T14:53:21.833 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:397ca17a:::benchmark_data_smithi194_186548_object68512:head# 2024-04-29T14:53:21.837 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:397cded8:::benchmark_data_smithi194_186548_object17168:head# 2024-04-29T14:53:21.838 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:397ed627:::benchmark_data_smithi194_186548_object49750:head# 2024-04-29T14:53:21.838 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:397fb871:::benchmark_data_smithi194_186548_object74008:head# 2024-04-29T14:53:21.838 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39806b6d:::benchmark_data_smithi194_186548_object61286:head# 2024-04-29T14:53:21.838 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3981646b:::benchmark_data_smithi194_186548_object48065:head# 2024-04-29T14:53:21.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39824538:::benchmark_data_smithi194_186548_object16738:head# 2024-04-29T14:53:21.844 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39828805:::benchmark_data_smithi194_186548_object13160:head# 2024-04-29T14:53:21.845 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3982b37f:::benchmark_data_smithi194_186548_object60047:head# 2024-04-29T14:53:21.846 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3983aeeb:::benchmark_data_smithi194_186548_object68333:head# 2024-04-29T14:53:21.846 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398487da:::benchmark_data_smithi194_186548_object82926:head# 2024-04-29T14:53:21.846 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39857f5b:::benchmark_data_smithi194_186548_object56638:head# 2024-04-29T14:53:21.847 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3985b4ca:::benchmark_data_smithi194_186548_object79496:head# 2024-04-29T14:53:21.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3986de10:::benchmark_data_smithi194_186548_object52990:head# 2024-04-29T14:53:21.848 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39888b7b:::benchmark_data_smithi194_186548_object28767:head# 2024-04-29T14:53:21.849 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3988f1e3:::benchmark_data_smithi194_186548_object46864:head# 2024-04-29T14:53:21.849 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398a95c9:::benchmark_data_smithi194_186548_object83447:head# 2024-04-29T14:53:21.850 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398a9c67:::benchmark_data_smithi194_186548_object28462:head# 2024-04-29T14:53:21.850 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398aceea:::benchmark_data_smithi194_186548_object46216:head# 2024-04-29T14:53:21.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398b2f79:::benchmark_data_smithi194_186548_object10929:head# 2024-04-29T14:53:21.851 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398b8267:::benchmark_data_smithi194_186548_object9300:head# 2024-04-29T14:53:21.852 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398bc821:::benchmark_data_smithi194_186548_object77764:head# 2024-04-29T14:53:21.852 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398c871f:::benchmark_data_smithi194_186548_object22019:head# 2024-04-29T14:53:21.853 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398dedd6:::benchmark_data_smithi194_186548_object19149:head# 2024-04-29T14:53:21.853 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398e9e8d:::benchmark_data_smithi194_186548_object51883:head# 2024-04-29T14:53:21.854 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398ea7b9:::benchmark_data_smithi194_186548_object27225:head# 2024-04-29T14:53:21.854 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398f0d53:::benchmark_data_smithi194_186548_object69599:head# 2024-04-29T14:53:21.854 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398f3409:::benchmark_data_smithi194_186548_object40249:head# 2024-04-29T14:53:21.855 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:398f6509:::benchmark_data_smithi194_186548_object49802:head# 2024-04-29T14:53:21.855 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39908d60:::benchmark_data_smithi194_186548_object61521:head# 2024-04-29T14:53:21.856 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3990a9be:::benchmark_data_smithi194_186548_object26564:head# 2024-04-29T14:53:21.857 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39910922:::benchmark_data_smithi194_186548_object12563:head# 2024-04-29T14:53:21.857 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39921449:::benchmark_data_smithi194_186548_object78028:head# 2024-04-29T14:53:21.858 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3992414f:::benchmark_data_smithi194_186548_object52467:head# 2024-04-29T14:53:21.858 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3992e4ea:::benchmark_data_smithi194_186548_object67202:head# 2024-04-29T14:53:21.859 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39946d85:::benchmark_data_smithi194_186548_object14241:head# 2024-04-29T14:53:21.859 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3994be9d:::benchmark_data_smithi194_186548_object80398:head# 2024-04-29T14:53:21.859 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3994df4d:::benchmark_data_smithi194_186548_object58775:head# 2024-04-29T14:53:21.860 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3995c8de:::benchmark_data_smithi194_186548_object16055:head# 2024-04-29T14:53:21.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3996f7cc:::benchmark_data_smithi194_186548_object6930:head# 2024-04-29T14:53:21.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399778b5:::benchmark_data_smithi194_186548_object78691:head# 2024-04-29T14:53:21.861 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39985b5b:::benchmark_data_smithi194_186548_object57029:head# 2024-04-29T14:53:21.862 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3998f7e6:::benchmark_data_smithi194_186548_object72867:head# 2024-04-29T14:53:21.862 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39995955:::benchmark_data_smithi194_186548_object80006:head# 2024-04-29T14:53:21.862 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3999932a:::benchmark_data_smithi194_186548_object46574:head# 2024-04-29T14:53:21.863 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399a9174:::benchmark_data_smithi194_186548_object40649:head# 2024-04-29T14:53:21.863 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399b1a81:::benchmark_data_smithi194_186548_object36314:head# 2024-04-29T14:53:21.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399bb253:::benchmark_data_smithi194_186548_object83605:head# 2024-04-29T14:53:21.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399c1d89:::benchmark_data_smithi194_186548_object29845:head# 2024-04-29T14:53:21.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399d4fd1:::benchmark_data_smithi194_186548_object24442:head# 2024-04-29T14:53:21.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399e5ef4:::benchmark_data_smithi194_186548_object40891:head# 2024-04-29T14:53:21.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399efd94:::benchmark_data_smithi194_186548_object64015:head# 2024-04-29T14:53:21.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:399fff18:::benchmark_data_smithi194_186548_object8179:head# 2024-04-29T14:53:21.870 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a046ae:::benchmark_data_smithi194_186548_object12316:head# 2024-04-29T14:53:21.871 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a06da6:::benchmark_data_smithi194_186548_object78313:head# 2024-04-29T14:53:21.873 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a166a9:::benchmark_data_smithi194_186548_object50552:head# 2024-04-29T14:53:21.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a29adf:::benchmark_data_smithi194_186548_object57256:head# 2024-04-29T14:53:21.876 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a2aa84:::benchmark_data_smithi194_186548_object45523:head# 2024-04-29T14:53:21.876 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a2b424:::benchmark_data_smithi194_186548_object80412:head# 2024-04-29T14:53:21.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a2d7d9:::benchmark_data_smithi194_186548_object84123:head# 2024-04-29T14:53:21.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a2f621:::benchmark_data_smithi194_186548_object75884:head# 2024-04-29T14:53:21.877 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a3778d:::benchmark_data_smithi194_186548_object16926:head# 2024-04-29T14:53:21.879 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a4c0f7:::benchmark_data_smithi194_186548_object75579:head# 2024-04-29T14:53:21.879 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a59f7b:::benchmark_data_smithi194_186548_object14316:head# 2024-04-29T14:53:21.879 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a6c9ef:::benchmark_data_smithi194_186548_object20952:head# 2024-04-29T14:53:21.880 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a77b45:::benchmark_data_smithi194_186548_object72262:head# 2024-04-29T14:53:21.880 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39a94420:::benchmark_data_smithi194_186548_object8311:head# 2024-04-29T14:53:21.882 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39aaafb4:::benchmark_data_smithi194_186548_object63015:head# 2024-04-29T14:53:21.882 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39aec591:::benchmark_data_smithi194_186548_object61103:head# 2024-04-29T14:53:21.882 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39aecf3f:::benchmark_data_smithi194_186548_object61130:head# 2024-04-29T14:53:21.883 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39af5e4c:::benchmark_data_smithi194_186548_object45614:head# 2024-04-29T14:53:21.883 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b01875:::benchmark_data_smithi194_186548_object78042:head# 2024-04-29T14:53:21.883 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b097d3:::benchmark_data_smithi194_186548_object15835:head# 2024-04-29T14:53:21.884 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b0b19e:::benchmark_data_smithi194_186548_object46400:head# 2024-04-29T14:53:21.884 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b0f323:::benchmark_data_smithi194_186548_object39348:head# 2024-04-29T14:53:21.885 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b1bdf0:::benchmark_data_smithi194_186548_object2256:head# 2024-04-29T14:53:21.885 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b1d11d:::benchmark_data_smithi194_186548_object32606:head# 2024-04-29T14:53:21.886 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b45833:::benchmark_data_smithi194_186548_object67550:head# 2024-04-29T14:53:21.886 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b4b473:::benchmark_data_smithi194_186548_object68045:head# 2024-04-29T14:53:21.886 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b4bd49:::benchmark_data_smithi194_186548_object48353:head# 2024-04-29T14:53:21.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b5ca8e:::benchmark_data_smithi194_186548_object12294:head# 2024-04-29T14:53:21.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b7952d:::benchmark_data_smithi194_186548_object68271:head# 2024-04-29T14:53:21.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b796ee:::benchmark_data_smithi194_186548_object78817:head# 2024-04-29T14:53:21.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b89b8a:::benchmark_data_smithi194_186548_object49517:head# 2024-04-29T14:53:21.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39b9cc9f:::benchmark_data_smithi194_186548_object44445:head# 2024-04-29T14:53:21.889 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39baec20:::benchmark_data_smithi194_186548_object62433:head# 2024-04-29T14:53:21.891 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39bb5932:::benchmark_data_smithi194_186548_object63615:head# 2024-04-29T14:53:21.892 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39bbcc6e:::benchmark_data_smithi194_186548_object9705:head# 2024-04-29T14:53:21.892 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39be1120:::benchmark_data_smithi194_186548_object76967:head# 2024-04-29T14:53:21.892 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39be65ec:::benchmark_data_smithi194_186548_object35033:head# 2024-04-29T14:53:21.892 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39befc5b:::benchmark_data_smithi194_186548_object66136:head# 2024-04-29T14:53:21.892 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39bf2475:::benchmark_data_smithi194_186548_object75080:head# 2024-04-29T14:53:21.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39bf5b2c:::benchmark_data_smithi194_186548_object54700:head# 2024-04-29T14:53:21.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39bfabe8:::benchmark_data_smithi194_186548_object25591:head# 2024-04-29T14:53:21.894 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39bfbfc5:::benchmark_data_smithi194_186548_object79303:head# 2024-04-29T14:53:21.896 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c21410:::benchmark_data_smithi194_186548_object43617:head# 2024-04-29T14:53:21.896 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c33e57:::benchmark_data_smithi194_186548_object65727:head# 2024-04-29T14:53:21.896 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c408a8:::benchmark_data_smithi194_186548_object76639:head# 2024-04-29T14:53:21.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c44e75:::benchmark_data_smithi194_186548_object39932:head# 2024-04-29T14:53:21.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c4af8b:::benchmark_data_smithi194_186548_object81110:head# 2024-04-29T14:53:21.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c4e1c4:::benchmark_data_smithi194_186548_object51697:head# 2024-04-29T14:53:21.900 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c5877f:::benchmark_data_smithi194_186548_object45240:head# 2024-04-29T14:53:21.900 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c667a7:::benchmark_data_smithi194_186548_object34669:head# 2024-04-29T14:53:21.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c7d4b0:::benchmark_data_smithi194_186548_object49335:head# 2024-04-29T14:53:21.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c8140e:::benchmark_data_smithi194_186548_object2264:head# 2024-04-29T14:53:21.902 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c82a97:::benchmark_data_smithi194_186548_object20121:head# 2024-04-29T14:53:21.903 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c8c90b:::benchmark_data_smithi194_186548_object44171:head# 2024-04-29T14:53:21.903 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39c94188:::benchmark_data_smithi194_186548_object45443:head# 2024-04-29T14:53:21.907 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39ca6a66:::benchmark_data_smithi194_186548_object85159:head# 2024-04-29T14:53:21.907 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cb758d:::benchmark_data_smithi194_186548_object45554:head# 2024-04-29T14:53:21.907 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cb8f8f:::benchmark_data_smithi194_186548_object26287:head# 2024-04-29T14:53:21.910 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cc1be8:::benchmark_data_smithi194_186548_object29306:head# 2024-04-29T14:53:21.910 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cc92a6:::benchmark_data_smithi194_186548_object52422:head# 2024-04-29T14:53:21.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cd490d:::benchmark_data_smithi194_186548_object41266:head# 2024-04-29T14:53:21.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39ce11e7:::benchmark_data_smithi194_186548_object7440:head# 2024-04-29T14:53:21.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cf6813:::benchmark_data_smithi194_186548_object72630:head# 2024-04-29T14:53:21.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cfdd99:::benchmark_data_smithi194_186548_object68675:head# 2024-04-29T14:53:21.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39cff183:::benchmark_data_smithi194_186548_object84589:head# 2024-04-29T14:53:21.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39d01eb5:::benchmark_data_smithi194_186548_object51907:head# 2024-04-29T14:53:21.915 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39d26fce:::benchmark_data_smithi194_186548_object68089:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39d466bf:::benchmark_data_smithi194_186548_object4076:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39d5d1d5:::benchmark_data_smithi194_186548_object18730:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39d638dd:::benchmark_data_smithi194_186548_object11404:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39d666d5:::benchmark_data_smithi194_186548_object70550:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39d7f441:::benchmark_data_smithi194_186548_object48205:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39dc1879:::benchmark_data_smithi194_186548_object62660:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39dd2cb0:::benchmark_data_smithi194_186548_object57022:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39deb020:::benchmark_data_smithi194_186548_object17419:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39df026c:::benchmark_data_smithi194_186548_object4497:head# 2024-04-29T14:53:21.920 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39df6b6c:::benchmark_data_smithi194_186548_object49977:head# 2024-04-29T14:53:21.921 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39dff3e1:::benchmark_data_smithi194_186548_object52669:head# 2024-04-29T14:53:21.921 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e1b7a0:::benchmark_data_smithi194_186548_object2788:head# 2024-04-29T14:53:21.921 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e2d2f5:::benchmark_data_smithi194_186548_object13433:head# 2024-04-29T14:53:21.922 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e32ec2:::benchmark_data_smithi194_186548_object42346:head# 2024-04-29T14:53:21.923 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e34311:::benchmark_data_smithi194_186548_object67440:head# 2024-04-29T14:53:21.923 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e4cdb5:::benchmark_data_smithi194_186548_object49816:head# 2024-04-29T14:53:21.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e5b614:::benchmark_data_smithi194_186548_object9665:head# 2024-04-29T14:53:21.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e6e1dc:::benchmark_data_smithi194_186548_object45843:head# 2024-04-29T14:53:21.926 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e700af:::benchmark_data_smithi194_186548_object77934:head# 2024-04-29T14:53:21.926 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e768bc:::benchmark_data_smithi194_186548_object49846:head# 2024-04-29T14:53:21.926 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e76d2f:::benchmark_data_smithi194_186548_object56423:head# 2024-04-29T14:53:21.927 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e981be:::benchmark_data_smithi194_186548_object40926:head# 2024-04-29T14:53:21.927 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e99dae:::benchmark_data_smithi194_186548_object18483:head# 2024-04-29T14:53:21.928 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39e9f2f7:::benchmark_data_smithi194_186548_object13918:head# 2024-04-29T14:53:21.928 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39eac07c:::benchmark_data_smithi194_186548_object8200:head# 2024-04-29T14:53:21.929 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39eb7057:::benchmark_data_smithi194_186548_object15541:head# 2024-04-29T14:53:21.929 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39ec225e:::benchmark_data_smithi194_186548_object44505:head# 2024-04-29T14:53:21.930 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39eda3b9:::benchmark_data_smithi194_186548_object72364:head# 2024-04-29T14:53:21.930 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39ee3572:::benchmark_data_smithi194_186548_object68237:head# 2024-04-29T14:53:21.932 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39ee47de:::benchmark_data_smithi194_186548_object18259:head# 2024-04-29T14:53:21.933 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39efc5ee:::benchmark_data_smithi194_186548_object20213:head# 2024-04-29T14:53:21.933 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f040bd:::benchmark_data_smithi194_186548_object15690:head# 2024-04-29T14:53:21.933 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f04adb:::benchmark_data_smithi194_186548_object4198:head# 2024-04-29T14:53:21.933 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f0c364:::benchmark_data_smithi194_186548_object54835:head# 2024-04-29T14:53:21.934 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f0d164:::benchmark_data_smithi194_186548_object55596:head# 2024-04-29T14:53:21.934 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f1bdc2:::benchmark_data_smithi194_186548_object42191:head# 2024-04-29T14:53:21.935 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f29aa3:::benchmark_data_smithi194_186548_object73249:head# 2024-04-29T14:53:21.935 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f2a983:::benchmark_data_smithi194_186548_object49499:head# 2024-04-29T14:53:21.937 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f401dc:::benchmark_data_smithi194_186548_object67668:head# 2024-04-29T14:53:21.942 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f47944:::benchmark_data_smithi194_186548_object58555:head# 2024-04-29T14:53:21.942 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f4f2f1:::benchmark_data_smithi194_186548_object79110:head# 2024-04-29T14:53:21.942 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f50ba5:::benchmark_data_smithi194_186548_object69489:head# 2024-04-29T14:53:21.943 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f56ae0:::benchmark_data_smithi194_186548_object70354:head# 2024-04-29T14:53:21.943 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f5984e:::benchmark_data_smithi194_186548_object1835:head# 2024-04-29T14:53:21.943 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f5b604:::benchmark_data_smithi194_186548_object35304:head# 2024-04-29T14:53:21.944 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f6aa32:::benchmark_data_smithi194_186548_object72282:head# 2024-04-29T14:53:21.944 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f6d5c7:::benchmark_data_smithi194_186548_object6538:head# 2024-04-29T14:53:21.944 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f72a60:::benchmark_data_smithi194_186548_object23689:head# 2024-04-29T14:53:21.945 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f76729:::benchmark_data_smithi194_186548_object30477:head# 2024-04-29T14:53:21.946 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f79f48:::benchmark_data_smithi194_186548_object79391:head# 2024-04-29T14:53:21.946 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f7c44d:::benchmark_data_smithi194_186548_object65203:head# 2024-04-29T14:53:21.946 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f94c59:::benchmark_data_smithi194_186548_object58965:head# 2024-04-29T14:53:21.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39f96a7f:::benchmark_data_smithi194_186548_object45469:head# 2024-04-29T14:53:21.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39fa8276:::benchmark_data_smithi194_186548_object78061:head# 2024-04-29T14:53:21.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39fab6b8:::benchmark_data_smithi194_186548_object31618:head# 2024-04-29T14:53:21.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39fbb453:::benchmark_data_smithi194_186548_object26874:head# 2024-04-29T14:53:21.954 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39fcde7f:::benchmark_data_smithi194_186548_object16760:head# 2024-04-29T14:53:21.956 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39fe1a87:::benchmark_data_smithi194_186548_object26103:head# 2024-04-29T14:53:21.963 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39fe4a8d:::benchmark_data_smithi194_186548_object55112:head# 2024-04-29T14:53:21.963 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:39fe9698:::benchmark_data_smithi194_186548_object2049:head# 2024-04-29T14:53:21.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a01a3d8:::benchmark_data_smithi194_186548_object43855:head# 2024-04-29T14:53:21.964 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a01b149:::benchmark_data_smithi194_186548_object76873:head# 2024-04-29T14:53:21.965 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a02751a:::benchmark_data_smithi194_186548_object55062:head# 2024-04-29T14:53:21.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a02e573:::benchmark_data_smithi194_186548_object36474:head# 2024-04-29T14:53:21.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a05857b:::benchmark_data_smithi194_186548_object44744:head# 2024-04-29T14:53:21.967 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a06a2d2:::benchmark_data_smithi194_186548_object42153:head# 2024-04-29T14:53:21.967 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a08b4f8:::benchmark_data_smithi194_186548_object34565:head# 2024-04-29T14:53:21.967 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a08d9c3:::benchmark_data_smithi194_186548_object65493:head# 2024-04-29T14:53:21.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a0baa29:::benchmark_data_smithi194_186548_object31402:head# 2024-04-29T14:53:21.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a0d3300:::benchmark_data_smithi194_186548_object4974:head# 2024-04-29T14:53:21.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a0de93b:::benchmark_data_smithi194_186548_object27581:head# 2024-04-29T14:53:21.969 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a0f352e:::benchmark_data_smithi194_186548_object41876:head# 2024-04-29T14:53:21.969 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1190ee:::benchmark_data_smithi194_186548_object78118:head# 2024-04-29T14:53:21.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a11b19b:::benchmark_data_smithi194_186548_object7030:head# 2024-04-29T14:53:21.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a11daa6:::benchmark_data_smithi194_186548_object83392:head# 2024-04-29T14:53:21.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a12b7c2:::benchmark_data_smithi194_186548_object11739:head# 2024-04-29T14:53:21.971 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a130403:::benchmark_data_smithi194_186548_object69331:head# 2024-04-29T14:53:21.971 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a13067b:::benchmark_data_smithi194_186548_object59002:head# 2024-04-29T14:53:21.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a13697b:::benchmark_data_smithi194_186548_object67226:head# 2024-04-29T14:53:21.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a13d524:::benchmark_data_smithi194_186548_object29235:head# 2024-04-29T14:53:21.973 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a146559:::benchmark_data_smithi194_186548_object70575:head# 2024-04-29T14:53:21.973 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1513f6:::benchmark_data_smithi194_186548_object56104:head# 2024-04-29T14:53:21.974 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a151c68:::benchmark_data_smithi194_186548_object55310:head# 2024-04-29T14:53:21.977 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a15221e:::benchmark_data_smithi194_186548_object42251:head# 2024-04-29T14:53:21.977 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a15319c:::benchmark_data_smithi194_186548_object49673:head# 2024-04-29T14:53:21.977 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a15adce:::benchmark_data_smithi194_186548_object72493:head# 2024-04-29T14:53:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:53:21.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:53:21.980 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a15f5be:::benchmark_data_smithi194_186548_object85025:head# 2024-04-29T14:53:21.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a16ad0e:::benchmark_data_smithi194_186548_object42655:head# 2024-04-29T14:53:21.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a18042c:::benchmark_data_smithi194_186548_object7972:head# 2024-04-29T14:53:21.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a185a25:::benchmark_data_smithi194_186548_object79107:head# 2024-04-29T14:53:21.982 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1895e9:::benchmark_data_smithi194_186548_object50908:head# 2024-04-29T14:53:21.982 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a18aae4:::benchmark_data_smithi194_186548_object33428:head# 2024-04-29T14:53:21.982 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a18cecf:::benchmark_data_smithi194_186548_object22028:head# 2024-04-29T14:53:21.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a19d0ca:::benchmark_data_smithi194_186548_object75614:head# 2024-04-29T14:53:21.986 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1a1d91:::benchmark_data_smithi194_186548_object65477:head# 2024-04-29T14:53:21.986 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1ab6d1:::benchmark_data_smithi194_186548_object35986:head# 2024-04-29T14:53:21.990 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1ad9ec:::benchmark_data_smithi194_186548_object42714:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1d1544:::benchmark_data_smithi194_186548_object247:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1dbbbc:::benchmark_data_smithi194_186548_object27827:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a1fd45a:::benchmark_data_smithi194_186548_object64982:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a208028:::benchmark_data_smithi194_186548_object70431:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a21b86b:::benchmark_data_smithi194_186548_object8230:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a248db0:::benchmark_data_smithi194_186548_object27915:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a249b2f:::benchmark_data_smithi194_186548_object49112:head# 2024-04-29T14:53:21.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a268a17:::benchmark_data_smithi194_186548_object17563:head# 2024-04-29T14:53:21.992 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a268ab5:::benchmark_data_smithi194_186548_object19893:head# 2024-04-29T14:53:21.992 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a276bde:::benchmark_data_smithi194_186548_object8050:head# 2024-04-29T14:53:21.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a28aa40:::benchmark_data_smithi194_186548_object85126:head# 2024-04-29T14:53:21.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a291d49:::benchmark_data_smithi194_186548_object18283:head# 2024-04-29T14:53:21.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a294895:::benchmark_data_smithi194_186548_object42062:head# 2024-04-29T14:53:21.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a2a66a5:::benchmark_data_smithi194_186548_object47041:head# 2024-04-29T14:53:21.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a2bb3f8:::benchmark_data_smithi194_186548_object53188:head# 2024-04-29T14:53:21.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a2c4b0a:::benchmark_data_smithi194_186548_object9945:head# 2024-04-29T14:53:21.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a2d37e5:::benchmark_data_smithi194_186548_object32023:head# 2024-04-29T14:53:21.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a2f1b32:::benchmark_data_smithi194_186548_object64414:head# 2024-04-29T14:53:22.003 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a2f5b57:::benchmark_data_smithi194_186548_object32891:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a2fa645:::benchmark_data_smithi194_186548_object26950:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a302e60:::benchmark_data_smithi194_186548_object48237:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a32f329:::benchmark_data_smithi194_186548_object71534:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a333a1f:::benchmark_data_smithi194_186548_object44321:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3470ca:::benchmark_data_smithi194_186548_object81501:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3592a1:::benchmark_data_smithi194_186548_object11766:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a35b64b:::benchmark_data_smithi194_186548_object48889:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a35b8e4:::benchmark_data_smithi194_186548_object58217:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a36b842:::benchmark_data_smithi194_186548_object58352:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3800bf:::benchmark_data_smithi194_186548_object67148:head# 2024-04-29T14:53:22.004 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a382119:::benchmark_data_smithi194_186548_object41650:head# 2024-04-29T14:53:22.005 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a383643:::benchmark_data_smithi194_186548_object37487:head# 2024-04-29T14:53:22.005 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a399cce:::benchmark_data_smithi194_186548_object6557:head# 2024-04-29T14:53:22.005 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3cb712:::benchmark_data_smithi194_186548_object45055:head# 2024-04-29T14:53:22.005 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3cc09d:::benchmark_data_smithi194_186548_object51842:head# 2024-04-29T14:53:22.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3e59cc:::benchmark_data_smithi194_186548_object34758:head# 2024-04-29T14:53:22.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3e7d3d:::benchmark_data_smithi194_186548_object2324:head# 2024-04-29T14:53:22.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3ec5dc:::benchmark_data_smithi194_186548_object74599:head# 2024-04-29T14:53:22.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3ed037:::benchmark_data_smithi194_186548_object64220:head# 2024-04-29T14:53:22.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3f0f08:::benchmark_data_smithi194_186548_object34244:head# 2024-04-29T14:53:22.013 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3f929e:::benchmark_data_smithi194_186548_object51871:head# 2024-04-29T14:53:22.013 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a3fa9bf:::benchmark_data_smithi194_186548_object28370:head# 2024-04-29T14:53:22.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4168fe:::benchmark_data_smithi194_186548_object81644:head# 2024-04-29T14:53:22.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a41ae91:::benchmark_data_smithi194_186548_object83767:head# 2024-04-29T14:53:22.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a42ea8a:::benchmark_data_smithi194_186548_object9389:head# 2024-04-29T14:53:22.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4404d1:::benchmark_data_smithi194_186548_object45125:head# 2024-04-29T14:53:22.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a446b2f:::benchmark_data_smithi194_186548_object63586:head# 2024-04-29T14:53:22.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a45c7a6:::benchmark_data_smithi194_186548_object21240:head# 2024-04-29T14:53:22.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a47397c:::benchmark_data_smithi194_186548_object25494:head# 2024-04-29T14:53:22.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a48224c:::benchmark_data_smithi194_186548_object8578:head# 2024-04-29T14:53:22.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a489403:::benchmark_data_smithi194_186548_object185:head# 2024-04-29T14:53:22.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4958fc:::benchmark_data_smithi194_186548_object40021:head# 2024-04-29T14:53:22.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4a0cb5:::benchmark_data_smithi194_186548_object37757:head# 2024-04-29T14:53:22.021 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4aa7c4:::benchmark_data_smithi194_186548_object81612:head# 2024-04-29T14:53:22.021 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4bb679:::benchmark_data_smithi194_186548_object65841:head# 2024-04-29T14:53:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4bb8f8:::benchmark_data_smithi194_186548_object6285:head# 2024-04-29T14:53:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4c089a:::benchmark_data_smithi194_186548_object52839:head# 2024-04-29T14:53:22.022 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4c4af3:::benchmark_data_smithi194_186548_object72817:head# 2024-04-29T14:53:22.023 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4d6688:::benchmark_data_smithi194_186548_object36919:head# 2024-04-29T14:53:22.023 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4e17f0:::benchmark_data_smithi194_186548_object71653:head# 2024-04-29T14:53:22.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4f1ab4:::benchmark_data_smithi194_186548_object25252:head# 2024-04-29T14:53:22.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a4fc811:::benchmark_data_smithi194_186548_object78204:head# 2024-04-29T14:53:22.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a53dc77:::benchmark_data_smithi194_186548_object83724:head# 2024-04-29T14:53:22.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a54b88d:::benchmark_data_smithi194_186548_object16175:head# 2024-04-29T14:53:22.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a54f2a1:::benchmark_data_smithi194_186548_object7087:head# 2024-04-29T14:53:22.026 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a55ad46:::benchmark_data_smithi194_186548_object66608:head# 2024-04-29T14:53:22.026 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a56381e:::benchmark_data_smithi194_186548_object38386:head# 2024-04-29T14:53:22.027 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a564b6f:::benchmark_data_smithi194_186548_object60752:head# 2024-04-29T14:53:22.031 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a56c3c2:::benchmark_data_smithi194_186548_object72074:head# 2024-04-29T14:53:22.040 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a571910:::benchmark_data_smithi194_186548_object62165:head# 2024-04-29T14:53:22.040 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a571fce:::benchmark_data_smithi194_186548_object59390:head# 2024-04-29T14:53:22.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a5728f8:::benchmark_data_smithi194_186548_object53404:head# 2024-04-29T14:53:22.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a57b3e5:::benchmark_data_smithi194_186548_object30532:head# 2024-04-29T14:53:22.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a58ebff:::benchmark_data_smithi194_186548_object42485:head# 2024-04-29T14:53:22.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a58f489:::benchmark_data_smithi194_186548_object68944:head# 2024-04-29T14:53:22.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a593bd2:::benchmark_data_smithi194_186548_object27073:head# 2024-04-29T14:53:22.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a59dc3d:::benchmark_data_smithi194_186548_object64886:head# 2024-04-29T14:53:22.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a5ace62:::benchmark_data_smithi194_186548_object25610:head# 2024-04-29T14:53:22.044 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a5b8e74:::benchmark_data_smithi194_186548_object411:head# 2024-04-29T14:53:22.045 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a5bf86b:::benchmark_data_smithi194_186548_object26132:head# 2024-04-29T14:53:22.045 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a5fc945:::benchmark_data_smithi194_186548_object22547:head# 2024-04-29T14:53:22.045 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a60a803:::benchmark_data_smithi194_186548_object67261:head# 2024-04-29T14:53:22.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a60c3dc:::benchmark_data_smithi194_186548_object50958:head# 2024-04-29T14:53:22.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a60ee4f:::benchmark_data_smithi194_186548_object57231:head# 2024-04-29T14:53:22.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a62532d:::benchmark_data_smithi194_186548_object6705:head# 2024-04-29T14:53:22.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a63dac5:::benchmark_data_smithi194_186548_object75268:head# 2024-04-29T14:53:22.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6411d2:::benchmark_data_smithi194_186548_object28846:head# 2024-04-29T14:53:22.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a645c57:::benchmark_data_smithi194_186548_object54661:head# 2024-04-29T14:53:22.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a650fb7:::benchmark_data_smithi194_186548_object37323:head# 2024-04-29T14:53:22.047 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6522dc:::benchmark_data_smithi194_186548_object56121:head# 2024-04-29T14:53:22.047 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a67c229:::benchmark_data_smithi194_186548_object25568:head# 2024-04-29T14:53:22.048 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6851a5:::benchmark_data_smithi194_186548_object48215:head# 2024-04-29T14:53:22.049 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a68b647:::benchmark_data_smithi194_186548_object32709:head# 2024-04-29T14:53:22.049 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6a1564:::benchmark_data_smithi194_186548_object41152:head# 2024-04-29T14:53:22.050 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6ad6e5:::benchmark_data_smithi194_186548_object37121:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6b463e:::benchmark_data_smithi194_186548_object1940:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6b8876:::benchmark_data_smithi194_186548_object79981:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6ced96:::benchmark_data_smithi194_186548_object15518:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a6d150e:::benchmark_data_smithi194_186548_object46653:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a700d1c:::benchmark_data_smithi194_186548_object31930:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7013c8:::benchmark_data_smithi194_186548_object33007:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a708c35:::benchmark_data_smithi194_186548_object40174:head# 2024-04-29T14:53:22.054 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a721c95:::benchmark_data_smithi194_186548_object29585:head# 2024-04-29T14:53:22.055 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a73386f:::benchmark_data_smithi194_186548_object507:head# 2024-04-29T14:53:22.055 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a75f04a:::benchmark_data_smithi194_186548_object744:head# 2024-04-29T14:53:22.055 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7699b3:::benchmark_data_smithi194_186548_object56806:head# 2024-04-29T14:53:22.055 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a773aff:::benchmark_data_smithi194_186548_object17579:head# 2024-04-29T14:53:22.056 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7754a2:::benchmark_data_smithi194_186548_object52977:head# 2024-04-29T14:53:22.056 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a785b3b:::benchmark_data_smithi194_186548_object568:head# 2024-04-29T14:53:22.056 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7b2c8a:::benchmark_data_smithi194_186548_object68780:head# 2024-04-29T14:53:22.057 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7b98b1:::benchmark_data_smithi194_186548_object11835:head# 2024-04-29T14:53:22.058 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7ba85e:::benchmark_data_smithi194_186548_object69336:head# 2024-04-29T14:53:22.060 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7c33a3:::benchmark_data_smithi194_186548_object39241:head# 2024-04-29T14:53:22.063 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7c4ab0:::benchmark_data_smithi194_186548_object83737:head# 2024-04-29T14:53:22.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7c9bd5:::benchmark_data_smithi194_186548_object69801:head# 2024-04-29T14:53:22.069 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7d99ad:::benchmark_data_smithi194_186548_object75058:head# 2024-04-29T14:53:22.069 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7ec590:::benchmark_data_smithi194_186548_object32108:head# 2024-04-29T14:53:22.071 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a7f349b:::benchmark_data_smithi194_186548_object9025:head# 2024-04-29T14:53:22.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a80b733:::benchmark_data_smithi194_186548_object21285:head# 2024-04-29T14:53:22.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8127a4:::benchmark_data_smithi194_186548_object59076:head# 2024-04-29T14:53:22.076 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a820db6:::benchmark_data_smithi194_186548_object28589:head# 2024-04-29T14:53:22.078 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a82584a:::benchmark_data_smithi194_186548_object63353:head# 2024-04-29T14:53:22.078 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8270ae:::benchmark_data_smithi194_186548_object25321:head# 2024-04-29T14:53:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a828538:::benchmark_data_smithi194_186548_object55669:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a884800:::benchmark_data_smithi194_186548_object58481:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8952af:::benchmark_data_smithi194_186548_object604:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a89cbf3:::benchmark_data_smithi194_186548_object3558:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8c07ab:::benchmark_data_smithi194_186548_object17612:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8c3231:::benchmark_data_smithi194_186548_object81358:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8c52fa:::benchmark_data_smithi194_186548_object21767:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8cedca:::benchmark_data_smithi194_186548_object1034:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8d2e92:::benchmark_data_smithi194_186548_object63688:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8d5702:::benchmark_data_smithi194_186548_object68642:head# 2024-04-29T14:53:22.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8db56c:::benchmark_data_smithi194_186548_object37615:head# 2024-04-29T14:53:22.090 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a8e2e57:::benchmark_data_smithi194_186548_object5303:head# 2024-04-29T14:53:22.090 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a90e187:::benchmark_data_smithi194_186548_object11828:head# 2024-04-29T14:53:22.090 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a91340d:::benchmark_data_smithi194_186548_object57331:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a920af5:::benchmark_data_smithi194_186548_object855:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9216be:::benchmark_data_smithi194_186548_object54638:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a92f006:::benchmark_data_smithi194_186548_object19109:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a93c46c:::benchmark_data_smithi194_186548_object29279:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a940b5e:::benchmark_data_smithi194_186548_object56742:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a94a50d:::benchmark_data_smithi194_186548_object14123:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a960821:::benchmark_data_smithi194_186548_object66456:head# 2024-04-29T14:53:22.095 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a96a785:::benchmark_data_smithi194_186548_object72120:head# 2024-04-29T14:53:22.097 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a96baf7:::benchmark_data_smithi194_186548_object38544:head# 2024-04-29T14:53:22.097 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a97f184:::benchmark_data_smithi194_186548_object44151:head# 2024-04-29T14:53:22.097 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9933a8:::benchmark_data_smithi194_186548_object11202:head# 2024-04-29T14:53:22.097 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a99ff52:::benchmark_data_smithi194_186548_object57990:head# 2024-04-29T14:53:22.098 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9ae5f2:::benchmark_data_smithi194_186548_object20040:head# 2024-04-29T14:53:22.098 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9b8977:::benchmark_data_smithi194_186548_object12532:head# 2024-04-29T14:53:22.099 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9e75b6:::benchmark_data_smithi194_186548_object33623:head# 2024-04-29T14:53:22.099 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9e96ec:::benchmark_data_smithi194_186548_object49687:head# 2024-04-29T14:53:22.099 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9f3a19:::benchmark_data_smithi194_186548_object7737:head# 2024-04-29T14:53:22.100 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3a9ff2c4:::benchmark_data_smithi194_186548_object24769:head# 2024-04-29T14:53:22.100 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa150da:::benchmark_data_smithi194_186548_object21264:head# 2024-04-29T14:53:22.101 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa1c953:::benchmark_data_smithi194_186548_object7409:head# 2024-04-29T14:53:22.103 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa358fc:::benchmark_data_smithi194_186548_object56630:head# 2024-04-29T14:53:22.103 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa3f301:::benchmark_data_smithi194_186548_object33076:head# 2024-04-29T14:53:22.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa54f97:::benchmark_data_smithi194_186548_object82102:head# 2024-04-29T14:53:22.107 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa5eee5:::benchmark_data_smithi194_186548_object5204:head# 2024-04-29T14:53:22.107 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa6b251:::benchmark_data_smithi194_186548_object56849:head# 2024-04-29T14:53:22.108 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa70488:::benchmark_data_smithi194_186548_object20860:head# 2024-04-29T14:53:22.108 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa72a49:::benchmark_data_smithi194_186548_object58516:head# 2024-04-29T14:53:22.109 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa7feec:::benchmark_data_smithi194_186548_object27747:head# 2024-04-29T14:53:22.109 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa86232:::benchmark_data_smithi194_186548_object12182:head# 2024-04-29T14:53:22.110 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aa9003e:::benchmark_data_smithi194_186548_object56476:head# 2024-04-29T14:53:22.110 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aaa2273:::benchmark_data_smithi194_186548_object16969:head# 2024-04-29T14:53:22.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aaa5d90:::benchmark_data_smithi194_186548_object36512:head# 2024-04-29T14:53:22.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aaae485:::benchmark_data_smithi194_186548_object14446:head# 2024-04-29T14:53:22.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aac1b6d:::benchmark_data_smithi194_186548_object30554:head# 2024-04-29T14:53:22.112 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aad36c0:::benchmark_data_smithi194_186548_object71374:head# 2024-04-29T14:53:22.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aadf44c:::benchmark_data_smithi194_186548_object30455:head# 2024-04-29T14:53:22.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aae7b0c:::benchmark_data_smithi194_186548_object51315:head# 2024-04-29T14:53:22.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aaf3594:::benchmark_data_smithi194_186548_object60157:head# 2024-04-29T14:53:22.114 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab0e886:::benchmark_data_smithi194_186548_object43699:head# 2024-04-29T14:53:22.114 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab30b48:::benchmark_data_smithi194_186548_object14025:head# 2024-04-29T14:53:22.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab3ac63:::benchmark_data_smithi194_186548_object27562:head# 2024-04-29T14:53:22.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab4cb95:::benchmark_data_smithi194_186548_object78677:head# 2024-04-29T14:53:22.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab6faf4:::benchmark_data_smithi194_186548_object16996:head# 2024-04-29T14:53:22.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab74a07:::benchmark_data_smithi194_186548_object66338:head# 2024-04-29T14:53:22.117 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab7fba1:::benchmark_data_smithi194_186548_object43735:head# 2024-04-29T14:53:22.117 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ab95f99:::benchmark_data_smithi194_186548_object22611:head# 2024-04-29T14:53:22.118 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aba84a4:::benchmark_data_smithi194_186548_object30993:head# 2024-04-29T14:53:22.118 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abaccdf:::benchmark_data_smithi194_186548_object38239:head# 2024-04-29T14:53:22.119 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abb9b38:::benchmark_data_smithi194_186548_object45808:head# 2024-04-29T14:53:22.121 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abc1c2c:::benchmark_data_smithi194_186548_object34946:head# 2024-04-29T14:53:22.121 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abc72bd:::benchmark_data_smithi194_186548_object26275:head# 2024-04-29T14:53:22.122 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abc7451:::benchmark_data_smithi194_186548_object80244:head# 2024-04-29T14:53:22.122 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abdd4e0:::benchmark_data_smithi194_186548_object62247:head# 2024-04-29T14:53:22.122 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abf3f92:::benchmark_data_smithi194_186548_object85124:head# 2024-04-29T14:53:22.122 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3abf74c9:::benchmark_data_smithi194_186548_object36643:head# 2024-04-29T14:53:22.122 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac069dc:::benchmark_data_smithi194_186548_object4437:head# 2024-04-29T14:53:22.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac0d537:::benchmark_data_smithi194_186548_object54423:head# 2024-04-29T14:53:22.123 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac17741:::benchmark_data_smithi194_186548_object48707:head# 2024-04-29T14:53:22.124 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac2af89:::benchmark_data_smithi194_186548_object33205:head# 2024-04-29T14:53:22.124 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac355a3:::benchmark_data_smithi194_186548_object72732:head# 2024-04-29T14:53:22.125 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac482ab:::benchmark_data_smithi194_186548_object73156:head# 2024-04-29T14:53:22.125 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac6c49c:::benchmark_data_smithi194_186548_object35453:head# 2024-04-29T14:53:22.125 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac756f5:::benchmark_data_smithi194_186548_object42134:head# 2024-04-29T14:53:22.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac7a5f8:::benchmark_data_smithi194_186548_object82321:head# 2024-04-29T14:53:22.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac8e77c:::benchmark_data_smithi194_186548_object18525:head# 2024-04-29T14:53:22.127 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ac944e2:::benchmark_data_smithi194_186548_object1457:head# 2024-04-29T14:53:22.128 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3accb109:::benchmark_data_smithi194_186548_object41292:head# 2024-04-29T14:53:22.128 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3acceeb7:::benchmark_data_smithi194_186548_object22749:head# 2024-04-29T14:53:22.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3acd0b3a:::benchmark_data_smithi194_186548_object47696:head# 2024-04-29T14:53:22.129 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3acd1e03:::benchmark_data_smithi194_186548_object8587:head# 2024-04-29T14:53:22.130 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ace5df3:::benchmark_data_smithi194_186548_object29242:head# 2024-04-29T14:53:22.130 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad1b399:::benchmark_data_smithi194_186548_object20891:head# 2024-04-29T14:53:22.130 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad200d0:::benchmark_data_smithi194_186548_object47842:head# 2024-04-29T14:53:22.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad2f3d7:::benchmark_data_smithi194_186548_object67855:head# 2024-04-29T14:53:22.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad5a1fa:::benchmark_data_smithi194_186548_object9993:head# 2024-04-29T14:53:22.132 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad5a487:::benchmark_data_smithi194_186548_object9646:head# 2024-04-29T14:53:22.132 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad5ef06:::benchmark_data_smithi194_186548_object77304:head# 2024-04-29T14:53:22.133 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad6b312:::benchmark_data_smithi194_186548_object32376:head# 2024-04-29T14:53:22.133 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad6f103:::benchmark_data_smithi194_186548_object41493:head# 2024-04-29T14:53:22.134 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad77957:::benchmark_data_smithi194_186548_object73080:head# 2024-04-29T14:53:22.134 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad97e4a:::benchmark_data_smithi194_186548_object14508:head# 2024-04-29T14:53:22.134 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ad9a4f9:::benchmark_data_smithi194_186548_object53733:head# 2024-04-29T14:53:22.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3adafa2b:::benchmark_data_smithi194_186548_object70087:head# 2024-04-29T14:53:22.136 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3adb5553:::benchmark_data_smithi194_186548_object3295:head# 2024-04-29T14:53:22.136 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3adc0075:::benchmark_data_smithi194_186548_object33371:head# 2024-04-29T14:53:22.136 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3adf1985:::benchmark_data_smithi194_186548_object28404:head# 2024-04-29T14:53:22.137 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3adf5e05:::benchmark_data_smithi194_186548_object69924:head# 2024-04-29T14:53:22.137 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae06bcc:::benchmark_data_smithi194_186548_object82957:head# 2024-04-29T14:53:22.137 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae16511:::benchmark_data_smithi194_186548_object2451:head# 2024-04-29T14:53:22.138 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae2178e:::benchmark_data_smithi194_186548_object39387:head# 2024-04-29T14:53:22.138 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae313d4:::benchmark_data_smithi194_186548_object20742:head# 2024-04-29T14:53:22.138 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae32133:::benchmark_data_smithi194_186548_object47280:head# 2024-04-29T14:53:22.139 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae458ef:::benchmark_data_smithi194_186548_object43248:head# 2024-04-29T14:53:22.139 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae47fc6:::benchmark_data_smithi194_186548_object66972:head# 2024-04-29T14:53:22.140 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae55de6:::benchmark_data_smithi194_186548_object25208:head# 2024-04-29T14:53:22.140 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae5e11a:::benchmark_data_smithi194_186548_object10643:head# 2024-04-29T14:53:22.141 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae66018:::benchmark_data_smithi194_186548_object5741:head# 2024-04-29T14:53:22.141 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae901af:::benchmark_data_smithi194_186548_object24359:head# 2024-04-29T14:53:22.142 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae90800:::benchmark_data_smithi194_186548_object22017:head# 2024-04-29T14:53:22.142 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae96322:::benchmark_data_smithi194_186548_object68886:head# 2024-04-29T14:53:22.142 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae9897a:::benchmark_data_smithi194_186548_object34812:head# 2024-04-29T14:53:22.143 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ae9bf85:::benchmark_data_smithi194_186548_object7656:head# 2024-04-29T14:53:22.143 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aec5f9a:::benchmark_data_smithi194_186548_object49981:head# 2024-04-29T14:53:22.144 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aee672b:::benchmark_data_smithi194_186548_object66171:head# 2024-04-29T14:53:22.144 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3aeeaec0:::benchmark_data_smithi194_186548_object18908:head# 2024-04-29T14:53:22.145 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af0121c:::benchmark_data_smithi194_186548_object26837:head# 2024-04-29T14:53:22.145 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af152b1:::benchmark_data_smithi194_186548_object38753:head# 2024-04-29T14:53:22.146 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af1a643:::benchmark_data_smithi194_186548_object61985:head# 2024-04-29T14:53:22.146 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af2018d:::benchmark_data_smithi194_186548_object56877:head# 2024-04-29T14:53:22.147 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af2e59a:::benchmark_data_smithi194_186548_object44557:head# 2024-04-29T14:53:22.147 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af30b7f:::benchmark_data_smithi194_186548_object62910:head# 2024-04-29T14:53:22.148 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af3e091:::benchmark_data_smithi194_186548_object61092:head# 2024-04-29T14:53:22.148 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af42210:::benchmark_data_smithi194_186548_object50418:head# 2024-04-29T14:53:22.148 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af52f52:::benchmark_data_smithi194_186548_object46026:head# 2024-04-29T14:53:22.149 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af6d385:::benchmark_data_smithi194_186548_object39080:head# 2024-04-29T14:53:22.149 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af6ee83:::benchmark_data_smithi194_186548_object41222:head# 2024-04-29T14:53:22.150 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3af74b1a:::benchmark_data_smithi194_186548_object7022:head# 2024-04-29T14:53:22.150 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3afa80f6:::benchmark_data_smithi194_186548_object83252:head# 2024-04-29T14:53:22.150 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3afa9c10:::benchmark_data_smithi194_186548_object183:head# 2024-04-29T14:53:22.151 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3afc2b92:::benchmark_data_smithi194_186548_object74317:head# 2024-04-29T14:53:22.151 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3afd16b6:::benchmark_data_smithi194_186548_object65764:head# 2024-04-29T14:53:22.152 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3afe7cc2:::benchmark_data_smithi194_186548_object43342:head# 2024-04-29T14:53:22.152 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b015b70:::benchmark_data_smithi194_186548_object21346:head# 2024-04-29T14:53:22.152 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b029371:::benchmark_data_smithi194_186548_object26123:head# 2024-04-29T14:53:22.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b02de22:::benchmark_data_smithi194_186548_object48753:head# 2024-04-29T14:53:22.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b03fdae:::benchmark_data_smithi194_186548_object6345:head# 2024-04-29T14:53:22.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b053a5f:::benchmark_data_smithi194_186548_object34085:head# 2024-04-29T14:53:22.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b05bc0a:::benchmark_data_smithi194_186548_object51280:head# 2024-04-29T14:53:22.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b06d272:::benchmark_data_smithi194_186548_object32268:head# 2024-04-29T14:53:22.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b07eb65:::benchmark_data_smithi194_186548_object82066:head# 2024-04-29T14:53:22.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b08521d:::benchmark_data_smithi194_186548_object60967:head# 2024-04-29T14:53:22.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0a3561:::benchmark_data_smithi194_186548_object29539:head# 2024-04-29T14:53:22.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0a3fb2:::benchmark_data_smithi194_186548_object71860:head# 2024-04-29T14:53:22.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0a4751:::benchmark_data_smithi194_186548_object8015:head# 2024-04-29T14:53:22.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0ae2e7:::benchmark_data_smithi194_186548_object43877:head# 2024-04-29T14:53:22.159 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0b780b:::benchmark_data_smithi194_186548_object42381:head# 2024-04-29T14:53:22.159 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0d61c6:::benchmark_data_smithi194_186548_object41315:head# 2024-04-29T14:53:22.159 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0e5d0a:::benchmark_data_smithi194_186548_object8697:head# 2024-04-29T14:53:22.159 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b0e7f4c:::benchmark_data_smithi194_186548_object31445:head# 2024-04-29T14:53:22.160 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1070f4:::benchmark_data_smithi194_186548_object9532:head# 2024-04-29T14:53:22.160 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b10ee5a:::benchmark_data_smithi194_186548_object6627:head# 2024-04-29T14:53:22.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b117ade:::benchmark_data_smithi194_186548_object16384:head# 2024-04-29T14:53:22.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1227f0:::benchmark_data_smithi194_186548_object14692:head# 2024-04-29T14:53:22.164 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1257c2:::benchmark_data_smithi194_186548_object21637:head# 2024-04-29T14:53:22.164 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b125be0:::benchmark_data_smithi194_186548_object70571:head# 2024-04-29T14:53:22.164 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b129e06:::benchmark_data_smithi194_186548_object64271:head# 2024-04-29T14:53:22.164 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b12e08e:::benchmark_data_smithi194_186548_object59035:head# 2024-04-29T14:53:22.165 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b13b2da:::benchmark_data_smithi194_186548_object46583:head# 2024-04-29T14:53:22.165 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b14416f:::benchmark_data_smithi194_186548_object28716:head# 2024-04-29T14:53:22.165 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b146fea:::benchmark_data_smithi194_186548_object57713:head# 2024-04-29T14:53:22.166 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b151765:::benchmark_data_smithi194_186548_object64097:head# 2024-04-29T14:53:22.166 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b19d5da:::benchmark_data_smithi194_186548_object40531:head# 2024-04-29T14:53:22.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b19df4c:::benchmark_data_smithi194_186548_object42879:head# 2024-04-29T14:53:22.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1a7ef9:::benchmark_data_smithi194_186548_object78266:head# 2024-04-29T14:53:22.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1c0b39:::benchmark_data_smithi194_186548_object6422:head# 2024-04-29T14:53:22.168 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1c73dc:::benchmark_data_smithi194_186548_object34254:head# 2024-04-29T14:53:22.168 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1d5644:::benchmark_data_smithi194_186548_object21395:head# 2024-04-29T14:53:22.170 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1e0d8d:::benchmark_data_smithi194_186548_object18777:head# 2024-04-29T14:53:22.170 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b1fbf6b:::benchmark_data_smithi194_186548_object79629:head# 2024-04-29T14:53:22.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b225254:::benchmark_data_smithi194_186548_object51341:head# 2024-04-29T14:53:22.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b225955:::benchmark_data_smithi194_186548_object71161:head# 2024-04-29T14:53:22.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b22f5a7:::benchmark_data_smithi194_186548_object54097:head# 2024-04-29T14:53:22.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b231dc5:::benchmark_data_smithi194_186548_object34762:head# 2024-04-29T14:53:22.171 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b244d4c:::benchmark_data_smithi194_186548_object45901:head# 2024-04-29T14:53:22.172 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2461d4:::benchmark_data_smithi194_186548_object56219:head# 2024-04-29T14:53:22.172 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b249b37:::benchmark_data_smithi194_186548_object30051:head# 2024-04-29T14:53:22.173 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b25dd27:::benchmark_data_smithi194_186548_object15889:head# 2024-04-29T14:53:22.173 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b25f991:::benchmark_data_smithi194_186548_object19044:head# 2024-04-29T14:53:22.173 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b266b3b:::benchmark_data_smithi194_186548_object61747:head# 2024-04-29T14:53:22.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b268c66:::benchmark_data_smithi194_186548_object30813:head# 2024-04-29T14:53:22.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b272461:::benchmark_data_smithi194_186548_object79763:head# 2024-04-29T14:53:22.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b274ad8:::benchmark_data_smithi194_186548_object5933:head# 2024-04-29T14:53:22.175 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b27995e:::benchmark_data_smithi194_186548_object8368:head# 2024-04-29T14:53:22.175 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b281e26:::benchmark_data_smithi194_186548_object11474:head# 2024-04-29T14:53:22.176 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b285ad2:::benchmark_data_smithi194_186548_object77993:head# 2024-04-29T14:53:22.176 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b29507d:::benchmark_data_smithi194_186548_object60205:head# 2024-04-29T14:53:22.176 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2a2ddb:::benchmark_data_smithi194_186548_object75490:head# 2024-04-29T14:53:22.177 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2a4ab5:::benchmark_data_smithi194_186548_object23873:head# 2024-04-29T14:53:22.177 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2a6358:::benchmark_data_smithi194_186548_object6324:head# 2024-04-29T14:53:22.178 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2a8445:::benchmark_data_smithi194_186548_object46650:head# 2024-04-29T14:53:22.178 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2ba408:::benchmark_data_smithi194_186548_object36620:head# 2024-04-29T14:53:22.178 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2ba76b:::benchmark_data_smithi194_186548_object77729:head# 2024-04-29T14:53:22.179 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2c6d32:::benchmark_data_smithi194_186548_object83600:head# 2024-04-29T14:53:22.179 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2d1dda:::benchmark_data_smithi194_186548_object75515:head# 2024-04-29T14:53:22.179 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b2e52a6:::benchmark_data_smithi194_186548_object75932:head# 2024-04-29T14:53:22.180 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b316160:::benchmark_data_smithi194_186548_object26282:head# 2024-04-29T14:53:22.180 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b31dae6:::benchmark_data_smithi194_186548_object39981:head# 2024-04-29T14:53:22.181 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b327a9d:::benchmark_data_smithi194_186548_object45779:head# 2024-04-29T14:53:22.181 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b330a20:::benchmark_data_smithi194_186548_object6949:head# 2024-04-29T14:53:22.182 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b346481:::benchmark_data_smithi194_186548_object39431:head# 2024-04-29T14:53:22.182 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b381f60:::benchmark_data_smithi194_186548_object82574:head# 2024-04-29T14:53:22.182 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b39a055:::benchmark_data_smithi194_186548_object48936:head# 2024-04-29T14:53:22.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3b795c:::benchmark_data_smithi194_186548_object66804:head# 2024-04-29T14:53:22.183 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3cb301:::benchmark_data_smithi194_186548_object53193:head# 2024-04-29T14:53:22.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3d5815:::benchmark_data_smithi194_186548_object72659:head# 2024-04-29T14:53:22.184 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3e0813:::benchmark_data_smithi194_186548_object8054:head# 2024-04-29T14:53:22.185 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3e0b78:::benchmark_data_smithi194_186548_object43193:head# 2024-04-29T14:53:22.185 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3ecb88:::benchmark_data_smithi194_186548_object50432:head# 2024-04-29T14:53:22.186 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3ef3a6:::benchmark_data_smithi194_186548_object11281:head# 2024-04-29T14:53:22.186 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3f0140:::benchmark_data_smithi194_186548_object47420:head# 2024-04-29T14:53:22.186 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b3f330b:::benchmark_data_smithi194_186548_object56305:head# 2024-04-29T14:53:22.187 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b410e6e:::benchmark_data_smithi194_186548_object53911:head# 2024-04-29T14:53:22.187 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b416a46:::benchmark_data_smithi194_186548_object16180:head# 2024-04-29T14:53:22.191 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b432979:::benchmark_data_smithi194_186548_object34322:head# 2024-04-29T14:53:22.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b436efe:::benchmark_data_smithi194_186548_object19066:head# 2024-04-29T14:53:22.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b444848:::benchmark_data_smithi194_186548_object18837:head# 2024-04-29T14:53:22.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b449c0b:::benchmark_data_smithi194_186548_object26620:head# 2024-04-29T14:53:22.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b46a1d0:::benchmark_data_smithi194_186548_object3595:head# 2024-04-29T14:53:22.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b46b6df:::benchmark_data_smithi194_186548_object40783:head# 2024-04-29T14:53:22.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b48c7ba:::benchmark_data_smithi194_186548_object35277:head# 2024-04-29T14:53:22.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b49a7d7:::benchmark_data_smithi194_186548_object28488:head# 2024-04-29T14:53:22.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4a16e1:::benchmark_data_smithi194_186548_object83393:head# 2024-04-29T14:53:22.197 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4a30fa:::benchmark_data_smithi194_186548_object63662:head# 2024-04-29T14:53:22.197 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4a6127:::benchmark_data_smithi194_186548_object31981:head# 2024-04-29T14:53:22.198 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4b0f45:::benchmark_data_smithi194_186548_object56987:head# 2024-04-29T14:53:22.198 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4bfe0b:::benchmark_data_smithi194_186548_object29293:head# 2024-04-29T14:53:22.199 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4c65af:::benchmark_data_smithi194_186548_object62370:head# 2024-04-29T14:53:22.199 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4ccfa4:::benchmark_data_smithi194_186548_object24395:head# 2024-04-29T14:53:22.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4cd86e:::benchmark_data_smithi194_186548_object38493:head# 2024-04-29T14:53:22.203 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4d528b:::benchmark_data_smithi194_186548_object84826:head# 2024-04-29T14:53:22.204 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4d6ba5:::benchmark_data_smithi194_186548_object5050:head# 2024-04-29T14:53:22.207 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4dd17d:::benchmark_data_smithi194_186548_object15903:head# 2024-04-29T14:53:22.207 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4df418:::benchmark_data_smithi194_186548_object22022:head# 2024-04-29T14:53:22.207 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4e1560:::benchmark_data_smithi194_186548_object32822:head# 2024-04-29T14:53:22.208 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b4ea4b0:::benchmark_data_smithi194_186548_object39457:head# 2024-04-29T14:53:22.209 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b519a93:::benchmark_data_smithi194_186548_object51642:head# 2024-04-29T14:53:22.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b522121:::benchmark_data_smithi194_186548_object15472:head# 2024-04-29T14:53:22.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b522d0e:::benchmark_data_smithi194_186548_object24962:head# 2024-04-29T14:53:22.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b532b7c:::benchmark_data_smithi194_186548_object4308:head# 2024-04-29T14:53:22.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b53a96f:::benchmark_data_smithi194_186548_object78237:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b53add1:::benchmark_data_smithi194_186548_object71200:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b53b07c:::benchmark_data_smithi194_186548_object33369:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b546ce8:::benchmark_data_smithi194_186548_object44618:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b54f88a:::benchmark_data_smithi194_186548_object46414:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b553ec3:::benchmark_data_smithi194_186548_object50991:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b560684:::benchmark_data_smithi194_186548_object37872:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5677fb:::benchmark_data_smithi194_186548_object58725:head# 2024-04-29T14:53:22.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b56e235:::benchmark_data_smithi194_186548_object51251:head# 2024-04-29T14:53:22.216 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b570b81:::benchmark_data_smithi194_186548_object72523:head# 2024-04-29T14:53:22.216 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b57e25d:::benchmark_data_smithi194_186548_object11491:head# 2024-04-29T14:53:22.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b587337:::benchmark_data_smithi194_186548_object70893:head# 2024-04-29T14:53:22.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b58b93b:::benchmark_data_smithi194_186548_object25597:head# 2024-04-29T14:53:22.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b591cbc:::benchmark_data_smithi194_186548_object19398:head# 2024-04-29T14:53:22.218 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b59a997:::benchmark_data_smithi194_186548_object56432:head# 2024-04-29T14:53:22.218 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5a64e5:::benchmark_data_smithi194_186548_object20093:head# 2024-04-29T14:53:22.219 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5a8c79:::benchmark_data_smithi194_186548_object16800:head# 2024-04-29T14:53:22.219 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5b0413:::benchmark_data_smithi194_186548_object60583:head# 2024-04-29T14:53:22.220 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5b58ba:::benchmark_data_smithi194_186548_object29236:head# 2024-04-29T14:53:22.220 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5bacf9:::benchmark_data_smithi194_186548_object7225:head# 2024-04-29T14:53:22.221 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5d1ea8:::benchmark_data_smithi194_186548_object51030:head# 2024-04-29T14:53:22.221 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5d93c8:::benchmark_data_smithi194_186548_object68337:head# 2024-04-29T14:53:22.225 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5e5b72:::benchmark_data_smithi194_186548_object27110:head# 2024-04-29T14:53:22.226 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5f5c55:::benchmark_data_smithi194_186548_object36272:head# 2024-04-29T14:53:22.226 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5fcb51:::benchmark_data_smithi194_186548_object84524:head# 2024-04-29T14:53:22.226 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b5feeaf:::benchmark_data_smithi194_186548_object56279:head# 2024-04-29T14:53:22.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b629389:::benchmark_data_smithi194_186548_object31879:head# 2024-04-29T14:53:22.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b62b34f:::benchmark_data_smithi194_186548_object48441:head# 2024-04-29T14:53:22.229 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b633233:::benchmark_data_smithi194_186548_object82573:head# 2024-04-29T14:53:22.229 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b633920:::benchmark_data_smithi194_186548_object8785:head# 2024-04-29T14:53:22.232 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b63c28f:::benchmark_data_smithi194_186548_object57490:head# 2024-04-29T14:53:22.232 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b644286:::benchmark_data_smithi194_186548_object12530:head# 2024-04-29T14:53:22.232 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b64b350:::benchmark_data_smithi194_186548_object16210:head# 2024-04-29T14:53:22.232 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b65c386:::benchmark_data_smithi194_186548_object34304:head# 2024-04-29T14:53:22.232 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b663894:::benchmark_data_smithi194_186548_object59837:head# 2024-04-29T14:53:22.232 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b66f152:::benchmark_data_smithi194_186548_object13658:head# 2024-04-29T14:53:22.234 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b67b2e9:::benchmark_data_smithi194_186548_object62643:head# 2024-04-29T14:53:22.234 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b681183:::benchmark_data_smithi194_186548_object32504:head# 2024-04-29T14:53:22.235 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6928cd:::benchmark_data_smithi194_186548_object25812:head# 2024-04-29T14:53:22.240 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b69767c:::benchmark_data_smithi194_186548_object49073:head# 2024-04-29T14:53:22.240 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b69a29e:::benchmark_data_smithi194_186548_object56124:head# 2024-04-29T14:53:22.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6a8d67:::benchmark_data_smithi194_186548_object73375:head# 2024-04-29T14:53:22.242 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6d0f5e:::benchmark_data_smithi194_186548_object25417:head# 2024-04-29T14:53:22.242 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6f0d07:::benchmark_data_smithi194_186548_object46293:head# 2024-04-29T14:53:22.243 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6f0e7b:::benchmark_data_smithi194_186548_object51443:head# 2024-04-29T14:53:22.243 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6f7bc1:::benchmark_data_smithi194_186548_object1959:head# 2024-04-29T14:53:22.244 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6f9eb3:::benchmark_data_smithi194_186548_object53667:head# 2024-04-29T14:53:22.244 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6fe1cf:::benchmark_data_smithi194_186548_object71261:head# 2024-04-29T14:53:22.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b6fe569:::benchmark_data_smithi194_186548_object41274:head# 2024-04-29T14:53:22.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b707c9c:::benchmark_data_smithi194_186548_object81389:head# 2024-04-29T14:53:22.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b70aa34:::benchmark_data_smithi194_186548_object69174:head# 2024-04-29T14:53:22.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b716ddc:::benchmark_data_smithi194_186548_object83820:head# 2024-04-29T14:53:22.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b717f4c:::benchmark_data_smithi194_186548_object31181:head# 2024-04-29T14:53:22.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b727665:::benchmark_data_smithi194_186548_object8679:head# 2024-04-29T14:53:22.253 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b72796e:::benchmark_data_smithi194_186548_object83127:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b72f6b4:::benchmark_data_smithi194_186548_object46883:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7309dc:::benchmark_data_smithi194_186548_object40185:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b752ccb:::benchmark_data_smithi194_186548_object32855:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b75ee19:::benchmark_data_smithi194_186548_object32173:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b76e27f:::benchmark_data_smithi194_186548_object40223:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b77cce8:::benchmark_data_smithi194_186548_object20648:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7953a7:::benchmark_data_smithi194_186548_object55712:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7a1557:::benchmark_data_smithi194_186548_object51469:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7b844e:::benchmark_data_smithi194_186548_object35159:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7c4ef3:::benchmark_data_smithi194_186548_object43869:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7cb9d8:::benchmark_data_smithi194_186548_object8631:head# 2024-04-29T14:53:22.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7d10ac:::benchmark_data_smithi194_186548_object54201:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7d9e7d:::benchmark_data_smithi194_186548_object55030:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7dcd4f:::benchmark_data_smithi194_186548_object12500:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b7ecca4:::benchmark_data_smithi194_186548_object6618:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b802a03:::benchmark_data_smithi194_186548_object64901:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b803f35:::benchmark_data_smithi194_186548_object45204:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b80e4f3:::benchmark_data_smithi194_186548_object18938:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b80fd1b:::benchmark_data_smithi194_186548_object62933:head# 2024-04-29T14:53:22.257 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b8255a3:::benchmark_data_smithi194_186548_object59620:head# 2024-04-29T14:53:22.258 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b82b5f4:::benchmark_data_smithi194_186548_object44536:head# 2024-04-29T14:53:22.258 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b82d17a:::benchmark_data_smithi194_186548_object4225:head# 2024-04-29T14:53:22.258 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b834760:::benchmark_data_smithi194_186548_object73876:head# 2024-04-29T14:53:22.263 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b8372df:::benchmark_data_smithi194_186548_object11981:head# 2024-04-29T14:53:22.264 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b847189:::benchmark_data_smithi194_186548_object57725:head# 2024-04-29T14:53:22.265 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b84d88a:::benchmark_data_smithi194_186548_object73054:head# 2024-04-29T14:53:22.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b84f097:::benchmark_data_smithi194_186548_object36450:head# 2024-04-29T14:53:22.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b85a206:::benchmark_data_smithi194_186548_object3039:head# 2024-04-29T14:53:22.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b85eaec:::benchmark_data_smithi194_186548_object78658:head# 2024-04-29T14:53:22.269 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b86bbfb:::benchmark_data_smithi194_186548_object4435:head# 2024-04-29T14:53:22.269 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b88edc6:::benchmark_data_smithi194_186548_object53515:head# 2024-04-29T14:53:22.269 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b88f85a:::benchmark_data_smithi194_186548_object47747:head# 2024-04-29T14:53:22.272 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b8a7b73:::benchmark_data_smithi194_186548_object69043:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b8acea4:::benchmark_data_smithi194_186548_object10317:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b8de23e:::benchmark_data_smithi194_186548_object67019:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b8ecefd:::benchmark_data_smithi194_186548_object6483:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b8f6637:::benchmark_data_smithi194_186548_object68436:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b91002f:::benchmark_data_smithi194_186548_object50223:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b914d1c:::benchmark_data_smithi194_186548_object76557:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b91dd0c:::benchmark_data_smithi194_186548_object46888:head# 2024-04-29T14:53:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b91ffd5:::benchmark_data_smithi194_186548_object11412:head# 2024-04-29T14:53:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b938cc2:::benchmark_data_smithi194_186548_object8418:head# 2024-04-29T14:53:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b93c6a6:::benchmark_data_smithi194_186548_object31520:head# 2024-04-29T14:53:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b943076:::benchmark_data_smithi194_186548_object39844:head# 2024-04-29T14:53:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b957469:::benchmark_data_smithi194_186548_object61625:head# 2024-04-29T14:53:22.278 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b958c65:::benchmark_data_smithi194_186548_object50574:head# 2024-04-29T14:53:22.278 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b963454:::benchmark_data_smithi194_186548_object37179:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b98e395:::benchmark_data_smithi194_186548_object70231:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b99bd5d:::benchmark_data_smithi194_186548_object40168:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b9ba2be:::benchmark_data_smithi194_186548_object84926:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b9c0099:::benchmark_data_smithi194_186548_object22552:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b9c899e:::benchmark_data_smithi194_186548_object40924:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b9e657d:::benchmark_data_smithi194_186548_object53015:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b9f23eb:::benchmark_data_smithi194_186548_object77136:head# 2024-04-29T14:53:22.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3b9fecb8:::benchmark_data_smithi194_186548_object59262:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba11549:::benchmark_data_smithi194_186548_object15029:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba1e3fa:::benchmark_data_smithi194_186548_object74043:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba208de:::benchmark_data_smithi194_186548_object76265:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba2ed65:::benchmark_data_smithi194_186548_object82531:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba39c40:::benchmark_data_smithi194_186548_object69545:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba40c81:::benchmark_data_smithi194_186548_object39811:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba46fc6:::benchmark_data_smithi194_186548_object33862:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba47a66:::benchmark_data_smithi194_186548_object76120:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba5bb72:::benchmark_data_smithi194_186548_object73000:head# 2024-04-29T14:53:22.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba640bc:::benchmark_data_smithi194_186548_object69106:head# 2024-04-29T14:53:22.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba6dbb5:::benchmark_data_smithi194_186548_object54012:head# 2024-04-29T14:53:22.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba88e98:::benchmark_data_smithi194_186548_object40905:head# 2024-04-29T14:53:22.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba8cce4:::benchmark_data_smithi194_186548_object44961:head# 2024-04-29T14:53:22.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ba918f9:::benchmark_data_smithi194_186548_object9356:head# 2024-04-29T14:53:22.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bab66d1:::benchmark_data_smithi194_186548_object34032:head# 2024-04-29T14:53:22.292 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3babd9c9:::benchmark_data_smithi194_186548_object73302:head# 2024-04-29T14:53:22.292 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bacaa96:::benchmark_data_smithi194_186548_object8046:head# 2024-04-29T14:53:22.293 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bad008d:::benchmark_data_smithi194_186548_object62470:head# 2024-04-29T14:53:22.293 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bae7731:::benchmark_data_smithi194_186548_object13643:head# 2024-04-29T14:53:22.294 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb0d472:::benchmark_data_smithi194_186548_object32348:head# 2024-04-29T14:53:22.295 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb17247:::benchmark_data_smithi194_186548_object59183:head# 2024-04-29T14:53:22.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb1912e:::benchmark_data_smithi194_186548_object1075:head# 2024-04-29T14:53:22.296 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb1adec:::benchmark_data_smithi194_186548_object63284:head# 2024-04-29T14:53:22.297 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb1b6de:::benchmark_data_smithi194_186548_object65531:head# 2024-04-29T14:53:22.297 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb1d90c:::benchmark_data_smithi194_186548_object31812:head# 2024-04-29T14:53:22.297 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb2d51a:::benchmark_data_smithi194_186548_object12873:head# 2024-04-29T14:53:22.297 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb333a8:::benchmark_data_smithi194_186548_object10085:head# 2024-04-29T14:53:22.298 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb3782f:::benchmark_data_smithi194_186548_object8101:head# 2024-04-29T14:53:22.298 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb3b70c:::benchmark_data_smithi194_186548_object9858:head# 2024-04-29T14:53:22.299 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb4d677:::benchmark_data_smithi194_186548_object23055:head# 2024-04-29T14:53:22.299 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb5e604:::benchmark_data_smithi194_186548_object61291:head# 2024-04-29T14:53:22.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb602a5:::benchmark_data_smithi194_186548_object47803:head# 2024-04-29T14:53:22.304 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb68cf3:::benchmark_data_smithi194_186548_object3684:head# 2024-04-29T14:53:22.304 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb82b03:::benchmark_data_smithi194_186548_object26182:head# 2024-04-29T14:53:22.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bb8e6fc:::benchmark_data_smithi194_186548_object15570:head# 2024-04-29T14:53:22.305 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bbaac34:::benchmark_data_smithi194_186548_object9306:head# 2024-04-29T14:53:22.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bbc0190:::benchmark_data_smithi194_186548_object19969:head# 2024-04-29T14:53:22.306 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bbd4678:::benchmark_data_smithi194_186548_object23943:head# 2024-04-29T14:53:22.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bbde058:::benchmark_data_smithi194_186548_object73419:head# 2024-04-29T14:53:22.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc04490:::benchmark_data_smithi194_186548_object66142:head# 2024-04-29T14:53:22.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc0a785:::benchmark_data_smithi194_186548_object50434:head# 2024-04-29T14:53:22.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc0c883:::benchmark_data_smithi194_186548_object27048:head# 2024-04-29T14:53:22.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc1fe3a:::benchmark_data_smithi194_186548_object50386:head# 2024-04-29T14:53:22.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc2d0f9:::benchmark_data_smithi194_186548_object2528:head# 2024-04-29T14:53:22.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc3bdad:::benchmark_data_smithi194_186548_object7783:head# 2024-04-29T14:53:22.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc518a0:::benchmark_data_smithi194_186548_object75061:head# 2024-04-29T14:53:22.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc5aed3:::benchmark_data_smithi194_186548_object6678:head# 2024-04-29T14:53:22.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc803ed:::benchmark_data_smithi194_186548_object44223:head# 2024-04-29T14:53:22.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc870eb:::benchmark_data_smithi194_186548_object63869:head# 2024-04-29T14:53:22.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc8b5f4:::benchmark_data_smithi194_186548_object32862:head# 2024-04-29T14:53:22.312 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc90386:::benchmark_data_smithi194_186548_object67008:head# 2024-04-29T14:53:22.313 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bc9b3d9:::benchmark_data_smithi194_186548_object49571:head# 2024-04-29T14:53:22.313 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bcad741:::benchmark_data_smithi194_186548_object63182:head# 2024-04-29T14:53:22.314 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bcb8baf:::benchmark_data_smithi194_186548_object51692:head# 2024-04-29T14:53:22.314 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bcc1ec8:::benchmark_data_smithi194_186548_object15515:head# 2024-04-29T14:53:22.315 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bcc58ae:::benchmark_data_smithi194_186548_object44701:head# 2024-04-29T14:53:22.315 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bccc6bb:::benchmark_data_smithi194_186548_object22205:head# 2024-04-29T14:53:22.315 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bce83d1:::benchmark_data_smithi194_186548_object82179:head# 2024-04-29T14:53:22.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd0e0c4:::benchmark_data_smithi194_186548_object12154:head# 2024-04-29T14:53:22.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd2a222:::benchmark_data_smithi194_186548_object277:head# 2024-04-29T14:53:22.317 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd3899e:::benchmark_data_smithi194_186548_object42443:head# 2024-04-29T14:53:22.317 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd3cbf2:::benchmark_data_smithi194_186548_object17275:head# 2024-04-29T14:53:22.318 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd40f0d:::benchmark_data_smithi194_186548_object56400:head# 2024-04-29T14:53:22.318 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd4cfc2:::benchmark_data_smithi194_186548_object60932:head# 2024-04-29T14:53:22.319 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd5c792:::benchmark_data_smithi194_186548_object34881:head# 2024-04-29T14:53:22.323 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd666fa:::benchmark_data_smithi194_186548_object26423:head# 2024-04-29T14:53:22.324 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd76fbc:::benchmark_data_smithi194_186548_object78933:head# 2024-04-29T14:53:22.324 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd77ebf:::benchmark_data_smithi194_186548_object32522:head# 2024-04-29T14:53:22.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd7c730:::benchmark_data_smithi194_186548_object30938:head# 2024-04-29T14:53:22.325 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd7da68:::benchmark_data_smithi194_186548_object60803:head# 2024-04-29T14:53:22.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bd869ee:::benchmark_data_smithi194_186548_object29534:head# 2024-04-29T14:53:22.326 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bdac5b6:::benchmark_data_smithi194_186548_object13806:head# 2024-04-29T14:53:22.327 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bdbd991:::benchmark_data_smithi194_186548_object29430:head# 2024-04-29T14:53:22.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bde6437:::benchmark_data_smithi194_186548_object39831:head# 2024-04-29T14:53:22.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bdfa6db:::benchmark_data_smithi194_186548_object39723:head# 2024-04-29T14:53:22.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bdfd7f5:::benchmark_data_smithi194_186548_object18699:head# 2024-04-29T14:53:22.331 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be0b7db:::benchmark_data_smithi194_186548_object60216:head# 2024-04-29T14:53:22.333 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be18a43:::benchmark_data_smithi194_186548_object12433:head# 2024-04-29T14:53:22.333 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be1b0ee:::benchmark_data_smithi194_186548_object58215:head# 2024-04-29T14:53:22.334 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be20071:::benchmark_data_smithi194_186548_object50547:head# 2024-04-29T14:53:22.334 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be2d57f:::benchmark_data_smithi194_186548_object78539:head# 2024-04-29T14:53:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be44b46:::benchmark_data_smithi194_186548_object60063:head# 2024-04-29T14:53:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be4704b:::benchmark_data_smithi194_186548_object60218:head# 2024-04-29T14:53:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be4fd2b:::benchmark_data_smithi194_186548_object16914:head# 2024-04-29T14:53:22.339 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be551b7:::benchmark_data_smithi194_186548_object22808:head# 2024-04-29T14:53:22.339 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be791c6:::benchmark_data_smithi194_186548_object62930:head# 2024-04-29T14:53:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be7f3f0:::benchmark_data_smithi194_186548_object84249:head# 2024-04-29T14:53:22.342 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be811f5:::benchmark_data_smithi194_186548_object7136:head# 2024-04-29T14:53:22.343 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be9cb1c:::benchmark_data_smithi194_186548_object39263:head# 2024-04-29T14:53:22.345 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3be9f7ac:::benchmark_data_smithi194_186548_object33502:head# 2024-04-29T14:53:22.345 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3beb07e2:::benchmark_data_smithi194_186548_object44185:head# 2024-04-29T14:53:22.347 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bec35db:::benchmark_data_smithi194_186548_object80687:head# 2024-04-29T14:53:22.347 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bee28be:::benchmark_data_smithi194_186548_object31113:head# 2024-04-29T14:53:22.347 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bee6f52:::benchmark_data_smithi194_186548_object59553:head# 2024-04-29T14:53:22.348 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bef88fd:::benchmark_data_smithi194_186548_object15257:head# 2024-04-29T14:53:22.351 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf0ea78:::benchmark_data_smithi194_186548_object32502:head# 2024-04-29T14:53:22.352 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf0f408:::benchmark_data_smithi194_186548_object34925:head# 2024-04-29T14:53:22.352 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf1447f:::benchmark_data_smithi194_186548_object19762:head# 2024-04-29T14:53:22.352 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf18b57:::benchmark_data_smithi194_186548_object57311:head# 2024-04-29T14:53:22.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf20186:::benchmark_data_smithi194_186548_object21788:head# 2024-04-29T14:53:22.356 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf21447:::benchmark_data_smithi194_186548_object81376:head# 2024-04-29T14:53:22.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf256f9:::benchmark_data_smithi194_186548_object55946:head# 2024-04-29T14:53:22.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf391dd:::benchmark_data_smithi194_186548_object69609:head# 2024-04-29T14:53:22.358 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf407b8:::benchmark_data_smithi194_186548_object26512:head# 2024-04-29T14:53:22.359 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:22 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 19136/208518 objects degraded (9.177%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:22.359 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf453ea:::benchmark_data_smithi194_186548_object23507:head# 2024-04-29T14:53:22.362 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf4b696:::benchmark_data_smithi194_186548_object21896:head# 2024-04-29T14:53:22.362 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf4b9b7:::benchmark_data_smithi194_186548_object55525:head# 2024-04-29T14:53:22.362 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf5fe02:::benchmark_data_smithi194_186548_object40886:head# 2024-04-29T14:53:22.363 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf6a5d8:::benchmark_data_smithi194_186548_object35182:head# 2024-04-29T14:53:22.363 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf7406a:::benchmark_data_smithi194_186548_object28111:head# 2024-04-29T14:53:22.363 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf741ed:::benchmark_data_smithi194_186548_object61767:head# 2024-04-29T14:53:22.363 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf79df5:::benchmark_data_smithi194_186548_object80892:head# 2024-04-29T14:53:22.363 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf90516:::benchmark_data_smithi194_186548_object67021:head# 2024-04-29T14:53:22.363 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bf99b7b:::benchmark_data_smithi194_186548_object73630:head# 2024-04-29T14:53:22.364 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bfc737d:::benchmark_data_smithi194_186548_object64437:head# 2024-04-29T14:53:22.365 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bfc8274:::benchmark_data_smithi194_186548_object14804:head# 2024-04-29T14:53:22.365 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bff9412:::benchmark_data_smithi194_186548_object64098:head# 2024-04-29T14:53:22.365 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bff9526:::benchmark_data_smithi194_186548_object54720:head# 2024-04-29T14:53:22.366 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bffc895:::benchmark_data_smithi194_186548_object2821:head# 2024-04-29T14:53:22.366 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3bffdaf8:::benchmark_data_smithi194_186548_object13634:head# 2024-04-29T14:53:22.366 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c006e01:::benchmark_data_smithi194_186548_object73278:head# 2024-04-29T14:53:22.367 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0099ef:::benchmark_data_smithi194_186548_object26946:head# 2024-04-29T14:53:22.370 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c01eeb3:::benchmark_data_smithi194_186548_object62192:head# 2024-04-29T14:53:22.370 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c020858:::benchmark_data_smithi194_186548_object30848:head# 2024-04-29T14:53:22.370 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c04553c:::benchmark_data_smithi194_186548_object595:head# 2024-04-29T14:53:22.371 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c045dbc:::benchmark_data_smithi194_186548_object62545:head# 2024-04-29T14:53:22.371 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c051945:::benchmark_data_smithi194_186548_object46028:head# 2024-04-29T14:53:22.371 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c072f5c:::benchmark_data_smithi194_186548_object40341:head# 2024-04-29T14:53:22.372 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c074d7f:::benchmark_data_smithi194_186548_object78363:head# 2024-04-29T14:53:22.373 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c088774:::benchmark_data_smithi194_186548_object5851:head# 2024-04-29T14:53:22.374 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c08dd35:::benchmark_data_smithi194_186548_object20423:head# 2024-04-29T14:53:22.374 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0a50c7:::benchmark_data_smithi194_186548_object10724:head# 2024-04-29T14:53:22.374 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0ad635:::benchmark_data_smithi194_186548_object41940:head# 2024-04-29T14:53:22.375 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0b0208:::benchmark_data_smithi194_186548_object74441:head# 2024-04-29T14:53:22.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0be8a5:::benchmark_data_smithi194_186548_object61451:head# 2024-04-29T14:53:22.376 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0d52ee:::benchmark_data_smithi194_186548_object35373:head# 2024-04-29T14:53:22.377 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0d86e3:::benchmark_data_smithi194_186548_object39515:head# 2024-04-29T14:53:22.377 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c0e73ef:::benchmark_data_smithi194_186548_object2156:head# 2024-04-29T14:53:22.377 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c103337:::benchmark_data_smithi194_186548_object8471:head# 2024-04-29T14:53:22.382 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1094cb:::benchmark_data_smithi194_186548_object21341:head# 2024-04-29T14:53:22.383 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c130b98:::benchmark_data_smithi194_186548_object61764:head# 2024-04-29T14:53:22.383 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c138c30:::benchmark_data_smithi194_186548_object5095:head# 2024-04-29T14:53:22.383 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c13d258:::benchmark_data_smithi194_186548_object67508:head# 2024-04-29T14:53:22.384 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c14004e:::benchmark_data_smithi194_186548_object23180:head# 2024-04-29T14:53:22.384 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c142b82:::benchmark_data_smithi194_186548_object7138:head# 2024-04-29T14:53:22.384 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c15ff2f:::benchmark_data_smithi194_186548_object75641:head# 2024-04-29T14:53:22.384 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c173fc1:::benchmark_data_smithi194_186548_object57772:head# 2024-04-29T14:53:22.389 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1774d7:::benchmark_data_smithi194_186548_object34908:head# 2024-04-29T14:53:22.390 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c17efde:::benchmark_data_smithi194_186548_object68572:head# 2024-04-29T14:53:22.390 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c182ce2:::benchmark_data_smithi194_186548_object78386:head# 2024-04-29T14:53:22.390 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c18432d:::benchmark_data_smithi194_186548_object43251:head# 2024-04-29T14:53:22.390 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c19500a:::benchmark_data_smithi194_186548_object70095:head# 2024-04-29T14:53:22.396 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1a8ffa:::benchmark_data_smithi194_186548_object61230:head# 2024-04-29T14:53:22.396 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1be0c3:::benchmark_data_smithi194_186548_object30371:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1eb0b7:::benchmark_data_smithi194_186548_object33723:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1ecec5:::benchmark_data_smithi194_186548_object50423:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1f6e76:::benchmark_data_smithi194_186548_object11242:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1fa31f:::benchmark_data_smithi194_186548_object76336:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1fbc71:::benchmark_data_smithi194_186548_object48240:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c1fd029:::benchmark_data_smithi194_186548_object56909:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c20e54e:::benchmark_data_smithi194_186548_object37151:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c221dd8:::benchmark_data_smithi194_186548_object36211:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2317d4:::benchmark_data_smithi194_186548_object45862:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c23e3f2:::benchmark_data_smithi194_186548_object70929:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c24bbc0:::benchmark_data_smithi194_186548_object78526:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c25ce42:::benchmark_data_smithi194_186548_object6755:head# 2024-04-29T14:53:22.397 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c25ea01:::benchmark_data_smithi194_186548_object5099:head# 2024-04-29T14:53:22.398 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2651bd:::benchmark_data_smithi194_186548_object80110:head# 2024-04-29T14:53:22.398 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c26c482:::benchmark_data_smithi194_186548_object65017:head# 2024-04-29T14:53:22.398 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c276b8f:::benchmark_data_smithi194_186548_object19266:head# 2024-04-29T14:53:22.399 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c27bd3b:::benchmark_data_smithi194_186548_object27862:head# 2024-04-29T14:53:22.399 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c286e57:::benchmark_data_smithi194_186548_object29892:head# 2024-04-29T14:53:22.399 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2985ea:::benchmark_data_smithi194_186548_object63641:head# 2024-04-29T14:53:22.399 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2a06cc:::benchmark_data_smithi194_186548_object84877:head# 2024-04-29T14:53:22.400 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2a4852:::benchmark_data_smithi194_186548_object62598:head# 2024-04-29T14:53:22.400 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2a8830:::benchmark_data_smithi194_186548_object58810:head# 2024-04-29T14:53:22.400 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2af1cc:::benchmark_data_smithi194_186548_object56676:head# 2024-04-29T14:53:22.401 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2c7741:::benchmark_data_smithi194_186548_object83938:head# 2024-04-29T14:53:22.401 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2ca6a4:::benchmark_data_smithi194_186548_object66702:head# 2024-04-29T14:53:22.401 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2cf9fb:::benchmark_data_smithi194_186548_object22857:head# 2024-04-29T14:53:22.402 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2cfeed:::benchmark_data_smithi194_186548_object71276:head# 2024-04-29T14:53:22.402 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2d8c8b:::benchmark_data_smithi194_186548_object63436:head# 2024-04-29T14:53:22.403 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2da9bf:::benchmark_data_smithi194_186548_object44463:head# 2024-04-29T14:53:22.403 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2de521:::benchmark_data_smithi194_186548_object84492:head# 2024-04-29T14:53:22.404 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2e6818:::benchmark_data_smithi194_186548_object47079:head# 2024-04-29T14:53:22.404 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2f0fdd:::benchmark_data_smithi194_186548_object33325:head# 2024-04-29T14:53:22.404 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c2f21c2:::benchmark_data_smithi194_186548_object84544:head# 2024-04-29T14:53:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c313931:::benchmark_data_smithi194_186548_object5696:head# 2024-04-29T14:53:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c324d19:::benchmark_data_smithi194_186548_object24404:head# 2024-04-29T14:53:22.407 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c326a3c:::benchmark_data_smithi194_186548_object6127:head# 2024-04-29T14:53:22.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c32bd8d:::benchmark_data_smithi194_186548_object48885:head# 2024-04-29T14:53:22.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c32bdd3:::benchmark_data_smithi194_186548_object29451:head# 2024-04-29T14:53:22.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c33f050:::benchmark_data_smithi194_186548_object12853:head# 2024-04-29T14:53:22.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c34c3d1:::benchmark_data_smithi194_186548_object830:head# 2024-04-29T14:53:22.411 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c34c8c2:::benchmark_data_smithi194_186548_object32737:head# 2024-04-29T14:53:22.412 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c35dec8:::benchmark_data_smithi194_186548_object47840:head# 2024-04-29T14:53:22.412 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c3b6840:::benchmark_data_smithi194_186548_object30244:head# 2024-04-29T14:53:22.412 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c3c2eec:::benchmark_data_smithi194_186548_object71988:head# 2024-04-29T14:53:22.413 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c3df5ac:::benchmark_data_smithi194_186548_object13110:head# 2024-04-29T14:53:22.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c3e39ce:::benchmark_data_smithi194_186548_object1126:head# 2024-04-29T14:53:22.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c3ed592:::benchmark_data_smithi194_186548_object22045:head# 2024-04-29T14:53:22.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c406a06:::benchmark_data_smithi194_186548_object1428:head# 2024-04-29T14:53:22.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c407381:::benchmark_data_smithi194_186548_object7170:head# 2024-04-29T14:53:22.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c411feb:::benchmark_data_smithi194_186548_object38911:head# 2024-04-29T14:53:22.416 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c41ee9c:::benchmark_data_smithi194_186548_object35289:head# 2024-04-29T14:53:22.417 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4224c2:::benchmark_data_smithi194_186548_object77333:head# 2024-04-29T14:53:22.417 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4343ef:::benchmark_data_smithi194_186548_object71473:head# 2024-04-29T14:53:22.417 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c43981e:::benchmark_data_smithi194_186548_object55521:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4452bf:::benchmark_data_smithi194_186548_object21858:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c44c84e:::benchmark_data_smithi194_186548_object4172:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4710cf:::benchmark_data_smithi194_186548_object59908:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c481476:::benchmark_data_smithi194_186548_object50007:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c482797:::benchmark_data_smithi194_186548_object46368:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4936d0:::benchmark_data_smithi194_186548_object5632:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c49842c:::benchmark_data_smithi194_186548_object2895:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4a80a5:::benchmark_data_smithi194_186548_object23059:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4a9b54:::benchmark_data_smithi194_186548_object37162:head# 2024-04-29T14:53:22.425 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4ab09f:::benchmark_data_smithi194_186548_object24772:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4b5af8:::benchmark_data_smithi194_186548_object17255:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4bc304:::benchmark_data_smithi194_186548_object21441:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4c35c1:::benchmark_data_smithi194_186548_object21505:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4c5148:::benchmark_data_smithi194_186548_object61733:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4d4be8:::benchmark_data_smithi194_186548_object64888:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4d86eb:::benchmark_data_smithi194_186548_object64391:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4d9374:::benchmark_data_smithi194_186548_object75740:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4dd8b0:::benchmark_data_smithi194_186548_object7213:head# 2024-04-29T14:53:22.429 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4e9b3d:::benchmark_data_smithi194_186548_object27525:head# 2024-04-29T14:53:22.430 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c4fcac5:::benchmark_data_smithi194_186548_object73379:head# 2024-04-29T14:53:22.430 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c506547:::benchmark_data_smithi194_186548_object24244:head# 2024-04-29T14:53:22.430 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c50dd47:::benchmark_data_smithi194_186548_object26999:head# 2024-04-29T14:53:22.431 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c52163e:::benchmark_data_smithi194_186548_object16897:head# 2024-04-29T14:53:22.431 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5277f3:::benchmark_data_smithi194_186548_object12827:head# 2024-04-29T14:53:22.431 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c538352:::benchmark_data_smithi194_186548_object68152:head# 2024-04-29T14:53:22.432 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5479f7:::benchmark_data_smithi194_186548_object83660:head# 2024-04-29T14:53:22.432 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c54aa83:::benchmark_data_smithi194_186548_object27563:head# 2024-04-29T14:53:22.433 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5586eb:::benchmark_data_smithi194_186548_object75004:head# 2024-04-29T14:53:22.433 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c558fa0:::benchmark_data_smithi194_186548_object31741:head# 2024-04-29T14:53:22.434 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c56f34a:::benchmark_data_smithi194_186548_object62976:head# 2024-04-29T14:53:22.434 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c57576f:::benchmark_data_smithi194_186548_object80161:head# 2024-04-29T14:53:22.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5760e3:::benchmark_data_smithi194_186548_object45882:head# 2024-04-29T14:53:22.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5a92e2:::benchmark_data_smithi194_186548_object40070:head# 2024-04-29T14:53:22.435 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5b856c:::benchmark_data_smithi194_186548_object31993:head# 2024-04-29T14:53:22.436 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5b9b14:::benchmark_data_smithi194_186548_object73988:head# 2024-04-29T14:53:22.436 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5c8e94:::benchmark_data_smithi194_186548_object66073:head# 2024-04-29T14:53:22.437 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5c93f7:::benchmark_data_smithi194_186548_object71623:head# 2024-04-29T14:53:22.437 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5cc25f:::benchmark_data_smithi194_186548_object48290:head# 2024-04-29T14:53:22.439 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5ef526:::benchmark_data_smithi194_186548_object33023:head# 2024-04-29T14:53:22.440 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5f2291:::benchmark_data_smithi194_186548_object21511:head# 2024-04-29T14:53:22.440 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c5f488b:::benchmark_data_smithi194_186548_object18461:head# 2024-04-29T14:53:22.441 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c613dc3:::benchmark_data_smithi194_186548_object44273:head# 2024-04-29T14:53:22.441 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6141b4:::benchmark_data_smithi194_186548_object22679:head# 2024-04-29T14:53:22.442 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c623548:::benchmark_data_smithi194_186548_object57694:head# 2024-04-29T14:53:22.442 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c62e7bc:::benchmark_data_smithi194_186548_object13123:head# 2024-04-29T14:53:22.443 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c64d1d8:::benchmark_data_smithi194_186548_object5245:head# 2024-04-29T14:53:22.443 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c65317f:::benchmark_data_smithi194_186548_object45155:head# 2024-04-29T14:53:22.444 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c658e1b:::benchmark_data_smithi194_186548_object38293:head# 2024-04-29T14:53:22.447 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c65a687:::benchmark_data_smithi194_186548_object69239:head# 2024-04-29T14:53:22.447 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c669a85:::benchmark_data_smithi194_186548_object30517:head# 2024-04-29T14:53:22.448 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c66f8f4:::benchmark_data_smithi194_186548_object3109:head# 2024-04-29T14:53:22.449 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c67393c:::benchmark_data_smithi194_186548_object54791:head# 2024-04-29T14:53:22.450 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6826cf:::benchmark_data_smithi194_186548_object7231:head# 2024-04-29T14:53:22.453 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c69198a:::benchmark_data_smithi194_186548_object20384:head# 2024-04-29T14:53:22.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c69ea96:::benchmark_data_smithi194_186548_object1733:head# 2024-04-29T14:53:22.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6a2a65:::benchmark_data_smithi194_186548_object70542:head# 2024-04-29T14:53:22.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6ae302:::benchmark_data_smithi194_186548_object6106:head# 2024-04-29T14:53:22.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6b0f27:::benchmark_data_smithi194_186548_object24821:head# 2024-04-29T14:53:22.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6b6a9e:::benchmark_data_smithi194_186548_object72015:head# 2024-04-29T14:53:22.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6d99cb:::benchmark_data_smithi194_186548_object64943:head# 2024-04-29T14:53:22.454 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6e4989:::benchmark_data_smithi194_186548_object72363:head# 2024-04-29T14:53:22.455 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6e7399:::benchmark_data_smithi194_186548_object64329:head# 2024-04-29T14:53:22.458 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6f0846:::benchmark_data_smithi194_186548_object20128:head# 2024-04-29T14:53:22.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c6f9db4:::benchmark_data_smithi194_186548_object46095:head# 2024-04-29T14:53:22.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c704e88:::benchmark_data_smithi194_186548_object58274:head# 2024-04-29T14:53:22.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c708c08:::benchmark_data_smithi194_186548_object48550:head# 2024-04-29T14:53:22.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c717fc8:::benchmark_data_smithi194_186548_object67781:head# 2024-04-29T14:53:22.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c718782:::benchmark_data_smithi194_186548_object75643:head# 2024-04-29T14:53:22.463 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7249aa:::benchmark_data_smithi194_186548_object26780:head# 2024-04-29T14:53:22.464 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c72f3a0:::benchmark_data_smithi194_186548_object84721:head# 2024-04-29T14:53:22.464 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7327fb:::benchmark_data_smithi194_186548_object14932:head# 2024-04-29T14:53:22.465 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c73b6b1:::benchmark_data_smithi194_186548_object58154:head# 2024-04-29T14:53:22.465 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c751e5d:::benchmark_data_smithi194_186548_object38889:head# 2024-04-29T14:53:22.467 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7527ca:::benchmark_data_smithi194_186548_object77295:head# 2024-04-29T14:53:22.468 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c75bcc2:::benchmark_data_smithi194_186548_object46466:head# 2024-04-29T14:53:22.468 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c75be1d:::benchmark_data_smithi194_186548_object8421:head# 2024-04-29T14:53:22.469 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c791d78:::benchmark_data_smithi194_186548_object41417:head# 2024-04-29T14:53:22.469 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7930de:::benchmark_data_smithi194_186548_object77153:head# 2024-04-29T14:53:22.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c79c6e4:::benchmark_data_smithi194_186548_object39607:head# 2024-04-29T14:53:22.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7a4e9e:::benchmark_data_smithi194_186548_object41828:head# 2024-04-29T14:53:22.470 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7b9f56:::benchmark_data_smithi194_186548_object714:head# 2024-04-29T14:53:22.471 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7baf55:::benchmark_data_smithi194_186548_object48214:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7c7fb2:::benchmark_data_smithi194_186548_object979:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7c8194:::benchmark_data_smithi194_186548_object15331:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7ce056:::benchmark_data_smithi194_186548_object68864:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7d5177:::benchmark_data_smithi194_186548_object50051:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7d6352:::benchmark_data_smithi194_186548_object43518:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7e5adf:::benchmark_data_smithi194_186548_object13384:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7e6d00:::benchmark_data_smithi194_186548_object20086:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7eb65d:::benchmark_data_smithi194_186548_object68734:head# 2024-04-29T14:53:22.475 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7ec5f8:::benchmark_data_smithi194_186548_object64047:head# 2024-04-29T14:53:22.476 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7ed4c0:::benchmark_data_smithi194_186548_object12462:head# 2024-04-29T14:53:22.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7f5add:::benchmark_data_smithi194_186548_object67471:head# 2024-04-29T14:53:22.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c7fdfe8:::benchmark_data_smithi194_186548_object55754:head# 2024-04-29T14:53:22.479 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c802f7a:::benchmark_data_smithi194_186548_object57121:head# 2024-04-29T14:53:22.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c818216:::benchmark_data_smithi194_186548_object7852:head# 2024-04-29T14:53:22.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c81e8b7:::benchmark_data_smithi194_186548_object45369:head# 2024-04-29T14:53:22.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c81ff81:::benchmark_data_smithi194_186548_object15545:head# 2024-04-29T14:53:22.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c82355c:::benchmark_data_smithi194_186548_object60972:head# 2024-04-29T14:53:22.480 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c84f2a6:::benchmark_data_smithi194_186548_object7232:head# 2024-04-29T14:53:22.483 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c85383b:::benchmark_data_smithi194_186548_object51983:head# 2024-04-29T14:53:22.483 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c85ddf0:::benchmark_data_smithi194_186548_object7010:head# 2024-04-29T14:53:22.484 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c88cd69:::benchmark_data_smithi194_186548_object1340:head# 2024-04-29T14:53:22.484 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c89b07f:::benchmark_data_smithi194_186548_object19114:head# 2024-04-29T14:53:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c89e7c0:::benchmark_data_smithi194_186548_object58498:head# 2024-04-29T14:53:22.486 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c8ad326:::benchmark_data_smithi194_186548_object58170:head# 2024-04-29T14:53:22.486 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c8bc5b8:::benchmark_data_smithi194_186548_object77651:head# 2024-04-29T14:53:22.488 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c8c3afc:::benchmark_data_smithi194_186548_object63859:head# 2024-04-29T14:53:22.490 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c8de803:::benchmark_data_smithi194_186548_object64140:head# 2024-04-29T14:53:22.490 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c8e8ce1:::benchmark_data_smithi194_186548_object2025:head# 2024-04-29T14:53:22.490 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c8f3b4c:::benchmark_data_smithi194_186548_object1744:head# 2024-04-29T14:53:22.490 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c908e58:::benchmark_data_smithi194_186548_object75663:head# 2024-04-29T14:53:22.491 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c911dc6:::benchmark_data_smithi194_186548_object76904:head# 2024-04-29T14:53:22.491 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9315c9:::benchmark_data_smithi194_186548_object41567:head# 2024-04-29T14:53:22.492 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c936f03:::benchmark_data_smithi194_186548_object31180:head# 2024-04-29T14:53:22.492 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c951eab:::benchmark_data_smithi194_186548_object73819:head# 2024-04-29T14:53:22.493 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c955368:::benchmark_data_smithi194_186548_object35094:head# 2024-04-29T14:53:22.493 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c967335:::benchmark_data_smithi194_186548_object56338:head# 2024-04-29T14:53:22.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c96818b:::benchmark_data_smithi194_186548_object21900:head# 2024-04-29T14:53:22.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c97826b:::benchmark_data_smithi194_186548_object74994:head# 2024-04-29T14:53:22.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c97b116:::benchmark_data_smithi194_186548_object36320:head# 2024-04-29T14:53:22.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c984b4f:::benchmark_data_smithi194_186548_object71061:head# 2024-04-29T14:53:22.500 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c98c6c0:::benchmark_data_smithi194_186548_object74710:head# 2024-04-29T14:53:22.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9a6ae3:::benchmark_data_smithi194_186548_object64442:head# 2024-04-29T14:53:22.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9a6d45:::benchmark_data_smithi194_186548_object65618:head# 2024-04-29T14:53:22.501 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9b48af:::benchmark_data_smithi194_186548_object52084:head# 2024-04-29T14:53:22.502 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9b4c92:::benchmark_data_smithi194_186548_object5179:head# 2024-04-29T14:53:22.502 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9b612e:::benchmark_data_smithi194_186548_object22217:head# 2024-04-29T14:53:22.503 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9bd851:::benchmark_data_smithi194_186548_object51042:head# 2024-04-29T14:53:22.503 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9bee89:::benchmark_data_smithi194_186548_object19275:head# 2024-04-29T14:53:22.504 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9d5c62:::benchmark_data_smithi194_186548_object29567:head# 2024-04-29T14:53:22.504 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3c9d7567:::benchmark_data_smithi194_186548_object36664:head# 2024-04-29T14:53:22.504 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca04631:::benchmark_data_smithi194_186548_object75298:head# 2024-04-29T14:53:22.505 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca0d6fa:::benchmark_data_smithi194_186548_object71295:head# 2024-04-29T14:53:22.505 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca109bc:::benchmark_data_smithi194_186548_object61977:head# 2024-04-29T14:53:22.506 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca12f35:::benchmark_data_smithi194_186548_object27122:head# 2024-04-29T14:53:22.506 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca3ad9c:::benchmark_data_smithi194_186548_object66492:head# 2024-04-29T14:53:22.507 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca47d37:::benchmark_data_smithi194_186548_object59519:head# 2024-04-29T14:53:22.507 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca48727:::benchmark_data_smithi194_186548_object55680:head# 2024-04-29T14:53:22.507 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca66f8a:::benchmark_data_smithi194_186548_object29226:head# 2024-04-29T14:53:22.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca811cf:::benchmark_data_smithi194_186548_object61064:head# 2024-04-29T14:53:22.510 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca86488:::benchmark_data_smithi194_186548_object24159:head# 2024-04-29T14:53:22.511 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca87b33:::benchmark_data_smithi194_186548_object206:head# 2024-04-29T14:53:22.511 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca8fbb1:::benchmark_data_smithi194_186548_object13184:head# 2024-04-29T14:53:22.511 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca991b8:::benchmark_data_smithi194_186548_object16317:head# 2024-04-29T14:53:22.512 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca9a0d6:::benchmark_data_smithi194_186548_object63964:head# 2024-04-29T14:53:22.513 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ca9b7d7:::benchmark_data_smithi194_186548_object69042:head# 2024-04-29T14:53:22.513 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3caabec5:::benchmark_data_smithi194_186548_object5147:head# 2024-04-29T14:53:22.522 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cab648c:::benchmark_data_smithi194_186548_object38657:head# 2024-04-29T14:53:22.522 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3caba960:::benchmark_data_smithi194_186548_object29822:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cad8e4f:::benchmark_data_smithi194_186548_object48778:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3caec840:::benchmark_data_smithi194_186548_object3523:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3caf2385:::benchmark_data_smithi194_186548_object29640:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb010ab:::benchmark_data_smithi194_186548_object80858:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb1072f:::benchmark_data_smithi194_186548_object15301:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb17388:::benchmark_data_smithi194_186548_object33044:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb31bfc:::benchmark_data_smithi194_186548_object53987:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb344f7:::benchmark_data_smithi194_186548_object80609:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb38f62:::benchmark_data_smithi194_186548_object74996:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb58559:::benchmark_data_smithi194_186548_object15739:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb83361:::benchmark_data_smithi194_186548_object49085:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cb9c071:::benchmark_data_smithi194_186548_object74730:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbb53b3:::benchmark_data_smithi194_186548_object57821:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbc152d:::benchmark_data_smithi194_186548_object12587:head# 2024-04-29T14:53:22.523 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbc6c82:::benchmark_data_smithi194_186548_object5223:head# 2024-04-29T14:53:22.528 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbcc00d:::benchmark_data_smithi194_186548_object12535:head# 2024-04-29T14:53:22.528 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbd0473:::benchmark_data_smithi194_186548_object37042:head# 2024-04-29T14:53:22.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbd58fb:::benchmark_data_smithi194_186548_object18474:head# 2024-04-29T14:53:22.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbd83ba:::benchmark_data_smithi194_186548_object10283:head# 2024-04-29T14:53:22.529 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbe932d:::benchmark_data_smithi194_186548_object47044:head# 2024-04-29T14:53:22.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbf5fd1:::benchmark_data_smithi194_186548_object24478:head# 2024-04-29T14:53:22.530 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbfef34:::benchmark_data_smithi194_186548_object40325:head# 2024-04-29T14:53:22.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cbff4af:::benchmark_data_smithi194_186548_object81530:head# 2024-04-29T14:53:22.531 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc01180:::benchmark_data_smithi194_186548_object32595:head# 2024-04-29T14:53:22.532 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc0d3af:::benchmark_data_smithi194_186548_object81134:head# 2024-04-29T14:53:22.532 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc1a22d:::benchmark_data_smithi194_186548_object12240:head# 2024-04-29T14:53:22.533 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc1ba0d:::benchmark_data_smithi194_186548_object63895:head# 2024-04-29T14:53:22.542 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc3bc71:::benchmark_data_smithi194_186548_object25582:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc42473:::benchmark_data_smithi194_186548_object62705:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc5490c:::benchmark_data_smithi194_186548_object64011:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc66e18:::benchmark_data_smithi194_186548_object49495:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc6ba14:::benchmark_data_smithi194_186548_object69067:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc6edc6:::benchmark_data_smithi194_186548_object40009:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc748be:::benchmark_data_smithi194_186548_object50183:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc7cf85:::benchmark_data_smithi194_186548_object23864:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc7d1b2:::benchmark_data_smithi194_186548_object39200:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cc8b5b9:::benchmark_data_smithi194_186548_object571:head# 2024-04-29T14:53:22.543 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ccb41ee:::benchmark_data_smithi194_186548_object84798:head# 2024-04-29T14:53:22.544 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ccbc710:::benchmark_data_smithi194_186548_object25486:head# 2024-04-29T14:53:22.544 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ccc57d6:::benchmark_data_smithi194_186548_object76681:head# 2024-04-29T14:53:22.544 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ccd8cb6:::benchmark_data_smithi194_186548_object21409:head# 2024-04-29T14:53:22.544 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ccda619:::benchmark_data_smithi194_186548_object32722:head# 2024-04-29T14:53:22.545 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ccdda2a:::benchmark_data_smithi194_186548_object11320:head# 2024-04-29T14:53:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd03b20:::benchmark_data_smithi194_186548_object66908:head# 2024-04-29T14:53:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd0c808:::benchmark_data_smithi194_186548_object16743:head# 2024-04-29T14:53:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd1876d:::benchmark_data_smithi194_186548_object58755:head# 2024-04-29T14:53:22.554 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd19bb5:::benchmark_data_smithi194_186548_object67622:head# 2024-04-29T14:53:22.555 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd1e2be:::benchmark_data_smithi194_186548_object54235:head# 2024-04-29T14:53:22.555 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd23d66:::benchmark_data_smithi194_186548_object15745:head# 2024-04-29T14:53:22.555 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd27aad:::benchmark_data_smithi194_186548_object79060:head# 2024-04-29T14:53:22.555 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd34fb9:::benchmark_data_smithi194_186548_object52680:head# 2024-04-29T14:53:22.555 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd39644:::benchmark_data_smithi194_186548_object6645:head# 2024-04-29T14:53:22.555 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd3ba7f:::benchmark_data_smithi194_186548_object65964:head# 2024-04-29T14:53:22.555 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd3f5f1:::benchmark_data_smithi194_186548_object76472:head# 2024-04-29T14:53:22.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd4a43c:::benchmark_data_smithi194_186548_object62801:head# 2024-04-29T14:53:22.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd71d5a:::benchmark_data_smithi194_186548_object75741:head# 2024-04-29T14:53:22.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd73e10:::benchmark_data_smithi194_186548_object52994:head# 2024-04-29T14:53:22.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd85907:::benchmark_data_smithi194_186548_object67050:head# 2024-04-29T14:53:22.556 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd8bc47:::benchmark_data_smithi194_186548_object73123:head# 2024-04-29T14:53:22.557 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd8db1b:::benchmark_data_smithi194_186548_object9588:head# 2024-04-29T14:53:22.557 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cd8f546:::benchmark_data_smithi194_186548_object19474:head# 2024-04-29T14:53:22.558 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cda4ce2:::benchmark_data_smithi194_186548_object27143:head# 2024-04-29T14:53:22.559 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cdb13e4:::benchmark_data_smithi194_186548_object24452:head# 2024-04-29T14:53:22.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cdc68df:::benchmark_data_smithi194_186548_object24532:head# 2024-04-29T14:53:22.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cdc77b5:::benchmark_data_smithi194_186548_object49182:head# 2024-04-29T14:53:22.560 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cddbf8e:::benchmark_data_smithi194_186548_object44914:head# 2024-04-29T14:53:22.565 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cde2948:::benchmark_data_smithi194_186548_object10213:head# 2024-04-29T14:53:22.568 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cde343f:::benchmark_data_smithi194_186548_object78283:head# 2024-04-29T14:53:22.569 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cde8b6c:::benchmark_data_smithi194_186548_object66314:head# 2024-04-29T14:53:22.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cdf068c:::benchmark_data_smithi194_186548_object76975:head# 2024-04-29T14:53:22.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce0bb85:::benchmark_data_smithi194_186548_object45910:head# 2024-04-29T14:53:22.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce1cc69:::benchmark_data_smithi194_186548_object30725:head# 2024-04-29T14:53:22.572 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce2b84f:::benchmark_data_smithi194_186548_object56572:head# 2024-04-29T14:53:22.573 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce2bbb9:::benchmark_data_smithi194_186548_object30671:head# 2024-04-29T14:53:22.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce3f848:::benchmark_data_smithi194_186548_object79062:head# 2024-04-29T14:53:22.574 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce3fab2:::benchmark_data_smithi194_186548_object16093:head# 2024-04-29T14:53:22.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:22 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 19136/208518 objects degraded (9.177%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:22.575 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce5304b:::benchmark_data_smithi194_186548_object48450:head# 2024-04-29T14:53:22.575 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce6dc24:::benchmark_data_smithi194_186548_object75409:head# 2024-04-29T14:53:22.575 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce6f678:::benchmark_data_smithi194_186548_object49228:head# 2024-04-29T14:53:22.575 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce75150:::benchmark_data_smithi194_186548_object42604:head# 2024-04-29T14:53:22.576 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce767cc:::benchmark_data_smithi194_186548_object6099:head# 2024-04-29T14:53:22.577 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce7d53e:::benchmark_data_smithi194_186548_object63816:head# 2024-04-29T14:53:22.577 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce890c3:::benchmark_data_smithi194_186548_object32020:head# 2024-04-29T14:53:22.577 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce93f6f:::benchmark_data_smithi194_186548_object73931:head# 2024-04-29T14:53:22.578 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ce9e4f2:::benchmark_data_smithi194_186548_object43319:head# 2024-04-29T14:53:22.580 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cea3f79:::benchmark_data_smithi194_186548_object35385:head# 2024-04-29T14:53:22.580 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ceae50e:::benchmark_data_smithi194_186548_object83896:head# 2024-04-29T14:53:22.581 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ceba365:::benchmark_data_smithi194_186548_object73430:head# 2024-04-29T14:53:22.581 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cecb8cd:::benchmark_data_smithi194_186548_object77536:head# 2024-04-29T14:53:22.583 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cee42ae:::benchmark_data_smithi194_186548_object83468:head# 2024-04-29T14:53:22.586 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ceefaba:::benchmark_data_smithi194_186548_object72764:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf0e736:::benchmark_data_smithi194_186548_object44439:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf0f96c:::benchmark_data_smithi194_186548_object84993:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf11796:::benchmark_data_smithi194_186548_object80167:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf16d49:::benchmark_data_smithi194_186548_object16282:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf1f49d:::benchmark_data_smithi194_186548_object30407:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf2049e:::benchmark_data_smithi194_186548_object6015:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf2393e:::benchmark_data_smithi194_186548_object1150:head# 2024-04-29T14:53:22.587 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf3e61a:::benchmark_data_smithi194_186548_object76133:head# 2024-04-29T14:53:22.588 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf4dca6:::benchmark_data_smithi194_186548_object79350:head# 2024-04-29T14:53:22.589 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf536cf:::benchmark_data_smithi194_186548_object46790:head# 2024-04-29T14:53:22.589 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf65b4c:::benchmark_data_smithi194_186548_object5704:head# 2024-04-29T14:53:22.589 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cf86024:::benchmark_data_smithi194_186548_object25152:head# 2024-04-29T14:53:22.590 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cfa03dd:::benchmark_data_smithi194_186548_object56887:head# 2024-04-29T14:53:22.591 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cfa7d75:::benchmark_data_smithi194_186548_object46833:head# 2024-04-29T14:53:22.593 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cfa96ff:::benchmark_data_smithi194_186548_object7199:head# 2024-04-29T14:53:22.593 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cfba487:::benchmark_data_smithi194_186548_object12533:head# 2024-04-29T14:53:22.594 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cfd6a8e:::benchmark_data_smithi194_186548_object3200:head# 2024-04-29T14:53:22.595 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3cff2da7:::benchmark_data_smithi194_186548_object65507:head# 2024-04-29T14:53:22.595 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0140ca:::benchmark_data_smithi194_186548_object56445:head# 2024-04-29T14:53:22.596 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d01c95d:::benchmark_data_smithi194_186548_object4784:head# 2024-04-29T14:53:22.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d024054:::benchmark_data_smithi194_186548_object73909:head# 2024-04-29T14:53:22.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d02a038:::benchmark_data_smithi194_186548_object48201:head# 2024-04-29T14:53:22.599 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d03a63e:::benchmark_data_smithi194_186548_object52315:head# 2024-04-29T14:53:22.600 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d04bae8:::benchmark_data_smithi194_186548_object54891:head# 2024-04-29T14:53:22.604 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d055b9f:::benchmark_data_smithi194_186548_object68772:head# 2024-04-29T14:53:22.609 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0643c2:::benchmark_data_smithi194_186548_object61034:head# 2024-04-29T14:53:22.610 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 109128 109112 97.4003 69.625 0.00455592 0.0102644 2024-04-29T14:53:22.617 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d06732e:::benchmark_data_smithi194_186548_object45855:head# 2024-04-29T14:53:22.620 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d075c64:::benchmark_data_smithi194_186548_object8459:head# 2024-04-29T14:53:22.621 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d09ad55:::benchmark_data_smithi194_186548_object34192:head# 2024-04-29T14:53:22.622 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0a045a:::benchmark_data_smithi194_186548_object62530:head# 2024-04-29T14:53:22.622 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0a89d6:::benchmark_data_smithi194_186548_object4844:head# 2024-04-29T14:53:22.623 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0ae772:::benchmark_data_smithi194_186548_object42753:head# 2024-04-29T14:53:22.623 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0b3cd1:::benchmark_data_smithi194_186548_object60522:head# 2024-04-29T14:53:22.625 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0bee17:::benchmark_data_smithi194_186548_object576:head# 2024-04-29T14:53:22.625 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0cab48:::benchmark_data_smithi194_186548_object11971:head# 2024-04-29T14:53:22.625 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0ce11e:::benchmark_data_smithi194_186548_object56672:head# 2024-04-29T14:53:22.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d0d11d0:::benchmark_data_smithi194_186548_object17643:head# 2024-04-29T14:53:22.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d100dd6:::benchmark_data_smithi194_186548_object76532:head# 2024-04-29T14:53:22.627 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d105937:::benchmark_data_smithi194_186548_object60356:head# 2024-04-29T14:53:22.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d10d66d:::benchmark_data_smithi194_186548_object65305:head# 2024-04-29T14:53:22.628 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d10fe01:::benchmark_data_smithi194_186548_object7377:head# 2024-04-29T14:53:22.629 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d114c7d:::benchmark_data_smithi194_186548_object54537:head# 2024-04-29T14:53:22.629 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d11755c:::benchmark_data_smithi194_186548_object27214:head# 2024-04-29T14:53:22.630 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1299a2:::benchmark_data_smithi194_186548_object73124:head# 2024-04-29T14:53:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d12b667:::benchmark_data_smithi194_186548_object65497:head# 2024-04-29T14:53:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d134752:::benchmark_data_smithi194_186548_object39401:head# 2024-04-29T14:53:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d15b34e:::benchmark_data_smithi194_186548_object48026:head# 2024-04-29T14:53:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d166fda:::benchmark_data_smithi194_186548_object1259:head# 2024-04-29T14:53:22.635 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d16dc38:::benchmark_data_smithi194_186548_object84534:head# 2024-04-29T14:53:22.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:22 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 19136/208518 objects degraded (9.177%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:22.636 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d179325:::benchmark_data_smithi194_186548_object28844:head# 2024-04-29T14:53:22.636 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d179b81:::benchmark_data_smithi194_186548_object73843:head# 2024-04-29T14:53:22.639 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d17c039:::benchmark_data_smithi194_186548_object49861:head# 2024-04-29T14:53:22.641 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d183273:::benchmark_data_smithi194_186548_object8756:head# 2024-04-29T14:53:22.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d18a1a2:::benchmark_data_smithi194_186548_object43924:head# 2024-04-29T14:53:22.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d196a3d:::benchmark_data_smithi194_186548_object82105:head# 2024-04-29T14:53:22.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d197a36:::benchmark_data_smithi194_186548_object64087:head# 2024-04-29T14:53:22.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d19ae86:::benchmark_data_smithi194_186548_object2011:head# 2024-04-29T14:53:22.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d19d5ad:::benchmark_data_smithi194_186548_object9889:head# 2024-04-29T14:53:22.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1a8534:::benchmark_data_smithi194_186548_object31365:head# 2024-04-29T14:53:22.642 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1b8b5c:::benchmark_data_smithi194_186548_object21577:head# 2024-04-29T14:53:22.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1bcc04:::benchmark_data_smithi194_186548_object61355:head# 2024-04-29T14:53:22.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1c3bbd:::benchmark_data_smithi194_186548_object31764:head# 2024-04-29T14:53:22.643 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1c7962:::benchmark_data_smithi194_186548_object6940:head# 2024-04-29T14:53:22.644 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1ce581:::benchmark_data_smithi194_186548_object48027:head# 2024-04-29T14:53:22.644 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1d9497:::benchmark_data_smithi194_186548_object22260:head# 2024-04-29T14:53:22.646 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1db253:::benchmark_data_smithi194_186548_object83806:head# 2024-04-29T14:53:22.647 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1e6b36:::benchmark_data_smithi194_186548_object71174:head# 2024-04-29T14:53:22.647 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d1f8e0f:::benchmark_data_smithi194_186548_object37318:head# 2024-04-29T14:53:22.648 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d206666:::benchmark_data_smithi194_186548_object24576:head# 2024-04-29T14:53:22.648 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d207ea2:::benchmark_data_smithi194_186548_object77827:head# 2024-04-29T14:53:22.649 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d210bea:::benchmark_data_smithi194_186548_object76405:head# 2024-04-29T14:53:22.649 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d235435:::benchmark_data_smithi194_186548_object80779:head# 2024-04-29T14:53:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d241a9a:::benchmark_data_smithi194_186548_object58315:head# 2024-04-29T14:53:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2485e0:::benchmark_data_smithi194_186548_object14085:head# 2024-04-29T14:53:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d26906b:::benchmark_data_smithi194_186548_object4360:head# 2024-04-29T14:53:22.653 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d26dc5e:::benchmark_data_smithi194_186548_object58140:head# 2024-04-29T14:53:22.653 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d274900:::benchmark_data_smithi194_186548_object22078:head# 2024-04-29T14:53:22.653 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d276633:::benchmark_data_smithi194_186548_object4970:head# 2024-04-29T14:53:22.653 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d27bbfc:::benchmark_data_smithi194_186548_object11937:head# 2024-04-29T14:53:22.653 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d28534f:::benchmark_data_smithi194_186548_object40378:head# 2024-04-29T14:53:22.654 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2876f1:::benchmark_data_smithi194_186548_object66225:head# 2024-04-29T14:53:22.654 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d29031b:::benchmark_data_smithi194_186548_object13625:head# 2024-04-29T14:53:22.655 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d290cf2:::benchmark_data_smithi194_186548_object21842:head# 2024-04-29T14:53:22.655 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d294921:::benchmark_data_smithi194_186548_object3602:head# 2024-04-29T14:53:22.656 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d299295:::benchmark_data_smithi194_186548_object66504:head# 2024-04-29T14:53:22.656 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2a9fe8:::benchmark_data_smithi194_186548_object60413:head# 2024-04-29T14:53:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2b415c:::benchmark_data_smithi194_186548_object47343:head# 2024-04-29T14:53:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2bde2c:::benchmark_data_smithi194_186548_object3652:head# 2024-04-29T14:53:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2bf735:::benchmark_data_smithi194_186548_object64562:head# 2024-04-29T14:53:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2f01a5:::benchmark_data_smithi194_186548_object25754:head# 2024-04-29T14:53:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2f5a2f:::benchmark_data_smithi194_186548_object25608:head# 2024-04-29T14:53:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d2faa26:::benchmark_data_smithi194_186548_object22272:head# 2024-04-29T14:53:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d30a3ae:::benchmark_data_smithi194_186548_object14474:head# 2024-04-29T14:53:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3222cf:::benchmark_data_smithi194_186548_object28554:head# 2024-04-29T14:53:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d323863:::benchmark_data_smithi194_186548_object1124:head# 2024-04-29T14:53:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d323be2:::benchmark_data_smithi194_186548_object79894:head# 2024-04-29T14:53:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3265c0:::benchmark_data_smithi194_186548_object46464:head# 2024-04-29T14:53:22.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d337c70:::benchmark_data_smithi194_186548_object59254:head# 2024-04-29T14:53:22.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d337ea3:::benchmark_data_smithi194_186548_object45570:head# 2024-04-29T14:53:22.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d33f6ae:::benchmark_data_smithi194_186548_object33784:head# 2024-04-29T14:53:22.677 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d348c39:::benchmark_data_smithi194_186548_object6956:head# 2024-04-29T14:53:22.678 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d34c408:::benchmark_data_smithi194_186548_object10045:head# 2024-04-29T14:53:22.678 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d35f9e0:::benchmark_data_smithi194_186548_object70467:head# 2024-04-29T14:53:22.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d368fd3:::benchmark_data_smithi194_186548_object30733:head# 2024-04-29T14:53:22.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d36f489:::benchmark_data_smithi194_186548_object33570:head# 2024-04-29T14:53:22.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d37d241:::benchmark_data_smithi194_186548_object77887:head# 2024-04-29T14:53:22.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d386718:::benchmark_data_smithi194_186548_object51703:head# 2024-04-29T14:53:22.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d38956f:::benchmark_data_smithi194_186548_object40478:head# 2024-04-29T14:53:22.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3a19fc:::benchmark_data_smithi194_186548_object25185:head# 2024-04-29T14:53:22.690 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3acd52:::benchmark_data_smithi194_186548_object49482:head# 2024-04-29T14:53:22.691 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3b530b:::benchmark_data_smithi194_186548_object13113:head# 2024-04-29T14:53:22.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3b8713:::benchmark_data_smithi194_186548_object4417:head# 2024-04-29T14:53:22.692 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3b8727:::benchmark_data_smithi194_186548_object2495:head# 2024-04-29T14:53:22.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3bab57:::benchmark_data_smithi194_186548_object51634:head# 2024-04-29T14:53:22.695 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3bcb9e:::benchmark_data_smithi194_186548_object66570:head# 2024-04-29T14:53:22.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3bfa0d:::benchmark_data_smithi194_186548_object66881:head# 2024-04-29T14:53:22.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3c7162:::benchmark_data_smithi194_186548_object7808:head# 2024-04-29T14:53:22.696 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3cce5b:::benchmark_data_smithi194_186548_object27600:head# 2024-04-29T14:53:22.697 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3d8dda:::benchmark_data_smithi194_186548_object67607:head# 2024-04-29T14:53:22.701 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3dead9:::benchmark_data_smithi194_186548_object9638:head# 2024-04-29T14:53:22.701 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3e6bda:::benchmark_data_smithi194_186548_object1379:head# 2024-04-29T14:53:22.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3e8acc:::benchmark_data_smithi194_186548_object67431:head# 2024-04-29T14:53:22.702 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3f1783:::benchmark_data_smithi194_186548_object27997:head# 2024-04-29T14:53:22.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d3f53da:::benchmark_data_smithi194_186548_object51837:head# 2024-04-29T14:53:22.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d44eae8:::benchmark_data_smithi194_186548_object46566:head# 2024-04-29T14:53:22.703 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d47a6e5:::benchmark_data_smithi194_186548_object7049:head# 2024-04-29T14:53:22.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4a7127:::benchmark_data_smithi194_186548_object14615:head# 2024-04-29T14:53:22.704 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4bd171:::benchmark_data_smithi194_186548_object75362:head# 2024-04-29T14:53:22.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4cf818:::benchmark_data_smithi194_186548_object30785:head# 2024-04-29T14:53:22.705 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4d58d1:::benchmark_data_smithi194_186548_object47330:head# 2024-04-29T14:53:22.706 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4db5f7:::benchmark_data_smithi194_186548_object13212:head# 2024-04-29T14:53:22.706 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4ec463:::benchmark_data_smithi194_186548_object4669:head# 2024-04-29T14:53:22.707 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4f4c58:::benchmark_data_smithi194_186548_object61336:head# 2024-04-29T14:53:22.707 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4f5fe2:::benchmark_data_smithi194_186548_object68691:head# 2024-04-29T14:53:22.711 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d4fafbf:::benchmark_data_smithi194_186548_object68208:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5089e0:::benchmark_data_smithi194_186548_object42043:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d510ce6:::benchmark_data_smithi194_186548_object11221:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d524967:::benchmark_data_smithi194_186548_object64929:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d534e4f:::benchmark_data_smithi194_186548_object9792:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d541640:::benchmark_data_smithi194_186548_object54258:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5545ce:::benchmark_data_smithi194_186548_object26200:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d555d5d:::benchmark_data_smithi194_186548_object61295:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d560042:::benchmark_data_smithi194_186548_object18671:head# 2024-04-29T14:53:22.712 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d578e48:::benchmark_data_smithi194_186548_object20696:head# 2024-04-29T14:53:22.713 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5855b6:::benchmark_data_smithi194_186548_object55181:head# 2024-04-29T14:53:22.713 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d59ae93:::benchmark_data_smithi194_186548_object29438:head# 2024-04-29T14:53:22.714 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d59f706:::benchmark_data_smithi194_186548_object61347:head# 2024-04-29T14:53:22.714 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5a40b8:::benchmark_data_smithi194_186548_object11110:head# 2024-04-29T14:53:22.715 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5a789b:::benchmark_data_smithi194_186548_object80776:head# 2024-04-29T14:53:22.715 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5b9dcf:::benchmark_data_smithi194_186548_object4649:head# 2024-04-29T14:53:22.715 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5c6386:::benchmark_data_smithi194_186548_object70848:head# 2024-04-29T14:53:22.716 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5c9304:::benchmark_data_smithi194_186548_object64857:head# 2024-04-29T14:53:22.716 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d5f6a14:::benchmark_data_smithi194_186548_object69988:head# 2024-04-29T14:53:22.717 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d60464e:::benchmark_data_smithi194_186548_object18584:head# 2024-04-29T14:53:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6050d7:::benchmark_data_smithi194_186548_object9851:head# 2024-04-29T14:53:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d60a91c:::benchmark_data_smithi194_186548_object20797:head# 2024-04-29T14:53:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6157b0:::benchmark_data_smithi194_186548_object16503:head# 2024-04-29T14:53:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d61ffc6:::benchmark_data_smithi194_186548_object75279:head# 2024-04-29T14:53:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6530c3:::benchmark_data_smithi194_186548_object65851:head# 2024-04-29T14:53:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d65b71b:::benchmark_data_smithi194_186548_object8026:head# 2024-04-29T14:53:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d66078d:::benchmark_data_smithi194_186548_object2468:head# 2024-04-29T14:53:22.724 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d669072:::benchmark_data_smithi194_186548_object16727:head# 2024-04-29T14:53:22.724 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6836fb:::benchmark_data_smithi194_186548_object33783:head# 2024-04-29T14:53:22.724 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d68dc64:::benchmark_data_smithi194_186548_object60683:head# 2024-04-29T14:53:22.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d696a42:::benchmark_data_smithi194_186548_object47236:head# 2024-04-29T14:53:22.728 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d697400:::benchmark_data_smithi194_186548_object7900:head# 2024-04-29T14:53:22.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6aed38:::benchmark_data_smithi194_186548_object52983:head# 2024-04-29T14:53:22.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6b7171:::benchmark_data_smithi194_186548_object13284:head# 2024-04-29T14:53:22.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6c08a3:::benchmark_data_smithi194_186548_object55942:head# 2024-04-29T14:53:22.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6c8a6a:::benchmark_data_smithi194_186548_object10457:head# 2024-04-29T14:53:22.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6c8aa9:::benchmark_data_smithi194_186548_object30211:head# 2024-04-29T14:53:22.729 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6dcb4d:::benchmark_data_smithi194_186548_object9035:head# 2024-04-29T14:53:22.730 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6e542d:::benchmark_data_smithi194_186548_object79328:head# 2024-04-29T14:53:22.730 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6e5a90:::benchmark_data_smithi194_186548_object76935:head# 2024-04-29T14:53:22.731 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6e967b:::benchmark_data_smithi194_186548_object54832:head# 2024-04-29T14:53:22.731 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6efdbe:::benchmark_data_smithi194_186548_object23601:head# 2024-04-29T14:53:22.731 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6f5fba:::benchmark_data_smithi194_186548_object58488:head# 2024-04-29T14:53:22.733 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6f8d9a:::benchmark_data_smithi194_186548_object83601:head# 2024-04-29T14:53:22.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d6ffe9f:::benchmark_data_smithi194_186548_object65774:head# 2024-04-29T14:53:22.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d705286:::benchmark_data_smithi194_186548_object28106:head# 2024-04-29T14:53:22.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d70798c:::benchmark_data_smithi194_186548_object62566:head# 2024-04-29T14:53:22.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d71f787:::benchmark_data_smithi194_186548_object23042:head# 2024-04-29T14:53:22.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d71fb96:::benchmark_data_smithi194_186548_object62975:head# 2024-04-29T14:53:22.734 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d72d73b:::benchmark_data_smithi194_186548_object85033:head# 2024-04-29T14:53:22.735 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d73f180:::benchmark_data_smithi194_186548_object13877:head# 2024-04-29T14:53:22.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d75b992:::benchmark_data_smithi194_186548_object82024:head# 2024-04-29T14:53:22.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7641f4:::benchmark_data_smithi194_186548_object53132:head# 2024-04-29T14:53:22.737 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d76ba0e:::benchmark_data_smithi194_186548_object51514:head# 2024-04-29T14:53:22.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:53:22.739 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d77c05e:::benchmark_data_smithi194_186548_object6193:head# 2024-04-29T14:53:22.741 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d77d017:::benchmark_data_smithi194_186548_object54377:head# 2024-04-29T14:53:22.741 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d78af23:::benchmark_data_smithi194_186548_object27029:head# 2024-04-29T14:53:22.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7a1d91:::benchmark_data_smithi194_186548_object50586:head# 2024-04-29T14:53:22.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7ab60a:::benchmark_data_smithi194_186548_object32183:head# 2024-04-29T14:53:22.744 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7ad04d:::benchmark_data_smithi194_186548_object67183:head# 2024-04-29T14:53:22.746 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7c529a:::benchmark_data_smithi194_186548_object44565:head# 2024-04-29T14:53:22.747 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7db755:::benchmark_data_smithi194_186548_object45725:head# 2024-04-29T14:53:22.747 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7e9564:::benchmark_data_smithi194_186548_object57699:head# 2024-04-29T14:53:22.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d7eab27:::benchmark_data_smithi194_186548_object50875:head# 2024-04-29T14:53:22.748 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d80b1fc:::benchmark_data_smithi194_186548_object44921:head# 2024-04-29T14:53:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d82a33b:::benchmark_data_smithi194_186548_object38779:head# 2024-04-29T14:53:22.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d836306:::benchmark_data_smithi194_186548_object74447:head# 2024-04-29T14:53:22.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d842a7c:::benchmark_data_smithi194_186548_object11486:head# 2024-04-29T14:53:22.752 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d845397:::benchmark_data_smithi194_186548_object22338:head# 2024-04-29T14:53:22.753 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d847bdb:::benchmark_data_smithi194_186548_object25596:head# 2024-04-29T14:53:22.753 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8498f7:::benchmark_data_smithi194_186548_object25243:head# 2024-04-29T14:53:22.754 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d84f688:::benchmark_data_smithi194_186548_object66378:head# 2024-04-29T14:53:22.754 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d84ffd2:::benchmark_data_smithi194_186548_object84266:head# 2024-04-29T14:53:22.757 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d854e48:::benchmark_data_smithi194_186548_object78747:head# 2024-04-29T14:53:22.759 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d85b845:::benchmark_data_smithi194_186548_object57986:head# 2024-04-29T14:53:22.759 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d87062f:::benchmark_data_smithi194_186548_object61382:head# 2024-04-29T14:53:22.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d872f4e:::benchmark_data_smithi194_186548_object2607:head# 2024-04-29T14:53:22.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d87a5dd:::benchmark_data_smithi194_186548_object51916:head# 2024-04-29T14:53:22.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d87c45f:::benchmark_data_smithi194_186548_object53649:head# 2024-04-29T14:53:22.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d87cdf8:::benchmark_data_smithi194_186548_object66047:head# 2024-04-29T14:53:22.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d87f4c4:::benchmark_data_smithi194_186548_object36299:head# 2024-04-29T14:53:22.760 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d882a87:::benchmark_data_smithi194_186548_object63708:head# 2024-04-29T14:53:22.764 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d89be94:::benchmark_data_smithi194_186548_object40479:head# 2024-04-29T14:53:22.764 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8a43d3:::benchmark_data_smithi194_186548_object13490:head# 2024-04-29T14:53:22.765 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8b2962:::benchmark_data_smithi194_186548_object56651:head# 2024-04-29T14:53:22.765 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8b9d1d:::benchmark_data_smithi194_186548_object6038:head# 2024-04-29T14:53:22.766 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8bc404:::benchmark_data_smithi194_186548_object80713:head# 2024-04-29T14:53:22.766 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8df902:::benchmark_data_smithi194_186548_object31345:head# 2024-04-29T14:53:22.767 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8e03a2:::benchmark_data_smithi194_186548_object19963:head# 2024-04-29T14:53:22.767 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8e2956:::benchmark_data_smithi194_186548_object41634:head# 2024-04-29T14:53:22.767 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8f011e:::benchmark_data_smithi194_186548_object26024:head# 2024-04-29T14:53:22.768 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d8f5a1e:::benchmark_data_smithi194_186548_object82513:head# 2024-04-29T14:53:22.768 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d90e7ed:::benchmark_data_smithi194_186548_object56662:head# 2024-04-29T14:53:22.770 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d91b055:::benchmark_data_smithi194_186548_object80451:head# 2024-04-29T14:53:22.770 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d91f849:::benchmark_data_smithi194_186548_object54566:head# 2024-04-29T14:53:22.770 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d941839:::benchmark_data_smithi194_186548_object40567:head# 2024-04-29T14:53:22.771 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9452f9:::benchmark_data_smithi194_186548_object9392:head# 2024-04-29T14:53:22.771 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d953b0e:::benchmark_data_smithi194_186548_object70084:head# 2024-04-29T14:53:22.772 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d95a257:::benchmark_data_smithi194_186548_object41648:head# 2024-04-29T14:53:22.772 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9653e2:::benchmark_data_smithi194_186548_object1265:head# 2024-04-29T14:53:22.773 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d965d69:::benchmark_data_smithi194_186548_object51581:head# 2024-04-29T14:53:22.773 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9698ad:::benchmark_data_smithi194_186548_object50294:head# 2024-04-29T14:53:22.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d977847:::benchmark_data_smithi194_186548_object53106:head# 2024-04-29T14:53:22.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d978e07:::benchmark_data_smithi194_186548_object53683:head# 2024-04-29T14:53:22.774 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d97c2b7:::benchmark_data_smithi194_186548_object72763:head# 2024-04-29T14:53:22.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d988c6e:::benchmark_data_smithi194_186548_object27606:head# 2024-04-29T14:53:22.775 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d988d9c:::benchmark_data_smithi194_186548_object65890:head# 2024-04-29T14:53:22.776 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d99a8f4:::benchmark_data_smithi194_186548_object50569:head# 2024-04-29T14:53:22.776 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9a117b:::benchmark_data_smithi194_186548_object40264:head# 2024-04-29T14:53:22.776 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9a7c4f:::benchmark_data_smithi194_186548_object62336:head# 2024-04-29T14:53:22.777 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9afa1e:::benchmark_data_smithi194_186548_object66438:head# 2024-04-29T14:53:22.777 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9d0628:::benchmark_data_smithi194_186548_object1370:head# 2024-04-29T14:53:22.778 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9d0b87:::benchmark_data_smithi194_186548_object74340:head# 2024-04-29T14:53:22.778 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9d0fb9:::benchmark_data_smithi194_186548_object84322:head# 2024-04-29T14:53:22.779 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3d9edffc:::benchmark_data_smithi194_186548_object15936:head# 2024-04-29T14:53:22.779 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3da0770c:::benchmark_data_smithi194_186548_object39345:head# 2024-04-29T14:53:22.779 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3da18dc1:::benchmark_data_smithi194_186548_object21424:head# 2024-04-29T14:53:22.780 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3da225a6:::benchmark_data_smithi194_186548_object35375:head# 2024-04-29T14:53:22.780 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3da313f1:::benchmark_data_smithi194_186548_object15644:head# 2024-04-29T14:53:22.781 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3da498b5:::benchmark_data_smithi194_186548_object30595:head# 2024-04-29T14:53:22.781 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3da6e67e:::benchmark_data_smithi194_186548_object17292:head# 2024-04-29T14:53:22.781 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3da9f2bf:::benchmark_data_smithi194_186548_object77089:head# 2024-04-29T14:53:22.782 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3daa0a3b:::benchmark_data_smithi194_186548_object81255:head# 2024-04-29T14:53:22.785 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dac29f3:::benchmark_data_smithi194_186548_object59199:head# 2024-04-29T14:53:22.785 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dacbe9a:::benchmark_data_smithi194_186548_object42167:head# 2024-04-29T14:53:22.786 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dacd6d4:::benchmark_data_smithi194_186548_object77226:head# 2024-04-29T14:53:22.786 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dadf137:::benchmark_data_smithi194_186548_object71341:head# 2024-04-29T14:53:22.786 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db00ff5:::benchmark_data_smithi194_186548_object53778:head# 2024-04-29T14:53:22.787 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db11024:::benchmark_data_smithi194_186548_object64164:head# 2024-04-29T14:53:22.788 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db143d1:::benchmark_data_smithi194_186548_object51794:head# 2024-04-29T14:53:22.788 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db1495d:::benchmark_data_smithi194_186548_object37202:head# 2024-04-29T14:53:22.788 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db1b120:::benchmark_data_smithi194_186548_object84279:head# 2024-04-29T14:53:22.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db28228:::benchmark_data_smithi194_186548_object67472:head# 2024-04-29T14:53:22.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db2aa5d:::benchmark_data_smithi194_186548_object58607:head# 2024-04-29T14:53:22.789 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db318c8:::benchmark_data_smithi194_186548_object21398:head# 2024-04-29T14:53:22.793 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db335a3:::benchmark_data_smithi194_186548_object9036:head# 2024-04-29T14:53:22.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db33c30:::benchmark_data_smithi194_186548_object8381:head# 2024-04-29T14:53:22.794 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db3b65a:::benchmark_data_smithi194_186548_object54132:head# 2024-04-29T14:53:22.796 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db62a74:::benchmark_data_smithi194_186548_object76499:head# 2024-04-29T14:53:22.798 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db63734:::benchmark_data_smithi194_186548_object66010:head# 2024-04-29T14:53:22.799 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db669f2:::benchmark_data_smithi194_186548_object50868:head# 2024-04-29T14:53:22.799 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db6a411:::benchmark_data_smithi194_186548_object76511:head# 2024-04-29T14:53:22.799 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db6cd68:::benchmark_data_smithi194_186548_object79358:head# 2024-04-29T14:53:22.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db6f654:::benchmark_data_smithi194_186548_object7801:head# 2024-04-29T14:53:22.800 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db6f9fc:::benchmark_data_smithi194_186548_object1140:head# 2024-04-29T14:53:22.803 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db76474:::benchmark_data_smithi194_186548_object79637:head# 2024-04-29T14:53:22.803 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db7ad78:::benchmark_data_smithi194_186548_object27987:head# 2024-04-29T14:53:22.803 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db9308b:::benchmark_data_smithi194_186548_object82039:head# 2024-04-29T14:53:22.804 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db9a909:::benchmark_data_smithi194_186548_object77093:head# 2024-04-29T14:53:22.804 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3db9f724:::benchmark_data_smithi194_186548_object47677:head# 2024-04-29T14:53:22.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dbb269a:::benchmark_data_smithi194_186548_object11522:head# 2024-04-29T14:53:22.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dbbc14e:::benchmark_data_smithi194_186548_object19407:head# 2024-04-29T14:53:22.805 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dbcac92:::benchmark_data_smithi194_186548_object55599:head# 2024-04-29T14:53:22.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dbdc302:::benchmark_data_smithi194_186548_object50935:head# 2024-04-29T14:53:22.806 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dbdd87d:::benchmark_data_smithi194_186548_object37372:head# 2024-04-29T14:53:22.807 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dbe758c:::benchmark_data_smithi194_186548_object75934:head# 2024-04-29T14:53:22.807 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc12db7:::benchmark_data_smithi194_186548_object40113:head# 2024-04-29T14:53:22.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc17223:::benchmark_data_smithi194_186548_object25498:head# 2024-04-29T14:53:22.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc420b7:::benchmark_data_smithi194_186548_object76648:head# 2024-04-29T14:53:22.808 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc42ea2:::benchmark_data_smithi194_186548_object46250:head# 2024-04-29T14:53:22.809 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc4c3f7:::benchmark_data_smithi194_186548_object18363:head# 2024-04-29T14:53:22.809 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc4cc56:::benchmark_data_smithi194_186548_object12825:head# 2024-04-29T14:53:22.810 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc61889:::benchmark_data_smithi194_186548_object74050:head# 2024-04-29T14:53:22.810 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc6567f:::benchmark_data_smithi194_186548_object43170:head# 2024-04-29T14:53:22.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc6dab6:::benchmark_data_smithi194_186548_object34139:head# 2024-04-29T14:53:22.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc7a23a:::benchmark_data_smithi194_186548_object38138:head# 2024-04-29T14:53:22.811 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc9577e:::benchmark_data_smithi194_186548_object34133:head# 2024-04-29T14:53:22.812 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dc9ec61:::benchmark_data_smithi194_186548_object1052:head# 2024-04-29T14:53:22.812 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dcafdb8:::benchmark_data_smithi194_186548_object54203:head# 2024-04-29T14:53:22.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dcc3f03:::benchmark_data_smithi194_186548_object38209:head# 2024-04-29T14:53:22.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dccd2a9:::benchmark_data_smithi194_186548_object15440:head# 2024-04-29T14:53:22.813 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dcd8165:::benchmark_data_smithi194_186548_object49176:head# 2024-04-29T14:53:22.814 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dcf586e:::benchmark_data_smithi194_186548_object33506:head# 2024-04-29T14:53:22.816 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dcf96d4:::benchmark_data_smithi194_186548_object78659:head# 2024-04-29T14:53:22.816 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dcf9944:::benchmark_data_smithi194_186548_object78975:head# 2024-04-29T14:53:22.816 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd0169a:::benchmark_data_smithi194_186548_object16262:head# 2024-04-29T14:53:22.817 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd0e7c1:::benchmark_data_smithi194_186548_object35592:head# 2024-04-29T14:53:22.817 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd113e6:::benchmark_data_smithi194_186548_object20160:head# 2024-04-29T14:53:22.819 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd1f788:::benchmark_data_smithi194_186548_object74234:head# 2024-04-29T14:53:22.819 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd3b792:::benchmark_data_smithi194_186548_object1905:head# 2024-04-29T14:53:22.819 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd41ef6:::benchmark_data_smithi194_186548_object71420:head# 2024-04-29T14:53:22.819 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd4fbc3:::benchmark_data_smithi194_186548_object16553:head# 2024-04-29T14:53:22.819 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd544e9:::benchmark_data_smithi194_186548_object19177:head# 2024-04-29T14:53:22.820 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd6d3dd:::benchmark_data_smithi194_186548_object83410:head# 2024-04-29T14:53:22.820 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd733ee:::benchmark_data_smithi194_186548_object48694:head# 2024-04-29T14:53:22.820 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd7a727:::benchmark_data_smithi194_186548_object46494:head# 2024-04-29T14:53:22.821 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd7bf75:::benchmark_data_smithi194_186548_object38650:head# 2024-04-29T14:53:22.821 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd815be:::benchmark_data_smithi194_186548_object20432:head# 2024-04-29T14:53:22.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd84947:::benchmark_data_smithi194_186548_object29172:head# 2024-04-29T14:53:22.822 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dd99950:::benchmark_data_smithi194_186548_object24892:head# 2024-04-29T14:53:22.823 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ddd2fa2:::benchmark_data_smithi194_186548_object31568:head# 2024-04-29T14:53:22.823 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ddf3a4d:::benchmark_data_smithi194_186548_object76823:head# 2024-04-29T14:53:22.824 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:53:22.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de185cf:::benchmark_data_smithi194_186548_object1912:head# 2024-04-29T14:53:22.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de250c6:::benchmark_data_smithi194_186548_object26891:head# 2024-04-29T14:53:22.824 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de2ad26:::benchmark_data_smithi194_186548_object80884:head# 2024-04-29T14:53:22.825 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de3b47b:::benchmark_data_smithi194_186548_object37522:head# 2024-04-29T14:53:22.825 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de3c4f6:::benchmark_data_smithi194_186548_object75425:head# 2024-04-29T14:53:22.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de40894:::benchmark_data_smithi194_186548_object38625:head# 2024-04-29T14:53:22.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de5d8b5:::benchmark_data_smithi194_186548_object40381:head# 2024-04-29T14:53:22.826 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de7fd1d:::benchmark_data_smithi194_186548_object68141:head# 2024-04-29T14:53:22.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de885bb:::benchmark_data_smithi194_186548_object27568:head# 2024-04-29T14:53:22.827 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de8a0eb:::benchmark_data_smithi194_186548_object28294:head# 2024-04-29T14:53:22.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de8ca91:::benchmark_data_smithi194_186548_object79582:head# 2024-04-29T14:53:22.828 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de964f4:::benchmark_data_smithi194_186548_object55199:head# 2024-04-29T14:53:22.829 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de9e729:::benchmark_data_smithi194_186548_object50953:head# 2024-04-29T14:53:22.829 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3de9ee2b:::benchmark_data_smithi194_186548_object78507:head# 2024-04-29T14:53:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:22.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:53:22.832 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dea0975:::benchmark_data_smithi194_186548_object43229:head# 2024-04-29T14:53:22.841 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dea1ade:::benchmark_data_smithi194_186548_object52928:head# 2024-04-29T14:53:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3deb3691:::benchmark_data_smithi194_186548_object79122:head# 2024-04-29T14:53:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df114f4:::benchmark_data_smithi194_186548_object83520:head# 2024-04-29T14:53:22.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df1a9e8:::benchmark_data_smithi194_186548_object67694:head# 2024-04-29T14:53:22.843 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df24606:::benchmark_data_smithi194_186548_object33663:head# 2024-04-29T14:53:22.845 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df2a029:::benchmark_data_smithi194_186548_object56088:head# 2024-04-29T14:53:22.845 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df38f2a:::benchmark_data_smithi194_186548_object45819:head# 2024-04-29T14:53:22.845 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df4d83f:::benchmark_data_smithi194_186548_object65740:head# 2024-04-29T14:53:22.845 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df5e562:::benchmark_data_smithi194_186548_object69525:head# 2024-04-29T14:53:22.845 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3df5e7bc:::benchmark_data_smithi194_186548_object78733:head# 2024-04-29T14:53:22.846 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dfac306:::benchmark_data_smithi194_186548_object9907:head# 2024-04-29T14:53:22.846 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dfc5efc:::benchmark_data_smithi194_186548_object66992:head# 2024-04-29T14:53:22.847 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dfe5e04:::benchmark_data_smithi194_186548_object21259:head# 2024-04-29T14:53:22.852 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dfea1ca:::benchmark_data_smithi194_186548_object10498:head# 2024-04-29T14:53:22.852 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3dfebb9c:::benchmark_data_smithi194_186548_object70916:head# 2024-04-29T14:53:22.852 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e0019b3:::benchmark_data_smithi194_186548_object31598:head# 2024-04-29T14:53:22.852 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e00ac01:::benchmark_data_smithi194_186548_object70210:head# 2024-04-29T14:53:22.857 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e00e38b:::benchmark_data_smithi194_186548_object27477:head# 2024-04-29T14:53:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e015d9a:::benchmark_data_smithi194_186548_object81800:head# 2024-04-29T14:53:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e03367a:::benchmark_data_smithi194_186548_object70657:head# 2024-04-29T14:53:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e033bab:::benchmark_data_smithi194_186548_object61989:head# 2024-04-29T14:53:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e04f756:::benchmark_data_smithi194_186548_object53830:head# 2024-04-29T14:53:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e056917:::benchmark_data_smithi194_186548_object16485:head# 2024-04-29T14:53:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e0686d3:::benchmark_data_smithi194_186548_object4125:head# 2024-04-29T14:53:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e08a827:::benchmark_data_smithi194_186548_object48806:head# 2024-04-29T14:53:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e097e38:::benchmark_data_smithi194_186548_object15682:head# 2024-04-29T14:53:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e09d078:::benchmark_data_smithi194_186548_object50412:head# 2024-04-29T14:53:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e0a68d2:::benchmark_data_smithi194_186548_object61969:head# 2024-04-29T14:53:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e0c979b:::benchmark_data_smithi194_186548_object47881:head# 2024-04-29T14:53:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e0d4456:::benchmark_data_smithi194_186548_object53952:head# 2024-04-29T14:53:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e0e28f7:::benchmark_data_smithi194_186548_object36310:head# 2024-04-29T14:53:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e0f5ffd:::benchmark_data_smithi194_186548_object19643:head# 2024-04-29T14:53:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e112b9f:::benchmark_data_smithi194_186548_object79114:head# 2024-04-29T14:53:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e123a32:::benchmark_data_smithi194_186548_object64062:head# 2024-04-29T14:53:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e12a72a:::benchmark_data_smithi194_186548_object52278:head# 2024-04-29T14:53:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e12cd7c:::benchmark_data_smithi194_186548_object30544:head# 2024-04-29T14:53:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e132474:::benchmark_data_smithi194_186548_object9403:head# 2024-04-29T14:53:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e137aee:::benchmark_data_smithi194_186548_object37499:head# 2024-04-29T14:53:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e13bd59:::benchmark_data_smithi194_186548_object54096:head# 2024-04-29T14:53:22.878 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e13eee7:::benchmark_data_smithi194_186548_object73672:head# 2024-04-29T14:53:22.880 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1517c8:::benchmark_data_smithi194_186548_object78706:head# 2024-04-29T14:53:22.881 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e16132c:::benchmark_data_smithi194_186548_object2484:head# 2024-04-29T14:53:22.881 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e17c69e:::benchmark_data_smithi194_186548_object63677:head# 2024-04-29T14:53:22.882 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e180ae7:::benchmark_data_smithi194_186548_object61443:head# 2024-04-29T14:53:22.882 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e18b22c:::benchmark_data_smithi194_186548_object75141:head# 2024-04-29T14:53:22.883 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e198ffe:::benchmark_data_smithi194_186548_object47122:head# 2024-04-29T14:53:22.883 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1a0f7e:::benchmark_data_smithi194_186548_object9427:head# 2024-04-29T14:53:22.884 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1a0fa7:::benchmark_data_smithi194_186548_object15289:head# 2024-04-29T14:53:22.884 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1b83bf:::benchmark_data_smithi194_186548_object20369:head# 2024-04-29T14:53:22.884 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1bf1fb:::benchmark_data_smithi194_186548_object68998:head# 2024-04-29T14:53:22.885 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1c1559:::benchmark_data_smithi194_186548_object8923:head# 2024-04-29T14:53:22.885 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1ccf2c:::benchmark_data_smithi194_186548_object61484:head# 2024-04-29T14:53:22.886 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1d4667:::benchmark_data_smithi194_186548_object4343:head# 2024-04-29T14:53:22.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e1f5723:::benchmark_data_smithi194_186548_object4517:head# 2024-04-29T14:53:22.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e20ca21:::benchmark_data_smithi194_186548_object23935:head# 2024-04-29T14:53:22.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e212fbf:::benchmark_data_smithi194_186548_object60234:head# 2024-04-29T14:53:22.887 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e218bde:::benchmark_data_smithi194_186548_object15751:head# 2024-04-29T14:53:22.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e21aee1:::benchmark_data_smithi194_186548_object80396:head# 2024-04-29T14:53:22.888 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e21ecfc:::benchmark_data_smithi194_186548_object26453:head# 2024-04-29T14:53:22.889 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2203bb:::benchmark_data_smithi194_186548_object17750:head# 2024-04-29T14:53:22.890 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e224159:::benchmark_data_smithi194_186548_object63221:head# 2024-04-29T14:53:22.890 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e22d1b4:::benchmark_data_smithi194_186548_object51913:head# 2024-04-29T14:53:22.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e22d269:::benchmark_data_smithi194_186548_object22663:head# 2024-04-29T14:53:22.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e23a2a6:::benchmark_data_smithi194_186548_object11982:head# 2024-04-29T14:53:22.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e259b47:::benchmark_data_smithi194_186548_object80853:head# 2024-04-29T14:53:22.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e25dec7:::benchmark_data_smithi194_186548_object39508:head# 2024-04-29T14:53:22.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2771bd:::benchmark_data_smithi194_186548_object45290:head# 2024-04-29T14:53:22.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e28d32b:::benchmark_data_smithi194_186548_object27545:head# 2024-04-29T14:53:22.893 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e28dab9:::benchmark_data_smithi194_186548_object36818:head# 2024-04-29T14:53:22.894 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e28db7b:::benchmark_data_smithi194_186548_object78904:head# 2024-04-29T14:53:22.894 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e293438:::benchmark_data_smithi194_186548_object67562:head# 2024-04-29T14:53:22.895 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2a8eff:::benchmark_data_smithi194_186548_object77855:head# 2024-04-29T14:53:22.895 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2c1f1c:::benchmark_data_smithi194_186548_object33340:head# 2024-04-29T14:53:22.896 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2cac11:::benchmark_data_smithi194_186548_object17846:head# 2024-04-29T14:53:22.896 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2cd2c7:::benchmark_data_smithi194_186548_object27800:head# 2024-04-29T14:53:22.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2d8b36:::benchmark_data_smithi194_186548_object82281:head# 2024-04-29T14:53:22.897 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2de6e6:::benchmark_data_smithi194_186548_object60115:head# 2024-04-29T14:53:22.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2e028d:::benchmark_data_smithi194_186548_object70633:head# 2024-04-29T14:53:22.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e2e5d80:::benchmark_data_smithi194_186548_object52192:head# 2024-04-29T14:53:22.898 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e30f329:::benchmark_data_smithi194_186548_object63314:head# 2024-04-29T14:53:22.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e323688:::benchmark_data_smithi194_186548_object84578:head# 2024-04-29T14:53:22.899 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e32aceb:::benchmark_data_smithi194_186548_object64178:head# 2024-04-29T14:53:22.900 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e33003b:::benchmark_data_smithi194_186548_object26278:head# 2024-04-29T14:53:22.901 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3342fd:::benchmark_data_smithi194_186548_object42525:head# 2024-04-29T14:53:22.902 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e336332:::benchmark_data_smithi194_186548_object29576:head# 2024-04-29T14:53:22.902 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e336592:::benchmark_data_smithi194_186548_object56959:head# 2024-04-29T14:53:22.904 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e33abae:::benchmark_data_smithi194_186548_object71844:head# 2024-04-29T14:53:22.910 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e33e36a:::benchmark_data_smithi194_186548_object50029:head# 2024-04-29T14:53:22.910 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e340d12:::benchmark_data_smithi194_186548_object52021:head# 2024-04-29T14:53:22.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3414c6:::benchmark_data_smithi194_186548_object55450:head# 2024-04-29T14:53:22.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e35513f:::benchmark_data_smithi194_186548_object985:head# 2024-04-29T14:53:22.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e364d23:::benchmark_data_smithi194_186548_object50050:head# 2024-04-29T14:53:22.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e37c0b9:::benchmark_data_smithi194_186548_object10134:head# 2024-04-29T14:53:22.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e37cf64:::benchmark_data_smithi194_186548_object50571:head# 2024-04-29T14:53:22.911 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e38b0ef:::benchmark_data_smithi194_186548_object51799:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e390757:::benchmark_data_smithi194_186548_object31513:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3a1382:::benchmark_data_smithi194_186548_object74938:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3aeb97:::benchmark_data_smithi194_186548_object3846:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3bef96:::benchmark_data_smithi194_186548_object656:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3d6988:::benchmark_data_smithi194_186548_object57663:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3e6b33:::benchmark_data_smithi194_186548_object70920:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3f4baf:::benchmark_data_smithi194_186548_object21689:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3f8251:::benchmark_data_smithi194_186548_object28238:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3f85c1:::benchmark_data_smithi194_186548_object8157:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e3fdf26:::benchmark_data_smithi194_186548_object69062:head# 2024-04-29T14:53:22.916 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e417d06:::benchmark_data_smithi194_186548_object4323:head# 2024-04-29T14:53:22.917 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e423cc1:::benchmark_data_smithi194_186548_object23135:head# 2024-04-29T14:53:22.917 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e42b9a7:::benchmark_data_smithi194_186548_object51756:head# 2024-04-29T14:53:22.917 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e437f33:::benchmark_data_smithi194_186548_object8805:head# 2024-04-29T14:53:22.918 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e444144:::benchmark_data_smithi194_186548_object33824:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e44e9de:::benchmark_data_smithi194_186548_object58619:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e46404e:::benchmark_data_smithi194_186548_object13226:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e48d246:::benchmark_data_smithi194_186548_object25817:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e4926d7:::benchmark_data_smithi194_186548_object14093:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e49e8cf:::benchmark_data_smithi194_186548_object83264:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e4b0028:::benchmark_data_smithi194_186548_object24469:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e4b97d5:::benchmark_data_smithi194_186548_object73339:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e4df5e5:::benchmark_data_smithi194_186548_object25789:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e4fc58f:::benchmark_data_smithi194_186548_object916:head# 2024-04-29T14:53:22.924 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e51e9df:::benchmark_data_smithi194_186548_object64526:head# 2024-04-29T14:53:22.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e53402a:::benchmark_data_smithi194_186548_object70285:head# 2024-04-29T14:53:22.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e53c3bc:::benchmark_data_smithi194_186548_object79719:head# 2024-04-29T14:53:22.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e53ce67:::benchmark_data_smithi194_186548_object3033:head# 2024-04-29T14:53:22.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e54a8ce:::benchmark_data_smithi194_186548_object8740:head# 2024-04-29T14:53:22.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e551924:::benchmark_data_smithi194_186548_object19635:head# 2024-04-29T14:53:22.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e55d1f2:::benchmark_data_smithi194_186548_object18625:head# 2024-04-29T14:53:22.925 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e568323:::benchmark_data_smithi194_186548_object24127:head# 2024-04-29T14:53:22.926 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e56dfa1:::benchmark_data_smithi194_186548_object46345:head# 2024-04-29T14:53:22.926 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e571a5f:::benchmark_data_smithi194_186548_object38976:head# 2024-04-29T14:53:22.927 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5722e6:::benchmark_data_smithi194_186548_object13548:head# 2024-04-29T14:53:22.927 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e573465:::benchmark_data_smithi194_186548_object45004:head# 2024-04-29T14:53:22.927 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e57c409:::benchmark_data_smithi194_186548_object10960:head# 2024-04-29T14:53:22.928 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e584da0:::benchmark_data_smithi194_186548_object50891:head# 2024-04-29T14:53:22.928 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e59dc1f:::benchmark_data_smithi194_186548_object56393:head# 2024-04-29T14:53:22.929 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e59e213:::benchmark_data_smithi194_186548_object5710:head# 2024-04-29T14:53:22.929 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e59f068:::benchmark_data_smithi194_186548_object1107:head# 2024-04-29T14:53:22.930 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5a3297:::benchmark_data_smithi194_186548_object16256:head# 2024-04-29T14:53:22.930 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5a74f5:::benchmark_data_smithi194_186548_object21807:head# 2024-04-29T14:53:22.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:22.932 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5c2b21:::benchmark_data_smithi194_186548_object24705:head# 2024-04-29T14:53:22.936 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5dade6:::benchmark_data_smithi194_186548_object60685:head# 2024-04-29T14:53:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5e3cd5:::benchmark_data_smithi194_186548_object16324:head# 2024-04-29T14:53:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5e9cc0:::benchmark_data_smithi194_186548_object1866:head# 2024-04-29T14:53:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e5f0aa2:::benchmark_data_smithi194_186548_object56450:head# 2024-04-29T14:53:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e611085:::benchmark_data_smithi194_186548_object83384:head# 2024-04-29T14:53:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e616daa:::benchmark_data_smithi194_186548_object30694:head# 2024-04-29T14:53:22.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e652d6b:::benchmark_data_smithi194_186548_object32744:head# 2024-04-29T14:53:22.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6613e1:::benchmark_data_smithi194_186548_object4803:head# 2024-04-29T14:53:22.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e66a239:::benchmark_data_smithi194_186548_object46911:head# 2024-04-29T14:53:22.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e66c288:::benchmark_data_smithi194_186548_object57433:head# 2024-04-29T14:53:22.947 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e679062:::benchmark_data_smithi194_186548_object7055:head# 2024-04-29T14:53:22.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e68515b:::benchmark_data_smithi194_186548_object24594:head# 2024-04-29T14:53:22.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e685a39:::benchmark_data_smithi194_186548_object62315:head# 2024-04-29T14:53:22.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6991e3:::benchmark_data_smithi194_186548_object33575:head# 2024-04-29T14:53:22.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6a9e7d:::benchmark_data_smithi194_186548_object78592:head# 2024-04-29T14:53:22.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6b1bd5:::benchmark_data_smithi194_186548_object33621:head# 2024-04-29T14:53:22.948 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6cac27:::benchmark_data_smithi194_186548_object59298:head# 2024-04-29T14:53:22.950 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6e0c0c:::benchmark_data_smithi194_186548_object85010:head# 2024-04-29T14:53:22.950 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6e61ec:::benchmark_data_smithi194_186548_object75798:head# 2024-04-29T14:53:22.950 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6ea296:::benchmark_data_smithi194_186548_object41334:head# 2024-04-29T14:53:22.950 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6ec53b:::benchmark_data_smithi194_186548_object11102:head# 2024-04-29T14:53:22.950 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e6f681a:::benchmark_data_smithi194_186548_object82501:head# 2024-04-29T14:53:22.950 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e702b21:::benchmark_data_smithi194_186548_object78034:head# 2024-04-29T14:53:22.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e705ce4:::benchmark_data_smithi194_186548_object71658:head# 2024-04-29T14:53:22.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e715e94:::benchmark_data_smithi194_186548_object25370:head# 2024-04-29T14:53:22.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e735244:::benchmark_data_smithi194_186548_object80337:head# 2024-04-29T14:53:22.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e73ee7e:::benchmark_data_smithi194_186548_object78446:head# 2024-04-29T14:53:22.959 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e74a4e0:::benchmark_data_smithi194_186548_object6368:head# 2024-04-29T14:53:22.960 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e74ed61:::benchmark_data_smithi194_186548_object48699:head# 2024-04-29T14:53:22.960 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e76d890:::benchmark_data_smithi194_186548_object11966:head# 2024-04-29T14:53:22.960 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e7ad6e1:::benchmark_data_smithi194_186548_object1111:head# 2024-04-29T14:53:22.960 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e7e45ad:::benchmark_data_smithi194_186548_object51073:head# 2024-04-29T14:53:22.960 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e7e8e1d:::benchmark_data_smithi194_186548_object55549:head# 2024-04-29T14:53:22.960 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e7f1b12:::benchmark_data_smithi194_186548_object72924:head# 2024-04-29T14:53:22.961 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e7fb040:::benchmark_data_smithi194_186548_object33263:head# 2024-04-29T14:53:22.961 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e801e34:::benchmark_data_smithi194_186548_object29752:head# 2024-04-29T14:53:22.961 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e811e64:::benchmark_data_smithi194_186548_object35025:head# 2024-04-29T14:53:22.962 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e822f66:::benchmark_data_smithi194_186548_object17982:head# 2024-04-29T14:53:22.963 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e82b56a:::benchmark_data_smithi194_186548_object51499:head# 2024-04-29T14:53:22.963 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8409d0:::benchmark_data_smithi194_186548_object25237:head# 2024-04-29T14:53:22.963 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e84a551:::benchmark_data_smithi194_186548_object10784:head# 2024-04-29T14:53:22.963 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e854192:::benchmark_data_smithi194_186548_object46887:head# 2024-04-29T14:53:22.965 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e85b3c5:::benchmark_data_smithi194_186548_object2473:head# 2024-04-29T14:53:22.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e860cfb:::benchmark_data_smithi194_186548_object1070:head# 2024-04-29T14:53:22.966 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e87300e:::benchmark_data_smithi194_186548_object3137:head# 2024-04-29T14:53:22.967 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e879c39:::benchmark_data_smithi194_186548_object41006:head# 2024-04-29T14:53:22.968 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e87a27b:::benchmark_data_smithi194_186548_object27935:head# 2024-04-29T14:53:22.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e88292e:::benchmark_data_smithi194_186548_object56342:head# 2024-04-29T14:53:22.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e88f15a:::benchmark_data_smithi194_186548_object25436:head# 2024-04-29T14:53:22.970 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8979c4:::benchmark_data_smithi194_186548_object75392:head# 2024-04-29T14:53:22.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e89bfa9:::benchmark_data_smithi194_186548_object67637:head# 2024-04-29T14:53:22.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e89e260:::benchmark_data_smithi194_186548_object51911:head# 2024-04-29T14:53:22.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e89f8db:::benchmark_data_smithi194_186548_object36006:head# 2024-04-29T14:53:22.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8a7654:::benchmark_data_smithi194_186548_object46088:head# 2024-04-29T14:53:22.972 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8b6c0b:::benchmark_data_smithi194_186548_object36822:head# 2024-04-29T14:53:22.973 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8b7713:::benchmark_data_smithi194_186548_object36936:head# 2024-04-29T14:53:22.978 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8b77b8:::benchmark_data_smithi194_186548_object48700:head# 2024-04-29T14:53:22.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8c85de:::benchmark_data_smithi194_186548_object58286:head# 2024-04-29T14:53:22.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8cff36:::benchmark_data_smithi194_186548_object26864:head# 2024-04-29T14:53:22.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8d6af8:::benchmark_data_smithi194_186548_object48972:head# 2024-04-29T14:53:22.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8edb01:::benchmark_data_smithi194_186548_object20780:head# 2024-04-29T14:53:22.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e8fa372:::benchmark_data_smithi194_186548_object43364:head# 2024-04-29T14:53:22.979 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e90ce5b:::benchmark_data_smithi194_186548_object22501:head# 2024-04-29T14:53:22.980 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e913a6d:::benchmark_data_smithi194_186548_object44832:head# 2024-04-29T14:53:22.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e919d05:::benchmark_data_smithi194_186548_object71060:head# 2024-04-29T14:53:22.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9218bd:::benchmark_data_smithi194_186548_object72279:head# 2024-04-29T14:53:22.981 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e92617b:::benchmark_data_smithi194_186548_object21406:head# 2024-04-29T14:53:22.982 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e92934b:::benchmark_data_smithi194_186548_object16622:head# 2024-04-29T14:53:22.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9353c8:::benchmark_data_smithi194_186548_object44085:head# 2024-04-29T14:53:22.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e940eff:::benchmark_data_smithi194_186548_object69827:head# 2024-04-29T14:53:22.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9430a6:::benchmark_data_smithi194_186548_object56989:head# 2024-04-29T14:53:22.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9460bc:::benchmark_data_smithi194_186548_object24759:head# 2024-04-29T14:53:22.983 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e94d324:::benchmark_data_smithi194_186548_object7914:head# 2024-04-29T14:53:22.984 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e95212a:::benchmark_data_smithi194_186548_object49490:head# 2024-04-29T14:53:22.985 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:22.987 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e956eeb:::benchmark_data_smithi194_186548_object48526:head# 2024-04-29T14:53:22.987 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e95df57:::benchmark_data_smithi194_186548_object56535:head# 2024-04-29T14:53:22.987 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9628d8:::benchmark_data_smithi194_186548_object54617:head# 2024-04-29T14:53:22.987 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e96ae4b:::benchmark_data_smithi194_186548_object36968:head# 2024-04-29T14:53:22.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e98075a:::benchmark_data_smithi194_186548_object82056:head# 2024-04-29T14:53:22.991 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e984813:::benchmark_data_smithi194_186548_object48796:head# 2024-04-29T14:53:22.992 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9885ef:::benchmark_data_smithi194_186548_object15234:head# 2024-04-29T14:53:22.992 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e99a10b:::benchmark_data_smithi194_186548_object82719:head# 2024-04-29T14:53:22.992 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9b2d19:::benchmark_data_smithi194_186548_object48820:head# 2024-04-29T14:53:22.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9b791d:::benchmark_data_smithi194_186548_object3331:head# 2024-04-29T14:53:22.993 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9d5486:::benchmark_data_smithi194_186548_object12494:head# 2024-04-29T14:53:22.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9d5782:::benchmark_data_smithi194_186548_object44070:head# 2024-04-29T14:53:22.994 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9d8dbb:::benchmark_data_smithi194_186548_object40500:head# 2024-04-29T14:53:22.995 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9d8fc2:::benchmark_data_smithi194_186548_object56029:head# 2024-04-29T14:53:22.999 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9e52f4:::benchmark_data_smithi194_186548_object51467:head# 2024-04-29T14:53:22.999 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9e8211:::benchmark_data_smithi194_186548_object74448:head# 2024-04-29T14:53:22.999 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9ea893:::benchmark_data_smithi194_186548_object56828:head# 2024-04-29T14:53:22.999 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9f3a52:::benchmark_data_smithi194_186548_object37877:head# 2024-04-29T14:53:23.000 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3e9ff69b:::benchmark_data_smithi194_186548_object61990:head# 2024-04-29T14:53:23.000 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea167a4:::benchmark_data_smithi194_186548_object57517:head# 2024-04-29T14:53:23.000 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea19146:::benchmark_data_smithi194_186548_object74658:head# 2024-04-29T14:53:23.006 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea24e6d:::benchmark_data_smithi194_186548_object71674:head# 2024-04-29T14:53:23.006 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea26007:::benchmark_data_smithi194_186548_object12876:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea36ba2:::benchmark_data_smithi194_186548_object80757:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea52fe4:::benchmark_data_smithi194_186548_object23698:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea56b71:::benchmark_data_smithi194_186548_object85251:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea625c3:::benchmark_data_smithi194_186548_object42387:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ea8acc6:::benchmark_data_smithi194_186548_object33955:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eaad17f:::benchmark_data_smithi194_186548_object35414:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eaae1c2:::benchmark_data_smithi194_186548_object21777:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eac2cd5:::benchmark_data_smithi194_186548_object25070:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ead09d7:::benchmark_data_smithi194_186548_object3280:head# 2024-04-29T14:53:23.007 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ead22a4:::benchmark_data_smithi194_186548_object58820:head# 2024-04-29T14:53:23.008 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eadba15:::benchmark_data_smithi194_186548_object4751:head# 2024-04-29T14:53:23.008 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eaf6165:::benchmark_data_smithi194_186548_object30767:head# 2024-04-29T14:53:23.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb093b5:::benchmark_data_smithi194_186548_object68666:head# 2024-04-29T14:53:23.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb2afcc:::benchmark_data_smithi194_186548_object12508:head# 2024-04-29T14:53:23.009 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb36d06:::benchmark_data_smithi194_186548_object79055:head# 2024-04-29T14:53:23.010 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb3d160:::benchmark_data_smithi194_186548_object82563:head# 2024-04-29T14:53:23.012 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb44902:::benchmark_data_smithi194_186548_object31489:head# 2024-04-29T14:53:23.013 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb47abe:::benchmark_data_smithi194_186548_object28559:head# 2024-04-29T14:53:23.014 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb61281:::benchmark_data_smithi194_186548_object14553:head# 2024-04-29T14:53:23.015 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb6b5ca:::benchmark_data_smithi194_186548_object63239:head# 2024-04-29T14:53:23.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb71ea8:::benchmark_data_smithi194_186548_object35903:head# 2024-04-29T14:53:23.016 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb776e4:::benchmark_data_smithi194_186548_object41664:head# 2024-04-29T14:53:23.018 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eb9f8f8:::benchmark_data_smithi194_186548_object68278:head# 2024-04-29T14:53:23.018 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eba1de1:::benchmark_data_smithi194_186548_object16431:head# 2024-04-29T14:53:23.019 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eba3657:::benchmark_data_smithi194_186548_object80035:head# 2024-04-29T14:53:23.019 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eba9d43:::benchmark_data_smithi194_186548_object68020:head# 2024-04-29T14:53:23.020 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebbdbfd:::benchmark_data_smithi194_186548_object71378:head# 2024-04-29T14:53:23.020 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebd16da:::benchmark_data_smithi194_186548_object46867:head# 2024-04-29T14:53:23.021 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebd5a9d:::benchmark_data_smithi194_186548_object25637:head# 2024-04-29T14:53:23.021 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebe2ad3:::benchmark_data_smithi194_186548_object3590:head# 2024-04-29T14:53:23.022 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebe5f5d:::benchmark_data_smithi194_186548_object44966:head# 2024-04-29T14:53:23.023 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebe936d:::benchmark_data_smithi194_186548_object48113:head# 2024-04-29T14:53:23.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebf67db:::benchmark_data_smithi194_186548_object18977:head# 2024-04-29T14:53:23.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebf7dbc:::benchmark_data_smithi194_186548_object73459:head# 2024-04-29T14:53:23.024 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebf89bc:::benchmark_data_smithi194_186548_object68280:head# 2024-04-29T14:53:23.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ebf9452:::benchmark_data_smithi194_186548_object62936:head# 2024-04-29T14:53:23.025 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec0537b:::benchmark_data_smithi194_186548_object47312:head# 2024-04-29T14:53:23.026 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec12560:::benchmark_data_smithi194_186548_object7103:head# 2024-04-29T14:53:23.026 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec126a0:::benchmark_data_smithi194_186548_object43688:head# 2024-04-29T14:53:23.026 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec1730a:::benchmark_data_smithi194_186548_object27742:head# 2024-04-29T14:53:23.027 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec35c97:::benchmark_data_smithi194_186548_object69389:head# 2024-04-29T14:53:23.027 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec3836d:::benchmark_data_smithi194_186548_object21491:head# 2024-04-29T14:53:23.028 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec38562:::benchmark_data_smithi194_186548_object64814:head# 2024-04-29T14:53:23.028 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec4fefb:::benchmark_data_smithi194_186548_object85220:head# 2024-04-29T14:53:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:23.029 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec52a9f:::benchmark_data_smithi194_186548_object30843:head# 2024-04-29T14:53:23.029 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec57dc0:::benchmark_data_smithi194_186548_object1991:head# 2024-04-29T14:53:23.029 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec60c4d:::benchmark_data_smithi194_186548_object3315:head# 2024-04-29T14:53:23.030 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec6224c:::benchmark_data_smithi194_186548_object264:head# 2024-04-29T14:53:23.030 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec665f4:::benchmark_data_smithi194_186548_object66281:head# 2024-04-29T14:53:23.031 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec6e5b4:::benchmark_data_smithi194_186548_object26472:head# 2024-04-29T14:53:23.031 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec93d98:::benchmark_data_smithi194_186548_object21835:head# 2024-04-29T14:53:23.032 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec97c8f:::benchmark_data_smithi194_186548_object77569:head# 2024-04-29T14:53:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:23.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:23.033 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ec9bf4f:::benchmark_data_smithi194_186548_object84274:head# 2024-04-29T14:53:23.033 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eca8d6d:::benchmark_data_smithi194_186548_object65055:head# 2024-04-29T14:53:23.033 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ecb7bd2:::benchmark_data_smithi194_186548_object11600:head# 2024-04-29T14:53:23.033 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ecc2534:::benchmark_data_smithi194_186548_object25567:head# 2024-04-29T14:53:23.033 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed00674:::benchmark_data_smithi194_186548_object73371:head# 2024-04-29T14:53:23.034 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed093dc:::benchmark_data_smithi194_186548_object28422:head# 2024-04-29T14:53:23.034 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed09eda:::benchmark_data_smithi194_186548_object49785:head# 2024-04-29T14:53:23.035 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed0aa52:::benchmark_data_smithi194_186548_object2797:head# 2024-04-29T14:53:23.035 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed0cd8a:::benchmark_data_smithi194_186548_object59601:head# 2024-04-29T14:53:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed100ed:::benchmark_data_smithi194_186548_object27123:head# 2024-04-29T14:53:23.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed2115c:::benchmark_data_smithi194_186548_object12689:head# 2024-04-29T14:53:23.041 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed24290:::benchmark_data_smithi194_186548_object68743:head# 2024-04-29T14:53:23.042 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed350cf:::benchmark_data_smithi194_186548_object71566:head# 2024-04-29T14:53:23.042 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed3762e:::benchmark_data_smithi194_186548_object69267:head# 2024-04-29T14:53:23.042 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed63b05:::benchmark_data_smithi194_186548_object77366:head# 2024-04-29T14:53:23.043 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed79e40:::benchmark_data_smithi194_186548_object47620:head# 2024-04-29T14:53:23.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed8a14d:::benchmark_data_smithi194_186548_object69569:head# 2024-04-29T14:53:23.046 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ed8dca7:::benchmark_data_smithi194_186548_object80325:head# 2024-04-29T14:53:23.047 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eda3386:::benchmark_data_smithi194_186548_object33990:head# 2024-04-29T14:53:23.048 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eda6686:::benchmark_data_smithi194_186548_object74341:head# 2024-04-29T14:53:23.057 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3edda7c5:::benchmark_data_smithi194_186548_object21197:head# 2024-04-29T14:53:23.057 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee08845:::benchmark_data_smithi194_186548_object56719:head# 2024-04-29T14:53:23.057 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee15538:::benchmark_data_smithi194_186548_object40137:head# 2024-04-29T14:53:23.058 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee17732:::benchmark_data_smithi194_186548_object40670:head# 2024-04-29T14:53:23.058 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee1ae43:::benchmark_data_smithi194_186548_object58006:head# 2024-04-29T14:53:23.058 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee257a2:::benchmark_data_smithi194_186548_object17528:head# 2024-04-29T14:53:23.061 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee29dc5:::benchmark_data_smithi194_186548_object80436:head# 2024-04-29T14:53:23.061 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee37607:::benchmark_data_smithi194_186548_object69376:head# 2024-04-29T14:53:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee3e38d:::benchmark_data_smithi194_186548_object23101:head# 2024-04-29T14:53:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee4ea4b:::benchmark_data_smithi194_186548_object49478:head# 2024-04-29T14:53:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee53afd:::benchmark_data_smithi194_186548_object45013:head# 2024-04-29T14:53:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee59d1a:::benchmark_data_smithi194_186548_object19649:head# 2024-04-29T14:53:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee680bf:::benchmark_data_smithi194_186548_object45275:head# 2024-04-29T14:53:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee6c48b:::benchmark_data_smithi194_186548_object40791:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee6e471:::benchmark_data_smithi194_186548_object35029:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee75817:::benchmark_data_smithi194_186548_object17841:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee79d8b:::benchmark_data_smithi194_186548_object45268:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee7ab0d:::benchmark_data_smithi194_186548_object37842:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee7d74c:::benchmark_data_smithi194_186548_object45398:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee81516:::benchmark_data_smithi194_186548_object61520:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ee85721:::benchmark_data_smithi194_186548_object40450:head# 2024-04-29T14:53:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eeb07a0:::benchmark_data_smithi194_186548_object30900:head# 2024-04-29T14:53:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eeb37d2:::benchmark_data_smithi194_186548_object23168:head# 2024-04-29T14:53:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eebd401:::benchmark_data_smithi194_186548_object24607:head# 2024-04-29T14:53:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eebf4f6:::benchmark_data_smithi194_186548_object35980:head# 2024-04-29T14:53:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eec0b51:::benchmark_data_smithi194_186548_object72797:head# 2024-04-29T14:53:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eec1bfc:::benchmark_data_smithi194_186548_object80663:head# 2024-04-29T14:53:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eecb75f:::benchmark_data_smithi194_186548_object80408:head# 2024-04-29T14:53:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eed5d99:::benchmark_data_smithi194_186548_object10727:head# 2024-04-29T14:53:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eedf048:::benchmark_data_smithi194_186548_object53359:head# 2024-04-29T14:53:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eefa70b:::benchmark_data_smithi194_186548_object39099:head# 2024-04-29T14:53:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef0d0a5:::benchmark_data_smithi194_186548_object15005:head# 2024-04-29T14:53:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef0e16b:::benchmark_data_smithi194_186548_object69875:head# 2024-04-29T14:53:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef1177f:::benchmark_data_smithi194_186548_object1922:head# 2024-04-29T14:53:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef146e4:::benchmark_data_smithi194_186548_object58296:head# 2024-04-29T14:53:23.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef2da05:::benchmark_data_smithi194_186548_object5673:head# 2024-04-29T14:53:23.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef30411:::benchmark_data_smithi194_186548_object58953:head# 2024-04-29T14:53:23.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef3176d:::benchmark_data_smithi194_186548_object4244:head# 2024-04-29T14:53:23.085 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef482c0:::benchmark_data_smithi194_186548_object43337:head# 2024-04-29T14:53:23.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef5b278:::benchmark_data_smithi194_186548_object3516:head# 2024-04-29T14:53:23.089 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef739b5:::benchmark_data_smithi194_186548_object3247:head# 2024-04-29T14:53:23.091 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef7b923:::benchmark_data_smithi194_186548_object54447:head# 2024-04-29T14:53:23.092 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef801dd:::benchmark_data_smithi194_186548_object77553:head# 2024-04-29T14:53:23.092 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef8a54c:::benchmark_data_smithi194_186548_object2503:head# 2024-04-29T14:53:23.093 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef94f7a:::benchmark_data_smithi194_186548_object72496:head# 2024-04-29T14:53:23.093 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef96d71:::benchmark_data_smithi194_186548_object82687:head# 2024-04-29T14:53:23.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ef9daab:::benchmark_data_smithi194_186548_object41932:head# 2024-04-29T14:53:23.102 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efa7e54:::benchmark_data_smithi194_186548_object35107:head# 2024-04-29T14:53:23.103 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efb8f83:::benchmark_data_smithi194_186548_object15341:head# 2024-04-29T14:53:23.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efba6a7:::benchmark_data_smithi194_186548_object38641:head# 2024-04-29T14:53:23.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efbb971:::benchmark_data_smithi194_186548_object71636:head# 2024-04-29T14:53:23.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efdb975:::benchmark_data_smithi194_186548_object83427:head# 2024-04-29T14:53:23.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efdd8f7:::benchmark_data_smithi194_186548_object15580:head# 2024-04-29T14:53:23.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efe3104:::benchmark_data_smithi194_186548_object81796:head# 2024-04-29T14:53:23.104 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efe606d:::benchmark_data_smithi194_186548_object11072:head# 2024-04-29T14:53:23.105 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3efed7b5:::benchmark_data_smithi194_186548_object8824:head# 2024-04-29T14:53:23.108 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3eff3ff4:::benchmark_data_smithi194_186548_object34099:head# 2024-04-29T14:53:23.109 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f004d77:::benchmark_data_smithi194_186548_object26009:head# 2024-04-29T14:53:23.110 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f00c599:::benchmark_data_smithi194_186548_object18106:head# 2024-04-29T14:53:23.110 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f01d99f:::benchmark_data_smithi194_186548_object30741:head# 2024-04-29T14:53:23.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0232f4:::benchmark_data_smithi194_186548_object74907:head# 2024-04-29T14:53:23.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0295e1:::benchmark_data_smithi194_186548_object57549:head# 2024-04-29T14:53:23.111 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f034e67:::benchmark_data_smithi194_186548_object19083:head# 2024-04-29T14:53:23.112 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f03639c:::benchmark_data_smithi194_186548_object70283:head# 2024-04-29T14:53:23.112 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f03b436:::benchmark_data_smithi194_186548_object8495:head# 2024-04-29T14:53:23.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f04008a:::benchmark_data_smithi194_186548_object58684:head# 2024-04-29T14:53:23.113 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f064177:::benchmark_data_smithi194_186548_object71869:head# 2024-04-29T14:53:23.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f08c8ef:::benchmark_data_smithi194_186548_object76264:head# 2024-04-29T14:53:23.115 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f08fa1b:::benchmark_data_smithi194_186548_object55460:head# 2024-04-29T14:53:23.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f09061a:::benchmark_data_smithi194_186548_object27801:head# 2024-04-29T14:53:23.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f09767f:::benchmark_data_smithi194_186548_object15479:head# 2024-04-29T14:53:23.116 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0979e8:::benchmark_data_smithi194_186548_object20440:head# 2024-04-29T14:53:23.117 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f099ba4:::benchmark_data_smithi194_186548_object72460:head# 2024-04-29T14:53:23.117 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f09d412:::benchmark_data_smithi194_186548_object22185:head# 2024-04-29T14:53:23.118 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0a7434:::benchmark_data_smithi194_186548_object10229:head# 2024-04-29T14:53:23.118 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0ae932:::benchmark_data_smithi194_186548_object65192:head# 2024-04-29T14:53:23.119 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0bd962:::benchmark_data_smithi194_186548_object80331:head# 2024-04-29T14:53:23.119 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0be93f:::benchmark_data_smithi194_186548_object82504:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0d6a69:::benchmark_data_smithi194_186548_object28253:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0ee168:::benchmark_data_smithi194_186548_object41975:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0f12af:::benchmark_data_smithi194_186548_object19371:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f0f697f:::benchmark_data_smithi194_186548_object32400:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f111122:::benchmark_data_smithi194_186548_object26945:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1147e7:::benchmark_data_smithi194_186548_object11505:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f12298e:::benchmark_data_smithi194_186548_object44779:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f123729:::benchmark_data_smithi194_186548_object35057:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f12e5c1:::benchmark_data_smithi194_186548_object51236:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f12f6e3:::benchmark_data_smithi194_186548_object8076:head# 2024-04-29T14:53:23.126 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f136604:::benchmark_data_smithi194_186548_object5491:head# 2024-04-29T14:53:23.129 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:53:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f156098:::benchmark_data_smithi194_186548_object69887:head# 2024-04-29T14:53:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1562d6:::benchmark_data_smithi194_186548_object17769:head# 2024-04-29T14:53:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f16a888:::benchmark_data_smithi194_186548_object32731:head# 2024-04-29T14:53:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1752b6:::benchmark_data_smithi194_186548_object3092:head# 2024-04-29T14:53:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f17ca23:::benchmark_data_smithi194_186548_object29384:head# 2024-04-29T14:53:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f186644:::benchmark_data_smithi194_186548_object68697:head# 2024-04-29T14:53:23.132 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f189833:::benchmark_data_smithi194_186548_object83140:head# 2024-04-29T14:53:23.132 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1910b1:::benchmark_data_smithi194_186548_object66743:head# 2024-04-29T14:53:23.132 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1a5a48:::benchmark_data_smithi194_186548_object64990:head# 2024-04-29T14:53:23.133 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1ad567:::benchmark_data_smithi194_186548_object28215:head# 2024-04-29T14:53:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1dbfb6:::benchmark_data_smithi194_186548_object24213:head# 2024-04-29T14:53:23.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f1df956:::benchmark_data_smithi194_186548_object30446:head# 2024-04-29T14:53:23.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f2047f3:::benchmark_data_smithi194_186548_object4319:head# 2024-04-29T14:53:23.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f209f4c:::benchmark_data_smithi194_186548_object43281:head# 2024-04-29T14:53:23.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f21cff7:::benchmark_data_smithi194_186548_object5531:head# 2024-04-29T14:53:23.135 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f23ea31:::benchmark_data_smithi194_186548_object16489:head# 2024-04-29T14:53:23.136 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f23fa08:::benchmark_data_smithi194_186548_object61775:head# 2024-04-29T14:53:23.138 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f24a521:::benchmark_data_smithi194_186548_object78594:head# 2024-04-29T14:53:23.139 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f2587a3:::benchmark_data_smithi194_186548_object74789:head# 2024-04-29T14:53:23.139 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f25e231:::benchmark_data_smithi194_186548_object21645:head# 2024-04-29T14:53:23.141 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f2617d2:::benchmark_data_smithi194_186548_object66372:head# 2024-04-29T14:53:23.142 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f267fbf:::benchmark_data_smithi194_186548_object62794:head# 2024-04-29T14:53:23.142 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f280e71:::benchmark_data_smithi194_186548_object36915:head# 2024-04-29T14:53:23.143 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f284d32:::benchmark_data_smithi194_186548_object20139:head# 2024-04-29T14:53:23.144 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f28b875:::benchmark_data_smithi194_186548_object2511:head# 2024-04-29T14:53:23.151 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f2c6442:::benchmark_data_smithi194_186548_object12407:head# 2024-04-29T14:53:23.151 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f2f0098:::benchmark_data_smithi194_186548_object1593:head# 2024-04-29T14:53:23.151 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f2f82a5:::benchmark_data_smithi194_186548_object21529:head# 2024-04-29T14:53:23.152 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f2f92f6:::benchmark_data_smithi194_186548_object76810:head# 2024-04-29T14:53:23.152 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f300176:::benchmark_data_smithi194_186548_object67093:head# 2024-04-29T14:53:23.153 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f300697:::benchmark_data_smithi194_186548_object14089:head# 2024-04-29T14:53:23.153 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f31270a:::benchmark_data_smithi194_186548_object28937:head# 2024-04-29T14:53:23.154 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f325fb6:::benchmark_data_smithi194_186548_object32431:head# 2024-04-29T14:53:23.155 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f33a421:::benchmark_data_smithi194_186548_object77990:head# 2024-04-29T14:53:23.155 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f3408da:::benchmark_data_smithi194_186548_object28991:head# 2024-04-29T14:53:23.155 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f354d5c:::benchmark_data_smithi194_186548_object58045:head# 2024-04-29T14:53:23.156 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f36d2ad:::benchmark_data_smithi194_186548_object59548:head# 2024-04-29T14:53:23.156 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f37e53f:::benchmark_data_smithi194_186548_object73195:head# 2024-04-29T14:53:23.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f37eee6:::benchmark_data_smithi194_186548_object59711:head# 2024-04-29T14:53:23.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f37f1bb:::benchmark_data_smithi194_186548_object60918:head# 2024-04-29T14:53:23.157 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f38f0b6:::benchmark_data_smithi194_186548_object5947:head# 2024-04-29T14:53:23.158 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f392ec0:::benchmark_data_smithi194_186548_object77760:head# 2024-04-29T14:53:23.160 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f3c4b1a:::benchmark_data_smithi194_186548_object53460:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f3d2690:::benchmark_data_smithi194_186548_object63814:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f3e301d:::benchmark_data_smithi194_186548_object8720:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f3f86f9:::benchmark_data_smithi194_186548_object3854:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f407909:::benchmark_data_smithi194_186548_object36123:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f40e080:::benchmark_data_smithi194_186548_object54642:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4179ae:::benchmark_data_smithi194_186548_object19230:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f42915f:::benchmark_data_smithi194_186548_object8380:head# 2024-04-29T14:53:23.161 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f44a77c:::benchmark_data_smithi194_186548_object33706:head# 2024-04-29T14:53:23.165 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f44b18c:::benchmark_data_smithi194_186548_object2646:head# 2024-04-29T14:53:23.165 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f44f2d9:::benchmark_data_smithi194_186548_object44776:head# 2024-04-29T14:53:23.166 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f46bdcc:::benchmark_data_smithi194_186548_object66750:head# 2024-04-29T14:53:23.166 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f470b2f:::benchmark_data_smithi194_186548_object69965:head# 2024-04-29T14:53:23.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f47813d:::benchmark_data_smithi194_186548_object36882:head# 2024-04-29T14:53:23.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f48612f:::benchmark_data_smithi194_186548_object28299:head# 2024-04-29T14:53:23.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f48b930:::benchmark_data_smithi194_186548_object34534:head# 2024-04-29T14:53:23.167 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f49f210:::benchmark_data_smithi194_186548_object55235:head# 2024-04-29T14:53:23.168 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4a7f16:::benchmark_data_smithi194_186548_object58993:head# 2024-04-29T14:53:23.169 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4a9b0b:::benchmark_data_smithi194_186548_object78667:head# 2024-04-29T14:53:23.169 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4afb44:::benchmark_data_smithi194_186548_object80413:head# 2024-04-29T14:53:23.169 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4cdae9:::benchmark_data_smithi194_186548_object49433:head# 2024-04-29T14:53:23.170 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4e51a6:::benchmark_data_smithi194_186548_object29117:head# 2024-04-29T14:53:23.172 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4e659b:::benchmark_data_smithi194_186548_object67130:head# 2024-04-29T14:53:23.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f4e922c:::benchmark_data_smithi194_186548_object31125:head# 2024-04-29T14:53:23.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f54bee0:::benchmark_data_smithi194_186548_object84116:head# 2024-04-29T14:53:23.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f557c4a:::benchmark_data_smithi194_186548_object21703:head# 2024-04-29T14:53:23.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f565214:::benchmark_data_smithi194_186548_object14924:head# 2024-04-29T14:53:23.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f572a64:::benchmark_data_smithi194_186548_object58173:head# 2024-04-29T14:53:23.174 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5789d1:::benchmark_data_smithi194_186548_object31482:head# 2024-04-29T14:53:23.178 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f589282:::benchmark_data_smithi194_186548_object23814:head# 2024-04-29T14:53:23.180 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5ac044:::benchmark_data_smithi194_186548_object15421:head# 2024-04-29T14:53:23.180 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5ad800:::benchmark_data_smithi194_186548_object20263:head# 2024-04-29T14:53:23.180 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5c4a3f:::benchmark_data_smithi194_186548_object49858:head# 2024-04-29T14:53:23.181 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5d45e6:::benchmark_data_smithi194_186548_object70045:head# 2024-04-29T14:53:23.181 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5da9ed:::benchmark_data_smithi194_186548_object57594:head# 2024-04-29T14:53:23.181 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5dc5b5:::benchmark_data_smithi194_186548_object82019:head# 2024-04-29T14:53:23.181 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:23.182 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5ea973:::benchmark_data_smithi194_186548_object15643:head# 2024-04-29T14:53:23.182 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f5fea68:::benchmark_data_smithi194_186548_object76144:head# 2024-04-29T14:53:23.182 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f60f8a7:::benchmark_data_smithi194_186548_object79575:head# 2024-04-29T14:53:23.185 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6180a2:::benchmark_data_smithi194_186548_object20151:head# 2024-04-29T14:53:23.188 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f61fc9e:::benchmark_data_smithi194_186548_object35433:head# 2024-04-29T14:53:23.188 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6274b4:::benchmark_data_smithi194_186548_object22427:head# 2024-04-29T14:53:23.188 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f62beff:::benchmark_data_smithi194_186548_object29572:head# 2024-04-29T14:53:23.188 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f632100:::benchmark_data_smithi194_186548_object48600:head# 2024-04-29T14:53:23.189 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f63eda8:::benchmark_data_smithi194_186548_object28894:head# 2024-04-29T14:53:23.191 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f64420f:::benchmark_data_smithi194_186548_object7258:head# 2024-04-29T14:53:23.191 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f64f982:::benchmark_data_smithi194_186548_object29994:head# 2024-04-29T14:53:23.192 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f65656d:::benchmark_data_smithi194_186548_object35170:head# 2024-04-29T14:53:23.192 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f65c5d4:::benchmark_data_smithi194_186548_object59065:head# 2024-04-29T14:53:23.192 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f66be2d:::benchmark_data_smithi194_186548_object34586:head# 2024-04-29T14:53:23.193 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f676b4f:::benchmark_data_smithi194_186548_object83740:head# 2024-04-29T14:53:23.193 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f67f774:::benchmark_data_smithi194_186548_object16221:head# 2024-04-29T14:53:23.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f68d5ed:::benchmark_data_smithi194_186548_object40785:head# 2024-04-29T14:53:23.194 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6a661c:::benchmark_data_smithi194_186548_object62183:head# 2024-04-29T14:53:23.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6a9991:::benchmark_data_smithi194_186548_object75416:head# 2024-04-29T14:53:23.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6b328e:::benchmark_data_smithi194_186548_object77787:head# 2024-04-29T14:53:23.195 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6b35ba:::benchmark_data_smithi194_186548_object14281:head# 2024-04-29T14:53:23.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6cb08b:::benchmark_data_smithi194_186548_object70444:head# 2024-04-29T14:53:23.196 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6cc874:::benchmark_data_smithi194_186548_object7350:head# 2024-04-29T14:53:23.197 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f6e2850:::benchmark_data_smithi194_186548_object39798:head# 2024-04-29T14:53:23.197 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f72cf08:::benchmark_data_smithi194_186548_object54246:head# 2024-04-29T14:53:23.197 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f73641b:::benchmark_data_smithi194_186548_object41826:head# 2024-04-29T14:53:23.198 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f737dee:::benchmark_data_smithi194_186548_object35333:head# 2024-04-29T14:53:23.200 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f743bad:::benchmark_data_smithi194_186548_object33718:head# 2024-04-29T14:53:23.201 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f767a94:::benchmark_data_smithi194_186548_object70782:head# 2024-04-29T14:53:23.201 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f76aa6f:::benchmark_data_smithi194_186548_object74469:head# 2024-04-29T14:53:23.202 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f77a400:::benchmark_data_smithi194_186548_object67428:head# 2024-04-29T14:53:23.206 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f77f21f:::benchmark_data_smithi194_186548_object27681:head# 2024-04-29T14:53:23.206 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f79a9dd:::benchmark_data_smithi194_186548_object76224:head# 2024-04-29T14:53:23.210 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f7d4184:::benchmark_data_smithi194_186548_object5094:head# 2024-04-29T14:53:23.210 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f7da8ac:::benchmark_data_smithi194_186548_object72498:head# 2024-04-29T14:53:23.211 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f7ebfed:::benchmark_data_smithi194_186548_object49207:head# 2024-04-29T14:53:23.212 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f7f362c:::benchmark_data_smithi194_186548_object42814:head# 2024-04-29T14:53:23.214 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8083a5:::benchmark_data_smithi194_186548_object69394:head# 2024-04-29T14:53:23.214 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f80c622:::benchmark_data_smithi194_186548_object67223:head# 2024-04-29T14:53:23.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8245c7:::benchmark_data_smithi194_186548_object78642:head# 2024-04-29T14:53:23.215 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f83b7ef:::benchmark_data_smithi194_186548_object28014:head# 2024-04-29T14:53:23.216 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f846c51:::benchmark_data_smithi194_186548_object17823:head# 2024-04-29T14:53:23.216 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8717bb:::benchmark_data_smithi194_186548_object11683:head# 2024-04-29T14:53:23.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f89c9b9:::benchmark_data_smithi194_186548_object51797:head# 2024-04-29T14:53:23.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8a63c3:::benchmark_data_smithi194_186548_object52898:head# 2024-04-29T14:53:23.217 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8ac15e:::benchmark_data_smithi194_186548_object63445:head# 2024-04-29T14:53:23.218 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8adc88:::benchmark_data_smithi194_186548_object76329:head# 2024-04-29T14:53:23.221 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8dd204:::benchmark_data_smithi194_186548_object32272:head# 2024-04-29T14:53:23.221 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8e14f7:::benchmark_data_smithi194_186548_object19895:head# 2024-04-29T14:53:23.221 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8e5c08:::benchmark_data_smithi194_186548_object2537:head# 2024-04-29T14:53:23.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8f9916:::benchmark_data_smithi194_186548_object64536:head# 2024-04-29T14:53:23.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f8ffb6c:::benchmark_data_smithi194_186548_object61993:head# 2024-04-29T14:53:23.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9264ef:::benchmark_data_smithi194_186548_object63183:head# 2024-04-29T14:53:23.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f92fcec:::benchmark_data_smithi194_186548_object45770:head# 2024-04-29T14:53:23.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9429ce:::benchmark_data_smithi194_186548_object56558:head# 2024-04-29T14:53:23.222 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f95f1e4:::benchmark_data_smithi194_186548_object5703:head# 2024-04-29T14:53:23.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f963909:::benchmark_data_smithi194_186548_object57458:head# 2024-04-29T14:53:23.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f964c03:::benchmark_data_smithi194_186548_object41693:head# 2024-04-29T14:53:23.223 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f969272:::benchmark_data_smithi194_186548_object8581:head# 2024-04-29T14:53:23.224 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f974578:::benchmark_data_smithi194_186548_object55482:head# 2024-04-29T14:53:23.224 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f98dd3a:::benchmark_data_smithi194_186548_object1696:head# 2024-04-29T14:53:23.225 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f996144:::benchmark_data_smithi194_186548_object31569:head# 2024-04-29T14:53:23.225 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f99f26f:::benchmark_data_smithi194_186548_object73453:head# 2024-04-29T14:53:23.226 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9a077a:::benchmark_data_smithi194_186548_object22049:head# 2024-04-29T14:53:23.226 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9a7659:::benchmark_data_smithi194_186548_object20417:head# 2024-04-29T14:53:23.226 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9d8ab0:::benchmark_data_smithi194_186548_object3275:head# 2024-04-29T14:53:23.227 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9deaa2:::benchmark_data_smithi194_186548_object42448:head# 2024-04-29T14:53:23.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9e1fd2:::benchmark_data_smithi194_186548_object48396:head# 2024-04-29T14:53:23.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9e2cd8:::benchmark_data_smithi194_186548_object21791:head# 2024-04-29T14:53:23.228 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3f9e3cd2:::benchmark_data_smithi194_186548_object27457:head# 2024-04-29T14:53:23.229 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa02ba0:::benchmark_data_smithi194_186548_object65703:head# 2024-04-29T14:53:23.229 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa06eb2:::benchmark_data_smithi194_186548_object57269:head# 2024-04-29T14:53:23.234 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa468d2:::benchmark_data_smithi194_186548_object27714:head# 2024-04-29T14:53:23.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa69307:::benchmark_data_smithi194_186548_object69383:head# 2024-04-29T14:53:23.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa70dc1:::benchmark_data_smithi194_186548_object57048:head# 2024-04-29T14:53:23.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa71871:::benchmark_data_smithi194_186548_object26829:head# 2024-04-29T14:53:23.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa855a4:::benchmark_data_smithi194_186548_object45391:head# 2024-04-29T14:53:23.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fa8c9c3:::benchmark_data_smithi194_186548_object63956:head# 2024-04-29T14:53:23.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fabbf9f:::benchmark_data_smithi194_186548_object34926:head# 2024-04-29T14:53:23.241 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3faf62bf:::benchmark_data_smithi194_186548_object74233:head# 2024-04-29T14:53:23.242 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fafa6c7:::benchmark_data_smithi194_186548_object48934:head# 2024-04-29T14:53:23.242 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb1030e:::benchmark_data_smithi194_186548_object18583:head# 2024-04-29T14:53:23.245 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb1649d:::benchmark_data_smithi194_186548_object35758:head# 2024-04-29T14:53:23.245 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb18c75:::benchmark_data_smithi194_186548_object10608:head# 2024-04-29T14:53:23.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb18fd4:::benchmark_data_smithi194_186548_object71167:head# 2024-04-29T14:53:23.247 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb4486e:::benchmark_data_smithi194_186548_object21391:head# 2024-04-29T14:53:23.248 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb553db:::benchmark_data_smithi194_186548_object75087:head# 2024-04-29T14:53:23.250 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb5dd4e:::benchmark_data_smithi194_186548_object59322:head# 2024-04-29T14:53:23.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb5eb2c:::benchmark_data_smithi194_186548_object29028:head# 2024-04-29T14:53:23.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb76325:::benchmark_data_smithi194_186548_object7986:head# 2024-04-29T14:53:23.251 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb8268a:::benchmark_data_smithi194_186548_object64057:head# 2024-04-29T14:53:23.254 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb95fff:::benchmark_data_smithi194_186548_object84562:head# 2024-04-29T14:53:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fb9716d:::benchmark_data_smithi194_186548_object3860:head# 2024-04-29T14:53:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbac187:::benchmark_data_smithi194_186548_object17953:head# 2024-04-29T14:53:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbaca18:::benchmark_data_smithi194_186548_object36642:head# 2024-04-29T14:53:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbb9dde:::benchmark_data_smithi194_186548_object77851:head# 2024-04-29T14:53:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbc1ea2:::benchmark_data_smithi194_186548_object31116:head# 2024-04-29T14:53:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbc42ab:::benchmark_data_smithi194_186548_object23202:head# 2024-04-29T14:53:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbc5818:::benchmark_data_smithi194_186548_object83236:head# 2024-04-29T14:53:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbc5a01:::benchmark_data_smithi194_186548_object80175:head# 2024-04-29T14:53:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbc7f91:::benchmark_data_smithi194_186548_object32793:head# 2024-04-29T14:53:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbe1eb3:::benchmark_data_smithi194_186548_object19785:head# 2024-04-29T14:53:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbe4a50:::benchmark_data_smithi194_186548_object46734:head# 2024-04-29T14:53:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbff07b:::benchmark_data_smithi194_186548_object82759:head# 2024-04-29T14:53:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fbff0b5:::benchmark_data_smithi194_186548_object18374:head# 2024-04-29T14:53:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc0769d:::benchmark_data_smithi194_186548_object27927:head# 2024-04-29T14:53:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc0d587:::benchmark_data_smithi194_186548_object44579:head# 2024-04-29T14:53:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc249fc:::benchmark_data_smithi194_186548_object899:head# 2024-04-29T14:53:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc2c8d6:::benchmark_data_smithi194_186548_object71184:head# 2024-04-29T14:53:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc32a22:::benchmark_data_smithi194_186548_object63006:head# 2024-04-29T14:53:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc33c2a:::benchmark_data_smithi194_186548_object32258:head# 2024-04-29T14:53:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc4a1e0:::benchmark_data_smithi194_186548_object22767:head# 2024-04-29T14:53:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc4c213:::benchmark_data_smithi194_186548_object33730:head# 2024-04-29T14:53:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc5c3d7:::benchmark_data_smithi194_186548_object43274:head# 2024-04-29T14:53:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc5c9e3:::benchmark_data_smithi194_186548_object21986:head# 2024-04-29T14:53:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc6e113:::benchmark_data_smithi194_186548_object48370:head# 2024-04-29T14:53:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fc77cad:::benchmark_data_smithi194_186548_object38165:head# 2024-04-29T14:53:23.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcb0a98:::benchmark_data_smithi194_186548_object33225:head# 2024-04-29T14:53:23.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcb4c85:::benchmark_data_smithi194_186548_object40149:head# 2024-04-29T14:53:23.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcb95c4:::benchmark_data_smithi194_186548_object6559:head# 2024-04-29T14:53:23.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcbf6af:::benchmark_data_smithi194_186548_object41434:head# 2024-04-29T14:53:23.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcc2701:::benchmark_data_smithi194_186548_object49226:head# 2024-04-29T14:53:23.273 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcc3f5b:::benchmark_data_smithi194_186548_object14859:head# 2024-04-29T14:53:23.274 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fccb03c:::benchmark_data_smithi194_186548_object7979:head# 2024-04-29T14:53:23.274 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcd4f9a:::benchmark_data_smithi194_186548_object14439:head# 2024-04-29T14:53:23.275 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcde971:::benchmark_data_smithi194_186548_object18149:head# 2024-04-29T14:53:23.275 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcea510:::benchmark_data_smithi194_186548_object37528:head# 2024-04-29T14:53:23.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fcf314f:::benchmark_data_smithi194_186548_object4926:head# 2024-04-29T14:53:23.276 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd2912c:::benchmark_data_smithi194_186548_object85210:head# 2024-04-29T14:53:23.277 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd323f0:::benchmark_data_smithi194_186548_object71056:head# 2024-04-29T14:53:23.279 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd43f26:::benchmark_data_smithi194_186548_object57607:head# 2024-04-29T14:53:23.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd477c3:::benchmark_data_smithi194_186548_object28054:head# 2024-04-29T14:53:23.280 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd50f3a:::benchmark_data_smithi194_186548_object75236:head# 2024-04-29T14:53:23.281 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd52307:::benchmark_data_smithi194_186548_object62553:head# 2024-04-29T14:53:23.281 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd62216:::benchmark_data_smithi194_186548_object66038:head# 2024-04-29T14:53:23.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd64eff:::benchmark_data_smithi194_186548_object40254:head# 2024-04-29T14:53:23.282 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd66bd4:::benchmark_data_smithi194_186548_object28584:head# 2024-04-29T14:53:23.283 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd673d0:::benchmark_data_smithi194_186548_object34048:head# 2024-04-29T14:53:23.283 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd789c3:::benchmark_data_smithi194_186548_object65536:head# 2024-04-29T14:53:23.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd820fc:::benchmark_data_smithi194_186548_object71552:head# 2024-04-29T14:53:23.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fd8f101:::benchmark_data_smithi194_186548_object24700:head# 2024-04-29T14:53:23.289 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fda3a7c:::benchmark_data_smithi194_186548_object59850:head# 2024-04-29T14:53:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fdad9c5:::benchmark_data_smithi194_186548_object76863:head# 2024-04-29T14:53:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fdbd481:::benchmark_data_smithi194_186548_object53706:head# 2024-04-29T14:53:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fddcf99:::benchmark_data_smithi194_186548_object14220:head# 2024-04-29T14:53:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fde4b1c:::benchmark_data_smithi194_186548_object34153:head# 2024-04-29T14:53:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fdf3010:::benchmark_data_smithi194_186548_object50087:head# 2024-04-29T14:53:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fdf953f:::benchmark_data_smithi194_186548_object256:head# 2024-04-29T14:53:23.290 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fdfa3f1:::benchmark_data_smithi194_186548_object78380:head# 2024-04-29T14:53:23.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fdfef52:::benchmark_data_smithi194_186548_object10429:head# 2024-04-29T14:53:23.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe07709:::benchmark_data_smithi194_186548_object73782:head# 2024-04-29T14:53:23.291 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe1047b:::benchmark_data_smithi194_186548_object21118:head# 2024-04-29T14:53:23.293 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe1c070:::benchmark_data_smithi194_186548_object8201:head# 2024-04-29T14:53:23.298 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe1e35e:::benchmark_data_smithi194_186548_object21814:head# 2024-04-29T14:53:23.298 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe1f622:::benchmark_data_smithi194_186548_object75491:head# 2024-04-29T14:53:23.298 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe1fd7e:::benchmark_data_smithi194_186548_object2544:head# 2024-04-29T14:53:23.298 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe2608b:::benchmark_data_smithi194_186548_object74579:head# 2024-04-29T14:53:23.299 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe2cc4c:::benchmark_data_smithi194_186548_object13783:head# 2024-04-29T14:53:23.299 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe2d0be:::benchmark_data_smithi194_186548_object63933:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe2ee6e:::benchmark_data_smithi194_186548_object77314:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe4234c:::benchmark_data_smithi194_186548_object8548:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe576d4:::benchmark_data_smithi194_186548_object30261:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe70cb1:::benchmark_data_smithi194_186548_object78125:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe7ada1:::benchmark_data_smithi194_186548_object18295:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe81970:::benchmark_data_smithi194_186548_object56094:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fe98989:::benchmark_data_smithi194_186548_object11531:head# 2024-04-29T14:53:23.303 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3febdd8d:::benchmark_data_smithi194_186548_object3148:head# 2024-04-29T14:53:23.304 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3febf1cd:::benchmark_data_smithi194_186548_object66153:head# 2024-04-29T14:53:23.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fece83d:::benchmark_data_smithi194_186548_object15891:head# 2024-04-29T14:53:23.307 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fed169b:::benchmark_data_smithi194_186548_object8172:head# 2024-04-29T14:53:23.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fedcd47:::benchmark_data_smithi194_186548_object47655:head# 2024-04-29T14:53:23.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff186a2:::benchmark_data_smithi194_186548_object36772:head# 2024-04-29T14:53:23.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff3a4df:::benchmark_data_smithi194_186548_object8399:head# 2024-04-29T14:53:23.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff47600:::benchmark_data_smithi194_186548_object25549:head# 2024-04-29T14:53:23.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff6f9c7:::benchmark_data_smithi194_186548_object71183:head# 2024-04-29T14:53:23.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff7fb3a:::benchmark_data_smithi194_186548_object2198:head# 2024-04-29T14:53:23.308 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff84865:::benchmark_data_smithi194_186548_object27309:head# 2024-04-29T14:53:23.309 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff9768d:::benchmark_data_smithi194_186548_object46057:head# 2024-04-29T14:53:23.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff9b687:::benchmark_data_smithi194_186548_object62903:head# 2024-04-29T14:53:23.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ff9e750:::benchmark_data_smithi194_186548_object39832:head# 2024-04-29T14:53:23.310 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ffcb00a:::benchmark_data_smithi194_186548_object29755:head# 2024-04-29T14:53:23.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ffccad9:::benchmark_data_smithi194_186548_object49055:head# 2024-04-29T14:53:23.311 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ffdc576:::benchmark_data_smithi194_186548_object12482:head# 2024-04-29T14:53:23.312 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ffdf668:::benchmark_data_smithi194_186548_object59044:head# 2024-04-29T14:53:23.312 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ffe9687:::benchmark_data_smithi194_186548_object34149:head# 2024-04-29T14:53:23.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3ffea926:::benchmark_data_smithi194_186548_object17469:head# 2024-04-29T14:53:23.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fff45b2:::benchmark_data_smithi194_186548_object26098:head# 2024-04-29T14:53:23.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fff7960:::benchmark_data_smithi194_186548_object39676:head# 2024-04-29T14:53:23.316 INFO:teuthology.orchestra.run.smithi194.stderr:Read #8:3fffef05:::benchmark_data_smithi194_186548_object12765:head# 2024-04-29T14:53:23.317 INFO:teuthology.orchestra.run.smithi194.stderr:Export successful 2024-04-29T14:53:23.320 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:23.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:23.379 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:23 smithi121 ceph-mon[44533]: pgmap v2612: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 18588/213194 objects degraded (8.719%); 9.2 MiB/s, 147 objects/s recovering 2024-04-29T14:53:23.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:23.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:23 smithi194 ceph-mon[44089]: pgmap v2612: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 18588/213194 objects degraded (8.719%); 9.2 MiB/s, 147 objects/s recovering 2024-04-29T14:53:23.472 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:23.506 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:23.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:23.607 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 16 110200 110184 96.9721 67 0.0252974 0.0103097 2024-04-29T14:53:23.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:23.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:23 smithi040 ceph-mon[39581]: pgmap v2612: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.5 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 18588/213194 objects degraded (8.719%); 9.2 MiB/s, 147 objects/s recovering 2024-04-29T14:53:23.647 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:23.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:53:24.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:24.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:24.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:24.151 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:24.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:24.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:53:24.294 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:24.346 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:24.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3736632319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:53:24.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3736632319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:53:24.403 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:24.504 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:24.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3736632319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:53:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:24.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:24.542 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.9 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-9 --journal-path /var/lib/ceph/osd/ceph-9/journal --force --op remove --pgid 8.c 2024-04-29T14:53:24.607 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 111076 111060 96.3857 54.75 0.00458574 0.0103514 2024-04-29T14:53:24.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:24.637 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:24.866 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.9/config 2024-04-29T14:53:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:25.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:25.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:25.195 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:25.222 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:25.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:25.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:25.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:25.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:25.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:25 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:25.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3736632319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:53:25.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:25 smithi040 ceph-mon[39581]: osdmap e1035: 12 total, 11 up, 9 in 2024-04-29T14:53:25.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:25 smithi040 ceph-mon[39581]: pgmap v2614: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 915 op/s; 18366/215604 objects degraded (8.518%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:53:25.358 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:25.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:25.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:25.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:25.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:25.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:25 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:25.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3736632319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:53:25.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:25 smithi121 ceph-mon[44533]: osdmap e1035: 12 total, 11 up, 9 in 2024-04-29T14:53:25.522 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:25 smithi121 ceph-mon[44533]: pgmap v2614: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 915 op/s; 18366/215604 objects degraded (8.518%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:53:25.523 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:25.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:25.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:25.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:25 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3736632319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:53:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:25 smithi194 ceph-mon[44089]: osdmap e1035: 12 total, 11 up, 9 in 2024-04-29T14:53:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:25 smithi194 ceph-mon[44089]: pgmap v2614: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 915 op/s; 18366/215604 objects degraded (8.518%); 8.8 MiB/s, 140 objects/s recovering 2024-04-29T14:53:25.600 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:53:25.600 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:53:25.607 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 112026 112010 95.8787 59.375 0.012297 0.010428 2024-04-29T14:53:25.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:25.986 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:25.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:26.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:53:26.136 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:26.349 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:53:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:26.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:53:26.448 INFO:teuthology.orchestra.run.smithi194.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:53:26.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:26.487 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:26.607 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 113111 113095 95.4993 67.8125 0.0144471 0.01047 2024-04-29T14:53:26.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:26 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 17976/220746 objects degraded (8.143%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:26.723 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:26 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 17976/220746 objects degraded (8.143%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:26 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 17976/220746 objects degraded (8.143%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:26.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:26.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:27.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:27.099 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:27.129 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:27.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:27.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:27.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:53:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:27.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:27.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:27.607 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 16 114109 114093 95.0577 62.375 0.00378947 0.0105055 2024-04-29T14:53:27.619 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:27.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:27 smithi194 ceph-mon[44089]: pgmap v2615: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 17976/220746 objects degraded (8.143%); 11 MiB/s, 170 objects/s recovering 2024-04-29T14:53:27.834 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:27.836 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:27 smithi040 ceph-mon[39581]: pgmap v2615: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 17976/220746 objects degraded (8.143%); 11 MiB/s, 170 objects/s recovering 2024-04-29T14:53:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:27.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:27 smithi121 ceph-mon[44533]: pgmap v2615: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 67 MiB/s wr, 1.07k op/s; 17976/220746 objects degraded (8.143%); 11 MiB/s, 170 objects/s recovering 2024-04-29T14:53:27.925 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:53:27.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:28.023 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:28.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:53:28.093 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:53:28.134 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:53:28.141 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:28.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:28.177 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:53:28.208 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:53:28.241 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:53:28.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:28.275 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:53:28.281 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:28.307 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:53:28.335 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:53:28.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:28.373 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:53:28.413 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:53:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:28.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:28.438 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:28.453 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:53:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:53:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:53:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:53:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:53:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:53:28.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:28.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 114983 114967 94.5257 54.625 0.0134727 0.0105777 2024-04-29T14:53:28.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1616542651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:28.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:28.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:28.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:28 smithi194 ceph-mon[44089]: pgmap v2616: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s; 17813/223384 objects degraded (7.974%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:53:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1616542651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:28.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:28 smithi040 ceph-mon[39581]: pgmap v2616: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s; 17813/223384 objects degraded (7.974%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:53:29.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1616542651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:28 smithi121 ceph-mon[44533]: pgmap v2616: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 904 op/s; 17813/223384 objects degraded (7.974%); 10 MiB/s, 161 objects/s recovering 2024-04-29T14:53:29.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 115972 115956 94.1008 61.8125 0.00461635 0.0106252 2024-04-29T14:53:29.634 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:29.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:29.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:53:29.792 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:30.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.047 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.047 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.28144.0:116439 8.1 e1035/1026 8:87bf6e12:::benchmark_data_smithi194_186548_object116438:head v 1035'7356, mlcod=1017'992)", 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:53:30.137755+0000", 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.011524085, 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011554619, 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:53:30.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.137755+0000", 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.137755+0000", 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.137758+0000", 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1669999999999999e-06 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.137809+0000", 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0596999999999997e-05 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.137823+0000", 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3959e-05 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.137835+0000", 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1980999999999999e-05 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.140639+0000", 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028039530000000001 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.140672+0000", 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3183999999999999e-05 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(8.1 1035/1026 [PushOp(8:821a8eb3:::benchmark_data_smithi194_186548_object15709:head, version: 1017'994, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:821a8eb3:::benchmark_data_smithi194_186548_object15709:head@1017'994, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-29T14:53:30.158 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:53:30.146176+0000", 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0031034489999999999, 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00323836, 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146176+0000", 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146176+0000", 2024-04-29T14:53:30.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146178+0000", 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9120000000000001e-06 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146228+0000", 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.045e-05 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146251+0000", 2024-04-29T14:53:30.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3099999999999999e-05 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146268+0000", 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6305999999999999e-05 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.148051+0000", 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017834630000000001 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.148067+0000", 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5818e-05 2024-04-29T14:53:30.161 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.28144.0:116441 8.1 e1035/1026)", 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:53:30.146604+0000", 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0026749199999999999, 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028782230000000001, 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "queued for pg", 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:53:30.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146604+0000", 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146604+0000", 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146606+0000", 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3370000000000001e-06 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146662+0000", 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6487999999999999e-05 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146677+0000", 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4622e-05 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.146683+0000", 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4389999999999997e-06 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.164 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.28144.0:116437 8.1 e1035/1026 8:8da7ea24:::benchmark_data_smithi194_186548_object116436:head v 1035'7355, mlcod=1017'992)", 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:53:30.134790+0000", 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.014489451, 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014748343000000001, 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134790+0000", 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134790+0000", 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134792+0000", 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.711e-06 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134837+0000", 2024-04-29T14:53:30.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.579e-05 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134851+0000", 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3641e-05 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134859+0000", 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0900000000000005e-06 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134932+0000", 2024-04-29T14:53:30.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2795999999999999e-05 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:53:30.134957+0000", 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4819000000000001e-05 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 4 2024-04-29T14:53:30.168 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:53:30.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 116856 116840 93.6027 55.25 0.00473475 0.0106819 2024-04-29T14:53:31.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:30 smithi194 ceph-mon[44089]: pgmap v2617: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 982 op/s; 17592/228166 objects degraded (7.710%); 9.6 MiB/s, 154 objects/s recovering 2024-04-29T14:53:31.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:30 smithi040 ceph-mon[39581]: pgmap v2617: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 982 op/s; 17592/228166 objects degraded (7.710%); 9.6 MiB/s, 154 objects/s recovering 2024-04-29T14:53:31.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:30 smithi121 ceph-mon[44533]: pgmap v2617: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 61 MiB/s wr, 982 op/s; 17592/228166 objects degraded (7.710%); 9.6 MiB/s, 154 objects/s recovering 2024-04-29T14:53:31.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 117836 117820 93.1931 61.25 0.0167125 0.0107286 2024-04-29T14:53:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:53:31.643 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:53:32.101 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:53:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:32.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:53:32.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:32.247 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:32.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:32.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:32.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:32.426 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:32.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:32.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:32.557 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:32.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:53:32.604965 min lat: 0.00105393 max lat: 17.3914 avg lat: 0.0107741 2024-04-29T14:53:32.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:53:32.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 118820 118804 92.797 61.5 0.00506159 0.0107741 2024-04-29T14:53:32.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:32 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 17031/232934 objects degraded (7.312%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:32.641 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:32.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:32 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 17031/232934 objects degraded (7.312%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:32.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:32 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 17031/232934 objects degraded (7.312%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:33.467 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:53:33.556 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:33.608 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 119739 119723 92.3596 57.4375 0.0040346 0.0108252 2024-04-29T14:53:33.644 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:53:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:33.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:33.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:33.806 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:33 smithi121 ceph-mon[44533]: pgmap v2618: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 986 op/s; 17031/232934 objects degraded (7.312%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:53:33.806 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3913732942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:33.809 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:33.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:33 smithi194 ceph-mon[44089]: pgmap v2618: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 986 op/s; 17031/232934 objects degraded (7.312%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:53:33.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3913732942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:33.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:33.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:33.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:33 smithi040 ceph-mon[39581]: pgmap v2618: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 986 op/s; 17031/232934 objects degraded (7.312%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:53:33.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3913732942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:33.939 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:33.980 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:34.609 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 120641 120625 91.9208 56.375 0.00702314 0.0108771 2024-04-29T14:53:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:34 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3913732942' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:34 smithi194 ceph-mon[44089]: osdmap e1036: 12 total, 11 up, 9 in 2024-04-29T14:53:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:34 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3913732942' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:34 smithi040 ceph-mon[39581]: osdmap e1036: 12 total, 11 up, 9 in 2024-04-29T14:53:34.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:34 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:34.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3913732942' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:34.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:34 smithi121 ceph-mon[44533]: osdmap e1036: 12 total, 11 up, 9 in 2024-04-29T14:53:35.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 16 121625 121609 91.5542 61.5 0.00839177 0.0109205 2024-04-29T14:53:35.756 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:35.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:35 smithi194 ceph-mon[44089]: pgmap v2620: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 987 op/s; 16815/235354 objects degraded (7.145%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:53:35.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:35 smithi040 ceph-mon[39581]: pgmap v2620: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 987 op/s; 16815/235354 objects degraded (7.145%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:53:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:35 smithi121 ceph-mon[44533]: pgmap v2620: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 62 MiB/s wr, 987 op/s; 16815/235354 objects degraded (7.145%); 9.7 MiB/s, 155 objects/s recovering 2024-04-29T14:53:36.609 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 122587 122571 91.18 60.125 0.00677375 0.0109641 2024-04-29T14:53:36.750 INFO:teuthology.orchestra.run.smithi194.stdout: marking collection for removal 2024-04-29T14:53:36.750 INFO:teuthology.orchestra.run.smithi194.stdout:setting '_remove' omap key 2024-04-29T14:53:36.750 INFO:teuthology.orchestra.run.smithi194.stdout:finish_remove_pgs 8.c_head removing 8.c 2024-04-29T14:53:36.888 INFO:teuthology.orchestra.run.smithi194.stdout:Remove successful 2024-04-29T14:53:37.009 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:53:37.049 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:37.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:37.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:37.204 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:37.229 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:37.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:37.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:53:37.381 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:37.515 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:37 smithi040 ceph-mon[39581]: pgmap v2621: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 60 MiB/s wr, 959 op/s; 16441/239950 objects degraded (6.852%); 9.6 MiB/s, 153 objects/s recovering 2024-04-29T14:53:37.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:37 smithi194 ceph-mon[44089]: pgmap v2621: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 60 MiB/s wr, 959 op/s; 16441/239950 objects degraded (6.852%); 9.6 MiB/s, 153 objects/s recovering 2024-04-29T14:53:37.609 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 16 123446 123430 90.7389 53.6875 0.00418183 0.011015 2024-04-29T14:53:37.631 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:53:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:37.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:53:37.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:53:37.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:37.768 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:37 smithi121 ceph-mon[44533]: pgmap v2621: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 60 MiB/s wr, 959 op/s; 16441/239950 objects degraded (6.852%); 9.6 MiB/s, 153 objects/s recovering 2024-04-29T14:53:37.769 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:37.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:37.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:37.897 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:37.947 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:38.609 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 124402 124386 90.3786 59.75 0.0135471 0.0110629 2024-04-29T14:53:38.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:38 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 16441/239950 objects degraded (6.852%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:38.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:38 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 16441/239950 objects degraded (6.852%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:38.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:38 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 16441/239950 objects degraded (6.852%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:39.473 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:53:39.499 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:53:39.530 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:53:39.559 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:53:39.591 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:53:39.609 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 125457 125441 90.0976 65.9375 0.016604 0.011097 2024-04-29T14:53:39.632 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:53:39.657 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:53:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:53:39.718 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:39.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:39.728 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:53:39.758 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:53:39.793 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:53:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:53:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:53:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:53:39.794 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:53:39.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:39 smithi040 ceph-mon[39581]: pgmap v2622: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 16282/242330 objects degraded (6.719%); 9.6 MiB/s, 153 objects/s recovering 2024-04-29T14:53:39.809 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/615767070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:39.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:39 smithi194 ceph-mon[44089]: pgmap v2622: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 16282/242330 objects degraded (6.719%); 9.6 MiB/s, 153 objects/s recovering 2024-04-29T14:53:39.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/615767070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:39.826 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:39.874 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:39.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:39 smithi121 ceph-mon[44533]: pgmap v2622: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 16282/242330 objects degraded (6.719%); 9.6 MiB/s, 153 objects/s recovering 2024-04-29T14:53:39.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/615767070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:39.911 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:39.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:40.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:40.054 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:40.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:40.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:40.183 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:53:40.227 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:40.609 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 126494 126478 89.8102 64.8125 0.00949663 0.0111323 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:40.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:41.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:41 smithi040 ceph-mon[39581]: pgmap v2623: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 16078/247128 objects degraded (6.506%); 9.5 MiB/s, 151 objects/s recovering 2024-04-29T14:53:41.587 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:53:41.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:41.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:53:41.609 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 127518 127502 89.5202 64 0.026925 0.011169 2024-04-29T14:53:41.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:41.727 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:41.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:41.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:41.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:41 smithi194 ceph-mon[44089]: pgmap v2623: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 16078/247128 objects degraded (6.506%); 9.5 MiB/s, 151 objects/s recovering 2024-04-29T14:53:41.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:41.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:41 smithi121 ceph-mon[44533]: pgmap v2623: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.5 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 59 MiB/s wr, 947 op/s; 16078/247128 objects degraded (6.506%); 9.5 MiB/s, 151 objects/s recovering 2024-04-29T14:53:41.906 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:41.941 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:41.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:42.046 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:53:42.161 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:53:42.610 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 90 1 128555 128554 89.2561 65.75 0.0125643 0.0112022 2024-04-29T14:53:43.610 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 90.0183 2024-04-29T14:53:43.610 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 128555 2024-04-29T14:53:43.610 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:53:43.610 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:53:43.610 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 89.2561 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 76.7611 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 352.25 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 0 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1428 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 1228.19 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 5636 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 0 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.0112022 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.194051 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 17.3914 2024-04-29T14:53:43.611 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.00105393 2024-04-29T14:53:43.616 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:53:43.670 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:53:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:43 smithi194 ceph-mon[44089]: pgmap v2624: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 957 op/s; 15535/252092 objects degraded (6.162%); 9.3 MiB/s, 149 objects/s recovering 2024-04-29T14:53:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:43 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 15535/252092 objects degraded (6.162%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2027377069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:43 smithi194 ceph-mon[44089]: from='client.37997 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:43.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:43 smithi040 ceph-mon[39581]: pgmap v2624: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 957 op/s; 15535/252092 objects degraded (6.162%); 9.3 MiB/s, 149 objects/s recovering 2024-04-29T14:53:43.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:43 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 15535/252092 objects degraded (6.162%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:43.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2027377069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:43.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:43 smithi040 ceph-mon[39581]: from='client.37997 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:43.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:43 smithi121 ceph-mon[44533]: pgmap v2624: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 60 MiB/s wr, 957 op/s; 15535/252092 objects degraded (6.162%); 9.3 MiB/s, 149 objects/s recovering 2024-04-29T14:53:43.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:43 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 15535/252092 objects degraded (6.162%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:43.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2027377069' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:43.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:43 smithi121 ceph-mon[44533]: from='client.37997 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:53:44.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:53:44.454 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:44.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:44.465 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.9 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-9 --journal-path /var/lib/ceph/osd/ceph-9/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.c.9 2024-04-29T14:53:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:53:44.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:44.602 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:44.670 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:44.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:44.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:44 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:44.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:44 smithi194 ceph-mon[44089]: from='client.37997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:53:44.683 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:44 smithi194 ceph-mon[44089]: osdmap e1037: 12 total, 11 up, 9 in 2024-04-29T14:53:44.779 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:44.836 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.9/config 2024-04-29T14:53:44.854 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:44.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:44 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:44.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:44 smithi040 ceph-mon[39581]: from='client.37997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:53:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:44 smithi040 ceph-mon[39581]: osdmap e1037: 12 total, 11 up, 9 in 2024-04-29T14:53:44.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:44 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:53:44.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:44 smithi121 ceph-mon[44533]: from='client.37997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:53:44.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:44 smithi121 ceph-mon[44533]: osdmap e1037: 12 total, 11 up, 9 in 2024-04-29T14:53:44.893 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:44.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:45.007 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:45.039 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:45.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:45 smithi194 ceph-mon[44089]: pgmap v2626: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 58 MiB/s wr, 939 op/s; 15244/253956 objects degraded (6.003%); 9.8 MiB/s, 157 objects/s recovering 2024-04-29T14:53:45.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:45.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:45 smithi040 ceph-mon[39581]: pgmap v2626: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 58 MiB/s wr, 939 op/s; 15244/253956 objects degraded (6.003%); 9.8 MiB/s, 157 objects/s recovering 2024-04-29T14:53:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:45 smithi121 ceph-mon[44533]: pgmap v2626: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.8 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 58 MiB/s wr, 939 op/s; 15244/253956 objects degraded (6.003%); 9.8 MiB/s, 157 objects/s recovering 2024-04-29T14:53:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:46.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:46.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:46.346 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:46 smithi040 ceph-mon[39581]: osdmap e1038: 12 total, 11 up, 9 in 2024-04-29T14:53:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:46 smithi040 ceph-mon[39581]: 8.d scrub starts 2024-04-29T14:53:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:46 smithi040 ceph-mon[39581]: pgmap v2628: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.8 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 127 B/s rd, 56 MiB/s wr, 1k op/s; 14736/254946 objects degraded (5.780%); 12 MiB/s, 193 objects/s recovering 2024-04-29T14:53:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:46 smithi194 ceph-mon[44089]: osdmap e1038: 12 total, 11 up, 9 in 2024-04-29T14:53:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:46 smithi194 ceph-mon[44089]: 8.d scrub starts 2024-04-29T14:53:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:46 smithi194 ceph-mon[44089]: pgmap v2628: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.8 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 127 B/s rd, 56 MiB/s wr, 1k op/s; 14736/254946 objects degraded (5.780%); 12 MiB/s, 193 objects/s recovering 2024-04-29T14:53:47.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:46 smithi121 ceph-mon[44533]: osdmap e1038: 12 total, 11 up, 9 in 2024-04-29T14:53:47.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:46 smithi121 ceph-mon[44533]: 8.d scrub starts 2024-04-29T14:53:47.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:46 smithi121 ceph-mon[44533]: pgmap v2628: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.8 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 127 B/s rd, 56 MiB/s wr, 1k op/s; 14736/254946 objects degraded (5.780%); 12 MiB/s, 193 objects/s recovering 2024-04-29T14:53:48.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:47 smithi194 ceph-mon[44089]: 8.d scrub ok 2024-04-29T14:53:48.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:47 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 14736/254946 objects degraded (5.780%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:48.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:47 smithi040 ceph-mon[39581]: 8.d scrub ok 2024-04-29T14:53:48.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:47 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 14736/254946 objects degraded (5.780%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:48.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:47 smithi121 ceph-mon[44533]: 8.d scrub ok 2024-04-29T14:53:48.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:47 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 14736/254946 objects degraded (5.780%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:48.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:53:48.570 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:48.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:48.584 INFO:teuthology.orchestra.run.smithi194.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:53:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:48.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:48.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:48.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:48.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:48 smithi194 ceph-mon[44089]: pgmap v2629: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 39 MiB/s wr, 829 op/s; 14511/253832 objects degraded (5.717%); 12 MiB/s, 195 objects/s recovering 2024-04-29T14:53:48.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:48.906 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:49.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:48 smithi040 ceph-mon[39581]: pgmap v2629: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 39 MiB/s wr, 829 op/s; 14511/253832 objects degraded (5.717%); 12 MiB/s, 195 objects/s recovering 2024-04-29T14:53:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:48 smithi121 ceph-mon[44533]: pgmap v2629: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.7 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 39 MiB/s wr, 829 op/s; 14511/253832 objects degraded (5.717%); 12 MiB/s, 195 objects/s recovering 2024-04-29T14:53:49.150 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:53:49.175 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:53:49.201 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:53:49.223 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:53:49.254 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:53:49.281 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:53:49.307 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:53:49.337 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:53:49.400 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:53:49.439 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:53:49.471 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:53:49.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:53:49.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:53:49.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:53:49.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:53:49.471 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:53:49.472 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:53:49.472 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:53:49.472 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:53:49.472 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:53:49.472 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:53:49.472 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:53:49.472 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:53:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3841158135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:49.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3841158135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3841158135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:50.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:50.769 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:50.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:50.877 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:53:50.920 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:51.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.060 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:50 smithi194 ceph-mon[44089]: pgmap v2630: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 842 op/s; 14272/248656 objects degraded (5.740%); 9.9 MiB/s, 157 objects/s recovering 2024-04-29T14:53:51.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:50 smithi040 ceph-mon[39581]: pgmap v2630: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 842 op/s; 14272/248656 objects degraded (5.740%); 9.9 MiB/s, 157 objects/s recovering 2024-04-29T14:53:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:51.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:50 smithi121 ceph-mon[44533]: pgmap v2630: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 7.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 20 MiB/s wr, 842 op/s; 14272/248656 objects degraded (5.740%); 9.9 MiB/s, 157 objects/s recovering 2024-04-29T14:53:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:53:51.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:53:51.701 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:51.701 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:51.701 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:51 smithi194 ceph-mon[44089]: osdmap e1039: 12 total, 11 up, 9 in 2024-04-29T14:53:51.701 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:51 smithi194 ceph-mon[44089]: 8.e scrub starts 2024-04-29T14:53:52.110 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:52.110 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:52.110 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:51 smithi040 ceph-mon[39581]: osdmap e1039: 12 total, 11 up, 9 in 2024-04-29T14:53:52.110 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:51 smithi040 ceph-mon[39581]: 8.e scrub starts 2024-04-29T14:53:52.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:52.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:52.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:51 smithi121 ceph-mon[44533]: osdmap e1039: 12 total, 11 up, 9 in 2024-04-29T14:53:52.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:51 smithi121 ceph-mon[44533]: 8.e scrub starts 2024-04-29T14:53:52.775 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:53:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:53:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:53:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:53:52.775 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:53:52.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:52 smithi040 ceph-mon[39581]: pgmap v2632: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 12 MiB/s wr, 1.04k op/s; 13633/243486 objects degraded (5.599%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:53:52.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:52 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 14272/248656 objects degraded (5.740%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:53.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:52 smithi194 ceph-mon[44089]: pgmap v2632: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 12 MiB/s wr, 1.04k op/s; 13633/243486 objects degraded (5.599%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:53:53.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:52 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 14272/248656 objects degraded (5.740%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:53.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:52 smithi121 ceph-mon[44533]: pgmap v2632: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 127 B/s rd, 12 MiB/s wr, 1.04k op/s; 13633/243486 objects degraded (5.599%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:53:53.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:52 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 14272/248656 objects degraded (5.740%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:53.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:53:53.765 INFO:teuthology.orchestra.run.smithi194.stdout:get_pg_num_history pg_num_history pg_num_history(e1019 pg_nums {1={14=1},7={655=16},8={1011=16}} deleted_pools 1009,7) 2024-04-29T14:53:53.765 INFO:teuthology.orchestra.run.smithi194.stdout:Importing pgid 8.c 2024-04-29T14:53:53.765 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3001a9c1:::benchmark_data_smithi194_186548_object20385:head# 2024-04-29T14:53:53.765 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.765 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3002e01d:::benchmark_data_smithi194_186548_object43396:head# 2024-04-29T14:53:53.765 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.765 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30031bd7:::benchmark_data_smithi194_186548_object85221:head# 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30032756:::benchmark_data_smithi194_186548_object75488:head# 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3004c823:::benchmark_data_smithi194_186548_object84981:head# 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30054e5e:::benchmark_data_smithi194_186548_object50697:head# 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3005da6d:::benchmark_data_smithi194_186548_object5106:head# 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3008989b:::benchmark_data_smithi194_186548_object67505:head# 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3008fa74:::benchmark_data_smithi194_186548_object26989:head# 2024-04-29T14:53:53.766 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:300a4e0e:::benchmark_data_smithi194_186548_object73691:head# 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:300baae3:::benchmark_data_smithi194_186548_object54938:head# 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:300d8b52:::benchmark_data_smithi194_186548_object27233:head# 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:300e002e:::benchmark_data_smithi194_186548_object59932:head# 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301062c2:::benchmark_data_smithi194_186548_object40164:head# 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3010b94c:::benchmark_data_smithi194_186548_object15048:head# 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3011bd7f:::benchmark_data_smithi194_186548_object44525:head# 2024-04-29T14:53:53.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3011db83:::benchmark_data_smithi194_186548_object43878:head# 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3013568f:::benchmark_data_smithi194_186548_object65175:head# 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30146ccb:::benchmark_data_smithi194_186548_object42660:head# 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30155c3b:::benchmark_data_smithi194_186548_object56698:head# 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301710c9:::benchmark_data_smithi194_186548_object56908:head# 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3017170e:::benchmark_data_smithi194_186548_object33389:head# 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3017633f:::benchmark_data_smithi194_186548_object75288:head# 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30179c5f:::benchmark_data_smithi194_186548_object73705:head# 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301a1f35:::benchmark_data_smithi194_186548_object73186:head# 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301b54cb:::benchmark_data_smithi194_186548_object17569:head# 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301bc4f8:::benchmark_data_smithi194_186548_object22895:head# 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301cf017:::benchmark_data_smithi194_186548_object8327:head# 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301d3a0c:::benchmark_data_smithi194_186548_object14001:head# 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301e04df:::benchmark_data_smithi194_186548_object21756:head# 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:301e3f1a:::benchmark_data_smithi194_186548_object59275:head# 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3020e805:::benchmark_data_smithi194_186548_object73480:head# 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3023a4ef:::benchmark_data_smithi194_186548_object62445:head# 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:302442f4:::benchmark_data_smithi194_186548_object13781:head# 2024-04-29T14:53:53.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30244e36:::benchmark_data_smithi194_186548_object20975:head# 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30249d96:::benchmark_data_smithi194_186548_object39106:head# 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3024d82a:::benchmark_data_smithi194_186548_object26942:head# 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:302510f2:::benchmark_data_smithi194_186548_object46162:head# 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3025c475:::benchmark_data_smithi194_186548_object64863:head# 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3025e44b:::benchmark_data_smithi194_186548_object22759:head# 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30275198:::benchmark_data_smithi194_186548_object25641:head# 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3027c1b4:::benchmark_data_smithi194_186548_object45871:head# 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3028142a:::benchmark_data_smithi194_186548_object56958:head# 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30295264:::benchmark_data_smithi194_186548_object76261:head# 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:302aa7c5:::benchmark_data_smithi194_186548_object15931:head# 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:302ffd6a:::benchmark_data_smithi194_186548_object74282:head# 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30305a80:::benchmark_data_smithi194_186548_object69495:head# 2024-04-29T14:53:53.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30326251:::benchmark_data_smithi194_186548_object67737:head# 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3032a4c5:::benchmark_data_smithi194_186548_object30090:head# 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3032aa50:::benchmark_data_smithi194_186548_object1674:head# 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30342c6d:::benchmark_data_smithi194_186548_object23929:head# 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303474ff:::benchmark_data_smithi194_186548_object51278:head# 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3034e2f5:::benchmark_data_smithi194_186548_object81398:head# 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30352c6d:::benchmark_data_smithi194_186548_object21805:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30367b68:::benchmark_data_smithi194_186548_object39511:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30377b10:::benchmark_data_smithi194_186548_object62663:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30391911:::benchmark_data_smithi194_186548_object85068:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3039768e:::benchmark_data_smithi194_186548_object11088:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3039e15d:::benchmark_data_smithi194_186548_object55976:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303a265b:::benchmark_data_smithi194_186548_object31794:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303a53a8:::benchmark_data_smithi194_186548_object44519:head# 2024-04-29T14:53:53.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303d26eb:::benchmark_data_smithi194_186548_object2589:head# 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303daf2f:::benchmark_data_smithi194_186548_object79914:head# 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303dff45:::benchmark_data_smithi194_186548_object64595:head# 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303e6956:::benchmark_data_smithi194_186548_object46926:head# 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303e80b7:::benchmark_data_smithi194_186548_object75469:head# 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303ede86:::benchmark_data_smithi194_186548_object51208:head# 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303f4f23:::benchmark_data_smithi194_186548_object59160:head# 2024-04-29T14:53:53.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:303f881d:::benchmark_data_smithi194_186548_object85185:head# 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30402b19:::benchmark_data_smithi194_186548_object24881:head# 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30403988:::benchmark_data_smithi194_186548_object58815:head# 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30407e5e:::benchmark_data_smithi194_186548_object2513:head# 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3040dfca:::benchmark_data_smithi194_186548_object69051:head# 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3042d061:::benchmark_data_smithi194_186548_object9235:head# 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30437437:::benchmark_data_smithi194_186548_object16095:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30449994:::benchmark_data_smithi194_186548_object17740:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304526c8:::benchmark_data_smithi194_186548_object33960:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30456894:::benchmark_data_smithi194_186548_object36112:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3045a224:::benchmark_data_smithi194_186548_object18352:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30469a19:::benchmark_data_smithi194_186548_object35151:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3047160d:::benchmark_data_smithi194_186548_object66445:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30471f20:::benchmark_data_smithi194_186548_object28449:head# 2024-04-29T14:53:53.777 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3047f5a9:::benchmark_data_smithi194_186548_object36059:head# 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30484c63:::benchmark_data_smithi194_186548_object81763:head# 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30498a61:::benchmark_data_smithi194_186548_object55960:head# 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304a6762:::benchmark_data_smithi194_186548_object22480:head# 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304a7e04:::benchmark_data_smithi194_186548_object391:head# 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304b9bef:::benchmark_data_smithi194_186548_object25484:head# 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.778 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304c467f:::benchmark_data_smithi194_186548_object13432:head# 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304d6b8e:::benchmark_data_smithi194_186548_object32154:head# 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304e0ea5:::benchmark_data_smithi194_186548_object72429:head# 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304e2542:::benchmark_data_smithi194_186548_object990:head# 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304e3839:::benchmark_data_smithi194_186548_object26899:head# 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304e8910:::benchmark_data_smithi194_186548_object71799:head# 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:304ef438:::benchmark_data_smithi194_186548_object71039:head# 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30511dd9:::benchmark_data_smithi194_186548_object26466:head# 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30512e5d:::benchmark_data_smithi194_186548_object12434:head# 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30514bcc:::benchmark_data_smithi194_186548_object30234:head# 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3051be27:::benchmark_data_smithi194_186548_object46428:head# 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3051be55:::benchmark_data_smithi194_186548_object23422:head# 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3051e746:::benchmark_data_smithi194_186548_object1228:head# 2024-04-29T14:53:53.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3052db28:::benchmark_data_smithi194_186548_object70557:head# 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3052de21:::benchmark_data_smithi194_186548_object54004:head# 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30530e7e:::benchmark_data_smithi194_186548_object84549:head# 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3054f6ff:::benchmark_data_smithi194_186548_object74284:head# 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3056d7f4:::benchmark_data_smithi194_186548_object26271:head# 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3057394d:::benchmark_data_smithi194_186548_object4514:head# 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3057b059:::benchmark_data_smithi194_186548_object25715:head# 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3057b26a:::benchmark_data_smithi194_186548_object47053:head# 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3057dc75:::benchmark_data_smithi194_186548_object53431:head# 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30587ea8:::benchmark_data_smithi194_186548_object72361:head# 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3058923e:::benchmark_data_smithi194_186548_object29313:head# 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3058cedc:::benchmark_data_smithi194_186548_object69339:head# 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30592d46:::benchmark_data_smithi194_186548_object73652:head# 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3059466e:::benchmark_data_smithi194_186548_object34792:head# 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305979c2:::benchmark_data_smithi194_186548_object18266:head# 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305ae4d0:::benchmark_data_smithi194_186548_object85181:head# 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305cb0d5:::benchmark_data_smithi194_186548_object67209:head# 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305cdec0:::benchmark_data_smithi194_186548_object39874:head# 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305ce0c8:::benchmark_data_smithi194_186548_object73414:head# 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305d049b:::benchmark_data_smithi194_186548_object48101:head# 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305d22e8:::benchmark_data_smithi194_186548_object85127:head# 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305dff14:::benchmark_data_smithi194_186548_object62631:head# 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305f20a6:::benchmark_data_smithi194_186548_object49971:head# 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305fc996:::benchmark_data_smithi194_186548_object10805:head# 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:305fd176:::benchmark_data_smithi194_186548_object26417:head# 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30614a41:::benchmark_data_smithi194_186548_object84380:head# 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3061cdf5:::benchmark_data_smithi194_186548_object48414:head# 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30625ac7:::benchmark_data_smithi194_186548_object78366:head# 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3062786f:::benchmark_data_smithi194_186548_object81898:head# 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3062d661:::benchmark_data_smithi194_186548_object27412:head# 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3063bdc2:::benchmark_data_smithi194_186548_object18551:head# 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3063fe7c:::benchmark_data_smithi194_186548_object78852:head# 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30644165:::benchmark_data_smithi194_186548_object11059:head# 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3066647e:::benchmark_data_smithi194_186548_object76367:head# 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30688284:::benchmark_data_smithi194_186548_object69192:head# 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3069a2a5:::benchmark_data_smithi194_186548_object38536:head# 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306a88ad:::benchmark_data_smithi194_186548_object16137:head# 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306afbfa:::benchmark_data_smithi194_186548_object20477:head# 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306c3499:::benchmark_data_smithi194_186548_object62492:head# 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306c9284:::benchmark_data_smithi194_186548_object5952:head# 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306d0866:::benchmark_data_smithi194_186548_object55840:head# 2024-04-29T14:53:53.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306d2551:::benchmark_data_smithi194_186548_object74390:head# 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306dd155:::benchmark_data_smithi194_186548_object40960:head# 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306e1177:::benchmark_data_smithi194_186548_object57987:head# 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306ea25e:::benchmark_data_smithi194_186548_object67115:head# 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:306ecde3:::benchmark_data_smithi194_186548_object15150:head# 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307046fa:::benchmark_data_smithi194_186548_object23489:head# 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3070d785:::benchmark_data_smithi194_186548_object33366:head# 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30712378:::benchmark_data_smithi194_186548_object13926:head# 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30714ca0:::benchmark_data_smithi194_186548_object20644:head# 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3071f99f:::benchmark_data_smithi194_186548_object11598:head# 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30723556:::benchmark_data_smithi194_186548_object16766:head# 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307437ac:::benchmark_data_smithi194_186548_object42403:head# 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30745fd2:::benchmark_data_smithi194_186548_object48788:head# 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3074b1fe:::benchmark_data_smithi194_186548_object62927:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3074dc86:::benchmark_data_smithi194_186548_object36424:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3075841e:::benchmark_data_smithi194_186548_object19513:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307649f1:::benchmark_data_smithi194_186548_object34991:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3077c474:::benchmark_data_smithi194_186548_object27691:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30781d92:::benchmark_data_smithi194_186548_object67315:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30798b1b:::benchmark_data_smithi194_186548_object14731:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3079b4eb:::benchmark_data_smithi194_186548_object17933:head# 2024-04-29T14:53:53.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307a7d1b:::benchmark_data_smithi194_186548_object1074:head# 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307d1089:::benchmark_data_smithi194_186548_object64782:head# 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307d5f14:::benchmark_data_smithi194_186548_object43747:head# 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307de8bc:::benchmark_data_smithi194_186548_object54153:head# 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307ea710:::benchmark_data_smithi194_186548_object42483:head# 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:307f9703:::benchmark_data_smithi194_186548_object61490:head# 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3080d7c0:::benchmark_data_smithi194_186548_object41593:head# 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.790 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30827c5f:::benchmark_data_smithi194_186548_object25384:head# 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30830cfe:::benchmark_data_smithi194_186548_object57150:head# 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30832345:::benchmark_data_smithi194_186548_object40274:head# 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3083e638:::benchmark_data_smithi194_186548_object16414:head# 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30882003:::benchmark_data_smithi194_186548_object73298:head# 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3088be82:::benchmark_data_smithi194_186548_object2596:head# 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3089f44f:::benchmark_data_smithi194_186548_object64479:head# 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3089f4a9:::benchmark_data_smithi194_186548_object72219:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308a2567:::benchmark_data_smithi194_186548_object54183:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308ab0bf:::benchmark_data_smithi194_186548_object43780:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308af0cd:::benchmark_data_smithi194_186548_object14191:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308b55a9:::benchmark_data_smithi194_186548_object31607:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308ba660:::benchmark_data_smithi194_186548_object80416:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308c2691:::benchmark_data_smithi194_186548_object26145:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308cfc41:::benchmark_data_smithi194_186548_object44516:head# 2024-04-29T14:53:53.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308e73f5:::benchmark_data_smithi194_186548_object25364:head# 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308eea60:::benchmark_data_smithi194_186548_object70055:head# 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:308f339b:::benchmark_data_smithi194_186548_object32972:head# 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309037ab:::benchmark_data_smithi194_186548_object72681:head# 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30905b2e:::benchmark_data_smithi194_186548_object14092:head# 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3090d2bc:::benchmark_data_smithi194_186548_object30698:head# 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30910a93:::benchmark_data_smithi194_186548_object20874:head# 2024-04-29T14:53:53.793 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309180c5:::benchmark_data_smithi194_186548_object80577:head# 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30939f35:::benchmark_data_smithi194_186548_object52591:head# 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30941345:::benchmark_data_smithi194_186548_object9034:head# 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30941bf2:::benchmark_data_smithi194_186548_object35681:head# 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3094c438:::benchmark_data_smithi194_186548_object64102:head# 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30963baa:::benchmark_data_smithi194_186548_object8127:head# 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30972536:::benchmark_data_smithi194_186548_object78226:head# 2024-04-29T14:53:53.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30979c74:::benchmark_data_smithi194_186548_object69294:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30992ffe:::benchmark_data_smithi194_186548_object55342:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3099726c:::benchmark_data_smithi194_186548_object73183:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309bc560:::benchmark_data_smithi194_186548_object69165:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309cb543:::benchmark_data_smithi194_186548_object32337:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309d217e:::benchmark_data_smithi194_186548_object68272:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309e1957:::benchmark_data_smithi194_186548_object37375:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309e4468:::benchmark_data_smithi194_186548_object46713:head# 2024-04-29T14:53:53.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:309ec3c2:::benchmark_data_smithi194_186548_object48088:head# 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a08174:::benchmark_data_smithi194_186548_object56324:head# 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a167da:::benchmark_data_smithi194_186548_object32674:head# 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a19cc6:::benchmark_data_smithi194_186548_object9346:head# 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a260c9:::benchmark_data_smithi194_186548_object51903:head# 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a291aa:::benchmark_data_smithi194_186548_object24235:head# 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a295e0:::benchmark_data_smithi194_186548_object11675:head# 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a4cc15:::benchmark_data_smithi194_186548_object23812:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a640cf:::benchmark_data_smithi194_186548_object60187:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a741fa:::benchmark_data_smithi194_186548_object35545:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30a9eac8:::benchmark_data_smithi194_186548_object16126:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ac2a03:::benchmark_data_smithi194_186548_object62468:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30aceb2a:::benchmark_data_smithi194_186548_object67720:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ada11b:::benchmark_data_smithi194_186548_object12941:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b0ec13:::benchmark_data_smithi194_186548_object58780:head# 2024-04-29T14:53:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b2f89a:::benchmark_data_smithi194_186548_object43181:head# 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b38020:::benchmark_data_smithi194_186548_object31835:head# 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b3d758:::benchmark_data_smithi194_186548_object34325:head# 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b59cbf:::benchmark_data_smithi194_186548_object77207:head# 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b5bdd2:::benchmark_data_smithi194_186548_object26475:head# 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b63798:::benchmark_data_smithi194_186548_object12924:head# 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b7da79:::benchmark_data_smithi194_186548_object25151:head# 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b84adb:::benchmark_data_smithi194_186548_object55705:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b8701e:::benchmark_data_smithi194_186548_object51927:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30b8742b:::benchmark_data_smithi194_186548_object71588:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30bc4b09:::benchmark_data_smithi194_186548_object2600:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30bf9293:::benchmark_data_smithi194_186548_object1623:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c0951f:::benchmark_data_smithi194_186548_object14275:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c16a07:::benchmark_data_smithi194_186548_object47432:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c1ce1c:::benchmark_data_smithi194_186548_object6281:head# 2024-04-29T14:53:53.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c20a1f:::benchmark_data_smithi194_186548_object38170:head# 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c2af27:::benchmark_data_smithi194_186548_object2137:head# 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c336bb:::benchmark_data_smithi194_186548_object34986:head# 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c46dde:::benchmark_data_smithi194_186548_object48427:head# 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c4adb7:::benchmark_data_smithi194_186548_object66866:head# 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c4dbe9:::benchmark_data_smithi194_186548_object67211:head# 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c4f8c6:::benchmark_data_smithi194_186548_object62882:head# 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c4fae3:::benchmark_data_smithi194_186548_object24654:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c5bf18:::benchmark_data_smithi194_186548_object56848:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c67be5:::benchmark_data_smithi194_186548_object41230:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c6b3b3:::benchmark_data_smithi194_186548_object14814:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c7c77b:::benchmark_data_smithi194_186548_object73962:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c860f2:::benchmark_data_smithi194_186548_object48891:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c91c15:::benchmark_data_smithi194_186548_object70956:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30c9a345:::benchmark_data_smithi194_186548_object77637:head# 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30cb31fb:::benchmark_data_smithi194_186548_object72311:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30cbb6fe:::benchmark_data_smithi194_186548_object56565:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30cd3b63:::benchmark_data_smithi194_186548_object54713:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ce1b5f:::benchmark_data_smithi194_186548_object48784:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d0677d:::benchmark_data_smithi194_186548_object6460:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d15856:::benchmark_data_smithi194_186548_object69434:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d16dfa:::benchmark_data_smithi194_186548_object19927:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d1c749:::benchmark_data_smithi194_186548_object17477:head# 2024-04-29T14:53:53.802 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d278de:::benchmark_data_smithi194_186548_object36130:head# 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d2b9a5:::benchmark_data_smithi194_186548_object49711:head# 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d3cb41:::benchmark_data_smithi194_186548_object53153:head# 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d41d99:::benchmark_data_smithi194_186548_object50083:head# 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d432d1:::benchmark_data_smithi194_186548_object59745:head# 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d53d3a:::benchmark_data_smithi194_186548_object68018:head# 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d61216:::benchmark_data_smithi194_186548_object60862:head# 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d6277b:::benchmark_data_smithi194_186548_object22250:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d660d9:::benchmark_data_smithi194_186548_object44528:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d6ddf8:::benchmark_data_smithi194_186548_object47739:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d71749:::benchmark_data_smithi194_186548_object44532:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d7efb4:::benchmark_data_smithi194_186548_object22947:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d83133:::benchmark_data_smithi194_186548_object13875:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d8efb5:::benchmark_data_smithi194_186548_object50827:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d92b7b:::benchmark_data_smithi194_186548_object97:head# 2024-04-29T14:53:53.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d9a7e5:::benchmark_data_smithi194_186548_object31816:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30d9d8d3:::benchmark_data_smithi194_186548_object29137:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30dae327:::benchmark_data_smithi194_186548_object7598:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30db51de:::benchmark_data_smithi194_186548_object46186:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ddf9bc:::benchmark_data_smithi194_186548_object80386:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30df37b0:::benchmark_data_smithi194_186548_object60022:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30df761a:::benchmark_data_smithi194_186548_object20455:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30dfd8f2:::benchmark_data_smithi194_186548_object37746:head# 2024-04-29T14:53:53.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e02bd1:::benchmark_data_smithi194_186548_object6122:head# 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e17f8d:::benchmark_data_smithi194_186548_object27566:head# 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e1c61c:::benchmark_data_smithi194_186548_object35017:head# 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e1cc69:::benchmark_data_smithi194_186548_object12986:head# 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e21bd6:::benchmark_data_smithi194_186548_object12639:head# 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e3bae8:::benchmark_data_smithi194_186548_object33255:head# 2024-04-29T14:53:53.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e3f64d:::benchmark_data_smithi194_186548_object3038:head# 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e48e39:::benchmark_data_smithi194_186548_object51670:head# 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e4c4bc:::benchmark_data_smithi194_186548_object3385:head# 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e4d2cc:::benchmark_data_smithi194_186548_object46840:head# 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e5d44b:::benchmark_data_smithi194_186548_object69729:head# 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e65c29:::benchmark_data_smithi194_186548_object80427:head# 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e729f8:::benchmark_data_smithi194_186548_object56965:head# 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e74ef0:::benchmark_data_smithi194_186548_object76361:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e7d6e9:::benchmark_data_smithi194_186548_object49796:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e8cc48:::benchmark_data_smithi194_186548_object61044:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e92e6d:::benchmark_data_smithi194_186548_object54380:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30e9ec20:::benchmark_data_smithi194_186548_object34109:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30eb0dd7:::benchmark_data_smithi194_186548_object11369:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ed169d:::benchmark_data_smithi194_186548_object8547:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ed37bd:::benchmark_data_smithi194_186548_object8507:head# 2024-04-29T14:53:53.808 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ed547c:::benchmark_data_smithi194_186548_object9416:head# 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30edf67a:::benchmark_data_smithi194_186548_object50312:head# 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ee4146:::benchmark_data_smithi194_186548_object67236:head# 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ee825f:::benchmark_data_smithi194_186548_object75330:head# 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f02ed9:::benchmark_data_smithi194_186548_object56626:head# 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f0df21:::benchmark_data_smithi194_186548_object76742:head# 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f1299f:::benchmark_data_smithi194_186548_object25344:head# 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.809 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f1ed91:::benchmark_data_smithi194_186548_object76027:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f25f36:::benchmark_data_smithi194_186548_object84977:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f2f1e8:::benchmark_data_smithi194_186548_object54034:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f42afa:::benchmark_data_smithi194_186548_object51970:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f48408:::benchmark_data_smithi194_186548_object52025:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f4d5d4:::benchmark_data_smithi194_186548_object58337:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f73fed:::benchmark_data_smithi194_186548_object36473:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f7aaa3:::benchmark_data_smithi194_186548_object12017:head# 2024-04-29T14:53:53.810 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f84ea5:::benchmark_data_smithi194_186548_object24033:head# 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30f897fc:::benchmark_data_smithi194_186548_object61209:head# 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30fa708d:::benchmark_data_smithi194_186548_object27301:head# 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30fbc677:::benchmark_data_smithi194_186548_object9031:head# 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30fcb96d:::benchmark_data_smithi194_186548_object2689:head# 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30fd263f:::benchmark_data_smithi194_186548_object26595:head# 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30fd3e01:::benchmark_data_smithi194_186548_object32932:head# 2024-04-29T14:53:53.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30fe41b1:::benchmark_data_smithi194_186548_object48636:head# 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ff364d:::benchmark_data_smithi194_186548_object44492:head# 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ff4010:::benchmark_data_smithi194_186548_object66194:head# 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:30ff8e98:::benchmark_data_smithi194_186548_object71856:head# 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31009bfe:::benchmark_data_smithi194_186548_object73274:head# 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3100c41f:::benchmark_data_smithi194_186548_object64441:head# 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3101a80e:::benchmark_data_smithi194_186548_object23938:head# 2024-04-29T14:53:53.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31029d07:::benchmark_data_smithi194_186548_object30287:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31035183:::benchmark_data_smithi194_186548_object27809:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31046d23:::benchmark_data_smithi194_186548_object84108:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3104b244:::benchmark_data_smithi194_186548_object51226:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31052056:::benchmark_data_smithi194_186548_object53397:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31056ccd:::benchmark_data_smithi194_186548_object48429:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31071b8c:::benchmark_data_smithi194_186548_object59600:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31072902:::benchmark_data_smithi194_186548_object27597:head# 2024-04-29T14:53:53.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31075fff:::benchmark_data_smithi194_186548_object72692:head# 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3107a595:::benchmark_data_smithi194_186548_object19204:head# 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3107ead5:::benchmark_data_smithi194_186548_object85331:head# 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3108716a:::benchmark_data_smithi194_186548_object19734:head# 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31088222:::benchmark_data_smithi194_186548_object8898:head# 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3108d688:::benchmark_data_smithi194_186548_object61065:head# 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31090591:::benchmark_data_smithi194_186548_object57836:head# 2024-04-29T14:53:53.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3109f285:::benchmark_data_smithi194_186548_object80654:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310a93b6:::benchmark_data_smithi194_186548_object24574:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310b0541:::benchmark_data_smithi194_186548_object16864:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310b07aa:::benchmark_data_smithi194_186548_object77083:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310b3413:::benchmark_data_smithi194_186548_object47902:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310d8780:::benchmark_data_smithi194_186548_object16672:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310e3a4b:::benchmark_data_smithi194_186548_object52503:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310f0d1d:::benchmark_data_smithi194_186548_object7177:head# 2024-04-29T14:53:53.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:310fe780:::benchmark_data_smithi194_186548_object77920:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311189fd:::benchmark_data_smithi194_186548_object65760:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3111d7ac:::benchmark_data_smithi194_186548_object43945:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31124791:::benchmark_data_smithi194_186548_object23424:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31128d7a:::benchmark_data_smithi194_186548_object35532:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311449e2:::benchmark_data_smithi194_186548_object37036:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311455eb:::benchmark_data_smithi194_186548_object26587:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3114f61b:::benchmark_data_smithi194_186548_object42950:head# 2024-04-29T14:53:53.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31165e35:::benchmark_data_smithi194_186548_object42600:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31169049:::benchmark_data_smithi194_186548_object25088:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31172c86:::benchmark_data_smithi194_186548_object44305:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31174f8e:::benchmark_data_smithi194_186548_object31217:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3118c2f8:::benchmark_data_smithi194_186548_object25901:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31190c06:::benchmark_data_smithi194_186548_object38663:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31199d44:::benchmark_data_smithi194_186548_object8183:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3119c8fc:::benchmark_data_smithi194_186548_object65014:head# 2024-04-29T14:53:53.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311a774d:::benchmark_data_smithi194_186548_object13415:head# 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311ac079:::benchmark_data_smithi194_186548_object11218:head# 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311b1e07:::benchmark_data_smithi194_186548_object38199:head# 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311c4138:::benchmark_data_smithi194_186548_object36350:head# 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311dd055:::benchmark_data_smithi194_186548_object50270:head# 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311ddda7:::benchmark_data_smithi194_186548_object62206:head# 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311e1544:::benchmark_data_smithi194_186548_object51723:head# 2024-04-29T14:53:53.818 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311e75b0:::benchmark_data_smithi194_186548_object64878:head# 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311ea181:::benchmark_data_smithi194_186548_object35290:head# 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311ea6fb:::benchmark_data_smithi194_186548_object76733:head# 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311f4951:::benchmark_data_smithi194_186548_object83875:head# 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:311f92c5:::benchmark_data_smithi194_186548_object19478:head# 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31203251:::benchmark_data_smithi194_186548_object15804:head# 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3121b0ed:::benchmark_data_smithi194_186548_object19468:head# 2024-04-29T14:53:53.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3121fcae:::benchmark_data_smithi194_186548_object57539:head# 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312235d0:::benchmark_data_smithi194_186548_object67347:head# 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312237c2:::benchmark_data_smithi194_186548_object56937:head# 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312285b8:::benchmark_data_smithi194_186548_object63821:head# 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31230706:::benchmark_data_smithi194_186548_object67586:head# 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3123d9c5:::benchmark_data_smithi194_186548_object26556:head# 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3125e007:::benchmark_data_smithi194_186548_object33600:head# 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31262d88:::benchmark_data_smithi194_186548_object49659:head# 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312821bc:::benchmark_data_smithi194_186548_object42718:head# 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3128f8e1:::benchmark_data_smithi194_186548_object77141:head# 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3128fe96:::benchmark_data_smithi194_186548_object40477:head# 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312b84cb:::benchmark_data_smithi194_186548_object57579:head# 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312d1acf:::benchmark_data_smithi194_186548_object10347:head# 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312d1da4:::benchmark_data_smithi194_186548_object10956:head# 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312e05b8:::benchmark_data_smithi194_186548_object48172:head# 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312e2580:::benchmark_data_smithi194_186548_object22854:head# 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:312eb1bf:::benchmark_data_smithi194_186548_object29495:head# 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31309eaf:::benchmark_data_smithi194_186548_object49001:head# 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3131f853:::benchmark_data_smithi194_186548_object33676:head# 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3132558d:::benchmark_data_smithi194_186548_object80441:head# 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313268ab:::benchmark_data_smithi194_186548_object80173:head# 2024-04-29T14:53:53.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313279cb:::benchmark_data_smithi194_186548_object2386:head# 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313396e0:::benchmark_data_smithi194_186548_object24085:head# 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31343ae4:::benchmark_data_smithi194_186548_object62459:head# 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31346cc5:::benchmark_data_smithi194_186548_object13324:head# 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31351c3a:::benchmark_data_smithi194_186548_object28320:head# 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313524b5:::benchmark_data_smithi194_186548_object53066:head# 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31353322:::benchmark_data_smithi194_186548_object32119:head# 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313644ed:::benchmark_data_smithi194_186548_object74986:head# 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313686a0:::benchmark_data_smithi194_186548_object68712:head# 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3136fd17:::benchmark_data_smithi194_186548_object85028:head# 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3138af48:::benchmark_data_smithi194_186548_object44338:head# 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3138e6fd:::benchmark_data_smithi194_186548_object33912:head# 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3139353a:::benchmark_data_smithi194_186548_object32661:head# 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31397ecd:::benchmark_data_smithi194_186548_object83088:head# 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3139a1fb:::benchmark_data_smithi194_186548_object76933:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313b2008:::benchmark_data_smithi194_186548_object48710:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313b952c:::benchmark_data_smithi194_186548_object23985:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313bccbe:::benchmark_data_smithi194_186548_object15597:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313bf906:::benchmark_data_smithi194_186548_object45895:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313c0f78:::benchmark_data_smithi194_186548_object75393:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313c7cb1:::benchmark_data_smithi194_186548_object81403:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313e30ae:::benchmark_data_smithi194_186548_object25687:head# 2024-04-29T14:53:53.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313e4de4:::benchmark_data_smithi194_186548_object64792:head# 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313e5698:::benchmark_data_smithi194_186548_object6560:head# 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313e67e8:::benchmark_data_smithi194_186548_object35513:head# 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313ecde2:::benchmark_data_smithi194_186548_object33117:head# 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313fd959:::benchmark_data_smithi194_186548_object7096:head# 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:313fff53:::benchmark_data_smithi194_186548_object82871:head# 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314028b9:::benchmark_data_smithi194_186548_object43300:head# 2024-04-29T14:53:53.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3140ff9d:::benchmark_data_smithi194_186548_object50415:head# 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31414e57:::benchmark_data_smithi194_186548_object78778:head# 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3141cde5:::benchmark_data_smithi194_186548_object15082:head# 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314301ef:::benchmark_data_smithi194_186548_object46226:head# 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3143f36d:::benchmark_data_smithi194_186548_object59034:head# 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3145370a:::benchmark_data_smithi194_186548_object19709:head# 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3145747f:::benchmark_data_smithi194_186548_object49833:head# 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.827 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3145b48a:::benchmark_data_smithi194_186548_object18884:head# 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31466651:::benchmark_data_smithi194_186548_object70618:head# 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3146df33:::benchmark_data_smithi194_186548_object77743:head# 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3147b845:::benchmark_data_smithi194_186548_object59646:head# 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3147d96e:::benchmark_data_smithi194_186548_object43210:head# 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3148eba7:::benchmark_data_smithi194_186548_object53687:head# 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3148f816:::benchmark_data_smithi194_186548_object85271:head# 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.828 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3149bb4b:::benchmark_data_smithi194_186548_object27402:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314b6c7f:::benchmark_data_smithi194_186548_object85070:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314b7885:::benchmark_data_smithi194_186548_object50819:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314ba73b:::benchmark_data_smithi194_186548_object26432:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314bf6fa:::benchmark_data_smithi194_186548_object16249:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314cb7ef:::benchmark_data_smithi194_186548_object12260:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314d971c:::benchmark_data_smithi194_186548_object73389:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314eabbc:::benchmark_data_smithi194_186548_object33501:head# 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:314f28d6:::benchmark_data_smithi194_186548_object56314:head# 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:315201e0:::benchmark_data_smithi194_186548_object10891:head# 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:315361a5:::benchmark_data_smithi194_186548_object15774:head# 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31536f17:::benchmark_data_smithi194_186548_object12722:head# 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31547a26:::benchmark_data_smithi194_186548_object69741:head# 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31554576:::benchmark_data_smithi194_186548_object17473:head# 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3155bd15:::benchmark_data_smithi194_186548_object67671:head# 2024-04-29T14:53:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3156173e:::benchmark_data_smithi194_186548_object51373:head# 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31567221:::benchmark_data_smithi194_186548_object57778:head# 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31570137:::benchmark_data_smithi194_186548_object26986:head# 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3157d8f0:::benchmark_data_smithi194_186548_object84713:head# 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3158e383:::benchmark_data_smithi194_186548_object75613:head# 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3158f80f:::benchmark_data_smithi194_186548_object24621:head# 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31591f2b:::benchmark_data_smithi194_186548_object82540:head# 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.831 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31596746:::benchmark_data_smithi194_186548_object71863:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31598846:::benchmark_data_smithi194_186548_object40205:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:315a9576:::benchmark_data_smithi194_186548_object26863:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:315b6755:::benchmark_data_smithi194_186548_object39898:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:315d84ed:::benchmark_data_smithi194_186548_object50934:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316048d0:::benchmark_data_smithi194_186548_object81002:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31606f7e:::benchmark_data_smithi194_186548_object56659:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31622739:::benchmark_data_smithi194_186548_object24578:head# 2024-04-29T14:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316307f2:::benchmark_data_smithi194_186548_object84820:head# 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31630b26:::benchmark_data_smithi194_186548_object14321:head# 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316529fb:::benchmark_data_smithi194_186548_object3796:head# 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316657f5:::benchmark_data_smithi194_186548_object50749:head# 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31669b20:::benchmark_data_smithi194_186548_object63905:head# 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3166becf:::benchmark_data_smithi194_186548_object314:head# 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3167f91f:::benchmark_data_smithi194_186548_object28507:head# 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3168a048:::benchmark_data_smithi194_186548_object17271:head# 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3168f877:::benchmark_data_smithi194_186548_object37585:head# 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3169880c:::benchmark_data_smithi194_186548_object61554:head# 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31699a2e:::benchmark_data_smithi194_186548_object73949:head# 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316d620b:::benchmark_data_smithi194_186548_object17585:head# 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316d7ef9:::benchmark_data_smithi194_186548_object78802:head# 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316f71dd:::benchmark_data_smithi194_186548_object1101:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316f9236:::benchmark_data_smithi194_186548_object4638:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:316fb2fb:::benchmark_data_smithi194_186548_object64111:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317015a2:::benchmark_data_smithi194_186548_object35081:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3170e6db:::benchmark_data_smithi194_186548_object16604:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3172c53d:::benchmark_data_smithi194_186548_object62400:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3172d8ec:::benchmark_data_smithi194_186548_object19741:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3173504a:::benchmark_data_smithi194_186548_object69221:head# 2024-04-29T14:53:53.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31737999:::benchmark_data_smithi194_186548_object1199:head# 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3173fc24:::benchmark_data_smithi194_186548_object51253:head# 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31745eab:::benchmark_data_smithi194_186548_object64862:head# 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3174af1e:::benchmark_data_smithi194_186548_object27273:head# 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31750b75:::benchmark_data_smithi194_186548_object54588:head# 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317604cc:::benchmark_data_smithi194_186548_object16027:head# 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31764866:::benchmark_data_smithi194_186548_object2212:head# 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31776391:::benchmark_data_smithi194_186548_object69894:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31778e39:::benchmark_data_smithi194_186548_object702:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3178174a:::benchmark_data_smithi194_186548_object33860:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31781f9e:::benchmark_data_smithi194_186548_object84915:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317826cb:::benchmark_data_smithi194_186548_object56505:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317b53fe:::benchmark_data_smithi194_186548_object16192:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317bb952:::benchmark_data_smithi194_186548_object74617:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317bd106:::benchmark_data_smithi194_186548_object35701:head# 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317cdfc4:::benchmark_data_smithi194_186548_object48368:head# 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317f8719:::benchmark_data_smithi194_186548_object39530:head# 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:317fe2fa:::benchmark_data_smithi194_186548_object60876:head# 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31800e09:::benchmark_data_smithi194_186548_object52437:head# 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3180f8a5:::benchmark_data_smithi194_186548_object13397:head# 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318151ee:::benchmark_data_smithi194_186548_object61349:head# 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31815a11:::benchmark_data_smithi194_186548_object66615:head# 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3181daed:::benchmark_data_smithi194_186548_object9820:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318236c4:::benchmark_data_smithi194_186548_object56140:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3182cd79:::benchmark_data_smithi194_186548_object77680:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31832af7:::benchmark_data_smithi194_186548_object9504:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3183cbd1:::benchmark_data_smithi194_186548_object1710:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3183dbdd:::benchmark_data_smithi194_186548_object12521:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31843183:::benchmark_data_smithi194_186548_object35865:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31849477:::benchmark_data_smithi194_186548_object15961:head# 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3184aa8b:::benchmark_data_smithi194_186548_object60435:head# 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31859b1a:::benchmark_data_smithi194_186548_object40491:head# 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3185af81:::benchmark_data_smithi194_186548_object54079:head# 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3186dba0:::benchmark_data_smithi194_186548_object14819:head# 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31886db0:::benchmark_data_smithi194_186548_object30267:head# 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31893905:::benchmark_data_smithi194_186548_object9250:head# 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3189a5d8:::benchmark_data_smithi194_186548_object17112:head# 2024-04-29T14:53:53.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3189fbaf:::benchmark_data_smithi194_186548_object21392:head# 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318a4c14:::benchmark_data_smithi194_186548_object4746:head# 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318c9962:::benchmark_data_smithi194_186548_object58572:head# 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318d8a96:::benchmark_data_smithi194_186548_object12390:head# 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318ed984:::benchmark_data_smithi194_186548_object75942:head# 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318f67bf:::benchmark_data_smithi194_186548_object27866:head# 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.841 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:318f79be:::benchmark_data_smithi194_186548_object19392:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3190ef32:::benchmark_data_smithi194_186548_object54674:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31915615:::benchmark_data_smithi194_186548_object22614:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31926c6c:::benchmark_data_smithi194_186548_object39719:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3192d26c:::benchmark_data_smithi194_186548_object64492:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31931294:::benchmark_data_smithi194_186548_object33302:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3193914a:::benchmark_data_smithi194_186548_object53369:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31952a16:::benchmark_data_smithi194_186548_object51437:head# 2024-04-29T14:53:53.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3195a51b:::benchmark_data_smithi194_186548_object52586:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3195b4b8:::benchmark_data_smithi194_186548_object37903:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3195b8c3:::benchmark_data_smithi194_186548_object1447:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31967861:::benchmark_data_smithi194_186548_object4723:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3198748e:::benchmark_data_smithi194_186548_object60924:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31989de9:::benchmark_data_smithi194_186548_object16004:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3199493a:::benchmark_data_smithi194_186548_object38817:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3199e61b:::benchmark_data_smithi194_186548_object54781:head# 2024-04-29T14:53:53.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:319a0694:::benchmark_data_smithi194_186548_object59186:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:319abe2e:::benchmark_data_smithi194_186548_object21309:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:319bc8ea:::benchmark_data_smithi194_186548_object13852:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:319e7f09:::benchmark_data_smithi194_186548_object73096:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:319ed150:::benchmark_data_smithi194_186548_object53047:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:319f5ca0:::benchmark_data_smithi194_186548_object67356:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:319f784a:::benchmark_data_smithi194_186548_object63083:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a04e80:::benchmark_data_smithi194_186548_object36777:head# 2024-04-29T14:53:53.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a25c80:::benchmark_data_smithi194_186548_object51917:head# 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a29a44:::benchmark_data_smithi194_186548_object44396:head# 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a2ad23:::benchmark_data_smithi194_186548_object20322:head# 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a2eefb:::benchmark_data_smithi194_186548_object35418:head# 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a41ec8:::benchmark_data_smithi194_186548_object17313:head# 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a69792:::benchmark_data_smithi194_186548_object54394:head# 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a6d2f9:::benchmark_data_smithi194_186548_object47679:head# 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a7b970:::benchmark_data_smithi194_186548_object2135:head# 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a9344e:::benchmark_data_smithi194_186548_object22865:head# 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a9c68c:::benchmark_data_smithi194_186548_object8644:head# 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31a9fbdd:::benchmark_data_smithi194_186548_object55751:head# 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31aa5bf8:::benchmark_data_smithi194_186548_object49091:head# 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31aa72be:::benchmark_data_smithi194_186548_object20134:head# 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ab09cc:::benchmark_data_smithi194_186548_object54151:head# 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31abc427:::benchmark_data_smithi194_186548_object10959:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ae1cf8:::benchmark_data_smithi194_186548_object58842:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31aed153:::benchmark_data_smithi194_186548_object61665:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31af6aa3:::benchmark_data_smithi194_186548_object3559:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31afa861:::benchmark_data_smithi194_186548_object31654:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b0c370:::benchmark_data_smithi194_186548_object11898:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b0e772:::benchmark_data_smithi194_186548_object58290:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b274f2:::benchmark_data_smithi194_186548_object28311:head# 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b374f4:::benchmark_data_smithi194_186548_object13373:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b3c1e3:::benchmark_data_smithi194_186548_object48621:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b40d9d:::benchmark_data_smithi194_186548_object25367:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b46325:::benchmark_data_smithi194_186548_object11109:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b4638b:::benchmark_data_smithi194_186548_object7950:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b4a810:::benchmark_data_smithi194_186548_object57357:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b4b4b2:::benchmark_data_smithi194_186548_object36672:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b4c166:::benchmark_data_smithi194_186548_object39776:head# 2024-04-29T14:53:53.848 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b52f7a:::benchmark_data_smithi194_186548_object68946:head# 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b575b4:::benchmark_data_smithi194_186548_object46309:head# 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b697c7:::benchmark_data_smithi194_186548_object78601:head# 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b6e2ef:::benchmark_data_smithi194_186548_object84795:head# 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b6f5e7:::benchmark_data_smithi194_186548_object108:head# 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b7128c:::benchmark_data_smithi194_186548_object12361:head# 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b73e1f:::benchmark_data_smithi194_186548_object70427:head# 2024-04-29T14:53:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b74f06:::benchmark_data_smithi194_186548_object47692:head# 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b8a434:::benchmark_data_smithi194_186548_object6633:head# 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b8c182:::benchmark_data_smithi194_186548_object42859:head# 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b90ccf:::benchmark_data_smithi194_186548_object43604:head# 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31b9ffa3:::benchmark_data_smithi194_186548_object33377:head# 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ba3202:::benchmark_data_smithi194_186548_object18:head# 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bb4f5d:::benchmark_data_smithi194_186548_object252:head# 2024-04-29T14:53:53.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bb50ba:::benchmark_data_smithi194_186548_object83682:head# 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bb9303:::benchmark_data_smithi194_186548_object29081:head# 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bbb8bf:::benchmark_data_smithi194_186548_object52085:head# 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bbbd29:::benchmark_data_smithi194_186548_object71395:head# 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bc213c:::benchmark_data_smithi194_186548_object41348:head# 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bc32ec:::benchmark_data_smithi194_186548_object66257:head# 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bc9ebf:::benchmark_data_smithi194_186548_object29663:head# 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bd14ff:::benchmark_data_smithi194_186548_object30043:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bdba95:::benchmark_data_smithi194_186548_object41049:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31beda00:::benchmark_data_smithi194_186548_object5745:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bfc24b:::benchmark_data_smithi194_186548_object813:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31bfdd27:::benchmark_data_smithi194_186548_object79099:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31c0da67:::benchmark_data_smithi194_186548_object8438:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31c24bc0:::benchmark_data_smithi194_186548_object49321:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31c2f1c6:::benchmark_data_smithi194_186548_object2206:head# 2024-04-29T14:53:53.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31c3f14f:::benchmark_data_smithi194_186548_object77389:head# 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31c56bc2:::benchmark_data_smithi194_186548_object68651:head# 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31c66334:::benchmark_data_smithi194_186548_object13374:head# 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31c8a899:::benchmark_data_smithi194_186548_object45223:head# 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ca3c7e:::benchmark_data_smithi194_186548_object27234:head# 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31cbb57d:::benchmark_data_smithi194_186548_object79711:head# 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31cc6e7e:::benchmark_data_smithi194_186548_object26951:head# 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ce02c0:::benchmark_data_smithi194_186548_object68276:head# 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ce0400:::benchmark_data_smithi194_186548_object42304:head# 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ceb25d:::benchmark_data_smithi194_186548_object46348:head# 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31cf0dcc:::benchmark_data_smithi194_186548_object23528:head# 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31cf81e2:::benchmark_data_smithi194_186548_object37755:head# 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.854 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d18746:::benchmark_data_smithi194_186548_object20439:head# 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d22cf8:::benchmark_data_smithi194_186548_object25487:head# 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d368ce:::benchmark_data_smithi194_186548_object45707:head# 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d37a42:::benchmark_data_smithi194_186548_object11905:head# 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d3a24e:::benchmark_data_smithi194_186548_object54552:head# 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d66c99:::benchmark_data_smithi194_186548_object7629:head# 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d78787:::benchmark_data_smithi194_186548_object9551:head# 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d8e80d:::benchmark_data_smithi194_186548_object29294:head# 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31d8fadb:::benchmark_data_smithi194_186548_object84701:head# 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31da180e:::benchmark_data_smithi194_186548_object80480:head# 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31da18bf:::benchmark_data_smithi194_186548_object21718:head# 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31db3503:::benchmark_data_smithi194_186548_object48775:head# 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31db8de6:::benchmark_data_smithi194_186548_object57210:head# 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31dc8028:::benchmark_data_smithi194_186548_object72016:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31dd3c16:::benchmark_data_smithi194_186548_object79773:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31df7bc3:::benchmark_data_smithi194_186548_object688:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31dfc311:::benchmark_data_smithi194_186548_object39027:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31dfe46f:::benchmark_data_smithi194_186548_object9386:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31dfeb0d:::benchmark_data_smithi194_186548_object19551:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e0c7ef:::benchmark_data_smithi194_186548_object20232:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e1a2d5:::benchmark_data_smithi194_186548_object61116:head# 2024-04-29T14:53:53.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e291d0:::benchmark_data_smithi194_186548_object61356:head# 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e531b5:::benchmark_data_smithi194_186548_object23060:head# 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e69d70:::benchmark_data_smithi194_186548_object17831:head# 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e6d39c:::benchmark_data_smithi194_186548_object42554:head# 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e6f228:::benchmark_data_smithi194_186548_object74675:head# 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e74480:::benchmark_data_smithi194_186548_object83695:head# 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e7a00b:::benchmark_data_smithi194_186548_object27912:head# 2024-04-29T14:53:53.858 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31e866db:::benchmark_data_smithi194_186548_object59670:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ea44ca:::benchmark_data_smithi194_186548_object65644:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31eb8fba:::benchmark_data_smithi194_186548_object9339:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ec2743:::benchmark_data_smithi194_186548_object5279:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ec3b28:::benchmark_data_smithi194_186548_object13617:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31ed9c7e:::benchmark_data_smithi194_186548_object30618:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31eec82c:::benchmark_data_smithi194_186548_object62597:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31efed0d:::benchmark_data_smithi194_186548_object79327:head# 2024-04-29T14:53:53.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f12101:::benchmark_data_smithi194_186548_object50202:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f39b31:::benchmark_data_smithi194_186548_object61101:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f43b79:::benchmark_data_smithi194_186548_object85194:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f49b4c:::benchmark_data_smithi194_186548_object70684:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f510e7:::benchmark_data_smithi194_186548_object18348:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f51e66:::benchmark_data_smithi194_186548_object39481:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f55c02:::benchmark_data_smithi194_186548_object58885:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f6670b:::benchmark_data_smithi194_186548_object46174:head# 2024-04-29T14:53:53.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f85332:::benchmark_data_smithi194_186548_object57264:head# 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f946a4:::benchmark_data_smithi194_186548_object55208:head# 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f96d64:::benchmark_data_smithi194_186548_object23454:head# 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31f9cfd8:::benchmark_data_smithi194_186548_object49853:head# 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fa1563:::benchmark_data_smithi194_186548_object73615:head# 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fa8a6c:::benchmark_data_smithi194_186548_object47382:head# 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fb22c2:::benchmark_data_smithi194_186548_object11476:head# 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fb891f:::benchmark_data_smithi194_186548_object4741:head# 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fc3175:::benchmark_data_smithi194_186548_object6189:head# 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fcb5f1:::benchmark_data_smithi194_186548_object45183:head# 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fdec56:::benchmark_data_smithi194_186548_object56318:head# 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:31fe818b:::benchmark_data_smithi194_186548_object11805:head# 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32017fcf:::benchmark_data_smithi194_186548_object67277:head# 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32019cf7:::benchmark_data_smithi194_186548_object1658:head# 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3203d4cb:::benchmark_data_smithi194_186548_object11909:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3205fd42:::benchmark_data_smithi194_186548_object33388:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3206a1ba:::benchmark_data_smithi194_186548_object77155:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3206add7:::benchmark_data_smithi194_186548_object1243:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3207023e:::benchmark_data_smithi194_186548_object67361:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32077440:::benchmark_data_smithi194_186548_object19405:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32088c31:::benchmark_data_smithi194_186548_object47195:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3208d036:::benchmark_data_smithi194_186548_object69646:head# 2024-04-29T14:53:53.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:320c44fb:::benchmark_data_smithi194_186548_object44591:head# 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:320c5fe9:::benchmark_data_smithi194_186548_object280:head# 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:320f1127:::benchmark_data_smithi194_186548_object7445:head# 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32123498:::benchmark_data_smithi194_186548_object62008:head# 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321348e1:::benchmark_data_smithi194_186548_object2171:head# 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321358d7:::benchmark_data_smithi194_186548_object15839:head# 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3213bc3c:::benchmark_data_smithi194_186548_object15784:head# 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32148a5a:::benchmark_data_smithi194_186548_object72191:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32149622:::benchmark_data_smithi194_186548_object81345:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32158c5f:::benchmark_data_smithi194_186548_object3089:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3215b250:::benchmark_data_smithi194_186548_object23755:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321677bb:::benchmark_data_smithi194_186548_object73765:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321702a5:::benchmark_data_smithi194_186548_object78321:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32180906:::benchmark_data_smithi194_186548_object6053:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3218b174:::benchmark_data_smithi194_186548_object63886:head# 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3218d4da:::benchmark_data_smithi194_186548_object30536:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32195192:::benchmark_data_smithi194_186548_object34901:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321ad3f5:::benchmark_data_smithi194_186548_object49942:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321be706:::benchmark_data_smithi194_186548_object26155:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321c5af9:::benchmark_data_smithi194_186548_object18911:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321c7aeb:::benchmark_data_smithi194_186548_object81964:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321e4d84:::benchmark_data_smithi194_186548_object27130:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:321f6ba0:::benchmark_data_smithi194_186548_object76812:head# 2024-04-29T14:53:53.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32203703:::benchmark_data_smithi194_186548_object7772:head# 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32204cee:::benchmark_data_smithi194_186548_object17316:head# 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3220a1a4:::benchmark_data_smithi194_186548_object82828:head# 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3220a54f:::benchmark_data_smithi194_186548_object38412:head# 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32214f12:::benchmark_data_smithi194_186548_object83126:head# 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32219561:::benchmark_data_smithi194_186548_object61310:head# 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3221faa2:::benchmark_data_smithi194_186548_object40868:head# 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3222090d:::benchmark_data_smithi194_186548_object22655:head# 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322534a9:::benchmark_data_smithi194_186548_object56015:head# 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32258c39:::benchmark_data_smithi194_186548_object26746:head# 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3225a784:::benchmark_data_smithi194_186548_object20809:head# 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3225f4d2:::benchmark_data_smithi194_186548_object50382:head# 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32274b45:::benchmark_data_smithi194_186548_object84907:head# 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3227d933:::benchmark_data_smithi194_186548_object34086:head# 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.868 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3227df34:::benchmark_data_smithi194_186548_object62416:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322853dd:::benchmark_data_smithi194_186548_object31029:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322887b3:::benchmark_data_smithi194_186548_object13099:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322b7df2:::benchmark_data_smithi194_186548_object6583:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322b7ec4:::benchmark_data_smithi194_186548_object7667:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322be446:::benchmark_data_smithi194_186548_object10671:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322d31d7:::benchmark_data_smithi194_186548_object12269:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322d3f17:::benchmark_data_smithi194_186548_object43497:head# 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322d6738:::benchmark_data_smithi194_186548_object27733:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322e41f4:::benchmark_data_smithi194_186548_object10553:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322e857b:::benchmark_data_smithi194_186548_object49717:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:322ee828:::benchmark_data_smithi194_186548_object50388:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32308b96:::benchmark_data_smithi194_186548_object41851:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3231083c:::benchmark_data_smithi194_186548_object43687:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3231272f:::benchmark_data_smithi194_186548_object29422:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32317e9d:::benchmark_data_smithi194_186548_object66116:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32328041:::benchmark_data_smithi194_186548_object36840:head# 2024-04-29T14:53:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3232c2ad:::benchmark_data_smithi194_186548_object271:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323328a0:::benchmark_data_smithi194_186548_object12799:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323356d8:::benchmark_data_smithi194_186548_object10389:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3233bd6c:::benchmark_data_smithi194_186548_object75138:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32345ae9:::benchmark_data_smithi194_186548_object17649:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32346947:::benchmark_data_smithi194_186548_object2887:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323479b6:::benchmark_data_smithi194_186548_object62059:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3234c1e3:::benchmark_data_smithi194_186548_object37622:head# 2024-04-29T14:53:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3236675e:::benchmark_data_smithi194_186548_object11634:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3236bb80:::benchmark_data_smithi194_186548_object53946:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32397dea:::benchmark_data_smithi194_186548_object18101:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323a744a:::benchmark_data_smithi194_186548_object19813:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323b9c3f:::benchmark_data_smithi194_186548_object27118:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323bd106:::benchmark_data_smithi194_186548_object44393:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323ca97e:::benchmark_data_smithi194_186548_object74861:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323ceb4f:::benchmark_data_smithi194_186548_object14517:head# 2024-04-29T14:53:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323d7665:::benchmark_data_smithi194_186548_object10803:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323efe60:::benchmark_data_smithi194_186548_object82818:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:323f845a:::benchmark_data_smithi194_186548_object45165:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:324070d7:::benchmark_data_smithi194_186548_object17951:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3241b70f:::benchmark_data_smithi194_186548_object66399:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3241e757:::benchmark_data_smithi194_186548_object65254:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32430610:::benchmark_data_smithi194_186548_object25426:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32433473:::benchmark_data_smithi194_186548_object78183:head# 2024-04-29T14:53:53.873 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32447614:::benchmark_data_smithi194_186548_object56926:head# 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3244debc:::benchmark_data_smithi194_186548_object65892:head# 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32472c52:::benchmark_data_smithi194_186548_object15131:head# 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32477eab:::benchmark_data_smithi194_186548_object41762:head# 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:324793c2:::benchmark_data_smithi194_186548_object78120:head# 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3247b82d:::benchmark_data_smithi194_186548_object32002:head# 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3248e8f6:::benchmark_data_smithi194_186548_object60317:head# 2024-04-29T14:53:53.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3249be59:::benchmark_data_smithi194_186548_object65347:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3249d072:::benchmark_data_smithi194_186548_object11891:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3249e863:::benchmark_data_smithi194_186548_object41316:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:324bb2af:::benchmark_data_smithi194_186548_object83364:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:324cdde5:::benchmark_data_smithi194_186548_object42778:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:324e327c:::benchmark_data_smithi194_186548_object39392:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:324f5b25:::benchmark_data_smithi194_186548_object26:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:324fa25d:::benchmark_data_smithi194_186548_object58042:head# 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32500dec:::benchmark_data_smithi194_186548_object84125:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32510592:::benchmark_data_smithi194_186548_object47853:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32513571:::benchmark_data_smithi194_186548_object69321:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32518ebb:::benchmark_data_smithi194_186548_object19665:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3251c0a5:::benchmark_data_smithi194_186548_object13901:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32522b98:::benchmark_data_smithi194_186548_object16702:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32525b2a:::benchmark_data_smithi194_186548_object33861:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3254c1b4:::benchmark_data_smithi194_186548_object80866:head# 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.876 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3255792d:::benchmark_data_smithi194_186548_object44990:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32562144:::benchmark_data_smithi194_186548_object63523:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32573830:::benchmark_data_smithi194_186548_object71761:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325743ed:::benchmark_data_smithi194_186548_object60364:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3257d712:::benchmark_data_smithi194_186548_object25493:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325aa194:::benchmark_data_smithi194_186548_object36753:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325b2ae0:::benchmark_data_smithi194_186548_object51048:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325b700f:::benchmark_data_smithi194_186548_object27228:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325c4e6d:::benchmark_data_smithi194_186548_object53007:head# 2024-04-29T14:53:53.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325ce492:::benchmark_data_smithi194_186548_object22376:head# 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325d6763:::benchmark_data_smithi194_186548_object2812:head# 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:325f0065:::benchmark_data_smithi194_186548_object19839:head# 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32612b5b:::benchmark_data_smithi194_186548_object26142:head# 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3261ffd1:::benchmark_data_smithi194_186548_object16061:head# 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3262364c:::benchmark_data_smithi194_186548_object66550:head# 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3262ba5e:::benchmark_data_smithi194_186548_object65800:head# 2024-04-29T14:53:53.878 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32641201:::benchmark_data_smithi194_186548_object21990:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32645ac2:::benchmark_data_smithi194_186548_object74581:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32651e25:::benchmark_data_smithi194_186548_object73944:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3265d7fd:::benchmark_data_smithi194_186548_object54894:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3265e6a3:::benchmark_data_smithi194_186548_object77719:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326607e9:::benchmark_data_smithi194_186548_object55017:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32666961:::benchmark_data_smithi194_186548_object15944:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32681861:::benchmark_data_smithi194_186548_object9457:head# 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326d08ca:::benchmark_data_smithi194_186548_object36697:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326ea9fd:::benchmark_data_smithi194_186548_object5613:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326ec0b4:::benchmark_data_smithi194_186548_object44261:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326ecba9:::benchmark_data_smithi194_186548_object52352:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326f1554:::benchmark_data_smithi194_186548_object79484:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326fbea6:::benchmark_data_smithi194_186548_object11686:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:326fc0e7:::benchmark_data_smithi194_186548_object59071:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3270384f:::benchmark_data_smithi194_186548_object29191:head# 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3270832f:::benchmark_data_smithi194_186548_object66384:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:327275a4:::benchmark_data_smithi194_186548_object25673:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32737c39:::benchmark_data_smithi194_186548_object82543:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3277109a:::benchmark_data_smithi194_186548_object30492:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3278332f:::benchmark_data_smithi194_186548_object33376:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32789967:::benchmark_data_smithi194_186548_object68476:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3279c8c5:::benchmark_data_smithi194_186548_object3419:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:327a39b8:::benchmark_data_smithi194_186548_object31506:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:327ab457:::benchmark_data_smithi194_186548_object74984:head# 2024-04-29T14:53:53.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:327c2918:::benchmark_data_smithi194_186548_object30988:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:327c5565:::benchmark_data_smithi194_186548_object23927:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:327fb560:::benchmark_data_smithi194_186548_object18086:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32800e12:::benchmark_data_smithi194_186548_object83498:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3280d627:::benchmark_data_smithi194_186548_object62135:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32815ea4:::benchmark_data_smithi194_186548_object38476:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3281b5ff:::benchmark_data_smithi194_186548_object2667:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3281c0f5:::benchmark_data_smithi194_186548_object65808:head# 2024-04-29T14:53:53.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32824d56:::benchmark_data_smithi194_186548_object18559:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32828f9b:::benchmark_data_smithi194_186548_object84070:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3282d730:::benchmark_data_smithi194_186548_object10625:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3283fa21:::benchmark_data_smithi194_186548_object49140:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3285707b:::benchmark_data_smithi194_186548_object55649:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3285deb5:::benchmark_data_smithi194_186548_object58081:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32877148:::benchmark_data_smithi194_186548_object3209:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328794ce:::benchmark_data_smithi194_186548_object44066:head# 2024-04-29T14:53:53.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328881fd:::benchmark_data_smithi194_186548_object5348:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3289234f:::benchmark_data_smithi194_186548_object35311:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32895935:::benchmark_data_smithi194_186548_object65217:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328a0797:::benchmark_data_smithi194_186548_object51593:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328a7532:::benchmark_data_smithi194_186548_object24707:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328b9524:::benchmark_data_smithi194_186548_object67707:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328be3c2:::benchmark_data_smithi194_186548_object19484:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328c08b7:::benchmark_data_smithi194_186548_object44976:head# 2024-04-29T14:53:53.884 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328cdba3:::benchmark_data_smithi194_186548_object74800:head# 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328e3d1e:::benchmark_data_smithi194_186548_object53041:head# 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328e405b:::benchmark_data_smithi194_186548_object33241:head# 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:328eb39f:::benchmark_data_smithi194_186548_object46784:head# 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329080d9:::benchmark_data_smithi194_186548_object36665:head# 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3291cc7e:::benchmark_data_smithi194_186548_object19819:head# 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329258ff:::benchmark_data_smithi194_186548_object31838:head# 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.885 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3292e3c0:::benchmark_data_smithi194_186548_object14223:head# 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32932128:::benchmark_data_smithi194_186548_object64918:head# 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32949ccd:::benchmark_data_smithi194_186548_object35462:head# 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3294ce70:::benchmark_data_smithi194_186548_object43171:head# 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32954d4c:::benchmark_data_smithi194_186548_object72610:head# 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3296494b:::benchmark_data_smithi194_186548_object68940:head# 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32969d83:::benchmark_data_smithi194_186548_object51751:head# 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.886 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3296a31e:::benchmark_data_smithi194_186548_object57024:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329754a9:::benchmark_data_smithi194_186548_object13168:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3297aaab:::benchmark_data_smithi194_186548_object65936:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3297ee6d:::benchmark_data_smithi194_186548_object71206:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32989443:::benchmark_data_smithi194_186548_object60211:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3299808f:::benchmark_data_smithi194_186548_object38236:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3299e726:::benchmark_data_smithi194_186548_object70653:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329a0f31:::benchmark_data_smithi194_186548_object69902:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329bd8c0:::benchmark_data_smithi194_186548_object17405:head# 2024-04-29T14:53:53.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329c3fe2:::benchmark_data_smithi194_186548_object72608:head# 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329d341b:::benchmark_data_smithi194_186548_object65311:head# 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329d4d4d:::benchmark_data_smithi194_186548_object24392:head# 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329ddd76:::benchmark_data_smithi194_186548_object85281:head# 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:329fe143:::benchmark_data_smithi194_186548_object11781:head# 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a09fbf:::benchmark_data_smithi194_186548_object1003:head# 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a0b69f:::benchmark_data_smithi194_186548_object26222:head# 2024-04-29T14:53:53.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:53.916 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a0cd85:::benchmark_data_smithi194_186548_object74737:head# 2024-04-29T14:53:53.916 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.916 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a15f43:::benchmark_data_smithi194_186548_object7636:head# 2024-04-29T14:53:53.916 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a1cb1a:::benchmark_data_smithi194_186548_object1050:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a36cdc:::benchmark_data_smithi194_186548_object78099:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a3dd55:::benchmark_data_smithi194_186548_object41920:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a4ca70:::benchmark_data_smithi194_186548_object14898:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a5d1e5:::benchmark_data_smithi194_186548_object35199:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a5e8d2:::benchmark_data_smithi194_186548_object50402:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a6054b:::benchmark_data_smithi194_186548_object52730:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a7cc01:::benchmark_data_smithi194_186548_object65380:head# 2024-04-29T14:53:53.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a825da:::benchmark_data_smithi194_186548_object67617:head# 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a8dfde:::benchmark_data_smithi194_186548_object4009:head# 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32a9dfa0:::benchmark_data_smithi194_186548_object39044:head# 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32aa6670:::benchmark_data_smithi194_186548_object32465:head# 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32aabbba:::benchmark_data_smithi194_186548_object11779:head# 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ab9a73:::benchmark_data_smithi194_186548_object78700:head# 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32abe0bf:::benchmark_data_smithi194_186548_object10910:head# 2024-04-29T14:53:53.918 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ac03b6:::benchmark_data_smithi194_186548_object54299:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b102a3:::benchmark_data_smithi194_186548_object55941:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b1689a:::benchmark_data_smithi194_186548_object37136:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b20c62:::benchmark_data_smithi194_186548_object20573:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b24677:::benchmark_data_smithi194_186548_object42912:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b27151:::benchmark_data_smithi194_186548_object7970:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b4f5c5:::benchmark_data_smithi194_186548_object84556:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b4f603:::benchmark_data_smithi194_186548_object80061:head# 2024-04-29T14:53:53.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b64eb2:::benchmark_data_smithi194_186548_object9014:head# 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b6a713:::benchmark_data_smithi194_186548_object25835:head# 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b6d3f0:::benchmark_data_smithi194_186548_object8731:head# 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b70f17:::benchmark_data_smithi194_186548_object1388:head# 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b71e28:::benchmark_data_smithi194_186548_object7614:head# 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b73d58:::benchmark_data_smithi194_186548_object69462:head# 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b78398:::benchmark_data_smithi194_186548_object59117:head# 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b82791:::benchmark_data_smithi194_186548_object63354:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b830a9:::benchmark_data_smithi194_186548_object21418:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b925de:::benchmark_data_smithi194_186548_object13359:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b95388:::benchmark_data_smithi194_186548_object81150:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32b96c78:::benchmark_data_smithi194_186548_object76352:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32bb1584:::benchmark_data_smithi194_186548_object1517:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32bcb7e3:::benchmark_data_smithi194_186548_object48790:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32bce8ce:::benchmark_data_smithi194_186548_object18650:head# 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32bd3b0e:::benchmark_data_smithi194_186548_object46:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32be00e4:::benchmark_data_smithi194_186548_object22335:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32bf3fe3:::benchmark_data_smithi194_186548_object77470:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c2fb2c:::benchmark_data_smithi194_186548_object17127:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c31223:::benchmark_data_smithi194_186548_object54567:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c379fe:::benchmark_data_smithi194_186548_object59749:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c3b8ea:::benchmark_data_smithi194_186548_object65434:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c4d676:::benchmark_data_smithi194_186548_object40671:head# 2024-04-29T14:53:53.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c743de:::benchmark_data_smithi194_186548_object29260:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c8c2c4:::benchmark_data_smithi194_186548_object59756:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32c8ed1a:::benchmark_data_smithi194_186548_object57249:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ca8e9c:::benchmark_data_smithi194_186548_object63871:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32cb0a9d:::benchmark_data_smithi194_186548_object79821:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32cb32f8:::benchmark_data_smithi194_186548_object42817:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32cbadb0:::benchmark_data_smithi194_186548_object145:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32cd10e8:::benchmark_data_smithi194_186548_object68810:head# 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32cd40cb:::benchmark_data_smithi194_186548_object10879:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32cd876a:::benchmark_data_smithi194_186548_object43820:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ceb591:::benchmark_data_smithi194_186548_object980:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32cf0a75:::benchmark_data_smithi194_186548_object46752:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d062f5:::benchmark_data_smithi194_186548_object29738:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d0edfe:::benchmark_data_smithi194_186548_object28027:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d12a04:::benchmark_data_smithi194_186548_object60095:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d1f0cc:::benchmark_data_smithi194_186548_object18195:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d33817:::benchmark_data_smithi194_186548_object4584:head# 2024-04-29T14:53:53.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d37b7f:::benchmark_data_smithi194_186548_object21266:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d47934:::benchmark_data_smithi194_186548_object34050:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d49f84:::benchmark_data_smithi194_186548_object50042:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d4a291:::benchmark_data_smithi194_186548_object4806:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d683bc:::benchmark_data_smithi194_186548_object80149:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d6df57:::benchmark_data_smithi194_186548_object41215:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d79a62:::benchmark_data_smithi194_186548_object460:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d8826e:::benchmark_data_smithi194_186548_object73537:head# 2024-04-29T14:53:53.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d893c3:::benchmark_data_smithi194_186548_object47433:head# 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d90d27:::benchmark_data_smithi194_186548_object23294:head# 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d9124e:::benchmark_data_smithi194_186548_object67258:head# 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32d94791:::benchmark_data_smithi194_186548_object28981:head# 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32da4131:::benchmark_data_smithi194_186548_object46155:head# 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32da6cc7:::benchmark_data_smithi194_186548_object61368:head# 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32de1328:::benchmark_data_smithi194_186548_object63023:head# 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32def3a8:::benchmark_data_smithi194_186548_object65089:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32defd84:::benchmark_data_smithi194_186548_object52715:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e04f2c:::benchmark_data_smithi194_186548_object3685:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e0deb3:::benchmark_data_smithi194_186548_object29886:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e107ea:::benchmark_data_smithi194_186548_object41788:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e1bc8f:::benchmark_data_smithi194_186548_object61873:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e1f19a:::benchmark_data_smithi194_186548_object52070:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e20bc4:::benchmark_data_smithi194_186548_object56049:head# 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.927 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e3354f:::benchmark_data_smithi194_186548_object11770:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e3ece2:::benchmark_data_smithi194_186548_object68108:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e45c5b:::benchmark_data_smithi194_186548_object78214:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e56849:::benchmark_data_smithi194_186548_object63379:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e66ed9:::benchmark_data_smithi194_186548_object84803:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e69263:::benchmark_data_smithi194_186548_object82771:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e6991f:::benchmark_data_smithi194_186548_object48305:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e6d850:::benchmark_data_smithi194_186548_object23289:head# 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.928 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e86981:::benchmark_data_smithi194_186548_object53414:head# 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32e9b46b:::benchmark_data_smithi194_186548_object66786:head# 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ed9176:::benchmark_data_smithi194_186548_object78153:head# 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ee069c:::benchmark_data_smithi194_186548_object57716:head# 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ef3f16:::benchmark_data_smithi194_186548_object67119:head# 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ef660e:::benchmark_data_smithi194_186548_object78956:head# 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ef8747:::benchmark_data_smithi194_186548_object52002:head# 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32efbb1d:::benchmark_data_smithi194_186548_object54361:head# 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f01f95:::benchmark_data_smithi194_186548_object42766:head# 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f2708e:::benchmark_data_smithi194_186548_object33412:head# 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f30a26:::benchmark_data_smithi194_186548_object33390:head# 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f33921:::benchmark_data_smithi194_186548_object84604:head# 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f35099:::benchmark_data_smithi194_186548_object47591:head# 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f37a79:::benchmark_data_smithi194_186548_object4358:head# 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f45f19:::benchmark_data_smithi194_186548_object45023:head# 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f462b1:::benchmark_data_smithi194_186548_object63068:head# 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f4ab36:::benchmark_data_smithi194_186548_object78484:head# 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f5cded:::benchmark_data_smithi194_186548_object62493:head# 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f5fb24:::benchmark_data_smithi194_186548_object55190:head# 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f625e6:::benchmark_data_smithi194_186548_object84586:head# 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f704fc:::benchmark_data_smithi194_186548_object49372:head# 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32f9b213:::benchmark_data_smithi194_186548_object73280:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32fdb283:::benchmark_data_smithi194_186548_object30845:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32fe636a:::benchmark_data_smithi194_186548_object11295:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ff5208:::benchmark_data_smithi194_186548_object60251:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:32ff925b:::benchmark_data_smithi194_186548_object6358:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3300d73b:::benchmark_data_smithi194_186548_object10811:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33018341:::benchmark_data_smithi194_186548_object56992:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33027b1d:::benchmark_data_smithi194_186548_object60464:head# 2024-04-29T14:53:53.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33033513:::benchmark_data_smithi194_186548_object64381:head# 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33036836:::benchmark_data_smithi194_186548_object82114:head# 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330396cf:::benchmark_data_smithi194_186548_object8932:head# 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3303edb4:::benchmark_data_smithi194_186548_object80047:head# 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3304c944:::benchmark_data_smithi194_186548_object26579:head# 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3304cde9:::benchmark_data_smithi194_186548_object69048:head# 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33056ff4:::benchmark_data_smithi194_186548_object62637:head# 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3306d4ee:::benchmark_data_smithi194_186548_object66507:head# 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330a8f51:::benchmark_data_smithi194_186548_object39699:head# 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330ab92f:::benchmark_data_smithi194_186548_object76447:head# 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330ac7ab:::benchmark_data_smithi194_186548_object65559:head# 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330d519d:::benchmark_data_smithi194_186548_object60884:head# 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330d9d22:::benchmark_data_smithi194_186548_object10170:head# 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330ea4c0:::benchmark_data_smithi194_186548_object80961:head# 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330ed23b:::benchmark_data_smithi194_186548_object41499:head# 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330f2522:::benchmark_data_smithi194_186548_object81383:head# 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330f5099:::benchmark_data_smithi194_186548_object64779:head# 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330f632c:::benchmark_data_smithi194_186548_object13921:head# 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:330ff981:::benchmark_data_smithi194_186548_object54939:head# 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33104af4:::benchmark_data_smithi194_186548_object5463:head# 2024-04-29T14:53:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331449d1:::benchmark_data_smithi194_186548_object51720:head# 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331593f9:::benchmark_data_smithi194_186548_object19280:head# 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3315b058:::benchmark_data_smithi194_186548_object32599:head# 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3315d335:::benchmark_data_smithi194_186548_object16627:head# 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3315d5cd:::benchmark_data_smithi194_186548_object5458:head# 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33199299:::benchmark_data_smithi194_186548_object25179:head# 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33199745:::benchmark_data_smithi194_186548_object43903:head# 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331b768a:::benchmark_data_smithi194_186548_object1594:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331c26a2:::benchmark_data_smithi194_186548_object48407:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331c384a:::benchmark_data_smithi194_186548_object5887:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331c5905:::benchmark_data_smithi194_186548_object20808:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331d8833:::benchmark_data_smithi194_186548_object78136:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331dd745:::benchmark_data_smithi194_186548_object43737:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331e0bec:::benchmark_data_smithi194_186548_object43279:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331fb178:::benchmark_data_smithi194_186548_object35729:head# 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:331fe400:::benchmark_data_smithi194_186548_object20619:head# 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3320967d:::benchmark_data_smithi194_186548_object36759:head# 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3321ca7d:::benchmark_data_smithi194_186548_object76402:head# 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3322098c:::benchmark_data_smithi194_186548_object49732:head# 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332213cb:::benchmark_data_smithi194_186548_object16468:head# 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3322a6fb:::benchmark_data_smithi194_186548_object3589:head# 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3322b3ed:::benchmark_data_smithi194_186548_object30058:head# 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3322dc1b:::benchmark_data_smithi194_186548_object72510:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3323082e:::benchmark_data_smithi194_186548_object82458:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33230cd0:::benchmark_data_smithi194_186548_object46905:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3324145c:::benchmark_data_smithi194_186548_object29096:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3324382a:::benchmark_data_smithi194_186548_object51191:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3324bab3:::benchmark_data_smithi194_186548_object20233:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33255508:::benchmark_data_smithi194_186548_object36751:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332610a6:::benchmark_data_smithi194_186548_object33401:head# 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33271b66:::benchmark_data_smithi194_186548_object70122:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33287821:::benchmark_data_smithi194_186548_object36260:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332a29eb:::benchmark_data_smithi194_186548_object79:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332a8743:::benchmark_data_smithi194_186548_object37126:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332a96ac:::benchmark_data_smithi194_186548_object33306:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332c9d8b:::benchmark_data_smithi194_186548_object2971:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332ca350:::benchmark_data_smithi194_186548_object28785:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332cafcd:::benchmark_data_smithi194_186548_object11461:head# 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332dcd32:::benchmark_data_smithi194_186548_object81961:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332ded8c:::benchmark_data_smithi194_186548_object11291:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:332fc83e:::benchmark_data_smithi194_186548_object40697:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3330a6a9:::benchmark_data_smithi194_186548_object69591:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3332707e:::benchmark_data_smithi194_186548_object34228:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3332c401:::benchmark_data_smithi194_186548_object25524:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3332ed61:::benchmark_data_smithi194_186548_object51900:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33340135:::benchmark_data_smithi194_186548_object65403:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333485ea:::benchmark_data_smithi194_186548_object76182:head# 2024-04-29T14:53:53.941 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33349a16:::benchmark_data_smithi194_186548_object36564:head# 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3334c319:::benchmark_data_smithi194_186548_object23611:head# 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3335c105:::benchmark_data_smithi194_186548_object21394:head# 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33366410:::benchmark_data_smithi194_186548_object11085:head# 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333773ed:::benchmark_data_smithi194_186548_object53447:head# 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3337f47c:::benchmark_data_smithi194_186548_object66056:head# 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3337fa4a:::benchmark_data_smithi194_186548_object72096:head# 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3338a6bf:::benchmark_data_smithi194_186548_object50460:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33394df0:::benchmark_data_smithi194_186548_object85034:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333a02b8:::benchmark_data_smithi194_186548_object50056:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333a517e:::benchmark_data_smithi194_186548_object48903:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333a7789:::benchmark_data_smithi194_186548_object35494:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333a7d39:::benchmark_data_smithi194_186548_object26884:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333aaea4:::benchmark_data_smithi194_186548_object50642:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333abda6:::benchmark_data_smithi194_186548_object82349:head# 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333ba326:::benchmark_data_smithi194_186548_object20348:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333bd8ba:::benchmark_data_smithi194_186548_object41302:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:333fd17e:::benchmark_data_smithi194_186548_object53601:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3340387a:::benchmark_data_smithi194_186548_object62857:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33405925:::benchmark_data_smithi194_186548_object22839:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3340ebb1:::benchmark_data_smithi194_186548_object66714:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3341fe63:::benchmark_data_smithi194_186548_object2613:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33425253:::benchmark_data_smithi194_186548_object54407:head# 2024-04-29T14:53:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33426e41:::benchmark_data_smithi194_186548_object5516:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3342ccde:::benchmark_data_smithi194_186548_object48344:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33435c84:::benchmark_data_smithi194_186548_object9757:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33443407:::benchmark_data_smithi194_186548_object17158:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3344d115:::benchmark_data_smithi194_186548_object6578:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334516be:::benchmark_data_smithi194_186548_object17919:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334572db:::benchmark_data_smithi194_186548_object19579:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3345cc9a:::benchmark_data_smithi194_186548_object4589:head# 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3346787b:::benchmark_data_smithi194_186548_object1413:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3346bfb8:::benchmark_data_smithi194_186548_object22257:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33473e68:::benchmark_data_smithi194_186548_object82272:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3347c440:::benchmark_data_smithi194_186548_object14375:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334a4d36:::benchmark_data_smithi194_186548_object61827:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334a7439:::benchmark_data_smithi194_186548_object22233:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334a7b9c:::benchmark_data_smithi194_186548_object26865:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334b2ec4:::benchmark_data_smithi194_186548_object69348:head# 2024-04-29T14:53:53.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334c3bb8:::benchmark_data_smithi194_186548_object80498:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334f2828:::benchmark_data_smithi194_186548_object24629:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:334fe63a:::benchmark_data_smithi194_186548_object32898:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33512911:::benchmark_data_smithi194_186548_object7534:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33530f84:::benchmark_data_smithi194_186548_object25731:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33545d10:::benchmark_data_smithi194_186548_object80007:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3354ad36:::benchmark_data_smithi194_186548_object75052:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3354ed29:::benchmark_data_smithi194_186548_object64142:head# 2024-04-29T14:53:53.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33550a02:::benchmark_data_smithi194_186548_object33139:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335559bf:::benchmark_data_smithi194_186548_object40602:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33557804:::benchmark_data_smithi194_186548_object9268:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3358e9ab:::benchmark_data_smithi194_186548_object8672:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335968a2:::benchmark_data_smithi194_186548_object11147:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335975fe:::benchmark_data_smithi194_186548_object73400:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33597a5c:::benchmark_data_smithi194_186548_object3821:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3359c314:::benchmark_data_smithi194_186548_object30337:head# 2024-04-29T14:53:53.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335b2815:::benchmark_data_smithi194_186548_object81892:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335d791a:::benchmark_data_smithi194_186548_object37544:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335df52b:::benchmark_data_smithi194_186548_object28783:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335e390e:::benchmark_data_smithi194_186548_object25037:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335e7f44:::benchmark_data_smithi194_186548_object38031:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335e9669:::benchmark_data_smithi194_186548_object78970:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335ea17d:::benchmark_data_smithi194_186548_object81729:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335f56a9:::benchmark_data_smithi194_186548_object30011:head# 2024-04-29T14:53:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:335fe782:::benchmark_data_smithi194_186548_object10535:head# 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3360f0fa:::benchmark_data_smithi194_186548_object44743:head# 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33611842:::benchmark_data_smithi194_186548_object3329:head# 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3361a846:::benchmark_data_smithi194_186548_object62807:head# 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33625ff3:::benchmark_data_smithi194_186548_object68915:head# 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33628686:::benchmark_data_smithi194_186548_object20188:head# 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3362c12d:::benchmark_data_smithi194_186548_object70027:head# 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.950 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33632c74:::benchmark_data_smithi194_186548_object20617:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33644258:::benchmark_data_smithi194_186548_object46676:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33652b86:::benchmark_data_smithi194_186548_object23412:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3365cec2:::benchmark_data_smithi194_186548_object19503:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3365f855:::benchmark_data_smithi194_186548_object53145:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3366cc51:::benchmark_data_smithi194_186548_object15187:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3368b2da:::benchmark_data_smithi194_186548_object80488:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336a116e:::benchmark_data_smithi194_186548_object54360:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336b098b:::benchmark_data_smithi194_186548_object14531:head# 2024-04-29T14:53:53.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336b9e79:::benchmark_data_smithi194_186548_object55219:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336c9afa:::benchmark_data_smithi194_186548_object57890:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336d16eb:::benchmark_data_smithi194_186548_object83203:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336ef86e:::benchmark_data_smithi194_186548_object22159:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336f21e0:::benchmark_data_smithi194_186548_object74021:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:336f8a57:::benchmark_data_smithi194_186548_object61829:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33707264:::benchmark_data_smithi194_186548_object18433:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33709d46:::benchmark_data_smithi194_186548_object57025:head# 2024-04-29T14:53:53.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33710478:::benchmark_data_smithi194_186548_object76997:head# 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3373cdfe:::benchmark_data_smithi194_186548_object39610:head# 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3373e53c:::benchmark_data_smithi194_186548_object21190:head# 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3373f99e:::benchmark_data_smithi194_186548_object67914:head# 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33745047:::benchmark_data_smithi194_186548_object69834:head# 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3374dbc6:::benchmark_data_smithi194_186548_object40499:head# 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33752c92:::benchmark_data_smithi194_186548_object61811:head# 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3375b14b:::benchmark_data_smithi194_186548_object9501:head# 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:337614e1:::benchmark_data_smithi194_186548_object742:head# 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33766235:::benchmark_data_smithi194_186548_object70451:head# 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3377e227:::benchmark_data_smithi194_186548_object7369:head# 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33782dee:::benchmark_data_smithi194_186548_object23209:head# 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33785366:::benchmark_data_smithi194_186548_object10295:head# 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3378b55e:::benchmark_data_smithi194_186548_object7172:head# 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33792375:::benchmark_data_smithi194_186548_object70317:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33798c4c:::benchmark_data_smithi194_186548_object21783:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:337aab06:::benchmark_data_smithi194_186548_object46933:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:337afe07:::benchmark_data_smithi194_186548_object18552:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:337db06e:::benchmark_data_smithi194_186548_object17440:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:337e024a:::benchmark_data_smithi194_186548_object12422:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:337e20ac:::benchmark_data_smithi194_186548_object82746:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:337f49fe:::benchmark_data_smithi194_186548_object62090:head# 2024-04-29T14:53:53.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3380b890:::benchmark_data_smithi194_186548_object24846:head# 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3381218e:::benchmark_data_smithi194_186548_object66591:head# 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3381bd02:::benchmark_data_smithi194_186548_object59385:head# 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33829805:::benchmark_data_smithi194_186548_object21581:head# 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3382a02a:::benchmark_data_smithi194_186548_object19550:head# 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3384bce3:::benchmark_data_smithi194_186548_object82979:head# 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3384daf0:::benchmark_data_smithi194_186548_object29936:head# 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33850762:::benchmark_data_smithi194_186548_object41992:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33858ecb:::benchmark_data_smithi194_186548_object36614:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3385aede:::benchmark_data_smithi194_186548_object49299:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3385cbde:::benchmark_data_smithi194_186548_object68372:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3387e92e:::benchmark_data_smithi194_186548_object54533:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33889029:::benchmark_data_smithi194_186548_object22172:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3388d0fe:::benchmark_data_smithi194_186548_object43509:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3388d29a:::benchmark_data_smithi194_186548_object79837:head# 2024-04-29T14:53:53.957 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3389d33d:::benchmark_data_smithi194_186548_object11123:head# 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338a2f33:::benchmark_data_smithi194_186548_object11026:head# 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338a5724:::benchmark_data_smithi194_186548_object10716:head# 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338a7022:::benchmark_data_smithi194_186548_object25193:head# 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338af070:::benchmark_data_smithi194_186548_object62987:head# 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338b25a0:::benchmark_data_smithi194_186548_object5025:head# 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338c598c:::benchmark_data_smithi194_186548_object2073:head# 2024-04-29T14:53:53.958 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338d5142:::benchmark_data_smithi194_186548_object32998:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338d7a34:::benchmark_data_smithi194_186548_object53692:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338e2abd:::benchmark_data_smithi194_186548_object44142:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338e4ce0:::benchmark_data_smithi194_186548_object60256:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:338fb119:::benchmark_data_smithi194_186548_object17239:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33908c03:::benchmark_data_smithi194_186548_object32310:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33908f5a:::benchmark_data_smithi194_186548_object71838:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3390dcbe:::benchmark_data_smithi194_186548_object72232:head# 2024-04-29T14:53:53.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33921013:::benchmark_data_smithi194_186548_object55743:head# 2024-04-29T14:53:53.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3392c9f3:::benchmark_data_smithi194_186548_object18967:head# 2024-04-29T14:53:53.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33931ad6:::benchmark_data_smithi194_186548_object73546:head# 2024-04-29T14:53:53.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33933555:::benchmark_data_smithi194_186548_object15170:head# 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:339420c3:::benchmark_data_smithi194_186548_object53379:head# 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3394bba8:::benchmark_data_smithi194_186548_object73242:head# 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3396c2ea:::benchmark_data_smithi194_186548_object14060:head# 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3396ebfd:::benchmark_data_smithi194_186548_object83375:head# 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3397f4e6:::benchmark_data_smithi194_186548_object64109:head# 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:339b5634:::benchmark_data_smithi194_186548_object30460:head# 2024-04-29T14:53:53.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:339b871d:::benchmark_data_smithi194_186548_object52338:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:339c6d5f:::benchmark_data_smithi194_186548_object4608:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:339d7007:::benchmark_data_smithi194_186548_object58835:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:339e0532:::benchmark_data_smithi194_186548_object24642:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:339fbf8a:::benchmark_data_smithi194_186548_object28348:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a11124:::benchmark_data_smithi194_186548_object15485:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a16e5c:::benchmark_data_smithi194_186548_object26565:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a31d2e:::benchmark_data_smithi194_186548_object79251:head# 2024-04-29T14:53:53.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a37ad6:::benchmark_data_smithi194_186548_object34735:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a4466c:::benchmark_data_smithi194_186548_object50665:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a4be74:::benchmark_data_smithi194_186548_object7421:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a5d2c2:::benchmark_data_smithi194_186548_object8377:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a810df:::benchmark_data_smithi194_186548_object55729:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a832a2:::benchmark_data_smithi194_186548_object77774:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a91c51:::benchmark_data_smithi194_186548_object52343:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a95b38:::benchmark_data_smithi194_186548_object48365:head# 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a996cd:::benchmark_data_smithi194_186548_object74960:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33a9a6f9:::benchmark_data_smithi194_186548_object16854:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33aa1996:::benchmark_data_smithi194_186548_object53559:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33aa2ba7:::benchmark_data_smithi194_186548_object5001:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33aa7c4f:::benchmark_data_smithi194_186548_object7523:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33aa935f:::benchmark_data_smithi194_186548_object21147:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ac9088:::benchmark_data_smithi194_186548_object54480:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ad0a70:::benchmark_data_smithi194_186548_object25028:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ad4a8b:::benchmark_data_smithi194_186548_object72293:head# 2024-04-29T14:53:53.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33adeffa:::benchmark_data_smithi194_186548_object67037:head# 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ae2e0a:::benchmark_data_smithi194_186548_object10817:head# 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33aee628:::benchmark_data_smithi194_186548_object71330:head# 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33af3399:::benchmark_data_smithi194_186548_object21188:head# 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33af5caa:::benchmark_data_smithi194_186548_object1713:head# 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b002a6:::benchmark_data_smithi194_186548_object41612:head# 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b0ff53:::benchmark_data_smithi194_186548_object69607:head# 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b14c33:::benchmark_data_smithi194_186548_object47593:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b1be02:::benchmark_data_smithi194_186548_object53585:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b1d1a1:::benchmark_data_smithi194_186548_object75539:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b1dc56:::benchmark_data_smithi194_186548_object84956:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b2ccc8:::benchmark_data_smithi194_186548_object56532:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b5507a:::benchmark_data_smithi194_186548_object6371:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b6ff15:::benchmark_data_smithi194_186548_object31521:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b85608:::benchmark_data_smithi194_186548_object45575:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b8f497:::benchmark_data_smithi194_186548_object83627:head# 2024-04-29T14:53:53.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33b9ed15:::benchmark_data_smithi194_186548_object71714:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ba244b:::benchmark_data_smithi194_186548_object7221:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33baa2fb:::benchmark_data_smithi194_186548_object35894:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33bac3b9:::benchmark_data_smithi194_186548_object45773:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33baca0c:::benchmark_data_smithi194_186548_object55415:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33bc6f03:::benchmark_data_smithi194_186548_object55050:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33bc9cfd:::benchmark_data_smithi194_186548_object14748:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33bcb10d:::benchmark_data_smithi194_186548_object17457:head# 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33bf2d89:::benchmark_data_smithi194_186548_object75213:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33bf9015:::benchmark_data_smithi194_186548_object373:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33bfd36b:::benchmark_data_smithi194_186548_object7082:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c12066:::benchmark_data_smithi194_186548_object28013:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c1f5c0:::benchmark_data_smithi194_186548_object70816:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c23996:::benchmark_data_smithi194_186548_object23107:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c2e3f5:::benchmark_data_smithi194_186548_object64154:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c2fed1:::benchmark_data_smithi194_186548_object54234:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c3c314:::benchmark_data_smithi194_186548_object71503:head# 2024-04-29T14:53:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c4b413:::benchmark_data_smithi194_186548_object32533:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c506c7:::benchmark_data_smithi194_186548_object69171:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c58c3d:::benchmark_data_smithi194_186548_object40276:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c8a10e:::benchmark_data_smithi194_186548_object47681:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c96a1a:::benchmark_data_smithi194_186548_object32009:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c97031:::benchmark_data_smithi194_186548_object37073:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c9ac13:::benchmark_data_smithi194_186548_object72958:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33c9e220:::benchmark_data_smithi194_186548_object31285:head# 2024-04-29T14:53:53.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33caec0d:::benchmark_data_smithi194_186548_object53771:head# 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33caf705:::benchmark_data_smithi194_186548_object56730:head# 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ccab80:::benchmark_data_smithi194_186548_object1163:head# 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d11c04:::benchmark_data_smithi194_186548_object33544:head# 2024-04-29T14:53:53.975 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d1298d:::benchmark_data_smithi194_186548_object66670:head# 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d1cfc8:::benchmark_data_smithi194_186548_object39231:head# 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d2cea7:::benchmark_data_smithi194_186548_object11343:head# 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d463b5:::benchmark_data_smithi194_186548_object41897:head# 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d541bf:::benchmark_data_smithi194_186548_object59127:head# 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d5db30:::benchmark_data_smithi194_186548_object33762:head# 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d62026:::benchmark_data_smithi194_186548_object66148:head# 2024-04-29T14:53:53.976 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d63ae2:::benchmark_data_smithi194_186548_object39644:head# 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d6433f:::benchmark_data_smithi194_186548_object56271:head# 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d6b3c8:::benchmark_data_smithi194_186548_object75207:head# 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d6e53a:::benchmark_data_smithi194_186548_object41666:head# 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d7a864:::benchmark_data_smithi194_186548_object19781:head# 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d9004e:::benchmark_data_smithi194_186548_object37693:head# 2024-04-29T14:53:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.978 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33d9b15e:::benchmark_data_smithi194_186548_object39004:head# 2024-04-29T14:53:53.978 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.978 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33da8e22:::benchmark_data_smithi194_186548_object47662:head# 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33db055b:::benchmark_data_smithi194_186548_object56972:head# 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33db7f5a:::benchmark_data_smithi194_186548_object66003:head# 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33dc108e:::benchmark_data_smithi194_186548_object37150:head# 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33dd4c5b:::benchmark_data_smithi194_186548_object28960:head# 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33de4966:::benchmark_data_smithi194_186548_object69342:head# 2024-04-29T14:53:53.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33df9beb:::benchmark_data_smithi194_186548_object62718:head# 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33dfefdf:::benchmark_data_smithi194_186548_object70492:head# 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33dff53d:::benchmark_data_smithi194_186548_object77763:head# 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e06feb:::benchmark_data_smithi194_186548_object5768:head# 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e0813c:::benchmark_data_smithi194_186548_object50946:head# 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e10e9f:::benchmark_data_smithi194_186548_object58476:head# 2024-04-29T14:53:53.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e14ab9:::benchmark_data_smithi194_186548_object2157:head# 2024-04-29T14:53:53.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e1f38e:::benchmark_data_smithi194_186548_object53864:head# 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e25b9c:::benchmark_data_smithi194_186548_object48167:head# 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e314d2:::benchmark_data_smithi194_186548_object34081:head# 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e3e11c:::benchmark_data_smithi194_186548_object81444:head# 2024-04-29T14:53:53.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e59543:::benchmark_data_smithi194_186548_object41154:head# 2024-04-29T14:53:53.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e75153:::benchmark_data_smithi194_186548_object80553:head# 2024-04-29T14:53:53.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e7a917:::benchmark_data_smithi194_186548_object18160:head# 2024-04-29T14:53:53.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e7b0d5:::benchmark_data_smithi194_186548_object55507:head# 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e7e78e:::benchmark_data_smithi194_186548_object43611:head# 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e865cf:::benchmark_data_smithi194_186548_object54557:head# 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e8ad1d:::benchmark_data_smithi194_186548_object64747:head# 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e8d801:::benchmark_data_smithi194_186548_object80882:head# 2024-04-29T14:53:53.985 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.985 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33e8e312:::benchmark_data_smithi194_186548_object18082:head# 2024-04-29T14:53:53.985 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.985 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ea52e5:::benchmark_data_smithi194_186548_object44207:head# 2024-04-29T14:53:53.985 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.986 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33eb48a1:::benchmark_data_smithi194_186548_object50399:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ed4b22:::benchmark_data_smithi194_186548_object47942:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ed59ac:::benchmark_data_smithi194_186548_object78809:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ef6db8:::benchmark_data_smithi194_186548_object3108:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ef7aa8:::benchmark_data_smithi194_186548_object55173:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33eff112:::benchmark_data_smithi194_186548_object36579:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33eff1b6:::benchmark_data_smithi194_186548_object55886:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f017c4:::benchmark_data_smithi194_186548_object81412:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f257dd:::benchmark_data_smithi194_186548_object38182:head# 2024-04-29T14:53:53.987 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.988 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f2a61b:::benchmark_data_smithi194_186548_object81753:head# 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f2f9d7:::benchmark_data_smithi194_186548_object69565:head# 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f5e074:::benchmark_data_smithi194_186548_object10420:head# 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f69c2a:::benchmark_data_smithi194_186548_object64123:head# 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f7093f:::benchmark_data_smithi194_186548_object1881:head# 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f8bf5c:::benchmark_data_smithi194_186548_object70611:head# 2024-04-29T14:53:53.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f8dd80:::benchmark_data_smithi194_186548_object21242:head# 2024-04-29T14:53:53.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f9452d:::benchmark_data_smithi194_186548_object70849:head# 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33f9fc88:::benchmark_data_smithi194_186548_object61457:head# 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33fa6cc7:::benchmark_data_smithi194_186548_object83099:head# 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33faaddc:::benchmark_data_smithi194_186548_object26463:head# 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33fcde1b:::benchmark_data_smithi194_186548_object66509:head# 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33fd24a2:::benchmark_data_smithi194_186548_object17274:head# 2024-04-29T14:53:53.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33fd8779:::benchmark_data_smithi194_186548_object51155:head# 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33fd8a49:::benchmark_data_smithi194_186548_object1849:head# 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33fdc713:::benchmark_data_smithi194_186548_object10752:head# 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33fe5613:::benchmark_data_smithi194_186548_object26328:head# 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ff94f1:::benchmark_data_smithi194_186548_object37517:head# 2024-04-29T14:53:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ff9e79:::benchmark_data_smithi194_186548_object13558:head# 2024-04-29T14:53:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:33ffe149:::benchmark_data_smithi194_186548_object36750:head# 2024-04-29T14:53:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3400ffe9:::benchmark_data_smithi194_186548_object38763:head# 2024-04-29T14:53:53.994 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.994 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34033664:::benchmark_data_smithi194_186548_object7927:head# 2024-04-29T14:53:53.994 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.994 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34042c26:::benchmark_data_smithi194_186548_object16955:head# 2024-04-29T14:53:53.994 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34063975:::benchmark_data_smithi194_186548_object23368:head# 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340649d0:::benchmark_data_smithi194_186548_object41918:head# 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3406e191:::benchmark_data_smithi194_186548_object12346:head# 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340717cc:::benchmark_data_smithi194_186548_object19557:head# 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34071947:::benchmark_data_smithi194_186548_object48134:head# 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3408ba50:::benchmark_data_smithi194_186548_object64914:head# 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3408e252:::benchmark_data_smithi194_186548_object21538:head# 2024-04-29T14:53:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340a9774:::benchmark_data_smithi194_186548_object84876:head# 2024-04-29T14:53:53.997 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340b6970:::benchmark_data_smithi194_186548_object79620:head# 2024-04-29T14:53:53.997 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340c7a7f:::benchmark_data_smithi194_186548_object84502:head# 2024-04-29T14:53:53.997 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340d2caa:::benchmark_data_smithi194_186548_object11034:head# 2024-04-29T14:53:53.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.998 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340d3c7d:::benchmark_data_smithi194_186548_object76477:head# 2024-04-29T14:53:53.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.998 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340d4ebb:::benchmark_data_smithi194_186548_object41985:head# 2024-04-29T14:53:53.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:53.998 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340ea11e:::benchmark_data_smithi194_186548_object3450:head# 2024-04-29T14:53:53.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:340ff022:::benchmark_data_smithi194_186548_object58708:head# 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34119443:::benchmark_data_smithi194_186548_object6652:head# 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34134734:::benchmark_data_smithi194_186548_object10620:head# 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34134fbd:::benchmark_data_smithi194_186548_object39325:head# 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3413b465:::benchmark_data_smithi194_186548_object19908:head# 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3413bfe4:::benchmark_data_smithi194_186548_object79583:head# 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3413fd34:::benchmark_data_smithi194_186548_object56033:head# 2024-04-29T14:53:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34142a67:::benchmark_data_smithi194_186548_object74980:head# 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34154c15:::benchmark_data_smithi194_186548_object12076:head# 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34157347:::benchmark_data_smithi194_186548_object71536:head# 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3416613d:::benchmark_data_smithi194_186548_object30965:head# 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:341735cf:::benchmark_data_smithi194_186548_object8762:head# 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3418cf2a:::benchmark_data_smithi194_186548_object27936:head# 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34191c85:::benchmark_data_smithi194_186548_object85074:head# 2024-04-29T14:53:54.004 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.006 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3419a2ff:::benchmark_data_smithi194_186548_object76196:head# 2024-04-29T14:53:54.006 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.006 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:341ac497:::benchmark_data_smithi194_186548_object67461:head# 2024-04-29T14:53:54.006 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:341b9e38:::benchmark_data_smithi194_186548_object82549:head# 2024-04-29T14:53:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:341ceaf7:::benchmark_data_smithi194_186548_object21894:head# 2024-04-29T14:53:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:341da6bc:::benchmark_data_smithi194_186548_object29113:head# 2024-04-29T14:53:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:341dea9d:::benchmark_data_smithi194_186548_object37331:head# 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:341ea21d:::benchmark_data_smithi194_186548_object21838:head# 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34202e16:::benchmark_data_smithi194_186548_object45723:head# 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342063f8:::benchmark_data_smithi194_186548_object36688:head# 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3420900c:::benchmark_data_smithi194_186548_object38029:head# 2024-04-29T14:53:54.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3421e577:::benchmark_data_smithi194_186548_object13472:head# 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34220ef4:::benchmark_data_smithi194_186548_object30720:head# 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34231007:::benchmark_data_smithi194_186548_object69386:head# 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342443b6:::benchmark_data_smithi194_186548_object38004:head# 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34253d77:::benchmark_data_smithi194_186548_object80652:head# 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34253e0b:::benchmark_data_smithi194_186548_object18529:head# 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342562b6:::benchmark_data_smithi194_186548_object55657:head# 2024-04-29T14:53:54.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.022 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3426ba6d:::benchmark_data_smithi194_186548_object82211:head# 2024-04-29T14:53:54.022 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.022 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3428497e:::benchmark_data_smithi194_186548_object76626:head# 2024-04-29T14:53:54.022 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.023 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342879d3:::benchmark_data_smithi194_186548_object6706:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3428b54e:::benchmark_data_smithi194_186548_object59522:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3429f700:::benchmark_data_smithi194_186548_object43590:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342ab245:::benchmark_data_smithi194_186548_object72752:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342c98d2:::benchmark_data_smithi194_186548_object57184:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342cf760:::benchmark_data_smithi194_186548_object12874:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342d2394:::benchmark_data_smithi194_186548_object57985:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342d5690:::benchmark_data_smithi194_186548_object25923:head# 2024-04-29T14:53:54.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342d9f65:::benchmark_data_smithi194_186548_object48911:head# 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:342e41b9:::benchmark_data_smithi194_186548_object34800:head# 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3430d18d:::benchmark_data_smithi194_186548_object36605:head# 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34313111:::benchmark_data_smithi194_186548_object53328:head# 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34317a15:::benchmark_data_smithi194_186548_object38358:head# 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34322c3a:::benchmark_data_smithi194_186548_object13607:head# 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3433364c:::benchmark_data_smithi194_186548_object4078:head# 2024-04-29T14:53:54.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.027 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34334ca0:::benchmark_data_smithi194_186548_object10453:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34335d98:::benchmark_data_smithi194_186548_object28113:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3435844c:::benchmark_data_smithi194_186548_object66197:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3435c94d:::benchmark_data_smithi194_186548_object56316:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3436176e:::benchmark_data_smithi194_186548_object61216:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:343962d5:::benchmark_data_smithi194_186548_object25335:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34398a39:::benchmark_data_smithi194_186548_object16258:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:343a8d43:::benchmark_data_smithi194_186548_object58397:head# 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.028 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:343aea4e:::benchmark_data_smithi194_186548_object51672:head# 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:343c9421:::benchmark_data_smithi194_186548_object15776:head# 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:343ce4da:::benchmark_data_smithi194_186548_object37541:head# 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:343d101a:::benchmark_data_smithi194_186548_object39474:head# 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:343f537f:::benchmark_data_smithi194_186548_object32764:head# 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3442b331:::benchmark_data_smithi194_186548_object33059:head# 2024-04-29T14:53:54.029 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34430185:::benchmark_data_smithi194_186548_object27330:head# 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34432c02:::benchmark_data_smithi194_186548_object58008:head# 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3443d87b:::benchmark_data_smithi194_186548_object60392:head# 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34442c0d:::benchmark_data_smithi194_186548_object49058:head# 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344523e5:::benchmark_data_smithi194_186548_object80651:head# 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34476834:::benchmark_data_smithi194_186548_object14011:head# 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3448fb2f:::benchmark_data_smithi194_186548_object58070:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344960c4:::benchmark_data_smithi194_186548_object52256:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344a0125:::benchmark_data_smithi194_186548_object25909:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344a321a:::benchmark_data_smithi194_186548_object69585:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344b1f7e:::benchmark_data_smithi194_186548_object40184:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344bc275:::benchmark_data_smithi194_186548_object73984:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344beaa3:::benchmark_data_smithi194_186548_object37098:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344d6e5d:::benchmark_data_smithi194_186548_object52141:head# 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344d99cb:::benchmark_data_smithi194_186548_object71302:head# 2024-04-29T14:53:54.034 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.034 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344f5b6e:::benchmark_data_smithi194_186548_object72501:head# 2024-04-29T14:53:54.034 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.034 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:344fbce6:::benchmark_data_smithi194_186548_object73027:head# 2024-04-29T14:53:54.034 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.034 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345021a6:::benchmark_data_smithi194_186548_object77876:head# 2024-04-29T14:53:54.034 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3450bc93:::benchmark_data_smithi194_186548_object17618:head# 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34521ddd:::benchmark_data_smithi194_186548_object7548:head# 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34526fe9:::benchmark_data_smithi194_186548_object15108:head# 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34536286:::benchmark_data_smithi194_186548_object75989:head# 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34540608:::benchmark_data_smithi194_186548_object63669:head# 2024-04-29T14:53:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345522cb:::benchmark_data_smithi194_186548_object52400:head# 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34563bcf:::benchmark_data_smithi194_186548_object73583:head# 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345810d2:::benchmark_data_smithi194_186548_object20910:head# 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34595865:::benchmark_data_smithi194_186548_object64474:head# 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34598399:::benchmark_data_smithi194_186548_object26458:head# 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3459fe63:::benchmark_data_smithi194_186548_object82738:head# 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345b3d4b:::benchmark_data_smithi194_186548_object44898:head# 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.038 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345c0240:::benchmark_data_smithi194_186548_object83055:head# 2024-04-29T14:53:54.039 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.041 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345cdd46:::benchmark_data_smithi194_186548_object79334:head# 2024-04-29T14:53:54.041 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.043 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345ced10:::benchmark_data_smithi194_186548_object66328:head# 2024-04-29T14:53:54.043 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.043 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345d1ea6:::benchmark_data_smithi194_186548_object2604:head# 2024-04-29T14:53:54.043 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.043 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345e9ead:::benchmark_data_smithi194_186548_object51376:head# 2024-04-29T14:53:54.043 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:345fb9f7:::benchmark_data_smithi194_186548_object48966:head# 2024-04-29T14:53:54.044 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34608bbf:::benchmark_data_smithi194_186548_object39871:head# 2024-04-29T14:53:54.045 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.045 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3460c6e2:::benchmark_data_smithi194_186548_object23616:head# 2024-04-29T14:53:54.045 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.045 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3461255f:::benchmark_data_smithi194_186548_object12778:head# 2024-04-29T14:53:54.045 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.047 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34614c44:::benchmark_data_smithi194_186548_object27494:head# 2024-04-29T14:53:54.048 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.048 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346215dd:::benchmark_data_smithi194_186548_object42505:head# 2024-04-29T14:53:54.048 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.048 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34626cab:::benchmark_data_smithi194_186548_object20482:head# 2024-04-29T14:53:54.048 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.049 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34627e0e:::benchmark_data_smithi194_186548_object42242:head# 2024-04-29T14:53:54.049 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3465ba09:::benchmark_data_smithi194_186548_object33349:head# 2024-04-29T14:53:54.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34663fea:::benchmark_data_smithi194_186548_object56561:head# 2024-04-29T14:53:54.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3466c853:::benchmark_data_smithi194_186548_object33099:head# 2024-04-29T14:53:54.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34684950:::benchmark_data_smithi194_186548_object50225:head# 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34695e06:::benchmark_data_smithi194_186548_object10296:head# 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346a1b04:::benchmark_data_smithi194_186548_object43230:head# 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346a8a95:::benchmark_data_smithi194_186548_object6757:head# 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346bad46:::benchmark_data_smithi194_186548_object83769:head# 2024-04-29T14:53:54.051 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346bd38e:::benchmark_data_smithi194_186548_object79149:head# 2024-04-29T14:53:54.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346bef76:::benchmark_data_smithi194_186548_object43710:head# 2024-04-29T14:53:54.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346c2177:::benchmark_data_smithi194_186548_object49529:head# 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346c62ce:::benchmark_data_smithi194_186548_object20842:head# 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346d4d5b:::benchmark_data_smithi194_186548_object70158:head# 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346dcfb0:::benchmark_data_smithi194_186548_object47944:head# 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:346e234d:::benchmark_data_smithi194_186548_object9028:head# 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34710df7:::benchmark_data_smithi194_186548_object25159:head# 2024-04-29T14:53:54.056 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3472835d:::benchmark_data_smithi194_186548_object556:head# 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3472cd3b:::benchmark_data_smithi194_186548_object11340:head# 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3474a5cf:::benchmark_data_smithi194_186548_object41272:head# 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3475133b:::benchmark_data_smithi194_186548_object83554:head# 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3475b1c2:::benchmark_data_smithi194_186548_object35687:head# 2024-04-29T14:53:54.057 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.059 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347662fa:::benchmark_data_smithi194_186548_object54280:head# 2024-04-29T14:53:54.059 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.059 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347a5bfd:::benchmark_data_smithi194_186548_object1381:head# 2024-04-29T14:53:54.059 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.060 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347af2fa:::benchmark_data_smithi194_186548_object23697:head# 2024-04-29T14:53:54.060 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.060 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347be683:::benchmark_data_smithi194_186548_object21289:head# 2024-04-29T14:53:54.060 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.060 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347d2007:::benchmark_data_smithi194_186548_object32131:head# 2024-04-29T14:53:54.060 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.061 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347e9d67:::benchmark_data_smithi194_186548_object43060:head# 2024-04-29T14:53:54.061 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.061 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347f5ba4:::benchmark_data_smithi194_186548_object18231:head# 2024-04-29T14:53:54.061 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.061 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347faddb:::benchmark_data_smithi194_186548_object26446:head# 2024-04-29T14:53:54.061 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.062 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:347feca8:::benchmark_data_smithi194_186548_object77099:head# 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348058c5:::benchmark_data_smithi194_186548_object84797:head# 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3480b11c:::benchmark_data_smithi194_186548_object56218:head# 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34810b29:::benchmark_data_smithi194_186548_object80631:head# 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34817e04:::benchmark_data_smithi194_186548_object49291:head# 2024-04-29T14:53:54.063 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3481dfaa:::benchmark_data_smithi194_186548_object69763:head# 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3482e456:::benchmark_data_smithi194_186548_object77007:head# 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34838e8c:::benchmark_data_smithi194_186548_object49323:head# 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34852540:::benchmark_data_smithi194_186548_object1631:head# 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34859f1e:::benchmark_data_smithi194_186548_object74253:head# 2024-04-29T14:53:54.064 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.065 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3485b59b:::benchmark_data_smithi194_186548_object84231:head# 2024-04-29T14:53:54.065 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.065 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3486c9b8:::benchmark_data_smithi194_186548_object15765:head# 2024-04-29T14:53:54.065 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.065 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3488197c:::benchmark_data_smithi194_186548_object35492:head# 2024-04-29T14:53:54.065 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.065 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348bdef2:::benchmark_data_smithi194_186548_object57109:head# 2024-04-29T14:53:54.066 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.066 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348cdd73:::benchmark_data_smithi194_186548_object7976:head# 2024-04-29T14:53:54.066 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.066 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348d6474:::benchmark_data_smithi194_186548_object20561:head# 2024-04-29T14:53:54.067 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.067 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348de1c0:::benchmark_data_smithi194_186548_object3061:head# 2024-04-29T14:53:54.067 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.067 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348e4ad7:::benchmark_data_smithi194_186548_object5394:head# 2024-04-29T14:53:54.067 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348f7e22:::benchmark_data_smithi194_186548_object60858:head# 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:348f97fe:::benchmark_data_smithi194_186548_object21983:head# 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34900e11:::benchmark_data_smithi194_186548_object56494:head# 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34908fc3:::benchmark_data_smithi194_186548_object39616:head# 2024-04-29T14:53:54.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.072 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3490f64e:::benchmark_data_smithi194_186548_object56187:head# 2024-04-29T14:53:54.073 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:53 smithi194 ceph-mon[44089]: 8.e scrub ok 2024-04-29T14:53:54.074 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3491c415:::benchmark_data_smithi194_186548_object32726:head# 2024-04-29T14:53:54.074 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.075 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3491ea6d:::benchmark_data_smithi194_186548_object52375:head# 2024-04-29T14:53:54.075 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.076 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34924ba1:::benchmark_data_smithi194_186548_object72547:head# 2024-04-29T14:53:54.076 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.076 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3492cd04:::benchmark_data_smithi194_186548_object21988:head# 2024-04-29T14:53:54.076 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3492dc1c:::benchmark_data_smithi194_186548_object34843:head# 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3492e174:::benchmark_data_smithi194_186548_object18242:head# 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34931f53:::benchmark_data_smithi194_186548_object41698:head# 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:349360ef:::benchmark_data_smithi194_186548_object22674:head# 2024-04-29T14:53:54.077 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.079 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34936d5d:::benchmark_data_smithi194_186548_object50600:head# 2024-04-29T14:53:54.079 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.079 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3493babc:::benchmark_data_smithi194_186548_object41490:head# 2024-04-29T14:53:54.079 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.079 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34952d11:::benchmark_data_smithi194_186548_object84201:head# 2024-04-29T14:53:54.079 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3495cc9e:::benchmark_data_smithi194_186548_object40114:head# 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3495cd65:::benchmark_data_smithi194_186548_object63779:head# 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3496a421:::benchmark_data_smithi194_186548_object75064:head# 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3498220f:::benchmark_data_smithi194_186548_object23895:head# 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:349834be:::benchmark_data_smithi194_186548_object60827:head# 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34984a6a:::benchmark_data_smithi194_186548_object15433:head# 2024-04-29T14:53:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3499096f:::benchmark_data_smithi194_186548_object1550:head# 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34990995:::benchmark_data_smithi194_186548_object10850:head# 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:349a4980:::benchmark_data_smithi194_186548_object16220:head# 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:349b3c07:::benchmark_data_smithi194_186548_object40242:head# 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:349d7f8e:::benchmark_data_smithi194_186548_object33584:head# 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:349f22ad:::benchmark_data_smithi194_186548_object10069:head# 2024-04-29T14:53:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:349f959a:::benchmark_data_smithi194_186548_object33368:head# 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34a27a43:::benchmark_data_smithi194_186548_object68429:head# 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34a7a753:::benchmark_data_smithi194_186548_object67251:head# 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34a821c8:::benchmark_data_smithi194_186548_object4887:head# 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34a8d7cb:::benchmark_data_smithi194_186548_object5663:head# 2024-04-29T14:53:54.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34a9360f:::benchmark_data_smithi194_186548_object41945:head# 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34a9696c:::benchmark_data_smithi194_186548_object40195:head# 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34a96c20:::benchmark_data_smithi194_186548_object80235:head# 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34aa5ad1:::benchmark_data_smithi194_186548_object54284:head# 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34aa6e1e:::benchmark_data_smithi194_186548_object60259:head# 2024-04-29T14:53:54.085 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34aad643:::benchmark_data_smithi194_186548_object17609:head# 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ab6419:::benchmark_data_smithi194_186548_object66932:head# 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ac13fe:::benchmark_data_smithi194_186548_object55791:head# 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ac9af0:::benchmark_data_smithi194_186548_object62172:head# 2024-04-29T14:53:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34acd06c:::benchmark_data_smithi194_186548_object5356:head# 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ad45b5:::benchmark_data_smithi194_186548_object30076:head# 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ad6d3e:::benchmark_data_smithi194_186548_object11222:head# 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34adaaef:::benchmark_data_smithi194_186548_object70608:head# 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ade4ce:::benchmark_data_smithi194_186548_object8096:head# 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ae7912:::benchmark_data_smithi194_186548_object37648:head# 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34aea253:::benchmark_data_smithi194_186548_object52673:head# 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34aea88a:::benchmark_data_smithi194_186548_object24279:head# 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34afb9ad:::benchmark_data_smithi194_186548_object47045:head# 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b00841:::benchmark_data_smithi194_186548_object23:head# 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b010e2:::benchmark_data_smithi194_186548_object59400:head# 2024-04-29T14:53:54.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b0bb94:::benchmark_data_smithi194_186548_object85059:head# 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b2226e:::benchmark_data_smithi194_186548_object77699:head# 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b29b10:::benchmark_data_smithi194_186548_object84845:head# 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b2be52:::benchmark_data_smithi194_186548_object59085:head# 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b3d629:::benchmark_data_smithi194_186548_object22556:head# 2024-04-29T14:53:54.097 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b4c427:::benchmark_data_smithi194_186548_object67830:head# 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b4d775:::benchmark_data_smithi194_186548_object12627:head# 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b4f5f2:::benchmark_data_smithi194_186548_object30171:head# 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b5a03b:::benchmark_data_smithi194_186548_object38628:head# 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b5bde0:::benchmark_data_smithi194_186548_object80093:head# 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b74272:::benchmark_data_smithi194_186548_object62774:head# 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b7427e:::benchmark_data_smithi194_186548_object6816:head# 2024-04-29T14:53:54.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34b9007a:::benchmark_data_smithi194_186548_object63589:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34bc8a50:::benchmark_data_smithi194_186548_object49859:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34bede5e:::benchmark_data_smithi194_186548_object47859:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34bfc26a:::benchmark_data_smithi194_186548_object9691:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c097cb:::benchmark_data_smithi194_186548_object4247:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c111d9:::benchmark_data_smithi194_186548_object75517:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c23c7e:::benchmark_data_smithi194_186548_object60714:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c31618:::benchmark_data_smithi194_186548_object65167:head# 2024-04-29T14:53:54.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c3f27d:::benchmark_data_smithi194_186548_object48866:head# 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c47f27:::benchmark_data_smithi194_186548_object80525:head# 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c49bd3:::benchmark_data_smithi194_186548_object17439:head# 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c60714:::benchmark_data_smithi194_186548_object2128:head# 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c64213:::benchmark_data_smithi194_186548_object68775:head# 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34c7cbce:::benchmark_data_smithi194_186548_object14097:head# 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34cb197c:::benchmark_data_smithi194_186548_object37882:head# 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34cbba8c:::benchmark_data_smithi194_186548_object46931:head# 2024-04-29T14:53:54.101 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.101 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34cdadb4:::benchmark_data_smithi194_186548_object31781:head# 2024-04-29T14:53:54.101 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.101 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34cdc2bd:::benchmark_data_smithi194_186548_object32426:head# 2024-04-29T14:53:54.101 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ce84db:::benchmark_data_smithi194_186548_object65015:head# 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34cf0495:::benchmark_data_smithi194_186548_object61557:head# 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34cf56c6:::benchmark_data_smithi194_186548_object82463:head# 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d02c56:::benchmark_data_smithi194_186548_object32207:head# 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d041de:::benchmark_data_smithi194_186548_object51037:head# 2024-04-29T14:53:54.102 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.103 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d0bd32:::benchmark_data_smithi194_186548_object10723:head# 2024-04-29T14:53:54.103 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.103 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d0ee6c:::benchmark_data_smithi194_186548_object2095:head# 2024-04-29T14:53:54.103 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.103 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d23c8b:::benchmark_data_smithi194_186548_object78890:head# 2024-04-29T14:53:54.104 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.104 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d3ab24:::benchmark_data_smithi194_186548_object3432:head# 2024-04-29T14:53:54.104 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d3d3a8:::benchmark_data_smithi194_186548_object59355:head# 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d448f5:::benchmark_data_smithi194_186548_object80225:head# 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d459a1:::benchmark_data_smithi194_186548_object70630:head# 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d47546:::benchmark_data_smithi194_186548_object21359:head# 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d592aa:::benchmark_data_smithi194_186548_object46099:head# 2024-04-29T14:53:54.115 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.116 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d5c6bd:::benchmark_data_smithi194_186548_object18703:head# 2024-04-29T14:53:54.116 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.116 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d5f8aa:::benchmark_data_smithi194_186548_object40877:head# 2024-04-29T14:53:54.116 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.116 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d66949:::benchmark_data_smithi194_186548_object54406:head# 2024-04-29T14:53:54.116 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.116 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d66c9b:::benchmark_data_smithi194_186548_object83940:head# 2024-04-29T14:53:54.117 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.117 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d6fb7b:::benchmark_data_smithi194_186548_object70754:head# 2024-04-29T14:53:54.117 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d77726:::benchmark_data_smithi194_186548_object9643:head# 2024-04-29T14:53:54.120 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.121 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d88de0:::benchmark_data_smithi194_186548_object13584:head# 2024-04-29T14:53:54.121 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.122 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d8abeb:::benchmark_data_smithi194_186548_object13618:head# 2024-04-29T14:53:54.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.122 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d9c40b:::benchmark_data_smithi194_186548_object54547:head# 2024-04-29T14:53:54.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34d9f105:::benchmark_data_smithi194_186548_object45973:head# 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34db46cd:::benchmark_data_smithi194_186548_object1865:head# 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34dccf3a:::benchmark_data_smithi194_186548_object56105:head# 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34dcf768:::benchmark_data_smithi194_186548_object22344:head# 2024-04-29T14:53:54.123 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:53 smithi040 ceph-mon[39581]: 8.e scrub ok 2024-04-29T14:53:54.128 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34dd0237:::benchmark_data_smithi194_186548_object20042:head# 2024-04-29T14:53:54.128 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.128 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34dd1594:::benchmark_data_smithi194_186548_object23651:head# 2024-04-29T14:53:54.128 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.133 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ddcb60:::benchmark_data_smithi194_186548_object11863:head# 2024-04-29T14:53:54.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.133 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34de02ac:::benchmark_data_smithi194_186548_object41602:head# 2024-04-29T14:53:54.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.134 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34de3009:::benchmark_data_smithi194_186548_object18492:head# 2024-04-29T14:53:54.134 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.134 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34dec983:::benchmark_data_smithi194_186548_object74029:head# 2024-04-29T14:53:54.134 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.134 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34dfb726:::benchmark_data_smithi194_186548_object15528:head# 2024-04-29T14:53:54.134 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34dfd4d6:::benchmark_data_smithi194_186548_object31332:head# 2024-04-29T14:53:54.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e0636c:::benchmark_data_smithi194_186548_object67174:head# 2024-04-29T14:53:54.136 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.136 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e1514c:::benchmark_data_smithi194_186548_object9690:head# 2024-04-29T14:53:54.136 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.136 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e225e8:::benchmark_data_smithi194_186548_object19180:head# 2024-04-29T14:53:54.136 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.136 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e395cb:::benchmark_data_smithi194_186548_object63425:head# 2024-04-29T14:53:54.136 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:53 smithi121 ceph-mon[44533]: 8.e scrub ok 2024-04-29T14:53:54.138 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e3f6ab:::benchmark_data_smithi194_186548_object10159:head# 2024-04-29T14:53:54.138 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.138 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e4ad41:::benchmark_data_smithi194_186548_object55994:head# 2024-04-29T14:53:54.138 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.138 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e4fe2f:::benchmark_data_smithi194_186548_object44603:head# 2024-04-29T14:53:54.138 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e7ce56:::benchmark_data_smithi194_186548_object65853:head# 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e8aab3:::benchmark_data_smithi194_186548_object51455:head# 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e8e8bc:::benchmark_data_smithi194_186548_object71789:head# 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e8f474:::benchmark_data_smithi194_186548_object73856:head# 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34e9d9d1:::benchmark_data_smithi194_186548_object21762:head# 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ea321e:::benchmark_data_smithi194_186548_object36488:head# 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34eb5460:::benchmark_data_smithi194_186548_object1820:head# 2024-04-29T14:53:54.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34eb7ed9:::benchmark_data_smithi194_186548_object31205:head# 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ecd4ec:::benchmark_data_smithi194_186548_object5346:head# 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ed05cf:::benchmark_data_smithi194_186548_object35079:head# 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ed1255:::benchmark_data_smithi194_186548_object20861:head# 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.141 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34ed5e61:::benchmark_data_smithi194_186548_object10054:head# 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f1f90d:::benchmark_data_smithi194_186548_object69910:head# 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f22d76:::benchmark_data_smithi194_186548_object9835:head# 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f2e041:::benchmark_data_smithi194_186548_object33095:head# 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f3223d:::benchmark_data_smithi194_186548_object5376:head# 2024-04-29T14:53:54.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.143 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f5501b:::benchmark_data_smithi194_186548_object65535:head# 2024-04-29T14:53:54.143 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.143 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f59c79:::benchmark_data_smithi194_186548_object72699:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f7cdd7:::benchmark_data_smithi194_186548_object52643:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f7e0b5:::benchmark_data_smithi194_186548_object7028:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f7f546:::benchmark_data_smithi194_186548_object21114:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f81e87:::benchmark_data_smithi194_186548_object70483:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f94bc1:::benchmark_data_smithi194_186548_object68474:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f9911f:::benchmark_data_smithi194_186548_object32841:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34f9f791:::benchmark_data_smithi194_186548_object44485:head# 2024-04-29T14:53:54.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.145 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fa2692:::benchmark_data_smithi194_186548_object34751:head# 2024-04-29T14:53:54.145 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.145 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fa936b:::benchmark_data_smithi194_186548_object56200:head# 2024-04-29T14:53:54.145 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.145 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fac20a:::benchmark_data_smithi194_186548_object76060:head# 2024-04-29T14:53:54.145 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.147 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fafe95:::benchmark_data_smithi194_186548_object20862:head# 2024-04-29T14:53:54.147 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.147 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fc6bc7:::benchmark_data_smithi194_186548_object63505:head# 2024-04-29T14:53:54.147 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.147 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fc903e:::benchmark_data_smithi194_186548_object76431:head# 2024-04-29T14:53:54.147 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fc9d83:::benchmark_data_smithi194_186548_object44861:head# 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fdeb22:::benchmark_data_smithi194_186548_object37124:head# 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:34fe1dc6:::benchmark_data_smithi194_186548_object46737:head# 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:350177b7:::benchmark_data_smithi194_186548_object45074:head# 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3501e14f:::benchmark_data_smithi194_186548_object51924:head# 2024-04-29T14:53:54.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.150 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3501ebc6:::benchmark_data_smithi194_186548_object5604:head# 2024-04-29T14:53:54.150 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.150 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3502c071:::benchmark_data_smithi194_186548_object10090:head# 2024-04-29T14:53:54.150 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.151 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35034672:::benchmark_data_smithi194_186548_object18076:head# 2024-04-29T14:53:54.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.151 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3503edcb:::benchmark_data_smithi194_186548_object15714:head# 2024-04-29T14:53:54.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3504f766:::benchmark_data_smithi194_186548_object16167:head# 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35059d78:::benchmark_data_smithi194_186548_object39124:head# 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3505f37d:::benchmark_data_smithi194_186548_object54181:head# 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35071646:::benchmark_data_smithi194_186548_object58114:head# 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3508e921:::benchmark_data_smithi194_186548_object81953:head# 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3509b99e:::benchmark_data_smithi194_186548_object59669:head# 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:350dcfb8:::benchmark_data_smithi194_186548_object64974:head# 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:350e8629:::benchmark_data_smithi194_186548_object51813:head# 2024-04-29T14:53:54.153 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:350ead35:::benchmark_data_smithi194_186548_object8393:head# 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:350fa287:::benchmark_data_smithi194_186548_object80743:head# 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35103203:::benchmark_data_smithi194_186548_object84772:head# 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35117913:::benchmark_data_smithi194_186548_object62966:head# 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35119a52:::benchmark_data_smithi194_186548_object2405:head# 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35128576:::benchmark_data_smithi194_186548_object45014:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35130532:::benchmark_data_smithi194_186548_object12672:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3514777c:::benchmark_data_smithi194_186548_object51044:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3517f31a:::benchmark_data_smithi194_186548_object53660:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35187ccc:::benchmark_data_smithi194_186548_object42201:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3519020d:::benchmark_data_smithi194_186548_object34739:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3519d13e:::benchmark_data_smithi194_186548_object81957:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:351bf5f8:::benchmark_data_smithi194_186548_object20918:head# 2024-04-29T14:53:54.157 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:351cd285:::benchmark_data_smithi194_186548_object5422:head# 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:351ce444:::benchmark_data_smithi194_186548_object77350:head# 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:351d051d:::benchmark_data_smithi194_186548_object28575:head# 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:351d2dbf:::benchmark_data_smithi194_186548_object80122:head# 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3520dad7:::benchmark_data_smithi194_186548_object12172:head# 2024-04-29T14:53:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352203f6:::benchmark_data_smithi194_186548_object62302:head# 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35247437:::benchmark_data_smithi194_186548_object84235:head# 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35248717:::benchmark_data_smithi194_186548_object6600:head# 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3527638b:::benchmark_data_smithi194_186548_object14723:head# 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3529822a:::benchmark_data_smithi194_186548_object10025:head# 2024-04-29T14:53:54.160 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35299155:::benchmark_data_smithi194_186548_object83461:head# 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352991fa:::benchmark_data_smithi194_186548_object23891:head# 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352a7a0e:::benchmark_data_smithi194_186548_object4752:head# 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352bdee3:::benchmark_data_smithi194_186548_object16995:head# 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352c1421:::benchmark_data_smithi194_186548_object43842:head# 2024-04-29T14:53:54.161 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.165 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352c31fc:::benchmark_data_smithi194_186548_object69149:head# 2024-04-29T14:53:54.165 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.165 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352c5fc5:::benchmark_data_smithi194_186548_object31886:head# 2024-04-29T14:53:54.165 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.165 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352ece50:::benchmark_data_smithi194_186548_object41389:head# 2024-04-29T14:53:54.165 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352f345a:::benchmark_data_smithi194_186548_object62245:head# 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:352f6711:::benchmark_data_smithi194_186548_object33805:head# 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35303124:::benchmark_data_smithi194_186548_object50655:head# 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35304f89:::benchmark_data_smithi194_186548_object72883:head# 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3530aca2:::benchmark_data_smithi194_186548_object6409:head# 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35315e2c:::benchmark_data_smithi194_186548_object83733:head# 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35322852:::benchmark_data_smithi194_186548_object28527:head# 2024-04-29T14:53:54.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35335966:::benchmark_data_smithi194_186548_object57809:head# 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3535b6de:::benchmark_data_smithi194_186548_object45291:head# 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35370420:::benchmark_data_smithi194_186548_object9101:head# 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35398ccd:::benchmark_data_smithi194_186548_object4960:head# 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353a1acd:::benchmark_data_smithi194_186548_object24518:head# 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353b16f4:::benchmark_data_smithi194_186548_object48342:head# 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353b52a4:::benchmark_data_smithi194_186548_object53076:head# 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353bd328:::benchmark_data_smithi194_186548_object49053:head# 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353cdce0:::benchmark_data_smithi194_186548_object50596:head# 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353d176b:::benchmark_data_smithi194_186548_object67468:head# 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353f2cb4:::benchmark_data_smithi194_186548_object33830:head# 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353f9ca4:::benchmark_data_smithi194_186548_object18233:head# 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:353ff0b7:::benchmark_data_smithi194_186548_object53101:head# 2024-04-29T14:53:54.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35405880:::benchmark_data_smithi194_186548_object57233:head# 2024-04-29T14:53:54.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3541832b:::benchmark_data_smithi194_186548_object54242:head# 2024-04-29T14:53:54.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.175 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3541cc44:::benchmark_data_smithi194_186548_object7373:head# 2024-04-29T14:53:54.175 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.175 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354371b1:::benchmark_data_smithi194_186548_object34528:head# 2024-04-29T14:53:54.175 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.175 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3544402e:::benchmark_data_smithi194_186548_object58527:head# 2024-04-29T14:53:54.175 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.175 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3545c949:::benchmark_data_smithi194_186548_object77079:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354626f5:::benchmark_data_smithi194_186548_object53819:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3546aeff:::benchmark_data_smithi194_186548_object59898:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354792ac:::benchmark_data_smithi194_186548_object15231:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3548a88b:::benchmark_data_smithi194_186548_object32104:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35491606:::benchmark_data_smithi194_186548_object15832:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3549438d:::benchmark_data_smithi194_186548_object27736:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35495bb7:::benchmark_data_smithi194_186548_object38938:head# 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.176 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354981b3:::benchmark_data_smithi194_186548_object80923:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354b3ebb:::benchmark_data_smithi194_186548_object75957:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354ba1d6:::benchmark_data_smithi194_186548_object52424:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354bc69b:::benchmark_data_smithi194_186548_object16114:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354be95f:::benchmark_data_smithi194_186548_object27463:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354d7d72:::benchmark_data_smithi194_186548_object77715:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354e1661:::benchmark_data_smithi194_186548_object1443:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:354f0947:::benchmark_data_smithi194_186548_object40701:head# 2024-04-29T14:53:54.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35519777:::benchmark_data_smithi194_186548_object37915:head# 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3552d2b5:::benchmark_data_smithi194_186548_object19425:head# 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35540870:::benchmark_data_smithi194_186548_object73420:head# 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35548d22:::benchmark_data_smithi194_186548_object13817:head# 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3554bbed:::benchmark_data_smithi194_186548_object72122:head# 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3554e9ec:::benchmark_data_smithi194_186548_object44584:head# 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35555356:::benchmark_data_smithi194_186548_object14212:head# 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35555eea:::benchmark_data_smithi194_186548_object46256:head# 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35569927:::benchmark_data_smithi194_186548_object42745:head# 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35591ac5:::benchmark_data_smithi194_186548_object55894:head# 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35593a15:::benchmark_data_smithi194_186548_object52322:head# 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3559edd5:::benchmark_data_smithi194_186548_object10869:head# 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:355a035e:::benchmark_data_smithi194_186548_object4151:head# 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:355ad35e:::benchmark_data_smithi194_186548_object11820:head# 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.179 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:355b334d:::benchmark_data_smithi194_186548_object82323:head# 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:355db0e3:::benchmark_data_smithi194_186548_object69099:head# 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:355fb14f:::benchmark_data_smithi194_186548_object10252:head# 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:355fecc0:::benchmark_data_smithi194_186548_object24389:head# 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35602b56:::benchmark_data_smithi194_186548_object57133:head# 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356103ba:::benchmark_data_smithi194_186548_object7211:head# 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35628420:::benchmark_data_smithi194_186548_object18155:head# 2024-04-29T14:53:54.180 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3562c2fc:::benchmark_data_smithi194_186548_object71622:head# 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35669f15:::benchmark_data_smithi194_186548_object24585:head# 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356804bd:::benchmark_data_smithi194_186548_object31457:head# 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35683ca6:::benchmark_data_smithi194_186548_object60116:head# 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3568472a:::benchmark_data_smithi194_186548_object68520:head# 2024-04-29T14:53:54.187 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3569475d:::benchmark_data_smithi194_186548_object63654:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356a83bf:::benchmark_data_smithi194_186548_object70576:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356aa8b2:::benchmark_data_smithi194_186548_object35760:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356c8dc0:::benchmark_data_smithi194_186548_object52800:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356d4948:::benchmark_data_smithi194_186548_object59779:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356d5081:::benchmark_data_smithi194_186548_object42080:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356ee0d6:::benchmark_data_smithi194_186548_object37534:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:356f4a90:::benchmark_data_smithi194_186548_object40414:head# 2024-04-29T14:53:54.188 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35702f61:::benchmark_data_smithi194_186548_object32915:head# 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357069fb:::benchmark_data_smithi194_186548_object55966:head# 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35710f98:::benchmark_data_smithi194_186548_object75898:head# 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35714785:::benchmark_data_smithi194_186548_object68221:head# 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357191db:::benchmark_data_smithi194_186548_object58822:head# 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3571a5f8:::benchmark_data_smithi194_186548_object6725:head# 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3572350b:::benchmark_data_smithi194_186548_object13097:head# 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.189 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3572b3be:::benchmark_data_smithi194_186548_object7190:head# 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35731a07:::benchmark_data_smithi194_186548_object76246:head# 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3573753b:::benchmark_data_smithi194_186548_object73273:head# 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357450a8:::benchmark_data_smithi194_186548_object82035:head# 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3574537a:::benchmark_data_smithi194_186548_object14286:head# 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35749dd0:::benchmark_data_smithi194_186548_object27888:head# 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3574e7a5:::benchmark_data_smithi194_186548_object5040:head# 2024-04-29T14:53:54.190 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357638bf:::benchmark_data_smithi194_186548_object62294:head# 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3576c773:::benchmark_data_smithi194_186548_object45103:head# 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3577b5bd:::benchmark_data_smithi194_186548_object45090:head# 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3579084e:::benchmark_data_smithi194_186548_object50284:head# 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357b612c:::benchmark_data_smithi194_186548_object71586:head# 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357bdb62:::benchmark_data_smithi194_186548_object26823:head# 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357cc084:::benchmark_data_smithi194_186548_object74212:head# 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.191 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357e41da:::benchmark_data_smithi194_186548_object37970:head# 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357e91ce:::benchmark_data_smithi194_186548_object67943:head# 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357ec402:::benchmark_data_smithi194_186548_object24767:head# 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357ee94d:::benchmark_data_smithi194_186548_object27327:head# 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:357f6171:::benchmark_data_smithi194_186548_object31961:head# 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35805770:::benchmark_data_smithi194_186548_object42732:head# 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35812c15:::benchmark_data_smithi194_186548_object48951:head# 2024-04-29T14:53:54.192 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35821d70:::benchmark_data_smithi194_186548_object6751:head# 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35825863:::benchmark_data_smithi194_186548_object16172:head# 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3583a327:::benchmark_data_smithi194_186548_object39208:head# 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3583d0d0:::benchmark_data_smithi194_186548_object13572:head# 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.201 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358649bd:::benchmark_data_smithi194_186548_object75477:head# 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35865036:::benchmark_data_smithi194_186548_object45539:head# 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3586be29:::benchmark_data_smithi194_186548_object81222:head# 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35876bdb:::benchmark_data_smithi194_186548_object41185:head# 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35884ad0:::benchmark_data_smithi194_186548_object32769:head# 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35885f30:::benchmark_data_smithi194_186548_object20830:head# 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358942ab:::benchmark_data_smithi194_186548_object35635:head# 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.202 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35897c75:::benchmark_data_smithi194_186548_object17624:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3589b12d:::benchmark_data_smithi194_186548_object42244:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358b5cc6:::benchmark_data_smithi194_186548_object17390:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358b6b53:::benchmark_data_smithi194_186548_object8071:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358b9c3e:::benchmark_data_smithi194_186548_object77554:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358c427f:::benchmark_data_smithi194_186548_object38198:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358d5f11:::benchmark_data_smithi194_186548_object30855:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358f1413:::benchmark_data_smithi194_186548_object60919:head# 2024-04-29T14:53:54.203 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:358fe012:::benchmark_data_smithi194_186548_object61463:head# 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35913331:::benchmark_data_smithi194_186548_object61125:head# 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3592f9e3:::benchmark_data_smithi194_186548_object48421:head# 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3593aa25:::benchmark_data_smithi194_186548_object68349:head# 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35942633:::benchmark_data_smithi194_186548_object65801:head# 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35975bc7:::benchmark_data_smithi194_186548_object51655:head# 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3597b3c7:::benchmark_data_smithi194_186548_object46329:head# 2024-04-29T14:53:54.204 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3597d379:::benchmark_data_smithi194_186548_object1606:head# 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35989517:::benchmark_data_smithi194_186548_object26480:head# 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359a0979:::benchmark_data_smithi194_186548_object5825:head# 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359a2f48:::benchmark_data_smithi194_186548_object6967:head# 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359b0b0e:::benchmark_data_smithi194_186548_object3190:head# 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359b2074:::benchmark_data_smithi194_186548_object11180:head# 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.205 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359be5de:::benchmark_data_smithi194_186548_object63195:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359d5748:::benchmark_data_smithi194_186548_object4468:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359e1b17:::benchmark_data_smithi194_186548_object4924:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359e4125:::benchmark_data_smithi194_186548_object26541:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:359e4a43:::benchmark_data_smithi194_186548_object31965:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a0c446:::benchmark_data_smithi194_186548_object48032:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a1b24b:::benchmark_data_smithi194_186548_object40259:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a1cfba:::benchmark_data_smithi194_186548_object12974:head# 2024-04-29T14:53:54.206 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a21b0d:::benchmark_data_smithi194_186548_object34757:head# 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a23f1e:::benchmark_data_smithi194_186548_object75461:head# 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a35b09:::benchmark_data_smithi194_186548_object59550:head# 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a38e17:::benchmark_data_smithi194_186548_object25666:head# 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a3d844:::benchmark_data_smithi194_186548_object2987:head# 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a60f95:::benchmark_data_smithi194_186548_object23016:head# 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a61bf2:::benchmark_data_smithi194_186548_object13195:head# 2024-04-29T14:53:54.207 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a76fba:::benchmark_data_smithi194_186548_object43479:head# 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a79fb5:::benchmark_data_smithi194_186548_object55137:head# 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a7bc77:::benchmark_data_smithi194_186548_object41537:head# 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35a8b441:::benchmark_data_smithi194_186548_object78195:head# 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35aa7880:::benchmark_data_smithi194_186548_object20084:head# 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35aa99ac:::benchmark_data_smithi194_186548_object11484:head# 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35abfb18:::benchmark_data_smithi194_186548_object38887:head# 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.208 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35aca256:::benchmark_data_smithi194_186548_object70273:head# 2024-04-29T14:53:54.209 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.209 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35acdd73:::benchmark_data_smithi194_186548_object21959:head# 2024-04-29T14:53:54.209 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.209 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35af1d4a:::benchmark_data_smithi194_186548_object12503:head# 2024-04-29T14:53:54.209 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.209 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35af9580:::benchmark_data_smithi194_186548_object56520:head# 2024-04-29T14:53:54.209 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.212 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b02064:::benchmark_data_smithi194_186548_object16194:head# 2024-04-29T14:53:54.212 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.212 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b1843c:::benchmark_data_smithi194_186548_object83016:head# 2024-04-29T14:53:54.212 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.212 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b212a2:::benchmark_data_smithi194_186548_object73597:head# 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b222b8:::benchmark_data_smithi194_186548_object68085:head# 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b284fa:::benchmark_data_smithi194_186548_object81316:head# 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b2b910:::benchmark_data_smithi194_186548_object20863:head# 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b2dcc0:::benchmark_data_smithi194_186548_object17821:head# 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b30374:::benchmark_data_smithi194_186548_object10610:head# 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b39fcd:::benchmark_data_smithi194_186548_object32549:head# 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.213 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b3a54e:::benchmark_data_smithi194_186548_object83539:head# 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b47992:::benchmark_data_smithi194_186548_object68545:head# 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b69109:::benchmark_data_smithi194_186548_object47225:head# 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b78e1e:::benchmark_data_smithi194_186548_object64379:head# 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b7d918:::benchmark_data_smithi194_186548_object71154:head# 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35b940f2:::benchmark_data_smithi194_186548_object14542:head# 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35bb3153:::benchmark_data_smithi194_186548_object78729:head# 2024-04-29T14:53:54.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35bbe57d:::benchmark_data_smithi194_186548_object62448:head# 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35bd65d6:::benchmark_data_smithi194_186548_object10866:head# 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c1a7ac:::benchmark_data_smithi194_186548_object36139:head# 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c266c6:::benchmark_data_smithi194_186548_object39291:head# 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c288cc:::benchmark_data_smithi194_186548_object80132:head# 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c2fcf9:::benchmark_data_smithi194_186548_object47516:head# 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c3b8cd:::benchmark_data_smithi194_186548_object47402:head# 2024-04-29T14:53:54.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c3c176:::benchmark_data_smithi194_186548_object2355:head# 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c4406e:::benchmark_data_smithi194_186548_object26685:head# 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c4ff96:::benchmark_data_smithi194_186548_object28832:head# 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c5c013:::benchmark_data_smithi194_186548_object10131:head# 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c6b393:::benchmark_data_smithi194_186548_object30907:head# 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c7539a:::benchmark_data_smithi194_186548_object29292:head# 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c75821:::benchmark_data_smithi194_186548_object1878:head# 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c7d00b:::benchmark_data_smithi194_186548_object50671:head# 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c80e39:::benchmark_data_smithi194_186548_object19701:head# 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c8313e:::benchmark_data_smithi194_186548_object75389:head# 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35c9050e:::benchmark_data_smithi194_186548_object24609:head# 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35cd4287:::benchmark_data_smithi194_186548_object6482:head# 2024-04-29T14:53:54.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.218 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:53:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35cded13:::benchmark_data_smithi194_186548_object10404:head# 2024-04-29T14:53:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35ce7e50:::benchmark_data_smithi194_186548_object66814:head# 2024-04-29T14:53:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.220 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35cf7d8d:::benchmark_data_smithi194_186548_object48451:head# 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d09ab5:::benchmark_data_smithi194_186548_object20881:head# 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d0d9ea:::benchmark_data_smithi194_186548_object26264:head# 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d28ef5:::benchmark_data_smithi194_186548_object70689:head# 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d36c60:::benchmark_data_smithi194_186548_object40615:head# 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d43d2f:::benchmark_data_smithi194_186548_object57421:head# 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d443fd:::benchmark_data_smithi194_186548_object54573:head# 2024-04-29T14:53:54.221 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d48d11:::benchmark_data_smithi194_186548_object62594:head# 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d5c95a:::benchmark_data_smithi194_186548_object57141:head# 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d65760:::benchmark_data_smithi194_186548_object51848:head# 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d6824d:::benchmark_data_smithi194_186548_object80044:head# 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d7cc42:::benchmark_data_smithi194_186548_object75792:head# 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d8269a:::benchmark_data_smithi194_186548_object19758:head# 2024-04-29T14:53:54.222 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:54.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:53:54.238 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d8324b:::benchmark_data_smithi194_186548_object66425:head# 2024-04-29T14:53:54.238 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d85e78:::benchmark_data_smithi194_186548_object34410:head# 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d8b8ef:::benchmark_data_smithi194_186548_object31571:head# 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d9999a:::benchmark_data_smithi194_186548_object32358:head# 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d9ef00:::benchmark_data_smithi194_186548_object16843:head# 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35d9f0c2:::benchmark_data_smithi194_186548_object11643:head# 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35dc224a:::benchmark_data_smithi194_186548_object59010:head# 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35dc8ac7:::benchmark_data_smithi194_186548_object25868:head# 2024-04-29T14:53:54.239 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35dcdff8:::benchmark_data_smithi194_186548_object66330:head# 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e068c2:::benchmark_data_smithi194_186548_object11275:head# 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e09951:::benchmark_data_smithi194_186548_object55534:head# 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e13309:::benchmark_data_smithi194_186548_object51967:head# 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e1a692:::benchmark_data_smithi194_186548_object8739:head# 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e25772:::benchmark_data_smithi194_186548_object65534:head# 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e44125:::benchmark_data_smithi194_186548_object47139:head# 2024-04-29T14:53:54.240 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e47ac2:::benchmark_data_smithi194_186548_object31485:head# 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e4a063:::benchmark_data_smithi194_186548_object80170:head# 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e4cc4f:::benchmark_data_smithi194_186548_object20025:head# 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e51e90:::benchmark_data_smithi194_186548_object79560:head# 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e6f04a:::benchmark_data_smithi194_186548_object52102:head# 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e7cc44:::benchmark_data_smithi194_186548_object64795:head# 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.241 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e7eb53:::benchmark_data_smithi194_186548_object33601:head# 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e8641f:::benchmark_data_smithi194_186548_object79227:head# 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35e9127a:::benchmark_data_smithi194_186548_object27304:head# 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35eb3701:::benchmark_data_smithi194_186548_object82963:head# 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35ecf1e4:::benchmark_data_smithi194_186548_object24252:head# 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35ede4b8:::benchmark_data_smithi194_186548_object24655:head# 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35ee9fdd:::benchmark_data_smithi194_186548_object47229:head# 2024-04-29T14:53:54.242 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35efdaaf:::benchmark_data_smithi194_186548_object49504:head# 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f0106a:::benchmark_data_smithi194_186548_object40067:head# 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f12f37:::benchmark_data_smithi194_186548_object62571:head# 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f1e0a7:::benchmark_data_smithi194_186548_object14919:head# 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f24423:::benchmark_data_smithi194_186548_object50987:head# 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f25b13:::benchmark_data_smithi194_186548_object50345:head# 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f2d130:::benchmark_data_smithi194_186548_object13649:head# 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f2f1b7:::benchmark_data_smithi194_186548_object11277:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f4801d:::benchmark_data_smithi194_186548_object27857:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f52acc:::benchmark_data_smithi194_186548_object50269:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f5782e:::benchmark_data_smithi194_186548_object52511:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f5f80f:::benchmark_data_smithi194_186548_object39572:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f5fce0:::benchmark_data_smithi194_186548_object53884:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f75f38:::benchmark_data_smithi194_186548_object71398:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35f97b67:::benchmark_data_smithi194_186548_object38012:head# 2024-04-29T14:53:54.244 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fa5ff8:::benchmark_data_smithi194_186548_object54669:head# 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fa62f0:::benchmark_data_smithi194_186548_object52945:head# 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fb6fe9:::benchmark_data_smithi194_186548_object26093:head# 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fba9a5:::benchmark_data_smithi194_186548_object73289:head# 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fc18fc:::benchmark_data_smithi194_186548_object3829:head# 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fc3f97:::benchmark_data_smithi194_186548_object47173:head# 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fcc97f:::benchmark_data_smithi194_186548_object10696:head# 2024-04-29T14:53:54.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fd5085:::benchmark_data_smithi194_186548_object24317:head# 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fe0a05:::benchmark_data_smithi194_186548_object60342:head# 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35fe7a9c:::benchmark_data_smithi194_186548_object68077:head# 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35ff5e86:::benchmark_data_smithi194_186548_object80920:head# 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:35ff77bf:::benchmark_data_smithi194_186548_object20130:head# 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36010499:::benchmark_data_smithi194_186548_object17783:head# 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.246 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36014c15:::benchmark_data_smithi194_186548_object62877:head# 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3601624c:::benchmark_data_smithi194_186548_object41405:head# 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3604757d:::benchmark_data_smithi194_186548_object35470:head# 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3605087c:::benchmark_data_smithi194_186548_object47960:head# 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36054b86:::benchmark_data_smithi194_186548_object64431:head# 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3605c431:::benchmark_data_smithi194_186548_object9126:head# 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3608dc5a:::benchmark_data_smithi194_186548_object47970:head# 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.247 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3608df98:::benchmark_data_smithi194_186548_object82883:head# 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360a429b:::benchmark_data_smithi194_186548_object18092:head# 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360aa57a:::benchmark_data_smithi194_186548_object76404:head# 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360ad5d9:::benchmark_data_smithi194_186548_object53805:head# 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360b2193:::benchmark_data_smithi194_186548_object30145:head# 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360b5f5e:::benchmark_data_smithi194_186548_object57781:head# 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360d6018:::benchmark_data_smithi194_186548_object39648:head# 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.248 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360e39e5:::benchmark_data_smithi194_186548_object50789:head# 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360ef44f:::benchmark_data_smithi194_186548_object9825:head# 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360f0518:::benchmark_data_smithi194_186548_object19526:head# 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360f1d57:::benchmark_data_smithi194_186548_object54417:head# 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360f654e:::benchmark_data_smithi194_186548_object5411:head# 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:360faa2b:::benchmark_data_smithi194_186548_object4981:head# 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3610198e:::benchmark_data_smithi194_186548_object55715:head# 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36105191:::benchmark_data_smithi194_186548_object18810:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36114bcb:::benchmark_data_smithi194_186548_object40485:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3611a355:::benchmark_data_smithi194_186548_object80062:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3611b2eb:::benchmark_data_smithi194_186548_object18523:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3611c105:::benchmark_data_smithi194_186548_object48274:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36146331:::benchmark_data_smithi194_186548_object17605:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36147486:::benchmark_data_smithi194_186548_object39296:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3615868b:::benchmark_data_smithi194_186548_object38692:head# 2024-04-29T14:53:54.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36162ae3:::benchmark_data_smithi194_186548_object84764:head# 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36174e05:::benchmark_data_smithi194_186548_object48064:head# 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3619eee6:::benchmark_data_smithi194_186548_object59564:head# 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:361ab0fe:::benchmark_data_smithi194_186548_object27503:head# 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:361b2ef1:::benchmark_data_smithi194_186548_object59950:head# 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:361da581:::benchmark_data_smithi194_186548_object28506:head# 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:361e47f2:::benchmark_data_smithi194_186548_object74864:head# 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.251 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:361f0758:::benchmark_data_smithi194_186548_object4827:head# 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:361fef65:::benchmark_data_smithi194_186548_object63197:head# 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36207e2a:::benchmark_data_smithi194_186548_object55383:head# 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36217520:::benchmark_data_smithi194_186548_object58599:head# 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3622446a:::benchmark_data_smithi194_186548_object51717:head# 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3622fa6e:::benchmark_data_smithi194_186548_object79189:head# 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3623064c:::benchmark_data_smithi194_186548_object2040:head# 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.252 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36237562:::benchmark_data_smithi194_186548_object16965:head# 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36237b00:::benchmark_data_smithi194_186548_object53895:head# 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3624abd0:::benchmark_data_smithi194_186548_object24225:head# 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3624c0f2:::benchmark_data_smithi194_186548_object68414:head# 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36254633:::benchmark_data_smithi194_186548_object45626:head# 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3625c9f5:::benchmark_data_smithi194_186548_object56158:head# 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36260819:::benchmark_data_smithi194_186548_object9770:head# 2024-04-29T14:53:54.253 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:362649a4:::benchmark_data_smithi194_186548_object35118:head# 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36265473:::benchmark_data_smithi194_186548_object54863:head# 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36286f9e:::benchmark_data_smithi194_186548_object39277:head# 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3629f1f0:::benchmark_data_smithi194_186548_object77707:head# 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:362b519e:::benchmark_data_smithi194_186548_object54065:head# 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:362b9929:::benchmark_data_smithi194_186548_object5141:head# 2024-04-29T14:53:54.277 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:362c2edc:::benchmark_data_smithi194_186548_object43807:head# 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:362d22b0:::benchmark_data_smithi194_186548_object28118:head# 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:362f417f:::benchmark_data_smithi194_186548_object83855:head# 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:363090ec:::benchmark_data_smithi194_186548_object46514:head# 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3630e698:::benchmark_data_smithi194_186548_object78037:head# 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:363122c9:::benchmark_data_smithi194_186548_object39872:head# 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3631e15a:::benchmark_data_smithi194_186548_object75453:head# 2024-04-29T14:53:54.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36334b84:::benchmark_data_smithi194_186548_object53292:head# 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3633a2fd:::benchmark_data_smithi194_186548_object45589:head# 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36349d7d:::benchmark_data_smithi194_186548_object50702:head# 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3635008b:::benchmark_data_smithi194_186548_object25852:head# 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3635afeb:::benchmark_data_smithi194_186548_object11837:head# 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:363604be:::benchmark_data_smithi194_186548_object17743:head# 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3636b3d9:::benchmark_data_smithi194_186548_object57166:head# 2024-04-29T14:53:54.279 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3636ca7c:::benchmark_data_smithi194_186548_object54593:head# 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3637ddf5:::benchmark_data_smithi194_186548_object14763:head# 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36388dae:::benchmark_data_smithi194_186548_object56628:head# 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:363cb3ec:::benchmark_data_smithi194_186548_object34114:head# 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:363e20b5:::benchmark_data_smithi194_186548_object82237:head# 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:363e6c13:::benchmark_data_smithi194_186548_object43538:head# 2024-04-29T14:53:54.280 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3640816b:::benchmark_data_smithi194_186548_object8047:head# 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3640e715:::benchmark_data_smithi194_186548_object68375:head# 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3642fa3c:::benchmark_data_smithi194_186548_object35042:head# 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3642fb77:::benchmark_data_smithi194_186548_object52714:head# 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36434c05:::benchmark_data_smithi194_186548_object13872:head# 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3644ccf3:::benchmark_data_smithi194_186548_object26987:head# 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3645892d:::benchmark_data_smithi194_186548_object48745:head# 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3647f0b3:::benchmark_data_smithi194_186548_object70764:head# 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3648cb69:::benchmark_data_smithi194_186548_object77157:head# 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3649647e:::benchmark_data_smithi194_186548_object55601:head# 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364995a5:::benchmark_data_smithi194_186548_object35571:head# 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364a7a7d:::benchmark_data_smithi194_186548_object3371:head# 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364bb14b:::benchmark_data_smithi194_186548_object17386:head# 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364c0025:::benchmark_data_smithi194_186548_object71145:head# 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.282 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364cd5f9:::benchmark_data_smithi194_186548_object23188:head# 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364d569c:::benchmark_data_smithi194_186548_object61464:head# 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364ed858:::benchmark_data_smithi194_186548_object3815:head# 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364ee803:::benchmark_data_smithi194_186548_object2686:head# 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.283 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:364f9d42:::benchmark_data_smithi194_186548_object31422:head# 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:365034dd:::benchmark_data_smithi194_186548_object23491:head# 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36506f10:::benchmark_data_smithi194_186548_object30911:head# 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3650a5c8:::benchmark_data_smithi194_186548_object13506:head# 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3650b7a0:::benchmark_data_smithi194_186548_object68297:head# 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.285 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:365170f5:::benchmark_data_smithi194_186548_object32593:head# 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36520594:::benchmark_data_smithi194_186548_object13660:head# 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36521bd8:::benchmark_data_smithi194_186548_object78954:head# 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:365292cc:::benchmark_data_smithi194_186548_object6901:head# 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36542533:::benchmark_data_smithi194_186548_object28859:head# 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36562782:::benchmark_data_smithi194_186548_object28525:head# 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.286 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3656716a:::benchmark_data_smithi194_186548_object42397:head# 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3658308e:::benchmark_data_smithi194_186548_object85237:head# 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:365aebf3:::benchmark_data_smithi194_186548_object40190:head# 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:365dd4a8:::benchmark_data_smithi194_186548_object13224:head# 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:365e00f2:::benchmark_data_smithi194_186548_object30854:head# 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:365fefd8:::benchmark_data_smithi194_186548_object71275:head# 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36602012:::benchmark_data_smithi194_186548_object81147:head# 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36608f22:::benchmark_data_smithi194_186548_object75928:head# 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3661dab0:::benchmark_data_smithi194_186548_object32329:head# 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36621a7a:::benchmark_data_smithi194_186548_object11255:head# 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3663d2b0:::benchmark_data_smithi194_186548_object21237:head# 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36647764:::benchmark_data_smithi194_186548_object28133:head# 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:366486ea:::benchmark_data_smithi194_186548_object26698:head# 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36649d76:::benchmark_data_smithi194_186548_object19140:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36654b23:::benchmark_data_smithi194_186548_object68865:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3666cca2:::benchmark_data_smithi194_186548_object42931:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36677946:::benchmark_data_smithi194_186548_object12961:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3668b42d:::benchmark_data_smithi194_186548_object59004:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36692b52:::benchmark_data_smithi194_186548_object81536:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3669d47f:::benchmark_data_smithi194_186548_object67117:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:366d04d5:::benchmark_data_smithi194_186548_object45925:head# 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.289 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:366ec008:::benchmark_data_smithi194_186548_object2673:head# 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367034db:::benchmark_data_smithi194_186548_object902:head# 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36704373:::benchmark_data_smithi194_186548_object61652:head# 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3670ab61:::benchmark_data_smithi194_186548_object75013:head# 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3670ad54:::benchmark_data_smithi194_186548_object24997:head# 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3670fa37:::benchmark_data_smithi194_186548_object81678:head# 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.290 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36713960:::benchmark_data_smithi194_186548_object72969:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367280b3:::benchmark_data_smithi194_186548_object74913:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3672c588:::benchmark_data_smithi194_186548_object16237:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3673ac72:::benchmark_data_smithi194_186548_object27388:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3674096c:::benchmark_data_smithi194_186548_object8829:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3675425e:::benchmark_data_smithi194_186548_object17348:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36756ee1:::benchmark_data_smithi194_186548_object56761:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367705f7:::benchmark_data_smithi194_186548_object24358:head# 2024-04-29T14:53:54.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3677537f:::benchmark_data_smithi194_186548_object44513:head# 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36792428:::benchmark_data_smithi194_186548_object18421:head# 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367b4c3c:::benchmark_data_smithi194_186548_object81989:head# 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367ca754:::benchmark_data_smithi194_186548_object6673:head# 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367cd5fb:::benchmark_data_smithi194_186548_object19714:head# 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367cf949:::benchmark_data_smithi194_186548_object1128:head# 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367d0d5d:::benchmark_data_smithi194_186548_object31823:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367d3157:::benchmark_data_smithi194_186548_object42976:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367e3e4d:::benchmark_data_smithi194_186548_object40207:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367f329c:::benchmark_data_smithi194_186548_object74205:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:367f81c1:::benchmark_data_smithi194_186548_object82626:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3680b24a:::benchmark_data_smithi194_186548_object23672:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36823814:::benchmark_data_smithi194_186548_object59787:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36825744:::benchmark_data_smithi194_186548_object70581:head# 2024-04-29T14:53:54.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36844a79:::benchmark_data_smithi194_186548_object42749:head# 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3684cbbb:::benchmark_data_smithi194_186548_object38248:head# 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3685c651:::benchmark_data_smithi194_186548_object4390:head# 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36862a6c:::benchmark_data_smithi194_186548_object15688:head# 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36868c78:::benchmark_data_smithi194_186548_object20899:head# 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3686e21d:::benchmark_data_smithi194_186548_object3292:head# 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36888d68:::benchmark_data_smithi194_186548_object67721:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3688f5e2:::benchmark_data_smithi194_186548_object14457:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36891ed0:::benchmark_data_smithi194_186548_object81751:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:368a8916:::benchmark_data_smithi194_186548_object75861:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:368b7d32:::benchmark_data_smithi194_186548_object1735:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:368de671:::benchmark_data_smithi194_186548_object78207:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:368df1cf:::benchmark_data_smithi194_186548_object13521:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:368ed2e5:::benchmark_data_smithi194_186548_object15011:head# 2024-04-29T14:53:54.295 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:368f540f:::benchmark_data_smithi194_186548_object15547:head# 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:368faab5:::benchmark_data_smithi194_186548_object53761:head# 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3690010a:::benchmark_data_smithi194_186548_object424:head# 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3692e119:::benchmark_data_smithi194_186548_object84091:head# 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36944868:::benchmark_data_smithi194_186548_object16383:head# 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3694aa65:::benchmark_data_smithi194_186548_object52320:head# 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.296 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3694d962:::benchmark_data_smithi194_186548_object63036:head# 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3694dee7:::benchmark_data_smithi194_186548_object28969:head# 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36959425:::benchmark_data_smithi194_186548_object3806:head# 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3695c1ae:::benchmark_data_smithi194_186548_object8262:head# 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369707aa:::benchmark_data_smithi194_186548_object13635:head# 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369777f4:::benchmark_data_smithi194_186548_object32169:head# 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3697aab6:::benchmark_data_smithi194_186548_object66311:head# 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.297 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3697d6ee:::benchmark_data_smithi194_186548_object75336:head# 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36989cc3:::benchmark_data_smithi194_186548_object27642:head# 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369a117e:::benchmark_data_smithi194_186548_object42798:head# 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369a9ac0:::benchmark_data_smithi194_186548_object16842:head# 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369c2512:::benchmark_data_smithi194_186548_object15045:head# 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369d1f3c:::benchmark_data_smithi194_186548_object6455:head# 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369d4d2b:::benchmark_data_smithi194_186548_object74062:head# 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369dc62e:::benchmark_data_smithi194_186548_object81905:head# 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369e764b:::benchmark_data_smithi194_186548_object51225:head# 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369ea310:::benchmark_data_smithi194_186548_object5026:head# 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:369ff66d:::benchmark_data_smithi194_186548_object46727:head# 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a0395b:::benchmark_data_smithi194_186548_object15856:head# 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a0b260:::benchmark_data_smithi194_186548_object23708:head# 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a11b13:::benchmark_data_smithi194_186548_object83374:head# 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.299 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a16abd:::benchmark_data_smithi194_186548_object53426:head# 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a2afdc:::benchmark_data_smithi194_186548_object2567:head# 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a2d5ee:::benchmark_data_smithi194_186548_object59992:head# 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a2ff86:::benchmark_data_smithi194_186548_object3560:head# 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a34771:::benchmark_data_smithi194_186548_object16756:head# 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a41aff:::benchmark_data_smithi194_186548_object38551:head# 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a42784:::benchmark_data_smithi194_186548_object15621:head# 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.300 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a4eac4:::benchmark_data_smithi194_186548_object57720:head# 2024-04-29T14:53:54.301 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:54.341 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a62ef6:::benchmark_data_smithi194_186548_object74162:head# 2024-04-29T14:53:54.341 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.341 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a92a46:::benchmark_data_smithi194_186548_object7183:head# 2024-04-29T14:53:54.341 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.341 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a967dd:::benchmark_data_smithi194_186548_object76519:head# 2024-04-29T14:53:54.341 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.341 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a99168:::benchmark_data_smithi194_186548_object6021:head# 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36a9b74c:::benchmark_data_smithi194_186548_object69046:head# 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36aa066a:::benchmark_data_smithi194_186548_object30893:head# 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ab218e:::benchmark_data_smithi194_186548_object77224:head# 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ad67a2:::benchmark_data_smithi194_186548_object50076:head# 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36adcbfd:::benchmark_data_smithi194_186548_object15402:head# 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36aeb29b:::benchmark_data_smithi194_186548_object5377:head# 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36aeed86:::benchmark_data_smithi194_186548_object61423:head# 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b00066:::benchmark_data_smithi194_186548_object52903:head# 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b09947:::benchmark_data_smithi194_186548_object73926:head# 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b0ce4f:::benchmark_data_smithi194_186548_object65169:head# 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b10175:::benchmark_data_smithi194_186548_object75296:head# 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b10b84:::benchmark_data_smithi194_186548_object53157:head# 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.343 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b3b65b:::benchmark_data_smithi194_186548_object32030:head# 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b60258:::benchmark_data_smithi194_186548_object31966:head# 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b63cbb:::benchmark_data_smithi194_186548_object37657:head# 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b78bcd:::benchmark_data_smithi194_186548_object13131:head# 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b7c5b6:::benchmark_data_smithi194_186548_object3401:head# 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b7f64e:::benchmark_data_smithi194_186548_object24205:head# 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36b81eac:::benchmark_data_smithi194_186548_object70078:head# 2024-04-29T14:53:54.344 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ba1fcf:::benchmark_data_smithi194_186548_object14727:head# 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36bac591:::benchmark_data_smithi194_186548_object5038:head# 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36bb3d2a:::benchmark_data_smithi194_186548_object55013:head# 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36bc0d7d:::benchmark_data_smithi194_186548_object68182:head# 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36bdb1a0:::benchmark_data_smithi194_186548_object10801:head# 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36be4999:::benchmark_data_smithi194_186548_object61892:head# 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36be6d16:::benchmark_data_smithi194_186548_object173:head# 2024-04-29T14:53:54.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36bedbe0:::benchmark_data_smithi194_186548_object47392:head# 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36bf08f9:::benchmark_data_smithi194_186548_object76205:head# 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c2908e:::benchmark_data_smithi194_186548_object69175:head# 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c2faa4:::benchmark_data_smithi194_186548_object20514:head# 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c3367f:::benchmark_data_smithi194_186548_object7535:head# 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c49f4b:::benchmark_data_smithi194_186548_object26524:head# 2024-04-29T14:53:54.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c60e9f:::benchmark_data_smithi194_186548_object33744:head# 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c64dc2:::benchmark_data_smithi194_186548_object4049:head# 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c65425:::benchmark_data_smithi194_186548_object5188:head# 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c7a5bd:::benchmark_data_smithi194_186548_object56674:head# 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c7de3d:::benchmark_data_smithi194_186548_object13923:head# 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.347 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c85fcc:::benchmark_data_smithi194_186548_object40957:head# 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c8c9c1:::benchmark_data_smithi194_186548_object26202:head# 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c90569:::benchmark_data_smithi194_186548_object18343:head# 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c98ca3:::benchmark_data_smithi194_186548_object81053:head# 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36c9f40f:::benchmark_data_smithi194_186548_object47103:head# 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36cad22d:::benchmark_data_smithi194_186548_object84128:head# 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36cc285e:::benchmark_data_smithi194_186548_object55163:head# 2024-04-29T14:53:54.348 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ccab9e:::benchmark_data_smithi194_186548_object32594:head# 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36cdf361:::benchmark_data_smithi194_186548_object1837:head# 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36cef757:::benchmark_data_smithi194_186548_object58015:head# 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36cf6838:::benchmark_data_smithi194_186548_object37671:head# 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d084af:::benchmark_data_smithi194_186548_object2439:head# 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d1520e:::benchmark_data_smithi194_186548_object11693:head# 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d1536a:::benchmark_data_smithi194_186548_object58888:head# 2024-04-29T14:53:54.349 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d1e5b2:::benchmark_data_smithi194_186548_object69638:head# 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d235eb:::benchmark_data_smithi194_186548_object14619:head# 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d36edb:::benchmark_data_smithi194_186548_object68556:head# 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d3c0c4:::benchmark_data_smithi194_186548_object73702:head# 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d5d57a:::benchmark_data_smithi194_186548_object62505:head# 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d7ca47:::benchmark_data_smithi194_186548_object84598:head# 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36d97f45:::benchmark_data_smithi194_186548_object10189:head# 2024-04-29T14:53:54.350 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36daf6e5:::benchmark_data_smithi194_186548_object78106:head# 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36db0979:::benchmark_data_smithi194_186548_object45289:head# 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36db1653:::benchmark_data_smithi194_186548_object71456:head# 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36db329c:::benchmark_data_smithi194_186548_object1629:head# 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36dc834b:::benchmark_data_smithi194_186548_object77605:head# 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36dd15f1:::benchmark_data_smithi194_186548_object72244:head# 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36dfb363:::benchmark_data_smithi194_186548_object48150:head# 2024-04-29T14:53:54.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36dfcd61:::benchmark_data_smithi194_186548_object21600:head# 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e1f967:::benchmark_data_smithi194_186548_object16338:head# 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e207b5:::benchmark_data_smithi194_186548_object67422:head# 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e3de5e:::benchmark_data_smithi194_186548_object6386:head# 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e412c4:::benchmark_data_smithi194_186548_object1874:head# 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e46443:::benchmark_data_smithi194_186548_object64394:head# 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e4d449:::benchmark_data_smithi194_186548_object5565:head# 2024-04-29T14:53:54.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e4f5ca:::benchmark_data_smithi194_186548_object7273:head# 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e53469:::benchmark_data_smithi194_186548_object62867:head# 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e5a2c9:::benchmark_data_smithi194_186548_object62677:head# 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e76767:::benchmark_data_smithi194_186548_object36171:head# 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e7c333:::benchmark_data_smithi194_186548_object72551:head# 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e7f1b5:::benchmark_data_smithi194_186548_object60173:head# 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e87f26:::benchmark_data_smithi194_186548_object31933:head# 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.353 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36e88d2b:::benchmark_data_smithi194_186548_object2617:head# 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ea4b46:::benchmark_data_smithi194_186548_object50052:head# 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36eafe19:::benchmark_data_smithi194_186548_object28445:head# 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36eafebc:::benchmark_data_smithi194_186548_object72278:head# 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ebf64c:::benchmark_data_smithi194_186548_object37649:head# 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ec6416:::benchmark_data_smithi194_186548_object77727:head# 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36edb0c1:::benchmark_data_smithi194_186548_object64273:head# 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.354 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ef847f:::benchmark_data_smithi194_186548_object70711:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ef9ebe:::benchmark_data_smithi194_186548_object71532:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f1baca:::benchmark_data_smithi194_186548_object24517:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f22d29:::benchmark_data_smithi194_186548_object8562:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f28a0c:::benchmark_data_smithi194_186548_object23169:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f2c12e:::benchmark_data_smithi194_186548_object62176:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f30941:::benchmark_data_smithi194_186548_object26577:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f34dbc:::benchmark_data_smithi194_186548_object22685:head# 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.355 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f35cb8:::benchmark_data_smithi194_186548_object30126:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f608d5:::benchmark_data_smithi194_186548_object42188:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f7b75e:::benchmark_data_smithi194_186548_object62094:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36f815b5:::benchmark_data_smithi194_186548_object12585:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36fabda1:::benchmark_data_smithi194_186548_object51609:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36fc2353:::benchmark_data_smithi194_186548_object438:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36fc826a:::benchmark_data_smithi194_186548_object5408:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36fc8958:::benchmark_data_smithi194_186548_object46819:head# 2024-04-29T14:53:54.356 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36fcbfd1:::benchmark_data_smithi194_186548_object12678:head# 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36fd507a:::benchmark_data_smithi194_186548_object2873:head# 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:36ff8a46:::benchmark_data_smithi194_186548_object22116:head# 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3700427d:::benchmark_data_smithi194_186548_object58446:head# 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3700cd7b:::benchmark_data_smithi194_186548_object80832:head# 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3701b562:::benchmark_data_smithi194_186548_object42333:head# 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3701cb33:::benchmark_data_smithi194_186548_object25401:head# 2024-04-29T14:53:54.357 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3702a9d5:::benchmark_data_smithi194_186548_object68803:head# 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37031b70:::benchmark_data_smithi194_186548_object7727:head# 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:370337ae:::benchmark_data_smithi194_186548_object24008:head# 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37038236:::benchmark_data_smithi194_186548_object7374:head# 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:370461ea:::benchmark_data_smithi194_186548_object34675:head# 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3706517c:::benchmark_data_smithi194_186548_object56555:head# 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3707b9bd:::benchmark_data_smithi194_186548_object72751:head# 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3709fb54:::benchmark_data_smithi194_186548_object16127:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:370aa10d:::benchmark_data_smithi194_186548_object73770:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:370acb6b:::benchmark_data_smithi194_186548_object55832:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:370b9187:::benchmark_data_smithi194_186548_object59446:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:370c0339:::benchmark_data_smithi194_186548_object55113:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:370f6f69:::benchmark_data_smithi194_186548_object74328:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37101bcd:::benchmark_data_smithi194_186548_object17972:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3710ac1f:::benchmark_data_smithi194_186548_object24829:head# 2024-04-29T14:53:54.359 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37114ff4:::benchmark_data_smithi194_186548_object61757:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37116624:::benchmark_data_smithi194_186548_object45127:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3714d8a5:::benchmark_data_smithi194_186548_object34791:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3714f7cd:::benchmark_data_smithi194_186548_object22806:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3715d394:::benchmark_data_smithi194_186548_object65775:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3716d4de:::benchmark_data_smithi194_186548_object74614:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3718b896:::benchmark_data_smithi194_186548_object75492:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371954d2:::benchmark_data_smithi194_186548_object53558:head# 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.360 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37197769:::benchmark_data_smithi194_186548_object31245:head# 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371af259:::benchmark_data_smithi194_186548_object61448:head# 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371afc76:::benchmark_data_smithi194_186548_object50532:head# 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371bab2a:::benchmark_data_smithi194_186548_object42418:head# 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371c26d5:::benchmark_data_smithi194_186548_object68312:head# 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371d5cf0:::benchmark_data_smithi194_186548_object44582:head# 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371e08dd:::benchmark_data_smithi194_186548_object72866:head# 2024-04-29T14:53:54.361 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371e80d1:::benchmark_data_smithi194_186548_object2306:head# 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:371f5ed6:::benchmark_data_smithi194_186548_object29636:head# 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3720c167:::benchmark_data_smithi194_186548_object21755:head# 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3720cf88:::benchmark_data_smithi194_186548_object56260:head# 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3720ee70:::benchmark_data_smithi194_186548_object55624:head# 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3720f62b:::benchmark_data_smithi194_186548_object49856:head# 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3721cf38:::benchmark_data_smithi194_186548_object83703:head# 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3723e102:::benchmark_data_smithi194_186548_object31801:head# 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37243bc9:::benchmark_data_smithi194_186548_object14724:head# 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3724c830:::benchmark_data_smithi194_186548_object19640:head# 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3726223d:::benchmark_data_smithi194_186548_object14774:head# 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3726250a:::benchmark_data_smithi194_186548_object55687:head# 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.363 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372629ea:::benchmark_data_smithi194_186548_object15879:head# 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37267e4b:::benchmark_data_smithi194_186548_object6846:head# 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37283069:::benchmark_data_smithi194_186548_object80497:head# 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372892a3:::benchmark_data_smithi194_186548_object22202:head# 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372993fa:::benchmark_data_smithi194_186548_object71098:head# 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372a0beb:::benchmark_data_smithi194_186548_object7308:head# 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372a89c2:::benchmark_data_smithi194_186548_object19945:head# 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.364 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372a9bdc:::benchmark_data_smithi194_186548_object34054:head# 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372be935:::benchmark_data_smithi194_186548_object60706:head# 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372d43b7:::benchmark_data_smithi194_186548_object73556:head# 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372f1ee8:::benchmark_data_smithi194_186548_object49892:head# 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:372fa598:::benchmark_data_smithi194_186548_object46356:head# 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3731e477:::benchmark_data_smithi194_186548_object8185:head# 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.365 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37343e8b:::benchmark_data_smithi194_186548_object75231:head# 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3734eed6:::benchmark_data_smithi194_186548_object41347:head# 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3734f6ae:::benchmark_data_smithi194_186548_object63470:head# 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3737385e:::benchmark_data_smithi194_186548_object76430:head# 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37375fb2:::benchmark_data_smithi194_186548_object75977:head# 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37384da8:::benchmark_data_smithi194_186548_object60788:head# 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3738dd1a:::benchmark_data_smithi194_186548_object20304:head# 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37397acd:::benchmark_data_smithi194_186548_object55059:head# 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:373a080a:::benchmark_data_smithi194_186548_object2967:head# 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:373a13dc:::benchmark_data_smithi194_186548_object84414:head# 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:373a23ab:::benchmark_data_smithi194_186548_object65135:head# 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:373a9837:::benchmark_data_smithi194_186548_object30044:head# 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:373c2b94:::benchmark_data_smithi194_186548_object46142:head# 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:373d897a:::benchmark_data_smithi194_186548_object5178:head# 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:373e0cbc:::benchmark_data_smithi194_186548_object45117:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37404871:::benchmark_data_smithi194_186548_object27056:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37409e63:::benchmark_data_smithi194_186548_object46068:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3740c94c:::benchmark_data_smithi194_186548_object73143:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3740f139:::benchmark_data_smithi194_186548_object25948:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37417842:::benchmark_data_smithi194_186548_object54128:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37436a1c:::benchmark_data_smithi194_186548_object60471:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3743b277:::benchmark_data_smithi194_186548_object8439:head# 2024-04-29T14:53:54.368 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3743d7d2:::benchmark_data_smithi194_186548_object60635:head# 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374438d9:::benchmark_data_smithi194_186548_object32656:head# 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37444ea0:::benchmark_data_smithi194_186548_object57561:head# 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37445b35:::benchmark_data_smithi194_186548_object36386:head# 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37447420:::benchmark_data_smithi194_186548_object964:head# 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37456aa3:::benchmark_data_smithi194_186548_object28225:head# 2024-04-29T14:53:54.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3745ee14:::benchmark_data_smithi194_186548_object68044:head# 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374632b1:::benchmark_data_smithi194_186548_object2126:head# 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37466f77:::benchmark_data_smithi194_186548_object51076:head# 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374699fc:::benchmark_data_smithi194_186548_object78758:head# 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37473fa5:::benchmark_data_smithi194_186548_object44592:head# 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37486580:::benchmark_data_smithi194_186548_object40255:head# 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3748815d:::benchmark_data_smithi194_186548_object40346:head# 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3748a310:::benchmark_data_smithi194_186548_object10852:head# 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3748b771:::benchmark_data_smithi194_186548_object56815:head# 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3749f69f:::benchmark_data_smithi194_186548_object2434:head# 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374b37f7:::benchmark_data_smithi194_186548_object80459:head# 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374b5981:::benchmark_data_smithi194_186548_object55007:head# 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374bacc8:::benchmark_data_smithi194_186548_object46616:head# 2024-04-29T14:53:54.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374cd659:::benchmark_data_smithi194_186548_object2510:head# 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:374cf03a:::benchmark_data_smithi194_186548_object78049:head# 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37509070:::benchmark_data_smithi194_186548_object26037:head# 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37530ddd:::benchmark_data_smithi194_186548_object29732:head# 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:375318d5:::benchmark_data_smithi194_186548_object28390:head# 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3755ae13:::benchmark_data_smithi194_186548_object1106:head# 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3756fe14:::benchmark_data_smithi194_186548_object53853:head# 2024-04-29T14:53:54.372 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3757ecea:::benchmark_data_smithi194_186548_object45478:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:375c0d4d:::benchmark_data_smithi194_186548_object33537:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:375c1fa8:::benchmark_data_smithi194_186548_object65965:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:375d33a7:::benchmark_data_smithi194_186548_object81138:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:375d7028:::benchmark_data_smithi194_186548_object63434:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:375ea99c:::benchmark_data_smithi194_186548_object84853:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3760b9f6:::benchmark_data_smithi194_186548_object57246:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3760c02b:::benchmark_data_smithi194_186548_object80910:head# 2024-04-29T14:53:54.373 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37621531:::benchmark_data_smithi194_186548_object72305:head# 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376249a3:::benchmark_data_smithi194_186548_object83995:head# 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37631a35:::benchmark_data_smithi194_186548_object65357:head# 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3764cb99:::benchmark_data_smithi194_186548_object40599:head# 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37654900:::benchmark_data_smithi194_186548_object77671:head# 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3766ef3a:::benchmark_data_smithi194_186548_object85304:head# 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376858c1:::benchmark_data_smithi194_186548_object84874:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3769b9f0:::benchmark_data_smithi194_186548_object10826:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376af6f0:::benchmark_data_smithi194_186548_object18654:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376cba4c:::benchmark_data_smithi194_186548_object19739:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376d4d4d:::benchmark_data_smithi194_186548_object33833:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376e676a:::benchmark_data_smithi194_186548_object67426:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376f715d:::benchmark_data_smithi194_186548_object53131:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:376fa38a:::benchmark_data_smithi194_186548_object60873:head# 2024-04-29T14:53:54.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3771e346:::benchmark_data_smithi194_186548_object65721:head# 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3772c199:::benchmark_data_smithi194_186548_object47474:head# 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3773e0ec:::benchmark_data_smithi194_186548_object42566:head# 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3774cc15:::benchmark_data_smithi194_186548_object84397:head# 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37758d2a:::benchmark_data_smithi194_186548_object60221:head# 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37769a53:::benchmark_data_smithi194_186548_object41438:head# 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3776f411:::benchmark_data_smithi194_186548_object62261:head# 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3779a97d:::benchmark_data_smithi194_186548_object79499:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3779b651:::benchmark_data_smithi194_186548_object31186:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3779fb22:::benchmark_data_smithi194_186548_object80410:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:377a8246:::benchmark_data_smithi194_186548_object59832:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:377b297c:::benchmark_data_smithi194_186548_object10499:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:377ba0a8:::benchmark_data_smithi194_186548_object11260:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:377c2982:::benchmark_data_smithi194_186548_object73295:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:377c4d66:::benchmark_data_smithi194_186548_object50720:head# 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:377c788c:::benchmark_data_smithi194_186548_object55716:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:377fab59:::benchmark_data_smithi194_186548_object29176:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3780246b:::benchmark_data_smithi194_186548_object54899:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3781fff7:::benchmark_data_smithi194_186548_object63020:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37821131:::benchmark_data_smithi194_186548_object72611:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3782d923:::benchmark_data_smithi194_186548_object58326:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37831918:::benchmark_data_smithi194_186548_object18883:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37831d12:::benchmark_data_smithi194_186548_object19784:head# 2024-04-29T14:53:54.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3783ba41:::benchmark_data_smithi194_186548_object83792:head# 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3784f1d0:::benchmark_data_smithi194_186548_object16270:head# 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:378671cf:::benchmark_data_smithi194_186548_object11377:head# 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3786e98b:::benchmark_data_smithi194_186548_object81508:head# 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37878d14:::benchmark_data_smithi194_186548_object17305:head# 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37891189:::benchmark_data_smithi194_186548_object34961:head# 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:378add1a:::benchmark_data_smithi194_186548_object66077:head# 2024-04-29T14:53:54.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:54.395 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:378b4f50:::benchmark_data_smithi194_186548_object48129:head# 2024-04-29T14:53:54.395 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.395 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:378cb668:::benchmark_data_smithi194_186548_object4014:head# 2024-04-29T14:53:54.395 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:378df738:::benchmark_data_smithi194_186548_object30807:head# 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:378fb840:::benchmark_data_smithi194_186548_object74063:head# 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:378fdce3:::benchmark_data_smithi194_186548_object36997:head# 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37902068:::benchmark_data_smithi194_186548_object19830:head# 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3790d36b:::benchmark_data_smithi194_186548_object13601:head# 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3790ddc8:::benchmark_data_smithi194_186548_object65242:head# 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37916e40:::benchmark_data_smithi194_186548_object30062:head# 2024-04-29T14:53:54.396 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3791c22e:::benchmark_data_smithi194_186548_object61124:head# 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3791ee3d:::benchmark_data_smithi194_186548_object53898:head# 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3792e83c:::benchmark_data_smithi194_186548_object26493:head# 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37937b47:::benchmark_data_smithi194_186548_object26572:head# 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3793d1ac:::benchmark_data_smithi194_186548_object72942:head# 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3793e378:::benchmark_data_smithi194_186548_object43215:head# 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37956146:::benchmark_data_smithi194_186548_object60645:head# 2024-04-29T14:53:54.397 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3795b032:::benchmark_data_smithi194_186548_object27980:head# 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37976ac3:::benchmark_data_smithi194_186548_object61917:head# 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37985543:::benchmark_data_smithi194_186548_object25390:head# 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37986fed:::benchmark_data_smithi194_186548_object65739:head# 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3798b20d:::benchmark_data_smithi194_186548_object25836:head# 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37993f91:::benchmark_data_smithi194_186548_object67745:head# 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:379cdfff:::benchmark_data_smithi194_186548_object21478:head# 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:379d5ed4:::benchmark_data_smithi194_186548_object36183:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:379e778a:::benchmark_data_smithi194_186548_object66710:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:379f2be2:::benchmark_data_smithi194_186548_object49635:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:379f40a5:::benchmark_data_smithi194_186548_object35839:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a01220:::benchmark_data_smithi194_186548_object19165:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a0af6c:::benchmark_data_smithi194_186548_object43312:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a15073:::benchmark_data_smithi194_186548_object24921:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a19691:::benchmark_data_smithi194_186548_object25365:head# 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a31cf5:::benchmark_data_smithi194_186548_object79044:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a40bc3:::benchmark_data_smithi194_186548_object29445:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a65213:::benchmark_data_smithi194_186548_object54997:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a66d6c:::benchmark_data_smithi194_186548_object75295:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a70667:::benchmark_data_smithi194_186548_object83085:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a70abd:::benchmark_data_smithi194_186548_object5615:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a758ce:::benchmark_data_smithi194_186548_object44640:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a821ce:::benchmark_data_smithi194_186548_object62467:head# 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a8458f:::benchmark_data_smithi194_186548_object65134:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a8c181:::benchmark_data_smithi194_186548_object80157:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a941fd:::benchmark_data_smithi194_186548_object76465:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37a9cd2e:::benchmark_data_smithi194_186548_object25029:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37aa7d83:::benchmark_data_smithi194_186548_object69849:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ac00fc:::benchmark_data_smithi194_186548_object33811:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ac9efd:::benchmark_data_smithi194_186548_object51885:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ace45e:::benchmark_data_smithi194_186548_object31484:head# 2024-04-29T14:53:54.401 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ae6173:::benchmark_data_smithi194_186548_object59411:head# 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37afba2a:::benchmark_data_smithi194_186548_object5971:head# 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37afd0b9:::benchmark_data_smithi194_186548_object75197:head# 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b19e5a:::benchmark_data_smithi194_186548_object60718:head# 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b222fe:::benchmark_data_smithi194_186548_object58586:head# 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b24f73:::benchmark_data_smithi194_186548_object76262:head# 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b2b951:::benchmark_data_smithi194_186548_object50203:head# 2024-04-29T14:53:54.402 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b39ced:::benchmark_data_smithi194_186548_object84988:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b429ba:::benchmark_data_smithi194_186548_object27564:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b42a28:::benchmark_data_smithi194_186548_object71530:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b441f3:::benchmark_data_smithi194_186548_object41001:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b4e3bc:::benchmark_data_smithi194_186548_object4730:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b50737:::benchmark_data_smithi194_186548_object68517:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b50965:::benchmark_data_smithi194_186548_object31870:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b59e57:::benchmark_data_smithi194_186548_object56381:head# 2024-04-29T14:53:54.403 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b5d369:::benchmark_data_smithi194_186548_object15009:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b60e84:::benchmark_data_smithi194_186548_object84590:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b64bd3:::benchmark_data_smithi194_186548_object60942:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b6569d:::benchmark_data_smithi194_186548_object39343:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b6f440:::benchmark_data_smithi194_186548_object74263:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b6f5d4:::benchmark_data_smithi194_186548_object5785:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b88ac3:::benchmark_data_smithi194_186548_object50567:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b97eb6:::benchmark_data_smithi194_186548_object35028:head# 2024-04-29T14:53:54.404 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37b9f0d1:::benchmark_data_smithi194_186548_object68481:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37bd3a77:::benchmark_data_smithi194_186548_object24554:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37bd4f3c:::benchmark_data_smithi194_186548_object85111:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37be7adf:::benchmark_data_smithi194_186548_object21447:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37bf002b:::benchmark_data_smithi194_186548_object2940:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c07937:::benchmark_data_smithi194_186548_object40262:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c244fa:::benchmark_data_smithi194_186548_object64630:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c2c310:::benchmark_data_smithi194_186548_object53195:head# 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.405 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c2dfe7:::benchmark_data_smithi194_186548_object4580:head# 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c30d75:::benchmark_data_smithi194_186548_object7178:head# 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c48f12:::benchmark_data_smithi194_186548_object49719:head# 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c54f66:::benchmark_data_smithi194_186548_object9264:head# 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c5ef35:::benchmark_data_smithi194_186548_object7479:head# 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c5f2ab:::benchmark_data_smithi194_186548_object70931:head# 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c61fb2:::benchmark_data_smithi194_186548_object29916:head# 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c6820f:::benchmark_data_smithi194_186548_object3346:head# 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37c9aa9f:::benchmark_data_smithi194_186548_object81216:head# 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ca7d2c:::benchmark_data_smithi194_186548_object4431:head# 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37cb10f9:::benchmark_data_smithi194_186548_object75565:head# 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37cb32d7:::benchmark_data_smithi194_186548_object54270:head# 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37cdd385:::benchmark_data_smithi194_186548_object9378:head# 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ced86f:::benchmark_data_smithi194_186548_object51475:head# 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d008b4:::benchmark_data_smithi194_186548_object50895:head# 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d024e1:::benchmark_data_smithi194_186548_object4469:head# 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d118a8:::benchmark_data_smithi194_186548_object81820:head# 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d18bcd:::benchmark_data_smithi194_186548_object26196:head# 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d1f8a0:::benchmark_data_smithi194_186548_object42312:head# 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d24a08:::benchmark_data_smithi194_186548_object46255:head# 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d29e68:::benchmark_data_smithi194_186548_object47892:head# 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d309bd:::benchmark_data_smithi194_186548_object67885:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d34333:::benchmark_data_smithi194_186548_object63137:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d4ab1f:::benchmark_data_smithi194_186548_object24450:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d4fbf3:::benchmark_data_smithi194_186548_object50884:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d5730c:::benchmark_data_smithi194_186548_object42198:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d57699:::benchmark_data_smithi194_186548_object56321:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d60963:::benchmark_data_smithi194_186548_object23432:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d662ad:::benchmark_data_smithi194_186548_object52382:head# 2024-04-29T14:53:54.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d6a4d7:::benchmark_data_smithi194_186548_object61861:head# 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d6f19d:::benchmark_data_smithi194_186548_object49026:head# 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d74a1b:::benchmark_data_smithi194_186548_object37973:head# 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d8a354:::benchmark_data_smithi194_186548_object69337:head# 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d8c73a:::benchmark_data_smithi194_186548_object29432:head# 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d92aa0:::benchmark_data_smithi194_186548_object45647:head# 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d9bd11:::benchmark_data_smithi194_186548_object40843:head# 2024-04-29T14:53:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37d9cb5f:::benchmark_data_smithi194_186548_object61397:head# 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37da7a7f:::benchmark_data_smithi194_186548_object41565:head# 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37dcaa9c:::benchmark_data_smithi194_186548_object27047:head# 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37dd5810:::benchmark_data_smithi194_186548_object20046:head# 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ddf112:::benchmark_data_smithi194_186548_object13834:head# 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37de41ab:::benchmark_data_smithi194_186548_object26452:head# 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37de6713:::benchmark_data_smithi194_186548_object29498:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37dfc68d:::benchmark_data_smithi194_186548_object38511:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e02b17:::benchmark_data_smithi194_186548_object70594:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e03feb:::benchmark_data_smithi194_186548_object53608:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e07063:::benchmark_data_smithi194_186548_object62032:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e0f4e3:::benchmark_data_smithi194_186548_object39688:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e11387:::benchmark_data_smithi194_186548_object35697:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e143f3:::benchmark_data_smithi194_186548_object9354:head# 2024-04-29T14:53:54.412 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e1526d:::benchmark_data_smithi194_186548_object32262:head# 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e23800:::benchmark_data_smithi194_186548_object33949:head# 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e2d94f:::benchmark_data_smithi194_186548_object46755:head# 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e301cd:::benchmark_data_smithi194_186548_object63250:head# 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e483d3:::benchmark_data_smithi194_186548_object367:head# 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e49474:::benchmark_data_smithi194_186548_object10053:head# 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.413 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e5291f:::benchmark_data_smithi194_186548_object21011:head# 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e61fa4:::benchmark_data_smithi194_186548_object8194:head# 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e63263:::benchmark_data_smithi194_186548_object30363:head# 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e690ba:::benchmark_data_smithi194_186548_object69519:head# 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e69936:::benchmark_data_smithi194_186548_object35702:head# 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e70e65:::benchmark_data_smithi194_186548_object138:head# 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e71753:::benchmark_data_smithi194_186548_object35381:head# 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.414 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e85513:::benchmark_data_smithi194_186548_object6723:head# 2024-04-29T14:53:54.415 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.418 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e8e4a8:::benchmark_data_smithi194_186548_object64737:head# 2024-04-29T14:53:54.418 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.418 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e9a396:::benchmark_data_smithi194_186548_object49063:head# 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e9dfc2:::benchmark_data_smithi194_186548_object38438:head# 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37e9ee9f:::benchmark_data_smithi194_186548_object38506:head# 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ea0bdb:::benchmark_data_smithi194_186548_object58066:head# 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ea6991:::benchmark_data_smithi194_186548_object52018:head# 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37eb447b:::benchmark_data_smithi194_186548_object50778:head# 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ebbc31:::benchmark_data_smithi194_186548_object48728:head# 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.419 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ec8fc9:::benchmark_data_smithi194_186548_object32279:head# 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ecd9a0:::benchmark_data_smithi194_186548_object76136:head# 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ee1d43:::benchmark_data_smithi194_186548_object5220:head# 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ee6b41:::benchmark_data_smithi194_186548_object2062:head# 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37efccd5:::benchmark_data_smithi194_186548_object70432:head# 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f48ff1:::benchmark_data_smithi194_186548_object58559:head# 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f5731c:::benchmark_data_smithi194_186548_object47926:head# 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.420 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f58574:::benchmark_data_smithi194_186548_object66204:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f67c1b:::benchmark_data_smithi194_186548_object29971:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f81190:::benchmark_data_smithi194_186548_object6965:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f8525b:::benchmark_data_smithi194_186548_object38228:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f910a3:::benchmark_data_smithi194_186548_object21628:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37f93eb1:::benchmark_data_smithi194_186548_object14020:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37fb7d08:::benchmark_data_smithi194_186548_object30365:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37fc0ca8:::benchmark_data_smithi194_186548_object21051:head# 2024-04-29T14:53:54.421 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37fd65b3:::benchmark_data_smithi194_186548_object41628:head# 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ff1669:::benchmark_data_smithi194_186548_object66501:head# 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:37ffb427:::benchmark_data_smithi194_186548_object83304:head# 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3800b524:::benchmark_data_smithi194_186548_object11041:head# 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3801e432:::benchmark_data_smithi194_186548_object20640:head# 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38026532:::benchmark_data_smithi194_186548_object16334:head# 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3802ca8b:::benchmark_data_smithi194_186548_object29228:head# 2024-04-29T14:53:54.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3802d0d3:::benchmark_data_smithi194_186548_object77547:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380437c2:::benchmark_data_smithi194_186548_object39197:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38045bf3:::benchmark_data_smithi194_186548_object22830:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3804db81:::benchmark_data_smithi194_186548_object21789:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3804edad:::benchmark_data_smithi194_186548_object33677:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38066868:::benchmark_data_smithi194_186548_object7648:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380769d1:::benchmark_data_smithi194_186548_object51920:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3807a381:::benchmark_data_smithi194_186548_object45715:head# 2024-04-29T14:53:54.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38097342:::benchmark_data_smithi194_186548_object8112:head# 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38099925:::benchmark_data_smithi194_186548_object3355:head# 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380be4bf:::benchmark_data_smithi194_186548_object3392:head# 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380c2679:::benchmark_data_smithi194_186548_object82607:head# 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380cb78f:::benchmark_data_smithi194_186548_object8610:head# 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380dd467:::benchmark_data_smithi194_186548_object33983:head# 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380e1125:::benchmark_data_smithi194_186548_object80069:head# 2024-04-29T14:53:54.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380f515e:::benchmark_data_smithi194_186548_object4110:head# 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:380ff313:::benchmark_data_smithi194_186548_object49792:head# 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38112fce:::benchmark_data_smithi194_186548_object16133:head# 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3813674f:::benchmark_data_smithi194_186548_object23150:head# 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381430bb:::benchmark_data_smithi194_186548_object65551:head# 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38143788:::benchmark_data_smithi194_186548_object61777:head# 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381447e4:::benchmark_data_smithi194_186548_object8373:head# 2024-04-29T14:53:54.426 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38149225:::benchmark_data_smithi194_186548_object23440:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3814a51c:::benchmark_data_smithi194_186548_object3083:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3814ce59:::benchmark_data_smithi194_186548_object72785:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38154f8c:::benchmark_data_smithi194_186548_object56391:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3815bc08:::benchmark_data_smithi194_186548_object70707:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38195597:::benchmark_data_smithi194_186548_object34033:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3819a509:::benchmark_data_smithi194_186548_object13414:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381b596b:::benchmark_data_smithi194_186548_object13512:head# 2024-04-29T14:53:54.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.428 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381b6803:::benchmark_data_smithi194_186548_object24558:head# 2024-04-29T14:53:54.428 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.430 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381bbe90:::benchmark_data_smithi194_186548_object69011:head# 2024-04-29T14:53:54.430 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.430 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381c32d6:::benchmark_data_smithi194_186548_object79950:head# 2024-04-29T14:53:54.430 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.431 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381c626b:::benchmark_data_smithi194_186548_object296:head# 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381d275f:::benchmark_data_smithi194_186548_object52330:head# 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381d693d:::benchmark_data_smithi194_186548_object14482:head# 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:381dff4f:::benchmark_data_smithi194_186548_object44468:head# 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38203092:::benchmark_data_smithi194_186548_object60007:head# 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3820b9a8:::benchmark_data_smithi194_186548_object40564:head# 2024-04-29T14:53:54.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.433 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3820fc5c:::benchmark_data_smithi194_186548_object4583:head# 2024-04-29T14:53:54.433 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.433 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38219dd1:::benchmark_data_smithi194_186548_object30016:head# 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38252fb1:::benchmark_data_smithi194_186548_object69941:head# 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38279128:::benchmark_data_smithi194_186548_object47272:head# 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38281d39:::benchmark_data_smithi194_186548_object39245:head# 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38289cff:::benchmark_data_smithi194_186548_object28049:head# 2024-04-29T14:53:54.434 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.435 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3828d227:::benchmark_data_smithi194_186548_object14969:head# 2024-04-29T14:53:54.435 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.435 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3829417a:::benchmark_data_smithi194_186548_object10540:head# 2024-04-29T14:53:54.435 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.438 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382977c2:::benchmark_data_smithi194_186548_object39762:head# 2024-04-29T14:53:54.438 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.439 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382a26df:::benchmark_data_smithi194_186548_object63356:head# 2024-04-29T14:53:54.439 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.439 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382a8ab1:::benchmark_data_smithi194_186548_object18663:head# 2024-04-29T14:53:54.439 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.441 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382c8a94:::benchmark_data_smithi194_186548_object37637:head# 2024-04-29T14:53:54.441 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.442 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382cb0b1:::benchmark_data_smithi194_186548_object73591:head# 2024-04-29T14:53:54.442 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.442 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382cd1a5:::benchmark_data_smithi194_186548_object23682:head# 2024-04-29T14:53:54.442 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.443 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382cf3b1:::benchmark_data_smithi194_186548_object29324:head# 2024-04-29T14:53:54.443 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.443 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382db19e:::benchmark_data_smithi194_186548_object72130:head# 2024-04-29T14:53:54.443 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.450 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382e8337:::benchmark_data_smithi194_186548_object22040:head# 2024-04-29T14:53:54.450 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.451 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:382f7a2f:::benchmark_data_smithi194_186548_object80849:head# 2024-04-29T14:53:54.451 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.451 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383042bf:::benchmark_data_smithi194_186548_object26712:head# 2024-04-29T14:53:54.451 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.451 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38309cbf:::benchmark_data_smithi194_186548_object77201:head# 2024-04-29T14:53:54.451 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3830b6ae:::benchmark_data_smithi194_186548_object52537:head# 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383104d2:::benchmark_data_smithi194_186548_object20767:head# 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3831a64f:::benchmark_data_smithi194_186548_object20966:head# 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3831b26b:::benchmark_data_smithi194_186548_object17052:head# 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38323b74:::benchmark_data_smithi194_186548_object49688:head# 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3833553d:::benchmark_data_smithi194_186548_object30021:head# 2024-04-29T14:53:54.453 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.454 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38347464:::benchmark_data_smithi194_186548_object34064:head# 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3835b00f:::benchmark_data_smithi194_186548_object19401:head# 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3838417b:::benchmark_data_smithi194_186548_object50003:head# 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383893ae:::benchmark_data_smithi194_186548_object13766:head# 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38399435:::benchmark_data_smithi194_186548_object62244:head# 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383a4ccb:::benchmark_data_smithi194_186548_object4277:head# 2024-04-29T14:53:54.455 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.456 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383a8edf:::benchmark_data_smithi194_186548_object29694:head# 2024-04-29T14:53:54.456 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.456 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383b8b6b:::benchmark_data_smithi194_186548_object85017:head# 2024-04-29T14:53:54.456 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.456 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383d03d8:::benchmark_data_smithi194_186548_object50814:head# 2024-04-29T14:53:54.457 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.457 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383dd688:::benchmark_data_smithi194_186548_object66146:head# 2024-04-29T14:53:54.457 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.457 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383df05a:::benchmark_data_smithi194_186548_object623:head# 2024-04-29T14:53:54.457 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.457 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383e46b0:::benchmark_data_smithi194_186548_object63214:head# 2024-04-29T14:53:54.457 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.464 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:383e7436:::benchmark_data_smithi194_186548_object61049:head# 2024-04-29T14:53:54.464 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.464 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38406819:::benchmark_data_smithi194_186548_object39925:head# 2024-04-29T14:53:54.464 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.464 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38409ad8:::benchmark_data_smithi194_186548_object63834:head# 2024-04-29T14:53:54.464 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.465 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384131fe:::benchmark_data_smithi194_186548_object33725:head# 2024-04-29T14:53:54.465 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.465 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38423652:::benchmark_data_smithi194_186548_object84558:head# 2024-04-29T14:53:54.466 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.466 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38425f7c:::benchmark_data_smithi194_186548_object25859:head# 2024-04-29T14:53:54.466 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.466 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3843075b:::benchmark_data_smithi194_186548_object52668:head# 2024-04-29T14:53:54.466 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38433ddf:::benchmark_data_smithi194_186548_object50876:head# 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3843e310:::benchmark_data_smithi194_186548_object78713:head# 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384583fb:::benchmark_data_smithi194_186548_object42143:head# 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3846020a:::benchmark_data_smithi194_186548_object21370:head# 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3846303e:::benchmark_data_smithi194_186548_object36366:head# 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38465472:::benchmark_data_smithi194_186548_object6221:head# 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3846a4d6:::benchmark_data_smithi194_186548_object27810:head# 2024-04-29T14:53:54.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.477 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3846e144:::benchmark_data_smithi194_186548_object12282:head# 2024-04-29T14:53:54.477 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.477 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38488f11:::benchmark_data_smithi194_186548_object34773:head# 2024-04-29T14:53:54.477 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.477 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3848b1db:::benchmark_data_smithi194_186548_object58728:head# 2024-04-29T14:53:54.477 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.477 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384a0539:::benchmark_data_smithi194_186548_object61750:head# 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384a6a9b:::benchmark_data_smithi194_186548_object12366:head# 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384aa113:::benchmark_data_smithi194_186548_object32867:head# 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384b3e69:::benchmark_data_smithi194_186548_object23771:head# 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384c6dcf:::benchmark_data_smithi194_186548_object65718:head# 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384cf9a6:::benchmark_data_smithi194_186548_object12584:head# 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384d0850:::benchmark_data_smithi194_186548_object20675:head# 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.478 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384d24a8:::benchmark_data_smithi194_186548_object75181:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:384ef537:::benchmark_data_smithi194_186548_object81029:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38513615:::benchmark_data_smithi194_186548_object14338:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38521c38:::benchmark_data_smithi194_186548_object22178:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38526846:::benchmark_data_smithi194_186548_object28409:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38535e46:::benchmark_data_smithi194_186548_object17018:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38549bf2:::benchmark_data_smithi194_186548_object55438:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38549d0c:::benchmark_data_smithi194_186548_object83035:head# 2024-04-29T14:53:54.479 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3854fd51:::benchmark_data_smithi194_186548_object9421:head# 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38558bc8:::benchmark_data_smithi194_186548_object36517:head# 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38584ed8:::benchmark_data_smithi194_186548_object70631:head# 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3858e05c:::benchmark_data_smithi194_186548_object69770:head# 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3859982e:::benchmark_data_smithi194_186548_object70680:head# 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385aca15:::benchmark_data_smithi194_186548_object6293:head# 2024-04-29T14:53:54.480 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.481 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385bd47b:::benchmark_data_smithi194_186548_object25108:head# 2024-04-29T14:53:54.481 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.481 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385c0b51:::benchmark_data_smithi194_186548_object75564:head# 2024-04-29T14:53:54.481 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.485 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385c5e76:::benchmark_data_smithi194_186548_object83540:head# 2024-04-29T14:53:54.485 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.485 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385c630a:::benchmark_data_smithi194_186548_object22083:head# 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385c8f2b:::benchmark_data_smithi194_186548_object14551:head# 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385c99fe:::benchmark_data_smithi194_186548_object78824:head# 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385e746f:::benchmark_data_smithi194_186548_object13899:head# 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385ec4ca:::benchmark_data_smithi194_186548_object83844:head# 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385eeb3c:::benchmark_data_smithi194_186548_object2237:head# 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385f947a:::benchmark_data_smithi194_186548_object75493:head# 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385faebb:::benchmark_data_smithi194_186548_object59130:head# 2024-04-29T14:53:54.487 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.487 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:385fed9d:::benchmark_data_smithi194_186548_object77248:head# 2024-04-29T14:53:54.487 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386048ee:::benchmark_data_smithi194_186548_object74976:head# 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386055ea:::benchmark_data_smithi194_186548_object76445:head# 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3860a792:::benchmark_data_smithi194_186548_object6950:head# 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38624f44:::benchmark_data_smithi194_186548_object52316:head# 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.492 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3862cbc5:::benchmark_data_smithi194_186548_object29037:head# 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38636931:::benchmark_data_smithi194_186548_object59957:head# 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3864a8e7:::benchmark_data_smithi194_186548_object54598:head# 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3865c7fd:::benchmark_data_smithi194_186548_object12560:head# 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38663d9c:::benchmark_data_smithi194_186548_object26919:head# 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3867d7ac:::benchmark_data_smithi194_186548_object37866:head# 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3868799c:::benchmark_data_smithi194_186548_object80639:head# 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386982db:::benchmark_data_smithi194_186548_object56673:head# 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386ae022:::benchmark_data_smithi194_186548_object20716:head# 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386af950:::benchmark_data_smithi194_186548_object84700:head# 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386c98b6:::benchmark_data_smithi194_186548_object83097:head# 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386ca29e:::benchmark_data_smithi194_186548_object32151:head# 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386ccde1:::benchmark_data_smithi194_186548_object43530:head# 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386d1e83:::benchmark_data_smithi194_186548_object23094:head# 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386e00f2:::benchmark_data_smithi194_186548_object32789:head# 2024-04-29T14:53:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386e5269:::benchmark_data_smithi194_186548_object48495:head# 2024-04-29T14:53:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386e67f2:::benchmark_data_smithi194_186548_object72387:head# 2024-04-29T14:53:54.495 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386e7893:::benchmark_data_smithi194_186548_object51360:head# 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:386fe11e:::benchmark_data_smithi194_186548_object5414:head# 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38701e4d:::benchmark_data_smithi194_186548_object55718:head# 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38704695:::benchmark_data_smithi194_186548_object41908:head# 2024-04-29T14:53:54.499 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:387185dd:::benchmark_data_smithi194_186548_object9409:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3871b2a2:::benchmark_data_smithi194_186548_object70221:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38733fa7:::benchmark_data_smithi194_186548_object26940:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38751c39:::benchmark_data_smithi194_186548_object72153:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:387727d9:::benchmark_data_smithi194_186548_object80196:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3879acd8:::benchmark_data_smithi194_186548_object39745:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:387cea6c:::benchmark_data_smithi194_186548_object18560:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:387e60a5:::benchmark_data_smithi194_186548_object80582:head# 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.500 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:387f6022:::benchmark_data_smithi194_186548_object29411:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:387f6e71:::benchmark_data_smithi194_186548_object78286:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:387f866e:::benchmark_data_smithi194_186548_object45039:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3880d432:::benchmark_data_smithi194_186548_object8702:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3880f1da:::benchmark_data_smithi194_186548_object66947:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38818b11:::benchmark_data_smithi194_186548_object80848:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3883eeeb:::benchmark_data_smithi194_186548_object9484:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38858cfe:::benchmark_data_smithi194_186548_object10201:head# 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388755b7:::benchmark_data_smithi194_186548_object48783:head# 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3887b7dd:::benchmark_data_smithi194_186548_object54214:head# 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3887e3fd:::benchmark_data_smithi194_186548_object32346:head# 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3888b4f1:::benchmark_data_smithi194_186548_object29468:head# 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3889e333:::benchmark_data_smithi194_186548_object82001:head# 2024-04-29T14:53:54.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.506 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388ba6b1:::benchmark_data_smithi194_186548_object73803:head# 2024-04-29T14:53:54.506 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.510 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388bc22b:::benchmark_data_smithi194_186548_object59982:head# 2024-04-29T14:53:54.510 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.510 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388c184c:::benchmark_data_smithi194_186548_object25691:head# 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388cee1e:::benchmark_data_smithi194_186548_object33685:head# 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388d12ca:::benchmark_data_smithi194_186548_object3058:head# 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388d27d8:::benchmark_data_smithi194_186548_object37490:head# 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388d93c0:::benchmark_data_smithi194_186548_object19948:head# 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388dc980:::benchmark_data_smithi194_186548_object80165:head# 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388e7e17:::benchmark_data_smithi194_186548_object48589:head# 2024-04-29T14:53:54.511 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:388f0e09:::benchmark_data_smithi194_186548_object59610:head# 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38911f4c:::benchmark_data_smithi194_186548_object65627:head# 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389143d2:::benchmark_data_smithi194_186548_object403:head# 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3891b1ce:::benchmark_data_smithi194_186548_object35919:head# 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.515 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3891f06f:::benchmark_data_smithi194_186548_object57488:head# 2024-04-29T14:53:54.516 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.516 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38926362:::benchmark_data_smithi194_186548_object61322:head# 2024-04-29T14:53:54.516 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.516 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389282db:::benchmark_data_smithi194_186548_object74929:head# 2024-04-29T14:53:54.516 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.516 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38948fa5:::benchmark_data_smithi194_186548_object70249:head# 2024-04-29T14:53:54.517 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.521 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3894e19f:::benchmark_data_smithi194_186548_object45832:head# 2024-04-29T14:53:54.521 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.521 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3894fd96:::benchmark_data_smithi194_186548_object65561:head# 2024-04-29T14:53:54.521 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38957468:::benchmark_data_smithi194_186548_object11379:head# 2024-04-29T14:53:54.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38963041:::benchmark_data_smithi194_186548_object41533:head# 2024-04-29T14:53:54.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38963623:::benchmark_data_smithi194_186548_object41373:head# 2024-04-29T14:53:54.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.523 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38963880:::benchmark_data_smithi194_186548_object63569:head# 2024-04-29T14:53:54.523 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.523 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3896717f:::benchmark_data_smithi194_186548_object47572:head# 2024-04-29T14:53:54.523 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.523 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3897e619:::benchmark_data_smithi194_186548_object74111:head# 2024-04-29T14:53:54.523 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.528 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389987d6:::benchmark_data_smithi194_186548_object70548:head# 2024-04-29T14:53:54.528 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.528 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3899dd36:::benchmark_data_smithi194_186548_object13542:head# 2024-04-29T14:53:54.528 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.529 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389a7fda:::benchmark_data_smithi194_186548_object25051:head# 2024-04-29T14:53:54.529 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.530 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389b8ab2:::benchmark_data_smithi194_186548_object70408:head# 2024-04-29T14:53:54.530 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.530 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389c7b85:::benchmark_data_smithi194_186548_object62659:head# 2024-04-29T14:53:54.530 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.530 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389cb91f:::benchmark_data_smithi194_186548_object42562:head# 2024-04-29T14:53:54.530 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389db280:::benchmark_data_smithi194_186548_object51075:head# 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389f84f1:::benchmark_data_smithi194_186548_object13795:head# 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389ff57e:::benchmark_data_smithi194_186548_object78971:head# 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:389ffdbd:::benchmark_data_smithi194_186548_object52612:head# 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a0759b:::benchmark_data_smithi194_186548_object15453:head# 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a10c27:::benchmark_data_smithi194_186548_object78048:head# 2024-04-29T14:53:54.534 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.535 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a1e765:::benchmark_data_smithi194_186548_object42730:head# 2024-04-29T14:53:54.536 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.536 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a2183b:::benchmark_data_smithi194_186548_object6157:head# 2024-04-29T14:53:54.536 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.536 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a2f653:::benchmark_data_smithi194_186548_object3881:head# 2024-04-29T14:53:54.536 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a46fb8:::benchmark_data_smithi194_186548_object26883:head# 2024-04-29T14:53:54.540 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a59098:::benchmark_data_smithi194_186548_object25140:head# 2024-04-29T14:53:54.540 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a71146:::benchmark_data_smithi194_186548_object56920:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a87452:::benchmark_data_smithi194_186548_object4698:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a99b86:::benchmark_data_smithi194_186548_object82795:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38a9cd6b:::benchmark_data_smithi194_186548_object31198:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38aa053e:::benchmark_data_smithi194_186548_object60056:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ad7bb7:::benchmark_data_smithi194_186548_object68492:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ae297e:::benchmark_data_smithi194_186548_object61883:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38af3399:::benchmark_data_smithi194_186548_object58799:head# 2024-04-29T14:53:54.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b135c0:::benchmark_data_smithi194_186548_object30235:head# 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b1803f:::benchmark_data_smithi194_186548_object68855:head# 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b32baf:::benchmark_data_smithi194_186548_object80291:head# 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b343c1:::benchmark_data_smithi194_186548_object58459:head# 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b4d317:::benchmark_data_smithi194_186548_object10438:head# 2024-04-29T14:53:54.542 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.543 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b523df:::benchmark_data_smithi194_186548_object85041:head# 2024-04-29T14:53:54.544 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.544 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b63c45:::benchmark_data_smithi194_186548_object52962:head# 2024-04-29T14:53:54.544 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.544 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b730a7:::benchmark_data_smithi194_186548_object42117:head# 2024-04-29T14:53:54.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b743ad:::benchmark_data_smithi194_186548_object79485:head# 2024-04-29T14:53:54.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b7861f:::benchmark_data_smithi194_186548_object43644:head# 2024-04-29T14:53:54.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b8aaf5:::benchmark_data_smithi194_186548_object74304:head# 2024-04-29T14:53:54.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.546 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38b98ab1:::benchmark_data_smithi194_186548_object58136:head# 2024-04-29T14:53:54.546 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.547 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bafb28:::benchmark_data_smithi194_186548_object70865:head# 2024-04-29T14:53:54.547 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.547 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bba71c:::benchmark_data_smithi194_186548_object63071:head# 2024-04-29T14:53:54.547 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bcc2b2:::benchmark_data_smithi194_186548_object28588:head# 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bd8978:::benchmark_data_smithi194_186548_object13917:head# 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bd90f0:::benchmark_data_smithi194_186548_object77132:head# 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bea73f:::benchmark_data_smithi194_186548_object49921:head# 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bf3e13:::benchmark_data_smithi194_186548_object18364:head# 2024-04-29T14:53:54.548 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.550 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bfb883:::benchmark_data_smithi194_186548_object63103:head# 2024-04-29T14:53:54.550 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.550 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38bfd72c:::benchmark_data_smithi194_186548_object83892:head# 2024-04-29T14:53:54.550 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c1939f:::benchmark_data_smithi194_186548_object19330:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c1a5d6:::benchmark_data_smithi194_186548_object19470:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c1b7a8:::benchmark_data_smithi194_186548_object10363:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c2e443:::benchmark_data_smithi194_186548_object6194:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c302dc:::benchmark_data_smithi194_186548_object33151:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c38c4d:::benchmark_data_smithi194_186548_object74158:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c3b830:::benchmark_data_smithi194_186548_object53095:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c3cc96:::benchmark_data_smithi194_186548_object67774:head# 2024-04-29T14:53:54.551 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.552 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c3d08f:::benchmark_data_smithi194_186548_object66211:head# 2024-04-29T14:53:54.552 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.552 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c3df87:::benchmark_data_smithi194_186548_object11671:head# 2024-04-29T14:53:54.552 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c3e19f:::benchmark_data_smithi194_186548_object84683:head# 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c3ea11:::benchmark_data_smithi194_186548_object21837:head# 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c4c1f0:::benchmark_data_smithi194_186548_object78546:head# 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c6198e:::benchmark_data_smithi194_186548_object49546:head# 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.556 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c62d75:::benchmark_data_smithi194_186548_object27571:head# 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38c80629:::benchmark_data_smithi194_186548_object57707:head# 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ca2bab:::benchmark_data_smithi194_186548_object25101:head# 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ca4852:::benchmark_data_smithi194_186548_object72879:head# 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ca5a16:::benchmark_data_smithi194_186548_object2679:head# 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38caeddb:::benchmark_data_smithi194_186548_object12962:head# 2024-04-29T14:53:54.557 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.558 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cb405a:::benchmark_data_smithi194_186548_object55333:head# 2024-04-29T14:53:54.558 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.558 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cb6586:::benchmark_data_smithi194_186548_object58604:head# 2024-04-29T14:53:54.558 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.564 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cba04c:::benchmark_data_smithi194_186548_object73089:head# 2024-04-29T14:53:54.564 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.564 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cdf40b:::benchmark_data_smithi194_186548_object24988:head# 2024-04-29T14:53:54.564 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.565 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cdf98d:::benchmark_data_smithi194_186548_object37824:head# 2024-04-29T14:53:54.565 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.565 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cf179c:::benchmark_data_smithi194_186548_object28141:head# 2024-04-29T14:53:54.565 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.570 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cf2848:::benchmark_data_smithi194_186548_object40707:head# 2024-04-29T14:53:54.570 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.570 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cf5401:::benchmark_data_smithi194_186548_object3394:head# 2024-04-29T14:53:54.570 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.570 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cf5e44:::benchmark_data_smithi194_186548_object34824:head# 2024-04-29T14:53:54.570 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cf7181:::benchmark_data_smithi194_186548_object38205:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38cf73d7:::benchmark_data_smithi194_186548_object76770:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d0a18d:::benchmark_data_smithi194_186548_object38157:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d1abd1:::benchmark_data_smithi194_186548_object20914:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d1b4d3:::benchmark_data_smithi194_186548_object40913:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d2a4c5:::benchmark_data_smithi194_186548_object41419:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d38507:::benchmark_data_smithi194_186548_object74682:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d3a4d7:::benchmark_data_smithi194_186548_object19767:head# 2024-04-29T14:53:54.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.572 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d45f7a:::benchmark_data_smithi194_186548_object6039:head# 2024-04-29T14:53:54.572 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.572 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d50778:::benchmark_data_smithi194_186548_object84858:head# 2024-04-29T14:53:54.572 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.577 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d5995b:::benchmark_data_smithi194_186548_object55462:head# 2024-04-29T14:53:54.578 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.579 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d5ddfe:::benchmark_data_smithi194_186548_object84480:head# 2024-04-29T14:53:54.579 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.579 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d5ed99:::benchmark_data_smithi194_186548_object63021:head# 2024-04-29T14:53:54.579 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.579 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d66279:::benchmark_data_smithi194_186548_object31596:head# 2024-04-29T14:53:54.579 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.581 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d67486:::benchmark_data_smithi194_186548_object64722:head# 2024-04-29T14:53:54.581 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.581 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d6f3ae:::benchmark_data_smithi194_186548_object73076:head# 2024-04-29T14:53:54.581 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.581 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d70b93:::benchmark_data_smithi194_186548_object31418:head# 2024-04-29T14:53:54.581 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.581 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d8b32b:::benchmark_data_smithi194_186548_object70229:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38d9c912:::benchmark_data_smithi194_186548_object40150:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38dfce11:::benchmark_data_smithi194_186548_object23028:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e01efc:::benchmark_data_smithi194_186548_object21426:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e075c5:::benchmark_data_smithi194_186548_object21103:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e0bd43:::benchmark_data_smithi194_186548_object59816:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e1527e:::benchmark_data_smithi194_186548_object56467:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e28acd:::benchmark_data_smithi194_186548_object48117:head# 2024-04-29T14:53:54.582 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e2ee71:::benchmark_data_smithi194_186548_object78529:head# 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e4384c:::benchmark_data_smithi194_186548_object5552:head# 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e52e61:::benchmark_data_smithi194_186548_object59367:head# 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e55a74:::benchmark_data_smithi194_186548_object13728:head# 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e59ae4:::benchmark_data_smithi194_186548_object63544:head# 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e687d4:::benchmark_data_smithi194_186548_object15425:head# 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e6ffb4:::benchmark_data_smithi194_186548_object36485:head# 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38e8883a:::benchmark_data_smithi194_186548_object2678:head# 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ea7f28:::benchmark_data_smithi194_186548_object73023:head# 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38eab5a9:::benchmark_data_smithi194_186548_object63879:head# 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38eace89:::benchmark_data_smithi194_186548_object62862:head# 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38eb795c:::benchmark_data_smithi194_186548_object49380:head# 2024-04-29T14:53:54.586 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.587 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ec6a27:::benchmark_data_smithi194_186548_object28441:head# 2024-04-29T14:53:54.587 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.594 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38eccec2:::benchmark_data_smithi194_186548_object36240:head# 2024-04-29T14:53:54.594 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.598 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38eced4b:::benchmark_data_smithi194_186548_object33975:head# 2024-04-29T14:53:54.598 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.598 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ed952d:::benchmark_data_smithi194_186548_object51699:head# 2024-04-29T14:53:54.598 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.598 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38edcabd:::benchmark_data_smithi194_186548_object24336:head# 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38edfd18:::benchmark_data_smithi194_186548_object3201:head# 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ee449c:::benchmark_data_smithi194_186548_object60669:head# 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ef4e22:::benchmark_data_smithi194_186548_object22109:head# 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f011cb:::benchmark_data_smithi194_186548_object78127:head# 2024-04-29T14:53:54.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.600 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f191de:::benchmark_data_smithi194_186548_object33056:head# 2024-04-29T14:53:54.600 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f26617:::benchmark_data_smithi194_186548_object81930:head# 2024-04-29T14:53:54.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f4a79d:::benchmark_data_smithi194_186548_object64270:head# 2024-04-29T14:53:54.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.615 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f5066a:::benchmark_data_smithi194_186548_object17844:head# 2024-04-29T14:53:54.616 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.616 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f53fef:::benchmark_data_smithi194_186548_object16279:head# 2024-04-29T14:53:54.616 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.616 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f657e4:::benchmark_data_smithi194_186548_object85032:head# 2024-04-29T14:53:54.616 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.616 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f6ed7c:::benchmark_data_smithi194_186548_object25680:head# 2024-04-29T14:53:54.617 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.617 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f72b2d:::benchmark_data_smithi194_186548_object16434:head# 2024-04-29T14:53:54.618 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.618 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f8015b:::benchmark_data_smithi194_186548_object52639:head# 2024-04-29T14:53:54.618 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.618 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f96737:::benchmark_data_smithi194_186548_object26529:head# 2024-04-29T14:53:54.618 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.619 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38f9fefb:::benchmark_data_smithi194_186548_object12236:head# 2024-04-29T14:53:54.619 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.619 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fa2219:::benchmark_data_smithi194_186548_object43886:head# 2024-04-29T14:53:54.619 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.619 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38faae0b:::benchmark_data_smithi194_186548_object14726:head# 2024-04-29T14:53:54.620 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.620 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fb0466:::benchmark_data_smithi194_186548_object68356:head# 2024-04-29T14:53:54.620 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.620 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fb35cb:::benchmark_data_smithi194_186548_object57895:head# 2024-04-29T14:53:54.620 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.621 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fb7b2d:::benchmark_data_smithi194_186548_object23754:head# 2024-04-29T14:53:54.621 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fb83d0:::benchmark_data_smithi194_186548_object48229:head# 2024-04-29T14:53:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fc4acb:::benchmark_data_smithi194_186548_object77516:head# 2024-04-29T14:53:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fd1a27:::benchmark_data_smithi194_186548_object10215:head# 2024-04-29T14:53:54.622 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fda7ff:::benchmark_data_smithi194_186548_object57837:head# 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fdd4a4:::benchmark_data_smithi194_186548_object59366:head# 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fe389e:::benchmark_data_smithi194_186548_object83245:head# 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38fe538d:::benchmark_data_smithi194_186548_object61822:head# 2024-04-29T14:53:54.623 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.624 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ff3b71:::benchmark_data_smithi194_186548_object26388:head# 2024-04-29T14:53:54.624 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.625 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ffc25d:::benchmark_data_smithi194_186548_object994:head# 2024-04-29T14:53:54.626 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.630 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:38ffe69e:::benchmark_data_smithi194_186548_object40571:head# 2024-04-29T14:53:54.631 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.631 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39011f27:::benchmark_data_smithi194_186548_object76245:head# 2024-04-29T14:53:54.631 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.632 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39012527:::benchmark_data_smithi194_186548_object29409:head# 2024-04-29T14:53:54.633 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.633 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3902003a:::benchmark_data_smithi194_186548_object73405:head# 2024-04-29T14:53:54.633 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.633 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39028290:::benchmark_data_smithi194_186548_object40194:head# 2024-04-29T14:53:54.633 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.633 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3902a9bd:::benchmark_data_smithi194_186548_object73601:head# 2024-04-29T14:53:54.633 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3902d12c:::benchmark_data_smithi194_186548_object68749:head# 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3904a24c:::benchmark_data_smithi194_186548_object9303:head# 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3904f71d:::benchmark_data_smithi194_186548_object5207:head# 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390518b3:::benchmark_data_smithi194_186548_object20502:head# 2024-04-29T14:53:54.634 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.635 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390552b3:::benchmark_data_smithi194_186548_object67153:head# 2024-04-29T14:53:54.635 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3905551b:::benchmark_data_smithi194_186548_object68457:head# 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39088bcd:::benchmark_data_smithi194_186548_object71732:head# 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390932c7:::benchmark_data_smithi194_186548_object55228:head# 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390992c5:::benchmark_data_smithi194_186548_object9152:head# 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390b2b21:::benchmark_data_smithi194_186548_object75191:head# 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390b8c72:::benchmark_data_smithi194_186548_object38608:head# 2024-04-29T14:53:54.636 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.646 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390c4e1d:::benchmark_data_smithi194_186548_object13857:head# 2024-04-29T14:53:54.646 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.646 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390d7542:::benchmark_data_smithi194_186548_object53496:head# 2024-04-29T14:53:54.646 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.647 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:390f2520:::benchmark_data_smithi194_186548_object55337:head# 2024-04-29T14:53:54.647 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.647 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39107268:::benchmark_data_smithi194_186548_object61919:head# 2024-04-29T14:53:54.647 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.648 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3910b976:::benchmark_data_smithi194_186548_object20936:head# 2024-04-29T14:53:54.648 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39124137:::benchmark_data_smithi194_186548_object72622:head# 2024-04-29T14:53:54.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39126096:::benchmark_data_smithi194_186548_object9345:head# 2024-04-29T14:53:54.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39128b58:::benchmark_data_smithi194_186548_object70978:head# 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39157b44:::benchmark_data_smithi194_186548_object57305:head# 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391a5403:::benchmark_data_smithi194_186548_object73551:head# 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391ac410:::benchmark_data_smithi194_186548_object57244:head# 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391b9978:::benchmark_data_smithi194_186548_object12821:head# 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391baf69:::benchmark_data_smithi194_186548_object64416:head# 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391d17f5:::benchmark_data_smithi194_186548_object74042:head# 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391d50e9:::benchmark_data_smithi194_186548_object58239:head# 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391e452c:::benchmark_data_smithi194_186548_object70305:head# 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391e5a7d:::benchmark_data_smithi194_186548_object45151:head# 2024-04-29T14:53:54.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.660 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391e73fe:::benchmark_data_smithi194_186548_object69617:head# 2024-04-29T14:53:54.660 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.660 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:391f61a8:::benchmark_data_smithi194_186548_object13225:head# 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39204f76:::benchmark_data_smithi194_186548_object40441:head# 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3920a189:::benchmark_data_smithi194_186548_object9050:head# 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3922e6bd:::benchmark_data_smithi194_186548_object41168:head# 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3924814d:::benchmark_data_smithi194_186548_object61978:head# 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39264078:::benchmark_data_smithi194_186548_object13076:head# 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39285f1b:::benchmark_data_smithi194_186548_object51289:head# 2024-04-29T14:53:54.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392893ca:::benchmark_data_smithi194_186548_object66538:head# 2024-04-29T14:53:54.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39290cea:::benchmark_data_smithi194_186548_object44850:head# 2024-04-29T14:53:54.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.663 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39295ff7:::benchmark_data_smithi194_186548_object1170:head# 2024-04-29T14:53:54.663 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.663 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3929fe55:::benchmark_data_smithi194_186548_object15491:head# 2024-04-29T14:53:54.663 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392a1243:::benchmark_data_smithi194_186548_object75374:head# 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392baf7d:::benchmark_data_smithi194_186548_object53111:head# 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392d07b1:::benchmark_data_smithi194_186548_object52139:head# 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392d08a4:::benchmark_data_smithi194_186548_object79064:head# 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392d1c2a:::benchmark_data_smithi194_186548_object66976:head# 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392e3757:::benchmark_data_smithi194_186548_object16515:head# 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.664 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392e9b8b:::benchmark_data_smithi194_186548_object44188:head# 2024-04-29T14:53:54.665 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.665 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392f0860:::benchmark_data_smithi194_186548_object67208:head# 2024-04-29T14:53:54.665 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.666 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:392f7bb6:::benchmark_data_smithi194_186548_object29185:head# 2024-04-29T14:53:54.666 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.666 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3930d89a:::benchmark_data_smithi194_186548_object19976:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3930f6e0:::benchmark_data_smithi194_186548_object11155:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3931e018:::benchmark_data_smithi194_186548_object85114:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39324982:::benchmark_data_smithi194_186548_object16600:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393337fd:::benchmark_data_smithi194_186548_object63959:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39341eb2:::benchmark_data_smithi194_186548_object18705:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3934b02a:::benchmark_data_smithi194_186548_object35565:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39350339:::benchmark_data_smithi194_186548_object10769:head# 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3936c173:::benchmark_data_smithi194_186548_object67385:head# 2024-04-29T14:53:54.668 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.668 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39375dc8:::benchmark_data_smithi194_186548_object27170:head# 2024-04-29T14:53:54.668 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.670 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3938c676:::benchmark_data_smithi194_186548_object4192:head# 2024-04-29T14:53:54.670 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39392b76:::benchmark_data_smithi194_186548_object4335:head# 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39396ab4:::benchmark_data_smithi194_186548_object52416:head# 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3939ca39:::benchmark_data_smithi194_186548_object75130:head# 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393ad217:::benchmark_data_smithi194_186548_object31244:head# 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393aef3d:::benchmark_data_smithi194_186548_object78294:head# 2024-04-29T14:53:54.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.674 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393b02a3:::benchmark_data_smithi194_186548_object14483:head# 2024-04-29T14:53:54.674 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.674 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393b453f:::benchmark_data_smithi194_186548_object20363:head# 2024-04-29T14:53:54.674 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393bb3a3:::benchmark_data_smithi194_186548_object65170:head# 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393c1822:::benchmark_data_smithi194_186548_object21397:head# 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393c183e:::benchmark_data_smithi194_186548_object11455:head# 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393c7352:::benchmark_data_smithi194_186548_object24418:head# 2024-04-29T14:53:54.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.676 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393c7ef4:::benchmark_data_smithi194_186548_object50835:head# 2024-04-29T14:53:54.676 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.676 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393e4462:::benchmark_data_smithi194_186548_object10282:head# 2024-04-29T14:53:54.676 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:393f541a:::benchmark_data_smithi194_186548_object67060:head# 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3940e462:::benchmark_data_smithi194_186548_object20169:head# 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3940f666:::benchmark_data_smithi194_186548_object66074:head# 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39410690:::benchmark_data_smithi194_186548_object9703:head# 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3941150c:::benchmark_data_smithi194_186548_object50577:head# 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3941568a:::benchmark_data_smithi194_186548_object63389:head# 2024-04-29T14:53:54.677 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3941821f:::benchmark_data_smithi194_186548_object61840:head# 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394344a2:::benchmark_data_smithi194_186548_object24188:head# 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39436a2e:::benchmark_data_smithi194_186548_object36223:head# 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394486f1:::benchmark_data_smithi194_186548_object38595:head# 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3944c0c4:::benchmark_data_smithi194_186548_object80253:head# 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3944c79f:::benchmark_data_smithi194_186548_object23599:head# 2024-04-29T14:53:54.681 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3945088d:::benchmark_data_smithi194_186548_object20225:head# 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3945c028:::benchmark_data_smithi194_186548_object28612:head# 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3945cbd7:::benchmark_data_smithi194_186548_object85335:head# 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394603a4:::benchmark_data_smithi194_186548_object78654:head# 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3947afe4:::benchmark_data_smithi194_186548_object23217:head# 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3947da9e:::benchmark_data_smithi194_186548_object3698:head# 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3948567e:::benchmark_data_smithi194_186548_object54401:head# 2024-04-29T14:53:54.682 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.683 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3948a90f:::benchmark_data_smithi194_186548_object43415:head# 2024-04-29T14:53:54.683 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.683 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3948da92:::benchmark_data_smithi194_186548_object40454:head# 2024-04-29T14:53:54.683 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.683 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394a590d:::benchmark_data_smithi194_186548_object49599:head# 2024-04-29T14:53:54.683 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394a8993:::benchmark_data_smithi194_186548_object81695:head# 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394bd1f1:::benchmark_data_smithi194_186548_object57513:head# 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394c9303:::benchmark_data_smithi194_186548_object62471:head# 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394cf157:::benchmark_data_smithi194_186548_object36108:head# 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394ddab4:::benchmark_data_smithi194_186548_object43241:head# 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394e56ee:::benchmark_data_smithi194_186548_object53466:head# 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394e6a91:::benchmark_data_smithi194_186548_object22921:head# 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394e84f9:::benchmark_data_smithi194_186548_object57521:head# 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394ea91d:::benchmark_data_smithi194_186548_object18871:head# 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394f3db7:::benchmark_data_smithi194_186548_object66781:head# 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:394f9626:::benchmark_data_smithi194_186548_object28056:head# 2024-04-29T14:53:54.686 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.688 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3951d457:::benchmark_data_smithi194_186548_object49575:head# 2024-04-29T14:53:54.688 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.689 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:395319ca:::benchmark_data_smithi194_186548_object55223:head# 2024-04-29T14:53:54.689 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.689 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:395400cb:::benchmark_data_smithi194_186548_object64512:head# 2024-04-29T14:53:54.689 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.697 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39542c69:::benchmark_data_smithi194_186548_object78991:head# 2024-04-29T14:53:54.697 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.697 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39544f26:::benchmark_data_smithi194_186548_object27683:head# 2024-04-29T14:53:54.697 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.697 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:395531ca:::benchmark_data_smithi194_186548_object57843:head# 2024-04-29T14:53:54.697 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.698 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3955a046:::benchmark_data_smithi194_186548_object21888:head# 2024-04-29T14:53:54.698 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.698 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3955ee48:::benchmark_data_smithi194_186548_object20850:head# 2024-04-29T14:53:54.698 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.699 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39563c0c:::benchmark_data_smithi194_186548_object9267:head# 2024-04-29T14:53:54.699 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.699 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39564d68:::benchmark_data_smithi194_186548_object46735:head# 2024-04-29T14:53:54.699 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.699 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3956e8c2:::benchmark_data_smithi194_186548_object51023:head# 2024-04-29T14:53:54.699 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.699 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3956f0f7:::benchmark_data_smithi194_186548_object47212:head# 2024-04-29T14:53:54.700 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3958a58a:::benchmark_data_smithi194_186548_object51285:head# 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3958d362:::benchmark_data_smithi194_186548_object6758:head# 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39590b3b:::benchmark_data_smithi194_186548_object56517:head# 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:395a6b51:::benchmark_data_smithi194_186548_object22096:head# 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:395ab122:::benchmark_data_smithi194_186548_object69871:head# 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:395d761c:::benchmark_data_smithi194_186548_object23870:head# 2024-04-29T14:53:54.701 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.702 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:395efc1e:::benchmark_data_smithi194_186548_object51472:head# 2024-04-29T14:53:54.702 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.702 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39606d92:::benchmark_data_smithi194_186548_object48017:head# 2024-04-29T14:53:54.702 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.702 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39606fb1:::benchmark_data_smithi194_186548_object60727:head# 2024-04-29T14:53:54.702 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.704 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39610924:::benchmark_data_smithi194_186548_object81107:head# 2024-04-29T14:53:54.704 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.705 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396177b4:::benchmark_data_smithi194_186548_object35231:head# 2024-04-29T14:53:54.705 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.705 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3961cc82:::benchmark_data_smithi194_186548_object55483:head# 2024-04-29T14:53:54.705 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.705 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39622044:::benchmark_data_smithi194_186548_object81526:head# 2024-04-29T14:53:54.705 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.706 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39628b13:::benchmark_data_smithi194_186548_object79793:head# 2024-04-29T14:53:54.706 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.706 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3962ab5f:::benchmark_data_smithi194_186548_object60421:head# 2024-04-29T14:53:54.706 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.706 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39630212:::benchmark_data_smithi194_186548_object24466:head# 2024-04-29T14:53:54.706 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39630aa0:::benchmark_data_smithi194_186548_object32245:head# 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3963eab4:::benchmark_data_smithi194_186548_object11490:head# 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396443db:::benchmark_data_smithi194_186548_object11666:head# 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39655323:::benchmark_data_smithi194_186548_object76164:head# 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39659e4b:::benchmark_data_smithi194_186548_object43763:head# 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3965f337:::benchmark_data_smithi194_186548_object68618:head# 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396736c0:::benchmark_data_smithi194_186548_object24475:head# 2024-04-29T14:53:54.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.710 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39675759:::benchmark_data_smithi194_186548_object4349:head# 2024-04-29T14:53:54.710 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39681af9:::benchmark_data_smithi194_186548_object18308:head# 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3968c666:::benchmark_data_smithi194_186548_object22518:head# 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39693899:::benchmark_data_smithi194_186548_object3412:head# 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3969db90:::benchmark_data_smithi194_186548_object77843:head# 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.712 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396aa8ce:::benchmark_data_smithi194_186548_object31849:head# 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396ac0b7:::benchmark_data_smithi194_186548_object77104:head# 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396b5584:::benchmark_data_smithi194_186548_object49928:head# 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396ba0a2:::benchmark_data_smithi194_186548_object54995:head# 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396d2501:::benchmark_data_smithi194_186548_object67998:head# 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396d7ac8:::benchmark_data_smithi194_186548_object51397:head# 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396dba54:::benchmark_data_smithi194_186548_object5876:head# 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.713 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396dd58a:::benchmark_data_smithi194_186548_object17513:head# 2024-04-29T14:53:54.714 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.714 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396df3e2:::benchmark_data_smithi194_186548_object60070:head# 2024-04-29T14:53:54.714 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.714 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396ebc1f:::benchmark_data_smithi194_186548_object24924:head# 2024-04-29T14:53:54.714 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.716 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396ee3c4:::benchmark_data_smithi194_186548_object78728:head# 2024-04-29T14:53:54.716 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.716 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396f1cf9:::benchmark_data_smithi194_186548_object2449:head# 2024-04-29T14:53:54.716 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.716 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396fc3b5:::benchmark_data_smithi194_186548_object14187:head# 2024-04-29T14:53:54.716 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:396fe504:::benchmark_data_smithi194_186548_object81581:head# 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39706c4b:::benchmark_data_smithi194_186548_object14365:head# 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39718b86:::benchmark_data_smithi194_186548_object60667:head# 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:397221e3:::benchmark_data_smithi194_186548_object61025:head# 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39723231:::benchmark_data_smithi194_186548_object66657:head# 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3972c3ea:::benchmark_data_smithi194_186548_object56361:head# 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3972d4c9:::benchmark_data_smithi194_186548_object32487:head# 2024-04-29T14:53:54.717 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3972f06d:::benchmark_data_smithi194_186548_object1252:head# 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39735473:::benchmark_data_smithi194_186548_object59417:head# 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3973d94c:::benchmark_data_smithi194_186548_object52026:head# 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3973dd14:::benchmark_data_smithi194_186548_object37309:head# 2024-04-29T14:53:54.718 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39763576:::benchmark_data_smithi194_186548_object80829:head# 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39786771:::benchmark_data_smithi194_186548_object22651:head# 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:397b1e1f:::benchmark_data_smithi194_186548_object67435:head# 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:397ca17a:::benchmark_data_smithi194_186548_object68512:head# 2024-04-29T14:53:54.719 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.720 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:397cded8:::benchmark_data_smithi194_186548_object17168:head# 2024-04-29T14:53:54.720 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:54.722 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:397ed627:::benchmark_data_smithi194_186548_object49750:head# 2024-04-29T14:53:54.722 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.725 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:397fb871:::benchmark_data_smithi194_186548_object74008:head# 2024-04-29T14:53:54.725 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.725 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39806b6d:::benchmark_data_smithi194_186548_object61286:head# 2024-04-29T14:53:54.725 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.726 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3981646b:::benchmark_data_smithi194_186548_object48065:head# 2024-04-29T14:53:54.726 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.726 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39824538:::benchmark_data_smithi194_186548_object16738:head# 2024-04-29T14:53:54.726 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.726 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39828805:::benchmark_data_smithi194_186548_object13160:head# 2024-04-29T14:53:54.726 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:54.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:54.728 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3982b37f:::benchmark_data_smithi194_186548_object60047:head# 2024-04-29T14:53:54.728 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.728 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3983aeeb:::benchmark_data_smithi194_186548_object68333:head# 2024-04-29T14:53:54.728 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398487da:::benchmark_data_smithi194_186548_object82926:head# 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39857f5b:::benchmark_data_smithi194_186548_object56638:head# 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3985b4ca:::benchmark_data_smithi194_186548_object79496:head# 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3986de10:::benchmark_data_smithi194_186548_object52990:head# 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39888b7b:::benchmark_data_smithi194_186548_object28767:head# 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3988f1e3:::benchmark_data_smithi194_186548_object46864:head# 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.729 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398a95c9:::benchmark_data_smithi194_186548_object83447:head# 2024-04-29T14:53:54.730 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398a9c67:::benchmark_data_smithi194_186548_object28462:head# 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398aceea:::benchmark_data_smithi194_186548_object46216:head# 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398b2f79:::benchmark_data_smithi194_186548_object10929:head# 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398b8267:::benchmark_data_smithi194_186548_object9300:head# 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398bc821:::benchmark_data_smithi194_186548_object77764:head# 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398c871f:::benchmark_data_smithi194_186548_object22019:head# 2024-04-29T14:53:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.732 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398dedd6:::benchmark_data_smithi194_186548_object19149:head# 2024-04-29T14:53:54.732 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.732 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398e9e8d:::benchmark_data_smithi194_186548_object51883:head# 2024-04-29T14:53:54.732 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398ea7b9:::benchmark_data_smithi194_186548_object27225:head# 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398f0d53:::benchmark_data_smithi194_186548_object69599:head# 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398f3409:::benchmark_data_smithi194_186548_object40249:head# 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:398f6509:::benchmark_data_smithi194_186548_object49802:head# 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39908d60:::benchmark_data_smithi194_186548_object61521:head# 2024-04-29T14:53:54.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3990a9be:::benchmark_data_smithi194_186548_object26564:head# 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39910922:::benchmark_data_smithi194_186548_object12563:head# 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39921449:::benchmark_data_smithi194_186548_object78028:head# 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3992414f:::benchmark_data_smithi194_186548_object52467:head# 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3992e4ea:::benchmark_data_smithi194_186548_object67202:head# 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39946d85:::benchmark_data_smithi194_186548_object14241:head# 2024-04-29T14:53:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.737 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3994be9d:::benchmark_data_smithi194_186548_object80398:head# 2024-04-29T14:53:54.737 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.737 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3994df4d:::benchmark_data_smithi194_186548_object58775:head# 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3995c8de:::benchmark_data_smithi194_186548_object16055:head# 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3996f7cc:::benchmark_data_smithi194_186548_object6930:head# 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399778b5:::benchmark_data_smithi194_186548_object78691:head# 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39985b5b:::benchmark_data_smithi194_186548_object57029:head# 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3998f7e6:::benchmark_data_smithi194_186548_object72867:head# 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39995955:::benchmark_data_smithi194_186548_object80006:head# 2024-04-29T14:53:54.738 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3999932a:::benchmark_data_smithi194_186548_object46574:head# 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399a9174:::benchmark_data_smithi194_186548_object40649:head# 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399b1a81:::benchmark_data_smithi194_186548_object36314:head# 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399bb253:::benchmark_data_smithi194_186548_object83605:head# 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399c1d89:::benchmark_data_smithi194_186548_object29845:head# 2024-04-29T14:53:54.739 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.740 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399d4fd1:::benchmark_data_smithi194_186548_object24442:head# 2024-04-29T14:53:54.740 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.740 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399e5ef4:::benchmark_data_smithi194_186548_object40891:head# 2024-04-29T14:53:54.740 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399efd94:::benchmark_data_smithi194_186548_object64015:head# 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:399fff18:::benchmark_data_smithi194_186548_object8179:head# 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a046ae:::benchmark_data_smithi194_186548_object12316:head# 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a06da6:::benchmark_data_smithi194_186548_object78313:head# 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a166a9:::benchmark_data_smithi194_186548_object50552:head# 2024-04-29T14:53:54.741 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.744 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a29adf:::benchmark_data_smithi194_186548_object57256:head# 2024-04-29T14:53:54.744 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.744 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a2aa84:::benchmark_data_smithi194_186548_object45523:head# 2024-04-29T14:53:54.744 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a2b424:::benchmark_data_smithi194_186548_object80412:head# 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a2d7d9:::benchmark_data_smithi194_186548_object84123:head# 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a2f621:::benchmark_data_smithi194_186548_object75884:head# 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a3778d:::benchmark_data_smithi194_186548_object16926:head# 2024-04-29T14:53:54.745 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.746 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a4c0f7:::benchmark_data_smithi194_186548_object75579:head# 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a59f7b:::benchmark_data_smithi194_186548_object14316:head# 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a6c9ef:::benchmark_data_smithi194_186548_object20952:head# 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a77b45:::benchmark_data_smithi194_186548_object72262:head# 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39a94420:::benchmark_data_smithi194_186548_object8311:head# 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39aaafb4:::benchmark_data_smithi194_186548_object63015:head# 2024-04-29T14:53:54.747 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.748 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39aec591:::benchmark_data_smithi194_186548_object61103:head# 2024-04-29T14:53:54.748 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.748 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39aecf3f:::benchmark_data_smithi194_186548_object61130:head# 2024-04-29T14:53:54.748 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.748 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39af5e4c:::benchmark_data_smithi194_186548_object45614:head# 2024-04-29T14:53:54.748 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.748 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b01875:::benchmark_data_smithi194_186548_object78042:head# 2024-04-29T14:53:54.749 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.754 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b097d3:::benchmark_data_smithi194_186548_object15835:head# 2024-04-29T14:53:54.754 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.754 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b0b19e:::benchmark_data_smithi194_186548_object46400:head# 2024-04-29T14:53:54.754 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.754 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b0f323:::benchmark_data_smithi194_186548_object39348:head# 2024-04-29T14:53:54.754 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b1bdf0:::benchmark_data_smithi194_186548_object2256:head# 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b1d11d:::benchmark_data_smithi194_186548_object32606:head# 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b45833:::benchmark_data_smithi194_186548_object67550:head# 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b4b473:::benchmark_data_smithi194_186548_object68045:head# 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b4bd49:::benchmark_data_smithi194_186548_object48353:head# 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b5ca8e:::benchmark_data_smithi194_186548_object12294:head# 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b7952d:::benchmark_data_smithi194_186548_object68271:head# 2024-04-29T14:53:54.755 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b796ee:::benchmark_data_smithi194_186548_object78817:head# 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b89b8a:::benchmark_data_smithi194_186548_object49517:head# 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39b9cc9f:::benchmark_data_smithi194_186548_object44445:head# 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39baec20:::benchmark_data_smithi194_186548_object62433:head# 2024-04-29T14:53:54.756 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:53:54.764 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39bb5932:::benchmark_data_smithi194_186548_object63615:head# 2024-04-29T14:53:54.764 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39bbcc6e:::benchmark_data_smithi194_186548_object9705:head# 2024-04-29T14:53:54.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39be1120:::benchmark_data_smithi194_186548_object76967:head# 2024-04-29T14:53:54.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39be65ec:::benchmark_data_smithi194_186548_object35033:head# 2024-04-29T14:53:54.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39befc5b:::benchmark_data_smithi194_186548_object66136:head# 2024-04-29T14:53:54.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.769 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39bf2475:::benchmark_data_smithi194_186548_object75080:head# 2024-04-29T14:53:54.769 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39bf5b2c:::benchmark_data_smithi194_186548_object54700:head# 2024-04-29T14:53:54.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39bfabe8:::benchmark_data_smithi194_186548_object25591:head# 2024-04-29T14:53:54.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.770 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39bfbfc5:::benchmark_data_smithi194_186548_object79303:head# 2024-04-29T14:53:54.770 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c21410:::benchmark_data_smithi194_186548_object43617:head# 2024-04-29T14:53:54.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c33e57:::benchmark_data_smithi194_186548_object65727:head# 2024-04-29T14:53:54.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c408a8:::benchmark_data_smithi194_186548_object76639:head# 2024-04-29T14:53:54.771 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.771 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c44e75:::benchmark_data_smithi194_186548_object39932:head# 2024-04-29T14:53:54.772 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c4af8b:::benchmark_data_smithi194_186548_object81110:head# 2024-04-29T14:53:54.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c4e1c4:::benchmark_data_smithi194_186548_object51697:head# 2024-04-29T14:53:54.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c5877f:::benchmark_data_smithi194_186548_object45240:head# 2024-04-29T14:53:54.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c667a7:::benchmark_data_smithi194_186548_object34669:head# 2024-04-29T14:53:54.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c7d4b0:::benchmark_data_smithi194_186548_object49335:head# 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c8140e:::benchmark_data_smithi194_186548_object2264:head# 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c82a97:::benchmark_data_smithi194_186548_object20121:head# 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c8c90b:::benchmark_data_smithi194_186548_object44171:head# 2024-04-29T14:53:54.782 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39c94188:::benchmark_data_smithi194_186548_object45443:head# 2024-04-29T14:53:54.783 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.783 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39ca6a66:::benchmark_data_smithi194_186548_object85159:head# 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cb758d:::benchmark_data_smithi194_186548_object45554:head# 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cb8f8f:::benchmark_data_smithi194_186548_object26287:head# 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cc1be8:::benchmark_data_smithi194_186548_object29306:head# 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.784 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cc92a6:::benchmark_data_smithi194_186548_object52422:head# 2024-04-29T14:53:54.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cd490d:::benchmark_data_smithi194_186548_object41266:head# 2024-04-29T14:53:54.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39ce11e7:::benchmark_data_smithi194_186548_object7440:head# 2024-04-29T14:53:54.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cf6813:::benchmark_data_smithi194_186548_object72630:head# 2024-04-29T14:53:54.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cfdd99:::benchmark_data_smithi194_186548_object68675:head# 2024-04-29T14:53:54.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39cff183:::benchmark_data_smithi194_186548_object84589:head# 2024-04-29T14:53:54.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39d01eb5:::benchmark_data_smithi194_186548_object51907:head# 2024-04-29T14:53:54.790 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39d26fce:::benchmark_data_smithi194_186548_object68089:head# 2024-04-29T14:53:54.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.796 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39d466bf:::benchmark_data_smithi194_186548_object4076:head# 2024-04-29T14:53:54.796 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.803 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39d5d1d5:::benchmark_data_smithi194_186548_object18730:head# 2024-04-29T14:53:54.803 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39d638dd:::benchmark_data_smithi194_186548_object11404:head# 2024-04-29T14:53:54.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39d666d5:::benchmark_data_smithi194_186548_object70550:head# 2024-04-29T14:53:54.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39d7f441:::benchmark_data_smithi194_186548_object48205:head# 2024-04-29T14:53:54.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39dc1879:::benchmark_data_smithi194_186548_object62660:head# 2024-04-29T14:53:54.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39dd2cb0:::benchmark_data_smithi194_186548_object57022:head# 2024-04-29T14:53:54.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.817 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:53:54.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39deb020:::benchmark_data_smithi194_186548_object17419:head# 2024-04-29T14:53:54.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.824 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39df026c:::benchmark_data_smithi194_186548_object4497:head# 2024-04-29T14:53:54.825 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:53:54.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39df6b6c:::benchmark_data_smithi194_186548_object49977:head# 2024-04-29T14:53:54.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39dff3e1:::benchmark_data_smithi194_186548_object52669:head# 2024-04-29T14:53:54.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e1b7a0:::benchmark_data_smithi194_186548_object2788:head# 2024-04-29T14:53:54.829 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.829 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e2d2f5:::benchmark_data_smithi194_186548_object13433:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e32ec2:::benchmark_data_smithi194_186548_object42346:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e34311:::benchmark_data_smithi194_186548_object67440:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e4cdb5:::benchmark_data_smithi194_186548_object49816:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e5b614:::benchmark_data_smithi194_186548_object9665:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e6e1dc:::benchmark_data_smithi194_186548_object45843:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e700af:::benchmark_data_smithi194_186548_object77934:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e768bc:::benchmark_data_smithi194_186548_object49846:head# 2024-04-29T14:53:54.830 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e76d2f:::benchmark_data_smithi194_186548_object56423:head# 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e981be:::benchmark_data_smithi194_186548_object40926:head# 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e99dae:::benchmark_data_smithi194_186548_object18483:head# 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39e9f2f7:::benchmark_data_smithi194_186548_object13918:head# 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39eac07c:::benchmark_data_smithi194_186548_object8200:head# 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39eb7057:::benchmark_data_smithi194_186548_object15541:head# 2024-04-29T14:53:54.832 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39ec225e:::benchmark_data_smithi194_186548_object44505:head# 2024-04-29T14:53:54.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39eda3b9:::benchmark_data_smithi194_186548_object72364:head# 2024-04-29T14:53:54.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39ee3572:::benchmark_data_smithi194_186548_object68237:head# 2024-04-29T14:53:54.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39ee47de:::benchmark_data_smithi194_186548_object18259:head# 2024-04-29T14:53:54.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39efc5ee:::benchmark_data_smithi194_186548_object20213:head# 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f040bd:::benchmark_data_smithi194_186548_object15690:head# 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f04adb:::benchmark_data_smithi194_186548_object4198:head# 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f0c364:::benchmark_data_smithi194_186548_object54835:head# 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f0d164:::benchmark_data_smithi194_186548_object55596:head# 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f1bdc2:::benchmark_data_smithi194_186548_object42191:head# 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f29aa3:::benchmark_data_smithi194_186548_object73249:head# 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f2a983:::benchmark_data_smithi194_186548_object49499:head# 2024-04-29T14:53:54.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f401dc:::benchmark_data_smithi194_186548_object67668:head# 2024-04-29T14:53:54.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f47944:::benchmark_data_smithi194_186548_object58555:head# 2024-04-29T14:53:54.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f4f2f1:::benchmark_data_smithi194_186548_object79110:head# 2024-04-29T14:53:54.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f50ba5:::benchmark_data_smithi194_186548_object69489:head# 2024-04-29T14:53:54.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f56ae0:::benchmark_data_smithi194_186548_object70354:head# 2024-04-29T14:53:54.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f5984e:::benchmark_data_smithi194_186548_object1835:head# 2024-04-29T14:53:54.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f5b604:::benchmark_data_smithi194_186548_object35304:head# 2024-04-29T14:53:54.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f6aa32:::benchmark_data_smithi194_186548_object72282:head# 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f6d5c7:::benchmark_data_smithi194_186548_object6538:head# 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f72a60:::benchmark_data_smithi194_186548_object23689:head# 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f76729:::benchmark_data_smithi194_186548_object30477:head# 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f79f48:::benchmark_data_smithi194_186548_object79391:head# 2024-04-29T14:53:54.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f7c44d:::benchmark_data_smithi194_186548_object65203:head# 2024-04-29T14:53:54.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f94c59:::benchmark_data_smithi194_186548_object58965:head# 2024-04-29T14:53:54.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39f96a7f:::benchmark_data_smithi194_186548_object45469:head# 2024-04-29T14:53:54.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39fa8276:::benchmark_data_smithi194_186548_object78061:head# 2024-04-29T14:53:54.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39fab6b8:::benchmark_data_smithi194_186548_object31618:head# 2024-04-29T14:53:54.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.847 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39fbb453:::benchmark_data_smithi194_186548_object26874:head# 2024-04-29T14:53:54.847 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39fcde7f:::benchmark_data_smithi194_186548_object16760:head# 2024-04-29T14:53:54.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39fe1a87:::benchmark_data_smithi194_186548_object26103:head# 2024-04-29T14:53:54.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.849 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39fe4a8d:::benchmark_data_smithi194_186548_object55112:head# 2024-04-29T14:53:54.849 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:39fe9698:::benchmark_data_smithi194_186548_object2049:head# 2024-04-29T14:53:54.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a01a3d8:::benchmark_data_smithi194_186548_object43855:head# 2024-04-29T14:53:54.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a01b149:::benchmark_data_smithi194_186548_object76873:head# 2024-04-29T14:53:54.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a02751a:::benchmark_data_smithi194_186548_object55062:head# 2024-04-29T14:53:54.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a02e573:::benchmark_data_smithi194_186548_object36474:head# 2024-04-29T14:53:54.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a05857b:::benchmark_data_smithi194_186548_object44744:head# 2024-04-29T14:53:54.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a06a2d2:::benchmark_data_smithi194_186548_object42153:head# 2024-04-29T14:53:54.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.852 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a08b4f8:::benchmark_data_smithi194_186548_object34565:head# 2024-04-29T14:53:54.852 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a08d9c3:::benchmark_data_smithi194_186548_object65493:head# 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a0baa29:::benchmark_data_smithi194_186548_object31402:head# 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a0d3300:::benchmark_data_smithi194_186548_object4974:head# 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a0de93b:::benchmark_data_smithi194_186548_object27581:head# 2024-04-29T14:53:54.853 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a0f352e:::benchmark_data_smithi194_186548_object41876:head# 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1190ee:::benchmark_data_smithi194_186548_object78118:head# 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a11b19b:::benchmark_data_smithi194_186548_object7030:head# 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a11daa6:::benchmark_data_smithi194_186548_object83392:head# 2024-04-29T14:53:54.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a12b7c2:::benchmark_data_smithi194_186548_object11739:head# 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a130403:::benchmark_data_smithi194_186548_object69331:head# 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a13067b:::benchmark_data_smithi194_186548_object59002:head# 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a13697b:::benchmark_data_smithi194_186548_object67226:head# 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a13d524:::benchmark_data_smithi194_186548_object29235:head# 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a146559:::benchmark_data_smithi194_186548_object70575:head# 2024-04-29T14:53:54.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1513f6:::benchmark_data_smithi194_186548_object56104:head# 2024-04-29T14:53:54.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.857 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a151c68:::benchmark_data_smithi194_186548_object55310:head# 2024-04-29T14:53:54.857 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a15221e:::benchmark_data_smithi194_186548_object42251:head# 2024-04-29T14:53:54.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a15319c:::benchmark_data_smithi194_186548_object49673:head# 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a15adce:::benchmark_data_smithi194_186548_object72493:head# 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a15f5be:::benchmark_data_smithi194_186548_object85025:head# 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a16ad0e:::benchmark_data_smithi194_186548_object42655:head# 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a18042c:::benchmark_data_smithi194_186548_object7972:head# 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a185a25:::benchmark_data_smithi194_186548_object79107:head# 2024-04-29T14:53:54.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1895e9:::benchmark_data_smithi194_186548_object50908:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a18aae4:::benchmark_data_smithi194_186548_object33428:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a18cecf:::benchmark_data_smithi194_186548_object22028:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a19d0ca:::benchmark_data_smithi194_186548_object75614:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1a1d91:::benchmark_data_smithi194_186548_object65477:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1ab6d1:::benchmark_data_smithi194_186548_object35986:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1ad9ec:::benchmark_data_smithi194_186548_object42714:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1d1544:::benchmark_data_smithi194_186548_object247:head# 2024-04-29T14:53:54.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1dbbbc:::benchmark_data_smithi194_186548_object27827:head# 2024-04-29T14:53:54.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a1fd45a:::benchmark_data_smithi194_186548_object64982:head# 2024-04-29T14:53:54.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a208028:::benchmark_data_smithi194_186548_object70431:head# 2024-04-29T14:53:54.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a21b86b:::benchmark_data_smithi194_186548_object8230:head# 2024-04-29T14:53:54.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a248db0:::benchmark_data_smithi194_186548_object27915:head# 2024-04-29T14:53:54.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.868 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:54.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a249b2f:::benchmark_data_smithi194_186548_object49112:head# 2024-04-29T14:53:54.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a268a17:::benchmark_data_smithi194_186548_object17563:head# 2024-04-29T14:53:54.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a268ab5:::benchmark_data_smithi194_186548_object19893:head# 2024-04-29T14:53:54.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a276bde:::benchmark_data_smithi194_186548_object8050:head# 2024-04-29T14:53:54.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a28aa40:::benchmark_data_smithi194_186548_object85126:head# 2024-04-29T14:53:54.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a291d49:::benchmark_data_smithi194_186548_object18283:head# 2024-04-29T14:53:54.874 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.874 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a294895:::benchmark_data_smithi194_186548_object42062:head# 2024-04-29T14:53:54.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.875 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a2a66a5:::benchmark_data_smithi194_186548_object47041:head# 2024-04-29T14:53:54.875 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a2bb3f8:::benchmark_data_smithi194_186548_object53188:head# 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a2c4b0a:::benchmark_data_smithi194_186548_object9945:head# 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a2d37e5:::benchmark_data_smithi194_186548_object32023:head# 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a2f1b32:::benchmark_data_smithi194_186548_object64414:head# 2024-04-29T14:53:54.877 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a2f5b57:::benchmark_data_smithi194_186548_object32891:head# 2024-04-29T14:53:54.879 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a2fa645:::benchmark_data_smithi194_186548_object26950:head# 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a302e60:::benchmark_data_smithi194_186548_object48237:head# 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a32f329:::benchmark_data_smithi194_186548_object71534:head# 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a333a1f:::benchmark_data_smithi194_186548_object44321:head# 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3470ca:::benchmark_data_smithi194_186548_object81501:head# 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3592a1:::benchmark_data_smithi194_186548_object11766:head# 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a35b64b:::benchmark_data_smithi194_186548_object48889:head# 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a35b8e4:::benchmark_data_smithi194_186548_object58217:head# 2024-04-29T14:53:54.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a36b842:::benchmark_data_smithi194_186548_object58352:head# 2024-04-29T14:53:54.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3800bf:::benchmark_data_smithi194_186548_object67148:head# 2024-04-29T14:53:54.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a382119:::benchmark_data_smithi194_186548_object41650:head# 2024-04-29T14:53:54.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a383643:::benchmark_data_smithi194_186548_object37487:head# 2024-04-29T14:53:54.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.888 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a399cce:::benchmark_data_smithi194_186548_object6557:head# 2024-04-29T14:53:54.888 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.890 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3cb712:::benchmark_data_smithi194_186548_object45055:head# 2024-04-29T14:53:54.890 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.890 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3cc09d:::benchmark_data_smithi194_186548_object51842:head# 2024-04-29T14:53:54.891 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.895 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3e59cc:::benchmark_data_smithi194_186548_object34758:head# 2024-04-29T14:53:54.895 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.895 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3e7d3d:::benchmark_data_smithi194_186548_object2324:head# 2024-04-29T14:53:54.895 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.895 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3ec5dc:::benchmark_data_smithi194_186548_object74599:head# 2024-04-29T14:53:54.895 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.896 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3ed037:::benchmark_data_smithi194_186548_object64220:head# 2024-04-29T14:53:54.896 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3f0f08:::benchmark_data_smithi194_186548_object34244:head# 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3f929e:::benchmark_data_smithi194_186548_object51871:head# 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a3fa9bf:::benchmark_data_smithi194_186548_object28370:head# 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4168fe:::benchmark_data_smithi194_186548_object81644:head# 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a41ae91:::benchmark_data_smithi194_186548_object83767:head# 2024-04-29T14:53:54.897 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.898 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a42ea8a:::benchmark_data_smithi194_186548_object9389:head# 2024-04-29T14:53:54.898 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.898 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4404d1:::benchmark_data_smithi194_186548_object45125:head# 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a446b2f:::benchmark_data_smithi194_186548_object63586:head# 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a45c7a6:::benchmark_data_smithi194_186548_object21240:head# 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a47397c:::benchmark_data_smithi194_186548_object25494:head# 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a48224c:::benchmark_data_smithi194_186548_object8578:head# 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a489403:::benchmark_data_smithi194_186548_object185:head# 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4958fc:::benchmark_data_smithi194_186548_object40021:head# 2024-04-29T14:53:54.900 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.900 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4a0cb5:::benchmark_data_smithi194_186548_object37757:head# 2024-04-29T14:53:54.901 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.901 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4aa7c4:::benchmark_data_smithi194_186548_object81612:head# 2024-04-29T14:53:54.901 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.901 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4bb679:::benchmark_data_smithi194_186548_object65841:head# 2024-04-29T14:53:54.901 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.906 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4bb8f8:::benchmark_data_smithi194_186548_object6285:head# 2024-04-29T14:53:54.906 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.906 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4c089a:::benchmark_data_smithi194_186548_object52839:head# 2024-04-29T14:53:54.906 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.913 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4c4af3:::benchmark_data_smithi194_186548_object72817:head# 2024-04-29T14:53:54.913 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.914 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4d6688:::benchmark_data_smithi194_186548_object36919:head# 2024-04-29T14:53:54.914 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.914 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4e17f0:::benchmark_data_smithi194_186548_object71653:head# 2024-04-29T14:53:54.914 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.914 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4f1ab4:::benchmark_data_smithi194_186548_object25252:head# 2024-04-29T14:53:54.914 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.915 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a4fc811:::benchmark_data_smithi194_186548_object78204:head# 2024-04-29T14:53:54.915 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.915 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a53dc77:::benchmark_data_smithi194_186548_object83724:head# 2024-04-29T14:53:54.915 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.915 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a54b88d:::benchmark_data_smithi194_186548_object16175:head# 2024-04-29T14:53:54.915 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a54f2a1:::benchmark_data_smithi194_186548_object7087:head# 2024-04-29T14:53:54.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a55ad46:::benchmark_data_smithi194_186548_object66608:head# 2024-04-29T14:53:54.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a56381e:::benchmark_data_smithi194_186548_object38386:head# 2024-04-29T14:53:54.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.922 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a564b6f:::benchmark_data_smithi194_186548_object60752:head# 2024-04-29T14:53:54.922 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.923 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a56c3c2:::benchmark_data_smithi194_186548_object72074:head# 2024-04-29T14:53:54.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a571910:::benchmark_data_smithi194_186548_object62165:head# 2024-04-29T14:53:54.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a571fce:::benchmark_data_smithi194_186548_object59390:head# 2024-04-29T14:53:54.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a5728f8:::benchmark_data_smithi194_186548_object53404:head# 2024-04-29T14:53:54.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a57b3e5:::benchmark_data_smithi194_186548_object30532:head# 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a58ebff:::benchmark_data_smithi194_186548_object42485:head# 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a58f489:::benchmark_data_smithi194_186548_object68944:head# 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a593bd2:::benchmark_data_smithi194_186548_object27073:head# 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a59dc3d:::benchmark_data_smithi194_186548_object64886:head# 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a5ace62:::benchmark_data_smithi194_186548_object25610:head# 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a5b8e74:::benchmark_data_smithi194_186548_object411:head# 2024-04-29T14:53:54.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a5bf86b:::benchmark_data_smithi194_186548_object26132:head# 2024-04-29T14:53:54.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a5fc945:::benchmark_data_smithi194_186548_object22547:head# 2024-04-29T14:53:54.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.942 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a60a803:::benchmark_data_smithi194_186548_object67261:head# 2024-04-29T14:53:54.942 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a60c3dc:::benchmark_data_smithi194_186548_object50958:head# 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a60ee4f:::benchmark_data_smithi194_186548_object57231:head# 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a62532d:::benchmark_data_smithi194_186548_object6705:head# 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a63dac5:::benchmark_data_smithi194_186548_object75268:head# 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6411d2:::benchmark_data_smithi194_186548_object28846:head# 2024-04-29T14:53:54.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a645c57:::benchmark_data_smithi194_186548_object54661:head# 2024-04-29T14:53:54.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a650fb7:::benchmark_data_smithi194_186548_object37323:head# 2024-04-29T14:53:54.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6522dc:::benchmark_data_smithi194_186548_object56121:head# 2024-04-29T14:53:54.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a67c229:::benchmark_data_smithi194_186548_object25568:head# 2024-04-29T14:53:54.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6851a5:::benchmark_data_smithi194_186548_object48215:head# 2024-04-29T14:53:54.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a68b647:::benchmark_data_smithi194_186548_object32709:head# 2024-04-29T14:53:54.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6a1564:::benchmark_data_smithi194_186548_object41152:head# 2024-04-29T14:53:54.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6ad6e5:::benchmark_data_smithi194_186548_object37121:head# 2024-04-29T14:53:54.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6b463e:::benchmark_data_smithi194_186548_object1940:head# 2024-04-29T14:53:54.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6b8876:::benchmark_data_smithi194_186548_object79981:head# 2024-04-29T14:53:54.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6ced96:::benchmark_data_smithi194_186548_object15518:head# 2024-04-29T14:53:54.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a6d150e:::benchmark_data_smithi194_186548_object46653:head# 2024-04-29T14:53:54.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a700d1c:::benchmark_data_smithi194_186548_object31930:head# 2024-04-29T14:53:54.962 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7013c8:::benchmark_data_smithi194_186548_object33007:head# 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a708c35:::benchmark_data_smithi194_186548_object40174:head# 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a721c95:::benchmark_data_smithi194_186548_object29585:head# 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a73386f:::benchmark_data_smithi194_186548_object507:head# 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a75f04a:::benchmark_data_smithi194_186548_object744:head# 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7699b3:::benchmark_data_smithi194_186548_object56806:head# 2024-04-29T14:53:54.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a773aff:::benchmark_data_smithi194_186548_object17579:head# 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7754a2:::benchmark_data_smithi194_186548_object52977:head# 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a785b3b:::benchmark_data_smithi194_186548_object568:head# 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7b2c8a:::benchmark_data_smithi194_186548_object68780:head# 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7b98b1:::benchmark_data_smithi194_186548_object11835:head# 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7ba85e:::benchmark_data_smithi194_186548_object69336:head# 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.965 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7c33a3:::benchmark_data_smithi194_186548_object39241:head# 2024-04-29T14:53:54.966 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.966 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7c4ab0:::benchmark_data_smithi194_186548_object83737:head# 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7c9bd5:::benchmark_data_smithi194_186548_object69801:head# 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7d99ad:::benchmark_data_smithi194_186548_object75058:head# 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7ec590:::benchmark_data_smithi194_186548_object32108:head# 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a7f349b:::benchmark_data_smithi194_186548_object9025:head# 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a80b733:::benchmark_data_smithi194_186548_object21285:head# 2024-04-29T14:53:54.967 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8127a4:::benchmark_data_smithi194_186548_object59076:head# 2024-04-29T14:53:54.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a820db6:::benchmark_data_smithi194_186548_object28589:head# 2024-04-29T14:53:54.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.968 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a82584a:::benchmark_data_smithi194_186548_object63353:head# 2024-04-29T14:53:54.968 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8270ae:::benchmark_data_smithi194_186548_object25321:head# 2024-04-29T14:53:54.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.977 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a828538:::benchmark_data_smithi194_186548_object55669:head# 2024-04-29T14:53:54.977 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a884800:::benchmark_data_smithi194_186548_object58481:head# 2024-04-29T14:53:54.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8952af:::benchmark_data_smithi194_186548_object604:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a89cbf3:::benchmark_data_smithi194_186548_object3558:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8c07ab:::benchmark_data_smithi194_186548_object17612:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8c3231:::benchmark_data_smithi194_186548_object81358:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8c52fa:::benchmark_data_smithi194_186548_object21767:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8cedca:::benchmark_data_smithi194_186548_object1034:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8d2e92:::benchmark_data_smithi194_186548_object63688:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8d5702:::benchmark_data_smithi194_186548_object68642:head# 2024-04-29T14:53:54.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8db56c:::benchmark_data_smithi194_186548_object37615:head# 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a8e2e57:::benchmark_data_smithi194_186548_object5303:head# 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a90e187:::benchmark_data_smithi194_186548_object11828:head# 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a91340d:::benchmark_data_smithi194_186548_object57331:head# 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a920af5:::benchmark_data_smithi194_186548_object855:head# 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9216be:::benchmark_data_smithi194_186548_object54638:head# 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a92f006:::benchmark_data_smithi194_186548_object19109:head# 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a93c46c:::benchmark_data_smithi194_186548_object29279:head# 2024-04-29T14:53:54.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a940b5e:::benchmark_data_smithi194_186548_object56742:head# 2024-04-29T14:53:54.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a94a50d:::benchmark_data_smithi194_186548_object14123:head# 2024-04-29T14:53:54.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a960821:::benchmark_data_smithi194_186548_object66456:head# 2024-04-29T14:53:54.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a96a785:::benchmark_data_smithi194_186548_object72120:head# 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a96baf7:::benchmark_data_smithi194_186548_object38544:head# 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a97f184:::benchmark_data_smithi194_186548_object44151:head# 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9933a8:::benchmark_data_smithi194_186548_object11202:head# 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a99ff52:::benchmark_data_smithi194_186548_object57990:head# 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9ae5f2:::benchmark_data_smithi194_186548_object20040:head# 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9b8977:::benchmark_data_smithi194_186548_object12532:head# 2024-04-29T14:53:54.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9e75b6:::benchmark_data_smithi194_186548_object33623:head# 2024-04-29T14:53:54.997 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9e96ec:::benchmark_data_smithi194_186548_object49687:head# 2024-04-29T14:53:54.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:54.998 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9f3a19:::benchmark_data_smithi194_186548_object7737:head# 2024-04-29T14:53:54.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.004 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3a9ff2c4:::benchmark_data_smithi194_186548_object24769:head# 2024-04-29T14:53:55.005 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.005 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa150da:::benchmark_data_smithi194_186548_object21264:head# 2024-04-29T14:53:55.005 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.005 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa1c953:::benchmark_data_smithi194_186548_object7409:head# 2024-04-29T14:53:55.005 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.006 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa358fc:::benchmark_data_smithi194_186548_object56630:head# 2024-04-29T14:53:55.006 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.006 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa3f301:::benchmark_data_smithi194_186548_object33076:head# 2024-04-29T14:53:55.006 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.006 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa54f97:::benchmark_data_smithi194_186548_object82102:head# 2024-04-29T14:53:55.006 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.010 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa5eee5:::benchmark_data_smithi194_186548_object5204:head# 2024-04-29T14:53:55.010 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.015 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa6b251:::benchmark_data_smithi194_186548_object56849:head# 2024-04-29T14:53:55.015 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.015 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa70488:::benchmark_data_smithi194_186548_object20860:head# 2024-04-29T14:53:55.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa72a49:::benchmark_data_smithi194_186548_object58516:head# 2024-04-29T14:53:55.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.019 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:53:55.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3760084016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:55.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:54 smithi040 ceph-mon[39581]: pgmap v2633: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 117 B/s rd, 5.0 MiB/s wr, 990 op/s; 13393/240720 objects degraded (5.564%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:53:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:55.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:53:55.024 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa7feec:::benchmark_data_smithi194_186548_object27747:head# 2024-04-29T14:53:55.024 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.037 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa86232:::benchmark_data_smithi194_186548_object12182:head# 2024-04-29T14:53:55.037 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.042 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aa9003e:::benchmark_data_smithi194_186548_object56476:head# 2024-04-29T14:53:55.042 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.043 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aaa2273:::benchmark_data_smithi194_186548_object16969:head# 2024-04-29T14:53:55.043 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aaa5d90:::benchmark_data_smithi194_186548_object36512:head# 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aaae485:::benchmark_data_smithi194_186548_object14446:head# 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aac1b6d:::benchmark_data_smithi194_186548_object30554:head# 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aad36c0:::benchmark_data_smithi194_186548_object71374:head# 2024-04-29T14:53:55.044 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aadf44c:::benchmark_data_smithi194_186548_object30455:head# 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aae7b0c:::benchmark_data_smithi194_186548_object51315:head# 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aaf3594:::benchmark_data_smithi194_186548_object60157:head# 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab0e886:::benchmark_data_smithi194_186548_object43699:head# 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab30b48:::benchmark_data_smithi194_186548_object14025:head# 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.049 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab3ac63:::benchmark_data_smithi194_186548_object27562:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab4cb95:::benchmark_data_smithi194_186548_object78677:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab6faf4:::benchmark_data_smithi194_186548_object16996:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab74a07:::benchmark_data_smithi194_186548_object66338:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab7fba1:::benchmark_data_smithi194_186548_object43735:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ab95f99:::benchmark_data_smithi194_186548_object22611:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aba84a4:::benchmark_data_smithi194_186548_object30993:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abaccdf:::benchmark_data_smithi194_186548_object38239:head# 2024-04-29T14:53:55.050 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.051 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abb9b38:::benchmark_data_smithi194_186548_object45808:head# 2024-04-29T14:53:55.051 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abc1c2c:::benchmark_data_smithi194_186548_object34946:head# 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abc72bd:::benchmark_data_smithi194_186548_object26275:head# 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abc7451:::benchmark_data_smithi194_186548_object80244:head# 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abdd4e0:::benchmark_data_smithi194_186548_object62247:head# 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abf3f92:::benchmark_data_smithi194_186548_object85124:head# 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3abf74c9:::benchmark_data_smithi194_186548_object36643:head# 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac069dc:::benchmark_data_smithi194_186548_object4437:head# 2024-04-29T14:53:55.053 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.053 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac0d537:::benchmark_data_smithi194_186548_object54423:head# 2024-04-29T14:53:55.053 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.059 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac17741:::benchmark_data_smithi194_186548_object48707:head# 2024-04-29T14:53:55.060 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.067 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac2af89:::benchmark_data_smithi194_186548_object33205:head# 2024-04-29T14:53:55.067 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.067 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac355a3:::benchmark_data_smithi194_186548_object72732:head# 2024-04-29T14:53:55.067 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac482ab:::benchmark_data_smithi194_186548_object73156:head# 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac6c49c:::benchmark_data_smithi194_186548_object35453:head# 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac756f5:::benchmark_data_smithi194_186548_object42134:head# 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac7a5f8:::benchmark_data_smithi194_186548_object82321:head# 2024-04-29T14:53:55.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.069 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac8e77c:::benchmark_data_smithi194_186548_object18525:head# 2024-04-29T14:53:55.069 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.069 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ac944e2:::benchmark_data_smithi194_186548_object1457:head# 2024-04-29T14:53:55.069 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.069 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3accb109:::benchmark_data_smithi194_186548_object41292:head# 2024-04-29T14:53:55.069 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.070 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3acceeb7:::benchmark_data_smithi194_186548_object22749:head# 2024-04-29T14:53:55.070 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.070 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3acd0b3a:::benchmark_data_smithi194_186548_object47696:head# 2024-04-29T14:53:55.070 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.070 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3acd1e03:::benchmark_data_smithi194_186548_object8587:head# 2024-04-29T14:53:55.070 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.070 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ace5df3:::benchmark_data_smithi194_186548_object29242:head# 2024-04-29T14:53:55.071 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.071 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad1b399:::benchmark_data_smithi194_186548_object20891:head# 2024-04-29T14:53:55.071 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.071 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad200d0:::benchmark_data_smithi194_186548_object47842:head# 2024-04-29T14:53:55.071 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3760084016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:55.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:54 smithi194 ceph-mon[44089]: pgmap v2633: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 117 B/s rd, 5.0 MiB/s wr, 990 op/s; 13393/240720 objects degraded (5.564%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:53:55.078 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad2f3d7:::benchmark_data_smithi194_186548_object67855:head# 2024-04-29T14:53:55.078 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.078 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad5a1fa:::benchmark_data_smithi194_186548_object9993:head# 2024-04-29T14:53:55.078 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.089 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad5a487:::benchmark_data_smithi194_186548_object9646:head# 2024-04-29T14:53:55.090 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.097 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad5ef06:::benchmark_data_smithi194_186548_object77304:head# 2024-04-29T14:53:55.097 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.101 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad6b312:::benchmark_data_smithi194_186548_object32376:head# 2024-04-29T14:53:55.101 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.105 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad6f103:::benchmark_data_smithi194_186548_object41493:head# 2024-04-29T14:53:55.105 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.105 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad77957:::benchmark_data_smithi194_186548_object73080:head# 2024-04-29T14:53:55.105 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.105 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad97e4a:::benchmark_data_smithi194_186548_object14508:head# 2024-04-29T14:53:55.105 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.106 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ad9a4f9:::benchmark_data_smithi194_186548_object53733:head# 2024-04-29T14:53:55.107 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.107 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3adafa2b:::benchmark_data_smithi194_186548_object70087:head# 2024-04-29T14:53:55.107 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.111 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3adb5553:::benchmark_data_smithi194_186548_object3295:head# 2024-04-29T14:53:55.112 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:53:55.126 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3adc0075:::benchmark_data_smithi194_186548_object33371:head# 2024-04-29T14:53:55.127 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.128 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3adf1985:::benchmark_data_smithi194_186548_object28404:head# 2024-04-29T14:53:55.128 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.129 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3adf5e05:::benchmark_data_smithi194_186548_object69924:head# 2024-04-29T14:53:55.130 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.130 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae06bcc:::benchmark_data_smithi194_186548_object82957:head# 2024-04-29T14:53:55.130 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.132 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae16511:::benchmark_data_smithi194_186548_object2451:head# 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae2178e:::benchmark_data_smithi194_186548_object39387:head# 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae313d4:::benchmark_data_smithi194_186548_object20742:head# 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae32133:::benchmark_data_smithi194_186548_object47280:head# 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae458ef:::benchmark_data_smithi194_186548_object43248:head# 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae47fc6:::benchmark_data_smithi194_186548_object66972:head# 2024-04-29T14:53:55.133 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae55de6:::benchmark_data_smithi194_186548_object25208:head# 2024-04-29T14:53:55.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3760084016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:53:55.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:54 smithi121 ceph-mon[44533]: pgmap v2633: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 117 B/s rd, 5.0 MiB/s wr, 990 op/s; 13393/240720 objects degraded (5.564%); 13 MiB/s, 213 objects/s recovering 2024-04-29T14:53:55.150 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae5e11a:::benchmark_data_smithi194_186548_object10643:head# 2024-04-29T14:53:55.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.152 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:55.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae66018:::benchmark_data_smithi194_186548_object5741:head# 2024-04-29T14:53:55.156 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.165 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae901af:::benchmark_data_smithi194_186548_object24359:head# 2024-04-29T14:53:55.165 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.165 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae90800:::benchmark_data_smithi194_186548_object22017:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae96322:::benchmark_data_smithi194_186548_object68886:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae9897a:::benchmark_data_smithi194_186548_object34812:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ae9bf85:::benchmark_data_smithi194_186548_object7656:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aec5f9a:::benchmark_data_smithi194_186548_object49981:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aee672b:::benchmark_data_smithi194_186548_object66171:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3aeeaec0:::benchmark_data_smithi194_186548_object18908:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af0121c:::benchmark_data_smithi194_186548_object26837:head# 2024-04-29T14:53:55.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af152b1:::benchmark_data_smithi194_186548_object38753:head# 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af1a643:::benchmark_data_smithi194_186548_object61985:head# 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af2018d:::benchmark_data_smithi194_186548_object56877:head# 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af2e59a:::benchmark_data_smithi194_186548_object44557:head# 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af30b7f:::benchmark_data_smithi194_186548_object62910:head# 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af3e091:::benchmark_data_smithi194_186548_object61092:head# 2024-04-29T14:53:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af42210:::benchmark_data_smithi194_186548_object50418:head# 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af52f52:::benchmark_data_smithi194_186548_object46026:head# 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af6d385:::benchmark_data_smithi194_186548_object39080:head# 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af6ee83:::benchmark_data_smithi194_186548_object41222:head# 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3af74b1a:::benchmark_data_smithi194_186548_object7022:head# 2024-04-29T14:53:55.171 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.173 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3afa80f6:::benchmark_data_smithi194_186548_object83252:head# 2024-04-29T14:53:55.173 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3afa9c10:::benchmark_data_smithi194_186548_object183:head# 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3afc2b92:::benchmark_data_smithi194_186548_object74317:head# 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3afd16b6:::benchmark_data_smithi194_186548_object65764:head# 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3afe7cc2:::benchmark_data_smithi194_186548_object43342:head# 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b015b70:::benchmark_data_smithi194_186548_object21346:head# 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b029371:::benchmark_data_smithi194_186548_object26123:head# 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.177 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b02de22:::benchmark_data_smithi194_186548_object48753:head# 2024-04-29T14:53:55.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b03fdae:::benchmark_data_smithi194_186548_object6345:head# 2024-04-29T14:53:55.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.178 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b053a5f:::benchmark_data_smithi194_186548_object34085:head# 2024-04-29T14:53:55.178 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.183 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b05bc0a:::benchmark_data_smithi194_186548_object51280:head# 2024-04-29T14:53:55.183 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.197 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b06d272:::benchmark_data_smithi194_186548_object32268:head# 2024-04-29T14:53:55.197 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.197 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b07eb65:::benchmark_data_smithi194_186548_object82066:head# 2024-04-29T14:53:55.197 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.199 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b08521d:::benchmark_data_smithi194_186548_object60967:head# 2024-04-29T14:53:55.199 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0a3561:::benchmark_data_smithi194_186548_object29539:head# 2024-04-29T14:53:55.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.214 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0a3fb2:::benchmark_data_smithi194_186548_object71860:head# 2024-04-29T14:53:55.214 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0a4751:::benchmark_data_smithi194_186548_object8015:head# 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0ae2e7:::benchmark_data_smithi194_186548_object43877:head# 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0b780b:::benchmark_data_smithi194_186548_object42381:head# 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0d61c6:::benchmark_data_smithi194_186548_object41315:head# 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0e5d0a:::benchmark_data_smithi194_186548_object8697:head# 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b0e7f4c:::benchmark_data_smithi194_186548_object31445:head# 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1070f4:::benchmark_data_smithi194_186548_object9532:head# 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.215 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b10ee5a:::benchmark_data_smithi194_186548_object6627:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b117ade:::benchmark_data_smithi194_186548_object16384:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1227f0:::benchmark_data_smithi194_186548_object14692:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1257c2:::benchmark_data_smithi194_186548_object21637:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b125be0:::benchmark_data_smithi194_186548_object70571:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b129e06:::benchmark_data_smithi194_186548_object64271:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b12e08e:::benchmark_data_smithi194_186548_object59035:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b13b2da:::benchmark_data_smithi194_186548_object46583:head# 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.216 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b14416f:::benchmark_data_smithi194_186548_object28716:head# 2024-04-29T14:53:55.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.217 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b146fea:::benchmark_data_smithi194_186548_object57713:head# 2024-04-29T14:53:55.217 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.223 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b151765:::benchmark_data_smithi194_186548_object64097:head# 2024-04-29T14:53:55.223 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.227 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b19d5da:::benchmark_data_smithi194_186548_object40531:head# 2024-04-29T14:53:55.227 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.227 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b19df4c:::benchmark_data_smithi194_186548_object42879:head# 2024-04-29T14:53:55.227 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.228 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1a7ef9:::benchmark_data_smithi194_186548_object78266:head# 2024-04-29T14:53:55.228 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.228 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1c0b39:::benchmark_data_smithi194_186548_object6422:head# 2024-04-29T14:53:55.228 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.229 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1c73dc:::benchmark_data_smithi194_186548_object34254:head# 2024-04-29T14:53:55.229 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.229 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1d5644:::benchmark_data_smithi194_186548_object21395:head# 2024-04-29T14:53:55.229 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1e0d8d:::benchmark_data_smithi194_186548_object18777:head# 2024-04-29T14:53:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b1fbf6b:::benchmark_data_smithi194_186548_object79629:head# 2024-04-29T14:53:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b225254:::benchmark_data_smithi194_186548_object51341:head# 2024-04-29T14:53:55.233 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.235 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b225955:::benchmark_data_smithi194_186548_object71161:head# 2024-04-29T14:53:55.235 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.235 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b22f5a7:::benchmark_data_smithi194_186548_object54097:head# 2024-04-29T14:53:55.235 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.235 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b231dc5:::benchmark_data_smithi194_186548_object34762:head# 2024-04-29T14:53:55.235 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b244d4c:::benchmark_data_smithi194_186548_object45901:head# 2024-04-29T14:53:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2461d4:::benchmark_data_smithi194_186548_object56219:head# 2024-04-29T14:53:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b249b37:::benchmark_data_smithi194_186548_object30051:head# 2024-04-29T14:53:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.236 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b25dd27:::benchmark_data_smithi194_186548_object15889:head# 2024-04-29T14:53:55.237 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.237 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b25f991:::benchmark_data_smithi194_186548_object19044:head# 2024-04-29T14:53:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b266b3b:::benchmark_data_smithi194_186548_object61747:head# 2024-04-29T14:53:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b268c66:::benchmark_data_smithi194_186548_object30813:head# 2024-04-29T14:53:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b272461:::benchmark_data_smithi194_186548_object79763:head# 2024-04-29T14:53:55.238 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.243 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b274ad8:::benchmark_data_smithi194_186548_object5933:head# 2024-04-29T14:53:55.243 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.245 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b27995e:::benchmark_data_smithi194_186548_object8368:head# 2024-04-29T14:53:55.245 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.249 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b281e26:::benchmark_data_smithi194_186548_object11474:head# 2024-04-29T14:53:55.250 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.259 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:53:55.259 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b285ad2:::benchmark_data_smithi194_186548_object77993:head# 2024-04-29T14:53:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b29507d:::benchmark_data_smithi194_186548_object60205:head# 2024-04-29T14:53:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2a2ddb:::benchmark_data_smithi194_186548_object75490:head# 2024-04-29T14:53:55.260 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.261 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2a4ab5:::benchmark_data_smithi194_186548_object23873:head# 2024-04-29T14:53:55.262 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:53:55.265 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2a6358:::benchmark_data_smithi194_186548_object6324:head# 2024-04-29T14:53:55.266 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.266 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2a8445:::benchmark_data_smithi194_186548_object46650:head# 2024-04-29T14:53:55.266 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.266 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2ba408:::benchmark_data_smithi194_186548_object36620:head# 2024-04-29T14:53:55.267 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.267 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2ba76b:::benchmark_data_smithi194_186548_object77729:head# 2024-04-29T14:53:55.267 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.267 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2c6d32:::benchmark_data_smithi194_186548_object83600:head# 2024-04-29T14:53:55.268 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.268 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2d1dda:::benchmark_data_smithi194_186548_object75515:head# 2024-04-29T14:53:55.268 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.268 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b2e52a6:::benchmark_data_smithi194_186548_object75932:head# 2024-04-29T14:53:55.268 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:55.269 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b316160:::benchmark_data_smithi194_186548_object26282:head# 2024-04-29T14:53:55.269 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.270 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b31dae6:::benchmark_data_smithi194_186548_object39981:head# 2024-04-29T14:53:55.270 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.270 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b327a9d:::benchmark_data_smithi194_186548_object45779:head# 2024-04-29T14:53:55.270 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:53:55.275 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b330a20:::benchmark_data_smithi194_186548_object6949:head# 2024-04-29T14:53:55.275 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.275 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b346481:::benchmark_data_smithi194_186548_object39431:head# 2024-04-29T14:53:55.275 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b381f60:::benchmark_data_smithi194_186548_object82574:head# 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b39a055:::benchmark_data_smithi194_186548_object48936:head# 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3b795c:::benchmark_data_smithi194_186548_object66804:head# 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3cb301:::benchmark_data_smithi194_186548_object53193:head# 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3d5815:::benchmark_data_smithi194_186548_object72659:head# 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3e0813:::benchmark_data_smithi194_186548_object8054:head# 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.276 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3e0b78:::benchmark_data_smithi194_186548_object43193:head# 2024-04-29T14:53:55.277 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.277 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3ecb88:::benchmark_data_smithi194_186548_object50432:head# 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3ef3a6:::benchmark_data_smithi194_186548_object11281:head# 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3f0140:::benchmark_data_smithi194_186548_object47420:head# 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b3f330b:::benchmark_data_smithi194_186548_object56305:head# 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b410e6e:::benchmark_data_smithi194_186548_object53911:head# 2024-04-29T14:53:55.278 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.281 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b416a46:::benchmark_data_smithi194_186548_object16180:head# 2024-04-29T14:53:55.281 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b432979:::benchmark_data_smithi194_186548_object34322:head# 2024-04-29T14:53:55.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b436efe:::benchmark_data_smithi194_186548_object19066:head# 2024-04-29T14:53:55.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b444848:::benchmark_data_smithi194_186548_object18837:head# 2024-04-29T14:53:55.287 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.287 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b449c0b:::benchmark_data_smithi194_186548_object26620:head# 2024-04-29T14:53:55.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b46a1d0:::benchmark_data_smithi194_186548_object3595:head# 2024-04-29T14:53:55.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b46b6df:::benchmark_data_smithi194_186548_object40783:head# 2024-04-29T14:53:55.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.288 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b48c7ba:::benchmark_data_smithi194_186548_object35277:head# 2024-04-29T14:53:55.288 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b49a7d7:::benchmark_data_smithi194_186548_object28488:head# 2024-04-29T14:53:55.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4a16e1:::benchmark_data_smithi194_186548_object83393:head# 2024-04-29T14:53:55.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4a30fa:::benchmark_data_smithi194_186548_object63662:head# 2024-04-29T14:53:55.291 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.291 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4a6127:::benchmark_data_smithi194_186548_object31981:head# 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4b0f45:::benchmark_data_smithi194_186548_object56987:head# 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4bfe0b:::benchmark_data_smithi194_186548_object29293:head# 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4c65af:::benchmark_data_smithi194_186548_object62370:head# 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4ccfa4:::benchmark_data_smithi194_186548_object24395:head# 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4cd86e:::benchmark_data_smithi194_186548_object38493:head# 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4d528b:::benchmark_data_smithi194_186548_object84826:head# 2024-04-29T14:53:55.292 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4d6ba5:::benchmark_data_smithi194_186548_object5050:head# 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4dd17d:::benchmark_data_smithi194_186548_object15903:head# 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4df418:::benchmark_data_smithi194_186548_object22022:head# 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4e1560:::benchmark_data_smithi194_186548_object32822:head# 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b4ea4b0:::benchmark_data_smithi194_186548_object39457:head# 2024-04-29T14:53:55.293 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b519a93:::benchmark_data_smithi194_186548_object51642:head# 2024-04-29T14:53:55.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.294 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b522121:::benchmark_data_smithi194_186548_object15472:head# 2024-04-29T14:53:55.294 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.298 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b522d0e:::benchmark_data_smithi194_186548_object24962:head# 2024-04-29T14:53:55.298 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.309 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b532b7c:::benchmark_data_smithi194_186548_object4308:head# 2024-04-29T14:53:55.310 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.316 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b53a96f:::benchmark_data_smithi194_186548_object78237:head# 2024-04-29T14:53:55.316 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.316 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b53add1:::benchmark_data_smithi194_186548_object71200:head# 2024-04-29T14:53:55.316 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.317 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b53b07c:::benchmark_data_smithi194_186548_object33369:head# 2024-04-29T14:53:55.318 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.325 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b546ce8:::benchmark_data_smithi194_186548_object44618:head# 2024-04-29T14:53:55.325 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.328 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b54f88a:::benchmark_data_smithi194_186548_object46414:head# 2024-04-29T14:53:55.328 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.329 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b553ec3:::benchmark_data_smithi194_186548_object50991:head# 2024-04-29T14:53:55.330 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.330 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b560684:::benchmark_data_smithi194_186548_object37872:head# 2024-04-29T14:53:55.330 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.341 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5677fb:::benchmark_data_smithi194_186548_object58725:head# 2024-04-29T14:53:55.341 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.342 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b56e235:::benchmark_data_smithi194_186548_object51251:head# 2024-04-29T14:53:55.342 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b570b81:::benchmark_data_smithi194_186548_object72523:head# 2024-04-29T14:53:55.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.345 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b57e25d:::benchmark_data_smithi194_186548_object11491:head# 2024-04-29T14:53:55.345 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b587337:::benchmark_data_smithi194_186548_object70893:head# 2024-04-29T14:53:55.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b58b93b:::benchmark_data_smithi194_186548_object25597:head# 2024-04-29T14:53:55.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b591cbc:::benchmark_data_smithi194_186548_object19398:head# 2024-04-29T14:53:55.346 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.346 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b59a997:::benchmark_data_smithi194_186548_object56432:head# 2024-04-29T14:53:55.347 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.347 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5a64e5:::benchmark_data_smithi194_186548_object20093:head# 2024-04-29T14:53:55.347 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5a8c79:::benchmark_data_smithi194_186548_object16800:head# 2024-04-29T14:53:55.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5b0413:::benchmark_data_smithi194_186548_object60583:head# 2024-04-29T14:53:55.351 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.351 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5b58ba:::benchmark_data_smithi194_186548_object29236:head# 2024-04-29T14:53:55.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.352 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5bacf9:::benchmark_data_smithi194_186548_object7225:head# 2024-04-29T14:53:55.352 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.358 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5d1ea8:::benchmark_data_smithi194_186548_object51030:head# 2024-04-29T14:53:55.358 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5d93c8:::benchmark_data_smithi194_186548_object68337:head# 2024-04-29T14:53:55.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.362 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5e5b72:::benchmark_data_smithi194_186548_object27110:head# 2024-04-29T14:53:55.362 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5f5c55:::benchmark_data_smithi194_186548_object36272:head# 2024-04-29T14:53:55.366 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.366 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5fcb51:::benchmark_data_smithi194_186548_object84524:head# 2024-04-29T14:53:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b5feeaf:::benchmark_data_smithi194_186548_object56279:head# 2024-04-29T14:53:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b629389:::benchmark_data_smithi194_186548_object31879:head# 2024-04-29T14:53:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b62b34f:::benchmark_data_smithi194_186548_object48441:head# 2024-04-29T14:53:55.367 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b633233:::benchmark_data_smithi194_186548_object82573:head# 2024-04-29T14:53:55.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b633920:::benchmark_data_smithi194_186548_object8785:head# 2024-04-29T14:53:55.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.369 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b63c28f:::benchmark_data_smithi194_186548_object57490:head# 2024-04-29T14:53:55.369 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b644286:::benchmark_data_smithi194_186548_object12530:head# 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b64b350:::benchmark_data_smithi194_186548_object16210:head# 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b65c386:::benchmark_data_smithi194_186548_object34304:head# 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b663894:::benchmark_data_smithi194_186548_object59837:head# 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b66f152:::benchmark_data_smithi194_186548_object13658:head# 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.370 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b67b2e9:::benchmark_data_smithi194_186548_object62643:head# 2024-04-29T14:53:55.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b681183:::benchmark_data_smithi194_186548_object32504:head# 2024-04-29T14:53:55.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.371 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6928cd:::benchmark_data_smithi194_186548_object25812:head# 2024-04-29T14:53:55.371 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:55.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b69767c:::benchmark_data_smithi194_186548_object49073:head# 2024-04-29T14:53:55.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b69a29e:::benchmark_data_smithi194_186548_object56124:head# 2024-04-29T14:53:55.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.374 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6a8d67:::benchmark_data_smithi194_186548_object73375:head# 2024-04-29T14:53:55.374 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6d0f5e:::benchmark_data_smithi194_186548_object25417:head# 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6f0d07:::benchmark_data_smithi194_186548_object46293:head# 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6f0e7b:::benchmark_data_smithi194_186548_object51443:head# 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6f7bc1:::benchmark_data_smithi194_186548_object1959:head# 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6f9eb3:::benchmark_data_smithi194_186548_object53667:head# 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6fe1cf:::benchmark_data_smithi194_186548_object71261:head# 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b6fe569:::benchmark_data_smithi194_186548_object41274:head# 2024-04-29T14:53:55.375 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b707c9c:::benchmark_data_smithi194_186548_object81389:head# 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b70aa34:::benchmark_data_smithi194_186548_object69174:head# 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b716ddc:::benchmark_data_smithi194_186548_object83820:head# 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b717f4c:::benchmark_data_smithi194_186548_object31181:head# 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b727665:::benchmark_data_smithi194_186548_object8679:head# 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b72796e:::benchmark_data_smithi194_186548_object83127:head# 2024-04-29T14:53:55.376 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.377 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b72f6b4:::benchmark_data_smithi194_186548_object46883:head# 2024-04-29T14:53:55.377 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.378 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7309dc:::benchmark_data_smithi194_186548_object40185:head# 2024-04-29T14:53:55.378 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.379 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b752ccb:::benchmark_data_smithi194_186548_object32855:head# 2024-04-29T14:53:55.379 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.380 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b75ee19:::benchmark_data_smithi194_186548_object32173:head# 2024-04-29T14:53:55.380 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.380 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b76e27f:::benchmark_data_smithi194_186548_object40223:head# 2024-04-29T14:53:55.380 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.380 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b77cce8:::benchmark_data_smithi194_186548_object20648:head# 2024-04-29T14:53:55.381 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.381 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7953a7:::benchmark_data_smithi194_186548_object55712:head# 2024-04-29T14:53:55.382 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.382 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7a1557:::benchmark_data_smithi194_186548_object51469:head# 2024-04-29T14:53:55.382 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.382 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7b844e:::benchmark_data_smithi194_186548_object35159:head# 2024-04-29T14:53:55.383 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.383 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7c4ef3:::benchmark_data_smithi194_186548_object43869:head# 2024-04-29T14:53:55.383 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.384 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7cb9d8:::benchmark_data_smithi194_186548_object8631:head# 2024-04-29T14:53:55.384 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.384 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7d10ac:::benchmark_data_smithi194_186548_object54201:head# 2024-04-29T14:53:55.384 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.384 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7d9e7d:::benchmark_data_smithi194_186548_object55030:head# 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7dcd4f:::benchmark_data_smithi194_186548_object12500:head# 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b7ecca4:::benchmark_data_smithi194_186548_object6618:head# 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b802a03:::benchmark_data_smithi194_186548_object64901:head# 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b803f35:::benchmark_data_smithi194_186548_object45204:head# 2024-04-29T14:53:55.385 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.386 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b80e4f3:::benchmark_data_smithi194_186548_object18938:head# 2024-04-29T14:53:55.386 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.386 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b80fd1b:::benchmark_data_smithi194_186548_object62933:head# 2024-04-29T14:53:55.386 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.386 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b8255a3:::benchmark_data_smithi194_186548_object59620:head# 2024-04-29T14:53:55.386 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.387 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b82b5f4:::benchmark_data_smithi194_186548_object44536:head# 2024-04-29T14:53:55.387 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.387 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b82d17a:::benchmark_data_smithi194_186548_object4225:head# 2024-04-29T14:53:55.388 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b834760:::benchmark_data_smithi194_186548_object73876:head# 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b8372df:::benchmark_data_smithi194_186548_object11981:head# 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b847189:::benchmark_data_smithi194_186548_object57725:head# 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b84d88a:::benchmark_data_smithi194_186548_object73054:head# 2024-04-29T14:53:55.393 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b84f097:::benchmark_data_smithi194_186548_object36450:head# 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b85a206:::benchmark_data_smithi194_186548_object3039:head# 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b85eaec:::benchmark_data_smithi194_186548_object78658:head# 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b86bbfb:::benchmark_data_smithi194_186548_object4435:head# 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b88edc6:::benchmark_data_smithi194_186548_object53515:head# 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.394 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b88f85a:::benchmark_data_smithi194_186548_object47747:head# 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b8a7b73:::benchmark_data_smithi194_186548_object69043:head# 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b8acea4:::benchmark_data_smithi194_186548_object10317:head# 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b8de23e:::benchmark_data_smithi194_186548_object67019:head# 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b8ecefd:::benchmark_data_smithi194_186548_object6483:head# 2024-04-29T14:53:55.395 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.397 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b8f6637:::benchmark_data_smithi194_186548_object68436:head# 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b91002f:::benchmark_data_smithi194_186548_object50223:head# 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b914d1c:::benchmark_data_smithi194_186548_object76557:head# 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b91dd0c:::benchmark_data_smithi194_186548_object46888:head# 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b91ffd5:::benchmark_data_smithi194_186548_object11412:head# 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b938cc2:::benchmark_data_smithi194_186548_object8418:head# 2024-04-29T14:53:55.398 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b93c6a6:::benchmark_data_smithi194_186548_object31520:head# 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b943076:::benchmark_data_smithi194_186548_object39844:head# 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b957469:::benchmark_data_smithi194_186548_object61625:head# 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b958c65:::benchmark_data_smithi194_186548_object50574:head# 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b963454:::benchmark_data_smithi194_186548_object37179:head# 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b98e395:::benchmark_data_smithi194_186548_object70231:head# 2024-04-29T14:53:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b99bd5d:::benchmark_data_smithi194_186548_object40168:head# 2024-04-29T14:53:55.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.400 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b9ba2be:::benchmark_data_smithi194_186548_object84926:head# 2024-04-29T14:53:55.400 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.406 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b9c0099:::benchmark_data_smithi194_186548_object22552:head# 2024-04-29T14:53:55.406 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b9c899e:::benchmark_data_smithi194_186548_object40924:head# 2024-04-29T14:53:55.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.407 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b9e657d:::benchmark_data_smithi194_186548_object53015:head# 2024-04-29T14:53:55.407 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b9f23eb:::benchmark_data_smithi194_186548_object77136:head# 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3b9fecb8:::benchmark_data_smithi194_186548_object59262:head# 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba11549:::benchmark_data_smithi194_186548_object15029:head# 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba1e3fa:::benchmark_data_smithi194_186548_object74043:head# 2024-04-29T14:53:55.408 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba208de:::benchmark_data_smithi194_186548_object76265:head# 2024-04-29T14:53:55.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.409 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba2ed65:::benchmark_data_smithi194_186548_object82531:head# 2024-04-29T14:53:55.409 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba39c40:::benchmark_data_smithi194_186548_object69545:head# 2024-04-29T14:53:55.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba40c81:::benchmark_data_smithi194_186548_object39811:head# 2024-04-29T14:53:55.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.410 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba46fc6:::benchmark_data_smithi194_186548_object33862:head# 2024-04-29T14:53:55.410 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba47a66:::benchmark_data_smithi194_186548_object76120:head# 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba5bb72:::benchmark_data_smithi194_186548_object73000:head# 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba640bc:::benchmark_data_smithi194_186548_object69106:head# 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba6dbb5:::benchmark_data_smithi194_186548_object54012:head# 2024-04-29T14:53:55.411 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba88e98:::benchmark_data_smithi194_186548_object40905:head# 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba8cce4:::benchmark_data_smithi194_186548_object44961:head# 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ba918f9:::benchmark_data_smithi194_186548_object9356:head# 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bab66d1:::benchmark_data_smithi194_186548_object34032:head# 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3babd9c9:::benchmark_data_smithi194_186548_object73302:head# 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bacaa96:::benchmark_data_smithi194_186548_object8046:head# 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.415 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bad008d:::benchmark_data_smithi194_186548_object62470:head# 2024-04-29T14:53:55.416 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.416 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bae7731:::benchmark_data_smithi194_186548_object13643:head# 2024-04-29T14:53:55.416 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.416 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb0d472:::benchmark_data_smithi194_186548_object32348:head# 2024-04-29T14:53:55.416 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.416 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb17247:::benchmark_data_smithi194_186548_object59183:head# 2024-04-29T14:53:55.417 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.417 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb1912e:::benchmark_data_smithi194_186548_object1075:head# 2024-04-29T14:53:55.417 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.417 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb1adec:::benchmark_data_smithi194_186548_object63284:head# 2024-04-29T14:53:55.417 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.417 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb1b6de:::benchmark_data_smithi194_186548_object65531:head# 2024-04-29T14:53:55.417 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb1d90c:::benchmark_data_smithi194_186548_object31812:head# 2024-04-29T14:53:55.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb2d51a:::benchmark_data_smithi194_186548_object12873:head# 2024-04-29T14:53:55.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb333a8:::benchmark_data_smithi194_186548_object10085:head# 2024-04-29T14:53:55.422 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.422 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb3782f:::benchmark_data_smithi194_186548_object8101:head# 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb3b70c:::benchmark_data_smithi194_186548_object9858:head# 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb4d677:::benchmark_data_smithi194_186548_object23055:head# 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb5e604:::benchmark_data_smithi194_186548_object61291:head# 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb602a5:::benchmark_data_smithi194_186548_object47803:head# 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb68cf3:::benchmark_data_smithi194_186548_object3684:head# 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb82b03:::benchmark_data_smithi194_186548_object26182:head# 2024-04-29T14:53:55.423 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bb8e6fc:::benchmark_data_smithi194_186548_object15570:head# 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bbaac34:::benchmark_data_smithi194_186548_object9306:head# 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bbc0190:::benchmark_data_smithi194_186548_object19969:head# 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bbd4678:::benchmark_data_smithi194_186548_object23943:head# 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bbde058:::benchmark_data_smithi194_186548_object73419:head# 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc04490:::benchmark_data_smithi194_186548_object66142:head# 2024-04-29T14:53:55.424 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.425 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc0a785:::benchmark_data_smithi194_186548_object50434:head# 2024-04-29T14:53:55.425 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.425 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc0c883:::benchmark_data_smithi194_186548_object27048:head# 2024-04-29T14:53:55.425 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.425 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc1fe3a:::benchmark_data_smithi194_186548_object50386:head# 2024-04-29T14:53:55.425 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.426 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc2d0f9:::benchmark_data_smithi194_186548_object2528:head# 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc3bdad:::benchmark_data_smithi194_186548_object7783:head# 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc518a0:::benchmark_data_smithi194_186548_object75061:head# 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc5aed3:::benchmark_data_smithi194_186548_object6678:head# 2024-04-29T14:53:55.427 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc803ed:::benchmark_data_smithi194_186548_object44223:head# 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc870eb:::benchmark_data_smithi194_186548_object63869:head# 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc8b5f4:::benchmark_data_smithi194_186548_object32862:head# 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc90386:::benchmark_data_smithi194_186548_object67008:head# 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bc9b3d9:::benchmark_data_smithi194_186548_object49571:head# 2024-04-29T14:53:55.428 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.432 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bcad741:::benchmark_data_smithi194_186548_object63182:head# 2024-04-29T14:53:55.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.432 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bcb8baf:::benchmark_data_smithi194_186548_object51692:head# 2024-04-29T14:53:55.432 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.439 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bcc1ec8:::benchmark_data_smithi194_186548_object15515:head# 2024-04-29T14:53:55.440 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.440 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bcc58ae:::benchmark_data_smithi194_186548_object44701:head# 2024-04-29T14:53:55.440 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.440 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bccc6bb:::benchmark_data_smithi194_186548_object22205:head# 2024-04-29T14:53:55.440 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.441 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bce83d1:::benchmark_data_smithi194_186548_object82179:head# 2024-04-29T14:53:55.441 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.441 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd0e0c4:::benchmark_data_smithi194_186548_object12154:head# 2024-04-29T14:53:55.441 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.441 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd2a222:::benchmark_data_smithi194_186548_object277:head# 2024-04-29T14:53:55.441 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.447 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd3899e:::benchmark_data_smithi194_186548_object42443:head# 2024-04-29T14:53:55.447 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.449 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd3cbf2:::benchmark_data_smithi194_186548_object17275:head# 2024-04-29T14:53:55.450 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.452 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd40f0d:::benchmark_data_smithi194_186548_object56400:head# 2024-04-29T14:53:55.453 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd4cfc2:::benchmark_data_smithi194_186548_object60932:head# 2024-04-29T14:53:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.458 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd5c792:::benchmark_data_smithi194_186548_object34881:head# 2024-04-29T14:53:55.458 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd666fa:::benchmark_data_smithi194_186548_object26423:head# 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd76fbc:::benchmark_data_smithi194_186548_object78933:head# 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd77ebf:::benchmark_data_smithi194_186548_object32522:head# 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd7c730:::benchmark_data_smithi194_186548_object30938:head# 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.460 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd7da68:::benchmark_data_smithi194_186548_object60803:head# 2024-04-29T14:53:55.461 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.461 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bd869ee:::benchmark_data_smithi194_186548_object29534:head# 2024-04-29T14:53:55.461 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.461 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bdac5b6:::benchmark_data_smithi194_186548_object13806:head# 2024-04-29T14:53:55.461 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.466 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bdbd991:::benchmark_data_smithi194_186548_object29430:head# 2024-04-29T14:53:55.466 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.466 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bde6437:::benchmark_data_smithi194_186548_object39831:head# 2024-04-29T14:53:55.466 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.466 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bdfa6db:::benchmark_data_smithi194_186548_object39723:head# 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bdfd7f5:::benchmark_data_smithi194_186548_object18699:head# 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be0b7db:::benchmark_data_smithi194_186548_object60216:head# 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be18a43:::benchmark_data_smithi194_186548_object12433:head# 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be1b0ee:::benchmark_data_smithi194_186548_object58215:head# 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be20071:::benchmark_data_smithi194_186548_object50547:head# 2024-04-29T14:53:55.467 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be2d57f:::benchmark_data_smithi194_186548_object78539:head# 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be44b46:::benchmark_data_smithi194_186548_object60063:head# 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be4704b:::benchmark_data_smithi194_186548_object60218:head# 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be4fd2b:::benchmark_data_smithi194_186548_object16914:head# 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be551b7:::benchmark_data_smithi194_186548_object22808:head# 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be791c6:::benchmark_data_smithi194_186548_object62930:head# 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.468 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be7f3f0:::benchmark_data_smithi194_186548_object84249:head# 2024-04-29T14:53:55.469 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.469 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be811f5:::benchmark_data_smithi194_186548_object7136:head# 2024-04-29T14:53:55.469 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.469 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be9cb1c:::benchmark_data_smithi194_186548_object39263:head# 2024-04-29T14:53:55.469 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.469 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3be9f7ac:::benchmark_data_smithi194_186548_object33502:head# 2024-04-29T14:53:55.469 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3beb07e2:::benchmark_data_smithi194_186548_object44185:head# 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bec35db:::benchmark_data_smithi194_186548_object80687:head# 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bee28be:::benchmark_data_smithi194_186548_object31113:head# 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bee6f52:::benchmark_data_smithi194_186548_object59553:head# 2024-04-29T14:53:55.470 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.473 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bef88fd:::benchmark_data_smithi194_186548_object15257:head# 2024-04-29T14:53:55.473 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.473 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf0ea78:::benchmark_data_smithi194_186548_object32502:head# 2024-04-29T14:53:55.473 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.475 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf0f408:::benchmark_data_smithi194_186548_object34925:head# 2024-04-29T14:53:55.475 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.475 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf1447f:::benchmark_data_smithi194_186548_object19762:head# 2024-04-29T14:53:55.475 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.475 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf18b57:::benchmark_data_smithi194_186548_object57311:head# 2024-04-29T14:53:55.475 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.475 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf20186:::benchmark_data_smithi194_186548_object21788:head# 2024-04-29T14:53:55.476 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.476 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf21447:::benchmark_data_smithi194_186548_object81376:head# 2024-04-29T14:53:55.476 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.477 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf256f9:::benchmark_data_smithi194_186548_object55946:head# 2024-04-29T14:53:55.477 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.485 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf391dd:::benchmark_data_smithi194_186548_object69609:head# 2024-04-29T14:53:55.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.486 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf407b8:::benchmark_data_smithi194_186548_object26512:head# 2024-04-29T14:53:55.486 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.488 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf453ea:::benchmark_data_smithi194_186548_object23507:head# 2024-04-29T14:53:55.488 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf4b696:::benchmark_data_smithi194_186548_object21896:head# 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf4b9b7:::benchmark_data_smithi194_186548_object55525:head# 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf5fe02:::benchmark_data_smithi194_186548_object40886:head# 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf6a5d8:::benchmark_data_smithi194_186548_object35182:head# 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf7406a:::benchmark_data_smithi194_186548_object28111:head# 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf741ed:::benchmark_data_smithi194_186548_object61767:head# 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.491 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf79df5:::benchmark_data_smithi194_186548_object80892:head# 2024-04-29T14:53:55.492 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf90516:::benchmark_data_smithi194_186548_object67021:head# 2024-04-29T14:53:55.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bf99b7b:::benchmark_data_smithi194_186548_object73630:head# 2024-04-29T14:53:55.493 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.493 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bfc737d:::benchmark_data_smithi194_186548_object64437:head# 2024-04-29T14:53:55.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.494 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bfc8274:::benchmark_data_smithi194_186548_object14804:head# 2024-04-29T14:53:55.494 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bff9412:::benchmark_data_smithi194_186548_object64098:head# 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bff9526:::benchmark_data_smithi194_186548_object54720:head# 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bffc895:::benchmark_data_smithi194_186548_object2821:head# 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3bffdaf8:::benchmark_data_smithi194_186548_object13634:head# 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.495 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c006e01:::benchmark_data_smithi194_186548_object73278:head# 2024-04-29T14:53:55.496 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.496 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0099ef:::benchmark_data_smithi194_186548_object26946:head# 2024-04-29T14:53:55.496 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.496 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c01eeb3:::benchmark_data_smithi194_186548_object62192:head# 2024-04-29T14:53:55.497 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.497 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c020858:::benchmark_data_smithi194_186548_object30848:head# 2024-04-29T14:53:55.497 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.497 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c04553c:::benchmark_data_smithi194_186548_object595:head# 2024-04-29T14:53:55.497 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.497 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c045dbc:::benchmark_data_smithi194_186548_object62545:head# 2024-04-29T14:53:55.498 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.498 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c051945:::benchmark_data_smithi194_186548_object46028:head# 2024-04-29T14:53:55.499 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.499 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c072f5c:::benchmark_data_smithi194_186548_object40341:head# 2024-04-29T14:53:55.499 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.499 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c074d7f:::benchmark_data_smithi194_186548_object78363:head# 2024-04-29T14:53:55.499 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.501 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c088774:::benchmark_data_smithi194_186548_object5851:head# 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c08dd35:::benchmark_data_smithi194_186548_object20423:head# 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0a50c7:::benchmark_data_smithi194_186548_object10724:head# 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0ad635:::benchmark_data_smithi194_186548_object41940:head# 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0b0208:::benchmark_data_smithi194_186548_object74441:head# 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0be8a5:::benchmark_data_smithi194_186548_object61451:head# 2024-04-29T14:53:55.502 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.506 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0d52ee:::benchmark_data_smithi194_186548_object35373:head# 2024-04-29T14:53:55.506 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.506 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0d86e3:::benchmark_data_smithi194_186548_object39515:head# 2024-04-29T14:53:55.506 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.511 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c0e73ef:::benchmark_data_smithi194_186548_object2156:head# 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c103337:::benchmark_data_smithi194_186548_object8471:head# 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1094cb:::benchmark_data_smithi194_186548_object21341:head# 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c130b98:::benchmark_data_smithi194_186548_object61764:head# 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c138c30:::benchmark_data_smithi194_186548_object5095:head# 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c13d258:::benchmark_data_smithi194_186548_object67508:head# 2024-04-29T14:53:55.512 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c14004e:::benchmark_data_smithi194_186548_object23180:head# 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c142b82:::benchmark_data_smithi194_186548_object7138:head# 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c15ff2f:::benchmark_data_smithi194_186548_object75641:head# 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c173fc1:::benchmark_data_smithi194_186548_object57772:head# 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1774d7:::benchmark_data_smithi194_186548_object34908:head# 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c17efde:::benchmark_data_smithi194_186548_object68572:head# 2024-04-29T14:53:55.513 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c182ce2:::benchmark_data_smithi194_186548_object78386:head# 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c18432d:::benchmark_data_smithi194_186548_object43251:head# 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c19500a:::benchmark_data_smithi194_186548_object70095:head# 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1a8ffa:::benchmark_data_smithi194_186548_object61230:head# 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1be0c3:::benchmark_data_smithi194_186548_object30371:head# 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1eb0b7:::benchmark_data_smithi194_186548_object33723:head# 2024-04-29T14:53:55.514 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.516 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1ecec5:::benchmark_data_smithi194_186548_object50423:head# 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1f6e76:::benchmark_data_smithi194_186548_object11242:head# 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1fa31f:::benchmark_data_smithi194_186548_object76336:head# 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1fbc71:::benchmark_data_smithi194_186548_object48240:head# 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c1fd029:::benchmark_data_smithi194_186548_object56909:head# 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c20e54e:::benchmark_data_smithi194_186548_object37151:head# 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.517 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c221dd8:::benchmark_data_smithi194_186548_object36211:head# 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2317d4:::benchmark_data_smithi194_186548_object45862:head# 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c23e3f2:::benchmark_data_smithi194_186548_object70929:head# 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c24bbc0:::benchmark_data_smithi194_186548_object78526:head# 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c25ce42:::benchmark_data_smithi194_186548_object6755:head# 2024-04-29T14:53:55.518 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.519 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c25ea01:::benchmark_data_smithi194_186548_object5099:head# 2024-04-29T14:53:55.519 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.519 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2651bd:::benchmark_data_smithi194_186548_object80110:head# 2024-04-29T14:53:55.519 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.519 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c26c482:::benchmark_data_smithi194_186548_object65017:head# 2024-04-29T14:53:55.519 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.520 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c276b8f:::benchmark_data_smithi194_186548_object19266:head# 2024-04-29T14:53:55.520 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.520 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c27bd3b:::benchmark_data_smithi194_186548_object27862:head# 2024-04-29T14:53:55.520 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.520 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c286e57:::benchmark_data_smithi194_186548_object29892:head# 2024-04-29T14:53:55.521 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.521 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2985ea:::benchmark_data_smithi194_186548_object63641:head# 2024-04-29T14:53:55.521 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2a06cc:::benchmark_data_smithi194_186548_object84877:head# 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2a4852:::benchmark_data_smithi194_186548_object62598:head# 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2a8830:::benchmark_data_smithi194_186548_object58810:head# 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2af1cc:::benchmark_data_smithi194_186548_object56676:head# 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2c7741:::benchmark_data_smithi194_186548_object83938:head# 2024-04-29T14:53:55.522 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.523 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2ca6a4:::benchmark_data_smithi194_186548_object66702:head# 2024-04-29T14:53:55.523 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2cf9fb:::benchmark_data_smithi194_186548_object22857:head# 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2cfeed:::benchmark_data_smithi194_186548_object71276:head# 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2d8c8b:::benchmark_data_smithi194_186548_object63436:head# 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2da9bf:::benchmark_data_smithi194_186548_object44463:head# 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2de521:::benchmark_data_smithi194_186548_object84492:head# 2024-04-29T14:53:55.524 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.528 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2e6818:::benchmark_data_smithi194_186548_object47079:head# 2024-04-29T14:53:55.528 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.529 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2f0fdd:::benchmark_data_smithi194_186548_object33325:head# 2024-04-29T14:53:55.529 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.529 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c2f21c2:::benchmark_data_smithi194_186548_object84544:head# 2024-04-29T14:53:55.529 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.531 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c313931:::benchmark_data_smithi194_186548_object5696:head# 2024-04-29T14:53:55.531 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.531 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c324d19:::benchmark_data_smithi194_186548_object24404:head# 2024-04-29T14:53:55.531 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.532 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c326a3c:::benchmark_data_smithi194_186548_object6127:head# 2024-04-29T14:53:55.532 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.532 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c32bd8d:::benchmark_data_smithi194_186548_object48885:head# 2024-04-29T14:53:55.532 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.532 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c32bdd3:::benchmark_data_smithi194_186548_object29451:head# 2024-04-29T14:53:55.532 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.532 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c33f050:::benchmark_data_smithi194_186548_object12853:head# 2024-04-29T14:53:55.533 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c34c3d1:::benchmark_data_smithi194_186548_object830:head# 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c34c8c2:::benchmark_data_smithi194_186548_object32737:head# 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c35dec8:::benchmark_data_smithi194_186548_object47840:head# 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c3b6840:::benchmark_data_smithi194_186548_object30244:head# 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c3c2eec:::benchmark_data_smithi194_186548_object71988:head# 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.535 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c3df5ac:::benchmark_data_smithi194_186548_object13110:head# 2024-04-29T14:53:55.536 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.536 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c3e39ce:::benchmark_data_smithi194_186548_object1126:head# 2024-04-29T14:53:55.536 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.536 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c3ed592:::benchmark_data_smithi194_186548_object22045:head# 2024-04-29T14:53:55.536 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.536 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c406a06:::benchmark_data_smithi194_186548_object1428:head# 2024-04-29T14:53:55.536 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.538 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c407381:::benchmark_data_smithi194_186548_object7170:head# 2024-04-29T14:53:55.538 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.538 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c411feb:::benchmark_data_smithi194_186548_object38911:head# 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c41ee9c:::benchmark_data_smithi194_186548_object35289:head# 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4224c2:::benchmark_data_smithi194_186548_object77333:head# 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4343ef:::benchmark_data_smithi194_186548_object71473:head# 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c43981e:::benchmark_data_smithi194_186548_object55521:head# 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4452bf:::benchmark_data_smithi194_186548_object21858:head# 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c44c84e:::benchmark_data_smithi194_186548_object4172:head# 2024-04-29T14:53:55.539 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4710cf:::benchmark_data_smithi194_186548_object59908:head# 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c481476:::benchmark_data_smithi194_186548_object50007:head# 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c482797:::benchmark_data_smithi194_186548_object46368:head# 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4936d0:::benchmark_data_smithi194_186548_object5632:head# 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c49842c:::benchmark_data_smithi194_186548_object2895:head# 2024-04-29T14:53:55.541 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.543 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4a80a5:::benchmark_data_smithi194_186548_object23059:head# 2024-04-29T14:53:55.543 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.544 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4a9b54:::benchmark_data_smithi194_186548_object37162:head# 2024-04-29T14:53:55.544 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4ab09f:::benchmark_data_smithi194_186548_object24772:head# 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4b5af8:::benchmark_data_smithi194_186548_object17255:head# 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4bc304:::benchmark_data_smithi194_186548_object21441:head# 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4c35c1:::benchmark_data_smithi194_186548_object21505:head# 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4c5148:::benchmark_data_smithi194_186548_object61733:head# 2024-04-29T14:53:55.545 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.546 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4d4be8:::benchmark_data_smithi194_186548_object64888:head# 2024-04-29T14:53:55.546 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.550 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4d86eb:::benchmark_data_smithi194_186548_object64391:head# 2024-04-29T14:53:55.550 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.550 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4d9374:::benchmark_data_smithi194_186548_object75740:head# 2024-04-29T14:53:55.550 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.555 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4dd8b0:::benchmark_data_smithi194_186548_object7213:head# 2024-04-29T14:53:55.555 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.558 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4e9b3d:::benchmark_data_smithi194_186548_object27525:head# 2024-04-29T14:53:55.559 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.559 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c4fcac5:::benchmark_data_smithi194_186548_object73379:head# 2024-04-29T14:53:55.559 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.559 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c506547:::benchmark_data_smithi194_186548_object24244:head# 2024-04-29T14:53:55.559 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.559 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c50dd47:::benchmark_data_smithi194_186548_object26999:head# 2024-04-29T14:53:55.560 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.560 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c52163e:::benchmark_data_smithi194_186548_object16897:head# 2024-04-29T14:53:55.560 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.560 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5277f3:::benchmark_data_smithi194_186548_object12827:head# 2024-04-29T14:53:55.560 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.560 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c538352:::benchmark_data_smithi194_186548_object68152:head# 2024-04-29T14:53:55.561 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.561 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5479f7:::benchmark_data_smithi194_186548_object83660:head# 2024-04-29T14:53:55.561 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.562 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c54aa83:::benchmark_data_smithi194_186548_object27563:head# 2024-04-29T14:53:55.563 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.563 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5586eb:::benchmark_data_smithi194_186548_object75004:head# 2024-04-29T14:53:55.563 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c558fa0:::benchmark_data_smithi194_186548_object31741:head# 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c56f34a:::benchmark_data_smithi194_186548_object62976:head# 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c57576f:::benchmark_data_smithi194_186548_object80161:head# 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5760e3:::benchmark_data_smithi194_186548_object45882:head# 2024-04-29T14:53:55.565 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.566 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5a92e2:::benchmark_data_smithi194_186548_object40070:head# 2024-04-29T14:53:55.566 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.566 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5b856c:::benchmark_data_smithi194_186548_object31993:head# 2024-04-29T14:53:55.566 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.566 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5b9b14:::benchmark_data_smithi194_186548_object73988:head# 2024-04-29T14:53:55.566 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.570 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5c8e94:::benchmark_data_smithi194_186548_object66073:head# 2024-04-29T14:53:55.571 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.579 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5c93f7:::benchmark_data_smithi194_186548_object71623:head# 2024-04-29T14:53:55.579 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.585 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5cc25f:::benchmark_data_smithi194_186548_object48290:head# 2024-04-29T14:53:55.585 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5ef526:::benchmark_data_smithi194_186548_object33023:head# 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5f2291:::benchmark_data_smithi194_186548_object21511:head# 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c5f488b:::benchmark_data_smithi194_186548_object18461:head# 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c613dc3:::benchmark_data_smithi194_186548_object44273:head# 2024-04-29T14:53:55.598 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6141b4:::benchmark_data_smithi194_186548_object22679:head# 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c623548:::benchmark_data_smithi194_186548_object57694:head# 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c62e7bc:::benchmark_data_smithi194_186548_object13123:head# 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c64d1d8:::benchmark_data_smithi194_186548_object5245:head# 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c65317f:::benchmark_data_smithi194_186548_object45155:head# 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.599 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c658e1b:::benchmark_data_smithi194_186548_object38293:head# 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c65a687:::benchmark_data_smithi194_186548_object69239:head# 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c669a85:::benchmark_data_smithi194_186548_object30517:head# 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c66f8f4:::benchmark_data_smithi194_186548_object3109:head# 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c67393c:::benchmark_data_smithi194_186548_object54791:head# 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6826cf:::benchmark_data_smithi194_186548_object7231:head# 2024-04-29T14:53:55.600 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c69198a:::benchmark_data_smithi194_186548_object20384:head# 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c69ea96:::benchmark_data_smithi194_186548_object1733:head# 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6a2a65:::benchmark_data_smithi194_186548_object70542:head# 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6ae302:::benchmark_data_smithi194_186548_object6106:head# 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6b0f27:::benchmark_data_smithi194_186548_object24821:head# 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6b6a9e:::benchmark_data_smithi194_186548_object72015:head# 2024-04-29T14:53:55.601 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6d99cb:::benchmark_data_smithi194_186548_object64943:head# 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6e4989:::benchmark_data_smithi194_186548_object72363:head# 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6e7399:::benchmark_data_smithi194_186548_object64329:head# 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6f0846:::benchmark_data_smithi194_186548_object20128:head# 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c6f9db4:::benchmark_data_smithi194_186548_object46095:head# 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c704e88:::benchmark_data_smithi194_186548_object58274:head# 2024-04-29T14:53:55.602 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.603 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c708c08:::benchmark_data_smithi194_186548_object48550:head# 2024-04-29T14:53:55.603 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c717fc8:::benchmark_data_smithi194_186548_object67781:head# 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c718782:::benchmark_data_smithi194_186548_object75643:head# 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7249aa:::benchmark_data_smithi194_186548_object26780:head# 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c72f3a0:::benchmark_data_smithi194_186548_object84721:head# 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7327fb:::benchmark_data_smithi194_186548_object14932:head# 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c73b6b1:::benchmark_data_smithi194_186548_object58154:head# 2024-04-29T14:53:55.605 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c751e5d:::benchmark_data_smithi194_186548_object38889:head# 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7527ca:::benchmark_data_smithi194_186548_object77295:head# 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c75bcc2:::benchmark_data_smithi194_186548_object46466:head# 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c75be1d:::benchmark_data_smithi194_186548_object8421:head# 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c791d78:::benchmark_data_smithi194_186548_object41417:head# 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7930de:::benchmark_data_smithi194_186548_object77153:head# 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.606 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c79c6e4:::benchmark_data_smithi194_186548_object39607:head# 2024-04-29T14:53:55.607 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.607 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7a4e9e:::benchmark_data_smithi194_186548_object41828:head# 2024-04-29T14:53:55.607 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.607 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7b9f56:::benchmark_data_smithi194_186548_object714:head# 2024-04-29T14:53:55.607 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.609 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7baf55:::benchmark_data_smithi194_186548_object48214:head# 2024-04-29T14:53:55.609 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.609 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7c7fb2:::benchmark_data_smithi194_186548_object979:head# 2024-04-29T14:53:55.609 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.609 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7c8194:::benchmark_data_smithi194_186548_object15331:head# 2024-04-29T14:53:55.609 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.609 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7ce056:::benchmark_data_smithi194_186548_object68864:head# 2024-04-29T14:53:55.610 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.610 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7d5177:::benchmark_data_smithi194_186548_object50051:head# 2024-04-29T14:53:55.610 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.610 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7d6352:::benchmark_data_smithi194_186548_object43518:head# 2024-04-29T14:53:55.610 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.610 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7e5adf:::benchmark_data_smithi194_186548_object13384:head# 2024-04-29T14:53:55.610 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.611 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7e6d00:::benchmark_data_smithi194_186548_object20086:head# 2024-04-29T14:53:55.611 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.611 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7eb65d:::benchmark_data_smithi194_186548_object68734:head# 2024-04-29T14:53:55.611 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.611 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7ec5f8:::benchmark_data_smithi194_186548_object64047:head# 2024-04-29T14:53:55.611 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7ed4c0:::benchmark_data_smithi194_186548_object12462:head# 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7f5add:::benchmark_data_smithi194_186548_object67471:head# 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c7fdfe8:::benchmark_data_smithi194_186548_object55754:head# 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c802f7a:::benchmark_data_smithi194_186548_object57121:head# 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c818216:::benchmark_data_smithi194_186548_object7852:head# 2024-04-29T14:53:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.613 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c81e8b7:::benchmark_data_smithi194_186548_object45369:head# 2024-04-29T14:53:55.613 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.613 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c81ff81:::benchmark_data_smithi194_186548_object15545:head# 2024-04-29T14:53:55.614 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.614 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c82355c:::benchmark_data_smithi194_186548_object60972:head# 2024-04-29T14:53:55.614 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.614 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c84f2a6:::benchmark_data_smithi194_186548_object7232:head# 2024-04-29T14:53:55.614 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.615 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c85383b:::benchmark_data_smithi194_186548_object51983:head# 2024-04-29T14:53:55.615 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.616 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c85ddf0:::benchmark_data_smithi194_186548_object7010:head# 2024-04-29T14:53:55.616 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.616 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c88cd69:::benchmark_data_smithi194_186548_object1340:head# 2024-04-29T14:53:55.617 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.617 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c89b07f:::benchmark_data_smithi194_186548_object19114:head# 2024-04-29T14:53:55.617 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.617 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c89e7c0:::benchmark_data_smithi194_186548_object58498:head# 2024-04-29T14:53:55.617 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.617 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c8ad326:::benchmark_data_smithi194_186548_object58170:head# 2024-04-29T14:53:55.618 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.618 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c8bc5b8:::benchmark_data_smithi194_186548_object77651:head# 2024-04-29T14:53:55.618 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.619 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c8c3afc:::benchmark_data_smithi194_186548_object63859:head# 2024-04-29T14:53:55.619 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c8de803:::benchmark_data_smithi194_186548_object64140:head# 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c8e8ce1:::benchmark_data_smithi194_186548_object2025:head# 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c8f3b4c:::benchmark_data_smithi194_186548_object1744:head# 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c908e58:::benchmark_data_smithi194_186548_object75663:head# 2024-04-29T14:53:55.620 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.621 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c911dc6:::benchmark_data_smithi194_186548_object76904:head# 2024-04-29T14:53:55.621 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.621 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9315c9:::benchmark_data_smithi194_186548_object41567:head# 2024-04-29T14:53:55.621 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.621 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c936f03:::benchmark_data_smithi194_186548_object31180:head# 2024-04-29T14:53:55.622 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.622 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c951eab:::benchmark_data_smithi194_186548_object73819:head# 2024-04-29T14:53:55.622 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.622 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c955368:::benchmark_data_smithi194_186548_object35094:head# 2024-04-29T14:53:55.622 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.623 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c967335:::benchmark_data_smithi194_186548_object56338:head# 2024-04-29T14:53:55.623 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.623 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c96818b:::benchmark_data_smithi194_186548_object21900:head# 2024-04-29T14:53:55.623 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.623 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c97826b:::benchmark_data_smithi194_186548_object74994:head# 2024-04-29T14:53:55.623 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c97b116:::benchmark_data_smithi194_186548_object36320:head# 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c984b4f:::benchmark_data_smithi194_186548_object71061:head# 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c98c6c0:::benchmark_data_smithi194_186548_object74710:head# 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9a6ae3:::benchmark_data_smithi194_186548_object64442:head# 2024-04-29T14:53:55.624 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.625 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9a6d45:::benchmark_data_smithi194_186548_object65618:head# 2024-04-29T14:53:55.625 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.625 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9b48af:::benchmark_data_smithi194_186548_object52084:head# 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9b4c92:::benchmark_data_smithi194_186548_object5179:head# 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9b612e:::benchmark_data_smithi194_186548_object22217:head# 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9bd851:::benchmark_data_smithi194_186548_object51042:head# 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9bee89:::benchmark_data_smithi194_186548_object19275:head# 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9d5c62:::benchmark_data_smithi194_186548_object29567:head# 2024-04-29T14:53:55.626 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.627 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3c9d7567:::benchmark_data_smithi194_186548_object36664:head# 2024-04-29T14:53:55.627 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.630 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca04631:::benchmark_data_smithi194_186548_object75298:head# 2024-04-29T14:53:55.631 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca0d6fa:::benchmark_data_smithi194_186548_object71295:head# 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca109bc:::benchmark_data_smithi194_186548_object61977:head# 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca12f35:::benchmark_data_smithi194_186548_object27122:head# 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca3ad9c:::benchmark_data_smithi194_186548_object66492:head# 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.632 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca47d37:::benchmark_data_smithi194_186548_object59519:head# 2024-04-29T14:53:55.633 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.634 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca48727:::benchmark_data_smithi194_186548_object55680:head# 2024-04-29T14:53:55.634 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.634 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca66f8a:::benchmark_data_smithi194_186548_object29226:head# 2024-04-29T14:53:55.634 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca811cf:::benchmark_data_smithi194_186548_object61064:head# 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca86488:::benchmark_data_smithi194_186548_object24159:head# 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca87b33:::benchmark_data_smithi194_186548_object206:head# 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca8fbb1:::benchmark_data_smithi194_186548_object13184:head# 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca991b8:::benchmark_data_smithi194_186548_object16317:head# 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca9a0d6:::benchmark_data_smithi194_186548_object63964:head# 2024-04-29T14:53:55.635 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ca9b7d7:::benchmark_data_smithi194_186548_object69042:head# 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3caabec5:::benchmark_data_smithi194_186548_object5147:head# 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cab648c:::benchmark_data_smithi194_186548_object38657:head# 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3caba960:::benchmark_data_smithi194_186548_object29822:head# 2024-04-29T14:53:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cad8e4f:::benchmark_data_smithi194_186548_object48778:head# 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3caec840:::benchmark_data_smithi194_186548_object3523:head# 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3caf2385:::benchmark_data_smithi194_186548_object29640:head# 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb010ab:::benchmark_data_smithi194_186548_object80858:head# 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb1072f:::benchmark_data_smithi194_186548_object15301:head# 2024-04-29T14:53:55.638 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.639 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb17388:::benchmark_data_smithi194_186548_object33044:head# 2024-04-29T14:53:55.639 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.639 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb31bfc:::benchmark_data_smithi194_186548_object53987:head# 2024-04-29T14:53:55.639 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.639 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb344f7:::benchmark_data_smithi194_186548_object80609:head# 2024-04-29T14:53:55.639 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.642 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb38f62:::benchmark_data_smithi194_186548_object74996:head# 2024-04-29T14:53:55.642 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.643 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb58559:::benchmark_data_smithi194_186548_object15739:head# 2024-04-29T14:53:55.643 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.643 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb83361:::benchmark_data_smithi194_186548_object49085:head# 2024-04-29T14:53:55.643 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.643 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cb9c071:::benchmark_data_smithi194_186548_object74730:head# 2024-04-29T14:53:55.643 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.648 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbb53b3:::benchmark_data_smithi194_186548_object57821:head# 2024-04-29T14:53:55.648 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbc152d:::benchmark_data_smithi194_186548_object12587:head# 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbc6c82:::benchmark_data_smithi194_186548_object5223:head# 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbcc00d:::benchmark_data_smithi194_186548_object12535:head# 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbd0473:::benchmark_data_smithi194_186548_object37042:head# 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbd58fb:::benchmark_data_smithi194_186548_object18474:head# 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbd83ba:::benchmark_data_smithi194_186548_object10283:head# 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.649 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbe932d:::benchmark_data_smithi194_186548_object47044:head# 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbf5fd1:::benchmark_data_smithi194_186548_object24478:head# 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbfef34:::benchmark_data_smithi194_186548_object40325:head# 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cbff4af:::benchmark_data_smithi194_186548_object81530:head# 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc01180:::benchmark_data_smithi194_186548_object32595:head# 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc0d3af:::benchmark_data_smithi194_186548_object81134:head# 2024-04-29T14:53:55.650 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc1a22d:::benchmark_data_smithi194_186548_object12240:head# 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc1ba0d:::benchmark_data_smithi194_186548_object63895:head# 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc3bc71:::benchmark_data_smithi194_186548_object25582:head# 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc42473:::benchmark_data_smithi194_186548_object62705:head# 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc5490c:::benchmark_data_smithi194_186548_object64011:head# 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc66e18:::benchmark_data_smithi194_186548_object49495:head# 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.651 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc6ba14:::benchmark_data_smithi194_186548_object69067:head# 2024-04-29T14:53:55.652 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.655 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc6edc6:::benchmark_data_smithi194_186548_object40009:head# 2024-04-29T14:53:55.655 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.655 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc748be:::benchmark_data_smithi194_186548_object50183:head# 2024-04-29T14:53:55.655 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.655 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc7cf85:::benchmark_data_smithi194_186548_object23864:head# 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc7d1b2:::benchmark_data_smithi194_186548_object39200:head# 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cc8b5b9:::benchmark_data_smithi194_186548_object571:head# 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ccb41ee:::benchmark_data_smithi194_186548_object84798:head# 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ccbc710:::benchmark_data_smithi194_186548_object25486:head# 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ccc57d6:::benchmark_data_smithi194_186548_object76681:head# 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.656 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ccd8cb6:::benchmark_data_smithi194_186548_object21409:head# 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ccda619:::benchmark_data_smithi194_186548_object32722:head# 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ccdda2a:::benchmark_data_smithi194_186548_object11320:head# 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd03b20:::benchmark_data_smithi194_186548_object66908:head# 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd0c808:::benchmark_data_smithi194_186548_object16743:head# 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd1876d:::benchmark_data_smithi194_186548_object58755:head# 2024-04-29T14:53:55.657 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd19bb5:::benchmark_data_smithi194_186548_object67622:head# 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd1e2be:::benchmark_data_smithi194_186548_object54235:head# 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd23d66:::benchmark_data_smithi194_186548_object15745:head# 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd27aad:::benchmark_data_smithi194_186548_object79060:head# 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd34fb9:::benchmark_data_smithi194_186548_object52680:head# 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.658 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd39644:::benchmark_data_smithi194_186548_object6645:head# 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd3ba7f:::benchmark_data_smithi194_186548_object65964:head# 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd3f5f1:::benchmark_data_smithi194_186548_object76472:head# 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd4a43c:::benchmark_data_smithi194_186548_object62801:head# 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd71d5a:::benchmark_data_smithi194_186548_object75741:head# 2024-04-29T14:53:55.659 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd73e10:::benchmark_data_smithi194_186548_object52994:head# 2024-04-29T14:53:55.661 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.661 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd85907:::benchmark_data_smithi194_186548_object67050:head# 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd8bc47:::benchmark_data_smithi194_186548_object73123:head# 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd8db1b:::benchmark_data_smithi194_186548_object9588:head# 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cd8f546:::benchmark_data_smithi194_186548_object19474:head# 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cda4ce2:::benchmark_data_smithi194_186548_object27143:head# 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cdb13e4:::benchmark_data_smithi194_186548_object24452:head# 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cdc68df:::benchmark_data_smithi194_186548_object24532:head# 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.662 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cdc77b5:::benchmark_data_smithi194_186548_object49182:head# 2024-04-29T14:53:55.663 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.663 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cddbf8e:::benchmark_data_smithi194_186548_object44914:head# 2024-04-29T14:53:55.663 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.663 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cde2948:::benchmark_data_smithi194_186548_object10213:head# 2024-04-29T14:53:55.663 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.663 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cde343f:::benchmark_data_smithi194_186548_object78283:head# 2024-04-29T14:53:55.663 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.666 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cde8b6c:::benchmark_data_smithi194_186548_object66314:head# 2024-04-29T14:53:55.666 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cdf068c:::benchmark_data_smithi194_186548_object76975:head# 2024-04-29T14:53:55.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce0bb85:::benchmark_data_smithi194_186548_object45910:head# 2024-04-29T14:53:55.667 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.667 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce1cc69:::benchmark_data_smithi194_186548_object30725:head# 2024-04-29T14:53:55.668 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.668 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce2b84f:::benchmark_data_smithi194_186548_object56572:head# 2024-04-29T14:53:55.668 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.668 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce2bbb9:::benchmark_data_smithi194_186548_object30671:head# 2024-04-29T14:53:55.668 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.668 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce3f848:::benchmark_data_smithi194_186548_object79062:head# 2024-04-29T14:53:55.668 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce3fab2:::benchmark_data_smithi194_186548_object16093:head# 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce5304b:::benchmark_data_smithi194_186548_object48450:head# 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce6dc24:::benchmark_data_smithi194_186548_object75409:head# 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce6f678:::benchmark_data_smithi194_186548_object49228:head# 2024-04-29T14:53:55.669 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce75150:::benchmark_data_smithi194_186548_object42604:head# 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce767cc:::benchmark_data_smithi194_186548_object6099:head# 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce7d53e:::benchmark_data_smithi194_186548_object63816:head# 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce890c3:::benchmark_data_smithi194_186548_object32020:head# 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.671 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce93f6f:::benchmark_data_smithi194_186548_object73931:head# 2024-04-29T14:53:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ce9e4f2:::benchmark_data_smithi194_186548_object43319:head# 2024-04-29T14:53:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cea3f79:::benchmark_data_smithi194_186548_object35385:head# 2024-04-29T14:53:55.672 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ceae50e:::benchmark_data_smithi194_186548_object83896:head# 2024-04-29T14:53:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.674 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ceba365:::benchmark_data_smithi194_186548_object73430:head# 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cecb8cd:::benchmark_data_smithi194_186548_object77536:head# 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cee42ae:::benchmark_data_smithi194_186548_object83468:head# 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ceefaba:::benchmark_data_smithi194_186548_object72764:head# 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf0e736:::benchmark_data_smithi194_186548_object44439:head# 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf0f96c:::benchmark_data_smithi194_186548_object84993:head# 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf11796:::benchmark_data_smithi194_186548_object80167:head# 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.675 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf16d49:::benchmark_data_smithi194_186548_object16282:head# 2024-04-29T14:53:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf1f49d:::benchmark_data_smithi194_186548_object30407:head# 2024-04-29T14:53:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf2049e:::benchmark_data_smithi194_186548_object6015:head# 2024-04-29T14:53:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf2393e:::benchmark_data_smithi194_186548_object1150:head# 2024-04-29T14:53:55.676 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.685 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf3e61a:::benchmark_data_smithi194_186548_object76133:head# 2024-04-29T14:53:55.685 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.686 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf4dca6:::benchmark_data_smithi194_186548_object79350:head# 2024-04-29T14:53:55.686 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.686 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf536cf:::benchmark_data_smithi194_186548_object46790:head# 2024-04-29T14:53:55.686 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.687 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf65b4c:::benchmark_data_smithi194_186548_object5704:head# 2024-04-29T14:53:55.687 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.687 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cf86024:::benchmark_data_smithi194_186548_object25152:head# 2024-04-29T14:53:55.687 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.687 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cfa03dd:::benchmark_data_smithi194_186548_object56887:head# 2024-04-29T14:53:55.688 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.688 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cfa7d75:::benchmark_data_smithi194_186548_object46833:head# 2024-04-29T14:53:55.688 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.688 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cfa96ff:::benchmark_data_smithi194_186548_object7199:head# 2024-04-29T14:53:55.688 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.688 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cfba487:::benchmark_data_smithi194_186548_object12533:head# 2024-04-29T14:53:55.688 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.692 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cfd6a8e:::benchmark_data_smithi194_186548_object3200:head# 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3cff2da7:::benchmark_data_smithi194_186548_object65507:head# 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0140ca:::benchmark_data_smithi194_186548_object56445:head# 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d01c95d:::benchmark_data_smithi194_186548_object4784:head# 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d024054:::benchmark_data_smithi194_186548_object73909:head# 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d02a038:::benchmark_data_smithi194_186548_object48201:head# 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.693 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d03a63e:::benchmark_data_smithi194_186548_object52315:head# 2024-04-29T14:53:55.694 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.694 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d04bae8:::benchmark_data_smithi194_186548_object54891:head# 2024-04-29T14:53:55.694 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.702 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d055b9f:::benchmark_data_smithi194_186548_object68772:head# 2024-04-29T14:53:55.702 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.703 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0643c2:::benchmark_data_smithi194_186548_object61034:head# 2024-04-29T14:53:55.703 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.706 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d06732e:::benchmark_data_smithi194_186548_object45855:head# 2024-04-29T14:53:55.706 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d075c64:::benchmark_data_smithi194_186548_object8459:head# 2024-04-29T14:53:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d09ad55:::benchmark_data_smithi194_186548_object34192:head# 2024-04-29T14:53:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.707 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0a045a:::benchmark_data_smithi194_186548_object62530:head# 2024-04-29T14:53:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0a89d6:::benchmark_data_smithi194_186548_object4844:head# 2024-04-29T14:53:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0ae772:::benchmark_data_smithi194_186548_object42753:head# 2024-04-29T14:53:55.708 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0b3cd1:::benchmark_data_smithi194_186548_object60522:head# 2024-04-29T14:53:55.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0bee17:::benchmark_data_smithi194_186548_object576:head# 2024-04-29T14:53:55.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.709 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0cab48:::benchmark_data_smithi194_186548_object11971:head# 2024-04-29T14:53:55.709 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.710 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0ce11e:::benchmark_data_smithi194_186548_object56672:head# 2024-04-29T14:53:55.710 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.710 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d0d11d0:::benchmark_data_smithi194_186548_object17643:head# 2024-04-29T14:53:55.711 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.711 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d100dd6:::benchmark_data_smithi194_186548_object76532:head# 2024-04-29T14:53:55.711 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.711 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d105937:::benchmark_data_smithi194_186548_object60356:head# 2024-04-29T14:53:55.711 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.711 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d10d66d:::benchmark_data_smithi194_186548_object65305:head# 2024-04-29T14:53:55.712 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.712 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d10fe01:::benchmark_data_smithi194_186548_object7377:head# 2024-04-29T14:53:55.712 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.719 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d114c7d:::benchmark_data_smithi194_186548_object54537:head# 2024-04-29T14:53:55.719 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.724 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d11755c:::benchmark_data_smithi194_186548_object27214:head# 2024-04-29T14:53:55.724 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1299a2:::benchmark_data_smithi194_186548_object73124:head# 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d12b667:::benchmark_data_smithi194_186548_object65497:head# 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d134752:::benchmark_data_smithi194_186548_object39401:head# 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d15b34e:::benchmark_data_smithi194_186548_object48026:head# 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d166fda:::benchmark_data_smithi194_186548_object1259:head# 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.727 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d16dc38:::benchmark_data_smithi194_186548_object84534:head# 2024-04-29T14:53:55.728 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.728 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d179325:::benchmark_data_smithi194_186548_object28844:head# 2024-04-29T14:53:55.728 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.728 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d179b81:::benchmark_data_smithi194_186548_object73843:head# 2024-04-29T14:53:55.728 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d17c039:::benchmark_data_smithi194_186548_object49861:head# 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d183273:::benchmark_data_smithi194_186548_object8756:head# 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d18a1a2:::benchmark_data_smithi194_186548_object43924:head# 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d196a3d:::benchmark_data_smithi194_186548_object82105:head# 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d197a36:::benchmark_data_smithi194_186548_object64087:head# 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.730 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d19ae86:::benchmark_data_smithi194_186548_object2011:head# 2024-04-29T14:53:55.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d19d5ad:::benchmark_data_smithi194_186548_object9889:head# 2024-04-29T14:53:55.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1a8534:::benchmark_data_smithi194_186548_object31365:head# 2024-04-29T14:53:55.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.731 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1b8b5c:::benchmark_data_smithi194_186548_object21577:head# 2024-04-29T14:53:55.731 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1bcc04:::benchmark_data_smithi194_186548_object61355:head# 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1c3bbd:::benchmark_data_smithi194_186548_object31764:head# 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1c7962:::benchmark_data_smithi194_186548_object6940:head# 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1ce581:::benchmark_data_smithi194_186548_object48027:head# 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1d9497:::benchmark_data_smithi194_186548_object22260:head# 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1db253:::benchmark_data_smithi194_186548_object83806:head# 2024-04-29T14:53:55.733 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1e6b36:::benchmark_data_smithi194_186548_object71174:head# 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d1f8e0f:::benchmark_data_smithi194_186548_object37318:head# 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d206666:::benchmark_data_smithi194_186548_object24576:head# 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d207ea2:::benchmark_data_smithi194_186548_object77827:head# 2024-04-29T14:53:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d210bea:::benchmark_data_smithi194_186548_object76405:head# 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d235435:::benchmark_data_smithi194_186548_object80779:head# 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d241a9a:::benchmark_data_smithi194_186548_object58315:head# 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2485e0:::benchmark_data_smithi194_186548_object14085:head# 2024-04-29T14:53:55.736 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d26906b:::benchmark_data_smithi194_186548_object4360:head# 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d26dc5e:::benchmark_data_smithi194_186548_object58140:head# 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d274900:::benchmark_data_smithi194_186548_object22078:head# 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d276633:::benchmark_data_smithi194_186548_object4970:head# 2024-04-29T14:53:55.737 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.741 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d27bbfc:::benchmark_data_smithi194_186548_object11937:head# 2024-04-29T14:53:55.741 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.741 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d28534f:::benchmark_data_smithi194_186548_object40378:head# 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2876f1:::benchmark_data_smithi194_186548_object66225:head# 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d29031b:::benchmark_data_smithi194_186548_object13625:head# 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d290cf2:::benchmark_data_smithi194_186548_object21842:head# 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d294921:::benchmark_data_smithi194_186548_object3602:head# 2024-04-29T14:53:55.742 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.743 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d299295:::benchmark_data_smithi194_186548_object66504:head# 2024-04-29T14:53:55.743 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.743 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2a9fe8:::benchmark_data_smithi194_186548_object60413:head# 2024-04-29T14:53:55.744 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.744 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2b415c:::benchmark_data_smithi194_186548_object47343:head# 2024-04-29T14:53:55.744 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.744 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2bde2c:::benchmark_data_smithi194_186548_object3652:head# 2024-04-29T14:53:55.744 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:55 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:53:55.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3760084016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:55.747 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:55 smithi194 ceph-mon[44089]: osdmap e1040: 12 total, 11 up, 9 in 2024-04-29T14:53:55.750 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2bf735:::benchmark_data_smithi194_186548_object64562:head# 2024-04-29T14:53:55.750 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.750 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2f01a5:::benchmark_data_smithi194_186548_object25754:head# 2024-04-29T14:53:55.750 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.750 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2f5a2f:::benchmark_data_smithi194_186548_object25608:head# 2024-04-29T14:53:55.750 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.753 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d2faa26:::benchmark_data_smithi194_186548_object22272:head# 2024-04-29T14:53:55.754 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d30a3ae:::benchmark_data_smithi194_186548_object14474:head# 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3222cf:::benchmark_data_smithi194_186548_object28554:head# 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d323863:::benchmark_data_smithi194_186548_object1124:head# 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d323be2:::benchmark_data_smithi194_186548_object79894:head# 2024-04-29T14:53:55.758 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3265c0:::benchmark_data_smithi194_186548_object46464:head# 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d337c70:::benchmark_data_smithi194_186548_object59254:head# 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d337ea3:::benchmark_data_smithi194_186548_object45570:head# 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d33f6ae:::benchmark_data_smithi194_186548_object33784:head# 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d348c39:::benchmark_data_smithi194_186548_object6956:head# 2024-04-29T14:53:55.760 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.761 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d34c408:::benchmark_data_smithi194_186548_object10045:head# 2024-04-29T14:53:55.761 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.761 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d35f9e0:::benchmark_data_smithi194_186548_object70467:head# 2024-04-29T14:53:55.761 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.761 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d368fd3:::benchmark_data_smithi194_186548_object30733:head# 2024-04-29T14:53:55.761 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.762 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d36f489:::benchmark_data_smithi194_186548_object33570:head# 2024-04-29T14:53:55.762 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.762 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d37d241:::benchmark_data_smithi194_186548_object77887:head# 2024-04-29T14:53:55.762 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.762 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d386718:::benchmark_data_smithi194_186548_object51703:head# 2024-04-29T14:53:55.762 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.762 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d38956f:::benchmark_data_smithi194_186548_object40478:head# 2024-04-29T14:53:55.763 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.763 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3a19fc:::benchmark_data_smithi194_186548_object25185:head# 2024-04-29T14:53:55.763 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3acd52:::benchmark_data_smithi194_186548_object49482:head# 2024-04-29T14:53:55.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3b530b:::benchmark_data_smithi194_186548_object13113:head# 2024-04-29T14:53:55.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3b8713:::benchmark_data_smithi194_186548_object4417:head# 2024-04-29T14:53:55.767 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.767 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3b8727:::benchmark_data_smithi194_186548_object2495:head# 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3bab57:::benchmark_data_smithi194_186548_object51634:head# 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3bcb9e:::benchmark_data_smithi194_186548_object66570:head# 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3bfa0d:::benchmark_data_smithi194_186548_object66881:head# 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3c7162:::benchmark_data_smithi194_186548_object7808:head# 2024-04-29T14:53:55.768 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3cce5b:::benchmark_data_smithi194_186548_object27600:head# 2024-04-29T14:53:55.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3d8dda:::benchmark_data_smithi194_186548_object67607:head# 2024-04-29T14:53:55.773 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.773 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3dead9:::benchmark_data_smithi194_186548_object9638:head# 2024-04-29T14:53:55.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.774 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3e6bda:::benchmark_data_smithi194_186548_object1379:head# 2024-04-29T14:53:55.774 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3e8acc:::benchmark_data_smithi194_186548_object67431:head# 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3f1783:::benchmark_data_smithi194_186548_object27997:head# 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d3f53da:::benchmark_data_smithi194_186548_object51837:head# 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d44eae8:::benchmark_data_smithi194_186548_object46566:head# 2024-04-29T14:53:55.775 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d47a6e5:::benchmark_data_smithi194_186548_object7049:head# 2024-04-29T14:53:55.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.776 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4a7127:::benchmark_data_smithi194_186548_object14615:head# 2024-04-29T14:53:55.776 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4bd171:::benchmark_data_smithi194_186548_object75362:head# 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4cf818:::benchmark_data_smithi194_186548_object30785:head# 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4d58d1:::benchmark_data_smithi194_186548_object47330:head# 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4db5f7:::benchmark_data_smithi194_186548_object13212:head# 2024-04-29T14:53:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4ec463:::benchmark_data_smithi194_186548_object4669:head# 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4f4c58:::benchmark_data_smithi194_186548_object61336:head# 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4f5fe2:::benchmark_data_smithi194_186548_object68691:head# 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d4fafbf:::benchmark_data_smithi194_186548_object68208:head# 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5089e0:::benchmark_data_smithi194_186548_object42043:head# 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d510ce6:::benchmark_data_smithi194_186548_object11221:head# 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d524967:::benchmark_data_smithi194_186548_object64929:head# 2024-04-29T14:53:55.780 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d534e4f:::benchmark_data_smithi194_186548_object9792:head# 2024-04-29T14:53:55.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d541640:::benchmark_data_smithi194_186548_object54258:head# 2024-04-29T14:53:55.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.781 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5545ce:::benchmark_data_smithi194_186548_object26200:head# 2024-04-29T14:53:55.781 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d555d5d:::benchmark_data_smithi194_186548_object61295:head# 2024-04-29T14:53:55.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.785 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d560042:::benchmark_data_smithi194_186548_object18671:head# 2024-04-29T14:53:55.785 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d578e48:::benchmark_data_smithi194_186548_object20696:head# 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5855b6:::benchmark_data_smithi194_186548_object55181:head# 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d59ae93:::benchmark_data_smithi194_186548_object29438:head# 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d59f706:::benchmark_data_smithi194_186548_object61347:head# 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5a40b8:::benchmark_data_smithi194_186548_object11110:head# 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5a789b:::benchmark_data_smithi194_186548_object80776:head# 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.786 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5b9dcf:::benchmark_data_smithi194_186548_object4649:head# 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5c6386:::benchmark_data_smithi194_186548_object70848:head# 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5c9304:::benchmark_data_smithi194_186548_object64857:head# 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d5f6a14:::benchmark_data_smithi194_186548_object69988:head# 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d60464e:::benchmark_data_smithi194_186548_object18584:head# 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.787 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6050d7:::benchmark_data_smithi194_186548_object9851:head# 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d60a91c:::benchmark_data_smithi194_186548_object20797:head# 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6157b0:::benchmark_data_smithi194_186548_object16503:head# 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d61ffc6:::benchmark_data_smithi194_186548_object75279:head# 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6530c3:::benchmark_data_smithi194_186548_object65851:head# 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d65b71b:::benchmark_data_smithi194_186548_object8026:head# 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d66078d:::benchmark_data_smithi194_186548_object2468:head# 2024-04-29T14:53:55.788 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d669072:::benchmark_data_smithi194_186548_object16727:head# 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6836fb:::benchmark_data_smithi194_186548_object33783:head# 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d68dc64:::benchmark_data_smithi194_186548_object60683:head# 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d696a42:::benchmark_data_smithi194_186548_object47236:head# 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d697400:::benchmark_data_smithi194_186548_object7900:head# 2024-04-29T14:53:55.789 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6aed38:::benchmark_data_smithi194_186548_object52983:head# 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6b7171:::benchmark_data_smithi194_186548_object13284:head# 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6c08a3:::benchmark_data_smithi194_186548_object55942:head# 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6c8a6a:::benchmark_data_smithi194_186548_object10457:head# 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6c8aa9:::benchmark_data_smithi194_186548_object30211:head# 2024-04-29T14:53:55.791 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6dcb4d:::benchmark_data_smithi194_186548_object9035:head# 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6e542d:::benchmark_data_smithi194_186548_object79328:head# 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6e5a90:::benchmark_data_smithi194_186548_object76935:head# 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6e967b:::benchmark_data_smithi194_186548_object54832:head# 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6efdbe:::benchmark_data_smithi194_186548_object23601:head# 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6f5fba:::benchmark_data_smithi194_186548_object58488:head# 2024-04-29T14:53:55.792 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6f8d9a:::benchmark_data_smithi194_186548_object83601:head# 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d6ffe9f:::benchmark_data_smithi194_186548_object65774:head# 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d705286:::benchmark_data_smithi194_186548_object28106:head# 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d70798c:::benchmark_data_smithi194_186548_object62566:head# 2024-04-29T14:53:55.794 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d71f787:::benchmark_data_smithi194_186548_object23042:head# 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d71fb96:::benchmark_data_smithi194_186548_object62975:head# 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d72d73b:::benchmark_data_smithi194_186548_object85033:head# 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d73f180:::benchmark_data_smithi194_186548_object13877:head# 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d75b992:::benchmark_data_smithi194_186548_object82024:head# 2024-04-29T14:53:55.795 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7641f4:::benchmark_data_smithi194_186548_object53132:head# 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d76ba0e:::benchmark_data_smithi194_186548_object51514:head# 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d77c05e:::benchmark_data_smithi194_186548_object6193:head# 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d77d017:::benchmark_data_smithi194_186548_object54377:head# 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d78af23:::benchmark_data_smithi194_186548_object27029:head# 2024-04-29T14:53:55.797 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7a1d91:::benchmark_data_smithi194_186548_object50586:head# 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7ab60a:::benchmark_data_smithi194_186548_object32183:head# 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7ad04d:::benchmark_data_smithi194_186548_object67183:head# 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7c529a:::benchmark_data_smithi194_186548_object44565:head# 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7db755:::benchmark_data_smithi194_186548_object45725:head# 2024-04-29T14:53:55.798 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7e9564:::benchmark_data_smithi194_186548_object57699:head# 2024-04-29T14:53:55.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.799 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d7eab27:::benchmark_data_smithi194_186548_object50875:head# 2024-04-29T14:53:55.799 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d80b1fc:::benchmark_data_smithi194_186548_object44921:head# 2024-04-29T14:53:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d82a33b:::benchmark_data_smithi194_186548_object38779:head# 2024-04-29T14:53:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d836306:::benchmark_data_smithi194_186548_object74447:head# 2024-04-29T14:53:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d842a7c:::benchmark_data_smithi194_186548_object11486:head# 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d845397:::benchmark_data_smithi194_186548_object22338:head# 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d847bdb:::benchmark_data_smithi194_186548_object25596:head# 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8498f7:::benchmark_data_smithi194_186548_object25243:head# 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d84f688:::benchmark_data_smithi194_186548_object66378:head# 2024-04-29T14:53:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d84ffd2:::benchmark_data_smithi194_186548_object84266:head# 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d854e48:::benchmark_data_smithi194_186548_object78747:head# 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d85b845:::benchmark_data_smithi194_186548_object57986:head# 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d87062f:::benchmark_data_smithi194_186548_object61382:head# 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d872f4e:::benchmark_data_smithi194_186548_object2607:head# 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d87a5dd:::benchmark_data_smithi194_186548_object51916:head# 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d87c45f:::benchmark_data_smithi194_186548_object53649:head# 2024-04-29T14:53:55.805 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d87cdf8:::benchmark_data_smithi194_186548_object66047:head# 2024-04-29T14:53:55.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.806 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d87f4c4:::benchmark_data_smithi194_186548_object36299:head# 2024-04-29T14:53:55.806 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.807 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d882a87:::benchmark_data_smithi194_186548_object63708:head# 2024-04-29T14:53:55.807 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.811 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d89be94:::benchmark_data_smithi194_186548_object40479:head# 2024-04-29T14:53:55.811 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8a43d3:::benchmark_data_smithi194_186548_object13490:head# 2024-04-29T14:53:55.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.812 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8b2962:::benchmark_data_smithi194_186548_object56651:head# 2024-04-29T14:53:55.812 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8b9d1d:::benchmark_data_smithi194_186548_object6038:head# 2024-04-29T14:53:55.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8bc404:::benchmark_data_smithi194_186548_object80713:head# 2024-04-29T14:53:55.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.813 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8df902:::benchmark_data_smithi194_186548_object31345:head# 2024-04-29T14:53:55.813 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8e03a2:::benchmark_data_smithi194_186548_object19963:head# 2024-04-29T14:53:55.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.814 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8e2956:::benchmark_data_smithi194_186548_object41634:head# 2024-04-29T14:53:55.814 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8f011e:::benchmark_data_smithi194_186548_object26024:head# 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d8f5a1e:::benchmark_data_smithi194_186548_object82513:head# 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d90e7ed:::benchmark_data_smithi194_186548_object56662:head# 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d91b055:::benchmark_data_smithi194_186548_object80451:head# 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d91f849:::benchmark_data_smithi194_186548_object54566:head# 2024-04-29T14:53:55.815 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d941839:::benchmark_data_smithi194_186548_object40567:head# 2024-04-29T14:53:55.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9452f9:::benchmark_data_smithi194_186548_object9392:head# 2024-04-29T14:53:55.816 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.816 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d953b0e:::benchmark_data_smithi194_186548_object70084:head# 2024-04-29T14:53:55.817 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.819 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d95a257:::benchmark_data_smithi194_186548_object41648:head# 2024-04-29T14:53:55.819 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9653e2:::benchmark_data_smithi194_186548_object1265:head# 2024-04-29T14:53:55.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.820 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d965d69:::benchmark_data_smithi194_186548_object51581:head# 2024-04-29T14:53:55.820 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9698ad:::benchmark_data_smithi194_186548_object50294:head# 2024-04-29T14:53:55.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d977847:::benchmark_data_smithi194_186548_object53106:head# 2024-04-29T14:53:55.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.821 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d978e07:::benchmark_data_smithi194_186548_object53683:head# 2024-04-29T14:53:55.821 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.822 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d97c2b7:::benchmark_data_smithi194_186548_object72763:head# 2024-04-29T14:53:55.822 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.823 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d988c6e:::benchmark_data_smithi194_186548_object27606:head# 2024-04-29T14:53:55.823 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d988d9c:::benchmark_data_smithi194_186548_object65890:head# 2024-04-29T14:53:55.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.826 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d99a8f4:::benchmark_data_smithi194_186548_object50569:head# 2024-04-29T14:53:55.826 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9a117b:::benchmark_data_smithi194_186548_object40264:head# 2024-04-29T14:53:55.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9a7c4f:::benchmark_data_smithi194_186548_object62336:head# 2024-04-29T14:53:55.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9afa1e:::benchmark_data_smithi194_186548_object66438:head# 2024-04-29T14:53:55.833 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.833 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9d0628:::benchmark_data_smithi194_186548_object1370:head# 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9d0b87:::benchmark_data_smithi194_186548_object74340:head# 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9d0fb9:::benchmark_data_smithi194_186548_object84322:head# 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3d9edffc:::benchmark_data_smithi194_186548_object15936:head# 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3da0770c:::benchmark_data_smithi194_186548_object39345:head# 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3da18dc1:::benchmark_data_smithi194_186548_object21424:head# 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3da225a6:::benchmark_data_smithi194_186548_object35375:head# 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.834 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3da313f1:::benchmark_data_smithi194_186548_object15644:head# 2024-04-29T14:53:55.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3da498b5:::benchmark_data_smithi194_186548_object30595:head# 2024-04-29T14:53:55.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3da6e67e:::benchmark_data_smithi194_186548_object17292:head# 2024-04-29T14:53:55.835 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.835 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3da9f2bf:::benchmark_data_smithi194_186548_object77089:head# 2024-04-29T14:53:55.836 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.836 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3daa0a3b:::benchmark_data_smithi194_186548_object81255:head# 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dac29f3:::benchmark_data_smithi194_186548_object59199:head# 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dacbe9a:::benchmark_data_smithi194_186548_object42167:head# 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dacd6d4:::benchmark_data_smithi194_186548_object77226:head# 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dadf137:::benchmark_data_smithi194_186548_object71341:head# 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db00ff5:::benchmark_data_smithi194_186548_object53778:head# 2024-04-29T14:53:55.837 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db11024:::benchmark_data_smithi194_186548_object64164:head# 2024-04-29T14:53:55.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db143d1:::benchmark_data_smithi194_186548_object51794:head# 2024-04-29T14:53:55.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.838 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db1495d:::benchmark_data_smithi194_186548_object37202:head# 2024-04-29T14:53:55.838 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db1b120:::benchmark_data_smithi194_186548_object84279:head# 2024-04-29T14:53:55.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db28228:::benchmark_data_smithi194_186548_object67472:head# 2024-04-29T14:53:55.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.839 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db2aa5d:::benchmark_data_smithi194_186548_object58607:head# 2024-04-29T14:53:55.839 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db318c8:::benchmark_data_smithi194_186548_object21398:head# 2024-04-29T14:53:55.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db335a3:::benchmark_data_smithi194_186548_object9036:head# 2024-04-29T14:53:55.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.840 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db33c30:::benchmark_data_smithi194_186548_object8381:head# 2024-04-29T14:53:55.840 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db3b65a:::benchmark_data_smithi194_186548_object54132:head# 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db62a74:::benchmark_data_smithi194_186548_object76499:head# 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db63734:::benchmark_data_smithi194_186548_object66010:head# 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db669f2:::benchmark_data_smithi194_186548_object50868:head# 2024-04-29T14:53:55.842 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db6a411:::benchmark_data_smithi194_186548_object76511:head# 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db6cd68:::benchmark_data_smithi194_186548_object79358:head# 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db6f654:::benchmark_data_smithi194_186548_object7801:head# 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db6f9fc:::benchmark_data_smithi194_186548_object1140:head# 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db76474:::benchmark_data_smithi194_186548_object79637:head# 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db7ad78:::benchmark_data_smithi194_186548_object27987:head# 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.843 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db9308b:::benchmark_data_smithi194_186548_object82039:head# 2024-04-29T14:53:55.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.844 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db9a909:::benchmark_data_smithi194_186548_object77093:head# 2024-04-29T14:53:55.844 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3db9f724:::benchmark_data_smithi194_186548_object47677:head# 2024-04-29T14:53:55.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dbb269a:::benchmark_data_smithi194_186548_object11522:head# 2024-04-29T14:53:55.845 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.845 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dbbc14e:::benchmark_data_smithi194_186548_object19407:head# 2024-04-29T14:53:55.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.846 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dbcac92:::benchmark_data_smithi194_186548_object55599:head# 2024-04-29T14:53:55.846 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dbdc302:::benchmark_data_smithi194_186548_object50935:head# 2024-04-29T14:53:55.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.850 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dbdd87d:::benchmark_data_smithi194_186548_object37372:head# 2024-04-29T14:53:55.850 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.851 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dbe758c:::benchmark_data_smithi194_186548_object75934:head# 2024-04-29T14:53:55.851 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc12db7:::benchmark_data_smithi194_186548_object40113:head# 2024-04-29T14:53:55.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.855 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc17223:::benchmark_data_smithi194_186548_object25498:head# 2024-04-29T14:53:55.855 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.856 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc420b7:::benchmark_data_smithi194_186548_object76648:head# 2024-04-29T14:53:55.856 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc42ea2:::benchmark_data_smithi194_186548_object46250:head# 2024-04-29T14:53:55.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc4c3f7:::benchmark_data_smithi194_186548_object18363:head# 2024-04-29T14:53:55.859 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.859 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc4cc56:::benchmark_data_smithi194_186548_object12825:head# 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc61889:::benchmark_data_smithi194_186548_object74050:head# 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc6567f:::benchmark_data_smithi194_186548_object43170:head# 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc6dab6:::benchmark_data_smithi194_186548_object34139:head# 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc7a23a:::benchmark_data_smithi194_186548_object38138:head# 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc9577e:::benchmark_data_smithi194_186548_object34133:head# 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dc9ec61:::benchmark_data_smithi194_186548_object1052:head# 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.860 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dcafdb8:::benchmark_data_smithi194_186548_object54203:head# 2024-04-29T14:53:55.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.861 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dcc3f03:::benchmark_data_smithi194_186548_object38209:head# 2024-04-29T14:53:55.861 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.862 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dccd2a9:::benchmark_data_smithi194_186548_object15440:head# 2024-04-29T14:53:55.862 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dcd8165:::benchmark_data_smithi194_186548_object49176:head# 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dcf586e:::benchmark_data_smithi194_186548_object33506:head# 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dcf96d4:::benchmark_data_smithi194_186548_object78659:head# 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dcf9944:::benchmark_data_smithi194_186548_object78975:head# 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd0169a:::benchmark_data_smithi194_186548_object16262:head# 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd0e7c1:::benchmark_data_smithi194_186548_object35592:head# 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.863 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd113e6:::benchmark_data_smithi194_186548_object20160:head# 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd1f788:::benchmark_data_smithi194_186548_object74234:head# 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd3b792:::benchmark_data_smithi194_186548_object1905:head# 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd41ef6:::benchmark_data_smithi194_186548_object71420:head# 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd4fbc3:::benchmark_data_smithi194_186548_object16553:head# 2024-04-29T14:53:55.864 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd544e9:::benchmark_data_smithi194_186548_object19177:head# 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd6d3dd:::benchmark_data_smithi194_186548_object83410:head# 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd733ee:::benchmark_data_smithi194_186548_object48694:head# 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd7a727:::benchmark_data_smithi194_186548_object46494:head# 2024-04-29T14:53:55.865 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd7bf75:::benchmark_data_smithi194_186548_object38650:head# 2024-04-29T14:53:55.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd815be:::benchmark_data_smithi194_186548_object20432:head# 2024-04-29T14:53:55.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.866 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd84947:::benchmark_data_smithi194_186548_object29172:head# 2024-04-29T14:53:55.866 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.867 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dd99950:::benchmark_data_smithi194_186548_object24892:head# 2024-04-29T14:53:55.867 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ddd2fa2:::benchmark_data_smithi194_186548_object31568:head# 2024-04-29T14:53:55.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ddf3a4d:::benchmark_data_smithi194_186548_object76823:head# 2024-04-29T14:53:55.869 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.869 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de185cf:::benchmark_data_smithi194_186548_object1912:head# 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de250c6:::benchmark_data_smithi194_186548_object26891:head# 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de2ad26:::benchmark_data_smithi194_186548_object80884:head# 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de3b47b:::benchmark_data_smithi194_186548_object37522:head# 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de3c4f6:::benchmark_data_smithi194_186548_object75425:head# 2024-04-29T14:53:55.870 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de40894:::benchmark_data_smithi194_186548_object38625:head# 2024-04-29T14:53:55.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.871 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de5d8b5:::benchmark_data_smithi194_186548_object40381:head# 2024-04-29T14:53:55.871 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.872 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de7fd1d:::benchmark_data_smithi194_186548_object68141:head# 2024-04-29T14:53:55.872 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.879 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de885bb:::benchmark_data_smithi194_186548_object27568:head# 2024-04-29T14:53:55.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.880 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de8a0eb:::benchmark_data_smithi194_186548_object28294:head# 2024-04-29T14:53:55.880 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de8ca91:::benchmark_data_smithi194_186548_object79582:head# 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de964f4:::benchmark_data_smithi194_186548_object55199:head# 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de9e729:::benchmark_data_smithi194_186548_object50953:head# 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3de9ee2b:::benchmark_data_smithi194_186548_object78507:head# 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.881 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dea0975:::benchmark_data_smithi194_186548_object43229:head# 2024-04-29T14:53:55.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dea1ade:::benchmark_data_smithi194_186548_object52928:head# 2024-04-29T14:53:55.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.882 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3deb3691:::benchmark_data_smithi194_186548_object79122:head# 2024-04-29T14:53:55.882 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.883 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df114f4:::benchmark_data_smithi194_186548_object83520:head# 2024-04-29T14:53:55.883 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.885 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df1a9e8:::benchmark_data_smithi194_186548_object67694:head# 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df24606:::benchmark_data_smithi194_186548_object33663:head# 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df2a029:::benchmark_data_smithi194_186548_object56088:head# 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df38f2a:::benchmark_data_smithi194_186548_object45819:head# 2024-04-29T14:53:55.887 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.889 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df4d83f:::benchmark_data_smithi194_186548_object65740:head# 2024-04-29T14:53:55.889 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.889 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df5e562:::benchmark_data_smithi194_186548_object69525:head# 2024-04-29T14:53:55.889 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:55.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:55.893 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3df5e7bc:::benchmark_data_smithi194_186548_object78733:head# 2024-04-29T14:53:55.893 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.894 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dfac306:::benchmark_data_smithi194_186548_object9907:head# 2024-04-29T14:53:55.895 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.895 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dfc5efc:::benchmark_data_smithi194_186548_object66992:head# 2024-04-29T14:53:55.895 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.895 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dfe5e04:::benchmark_data_smithi194_186548_object21259:head# 2024-04-29T14:53:55.895 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.895 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dfea1ca:::benchmark_data_smithi194_186548_object10498:head# 2024-04-29T14:53:55.895 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.896 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3dfebb9c:::benchmark_data_smithi194_186548_object70916:head# 2024-04-29T14:53:55.896 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.896 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e0019b3:::benchmark_data_smithi194_186548_object31598:head# 2024-04-29T14:53:55.896 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.896 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e00ac01:::benchmark_data_smithi194_186548_object70210:head# 2024-04-29T14:53:55.896 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e00e38b:::benchmark_data_smithi194_186548_object27477:head# 2024-04-29T14:53:55.897 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e015d9a:::benchmark_data_smithi194_186548_object81800:head# 2024-04-29T14:53:55.897 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.897 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e03367a:::benchmark_data_smithi194_186548_object70657:head# 2024-04-29T14:53:55.898 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.898 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e033bab:::benchmark_data_smithi194_186548_object61989:head# 2024-04-29T14:53:55.898 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.898 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e04f756:::benchmark_data_smithi194_186548_object53830:head# 2024-04-29T14:53:55.898 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e056917:::benchmark_data_smithi194_186548_object16485:head# 2024-04-29T14:53:55.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e0686d3:::benchmark_data_smithi194_186548_object4125:head# 2024-04-29T14:53:55.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.899 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e08a827:::benchmark_data_smithi194_186548_object48806:head# 2024-04-29T14:53:55.899 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e097e38:::benchmark_data_smithi194_186548_object15682:head# 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e09d078:::benchmark_data_smithi194_186548_object50412:head# 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e0a68d2:::benchmark_data_smithi194_186548_object61969:head# 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e0c979b:::benchmark_data_smithi194_186548_object47881:head# 2024-04-29T14:53:55.900 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.901 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e0d4456:::benchmark_data_smithi194_186548_object53952:head# 2024-04-29T14:53:55.901 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e0e28f7:::benchmark_data_smithi194_186548_object36310:head# 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e0f5ffd:::benchmark_data_smithi194_186548_object19643:head# 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e112b9f:::benchmark_data_smithi194_186548_object79114:head# 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e123a32:::benchmark_data_smithi194_186548_object64062:head# 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e12a72a:::benchmark_data_smithi194_186548_object52278:head# 2024-04-29T14:53:55.902 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.903 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e12cd7c:::benchmark_data_smithi194_186548_object30544:head# 2024-04-29T14:53:55.903 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.903 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e132474:::benchmark_data_smithi194_186548_object9403:head# 2024-04-29T14:53:55.903 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.904 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e137aee:::benchmark_data_smithi194_186548_object37499:head# 2024-04-29T14:53:55.904 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.904 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e13bd59:::benchmark_data_smithi194_186548_object54096:head# 2024-04-29T14:53:55.904 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.904 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e13eee7:::benchmark_data_smithi194_186548_object73672:head# 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1517c8:::benchmark_data_smithi194_186548_object78706:head# 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e16132c:::benchmark_data_smithi194_186548_object2484:head# 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e17c69e:::benchmark_data_smithi194_186548_object63677:head# 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e180ae7:::benchmark_data_smithi194_186548_object61443:head# 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e18b22c:::benchmark_data_smithi194_186548_object75141:head# 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.905 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e198ffe:::benchmark_data_smithi194_186548_object47122:head# 2024-04-29T14:53:55.906 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1a0f7e:::benchmark_data_smithi194_186548_object9427:head# 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1a0fa7:::benchmark_data_smithi194_186548_object15289:head# 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1b83bf:::benchmark_data_smithi194_186548_object20369:head# 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1bf1fb:::benchmark_data_smithi194_186548_object68998:head# 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.907 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1c1559:::benchmark_data_smithi194_186548_object8923:head# 2024-04-29T14:53:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1ccf2c:::benchmark_data_smithi194_186548_object61484:head# 2024-04-29T14:53:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1d4667:::benchmark_data_smithi194_186548_object4343:head# 2024-04-29T14:53:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e1f5723:::benchmark_data_smithi194_186548_object4517:head# 2024-04-29T14:53:55.908 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.909 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e20ca21:::benchmark_data_smithi194_186548_object23935:head# 2024-04-29T14:53:55.909 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.909 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e212fbf:::benchmark_data_smithi194_186548_object60234:head# 2024-04-29T14:53:55.909 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.909 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e218bde:::benchmark_data_smithi194_186548_object15751:head# 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e21aee1:::benchmark_data_smithi194_186548_object80396:head# 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e21ecfc:::benchmark_data_smithi194_186548_object26453:head# 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2203bb:::benchmark_data_smithi194_186548_object17750:head# 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e224159:::benchmark_data_smithi194_186548_object63221:head# 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e22d1b4:::benchmark_data_smithi194_186548_object51913:head# 2024-04-29T14:53:55.910 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.911 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e22d269:::benchmark_data_smithi194_186548_object22663:head# 2024-04-29T14:53:55.911 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.911 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e23a2a6:::benchmark_data_smithi194_186548_object11982:head# 2024-04-29T14:53:55.911 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.912 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e259b47:::benchmark_data_smithi194_186548_object80853:head# 2024-04-29T14:53:55.912 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.912 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e25dec7:::benchmark_data_smithi194_186548_object39508:head# 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2771bd:::benchmark_data_smithi194_186548_object45290:head# 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e28d32b:::benchmark_data_smithi194_186548_object27545:head# 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e28dab9:::benchmark_data_smithi194_186548_object36818:head# 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e28db7b:::benchmark_data_smithi194_186548_object78904:head# 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e293438:::benchmark_data_smithi194_186548_object67562:head# 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.913 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2a8eff:::benchmark_data_smithi194_186548_object77855:head# 2024-04-29T14:53:55.914 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.914 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2c1f1c:::benchmark_data_smithi194_186548_object33340:head# 2024-04-29T14:53:55.914 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.914 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2cac11:::benchmark_data_smithi194_186548_object17846:head# 2024-04-29T14:53:55.914 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.914 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2cd2c7:::benchmark_data_smithi194_186548_object27800:head# 2024-04-29T14:53:55.914 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.915 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2d8b36:::benchmark_data_smithi194_186548_object82281:head# 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2de6e6:::benchmark_data_smithi194_186548_object60115:head# 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2e028d:::benchmark_data_smithi194_186548_object70633:head# 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e2e5d80:::benchmark_data_smithi194_186548_object52192:head# 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e30f329:::benchmark_data_smithi194_186548_object63314:head# 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.916 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e323688:::benchmark_data_smithi194_186548_object84578:head# 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e32aceb:::benchmark_data_smithi194_186548_object64178:head# 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e33003b:::benchmark_data_smithi194_186548_object26278:head# 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3342fd:::benchmark_data_smithi194_186548_object42525:head# 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e336332:::benchmark_data_smithi194_186548_object29576:head# 2024-04-29T14:53:55.917 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e336592:::benchmark_data_smithi194_186548_object56959:head# 2024-04-29T14:53:55.919 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.919 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e33abae:::benchmark_data_smithi194_186548_object71844:head# 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e33e36a:::benchmark_data_smithi194_186548_object50029:head# 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e340d12:::benchmark_data_smithi194_186548_object52021:head# 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3414c6:::benchmark_data_smithi194_186548_object55450:head# 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e35513f:::benchmark_data_smithi194_186548_object985:head# 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e364d23:::benchmark_data_smithi194_186548_object50050:head# 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.920 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e37c0b9:::benchmark_data_smithi194_186548_object10134:head# 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e37cf64:::benchmark_data_smithi194_186548_object50571:head# 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e38b0ef:::benchmark_data_smithi194_186548_object51799:head# 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e390757:::benchmark_data_smithi194_186548_object31513:head# 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3a1382:::benchmark_data_smithi194_186548_object74938:head# 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3aeb97:::benchmark_data_smithi194_186548_object3846:head# 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3bef96:::benchmark_data_smithi194_186548_object656:head# 2024-04-29T14:53:55.921 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3d6988:::benchmark_data_smithi194_186548_object57663:head# 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3e6b33:::benchmark_data_smithi194_186548_object70920:head# 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3f4baf:::benchmark_data_smithi194_186548_object21689:head# 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3f8251:::benchmark_data_smithi194_186548_object28238:head# 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3f85c1:::benchmark_data_smithi194_186548_object8157:head# 2024-04-29T14:53:55.924 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e3fdf26:::benchmark_data_smithi194_186548_object69062:head# 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e417d06:::benchmark_data_smithi194_186548_object4323:head# 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e423cc1:::benchmark_data_smithi194_186548_object23135:head# 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e42b9a7:::benchmark_data_smithi194_186548_object51756:head# 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e437f33:::benchmark_data_smithi194_186548_object8805:head# 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e444144:::benchmark_data_smithi194_186548_object33824:head# 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e44e9de:::benchmark_data_smithi194_186548_object58619:head# 2024-04-29T14:53:55.925 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e46404e:::benchmark_data_smithi194_186548_object13226:head# 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e48d246:::benchmark_data_smithi194_186548_object25817:head# 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e4926d7:::benchmark_data_smithi194_186548_object14093:head# 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e49e8cf:::benchmark_data_smithi194_186548_object83264:head# 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e4b0028:::benchmark_data_smithi194_186548_object24469:head# 2024-04-29T14:53:55.926 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e4b97d5:::benchmark_data_smithi194_186548_object73339:head# 2024-04-29T14:53:55.929 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.929 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e4df5e5:::benchmark_data_smithi194_186548_object25789:head# 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e4fc58f:::benchmark_data_smithi194_186548_object916:head# 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e51e9df:::benchmark_data_smithi194_186548_object64526:head# 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e53402a:::benchmark_data_smithi194_186548_object70285:head# 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e53c3bc:::benchmark_data_smithi194_186548_object79719:head# 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e53ce67:::benchmark_data_smithi194_186548_object3033:head# 2024-04-29T14:53:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e54a8ce:::benchmark_data_smithi194_186548_object8740:head# 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e551924:::benchmark_data_smithi194_186548_object19635:head# 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e55d1f2:::benchmark_data_smithi194_186548_object18625:head# 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e568323:::benchmark_data_smithi194_186548_object24127:head# 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e56dfa1:::benchmark_data_smithi194_186548_object46345:head# 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e571a5f:::benchmark_data_smithi194_186548_object38976:head# 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.931 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5722e6:::benchmark_data_smithi194_186548_object13548:head# 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e573465:::benchmark_data_smithi194_186548_object45004:head# 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e57c409:::benchmark_data_smithi194_186548_object10960:head# 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e584da0:::benchmark_data_smithi194_186548_object50891:head# 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e59dc1f:::benchmark_data_smithi194_186548_object56393:head# 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e59e213:::benchmark_data_smithi194_186548_object5710:head# 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.932 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e59f068:::benchmark_data_smithi194_186548_object1107:head# 2024-04-29T14:53:55.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.933 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5a3297:::benchmark_data_smithi194_186548_object16256:head# 2024-04-29T14:53:55.933 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5a74f5:::benchmark_data_smithi194_186548_object21807:head# 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5c2b21:::benchmark_data_smithi194_186548_object24705:head# 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5dade6:::benchmark_data_smithi194_186548_object60685:head# 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5e3cd5:::benchmark_data_smithi194_186548_object16324:head# 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5e9cc0:::benchmark_data_smithi194_186548_object1866:head# 2024-04-29T14:53:55.936 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e5f0aa2:::benchmark_data_smithi194_186548_object56450:head# 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e611085:::benchmark_data_smithi194_186548_object83384:head# 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e616daa:::benchmark_data_smithi194_186548_object30694:head# 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e652d6b:::benchmark_data_smithi194_186548_object32744:head# 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6613e1:::benchmark_data_smithi194_186548_object4803:head# 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.937 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e66a239:::benchmark_data_smithi194_186548_object46911:head# 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e66c288:::benchmark_data_smithi194_186548_object57433:head# 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e679062:::benchmark_data_smithi194_186548_object7055:head# 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e68515b:::benchmark_data_smithi194_186548_object24594:head# 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e685a39:::benchmark_data_smithi194_186548_object62315:head# 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6991e3:::benchmark_data_smithi194_186548_object33575:head# 2024-04-29T14:53:55.938 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6a9e7d:::benchmark_data_smithi194_186548_object78592:head# 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6b1bd5:::benchmark_data_smithi194_186548_object33621:head# 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6cac27:::benchmark_data_smithi194_186548_object59298:head# 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6e0c0c:::benchmark_data_smithi194_186548_object85010:head# 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6e61ec:::benchmark_data_smithi194_186548_object75798:head# 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6ea296:::benchmark_data_smithi194_186548_object41334:head# 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.939 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6ec53b:::benchmark_data_smithi194_186548_object11102:head# 2024-04-29T14:53:55.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e6f681a:::benchmark_data_smithi194_186548_object82501:head# 2024-04-29T14:53:55.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.940 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e702b21:::benchmark_data_smithi194_186548_object78034:head# 2024-04-29T14:53:55.940 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e705ce4:::benchmark_data_smithi194_186548_object71658:head# 2024-04-29T14:53:55.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e715e94:::benchmark_data_smithi194_186548_object25370:head# 2024-04-29T14:53:55.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.943 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e735244:::benchmark_data_smithi194_186548_object80337:head# 2024-04-29T14:53:55.943 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e73ee7e:::benchmark_data_smithi194_186548_object78446:head# 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e74a4e0:::benchmark_data_smithi194_186548_object6368:head# 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e74ed61:::benchmark_data_smithi194_186548_object48699:head# 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e76d890:::benchmark_data_smithi194_186548_object11966:head# 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e7ad6e1:::benchmark_data_smithi194_186548_object1111:head# 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.944 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e7e45ad:::benchmark_data_smithi194_186548_object51073:head# 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e7e8e1d:::benchmark_data_smithi194_186548_object55549:head# 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e7f1b12:::benchmark_data_smithi194_186548_object72924:head# 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e7fb040:::benchmark_data_smithi194_186548_object33263:head# 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e801e34:::benchmark_data_smithi194_186548_object29752:head# 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e811e64:::benchmark_data_smithi194_186548_object35025:head# 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e822f66:::benchmark_data_smithi194_186548_object17982:head# 2024-04-29T14:53:55.945 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e82b56a:::benchmark_data_smithi194_186548_object51499:head# 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8409d0:::benchmark_data_smithi194_186548_object25237:head# 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e84a551:::benchmark_data_smithi194_186548_object10784:head# 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e854192:::benchmark_data_smithi194_186548_object46887:head# 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e85b3c5:::benchmark_data_smithi194_186548_object2473:head# 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.946 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e860cfb:::benchmark_data_smithi194_186548_object1070:head# 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e87300e:::benchmark_data_smithi194_186548_object3137:head# 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e879c39:::benchmark_data_smithi194_186548_object41006:head# 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e87a27b:::benchmark_data_smithi194_186548_object27935:head# 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e88292e:::benchmark_data_smithi194_186548_object56342:head# 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e88f15a:::benchmark_data_smithi194_186548_object25436:head# 2024-04-29T14:53:55.947 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8979c4:::benchmark_data_smithi194_186548_object75392:head# 2024-04-29T14:53:55.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.948 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e89bfa9:::benchmark_data_smithi194_186548_object67637:head# 2024-04-29T14:53:55.948 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e89e260:::benchmark_data_smithi194_186548_object51911:head# 2024-04-29T14:53:55.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e89f8db:::benchmark_data_smithi194_186548_object36006:head# 2024-04-29T14:53:55.951 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.951 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8a7654:::benchmark_data_smithi194_186548_object46088:head# 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8b6c0b:::benchmark_data_smithi194_186548_object36822:head# 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8b7713:::benchmark_data_smithi194_186548_object36936:head# 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8b77b8:::benchmark_data_smithi194_186548_object48700:head# 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8c85de:::benchmark_data_smithi194_186548_object58286:head# 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8cff36:::benchmark_data_smithi194_186548_object26864:head# 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8d6af8:::benchmark_data_smithi194_186548_object48972:head# 2024-04-29T14:53:55.952 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8edb01:::benchmark_data_smithi194_186548_object20780:head# 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e8fa372:::benchmark_data_smithi194_186548_object43364:head# 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e90ce5b:::benchmark_data_smithi194_186548_object22501:head# 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e913a6d:::benchmark_data_smithi194_186548_object44832:head# 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e919d05:::benchmark_data_smithi194_186548_object71060:head# 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9218bd:::benchmark_data_smithi194_186548_object72279:head# 2024-04-29T14:53:55.953 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e92617b:::benchmark_data_smithi194_186548_object21406:head# 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e92934b:::benchmark_data_smithi194_186548_object16622:head# 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9353c8:::benchmark_data_smithi194_186548_object44085:head# 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e940eff:::benchmark_data_smithi194_186548_object69827:head# 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9430a6:::benchmark_data_smithi194_186548_object56989:head# 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9460bc:::benchmark_data_smithi194_186548_object24759:head# 2024-04-29T14:53:55.954 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e94d324:::benchmark_data_smithi194_186548_object7914:head# 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e95212a:::benchmark_data_smithi194_186548_object49490:head# 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e956eeb:::benchmark_data_smithi194_186548_object48526:head# 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e95df57:::benchmark_data_smithi194_186548_object56535:head# 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9628d8:::benchmark_data_smithi194_186548_object54617:head# 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e96ae4b:::benchmark_data_smithi194_186548_object36968:head# 2024-04-29T14:53:55.955 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e98075a:::benchmark_data_smithi194_186548_object82056:head# 2024-04-29T14:53:55.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.956 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e984813:::benchmark_data_smithi194_186548_object48796:head# 2024-04-29T14:53:55.956 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.959 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9885ef:::benchmark_data_smithi194_186548_object15234:head# 2024-04-29T14:53:55.959 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e99a10b:::benchmark_data_smithi194_186548_object82719:head# 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9b2d19:::benchmark_data_smithi194_186548_object48820:head# 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9b791d:::benchmark_data_smithi194_186548_object3331:head# 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9d5486:::benchmark_data_smithi194_186548_object12494:head# 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9d5782:::benchmark_data_smithi194_186548_object44070:head# 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9d8dbb:::benchmark_data_smithi194_186548_object40500:head# 2024-04-29T14:53:55.960 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9d8fc2:::benchmark_data_smithi194_186548_object56029:head# 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9e52f4:::benchmark_data_smithi194_186548_object51467:head# 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9e8211:::benchmark_data_smithi194_186548_object74448:head# 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9ea893:::benchmark_data_smithi194_186548_object56828:head# 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9f3a52:::benchmark_data_smithi194_186548_object37877:head# 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3e9ff69b:::benchmark_data_smithi194_186548_object61990:head# 2024-04-29T14:53:55.961 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea167a4:::benchmark_data_smithi194_186548_object57517:head# 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea19146:::benchmark_data_smithi194_186548_object74658:head# 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea24e6d:::benchmark_data_smithi194_186548_object71674:head# 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea26007:::benchmark_data_smithi194_186548_object12876:head# 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea36ba2:::benchmark_data_smithi194_186548_object80757:head# 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea52fe4:::benchmark_data_smithi194_186548_object23698:head# 2024-04-29T14:53:55.962 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea56b71:::benchmark_data_smithi194_186548_object85251:head# 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea625c3:::benchmark_data_smithi194_186548_object42387:head# 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ea8acc6:::benchmark_data_smithi194_186548_object33955:head# 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eaad17f:::benchmark_data_smithi194_186548_object35414:head# 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eaae1c2:::benchmark_data_smithi194_186548_object21777:head# 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eac2cd5:::benchmark_data_smithi194_186548_object25070:head# 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.963 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ead09d7:::benchmark_data_smithi194_186548_object3280:head# 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ead22a4:::benchmark_data_smithi194_186548_object58820:head# 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eadba15:::benchmark_data_smithi194_186548_object4751:head# 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eaf6165:::benchmark_data_smithi194_186548_object30767:head# 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb093b5:::benchmark_data_smithi194_186548_object68666:head# 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb2afcc:::benchmark_data_smithi194_186548_object12508:head# 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.964 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb36d06:::benchmark_data_smithi194_186548_object79055:head# 2024-04-29T14:53:55.965 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb3d160:::benchmark_data_smithi194_186548_object82563:head# 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb44902:::benchmark_data_smithi194_186548_object31489:head# 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb47abe:::benchmark_data_smithi194_186548_object28559:head# 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb61281:::benchmark_data_smithi194_186548_object14553:head# 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb6b5ca:::benchmark_data_smithi194_186548_object63239:head# 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb71ea8:::benchmark_data_smithi194_186548_object35903:head# 2024-04-29T14:53:55.969 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb776e4:::benchmark_data_smithi194_186548_object41664:head# 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eb9f8f8:::benchmark_data_smithi194_186548_object68278:head# 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eba1de1:::benchmark_data_smithi194_186548_object16431:head# 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eba3657:::benchmark_data_smithi194_186548_object80035:head# 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eba9d43:::benchmark_data_smithi194_186548_object68020:head# 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebbdbfd:::benchmark_data_smithi194_186548_object71378:head# 2024-04-29T14:53:55.970 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebd16da:::benchmark_data_smithi194_186548_object46867:head# 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebd5a9d:::benchmark_data_smithi194_186548_object25637:head# 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebe2ad3:::benchmark_data_smithi194_186548_object3590:head# 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebe5f5d:::benchmark_data_smithi194_186548_object44966:head# 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebe936d:::benchmark_data_smithi194_186548_object48113:head# 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebf67db:::benchmark_data_smithi194_186548_object18977:head# 2024-04-29T14:53:55.971 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebf7dbc:::benchmark_data_smithi194_186548_object73459:head# 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebf89bc:::benchmark_data_smithi194_186548_object68280:head# 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ebf9452:::benchmark_data_smithi194_186548_object62936:head# 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec0537b:::benchmark_data_smithi194_186548_object47312:head# 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec12560:::benchmark_data_smithi194_186548_object7103:head# 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec126a0:::benchmark_data_smithi194_186548_object43688:head# 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.972 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec1730a:::benchmark_data_smithi194_186548_object27742:head# 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec35c97:::benchmark_data_smithi194_186548_object69389:head# 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec3836d:::benchmark_data_smithi194_186548_object21491:head# 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec38562:::benchmark_data_smithi194_186548_object64814:head# 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec4fefb:::benchmark_data_smithi194_186548_object85220:head# 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec52a9f:::benchmark_data_smithi194_186548_object30843:head# 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.973 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec57dc0:::benchmark_data_smithi194_186548_object1991:head# 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec60c4d:::benchmark_data_smithi194_186548_object3315:head# 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec6224c:::benchmark_data_smithi194_186548_object264:head# 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec665f4:::benchmark_data_smithi194_186548_object66281:head# 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec6e5b4:::benchmark_data_smithi194_186548_object26472:head# 2024-04-29T14:53:55.974 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.978 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec93d98:::benchmark_data_smithi194_186548_object21835:head# 2024-04-29T14:53:55.978 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec97c8f:::benchmark_data_smithi194_186548_object77569:head# 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ec9bf4f:::benchmark_data_smithi194_186548_object84274:head# 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eca8d6d:::benchmark_data_smithi194_186548_object65055:head# 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ecb7bd2:::benchmark_data_smithi194_186548_object11600:head# 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ecc2534:::benchmark_data_smithi194_186548_object25567:head# 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed00674:::benchmark_data_smithi194_186548_object73371:head# 2024-04-29T14:53:55.979 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed093dc:::benchmark_data_smithi194_186548_object28422:head# 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed09eda:::benchmark_data_smithi194_186548_object49785:head# 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed0aa52:::benchmark_data_smithi194_186548_object2797:head# 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed0cd8a:::benchmark_data_smithi194_186548_object59601:head# 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed100ed:::benchmark_data_smithi194_186548_object27123:head# 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed2115c:::benchmark_data_smithi194_186548_object12689:head# 2024-04-29T14:53:55.980 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed24290:::benchmark_data_smithi194_186548_object68743:head# 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed350cf:::benchmark_data_smithi194_186548_object71566:head# 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed3762e:::benchmark_data_smithi194_186548_object69267:head# 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed63b05:::benchmark_data_smithi194_186548_object77366:head# 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed79e40:::benchmark_data_smithi194_186548_object47620:head# 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed8a14d:::benchmark_data_smithi194_186548_object69569:head# 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ed8dca7:::benchmark_data_smithi194_186548_object80325:head# 2024-04-29T14:53:55.981 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eda3386:::benchmark_data_smithi194_186548_object33990:head# 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eda6686:::benchmark_data_smithi194_186548_object74341:head# 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3edda7c5:::benchmark_data_smithi194_186548_object21197:head# 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee08845:::benchmark_data_smithi194_186548_object56719:head# 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee15538:::benchmark_data_smithi194_186548_object40137:head# 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee17732:::benchmark_data_smithi194_186548_object40670:head# 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee1ae43:::benchmark_data_smithi194_186548_object58006:head# 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee257a2:::benchmark_data_smithi194_186548_object17528:head# 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee29dc5:::benchmark_data_smithi194_186548_object80436:head# 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee37607:::benchmark_data_smithi194_186548_object69376:head# 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee3e38d:::benchmark_data_smithi194_186548_object23101:head# 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee4ea4b:::benchmark_data_smithi194_186548_object49478:head# 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee53afd:::benchmark_data_smithi194_186548_object45013:head# 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee59d1a:::benchmark_data_smithi194_186548_object19649:head# 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee680bf:::benchmark_data_smithi194_186548_object45275:head# 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee6c48b:::benchmark_data_smithi194_186548_object40791:head# 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee6e471:::benchmark_data_smithi194_186548_object35029:head# 2024-04-29T14:53:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee75817:::benchmark_data_smithi194_186548_object17841:head# 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee79d8b:::benchmark_data_smithi194_186548_object45268:head# 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee7ab0d:::benchmark_data_smithi194_186548_object37842:head# 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee7d74c:::benchmark_data_smithi194_186548_object45398:head# 2024-04-29T14:53:55.988 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee81516:::benchmark_data_smithi194_186548_object61520:head# 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ee85721:::benchmark_data_smithi194_186548_object40450:head# 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eeb07a0:::benchmark_data_smithi194_186548_object30900:head# 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eeb37d2:::benchmark_data_smithi194_186548_object23168:head# 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eebd401:::benchmark_data_smithi194_186548_object24607:head# 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eebf4f6:::benchmark_data_smithi194_186548_object35980:head# 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.989 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eec0b51:::benchmark_data_smithi194_186548_object72797:head# 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eec1bfc:::benchmark_data_smithi194_186548_object80663:head# 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eecb75f:::benchmark_data_smithi194_186548_object80408:head# 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eed5d99:::benchmark_data_smithi194_186548_object10727:head# 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eedf048:::benchmark_data_smithi194_186548_object53359:head# 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eefa70b:::benchmark_data_smithi194_186548_object39099:head# 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.990 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef0d0a5:::benchmark_data_smithi194_186548_object15005:head# 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef0e16b:::benchmark_data_smithi194_186548_object69875:head# 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef1177f:::benchmark_data_smithi194_186548_object1922:head# 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef146e4:::benchmark_data_smithi194_186548_object58296:head# 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef2da05:::benchmark_data_smithi194_186548_object5673:head# 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef30411:::benchmark_data_smithi194_186548_object58953:head# 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef3176d:::benchmark_data_smithi194_186548_object4244:head# 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.991 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef482c0:::benchmark_data_smithi194_186548_object43337:head# 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef5b278:::benchmark_data_smithi194_186548_object3516:head# 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef739b5:::benchmark_data_smithi194_186548_object3247:head# 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef7b923:::benchmark_data_smithi194_186548_object54447:head# 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef801dd:::benchmark_data_smithi194_186548_object77553:head# 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef8a54c:::benchmark_data_smithi194_186548_object2503:head# 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.992 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef94f7a:::benchmark_data_smithi194_186548_object72496:head# 2024-04-29T14:53:55.993 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef96d71:::benchmark_data_smithi194_186548_object82687:head# 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ef9daab:::benchmark_data_smithi194_186548_object41932:head# 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efa7e54:::benchmark_data_smithi194_186548_object35107:head# 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efb8f83:::benchmark_data_smithi194_186548_object15341:head# 2024-04-29T14:53:55.994 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efba6a7:::benchmark_data_smithi194_186548_object38641:head# 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efbb971:::benchmark_data_smithi194_186548_object71636:head# 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efdb975:::benchmark_data_smithi194_186548_object83427:head# 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efdd8f7:::benchmark_data_smithi194_186548_object15580:head# 2024-04-29T14:53:55.995 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.995 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:55.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efe3104:::benchmark_data_smithi194_186548_object81796:head# 2024-04-29T14:53:55.997 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efe606d:::benchmark_data_smithi194_186548_object11072:head# 2024-04-29T14:53:55.997 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3efed7b5:::benchmark_data_smithi194_186548_object8824:head# 2024-04-29T14:53:55.997 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.997 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3eff3ff4:::benchmark_data_smithi194_186548_object34099:head# 2024-04-29T14:53:55.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.998 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f004d77:::benchmark_data_smithi194_186548_object26009:head# 2024-04-29T14:53:55.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.998 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f00c599:::benchmark_data_smithi194_186548_object18106:head# 2024-04-29T14:53:55.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:55.998 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f01d99f:::benchmark_data_smithi194_186548_object30741:head# 2024-04-29T14:53:55.998 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.003 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0232f4:::benchmark_data_smithi194_186548_object74907:head# 2024-04-29T14:53:56.003 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.005 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0295e1:::benchmark_data_smithi194_186548_object57549:head# 2024-04-29T14:53:56.005 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.008 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f034e67:::benchmark_data_smithi194_186548_object19083:head# 2024-04-29T14:53:56.008 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.011 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f03639c:::benchmark_data_smithi194_186548_object70283:head# 2024-04-29T14:53:56.011 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.015 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f03b436:::benchmark_data_smithi194_186548_object8495:head# 2024-04-29T14:53:56.015 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.015 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f04008a:::benchmark_data_smithi194_186548_object58684:head# 2024-04-29T14:53:56.015 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.015 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f064177:::benchmark_data_smithi194_186548_object71869:head# 2024-04-29T14:53:56.015 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f08c8ef:::benchmark_data_smithi194_186548_object76264:head# 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f08fa1b:::benchmark_data_smithi194_186548_object55460:head# 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f09061a:::benchmark_data_smithi194_186548_object27801:head# 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f09767f:::benchmark_data_smithi194_186548_object15479:head# 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0979e8:::benchmark_data_smithi194_186548_object20440:head# 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f099ba4:::benchmark_data_smithi194_186548_object72460:head# 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.016 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f09d412:::benchmark_data_smithi194_186548_object22185:head# 2024-04-29T14:53:56.017 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.017 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0a7434:::benchmark_data_smithi194_186548_object10229:head# 2024-04-29T14:53:56.017 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.017 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0ae932:::benchmark_data_smithi194_186548_object65192:head# 2024-04-29T14:53:56.017 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.017 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0bd962:::benchmark_data_smithi194_186548_object80331:head# 2024-04-29T14:53:56.017 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.018 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0be93f:::benchmark_data_smithi194_186548_object82504:head# 2024-04-29T14:53:56.018 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.018 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0d6a69:::benchmark_data_smithi194_186548_object28253:head# 2024-04-29T14:53:56.018 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.018 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0ee168:::benchmark_data_smithi194_186548_object41975:head# 2024-04-29T14:53:56.018 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.019 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0f12af:::benchmark_data_smithi194_186548_object19371:head# 2024-04-29T14:53:56.019 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.019 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f0f697f:::benchmark_data_smithi194_186548_object32400:head# 2024-04-29T14:53:56.019 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f111122:::benchmark_data_smithi194_186548_object26945:head# 2024-04-29T14:53:56.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1147e7:::benchmark_data_smithi194_186548_object11505:head# 2024-04-29T14:53:56.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.020 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f12298e:::benchmark_data_smithi194_186548_object44779:head# 2024-04-29T14:53:56.020 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f123729:::benchmark_data_smithi194_186548_object35057:head# 2024-04-29T14:53:56.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f12e5c1:::benchmark_data_smithi194_186548_object51236:head# 2024-04-29T14:53:56.021 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.021 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f12f6e3:::benchmark_data_smithi194_186548_object8076:head# 2024-04-29T14:53:56.022 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f136604:::benchmark_data_smithi194_186548_object5491:head# 2024-04-29T14:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f156098:::benchmark_data_smithi194_186548_object69887:head# 2024-04-29T14:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.030 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1562d6:::benchmark_data_smithi194_186548_object17769:head# 2024-04-29T14:53:56.030 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.030 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f16a888:::benchmark_data_smithi194_186548_object32731:head# 2024-04-29T14:53:56.030 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.031 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1752b6:::benchmark_data_smithi194_186548_object3092:head# 2024-04-29T14:53:56.031 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.031 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f17ca23:::benchmark_data_smithi194_186548_object29384:head# 2024-04-29T14:53:56.031 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.031 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f186644:::benchmark_data_smithi194_186548_object68697:head# 2024-04-29T14:53:56.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f189833:::benchmark_data_smithi194_186548_object83140:head# 2024-04-29T14:53:56.032 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.032 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1910b1:::benchmark_data_smithi194_186548_object66743:head# 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1a5a48:::benchmark_data_smithi194_186548_object64990:head# 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1ad567:::benchmark_data_smithi194_186548_object28215:head# 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1dbfb6:::benchmark_data_smithi194_186548_object24213:head# 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f1df956:::benchmark_data_smithi194_186548_object30446:head# 2024-04-29T14:53:56.033 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.034 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f2047f3:::benchmark_data_smithi194_186548_object4319:head# 2024-04-29T14:53:56.034 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.034 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f209f4c:::benchmark_data_smithi194_186548_object43281:head# 2024-04-29T14:53:56.034 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.034 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f21cff7:::benchmark_data_smithi194_186548_object5531:head# 2024-04-29T14:53:56.035 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.035 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f23ea31:::benchmark_data_smithi194_186548_object16489:head# 2024-04-29T14:53:56.035 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.035 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f23fa08:::benchmark_data_smithi194_186548_object61775:head# 2024-04-29T14:53:56.035 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.039 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f24a521:::benchmark_data_smithi194_186548_object78594:head# 2024-04-29T14:53:56.039 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.042 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f2587a3:::benchmark_data_smithi194_186548_object74789:head# 2024-04-29T14:53:56.043 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.043 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f25e231:::benchmark_data_smithi194_186548_object21645:head# 2024-04-29T14:53:56.043 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.043 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f2617d2:::benchmark_data_smithi194_186548_object66372:head# 2024-04-29T14:53:56.044 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f267fbf:::benchmark_data_smithi194_186548_object62794:head# 2024-04-29T14:53:56.044 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.044 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f280e71:::benchmark_data_smithi194_186548_object36915:head# 2024-04-29T14:53:56.045 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.045 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f284d32:::benchmark_data_smithi194_186548_object20139:head# 2024-04-29T14:53:56.045 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.046 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:56.046 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f28b875:::benchmark_data_smithi194_186548_object2511:head# 2024-04-29T14:53:56.046 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f2c6442:::benchmark_data_smithi194_186548_object12407:head# 2024-04-29T14:53:56.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.052 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f2f0098:::benchmark_data_smithi194_186548_object1593:head# 2024-04-29T14:53:56.052 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.064 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f2f82a5:::benchmark_data_smithi194_186548_object21529:head# 2024-04-29T14:53:56.064 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.066 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f2f92f6:::benchmark_data_smithi194_186548_object76810:head# 2024-04-29T14:53:56.066 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.066 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f300176:::benchmark_data_smithi194_186548_object67093:head# 2024-04-29T14:53:56.067 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.067 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f300697:::benchmark_data_smithi194_186548_object14089:head# 2024-04-29T14:53:56.067 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.067 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f31270a:::benchmark_data_smithi194_186548_object28937:head# 2024-04-29T14:53:56.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f325fb6:::benchmark_data_smithi194_186548_object32431:head# 2024-04-29T14:53:56.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f33a421:::benchmark_data_smithi194_186548_object77990:head# 2024-04-29T14:53:56.068 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.068 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f3408da:::benchmark_data_smithi194_186548_object28991:head# 2024-04-29T14:53:56.069 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.071 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f354d5c:::benchmark_data_smithi194_186548_object58045:head# 2024-04-29T14:53:56.071 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.077 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f36d2ad:::benchmark_data_smithi194_186548_object59548:head# 2024-04-29T14:53:56.077 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.078 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f37e53f:::benchmark_data_smithi194_186548_object73195:head# 2024-04-29T14:53:56.078 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.079 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f37eee6:::benchmark_data_smithi194_186548_object59711:head# 2024-04-29T14:53:56.080 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.081 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f37f1bb:::benchmark_data_smithi194_186548_object60918:head# 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f38f0b6:::benchmark_data_smithi194_186548_object5947:head# 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f392ec0:::benchmark_data_smithi194_186548_object77760:head# 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f3c4b1a:::benchmark_data_smithi194_186548_object53460:head# 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f3d2690:::benchmark_data_smithi194_186548_object63814:head# 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f3e301d:::benchmark_data_smithi194_186548_object8720:head# 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.082 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f3f86f9:::benchmark_data_smithi194_186548_object3854:head# 2024-04-29T14:53:56.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.083 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f407909:::benchmark_data_smithi194_186548_object36123:head# 2024-04-29T14:53:56.083 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f40e080:::benchmark_data_smithi194_186548_object54642:head# 2024-04-29T14:53:56.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4179ae:::benchmark_data_smithi194_186548_object19230:head# 2024-04-29T14:53:56.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.084 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f42915f:::benchmark_data_smithi194_186548_object8380:head# 2024-04-29T14:53:56.084 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f44a77c:::benchmark_data_smithi194_186548_object33706:head# 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f44b18c:::benchmark_data_smithi194_186548_object2646:head# 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f44f2d9:::benchmark_data_smithi194_186548_object44776:head# 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f46bdcc:::benchmark_data_smithi194_186548_object66750:head# 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f470b2f:::benchmark_data_smithi194_186548_object69965:head# 2024-04-29T14:53:56.091 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f47813d:::benchmark_data_smithi194_186548_object36882:head# 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f48612f:::benchmark_data_smithi194_186548_object28299:head# 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f48b930:::benchmark_data_smithi194_186548_object34534:head# 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f49f210:::benchmark_data_smithi194_186548_object55235:head# 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4a7f16:::benchmark_data_smithi194_186548_object58993:head# 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4a9b0b:::benchmark_data_smithi194_186548_object78667:head# 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.092 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4afb44:::benchmark_data_smithi194_186548_object80413:head# 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4cdae9:::benchmark_data_smithi194_186548_object49433:head# 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4e51a6:::benchmark_data_smithi194_186548_object29117:head# 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4e659b:::benchmark_data_smithi194_186548_object67130:head# 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f4e922c:::benchmark_data_smithi194_186548_object31125:head# 2024-04-29T14:53:56.093 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.094 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f54bee0:::benchmark_data_smithi194_186548_object84116:head# 2024-04-29T14:53:56.094 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.094 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f557c4a:::benchmark_data_smithi194_186548_object21703:head# 2024-04-29T14:53:56.094 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.094 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f565214:::benchmark_data_smithi194_186548_object14924:head# 2024-04-29T14:53:56.094 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f572a64:::benchmark_data_smithi194_186548_object58173:head# 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5789d1:::benchmark_data_smithi194_186548_object31482:head# 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f589282:::benchmark_data_smithi194_186548_object23814:head# 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5ac044:::benchmark_data_smithi194_186548_object15421:head# 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.098 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5ad800:::benchmark_data_smithi194_186548_object20263:head# 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5c4a3f:::benchmark_data_smithi194_186548_object49858:head# 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5d45e6:::benchmark_data_smithi194_186548_object70045:head# 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5da9ed:::benchmark_data_smithi194_186548_object57594:head# 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5dc5b5:::benchmark_data_smithi194_186548_object82019:head# 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5ea973:::benchmark_data_smithi194_186548_object15643:head# 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f5fea68:::benchmark_data_smithi194_186548_object76144:head# 2024-04-29T14:53:56.099 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f60f8a7:::benchmark_data_smithi194_186548_object79575:head# 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6180a2:::benchmark_data_smithi194_186548_object20151:head# 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f61fc9e:::benchmark_data_smithi194_186548_object35433:head# 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6274b4:::benchmark_data_smithi194_186548_object22427:head# 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f62beff:::benchmark_data_smithi194_186548_object29572:head# 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f632100:::benchmark_data_smithi194_186548_object48600:head# 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.100 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f63eda8:::benchmark_data_smithi194_186548_object28894:head# 2024-04-29T14:53:56.101 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.101 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f64420f:::benchmark_data_smithi194_186548_object7258:head# 2024-04-29T14:53:56.101 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f64f982:::benchmark_data_smithi194_186548_object29994:head# 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f65656d:::benchmark_data_smithi194_186548_object35170:head# 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f65c5d4:::benchmark_data_smithi194_186548_object59065:head# 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f66be2d:::benchmark_data_smithi194_186548_object34586:head# 2024-04-29T14:53:56.105 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f676b4f:::benchmark_data_smithi194_186548_object83740:head# 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f67f774:::benchmark_data_smithi194_186548_object16221:head# 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f68d5ed:::benchmark_data_smithi194_186548_object40785:head# 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6a661c:::benchmark_data_smithi194_186548_object62183:head# 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6a9991:::benchmark_data_smithi194_186548_object75416:head# 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6b328e:::benchmark_data_smithi194_186548_object77787:head# 2024-04-29T14:53:56.106 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.107 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6b35ba:::benchmark_data_smithi194_186548_object14281:head# 2024-04-29T14:53:56.107 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.107 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6cb08b:::benchmark_data_smithi194_186548_object70444:head# 2024-04-29T14:53:56.107 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.107 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6cc874:::benchmark_data_smithi194_186548_object7350:head# 2024-04-29T14:53:56.107 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.109 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f6e2850:::benchmark_data_smithi194_186548_object39798:head# 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f72cf08:::benchmark_data_smithi194_186548_object54246:head# 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f73641b:::benchmark_data_smithi194_186548_object41826:head# 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f737dee:::benchmark_data_smithi194_186548_object35333:head# 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f743bad:::benchmark_data_smithi194_186548_object33718:head# 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f767a94:::benchmark_data_smithi194_186548_object70782:head# 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.110 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f76aa6f:::benchmark_data_smithi194_186548_object74469:head# 2024-04-29T14:53:56.111 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.111 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f77a400:::benchmark_data_smithi194_186548_object67428:head# 2024-04-29T14:53:56.111 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.111 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f77f21f:::benchmark_data_smithi194_186548_object27681:head# 2024-04-29T14:53:56.111 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.118 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f79a9dd:::benchmark_data_smithi194_186548_object76224:head# 2024-04-29T14:53:56.118 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.118 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f7d4184:::benchmark_data_smithi194_186548_object5094:head# 2024-04-29T14:53:56.118 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.118 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f7da8ac:::benchmark_data_smithi194_186548_object72498:head# 2024-04-29T14:53:56.118 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f7ebfed:::benchmark_data_smithi194_186548_object49207:head# 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f7f362c:::benchmark_data_smithi194_186548_object42814:head# 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8083a5:::benchmark_data_smithi194_186548_object69394:head# 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f80c622:::benchmark_data_smithi194_186548_object67223:head# 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8245c7:::benchmark_data_smithi194_186548_object78642:head# 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f83b7ef:::benchmark_data_smithi194_186548_object28014:head# 2024-04-29T14:53:56.119 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f846c51:::benchmark_data_smithi194_186548_object17823:head# 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8717bb:::benchmark_data_smithi194_186548_object11683:head# 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f89c9b9:::benchmark_data_smithi194_186548_object51797:head# 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8a63c3:::benchmark_data_smithi194_186548_object52898:head# 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8ac15e:::benchmark_data_smithi194_186548_object63445:head# 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8adc88:::benchmark_data_smithi194_186548_object76329:head# 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.120 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8dd204:::benchmark_data_smithi194_186548_object32272:head# 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8e14f7:::benchmark_data_smithi194_186548_object19895:head# 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8e5c08:::benchmark_data_smithi194_186548_object2537:head# 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8f9916:::benchmark_data_smithi194_186548_object64536:head# 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f8ffb6c:::benchmark_data_smithi194_186548_object61993:head# 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9264ef:::benchmark_data_smithi194_186548_object63183:head# 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.121 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f92fcec:::benchmark_data_smithi194_186548_object45770:head# 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9429ce:::benchmark_data_smithi194_186548_object56558:head# 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f95f1e4:::benchmark_data_smithi194_186548_object5703:head# 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f963909:::benchmark_data_smithi194_186548_object57458:head# 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f964c03:::benchmark_data_smithi194_186548_object41693:head# 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f969272:::benchmark_data_smithi194_186548_object8581:head# 2024-04-29T14:53:56.122 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.123 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f974578:::benchmark_data_smithi194_186548_object55482:head# 2024-04-29T14:53:56.123 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.123 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f98dd3a:::benchmark_data_smithi194_186548_object1696:head# 2024-04-29T14:53:56.123 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.123 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f996144:::benchmark_data_smithi194_186548_object31569:head# 2024-04-29T14:53:56.123 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:55 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:53:56.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3760084016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:56.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:55 smithi040 ceph-mon[39581]: osdmap e1040: 12 total, 11 up, 9 in 2024-04-29T14:53:56.125 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f99f26f:::benchmark_data_smithi194_186548_object73453:head# 2024-04-29T14:53:56.125 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.125 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9a077a:::benchmark_data_smithi194_186548_object22049:head# 2024-04-29T14:53:56.125 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9a7659:::benchmark_data_smithi194_186548_object20417:head# 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9d8ab0:::benchmark_data_smithi194_186548_object3275:head# 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9deaa2:::benchmark_data_smithi194_186548_object42448:head# 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9e1fd2:::benchmark_data_smithi194_186548_object48396:head# 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9e2cd8:::benchmark_data_smithi194_186548_object21791:head# 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3f9e3cd2:::benchmark_data_smithi194_186548_object27457:head# 2024-04-29T14:53:56.126 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.127 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa02ba0:::benchmark_data_smithi194_186548_object65703:head# 2024-04-29T14:53:56.127 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.127 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa06eb2:::benchmark_data_smithi194_186548_object57269:head# 2024-04-29T14:53:56.127 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.127 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa468d2:::benchmark_data_smithi194_186548_object27714:head# 2024-04-29T14:53:56.127 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.128 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa69307:::benchmark_data_smithi194_186548_object69383:head# 2024-04-29T14:53:56.128 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.128 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa70dc1:::benchmark_data_smithi194_186548_object57048:head# 2024-04-29T14:53:56.128 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.129 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa71871:::benchmark_data_smithi194_186548_object26829:head# 2024-04-29T14:53:56.129 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.129 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa855a4:::benchmark_data_smithi194_186548_object45391:head# 2024-04-29T14:53:56.129 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.129 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fa8c9c3:::benchmark_data_smithi194_186548_object63956:head# 2024-04-29T14:53:56.129 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.130 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fabbf9f:::benchmark_data_smithi194_186548_object34926:head# 2024-04-29T14:53:56.130 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.130 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3faf62bf:::benchmark_data_smithi194_186548_object74233:head# 2024-04-29T14:53:56.131 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.131 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fafa6c7:::benchmark_data_smithi194_186548_object48934:head# 2024-04-29T14:53:56.131 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.131 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb1030e:::benchmark_data_smithi194_186548_object18583:head# 2024-04-29T14:53:56.131 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.132 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb1649d:::benchmark_data_smithi194_186548_object35758:head# 2024-04-29T14:53:56.132 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.132 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb18c75:::benchmark_data_smithi194_186548_object10608:head# 2024-04-29T14:53:56.132 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.132 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb18fd4:::benchmark_data_smithi194_186548_object71167:head# 2024-04-29T14:53:56.132 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.132 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:56.134 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb4486e:::benchmark_data_smithi194_186548_object21391:head# 2024-04-29T14:53:56.134 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.134 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb553db:::benchmark_data_smithi194_186548_object75087:head# 2024-04-29T14:53:56.134 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.134 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb5dd4e:::benchmark_data_smithi194_186548_object59322:head# 2024-04-29T14:53:56.134 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.134 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb5eb2c:::benchmark_data_smithi194_186548_object29028:head# 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb76325:::benchmark_data_smithi194_186548_object7986:head# 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb8268a:::benchmark_data_smithi194_186548_object64057:head# 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb95fff:::benchmark_data_smithi194_186548_object84562:head# 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fb9716d:::benchmark_data_smithi194_186548_object3860:head# 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbac187:::benchmark_data_smithi194_186548_object17953:head# 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.135 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbaca18:::benchmark_data_smithi194_186548_object36642:head# 2024-04-29T14:53:56.136 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:55 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:53:56.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3760084016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:53:56.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:55 smithi121 ceph-mon[44533]: osdmap e1040: 12 total, 11 up, 9 in 2024-04-29T14:53:56.138 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbb9dde:::benchmark_data_smithi194_186548_object77851:head# 2024-04-29T14:53:56.138 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.138 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbc1ea2:::benchmark_data_smithi194_186548_object31116:head# 2024-04-29T14:53:56.138 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.138 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbc42ab:::benchmark_data_smithi194_186548_object23202:head# 2024-04-29T14:53:56.138 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.138 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbc5818:::benchmark_data_smithi194_186548_object83236:head# 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbc5a01:::benchmark_data_smithi194_186548_object80175:head# 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbc7f91:::benchmark_data_smithi194_186548_object32793:head# 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbe1eb3:::benchmark_data_smithi194_186548_object19785:head# 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbe4a50:::benchmark_data_smithi194_186548_object46734:head# 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbff07b:::benchmark_data_smithi194_186548_object82759:head# 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fbff0b5:::benchmark_data_smithi194_186548_object18374:head# 2024-04-29T14:53:56.139 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.140 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc0769d:::benchmark_data_smithi194_186548_object27927:head# 2024-04-29T14:53:56.140 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:56.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:56.142 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc0d587:::benchmark_data_smithi194_186548_object44579:head# 2024-04-29T14:53:56.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.142 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc249fc:::benchmark_data_smithi194_186548_object899:head# 2024-04-29T14:53:56.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.142 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc2c8d6:::benchmark_data_smithi194_186548_object71184:head# 2024-04-29T14:53:56.142 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.143 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc32a22:::benchmark_data_smithi194_186548_object63006:head# 2024-04-29T14:53:56.143 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.143 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc33c2a:::benchmark_data_smithi194_186548_object32258:head# 2024-04-29T14:53:56.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc4a1e0:::benchmark_data_smithi194_186548_object22767:head# 2024-04-29T14:53:56.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc4c213:::benchmark_data_smithi194_186548_object33730:head# 2024-04-29T14:53:56.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.144 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc5c3d7:::benchmark_data_smithi194_186548_object43274:head# 2024-04-29T14:53:56.144 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc5c9e3:::benchmark_data_smithi194_186548_object21986:head# 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc6e113:::benchmark_data_smithi194_186548_object48370:head# 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fc77cad:::benchmark_data_smithi194_186548_object38165:head# 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcb0a98:::benchmark_data_smithi194_186548_object33225:head# 2024-04-29T14:53:56.145 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.146 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcb4c85:::benchmark_data_smithi194_186548_object40149:head# 2024-04-29T14:53:56.146 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.146 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcb95c4:::benchmark_data_smithi194_186548_object6559:head# 2024-04-29T14:53:56.146 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.146 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcbf6af:::benchmark_data_smithi194_186548_object41434:head# 2024-04-29T14:53:56.146 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.147 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcc2701:::benchmark_data_smithi194_186548_object49226:head# 2024-04-29T14:53:56.147 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.147 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcc3f5b:::benchmark_data_smithi194_186548_object14859:head# 2024-04-29T14:53:56.147 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.148 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fccb03c:::benchmark_data_smithi194_186548_object7979:head# 2024-04-29T14:53:56.148 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.148 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcd4f9a:::benchmark_data_smithi194_186548_object14439:head# 2024-04-29T14:53:56.148 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.148 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcde971:::benchmark_data_smithi194_186548_object18149:head# 2024-04-29T14:53:56.148 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcea510:::benchmark_data_smithi194_186548_object37528:head# 2024-04-29T14:53:56.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fcf314f:::benchmark_data_smithi194_186548_object4926:head# 2024-04-29T14:53:56.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.149 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd2912c:::benchmark_data_smithi194_186548_object85210:head# 2024-04-29T14:53:56.149 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd323f0:::benchmark_data_smithi194_186548_object71056:head# 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd43f26:::benchmark_data_smithi194_186548_object57607:head# 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd477c3:::benchmark_data_smithi194_186548_object28054:head# 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd50f3a:::benchmark_data_smithi194_186548_object75236:head# 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd52307:::benchmark_data_smithi194_186548_object62553:head# 2024-04-29T14:53:56.151 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd62216:::benchmark_data_smithi194_186548_object66038:head# 2024-04-29T14:53:56.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd64eff:::benchmark_data_smithi194_186548_object40254:head# 2024-04-29T14:53:56.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.152 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd66bd4:::benchmark_data_smithi194_186548_object28584:head# 2024-04-29T14:53:56.152 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd673d0:::benchmark_data_smithi194_186548_object34048:head# 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd789c3:::benchmark_data_smithi194_186548_object65536:head# 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd820fc:::benchmark_data_smithi194_186548_object71552:head# 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fd8f101:::benchmark_data_smithi194_186548_object24700:head# 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.154 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fda3a7c:::benchmark_data_smithi194_186548_object59850:head# 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fdad9c5:::benchmark_data_smithi194_186548_object76863:head# 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fdbd481:::benchmark_data_smithi194_186548_object53706:head# 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fddcf99:::benchmark_data_smithi194_186548_object14220:head# 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fde4b1c:::benchmark_data_smithi194_186548_object34153:head# 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fdf3010:::benchmark_data_smithi194_186548_object50087:head# 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fdf953f:::benchmark_data_smithi194_186548_object256:head# 2024-04-29T14:53:56.155 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.156 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fdfa3f1:::benchmark_data_smithi194_186548_object78380:head# 2024-04-29T14:53:56.156 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.158 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fdfef52:::benchmark_data_smithi194_186548_object10429:head# 2024-04-29T14:53:56.158 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.158 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe07709:::benchmark_data_smithi194_186548_object73782:head# 2024-04-29T14:53:56.158 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe1047b:::benchmark_data_smithi194_186548_object21118:head# 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe1c070:::benchmark_data_smithi194_186548_object8201:head# 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe1e35e:::benchmark_data_smithi194_186548_object21814:head# 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe1f622:::benchmark_data_smithi194_186548_object75491:head# 2024-04-29T14:53:56.159 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe1fd7e:::benchmark_data_smithi194_186548_object2544:head# 2024-04-29T14:53:56.160 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe2608b:::benchmark_data_smithi194_186548_object74579:head# 2024-04-29T14:53:56.160 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.160 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe2cc4c:::benchmark_data_smithi194_186548_object13783:head# 2024-04-29T14:53:56.161 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.161 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe2d0be:::benchmark_data_smithi194_186548_object63933:head# 2024-04-29T14:53:56.161 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe2ee6e:::benchmark_data_smithi194_186548_object77314:head# 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe4234c:::benchmark_data_smithi194_186548_object8548:head# 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe576d4:::benchmark_data_smithi194_186548_object30261:head# 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe70cb1:::benchmark_data_smithi194_186548_object78125:head# 2024-04-29T14:53:56.162 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.163 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe7ada1:::benchmark_data_smithi194_186548_object18295:head# 2024-04-29T14:53:56.163 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.163 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe81970:::benchmark_data_smithi194_186548_object56094:head# 2024-04-29T14:53:56.163 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.163 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fe98989:::benchmark_data_smithi194_186548_object11531:head# 2024-04-29T14:53:56.164 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.164 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3febdd8d:::benchmark_data_smithi194_186548_object3148:head# 2024-04-29T14:53:56.164 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.164 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3febf1cd:::benchmark_data_smithi194_186548_object66153:head# 2024-04-29T14:53:56.164 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.164 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fece83d:::benchmark_data_smithi194_186548_object15891:head# 2024-04-29T14:53:56.164 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fed169b:::benchmark_data_smithi194_186548_object8172:head# 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fedcd47:::benchmark_data_smithi194_186548_object47655:head# 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff186a2:::benchmark_data_smithi194_186548_object36772:head# 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff3a4df:::benchmark_data_smithi194_186548_object8399:head# 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff47600:::benchmark_data_smithi194_186548_object25549:head# 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff6f9c7:::benchmark_data_smithi194_186548_object71183:head# 2024-04-29T14:53:56.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.167 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff7fb3a:::benchmark_data_smithi194_186548_object2198:head# 2024-04-29T14:53:56.167 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff84865:::benchmark_data_smithi194_186548_object27309:head# 2024-04-29T14:53:56.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff9768d:::benchmark_data_smithi194_186548_object46057:head# 2024-04-29T14:53:56.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.168 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff9b687:::benchmark_data_smithi194_186548_object62903:head# 2024-04-29T14:53:56.168 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ff9e750:::benchmark_data_smithi194_186548_object39832:head# 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ffcb00a:::benchmark_data_smithi194_186548_object29755:head# 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ffccad9:::benchmark_data_smithi194_186548_object49055:head# 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ffdc576:::benchmark_data_smithi194_186548_object12482:head# 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ffdf668:::benchmark_data_smithi194_186548_object59044:head# 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ffe9687:::benchmark_data_smithi194_186548_object34149:head# 2024-04-29T14:53:56.169 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.170 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3ffea926:::benchmark_data_smithi194_186548_object17469:head# 2024-04-29T14:53:56.170 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.170 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fff45b2:::benchmark_data_smithi194_186548_object26098:head# 2024-04-29T14:53:56.170 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.171 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fff7960:::benchmark_data_smithi194_186548_object39676:head# 2024-04-29T14:53:56.171 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.172 INFO:teuthology.orchestra.run.smithi194.stdout:Write #8:3fffef05:::benchmark_data_smithi194_186548_object12765:head# 2024-04-29T14:53:56.172 INFO:teuthology.orchestra.run.smithi194.stdout:snapset 0=[]:{} 2024-04-29T14:53:56.174 INFO:teuthology.orchestra.run.smithi194.stdout:write_pg epoch 1019 info 8.c( v 1019'5334 (0'0,1019'5334] local-lis/les=1011/1012 n=5334 ec=1011/1011 lis/c=1011/1011 les/c/f=1012/1012/0 sis=1011) 2024-04-29T14:53:56.219 INFO:teuthology.orchestra.run.smithi194.stdout:Import successful 2024-04-29T14:53:56.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:56.295 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:56.559 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:56.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:56.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:56.697 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:56.718 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:56.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:56.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:56 smithi194 ceph-mon[44089]: pgmap v2635: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 12923/235910 objects degraded (5.478%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:53:56.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:53:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:56.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:53:56.944 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:56 smithi121 ceph-mon[44533]: pgmap v2635: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 12923/235910 objects degraded (5.478%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:53:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:56.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:53:56.989 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:56 smithi040 ceph-mon[39581]: pgmap v2635: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.2 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 12923/235910 objects degraded (5.478%); 12 MiB/s, 198 objects/s recovering 2024-04-29T14:53:57.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:57.094 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:57.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:57.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:57.224 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:53:57.280 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:57.308 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:53:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:57.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:53:57.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:53:57.452 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:53:57.456 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:57.481 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:53:57.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:53:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:57.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:53:57.509 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:53:57.542 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:53:57.577 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:53:57.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:57.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:53:57.636 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:53:57.644 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:57.665 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:53:57.692 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:53:57.719 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:53:57.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:53:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:57.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:53:57.750 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:53:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:53:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:53:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:53:57.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:53:57.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1350553336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:57.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:57 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 12923/235910 objects degraded (5.478%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:57.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:53:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:57.951 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:53:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:57.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:53:57.963 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/exp.8.c.9 2024-04-29T14:53:58.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:53:58.071 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-29T14:53:58.071 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:53:58.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:53:58.116 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1350553336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:58.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:57 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 12923/235910 objects degraded (5.478%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:58.121 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:58.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1350553336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:53:58.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:57 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 12923/235910 objects degraded (5.478%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:53:58.716 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:53:58.716 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:53:59.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:53:59.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:58 smithi194 ceph-mon[44089]: pgmap v2636: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 945 op/s; 12746/233528 objects degraded (5.458%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:53:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:53:59.089 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.089 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.089 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.089 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:53:59.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:58 smithi040 ceph-mon[39581]: pgmap v2636: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 945 op/s; 12746/233528 objects degraded (5.458%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:53:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:59.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:53:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:53:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:58 smithi121 ceph-mon[44533]: pgmap v2636: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.1 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 945 op/s; 12746/233528 objects degraded (5.458%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:53:59.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:53:59.232 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:59.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:53:59.594 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:53:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:53:59.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:53:59.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:53:59.736 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:53:59.768 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:53:59 smithi194 ceph-mon[44089]: osdmap e1041: 12 total, 11 up, 9 in 2024-04-29T14:53:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:53:59.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:53:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:53:59.980 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:54:00.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:53:59 smithi040 ceph-mon[39581]: osdmap e1041: 12 total, 11 up, 9 in 2024-04-29T14:54:00.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:53:59 smithi121 ceph-mon[44533]: osdmap e1041: 12 total, 11 up, 9 in 2024-04-29T14:54:00.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:00.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:00.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:00.296 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:00.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:00.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:00.750 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:00.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:00.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:00 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:00.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:00 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:00.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:00 smithi121 ceph-mon[44533]: pgmap v2638: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 12483/228688 objects degraded (5.459%); 9.0 MiB/s, 143 objects/s recovering 2024-04-29T14:54:00.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:00.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:00 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:00 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:00 smithi194 ceph-mon[44089]: pgmap v2638: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 12483/228688 objects degraded (5.459%); 9.0 MiB/s, 143 objects/s recovering 2024-04-29T14:54:01.080 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:00 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:01.080 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:00 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:01.080 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:00 smithi040 ceph-mon[39581]: pgmap v2638: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 7.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 924 op/s; 12483/228688 objects degraded (5.459%); 9.0 MiB/s, 143 objects/s recovering 2024-04-29T14:54:01.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:01.118 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:01.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:01.363 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:01.412 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:01.552 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:01.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:01.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:01.708 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:01.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:54:01.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:01 smithi194 ceph-mon[44089]: osdmap e1042: 12 total, 11 up, 9 in 2024-04-29T14:54:02.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:01 smithi040 ceph-mon[39581]: osdmap e1042: 12 total, 11 up, 9 in 2024-04-29T14:54:02.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:01 smithi121 ceph-mon[44533]: osdmap e1042: 12 total, 11 up, 9 in 2024-04-29T14:54:02.286 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:54:02.286 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:54:03.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:02 smithi194 ceph-mon[44089]: pgmap v2640: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 968 op/s; 11829/223548 objects degraded (5.291%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:54:03.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:02 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 12483/228688 objects degraded (5.459%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:03.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:02 smithi040 ceph-mon[39581]: pgmap v2640: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 968 op/s; 11829/223548 objects degraded (5.291%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:54:03.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:02 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 12483/228688 objects degraded (5.459%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:03.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:02 smithi121 ceph-mon[44533]: pgmap v2640: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 968 op/s; 11829/223548 objects degraded (5.291%); 13 MiB/s, 205 objects/s recovering 2024-04-29T14:54:03.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:02 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 12483/228688 objects degraded (5.459%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:03.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:54:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:03.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:03.482 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:03.527 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:03.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:03.569 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:54:03.669 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:03.722 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:03.842 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:03.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:03.949 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:04.011 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:04.118 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:04.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:04.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:05.041 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:05.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:05.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:05.540 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:54:05.540 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:54:05.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:05 smithi194 ceph-mon[44089]: pgmap v2641: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 931 op/s; 11560/220998 objects degraded (5.231%); 11 MiB/s, 170 objects/s recovering 2024-04-29T14:54:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:05 smithi040 ceph-mon[39581]: pgmap v2641: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 931 op/s; 11560/220998 objects degraded (5.231%); 11 MiB/s, 170 objects/s recovering 2024-04-29T14:54:05.882 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:54:05.883 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:54:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:05 smithi121 ceph-mon[44533]: pgmap v2641: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 931 op/s; 11560/220998 objects degraded (5.231%); 11 MiB/s, 170 objects/s recovering 2024-04-29T14:54:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:54:06.492 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:06.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:06.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:06.644 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:06.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:06.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:06.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:06 smithi121 ceph-mon[44533]: from='client.28249 v1:172.21.15.40:0/1279093923' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:06.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1353835703' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:54:06.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:06 smithi121 ceph-mon[44533]: from='client.38015 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:54:06.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:06 smithi194 ceph-mon[44089]: from='client.28249 v1:172.21.15.40:0/1279093923' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:06.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1353835703' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:54:06.700 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:06 smithi194 ceph-mon[44089]: from='client.38015 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:54:06.741 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:54:06.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:06.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:06.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:06.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:06 smithi040 ceph-mon[39581]: from='client.28249 v1:172.21.15.40:0/1279093923' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:06.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1353835703' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:54:06.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:06 smithi040 ceph-mon[39581]: from='client.38015 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:54:06.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:07.002 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:07.033 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:07.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:07.140 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:07.202 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:07.213 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:54:07.235 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:07.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:07.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:07.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:07.377 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:07.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:07 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:54:07.512 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:07 smithi194 ceph-mon[44089]: pgmap v2642: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 11068/216540 objects degraded (5.111%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:07.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:07 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:07.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:07 smithi194 ceph-mon[44089]: from='client.38015 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:54:07.513 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:07 smithi194 ceph-mon[44089]: osdmap e1043: 12 total, 11 up, 9 in 2024-04-29T14:54:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:07 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:54:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:07 smithi040 ceph-mon[39581]: pgmap v2642: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 11068/216540 objects degraded (5.111%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:07 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:07 smithi040 ceph-mon[39581]: from='client.38015 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:54:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:07 smithi040 ceph-mon[39581]: osdmap e1043: 12 total, 11 up, 9 in 2024-04-29T14:54:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:07 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:54:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:07 smithi121 ceph-mon[44533]: pgmap v2642: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.6 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 11068/216540 objects degraded (5.111%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:07 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:07 smithi121 ceph-mon[44533]: from='client.38015 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:54:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:07 smithi121 ceph-mon[44533]: osdmap e1043: 12 total, 11 up, 9 in 2024-04-29T14:54:07.949 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:07.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:08.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:08.106 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:08.568 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:08.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:08.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:08 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:54:08.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:08 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 11068/216540 objects degraded (5.111%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:08.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:08.707 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:08 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:54:08.707 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:08 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 11068/216540 objects degraded (5.111%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:08.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:08 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:54:08.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:08 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 11068/216540 objects degraded (5.111%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:08.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:08.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:08.744 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2641,"stamp":"2024-04-29T14:54:04.351316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7242056248,"num_objects":110499,"num_object_clones":0,"num_object_copies":220998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110499,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":146796,"num_write_kb":8230459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9976,"num_bytes_recovered":656572128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146754,"ondisk_log_size":146754,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":20137120,"kb_used_data":13556228,"kb_used_omap":666,"kb_used_meta":6580069,"kb_avail":823569248,"statfs":{"total":863955320832,"available":843334909952,"internally_reserved":0,"allocated":13881577472,"data_stored":13877878351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":682449,"internal_metadata":6737991215},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-488636416,"num_objects":-7456,"num_object_clones":0,"num_object_copies":-14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1363,"num_bytes_recovered":89325568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001526"},"pg_stats":[{"pgid":"8.f","version":"1042'9216","reported_seq":9287,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:01.646971+0000","last_change":"2024-04-29T14:51:58.472775+0000","last_active":"2024-04-29T14:54:01.646971+0000","last_peered":"2024-04-29T14:54:01.646971+0000","last_clean":"2024-04-29T14:54:01.646971+0000","last_became_active":"2024-04-29T14:51:58.471518+0000","last_became_peered":"2024-04-29T14:51:58.471518+0000","last_unstale":"2024-04-29T14:54:01.646971+0000","last_undegraded":"2024-04-29T14:54:01.646971+0000","last_fullsized":"2024-04-29T14:54:01.646971+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455606272,"num_objects":6952,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"1042'9487","reported_seq":9642,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:04.283614+0000","last_change":"2024-04-29T14:53:52.432694+0000","last_active":"2024-04-29T14:54:04.283614+0000","last_peered":"2024-04-29T14:54:04.283614+0000","last_clean":"2024-04-29T14:54:04.283614+0000","last_became_active":"2024-04-29T14:51:58.468700+0000","last_became_peered":"2024-04-29T14:51:58.468700+0000","last_unstale":"2024-04-29T14:54:04.283614+0000","last_undegraded":"2024-04-29T14:54:04.283614+0000","last_fullsized":"2024-04-29T14:54:04.283614+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1039'8698","last_scrub_stamp":"2024-04-29T14:53:52.432520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:53:52.432520+0000","objects_scrubbed":7584,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:52.682433+0000","scrub_duration":1228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443219968,"num_objects":6763,"num_object_clones":0,"num_object_copies":13526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9487,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"1042'9247","reported_seq":9407,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:01.241607+0000","last_change":"2024-04-29T14:53:47.116279+0000","last_active":"2024-04-29T14:54:01.241607+0000","last_peered":"2024-04-29T14:54:01.241607+0000","last_clean":"2024-04-29T14:54:01.241607+0000","last_became_active":"2024-04-29T14:51:58.469439+0000","last_became_peered":"2024-04-29T14:51:58.469439+0000","last_unstale":"2024-04-29T14:54:01.241607+0000","last_undegraded":"2024-04-29T14:54:01.241607+0000","last_fullsized":"2024-04-29T14:54:01.241607+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1038'8337","last_scrub_stamp":"2024-04-29T14:53:47.116014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:53:47.116014+0000","objects_scrubbed":7920,"log_size":9247,"log_dups_size":0,"ondisk_log_size":9247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:08.848827+0000","scrub_duration":1109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457441304,"num_objects":6981,"num_object_clones":0,"num_object_copies":13962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9247,"num_write_kb":519233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"1041'9066","reported_seq":14333,"reported_epoch":1041,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:00.706742+0000","last_change":"2024-04-29T14:52:59.877502+0000","last_active":"2024-04-29T14:54:00.706742+0000","last_peered":"2024-04-29T14:54:00.706742+0000","last_clean":"2024-04-29T14:52:42.460725+0000","last_became_active":"2024-04-29T14:52:59.820433+0000","last_became_peered":"2024-04-29T14:52:59.820433+0000","last_unstale":"2024-04-29T14:54:00.706742+0000","last_undegraded":"2024-04-29T14:52:59.690774+0000","last_fullsized":"2024-04-29T14:54:00.706742+0000","mapping_epoch":1026,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9066,"log_dups_size":0,"ondisk_log_size":9066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452198400,"num_objects":6900,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2597,"num_bytes_recovered":170196992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8"],"object_location_counts":[{"shards":"3,8","objects":4163},{"shards":"8","objects":2737}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"1041'9009","reported_seq":9078,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-04-29T14:54:00.648003+0000","last_change":"2024-04-29T14:51:58.469525+0000","last_active":"2024-04-29T14:54:00.648003+0000","last_peered":"2024-04-29T14:54:00.648003+0000","last_clean":"2024-04-29T14:54:00.648003+0000","last_became_active":"2024-04-29T14:51:58.469253+0000","last_became_peered":"2024-04-29T14:51:58.469253+0000","last_unstale":"2024-04-29T14:54:00.648003+0000","last_undegraded":"2024-04-29T14:54:00.648003+0000","last_fullsized":"2024-04-29T14:54:00.648003+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456851456,"num_objects":6971,"num_object_clones":0,"num_object_copies":13942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"1041'9105","reported_seq":13874,"reported_epoch":1041,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:00.722093+0000","last_change":"2024-04-29T14:52:59.845465+0000","last_active":"2024-04-29T14:54:00.722093+0000","last_peered":"2024-04-29T14:54:00.722093+0000","last_clean":"2024-04-29T14:52:42.470937+0000","last_became_active":"2024-04-29T14:52:59.816964+0000","last_became_peered":"2024-04-29T14:52:59.816964+0000","last_unstale":"2024-04-29T14:54:00.722093+0000","last_undegraded":"2024-04-29T14:52:59.688731+0000","last_fullsized":"2024-04-29T14:54:00.722093+0000","mapping_epoch":1026,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458293248,"num_objects":6993,"num_object_clones":0,"num_object_copies":13986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9105,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2348,"num_bytes_recovered":153878528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8"],"object_location_counts":[{"shards":"3,8","objects":3990},{"shards":"8","objects":3003}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"1042'9386","reported_seq":9457,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:03.666336+0000","last_change":"2024-04-29T14:51:58.470103+0000","last_active":"2024-04-29T14:54:03.666336+0000","last_peered":"2024-04-29T14:54:03.666336+0000","last_clean":"2024-04-29T14:54:03.666336+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:54:03.666336+0000","last_undegraded":"2024-04-29T14:54:03.666336+0000","last_fullsized":"2024-04-29T14:54:03.666336+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448004096,"num_objects":6836,"num_object_clones":0,"num_object_copies":13672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":519104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"1042'9251","reported_seq":9322,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:01.601622+0000","last_change":"2024-04-29T14:51:58.473278+0000","last_active":"2024-04-29T14:54:01.601622+0000","last_peered":"2024-04-29T14:54:01.601622+0000","last_clean":"2024-04-29T14:54:01.601622+0000","last_became_active":"2024-04-29T14:51:58.473066+0000","last_became_peered":"2024-04-29T14:51:58.473066+0000","last_unstale":"2024-04-29T14:54:01.601622+0000","last_undegraded":"2024-04-29T14:54:01.601622+0000","last_fullsized":"2024-04-29T14:54:01.601622+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449773568,"num_objects":6863,"num_object_clones":0,"num_object_copies":13726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"1041'8985","reported_seq":9054,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-04-29T14:53:59.480796+0000","last_change":"2024-04-29T14:51:58.472827+0000","last_active":"2024-04-29T14:53:59.480796+0000","last_peered":"2024-04-29T14:53:59.480796+0000","last_clean":"2024-04-29T14:53:59.480796+0000","last_became_active":"2024-04-29T14:51:58.472520+0000","last_became_peered":"2024-04-29T14:51:58.472520+0000","last_unstale":"2024-04-29T14:53:59.480796+0000","last_undegraded":"2024-04-29T14:53:59.480796+0000","last_fullsized":"2024-04-29T14:53:59.480796+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458031104,"num_objects":6989,"num_object_clones":0,"num_object_copies":13978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"1041'8990","reported_seq":9059,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-04-29T14:53:59.472930+0000","last_change":"2024-04-29T14:51:58.473051+0000","last_active":"2024-04-29T14:53:59.472930+0000","last_peered":"2024-04-29T14:53:59.472930+0000","last_clean":"2024-04-29T14:53:59.472930+0000","last_became_active":"2024-04-29T14:51:58.472757+0000","last_became_peered":"2024-04-29T14:51:58.472757+0000","last_unstale":"2024-04-29T14:53:59.472930+0000","last_undegraded":"2024-04-29T14:53:59.472930+0000","last_fullsized":"2024-04-29T14:53:59.472930+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":8990,"log_dups_size":0,"ondisk_log_size":8990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456785920,"num_objects":6970,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"1042'9432","reported_seq":15268,"reported_epoch":1042,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:03.662376+0000","last_change":"2024-04-29T14:52:59.834273+0000","last_active":"2024-04-29T14:54:03.662376+0000","last_peered":"2024-04-29T14:54:03.662376+0000","last_clean":"2024-04-29T14:52:42.476364+0000","last_became_active":"2024-04-29T14:52:59.825361+0000","last_became_peered":"2024-04-29T14:52:59.825361+0000","last_unstale":"2024-04-29T14:54:03.662376+0000","last_undegraded":"2024-04-29T14:52:59.673133+0000","last_fullsized":"2024-04-29T14:54:03.662376+0000","mapping_epoch":1026,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444596224,"num_objects":6784,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2880,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2572},{"shards":"5,10","objects":4212}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"1042'9085","reported_seq":9156,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:01.653931+0000","last_change":"2024-04-29T14:51:58.469033+0000","last_active":"2024-04-29T14:54:01.653931+0000","last_peered":"2024-04-29T14:54:01.653931+0000","last_clean":"2024-04-29T14:54:01.653931+0000","last_became_active":"2024-04-29T14:51:58.468800+0000","last_became_peered":"2024-04-29T14:51:58.468800+0000","last_unstale":"2024-04-29T14:54:01.653931+0000","last_undegraded":"2024-04-29T14:54:01.653931+0000","last_fullsized":"2024-04-29T14:54:01.653931+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448987136,"num_objects":6851,"num_object_clones":0,"num_object_copies":13702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9085,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"1041'8978","reported_seq":9047,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-04-29T14:53:59.449840+0000","last_change":"2024-04-29T14:51:58.473947+0000","last_active":"2024-04-29T14:53:59.449840+0000","last_peered":"2024-04-29T14:53:59.449840+0000","last_clean":"2024-04-29T14:53:59.449840+0000","last_became_active":"2024-04-29T14:51:58.473704+0000","last_became_peered":"2024-04-29T14:51:58.473704+0000","last_unstale":"2024-04-29T14:53:59.449840+0000","last_undegraded":"2024-04-29T14:53:59.449840+0000","last_fullsized":"2024-04-29T14:53:59.449840+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":8978,"log_dups_size":0,"ondisk_log_size":8978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457310208,"num_objects":6978,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":510592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"1042'9278","reported_seq":13629,"reported_epoch":1042,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:01.643659+0000","last_change":"2024-04-29T14:52:59.848634+0000","last_active":"2024-04-29T14:54:01.643659+0000","last_peered":"2024-04-29T14:54:01.643659+0000","last_clean":"2024-04-29T14:52:42.467078+0000","last_became_active":"2024-04-29T14:52:59.811137+0000","last_became_peered":"2024-04-29T14:52:59.811137+0000","last_unstale":"2024-04-29T14:54:01.643659+0000","last_undegraded":"2024-04-29T14:52:59.676485+0000","last_fullsized":"2024-04-29T14:54:01.643659+0000","mapping_epoch":1026,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457572352,"num_objects":6982,"num_object_clones":0,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9278,"num_write_kb":520320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2137,"num_bytes_recovered":140050432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":3248},{"shards":"1,10","objects":3734}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"1042'9285","reported_seq":9357,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:03.649983+0000","last_change":"2024-04-29T14:51:58.472256+0000","last_active":"2024-04-29T14:54:03.649983+0000","last_peered":"2024-04-29T14:54:03.649983+0000","last_clean":"2024-04-29T14:54:03.649983+0000","last_became_active":"2024-04-29T14:51:58.471940+0000","last_became_peered":"2024-04-29T14:51:58.471940+0000","last_unstale":"2024-04-29T14:54:03.649983+0000","last_undegraded":"2024-04-29T14:54:03.649983+0000","last_fullsized":"2024-04-29T14:54:03.649983+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447545344,"num_objects":6829,"num_object_clones":0,"num_object_copies":13658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9285,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"1041'8815","reported_seq":8884,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-04-29T14:53:59.503733+0000","last_change":"2024-04-29T14:51:58.472764+0000","last_active":"2024-04-29T14:53:59.503733+0000","last_peered":"2024-04-29T14:53:59.503733+0000","last_clean":"2024-04-29T14:53:59.503733+0000","last_became_active":"2024-04-29T14:51:58.472489+0000","last_became_peered":"2024-04-29T14:51:58.472489+0000","last_unstale":"2024-04-29T14:53:59.503733+0000","last_undegraded":"2024-04-29T14:53:59.503733+0000","last_fullsized":"2024-04-29T14:53:59.503733+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449249280,"num_objects":6855,"num_object_clones":0,"num_object_copies":13710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8815,"num_write_kb":501440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2090,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-04-29T14:54:00.808176+0000","last_change":"2024-04-29T14:53:05.067880+0000","last_active":"2024-04-29T14:54:00.808176+0000","last_peered":"2024-04-29T14:54:00.808176+0000","last_clean":"2024-04-29T14:54:00.808176+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:54:00.808176+0000","last_undegraded":"2024-04-29T14:54:00.808176+0000","last_fullsized":"2024-04-29T14:54:00.808176+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:53:05.067722+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:53:05.067722+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:05.530532+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7241465880,"num_objects":110497,"num_object_clones":0,"num_object_copies":220994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146615,"num_write_kb":8227521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9962,"num_bytes_recovered":652869632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15005261824,"data_stored":15005253680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146615,"ondisk_log_size":146615,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122614,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3957948,"kb_used_data":2925808,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":89787204,"statfs":{"total":95995035648,"available":91942096896,"internally_reserved":0,"allocated":2996027392,"data_stored":2995621535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1056897472},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423193,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1527804,"kb_used_data":1342576,"kb_used_omap":4,"kb_used_meta":185211,"kb_avail":92217348,"statfs":{"total":95995035648,"available":94430564352,"internally_reserved":0,"allocated":1374797824,"data_stored":1374391987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":189656768},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234569,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1790036,"kb_used_data":888392,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":91955116,"statfs":{"total":95995035648,"available":94162038784,"internally_reserved":0,"allocated":909713408,"data_stored":909293683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":923269578},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988996,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92756156,"statfs":{"total":95995035648,"available":94982303744,"internally_reserved":0,"allocated":1302528,"data_stored":893139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575450,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2332040,"kb_used_data":1783036,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":91413112,"statfs":{"total":95995035648,"available":93607026688,"internally_reserved":0,"allocated":1825828864,"data_stored":1825416763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738988,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1827832,"kb_used_data":890032,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":91917320,"statfs":{"total":95995035648,"available":94123335680,"internally_reserved":0,"allocated":911392768,"data_stored":910986931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183008,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2042564,"kb_used_data":1393596,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91702588,"statfs":{"total":95995035648,"available":93903450112,"internally_reserved":0,"allocated":1427042304,"data_stored":1426630203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":664533448},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986773,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2752964,"kb_used_data":1729148,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":90992188,"statfs":{"total":95995035648,"available":93176000512,"internally_reserved":0,"allocated":1770647552,"data_stored":1770235451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1048377797},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653334,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2916936,"kb_used_data":2602368,"kb_used_omap":651,"kb_used_meta":313908,"kb_avail":90828216,"statfs":{"total":95995035648,"available":93008093184,"internally_reserved":0,"allocated":2664824832,"data_stored":2664408659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":321442568},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":908263424,"data_stored":908263424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1811742720,"data_stored":1811742720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1423638528,"data_stored":1423638528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1768226816,"data_stored":1768226816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2660110336,"data_stored":2660106264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":908333056,"data_stored":908328984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1367212032,"data_stored":1367212032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1349648384,"data_stored":1349648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2808086528,"data_stored":2808086528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:54:08.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:54:08.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:08.881 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:09.016 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:09.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:09.121 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:09.173 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:09.208 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:09.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:09.229 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:54:09.265 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:54:09.297 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:54:09.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:09.332 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:54:09.372 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:09.373 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:54:09.409 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:54:09.442 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:54:09.505 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:54:09.515 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:54:09.540 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:54:09.540 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:54:09.549 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:54:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:54:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:54:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:54:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:54:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:54:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:54:09.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:09 smithi040 ceph-mon[39581]: pgmap v2644: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 961 op/s; 10866/213298 objects degraded (5.094%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:54:09.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/135136237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:09.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:09 smithi194 ceph-mon[44089]: pgmap v2644: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 961 op/s; 10866/213298 objects degraded (5.094%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:54:09.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/135136237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:09.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:09 smithi121 ceph-mon[44533]: pgmap v2644: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 6.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 961 op/s; 10866/213298 objects degraded (5.094%); 13 MiB/s, 202 objects/s recovering 2024-04-29T14:54:09.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/135136237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:10.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:54:10.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:10 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:10 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:10.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:10 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:11.370 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:11.380 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:11.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:11.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:11.544 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:11.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:11.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:11.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:11.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:11 smithi040 ceph-mon[39581]: pgmap v2645: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 6.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 970 op/s; 10584/208254 objects degraded (5.082%); 9.4 MiB/s, 149 objects/s recovering 2024-04-29T14:54:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:11 smithi194 ceph-mon[44089]: pgmap v2645: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 6.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 970 op/s; 10584/208254 objects degraded (5.082%); 9.4 MiB/s, 149 objects/s recovering 2024-04-29T14:54:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:11 smithi121 ceph-mon[44533]: pgmap v2645: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+degraded, 12 active+clean; 6.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 970 op/s; 10584/208254 objects degraded (5.082%); 9.4 MiB/s, 149 objects/s recovering 2024-04-29T14:54:12.482 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:54:12.482 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:54:12.797 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:12.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:12.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:12.946 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:12.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:12.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:13.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:13.139 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:13.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:13.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:13.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:54:13.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:13.322 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:13.441 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:13.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:13.556 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:13.602 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:13.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:13 smithi040 ceph-mon[39581]: pgmap v2646: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 9849/203254 objects degraded (4.846%); 12 MiB/s, 197 objects/s recovering 2024-04-29T14:54:13.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:13 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 9849/203254 objects degraded (4.846%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:13.659 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:13.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:13.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:13 smithi194 ceph-mon[44089]: pgmap v2646: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 9849/203254 objects degraded (4.846%); 12 MiB/s, 197 objects/s recovering 2024-04-29T14:54:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:13 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 9849/203254 objects degraded (4.846%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:13.827 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:13.850 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:13.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:13 smithi121 ceph-mon[44533]: pgmap v2646: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 9849/203254 objects degraded (4.846%); 12 MiB/s, 197 objects/s recovering 2024-04-29T14:54:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:13 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 9849/203254 objects degraded (4.846%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:13.960 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:14.000 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:14.028 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:14.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:14.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:14.179 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:14.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:14.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:14.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:14.346 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:14.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:14.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:14.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:54:14.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:14.529 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:14.550 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:14.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:14.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:15.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:15 smithi194 ceph-mon[44089]: pgmap v2647: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 9541/200664 objects degraded (4.755%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:54:15.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:15 smithi040 ceph-mon[39581]: pgmap v2647: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 9541/200664 objects degraded (4.755%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:54:15.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:15 smithi121 ceph-mon[44533]: pgmap v2647: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.1 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 9541/200664 objects degraded (4.755%); 13 MiB/s, 201 objects/s recovering 2024-04-29T14:54:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:16 smithi194 ceph-mon[44089]: osdmap e1044: 12 total, 11 up, 9 in 2024-04-29T14:54:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:16.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:16.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:16.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:16 smithi040 ceph-mon[39581]: osdmap e1044: 12 total, 11 up, 9 in 2024-04-29T14:54:16.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:16.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:16.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:16.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:16.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:16 smithi121 ceph-mon[44533]: osdmap e1044: 12 total, 11 up, 9 in 2024-04-29T14:54:16.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:16.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:17.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:17 smithi040 ceph-mon[39581]: pgmap v2649: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 8990/196296 objects degraded (4.580%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:17.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:17 smithi040 ceph-mon[39581]: osdmap e1045: 12 total, 11 up, 9 in 2024-04-29T14:54:17.523 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:17 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:17.574 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:17.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:17.614 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:54:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:54:17.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:17.750 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:17.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:17 smithi194 ceph-mon[44089]: pgmap v2649: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 8990/196296 objects degraded (4.580%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:17.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:17 smithi194 ceph-mon[44089]: osdmap e1045: 12 total, 11 up, 9 in 2024-04-29T14:54:17.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:17 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:17 smithi121 ceph-mon[44533]: pgmap v2649: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 6.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.02k op/s; 8990/196296 objects degraded (4.580%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:17 smithi121 ceph-mon[44533]: osdmap e1045: 12 total, 11 up, 9 in 2024-04-29T14:54:17.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:17 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:18.716 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:18.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:18 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 8990/196296 objects degraded (4.580%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:18.827 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:18.860 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:18.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:18 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 8990/196296 objects degraded (4.580%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:18.887 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:18.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:18 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 8990/196296 objects degraded (4.580%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:18.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:18.995 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:19.031 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:19.066 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:19.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:19.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:19.208 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:19.246 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:19.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:19.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:19.412 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:19.416 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2650,"stamp":"2024-04-29T14:54:16.515780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6411452984,"num_objects":97825,"num_object_clones":0,"num_object_copies":195650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97825,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":159470,"num_write_kb":8230459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12546,"num_bytes_recovered":824999648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159028,"ondisk_log_size":159028,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":18727948,"kb_used_data":12105344,"kb_used_omap":666,"kb_used_meta":6621733,"kb_avail":824978420,"statfs":{"total":863955320832,"available":844777902080,"internally_reserved":0,"allocated":12395872256,"data_stored":12392642575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":682449,"internal_metadata":6780655151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,7,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-578289664,"num_objects":-8824,"num_object_clones":0,"num_object_copies":-17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8824,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":122945536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.163669"},"pg_stats":[{"pgid":"8.f","version":"1043'9854","reported_seq":9927,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:11.709976+0000","last_change":"2024-04-29T14:51:58.472775+0000","last_active":"2024-04-29T14:54:11.709976+0000","last_peered":"2024-04-29T14:54:11.709976+0000","last_clean":"2024-04-29T14:54:11.709976+0000","last_became_active":"2024-04-29T14:51:58.471518+0000","last_became_peered":"2024-04-29T14:51:58.471518+0000","last_unstale":"2024-04-29T14:54:11.709976+0000","last_undegraded":"2024-04-29T14:54:11.709976+0000","last_fullsized":"2024-04-29T14:54:11.709976+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9854,"log_dups_size":0,"ondisk_log_size":9854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413794304,"num_objects":6314,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9854,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"1043'10118","reported_seq":10276,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:14.333989+0000","last_change":"2024-04-29T14:53:52.432694+0000","last_active":"2024-04-29T14:54:14.333989+0000","last_peered":"2024-04-29T14:54:14.333989+0000","last_clean":"2024-04-29T14:54:14.333989+0000","last_became_active":"2024-04-29T14:51:58.468700+0000","last_became_peered":"2024-04-29T14:51:58.468700+0000","last_unstale":"2024-04-29T14:54:14.333989+0000","last_undegraded":"2024-04-29T14:54:14.333989+0000","last_fullsized":"2024-04-29T14:54:14.333989+0000","mapping_epoch":1011,"log_start":"1016'100","ondisk_log_start":"1016'100","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1039'8698","last_scrub_stamp":"2024-04-29T14:53:52.432520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:53:52.432520+0000","objects_scrubbed":7584,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:52.682433+0000","scrub_duration":1228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401866752,"num_objects":6132,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"1044'10237","reported_seq":10401,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-29T14:54:16.438364+0000","last_change":"2024-04-29T14:53:47.116279+0000","last_active":"2024-04-29T14:54:16.438364+0000","last_peered":"2024-04-29T14:54:16.438364+0000","last_clean":"2024-04-29T14:54:16.438364+0000","last_became_active":"2024-04-29T14:51:58.469439+0000","last_became_peered":"2024-04-29T14:51:58.469439+0000","last_unstale":"2024-04-29T14:54:16.438364+0000","last_undegraded":"2024-04-29T14:54:16.438364+0000","last_fullsized":"2024-04-29T14:54:16.438364+0000","mapping_epoch":1011,"log_start":"1016'200","ondisk_log_start":"1016'200","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1038'8337","last_scrub_stamp":"2024-04-29T14:53:47.116014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:53:47.116014+0000","objects_scrubbed":7920,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:08.848827+0000","scrub_duration":1109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392560664,"num_objects":5991,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10237,"num_write_kb":519233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"1044'10048","reported_seq":16986,"reported_epoch":1044,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:15.736257+0000","last_change":"2024-04-29T14:52:59.877502+0000","last_active":"2024-04-29T14:54:15.736257+0000","last_peered":"2024-04-29T14:54:15.736257+0000","last_clean":"2024-04-29T14:52:42.460725+0000","last_became_active":"2024-04-29T14:52:59.820433+0000","last_became_peered":"2024-04-29T14:52:59.820433+0000","last_unstale":"2024-04-29T14:54:15.736257+0000","last_undegraded":"2024-04-29T14:52:59.690774+0000","last_fullsized":"2024-04-29T14:54:15.736257+0000","mapping_epoch":1026,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387842048,"num_objects":5918,"num_object_clones":0,"num_object_copies":11836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10048,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3429,"num_bytes_recovered":224722944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8"],"object_location_counts":[{"shards":"3,8","objects":4013},{"shards":"8","objects":1905}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"1044'9952","reported_seq":10028,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-29T14:54:15.739803+0000","last_change":"2024-04-29T14:51:58.469525+0000","last_active":"2024-04-29T14:54:15.739803+0000","last_peered":"2024-04-29T14:54:15.739803+0000","last_clean":"2024-04-29T14:54:15.739803+0000","last_became_active":"2024-04-29T14:51:58.469253+0000","last_became_peered":"2024-04-29T14:51:58.469253+0000","last_unstale":"2024-04-29T14:54:15.739803+0000","last_undegraded":"2024-04-29T14:54:15.739803+0000","last_fullsized":"2024-04-29T14:54:15.739803+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395051008,"num_objects":6028,"num_object_clones":0,"num_object_copies":12056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"1044'10066","reported_seq":16336,"reported_epoch":1044,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:15.736745+0000","last_change":"2024-04-29T14:52:59.845465+0000","last_active":"2024-04-29T14:54:15.736745+0000","last_peered":"2024-04-29T14:54:15.736745+0000","last_clean":"2024-04-29T14:52:42.470937+0000","last_became_active":"2024-04-29T14:52:59.816964+0000","last_became_peered":"2024-04-29T14:52:59.816964+0000","last_unstale":"2024-04-29T14:54:15.736745+0000","last_undegraded":"2024-04-29T14:52:59.688731+0000","last_fullsized":"2024-04-29T14:54:15.736745+0000","mapping_epoch":1026,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395313152,"num_objects":6032,"num_object_clones":0,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10066,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3095,"num_bytes_recovered":202833920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8"],"object_location_counts":[{"shards":"3,8","objects":3776},{"shards":"8","objects":2256}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"1043'10032","reported_seq":10106,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:13.669620+0000","last_change":"2024-04-29T14:51:58.470103+0000","last_active":"2024-04-29T14:54:13.669620+0000","last_peered":"2024-04-29T14:54:13.669620+0000","last_clean":"2024-04-29T14:54:13.669620+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:54:13.669620+0000","last_undegraded":"2024-04-29T14:54:13.669620+0000","last_fullsized":"2024-04-29T14:54:13.669620+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405667840,"num_objects":6190,"num_object_clones":0,"num_object_copies":12380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10032,"num_write_kb":519104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"1043'9906","reported_seq":9980,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:11.710195+0000","last_change":"2024-04-29T14:51:58.473278+0000","last_active":"2024-04-29T14:54:11.710195+0000","last_peered":"2024-04-29T14:54:11.710195+0000","last_clean":"2024-04-29T14:54:11.710195+0000","last_became_active":"2024-04-29T14:51:58.473066+0000","last_became_peered":"2024-04-29T14:51:58.473066+0000","last_unstale":"2024-04-29T14:54:11.710195+0000","last_undegraded":"2024-04-29T14:54:11.710195+0000","last_fullsized":"2024-04-29T14:54:11.710195+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9906,"log_dups_size":0,"ondisk_log_size":9906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406847488,"num_objects":6208,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"1043'9848","reported_seq":9922,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:14.485354+0000","last_change":"2024-04-29T14:51:58.472827+0000","last_active":"2024-04-29T14:54:14.485354+0000","last_peered":"2024-04-29T14:54:14.485354+0000","last_clean":"2024-04-29T14:54:14.485354+0000","last_became_active":"2024-04-29T14:51:58.472520+0000","last_became_peered":"2024-04-29T14:51:58.472520+0000","last_unstale":"2024-04-29T14:54:14.485354+0000","last_undegraded":"2024-04-29T14:54:14.485354+0000","last_fullsized":"2024-04-29T14:54:14.485354+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9848,"log_dups_size":0,"ondisk_log_size":9848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401473536,"num_objects":6126,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"1043'9870","reported_seq":9944,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:14.460979+0000","last_change":"2024-04-29T14:51:58.473051+0000","last_active":"2024-04-29T14:54:14.460979+0000","last_peered":"2024-04-29T14:54:14.460979+0000","last_clean":"2024-04-29T14:54:14.460979+0000","last_became_active":"2024-04-29T14:51:58.472757+0000","last_became_peered":"2024-04-29T14:51:58.472757+0000","last_unstale":"2024-04-29T14:54:14.460979+0000","last_undegraded":"2024-04-29T14:54:14.460979+0000","last_fullsized":"2024-04-29T14:54:14.460979+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9870,"log_dups_size":0,"ondisk_log_size":9870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399114240,"num_objects":6090,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"1043'10102","reported_seq":17121,"reported_epoch":1043,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:13.677650+0000","last_change":"2024-04-29T14:52:59.834273+0000","last_active":"2024-04-29T14:54:13.677650+0000","last_peered":"2024-04-29T14:54:13.677650+0000","last_clean":"2024-04-29T14:52:42.476364+0000","last_became_active":"2024-04-29T14:52:59.825361+0000","last_became_peered":"2024-04-29T14:52:59.825361+0000","last_unstale":"2024-04-29T14:54:13.677650+0000","last_undegraded":"2024-04-29T14:52:59.673133+0000","last_fullsized":"2024-04-29T14:54:13.677650+0000","mapping_epoch":1026,"log_start":"1016'100","ondisk_log_start":"1016'100","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400687104,"num_objects":6114,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10102,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3470,"num_bytes_recovered":227409920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1982},{"shards":"5,10","objects":4132}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"1043'9710","reported_seq":9784,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:11.689179+0000","last_change":"2024-04-29T14:51:58.469033+0000","last_active":"2024-04-29T14:54:11.689179+0000","last_peered":"2024-04-29T14:54:11.689179+0000","last_clean":"2024-04-29T14:54:11.689179+0000","last_became_active":"2024-04-29T14:51:58.468800+0000","last_became_peered":"2024-04-29T14:51:58.468800+0000","last_unstale":"2024-04-29T14:54:11.689179+0000","last_undegraded":"2024-04-29T14:54:11.689179+0000","last_fullsized":"2024-04-29T14:54:11.689179+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408027136,"num_objects":6226,"num_object_clones":0,"num_object_copies":12452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9710,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"1043'9944","reported_seq":10018,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:14.525366+0000","last_change":"2024-04-29T14:51:58.473947+0000","last_active":"2024-04-29T14:54:14.525366+0000","last_peered":"2024-04-29T14:54:14.525366+0000","last_clean":"2024-04-29T14:54:14.525366+0000","last_became_active":"2024-04-29T14:51:58.473704+0000","last_became_peered":"2024-04-29T14:51:58.473704+0000","last_unstale":"2024-04-29T14:54:14.525366+0000","last_undegraded":"2024-04-29T14:54:14.525366+0000","last_fullsized":"2024-04-29T14:54:14.525366+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394002432,"num_objects":6012,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":510592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"1043'9883","reported_seq":15039,"reported_epoch":1043,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:11.706151+0000","last_change":"2024-04-29T14:52:59.848634+0000","last_active":"2024-04-29T14:54:11.706151+0000","last_peered":"2024-04-29T14:54:11.706151+0000","last_clean":"2024-04-29T14:52:42.467078+0000","last_became_active":"2024-04-29T14:52:59.811137+0000","last_became_peered":"2024-04-29T14:52:59.811137+0000","last_unstale":"2024-04-29T14:54:11.706151+0000","last_undegraded":"2024-04-29T14:52:59.676485+0000","last_fullsized":"2024-04-29T14:54:11.706151+0000","mapping_epoch":1026,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9883,"log_dups_size":0,"ondisk_log_size":9883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417923072,"num_objects":6377,"num_object_clones":0,"num_object_copies":12754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":520320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2538,"num_bytes_recovered":166330368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2847},{"shards":"1,10","objects":3530}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"1043'9927","reported_seq":10001,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:13.659582+0000","last_change":"2024-04-29T14:51:58.472256+0000","last_active":"2024-04-29T14:54:13.659582+0000","last_peered":"2024-04-29T14:54:13.659582+0000","last_clean":"2024-04-29T14:54:13.659582+0000","last_became_active":"2024-04-29T14:51:58.471940+0000","last_became_peered":"2024-04-29T14:51:58.471940+0000","last_unstale":"2024-04-29T14:54:13.659582+0000","last_undegraded":"2024-04-29T14:54:13.659582+0000","last_fullsized":"2024-04-29T14:54:13.659582+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405471232,"num_objects":6187,"num_object_clones":0,"num_object_copies":12374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"1043'9792","reported_seq":9865,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-04-29T14:54:14.544489+0000","last_change":"2024-04-29T14:51:58.472764+0000","last_active":"2024-04-29T14:54:14.544489+0000","last_peered":"2024-04-29T14:54:14.544489+0000","last_clean":"2024-04-29T14:54:14.544489+0000","last_became_active":"2024-04-29T14:51:58.472489+0000","last_became_peered":"2024-04-29T14:51:58.472489+0000","last_unstale":"2024-04-29T14:54:14.544489+0000","last_undegraded":"2024-04-29T14:54:14.544489+0000","last_fullsized":"2024-04-29T14:54:14.544489+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385220608,"num_objects":5878,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":501440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2103,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-29T14:54:15.520636+0000","last_change":"2024-04-29T14:54:06.451143+0000","last_active":"2024-04-29T14:54:15.520636+0000","last_peered":"2024-04-29T14:54:15.520636+0000","last_clean":"2024-04-29T14:54:15.520636+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:54:15.520636+0000","last_undegraded":"2024-04-29T14:54:15.520636+0000","last_fullsized":"2024-04-29T14:54:15.520636+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:54:06.451003+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:54:06.451003+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:09.012234+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6410862616,"num_objects":97823,"num_object_clones":0,"num_object_copies":195646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159289,"num_write_kb":8227521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12532,"num_bytes_recovered":821297152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13543809024,"data_stored":13543800880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158889,"ondisk_log_size":158889,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384256,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3597272,"kb_used_data":2569296,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":90147880,"statfs":{"total":95995035648,"available":92311429120,"internally_reserved":0,"allocated":2630959104,"data_stored":2630603319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1052637632},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423196,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1358232,"kb_used_data":1159120,"kb_used_omap":4,"kb_used_meta":199099,"kb_avail":92386920,"statfs":{"total":95995035648,"available":94604206080,"internally_reserved":0,"allocated":1186938880,"data_stored":1186583115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":203878080},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234572,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1697332,"kb_used_data":769320,"kb_used_omap":1,"kb_used_meta":927998,"kb_avail":92047820,"statfs":{"total":95995035648,"available":94256967680,"internally_reserved":0,"allocated":787783680,"data_stored":787414027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":950270410},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989020,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92756132,"statfs":{"total":95995035648,"available":94982279168,"internally_reserved":0,"allocated":1204224,"data_stored":844907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575452,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2160920,"kb_used_data":1620240,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91584232,"statfs":{"total":95995035648,"available":93782253568,"internally_reserved":0,"allocated":1659125760,"data_stored":1658769995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738991,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712808,"kb_used_data":766748,"kb_used_omap":1,"kb_used_meta":946046,"kb_avail":92032344,"statfs":{"total":95995035648,"available":94241120256,"internally_reserved":0,"allocated":785149952,"data_stored":784787923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968751557},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959256,"kb_used_data":1296400,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91785896,"statfs":{"total":95995035648,"available":93988757504,"internally_reserved":0,"allocated":1327513600,"data_stored":1327157835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":678754760},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986775,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2589208,"kb_used_data":1569552,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":91155944,"statfs":{"total":95995035648,"available":93343686656,"internally_reserved":0,"allocated":1607221248,"data_stored":1606865483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1044117957},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653336,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2663900,"kb_used_data":2353492,"kb_used_omap":651,"kb_used_meta":309748,"kb_avail":91081252,"statfs":{"total":95995035648,"available":93267202048,"internally_reserved":0,"allocated":2409975808,"data_stored":2409615971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":317182728},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":782565376,"data_stored":782565376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1646526464,"data_stored":1646526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1322647552,"data_stored":1322647552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1606287360,"data_stored":1606287360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2403405824,"data_stored":2403401752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":784470016,"data_stored":784465944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1178206208,"data_stored":1178206208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1349648384,"data_stored":1349648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2470051840,"data_stored":2470051840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:54:19.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:54:19.493 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:19.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:19.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:19 smithi121 ceph-mon[44533]: from='client.38027 v1:172.21.15.40:0/251548464' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:19 smithi121 ceph-mon[44533]: pgmap v2651: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 952 op/s; 8788/193004 objects degraded (4.553%); 14 MiB/s, 224 objects/s recovering 2024-04-29T14:54:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:19 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:19.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:19 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:54:19.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:19.641 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:19.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:19.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:19.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:19 smithi040 ceph-mon[39581]: from='client.38027 v1:172.21.15.40:0/251548464' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:19.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:19 smithi040 ceph-mon[39581]: pgmap v2651: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 952 op/s; 8788/193004 objects degraded (4.553%); 14 MiB/s, 224 objects/s recovering 2024-04-29T14:54:19.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:19.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:19 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:19.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:19 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:54:19.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:19 smithi194 ceph-mon[44089]: from='client.38027 v1:172.21.15.40:0/251548464' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:19 smithi194 ceph-mon[44089]: pgmap v2651: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 952 op/s; 8788/193004 objects degraded (4.553%); 14 MiB/s, 224 objects/s recovering 2024-04-29T14:54:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:19 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:19 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:54:19.848 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:19.876 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:19.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:19.981 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:20.018 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:20.046 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:20.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:20.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:21 smithi194 ceph-mon[44089]: pgmap v2652: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 8450/188028 objects degraded (4.494%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:54:21.864 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:54:21.864 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:54:21.864 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:54:21.864 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:54:21.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:21 smithi040 ceph-mon[39581]: pgmap v2652: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 8450/188028 objects degraded (4.494%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:54:21.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:21 smithi121 ceph-mon[44533]: pgmap v2652: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 951 op/s; 8450/188028 objects degraded (4.494%); 11 MiB/s, 174 objects/s recovering 2024-04-29T14:54:22.093 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:22.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:22.200 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:22.255 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:22.300 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:22.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:22.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:22.442 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:22.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:22.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:22.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:22.655 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:22.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:22.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:54:22.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:22.785 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:22.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:22.811 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:22.862 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:22.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:22.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:22.949 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:22.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:23.077 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:23.111 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:23.148 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:23.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:23.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:23.298 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:23.338 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:23.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:23.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:23.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:23 smithi194 ceph-mon[44089]: pgmap v2653: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 7693/183206 objects degraded (4.199%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:54:23.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:23 smithi194 ceph-mon[44089]: osdmap e1046: 12 total, 11 up, 9 in 2024-04-29T14:54:23.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:23 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 7693/183206 objects degraded (4.199%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:23.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:23.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:23.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:23 smithi040 ceph-mon[39581]: pgmap v2653: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 7693/183206 objects degraded (4.199%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:54:23.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:23 smithi040 ceph-mon[39581]: osdmap e1046: 12 total, 11 up, 9 in 2024-04-29T14:54:23.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:23 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 7693/183206 objects degraded (4.199%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:23 smithi121 ceph-mon[44533]: pgmap v2653: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.6 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 7693/183206 objects degraded (4.199%); 14 MiB/s, 230 objects/s recovering 2024-04-29T14:54:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:23 smithi121 ceph-mon[44533]: osdmap e1046: 12 total, 11 up, 9 in 2024-04-29T14:54:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:23 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 7693/183206 objects degraded (4.199%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:25.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:25.583 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:25.632 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:25.676 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:25.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:25.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:25.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:25 smithi194 ceph-mon[44089]: pgmap v2655: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 980 op/s; 7314/180604 objects degraded (4.050%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:25.830 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:25 smithi040 ceph-mon[39581]: pgmap v2655: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 980 op/s; 7314/180604 objects degraded (4.050%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:25.831 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:25 smithi121 ceph-mon[44533]: pgmap v2655: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 980 op/s; 7314/180604 objects degraded (4.050%); 13 MiB/s, 209 objects/s recovering 2024-04-29T14:54:26.395 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:26.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:26.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:26.538 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:26.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:26.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:26.722 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:27.781 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:27.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:27 smithi194 ceph-mon[44089]: pgmap v2656: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 990 op/s; 6682/176160 objects degraded (3.793%); 15 MiB/s, 234 objects/s recovering 2024-04-29T14:54:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:27 smithi040 ceph-mon[39581]: pgmap v2656: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 990 op/s; 6682/176160 objects degraded (3.793%); 15 MiB/s, 234 objects/s recovering 2024-04-29T14:54:27.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:27 smithi121 ceph-mon[44533]: pgmap v2656: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 990 op/s; 6682/176160 objects degraded (3.793%); 15 MiB/s, 234 objects/s recovering 2024-04-29T14:54:27.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:27.959 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:28.829 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:54:28.829 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:54:28.829 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:54:28.829 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:54:28.829 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:54:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:54:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:28 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 6682/176160 objects degraded (3.793%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2477794741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:28 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:28 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 6682/176160 objects degraded (3.793%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2477794741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:28.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:28 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:28 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 6682/176160 objects degraded (3.793%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2477794741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:28 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: pgmap v2657: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 6464/172882 objects degraded (3.739%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: pgmap v2657: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 6464/172882 objects degraded (3.739%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:29.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: pgmap v2657: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 6464/172882 objects degraded (3.739%); 15 MiB/s, 232 objects/s recovering 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:31.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:31 smithi040 ceph-mon[39581]: pgmap v2658: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 6095/167310 objects degraded (3.643%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:54:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:31 smithi121 ceph-mon[44533]: pgmap v2658: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 6095/167310 objects degraded (3.643%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:54:32.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:31 smithi194 ceph-mon[44089]: pgmap v2658: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 5.1 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.03k op/s; 6095/167310 objects degraded (3.643%); 15 MiB/s, 235 objects/s recovering 2024-04-29T14:54:32.415 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:54:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:32.423 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:54:32.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:32.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:32.578 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:32.851 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:32.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:32.958 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:32.997 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:33.035 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:33.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:33.139 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:33.227 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:33.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:33.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:33.366 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:33.625 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:54:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:33 smithi121 ceph-mon[44533]: pgmap v2659: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5162/162042 objects degraded (3.186%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:54:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:33 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 5162/162042 objects degraded (3.186%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1976666404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:54:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:33 smithi121 ceph-mon[44533]: from='client.38039 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:54:33.650 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:33 smithi040 ceph-mon[39581]: pgmap v2659: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5162/162042 objects degraded (3.186%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:54:33.650 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:33 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 5162/162042 objects degraded (3.186%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:33.650 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1976666404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:54:33.650 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:33 smithi040 ceph-mon[39581]: from='client.38039 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:54:33.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:33.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:33.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:34.014 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:34.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:33 smithi194 ceph-mon[44089]: pgmap v2659: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5162/162042 objects degraded (3.186%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:54:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:33 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 5162/162042 objects degraded (3.186%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1976666404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:54:34.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:33 smithi194 ceph-mon[44089]: from='client.38039 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:54:34.119 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:34.163 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:34.193 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:34.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:34.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:34.343 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:34.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:54:34.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:34 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:34.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:34 smithi040 ceph-mon[39581]: from='client.38039 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:54:34.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:34 smithi040 ceph-mon[39581]: osdmap e1047: 12 total, 11 up, 9 in 2024-04-29T14:54:34.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:34 smithi040 ceph-mon[39581]: pgmap v2661: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4747/159290 objects degraded (2.980%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:54:34.707 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:34.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:34.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:34.859 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:34.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:34 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:34.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:34 smithi121 ceph-mon[44533]: from='client.38039 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:54:34.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:34 smithi121 ceph-mon[44533]: osdmap e1047: 12 total, 11 up, 9 in 2024-04-29T14:54:34.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:34 smithi121 ceph-mon[44533]: pgmap v2661: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4747/159290 objects degraded (2.980%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:54:34.895 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:34.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:34.942 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:54:34.943 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:54:35.001 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:35.040 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:34 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:35.040 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:34 smithi194 ceph-mon[44089]: from='client.38039 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:54:35.040 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:34 smithi194 ceph-mon[44089]: osdmap e1047: 12 total, 11 up, 9 in 2024-04-29T14:54:35.040 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:34 smithi194 ceph-mon[44089]: pgmap v2661: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 4747/159290 objects degraded (2.980%); 16 MiB/s, 256 objects/s recovering 2024-04-29T14:54:35.042 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:35.067 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:35.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:35.087 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:54:35.087 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:54:35.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:35.215 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:35.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:35 smithi040 ceph-mon[39581]: from='client.28261 v1:172.21.15.40:0/1084610904' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:35.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:35 smithi121 ceph-mon[44533]: from='client.28261 v1:172.21.15.40:0/1084610904' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:36.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:35 smithi194 ceph-mon[44089]: from='client.28261 v1:172.21.15.40:0/1084610904' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:54:36.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:54:36.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:36 smithi040 ceph-mon[39581]: pgmap v2662: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 4013/154686 objects degraded (2.594%); 17 MiB/s, 266 objects/s recovering 2024-04-29T14:54:36.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:36 smithi121 ceph-mon[44533]: pgmap v2662: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 4013/154686 objects degraded (2.594%); 17 MiB/s, 266 objects/s recovering 2024-04-29T14:54:37.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:36 smithi194 ceph-mon[44089]: pgmap v2662: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.7 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 4013/154686 objects degraded (2.594%); 17 MiB/s, 266 objects/s recovering 2024-04-29T14:54:37.729 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2661,"stamp":"2024-04-29T14:54:34.361823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5220008504,"num_objects":79645,"num_object_clones":0,"num_object_copies":159290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79645,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":177650,"num_write_kb":8230459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16789,"num_bytes_recovered":1103068896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161008,"ondisk_log_size":161008,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":16764872,"kb_used_data":10002040,"kb_used_omap":666,"kb_used_meta":6761957,"kb_avail":826941496,"statfs":{"total":863955320832,"available":846788091904,"internally_reserved":0,"allocated":10242088960,"data_stored":10238696815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":682449,"internal_metadata":6924244527},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-698417152,"num_objects":-10657,"num_object_clones":0,"num_object_copies":-21314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2567,"num_bytes_recovered":168230912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006127"},"pg_stats":[{"pgid":"8.f","version":"1046'11192","reported_seq":11272,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:31.738728+0000","last_change":"2024-04-29T14:51:58.472775+0000","last_active":"2024-04-29T14:54:31.738728+0000","last_peered":"2024-04-29T14:54:31.738728+0000","last_clean":"2024-04-29T14:54:31.738728+0000","last_became_active":"2024-04-29T14:51:58.471518+0000","last_became_peered":"2024-04-29T14:51:58.471518+0000","last_unstale":"2024-04-29T14:54:31.738728+0000","last_undegraded":"2024-04-29T14:54:31.738728+0000","last_fullsized":"2024-04-29T14:54:31.738728+0000","mapping_epoch":1011,"log_start":"1017'1100","ondisk_log_start":"1017'1100","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326107136,"num_objects":4976,"num_object_clones":0,"num_object_copies":9952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11192,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"1047'11448","reported_seq":11614,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-04-29T14:54:34.298362+0000","last_change":"2024-04-29T14:53:52.432694+0000","last_active":"2024-04-29T14:54:34.298362+0000","last_peered":"2024-04-29T14:54:34.298362+0000","last_clean":"2024-04-29T14:54:34.298362+0000","last_became_active":"2024-04-29T14:51:58.468700+0000","last_became_peered":"2024-04-29T14:51:58.468700+0000","last_unstale":"2024-04-29T14:54:34.298362+0000","last_undegraded":"2024-04-29T14:54:34.298362+0000","last_fullsized":"2024-04-29T14:54:34.298362+0000","mapping_epoch":1011,"log_start":"1017'1400","ondisk_log_start":"1017'1400","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1039'8698","last_scrub_stamp":"2024-04-29T14:53:52.432520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:53:52.432520+0000","objects_scrubbed":7584,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:52.682433+0000","scrub_duration":1228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314703872,"num_objects":4802,"num_object_clones":0,"num_object_copies":9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"1046'11184","reported_seq":11352,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:31.471372+0000","last_change":"2024-04-29T14:53:47.116279+0000","last_active":"2024-04-29T14:54:31.471372+0000","last_peered":"2024-04-29T14:54:31.471372+0000","last_clean":"2024-04-29T14:54:31.471372+0000","last_became_active":"2024-04-29T14:51:58.469439+0000","last_became_peered":"2024-04-29T14:51:58.469439+0000","last_unstale":"2024-04-29T14:54:31.471372+0000","last_undegraded":"2024-04-29T14:54:31.471372+0000","last_fullsized":"2024-04-29T14:54:31.471372+0000","mapping_epoch":1011,"log_start":"1017'1100","ondisk_log_start":"1017'1100","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1038'8337","last_scrub_stamp":"2024-04-29T14:53:47.116014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:53:47.116014+0000","objects_scrubbed":7920,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:08.848827+0000","scrub_duration":1109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330498072,"num_objects":5044,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11184,"num_write_kb":519233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"1046'10948","reported_seq":19846,"reported_epoch":1046,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:30.763854+0000","last_change":"2024-04-29T14:52:59.877502+0000","last_active":"2024-04-29T14:54:30.763854+0000","last_peered":"2024-04-29T14:54:30.763854+0000","last_clean":"2024-04-29T14:52:42.460725+0000","last_became_active":"2024-04-29T14:52:59.820433+0000","last_became_peered":"2024-04-29T14:52:59.820433+0000","last_unstale":"2024-04-29T14:54:30.763854+0000","last_undegraded":"2024-04-29T14:52:59.690774+0000","last_fullsized":"2024-04-29T14:54:30.763854+0000","mapping_epoch":1026,"log_start":"1016'900","ondisk_log_start":"1016'900","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328859648,"num_objects":5018,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4407,"num_bytes_recovered":288817152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8"],"object_location_counts":[{"shards":"3,8","objects":4091},{"shards":"8","objects":927}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"1046'10909","reported_seq":10989,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:30.734970+0000","last_change":"2024-04-29T14:51:58.469525+0000","last_active":"2024-04-29T14:54:30.734970+0000","last_peered":"2024-04-29T14:54:30.734970+0000","last_clean":"2024-04-29T14:54:30.734970+0000","last_became_active":"2024-04-29T14:51:58.469253+0000","last_became_peered":"2024-04-29T14:51:58.469253+0000","last_unstale":"2024-04-29T14:54:30.734970+0000","last_undegraded":"2024-04-29T14:54:30.734970+0000","last_fullsized":"2024-04-29T14:54:30.734970+0000","mapping_epoch":1011,"log_start":"1017'900","ondisk_log_start":"1017'900","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332333056,"num_objects":5071,"num_object_clones":0,"num_object_copies":10142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"1046'11034","reported_seq":19083,"reported_epoch":1046,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:30.763494+0000","last_change":"2024-04-29T14:52:59.845465+0000","last_active":"2024-04-29T14:54:30.763494+0000","last_peered":"2024-04-29T14:54:30.763494+0000","last_clean":"2024-04-29T14:52:42.470937+0000","last_became_active":"2024-04-29T14:52:59.816964+0000","last_became_peered":"2024-04-29T14:52:59.816964+0000","last_unstale":"2024-04-29T14:54:30.763494+0000","last_undegraded":"2024-04-29T14:52:59.688731+0000","last_fullsized":"2024-04-29T14:54:30.763494+0000","mapping_epoch":1026,"log_start":"1017'1000","ondisk_log_start":"1017'1000","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331874304,"num_objects":5064,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11034,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3982,"num_bytes_recovered":260964352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8"],"object_location_counts":[{"shards":"3,8","objects":3695},{"shards":"8","objects":1369}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"1047'11340","reported_seq":11422,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-04-29T14:54:33.680608+0000","last_change":"2024-04-29T14:51:58.470103+0000","last_active":"2024-04-29T14:54:33.680608+0000","last_peered":"2024-04-29T14:54:33.680608+0000","last_clean":"2024-04-29T14:54:33.680608+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:54:33.680608+0000","last_undegraded":"2024-04-29T14:54:33.680608+0000","last_fullsized":"2024-04-29T14:54:33.680608+0000","mapping_epoch":1011,"log_start":"1017'1300","ondisk_log_start":"1017'1300","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319946752,"num_objects":4882,"num_object_clones":0,"num_object_copies":9764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11340,"num_write_kb":519104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"1046'11189","reported_seq":11269,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:31.699954+0000","last_change":"2024-04-29T14:51:58.473278+0000","last_active":"2024-04-29T14:54:31.699954+0000","last_peered":"2024-04-29T14:54:31.699954+0000","last_clean":"2024-04-29T14:54:31.699954+0000","last_became_active":"2024-04-29T14:51:58.473066+0000","last_became_peered":"2024-04-29T14:51:58.473066+0000","last_unstale":"2024-04-29T14:54:31.699954+0000","last_undegraded":"2024-04-29T14:54:31.699954+0000","last_fullsized":"2024-04-29T14:54:31.699954+0000","mapping_epoch":1011,"log_start":"1017'1100","ondisk_log_start":"1017'1100","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322764800,"num_objects":4925,"num_object_clones":0,"num_object_copies":9850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11189,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"1046'10866","reported_seq":10946,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:29.584270+0000","last_change":"2024-04-29T14:51:58.472827+0000","last_active":"2024-04-29T14:54:29.584270+0000","last_peered":"2024-04-29T14:54:29.584270+0000","last_clean":"2024-04-29T14:54:29.584270+0000","last_became_active":"2024-04-29T14:51:58.472520+0000","last_became_peered":"2024-04-29T14:51:58.472520+0000","last_unstale":"2024-04-29T14:54:29.584270+0000","last_undegraded":"2024-04-29T14:54:29.584270+0000","last_fullsized":"2024-04-29T14:54:29.584270+0000","mapping_epoch":1011,"log_start":"1016'800","ondisk_log_start":"1016'800","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334757888,"num_objects":5108,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10866,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"1046'10807","reported_seq":10887,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:29.572861+0000","last_change":"2024-04-29T14:51:58.473051+0000","last_active":"2024-04-29T14:54:29.572861+0000","last_peered":"2024-04-29T14:54:29.572861+0000","last_clean":"2024-04-29T14:54:29.572861+0000","last_became_active":"2024-04-29T14:51:58.472757+0000","last_became_peered":"2024-04-29T14:51:58.472757+0000","last_unstale":"2024-04-29T14:54:29.572861+0000","last_undegraded":"2024-04-29T14:54:29.572861+0000","last_fullsized":"2024-04-29T14:54:29.572861+0000","mapping_epoch":1011,"log_start":"1016'800","ondisk_log_start":"1016'800","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337707008,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"1047'11464","reported_seq":21493,"reported_epoch":1047,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:33.692108+0000","last_change":"2024-04-29T14:52:59.834273+0000","last_active":"2024-04-29T14:54:33.692108+0000","last_peered":"2024-04-29T14:54:33.692108+0000","last_clean":"2024-04-29T14:52:42.476364+0000","last_became_active":"2024-04-29T14:52:59.825361+0000","last_became_peered":"2024-04-29T14:52:59.825361+0000","last_unstale":"2024-04-29T14:54:33.692108+0000","last_undegraded":"2024-04-29T14:52:59.673133+0000","last_fullsized":"2024-04-29T14:54:33.692108+0000","mapping_epoch":1026,"log_start":"1017'1400","ondisk_log_start":"1017'1400","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311427072,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11464,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4971,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":481},{"shards":"5,10","objects":4271}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"1046'10989","reported_seq":11069,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:31.722721+0000","last_change":"2024-04-29T14:51:58.469033+0000","last_active":"2024-04-29T14:54:31.722721+0000","last_peered":"2024-04-29T14:54:31.722721+0000","last_clean":"2024-04-29T14:54:31.722721+0000","last_became_active":"2024-04-29T14:51:58.468800+0000","last_became_peered":"2024-04-29T14:51:58.468800+0000","last_unstale":"2024-04-29T14:54:31.722721+0000","last_undegraded":"2024-04-29T14:54:31.722721+0000","last_fullsized":"2024-04-29T14:54:31.722721+0000","mapping_epoch":1011,"log_start":"1016'900","ondisk_log_start":"1016'900","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324206592,"num_objects":4947,"num_object_clones":0,"num_object_copies":9894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10989,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"1046'10875","reported_seq":10955,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:29.523841+0000","last_change":"2024-04-29T14:51:58.473947+0000","last_active":"2024-04-29T14:54:29.523841+0000","last_peered":"2024-04-29T14:54:29.523841+0000","last_clean":"2024-04-29T14:54:29.523841+0000","last_became_active":"2024-04-29T14:51:58.473704+0000","last_became_peered":"2024-04-29T14:51:58.473704+0000","last_unstale":"2024-04-29T14:54:29.523841+0000","last_undegraded":"2024-04-29T14:54:29.523841+0000","last_fullsized":"2024-04-29T14:54:29.523841+0000","mapping_epoch":1011,"log_start":"1016'800","ondisk_log_start":"1016'800","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332988416,"num_objects":5081,"num_object_clones":0,"num_object_copies":10162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10875,"num_write_kb":510592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"1046'11201","reported_seq":18117,"reported_epoch":1046,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:31.750617+0000","last_change":"2024-04-29T14:52:59.848634+0000","last_active":"2024-04-29T14:54:31.750617+0000","last_peered":"2024-04-29T14:54:31.750617+0000","last_clean":"2024-04-29T14:52:42.467078+0000","last_became_active":"2024-04-29T14:52:59.811137+0000","last_became_peered":"2024-04-29T14:52:59.811137+0000","last_unstale":"2024-04-29T14:54:31.750617+0000","last_undegraded":"2024-04-29T14:52:59.676485+0000","last_fullsized":"2024-04-29T14:54:31.750617+0000","mapping_epoch":1026,"log_start":"1017'1200","ondisk_log_start":"1017'1200","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331546624,"num_objects":5059,"num_object_clones":0,"num_object_copies":10118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11201,"num_write_kb":520320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3415,"num_bytes_recovered":223805440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1970},{"shards":"1,10","objects":3089}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"1047'11250","reported_seq":11332,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-04-29T14:54:33.667463+0000","last_change":"2024-04-29T14:51:58.472256+0000","last_active":"2024-04-29T14:54:33.667463+0000","last_peered":"2024-04-29T14:54:33.667463+0000","last_clean":"2024-04-29T14:54:33.667463+0000","last_became_active":"2024-04-29T14:51:58.471940+0000","last_became_peered":"2024-04-29T14:51:58.471940+0000","last_unstale":"2024-04-29T14:54:33.667463+0000","last_undegraded":"2024-04-29T14:54:33.667463+0000","last_fullsized":"2024-04-29T14:54:33.667463+0000","mapping_epoch":1011,"log_start":"1017'1200","ondisk_log_start":"1017'1200","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"1046'10773","reported_seq":10853,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:29.537305+0000","last_change":"2024-04-29T14:51:58.472764+0000","last_active":"2024-04-29T14:54:29.537305+0000","last_peered":"2024-04-29T14:54:29.537305+0000","last_clean":"2024-04-29T14:54:29.537305+0000","last_became_active":"2024-04-29T14:51:58.472489+0000","last_became_peered":"2024-04-29T14:51:58.472489+0000","last_unstale":"2024-04-29T14:54:29.537305+0000","last_undegraded":"2024-04-29T14:54:29.537305+0000","last_fullsized":"2024-04-29T14:54:29.537305+0000","mapping_epoch":1011,"log_start":"1016'700","ondisk_log_start":"1016'700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320929792,"num_objects":4897,"num_object_clones":0,"num_object_copies":9794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10773,"num_write_kb":501440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2107,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-04-29T14:54:22.549693+0000","last_change":"2024-04-29T14:54:06.451143+0000","last_active":"2024-04-29T14:54:22.549693+0000","last_peered":"2024-04-29T14:54:22.549693+0000","last_clean":"2024-04-29T14:54:22.549693+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:54:22.549693+0000","last_undegraded":"2024-04-29T14:54:22.549693+0000","last_fullsized":"2024-04-29T14:54:22.549693+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:54:06.451003+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:54:06.451003+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:09.012234+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5219418136,"num_objects":79643,"num_object_clones":0,"num_object_copies":159286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177469,"num_write_kb":8227521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16775,"num_bytes_recovered":1099366400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11409235968,"data_stored":11409227824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122620,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384259,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3287676,"kb_used_data":2223476,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":90457476,"statfs":{"total":95995035648,"available":92628455424,"internally_reserved":0,"allocated":2276839424,"data_stored":2276464897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1089599936},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423199,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172284,"kb_used_data":977332,"kb_used_omap":4,"kb_used_meta":194939,"kb_avail":92572868,"statfs":{"total":95995035648,"available":94794616832,"internally_reserved":0,"allocated":1000787968,"data_stored":1000413461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":199618240},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234575,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1563336,"kb_used_data":643648,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92181816,"statfs":{"total":95995035648,"available":94394179584,"internally_reserved":0,"allocated":659095552,"data_stored":658713421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":941750730},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988936,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92756216,"statfs":{"total":95995035648,"available":94982365184,"internally_reserved":0,"allocated":1241088,"data_stored":863029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575456,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1832636,"kb_used_data":1287796,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91912516,"statfs":{"total":95995035648,"available":94118416384,"internally_reserved":0,"allocated":1318703104,"data_stored":1318328597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738994,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1596604,"kb_used_data":644980,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92148548,"statfs":{"total":95995035648,"available":94360113152,"internally_reserved":0,"allocated":660459520,"data_stored":660085013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183014,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1861500,"kb_used_data":1132020,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":91883652,"statfs":{"total":95995035648,"available":94088859648,"internally_reserved":0,"allocated":1159188480,"data_stored":1158813973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":746977736},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986779,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2260236,"kb_used_data":1236416,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":91484916,"statfs":{"total":95995035648,"available":93680553984,"internally_reserved":0,"allocated":1266089984,"data_stored":1265709163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1048377797},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653340,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201664,"kb_used_data":1855160,"kb_used_omap":651,"kb_used_meta":345844,"kb_avail":91543488,"statfs":{"total":95995035648,"available":93740531712,"internally_reserved":0,"allocated":1899683840,"data_stored":1899305261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":354145032},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":656867328,"data_stored":656867328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1304625152,"data_stored":1304625152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1154220032,"data_stored":1154220032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1264582656,"data_stored":1264582656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1891176448,"data_stored":1891172376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":655888384,"data_stored":655884312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":992804864,"data_stored":992804864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1349648384,"data_stored":1349648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2139422720,"data_stored":2139422720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:54:37.730 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.c is not active+clean 2024-04-29T14:54:37.730 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.c', 'version': "1046'10948", 'reported_seq': 19846, 'reported_epoch': 1046, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:54:30.763854+0000', 'last_change': '2024-04-29T14:52:59.877502+0000', 'last_active': '2024-04-29T14:54:30.763854+0000', 'last_peered': '2024-04-29T14:54:30.763854+0000', 'last_clean': '2024-04-29T14:52:42.460725+0000', 'last_became_active': '2024-04-29T14:52:59.820433+0000', 'last_became_peered': '2024-04-29T14:52:59.820433+0000', 'last_unstale': '2024-04-29T14:54:30.763854+0000', 'last_undegraded': '2024-04-29T14:52:59.690774+0000', 'last_fullsized': '2024-04-29T14:54:30.763854+0000', 'mapping_epoch': 1026, 'log_start': "1016'900", 'ondisk_log_start': "1016'900", 'created': 1011, 'last_epoch_clean': 1012, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_clean_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 328859648, 'num_objects': 5018, 'num_object_clones': 0, 'num_object_copies': 10036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 927, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10948, 'num_write_kb': 510912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4407, 'num_bytes_recovered': 288817152, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 3], 'acting': [8, 3], 'avail_no_missing': ['8'], 'object_location_counts': [{'shards': '3,8', 'objects': 4091}, {'shards': '8', 'objects': 927}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-29T14:54:37.731 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.a is not active+clean 2024-04-29T14:54:37.731 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.a', 'version': "1046'11034", 'reported_seq': 19083, 'reported_epoch': 1046, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:54:30.763494+0000', 'last_change': '2024-04-29T14:52:59.845465+0000', 'last_active': '2024-04-29T14:54:30.763494+0000', 'last_peered': '2024-04-29T14:54:30.763494+0000', 'last_clean': '2024-04-29T14:52:42.470937+0000', 'last_became_active': '2024-04-29T14:52:59.816964+0000', 'last_became_peered': '2024-04-29T14:52:59.816964+0000', 'last_unstale': '2024-04-29T14:54:30.763494+0000', 'last_undegraded': '2024-04-29T14:52:59.688731+0000', 'last_fullsized': '2024-04-29T14:54:30.763494+0000', 'mapping_epoch': 1026, 'log_start': "1017'1000", 'ondisk_log_start': "1017'1000", 'created': 1011, 'last_epoch_clean': 1012, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_clean_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 331874304, 'num_objects': 5064, 'num_object_clones': 0, 'num_object_copies': 10128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11034, 'num_write_kb': 515136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3982, 'num_bytes_recovered': 260964352, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 3], 'acting': [8, 3], 'avail_no_missing': ['8'], 'object_location_counts': [{'shards': '3,8', 'objects': 3695}, {'shards': '8', 'objects': 1369}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-29T14:54:37.731 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.4 is not active+clean 2024-04-29T14:54:37.731 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.4', 'version': "1047'11464", 'reported_seq': 21493, 'reported_epoch': 1047, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:54:33.692108+0000', 'last_change': '2024-04-29T14:52:59.834273+0000', 'last_active': '2024-04-29T14:54:33.692108+0000', 'last_peered': '2024-04-29T14:54:33.692108+0000', 'last_clean': '2024-04-29T14:52:42.476364+0000', 'last_became_active': '2024-04-29T14:52:59.825361+0000', 'last_became_peered': '2024-04-29T14:52:59.825361+0000', 'last_unstale': '2024-04-29T14:54:33.692108+0000', 'last_undegraded': '2024-04-29T14:52:59.673133+0000', 'last_fullsized': '2024-04-29T14:54:33.692108+0000', 'mapping_epoch': 1026, 'log_start': "1017'1400", 'ondisk_log_start': "1017'1400", 'created': 1011, 'last_epoch_clean': 1012, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_clean_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 311427072, 'num_objects': 4752, 'num_object_clones': 0, 'num_object_copies': 9504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 481, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11464, 'num_write_kb': 518912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4971, 'num_bytes_recovered': 325779456, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 10], 'acting': [5, 10], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 481}, {'shards': '5,10', 'objects': 4271}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-29T14:54:37.731 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.1 is not active+clean 2024-04-29T14:54:37.731 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.1', 'version': "1046'11201", 'reported_seq': 18117, 'reported_epoch': 1046, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-29T14:54:31.750617+0000', 'last_change': '2024-04-29T14:52:59.848634+0000', 'last_active': '2024-04-29T14:54:31.750617+0000', 'last_peered': '2024-04-29T14:54:31.750617+0000', 'last_clean': '2024-04-29T14:52:42.467078+0000', 'last_became_active': '2024-04-29T14:52:59.811137+0000', 'last_became_peered': '2024-04-29T14:52:59.811137+0000', 'last_unstale': '2024-04-29T14:54:31.750617+0000', 'last_undegraded': '2024-04-29T14:52:59.676485+0000', 'last_fullsized': '2024-04-29T14:54:31.750617+0000', 'mapping_epoch': 1026, 'log_start': "1017'1200", 'ondisk_log_start': "1017'1200", 'created': 1011, 'last_epoch_clean': 1012, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'last_clean_scrub_stamp': '2024-04-29T14:51:57.451341+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 331546624, 'num_objects': 5059, 'num_object_clones': 0, 'num_object_copies': 10118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11201, 'num_write_kb': 520320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3415, 'num_bytes_recovered': 223805440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 10], 'acting': [1, 10], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1970}, {'shards': '1,10', 'objects': 3089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-29T14:54:37.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-29T14:54:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:37.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:37.901 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:37.932 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:37.960 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:37.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:38.066 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:38.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:37 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 4013/154686 objects degraded (2.594%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:38.104 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:38.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:37 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 4013/154686 objects degraded (2.594%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:38.134 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:38.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:37 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 4013/154686 objects degraded (2.594%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:38.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:38.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:54:38.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:38.278 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:38.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:38.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:38.467 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:38.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:38.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:38.606 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:54:38.647 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:38.672 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:54:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:38.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:54:38.779 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:38.824 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:38.963 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:38 smithi194 ceph-mon[44089]: pgmap v2663: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.6 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 3762/151358 objects degraded (2.485%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:54:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:38 smithi040 ceph-mon[39581]: pgmap v2663: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.6 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 3762/151358 objects degraded (2.485%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:54:39.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:38 smithi121 ceph-mon[44533]: pgmap v2663: 17 pgs: 4 active+recovering+degraded, 13 active+clean; 4.6 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 3762/151358 objects degraded (2.485%); 17 MiB/s, 270 objects/s recovering 2024-04-29T14:54:40.103 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:40.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:40.203 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:40.208 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:40.243 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:40.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:40.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:40.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:40.423 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:40.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:40.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:40.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:40.608 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:40.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:40.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:40.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:40.779 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:40.806 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:40.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:40.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:40.976 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:41.004 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:41.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:41.087 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:41 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:41.087 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:41 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:41.087 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:41 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:41.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:41.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:41 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:41.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:41 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:41.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:41 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:41.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:41 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:41.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:41 smithi040 ceph-mon[39581]: osdmap e1048: 12 total, 11 up, 9 in 2024-04-29T14:54:41.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:41 smithi040 ceph-mon[39581]: pgmap v2665: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 3281/145846 objects degraded (2.250%); 15 MiB/s, 234 objects/s recovering 2024-04-29T14:54:41.159 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:41.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:41 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:41.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:41 smithi121 ceph-mon[44533]: osdmap e1048: 12 total, 11 up, 9 in 2024-04-29T14:54:41.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:41 smithi121 ceph-mon[44533]: pgmap v2665: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 3281/145846 objects degraded (2.250%); 15 MiB/s, 234 objects/s recovering 2024-04-29T14:54:41.473 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:41.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:41 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:41.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:41 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:41.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:41 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:41.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:41 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:41.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:41 smithi194 ceph-mon[44089]: osdmap e1048: 12 total, 11 up, 9 in 2024-04-29T14:54:41.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:41 smithi194 ceph-mon[44089]: pgmap v2665: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.5 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 3281/145846 objects degraded (2.250%); 15 MiB/s, 234 objects/s recovering 2024-04-29T14:54:43.693 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: pgmap v2666: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.26k op/s; 2209/140050 objects degraded (1.577%); 21 MiB/s, 337 objects/s recovering 2024-04-29T14:54:43.693 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 2209/140050 objects degraded (1.577%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:43.693 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:54:43.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:54:43.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:54:43.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:43.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:43.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:43.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:43 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:54:43.695 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:43.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:43.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:43.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: pgmap v2666: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.26k op/s; 2209/140050 objects degraded (1.577%); 21 MiB/s, 337 objects/s recovering 2024-04-29T14:54:43.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 2209/140050 objects degraded (1.577%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:43.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:54:43.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:54:43.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:54:43.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:43.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:43.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:43.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:43 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:54:43.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: pgmap v2666: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.26k op/s; 2209/140050 objects degraded (1.577%); 21 MiB/s, 337 objects/s recovering 2024-04-29T14:54:43.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 2209/140050 objects degraded (1.577%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:43.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:54:43.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:54:43.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:54:43.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:43.837 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:43.837 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:43.837 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:43 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T14:54:43.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:43.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:43.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:43.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:44.010 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:44.034 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:44.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:44.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:45.241 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:54:45.272 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:54:45.306 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:54:45.341 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:54:45.379 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:54:45.417 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:54:45.452 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:54:45.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:54:45.514 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:54:45.552 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:54:45.583 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:54:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:54:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:54:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:54:45.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:45 smithi040 ceph-mon[39581]: pgmap v2667: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 2209/137914 objects degraded (1.602%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:54:45.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1602403758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:45 smithi194 ceph-mon[44089]: pgmap v2667: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 2209/137914 objects degraded (1.602%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:54:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1602403758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:45.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:45 smithi121 ceph-mon[44533]: pgmap v2667: 17 pgs: 3 active+recovering+degraded, 14 active+clean; 4.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 2209/137914 objects degraded (1.602%); 16 MiB/s, 253 objects/s recovering 2024-04-29T14:54:45.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1602403758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:46 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:46 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:46.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:46 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:47.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:47.425 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:47.461 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:47.494 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:47.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:47.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:47.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:47.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:47 smithi040 ceph-mon[39581]: pgmap v2668: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 1645/132286 objects degraded (1.244%); 15 MiB/s, 236 objects/s recovering 2024-04-29T14:54:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:47.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:47.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:47.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:47.813 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:47.813 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:47 smithi121 ceph-mon[44533]: pgmap v2668: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 1645/132286 objects degraded (1.244%); 15 MiB/s, 236 objects/s recovering 2024-04-29T14:54:47.816 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:47 smithi194 ceph-mon[44089]: pgmap v2668: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 1645/132286 objects degraded (1.244%); 15 MiB/s, 236 objects/s recovering 2024-04-29T14:54:47.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:47.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:47.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:47.988 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:48.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:48.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:48.202 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:48.693 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:54:48.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:48 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 1645/132286 objects degraded (1.244%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:48.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2007318633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:54:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:48 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 1645/132286 objects degraded (1.244%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2007318633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:54:48.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:48 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 1645/132286 objects degraded (1.244%), 2 pgs degraded (PG_DEGRADED) 2024-04-29T14:54:48.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2007318633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:54:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:54:49.291 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:54:49.374 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:49.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:49.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:49.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:49 smithi040 ceph-mon[39581]: pgmap v2669: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.9 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 1374/128586 objects degraded (1.069%); 15 MiB/s, 238 objects/s recovering 2024-04-29T14:54:49.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:54:49.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:54:49.518 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:49 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:49.519 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2007318633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:54:49.519 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:49 smithi040 ceph-mon[39581]: osdmap e1049: 12 total, 11 up, 9 in 2024-04-29T14:54:49.522 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:49.715 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:49.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:49.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:49.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:49 smithi194 ceph-mon[44089]: pgmap v2669: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.9 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 1374/128586 objects degraded (1.069%); 15 MiB/s, 238 objects/s recovering 2024-04-29T14:54:49.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:54:49.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:54:49.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:49 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:49.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2007318633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:54:49.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:49 smithi194 ceph-mon[44089]: osdmap e1049: 12 total, 11 up, 9 in 2024-04-29T14:54:49.860 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:49.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:49 smithi121 ceph-mon[44533]: pgmap v2669: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.9 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.14k op/s; 1374/128586 objects degraded (1.069%); 15 MiB/s, 238 objects/s recovering 2024-04-29T14:54:49.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:54:49.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:54:49.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:49 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:54:49.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2007318633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:54:49.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:49 smithi121 ceph-mon[44533]: osdmap e1049: 12 total, 11 up, 9 in 2024-04-29T14:54:49.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:49.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:50.003 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:50.038 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:50.068 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:50.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:50.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:50.216 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:50.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:50.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:50.713 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:50.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:50.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:50.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:50.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:50.896 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:51.005 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:54:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:51.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:54:51.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:51.141 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:51.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:51.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:51.290 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:51.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:54:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:51.443 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:51.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:51.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:51 smithi121 ceph-mon[44533]: pgmap v2671: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 1374/122680 objects degraded (1.120%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:54:51.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:51 smithi121 ceph-mon[44533]: 8.8 scrub starts 2024-04-29T14:54:51.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:51 smithi040 ceph-mon[39581]: pgmap v2671: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 1374/122680 objects degraded (1.120%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:54:51.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:51 smithi040 ceph-mon[39581]: 8.8 scrub starts 2024-04-29T14:54:51.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:51.603 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:54:51.603 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1374/122680 objects degraded (1.120%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":3165,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1049,"num_osds":12,"num_up_osds":11,"osd_up_since":1714402362,"num_in_osds":9,"osd_in_since":1714402378,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":17,"num_pools":2,"num_objects":61340,"data_bytes":4020372024,"bytes_used":15032954880,"bytes_avail":848922365952,"bytes_total":863955320832,"degraded_objects":1374,"degraded_total":122680,"degraded_ratio":0.011199869579393545,"recovering_objects_per_sec":190,"recovering_bytes_per_sec":12495237,"recovering_keys_per_sec":0,"num_objects_recovered":1907,"num_bytes_recovered":124977152,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1158},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.40:8443/"}},"servicemap":{"epoch":74,"modified":"2024-04-29T14:53:10.340179+0000","services":{}},"progress_events":{}} 2024-04-29T14:54:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:51.657 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:54:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T14:54:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T14:54:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:54:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:154954 7.c 7:3543cd94:::benchmark_data_smithi194_147200_object40042:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 588.16613435900001, 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041657131999999999, 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154954 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477049+0000", 2024-04-29T14:54:51.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477051+0000", 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.807e-06 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477056+0000", 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1760000000000001e-06 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477067+0000", 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0828e-05 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.477074+0000", 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0990000000000001e-06 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489561+0000", 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012487242000000001 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489704+0000", 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014225699999999999 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.489831+0000", 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012778299999999999 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.497870+0000", 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080388730000000002 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.515675+0000", 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017804868000000001 2024-04-29T14:54:51.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.515755+0000", 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9532999999999999e-05 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518706+0000", 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029516659999999999 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.664 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:154957 7.c 7:39c117cf:::benchmark_data_smithi194_147200_object40045:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 588.16088831800005, 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039817678000000002, 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.665 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 154957 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482295+0000", 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482296+0000", 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.054e-06 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482304+0000", 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5410000000000003e-06 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482329+0000", 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4757999999999999e-05 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.482336+0000", 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6459999999999993e-06 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.493836+0000", 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011499568 2024-04-29T14:54:51.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.493962+0000", 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012657 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.494045+0000", 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3016000000000006e-05 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.498011+0000", 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039652560000000003 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518780+0000", 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020769699999999999 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.518839+0000", 2024-04-29T14:54:51.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8284e-05 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.522113+0000", 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003274285 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.672 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155044 7.c 7:3e86b522:::benchmark_data_smithi194_147200_object40132:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 587.88562876599997, 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038996208999999997, 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155044 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757555+0000", 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757557+0000", 2024-04-29T14:54:51.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.447e-06 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757565+0000", 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2880000000000004e-06 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757578+0000", 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3828999999999999e-05 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.757590+0000", 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1832e-05 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766688+0000", 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090977279999999994 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766861+0000", 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017302100000000001 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.766993+0000", 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000131521 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.787943+0000", 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020950180999999998 2024-04-29T14:54:51.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.793252+0000", 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053089900000000004 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.793316+0000", 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4738000000000003e-05 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:03.796551+0000", 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032346340000000001 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155579 7.c 7:3ca30ab7:::benchmark_data_smithi194_147200_object40667:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 584.44386895100001, 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040246949999999997, 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155579 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199315+0000", 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199317+0000", 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0700000000000001e-06 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199322+0000", 2024-04-29T14:54:51.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.604e-06 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199332+0000", 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9869999999999994e-06 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.199341+0000", 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1009999999999998e-06 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211121+0000", 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011779092 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211232+0000", 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111634 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.211319+0000", 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7120999999999995e-05 2024-04-29T14:54:51.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.216006+0000", 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046872049999999998 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.236426+0000", 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020419567 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.236500+0000", 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4419999999999996e-05 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:07.239562+0000", 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003061149 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:155964 7.c 7:342ad242:::benchmark_data_smithi194_147200_object41052:head [delete] snapc 0=[] ondisk+write+known_if_redirected e845)", 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:54:51.685 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 581.85119004000001, 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038617988999999998, 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 155964 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791994+0000", 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.791996+0000", 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5220000000000002e-06 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792002+0000", 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0700000000000003e-06 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792012+0000", 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0268000000000001e-05 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.792017+0000", 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.994e-06 2024-04-29T14:54:51.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799209+0000", 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0071913070000000001 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799312+0000", 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010362899999999999 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.799433+0000", 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120448 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.804509+0000", 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050758950000000004 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.827370+0000", 2024-04-29T14:54:51.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022861757 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.827437+0000", 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6568000000000004e-05 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:09.830612+0000", 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003174531 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:156496 7.c 7:335b5fcf:::benchmark_data_smithi194_147200_object41584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e846)", 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 577.65150077400006, 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040110869, 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 156496 2024-04-29T14:54:51.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991683+0000", 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991686+0000", 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.613e-06 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991699+0000", 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2316e-05 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991715+0000", 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6147999999999999e-05 2024-04-29T14:54:51.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:13.991730+0000", 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4970000000000001e-05 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001539+0000", 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098087069999999998 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001670+0000", 2024-04-29T14:54:51.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013161299999999999 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.001804+0000", 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013356300000000001 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.014002+0000", 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012198525 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.028772+0000", 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014770086 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.028824+0000", 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1232e-05 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:14.031794+0000", 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029700960000000002 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:157905 7.c 7:3da9029e:::benchmark_data_smithi194_147200_object42993:head [delete] snapc 0=[] ondisk+write+known_if_redirected e850)", 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 567.84196874899999, 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041407046000000003, 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 157905 2024-04-29T14:54:51.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801215+0000", 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801218+0000", 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7939999999999998e-06 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801226+0000", 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4090000000000003e-06 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801238+0000", 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2238999999999999e-05 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.801247+0000", 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8240000000000006e-06 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812668+0000", 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011420397000000001 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812852+0000", 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018423699999999999 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.812973+0000", 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121585 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.824943+0000", 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011969484000000001 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.840398+0000", 2024-04-29T14:54:51.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01545529 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.840441+0000", 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2951000000000003e-05 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:45:23.842622+0000", 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021808359999999998 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164264 7.c 7:3a985e73:::benchmark_data_smithi194_147200_object49352:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 508.47210568899999, 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010643537999999999, 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164264 2024-04-29T14:54:51.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171078+0000", 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171082+0000", 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7680000000000002e-06 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171093+0000", 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0937e-05 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171114+0000", 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1294999999999999e-05 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171127+0000", 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3363999999999999e-05 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171212+0000", 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4429000000000001e-05 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171412+0000", 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020048099999999999 2024-04-29T14:54:51.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.171548+0000", 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013562900000000001 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181610+0000", 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010061746 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181626+0000", 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5973000000000001e-05 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181674+0000", 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8105000000000001e-05 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181721+0000", 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7811e-05 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164265 7.c 7:3e4dad70:::benchmark_data_smithi194_147200_object49353:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 508.46865308000002, 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019274217, 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164265 2024-04-29T14:54:51.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174531+0000", 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174534+0000", 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4429999999999999e-06 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174545+0000", 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0548e-05 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174563+0000", 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8488999999999999e-05 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.174581+0000", 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7569000000000001e-05 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176564+0000", 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019836049999999998 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176739+0000", 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017455199999999999 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 0", 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.176878+0000", 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013903 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.181767+0000", 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048891000000000004 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193663+0000", 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011896105000000001 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193730+0000", 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7065999999999998e-05 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:23.193805+0000", 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4709999999999995e-05 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164330 7.e 7:7a9f05b1:::benchmark_data_smithi194_147200_object49418:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e869)", 2024-04-29T14:54:51.715 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 505.92264548700001, 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15689136400000001, 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164330 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:54:51.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720538+0000", 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720541+0000", 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7659999999999999e-06 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720547+0000", 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2770000000000002e-06 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720561+0000", 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3909e-05 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720571+0000", 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0066e-05 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720646+0000", 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4630999999999997e-05 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720659+0000", 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3281e-05 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872536+0000", 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15187671799999999 2024-04-29T14:54:51.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872672+0000", 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013665500000000001 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9,10", 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.872793+0000", 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012021 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877184+0000", 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043910989999999999 2024-04-29T14:54:51.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877252+0000", 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8238000000000006e-05 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877270+0000", 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7705000000000001e-05 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877326+0000", 2024-04-29T14:54:51.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6563e-05 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877430+0000", 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010324599999999999 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.26998.0:164350 7.e 7:7e0a46a8:::benchmark_data_smithi194_147200_object49438:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e869)", 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 505.92259497399999, 2024-04-29T14:54:51.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.16736200300000001, 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.26998", 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 164350 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720589+0000", 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8400000000000003e-07 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720593+0000", 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2389999999999998e-06 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720599+0000", 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3350000000000003e-06 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720604+0000", 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2959999999999998e-06 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720698+0000", 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3924000000000004e-05 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.720704+0000", 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1480000000000003e-06 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877463+0000", 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.156758704 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877555+0000", 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1974999999999999e-05 2024-04-29T14:54:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9,10", 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.877640+0000", 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5377999999999995e-05 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.881807+0000", 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041667179999999998 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.881835+0000", 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7815999999999999e-05 2024-04-29T14:54:51.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887756+0000", 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059212279999999997 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887822+0000", 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5743999999999999e-05 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:25.887951+0000", 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012871399999999999 2024-04-29T14:54:51.728 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:164988 7.c e879/877 7:333ea8eb:::benchmark_data_smithi194_147200_object50076:head v 879'10294, mlcod=879'10291)", 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 487.93084963000001, 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012683506000000001, 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712334+0000", 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712335+0000", 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4799999999999998e-07 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712340+0000", 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9910000000000002e-06 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712355+0000", 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5591e-05 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.712360+0000", 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1139999999999997e-06 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.716296+0000", 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039360840000000003 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.716327+0000", 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0960999999999997e-05 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.724966+0000", 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086382809999999994 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:43.725018+0000", 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1935999999999999e-05 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:165418 7.a e882/877 7:5a70a2d7:::benchmark_data_smithi194_147200_object50506:head v 882'17678, mlcod=881'17676)", 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 483.78066427200002, 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011664525, 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.733 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862519+0000", 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862520+0000", 2024-04-29T14:54:51.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1699999999999999e-07 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862527+0000", 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0779999999999998e-06 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862543+0000", 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5698000000000001e-05 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862553+0000", 2024-04-29T14:54:51.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0301e-05 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862624+0000", 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.148e-05 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.862657+0000", 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2861000000000003e-05 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.874126+0000", 2024-04-29T14:54:51.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011468580000000001 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.874184+0000", 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8109999999999998e-05 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.26998.0:165421 7.2 e882/877 7:4462835d:::benchmark_data_smithi194_147200_object50509:head v 882'10201, mlcod=881'10199)", 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 483.774044167, 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013165216, 2024-04-29T14:54:51.737 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869139+0000", 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869140+0000", 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8799999999999998e-07 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869147+0000", 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1590000000000004e-06 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869167+0000", 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.014e-05 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.869177+0000", 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0088999999999999e-05 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.873690+0000", 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045127489999999999 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.873721+0000", 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1281000000000001e-05 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.882230+0000", 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0085086999999999992 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:46:47.882305+0000", 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4709999999999995e-05 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:54:51.741 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:54:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:51.742 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:51.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:51.755 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:54:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:51 smithi194 ceph-mon[44089]: pgmap v2671: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 1374/122680 objects degraded (1.120%); 12 MiB/s, 190 objects/s recovering 2024-04-29T14:54:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:51 smithi194 ceph-mon[44089]: 8.8 scrub starts 2024-04-29T14:54:51.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:51.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:51.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:52.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:52.052 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:52.088 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:52.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:52.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:54:52.243 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:52.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:54:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:52.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:54:52.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:54:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:52.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:52 smithi040 ceph-mon[39581]: 8.8 scrub ok 2024-04-29T14:54:52.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2843158093' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:54:52.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:52 smithi040 ceph-mon[39581]: 8.4 scrub starts 2024-04-29T14:54:52.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:52 smithi121 ceph-mon[44533]: 8.8 scrub ok 2024-04-29T14:54:52.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2843158093' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:54:52.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:52 smithi121 ceph-mon[44533]: 8.4 scrub starts 2024-04-29T14:54:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:52 smithi194 ceph-mon[44089]: 8.8 scrub ok 2024-04-29T14:54:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2843158093' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-29T14:54:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:52 smithi194 ceph-mon[44089]: 8.4 scrub starts 2024-04-29T14:54:53.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:53 smithi194 ceph-mon[44089]: pgmap v2672: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/116690 objects degraded (0.719%); 8.6 MiB/s, 136 objects/s recovering 2024-04-29T14:54:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:53 smithi194 ceph-mon[44089]: 8.4 scrub ok 2024-04-29T14:54:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:53 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 839/116690 objects degraded (0.719%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:54:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:53 smithi194 ceph-mon[44089]: 8.e scrub starts 2024-04-29T14:54:53.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:53 smithi040 ceph-mon[39581]: pgmap v2672: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/116690 objects degraded (0.719%); 8.6 MiB/s, 136 objects/s recovering 2024-04-29T14:54:53.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:53 smithi040 ceph-mon[39581]: 8.4 scrub ok 2024-04-29T14:54:53.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:53 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 839/116690 objects degraded (0.719%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:54:53.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:53 smithi040 ceph-mon[39581]: 8.e scrub starts 2024-04-29T14:54:53.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:53 smithi121 ceph-mon[44533]: pgmap v2672: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/116690 objects degraded (0.719%); 8.6 MiB/s, 136 objects/s recovering 2024-04-29T14:54:53.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:53 smithi121 ceph-mon[44533]: 8.4 scrub ok 2024-04-29T14:54:53.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:53 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 839/116690 objects degraded (0.719%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:54:53.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:53 smithi121 ceph-mon[44533]: 8.e scrub starts 2024-04-29T14:54:54.003 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:54:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:54.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:54:54.034 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-29T14:54:54.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:54:54.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:54:54.168 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:54.629 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:54.638 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:54:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:54.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:54:54.680 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:54:54.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:54.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:54 smithi040 ceph-mon[39581]: 8.e scrub ok 2024-04-29T14:54:54.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:54 smithi040 ceph-mon[39581]: 8.c scrub starts 2024-04-29T14:54:54.786 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:54.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:54 smithi194 ceph-mon[44089]: 8.e scrub ok 2024-04-29T14:54:54.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:54 smithi194 ceph-mon[44089]: 8.c scrub starts 2024-04-29T14:54:54.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:54 smithi121 ceph-mon[44533]: 8.e scrub ok 2024-04-29T14:54:54.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:54 smithi121 ceph-mon[44533]: 8.c scrub starts 2024-04-29T14:54:54.927 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:54:55.064 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:54:55.158 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:55.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:55.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:55.486 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:55 smithi040 ceph-mon[39581]: pgmap v2673: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/114452 objects degraded (0.733%); 8.6 MiB/s, 136 objects/s recovering 2024-04-29T14:54:55.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:55 smithi194 ceph-mon[44089]: pgmap v2673: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/114452 objects degraded (0.733%); 8.6 MiB/s, 136 objects/s recovering 2024-04-29T14:54:55.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:55 smithi194 ceph-mon[44089]: 8.c scrub ok 2024-04-29T14:54:55.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:55 smithi040 ceph-mon[39581]: 8.c scrub ok 2024-04-29T14:54:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:55 smithi121 ceph-mon[44533]: pgmap v2673: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/114452 objects degraded (0.733%); 8.6 MiB/s, 136 objects/s recovering 2024-04-29T14:54:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:55 smithi121 ceph-mon[44533]: 8.c scrub ok 2024-04-29T14:54:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:56.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:56.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:56.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:57.052 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:57.095 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:57.128 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:57.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:57.232 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:54:57.274 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:57.311 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:54:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:57.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:54:57.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:57.462 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:57.467 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:54:57.467 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:54:57.467 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:54:57.467 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:54:57.487 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:57.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:57.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:57 smithi040 ceph-mon[39581]: pgmap v2674: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/108788 objects degraded (0.771%); 5.0 MiB/s, 80 objects/s recovering 2024-04-29T14:54:57.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:57 smithi040 ceph-mon[39581]: 8.b scrub starts 2024-04-29T14:54:57.495 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:57 smithi040 ceph-mon[39581]: 8.b scrub ok 2024-04-29T14:54:57.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:57 smithi194 ceph-mon[44089]: pgmap v2674: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/108788 objects degraded (0.771%); 5.0 MiB/s, 80 objects/s recovering 2024-04-29T14:54:57.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:57 smithi194 ceph-mon[44089]: 8.b scrub starts 2024-04-29T14:54:57.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:57 smithi194 ceph-mon[44089]: 8.b scrub ok 2024-04-29T14:54:57.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:57.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:57.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:54:57.811 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:57.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:57 smithi121 ceph-mon[44533]: pgmap v2674: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.3 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 839/108788 objects degraded (0.771%); 5.0 MiB/s, 80 objects/s recovering 2024-04-29T14:54:57.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:57 smithi121 ceph-mon[44533]: 8.b scrub starts 2024-04-29T14:54:57.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:57 smithi121 ceph-mon[44533]: 8.b scrub ok 2024-04-29T14:54:57.975 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:54:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:57.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:54:58.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:58.126 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:58.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:58.189 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:54:58.226 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:54:58.252 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:54:58.260 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:54:58.275 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:54:58.297 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:54:58.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:54:58.307 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:54:58.344 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:54:58.351 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:54:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:58.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:54:58.380 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:54:58.419 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:54:58.455 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:54:58.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:54:58.488 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:54:58.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:54:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:54:58.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:54:58.492 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:54:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:58.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:54:58.578 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 839/108788 objects degraded (0.771%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:54:58.578 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2867711035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:58.578 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:58.578 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:54:58.578 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:54:58.578 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:54:58.579 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:58.579 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:58.579 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:58.579 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:58.579 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:54:58.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:54:58.693 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:54:58.693 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:58.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:54:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:58.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:54:58.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:54:58.866 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 839/108788 objects degraded (0.771%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:54:58.866 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2867711035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:58.866 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:58.866 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:54:58.866 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:54:58.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:54:58.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:58.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:58.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:58.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:58.867 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:54:58.868 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 839/108788 objects degraded (0.771%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:54:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2867711035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:54:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:54:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:54:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:54:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:54:58.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:54:59.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:54:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:54:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:54:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:54:59.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:54:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:54:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:54:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:54:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:00.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:55:00.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:55:00.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:00.774 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:00.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:00.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:00.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:00.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:01.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:01.127 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:01.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:01.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:01.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:01.319 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:01.499 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:01 smithi194 ceph-mon[44089]: from='client.38060 v1:172.21.15.40:0/463198806' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:55:01.499 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:01 smithi194 ceph-mon[44089]: pgmap v2676: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s; 447/99232 objects degraded (0.450%); 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:55:01.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:01 smithi040 ceph-mon[39581]: from='client.38060 v1:172.21.15.40:0/463198806' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:55:01.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:01 smithi040 ceph-mon[39581]: pgmap v2676: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s; 447/99232 objects degraded (0.450%); 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:55:01.790 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2675,"stamp":"2024-04-29T14:54:58.368637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3441754680,"num_objects":52511,"num_object_clones":0,"num_object_copies":105022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52511,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":204784,"num_write_kb":8230459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21089,"num_bytes_recovered":1384873696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":13661820,"kb_used_data":6861296,"kb_used_omap":666,"kb_used_meta":6799461,"kb_avail":830044548,"statfs":{"total":863955320832,"available":849965617152,"internally_reserved":0,"allocated":7025967104,"data_stored":7022412145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":682449,"internal_metadata":6962648623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,8,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-772145152,"num_objects":-11782,"num_object_clones":0,"num_object_copies":-23564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":927,"num_bytes_recovered":60751872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003678"},"pg_stats":[{"pgid":"8.f","version":"1049'12988","reported_seq":13074,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:56.784340+0000","last_change":"2024-04-29T14:51:58.472775+0000","last_active":"2024-04-29T14:54:56.784340+0000","last_peered":"2024-04-29T14:54:56.784340+0000","last_clean":"2024-04-29T14:54:56.784340+0000","last_became_active":"2024-04-29T14:51:58.471518+0000","last_became_peered":"2024-04-29T14:51:58.471518+0000","last_unstale":"2024-04-29T14:54:56.784340+0000","last_undegraded":"2024-04-29T14:54:56.784340+0000","last_fullsized":"2024-04-29T14:54:56.784340+0000","mapping_epoch":1011,"log_start":"1017'2900","ondisk_log_start":"1017'2900","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208404480,"num_objects":3180,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12988,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"1049'12848","reported_seq":13061,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:54.315811+0000","last_change":"2024-04-29T14:54:53.972810+0000","last_active":"2024-04-29T14:54:54.315811+0000","last_peered":"2024-04-29T14:54:54.315811+0000","last_clean":"2024-04-29T14:54:54.315811+0000","last_became_active":"2024-04-29T14:51:58.468700+0000","last_became_peered":"2024-04-29T14:51:58.468700+0000","last_unstale":"2024-04-29T14:54:54.315811+0000","last_undegraded":"2024-04-29T14:54:54.315811+0000","last_fullsized":"2024-04-29T14:54:54.315811+0000","mapping_epoch":1011,"log_start":"1017'2800","ondisk_log_start":"1017'2800","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12818","last_scrub_stamp":"2024-04-29T14:54:53.972600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:53.972600+0000","objects_scrubbed":3458,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:03.208154+0000","scrub_duration":713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222953472,"num_objects":3402,"num_object_clones":0,"num_object_copies":6804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12848,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"1049'13025","reported_seq":13199,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:56.412076+0000","last_change":"2024-04-29T14:53:47.116279+0000","last_active":"2024-04-29T14:54:56.412076+0000","last_peered":"2024-04-29T14:54:56.412076+0000","last_clean":"2024-04-29T14:54:56.412076+0000","last_became_active":"2024-04-29T14:51:58.469439+0000","last_became_peered":"2024-04-29T14:51:58.469439+0000","last_unstale":"2024-04-29T14:54:56.412076+0000","last_undegraded":"2024-04-29T14:54:56.412076+0000","last_fullsized":"2024-04-29T14:54:56.412076+0000","mapping_epoch":1011,"log_start":"1017'3000","ondisk_log_start":"1017'3000","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1038'8337","last_scrub_stamp":"2024-04-29T14:53:47.116014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:53:47.116014+0000","objects_scrubbed":7920,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:08.848827+0000","scrub_duration":1109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209846296,"num_objects":3203,"num_object_clones":0,"num_object_copies":6406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13025,"num_write_kb":519233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"1049'12804","reported_seq":23605,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:55.754272+0000","last_change":"2024-04-29T14:54:55.056447+0000","last_active":"2024-04-29T14:54:55.754272+0000","last_peered":"2024-04-29T14:54:55.754272+0000","last_clean":"2024-04-29T14:54:55.754272+0000","last_became_active":"2024-04-29T14:52:59.820433+0000","last_became_peered":"2024-04-29T14:52:59.820433+0000","last_unstale":"2024-04-29T14:54:55.754272+0000","last_undegraded":"2024-04-29T14:54:55.754272+0000","last_fullsized":"2024-04-29T14:54:55.754272+0000","mapping_epoch":1026,"log_start":"1017'2800","ondisk_log_start":"1017'2800","created":1011,"last_epoch_clean":1048,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12753","last_scrub_stamp":"2024-04-29T14:54:55.056281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:55.056281+0000","objects_scrubbed":3247,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:23.980689+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207224832,"num_objects":3162,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12804,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5334,"num_bytes_recovered":349569024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"1049'12634","reported_seq":12719,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:55.762517+0000","last_change":"2024-04-29T14:51:58.469525+0000","last_active":"2024-04-29T14:54:55.762517+0000","last_peered":"2024-04-29T14:54:55.762517+0000","last_clean":"2024-04-29T14:54:55.762517+0000","last_became_active":"2024-04-29T14:51:58.469253+0000","last_became_peered":"2024-04-29T14:51:58.469253+0000","last_unstale":"2024-04-29T14:54:55.762517+0000","last_undegraded":"2024-04-29T14:54:55.762517+0000","last_fullsized":"2024-04-29T14:54:55.762517+0000","mapping_epoch":1011,"log_start":"1017'2600","ondisk_log_start":"1017'2600","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219283456,"num_objects":3346,"num_object_clones":0,"num_object_copies":6692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12634,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"1049'12806","reported_seq":23602,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:55.754092+0000","last_change":"2024-04-29T14:54:47.523341+0000","last_active":"2024-04-29T14:54:55.754092+0000","last_peered":"2024-04-29T14:54:55.754092+0000","last_clean":"2024-04-29T14:54:55.754092+0000","last_became_active":"2024-04-29T14:52:59.816964+0000","last_became_peered":"2024-04-29T14:52:59.816964+0000","last_unstale":"2024-04-29T14:54:55.754092+0000","last_undegraded":"2024-04-29T14:54:55.754092+0000","last_fullsized":"2024-04-29T14:54:55.754092+0000","mapping_epoch":1026,"log_start":"1017'2800","ondisk_log_start":"1017'2800","created":1011,"last_epoch_clean":1048,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215744512,"num_objects":3292,"num_object_clones":0,"num_object_copies":6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5351,"num_bytes_recovered":350683136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"1049'12792","reported_seq":12922,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:53.714448+0000","last_change":"2024-04-29T14:54:51.498258+0000","last_active":"2024-04-29T14:54:53.714448+0000","last_peered":"2024-04-29T14:54:53.714448+0000","last_clean":"2024-04-29T14:54:53.714448+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:54:53.714448+0000","last_undegraded":"2024-04-29T14:54:53.714448+0000","last_fullsized":"2024-04-29T14:54:53.714448+0000","mapping_epoch":1011,"log_start":"1017'2700","ondisk_log_start":"1017'2700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12636","last_scrub_stamp":"2024-04-29T14:54:51.498114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:51.498114+0000","objects_scrubbed":3603,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:57.798038+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12792,"num_write_kb":519104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"1049'12974","reported_seq":13060,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:56.764561+0000","last_change":"2024-04-29T14:51:58.473278+0000","last_active":"2024-04-29T14:54:56.764561+0000","last_peered":"2024-04-29T14:54:56.764561+0000","last_clean":"2024-04-29T14:54:56.764561+0000","last_became_active":"2024-04-29T14:51:58.473066+0000","last_became_peered":"2024-04-29T14:51:58.473066+0000","last_unstale":"2024-04-29T14:54:56.764561+0000","last_undegraded":"2024-04-29T14:54:56.764561+0000","last_fullsized":"2024-04-29T14:54:56.764561+0000","mapping_epoch":1011,"log_start":"1017'2900","ondisk_log_start":"1017'2900","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205783040,"num_objects":3140,"num_object_clones":0,"num_object_copies":6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12974,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"1049'12616","reported_seq":12702,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:54.604969+0000","last_change":"2024-04-29T14:51:58.472827+0000","last_active":"2024-04-29T14:54:54.604969+0000","last_peered":"2024-04-29T14:54:54.604969+0000","last_clean":"2024-04-29T14:54:54.604969+0000","last_became_active":"2024-04-29T14:51:58.472520+0000","last_became_peered":"2024-04-29T14:51:58.472520+0000","last_unstale":"2024-04-29T14:54:54.604969+0000","last_undegraded":"2024-04-29T14:54:54.604969+0000","last_fullsized":"2024-04-29T14:54:54.604969+0000","mapping_epoch":1011,"log_start":"1017'2600","ondisk_log_start":"1017'2600","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220069888,"num_objects":3358,"num_object_clones":0,"num_object_copies":6716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12616,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"1049'12559","reported_seq":12645,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:54.621981+0000","last_change":"2024-04-29T14:51:58.473051+0000","last_active":"2024-04-29T14:54:54.621981+0000","last_peered":"2024-04-29T14:54:54.621981+0000","last_clean":"2024-04-29T14:54:54.621981+0000","last_became_active":"2024-04-29T14:51:58.472757+0000","last_became_peered":"2024-04-29T14:51:58.472757+0000","last_unstale":"2024-04-29T14:54:54.621981+0000","last_undegraded":"2024-04-29T14:54:54.621981+0000","last_fullsized":"2024-04-29T14:54:54.621981+0000","mapping_epoch":1011,"log_start":"1017'2500","ondisk_log_start":"1017'2500","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222887936,"num_objects":3401,"num_object_clones":0,"num_object_copies":6802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12559,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"1049'12907","reported_seq":23948,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:53.701720+0000","last_change":"2024-04-29T14:54:52.411446+0000","last_active":"2024-04-29T14:54:53.701720+0000","last_peered":"2024-04-29T14:54:53.701720+0000","last_clean":"2024-04-29T14:54:53.701720+0000","last_became_active":"2024-04-29T14:52:59.825361+0000","last_became_peered":"2024-04-29T14:52:59.825361+0000","last_unstale":"2024-04-29T14:54:53.701720+0000","last_undegraded":"2024-04-29T14:54:53.701720+0000","last_fullsized":"2024-04-29T14:54:53.701720+0000","mapping_epoch":1026,"log_start":"1017'2900","ondisk_log_start":"1017'2900","created":1011,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12798","last_scrub_stamp":"2024-04-29T14:54:52.411348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:52.411348+0000","objects_scrubbed":3449,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:19.646039+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216858624,"num_objects":3309,"num_object_clones":0,"num_object_copies":6618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12907,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5452,"num_bytes_recovered":357302272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"1049'12776","reported_seq":12862,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:56.783336+0000","last_change":"2024-04-29T14:51:58.469033+0000","last_active":"2024-04-29T14:54:56.783336+0000","last_peered":"2024-04-29T14:54:56.783336+0000","last_clean":"2024-04-29T14:54:56.783336+0000","last_became_active":"2024-04-29T14:51:58.468800+0000","last_became_peered":"2024-04-29T14:51:58.468800+0000","last_unstale":"2024-04-29T14:54:56.783336+0000","last_undegraded":"2024-04-29T14:54:56.783336+0000","last_fullsized":"2024-04-29T14:54:56.783336+0000","mapping_epoch":1011,"log_start":"1017'2700","ondisk_log_start":"1017'2700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207093760,"num_objects":3160,"num_object_clones":0,"num_object_copies":6320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12776,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"1049'12709","reported_seq":12795,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:54.612955+0000","last_change":"2024-04-29T14:51:58.473947+0000","last_active":"2024-04-29T14:54:54.612955+0000","last_peered":"2024-04-29T14:54:54.612955+0000","last_clean":"2024-04-29T14:54:54.612955+0000","last_became_active":"2024-04-29T14:51:58.473704+0000","last_became_peered":"2024-04-29T14:51:58.473704+0000","last_unstale":"2024-04-29T14:54:54.612955+0000","last_undegraded":"2024-04-29T14:54:54.612955+0000","last_fullsized":"2024-04-29T14:54:54.612955+0000","mapping_epoch":1011,"log_start":"1017'2700","ondisk_log_start":"1017'2700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212795392,"num_objects":3247,"num_object_clones":0,"num_object_copies":6494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12709,"num_write_kb":510592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"1049'13060","reported_seq":23028,"reported_epoch":1049,"state":"active+recovering+degraded","last_fresh":"2024-04-29T14:54:56.796090+0000","last_change":"2024-04-29T14:52:59.848634+0000","last_active":"2024-04-29T14:54:56.796090+0000","last_peered":"2024-04-29T14:54:56.796090+0000","last_clean":"2024-04-29T14:52:42.467078+0000","last_became_active":"2024-04-29T14:52:59.811137+0000","last_became_peered":"2024-04-29T14:52:59.811137+0000","last_unstale":"2024-04-29T14:54:56.796090+0000","last_undegraded":"2024-04-29T14:52:59.676485+0000","last_fullsized":"2024-04-29T14:54:56.796090+0000","mapping_epoch":1026,"log_start":"1017'3000","ondisk_log_start":"1017'3000","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":6400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13060,"num_write_kb":520320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4938,"num_bytes_recovered":323616768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":447},{"shards":"1,10","objects":2753}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"1049'12691","reported_seq":12776,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:53.426626+0000","last_change":"2024-04-29T14:51:58.472256+0000","last_active":"2024-04-29T14:54:53.426626+0000","last_peered":"2024-04-29T14:54:53.426626+0000","last_clean":"2024-04-29T14:54:53.426626+0000","last_became_active":"2024-04-29T14:51:58.471940+0000","last_became_peered":"2024-04-29T14:51:58.471940+0000","last_unstale":"2024-04-29T14:54:53.426626+0000","last_undegraded":"2024-04-29T14:54:53.426626+0000","last_fullsized":"2024-04-29T14:54:53.426626+0000","mapping_epoch":1011,"log_start":"1017'2600","ondisk_log_start":"1017'2600","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224329728,"num_objects":3423,"num_object_clones":0,"num_object_copies":6846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12691,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"1049'12414","reported_seq":12500,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:54.598351+0000","last_change":"2024-04-29T14:51:58.472764+0000","last_active":"2024-04-29T14:54:54.598351+0000","last_peered":"2024-04-29T14:54:54.598351+0000","last_clean":"2024-04-29T14:54:54.598351+0000","last_became_active":"2024-04-29T14:51:58.472489+0000","last_became_peered":"2024-04-29T14:51:58.472489+0000","last_unstale":"2024-04-29T14:54:54.598351+0000","last_undegraded":"2024-04-29T14:54:54.598351+0000","last_fullsized":"2024-04-29T14:54:54.598351+0000","mapping_epoch":1011,"log_start":"1017'2400","ondisk_log_start":"1017'2400","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213385216,"num_objects":3256,"num_object_clones":0,"num_object_copies":6512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":501440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2113,"reported_epoch":1049,"state":"active+clean","last_fresh":"2024-04-29T14:54:48.690899+0000","last_change":"2024-04-29T14:54:06.451143+0000","last_active":"2024-04-29T14:54:48.690899+0000","last_peered":"2024-04-29T14:54:48.690899+0000","last_clean":"2024-04-29T14:54:48.690899+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:54:48.690899+0000","last_undegraded":"2024-04-29T14:54:48.690899+0000","last_fullsized":"2024-04-29T14:54:48.690899+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:54:06.451003+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:54:06.451003+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:09.012234+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3441164312,"num_objects":52509,"num_object_clones":0,"num_object_copies":105018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":447,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204603,"num_write_kb":8227521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":1381171200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8249483264,"data_stored":8249475120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122625,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384264,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2594976,"kb_used_data":1521112,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":91150176,"statfs":{"total":95995035648,"available":93337780224,"internally_reserved":0,"allocated":1557618688,"data_stored":1557225369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1099561408},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423204,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":851424,"kb_used_data":634264,"kb_used_omap":4,"kb_used_meta":217147,"kb_avail":92893728,"statfs":{"total":95995035648,"available":95123177472,"internally_reserved":0,"allocated":649486336,"data_stored":649093037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":222359232},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234580,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328172,"kb_used_data":412644,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92416980,"statfs":{"total":95995035648,"available":94634987520,"internally_reserved":0,"allocated":422547456,"data_stored":422146533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":937490890},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988972,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92756180,"statfs":{"total":95995035648,"available":94982328320,"internally_reserved":0,"allocated":1277952,"data_stored":881101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575461,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1365408,"kb_used_data":828888,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92379744,"statfs":{"total":95995035648,"available":94596857856,"internally_reserved":0,"allocated":848781312,"data_stored":848388013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739000,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1357856,"kb_used_data":406232,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92387296,"statfs":{"total":95995035648,"available":94604591104,"internally_reserved":0,"allocated":415981568,"data_stored":415588269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183018,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1608480,"kb_used_data":883160,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92136672,"statfs":{"total":95995035648,"available":94347952128,"internally_reserved":0,"allocated":904355840,"data_stored":903962541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":742717896},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986783,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1902368,"kb_used_data":874392,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":91842784,"statfs":{"total":95995035648,"available":94047010816,"internally_reserved":0,"allocated":895377408,"data_stored":894984109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1052637637},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653344,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1664164,"kb_used_data":1299356,"kb_used_omap":651,"kb_used_meta":363892,"kb_avail":92080988,"statfs":{"total":95995035648,"available":94290931712,"internally_reserved":0,"allocated":1330540544,"data_stored":1330143173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":372626184},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":411959296,"data_stored":411959296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":830930944,"data_stored":830930944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":895287296,"data_stored":895287296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":893059072,"data_stored":893059072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1329467392,"data_stored":1329463320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":419106816,"data_stored":419102744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":642252800,"data_stored":642252800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1349648384,"data_stored":1349648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1477771264,"data_stored":1477771264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:55:01.791 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:01.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:01.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:01 smithi121 ceph-mon[44533]: from='client.38060 v1:172.21.15.40:0/463198806' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:55:01.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:01 smithi121 ceph-mon[44533]: pgmap v2676: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.0 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s; 447/99232 objects degraded (0.450%); 5.0 MiB/s, 79 objects/s recovering 2024-04-29T14:55:01.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:01.931 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:01.966 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:01.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:02.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:02.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:02.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:02.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:02.283 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:02.311 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:02.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:02.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:02.481 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:02.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:02.514 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:55:02.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2267074256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:02.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:02 smithi040 ceph-mon[39581]: 8.0 scrub starts 2024-04-29T14:55:02.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:02.626 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:02.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:55:02.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2267074256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:02.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:02 smithi194 ceph-mon[44089]: 8.0 scrub starts 2024-04-29T14:55:02.840 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:02.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2267074256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:02 smithi121 ceph-mon[44533]: 8.0 scrub starts 2024-04-29T14:55:02.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:02.979 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:55:02.979 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:55:02.996 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:03.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:03.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:03.617 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:03.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:03 smithi194 ceph-mon[44089]: pgmap v2677: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s; 74/93330 objects degraded (0.079%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:55:03.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:03 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:55:03.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2267074256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:03.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:03 smithi194 ceph-mon[44089]: osdmap e1050: 12 total, 11 up, 9 in 2024-04-29T14:55:03.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:03 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 74/93330 objects degraded (0.079%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:55:03.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:03 smithi194 ceph-mon[44089]: 8.0 scrub ok 2024-04-29T14:55:03.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:03 smithi194 ceph-mon[44089]: 8.7 scrub starts 2024-04-29T14:55:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:03 smithi040 ceph-mon[39581]: pgmap v2677: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s; 74/93330 objects degraded (0.079%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:55:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:03 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:55:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2267074256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:03 smithi040 ceph-mon[39581]: osdmap e1050: 12 total, 11 up, 9 in 2024-04-29T14:55:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:03 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 74/93330 objects degraded (0.079%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:55:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:03 smithi040 ceph-mon[39581]: 8.0 scrub ok 2024-04-29T14:55:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:03 smithi040 ceph-mon[39581]: 8.7 scrub starts 2024-04-29T14:55:03.883 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:55:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:03 smithi121 ceph-mon[44533]: pgmap v2677: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.22k op/s; 74/93330 objects degraded (0.079%); 6.8 MiB/s, 108 objects/s recovering 2024-04-29T14:55:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:03 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:55:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2267074256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:03 smithi121 ceph-mon[44533]: osdmap e1050: 12 total, 11 up, 9 in 2024-04-29T14:55:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:03 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 74/93330 objects degraded (0.079%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:55:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:03 smithi121 ceph-mon[44533]: 8.0 scrub ok 2024-04-29T14:55:03.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:03 smithi121 ceph-mon[44533]: 8.7 scrub starts 2024-04-29T14:55:03.979 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:03.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:04.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:04.130 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:04.264 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:55:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:04.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:04.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:04.745 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:04.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:04.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:04.792 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:55:04.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:04 smithi194 ceph-mon[44089]: 8.a scrub starts 2024-04-29T14:55:04.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:04 smithi194 ceph-mon[44089]: 8.6 scrub starts 2024-04-29T14:55:04.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:04 smithi194 ceph-mon[44089]: 8.7 scrub ok 2024-04-29T14:55:04.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:04 smithi194 ceph-mon[44089]: 8.1 scrub starts 2024-04-29T14:55:04.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:04 smithi194 ceph-mon[44089]: 8.a scrub ok 2024-04-29T14:55:04.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:04 smithi194 ceph-mon[44089]: 8.6 scrub ok 2024-04-29T14:55:04.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:04 smithi040 ceph-mon[39581]: 8.a scrub starts 2024-04-29T14:55:04.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:04 smithi040 ceph-mon[39581]: 8.6 scrub starts 2024-04-29T14:55:04.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:04 smithi040 ceph-mon[39581]: 8.7 scrub ok 2024-04-29T14:55:04.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:04 smithi040 ceph-mon[39581]: 8.1 scrub starts 2024-04-29T14:55:04.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:04 smithi040 ceph-mon[39581]: 8.a scrub ok 2024-04-29T14:55:04.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:04 smithi040 ceph-mon[39581]: 8.6 scrub ok 2024-04-29T14:55:04.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:04 smithi121 ceph-mon[44533]: 8.a scrub starts 2024-04-29T14:55:04.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:04 smithi121 ceph-mon[44533]: 8.6 scrub starts 2024-04-29T14:55:04.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:04 smithi121 ceph-mon[44533]: 8.7 scrub ok 2024-04-29T14:55:04.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:04 smithi121 ceph-mon[44533]: 8.1 scrub starts 2024-04-29T14:55:04.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:04 smithi121 ceph-mon[44533]: 8.a scrub ok 2024-04-29T14:55:04.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:04 smithi121 ceph-mon[44533]: 8.6 scrub ok 2024-04-29T14:55:04.907 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:04.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:04.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:05.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:05.109 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:05.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:05.265 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:05.301 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:05.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:55:05.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:05 smithi194 ceph-mon[44089]: pgmap v2679: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 74/91098 objects degraded (0.081%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:55:05.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:05 smithi194 ceph-mon[44089]: 8.1 scrub ok 2024-04-29T14:55:05.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:05 smithi194 ceph-mon[44089]: 8.2 scrub starts 2024-04-29T14:55:05.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:05 smithi194 ceph-mon[44089]: 8.3 deep-scrub starts 2024-04-29T14:55:05.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:05 smithi194 ceph-mon[44089]: 8.2 scrub ok 2024-04-29T14:55:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:05 smithi040 ceph-mon[39581]: pgmap v2679: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 74/91098 objects degraded (0.081%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:55:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:05 smithi040 ceph-mon[39581]: 8.1 scrub ok 2024-04-29T14:55:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:05 smithi040 ceph-mon[39581]: 8.2 scrub starts 2024-04-29T14:55:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:05 smithi040 ceph-mon[39581]: 8.3 deep-scrub starts 2024-04-29T14:55:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:05 smithi040 ceph-mon[39581]: 8.2 scrub ok 2024-04-29T14:55:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:05 smithi121 ceph-mon[44533]: pgmap v2679: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.8 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 74/91098 objects degraded (0.081%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:55:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:05 smithi121 ceph-mon[44533]: 8.1 scrub ok 2024-04-29T14:55:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:05 smithi121 ceph-mon[44533]: 8.2 scrub starts 2024-04-29T14:55:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:05 smithi121 ceph-mon[44533]: 8.3 deep-scrub starts 2024-04-29T14:55:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:05 smithi121 ceph-mon[44533]: 8.2 scrub ok 2024-04-29T14:55:06.085 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:55:06.085 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:55:06.085 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:55:06.085 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:55:06.085 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:55:06.085 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:55:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:06.121 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:55:06.166 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:55:06.195 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:55:06.245 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:55:06.271 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:55:06.271 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:55:06.271 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:55:06.271 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:55:06.271 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:55:06.272 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:55:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3638667386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: 8.f scrub starts 2024-04-29T14:55:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3638667386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: 8.f scrub starts 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3638667386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: 8.f scrub starts 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:07.212 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:07.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:07.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:07.357 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:07 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:07 smithi194 ceph-mon[44089]: pgmap v2680: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 2.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 74/85082 objects degraded (0.087%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:55:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:07 smithi194 ceph-mon[44089]: 8.f scrub ok 2024-04-29T14:55:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:07 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:07 smithi040 ceph-mon[39581]: pgmap v2680: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 2.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 74/85082 objects degraded (0.087%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:55:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:07 smithi040 ceph-mon[39581]: 8.f scrub ok 2024-04-29T14:55:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:07 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:07 smithi121 ceph-mon[44533]: pgmap v2680: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 2.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s; 74/85082 objects degraded (0.087%); 4.8 MiB/s, 76 objects/s recovering 2024-04-29T14:55:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:07 smithi121 ceph-mon[44533]: 8.f scrub ok 2024-04-29T14:55:08.418 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:08.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:08.524 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:08.561 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:08 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 74/85082 objects degraded (0.087%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:55:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:08 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 74/85082 objects degraded (0.087%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:55:08.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:08 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 74/85082 objects degraded (0.087%), 1 pg degraded (PG_DEGRADED) 2024-04-29T14:55:09.316 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:09.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:09.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:09.458 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:09.514 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:09.517 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:09.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:09.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:09 smithi194 ceph-mon[44089]: pgmap v2681: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:55:09.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:09 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 74/85082 objects degraded (0.087%), 1 pg degraded) 2024-04-29T14:55:09.545 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:09 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:55:09.624 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:09.662 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:09.769 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:09.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:09.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:09.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:09 smithi040 ceph-mon[39581]: pgmap v2681: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:55:09.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:09 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 74/85082 objects degraded (0.087%), 1 pg degraded) 2024-04-29T14:55:09.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:09 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:55:09.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:09 smithi121 ceph-mon[44533]: pgmap v2681: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 2.5 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:55:09.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:09 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 74/85082 objects degraded (0.087%), 1 pg degraded) 2024-04-29T14:55:09.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:09 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:55:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:10.134 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:55:10.134 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:55:10.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:55:10.134 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:55:10.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173677 7.f 7.5de73dcf (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:49.669567+0000", 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 440.45648101799998, 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.88943558899999997, 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173677 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669567+0000", 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669567+0000", 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669569+0000", 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9259999999999999e-06 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669574+0000", 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.904e-06 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669585+0000", 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1327e-05 2024-04-29T14:55:10.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669592+0000", 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4799999999999998e-06 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669643+0000", 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1320999999999999e-05 2024-04-29T14:55:10.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.669653+0000", 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3570000000000004e-06 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.558989+0000", 2024-04-29T14:55:10.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.88933647800000004 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559003+0000", 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3796e-05 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173677 7.f 7.5de73dcf (undecoded) ondisk+retry+write+known_if_redirected e910)", 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:49.933187+0000", 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 440.19286112499998, 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62587116700000001, 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.142 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173677 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933187+0000", 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933187+0000", 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933189+0000", 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0889999999999998e-06 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933194+0000", 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4170000000000002e-06 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933202+0000", 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0250000000000002e-06 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933212+0000", 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0618000000000001e-05 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933265+0000", 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2488000000000003e-05 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933273+0000", 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5019999999999992e-06 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559030+0000", 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62575718300000005 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559058+0000", 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7844999999999999e-05 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173696 7.f 7.435a06af (undecoded) ondisk+retry+write+known_if_redirected e910)", 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:49.933229+0000", 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 440.19281922900001, 2024-04-29T14:55:10.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62588960400000004, 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173696 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933229+0000", 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933229+0000", 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933230+0000", 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4799999999999997e-07 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933233+0000", 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2449999999999998e-06 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933239+0000", 2024-04-29T14:55:10.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8080000000000001e-06 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933245+0000", 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0430000000000004e-06 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933296+0000", 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1267000000000002e-05 2024-04-29T14:55:10.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:49.933304+0000", 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.481e-06 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559107+0000", 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.62580390200000002 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559119+0000", 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.111e-05 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173702 7.f 7.a3ceca0f (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:55:10.154 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.068392+0000", 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 440.057655974, 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49078070699999998, 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173702 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.155 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068392+0000", 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068392+0000", 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068396+0000", 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.473e-06 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068408+0000", 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2313000000000001e-05 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068430+0000", 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1793999999999998e-05 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068441+0000", 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1674e-05 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068511+0000", 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9958999999999999e-05 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.068521+0000", 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5100000000000004e-06 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559161+0000", 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49064043299999999 2024-04-29T14:55:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559173+0000", 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1551e-05 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173717 7.f 7.d56abeef (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:55:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.379996+0000", 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 439.74605226400001, 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17923861099999999, 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173717 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.379996+0000", 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.379996+0000", 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.379999+0000", 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.875e-06 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380007+0000", 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1850000000000004e-06 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380023+0000", 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5628e-05 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380030+0000", 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8569999999999995e-06 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380098+0000", 2024-04-29T14:55:10.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7161000000000003e-05 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.380111+0000", 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3012e-05 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559225+0000", 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.179114256 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559235+0000", 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6369999999999994e-06 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173718 7.f 7.3bfc8dbf (undecoded) ondisk+write+known_if_redirected e910)", 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:50.383765+0000", 2024-04-29T14:55:10.166 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 439.74228299599997, 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17552379800000001, 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173718 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383765+0000", 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383765+0000", 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383767+0000", 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3489999999999999e-06 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383771+0000", 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9289999999999997e-06 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383782+0000", 2024-04-29T14:55:10.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0034000000000001e-05 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383787+0000", 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8139999999999997e-06 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383833+0000", 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5291999999999999e-05 2024-04-29T14:55:10.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.383840+0000", 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4970000000000004e-06 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559278+0000", 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17543802999999999 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:50.559289+0000", 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0852999999999999e-05 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173677 7.f 7:f3bce7ba:::benchmark_data_smithi194_147200_object58765:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e911)", 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.578733+0000", 2024-04-29T14:55:10.172 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 438.54731564299999, 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51027307300000002, 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173677 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578733+0000", 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578733+0000", 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578735+0000", 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3199999999999998e-06 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578742+0000", 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6610000000000003e-06 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578754+0000", 2024-04-29T14:55:10.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2435e-05 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578765+0000", 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1103e-05 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578848+0000", 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2929000000000005e-05 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578871+0000", 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3156e-05 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.076660+0000", 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.497788964 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.076825+0000", 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016460699999999999 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.076942+0000", 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011712799999999999 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.088840+0000", 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011898021 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.088879+0000", 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.888e-05 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.088892+0000", 2024-04-29T14:55:10.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2917e-05 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.088936+0000", 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4230999999999997e-05 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089006+0000", 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9721000000000006e-05 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.180 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173696 7.f 7:f5605ac2:::benchmark_data_smithi194_147200_object58784:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e911)", 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.578785+0000", 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 438.54726284899999, 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51646160900000004, 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.181 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173696 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578785+0000", 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578785+0000", 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578786+0000", 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000005e-07 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578790+0000", 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8279999999999996e-06 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578796+0000", 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8130000000000001e-06 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578801+0000", 2024-04-29T14:55:10.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1699999999999996e-06 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578913+0000", 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112695 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578922+0000", 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1389999999999995e-06 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.079998+0000", 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50107606000000005 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.080122+0000", 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000123948 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.080270+0000", 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000148166 2024-04-29T14:55:10.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089015+0000", 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0087448930000000001 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089154+0000", 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013937699999999999 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.095122+0000", 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005968307 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.095162+0000", 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9858999999999999e-05 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.095247+0000", 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4728999999999995e-05 2024-04-29T14:55:10.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173702 7.f 7:f05373c5:::benchmark_data_smithi194_147200_object58790:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e911)", 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.578821+0000", 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 438.54722699299998, 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51996605799999995, 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173702 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578821+0000", 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578821+0000", 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578822+0000", 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.68e-07 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578824+0000", 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6189999999999998e-06 2024-04-29T14:55:10.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578830+0000", 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7930000000000003e-06 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578836+0000", 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4389999999999999e-06 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578959+0000", 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012356800000000001 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578966+0000", 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2860000000000002e-06 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.082945+0000", 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50397858200000001 2024-04-29T14:55:10.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.083066+0000", 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120899 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.083227+0000", 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000160939 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089022+0000", 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057946350000000002 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.095313+0000", 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062915769999999996 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.098606+0000", 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003293364 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.098671+0000", 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4747999999999997e-05 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.098787+0000", 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116041 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173717 7.f 7:f77d56ab:::benchmark_data_smithi194_147200_object58805:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e911)", 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.578842+0000", 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 438.54720649699999, 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52547145299999998, 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173717 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.197 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578842+0000", 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578842+0000", 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578842+0000", 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.51e-07 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578844+0000", 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.108e-06 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578850+0000", 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7810000000000002e-06 2024-04-29T14:55:10.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578856+0000", 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8939999999999998e-06 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.579009+0000", 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015299000000000001 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.579018+0000", 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9339999999999995e-06 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.085878+0000", 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50685994400000001 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.085980+0000", 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000102295 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.086120+0000", 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014019999999999999 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089164+0000", 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003044073 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.098559+0000", 2024-04-29T14:55:10.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093943550000000001 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.104144+0000", 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005585423 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.104206+0000", 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1551999999999996e-05 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.104313+0000", 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107453 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173718 7.f 7:fdb13fdc:::benchmark_data_smithi194_147200_object58806:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e911)", 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.578861+0000", 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 438.54718712300001, 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53251540799999997, 2024-04-29T14:55:10.204 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173718 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578861+0000", 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578861+0000", 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578861+0000", 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.03e-07 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578864+0000", 2024-04-29T14:55:10.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3740000000000001e-06 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578869+0000", 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6160000000000001e-06 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578875+0000", 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0849999999999996e-06 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.579070+0000", 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000195317 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.579077+0000", 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3239999999999997e-06 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089185+0000", 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51010744299999999 2024-04-29T14:55:10.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089270+0000", 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5408000000000006e-05 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.089350+0000", 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0154999999999994e-05 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.095332+0000", 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059816340000000004 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.103365+0000", 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0080327899999999997 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.111266+0000", 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079009349999999996 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.111305+0000", 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9060000000000002e-05 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.111376+0000", 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1864000000000002e-05 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.211 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:173727 7.f 7:f24b0e3f:::benchmark_data_smithi194_147200_object58815:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e911)", 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:51.578905+0000", 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 438.54714311800001, 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53267840300000002, 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173727 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578905+0000", 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578905+0000", 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578906+0000", 2024-04-29T14:55:10.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.540000000000001e-07 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578910+0000", 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0459999999999999e-06 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578919+0000", 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3989999999999993e-06 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.578929+0000", 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8940000000000005e-06 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.579106+0000", 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017704 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for peered", 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:51.579111+0000", 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7299999999999996e-06 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.092192+0000", 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51308076499999999 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.092333+0000", 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014099499999999999 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.092419+0000", 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6379999999999996e-05 2024-04-29T14:55:10.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.101686+0000", 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0092667050000000001 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.103417+0000", 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017308099999999999 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.111423+0000", 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0080064869999999996 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.111478+0000", 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4289000000000001e-05 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:52.111583+0000", 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010590900000000001 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.218 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:174639 7.f 7:fb335220:::benchmark_data_smithi194_147200_object59727:head [delete] snapc 0=[] ondisk+write+known_if_redirected e917)", 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.702600+0000", 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 432.42344792300003, 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232552334, 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.219 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174639 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.702600+0000", 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.702600+0000", 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.702603+0000", 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.914e-06 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.702611+0000", 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2930000000000001e-06 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.702624+0000", 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2163e-05 2024-04-29T14:55:10.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.702631+0000", 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5140000000000004e-06 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.781370+0000", 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.078738683000000004 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.781477+0000", 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106772 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.781596+0000", 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000119726 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.934895+0000", 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15329849700000001 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.934922+0000", 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6857999999999999e-05 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.934974+0000", 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2710000000000002e-05 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.935039+0000", 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4925000000000006e-05 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.935153+0000", 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000113279 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:174656 7.7 7:ebc52bdd:::benchmark_data_smithi194_147200_object59744:head [delete] snapc 0=[] ondisk+write+known_if_redirected e917)", 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.764653+0000", 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 432.36139507600001, 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.170370933, 2024-04-29T14:55:10.225 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174656 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.764653+0000", 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.764653+0000", 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.764656+0000", 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.078e-06 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.764670+0000", 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3920000000000001e-05 2024-04-29T14:55:10.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.764682+0000", 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1747e-05 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.764690+0000", 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1450000000000008e-06 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.784212+0000", 2024-04-29T14:55:10.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019521941000000001 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.784300+0000", 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669000000000004e-05 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.784373+0000", 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3553000000000005e-05 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.934905+0000", 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15053140400000001 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.934952+0000", 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7104000000000002e-05 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.934982+0000", 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0621999999999999e-05 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.935024+0000", 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1749999999999998e-05 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.26998.0:174664 7.f 7:f81cda00:::benchmark_data_smithi194_147200_object59752:head [delete] snapc 0=[] ondisk+write+known_if_redirected e917)", 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:47:57.810105+0000", 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 432.31594344299998, 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20538099000000001, 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.231 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.26998", 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 174664 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.810105+0000", 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.810105+0000", 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.810108+0000", 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8140000000000001e-06 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.810115+0000", 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.810128+0000", 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2604e-05 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.810139+0000", 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0715e-05 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.935456+0000", 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12531725599999999 2024-04-29T14:55:10.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.935578+0000", 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012214699999999999 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 0,9", 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:57.935714+0000", 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013584099999999999 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:58.015174+0000", 2024-04-29T14:55:10.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.079459806999999993 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:58.015231+0000", 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7462e-05 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:58.015285+0000", 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3396999999999997e-05 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:58.015353+0000", 2024-04-29T14:55:10.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8584999999999995e-05 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:47:58.015486+0000", 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013248099999999999 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162440 8.a 8:559374cd:::benchmark_data_smithi194_186548_object33882:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.017140+0000", 2024-04-29T14:55:10.237 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 53.108908659999997, 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21948063300000001, 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162440 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017140+0000", 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017140+0000", 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017142+0000", 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.08e-06 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017146+0000", 2024-04-29T14:55:10.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5859999999999998e-06 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017157+0000", 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0755e-05 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017165+0000", 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9310000000000008e-06 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.022286+0000", 2024-04-29T14:55:10.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051215369999999998 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.022399+0000", 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011301799999999999 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.022498+0000", 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8815999999999997e-05 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.026739+0000", 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042403019999999996 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235303+0000", 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.208564257 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235360+0000", 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7552000000000001e-05 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236620+0000", 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001259799 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162470 8.c 8:3138e6fd:::benchmark_data_smithi194_186548_object33912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.039232+0000", 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 53.086816321000001, 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.197422704, 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.243 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162470 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039232+0000", 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039232+0000", 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039234+0000", 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5150000000000001e-06 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039241+0000", 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8410000000000002e-06 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039255+0000", 2024-04-29T14:55:10.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.375e-05 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039264+0000", 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2410000000000001e-06 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039324+0000", 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9772999999999998e-05 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039465+0000", 2024-04-29T14:55:10.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014109300000000001 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039594+0000", 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012870500000000001 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.044220+0000", 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046258159999999996 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235552+0000", 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.191332421 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235617+0000", 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4801999999999994e-05 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236655+0000", 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010377470000000001 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162482 8.a 8:54bedb5a:::benchmark_data_smithi194_186548_object33924:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.050747+0000", 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 53.075301668999998, 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.186855457, 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162482 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.249 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050747+0000", 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050747+0000", 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050749+0000", 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9390000000000002e-06 2024-04-29T14:55:10.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050758+0000", 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5900000000000008e-06 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050774+0000", 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6143000000000001e-05 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050782+0000", 2024-04-29T14:55:10.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0090000000000007e-06 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050833+0000", 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0865000000000002e-05 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050952+0000", 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118418 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.051037+0000", 2024-04-29T14:55:10.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5702e-05 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.056976+0000", 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059383190000000001 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236650+0000", 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17967455800000001 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236691+0000", 2024-04-29T14:55:10.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0961000000000003e-05 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.237602+0000", 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00091095300000000004 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162492 8.a 8:5a0e10d2:::benchmark_data_smithi194_186548_object33934:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.059901+0000", 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 53.066147031, 2024-04-29T14:55:10.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18326977699999999, 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162492 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059901+0000", 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059901+0000", 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059904+0000", 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3580000000000001e-06 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059910+0000", 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5860000000000001e-06 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059925+0000", 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4623999999999999e-05 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059934+0000", 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9970000000000004e-06 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059996+0000", 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2577999999999996e-05 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.060132+0000", 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000135563 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.060268+0000", 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013622299999999999 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.065270+0000", 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050021170000000004 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.242031+0000", 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17676051300000001 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.242066+0000", 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4816e-05 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.243171+0000", 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011054019999999999 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162518 8.c 8:304526c8:::benchmark_data_smithi194_186548_object33960:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.080553+0000", 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 53.045495631999998, 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16836797100000001, 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162518 2024-04-29T14:55:10.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080553+0000", 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080553+0000", 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080554+0000", 2024-04-29T14:55:10.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.517e-06 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080559+0000", 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8239999999999999e-06 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080569+0000", 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0045e-05 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080576+0000", 2024-04-29T14:55:10.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1300000000000003e-06 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.084319+0000", 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003743269 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.084461+0000", 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014201799999999999 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.084597+0000", 2024-04-29T14:55:10.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013583900000000001 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.088351+0000", 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037542169999999998 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.248256+0000", 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.159904088 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.248293+0000", 2024-04-29T14:55:10.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7840999999999999e-05 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.248921+0000", 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062718300000000002 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:55:10.265 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:55:10.549 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:10 smithi194 ceph-mon[44089]: 8.d scrub starts 2024-04-29T14:55:10.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:10 smithi040 ceph-mon[39581]: 8.d scrub starts 2024-04-29T14:55:10.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:10 smithi121 ceph-mon[44533]: 8.d scrub starts 2024-04-29T14:55:11.010 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:11.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:55:11.058 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:55:11.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:55:11.118 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:11.150 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:11.172 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:11.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:11.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:11.318 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:11.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:11.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:11.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:11.489 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:11.521 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:55:11.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:11 smithi121 ceph-mon[44533]: pgmap v2682: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:55:11.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:11 smithi121 ceph-mon[44533]: 8.5 scrub starts 2024-04-29T14:55:11.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:11 smithi121 ceph-mon[44533]: 8.d scrub ok 2024-04-29T14:55:11.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:11 smithi121 ceph-mon[44533]: 8.5 scrub ok 2024-04-29T14:55:11.667 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:11.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:11 smithi040 ceph-mon[39581]: pgmap v2682: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:55:11.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:11 smithi040 ceph-mon[39581]: 8.5 scrub starts 2024-04-29T14:55:11.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:11 smithi040 ceph-mon[39581]: 8.d scrub ok 2024-04-29T14:55:11.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:11 smithi040 ceph-mon[39581]: 8.5 scrub ok 2024-04-29T14:55:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:11.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:11.779 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:11.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:11 smithi194 ceph-mon[44089]: pgmap v2682: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.3 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.15k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-04-29T14:55:11.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:11 smithi194 ceph-mon[44089]: 8.5 scrub starts 2024-04-29T14:55:11.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:11 smithi194 ceph-mon[44089]: 8.d scrub ok 2024-04-29T14:55:11.807 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:11 smithi194 ceph-mon[44089]: 8.5 scrub ok 2024-04-29T14:55:11.810 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:11.830 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:11.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:11.934 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:11.963 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:11.997 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:12.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:12.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:12.135 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:13.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:55:13.061 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:55:13.065 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:55:13.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:13.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:13.203 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:13.234 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:13.271 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:13.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:13.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:13.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:13 smithi194 ceph-mon[44089]: pgmap v2683: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 474 KiB/s, 7 objects/s recovering 2024-04-29T14:55:13.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:13 smithi194 ceph-mon[44089]: 8.9 scrub starts 2024-04-29T14:55:13.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:13 smithi194 ceph-mon[44089]: 8.9 scrub ok 2024-04-29T14:55:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:13 smithi040 ceph-mon[39581]: pgmap v2683: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 474 KiB/s, 7 objects/s recovering 2024-04-29T14:55:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:13 smithi040 ceph-mon[39581]: 8.9 scrub starts 2024-04-29T14:55:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:13 smithi040 ceph-mon[39581]: 8.9 scrub ok 2024-04-29T14:55:13.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:13 smithi121 ceph-mon[44533]: pgmap v2683: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 474 KiB/s, 7 objects/s recovering 2024-04-29T14:55:13.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:13 smithi121 ceph-mon[44533]: 8.9 scrub starts 2024-04-29T14:55:13.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:13 smithi121 ceph-mon[44533]: 8.9 scrub ok 2024-04-29T14:55:14.028 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28144.0:225459 8.3 e1051/1011 8:cff20144:::benchmark_data_smithi194_186548_object96901:head v 1051'15327, mlcod=1051'15317)", 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:55:14.008369+0000", 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.015974499, 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015995669000000001, 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:55:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.008369+0000", 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.008369+0000", 2024-04-29T14:55:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.008370+0000", 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2139999999999999e-06 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.008377+0000", 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9750000000000001e-06 2024-04-29T14:55:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.008387+0000", 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0285000000000001e-05 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.008393+0000", 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3899999999999998e-06 2024-04-29T14:55:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015715+0000", 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073212640000000001 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015733+0000", 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8615999999999999e-05 2024-04-29T14:55:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28144.0:225462 8.3 e1051/1011 8:c97d68db:::benchmark_data_smithi194_186548_object96904:head v 1051'15328, mlcod=1051'15319)", 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:55:14.012609+0000", 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.01173402, 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011796516999999999, 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:55:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.012609+0000", 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:55:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.012609+0000", 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.012611+0000", 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0219999999999999e-06 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.012622+0000", 2024-04-29T14:55:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0689e-05 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.012630+0000", 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4530000000000002e-06 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.012641+0000", 2024-04-29T14:55:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0655999999999999e-05 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019743+0000", 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071017110000000001 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019760+0000", 2024-04-29T14:55:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7676000000000002e-05 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28144.0:225470 8.3 e1051/1011 8:c4bcdb75:::benchmark_data_smithi194_186548_object96912:head v 1051'15329, mlcod=1051'15319)", 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:55:14.015935+0000", 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0084083639999999998, 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085075789999999995, 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:55:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015935+0000", 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015935+0000", 2024-04-29T14:55:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015936+0000", 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5600000000000001e-06 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015944+0000", 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5220000000000001e-06 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015955+0000", 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1358e-05 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.015962+0000", 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9979999999999997e-06 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023595+0000", 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076330809999999999 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023611+0000", 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6082e-05 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28144.0:225488 8.3 e1051/1011)", 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:55:14.019586+0000", 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0047565910000000001, 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048879930000000002, 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:55:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019586+0000", 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019586+0000", 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019588+0000", 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.192e-06 2024-04-29T14:55:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019596+0000", 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6629999999999999e-06 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019605+0000", 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6610000000000006e-06 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.019611+0000", 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8810000000000001e-06 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28144.0:225497 8.3 e1051/1011)", 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:55:14.023692+0000", 2024-04-29T14:55:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00065133300000000003, 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080974599999999997, 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023692+0000", 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023692+0000", 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023693+0000", 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.178e-06 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023699+0000", 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9089999999999997e-06 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023710+0000", 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1187e-05 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:55:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:55:14.023716+0000", 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3770000000000001e-06 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 5 2024-04-29T14:55:14.051 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:55:14.381 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:14.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:14.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:14.523 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:14.652 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:14 smithi040 ceph-mon[39581]: from='client.38069 v1:172.21.15.40:0/3529113850' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:55:14.652 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:14.652 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:14.652 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:14 smithi040 ceph-mon[39581]: osdmap e1051: 12 total, 11 up, 9 in 2024-04-29T14:55:14.757 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:14 smithi121 ceph-mon[44533]: from='client.38069 v1:172.21.15.40:0/3529113850' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:55:14.757 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:14.757 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:14.757 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:14 smithi121 ceph-mon[44533]: osdmap e1051: 12 total, 11 up, 9 in 2024-04-29T14:55:14.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:14.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:55:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:14.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:14.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:14 smithi194 ceph-mon[44089]: from='client.38069 v1:172.21.15.40:0/3529113850' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:55:14.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:14.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:14.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:14 smithi194 ceph-mon[44089]: osdmap e1051: 12 total, 11 up, 9 in 2024-04-29T14:55:14.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:14.952 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:14.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:15.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:15.131 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:15.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:15.248 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:55:15.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:15.296 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:15.317 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:15.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:15.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:15.456 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:15.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:15.562 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:15.563 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2683,"stamp":"2024-04-29T14:55:12.371606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2325479992,"num_objects":35478,"num_object_clones":0,"num_object_copies":70956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35478,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":221817,"num_write_kb":8230459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21536,"num_bytes_recovered":1414168288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161011,"ondisk_log_size":161011,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":11590184,"kb_used_data":4678876,"kb_used_omap":666,"kb_used_meta":6910501,"kb_avail":832116184,"statfs":{"total":863955320832,"available":852086972416,"internally_reserved":0,"allocated":4791169024,"data_stored":4787557239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":682449,"internal_metadata":7076353583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,5,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-733151232,"num_objects":-11187,"num_object_clones":0,"num_object_copies":-22374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":4849664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002092"},"pg_stats":[{"pgid":"8.f","version":"1050'14018","reported_seq":14139,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:11.894714+0000","last_change":"2024-04-29T14:55:06.485189+0000","last_active":"2024-04-29T14:55:11.894714+0000","last_peered":"2024-04-29T14:55:11.894714+0000","last_clean":"2024-04-29T14:55:11.894714+0000","last_became_active":"2024-04-29T14:51:58.471518+0000","last_became_peered":"2024-04-29T14:51:58.471518+0000","last_unstale":"2024-04-29T14:55:11.894714+0000","last_undegraded":"2024-04-29T14:55:11.894714+0000","last_fullsized":"2024-04-29T14:55:11.894714+0000","mapping_epoch":1011,"log_start":"1018'4000","ondisk_log_start":"1018'4000","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'13688","last_scrub_stamp":"2024-04-29T14:55:06.485008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:06.485008+0000","objects_scrubbed":2495,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:26.095563+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14018,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"1050'13927","reported_seq":14142,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:09.355871+0000","last_change":"2024-04-29T14:54:53.972810+0000","last_active":"2024-04-29T14:55:09.355871+0000","last_peered":"2024-04-29T14:55:09.355871+0000","last_clean":"2024-04-29T14:55:09.355871+0000","last_became_active":"2024-04-29T14:51:58.468700+0000","last_became_peered":"2024-04-29T14:51:58.468700+0000","last_unstale":"2024-04-29T14:55:09.355871+0000","last_undegraded":"2024-04-29T14:55:09.355871+0000","last_fullsized":"2024-04-29T14:55:09.355871+0000","mapping_epoch":1011,"log_start":"1018'3900","ondisk_log_start":"1018'3900","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12818","last_scrub_stamp":"2024-04-29T14:54:53.972600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:53.972600+0000","objects_scrubbed":3458,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:03.208154+0000","scrub_duration":713,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152240128,"num_objects":2323,"num_object_clones":0,"num_object_copies":4646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13927,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"1050'14060","reported_seq":14266,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:11.454407+0000","last_change":"2024-04-29T14:55:10.874877+0000","last_active":"2024-04-29T14:55:11.454407+0000","last_peered":"2024-04-29T14:55:11.454407+0000","last_clean":"2024-04-29T14:55:11.454407+0000","last_became_active":"2024-04-29T14:51:58.469439+0000","last_became_peered":"2024-04-29T14:51:58.469439+0000","last_unstale":"2024-04-29T14:55:11.454407+0000","last_undegraded":"2024-04-29T14:55:11.454407+0000","last_fullsized":"2024-04-29T14:55:11.454407+0000","mapping_epoch":1011,"log_start":"1018'4000","ondisk_log_start":"1018'4000","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'14021","last_scrub_stamp":"2024-04-29T14:55:10.874616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:10.874616+0000","objects_scrubbed":2225,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:17.881992+0000","scrub_duration":464,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142016536,"num_objects":2168,"num_object_clones":0,"num_object_copies":4336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14060,"num_write_kb":519233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"1050'13823","reported_seq":24627,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:10.815557+0000","last_change":"2024-04-29T14:54:55.056447+0000","last_active":"2024-04-29T14:55:10.815557+0000","last_peered":"2024-04-29T14:55:10.815557+0000","last_clean":"2024-04-29T14:55:10.815557+0000","last_became_active":"2024-04-29T14:52:59.820433+0000","last_became_peered":"2024-04-29T14:52:59.820433+0000","last_unstale":"2024-04-29T14:55:10.815557+0000","last_undegraded":"2024-04-29T14:55:10.815557+0000","last_fullsized":"2024-04-29T14:55:10.815557+0000","mapping_epoch":1026,"log_start":"1018'3800","ondisk_log_start":"1018'3800","created":1011,"last_epoch_clean":1048,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12753","last_scrub_stamp":"2024-04-29T14:54:55.056281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:55.056281+0000","objects_scrubbed":3247,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:23.980689+0000","scrub_duration":758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140443648,"num_objects":2143,"num_object_clones":0,"num_object_copies":4286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13823,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5334,"num_bytes_recovered":349569024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"1050'13721","reported_seq":13851,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:10.769198+0000","last_change":"2024-04-29T14:54:56.941725+0000","last_active":"2024-04-29T14:55:10.769198+0000","last_peered":"2024-04-29T14:55:10.769198+0000","last_clean":"2024-04-29T14:55:10.769198+0000","last_became_active":"2024-04-29T14:51:58.469253+0000","last_became_peered":"2024-04-29T14:51:58.469253+0000","last_unstale":"2024-04-29T14:55:10.769198+0000","last_undegraded":"2024-04-29T14:55:10.769198+0000","last_fullsized":"2024-04-29T14:55:10.769198+0000","mapping_epoch":1011,"log_start":"1018'3700","ondisk_log_start":"1018'3700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12718","last_scrub_stamp":"2024-04-29T14:54:56.941518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:56.941518+0000","objects_scrubbed":3270,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:08.178484+0000","scrub_duration":572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148045824,"num_objects":2259,"num_object_clones":0,"num_object_copies":4518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13721,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"1050'13937","reported_seq":24770,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:10.791543+0000","last_change":"2024-04-29T14:55:03.993939+0000","last_active":"2024-04-29T14:55:10.791543+0000","last_peered":"2024-04-29T14:55:10.791543+0000","last_clean":"2024-04-29T14:55:10.791543+0000","last_became_active":"2024-04-29T14:52:59.816964+0000","last_became_peered":"2024-04-29T14:52:59.816964+0000","last_unstale":"2024-04-29T14:55:10.791543+0000","last_undegraded":"2024-04-29T14:55:10.791543+0000","last_fullsized":"2024-04-29T14:55:10.791543+0000","mapping_epoch":1026,"log_start":"1018'3900","ondisk_log_start":"1018'3900","created":1011,"last_epoch_clean":1048,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'13467","last_scrub_stamp":"2024-04-29T14:55:03.993782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:03.993782+0000","objects_scrubbed":2646,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:08.028870+0000","scrub_duration":610,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141623296,"num_objects":2161,"num_object_clones":0,"num_object_copies":4322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13937,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5351,"num_bytes_recovered":350683136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"1050'13898","reported_seq":14030,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:08.736554+0000","last_change":"2024-04-29T14:54:51.498258+0000","last_active":"2024-04-29T14:55:08.736554+0000","last_peered":"2024-04-29T14:55:08.736554+0000","last_clean":"2024-04-29T14:55:08.736554+0000","last_became_active":"2024-04-29T14:51:58.469814+0000","last_became_peered":"2024-04-29T14:51:58.469814+0000","last_unstale":"2024-04-29T14:55:08.736554+0000","last_undegraded":"2024-04-29T14:55:08.736554+0000","last_fullsized":"2024-04-29T14:55:08.736554+0000","mapping_epoch":1011,"log_start":"1018'3800","ondisk_log_start":"1018'3800","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12636","last_scrub_stamp":"2024-04-29T14:54:51.498114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:51.498114+0000","objects_scrubbed":3603,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:57.798038+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152305664,"num_objects":2324,"num_object_clones":0,"num_object_copies":4648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13898,"num_write_kb":519104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"1050'13990","reported_seq":14115,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:11.900051+0000","last_change":"2024-04-29T14:55:03.624900+0000","last_active":"2024-04-29T14:55:11.900051+0000","last_peered":"2024-04-29T14:55:11.900051+0000","last_clean":"2024-04-29T14:55:11.900051+0000","last_became_active":"2024-04-29T14:51:58.473066+0000","last_became_peered":"2024-04-29T14:51:58.473066+0000","last_unstale":"2024-04-29T14:55:11.900051+0000","last_undegraded":"2024-04-29T14:55:11.900051+0000","last_fullsized":"2024-04-29T14:55:11.900051+0000","mapping_epoch":1011,"log_start":"1018'3900","ondisk_log_start":"1018'3900","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'13459","last_scrub_stamp":"2024-04-29T14:55:03.624762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:03.624762+0000","objects_scrubbed":2676,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:04.374288+0000","scrub_duration":701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139198464,"num_objects":2124,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13990,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"1050'13742","reported_seq":13864,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:09.634961+0000","last_change":"2024-04-29T14:55:04.099157+0000","last_active":"2024-04-29T14:55:09.634961+0000","last_peered":"2024-04-29T14:55:09.634961+0000","last_clean":"2024-04-29T14:55:09.634961+0000","last_became_active":"2024-04-29T14:51:58.472520+0000","last_became_peered":"2024-04-29T14:51:58.472520+0000","last_unstale":"2024-04-29T14:55:09.634961+0000","last_undegraded":"2024-04-29T14:55:09.634961+0000","last_fullsized":"2024-04-29T14:55:09.634961+0000","mapping_epoch":1011,"log_start":"1018'3700","ondisk_log_start":"1018'3700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'13371","last_scrub_stamp":"2024-04-29T14:55:04.098898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:04.098898+0000","objects_scrubbed":2617,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:26.016202+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146276352,"num_objects":2232,"num_object_clones":0,"num_object_copies":4464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13742,"num_write_kb":511168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"1050'13693","reported_seq":13781,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:09.642529+0000","last_change":"2024-04-29T14:51:58.473051+0000","last_active":"2024-04-29T14:55:09.642529+0000","last_peered":"2024-04-29T14:55:09.642529+0000","last_clean":"2024-04-29T14:55:09.642529+0000","last_became_active":"2024-04-29T14:51:58.472757+0000","last_became_peered":"2024-04-29T14:51:58.472757+0000","last_unstale":"2024-04-29T14:55:09.642529+0000","last_undegraded":"2024-04-29T14:55:09.642529+0000","last_fullsized":"2024-04-29T14:55:09.642529+0000","mapping_epoch":1011,"log_start":"1018'3600","ondisk_log_start":"1018'3600","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148570112,"num_objects":2267,"num_object_clones":0,"num_object_copies":4534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13693,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"1050'13952","reported_seq":24995,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:08.748169+0000","last_change":"2024-04-29T14:54:52.411446+0000","last_active":"2024-04-29T14:55:08.748169+0000","last_peered":"2024-04-29T14:55:08.748169+0000","last_clean":"2024-04-29T14:55:08.748169+0000","last_became_active":"2024-04-29T14:52:59.825361+0000","last_became_peered":"2024-04-29T14:52:59.825361+0000","last_unstale":"2024-04-29T14:55:08.748169+0000","last_undegraded":"2024-04-29T14:55:08.748169+0000","last_fullsized":"2024-04-29T14:55:08.748169+0000","mapping_epoch":1026,"log_start":"1018'3900","ondisk_log_start":"1018'3900","created":1011,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'12798","last_scrub_stamp":"2024-04-29T14:54:52.411348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:54:52.411348+0000","objects_scrubbed":3449,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:19.646039+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148373504,"num_objects":2264,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13952,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5452,"num_bytes_recovered":357302272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"1050'14877","reported_seq":15009,"reported_epoch":1050,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T14:55:11.898407+0000","last_change":"2024-04-29T14:55:04.915855+0000","last_active":"2024-04-29T14:55:11.898407+0000","last_peered":"2024-04-29T14:55:11.898407+0000","last_clean":"2024-04-29T14:55:11.898407+0000","last_became_active":"2024-04-29T14:51:58.468800+0000","last_became_peered":"2024-04-29T14:51:58.468800+0000","last_unstale":"2024-04-29T14:55:11.898407+0000","last_undegraded":"2024-04-29T14:55:11.898407+0000","last_fullsized":"2024-04-29T14:55:11.898407+0000","mapping_epoch":1011,"log_start":"1019'4800","ondisk_log_start":"1019'4800","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":1036,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":4190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13841,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"1050'13720","reported_seq":13843,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:09.657605+0000","last_change":"2024-04-29T14:55:05.036341+0000","last_active":"2024-04-29T14:55:09.657605+0000","last_peered":"2024-04-29T14:55:09.657605+0000","last_clean":"2024-04-29T14:55:09.657605+0000","last_became_active":"2024-04-29T14:51:58.473704+0000","last_became_peered":"2024-04-29T14:51:58.473704+0000","last_unstale":"2024-04-29T14:55:09.657605+0000","last_undegraded":"2024-04-29T14:55:09.657605+0000","last_fullsized":"2024-04-29T14:55:09.657605+0000","mapping_epoch":1011,"log_start":"1018'3700","ondisk_log_start":"1018'3700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'13402","last_scrub_stamp":"2024-04-29T14:55:05.036160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:05.036160+0000","objects_scrubbed":2563,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:20.467169+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146538496,"num_objects":2236,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13720,"num_write_kb":510592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"1050'14073","reported_seq":24974,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:11.789309+0000","last_change":"2024-04-29T14:55:04.391793+0000","last_active":"2024-04-29T14:55:11.789309+0000","last_peered":"2024-04-29T14:55:11.789309+0000","last_clean":"2024-04-29T14:55:11.789309+0000","last_became_active":"2024-04-29T14:52:59.811137+0000","last_became_peered":"2024-04-29T14:52:59.811137+0000","last_unstale":"2024-04-29T14:55:11.789309+0000","last_undegraded":"2024-04-29T14:55:11.789309+0000","last_fullsized":"2024-04-29T14:55:11.789309+0000","mapping_epoch":1026,"log_start":"1018'4000","ondisk_log_start":"1018'4000","created":1011,"last_epoch_clean":1050,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'13595","last_scrub_stamp":"2024-04-29T14:55:04.391585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:04.391585+0000","objects_scrubbed":2678,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:21.869303+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143327232,"num_objects":2187,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14073,"num_write_kb":520320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5385,"num_bytes_recovered":352911360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"1050'13772","reported_seq":13895,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:08.422217+0000","last_change":"2024-04-29T14:55:02.763510+0000","last_active":"2024-04-29T14:55:08.422217+0000","last_peered":"2024-04-29T14:55:08.422217+0000","last_clean":"2024-04-29T14:55:08.422217+0000","last_became_active":"2024-04-29T14:51:58.471940+0000","last_became_peered":"2024-04-29T14:51:58.471940+0000","last_unstale":"2024-04-29T14:55:08.422217+0000","last_undegraded":"2024-04-29T14:55:08.422217+0000","last_fullsized":"2024-04-29T14:55:08.422217+0000","mapping_epoch":1011,"log_start":"1018'3700","ondisk_log_start":"1018'3700","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"1050'13393","last_scrub_stamp":"2024-04-29T14:55:02.763158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:55:02.763158+0000","objects_scrubbed":2745,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:56:06.732386+0000","scrub_duration":689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153485312,"num_objects":2342,"num_object_clones":0,"num_object_copies":4684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13772,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"1050'13469","reported_seq":13557,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:09.649314+0000","last_change":"2024-04-29T14:51:58.472764+0000","last_active":"2024-04-29T14:55:09.649314+0000","last_peered":"2024-04-29T14:55:09.649314+0000","last_clean":"2024-04-29T14:55:09.649314+0000","last_became_active":"2024-04-29T14:51:58.472489+0000","last_became_peered":"2024-04-29T14:51:58.472489+0000","last_unstale":"2024-04-29T14:55:09.649314+0000","last_undegraded":"2024-04-29T14:55:09.649314+0000","last_fullsized":"2024-04-29T14:55:09.649314+0000","mapping_epoch":1011,"log_start":"1017'3400","ondisk_log_start":"1017'3400","created":1011,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:51:57.451341+0000","last_clean_scrub_stamp":"2024-04-29T14:51:57.451341+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144244736,"num_objects":2201,"num_object_clones":0,"num_object_copies":4402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13469,"num_write_kb":501440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2115,"reported_epoch":1050,"state":"active+clean","last_fresh":"2024-04-29T14:55:02.511313+0000","last_change":"2024-04-29T14:54:06.451143+0000","last_active":"2024-04-29T14:55:02.511313+0000","last_peered":"2024-04-29T14:55:02.511313+0000","last_clean":"2024-04-29T14:55:02.511313+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:55:02.511313+0000","last_undegraded":"2024-04-29T14:55:02.511313+0000","last_fullsized":"2024-04-29T14:55:02.511313+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:54:06.451003+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:54:06.451003+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:55:09.012234+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2324889624,"num_objects":35476,"num_object_clones":0,"num_object_copies":70952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":221636,"num_write_kb":8227521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21522,"num_bytes_recovered":1410465792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6050684928,"data_stored":6050676784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122628,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2162924,"kb_used_data":1058596,"kb_used_omap":1,"kb_used_meta":1104318,"kb_avail":91582228,"statfs":{"total":95995035648,"available":93780201472,"internally_reserved":0,"allocated":1084002304,"data_stored":1083602671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1130822080},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423207,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638188,"kb_used_data":425188,"kb_used_omap":4,"kb_used_meta":212987,"kb_avail":93106964,"statfs":{"total":95995035648,"available":95341531136,"internally_reserved":0,"allocated":435392512,"data_stored":434992899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":218099392},"hb_peers":[0,1,2,3,4,5,6,7,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234583,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200376,"kb_used_data":276464,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92544776,"statfs":{"total":95995035648,"available":94765850624,"internally_reserved":0,"allocated":283099136,"data_stored":282691899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":946010570},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988984,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92756168,"statfs":{"total":95995035648,"available":94982316032,"internally_reserved":0,"allocated":1290240,"data_stored":887075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011415494},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575464,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139820,"kb_used_data":563044,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92605332,"statfs":{"total":95995035648,"available":94827859968,"internally_reserved":0,"allocated":576557056,"data_stored":576157443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739003,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233516,"kb_used_data":273572,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92511636,"statfs":{"total":95995035648,"available":94731915264,"internally_reserved":0,"allocated":280137728,"data_stored":279738115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183021,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1344364,"kb_used_data":600996,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92400788,"statfs":{"total":95995035648,"available":94618406912,"internally_reserved":0,"allocated":615419904,"data_stored":615020291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":761199048},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986786,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1623340,"kb_used_data":595364,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":92121812,"statfs":{"total":95995035648,"available":94332735488,"internally_reserved":0,"allocated":609652736,"data_stored":609253123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1052637637},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653347,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258672,"kb_used_data":884392,"kb_used_omap":651,"kb_used_meta":373620,"kb_avail":92486480,"statfs":{"total":95995035648,"available":94706155520,"internally_reserved":0,"allocated":905617408,"data_stored":905213723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":382587656},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277020672,"data_stored":277020672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":560594944,"data_stored":560594944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":609353728,"data_stored":609353728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":607649792,"data_stored":607649792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":904335360,"data_stored":904331288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":280891392,"data_stored":280887320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":429981696,"data_stored":429981696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1349648384,"data_stored":1349648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1031208960,"data_stored":1031208960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:55:15.564 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:55:15.565 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-29T14:55:15.640 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:55:15.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:15 smithi040 ceph-mon[39581]: pgmap v2685: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 474 KiB/s, 7 objects/s recovering 2024-04-29T14:55:15.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4294329130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:55:15.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:15 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:55:15.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:15 smithi121 ceph-mon[44533]: pgmap v2685: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 474 KiB/s, 7 objects/s recovering 2024-04-29T14:55:15.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4294329130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:55:15.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:15 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:55:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:15 smithi194 ceph-mon[44089]: pgmap v2685: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.1 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 474 KiB/s, 7 objects/s recovering 2024-04-29T14:55:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4294329130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:55:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:15 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:55:16.012 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:16.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:16.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:16.154 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:16.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:16.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:16.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:16 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:55:16.860 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:16 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:16.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4294329130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:55:16.861 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:16 smithi121 ceph-mon[44533]: osdmap e1052: 12 total, 11 up, 9 in 2024-04-29T14:55:16.862 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:16.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:16 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:55:16.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:16 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:16.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4294329130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:55:16.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:16 smithi040 ceph-mon[39581]: osdmap e1052: 12 total, 11 up, 9 in 2024-04-29T14:55:16.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:16.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:17.019 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:55:17.052 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:17.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:16 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:55:17.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:16 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:17.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4294329130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:55:17.055 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:16 smithi194 ceph-mon[44089]: osdmap e1052: 12 total, 11 up, 9 in 2024-04-29T14:55:17.077 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:55:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:17.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:55:17.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:17.223 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:17.321 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:55:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:55:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:55:17.322 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:55:17.322 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:55:17.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:17 smithi121 ceph-mon[44533]: pgmap v2687: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:55:17.679 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:17.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:17.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:17 smithi040 ceph-mon[39581]: pgmap v2687: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:55:18.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:17 smithi194 ceph-mon[44089]: pgmap v2687: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.9 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:55:18.291 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:55:18.315 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:55:18.340 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:55:18.364 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:55:18.393 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:55:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:55:18.443 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:55:18.478 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:55:18.511 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:55:18.546 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:55:18.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:55:18.570 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:55:18.570 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:55:18.570 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:55:18.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:55:18.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:55:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:18 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:18 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:18 smithi040 ceph-mon[39581]: osdmap e1053: 12 total, 11 up, 9 in 2024-04-29T14:55:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3077077145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:18 smithi040 ceph-mon[39581]: pgmap v2689: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 869 op/s 2024-04-29T14:55:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:18 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:18 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:18 smithi194 ceph-mon[44089]: osdmap e1053: 12 total, 11 up, 9 in 2024-04-29T14:55:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3077077145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:18 smithi194 ceph-mon[44089]: pgmap v2689: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 869 op/s 2024-04-29T14:55:19.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:19.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:19.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:18 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:19.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:18 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:19.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:18 smithi121 ceph-mon[44533]: osdmap e1053: 12 total, 11 up, 9 in 2024-04-29T14:55:19.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3077077145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:19.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:18 smithi121 ceph-mon[44533]: pgmap v2689: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.8 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 869 op/s 2024-04-29T14:55:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:55:19.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:55:19.550 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:19.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:55:19.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:19.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:19.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:19.750 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:19.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:19.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:19.935 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:19.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:19.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:55:20.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.113 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:20.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:20.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:20.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:20.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:20.422 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:20.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:20 smithi040 ceph-mon[39581]: 8.3 deep-scrub ok 2024-04-29T14:55:20.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:20 smithi040 ceph-mon[39581]: pgmap v2690: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.7 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:55:20.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:20 smithi121 ceph-mon[44533]: 8.3 deep-scrub ok 2024-04-29T14:55:20.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:20 smithi121 ceph-mon[44533]: pgmap v2690: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.7 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:55:20.985 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:20.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:21.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:20 smithi194 ceph-mon[44089]: 8.3 deep-scrub ok 2024-04-29T14:55:21.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:20 smithi194 ceph-mon[44089]: pgmap v2690: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.7 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:55:21.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:21.136 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:21.342 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:21.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:21.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:21.466 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:55:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:21.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:21.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:21.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:21.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:21.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:21.899 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:21.948 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:22.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:22.057 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:22.057 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:22.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:22.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:22.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:22.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:22.207 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:22.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:22.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:22.307 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:55:22.307 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:55:22.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:22.380 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:22.781 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:55:23.032 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:22 smithi121 ceph-mon[44533]: osdmap e1054: 12 total, 11 up, 9 in 2024-04-29T14:55:23.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4288263889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:23.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:22 smithi121 ceph-mon[44533]: pgmap v2692: 17 pgs: 17 active+clean; 1.5 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-29T14:55:23.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:55:23.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:22 smithi194 ceph-mon[44089]: osdmap e1054: 12 total, 11 up, 9 in 2024-04-29T14:55:23.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4288263889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:23.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:22 smithi194 ceph-mon[44089]: pgmap v2692: 17 pgs: 17 active+clean; 1.5 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-29T14:55:23.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:22 smithi040 ceph-mon[39581]: osdmap e1054: 12 total, 11 up, 9 in 2024-04-29T14:55:23.104 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4288263889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:23.104 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:22 smithi040 ceph-mon[39581]: pgmap v2692: 17 pgs: 17 active+clean; 1.5 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-29T14:55:23.745 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:23.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:23.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:23.896 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:24.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:23 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:24.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4288263889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:24.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:23 smithi194 ceph-mon[44089]: osdmap e1055: 12 total, 11 up, 9 in 2024-04-29T14:55:24.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:24.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:24.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:23 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:24.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4288263889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:24.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:23 smithi040 ceph-mon[39581]: osdmap e1055: 12 total, 11 up, 9 in 2024-04-29T14:55:24.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:24.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:24.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:23 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:24.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4288263889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:24.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:23 smithi121 ceph-mon[44533]: osdmap e1055: 12 total, 11 up, 9 in 2024-04-29T14:55:24.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:24.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:25.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:24 smithi194 ceph-mon[44089]: osdmap e1056: 12 total, 11 up, 9 in 2024-04-29T14:55:25.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:24 smithi194 ceph-mon[44089]: pgmap v2695: 17 pgs: 17 active+clean; 1.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:55:25.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:24 smithi040 ceph-mon[39581]: osdmap e1056: 12 total, 11 up, 9 in 2024-04-29T14:55:25.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:24 smithi040 ceph-mon[39581]: pgmap v2695: 17 pgs: 17 active+clean; 1.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:55:25.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:24 smithi121 ceph-mon[44533]: osdmap e1056: 12 total, 11 up, 9 in 2024-04-29T14:55:25.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:24 smithi121 ceph-mon[44533]: pgmap v2695: 17 pgs: 17 active+clean; 1.4 GiB data, 9.6 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:55:25.459 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:25.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:25.484 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:25.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:26.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:26.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:27.027 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:27.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:27.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:27.166 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:55:27.198 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:27.223 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:55:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:27.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:55:27.328 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:55:27.366 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:27.402 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:55:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:27.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:55:27.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:55:27.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:27.567 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:27.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:27 smithi194 ceph-mon[44089]: pgmap v2696: 17 pgs: 17 active+clean; 1.3 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:55:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:27.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:27 smithi040 ceph-mon[39581]: pgmap v2696: 17 pgs: 17 active+clean; 1.3 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:55:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:27.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:27.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:27.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:27 smithi121 ceph-mon[44533]: pgmap v2696: 17 pgs: 17 active+clean; 1.3 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-29T14:55:27.881 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:27.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:27.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:28.034 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:55:28.069 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:28.095 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:55:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:28.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:55:28.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:28.245 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:28.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:28.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:28.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:28.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:28.419 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:28.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:28.555 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:55:28.586 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:55:28.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:28.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:55:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:28.653 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:55:28.693 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:28.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:28.721 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:55:28.759 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.28144.0:242933 8.2 e1057/1011 8:4c07f064:::benchmark_data_smithi194_186548_object114375:head v 1057'15066, mlcod=1056'15064)", 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:55:28.751507+0000", 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.001932853, 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001955975, 2024-04-29T14:55:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751507+0000", 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:55:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751507+0000", 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751507+0000", 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6199999999999998e-07 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751514+0000", 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.618e-06 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751528+0000", 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.361e-05 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751537+0000", 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0389999999999993e-06 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751606+0000", 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9023000000000006e-05 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:55:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:55:28.751635+0000", 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9451999999999999e-05 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-04-29T14:55:28.765 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:55:28.781 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:28.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:28 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:28.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:28 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:28.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:28 smithi040 ceph-mon[39581]: pgmap v2697: 17 pgs: 17 active+clean; 1.1 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:55:28.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/265529196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:28.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:28.803 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:55:28.857 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:55:28.876 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:28.910 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:55:28.945 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:55:28.971 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:55:28.971 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:55:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:55:29.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:28 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:29.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:28 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:29.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:28 smithi194 ceph-mon[44089]: pgmap v2697: 17 pgs: 17 active+clean; 1.1 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:55:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/265529196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:29.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:28 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:29.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:28 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:29.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:28 smithi121 ceph-mon[44533]: pgmap v2697: 17 pgs: 17 active+clean; 1.1 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:55:29.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/265529196' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:29.777 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:29.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:29.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:29.931 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:29.962 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:29.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:29.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:30.068 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osdmap e1057: 12 total, 11 up, 9 in 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:29 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.097 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osdmap e1057: 12 total, 11 up, 9 in 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:29 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:30.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osdmap e1057: 12 total, 11 up, 9 in 2024-04-29T14:55:30.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:29 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:30.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:55:30.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:55:30.270 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:31.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:30 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:31.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:30 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:31.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:30 smithi194 ceph-mon[44089]: osdmap e1058: 12 total, 11 up, 9 in 2024-04-29T14:55:31.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:30 smithi194 ceph-mon[44089]: pgmap v2700: 17 pgs: 17 active+clean; 953 MiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-29T14:55:31.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:30 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:31.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:30 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:31.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:30 smithi040 ceph-mon[39581]: osdmap e1058: 12 total, 11 up, 9 in 2024-04-29T14:55:31.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:30 smithi040 ceph-mon[39581]: pgmap v2700: 17 pgs: 17 active+clean; 953 MiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-29T14:55:31.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:30 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:31.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:30 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:31.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:30 smithi121 ceph-mon[44533]: osdmap e1058: 12 total, 11 up, 9 in 2024-04-29T14:55:31.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:30 smithi121 ceph-mon[44533]: pgmap v2700: 17 pgs: 17 active+clean; 953 MiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-29T14:55:31.163 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:31.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:31.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:31.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:31.313 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:31.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:31.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:31.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:31.495 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:31.758 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:31.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:31.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:55:31.863 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:32.088 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:32.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:32.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:32.236 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:32.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:55:32.265 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:32.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:32.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:32.456 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:32.500 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:32.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:32.527 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:32.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:32.657 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:32.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:32.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:32.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:32.826 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:32.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:32.966 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:33.000 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:33.031 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:33.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:33.140 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:33.171 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:33.196 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:33.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:33.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:33.357 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:33 smithi040 ceph-mon[39581]: pgmap v2701: 17 pgs: 17 active+clean; 754 MiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.45k op/s 2024-04-29T14:55:33.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2979938771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:55:33.712 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:55:33.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:33 smithi194 ceph-mon[44089]: pgmap v2701: 17 pgs: 17 active+clean; 754 MiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.45k op/s 2024-04-29T14:55:33.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2979938771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:55:33.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:33 smithi121 ceph-mon[44533]: pgmap v2701: 17 pgs: 17 active+clean; 754 MiB data, 8.2 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 1.45k op/s 2024-04-29T14:55:33.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2979938771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:55:34.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:34 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2979938771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:55:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:34 smithi194 ceph-mon[44089]: osdmap e1059: 12 total, 11 up, 9 in 2024-04-29T14:55:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:34 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2979938771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:55:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:34 smithi040 ceph-mon[39581]: osdmap e1059: 12 total, 11 up, 9 in 2024-04-29T14:55:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:34 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2979938771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:55:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:34 smithi121 ceph-mon[44533]: osdmap e1059: 12 total, 11 up, 9 in 2024-04-29T14:55:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:35.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:35.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:35.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:35.561 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:35.587 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:35.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:35.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:35 smithi040 ceph-mon[39581]: pgmap v2703: 17 pgs: 17 active+clean; 678 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:55:35.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:35.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:35 smithi121 ceph-mon[44533]: pgmap v2703: 17 pgs: 17 active+clean; 678 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:55:35.727 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:35.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:35.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:35.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:35.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:35 smithi194 ceph-mon[44089]: pgmap v2703: 17 pgs: 17 active+clean; 678 MiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:55:35.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:35.891 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:35.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:35.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:36.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:36.197 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:36.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:36.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:36.340 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:36.366 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:36.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:36.471 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:55:36.502 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:36.530 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:55:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:36.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:55:36.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:36.685 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:36.710 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:36.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:36.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:36.877 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:36.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:36.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:37.035 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:37.058 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:37.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:37.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:37.202 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:37.230 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:37.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:37.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:37.372 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:37.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:37.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:37.770 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:37 smithi121 ceph-mon[44533]: pgmap v2704: 17 pgs: 17 active+clean; 480 MiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:55:37.779 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:37.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:37.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:37 smithi194 ceph-mon[44089]: pgmap v2704: 17 pgs: 17 active+clean; 480 MiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:55:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:37 smithi040 ceph-mon[39581]: pgmap v2704: 17 pgs: 17 active+clean; 480 MiB data, 7.7 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-29T14:55:37.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:37.977 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:37.993 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:55:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:55:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:55:37.993 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:55:37.993 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:55:38.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:55:38.449 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:38.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:38.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:38.604 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:38.717 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:38.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:38.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:38.861 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:38.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:38.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:38.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:39.025 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:39.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:39.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:39.197 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:39.230 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:39.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:39.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:39.375 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:39.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:39.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:39.515 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:39.553 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:39.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:39 smithi121 ceph-mon[44533]: pgmap v2705: 17 pgs: 17 active+clean; 357 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:55:39.721 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:55:39.753 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:55:39.780 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:55:39.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:39 smithi194 ceph-mon[44089]: pgmap v2705: 17 pgs: 17 active+clean; 357 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:55:39.857 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:55:39.858 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:55:39.865 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 128555 objects 2024-04-29T14:55:39.865 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :116.248 2024-04-29T14:55:39.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:39 smithi040 ceph-mon[39581]: pgmap v2705: 17 pgs: 17 active+clean; 357 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-29T14:55:39.874 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-04-29T14:55:39.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-04-29T14:55:39.878 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:55:39.899 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:55:39.919 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:55:39.939 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:55:39.960 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:55:39.980 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:55:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:55:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:55:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:55:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:55:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:55:40.797 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/439426659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:40.797 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.797 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.797 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:40 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/439426659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:40 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/439426659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:40.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:40 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:41.044 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:41.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:41.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:41.177 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:55:41.650 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:55:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:41 smithi194 ceph-mon[44089]: pgmap v2706: 17 pgs: 17 active+clean; 156 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-29T14:55:41.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:41 smithi040 ceph-mon[39581]: pgmap v2706: 17 pgs: 17 active+clean; 156 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-29T14:55:41.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:41 smithi121 ceph-mon[44533]: pgmap v2706: 17 pgs: 17 active+clean; 156 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.27k op/s 2024-04-29T14:55:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:41.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:42.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:42.065 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:42.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:42.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:42.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:42.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:42.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:42.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:42.374 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:42.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:42.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:42.442 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:42.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:55:42.526 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:55:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:42.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:55:42.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:43.027 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:43.036 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:43.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:43.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:43.170 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:43.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:43.399 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:43.432 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:43.467 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:43.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:43.541 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:43 smithi040 ceph-mon[39581]: pgmap v2707: 17 pgs: 17 active+clean; 22 MiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-29T14:55:43.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:43.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:55:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:43 smithi194 ceph-mon[44089]: pgmap v2707: 17 pgs: 17 active+clean; 22 MiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-29T14:55:43.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:43 smithi121 ceph-mon[44533]: pgmap v2707: 17 pgs: 17 active+clean; 22 MiB data, 6.8 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-29T14:55:44.152 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:44.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:44.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:44.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1134596430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:44.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1134596430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:44.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1134596430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:55:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:44.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:45.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:45.071 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:45.094 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:45.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:45.198 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:45.228 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:45.327 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:45.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:45.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:45.465 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:45.487 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:45.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:45.591 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:55:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:55:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:45.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:55:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:55:45.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:45.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:45 smithi194 ceph-mon[44089]: pgmap v2708: 17 pgs: 17 active+clean; 5.3 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:55:45.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:45 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:55:45.776 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:45 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:55:45.776 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1134596430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:45.776 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:45 smithi194 ceph-mon[44089]: osdmap e1060: 12 total, 11 up, 9 in 2024-04-29T14:55:45.778 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:55:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:45.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:55:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:45 smithi040 ceph-mon[39581]: pgmap v2708: 17 pgs: 17 active+clean; 5.3 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:55:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:45 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:55:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:45 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:55:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1134596430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:45 smithi040 ceph-mon[39581]: osdmap e1060: 12 total, 11 up, 9 in 2024-04-29T14:55:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:45 smithi121 ceph-mon[44533]: pgmap v2708: 17 pgs: 17 active+clean; 5.3 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:55:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:45 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:55:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:45 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:55:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1134596430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:55:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:45 smithi121 ceph-mon[44533]: osdmap e1060: 12 total, 11 up, 9 in 2024-04-29T14:55:45.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:45.933 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:45.954 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:45.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:46.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:46.089 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:46.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:46.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:46.244 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:46.408 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:55:46.408 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:55:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:47.194 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:47.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:47.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:47.332 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:47.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:47.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:47.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:47.488 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:47.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:47 smithi121 ceph-mon[44533]: pgmap v2710: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 767 op/s 2024-04-29T14:55:47.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:47 smithi121 ceph-mon[44533]: osdmap e1061: 12 total, 11 up, 9 in 2024-04-29T14:55:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:47.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:47.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:47.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:47 smithi040 ceph-mon[39581]: pgmap v2710: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 767 op/s 2024-04-29T14:55:47.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:47 smithi040 ceph-mon[39581]: osdmap e1061: 12 total, 11 up, 9 in 2024-04-29T14:55:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:47 smithi194 ceph-mon[44089]: pgmap v2710: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 767 op/s 2024-04-29T14:55:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:47 smithi194 ceph-mon[44089]: osdmap e1061: 12 total, 11 up, 9 in 2024-04-29T14:55:48.074 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:48.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:48.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T14:55:48.081 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:48.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:48.206 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:48.331 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:48.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:48.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:48.466 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:48.591 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:48.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:48.695 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:49.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:55:49.503 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:49 smithi040 ceph-mon[39581]: pgmap v2712: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 309 op/s 2024-04-29T14:55:49.503 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/177557166' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:55:49.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:49 smithi194 ceph-mon[44089]: pgmap v2712: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 309 op/s 2024-04-29T14:55:49.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/177557166' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:55:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:49 smithi121 ceph-mon[44533]: pgmap v2712: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 309 op/s 2024-04-29T14:55:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/177557166' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T14:55:50.002 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_6' removed 2024-04-29T14:55:50.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/177557166' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:55:50.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:50 smithi040 ceph-mon[39581]: osdmap e1062: 12 total, 11 up, 9 in 2024-04-29T14:55:50.697 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-29T14:55:50.698 INFO:tasks.radosbench:Beginning radosbench... 2024-04-29T14:55:50.698 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-04-29T14:55:50.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-04-29T14:55:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/177557166' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:55:50.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:50 smithi194 ceph-mon[44089]: osdmap e1062: 12 total, 11 up, 9 in 2024-04-29T14:55:50.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/177557166' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T14:55:50.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:50 smithi121 ceph-mon[44533]: osdmap e1062: 12 total, 11 up, 9 in 2024-04-29T14:55:51.003 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:51.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:51.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:51.139 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:51.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:51.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:51.251 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:51.264 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:55:51.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:51.289 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:51.303 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:55:51.303 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:51.309 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:51.328 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:55:51.348 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:55:51.368 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:55:51.388 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:55:51.408 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:55:51.428 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:55:51.449 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:55:51.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:55:51.490 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:55:51.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:55:51.490 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:55:51.491 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:55:51.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:51 smithi040 ceph-mon[39581]: pgmap v2714: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:51.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3596858875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:51.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:51.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:51.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:51 smithi194 ceph-mon[44089]: pgmap v2714: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3596858875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:51.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:51 smithi121 ceph-mon[44533]: pgmap v2714: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3596858875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:55:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:51.898 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:52.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:52.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:55:52.161 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:52.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:55:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:52.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:55:52.284 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:55:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:52.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:55:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:53 smithi040 ceph-mon[39581]: pgmap v2715: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:53.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:53 smithi194 ceph-mon[44089]: pgmap v2715: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:53 smithi121 ceph-mon[44533]: pgmap v2715: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:55:54.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:55:55.566 INFO:tasks.thrashosds.thrasher:Reviving osd 9 2024-04-29T14:55:55.566 INFO:tasks.cephadm.osd.9:Restarting osd.9 (starting--it wasn't running)... 2024-04-29T14:55:55.566 DEBUG:teuthology.orchestra.run.smithi194:osd.9> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.9.service 2024-04-29T14:55:55.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl start ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.9 2024-04-29T14:55:55.603 INFO:journalctl@ceph.osd.9.smithi194.stdout:-- Logs begin at Mon 2024-04-29 13:48:49 UTC. -- 2024-04-29T14:55:55.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:55 smithi194 ceph-mon[44089]: pgmap v2716: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:55.825 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:55 smithi194 systemd[1]: Starting Ceph osd.9 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:55:55.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:55 smithi040 ceph-mon[39581]: pgmap v2716: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:55 smithi121 ceph-mon[44533]: pgmap v2716: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:55.995 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:55:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:55.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:55:56.100 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:56.132 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:56.141 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:55:56.141 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:55:56.141 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T14:55:56.141 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:55:56.156 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:56.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:56.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:56.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:56.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:56.474 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:56.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:56.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:56 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:55:56.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:56 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:55:56.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:56 smithi040 ceph-mon[39581]: osdmap e1063: 12 total, 11 up, 9 in 2024-04-29T14:55:56.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:56.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:55:56.648 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:56.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:56.752 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:55:56.781 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:56.803 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:55:56.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:56 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:55:56.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:56 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:55:56.805 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:56 smithi194 ceph-mon[44089]: osdmap e1063: 12 total, 11 up, 9 in 2024-04-29T14:55:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:56.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:55:56.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:56 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:55:56.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:56 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:55:56.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:56 smithi121 ceph-mon[44533]: osdmap e1063: 12 total, 11 up, 9 in 2024-04-29T14:55:56.906 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:55:56.933 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:56.957 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:55:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:56.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:55:57.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:55:57.092 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:57.114 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:55:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:57.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:55:57.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T14:55:57.218 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:55:57.265 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:57.339 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:55:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:57.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:55:57.443 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:57.483 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:57.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:55:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-9 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-9/block 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-29T14:55:57.540 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:57 smithi194 bash[195027]: --> ceph-volume raw activate successful for osd ID: 9 2024-04-29T14:55:57.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:55:57.644 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:57.666 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:55:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:57.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:55:57.759 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:57 smithi040 ceph-mon[39581]: pgmap v2718: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:57.759 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2573799611' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:55:57.770 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:55:57.799 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:57 smithi194 ceph-mon[44089]: pgmap v2718: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2573799611' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:55:57.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:57 smithi121 ceph-mon[44533]: pgmap v2718: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:57.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2573799611' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:55:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:55:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:58.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:55:58.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:55:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:58.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:58 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:58.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2573799611' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:55:58.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:58 smithi040 ceph-mon[39581]: osdmap e1064: 12 total, 11 up, 9 in 2024-04-29T14:55:58.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3028365131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-29T14:55:58.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:58 smithi040 ceph-mon[39581]: from='client.28351 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-29T14:55:58.645 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:55:58.673 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:58 smithi194 bash[195407]: d63e9d798e9d40c7d3c109a703cd2e05c112dcc044453f2e4e56785c3ac163bf 2024-04-29T14:55:58.673 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:58 smithi194 systemd[1]: Started Ceph osd.9 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T14:55:58.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:58 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:58.673 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2573799611' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:55:58.674 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:58 smithi194 ceph-mon[44089]: osdmap e1064: 12 total, 11 up, 9 in 2024-04-29T14:55:58.674 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3028365131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-29T14:55:58.674 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:58 smithi194 ceph-mon[44089]: from='client.28351 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-29T14:55:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_7' created 2024-04-29T14:55:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:58 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:55:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2573799611' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:55:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:58 smithi121 ceph-mon[44533]: osdmap e1064: 12 total, 11 up, 9 in 2024-04-29T14:55:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3028365131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-29T14:55:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:58 smithi121 ceph-mon[44533]: from='client.28351 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-29T14:55:58.938 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:55:59.324 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:55:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T14:55:59.033+0000 7fb557b797c0 -1 Falling back to public interface 2024-04-29T14:55:59.633 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:55:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:55:59.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:55:59.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:55:59.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:59 smithi194 ceph-mon[44089]: pgmap v2720: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:59.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:59 smithi194 ceph-mon[44089]: from='client.28351 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-29T14:55:59.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:59 smithi194 ceph-mon[44089]: osdmap e1065: 12 total, 11 up, 9 in 2024-04-29T14:55:59.765 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:55:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:55:59.768 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:55:59.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:59 smithi040 ceph-mon[39581]: pgmap v2720: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:59.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:59 smithi040 ceph-mon[39581]: from='client.28351 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-29T14:55:59.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:59 smithi040 ceph-mon[39581]: osdmap e1065: 12 total, 11 up, 9 in 2024-04-29T14:55:59.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:55:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:55:59.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-04-29T14:55:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:59 smithi121 ceph-mon[44533]: pgmap v2720: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:55:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:59 smithi121 ceph-mon[44533]: from='client.28351 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-29T14:55:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:59 smithi121 ceph-mon[44533]: osdmap e1065: 12 total, 11 up, 9 in 2024-04-29T14:55:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:55:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:56:00.318 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:00.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:00.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:00.504 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:00.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:00.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:56:00.643 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:00.665 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:56:00.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:00.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:56:00.769 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:01.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:00 smithi040 ceph-mon[39581]: osdmap e1066: 12 total, 11 up, 9 in 2024-04-29T14:56:01.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:01.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:01.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:00 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:01.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:00 smithi040 ceph-mon[39581]: pgmap v2723: 17 pgs: 5 creating+peering, 9 unknown, 3 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:00 smithi194 ceph-mon[44089]: osdmap e1066: 12 total, 11 up, 9 in 2024-04-29T14:56:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:01.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:01.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:00 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:01.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:00 smithi194 ceph-mon[44089]: pgmap v2723: 17 pgs: 5 creating+peering, 9 unknown, 3 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:no valid command found; 10 closest matches: 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:0 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:1 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:2 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:abort 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:assert 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:config diff 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:config diff get 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:config get 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:config help [] 2024-04-29T14:56:01.088 INFO:teuthology.orchestra.run.smithi194.stderr:config set ... 2024-04-29T14:56:01.089 INFO:teuthology.orchestra.run.smithi194.stderr:admin_socket: invalid command 2024-04-29T14:56:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:00 smithi121 ceph-mon[44533]: osdmap e1066: 12 total, 11 up, 9 in 2024-04-29T14:56:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:00 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:00 smithi121 ceph-mon[44533]: pgmap v2723: 17 pgs: 5 creating+peering, 9 unknown, 3 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:01.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:01.755 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:01.787 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:01.912 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:01.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T14:56:01.943 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-9, ['dump_ops_in_flight'] 2024-04-29T14:56:01.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osdmap e1067: 12 total, 11 up, 9 in 2024-04-29T14:56:01.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:56:01.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/567810114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:01.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:01.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:02.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osdmap e1067: 12 total, 11 up, 9 in 2024-04-29T14:56:02.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:56:02.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/567810114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:02.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.047 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.049 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:02.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:02.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:02.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osdmap e1067: 12 total, 11 up, 9 in 2024-04-29T14:56:02.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-29T14:56:02.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/567810114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:02.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.175 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:02.202 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:02.221 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:02.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:02.264 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:56:02.264 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:56:02.325 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:02.352 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:02.396 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:02.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:02.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:02.531 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:02.574 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:56:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T14:56:02.338+0000 7fb557b797c0 -1 osd.9 1019 log_to_monitors true 2024-04-29T14:56:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:02 smithi040 ceph-mon[39581]: from='osd.9 v1:172.21.15.194:6804/3807855610' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:56:02.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:02 smithi040 ceph-mon[39581]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:56:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:02 smithi040 ceph-mon[39581]: pgmap v2725: 17 pgs: 5 creating+peering, 4 unknown, 8 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:02.980 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:02.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:03.007 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T14:56:03.074 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 14:56:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T14:56:02.742+0000 7fb548fd1700 -1 osd.9 1068 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T14:56:03.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:02 smithi194 ceph-mon[44089]: from='osd.9 v1:172.21.15.194:6804/3807855610' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:56:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:02 smithi194 ceph-mon[44089]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:56:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:02 smithi194 ceph-mon[44089]: pgmap v2725: 17 pgs: 5 creating+peering, 4 unknown, 8 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:03.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:03.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:03.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:02 smithi121 ceph-mon[44533]: from='osd.9 v1:172.21.15.194:6804/3807855610' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:56:03.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:02 smithi121 ceph-mon[44533]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-29T14:56:03.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:02 smithi121 ceph-mon[44533]: pgmap v2725: 17 pgs: 5 creating+peering, 4 unknown, 8 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:03.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:56:03.143 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:56:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:56:03.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:03.272 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:03.291 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:03.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:03.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:03.423 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:03.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:03.584 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:03.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:56:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:03.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:56:03.839 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:56:03.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:04.068 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:04.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:03 smithi040 ceph-mon[39581]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-29T14:56:04.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:03 smithi040 ceph-mon[39581]: osdmap e1068: 12 total, 11 up, 9 in 2024-04-29T14:56:04.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:03 smithi040 ceph-mon[39581]: from='osd.9 v1:172.21.15.194:6804/3807855610' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:56:04.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:03 smithi040 ceph-mon[39581]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:56:04.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:03 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:56:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:04.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:03 smithi194 ceph-mon[44089]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-29T14:56:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:03 smithi194 ceph-mon[44089]: osdmap e1068: 12 total, 11 up, 9 in 2024-04-29T14:56:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:03 smithi194 ceph-mon[44089]: from='osd.9 v1:172.21.15.194:6804/3807855610' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:56:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:03 smithi194 ceph-mon[44089]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:56:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:03 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:56:04.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:03 smithi121 ceph-mon[44533]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-29T14:56:04.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:03 smithi121 ceph-mon[44533]: osdmap e1068: 12 total, 11 up, 9 in 2024-04-29T14:56:04.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:03 smithi121 ceph-mon[44533]: from='osd.9 v1:172.21.15.194:6804/3807855610' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:56:04.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:03 smithi121 ceph-mon[44533]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-29T14:56:04.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:03 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:56:04.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:04.675 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:04.711 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:04.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:04.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:04 smithi040 ceph-mon[39581]: osd.9 v1:172.21.15.194:6804/3807855610 boot 2024-04-29T14:56:04.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:04 smithi040 ceph-mon[39581]: osdmap e1069: 12 total, 12 up, 9 in 2024-04-29T14:56:04.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:04 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:56:04.990 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:04 smithi040 ceph-mon[39581]: pgmap v2728: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:05.053 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:04 smithi194 ceph-mon[44089]: osd.9 v1:172.21.15.194:6804/3807855610 boot 2024-04-29T14:56:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:04 smithi194 ceph-mon[44089]: osdmap e1069: 12 total, 12 up, 9 in 2024-04-29T14:56:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:04 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:56:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:04 smithi194 ceph-mon[44089]: pgmap v2728: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:05.085 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:05.105 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:05.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:05.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:04 smithi121 ceph-mon[44533]: osd.9 v1:172.21.15.194:6804/3807855610 boot 2024-04-29T14:56:05.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:04 smithi121 ceph-mon[44533]: osdmap e1069: 12 total, 12 up, 9 in 2024-04-29T14:56:05.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:04 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-29T14:56:05.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:04 smithi121 ceph-mon[44533]: pgmap v2728: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:05.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:05.241 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:05.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:05.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:05.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:56:05.397 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:05.417 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:56:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:05.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:56:05.521 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:05.553 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:05.574 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:05.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:05.679 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:05.708 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:05.728 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:05.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:05.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:05 smithi194 ceph-mon[44089]: osdmap e1070: 12 total, 12 up, 9 in 2024-04-29T14:56:05.832 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:05.840 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:56:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:05.895 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:05.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:06.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:06.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:06.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:06.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:06.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:05 smithi040 ceph-mon[39581]: osdmap e1070: 12 total, 12 up, 9 in 2024-04-29T14:56:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:06.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:06.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:06.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:06.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:05 smithi121 ceph-mon[44533]: osdmap e1070: 12 total, 12 up, 9 in 2024-04-29T14:56:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:06.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:06.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:06.288 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:06.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:06.541 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:06.573 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:06.594 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:06.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:06.698 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:06 smithi194 ceph-mon[44089]: pgmap v2730: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:06.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:06 smithi040 ceph-mon[39581]: pgmap v2730: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:06.944 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:56:07.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:06 smithi121 ceph-mon[44533]: pgmap v2730: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:07.231 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:56:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:07.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:07.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:07.714 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:07.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:07.751 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-04-29T14:56:07.849 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4227998582' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:56:07.880 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:08.003 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:08.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:08.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:08.005 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4227998582' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:56:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:08.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:08.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:08.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:08.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:08.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4227998582' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-29T14:56:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:08.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:08.339 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:08.466 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:08.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:08.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:08.598 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:08.618 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:08.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:08.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4227998582' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:56:08.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:08 smithi194 ceph-mon[44089]: osdmap e1071: 12 total, 12 up, 9 in 2024-04-29T14:56:08.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:08 smithi194 ceph-mon[44089]: pgmap v2732: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:08.873 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:56:08.873 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:56:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:08.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:08.989 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:09.018 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:09.055 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:09.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:09.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4227998582' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:56:09.090 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:08 smithi040 ceph-mon[39581]: osdmap e1071: 12 total, 12 up, 9 in 2024-04-29T14:56:09.091 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:08 smithi040 ceph-mon[39581]: pgmap v2732: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:09.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4227998582' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-29T14:56:09.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:08 smithi121 ceph-mon[44533]: osdmap e1071: 12 total, 12 up, 9 in 2024-04-29T14:56:09.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:08 smithi121 ceph-mon[44533]: pgmap v2732: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:09.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:09.187 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:09.206 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:09.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:09.253 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:56:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:56:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:56:09.253 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:56:09.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:09.961 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:09 smithi194 ceph-mon[44089]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:56:09.988 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T14:56:09.988 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 4, 7, 10, 6, 1, 5, 8, 3] 2024-04-29T14:56:09.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 0 2024-04-29T14:56:10.016 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:09 smithi040 ceph-mon[39581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:56:10.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:09 smithi121 ceph-mon[44533]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-29T14:56:10.490 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:10.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:10.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:10.627 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:10.648 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:10.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:10.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:10.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:10 smithi121 ceph-mon[44533]: pgmap v2733: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:10.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:10 smithi040 ceph-mon[39581]: pgmap v2733: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:10.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:11.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:10 smithi194 ceph-mon[44089]: pgmap v2733: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 798 GiB / 805 GiB avail 2024-04-29T14:56:11.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:11.114 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:11.451 DEBUG:teuthology.orchestra.run.smithi194:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-04-29T14:56:11.455 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-29T14:56:11.455 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-29T14:56:11.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:hints = 1 2024-04-29T14:56:11.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-29T14:56:11.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object prefix: benchmark_data_smithi194_196924 2024-04-29T14:56:11.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:56:11.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 0 0 0 0 0 0 - 0 2024-04-29T14:56:11.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:56:11.667 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:11.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:11.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:11.772 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:11.782 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:56:11.832 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:11.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:11 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:11.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:11 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:11.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:11 smithi040 ceph-mon[39581]: osdmap e1072: 12 total, 12 up, 9 in 2024-04-29T14:56:11.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/769892426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:56:12.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:11 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:12.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:11 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:12.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:11 smithi194 ceph-mon[44089]: osdmap e1072: 12 total, 12 up, 9 in 2024-04-29T14:56:12.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/769892426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:56:12.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:11 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:12.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:11 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:12.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:11 smithi121 ceph-mon[44533]: osdmap e1072: 12 total, 12 up, 9 in 2024-04-29T14:56:12.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/769892426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:56:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:12.511 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:12.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:12.527 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:56:12.549 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:56:12.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 1 16 5484 5468 341.728 341.75 0.00145422 0.00280285 2024-04-29T14:56:12.568 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:56:12.587 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:56:12.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:56:12.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:56:12.627 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:56:12.638 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:12.650 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:56:12.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:56:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:12.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:56:12.669 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:56:12.689 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:56:12.718 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:56:12.749 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:56:12.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:12.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:12 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:56:12.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/769892426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:56:12.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:12 smithi040 ceph-mon[39581]: osdmap e1073: 12 total, 12 up, 9 in 2024-04-29T14:56:12.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:12 smithi040 ceph-mon[39581]: pgmap v2736: 17 pgs: 17 active+clean; 18 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 3.0 MiB/s wr, 47 op/s 2024-04-29T14:56:12.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2042195842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:12.783 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:56:12.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:56:12.798 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:12.844 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:12.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:12.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:12.992 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:13.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:12 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:56:13.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/769892426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:56:13.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:12 smithi194 ceph-mon[44089]: osdmap e1073: 12 total, 12 up, 9 in 2024-04-29T14:56:13.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:12 smithi194 ceph-mon[44089]: pgmap v2736: 17 pgs: 17 active+clean; 18 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 3.0 MiB/s wr, 47 op/s 2024-04-29T14:56:13.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2042195842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:13.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:12 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:56:13.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/769892426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:56:13.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:12 smithi121 ceph-mon[44533]: osdmap e1073: 12 total, 12 up, 9 in 2024-04-29T14:56:13.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:12 smithi121 ceph-mon[44533]: pgmap v2736: 17 pgs: 17 active+clean; 18 MiB data, 6.7 GiB used, 798 GiB / 805 GiB avail; 3.0 MiB/s wr, 47 op/s 2024-04-29T14:56:13.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2042195842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:13.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 2 15 10085 10070 314.663 287.625 0.00582429 0.0031742 2024-04-29T14:56:14.055 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.059 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.059 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:14.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:14.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:14.195 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:14.219 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:14.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:14.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:14.350 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:14.369 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:14.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:14.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:14.515 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:14.543 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:14.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:14.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 3 16 14128 14112 293.978 252.625 0.00238962 0.00339869 2024-04-29T14:56:14.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:15.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:14 smithi040 ceph-mon[39581]: pgmap v2737: 17 pgs: 17 active+clean; 395 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 59 MiB/s wr, 950 op/s 2024-04-29T14:56:15.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:14 smithi194 ceph-mon[44089]: pgmap v2737: 17 pgs: 17 active+clean; 395 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 59 MiB/s wr, 950 op/s 2024-04-29T14:56:15.124 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:15.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:14 smithi121 ceph-mon[44533]: pgmap v2737: 17 pgs: 17 active+clean; 395 MiB data, 7.0 GiB used, 798 GiB / 805 GiB avail; 59 MiB/s wr, 950 op/s 2024-04-29T14:56:15.148 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:15.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:15.253 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:15.286 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:15.309 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:15.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:15.413 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:56:15.445 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:15.470 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:56:15.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:15.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:56:15.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 4 15 18549 18534 289.572 276.375 0.00359228 0.00345063 2024-04-29T14:56:15.575 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:15.608 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:15.631 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:15.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:15.737 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:16.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 5 16 22710 22694 283.653 260 0.00163762 0.00352338 2024-04-29T14:56:17.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:56:17.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 6 16 26519 26503 276.052 238.062 0.00344501 0.00362019 2024-04-29T14:56:17.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:17 smithi040 ceph-mon[39581]: pgmap v2738: 17 pgs: 17 active+clean; 593 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:56:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:17 smithi194 ceph-mon[44089]: pgmap v2738: 17 pgs: 17 active+clean; 593 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:56:17.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:17 smithi121 ceph-mon[44533]: pgmap v2738: 17 pgs: 17 active+clean; 593 MiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-29T14:56:18.138 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:18.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:18.247 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:56:18.281 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:18.307 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:56:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:18.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:56:18.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:18.454 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:18.553 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 7 16 30039 30023 268.042 220 0.00237235 0.00372907 2024-04-29T14:56:18.561 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:18.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:18.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:56:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:18.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:56:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:18.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:56:18.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:18.888 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:18.997 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:18 smithi040 ceph-mon[39581]: pgmap v2739: 17 pgs: 17 active+clean; 938 MiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-04-29T14:56:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:18 smithi194 ceph-mon[44089]: pgmap v2739: 17 pgs: 17 active+clean; 938 MiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-04-29T14:56:19.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:18 smithi121 ceph-mon[44533]: pgmap v2739: 17 pgs: 17 active+clean; 938 MiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-04-29T14:56:19.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 8 15 33169 33154 258.996 195.688 0.00208721 0.00385865 2024-04-29T14:56:19.946 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:19.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:20.037 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:20.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:20.091 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:20.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:56:20.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 9 16 36243 36227 251.557 192.062 0.0064347 0.00397354 2024-04-29T14:56:21.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 10 16 39244 39228 245.156 187.562 0.00775172 0.00407678 2024-04-29T14:56:21.682 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:21.686 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:21 smithi040 ceph-mon[39581]: pgmap v2740: 17 pgs: 17 active+clean; 1.7 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-29T14:56:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:21.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:21.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:21 smithi194 ceph-mon[44089]: pgmap v2740: 17 pgs: 17 active+clean; 1.7 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-29T14:56:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:21.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:21.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:21 smithi121 ceph-mon[44533]: pgmap v2740: 17 pgs: 17 active+clean; 1.7 GiB data, 9.9 GiB used, 795 GiB / 805 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-29T14:56:21.979 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:56:22.011 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:22.035 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:56:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:22.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:56:22.141 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:22.172 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:22.195 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:22.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:22.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:22.349 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:22.378 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:22.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:22.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:22.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:22.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:22.539 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:22.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 11 16 42055 42039 238.84 175.688 0.00320356 0.00418532 2024-04-29T14:56:22.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:22.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:22.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:22.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:23.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 12 16 44537 44521 231.863 155.125 0.00298293 0.0043117 2024-04-29T14:56:23.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: osdmap e1074: 12 total, 12 up, 9 in 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:23.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: osdmap e1074: 12 total, 12 up, 9 in 2024-04-29T14:56:23.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:23.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: osdmap e1074: 12 total, 12 up, 9 in 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:23.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:23.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:23.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:24.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:23 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:56:23.844+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1075 definitely_dead 0 2024-04-29T14:56:24.545 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:56:24.546 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:56:24.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.546 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:46993 9.1 9:8264427b:::benchmark_data_smithi194_196924_object46992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.546 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.466696+0000", 2024-04-29T14:56:24.546 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.0073111809999999999, 2024-04-29T14:56:24.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0073445990000000003, 2024-04-29T14:56:24.546 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 46993 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466696+0000", 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466696+0000", 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466697+0000", 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.172e-06 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466730+0000", 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3068999999999998e-05 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466751+0000", 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0458999999999998e-05 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466759+0000", 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1040000000000007e-06 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.471657+0000", 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048978779999999996 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.471794+0000", 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013708899999999999 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:56:24.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.471902+0000", 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010826499999999999 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:46995 9.2 9:41f74106:::benchmark_data_smithi194_196924_object46994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.466916+0000", 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.007091572, 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.007213807, 2024-04-29T14:56:24.552 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 46995 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466916+0000", 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466916+0000", 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466917+0000", 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9999999999999996e-07 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466965+0000", 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8077000000000001e-05 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466972+0000", 2024-04-29T14:56:24.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.221e-06 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466978+0000", 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9699999999999996e-06 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468685+0000", 2024-04-29T14:56:24.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017070480000000001 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468812+0000", 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012757400000000001 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468898+0000", 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5661000000000005e-05 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470138+0000", 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012401790000000001 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:46999 9.1 9:8de07871:::benchmark_data_smithi194_196924_object46998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.468263+0000", 2024-04-29T14:56:24.558 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.0057445079999999997, 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059526329999999997, 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 46999 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468263+0000", 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468263+0000", 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468264+0000", 2024-04-29T14:56:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0920000000000001e-06 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468329+0000", 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5214000000000003e-05 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468335+0000", 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.604e-06 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.468339+0000", 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1280000000000001e-06 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.473056+0000", 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0047175669999999998 2024-04-29T14:56:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.473150+0000", 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3560000000000005e-05 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.473259+0000", 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010967 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:47004 9.c 9:36e28e80:::benchmark_data_smithi194_196924_object47003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.470066+0000", 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.0039412809999999996, 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004238777, 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 47004 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470066+0000", 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470066+0000", 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470068+0000", 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.61e-06 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470183+0000", 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115204 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470191+0000", 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1489999999999994e-06 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470198+0000", 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.968e-06 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.471576+0000", 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0013784450000000001 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.471684+0000", 2024-04-29T14:56:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107747 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 4", 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.471816+0000", 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013227000000000001 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:47008 9.2 9:4aa274d8:::benchmark_data_smithi194_196924_object47007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.470549+0000", 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.003458034, 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0038652349999999999, 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 47008 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470549+0000", 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470549+0000", 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470551+0000", 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5290000000000001e-06 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470582+0000", 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1183999999999998e-05 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470592+0000", 2024-04-29T14:56:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0526999999999999e-05 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470601+0000", 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2670000000000006e-06 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470645+0000", 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4815999999999999e-05 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470764+0000", 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118422 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470884+0000", 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000120405 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-04-29T14:56:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472258+0000", 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0013736919999999999 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(client.28381.0:46992 9.1 e1075/1065)", 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.472550+0000", 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.0014575160000000001, 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019490460000000001, 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472550+0000", 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472550+0000", 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472551+0000", 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4200000000000004e-07 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472555+0000", 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9820000000000002e-06 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472565+0000", 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.978e-06 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472572+0000", 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9890000000000003e-06 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:47009 9.3 9:cac72c1a:::benchmark_data_smithi194_196924_object47008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.472595+0000", 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.001412593, 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019580550000000002, 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 47009 2024-04-29T14:56:24.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472595+0000", 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472595+0000", 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472596+0000", 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0380000000000001e-06 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472625+0000", 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9810000000000001e-05 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472636+0000", 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0655e-05 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472643+0000", 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0169999999999999e-06 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472699+0000", 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5787e-05 2024-04-29T14:56:24.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472836+0000", 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013727999999999999 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 5", 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472959+0000", 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000122719 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:47010 9.2 9:41dad37f:::benchmark_data_smithi194_196924_object47009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.472659+0000", 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.0013483900000000001, 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019929790000000002, 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.584 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 47010 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472659+0000", 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472659+0000", 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472660+0000", 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4300000000000002e-07 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472738+0000", 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8188000000000004e-05 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472758+0000", 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9814e-05 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472765+0000", 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6699999999999994e-06 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472844+0000", 2024-04-29T14:56:24.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8883999999999999e-05 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.472962+0000", 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118031 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 1", 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.473142+0000", 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000180378 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.588 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.28381.0:46992 9.1 9:88d89ab4:::benchmark_data_smithi194_196924_object46991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e1075)", 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:56:24.466483+0000", 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.0075242479999999999, 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082691499999999994, 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.28381", 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 46992 2024-04-29T14:56:24.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466483+0000", 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466483+0000", 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466484+0000", 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.477e-06 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466515+0000", 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.04e-05 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466523+0000", 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6149999999999997e-06 2024-04-29T14:56:24.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.466531+0000", 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.418e-06 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470048+0000", 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035174889999999999 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-04-29T14:56:24.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470167+0000", 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118353 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 6", 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:56:24.470309+0000", 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014278699999999999 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 9 2024-04-29T14:56:24.593 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:56:24.594 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 13 16 47227 47211 226.959 168.125 0.00497277 0.00440427 2024-04-29T14:56:24.632 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.0. 2024-04-29T14:56:24.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:24 smithi194 ceph-mon[44089]: osdmap e1075: 12 total, 12 up, 9 in 2024-04-29T14:56:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2031353046' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-29T14:56:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:24 smithi194 ceph-mon[44089]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-29T14:56:24.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:24 smithi040 ceph-mon[39581]: osdmap e1075: 12 total, 12 up, 9 in 2024-04-29T14:56:24.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:24.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:24.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2031353046' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-29T14:56:24.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:24 smithi040 ceph-mon[39581]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-29T14:56:24.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:24 smithi121 ceph-mon[44533]: osdmap e1075: 12 total, 12 up, 9 in 2024-04-29T14:56:24.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2031353046' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-29T14:56:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:24 smithi121 ceph-mon[44533]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-29T14:56:25.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 14 16 49966 49950 222.974 171.188 0.00663858 0.00448354 2024-04-29T14:56:25.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:25 smithi194 ceph-mon[44089]: pgmap v2744: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 252 MiB/s wr, 4.04k op/s 2024-04-29T14:56:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2031353046' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-29T14:56:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:25 smithi194 ceph-mon[44089]: osdmap e1076: 12 total, 12 up, 8 in 2024-04-29T14:56:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:25 smithi040 ceph-mon[39581]: pgmap v2744: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 252 MiB/s wr, 4.04k op/s 2024-04-29T14:56:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2031353046' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-29T14:56:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:25 smithi040 ceph-mon[39581]: osdmap e1076: 12 total, 12 up, 8 in 2024-04-29T14:56:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:25 smithi121 ceph-mon[44533]: pgmap v2744: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 252 MiB/s wr, 4.04k op/s 2024-04-29T14:56:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2031353046' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-29T14:56:25.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:25 smithi121 ceph-mon[44533]: osdmap e1076: 12 total, 12 up, 8 in 2024-04-29T14:56:26.002 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:26.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:56:26.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:26.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:26.168 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:26.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 15 16 52582 52566 219.008 163.5 0.00805907 0.00456411 2024-04-29T14:56:26.670 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:26.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:27.026 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:27.059 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:27.083 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:27.086 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:27.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:27.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 16 15 54634 54619 213.339 128.312 0.00635018 0.00468629 2024-04-29T14:56:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:27 smithi194 ceph-mon[44089]: pgmap v2746: 17 pgs: 17 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 166 MiB/s wr, 2.66k op/s 2024-04-29T14:56:27.874 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:27.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:27 smithi040 ceph-mon[39581]: pgmap v2746: 17 pgs: 17 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 166 MiB/s wr, 2.66k op/s 2024-04-29T14:56:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:27.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:27.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:27 smithi121 ceph-mon[44533]: pgmap v2746: 17 pgs: 17 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 166 MiB/s wr, 2.66k op/s 2024-04-29T14:56:27.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:28.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 17 16 56847 56831 208.922 138.25 0.0056029 0.00478499 2024-04-29T14:56:29.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1607043140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:56:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:28 smithi194 ceph-mon[44089]: from='client.38168 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:56:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:28 smithi194 ceph-mon[44089]: pgmap v2747: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-29T14:56:29.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1607043140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:56:29.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:28 smithi040 ceph-mon[39581]: from='client.38168 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:56:29.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:28 smithi040 ceph-mon[39581]: pgmap v2747: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-29T14:56:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1607043140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:56:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:28 smithi121 ceph-mon[44533]: from='client.38168 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:56:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:28 smithi121 ceph-mon[44533]: pgmap v2747: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-29T14:56:29.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 18 16 59231 59215 205.592 149 0.00670323 0.00486216 2024-04-29T14:56:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:56:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:29 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:56:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:29 smithi194 ceph-mon[44089]: from='client.38168 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:56:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:29 smithi194 ceph-mon[44089]: osdmap e1077: 12 total, 12 up, 8 in 2024-04-29T14:56:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:29 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:56:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:29 smithi040 ceph-mon[39581]: from='client.38168 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:56:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:29 smithi040 ceph-mon[39581]: osdmap e1077: 12 total, 12 up, 8 in 2024-04-29T14:56:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:30.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:29 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:56:30.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:29 smithi121 ceph-mon[44533]: from='client.38168 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:56:30.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:29 smithi121 ceph-mon[44533]: osdmap e1077: 12 total, 12 up, 8 in 2024-04-29T14:56:30.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:30.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:30.364 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:30.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:30.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:30.506 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:30.554 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 19 16 61476 61460 202.156 140.312 0.0152462 0.00494475 2024-04-29T14:56:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:30.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:30.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:30.748 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:30.882 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:30.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:30.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:31.046 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:30 smithi121 ceph-mon[44533]: osdmap e1078: 12 total, 12 up, 8 in 2024-04-29T14:56:31.046 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:30 smithi121 ceph-mon[44533]: pgmap v2750: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-29T14:56:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:31.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:30 smithi194 ceph-mon[44089]: osdmap e1078: 12 total, 12 up, 8 in 2024-04-29T14:56:31.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:30 smithi194 ceph-mon[44089]: pgmap v2750: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-29T14:56:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:31.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:31.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:30 smithi040 ceph-mon[39581]: osdmap e1078: 12 total, 12 up, 8 in 2024-04-29T14:56:31.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:30 smithi040 ceph-mon[39581]: pgmap v2750: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-29T14:56:31.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:31.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:56:31.551338 min lat: 0.000922662 max lat: 0.217889 avg lat: 0.00502112 2024-04-29T14:56:31.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:56:31.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 20 16 63729 63713 199.088 140.812 0.00329699 0.00502112 2024-04-29T14:56:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:31.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:31.716 INFO:tasks.thrashosds.thrasher:in_osds: [4, 7, 10, 6, 1, 5, 8, 3] out_osds: [11, 2, 9, 0] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 10, 11, 2, 8, 5, 9] 2024-04-29T14:56:31.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-29T14:56:31.744 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:56:31.784 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:56:31.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:31.818 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:56:31.831 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:31.851 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:56:31.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:31.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2419573870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:31.884 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:56:31.917 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:56:31.959 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:56:31.966 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:31.996 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:32.001 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:56:32.031 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:32.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2419573870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:32.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:32.038 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:56:32.070 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:56:32.098 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:56:32.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:32.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2419573870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:32.142 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:56:32.142 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:56:32.142 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:56:32.143 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:56:32.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:56:32.144 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:56:32.185 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:32.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:32.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:32.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:32.352 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:32.376 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:32.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:32.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:32.513 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:32.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 21 16 65775 65759 195.696 127.875 0.00238552 0.0051063 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:32.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:32.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: pgmap v2751: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-04-29T14:56:32.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:32 smithi040 ceph-mon[39581]: osdmap e1079: 12 total, 12 up, 8 in 2024-04-29T14:56:33.068 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:33.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: pgmap v2751: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-04-29T14:56:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:32 smithi194 ceph-mon[44089]: osdmap e1079: 12 total, 12 up, 8 in 2024-04-29T14:56:33.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: pgmap v2751: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-04-29T14:56:33.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:32 smithi121 ceph-mon[44533]: osdmap e1079: 12 total, 12 up, 8 in 2024-04-29T14:56:33.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:33.423 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:33.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:33.529 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:33.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 22 16 67811 67795 192.584 127.25 0.00944643 0.00519091 2024-04-29T14:56:33.563 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:33.589 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:33.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:33.695 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:33.727 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:33.752 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:33.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:33.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:33 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:33 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:33.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:33.914 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:33.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:33.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:34.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:34.093 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:34.132 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:34.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:34.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:33 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:34.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:34.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:34.274 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:34.297 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:34.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:34.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:34.461 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:34.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:34.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:34.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 23 16 69923 69907 189.95 132 0.0102285 0.00526328 2024-04-29T14:56:34.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:34.628 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:34.781 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:34 smithi121 ceph-mon[44533]: pgmap v2753: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-29T14:56:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:34.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:34.903 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:34 smithi040 ceph-mon[39581]: pgmap v2753: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-29T14:56:34.952 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:35.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:35.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:34 smithi194 ceph-mon[44089]: pgmap v2753: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-29T14:56:35.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:56:35.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:35.185 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:35.209 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:35.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:35.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:35.360 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:35.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:35.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:35.447 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:56:35.447 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:56:35.501 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:35.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 24 16 71938 71922 187.282 125.938 0.00422307 0.00533772 2024-04-29T14:56:35.558 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:36.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 25 16 73988 73972 184.915 128.125 0.012379 0.0054064 2024-04-29T14:56:36.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:56:36.931 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:36.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:37.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:37.081 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:37.101 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:37.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:37.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:37.238 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:37.262 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:37.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:37.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:37.420 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:37.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:37.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:37.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 26 16 75969 75953 182.565 123.812 0.00561082 0.00547573 2024-04-29T14:56:37.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:37.620 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:37.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:37.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:37.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:37.814 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:37 smithi121 ceph-mon[44533]: pgmap v2754: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-04-29T14:56:37.816 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:37 smithi194 ceph-mon[44089]: pgmap v2754: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-04-29T14:56:37.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:37.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:37 smithi040 ceph-mon[39581]: pgmap v2754: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-04-29T14:56:37.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:37.987 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:38.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:38.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:38.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:38.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:56:38.125 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:38.152 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:38.176 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:38.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:38.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:38.321 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:38.450 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:38.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:38.499 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:38.511 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:38.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 27 16 77762 77746 179.953 112.062 0.0165311 0.00555521 2024-04-29T14:56:38.556 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:38.584 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:38.605 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:38.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:38.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:38.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:38 smithi040 ceph-mon[39581]: osdmap e1080: 12 total, 12 up, 8 in 2024-04-29T14:56:38.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:38 smithi040 ceph-mon[39581]: pgmap v2756: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-04-29T14:56:38.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:56:38.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:56:38.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:56:38.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:38 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:56:38.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:38 smithi121 ceph-mon[44533]: osdmap e1080: 12 total, 12 up, 8 in 2024-04-29T14:56:38.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:38 smithi121 ceph-mon[44533]: pgmap v2756: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-04-29T14:56:38.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:56:38.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:56:38.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:56:38.729 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:38 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:56:38.739 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:38.762 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:38.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:38.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:38.913 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:38.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:38.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:39.044 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:39.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:38 smithi194 ceph-mon[44089]: osdmap e1080: 12 total, 12 up, 8 in 2024-04-29T14:56:39.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:38 smithi194 ceph-mon[44089]: pgmap v2756: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-04-29T14:56:39.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:56:39.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:56:39.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:56:39.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:38 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:56:39.073 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:39.081 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:39.095 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:39.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:39.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:39.350 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:39.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:39.455 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:56:39.486 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:39.510 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:56:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:39.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:56:39.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 28 16 79588 79572 177.602 114.125 0.00935223 0.00562894 2024-04-29T14:56:39.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:39.651 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:39.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:39.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:39.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:39.837 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:40.045 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:40.047 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:39 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:56:40.048 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:39 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:56:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:40.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:40.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:39 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:56:40.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:39 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:56:40.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:39 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:56:40.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:39 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:56:40.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:40.194 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:40.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:40.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:40.327 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:40.360 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:40.384 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:40.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:40.488 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:40.517 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:40.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:40.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 29 16 81479 81463 175.553 118.188 0.0120669 0.0056947 2024-04-29T14:56:40.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:40.688 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:40.721 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:40.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:40.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:40.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:40.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:40 smithi194 ceph-mon[44089]: pgmap v2757: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-29T14:56:40.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:40.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:40.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:40.868 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:40 smithi040 ceph-mon[39581]: pgmap v2757: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-29T14:56:40.872 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:41.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:41.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:41.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:40 smithi121 ceph-mon[44533]: pgmap v2757: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-29T14:56:41.555 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 30 16 83379 83363 173.659 118.75 0.010744 0.00575682 2024-04-29T14:56:42.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:41 smithi194 ceph-mon[44089]: osdmap e1081: 12 total, 12 up, 8 in 2024-04-29T14:56:42.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:41 smithi040 ceph-mon[39581]: osdmap e1081: 12 total, 12 up, 8 in 2024-04-29T14:56:42.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:41 smithi121 ceph-mon[44533]: osdmap e1081: 12 total, 12 up, 8 in 2024-04-29T14:56:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:42.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:42.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 31 15 85279 85264 171.89 118.812 0.00357638 0.00581604 2024-04-29T14:56:42.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:56:42.607 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:42.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:56:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:42.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:56:42.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:42.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:42 smithi121 ceph-mon[44533]: pgmap v2759: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:56:42.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:42 smithi121 ceph-mon[44533]: osdmap e1082: 12 total, 12 up, 8 in 2024-04-29T14:56:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:42 smithi194 ceph-mon[44089]: pgmap v2759: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:56:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:42 smithi194 ceph-mon[44089]: osdmap e1082: 12 total, 12 up, 8 in 2024-04-29T14:56:43.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:42 smithi040 ceph-mon[39581]: pgmap v2759: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:56:43.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:42 smithi040 ceph-mon[39581]: osdmap e1082: 12 total, 12 up, 8 in 2024-04-29T14:56:43.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 32 16 87063 87047 170 111.438 0.00829877 0.00588093 2024-04-29T14:56:44.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 33 16 88725 88709 167.996 103.875 0.00610148 0.005951 2024-04-29T14:56:44.852 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:44.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:44.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:45.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:45.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:45.181 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:45.458 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:45 smithi040 ceph-mon[39581]: pgmap v2761: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-29T14:56:45.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 34 16 90482 90466 166.285 109.812 0.0112136 0.00601221 2024-04-29T14:56:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:45.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:45.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:45 smithi194 ceph-mon[44089]: pgmap v2761: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-29T14:56:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:45 smithi121 ceph-mon[44533]: pgmap v2761: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-04-29T14:56:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:46.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:46.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 35 16 92257 92241 164.703 110.938 0.0134093 0.0060698 2024-04-29T14:56:46.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:46.627 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:46.896 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:46.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:47.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:47.035 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:47.426 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:56:47.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:47 smithi040 ceph-mon[39581]: pgmap v2762: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-04-29T14:56:47.460 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1274993741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:47.461 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:56:47.480 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:56:47.480 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:56:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:47.496 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:56:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:47.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:47.525 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:56:47.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 36 16 93967 93951 163.096 106.875 0.014572 0.00612996 2024-04-29T14:56:47.560 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:56:47.597 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:56:47.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:47.626 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:56:47.629 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:47.652 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:56:47.678 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:56:47.684 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:47.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:47.708 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:56:47.740 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:56:47.784 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:56:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:56:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:56:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:56:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:56:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:56:47.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:47.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:47 smithi194 ceph-mon[44089]: pgmap v2762: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-04-29T14:56:47.817 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1274993741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:47.820 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:47.841 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:47.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:47.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:47 smithi121 ceph-mon[44533]: pgmap v2762: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-04-29T14:56:47.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1274993741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:56:47.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:47.988 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:48.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:48.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:56:48.172 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:48.214 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:56:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:48.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:56:48.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:48.368 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:48.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:48.502 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:48.531 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:48.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:48.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 37 16 95498 95482 161.274 95.6875 0.00882206 0.00619916 2024-04-29T14:56:48.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:48.700 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:48.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:48.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osdmap e1083: 12 total, 12 up, 8 in 2024-04-29T14:56:48.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:48.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:48 smithi194 ceph-mon[44089]: pgmap v2764: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-29T14:56:48.848 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:48.874 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:48.898 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:48.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:49.005 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:49.034 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:49.055 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:49.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osdmap e1083: 12 total, 12 up, 8 in 2024-04-29T14:56:49.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:48 smithi040 ceph-mon[39581]: pgmap v2764: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-29T14:56:49.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osdmap e1083: 12 total, 12 up, 8 in 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:56:49.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:48 smithi121 ceph-mon[44533]: pgmap v2764: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-29T14:56:49.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:49.203 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:49.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 38 16 97105 97089 159.673 100.438 0.00530742 0.00626128 2024-04-29T14:56:49.780 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:56:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:49 smithi194 ceph-mon[44089]: osdmap e1084: 12 total, 12 up, 8 in 2024-04-29T14:56:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/592740898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:56:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/341349178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:56:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:49 smithi194 ceph-mon[44089]: from='client.38180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:56:50.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:50.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:50.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:49 smithi040 ceph-mon[39581]: osdmap e1084: 12 total, 12 up, 8 in 2024-04-29T14:56:50.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/592740898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:56:50.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/341349178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:56:50.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:49 smithi040 ceph-mon[39581]: from='client.38180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:56:50.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:50.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:50.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:49 smithi121 ceph-mon[44533]: osdmap e1084: 12 total, 12 up, 8 in 2024-04-29T14:56:50.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/592740898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:56:50.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/341349178' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:56:50.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:49 smithi121 ceph-mon[44533]: from='client.38180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T14:56:50.415 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.4 to 1 (10000) 2024-04-29T14:56:50.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 39 16 98724 98708 158.173 101.188 0.0113688 0.00632078 2024-04-29T14:56:50.761 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:50.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:50.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:56:50.867 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:50.898 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:50.922 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:50.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:51.028 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/592740898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: from='client.38180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: osdmap e1085: 12 total, 12 up, 8 in 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:56:51.056 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:50 smithi194 ceph-mon[44089]: pgmap v2767: 17 pgs: 17 active+clean; 5.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-29T14:56:51.058 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:51.081 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:51.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:51.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:56:51.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:56:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/592740898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:56:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: from='client.38180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:56:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: osdmap e1085: 12 total, 12 up, 8 in 2024-04-29T14:56:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:56:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:56:51.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:50 smithi040 ceph-mon[39581]: pgmap v2767: 17 pgs: 17 active+clean; 5.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-29T14:56:51.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:56:51.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:56:51.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/592740898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:56:51.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: from='client.38180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T14:56:51.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: osdmap e1085: 12 total, 12 up, 8 in 2024-04-29T14:56:51.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:56:51.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:56:51.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:50 smithi121 ceph-mon[44533]: pgmap v2767: 17 pgs: 17 active+clean; 5.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-29T14:56:51.185 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:51.215 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:56:51.236 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:51.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:51.340 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:51.368 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:51.392 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:51.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:51.498 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:51.528 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:51.548 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:51.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:51.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:56:51.552948 min lat: 0.000922662 max lat: 0.217889 avg lat: 0.00638366 2024-04-29T14:56:51.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:56:51.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 40 16 100251 100235 156.605 95.4375 0.0114519 0.00638366 2024-04-29T14:56:51.654 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:51.684 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:51.708 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:51.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:51.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-29T14:56:51.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:51 smithi121 ceph-mon[44533]: osdmap e1086: 12 total, 12 up, 8 in 2024-04-29T14:56:51.888 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:56:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:51.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:56:51.994 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:56:52.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:51 smithi040 ceph-mon[39581]: osdmap e1086: 12 total, 12 up, 8 in 2024-04-29T14:56:52.021 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:51 smithi194 ceph-mon[44089]: osdmap e1086: 12 total, 12 up, 8 in 2024-04-29T14:56:52.024 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:56:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:52.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:56:52.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:52.200 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:52.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:52.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:52.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:52.461 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:52.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 41 16 101758 101742 155.082 94.1875 0.00884537 0.00644696 2024-04-29T14:56:52.805 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:52 smithi121 ceph-mon[44533]: pgmap v2769: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:56:52.805 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:52 smithi121 ceph-mon[44533]: osdmap e1087: 12 total, 12 up, 8 in 2024-04-29T14:56:52.916 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:52.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:53.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:53.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:52 smithi194 ceph-mon[44089]: pgmap v2769: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:56:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:52 smithi194 ceph-mon[44089]: osdmap e1087: 12 total, 12 up, 8 in 2024-04-29T14:56:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:52 smithi040 ceph-mon[39581]: pgmap v2769: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-04-29T14:56:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:52 smithi040 ceph-mon[39581]: osdmap e1087: 12 total, 12 up, 8 in 2024-04-29T14:56:53.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:53.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:53.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:53.379 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:53.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:53.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:53.534 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:53.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 42 16 103231 103215 153.582 92.0625 0.00591794 0.00650899 2024-04-29T14:56:53.562 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:53.584 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:53.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:53.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:53.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:53.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:53.854 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:56:53.854 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:56:53.855 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T14:56:53.855 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:56:53.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:56:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:54.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:56:54.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 43 16 104746 104730 152.212 94.6875 0.0108461 0.00656822 2024-04-29T14:56:54.579 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:56:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:54.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:56:54.587 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.587 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.587 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:54.587 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:54.587 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T14:56:54.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:56:54.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:54.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:54.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.728 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:54.808 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:56:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:54.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:56:54.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:56:54.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:56:54.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:56:54.914 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:54.940 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:54.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:56:54.962 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:54.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:54.975 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:56:55.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:55.113 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:55.161 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:55.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:55.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:55.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:56:55.393 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:56:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:55.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:56:55.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 44 16 105913 105897 150.41 72.9375 0.00692938 0.00664726 2024-04-29T14:56:55.609 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:56:55.639 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:55.659 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:56:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:55.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:56:55.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:55 smithi040 ceph-mon[39581]: pgmap v2771: 17 pgs: 17 active+clean; 6.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-29T14:56:55.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:56:55.807 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:55 smithi121 ceph-mon[44533]: pgmap v2771: 17 pgs: 17 active+clean; 6.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-29T14:56:55.811 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:55.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:55 smithi194 ceph-mon[44089]: pgmap v2771: 17 pgs: 17 active+clean; 6.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-29T14:56:55.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:56:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:55.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:56:55.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:56:56.012 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:56.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:56:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:56.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:56:56.457 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.7 to 1 (10000) 2024-04-29T14:56:56.481 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:56 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3422149596' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:56:56.484 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:56.512 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:56.536 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:56.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:56.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 45 16 107391 107375 149.12 92.375 0.00754326 0.00670442 2024-04-29T14:56:56.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:56 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3422149596' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:56:56.640 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:56.674 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:56.699 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:56.702 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:56 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3422149596' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T14:56:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:56.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:56.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:56.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:56.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:56.924 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:56:57.011 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:56:57.039 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:56:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:57.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:56:57.165 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:56:57.181 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:56:57.181 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:56:57.181 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:56:57.182 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:56:57.182 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:56:57.194 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:57.216 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:56:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:57.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:56:57.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:56:57.350 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:57.372 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:56:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:57.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:56:57.462 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:57 smithi121 ceph-mon[44533]: pgmap v2772: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:56:57.462 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3422149596' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:56:57.462 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:57 smithi121 ceph-mon[44533]: osdmap e1088: 12 total, 12 up, 8 in 2024-04-29T14:56:57.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:56:57.529 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:57.556 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 46 16 108790 108774 147.779 87.4375 0.00379191 0.00676552 2024-04-29T14:56:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:57 smithi194 ceph-mon[44089]: pgmap v2772: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:56:57.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3422149596' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:56:57.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:57 smithi194 ceph-mon[44089]: osdmap e1088: 12 total, 12 up, 8 in 2024-04-29T14:56:57.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:57 smithi040 ceph-mon[39581]: pgmap v2772: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:56:57.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3422149596' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T14:56:57.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:57 smithi040 ceph-mon[39581]: osdmap e1088: 12 total, 12 up, 8 in 2024-04-29T14:56:57.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T14:56:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:56:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:56:58.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:56:58.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:56:58.146 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:56:58.465 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:56:58.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 47 16 110192 110176 146.499 87.625 0.00450888 0.00682411 2024-04-29T14:56:59.063 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:58 smithi121 ceph-mon[44533]: osdmap e1089: 12 total, 12 up, 8 in 2024-04-29T14:56:59.064 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:56:58 smithi121 ceph-mon[44533]: pgmap v2775: 17 pgs: 17 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:56:59.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:58 smithi194 ceph-mon[44089]: osdmap e1089: 12 total, 12 up, 8 in 2024-04-29T14:56:59.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:56:58 smithi194 ceph-mon[44089]: pgmap v2775: 17 pgs: 17 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:56:59.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:58 smithi040 ceph-mon[39581]: osdmap e1089: 12 total, 12 up, 8 in 2024-04-29T14:56:59.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:56:58 smithi040 ceph-mon[39581]: pgmap v2775: 17 pgs: 17 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-29T14:56:59.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 48 15 111594 111579 145.274 87.6875 0.00995111 0.00688188 2024-04-29T14:57:00.001 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:00.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:00.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-29T14:57:00.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:00.161 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:00.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 49 16 113062 113046 144.18 91.6875 0.00632238 0.00693429 2024-04-29T14:57:01.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 50 16 114421 114405 142.995 84.9375 0.00605443 0.00699149 2024-04-29T14:57:01.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:01 smithi194 ceph-mon[44089]: pgmap v2776: 17 pgs: 17 active+clean; 6.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:57:01.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:01 smithi194 ceph-mon[44089]: 9.0 scrub starts 2024-04-29T14:57:01.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:01 smithi040 ceph-mon[39581]: pgmap v2776: 17 pgs: 17 active+clean; 6.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:57:01.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:01 smithi040 ceph-mon[39581]: 9.0 scrub starts 2024-04-29T14:57:01.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:01 smithi121 ceph-mon[44533]: pgmap v2776: 17 pgs: 17 active+clean; 6.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-29T14:57:01.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:01 smithi121 ceph-mon[44533]: 9.0 scrub starts 2024-04-29T14:57:02.417 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:02.479 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:57:02.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 51 16 115749 115733 141.819 83 0.00509156 0.00704937 2024-04-29T14:57:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:57:02.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:57:02.646 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:57:02.673 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:57:02.712 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:57:02.741 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:57:02.818 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:57:02.819 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:57:02.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:02 smithi194 ceph-mon[44089]: 9.0 scrub ok 2024-04-29T14:57:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1175708176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:02.841 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:57:02.868 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:57:02.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:02 smithi040 ceph-mon[39581]: 9.0 scrub ok 2024-04-29T14:57:02.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1175708176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:02 smithi121 ceph-mon[44533]: 9.0 scrub ok 2024-04-29T14:57:02.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1175708176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:02.897 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:57:02.927 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:57:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:57:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:57:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:03.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:03.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:03.142 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:03.162 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:03.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:03.266 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:03.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: pgmap v2777: 17 pgs: 17 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-29T14:57:03.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:03.469 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1175708176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osdmap e1090: 12 total, 12 up, 8 in 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3927743143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osdmap e1091: 12 total, 12 up, 8 in 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.470 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.471 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:03 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 52 16 117070 117054 140.679 82.5625 0.00934035 0.00710706 2024-04-29T14:57:03.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: pgmap v2777: 17 pgs: 17 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1175708176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osdmap e1090: 12 total, 12 up, 8 in 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3927743143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osdmap e1091: 12 total, 12 up, 8 in 2024-04-29T14:57:03.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:03 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: pgmap v2777: 17 pgs: 17 active+clean; 6.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-29T14:57:03.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:03.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1175708176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:57:03.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osdmap e1090: 12 total, 12 up, 8 in 2024-04-29T14:57:03.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3927743143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osdmap e1091: 12 total, 12 up, 8 in 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:03 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:03.995 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:57:03.995 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:57:04.141 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:04.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:04.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:04.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 53 16 118451 118435 139.653 86.3125 0.00659129 0.00715913 2024-04-29T14:57:04.741 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:04.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T14:57:04.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:04.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:57:05.277 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:05.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:05.388 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:05.418 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:05.439 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:05.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:05.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:05.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 54 16 119779 119763 138.604 83 0.0241093 0.00721296 2024-04-29T14:57:05.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:05 smithi040 ceph-mon[39581]: pgmap v2780: 17 pgs: 17 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-04-29T14:57:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:05 smithi194 ceph-mon[44089]: pgmap v2780: 17 pgs: 17 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-04-29T14:57:05.836 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:05.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:05.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:05 smithi121 ceph-mon[44533]: pgmap v2780: 17 pgs: 17 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-04-29T14:57:05.942 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:05.971 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:05.991 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:05.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:06.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:06.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:06.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:06.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:06.328 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:06.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 55 16 121133 121117 137.622 84.625 0.00442307 0.00726435 2024-04-29T14:57:07.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:57:07.557 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 56 16 122243 122227 136.403 69.375 0.006657 0.00732991 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: pgmap v2781: 17 pgs: 17 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3098345881,config=1981513354,health=3340005642,kv=3809138020}) 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=611521278,logm=2592517319}) 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3723134853}) 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2632278819}) 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2997336564}) 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295105320}) 2024-04-29T14:57:07.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546633381}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3925432997}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3065956786}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1536575879}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128220276}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685737108}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910170449}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2694925590}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3611685705}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663369985}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4031903015}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982699860}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230695012}) 2024-04-29T14:57:07.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2681831352}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361587477}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704013713}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637883195}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613636482}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931586934}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4084247711}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2650697932}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591478670}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=865288973}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=662340851}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559158536}) 2024-04-29T14:57:07.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=624894090}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186743360}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183511763}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2946456152}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1154512875}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750073050}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228928327}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1351263771}) 2024-04-29T14:57:07.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792168450}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062572667}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098150057}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1624362021}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412916003}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150394051}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452199914}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3621224571}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3086658955}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414649977}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824332515}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3678380845}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2990330608}) 2024-04-29T14:57:07.628 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110448943}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=748617627}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940997881}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175551702}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1105525173}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1652860732}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167089155}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3221901342}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=872235178}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2240587516}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1744131865}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730802811}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294730576}) 2024-04-29T14:57:07.629 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891850877}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351216322}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=740110496}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940489493}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3075490923}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128467862}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581398742}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=348577547}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2430600544}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780518374}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466198082}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413974435}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3570577766}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1012038546}) 2024-04-29T14:57:07.630 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682663207}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661216267}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316864215}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1598049726}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520206217}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041390581}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=766350658}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3866135332}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390277884}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973586042}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3475875915}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2184374186}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2891871717}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1649394639}) 2024-04-29T14:57:07.631 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083722219}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101055220}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242002031}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103733945}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887501192}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191326624}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921937127}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3582370111}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=830136401}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705949120}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317922874}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477127474}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597993882}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3846367916}) 2024-04-29T14:57:07.632 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753805313}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734663818}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785980462}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1176167622}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530787253}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2368086517}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3597729875}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858676009}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2737787084}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130359446}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383297868}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242792265}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888528328}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=954781107,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3782610014}) 2024-04-29T14:57:07.633 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3197155243}) 2024-04-29T14:57:07.634 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3153805377}) 2024-04-29T14:57:07.634 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=78,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=4} crc {mon_config_key=3388292988,monmap=455069592,osd_metadata=1520689391,osd_pg_creating=1682327481,osdmap=653896000}) 2024-04-29T14:57:07.634 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1742620127}) 2024-04-29T14:57:07.634 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=2442852602}) 2024-04-29T14:57:07.734 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: pgmap v2781: 17 pgs: 17 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3098345881,config=1981513354,health=3340005642,kv=3809138020}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=611521278,logm=2592517319}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3723134853}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2632278819}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2997336564}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295105320}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546633381}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3925432997}) 2024-04-29T14:57:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3065956786}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1536575879}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128220276}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685737108}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910170449}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2694925590}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3611685705}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663369985}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4031903015}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982699860}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230695012}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2681831352}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361587477}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704013713}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637883195}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613636482}) 2024-04-29T14:57:07.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931586934}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4084247711}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2650697932}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591478670}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=865288973}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=662340851}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559158536}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=624894090}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186743360}) 2024-04-29T14:57:07.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183511763}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2946456152}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1154512875}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750073050}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228928327}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1351263771}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792168450}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062572667}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098150057}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1624362021}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412916003}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150394051}) 2024-04-29T14:57:07.828 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452199914}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3621224571}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3086658955}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414649977}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824332515}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3678380845}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2990330608}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110448943}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=748617627}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940997881}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175551702}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1105525173}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1652860732}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167089155}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3221901342}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=872235178}) 2024-04-29T14:57:07.829 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2240587516}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1744131865}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730802811}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294730576}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891850877}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351216322}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=740110496}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940489493}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3075490923}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128467862}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581398742}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=348577547}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2430600544}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780518374}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466198082}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413974435}) 2024-04-29T14:57:07.830 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3570577766}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1012038546}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682663207}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661216267}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316864215}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1598049726}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520206217}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041390581}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=766350658}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3866135332}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390277884}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973586042}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3475875915}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2184374186}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2891871717}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1649394639}) 2024-04-29T14:57:07.831 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083722219}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101055220}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242002031}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103733945}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887501192}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191326624}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921937127}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3582370111}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=830136401}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705949120}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317922874}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477127474}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597993882}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3846367916}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753805313}) 2024-04-29T14:57:07.832 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734663818}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785980462}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1176167622}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530787253}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2368086517}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3597729875}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858676009}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2737787084}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130359446}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383297868}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242792265}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888528328}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=954781107,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3782610014}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3197155243}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3153805377}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=78,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=4} crc {mon_config_key=3388292988,monmap=455069592,osd_metadata=1520689391,osd_pg_creating=1682327481,osdmap=653896000}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1742620127}) 2024-04-29T14:57:07.833 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=2442852602}) 2024-04-29T14:57:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: pgmap v2781: 17 pgs: 17 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-29T14:57:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3098345881,config=1981513354,health=3340005642,kv=3809138020}) 2024-04-29T14:57:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=611521278,logm=2592517319}) 2024-04-29T14:57:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3723134853}) 2024-04-29T14:57:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2632278819}) 2024-04-29T14:57:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2997336564}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295105320}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546633381}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3925432997}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3065956786}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1536575879}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128220276}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685737108}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910170449}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2694925590}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3611685705}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1663369985}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4031903015}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982699860}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2230695012}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2681831352}) 2024-04-29T14:57:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361587477}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704013713}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637883195}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613636482}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931586934}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4084247711}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2650697932}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591478670}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=865288973}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=662340851}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559158536}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=624894090}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186743360}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183511763}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2946456152}) 2024-04-29T14:57:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1154512875}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750073050}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228928327}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1351263771}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792168450}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062572667}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098150057}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1624362021}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412916003}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150394051}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452199914}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3621224571}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3086658955}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414649977}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824332515}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3678380845}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2990330608}) 2024-04-29T14:57:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4110448943}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=748617627}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940997881}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175551702}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1105525173}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1652860732}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167089155}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3221901342}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=872235178}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2240587516}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1744131865}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730802811}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294730576}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891850877}) 2024-04-29T14:57:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351216322}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=740110496}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2940489493}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3075490923}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128467862}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581398742}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=348577547}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2430600544}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780518374}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466198082}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413974435}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3570577766}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1012038546}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682663207}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661216267}) 2024-04-29T14:57:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316864215}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1598049726}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520206217}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2041390581}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=766350658}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3866135332}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390277884}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973586042}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3475875915}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2184374186}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2891871717}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1649394639}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083722219}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101055220}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242002031}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103733945}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887501192}) 2024-04-29T14:57:07.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191326624}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921937127}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3582370111}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=830136401}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705949120}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317922874}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477127474}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597993882}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3846367916}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753805313}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734663818}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785980462}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1176167622}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=530787253}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2368086517}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3597729875}) 2024-04-29T14:57:07.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858676009}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2737787084}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130359446}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383297868}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242792265}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888528328}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=954781107,mdsmap=2840952586,mgr=489469112,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=3782610014}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3197155243}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3153805377}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=78,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=4} crc {mon_config_key=3388292988,monmap=455069592,osd_metadata=1520689391,osd_pg_creating=1682327481,osdmap=653896000}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1742620127}) 2024-04-29T14:57:07.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=2442852602}) 2024-04-29T14:57:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:08.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:08.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:08.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:08.281 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:08.418 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:08.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:57:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:08.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:08.511 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.10 to 1 (10000) 2024-04-29T14:57:08.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:08.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 57 16 123552 123536 135.445 81.8125 0.00675026 0.00738182 2024-04-29T14:57:08.561 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:08 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4031078141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:57:08.568 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:08.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:08.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:08.763 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:08.790 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:08 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4031078141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:57:08.792 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:08.817 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:08.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:08.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:08 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4031078141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T14:57:08.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:08.962 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:09.001 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:57:09.001 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:57:09.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 58 16 124842 124826 134.5 80.625 0.0115817 0.00743313 2024-04-29T14:57:09.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:09 smithi121 ceph-mon[44533]: pgmap v2782: 17 pgs: 17 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:57:09.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4031078141' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:57:09.718 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:09 smithi121 ceph-mon[44533]: osdmap e1092: 12 total, 12 up, 8 in 2024-04-29T14:57:09.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T14:57:09.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:09 smithi194 ceph-mon[44089]: pgmap v2782: 17 pgs: 17 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:57:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4031078141' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:57:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:09 smithi194 ceph-mon[44089]: osdmap e1092: 12 total, 12 up, 8 in 2024-04-29T14:57:09.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:09 smithi040 ceph-mon[39581]: pgmap v2782: 17 pgs: 17 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-29T14:57:09.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4031078141' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T14:57:09.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:09 smithi040 ceph-mon[39581]: osdmap e1092: 12 total, 12 up, 8 in 2024-04-29T14:57:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:57:10.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 59 16 126084 126068 133.536 77.625 0.00521467 0.00748666 2024-04-29T14:57:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:10.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:10.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:10.898 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:10.921 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:10.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:11.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:11.374 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-29T14:57:11.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:57:11.554534 min lat: 0.000922662 max lat: 0.246703 avg lat: 0.00753705 2024-04-29T14:57:11.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:57:11.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 60 16 127378 127362 132.658 80.875 0.012596 0.00753705 2024-04-29T14:57:11.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:11 smithi040 ceph-mon[39581]: pgmap v2784: 17 pgs: 17 active+clean; 7.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:57:11.609 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:11.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:11.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:11.724 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:11 smithi121 ceph-mon[44533]: pgmap v2784: 17 pgs: 17 active+clean; 7.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:57:11.752 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:11.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:11 smithi194 ceph-mon[44089]: pgmap v2784: 17 pgs: 17 active+clean; 7.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-29T14:57:11.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:11.928 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:12.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:12.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:12.222 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:12.291 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:12.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:12.501 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:12.531 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:12.556 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:12.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 61 16 128616 128600 131.752 77.375 0.00476579 0.00758867 2024-04-29T14:57:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:12.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:12.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:12.690 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:12.711 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:12.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:12.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:12.849 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:12.951 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:12.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:13.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:57:13.089 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:57:13.130 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:13.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:13.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:13.265 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:13.294 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:13.317 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:13.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:13.421 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:13.449 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:13.470 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:13.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:13.543 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:57:13.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 62 15 129826 129811 130.847 75.6875 0.00722227 0.00764082 2024-04-29T14:57:13.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:13 smithi040 ceph-mon[39581]: pgmap v2785: 17 pgs: 17 active+clean; 7.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-29T14:57:13.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3685851608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:57:13.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:13 smithi194 ceph-mon[44089]: pgmap v2785: 17 pgs: 17 active+clean; 7.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-29T14:57:13.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3685851608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:57:13.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:13.612 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:13 smithi121 ceph-mon[44533]: pgmap v2785: 17 pgs: 17 active+clean; 7.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-29T14:57:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3685851608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:57:14.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 63 16 131127 131111 130.06 81.25 0.0215631 0.00768765 2024-04-29T14:57:14.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:14 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:14.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3685851608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:57:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:14 smithi194 ceph-mon[44089]: osdmap e1093: 12 total, 12 up, 8 in 2024-04-29T14:57:14.863 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:14 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:14.863 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3685851608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:57:14.863 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:14 smithi121 ceph-mon[44533]: osdmap e1093: 12 total, 12 up, 8 in 2024-04-29T14:57:14.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:14 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:14.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3685851608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:57:14.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:14 smithi040 ceph-mon[39581]: osdmap e1093: 12 total, 12 up, 8 in 2024-04-29T14:57:14.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T14:57:14.982 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:14.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:15.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:15.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:15.340 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:57:15.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:15.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:15 smithi121 ceph-mon[44533]: pgmap v2787: 17 pgs: 17 active+clean; 7.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-04-29T14:57:15.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 64 16 132285 132269 129.159 72.375 0.013663 0.00774083 2024-04-29T14:57:15.610 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:15.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:15.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:15 smithi040 ceph-mon[39581]: pgmap v2787: 17 pgs: 17 active+clean; 7.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-04-29T14:57:15.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:15.775 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:15 smithi194 ceph-mon[44089]: pgmap v2787: 17 pgs: 17 active+clean; 7.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-04-29T14:57:16.250 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:16.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:16.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:16.390 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:16.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:16.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:16.533 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:16.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 65 16 133348 133332 128.194 66.4375 0.0196781 0.00779904 2024-04-29T14:57:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:16.747 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:16.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:16.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:16.882 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:17.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:17.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:17.258 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:17.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:17 smithi121 ceph-mon[44533]: pgmap v2788: 17 pgs: 17 active+clean; 7.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.38k op/s 2024-04-29T14:57:17.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 66 16 134614 134598 127.45 79.125 0.0129321 0.00784444 2024-04-29T14:57:17.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:17 smithi040 ceph-mon[39581]: pgmap v2788: 17 pgs: 17 active+clean; 7.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.38k op/s 2024-04-29T14:57:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:17.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:17 smithi194 ceph-mon[44089]: pgmap v2788: 17 pgs: 17 active+clean; 7.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.38k op/s 2024-04-29T14:57:17.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:17.895 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:18.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:18.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:18.114 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:57:18.155 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:57:18.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:57:18.187 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:57:18.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:18.223 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:57:18.269 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:57:18.274 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:18.307 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:57:18.344 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:18.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:18.354 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:57:18.401 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:57:18.436 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:57:18.451 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:18.481 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:18.482 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:57:18.501 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:18.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:18.518 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:57:18.557 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:57:18.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 67 16 135760 135744 126.617 71.625 0.0142958 0.00789682 2024-04-29T14:57:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:57:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:57:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:57:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:57:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:57:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:57:18.567 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:57:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:57:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:57:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:57:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:57:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:57:18.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:18.669 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:18.683 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:57:18.683 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:230993 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.85501011000002, 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076782375, 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817202+0000", 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817203+0000", 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.203e-06 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817208+0000", 2024-04-29T14:57:18.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999996e-06 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817218+0000", 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5929999999999995e-06 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.817224+0000", 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7530000000000004e-06 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893964+0000", 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076739872000000001 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893972+0000", 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0450000000000001e-06 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:51.893985+0000", 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2196e-05 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231043 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.66741073100002, 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.084551465000000006, 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004802+0000", 2024-04-29T14:57:18.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004804+0000", 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2709999999999999e-06 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004816+0000", 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1864e-05 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004832+0000", 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6670000000000001e-05 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.004845+0000", 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2401e-05 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089323+0000", 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.084478445999999999 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089334+0000", 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1250000000000001e-05 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089353+0000", 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8563000000000002e-05 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231044 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.663094731, 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080603953000000006, 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009118+0000", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009119+0000", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.866e-06 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009125+0000", 2024-04-29T14:57:18.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.015e-06 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009136+0000", 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0185e-05 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.009143+0000", 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1029999999999996e-06 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089391+0000", 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080247936000000006 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089398+0000", 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089721+0000", 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032323900000000001 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231045 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:57:18.694 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.658667165, 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076311253999999995, 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013545+0000", 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013547+0000", 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.624e-06 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013553+0000", 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9710000000000001e-06 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013562+0000", 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7529999999999996e-06 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.013570+0000", 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2590000000000003e-06 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089749+0000", 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076179350000000007 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089757+0000", 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8290000000000008e-06 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.089856+0000", 2024-04-29T14:57:18.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9468000000000004e-05 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231093 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.47815111199998, 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082382412000000002, 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:57:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194061+0000", 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194063+0000", 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4720000000000001e-06 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194068+0000", 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.558e-06 2024-04-29T14:57:18.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194078+0000", 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9629999999999993e-06 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.194085+0000", 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3579999999999999e-06 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276425+0000", 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082340872999999995 2024-04-29T14:57:18.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276432+0000", 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5309999999999998e-06 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276444+0000", 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1657e-05 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231094 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.47474502900002, 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079185309999999995, 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197467+0000", 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197469+0000", 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5579999999999999e-06 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197475+0000", 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.241e-06 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197487+0000", 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1905e-05 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.197495+0000", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5099999999999998e-06 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276469+0000", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078973186000000001 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276474+0000", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7660000000000004e-06 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276653+0000", 2024-04-29T14:57:18.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017814400000000001 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231095 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.47184162600001, 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076347655, 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200371+0000", 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200372+0000", 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.606e-06 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200378+0000", 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0020000000000003e-06 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200390+0000", 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1772e-05 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.200399+0000", 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7050000000000005e-06 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276664+0000", 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076265239999999998 2024-04-29T14:57:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276668+0000", 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9620000000000004e-06 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.276718+0000", 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0368000000000003e-05 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.708 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231143 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.29126970800002, 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076904044000000005, 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380943+0000", 2024-04-29T14:57:18.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380944+0000", 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2550000000000001e-06 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380949+0000", 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8910000000000003e-06 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380958+0000", 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5729999999999997e-06 2024-04-29T14:57:18.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.380966+0000", 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6469999999999998e-06 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457786+0000", 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076820090999999993 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457792+0000", 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0270000000000001e-06 2024-04-29T14:57:18.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.457847+0000", 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4559999999999999e-05 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:231168 7.c e943/939 ondisk, result = 0)", 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 506.19850365399998, 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078192824999999994, 2024-04-29T14:57:18.712 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473709+0000", 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473711+0000", 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4590000000000001e-06 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473719+0000", 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0930000000000003e-06 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473730+0000", 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0621e-05 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.473738+0000", 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5310000000000001e-06 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551873+0000", 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078134561000000005 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551883+0000", 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0236e-05 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:52.551901+0000", 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8323999999999999e-05 2024-04-29T14:57:18.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232295 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 502.630217738, 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076427287999999996, 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:57:18.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041995+0000", 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.041996+0000", 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8369999999999999e-06 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042004+0000", 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6930000000000007e-06 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042015+0000", 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1385e-05 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042024+0000", 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1200000000000002e-06 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118396+0000", 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076372851000000005 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118406+0000", 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8090000000000004e-06 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118422+0000", 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5593000000000001e-05 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232296 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 502.63006406, 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076313084000000003, 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042148+0000", 2024-04-29T14:57:18.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042149+0000", 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5799999999999999e-07 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042152+0000", 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.078e-06 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042159+0000", 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3309999999999998e-06 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.042164+0000", 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0719999999999999e-06 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118442+0000", 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076277242999999995 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118448+0000", 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5119999999999997e-06 2024-04-29T14:57:18.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.118461+0000", 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3190000000000001e-05 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178706 7.c 7:34fb7533:::benchmark_data_smithi194_147200_object63794:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 502.35068003800001, 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093309815000000004, 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:18.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178706 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321532+0000", 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321534+0000", 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0490000000000002e-06 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321540+0000", 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9490000000000001e-06 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321552+0000", 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1925e-05 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.321574+0000", 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2342999999999999e-05 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.332747+0000", 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011172127 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.332889+0000", 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014268700000000001 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.333033+0000", 2024-04-29T14:57:18.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000144041 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.336903+0000", 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038697530000000001 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414719+0000", 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077815881000000003 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414779+0000", 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0061e-05 2024-04-29T14:57:18.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414842+0000", 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2998999999999998e-05 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178709 7.c 7:30b10f7c:::benchmark_data_smithi194_147200_object63797:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 502.34256732300003, 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085420788999999997, 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:18.728 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178709 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329645+0000", 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329648+0000", 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6699999999999998e-06 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329655+0000", 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0949999999999998e-06 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329672+0000", 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7569000000000001e-05 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.329682+0000", 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8660000000000001e-06 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.336948+0000", 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007265886 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.337066+0000", 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011782600000000001 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.337188+0000", 2024-04-29T14:57:18.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012226199999999999 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414891+0000", 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077703373000000006 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.414960+0000", 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8461999999999996e-05 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.415006+0000", 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6026999999999998e-05 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.415066+0000", 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9753000000000002e-05 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178745 7.c 7:37c53f67:::benchmark_data_smithi194_147200_object63833:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 502.24640241100002, 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078737684000000002, 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:18.733 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178745 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425810+0000", 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425811+0000", 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4419999999999999e-06 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425816+0000", 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5249999999999999e-06 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425827+0000", 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0712e-05 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.425833+0000", 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2750000000000001e-06 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432006+0000", 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061731959999999997 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432085+0000", 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8881000000000006e-05 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.432164+0000", 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9328999999999999e-05 2024-04-29T14:57:18.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504359+0000", 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.072195230999999999 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504427+0000", 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7816000000000003e-05 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504482+0000", 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4471000000000001e-05 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.504548+0000", 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5806e-05 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178796 7.c 7:31d4dc8d:::benchmark_data_smithi194_147200_object63884:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 502.05645893000002, 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.081790044000000006, 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:18.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178796 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615753+0000", 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615755+0000", 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5829999999999999e-06 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615760+0000", 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000001e-06 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615774+0000", 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3879999999999999e-05 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.615780+0000", 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.105e-06 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619804+0000", 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004023357 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619881+0000", 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7272000000000001e-05 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.619956+0000", 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5086000000000006e-05 2024-04-29T14:57:18.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.623354+0000", 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003398252 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697462+0000", 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.074107585000000004 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697500+0000", 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8003999999999997e-05 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:56.697543+0000", 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3529999999999998e-05 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.6.0:232772 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 501.11070758400001, 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077453872000000007, 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561505+0000", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561506+0000", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.782e-06 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561515+0000", 2024-04-29T14:57:18.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1289999999999996e-06 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561529+0000", 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4644000000000001e-05 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.561539+0000", 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5380000000000008e-06 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638939+0000", 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077400374999999993 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638947+0000", 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8569999999999995e-06 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.638959+0000", 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1547e-05 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178971 7.c 7:3606b2fb:::benchmark_data_smithi194_147200_object64059:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:57:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 500.844662234, 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090073625000000004, 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178971 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827550+0000", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827552+0000", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5999999999999999e-06 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827556+0000", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7849999999999999e-06 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827565+0000", 2024-04-29T14:57:18.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7970000000000006e-06 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.827571+0000", 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9120000000000003e-06 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.833976+0000", 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064046939999999998 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.834060+0000", 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4618999999999997e-05 2024-04-29T14:57:18.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.834144+0000", 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3987999999999996e-05 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917478+0000", 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.083333661000000003 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917526+0000", 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7812000000000002e-05 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917563+0000", 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6936e-05 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917624+0000", 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0820999999999999e-05 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26998.0:178972 7.c 7:34cd9e3a:::benchmark_data_smithi194_147200_object64060:head [delete] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 500.84386451, 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089382420000000004, 2024-04-29T14:57:18.751 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26998", 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1247748610", 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 178972 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828348+0000", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828349+0000", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.257e-06 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828353+0000", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3440000000000003e-06 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828363+0000", 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0026e-05 2024-04-29T14:57:18.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.828370+0000", 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9700000000000002e-06 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837577+0000", 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092070030000000001 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837702+0000", 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012509000000000001 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837849+0000", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014664800000000001 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917659+0000", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079810324000000002 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917667+0000", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7069999999999992e-06 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917696+0000", 2024-04-29T14:57:18.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8530999999999998e-05 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917730+0000", 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26998.0:178971 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 500.83465586099999, 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080073583000000004, 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837556+0000", 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837557+0000", 2024-04-29T14:57:18.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.066e-06 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837562+0000", 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8040000000000001e-06 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837572+0000", 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5019999999999998e-06 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.837579+0000", 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1620000000000002e-06 2024-04-29T14:57:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917515+0000", 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079936129999999994 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917522+0000", 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6059999999999999e-06 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917630+0000", 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108313 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26998.0:178972 7.c e945/939 ondisk, result = 0)", 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 500.83142673200001, 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077106775000000002, 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840786+0000", 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840787+0000", 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3820000000000001e-06 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840791+0000", 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9169999999999999e-06 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:18.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840800+0000", 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1640000000000007e-06 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.840806+0000", 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.648e-06 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917651+0000", 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076845573 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917656+0000", 2024-04-29T14:57:18.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4549999999999997e-06 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:48:57.917892+0000", 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236636 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:18.763 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:57:18.764 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.6 to 1 (10000) 2024-04-29T14:57:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:18.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:18.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2411520173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:18 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:18 smithi040 ceph-mon[39581]: pgmap v2789: 17 pgs: 17 active+clean; 8.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-29T14:57:18.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3833672803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:57:18.900 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:19.073 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2411520173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:18 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:18 smithi194 ceph-mon[44089]: pgmap v2789: 17 pgs: 17 active+clean; 8.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-29T14:57:19.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3833672803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:57:19.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:19.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:19.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2411520173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:19.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:18 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:18 smithi121 ceph-mon[44533]: pgmap v2789: 17 pgs: 17 active+clean; 8.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-29T14:57:19.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3833672803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T14:57:19.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:19.216 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:19.237 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:19.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:19.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:19.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 68 16 136961 136945 125.858 75.0625 0.0157856 0.00794395 2024-04-29T14:57:19.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.745 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:19.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3833672803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:57:19.746 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:19 smithi121 ceph-mon[44533]: osdmap e1094: 12 total, 12 up, 8 in 2024-04-29T14:57:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3833672803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:57:20.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:19 smithi194 ceph-mon[44089]: osdmap e1094: 12 total, 12 up, 8 in 2024-04-29T14:57:20.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:20.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:20.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3833672803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T14:57:20.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:19 smithi040 ceph-mon[39581]: osdmap e1094: 12 total, 12 up, 8 in 2024-04-29T14:57:20.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T14:57:20.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:20.251 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:20.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 69 16 138164 138148 125.124 75.1875 0.018845 0.00799052 2024-04-29T14:57:20.609 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:57:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:20.908 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:20 smithi040 ceph-mon[39581]: pgmap v2791: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:57:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:20.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:20.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-29T14:57:21.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:21.038 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:20 smithi194 ceph-mon[44089]: pgmap v2791: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:57:21.040 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:21.060 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:21.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:21.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:20 smithi121 ceph-mon[44533]: pgmap v2791: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-29T14:57:21.164 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:21.192 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:21.214 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:21.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:21.318 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:21.349 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:21.370 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:21.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:21.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:21.543 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:21.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 70 16 139336 139320 124.383 73.25 0.0205408 0.0080382 2024-04-29T14:57:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:21.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:21.682 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:21.709 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:21.731 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:21.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:21.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:22.083 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:22.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:22.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:22.223 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:22.557 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:22.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 71 16 140543 140527 123.693 75.4375 0.0111941 0.00808321 2024-04-29T14:57:22.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:22.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:22.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:22.712 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:22.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:22.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:22.926 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:22.944 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:57:22.944 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28381.0:140954 9.5 e1094/1065 9:aa08a59e:::benchmark_data_smithi194_196924_object140953:head v 1094'8862, mlcod=1094'8858)", 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:57:22.933593+0000", 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0065438190000000002, 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065810879999999997, 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.933593+0000", 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:22.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.933593+0000", 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.933595+0000", 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.413e-06 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.933624+0000", 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8812000000000001e-05 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.933638+0000", 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4263999999999999e-05 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.933648+0000", 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0013999999999999e-05 2024-04-29T14:57:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.938735+0000", 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050869449999999998 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:22.938758+0000", 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2940999999999999e-05 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:57:22.949 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 1 2024-04-29T14:57:22.949 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:57:22.966 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:22.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:23.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:23.120 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:23.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:23.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:23.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:23.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 72 16 141693 141677 122.974 71.875 0.0252311 0.00813018 2024-04-29T14:57:23.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:23.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:57:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:23.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:23 smithi194 ceph-mon[44089]: pgmap v2792: 17 pgs: 17 active+clean; 8.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-29T14:57:23.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:23 smithi040 ceph-mon[39581]: pgmap v2792: 17 pgs: 17 active+clean; 8.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-29T14:57:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:23 smithi121 ceph-mon[44533]: pgmap v2792: 17 pgs: 17 active+clean; 8.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-29T14:57:24.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T14:57:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:57:24.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 73 16 142886 142870 122.31 74.5625 0.0222891 0.00817465 2024-04-29T14:57:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:24.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:24.812 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3805847840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:57:24.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3805847840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:57:24.885 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:57:24.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3805847840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:57:24.911 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:24.938 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:24.960 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:24.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:25.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:25.137 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:25.175 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:25.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:25.287 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:25.321 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:25.341 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:25.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:25.447 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:25 smithi040 ceph-mon[39581]: pgmap v2793: 17 pgs: 17 active+clean; 8.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-29T14:57:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:25 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3805847840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:57:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:25 smithi040 ceph-mon[39581]: osdmap e1095: 12 total, 12 up, 8 in 2024-04-29T14:57:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:25 smithi040 ceph-mon[39581]: 9.b scrub starts 2024-04-29T14:57:25.484 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:25.511 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:25.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:25.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 74 16 143920 143904 121.531 64.625 0.0147784 0.00822687 2024-04-29T14:57:25.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:25 smithi194 ceph-mon[44089]: pgmap v2793: 17 pgs: 17 active+clean; 8.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-29T14:57:25.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:25 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:25.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3805847840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:57:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:25 smithi194 ceph-mon[44089]: osdmap e1095: 12 total, 12 up, 8 in 2024-04-29T14:57:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:25.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:25 smithi194 ceph-mon[44089]: 9.b scrub starts 2024-04-29T14:57:25.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:25.660 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:25 smithi121 ceph-mon[44533]: pgmap v2793: 17 pgs: 17 active+clean; 8.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-29T14:57:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:25 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3805847840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:57:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:25 smithi121 ceph-mon[44533]: osdmap e1095: 12 total, 12 up, 8 in 2024-04-29T14:57:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:25.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:25.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:25 smithi121 ceph-mon[44533]: 9.b scrub starts 2024-04-29T14:57:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:26.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:26.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 75 16 145025 145009 120.831 69.0625 0.00670102 0.00827197 2024-04-29T14:57:26.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:26 smithi040 ceph-mon[39581]: 9.2 scrub starts 2024-04-29T14:57:26.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:26 smithi040 ceph-mon[39581]: osdmap e1096: 12 total, 12 up, 8 in 2024-04-29T14:57:26.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:26 smithi040 ceph-mon[39581]: 9.b scrub ok 2024-04-29T14:57:26.653 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:26.682 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:26.701 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:26.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:26.758 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:26.805 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:26.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:26 smithi194 ceph-mon[44089]: 9.2 scrub starts 2024-04-29T14:57:26.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:26 smithi194 ceph-mon[44089]: osdmap e1096: 12 total, 12 up, 8 in 2024-04-29T14:57:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:26 smithi194 ceph-mon[44089]: 9.b scrub ok 2024-04-29T14:57:26.832 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:26.857 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:26.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:26.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:26 smithi121 ceph-mon[44533]: 9.2 scrub starts 2024-04-29T14:57:26.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:26 smithi121 ceph-mon[44533]: osdmap e1096: 12 total, 12 up, 8 in 2024-04-29T14:57:26.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:26 smithi121 ceph-mon[44533]: 9.b scrub ok 2024-04-29T14:57:26.961 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:26.989 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:27.010 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:27.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:27.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:57:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:27.182 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:57:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:27.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:57:27.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:57:27.335 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-04-29T14:57:27.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 76 16 145944 145928 119.997 57.4375 0.0101727 0.00833154 2024-04-29T14:57:27.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:57:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:27.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:57:27.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:27.770 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:27.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:27 smithi121 ceph-mon[44533]: 9.5 scrub starts 2024-04-29T14:57:27.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:27 smithi121 ceph-mon[44533]: pgmap v2796: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-29T14:57:27.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:27 smithi121 ceph-mon[44533]: 9.2 scrub ok 2024-04-29T14:57:27.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:27 smithi121 ceph-mon[44533]: 9.e scrub starts 2024-04-29T14:57:27.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:27 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4111068483' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:57:27.817 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:27 smithi121 ceph-mon[44533]: from='client.28465 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:57:27.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:27 smithi194 ceph-mon[44089]: 9.5 scrub starts 2024-04-29T14:57:27.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:27 smithi194 ceph-mon[44089]: pgmap v2796: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-29T14:57:27.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:27 smithi194 ceph-mon[44089]: 9.2 scrub ok 2024-04-29T14:57:27.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:27 smithi194 ceph-mon[44089]: 9.e scrub starts 2024-04-29T14:57:27.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:27 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4111068483' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:57:27.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:27 smithi194 ceph-mon[44089]: from='client.28465 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:57:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:27.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:27 smithi040 ceph-mon[39581]: 9.5 scrub starts 2024-04-29T14:57:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:27 smithi040 ceph-mon[39581]: pgmap v2796: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-29T14:57:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:27 smithi040 ceph-mon[39581]: 9.2 scrub ok 2024-04-29T14:57:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:27 smithi040 ceph-mon[39581]: 9.e scrub starts 2024-04-29T14:57:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:27 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4111068483' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:57:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:27 smithi040 ceph-mon[39581]: from='client.28465 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T14:57:27.933 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:27.961 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:27.982 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:27.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:28.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:28.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:57:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:28.207 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:28.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:28.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:28.386 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:28.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 77 16 146830 146814 119.158 55.375 0.0151787 0.00839031 2024-04-29T14:57:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:28.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:28.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:28 smithi121 ceph-mon[44533]: 9.a scrub starts 2024-04-29T14:57:28.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:28 smithi121 ceph-mon[44533]: from='client.28465 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:57:28.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:28 smithi121 ceph-mon[44533]: osdmap e1097: 12 total, 12 up, 8 in 2024-04-29T14:57:28.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:28.743 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:28.766 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:28.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:28.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:28 smithi194 ceph-mon[44089]: 9.a scrub starts 2024-04-29T14:57:28.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:28 smithi194 ceph-mon[44089]: from='client.28465 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:57:28.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:28 smithi194 ceph-mon[44089]: osdmap e1097: 12 total, 12 up, 8 in 2024-04-29T14:57:28.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:28.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:28 smithi040 ceph-mon[39581]: 9.a scrub starts 2024-04-29T14:57:28.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:28 smithi040 ceph-mon[39581]: from='client.28465 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T14:57:28.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:28 smithi040 ceph-mon[39581]: osdmap e1097: 12 total, 12 up, 8 in 2024-04-29T14:57:28.904 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:28.926 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:28.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:29.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:57:29.072 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:29.205 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:57:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:29.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:57:29.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:29.470 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:29.492 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:29.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:29.559 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 78 16 147781 147765 118.392 59.4375 0.0136538 0.00844481 2024-04-29T14:57:29.596 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:29.624 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:29.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:29 smithi040 ceph-mon[39581]: 9.9 scrub starts 2024-04-29T14:57:29.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:29 smithi040 ceph-mon[39581]: 9.c scrub starts 2024-04-29T14:57:29.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:29 smithi040 ceph-mon[39581]: pgmap v2798: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-04-29T14:57:29.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:29 smithi040 ceph-mon[39581]: 9.e scrub ok 2024-04-29T14:57:29.647 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:29.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:29.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:29.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:29 smithi194 ceph-mon[44089]: 9.9 scrub starts 2024-04-29T14:57:29.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:29 smithi194 ceph-mon[44089]: 9.c scrub starts 2024-04-29T14:57:29.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:29 smithi194 ceph-mon[44089]: pgmap v2798: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-04-29T14:57:29.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:29 smithi194 ceph-mon[44089]: 9.e scrub ok 2024-04-29T14:57:29.850 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:29 smithi121 ceph-mon[44533]: 9.9 scrub starts 2024-04-29T14:57:29.850 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:29 smithi121 ceph-mon[44533]: 9.c scrub starts 2024-04-29T14:57:29.850 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:29 smithi121 ceph-mon[44533]: pgmap v2798: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s 2024-04-29T14:57:29.850 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:29 smithi121 ceph-mon[44533]: 9.e scrub ok 2024-04-29T14:57:29.851 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:29.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:29.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:57:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T14:57:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T14:57:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T14:57:29.937 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:57:29.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:30.038 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:30.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 79 16 148771 148755 117.676 61.875 0.0138953 0.0084963 2024-04-29T14:57:30.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:30 smithi194 ceph-mon[44089]: 9.5 scrub ok 2024-04-29T14:57:30.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:30 smithi194 ceph-mon[44089]: 9.3 scrub starts 2024-04-29T14:57:30.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:30 smithi194 ceph-mon[44089]: 9.a scrub ok 2024-04-29T14:57:30.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:30 smithi194 ceph-mon[44089]: 9.d scrub starts 2024-04-29T14:57:30.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:30 smithi194 ceph-mon[44089]: 9.9 scrub ok 2024-04-29T14:57:30.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:30 smithi194 ceph-mon[44089]: 9.7 scrub starts 2024-04-29T14:57:30.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:30 smithi040 ceph-mon[39581]: 9.5 scrub ok 2024-04-29T14:57:30.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:30 smithi040 ceph-mon[39581]: 9.3 scrub starts 2024-04-29T14:57:30.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:30 smithi040 ceph-mon[39581]: 9.a scrub ok 2024-04-29T14:57:30.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:30 smithi040 ceph-mon[39581]: 9.d scrub starts 2024-04-29T14:57:30.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:30 smithi040 ceph-mon[39581]: 9.9 scrub ok 2024-04-29T14:57:30.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:30 smithi040 ceph-mon[39581]: 9.7 scrub starts 2024-04-29T14:57:30.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:30 smithi121 ceph-mon[44533]: 9.5 scrub ok 2024-04-29T14:57:30.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:30 smithi121 ceph-mon[44533]: 9.3 scrub starts 2024-04-29T14:57:30.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:30 smithi121 ceph-mon[44533]: 9.a scrub ok 2024-04-29T14:57:30.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:30 smithi121 ceph-mon[44533]: 9.d scrub starts 2024-04-29T14:57:30.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:30 smithi121 ceph-mon[44533]: 9.9 scrub ok 2024-04-29T14:57:30.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:30 smithi121 ceph-mon[44533]: 9.7 scrub starts 2024-04-29T14:57:31.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T14:57:31.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:2024-04-29 14:57:31.556281 min lat: 0.000922662 max lat: 0.246703 avg lat: 0.00855787 2024-04-29T14:57:31.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-29T14:57:31.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 80 16 149570 149554 116.83 49.9375 0.0109052 0.00855787 2024-04-29T14:57:31.621 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:57:31.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:31 smithi194 ceph-mon[44089]: pgmap v2799: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 997 op/s 2024-04-29T14:57:31.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:31 smithi194 ceph-mon[44089]: 9.c scrub ok 2024-04-29T14:57:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:31 smithi040 ceph-mon[39581]: pgmap v2799: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 997 op/s 2024-04-29T14:57:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:31 smithi040 ceph-mon[39581]: 9.c scrub ok 2024-04-29T14:57:31.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:31 smithi121 ceph-mon[44533]: pgmap v2799: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 8.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 997 op/s 2024-04-29T14:57:31.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:31 smithi121 ceph-mon[44533]: 9.c scrub ok 2024-04-29T14:57:32.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-29T14:57:32.341 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:32.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:32.448 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:32.498 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:32.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 81 16 150618 150602 116.196 65.5 0.00904058 0.00860468 2024-04-29T14:57:32.576 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:32.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:32.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:32 smithi121 ceph-mon[44533]: 9.3 scrub ok 2024-04-29T14:57:32.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:32 smithi121 ceph-mon[44533]: 9.7 scrub ok 2024-04-29T14:57:32.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:32 smithi121 ceph-mon[44533]: 9.d scrub ok 2024-04-29T14:57:32.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:32 smithi121 ceph-mon[44533]: 9.4 scrub starts 2024-04-29T14:57:32.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:32 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:32.609 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:32 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:32.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:57:32.737 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:32.829 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:57:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:32.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:32 smithi040 ceph-mon[39581]: 9.3 scrub ok 2024-04-29T14:57:32.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:32 smithi040 ceph-mon[39581]: 9.7 scrub ok 2024-04-29T14:57:32.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:32 smithi040 ceph-mon[39581]: 9.d scrub ok 2024-04-29T14:57:32.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:32 smithi040 ceph-mon[39581]: 9.4 scrub starts 2024-04-29T14:57:32.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:32 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:32.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:32 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:32.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:57:32.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:32.991 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:33.039 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:33.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:32 smithi194 ceph-mon[44089]: 9.3 scrub ok 2024-04-29T14:57:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:32 smithi194 ceph-mon[44089]: 9.7 scrub ok 2024-04-29T14:57:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:32 smithi194 ceph-mon[44089]: 9.d scrub ok 2024-04-29T14:57:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:32 smithi194 ceph-mon[44089]: 9.4 scrub starts 2024-04-29T14:57:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:32 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:33.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:32 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:33.115 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:33.144 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:33.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 82 16 151633 151617 115.552 63.4375 0.029441 0.00865293 2024-04-29T14:57:33.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:33 smithi040 ceph-mon[39581]: pgmap v2800: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 937 op/s 2024-04-29T14:57:33.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:33 smithi040 ceph-mon[39581]: 9.1 scrub starts 2024-04-29T14:57:33.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:33 smithi040 ceph-mon[39581]: osdmap e1098: 12 total, 12 up, 8 in 2024-04-29T14:57:33.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:33 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:33.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:33 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:33.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:33 smithi121 ceph-mon[44533]: pgmap v2800: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 937 op/s 2024-04-29T14:57:33.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:33 smithi121 ceph-mon[44533]: 9.1 scrub starts 2024-04-29T14:57:33.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:33 smithi121 ceph-mon[44533]: osdmap e1098: 12 total, 12 up, 8 in 2024-04-29T14:57:33.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:33 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:33.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:33 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:33.894 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:57:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T14:57:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T14:57:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:57:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55786 8.d 8:b7ad01c4:::benchmark_data_smithi194_186548_object55785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.479136+0000", 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.40727181900002, 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.206405844, 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55786 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479136+0000", 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479136+0000", 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479144+0000", 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3610000000000001e-06 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479206+0000", 2024-04-29T14:57:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1859000000000005e-05 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479223+0000", 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7119e-05 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479232+0000", 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3710000000000006e-06 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479282+0000", 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0160000000000001e-05 2024-04-29T14:57:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479387+0000", 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104498 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479547+0000", 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001605 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.481617+0000", 2024-04-29T14:57:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002069843 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.685418+0000", 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20380108299999999 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.685487+0000", 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8468999999999998e-05 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.685541+0000", 2024-04-29T14:57:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4580999999999998e-05 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55798 8.d 8:b0e864d1:::benchmark_data_smithi194_186548_object55797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.483978+0000", 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.40242910500001, 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20377042300000001, 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55798 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.483978+0000", 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.483978+0000", 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.483980+0000", 2024-04-29T14:57:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.875e-06 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484041+0000", 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0939e-05 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484052+0000", 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1252e-05 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484061+0000", 2024-04-29T14:57:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2239999999999995e-06 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484112+0000", 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1056e-05 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484221+0000", 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010945799999999999 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484348+0000", 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012649100000000001 2024-04-29T14:57:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488272+0000", 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039242139999999997 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687645+0000", 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19937343900000001 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687704+0000", 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8479e-05 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687749+0000", 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4996000000000001e-05 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55800 8.d 8:b9d6e6f9:::benchmark_data_smithi194_186548_object55799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.484076+0000", 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.402331355, 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383693, 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55800 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484076+0000", 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484076+0000", 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484077+0000", 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9699999999999995e-07 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484126+0000", 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9508000000000001e-05 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484134+0000", 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5079999999999999e-06 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484140+0000", 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5150000000000003e-06 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.486180+0000", 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020398460000000001 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.486327+0000", 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014636599999999999 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.486467+0000", 2024-04-29T14:57:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014013799999999999 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.490065+0000", 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035985079999999998 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687807+0000", 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19774185 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687867+0000", 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0146000000000003e-05 2024-04-29T14:57:33.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687913+0000", 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5748000000000003e-05 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55801 8.d 8:bb82fe4c:::benchmark_data_smithi194_186548_object55800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.484152+0000", 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.40225563799999, 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20525239100000001, 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.910 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55801 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484152+0000", 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484152+0000", 2024-04-29T14:57:33.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484153+0000", 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6000000000000002e-07 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484239+0000", 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6113999999999996e-05 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484252+0000", 2024-04-29T14:57:33.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3665999999999999e-05 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484258+0000", 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9880000000000001e-06 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488314+0000", 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040557700000000002 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488466+0000", 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015224100000000001 2024-04-29T14:57:33.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488605+0000", 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013902700000000001 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.490212+0000", 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016065820000000001 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689307+0000", 2024-04-29T14:57:33.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19909527899999999 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689362+0000", 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4366000000000001e-05 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689404+0000", 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2497999999999999e-05 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55825 8.d 8:b6e45766:::benchmark_data_smithi194_186548_object55824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.915 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.494640+0000", 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.391767147, 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.194866867, 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55825 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494640+0000", 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494640+0000", 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494643+0000", 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.931e-06 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494713+0000", 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9857999999999994e-05 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494729+0000", 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6413e-05 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494737+0000", 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8439999999999998e-06 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494799+0000", 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1389000000000005e-05 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494904+0000", 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104845 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494992+0000", 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8125000000000001e-05 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.497062+0000", 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020707059999999998 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689449+0000", 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19238635400000001 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689481+0000", 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2147000000000002e-05 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689507+0000", 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6254999999999998e-05 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.920 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55843 8.d 8:b8b16e17:::benchmark_data_smithi194_186548_object55842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.501883+0000", 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.38452420700003, 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.187756907, 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55843 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501883+0000", 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501883+0000", 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501886+0000", 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5799999999999999e-06 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501966+0000", 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0462000000000003e-05 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501979+0000", 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2486e-05 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501987+0000", 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5439999999999999e-06 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.502046+0000", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8774999999999999e-05 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.502144+0000", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8028000000000004e-05 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.502242+0000", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7589000000000004e-05 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.503867+0000", 2024-04-29T14:57:33.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001625004 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689551+0000", 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18568444200000001 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689604+0000", 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2485999999999999e-05 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689640+0000", 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6511000000000003e-05 2024-04-29T14:57:33.925 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55847 8.d 8:b2675a97:::benchmark_data_smithi194_186548_object55846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.504082+0000", 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.38232583000001, 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18729451899999999, 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55847 2024-04-29T14:57:33.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504082+0000", 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504082+0000", 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504083+0000", 2024-04-29T14:57:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7129999999999999e-06 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504162+0000", 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8981999999999997e-05 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504174+0000", 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1877e-05 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504186+0000", 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1912999999999999e-05 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504240+0000", 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3683000000000001e-05 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504375+0000", 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000135337 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504523+0000", 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014781700000000001 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506003+0000", 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00148029 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691301+0000", 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18529801800000001 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691348+0000", 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6669000000000003e-05 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691376+0000", 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8220000000000001e-05 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55853 8.d 8:b6da80e1:::benchmark_data_smithi194_186548_object55852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.506457+0000", 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.37995009299999, 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.185007753, 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55853 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506457+0000", 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506457+0000", 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506459+0000", 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8369999999999999e-06 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506548+0000", 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8329000000000001e-05 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506561+0000", 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3273e-05 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506568+0000", 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5279999999999998e-06 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506616+0000", 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7604e-05 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506725+0000", 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108674 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506843+0000", 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011833900000000001 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.508399+0000", 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015563759999999999 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691410+0000", 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18301077299999999 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691441+0000", 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0508999999999999e-05 2024-04-29T14:57:33.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691465+0000", 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4511e-05 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55865 8.d 8:b5fea464:::benchmark_data_smithi194_186548_object55864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.512697+0000", 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.37371027099999, 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17965973299999999, 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.936 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55865 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512697+0000", 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512697+0000", 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512699+0000", 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3039999999999999e-06 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512778+0000", 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8614000000000003e-05 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512791+0000", 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3050999999999999e-05 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512798+0000", 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3529999999999997e-06 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512855+0000", 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6495e-05 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.513015+0000", 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016034400000000001 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.513179+0000", 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016350800000000001 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.516866+0000", 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036867520000000002 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692265+0000", 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.175399061 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692316+0000", 2024-04-29T14:57:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1211999999999997e-05 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692357+0000", 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1038999999999998e-05 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55870 8.d 8:b73c258c:::benchmark_data_smithi194_186548_object55869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.514805+0000", 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.37160209799998, 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17769389799999999, 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55870 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514805+0000", 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514805+0000", 2024-04-29T14:57:33.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514808+0000", 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2330000000000001e-06 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514921+0000", 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113451 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514932+0000", 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1071e-05 2024-04-29T14:57:33.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514941+0000", 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7229999999999993e-06 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514995+0000", 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3783999999999999e-05 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515123+0000", 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128608 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515244+0000", 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120907 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.518630+0000", 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033863610000000001 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692404+0000", 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17377337800000001 2024-04-29T14:57:33.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692459+0000", 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4839000000000001e-05 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692499+0000", 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0543000000000001e-05 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.946 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55874 8.d 8:b6a90c97:::benchmark_data_smithi194_186548_object55873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.515297+0000", 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.37110996799998, 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.178947574, 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55874 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515297+0000", 2024-04-29T14:57:33.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515297+0000", 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515299+0000", 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3090000000000001e-06 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515408+0000", 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109038 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515419+0000", 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1345999999999999e-05 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515427+0000", 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7549999999999994e-06 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.516892+0000", 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146514 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.517033+0000", 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014117000000000001 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.517174+0000", 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014041300000000001 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.518748+0000", 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015739149999999999 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.694148+0000", 2024-04-29T14:57:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.175400532 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.694204+0000", 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5454999999999997e-05 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.694245+0000", 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501000000000001e-05 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55881 8.d 8:b86e0694:::benchmark_data_smithi194_186548_object55880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.520125+0000", 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.36628253399999, 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17558306000000001, 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55881 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520125+0000", 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520125+0000", 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520127+0000", 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3420000000000002e-06 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520197+0000", 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9814000000000006e-05 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520212+0000", 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4543e-05 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520221+0000", 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0699999999999996e-06 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520275+0000", 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3844999999999997e-05 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520279+0000", 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5310000000000003e-06 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.522866+0000", 2024-04-29T14:57:33.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025868760000000001 2024-04-29T14:57:33.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.522988+0000", 2024-04-29T14:57:33.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012194 2024-04-29T14:57:33.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.523105+0000", 2024-04-29T14:57:33.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116737 2024-04-29T14:57:33.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.524986+0000", 2024-04-29T14:57:33.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018813829999999999 2024-04-29T14:57:33.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695621+0000", 2024-04-29T14:57:33.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17063498299999999 2024-04-29T14:57:33.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695676+0000", 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5096000000000002e-05 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695708+0000", 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1900000000000003e-05 2024-04-29T14:57:33.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.961 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.961 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.961 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55896 8.d 8:b67bbaf5:::benchmark_data_smithi194_186548_object55895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:57:33.961 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.534147+0000", 2024-04-29T14:57:33.961 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.35226004100002, 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.161730709, 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55896 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.962 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534147+0000", 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534147+0000", 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534150+0000", 2024-04-29T14:57:33.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8100000000000002e-06 2024-04-29T14:57:33.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534212+0000", 2024-04-29T14:57:33.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1506000000000004e-05 2024-04-29T14:57:33.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534225+0000", 2024-04-29T14:57:33.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3664e-05 2024-04-29T14:57:33.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534233+0000", 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.79e-06 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534280+0000", 2024-04-29T14:57:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6791e-05 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534382+0000", 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102073 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534468+0000", 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5833000000000003e-05 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.538324+0000", 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038565439999999999 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695779+0000", 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.157454439 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695835+0000", 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5769000000000001e-05 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695878+0000", 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3489999999999999e-05 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55898 8.d 8:b5f4d5c8:::benchmark_data_smithi194_186548_object55897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.534722+0000", 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.35168577000002, 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16130372000000001, 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55898 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534722+0000", 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534722+0000", 2024-04-29T14:57:33.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534723+0000", 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.53e-06 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534784+0000", 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1001000000000001e-05 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534794+0000", 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7550000000000006e-06 2024-04-29T14:57:33.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534800+0000", 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3659999999999997e-06 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536179+0000", 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013789379999999999 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536294+0000", 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011436499999999999 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536387+0000", 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3579000000000006e-05 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.538329+0000", 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001941952 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695920+0000", 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.157591237 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695988+0000", 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7735000000000001e-05 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696025+0000", 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7261999999999997e-05 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55899 8.d 8:b55d3b13:::benchmark_data_smithi194_186548_object55898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:57:33.974 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.536001+0000", 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.350406604, 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16021872400000001, 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55899 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536001+0000", 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536001+0000", 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536002+0000", 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.119e-06 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536073+0000", 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0591000000000002e-05 2024-04-29T14:57:33.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536080+0000", 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5519999999999999e-06 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536085+0000", 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0849999999999996e-06 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.537244+0000", 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011585149999999999 2024-04-29T14:57:33.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.537339+0000", 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5093999999999995e-05 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.537420+0000", 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1573e-05 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.538506+0000", 2024-04-29T14:57:33.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001085981 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696128+0000", 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15762129799999999 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696180+0000", 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2200999999999997e-05 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696220+0000", 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9715000000000002e-05 2024-04-29T14:57:33.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55934 8.d 8:b0f18902:::benchmark_data_smithi194_186548_object55933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.616473+0000", 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 305.26993394700003, 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079838084000000004, 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55934 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.980 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616473+0000", 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616473+0000", 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616476+0000", 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.469e-06 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616521+0000", 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4690000000000001e-05 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616533+0000", 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2427e-05 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616557+0000", 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4187999999999999e-05 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616623+0000", 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5411000000000001e-05 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616780+0000", 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000157763 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.616940+0000", 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015942199999999999 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.618592+0000", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001651703 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696249+0000", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.077657725999999996 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696279+0000", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0107000000000002e-05 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696312+0000", 2024-04-29T14:57:33.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2178000000000003e-05 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28381.0:94772 9.5 e1083/1065 9:a1b84a53:::benchmark_data_smithi194_196924_object94771:head v 1083'5988, mlcod=1083'5986)", 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:56:48.057880+0000", 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.828527661000003, 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.062241263999999998, 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.057880+0000", 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.057880+0000", 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.057881+0000", 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.057e-06 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.057900+0000", 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9213000000000001e-05 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.057906+0000", 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7049999999999996e-06 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.057909+0000", 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.675e-06 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058061+0000", 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015150600000000001 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058076+0000", 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4919999999999999e-05 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.120087+0000", 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.062011082000000002 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.120121+0000", 2024-04-29T14:57:33.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4106000000000002e-05 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28381.0:94774 9.5 e1083/1065 9:acf7d240:::benchmark_data_smithi194_196924_object94773:head v 1083'5989, mlcod=1083'5986)", 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:56:48.058436+0000", 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.827971886999997, 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.061720745, 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058436+0000", 2024-04-29T14:57:33.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058436+0000", 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058437+0000", 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.294e-06 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058462+0000", 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4774e-05 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.108703+0000", 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.050241115000000003 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.108723+0000", 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0681000000000001e-05 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.111274+0000", 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025503779999999998 2024-04-29T14:57:33.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.111290+0000", 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6422e-05 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.120124+0000", 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088338549999999998 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.120156+0000", 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2225999999999999e-05 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.992 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:94781 9.d 9:b7141cca:::benchmark_data_smithi194_196924_object94780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1083)", 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:56:48.058649+0000", 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.82775822, 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.068908264999999996, 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94781 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058649+0000", 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058649+0000", 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058651+0000", 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.198e-06 2024-04-29T14:57:33.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.058709+0000", 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7476999999999999e-05 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.108680+0000", 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.049971338999999997 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.108698+0000", 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7479999999999999e-05 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.117984+0000", 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092859029999999999 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.118135+0000", 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151063 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.118236+0000", 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010100899999999999 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.127407+0000", 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091710579999999993 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.127449+0000", 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2416999999999997e-05 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.127508+0000", 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9274000000000002e-05 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:33.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:56:48.127557+0000", 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9047e-05 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:147374 9.e 9:76328fe0:::benchmark_data_smithi194_196924_object147373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1097)", 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:57:29.150801+0000", 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7356061240000002, 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.061592119000000001, 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:57:33.998 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 147374 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.150801+0000", 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.150801+0000", 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:33.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.150804+0000", 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.576e-06 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.150834+0000", 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0555000000000001e-05 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.150846+0000", 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1482e-05 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:57:34.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.150854+0000", 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9070000000000007e-06 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.152817+0000", 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001963574 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.152954+0000", 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136336 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.153083+0000", 2024-04-29T14:57:34.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012956799999999999 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.162197+0000", 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091138550000000006 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.212302+0000", 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.050105031000000001 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.212351+0000", 2024-04-29T14:57:34.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8748999999999997e-05 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:57:29.212393+0000", 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2485999999999999e-05 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:57:34.003 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:57:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:33 smithi194 ceph-mon[44089]: pgmap v2800: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 937 op/s 2024-04-29T14:57:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:33 smithi194 ceph-mon[44089]: 9.1 scrub starts 2024-04-29T14:57:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:33 smithi194 ceph-mon[44089]: osdmap e1098: 12 total, 12 up, 8 in 2024-04-29T14:57:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:33 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:34.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:33 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:34.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 83 16 152720 152704 114.979 67.9375 0.0129437 0.00869556 2024-04-29T14:57:34.674 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:34.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T14:57:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:34.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:34.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:57:34.842 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:34.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:57:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:34 smithi121 ceph-mon[44533]: osdmap e1099: 12 total, 12 up, 8 in 2024-04-29T14:57:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:34 smithi121 ceph-mon[44533]: 9.4 scrub ok 2024-04-29T14:57:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:34.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:57:34.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:34.999 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:57:35.036 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:35.062 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:34 smithi040 ceph-mon[39581]: osdmap e1099: 12 total, 12 up, 8 in 2024-04-29T14:57:35.062 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:34 smithi040 ceph-mon[39581]: 9.4 scrub ok 2024-04-29T14:57:35.062 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:35.062 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:35.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:35.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:34 smithi194 ceph-mon[44089]: osdmap e1099: 12 total, 12 up, 8 in 2024-04-29T14:57:35.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:34 smithi194 ceph-mon[44089]: 9.4 scrub ok 2024-04-29T14:57:35.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:35.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:35.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:35.177 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:35.205 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:35.281 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:35.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:35.323 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:57:35.358 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:57:35.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:57:35.396 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:57:35.430 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:57:35.440 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:35.460 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:57:35.499 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:57:35.539 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:57:35.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 84 16 153811 153795 114.422 68.1875 0.0109678 0.0087383 2024-04-29T14:57:35.574 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:57:35.599 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:57:35.624 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:57:35.652 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:57:35.681 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:57:35.683 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:57:35.683 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:57:35.683 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:57:35.683 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:57:35.684 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:57:35.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:35 smithi121 ceph-mon[44533]: 9.1 scrub ok 2024-04-29T14:57:35.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:35 smithi121 ceph-mon[44533]: pgmap v2803: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:57:35.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:35 smithi121 ceph-mon[44533]: osdmap e1100: 12 total, 12 up, 8 in 2024-04-29T14:57:35.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/513096199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:36.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:35 smithi040 ceph-mon[39581]: 9.1 scrub ok 2024-04-29T14:57:36.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:35 smithi040 ceph-mon[39581]: pgmap v2803: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:57:36.038 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:35 smithi040 ceph-mon[39581]: osdmap e1100: 12 total, 12 up, 8 in 2024-04-29T14:57:36.038 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/513096199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:36.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:35 smithi194 ceph-mon[44089]: 9.1 scrub ok 2024-04-29T14:57:36.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:35 smithi194 ceph-mon[44089]: pgmap v2803: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-29T14:57:36.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:35 smithi194 ceph-mon[44089]: osdmap e1100: 12 total, 12 up, 8 in 2024-04-29T14:57:36.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/513096199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:36.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 85 16 154844 154828 113.835 64.5625 0.0105469 0.00878317 2024-04-29T14:57:36.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: 9.f scrub starts 2024-04-29T14:57:36.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.772 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: 9.f scrub starts 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:36.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: 9.f scrub starts 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:37.392 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:57:37.392 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:57:37.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 86 16 155836 155820 113.232 62 0.00697756 0.00882935 2024-04-29T14:57:37.598 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:57:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:37.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:57:37.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:57:37.766 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:37.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:57:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:37 smithi194 ceph-mon[44089]: 9.6 scrub starts 2024-04-29T14:57:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:37 smithi194 ceph-mon[44089]: pgmap v2805: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-04-29T14:57:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:37.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:57:37.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:37 smithi040 ceph-mon[39581]: 9.6 scrub starts 2024-04-29T14:57:37.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:37 smithi040 ceph-mon[39581]: pgmap v2805: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-04-29T14:57:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:37 smithi121 ceph-mon[44533]: 9.6 scrub starts 2024-04-29T14:57:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:37 smithi121 ceph-mon[44533]: pgmap v2805: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-04-29T14:57:37.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:37.964 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:38.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:38.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T14:57:38.279 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:57:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:38.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 87 16 156870 156854 112.673 64.625 0.0103632 0.00887352 2024-04-29T14:57:38.587 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:57:39.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:38 smithi194 ceph-mon[44089]: 9.f scrub ok 2024-04-29T14:57:39.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:38 smithi194 ceph-mon[44089]: pgmap v2806: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 52 MiB/s wr, 824 op/s 2024-04-29T14:57:39.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1781813985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:57:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:38 smithi040 ceph-mon[39581]: 9.f scrub ok 2024-04-29T14:57:39.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:38 smithi040 ceph-mon[39581]: pgmap v2806: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 52 MiB/s wr, 824 op/s 2024-04-29T14:57:39.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1781813985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:57:39.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:38 smithi121 ceph-mon[44533]: 9.f scrub ok 2024-04-29T14:57:39.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:38 smithi121 ceph-mon[44533]: pgmap v2806: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 52 MiB/s wr, 824 op/s 2024-04-29T14:57:39.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1781813985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T14:57:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:57:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:39.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:57:39.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:39.444 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:39.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:39.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:39.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 88 16 157916 157900 112.136 65.375 0.00857132 0.00891633 2024-04-29T14:57:39.623 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:39.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:39.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:39.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:39.806 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:39.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:39.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:39.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:39 smithi121 ceph-mon[44533]: 9.8 scrub starts 2024-04-29T14:57:39.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:39 smithi121 ceph-mon[44533]: 9.6 scrub ok 2024-04-29T14:57:39.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1781813985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:57:39.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:39 smithi121 ceph-mon[44533]: osdmap e1101: 12 total, 12 up, 8 in 2024-04-29T14:57:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.5 to 1 (10000) 2024-04-29T14:57:39.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:39.997 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:40.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:40.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:39 smithi194 ceph-mon[44089]: 9.8 scrub starts 2024-04-29T14:57:40.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:39 smithi194 ceph-mon[44089]: 9.6 scrub ok 2024-04-29T14:57:40.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1781813985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:57:40.070 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:39 smithi194 ceph-mon[44089]: osdmap e1101: 12 total, 12 up, 8 in 2024-04-29T14:57:40.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:39 smithi040 ceph-mon[39581]: 9.8 scrub starts 2024-04-29T14:57:40.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:39 smithi040 ceph-mon[39581]: 9.6 scrub ok 2024-04-29T14:57:40.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1781813985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T14:57:40.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:39 smithi040 ceph-mon[39581]: osdmap e1101: 12 total, 12 up, 8 in 2024-04-29T14:57:40.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:40.209 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:40.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:40.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:40.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:40.416 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:40.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 89 16 158917 158901 111.579 62.5625 0.0197184 0.00896055 2024-04-29T14:57:40.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-29T14:57:40.955 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:40 smithi040 ceph-mon[39581]: pgmap v2808: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:57:40.972 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:40.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:41.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:40 smithi194 ceph-mon[44089]: pgmap v2808: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:57:41.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:41.113 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:41.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:40 smithi121 ceph-mon[44533]: pgmap v2808: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-29T14:57:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:41.560 INFO:tasks.radosbench.radosbench.2.smithi194.stdout: 90 16 160003 159987 111.093 67.875 0.0232372 0.00899945 2024-04-29T14:57:41.792 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:57:41.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:57:41.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:41 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:41.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:41 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:41.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:41 smithi040 ceph-mon[39581]: 9.8 scrub ok 2024-04-29T14:57:41.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:41 smithi040 ceph-mon[39581]: osdmap e1102: 12 total, 12 up, 8 in 2024-04-29T14:57:41.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1537501800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:57:42.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:41 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:42.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:41 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:42.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:41 smithi194 ceph-mon[44089]: 9.8 scrub ok 2024-04-29T14:57:42.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:41 smithi194 ceph-mon[44089]: osdmap e1102: 12 total, 12 up, 8 in 2024-04-29T14:57:42.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1537501800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:57:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:41 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:41 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:41 smithi121 ceph-mon[44533]: 9.8 scrub ok 2024-04-29T14:57:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:41 smithi121 ceph-mon[44533]: osdmap e1102: 12 total, 12 up, 8 in 2024-04-29T14:57:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1537501800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:57:42.468 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:42.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:42.561 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total time run: 90.0085 2024-04-29T14:57:42.561 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Total writes made: 160003 2024-04-29T14:57:42.561 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Write size: 65536 2024-04-29T14:57:42.561 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Object size: 65536 2024-04-29T14:57:42.561 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Bandwidth (MB/sec): 111.103 2024-04-29T14:57:42.561 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Bandwidth: 57.6724 2024-04-29T14:57:42.561 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max bandwidth (MB/sec): 341.75 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min bandwidth (MB/sec): 49.9375 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average IOPS: 1777 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev IOPS: 922.758 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max IOPS: 5468 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min IOPS: 799 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Average Latency(s): 0.00900016 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Stddev Latency(s): 0.00839311 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Max latency(s): 0.246703 2024-04-29T14:57:42.562 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Min latency(s): 0.000922662 2024-04-29T14:57:42.569 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Cleaning up (deleting benchmark objects) 2024-04-29T14:57:42.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:57:42.630 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:42.666 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:57:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:42.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:57:42.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:42.813 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:42.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:42.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:42.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:42 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:57:42.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:42 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:57:42.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1537501800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:57:42.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:42 smithi121 ceph-mon[44533]: osdmap e1103: 12 total, 12 up, 8 in 2024-04-29T14:57:42.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:42 smithi121 ceph-mon[44533]: pgmap v2811: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-29T14:57:42.941 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:42.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:43.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:43.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:43.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:42 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:57:43.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:42 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:57:43.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1537501800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:57:43.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:42 smithi040 ceph-mon[39581]: osdmap e1103: 12 total, 12 up, 8 in 2024-04-29T14:57:43.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:42 smithi040 ceph-mon[39581]: pgmap v2811: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-29T14:57:43.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:43.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:42 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:57:43.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:42 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:57:43.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1537501800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:57:43.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:42 smithi194 ceph-mon[44089]: osdmap e1103: 12 total, 12 up, 8 in 2024-04-29T14:57:43.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:42 smithi194 ceph-mon[44089]: pgmap v2811: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-29T14:57:43.092 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:43.104 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:57:43.104 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:57:43.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:43.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:43.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:43.265 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:43.398 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:43.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:43.413 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:43.502 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:43.540 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:43.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:43.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:44.028 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:44.054 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T14:57:44.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:44.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:44.171 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:44.202 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:44.225 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:44.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:44.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:44.376 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:44.377 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:57:44.405 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:44.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:44.517 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:44.545 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:44.674 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:44.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:44.780 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:44.814 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.8 to 1 (10000) 2024-04-29T14:57:44.891 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:44.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:44.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: osdmap e1104: 12 total, 12 up, 8 in 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: pgmap v2813: 17 pgs: 17 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 1.11k op/s 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2166424994' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:57:45.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:44 smithi040 ceph-mon[39581]: from='client.38228 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:57:45.037 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:45.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:45.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:45.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:57:45.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: osdmap e1104: 12 total, 12 up, 8 in 2024-04-29T14:57:45.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:57:45.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: pgmap v2813: 17 pgs: 17 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 1.11k op/s 2024-04-29T14:57:45.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2166424994' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:57:45.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:44 smithi194 ceph-mon[44089]: from='client.38228 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: osdmap e1104: 12 total, 12 up, 8 in 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: pgmap v2813: 17 pgs: 17 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 1.11k op/s 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2166424994' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:57:45.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:44 smithi121 ceph-mon[44533]: from='client.38228 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T14:57:45.170 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:45.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:45.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:57:45.319 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:45.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:57:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:57:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:45.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:57:45.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:45.715 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:45.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:45.823 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:45.859 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:45.884 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:45.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:45.958 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:45.959 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:45.959 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:45 smithi040 ceph-mon[39581]: from='client.38228 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:57:45.959 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:45 smithi040 ceph-mon[39581]: osdmap e1105: 12 total, 12 up, 8 in 2024-04-29T14:57:45.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:45.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-29T14:57:46.072 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:46.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:46.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:46.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:45 smithi194 ceph-mon[44089]: from='client.38228 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:57:46.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:45 smithi194 ceph-mon[44089]: osdmap e1105: 12 total, 12 up, 8 in 2024-04-29T14:57:46.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:46.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:46.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:45 smithi121 ceph-mon[44533]: from='client.38228 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T14:57:46.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:45 smithi121 ceph-mon[44533]: osdmap e1105: 12 total, 12 up, 8 in 2024-04-29T14:57:46.144 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:46.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:46.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:57:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:46.572 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:57:46.572 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:57:46.939 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:46 smithi194 ceph-mon[44089]: pgmap v2815: 17 pgs: 17 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 853 op/s 2024-04-29T14:57:47.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:46 smithi040 ceph-mon[39581]: pgmap v2815: 17 pgs: 17 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 853 op/s 2024-04-29T14:57:47.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:46 smithi121 ceph-mon[44533]: pgmap v2815: 17 pgs: 17 active+clean; 9.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 853 op/s 2024-04-29T14:57:47.335 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T14:57:47.642 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:57:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:57:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:57:47.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/263509534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:47.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:47.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:47 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/263509534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:48.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:47 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.213 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/263509534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:48.213 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.213 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:47 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.621 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:48.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:48.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:48 smithi040 ceph-mon[39581]: pgmap v2816: 17 pgs: 17 active+clean; 9.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 154 B/s rd, 23 MiB/s wr, 754 op/s 2024-04-29T14:57:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:57:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:48.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:57:48.990 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:49.018 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.019 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:48 smithi194 ceph-mon[44089]: pgmap v2816: 17 pgs: 17 active+clean; 9.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 154 B/s rd, 23 MiB/s wr, 754 op/s 2024-04-29T14:57:49.021 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:49.127 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:49.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:48 smithi121 ceph-mon[44533]: pgmap v2816: 17 pgs: 17 active+clean; 9.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 154 B/s rd, 23 MiB/s wr, 754 op/s 2024-04-29T14:57:49.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:49.290 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:49.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:49.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:49.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:49.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:49.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:49.505 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:49.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:49.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:49.671 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:49.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:49.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:49.744 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:57:49.744 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:57:49.744 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:57:49.744 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:57:49.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:49.853 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:50.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:50.210 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:50.245 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:50.479 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:50.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:50.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:50.505 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T14:57:50.585 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:50.619 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:50.639 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:50.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:50.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:57:50.745 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:50.779 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:50.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:50.858 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:57:50.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:57:50.957 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:50.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:57:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:50.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:57:51.099 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:51.182 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:51.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:51.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:51.332 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:51.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:51.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:51.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:51.480 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:51 smithi194 ceph-mon[44089]: pgmap v2817: 17 pgs: 17 active+clean; 9.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-29T14:57:51.480 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3281956085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:51.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:51.526 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:51.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:51.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:51.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:51 smithi121 ceph-mon[44533]: pgmap v2817: 17 pgs: 17 active+clean; 9.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-29T14:57:51.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3281956085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:51.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:51 smithi121 ceph-mon[44533]: from='client.38237 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:51.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:57:51.700 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:51.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:51 smithi194 ceph-mon[44089]: from='client.38237 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:51 smithi040 ceph-mon[39581]: pgmap v2817: 17 pgs: 17 active+clean; 9.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-29T14:57:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3281956085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:51 smithi040 ceph-mon[39581]: from='client.38237 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:57:51.908 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:57:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:51.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:57:52.012 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:57:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:52.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:52 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:52.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:52 smithi194 ceph-mon[44089]: from='client.38237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:57:52.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:52 smithi194 ceph-mon[44089]: osdmap e1106: 12 total, 12 up, 8 in 2024-04-29T14:57:52.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:52.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:52 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:52 smithi040 ceph-mon[39581]: from='client.38237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:57:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:52 smithi040 ceph-mon[39581]: osdmap e1106: 12 total, 12 up, 8 in 2024-04-29T14:57:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:52.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:52 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:57:52.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:52 smithi121 ceph-mon[44533]: from='client.38237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:57:52.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:52 smithi121 ceph-mon[44533]: osdmap e1106: 12 total, 12 up, 8 in 2024-04-29T14:57:52.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:52.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:53.067 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:57:53.156 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:57:53.156 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:57:53.156 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:57:53.156 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:57:53.156 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:57:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:53 smithi040 ceph-mon[39581]: pgmap v2819: 17 pgs: 17 active+clean; 9.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 2.2 MiB/s wr, 952 op/s 2024-04-29T14:57:53.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:53 smithi040 ceph-mon[39581]: osdmap e1107: 12 total, 12 up, 8 in 2024-04-29T14:57:53.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:57:53.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:57:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:57:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:53.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:57:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:53 smithi194 ceph-mon[44089]: pgmap v2819: 17 pgs: 17 active+clean; 9.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 2.2 MiB/s wr, 952 op/s 2024-04-29T14:57:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:53 smithi194 ceph-mon[44089]: osdmap e1107: 12 total, 12 up, 8 in 2024-04-29T14:57:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:57:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:57:53.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:53.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:53 smithi121 ceph-mon[44533]: pgmap v2819: 17 pgs: 17 active+clean; 9.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 2.2 MiB/s wr, 952 op/s 2024-04-29T14:57:53.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:53 smithi121 ceph-mon[44533]: osdmap e1107: 12 total, 12 up, 8 in 2024-04-29T14:57:53.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:57:53.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:57:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:53.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:54.078 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:54.243 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:54.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:54.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:54.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:57:54.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:57:54.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:57:54.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:54 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:57:54.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:54 smithi040 ceph-mon[39581]: osdmap e1108: 12 total, 12 up, 8 in 2024-04-29T14:57:54.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:57:54.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:57:54.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:57:54.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:54 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:57:54.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:54 smithi194 ceph-mon[44089]: osdmap e1108: 12 total, 12 up, 8 in 2024-04-29T14:57:54.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:57:54.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:57:54.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:57:54.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:54 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:57:54.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:54 smithi121 ceph-mon[44533]: osdmap e1108: 12 total, 12 up, 8 in 2024-04-29T14:57:55.201 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T14:57:55.513 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:57:55.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:55 smithi040 ceph-mon[39581]: pgmap v2822: 17 pgs: 17 active+clean; 9.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-29T14:57:55.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1765086722' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:57:55.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:55 smithi040 ceph-mon[39581]: from='client.28501 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:57:55.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:57:55.779 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-04-29T14:57:55.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:55 smithi194 ceph-mon[44089]: pgmap v2822: 17 pgs: 17 active+clean; 9.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-29T14:57:55.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1765086722' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:57:55.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:55 smithi194 ceph-mon[44089]: from='client.28501 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:57:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:55 smithi121 ceph-mon[44533]: pgmap v2822: 17 pgs: 17 active+clean; 9.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-29T14:57:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1765086722' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:57:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:55 smithi121 ceph-mon[44533]: from='client.28501 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T14:57:56.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:56.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:56.304 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:57:56.338 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:56.365 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:57:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:56.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:57:56.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:56.528 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:56.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:56.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:56.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:57:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:56.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:56 smithi121 ceph-mon[44533]: from='client.28501 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T14:57:56.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:56 smithi121 ceph-mon[44533]: osdmap e1109: 12 total, 12 up, 8 in 2024-04-29T14:57:56.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:57:56.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:57:56.777 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:56 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:57:56.957 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:57:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:56.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:57:57.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:57.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:56 smithi194 ceph-mon[44089]: from='client.28501 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T14:57:57.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:56 smithi194 ceph-mon[44089]: osdmap e1109: 12 total, 12 up, 8 in 2024-04-29T14:57:57.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:57:57.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:57:57.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:56 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:57:57.091 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:57.112 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:57.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:57.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:56 smithi040 ceph-mon[39581]: from='client.28501 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T14:57:57.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:56 smithi040 ceph-mon[39581]: osdmap e1109: 12 total, 12 up, 8 in 2024-04-29T14:57:57.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:57:57.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:57:57.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:56 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:57:57.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:57:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:57.811 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:57:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:57.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:57:57.899 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:57:57.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:57:57.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:57:57.948 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:57:57.971 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:57:57.974 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:57.998 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:57:58.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:57:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:58.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:57:58.032 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:57:58.064 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:57:58.096 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:57:58.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:57:58.126 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:57:58.149 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:58.156 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:57:58.183 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:57:58.207 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:57:58.344 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:57:58.344 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:57:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:57:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:58.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:57:58.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2922507859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:57:58.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:57:58.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:57:58.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.717 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:57:58.717 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:58.744 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.744 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.744 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2922507859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:58.744 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.744 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.745 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:57:58.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:57:58.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:57:58.746 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.747 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:58.755 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-29T14:57:58.756 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-29T14:57:58.756 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-29T14:57:58.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T14:57:58.772 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:57:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:58.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:57:58.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:58.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2922507859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:57:58.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:57:58.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:57:58.929 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:58.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:58.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:59.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:57:59.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T14:57:59.137 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:59.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:57:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:59.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:57:59.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:57:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:59.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:57:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:59.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:57:59.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:57:59.494 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:59.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:57:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:59.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:57:59.637 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:59.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.638 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.639 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:57:59 smithi121 ceph-mon[44533]: pgmap v2825: 17 pgs: 17 active+clean; 8.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 937 op/s 2024-04-29T14:57:59.665 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:59.688 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:59.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:57:59 smithi194 ceph-mon[44089]: pgmap v2825: 17 pgs: 17 active+clean; 8.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 937 op/s 2024-04-29T14:57:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:59.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:59.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:59.822 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:59.844 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:57:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:57:59.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:57:59.858 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:57:59.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:57:59 smithi040 ceph-mon[39581]: pgmap v2825: 17 pgs: 17 active+clean; 8.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 937 op/s 2024-04-29T14:57:59.948 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:57:59.977 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:57:59.997 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:00.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:00.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:00.136 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:00.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:00.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:00.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:00.347 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:00.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:00.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:00.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:00.532 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:00.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:00.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:00.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:00.721 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:00.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:00.857 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:58:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:01.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:58:01.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:01 smithi040 ceph-mon[39581]: pgmap v2826: 17 pgs: 17 active+clean; 8.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:58:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:01.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:58:01.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:58:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:01.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:01 smithi194 ceph-mon[44089]: pgmap v2826: 17 pgs: 17 active+clean; 8.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:01.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:01 smithi121 ceph-mon[44533]: pgmap v2826: 17 pgs: 17 active+clean; 8.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:02.181 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:58:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:02.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:58:02.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:02.346 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:02.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:02.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:02.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:02.506 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:02.526 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:03.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:03 smithi194 ceph-mon[44089]: pgmap v2827: 17 pgs: 17 active+clean; 8.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:58:03.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:03 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:03.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:03 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:03 smithi040 ceph-mon[39581]: pgmap v2827: 17 pgs: 17 active+clean; 8.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:58:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:03 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:03 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:03.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:03 smithi121 ceph-mon[44533]: pgmap v2827: 17 pgs: 17 active+clean; 8.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-29T14:58:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:03 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:03.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:03 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:04.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:04 smithi040 ceph-mon[39581]: osdmap e1110: 12 total, 12 up, 8 in 2024-04-29T14:58:04.790 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:04.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:04.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:04 smithi194 ceph-mon[44089]: osdmap e1110: 12 total, 12 up, 8 in 2024-04-29T14:58:04.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:04 smithi121 ceph-mon[44533]: osdmap e1110: 12 total, 12 up, 8 in 2024-04-29T14:58:04.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:04.935 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:05.193 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:05.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:05.301 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:05.329 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:05.350 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:05.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:05.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:05.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:05 smithi040 ceph-mon[39581]: pgmap v2829: 17 pgs: 17 active+clean; 8.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:05.519 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:05.631 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:58:05.631 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":1110,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T14:58:03.441675+0000","last_up_change":"2024-04-29T14:56:03.723063+0000","last_in_change":"2024-04-29T14:56:24.512006+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":31,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"796","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":10,"score_stable":10,"optimal_score":0.20000000298023224,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}},{"pool":9,"pool_name":"unique_pool_7","create_time":"2024-04-29T14:55:57.720939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1071","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.130000114440918,"score_stable":3.130000114440918,"optimal_score":0.80000001192092896,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":0,"weight":0,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":937,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":4178456629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":4178456629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":4178456629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":4178456629}]},"public_addr":"172.21.15.40:6805/4178456629","cluster_addr":"172.21.15.40:6806/4178456629","heartbeat_back_addr":"172.21.15.40:6808/4178456629","heartbeat_front_addr":"172.21.15.40:6807/4178456629","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":230,"up_from":260,"up_thru":874,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3416690575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3416690575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3416690575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3416690575}]},"public_addr":"172.21.15.40:6809/3416690575","cluster_addr":"172.21.15.40:6810/3416690575","heartbeat_back_addr":"172.21.15.40:6812/3416690575","heartbeat_front_addr":"172.21.15.40:6811/3416690575","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1011,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":388,"last_clean_end":457,"up_from":561,"up_thru":1065,"down_at":459,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2522198013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2522198013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2522198013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2522198013}]},"public_addr":"172.21.15.121:6804/2522198013","cluster_addr":"172.21.15.121:6805/2522198013","heartbeat_back_addr":"172.21.15.121:6807/2522198013","heartbeat_front_addr":"172.21.15.121:6806/2522198013","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":64,"last_clean_end":320,"up_from":354,"up_thru":1065,"down_at":321,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":165633203}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":165633203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":165633203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":165633203}]},"public_addr":"172.21.15.194:6800/165633203","cluster_addr":"172.21.15.194:6801/165633203","heartbeat_back_addr":"172.21.15.194:6803/165633203","heartbeat_front_addr":"172.21.15.194:6802/165633203","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":72,"last_clean_end":1019,"up_from":1069,"up_thru":1011,"down_at":1020,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":3807855610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":3807855610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":3807855610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":3807855610}]},"public_addr":"172.21.15.194:6804/3807855610","cluster_addr":"172.21.15.194:6805/3807855610","heartbeat_back_addr":"172.21.15.194:6807/3807855610","heartbeat_front_addr":"172.21.15.194:6806/3807855610","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":0,"weight":0,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":524,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-29T14:57:33.928380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:57:51.616701+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":51551,"last_purged_snaps_scrub":"2024-04-29T14:57:40.164516+0000","dead_epoch":230},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:56:49.917864+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:57:44.583783+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:57:31.834159+0000","dead_epoch":457},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:57:25.012414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:57:43.580034+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:57:53.081764+0000","dead_epoch":320},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-29T14:57:33.264437+0000","dead_epoch":1019},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T14:57:41.469517+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":28165,"last_purged_snaps_scrub":"2024-04-29T14:58:02.524540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T14:58:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:05 smithi194 ceph-mon[44089]: pgmap v2829: 17 pgs: 17 active+clean; 8.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:05 smithi121 ceph-mon[44533]: pgmap v2829: 17 pgs: 17 active+clean; 8.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:06.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/700806057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:58:06.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3580860982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:06.492 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:06 smithi040 ceph-mon[39581]: from='client.28513 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:58:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/700806057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:58:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3580860982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:06 smithi194 ceph-mon[44089]: from='client.28513 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:06.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/700806057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T14:58:06.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3580860982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:06.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:06 smithi121 ceph-mon[44533]: from='client.28513 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:06.976 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:07.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:07.513 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:07.540 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:07.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:07.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:07.730 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:07.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:07.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:07 smithi040 ceph-mon[39581]: pgmap v2830: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:07.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:07 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:07.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:07 smithi040 ceph-mon[39581]: from='client.28513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:07.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:07 smithi040 ceph-mon[39581]: osdmap e1111: 12 total, 12 up, 8 in 2024-04-29T14:58:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:07.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:07 smithi121 ceph-mon[44533]: pgmap v2830: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:07.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:07 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:07.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:07 smithi121 ceph-mon[44533]: from='client.28513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:07.788 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:07 smithi121 ceph-mon[44533]: osdmap e1111: 12 total, 12 up, 8 in 2024-04-29T14:58:07.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:07 smithi194 ceph-mon[44089]: pgmap v2830: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:07 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:07 smithi194 ceph-mon[44089]: from='client.28513 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:07 smithi194 ceph-mon[44089]: osdmap e1111: 12 total, 12 up, 8 in 2024-04-29T14:58:07.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:07.943 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:08.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:08.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:08.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:08.139 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-29T14:58:08.160 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:08.220 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:08.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:08.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:08.361 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:08.574 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:08.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:08.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:08.979 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:08.983 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:08 smithi040 ceph-mon[39581]: pgmap v2832: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 904 op/s 2024-04-29T14:58:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:08.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:09.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:08 smithi194 ceph-mon[44089]: pgmap v2832: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 904 op/s 2024-04-29T14:58:09.085 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:09.113 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:09.135 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:09.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:09.143 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:08 smithi121 ceph-mon[44533]: pgmap v2832: 17 pgs: 17 active+clean; 8.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 904 op/s 2024-04-29T14:58:09.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:09.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:09.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:09.425 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:58:09.425 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:58:09.435 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:09.443 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:58:09.463 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:09.484 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:09.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:09.589 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:09.618 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:09.638 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:09.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:09.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:58:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:10.584 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T14:58:10.596 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:10.986 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:58:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:10.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:58:11.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:11.246 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:11.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:58:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:58:11.356 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:58:11.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:11.389 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:58:11.400 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:58:11.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:11.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:58:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:58:11.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:11 smithi040 ceph-mon[39581]: pgmap v2833: 17 pgs: 17 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:58:11.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3751297905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:11.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:58:11.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:11.547 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:58:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:58:11.589 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:11.596 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:58:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:58:11.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:58:11.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:58:11.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:58:11.620 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:58:11.621 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:58:11.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:11.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:11.732 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:11.760 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:11 smithi194 ceph-mon[44089]: pgmap v2833: 17 pgs: 17 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:58:11.760 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3751297905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:11.760 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:11.761 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:11.782 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:11.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:11.887 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:58:11.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:11 smithi121 ceph-mon[44533]: pgmap v2833: 17 pgs: 17 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T14:58:11.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3751297905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:11.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:11.915 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:11.944 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:58:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:11.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:58:12.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:12.082 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:12.108 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:12.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:12.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:12.256 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:12.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.487 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.488 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.488 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.488 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.488 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.488 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.488 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:12.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:13.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:13 smithi040 ceph-mon[39581]: pgmap v2834: 17 pgs: 17 active+clean; 7.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:13 smithi194 ceph-mon[44089]: pgmap v2834: 17 pgs: 17 active+clean; 7.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:13 smithi121 ceph-mon[44533]: pgmap v2834: 17 pgs: 17 active+clean; 7.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:14.813 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:14.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:14.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:14.958 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:15.111 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:15.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:15.219 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:15.249 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:15.270 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:15.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:15.374 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:58:15.404 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:15.426 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:58:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:15.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:58:15.534 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:15.561 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:15.581 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:15.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:15.686 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:15.714 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:15.737 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:15.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:15 smithi194 ceph-mon[44089]: pgmap v2835: 17 pgs: 17 active+clean; 7.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:15.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:15.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:15 smithi040 ceph-mon[39581]: pgmap v2835: 17 pgs: 17 active+clean; 7.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:15.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:15 smithi121 ceph-mon[44533]: pgmap v2835: 17 pgs: 17 active+clean; 7.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:15.886 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:15.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:15.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:16.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:16.058 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:16.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:16.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:16.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:16.260 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:16.289 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:16.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:16.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:16.436 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:16.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1177470603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:58:16.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:16 smithi040 ceph-mon[39581]: from='client.28522 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:58:16.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:16 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:16.507 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:16 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:58:16.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:58:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:16.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:16.685 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:16.713 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:16.736 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:16.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1177470603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:58:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:16 smithi194 ceph-mon[44089]: from='client.28522 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:58:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:16 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:16 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:16.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:16.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1177470603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:58:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:16 smithi121 ceph-mon[44533]: from='client.28522 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:58:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:16 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:16 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:16.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:17.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:17.095 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:17.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:17.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:17.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:17.305 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:17.347 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:17.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:17.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:17.507 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:17.547 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:17.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:17.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:17.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:17 smithi121 ceph-mon[44533]: pgmap v2836: 17 pgs: 17 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:17.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:17 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:17.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:17 smithi121 ceph-mon[44533]: from='client.28522 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:58:17.554 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:17 smithi121 ceph-mon[44533]: osdmap e1112: 12 total, 12 up, 8 in 2024-04-29T14:58:17.654 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:17.683 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:17.705 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:17.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:17.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:17 smithi194 ceph-mon[44089]: pgmap v2836: 17 pgs: 17 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:17 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:17 smithi194 ceph-mon[44089]: from='client.28522 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:58:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:17 smithi194 ceph-mon[44089]: osdmap e1112: 12 total, 12 up, 8 in 2024-04-29T14:58:17.836 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:17.859 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:17.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:17 smithi040 ceph-mon[39581]: pgmap v2836: 17 pgs: 17 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:17 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:17 smithi040 ceph-mon[39581]: from='client.28522 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:58:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:17 smithi040 ceph-mon[39581]: osdmap e1112: 12 total, 12 up, 8 in 2024-04-29T14:58:17.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:17.997 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:18.034 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:58:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:58:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:58:18.034 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:58:18.034 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:58:18.374 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:18.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:18.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:19 smithi194 ceph-mon[44089]: pgmap v2838: 17 pgs: 17 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:19.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:19 smithi040 ceph-mon[39581]: pgmap v2838: 17 pgs: 17 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:19.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:19 smithi121 ceph-mon[44533]: pgmap v2838: 17 pgs: 17 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:19.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T14:58:19.926 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:19.929 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:19.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:20.034 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:20.061 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:20.081 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:20.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:20.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:20.248 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:20.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:20.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:20.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:20.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:20.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:20.641 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:20.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:20.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:20.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:20.833 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:20.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:20.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:20.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:21.010 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:21.042 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:21.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:21.149 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:21.176 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:21.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:58:21.198 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:21.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:21.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:21.355 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:21.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:21.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:21.496 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:21.524 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:21.545 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:21.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:21.641 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:21 smithi121 ceph-mon[44533]: pgmap v2839: 17 pgs: 17 active+clean; 7.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:21.641 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:21 smithi121 ceph-mon[44533]: 9.0 scrub starts 2024-04-29T14:58:21.652 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:21.688 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:21.707 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:21.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:21.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:21.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:21 smithi194 ceph-mon[44089]: pgmap v2839: 17 pgs: 17 active+clean; 7.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:21.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:21 smithi194 ceph-mon[44089]: 9.0 scrub starts 2024-04-29T14:58:21.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:21 smithi040 ceph-mon[39581]: pgmap v2839: 17 pgs: 17 active+clean; 7.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:21.860 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:21 smithi040 ceph-mon[39581]: 9.0 scrub starts 2024-04-29T14:58:21.863 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:23 smithi194 ceph-mon[44089]: pgmap v2840: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:23 smithi194 ceph-mon[44089]: 9.0 scrub ok 2024-04-29T14:58:23.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:23 smithi040 ceph-mon[39581]: pgmap v2840: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:23 smithi040 ceph-mon[39581]: 9.0 scrub ok 2024-04-29T14:58:23.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:23 smithi121 ceph-mon[44533]: pgmap v2840: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:23.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:23 smithi121 ceph-mon[44533]: 9.0 scrub ok 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:58:24.570 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:58:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:58:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:58:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:58:24.821 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:24.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4170063231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4170063231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.826 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:24.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:24.840 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4170063231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:24.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:24.956 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:24.977 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:24.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:25.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:25.128 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:25.590 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:25.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:25.706 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:25.734 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:25.754 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:25.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:25 smithi194 ceph-mon[44089]: pgmap v2841: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:25.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:25 smithi040 ceph-mon[39581]: pgmap v2841: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:25.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:25 smithi121 ceph-mon[44533]: pgmap v2841: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:25.891 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:27.374 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:27.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:27.483 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:58:27.511 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:27.534 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:58:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:27.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:58:27.624 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:58:27.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:27 smithi040 ceph-mon[39581]: pgmap v2842: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:58:27.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:27.692 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:27.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:27.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:27 smithi194 ceph-mon[44089]: pgmap v2842: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:58:27.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:27.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:27 smithi121 ceph-mon[44533]: pgmap v2842: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T14:58:27.890 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:58:27.890 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:58:27.899 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:27.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:27.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:28.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:28.079 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:28.140 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T14:58:28.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:58:29.024 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:28 smithi040 ceph-mon[39581]: pgmap v2843: 17 pgs: 17 active+clean; 6.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1k op/s 2024-04-29T14:58:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:28 smithi194 ceph-mon[44089]: pgmap v2843: 17 pgs: 17 active+clean; 6.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1k op/s 2024-04-29T14:58:29.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:28 smithi121 ceph-mon[44533]: pgmap v2843: 17 pgs: 17 active+clean; 6.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1k op/s 2024-04-29T14:58:29.547 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T14:58:29.549 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:29.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:29.571 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:29.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:29.801 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:29.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:29.864 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:58:29.906 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:29.947 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:31.500 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:58:31.732 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:31 smithi194 ceph-mon[44089]: pgmap v2844: 17 pgs: 17 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:31.732 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1820872515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:31.732 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:31 smithi194 ceph-mon[44089]: from='client.38255 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:31 smithi040 ceph-mon[39581]: pgmap v2844: 17 pgs: 17 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1820872515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:31 smithi040 ceph-mon[39581]: from='client.38255 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:31.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:31 smithi121 ceph-mon[44533]: pgmap v2844: 17 pgs: 17 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:31.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1820872515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:31.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:31 smithi121 ceph-mon[44533]: from='client.38255 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:32.318 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:32.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:32.342 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:58:32.342 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:58:32.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:32.478 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:32.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:32 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:58:32.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:32 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T14:58:32.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:32 smithi040 ceph-mon[39581]: from='client.38255 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:32.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:32 smithi040 ceph-mon[39581]: osdmap e1113: 12 total, 12 up, 8 in 2024-04-29T14:58:32.734 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:32 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:58:32.734 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:32 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T14:58:32.734 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:32 smithi194 ceph-mon[44089]: from='client.38255 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:32.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:32 smithi194 ceph-mon[44089]: osdmap e1113: 12 total, 12 up, 8 in 2024-04-29T14:58:32.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:32 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:58:32.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:32 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T14:58:32.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:32 smithi121 ceph-mon[44533]: from='client.38255 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:32.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:32 smithi121 ceph-mon[44533]: osdmap e1113: 12 total, 12 up, 8 in 2024-04-29T14:58:33.037 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:33.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:33.059 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:33.141 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T14:58:33.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:58:33.176 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:33.282 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:58:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:33.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:58:33.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:33.420 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:33.459 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:58:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:33.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:33.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:33.618 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:33 smithi040 ceph-mon[39581]: pgmap v2846: 17 pgs: 17 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:33.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:33.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:33.779 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:33 smithi194 ceph-mon[44089]: pgmap v2846: 17 pgs: 17 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:33.807 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:33.834 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:33.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:33.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:33 smithi121 ceph-mon[44533]: pgmap v2846: 17 pgs: 17 active+clean; 6.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-29T14:58:33.941 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:33.969 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:33.993 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:33.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:34.098 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:34.128 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:34.188 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:58:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:58:35.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:58:35.210 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:35.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:35.314 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:35.353 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:35.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2846,"stamp":"2024-04-29T14:58:32.425161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7021855288,"num_objects":107139,"num_object_clones":0,"num_object_copies":214278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107139,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":213052,"num_write_kb":10243131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161110,"ondisk_log_size":161110,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20479416,"kb_used_data":13731000,"kb_used_omap":664,"kb_used_meta":6747431,"kb_avail":729481800,"statfs":{"total":767960285184,"available":746989363200,"internally_reserved":0,"allocated":14060544000,"data_stored":14057252580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":680854,"internal_metadata":6909369450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-744030208,"num_objects":-11353,"num_object_clones":0,"num_object_copies":-22706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003227"},"pg_stats":[{"pgid":"9.f","version":"1112'13240","reported_seq":13449,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:29.093046+0000","last_change":"2024-04-29T14:57:37.360585+0000","last_active":"2024-04-29T14:58:29.093046+0000","last_peered":"2024-04-29T14:58:29.093046+0000","last_clean":"2024-04-29T14:58:29.093046+0000","last_became_active":"2024-04-29T14:55:59.715306+0000","last_became_peered":"2024-04-29T14:55:59.715306+0000","last_unstale":"2024-04-29T14:58:29.093046+0000","last_undegraded":"2024-04-29T14:58:29.093046+0000","last_fullsized":"2024-04-29T14:58:29.093046+0000","mapping_epoch":1065,"log_start":"1076'3200","ondisk_log_start":"1076'3200","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1100'9730","last_scrub_stamp":"2024-04-29T14:57:37.360464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:37.360464+0000","objects_scrubbed":9655,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.217197+0000","scrub_duration":2275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441712640,"num_objects":6740,"num_object_clones":0,"num_object_copies":13480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13240,"num_write_kb":639360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"1112'13485","reported_seq":13689,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:31.130009+0000","last_change":"2024-04-29T14:57:28.787035+0000","last_active":"2024-04-29T14:58:31.130009+0000","last_peered":"2024-04-29T14:58:31.130009+0000","last_clean":"2024-04-29T14:58:31.130009+0000","last_became_active":"2024-04-29T14:55:59.715887+0000","last_became_peered":"2024-04-29T14:55:59.715887+0000","last_unstale":"2024-04-29T14:58:31.130009+0000","last_undegraded":"2024-04-29T14:58:31.130009+0000","last_fullsized":"2024-04-29T14:58:31.130009+0000","mapping_epoch":1065,"log_start":"1076'3400","ondisk_log_start":"1076'3400","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9237","last_scrub_stamp":"2024-04-29T14:57:28.786814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:28.786814+0000","objects_scrubbed":9187,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:58.780518+0000","scrub_duration":2102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434700288,"num_objects":6633,"num_object_clones":0,"num_object_copies":13266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13485,"num_write_kb":643776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"1112'13353","reported_seq":13557,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:31.152590+0000","last_change":"2024-04-29T14:57:31.565364+0000","last_active":"2024-04-29T14:58:31.152590+0000","last_peered":"2024-04-29T14:58:31.152590+0000","last_clean":"2024-04-29T14:58:31.152590+0000","last_became_active":"2024-04-29T14:55:59.715955+0000","last_became_peered":"2024-04-29T14:55:59.715955+0000","last_unstale":"2024-04-29T14:58:31.152590+0000","last_undegraded":"2024-04-29T14:58:31.152590+0000","last_fullsized":"2024-04-29T14:58:31.152590+0000","mapping_epoch":1065,"log_start":"1076'3300","ondisk_log_start":"1076'3300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9317","last_scrub_stamp":"2024-04-29T14:57:31.565106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:31.565106+0000","objects_scrubbed":9272,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.564312+0000","scrub_duration":1916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434241560,"num_objects":6627,"num_object_clones":0,"num_object_copies":13254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13353,"num_write_kb":639297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"1112'12961","reported_seq":13163,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:28.612256+0000","last_change":"2024-04-29T14:57:30.415164+0000","last_active":"2024-04-29T14:58:28.612256+0000","last_peered":"2024-04-29T14:58:28.612256+0000","last_clean":"2024-04-29T14:58:28.612256+0000","last_became_active":"2024-04-29T14:55:59.711280+0000","last_became_peered":"2024-04-29T14:55:59.711280+0000","last_unstale":"2024-04-29T14:58:28.612256+0000","last_undegraded":"2024-04-29T14:58:28.612256+0000","last_fullsized":"2024-04-29T14:58:28.612256+0000","mapping_epoch":1065,"log_start":"1076'2900","ondisk_log_start":"1076'2900","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9070","last_scrub_stamp":"2024-04-29T14:57:30.415030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:30.415030+0000","objects_scrubbed":8993,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:33.371993+0000","scrub_duration":2038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431423488,"num_objects":6583,"num_object_clones":0,"num_object_copies":13166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12961,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"1112'13440","reported_seq":13642,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:30.815970+0000","last_change":"2024-04-29T14:57:26.289644+0000","last_active":"2024-04-29T14:58:30.815970+0000","last_peered":"2024-04-29T14:58:30.815970+0000","last_clean":"2024-04-29T14:58:30.815970+0000","last_became_active":"2024-04-29T14:55:59.713356+0000","last_became_peered":"2024-04-29T14:55:59.713356+0000","last_unstale":"2024-04-29T14:58:30.815970+0000","last_undegraded":"2024-04-29T14:58:30.815970+0000","last_fullsized":"2024-04-29T14:58:30.815970+0000","mapping_epoch":1065,"log_start":"1076'3400","ondisk_log_start":"1076'3400","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1096'9086","last_scrub_stamp":"2024-04-29T14:57:26.289482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:26.289482+0000","objects_scrubbed":9033,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.542697+0000","scrub_duration":1170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440664064,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13440,"num_write_kb":645248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"1112'13270","reported_seq":13474,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:29.083118+0000","last_change":"2024-04-29T14:57:29.553496+0000","last_active":"2024-04-29T14:58:29.083118+0000","last_peered":"2024-04-29T14:58:29.083118+0000","last_clean":"2024-04-29T14:58:29.083118+0000","last_became_active":"2024-04-29T14:55:59.715266+0000","last_became_peered":"2024-04-29T14:55:59.715266+0000","last_unstale":"2024-04-29T14:58:29.083118+0000","last_undegraded":"2024-04-29T14:58:29.083118+0000","last_fullsized":"2024-04-29T14:58:29.083118+0000","mapping_epoch":1065,"log_start":"1076'3200","ondisk_log_start":"1076'3200","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9276","last_scrub_stamp":"2024-04-29T14:57:29.553192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:29.553192+0000","objects_scrubbed":9211,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:59.188755+0000","scrub_duration":2353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444989440,"num_objects":6790,"num_object_clones":0,"num_object_copies":13580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13270,"num_write_kb":641920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"1112'13545","reported_seq":13751,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:29.788266+0000","last_change":"2024-04-29T14:57:29.673784+0000","last_active":"2024-04-29T14:58:29.788266+0000","last_peered":"2024-04-29T14:58:29.788266+0000","last_clean":"2024-04-29T14:58:29.788266+0000","last_became_active":"2024-04-29T14:55:59.713461+0000","last_became_peered":"2024-04-29T14:55:59.713461+0000","last_unstale":"2024-04-29T14:58:29.788266+0000","last_undegraded":"2024-04-29T14:58:29.788266+0000","last_fullsized":"2024-04-29T14:58:29.788266+0000","mapping_epoch":1065,"log_start":"1076'3500","ondisk_log_start":"1076'3500","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9466","last_scrub_stamp":"2024-04-29T14:57:29.673626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:29.673626+0000","objects_scrubbed":9432,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:47.159905+0000","scrub_duration":1388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444530688,"num_objects":6783,"num_object_clones":0,"num_object_copies":13566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13545,"num_write_kb":650496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"1112'13053","reported_seq":13256,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:29.776368+0000","last_change":"2024-04-29T14:57:31.535038+0000","last_active":"2024-04-29T14:58:29.776368+0000","last_peered":"2024-04-29T14:58:29.776368+0000","last_clean":"2024-04-29T14:58:29.776368+0000","last_became_active":"2024-04-29T14:55:59.716261+0000","last_became_peered":"2024-04-29T14:55:59.716261+0000","last_unstale":"2024-04-29T14:58:29.776368+0000","last_undegraded":"2024-04-29T14:58:29.776368+0000","last_fullsized":"2024-04-29T14:58:29.776368+0000","mapping_epoch":1065,"log_start":"1076'3000","ondisk_log_start":"1076'3000","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9138","last_scrub_stamp":"2024-04-29T14:57:31.534865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:31.534865+0000","objects_scrubbed":9114,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:39.925643+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433520640,"num_objects":6615,"num_object_clones":0,"num_object_copies":13230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13053,"num_write_kb":629376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"1112'13064","reported_seq":13271,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:29.082258+0000","last_change":"2024-04-29T14:57:38.146397+0000","last_active":"2024-04-29T14:58:29.082258+0000","last_peered":"2024-04-29T14:58:29.082258+0000","last_clean":"2024-04-29T14:58:29.082258+0000","last_became_active":"2024-04-29T14:55:59.713183+0000","last_became_peered":"2024-04-29T14:55:59.713183+0000","last_unstale":"2024-04-29T14:58:29.082258+0000","last_undegraded":"2024-04-29T14:58:29.082258+0000","last_fullsized":"2024-04-29T14:58:29.082258+0000","mapping_epoch":1065,"log_start":"1076'3000","ondisk_log_start":"1076'3000","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1100'9635","last_scrub_stamp":"2024-04-29T14:57:38.146145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:38.146145+0000","objects_scrubbed":9573,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:47.250208+0000","scrub_duration":2076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435945472,"num_objects":6652,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13064,"num_write_kb":630912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"1112'13400","reported_seq":13603,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:29.080766+0000","last_change":"2024-04-29T14:57:29.030253+0000","last_active":"2024-04-29T14:58:29.080766+0000","last_peered":"2024-04-29T14:58:29.080766+0000","last_clean":"2024-04-29T14:58:29.080766+0000","last_became_active":"2024-04-29T14:55:59.714571+0000","last_became_peered":"2024-04-29T14:55:59.714571+0000","last_unstale":"2024-04-29T14:58:29.080766+0000","last_undegraded":"2024-04-29T14:58:29.080766+0000","last_fullsized":"2024-04-29T14:58:29.080766+0000","mapping_epoch":1065,"log_start":"1076'3300","ondisk_log_start":"1076'3300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9262","last_scrub_stamp":"2024-04-29T14:57:29.030022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:29.030022+0000","objects_scrubbed":9178,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:47.154711+0000","scrub_duration":2818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440401920,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13400,"num_write_kb":643840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"1113'13458","reported_seq":13668,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-04-29T14:58:32.134040+0000","last_change":"2024-04-29T14:57:33.716716+0000","last_active":"2024-04-29T14:58:32.134040+0000","last_peered":"2024-04-29T14:58:32.134040+0000","last_clean":"2024-04-29T14:58:32.134040+0000","last_became_active":"2024-04-29T14:55:59.712951+0000","last_became_peered":"2024-04-29T14:55:59.712951+0000","last_unstale":"2024-04-29T14:58:32.134040+0000","last_undegraded":"2024-04-29T14:58:32.134040+0000","last_fullsized":"2024-04-29T14:58:32.134040+0000","mapping_epoch":1065,"log_start":"1076'3400","ondisk_log_start":"1076'3400","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'9531","last_scrub_stamp":"2024-04-29T14:57:33.716542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:33.716542+0000","objects_scrubbed":9458,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:36.398559+0000","scrub_duration":2138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432144384,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13458,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"1112'13172","reported_seq":13376,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:28.602335+0000","last_change":"2024-04-29T14:57:31.345869+0000","last_active":"2024-04-29T14:58:28.602335+0000","last_peered":"2024-04-29T14:58:28.602335+0000","last_clean":"2024-04-29T14:58:28.602335+0000","last_became_active":"2024-04-29T14:55:59.713602+0000","last_became_peered":"2024-04-29T14:55:59.713602+0000","last_unstale":"2024-04-29T14:58:28.602335+0000","last_undegraded":"2024-04-29T14:58:28.602335+0000","last_fullsized":"2024-04-29T14:58:28.602335+0000","mapping_epoch":1065,"log_start":"1076'3100","ondisk_log_start":"1076'3100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9339","last_scrub_stamp":"2024-04-29T14:57:31.345750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:31.345750+0000","objects_scrubbed":9280,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:37.335068+0000","scrub_duration":1937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446693376,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13172,"num_write_kb":639616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"1112'13164","reported_seq":13365,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:28.627373+0000","last_change":"2024-04-29T14:57:26.655236+0000","last_active":"2024-04-29T14:58:28.627373+0000","last_peered":"2024-04-29T14:58:28.627373+0000","last_clean":"2024-04-29T14:58:28.627373+0000","last_became_active":"2024-04-29T14:55:59.712436+0000","last_became_peered":"2024-04-29T14:55:59.712436+0000","last_unstale":"2024-04-29T14:58:28.627373+0000","last_undegraded":"2024-04-29T14:58:28.627373+0000","last_fullsized":"2024-04-29T14:58:28.627373+0000","mapping_epoch":1065,"log_start":"1076'3100","ondisk_log_start":"1076'3100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1096'8996","last_scrub_stamp":"2024-04-29T14:57:26.655091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:26.655091+0000","objects_scrubbed":8946,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:44.764007+0000","scrub_duration":1321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440008704,"num_objects":6714,"num_object_clones":0,"num_object_copies":13428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":636096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"1112'13380","reported_seq":13587,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:28.618240+0000","last_change":"2024-04-29T14:57:34.381161+0000","last_active":"2024-04-29T14:58:28.618240+0000","last_peered":"2024-04-29T14:58:28.618240+0000","last_clean":"2024-04-29T14:58:28.618240+0000","last_became_active":"2024-04-29T14:55:59.761514+0000","last_became_peered":"2024-04-29T14:55:59.761514+0000","last_unstale":"2024-04-29T14:58:28.618240+0000","last_undegraded":"2024-04-29T14:58:28.618240+0000","last_fullsized":"2024-04-29T14:58:28.618240+0000","mapping_epoch":1065,"log_start":"1076'3300","ondisk_log_start":"1076'3300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'9582","last_scrub_stamp":"2024-04-29T14:57:34.381039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:34.381039+0000","objects_scrubbed":9521,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.357505+0000","scrub_duration":1949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13380,"num_write_kb":643072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"1113'13571","reported_seq":13839,"reported_epoch":1113,"state":"active+clean","last_fresh":"2024-04-29T14:58:32.138563+0000","last_change":"2024-04-29T14:58:22.484274+0000","last_active":"2024-04-29T14:58:32.138563+0000","last_peered":"2024-04-29T14:58:32.138563+0000","last_clean":"2024-04-29T14:58:32.138563+0000","last_became_active":"2024-04-29T14:55:59.760008+0000","last_became_peered":"2024-04-29T14:55:59.760008+0000","last_unstale":"2024-04-29T14:58:32.138563+0000","last_undegraded":"2024-04-29T14:58:32.138563+0000","last_fullsized":"2024-04-29T14:58:32.138563+0000","mapping_epoch":1065,"log_start":"1076'3500","ondisk_log_start":"1076'3500","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'12866","last_scrub_stamp":"2024-04-29T14:58:22.484089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:58:22.484089+0000","objects_scrubbed":7299,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:40.606858+0000","scrub_duration":1410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429588480,"num_objects":6555,"num_object_clones":0,"num_object_copies":13110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13571,"num_write_kb":644032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"1112'13315","reported_seq":13526,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:29.103398+0000","last_change":"2024-04-29T14:57:40.651846+0000","last_active":"2024-04-29T14:58:29.103398+0000","last_peered":"2024-04-29T14:58:29.103398+0000","last_clean":"2024-04-29T14:58:29.103398+0000","last_became_active":"2024-04-29T14:55:59.714624+0000","last_became_peered":"2024-04-29T14:55:59.714624+0000","last_unstale":"2024-04-29T14:58:29.103398+0000","last_undegraded":"2024-04-29T14:58:29.103398+0000","last_fullsized":"2024-04-29T14:58:29.103398+0000","mapping_epoch":1065,"log_start":"1076'3300","ondisk_log_start":"1076'3300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1101'10023","last_scrub_stamp":"2024-04-29T14:57:40.651515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:40.651515+0000","objects_scrubbed":9940,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:51.580017+0000","scrub_duration":2555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450560000,"num_objects":6875,"num_object_clones":0,"num_object_copies":13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13315,"num_write_kb":646080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2265,"reported_epoch":1112,"state":"active+clean","last_fresh":"2024-04-29T14:58:16.539946+0000","last_change":"2024-04-29T14:57:43.962180+0000","last_active":"2024-04-29T14:58:16.539946+0000","last_peered":"2024-04-29T14:58:16.539946+0000","last_clean":"2024-04-29T14:58:16.539946+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:58:16.539946+0000","last_undegraded":"2024-04-29T14:58:16.539946+0000","last_fullsized":"2024-04-29T14:58:16.539946+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:57:43.962100+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:57:43.962100+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:52.412069+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7021264920,"num_objects":107137,"num_object_clones":0,"num_object_copies":214274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212871,"num_write_kb":10240193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13961601024,"data_stored":13961592880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160971,"ondisk_log_size":160971,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122668,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3397308,"kb_used_data":2152756,"kb_used_omap":1,"kb_used_meta":1244542,"kb_avail":90347844,"statfs":{"total":95995035648,"available":92516192256,"internally_reserved":0,"allocated":2204422144,"data_stored":2204012605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1274411456},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":4591320039454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423248,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681340,"kb_used_data":432244,"kb_used_omap":4,"kb_used_meta":249083,"kb_avail":93063812,"statfs":{"total":95995035648,"available":95297343488,"internally_reserved":0,"allocated":442617856,"data_stored":442208337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":255061696},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234623,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3138056,"kb_used_data":2123968,"kb_used_omap":1,"kb_used_meta":1014078,"kb_avail":90607096,"statfs":{"total":95995035648,"available":92781666304,"internally_reserved":0,"allocated":2174943232,"data_stored":2174526089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1038416330},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528964,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1921220,"kb_used_data":857148,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":91823932,"statfs":{"total":95995035648,"available":94027706368,"internally_reserved":0,"allocated":877719552,"data_stored":877306481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1089599942},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3285888,"kb_used_data":2542520,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":90459264,"statfs":{"total":95995035648,"available":92630286336,"internally_reserved":0,"allocated":2603540480,"data_stored":2603126889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359739043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183061,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2537532,"kb_used_data":1721972,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91207620,"statfs":{"total":95995035648,"available":93396602880,"internally_reserved":0,"allocated":1763299328,"data_stored":1762889809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":835123656},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986827,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1940604,"kb_used_data":836276,"kb_used_omap":1,"kb_used_meta":1104318,"kb_avail":91804548,"statfs":{"total":95995035648,"available":94007857152,"internally_reserved":0,"allocated":856346624,"data_stored":855937105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1130822085},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653387,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3577468,"kb_used_data":3064116,"kb_used_omap":651,"kb_used_meta":512436,"kb_avail":90167684,"statfs":{"total":95995035648,"available":92331708416,"internally_reserved":0,"allocated":3137654784,"data_stored":3137245265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":524735240},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2572488704,"data_stored":2572484632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1758199808,"data_stored":1758199808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":854327296,"data_stored":854327296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3104243712,"data_stored":3104243712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":868679680,"data_stored":868679680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2167738368,"data_stored":2167734296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":440598528,"data_stored":440598528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2195324928,"data_stored":2195324928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:58:35.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T14:58:35.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:35.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:35 smithi194 ceph-mon[44089]: from='client.28537 v1:172.21.15.40:0/2292962855' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:58:35.514 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:35 smithi194 ceph-mon[44089]: pgmap v2847: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:35.547 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:35.565 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:58:35.565 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:58:35.584 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:35.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:35.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:35 smithi040 ceph-mon[39581]: from='client.28537 v1:172.21.15.40:0/2292962855' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:58:35.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:35 smithi040 ceph-mon[39581]: pgmap v2847: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:35.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:35.742 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:35.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:35.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:35 smithi121 ceph-mon[44533]: from='client.28537 v1:172.21.15.40:0/2292962855' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:58:35.787 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:35 smithi121 ceph-mon[44533]: pgmap v2847: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:35.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:35.890 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:35.959 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:35.963 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:35.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:36.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:36.122 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:36.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:36.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:36.275 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:36.323 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:36.339 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T14:58:36.479 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:36.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:36.588 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:36.620 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:36.643 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:58:36.647 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:36.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:36.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:36.830 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:36.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:36.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:36.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:36.922 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:58:36.947 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:58:36.972 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:36.981 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:58:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:37.019 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:58:37.059 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:58:37.095 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:58:37.142 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:58:37.153 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:58:37.178 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:58:37.209 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:58:37.241 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:58:37.271 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:58:37.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:58:37.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:58:37.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:58:37.277 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:58:37.278 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: pgmap v2848: 17 pgs: 17 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/358993377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: pgmap v2848: 17 pgs: 17 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/358993377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.827 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: pgmap v2848: 17 pgs: 17 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/358993377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:37.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.473 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:38.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:38.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:38.638 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:38.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:38.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:38.799 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:58:38.799 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T14:58:38.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T14:58:38.799 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T14:58:38.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.799 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162440 8.a 8:559374cd:::benchmark_data_smithi194_186548_object33882:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.017140+0000", 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 261.77504790099999, 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21948063300000001, 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:58:38.800 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162440 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017140+0000", 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017140+0000", 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017142+0000", 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.08e-06 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017146+0000", 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5859999999999998e-06 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017157+0000", 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0755e-05 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.017165+0000", 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9310000000000008e-06 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.022286+0000", 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051215369999999998 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.022399+0000", 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011301799999999999 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.022498+0000", 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8815999999999997e-05 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.026739+0000", 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042403019999999996 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235303+0000", 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.208564257 2024-04-29T14:58:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235360+0000", 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7552000000000001e-05 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236620+0000", 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001259799 2024-04-29T14:58:38.807 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162470 8.c 8:3138e6fd:::benchmark_data_smithi194_186548_object33912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.039232+0000", 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 261.75295556200001, 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.197422704, 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.808 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162470 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039232+0000", 2024-04-29T14:58:38.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039232+0000", 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039234+0000", 2024-04-29T14:58:38.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5150000000000001e-06 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039241+0000", 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8410000000000002e-06 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039255+0000", 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.375e-05 2024-04-29T14:58:38.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039264+0000", 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2410000000000001e-06 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039324+0000", 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9772999999999998e-05 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039465+0000", 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014109300000000001 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.039594+0000", 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012870500000000001 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.044220+0000", 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046258159999999996 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235552+0000", 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.191332421 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.235617+0000", 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4801999999999994e-05 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236655+0000", 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010377470000000001 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162482 8.a 8:54bedb5a:::benchmark_data_smithi194_186548_object33924:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.050747+0000", 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 261.74144090999999, 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.186855457, 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:58:38.816 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162482 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050747+0000", 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050747+0000", 2024-04-29T14:58:38.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050749+0000", 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9390000000000002e-06 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050758+0000", 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5900000000000008e-06 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050774+0000", 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6143000000000001e-05 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050782+0000", 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0090000000000007e-06 2024-04-29T14:58:38.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050833+0000", 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0865000000000002e-05 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.050952+0000", 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118418 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.051037+0000", 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5702e-05 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.056976+0000", 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059383190000000001 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236650+0000", 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17967455800000001 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.236691+0000", 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0961000000000003e-05 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.237602+0000", 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00091095300000000004 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162492 8.a 8:5a0e10d2:::benchmark_data_smithi194_186548_object33934:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.059901+0000", 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 261.73228627200001, 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18326977699999999, 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.823 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162492 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059901+0000", 2024-04-29T14:58:38.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059901+0000", 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059904+0000", 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3580000000000001e-06 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059910+0000", 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5860000000000001e-06 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059925+0000", 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4623999999999999e-05 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059934+0000", 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9970000000000004e-06 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.059996+0000", 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2577999999999996e-05 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.060132+0000", 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000135563 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.060268+0000", 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013622299999999999 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.065270+0000", 2024-04-29T14:58:38.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050021170000000004 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.242031+0000", 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.17676051300000001 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.242066+0000", 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4816e-05 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.243171+0000", 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011054019999999999 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:162518 8.c 8:304526c8:::benchmark_data_smithi194_186548_object33960:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1045)", 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:54:17.080553+0000", 2024-04-29T14:58:38.830 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 261.71163487299998, 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16836797100000001, 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 162518 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080553+0000", 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080553+0000", 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080554+0000", 2024-04-29T14:58:38.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.517e-06 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080559+0000", 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8239999999999999e-06 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080569+0000", 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0045e-05 2024-04-29T14:58:38.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.080576+0000", 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1300000000000003e-06 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.084319+0000", 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003743269 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.084461+0000", 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014201799999999999 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 3", 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.084597+0000", 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013583900000000001 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.088351+0000", 2024-04-29T14:58:38.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037542169999999998 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.248256+0000", 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.159904088 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.248293+0000", 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7840999999999999e-05 2024-04-29T14:58:38.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:54:17.248921+0000", 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062718300000000002 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:172029 9.b 9:daf5c37c:::benchmark_data_smithi194_196924_object12023:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1107)", 2024-04-29T14:58:38.837 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:53.201616+0000", 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.590571476000001, 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021795939, 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172029 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.838 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201616+0000", 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201616+0000", 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201617+0000", 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.502e-06 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201623+0000", 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4419999999999997e-06 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201633+0000", 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0339e-05 2024-04-29T14:58:38.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201639+0000", 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.139e-06 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.208839+0000", 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0071991360000000001 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.841 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.208933+0000", 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4938999999999993e-05 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.209016+0000", 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2107999999999994e-05 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.218256+0000", 2024-04-29T14:58:38.842 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0092408539999999997 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.223303+0000", 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050461209999999998 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.223360+0000", 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7411000000000003e-05 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.843 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.223412+0000", 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1947999999999998e-05 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:172030 9.b 9:d4a845b9:::benchmark_data_smithi194_196924_object12024:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1107)", 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:53.201657+0000", 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.590530022000003, 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022506298000000001, 2024-04-29T14:58:38.844 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 172030 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201657+0000", 2024-04-29T14:58:38.845 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201657+0000", 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201658+0000", 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6599999999999995e-07 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.846 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201661+0000", 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4490000000000002e-06 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201666+0000", 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0810000000000001e-06 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.201669+0000", 2024-04-29T14:58:38.847 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7000000000000002e-06 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.213543+0000", 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011873083 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.213659+0000", 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116514 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.848 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.213755+0000", 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6360999999999994e-05 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.223250+0000", 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094941910000000008 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.224066+0000", 2024-04-29T14:58:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081673800000000001 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.224122+0000", 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5198999999999998e-05 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:53.224164+0000", 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2215999999999997e-05 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.850 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:173871 9.b 9:dacacd7c:::benchmark_data_smithi194_196924_object13865:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1108)", 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:54.992346+0000", 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 43.799841565999998, 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.058206252999999999, 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.851 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173871 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.992346+0000", 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.992346+0000", 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.852 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.992348+0000", 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.933e-06 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.992354+0000", 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7919999999999998e-06 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.853 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.992363+0000", 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0950000000000002e-06 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.992368+0000", 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7579999999999998e-06 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.998434+0000", 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060652010000000001 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.854 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.998540+0000", 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106057 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:54.999163+0000", 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062303599999999999 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.855 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.003521+0000", 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043577950000000002 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.050468+0000", 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.046946981999999998 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.050514+0000", 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6119999999999999e-05 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.856 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.050552+0000", 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8483999999999999e-05 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:173908 9.b 9:d74b1b9e:::benchmark_data_smithi194_196924_object13902:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1108)", 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:55.032758+0000", 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 43.759429036, 2024-04-29T14:58:38.857 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021259191, 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173908 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.858 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032758+0000", 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032758+0000", 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032760+0000", 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8500000000000001e-06 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.859 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032765+0000", 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1070000000000004e-06 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032775+0000", 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.499e-06 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032781+0000", 2024-04-29T14:58:38.860 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9340000000000003e-06 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032819+0000", 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7898000000000002e-05 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032894+0000", 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5121000000000001e-05 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.861 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.032997+0000", 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000102853 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.035823+0000", 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028261789999999998 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.053954+0000", 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018131540000000002 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.862 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.053989+0000", 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4511000000000002e-05 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.054018+0000", 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8699000000000001e-05 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.863 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:173917 9.b 9:d7778385:::benchmark_data_smithi194_196924_object13911:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1108)", 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:55.042304+0000", 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 43.749883631000003, 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021304146, 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173917 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042304+0000", 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042304+0000", 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.865 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042307+0000", 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8789999999999999e-06 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042312+0000", 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.733e-06 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042324+0000", 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1999e-05 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.866 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042333+0000", 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2819999999999996e-06 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042389+0000", 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5800000000000001e-05 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042496+0000", 2024-04-29T14:58:38.867 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107923 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042571+0000", 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4461000000000004e-05 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.050567+0000", 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079964579999999997 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.868 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.063501+0000", 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01293332 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.063558+0000", 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7661000000000002e-05 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.063608+0000", 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9629999999999997e-05 2024-04-29T14:58:38.869 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:173918 9.b 9:ddf6d511:::benchmark_data_smithi194_196924_object13912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1108)", 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:55.042936+0000", 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 43.749251229999999, 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020831342999999999, 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.870 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 173918 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042936+0000", 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.871 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042936+0000", 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042938+0000", 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.491e-06 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042943+0000", 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0590000000000002e-06 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.872 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042953+0000", 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0671e-05 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.042961+0000", 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2250000000000003e-06 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.046297+0000", 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0033362050000000001 2024-04-29T14:58:38.873 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.046422+0000", 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012464900000000001 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.046511+0000", 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9581999999999997e-05 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.874 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.050572+0000", 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004060854 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.063674+0000", 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013102244000000001 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.063728+0000", 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3868e-05 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.875 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:55.063768+0000", 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9495e-05 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:193572 9.b 9:de9db945:::benchmark_data_smithi194_196924_object33566:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1111)", 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:12.724989+0000", 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.067197974999999, 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021109921, 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.876 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 193572 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.724989+0000", 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.724989+0000", 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.724991+0000", 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3230000000000001e-06 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.724996+0000", 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7210000000000002e-06 2024-04-29T14:58:38.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725007+0000", 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1208e-05 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725014+0000", 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7419999999999997e-06 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.734550+0000", 2024-04-29T14:58:38.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009535488 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.734676+0000", 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126332 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.734805+0000", 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000128551 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.745998+0000", 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011193244999999999 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.746007+0000", 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5469999999999997e-06 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.746053+0000", 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6749000000000003e-05 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.881 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.746099+0000", 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6014999999999999e-05 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:193574 9.b 9:de93cd0a:::benchmark_data_smithi194_196924_object33568:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1111)", 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:12.725205+0000", 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.066982204999999, 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.024646649, 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.882 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 193574 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725205+0000", 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725205+0000", 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725207+0000", 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.516e-06 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725212+0000", 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2739999999999998e-06 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725225+0000", 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.279e-05 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.725233+0000", 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7859999999999997e-06 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.738209+0000", 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012976524999999999 2024-04-29T14:58:38.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.738315+0000", 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106028 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.738434+0000", 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118528 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.886 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.746142+0000", 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077084320000000003 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.749752+0000", 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036103099999999998 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.749805+0000", 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2759000000000001e-05 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:12.749852+0000", 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6700999999999999e-05 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:202913 9.b 9:d9183045:::benchmark_data_smithi194_196924_object42907:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:21.000239+0000", 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.791948325, 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021473721000000001, 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.888 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 202913 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000239+0000", 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000239+0000", 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000241+0000", 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5579999999999999e-06 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000245+0000", 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7369999999999997e-06 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000255+0000", 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7100000000000002e-06 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000262+0000", 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9249999999999998e-06 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.007565+0000", 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073029929999999998 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.007654+0000", 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8947000000000001e-05 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.007766+0000", 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000111535 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.011450+0000", 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036849420000000001 2024-04-29T14:58:38.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021606+0000", 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010155637 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021665+0000", 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8464999999999997e-05 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021713+0000", 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8272000000000001e-05 2024-04-29T14:58:38.893 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:202914 9.b 9:d34f870a:::benchmark_data_smithi194_196924_object42908:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:21.000341+0000", 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.791846592999999, 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021530823000000001, 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.894 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 202914 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000341+0000", 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000341+0000", 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.895 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000342+0000", 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000348+0000", 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3179999999999997e-06 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000358+0000", 2024-04-29T14:58:38.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0644e-05 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000366+0000", 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9310000000000008e-06 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.011466+0000", 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011100133 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.011572+0000", 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010601499999999999 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.011700+0000", 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012778400000000001 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021777+0000", 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010076536000000001 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021789+0000", 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2619999999999999e-05 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021833+0000", 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3464000000000003e-05 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021872+0000", 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8856000000000002e-05 2024-04-29T14:58:38.899 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:202915 9.b 9:d0224501:::benchmark_data_smithi194_196924_object42909:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:21.000380+0000", 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.791807516999999, 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025150183999999999, 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.900 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 202915 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000380+0000", 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000380+0000", 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000381+0000", 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6400000000000001e-07 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000384+0000", 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4120000000000001e-06 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000389+0000", 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3120000000000001e-06 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.000395+0000", 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2889999999999997e-06 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.015069+0000", 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014674662999999999 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.015149+0000", 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0031999999999995e-05 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.015219+0000", 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9096000000000004e-05 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.021879+0000", 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0066606369999999996 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.025463+0000", 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035839079999999998 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.025499+0000", 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6205000000000003e-05 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.025530+0000", 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0865999999999999e-05 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:203651 9.b 9:d03a030c:::benchmark_data_smithi194_196924_object43645:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:21.652730+0000", 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.139457085, 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021010694, 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 203651 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.906 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.652730+0000", 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.652730+0000", 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.652732+0000", 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.305e-06 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.652737+0000", 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.04e-06 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.652748+0000", 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0832000000000001e-05 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.652755+0000", 2024-04-29T14:58:38.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1949999999999997e-06 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.656200+0000", 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034456159999999999 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.656309+0000", 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108501 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.656384+0000", 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4773999999999999e-05 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.664497+0000", 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0081138070000000007 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.673624+0000", 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009126136 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.673686+0000", 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2665999999999999e-05 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:21.673741+0000", 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4821999999999998e-05 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:206182 9.b 9:d8dc7b78:::benchmark_data_smithi194_196924_object46176:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:23.969070+0000", 2024-04-29T14:58:38.911 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.823117075000001, 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022384607000000001, 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 206182 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.969070+0000", 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.969070+0000", 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.969071+0000", 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8800000000000002e-07 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.969074+0000", 2024-04-29T14:58:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8700000000000001e-06 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.969079+0000", 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3190000000000002e-06 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.969084+0000", 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1760000000000001e-06 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.978805+0000", 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0097204609999999997 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.978928+0000", 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012262800000000001 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.979043+0000", 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011518000000000001 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.991346+0000", 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012303657000000001 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.991360+0000", 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3895e-05 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.991411+0000", 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0961000000000002e-05 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:23.991455+0000", 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3772e-05 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:222714 9.b 9:d4f32ebd:::benchmark_data_smithi194_196924_object62708:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1113)", 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:38.463373+0000", 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.32881485999999999, 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022432404999999999, 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.917 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 222714 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.463373+0000", 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.463373+0000", 2024-04-29T14:58:38.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.463375+0000", 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0190000000000001e-06 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.463381+0000", 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3239999999999999e-06 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.463394+0000", 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3550000000000001e-05 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.463405+0000", 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0812000000000001e-05 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.472434+0000", 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090290260000000008 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.472562+0000", 2024-04-29T14:58:38.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012812199999999999 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.472652+0000", 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9759000000000006e-05 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.480672+0000", 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0080197099999999993 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.485691+0000", 2024-04-29T14:58:38.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050190850000000004 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.485754+0000", 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3404999999999993e-05 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.485805+0000", 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0593000000000002e-05 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:222717 9.b 9:d2f634df:::benchmark_data_smithi194_196924_object62711:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1113)", 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:58:38.465256+0000", 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.326931001, 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.021619755000000001, 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 222717 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T14:58:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.465256+0000", 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.465256+0000", 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.465259+0000", 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1160000000000002e-06 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.465264+0000", 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8610000000000003e-06 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.465277+0000", 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2629000000000001e-05 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.465285+0000", 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5329999999999997e-06 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.476731+0000", 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011446207 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.476855+0000", 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000124237 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 10", 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.476978+0000", 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012314499999999999 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.480677+0000", 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003698705 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.486778+0000", 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061006660000000002 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.486827+0000", 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9895999999999998e-05 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T14:58:38.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:58:38.486876+0000", 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8760000000000001e-05 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T14:58:38.928 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:58:38.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:38.929 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.929 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.929 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:38 smithi121 ceph-mon[44533]: 9.4 scrub starts 2024-04-29T14:58:38.929 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:38 smithi040 ceph-mon[39581]: 9.4 scrub starts 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:38 smithi194 ceph-mon[44089]: 9.4 scrub starts 2024-04-29T14:58:38.930 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:38.976 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:39.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:39.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:39.274 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:39.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:39.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:39.335 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:39.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:39.478 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:39.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:39.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:39.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T14:58:39.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:39.688 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:39.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:39.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:39.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:39 smithi194 ceph-mon[44089]: pgmap v2849: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:39.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:39 smithi194 ceph-mon[44089]: 9.c scrub starts 2024-04-29T14:58:39.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:39 smithi194 ceph-mon[44089]: 9.4 scrub ok 2024-04-29T14:58:39.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:39 smithi040 ceph-mon[39581]: pgmap v2849: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:39 smithi040 ceph-mon[39581]: 9.c scrub starts 2024-04-29T14:58:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:39 smithi040 ceph-mon[39581]: 9.4 scrub ok 2024-04-29T14:58:39.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:39 smithi121 ceph-mon[44533]: pgmap v2849: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-29T14:58:39.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:39 smithi121 ceph-mon[44533]: 9.c scrub starts 2024-04-29T14:58:39.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:39 smithi121 ceph-mon[44533]: 9.4 scrub ok 2024-04-29T14:58:39.937 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:39.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:39.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:40.044 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:58:40.097 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:40.128 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:40.149 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:40.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:40.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:40.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:40.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:40.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:40.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:40 smithi194 ceph-mon[44089]: 9.c scrub ok 2024-04-29T14:58:40.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:40 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:40.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:40 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:40.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:40.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:40 smithi121 ceph-mon[44533]: 9.c scrub ok 2024-04-29T14:58:40.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:40 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:40.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:40 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:40.696 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:58:40.727 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:40.747 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:58:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:40.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:58:40.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:40.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:40 smithi040 ceph-mon[39581]: 9.c scrub ok 2024-04-29T14:58:40.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:40 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:40.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:40 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:40.918 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:41.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:41.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:41.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:41.214 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:41.567 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:58:41.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:41.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:41.789 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:58:41.822 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:41 smithi040 ceph-mon[39581]: pgmap v2850: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:41.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:41 smithi040 ceph-mon[39581]: osdmap e1114: 12 total, 12 up, 8 in 2024-04-29T14:58:41.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4195214738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:58:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:41 smithi194 ceph-mon[44089]: pgmap v2850: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:41.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:41 smithi194 ceph-mon[44089]: osdmap e1114: 12 total, 12 up, 8 in 2024-04-29T14:58:41.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4195214738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:58:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:41.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:41 smithi121 ceph-mon[44533]: pgmap v2850: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T14:58:41.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:41 smithi121 ceph-mon[44533]: osdmap e1114: 12 total, 12 up, 8 in 2024-04-29T14:58:41.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4195214738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:58:41.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:58:42.536 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T14:58:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T14:58:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:227002 9.8 9:179fc0b7:::benchmark_data_smithi194_196924_object66996:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1115)", 2024-04-29T14:58:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:58:42.517916+0000", 2024-04-29T14:58:42.536 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.010965671, 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010989252, 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227002 2024-04-29T14:58:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517916+0000", 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517916+0000", 2024-04-29T14:58:42.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517917+0000", 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2699999999999999e-07 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517922+0000", 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2209999999999996e-06 2024-04-29T14:58:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517928+0000", 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9909999999999999e-06 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517933+0000", 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3369999999999999e-06 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.518148+0000", 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021514099999999999 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.518214+0000", 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5648999999999994e-05 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:58:42.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.518279+0000", 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4665000000000005e-05 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527474+0000", 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091953290000000003 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:227004 9.8 9:1a77fb72:::benchmark_data_smithi194_196924_object66998:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1115)", 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:58:42.517940+0000", 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.010941813, 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011019543, 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T14:58:42.543 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227004 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517940+0000", 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517940+0000", 2024-04-29T14:58:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517941+0000", 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.37e-07 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517949+0000", 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1960000000000008e-06 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517955+0000", 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.049e-06 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.517960+0000", 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2900000000000002e-06 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.522122+0000", 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041621109999999996 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.522234+0000", 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011162699999999999 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:58:42.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.522308+0000", 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4509000000000001e-05 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527481+0000", 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051723619999999998 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:227018 9.8 9:1e44815f:::benchmark_data_smithi194_196924_object67012:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1115)", 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:58:42.527749+0000", 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0011327150000000001, 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012565079999999999, 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 227018 2024-04-29T14:58:42.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527749+0000", 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527749+0000", 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527751+0000", 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6440000000000001e-06 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527757+0000", 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3539999999999996e-06 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T14:58:42.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527773+0000", 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5600999999999999e-05 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527781+0000", 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5019999999999992e-06 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527842+0000", 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1317999999999998e-05 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.527974+0000", 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013180900000000001 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:42.528093+0000", 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118973 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:58:42.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T14:58:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-29T14:58:42.554 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 3 2024-04-29T14:58:42.554 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T14:58:42.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:42 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:42.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4195214738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:58:42.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:42 smithi194 ceph-mon[44089]: osdmap e1115: 12 total, 12 up, 8 in 2024-04-29T14:58:42.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:42 smithi194 ceph-mon[44089]: 9.3 scrub starts 2024-04-29T14:58:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:42 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4195214738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:58:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:42 smithi040 ceph-mon[39581]: osdmap e1115: 12 total, 12 up, 8 in 2024-04-29T14:58:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:42 smithi040 ceph-mon[39581]: 9.3 scrub starts 2024-04-29T14:58:42.879 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:58:42.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:42 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:42.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4195214738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:58:42.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:42 smithi121 ceph-mon[44533]: osdmap e1115: 12 total, 12 up, 8 in 2024-04-29T14:58:42.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:42 smithi121 ceph-mon[44533]: 9.3 scrub starts 2024-04-29T14:58:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:42.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:58:42.990 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:43.019 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:43.040 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:43.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:43.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:43.192 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:43.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T14:58:43.347 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:43.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:43.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:43.484 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:43.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:43.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:43.721 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:58:43.800 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:43.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:43 smithi040 ceph-mon[39581]: pgmap v2853: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:58:43.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:43 smithi121 ceph-mon[44533]: pgmap v2853: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:58:44.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:44.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:44.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:43 smithi194 ceph-mon[44089]: pgmap v2853: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-29T14:58:44.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:58:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:44.211 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:58:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:44.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:58:44.284 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:44.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:44.361 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:44.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:44.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:44.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:44.748 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:45.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:58:45.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:45 smithi040 ceph-mon[39581]: pgmap v2854: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:45.788 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:45.788 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:45 smithi121 ceph-mon[44533]: pgmap v2854: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:45.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:45.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:46.043 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T14:58:46.044 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T14:58:46.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:45 smithi194 ceph-mon[44089]: pgmap v2854: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:46.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:46.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:46.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:46 smithi040 ceph-mon[39581]: osdmap e1116: 12 total, 12 up, 8 in 2024-04-29T14:58:46.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:46 smithi121 ceph-mon[44533]: osdmap e1116: 12 total, 12 up, 8 in 2024-04-29T14:58:46.963 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2855,"stamp":"2024-04-29T14:58:45.602346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6039798328,"num_objects":92154,"num_object_clones":0,"num_object_copies":184308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92154,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":228037,"num_write_kb":10243131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160895,"ondisk_log_size":160895,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18637644,"kb_used_data":11879316,"kb_used_omap":664,"kb_used_meta":6757159,"kb_avail":731323572,"statfs":{"total":767960285184,"available":748875337728,"internally_reserved":0,"allocated":12164419584,"data_stored":12161450353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":680854,"internal_metadata":6919330922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-606601216,"num_objects":-9256,"num_object_clones":0,"num_object_copies":-18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.176015"},"pg_stats":[{"pgid":"9.f","version":"1115'14276","reported_seq":14491,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:44.130125+0000","last_change":"2024-04-29T14:57:37.360585+0000","last_active":"2024-04-29T14:58:44.130125+0000","last_peered":"2024-04-29T14:58:44.130125+0000","last_clean":"2024-04-29T14:58:44.130125+0000","last_became_active":"2024-04-29T14:55:59.715306+0000","last_became_peered":"2024-04-29T14:55:59.715306+0000","last_unstale":"2024-04-29T14:58:44.130125+0000","last_undegraded":"2024-04-29T14:58:44.130125+0000","last_fullsized":"2024-04-29T14:58:44.130125+0000","mapping_epoch":1065,"log_start":"1079'4200","ondisk_log_start":"1079'4200","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1100'9730","last_scrub_stamp":"2024-04-29T14:57:37.360464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:37.360464+0000","objects_scrubbed":9655,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.217197+0000","scrub_duration":2275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373817344,"num_objects":5704,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14276,"num_write_kb":639360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"1114'14206","reported_seq":14414,"reported_epoch":1114,"state":"active+clean","last_fresh":"2024-04-29T14:58:41.120453+0000","last_change":"2024-04-29T14:57:28.787035+0000","last_active":"2024-04-29T14:58:41.120453+0000","last_peered":"2024-04-29T14:58:41.120453+0000","last_clean":"2024-04-29T14:58:41.120453+0000","last_became_active":"2024-04-29T14:55:59.715887+0000","last_became_peered":"2024-04-29T14:55:59.715887+0000","last_unstale":"2024-04-29T14:58:41.120453+0000","last_undegraded":"2024-04-29T14:58:41.120453+0000","last_fullsized":"2024-04-29T14:58:41.120453+0000","mapping_epoch":1065,"log_start":"1079'4200","ondisk_log_start":"1079'4200","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9237","last_scrub_stamp":"2024-04-29T14:57:28.786814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:28.786814+0000","objects_scrubbed":9187,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:58.780518+0000","scrub_duration":2102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387448832,"num_objects":5912,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14206,"num_write_kb":643776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"1114'14061","reported_seq":14270,"reported_epoch":1114,"state":"active+clean","last_fresh":"2024-04-29T14:58:41.134291+0000","last_change":"2024-04-29T14:57:31.565364+0000","last_active":"2024-04-29T14:58:41.134291+0000","last_peered":"2024-04-29T14:58:41.134291+0000","last_clean":"2024-04-29T14:58:41.134291+0000","last_became_active":"2024-04-29T14:55:59.715955+0000","last_became_peered":"2024-04-29T14:55:59.715955+0000","last_unstale":"2024-04-29T14:58:41.134291+0000","last_undegraded":"2024-04-29T14:58:41.134291+0000","last_fullsized":"2024-04-29T14:58:41.134291+0000","mapping_epoch":1065,"log_start":"1078'4000","ondisk_log_start":"1078'4000","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9317","last_scrub_stamp":"2024-04-29T14:57:31.565106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:31.565106+0000","objects_scrubbed":9272,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.564312+0000","scrub_duration":1916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387842072,"num_objects":5919,"num_object_clones":0,"num_object_copies":11838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14061,"num_write_kb":639297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"1115'13944","reported_seq":14221,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:43.630132+0000","last_change":"2024-04-29T14:58:39.827411+0000","last_active":"2024-04-29T14:58:43.630132+0000","last_peered":"2024-04-29T14:58:43.630132+0000","last_clean":"2024-04-29T14:58:43.630132+0000","last_became_active":"2024-04-29T14:55:59.711280+0000","last_became_peered":"2024-04-29T14:55:59.711280+0000","last_unstale":"2024-04-29T14:58:43.630132+0000","last_undegraded":"2024-04-29T14:58:43.630132+0000","last_fullsized":"2024-04-29T14:58:43.630132+0000","mapping_epoch":1065,"log_start":"1078'3900","ondisk_log_start":"1078'3900","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'13711","last_scrub_stamp":"2024-04-29T14:58:39.827190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:58:39.827190+0000","objects_scrubbed":5882,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:41.095592+0000","scrub_duration":1340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367001600,"num_objects":5600,"num_object_clones":0,"num_object_copies":11200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13944,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"1114'14143","reported_seq":14349,"reported_epoch":1114,"state":"active+clean","last_fresh":"2024-04-29T14:58:40.843923+0000","last_change":"2024-04-29T14:57:26.289644+0000","last_active":"2024-04-29T14:58:40.843923+0000","last_peered":"2024-04-29T14:58:40.843923+0000","last_clean":"2024-04-29T14:58:40.843923+0000","last_became_active":"2024-04-29T14:55:59.713356+0000","last_became_peered":"2024-04-29T14:55:59.713356+0000","last_unstale":"2024-04-29T14:58:40.843923+0000","last_undegraded":"2024-04-29T14:58:40.843923+0000","last_fullsized":"2024-04-29T14:58:40.843923+0000","mapping_epoch":1065,"log_start":"1078'4100","ondisk_log_start":"1078'4100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1096'9086","last_scrub_stamp":"2024-04-29T14:57:26.289482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:26.289482+0000","objects_scrubbed":9033,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.542697+0000","scrub_duration":1170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394592256,"num_objects":6021,"num_object_clones":0,"num_object_copies":12042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14143,"num_write_kb":645248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"1115'14313","reported_seq":14523,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:44.091924+0000","last_change":"2024-04-29T14:57:29.553496+0000","last_active":"2024-04-29T14:58:44.091924+0000","last_peered":"2024-04-29T14:58:44.091924+0000","last_clean":"2024-04-29T14:58:44.091924+0000","last_became_active":"2024-04-29T14:55:59.715266+0000","last_became_peered":"2024-04-29T14:55:59.715266+0000","last_unstale":"2024-04-29T14:58:44.091924+0000","last_undegraded":"2024-04-29T14:58:44.091924+0000","last_fullsized":"2024-04-29T14:58:44.091924+0000","mapping_epoch":1065,"log_start":"1079'4300","ondisk_log_start":"1079'4300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9276","last_scrub_stamp":"2024-04-29T14:57:29.553192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:29.553192+0000","objects_scrubbed":9211,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:59.188755+0000","scrub_duration":2353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376635392,"num_objects":5747,"num_object_clones":0,"num_object_copies":11494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14313,"num_write_kb":641920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"1115'14577","reported_seq":14789,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:44.775602+0000","last_change":"2024-04-29T14:57:29.673784+0000","last_active":"2024-04-29T14:58:44.775602+0000","last_peered":"2024-04-29T14:58:44.775602+0000","last_clean":"2024-04-29T14:58:44.775602+0000","last_became_active":"2024-04-29T14:55:59.713461+0000","last_became_peered":"2024-04-29T14:55:59.713461+0000","last_unstale":"2024-04-29T14:58:44.775602+0000","last_undegraded":"2024-04-29T14:58:44.775602+0000","last_fullsized":"2024-04-29T14:58:44.775602+0000","mapping_epoch":1065,"log_start":"1079'4500","ondisk_log_start":"1079'4500","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9466","last_scrub_stamp":"2024-04-29T14:57:29.673626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:29.673626+0000","objects_scrubbed":9432,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:47.159905+0000","scrub_duration":1388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376897536,"num_objects":5751,"num_object_clones":0,"num_object_copies":11502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":650496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"1115'14104","reported_seq":14312,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:44.782386+0000","last_change":"2024-04-29T14:57:31.535038+0000","last_active":"2024-04-29T14:58:44.782386+0000","last_peered":"2024-04-29T14:58:44.782386+0000","last_clean":"2024-04-29T14:58:44.782386+0000","last_became_active":"2024-04-29T14:55:59.716261+0000","last_became_peered":"2024-04-29T14:55:59.716261+0000","last_unstale":"2024-04-29T14:58:44.782386+0000","last_undegraded":"2024-04-29T14:58:44.782386+0000","last_fullsized":"2024-04-29T14:58:44.782386+0000","mapping_epoch":1065,"log_start":"1079'4100","ondisk_log_start":"1079'4100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9138","last_scrub_stamp":"2024-04-29T14:57:31.534865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:31.534865+0000","objects_scrubbed":9114,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:46.579381+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364642304,"num_objects":5564,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14104,"num_write_kb":629376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"1115'14131","reported_seq":14344,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:44.139058+0000","last_change":"2024-04-29T14:57:38.146397+0000","last_active":"2024-04-29T14:58:44.139058+0000","last_peered":"2024-04-29T14:58:44.139058+0000","last_clean":"2024-04-29T14:58:44.139058+0000","last_became_active":"2024-04-29T14:55:59.713183+0000","last_became_peered":"2024-04-29T14:55:59.713183+0000","last_unstale":"2024-04-29T14:58:44.139058+0000","last_undegraded":"2024-04-29T14:58:44.139058+0000","last_fullsized":"2024-04-29T14:58:44.139058+0000","mapping_epoch":1065,"log_start":"1079'4100","ondisk_log_start":"1079'4100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1100'9635","last_scrub_stamp":"2024-04-29T14:57:38.146145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:38.146145+0000","objects_scrubbed":9573,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:47.250208+0000","scrub_duration":2076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366018560,"num_objects":5585,"num_object_clones":0,"num_object_copies":11170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14131,"num_write_kb":630912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"1115'14466","reported_seq":14675,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:44.130287+0000","last_change":"2024-04-29T14:57:29.030253+0000","last_active":"2024-04-29T14:58:44.130287+0000","last_peered":"2024-04-29T14:58:44.130287+0000","last_clean":"2024-04-29T14:58:44.130287+0000","last_became_active":"2024-04-29T14:55:59.714571+0000","last_became_peered":"2024-04-29T14:55:59.714571+0000","last_unstale":"2024-04-29T14:58:44.130287+0000","last_undegraded":"2024-04-29T14:58:44.130287+0000","last_fullsized":"2024-04-29T14:58:44.130287+0000","mapping_epoch":1065,"log_start":"1079'4400","ondisk_log_start":"1079'4400","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9262","last_scrub_stamp":"2024-04-29T14:57:29.030022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:29.030022+0000","objects_scrubbed":9178,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:47.154711+0000","scrub_duration":2818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370540544,"num_objects":5654,"num_object_clones":0,"num_object_copies":11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14466,"num_write_kb":643840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"1115'14141","reported_seq":14426,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:42.097900+0000","last_change":"2024-04-29T14:58:38.647959+0000","last_active":"2024-04-29T14:58:42.097900+0000","last_peered":"2024-04-29T14:58:42.097900+0000","last_clean":"2024-04-29T14:58:42.097900+0000","last_became_active":"2024-04-29T14:55:59.712951+0000","last_became_peered":"2024-04-29T14:55:59.712951+0000","last_unstale":"2024-04-29T14:58:42.097900+0000","last_undegraded":"2024-04-29T14:58:42.097900+0000","last_fullsized":"2024-04-29T14:58:42.097900+0000","mapping_epoch":1065,"log_start":"1079'4100","ondisk_log_start":"1079'4100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'13910","last_scrub_stamp":"2024-04-29T14:58:38.647835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:58:38.647835+0000","objects_scrubbed":6207,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:40.840276+0000","scrub_duration":1493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387383296,"num_objects":5911,"num_object_clones":0,"num_object_copies":11822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14141,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"1115'14241","reported_seq":14454,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:43.631190+0000","last_change":"2024-04-29T14:58:41.585585+0000","last_active":"2024-04-29T14:58:43.631190+0000","last_peered":"2024-04-29T14:58:43.631190+0000","last_clean":"2024-04-29T14:58:43.631190+0000","last_became_active":"2024-04-29T14:55:59.713602+0000","last_became_peered":"2024-04-29T14:55:59.713602+0000","last_unstale":"2024-04-29T14:58:43.631190+0000","last_undegraded":"2024-04-29T14:58:43.631190+0000","last_fullsized":"2024-04-29T14:58:43.631190+0000","mapping_epoch":1065,"log_start":"1079'4200","ondisk_log_start":"1079'4200","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1097'9339","last_scrub_stamp":"2024-04-29T14:57:31.345750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:31.345750+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:46.569514+0000","scrub_duration":1937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376635392,"num_objects":5747,"num_object_clones":0,"num_object_copies":11494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14241,"num_write_kb":639616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"1115'14184","reported_seq":14391,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:43.588396+0000","last_change":"2024-04-29T14:57:26.655236+0000","last_active":"2024-04-29T14:58:43.588396+0000","last_peered":"2024-04-29T14:58:43.588396+0000","last_clean":"2024-04-29T14:58:43.588396+0000","last_became_active":"2024-04-29T14:55:59.712436+0000","last_became_peered":"2024-04-29T14:55:59.712436+0000","last_unstale":"2024-04-29T14:58:43.588396+0000","last_undegraded":"2024-04-29T14:58:43.588396+0000","last_fullsized":"2024-04-29T14:58:43.588396+0000","mapping_epoch":1065,"log_start":"1078'4100","ondisk_log_start":"1078'4100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1096'8996","last_scrub_stamp":"2024-04-29T14:57:26.655091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:26.655091+0000","objects_scrubbed":8946,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:44.764007+0000","scrub_duration":1321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373161984,"num_objects":5694,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14184,"num_write_kb":636096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"1115'14440","reported_seq":14653,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:43.591284+0000","last_change":"2024-04-29T14:57:34.381161+0000","last_active":"2024-04-29T14:58:43.591284+0000","last_peered":"2024-04-29T14:58:43.591284+0000","last_clean":"2024-04-29T14:58:43.591284+0000","last_became_active":"2024-04-29T14:55:59.761514+0000","last_became_peered":"2024-04-29T14:55:59.761514+0000","last_unstale":"2024-04-29T14:58:43.591284+0000","last_undegraded":"2024-04-29T14:58:43.591284+0000","last_fullsized":"2024-04-29T14:58:43.591284+0000","mapping_epoch":1065,"log_start":"1079'4400","ondisk_log_start":"1079'4400","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'9582","last_scrub_stamp":"2024-04-29T14:57:34.381039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:34.381039+0000","objects_scrubbed":9521,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:53.357505+0000","scrub_duration":1949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370671616,"num_objects":5656,"num_object_clones":0,"num_object_copies":11312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14440,"num_write_kb":643072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"1115'14252","reported_seq":14524,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:42.128869+0000","last_change":"2024-04-29T14:58:22.484274+0000","last_active":"2024-04-29T14:58:42.128869+0000","last_peered":"2024-04-29T14:58:42.128869+0000","last_clean":"2024-04-29T14:58:42.128869+0000","last_became_active":"2024-04-29T14:55:59.760008+0000","last_became_peered":"2024-04-29T14:55:59.760008+0000","last_unstale":"2024-04-29T14:58:42.128869+0000","last_undegraded":"2024-04-29T14:58:42.128869+0000","last_fullsized":"2024-04-29T14:58:42.128869+0000","mapping_epoch":1065,"log_start":"1079'4200","ondisk_log_start":"1079'4200","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'12866","last_scrub_stamp":"2024-04-29T14:58:22.484089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:58:22.484089+0000","objects_scrubbed":7299,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:40.606858+0000","scrub_duration":1410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384958464,"num_objects":5874,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14252,"num_write_kb":644032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"1115'14377","reported_seq":14593,"reported_epoch":1115,"state":"active+clean","last_fresh":"2024-04-29T14:58:44.139960+0000","last_change":"2024-04-29T14:57:40.651846+0000","last_active":"2024-04-29T14:58:44.139960+0000","last_peered":"2024-04-29T14:58:44.139960+0000","last_clean":"2024-04-29T14:58:44.139960+0000","last_became_active":"2024-04-29T14:55:59.714624+0000","last_became_peered":"2024-04-29T14:55:59.714624+0000","last_unstale":"2024-04-29T14:58:44.139960+0000","last_undegraded":"2024-04-29T14:58:44.139960+0000","last_fullsized":"2024-04-29T14:58:44.139960+0000","mapping_epoch":1065,"log_start":"1079'4300","ondisk_log_start":"1079'4300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1101'10023","last_scrub_stamp":"2024-04-29T14:57:40.651515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:57:40.651515+0000","objects_scrubbed":9940,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:51.580017+0000","scrub_duration":2555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380960768,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14377,"num_write_kb":646080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2269,"reported_epoch":1114,"state":"active+clean","last_fresh":"2024-04-29T14:58:40.559061+0000","last_change":"2024-04-29T14:57:43.962180+0000","last_active":"2024-04-29T14:58:40.559061+0000","last_peered":"2024-04-29T14:58:40.559061+0000","last_clean":"2024-04-29T14:58:40.559061+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T14:58:40.559061+0000","last_undegraded":"2024-04-29T14:58:40.559061+0000","last_fullsized":"2024-04-29T14:58:40.559061+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:57:43.962100+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:57:43.962100+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:58:52.412069+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6039207960,"num_objects":92152,"num_object_clones":0,"num_object_copies":184304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227856,"num_write_kb":10240193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12136947712,"data_stored":12136939568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160756,"ondisk_log_size":160756,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384310,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3070888,"kb_used_data":1818016,"kb_used_omap":1,"kb_used_meta":1252862,"kb_avail":90674264,"statfs":{"total":95995035648,"available":92850446336,"internally_reserved":0,"allocated":1861648384,"data_stored":1861283068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1282931136},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":4591320039457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":640328,"kb_used_data":387072,"kb_used_omap":4,"kb_used_meta":253243,"kb_avail":93104824,"statfs":{"total":95995035648,"available":95339339776,"internally_reserved":0,"allocated":396361728,"data_stored":395945895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":259321536},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234625,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2924580,"kb_used_data":1896416,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":90820572,"statfs":{"total":95995035648,"available":93000265728,"internally_reserved":0,"allocated":1941929984,"data_stored":1941563378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1052637642},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528966,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1826528,"kb_used_data":758172,"kb_used_omap":1,"kb_used_meta":1068222,"kb_avail":91918624,"statfs":{"total":95995035648,"available":94124670976,"internally_reserved":0,"allocated":776368128,"data_stored":776005594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1093859782},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575506,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2996396,"kb_used_data":2252900,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":90748756,"statfs":{"total":95995035648,"available":92926726144,"internally_reserved":0,"allocated":2306969600,"data_stored":2306600232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359739045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2269800,"kb_used_data":1454240,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91475352,"statfs":{"total":95995035648,"available":93670760448,"internally_reserved":0,"allocated":1489141760,"data_stored":1488776464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":835123656},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986830,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1840668,"kb_used_data":748820,"kb_used_omap":1,"kb_used_meta":1091838,"kb_avail":91904484,"statfs":{"total":95995035648,"available":94110191616,"internally_reserved":0,"allocated":766791680,"data_stored":766432698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1118042565},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653390,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3068456,"kb_used_data":2563680,"kb_used_omap":651,"kb_used_meta":504116,"kb_avail":90676696,"statfs":{"total":95995035648,"available":92852936704,"internally_reserved":0,"allocated":2625208320,"data_stored":2624843024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":516215560},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2305167360,"data_stored":2305163288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1487339520,"data_stored":1487339520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":766050304,"data_stored":766050304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2621571072,"data_stored":2621571072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":774897664,"data_stored":774897664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1933447168,"data_stored":1933443096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":394526720,"data_stored":394526720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1853947904,"data_stored":1853947904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T14:58:46.964 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T14:58:47.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:46 smithi194 ceph-mon[44089]: osdmap e1116: 12 total, 12 up, 8 in 2024-04-29T14:58:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:47.515 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:47.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:47.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:47.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:47 smithi040 ceph-mon[39581]: from='client.28555 v1:172.21.15.40:0/3668638272' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:58:47.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:47 smithi040 ceph-mon[39581]: pgmap v2856: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-29T14:58:47.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:47 smithi121 ceph-mon[44533]: from='client.28555 v1:172.21.15.40:0/3668638272' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:58:47.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:47 smithi121 ceph-mon[44533]: pgmap v2856: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-29T14:58:48.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:47 smithi194 ceph-mon[44089]: from='client.28555 v1:172.21.15.40:0/3668638272' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T14:58:48.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:47 smithi194 ceph-mon[44089]: pgmap v2856: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-29T14:58:49.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2019202719' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:49.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:48 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2019202719' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:48 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2019202719' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:48 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:58:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:58:49.520 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:58:49.521 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:58:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:49 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:49 smithi121 ceph-mon[44533]: pgmap v2857: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:58:49.730 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:49.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:49.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:49.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:49.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:49.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:49 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:49.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:49 smithi040 ceph-mon[39581]: pgmap v2857: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:58:50.009 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:58:50.009 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:58:50.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:49 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:49 smithi194 ceph-mon[44089]: pgmap v2857: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:58:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:50.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:50.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:50.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:50.299 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:50.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:50.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:50.581 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:50.611 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:50.632 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:50.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:50.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:50.778 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:51.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:51.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:51.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T14:58:51.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:51.537 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:51.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:51.641 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:51.669 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:51.695 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:51.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:51.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:51.867 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:58:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:51.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:51 smithi040 ceph-mon[39581]: pgmap v2858: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:58:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:51.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:51.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:51 smithi121 ceph-mon[44533]: pgmap v2858: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:58:51.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:52.053 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:52.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:51 smithi194 ceph-mon[44089]: pgmap v2858: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-29T14:58:52.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:52.205 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:52.226 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:52.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:52.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:52.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:52.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:52.552 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:52.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:58:52.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:52.710 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2723998740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:52 smithi121 ceph-mon[44533]: from='client.28564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:52 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:52.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:52 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:52.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:58:52.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2723998740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:52.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:52 smithi040 ceph-mon[39581]: from='client.28564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:52.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:52 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:52.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:52 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2723998740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:52 smithi194 ceph-mon[44089]: from='client.28564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:58:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:52 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:52 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:53.754 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:53 smithi121 ceph-mon[44533]: pgmap v2859: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:58:53.754 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:53 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:53.754 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:53 smithi121 ceph-mon[44533]: from='client.28564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:53.754 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:53 smithi121 ceph-mon[44533]: osdmap e1117: 12 total, 12 up, 8 in 2024-04-29T14:58:53.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:53.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:53.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:53.924 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:54.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:53 smithi194 ceph-mon[44089]: pgmap v2859: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:58:54.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:53 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:54.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:53 smithi194 ceph-mon[44089]: from='client.28564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:54.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:53 smithi194 ceph-mon[44089]: osdmap e1117: 12 total, 12 up, 8 in 2024-04-29T14:58:54.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:54.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:54.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:53 smithi040 ceph-mon[39581]: pgmap v2859: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:58:54.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:53 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:58:54.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:53 smithi040 ceph-mon[39581]: from='client.28564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:58:54.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:53 smithi040 ceph-mon[39581]: osdmap e1117: 12 total, 12 up, 8 in 2024-04-29T14:58:54.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:54.244 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:54.269 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:54.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:54.373 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:58:54.402 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:54.422 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:58:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:54.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:58:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:54.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:58:54.542 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T14:58:54.542 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:58:54.561 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:54.605 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:58:54.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:54.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:58:54.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:54.740 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:54.763 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:54.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:54.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:58:54.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:54.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:54.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:54.917 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:55.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:58:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:55.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:58:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:55.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:58:55.178 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:55.855 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T14:58:55.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:58:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:55.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:58:55.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:55 smithi040 ceph-mon[39581]: pgmap v2861: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:55.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:55 smithi040 ceph-mon[39581]: osdmap e1118: 12 total, 12 up, 8 in 2024-04-29T14:58:55.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:58:56.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:55 smithi121 ceph-mon[44533]: pgmap v2861: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:56.012 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:55 smithi121 ceph-mon[44533]: osdmap e1118: 12 total, 12 up, 8 in 2024-04-29T14:58:56.012 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:56.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:58:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:56.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:58:56.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:55 smithi194 ceph-mon[44089]: pgmap v2861: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:58:56.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:55 smithi194 ceph-mon[44089]: osdmap e1118: 12 total, 12 up, 8 in 2024-04-29T14:58:56.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:58:56.173 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:58:56.191 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:56.326 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:58:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:56.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:58:56.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:58:56.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:56.479 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:56.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:56.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:56.617 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:58:56.645 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:56.701 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:58:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:56.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:58:56.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:56.847 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:56 smithi121 ceph-mon[44533]: pgmap v2863: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:56.851 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:56.857 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:58:56.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:56.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:56.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:58:57.042 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:58:57.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:56 smithi194 ceph-mon[44089]: pgmap v2863: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:57.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:57.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:58:57.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:56 smithi040 ceph-mon[39581]: pgmap v2863: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-29T14:58:57.180 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:57.209 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:57.582 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:58:57.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:58:57.627 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:58:57.665 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:58:57.695 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:58:57.732 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:58:57.760 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:58:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:57.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:58:57.784 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:58:57.819 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:58:57.848 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:58:57.864 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:58:57.873 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:58:57.897 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:57.902 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:58:57.928 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:58:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:58:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:58:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:58:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:58:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:58:57.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/113345858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:58.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/113345858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:58.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/113345858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:58:58.147 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:58:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:58.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:58:58.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:58:58.288 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:58.515 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:58:58.515 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:58:58.515 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T14:58:58.515 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:58.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:58.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:58.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:58:58.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:58:58.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: pgmap v2864: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-04-29T14:58:58.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:58 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:58:59.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:59.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: pgmap v2864: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-04-29T14:58:59.120 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:58 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:58:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: pgmap v2864: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-04-29T14:58:59.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:58 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:58:59.649 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:58:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:58:59.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:58:59.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:58:59.797 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:58:59.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:59 smithi040 ceph-mon[39581]: osdmap e1119: 12 total, 12 up, 8 in 2024-04-29T14:58:59.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:59 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:58:59.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:59 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:58:59.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:58:59.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:58:59.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:58:59 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:59:00.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:59 smithi194 ceph-mon[44089]: osdmap e1119: 12 total, 12 up, 8 in 2024-04-29T14:59:00.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:59 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:59:00.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:59 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:59:00.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:59:00.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:59:00.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:58:59 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:59:00.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:59 smithi121 ceph-mon[44533]: osdmap e1119: 12 total, 12 up, 8 in 2024-04-29T14:59:00.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:59 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:59:00.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:59 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:59:00.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:59:00.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:59:00.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:58:59 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:59:00.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T14:59:00.847 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:00.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:00.951 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:00.978 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:01.004 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:01.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:01.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:00 smithi194 ceph-mon[44089]: osdmap e1120: 12 total, 12 up, 8 in 2024-04-29T14:59:01.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:00 smithi194 ceph-mon[44089]: pgmap v2867: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-29T14:59:01.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:01.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:00 smithi040 ceph-mon[39581]: osdmap e1120: 12 total, 12 up, 8 in 2024-04-29T14:59:01.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:00 smithi040 ceph-mon[39581]: pgmap v2867: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-29T14:59:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:00 smithi121 ceph-mon[44533]: osdmap e1120: 12 total, 12 up, 8 in 2024-04-29T14:59:01.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:00 smithi121 ceph-mon[44533]: pgmap v2867: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-29T14:59:01.161 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:01.169 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:59:01.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:01.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:01.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:01.336 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:01.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:01.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:01.526 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:01.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:01.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:01.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:59:01.663 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:01.695 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:01.800 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:01.803 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:59:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:01.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:01.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:01.968 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:02.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:02.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:02.061 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4118857862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:02.113 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:02.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4118857862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:02.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4118857862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:02.148 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:02.699 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:02.700 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:02.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:02.805 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:02 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4118857862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:59:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:02 smithi194 ceph-mon[44089]: osdmap e1121: 12 total, 12 up, 8 in 2024-04-29T14:59:02.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:02 smithi194 ceph-mon[44089]: pgmap v2869: 17 pgs: 17 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-29T14:59:02.835 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:02.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:02 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:02.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4118857862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:59:02.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:02 smithi040 ceph-mon[39581]: osdmap e1121: 12 total, 12 up, 8 in 2024-04-29T14:59:02.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:02 smithi040 ceph-mon[39581]: pgmap v2869: 17 pgs: 17 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-29T14:59:03.049 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:03.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:03.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:02 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:03.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4118857862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:59:03.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:02 smithi121 ceph-mon[44533]: osdmap e1121: 12 total, 12 up, 8 in 2024-04-29T14:59:03.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:02 smithi121 ceph-mon[44533]: pgmap v2869: 17 pgs: 17 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-29T14:59:03.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:03.181 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:03.336 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:03.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:03.443 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:03.477 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:03.633 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:03.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:03.645 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T14:59:03.645 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T14:59:03.645 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-04-29T14:59:03.645 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-04-29T14:59:03.645 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T14:59:03.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1 2024-04-29T14:59:03.778 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:04.134 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-29T14:59:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:04.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.1' 2024-04-29T14:59:04.239 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:04.272 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:04.525 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:04.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:04.536 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T14:59:04.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:04.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:59:04.698 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:04.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:04.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:04.764 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:59:04.821 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1: {} 2024-04-29T14:59:04.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:04.860 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:59:04.886 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:04.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:04.899 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:59:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T14:59:04.944 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:59:04.984 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:59:04.993 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:05.017 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:59:05.024 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:05.046 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:59:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:05.102 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:59:05.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:59:05.160 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:59:05.189 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:59:05.191 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:59:05.191 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-29T14:59:05.191 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:59:05.191 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:59:05.191 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:59:05.191 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:59:05.191 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:59:05.192 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:59:05.192 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:59:05.192 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:59:05.192 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:59:05.192 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:59:05.233 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:05.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:05.337 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:05.365 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: 9.b scrub starts 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: pgmap v2870: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1762010664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: 9.1 scrub starts 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.500 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.501 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.501 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.501 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.501 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.501 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: 9.b scrub ok 2024-04-29T14:59:05.501 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:05 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.501 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: 9.b scrub starts 2024-04-29T14:59:05.502 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: pgmap v2870: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:59:05.502 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1762010664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:05.502 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: 9.1 scrub starts 2024-04-29T14:59:05.502 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.568 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:05.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:05.672 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:05.712 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:05.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: 9.b scrub ok 2024-04-29T14:59:05.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:05 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: 9.b scrub starts 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: pgmap v2870: 17 pgs: 17 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.55k op/s 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1762010664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: 9.1 scrub starts 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: 9.b scrub ok 2024-04-29T14:59:05.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:05 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:05.966 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:05.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:06.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:06.144 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:06.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:06.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:06.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:06.329 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:06.545 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:06.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:06.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:06.727 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:06.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:06.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:06.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:06 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T14:59:06.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:06 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:06 smithi040 ceph-mon[39581]: 9.5 scrub starts 2024-04-29T14:59:06.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:06 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T14:59:06.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:06 smithi040 ceph-mon[39581]: 9.1 scrub ok 2024-04-29T14:59:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:06 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T14:59:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:06 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:06 smithi194 ceph-mon[44089]: 9.5 scrub starts 2024-04-29T14:59:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:06 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T14:59:06.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:06 smithi194 ceph-mon[44089]: 9.1 scrub ok 2024-04-29T14:59:06.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:06.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:06 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T14:59:06.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:06 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:06.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:06 smithi121 ceph-mon[44533]: 9.5 scrub starts 2024-04-29T14:59:06.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:06 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T14:59:06.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:06 smithi121 ceph-mon[44533]: 9.1 scrub ok 2024-04-29T14:59:06.926 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:06.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:06.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:06.966 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-29T14:59:06.966 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 3, 4, 6, 7, 10, 11, 2, 8, 5, 9] 2024-04-29T14:59:06.966 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-29T14:59:06.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.1 2024-04-29T14:59:07.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:07.124 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:59:07 smithi040 systemd[1]: Stopping Ceph osd.1 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T14:59:07.137 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:07.172 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:07.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:07.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:07.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:07.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:07.390 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T14:59:07.390 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:59:07.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:07.543 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:07.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:07.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:07.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:07.762 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:07.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:07 smithi194 ceph-mon[44089]: 9.d scrub starts 2024-04-29T14:59:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:07 smithi194 ceph-mon[44089]: pgmap v2871: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:59:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:07 smithi194 ceph-mon[44089]: 9.5 scrub ok 2024-04-29T14:59:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:07.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:07.837 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:07 smithi121 ceph-mon[44533]: 9.d scrub starts 2024-04-29T14:59:07.837 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:07 smithi121 ceph-mon[44533]: pgmap v2871: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:59:07.837 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:07 smithi121 ceph-mon[44533]: 9.5 scrub ok 2024-04-29T14:59:07.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:07 smithi040 ceph-mon[39581]: 9.d scrub starts 2024-04-29T14:59:07.852 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:07 smithi040 ceph-mon[39581]: pgmap v2871: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-29T14:59:07.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:07 smithi040 ceph-mon[39581]: 9.5 scrub ok 2024-04-29T14:59:07.937 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:08.117 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:59:07 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[66356]: 2024-04-29T14:59:07.846+0000 7fce48a88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T14:59:08.118 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:59:07 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[66356]: 2024-04-29T14:59:07.846+0000 7fce48a88700 -1 osd.1 1121 *** Got signal Terminated *** 2024-04-29T14:59:08.118 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:59:07 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[66356]: 2024-04-29T14:59:07.846+0000 7fce48a88700 -1 osd.1 1121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T14:59:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:08.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:08.220 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:08.261 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:08.286 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:08.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:08.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-29T14:59:08.399 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:08.429 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:08.452 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:08.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:08.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:08.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:08 smithi194 ceph-mon[44089]: 9.e scrub starts 2024-04-29T14:59:08.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:08 smithi194 ceph-mon[44089]: 9.d scrub ok 2024-04-29T14:59:08.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:08 smithi194 ceph-mon[44089]: 9.3 scrub starts 2024-04-29T14:59:08.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:08 smithi194 ceph-mon[44089]: osd.1 marked itself down and dead 2024-04-29T14:59:08.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:59:08.610 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:08.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:08.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:08.742 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:08.770 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:08.792 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:08.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:08.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:08 smithi040 ceph-mon[39581]: 9.e scrub starts 2024-04-29T14:59:08.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:08 smithi040 ceph-mon[39581]: 9.d scrub ok 2024-04-29T14:59:08.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:08 smithi040 ceph-mon[39581]: 9.3 scrub starts 2024-04-29T14:59:08.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:08 smithi040 ceph-mon[39581]: osd.1 marked itself down and dead 2024-04-29T14:59:08.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:08 smithi121 ceph-mon[44533]: 9.e scrub starts 2024-04-29T14:59:08.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:08 smithi121 ceph-mon[44533]: 9.d scrub ok 2024-04-29T14:59:08.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:08 smithi121 ceph-mon[44533]: 9.3 scrub starts 2024-04-29T14:59:08.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:08 smithi121 ceph-mon[44533]: osd.1 marked itself down and dead 2024-04-29T14:59:08.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:08.924 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:08.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:08.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:09.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:09.075 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:09.323 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:09.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:09.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:09.456 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:09.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:09.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: 9.e scrub ok 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: 9.6 scrub starts 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: pgmap v2872: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: 9.3 scrub ok 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: osdmap e1122: 12 total, 11 up, 8 in 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: 9.6 scrub ok 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:59:09.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:09 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:59:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:09.797 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: 9.e scrub ok 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: 9.6 scrub starts 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: pgmap v2872: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: 9.3 scrub ok 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: osdmap e1122: 12 total, 11 up, 8 in 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: 9.6 scrub ok 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:59:09.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:09 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:59:09.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: 9.e scrub ok 2024-04-29T14:59:09.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: 9.6 scrub starts 2024-04-29T14:59:09.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: pgmap v2872: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 906 op/s 2024-04-29T14:59:09.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: 9.3 scrub ok 2024-04-29T14:59:09.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-29T14:59:09.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: osdmap e1122: 12 total, 11 up, 8 in 2024-04-29T14:59:09.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: 9.6 scrub ok 2024-04-29T14:59:09.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:59:09.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:09 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:59:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1252635541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:59:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:10 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:59:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1252635541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:59:10.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:10 smithi194 ceph-mon[44089]: osdmap e1123: 12 total, 11 up, 8 in 2024-04-29T14:59:10.831 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1252635541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:59:10.831 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:10 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:59:10.831 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1252635541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:59:10.831 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:10 smithi040 ceph-mon[39581]: osdmap e1123: 12 total, 11 up, 8 in 2024-04-29T14:59:10.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1252635541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:59:10.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:10 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T14:59:10.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1252635541' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T14:59:10.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:10 smithi121 ceph-mon[44533]: osdmap e1123: 12 total, 11 up, 8 in 2024-04-29T14:59:11.124 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 14:59:10 smithi040 bash[384216]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1 2024-04-29T14:59:11.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:11 smithi194 ceph-mon[44089]: 9.8 scrub starts 2024-04-29T14:59:11.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:11 smithi194 ceph-mon[44089]: pgmap v2875: 17 pgs: 2 peering, 2 stale+active+clean, 1 active+clean+scrubbing, 12 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:59:11.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:11 smithi194 ceph-mon[44089]: 9.8 scrub ok 2024-04-29T14:59:11.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:11 smithi194 ceph-mon[44089]: 9.7 scrub starts 2024-04-29T14:59:11.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:11 smithi194 ceph-mon[44089]: 9.7 scrub ok 2024-04-29T14:59:11.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:11 smithi040 ceph-mon[39581]: 9.8 scrub starts 2024-04-29T14:59:11.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:11 smithi040 ceph-mon[39581]: pgmap v2875: 17 pgs: 2 peering, 2 stale+active+clean, 1 active+clean+scrubbing, 12 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:59:11.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:11 smithi040 ceph-mon[39581]: 9.8 scrub ok 2024-04-29T14:59:11.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:11 smithi040 ceph-mon[39581]: 9.7 scrub starts 2024-04-29T14:59:11.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:11 smithi040 ceph-mon[39581]: 9.7 scrub ok 2024-04-29T14:59:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:11 smithi121 ceph-mon[44533]: 9.8 scrub starts 2024-04-29T14:59:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:11 smithi121 ceph-mon[44533]: pgmap v2875: 17 pgs: 2 peering, 2 stale+active+clean, 1 active+clean+scrubbing, 12 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-29T14:59:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:11 smithi121 ceph-mon[44533]: 9.8 scrub ok 2024-04-29T14:59:11.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:11 smithi121 ceph-mon[44533]: 9.7 scrub starts 2024-04-29T14:59:11.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:11 smithi121 ceph-mon[44533]: 9.7 scrub ok 2024-04-29T14:59:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:11.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:12.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:12.129 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:12.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:12.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:12.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:12.288 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:12.309 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:12.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:12.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:12.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:12.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:12.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:59:12.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:59:12.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:12 smithi194 ceph-mon[44089]: osdmap e1124: 12 total, 11 up, 8 in 2024-04-29T14:59:12.591 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:59:12.591 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:59:12.591 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:12 smithi040 ceph-mon[39581]: osdmap e1124: 12 total, 11 up, 8 in 2024-04-29T14:59:12.599 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:12.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:12.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:12.725 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:12.757 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:12.782 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:12.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:12.886 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:59:12.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:59:12.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:12 smithi121 ceph-mon[44533]: osdmap e1124: 12 total, 11 up, 8 in 2024-04-29T14:59:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:13.042 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:13.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:13.147 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:13.179 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:13.199 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:13.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:13.303 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:13.332 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:13.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:13.457 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:13.485 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:13.505 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:13.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:13.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:13.642 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:13.664 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:13.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:13.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:13.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:13 smithi121 ceph-mon[44533]: pgmap v2877: 17 pgs: 3 undersized+degraded+peered, 2 peering, 12 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 12210/129312 objects degraded (9.442%) 2024-04-29T14:59:13.797 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:13 smithi121 ceph-mon[44533]: Health check failed: Degraded data redundancy: 12210/129312 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:13.799 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:13.820 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:13.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:13 smithi194 ceph-mon[44089]: pgmap v2877: 17 pgs: 3 undersized+degraded+peered, 2 peering, 12 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 12210/129312 objects degraded (9.442%) 2024-04-29T14:59:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:13 smithi194 ceph-mon[44089]: Health check failed: Degraded data redundancy: 12210/129312 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:13 smithi040 ceph-mon[39581]: pgmap v2877: 17 pgs: 3 undersized+degraded+peered, 2 peering, 12 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 12210/129312 objects degraded (9.442%) 2024-04-29T14:59:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:13 smithi040 ceph-mon[39581]: Health check failed: Degraded data redundancy: 12210/129312 objects degraded (9.442%), 3 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:13.924 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:13.957 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:13.979 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:13.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:14.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T14:59:14.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:14.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:14.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:14.241 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:14.273 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:14.294 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:14.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:14.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:14.431 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:14.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:14.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:14.556 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:14.588 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:14.613 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:14.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:14.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:14.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:59:14.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:59:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:59:14.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:59:14.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:59:14.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:59:15.208 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:59:15.208 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:59:15.498 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:15.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:15.603 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:15.606 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:59:15.606 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:59:15.606 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:59:15.606 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:59:15.606 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:59:15.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:59:15.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:59:15.607 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:59:15.609 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:59:15.629 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:59:15.631 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:15.651 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:59:15.652 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:59:15.652 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T14:59:15.652 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:59:15.652 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:59:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:59:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:59:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:59:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:59:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:59:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:59:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:59:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:59:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:15.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:15.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:15.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:15.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:15 smithi121 ceph-mon[44533]: pgmap v2878: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 673 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:15.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:15 smithi121 ceph-mon[44533]: osdmap e1125: 12 total, 11 up, 8 in 2024-04-29T14:59:15.789 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:15 smithi121 ceph-mon[44533]: 9.9 scrub starts 2024-04-29T14:59:15.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:15 smithi121 ceph-mon[44533]: 9.9 scrub ok 2024-04-29T14:59:15.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3361659260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:15.790 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:15 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:15.791 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:15.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:15.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:15.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:15 smithi194 ceph-mon[44089]: pgmap v2878: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 673 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:15.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:15 smithi194 ceph-mon[44089]: osdmap e1125: 12 total, 11 up, 8 in 2024-04-29T14:59:15.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:15 smithi194 ceph-mon[44089]: 9.9 scrub starts 2024-04-29T14:59:15.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:15 smithi194 ceph-mon[44089]: 9.9 scrub ok 2024-04-29T14:59:15.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3361659260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:15.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:15 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:15.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:15 smithi040 ceph-mon[39581]: pgmap v2878: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 673 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:15.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:15 smithi040 ceph-mon[39581]: osdmap e1125: 12 total, 11 up, 8 in 2024-04-29T14:59:15.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:15 smithi040 ceph-mon[39581]: 9.9 scrub starts 2024-04-29T14:59:15.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:15 smithi040 ceph-mon[39581]: 9.9 scrub ok 2024-04-29T14:59:15.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3361659260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:15.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:15 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:15.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:15.948 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:16.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-29T14:59:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:16.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:16.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.817 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.818 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:16 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:16 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.838 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:16.859 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:16.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:16.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:16 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:16.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:16.994 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:17.014 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:17.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:17.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:17.146 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:17.165 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:17.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:59:17.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:59:17.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:17.298 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:17.336 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:17.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:17.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:17.474 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:17.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:17.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:17.628 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:17.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:17.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:17.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:17.753 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:17.783 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:17.804 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:17.806 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:17 smithi194 ceph-mon[44089]: pgmap v2880: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 110 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:17.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:17.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:17.821 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:17 smithi040 ceph-mon[39581]: pgmap v2880: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 110 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:17.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:17 smithi121 ceph-mon[44533]: pgmap v2880: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 110 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:17.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:17.939 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:17.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:17.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:18.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:18.095 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:18.584 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:19.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:59:19.747 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:19.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:19.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:19 smithi194 ceph-mon[44089]: pgmap v2881: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 91 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:19.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:19 smithi040 ceph-mon[39581]: pgmap v2881: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 91 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:19.880 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:19 smithi121 ceph-mon[44533]: pgmap v2881: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 91 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:20.630 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:20.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:20.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:20.769 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:20.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:20.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:20.894 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:20.927 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:20.948 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:20.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:21.052 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:21.082 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:21.102 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:21.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:21.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:21.235 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:21.255 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:21.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:21.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:21.393 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:21.416 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:21.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:21.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:21.553 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:21.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:21.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:21.709 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:21.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:21.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:21.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:21 smithi194 ceph-mon[44089]: pgmap v2882: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 4 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:21.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:59:21.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:59:21.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:21 smithi040 ceph-mon[39581]: pgmap v2882: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 4 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:21.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:59:21.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:59:21.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:21.847 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.1.service' 2024-04-29T14:59:21.866 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:21.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:21 smithi121 ceph-mon[44533]: pgmap v2882: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 4 op/s; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:21.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:59:21.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:59:21.887 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:21.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:21.892 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T14:59:21.892 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-29T14:59:21.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd down 1 2024-04-29T14:59:21.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:22.119 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:22.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:22.143 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:22.158 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:59:22.158 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:59:22.159 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:59:22.159 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:59:22.159 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:59:22.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:22.256 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:22.406 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:22.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:22.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:22.543 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:22.557 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T14:59:22.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:22.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:22.669 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:22.701 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:22.723 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:22.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:22 smithi194 ceph-mon[44089]: osdmap e1126: 12 total, 11 up, 8 in 2024-04-29T14:59:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2281966509' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:59:22.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:22.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:22 smithi040 ceph-mon[39581]: osdmap e1126: 12 total, 11 up, 8 in 2024-04-29T14:59:22.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2281966509' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:59:22.856 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:22.876 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:22.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:22.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:22 smithi121 ceph-mon[44533]: osdmap e1126: 12 total, 11 up, 8 in 2024-04-29T14:59:22.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2281966509' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T14:59:22.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:23.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-29T14:59:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:23 smithi194 ceph-mon[44089]: pgmap v2884: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:23 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2281966509' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:59:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:23 smithi194 ceph-mon[44089]: osdmap e1127: 12 total, 11 up, 8 in 2024-04-29T14:59:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:23 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 24323/129240 objects degraded (18.820%), 6 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:23 smithi040 ceph-mon[39581]: pgmap v2884: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:23 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2281966509' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:59:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:23 smithi040 ceph-mon[39581]: osdmap e1127: 12 total, 11 up, 8 in 2024-04-29T14:59:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:23 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 24323/129240 objects degraded (18.820%), 6 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:23 smithi121 ceph-mon[44533]: pgmap v2884: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:23 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2281966509' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T14:59:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:23 smithi121 ceph-mon[44533]: osdmap e1127: 12 total, 11 up, 8 in 2024-04-29T14:59:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:23 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 24323/129240 objects degraded (18.820%), 6 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:24.568 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:25.460 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:25.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:25.565 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:25.596 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:25.603 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:59:25.636 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:59:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:25.637 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:25.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:25.642 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:59:25.663 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:59:25.683 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:59:25.703 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:59:25.723 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:59:25.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:25.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:25 smithi040 ceph-mon[39581]: pgmap v2886: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:25.745 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:59:25.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T14:59:25.765 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:59:25.769 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:25.785 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:25.789 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:59:25.791 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:25.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:25.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:59:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:59:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:59:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:59:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:25 smithi194 ceph-mon[44089]: pgmap v2886: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:25.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:25 smithi121 ceph-mon[44533]: pgmap v2886: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:25.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:25.926 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:25.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:25.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:26.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:26.083 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:26.105 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:26.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:26.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:26.238 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:26.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3344832057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:26.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:26 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3344832057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:26 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3344832057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:26.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:26.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:26 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:59:27.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:59:27.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:27 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:27.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:27 smithi194 ceph-mon[44089]: pgmap v2887: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:27 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:27 smithi040 ceph-mon[39581]: pgmap v2887: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:27.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:27 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:27.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:27 smithi121 ceph-mon[44533]: pgmap v2887: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:28.439 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:28 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:59:28.402+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1127 definitely_dead 0 2024-04-29T14:59:28.689 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:28.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:28.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:28.827 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:28.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:28.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:28.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:28.985 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:28.997 INFO:teuthology.orchestra.run.smithi040.stderr:osd.1 is already down. 2024-04-29T14:59:29.006 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:29.009 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:29.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2579210040' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-29T14:59:29.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:28 smithi194 ceph-mon[44089]: pgmap v2888: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:29.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:29.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2579210040' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-29T14:59:29.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:28 smithi040 ceph-mon[39581]: pgmap v2888: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2579210040' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-29T14:59:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:28 smithi121 ceph-mon[44533]: pgmap v2888: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:29.142 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:29.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:29.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:29.172 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:29.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:29.302 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:29.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:59:29.627 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [4, 7, 10, 6, 1, 5, 8, 3] 2024-04-29T14:59:29.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd out 1 2024-04-29T14:59:29.853 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:29.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:29.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:29.990 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:30.011 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:30.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:30.115 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:30.130 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:59:30.130 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:59:30.145 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:30.165 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:30.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:30.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:30.297 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:30.317 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:30.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:30.420 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:30.448 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:30.466 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:30.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:30.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:30.602 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:30.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:30.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:30.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:30.759 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:30.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:30.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:30.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:30.914 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:30.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:30.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:31.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:31.066 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:31.085 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:31.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:31.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:31.215 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:31.234 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:31.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:31.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:31.368 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:31.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:31 smithi040 ceph-mon[39581]: pgmap v2889: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:31.755 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:31.757 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:31.795 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-29T14:59:31.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:31 smithi194 ceph-mon[44089]: pgmap v2889: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:31 smithi121 ceph-mon[44533]: pgmap v2889: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:32.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:32.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:32.701 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:32.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:32.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:32.834 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:33.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:33 smithi194 ceph-mon[44089]: pgmap v2890: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:33.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:33 smithi040 ceph-mon[39581]: pgmap v2890: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:33.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:33 smithi121 ceph-mon[44533]: pgmap v2890: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:34.479 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:34.485 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:34.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:marked out osd.1. 2024-04-29T14:59:34.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T14:59:34.328+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1127 definitely_dead 0 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/382521764' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:59:34.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:34 smithi040 ceph-mon[39581]: Client client.admin marked osd.1 out, after it was down for 25 seconds 2024-04-29T14:59:34.624 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:34.647 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:34.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:34.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:34.783 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/382521764' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:59:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:34 smithi194 ceph-mon[44089]: Client client.admin marked osd.1 out, after it was down for 25 seconds 2024-04-29T14:59:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T14:59:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T14:59:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T14:59:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T14:59:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T14:59:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T14:59:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T14:59:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/382521764' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-29T14:59:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:34 smithi121 ceph-mon[44533]: Client client.admin marked osd.1 out, after it was down for 25 seconds 2024-04-29T14:59:35.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:35 smithi040 ceph-mon[39581]: pgmap v2891: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:35.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:35 smithi040 ceph-mon[39581]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:59:35.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/382521764' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-29T14:59:35.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:35 smithi040 ceph-mon[39581]: osdmap e1128: 12 total, 11 up, 7 in 2024-04-29T14:59:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:35 smithi194 ceph-mon[44089]: pgmap v2891: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:35 smithi194 ceph-mon[44089]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:59:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/382521764' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-29T14:59:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:35 smithi194 ceph-mon[44089]: osdmap e1128: 12 total, 11 up, 7 in 2024-04-29T14:59:35.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:35 smithi121 ceph-mon[44533]: pgmap v2891: 17 pgs: 6 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 24323/129240 objects degraded (18.820%) 2024-04-29T14:59:35.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:35 smithi121 ceph-mon[44533]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-29T14:59:35.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/382521764' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-29T14:59:35.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:35 smithi121 ceph-mon[44533]: osdmap e1128: 12 total, 11 up, 7 in 2024-04-29T14:59:36.309 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:59:36.329 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:59:36.348 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:59:36.369 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:59:36.389 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:59:36.409 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:59:36.429 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:59:36.451 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:59:36.472 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:59:36.492 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:59:36.512 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:59:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:59:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T14:59:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:59:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:59:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:59:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:59:36.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:59:36.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:59:36.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:59:36.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:59:36.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:59:36.513 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:59:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:36 smithi194 ceph-mon[44089]: osdmap e1129: 12 total, 11 up, 7 in 2024-04-29T14:59:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:36 smithi194 ceph-mon[44089]: from='client.18921 v1:172.21.15.40:0/2862326736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:36 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:36.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:36 smithi040 ceph-mon[39581]: osdmap e1129: 12 total, 11 up, 7 in 2024-04-29T14:59:36.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:36 smithi040 ceph-mon[39581]: from='client.18921 v1:172.21.15.40:0/2862326736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:36.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:36.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:36 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:36.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:36 smithi121 ceph-mon[44533]: osdmap e1129: 12 total, 11 up, 7 in 2024-04-29T14:59:36.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:36 smithi121 ceph-mon[44533]: from='client.18921 v1:172.21.15.40:0/2862326736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:36.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:36.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:36 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.001 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:59:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-04-29T14:59:37.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDPGPushReply(9.2 1130/1129 [PushReplyOp(9:406e6631:::benchmark_data_smithi194_196924_object115109:head)])", 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-04-29T14:59:36.996534+0000", 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.5463999999999998e-05, 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6589999999999998e-05, 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:59:36.996534+0000", 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:59:36.996534+0000", 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-04-29T14:59:37.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:59:36.996534+0000", 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.82e-07 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:59:36.996537+0000", 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1219999999999999e-06 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:59:36.996545+0000", 2024-04-29T14:59:37.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-06 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:59:36.996552+0000", 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8530000000000003e-06 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-04-29T14:59:36.996588+0000", 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6480000000000003e-05 2024-04-29T14:59:37.005 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:59:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-29T14:59:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:59:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-29T14:59:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-29T14:59:37.006 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-04-29T14:59:37.006 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: pgmap v2894: 17 pgs: 2 remapped+peering, 4 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 16213/129240 objects degraded (12.545%) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 16213/129240 objects degraded (12.545%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: 9.0 starting backfill to osd.8 from (0'0,0'0] MAX to 1121'16096 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: osdmap e1130: 12 total, 11 up, 7 in 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 1121'15904 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: 9.4 starting backfill to osd.8 from (0'0,0'0] MAX to 1121'15952 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 1121'16012 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: 9.f starting backfill to osd.10 from (0'0,0'0] MAX to 1121'15889 2024-04-29T14:59:37.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:37 smithi194 ceph-mon[44089]: 9.d starting backfill to osd.5 from (0'0,0'0] MAX to 1121'15900 2024-04-29T14:59:37.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: pgmap v2894: 17 pgs: 2 remapped+peering, 4 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 16213/129240 objects degraded (12.545%) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 16213/129240 objects degraded (12.545%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: 9.0 starting backfill to osd.8 from (0'0,0'0] MAX to 1121'16096 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: osdmap e1130: 12 total, 11 up, 7 in 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 1121'15904 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: 9.4 starting backfill to osd.8 from (0'0,0'0] MAX to 1121'15952 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 1121'16012 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: 9.f starting backfill to osd.10 from (0'0,0'0] MAX to 1121'15889 2024-04-29T14:59:37.871 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:37 smithi040 ceph-mon[39581]: 9.d starting backfill to osd.5 from (0'0,0'0] MAX to 1121'15900 2024-04-29T14:59:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: pgmap v2894: 17 pgs: 2 remapped+peering, 4 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 16213/129240 objects degraded (12.545%) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 16213/129240 objects degraded (12.545%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: 9.0 starting backfill to osd.8 from (0'0,0'0] MAX to 1121'16096 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: osdmap e1130: 12 total, 11 up, 7 in 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 1121'15904 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: 9.4 starting backfill to osd.8 from (0'0,0'0] MAX to 1121'15952 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 1121'16012 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: 9.f starting backfill to osd.10 from (0'0,0'0] MAX to 1121'15889 2024-04-29T14:59:37.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:37 smithi121 ceph-mon[44533]: 9.d starting backfill to osd.5 from (0'0,0'0] MAX to 1121'15900 2024-04-29T14:59:37.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-29T14:59:38.333 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-04-29T14:59:38.333 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi040.front.sepia.ceph.com 2024-04-29T14:59:38.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.1 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-04-29T14:59:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:38.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:38.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:38.471 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:38.677 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.1/config 2024-04-29T14:59:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:59:38.714 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:59:39.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:39 smithi040 ceph-mon[39581]: pgmap v2896: 17 pgs: 3 remapped+peering, 3 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 12113/129240 objects degraded (9.372%) 2024-04-29T14:59:39.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:39 smithi194 ceph-mon[44089]: pgmap v2896: 17 pgs: 3 remapped+peering, 3 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 12113/129240 objects degraded (9.372%) 2024-04-29T14:59:39.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:39 smithi121 ceph-mon[44533]: pgmap v2896: 17 pgs: 3 remapped+peering, 3 undersized+degraded+peered, 11 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 12113/129240 objects degraded (9.372%) 2024-04-29T14:59:39.922 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:39.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:40.026 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:40.055 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:40.077 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:40.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:40.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:40.212 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:40.287 INFO:teuthology.orchestra.run.smithi040.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:59:40.333 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:40.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:40.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:40.473 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:40.494 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:40.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:40.570 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T14:59:40.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:40.635 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:40.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:40.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:40.714 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T14:59:40.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:40.803 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:40.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2004411930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:59:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:40.831 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2004411930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:59:40.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:40.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:40.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2004411930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T14:59:40.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:40.964 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:41.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:41 smithi194 ceph-mon[44089]: pgmap v2897: 17 pgs: 3 undersized+degraded+remapped+backfilling+peered, 3 remapped+peering, 11 active+clean; 3.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 8948/129240 objects degraded (6.924%); 33 MiB/s, 527 objects/s recovering 2024-04-29T14:59:41.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:41 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:41.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2004411930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:59:41.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:41 smithi194 ceph-mon[44089]: osdmap e1131: 12 total, 11 up, 7 in 2024-04-29T14:59:41.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:41 smithi040 ceph-mon[39581]: pgmap v2897: 17 pgs: 3 undersized+degraded+remapped+backfilling+peered, 3 remapped+peering, 11 active+clean; 3.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 8948/129240 objects degraded (6.924%); 33 MiB/s, 527 objects/s recovering 2024-04-29T14:59:41.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:41 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:41.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2004411930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:59:41.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:41 smithi040 ceph-mon[39581]: osdmap e1131: 12 total, 11 up, 7 in 2024-04-29T14:59:41.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:41 smithi121 ceph-mon[44533]: pgmap v2897: 17 pgs: 3 undersized+degraded+remapped+backfilling+peered, 3 remapped+peering, 11 active+clean; 3.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 8948/129240 objects degraded (6.924%); 33 MiB/s, 527 objects/s recovering 2024-04-29T14:59:41.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:41 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:41.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2004411930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T14:59:41.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:41 smithi121 ceph-mon[44533]: osdmap e1131: 12 total, 11 up, 7 in 2024-04-29T14:59:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:42 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 15169/129240 objects degraded (11.737%), 6 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:42.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:42 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 15169/129240 objects degraded (11.737%), 6 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:43.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:42 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 15169/129240 objects degraded (11.737%), 6 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:43.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:43 smithi040 ceph-mon[39581]: pgmap v2899: 17 pgs: 6 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 15169/129240 objects degraded (11.737%); 83 MiB/s, 1.33k objects/s recovering 2024-04-29T14:59:43.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:43 smithi121 ceph-mon[44533]: pgmap v2899: 17 pgs: 6 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 15169/129240 objects degraded (11.737%); 83 MiB/s, 1.33k objects/s recovering 2024-04-29T14:59:43.996 INFO:teuthology.orchestra.run.smithi040.stdout:9.a 2024-04-29T14:59:43.997 INFO:teuthology.orchestra.run.smithi040.stdout:9.0 2024-04-29T14:59:43.997 INFO:teuthology.orchestra.run.smithi040.stdout:9.2 2024-04-29T14:59:43.997 INFO:teuthology.orchestra.run.smithi040.stdout:9.4 2024-04-29T14:59:43.997 INFO:teuthology.orchestra.run.smithi040.stdout:9.d 2024-04-29T14:59:43.997 INFO:teuthology.orchestra.run.smithi040.stdout:9.f 2024-04-29T14:59:44.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:43 smithi194 ceph-mon[44089]: pgmap v2899: 17 pgs: 6 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 15169/129240 objects degraded (11.737%); 83 MiB/s, 1.33k objects/s recovering 2024-04-29T14:59:45.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.1 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 9.2 --file /var/log/ceph/exp.9.2.1 2024-04-29T14:59:45.757 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:45.759 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:45 smithi040 ceph-mon[39581]: pgmap v2900: 17 pgs: 6 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 8263/129240 objects degraded (6.394%); 125 MiB/s, 2.01k objects/s recovering 2024-04-29T14:59:45.760 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.1/config 2024-04-29T14:59:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:45.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:45.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:45 smithi121 ceph-mon[44533]: pgmap v2900: 17 pgs: 6 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 8263/129240 objects degraded (6.394%); 125 MiB/s, 2.01k objects/s recovering 2024-04-29T14:59:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:46.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:45 smithi194 ceph-mon[44089]: pgmap v2900: 17 pgs: 6 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 8263/129240 objects degraded (6.394%); 125 MiB/s, 2.01k objects/s recovering 2024-04-29T14:59:46.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T14:59:46.758 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:46 smithi040 ceph-mon[39581]: osdmap e1132: 12 total, 11 up, 7 in 2024-04-29T14:59:46.759 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:46 smithi040 ceph-mon[39581]: osdmap e1133: 12 total, 11 up, 7 in 2024-04-29T14:59:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:46 smithi121 ceph-mon[44533]: osdmap e1132: 12 total, 11 up, 7 in 2024-04-29T14:59:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:46 smithi121 ceph-mon[44533]: osdmap e1133: 12 total, 11 up, 7 in 2024-04-29T14:59:47.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:46 smithi194 ceph-mon[44089]: osdmap e1132: 12 total, 11 up, 7 in 2024-04-29T14:59:47.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:46 smithi194 ceph-mon[44089]: osdmap e1133: 12 total, 11 up, 7 in 2024-04-29T14:59:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:47.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:47.453 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:47.485 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:47.507 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:47.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:47.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:47.687 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:47.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T14:59:47.791 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T14:59:47.791 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T14:59:47.791 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T14:59:47.791 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T14:59:47.795 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:47.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:47 smithi194 ceph-mon[44089]: pgmap v2902: 17 pgs: 1 undersized+remapped+peered, 1 peering, 4 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 3926/129240 objects degraded (3.038%); 159 MiB/s, 2.55k objects/s recovering 2024-04-29T14:59:47.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:47 smithi194 ceph-mon[44089]: osdmap e1134: 12 total, 11 up, 7 in 2024-04-29T14:59:47.828 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:47.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:47 smithi121 ceph-mon[44533]: pgmap v2902: 17 pgs: 1 undersized+remapped+peered, 1 peering, 4 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 3926/129240 objects degraded (3.038%); 159 MiB/s, 2.55k objects/s recovering 2024-04-29T14:59:47.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:47 smithi121 ceph-mon[44533]: osdmap e1134: 12 total, 11 up, 7 in 2024-04-29T14:59:48.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:47 smithi040 ceph-mon[39581]: pgmap v2902: 17 pgs: 1 undersized+remapped+peered, 1 peering, 4 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 3926/129240 objects degraded (3.038%); 159 MiB/s, 2.55k objects/s recovering 2024-04-29T14:59:48.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:47 smithi040 ceph-mon[39581]: osdmap e1134: 12 total, 11 up, 7 in 2024-04-29T14:59:48.182 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:48.184 INFO:teuthology.orchestra.run.smithi040.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:59:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:48.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:48.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:48.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:48.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:48.590 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:48.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:48.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:48.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:48 smithi040 ceph-mon[39581]: Health check update: Degraded data redundancy: 3926/129240 objects degraded (3.038%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:48.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T14:59:48.758 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:48.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:48 smithi121 ceph-mon[44533]: Health check update: Degraded data redundancy: 3926/129240 objects degraded (3.038%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:49.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:48 smithi194 ceph-mon[44089]: Health check update: Degraded data redundancy: 3926/129240 objects degraded (3.038%), 4 pgs degraded (PG_DEGRADED) 2024-04-29T14:59:49.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-29T14:59:49.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:49 smithi040 ceph-mon[39581]: pgmap v2905: 17 pgs: 2 undersized+remapped+peered, 1 peering, 3 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2042/129240 objects degraded (1.580%); 137 MiB/s, 2.19k objects/s recovering 2024-04-29T14:59:50.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:49 smithi194 ceph-mon[44089]: pgmap v2905: 17 pgs: 2 undersized+remapped+peered, 1 peering, 3 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2042/129240 objects degraded (1.580%); 137 MiB/s, 2.19k objects/s recovering 2024-04-29T14:59:50.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:49 smithi121 ceph-mon[44533]: pgmap v2905: 17 pgs: 2 undersized+remapped+peered, 1 peering, 3 undersized+degraded+remapped+backfilling+peered, 11 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2042/129240 objects degraded (1.580%); 137 MiB/s, 2.19k objects/s recovering 2024-04-29T14:59:50.633 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:50.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:50 smithi040 ceph-mon[39581]: pgmap v2906: 17 pgs: 2 undersized+remapped+peered, 1 peering, 14 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 160 op/s; 86 MiB/s, 1.38k objects/s recovering 2024-04-29T14:59:50.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:50 smithi040 ceph-mon[39581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2042/129240 objects degraded (1.580%), 3 pgs degraded) 2024-04-29T14:59:51.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:50 smithi194 ceph-mon[44089]: pgmap v2906: 17 pgs: 2 undersized+remapped+peered, 1 peering, 14 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 160 op/s; 86 MiB/s, 1.38k objects/s recovering 2024-04-29T14:59:51.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:50 smithi194 ceph-mon[44089]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2042/129240 objects degraded (1.580%), 3 pgs degraded) 2024-04-29T14:59:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:50 smithi121 ceph-mon[44533]: pgmap v2906: 17 pgs: 2 undersized+remapped+peered, 1 peering, 14 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 160 op/s; 86 MiB/s, 1.38k objects/s recovering 2024-04-29T14:59:51.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:50 smithi121 ceph-mon[44533]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2042/129240 objects degraded (1.580%), 3 pgs degraded) 2024-04-29T14:59:51.417 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T14:59:51.438 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T14:59:51.472 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T14:59:51.513 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T14:59:51.543 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T14:59:51.584 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T14:59:51.622 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T14:59:51.643 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T14:59:51.665 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T14:59:51.688 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T14:59:51.710 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T14:59:51.710 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T14:59:51.710 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T14:59:51.710 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T14:59:51.710 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T14:59:51.710 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T14:59:51.710 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T14:59:51.711 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T14:59:51.711 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T14:59:51.711 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T14:59:51.711 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T14:59:51.711 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T14:59:51.711 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T14:59:51.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/292189788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:51.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:51.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/292189788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:52.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/292189788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T14:59:52.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T14:59:52.599 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T14:59:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:Exporting 9.2 info 9.2( v 1121'15904 (1083'5900,1121'15904] local-lis/les=1065/1066 n=3974 ec=1065/1065 lis/c=1065/1065 les/c/f=1066/1066/0 sis=1065) 2024-04-29T14:59:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40017731:::benchmark_data_smithi194_196924_object139562:head# 2024-04-29T14:59:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4001aa42:::benchmark_data_smithi194_196924_object142247:head# 2024-04-29T14:59:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40029627:::benchmark_data_smithi194_196924_object102988:head# 2024-04-29T14:59:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4003f7ea:::benchmark_data_smithi194_196924_object106917:head# 2024-04-29T14:59:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4004d802:::benchmark_data_smithi194_196924_object146458:head# 2024-04-29T14:59:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4005d108:::benchmark_data_smithi194_196924_object157693:head# 2024-04-29T14:59:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:400a40f9:::benchmark_data_smithi194_196924_object123341:head# 2024-04-29T14:59:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:400aba72:::benchmark_data_smithi194_196924_object103957:head# 2024-04-29T14:59:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:400b75ed:::benchmark_data_smithi194_196924_object141664:head# 2024-04-29T14:59:52.660 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:400cad17:::benchmark_data_smithi194_196924_object96203:head# 2024-04-29T14:59:52.660 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:400cd5c2:::benchmark_data_smithi194_196924_object138892:head# 2024-04-29T14:59:52.661 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:400df348:::benchmark_data_smithi194_196924_object135565:head# 2024-04-29T14:59:52.661 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:400eb2b7:::benchmark_data_smithi194_196924_object114768:head# 2024-04-29T14:59:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4011b6ab:::benchmark_data_smithi194_196924_object144287:head# 2024-04-29T14:59:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4013b8b6:::benchmark_data_smithi194_196924_object143665:head# 2024-04-29T14:59:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4013f288:::benchmark_data_smithi194_196924_object132940:head# 2024-04-29T14:59:52.663 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4014ccbb:::benchmark_data_smithi194_196924_object124150:head# 2024-04-29T14:59:52.665 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:401591ec:::benchmark_data_smithi194_196924_object155809:head# 2024-04-29T14:59:52.665 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40178eb3:::benchmark_data_smithi194_196924_object114232:head# 2024-04-29T14:59:52.665 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4017fd06:::benchmark_data_smithi194_196924_object119273:head# 2024-04-29T14:59:52.666 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40180621:::benchmark_data_smithi194_196924_object101811:head# 2024-04-29T14:59:52.666 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4018738c:::benchmark_data_smithi194_196924_object103666:head# 2024-04-29T14:59:52.666 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:401afddd:::benchmark_data_smithi194_196924_object150888:head# 2024-04-29T14:59:52.667 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:401cc8a1:::benchmark_data_smithi194_196924_object119323:head# 2024-04-29T14:59:52.667 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:401de47a:::benchmark_data_smithi194_196924_object147500:head# 2024-04-29T14:59:52.668 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:401e7678:::benchmark_data_smithi194_196924_object157873:head# 2024-04-29T14:59:52.668 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:401f0566:::benchmark_data_smithi194_196924_object97111:head# 2024-04-29T14:59:52.669 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40202dc7:::benchmark_data_smithi194_196924_object143317:head# 2024-04-29T14:59:52.669 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40213a71:::benchmark_data_smithi194_196924_object98745:head# 2024-04-29T14:59:52.669 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4021f7e9:::benchmark_data_smithi194_196924_object104611:head# 2024-04-29T14:59:52.670 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4024afae:::benchmark_data_smithi194_196924_object100841:head# 2024-04-29T14:59:52.670 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4024bc0f:::benchmark_data_smithi194_196924_object112871:head# 2024-04-29T14:59:52.671 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4027164c:::benchmark_data_smithi194_196924_object136833:head# 2024-04-29T14:59:52.671 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4029aa73:::benchmark_data_smithi194_196924_object142969:head# 2024-04-29T14:59:52.672 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402b17ce:::benchmark_data_smithi194_196924_object137827:head# 2024-04-29T14:59:52.672 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402cc30f:::benchmark_data_smithi194_196924_object140538:head# 2024-04-29T14:59:52.673 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402db6ca:::benchmark_data_smithi194_196924_object153577:head# 2024-04-29T14:59:52.673 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402e2d59:::benchmark_data_smithi194_196924_object152693:head# 2024-04-29T14:59:52.674 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402e41e0:::benchmark_data_smithi194_196924_object144797:head# 2024-04-29T14:59:52.674 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402e5a79:::benchmark_data_smithi194_196924_object157139:head# 2024-04-29T14:59:52.675 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402f1161:::benchmark_data_smithi194_196924_object148124:head# 2024-04-29T14:59:52.676 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:402f7e8b:::benchmark_data_smithi194_196924_object100572:head# 2024-04-29T14:59:52.676 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40304b51:::benchmark_data_smithi194_196924_object112895:head# 2024-04-29T14:59:52.676 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:403064d1:::benchmark_data_smithi194_196924_object156498:head# 2024-04-29T14:59:52.677 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4030d584:::benchmark_data_smithi194_196924_object136880:head# 2024-04-29T14:59:52.677 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40310fb1:::benchmark_data_smithi194_196924_object107940:head# 2024-04-29T14:59:52.678 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40315a79:::benchmark_data_smithi194_196924_object109812:head# 2024-04-29T14:59:52.678 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4032cc55:::benchmark_data_smithi194_196924_object138109:head# 2024-04-29T14:59:52.678 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4033013e:::benchmark_data_smithi194_196924_object128016:head# 2024-04-29T14:59:52.679 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4033376b:::benchmark_data_smithi194_196924_object110778:head# 2024-04-29T14:59:52.679 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4033ad9f:::benchmark_data_smithi194_196924_object147568:head# 2024-04-29T14:59:52.680 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:403574b6:::benchmark_data_smithi194_196924_object159160:head# 2024-04-29T14:59:52.680 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4035aa10:::benchmark_data_smithi194_196924_object144026:head# 2024-04-29T14:59:52.681 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4035e472:::benchmark_data_smithi194_196924_object148439:head# 2024-04-29T14:59:52.681 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40366012:::benchmark_data_smithi194_196924_object97436:head# 2024-04-29T14:59:52.681 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4036b563:::benchmark_data_smithi194_196924_object143190:head# 2024-04-29T14:59:52.682 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40380e46:::benchmark_data_smithi194_196924_object119983:head# 2024-04-29T14:59:52.682 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:403ad1e2:::benchmark_data_smithi194_196924_object95975:head# 2024-04-29T14:59:52.682 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:403be672:::benchmark_data_smithi194_196924_object156333:head# 2024-04-29T14:59:52.683 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:403cb3e0:::benchmark_data_smithi194_196924_object98744:head# 2024-04-29T14:59:52.683 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:403cb6c7:::benchmark_data_smithi194_196924_object130644:head# 2024-04-29T14:59:52.684 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:403d54c5:::benchmark_data_smithi194_196924_object116977:head# 2024-04-29T14:59:52.684 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4040c71a:::benchmark_data_smithi194_196924_object121606:head# 2024-04-29T14:59:52.684 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4040dd3d:::benchmark_data_smithi194_196924_object129766:head# 2024-04-29T14:59:52.685 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40414d1d:::benchmark_data_smithi194_196924_object126689:head# 2024-04-29T14:59:52.686 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4041b2ff:::benchmark_data_smithi194_196924_object112319:head# 2024-04-29T14:59:52.686 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4041bf94:::benchmark_data_smithi194_196924_object97726:head# 2024-04-29T14:59:52.686 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4041ea51:::benchmark_data_smithi194_196924_object113272:head# 2024-04-29T14:59:52.687 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40426f0d:::benchmark_data_smithi194_196924_object128560:head# 2024-04-29T14:59:52.687 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404297b4:::benchmark_data_smithi194_196924_object122071:head# 2024-04-29T14:59:52.688 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4042e074:::benchmark_data_smithi194_196924_object97574:head# 2024-04-29T14:59:52.688 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40438855:::benchmark_data_smithi194_196924_object97320:head# 2024-04-29T14:59:52.688 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404407b4:::benchmark_data_smithi194_196924_object129050:head# 2024-04-29T14:59:52.689 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404409f7:::benchmark_data_smithi194_196924_object158501:head# 2024-04-29T14:59:52.689 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404427ac:::benchmark_data_smithi194_196924_object96329:head# 2024-04-29T14:59:52.690 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40446f80:::benchmark_data_smithi194_196924_object126500:head# 2024-04-29T14:59:52.690 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404487b7:::benchmark_data_smithi194_196924_object126914:head# 2024-04-29T14:59:52.691 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4045e3ce:::benchmark_data_smithi194_196924_object97332:head# 2024-04-29T14:59:52.691 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40460c97:::benchmark_data_smithi194_196924_object130207:head# 2024-04-29T14:59:52.692 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40464259:::benchmark_data_smithi194_196924_object101463:head# 2024-04-29T14:59:52.692 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404785db:::benchmark_data_smithi194_196924_object124472:head# 2024-04-29T14:59:52.693 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404cecc3:::benchmark_data_smithi194_196924_object139534:head# 2024-04-29T14:59:52.693 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404d7b26:::benchmark_data_smithi194_196924_object156878:head# 2024-04-29T14:59:52.693 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404da09d:::benchmark_data_smithi194_196924_object131555:head# 2024-04-29T14:59:52.694 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404e7694:::benchmark_data_smithi194_196924_object100058:head# 2024-04-29T14:59:52.695 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:404f960c:::benchmark_data_smithi194_196924_object120008:head# 2024-04-29T14:59:52.697 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40514bea:::benchmark_data_smithi194_196924_object157346:head# 2024-04-29T14:59:52.697 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40516d9c:::benchmark_data_smithi194_196924_object123148:head# 2024-04-29T14:59:52.697 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4052856f:::benchmark_data_smithi194_196924_object149337:head# 2024-04-29T14:59:52.698 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40570e02:::benchmark_data_smithi194_196924_object128757:head# 2024-04-29T14:59:52.698 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:405a5bfc:::benchmark_data_smithi194_196924_object151412:head# 2024-04-29T14:59:52.699 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:405ba788:::benchmark_data_smithi194_196924_object139527:head# 2024-04-29T14:59:52.699 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:405cf5cb:::benchmark_data_smithi194_196924_object132890:head# 2024-04-29T14:59:52.699 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:405ef8e0:::benchmark_data_smithi194_196924_object106505:head# 2024-04-29T14:59:52.700 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:405f9acb:::benchmark_data_smithi194_196924_object144954:head# 2024-04-29T14:59:52.700 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:405fd5b8:::benchmark_data_smithi194_196924_object140240:head# 2024-04-29T14:59:52.700 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40601fd7:::benchmark_data_smithi194_196924_object146371:head# 2024-04-29T14:59:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406025ef:::benchmark_data_smithi194_196924_object126130:head# 2024-04-29T14:59:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406029f0:::benchmark_data_smithi194_196924_object103281:head# 2024-04-29T14:59:52.702 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40614826:::benchmark_data_smithi194_196924_object112839:head# 2024-04-29T14:59:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40619c68:::benchmark_data_smithi194_196924_object120567:head# 2024-04-29T14:59:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40641e48:::benchmark_data_smithi194_196924_object102505:head# 2024-04-29T14:59:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40650bd0:::benchmark_data_smithi194_196924_object153230:head# 2024-04-29T14:59:52.704 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406871eb:::benchmark_data_smithi194_196924_object101198:head# 2024-04-29T14:59:52.704 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4068d34a:::benchmark_data_smithi194_196924_object121110:head# 2024-04-29T14:59:52.705 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406998d8:::benchmark_data_smithi194_196924_object117560:head# 2024-04-29T14:59:52.705 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406a0fc2:::benchmark_data_smithi194_196924_object121838:head# 2024-04-29T14:59:52.706 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406ac8d4:::benchmark_data_smithi194_196924_object104174:head# 2024-04-29T14:59:52.706 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406b4157:::benchmark_data_smithi194_196924_object148760:head# 2024-04-29T14:59:52.706 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406b5f38:::benchmark_data_smithi194_196924_object120040:head# 2024-04-29T14:59:52.707 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406db30f:::benchmark_data_smithi194_196924_object100911:head# 2024-04-29T14:59:52.707 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406e6197:::benchmark_data_smithi194_196924_object115797:head# 2024-04-29T14:59:52.708 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406e6631:::benchmark_data_smithi194_196924_object115109:head# 2024-04-29T14:59:52.708 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406ec7d6:::benchmark_data_smithi194_196924_object144536:head# 2024-04-29T14:59:52.708 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406f30c1:::benchmark_data_smithi194_196924_object146287:head# 2024-04-29T14:59:52.709 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:406f7138:::benchmark_data_smithi194_196924_object124794:head# 2024-04-29T14:59:52.709 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40701dc0:::benchmark_data_smithi194_196924_object146954:head# 2024-04-29T14:59:52.710 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40706b7b:::benchmark_data_smithi194_196924_object125614:head# 2024-04-29T14:59:52.710 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4070a052:::benchmark_data_smithi194_196924_object125243:head# 2024-04-29T14:59:52.710 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4070d44f:::benchmark_data_smithi194_196924_object130513:head# 2024-04-29T14:59:52.711 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40734d26:::benchmark_data_smithi194_196924_object112702:head# 2024-04-29T14:59:52.711 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40744aee:::benchmark_data_smithi194_196924_object140388:head# 2024-04-29T14:59:52.712 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4075b4ea:::benchmark_data_smithi194_196924_object127663:head# 2024-04-29T14:59:52.712 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4075f79d:::benchmark_data_smithi194_196924_object117139:head# 2024-04-29T14:59:52.713 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4076d256:::benchmark_data_smithi194_196924_object158247:head# 2024-04-29T14:59:52.713 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:407756ae:::benchmark_data_smithi194_196924_object146491:head# 2024-04-29T14:59:52.714 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4077e945:::benchmark_data_smithi194_196924_object124218:head# 2024-04-29T14:59:52.714 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4077ed58:::benchmark_data_smithi194_196924_object108061:head# 2024-04-29T14:59:52.714 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:407a4e63:::benchmark_data_smithi194_196924_object141512:head# 2024-04-29T14:59:52.715 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:407aa3af:::benchmark_data_smithi194_196924_object110281:head# 2024-04-29T14:59:52.715 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:407d8e51:::benchmark_data_smithi194_196924_object126622:head# 2024-04-29T14:59:52.716 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:407eb087:::benchmark_data_smithi194_196924_object132150:head# 2024-04-29T14:59:52.717 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:407eb807:::benchmark_data_smithi194_196924_object126451:head# 2024-04-29T14:59:52.717 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:407f34a7:::benchmark_data_smithi194_196924_object113338:head# 2024-04-29T14:59:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4080ede9:::benchmark_data_smithi194_196924_object148235:head# 2024-04-29T14:59:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4081a642:::benchmark_data_smithi194_196924_object97397:head# 2024-04-29T14:59:52.719 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40833d75:::benchmark_data_smithi194_196924_object118046:head# 2024-04-29T14:59:52.719 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40861d93:::benchmark_data_smithi194_196924_object128101:head# 2024-04-29T14:59:52.719 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:408693de:::benchmark_data_smithi194_196924_object128030:head# 2024-04-29T14:59:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:408827d5:::benchmark_data_smithi194_196924_object104517:head# 2024-04-29T14:59:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40893378:::benchmark_data_smithi194_196924_object147995:head# 2024-04-29T14:59:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:408d4ddc:::benchmark_data_smithi194_196924_object118069:head# 2024-04-29T14:59:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:408d74e6:::benchmark_data_smithi194_196924_object129456:head# 2024-04-29T14:59:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:408dfe72:::benchmark_data_smithi194_196924_object155910:head# 2024-04-29T14:59:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:408edad6:::benchmark_data_smithi194_196924_object159596:head# 2024-04-29T14:59:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:408f61c9:::benchmark_data_smithi194_196924_object125548:head# 2024-04-29T14:59:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4094a335:::benchmark_data_smithi194_196924_object97324:head# 2024-04-29T14:59:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40971c08:::benchmark_data_smithi194_196924_object138867:head# 2024-04-29T14:59:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4097b0a7:::benchmark_data_smithi194_196924_object97357:head# 2024-04-29T14:59:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4098159b:::benchmark_data_smithi194_196924_object116459:head# 2024-04-29T14:59:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40982b2f:::benchmark_data_smithi194_196924_object130200:head# 2024-04-29T14:59:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40983e2c:::benchmark_data_smithi194_196924_object148281:head# 2024-04-29T14:59:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4099d74d:::benchmark_data_smithi194_196924_object124066:head# 2024-04-29T14:59:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:409a3029:::benchmark_data_smithi194_196924_object104243:head# 2024-04-29T14:59:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:409a3aad:::benchmark_data_smithi194_196924_object158638:head# 2024-04-29T14:59:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:409b19de:::benchmark_data_smithi194_196924_object152481:head# 2024-04-29T14:59:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:409e6503:::benchmark_data_smithi194_196924_object110657:head# 2024-04-29T14:59:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:409f5f2e:::benchmark_data_smithi194_196924_object98583:head# 2024-04-29T14:59:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a100eb:::benchmark_data_smithi194_196924_object107037:head# 2024-04-29T14:59:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a1adf4:::benchmark_data_smithi194_196924_object120159:head# 2024-04-29T14:59:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a22d95:::benchmark_data_smithi194_196924_object98016:head# 2024-04-29T14:59:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a24479:::benchmark_data_smithi194_196924_object110277:head# 2024-04-29T14:59:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a41a84:::benchmark_data_smithi194_196924_object101540:head# 2024-04-29T14:59:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a57a99:::benchmark_data_smithi194_196924_object129515:head# 2024-04-29T14:59:52.731 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a5d261:::benchmark_data_smithi194_196924_object132250:head# 2024-04-29T14:59:52.731 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a5eb42:::benchmark_data_smithi194_196924_object104860:head# 2024-04-29T14:59:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a6bd04:::benchmark_data_smithi194_196924_object153179:head# 2024-04-29T14:59:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a787d2:::benchmark_data_smithi194_196924_object112717:head# 2024-04-29T14:59:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a7a620:::benchmark_data_smithi194_196924_object100990:head# 2024-04-29T14:59:52.733 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a7f40c:::benchmark_data_smithi194_196924_object121479:head# 2024-04-29T14:59:52.733 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a85b34:::benchmark_data_smithi194_196924_object149501:head# 2024-04-29T14:59:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a879df:::benchmark_data_smithi194_196924_object131702:head# 2024-04-29T14:59:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40a989cc:::benchmark_data_smithi194_196924_object145551:head# 2024-04-29T14:59:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40aac509:::benchmark_data_smithi194_196924_object96113:head# 2024-04-29T14:59:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40ac25e6:::benchmark_data_smithi194_196924_object110327:head# 2024-04-29T14:59:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40acdc68:::benchmark_data_smithi194_196924_object150116:head# 2024-04-29T14:59:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40aeb4b4:::benchmark_data_smithi194_196924_object130003:head# 2024-04-29T14:59:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b05f11:::benchmark_data_smithi194_196924_object109778:head# 2024-04-29T14:59:52.738 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b15d75:::benchmark_data_smithi194_196924_object123259:head# 2024-04-29T14:59:52.738 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b1bb55:::benchmark_data_smithi194_196924_object139256:head# 2024-04-29T14:59:52.739 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b1db12:::benchmark_data_smithi194_196924_object123157:head# 2024-04-29T14:59:52.739 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b1fa8c:::benchmark_data_smithi194_196924_object99634:head# 2024-04-29T14:59:52.739 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b299b3:::benchmark_data_smithi194_196924_object114150:head# 2024-04-29T14:59:52.740 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b4fbd8:::benchmark_data_smithi194_196924_object108412:head# 2024-04-29T14:59:52.740 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b50d69:::benchmark_data_smithi194_196924_object149876:head# 2024-04-29T14:59:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b5ab8b:::benchmark_data_smithi194_196924_object98400:head# 2024-04-29T14:59:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40b7fdc7:::benchmark_data_smithi194_196924_object138617:head# 2024-04-29T14:59:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40babddc:::benchmark_data_smithi194_196924_object154103:head# 2024-04-29T14:59:52.742 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40bb0780:::benchmark_data_smithi194_196924_object118133:head# 2024-04-29T14:59:52.742 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40bd169b:::benchmark_data_smithi194_196924_object120266:head# 2024-04-29T14:59:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40bd1a46:::benchmark_data_smithi194_196924_object111241:head# 2024-04-29T14:59:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40beb6b4:::benchmark_data_smithi194_196924_object122108:head# 2024-04-29T14:59:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40c0fa09:::benchmark_data_smithi194_196924_object152867:head# 2024-04-29T14:59:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40c2eb54:::benchmark_data_smithi194_196924_object143900:head# 2024-04-29T14:59:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40c61fff:::benchmark_data_smithi194_196924_object148744:head# 2024-04-29T14:59:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40c719f8:::benchmark_data_smithi194_196924_object122448:head# 2024-04-29T14:59:52.746 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40c747d4:::benchmark_data_smithi194_196924_object128923:head# 2024-04-29T14:59:52.746 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40c7bb7f:::benchmark_data_smithi194_196924_object110870:head# 2024-04-29T14:59:52.746 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40ced2f9:::benchmark_data_smithi194_196924_object109966:head# 2024-04-29T14:59:52.747 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40cf775e:::benchmark_data_smithi194_196924_object119947:head# 2024-04-29T14:59:52.747 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40d1f962:::benchmark_data_smithi194_196924_object155275:head# 2024-04-29T14:59:52.747 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40d37ac9:::benchmark_data_smithi194_196924_object146191:head# 2024-04-29T14:59:52.748 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40d62cbd:::benchmark_data_smithi194_196924_object141003:head# 2024-04-29T14:59:52.748 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40d74936:::benchmark_data_smithi194_196924_object95759:head# 2024-04-29T14:59:52.749 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40d910f6:::benchmark_data_smithi194_196924_object147302:head# 2024-04-29T14:59:52.749 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40d93e38:::benchmark_data_smithi194_196924_object154164:head# 2024-04-29T14:59:52.750 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40daacd9:::benchmark_data_smithi194_196924_object153904:head# 2024-04-29T14:59:52.750 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40dbbcf7:::benchmark_data_smithi194_196924_object141312:head# 2024-04-29T14:59:52.751 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40dbf7e3:::benchmark_data_smithi194_196924_object122697:head# 2024-04-29T14:59:52.751 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40dc5495:::benchmark_data_smithi194_196924_object138878:head# 2024-04-29T14:59:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40dc6f75:::benchmark_data_smithi194_196924_object137661:head# 2024-04-29T14:59:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40dda1b9:::benchmark_data_smithi194_196924_object152682:head# 2024-04-29T14:59:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40de08b1:::benchmark_data_smithi194_196924_object107354:head# 2024-04-29T14:59:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40de4827:::benchmark_data_smithi194_196924_object128384:head# 2024-04-29T14:59:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40e098d2:::benchmark_data_smithi194_196924_object155846:head# 2024-04-29T14:59:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40e51123:::benchmark_data_smithi194_196924_object126428:head# 2024-04-29T14:59:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40e83302:::benchmark_data_smithi194_196924_object136232:head# 2024-04-29T14:59:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40e870d1:::benchmark_data_smithi194_196924_object152745:head# 2024-04-29T14:59:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40e89fae:::benchmark_data_smithi194_196924_object112898:head# 2024-04-29T14:59:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40e8b02c:::benchmark_data_smithi194_196924_object149221:head# 2024-04-29T14:59:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40ebb7ed:::benchmark_data_smithi194_196924_object106745:head# 2024-04-29T14:59:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40ebed08:::benchmark_data_smithi194_196924_object100027:head# 2024-04-29T14:59:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40ecb8d5:::benchmark_data_smithi194_196924_object136938:head# 2024-04-29T14:59:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40ed8845:::benchmark_data_smithi194_196924_object116409:head# 2024-04-29T14:59:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40f17dd8:::benchmark_data_smithi194_196924_object133318:head# 2024-04-29T14:59:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40f1e425:::benchmark_data_smithi194_196924_object115495:head# 2024-04-29T14:59:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40f28493:::benchmark_data_smithi194_196924_object156830:head# 2024-04-29T14:59:52.759 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40f2fd94:::benchmark_data_smithi194_196924_object156962:head# 2024-04-29T14:59:52.759 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40f395b1:::benchmark_data_smithi194_196924_object95938:head# 2024-04-29T14:59:52.760 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40f81e56:::benchmark_data_smithi194_196924_object99178:head# 2024-04-29T14:59:52.761 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40f8cfd0:::benchmark_data_smithi194_196924_object138533:head# 2024-04-29T14:59:52.761 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40fbf379:::benchmark_data_smithi194_196924_object134685:head# 2024-04-29T14:59:52.762 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40fd6a5f:::benchmark_data_smithi194_196924_object137337:head# 2024-04-29T14:59:52.762 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40fdd6bf:::benchmark_data_smithi194_196924_object102022:head# 2024-04-29T14:59:52.762 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:40fe11b0:::benchmark_data_smithi194_196924_object112971:head# 2024-04-29T14:59:52.763 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41008e51:::benchmark_data_smithi194_196924_object97669:head# 2024-04-29T14:59:52.763 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4100bc5e:::benchmark_data_smithi194_196924_object136348:head# 2024-04-29T14:59:52.763 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41017ac9:::benchmark_data_smithi194_196924_object134123:head# 2024-04-29T14:59:52.764 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41040689:::benchmark_data_smithi194_196924_object153720:head# 2024-04-29T14:59:52.765 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4104ea44:::benchmark_data_smithi194_196924_object99786:head# 2024-04-29T14:59:52.765 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4106806f:::benchmark_data_smithi194_196924_object150843:head# 2024-04-29T14:59:52.766 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4106e2a5:::benchmark_data_smithi194_196924_object126859:head# 2024-04-29T14:59:52.766 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41089211:::benchmark_data_smithi194_196924_object151991:head# 2024-04-29T14:59:52.767 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:410945a1:::benchmark_data_smithi194_196924_object138907:head# 2024-04-29T14:59:52.767 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41096871:::benchmark_data_smithi194_196924_object113470:head# 2024-04-29T14:59:52.768 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41098811:::benchmark_data_smithi194_196924_object123185:head# 2024-04-29T14:59:52.768 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:410b3885:::benchmark_data_smithi194_196924_object131047:head# 2024-04-29T14:59:52.769 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:410b9146:::benchmark_data_smithi194_196924_object139509:head# 2024-04-29T14:59:52.769 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:410d3442:::benchmark_data_smithi194_196924_object120852:head# 2024-04-29T14:59:52.769 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:410d3bd1:::benchmark_data_smithi194_196924_object134693:head# 2024-04-29T14:59:52.770 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:410dfe1d:::benchmark_data_smithi194_196924_object141606:head# 2024-04-29T14:59:52.771 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:410e11de:::benchmark_data_smithi194_196924_object138955:head# 2024-04-29T14:59:52.771 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4110cbcb:::benchmark_data_smithi194_196924_object149459:head# 2024-04-29T14:59:52.771 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41135c91:::benchmark_data_smithi194_196924_object158593:head# 2024-04-29T14:59:52.772 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4113b86e:::benchmark_data_smithi194_196924_object108414:head# 2024-04-29T14:59:52.772 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4114d6fd:::benchmark_data_smithi194_196924_object111941:head# 2024-04-29T14:59:52.773 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4114f8e1:::benchmark_data_smithi194_196924_object105061:head# 2024-04-29T14:59:52.773 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4116fc86:::benchmark_data_smithi194_196924_object133411:head# 2024-04-29T14:59:52.774 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41171918:::benchmark_data_smithi194_196924_object135726:head# 2024-04-29T14:59:52.774 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4117d3d3:::benchmark_data_smithi194_196924_object153127:head# 2024-04-29T14:59:52.774 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4118be51:::benchmark_data_smithi194_196924_object122483:head# 2024-04-29T14:59:52.775 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:411ac003:::benchmark_data_smithi194_196924_object157097:head# 2024-04-29T14:59:52.775 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:411ae5b6:::benchmark_data_smithi194_196924_object122081:head# 2024-04-29T14:59:52.776 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:411bc292:::benchmark_data_smithi194_196924_object96753:head# 2024-04-29T14:59:52.776 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:411bc4e3:::benchmark_data_smithi194_196924_object157161:head# 2024-04-29T14:59:52.777 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:411bcab3:::benchmark_data_smithi194_196924_object151075:head# 2024-04-29T14:59:52.777 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:411d8ab1:::benchmark_data_smithi194_196924_object108944:head# 2024-04-29T14:59:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:411f96db:::benchmark_data_smithi194_196924_object134535:head# 2024-04-29T14:59:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41203c8e:::benchmark_data_smithi194_196924_object122810:head# 2024-04-29T14:59:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4120b523:::benchmark_data_smithi194_196924_object135430:head# 2024-04-29T14:59:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4122f48c:::benchmark_data_smithi194_196924_object139385:head# 2024-04-29T14:59:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41242942:::benchmark_data_smithi194_196924_object148879:head# 2024-04-29T14:59:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:412622c4:::benchmark_data_smithi194_196924_object136586:head# 2024-04-29T14:59:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4126ce60:::benchmark_data_smithi194_196924_object119671:head# 2024-04-29T14:59:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41285392:::benchmark_data_smithi194_196924_object115310:head# 2024-04-29T14:59:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41288288:::benchmark_data_smithi194_196924_object122186:head# 2024-04-29T14:59:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:412a4bf9:::benchmark_data_smithi194_196924_object105856:head# 2024-04-29T14:59:52.782 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:412b8f00:::benchmark_data_smithi194_196924_object157383:head# 2024-04-29T14:59:52.782 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:412da477:::benchmark_data_smithi194_196924_object135690:head# 2024-04-29T14:59:52.782 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:413002ff:::benchmark_data_smithi194_196924_object110025:head# 2024-04-29T14:59:52.783 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41347883:::benchmark_data_smithi194_196924_object130176:head# 2024-04-29T14:59:52.783 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4135e73f:::benchmark_data_smithi194_196924_object146093:head# 2024-04-29T14:59:52.784 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41394c2c:::benchmark_data_smithi194_196924_object148255:head# 2024-04-29T14:59:52.784 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:413b1334:::benchmark_data_smithi194_196924_object151813:head# 2024-04-29T14:59:52.785 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:413b67bd:::benchmark_data_smithi194_196924_object126283:head# 2024-04-29T14:59:52.785 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:413c60cd:::benchmark_data_smithi194_196924_object140813:head# 2024-04-29T14:59:52.785 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:413f994d:::benchmark_data_smithi194_196924_object131874:head# 2024-04-29T14:59:52.786 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41402d88:::benchmark_data_smithi194_196924_object159555:head# 2024-04-29T14:59:52.786 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41404fc0:::benchmark_data_smithi194_196924_object156935:head# 2024-04-29T14:59:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:414073f1:::benchmark_data_smithi194_196924_object133826:head# 2024-04-29T14:59:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4145db32:::benchmark_data_smithi194_196924_object129335:head# 2024-04-29T14:59:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41490c86:::benchmark_data_smithi194_196924_object116088:head# 2024-04-29T14:59:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4149b9be:::benchmark_data_smithi194_196924_object102010:head# 2024-04-29T14:59:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4149bd1c:::benchmark_data_smithi194_196924_object132531:head# 2024-04-29T14:59:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:414a2a4f:::benchmark_data_smithi194_196924_object114135:head# 2024-04-29T14:59:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:414b503e:::benchmark_data_smithi194_196924_object99573:head# 2024-04-29T14:59:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:414d40ba:::benchmark_data_smithi194_196924_object144528:head# 2024-04-29T14:59:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:414f03aa:::benchmark_data_smithi194_196924_object108029:head# 2024-04-29T14:59:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:414f0555:::benchmark_data_smithi194_196924_object149571:head# 2024-04-29T14:59:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41501f47:::benchmark_data_smithi194_196924_object108709:head# 2024-04-29T14:59:52.793 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4150709f:::benchmark_data_smithi194_196924_object122851:head# 2024-04-29T14:59:52.793 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415075ed:::benchmark_data_smithi194_196924_object135214:head# 2024-04-29T14:59:52.793 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41514a95:::benchmark_data_smithi194_196924_object127952:head# 2024-04-29T14:59:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4152546d:::benchmark_data_smithi194_196924_object138130:head# 2024-04-29T14:59:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4152cd82:::benchmark_data_smithi194_196924_object136320:head# 2024-04-29T14:59:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415677c2:::benchmark_data_smithi194_196924_object133388:head# 2024-04-29T14:59:52.795 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415718bd:::benchmark_data_smithi194_196924_object132958:head# 2024-04-29T14:59:52.795 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4158144d:::benchmark_data_smithi194_196924_object144072:head# 2024-04-29T14:59:52.796 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41582afe:::benchmark_data_smithi194_196924_object125273:head# 2024-04-29T14:59:52.796 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4158bc75:::benchmark_data_smithi194_196924_object157606:head# 2024-04-29T14:59:52.797 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415ac31a:::benchmark_data_smithi194_196924_object133013:head# 2024-04-29T14:59:52.797 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415bbe42:::benchmark_data_smithi194_196924_object137890:head# 2024-04-29T14:59:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415ccb8f:::benchmark_data_smithi194_196924_object122606:head# 2024-04-29T14:59:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415d4277:::benchmark_data_smithi194_196924_object101996:head# 2024-04-29T14:59:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:415f3409:::benchmark_data_smithi194_196924_object104508:head# 2024-04-29T14:59:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416078a0:::benchmark_data_smithi194_196924_object95965:head# 2024-04-29T14:59:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4160b5d5:::benchmark_data_smithi194_196924_object146435:head# 2024-04-29T14:59:52.800 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41613d91:::benchmark_data_smithi194_196924_object128530:head# 2024-04-29T14:59:52.800 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4161991c:::benchmark_data_smithi194_196924_object105515:head# 2024-04-29T14:59:52.801 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4161e5f3:::benchmark_data_smithi194_196924_object148151:head# 2024-04-29T14:59:52.801 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4162a7ee:::benchmark_data_smithi194_196924_object104162:head# 2024-04-29T14:59:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416467cd:::benchmark_data_smithi194_196924_object99216:head# 2024-04-29T14:59:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4167a095:::benchmark_data_smithi194_196924_object141515:head# 2024-04-29T14:59:52.803 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4167a20a:::benchmark_data_smithi194_196924_object144624:head# 2024-04-29T14:59:52.803 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41691443:::benchmark_data_smithi194_196924_object108810:head# 2024-04-29T14:59:52.803 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416966e4:::benchmark_data_smithi194_196924_object96238:head# 2024-04-29T14:59:52.804 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4169d381:::benchmark_data_smithi194_196924_object102986:head# 2024-04-29T14:59:52.804 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416a0767:::benchmark_data_smithi194_196924_object116631:head# 2024-04-29T14:59:52.805 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416a86ee:::benchmark_data_smithi194_196924_object130137:head# 2024-04-29T14:59:52.805 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416b9248:::benchmark_data_smithi194_196924_object135812:head# 2024-04-29T14:59:52.806 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416c37e9:::benchmark_data_smithi194_196924_object116920:head# 2024-04-29T14:59:52.806 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416c7179:::benchmark_data_smithi194_196924_object109517:head# 2024-04-29T14:59:52.807 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416d34bb:::benchmark_data_smithi194_196924_object126074:head# 2024-04-29T14:59:52.807 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416e7f7d:::benchmark_data_smithi194_196924_object99461:head# 2024-04-29T14:59:52.808 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416f62ab:::benchmark_data_smithi194_196924_object146122:head# 2024-04-29T14:59:52.808 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:416facd2:::benchmark_data_smithi194_196924_object133809:head# 2024-04-29T14:59:52.809 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4170d6e3:::benchmark_data_smithi194_196924_object130926:head# 2024-04-29T14:59:52.809 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4171c28c:::benchmark_data_smithi194_196924_object118474:head# 2024-04-29T14:59:52.811 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4173c9ac:::benchmark_data_smithi194_196924_object112995:head# 2024-04-29T14:59:52.811 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T14:59:52.811 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4173caa0:::benchmark_data_smithi194_196924_object107071:head# 2024-04-29T14:59:52.811 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41746bf4:::benchmark_data_smithi194_196924_object154570:head# 2024-04-29T14:59:52.812 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:417688a1:::benchmark_data_smithi194_196924_object133276:head# 2024-04-29T14:59:52.812 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:417781aa:::benchmark_data_smithi194_196924_object98352:head# 2024-04-29T14:59:52.813 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4178fb9b:::benchmark_data_smithi194_196924_object138523:head# 2024-04-29T14:59:52.813 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41793ed3:::benchmark_data_smithi194_196924_object98648:head# 2024-04-29T14:59:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:52.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T14:59:52.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.815 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:52.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:52 smithi040 ceph-mon[39581]: pgmap v2907: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 76 MiB/s, 1.21k objects/s recovering 2024-04-29T14:59:52.816 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4179f8dd:::benchmark_data_smithi194_196924_object141527:head# 2024-04-29T14:59:52.816 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:417a7f8e:::benchmark_data_smithi194_196924_object137128:head# 2024-04-29T14:59:52.817 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:417e1e2f:::benchmark_data_smithi194_196924_object158670:head# 2024-04-29T14:59:52.817 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:417f7b28:::benchmark_data_smithi194_196924_object116174:head# 2024-04-29T14:59:52.817 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4180d6b2:::benchmark_data_smithi194_196924_object117355:head# 2024-04-29T14:59:52.817 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4181961c:::benchmark_data_smithi194_196924_object104253:head# 2024-04-29T14:59:52.817 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41824c14:::benchmark_data_smithi194_196924_object116432:head# 2024-04-29T14:59:52.818 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41826f28:::benchmark_data_smithi194_196924_object102202:head# 2024-04-29T14:59:52.818 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41836ed1:::benchmark_data_smithi194_196924_object149005:head# 2024-04-29T14:59:52.818 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:418599a7:::benchmark_data_smithi194_196924_object104425:head# 2024-04-29T14:59:52.819 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4188de69:::benchmark_data_smithi194_196924_object102850:head# 2024-04-29T14:59:52.819 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4188e546:::benchmark_data_smithi194_196924_object96088:head# 2024-04-29T14:59:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41898356:::benchmark_data_smithi194_196924_object125454:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:418b2cf9:::benchmark_data_smithi194_196924_object106310:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:418b369e:::benchmark_data_smithi194_196924_object96018:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:418df071:::benchmark_data_smithi194_196924_object152710:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:418e2512:::benchmark_data_smithi194_196924_object118041:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41918b97:::benchmark_data_smithi194_196924_object133264:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:419239a5:::benchmark_data_smithi194_196924_object144038:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41931a87:::benchmark_data_smithi194_196924_object141303:head# 2024-04-29T14:59:52.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41935aff:::benchmark_data_smithi194_196924_object138647:head# 2024-04-29T14:59:52.826 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41949e78:::benchmark_data_smithi194_196924_object102570:head# 2024-04-29T14:59:52.826 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4195d743:::benchmark_data_smithi194_196924_object99942:head# 2024-04-29T14:59:52.827 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4199b0b3:::benchmark_data_smithi194_196924_object122562:head# 2024-04-29T14:59:52.827 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:419a1ba4:::benchmark_data_smithi194_196924_object117772:head# 2024-04-29T14:59:52.827 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:419bb6b8:::benchmark_data_smithi194_196924_object98222:head# 2024-04-29T14:59:52.828 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:419c05c2:::benchmark_data_smithi194_196924_object122158:head# 2024-04-29T14:59:52.828 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:419c3f7c:::benchmark_data_smithi194_196924_object143211:head# 2024-04-29T14:59:52.829 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:419d167a:::benchmark_data_smithi194_196924_object153550:head# 2024-04-29T14:59:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:419d4417:::benchmark_data_smithi194_196924_object105972:head# 2024-04-29T14:59:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a476f6:::benchmark_data_smithi194_196924_object157388:head# 2024-04-29T14:59:52.831 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a4cc1f:::benchmark_data_smithi194_196924_object106711:head# 2024-04-29T14:59:52.831 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a580c9:::benchmark_data_smithi194_196924_object141234:head# 2024-04-29T14:59:52.832 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a5aef8:::benchmark_data_smithi194_196924_object105807:head# 2024-04-29T14:59:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a68004:::benchmark_data_smithi194_196924_object138217:head# 2024-04-29T14:59:52.834 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a6b996:::benchmark_data_smithi194_196924_object150850:head# 2024-04-29T14:59:52.836 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a6e0a6:::benchmark_data_smithi194_196924_object158474:head# 2024-04-29T14:59:52.836 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a8da13:::benchmark_data_smithi194_196924_object131203:head# 2024-04-29T14:59:52.837 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a9ab1c:::benchmark_data_smithi194_196924_object108087:head# 2024-04-29T14:59:52.837 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41a9c2b0:::benchmark_data_smithi194_196924_object151891:head# 2024-04-29T14:59:52.837 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41aa84a0:::benchmark_data_smithi194_196924_object131568:head# 2024-04-29T14:59:52.838 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41aaad27:::benchmark_data_smithi194_196924_object137459:head# 2024-04-29T14:59:52.838 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ab5e2c:::benchmark_data_smithi194_196924_object121387:head# 2024-04-29T14:59:52.839 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ab9cd7:::benchmark_data_smithi194_196924_object105518:head# 2024-04-29T14:59:52.839 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ac5a8c:::benchmark_data_smithi194_196924_object150172:head# 2024-04-29T14:59:52.840 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41add528:::benchmark_data_smithi194_196924_object95948:head# 2024-04-29T14:59:52.840 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41af787b:::benchmark_data_smithi194_196924_object125638:head# 2024-04-29T14:59:52.843 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41b12e93:::benchmark_data_smithi194_196924_object115815:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41b142e1:::benchmark_data_smithi194_196924_object124902:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41b39240:::benchmark_data_smithi194_196924_object108840:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41b39b44:::benchmark_data_smithi194_196924_object123840:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41b8497f:::benchmark_data_smithi194_196924_object129299:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41b8557b:::benchmark_data_smithi194_196924_object135375:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41b99c12:::benchmark_data_smithi194_196924_object148324:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41bae27d:::benchmark_data_smithi194_196924_object158382:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41bcdd90:::benchmark_data_smithi194_196924_object97268:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41be0782:::benchmark_data_smithi194_196924_object96475:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41bf747d:::benchmark_data_smithi194_196924_object100942:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41bfb903:::benchmark_data_smithi194_196924_object115046:head# 2024-04-29T14:59:52.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41bfd2ae:::benchmark_data_smithi194_196924_object136309:head# 2024-04-29T14:59:52.851 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c00157:::benchmark_data_smithi194_196924_object147146:head# 2024-04-29T14:59:52.851 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c361d3:::benchmark_data_smithi194_196924_object152087:head# 2024-04-29T14:59:52.851 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c3f8ec:::benchmark_data_smithi194_196924_object137179:head# 2024-04-29T14:59:52.853 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c42fa6:::benchmark_data_smithi194_196924_object152419:head# 2024-04-29T14:59:52.853 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c4e47e:::benchmark_data_smithi194_196924_object107741:head# 2024-04-29T14:59:52.853 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c530d2:::benchmark_data_smithi194_196924_object105267:head# 2024-04-29T14:59:52.854 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c6c308:::benchmark_data_smithi194_196924_object97893:head# 2024-04-29T14:59:52.854 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c74332:::benchmark_data_smithi194_196924_object139104:head# 2024-04-29T14:59:52.854 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c8daf2:::benchmark_data_smithi194_196924_object119658:head# 2024-04-29T14:59:52.855 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c94199:::benchmark_data_smithi194_196924_object99682:head# 2024-04-29T14:59:52.855 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41c9900d:::benchmark_data_smithi194_196924_object155098:head# 2024-04-29T14:59:52.856 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41cd2f56:::benchmark_data_smithi194_196924_object118516:head# 2024-04-29T14:59:52.856 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41cd31d3:::benchmark_data_smithi194_196924_object121993:head# 2024-04-29T14:59:52.857 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ce07c3:::benchmark_data_smithi194_196924_object158772:head# 2024-04-29T14:59:52.857 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d04577:::benchmark_data_smithi194_196924_object114816:head# 2024-04-29T14:59:52.858 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d14608:::benchmark_data_smithi194_196924_object133808:head# 2024-04-29T14:59:52.858 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d19f95:::benchmark_data_smithi194_196924_object129441:head# 2024-04-29T14:59:52.858 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d274a7:::benchmark_data_smithi194_196924_object117554:head# 2024-04-29T14:59:52.862 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d3cb62:::benchmark_data_smithi194_196924_object114160:head# 2024-04-29T14:59:52.862 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d4678a:::benchmark_data_smithi194_196924_object146671:head# 2024-04-29T14:59:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d50440:::benchmark_data_smithi194_196924_object95837:head# 2024-04-29T14:59:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d51def:::benchmark_data_smithi194_196924_object95599:head# 2024-04-29T14:59:52.864 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d55ce3:::benchmark_data_smithi194_196924_object127745:head# 2024-04-29T14:59:52.864 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d7d6ba:::benchmark_data_smithi194_196924_object96750:head# 2024-04-29T14:59:52.864 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d8be6c:::benchmark_data_smithi194_196924_object131821:head# 2024-04-29T14:59:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d941c4:::benchmark_data_smithi194_196924_object115220:head# 2024-04-29T14:59:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41d95007:::benchmark_data_smithi194_196924_object98149:head# 2024-04-29T14:59:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41da5c4a:::benchmark_data_smithi194_196924_object118387:head# 2024-04-29T14:59:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41da72b2:::benchmark_data_smithi194_196924_object128636:head# 2024-04-29T14:59:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41dadb64:::benchmark_data_smithi194_196924_object97774:head# 2024-04-29T14:59:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41db2570:::benchmark_data_smithi194_196924_object139007:head# 2024-04-29T14:59:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41dbb2d9:::benchmark_data_smithi194_196924_object124140:head# 2024-04-29T14:59:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41dbebb9:::benchmark_data_smithi194_196924_object115570:head# 2024-04-29T14:59:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41dcd8a7:::benchmark_data_smithi194_196924_object158737:head# 2024-04-29T14:59:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41dd9ef3:::benchmark_data_smithi194_196924_object112584:head# 2024-04-29T14:59:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41de7bcf:::benchmark_data_smithi194_196924_object113073:head# 2024-04-29T14:59:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41df89ff:::benchmark_data_smithi194_196924_object148293:head# 2024-04-29T14:59:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41dff837:::benchmark_data_smithi194_196924_object122902:head# 2024-04-29T14:59:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e0d801:::benchmark_data_smithi194_196924_object109808:head# 2024-04-29T14:59:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e4e436:::benchmark_data_smithi194_196924_object117916:head# 2024-04-29T14:59:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e58772:::benchmark_data_smithi194_196924_object146249:head# 2024-04-29T14:59:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e7a25a:::benchmark_data_smithi194_196924_object121695:head# 2024-04-29T14:59:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e830b1:::benchmark_data_smithi194_196924_object121504:head# 2024-04-29T14:59:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e833ea:::benchmark_data_smithi194_196924_object109290:head# 2024-04-29T14:59:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e8762e:::benchmark_data_smithi194_196924_object126403:head# 2024-04-29T14:59:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e91dce:::benchmark_data_smithi194_196924_object148556:head# 2024-04-29T14:59:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e9346f:::benchmark_data_smithi194_196924_object140302:head# 2024-04-29T14:59:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e945c1:::benchmark_data_smithi194_196924_object125356:head# 2024-04-29T14:59:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41e9feec:::benchmark_data_smithi194_196924_object103216:head# 2024-04-29T14:59:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ea274e:::benchmark_data_smithi194_196924_object134479:head# 2024-04-29T14:59:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41eae441:::benchmark_data_smithi194_196924_object128963:head# 2024-04-29T14:59:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ecd118:::benchmark_data_smithi194_196924_object114100:head# 2024-04-29T14:59:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ef6acd:::benchmark_data_smithi194_196924_object97539:head# 2024-04-29T14:59:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f07b97:::benchmark_data_smithi194_196924_object132820:head# 2024-04-29T14:59:52.880 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f0b672:::benchmark_data_smithi194_196924_object138780:head# 2024-04-29T14:59:52.880 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f161b6:::benchmark_data_smithi194_196924_object120612:head# 2024-04-29T14:59:52.881 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f1f8b6:::benchmark_data_smithi194_196924_object108286:head# 2024-04-29T14:59:52.881 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f21a36:::benchmark_data_smithi194_196924_object129406:head# 2024-04-29T14:59:52.882 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f347d0:::benchmark_data_smithi194_196924_object146118:head# 2024-04-29T14:59:52.882 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f37f85:::benchmark_data_smithi194_196924_object139922:head# 2024-04-29T14:59:52.882 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f46e68:::benchmark_data_smithi194_196924_object111164:head# 2024-04-29T14:59:52.883 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f77fd1:::benchmark_data_smithi194_196924_object143612:head# 2024-04-29T14:59:52.883 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f7ea35:::benchmark_data_smithi194_196924_object108507:head# 2024-04-29T14:59:52.884 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f801d1:::benchmark_data_smithi194_196924_object120995:head# 2024-04-29T14:59:52.884 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f86cb7:::benchmark_data_smithi194_196924_object108897:head# 2024-04-29T14:59:52.885 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f91810:::benchmark_data_smithi194_196924_object138441:head# 2024-04-29T14:59:52.885 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41f93bac:::benchmark_data_smithi194_196924_object112902:head# 2024-04-29T14:59:52.885 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fa0c14:::benchmark_data_smithi194_196924_object155132:head# 2024-04-29T14:59:52.886 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fa5c1b:::benchmark_data_smithi194_196924_object145807:head# 2024-04-29T14:59:52.886 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fbee23:::benchmark_data_smithi194_196924_object153667:head# 2024-04-29T14:59:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fc4e85:::benchmark_data_smithi194_196924_object141161:head# 2024-04-29T14:59:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fc98e2:::benchmark_data_smithi194_196924_object100887:head# 2024-04-29T14:59:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fca02f:::benchmark_data_smithi194_196924_object138991:head# 2024-04-29T14:59:52.888 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fd116f:::benchmark_data_smithi194_196924_object142628:head# 2024-04-29T14:59:52.888 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fdcfa8:::benchmark_data_smithi194_196924_object101945:head# 2024-04-29T14:59:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41fed233:::benchmark_data_smithi194_196924_object136245:head# 2024-04-29T14:59:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ff8dab:::benchmark_data_smithi194_196924_object143683:head# 2024-04-29T14:59:52.890 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:41ffdedf:::benchmark_data_smithi194_196924_object133402:head# 2024-04-29T14:59:52.891 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42025e1d:::benchmark_data_smithi194_196924_object96123:head# 2024-04-29T14:59:52.891 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4202ad20:::benchmark_data_smithi194_196924_object107033:head# 2024-04-29T14:59:52.891 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4202db42:::benchmark_data_smithi194_196924_object139830:head# 2024-04-29T14:59:52.891 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4205c7bc:::benchmark_data_smithi194_196924_object116739:head# 2024-04-29T14:59:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4206f34f:::benchmark_data_smithi194_196924_object116784:head# 2024-04-29T14:59:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:420701d0:::benchmark_data_smithi194_196924_object147403:head# 2024-04-29T14:59:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42072aad:::benchmark_data_smithi194_196924_object139502:head# 2024-04-29T14:59:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4207794b:::benchmark_data_smithi194_196924_object109371:head# 2024-04-29T14:59:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42080698:::benchmark_data_smithi194_196924_object144912:head# 2024-04-29T14:59:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:420a1fd1:::benchmark_data_smithi194_196924_object122035:head# 2024-04-29T14:59:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:420a4163:::benchmark_data_smithi194_196924_object153776:head# 2024-04-29T14:59:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:420a7123:::benchmark_data_smithi194_196924_object121239:head# 2024-04-29T14:59:52.896 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:420d6b04:::benchmark_data_smithi194_196924_object103402:head# 2024-04-29T14:59:52.896 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:420d6d30:::benchmark_data_smithi194_196924_object112377:head# 2024-04-29T14:59:52.897 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:420f0d46:::benchmark_data_smithi194_196924_object115586:head# 2024-04-29T14:59:52.897 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42107c55:::benchmark_data_smithi194_196924_object99099:head# 2024-04-29T14:59:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42110ef6:::benchmark_data_smithi194_196924_object97473:head# 2024-04-29T14:59:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:421173e4:::benchmark_data_smithi194_196924_object97925:head# 2024-04-29T14:59:52.899 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:421200e0:::benchmark_data_smithi194_196924_object119637:head# 2024-04-29T14:59:52.899 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42135e99:::benchmark_data_smithi194_196924_object114457:head# 2024-04-29T14:59:52.899 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42149b3e:::benchmark_data_smithi194_196924_object108854:head# 2024-04-29T14:59:52.900 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4214a35f:::benchmark_data_smithi194_196924_object98538:head# 2024-04-29T14:59:52.900 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4216f8fe:::benchmark_data_smithi194_196924_object153532:head# 2024-04-29T14:59:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4218d608:::benchmark_data_smithi194_196924_object117188:head# 2024-04-29T14:59:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42194869:::benchmark_data_smithi194_196924_object114016:head# 2024-04-29T14:59:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:421bba27:::benchmark_data_smithi194_196924_object99464:head# 2024-04-29T14:59:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:421dabc7:::benchmark_data_smithi194_196924_object112732:head# 2024-04-29T14:59:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4220565a:::benchmark_data_smithi194_196924_object143035:head# 2024-04-29T14:59:52.903 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42225fd2:::benchmark_data_smithi194_196924_object148520:head# 2024-04-29T14:59:52.903 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42235d35:::benchmark_data_smithi194_196924_object121284:head# 2024-04-29T14:59:52.904 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4225d970:::benchmark_data_smithi194_196924_object146756:head# 2024-04-29T14:59:52.904 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42291be7:::benchmark_data_smithi194_196924_object145553:head# 2024-04-29T14:59:52.905 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:422d9e06:::benchmark_data_smithi194_196924_object129563:head# 2024-04-29T14:59:52.905 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:422e657e:::benchmark_data_smithi194_196924_object130924:head# 2024-04-29T14:59:52.905 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:422f379b:::benchmark_data_smithi194_196924_object99635:head# 2024-04-29T14:59:52.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4230ddcd:::benchmark_data_smithi194_196924_object153969:head# 2024-04-29T14:59:52.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4234008f:::benchmark_data_smithi194_196924_object105562:head# 2024-04-29T14:59:52.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4235aed6:::benchmark_data_smithi194_196924_object130675:head# 2024-04-29T14:59:52.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4235f4c4:::benchmark_data_smithi194_196924_object156651:head# 2024-04-29T14:59:52.908 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4236555d:::benchmark_data_smithi194_196924_object106070:head# 2024-04-29T14:59:52.908 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42368bd8:::benchmark_data_smithi194_196924_object119156:head# 2024-04-29T14:59:52.908 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42370fdc:::benchmark_data_smithi194_196924_object150761:head# 2024-04-29T14:59:52.909 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42371da5:::benchmark_data_smithi194_196924_object98963:head# 2024-04-29T14:59:52.909 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4237757f:::benchmark_data_smithi194_196924_object143711:head# 2024-04-29T14:59:52.910 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42377fc9:::benchmark_data_smithi194_196924_object141680:head# 2024-04-29T14:59:52.910 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42387354:::benchmark_data_smithi194_196924_object103062:head# 2024-04-29T14:59:52.911 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42389743:::benchmark_data_smithi194_196924_object159830:head# 2024-04-29T14:59:52.911 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4238aa64:::benchmark_data_smithi194_196924_object132430:head# 2024-04-29T14:59:52.912 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:423a9c7f:::benchmark_data_smithi194_196924_object136151:head# 2024-04-29T14:59:52.912 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:423ac7ca:::benchmark_data_smithi194_196924_object113372:head# 2024-04-29T14:59:52.913 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:423b99bc:::benchmark_data_smithi194_196924_object134989:head# 2024-04-29T14:59:52.913 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:423c6042:::benchmark_data_smithi194_196924_object112527:head# 2024-04-29T14:59:52.913 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:423f510e:::benchmark_data_smithi194_196924_object156260:head# 2024-04-29T14:59:52.914 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:423f8113:::benchmark_data_smithi194_196924_object150010:head# 2024-04-29T14:59:52.914 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:423fc0b4:::benchmark_data_smithi194_196924_object157672:head# 2024-04-29T14:59:52.915 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4240b4d5:::benchmark_data_smithi194_196924_object107790:head# 2024-04-29T14:59:52.915 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4240de17:::benchmark_data_smithi194_196924_object141125:head# 2024-04-29T14:59:52.915 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:52.916 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424109ef:::benchmark_data_smithi194_196924_object154953:head# 2024-04-29T14:59:52.917 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424122e3:::benchmark_data_smithi194_196924_object156464:head# 2024-04-29T14:59:52.917 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424224c6:::benchmark_data_smithi194_196924_object153374:head# 2024-04-29T14:59:52.918 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42435855:::benchmark_data_smithi194_196924_object144692:head# 2024-04-29T14:59:52.919 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424482af:::benchmark_data_smithi194_196924_object104851:head# 2024-04-29T14:59:52.919 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42456332:::benchmark_data_smithi194_196924_object148907:head# 2024-04-29T14:59:52.919 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4247256f:::benchmark_data_smithi194_196924_object108117:head# 2024-04-29T14:59:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4247c8a7:::benchmark_data_smithi194_196924_object159532:head# 2024-04-29T14:59:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424809e5:::benchmark_data_smithi194_196924_object115745:head# 2024-04-29T14:59:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42486414:::benchmark_data_smithi194_196924_object99278:head# 2024-04-29T14:59:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4248e55f:::benchmark_data_smithi194_196924_object121380:head# 2024-04-29T14:59:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4248ea55:::benchmark_data_smithi194_196924_object135681:head# 2024-04-29T14:59:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424a03a7:::benchmark_data_smithi194_196924_object102253:head# 2024-04-29T14:59:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424b7f1b:::benchmark_data_smithi194_196924_object149858:head# 2024-04-29T14:59:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424bc7d3:::benchmark_data_smithi194_196924_object157206:head# 2024-04-29T14:59:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424db7a1:::benchmark_data_smithi194_196924_object96942:head# 2024-04-29T14:59:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424e06f8:::benchmark_data_smithi194_196924_object132534:head# 2024-04-29T14:59:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:424f1256:::benchmark_data_smithi194_196924_object158046:head# 2024-04-29T14:59:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4251e80e:::benchmark_data_smithi194_196924_object116043:head# 2024-04-29T14:59:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42521d09:::benchmark_data_smithi194_196924_object121303:head# 2024-04-29T14:59:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42521fd7:::benchmark_data_smithi194_196924_object120669:head# 2024-04-29T14:59:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:425257f3:::benchmark_data_smithi194_196924_object128409:head# 2024-04-29T14:59:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:425344c8:::benchmark_data_smithi194_196924_object126458:head# 2024-04-29T14:59:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42534c22:::benchmark_data_smithi194_196924_object104246:head# 2024-04-29T14:59:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4255fdc9:::benchmark_data_smithi194_196924_object120564:head# 2024-04-29T14:59:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42574f76:::benchmark_data_smithi194_196924_object120665:head# 2024-04-29T14:59:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42590eea:::benchmark_data_smithi194_196924_object119291:head# 2024-04-29T14:59:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4259fca3:::benchmark_data_smithi194_196924_object115610:head# 2024-04-29T14:59:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:425c87d8:::benchmark_data_smithi194_196924_object113623:head# 2024-04-29T14:59:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:425f1264:::benchmark_data_smithi194_196924_object156131:head# 2024-04-29T14:59:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42617ec4:::benchmark_data_smithi194_196924_object158472:head# 2024-04-29T14:59:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4263a9ea:::benchmark_data_smithi194_196924_object117513:head# 2024-04-29T14:59:52.930 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42644dd9:::benchmark_data_smithi194_196924_object102588:head# 2024-04-29T14:59:52.930 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:426526a9:::benchmark_data_smithi194_196924_object131887:head# 2024-04-29T14:59:52.931 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:426592df:::benchmark_data_smithi194_196924_object142713:head# 2024-04-29T14:59:52.931 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4265de3e:::benchmark_data_smithi194_196924_object98160:head# 2024-04-29T14:59:52.932 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4266722a:::benchmark_data_smithi194_196924_object100128:head# 2024-04-29T14:59:52.932 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4266dd0b:::benchmark_data_smithi194_196924_object130999:head# 2024-04-29T14:59:52.932 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42677872:::benchmark_data_smithi194_196924_object119224:head# 2024-04-29T14:59:52.933 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42677cb5:::benchmark_data_smithi194_196924_object111281:head# 2024-04-29T14:59:52.933 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4267de64:::benchmark_data_smithi194_196924_object117536:head# 2024-04-29T14:59:52.934 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42681602:::benchmark_data_smithi194_196924_object139013:head# 2024-04-29T14:59:52.935 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:426832a3:::benchmark_data_smithi194_196924_object111543:head# 2024-04-29T14:59:52.935 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42688d92:::benchmark_data_smithi194_196924_object117032:head# 2024-04-29T14:59:52.935 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4269852a:::benchmark_data_smithi194_196924_object126336:head# 2024-04-29T14:59:52.936 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:426c7cb0:::benchmark_data_smithi194_196924_object101646:head# 2024-04-29T14:59:52.938 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:426cc5d9:::benchmark_data_smithi194_196924_object124018:head# 2024-04-29T14:59:52.938 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:426cdea0:::benchmark_data_smithi194_196924_object141728:head# 2024-04-29T14:59:52.938 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:426e5a4d:::benchmark_data_smithi194_196924_object143607:head# 2024-04-29T14:59:52.939 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4270e64c:::benchmark_data_smithi194_196924_object159342:head# 2024-04-29T14:59:52.939 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:427137c5:::benchmark_data_smithi194_196924_object102681:head# 2024-04-29T14:59:52.940 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:427165d1:::benchmark_data_smithi194_196924_object141452:head# 2024-04-29T14:59:52.940 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4273ac4d:::benchmark_data_smithi194_196924_object148334:head# 2024-04-29T14:59:52.940 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4274f52a:::benchmark_data_smithi194_196924_object113162:head# 2024-04-29T14:59:52.941 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4275abad:::benchmark_data_smithi194_196924_object147169:head# 2024-04-29T14:59:52.942 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4275acb4:::benchmark_data_smithi194_196924_object154018:head# 2024-04-29T14:59:52.942 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4276335b:::benchmark_data_smithi194_196924_object155628:head# 2024-04-29T14:59:52.942 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4279ba4c:::benchmark_data_smithi194_196924_object138534:head# 2024-04-29T14:59:52.943 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4279dde3:::benchmark_data_smithi194_196924_object139962:head# 2024-04-29T14:59:52.944 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:427dbc86:::benchmark_data_smithi194_196924_object135606:head# 2024-04-29T14:59:52.945 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:427e577d:::benchmark_data_smithi194_196924_object102189:head# 2024-04-29T14:59:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:427f393f:::benchmark_data_smithi194_196924_object145489:head# 2024-04-29T14:59:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:427f3d5a:::benchmark_data_smithi194_196924_object109687:head# 2024-04-29T14:59:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4280bd2b:::benchmark_data_smithi194_196924_object136617:head# 2024-04-29T14:59:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:428257a2:::benchmark_data_smithi194_196924_object123646:head# 2024-04-29T14:59:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4284d56b:::benchmark_data_smithi194_196924_object132342:head# 2024-04-29T14:59:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42854f8f:::benchmark_data_smithi194_196924_object122375:head# 2024-04-29T14:59:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4287a7c4:::benchmark_data_smithi194_196924_object148940:head# 2024-04-29T14:59:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42882a22:::benchmark_data_smithi194_196924_object142526:head# 2024-04-29T14:59:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:428836c7:::benchmark_data_smithi194_196924_object123115:head# 2024-04-29T14:59:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4288c000:::benchmark_data_smithi194_196924_object129413:head# 2024-04-29T14:59:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:428cb464:::benchmark_data_smithi194_196924_object131543:head# 2024-04-29T14:59:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:428d2d32:::benchmark_data_smithi194_196924_object136029:head# 2024-04-29T14:59:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:428e63c9:::benchmark_data_smithi194_196924_object139761:head# 2024-04-29T14:59:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:428ebd8b:::benchmark_data_smithi194_196924_object155690:head# 2024-04-29T14:59:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:428f0546:::benchmark_data_smithi194_196924_object127594:head# 2024-04-29T14:59:52.952 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4290a2a1:::benchmark_data_smithi194_196924_object124204:head# 2024-04-29T14:59:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4291d0a1:::benchmark_data_smithi194_196924_object120511:head# 2024-04-29T14:59:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42951fb7:::benchmark_data_smithi194_196924_object134507:head# 2024-04-29T14:59:52.954 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4296240f:::benchmark_data_smithi194_196924_object97180:head# 2024-04-29T14:59:52.954 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4296259e:::benchmark_data_smithi194_196924_object142888:head# 2024-04-29T14:59:52.954 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:429632c4:::benchmark_data_smithi194_196924_object126824:head# 2024-04-29T14:59:52.955 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4297f2b9:::benchmark_data_smithi194_196924_object130325:head# 2024-04-29T14:59:52.955 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:429ce803:::benchmark_data_smithi194_196924_object97590:head# 2024-04-29T14:59:52.955 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:429e2ff6:::benchmark_data_smithi194_196924_object139982:head# 2024-04-29T14:59:52.956 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:429e665f:::benchmark_data_smithi194_196924_object144806:head# 2024-04-29T14:59:52.956 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:429f13e8:::benchmark_data_smithi194_196924_object114788:head# 2024-04-29T14:59:52.957 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42a15db0:::benchmark_data_smithi194_196924_object118881:head# 2024-04-29T14:59:52.957 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42a50324:::benchmark_data_smithi194_196924_object155354:head# 2024-04-29T14:59:52.957 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42a516d6:::benchmark_data_smithi194_196924_object135127:head# 2024-04-29T14:59:52.958 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42a517d1:::benchmark_data_smithi194_196924_object153998:head# 2024-04-29T14:59:52.958 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42a58b5b:::benchmark_data_smithi194_196924_object118842:head# 2024-04-29T14:59:52.959 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42a5e82c:::benchmark_data_smithi194_196924_object116031:head# 2024-04-29T14:59:52.959 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42a86c94:::benchmark_data_smithi194_196924_object137140:head# 2024-04-29T14:59:52.960 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ab15b0:::benchmark_data_smithi194_196924_object134611:head# 2024-04-29T14:59:52.960 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ac3674:::benchmark_data_smithi194_196924_object132648:head# 2024-04-29T14:59:52.961 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42adc987:::benchmark_data_smithi194_196924_object144170:head# 2024-04-29T14:59:52.961 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42af7f8f:::benchmark_data_smithi194_196924_object101879:head# 2024-04-29T14:59:52.961 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b2191c:::benchmark_data_smithi194_196924_object135188:head# 2024-04-29T14:59:52.962 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b2fb41:::benchmark_data_smithi194_196924_object139409:head# 2024-04-29T14:59:52.962 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b3a20f:::benchmark_data_smithi194_196924_object154485:head# 2024-04-29T14:59:52.963 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b44e2f:::benchmark_data_smithi194_196924_object112626:head# 2024-04-29T14:59:52.963 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b49f73:::benchmark_data_smithi194_196924_object155079:head# 2024-04-29T14:59:52.963 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b4d4c3:::benchmark_data_smithi194_196924_object159671:head# 2024-04-29T14:59:52.964 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b52889:::benchmark_data_smithi194_196924_object137471:head# 2024-04-29T14:59:52.964 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b563d2:::benchmark_data_smithi194_196924_object124854:head# 2024-04-29T14:59:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b5c1b2:::benchmark_data_smithi194_196924_object101305:head# 2024-04-29T14:59:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b65aea:::benchmark_data_smithi194_196924_object159553:head# 2024-04-29T14:59:52.966 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b70d61:::benchmark_data_smithi194_196924_object156632:head# 2024-04-29T14:59:52.966 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b8b3fd:::benchmark_data_smithi194_196924_object121968:head# 2024-04-29T14:59:52.967 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b97a9b:::benchmark_data_smithi194_196924_object131341:head# 2024-04-29T14:59:52.967 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42b9a346:::benchmark_data_smithi194_196924_object111704:head# 2024-04-29T14:59:52.967 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ba441e:::benchmark_data_smithi194_196924_object103700:head# 2024-04-29T14:59:52.968 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:52.968 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42baea0c:::benchmark_data_smithi194_196924_object132877:head# 2024-04-29T14:59:52.968 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42bc2bc6:::benchmark_data_smithi194_196924_object134691:head# 2024-04-29T14:59:52.969 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42bd5410:::benchmark_data_smithi194_196924_object98925:head# 2024-04-29T14:59:52.970 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42bd6928:::benchmark_data_smithi194_196924_object134978:head# 2024-04-29T14:59:52.970 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42bd7f43:::benchmark_data_smithi194_196924_object148864:head# 2024-04-29T14:59:52.970 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42be1df7:::benchmark_data_smithi194_196924_object104942:head# 2024-04-29T14:59:52.971 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42c1c78f:::benchmark_data_smithi194_196924_object155125:head# 2024-04-29T14:59:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:52.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:52.972 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42c3b17a:::benchmark_data_smithi194_196924_object121513:head# 2024-04-29T14:59:52.973 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42c4dc0a:::benchmark_data_smithi194_196924_object143404:head# 2024-04-29T14:59:52.973 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42c52013:::benchmark_data_smithi194_196924_object144392:head# 2024-04-29T14:59:52.973 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42c79df7:::benchmark_data_smithi194_196924_object106854:head# 2024-04-29T14:59:52.973 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42c82fc8:::benchmark_data_smithi194_196924_object143490:head# 2024-04-29T14:59:52.974 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ca5564:::benchmark_data_smithi194_196924_object145000:head# 2024-04-29T14:59:52.974 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42caaf15:::benchmark_data_smithi194_196924_object153705:head# 2024-04-29T14:59:52.975 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42cabfad:::benchmark_data_smithi194_196924_object136100:head# 2024-04-29T14:59:52.975 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42cb0693:::benchmark_data_smithi194_196924_object153644:head# 2024-04-29T14:59:52.976 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42cd5997:::benchmark_data_smithi194_196924_object144320:head# 2024-04-29T14:59:52.976 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42cdccc7:::benchmark_data_smithi194_196924_object135892:head# 2024-04-29T14:59:52.977 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ceaa52:::benchmark_data_smithi194_196924_object149769:head# 2024-04-29T14:59:52.977 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42cee493:::benchmark_data_smithi194_196924_object135557:head# 2024-04-29T14:59:52.977 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42cf7a9d:::benchmark_data_smithi194_196924_object104336:head# 2024-04-29T14:59:52.978 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42d05007:::benchmark_data_smithi194_196924_object100862:head# 2024-04-29T14:59:52.978 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42d27743:::benchmark_data_smithi194_196924_object123724:head# 2024-04-29T14:59:52.979 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42d302f7:::benchmark_data_smithi194_196924_object108637:head# 2024-04-29T14:59:52.979 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42d997cd:::benchmark_data_smithi194_196924_object113682:head# 2024-04-29T14:59:52.980 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42da9677:::benchmark_data_smithi194_196924_object136812:head# 2024-04-29T14:59:52.980 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42dbc372:::benchmark_data_smithi194_196924_object129105:head# 2024-04-29T14:59:52.980 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42dc5f60:::benchmark_data_smithi194_196924_object139005:head# 2024-04-29T14:59:52.981 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ddea82:::benchmark_data_smithi194_196924_object126120:head# 2024-04-29T14:59:52.981 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42de2fc1:::benchmark_data_smithi194_196924_object128277:head# 2024-04-29T14:59:52.982 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42de7690:::benchmark_data_smithi194_196924_object117729:head# 2024-04-29T14:59:52.982 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e26d06:::benchmark_data_smithi194_196924_object145345:head# 2024-04-29T14:59:52.983 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e3720e:::benchmark_data_smithi194_196924_object102478:head# 2024-04-29T14:59:52.983 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e46493:::benchmark_data_smithi194_196924_object122265:head# 2024-04-29T14:59:52.984 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e7113d:::benchmark_data_smithi194_196924_object146472:head# 2024-04-29T14:59:52.985 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e74e35:::benchmark_data_smithi194_196924_object130184:head# 2024-04-29T14:59:52.986 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e843b6:::benchmark_data_smithi194_196924_object134650:head# 2024-04-29T14:59:52.986 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e91f48:::benchmark_data_smithi194_196924_object99419:head# 2024-04-29T14:59:52.987 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42e96fd0:::benchmark_data_smithi194_196924_object154571:head# 2024-04-29T14:59:52.987 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ea5d74:::benchmark_data_smithi194_196924_object107247:head# 2024-04-29T14:59:52.987 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42edcea1:::benchmark_data_smithi194_196924_object96738:head# 2024-04-29T14:59:52.987 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42eefcb9:::benchmark_data_smithi194_196924_object133040:head# 2024-04-29T14:59:52.988 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f18005:::benchmark_data_smithi194_196924_object128488:head# 2024-04-29T14:59:52.988 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f1fe1f:::benchmark_data_smithi194_196924_object159403:head# 2024-04-29T14:59:52.989 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f39242:::benchmark_data_smithi194_196924_object124229:head# 2024-04-29T14:59:52.989 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f59a1a:::benchmark_data_smithi194_196924_object144534:head# 2024-04-29T14:59:52.989 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f628cc:::benchmark_data_smithi194_196924_object120399:head# 2024-04-29T14:59:52.990 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f77885:::benchmark_data_smithi194_196924_object130337:head# 2024-04-29T14:59:52.990 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f7abc4:::benchmark_data_smithi194_196924_object131076:head# 2024-04-29T14:59:52.991 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42f948fa:::benchmark_data_smithi194_196924_object96306:head# 2024-04-29T14:59:52.991 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42facbef:::benchmark_data_smithi194_196924_object122233:head# 2024-04-29T14:59:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42fb5ae0:::benchmark_data_smithi194_196924_object110642:head# 2024-04-29T14:59:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42fbbfd1:::benchmark_data_smithi194_196924_object101443:head# 2024-04-29T14:59:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42fe2d3a:::benchmark_data_smithi194_196924_object132472:head# 2024-04-29T14:59:52.993 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ff79f6:::benchmark_data_smithi194_196924_object117003:head# 2024-04-29T14:59:52.993 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:42ff968f:::benchmark_data_smithi194_196924_object100770:head# 2024-04-29T14:59:52.994 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430051d1:::benchmark_data_smithi194_196924_object152382:head# 2024-04-29T14:59:52.994 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4300b732:::benchmark_data_smithi194_196924_object119184:head# 2024-04-29T14:59:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4301b916:::benchmark_data_smithi194_196924_object109433:head# 2024-04-29T14:59:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43027dfa:::benchmark_data_smithi194_196924_object113280:head# 2024-04-29T14:59:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4302e5dc:::benchmark_data_smithi194_196924_object151614:head# 2024-04-29T14:59:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430491d3:::benchmark_data_smithi194_196924_object109992:head# 2024-04-29T14:59:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4304b56e:::benchmark_data_smithi194_196924_object156923:head# 2024-04-29T14:59:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4304cfdb:::benchmark_data_smithi194_196924_object157640:head# 2024-04-29T14:59:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430670d2:::benchmark_data_smithi194_196924_object159675:head# 2024-04-29T14:59:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43074d06:::benchmark_data_smithi194_196924_object159870:head# 2024-04-29T14:59:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4308ad42:::benchmark_data_smithi194_196924_object155764:head# 2024-04-29T14:59:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4309fc2f:::benchmark_data_smithi194_196924_object137733:head# 2024-04-29T14:59:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430ae4da:::benchmark_data_smithi194_196924_object130555:head# 2024-04-29T14:59:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430b2b66:::benchmark_data_smithi194_196924_object106035:head# 2024-04-29T14:59:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430b380a:::benchmark_data_smithi194_196924_object156019:head# 2024-04-29T14:59:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430d1a27:::benchmark_data_smithi194_196924_object126481:head# 2024-04-29T14:59:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:430f49f1:::benchmark_data_smithi194_196924_object99746:head# 2024-04-29T14:59:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:431195bf:::benchmark_data_smithi194_196924_object122263:head# 2024-04-29T14:59:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4313c640:::benchmark_data_smithi194_196924_object134405:head# 2024-04-29T14:59:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4313eba1:::benchmark_data_smithi194_196924_object99554:head# 2024-04-29T14:59:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4313feef:::benchmark_data_smithi194_196924_object105488:head# 2024-04-29T14:59:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43147af5:::benchmark_data_smithi194_196924_object158072:head# 2024-04-29T14:59:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:431494d7:::benchmark_data_smithi194_196924_object105730:head# 2024-04-29T14:59:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4314cf4f:::benchmark_data_smithi194_196924_object134355:head# 2024-04-29T14:59:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4317865e:::benchmark_data_smithi194_196924_object140757:head# 2024-04-29T14:59:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4317e11c:::benchmark_data_smithi194_196924_object111474:head# 2024-04-29T14:59:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:431ccb89:::benchmark_data_smithi194_196924_object134939:head# 2024-04-29T14:59:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:431d8e4d:::benchmark_data_smithi194_196924_object153385:head# 2024-04-29T14:59:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432097f5:::benchmark_data_smithi194_196924_object135880:head# 2024-04-29T14:59:53.007 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4320f9f6:::benchmark_data_smithi194_196924_object99826:head# 2024-04-29T14:59:53.007 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43242341:::benchmark_data_smithi194_196924_object132690:head# 2024-04-29T14:59:53.008 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43242fff:::benchmark_data_smithi194_196924_object132078:head# 2024-04-29T14:59:53.008 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4324e08b:::benchmark_data_smithi194_196924_object155173:head# 2024-04-29T14:59:53.009 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43252a9b:::benchmark_data_smithi194_196924_object112990:head# 2024-04-29T14:59:53.009 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432620aa:::benchmark_data_smithi194_196924_object122782:head# 2024-04-29T14:59:53.010 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43279f6d:::benchmark_data_smithi194_196924_object132786:head# 2024-04-29T14:59:53.010 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43280677:::benchmark_data_smithi194_196924_object100598:head# 2024-04-29T14:59:53.011 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43291fd3:::benchmark_data_smithi194_196924_object116342:head# 2024-04-29T14:59:53.011 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43294aa8:::benchmark_data_smithi194_196924_object156748:head# 2024-04-29T14:59:53.011 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432aeafb:::benchmark_data_smithi194_196924_object155408:head# 2024-04-29T14:59:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432b8763:::benchmark_data_smithi194_196924_object151019:head# 2024-04-29T14:59:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432b97d9:::benchmark_data_smithi194_196924_object107088:head# 2024-04-29T14:59:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432be625:::benchmark_data_smithi194_196924_object106500:head# 2024-04-29T14:59:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432c6bfd:::benchmark_data_smithi194_196924_object100909:head# 2024-04-29T14:59:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432d25be:::benchmark_data_smithi194_196924_object121103:head# 2024-04-29T14:59:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432de5c7:::benchmark_data_smithi194_196924_object128861:head# 2024-04-29T14:59:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432df02f:::benchmark_data_smithi194_196924_object101470:head# 2024-04-29T14:59:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432e8c1a:::benchmark_data_smithi194_196924_object127341:head# 2024-04-29T14:59:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:432ef306:::benchmark_data_smithi194_196924_object126080:head# 2024-04-29T14:59:53.016 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4330f95e:::benchmark_data_smithi194_196924_object153237:head# 2024-04-29T14:59:53.016 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:433312a6:::benchmark_data_smithi194_196924_object96071:head# 2024-04-29T14:59:53.016 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4333a102:::benchmark_data_smithi194_196924_object128322:head# 2024-04-29T14:59:53.017 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43347c46:::benchmark_data_smithi194_196924_object96909:head# 2024-04-29T14:59:53.017 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43356672:::benchmark_data_smithi194_196924_object113953:head# 2024-04-29T14:59:53.018 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4338929b:::benchmark_data_smithi194_196924_object136290:head# 2024-04-29T14:59:53.018 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43389393:::benchmark_data_smithi194_196924_object111570:head# 2024-04-29T14:59:53.018 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4338dc70:::benchmark_data_smithi194_196924_object151162:head# 2024-04-29T14:59:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4339ca2f:::benchmark_data_smithi194_196924_object121697:head# 2024-04-29T14:59:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:433a833b:::benchmark_data_smithi194_196924_object140046:head# 2024-04-29T14:59:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:433c2914:::benchmark_data_smithi194_196924_object105310:head# 2024-04-29T14:59:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:433d2071:::benchmark_data_smithi194_196924_object147481:head# 2024-04-29T14:59:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:433f100d:::benchmark_data_smithi194_196924_object149344:head# 2024-04-29T14:59:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:433f65d4:::benchmark_data_smithi194_196924_object151836:head# 2024-04-29T14:59:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:434166c2:::benchmark_data_smithi194_196924_object119275:head# 2024-04-29T14:59:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4341d546:::benchmark_data_smithi194_196924_object142566:head# 2024-04-29T14:59:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:434450ef:::benchmark_data_smithi194_196924_object97262:head# 2024-04-29T14:59:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4345c618:::benchmark_data_smithi194_196924_object117369:head# 2024-04-29T14:59:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4347af97:::benchmark_data_smithi194_196924_object155194:head# 2024-04-29T14:59:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:434a5c60:::benchmark_data_smithi194_196924_object123739:head# 2024-04-29T14:59:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:434d8647:::benchmark_data_smithi194_196924_object103223:head# 2024-04-29T14:59:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:434dcd2b:::benchmark_data_smithi194_196924_object99601:head# 2024-04-29T14:59:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43522f2c:::benchmark_data_smithi194_196924_object113706:head# 2024-04-29T14:59:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43529c80:::benchmark_data_smithi194_196924_object157993:head# 2024-04-29T14:59:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43533b89:::benchmark_data_smithi194_196924_object153604:head# 2024-04-29T14:59:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43542ac8:::benchmark_data_smithi194_196924_object107237:head# 2024-04-29T14:59:53.027 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4357a0af:::benchmark_data_smithi194_196924_object123100:head# 2024-04-29T14:59:53.027 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4358c392:::benchmark_data_smithi194_196924_object153255:head# 2024-04-29T14:59:53.027 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4358d38a:::benchmark_data_smithi194_196924_object148656:head# 2024-04-29T14:59:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435aa72f:::benchmark_data_smithi194_196924_object156591:head# 2024-04-29T14:59:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435aac65:::benchmark_data_smithi194_196924_object109779:head# 2024-04-29T14:59:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435b51e7:::benchmark_data_smithi194_196924_object154039:head# 2024-04-29T14:59:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435bd122:::benchmark_data_smithi194_196924_object113421:head# 2024-04-29T14:59:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435c4323:::benchmark_data_smithi194_196924_object120763:head# 2024-04-29T14:59:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435ce4eb:::benchmark_data_smithi194_196924_object112689:head# 2024-04-29T14:59:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435dadad:::benchmark_data_smithi194_196924_object97560:head# 2024-04-29T14:59:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435e00c7:::benchmark_data_smithi194_196924_object119144:head# 2024-04-29T14:59:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435e881a:::benchmark_data_smithi194_196924_object136866:head# 2024-04-29T14:59:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:435ebb08:::benchmark_data_smithi194_196924_object115708:head# 2024-04-29T14:59:53.032 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43606ec1:::benchmark_data_smithi194_196924_object147642:head# 2024-04-29T14:59:53.032 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4360b867:::benchmark_data_smithi194_196924_object146437:head# 2024-04-29T14:59:53.032 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4360dd87:::benchmark_data_smithi194_196924_object108340:head# 2024-04-29T14:59:53.033 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4361841f:::benchmark_data_smithi194_196924_object118681:head# 2024-04-29T14:59:53.033 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4361c996:::benchmark_data_smithi194_196924_object123172:head# 2024-04-29T14:59:53.034 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43622105:::benchmark_data_smithi194_196924_object108046:head# 2024-04-29T14:59:53.034 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4364e03f:::benchmark_data_smithi194_196924_object122682:head# 2024-04-29T14:59:53.035 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43653f14:::benchmark_data_smithi194_196924_object136945:head# 2024-04-29T14:59:53.035 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4365f93e:::benchmark_data_smithi194_196924_object143382:head# 2024-04-29T14:59:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4366352c:::benchmark_data_smithi194_196924_object108750:head# 2024-04-29T14:59:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4366431f:::benchmark_data_smithi194_196924_object159356:head# 2024-04-29T14:59:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4366ec39:::benchmark_data_smithi194_196924_object157210:head# 2024-04-29T14:59:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43672458:::benchmark_data_smithi194_196924_object159982:head# 2024-04-29T14:59:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4368a4f3:::benchmark_data_smithi194_196924_object98602:head# 2024-04-29T14:59:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:436b9ba9:::benchmark_data_smithi194_196924_object122084:head# 2024-04-29T14:59:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:436c6990:::benchmark_data_smithi194_196924_object143558:head# 2024-04-29T14:59:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:436d486c:::benchmark_data_smithi194_196924_object125367:head# 2024-04-29T14:59:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43704349:::benchmark_data_smithi194_196924_object111979:head# 2024-04-29T14:59:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43705de5:::benchmark_data_smithi194_196924_object116188:head# 2024-04-29T14:59:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43731f42:::benchmark_data_smithi194_196924_object111604:head# 2024-04-29T14:59:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:437391d7:::benchmark_data_smithi194_196924_object117485:head# 2024-04-29T14:59:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4374dc69:::benchmark_data_smithi194_196924_object119719:head# 2024-04-29T14:59:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4375db94:::benchmark_data_smithi194_196924_object95672:head# 2024-04-29T14:59:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43762053:::benchmark_data_smithi194_196924_object151210:head# 2024-04-29T14:59:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4376b0ad:::benchmark_data_smithi194_196924_object126349:head# 2024-04-29T14:59:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43771643:::benchmark_data_smithi194_196924_object134941:head# 2024-04-29T14:59:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43787923:::benchmark_data_smithi194_196924_object119443:head# 2024-04-29T14:59:53.044 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:437b95a2:::benchmark_data_smithi194_196924_object146966:head# 2024-04-29T14:59:53.044 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:437c5677:::benchmark_data_smithi194_196924_object110064:head# 2024-04-29T14:59:53.044 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:437eae29:::benchmark_data_smithi194_196924_object124104:head# 2024-04-29T14:59:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4380cf8d:::benchmark_data_smithi194_196924_object102420:head# 2024-04-29T14:59:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43815bf1:::benchmark_data_smithi194_196924_object126972:head# 2024-04-29T14:59:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4388385a:::benchmark_data_smithi194_196924_object119535:head# 2024-04-29T14:59:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43896acc:::benchmark_data_smithi194_196924_object148193:head# 2024-04-29T14:59:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:438b9241:::benchmark_data_smithi194_196924_object154631:head# 2024-04-29T14:59:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:438d9dbb:::benchmark_data_smithi194_196924_object125108:head# 2024-04-29T14:59:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:438e5311:::benchmark_data_smithi194_196924_object111003:head# 2024-04-29T14:59:53.048 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:438e6c19:::benchmark_data_smithi194_196924_object116099:head# 2024-04-29T14:59:53.048 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:438feb41:::benchmark_data_smithi194_196924_object114700:head# 2024-04-29T14:59:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43903b42:::benchmark_data_smithi194_196924_object141122:head# 2024-04-29T14:59:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43930059:::benchmark_data_smithi194_196924_object145937:head# 2024-04-29T14:59:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439570f5:::benchmark_data_smithi194_196924_object133461:head# 2024-04-29T14:59:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43965ff4:::benchmark_data_smithi194_196924_object148243:head# 2024-04-29T14:59:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43974059:::benchmark_data_smithi194_196924_object153721:head# 2024-04-29T14:59:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4397415e:::benchmark_data_smithi194_196924_object131104:head# 2024-04-29T14:59:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4398c5fc:::benchmark_data_smithi194_196924_object115560:head# 2024-04-29T14:59:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439a99fa:::benchmark_data_smithi194_196924_object151753:head# 2024-04-29T14:59:53.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439aad57:::benchmark_data_smithi194_196924_object156409:head# 2024-04-29T14:59:53.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439b4256:::benchmark_data_smithi194_196924_object144254:head# 2024-04-29T14:59:53.053 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439c0957:::benchmark_data_smithi194_196924_object97981:head# 2024-04-29T14:59:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439cc738:::benchmark_data_smithi194_196924_object152938:head# 2024-04-29T14:59:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439e2b7a:::benchmark_data_smithi194_196924_object134403:head# 2024-04-29T14:59:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:439ec08b:::benchmark_data_smithi194_196924_object131373:head# 2024-04-29T14:59:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a23088:::benchmark_data_smithi194_196924_object136385:head# 2024-04-29T14:59:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a34eb1:::benchmark_data_smithi194_196924_object99186:head# 2024-04-29T14:59:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a3565e:::benchmark_data_smithi194_196924_object126543:head# 2024-04-29T14:59:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a92ce8:::benchmark_data_smithi194_196924_object115100:head# 2024-04-29T14:59:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a92d2f:::benchmark_data_smithi194_196924_object107419:head# 2024-04-29T14:59:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a98d3b:::benchmark_data_smithi194_196924_object128847:head# 2024-04-29T14:59:53.058 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a9b239:::benchmark_data_smithi194_196924_object125631:head# 2024-04-29T14:59:53.058 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43a9dde1:::benchmark_data_smithi194_196924_object156042:head# 2024-04-29T14:59:53.059 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ab822e:::benchmark_data_smithi194_196924_object147346:head# 2024-04-29T14:59:53.059 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ac4cd0:::benchmark_data_smithi194_196924_object109024:head# 2024-04-29T14:59:53.060 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ae2daa:::benchmark_data_smithi194_196924_object134480:head# 2024-04-29T14:59:53.060 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43af872a:::benchmark_data_smithi194_196924_object142890:head# 2024-04-29T14:59:53.060 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b14dd7:::benchmark_data_smithi194_196924_object155963:head# 2024-04-29T14:59:53.061 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b21d42:::benchmark_data_smithi194_196924_object102839:head# 2024-04-29T14:59:53.062 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b340df:::benchmark_data_smithi194_196924_object157833:head# 2024-04-29T14:59:53.062 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b461ed:::benchmark_data_smithi194_196924_object104191:head# 2024-04-29T14:59:53.062 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b525d5:::benchmark_data_smithi194_196924_object139842:head# 2024-04-29T14:59:53.063 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b60350:::benchmark_data_smithi194_196924_object155119:head# 2024-04-29T14:59:53.063 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b679f2:::benchmark_data_smithi194_196924_object118575:head# 2024-04-29T14:59:53.064 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43b9ceb9:::benchmark_data_smithi194_196924_object107704:head# 2024-04-29T14:59:53.064 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ba38d4:::benchmark_data_smithi194_196924_object114777:head# 2024-04-29T14:59:53.064 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ba8031:::benchmark_data_smithi194_196924_object116444:head# 2024-04-29T14:59:53.065 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43baa22c:::benchmark_data_smithi194_196924_object140619:head# 2024-04-29T14:59:53.065 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43bc7e9f:::benchmark_data_smithi194_196924_object106181:head# 2024-04-29T14:59:53.066 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c0f952:::benchmark_data_smithi194_196924_object147497:head# 2024-04-29T14:59:53.066 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c1aed6:::benchmark_data_smithi194_196924_object124356:head# 2024-04-29T14:59:53.067 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c218fa:::benchmark_data_smithi194_196924_object119298:head# 2024-04-29T14:59:53.067 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c2b98f:::benchmark_data_smithi194_196924_object120892:head# 2024-04-29T14:59:53.067 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c3edaa:::benchmark_data_smithi194_196924_object130667:head# 2024-04-29T14:59:53.068 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c45f50:::benchmark_data_smithi194_196924_object146698:head# 2024-04-29T14:59:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c60989:::benchmark_data_smithi194_196924_object152185:head# 2024-04-29T14:59:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c6b3ee:::benchmark_data_smithi194_196924_object140831:head# 2024-04-29T14:59:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c998c5:::benchmark_data_smithi194_196924_object109712:head# 2024-04-29T14:59:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43c9d385:::benchmark_data_smithi194_196924_object106162:head# 2024-04-29T14:59:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ca1ae0:::benchmark_data_smithi194_196924_object118714:head# 2024-04-29T14:59:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43cac3e3:::benchmark_data_smithi194_196924_object137625:head# 2024-04-29T14:59:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43cae237:::benchmark_data_smithi194_196924_object143596:head# 2024-04-29T14:59:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43cb0d83:::benchmark_data_smithi194_196924_object146311:head# 2024-04-29T14:59:53.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43cb19c4:::benchmark_data_smithi194_196924_object145705:head# 2024-04-29T14:59:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43cc2091:::benchmark_data_smithi194_196924_object159827:head# 2024-04-29T14:59:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ccde06:::benchmark_data_smithi194_196924_object137563:head# 2024-04-29T14:59:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43cdb833:::benchmark_data_smithi194_196924_object139154:head# 2024-04-29T14:59:53.074 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43cf6fc6:::benchmark_data_smithi194_196924_object146848:head# 2024-04-29T14:59:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:52 smithi194 ceph-mon[44089]: pgmap v2907: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 76 MiB/s, 1.21k objects/s recovering 2024-04-29T14:59:53.076 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d1e12e:::benchmark_data_smithi194_196924_object116794:head# 2024-04-29T14:59:53.076 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d3d171:::benchmark_data_smithi194_196924_object146448:head# 2024-04-29T14:59:53.076 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d446f4:::benchmark_data_smithi194_196924_object108968:head# 2024-04-29T14:59:53.076 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d4a41f:::benchmark_data_smithi194_196924_object109532:head# 2024-04-29T14:59:53.077 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d60bc0:::benchmark_data_smithi194_196924_object111908:head# 2024-04-29T14:59:53.077 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d7815b:::benchmark_data_smithi194_196924_object110478:head# 2024-04-29T14:59:53.077 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d8afc3:::benchmark_data_smithi194_196924_object120474:head# 2024-04-29T14:59:53.078 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43d9d21f:::benchmark_data_smithi194_196924_object153576:head# 2024-04-29T14:59:53.078 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43dbcadb:::benchmark_data_smithi194_196924_object146656:head# 2024-04-29T14:59:53.079 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ddfac6:::benchmark_data_smithi194_196924_object133205:head# 2024-04-29T14:59:53.079 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43dfabf6:::benchmark_data_smithi194_196924_object116800:head# 2024-04-29T14:59:53.080 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43dfe251:::benchmark_data_smithi194_196924_object130941:head# 2024-04-29T14:59:53.080 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e4af9b:::benchmark_data_smithi194_196924_object129822:head# 2024-04-29T14:59:53.080 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e4f45b:::benchmark_data_smithi194_196924_object145432:head# 2024-04-29T14:59:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e521fc:::benchmark_data_smithi194_196924_object97826:head# 2024-04-29T14:59:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e5440f:::benchmark_data_smithi194_196924_object128647:head# 2024-04-29T14:59:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e6b3ef:::benchmark_data_smithi194_196924_object159364:head# 2024-04-29T14:59:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e735bc:::benchmark_data_smithi194_196924_object95926:head# 2024-04-29T14:59:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e81018:::benchmark_data_smithi194_196924_object96277:head# 2024-04-29T14:59:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43e9750c:::benchmark_data_smithi194_196924_object140489:head# 2024-04-29T14:59:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43eae882:::benchmark_data_smithi194_196924_object120636:head# 2024-04-29T14:59:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43eb8725:::benchmark_data_smithi194_196924_object158795:head# 2024-04-29T14:59:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ed706b:::benchmark_data_smithi194_196924_object112401:head# 2024-04-29T14:59:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ee1a28:::benchmark_data_smithi194_196924_object139682:head# 2024-04-29T14:59:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43ee7eca:::benchmark_data_smithi194_196924_object134738:head# 2024-04-29T14:59:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43eea6c5:::benchmark_data_smithi194_196924_object141787:head# 2024-04-29T14:59:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f02721:::benchmark_data_smithi194_196924_object148079:head# 2024-04-29T14:59:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f04dc3:::benchmark_data_smithi194_196924_object136649:head# 2024-04-29T14:59:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f11793:::benchmark_data_smithi194_196924_object117478:head# 2024-04-29T14:59:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f124b1:::benchmark_data_smithi194_196924_object119861:head# 2024-04-29T14:59:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f26110:::benchmark_data_smithi194_196924_object106574:head# 2024-04-29T14:59:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f30868:::benchmark_data_smithi194_196924_object154081:head# 2024-04-29T14:59:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f35cad:::benchmark_data_smithi194_196924_object116225:head# 2024-04-29T14:59:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f418d9:::benchmark_data_smithi194_196924_object100985:head# 2024-04-29T14:59:53.089 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f4d35e:::benchmark_data_smithi194_196924_object150596:head# 2024-04-29T14:59:53.089 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f588c7:::benchmark_data_smithi194_196924_object119079:head# 2024-04-29T14:59:53.090 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f66691:::benchmark_data_smithi194_196924_object148267:head# 2024-04-29T14:59:53.090 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f70470:::benchmark_data_smithi194_196924_object111025:head# 2024-04-29T14:59:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43f8c06e:::benchmark_data_smithi194_196924_object125767:head# 2024-04-29T14:59:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43faeba0:::benchmark_data_smithi194_196924_object111061:head# 2024-04-29T14:59:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43fb45fc:::benchmark_data_smithi194_196924_object108827:head# 2024-04-29T14:59:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43fb5f33:::benchmark_data_smithi194_196924_object127576:head# 2024-04-29T14:59:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43fb6eb4:::benchmark_data_smithi194_196924_object140458:head# 2024-04-29T14:59:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43fb99c9:::benchmark_data_smithi194_196924_object112934:head# 2024-04-29T14:59:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43fbb239:::benchmark_data_smithi194_196924_object153167:head# 2024-04-29T14:59:53.094 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:43fcfe49:::benchmark_data_smithi194_196924_object139834:head# 2024-04-29T14:59:53.094 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44013c6c:::benchmark_data_smithi194_196924_object137341:head# 2024-04-29T14:59:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44026e32:::benchmark_data_smithi194_196924_object145030:head# 2024-04-29T14:59:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44031eb5:::benchmark_data_smithi194_196924_object117669:head# 2024-04-29T14:59:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4403e0fd:::benchmark_data_smithi194_196924_object108025:head# 2024-04-29T14:59:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4404390c:::benchmark_data_smithi194_196924_object143302:head# 2024-04-29T14:59:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:440446a3:::benchmark_data_smithi194_196924_object98690:head# 2024-04-29T14:59:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4404db8c:::benchmark_data_smithi194_196924_object118294:head# 2024-04-29T14:59:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44050ec5:::benchmark_data_smithi194_196924_object117329:head# 2024-04-29T14:59:53.098 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4407bdcf:::benchmark_data_smithi194_196924_object121010:head# 2024-04-29T14:59:53.098 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4408ae04:::benchmark_data_smithi194_196924_object119525:head# 2024-04-29T14:59:53.099 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4408deaf:::benchmark_data_smithi194_196924_object135682:head# 2024-04-29T14:59:53.099 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:440a40ca:::benchmark_data_smithi194_196924_object140864:head# 2024-04-29T14:59:53.100 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:440b7265:::benchmark_data_smithi194_196924_object140185:head# 2024-04-29T14:59:53.100 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:440bb8ff:::benchmark_data_smithi194_196924_object96567:head# 2024-04-29T14:59:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:440ea372:::benchmark_data_smithi194_196924_object154530:head# 2024-04-29T14:59:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441022f8:::benchmark_data_smithi194_196924_object107513:head# 2024-04-29T14:59:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4410477e:::benchmark_data_smithi194_196924_object153960:head# 2024-04-29T14:59:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44109e89:::benchmark_data_smithi194_196924_object119409:head# 2024-04-29T14:59:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4411feab:::benchmark_data_smithi194_196924_object113082:head# 2024-04-29T14:59:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441259ee:::benchmark_data_smithi194_196924_object103946:head# 2024-04-29T14:59:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44128e28:::benchmark_data_smithi194_196924_object151835:head# 2024-04-29T14:59:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44146550:::benchmark_data_smithi194_196924_object146434:head# 2024-04-29T14:59:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44149b9b:::benchmark_data_smithi194_196924_object113318:head# 2024-04-29T14:59:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4415264e:::benchmark_data_smithi194_196924_object132664:head# 2024-04-29T14:59:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44165f8c:::benchmark_data_smithi194_196924_object134828:head# 2024-04-29T14:59:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44166f48:::benchmark_data_smithi194_196924_object151410:head# 2024-04-29T14:59:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44184e4b:::benchmark_data_smithi194_196924_object116629:head# 2024-04-29T14:59:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441886ee:::benchmark_data_smithi194_196924_object99977:head# 2024-04-29T14:59:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441a842a:::benchmark_data_smithi194_196924_object116012:head# 2024-04-29T14:59:53.108 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441ab324:::benchmark_data_smithi194_196924_object97033:head# 2024-04-29T14:59:53.108 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441bb656:::benchmark_data_smithi194_196924_object153152:head# 2024-04-29T14:59:53.109 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441ccdf4:::benchmark_data_smithi194_196924_object105223:head# 2024-04-29T14:59:53.109 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441ceb49:::benchmark_data_smithi194_196924_object155720:head# 2024-04-29T14:59:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441db8c1:::benchmark_data_smithi194_196924_object144328:head# 2024-04-29T14:59:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441f6cac:::benchmark_data_smithi194_196924_object159250:head# 2024-04-29T14:59:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:441fe75e:::benchmark_data_smithi194_196924_object120417:head# 2024-04-29T14:59:53.111 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44210ceb:::benchmark_data_smithi194_196924_object114186:head# 2024-04-29T14:59:53.111 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44219e66:::benchmark_data_smithi194_196924_object143286:head# 2024-04-29T14:59:53.112 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44227a63:::benchmark_data_smithi194_196924_object102460:head# 2024-04-29T14:59:53.112 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44236dea:::benchmark_data_smithi194_196924_object150745:head# 2024-04-29T14:59:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:442370e9:::benchmark_data_smithi194_196924_object105889:head# 2024-04-29T14:59:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:442513e6:::benchmark_data_smithi194_196924_object134188:head# 2024-04-29T14:59:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4425c971:::benchmark_data_smithi194_196924_object119788:head# 2024-04-29T14:59:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44266e6a:::benchmark_data_smithi194_196924_object158412:head# 2024-04-29T14:59:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4426bf27:::benchmark_data_smithi194_196924_object151573:head# 2024-04-29T14:59:53.115 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4426d64d:::benchmark_data_smithi194_196924_object152385:head# 2024-04-29T14:59:53.115 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4426f169:::benchmark_data_smithi194_196924_object142269:head# 2024-04-29T14:59:53.116 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4429c3d2:::benchmark_data_smithi194_196924_object96066:head# 2024-04-29T14:59:53.116 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:442cf589:::benchmark_data_smithi194_196924_object99943:head# 2024-04-29T14:59:53.117 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:442e2df5:::benchmark_data_smithi194_196924_object141900:head# 2024-04-29T14:59:53.118 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:442e5860:::benchmark_data_smithi194_196924_object154955:head# 2024-04-29T14:59:53.118 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:442ebebb:::benchmark_data_smithi194_196924_object121750:head# 2024-04-29T14:59:53.118 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4430fedd:::benchmark_data_smithi194_196924_object146818:head# 2024-04-29T14:59:53.119 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44316d08:::benchmark_data_smithi194_196924_object111161:head# 2024-04-29T14:59:53.120 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4431a513:::benchmark_data_smithi194_196924_object101784:head# 2024-04-29T14:59:53.121 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4431d391:::benchmark_data_smithi194_196924_object101927:head# 2024-04-29T14:59:53.122 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4437660c:::benchmark_data_smithi194_196924_object150694:head# 2024-04-29T14:59:53.122 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4437e396:::benchmark_data_smithi194_196924_object140590:head# 2024-04-29T14:59:53.123 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44383fbd:::benchmark_data_smithi194_196924_object156842:head# 2024-04-29T14:59:53.123 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4439141e:::benchmark_data_smithi194_196924_object125292:head# 2024-04-29T14:59:53.123 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:443a8d62:::benchmark_data_smithi194_196924_object152942:head# 2024-04-29T14:59:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:443d74bd:::benchmark_data_smithi194_196924_object107874:head# 2024-04-29T14:59:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:443e7229:::benchmark_data_smithi194_196924_object146748:head# 2024-04-29T14:59:53.125 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:443fe318:::benchmark_data_smithi194_196924_object148422:head# 2024-04-29T14:59:53.125 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:443fe33e:::benchmark_data_smithi194_196924_object106541:head# 2024-04-29T14:59:53.125 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4440119c:::benchmark_data_smithi194_196924_object116645:head# 2024-04-29T14:59:53.126 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4442f80c:::benchmark_data_smithi194_196924_object126621:head# 2024-04-29T14:59:53.126 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4443240f:::benchmark_data_smithi194_196924_object130622:head# 2024-04-29T14:59:53.127 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4443d543:::benchmark_data_smithi194_196924_object159473:head# 2024-04-29T14:59:53.127 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44445e3f:::benchmark_data_smithi194_196924_object129533:head# 2024-04-29T14:59:53.128 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4445ed49:::benchmark_data_smithi194_196924_object115929:head# 2024-04-29T14:59:53.128 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44460523:::benchmark_data_smithi194_196924_object138454:head# 2024-04-29T14:59:53.129 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4446e06e:::benchmark_data_smithi194_196924_object106824:head# 2024-04-29T14:59:53.129 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:444757c5:::benchmark_data_smithi194_196924_object124701:head# 2024-04-29T14:59:53.130 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:444829a0:::benchmark_data_smithi194_196924_object109937:head# 2024-04-29T14:59:53.130 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:444952ef:::benchmark_data_smithi194_196924_object145458:head# 2024-04-29T14:59:53.131 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:444b73ab:::benchmark_data_smithi194_196924_object127255:head# 2024-04-29T14:59:53.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T14:59:53.132 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:52 smithi121 ceph-mon[44533]: pgmap v2907: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 76 MiB/s, 1.21k objects/s recovering 2024-04-29T14:59:53.132 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:444c544a:::benchmark_data_smithi194_196924_object105028:head# 2024-04-29T14:59:53.133 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:444f7cd5:::benchmark_data_smithi194_196924_object104256:head# 2024-04-29T14:59:53.133 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4450032b:::benchmark_data_smithi194_196924_object151827:head# 2024-04-29T14:59:53.133 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44502d03:::benchmark_data_smithi194_196924_object138649:head# 2024-04-29T14:59:53.133 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445171e1:::benchmark_data_smithi194_196924_object100333:head# 2024-04-29T14:59:53.134 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4452b572:::benchmark_data_smithi194_196924_object159992:head# 2024-04-29T14:59:53.134 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4453aa47:::benchmark_data_smithi194_196924_object101382:head# 2024-04-29T14:59:53.134 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44543e0d:::benchmark_data_smithi194_196924_object114742:head# 2024-04-29T14:59:53.135 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4454c238:::benchmark_data_smithi194_196924_object141365:head# 2024-04-29T14:59:53.135 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4456232b:::benchmark_data_smithi194_196924_object147728:head# 2024-04-29T14:59:53.136 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445699fb:::benchmark_data_smithi194_196924_object142952:head# 2024-04-29T14:59:53.136 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44575bc0:::benchmark_data_smithi194_196924_object96471:head# 2024-04-29T14:59:53.137 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445a8843:::benchmark_data_smithi194_196924_object122832:head# 2024-04-29T14:59:53.137 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445b007c:::benchmark_data_smithi194_196924_object111892:head# 2024-04-29T14:59:53.138 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445c4906:::benchmark_data_smithi194_196924_object117252:head# 2024-04-29T14:59:53.138 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445c9f5a:::benchmark_data_smithi194_196924_object159821:head# 2024-04-29T14:59:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445d7031:::benchmark_data_smithi194_196924_object117553:head# 2024-04-29T14:59:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445ee788:::benchmark_data_smithi194_196924_object134771:head# 2024-04-29T14:59:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:445f7d2f:::benchmark_data_smithi194_196924_object119937:head# 2024-04-29T14:59:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:446243cb:::benchmark_data_smithi194_196924_object127153:head# 2024-04-29T14:59:53.141 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44628caa:::benchmark_data_smithi194_196924_object141901:head# 2024-04-29T14:59:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44632439:::benchmark_data_smithi194_196924_object145400:head# 2024-04-29T14:59:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44639a42:::benchmark_data_smithi194_196924_object133793:head# 2024-04-29T14:59:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4463d0b6:::benchmark_data_smithi194_196924_object140756:head# 2024-04-29T14:59:53.144 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4464b209:::benchmark_data_smithi194_196924_object152755:head# 2024-04-29T14:59:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4465008c:::benchmark_data_smithi194_196924_object137238:head# 2024-04-29T14:59:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44666bd2:::benchmark_data_smithi194_196924_object107252:head# 2024-04-29T14:59:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4468178a:::benchmark_data_smithi194_196924_object136715:head# 2024-04-29T14:59:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4469eae1:::benchmark_data_smithi194_196924_object138899:head# 2024-04-29T14:59:53.147 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:446b59d6:::benchmark_data_smithi194_196924_object149756:head# 2024-04-29T14:59:53.147 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:446bb961:::benchmark_data_smithi194_196924_object154372:head# 2024-04-29T14:59:53.148 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:446bfecc:::benchmark_data_smithi194_196924_object113435:head# 2024-04-29T14:59:53.148 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:446cb014:::benchmark_data_smithi194_196924_object123888:head# 2024-04-29T14:59:53.148 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:446ddc7c:::benchmark_data_smithi194_196924_object143265:head# 2024-04-29T14:59:53.149 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44700b4b:::benchmark_data_smithi194_196924_object159223:head# 2024-04-29T14:59:53.149 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44713fe7:::benchmark_data_smithi194_196924_object112167:head# 2024-04-29T14:59:53.150 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:447161aa:::benchmark_data_smithi194_196924_object149035:head# 2024-04-29T14:59:53.150 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44724569:::benchmark_data_smithi194_196924_object158789:head# 2024-04-29T14:59:53.151 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4474039a:::benchmark_data_smithi194_196924_object120453:head# 2024-04-29T14:59:53.151 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44742608:::benchmark_data_smithi194_196924_object102135:head# 2024-04-29T14:59:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44748cbc:::benchmark_data_smithi194_196924_object141541:head# 2024-04-29T14:59:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4476f911:::benchmark_data_smithi194_196924_object149687:head# 2024-04-29T14:59:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44770227:::benchmark_data_smithi194_196924_object115259:head# 2024-04-29T14:59:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44787484:::benchmark_data_smithi194_196924_object128319:head# 2024-04-29T14:59:53.155 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:447988d5:::benchmark_data_smithi194_196924_object133776:head# 2024-04-29T14:59:53.155 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:447acb1c:::benchmark_data_smithi194_196924_object97025:head# 2024-04-29T14:59:53.156 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:447ae6a3:::benchmark_data_smithi194_196924_object142153:head# 2024-04-29T14:59:53.156 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:447f1c21:::benchmark_data_smithi194_196924_object158582:head# 2024-04-29T14:59:53.156 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4481541b:::benchmark_data_smithi194_196924_object118090:head# 2024-04-29T14:59:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44818035:::benchmark_data_smithi194_196924_object114752:head# 2024-04-29T14:59:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44826061:::benchmark_data_smithi194_196924_object114967:head# 2024-04-29T14:59:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4482cd6e:::benchmark_data_smithi194_196924_object144627:head# 2024-04-29T14:59:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4482ecee:::benchmark_data_smithi194_196924_object102192:head# 2024-04-29T14:59:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44844c3c:::benchmark_data_smithi194_196924_object105429:head# 2024-04-29T14:59:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4484e83f:::benchmark_data_smithi194_196924_object108224:head# 2024-04-29T14:59:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44892bc8:::benchmark_data_smithi194_196924_object114928:head# 2024-04-29T14:59:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:448931e0:::benchmark_data_smithi194_196924_object122302:head# 2024-04-29T14:59:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44898d5f:::benchmark_data_smithi194_196924_object130373:head# 2024-04-29T14:59:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:448b473d:::benchmark_data_smithi194_196924_object156339:head# 2024-04-29T14:59:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:448bb944:::benchmark_data_smithi194_196924_object113334:head# 2024-04-29T14:59:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:448cccc7:::benchmark_data_smithi194_196924_object124722:head# 2024-04-29T14:59:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:448d2808:::benchmark_data_smithi194_196924_object139866:head# 2024-04-29T14:59:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:448fff50:::benchmark_data_smithi194_196924_object96788:head# 2024-04-29T14:59:53.166 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4493baaf:::benchmark_data_smithi194_196924_object105724:head# 2024-04-29T14:59:53.166 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4493cdcb:::benchmark_data_smithi194_196924_object115254:head# 2024-04-29T14:59:53.167 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4494e780:::benchmark_data_smithi194_196924_object125039:head# 2024-04-29T14:59:53.167 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44967027:::benchmark_data_smithi194_196924_object126688:head# 2024-04-29T14:59:53.167 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4496c896:::benchmark_data_smithi194_196924_object128199:head# 2024-04-29T14:59:53.168 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4497eac7:::benchmark_data_smithi194_196924_object107717:head# 2024-04-29T14:59:53.168 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:449a2c3d:::benchmark_data_smithi194_196924_object123360:head# 2024-04-29T14:59:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:449be2b1:::benchmark_data_smithi194_196924_object139907:head# 2024-04-29T14:59:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:449c8d50:::benchmark_data_smithi194_196924_object134863:head# 2024-04-29T14:59:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:449d3760:::benchmark_data_smithi194_196924_object154354:head# 2024-04-29T14:59:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:449dc5d5:::benchmark_data_smithi194_196924_object104542:head# 2024-04-29T14:59:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:449f1f90:::benchmark_data_smithi194_196924_object156559:head# 2024-04-29T14:59:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:449fb9ce:::benchmark_data_smithi194_196924_object105554:head# 2024-04-29T14:59:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a08237:::benchmark_data_smithi194_196924_object129790:head# 2024-04-29T14:59:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a0b7e4:::benchmark_data_smithi194_196924_object98545:head# 2024-04-29T14:59:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a0bb6a:::benchmark_data_smithi194_196924_object116736:head# 2024-04-29T14:59:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a1091a:::benchmark_data_smithi194_196924_object103586:head# 2024-04-29T14:59:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a2679f:::benchmark_data_smithi194_196924_object108535:head# 2024-04-29T14:59:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a32568:::benchmark_data_smithi194_196924_object105998:head# 2024-04-29T14:59:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a3b294:::benchmark_data_smithi194_196924_object123752:head# 2024-04-29T14:59:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a3d629:::benchmark_data_smithi194_196924_object148288:head# 2024-04-29T14:59:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a41133:::benchmark_data_smithi194_196924_object147912:head# 2024-04-29T14:59:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a45e4e:::benchmark_data_smithi194_196924_object128654:head# 2024-04-29T14:59:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a6914e:::benchmark_data_smithi194_196924_object124387:head# 2024-04-29T14:59:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a69ae9:::benchmark_data_smithi194_196924_object108418:head# 2024-04-29T14:59:53.177 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a6ff47:::benchmark_data_smithi194_196924_object156007:head# 2024-04-29T14:59:53.177 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a7302a:::benchmark_data_smithi194_196924_object101490:head# 2024-04-29T14:59:53.178 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a77d1a:::benchmark_data_smithi194_196924_object114720:head# 2024-04-29T14:59:53.179 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a7fddf:::benchmark_data_smithi194_196924_object102326:head# 2024-04-29T14:59:53.179 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a80b5a:::benchmark_data_smithi194_196924_object126646:head# 2024-04-29T14:59:53.180 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44a8163f:::benchmark_data_smithi194_196924_object119587:head# 2024-04-29T14:59:53.180 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ad6416:::benchmark_data_smithi194_196924_object112234:head# 2024-04-29T14:59:53.181 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44adf9f9:::benchmark_data_smithi194_196924_object96105:head# 2024-04-29T14:59:53.181 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44aee4db:::benchmark_data_smithi194_196924_object149132:head# 2024-04-29T14:59:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b1292e:::benchmark_data_smithi194_196924_object154584:head# 2024-04-29T14:59:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b13a39:::benchmark_data_smithi194_196924_object158891:head# 2024-04-29T14:59:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b265ae:::benchmark_data_smithi194_196924_object150713:head# 2024-04-29T14:59:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b34dfd:::benchmark_data_smithi194_196924_object137551:head# 2024-04-29T14:59:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b40a6d:::benchmark_data_smithi194_196924_object103792:head# 2024-04-29T14:59:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b455ec:::benchmark_data_smithi194_196924_object143628:head# 2024-04-29T14:59:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b7144a:::benchmark_data_smithi194_196924_object100009:head# 2024-04-29T14:59:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b71db6:::benchmark_data_smithi194_196924_object112880:head# 2024-04-29T14:59:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b76f47:::benchmark_data_smithi194_196924_object120018:head# 2024-04-29T14:59:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b836c3:::benchmark_data_smithi194_196924_object147100:head# 2024-04-29T14:59:53.187 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b8bf78:::benchmark_data_smithi194_196924_object125330:head# 2024-04-29T14:59:53.187 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44b9ae2a:::benchmark_data_smithi194_196924_object143634:head# 2024-04-29T14:59:53.188 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ba93c5:::benchmark_data_smithi194_196924_object147827:head# 2024-04-29T14:59:53.188 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44bbf4cd:::benchmark_data_smithi194_196924_object117311:head# 2024-04-29T14:59:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44be117f:::benchmark_data_smithi194_196924_object99570:head# 2024-04-29T14:59:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44bf520e:::benchmark_data_smithi194_196924_object153555:head# 2024-04-29T14:59:53.190 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44bf82a2:::benchmark_data_smithi194_196924_object136499:head# 2024-04-29T14:59:53.190 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44bf8855:::benchmark_data_smithi194_196924_object139668:head# 2024-04-29T14:59:53.191 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44c1d16d:::benchmark_data_smithi194_196924_object127236:head# 2024-04-29T14:59:53.191 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44c20ecd:::benchmark_data_smithi194_196924_object109886:head# 2024-04-29T14:59:53.191 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44c25a12:::benchmark_data_smithi194_196924_object135927:head# 2024-04-29T14:59:53.193 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44c53284:::benchmark_data_smithi194_196924_object145475:head# 2024-04-29T14:59:53.193 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44c772d2:::benchmark_data_smithi194_196924_object136375:head# 2024-04-29T14:59:53.194 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44c7dfe0:::benchmark_data_smithi194_196924_object149911:head# 2024-04-29T14:59:53.194 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44c997a8:::benchmark_data_smithi194_196924_object138920:head# 2024-04-29T14:59:53.195 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ca5179:::benchmark_data_smithi194_196924_object106393:head# 2024-04-29T14:59:53.196 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ca937b:::benchmark_data_smithi194_196924_object159836:head# 2024-04-29T14:59:53.196 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44caa6d0:::benchmark_data_smithi194_196924_object114574:head# 2024-04-29T14:59:53.196 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44cab274:::benchmark_data_smithi194_196924_object148246:head# 2024-04-29T14:59:53.197 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44cb12e8:::benchmark_data_smithi194_196924_object146405:head# 2024-04-29T14:59:53.197 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44cb7755:::benchmark_data_smithi194_196924_object120673:head# 2024-04-29T14:59:53.198 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44cca125:::benchmark_data_smithi194_196924_object112241:head# 2024-04-29T14:59:53.198 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ccfae2:::benchmark_data_smithi194_196924_object113740:head# 2024-04-29T14:59:53.199 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44cd00f4:::benchmark_data_smithi194_196924_object139610:head# 2024-04-29T14:59:53.200 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ce7e63:::benchmark_data_smithi194_196924_object134093:head# 2024-04-29T14:59:53.200 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ceab0b:::benchmark_data_smithi194_196924_object132139:head# 2024-04-29T14:59:53.201 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44cec516:::benchmark_data_smithi194_196924_object111262:head# 2024-04-29T14:59:53.201 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44cf5f13:::benchmark_data_smithi194_196924_object111626:head# 2024-04-29T14:59:53.202 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d080d1:::benchmark_data_smithi194_196924_object131148:head# 2024-04-29T14:59:53.202 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d457f4:::benchmark_data_smithi194_196924_object97104:head# 2024-04-29T14:59:53.203 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d4cbbd:::benchmark_data_smithi194_196924_object105489:head# 2024-04-29T14:59:53.203 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d4eff1:::benchmark_data_smithi194_196924_object132765:head# 2024-04-29T14:59:53.204 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d519ad:::benchmark_data_smithi194_196924_object106820:head# 2024-04-29T14:59:53.204 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d5e2e6:::benchmark_data_smithi194_196924_object113013:head# 2024-04-29T14:59:53.205 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d63991:::benchmark_data_smithi194_196924_object95498:head# 2024-04-29T14:59:53.205 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d64d39:::benchmark_data_smithi194_196924_object146101:head# 2024-04-29T14:59:53.206 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d67329:::benchmark_data_smithi194_196924_object106590:head# 2024-04-29T14:59:53.207 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d6e7d8:::benchmark_data_smithi194_196924_object137881:head# 2024-04-29T14:59:53.207 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d78237:::benchmark_data_smithi194_196924_object133031:head# 2024-04-29T14:59:53.207 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d79398:::benchmark_data_smithi194_196924_object105923:head# 2024-04-29T14:59:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d82259:::benchmark_data_smithi194_196924_object125611:head# 2024-04-29T14:59:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d8900a:::benchmark_data_smithi194_196924_object120275:head# 2024-04-29T14:59:53.209 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44d98389:::benchmark_data_smithi194_196924_object144641:head# 2024-04-29T14:59:53.209 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44dc67ec:::benchmark_data_smithi194_196924_object117134:head# 2024-04-29T14:59:53.209 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44dcf291:::benchmark_data_smithi194_196924_object107199:head# 2024-04-29T14:59:53.210 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44dec746:::benchmark_data_smithi194_196924_object116368:head# 2024-04-29T14:59:53.210 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44df609a:::benchmark_data_smithi194_196924_object159681:head# 2024-04-29T14:59:53.211 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e0425a:::benchmark_data_smithi194_196924_object131950:head# 2024-04-29T14:59:53.211 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e11142:::benchmark_data_smithi194_196924_object104529:head# 2024-04-29T14:59:53.212 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e24922:::benchmark_data_smithi194_196924_object126476:head# 2024-04-29T14:59:53.212 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e253e7:::benchmark_data_smithi194_196924_object106410:head# 2024-04-29T14:59:53.213 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e492d4:::benchmark_data_smithi194_196924_object118040:head# 2024-04-29T14:59:53.213 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e5b0f5:::benchmark_data_smithi194_196924_object150285:head# 2024-04-29T14:59:53.214 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e8b567:::benchmark_data_smithi194_196924_object156635:head# 2024-04-29T14:59:53.214 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44e9b1b8:::benchmark_data_smithi194_196924_object100885:head# 2024-04-29T14:59:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ebff0a:::benchmark_data_smithi194_196924_object146565:head# 2024-04-29T14:59:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ec308b:::benchmark_data_smithi194_196924_object135007:head# 2024-04-29T14:59:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ed8efa:::benchmark_data_smithi194_196924_object102120:head# 2024-04-29T14:59:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ed9c63:::benchmark_data_smithi194_196924_object96354:head# 2024-04-29T14:59:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ef6a13:::benchmark_data_smithi194_196924_object100001:head# 2024-04-29T14:59:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44f35723:::benchmark_data_smithi194_196924_object151333:head# 2024-04-29T14:59:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44f4b296:::benchmark_data_smithi194_196924_object146166:head# 2024-04-29T14:59:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44f58240:::benchmark_data_smithi194_196924_object146076:head# 2024-04-29T14:59:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44f97cfe:::benchmark_data_smithi194_196924_object95499:head# 2024-04-29T14:59:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fbe718:::benchmark_data_smithi194_196924_object119363:head# 2024-04-29T14:59:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fbfa83:::benchmark_data_smithi194_196924_object115774:head# 2024-04-29T14:59:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fc25da:::benchmark_data_smithi194_196924_object123011:head# 2024-04-29T14:59:53.220 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fc799c:::benchmark_data_smithi194_196924_object101119:head# 2024-04-29T14:59:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fce403:::benchmark_data_smithi194_196924_object126467:head# 2024-04-29T14:59:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fdc92c:::benchmark_data_smithi194_196924_object136769:head# 2024-04-29T14:59:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fe499f:::benchmark_data_smithi194_196924_object150906:head# 2024-04-29T14:59:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44feafe7:::benchmark_data_smithi194_196924_object126548:head# 2024-04-29T14:59:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44fef1ba:::benchmark_data_smithi194_196924_object110265:head# 2024-04-29T14:59:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:44ff9aac:::benchmark_data_smithi194_196924_object153201:head# 2024-04-29T14:59:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45008701:::benchmark_data_smithi194_196924_object124570:head# 2024-04-29T14:59:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4503212a:::benchmark_data_smithi194_196924_object139068:head# 2024-04-29T14:59:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:450364bc:::benchmark_data_smithi194_196924_object153897:head# 2024-04-29T14:59:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45058f67:::benchmark_data_smithi194_196924_object156938:head# 2024-04-29T14:59:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45063c8a:::benchmark_data_smithi194_196924_object115378:head# 2024-04-29T14:59:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4506e2dd:::benchmark_data_smithi194_196924_object146288:head# 2024-04-29T14:59:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45088661:::benchmark_data_smithi194_196924_object100590:head# 2024-04-29T14:59:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:53.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:450896c8:::benchmark_data_smithi194_196924_object110968:head# 2024-04-29T14:59:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4508bec3:::benchmark_data_smithi194_196924_object156248:head# 2024-04-29T14:59:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45095ada:::benchmark_data_smithi194_196924_object154791:head# 2024-04-29T14:59:53.228 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4509f787:::benchmark_data_smithi194_196924_object132796:head# 2024-04-29T14:59:53.228 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:450c7c0f:::benchmark_data_smithi194_196924_object111487:head# 2024-04-29T14:59:53.228 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45104d74:::benchmark_data_smithi194_196924_object124077:head# 2024-04-29T14:59:53.229 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45115483:::benchmark_data_smithi194_196924_object122476:head# 2024-04-29T14:59:53.229 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45135c07:::benchmark_data_smithi194_196924_object106078:head# 2024-04-29T14:59:53.230 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:451445b0:::benchmark_data_smithi194_196924_object139580:head# 2024-04-29T14:59:53.230 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4514e830:::benchmark_data_smithi194_196924_object96744:head# 2024-04-29T14:59:53.230 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4517332a:::benchmark_data_smithi194_196924_object132368:head# 2024-04-29T14:59:53.231 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4517914f:::benchmark_data_smithi194_196924_object101515:head# 2024-04-29T14:59:53.231 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:451893ca:::benchmark_data_smithi194_196924_object151783:head# 2024-04-29T14:59:53.232 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:451a18ce:::benchmark_data_smithi194_196924_object144788:head# 2024-04-29T14:59:53.232 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:451d417e:::benchmark_data_smithi194_196924_object99600:head# 2024-04-29T14:59:53.233 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:451ea710:::benchmark_data_smithi194_196924_object133558:head# 2024-04-29T14:59:53.233 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:451fc674:::benchmark_data_smithi194_196924_object156123:head# 2024-04-29T14:59:53.234 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:452131ab:::benchmark_data_smithi194_196924_object156375:head# 2024-04-29T14:59:53.234 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4521778a:::benchmark_data_smithi194_196924_object144028:head# 2024-04-29T14:59:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4522954e:::benchmark_data_smithi194_196924_object139900:head# 2024-04-29T14:59:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4522bc32:::benchmark_data_smithi194_196924_object99835:head# 2024-04-29T14:59:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4522f031:::benchmark_data_smithi194_196924_object142725:head# 2024-04-29T14:59:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45233299:::benchmark_data_smithi194_196924_object158848:head# 2024-04-29T14:59:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4523f49e:::benchmark_data_smithi194_196924_object134264:head# 2024-04-29T14:59:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4525d184:::benchmark_data_smithi194_196924_object107469:head# 2024-04-29T14:59:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4526027e:::benchmark_data_smithi194_196924_object96621:head# 2024-04-29T14:59:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4526ea45:::benchmark_data_smithi194_196924_object157496:head# 2024-04-29T14:59:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4527abf9:::benchmark_data_smithi194_196924_object132419:head# 2024-04-29T14:59:53.238 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45296822:::benchmark_data_smithi194_196924_object151192:head# 2024-04-29T14:59:53.238 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:452b1220:::benchmark_data_smithi194_196924_object96789:head# 2024-04-29T14:59:53.239 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:452b200f:::benchmark_data_smithi194_196924_object102656:head# 2024-04-29T14:59:53.239 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:452b781d:::benchmark_data_smithi194_196924_object103401:head# 2024-04-29T14:59:53.240 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:452e4983:::benchmark_data_smithi194_196924_object151510:head# 2024-04-29T14:59:53.240 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453004d7:::benchmark_data_smithi194_196924_object120800:head# 2024-04-29T14:59:53.240 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45307b33:::benchmark_data_smithi194_196924_object148109:head# 2024-04-29T14:59:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4530a2d0:::benchmark_data_smithi194_196924_object100491:head# 2024-04-29T14:59:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45312483:::benchmark_data_smithi194_196924_object111614:head# 2024-04-29T14:59:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4531731a:::benchmark_data_smithi194_196924_object102921:head# 2024-04-29T14:59:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45340b4d:::benchmark_data_smithi194_196924_object120524:head# 2024-04-29T14:59:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45357fa2:::benchmark_data_smithi194_196924_object157309:head# 2024-04-29T14:59:53.247 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4536eccb:::benchmark_data_smithi194_196924_object117710:head# 2024-04-29T14:59:53.247 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453726fa:::benchmark_data_smithi194_196924_object111701:head# 2024-04-29T14:59:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45388f48:::benchmark_data_smithi194_196924_object103543:head# 2024-04-29T14:59:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4538e65d:::benchmark_data_smithi194_196924_object141053:head# 2024-04-29T14:59:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453983be:::benchmark_data_smithi194_196924_object159926:head# 2024-04-29T14:59:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453a3ec2:::benchmark_data_smithi194_196924_object136435:head# 2024-04-29T14:59:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453ac4ff:::benchmark_data_smithi194_196924_object101420:head# 2024-04-29T14:59:53.250 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453af4d1:::benchmark_data_smithi194_196924_object99773:head# 2024-04-29T14:59:53.250 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453b3f41:::benchmark_data_smithi194_196924_object134682:head# 2024-04-29T14:59:53.251 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453f1870:::benchmark_data_smithi194_196924_object107616:head# 2024-04-29T14:59:53.251 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453fd59d:::benchmark_data_smithi194_196924_object135917:head# 2024-04-29T14:59:53.252 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:453ffac8:::benchmark_data_smithi194_196924_object144823:head# 2024-04-29T14:59:53.252 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45410c27:::benchmark_data_smithi194_196924_object122254:head# 2024-04-29T14:59:53.252 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454254b1:::benchmark_data_smithi194_196924_object147614:head# 2024-04-29T14:59:53.253 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454355dd:::benchmark_data_smithi194_196924_object98035:head# 2024-04-29T14:59:53.253 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454496e7:::benchmark_data_smithi194_196924_object154756:head# 2024-04-29T14:59:53.254 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4545943d:::benchmark_data_smithi194_196924_object147762:head# 2024-04-29T14:59:53.254 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4545ce1b:::benchmark_data_smithi194_196924_object96234:head# 2024-04-29T14:59:53.255 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4547353e:::benchmark_data_smithi194_196924_object118573:head# 2024-04-29T14:59:53.255 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45480092:::benchmark_data_smithi194_196924_object143542:head# 2024-04-29T14:59:53.256 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45481bf5:::benchmark_data_smithi194_196924_object146510:head# 2024-04-29T14:59:53.256 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454931e8:::benchmark_data_smithi194_196924_object121064:head# 2024-04-29T14:59:53.257 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4549aaf8:::benchmark_data_smithi194_196924_object120756:head# 2024-04-29T14:59:53.257 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454ac28d:::benchmark_data_smithi194_196924_object101783:head# 2024-04-29T14:59:53.258 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454b4516:::benchmark_data_smithi194_196924_object136909:head# 2024-04-29T14:59:53.258 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454b8ded:::benchmark_data_smithi194_196924_object142555:head# 2024-04-29T14:59:53.258 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454c1e08:::benchmark_data_smithi194_196924_object145938:head# 2024-04-29T14:59:53.259 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454cda1f:::benchmark_data_smithi194_196924_object159323:head# 2024-04-29T14:59:53.259 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454d7c2a:::benchmark_data_smithi194_196924_object124338:head# 2024-04-29T14:59:53.260 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454d92f1:::benchmark_data_smithi194_196924_object126183:head# 2024-04-29T14:59:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454e36fa:::benchmark_data_smithi194_196924_object159625:head# 2024-04-29T14:59:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454e4582:::benchmark_data_smithi194_196924_object110595:head# 2024-04-29T14:59:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:454f8f4e:::benchmark_data_smithi194_196924_object150926:head# 2024-04-29T14:59:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45502a02:::benchmark_data_smithi194_196924_object159373:head# 2024-04-29T14:59:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45517b46:::benchmark_data_smithi194_196924_object98936:head# 2024-04-29T14:59:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45518ed4:::benchmark_data_smithi194_196924_object131070:head# 2024-04-29T14:59:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4552308d:::benchmark_data_smithi194_196924_object135543:head# 2024-04-29T14:59:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45533992:::benchmark_data_smithi194_196924_object106543:head# 2024-04-29T14:59:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455363a7:::benchmark_data_smithi194_196924_object144024:head# 2024-04-29T14:59:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45536952:::benchmark_data_smithi194_196924_object129757:head# 2024-04-29T14:59:53.265 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45538b21:::benchmark_data_smithi194_196924_object113537:head# 2024-04-29T14:59:53.265 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455482b8:::benchmark_data_smithi194_196924_object105173:head# 2024-04-29T14:59:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455515d7:::benchmark_data_smithi194_196924_object133338:head# 2024-04-29T14:59:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4555d1e7:::benchmark_data_smithi194_196924_object108375:head# 2024-04-29T14:59:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4556afe7:::benchmark_data_smithi194_196924_object146798:head# 2024-04-29T14:59:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4558dec7:::benchmark_data_smithi194_196924_object123235:head# 2024-04-29T14:59:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455960d3:::benchmark_data_smithi194_196924_object136111:head# 2024-04-29T14:59:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455a5329:::benchmark_data_smithi194_196924_object112374:head# 2024-04-29T14:59:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455b72fe:::benchmark_data_smithi194_196924_object137453:head# 2024-04-29T14:59:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455b8b80:::benchmark_data_smithi194_196924_object128065:head# 2024-04-29T14:59:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455c743b:::benchmark_data_smithi194_196924_object113027:head# 2024-04-29T14:59:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455d4754:::benchmark_data_smithi194_196924_object118778:head# 2024-04-29T14:59:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455da13d:::benchmark_data_smithi194_196924_object159222:head# 2024-04-29T14:59:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:455f6eb2:::benchmark_data_smithi194_196924_object108673:head# 2024-04-29T14:59:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4561e196:::benchmark_data_smithi194_196924_object144565:head# 2024-04-29T14:59:53.273 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45623a13:::benchmark_data_smithi194_196924_object159539:head# 2024-04-29T14:59:53.273 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45632671:::benchmark_data_smithi194_196924_object152139:head# 2024-04-29T14:59:53.274 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45642af6:::benchmark_data_smithi194_196924_object98453:head# 2024-04-29T14:59:53.274 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45657a78:::benchmark_data_smithi194_196924_object136016:head# 2024-04-29T14:59:53.274 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4565dc18:::benchmark_data_smithi194_196924_object120157:head# 2024-04-29T14:59:53.275 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4568d63d:::benchmark_data_smithi194_196924_object126279:head# 2024-04-29T14:59:53.275 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4568e0f6:::benchmark_data_smithi194_196924_object141249:head# 2024-04-29T14:59:53.276 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4569a3dc:::benchmark_data_smithi194_196924_object95408:head# 2024-04-29T14:59:53.276 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456ae786:::benchmark_data_smithi194_196924_object154562:head# 2024-04-29T14:59:53.276 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456b252b:::benchmark_data_smithi194_196924_object101506:head# 2024-04-29T14:59:53.277 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456b95e7:::benchmark_data_smithi194_196924_object97317:head# 2024-04-29T14:59:53.277 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456ba87b:::benchmark_data_smithi194_196924_object122395:head# 2024-04-29T14:59:53.278 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456bf385:::benchmark_data_smithi194_196924_object151326:head# 2024-04-29T14:59:53.278 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456d93d7:::benchmark_data_smithi194_196924_object145862:head# 2024-04-29T14:59:53.279 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456e9bc6:::benchmark_data_smithi194_196924_object151775:head# 2024-04-29T14:59:53.279 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456f6000:::benchmark_data_smithi194_196924_object110519:head# 2024-04-29T14:59:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:456f9c8a:::benchmark_data_smithi194_196924_object117526:head# 2024-04-29T14:59:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457021a4:::benchmark_data_smithi194_196924_object131492:head# 2024-04-29T14:59:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4571a34f:::benchmark_data_smithi194_196924_object127325:head# 2024-04-29T14:59:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4572868e:::benchmark_data_smithi194_196924_object158669:head# 2024-04-29T14:59:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4572be97:::benchmark_data_smithi194_196924_object152846:head# 2024-04-29T14:59:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4572e628:::benchmark_data_smithi194_196924_object141786:head# 2024-04-29T14:59:53.282 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45736c5e:::benchmark_data_smithi194_196924_object121985:head# 2024-04-29T14:59:53.282 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4573b631:::benchmark_data_smithi194_196924_object140139:head# 2024-04-29T14:59:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457451ad:::benchmark_data_smithi194_196924_object142847:head# 2024-04-29T14:59:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4574f587:::benchmark_data_smithi194_196924_object142013:head# 2024-04-29T14:59:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45756aab:::benchmark_data_smithi194_196924_object159138:head# 2024-04-29T14:59:53.284 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4575cf47:::benchmark_data_smithi194_196924_object103352:head# 2024-04-29T14:59:53.284 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45768be0:::benchmark_data_smithi194_196924_object152042:head# 2024-04-29T14:59:53.285 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4577068f:::benchmark_data_smithi194_196924_object137462:head# 2024-04-29T14:59:53.285 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45772add:::benchmark_data_smithi194_196924_object134445:head# 2024-04-29T14:59:53.285 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45774f90:::benchmark_data_smithi194_196924_object125674:head# 2024-04-29T14:59:53.286 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4577d5d7:::benchmark_data_smithi194_196924_object129235:head# 2024-04-29T14:59:53.286 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45798687:::benchmark_data_smithi194_196924_object127226:head# 2024-04-29T14:59:53.287 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457a92c0:::benchmark_data_smithi194_196924_object149550:head# 2024-04-29T14:59:53.287 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457b0367:::benchmark_data_smithi194_196924_object95932:head# 2024-04-29T14:59:53.288 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457b2cbe:::benchmark_data_smithi194_196924_object129704:head# 2024-04-29T14:59:53.288 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457b3e1e:::benchmark_data_smithi194_196924_object107343:head# 2024-04-29T14:59:53.288 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457e522a:::benchmark_data_smithi194_196924_object97852:head# 2024-04-29T14:59:53.289 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457e7a4d:::benchmark_data_smithi194_196924_object121796:head# 2024-04-29T14:59:53.289 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:457efe01:::benchmark_data_smithi194_196924_object95540:head# 2024-04-29T14:59:53.290 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45839c75:::benchmark_data_smithi194_196924_object148508:head# 2024-04-29T14:59:53.290 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:458440b3:::benchmark_data_smithi194_196924_object109283:head# 2024-04-29T14:59:53.290 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45844e2f:::benchmark_data_smithi194_196924_object159201:head# 2024-04-29T14:59:53.291 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4585a349:::benchmark_data_smithi194_196924_object130770:head# 2024-04-29T14:59:53.291 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:458619fc:::benchmark_data_smithi194_196924_object141042:head# 2024-04-29T14:59:53.292 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4586a4f3:::benchmark_data_smithi194_196924_object113092:head# 2024-04-29T14:59:53.292 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:458913de:::benchmark_data_smithi194_196924_object107129:head# 2024-04-29T14:59:53.293 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:458925c9:::benchmark_data_smithi194_196924_object143383:head# 2024-04-29T14:59:53.293 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:458c58ea:::benchmark_data_smithi194_196924_object114024:head# 2024-04-29T14:59:53.294 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:458e7975:::benchmark_data_smithi194_196924_object105376:head# 2024-04-29T14:59:53.294 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:459165f4:::benchmark_data_smithi194_196924_object133738:head# 2024-04-29T14:59:53.295 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45918a1c:::benchmark_data_smithi194_196924_object114359:head# 2024-04-29T14:59:53.295 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4593e162:::benchmark_data_smithi194_196924_object154625:head# 2024-04-29T14:59:53.295 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4594c6a0:::benchmark_data_smithi194_196924_object102899:head# 2024-04-29T14:59:53.296 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4596acd4:::benchmark_data_smithi194_196924_object134603:head# 2024-04-29T14:59:53.296 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45971702:::benchmark_data_smithi194_196924_object112188:head# 2024-04-29T14:59:53.297 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45999074:::benchmark_data_smithi194_196924_object98134:head# 2024-04-29T14:59:53.297 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:459b2826:::benchmark_data_smithi194_196924_object133821:head# 2024-04-29T14:59:53.298 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:459bd592:::benchmark_data_smithi194_196924_object127824:head# 2024-04-29T14:59:53.298 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:459e7b2c:::benchmark_data_smithi194_196924_object149530:head# 2024-04-29T14:59:53.299 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:459ea8d4:::benchmark_data_smithi194_196924_object96829:head# 2024-04-29T14:59:53.299 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:459ed970:::benchmark_data_smithi194_196924_object155388:head# 2024-04-29T14:59:53.300 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45a12a29:::benchmark_data_smithi194_196924_object150629:head# 2024-04-29T14:59:53.300 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45a3c32a:::benchmark_data_smithi194_196924_object110200:head# 2024-04-29T14:59:53.300 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45a45be3:::benchmark_data_smithi194_196924_object101589:head# 2024-04-29T14:59:53.301 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45a5e87c:::benchmark_data_smithi194_196924_object117975:head# 2024-04-29T14:59:53.301 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45a67b6c:::benchmark_data_smithi194_196924_object120561:head# 2024-04-29T14:59:53.302 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45a749ce:::benchmark_data_smithi194_196924_object146901:head# 2024-04-29T14:59:53.302 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45a96be5:::benchmark_data_smithi194_196924_object148544:head# 2024-04-29T14:59:53.302 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45ab0eeb:::benchmark_data_smithi194_196924_object97880:head# 2024-04-29T14:59:53.303 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45aee2c2:::benchmark_data_smithi194_196924_object120527:head# 2024-04-29T14:59:53.303 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45af8f5c:::benchmark_data_smithi194_196924_object114278:head# 2024-04-29T14:59:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45af9f98:::benchmark_data_smithi194_196924_object157539:head# 2024-04-29T14:59:53.304 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b01b33:::benchmark_data_smithi194_196924_object112713:head# 2024-04-29T14:59:53.305 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b245a9:::benchmark_data_smithi194_196924_object137258:head# 2024-04-29T14:59:53.305 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b3f6ab:::benchmark_data_smithi194_196924_object100198:head# 2024-04-29T14:59:53.306 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b51537:::benchmark_data_smithi194_196924_object115177:head# 2024-04-29T14:59:53.306 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b5b47b:::benchmark_data_smithi194_196924_object123675:head# 2024-04-29T14:59:53.307 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b617df:::benchmark_data_smithi194_196924_object158236:head# 2024-04-29T14:59:53.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b6938d:::benchmark_data_smithi194_196924_object151190:head# 2024-04-29T14:59:53.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45b8b937:::benchmark_data_smithi194_196924_object120781:head# 2024-04-29T14:59:53.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45ba3ad7:::benchmark_data_smithi194_196924_object114823:head# 2024-04-29T14:59:53.309 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45bad676:::benchmark_data_smithi194_196924_object134439:head# 2024-04-29T14:59:53.309 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45baffeb:::benchmark_data_smithi194_196924_object147521:head# 2024-04-29T14:59:53.310 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45bc4327:::benchmark_data_smithi194_196924_object141986:head# 2024-04-29T14:59:53.310 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45bd6609:::benchmark_data_smithi194_196924_object123119:head# 2024-04-29T14:59:53.311 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45be3df4:::benchmark_data_smithi194_196924_object102748:head# 2024-04-29T14:59:53.312 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45be4af9:::benchmark_data_smithi194_196924_object135782:head# 2024-04-29T14:59:53.312 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45be7311:::benchmark_data_smithi194_196924_object150931:head# 2024-04-29T14:59:53.312 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45be86bd:::benchmark_data_smithi194_196924_object128629:head# 2024-04-29T14:59:53.313 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45be9652:::benchmark_data_smithi194_196924_object139222:head# 2024-04-29T14:59:53.313 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45be99a5:::benchmark_data_smithi194_196924_object118455:head# 2024-04-29T14:59:53.314 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45bee3e6:::benchmark_data_smithi194_196924_object117301:head# 2024-04-29T14:59:53.314 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45bfc9bd:::benchmark_data_smithi194_196924_object109294:head# 2024-04-29T14:59:53.315 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c28d9c:::benchmark_data_smithi194_196924_object114665:head# 2024-04-29T14:59:53.315 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c3100a:::benchmark_data_smithi194_196924_object151199:head# 2024-04-29T14:59:53.316 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c327df:::benchmark_data_smithi194_196924_object159289:head# 2024-04-29T14:59:53.316 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c351c8:::benchmark_data_smithi194_196924_object135357:head# 2024-04-29T14:59:53.317 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c583ae:::benchmark_data_smithi194_196924_object144725:head# 2024-04-29T14:59:53.317 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c62849:::benchmark_data_smithi194_196924_object109575:head# 2024-04-29T14:59:53.317 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c688fd:::benchmark_data_smithi194_196924_object102147:head# 2024-04-29T14:59:53.318 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c6e1e9:::benchmark_data_smithi194_196924_object138022:head# 2024-04-29T14:59:53.318 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45c8e00f:::benchmark_data_smithi194_196924_object115418:head# 2024-04-29T14:59:53.319 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45ca8d70:::benchmark_data_smithi194_196924_object122742:head# 2024-04-29T14:59:53.319 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cae475:::benchmark_data_smithi194_196924_object121197:head# 2024-04-29T14:59:53.320 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cc1407:::benchmark_data_smithi194_196924_object152899:head# 2024-04-29T14:59:53.321 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cc6671:::benchmark_data_smithi194_196924_object140915:head# 2024-04-29T14:59:53.321 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cca6df:::benchmark_data_smithi194_196924_object108556:head# 2024-04-29T14:59:53.321 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cd7124:::benchmark_data_smithi194_196924_object152617:head# 2024-04-29T14:59:53.321 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cd9a03:::benchmark_data_smithi194_196924_object131578:head# 2024-04-29T14:59:53.322 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cebb22:::benchmark_data_smithi194_196924_object101624:head# 2024-04-29T14:59:53.322 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45cf84ca:::benchmark_data_smithi194_196924_object158130:head# 2024-04-29T14:59:53.323 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d1590d:::benchmark_data_smithi194_196924_object147377:head# 2024-04-29T14:59:53.323 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d194c5:::benchmark_data_smithi194_196924_object140346:head# 2024-04-29T14:59:53.324 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d1b49f:::benchmark_data_smithi194_196924_object154096:head# 2024-04-29T14:59:53.324 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d28421:::benchmark_data_smithi194_196924_object139309:head# 2024-04-29T14:59:53.325 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d37183:::benchmark_data_smithi194_196924_object136512:head# 2024-04-29T14:59:53.325 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d476ec:::benchmark_data_smithi194_196924_object131025:head# 2024-04-29T14:59:53.326 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d4aa34:::benchmark_data_smithi194_196924_object141952:head# 2024-04-29T14:59:53.326 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d4e7b2:::benchmark_data_smithi194_196924_object110545:head# 2024-04-29T14:59:53.327 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d509a1:::benchmark_data_smithi194_196924_object152427:head# 2024-04-29T14:59:53.327 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d509c5:::benchmark_data_smithi194_196924_object147205:head# 2024-04-29T14:59:53.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T14:59:53.328 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d55ca1:::benchmark_data_smithi194_196924_object101194:head# 2024-04-29T14:59:53.328 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d55dec:::benchmark_data_smithi194_196924_object96655:head# 2024-04-29T14:59:53.329 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d632b2:::benchmark_data_smithi194_196924_object151469:head# 2024-04-29T14:59:53.329 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d6dc0f:::benchmark_data_smithi194_196924_object153730:head# 2024-04-29T14:59:53.329 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d6ff1b:::benchmark_data_smithi194_196924_object155329:head# 2024-04-29T14:59:53.329 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d700ef:::benchmark_data_smithi194_196924_object128541:head# 2024-04-29T14:59:53.330 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45d899f3:::benchmark_data_smithi194_196924_object120938:head# 2024-04-29T14:59:53.330 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45da56d7:::benchmark_data_smithi194_196924_object148820:head# 2024-04-29T14:59:53.331 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45de4c10:::benchmark_data_smithi194_196924_object106221:head# 2024-04-29T14:59:53.331 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45dfd5bf:::benchmark_data_smithi194_196924_object155715:head# 2024-04-29T14:59:53.332 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e0799a:::benchmark_data_smithi194_196924_object112103:head# 2024-04-29T14:59:53.332 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e0f73f:::benchmark_data_smithi194_196924_object118469:head# 2024-04-29T14:59:53.332 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e1a4be:::benchmark_data_smithi194_196924_object123229:head# 2024-04-29T14:59:53.333 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e2dec4:::benchmark_data_smithi194_196924_object140448:head# 2024-04-29T14:59:53.333 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e47852:::benchmark_data_smithi194_196924_object129288:head# 2024-04-29T14:59:53.334 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e51421:::benchmark_data_smithi194_196924_object102452:head# 2024-04-29T14:59:53.334 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e5c084:::benchmark_data_smithi194_196924_object102214:head# 2024-04-29T14:59:53.334 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e63e6c:::benchmark_data_smithi194_196924_object101367:head# 2024-04-29T14:59:53.335 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e66ab4:::benchmark_data_smithi194_196924_object158620:head# 2024-04-29T14:59:53.335 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e6a08a:::benchmark_data_smithi194_196924_object136518:head# 2024-04-29T14:59:53.336 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e7bb02:::benchmark_data_smithi194_196924_object97636:head# 2024-04-29T14:59:53.336 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e82a30:::benchmark_data_smithi194_196924_object103328:head# 2024-04-29T14:59:53.337 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e85bf7:::benchmark_data_smithi194_196924_object138512:head# 2024-04-29T14:59:53.337 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e87128:::benchmark_data_smithi194_196924_object159224:head# 2024-04-29T14:59:53.338 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45e88d20:::benchmark_data_smithi194_196924_object116273:head# 2024-04-29T14:59:53.338 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f0a688:::benchmark_data_smithi194_196924_object142573:head# 2024-04-29T14:59:53.339 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f13cfd:::benchmark_data_smithi194_196924_object120869:head# 2024-04-29T14:59:53.339 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f28466:::benchmark_data_smithi194_196924_object139885:head# 2024-04-29T14:59:53.339 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f35ccf:::benchmark_data_smithi194_196924_object129728:head# 2024-04-29T14:59:53.340 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f3e5fe:::benchmark_data_smithi194_196924_object154168:head# 2024-04-29T14:59:53.340 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f46865:::benchmark_data_smithi194_196924_object138094:head# 2024-04-29T14:59:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f4f51b:::benchmark_data_smithi194_196924_object157301:head# 2024-04-29T14:59:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f69577:::benchmark_data_smithi194_196924_object127474:head# 2024-04-29T14:59:53.342 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f72158:::benchmark_data_smithi194_196924_object156977:head# 2024-04-29T14:59:53.342 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45f7a9c7:::benchmark_data_smithi194_196924_object142709:head# 2024-04-29T14:59:53.343 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fa9348:::benchmark_data_smithi194_196924_object120778:head# 2024-04-29T14:59:53.344 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45faa9c7:::benchmark_data_smithi194_196924_object131857:head# 2024-04-29T14:59:53.344 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fc3dff:::benchmark_data_smithi194_196924_object104264:head# 2024-04-29T14:59:53.345 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fc6d68:::benchmark_data_smithi194_196924_object98249:head# 2024-04-29T14:59:53.345 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fcba32:::benchmark_data_smithi194_196924_object108132:head# 2024-04-29T14:59:53.346 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fce540:::benchmark_data_smithi194_196924_object139238:head# 2024-04-29T14:59:53.346 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fcf271:::benchmark_data_smithi194_196924_object120632:head# 2024-04-29T14:59:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fd7489:::benchmark_data_smithi194_196924_object158095:head# 2024-04-29T14:59:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:45fdd0f7:::benchmark_data_smithi194_196924_object116440:head# 2024-04-29T14:59:53.348 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46005925:::benchmark_data_smithi194_196924_object133922:head# 2024-04-29T14:59:53.348 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4600a1b4:::benchmark_data_smithi194_196924_object106733:head# 2024-04-29T14:59:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4600dbe1:::benchmark_data_smithi194_196924_object115701:head# 2024-04-29T14:59:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46018e3a:::benchmark_data_smithi194_196924_object106517:head# 2024-04-29T14:59:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4603e513:::benchmark_data_smithi194_196924_object126565:head# 2024-04-29T14:59:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46052b03:::benchmark_data_smithi194_196924_object140486:head# 2024-04-29T14:59:53.351 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4606ba8a:::benchmark_data_smithi194_196924_object147386:head# 2024-04-29T14:59:53.351 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4606d5b7:::benchmark_data_smithi194_196924_object103394:head# 2024-04-29T14:59:53.352 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46072bca:::benchmark_data_smithi194_196924_object138199:head# 2024-04-29T14:59:53.352 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46074bcc:::benchmark_data_smithi194_196924_object142521:head# 2024-04-29T14:59:53.352 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46076a94:::benchmark_data_smithi194_196924_object109163:head# 2024-04-29T14:59:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4607840c:::benchmark_data_smithi194_196924_object123818:head# 2024-04-29T14:59:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4607a94f:::benchmark_data_smithi194_196924_object158010:head# 2024-04-29T14:59:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460806a7:::benchmark_data_smithi194_196924_object138378:head# 2024-04-29T14:59:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46083b33:::benchmark_data_smithi194_196924_object150838:head# 2024-04-29T14:59:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46090aeb:::benchmark_data_smithi194_196924_object114705:head# 2024-04-29T14:59:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46095bee:::benchmark_data_smithi194_196924_object107309:head# 2024-04-29T14:59:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46096bb5:::benchmark_data_smithi194_196924_object158288:head# 2024-04-29T14:59:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460973b1:::benchmark_data_smithi194_196924_object140079:head# 2024-04-29T14:59:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4609e98f:::benchmark_data_smithi194_196924_object125440:head# 2024-04-29T14:59:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460a7aa9:::benchmark_data_smithi194_196924_object110055:head# 2024-04-29T14:59:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460a91ca:::benchmark_data_smithi194_196924_object149593:head# 2024-04-29T14:59:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460b1131:::benchmark_data_smithi194_196924_object99011:head# 2024-04-29T14:59:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460b2a73:::benchmark_data_smithi194_196924_object137211:head# 2024-04-29T14:59:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460b3033:::benchmark_data_smithi194_196924_object119278:head# 2024-04-29T14:59:53.360 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460b336b:::benchmark_data_smithi194_196924_object136004:head# 2024-04-29T14:59:53.360 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460b5b05:::benchmark_data_smithi194_196924_object150620:head# 2024-04-29T14:59:53.361 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460bae3a:::benchmark_data_smithi194_196924_object97994:head# 2024-04-29T14:59:53.361 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460be435:::benchmark_data_smithi194_196924_object123482:head# 2024-04-29T14:59:53.361 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460c1efb:::benchmark_data_smithi194_196924_object104723:head# 2024-04-29T14:59:53.362 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460c8bc9:::benchmark_data_smithi194_196924_object155127:head# 2024-04-29T14:59:53.362 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460dd167:::benchmark_data_smithi194_196924_object122768:head# 2024-04-29T14:59:53.363 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460e566a:::benchmark_data_smithi194_196924_object129417:head# 2024-04-29T14:59:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460fb6af:::benchmark_data_smithi194_196924_object131140:head# 2024-04-29T14:59:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:460fc496:::benchmark_data_smithi194_196924_object158684:head# 2024-04-29T14:59:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4610c221:::benchmark_data_smithi194_196924_object136311:head# 2024-04-29T14:59:53.365 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4610d092:::benchmark_data_smithi194_196924_object120142:head# 2024-04-29T14:59:53.365 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:461118ee:::benchmark_data_smithi194_196924_object144980:head# 2024-04-29T14:59:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:461131f9:::benchmark_data_smithi194_196924_object159313:head# 2024-04-29T14:59:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4611c2ea:::benchmark_data_smithi194_196924_object112403:head# 2024-04-29T14:59:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46126934:::benchmark_data_smithi194_196924_object123957:head# 2024-04-29T14:59:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46134089:::benchmark_data_smithi194_196924_object122411:head# 2024-04-29T14:59:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4613a78e:::benchmark_data_smithi194_196924_object146763:head# 2024-04-29T14:59:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46145db6:::benchmark_data_smithi194_196924_object122427:head# 2024-04-29T14:59:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46148242:::benchmark_data_smithi194_196924_object123443:head# 2024-04-29T14:59:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4616a254:::benchmark_data_smithi194_196924_object159662:head# 2024-04-29T14:59:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4617571c:::benchmark_data_smithi194_196924_object146366:head# 2024-04-29T14:59:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4617cc1a:::benchmark_data_smithi194_196924_object130794:head# 2024-04-29T14:59:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:461873e6:::benchmark_data_smithi194_196924_object100008:head# 2024-04-29T14:59:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4618f477:::benchmark_data_smithi194_196924_object135079:head# 2024-04-29T14:59:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:461a075a:::benchmark_data_smithi194_196924_object122459:head# 2024-04-29T14:59:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:461af4bf:::benchmark_data_smithi194_196924_object109949:head# 2024-04-29T14:59:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:461c6ec0:::benchmark_data_smithi194_196924_object148135:head# 2024-04-29T14:59:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:461f4be8:::benchmark_data_smithi194_196924_object125744:head# 2024-04-29T14:59:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46206a54:::benchmark_data_smithi194_196924_object114020:head# 2024-04-29T14:59:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4620f33f:::benchmark_data_smithi194_196924_object153679:head# 2024-04-29T14:59:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46216290:::benchmark_data_smithi194_196924_object142147:head# 2024-04-29T14:59:53.374 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:462182bd:::benchmark_data_smithi194_196924_object158189:head# 2024-04-29T14:59:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46229625:::benchmark_data_smithi194_196924_object159159:head# 2024-04-29T14:59:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4622fd5f:::benchmark_data_smithi194_196924_object112835:head# 2024-04-29T14:59:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4623dc3e:::benchmark_data_smithi194_196924_object114240:head# 2024-04-29T14:59:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46247491:::benchmark_data_smithi194_196924_object142017:head# 2024-04-29T14:59:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46264c5e:::benchmark_data_smithi194_196924_object123164:head# 2024-04-29T14:59:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4626a6f6:::benchmark_data_smithi194_196924_object117011:head# 2024-04-29T14:59:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46273c3f:::benchmark_data_smithi194_196924_object137119:head# 2024-04-29T14:59:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4628d197:::benchmark_data_smithi194_196924_object102801:head# 2024-04-29T14:59:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:462a166c:::benchmark_data_smithi194_196924_object121631:head# 2024-04-29T14:59:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:462d8dc8:::benchmark_data_smithi194_196924_object134092:head# 2024-04-29T14:59:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:462ddde7:::benchmark_data_smithi194_196924_object101648:head# 2024-04-29T14:59:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:462dff1b:::benchmark_data_smithi194_196924_object141072:head# 2024-04-29T14:59:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46307e2f:::benchmark_data_smithi194_196924_object142999:head# 2024-04-29T14:59:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463156d4:::benchmark_data_smithi194_196924_object142721:head# 2024-04-29T14:59:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463412d0:::benchmark_data_smithi194_196924_object123103:head# 2024-04-29T14:59:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463439a8:::benchmark_data_smithi194_196924_object116683:head# 2024-04-29T14:59:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46357b27:::benchmark_data_smithi194_196924_object150981:head# 2024-04-29T14:59:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4637eb4b:::benchmark_data_smithi194_196924_object140176:head# 2024-04-29T14:59:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46383b23:::benchmark_data_smithi194_196924_object125022:head# 2024-04-29T14:59:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4638d7b5:::benchmark_data_smithi194_196924_object150052:head# 2024-04-29T14:59:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4638fcbc:::benchmark_data_smithi194_196924_object106180:head# 2024-04-29T14:59:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4639156e:::benchmark_data_smithi194_196924_object102644:head# 2024-04-29T14:59:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463953c5:::benchmark_data_smithi194_196924_object151092:head# 2024-04-29T14:59:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463979b3:::benchmark_data_smithi194_196924_object113669:head# 2024-04-29T14:59:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463c47ef:::benchmark_data_smithi194_196924_object120955:head# 2024-04-29T14:59:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463cea5c:::benchmark_data_smithi194_196924_object119405:head# 2024-04-29T14:59:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463d0a93:::benchmark_data_smithi194_196924_object123198:head# 2024-04-29T14:59:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463d5592:::benchmark_data_smithi194_196924_object133468:head# 2024-04-29T14:59:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:463e3cf5:::benchmark_data_smithi194_196924_object131510:head# 2024-04-29T14:59:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464140e9:::benchmark_data_smithi194_196924_object137382:head# 2024-04-29T14:59:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464243eb:::benchmark_data_smithi194_196924_object151854:head# 2024-04-29T14:59:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46441bde:::benchmark_data_smithi194_196924_object125830:head# 2024-04-29T14:59:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4644f2d0:::benchmark_data_smithi194_196924_object101812:head# 2024-04-29T14:59:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46457764:::benchmark_data_smithi194_196924_object128816:head# 2024-04-29T14:59:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464588cc:::benchmark_data_smithi194_196924_object146145:head# 2024-04-29T14:59:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4645a8e2:::benchmark_data_smithi194_196924_object133627:head# 2024-04-29T14:59:53.390 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46462e9b:::benchmark_data_smithi194_196924_object114227:head# 2024-04-29T14:59:53.390 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46480d22:::benchmark_data_smithi194_196924_object123426:head# 2024-04-29T14:59:53.391 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464b1467:::benchmark_data_smithi194_196924_object152994:head# 2024-04-29T14:59:53.391 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464b2a2d:::benchmark_data_smithi194_196924_object157240:head# 2024-04-29T14:59:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464b4625:::benchmark_data_smithi194_196924_object107789:head# 2024-04-29T14:59:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464b5513:::benchmark_data_smithi194_196924_object142431:head# 2024-04-29T14:59:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464d1d74:::benchmark_data_smithi194_196924_object138548:head# 2024-04-29T14:59:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464e9a4d:::benchmark_data_smithi194_196924_object116646:head# 2024-04-29T14:59:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464ef374:::benchmark_data_smithi194_196924_object159190:head# 2024-04-29T14:59:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:464f6208:::benchmark_data_smithi194_196924_object113408:head# 2024-04-29T14:59:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465121a5:::benchmark_data_smithi194_196924_object137867:head# 2024-04-29T14:59:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46517ba1:::benchmark_data_smithi194_196924_object149711:head# 2024-04-29T14:59:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4651b16e:::benchmark_data_smithi194_196924_object117807:head# 2024-04-29T14:59:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4652a38c:::benchmark_data_smithi194_196924_object154612:head# 2024-04-29T14:59:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46534065:::benchmark_data_smithi194_196924_object101510:head# 2024-04-29T14:59:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4654b318:::benchmark_data_smithi194_196924_object109484:head# 2024-04-29T14:59:53.397 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46558f0a:::benchmark_data_smithi194_196924_object124183:head# 2024-04-29T14:59:53.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T14:59:53.398 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4657dff9:::benchmark_data_smithi194_196924_object153714:head# 2024-04-29T14:59:53.399 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4659d3d6:::benchmark_data_smithi194_196924_object121092:head# 2024-04-29T14:59:53.399 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4659f96c:::benchmark_data_smithi194_196924_object131744:head# 2024-04-29T14:59:53.399 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465a74ad:::benchmark_data_smithi194_196924_object140930:head# 2024-04-29T14:59:53.399 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465af1d0:::benchmark_data_smithi194_196924_object139892:head# 2024-04-29T14:59:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465c4dd7:::benchmark_data_smithi194_196924_object142280:head# 2024-04-29T14:59:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465c6760:::benchmark_data_smithi194_196924_object135716:head# 2024-04-29T14:59:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465ce243:::benchmark_data_smithi194_196924_object138723:head# 2024-04-29T14:59:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465cee67:::benchmark_data_smithi194_196924_object151195:head# 2024-04-29T14:59:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465dd7d7:::benchmark_data_smithi194_196924_object158940:head# 2024-04-29T14:59:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:465f456a:::benchmark_data_smithi194_196924_object96615:head# 2024-04-29T14:59:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:53.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T14:59:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4660b7c9:::benchmark_data_smithi194_196924_object157989:head# 2024-04-29T14:59:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46623fb1:::benchmark_data_smithi194_196924_object101087:head# 2024-04-29T14:59:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4663d8c3:::benchmark_data_smithi194_196924_object143750:head# 2024-04-29T14:59:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46654cac:::benchmark_data_smithi194_196924_object130091:head# 2024-04-29T14:59:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4665eecb:::benchmark_data_smithi194_196924_object101010:head# 2024-04-29T14:59:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46664b84:::benchmark_data_smithi194_196924_object104265:head# 2024-04-29T14:59:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46678086:::benchmark_data_smithi194_196924_object109621:head# 2024-04-29T14:59:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4668216a:::benchmark_data_smithi194_196924_object102546:head# 2024-04-29T14:59:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46683397:::benchmark_data_smithi194_196924_object122952:head# 2024-04-29T14:59:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46691db4:::benchmark_data_smithi194_196924_object107735:head# 2024-04-29T14:59:53.407 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4669a24e:::benchmark_data_smithi194_196924_object108251:head# 2024-04-29T14:59:53.407 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4669f0a1:::benchmark_data_smithi194_196924_object130182:head# 2024-04-29T14:59:53.408 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:466b7e90:::benchmark_data_smithi194_196924_object95421:head# 2024-04-29T14:59:53.408 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:466bce5e:::benchmark_data_smithi194_196924_object127897:head# 2024-04-29T14:59:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:466c9caf:::benchmark_data_smithi194_196924_object143656:head# 2024-04-29T14:59:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:466d9ca0:::benchmark_data_smithi194_196924_object155939:head# 2024-04-29T14:59:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467083e3:::benchmark_data_smithi194_196924_object153055:head# 2024-04-29T14:59:53.410 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467175cd:::benchmark_data_smithi194_196924_object138017:head# 2024-04-29T14:59:53.410 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4671f571:::benchmark_data_smithi194_196924_object127017:head# 2024-04-29T14:59:53.411 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46725e8b:::benchmark_data_smithi194_196924_object120191:head# 2024-04-29T14:59:53.411 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46728e00:::benchmark_data_smithi194_196924_object145516:head# 2024-04-29T14:59:53.411 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4672f849:::benchmark_data_smithi194_196924_object106020:head# 2024-04-29T14:59:53.412 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4673d2aa:::benchmark_data_smithi194_196924_object110187:head# 2024-04-29T14:59:53.412 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46745d09:::benchmark_data_smithi194_196924_object100006:head# 2024-04-29T14:59:53.413 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4674fdcd:::benchmark_data_smithi194_196924_object141230:head# 2024-04-29T14:59:53.413 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46756504:::benchmark_data_smithi194_196924_object152572:head# 2024-04-29T14:59:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46756c3f:::benchmark_data_smithi194_196924_object139714:head# 2024-04-29T14:59:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46776c7e:::benchmark_data_smithi194_196924_object153848:head# 2024-04-29T14:59:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46787026:::benchmark_data_smithi194_196924_object149173:head# 2024-04-29T14:59:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467a1d9a:::benchmark_data_smithi194_196924_object106906:head# 2024-04-29T14:59:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467b2565:::benchmark_data_smithi194_196924_object142901:head# 2024-04-29T14:59:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467b884a:::benchmark_data_smithi194_196924_object104012:head# 2024-04-29T14:59:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467ba1e4:::benchmark_data_smithi194_196924_object141639:head# 2024-04-29T14:59:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467be846:::benchmark_data_smithi194_196924_object107295:head# 2024-04-29T14:59:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467d44b7:::benchmark_data_smithi194_196924_object129577:head# 2024-04-29T14:59:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467dc146:::benchmark_data_smithi194_196924_object159707:head# 2024-04-29T14:59:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:467e69a9:::benchmark_data_smithi194_196924_object126742:head# 2024-04-29T14:59:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46815941:::benchmark_data_smithi194_196924_object101509:head# 2024-04-29T14:59:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4681925f:::benchmark_data_smithi194_196924_object147516:head# 2024-04-29T14:59:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4683c7d1:::benchmark_data_smithi194_196924_object142868:head# 2024-04-29T14:59:53.423 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468516d5:::benchmark_data_smithi194_196924_object123506:head# 2024-04-29T14:59:53.423 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4685da09:::benchmark_data_smithi194_196924_object139598:head# 2024-04-29T14:59:53.423 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468606de:::benchmark_data_smithi194_196924_object116255:head# 2024-04-29T14:59:53.424 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4687e058:::benchmark_data_smithi194_196924_object133614:head# 2024-04-29T14:59:53.424 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4688b2bb:::benchmark_data_smithi194_196924_object133316:head# 2024-04-29T14:59:53.425 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46892a46:::benchmark_data_smithi194_196924_object149442:head# 2024-04-29T14:59:53.425 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468942c2:::benchmark_data_smithi194_196924_object114257:head# 2024-04-29T14:59:53.426 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4689a805:::benchmark_data_smithi194_196924_object142082:head# 2024-04-29T14:59:53.426 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468a17d2:::benchmark_data_smithi194_196924_object117164:head# 2024-04-29T14:59:53.427 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468ab0a7:::benchmark_data_smithi194_196924_object159262:head# 2024-04-29T14:59:53.427 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468b6b8b:::benchmark_data_smithi194_196924_object138947:head# 2024-04-29T14:59:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468d12ad:::benchmark_data_smithi194_196924_object141165:head# 2024-04-29T14:59:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468e1cf5:::benchmark_data_smithi194_196924_object128494:head# 2024-04-29T14:59:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468e88e4:::benchmark_data_smithi194_196924_object135252:head# 2024-04-29T14:59:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:468f82ef:::benchmark_data_smithi194_196924_object118371:head# 2024-04-29T14:59:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46900fc0:::benchmark_data_smithi194_196924_object116876:head# 2024-04-29T14:59:53.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469114db:::benchmark_data_smithi194_196924_object155110:head# 2024-04-29T14:59:53.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4691fb58:::benchmark_data_smithi194_196924_object115026:head# 2024-04-29T14:59:53.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4692ceed:::benchmark_data_smithi194_196924_object143013:head# 2024-04-29T14:59:53.431 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4692e948:::benchmark_data_smithi194_196924_object105140:head# 2024-04-29T14:59:53.432 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46963fd4:::benchmark_data_smithi194_196924_object110121:head# 2024-04-29T14:59:53.432 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46973475:::benchmark_data_smithi194_196924_object110361:head# 2024-04-29T14:59:53.432 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46987a88:::benchmark_data_smithi194_196924_object97808:head# 2024-04-29T14:59:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46999489:::benchmark_data_smithi194_196924_object96140:head# 2024-04-29T14:59:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469aae19:::benchmark_data_smithi194_196924_object122528:head# 2024-04-29T14:59:53.434 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469bcc2a:::benchmark_data_smithi194_196924_object106699:head# 2024-04-29T14:59:53.434 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469bfd54:::benchmark_data_smithi194_196924_object159191:head# 2024-04-29T14:59:53.434 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469c8aab:::benchmark_data_smithi194_196924_object118162:head# 2024-04-29T14:59:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469e04df:::benchmark_data_smithi194_196924_object133936:head# 2024-04-29T14:59:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469e42bb:::benchmark_data_smithi194_196924_object118814:head# 2024-04-29T14:59:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469e4df2:::benchmark_data_smithi194_196924_object149193:head# 2024-04-29T14:59:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:469f388e:::benchmark_data_smithi194_196924_object119756:head# 2024-04-29T14:59:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a031f3:::benchmark_data_smithi194_196924_object154784:head# 2024-04-29T14:59:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a04b1f:::benchmark_data_smithi194_196924_object133901:head# 2024-04-29T14:59:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a0bb6f:::benchmark_data_smithi194_196924_object132542:head# 2024-04-29T14:59:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a0fa19:::benchmark_data_smithi194_196924_object107115:head# 2024-04-29T14:59:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a29dd7:::benchmark_data_smithi194_196924_object110697:head# 2024-04-29T14:59:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a32976:::benchmark_data_smithi194_196924_object155711:head# 2024-04-29T14:59:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a4efd5:::benchmark_data_smithi194_196924_object121435:head# 2024-04-29T14:59:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a715a8:::benchmark_data_smithi194_196924_object120410:head# 2024-04-29T14:59:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a83cc2:::benchmark_data_smithi194_196924_object128008:head# 2024-04-29T14:59:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a8726c:::benchmark_data_smithi194_196924_object125796:head# 2024-04-29T14:59:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a8d31e:::benchmark_data_smithi194_196924_object133431:head# 2024-04-29T14:59:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a96cf6:::benchmark_data_smithi194_196924_object154651:head# 2024-04-29T14:59:53.442 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46a9fd33:::benchmark_data_smithi194_196924_object113443:head# 2024-04-29T14:59:53.442 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ae1ec1:::benchmark_data_smithi194_196924_object101008:head# 2024-04-29T14:59:53.443 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46b20473:::benchmark_data_smithi194_196924_object128756:head# 2024-04-29T14:59:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46b25fac:::benchmark_data_smithi194_196924_object111403:head# 2024-04-29T14:59:53.446 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46b266c6:::benchmark_data_smithi194_196924_object148173:head# 2024-04-29T14:59:53.446 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46b40f86:::benchmark_data_smithi194_196924_object125628:head# 2024-04-29T14:59:53.446 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46b5c6f1:::benchmark_data_smithi194_196924_object121744:head# 2024-04-29T14:59:53.447 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46b5e7be:::benchmark_data_smithi194_196924_object117665:head# 2024-04-29T14:59:53.447 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46b86682:::benchmark_data_smithi194_196924_object130799:head# 2024-04-29T14:59:53.448 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ba1a9a:::benchmark_data_smithi194_196924_object115546:head# 2024-04-29T14:59:53.448 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ba27d9:::benchmark_data_smithi194_196924_object149049:head# 2024-04-29T14:59:53.448 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46bb5ea4:::benchmark_data_smithi194_196924_object97523:head# 2024-04-29T14:59:53.449 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46bc28a6:::benchmark_data_smithi194_196924_object108888:head# 2024-04-29T14:59:53.449 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46be338a:::benchmark_data_smithi194_196924_object106712:head# 2024-04-29T14:59:53.450 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46be686e:::benchmark_data_smithi194_196924_object133376:head# 2024-04-29T14:59:53.450 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46bf9d9c:::benchmark_data_smithi194_196924_object108121:head# 2024-04-29T14:59:53.450 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46bfdbf0:::benchmark_data_smithi194_196924_object133548:head# 2024-04-29T14:59:53.451 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c0022e:::benchmark_data_smithi194_196924_object112486:head# 2024-04-29T14:59:53.451 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c0b0b8:::benchmark_data_smithi194_196924_object122488:head# 2024-04-29T14:59:53.452 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c2af14:::benchmark_data_smithi194_196924_object124252:head# 2024-04-29T14:59:53.452 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c33195:::benchmark_data_smithi194_196924_object124197:head# 2024-04-29T14:59:53.453 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c349dd:::benchmark_data_smithi194_196924_object119925:head# 2024-04-29T14:59:53.453 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c49e43:::benchmark_data_smithi194_196924_object142805:head# 2024-04-29T14:59:53.453 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c7499a:::benchmark_data_smithi194_196924_object96712:head# 2024-04-29T14:59:53.454 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c7608e:::benchmark_data_smithi194_196924_object130598:head# 2024-04-29T14:59:53.454 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c81325:::benchmark_data_smithi194_196924_object105657:head# 2024-04-29T14:59:53.454 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c8de62:::benchmark_data_smithi194_196924_object143839:head# 2024-04-29T14:59:53.455 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46c9fd81:::benchmark_data_smithi194_196924_object123104:head# 2024-04-29T14:59:53.456 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ca0a4a:::benchmark_data_smithi194_196924_object104936:head# 2024-04-29T14:59:53.456 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46caab66:::benchmark_data_smithi194_196924_object112605:head# 2024-04-29T14:59:53.457 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46cb335a:::benchmark_data_smithi194_196924_object129881:head# 2024-04-29T14:59:53.457 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46cb7bc1:::benchmark_data_smithi194_196924_object118554:head# 2024-04-29T14:59:53.457 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46cbf22c:::benchmark_data_smithi194_196924_object140998:head# 2024-04-29T14:59:53.458 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46cc3dcb:::benchmark_data_smithi194_196924_object145696:head# 2024-04-29T14:59:53.458 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46cd2e04:::benchmark_data_smithi194_196924_object129216:head# 2024-04-29T14:59:53.459 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46cd6338:::benchmark_data_smithi194_196924_object154533:head# 2024-04-29T14:59:53.459 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ce9b6d:::benchmark_data_smithi194_196924_object131635:head# 2024-04-29T14:59:53.460 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d0e33a:::benchmark_data_smithi194_196924_object129452:head# 2024-04-29T14:59:53.460 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d1efa1:::benchmark_data_smithi194_196924_object153562:head# 2024-04-29T14:59:53.461 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d2e4a6:::benchmark_data_smithi194_196924_object108049:head# 2024-04-29T14:59:53.461 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d3b1db:::benchmark_data_smithi194_196924_object114523:head# 2024-04-29T14:59:53.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d3f82e:::benchmark_data_smithi194_196924_object101897:head# 2024-04-29T14:59:53.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d4791c:::benchmark_data_smithi194_196924_object112582:head# 2024-04-29T14:59:53.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d4c8e4:::benchmark_data_smithi194_196924_object146960:head# 2024-04-29T14:59:53.463 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d60e99:::benchmark_data_smithi194_196924_object150833:head# 2024-04-29T14:59:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d76206:::benchmark_data_smithi194_196924_object132059:head# 2024-04-29T14:59:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d78e4c:::benchmark_data_smithi194_196924_object145754:head# 2024-04-29T14:59:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d85571:::benchmark_data_smithi194_196924_object120743:head# 2024-04-29T14:59:53.465 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46d8c6fe:::benchmark_data_smithi194_196924_object106350:head# 2024-04-29T14:59:53.466 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46db4525:::benchmark_data_smithi194_196924_object158566:head# 2024-04-29T14:59:53.466 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46db610c:::benchmark_data_smithi194_196924_object116154:head# 2024-04-29T14:59:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46dbb228:::benchmark_data_smithi194_196924_object146398:head# 2024-04-29T14:59:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46dffd46:::benchmark_data_smithi194_196924_object105371:head# 2024-04-29T14:59:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e0e3ee:::benchmark_data_smithi194_196924_object118391:head# 2024-04-29T14:59:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e18665:::benchmark_data_smithi194_196924_object110605:head# 2024-04-29T14:59:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e1e5b3:::benchmark_data_smithi194_196924_object148299:head# 2024-04-29T14:59:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e38b2a:::benchmark_data_smithi194_196924_object98709:head# 2024-04-29T14:59:53.470 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e3ca82:::benchmark_data_smithi194_196924_object144446:head# 2024-04-29T14:59:53.470 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e3df9f:::benchmark_data_smithi194_196924_object104963:head# 2024-04-29T14:59:53.470 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e4d0de:::benchmark_data_smithi194_196924_object138095:head# 2024-04-29T14:59:53.471 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e5bec9:::benchmark_data_smithi194_196924_object132693:head# 2024-04-29T14:59:53.471 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e62b86:::benchmark_data_smithi194_196924_object129454:head# 2024-04-29T14:59:53.472 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e80180:::benchmark_data_smithi194_196924_object114668:head# 2024-04-29T14:59:53.472 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46e92c4f:::benchmark_data_smithi194_196924_object137301:head# 2024-04-29T14:59:53.473 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ea054e:::benchmark_data_smithi194_196924_object158445:head# 2024-04-29T14:59:53.473 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ea46fd:::benchmark_data_smithi194_196924_object97228:head# 2024-04-29T14:59:53.473 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46eb6ef0:::benchmark_data_smithi194_196924_object109159:head# 2024-04-29T14:59:53.474 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ed615d:::benchmark_data_smithi194_196924_object148911:head# 2024-04-29T14:59:53.474 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ee0387:::benchmark_data_smithi194_196924_object127523:head# 2024-04-29T14:59:53.475 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ee543c:::benchmark_data_smithi194_196924_object137091:head# 2024-04-29T14:59:53.476 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ee889a:::benchmark_data_smithi194_196924_object144829:head# 2024-04-29T14:59:53.477 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ef31ff:::benchmark_data_smithi194_196924_object150323:head# 2024-04-29T14:59:53.477 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ef47f0:::benchmark_data_smithi194_196924_object158107:head# 2024-04-29T14:59:53.478 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ef5c69:::benchmark_data_smithi194_196924_object159903:head# 2024-04-29T14:59:53.478 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f03487:::benchmark_data_smithi194_196924_object129535:head# 2024-04-29T14:59:53.478 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f0d771:::benchmark_data_smithi194_196924_object119419:head# 2024-04-29T14:59:53.479 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f1415f:::benchmark_data_smithi194_196924_object108180:head# 2024-04-29T14:59:53.479 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f1da6c:::benchmark_data_smithi194_196924_object124712:head# 2024-04-29T14:59:53.480 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f26b4a:::benchmark_data_smithi194_196924_object136296:head# 2024-04-29T14:59:53.480 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f29778:::benchmark_data_smithi194_196924_object155536:head# 2024-04-29T14:59:53.480 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f2fd25:::benchmark_data_smithi194_196924_object98097:head# 2024-04-29T14:59:53.481 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f48e57:::benchmark_data_smithi194_196924_object95777:head# 2024-04-29T14:59:53.481 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f7836e:::benchmark_data_smithi194_196924_object116372:head# 2024-04-29T14:59:53.482 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f83c3c:::benchmark_data_smithi194_196924_object144872:head# 2024-04-29T14:59:53.482 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f989e8:::benchmark_data_smithi194_196924_object125415:head# 2024-04-29T14:59:53.483 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f9a5f7:::benchmark_data_smithi194_196924_object138795:head# 2024-04-29T14:59:53.484 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46f9c6f9:::benchmark_data_smithi194_196924_object116730:head# 2024-04-29T14:59:53.484 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46fab54a:::benchmark_data_smithi194_196924_object158969:head# 2024-04-29T14:59:53.485 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46fba497:::benchmark_data_smithi194_196924_object153575:head# 2024-04-29T14:59:53.485 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46fbabf3:::benchmark_data_smithi194_196924_object157711:head# 2024-04-29T14:59:53.485 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46fc6638:::benchmark_data_smithi194_196924_object110158:head# 2024-04-29T14:59:53.486 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46fcb0da:::benchmark_data_smithi194_196924_object159353:head# 2024-04-29T14:59:53.486 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46fd0630:::benchmark_data_smithi194_196924_object110244:head# 2024-04-29T14:59:53.487 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:46ff6e79:::benchmark_data_smithi194_196924_object145819:head# 2024-04-29T14:59:53.487 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4700ff77:::benchmark_data_smithi194_196924_object119882:head# 2024-04-29T14:59:53.488 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47018fe7:::benchmark_data_smithi194_196924_object118860:head# 2024-04-29T14:59:53.488 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47021004:::benchmark_data_smithi194_196924_object123991:head# 2024-04-29T14:59:53.488 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4702ebe6:::benchmark_data_smithi194_196924_object158688:head# 2024-04-29T14:59:53.489 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4702ec41:::benchmark_data_smithi194_196924_object118386:head# 2024-04-29T14:59:53.489 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47037174:::benchmark_data_smithi194_196924_object122049:head# 2024-04-29T14:59:53.489 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47072a74:::benchmark_data_smithi194_196924_object146745:head# 2024-04-29T14:59:53.490 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4707c90d:::benchmark_data_smithi194_196924_object142870:head# 2024-04-29T14:59:53.490 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4708df02:::benchmark_data_smithi194_196924_object158168:head# 2024-04-29T14:59:53.491 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:470902bc:::benchmark_data_smithi194_196924_object110800:head# 2024-04-29T14:59:53.491 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:470acb05:::benchmark_data_smithi194_196924_object128361:head# 2024-04-29T14:59:53.492 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:470adaf9:::benchmark_data_smithi194_196924_object118643:head# 2024-04-29T14:59:53.492 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:470c010f:::benchmark_data_smithi194_196924_object119520:head# 2024-04-29T14:59:53.492 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:470d9939:::benchmark_data_smithi194_196924_object155672:head# 2024-04-29T14:59:53.493 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47110c42:::benchmark_data_smithi194_196924_object117222:head# 2024-04-29T14:59:53.493 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47121fbe:::benchmark_data_smithi194_196924_object141029:head# 2024-04-29T14:59:53.494 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:471275f6:::benchmark_data_smithi194_196924_object124771:head# 2024-04-29T14:59:53.494 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4712f8bb:::benchmark_data_smithi194_196924_object151308:head# 2024-04-29T14:59:53.494 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4713135d:::benchmark_data_smithi194_196924_object103717:head# 2024-04-29T14:59:53.495 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:471340ef:::benchmark_data_smithi194_196924_object145449:head# 2024-04-29T14:59:53.496 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47135466:::benchmark_data_smithi194_196924_object148587:head# 2024-04-29T14:59:53.496 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47157e4e:::benchmark_data_smithi194_196924_object95676:head# 2024-04-29T14:59:53.496 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4715e99b:::benchmark_data_smithi194_196924_object130543:head# 2024-04-29T14:59:53.497 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4716410d:::benchmark_data_smithi194_196924_object156523:head# 2024-04-29T14:59:53.497 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:471680af:::benchmark_data_smithi194_196924_object108275:head# 2024-04-29T14:59:53.498 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47176bc5:::benchmark_data_smithi194_196924_object135707:head# 2024-04-29T14:59:53.498 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4718ec43:::benchmark_data_smithi194_196924_object147255:head# 2024-04-29T14:59:53.499 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:471905da:::benchmark_data_smithi194_196924_object105237:head# 2024-04-29T14:59:53.499 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:471d2346:::benchmark_data_smithi194_196924_object98635:head# 2024-04-29T14:59:53.500 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:471fd331:::benchmark_data_smithi194_196924_object140751:head# 2024-04-29T14:59:53.500 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4720fe37:::benchmark_data_smithi194_196924_object124543:head# 2024-04-29T14:59:53.500 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472244fb:::benchmark_data_smithi194_196924_object125363:head# 2024-04-29T14:59:53.501 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472301c9:::benchmark_data_smithi194_196924_object132008:head# 2024-04-29T14:59:53.502 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472384ff:::benchmark_data_smithi194_196924_object135483:head# 2024-04-29T14:59:53.502 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4724259e:::benchmark_data_smithi194_196924_object152646:head# 2024-04-29T14:59:53.503 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472494d5:::benchmark_data_smithi194_196924_object154899:head# 2024-04-29T14:59:53.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:53.503 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47254cc4:::benchmark_data_smithi194_196924_object149806:head# 2024-04-29T14:59:53.504 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47269e9d:::benchmark_data_smithi194_196924_object123728:head# 2024-04-29T14:59:53.506 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4726c622:::benchmark_data_smithi194_196924_object98276:head# 2024-04-29T14:59:53.506 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47272d34:::benchmark_data_smithi194_196924_object109228:head# 2024-04-29T14:59:53.506 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4727c57f:::benchmark_data_smithi194_196924_object153738:head# 2024-04-29T14:59:53.507 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47289bb2:::benchmark_data_smithi194_196924_object150043:head# 2024-04-29T14:59:53.507 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4729b029:::benchmark_data_smithi194_196924_object135381:head# 2024-04-29T14:59:53.507 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472b9811:::benchmark_data_smithi194_196924_object134847:head# 2024-04-29T14:59:53.507 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472c0494:::benchmark_data_smithi194_196924_object102216:head# 2024-04-29T14:59:53.507 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472c43f2:::benchmark_data_smithi194_196924_object124463:head# 2024-04-29T14:59:53.508 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472c7ab6:::benchmark_data_smithi194_196924_object159126:head# 2024-04-29T14:59:53.508 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472d4433:::benchmark_data_smithi194_196924_object135710:head# 2024-04-29T14:59:53.509 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472ddd31:::benchmark_data_smithi194_196924_object147539:head# 2024-04-29T14:59:53.509 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:472fc010:::benchmark_data_smithi194_196924_object97939:head# 2024-04-29T14:59:53.509 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4730e776:::benchmark_data_smithi194_196924_object111029:head# 2024-04-29T14:59:53.510 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473348b4:::benchmark_data_smithi194_196924_object137349:head# 2024-04-29T14:59:53.510 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473421cc:::benchmark_data_smithi194_196924_object131873:head# 2024-04-29T14:59:53.511 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473424f7:::benchmark_data_smithi194_196924_object129843:head# 2024-04-29T14:59:53.511 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473765e4:::benchmark_data_smithi194_196924_object99315:head# 2024-04-29T14:59:53.512 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4737c041:::benchmark_data_smithi194_196924_object142787:head# 2024-04-29T14:59:53.512 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47388e70:::benchmark_data_smithi194_196924_object153845:head# 2024-04-29T14:59:53.513 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4739612b:::benchmark_data_smithi194_196924_object103888:head# 2024-04-29T14:59:53.513 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473acfdc:::benchmark_data_smithi194_196924_object132122:head# 2024-04-29T14:59:53.513 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473ae9d5:::benchmark_data_smithi194_196924_object153414:head# 2024-04-29T14:59:53.514 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473bffec:::benchmark_data_smithi194_196924_object131759:head# 2024-04-29T14:59:53.514 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473d2f80:::benchmark_data_smithi194_196924_object153408:head# 2024-04-29T14:59:53.515 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473d3fc5:::benchmark_data_smithi194_196924_object126796:head# 2024-04-29T14:59:53.516 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473f9979:::benchmark_data_smithi194_196924_object138629:head# 2024-04-29T14:59:53.516 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:473fe01d:::benchmark_data_smithi194_196924_object101804:head# 2024-04-29T14:59:53.517 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:474169a6:::benchmark_data_smithi194_196924_object119804:head# 2024-04-29T14:59:53.520 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47438a7d:::benchmark_data_smithi194_196924_object132033:head# 2024-04-29T14:59:53.521 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47442112:::benchmark_data_smithi194_196924_object133943:head# 2024-04-29T14:59:53.521 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47468daa:::benchmark_data_smithi194_196924_object98324:head# 2024-04-29T14:59:53.521 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4747a373:::benchmark_data_smithi194_196924_object137108:head# 2024-04-29T14:59:53.521 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47484b66:::benchmark_data_smithi194_196924_object113396:head# 2024-04-29T14:59:53.521 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47487d29:::benchmark_data_smithi194_196924_object125948:head# 2024-04-29T14:59:53.521 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:474aa56e:::benchmark_data_smithi194_196924_object145249:head# 2024-04-29T14:59:53.523 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:474b94d3:::benchmark_data_smithi194_196924_object143024:head# 2024-04-29T14:59:53.523 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:474ce03c:::benchmark_data_smithi194_196924_object130868:head# 2024-04-29T14:59:53.523 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:474e29fe:::benchmark_data_smithi194_196924_object153447:head# 2024-04-29T14:59:53.523 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47508761:::benchmark_data_smithi194_196924_object135630:head# 2024-04-29T14:59:53.523 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47527207:::benchmark_data_smithi194_196924_object121494:head# 2024-04-29T14:59:53.524 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4752c1b3:::benchmark_data_smithi194_196924_object111502:head# 2024-04-29T14:59:53.524 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4755e01b:::benchmark_data_smithi194_196924_object121646:head# 2024-04-29T14:59:53.524 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47561abe:::benchmark_data_smithi194_196924_object123851:head# 2024-04-29T14:59:53.524 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4756ea44:::benchmark_data_smithi194_196924_object144729:head# 2024-04-29T14:59:53.525 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:475853c4:::benchmark_data_smithi194_196924_object148628:head# 2024-04-29T14:59:53.525 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:475a1180:::benchmark_data_smithi194_196924_object121275:head# 2024-04-29T14:59:53.526 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:475c362a:::benchmark_data_smithi194_196924_object130345:head# 2024-04-29T14:59:53.526 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:475e5145:::benchmark_data_smithi194_196924_object107574:head# 2024-04-29T14:59:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:475f8a38:::benchmark_data_smithi194_196924_object111764:head# 2024-04-29T14:59:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:475fca31:::benchmark_data_smithi194_196924_object150317:head# 2024-04-29T14:59:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:475fe450:::benchmark_data_smithi194_196924_object155423:head# 2024-04-29T14:59:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47603e6e:::benchmark_data_smithi194_196924_object113588:head# 2024-04-29T14:59:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47608d26:::benchmark_data_smithi194_196924_object131863:head# 2024-04-29T14:59:53.531 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47617a99:::benchmark_data_smithi194_196924_object113775:head# 2024-04-29T14:59:53.531 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47617f8b:::benchmark_data_smithi194_196924_object153896:head# 2024-04-29T14:59:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4761b33d:::benchmark_data_smithi194_196924_object141191:head# 2024-04-29T14:59:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4761c247:::benchmark_data_smithi194_196924_object96538:head# 2024-04-29T14:59:53.534 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:476270c8:::benchmark_data_smithi194_196924_object131626:head# 2024-04-29T14:59:53.537 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:53.539 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47634253:::benchmark_data_smithi194_196924_object102613:head# 2024-04-29T14:59:53.539 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47648851:::benchmark_data_smithi194_196924_object142079:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4766bb5d:::benchmark_data_smithi194_196924_object157415:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47699c60:::benchmark_data_smithi194_196924_object147027:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4769ba60:::benchmark_data_smithi194_196924_object145224:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4769e108:::benchmark_data_smithi194_196924_object110602:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:476ab735:::benchmark_data_smithi194_196924_object129163:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:476afb01:::benchmark_data_smithi194_196924_object141658:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:476c3153:::benchmark_data_smithi194_196924_object114171:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:476c6792:::benchmark_data_smithi194_196924_object112962:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:476e4fee:::benchmark_data_smithi194_196924_object107901:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47708f83:::benchmark_data_smithi194_196924_object151738:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47713bdf:::benchmark_data_smithi194_196924_object123663:head# 2024-04-29T14:59:53.540 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47716cf6:::benchmark_data_smithi194_196924_object117155:head# 2024-04-29T14:59:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4771b3e1:::benchmark_data_smithi194_196924_object145804:head# 2024-04-29T14:59:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4771ffa1:::benchmark_data_smithi194_196924_object103466:head# 2024-04-29T14:59:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4772b835:::benchmark_data_smithi194_196924_object128103:head# 2024-04-29T14:59:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4772e5de:::benchmark_data_smithi194_196924_object102632:head# 2024-04-29T14:59:53.542 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47736e67:::benchmark_data_smithi194_196924_object100484:head# 2024-04-29T14:59:53.542 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4775e21b:::benchmark_data_smithi194_196924_object137320:head# 2024-04-29T14:59:53.542 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4776f551:::benchmark_data_smithi194_196924_object150627:head# 2024-04-29T14:59:53.543 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4776f60b:::benchmark_data_smithi194_196924_object131999:head# 2024-04-29T14:59:53.543 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4778efbc:::benchmark_data_smithi194_196924_object97408:head# 2024-04-29T14:59:53.544 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4779244d:::benchmark_data_smithi194_196924_object151322:head# 2024-04-29T14:59:53.544 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4779a91e:::benchmark_data_smithi194_196924_object99312:head# 2024-04-29T14:59:53.545 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:477bb02f:::benchmark_data_smithi194_196924_object110096:head# 2024-04-29T14:59:53.545 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:477ceede:::benchmark_data_smithi194_196924_object123995:head# 2024-04-29T14:59:53.546 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:477d9546:::benchmark_data_smithi194_196924_object141928:head# 2024-04-29T14:59:53.546 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47808345:::benchmark_data_smithi194_196924_object118302:head# 2024-04-29T14:59:53.546 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4780f937:::benchmark_data_smithi194_196924_object159351:head# 2024-04-29T14:59:53.547 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478121fd:::benchmark_data_smithi194_196924_object101021:head# 2024-04-29T14:59:53.547 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4781d63d:::benchmark_data_smithi194_196924_object117149:head# 2024-04-29T14:59:53.548 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47829c72:::benchmark_data_smithi194_196924_object134920:head# 2024-04-29T14:59:53.548 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47844316:::benchmark_data_smithi194_196924_object109160:head# 2024-04-29T14:59:53.549 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4785fa71:::benchmark_data_smithi194_196924_object107974:head# 2024-04-29T14:59:53.549 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4788c93d:::benchmark_data_smithi194_196924_object155474:head# 2024-04-29T14:59:53.550 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478910ff:::benchmark_data_smithi194_196924_object149916:head# 2024-04-29T14:59:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4789880f:::benchmark_data_smithi194_196924_object97466:head# 2024-04-29T14:59:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478b5327:::benchmark_data_smithi194_196924_object151445:head# 2024-04-29T14:59:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478ca298:::benchmark_data_smithi194_196924_object130341:head# 2024-04-29T14:59:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478cf2ef:::benchmark_data_smithi194_196924_object126314:head# 2024-04-29T14:59:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478d6658:::benchmark_data_smithi194_196924_object99045:head# 2024-04-29T14:59:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478da596:::benchmark_data_smithi194_196924_object109058:head# 2024-04-29T14:59:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478db77f:::benchmark_data_smithi194_196924_object143994:head# 2024-04-29T14:59:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:478df825:::benchmark_data_smithi194_196924_object101154:head# 2024-04-29T14:59:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47913361:::benchmark_data_smithi194_196924_object113547:head# 2024-04-29T14:59:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4791850f:::benchmark_data_smithi194_196924_object101880:head# 2024-04-29T14:59:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47918eb6:::benchmark_data_smithi194_196924_object122653:head# 2024-04-29T14:59:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47921e56:::benchmark_data_smithi194_196924_object104385:head# 2024-04-29T14:59:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4793a26b:::benchmark_data_smithi194_196924_object125599:head# 2024-04-29T14:59:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4794d842:::benchmark_data_smithi194_196924_object152919:head# 2024-04-29T14:59:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47968480:::benchmark_data_smithi194_196924_object102621:head# 2024-04-29T14:59:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47970b34:::benchmark_data_smithi194_196924_object131584:head# 2024-04-29T14:59:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4799cfbe:::benchmark_data_smithi194_196924_object142583:head# 2024-04-29T14:59:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:479a85cc:::benchmark_data_smithi194_196924_object147352:head# 2024-04-29T14:59:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:479d16f5:::benchmark_data_smithi194_196924_object155116:head# 2024-04-29T14:59:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:479e88a1:::benchmark_data_smithi194_196924_object158942:head# 2024-04-29T14:59:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:479f568b:::benchmark_data_smithi194_196924_object112615:head# 2024-04-29T14:59:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:479f6169:::benchmark_data_smithi194_196924_object153757:head# 2024-04-29T14:59:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:479f84e2:::benchmark_data_smithi194_196924_object96542:head# 2024-04-29T14:59:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:479f8ac6:::benchmark_data_smithi194_196924_object107516:head# 2024-04-29T14:59:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a03956:::benchmark_data_smithi194_196924_object114923:head# 2024-04-29T14:59:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a16e96:::benchmark_data_smithi194_196924_object146857:head# 2024-04-29T14:59:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a281ec:::benchmark_data_smithi194_196924_object157637:head# 2024-04-29T14:59:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a29461:::benchmark_data_smithi194_196924_object147919:head# 2024-04-29T14:59:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a29712:::benchmark_data_smithi194_196924_object98789:head# 2024-04-29T14:59:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a3be29:::benchmark_data_smithi194_196924_object115402:head# 2024-04-29T14:59:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a46177:::benchmark_data_smithi194_196924_object159584:head# 2024-04-29T14:59:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a51fcb:::benchmark_data_smithi194_196924_object143177:head# 2024-04-29T14:59:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a58dab:::benchmark_data_smithi194_196924_object126052:head# 2024-04-29T14:59:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a6dd09:::benchmark_data_smithi194_196924_object133171:head# 2024-04-29T14:59:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a712fc:::benchmark_data_smithi194_196924_object119514:head# 2024-04-29T14:59:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47a9c505:::benchmark_data_smithi194_196924_object102383:head# 2024-04-29T14:59:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ac8c0c:::benchmark_data_smithi194_196924_object118172:head# 2024-04-29T14:59:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47add173:::benchmark_data_smithi194_196924_object111714:head# 2024-04-29T14:59:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47af192e:::benchmark_data_smithi194_196924_object138982:head# 2024-04-29T14:59:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47af1e89:::benchmark_data_smithi194_196924_object106950:head# 2024-04-29T14:59:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b0a98d:::benchmark_data_smithi194_196924_object153651:head# 2024-04-29T14:59:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b30389:::benchmark_data_smithi194_196924_object99892:head# 2024-04-29T14:59:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b31476:::benchmark_data_smithi194_196924_object159066:head# 2024-04-29T14:59:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b344ec:::benchmark_data_smithi194_196924_object131029:head# 2024-04-29T14:59:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b4b2ef:::benchmark_data_smithi194_196924_object131408:head# 2024-04-29T14:59:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b511e2:::benchmark_data_smithi194_196924_object115400:head# 2024-04-29T14:59:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b56083:::benchmark_data_smithi194_196924_object137910:head# 2024-04-29T14:59:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b66c62:::benchmark_data_smithi194_196924_object131537:head# 2024-04-29T14:59:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b6d252:::benchmark_data_smithi194_196924_object157058:head# 2024-04-29T14:59:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b79126:::benchmark_data_smithi194_196924_object118748:head# 2024-04-29T14:59:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b7cf6b:::benchmark_data_smithi194_196924_object103176:head# 2024-04-29T14:59:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b8368c:::benchmark_data_smithi194_196924_object98887:head# 2024-04-29T14:59:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47b9cda1:::benchmark_data_smithi194_196924_object105259:head# 2024-04-29T14:59:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ba8346:::benchmark_data_smithi194_196924_object112790:head# 2024-04-29T14:59:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ba851b:::benchmark_data_smithi194_196924_object109474:head# 2024-04-29T14:59:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47bc899e:::benchmark_data_smithi194_196924_object145735:head# 2024-04-29T14:59:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c1383e:::benchmark_data_smithi194_196924_object118397:head# 2024-04-29T14:59:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c16ee5:::benchmark_data_smithi194_196924_object158222:head# 2024-04-29T14:59:53.578 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c23412:::benchmark_data_smithi194_196924_object108116:head# 2024-04-29T14:59:53.578 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c5fc50:::benchmark_data_smithi194_196924_object98738:head# 2024-04-29T14:59:53.579 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c6e238:::benchmark_data_smithi194_196924_object155854:head# 2024-04-29T14:59:53.579 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c6e977:::benchmark_data_smithi194_196924_object139871:head# 2024-04-29T14:59:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c87c3d:::benchmark_data_smithi194_196924_object159644:head# 2024-04-29T14:59:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47c8d9a1:::benchmark_data_smithi194_196924_object123788:head# 2024-04-29T14:59:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ca0680:::benchmark_data_smithi194_196924_object107297:head# 2024-04-29T14:59:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47cb3325:::benchmark_data_smithi194_196924_object132806:head# 2024-04-29T14:59:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ccad5c:::benchmark_data_smithi194_196924_object138041:head# 2024-04-29T14:59:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47cef32a:::benchmark_data_smithi194_196924_object141643:head# 2024-04-29T14:59:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47cf1953:::benchmark_data_smithi194_196924_object101227:head# 2024-04-29T14:59:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d0be17:::benchmark_data_smithi194_196924_object107724:head# 2024-04-29T14:59:53.583 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d0ea88:::benchmark_data_smithi194_196924_object115811:head# 2024-04-29T14:59:53.583 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d1b434:::benchmark_data_smithi194_196924_object116266:head# 2024-04-29T14:59:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d24844:::benchmark_data_smithi194_196924_object127160:head# 2024-04-29T14:59:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d37161:::benchmark_data_smithi194_196924_object96727:head# 2024-04-29T14:59:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d3ec8d:::benchmark_data_smithi194_196924_object98875:head# 2024-04-29T14:59:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d40c65:::benchmark_data_smithi194_196924_object157238:head# 2024-04-29T14:59:53.586 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d4e6ab:::benchmark_data_smithi194_196924_object113235:head# 2024-04-29T14:59:53.586 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d67774:::benchmark_data_smithi194_196924_object124911:head# 2024-04-29T14:59:53.586 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d6a697:::benchmark_data_smithi194_196924_object110328:head# 2024-04-29T14:59:53.587 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d91ae1:::benchmark_data_smithi194_196924_object135428:head# 2024-04-29T14:59:53.587 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d92cad:::benchmark_data_smithi194_196924_object152090:head# 2024-04-29T14:59:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47d96948:::benchmark_data_smithi194_196924_object143412:head# 2024-04-29T14:59:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47da15ec:::benchmark_data_smithi194_196924_object130384:head# 2024-04-29T14:59:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47da6755:::benchmark_data_smithi194_196924_object131552:head# 2024-04-29T14:59:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47db21c2:::benchmark_data_smithi194_196924_object140641:head# 2024-04-29T14:59:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47db9a4e:::benchmark_data_smithi194_196924_object157046:head# 2024-04-29T14:59:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47df1794:::benchmark_data_smithi194_196924_object158720:head# 2024-04-29T14:59:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47df98b6:::benchmark_data_smithi194_196924_object100142:head# 2024-04-29T14:59:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e09007:::benchmark_data_smithi194_196924_object98307:head# 2024-04-29T14:59:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e19e67:::benchmark_data_smithi194_196924_object98473:head# 2024-04-29T14:59:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e25037:::benchmark_data_smithi194_196924_object114893:head# 2024-04-29T14:59:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e54215:::benchmark_data_smithi194_196924_object155573:head# 2024-04-29T14:59:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e584f5:::benchmark_data_smithi194_196924_object112013:head# 2024-04-29T14:59:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e6fb5d:::benchmark_data_smithi194_196924_object123232:head# 2024-04-29T14:59:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e8436b:::benchmark_data_smithi194_196924_object131981:head# 2024-04-29T14:59:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e8b7e9:::benchmark_data_smithi194_196924_object101522:head# 2024-04-29T14:59:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e957b6:::benchmark_data_smithi194_196924_object120449:head# 2024-04-29T14:59:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e99a59:::benchmark_data_smithi194_196924_object138356:head# 2024-04-29T14:59:53.596 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47e9f12c:::benchmark_data_smithi194_196924_object153103:head# 2024-04-29T14:59:53.596 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ea390b:::benchmark_data_smithi194_196924_object150557:head# 2024-04-29T14:59:53.596 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47eadfc3:::benchmark_data_smithi194_196924_object138079:head# 2024-04-29T14:59:53.597 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ecf053:::benchmark_data_smithi194_196924_object135248:head# 2024-04-29T14:59:53.597 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ee0e1a:::benchmark_data_smithi194_196924_object153851:head# 2024-04-29T14:59:53.598 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ee693f:::benchmark_data_smithi194_196924_object145481:head# 2024-04-29T14:59:53.598 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ef3452:::benchmark_data_smithi194_196924_object100663:head# 2024-04-29T14:59:53.599 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47f3c0ee:::benchmark_data_smithi194_196924_object134817:head# 2024-04-29T14:59:53.599 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47f4196d:::benchmark_data_smithi194_196924_object125283:head# 2024-04-29T14:59:53.599 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47f663dc:::benchmark_data_smithi194_196924_object128744:head# 2024-04-29T14:59:53.600 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47f83335:::benchmark_data_smithi194_196924_object150985:head# 2024-04-29T14:59:53.600 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47fa77b8:::benchmark_data_smithi194_196924_object114433:head# 2024-04-29T14:59:53.601 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47fb6f4f:::benchmark_data_smithi194_196924_object112900:head# 2024-04-29T14:59:53.601 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47fd9929:::benchmark_data_smithi194_196924_object157799:head# 2024-04-29T14:59:53.602 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47fe37aa:::benchmark_data_smithi194_196924_object140229:head# 2024-04-29T14:59:53.602 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:47ff2b33:::benchmark_data_smithi194_196924_object144737:head# 2024-04-29T14:59:53.603 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:480329c5:::benchmark_data_smithi194_196924_object118213:head# 2024-04-29T14:59:53.603 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:480710a7:::benchmark_data_smithi194_196924_object146615:head# 2024-04-29T14:59:53.603 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4808e686:::benchmark_data_smithi194_196924_object133349:head# 2024-04-29T14:59:53.604 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:480a1ffc:::benchmark_data_smithi194_196924_object128844:head# 2024-04-29T14:59:53.604 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:480a4382:::benchmark_data_smithi194_196924_object141702:head# 2024-04-29T14:59:53.605 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:480b4856:::benchmark_data_smithi194_196924_object97341:head# 2024-04-29T14:59:53.606 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:480f7dda:::benchmark_data_smithi194_196924_object113344:head# 2024-04-29T14:59:53.606 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48104e37:::benchmark_data_smithi194_196924_object152292:head# 2024-04-29T14:59:53.606 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:481285ce:::benchmark_data_smithi194_196924_object121396:head# 2024-04-29T14:59:53.607 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48128fc0:::benchmark_data_smithi194_196924_object129674:head# 2024-04-29T14:59:53.607 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:481373e1:::benchmark_data_smithi194_196924_object143144:head# 2024-04-29T14:59:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4813e0ea:::benchmark_data_smithi194_196924_object142360:head# 2024-04-29T14:59:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48179ed9:::benchmark_data_smithi194_196924_object117391:head# 2024-04-29T14:59:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4817b13a:::benchmark_data_smithi194_196924_object119167:head# 2024-04-29T14:59:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48197240:::benchmark_data_smithi194_196924_object112363:head# 2024-04-29T14:59:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:481ac5f1:::benchmark_data_smithi194_196924_object101254:head# 2024-04-29T14:59:53.610 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:481d462d:::benchmark_data_smithi194_196924_object112870:head# 2024-04-29T14:59:53.610 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:481f19b9:::benchmark_data_smithi194_196924_object151434:head# 2024-04-29T14:59:53.611 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4820e0a7:::benchmark_data_smithi194_196924_object144727:head# 2024-04-29T14:59:53.611 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48210059:::benchmark_data_smithi194_196924_object136842:head# 2024-04-29T14:59:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482116bc:::benchmark_data_smithi194_196924_object155352:head# 2024-04-29T14:59:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4821e826:::benchmark_data_smithi194_196924_object121452:head# 2024-04-29T14:59:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4821f281:::benchmark_data_smithi194_196924_object120794:head# 2024-04-29T14:59:53.613 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4823c071:::benchmark_data_smithi194_196924_object100923:head# 2024-04-29T14:59:53.613 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4824f345:::benchmark_data_smithi194_196924_object155947:head# 2024-04-29T14:59:53.614 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482521d4:::benchmark_data_smithi194_196924_object129947:head# 2024-04-29T14:59:53.614 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4825b582:::benchmark_data_smithi194_196924_object144367:head# 2024-04-29T14:59:53.614 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4827717f:::benchmark_data_smithi194_196924_object157237:head# 2024-04-29T14:59:53.615 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48277f0c:::benchmark_data_smithi194_196924_object98681:head# 2024-04-29T14:59:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4827acbc:::benchmark_data_smithi194_196924_object123781:head# 2024-04-29T14:59:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4828433a:::benchmark_data_smithi194_196924_object103536:head# 2024-04-29T14:59:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4828cae6:::benchmark_data_smithi194_196924_object115200:head# 2024-04-29T14:59:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4829cf9a:::benchmark_data_smithi194_196924_object143153:head# 2024-04-29T14:59:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4829ee3e:::benchmark_data_smithi194_196924_object158825:head# 2024-04-29T14:59:53.618 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482ad1f1:::benchmark_data_smithi194_196924_object159108:head# 2024-04-29T14:59:53.618 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482afb89:::benchmark_data_smithi194_196924_object132178:head# 2024-04-29T14:59:53.618 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482b42b9:::benchmark_data_smithi194_196924_object132743:head# 2024-04-29T14:59:53.619 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482b991c:::benchmark_data_smithi194_196924_object132727:head# 2024-04-29T14:59:53.619 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482c032e:::benchmark_data_smithi194_196924_object110982:head# 2024-04-29T14:59:53.620 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482d2feb:::benchmark_data_smithi194_196924_object118180:head# 2024-04-29T14:59:53.620 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482ec72a:::benchmark_data_smithi194_196924_object114367:head# 2024-04-29T14:59:53.621 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:482f6319:::benchmark_data_smithi194_196924_object112398:head# 2024-04-29T14:59:53.621 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48304ad9:::benchmark_data_smithi194_196924_object124242:head# 2024-04-29T14:59:53.622 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48306a67:::benchmark_data_smithi194_196924_object100464:head# 2024-04-29T14:59:53.623 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48326cf5:::benchmark_data_smithi194_196924_object152170:head# 2024-04-29T14:59:53.623 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48345456:::benchmark_data_smithi194_196924_object121268:head# 2024-04-29T14:59:53.624 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4834ccf4:::benchmark_data_smithi194_196924_object121417:head# 2024-04-29T14:59:53.624 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:483564e4:::benchmark_data_smithi194_196924_object152861:head# 2024-04-29T14:59:53.624 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4835ffed:::benchmark_data_smithi194_196924_object109679:head# 2024-04-29T14:59:53.625 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:483bb230:::benchmark_data_smithi194_196924_object151888:head# 2024-04-29T14:59:53.625 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:483c9736:::benchmark_data_smithi194_196924_object133038:head# 2024-04-29T14:59:53.626 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:483d1b03:::benchmark_data_smithi194_196924_object143322:head# 2024-04-29T14:59:53.626 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:483e1ce0:::benchmark_data_smithi194_196924_object145929:head# 2024-04-29T14:59:53.627 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:483e99b9:::benchmark_data_smithi194_196924_object129913:head# 2024-04-29T14:59:53.627 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484170f5:::benchmark_data_smithi194_196924_object108135:head# 2024-04-29T14:59:53.627 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4843211a:::benchmark_data_smithi194_196924_object96966:head# 2024-04-29T14:59:53.628 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484367e9:::benchmark_data_smithi194_196924_object134222:head# 2024-04-29T14:59:53.628 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48443935:::benchmark_data_smithi194_196924_object103826:head# 2024-04-29T14:59:53.629 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4844839c:::benchmark_data_smithi194_196924_object150727:head# 2024-04-29T14:59:53.629 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4847cf81:::benchmark_data_smithi194_196924_object116677:head# 2024-04-29T14:59:53.629 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4849f427:::benchmark_data_smithi194_196924_object109929:head# 2024-04-29T14:59:53.630 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484a2302:::benchmark_data_smithi194_196924_object105776:head# 2024-04-29T14:59:53.630 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484a2dfa:::benchmark_data_smithi194_196924_object109753:head# 2024-04-29T14:59:53.631 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484ae8d5:::benchmark_data_smithi194_196924_object151384:head# 2024-04-29T14:59:53.631 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484c6353:::benchmark_data_smithi194_196924_object118524:head# 2024-04-29T14:59:53.632 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484d53c4:::benchmark_data_smithi194_196924_object136009:head# 2024-04-29T14:59:53.632 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484dd3e3:::benchmark_data_smithi194_196924_object137707:head# 2024-04-29T14:59:53.632 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484ea02a:::benchmark_data_smithi194_196924_object140395:head# 2024-04-29T14:59:53.633 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:484f181d:::benchmark_data_smithi194_196924_object111739:head# 2024-04-29T14:59:53.633 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485013a3:::benchmark_data_smithi194_196924_object159919:head# 2024-04-29T14:59:53.634 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4851f03b:::benchmark_data_smithi194_196924_object138914:head# 2024-04-29T14:59:53.634 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4851fb00:::benchmark_data_smithi194_196924_object158355:head# 2024-04-29T14:59:53.634 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48522c85:::benchmark_data_smithi194_196924_object95618:head# 2024-04-29T14:59:53.635 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48558100:::benchmark_data_smithi194_196924_object138707:head# 2024-04-29T14:59:53.635 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4855bb13:::benchmark_data_smithi194_196924_object110270:head# 2024-04-29T14:59:53.636 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48566ede:::benchmark_data_smithi194_196924_object112536:head# 2024-04-29T14:59:53.636 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48566f2e:::benchmark_data_smithi194_196924_object136611:head# 2024-04-29T14:59:53.637 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485671d9:::benchmark_data_smithi194_196924_object100670:head# 2024-04-29T14:59:53.637 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4856a450:::benchmark_data_smithi194_196924_object105303:head# 2024-04-29T14:59:53.637 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4856d440:::benchmark_data_smithi194_196924_object102019:head# 2024-04-29T14:59:53.638 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4858904e:::benchmark_data_smithi194_196924_object119092:head# 2024-04-29T14:59:53.638 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485893ea:::benchmark_data_smithi194_196924_object143462:head# 2024-04-29T14:59:53.639 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4859d92f:::benchmark_data_smithi194_196924_object148713:head# 2024-04-29T14:59:53.639 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4859e13d:::benchmark_data_smithi194_196924_object125833:head# 2024-04-29T14:59:53.639 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485b4193:::benchmark_data_smithi194_196924_object114078:head# 2024-04-29T14:59:53.640 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485c4e75:::benchmark_data_smithi194_196924_object128907:head# 2024-04-29T14:59:53.641 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485ca464:::benchmark_data_smithi194_196924_object141514:head# 2024-04-29T14:59:53.641 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485d8af6:::benchmark_data_smithi194_196924_object138799:head# 2024-04-29T14:59:53.642 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485dab10:::benchmark_data_smithi194_196924_object136619:head# 2024-04-29T14:59:53.642 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:485e8537:::benchmark_data_smithi194_196924_object138397:head# 2024-04-29T14:59:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48601e21:::benchmark_data_smithi194_196924_object136020:head# 2024-04-29T14:59:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4860710d:::benchmark_data_smithi194_196924_object155608:head# 2024-04-29T14:59:53.644 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48617253:::benchmark_data_smithi194_196924_object140347:head# 2024-04-29T14:59:53.644 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48620033:::benchmark_data_smithi194_196924_object97631:head# 2024-04-29T14:59:53.644 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48639a35:::benchmark_data_smithi194_196924_object112714:head# 2024-04-29T14:59:53.645 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4863b42d:::benchmark_data_smithi194_196924_object115706:head# 2024-04-29T14:59:53.645 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4863cd3c:::benchmark_data_smithi194_196924_object104364:head# 2024-04-29T14:59:53.646 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4863ef6b:::benchmark_data_smithi194_196924_object101033:head# 2024-04-29T14:59:53.646 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4866b55b:::benchmark_data_smithi194_196924_object102809:head# 2024-04-29T14:59:53.646 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4867629b:::benchmark_data_smithi194_196924_object124238:head# 2024-04-29T14:59:53.647 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:486a3356:::benchmark_data_smithi194_196924_object131982:head# 2024-04-29T14:59:53.647 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:486c3f3d:::benchmark_data_smithi194_196924_object120167:head# 2024-04-29T14:59:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:486ca037:::benchmark_data_smithi194_196924_object111881:head# 2024-04-29T14:59:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:486db1f9:::benchmark_data_smithi194_196924_object127201:head# 2024-04-29T14:59:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4871437a:::benchmark_data_smithi194_196924_object124210:head# 2024-04-29T14:59:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48730f6d:::benchmark_data_smithi194_196924_object145293:head# 2024-04-29T14:59:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4873ddc8:::benchmark_data_smithi194_196924_object154161:head# 2024-04-29T14:59:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4875064a:::benchmark_data_smithi194_196924_object104905:head# 2024-04-29T14:59:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:487704a0:::benchmark_data_smithi194_196924_object123856:head# 2024-04-29T14:59:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:487a477a:::benchmark_data_smithi194_196924_object109080:head# 2024-04-29T14:59:53.651 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:487a829e:::benchmark_data_smithi194_196924_object143151:head# 2024-04-29T14:59:53.651 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:487cff82:::benchmark_data_smithi194_196924_object121952:head# 2024-04-29T14:59:53.652 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:487def7e:::benchmark_data_smithi194_196924_object138236:head# 2024-04-29T14:59:53.652 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:487f0bfe:::benchmark_data_smithi194_196924_object126573:head# 2024-04-29T14:59:53.652 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48809d6d:::benchmark_data_smithi194_196924_object147636:head# 2024-04-29T14:59:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48819cc1:::benchmark_data_smithi194_196924_object152577:head# 2024-04-29T14:59:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488214b9:::benchmark_data_smithi194_196924_object107982:head# 2024-04-29T14:59:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48833904:::benchmark_data_smithi194_196924_object121250:head# 2024-04-29T14:59:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48843cee:::benchmark_data_smithi194_196924_object133160:head# 2024-04-29T14:59:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48850cc7:::benchmark_data_smithi194_196924_object140082:head# 2024-04-29T14:59:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48852ea9:::benchmark_data_smithi194_196924_object146628:head# 2024-04-29T14:59:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48858cc2:::benchmark_data_smithi194_196924_object159591:head# 2024-04-29T14:59:53.656 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48881a8e:::benchmark_data_smithi194_196924_object114771:head# 2024-04-29T14:59:53.656 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488a38d7:::benchmark_data_smithi194_196924_object153199:head# 2024-04-29T14:59:53.656 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488b4fb9:::benchmark_data_smithi194_196924_object102697:head# 2024-04-29T14:59:53.657 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488b901c:::benchmark_data_smithi194_196924_object106902:head# 2024-04-29T14:59:53.657 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488cc0b0:::benchmark_data_smithi194_196924_object141052:head# 2024-04-29T14:59:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488d9ece:::benchmark_data_smithi194_196924_object159479:head# 2024-04-29T14:59:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488ea557:::benchmark_data_smithi194_196924_object121694:head# 2024-04-29T14:59:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488eb328:::benchmark_data_smithi194_196924_object135796:head# 2024-04-29T14:59:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488ed7ac:::benchmark_data_smithi194_196924_object129008:head# 2024-04-29T14:59:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:488ef04f:::benchmark_data_smithi194_196924_object138510:head# 2024-04-29T14:59:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48902afc:::benchmark_data_smithi194_196924_object126254:head# 2024-04-29T14:59:53.661 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4892395e:::benchmark_data_smithi194_196924_object130952:head# 2024-04-29T14:59:53.661 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4893b64f:::benchmark_data_smithi194_196924_object104537:head# 2024-04-29T14:59:53.661 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48950ef1:::benchmark_data_smithi194_196924_object104795:head# 2024-04-29T14:59:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:489801cf:::benchmark_data_smithi194_196924_object104065:head# 2024-04-29T14:59:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4898e03c:::benchmark_data_smithi194_196924_object121641:head# 2024-04-29T14:59:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:489a0f2c:::benchmark_data_smithi194_196924_object133140:head# 2024-04-29T14:59:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:489a3c2b:::benchmark_data_smithi194_196924_object157482:head# 2024-04-29T14:59:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:489a63bc:::benchmark_data_smithi194_196924_object112028:head# 2024-04-29T14:59:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:489aa784:::benchmark_data_smithi194_196924_object157886:head# 2024-04-29T14:59:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:489da0b2:::benchmark_data_smithi194_196924_object96791:head# 2024-04-29T14:59:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:489f7e35:::benchmark_data_smithi194_196924_object135120:head# 2024-04-29T14:59:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48a0521b:::benchmark_data_smithi194_196924_object110936:head# 2024-04-29T14:59:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48a07ac2:::benchmark_data_smithi194_196924_object118169:head# 2024-04-29T14:59:53.666 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48a53691:::benchmark_data_smithi194_196924_object158333:head# 2024-04-29T14:59:53.666 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48a69569:::benchmark_data_smithi194_196924_object115632:head# 2024-04-29T14:59:53.667 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48a7c9a1:::benchmark_data_smithi194_196924_object96134:head# 2024-04-29T14:59:53.667 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48a8a873:::benchmark_data_smithi194_196924_object139185:head# 2024-04-29T14:59:53.667 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48a8d7c7:::benchmark_data_smithi194_196924_object103953:head# 2024-04-29T14:59:53.668 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ab726a:::benchmark_data_smithi194_196924_object150188:head# 2024-04-29T14:59:53.668 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ac3dac:::benchmark_data_smithi194_196924_object106674:head# 2024-04-29T14:59:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48adb247:::benchmark_data_smithi194_196924_object136694:head# 2024-04-29T14:59:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48aee2a0:::benchmark_data_smithi194_196924_object154249:head# 2024-04-29T14:59:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48af1773:::benchmark_data_smithi194_196924_object104177:head# 2024-04-29T14:59:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b16245:::benchmark_data_smithi194_196924_object136785:head# 2024-04-29T14:59:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b170e4:::benchmark_data_smithi194_196924_object136552:head# 2024-04-29T14:59:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b1f1bb:::benchmark_data_smithi194_196924_object114694:head# 2024-04-29T14:59:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b2434e:::benchmark_data_smithi194_196924_object103097:head# 2024-04-29T14:59:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b2873a:::benchmark_data_smithi194_196924_object136393:head# 2024-04-29T14:59:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b385e4:::benchmark_data_smithi194_196924_object153102:head# 2024-04-29T14:59:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b543d9:::benchmark_data_smithi194_196924_object147805:head# 2024-04-29T14:59:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b58f13:::benchmark_data_smithi194_196924_object108716:head# 2024-04-29T14:59:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b6d7fe:::benchmark_data_smithi194_196924_object101160:head# 2024-04-29T14:59:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b6fdcb:::benchmark_data_smithi194_196924_object123437:head# 2024-04-29T14:59:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b8703d:::benchmark_data_smithi194_196924_object156754:head# 2024-04-29T14:59:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b8c475:::benchmark_data_smithi194_196924_object140155:head# 2024-04-29T14:59:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48b97193:::benchmark_data_smithi194_196924_object97483:head# 2024-04-29T14:59:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ba3331:::benchmark_data_smithi194_196924_object150639:head# 2024-04-29T14:59:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48bc034c:::benchmark_data_smithi194_196924_object121488:head# 2024-04-29T14:59:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48bd72f9:::benchmark_data_smithi194_196924_object114386:head# 2024-04-29T14:59:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48bdf82c:::benchmark_data_smithi194_196924_object122083:head# 2024-04-29T14:59:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48bfed6c:::benchmark_data_smithi194_196924_object128732:head# 2024-04-29T14:59:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48c07067:::benchmark_data_smithi194_196924_object116508:head# 2024-04-29T14:59:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48c10175:::benchmark_data_smithi194_196924_object125384:head# 2024-04-29T14:59:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48c1bef5:::benchmark_data_smithi194_196924_object147537:head# 2024-04-29T14:59:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48c1d3fa:::benchmark_data_smithi194_196924_object152914:head# 2024-04-29T14:59:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48c4abed:::benchmark_data_smithi194_196924_object157456:head# 2024-04-29T14:59:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48c57d21:::benchmark_data_smithi194_196924_object115725:head# 2024-04-29T14:59:53.683 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ca6977:::benchmark_data_smithi194_196924_object105927:head# 2024-04-29T14:59:53.683 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48cb5168:::benchmark_data_smithi194_196924_object101492:head# 2024-04-29T14:59:53.683 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48cb7384:::benchmark_data_smithi194_196924_object122661:head# 2024-04-29T14:59:53.684 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48cbab6a:::benchmark_data_smithi194_196924_object121198:head# 2024-04-29T14:59:53.684 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48cc77a0:::benchmark_data_smithi194_196924_object123913:head# 2024-04-29T14:59:53.685 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48cdf85e:::benchmark_data_smithi194_196924_object134578:head# 2024-04-29T14:59:53.685 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ce9958:::benchmark_data_smithi194_196924_object130153:head# 2024-04-29T14:59:53.685 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d156ad:::benchmark_data_smithi194_196924_object112467:head# 2024-04-29T14:59:53.686 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d16638:::benchmark_data_smithi194_196924_object148292:head# 2024-04-29T14:59:53.686 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d1d5b0:::benchmark_data_smithi194_196924_object149509:head# 2024-04-29T14:59:53.687 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d2c1a3:::benchmark_data_smithi194_196924_object126566:head# 2024-04-29T14:59:53.687 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d3018a:::benchmark_data_smithi194_196924_object96275:head# 2024-04-29T14:59:53.688 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d44df5:::benchmark_data_smithi194_196924_object141457:head# 2024-04-29T14:59:53.688 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d58700:::benchmark_data_smithi194_196924_object153106:head# 2024-04-29T14:59:53.689 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d670b5:::benchmark_data_smithi194_196924_object148641:head# 2024-04-29T14:59:53.689 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d77eba:::benchmark_data_smithi194_196924_object131720:head# 2024-04-29T14:59:53.690 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d7d1ff:::benchmark_data_smithi194_196924_object129057:head# 2024-04-29T14:59:53.691 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d95561:::benchmark_data_smithi194_196924_object153197:head# 2024-04-29T14:59:53.691 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48d95d76:::benchmark_data_smithi194_196924_object156540:head# 2024-04-29T14:59:53.692 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48da8fc6:::benchmark_data_smithi194_196924_object125632:head# 2024-04-29T14:59:53.692 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48daf27a:::benchmark_data_smithi194_196924_object124540:head# 2024-04-29T14:59:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48dc95e8:::benchmark_data_smithi194_196924_object117195:head# 2024-04-29T14:59:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48dccf61:::benchmark_data_smithi194_196924_object141212:head# 2024-04-29T14:59:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48def1b8:::benchmark_data_smithi194_196924_object102928:head# 2024-04-29T14:59:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e16506:::benchmark_data_smithi194_196924_object123986:head# 2024-04-29T14:59:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e1b0d3:::benchmark_data_smithi194_196924_object151137:head# 2024-04-29T14:59:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e20876:::benchmark_data_smithi194_196924_object105126:head# 2024-04-29T14:59:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e2fb4a:::benchmark_data_smithi194_196924_object140546:head# 2024-04-29T14:59:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e394dc:::benchmark_data_smithi194_196924_object147830:head# 2024-04-29T14:59:53.696 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e5ec65:::benchmark_data_smithi194_196924_object110421:head# 2024-04-29T14:59:53.696 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e7a5cf:::benchmark_data_smithi194_196924_object104660:head# 2024-04-29T14:59:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e845cb:::benchmark_data_smithi194_196924_object100146:head# 2024-04-29T14:59:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e854b2:::benchmark_data_smithi194_196924_object151303:head# 2024-04-29T14:59:53.698 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e8cce4:::benchmark_data_smithi194_196924_object133833:head# 2024-04-29T14:59:53.698 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48e938f6:::benchmark_data_smithi194_196924_object120409:head# 2024-04-29T14:59:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ec0313:::benchmark_data_smithi194_196924_object95741:head# 2024-04-29T14:59:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ec9bb8:::benchmark_data_smithi194_196924_object126978:head# 2024-04-29T14:59:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ed74e8:::benchmark_data_smithi194_196924_object134959:head# 2024-04-29T14:59:53.701 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48edbe1c:::benchmark_data_smithi194_196924_object106566:head# 2024-04-29T14:59:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ee638d:::benchmark_data_smithi194_196924_object112314:head# 2024-04-29T14:59:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48eeb91c:::benchmark_data_smithi194_196924_object151673:head# 2024-04-29T14:59:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ef5521:::benchmark_data_smithi194_196924_object116595:head# 2024-04-29T14:59:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ef9025:::benchmark_data_smithi194_196924_object116144:head# 2024-04-29T14:59:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48f2d645:::benchmark_data_smithi194_196924_object128748:head# 2024-04-29T14:59:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48f36666:::benchmark_data_smithi194_196924_object131814:head# 2024-04-29T14:59:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48f984c6:::benchmark_data_smithi194_196924_object135904:head# 2024-04-29T14:59:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48f9fbff:::benchmark_data_smithi194_196924_object123269:head# 2024-04-29T14:59:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fb95db:::benchmark_data_smithi194_196924_object157863:head# 2024-04-29T14:59:53.706 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fbe1b7:::benchmark_data_smithi194_196924_object128481:head# 2024-04-29T14:59:53.706 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fd3ab1:::benchmark_data_smithi194_196924_object129988:head# 2024-04-29T14:59:53.706 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fd6396:::benchmark_data_smithi194_196924_object138435:head# 2024-04-29T14:59:53.707 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fdff61:::benchmark_data_smithi194_196924_object124043:head# 2024-04-29T14:59:53.707 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fe4d0e:::benchmark_data_smithi194_196924_object158792:head# 2024-04-29T14:59:53.708 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fe52a5:::benchmark_data_smithi194_196924_object114310:head# 2024-04-29T14:59:53.708 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48fe96e9:::benchmark_data_smithi194_196924_object150610:head# 2024-04-29T14:59:53.710 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ff0024:::benchmark_data_smithi194_196924_object132708:head# 2024-04-29T14:59:53.710 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:48ffafb4:::benchmark_data_smithi194_196924_object153462:head# 2024-04-29T14:59:53.710 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4900b7f2:::benchmark_data_smithi194_196924_object116209:head# 2024-04-29T14:59:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4900d970:::benchmark_data_smithi194_196924_object112197:head# 2024-04-29T14:59:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49032da3:::benchmark_data_smithi194_196924_object111022:head# 2024-04-29T14:59:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:490387ee:::benchmark_data_smithi194_196924_object148964:head# 2024-04-29T14:59:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49040e6e:::benchmark_data_smithi194_196924_object153422:head# 2024-04-29T14:59:53.713 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4904b78d:::benchmark_data_smithi194_196924_object159304:head# 2024-04-29T14:59:53.714 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49051964:::benchmark_data_smithi194_196924_object102849:head# 2024-04-29T14:59:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49052826:::benchmark_data_smithi194_196924_object120415:head# 2024-04-29T14:59:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49057ddf:::benchmark_data_smithi194_196924_object140862:head# 2024-04-29T14:59:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49067222:::benchmark_data_smithi194_196924_object144896:head# 2024-04-29T14:59:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4907323b:::benchmark_data_smithi194_196924_object122034:head# 2024-04-29T14:59:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4907e0c4:::benchmark_data_smithi194_196924_object96253:head# 2024-04-29T14:59:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:490a0871:::benchmark_data_smithi194_196924_object106367:head# 2024-04-29T14:59:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:490a992a:::benchmark_data_smithi194_196924_object158417:head# 2024-04-29T14:59:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:490c8372:::benchmark_data_smithi194_196924_object105860:head# 2024-04-29T14:59:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:490d659c:::benchmark_data_smithi194_196924_object105626:head# 2024-04-29T14:59:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:490fcae1:::benchmark_data_smithi194_196924_object106376:head# 2024-04-29T14:59:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4910e240:::benchmark_data_smithi194_196924_object140826:head# 2024-04-29T14:59:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4910e33a:::benchmark_data_smithi194_196924_object100278:head# 2024-04-29T14:59:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49138a5a:::benchmark_data_smithi194_196924_object155760:head# 2024-04-29T14:59:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49158abb:::benchmark_data_smithi194_196924_object112065:head# 2024-04-29T14:59:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49161cad:::benchmark_data_smithi194_196924_object155782:head# 2024-04-29T14:59:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4916b195:::benchmark_data_smithi194_196924_object126983:head# 2024-04-29T14:59:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:491755f8:::benchmark_data_smithi194_196924_object131587:head# 2024-04-29T14:59:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:491ab4b9:::benchmark_data_smithi194_196924_object137127:head# 2024-04-29T14:59:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:491b06e8:::benchmark_data_smithi194_196924_object151116:head# 2024-04-29T14:59:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:491b194e:::benchmark_data_smithi194_196924_object109269:head# 2024-04-29T14:59:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:491b5388:::benchmark_data_smithi194_196924_object129278:head# 2024-04-29T14:59:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:491efe68:::benchmark_data_smithi194_196924_object132594:head# 2024-04-29T14:59:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4921dbbc:::benchmark_data_smithi194_196924_object104457:head# 2024-04-29T14:59:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:492334e6:::benchmark_data_smithi194_196924_object131844:head# 2024-04-29T14:59:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49247e90:::benchmark_data_smithi194_196924_object108111:head# 2024-04-29T14:59:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49258450:::benchmark_data_smithi194_196924_object102895:head# 2024-04-29T14:59:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49263aa5:::benchmark_data_smithi194_196924_object156663:head# 2024-04-29T14:59:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4926badc:::benchmark_data_smithi194_196924_object134000:head# 2024-04-29T14:59:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49271a37:::benchmark_data_smithi194_196924_object109495:head# 2024-04-29T14:59:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4929e321:::benchmark_data_smithi194_196924_object143606:head# 2024-04-29T14:59:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:492ae1f7:::benchmark_data_smithi194_196924_object149041:head# 2024-04-29T14:59:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:492b2032:::benchmark_data_smithi194_196924_object152666:head# 2024-04-29T14:59:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:492dd943:::benchmark_data_smithi194_196924_object145012:head# 2024-04-29T14:59:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:492e9e6b:::benchmark_data_smithi194_196924_object141934:head# 2024-04-29T14:59:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4930388d:::benchmark_data_smithi194_196924_object114922:head# 2024-04-29T14:59:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49312fcb:::benchmark_data_smithi194_196924_object105070:head# 2024-04-29T14:59:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49314fd2:::benchmark_data_smithi194_196924_object155147:head# 2024-04-29T14:59:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4933205d:::benchmark_data_smithi194_196924_object103405:head# 2024-04-29T14:59:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4933358e:::benchmark_data_smithi194_196924_object140637:head# 2024-04-29T14:59:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4936282d:::benchmark_data_smithi194_196924_object152568:head# 2024-04-29T14:59:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4939d5b4:::benchmark_data_smithi194_196924_object153763:head# 2024-04-29T14:59:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493a6674:::benchmark_data_smithi194_196924_object156836:head# 2024-04-29T14:59:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493ab546:::benchmark_data_smithi194_196924_object136640:head# 2024-04-29T14:59:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493c024d:::benchmark_data_smithi194_196924_object147546:head# 2024-04-29T14:59:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493c6b96:::benchmark_data_smithi194_196924_object144360:head# 2024-04-29T14:59:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493ce39a:::benchmark_data_smithi194_196924_object133897:head# 2024-04-29T14:59:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493d6757:::benchmark_data_smithi194_196924_object102286:head# 2024-04-29T14:59:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493dcf86:::benchmark_data_smithi194_196924_object151579:head# 2024-04-29T14:59:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:493e5caa:::benchmark_data_smithi194_196924_object124710:head# 2024-04-29T14:59:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49401cd3:::benchmark_data_smithi194_196924_object106786:head# 2024-04-29T14:59:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4941270a:::benchmark_data_smithi194_196924_object135014:head# 2024-04-29T14:59:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:494181e9:::benchmark_data_smithi194_196924_object128014:head# 2024-04-29T14:59:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49422b7b:::benchmark_data_smithi194_196924_object108927:head# 2024-04-29T14:59:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4942fd55:::benchmark_data_smithi194_196924_object135645:head# 2024-04-29T14:59:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49434bd8:::benchmark_data_smithi194_196924_object149033:head# 2024-04-29T14:59:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4944f6fc:::benchmark_data_smithi194_196924_object136007:head# 2024-04-29T14:59:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49453d62:::benchmark_data_smithi194_196924_object148800:head# 2024-04-29T14:59:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4945ea35:::benchmark_data_smithi194_196924_object151884:head# 2024-04-29T14:59:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4946a376:::benchmark_data_smithi194_196924_object128883:head# 2024-04-29T14:59:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4947cf1c:::benchmark_data_smithi194_196924_object138989:head# 2024-04-29T14:59:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49483fc0:::benchmark_data_smithi194_196924_object120680:head# 2024-04-29T14:59:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:494c6792:::benchmark_data_smithi194_196924_object103166:head# 2024-04-29T14:59:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:494cb512:::benchmark_data_smithi194_196924_object148846:head# 2024-04-29T14:59:53.751 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:494d9797:::benchmark_data_smithi194_196924_object122086:head# 2024-04-29T14:59:53.751 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:494dcbb0:::benchmark_data_smithi194_196924_object156628:head# 2024-04-29T14:59:53.752 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:494e7321:::benchmark_data_smithi194_196924_object100268:head# 2024-04-29T14:59:53.752 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:494ed20b:::benchmark_data_smithi194_196924_object102634:head# 2024-04-29T14:59:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49509e6f:::benchmark_data_smithi194_196924_object133346:head# 2024-04-29T14:59:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4950ab7f:::benchmark_data_smithi194_196924_object111099:head# 2024-04-29T14:59:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4950c00a:::benchmark_data_smithi194_196924_object122209:head# 2024-04-29T14:59:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49510448:::benchmark_data_smithi194_196924_object145167:head# 2024-04-29T14:59:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49513485:::benchmark_data_smithi194_196924_object156684:head# 2024-04-29T14:59:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:495299ec:::benchmark_data_smithi194_196924_object132649:head# 2024-04-29T14:59:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4953e009:::benchmark_data_smithi194_196924_object152614:head# 2024-04-29T14:59:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4955aad2:::benchmark_data_smithi194_196924_object97389:head# 2024-04-29T14:59:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4955db51:::benchmark_data_smithi194_196924_object142803:head# 2024-04-29T14:59:53.757 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4956f680:::benchmark_data_smithi194_196924_object133966:head# 2024-04-29T14:59:53.757 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:495779ac:::benchmark_data_smithi194_196924_object106328:head# 2024-04-29T14:59:53.758 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:495acd06:::benchmark_data_smithi194_196924_object127970:head# 2024-04-29T14:59:53.758 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:495c00d1:::benchmark_data_smithi194_196924_object121141:head# 2024-04-29T14:59:53.758 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:495fe8c0:::benchmark_data_smithi194_196924_object151995:head# 2024-04-29T14:59:53.759 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:496234c9:::benchmark_data_smithi194_196924_object152722:head# 2024-04-29T14:59:53.759 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49652365:::benchmark_data_smithi194_196924_object157012:head# 2024-04-29T14:59:53.759 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49661954:::benchmark_data_smithi194_196924_object122103:head# 2024-04-29T14:59:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49665dbc:::benchmark_data_smithi194_196924_object134395:head# 2024-04-29T14:59:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4968dac8:::benchmark_data_smithi194_196924_object139932:head# 2024-04-29T14:59:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4968dba7:::benchmark_data_smithi194_196924_object97970:head# 2024-04-29T14:59:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49694e5d:::benchmark_data_smithi194_196924_object109426:head# 2024-04-29T14:59:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49697f84:::benchmark_data_smithi194_196924_object121932:head# 2024-04-29T14:59:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4969e932:::benchmark_data_smithi194_196924_object140806:head# 2024-04-29T14:59:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:496a3291:::benchmark_data_smithi194_196924_object127896:head# 2024-04-29T14:59:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:496b0c99:::benchmark_data_smithi194_196924_object114376:head# 2024-04-29T14:59:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:496d0ca9:::benchmark_data_smithi194_196924_object114517:head# 2024-04-29T14:59:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:496d1ed5:::benchmark_data_smithi194_196924_object131256:head# 2024-04-29T14:59:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:496e4f58:::benchmark_data_smithi194_196924_object138292:head# 2024-04-29T14:59:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:496f5c93:::benchmark_data_smithi194_196924_object152702:head# 2024-04-29T14:59:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49739a19:::benchmark_data_smithi194_196924_object149604:head# 2024-04-29T14:59:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4973fda7:::benchmark_data_smithi194_196924_object96764:head# 2024-04-29T14:59:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:497416c7:::benchmark_data_smithi194_196924_object113023:head# 2024-04-29T14:59:53.770 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49746831:::benchmark_data_smithi194_196924_object111384:head# 2024-04-29T14:59:53.770 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49749e34:::benchmark_data_smithi194_196924_object113094:head# 2024-04-29T14:59:53.771 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4974da10:::benchmark_data_smithi194_196924_object157006:head# 2024-04-29T14:59:53.771 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4974fb13:::benchmark_data_smithi194_196924_object134746:head# 2024-04-29T14:59:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4976ef16:::benchmark_data_smithi194_196924_object99163:head# 2024-04-29T14:59:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4978fec8:::benchmark_data_smithi194_196924_object131058:head# 2024-04-29T14:59:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:497ea2d4:::benchmark_data_smithi194_196924_object136424:head# 2024-04-29T14:59:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:497ed9a4:::benchmark_data_smithi194_196924_object123090:head# 2024-04-29T14:59:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498185e6:::benchmark_data_smithi194_196924_object96590:head# 2024-04-29T14:59:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498270b3:::benchmark_data_smithi194_196924_object123742:head# 2024-04-29T14:59:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4983d037:::benchmark_data_smithi194_196924_object148456:head# 2024-04-29T14:59:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4984e19b:::benchmark_data_smithi194_196924_object104569:head# 2024-04-29T14:59:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4986e0a8:::benchmark_data_smithi194_196924_object97157:head# 2024-04-29T14:59:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4988b71e:::benchmark_data_smithi194_196924_object139113:head# 2024-04-29T14:59:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49891123:::benchmark_data_smithi194_196924_object125068:head# 2024-04-29T14:59:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4989133c:::benchmark_data_smithi194_196924_object109890:head# 2024-04-29T14:59:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49899b11:::benchmark_data_smithi194_196924_object152293:head# 2024-04-29T14:59:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4989baed:::benchmark_data_smithi194_196924_object101268:head# 2024-04-29T14:59:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498bff9b:::benchmark_data_smithi194_196924_object135232:head# 2024-04-29T14:59:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498c94ad:::benchmark_data_smithi194_196924_object140353:head# 2024-04-29T14:59:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498d2936:::benchmark_data_smithi194_196924_object103385:head# 2024-04-29T14:59:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498d51c4:::benchmark_data_smithi194_196924_object100185:head# 2024-04-29T14:59:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498dd66b:::benchmark_data_smithi194_196924_object100879:head# 2024-04-29T14:59:53.781 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:498f189f:::benchmark_data_smithi194_196924_object138929:head# 2024-04-29T14:59:53.781 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4990d614:::benchmark_data_smithi194_196924_object114178:head# 2024-04-29T14:59:53.782 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4990db49:::benchmark_data_smithi194_196924_object104115:head# 2024-04-29T14:59:53.782 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49923700:::benchmark_data_smithi194_196924_object110095:head# 2024-04-29T14:59:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499274f9:::benchmark_data_smithi194_196924_object131096:head# 2024-04-29T14:59:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49939454:::benchmark_data_smithi194_196924_object127122:head# 2024-04-29T14:59:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4993f09b:::benchmark_data_smithi194_196924_object146535:head# 2024-04-29T14:59:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4994264c:::benchmark_data_smithi194_196924_object115017:head# 2024-04-29T14:59:53.785 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49964fab:::benchmark_data_smithi194_196924_object156468:head# 2024-04-29T14:59:53.785 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499674ba:::benchmark_data_smithi194_196924_object122052:head# 2024-04-29T14:59:53.785 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4998b9d1:::benchmark_data_smithi194_196924_object140951:head# 2024-04-29T14:59:53.786 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4998bd08:::benchmark_data_smithi194_196924_object122869:head# 2024-04-29T14:59:53.787 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499a8805:::benchmark_data_smithi194_196924_object101992:head# 2024-04-29T14:59:53.787 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499b9f13:::benchmark_data_smithi194_196924_object128335:head# 2024-04-29T14:59:53.787 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499d398d:::benchmark_data_smithi194_196924_object110947:head# 2024-04-29T14:59:53.788 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499d50f4:::benchmark_data_smithi194_196924_object125889:head# 2024-04-29T14:59:53.788 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499e7fa2:::benchmark_data_smithi194_196924_object148644:head# 2024-04-29T14:59:53.789 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:499fe4ce:::benchmark_data_smithi194_196924_object118240:head# 2024-04-29T14:59:53.789 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49a06bc5:::benchmark_data_smithi194_196924_object124909:head# 2024-04-29T14:59:53.790 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49a332c8:::benchmark_data_smithi194_196924_object121125:head# 2024-04-29T14:59:53.791 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:53.791 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49a44a95:::benchmark_data_smithi194_196924_object134736:head# 2024-04-29T14:59:53.792 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49a5e80e:::benchmark_data_smithi194_196924_object109663:head# 2024-04-29T14:59:53.792 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49a61da9:::benchmark_data_smithi194_196924_object108099:head# 2024-04-29T14:59:53.793 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49a89238:::benchmark_data_smithi194_196924_object98794:head# 2024-04-29T14:59:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:53.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:53.795 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49aa266e:::benchmark_data_smithi194_196924_object159436:head# 2024-04-29T14:59:53.795 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49aa8b32:::benchmark_data_smithi194_196924_object103850:head# 2024-04-29T14:59:53.796 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ab9aaa:::benchmark_data_smithi194_196924_object98306:head# 2024-04-29T14:59:53.796 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ac66ba:::benchmark_data_smithi194_196924_object141089:head# 2024-04-29T14:59:53.797 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ac8d27:::benchmark_data_smithi194_196924_object138514:head# 2024-04-29T14:59:53.797 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ad632f:::benchmark_data_smithi194_196924_object122113:head# 2024-04-29T14:59:53.797 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ada826:::benchmark_data_smithi194_196924_object151391:head# 2024-04-29T14:59:53.798 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49afe27b:::benchmark_data_smithi194_196924_object154595:head# 2024-04-29T14:59:53.798 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b1f3bb:::benchmark_data_smithi194_196924_object97819:head# 2024-04-29T14:59:53.799 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b3c380:::benchmark_data_smithi194_196924_object146196:head# 2024-04-29T14:59:53.799 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b442d8:::benchmark_data_smithi194_196924_object142862:head# 2024-04-29T14:59:53.800 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b45e14:::benchmark_data_smithi194_196924_object156209:head# 2024-04-29T14:59:53.800 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b4b956:::benchmark_data_smithi194_196924_object132985:head# 2024-04-29T14:59:53.800 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b69a5f:::benchmark_data_smithi194_196924_object123776:head# 2024-04-29T14:59:53.801 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b6e4e4:::benchmark_data_smithi194_196924_object145843:head# 2024-04-29T14:59:53.801 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b74674:::benchmark_data_smithi194_196924_object113077:head# 2024-04-29T14:59:53.802 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b76645:::benchmark_data_smithi194_196924_object121722:head# 2024-04-29T14:59:53.802 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b7bbac:::benchmark_data_smithi194_196924_object95645:head# 2024-04-29T14:59:53.802 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b859e0:::benchmark_data_smithi194_196924_object145251:head# 2024-04-29T14:59:53.803 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49b8f2a1:::benchmark_data_smithi194_196924_object96982:head# 2024-04-29T14:59:53.803 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ba0372:::benchmark_data_smithi194_196924_object151325:head# 2024-04-29T14:59:53.804 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bb4fa0:::benchmark_data_smithi194_196924_object146003:head# 2024-04-29T14:59:53.804 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bbe0c0:::benchmark_data_smithi194_196924_object149027:head# 2024-04-29T14:59:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bd042c:::benchmark_data_smithi194_196924_object127066:head# 2024-04-29T14:59:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bd1619:::benchmark_data_smithi194_196924_object105823:head# 2024-04-29T14:59:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bd1dcb:::benchmark_data_smithi194_196924_object128427:head# 2024-04-29T14:59:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49be0c34:::benchmark_data_smithi194_196924_object135187:head# 2024-04-29T14:59:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bf4595:::benchmark_data_smithi194_196924_object120493:head# 2024-04-29T14:59:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bf8fed:::benchmark_data_smithi194_196924_object150072:head# 2024-04-29T14:59:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49bfca29:::benchmark_data_smithi194_196924_object102931:head# 2024-04-29T14:59:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49c29316:::benchmark_data_smithi194_196924_object106834:head# 2024-04-29T14:59:53.809 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49c36cfe:::benchmark_data_smithi194_196924_object154981:head# 2024-04-29T14:59:53.809 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49c38038:::benchmark_data_smithi194_196924_object157136:head# 2024-04-29T14:59:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49c44f13:::benchmark_data_smithi194_196924_object132034:head# 2024-04-29T14:59:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49c90ee9:::benchmark_data_smithi194_196924_object108520:head# 2024-04-29T14:59:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ca1203:::benchmark_data_smithi194_196924_object114683:head# 2024-04-29T14:59:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ca28ef:::benchmark_data_smithi194_196924_object114001:head# 2024-04-29T14:59:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ca67bb:::benchmark_data_smithi194_196924_object112635:head# 2024-04-29T14:59:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ca9deb:::benchmark_data_smithi194_196924_object126063:head# 2024-04-29T14:59:53.813 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49cb7d3f:::benchmark_data_smithi194_196924_object132121:head# 2024-04-29T14:59:53.813 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ccf0fe:::benchmark_data_smithi194_196924_object119932:head# 2024-04-29T14:59:53.813 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49cf5ecc:::benchmark_data_smithi194_196924_object104074:head# 2024-04-29T14:59:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49cfc46c:::benchmark_data_smithi194_196924_object151742:head# 2024-04-29T14:59:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49d06520:::benchmark_data_smithi194_196924_object103081:head# 2024-04-29T14:59:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49d0668d:::benchmark_data_smithi194_196924_object135876:head# 2024-04-29T14:59:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49d195da:::benchmark_data_smithi194_196924_object145769:head# 2024-04-29T14:59:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49d44ce4:::benchmark_data_smithi194_196924_object141139:head# 2024-04-29T14:59:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49d97942:::benchmark_data_smithi194_196924_object151595:head# 2024-04-29T14:59:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49da337b:::benchmark_data_smithi194_196924_object129407:head# 2024-04-29T14:59:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49db0a09:::benchmark_data_smithi194_196924_object101851:head# 2024-04-29T14:59:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49db2693:::benchmark_data_smithi194_196924_object110141:head# 2024-04-29T14:59:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49dd3501:::benchmark_data_smithi194_196924_object138423:head# 2024-04-29T14:59:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49df40cf:::benchmark_data_smithi194_196924_object140285:head# 2024-04-29T14:59:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e0695d:::benchmark_data_smithi194_196924_object145331:head# 2024-04-29T14:59:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e15d1e:::benchmark_data_smithi194_196924_object157717:head# 2024-04-29T14:59:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e1c1df:::benchmark_data_smithi194_196924_object122479:head# 2024-04-29T14:59:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e31ecd:::benchmark_data_smithi194_196924_object141495:head# 2024-04-29T14:59:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e3b489:::benchmark_data_smithi194_196924_object136072:head# 2024-04-29T14:59:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e4b124:::benchmark_data_smithi194_196924_object156987:head# 2024-04-29T14:59:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e5d4f3:::benchmark_data_smithi194_196924_object150027:head# 2024-04-29T14:59:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e5dca3:::benchmark_data_smithi194_196924_object131684:head# 2024-04-29T14:59:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49e6369c:::benchmark_data_smithi194_196924_object112378:head# 2024-04-29T14:59:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ebeb1c:::benchmark_data_smithi194_196924_object104480:head# 2024-04-29T14:59:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ed5f04:::benchmark_data_smithi194_196924_object147842:head# 2024-04-29T14:59:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49eff11b:::benchmark_data_smithi194_196924_object126886:head# 2024-04-29T14:59:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f051a2:::benchmark_data_smithi194_196924_object122042:head# 2024-04-29T14:59:53.824 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f0bda4:::benchmark_data_smithi194_196924_object156040:head# 2024-04-29T14:59:53.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f21b73:::benchmark_data_smithi194_196924_object146673:head# 2024-04-29T14:59:53.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f2ba7e:::benchmark_data_smithi194_196924_object154371:head# 2024-04-29T14:59:53.825 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f4dabf:::benchmark_data_smithi194_196924_object115712:head# 2024-04-29T14:59:53.826 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f769c1:::benchmark_data_smithi194_196924_object151478:head# 2024-04-29T14:59:53.826 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f8398a:::benchmark_data_smithi194_196924_object133439:head# 2024-04-29T14:59:53.827 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49f8e3b7:::benchmark_data_smithi194_196924_object157871:head# 2024-04-29T14:59:53.827 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49fa0cfd:::benchmark_data_smithi194_196924_object122862:head# 2024-04-29T14:59:53.828 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49fb0e7b:::benchmark_data_smithi194_196924_object150858:head# 2024-04-29T14:59:53.828 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49feddb0:::benchmark_data_smithi194_196924_object104828:head# 2024-04-29T14:59:53.829 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:49ffe727:::benchmark_data_smithi194_196924_object151455:head# 2024-04-29T14:59:53.829 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0178aa:::benchmark_data_smithi194_196924_object96711:head# 2024-04-29T14:59:53.829 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a020566:::benchmark_data_smithi194_196924_object102596:head# 2024-04-29T14:59:53.830 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a024747:::benchmark_data_smithi194_196924_object104039:head# 2024-04-29T14:59:53.830 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0278a0:::benchmark_data_smithi194_196924_object116496:head# 2024-04-29T14:59:53.831 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a02882d:::benchmark_data_smithi194_196924_object152107:head# 2024-04-29T14:59:53.831 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a050a95:::benchmark_data_smithi194_196924_object140783:head# 2024-04-29T14:59:53.832 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a06c351:::benchmark_data_smithi194_196924_object106522:head# 2024-04-29T14:59:53.832 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a075a26:::benchmark_data_smithi194_196924_object136252:head# 2024-04-29T14:59:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a088fcb:::benchmark_data_smithi194_196924_object156448:head# 2024-04-29T14:59:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a090325:::benchmark_data_smithi194_196924_object133675:head# 2024-04-29T14:59:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a09aee6:::benchmark_data_smithi194_196924_object144590:head# 2024-04-29T14:59:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0a6407:::benchmark_data_smithi194_196924_object159773:head# 2024-04-29T14:59:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0b19c4:::benchmark_data_smithi194_196924_object122173:head# 2024-04-29T14:59:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0b3c1a:::benchmark_data_smithi194_196924_object142744:head# 2024-04-29T14:59:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0b4c4e:::benchmark_data_smithi194_196924_object96121:head# 2024-04-29T14:59:53.836 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0b5cda:::benchmark_data_smithi194_196924_object135182:head# 2024-04-29T14:59:53.836 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0c3c0a:::benchmark_data_smithi194_196924_object155063:head# 2024-04-29T14:59:53.837 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0c5886:::benchmark_data_smithi194_196924_object111098:head# 2024-04-29T14:59:53.837 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0d13df:::benchmark_data_smithi194_196924_object119150:head# 2024-04-29T14:59:53.838 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0d39b0:::benchmark_data_smithi194_196924_object145683:head# 2024-04-29T14:59:53.838 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0eff1a:::benchmark_data_smithi194_196924_object121353:head# 2024-04-29T14:59:53.839 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0f1c11:::benchmark_data_smithi194_196924_object133596:head# 2024-04-29T14:59:53.839 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0f49ce:::benchmark_data_smithi194_196924_object130816:head# 2024-04-29T14:59:53.840 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0f5aaf:::benchmark_data_smithi194_196924_object157426:head# 2024-04-29T14:59:53.840 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a0f69d8:::benchmark_data_smithi194_196924_object146120:head# 2024-04-29T14:59:53.840 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a102a24:::benchmark_data_smithi194_196924_object105260:head# 2024-04-29T14:59:53.841 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a10be3d:::benchmark_data_smithi194_196924_object117054:head# 2024-04-29T14:59:53.842 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a125b06:::benchmark_data_smithi194_196924_object103593:head# 2024-04-29T14:59:53.842 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a130084:::benchmark_data_smithi194_196924_object138322:head# 2024-04-29T14:59:53.842 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a130415:::benchmark_data_smithi194_196924_object133673:head# 2024-04-29T14:59:53.843 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1424d9:::benchmark_data_smithi194_196924_object146971:head# 2024-04-29T14:59:53.843 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1457da:::benchmark_data_smithi194_196924_object141915:head# 2024-04-29T14:59:53.844 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a164067:::benchmark_data_smithi194_196924_object141490:head# 2024-04-29T14:59:53.844 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a16940b:::benchmark_data_smithi194_196924_object153443:head# 2024-04-29T14:59:53.845 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a19bf9d:::benchmark_data_smithi194_196924_object113356:head# 2024-04-29T14:59:53.845 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1a851b:::benchmark_data_smithi194_196924_object102600:head# 2024-04-29T14:59:53.845 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1aa372:::benchmark_data_smithi194_196924_object134723:head# 2024-04-29T14:59:53.846 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1acec9:::benchmark_data_smithi194_196924_object133888:head# 2024-04-29T14:59:53.846 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1b8a22:::benchmark_data_smithi194_196924_object117123:head# 2024-04-29T14:59:53.847 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1bbab7:::benchmark_data_smithi194_196924_object115727:head# 2024-04-29T14:59:53.847 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1e57cd:::benchmark_data_smithi194_196924_object158929:head# 2024-04-29T14:59:53.847 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1e8704:::benchmark_data_smithi194_196924_object126362:head# 2024-04-29T14:59:53.848 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1e9f6b:::benchmark_data_smithi194_196924_object123299:head# 2024-04-29T14:59:53.848 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a1f31da:::benchmark_data_smithi194_196924_object141770:head# 2024-04-29T14:59:53.849 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a207c61:::benchmark_data_smithi194_196924_object152675:head# 2024-04-29T14:59:53.849 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a20eb15:::benchmark_data_smithi194_196924_object138186:head# 2024-04-29T14:59:53.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a2203f3:::benchmark_data_smithi194_196924_object123055:head# 2024-04-29T14:59:53.850 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a220938:::benchmark_data_smithi194_196924_object115536:head# 2024-04-29T14:59:53.851 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a2365ed:::benchmark_data_smithi194_196924_object99564:head# 2024-04-29T14:59:53.851 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a23c807:::benchmark_data_smithi194_196924_object108830:head# 2024-04-29T14:59:53.852 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a23f982:::benchmark_data_smithi194_196924_object124291:head# 2024-04-29T14:59:53.852 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a25b8f3:::benchmark_data_smithi194_196924_object130178:head# 2024-04-29T14:59:53.853 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a260e7d:::benchmark_data_smithi194_196924_object102058:head# 2024-04-29T14:59:53.853 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a28450d:::benchmark_data_smithi194_196924_object145078:head# 2024-04-29T14:59:53.854 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a2bbe79:::benchmark_data_smithi194_196924_object104066:head# 2024-04-29T14:59:53.854 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a2ce73c:::benchmark_data_smithi194_196924_object149449:head# 2024-04-29T14:59:53.854 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a2f1a08:::benchmark_data_smithi194_196924_object128580:head# 2024-04-29T14:59:53.855 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a30aeb3:::benchmark_data_smithi194_196924_object103116:head# 2024-04-29T14:59:53.855 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a310987:::benchmark_data_smithi194_196924_object97985:head# 2024-04-29T14:59:53.856 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3201a7:::benchmark_data_smithi194_196924_object118717:head# 2024-04-29T14:59:53.856 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a338abd:::benchmark_data_smithi194_196924_object137797:head# 2024-04-29T14:59:53.856 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3431f5:::benchmark_data_smithi194_196924_object133777:head# 2024-04-29T14:59:53.857 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a35eb38:::benchmark_data_smithi194_196924_object109951:head# 2024-04-29T14:59:53.857 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a376ecd:::benchmark_data_smithi194_196924_object98767:head# 2024-04-29T14:59:53.858 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a37e94e:::benchmark_data_smithi194_196924_object114945:head# 2024-04-29T14:59:53.858 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a38b619:::benchmark_data_smithi194_196924_object105113:head# 2024-04-29T14:59:53.858 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a39ef51:::benchmark_data_smithi194_196924_object120358:head# 2024-04-29T14:59:53.859 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3a580a:::benchmark_data_smithi194_196924_object99739:head# 2024-04-29T14:59:53.859 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3b179e:::benchmark_data_smithi194_196924_object115103:head# 2024-04-29T14:59:53.860 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3d3b4f:::benchmark_data_smithi194_196924_object102635:head# 2024-04-29T14:59:53.860 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3d5a24:::benchmark_data_smithi194_196924_object146425:head# 2024-04-29T14:59:53.861 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3eb348:::benchmark_data_smithi194_196924_object124938:head# 2024-04-29T14:59:53.861 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a3f5dc7:::benchmark_data_smithi194_196924_object140389:head# 2024-04-29T14:59:53.862 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a41dc42:::benchmark_data_smithi194_196924_object134219:head# 2024-04-29T14:59:53.862 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a42086e:::benchmark_data_smithi194_196924_object99463:head# 2024-04-29T14:59:53.863 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a420cdf:::benchmark_data_smithi194_196924_object136418:head# 2024-04-29T14:59:53.863 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a42d36d:::benchmark_data_smithi194_196924_object148149:head# 2024-04-29T14:59:53.864 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a435ae6:::benchmark_data_smithi194_196924_object135484:head# 2024-04-29T14:59:53.864 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a450570:::benchmark_data_smithi194_196924_object114234:head# 2024-04-29T14:59:53.864 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a46688c:::benchmark_data_smithi194_196924_object138574:head# 2024-04-29T14:59:53.865 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4671f4:::benchmark_data_smithi194_196924_object110556:head# 2024-04-29T14:59:53.865 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a46adf8:::benchmark_data_smithi194_196924_object110059:head# 2024-04-29T14:59:53.866 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a48f3a1:::benchmark_data_smithi194_196924_object116691:head# 2024-04-29T14:59:53.866 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4a17f9:::benchmark_data_smithi194_196924_object120914:head# 2024-04-29T14:59:53.867 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4a624c:::benchmark_data_smithi194_196924_object125199:head# 2024-04-29T14:59:53.867 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4b1643:::benchmark_data_smithi194_196924_object113465:head# 2024-04-29T14:59:53.868 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4bf4e3:::benchmark_data_smithi194_196924_object146116:head# 2024-04-29T14:59:53.868 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4c4f5d:::benchmark_data_smithi194_196924_object152707:head# 2024-04-29T14:59:53.869 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4cb9e1:::benchmark_data_smithi194_196924_object148671:head# 2024-04-29T14:59:53.869 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4e7e34:::benchmark_data_smithi194_196924_object132978:head# 2024-04-29T14:59:53.870 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a4e84f6:::benchmark_data_smithi194_196924_object97121:head# 2024-04-29T14:59:53.870 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a506cc8:::benchmark_data_smithi194_196924_object151117:head# 2024-04-29T14:59:53.870 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a50908c:::benchmark_data_smithi194_196924_object110819:head# 2024-04-29T14:59:53.871 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a50a715:::benchmark_data_smithi194_196924_object111280:head# 2024-04-29T14:59:53.871 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a50edf5:::benchmark_data_smithi194_196924_object113869:head# 2024-04-29T14:59:53.872 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a519d02:::benchmark_data_smithi194_196924_object103489:head# 2024-04-29T14:59:53.872 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a51d541:::benchmark_data_smithi194_196924_object128957:head# 2024-04-29T14:59:53.872 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a530037:::benchmark_data_smithi194_196924_object127253:head# 2024-04-29T14:59:53.873 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a560ceb:::benchmark_data_smithi194_196924_object157600:head# 2024-04-29T14:59:53.873 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5683cb:::benchmark_data_smithi194_196924_object96667:head# 2024-04-29T14:59:53.873 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a59d8cc:::benchmark_data_smithi194_196924_object152173:head# 2024-04-29T14:59:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5ade6b:::benchmark_data_smithi194_196924_object111228:head# 2024-04-29T14:59:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5b4c37:::benchmark_data_smithi194_196924_object137848:head# 2024-04-29T14:59:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5bb606:::benchmark_data_smithi194_196924_object119569:head# 2024-04-29T14:59:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5c0eff:::benchmark_data_smithi194_196924_object123794:head# 2024-04-29T14:59:53.876 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5cc67a:::benchmark_data_smithi194_196924_object95946:head# 2024-04-29T14:59:53.876 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5d2c8c:::benchmark_data_smithi194_196924_object123654:head# 2024-04-29T14:59:53.876 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5d74c1:::benchmark_data_smithi194_196924_object151307:head# 2024-04-29T14:59:53.877 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a5df602:::benchmark_data_smithi194_196924_object116427:head# 2024-04-29T14:59:53.877 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a62356a:::benchmark_data_smithi194_196924_object153110:head# 2024-04-29T14:59:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a632246:::benchmark_data_smithi194_196924_object158081:head# 2024-04-29T14:59:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a64cead:::benchmark_data_smithi194_196924_object97938:head# 2024-04-29T14:59:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a64eea9:::benchmark_data_smithi194_196924_object157474:head# 2024-04-29T14:59:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a658749:::benchmark_data_smithi194_196924_object149880:head# 2024-04-29T14:59:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a676f35:::benchmark_data_smithi194_196924_object103296:head# 2024-04-29T14:59:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a677d1b:::benchmark_data_smithi194_196924_object120166:head# 2024-04-29T14:59:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a67d5fe:::benchmark_data_smithi194_196924_object112788:head# 2024-04-29T14:59:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a67f7d4:::benchmark_data_smithi194_196924_object107290:head# 2024-04-29T14:59:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a699e8e:::benchmark_data_smithi194_196924_object149613:head# 2024-04-29T14:59:53.882 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a6ba52e:::benchmark_data_smithi194_196924_object99434:head# 2024-04-29T14:59:53.882 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a6bec77:::benchmark_data_smithi194_196924_object135119:head# 2024-04-29T14:59:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a6ef6f5:::benchmark_data_smithi194_196924_object115763:head# 2024-04-29T14:59:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a6f37e8:::benchmark_data_smithi194_196924_object139930:head# 2024-04-29T14:59:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a6f45ad:::benchmark_data_smithi194_196924_object155380:head# 2024-04-29T14:59:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a6fd5ba:::benchmark_data_smithi194_196924_object120643:head# 2024-04-29T14:59:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a709d15:::benchmark_data_smithi194_196924_object116575:head# 2024-04-29T14:59:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a735129:::benchmark_data_smithi194_196924_object108735:head# 2024-04-29T14:59:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a754117:::benchmark_data_smithi194_196924_object136653:head# 2024-04-29T14:59:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a75987f:::benchmark_data_smithi194_196924_object98493:head# 2024-04-29T14:59:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a75a97a:::benchmark_data_smithi194_196924_object132956:head# 2024-04-29T14:59:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a762a6b:::benchmark_data_smithi194_196924_object154942:head# 2024-04-29T14:59:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a775adf:::benchmark_data_smithi194_196924_object101957:head# 2024-04-29T14:59:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a779a15:::benchmark_data_smithi194_196924_object117456:head# 2024-04-29T14:59:53.888 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a77eb67:::benchmark_data_smithi194_196924_object143714:head# 2024-04-29T14:59:53.888 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a785e25:::benchmark_data_smithi194_196924_object138120:head# 2024-04-29T14:59:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a7ae1e3:::benchmark_data_smithi194_196924_object139122:head# 2024-04-29T14:59:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a7b10c5:::benchmark_data_smithi194_196924_object112888:head# 2024-04-29T14:59:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a7d41b1:::benchmark_data_smithi194_196924_object125766:head# 2024-04-29T14:59:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a7dd0a0:::benchmark_data_smithi194_196924_object132418:head# 2024-04-29T14:59:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a7ea6f5:::benchmark_data_smithi194_196924_object146661:head# 2024-04-29T14:59:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a7f9e61:::benchmark_data_smithi194_196924_object151628:head# 2024-04-29T14:59:53.892 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8277a9:::benchmark_data_smithi194_196924_object122815:head# 2024-04-29T14:59:53.892 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a83232b:::benchmark_data_smithi194_196924_object120257:head# 2024-04-29T14:59:53.892 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a837b96:::benchmark_data_smithi194_196924_object96358:head# 2024-04-29T14:59:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8386ff:::benchmark_data_smithi194_196924_object138975:head# 2024-04-29T14:59:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a83a606:::benchmark_data_smithi194_196924_object144647:head# 2024-04-29T14:59:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a84ee2e:::benchmark_data_smithi194_196924_object149549:head# 2024-04-29T14:59:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a85d79b:::benchmark_data_smithi194_196924_object157928:head# 2024-04-29T14:59:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8831a5:::benchmark_data_smithi194_196924_object127260:head# 2024-04-29T14:59:53.895 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8834e5:::benchmark_data_smithi194_196924_object112062:head# 2024-04-29T14:59:53.895 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8a818b:::benchmark_data_smithi194_196924_object142756:head# 2024-04-29T14:59:53.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T14:59:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8d1509:::benchmark_data_smithi194_196924_object138804:head# 2024-04-29T14:59:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8d361d:::benchmark_data_smithi194_196924_object156948:head# 2024-04-29T14:59:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8e5c39:::benchmark_data_smithi194_196924_object140728:head# 2024-04-29T14:59:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a8e75e1:::benchmark_data_smithi194_196924_object98817:head# 2024-04-29T14:59:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9040aa:::benchmark_data_smithi194_196924_object113216:head# 2024-04-29T14:59:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9063c9:::benchmark_data_smithi194_196924_object108334:head# 2024-04-29T14:59:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a926b75:::benchmark_data_smithi194_196924_object115505:head# 2024-04-29T14:59:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a927349:::benchmark_data_smithi194_196924_object145684:head# 2024-04-29T14:59:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a92813d:::benchmark_data_smithi194_196924_object104460:head# 2024-04-29T14:59:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a94a66f:::benchmark_data_smithi194_196924_object139960:head# 2024-04-29T14:59:53.900 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a966109:::benchmark_data_smithi194_196924_object99879:head# 2024-04-29T14:59:53.900 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a97fd08:::benchmark_data_smithi194_196924_object122784:head# 2024-04-29T14:59:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a988006:::benchmark_data_smithi194_196924_object141204:head# 2024-04-29T14:59:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a989987:::benchmark_data_smithi194_196924_object121511:head# 2024-04-29T14:59:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a990411:::benchmark_data_smithi194_196924_object114210:head# 2024-04-29T14:59:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a99ab15:::benchmark_data_smithi194_196924_object113835:head# 2024-04-29T14:59:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9a24a0:::benchmark_data_smithi194_196924_object132389:head# 2024-04-29T14:59:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9b56e9:::benchmark_data_smithi194_196924_object116895:head# 2024-04-29T14:59:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9b59fd:::benchmark_data_smithi194_196924_object98716:head# 2024-04-29T14:59:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9dc289:::benchmark_data_smithi194_196924_object155279:head# 2024-04-29T14:59:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9e28b4:::benchmark_data_smithi194_196924_object127955:head# 2024-04-29T14:59:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9ecfa3:::benchmark_data_smithi194_196924_object125438:head# 2024-04-29T14:59:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4a9fab59:::benchmark_data_smithi194_196924_object112950:head# 2024-04-29T14:59:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa0c2c3:::benchmark_data_smithi194_196924_object106875:head# 2024-04-29T14:59:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa0daaf:::benchmark_data_smithi194_196924_object99701:head# 2024-04-29T14:59:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa13a07:::benchmark_data_smithi194_196924_object96477:head# 2024-04-29T14:59:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa1d90d:::benchmark_data_smithi194_196924_object102516:head# 2024-04-29T14:59:53.908 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa2390a:::benchmark_data_smithi194_196924_object108331:head# 2024-04-29T14:59:53.908 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa27b54:::benchmark_data_smithi194_196924_object147867:head# 2024-04-29T14:59:53.908 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa46a66:::benchmark_data_smithi194_196924_object145563:head# 2024-04-29T14:59:53.909 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa5a272:::benchmark_data_smithi194_196924_object99762:head# 2024-04-29T14:59:53.909 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa5ce06:::benchmark_data_smithi194_196924_object132669:head# 2024-04-29T14:59:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aa7d053:::benchmark_data_smithi194_196924_object154397:head# 2024-04-29T14:59:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aab8690:::benchmark_data_smithi194_196924_object141124:head# 2024-04-29T14:59:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aabf827:::benchmark_data_smithi194_196924_object152187:head# 2024-04-29T14:59:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aad4ace:::benchmark_data_smithi194_196924_object116191:head# 2024-04-29T14:59:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aaef6ba:::benchmark_data_smithi194_196924_object132668:head# 2024-04-29T14:59:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aaf148e:::benchmark_data_smithi194_196924_object119742:head# 2024-04-29T14:59:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aaf6a3a:::benchmark_data_smithi194_196924_object108056:head# 2024-04-29T14:59:53.913 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab0de37:::benchmark_data_smithi194_196924_object109157:head# 2024-04-29T14:59:53.913 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab16fca:::benchmark_data_smithi194_196924_object115490:head# 2024-04-29T14:59:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab2d506:::benchmark_data_smithi194_196924_object119286:head# 2024-04-29T14:59:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab41411:::benchmark_data_smithi194_196924_object136839:head# 2024-04-29T14:59:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab5a8f4:::benchmark_data_smithi194_196924_object124514:head# 2024-04-29T14:59:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab7b053:::benchmark_data_smithi194_196924_object121302:head# 2024-04-29T14:59:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab7db70:::benchmark_data_smithi194_196924_object154040:head# 2024-04-29T14:59:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab7de5e:::benchmark_data_smithi194_196924_object146331:head# 2024-04-29T14:59:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab7e7dd:::benchmark_data_smithi194_196924_object145266:head# 2024-04-29T14:59:53.917 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ab7faa4:::benchmark_data_smithi194_196924_object143122:head# 2024-04-29T14:59:53.917 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aba024e:::benchmark_data_smithi194_196924_object96933:head# 2024-04-29T14:59:53.917 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4abb16a0:::benchmark_data_smithi194_196924_object97175:head# 2024-04-29T14:59:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4abbd993:::benchmark_data_smithi194_196924_object108792:head# 2024-04-29T14:59:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4abe7eca:::benchmark_data_smithi194_196924_object154122:head# 2024-04-29T14:59:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ac019cc:::benchmark_data_smithi194_196924_object153727:head# 2024-04-29T14:59:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ac05b76:::benchmark_data_smithi194_196924_object152640:head# 2024-04-29T14:59:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ac06b7f:::benchmark_data_smithi194_196924_object104858:head# 2024-04-29T14:59:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ac826dd:::benchmark_data_smithi194_196924_object159321:head# 2024-04-29T14:59:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ac8fe52:::benchmark_data_smithi194_196924_object153225:head# 2024-04-29T14:59:53.921 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4accaa16:::benchmark_data_smithi194_196924_object116681:head# 2024-04-29T14:59:53.921 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4acdb53d:::benchmark_data_smithi194_196924_object150757:head# 2024-04-29T14:59:53.922 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aced5c8:::benchmark_data_smithi194_196924_object97238:head# 2024-04-29T14:59:53.922 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ad030f4:::benchmark_data_smithi194_196924_object110090:head# 2024-04-29T14:59:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ad4508e:::benchmark_data_smithi194_196924_object111222:head# 2024-04-29T14:59:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ad59305:::benchmark_data_smithi194_196924_object150407:head# 2024-04-29T14:59:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ad643af:::benchmark_data_smithi194_196924_object157173:head# 2024-04-29T14:59:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ad6d1da:::benchmark_data_smithi194_196924_object127466:head# 2024-04-29T14:59:53.925 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ad74bff:::benchmark_data_smithi194_196924_object133775:head# 2024-04-29T14:59:53.925 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ad98a76:::benchmark_data_smithi194_196924_object126652:head# 2024-04-29T14:59:53.926 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4adbedef:::benchmark_data_smithi194_196924_object137106:head# 2024-04-29T14:59:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4adc030a:::benchmark_data_smithi194_196924_object135888:head# 2024-04-29T14:59:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4add2a56:::benchmark_data_smithi194_196924_object154798:head# 2024-04-29T14:59:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4add3504:::benchmark_data_smithi194_196924_object131101:head# 2024-04-29T14:59:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4addea79:::benchmark_data_smithi194_196924_object139541:head# 2024-04-29T14:59:53.929 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:53.929 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae13ea9:::benchmark_data_smithi194_196924_object152197:head# 2024-04-29T14:59:53.929 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae1dc75:::benchmark_data_smithi194_196924_object115603:head# 2024-04-29T14:59:53.929 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae270aa:::benchmark_data_smithi194_196924_object97874:head# 2024-04-29T14:59:53.930 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae41474:::benchmark_data_smithi194_196924_object116053:head# 2024-04-29T14:59:53.930 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae5b7e5:::benchmark_data_smithi194_196924_object124584:head# 2024-04-29T14:59:53.931 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae7a39b:::benchmark_data_smithi194_196924_object143827:head# 2024-04-29T14:59:53.931 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae85547:::benchmark_data_smithi194_196924_object124610:head# 2024-04-29T14:59:53.932 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ae91748:::benchmark_data_smithi194_196924_object119099:head# 2024-04-29T14:59:53.932 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aeb257c:::benchmark_data_smithi194_196924_object100571:head# 2024-04-29T14:59:53.933 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aeb6387:::benchmark_data_smithi194_196924_object106028:head# 2024-04-29T14:59:53.933 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aec448f:::benchmark_data_smithi194_196924_object130426:head# 2024-04-29T14:59:53.933 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aecf7cb:::benchmark_data_smithi194_196924_object108222:head# 2024-04-29T14:59:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aedacdc:::benchmark_data_smithi194_196924_object97125:head# 2024-04-29T14:59:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aedb1de:::benchmark_data_smithi194_196924_object109247:head# 2024-04-29T14:59:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aedf2df:::benchmark_data_smithi194_196924_object111530:head# 2024-04-29T14:59:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aee111c:::benchmark_data_smithi194_196924_object95399:head# 2024-04-29T14:59:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aef0362:::benchmark_data_smithi194_196924_object146973:head# 2024-04-29T14:59:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aef489c:::benchmark_data_smithi194_196924_object152328:head# 2024-04-29T14:59:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aef9f67:::benchmark_data_smithi194_196924_object148729:head# 2024-04-29T14:59:53.937 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af186ef:::benchmark_data_smithi194_196924_object144904:head# 2024-04-29T14:59:53.937 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af1e1d5:::benchmark_data_smithi194_196924_object152452:head# 2024-04-29T14:59:53.938 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af2ca5f:::benchmark_data_smithi194_196924_object110444:head# 2024-04-29T14:59:53.938 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af2f322:::benchmark_data_smithi194_196924_object132712:head# 2024-04-29T14:59:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af436cf:::benchmark_data_smithi194_196924_object141914:head# 2024-04-29T14:59:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af48ab6:::benchmark_data_smithi194_196924_object104573:head# 2024-04-29T14:59:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af589c2:::benchmark_data_smithi194_196924_object117029:head# 2024-04-29T14:59:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af5da7d:::benchmark_data_smithi194_196924_object122377:head# 2024-04-29T14:59:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af82584:::benchmark_data_smithi194_196924_object97780:head# 2024-04-29T14:59:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af85f3a:::benchmark_data_smithi194_196924_object152340:head# 2024-04-29T14:59:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af911a1:::benchmark_data_smithi194_196924_object111416:head# 2024-04-29T14:59:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4af93810:::benchmark_data_smithi194_196924_object102733:head# 2024-04-29T14:59:53.942 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4afaef05:::benchmark_data_smithi194_196924_object119380:head# 2024-04-29T14:59:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4afb4416:::benchmark_data_smithi194_196924_object99761:head# 2024-04-29T14:59:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4afc5342:::benchmark_data_smithi194_196924_object138526:head# 2024-04-29T14:59:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4afd4d97:::benchmark_data_smithi194_196924_object122275:head# 2024-04-29T14:59:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4afd72c3:::benchmark_data_smithi194_196924_object105435:head# 2024-04-29T14:59:53.945 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4afe4769:::benchmark_data_smithi194_196924_object103107:head# 2024-04-29T14:59:53.945 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4afee656:::benchmark_data_smithi194_196924_object152667:head# 2024-04-29T14:59:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4aff28e7:::benchmark_data_smithi194_196924_object144960:head# 2024-04-29T14:59:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4affe136:::benchmark_data_smithi194_196924_object99130:head# 2024-04-29T14:59:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b00ecf5:::benchmark_data_smithi194_196924_object132387:head# 2024-04-29T14:59:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b01af93:::benchmark_data_smithi194_196924_object116578:head# 2024-04-29T14:59:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b045958:::benchmark_data_smithi194_196924_object112517:head# 2024-04-29T14:59:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b049323:::benchmark_data_smithi194_196924_object146978:head# 2024-04-29T14:59:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0525bd:::benchmark_data_smithi194_196924_object147509:head# 2024-04-29T14:59:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b054252:::benchmark_data_smithi194_196924_object118798:head# 2024-04-29T14:59:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b05f2e4:::benchmark_data_smithi194_196924_object124733:head# 2024-04-29T14:59:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b07c915:::benchmark_data_smithi194_196924_object118509:head# 2024-04-29T14:59:53.950 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T14:59:53.950 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b07d4b3:::benchmark_data_smithi194_196924_object140253:head# 2024-04-29T14:59:53.950 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0862e9:::benchmark_data_smithi194_196924_object156976:head# 2024-04-29T14:59:53.951 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b09a733:::benchmark_data_smithi194_196924_object122999:head# 2024-04-29T14:59:53.951 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b09cc94:::benchmark_data_smithi194_196924_object103749:head# 2024-04-29T14:59:53.951 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0bbce3:::benchmark_data_smithi194_196924_object153324:head# 2024-04-29T14:59:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0c7e29:::benchmark_data_smithi194_196924_object96685:head# 2024-04-29T14:59:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0d475e:::benchmark_data_smithi194_196924_object147652:head# 2024-04-29T14:59:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0d9c1b:::benchmark_data_smithi194_196924_object132654:head# 2024-04-29T14:59:53.954 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0df60d:::benchmark_data_smithi194_196924_object147372:head# 2024-04-29T14:59:53.954 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b0f466e:::benchmark_data_smithi194_196924_object141375:head# 2024-04-29T14:59:53.954 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b1026e2:::benchmark_data_smithi194_196924_object112343:head# 2024-04-29T14:59:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:53.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T14:59:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b10c68b:::benchmark_data_smithi194_196924_object152764:head# 2024-04-29T14:59:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b10e08c:::benchmark_data_smithi194_196924_object100679:head# 2024-04-29T14:59:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b11c5e1:::benchmark_data_smithi194_196924_object113717:head# 2024-04-29T14:59:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b138eb7:::benchmark_data_smithi194_196924_object103983:head# 2024-04-29T14:59:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b13a9e8:::benchmark_data_smithi194_196924_object126272:head# 2024-04-29T14:59:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b150218:::benchmark_data_smithi194_196924_object138832:head# 2024-04-29T14:59:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b155b64:::benchmark_data_smithi194_196924_object114878:head# 2024-04-29T14:59:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b158397:::benchmark_data_smithi194_196924_object137259:head# 2024-04-29T14:59:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b168135:::benchmark_data_smithi194_196924_object107024:head# 2024-04-29T14:59:53.960 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b1892d8:::benchmark_data_smithi194_196924_object117878:head# 2024-04-29T14:59:53.960 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b199920:::benchmark_data_smithi194_196924_object98200:head# 2024-04-29T14:59:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b199a6e:::benchmark_data_smithi194_196924_object135082:head# 2024-04-29T14:59:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b1aef67:::benchmark_data_smithi194_196924_object122911:head# 2024-04-29T14:59:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b1ba3e4:::benchmark_data_smithi194_196924_object153464:head# 2024-04-29T14:59:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b1c9803:::benchmark_data_smithi194_196924_object151703:head# 2024-04-29T14:59:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b1caace:::benchmark_data_smithi194_196924_object146875:head# 2024-04-29T14:59:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b239778:::benchmark_data_smithi194_196924_object99251:head# 2024-04-29T14:59:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b252fe2:::benchmark_data_smithi194_196924_object138224:head# 2024-04-29T14:59:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b254feb:::benchmark_data_smithi194_196924_object147162:head# 2024-04-29T14:59:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b2639d4:::benchmark_data_smithi194_196924_object148203:head# 2024-04-29T14:59:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b275120:::benchmark_data_smithi194_196924_object148465:head# 2024-04-29T14:59:53.965 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b27c976:::benchmark_data_smithi194_196924_object157727:head# 2024-04-29T14:59:53.965 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b28c62b:::benchmark_data_smithi194_196924_object116981:head# 2024-04-29T14:59:53.965 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b28e85e:::benchmark_data_smithi194_196924_object133112:head# 2024-04-29T14:59:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b29f4d6:::benchmark_data_smithi194_196924_object113559:head# 2024-04-29T14:59:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b2b9efa:::benchmark_data_smithi194_196924_object102771:head# 2024-04-29T14:59:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b2ec8d4:::benchmark_data_smithi194_196924_object119211:head# 2024-04-29T14:59:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b2ed018:::benchmark_data_smithi194_196924_object143977:head# 2024-04-29T14:59:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b2f6670:::benchmark_data_smithi194_196924_object125396:head# 2024-04-29T14:59:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b2f96af:::benchmark_data_smithi194_196924_object95735:head# 2024-04-29T14:59:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b320270:::benchmark_data_smithi194_196924_object105013:head# 2024-04-29T14:59:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b32f32e:::benchmark_data_smithi194_196924_object144981:head# 2024-04-29T14:59:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b341117:::benchmark_data_smithi194_196924_object124675:head# 2024-04-29T14:59:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b369425:::benchmark_data_smithi194_196924_object131126:head# 2024-04-29T14:59:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b37675c:::benchmark_data_smithi194_196924_object159079:head# 2024-04-29T14:59:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b37abee:::benchmark_data_smithi194_196924_object139301:head# 2024-04-29T14:59:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b392df3:::benchmark_data_smithi194_196924_object158300:head# 2024-04-29T14:59:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3a5715:::benchmark_data_smithi194_196924_object157965:head# 2024-04-29T14:59:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3af547:::benchmark_data_smithi194_196924_object134249:head# 2024-04-29T14:59:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3b690f:::benchmark_data_smithi194_196924_object153481:head# 2024-04-29T14:59:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3beda3:::benchmark_data_smithi194_196924_object156603:head# 2024-04-29T14:59:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3d5143:::benchmark_data_smithi194_196924_object133125:head# 2024-04-29T14:59:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3ea7dd:::benchmark_data_smithi194_196924_object147894:head# 2024-04-29T14:59:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3eed62:::benchmark_data_smithi194_196924_object158426:head# 2024-04-29T14:59:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3f0c60:::benchmark_data_smithi194_196924_object153003:head# 2024-04-29T14:59:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3f225d:::benchmark_data_smithi194_196924_object132676:head# 2024-04-29T14:59:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b3f60e4:::benchmark_data_smithi194_196924_object138270:head# 2024-04-29T14:59:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b405ab1:::benchmark_data_smithi194_196924_object99852:head# 2024-04-29T14:59:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b40d972:::benchmark_data_smithi194_196924_object146080:head# 2024-04-29T14:59:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b42c3d9:::benchmark_data_smithi194_196924_object129148:head# 2024-04-29T14:59:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b42faf5:::benchmark_data_smithi194_196924_object137608:head# 2024-04-29T14:59:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b4327f8:::benchmark_data_smithi194_196924_object149987:head# 2024-04-29T14:59:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b456d90:::benchmark_data_smithi194_196924_object139906:head# 2024-04-29T14:59:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b45d89b:::benchmark_data_smithi194_196924_object119108:head# 2024-04-29T14:59:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b47cae0:::benchmark_data_smithi194_196924_object130133:head# 2024-04-29T14:59:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b493e5e:::benchmark_data_smithi194_196924_object107557:head# 2024-04-29T14:59:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b4a913b:::benchmark_data_smithi194_196924_object143134:head# 2024-04-29T14:59:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b4cc43b:::benchmark_data_smithi194_196924_object102469:head# 2024-04-29T14:59:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b4d1388:::benchmark_data_smithi194_196924_object121016:head# 2024-04-29T14:59:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b4e54ad:::benchmark_data_smithi194_196924_object144120:head# 2024-04-29T14:59:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b4eda8c:::benchmark_data_smithi194_196924_object153210:head# 2024-04-29T14:59:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b4f12d3:::benchmark_data_smithi194_196924_object153883:head# 2024-04-29T14:59:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b50fdb7:::benchmark_data_smithi194_196924_object136783:head# 2024-04-29T14:59:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b516455:::benchmark_data_smithi194_196924_object156320:head# 2024-04-29T14:59:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b53fb7c:::benchmark_data_smithi194_196924_object125471:head# 2024-04-29T14:59:53.984 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b53ff80:::benchmark_data_smithi194_196924_object134344:head# 2024-04-29T14:59:53.984 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b54482e:::benchmark_data_smithi194_196924_object111719:head# 2024-04-29T14:59:53.987 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b548fb9:::benchmark_data_smithi194_196924_object127885:head# 2024-04-29T14:59:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b561180:::benchmark_data_smithi194_196924_object117414:head# 2024-04-29T14:59:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b57ec7d:::benchmark_data_smithi194_196924_object110724:head# 2024-04-29T14:59:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b5864c4:::benchmark_data_smithi194_196924_object131849:head# 2024-04-29T14:59:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b595a8c:::benchmark_data_smithi194_196924_object154137:head# 2024-04-29T14:59:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b597ab4:::benchmark_data_smithi194_196924_object110332:head# 2024-04-29T14:59:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b5aa1d1:::benchmark_data_smithi194_196924_object130640:head# 2024-04-29T14:59:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b5ad66d:::benchmark_data_smithi194_196924_object155205:head# 2024-04-29T14:59:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b5c54d1:::benchmark_data_smithi194_196924_object135704:head# 2024-04-29T14:59:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b5cc9ed:::benchmark_data_smithi194_196924_object98562:head# 2024-04-29T14:59:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b5e1088:::benchmark_data_smithi194_196924_object150398:head# 2024-04-29T14:59:53.992 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b5eef88:::benchmark_data_smithi194_196924_object137134:head# 2024-04-29T14:59:53.992 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b601178:::benchmark_data_smithi194_196924_object125613:head# 2024-04-29T14:59:53.993 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b607b2c:::benchmark_data_smithi194_196924_object99326:head# 2024-04-29T14:59:53.993 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b610980:::benchmark_data_smithi194_196924_object152352:head# 2024-04-29T14:59:53.994 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b623de5:::benchmark_data_smithi194_196924_object139622:head# 2024-04-29T14:59:53.994 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b625b01:::benchmark_data_smithi194_196924_object146666:head# 2024-04-29T14:59:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b63a2df:::benchmark_data_smithi194_196924_object136873:head# 2024-04-29T14:59:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b647aae:::benchmark_data_smithi194_196924_object158036:head# 2024-04-29T14:59:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6562a2:::benchmark_data_smithi194_196924_object99441:head# 2024-04-29T14:59:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b66b071:::benchmark_data_smithi194_196924_object107230:head# 2024-04-29T14:59:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b67263c:::benchmark_data_smithi194_196924_object142328:head# 2024-04-29T14:59:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b681376:::benchmark_data_smithi194_196924_object135236:head# 2024-04-29T14:59:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b68362a:::benchmark_data_smithi194_196924_object101205:head# 2024-04-29T14:59:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6842b9:::benchmark_data_smithi194_196924_object146103:head# 2024-04-29T14:59:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b694116:::benchmark_data_smithi194_196924_object116423:head# 2024-04-29T14:59:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b69f010:::benchmark_data_smithi194_196924_object140058:head# 2024-04-29T14:59:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6a818a:::benchmark_data_smithi194_196924_object104649:head# 2024-04-29T14:59:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6b7508:::benchmark_data_smithi194_196924_object156325:head# 2024-04-29T14:59:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6c60cb:::benchmark_data_smithi194_196924_object121909:head# 2024-04-29T14:59:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6cf62d:::benchmark_data_smithi194_196924_object122502:head# 2024-04-29T14:59:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6e1c7c:::benchmark_data_smithi194_196924_object128687:head# 2024-04-29T14:59:54.002 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b6f650e:::benchmark_data_smithi194_196924_object118434:head# 2024-04-29T14:59:54.002 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b742407:::benchmark_data_smithi194_196924_object114555:head# 2024-04-29T14:59:54.003 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7497dc:::benchmark_data_smithi194_196924_object152050:head# 2024-04-29T14:59:54.003 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b75108e:::benchmark_data_smithi194_196924_object138699:head# 2024-04-29T14:59:54.003 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b771aa6:::benchmark_data_smithi194_196924_object113199:head# 2024-04-29T14:59:54.004 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b783a67:::benchmark_data_smithi194_196924_object118086:head# 2024-04-29T14:59:54.004 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7902be:::benchmark_data_smithi194_196924_object154143:head# 2024-04-29T14:59:54.004 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b790c3f:::benchmark_data_smithi194_196924_object107957:head# 2024-04-29T14:59:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7ac03c:::benchmark_data_smithi194_196924_object150128:head# 2024-04-29T14:59:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7b7dc9:::benchmark_data_smithi194_196924_object144728:head# 2024-04-29T14:59:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7c0e04:::benchmark_data_smithi194_196924_object103978:head# 2024-04-29T14:59:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7c6461:::benchmark_data_smithi194_196924_object136392:head# 2024-04-29T14:59:54.007 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7d905c:::benchmark_data_smithi194_196924_object139187:head# 2024-04-29T14:59:54.007 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7e1c72:::benchmark_data_smithi194_196924_object104991:head# 2024-04-29T14:59:54.008 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b7ee92d:::benchmark_data_smithi194_196924_object152016:head# 2024-04-29T14:59:54.008 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b81aedf:::benchmark_data_smithi194_196924_object158219:head# 2024-04-29T14:59:54.008 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b82cbc9:::benchmark_data_smithi194_196924_object115357:head# 2024-04-29T14:59:54.009 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b847806:::benchmark_data_smithi194_196924_object125252:head# 2024-04-29T14:59:54.009 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b849f95:::benchmark_data_smithi194_196924_object132819:head# 2024-04-29T14:59:54.010 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b87b3b3:::benchmark_data_smithi194_196924_object159065:head# 2024-04-29T14:59:54.011 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b87c5e7:::benchmark_data_smithi194_196924_object111864:head# 2024-04-29T14:59:54.011 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b88d789:::benchmark_data_smithi194_196924_object150223:head# 2024-04-29T14:59:54.011 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b89ef3c:::benchmark_data_smithi194_196924_object156089:head# 2024-04-29T14:59:54.012 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b8bb409:::benchmark_data_smithi194_196924_object136767:head# 2024-04-29T14:59:54.012 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b8d3ca2:::benchmark_data_smithi194_196924_object114298:head# 2024-04-29T14:59:54.013 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b8d9e6e:::benchmark_data_smithi194_196924_object120929:head# 2024-04-29T14:59:54.013 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b8ebb58:::benchmark_data_smithi194_196924_object159043:head# 2024-04-29T14:59:54.013 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b90a71f:::benchmark_data_smithi194_196924_object110191:head# 2024-04-29T14:59:54.014 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b923c33:::benchmark_data_smithi194_196924_object135741:head# 2024-04-29T14:59:54.014 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b964b3a:::benchmark_data_smithi194_196924_object127826:head# 2024-04-29T14:59:54.014 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b967afc:::benchmark_data_smithi194_196924_object122674:head# 2024-04-29T14:59:54.015 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b97084e:::benchmark_data_smithi194_196924_object124929:head# 2024-04-29T14:59:54.015 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b971d84:::benchmark_data_smithi194_196924_object145491:head# 2024-04-29T14:59:54.016 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b97a190:::benchmark_data_smithi194_196924_object120454:head# 2024-04-29T14:59:54.016 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b9829e5:::benchmark_data_smithi194_196924_object106032:head# 2024-04-29T14:59:54.017 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b9aedb6:::benchmark_data_smithi194_196924_object145017:head# 2024-04-29T14:59:54.017 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b9bed3a:::benchmark_data_smithi194_196924_object108202:head# 2024-04-29T14:59:54.017 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b9d3c6f:::benchmark_data_smithi194_196924_object102742:head# 2024-04-29T14:59:54.018 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b9d6322:::benchmark_data_smithi194_196924_object106444:head# 2024-04-29T14:59:54.018 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b9da915:::benchmark_data_smithi194_196924_object123406:head# 2024-04-29T14:59:54.019 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4b9fa3a2:::benchmark_data_smithi194_196924_object137942:head# 2024-04-29T14:59:54.020 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba1a82b:::benchmark_data_smithi194_196924_object140412:head# 2024-04-29T14:59:54.020 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba207b5:::benchmark_data_smithi194_196924_object118076:head# 2024-04-29T14:59:54.020 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba23648:::benchmark_data_smithi194_196924_object141175:head# 2024-04-29T14:59:54.020 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba35579:::benchmark_data_smithi194_196924_object127886:head# 2024-04-29T14:59:54.021 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba3901c:::benchmark_data_smithi194_196924_object140038:head# 2024-04-29T14:59:54.021 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba3afad:::benchmark_data_smithi194_196924_object119311:head# 2024-04-29T14:59:54.022 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba717e6:::benchmark_data_smithi194_196924_object158192:head# 2024-04-29T14:59:54.022 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba81c4f:::benchmark_data_smithi194_196924_object135887:head# 2024-04-29T14:59:54.022 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ba99534:::benchmark_data_smithi194_196924_object139545:head# 2024-04-29T14:59:54.023 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bac71b9:::benchmark_data_smithi194_196924_object141405:head# 2024-04-29T14:59:54.023 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bad68f3:::benchmark_data_smithi194_196924_object127188:head# 2024-04-29T14:59:54.024 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bada802:::benchmark_data_smithi194_196924_object131036:head# 2024-04-29T14:59:54.024 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb0e7dc:::benchmark_data_smithi194_196924_object101406:head# 2024-04-29T14:59:54.025 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb29700:::benchmark_data_smithi194_196924_object104654:head# 2024-04-29T14:59:54.030 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb2a208:::benchmark_data_smithi194_196924_object152100:head# 2024-04-29T14:59:54.030 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb4a28f:::benchmark_data_smithi194_196924_object135618:head# 2024-04-29T14:59:54.031 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb62a85:::benchmark_data_smithi194_196924_object154246:head# 2024-04-29T14:59:54.031 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb837fe:::benchmark_data_smithi194_196924_object126103:head# 2024-04-29T14:59:54.032 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb891b6:::benchmark_data_smithi194_196924_object100186:head# 2024-04-29T14:59:54.032 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bb9c41e:::benchmark_data_smithi194_196924_object117162:head# 2024-04-29T14:59:54.033 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bba9189:::benchmark_data_smithi194_196924_object152258:head# 2024-04-29T14:59:54.033 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bbabfdd:::benchmark_data_smithi194_196924_object151550:head# 2024-04-29T14:59:54.034 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bbdcf21:::benchmark_data_smithi194_196924_object102873:head# 2024-04-29T14:59:54.034 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bbf7d2b:::benchmark_data_smithi194_196924_object149590:head# 2024-04-29T14:59:54.035 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc0c41c:::benchmark_data_smithi194_196924_object145940:head# 2024-04-29T14:59:54.035 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc179b0:::benchmark_data_smithi194_196924_object110204:head# 2024-04-29T14:59:54.036 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc18683:::benchmark_data_smithi194_196924_object126773:head# 2024-04-29T14:59:54.036 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc305c6:::benchmark_data_smithi194_196924_object131583:head# 2024-04-29T14:59:54.036 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc443e3:::benchmark_data_smithi194_196924_object156590:head# 2024-04-29T14:59:54.037 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc48c94:::benchmark_data_smithi194_196924_object121381:head# 2024-04-29T14:59:54.037 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc4f6eb:::benchmark_data_smithi194_196924_object152424:head# 2024-04-29T14:59:54.038 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc5117e:::benchmark_data_smithi194_196924_object110635:head# 2024-04-29T14:59:54.038 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc5158d:::benchmark_data_smithi194_196924_object157390:head# 2024-04-29T14:59:54.038 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc554bb:::benchmark_data_smithi194_196924_object148451:head# 2024-04-29T14:59:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc57456:::benchmark_data_smithi194_196924_object114088:head# 2024-04-29T14:59:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc5c7d8:::benchmark_data_smithi194_196924_object119385:head# 2024-04-29T14:59:54.040 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc5cdc1:::benchmark_data_smithi194_196924_object125009:head# 2024-04-29T14:59:54.040 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc5eaf9:::benchmark_data_smithi194_196924_object128626:head# 2024-04-29T14:59:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc65354:::benchmark_data_smithi194_196924_object156038:head# 2024-04-29T14:59:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc73ae7:::benchmark_data_smithi194_196924_object103861:head# 2024-04-29T14:59:54.042 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc757de:::benchmark_data_smithi194_196924_object120259:head# 2024-04-29T14:59:54.042 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc809b6:::benchmark_data_smithi194_196924_object144963:head# 2024-04-29T14:59:54.043 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc87a6f:::benchmark_data_smithi194_196924_object151046:head# 2024-04-29T14:59:54.043 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bc8dc80:::benchmark_data_smithi194_196924_object111616:head# 2024-04-29T14:59:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bca0668:::benchmark_data_smithi194_196924_object129020:head# 2024-04-29T14:59:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bca5064:::benchmark_data_smithi194_196924_object131296:head# 2024-04-29T14:59:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bcb5280:::benchmark_data_smithi194_196924_object139879:head# 2024-04-29T14:59:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bcbe56b:::benchmark_data_smithi194_196924_object99867:head# 2024-04-29T14:59:54.046 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bcc05b8:::benchmark_data_smithi194_196924_object103135:head# 2024-04-29T14:59:54.046 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bce90e3:::benchmark_data_smithi194_196924_object132147:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bcf5074:::benchmark_data_smithi194_196924_object126997:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bcf8683:::benchmark_data_smithi194_196924_object105790:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bd15a19:::benchmark_data_smithi194_196924_object105419:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bd727e2:::benchmark_data_smithi194_196924_object132417:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bd767ba:::benchmark_data_smithi194_196924_object126104:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bd7c61c:::benchmark_data_smithi194_196924_object120653:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bd7ec29:::benchmark_data_smithi194_196924_object141857:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bd96832:::benchmark_data_smithi194_196924_object127428:head# 2024-04-29T14:59:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bda7a02:::benchmark_data_smithi194_196924_object149650:head# 2024-04-29T14:59:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bda7c68:::benchmark_data_smithi194_196924_object121317:head# 2024-04-29T14:59:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bda7ec8:::benchmark_data_smithi194_196924_object154856:head# 2024-04-29T14:59:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bdadf23:::benchmark_data_smithi194_196924_object108303:head# 2024-04-29T14:59:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bdba08c:::benchmark_data_smithi194_196924_object143890:head# 2024-04-29T14:59:54.054 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bdd521e:::benchmark_data_smithi194_196924_object102966:head# 2024-04-29T14:59:54.054 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bdf0041:::benchmark_data_smithi194_196924_object114735:head# 2024-04-29T14:59:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bdfda26:::benchmark_data_smithi194_196924_object121030:head# 2024-04-29T14:59:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be0c925:::benchmark_data_smithi194_196924_object131522:head# 2024-04-29T14:59:54.056 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be15de6:::benchmark_data_smithi194_196924_object131033:head# 2024-04-29T14:59:54.056 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:54.056 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be1e897:::benchmark_data_smithi194_196924_object107019:head# 2024-04-29T14:59:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be1fa8e:::benchmark_data_smithi194_196924_object106599:head# 2024-04-29T14:59:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be386d5:::benchmark_data_smithi194_196924_object146766:head# 2024-04-29T14:59:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be3f2ec:::benchmark_data_smithi194_196924_object103601:head# 2024-04-29T14:59:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be4ed8f:::benchmark_data_smithi194_196924_object99916:head# 2024-04-29T14:59:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be684b4:::benchmark_data_smithi194_196924_object104254:head# 2024-04-29T14:59:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be735f4:::benchmark_data_smithi194_196924_object133345:head# 2024-04-29T14:59:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be90471:::benchmark_data_smithi194_196924_object147388:head# 2024-04-29T14:59:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4be96ff7:::benchmark_data_smithi194_196924_object129907:head# 2024-04-29T14:59:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bea0f21:::benchmark_data_smithi194_196924_object100792:head# 2024-04-29T14:59:54.063 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4beb7332:::benchmark_data_smithi194_196924_object151352:head# 2024-04-29T14:59:54.063 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4beb7ed6:::benchmark_data_smithi194_196924_object107345:head# 2024-04-29T14:59:54.063 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4beb89f2:::benchmark_data_smithi194_196924_object115977:head# 2024-04-29T14:59:54.064 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bedf680:::benchmark_data_smithi194_196924_object143926:head# 2024-04-29T14:59:54.064 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf0e532:::benchmark_data_smithi194_196924_object152252:head# 2024-04-29T14:59:54.064 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf11f2b:::benchmark_data_smithi194_196924_object101557:head# 2024-04-29T14:59:54.065 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf210e7:::benchmark_data_smithi194_196924_object122508:head# 2024-04-29T14:59:54.065 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf339d3:::benchmark_data_smithi194_196924_object107319:head# 2024-04-29T14:59:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf50151:::benchmark_data_smithi194_196924_object125176:head# 2024-04-29T14:59:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf502b9:::benchmark_data_smithi194_196924_object124311:head# 2024-04-29T14:59:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf56f52:::benchmark_data_smithi194_196924_object95740:head# 2024-04-29T14:59:54.067 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf582c2:::benchmark_data_smithi194_196924_object108156:head# 2024-04-29T14:59:54.067 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf5f25d:::benchmark_data_smithi194_196924_object148835:head# 2024-04-29T14:59:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf79e2e:::benchmark_data_smithi194_196924_object116138:head# 2024-04-29T14:59:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf7edfb:::benchmark_data_smithi194_196924_object143476:head# 2024-04-29T14:59:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf93f59:::benchmark_data_smithi194_196924_object133540:head# 2024-04-29T14:59:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bf99e0c:::benchmark_data_smithi194_196924_object128815:head# 2024-04-29T14:59:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bfb58a0:::benchmark_data_smithi194_196924_object152065:head# 2024-04-29T14:59:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bfc47f0:::benchmark_data_smithi194_196924_object138692:head# 2024-04-29T14:59:54.071 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bfd0e85:::benchmark_data_smithi194_196924_object159535:head# 2024-04-29T14:59:54.071 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bfe0f73:::benchmark_data_smithi194_196924_object106137:head# 2024-04-29T14:59:54.071 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bfea78d:::benchmark_data_smithi194_196924_object148233:head# 2024-04-29T14:59:54.072 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4bff26ee:::benchmark_data_smithi194_196924_object113251:head# 2024-04-29T14:59:54.072 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c00ade7:::benchmark_data_smithi194_196924_object153556:head# 2024-04-29T14:59:54.073 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c01ca76:::benchmark_data_smithi194_196924_object146436:head# 2024-04-29T14:59:54.073 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c01db8a:::benchmark_data_smithi194_196924_object107603:head# 2024-04-29T14:59:54.073 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c034bdd:::benchmark_data_smithi194_196924_object148189:head# 2024-04-29T14:59:54.074 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c04c15b:::benchmark_data_smithi194_196924_object108744:head# 2024-04-29T14:59:54.074 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c0709b1:::benchmark_data_smithi194_196924_object111721:head# 2024-04-29T14:59:54.074 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c0b938c:::benchmark_data_smithi194_196924_object104139:head# 2024-04-29T14:59:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c0cc73e:::benchmark_data_smithi194_196924_object152856:head# 2024-04-29T14:59:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c0e19ee:::benchmark_data_smithi194_196924_object126678:head# 2024-04-29T14:59:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c0e48bc:::benchmark_data_smithi194_196924_object119440:head# 2024-04-29T14:59:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c0eeeee:::benchmark_data_smithi194_196924_object117411:head# 2024-04-29T14:59:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c0f3c6c:::benchmark_data_smithi194_196924_object124331:head# 2024-04-29T14:59:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c100cb4:::benchmark_data_smithi194_196924_object131910:head# 2024-04-29T14:59:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c106007:::benchmark_data_smithi194_196924_object107322:head# 2024-04-29T14:59:54.078 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c106e6f:::benchmark_data_smithi194_196924_object127616:head# 2024-04-29T14:59:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c11475e:::benchmark_data_smithi194_196924_object106804:head# 2024-04-29T14:59:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1270f7:::benchmark_data_smithi194_196924_object118254:head# 2024-04-29T14:59:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c139ab2:::benchmark_data_smithi194_196924_object102221:head# 2024-04-29T14:59:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c141d65:::benchmark_data_smithi194_196924_object140490:head# 2024-04-29T14:59:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c154503:::benchmark_data_smithi194_196924_object102941:head# 2024-04-29T14:59:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1731a4:::benchmark_data_smithi194_196924_object153004:head# 2024-04-29T14:59:54.083 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c197258:::benchmark_data_smithi194_196924_object147422:head# 2024-04-29T14:59:54.083 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1985fe:::benchmark_data_smithi194_196924_object134321:head# 2024-04-29T14:59:54.083 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1ab4a2:::benchmark_data_smithi194_196924_object122329:head# 2024-04-29T14:59:54.083 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1b3802:::benchmark_data_smithi194_196924_object122315:head# 2024-04-29T14:59:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1bd9ff:::benchmark_data_smithi194_196924_object150331:head# 2024-04-29T14:59:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1d0eef:::benchmark_data_smithi194_196924_object152104:head# 2024-04-29T14:59:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1f2a5b:::benchmark_data_smithi194_196924_object125954:head# 2024-04-29T14:59:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c1f52a3:::benchmark_data_smithi194_196924_object145606:head# 2024-04-29T14:59:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c290fc2:::benchmark_data_smithi194_196924_object140735:head# 2024-04-29T14:59:54.085 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2aa971:::benchmark_data_smithi194_196924_object106152:head# 2024-04-29T14:59:54.086 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2ae4ed:::benchmark_data_smithi194_196924_object101156:head# 2024-04-29T14:59:54.086 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2b4fb5:::benchmark_data_smithi194_196924_object109796:head# 2024-04-29T14:59:54.087 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2b5b49:::benchmark_data_smithi194_196924_object136080:head# 2024-04-29T14:59:54.087 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2bec8d:::benchmark_data_smithi194_196924_object98702:head# 2024-04-29T14:59:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2c2eaa:::benchmark_data_smithi194_196924_object141194:head# 2024-04-29T14:59:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2d262c:::benchmark_data_smithi194_196924_object140326:head# 2024-04-29T14:59:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2dc297:::benchmark_data_smithi194_196924_object148018:head# 2024-04-29T14:59:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2dc438:::benchmark_data_smithi194_196924_object125660:head# 2024-04-29T14:59:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2e120a:::benchmark_data_smithi194_196924_object132926:head# 2024-04-29T14:59:54.090 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c2f907f:::benchmark_data_smithi194_196924_object128383:head# 2024-04-29T14:59:54.090 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c339388:::benchmark_data_smithi194_196924_object125932:head# 2024-04-29T14:59:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c35f54f:::benchmark_data_smithi194_196924_object159349:head# 2024-04-29T14:59:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c37191a:::benchmark_data_smithi194_196924_object142523:head# 2024-04-29T14:59:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c380316:::benchmark_data_smithi194_196924_object98432:head# 2024-04-29T14:59:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c38c302:::benchmark_data_smithi194_196924_object153379:head# 2024-04-29T14:59:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c3df8a5:::benchmark_data_smithi194_196924_object134831:head# 2024-04-29T14:59:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c408b3b:::benchmark_data_smithi194_196924_object154395:head# 2024-04-29T14:59:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c41d538:::benchmark_data_smithi194_196924_object146086:head# 2024-04-29T14:59:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c43bb45:::benchmark_data_smithi194_196924_object110301:head# 2024-04-29T14:59:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c486d0f:::benchmark_data_smithi194_196924_object123464:head# 2024-04-29T14:59:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c4a7cbc:::benchmark_data_smithi194_196924_object114976:head# 2024-04-29T14:59:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c4c9c12:::benchmark_data_smithi194_196924_object101665:head# 2024-04-29T14:59:54.097 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c4d284a:::benchmark_data_smithi194_196924_object118929:head# 2024-04-29T14:59:54.098 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c4e5e1e:::benchmark_data_smithi194_196924_object119761:head# 2024-04-29T14:59:54.098 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c51b90b:::benchmark_data_smithi194_196924_object127248:head# 2024-04-29T14:59:54.098 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c526c30:::benchmark_data_smithi194_196924_object128884:head# 2024-04-29T14:59:54.099 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c52eb33:::benchmark_data_smithi194_196924_object117835:head# 2024-04-29T14:59:54.100 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c54e30c:::benchmark_data_smithi194_196924_object110428:head# 2024-04-29T14:59:54.100 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c55a6dc:::benchmark_data_smithi194_196924_object139361:head# 2024-04-29T14:59:54.100 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c572c86:::benchmark_data_smithi194_196924_object114901:head# 2024-04-29T14:59:54.100 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c577550:::benchmark_data_smithi194_196924_object155139:head# 2024-04-29T14:59:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c578c73:::benchmark_data_smithi194_196924_object141482:head# 2024-04-29T14:59:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c590e0c:::benchmark_data_smithi194_196924_object95549:head# 2024-04-29T14:59:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c596483:::benchmark_data_smithi194_196924_object109640:head# 2024-04-29T14:59:54.103 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c5c271b:::benchmark_data_smithi194_196924_object105197:head# 2024-04-29T14:59:54.103 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c5ca21a:::benchmark_data_smithi194_196924_object131116:head# 2024-04-29T14:59:54.103 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c5cd63b:::benchmark_data_smithi194_196924_object150923:head# 2024-04-29T14:59:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c5d3ce6:::benchmark_data_smithi194_196924_object117232:head# 2024-04-29T14:59:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c5e79ca:::benchmark_data_smithi194_196924_object139952:head# 2024-04-29T14:59:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c5f1b39:::benchmark_data_smithi194_196924_object105296:head# 2024-04-29T14:59:54.105 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c5f6b8b:::benchmark_data_smithi194_196924_object115228:head# 2024-04-29T14:59:54.105 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:54.105 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c60f4df:::benchmark_data_smithi194_196924_object107595:head# 2024-04-29T14:59:54.105 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c6164da:::benchmark_data_smithi194_196924_object141385:head# 2024-04-29T14:59:54.106 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c61b2f5:::benchmark_data_smithi194_196924_object101888:head# 2024-04-29T14:59:54.106 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c627b3a:::benchmark_data_smithi194_196924_object146203:head# 2024-04-29T14:59:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c63c76f:::benchmark_data_smithi194_196924_object115557:head# 2024-04-29T14:59:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c64642c:::benchmark_data_smithi194_196924_object126829:head# 2024-04-29T14:59:54.108 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c6605c6:::benchmark_data_smithi194_196924_object133025:head# 2024-04-29T14:59:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:54.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c679cc1:::benchmark_data_smithi194_196924_object149846:head# 2024-04-29T14:59:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c68f68b:::benchmark_data_smithi194_196924_object114471:head# 2024-04-29T14:59:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c697265:::benchmark_data_smithi194_196924_object140244:head# 2024-04-29T14:59:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c6e1275:::benchmark_data_smithi194_196924_object108782:head# 2024-04-29T14:59:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c6ede89:::benchmark_data_smithi194_196924_object119862:head# 2024-04-29T14:59:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c6f8e46:::benchmark_data_smithi194_196924_object136292:head# 2024-04-29T14:59:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c6f9821:::benchmark_data_smithi194_196924_object147609:head# 2024-04-29T14:59:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c70b1a1:::benchmark_data_smithi194_196924_object129587:head# 2024-04-29T14:59:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c712de1:::benchmark_data_smithi194_196924_object112526:head# 2024-04-29T14:59:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c7130fb:::benchmark_data_smithi194_196924_object132539:head# 2024-04-29T14:59:54.113 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c71684e:::benchmark_data_smithi194_196924_object135147:head# 2024-04-29T14:59:54.113 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c728081:::benchmark_data_smithi194_196924_object154091:head# 2024-04-29T14:59:54.113 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c735bc3:::benchmark_data_smithi194_196924_object140894:head# 2024-04-29T14:59:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c73b84d:::benchmark_data_smithi194_196924_object127703:head# 2024-04-29T14:59:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c73f946:::benchmark_data_smithi194_196924_object158146:head# 2024-04-29T14:59:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c746029:::benchmark_data_smithi194_196924_object123160:head# 2024-04-29T14:59:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c76edec:::benchmark_data_smithi194_196924_object117871:head# 2024-04-29T14:59:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c77aed9:::benchmark_data_smithi194_196924_object147935:head# 2024-04-29T14:59:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c780c6c:::benchmark_data_smithi194_196924_object141038:head# 2024-04-29T14:59:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c792228:::benchmark_data_smithi194_196924_object145386:head# 2024-04-29T14:59:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c7cb3ff:::benchmark_data_smithi194_196924_object135212:head# 2024-04-29T14:59:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c7d01ec:::benchmark_data_smithi194_196924_object113211:head# 2024-04-29T14:59:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c7d7e5a:::benchmark_data_smithi194_196924_object137149:head# 2024-04-29T14:59:54.118 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c8065bc:::benchmark_data_smithi194_196924_object108129:head# 2024-04-29T14:59:54.118 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c839edf:::benchmark_data_smithi194_196924_object103683:head# 2024-04-29T14:59:54.118 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c83b571:::benchmark_data_smithi194_196924_object154334:head# 2024-04-29T14:59:54.119 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c855e45:::benchmark_data_smithi194_196924_object106808:head# 2024-04-29T14:59:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c878845:::benchmark_data_smithi194_196924_object152740:head# 2024-04-29T14:59:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c87ca80:::benchmark_data_smithi194_196924_object148780:head# 2024-04-29T14:59:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c87ecd1:::benchmark_data_smithi194_196924_object142880:head# 2024-04-29T14:59:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c883d0b:::benchmark_data_smithi194_196924_object117649:head# 2024-04-29T14:59:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c8868fa:::benchmark_data_smithi194_196924_object144661:head# 2024-04-29T14:59:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c8a73f7:::benchmark_data_smithi194_196924_object141811:head# 2024-04-29T14:59:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c8c5f52:::benchmark_data_smithi194_196924_object141459:head# 2024-04-29T14:59:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c8deb3d:::benchmark_data_smithi194_196924_object96053:head# 2024-04-29T14:59:54.123 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c8e7fbf:::benchmark_data_smithi194_196924_object124709:head# 2024-04-29T14:59:54.123 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c8ec80a:::benchmark_data_smithi194_196924_object150969:head# 2024-04-29T14:59:54.124 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c91123d:::benchmark_data_smithi194_196924_object117557:head# 2024-04-29T14:59:54.124 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c915876:::benchmark_data_smithi194_196924_object136600:head# 2024-04-29T14:59:54.125 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c92115e:::benchmark_data_smithi194_196924_object120017:head# 2024-04-29T14:59:54.125 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c92c028:::benchmark_data_smithi194_196924_object153922:head# 2024-04-29T14:59:54.125 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c92cf26:::benchmark_data_smithi194_196924_object157485:head# 2024-04-29T14:59:54.126 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c93e2f2:::benchmark_data_smithi194_196924_object125161:head# 2024-04-29T14:59:54.126 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c940be3:::benchmark_data_smithi194_196924_object112347:head# 2024-04-29T14:59:54.127 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c96ecc2:::benchmark_data_smithi194_196924_object111104:head# 2024-04-29T14:59:54.127 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c97e99d:::benchmark_data_smithi194_196924_object159624:head# 2024-04-29T14:59:54.129 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c99d5c5:::benchmark_data_smithi194_196924_object121816:head# 2024-04-29T14:59:54.129 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c9c6284:::benchmark_data_smithi194_196924_object107301:head# 2024-04-29T14:59:54.131 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4c9d4b8f:::benchmark_data_smithi194_196924_object142761:head# 2024-04-29T14:59:54.131 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ca11530:::benchmark_data_smithi194_196924_object138099:head# 2024-04-29T14:59:54.131 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ca1e149:::benchmark_data_smithi194_196924_object107890:head# 2024-04-29T14:59:54.131 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ca24816:::benchmark_data_smithi194_196924_object157593:head# 2024-04-29T14:59:54.131 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ca59d29:::benchmark_data_smithi194_196924_object125496:head# 2024-04-29T14:59:54.131 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ca8ed09:::benchmark_data_smithi194_196924_object119008:head# 2024-04-29T14:59:54.132 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ca98a81:::benchmark_data_smithi194_196924_object133223:head# 2024-04-29T14:59:54.132 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ca9ecb6:::benchmark_data_smithi194_196924_object117240:head# 2024-04-29T14:59:54.132 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4caa9624:::benchmark_data_smithi194_196924_object115724:head# 2024-04-29T14:59:54.133 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4caac754:::benchmark_data_smithi194_196924_object151698:head# 2024-04-29T14:59:54.133 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cac4771:::benchmark_data_smithi194_196924_object150846:head# 2024-04-29T14:59:54.134 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb02298:::benchmark_data_smithi194_196924_object129098:head# 2024-04-29T14:59:54.134 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb30f13:::benchmark_data_smithi194_196924_object112021:head# 2024-04-29T14:59:54.134 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb31c4e:::benchmark_data_smithi194_196924_object106081:head# 2024-04-29T14:59:54.135 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb60272:::benchmark_data_smithi194_196924_object133542:head# 2024-04-29T14:59:54.135 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb69995:::benchmark_data_smithi194_196924_object148722:head# 2024-04-29T14:59:54.136 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb69d96:::benchmark_data_smithi194_196924_object119816:head# 2024-04-29T14:59:54.136 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb7a9ed:::benchmark_data_smithi194_196924_object146989:head# 2024-04-29T14:59:54.137 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb80e86:::benchmark_data_smithi194_196924_object122247:head# 2024-04-29T14:59:54.138 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb90775:::benchmark_data_smithi194_196924_object159370:head# 2024-04-29T14:59:54.138 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cb99efc:::benchmark_data_smithi194_196924_object154960:head# 2024-04-29T14:59:54.138 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cbcc5d2:::benchmark_data_smithi194_196924_object107108:head# 2024-04-29T14:59:54.139 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cbdf61f:::benchmark_data_smithi194_196924_object159001:head# 2024-04-29T14:59:54.139 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cbf22dc:::benchmark_data_smithi194_196924_object142010:head# 2024-04-29T14:59:54.140 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cbf26a2:::benchmark_data_smithi194_196924_object108853:head# 2024-04-29T14:59:54.140 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc0469a:::benchmark_data_smithi194_196924_object95566:head# 2024-04-29T14:59:54.140 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc0cc1b:::benchmark_data_smithi194_196924_object126079:head# 2024-04-29T14:59:54.141 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc25fd0:::benchmark_data_smithi194_196924_object149477:head# 2024-04-29T14:59:54.141 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc386bc:::benchmark_data_smithi194_196924_object114051:head# 2024-04-29T14:59:54.142 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc395a8:::benchmark_data_smithi194_196924_object103202:head# 2024-04-29T14:59:54.142 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc5bdcd:::benchmark_data_smithi194_196924_object152334:head# 2024-04-29T14:59:54.143 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc8c585:::benchmark_data_smithi194_196924_object150901:head# 2024-04-29T14:59:54.143 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc8c59a:::benchmark_data_smithi194_196924_object131481:head# 2024-04-29T14:59:54.143 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc910dd:::benchmark_data_smithi194_196924_object95831:head# 2024-04-29T14:59:54.144 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cc9fc90:::benchmark_data_smithi194_196924_object142181:head# 2024-04-29T14:59:54.144 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cca1064:::benchmark_data_smithi194_196924_object144916:head# 2024-04-29T14:59:54.145 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cca7a3e:::benchmark_data_smithi194_196924_object106136:head# 2024-04-29T14:59:54.145 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cccf1e4:::benchmark_data_smithi194_196924_object96095:head# 2024-04-29T14:59:54.146 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ccd48f2:::benchmark_data_smithi194_196924_object95794:head# 2024-04-29T14:59:54.146 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ccd4e7e:::benchmark_data_smithi194_196924_object157645:head# 2024-04-29T14:59:54.147 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ccde8f0:::benchmark_data_smithi194_196924_object141009:head# 2024-04-29T14:59:54.147 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ccdf090:::benchmark_data_smithi194_196924_object106700:head# 2024-04-29T14:59:54.148 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ccfa033:::benchmark_data_smithi194_196924_object115986:head# 2024-04-29T14:59:54.148 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cd20097:::benchmark_data_smithi194_196924_object147650:head# 2024-04-29T14:59:54.149 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cd2c233:::benchmark_data_smithi194_196924_object107831:head# 2024-04-29T14:59:54.149 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cd576f1:::benchmark_data_smithi194_196924_object102863:head# 2024-04-29T14:59:54.149 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cd6a461:::benchmark_data_smithi194_196924_object116947:head# 2024-04-29T14:59:54.150 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cd71444:::benchmark_data_smithi194_196924_object135959:head# 2024-04-29T14:59:54.150 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cda1fc8:::benchmark_data_smithi194_196924_object139233:head# 2024-04-29T14:59:54.151 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cdd6962:::benchmark_data_smithi194_196924_object104531:head# 2024-04-29T14:59:54.151 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cde8b91:::benchmark_data_smithi194_196924_object157367:head# 2024-04-29T14:59:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce17798:::benchmark_data_smithi194_196924_object136874:head# 2024-04-29T14:59:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce1f6ae:::benchmark_data_smithi194_196924_object136794:head# 2024-04-29T14:59:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce20c1c:::benchmark_data_smithi194_196924_object154473:head# 2024-04-29T14:59:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce33e4c:::benchmark_data_smithi194_196924_object114725:head# 2024-04-29T14:59:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce41912:::benchmark_data_smithi194_196924_object111316:head# 2024-04-29T14:59:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce49992:::benchmark_data_smithi194_196924_object116687:head# 2024-04-29T14:59:54.155 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce4d5fc:::benchmark_data_smithi194_196924_object100691:head# 2024-04-29T14:59:54.155 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce54cb1:::benchmark_data_smithi194_196924_object152232:head# 2024-04-29T14:59:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce5a51e:::benchmark_data_smithi194_196924_object130059:head# 2024-04-29T14:59:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce7b5e9:::benchmark_data_smithi194_196924_object156407:head# 2024-04-29T14:59:54.157 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ce863d9:::benchmark_data_smithi194_196924_object148756:head# 2024-04-29T14:59:54.157 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ceb6e1d:::benchmark_data_smithi194_196924_object112976:head# 2024-04-29T14:59:54.157 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ceb975b:::benchmark_data_smithi194_196924_object127572:head# 2024-04-29T14:59:54.158 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cee22b2:::benchmark_data_smithi194_196924_object129793:head# 2024-04-29T14:59:54.158 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ceeb697:::benchmark_data_smithi194_196924_object138281:head# 2024-04-29T14:59:54.158 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cefd4a6:::benchmark_data_smithi194_196924_object107379:head# 2024-04-29T14:59:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ceff6fc:::benchmark_data_smithi194_196924_object118419:head# 2024-04-29T14:59:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf0cac6:::benchmark_data_smithi194_196924_object98223:head# 2024-04-29T14:59:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf28376:::benchmark_data_smithi194_196924_object148487:head# 2024-04-29T14:59:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf33a55:::benchmark_data_smithi194_196924_object141113:head# 2024-04-29T14:59:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf38536:::benchmark_data_smithi194_196924_object140213:head# 2024-04-29T14:59:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf5976f:::benchmark_data_smithi194_196924_object123051:head# 2024-04-29T14:59:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf70d4e:::benchmark_data_smithi194_196924_object145114:head# 2024-04-29T14:59:54.162 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf7402d:::benchmark_data_smithi194_196924_object150408:head# 2024-04-29T14:59:54.162 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf83dae:::benchmark_data_smithi194_196924_object112775:head# 2024-04-29T14:59:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cf9b906:::benchmark_data_smithi194_196924_object135708:head# 2024-04-29T14:59:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cfa321c:::benchmark_data_smithi194_196924_object129569:head# 2024-04-29T14:59:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cfc653e:::benchmark_data_smithi194_196924_object103472:head# 2024-04-29T14:59:54.164 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4cfe45be:::benchmark_data_smithi194_196924_object111480:head# 2024-04-29T14:59:54.164 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d00a0fa:::benchmark_data_smithi194_196924_object126340:head# 2024-04-29T14:59:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d021983:::benchmark_data_smithi194_196924_object156303:head# 2024-04-29T14:59:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d02b270:::benchmark_data_smithi194_196924_object97648:head# 2024-04-29T14:59:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d050e7f:::benchmark_data_smithi194_196924_object97823:head# 2024-04-29T14:59:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d05a6c7:::benchmark_data_smithi194_196924_object149585:head# 2024-04-29T14:59:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d05b40e:::benchmark_data_smithi194_196924_object111226:head# 2024-04-29T14:59:54.167 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d07c00a:::benchmark_data_smithi194_196924_object108582:head# 2024-04-29T14:59:54.167 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d086b5f:::benchmark_data_smithi194_196924_object114255:head# 2024-04-29T14:59:54.167 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d0b593d:::benchmark_data_smithi194_196924_object108281:head# 2024-04-29T14:59:54.168 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d0be5ca:::benchmark_data_smithi194_196924_object134386:head# 2024-04-29T14:59:54.168 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d0c2f49:::benchmark_data_smithi194_196924_object154028:head# 2024-04-29T14:59:54.168 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d0d38cd:::benchmark_data_smithi194_196924_object122324:head# 2024-04-29T14:59:54.169 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d0dd7df:::benchmark_data_smithi194_196924_object107231:head# 2024-04-29T14:59:54.169 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d0f5ed0:::benchmark_data_smithi194_196924_object114377:head# 2024-04-29T14:59:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d100fd9:::benchmark_data_smithi194_196924_object119010:head# 2024-04-29T14:59:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d13122a:::benchmark_data_smithi194_196924_object102892:head# 2024-04-29T14:59:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d179596:::benchmark_data_smithi194_196924_object107853:head# 2024-04-29T14:59:54.171 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1851d7:::benchmark_data_smithi194_196924_object155517:head# 2024-04-29T14:59:54.171 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1911cc:::benchmark_data_smithi194_196924_object117321:head# 2024-04-29T14:59:54.172 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d19dd8b:::benchmark_data_smithi194_196924_object106603:head# 2024-04-29T14:59:54.172 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1a4c3e:::benchmark_data_smithi194_196924_object99813:head# 2024-04-29T14:59:54.173 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1a9751:::benchmark_data_smithi194_196924_object117627:head# 2024-04-29T14:59:54.173 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1b1620:::benchmark_data_smithi194_196924_object105767:head# 2024-04-29T14:59:54.173 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1d3bd2:::benchmark_data_smithi194_196924_object111906:head# 2024-04-29T14:59:54.174 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1e7bb6:::benchmark_data_smithi194_196924_object146812:head# 2024-04-29T14:59:54.174 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d1e809d:::benchmark_data_smithi194_196924_object157623:head# 2024-04-29T14:59:54.175 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d205899:::benchmark_data_smithi194_196924_object124182:head# 2024-04-29T14:59:54.175 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d20f81c:::benchmark_data_smithi194_196924_object157960:head# 2024-04-29T14:59:54.176 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d211a9d:::benchmark_data_smithi194_196924_object157591:head# 2024-04-29T14:59:54.176 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d214717:::benchmark_data_smithi194_196924_object121413:head# 2024-04-29T14:59:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d224bb8:::benchmark_data_smithi194_196924_object117692:head# 2024-04-29T14:59:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d231885:::benchmark_data_smithi194_196924_object118336:head# 2024-04-29T14:59:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d2695c1:::benchmark_data_smithi194_196924_object131855:head# 2024-04-29T14:59:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d26be2d:::benchmark_data_smithi194_196924_object120065:head# 2024-04-29T14:59:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d26dd85:::benchmark_data_smithi194_196924_object139191:head# 2024-04-29T14:59:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d272998:::benchmark_data_smithi194_196924_object141721:head# 2024-04-29T14:59:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d280383:::benchmark_data_smithi194_196924_object158143:head# 2024-04-29T14:59:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d294e5c:::benchmark_data_smithi194_196924_object124446:head# 2024-04-29T14:59:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d2953f5:::benchmark_data_smithi194_196924_object118821:head# 2024-04-29T14:59:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d295b5b:::benchmark_data_smithi194_196924_object151281:head# 2024-04-29T14:59:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d29913d:::benchmark_data_smithi194_196924_object141627:head# 2024-04-29T14:59:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d2d55b8:::benchmark_data_smithi194_196924_object152439:head# 2024-04-29T14:59:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3089c0:::benchmark_data_smithi194_196924_object128822:head# 2024-04-29T14:59:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d312240:::benchmark_data_smithi194_196924_object138722:head# 2024-04-29T14:59:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3290a9:::benchmark_data_smithi194_196924_object101433:head# 2024-04-29T14:59:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3312df:::benchmark_data_smithi194_196924_object98445:head# 2024-04-29T14:59:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d341b2e:::benchmark_data_smithi194_196924_object109560:head# 2024-04-29T14:59:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d34b9f2:::benchmark_data_smithi194_196924_object141307:head# 2024-04-29T14:59:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d35ab3c:::benchmark_data_smithi194_196924_object152775:head# 2024-04-29T14:59:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d37122a:::benchmark_data_smithi194_196924_object159057:head# 2024-04-29T14:59:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d37a329:::benchmark_data_smithi194_196924_object127674:head# 2024-04-29T14:59:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3a6ede:::benchmark_data_smithi194_196924_object116750:head# 2024-04-29T14:59:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3b494a:::benchmark_data_smithi194_196924_object131145:head# 2024-04-29T14:59:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3d7182:::benchmark_data_smithi194_196924_object95963:head# 2024-04-29T14:59:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3db4c4:::benchmark_data_smithi194_196924_object101552:head# 2024-04-29T14:59:54.187 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3e0d08:::benchmark_data_smithi194_196924_object147780:head# 2024-04-29T14:59:54.187 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d3f2914:::benchmark_data_smithi194_196924_object152554:head# 2024-04-29T14:59:54.187 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d401edd:::benchmark_data_smithi194_196924_object144992:head# 2024-04-29T14:59:54.188 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d40f4e1:::benchmark_data_smithi194_196924_object118968:head# 2024-04-29T14:59:54.188 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d42c6d7:::benchmark_data_smithi194_196924_object155974:head# 2024-04-29T14:59:54.188 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d442aec:::benchmark_data_smithi194_196924_object113035:head# 2024-04-29T14:59:54.189 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d454491:::benchmark_data_smithi194_196924_object146663:head# 2024-04-29T14:59:54.189 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d455d4e:::benchmark_data_smithi194_196924_object122927:head# 2024-04-29T14:59:54.190 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d459473:::benchmark_data_smithi194_196924_object100122:head# 2024-04-29T14:59:54.190 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d45be9e:::benchmark_data_smithi194_196924_object119039:head# 2024-04-29T14:59:54.191 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d47289a:::benchmark_data_smithi194_196924_object121020:head# 2024-04-29T14:59:54.191 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d486221:::benchmark_data_smithi194_196924_object129291:head# 2024-04-29T14:59:54.192 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d490462:::benchmark_data_smithi194_196924_object112494:head# 2024-04-29T14:59:54.192 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d49a7f9:::benchmark_data_smithi194_196924_object136310:head# 2024-04-29T14:59:54.192 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d49c2b4:::benchmark_data_smithi194_196924_object126165:head# 2024-04-29T14:59:54.193 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d4a2118:::benchmark_data_smithi194_196924_object121581:head# 2024-04-29T14:59:54.193 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d4c59bd:::benchmark_data_smithi194_196924_object138507:head# 2024-04-29T14:59:54.194 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d4c7e77:::benchmark_data_smithi194_196924_object149984:head# 2024-04-29T14:59:54.194 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d4c88f4:::benchmark_data_smithi194_196924_object143857:head# 2024-04-29T14:59:54.194 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d4c996f:::benchmark_data_smithi194_196924_object157869:head# 2024-04-29T14:59:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d4d5b6a:::benchmark_data_smithi194_196924_object122676:head# 2024-04-29T14:59:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d4ebbf7:::benchmark_data_smithi194_196924_object114197:head# 2024-04-29T14:59:54.196 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d51620a:::benchmark_data_smithi194_196924_object124213:head# 2024-04-29T14:59:54.196 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d53c411:::benchmark_data_smithi194_196924_object153933:head# 2024-04-29T14:59:54.197 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d5474f5:::benchmark_data_smithi194_196924_object112774:head# 2024-04-29T14:59:54.197 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d54d8e4:::benchmark_data_smithi194_196924_object105893:head# 2024-04-29T14:59:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d554126:::benchmark_data_smithi194_196924_object124141:head# 2024-04-29T14:59:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d55924a:::benchmark_data_smithi194_196924_object145031:head# 2024-04-29T14:59:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d57ebf9:::benchmark_data_smithi194_196924_object96833:head# 2024-04-29T14:59:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d58122c:::benchmark_data_smithi194_196924_object119691:head# 2024-04-29T14:59:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d591004:::benchmark_data_smithi194_196924_object101964:head# 2024-04-29T14:59:54.200 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d5c1317:::benchmark_data_smithi194_196924_object99226:head# 2024-04-29T14:59:54.200 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d5c95cf:::benchmark_data_smithi194_196924_object126595:head# 2024-04-29T14:59:54.201 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d5df3f2:::benchmark_data_smithi194_196924_object136452:head# 2024-04-29T14:59:54.201 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d612f54:::benchmark_data_smithi194_196924_object106227:head# 2024-04-29T14:59:54.202 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d6152fe:::benchmark_data_smithi194_196924_object119609:head# 2024-04-29T14:59:54.202 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d61f72a:::benchmark_data_smithi194_196924_object133419:head# 2024-04-29T14:59:54.203 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d62fa61:::benchmark_data_smithi194_196924_object104887:head# 2024-04-29T14:59:54.203 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d64c4b3:::benchmark_data_smithi194_196924_object138798:head# 2024-04-29T14:59:54.203 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d66317b:::benchmark_data_smithi194_196924_object119472:head# 2024-04-29T14:59:54.204 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d668acd:::benchmark_data_smithi194_196924_object144669:head# 2024-04-29T14:59:54.204 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d67e822:::benchmark_data_smithi194_196924_object158647:head# 2024-04-29T14:59:54.204 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d68f449:::benchmark_data_smithi194_196924_object123661:head# 2024-04-29T14:59:54.205 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d6b917c:::benchmark_data_smithi194_196924_object107392:head# 2024-04-29T14:59:54.206 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d6e5d71:::benchmark_data_smithi194_196924_object107106:head# 2024-04-29T14:59:54.206 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d6ec20e:::benchmark_data_smithi194_196924_object101514:head# 2024-04-29T14:59:54.206 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d6f29d3:::benchmark_data_smithi194_196924_object154523:head# 2024-04-29T14:59:54.206 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d71431c:::benchmark_data_smithi194_196924_object116701:head# 2024-04-29T14:59:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d7176d2:::benchmark_data_smithi194_196924_object112908:head# 2024-04-29T14:59:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d71dfd4:::benchmark_data_smithi194_196924_object156805:head# 2024-04-29T14:59:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d723531:::benchmark_data_smithi194_196924_object117520:head# 2024-04-29T14:59:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d72685c:::benchmark_data_smithi194_196924_object124468:head# 2024-04-29T14:59:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d72a417:::benchmark_data_smithi194_196924_object114348:head# 2024-04-29T14:59:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d72a471:::benchmark_data_smithi194_196924_object110707:head# 2024-04-29T14:59:54.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d72bcec:::benchmark_data_smithi194_196924_object101561:head# 2024-04-29T14:59:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d730f56:::benchmark_data_smithi194_196924_object134035:head# 2024-04-29T14:59:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d73a5a7:::benchmark_data_smithi194_196924_object106386:head# 2024-04-29T14:59:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d73c9d4:::benchmark_data_smithi194_196924_object159601:head# 2024-04-29T14:59:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d749aec:::benchmark_data_smithi194_196924_object109674:head# 2024-04-29T14:59:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d764ea3:::benchmark_data_smithi194_196924_object95663:head# 2024-04-29T14:59:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d78d526:::benchmark_data_smithi194_196924_object140126:head# 2024-04-29T14:59:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d7a9248:::benchmark_data_smithi194_196924_object110307:head# 2024-04-29T14:59:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d7aab1b:::benchmark_data_smithi194_196924_object109254:head# 2024-04-29T14:59:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d7b614a:::benchmark_data_smithi194_196924_object154506:head# 2024-04-29T14:59:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d7d84f0:::benchmark_data_smithi194_196924_object103715:head# 2024-04-29T14:59:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d7e4fa3:::benchmark_data_smithi194_196924_object96849:head# 2024-04-29T14:59:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d82937c:::benchmark_data_smithi194_196924_object116465:head# 2024-04-29T14:59:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8447fe:::benchmark_data_smithi194_196924_object118584:head# 2024-04-29T14:59:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8555ed:::benchmark_data_smithi194_196924_object138752:head# 2024-04-29T14:59:54.216 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d858ba9:::benchmark_data_smithi194_196924_object142213:head# 2024-04-29T14:59:54.216 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d862d65:::benchmark_data_smithi194_196924_object133636:head# 2024-04-29T14:59:54.216 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d863958:::benchmark_data_smithi194_196924_object150298:head# 2024-04-29T14:59:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d86680f:::benchmark_data_smithi194_196924_object123773:head# 2024-04-29T14:59:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d886004:::benchmark_data_smithi194_196924_object116827:head# 2024-04-29T14:59:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d888e67:::benchmark_data_smithi194_196924_object155689:head# 2024-04-29T14:59:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d889db7:::benchmark_data_smithi194_196924_object119625:head# 2024-04-29T14:59:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8a00e3:::benchmark_data_smithi194_196924_object151616:head# 2024-04-29T14:59:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8a4829:::benchmark_data_smithi194_196924_object153436:head# 2024-04-29T14:59:54.220 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8b0582:::benchmark_data_smithi194_196924_object125713:head# 2024-04-29T14:59:54.220 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8c5e27:::benchmark_data_smithi194_196924_object154794:head# 2024-04-29T14:59:54.220 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8c76b3:::benchmark_data_smithi194_196924_object136450:head# 2024-04-29T14:59:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8e760c:::benchmark_data_smithi194_196924_object150409:head# 2024-04-29T14:59:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8f5838:::benchmark_data_smithi194_196924_object141098:head# 2024-04-29T14:59:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d8fca79:::benchmark_data_smithi194_196924_object159754:head# 2024-04-29T14:59:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d94c0d9:::benchmark_data_smithi194_196924_object109527:head# 2024-04-29T14:59:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d94d971:::benchmark_data_smithi194_196924_object118349:head# 2024-04-29T14:59:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d950ba2:::benchmark_data_smithi194_196924_object125014:head# 2024-04-29T14:59:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d959f07:::benchmark_data_smithi194_196924_object95438:head# 2024-04-29T14:59:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d95b031:::benchmark_data_smithi194_196924_object97279:head# 2024-04-29T14:59:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9608ed:::benchmark_data_smithi194_196924_object139723:head# 2024-04-29T14:59:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d96b824:::benchmark_data_smithi194_196924_object154038:head# 2024-04-29T14:59:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d975ee5:::benchmark_data_smithi194_196924_object105484:head# 2024-04-29T14:59:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d97e487:::benchmark_data_smithi194_196924_object111623:head# 2024-04-29T14:59:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d98dbcf:::benchmark_data_smithi194_196924_object138580:head# 2024-04-29T14:59:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d99520b:::benchmark_data_smithi194_196924_object95873:head# 2024-04-29T14:59:54.227 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9a11d3:::benchmark_data_smithi194_196924_object130915:head# 2024-04-29T14:59:54.227 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9b6800:::benchmark_data_smithi194_196924_object159594:head# 2024-04-29T14:59:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9c469d:::benchmark_data_smithi194_196924_object118566:head# 2024-04-29T14:59:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9c5ebe:::benchmark_data_smithi194_196924_object97964:head# 2024-04-29T14:59:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9cf4e4:::benchmark_data_smithi194_196924_object124775:head# 2024-04-29T14:59:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9da0d8:::benchmark_data_smithi194_196924_object158911:head# 2024-04-29T14:59:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9de951:::benchmark_data_smithi194_196924_object156508:head# 2024-04-29T14:59:54.230 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9e6f78:::benchmark_data_smithi194_196924_object100907:head# 2024-04-29T14:59:54.230 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9ea2e7:::benchmark_data_smithi194_196924_object127171:head# 2024-04-29T14:59:54.230 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9eca67:::benchmark_data_smithi194_196924_object113734:head# 2024-04-29T14:59:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9ef074:::benchmark_data_smithi194_196924_object127570:head# 2024-04-29T14:59:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4d9ef479:::benchmark_data_smithi194_196924_object109582:head# 2024-04-29T14:59:54.232 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da0933d:::benchmark_data_smithi194_196924_object98987:head# 2024-04-29T14:59:54.232 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da2dc73:::benchmark_data_smithi194_196924_object101426:head# 2024-04-29T14:59:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da55b21:::benchmark_data_smithi194_196924_object127855:head# 2024-04-29T14:59:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da62715:::benchmark_data_smithi194_196924_object148500:head# 2024-04-29T14:59:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da6afe0:::benchmark_data_smithi194_196924_object146449:head# 2024-04-29T14:59:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da7857f:::benchmark_data_smithi194_196924_object143530:head# 2024-04-29T14:59:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da79230:::benchmark_data_smithi194_196924_object132385:head# 2024-04-29T14:59:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4da855ef:::benchmark_data_smithi194_196924_object152939:head# 2024-04-29T14:59:54.235 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dacb21f:::benchmark_data_smithi194_196924_object141231:head# 2024-04-29T14:59:54.235 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dae6ed0:::benchmark_data_smithi194_196924_object124652:head# 2024-04-29T14:59:54.236 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db20f51:::benchmark_data_smithi194_196924_object151252:head# 2024-04-29T14:59:54.236 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db36c07:::benchmark_data_smithi194_196924_object131074:head# 2024-04-29T14:59:54.236 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db3ff50:::benchmark_data_smithi194_196924_object106188:head# 2024-04-29T14:59:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db4e9dc:::benchmark_data_smithi194_196924_object122404:head# 2024-04-29T14:59:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db5c8ec:::benchmark_data_smithi194_196924_object102679:head# 2024-04-29T14:59:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db6c065:::benchmark_data_smithi194_196924_object110150:head# 2024-04-29T14:59:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db6e1d0:::benchmark_data_smithi194_196924_object108283:head# 2024-04-29T14:59:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4db7a1bf:::benchmark_data_smithi194_196924_object95721:head# 2024-04-29T14:59:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dbbd87a:::benchmark_data_smithi194_196924_object152435:head# 2024-04-29T14:59:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dbc9972:::benchmark_data_smithi194_196924_object102700:head# 2024-04-29T14:59:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dbcbbfd:::benchmark_data_smithi194_196924_object139751:head# 2024-04-29T14:59:54.241 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dbdbb44:::benchmark_data_smithi194_196924_object114324:head# 2024-04-29T14:59:54.241 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dc00f48:::benchmark_data_smithi194_196924_object139552:head# 2024-04-29T14:59:54.241 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:54.242 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dc18dbf:::benchmark_data_smithi194_196924_object139870:head# 2024-04-29T14:59:54.242 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dc4df5a:::benchmark_data_smithi194_196924_object146917:head# 2024-04-29T14:59:54.242 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dc4f43b:::benchmark_data_smithi194_196924_object148048:head# 2024-04-29T14:59:54.243 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dc60887:::benchmark_data_smithi194_196924_object115512:head# 2024-04-29T14:59:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dc66a4d:::benchmark_data_smithi194_196924_object122561:head# 2024-04-29T14:59:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dca2290:::benchmark_data_smithi194_196924_object149832:head# 2024-04-29T14:59:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dcb2ac8:::benchmark_data_smithi194_196924_object138169:head# 2024-04-29T14:59:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dcb89b1:::benchmark_data_smithi194_196924_object114168:head# 2024-04-29T14:59:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dccb574:::benchmark_data_smithi194_196924_object97567:head# 2024-04-29T14:59:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dccf0aa:::benchmark_data_smithi194_196924_object142390:head# 2024-04-29T14:59:54.245 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dce0606:::benchmark_data_smithi194_196924_object120822:head# 2024-04-29T14:59:54.246 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dce5b45:::benchmark_data_smithi194_196924_object131035:head# 2024-04-29T14:59:54.246 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dcf645a:::benchmark_data_smithi194_196924_object149632:head# 2024-04-29T14:59:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dcfc5bf:::benchmark_data_smithi194_196924_object154749:head# 2024-04-29T14:59:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd095eb:::benchmark_data_smithi194_196924_object141001:head# 2024-04-29T14:59:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd0efc0:::benchmark_data_smithi194_196924_object110288:head# 2024-04-29T14:59:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd169c4:::benchmark_data_smithi194_196924_object145944:head# 2024-04-29T14:59:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd1963b:::benchmark_data_smithi194_196924_object133127:head# 2024-04-29T14:59:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd20fe4:::benchmark_data_smithi194_196924_object136419:head# 2024-04-29T14:59:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd218ec:::benchmark_data_smithi194_196924_object127233:head# 2024-04-29T14:59:54.250 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd3c3c3:::benchmark_data_smithi194_196924_object106920:head# 2024-04-29T14:59:54.250 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd45a72:::benchmark_data_smithi194_196924_object141885:head# 2024-04-29T14:59:54.251 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd59d6a:::benchmark_data_smithi194_196924_object116627:head# 2024-04-29T14:59:54.251 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd5c9f4:::benchmark_data_smithi194_196924_object112262:head# 2024-04-29T14:59:54.251 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd5d288:::benchmark_data_smithi194_196924_object113185:head# 2024-04-29T14:59:54.252 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd7b63c:::benchmark_data_smithi194_196924_object138074:head# 2024-04-29T14:59:54.252 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd93a96:::benchmark_data_smithi194_196924_object134087:head# 2024-04-29T14:59:54.253 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dd9c6e0:::benchmark_data_smithi194_196924_object107488:head# 2024-04-29T14:59:54.253 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ddae920:::benchmark_data_smithi194_196924_object154554:head# 2024-04-29T14:59:54.253 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ddd199a:::benchmark_data_smithi194_196924_object156700:head# 2024-04-29T14:59:54.254 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ddda9ec:::benchmark_data_smithi194_196924_object151147:head# 2024-04-29T14:59:54.254 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ddf9529:::benchmark_data_smithi194_196924_object149831:head# 2024-04-29T14:59:54.255 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de0fd9f:::benchmark_data_smithi194_196924_object154904:head# 2024-04-29T14:59:54.255 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de2371d:::benchmark_data_smithi194_196924_object158469:head# 2024-04-29T14:59:54.256 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de302ec:::benchmark_data_smithi194_196924_object98164:head# 2024-04-29T14:59:54.256 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de36f82:::benchmark_data_smithi194_196924_object101097:head# 2024-04-29T14:59:54.257 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de507e5:::benchmark_data_smithi194_196924_object141828:head# 2024-04-29T14:59:54.257 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de56161:::benchmark_data_smithi194_196924_object148769:head# 2024-04-29T14:59:54.257 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de581e2:::benchmark_data_smithi194_196924_object137536:head# 2024-04-29T14:59:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4de6c12b:::benchmark_data_smithi194_196924_object122638:head# 2024-04-29T14:59:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4deaf81b:::benchmark_data_smithi194_196924_object110826:head# 2024-04-29T14:59:54.259 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dedfa6b:::benchmark_data_smithi194_196924_object98741:head# 2024-04-29T14:59:54.259 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4def9531:::benchmark_data_smithi194_196924_object144434:head# 2024-04-29T14:59:54.259 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df119b0:::benchmark_data_smithi194_196924_object110287:head# 2024-04-29T14:59:54.260 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df33d4e:::benchmark_data_smithi194_196924_object139097:head# 2024-04-29T14:59:54.261 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df369e2:::benchmark_data_smithi194_196924_object132218:head# 2024-04-29T14:59:54.261 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df402e7:::benchmark_data_smithi194_196924_object123243:head# 2024-04-29T14:59:54.261 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df412fa:::benchmark_data_smithi194_196924_object149488:head# 2024-04-29T14:59:54.262 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df44aba:::benchmark_data_smithi194_196924_object104803:head# 2024-04-29T14:59:54.262 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df55c54:::benchmark_data_smithi194_196924_object151250:head# 2024-04-29T14:59:54.263 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4df9d406:::benchmark_data_smithi194_196924_object154153:head# 2024-04-29T14:59:54.263 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dfa4b14:::benchmark_data_smithi194_196924_object104426:head# 2024-04-29T14:59:54.264 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dfac436:::benchmark_data_smithi194_196924_object125348:head# 2024-04-29T14:59:54.264 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dfb0bb0:::benchmark_data_smithi194_196924_object147272:head# 2024-04-29T14:59:54.265 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dfcc513:::benchmark_data_smithi194_196924_object131127:head# 2024-04-29T14:59:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dfea619:::benchmark_data_smithi194_196924_object142069:head# 2024-04-29T14:59:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dfeb0a1:::benchmark_data_smithi194_196924_object123789:head# 2024-04-29T14:59:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4dfeb295:::benchmark_data_smithi194_196924_object142797:head# 2024-04-29T14:59:54.267 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e011b73:::benchmark_data_smithi194_196924_object125555:head# 2024-04-29T14:59:54.267 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e01e3d8:::benchmark_data_smithi194_196924_object133134:head# 2024-04-29T14:59:54.267 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e0326d0:::benchmark_data_smithi194_196924_object150961:head# 2024-04-29T14:59:54.268 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e04d6b2:::benchmark_data_smithi194_196924_object103638:head# 2024-04-29T14:59:54.268 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:54.268 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e0783a4:::benchmark_data_smithi194_196924_object144553:head# 2024-04-29T14:59:54.269 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e08008f:::benchmark_data_smithi194_196924_object112666:head# 2024-04-29T14:59:54.269 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e0842ce:::benchmark_data_smithi194_196924_object142927:head# 2024-04-29T14:59:54.270 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e09583e:::benchmark_data_smithi194_196924_object132897:head# 2024-04-29T14:59:54.270 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e09c16b:::benchmark_data_smithi194_196924_object105870:head# 2024-04-29T14:59:54.271 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e0d6ce4:::benchmark_data_smithi194_196924_object134819:head# 2024-04-29T14:59:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:54.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:54.272 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e0f8da1:::benchmark_data_smithi194_196924_object123208:head# 2024-04-29T14:59:54.272 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e12f7f3:::benchmark_data_smithi194_196924_object137934:head# 2024-04-29T14:59:54.273 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e146055:::benchmark_data_smithi194_196924_object106851:head# 2024-04-29T14:59:54.273 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e14dcb1:::benchmark_data_smithi194_196924_object135629:head# 2024-04-29T14:59:54.273 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1519f9:::benchmark_data_smithi194_196924_object150136:head# 2024-04-29T14:59:54.276 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e152a48:::benchmark_data_smithi194_196924_object134551:head# 2024-04-29T14:59:54.276 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e16b800:::benchmark_data_smithi194_196924_object131551:head# 2024-04-29T14:59:54.276 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e182b25:::benchmark_data_smithi194_196924_object135435:head# 2024-04-29T14:59:54.276 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e196aa8:::benchmark_data_smithi194_196924_object149991:head# 2024-04-29T14:59:54.277 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e198468:::benchmark_data_smithi194_196924_object118139:head# 2024-04-29T14:59:54.277 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1b099e:::benchmark_data_smithi194_196924_object157627:head# 2024-04-29T14:59:54.278 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1c7609:::benchmark_data_smithi194_196924_object113646:head# 2024-04-29T14:59:54.278 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1c9a7a:::benchmark_data_smithi194_196924_object131582:head# 2024-04-29T14:59:54.278 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1cb27f:::benchmark_data_smithi194_196924_object129544:head# 2024-04-29T14:59:54.279 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1d6e2e:::benchmark_data_smithi194_196924_object134001:head# 2024-04-29T14:59:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1de130:::benchmark_data_smithi194_196924_object113182:head# 2024-04-29T14:59:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e1e1fdf:::benchmark_data_smithi194_196924_object137556:head# 2024-04-29T14:59:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e211ffe:::benchmark_data_smithi194_196924_object120970:head# 2024-04-29T14:59:54.281 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e21335a:::benchmark_data_smithi194_196924_object121253:head# 2024-04-29T14:59:54.281 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e226026:::benchmark_data_smithi194_196924_object100689:head# 2024-04-29T14:59:54.282 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e23b245:::benchmark_data_smithi194_196924_object139124:head# 2024-04-29T14:59:54.282 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e2466a5:::benchmark_data_smithi194_196924_object130725:head# 2024-04-29T14:59:54.283 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e24c0c1:::benchmark_data_smithi194_196924_object105963:head# 2024-04-29T14:59:54.283 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e271e67:::benchmark_data_smithi194_196924_object120261:head# 2024-04-29T14:59:54.283 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e279810:::benchmark_data_smithi194_196924_object130277:head# 2024-04-29T14:59:54.284 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e283ec9:::benchmark_data_smithi194_196924_object116268:head# 2024-04-29T14:59:54.284 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e297229:::benchmark_data_smithi194_196924_object155592:head# 2024-04-29T14:59:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e2a4a9f:::benchmark_data_smithi194_196924_object121133:head# 2024-04-29T14:59:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e2ab65a:::benchmark_data_smithi194_196924_object154769:head# 2024-04-29T14:59:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e2b5fc5:::benchmark_data_smithi194_196924_object111829:head# 2024-04-29T14:59:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e2d2bda:::benchmark_data_smithi194_196924_object96482:head# 2024-04-29T14:59:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e2fc46a:::benchmark_data_smithi194_196924_object145555:head# 2024-04-29T14:59:54.287 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e310c4b:::benchmark_data_smithi194_196924_object96906:head# 2024-04-29T14:59:54.287 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e3239d1:::benchmark_data_smithi194_196924_object110384:head# 2024-04-29T14:59:54.288 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e32fddc:::benchmark_data_smithi194_196924_object138869:head# 2024-04-29T14:59:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e333fbe:::benchmark_data_smithi194_196924_object107413:head# 2024-04-29T14:59:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e346b27:::benchmark_data_smithi194_196924_object159236:head# 2024-04-29T14:59:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e35e6d4:::benchmark_data_smithi194_196924_object139249:head# 2024-04-29T14:59:54.290 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e3624a9:::benchmark_data_smithi194_196924_object147049:head# 2024-04-29T14:59:54.290 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e37120f:::benchmark_data_smithi194_196924_object124473:head# 2024-04-29T14:59:54.290 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e374cf1:::benchmark_data_smithi194_196924_object131328:head# 2024-04-29T14:59:54.291 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e39f860:::benchmark_data_smithi194_196924_object123000:head# 2024-04-29T14:59:54.291 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e3a210d:::benchmark_data_smithi194_196924_object132253:head# 2024-04-29T14:59:54.292 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e3c8b72:::benchmark_data_smithi194_196924_object122342:head# 2024-04-29T14:59:54.292 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e3da5f1:::benchmark_data_smithi194_196924_object118623:head# 2024-04-29T14:59:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e3e7d84:::benchmark_data_smithi194_196924_object102687:head# 2024-04-29T14:59:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e3fb17b:::benchmark_data_smithi194_196924_object156914:head# 2024-04-29T14:59:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e41aeba:::benchmark_data_smithi194_196924_object101687:head# 2024-04-29T14:59:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e41c863:::benchmark_data_smithi194_196924_object140979:head# 2024-04-29T14:59:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e44cdd1:::benchmark_data_smithi194_196924_object108730:head# 2024-04-29T14:59:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e455c8d:::benchmark_data_smithi194_196924_object96356:head# 2024-04-29T14:59:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e470889:::benchmark_data_smithi194_196924_object157929:head# 2024-04-29T14:59:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e477f04:::benchmark_data_smithi194_196924_object99665:head# 2024-04-29T14:59:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e47ca92:::benchmark_data_smithi194_196924_object135556:head# 2024-04-29T14:59:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e482732:::benchmark_data_smithi194_196924_object115056:head# 2024-04-29T14:59:54.297 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e4930ae:::benchmark_data_smithi194_196924_object145893:head# 2024-04-29T14:59:54.297 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e4a1add:::benchmark_data_smithi194_196924_object140852:head# 2024-04-29T14:59:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e4c251f:::benchmark_data_smithi194_196924_object124936:head# 2024-04-29T14:59:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e4c8d92:::benchmark_data_smithi194_196924_object115664:head# 2024-04-29T14:59:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e4f1f53:::benchmark_data_smithi194_196924_object122376:head# 2024-04-29T14:59:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e537cf4:::benchmark_data_smithi194_196924_object106176:head# 2024-04-29T14:59:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e546d7e:::benchmark_data_smithi194_196924_object153424:head# 2024-04-29T14:59:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e54aed7:::benchmark_data_smithi194_196924_object145801:head# 2024-04-29T14:59:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e554bd2:::benchmark_data_smithi194_196924_object157289:head# 2024-04-29T14:59:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e55922e:::benchmark_data_smithi194_196924_object135402:head# 2024-04-29T14:59:54.302 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e55f5da:::benchmark_data_smithi194_196924_object100412:head# 2024-04-29T14:59:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5769e1:::benchmark_data_smithi194_196924_object153767:head# 2024-04-29T14:59:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e59dd80:::benchmark_data_smithi194_196924_object148677:head# 2024-04-29T14:59:54.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5be809:::benchmark_data_smithi194_196924_object146831:head# 2024-04-29T14:59:54.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5c7fad:::benchmark_data_smithi194_196924_object148400:head# 2024-04-29T14:59:54.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5cb7c5:::benchmark_data_smithi194_196924_object102701:head# 2024-04-29T14:59:54.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5d07e0:::benchmark_data_smithi194_196924_object105403:head# 2024-04-29T14:59:54.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5d5226:::benchmark_data_smithi194_196924_object115044:head# 2024-04-29T14:59:54.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5d5692:::benchmark_data_smithi194_196924_object125665:head# 2024-04-29T14:59:54.308 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5d5a19:::benchmark_data_smithi194_196924_object144091:head# 2024-04-29T14:59:54.309 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5d9847:::benchmark_data_smithi194_196924_object145016:head# 2024-04-29T14:59:54.309 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5e2435:::benchmark_data_smithi194_196924_object130035:head# 2024-04-29T14:59:54.309 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5f56f5:::benchmark_data_smithi194_196924_object147764:head# 2024-04-29T14:59:54.310 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5f7cc0:::benchmark_data_smithi194_196924_object124897:head# 2024-04-29T14:59:54.310 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5fc5e4:::benchmark_data_smithi194_196924_object139090:head# 2024-04-29T14:59:54.310 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e5fc6ec:::benchmark_data_smithi194_196924_object129812:head# 2024-04-29T14:59:54.311 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e63286e:::benchmark_data_smithi194_196924_object134677:head# 2024-04-29T14:59:54.311 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e647e12:::benchmark_data_smithi194_196924_object102819:head# 2024-04-29T14:59:54.312 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e65bf8e:::benchmark_data_smithi194_196924_object106212:head# 2024-04-29T14:59:54.312 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e6b63f6:::benchmark_data_smithi194_196924_object116402:head# 2024-04-29T14:59:54.313 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e6bb1b8:::benchmark_data_smithi194_196924_object126086:head# 2024-04-29T14:59:54.313 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e6c3c74:::benchmark_data_smithi194_196924_object98759:head# 2024-04-29T14:59:54.314 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e6f6162:::benchmark_data_smithi194_196924_object97301:head# 2024-04-29T14:59:54.314 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e6fc834:::benchmark_data_smithi194_196924_object138387:head# 2024-04-29T14:59:54.315 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e708e1e:::benchmark_data_smithi194_196924_object116034:head# 2024-04-29T14:59:54.315 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e70c45a:::benchmark_data_smithi194_196924_object135843:head# 2024-04-29T14:59:54.316 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e746840:::benchmark_data_smithi194_196924_object119854:head# 2024-04-29T14:59:54.316 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e7783cd:::benchmark_data_smithi194_196924_object152430:head# 2024-04-29T14:59:54.317 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e77cf28:::benchmark_data_smithi194_196924_object97643:head# 2024-04-29T14:59:54.317 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e77ee9b:::benchmark_data_smithi194_196924_object109004:head# 2024-04-29T14:59:54.318 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e789787:::benchmark_data_smithi194_196924_object111438:head# 2024-04-29T14:59:54.318 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e793c46:::benchmark_data_smithi194_196924_object105901:head# 2024-04-29T14:59:54.319 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e798c81:::benchmark_data_smithi194_196924_object105117:head# 2024-04-29T14:59:54.319 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e79a92b:::benchmark_data_smithi194_196924_object152083:head# 2024-04-29T14:59:54.319 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e79ccfb:::benchmark_data_smithi194_196924_object140567:head# 2024-04-29T14:59:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e7b04ff:::benchmark_data_smithi194_196924_object156699:head# 2024-04-29T14:59:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e7c0f8b:::benchmark_data_smithi194_196924_object134005:head# 2024-04-29T14:59:54.321 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e7ca61a:::benchmark_data_smithi194_196924_object110906:head# 2024-04-29T14:59:54.321 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e7ea8e7:::benchmark_data_smithi194_196924_object127840:head# 2024-04-29T14:59:54.322 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e7f8fb5:::benchmark_data_smithi194_196924_object115873:head# 2024-04-29T14:59:54.322 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8046ec:::benchmark_data_smithi194_196924_object126088:head# 2024-04-29T14:59:54.323 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e80f931:::benchmark_data_smithi194_196924_object147280:head# 2024-04-29T14:59:54.323 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e839daf:::benchmark_data_smithi194_196924_object111560:head# 2024-04-29T14:59:54.324 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e848c67:::benchmark_data_smithi194_196924_object153838:head# 2024-04-29T14:59:54.324 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8572a7:::benchmark_data_smithi194_196924_object135485:head# 2024-04-29T14:59:54.325 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e85b51c:::benchmark_data_smithi194_196924_object116634:head# 2024-04-29T14:59:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8628c4:::benchmark_data_smithi194_196924_object103014:head# 2024-04-29T14:59:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8806e8:::benchmark_data_smithi194_196924_object136664:head# 2024-04-29T14:59:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8818f2:::benchmark_data_smithi194_196924_object115246:head# 2024-04-29T14:59:54.329 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8bfc60:::benchmark_data_smithi194_196924_object154629:head# 2024-04-29T14:59:54.329 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8c4db7:::benchmark_data_smithi194_196924_object118664:head# 2024-04-29T14:59:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8dce47:::benchmark_data_smithi194_196924_object95773:head# 2024-04-29T14:59:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8e6855:::benchmark_data_smithi194_196924_object104958:head# 2024-04-29T14:59:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8e7335:::benchmark_data_smithi194_196924_object129644:head# 2024-04-29T14:59:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e8ebd10:::benchmark_data_smithi194_196924_object109975:head# 2024-04-29T14:59:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9148ab:::benchmark_data_smithi194_196924_object143543:head# 2024-04-29T14:59:54.332 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e91bb9d:::benchmark_data_smithi194_196924_object159122:head# 2024-04-29T14:59:54.332 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e94f9b8:::benchmark_data_smithi194_196924_object130377:head# 2024-04-29T14:59:54.332 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e96d49c:::benchmark_data_smithi194_196924_object99645:head# 2024-04-29T14:59:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e98a57c:::benchmark_data_smithi194_196924_object96325:head# 2024-04-29T14:59:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9a5fc3:::benchmark_data_smithi194_196924_object131542:head# 2024-04-29T14:59:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9ac15d:::benchmark_data_smithi194_196924_object130857:head# 2024-04-29T14:59:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9ad557:::benchmark_data_smithi194_196924_object134684:head# 2024-04-29T14:59:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9bf66b:::benchmark_data_smithi194_196924_object102141:head# 2024-04-29T14:59:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9cf51d:::benchmark_data_smithi194_196924_object117587:head# 2024-04-29T14:59:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9d05e2:::benchmark_data_smithi194_196924_object129696:head# 2024-04-29T14:59:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9f13d4:::benchmark_data_smithi194_196924_object112173:head# 2024-04-29T14:59:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4e9f625b:::benchmark_data_smithi194_196924_object141359:head# 2024-04-29T14:59:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea27255:::benchmark_data_smithi194_196924_object117171:head# 2024-04-29T14:59:54.337 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea394a8:::benchmark_data_smithi194_196924_object110619:head# 2024-04-29T14:59:54.338 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea51e70:::benchmark_data_smithi194_196924_object111646:head# 2024-04-29T14:59:54.338 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea7440b:::benchmark_data_smithi194_196924_object140989:head# 2024-04-29T14:59:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea79307:::benchmark_data_smithi194_196924_object157986:head# 2024-04-29T14:59:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea7ea7b:::benchmark_data_smithi194_196924_object129570:head# 2024-04-29T14:59:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea82b63:::benchmark_data_smithi194_196924_object100407:head# 2024-04-29T14:59:54.340 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea8779d:::benchmark_data_smithi194_196924_object155163:head# 2024-04-29T14:59:54.340 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ea8f6dd:::benchmark_data_smithi194_196924_object139427:head# 2024-04-29T14:59:54.340 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eab98e1:::benchmark_data_smithi194_196924_object153552:head# 2024-04-29T14:59:54.341 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eac9fb4:::benchmark_data_smithi194_196924_object116700:head# 2024-04-29T14:59:54.341 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eace7cd:::benchmark_data_smithi194_196924_object111695:head# 2024-04-29T14:59:54.342 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eaea503:::benchmark_data_smithi194_196924_object155704:head# 2024-04-29T14:59:54.342 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eaf1159:::benchmark_data_smithi194_196924_object130002:head# 2024-04-29T14:59:54.342 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb066dc:::benchmark_data_smithi194_196924_object102838:head# 2024-04-29T14:59:54.343 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb0a1a7:::benchmark_data_smithi194_196924_object151831:head# 2024-04-29T14:59:54.343 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb1eb07:::benchmark_data_smithi194_196924_object104354:head# 2024-04-29T14:59:54.343 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb31094:::benchmark_data_smithi194_196924_object108444:head# 2024-04-29T14:59:54.344 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb33cfa:::benchmark_data_smithi194_196924_object159456:head# 2024-04-29T14:59:54.344 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb40b3d:::benchmark_data_smithi194_196924_object155902:head# 2024-04-29T14:59:54.345 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb49efa:::benchmark_data_smithi194_196924_object125742:head# 2024-04-29T14:59:54.345 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb61f0c:::benchmark_data_smithi194_196924_object108862:head# 2024-04-29T14:59:54.345 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb6330a:::benchmark_data_smithi194_196924_object153369:head# 2024-04-29T14:59:54.347 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb88921:::benchmark_data_smithi194_196924_object147191:head# 2024-04-29T14:59:54.347 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb8ab54:::benchmark_data_smithi194_196924_object98450:head# 2024-04-29T14:59:54.347 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb90952:::benchmark_data_smithi194_196924_object144932:head# 2024-04-29T14:59:54.349 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eb97d98:::benchmark_data_smithi194_196924_object101023:head# 2024-04-29T14:59:54.349 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eba4a69:::benchmark_data_smithi194_196924_object103691:head# 2024-04-29T14:59:54.350 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ebb098a:::benchmark_data_smithi194_196924_object109774:head# 2024-04-29T14:59:54.350 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ebc3ae8:::benchmark_data_smithi194_196924_object102181:head# 2024-04-29T14:59:54.350 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ebc97bc:::benchmark_data_smithi194_196924_object150649:head# 2024-04-29T14:59:54.351 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ebcebce:::benchmark_data_smithi194_196924_object152325:head# 2024-04-29T14:59:54.351 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ebdab59:::benchmark_data_smithi194_196924_object110875:head# 2024-04-29T14:59:54.352 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ebf4961:::benchmark_data_smithi194_196924_object145271:head# 2024-04-29T14:59:54.354 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec0a2dc:::benchmark_data_smithi194_196924_object159547:head# 2024-04-29T14:59:54.354 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec2e7ee:::benchmark_data_smithi194_196924_object116070:head# 2024-04-29T14:59:54.354 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec2e9d7:::benchmark_data_smithi194_196924_object120843:head# 2024-04-29T14:59:54.355 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec344c7:::benchmark_data_smithi194_196924_object141734:head# 2024-04-29T14:59:54.355 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec4eacd:::benchmark_data_smithi194_196924_object118582:head# 2024-04-29T14:59:54.355 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec56041:::benchmark_data_smithi194_196924_object158194:head# 2024-04-29T14:59:54.356 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec5b3c5:::benchmark_data_smithi194_196924_object131088:head# 2024-04-29T14:59:54.356 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec68de7:::benchmark_data_smithi194_196924_object142657:head# 2024-04-29T14:59:54.357 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec7c203:::benchmark_data_smithi194_196924_object122487:head# 2024-04-29T14:59:54.357 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec81d90:::benchmark_data_smithi194_196924_object159802:head# 2024-04-29T14:59:54.358 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec89d6c:::benchmark_data_smithi194_196924_object139744:head# 2024-04-29T14:59:54.359 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec8b9f8:::benchmark_data_smithi194_196924_object129477:head# 2024-04-29T14:59:54.360 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec8fc5b:::benchmark_data_smithi194_196924_object123678:head# 2024-04-29T14:59:54.360 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ec982c2:::benchmark_data_smithi194_196924_object98623:head# 2024-04-29T14:59:54.360 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eca1a83:::benchmark_data_smithi194_196924_object144012:head# 2024-04-29T14:59:54.361 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ecaaa6c:::benchmark_data_smithi194_196924_object138279:head# 2024-04-29T14:59:54.361 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ecad55c:::benchmark_data_smithi194_196924_object136667:head# 2024-04-29T14:59:54.362 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ecbb392:::benchmark_data_smithi194_196924_object100092:head# 2024-04-29T14:59:54.362 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ecc3367:::benchmark_data_smithi194_196924_object111961:head# 2024-04-29T14:59:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ecd9ba8:::benchmark_data_smithi194_196924_object131113:head# 2024-04-29T14:59:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ece1b61:::benchmark_data_smithi194_196924_object153428:head# 2024-04-29T14:59:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eced644:::benchmark_data_smithi194_196924_object122045:head# 2024-04-29T14:59:54.364 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ed013a8:::benchmark_data_smithi194_196924_object120654:head# 2024-04-29T14:59:54.364 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ed15990:::benchmark_data_smithi194_196924_object139098:head# 2024-04-29T14:59:54.365 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ed17470:::benchmark_data_smithi194_196924_object131734:head# 2024-04-29T14:59:54.368 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ed50866:::benchmark_data_smithi194_196924_object132936:head# 2024-04-29T14:59:54.368 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ed79a12:::benchmark_data_smithi194_196924_object97953:head# 2024-04-29T14:59:54.368 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ed7c93d:::benchmark_data_smithi194_196924_object125828:head# 2024-04-29T14:59:54.369 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eda0e28:::benchmark_data_smithi194_196924_object98158:head# 2024-04-29T14:59:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4edc1114:::benchmark_data_smithi194_196924_object129751:head# 2024-04-29T14:59:54.372 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4edd47c1:::benchmark_data_smithi194_196924_object104852:head# 2024-04-29T14:59:54.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:54.374 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4edfb342:::benchmark_data_smithi194_196924_object100507:head# 2024-04-29T14:59:54.375 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee01f3a:::benchmark_data_smithi194_196924_object126009:head# 2024-04-29T14:59:54.375 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee042d1:::benchmark_data_smithi194_196924_object117737:head# 2024-04-29T14:59:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee047ad:::benchmark_data_smithi194_196924_object135502:head# 2024-04-29T14:59:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee06bb7:::benchmark_data_smithi194_196924_object139678:head# 2024-04-29T14:59:54.377 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee12864:::benchmark_data_smithi194_196924_object141380:head# 2024-04-29T14:59:54.377 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee2f11a:::benchmark_data_smithi194_196924_object134902:head# 2024-04-29T14:59:54.378 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee31881:::benchmark_data_smithi194_196924_object141601:head# 2024-04-29T14:59:54.378 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee41aab:::benchmark_data_smithi194_196924_object128650:head# 2024-04-29T14:59:54.379 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee4fc43:::benchmark_data_smithi194_196924_object143627:head# 2024-04-29T14:59:54.379 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee612d1:::benchmark_data_smithi194_196924_object100466:head# 2024-04-29T14:59:54.379 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee6d14b:::benchmark_data_smithi194_196924_object152508:head# 2024-04-29T14:59:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee72ce0:::benchmark_data_smithi194_196924_object118812:head# 2024-04-29T14:59:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ee92a54:::benchmark_data_smithi194_196924_object105139:head# 2024-04-29T14:59:54.381 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eea57e5:::benchmark_data_smithi194_196924_object127067:head# 2024-04-29T14:59:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eebba1a:::benchmark_data_smithi194_196924_object151327:head# 2024-04-29T14:59:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eec433b:::benchmark_data_smithi194_196924_object114033:head# 2024-04-29T14:59:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eed8dd4:::benchmark_data_smithi194_196924_object113748:head# 2024-04-29T14:59:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eedf9e7:::benchmark_data_smithi194_196924_object102818:head# 2024-04-29T14:59:54.383 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eef6cb2:::benchmark_data_smithi194_196924_object132114:head# 2024-04-29T14:59:54.383 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eefa933:::benchmark_data_smithi194_196924_object158601:head# 2024-04-29T14:59:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eeffe83:::benchmark_data_smithi194_196924_object137168:head# 2024-04-29T14:59:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ef21f56:::benchmark_data_smithi194_196924_object122307:head# 2024-04-29T14:59:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ef3a412:::benchmark_data_smithi194_196924_object99403:head# 2024-04-29T14:59:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ef6931e:::benchmark_data_smithi194_196924_object142513:head# 2024-04-29T14:59:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ef7149d:::benchmark_data_smithi194_196924_object114231:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efa2f88:::benchmark_data_smithi194_196924_object158973:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efa3599:::benchmark_data_smithi194_196924_object153792:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efb0046:::benchmark_data_smithi194_196924_object116805:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efb152d:::benchmark_data_smithi194_196924_object153517:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efb3368:::benchmark_data_smithi194_196924_object151388:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efcdcfd:::benchmark_data_smithi194_196924_object105822:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efd49c3:::benchmark_data_smithi194_196924_object142188:head# 2024-04-29T14:59:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efd9ebd:::benchmark_data_smithi194_196924_object109119:head# 2024-04-29T14:59:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efe81cc:::benchmark_data_smithi194_196924_object128018:head# 2024-04-29T14:59:54.395 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eff319e:::benchmark_data_smithi194_196924_object102930:head# 2024-04-29T14:59:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4eff6915:::benchmark_data_smithi194_196924_object119869:head# 2024-04-29T14:59:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4effd7b4:::benchmark_data_smithi194_196924_object137450:head# 2024-04-29T14:59:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4effec28:::benchmark_data_smithi194_196924_object155051:head# 2024-04-29T14:59:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4efff612:::benchmark_data_smithi194_196924_object142796:head# 2024-04-29T14:59:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0037c8:::benchmark_data_smithi194_196924_object152378:head# 2024-04-29T14:59:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0130cd:::benchmark_data_smithi194_196924_object112999:head# 2024-04-29T14:59:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0147e4:::benchmark_data_smithi194_196924_object116325:head# 2024-04-29T14:59:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f014812:::benchmark_data_smithi194_196924_object121500:head# 2024-04-29T14:59:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f02d595:::benchmark_data_smithi194_196924_object150756:head# 2024-04-29T14:59:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f036539:::benchmark_data_smithi194_196924_object124335:head# 2024-04-29T14:59:54.399 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f03704d:::benchmark_data_smithi194_196924_object109805:head# 2024-04-29T14:59:54.399 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f044a56:::benchmark_data_smithi194_196924_object150488:head# 2024-04-29T14:59:54.400 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f057564:::benchmark_data_smithi194_196924_object115493:head# 2024-04-29T14:59:54.400 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f067abc:::benchmark_data_smithi194_196924_object98079:head# 2024-04-29T14:59:54.401 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f07f0e8:::benchmark_data_smithi194_196924_object102667:head# 2024-04-29T14:59:54.401 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f080b03:::benchmark_data_smithi194_196924_object97242:head# 2024-04-29T14:59:54.401 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f09ecf6:::benchmark_data_smithi194_196924_object146319:head# 2024-04-29T14:59:54.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0a4846:::benchmark_data_smithi194_196924_object117751:head# 2024-04-29T14:59:54.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0b11ca:::benchmark_data_smithi194_196924_object129150:head# 2024-04-29T14:59:54.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0bdeb4:::benchmark_data_smithi194_196924_object117075:head# 2024-04-29T14:59:54.403 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0cc654:::benchmark_data_smithi194_196924_object153174:head# 2024-04-29T14:59:54.404 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0cf13e:::benchmark_data_smithi194_196924_object146700:head# 2024-04-29T14:59:54.404 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0d8812:::benchmark_data_smithi194_196924_object130826:head# 2024-04-29T14:59:54.405 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f0ff84d:::benchmark_data_smithi194_196924_object154531:head# 2024-04-29T14:59:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f10121e:::benchmark_data_smithi194_196924_object98585:head# 2024-04-29T14:59:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f12a8be:::benchmark_data_smithi194_196924_object107314:head# 2024-04-29T14:59:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f12c428:::benchmark_data_smithi194_196924_object135237:head# 2024-04-29T14:59:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f12c7b7:::benchmark_data_smithi194_196924_object146428:head# 2024-04-29T14:59:54.408 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f1307c1:::benchmark_data_smithi194_196924_object144268:head# 2024-04-29T14:59:54.408 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f1565ae:::benchmark_data_smithi194_196924_object144466:head# 2024-04-29T14:59:54.409 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f16e7be:::benchmark_data_smithi194_196924_object101165:head# 2024-04-29T14:59:54.409 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f1741b7:::benchmark_data_smithi194_196924_object101336:head# 2024-04-29T14:59:54.409 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f1aedb7:::benchmark_data_smithi194_196924_object120538:head# 2024-04-29T14:59:54.410 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f1f5508:::benchmark_data_smithi194_196924_object134359:head# 2024-04-29T14:59:54.411 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f1fc1f6:::benchmark_data_smithi194_196924_object139160:head# 2024-04-29T14:59:54.411 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f208792:::benchmark_data_smithi194_196924_object122556:head# 2024-04-29T14:59:54.411 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f211602:::benchmark_data_smithi194_196924_object139798:head# 2024-04-29T14:59:54.411 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f214a35:::benchmark_data_smithi194_196924_object125588:head# 2024-04-29T14:59:54.412 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f223f7a:::benchmark_data_smithi194_196924_object108123:head# 2024-04-29T14:59:54.412 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f231da9:::benchmark_data_smithi194_196924_object103742:head# 2024-04-29T14:59:54.413 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f233fa1:::benchmark_data_smithi194_196924_object159846:head# 2024-04-29T14:59:54.413 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f247ed1:::benchmark_data_smithi194_196924_object127057:head# 2024-04-29T14:59:54.414 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f26f934:::benchmark_data_smithi194_196924_object107182:head# 2024-04-29T14:59:54.414 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f2708db:::benchmark_data_smithi194_196924_object103722:head# 2024-04-29T14:59:54.414 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f293ffd:::benchmark_data_smithi194_196924_object131538:head# 2024-04-29T14:59:54.415 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f29c36c:::benchmark_data_smithi194_196924_object145141:head# 2024-04-29T14:59:54.415 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f2aa3f5:::benchmark_data_smithi194_196924_object138643:head# 2024-04-29T14:59:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f2c0f32:::benchmark_data_smithi194_196924_object156869:head# 2024-04-29T14:59:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f2ec060:::benchmark_data_smithi194_196924_object106897:head# 2024-04-29T14:59:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f2eceab:::benchmark_data_smithi194_196924_object124383:head# 2024-04-29T14:59:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3042bc:::benchmark_data_smithi194_196924_object116798:head# 2024-04-29T14:59:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f305ed3:::benchmark_data_smithi194_196924_object125709:head# 2024-04-29T14:59:54.418 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3186f0:::benchmark_data_smithi194_196924_object128558:head# 2024-04-29T14:59:54.418 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f321b72:::benchmark_data_smithi194_196924_object124958:head# 2024-04-29T14:59:54.418 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f332945:::benchmark_data_smithi194_196924_object121415:head# 2024-04-29T14:59:54.419 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f334dba:::benchmark_data_smithi194_196924_object126802:head# 2024-04-29T14:59:54.419 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f335016:::benchmark_data_smithi194_196924_object118813:head# 2024-04-29T14:59:54.420 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f33b965:::benchmark_data_smithi194_196924_object151761:head# 2024-04-29T14:59:54.420 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3472d6:::benchmark_data_smithi194_196924_object150916:head# 2024-04-29T14:59:54.420 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f35b019:::benchmark_data_smithi194_196924_object120789:head# 2024-04-29T14:59:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f35e2cb:::benchmark_data_smithi194_196924_object127451:head# 2024-04-29T14:59:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f37e63c:::benchmark_data_smithi194_196924_object111734:head# 2024-04-29T14:59:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f37f8f1:::benchmark_data_smithi194_196924_object128522:head# 2024-04-29T14:59:54.422 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f39a06d:::benchmark_data_smithi194_196924_object115928:head# 2024-04-29T14:59:54.422 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3a61f8:::benchmark_data_smithi194_196924_object114335:head# 2024-04-29T14:59:54.423 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3b01fc:::benchmark_data_smithi194_196924_object104971:head# 2024-04-29T14:59:54.423 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3c1cca:::benchmark_data_smithi194_196924_object108300:head# 2024-04-29T14:59:54.424 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3d856e:::benchmark_data_smithi194_196924_object112270:head# 2024-04-29T14:59:54.424 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f3ea510:::benchmark_data_smithi194_196924_object117352:head# 2024-04-29T14:59:54.425 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:54.428 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f408271:::benchmark_data_smithi194_196924_object147864:head# 2024-04-29T14:59:54.428 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f410dfa:::benchmark_data_smithi194_196924_object110584:head# 2024-04-29T14:59:54.428 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f43f283:::benchmark_data_smithi194_196924_object140593:head# 2024-04-29T14:59:54.428 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f44064e:::benchmark_data_smithi194_196924_object157109:head# 2024-04-29T14:59:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f445ccc:::benchmark_data_smithi194_196924_object143995:head# 2024-04-29T14:59:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4530ff:::benchmark_data_smithi194_196924_object103843:head# 2024-04-29T14:59:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f462c97:::benchmark_data_smithi194_196924_object135512:head# 2024-04-29T14:59:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f464ea3:::benchmark_data_smithi194_196924_object146169:head# 2024-04-29T14:59:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4703a2:::benchmark_data_smithi194_196924_object159715:head# 2024-04-29T14:59:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f471f23:::benchmark_data_smithi194_196924_object115548:head# 2024-04-29T14:59:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f478a80:::benchmark_data_smithi194_196924_object97947:head# 2024-04-29T14:59:54.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f48b6a0:::benchmark_data_smithi194_196924_object101113:head# 2024-04-29T14:59:54.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f49a135:::benchmark_data_smithi194_196924_object150763:head# 2024-04-29T14:59:54.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4c5a59:::benchmark_data_smithi194_196924_object122399:head# 2024-04-29T14:59:54.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4cef85:::benchmark_data_smithi194_196924_object97440:head# 2024-04-29T14:59:54.430 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4e05b3:::benchmark_data_smithi194_196924_object157723:head# 2024-04-29T14:59:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4ef104:::benchmark_data_smithi194_196924_object117570:head# 2024-04-29T14:59:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4f6725:::benchmark_data_smithi194_196924_object119519:head# 2024-04-29T14:59:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f4fdc1f:::benchmark_data_smithi194_196924_object139839:head# 2024-04-29T14:59:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f516930:::benchmark_data_smithi194_196924_object145704:head# 2024-04-29T14:59:54.433 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f518907:::benchmark_data_smithi194_196924_object154341:head# 2024-04-29T14:59:54.433 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f51cbbc:::benchmark_data_smithi194_196924_object155651:head# 2024-04-29T14:59:54.433 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5340b8:::benchmark_data_smithi194_196924_object97027:head# 2024-04-29T14:59:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f53ae62:::benchmark_data_smithi194_196924_object151575:head# 2024-04-29T14:59:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f53d6dd:::benchmark_data_smithi194_196924_object105934:head# 2024-04-29T14:59:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f54b53a:::benchmark_data_smithi194_196924_object136101:head# 2024-04-29T14:59:54.435 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5638e2:::benchmark_data_smithi194_196924_object117852:head# 2024-04-29T14:59:54.435 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f575abf:::benchmark_data_smithi194_196924_object102358:head# 2024-04-29T14:59:54.435 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f595890:::benchmark_data_smithi194_196924_object105938:head# 2024-04-29T14:59:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f597286:::benchmark_data_smithi194_196924_object96751:head# 2024-04-29T14:59:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5ae8cf:::benchmark_data_smithi194_196924_object97485:head# 2024-04-29T14:59:54.438 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5b0350:::benchmark_data_smithi194_196924_object122402:head# 2024-04-29T14:59:54.438 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5b14a7:::benchmark_data_smithi194_196924_object155401:head# 2024-04-29T14:59:54.438 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5b6827:::benchmark_data_smithi194_196924_object144122:head# 2024-04-29T14:59:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5b7a1d:::benchmark_data_smithi194_196924_object124215:head# 2024-04-29T14:59:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5bfb56:::benchmark_data_smithi194_196924_object105862:head# 2024-04-29T14:59:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5c74e6:::benchmark_data_smithi194_196924_object130135:head# 2024-04-29T14:59:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5ced61:::benchmark_data_smithi194_196924_object156882:head# 2024-04-29T14:59:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5d0eda:::benchmark_data_smithi194_196924_object150225:head# 2024-04-29T14:59:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5d476a:::benchmark_data_smithi194_196924_object106097:head# 2024-04-29T14:59:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5d71f7:::benchmark_data_smithi194_196924_object142638:head# 2024-04-29T14:59:54.442 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5d91d3:::benchmark_data_smithi194_196924_object95723:head# 2024-04-29T14:59:54.442 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f5f0ced:::benchmark_data_smithi194_196924_object96650:head# 2024-04-29T14:59:54.443 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6066d0:::benchmark_data_smithi194_196924_object114520:head# 2024-04-29T14:59:54.443 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f608f74:::benchmark_data_smithi194_196924_object112757:head# 2024-04-29T14:59:54.443 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f60de22:::benchmark_data_smithi194_196924_object155245:head# 2024-04-29T14:59:54.443 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6233f6:::benchmark_data_smithi194_196924_object117780:head# 2024-04-29T14:59:54.444 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f638667:::benchmark_data_smithi194_196924_object123155:head# 2024-04-29T14:59:54.444 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f642472:::benchmark_data_smithi194_196924_object153231:head# 2024-04-29T14:59:54.445 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f64ab3d:::benchmark_data_smithi194_196924_object145668:head# 2024-04-29T14:59:54.445 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f66d9ef:::benchmark_data_smithi194_196924_object127611:head# 2024-04-29T14:59:54.445 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f67eae2:::benchmark_data_smithi194_196924_object123723:head# 2024-04-29T14:59:54.446 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f692640:::benchmark_data_smithi194_196924_object111588:head# 2024-04-29T14:59:54.446 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6ae4bb:::benchmark_data_smithi194_196924_object118215:head# 2024-04-29T14:59:54.447 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6c79f1:::benchmark_data_smithi194_196924_object142921:head# 2024-04-29T14:59:54.447 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6cb195:::benchmark_data_smithi194_196924_object107153:head# 2024-04-29T14:59:54.447 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6cce49:::benchmark_data_smithi194_196924_object116605:head# 2024-04-29T14:59:54.448 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6cfe87:::benchmark_data_smithi194_196924_object109213:head# 2024-04-29T14:59:54.448 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6dc379:::benchmark_data_smithi194_196924_object151773:head# 2024-04-29T14:59:54.449 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6e89f2:::benchmark_data_smithi194_196924_object99304:head# 2024-04-29T14:59:54.449 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f6fcc4b:::benchmark_data_smithi194_196924_object137163:head# 2024-04-29T14:59:54.450 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f701334:::benchmark_data_smithi194_196924_object126176:head# 2024-04-29T14:59:54.450 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f72187c:::benchmark_data_smithi194_196924_object105109:head# 2024-04-29T14:59:54.450 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f73e368:::benchmark_data_smithi194_196924_object147833:head# 2024-04-29T14:59:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f73ec24:::benchmark_data_smithi194_196924_object142692:head# 2024-04-29T14:59:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f751dd2:::benchmark_data_smithi194_196924_object99145:head# 2024-04-29T14:59:54.452 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f763efa:::benchmark_data_smithi194_196924_object138690:head# 2024-04-29T14:59:54.452 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f76c2a1:::benchmark_data_smithi194_196924_object134192:head# 2024-04-29T14:59:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f76c36e:::benchmark_data_smithi194_196924_object98568:head# 2024-04-29T14:59:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f7b4094:::benchmark_data_smithi194_196924_object125079:head# 2024-04-29T14:59:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f7bc34c:::benchmark_data_smithi194_196924_object135873:head# 2024-04-29T14:59:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f7d6467:::benchmark_data_smithi194_196924_object97906:head# 2024-04-29T14:59:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f7e2af7:::benchmark_data_smithi194_196924_object98906:head# 2024-04-29T14:59:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f7f5b58:::benchmark_data_smithi194_196924_object134962:head# 2024-04-29T14:59:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f8423d6:::benchmark_data_smithi194_196924_object118609:head# 2024-04-29T14:59:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f844534:::benchmark_data_smithi194_196924_object104994:head# 2024-04-29T14:59:54.456 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f85b2df:::benchmark_data_smithi194_196924_object96449:head# 2024-04-29T14:59:54.456 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f85d9df:::benchmark_data_smithi194_196924_object129474:head# 2024-04-29T14:59:54.456 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f85ebbf:::benchmark_data_smithi194_196924_object126257:head# 2024-04-29T14:59:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f861a30:::benchmark_data_smithi194_196924_object152021:head# 2024-04-29T14:59:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f879de8:::benchmark_data_smithi194_196924_object109576:head# 2024-04-29T14:59:54.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T14:59:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f88d56f:::benchmark_data_smithi194_196924_object128960:head# 2024-04-29T14:59:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f8a0aad:::benchmark_data_smithi194_196924_object130723:head# 2024-04-29T14:59:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f8ba089:::benchmark_data_smithi194_196924_object150475:head# 2024-04-29T14:59:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f8cc660:::benchmark_data_smithi194_196924_object124450:head# 2024-04-29T14:59:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f8e6171:::benchmark_data_smithi194_196924_object143964:head# 2024-04-29T14:59:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f8ea5cf:::benchmark_data_smithi194_196924_object141421:head# 2024-04-29T14:59:54.460 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f8f7974:::benchmark_data_smithi194_196924_object157077:head# 2024-04-29T14:59:54.460 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f90beb7:::benchmark_data_smithi194_196924_object121236:head# 2024-04-29T14:59:54.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:54.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T14:59:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9110af:::benchmark_data_smithi194_196924_object131316:head# 2024-04-29T14:59:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f91bbe1:::benchmark_data_smithi194_196924_object156531:head# 2024-04-29T14:59:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f926fde:::benchmark_data_smithi194_196924_object131701:head# 2024-04-29T14:59:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f93f6db:::benchmark_data_smithi194_196924_object111482:head# 2024-04-29T14:59:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9570d4:::benchmark_data_smithi194_196924_object128210:head# 2024-04-29T14:59:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f959ad0:::benchmark_data_smithi194_196924_object155108:head# 2024-04-29T14:59:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f981dac:::benchmark_data_smithi194_196924_object109300:head# 2024-04-29T14:59:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f98c279:::benchmark_data_smithi194_196924_object126137:head# 2024-04-29T14:59:54.464 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f990858:::benchmark_data_smithi194_196924_object122067:head# 2024-04-29T14:59:54.464 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f99776a:::benchmark_data_smithi194_196924_object116968:head# 2024-04-29T14:59:54.465 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f99a0f4:::benchmark_data_smithi194_196924_object137929:head# 2024-04-29T14:59:54.465 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9a5dba:::benchmark_data_smithi194_196924_object119538:head# 2024-04-29T14:59:54.466 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9a8614:::benchmark_data_smithi194_196924_object153962:head# 2024-04-29T14:59:54.466 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9b015d:::benchmark_data_smithi194_196924_object103453:head# 2024-04-29T14:59:54.467 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9cd40d:::benchmark_data_smithi194_196924_object127216:head# 2024-04-29T14:59:54.467 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9db4ee:::benchmark_data_smithi194_196924_object101802:head# 2024-04-29T14:59:54.467 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9e9838:::benchmark_data_smithi194_196924_object107524:head# 2024-04-29T14:59:54.468 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9f5c7b:::benchmark_data_smithi194_196924_object156180:head# 2024-04-29T14:59:54.468 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4f9fd4cd:::benchmark_data_smithi194_196924_object133537:head# 2024-04-29T14:59:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa08521:::benchmark_data_smithi194_196924_object155389:head# 2024-04-29T14:59:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa26b0c:::benchmark_data_smithi194_196924_object101887:head# 2024-04-29T14:59:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa29ab3:::benchmark_data_smithi194_196924_object138685:head# 2024-04-29T14:59:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa29cc3:::benchmark_data_smithi194_196924_object129623:head# 2024-04-29T14:59:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa410f7:::benchmark_data_smithi194_196924_object155071:head# 2024-04-29T14:59:54.471 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa5f2ed:::benchmark_data_smithi194_196924_object133879:head# 2024-04-29T14:59:54.471 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa622ed:::benchmark_data_smithi194_196924_object125813:head# 2024-04-29T14:59:54.472 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa664dc:::benchmark_data_smithi194_196924_object102662:head# 2024-04-29T14:59:54.473 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa781bc:::benchmark_data_smithi194_196924_object140836:head# 2024-04-29T14:59:54.473 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa8418c:::benchmark_data_smithi194_196924_object142042:head# 2024-04-29T14:59:54.473 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa8c952:::benchmark_data_smithi194_196924_object100339:head# 2024-04-29T14:59:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fa9b52e:::benchmark_data_smithi194_196924_object105327:head# 2024-04-29T14:59:54.474 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fac8233:::benchmark_data_smithi194_196924_object149995:head# 2024-04-29T14:59:54.475 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fad7f54:::benchmark_data_smithi194_196924_object127100:head# 2024-04-29T14:59:54.475 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4faee032:::benchmark_data_smithi194_196924_object117415:head# 2024-04-29T14:59:54.476 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fafb52f:::benchmark_data_smithi194_196924_object110061:head# 2024-04-29T14:59:54.476 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb00f65:::benchmark_data_smithi194_196924_object107148:head# 2024-04-29T14:59:54.477 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb10af0:::benchmark_data_smithi194_196924_object140795:head# 2024-04-29T14:59:54.477 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb1477b:::benchmark_data_smithi194_196924_object119927:head# 2024-04-29T14:59:54.477 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb18ebb:::benchmark_data_smithi194_196924_object157071:head# 2024-04-29T14:59:54.478 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb3016c:::benchmark_data_smithi194_196924_object138908:head# 2024-04-29T14:59:54.478 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb3e6f2:::benchmark_data_smithi194_196924_object114779:head# 2024-04-29T14:59:54.479 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb4a69d:::benchmark_data_smithi194_196924_object152011:head# 2024-04-29T14:59:54.479 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb6f75d:::benchmark_data_smithi194_196924_object154138:head# 2024-04-29T14:59:54.480 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb70a5c:::benchmark_data_smithi194_196924_object159967:head# 2024-04-29T14:59:54.481 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fb746a0:::benchmark_data_smithi194_196924_object152186:head# 2024-04-29T14:59:54.481 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fba793f:::benchmark_data_smithi194_196924_object157253:head# 2024-04-29T14:59:54.481 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fbbb5c7:::benchmark_data_smithi194_196924_object107896:head# 2024-04-29T14:59:54.482 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fbc25d9:::benchmark_data_smithi194_196924_object139665:head# 2024-04-29T14:59:54.482 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fbc9b10:::benchmark_data_smithi194_196924_object157223:head# 2024-04-29T14:59:54.483 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fbe7461:::benchmark_data_smithi194_196924_object95917:head# 2024-04-29T14:59:54.483 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fbf992f:::benchmark_data_smithi194_196924_object111955:head# 2024-04-29T14:59:54.484 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc026e4:::benchmark_data_smithi194_196924_object98151:head# 2024-04-29T14:59:54.484 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc2b9da:::benchmark_data_smithi194_196924_object159559:head# 2024-04-29T14:59:54.485 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc2db7e:::benchmark_data_smithi194_196924_object119704:head# 2024-04-29T14:59:54.485 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc4ad3e:::benchmark_data_smithi194_196924_object124753:head# 2024-04-29T14:59:54.485 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc59513:::benchmark_data_smithi194_196924_object151872:head# 2024-04-29T14:59:54.486 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc6047f:::benchmark_data_smithi194_196924_object121301:head# 2024-04-29T14:59:54.486 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc617b3:::benchmark_data_smithi194_196924_object118148:head# 2024-04-29T14:59:54.487 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fc8c78c:::benchmark_data_smithi194_196924_object131525:head# 2024-04-29T14:59:54.487 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fca440e:::benchmark_data_smithi194_196924_object158027:head# 2024-04-29T14:59:54.488 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fcc0be2:::benchmark_data_smithi194_196924_object139489:head# 2024-04-29T14:59:54.488 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fccfd15:::benchmark_data_smithi194_196924_object95718:head# 2024-04-29T14:59:54.489 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fcd5ec2:::benchmark_data_smithi194_196924_object155047:head# 2024-04-29T14:59:54.489 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fce350d:::benchmark_data_smithi194_196924_object140726:head# 2024-04-29T14:59:54.489 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fcf18a6:::benchmark_data_smithi194_196924_object154705:head# 2024-04-29T14:59:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd1160a:::benchmark_data_smithi194_196924_object158832:head# 2024-04-29T14:59:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd129dc:::benchmark_data_smithi194_196924_object106093:head# 2024-04-29T14:59:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd2af88:::benchmark_data_smithi194_196924_object123099:head# 2024-04-29T14:59:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd35621:::benchmark_data_smithi194_196924_object139535:head# 2024-04-29T14:59:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd43f36:::benchmark_data_smithi194_196924_object129776:head# 2024-04-29T14:59:54.492 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd4c29e:::benchmark_data_smithi194_196924_object96713:head# 2024-04-29T14:59:54.492 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd659db:::benchmark_data_smithi194_196924_object123633:head# 2024-04-29T14:59:54.492 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd735e6:::benchmark_data_smithi194_196924_object148942:head# 2024-04-29T14:59:54.493 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd8bbae:::benchmark_data_smithi194_196924_object99327:head# 2024-04-29T14:59:54.493 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fd9e26c:::benchmark_data_smithi194_196924_object95701:head# 2024-04-29T14:59:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fdb4121:::benchmark_data_smithi194_196924_object97284:head# 2024-04-29T14:59:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fdb452c:::benchmark_data_smithi194_196924_object132140:head# 2024-04-29T14:59:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fdf4ee5:::benchmark_data_smithi194_196924_object130525:head# 2024-04-29T14:59:54.495 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fdf8270:::benchmark_data_smithi194_196924_object149515:head# 2024-04-29T14:59:54.495 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe035c4:::benchmark_data_smithi194_196924_object112444:head# 2024-04-29T14:59:54.496 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe15bf1:::benchmark_data_smithi194_196924_object115654:head# 2024-04-29T14:59:54.496 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe18403:::benchmark_data_smithi194_196924_object99125:head# 2024-04-29T14:59:54.496 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe1b980:::benchmark_data_smithi194_196924_object135409:head# 2024-04-29T14:59:54.497 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe44ca6:::benchmark_data_smithi194_196924_object127893:head# 2024-04-29T14:59:54.497 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe4ee75:::benchmark_data_smithi194_196924_object106496:head# 2024-04-29T14:59:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe56887:::benchmark_data_smithi194_196924_object156456:head# 2024-04-29T14:59:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe616d2:::benchmark_data_smithi194_196924_object121615:head# 2024-04-29T14:59:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe6c0a4:::benchmark_data_smithi194_196924_object118994:head# 2024-04-29T14:59:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe7e6ee:::benchmark_data_smithi194_196924_object153322:head# 2024-04-29T14:59:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe894f7:::benchmark_data_smithi194_196924_object123368:head# 2024-04-29T14:59:54.500 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe8e399:::benchmark_data_smithi194_196924_object158785:head# 2024-04-29T14:59:54.500 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe9e038:::benchmark_data_smithi194_196924_object125513:head# 2024-04-29T14:59:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fe9e96d:::benchmark_data_smithi194_196924_object139696:head# 2024-04-29T14:59:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4febd3da:::benchmark_data_smithi194_196924_object152757:head# 2024-04-29T14:59:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fec9de8:::benchmark_data_smithi194_196924_object148003:head# 2024-04-29T14:59:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4feca833:::benchmark_data_smithi194_196924_object96146:head# 2024-04-29T14:59:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fed50bd:::benchmark_data_smithi194_196924_object121589:head# 2024-04-29T14:59:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fed51bc:::benchmark_data_smithi194_196924_object132125:head# 2024-04-29T14:59:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fee1be5:::benchmark_data_smithi194_196924_object130150:head# 2024-04-29T14:59:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4fee4a09:::benchmark_data_smithi194_196924_object142684:head# 2024-04-29T14:59:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff0165d:::benchmark_data_smithi194_196924_object97875:head# 2024-04-29T14:59:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff0f37f:::benchmark_data_smithi194_196924_object148521:head# 2024-04-29T14:59:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff1f61b:::benchmark_data_smithi194_196924_object155880:head# 2024-04-29T14:59:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff21035:::benchmark_data_smithi194_196924_object114971:head# 2024-04-29T14:59:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff240a1:::benchmark_data_smithi194_196924_object157961:head# 2024-04-29T14:59:54.506 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff2d6d6:::benchmark_data_smithi194_196924_object157878:head# 2024-04-29T14:59:54.506 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff5e42f:::benchmark_data_smithi194_196924_object150684:head# 2024-04-29T14:59:54.506 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff64e5d:::benchmark_data_smithi194_196924_object121201:head# 2024-04-29T14:59:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff6ef0a:::benchmark_data_smithi194_196924_object118758:head# 2024-04-29T14:59:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff7d812:::benchmark_data_smithi194_196924_object136892:head# 2024-04-29T14:59:54.508 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff7f6c2:::benchmark_data_smithi194_196924_object98185:head# 2024-04-29T14:59:54.508 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ff94091:::benchmark_data_smithi194_196924_object158625:head# 2024-04-29T14:59:54.508 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ffaa82a:::benchmark_data_smithi194_196924_object117385:head# 2024-04-29T14:59:54.509 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ffc3fe3:::benchmark_data_smithi194_196924_object143170:head# 2024-04-29T14:59:54.509 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ffd4245:::benchmark_data_smithi194_196924_object117169:head# 2024-04-29T14:59:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ffdbc38:::benchmark_data_smithi194_196924_object130719:head# 2024-04-29T14:59:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ffea7d1:::benchmark_data_smithi194_196924_object147394:head# 2024-04-29T14:59:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:Read #9:4ffed6a1:::benchmark_data_smithi194_196924_object121710:head# 2024-04-29T14:59:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:Export successful 2024-04-29T14:59:54.561 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:54.590 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:54.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T14:59:54.615 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:54.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:54.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:54.751 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:54.775 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:54.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:54.882 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:54.914 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:54.935 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:54.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:55.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T14:59:55.093 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:55.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T14:59:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:55.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T14:59:55.241 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:55.269 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:55.293 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:55.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:55.397 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:55.428 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:55.452 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:55.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:55.513 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T14:59:55.513 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T14:59:55.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:55 smithi194 ceph-mon[44089]: pgmap v2908: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 31 MiB/s, 490 objects/s recovering 2024-04-29T14:59:55.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:55 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4127756966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:55.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:55 smithi194 ceph-mon[44089]: from='client.28642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:55.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T14:59:55.522 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T14:59:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T14:59:55.557 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T14:59:55.585 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:55.610 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T14:59:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:55.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T14:59:55.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:55 smithi040 ceph-mon[39581]: pgmap v2908: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 31 MiB/s, 490 objects/s recovering 2024-04-29T14:59:55.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:55 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4127756966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:55.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:55 smithi040 ceph-mon[39581]: from='client.28642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:55.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T14:59:55.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T14:59:55.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:55.746 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:55.773 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:55.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:55.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:55.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:55 smithi121 ceph-mon[44533]: pgmap v2908: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 31 MiB/s, 490 objects/s recovering 2024-04-29T14:59:55.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:55 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4127756966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:55.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:55 smithi121 ceph-mon[44533]: from='client.28642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T14:59:55.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T14:59:55.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T14:59:55.920 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:55.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:55.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:56.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T14:59:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:56.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.1 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 9.2 2024-04-29T14:59:56.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:56 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:56.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:56 smithi194 ceph-mon[44089]: from='client.28642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:59:56.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:56 smithi194 ceph-mon[44089]: osdmap e1135: 12 total, 11 up, 7 in 2024-04-29T14:59:56.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:56 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:56.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:56 smithi040 ceph-mon[39581]: from='client.28642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:59:56.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:56 smithi040 ceph-mon[39581]: osdmap e1135: 12 total, 11 up, 7 in 2024-04-29T14:59:56.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:56 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T14:59:56.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:56 smithi121 ceph-mon[44533]: from='client.28642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T14:59:56.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:56 smithi121 ceph-mon[44533]: osdmap e1135: 12 total, 11 up, 7 in 2024-04-29T14:59:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.1/config 2024-04-29T14:59:56.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-29T14:59:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T14:59:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:56.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T14:59:57.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:57.069 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:57.398 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T14:59:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:57.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:57.824 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:57.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:57 smithi194 ceph-mon[44089]: pgmap v2910: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 14 MiB/s, 230 objects/s recovering 2024-04-29T14:59:57.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:57 smithi194 ceph-mon[44089]: 9.4 scrub starts 2024-04-29T14:59:57.853 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:57.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:57 smithi040 ceph-mon[39581]: pgmap v2910: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 14 MiB/s, 230 objects/s recovering 2024-04-29T14:59:57.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:57 smithi040 ceph-mon[39581]: 9.4 scrub starts 2024-04-29T14:59:57.875 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:57.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:57.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:57.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:57 smithi121 ceph-mon[44533]: pgmap v2910: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 14 MiB/s, 230 objects/s recovering 2024-04-29T14:59:57.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:57 smithi121 ceph-mon[44533]: 9.4 scrub starts 2024-04-29T14:59:57.981 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:58.009 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:58.035 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:58.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:58.140 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T14:59:58.170 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:58.195 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T14:59:58.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:58.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T14:59:58.202 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T14:59:58.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T14:59:58.330 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:58.458 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T14:59:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:58.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T14:59:58.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T14:59:58.601 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:58 smithi040 ceph-mon[39581]: 9.4 scrub ok 2024-04-29T14:59:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:58.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:58 smithi194 ceph-mon[44089]: 9.4 scrub ok 2024-04-29T14:59:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:58 smithi121 ceph-mon[44533]: 9.4 scrub ok 2024-04-29T14:59:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T14:59:59.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T14:59:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:59.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T14:59:59.605 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T14:59:59.634 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T14:59:59.657 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T14:59:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T14:59:59.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T14:59:59.726 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T14:59:59.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T14:59:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 14:59:59 smithi194 ceph-mon[44089]: pgmap v2911: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 13 MiB/s, 203 objects/s recovering 2024-04-29T14:59:59.825 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 14:59:59 smithi040 ceph-mon[39581]: pgmap v2911: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 13 MiB/s, 203 objects/s recovering 2024-04-29T14:59:59.836 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 14:59:59 smithi121 ceph-mon[44533]: pgmap v2911: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 13 MiB/s, 203 objects/s recovering 2024-04-29T14:59:59.842 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:00.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:00.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:00.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:00 smithi194 ceph-mon[44089]: overall HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T15:00:00.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:00.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:00 smithi040 ceph-mon[39581]: overall HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T15:00:00.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:00 smithi121 ceph-mon[44533]: overall HEALTH_WARN nodeep-scrub flag(s) set 2024-04-29T15:00:00.904 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:01.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:01.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:01.176 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:01.205 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:01.225 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:01.227 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:01.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:01.256 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:01.278 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:01.302 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:01.330 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:01.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:01.360 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:01.395 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:01.403 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:01.424 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:01.446 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:01.468 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:01.497 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:01.497 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:01.497 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:01.497 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:01.497 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:01.497 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:01.497 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:01.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:01.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:01.632 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:01.663 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:01.683 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:01.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:01.698 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T15:00:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T15:00:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-29T15:00:01.698 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T15:00:01.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:01 smithi121 ceph-mon[44533]: pgmap v2912: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-04-29T15:00:01.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4203882336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:01.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:01.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:01 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:01.787 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:01.814 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:01.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:01 smithi194 ceph-mon[44089]: pgmap v2912: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-04-29T15:00:01.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4203882336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:01.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:01.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:01 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:01.837 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:01.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:01 smithi040 ceph-mon[39581]: pgmap v2912: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-04-29T15:00:01.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4203882336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:01.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:01.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:01 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:01.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:01.945 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:01.975 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:01.998 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:02.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:02.101 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:02.129 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:02.150 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:02.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:02.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:02.324 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:02.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:02.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:02.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-29T15:00:02.545 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:02.589 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:02.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:02.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: 9.a scrub starts 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.751 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:02 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.758 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:02.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:02.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:02.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: 9.a scrub starts 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:02 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:02.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:02.990 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:03.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:03.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: 9.a scrub starts 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:02 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:03.141 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:03.171 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:03.197 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:03.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:03.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:03.331 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:03.355 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:03.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:03.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:03.492 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:03.548 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:03.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:03.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:03 smithi121 ceph-mon[44533]: 9.a scrub ok 2024-04-29T15:00:03.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:03 smithi121 ceph-mon[44533]: 9.0 scrub starts 2024-04-29T15:00:03.626 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:03 smithi121 ceph-mon[44533]: pgmap v2913: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-04-29T15:00:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:03 smithi121 ceph-mon[44533]: 9.2 scrub starts 2024-04-29T15:00:03.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:03 smithi121 ceph-mon[44533]: 9.0 scrub ok 2024-04-29T15:00:03.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:03.718 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:03 smithi040 ceph-mon[39581]: 9.a scrub ok 2024-04-29T15:00:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:03 smithi040 ceph-mon[39581]: 9.0 scrub starts 2024-04-29T15:00:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:03 smithi040 ceph-mon[39581]: pgmap v2913: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-04-29T15:00:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:03 smithi040 ceph-mon[39581]: 9.2 scrub starts 2024-04-29T15:00:03.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:03 smithi040 ceph-mon[39581]: 9.0 scrub ok 2024-04-29T15:00:03.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:03.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:04.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:04.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:03 smithi194 ceph-mon[44089]: 9.a scrub ok 2024-04-29T15:00:04.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:03 smithi194 ceph-mon[44089]: 9.0 scrub starts 2024-04-29T15:00:04.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:03 smithi194 ceph-mon[44089]: pgmap v2913: 17 pgs: 17 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-04-29T15:00:04.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:03 smithi194 ceph-mon[44089]: 9.2 scrub starts 2024-04-29T15:00:04.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:03 smithi194 ceph-mon[44089]: 9.0 scrub ok 2024-04-29T15:00:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:04.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:04.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:04.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T15:00:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T15:00:05.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: 9.f scrub starts 2024-04-29T15:00:05.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: 9.2 scrub ok 2024-04-29T15:00:05.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:05.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:05.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: osdmap e1136: 12 total, 11 up, 7 in 2024-04-29T15:00:05.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: 9.c scrub starts 2024-04-29T15:00:05.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: 9.f scrub ok 2024-04-29T15:00:05.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2826847533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:05.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:04 smithi194 ceph-mon[44089]: pgmap v2915: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T15:00:05.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: 9.f scrub starts 2024-04-29T15:00:05.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: 9.2 scrub ok 2024-04-29T15:00:05.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:05.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:05.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: osdmap e1136: 12 total, 11 up, 7 in 2024-04-29T15:00:05.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: 9.c scrub starts 2024-04-29T15:00:05.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: 9.f scrub ok 2024-04-29T15:00:05.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2826847533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:05.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:04 smithi040 ceph-mon[39581]: pgmap v2915: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T15:00:05.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: 9.f scrub starts 2024-04-29T15:00:05.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: 9.2 scrub ok 2024-04-29T15:00:05.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:05.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:05.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: osdmap e1136: 12 total, 11 up, 7 in 2024-04-29T15:00:05.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: 9.c scrub starts 2024-04-29T15:00:05.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: 9.f scrub ok 2024-04-29T15:00:05.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2826847533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:05.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:04 smithi121 ceph-mon[44533]: pgmap v2915: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T15:00:05.257 INFO:teuthology.orchestra.run.smithi040.stdout: marking collection for removal 2024-04-29T15:00:05.278 INFO:teuthology.orchestra.run.smithi040.stdout:setting '_remove' omap key 2024-04-29T15:00:05.279 INFO:teuthology.orchestra.run.smithi040.stdout:finish_remove_pgs 9.2_head removing 9.2 2024-04-29T15:00:05.361 INFO:teuthology.orchestra.run.smithi040.stdout:Remove successful 2024-04-29T15:00:06.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:05 smithi194 ceph-mon[44089]: 9.c scrub ok 2024-04-29T15:00:06.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:05 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:00:06.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:05 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T15:00:06.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2826847533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:06.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:05 smithi194 ceph-mon[44089]: osdmap e1137: 12 total, 11 up, 7 in 2024-04-29T15:00:06.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:05 smithi040 ceph-mon[39581]: 9.c scrub ok 2024-04-29T15:00:06.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:05 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:00:06.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:05 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T15:00:06.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2826847533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:06.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:05 smithi040 ceph-mon[39581]: osdmap e1137: 12 total, 11 up, 7 in 2024-04-29T15:00:06.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:05 smithi121 ceph-mon[44533]: 9.c scrub ok 2024-04-29T15:00:06.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:05 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:00:06.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:05 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T15:00:06.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2826847533' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:06.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:05 smithi121 ceph-mon[44533]: osdmap e1137: 12 total, 11 up, 7 in 2024-04-29T15:00:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:06.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:06.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:06 smithi121 ceph-mon[44533]: pgmap v2917: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T15:00:06.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell -n osd.1 --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.9.2.1 2024-04-29T15:00:07.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:06 smithi194 ceph-mon[44089]: pgmap v2917: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T15:00:07.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:06 smithi040 ceph-mon[39581]: pgmap v2917: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-29T15:00:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:07.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/osd.1/config 2024-04-29T15:00:07.275 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:07.302 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:07.323 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:07.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:07.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:07.458 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:07.483 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:07.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:07.517 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T15:00:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-29T15:00:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-29T15:00:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-29T15:00:07.518 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T15:00:07.591 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:07.619 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:07.645 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:07.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:07.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:07.785 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:08.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:08.144 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:08.175 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:08.197 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:08.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:08.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T15:00:08.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:08.331 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:08.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:08 smithi121 ceph-mon[44533]: pgmap v2918: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 742 op/s 2024-04-29T15:00:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:08.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:08.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:08.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-29T15:00:08.992 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:09.021 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:09.042 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:09.042 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:08 smithi040 ceph-mon[39581]: pgmap v2918: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 742 op/s 2024-04-29T15:00:09.045 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:08 smithi194 ceph-mon[44089]: pgmap v2918: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 742 op/s 2024-04-29T15:00:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:09.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:09.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:09.175 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:09.497 INFO:teuthology.orchestra.run.smithi040.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-29T15:00:10.212 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:10.234 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:10.259 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:10.287 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:10.316 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:10.325 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:10.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:10.351 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:10.381 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:10.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1701650837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:10.404 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:10.426 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:10.430 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:10.456 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:10.459 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:10.486 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:10.487 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:10.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:10.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:10.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:10.488 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:10.490 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:10.491 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:10.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:10.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1701650837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:10.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1701650837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:10.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:10.627 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:10.649 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:10.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:10.672 INFO:teuthology.orchestra.run.smithi040.stdout:get_pg_num_history pg_num_history pg_num_history(e1121 pg_nums {1={14=1},9={1065=16}} deleted_pools ) 2024-04-29T15:00:10.673 INFO:teuthology.orchestra.run.smithi040.stdout:Importing pgid 9.2 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40017731:::benchmark_data_smithi194_196924_object139562:head# 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4001aa42:::benchmark_data_smithi194_196924_object142247:head# 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40029627:::benchmark_data_smithi194_196924_object102988:head# 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4003f7ea:::benchmark_data_smithi194_196924_object106917:head# 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4004d802:::benchmark_data_smithi194_196924_object146458:head# 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4005d108:::benchmark_data_smithi194_196924_object157693:head# 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:400a40f9:::benchmark_data_smithi194_196924_object123341:head# 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:400aba72:::benchmark_data_smithi194_196924_object103957:head# 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:400b75ed:::benchmark_data_smithi194_196924_object141664:head# 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:400cad17:::benchmark_data_smithi194_196924_object96203:head# 2024-04-29T15:00:10.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:400cd5c2:::benchmark_data_smithi194_196924_object138892:head# 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:400df348:::benchmark_data_smithi194_196924_object135565:head# 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:400eb2b7:::benchmark_data_smithi194_196924_object114768:head# 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4011b6ab:::benchmark_data_smithi194_196924_object144287:head# 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4013b8b6:::benchmark_data_smithi194_196924_object143665:head# 2024-04-29T15:00:10.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4013f288:::benchmark_data_smithi194_196924_object132940:head# 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4014ccbb:::benchmark_data_smithi194_196924_object124150:head# 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:401591ec:::benchmark_data_smithi194_196924_object155809:head# 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40178eb3:::benchmark_data_smithi194_196924_object114232:head# 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4017fd06:::benchmark_data_smithi194_196924_object119273:head# 2024-04-29T15:00:10.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.702 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40180621:::benchmark_data_smithi194_196924_object101811:head# 2024-04-29T15:00:10.702 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.702 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4018738c:::benchmark_data_smithi194_196924_object103666:head# 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:401afddd:::benchmark_data_smithi194_196924_object150888:head# 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:401cc8a1:::benchmark_data_smithi194_196924_object119323:head# 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:401de47a:::benchmark_data_smithi194_196924_object147500:head# 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:401e7678:::benchmark_data_smithi194_196924_object157873:head# 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.703 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:401f0566:::benchmark_data_smithi194_196924_object97111:head# 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40202dc7:::benchmark_data_smithi194_196924_object143317:head# 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40213a71:::benchmark_data_smithi194_196924_object98745:head# 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4021f7e9:::benchmark_data_smithi194_196924_object104611:head# 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4024afae:::benchmark_data_smithi194_196924_object100841:head# 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4024bc0f:::benchmark_data_smithi194_196924_object112871:head# 2024-04-29T15:00:10.704 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4027164c:::benchmark_data_smithi194_196924_object136833:head# 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4029aa73:::benchmark_data_smithi194_196924_object142969:head# 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402b17ce:::benchmark_data_smithi194_196924_object137827:head# 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402cc30f:::benchmark_data_smithi194_196924_object140538:head# 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402db6ca:::benchmark_data_smithi194_196924_object153577:head# 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.705 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402e2d59:::benchmark_data_smithi194_196924_object152693:head# 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402e41e0:::benchmark_data_smithi194_196924_object144797:head# 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402e5a79:::benchmark_data_smithi194_196924_object157139:head# 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402f1161:::benchmark_data_smithi194_196924_object148124:head# 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:402f7e8b:::benchmark_data_smithi194_196924_object100572:head# 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.706 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40304b51:::benchmark_data_smithi194_196924_object112895:head# 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:403064d1:::benchmark_data_smithi194_196924_object156498:head# 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4030d584:::benchmark_data_smithi194_196924_object136880:head# 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40310fb1:::benchmark_data_smithi194_196924_object107940:head# 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40315a79:::benchmark_data_smithi194_196924_object109812:head# 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4032cc55:::benchmark_data_smithi194_196924_object138109:head# 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4033013e:::benchmark_data_smithi194_196924_object128016:head# 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4033376b:::benchmark_data_smithi194_196924_object110778:head# 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4033ad9f:::benchmark_data_smithi194_196924_object147568:head# 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:403574b6:::benchmark_data_smithi194_196924_object159160:head# 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4035aa10:::benchmark_data_smithi194_196924_object144026:head# 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.708 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4035e472:::benchmark_data_smithi194_196924_object148439:head# 2024-04-29T15:00:10.709 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.709 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40366012:::benchmark_data_smithi194_196924_object97436:head# 2024-04-29T15:00:10.709 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.709 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4036b563:::benchmark_data_smithi194_196924_object143190:head# 2024-04-29T15:00:10.709 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40380e46:::benchmark_data_smithi194_196924_object119983:head# 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:403ad1e2:::benchmark_data_smithi194_196924_object95975:head# 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:403be672:::benchmark_data_smithi194_196924_object156333:head# 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:403cb3e0:::benchmark_data_smithi194_196924_object98744:head# 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:403cb6c7:::benchmark_data_smithi194_196924_object130644:head# 2024-04-29T15:00:10.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:403d54c5:::benchmark_data_smithi194_196924_object116977:head# 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4040c71a:::benchmark_data_smithi194_196924_object121606:head# 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4040dd3d:::benchmark_data_smithi194_196924_object129766:head# 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40414d1d:::benchmark_data_smithi194_196924_object126689:head# 2024-04-29T15:00:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4041b2ff:::benchmark_data_smithi194_196924_object112319:head# 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4041bf94:::benchmark_data_smithi194_196924_object97726:head# 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4041ea51:::benchmark_data_smithi194_196924_object113272:head# 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40426f0d:::benchmark_data_smithi194_196924_object128560:head# 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404297b4:::benchmark_data_smithi194_196924_object122071:head# 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4042e074:::benchmark_data_smithi194_196924_object97574:head# 2024-04-29T15:00:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40438855:::benchmark_data_smithi194_196924_object97320:head# 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404407b4:::benchmark_data_smithi194_196924_object129050:head# 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404409f7:::benchmark_data_smithi194_196924_object158501:head# 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404427ac:::benchmark_data_smithi194_196924_object96329:head# 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40446f80:::benchmark_data_smithi194_196924_object126500:head# 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404487b7:::benchmark_data_smithi194_196924_object126914:head# 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4045e3ce:::benchmark_data_smithi194_196924_object97332:head# 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40460c97:::benchmark_data_smithi194_196924_object130207:head# 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40464259:::benchmark_data_smithi194_196924_object101463:head# 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404785db:::benchmark_data_smithi194_196924_object124472:head# 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404cecc3:::benchmark_data_smithi194_196924_object139534:head# 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404d7b26:::benchmark_data_smithi194_196924_object156878:head# 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404da09d:::benchmark_data_smithi194_196924_object131555:head# 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404e7694:::benchmark_data_smithi194_196924_object100058:head# 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:404f960c:::benchmark_data_smithi194_196924_object120008:head# 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40514bea:::benchmark_data_smithi194_196924_object157346:head# 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40516d9c:::benchmark_data_smithi194_196924_object123148:head# 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4052856f:::benchmark_data_smithi194_196924_object149337:head# 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40570e02:::benchmark_data_smithi194_196924_object128757:head# 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:405a5bfc:::benchmark_data_smithi194_196924_object151412:head# 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:405ba788:::benchmark_data_smithi194_196924_object139527:head# 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:405cf5cb:::benchmark_data_smithi194_196924_object132890:head# 2024-04-29T15:00:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:405ef8e0:::benchmark_data_smithi194_196924_object106505:head# 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:405f9acb:::benchmark_data_smithi194_196924_object144954:head# 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:405fd5b8:::benchmark_data_smithi194_196924_object140240:head# 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40601fd7:::benchmark_data_smithi194_196924_object146371:head# 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406025ef:::benchmark_data_smithi194_196924_object126130:head# 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406029f0:::benchmark_data_smithi194_196924_object103281:head# 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40614826:::benchmark_data_smithi194_196924_object112839:head# 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40619c68:::benchmark_data_smithi194_196924_object120567:head# 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40641e48:::benchmark_data_smithi194_196924_object102505:head# 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40650bd0:::benchmark_data_smithi194_196924_object153230:head# 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406871eb:::benchmark_data_smithi194_196924_object101198:head# 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4068d34a:::benchmark_data_smithi194_196924_object121110:head# 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406998d8:::benchmark_data_smithi194_196924_object117560:head# 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406a0fc2:::benchmark_data_smithi194_196924_object121838:head# 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406ac8d4:::benchmark_data_smithi194_196924_object104174:head# 2024-04-29T15:00:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406b4157:::benchmark_data_smithi194_196924_object148760:head# 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406b5f38:::benchmark_data_smithi194_196924_object120040:head# 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406db30f:::benchmark_data_smithi194_196924_object100911:head# 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406e6197:::benchmark_data_smithi194_196924_object115797:head# 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406e6631:::benchmark_data_smithi194_196924_object115109:head# 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406ec7d6:::benchmark_data_smithi194_196924_object144536:head# 2024-04-29T15:00:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.743 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406f30c1:::benchmark_data_smithi194_196924_object146287:head# 2024-04-29T15:00:10.743 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.743 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:406f7138:::benchmark_data_smithi194_196924_object124794:head# 2024-04-29T15:00:10.743 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.743 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40701dc0:::benchmark_data_smithi194_196924_object146954:head# 2024-04-29T15:00:10.743 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40706b7b:::benchmark_data_smithi194_196924_object125614:head# 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4070a052:::benchmark_data_smithi194_196924_object125243:head# 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4070d44f:::benchmark_data_smithi194_196924_object130513:head# 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40734d26:::benchmark_data_smithi194_196924_object112702:head# 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40744aee:::benchmark_data_smithi194_196924_object140388:head# 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4075b4ea:::benchmark_data_smithi194_196924_object127663:head# 2024-04-29T15:00:10.744 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4075f79d:::benchmark_data_smithi194_196924_object117139:head# 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4076d256:::benchmark_data_smithi194_196924_object158247:head# 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:407756ae:::benchmark_data_smithi194_196924_object146491:head# 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4077e945:::benchmark_data_smithi194_196924_object124218:head# 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4077ed58:::benchmark_data_smithi194_196924_object108061:head# 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:407a4e63:::benchmark_data_smithi194_196924_object141512:head# 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.745 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:407aa3af:::benchmark_data_smithi194_196924_object110281:head# 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:407d8e51:::benchmark_data_smithi194_196924_object126622:head# 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:407eb087:::benchmark_data_smithi194_196924_object132150:head# 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:407eb807:::benchmark_data_smithi194_196924_object126451:head# 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:407f34a7:::benchmark_data_smithi194_196924_object113338:head# 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4080ede9:::benchmark_data_smithi194_196924_object148235:head# 2024-04-29T15:00:10.746 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4081a642:::benchmark_data_smithi194_196924_object97397:head# 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40833d75:::benchmark_data_smithi194_196924_object118046:head# 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40861d93:::benchmark_data_smithi194_196924_object128101:head# 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:408693de:::benchmark_data_smithi194_196924_object128030:head# 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:408827d5:::benchmark_data_smithi194_196924_object104517:head# 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40893378:::benchmark_data_smithi194_196924_object147995:head# 2024-04-29T15:00:10.747 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:408d4ddc:::benchmark_data_smithi194_196924_object118069:head# 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:408d74e6:::benchmark_data_smithi194_196924_object129456:head# 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:408dfe72:::benchmark_data_smithi194_196924_object155910:head# 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:408edad6:::benchmark_data_smithi194_196924_object159596:head# 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:408f61c9:::benchmark_data_smithi194_196924_object125548:head# 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4094a335:::benchmark_data_smithi194_196924_object97324:head# 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.748 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40971c08:::benchmark_data_smithi194_196924_object138867:head# 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4097b0a7:::benchmark_data_smithi194_196924_object97357:head# 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4098159b:::benchmark_data_smithi194_196924_object116459:head# 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40982b2f:::benchmark_data_smithi194_196924_object130200:head# 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40983e2c:::benchmark_data_smithi194_196924_object148281:head# 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4099d74d:::benchmark_data_smithi194_196924_object124066:head# 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:409a3029:::benchmark_data_smithi194_196924_object104243:head# 2024-04-29T15:00:10.749 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:409a3aad:::benchmark_data_smithi194_196924_object158638:head# 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:409b19de:::benchmark_data_smithi194_196924_object152481:head# 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:409e6503:::benchmark_data_smithi194_196924_object110657:head# 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:409f5f2e:::benchmark_data_smithi194_196924_object98583:head# 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a100eb:::benchmark_data_smithi194_196924_object107037:head# 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a1adf4:::benchmark_data_smithi194_196924_object120159:head# 2024-04-29T15:00:10.750 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a22d95:::benchmark_data_smithi194_196924_object98016:head# 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a24479:::benchmark_data_smithi194_196924_object110277:head# 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a41a84:::benchmark_data_smithi194_196924_object101540:head# 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a57a99:::benchmark_data_smithi194_196924_object129515:head# 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a5d261:::benchmark_data_smithi194_196924_object132250:head# 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a5eb42:::benchmark_data_smithi194_196924_object104860:head# 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a6bd04:::benchmark_data_smithi194_196924_object153179:head# 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a787d2:::benchmark_data_smithi194_196924_object112717:head# 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a7a620:::benchmark_data_smithi194_196924_object100990:head# 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a7f40c:::benchmark_data_smithi194_196924_object121479:head# 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a85b34:::benchmark_data_smithi194_196924_object149501:head# 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a879df:::benchmark_data_smithi194_196924_object131702:head# 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40a989cc:::benchmark_data_smithi194_196924_object145551:head# 2024-04-29T15:00:10.752 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40aac509:::benchmark_data_smithi194_196924_object96113:head# 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40ac25e6:::benchmark_data_smithi194_196924_object110327:head# 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40acdc68:::benchmark_data_smithi194_196924_object150116:head# 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40aeb4b4:::benchmark_data_smithi194_196924_object130003:head# 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b05f11:::benchmark_data_smithi194_196924_object109778:head# 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b15d75:::benchmark_data_smithi194_196924_object123259:head# 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.753 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b1bb55:::benchmark_data_smithi194_196924_object139256:head# 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b1db12:::benchmark_data_smithi194_196924_object123157:head# 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b1fa8c:::benchmark_data_smithi194_196924_object99634:head# 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b299b3:::benchmark_data_smithi194_196924_object114150:head# 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b4fbd8:::benchmark_data_smithi194_196924_object108412:head# 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b50d69:::benchmark_data_smithi194_196924_object149876:head# 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b5ab8b:::benchmark_data_smithi194_196924_object98400:head# 2024-04-29T15:00:10.754 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40b7fdc7:::benchmark_data_smithi194_196924_object138617:head# 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40babddc:::benchmark_data_smithi194_196924_object154103:head# 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40bb0780:::benchmark_data_smithi194_196924_object118133:head# 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40bd169b:::benchmark_data_smithi194_196924_object120266:head# 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40bd1a46:::benchmark_data_smithi194_196924_object111241:head# 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40beb6b4:::benchmark_data_smithi194_196924_object122108:head# 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40c0fa09:::benchmark_data_smithi194_196924_object152867:head# 2024-04-29T15:00:10.755 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40c2eb54:::benchmark_data_smithi194_196924_object143900:head# 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40c61fff:::benchmark_data_smithi194_196924_object148744:head# 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40c719f8:::benchmark_data_smithi194_196924_object122448:head# 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40c747d4:::benchmark_data_smithi194_196924_object128923:head# 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40c7bb7f:::benchmark_data_smithi194_196924_object110870:head# 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40ced2f9:::benchmark_data_smithi194_196924_object109966:head# 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.756 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40cf775e:::benchmark_data_smithi194_196924_object119947:head# 2024-04-29T15:00:10.757 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.757 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40d1f962:::benchmark_data_smithi194_196924_object155275:head# 2024-04-29T15:00:10.757 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.757 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:10.772 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40d37ac9:::benchmark_data_smithi194_196924_object146191:head# 2024-04-29T15:00:10.772 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40d62cbd:::benchmark_data_smithi194_196924_object141003:head# 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40d74936:::benchmark_data_smithi194_196924_object95759:head# 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40d910f6:::benchmark_data_smithi194_196924_object147302:head# 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40d93e38:::benchmark_data_smithi194_196924_object154164:head# 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40daacd9:::benchmark_data_smithi194_196924_object153904:head# 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40dbbcf7:::benchmark_data_smithi194_196924_object141312:head# 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.773 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40dbf7e3:::benchmark_data_smithi194_196924_object122697:head# 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40dc5495:::benchmark_data_smithi194_196924_object138878:head# 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40dc6f75:::benchmark_data_smithi194_196924_object137661:head# 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40dda1b9:::benchmark_data_smithi194_196924_object152682:head# 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40de08b1:::benchmark_data_smithi194_196924_object107354:head# 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40de4827:::benchmark_data_smithi194_196924_object128384:head# 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40e098d2:::benchmark_data_smithi194_196924_object155846:head# 2024-04-29T15:00:10.774 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40e51123:::benchmark_data_smithi194_196924_object126428:head# 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40e83302:::benchmark_data_smithi194_196924_object136232:head# 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40e870d1:::benchmark_data_smithi194_196924_object152745:head# 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40e89fae:::benchmark_data_smithi194_196924_object112898:head# 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40e8b02c:::benchmark_data_smithi194_196924_object149221:head# 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40ebb7ed:::benchmark_data_smithi194_196924_object106745:head# 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40ebed08:::benchmark_data_smithi194_196924_object100027:head# 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40ecb8d5:::benchmark_data_smithi194_196924_object136938:head# 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40ed8845:::benchmark_data_smithi194_196924_object116409:head# 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40f17dd8:::benchmark_data_smithi194_196924_object133318:head# 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40f1e425:::benchmark_data_smithi194_196924_object115495:head# 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40f28493:::benchmark_data_smithi194_196924_object156830:head# 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40f2fd94:::benchmark_data_smithi194_196924_object156962:head# 2024-04-29T15:00:10.776 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40f395b1:::benchmark_data_smithi194_196924_object95938:head# 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40f81e56:::benchmark_data_smithi194_196924_object99178:head# 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40f8cfd0:::benchmark_data_smithi194_196924_object138533:head# 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40fbf379:::benchmark_data_smithi194_196924_object134685:head# 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40fd6a5f:::benchmark_data_smithi194_196924_object137337:head# 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40fdd6bf:::benchmark_data_smithi194_196924_object102022:head# 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:40fe11b0:::benchmark_data_smithi194_196924_object112971:head# 2024-04-29T15:00:10.777 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41008e51:::benchmark_data_smithi194_196924_object97669:head# 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4100bc5e:::benchmark_data_smithi194_196924_object136348:head# 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41017ac9:::benchmark_data_smithi194_196924_object134123:head# 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41040689:::benchmark_data_smithi194_196924_object153720:head# 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4104ea44:::benchmark_data_smithi194_196924_object99786:head# 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4106806f:::benchmark_data_smithi194_196924_object150843:head# 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.778 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4106e2a5:::benchmark_data_smithi194_196924_object126859:head# 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41089211:::benchmark_data_smithi194_196924_object151991:head# 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:410945a1:::benchmark_data_smithi194_196924_object138907:head# 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41096871:::benchmark_data_smithi194_196924_object113470:head# 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41098811:::benchmark_data_smithi194_196924_object123185:head# 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:410b3885:::benchmark_data_smithi194_196924_object131047:head# 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:410b9146:::benchmark_data_smithi194_196924_object139509:head# 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.779 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:410d3442:::benchmark_data_smithi194_196924_object120852:head# 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:410d3bd1:::benchmark_data_smithi194_196924_object134693:head# 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:410dfe1d:::benchmark_data_smithi194_196924_object141606:head# 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:410e11de:::benchmark_data_smithi194_196924_object138955:head# 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4110cbcb:::benchmark_data_smithi194_196924_object149459:head# 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41135c91:::benchmark_data_smithi194_196924_object158593:head# 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4113b86e:::benchmark_data_smithi194_196924_object108414:head# 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.780 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4114d6fd:::benchmark_data_smithi194_196924_object111941:head# 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4114f8e1:::benchmark_data_smithi194_196924_object105061:head# 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4116fc86:::benchmark_data_smithi194_196924_object133411:head# 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41171918:::benchmark_data_smithi194_196924_object135726:head# 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4117d3d3:::benchmark_data_smithi194_196924_object153127:head# 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4118be51:::benchmark_data_smithi194_196924_object122483:head# 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:411ac003:::benchmark_data_smithi194_196924_object157097:head# 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.781 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:411ae5b6:::benchmark_data_smithi194_196924_object122081:head# 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:411bc292:::benchmark_data_smithi194_196924_object96753:head# 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:411bc4e3:::benchmark_data_smithi194_196924_object157161:head# 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:411bcab3:::benchmark_data_smithi194_196924_object151075:head# 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:411d8ab1:::benchmark_data_smithi194_196924_object108944:head# 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:411f96db:::benchmark_data_smithi194_196924_object134535:head# 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41203c8e:::benchmark_data_smithi194_196924_object122810:head# 2024-04-29T15:00:10.782 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4120b523:::benchmark_data_smithi194_196924_object135430:head# 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4122f48c:::benchmark_data_smithi194_196924_object139385:head# 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41242942:::benchmark_data_smithi194_196924_object148879:head# 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:412622c4:::benchmark_data_smithi194_196924_object136586:head# 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4126ce60:::benchmark_data_smithi194_196924_object119671:head# 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41285392:::benchmark_data_smithi194_196924_object115310:head# 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41288288:::benchmark_data_smithi194_196924_object122186:head# 2024-04-29T15:00:10.783 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:412a4bf9:::benchmark_data_smithi194_196924_object105856:head# 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:412b8f00:::benchmark_data_smithi194_196924_object157383:head# 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:412da477:::benchmark_data_smithi194_196924_object135690:head# 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:413002ff:::benchmark_data_smithi194_196924_object110025:head# 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41347883:::benchmark_data_smithi194_196924_object130176:head# 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4135e73f:::benchmark_data_smithi194_196924_object146093:head# 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41394c2c:::benchmark_data_smithi194_196924_object148255:head# 2024-04-29T15:00:10.784 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:413b1334:::benchmark_data_smithi194_196924_object151813:head# 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:413b67bd:::benchmark_data_smithi194_196924_object126283:head# 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:413c60cd:::benchmark_data_smithi194_196924_object140813:head# 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:413f994d:::benchmark_data_smithi194_196924_object131874:head# 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41402d88:::benchmark_data_smithi194_196924_object159555:head# 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41404fc0:::benchmark_data_smithi194_196924_object156935:head# 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:414073f1:::benchmark_data_smithi194_196924_object133826:head# 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.785 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4145db32:::benchmark_data_smithi194_196924_object129335:head# 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41490c86:::benchmark_data_smithi194_196924_object116088:head# 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4149b9be:::benchmark_data_smithi194_196924_object102010:head# 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4149bd1c:::benchmark_data_smithi194_196924_object132531:head# 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:414a2a4f:::benchmark_data_smithi194_196924_object114135:head# 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:414b503e:::benchmark_data_smithi194_196924_object99573:head# 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:414d40ba:::benchmark_data_smithi194_196924_object144528:head# 2024-04-29T15:00:10.786 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:414f03aa:::benchmark_data_smithi194_196924_object108029:head# 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:414f0555:::benchmark_data_smithi194_196924_object149571:head# 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41501f47:::benchmark_data_smithi194_196924_object108709:head# 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4150709f:::benchmark_data_smithi194_196924_object122851:head# 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415075ed:::benchmark_data_smithi194_196924_object135214:head# 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41514a95:::benchmark_data_smithi194_196924_object127952:head# 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4152546d:::benchmark_data_smithi194_196924_object138130:head# 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4152cd82:::benchmark_data_smithi194_196924_object136320:head# 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415677c2:::benchmark_data_smithi194_196924_object133388:head# 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415718bd:::benchmark_data_smithi194_196924_object132958:head# 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4158144d:::benchmark_data_smithi194_196924_object144072:head# 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41582afe:::benchmark_data_smithi194_196924_object125273:head# 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4158bc75:::benchmark_data_smithi194_196924_object157606:head# 2024-04-29T15:00:10.788 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.796 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:10.799 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415ac31a:::benchmark_data_smithi194_196924_object133013:head# 2024-04-29T15:00:10.799 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.799 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415bbe42:::benchmark_data_smithi194_196924_object137890:head# 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415ccb8f:::benchmark_data_smithi194_196924_object122606:head# 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415d4277:::benchmark_data_smithi194_196924_object101996:head# 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:415f3409:::benchmark_data_smithi194_196924_object104508:head# 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416078a0:::benchmark_data_smithi194_196924_object95965:head# 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4160b5d5:::benchmark_data_smithi194_196924_object146435:head# 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41613d91:::benchmark_data_smithi194_196924_object128530:head# 2024-04-29T15:00:10.800 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4161991c:::benchmark_data_smithi194_196924_object105515:head# 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4161e5f3:::benchmark_data_smithi194_196924_object148151:head# 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4162a7ee:::benchmark_data_smithi194_196924_object104162:head# 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416467cd:::benchmark_data_smithi194_196924_object99216:head# 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4167a095:::benchmark_data_smithi194_196924_object141515:head# 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4167a20a:::benchmark_data_smithi194_196924_object144624:head# 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41691443:::benchmark_data_smithi194_196924_object108810:head# 2024-04-29T15:00:10.801 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416966e4:::benchmark_data_smithi194_196924_object96238:head# 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4169d381:::benchmark_data_smithi194_196924_object102986:head# 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416a0767:::benchmark_data_smithi194_196924_object116631:head# 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416a86ee:::benchmark_data_smithi194_196924_object130137:head# 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416b9248:::benchmark_data_smithi194_196924_object135812:head# 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416c37e9:::benchmark_data_smithi194_196924_object116920:head# 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416c7179:::benchmark_data_smithi194_196924_object109517:head# 2024-04-29T15:00:10.802 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416d34bb:::benchmark_data_smithi194_196924_object126074:head# 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416e7f7d:::benchmark_data_smithi194_196924_object99461:head# 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416f62ab:::benchmark_data_smithi194_196924_object146122:head# 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:416facd2:::benchmark_data_smithi194_196924_object133809:head# 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4170d6e3:::benchmark_data_smithi194_196924_object130926:head# 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4171c28c:::benchmark_data_smithi194_196924_object118474:head# 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4173c9ac:::benchmark_data_smithi194_196924_object112995:head# 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.803 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4173caa0:::benchmark_data_smithi194_196924_object107071:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41746bf4:::benchmark_data_smithi194_196924_object154570:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:417688a1:::benchmark_data_smithi194_196924_object133276:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:417781aa:::benchmark_data_smithi194_196924_object98352:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4178fb9b:::benchmark_data_smithi194_196924_object138523:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41793ed3:::benchmark_data_smithi194_196924_object98648:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4179f8dd:::benchmark_data_smithi194_196924_object141527:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:417a7f8e:::benchmark_data_smithi194_196924_object137128:head# 2024-04-29T15:00:10.804 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:417e1e2f:::benchmark_data_smithi194_196924_object158670:head# 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:417f7b28:::benchmark_data_smithi194_196924_object116174:head# 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4180d6b2:::benchmark_data_smithi194_196924_object117355:head# 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4181961c:::benchmark_data_smithi194_196924_object104253:head# 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41824c14:::benchmark_data_smithi194_196924_object116432:head# 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41826f28:::benchmark_data_smithi194_196924_object102202:head# 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41836ed1:::benchmark_data_smithi194_196924_object149005:head# 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.805 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:418599a7:::benchmark_data_smithi194_196924_object104425:head# 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4188de69:::benchmark_data_smithi194_196924_object102850:head# 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4188e546:::benchmark_data_smithi194_196924_object96088:head# 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41898356:::benchmark_data_smithi194_196924_object125454:head# 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:418b2cf9:::benchmark_data_smithi194_196924_object106310:head# 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:418b369e:::benchmark_data_smithi194_196924_object96018:head# 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:418df071:::benchmark_data_smithi194_196924_object152710:head# 2024-04-29T15:00:10.806 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:418e2512:::benchmark_data_smithi194_196924_object118041:head# 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41918b97:::benchmark_data_smithi194_196924_object133264:head# 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:419239a5:::benchmark_data_smithi194_196924_object144038:head# 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41931a87:::benchmark_data_smithi194_196924_object141303:head# 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41935aff:::benchmark_data_smithi194_196924_object138647:head# 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41949e78:::benchmark_data_smithi194_196924_object102570:head# 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4195d743:::benchmark_data_smithi194_196924_object99942:head# 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.807 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4199b0b3:::benchmark_data_smithi194_196924_object122562:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:419a1ba4:::benchmark_data_smithi194_196924_object117772:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:419bb6b8:::benchmark_data_smithi194_196924_object98222:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:419c05c2:::benchmark_data_smithi194_196924_object122158:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:419c3f7c:::benchmark_data_smithi194_196924_object143211:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:419d167a:::benchmark_data_smithi194_196924_object153550:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:419d4417:::benchmark_data_smithi194_196924_object105972:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a476f6:::benchmark_data_smithi194_196924_object157388:head# 2024-04-29T15:00:10.808 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a4cc1f:::benchmark_data_smithi194_196924_object106711:head# 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a580c9:::benchmark_data_smithi194_196924_object141234:head# 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a5aef8:::benchmark_data_smithi194_196924_object105807:head# 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a68004:::benchmark_data_smithi194_196924_object138217:head# 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a6b996:::benchmark_data_smithi194_196924_object150850:head# 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a6e0a6:::benchmark_data_smithi194_196924_object158474:head# 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a8da13:::benchmark_data_smithi194_196924_object131203:head# 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.809 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a9ab1c:::benchmark_data_smithi194_196924_object108087:head# 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41a9c2b0:::benchmark_data_smithi194_196924_object151891:head# 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41aa84a0:::benchmark_data_smithi194_196924_object131568:head# 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41aaad27:::benchmark_data_smithi194_196924_object137459:head# 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ab5e2c:::benchmark_data_smithi194_196924_object121387:head# 2024-04-29T15:00:10.810 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:10.819 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:10.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ab9cd7:::benchmark_data_smithi194_196924_object105518:head# 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ac5a8c:::benchmark_data_smithi194_196924_object150172:head# 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41add528:::benchmark_data_smithi194_196924_object95948:head# 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41af787b:::benchmark_data_smithi194_196924_object125638:head# 2024-04-29T15:00:10.823 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41b12e93:::benchmark_data_smithi194_196924_object115815:head# 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41b142e1:::benchmark_data_smithi194_196924_object124902:head# 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41b39240:::benchmark_data_smithi194_196924_object108840:head# 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41b39b44:::benchmark_data_smithi194_196924_object123840:head# 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41b8497f:::benchmark_data_smithi194_196924_object129299:head# 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41b8557b:::benchmark_data_smithi194_196924_object135375:head# 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41b99c12:::benchmark_data_smithi194_196924_object148324:head# 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.824 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41bae27d:::benchmark_data_smithi194_196924_object158382:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41bcdd90:::benchmark_data_smithi194_196924_object97268:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41be0782:::benchmark_data_smithi194_196924_object96475:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41bf747d:::benchmark_data_smithi194_196924_object100942:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41bfb903:::benchmark_data_smithi194_196924_object115046:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41bfd2ae:::benchmark_data_smithi194_196924_object136309:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c00157:::benchmark_data_smithi194_196924_object147146:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c361d3:::benchmark_data_smithi194_196924_object152087:head# 2024-04-29T15:00:10.825 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c3f8ec:::benchmark_data_smithi194_196924_object137179:head# 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c42fa6:::benchmark_data_smithi194_196924_object152419:head# 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c4e47e:::benchmark_data_smithi194_196924_object107741:head# 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c530d2:::benchmark_data_smithi194_196924_object105267:head# 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c6c308:::benchmark_data_smithi194_196924_object97893:head# 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c74332:::benchmark_data_smithi194_196924_object139104:head# 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c8daf2:::benchmark_data_smithi194_196924_object119658:head# 2024-04-29T15:00:10.826 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c94199:::benchmark_data_smithi194_196924_object99682:head# 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41c9900d:::benchmark_data_smithi194_196924_object155098:head# 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41cd2f56:::benchmark_data_smithi194_196924_object118516:head# 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41cd31d3:::benchmark_data_smithi194_196924_object121993:head# 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ce07c3:::benchmark_data_smithi194_196924_object158772:head# 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d04577:::benchmark_data_smithi194_196924_object114816:head# 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d14608:::benchmark_data_smithi194_196924_object133808:head# 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d19f95:::benchmark_data_smithi194_196924_object129441:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d274a7:::benchmark_data_smithi194_196924_object117554:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d3cb62:::benchmark_data_smithi194_196924_object114160:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d4678a:::benchmark_data_smithi194_196924_object146671:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d50440:::benchmark_data_smithi194_196924_object95837:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d51def:::benchmark_data_smithi194_196924_object95599:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d55ce3:::benchmark_data_smithi194_196924_object127745:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d7d6ba:::benchmark_data_smithi194_196924_object96750:head# 2024-04-29T15:00:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d8be6c:::benchmark_data_smithi194_196924_object131821:head# 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d941c4:::benchmark_data_smithi194_196924_object115220:head# 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41d95007:::benchmark_data_smithi194_196924_object98149:head# 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41da5c4a:::benchmark_data_smithi194_196924_object118387:head# 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41da72b2:::benchmark_data_smithi194_196924_object128636:head# 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41dadb64:::benchmark_data_smithi194_196924_object97774:head# 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41db2570:::benchmark_data_smithi194_196924_object139007:head# 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.829 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41dbb2d9:::benchmark_data_smithi194_196924_object124140:head# 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41dbebb9:::benchmark_data_smithi194_196924_object115570:head# 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41dcd8a7:::benchmark_data_smithi194_196924_object158737:head# 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41dd9ef3:::benchmark_data_smithi194_196924_object112584:head# 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41de7bcf:::benchmark_data_smithi194_196924_object113073:head# 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41df89ff:::benchmark_data_smithi194_196924_object148293:head# 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41dff837:::benchmark_data_smithi194_196924_object122902:head# 2024-04-29T15:00:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e0d801:::benchmark_data_smithi194_196924_object109808:head# 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e4e436:::benchmark_data_smithi194_196924_object117916:head# 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e58772:::benchmark_data_smithi194_196924_object146249:head# 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e7a25a:::benchmark_data_smithi194_196924_object121695:head# 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e830b1:::benchmark_data_smithi194_196924_object121504:head# 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e833ea:::benchmark_data_smithi194_196924_object109290:head# 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e8762e:::benchmark_data_smithi194_196924_object126403:head# 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.831 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e91dce:::benchmark_data_smithi194_196924_object148556:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e9346f:::benchmark_data_smithi194_196924_object140302:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e945c1:::benchmark_data_smithi194_196924_object125356:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41e9feec:::benchmark_data_smithi194_196924_object103216:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ea274e:::benchmark_data_smithi194_196924_object134479:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41eae441:::benchmark_data_smithi194_196924_object128963:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ecd118:::benchmark_data_smithi194_196924_object114100:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ef6acd:::benchmark_data_smithi194_196924_object97539:head# 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.832 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f07b97:::benchmark_data_smithi194_196924_object132820:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f0b672:::benchmark_data_smithi194_196924_object138780:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f161b6:::benchmark_data_smithi194_196924_object120612:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f1f8b6:::benchmark_data_smithi194_196924_object108286:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f21a36:::benchmark_data_smithi194_196924_object129406:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f347d0:::benchmark_data_smithi194_196924_object146118:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f37f85:::benchmark_data_smithi194_196924_object139922:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f46e68:::benchmark_data_smithi194_196924_object111164:head# 2024-04-29T15:00:10.833 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.834 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f77fd1:::benchmark_data_smithi194_196924_object143612:head# 2024-04-29T15:00:10.834 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.834 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f7ea35:::benchmark_data_smithi194_196924_object108507:head# 2024-04-29T15:00:10.834 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.834 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f801d1:::benchmark_data_smithi194_196924_object120995:head# 2024-04-29T15:00:10.834 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.841 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f86cb7:::benchmark_data_smithi194_196924_object108897:head# 2024-04-29T15:00:10.841 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.841 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f91810:::benchmark_data_smithi194_196924_object138441:head# 2024-04-29T15:00:10.841 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.841 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41f93bac:::benchmark_data_smithi194_196924_object112902:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fa0c14:::benchmark_data_smithi194_196924_object155132:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fa5c1b:::benchmark_data_smithi194_196924_object145807:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fbee23:::benchmark_data_smithi194_196924_object153667:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fc4e85:::benchmark_data_smithi194_196924_object141161:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fc98e2:::benchmark_data_smithi194_196924_object100887:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fca02f:::benchmark_data_smithi194_196924_object138991:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fd116f:::benchmark_data_smithi194_196924_object142628:head# 2024-04-29T15:00:10.842 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fdcfa8:::benchmark_data_smithi194_196924_object101945:head# 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41fed233:::benchmark_data_smithi194_196924_object136245:head# 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ff8dab:::benchmark_data_smithi194_196924_object143683:head# 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:41ffdedf:::benchmark_data_smithi194_196924_object133402:head# 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42025e1d:::benchmark_data_smithi194_196924_object96123:head# 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4202ad20:::benchmark_data_smithi194_196924_object107033:head# 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4202db42:::benchmark_data_smithi194_196924_object139830:head# 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.843 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4205c7bc:::benchmark_data_smithi194_196924_object116739:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4206f34f:::benchmark_data_smithi194_196924_object116784:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:420701d0:::benchmark_data_smithi194_196924_object147403:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42072aad:::benchmark_data_smithi194_196924_object139502:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4207794b:::benchmark_data_smithi194_196924_object109371:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42080698:::benchmark_data_smithi194_196924_object144912:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:420a1fd1:::benchmark_data_smithi194_196924_object122035:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:420a4163:::benchmark_data_smithi194_196924_object153776:head# 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.844 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:420a7123:::benchmark_data_smithi194_196924_object121239:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:420d6b04:::benchmark_data_smithi194_196924_object103402:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:420d6d30:::benchmark_data_smithi194_196924_object112377:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:420f0d46:::benchmark_data_smithi194_196924_object115586:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42107c55:::benchmark_data_smithi194_196924_object99099:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42110ef6:::benchmark_data_smithi194_196924_object97473:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:421173e4:::benchmark_data_smithi194_196924_object97925:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:421200e0:::benchmark_data_smithi194_196924_object119637:head# 2024-04-29T15:00:10.845 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42135e99:::benchmark_data_smithi194_196924_object114457:head# 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42149b3e:::benchmark_data_smithi194_196924_object108854:head# 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4214a35f:::benchmark_data_smithi194_196924_object98538:head# 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4216f8fe:::benchmark_data_smithi194_196924_object153532:head# 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4218d608:::benchmark_data_smithi194_196924_object117188:head# 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42194869:::benchmark_data_smithi194_196924_object114016:head# 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:421bba27:::benchmark_data_smithi194_196924_object99464:head# 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:421dabc7:::benchmark_data_smithi194_196924_object112732:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4220565a:::benchmark_data_smithi194_196924_object143035:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42225fd2:::benchmark_data_smithi194_196924_object148520:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42235d35:::benchmark_data_smithi194_196924_object121284:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4225d970:::benchmark_data_smithi194_196924_object146756:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42291be7:::benchmark_data_smithi194_196924_object145553:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:422d9e06:::benchmark_data_smithi194_196924_object129563:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:422e657e:::benchmark_data_smithi194_196924_object130924:head# 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.847 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:422f379b:::benchmark_data_smithi194_196924_object99635:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4230ddcd:::benchmark_data_smithi194_196924_object153969:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4234008f:::benchmark_data_smithi194_196924_object105562:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4235aed6:::benchmark_data_smithi194_196924_object130675:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4235f4c4:::benchmark_data_smithi194_196924_object156651:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4236555d:::benchmark_data_smithi194_196924_object106070:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42368bd8:::benchmark_data_smithi194_196924_object119156:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42370fdc:::benchmark_data_smithi194_196924_object150761:head# 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.848 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42371da5:::benchmark_data_smithi194_196924_object98963:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4237757f:::benchmark_data_smithi194_196924_object143711:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42377fc9:::benchmark_data_smithi194_196924_object141680:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42387354:::benchmark_data_smithi194_196924_object103062:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42389743:::benchmark_data_smithi194_196924_object159830:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4238aa64:::benchmark_data_smithi194_196924_object132430:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:423a9c7f:::benchmark_data_smithi194_196924_object136151:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:423ac7ca:::benchmark_data_smithi194_196924_object113372:head# 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.849 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:423b99bc:::benchmark_data_smithi194_196924_object134989:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:423c6042:::benchmark_data_smithi194_196924_object112527:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:423f510e:::benchmark_data_smithi194_196924_object156260:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:423f8113:::benchmark_data_smithi194_196924_object150010:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:423fc0b4:::benchmark_data_smithi194_196924_object157672:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4240b4d5:::benchmark_data_smithi194_196924_object107790:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4240de17:::benchmark_data_smithi194_196924_object141125:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424109ef:::benchmark_data_smithi194_196924_object154953:head# 2024-04-29T15:00:10.850 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424122e3:::benchmark_data_smithi194_196924_object156464:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424224c6:::benchmark_data_smithi194_196924_object153374:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42435855:::benchmark_data_smithi194_196924_object144692:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424482af:::benchmark_data_smithi194_196924_object104851:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42456332:::benchmark_data_smithi194_196924_object148907:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4247256f:::benchmark_data_smithi194_196924_object108117:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4247c8a7:::benchmark_data_smithi194_196924_object159532:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424809e5:::benchmark_data_smithi194_196924_object115745:head# 2024-04-29T15:00:10.851 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.861 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42486414:::benchmark_data_smithi194_196924_object99278:head# 2024-04-29T15:00:10.861 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.861 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4248e55f:::benchmark_data_smithi194_196924_object121380:head# 2024-04-29T15:00:10.861 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4248ea55:::benchmark_data_smithi194_196924_object135681:head# 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424a03a7:::benchmark_data_smithi194_196924_object102253:head# 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424b7f1b:::benchmark_data_smithi194_196924_object149858:head# 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424bc7d3:::benchmark_data_smithi194_196924_object157206:head# 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424db7a1:::benchmark_data_smithi194_196924_object96942:head# 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424e06f8:::benchmark_data_smithi194_196924_object132534:head# 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:424f1256:::benchmark_data_smithi194_196924_object158046:head# 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.862 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4251e80e:::benchmark_data_smithi194_196924_object116043:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42521d09:::benchmark_data_smithi194_196924_object121303:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42521fd7:::benchmark_data_smithi194_196924_object120669:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:425257f3:::benchmark_data_smithi194_196924_object128409:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:425344c8:::benchmark_data_smithi194_196924_object126458:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42534c22:::benchmark_data_smithi194_196924_object104246:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4255fdc9:::benchmark_data_smithi194_196924_object120564:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42574f76:::benchmark_data_smithi194_196924_object120665:head# 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.863 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42590eea:::benchmark_data_smithi194_196924_object119291:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4259fca3:::benchmark_data_smithi194_196924_object115610:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:425c87d8:::benchmark_data_smithi194_196924_object113623:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:425f1264:::benchmark_data_smithi194_196924_object156131:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42617ec4:::benchmark_data_smithi194_196924_object158472:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4263a9ea:::benchmark_data_smithi194_196924_object117513:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42644dd9:::benchmark_data_smithi194_196924_object102588:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:426526a9:::benchmark_data_smithi194_196924_object131887:head# 2024-04-29T15:00:10.864 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:426592df:::benchmark_data_smithi194_196924_object142713:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4265de3e:::benchmark_data_smithi194_196924_object98160:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4266722a:::benchmark_data_smithi194_196924_object100128:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4266dd0b:::benchmark_data_smithi194_196924_object130999:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42677872:::benchmark_data_smithi194_196924_object119224:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42677cb5:::benchmark_data_smithi194_196924_object111281:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4267de64:::benchmark_data_smithi194_196924_object117536:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42681602:::benchmark_data_smithi194_196924_object139013:head# 2024-04-29T15:00:10.865 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:426832a3:::benchmark_data_smithi194_196924_object111543:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42688d92:::benchmark_data_smithi194_196924_object117032:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4269852a:::benchmark_data_smithi194_196924_object126336:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:426c7cb0:::benchmark_data_smithi194_196924_object101646:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:426cc5d9:::benchmark_data_smithi194_196924_object124018:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:426cdea0:::benchmark_data_smithi194_196924_object141728:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:426e5a4d:::benchmark_data_smithi194_196924_object143607:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4270e64c:::benchmark_data_smithi194_196924_object159342:head# 2024-04-29T15:00:10.866 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:427137c5:::benchmark_data_smithi194_196924_object102681:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:427165d1:::benchmark_data_smithi194_196924_object141452:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4273ac4d:::benchmark_data_smithi194_196924_object148334:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4274f52a:::benchmark_data_smithi194_196924_object113162:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4275abad:::benchmark_data_smithi194_196924_object147169:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4275acb4:::benchmark_data_smithi194_196924_object154018:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4276335b:::benchmark_data_smithi194_196924_object155628:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4279ba4c:::benchmark_data_smithi194_196924_object138534:head# 2024-04-29T15:00:10.867 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4279dde3:::benchmark_data_smithi194_196924_object139962:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:427dbc86:::benchmark_data_smithi194_196924_object135606:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:427e577d:::benchmark_data_smithi194_196924_object102189:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:427f393f:::benchmark_data_smithi194_196924_object145489:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:427f3d5a:::benchmark_data_smithi194_196924_object109687:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4280bd2b:::benchmark_data_smithi194_196924_object136617:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:428257a2:::benchmark_data_smithi194_196924_object123646:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4284d56b:::benchmark_data_smithi194_196924_object132342:head# 2024-04-29T15:00:10.868 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42854f8f:::benchmark_data_smithi194_196924_object122375:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4287a7c4:::benchmark_data_smithi194_196924_object148940:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42882a22:::benchmark_data_smithi194_196924_object142526:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:428836c7:::benchmark_data_smithi194_196924_object123115:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4288c000:::benchmark_data_smithi194_196924_object129413:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:428cb464:::benchmark_data_smithi194_196924_object131543:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:428d2d32:::benchmark_data_smithi194_196924_object136029:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:428e63c9:::benchmark_data_smithi194_196924_object139761:head# 2024-04-29T15:00:10.869 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:428ebd8b:::benchmark_data_smithi194_196924_object155690:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:428f0546:::benchmark_data_smithi194_196924_object127594:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4290a2a1:::benchmark_data_smithi194_196924_object124204:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4291d0a1:::benchmark_data_smithi194_196924_object120511:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42951fb7:::benchmark_data_smithi194_196924_object134507:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4296240f:::benchmark_data_smithi194_196924_object97180:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4296259e:::benchmark_data_smithi194_196924_object142888:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:429632c4:::benchmark_data_smithi194_196924_object126824:head# 2024-04-29T15:00:10.870 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4297f2b9:::benchmark_data_smithi194_196924_object130325:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:429ce803:::benchmark_data_smithi194_196924_object97590:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:429e2ff6:::benchmark_data_smithi194_196924_object139982:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:429e665f:::benchmark_data_smithi194_196924_object144806:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:429f13e8:::benchmark_data_smithi194_196924_object114788:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42a15db0:::benchmark_data_smithi194_196924_object118881:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42a50324:::benchmark_data_smithi194_196924_object155354:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42a516d6:::benchmark_data_smithi194_196924_object135127:head# 2024-04-29T15:00:10.871 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.883 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42a517d1:::benchmark_data_smithi194_196924_object153998:head# 2024-04-29T15:00:10.883 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.883 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42a58b5b:::benchmark_data_smithi194_196924_object118842:head# 2024-04-29T15:00:10.883 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.883 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42a5e82c:::benchmark_data_smithi194_196924_object116031:head# 2024-04-29T15:00:10.883 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.883 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42a86c94:::benchmark_data_smithi194_196924_object137140:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ab15b0:::benchmark_data_smithi194_196924_object134611:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ac3674:::benchmark_data_smithi194_196924_object132648:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42adc987:::benchmark_data_smithi194_196924_object144170:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42af7f8f:::benchmark_data_smithi194_196924_object101879:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b2191c:::benchmark_data_smithi194_196924_object135188:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b2fb41:::benchmark_data_smithi194_196924_object139409:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b3a20f:::benchmark_data_smithi194_196924_object154485:head# 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.884 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b44e2f:::benchmark_data_smithi194_196924_object112626:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b49f73:::benchmark_data_smithi194_196924_object155079:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b4d4c3:::benchmark_data_smithi194_196924_object159671:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b52889:::benchmark_data_smithi194_196924_object137471:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b563d2:::benchmark_data_smithi194_196924_object124854:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b5c1b2:::benchmark_data_smithi194_196924_object101305:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b65aea:::benchmark_data_smithi194_196924_object159553:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b70d61:::benchmark_data_smithi194_196924_object156632:head# 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.885 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b8b3fd:::benchmark_data_smithi194_196924_object121968:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b97a9b:::benchmark_data_smithi194_196924_object131341:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42b9a346:::benchmark_data_smithi194_196924_object111704:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ba441e:::benchmark_data_smithi194_196924_object103700:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42baea0c:::benchmark_data_smithi194_196924_object132877:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42bc2bc6:::benchmark_data_smithi194_196924_object134691:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42bd5410:::benchmark_data_smithi194_196924_object98925:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42bd6928:::benchmark_data_smithi194_196924_object134978:head# 2024-04-29T15:00:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42bd7f43:::benchmark_data_smithi194_196924_object148864:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42be1df7:::benchmark_data_smithi194_196924_object104942:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42c1c78f:::benchmark_data_smithi194_196924_object155125:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42c3b17a:::benchmark_data_smithi194_196924_object121513:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42c4dc0a:::benchmark_data_smithi194_196924_object143404:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42c52013:::benchmark_data_smithi194_196924_object144392:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42c79df7:::benchmark_data_smithi194_196924_object106854:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42c82fc8:::benchmark_data_smithi194_196924_object143490:head# 2024-04-29T15:00:10.887 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ca5564:::benchmark_data_smithi194_196924_object145000:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42caaf15:::benchmark_data_smithi194_196924_object153705:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42cabfad:::benchmark_data_smithi194_196924_object136100:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42cb0693:::benchmark_data_smithi194_196924_object153644:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42cd5997:::benchmark_data_smithi194_196924_object144320:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42cdccc7:::benchmark_data_smithi194_196924_object135892:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ceaa52:::benchmark_data_smithi194_196924_object149769:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42cee493:::benchmark_data_smithi194_196924_object135557:head# 2024-04-29T15:00:10.888 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42cf7a9d:::benchmark_data_smithi194_196924_object104336:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42d05007:::benchmark_data_smithi194_196924_object100862:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42d27743:::benchmark_data_smithi194_196924_object123724:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42d302f7:::benchmark_data_smithi194_196924_object108637:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42d997cd:::benchmark_data_smithi194_196924_object113682:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42da9677:::benchmark_data_smithi194_196924_object136812:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42dbc372:::benchmark_data_smithi194_196924_object129105:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42dc5f60:::benchmark_data_smithi194_196924_object139005:head# 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ddea82:::benchmark_data_smithi194_196924_object126120:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42de2fc1:::benchmark_data_smithi194_196924_object128277:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42de7690:::benchmark_data_smithi194_196924_object117729:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e26d06:::benchmark_data_smithi194_196924_object145345:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e3720e:::benchmark_data_smithi194_196924_object102478:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e46493:::benchmark_data_smithi194_196924_object122265:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e7113d:::benchmark_data_smithi194_196924_object146472:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e74e35:::benchmark_data_smithi194_196924_object130184:head# 2024-04-29T15:00:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e843b6:::benchmark_data_smithi194_196924_object134650:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e91f48:::benchmark_data_smithi194_196924_object99419:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42e96fd0:::benchmark_data_smithi194_196924_object154571:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ea5d74:::benchmark_data_smithi194_196924_object107247:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42edcea1:::benchmark_data_smithi194_196924_object96738:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42eefcb9:::benchmark_data_smithi194_196924_object133040:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f18005:::benchmark_data_smithi194_196924_object128488:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f1fe1f:::benchmark_data_smithi194_196924_object159403:head# 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f39242:::benchmark_data_smithi194_196924_object124229:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f59a1a:::benchmark_data_smithi194_196924_object144534:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f628cc:::benchmark_data_smithi194_196924_object120399:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f77885:::benchmark_data_smithi194_196924_object130337:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f7abc4:::benchmark_data_smithi194_196924_object131076:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42f948fa:::benchmark_data_smithi194_196924_object96306:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42facbef:::benchmark_data_smithi194_196924_object122233:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42fb5ae0:::benchmark_data_smithi194_196924_object110642:head# 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.892 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42fbbfd1:::benchmark_data_smithi194_196924_object101443:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42fe2d3a:::benchmark_data_smithi194_196924_object132472:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ff79f6:::benchmark_data_smithi194_196924_object117003:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:42ff968f:::benchmark_data_smithi194_196924_object100770:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430051d1:::benchmark_data_smithi194_196924_object152382:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4300b732:::benchmark_data_smithi194_196924_object119184:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4301b916:::benchmark_data_smithi194_196924_object109433:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43027dfa:::benchmark_data_smithi194_196924_object113280:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4302e5dc:::benchmark_data_smithi194_196924_object151614:head# 2024-04-29T15:00:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430491d3:::benchmark_data_smithi194_196924_object109992:head# 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4304b56e:::benchmark_data_smithi194_196924_object156923:head# 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4304cfdb:::benchmark_data_smithi194_196924_object157640:head# 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430670d2:::benchmark_data_smithi194_196924_object159675:head# 2024-04-29T15:00:10.894 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43074d06:::benchmark_data_smithi194_196924_object159870:head# 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4308ad42:::benchmark_data_smithi194_196924_object155764:head# 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4309fc2f:::benchmark_data_smithi194_196924_object137733:head# 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430ae4da:::benchmark_data_smithi194_196924_object130555:head# 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430b2b66:::benchmark_data_smithi194_196924_object106035:head# 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430b380a:::benchmark_data_smithi194_196924_object156019:head# 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430d1a27:::benchmark_data_smithi194_196924_object126481:head# 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.905 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:430f49f1:::benchmark_data_smithi194_196924_object99746:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:431195bf:::benchmark_data_smithi194_196924_object122263:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4313c640:::benchmark_data_smithi194_196924_object134405:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4313eba1:::benchmark_data_smithi194_196924_object99554:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4313feef:::benchmark_data_smithi194_196924_object105488:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43147af5:::benchmark_data_smithi194_196924_object158072:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:431494d7:::benchmark_data_smithi194_196924_object105730:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4314cf4f:::benchmark_data_smithi194_196924_object134355:head# 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.906 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4317865e:::benchmark_data_smithi194_196924_object140757:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4317e11c:::benchmark_data_smithi194_196924_object111474:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:431ccb89:::benchmark_data_smithi194_196924_object134939:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:431d8e4d:::benchmark_data_smithi194_196924_object153385:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432097f5:::benchmark_data_smithi194_196924_object135880:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4320f9f6:::benchmark_data_smithi194_196924_object99826:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43242341:::benchmark_data_smithi194_196924_object132690:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43242fff:::benchmark_data_smithi194_196924_object132078:head# 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.907 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4324e08b:::benchmark_data_smithi194_196924_object155173:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43252a9b:::benchmark_data_smithi194_196924_object112990:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432620aa:::benchmark_data_smithi194_196924_object122782:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43279f6d:::benchmark_data_smithi194_196924_object132786:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43280677:::benchmark_data_smithi194_196924_object100598:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43291fd3:::benchmark_data_smithi194_196924_object116342:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43294aa8:::benchmark_data_smithi194_196924_object156748:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432aeafb:::benchmark_data_smithi194_196924_object155408:head# 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.908 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432b8763:::benchmark_data_smithi194_196924_object151019:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432b97d9:::benchmark_data_smithi194_196924_object107088:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432be625:::benchmark_data_smithi194_196924_object106500:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432c6bfd:::benchmark_data_smithi194_196924_object100909:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432d25be:::benchmark_data_smithi194_196924_object121103:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432de5c7:::benchmark_data_smithi194_196924_object128861:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432df02f:::benchmark_data_smithi194_196924_object101470:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432e8c1a:::benchmark_data_smithi194_196924_object127341:head# 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.909 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:432ef306:::benchmark_data_smithi194_196924_object126080:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4330f95e:::benchmark_data_smithi194_196924_object153237:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:433312a6:::benchmark_data_smithi194_196924_object96071:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4333a102:::benchmark_data_smithi194_196924_object128322:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43347c46:::benchmark_data_smithi194_196924_object96909:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43356672:::benchmark_data_smithi194_196924_object113953:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4338929b:::benchmark_data_smithi194_196924_object136290:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43389393:::benchmark_data_smithi194_196924_object111570:head# 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.910 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4338dc70:::benchmark_data_smithi194_196924_object151162:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4339ca2f:::benchmark_data_smithi194_196924_object121697:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:433a833b:::benchmark_data_smithi194_196924_object140046:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:433c2914:::benchmark_data_smithi194_196924_object105310:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:433d2071:::benchmark_data_smithi194_196924_object147481:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:433f100d:::benchmark_data_smithi194_196924_object149344:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:433f65d4:::benchmark_data_smithi194_196924_object151836:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:434166c2:::benchmark_data_smithi194_196924_object119275:head# 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.911 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4341d546:::benchmark_data_smithi194_196924_object142566:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:434450ef:::benchmark_data_smithi194_196924_object97262:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4345c618:::benchmark_data_smithi194_196924_object117369:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4347af97:::benchmark_data_smithi194_196924_object155194:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:434a5c60:::benchmark_data_smithi194_196924_object123739:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:434d8647:::benchmark_data_smithi194_196924_object103223:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:434dcd2b:::benchmark_data_smithi194_196924_object99601:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43522f2c:::benchmark_data_smithi194_196924_object113706:head# 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.912 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43529c80:::benchmark_data_smithi194_196924_object157993:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43533b89:::benchmark_data_smithi194_196924_object153604:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43542ac8:::benchmark_data_smithi194_196924_object107237:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4357a0af:::benchmark_data_smithi194_196924_object123100:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4358c392:::benchmark_data_smithi194_196924_object153255:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4358d38a:::benchmark_data_smithi194_196924_object148656:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435aa72f:::benchmark_data_smithi194_196924_object156591:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435aac65:::benchmark_data_smithi194_196924_object109779:head# 2024-04-29T15:00:10.913 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435b51e7:::benchmark_data_smithi194_196924_object154039:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435bd122:::benchmark_data_smithi194_196924_object113421:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435c4323:::benchmark_data_smithi194_196924_object120763:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435ce4eb:::benchmark_data_smithi194_196924_object112689:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435dadad:::benchmark_data_smithi194_196924_object97560:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435e00c7:::benchmark_data_smithi194_196924_object119144:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435e881a:::benchmark_data_smithi194_196924_object136866:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:435ebb08:::benchmark_data_smithi194_196924_object115708:head# 2024-04-29T15:00:10.914 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.915 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43606ec1:::benchmark_data_smithi194_196924_object147642:head# 2024-04-29T15:00:10.915 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.915 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4360b867:::benchmark_data_smithi194_196924_object146437:head# 2024-04-29T15:00:10.915 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.915 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4360dd87:::benchmark_data_smithi194_196924_object108340:head# 2024-04-29T15:00:10.915 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4361841f:::benchmark_data_smithi194_196924_object118681:head# 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4361c996:::benchmark_data_smithi194_196924_object123172:head# 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43622105:::benchmark_data_smithi194_196924_object108046:head# 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4364e03f:::benchmark_data_smithi194_196924_object122682:head# 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43653f14:::benchmark_data_smithi194_196924_object136945:head# 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4365f93e:::benchmark_data_smithi194_196924_object143382:head# 2024-04-29T15:00:10.926 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4366352c:::benchmark_data_smithi194_196924_object108750:head# 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4366431f:::benchmark_data_smithi194_196924_object159356:head# 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4366ec39:::benchmark_data_smithi194_196924_object157210:head# 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43672458:::benchmark_data_smithi194_196924_object159982:head# 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4368a4f3:::benchmark_data_smithi194_196924_object98602:head# 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:436b9ba9:::benchmark_data_smithi194_196924_object122084:head# 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:436c6990:::benchmark_data_smithi194_196924_object143558:head# 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.927 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:436d486c:::benchmark_data_smithi194_196924_object125367:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43704349:::benchmark_data_smithi194_196924_object111979:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43705de5:::benchmark_data_smithi194_196924_object116188:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43731f42:::benchmark_data_smithi194_196924_object111604:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:437391d7:::benchmark_data_smithi194_196924_object117485:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4374dc69:::benchmark_data_smithi194_196924_object119719:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4375db94:::benchmark_data_smithi194_196924_object95672:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43762053:::benchmark_data_smithi194_196924_object151210:head# 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.928 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4376b0ad:::benchmark_data_smithi194_196924_object126349:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43771643:::benchmark_data_smithi194_196924_object134941:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43787923:::benchmark_data_smithi194_196924_object119443:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:437b95a2:::benchmark_data_smithi194_196924_object146966:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:437c5677:::benchmark_data_smithi194_196924_object110064:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:437eae29:::benchmark_data_smithi194_196924_object124104:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4380cf8d:::benchmark_data_smithi194_196924_object102420:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43815bf1:::benchmark_data_smithi194_196924_object126972:head# 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4388385a:::benchmark_data_smithi194_196924_object119535:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43896acc:::benchmark_data_smithi194_196924_object148193:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:438b9241:::benchmark_data_smithi194_196924_object154631:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:438d9dbb:::benchmark_data_smithi194_196924_object125108:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:438e5311:::benchmark_data_smithi194_196924_object111003:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:438e6c19:::benchmark_data_smithi194_196924_object116099:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:438feb41:::benchmark_data_smithi194_196924_object114700:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43903b42:::benchmark_data_smithi194_196924_object141122:head# 2024-04-29T15:00:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43930059:::benchmark_data_smithi194_196924_object145937:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439570f5:::benchmark_data_smithi194_196924_object133461:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43965ff4:::benchmark_data_smithi194_196924_object148243:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43974059:::benchmark_data_smithi194_196924_object153721:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4397415e:::benchmark_data_smithi194_196924_object131104:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4398c5fc:::benchmark_data_smithi194_196924_object115560:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439a99fa:::benchmark_data_smithi194_196924_object151753:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439aad57:::benchmark_data_smithi194_196924_object156409:head# 2024-04-29T15:00:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439b4256:::benchmark_data_smithi194_196924_object144254:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439c0957:::benchmark_data_smithi194_196924_object97981:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439cc738:::benchmark_data_smithi194_196924_object152938:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439e2b7a:::benchmark_data_smithi194_196924_object134403:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:439ec08b:::benchmark_data_smithi194_196924_object131373:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a23088:::benchmark_data_smithi194_196924_object136385:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a34eb1:::benchmark_data_smithi194_196924_object99186:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a3565e:::benchmark_data_smithi194_196924_object126543:head# 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.932 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a92ce8:::benchmark_data_smithi194_196924_object115100:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a92d2f:::benchmark_data_smithi194_196924_object107419:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a98d3b:::benchmark_data_smithi194_196924_object128847:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a9b239:::benchmark_data_smithi194_196924_object125631:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43a9dde1:::benchmark_data_smithi194_196924_object156042:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ab822e:::benchmark_data_smithi194_196924_object147346:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ac4cd0:::benchmark_data_smithi194_196924_object109024:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ae2daa:::benchmark_data_smithi194_196924_object134480:head# 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.933 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43af872a:::benchmark_data_smithi194_196924_object142890:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b14dd7:::benchmark_data_smithi194_196924_object155963:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b21d42:::benchmark_data_smithi194_196924_object102839:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b340df:::benchmark_data_smithi194_196924_object157833:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b461ed:::benchmark_data_smithi194_196924_object104191:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b525d5:::benchmark_data_smithi194_196924_object139842:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b60350:::benchmark_data_smithi194_196924_object155119:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b679f2:::benchmark_data_smithi194_196924_object118575:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43b9ceb9:::benchmark_data_smithi194_196924_object107704:head# 2024-04-29T15:00:10.934 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ba38d4:::benchmark_data_smithi194_196924_object114777:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ba8031:::benchmark_data_smithi194_196924_object116444:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43baa22c:::benchmark_data_smithi194_196924_object140619:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43bc7e9f:::benchmark_data_smithi194_196924_object106181:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c0f952:::benchmark_data_smithi194_196924_object147497:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c1aed6:::benchmark_data_smithi194_196924_object124356:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c218fa:::benchmark_data_smithi194_196924_object119298:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c2b98f:::benchmark_data_smithi194_196924_object120892:head# 2024-04-29T15:00:10.935 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c3edaa:::benchmark_data_smithi194_196924_object130667:head# 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c45f50:::benchmark_data_smithi194_196924_object146698:head# 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c60989:::benchmark_data_smithi194_196924_object152185:head# 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c6b3ee:::benchmark_data_smithi194_196924_object140831:head# 2024-04-29T15:00:10.936 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.940 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c998c5:::benchmark_data_smithi194_196924_object109712:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43c9d385:::benchmark_data_smithi194_196924_object106162:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ca1ae0:::benchmark_data_smithi194_196924_object118714:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43cac3e3:::benchmark_data_smithi194_196924_object137625:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43cae237:::benchmark_data_smithi194_196924_object143596:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43cb0d83:::benchmark_data_smithi194_196924_object146311:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43cb19c4:::benchmark_data_smithi194_196924_object145705:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43cc2091:::benchmark_data_smithi194_196924_object159827:head# 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.941 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ccde06:::benchmark_data_smithi194_196924_object137563:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43cdb833:::benchmark_data_smithi194_196924_object139154:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43cf6fc6:::benchmark_data_smithi194_196924_object146848:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d1e12e:::benchmark_data_smithi194_196924_object116794:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d3d171:::benchmark_data_smithi194_196924_object146448:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d446f4:::benchmark_data_smithi194_196924_object108968:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d4a41f:::benchmark_data_smithi194_196924_object109532:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d60bc0:::benchmark_data_smithi194_196924_object111908:head# 2024-04-29T15:00:10.942 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d7815b:::benchmark_data_smithi194_196924_object110478:head# 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d8afc3:::benchmark_data_smithi194_196924_object120474:head# 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43d9d21f:::benchmark_data_smithi194_196924_object153576:head# 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43dbcadb:::benchmark_data_smithi194_196924_object146656:head# 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ddfac6:::benchmark_data_smithi194_196924_object133205:head# 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43dfabf6:::benchmark_data_smithi194_196924_object116800:head# 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43dfe251:::benchmark_data_smithi194_196924_object130941:head# 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.943 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e4af9b:::benchmark_data_smithi194_196924_object129822:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e4f45b:::benchmark_data_smithi194_196924_object145432:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e521fc:::benchmark_data_smithi194_196924_object97826:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e5440f:::benchmark_data_smithi194_196924_object128647:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e6b3ef:::benchmark_data_smithi194_196924_object159364:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e735bc:::benchmark_data_smithi194_196924_object95926:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e81018:::benchmark_data_smithi194_196924_object96277:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43e9750c:::benchmark_data_smithi194_196924_object140489:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43eae882:::benchmark_data_smithi194_196924_object120636:head# 2024-04-29T15:00:10.944 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43eb8725:::benchmark_data_smithi194_196924_object158795:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ed706b:::benchmark_data_smithi194_196924_object112401:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ee1a28:::benchmark_data_smithi194_196924_object139682:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43ee7eca:::benchmark_data_smithi194_196924_object134738:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43eea6c5:::benchmark_data_smithi194_196924_object141787:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f02721:::benchmark_data_smithi194_196924_object148079:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f04dc3:::benchmark_data_smithi194_196924_object136649:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f11793:::benchmark_data_smithi194_196924_object117478:head# 2024-04-29T15:00:10.945 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.951 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f124b1:::benchmark_data_smithi194_196924_object119861:head# 2024-04-29T15:00:10.951 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.951 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f26110:::benchmark_data_smithi194_196924_object106574:head# 2024-04-29T15:00:10.951 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.951 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f30868:::benchmark_data_smithi194_196924_object154081:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f35cad:::benchmark_data_smithi194_196924_object116225:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f418d9:::benchmark_data_smithi194_196924_object100985:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f4d35e:::benchmark_data_smithi194_196924_object150596:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f588c7:::benchmark_data_smithi194_196924_object119079:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f66691:::benchmark_data_smithi194_196924_object148267:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f70470:::benchmark_data_smithi194_196924_object111025:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43f8c06e:::benchmark_data_smithi194_196924_object125767:head# 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43faeba0:::benchmark_data_smithi194_196924_object111061:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43fb45fc:::benchmark_data_smithi194_196924_object108827:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43fb5f33:::benchmark_data_smithi194_196924_object127576:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43fb6eb4:::benchmark_data_smithi194_196924_object140458:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43fb99c9:::benchmark_data_smithi194_196924_object112934:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43fbb239:::benchmark_data_smithi194_196924_object153167:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:43fcfe49:::benchmark_data_smithi194_196924_object139834:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44013c6c:::benchmark_data_smithi194_196924_object137341:head# 2024-04-29T15:00:10.953 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44026e32:::benchmark_data_smithi194_196924_object145030:head# 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44031eb5:::benchmark_data_smithi194_196924_object117669:head# 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4403e0fd:::benchmark_data_smithi194_196924_object108025:head# 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4404390c:::benchmark_data_smithi194_196924_object143302:head# 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:440446a3:::benchmark_data_smithi194_196924_object98690:head# 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4404db8c:::benchmark_data_smithi194_196924_object118294:head# 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44050ec5:::benchmark_data_smithi194_196924_object117329:head# 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4407bdcf:::benchmark_data_smithi194_196924_object121010:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4408ae04:::benchmark_data_smithi194_196924_object119525:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4408deaf:::benchmark_data_smithi194_196924_object135682:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:440a40ca:::benchmark_data_smithi194_196924_object140864:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:440b7265:::benchmark_data_smithi194_196924_object140185:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:440bb8ff:::benchmark_data_smithi194_196924_object96567:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:440ea372:::benchmark_data_smithi194_196924_object154530:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441022f8:::benchmark_data_smithi194_196924_object107513:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4410477e:::benchmark_data_smithi194_196924_object153960:head# 2024-04-29T15:00:10.955 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44109e89:::benchmark_data_smithi194_196924_object119409:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4411feab:::benchmark_data_smithi194_196924_object113082:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441259ee:::benchmark_data_smithi194_196924_object103946:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44128e28:::benchmark_data_smithi194_196924_object151835:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44146550:::benchmark_data_smithi194_196924_object146434:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44149b9b:::benchmark_data_smithi194_196924_object113318:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4415264e:::benchmark_data_smithi194_196924_object132664:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44165f8c:::benchmark_data_smithi194_196924_object134828:head# 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.956 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44166f48:::benchmark_data_smithi194_196924_object151410:head# 2024-04-29T15:00:10.957 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.957 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44184e4b:::benchmark_data_smithi194_196924_object116629:head# 2024-04-29T15:00:10.957 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.962 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441886ee:::benchmark_data_smithi194_196924_object99977:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441a842a:::benchmark_data_smithi194_196924_object116012:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441ab324:::benchmark_data_smithi194_196924_object97033:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441bb656:::benchmark_data_smithi194_196924_object153152:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441ccdf4:::benchmark_data_smithi194_196924_object105223:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441ceb49:::benchmark_data_smithi194_196924_object155720:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441db8c1:::benchmark_data_smithi194_196924_object144328:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441f6cac:::benchmark_data_smithi194_196924_object159250:head# 2024-04-29T15:00:10.963 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:441fe75e:::benchmark_data_smithi194_196924_object120417:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44210ceb:::benchmark_data_smithi194_196924_object114186:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44219e66:::benchmark_data_smithi194_196924_object143286:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44227a63:::benchmark_data_smithi194_196924_object102460:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44236dea:::benchmark_data_smithi194_196924_object150745:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:442370e9:::benchmark_data_smithi194_196924_object105889:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:442513e6:::benchmark_data_smithi194_196924_object134188:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4425c971:::benchmark_data_smithi194_196924_object119788:head# 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.964 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44266e6a:::benchmark_data_smithi194_196924_object158412:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4426bf27:::benchmark_data_smithi194_196924_object151573:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4426d64d:::benchmark_data_smithi194_196924_object152385:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4426f169:::benchmark_data_smithi194_196924_object142269:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4429c3d2:::benchmark_data_smithi194_196924_object96066:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:442cf589:::benchmark_data_smithi194_196924_object99943:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:442e2df5:::benchmark_data_smithi194_196924_object141900:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:442e5860:::benchmark_data_smithi194_196924_object154955:head# 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.965 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:442ebebb:::benchmark_data_smithi194_196924_object121750:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4430fedd:::benchmark_data_smithi194_196924_object146818:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44316d08:::benchmark_data_smithi194_196924_object111161:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4431a513:::benchmark_data_smithi194_196924_object101784:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4431d391:::benchmark_data_smithi194_196924_object101927:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4437660c:::benchmark_data_smithi194_196924_object150694:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4437e396:::benchmark_data_smithi194_196924_object140590:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44383fbd:::benchmark_data_smithi194_196924_object156842:head# 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4439141e:::benchmark_data_smithi194_196924_object125292:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:443a8d62:::benchmark_data_smithi194_196924_object152942:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:443d74bd:::benchmark_data_smithi194_196924_object107874:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:443e7229:::benchmark_data_smithi194_196924_object146748:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:443fe318:::benchmark_data_smithi194_196924_object148422:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:443fe33e:::benchmark_data_smithi194_196924_object106541:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4440119c:::benchmark_data_smithi194_196924_object116645:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4442f80c:::benchmark_data_smithi194_196924_object126621:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4443240f:::benchmark_data_smithi194_196924_object130622:head# 2024-04-29T15:00:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4443d543:::benchmark_data_smithi194_196924_object159473:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44445e3f:::benchmark_data_smithi194_196924_object129533:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4445ed49:::benchmark_data_smithi194_196924_object115929:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44460523:::benchmark_data_smithi194_196924_object138454:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4446e06e:::benchmark_data_smithi194_196924_object106824:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:444757c5:::benchmark_data_smithi194_196924_object124701:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:444829a0:::benchmark_data_smithi194_196924_object109937:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:444952ef:::benchmark_data_smithi194_196924_object145458:head# 2024-04-29T15:00:10.968 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:444b73ab:::benchmark_data_smithi194_196924_object127255:head# 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:444c544a:::benchmark_data_smithi194_196924_object105028:head# 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:444f7cd5:::benchmark_data_smithi194_196924_object104256:head# 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4450032b:::benchmark_data_smithi194_196924_object151827:head# 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44502d03:::benchmark_data_smithi194_196924_object138649:head# 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445171e1:::benchmark_data_smithi194_196924_object100333:head# 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4452b572:::benchmark_data_smithi194_196924_object159992:head# 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.975 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4453aa47:::benchmark_data_smithi194_196924_object101382:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44543e0d:::benchmark_data_smithi194_196924_object114742:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4454c238:::benchmark_data_smithi194_196924_object141365:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4456232b:::benchmark_data_smithi194_196924_object147728:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445699fb:::benchmark_data_smithi194_196924_object142952:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44575bc0:::benchmark_data_smithi194_196924_object96471:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445a8843:::benchmark_data_smithi194_196924_object122832:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445b007c:::benchmark_data_smithi194_196924_object111892:head# 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.976 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445c4906:::benchmark_data_smithi194_196924_object117252:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445c9f5a:::benchmark_data_smithi194_196924_object159821:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445d7031:::benchmark_data_smithi194_196924_object117553:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445ee788:::benchmark_data_smithi194_196924_object134771:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:445f7d2f:::benchmark_data_smithi194_196924_object119937:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:446243cb:::benchmark_data_smithi194_196924_object127153:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44628caa:::benchmark_data_smithi194_196924_object141901:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44632439:::benchmark_data_smithi194_196924_object145400:head# 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.977 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44639a42:::benchmark_data_smithi194_196924_object133793:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4463d0b6:::benchmark_data_smithi194_196924_object140756:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4464b209:::benchmark_data_smithi194_196924_object152755:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4465008c:::benchmark_data_smithi194_196924_object137238:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44666bd2:::benchmark_data_smithi194_196924_object107252:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4468178a:::benchmark_data_smithi194_196924_object136715:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4469eae1:::benchmark_data_smithi194_196924_object138899:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:446b59d6:::benchmark_data_smithi194_196924_object149756:head# 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.978 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:446bb961:::benchmark_data_smithi194_196924_object154372:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:446bfecc:::benchmark_data_smithi194_196924_object113435:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:446cb014:::benchmark_data_smithi194_196924_object123888:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:446ddc7c:::benchmark_data_smithi194_196924_object143265:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44700b4b:::benchmark_data_smithi194_196924_object159223:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44713fe7:::benchmark_data_smithi194_196924_object112167:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:447161aa:::benchmark_data_smithi194_196924_object149035:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44724569:::benchmark_data_smithi194_196924_object158789:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4474039a:::benchmark_data_smithi194_196924_object120453:head# 2024-04-29T15:00:10.979 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44742608:::benchmark_data_smithi194_196924_object102135:head# 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44748cbc:::benchmark_data_smithi194_196924_object141541:head# 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4476f911:::benchmark_data_smithi194_196924_object149687:head# 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44770227:::benchmark_data_smithi194_196924_object115259:head# 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44787484:::benchmark_data_smithi194_196924_object128319:head# 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:447988d5:::benchmark_data_smithi194_196924_object133776:head# 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:447acb1c:::benchmark_data_smithi194_196924_object97025:head# 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.980 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:447ae6a3:::benchmark_data_smithi194_196924_object142153:head# 2024-04-29T15:00:10.981 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.981 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:447f1c21:::benchmark_data_smithi194_196924_object158582:head# 2024-04-29T15:00:10.981 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.981 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4481541b:::benchmark_data_smithi194_196924_object118090:head# 2024-04-29T15:00:10.981 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44818035:::benchmark_data_smithi194_196924_object114752:head# 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44826061:::benchmark_data_smithi194_196924_object114967:head# 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4482cd6e:::benchmark_data_smithi194_196924_object144627:head# 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4482ecee:::benchmark_data_smithi194_196924_object102192:head# 2024-04-29T15:00:10.994 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44844c3c:::benchmark_data_smithi194_196924_object105429:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4484e83f:::benchmark_data_smithi194_196924_object108224:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44892bc8:::benchmark_data_smithi194_196924_object114928:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:448931e0:::benchmark_data_smithi194_196924_object122302:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44898d5f:::benchmark_data_smithi194_196924_object130373:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:448b473d:::benchmark_data_smithi194_196924_object156339:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:448bb944:::benchmark_data_smithi194_196924_object113334:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:448cccc7:::benchmark_data_smithi194_196924_object124722:head# 2024-04-29T15:00:10.995 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:448d2808:::benchmark_data_smithi194_196924_object139866:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:448fff50:::benchmark_data_smithi194_196924_object96788:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4493baaf:::benchmark_data_smithi194_196924_object105724:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4493cdcb:::benchmark_data_smithi194_196924_object115254:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4494e780:::benchmark_data_smithi194_196924_object125039:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44967027:::benchmark_data_smithi194_196924_object126688:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4496c896:::benchmark_data_smithi194_196924_object128199:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4497eac7:::benchmark_data_smithi194_196924_object107717:head# 2024-04-29T15:00:10.996 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:449a2c3d:::benchmark_data_smithi194_196924_object123360:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:449be2b1:::benchmark_data_smithi194_196924_object139907:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:449c8d50:::benchmark_data_smithi194_196924_object134863:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:449d3760:::benchmark_data_smithi194_196924_object154354:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:449dc5d5:::benchmark_data_smithi194_196924_object104542:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:449f1f90:::benchmark_data_smithi194_196924_object156559:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:449fb9ce:::benchmark_data_smithi194_196924_object105554:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a08237:::benchmark_data_smithi194_196924_object129790:head# 2024-04-29T15:00:10.997 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a0b7e4:::benchmark_data_smithi194_196924_object98545:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a0bb6a:::benchmark_data_smithi194_196924_object116736:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a1091a:::benchmark_data_smithi194_196924_object103586:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a2679f:::benchmark_data_smithi194_196924_object108535:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a32568:::benchmark_data_smithi194_196924_object105998:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a3b294:::benchmark_data_smithi194_196924_object123752:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a3d629:::benchmark_data_smithi194_196924_object148288:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a41133:::benchmark_data_smithi194_196924_object147912:head# 2024-04-29T15:00:10.998 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a45e4e:::benchmark_data_smithi194_196924_object128654:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a6914e:::benchmark_data_smithi194_196924_object124387:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a69ae9:::benchmark_data_smithi194_196924_object108418:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a6ff47:::benchmark_data_smithi194_196924_object156007:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a7302a:::benchmark_data_smithi194_196924_object101490:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a77d1a:::benchmark_data_smithi194_196924_object114720:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a7fddf:::benchmark_data_smithi194_196924_object102326:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a80b5a:::benchmark_data_smithi194_196924_object126646:head# 2024-04-29T15:00:10.999 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44a8163f:::benchmark_data_smithi194_196924_object119587:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ad6416:::benchmark_data_smithi194_196924_object112234:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44adf9f9:::benchmark_data_smithi194_196924_object96105:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44aee4db:::benchmark_data_smithi194_196924_object149132:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b1292e:::benchmark_data_smithi194_196924_object154584:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b13a39:::benchmark_data_smithi194_196924_object158891:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b265ae:::benchmark_data_smithi194_196924_object150713:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b34dfd:::benchmark_data_smithi194_196924_object137551:head# 2024-04-29T15:00:11.000 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b40a6d:::benchmark_data_smithi194_196924_object103792:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b455ec:::benchmark_data_smithi194_196924_object143628:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b7144a:::benchmark_data_smithi194_196924_object100009:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b71db6:::benchmark_data_smithi194_196924_object112880:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b76f47:::benchmark_data_smithi194_196924_object120018:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b836c3:::benchmark_data_smithi194_196924_object147100:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b8bf78:::benchmark_data_smithi194_196924_object125330:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44b9ae2a:::benchmark_data_smithi194_196924_object143634:head# 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.001 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ba93c5:::benchmark_data_smithi194_196924_object147827:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44bbf4cd:::benchmark_data_smithi194_196924_object117311:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44be117f:::benchmark_data_smithi194_196924_object99570:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44bf520e:::benchmark_data_smithi194_196924_object153555:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44bf82a2:::benchmark_data_smithi194_196924_object136499:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44bf8855:::benchmark_data_smithi194_196924_object139668:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44c1d16d:::benchmark_data_smithi194_196924_object127236:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44c20ecd:::benchmark_data_smithi194_196924_object109886:head# 2024-04-29T15:00:11.002 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44c25a12:::benchmark_data_smithi194_196924_object135927:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44c53284:::benchmark_data_smithi194_196924_object145475:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44c772d2:::benchmark_data_smithi194_196924_object136375:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44c7dfe0:::benchmark_data_smithi194_196924_object149911:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44c997a8:::benchmark_data_smithi194_196924_object138920:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ca5179:::benchmark_data_smithi194_196924_object106393:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ca937b:::benchmark_data_smithi194_196924_object159836:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44caa6d0:::benchmark_data_smithi194_196924_object114574:head# 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.003 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44cab274:::benchmark_data_smithi194_196924_object148246:head# 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44cb12e8:::benchmark_data_smithi194_196924_object146405:head# 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44cb7755:::benchmark_data_smithi194_196924_object120673:head# 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44cca125:::benchmark_data_smithi194_196924_object112241:head# 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ccfae2:::benchmark_data_smithi194_196924_object113740:head# 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44cd00f4:::benchmark_data_smithi194_196924_object139610:head# 2024-04-29T15:00:11.004 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.005 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:11.017 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ce7e63:::benchmark_data_smithi194_196924_object134093:head# 2024-04-29T15:00:11.017 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.017 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ceab0b:::benchmark_data_smithi194_196924_object132139:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44cec516:::benchmark_data_smithi194_196924_object111262:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44cf5f13:::benchmark_data_smithi194_196924_object111626:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d080d1:::benchmark_data_smithi194_196924_object131148:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d457f4:::benchmark_data_smithi194_196924_object97104:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d4cbbd:::benchmark_data_smithi194_196924_object105489:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d4eff1:::benchmark_data_smithi194_196924_object132765:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d519ad:::benchmark_data_smithi194_196924_object106820:head# 2024-04-29T15:00:11.018 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d5e2e6:::benchmark_data_smithi194_196924_object113013:head# 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d63991:::benchmark_data_smithi194_196924_object95498:head# 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d64d39:::benchmark_data_smithi194_196924_object146101:head# 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d67329:::benchmark_data_smithi194_196924_object106590:head# 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d6e7d8:::benchmark_data_smithi194_196924_object137881:head# 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d78237:::benchmark_data_smithi194_196924_object133031:head# 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d79398:::benchmark_data_smithi194_196924_object105923:head# 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.019 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d82259:::benchmark_data_smithi194_196924_object125611:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d8900a:::benchmark_data_smithi194_196924_object120275:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44d98389:::benchmark_data_smithi194_196924_object144641:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44dc67ec:::benchmark_data_smithi194_196924_object117134:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44dcf291:::benchmark_data_smithi194_196924_object107199:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44dec746:::benchmark_data_smithi194_196924_object116368:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44df609a:::benchmark_data_smithi194_196924_object159681:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e0425a:::benchmark_data_smithi194_196924_object131950:head# 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.020 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e11142:::benchmark_data_smithi194_196924_object104529:head# 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e24922:::benchmark_data_smithi194_196924_object126476:head# 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e253e7:::benchmark_data_smithi194_196924_object106410:head# 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e492d4:::benchmark_data_smithi194_196924_object118040:head# 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e5b0f5:::benchmark_data_smithi194_196924_object150285:head# 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e8b567:::benchmark_data_smithi194_196924_object156635:head# 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44e9b1b8:::benchmark_data_smithi194_196924_object100885:head# 2024-04-29T15:00:11.021 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ebff0a:::benchmark_data_smithi194_196924_object146565:head# 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ec308b:::benchmark_data_smithi194_196924_object135007:head# 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ed8efa:::benchmark_data_smithi194_196924_object102120:head# 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ed9c63:::benchmark_data_smithi194_196924_object96354:head# 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ef6a13:::benchmark_data_smithi194_196924_object100001:head# 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44f35723:::benchmark_data_smithi194_196924_object151333:head# 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44f4b296:::benchmark_data_smithi194_196924_object146166:head# 2024-04-29T15:00:11.022 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44f58240:::benchmark_data_smithi194_196924_object146076:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44f97cfe:::benchmark_data_smithi194_196924_object95499:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fbe718:::benchmark_data_smithi194_196924_object119363:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fbfa83:::benchmark_data_smithi194_196924_object115774:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fc25da:::benchmark_data_smithi194_196924_object123011:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fc799c:::benchmark_data_smithi194_196924_object101119:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fce403:::benchmark_data_smithi194_196924_object126467:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fdc92c:::benchmark_data_smithi194_196924_object136769:head# 2024-04-29T15:00:11.023 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fe499f:::benchmark_data_smithi194_196924_object150906:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44feafe7:::benchmark_data_smithi194_196924_object126548:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44fef1ba:::benchmark_data_smithi194_196924_object110265:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:44ff9aac:::benchmark_data_smithi194_196924_object153201:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45008701:::benchmark_data_smithi194_196924_object124570:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4503212a:::benchmark_data_smithi194_196924_object139068:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:450364bc:::benchmark_data_smithi194_196924_object153897:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45058f67:::benchmark_data_smithi194_196924_object156938:head# 2024-04-29T15:00:11.024 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45063c8a:::benchmark_data_smithi194_196924_object115378:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4506e2dd:::benchmark_data_smithi194_196924_object146288:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45088661:::benchmark_data_smithi194_196924_object100590:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:450896c8:::benchmark_data_smithi194_196924_object110968:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4508bec3:::benchmark_data_smithi194_196924_object156248:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45095ada:::benchmark_data_smithi194_196924_object154791:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4509f787:::benchmark_data_smithi194_196924_object132796:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:450c7c0f:::benchmark_data_smithi194_196924_object111487:head# 2024-04-29T15:00:11.025 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45104d74:::benchmark_data_smithi194_196924_object124077:head# 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45115483:::benchmark_data_smithi194_196924_object122476:head# 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45135c07:::benchmark_data_smithi194_196924_object106078:head# 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:451445b0:::benchmark_data_smithi194_196924_object139580:head# 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4514e830:::benchmark_data_smithi194_196924_object96744:head# 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4517332a:::benchmark_data_smithi194_196924_object132368:head# 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4517914f:::benchmark_data_smithi194_196924_object101515:head# 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.026 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:451893ca:::benchmark_data_smithi194_196924_object151783:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:451a18ce:::benchmark_data_smithi194_196924_object144788:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:451d417e:::benchmark_data_smithi194_196924_object99600:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:451ea710:::benchmark_data_smithi194_196924_object133558:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:451fc674:::benchmark_data_smithi194_196924_object156123:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:452131ab:::benchmark_data_smithi194_196924_object156375:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4521778a:::benchmark_data_smithi194_196924_object144028:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4522954e:::benchmark_data_smithi194_196924_object139900:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4522bc32:::benchmark_data_smithi194_196924_object99835:head# 2024-04-29T15:00:11.027 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4522f031:::benchmark_data_smithi194_196924_object142725:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45233299:::benchmark_data_smithi194_196924_object158848:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4523f49e:::benchmark_data_smithi194_196924_object134264:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4525d184:::benchmark_data_smithi194_196924_object107469:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4526027e:::benchmark_data_smithi194_196924_object96621:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4526ea45:::benchmark_data_smithi194_196924_object157496:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4527abf9:::benchmark_data_smithi194_196924_object132419:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45296822:::benchmark_data_smithi194_196924_object151192:head# 2024-04-29T15:00:11.028 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:452b1220:::benchmark_data_smithi194_196924_object96789:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:452b200f:::benchmark_data_smithi194_196924_object102656:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:452b781d:::benchmark_data_smithi194_196924_object103401:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:452e4983:::benchmark_data_smithi194_196924_object151510:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453004d7:::benchmark_data_smithi194_196924_object120800:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45307b33:::benchmark_data_smithi194_196924_object148109:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4530a2d0:::benchmark_data_smithi194_196924_object100491:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45312483:::benchmark_data_smithi194_196924_object111614:head# 2024-04-29T15:00:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.030 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4531731a:::benchmark_data_smithi194_196924_object102921:head# 2024-04-29T15:00:11.030 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45340b4d:::benchmark_data_smithi194_196924_object120524:head# 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45357fa2:::benchmark_data_smithi194_196924_object157309:head# 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4536eccb:::benchmark_data_smithi194_196924_object117710:head# 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453726fa:::benchmark_data_smithi194_196924_object111701:head# 2024-04-29T15:00:11.044 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45388f48:::benchmark_data_smithi194_196924_object103543:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4538e65d:::benchmark_data_smithi194_196924_object141053:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453983be:::benchmark_data_smithi194_196924_object159926:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453a3ec2:::benchmark_data_smithi194_196924_object136435:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453ac4ff:::benchmark_data_smithi194_196924_object101420:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453af4d1:::benchmark_data_smithi194_196924_object99773:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453b3f41:::benchmark_data_smithi194_196924_object134682:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453f1870:::benchmark_data_smithi194_196924_object107616:head# 2024-04-29T15:00:11.045 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453fd59d:::benchmark_data_smithi194_196924_object135917:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:453ffac8:::benchmark_data_smithi194_196924_object144823:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45410c27:::benchmark_data_smithi194_196924_object122254:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454254b1:::benchmark_data_smithi194_196924_object147614:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454355dd:::benchmark_data_smithi194_196924_object98035:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454496e7:::benchmark_data_smithi194_196924_object154756:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4545943d:::benchmark_data_smithi194_196924_object147762:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4545ce1b:::benchmark_data_smithi194_196924_object96234:head# 2024-04-29T15:00:11.046 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4547353e:::benchmark_data_smithi194_196924_object118573:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45480092:::benchmark_data_smithi194_196924_object143542:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45481bf5:::benchmark_data_smithi194_196924_object146510:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454931e8:::benchmark_data_smithi194_196924_object121064:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4549aaf8:::benchmark_data_smithi194_196924_object120756:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454ac28d:::benchmark_data_smithi194_196924_object101783:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454b4516:::benchmark_data_smithi194_196924_object136909:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454b8ded:::benchmark_data_smithi194_196924_object142555:head# 2024-04-29T15:00:11.047 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454c1e08:::benchmark_data_smithi194_196924_object145938:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454cda1f:::benchmark_data_smithi194_196924_object159323:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454d7c2a:::benchmark_data_smithi194_196924_object124338:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454d92f1:::benchmark_data_smithi194_196924_object126183:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454e36fa:::benchmark_data_smithi194_196924_object159625:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454e4582:::benchmark_data_smithi194_196924_object110595:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:454f8f4e:::benchmark_data_smithi194_196924_object150926:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45502a02:::benchmark_data_smithi194_196924_object159373:head# 2024-04-29T15:00:11.048 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45517b46:::benchmark_data_smithi194_196924_object98936:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45518ed4:::benchmark_data_smithi194_196924_object131070:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4552308d:::benchmark_data_smithi194_196924_object135543:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45533992:::benchmark_data_smithi194_196924_object106543:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455363a7:::benchmark_data_smithi194_196924_object144024:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45536952:::benchmark_data_smithi194_196924_object129757:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45538b21:::benchmark_data_smithi194_196924_object113537:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455482b8:::benchmark_data_smithi194_196924_object105173:head# 2024-04-29T15:00:11.049 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455515d7:::benchmark_data_smithi194_196924_object133338:head# 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4555d1e7:::benchmark_data_smithi194_196924_object108375:head# 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4556afe7:::benchmark_data_smithi194_196924_object146798:head# 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4558dec7:::benchmark_data_smithi194_196924_object123235:head# 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455960d3:::benchmark_data_smithi194_196924_object136111:head# 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455a5329:::benchmark_data_smithi194_196924_object112374:head# 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455b72fe:::benchmark_data_smithi194_196924_object137453:head# 2024-04-29T15:00:11.050 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455b8b80:::benchmark_data_smithi194_196924_object128065:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455c743b:::benchmark_data_smithi194_196924_object113027:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455d4754:::benchmark_data_smithi194_196924_object118778:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455da13d:::benchmark_data_smithi194_196924_object159222:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:455f6eb2:::benchmark_data_smithi194_196924_object108673:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4561e196:::benchmark_data_smithi194_196924_object144565:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45623a13:::benchmark_data_smithi194_196924_object159539:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45632671:::benchmark_data_smithi194_196924_object152139:head# 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.051 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45642af6:::benchmark_data_smithi194_196924_object98453:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45657a78:::benchmark_data_smithi194_196924_object136016:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4565dc18:::benchmark_data_smithi194_196924_object120157:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4568d63d:::benchmark_data_smithi194_196924_object126279:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4568e0f6:::benchmark_data_smithi194_196924_object141249:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4569a3dc:::benchmark_data_smithi194_196924_object95408:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456ae786:::benchmark_data_smithi194_196924_object154562:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456b252b:::benchmark_data_smithi194_196924_object101506:head# 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.052 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456b95e7:::benchmark_data_smithi194_196924_object97317:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456ba87b:::benchmark_data_smithi194_196924_object122395:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456bf385:::benchmark_data_smithi194_196924_object151326:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456d93d7:::benchmark_data_smithi194_196924_object145862:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456e9bc6:::benchmark_data_smithi194_196924_object151775:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456f6000:::benchmark_data_smithi194_196924_object110519:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:456f9c8a:::benchmark_data_smithi194_196924_object117526:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457021a4:::benchmark_data_smithi194_196924_object131492:head# 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.053 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4571a34f:::benchmark_data_smithi194_196924_object127325:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4572868e:::benchmark_data_smithi194_196924_object158669:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4572be97:::benchmark_data_smithi194_196924_object152846:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4572e628:::benchmark_data_smithi194_196924_object141786:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45736c5e:::benchmark_data_smithi194_196924_object121985:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4573b631:::benchmark_data_smithi194_196924_object140139:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457451ad:::benchmark_data_smithi194_196924_object142847:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4574f587:::benchmark_data_smithi194_196924_object142013:head# 2024-04-29T15:00:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45756aab:::benchmark_data_smithi194_196924_object159138:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4575cf47:::benchmark_data_smithi194_196924_object103352:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45768be0:::benchmark_data_smithi194_196924_object152042:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4577068f:::benchmark_data_smithi194_196924_object137462:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45772add:::benchmark_data_smithi194_196924_object134445:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45774f90:::benchmark_data_smithi194_196924_object125674:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4577d5d7:::benchmark_data_smithi194_196924_object129235:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45798687:::benchmark_data_smithi194_196924_object127226:head# 2024-04-29T15:00:11.055 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457a92c0:::benchmark_data_smithi194_196924_object149550:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457b0367:::benchmark_data_smithi194_196924_object95932:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457b2cbe:::benchmark_data_smithi194_196924_object129704:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457b3e1e:::benchmark_data_smithi194_196924_object107343:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457e522a:::benchmark_data_smithi194_196924_object97852:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457e7a4d:::benchmark_data_smithi194_196924_object121796:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:457efe01:::benchmark_data_smithi194_196924_object95540:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45839c75:::benchmark_data_smithi194_196924_object148508:head# 2024-04-29T15:00:11.056 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:458440b3:::benchmark_data_smithi194_196924_object109283:head# 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45844e2f:::benchmark_data_smithi194_196924_object159201:head# 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4585a349:::benchmark_data_smithi194_196924_object130770:head# 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:458619fc:::benchmark_data_smithi194_196924_object141042:head# 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4586a4f3:::benchmark_data_smithi194_196924_object113092:head# 2024-04-29T15:00:11.057 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.074 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:458913de:::benchmark_data_smithi194_196924_object107129:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:458925c9:::benchmark_data_smithi194_196924_object143383:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:458c58ea:::benchmark_data_smithi194_196924_object114024:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:458e7975:::benchmark_data_smithi194_196924_object105376:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:459165f4:::benchmark_data_smithi194_196924_object133738:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45918a1c:::benchmark_data_smithi194_196924_object114359:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4593e162:::benchmark_data_smithi194_196924_object154625:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4594c6a0:::benchmark_data_smithi194_196924_object102899:head# 2024-04-29T15:00:11.075 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4596acd4:::benchmark_data_smithi194_196924_object134603:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45971702:::benchmark_data_smithi194_196924_object112188:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45999074:::benchmark_data_smithi194_196924_object98134:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:459b2826:::benchmark_data_smithi194_196924_object133821:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:459bd592:::benchmark_data_smithi194_196924_object127824:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:459e7b2c:::benchmark_data_smithi194_196924_object149530:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:459ea8d4:::benchmark_data_smithi194_196924_object96829:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:459ed970:::benchmark_data_smithi194_196924_object155388:head# 2024-04-29T15:00:11.076 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45a12a29:::benchmark_data_smithi194_196924_object150629:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45a3c32a:::benchmark_data_smithi194_196924_object110200:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45a45be3:::benchmark_data_smithi194_196924_object101589:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45a5e87c:::benchmark_data_smithi194_196924_object117975:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45a67b6c:::benchmark_data_smithi194_196924_object120561:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45a749ce:::benchmark_data_smithi194_196924_object146901:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45a96be5:::benchmark_data_smithi194_196924_object148544:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45ab0eeb:::benchmark_data_smithi194_196924_object97880:head# 2024-04-29T15:00:11.077 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45aee2c2:::benchmark_data_smithi194_196924_object120527:head# 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45af8f5c:::benchmark_data_smithi194_196924_object114278:head# 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45af9f98:::benchmark_data_smithi194_196924_object157539:head# 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b01b33:::benchmark_data_smithi194_196924_object112713:head# 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b245a9:::benchmark_data_smithi194_196924_object137258:head# 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b3f6ab:::benchmark_data_smithi194_196924_object100198:head# 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b51537:::benchmark_data_smithi194_196924_object115177:head# 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.078 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b5b47b:::benchmark_data_smithi194_196924_object123675:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b617df:::benchmark_data_smithi194_196924_object158236:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b6938d:::benchmark_data_smithi194_196924_object151190:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45b8b937:::benchmark_data_smithi194_196924_object120781:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45ba3ad7:::benchmark_data_smithi194_196924_object114823:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45bad676:::benchmark_data_smithi194_196924_object134439:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45baffeb:::benchmark_data_smithi194_196924_object147521:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45bc4327:::benchmark_data_smithi194_196924_object141986:head# 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45bd6609:::benchmark_data_smithi194_196924_object123119:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45be3df4:::benchmark_data_smithi194_196924_object102748:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45be4af9:::benchmark_data_smithi194_196924_object135782:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45be7311:::benchmark_data_smithi194_196924_object150931:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45be86bd:::benchmark_data_smithi194_196924_object128629:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45be9652:::benchmark_data_smithi194_196924_object139222:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45be99a5:::benchmark_data_smithi194_196924_object118455:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45bee3e6:::benchmark_data_smithi194_196924_object117301:head# 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.080 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45bfc9bd:::benchmark_data_smithi194_196924_object109294:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c28d9c:::benchmark_data_smithi194_196924_object114665:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c3100a:::benchmark_data_smithi194_196924_object151199:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c327df:::benchmark_data_smithi194_196924_object159289:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c351c8:::benchmark_data_smithi194_196924_object135357:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c583ae:::benchmark_data_smithi194_196924_object144725:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c62849:::benchmark_data_smithi194_196924_object109575:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c688fd:::benchmark_data_smithi194_196924_object102147:head# 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.081 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c6e1e9:::benchmark_data_smithi194_196924_object138022:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45c8e00f:::benchmark_data_smithi194_196924_object115418:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45ca8d70:::benchmark_data_smithi194_196924_object122742:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cae475:::benchmark_data_smithi194_196924_object121197:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cc1407:::benchmark_data_smithi194_196924_object152899:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cc6671:::benchmark_data_smithi194_196924_object140915:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cca6df:::benchmark_data_smithi194_196924_object108556:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cd7124:::benchmark_data_smithi194_196924_object152617:head# 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.082 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cd9a03:::benchmark_data_smithi194_196924_object131578:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cebb22:::benchmark_data_smithi194_196924_object101624:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45cf84ca:::benchmark_data_smithi194_196924_object158130:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d1590d:::benchmark_data_smithi194_196924_object147377:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d194c5:::benchmark_data_smithi194_196924_object140346:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d1b49f:::benchmark_data_smithi194_196924_object154096:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d28421:::benchmark_data_smithi194_196924_object139309:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d37183:::benchmark_data_smithi194_196924_object136512:head# 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.083 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d476ec:::benchmark_data_smithi194_196924_object131025:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d4aa34:::benchmark_data_smithi194_196924_object141952:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d4e7b2:::benchmark_data_smithi194_196924_object110545:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d509a1:::benchmark_data_smithi194_196924_object152427:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d509c5:::benchmark_data_smithi194_196924_object147205:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d55ca1:::benchmark_data_smithi194_196924_object101194:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d55dec:::benchmark_data_smithi194_196924_object96655:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d632b2:::benchmark_data_smithi194_196924_object151469:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d6dc0f:::benchmark_data_smithi194_196924_object153730:head# 2024-04-29T15:00:11.084 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d6ff1b:::benchmark_data_smithi194_196924_object155329:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d700ef:::benchmark_data_smithi194_196924_object128541:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45d899f3:::benchmark_data_smithi194_196924_object120938:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45da56d7:::benchmark_data_smithi194_196924_object148820:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45de4c10:::benchmark_data_smithi194_196924_object106221:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45dfd5bf:::benchmark_data_smithi194_196924_object155715:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e0799a:::benchmark_data_smithi194_196924_object112103:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e0f73f:::benchmark_data_smithi194_196924_object118469:head# 2024-04-29T15:00:11.085 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e1a4be:::benchmark_data_smithi194_196924_object123229:head# 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e2dec4:::benchmark_data_smithi194_196924_object140448:head# 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e47852:::benchmark_data_smithi194_196924_object129288:head# 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e51421:::benchmark_data_smithi194_196924_object102452:head# 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e5c084:::benchmark_data_smithi194_196924_object102214:head# 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e63e6c:::benchmark_data_smithi194_196924_object101367:head# 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e66ab4:::benchmark_data_smithi194_196924_object158620:head# 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.086 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e6a08a:::benchmark_data_smithi194_196924_object136518:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e7bb02:::benchmark_data_smithi194_196924_object97636:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e82a30:::benchmark_data_smithi194_196924_object103328:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e85bf7:::benchmark_data_smithi194_196924_object138512:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e87128:::benchmark_data_smithi194_196924_object159224:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45e88d20:::benchmark_data_smithi194_196924_object116273:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f0a688:::benchmark_data_smithi194_196924_object142573:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f13cfd:::benchmark_data_smithi194_196924_object120869:head# 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f28466:::benchmark_data_smithi194_196924_object139885:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f35ccf:::benchmark_data_smithi194_196924_object129728:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f3e5fe:::benchmark_data_smithi194_196924_object154168:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f46865:::benchmark_data_smithi194_196924_object138094:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f4f51b:::benchmark_data_smithi194_196924_object157301:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f69577:::benchmark_data_smithi194_196924_object127474:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f72158:::benchmark_data_smithi194_196924_object156977:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45f7a9c7:::benchmark_data_smithi194_196924_object142709:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fa9348:::benchmark_data_smithi194_196924_object120778:head# 2024-04-29T15:00:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45faa9c7:::benchmark_data_smithi194_196924_object131857:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fc3dff:::benchmark_data_smithi194_196924_object104264:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fc6d68:::benchmark_data_smithi194_196924_object98249:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fcba32:::benchmark_data_smithi194_196924_object108132:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fce540:::benchmark_data_smithi194_196924_object139238:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fcf271:::benchmark_data_smithi194_196924_object120632:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fd7489:::benchmark_data_smithi194_196924_object158095:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:45fdd0f7:::benchmark_data_smithi194_196924_object116440:head# 2024-04-29T15:00:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46005925:::benchmark_data_smithi194_196924_object133922:head# 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4600a1b4:::benchmark_data_smithi194_196924_object106733:head# 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4600dbe1:::benchmark_data_smithi194_196924_object115701:head# 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46018e3a:::benchmark_data_smithi194_196924_object106517:head# 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4603e513:::benchmark_data_smithi194_196924_object126565:head# 2024-04-29T15:00:11.090 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.097 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46052b03:::benchmark_data_smithi194_196924_object140486:head# 2024-04-29T15:00:11.097 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.097 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4606ba8a:::benchmark_data_smithi194_196924_object147386:head# 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4606d5b7:::benchmark_data_smithi194_196924_object103394:head# 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46072bca:::benchmark_data_smithi194_196924_object138199:head# 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46074bcc:::benchmark_data_smithi194_196924_object142521:head# 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46076a94:::benchmark_data_smithi194_196924_object109163:head# 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4607840c:::benchmark_data_smithi194_196924_object123818:head# 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4607a94f:::benchmark_data_smithi194_196924_object158010:head# 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.098 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460806a7:::benchmark_data_smithi194_196924_object138378:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46083b33:::benchmark_data_smithi194_196924_object150838:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46090aeb:::benchmark_data_smithi194_196924_object114705:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46095bee:::benchmark_data_smithi194_196924_object107309:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46096bb5:::benchmark_data_smithi194_196924_object158288:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460973b1:::benchmark_data_smithi194_196924_object140079:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4609e98f:::benchmark_data_smithi194_196924_object125440:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460a7aa9:::benchmark_data_smithi194_196924_object110055:head# 2024-04-29T15:00:11.099 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460a91ca:::benchmark_data_smithi194_196924_object149593:head# 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460b1131:::benchmark_data_smithi194_196924_object99011:head# 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460b2a73:::benchmark_data_smithi194_196924_object137211:head# 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460b3033:::benchmark_data_smithi194_196924_object119278:head# 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460b336b:::benchmark_data_smithi194_196924_object136004:head# 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460b5b05:::benchmark_data_smithi194_196924_object150620:head# 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460bae3a:::benchmark_data_smithi194_196924_object97994:head# 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.100 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460be435:::benchmark_data_smithi194_196924_object123482:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460c1efb:::benchmark_data_smithi194_196924_object104723:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460c8bc9:::benchmark_data_smithi194_196924_object155127:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460dd167:::benchmark_data_smithi194_196924_object122768:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460e566a:::benchmark_data_smithi194_196924_object129417:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460fb6af:::benchmark_data_smithi194_196924_object131140:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:460fc496:::benchmark_data_smithi194_196924_object158684:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4610c221:::benchmark_data_smithi194_196924_object136311:head# 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4610d092:::benchmark_data_smithi194_196924_object120142:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:461118ee:::benchmark_data_smithi194_196924_object144980:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:461131f9:::benchmark_data_smithi194_196924_object159313:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4611c2ea:::benchmark_data_smithi194_196924_object112403:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46126934:::benchmark_data_smithi194_196924_object123957:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46134089:::benchmark_data_smithi194_196924_object122411:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4613a78e:::benchmark_data_smithi194_196924_object146763:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46145db6:::benchmark_data_smithi194_196924_object122427:head# 2024-04-29T15:00:11.102 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46148242:::benchmark_data_smithi194_196924_object123443:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4616a254:::benchmark_data_smithi194_196924_object159662:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4617571c:::benchmark_data_smithi194_196924_object146366:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4617cc1a:::benchmark_data_smithi194_196924_object130794:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:461873e6:::benchmark_data_smithi194_196924_object100008:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4618f477:::benchmark_data_smithi194_196924_object135079:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:461a075a:::benchmark_data_smithi194_196924_object122459:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:461af4bf:::benchmark_data_smithi194_196924_object109949:head# 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:461c6ec0:::benchmark_data_smithi194_196924_object148135:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:461f4be8:::benchmark_data_smithi194_196924_object125744:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46206a54:::benchmark_data_smithi194_196924_object114020:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4620f33f:::benchmark_data_smithi194_196924_object153679:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46216290:::benchmark_data_smithi194_196924_object142147:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:462182bd:::benchmark_data_smithi194_196924_object158189:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46229625:::benchmark_data_smithi194_196924_object159159:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4622fd5f:::benchmark_data_smithi194_196924_object112835:head# 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.104 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4623dc3e:::benchmark_data_smithi194_196924_object114240:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46247491:::benchmark_data_smithi194_196924_object142017:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46264c5e:::benchmark_data_smithi194_196924_object123164:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4626a6f6:::benchmark_data_smithi194_196924_object117011:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46273c3f:::benchmark_data_smithi194_196924_object137119:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4628d197:::benchmark_data_smithi194_196924_object102801:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:462a166c:::benchmark_data_smithi194_196924_object121631:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:462d8dc8:::benchmark_data_smithi194_196924_object134092:head# 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.105 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:462ddde7:::benchmark_data_smithi194_196924_object101648:head# 2024-04-29T15:00:11.106 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.106 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:462dff1b:::benchmark_data_smithi194_196924_object141072:head# 2024-04-29T15:00:11.106 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46307e2f:::benchmark_data_smithi194_196924_object142999:head# 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463156d4:::benchmark_data_smithi194_196924_object142721:head# 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463412d0:::benchmark_data_smithi194_196924_object123103:head# 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463439a8:::benchmark_data_smithi194_196924_object116683:head# 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46357b27:::benchmark_data_smithi194_196924_object150981:head# 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4637eb4b:::benchmark_data_smithi194_196924_object140176:head# 2024-04-29T15:00:11.109 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46383b23:::benchmark_data_smithi194_196924_object125022:head# 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4638d7b5:::benchmark_data_smithi194_196924_object150052:head# 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4638fcbc:::benchmark_data_smithi194_196924_object106180:head# 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4639156e:::benchmark_data_smithi194_196924_object102644:head# 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463953c5:::benchmark_data_smithi194_196924_object151092:head# 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463979b3:::benchmark_data_smithi194_196924_object113669:head# 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463c47ef:::benchmark_data_smithi194_196924_object120955:head# 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.110 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463cea5c:::benchmark_data_smithi194_196924_object119405:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463d0a93:::benchmark_data_smithi194_196924_object123198:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463d5592:::benchmark_data_smithi194_196924_object133468:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:463e3cf5:::benchmark_data_smithi194_196924_object131510:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464140e9:::benchmark_data_smithi194_196924_object137382:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464243eb:::benchmark_data_smithi194_196924_object151854:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46441bde:::benchmark_data_smithi194_196924_object125830:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4644f2d0:::benchmark_data_smithi194_196924_object101812:head# 2024-04-29T15:00:11.111 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46457764:::benchmark_data_smithi194_196924_object128816:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464588cc:::benchmark_data_smithi194_196924_object146145:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4645a8e2:::benchmark_data_smithi194_196924_object133627:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46462e9b:::benchmark_data_smithi194_196924_object114227:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46480d22:::benchmark_data_smithi194_196924_object123426:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464b1467:::benchmark_data_smithi194_196924_object152994:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464b2a2d:::benchmark_data_smithi194_196924_object157240:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464b4625:::benchmark_data_smithi194_196924_object107789:head# 2024-04-29T15:00:11.112 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464b5513:::benchmark_data_smithi194_196924_object142431:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464d1d74:::benchmark_data_smithi194_196924_object138548:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464e9a4d:::benchmark_data_smithi194_196924_object116646:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464ef374:::benchmark_data_smithi194_196924_object159190:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:464f6208:::benchmark_data_smithi194_196924_object113408:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465121a5:::benchmark_data_smithi194_196924_object137867:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46517ba1:::benchmark_data_smithi194_196924_object149711:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4651b16e:::benchmark_data_smithi194_196924_object117807:head# 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.113 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4652a38c:::benchmark_data_smithi194_196924_object154612:head# 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46534065:::benchmark_data_smithi194_196924_object101510:head# 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4654b318:::benchmark_data_smithi194_196924_object109484:head# 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46558f0a:::benchmark_data_smithi194_196924_object124183:head# 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4657dff9:::benchmark_data_smithi194_196924_object153714:head# 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4659d3d6:::benchmark_data_smithi194_196924_object121092:head# 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.114 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4659f96c:::benchmark_data_smithi194_196924_object131744:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465a74ad:::benchmark_data_smithi194_196924_object140930:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465af1d0:::benchmark_data_smithi194_196924_object139892:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465c4dd7:::benchmark_data_smithi194_196924_object142280:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465c6760:::benchmark_data_smithi194_196924_object135716:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465ce243:::benchmark_data_smithi194_196924_object138723:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465cee67:::benchmark_data_smithi194_196924_object151195:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465dd7d7:::benchmark_data_smithi194_196924_object158940:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:465f456a:::benchmark_data_smithi194_196924_object96615:head# 2024-04-29T15:00:11.115 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4660b7c9:::benchmark_data_smithi194_196924_object157989:head# 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46623fb1:::benchmark_data_smithi194_196924_object101087:head# 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4663d8c3:::benchmark_data_smithi194_196924_object143750:head# 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46654cac:::benchmark_data_smithi194_196924_object130091:head# 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4665eecb:::benchmark_data_smithi194_196924_object101010:head# 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46664b84:::benchmark_data_smithi194_196924_object104265:head# 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46678086:::benchmark_data_smithi194_196924_object109621:head# 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.116 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4668216a:::benchmark_data_smithi194_196924_object102546:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46683397:::benchmark_data_smithi194_196924_object122952:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46691db4:::benchmark_data_smithi194_196924_object107735:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4669a24e:::benchmark_data_smithi194_196924_object108251:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4669f0a1:::benchmark_data_smithi194_196924_object130182:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:466b7e90:::benchmark_data_smithi194_196924_object95421:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:466bce5e:::benchmark_data_smithi194_196924_object127897:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:466c9caf:::benchmark_data_smithi194_196924_object143656:head# 2024-04-29T15:00:11.117 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:466d9ca0:::benchmark_data_smithi194_196924_object155939:head# 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467083e3:::benchmark_data_smithi194_196924_object153055:head# 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467175cd:::benchmark_data_smithi194_196924_object138017:head# 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4671f571:::benchmark_data_smithi194_196924_object127017:head# 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46725e8b:::benchmark_data_smithi194_196924_object120191:head# 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46728e00:::benchmark_data_smithi194_196924_object145516:head# 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.118 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4672f849:::benchmark_data_smithi194_196924_object106020:head# 2024-04-29T15:00:11.119 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.119 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4673d2aa:::benchmark_data_smithi194_196924_object110187:head# 2024-04-29T15:00:11.119 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.119 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46745d09:::benchmark_data_smithi194_196924_object100006:head# 2024-04-29T15:00:11.119 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4674fdcd:::benchmark_data_smithi194_196924_object141230:head# 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46756504:::benchmark_data_smithi194_196924_object152572:head# 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46756c3f:::benchmark_data_smithi194_196924_object139714:head# 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46776c7e:::benchmark_data_smithi194_196924_object153848:head# 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46787026:::benchmark_data_smithi194_196924_object149173:head# 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467a1d9a:::benchmark_data_smithi194_196924_object106906:head# 2024-04-29T15:00:11.128 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467b2565:::benchmark_data_smithi194_196924_object142901:head# 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467b884a:::benchmark_data_smithi194_196924_object104012:head# 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467ba1e4:::benchmark_data_smithi194_196924_object141639:head# 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467be846:::benchmark_data_smithi194_196924_object107295:head# 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467d44b7:::benchmark_data_smithi194_196924_object129577:head# 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467dc146:::benchmark_data_smithi194_196924_object159707:head# 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:467e69a9:::benchmark_data_smithi194_196924_object126742:head# 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.129 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46815941:::benchmark_data_smithi194_196924_object101509:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4681925f:::benchmark_data_smithi194_196924_object147516:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4683c7d1:::benchmark_data_smithi194_196924_object142868:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468516d5:::benchmark_data_smithi194_196924_object123506:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4685da09:::benchmark_data_smithi194_196924_object139598:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468606de:::benchmark_data_smithi194_196924_object116255:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4687e058:::benchmark_data_smithi194_196924_object133614:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4688b2bb:::benchmark_data_smithi194_196924_object133316:head# 2024-04-29T15:00:11.130 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46892a46:::benchmark_data_smithi194_196924_object149442:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468942c2:::benchmark_data_smithi194_196924_object114257:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4689a805:::benchmark_data_smithi194_196924_object142082:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468a17d2:::benchmark_data_smithi194_196924_object117164:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468ab0a7:::benchmark_data_smithi194_196924_object159262:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468b6b8b:::benchmark_data_smithi194_196924_object138947:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468d12ad:::benchmark_data_smithi194_196924_object141165:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468e1cf5:::benchmark_data_smithi194_196924_object128494:head# 2024-04-29T15:00:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468e88e4:::benchmark_data_smithi194_196924_object135252:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:468f82ef:::benchmark_data_smithi194_196924_object118371:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46900fc0:::benchmark_data_smithi194_196924_object116876:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469114db:::benchmark_data_smithi194_196924_object155110:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4691fb58:::benchmark_data_smithi194_196924_object115026:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4692ceed:::benchmark_data_smithi194_196924_object143013:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4692e948:::benchmark_data_smithi194_196924_object105140:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46963fd4:::benchmark_data_smithi194_196924_object110121:head# 2024-04-29T15:00:11.132 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46973475:::benchmark_data_smithi194_196924_object110361:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46987a88:::benchmark_data_smithi194_196924_object97808:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46999489:::benchmark_data_smithi194_196924_object96140:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469aae19:::benchmark_data_smithi194_196924_object122528:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469bcc2a:::benchmark_data_smithi194_196924_object106699:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469bfd54:::benchmark_data_smithi194_196924_object159191:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469c8aab:::benchmark_data_smithi194_196924_object118162:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469e04df:::benchmark_data_smithi194_196924_object133936:head# 2024-04-29T15:00:11.133 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469e42bb:::benchmark_data_smithi194_196924_object118814:head# 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469e4df2:::benchmark_data_smithi194_196924_object149193:head# 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:469f388e:::benchmark_data_smithi194_196924_object119756:head# 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a031f3:::benchmark_data_smithi194_196924_object154784:head# 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a04b1f:::benchmark_data_smithi194_196924_object133901:head# 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a0bb6f:::benchmark_data_smithi194_196924_object132542:head# 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a0fa19:::benchmark_data_smithi194_196924_object107115:head# 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.134 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a29dd7:::benchmark_data_smithi194_196924_object110697:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a32976:::benchmark_data_smithi194_196924_object155711:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a4efd5:::benchmark_data_smithi194_196924_object121435:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a715a8:::benchmark_data_smithi194_196924_object120410:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a83cc2:::benchmark_data_smithi194_196924_object128008:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a8726c:::benchmark_data_smithi194_196924_object125796:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a8d31e:::benchmark_data_smithi194_196924_object133431:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a96cf6:::benchmark_data_smithi194_196924_object154651:head# 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.135 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46a9fd33:::benchmark_data_smithi194_196924_object113443:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ae1ec1:::benchmark_data_smithi194_196924_object101008:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46b20473:::benchmark_data_smithi194_196924_object128756:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46b25fac:::benchmark_data_smithi194_196924_object111403:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46b266c6:::benchmark_data_smithi194_196924_object148173:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46b40f86:::benchmark_data_smithi194_196924_object125628:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46b5c6f1:::benchmark_data_smithi194_196924_object121744:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46b5e7be:::benchmark_data_smithi194_196924_object117665:head# 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.136 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46b86682:::benchmark_data_smithi194_196924_object130799:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ba1a9a:::benchmark_data_smithi194_196924_object115546:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ba27d9:::benchmark_data_smithi194_196924_object149049:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46bb5ea4:::benchmark_data_smithi194_196924_object97523:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46bc28a6:::benchmark_data_smithi194_196924_object108888:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46be338a:::benchmark_data_smithi194_196924_object106712:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46be686e:::benchmark_data_smithi194_196924_object133376:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46bf9d9c:::benchmark_data_smithi194_196924_object108121:head# 2024-04-29T15:00:11.137 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.138 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46bfdbf0:::benchmark_data_smithi194_196924_object133548:head# 2024-04-29T15:00:11.138 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.138 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c0022e:::benchmark_data_smithi194_196924_object112486:head# 2024-04-29T15:00:11.138 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.149 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c0b0b8:::benchmark_data_smithi194_196924_object122488:head# 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c2af14:::benchmark_data_smithi194_196924_object124252:head# 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c33195:::benchmark_data_smithi194_196924_object124197:head# 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c349dd:::benchmark_data_smithi194_196924_object119925:head# 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c49e43:::benchmark_data_smithi194_196924_object142805:head# 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c7499a:::benchmark_data_smithi194_196924_object96712:head# 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c7608e:::benchmark_data_smithi194_196924_object130598:head# 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.150 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c81325:::benchmark_data_smithi194_196924_object105657:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c8de62:::benchmark_data_smithi194_196924_object143839:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46c9fd81:::benchmark_data_smithi194_196924_object123104:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ca0a4a:::benchmark_data_smithi194_196924_object104936:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46caab66:::benchmark_data_smithi194_196924_object112605:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46cb335a:::benchmark_data_smithi194_196924_object129881:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46cb7bc1:::benchmark_data_smithi194_196924_object118554:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46cbf22c:::benchmark_data_smithi194_196924_object140998:head# 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46cc3dcb:::benchmark_data_smithi194_196924_object145696:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46cd2e04:::benchmark_data_smithi194_196924_object129216:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46cd6338:::benchmark_data_smithi194_196924_object154533:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ce9b6d:::benchmark_data_smithi194_196924_object131635:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d0e33a:::benchmark_data_smithi194_196924_object129452:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d1efa1:::benchmark_data_smithi194_196924_object153562:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d2e4a6:::benchmark_data_smithi194_196924_object108049:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d3b1db:::benchmark_data_smithi194_196924_object114523:head# 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.152 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d3f82e:::benchmark_data_smithi194_196924_object101897:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d4791c:::benchmark_data_smithi194_196924_object112582:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d4c8e4:::benchmark_data_smithi194_196924_object146960:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d60e99:::benchmark_data_smithi194_196924_object150833:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d76206:::benchmark_data_smithi194_196924_object132059:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d78e4c:::benchmark_data_smithi194_196924_object145754:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d85571:::benchmark_data_smithi194_196924_object120743:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46d8c6fe:::benchmark_data_smithi194_196924_object106350:head# 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.153 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46db4525:::benchmark_data_smithi194_196924_object158566:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46db610c:::benchmark_data_smithi194_196924_object116154:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46dbb228:::benchmark_data_smithi194_196924_object146398:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46dffd46:::benchmark_data_smithi194_196924_object105371:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e0e3ee:::benchmark_data_smithi194_196924_object118391:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e18665:::benchmark_data_smithi194_196924_object110605:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e1e5b3:::benchmark_data_smithi194_196924_object148299:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e38b2a:::benchmark_data_smithi194_196924_object98709:head# 2024-04-29T15:00:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e3ca82:::benchmark_data_smithi194_196924_object144446:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e3df9f:::benchmark_data_smithi194_196924_object104963:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e4d0de:::benchmark_data_smithi194_196924_object138095:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e5bec9:::benchmark_data_smithi194_196924_object132693:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e62b86:::benchmark_data_smithi194_196924_object129454:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e80180:::benchmark_data_smithi194_196924_object114668:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46e92c4f:::benchmark_data_smithi194_196924_object137301:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ea054e:::benchmark_data_smithi194_196924_object158445:head# 2024-04-29T15:00:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ea46fd:::benchmark_data_smithi194_196924_object97228:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46eb6ef0:::benchmark_data_smithi194_196924_object109159:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ed615d:::benchmark_data_smithi194_196924_object148911:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ee0387:::benchmark_data_smithi194_196924_object127523:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ee543c:::benchmark_data_smithi194_196924_object137091:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ee889a:::benchmark_data_smithi194_196924_object144829:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ef31ff:::benchmark_data_smithi194_196924_object150323:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ef47f0:::benchmark_data_smithi194_196924_object158107:head# 2024-04-29T15:00:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ef5c69:::benchmark_data_smithi194_196924_object159903:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f03487:::benchmark_data_smithi194_196924_object129535:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f0d771:::benchmark_data_smithi194_196924_object119419:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f1415f:::benchmark_data_smithi194_196924_object108180:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f1da6c:::benchmark_data_smithi194_196924_object124712:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f26b4a:::benchmark_data_smithi194_196924_object136296:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f29778:::benchmark_data_smithi194_196924_object155536:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f2fd25:::benchmark_data_smithi194_196924_object98097:head# 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f48e57:::benchmark_data_smithi194_196924_object95777:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f7836e:::benchmark_data_smithi194_196924_object116372:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f83c3c:::benchmark_data_smithi194_196924_object144872:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f989e8:::benchmark_data_smithi194_196924_object125415:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f9a5f7:::benchmark_data_smithi194_196924_object138795:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46f9c6f9:::benchmark_data_smithi194_196924_object116730:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46fab54a:::benchmark_data_smithi194_196924_object158969:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46fba497:::benchmark_data_smithi194_196924_object153575:head# 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46fbabf3:::benchmark_data_smithi194_196924_object157711:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46fc6638:::benchmark_data_smithi194_196924_object110158:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46fcb0da:::benchmark_data_smithi194_196924_object159353:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46fd0630:::benchmark_data_smithi194_196924_object110244:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:46ff6e79:::benchmark_data_smithi194_196924_object145819:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4700ff77:::benchmark_data_smithi194_196924_object119882:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47018fe7:::benchmark_data_smithi194_196924_object118860:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47021004:::benchmark_data_smithi194_196924_object123991:head# 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4702ebe6:::benchmark_data_smithi194_196924_object158688:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4702ec41:::benchmark_data_smithi194_196924_object118386:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47037174:::benchmark_data_smithi194_196924_object122049:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47072a74:::benchmark_data_smithi194_196924_object146745:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4707c90d:::benchmark_data_smithi194_196924_object142870:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4708df02:::benchmark_data_smithi194_196924_object158168:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:470902bc:::benchmark_data_smithi194_196924_object110800:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:470acb05:::benchmark_data_smithi194_196924_object128361:head# 2024-04-29T15:00:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:470adaf9:::benchmark_data_smithi194_196924_object118643:head# 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:470c010f:::benchmark_data_smithi194_196924_object119520:head# 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:470d9939:::benchmark_data_smithi194_196924_object155672:head# 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47110c42:::benchmark_data_smithi194_196924_object117222:head# 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47121fbe:::benchmark_data_smithi194_196924_object141029:head# 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:471275f6:::benchmark_data_smithi194_196924_object124771:head# 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4712f8bb:::benchmark_data_smithi194_196924_object151308:head# 2024-04-29T15:00:11.173 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4713135d:::benchmark_data_smithi194_196924_object103717:head# 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:471340ef:::benchmark_data_smithi194_196924_object145449:head# 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47135466:::benchmark_data_smithi194_196924_object148587:head# 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47157e4e:::benchmark_data_smithi194_196924_object95676:head# 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4715e99b:::benchmark_data_smithi194_196924_object130543:head# 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4716410d:::benchmark_data_smithi194_196924_object156523:head# 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:471680af:::benchmark_data_smithi194_196924_object108275:head# 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.174 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47176bc5:::benchmark_data_smithi194_196924_object135707:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4718ec43:::benchmark_data_smithi194_196924_object147255:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:471905da:::benchmark_data_smithi194_196924_object105237:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:471d2346:::benchmark_data_smithi194_196924_object98635:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:471fd331:::benchmark_data_smithi194_196924_object140751:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4720fe37:::benchmark_data_smithi194_196924_object124543:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472244fb:::benchmark_data_smithi194_196924_object125363:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472301c9:::benchmark_data_smithi194_196924_object132008:head# 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.175 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472384ff:::benchmark_data_smithi194_196924_object135483:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4724259e:::benchmark_data_smithi194_196924_object152646:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472494d5:::benchmark_data_smithi194_196924_object154899:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47254cc4:::benchmark_data_smithi194_196924_object149806:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47269e9d:::benchmark_data_smithi194_196924_object123728:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4726c622:::benchmark_data_smithi194_196924_object98276:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47272d34:::benchmark_data_smithi194_196924_object109228:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4727c57f:::benchmark_data_smithi194_196924_object153738:head# 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.176 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47289bb2:::benchmark_data_smithi194_196924_object150043:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4729b029:::benchmark_data_smithi194_196924_object135381:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472b9811:::benchmark_data_smithi194_196924_object134847:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472c0494:::benchmark_data_smithi194_196924_object102216:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472c43f2:::benchmark_data_smithi194_196924_object124463:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472c7ab6:::benchmark_data_smithi194_196924_object159126:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472d4433:::benchmark_data_smithi194_196924_object135710:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472ddd31:::benchmark_data_smithi194_196924_object147539:head# 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.177 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:472fc010:::benchmark_data_smithi194_196924_object97939:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4730e776:::benchmark_data_smithi194_196924_object111029:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473348b4:::benchmark_data_smithi194_196924_object137349:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473421cc:::benchmark_data_smithi194_196924_object131873:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473424f7:::benchmark_data_smithi194_196924_object129843:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473765e4:::benchmark_data_smithi194_196924_object99315:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4737c041:::benchmark_data_smithi194_196924_object142787:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47388e70:::benchmark_data_smithi194_196924_object153845:head# 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.178 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4739612b:::benchmark_data_smithi194_196924_object103888:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473acfdc:::benchmark_data_smithi194_196924_object132122:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473ae9d5:::benchmark_data_smithi194_196924_object153414:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473bffec:::benchmark_data_smithi194_196924_object131759:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473d2f80:::benchmark_data_smithi194_196924_object153408:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473d3fc5:::benchmark_data_smithi194_196924_object126796:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473f9979:::benchmark_data_smithi194_196924_object138629:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:473fe01d:::benchmark_data_smithi194_196924_object101804:head# 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.179 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:474169a6:::benchmark_data_smithi194_196924_object119804:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47438a7d:::benchmark_data_smithi194_196924_object132033:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47442112:::benchmark_data_smithi194_196924_object133943:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47468daa:::benchmark_data_smithi194_196924_object98324:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4747a373:::benchmark_data_smithi194_196924_object137108:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47484b66:::benchmark_data_smithi194_196924_object113396:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47487d29:::benchmark_data_smithi194_196924_object125948:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:474aa56e:::benchmark_data_smithi194_196924_object145249:head# 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.180 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:474b94d3:::benchmark_data_smithi194_196924_object143024:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:474ce03c:::benchmark_data_smithi194_196924_object130868:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:474e29fe:::benchmark_data_smithi194_196924_object153447:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47508761:::benchmark_data_smithi194_196924_object135630:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47527207:::benchmark_data_smithi194_196924_object121494:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4752c1b3:::benchmark_data_smithi194_196924_object111502:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4755e01b:::benchmark_data_smithi194_196924_object121646:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47561abe:::benchmark_data_smithi194_196924_object123851:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4756ea44:::benchmark_data_smithi194_196924_object144729:head# 2024-04-29T15:00:11.181 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:475853c4:::benchmark_data_smithi194_196924_object148628:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:475a1180:::benchmark_data_smithi194_196924_object121275:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:475c362a:::benchmark_data_smithi194_196924_object130345:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:475e5145:::benchmark_data_smithi194_196924_object107574:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:475f8a38:::benchmark_data_smithi194_196924_object111764:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:475fca31:::benchmark_data_smithi194_196924_object150317:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:475fe450:::benchmark_data_smithi194_196924_object155423:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47603e6e:::benchmark_data_smithi194_196924_object113588:head# 2024-04-29T15:00:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47608d26:::benchmark_data_smithi194_196924_object131863:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47617a99:::benchmark_data_smithi194_196924_object113775:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47617f8b:::benchmark_data_smithi194_196924_object153896:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4761b33d:::benchmark_data_smithi194_196924_object141191:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4761c247:::benchmark_data_smithi194_196924_object96538:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:476270c8:::benchmark_data_smithi194_196924_object131626:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47634253:::benchmark_data_smithi194_196924_object102613:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47648851:::benchmark_data_smithi194_196924_object142079:head# 2024-04-29T15:00:11.183 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4766bb5d:::benchmark_data_smithi194_196924_object157415:head# 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47699c60:::benchmark_data_smithi194_196924_object147027:head# 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4769ba60:::benchmark_data_smithi194_196924_object145224:head# 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4769e108:::benchmark_data_smithi194_196924_object110602:head# 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:476ab735:::benchmark_data_smithi194_196924_object129163:head# 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:476afb01:::benchmark_data_smithi194_196924_object141658:head# 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:476c3153:::benchmark_data_smithi194_196924_object114171:head# 2024-04-29T15:00:11.184 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:11.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:11.196 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:476c6792:::benchmark_data_smithi194_196924_object112962:head# 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:476e4fee:::benchmark_data_smithi194_196924_object107901:head# 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47708f83:::benchmark_data_smithi194_196924_object151738:head# 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47713bdf:::benchmark_data_smithi194_196924_object123663:head# 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47716cf6:::benchmark_data_smithi194_196924_object117155:head# 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4771b3e1:::benchmark_data_smithi194_196924_object145804:head# 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4771ffa1:::benchmark_data_smithi194_196924_object103466:head# 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.197 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4772b835:::benchmark_data_smithi194_196924_object128103:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4772e5de:::benchmark_data_smithi194_196924_object102632:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47736e67:::benchmark_data_smithi194_196924_object100484:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4775e21b:::benchmark_data_smithi194_196924_object137320:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4776f551:::benchmark_data_smithi194_196924_object150627:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4776f60b:::benchmark_data_smithi194_196924_object131999:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4778efbc:::benchmark_data_smithi194_196924_object97408:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4779244d:::benchmark_data_smithi194_196924_object151322:head# 2024-04-29T15:00:11.198 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4779a91e:::benchmark_data_smithi194_196924_object99312:head# 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:477bb02f:::benchmark_data_smithi194_196924_object110096:head# 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:477ceede:::benchmark_data_smithi194_196924_object123995:head# 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:477d9546:::benchmark_data_smithi194_196924_object141928:head# 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47808345:::benchmark_data_smithi194_196924_object118302:head# 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4780f937:::benchmark_data_smithi194_196924_object159351:head# 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478121fd:::benchmark_data_smithi194_196924_object101021:head# 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4781d63d:::benchmark_data_smithi194_196924_object117149:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47829c72:::benchmark_data_smithi194_196924_object134920:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47844316:::benchmark_data_smithi194_196924_object109160:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4785fa71:::benchmark_data_smithi194_196924_object107974:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4788c93d:::benchmark_data_smithi194_196924_object155474:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478910ff:::benchmark_data_smithi194_196924_object149916:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4789880f:::benchmark_data_smithi194_196924_object97466:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478b5327:::benchmark_data_smithi194_196924_object151445:head# 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.200 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478ca298:::benchmark_data_smithi194_196924_object130341:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478cf2ef:::benchmark_data_smithi194_196924_object126314:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478d6658:::benchmark_data_smithi194_196924_object99045:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478da596:::benchmark_data_smithi194_196924_object109058:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478db77f:::benchmark_data_smithi194_196924_object143994:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:478df825:::benchmark_data_smithi194_196924_object101154:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47913361:::benchmark_data_smithi194_196924_object113547:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4791850f:::benchmark_data_smithi194_196924_object101880:head# 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.201 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47918eb6:::benchmark_data_smithi194_196924_object122653:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47921e56:::benchmark_data_smithi194_196924_object104385:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4793a26b:::benchmark_data_smithi194_196924_object125599:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4794d842:::benchmark_data_smithi194_196924_object152919:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47968480:::benchmark_data_smithi194_196924_object102621:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47970b34:::benchmark_data_smithi194_196924_object131584:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4799cfbe:::benchmark_data_smithi194_196924_object142583:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:479a85cc:::benchmark_data_smithi194_196924_object147352:head# 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.202 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:479d16f5:::benchmark_data_smithi194_196924_object155116:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:479e88a1:::benchmark_data_smithi194_196924_object158942:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:479f568b:::benchmark_data_smithi194_196924_object112615:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:479f6169:::benchmark_data_smithi194_196924_object153757:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:479f84e2:::benchmark_data_smithi194_196924_object96542:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:479f8ac6:::benchmark_data_smithi194_196924_object107516:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a03956:::benchmark_data_smithi194_196924_object114923:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a16e96:::benchmark_data_smithi194_196924_object146857:head# 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.203 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a281ec:::benchmark_data_smithi194_196924_object157637:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a29461:::benchmark_data_smithi194_196924_object147919:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a29712:::benchmark_data_smithi194_196924_object98789:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a3be29:::benchmark_data_smithi194_196924_object115402:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a46177:::benchmark_data_smithi194_196924_object159584:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a51fcb:::benchmark_data_smithi194_196924_object143177:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a58dab:::benchmark_data_smithi194_196924_object126052:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a6dd09:::benchmark_data_smithi194_196924_object133171:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a712fc:::benchmark_data_smithi194_196924_object119514:head# 2024-04-29T15:00:11.204 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47a9c505:::benchmark_data_smithi194_196924_object102383:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ac8c0c:::benchmark_data_smithi194_196924_object118172:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47add173:::benchmark_data_smithi194_196924_object111714:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47af192e:::benchmark_data_smithi194_196924_object138982:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47af1e89:::benchmark_data_smithi194_196924_object106950:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b0a98d:::benchmark_data_smithi194_196924_object153651:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b30389:::benchmark_data_smithi194_196924_object99892:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b31476:::benchmark_data_smithi194_196924_object159066:head# 2024-04-29T15:00:11.205 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b344ec:::benchmark_data_smithi194_196924_object131029:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b4b2ef:::benchmark_data_smithi194_196924_object131408:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b511e2:::benchmark_data_smithi194_196924_object115400:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b56083:::benchmark_data_smithi194_196924_object137910:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b66c62:::benchmark_data_smithi194_196924_object131537:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b6d252:::benchmark_data_smithi194_196924_object157058:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b79126:::benchmark_data_smithi194_196924_object118748:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b7cf6b:::benchmark_data_smithi194_196924_object103176:head# 2024-04-29T15:00:11.206 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b8368c:::benchmark_data_smithi194_196924_object98887:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47b9cda1:::benchmark_data_smithi194_196924_object105259:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ba8346:::benchmark_data_smithi194_196924_object112790:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ba851b:::benchmark_data_smithi194_196924_object109474:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47bc899e:::benchmark_data_smithi194_196924_object145735:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c1383e:::benchmark_data_smithi194_196924_object118397:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c16ee5:::benchmark_data_smithi194_196924_object158222:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c23412:::benchmark_data_smithi194_196924_object108116:head# 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.207 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c5fc50:::benchmark_data_smithi194_196924_object98738:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c6e238:::benchmark_data_smithi194_196924_object155854:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c6e977:::benchmark_data_smithi194_196924_object139871:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c87c3d:::benchmark_data_smithi194_196924_object159644:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47c8d9a1:::benchmark_data_smithi194_196924_object123788:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ca0680:::benchmark_data_smithi194_196924_object107297:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47cb3325:::benchmark_data_smithi194_196924_object132806:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ccad5c:::benchmark_data_smithi194_196924_object138041:head# 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.208 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47cef32a:::benchmark_data_smithi194_196924_object141643:head# 2024-04-29T15:00:11.209 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.209 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47cf1953:::benchmark_data_smithi194_196924_object101227:head# 2024-04-29T15:00:11.209 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.222 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d0be17:::benchmark_data_smithi194_196924_object107724:head# 2024-04-29T15:00:11.222 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.222 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d0ea88:::benchmark_data_smithi194_196924_object115811:head# 2024-04-29T15:00:11.222 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.222 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d1b434:::benchmark_data_smithi194_196924_object116266:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d24844:::benchmark_data_smithi194_196924_object127160:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d37161:::benchmark_data_smithi194_196924_object96727:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d3ec8d:::benchmark_data_smithi194_196924_object98875:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d40c65:::benchmark_data_smithi194_196924_object157238:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d4e6ab:::benchmark_data_smithi194_196924_object113235:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d67774:::benchmark_data_smithi194_196924_object124911:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d6a697:::benchmark_data_smithi194_196924_object110328:head# 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.223 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d91ae1:::benchmark_data_smithi194_196924_object135428:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d92cad:::benchmark_data_smithi194_196924_object152090:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47d96948:::benchmark_data_smithi194_196924_object143412:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47da15ec:::benchmark_data_smithi194_196924_object130384:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47da6755:::benchmark_data_smithi194_196924_object131552:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47db21c2:::benchmark_data_smithi194_196924_object140641:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47db9a4e:::benchmark_data_smithi194_196924_object157046:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47df1794:::benchmark_data_smithi194_196924_object158720:head# 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.224 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47df98b6:::benchmark_data_smithi194_196924_object100142:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e09007:::benchmark_data_smithi194_196924_object98307:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e19e67:::benchmark_data_smithi194_196924_object98473:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e25037:::benchmark_data_smithi194_196924_object114893:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e54215:::benchmark_data_smithi194_196924_object155573:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e584f5:::benchmark_data_smithi194_196924_object112013:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e6fb5d:::benchmark_data_smithi194_196924_object123232:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e8436b:::benchmark_data_smithi194_196924_object131981:head# 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.225 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e8b7e9:::benchmark_data_smithi194_196924_object101522:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e957b6:::benchmark_data_smithi194_196924_object120449:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e99a59:::benchmark_data_smithi194_196924_object138356:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47e9f12c:::benchmark_data_smithi194_196924_object153103:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ea390b:::benchmark_data_smithi194_196924_object150557:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47eadfc3:::benchmark_data_smithi194_196924_object138079:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ecf053:::benchmark_data_smithi194_196924_object135248:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ee0e1a:::benchmark_data_smithi194_196924_object153851:head# 2024-04-29T15:00:11.226 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ee693f:::benchmark_data_smithi194_196924_object145481:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ef3452:::benchmark_data_smithi194_196924_object100663:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47f3c0ee:::benchmark_data_smithi194_196924_object134817:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47f4196d:::benchmark_data_smithi194_196924_object125283:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47f663dc:::benchmark_data_smithi194_196924_object128744:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47f83335:::benchmark_data_smithi194_196924_object150985:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47fa77b8:::benchmark_data_smithi194_196924_object114433:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47fb6f4f:::benchmark_data_smithi194_196924_object112900:head# 2024-04-29T15:00:11.227 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47fd9929:::benchmark_data_smithi194_196924_object157799:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47fe37aa:::benchmark_data_smithi194_196924_object140229:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:47ff2b33:::benchmark_data_smithi194_196924_object144737:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:480329c5:::benchmark_data_smithi194_196924_object118213:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:480710a7:::benchmark_data_smithi194_196924_object146615:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4808e686:::benchmark_data_smithi194_196924_object133349:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:480a1ffc:::benchmark_data_smithi194_196924_object128844:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:480a4382:::benchmark_data_smithi194_196924_object141702:head# 2024-04-29T15:00:11.228 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:480b4856:::benchmark_data_smithi194_196924_object97341:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:480f7dda:::benchmark_data_smithi194_196924_object113344:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48104e37:::benchmark_data_smithi194_196924_object152292:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:481285ce:::benchmark_data_smithi194_196924_object121396:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48128fc0:::benchmark_data_smithi194_196924_object129674:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:481373e1:::benchmark_data_smithi194_196924_object143144:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4813e0ea:::benchmark_data_smithi194_196924_object142360:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48179ed9:::benchmark_data_smithi194_196924_object117391:head# 2024-04-29T15:00:11.229 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4817b13a:::benchmark_data_smithi194_196924_object119167:head# 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48197240:::benchmark_data_smithi194_196924_object112363:head# 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:481ac5f1:::benchmark_data_smithi194_196924_object101254:head# 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:481d462d:::benchmark_data_smithi194_196924_object112870:head# 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:481f19b9:::benchmark_data_smithi194_196924_object151434:head# 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4820e0a7:::benchmark_data_smithi194_196924_object144727:head# 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48210059:::benchmark_data_smithi194_196924_object136842:head# 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.230 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482116bc:::benchmark_data_smithi194_196924_object155352:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4821e826:::benchmark_data_smithi194_196924_object121452:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4821f281:::benchmark_data_smithi194_196924_object120794:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4823c071:::benchmark_data_smithi194_196924_object100923:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4824f345:::benchmark_data_smithi194_196924_object155947:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482521d4:::benchmark_data_smithi194_196924_object129947:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4825b582:::benchmark_data_smithi194_196924_object144367:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4827717f:::benchmark_data_smithi194_196924_object157237:head# 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.231 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48277f0c:::benchmark_data_smithi194_196924_object98681:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4827acbc:::benchmark_data_smithi194_196924_object123781:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4828433a:::benchmark_data_smithi194_196924_object103536:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4828cae6:::benchmark_data_smithi194_196924_object115200:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4829cf9a:::benchmark_data_smithi194_196924_object143153:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4829ee3e:::benchmark_data_smithi194_196924_object158825:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482ad1f1:::benchmark_data_smithi194_196924_object159108:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482afb89:::benchmark_data_smithi194_196924_object132178:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482b42b9:::benchmark_data_smithi194_196924_object132743:head# 2024-04-29T15:00:11.232 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482b991c:::benchmark_data_smithi194_196924_object132727:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482c032e:::benchmark_data_smithi194_196924_object110982:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482d2feb:::benchmark_data_smithi194_196924_object118180:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482ec72a:::benchmark_data_smithi194_196924_object114367:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:482f6319:::benchmark_data_smithi194_196924_object112398:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48304ad9:::benchmark_data_smithi194_196924_object124242:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48306a67:::benchmark_data_smithi194_196924_object100464:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48326cf5:::benchmark_data_smithi194_196924_object152170:head# 2024-04-29T15:00:11.233 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48345456:::benchmark_data_smithi194_196924_object121268:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4834ccf4:::benchmark_data_smithi194_196924_object121417:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:483564e4:::benchmark_data_smithi194_196924_object152861:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4835ffed:::benchmark_data_smithi194_196924_object109679:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:483bb230:::benchmark_data_smithi194_196924_object151888:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:483c9736:::benchmark_data_smithi194_196924_object133038:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:483d1b03:::benchmark_data_smithi194_196924_object143322:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:483e1ce0:::benchmark_data_smithi194_196924_object145929:head# 2024-04-29T15:00:11.234 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:483e99b9:::benchmark_data_smithi194_196924_object129913:head# 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484170f5:::benchmark_data_smithi194_196924_object108135:head# 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4843211a:::benchmark_data_smithi194_196924_object96966:head# 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484367e9:::benchmark_data_smithi194_196924_object134222:head# 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48443935:::benchmark_data_smithi194_196924_object103826:head# 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4844839c:::benchmark_data_smithi194_196924_object150727:head# 2024-04-29T15:00:11.235 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4847cf81:::benchmark_data_smithi194_196924_object116677:head# 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4849f427:::benchmark_data_smithi194_196924_object109929:head# 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484a2302:::benchmark_data_smithi194_196924_object105776:head# 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484a2dfa:::benchmark_data_smithi194_196924_object109753:head# 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484ae8d5:::benchmark_data_smithi194_196924_object151384:head# 2024-04-29T15:00:11.248 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484c6353:::benchmark_data_smithi194_196924_object118524:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484d53c4:::benchmark_data_smithi194_196924_object136009:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484dd3e3:::benchmark_data_smithi194_196924_object137707:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484ea02a:::benchmark_data_smithi194_196924_object140395:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:484f181d:::benchmark_data_smithi194_196924_object111739:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485013a3:::benchmark_data_smithi194_196924_object159919:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4851f03b:::benchmark_data_smithi194_196924_object138914:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4851fb00:::benchmark_data_smithi194_196924_object158355:head# 2024-04-29T15:00:11.249 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48522c85:::benchmark_data_smithi194_196924_object95618:head# 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48558100:::benchmark_data_smithi194_196924_object138707:head# 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4855bb13:::benchmark_data_smithi194_196924_object110270:head# 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48566ede:::benchmark_data_smithi194_196924_object112536:head# 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48566f2e:::benchmark_data_smithi194_196924_object136611:head# 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485671d9:::benchmark_data_smithi194_196924_object100670:head# 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4856a450:::benchmark_data_smithi194_196924_object105303:head# 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.250 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4856d440:::benchmark_data_smithi194_196924_object102019:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4858904e:::benchmark_data_smithi194_196924_object119092:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485893ea:::benchmark_data_smithi194_196924_object143462:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4859d92f:::benchmark_data_smithi194_196924_object148713:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4859e13d:::benchmark_data_smithi194_196924_object125833:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485b4193:::benchmark_data_smithi194_196924_object114078:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485c4e75:::benchmark_data_smithi194_196924_object128907:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485ca464:::benchmark_data_smithi194_196924_object141514:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485d8af6:::benchmark_data_smithi194_196924_object138799:head# 2024-04-29T15:00:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485dab10:::benchmark_data_smithi194_196924_object136619:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:485e8537:::benchmark_data_smithi194_196924_object138397:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48601e21:::benchmark_data_smithi194_196924_object136020:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4860710d:::benchmark_data_smithi194_196924_object155608:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48617253:::benchmark_data_smithi194_196924_object140347:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48620033:::benchmark_data_smithi194_196924_object97631:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48639a35:::benchmark_data_smithi194_196924_object112714:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4863b42d:::benchmark_data_smithi194_196924_object115706:head# 2024-04-29T15:00:11.252 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4863cd3c:::benchmark_data_smithi194_196924_object104364:head# 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4863ef6b:::benchmark_data_smithi194_196924_object101033:head# 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4866b55b:::benchmark_data_smithi194_196924_object102809:head# 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4867629b:::benchmark_data_smithi194_196924_object124238:head# 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:486a3356:::benchmark_data_smithi194_196924_object131982:head# 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:486c3f3d:::benchmark_data_smithi194_196924_object120167:head# 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:486ca037:::benchmark_data_smithi194_196924_object111881:head# 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.253 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:486db1f9:::benchmark_data_smithi194_196924_object127201:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4871437a:::benchmark_data_smithi194_196924_object124210:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48730f6d:::benchmark_data_smithi194_196924_object145293:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4873ddc8:::benchmark_data_smithi194_196924_object154161:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4875064a:::benchmark_data_smithi194_196924_object104905:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:487704a0:::benchmark_data_smithi194_196924_object123856:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:487a477a:::benchmark_data_smithi194_196924_object109080:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:487a829e:::benchmark_data_smithi194_196924_object143151:head# 2024-04-29T15:00:11.254 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:487cff82:::benchmark_data_smithi194_196924_object121952:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:487def7e:::benchmark_data_smithi194_196924_object138236:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:487f0bfe:::benchmark_data_smithi194_196924_object126573:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48809d6d:::benchmark_data_smithi194_196924_object147636:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48819cc1:::benchmark_data_smithi194_196924_object152577:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488214b9:::benchmark_data_smithi194_196924_object107982:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48833904:::benchmark_data_smithi194_196924_object121250:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48843cee:::benchmark_data_smithi194_196924_object133160:head# 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48850cc7:::benchmark_data_smithi194_196924_object140082:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48852ea9:::benchmark_data_smithi194_196924_object146628:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48858cc2:::benchmark_data_smithi194_196924_object159591:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48881a8e:::benchmark_data_smithi194_196924_object114771:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488a38d7:::benchmark_data_smithi194_196924_object153199:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488b4fb9:::benchmark_data_smithi194_196924_object102697:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488b901c:::benchmark_data_smithi194_196924_object106902:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488cc0b0:::benchmark_data_smithi194_196924_object141052:head# 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488d9ece:::benchmark_data_smithi194_196924_object159479:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488ea557:::benchmark_data_smithi194_196924_object121694:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488eb328:::benchmark_data_smithi194_196924_object135796:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488ed7ac:::benchmark_data_smithi194_196924_object129008:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:488ef04f:::benchmark_data_smithi194_196924_object138510:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48902afc:::benchmark_data_smithi194_196924_object126254:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4892395e:::benchmark_data_smithi194_196924_object130952:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4893b64f:::benchmark_data_smithi194_196924_object104537:head# 2024-04-29T15:00:11.257 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48950ef1:::benchmark_data_smithi194_196924_object104795:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:489801cf:::benchmark_data_smithi194_196924_object104065:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4898e03c:::benchmark_data_smithi194_196924_object121641:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:489a0f2c:::benchmark_data_smithi194_196924_object133140:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:489a3c2b:::benchmark_data_smithi194_196924_object157482:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:489a63bc:::benchmark_data_smithi194_196924_object112028:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:489aa784:::benchmark_data_smithi194_196924_object157886:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:489da0b2:::benchmark_data_smithi194_196924_object96791:head# 2024-04-29T15:00:11.258 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:489f7e35:::benchmark_data_smithi194_196924_object135120:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48a0521b:::benchmark_data_smithi194_196924_object110936:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48a07ac2:::benchmark_data_smithi194_196924_object118169:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48a53691:::benchmark_data_smithi194_196924_object158333:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48a69569:::benchmark_data_smithi194_196924_object115632:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48a7c9a1:::benchmark_data_smithi194_196924_object96134:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48a8a873:::benchmark_data_smithi194_196924_object139185:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48a8d7c7:::benchmark_data_smithi194_196924_object103953:head# 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.259 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ab726a:::benchmark_data_smithi194_196924_object150188:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ac3dac:::benchmark_data_smithi194_196924_object106674:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48adb247:::benchmark_data_smithi194_196924_object136694:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48aee2a0:::benchmark_data_smithi194_196924_object154249:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48af1773:::benchmark_data_smithi194_196924_object104177:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b16245:::benchmark_data_smithi194_196924_object136785:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b170e4:::benchmark_data_smithi194_196924_object136552:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b1f1bb:::benchmark_data_smithi194_196924_object114694:head# 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b2434e:::benchmark_data_smithi194_196924_object103097:head# 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b2873a:::benchmark_data_smithi194_196924_object136393:head# 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b385e4:::benchmark_data_smithi194_196924_object153102:head# 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b543d9:::benchmark_data_smithi194_196924_object147805:head# 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b58f13:::benchmark_data_smithi194_196924_object108716:head# 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b6d7fe:::benchmark_data_smithi194_196924_object101160:head# 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b6fdcb:::benchmark_data_smithi194_196924_object123437:head# 2024-04-29T15:00:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b8703d:::benchmark_data_smithi194_196924_object156754:head# 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b8c475:::benchmark_data_smithi194_196924_object140155:head# 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48b97193:::benchmark_data_smithi194_196924_object97483:head# 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ba3331:::benchmark_data_smithi194_196924_object150639:head# 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48bc034c:::benchmark_data_smithi194_196924_object121488:head# 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48bd72f9:::benchmark_data_smithi194_196924_object114386:head# 2024-04-29T15:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48bdf82c:::benchmark_data_smithi194_196924_object122083:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48bfed6c:::benchmark_data_smithi194_196924_object128732:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48c07067:::benchmark_data_smithi194_196924_object116508:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48c10175:::benchmark_data_smithi194_196924_object125384:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48c1bef5:::benchmark_data_smithi194_196924_object147537:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48c1d3fa:::benchmark_data_smithi194_196924_object152914:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48c4abed:::benchmark_data_smithi194_196924_object157456:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48c57d21:::benchmark_data_smithi194_196924_object115725:head# 2024-04-29T15:00:11.271 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ca6977:::benchmark_data_smithi194_196924_object105927:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48cb5168:::benchmark_data_smithi194_196924_object101492:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48cb7384:::benchmark_data_smithi194_196924_object122661:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48cbab6a:::benchmark_data_smithi194_196924_object121198:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48cc77a0:::benchmark_data_smithi194_196924_object123913:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48cdf85e:::benchmark_data_smithi194_196924_object134578:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ce9958:::benchmark_data_smithi194_196924_object130153:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d156ad:::benchmark_data_smithi194_196924_object112467:head# 2024-04-29T15:00:11.272 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d16638:::benchmark_data_smithi194_196924_object148292:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d1d5b0:::benchmark_data_smithi194_196924_object149509:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d2c1a3:::benchmark_data_smithi194_196924_object126566:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d3018a:::benchmark_data_smithi194_196924_object96275:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d44df5:::benchmark_data_smithi194_196924_object141457:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d58700:::benchmark_data_smithi194_196924_object153106:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d670b5:::benchmark_data_smithi194_196924_object148641:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d77eba:::benchmark_data_smithi194_196924_object131720:head# 2024-04-29T15:00:11.273 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d7d1ff:::benchmark_data_smithi194_196924_object129057:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d95561:::benchmark_data_smithi194_196924_object153197:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48d95d76:::benchmark_data_smithi194_196924_object156540:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48da8fc6:::benchmark_data_smithi194_196924_object125632:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48daf27a:::benchmark_data_smithi194_196924_object124540:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48dc95e8:::benchmark_data_smithi194_196924_object117195:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48dccf61:::benchmark_data_smithi194_196924_object141212:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48def1b8:::benchmark_data_smithi194_196924_object102928:head# 2024-04-29T15:00:11.274 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e16506:::benchmark_data_smithi194_196924_object123986:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e1b0d3:::benchmark_data_smithi194_196924_object151137:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e20876:::benchmark_data_smithi194_196924_object105126:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e2fb4a:::benchmark_data_smithi194_196924_object140546:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e394dc:::benchmark_data_smithi194_196924_object147830:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e5ec65:::benchmark_data_smithi194_196924_object110421:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e7a5cf:::benchmark_data_smithi194_196924_object104660:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e845cb:::benchmark_data_smithi194_196924_object100146:head# 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.275 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e854b2:::benchmark_data_smithi194_196924_object151303:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e8cce4:::benchmark_data_smithi194_196924_object133833:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48e938f6:::benchmark_data_smithi194_196924_object120409:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ec0313:::benchmark_data_smithi194_196924_object95741:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ec9bb8:::benchmark_data_smithi194_196924_object126978:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ed74e8:::benchmark_data_smithi194_196924_object134959:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48edbe1c:::benchmark_data_smithi194_196924_object106566:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ee638d:::benchmark_data_smithi194_196924_object112314:head# 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.276 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48eeb91c:::benchmark_data_smithi194_196924_object151673:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ef5521:::benchmark_data_smithi194_196924_object116595:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ef9025:::benchmark_data_smithi194_196924_object116144:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48f2d645:::benchmark_data_smithi194_196924_object128748:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48f36666:::benchmark_data_smithi194_196924_object131814:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48f984c6:::benchmark_data_smithi194_196924_object135904:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48f9fbff:::benchmark_data_smithi194_196924_object123269:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fb95db:::benchmark_data_smithi194_196924_object157863:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fbe1b7:::benchmark_data_smithi194_196924_object128481:head# 2024-04-29T15:00:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fd3ab1:::benchmark_data_smithi194_196924_object129988:head# 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fd6396:::benchmark_data_smithi194_196924_object138435:head# 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fdff61:::benchmark_data_smithi194_196924_object124043:head# 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fe4d0e:::benchmark_data_smithi194_196924_object158792:head# 2024-04-29T15:00:11.278 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.285 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fe52a5:::benchmark_data_smithi194_196924_object114310:head# 2024-04-29T15:00:11.285 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.285 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48fe96e9:::benchmark_data_smithi194_196924_object150610:head# 2024-04-29T15:00:11.285 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.285 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ff0024:::benchmark_data_smithi194_196924_object132708:head# 2024-04-29T15:00:11.285 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.285 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:48ffafb4:::benchmark_data_smithi194_196924_object153462:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4900b7f2:::benchmark_data_smithi194_196924_object116209:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4900d970:::benchmark_data_smithi194_196924_object112197:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49032da3:::benchmark_data_smithi194_196924_object111022:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:490387ee:::benchmark_data_smithi194_196924_object148964:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49040e6e:::benchmark_data_smithi194_196924_object153422:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4904b78d:::benchmark_data_smithi194_196924_object159304:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49051964:::benchmark_data_smithi194_196924_object102849:head# 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.286 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49052826:::benchmark_data_smithi194_196924_object120415:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49057ddf:::benchmark_data_smithi194_196924_object140862:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49067222:::benchmark_data_smithi194_196924_object144896:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4907323b:::benchmark_data_smithi194_196924_object122034:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4907e0c4:::benchmark_data_smithi194_196924_object96253:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:490a0871:::benchmark_data_smithi194_196924_object106367:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:490a992a:::benchmark_data_smithi194_196924_object158417:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:490c8372:::benchmark_data_smithi194_196924_object105860:head# 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.287 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:490d659c:::benchmark_data_smithi194_196924_object105626:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:490fcae1:::benchmark_data_smithi194_196924_object106376:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4910e240:::benchmark_data_smithi194_196924_object140826:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4910e33a:::benchmark_data_smithi194_196924_object100278:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49138a5a:::benchmark_data_smithi194_196924_object155760:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49158abb:::benchmark_data_smithi194_196924_object112065:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49161cad:::benchmark_data_smithi194_196924_object155782:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4916b195:::benchmark_data_smithi194_196924_object126983:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:491755f8:::benchmark_data_smithi194_196924_object131587:head# 2024-04-29T15:00:11.288 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:491ab4b9:::benchmark_data_smithi194_196924_object137127:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:491b06e8:::benchmark_data_smithi194_196924_object151116:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:491b194e:::benchmark_data_smithi194_196924_object109269:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:491b5388:::benchmark_data_smithi194_196924_object129278:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:491efe68:::benchmark_data_smithi194_196924_object132594:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4921dbbc:::benchmark_data_smithi194_196924_object104457:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:492334e6:::benchmark_data_smithi194_196924_object131844:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49247e90:::benchmark_data_smithi194_196924_object108111:head# 2024-04-29T15:00:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49258450:::benchmark_data_smithi194_196924_object102895:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49263aa5:::benchmark_data_smithi194_196924_object156663:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4926badc:::benchmark_data_smithi194_196924_object134000:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49271a37:::benchmark_data_smithi194_196924_object109495:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4929e321:::benchmark_data_smithi194_196924_object143606:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:492ae1f7:::benchmark_data_smithi194_196924_object149041:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:492b2032:::benchmark_data_smithi194_196924_object152666:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:492dd943:::benchmark_data_smithi194_196924_object145012:head# 2024-04-29T15:00:11.290 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:492e9e6b:::benchmark_data_smithi194_196924_object141934:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4930388d:::benchmark_data_smithi194_196924_object114922:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49312fcb:::benchmark_data_smithi194_196924_object105070:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49314fd2:::benchmark_data_smithi194_196924_object155147:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4933205d:::benchmark_data_smithi194_196924_object103405:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4933358e:::benchmark_data_smithi194_196924_object140637:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4936282d:::benchmark_data_smithi194_196924_object152568:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4939d5b4:::benchmark_data_smithi194_196924_object153763:head# 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.291 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493a6674:::benchmark_data_smithi194_196924_object156836:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493ab546:::benchmark_data_smithi194_196924_object136640:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493c024d:::benchmark_data_smithi194_196924_object147546:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493c6b96:::benchmark_data_smithi194_196924_object144360:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493ce39a:::benchmark_data_smithi194_196924_object133897:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493d6757:::benchmark_data_smithi194_196924_object102286:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493dcf86:::benchmark_data_smithi194_196924_object151579:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:493e5caa:::benchmark_data_smithi194_196924_object124710:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49401cd3:::benchmark_data_smithi194_196924_object106786:head# 2024-04-29T15:00:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.293 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4941270a:::benchmark_data_smithi194_196924_object135014:head# 2024-04-29T15:00:11.293 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:494181e9:::benchmark_data_smithi194_196924_object128014:head# 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49422b7b:::benchmark_data_smithi194_196924_object108927:head# 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4942fd55:::benchmark_data_smithi194_196924_object135645:head# 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49434bd8:::benchmark_data_smithi194_196924_object149033:head# 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4944f6fc:::benchmark_data_smithi194_196924_object136007:head# 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49453d62:::benchmark_data_smithi194_196924_object148800:head# 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4945ea35:::benchmark_data_smithi194_196924_object151884:head# 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4946a376:::benchmark_data_smithi194_196924_object128883:head# 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4947cf1c:::benchmark_data_smithi194_196924_object138989:head# 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49483fc0:::benchmark_data_smithi194_196924_object120680:head# 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:494c6792:::benchmark_data_smithi194_196924_object103166:head# 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.300 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:494cb512:::benchmark_data_smithi194_196924_object148846:head# 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:494d9797:::benchmark_data_smithi194_196924_object122086:head# 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:494dcbb0:::benchmark_data_smithi194_196924_object156628:head# 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:494e7321:::benchmark_data_smithi194_196924_object100268:head# 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:494ed20b:::benchmark_data_smithi194_196924_object102634:head# 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49509e6f:::benchmark_data_smithi194_196924_object133346:head# 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4950ab7f:::benchmark_data_smithi194_196924_object111099:head# 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.301 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4950c00a:::benchmark_data_smithi194_196924_object122209:head# 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49510448:::benchmark_data_smithi194_196924_object145167:head# 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49513485:::benchmark_data_smithi194_196924_object156684:head# 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:495299ec:::benchmark_data_smithi194_196924_object132649:head# 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4953e009:::benchmark_data_smithi194_196924_object152614:head# 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4955aad2:::benchmark_data_smithi194_196924_object97389:head# 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4955db51:::benchmark_data_smithi194_196924_object142803:head# 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.302 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4956f680:::benchmark_data_smithi194_196924_object133966:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:495779ac:::benchmark_data_smithi194_196924_object106328:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:495acd06:::benchmark_data_smithi194_196924_object127970:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:495c00d1:::benchmark_data_smithi194_196924_object121141:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:495fe8c0:::benchmark_data_smithi194_196924_object151995:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:496234c9:::benchmark_data_smithi194_196924_object152722:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49652365:::benchmark_data_smithi194_196924_object157012:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49661954:::benchmark_data_smithi194_196924_object122103:head# 2024-04-29T15:00:11.303 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49665dbc:::benchmark_data_smithi194_196924_object134395:head# 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4968dac8:::benchmark_data_smithi194_196924_object139932:head# 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4968dba7:::benchmark_data_smithi194_196924_object97970:head# 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49694e5d:::benchmark_data_smithi194_196924_object109426:head# 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49697f84:::benchmark_data_smithi194_196924_object121932:head# 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4969e932:::benchmark_data_smithi194_196924_object140806:head# 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:496a3291:::benchmark_data_smithi194_196924_object127896:head# 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:496b0c99:::benchmark_data_smithi194_196924_object114376:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:496d0ca9:::benchmark_data_smithi194_196924_object114517:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:496d1ed5:::benchmark_data_smithi194_196924_object131256:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:496e4f58:::benchmark_data_smithi194_196924_object138292:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:496f5c93:::benchmark_data_smithi194_196924_object152702:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49739a19:::benchmark_data_smithi194_196924_object149604:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4973fda7:::benchmark_data_smithi194_196924_object96764:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:497416c7:::benchmark_data_smithi194_196924_object113023:head# 2024-04-29T15:00:11.305 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.306 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49746831:::benchmark_data_smithi194_196924_object111384:head# 2024-04-29T15:00:11.306 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.306 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49749e34:::benchmark_data_smithi194_196924_object113094:head# 2024-04-29T15:00:11.306 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.306 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4974da10:::benchmark_data_smithi194_196924_object157006:head# 2024-04-29T15:00:11.306 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4974fb13:::benchmark_data_smithi194_196924_object134746:head# 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4976ef16:::benchmark_data_smithi194_196924_object99163:head# 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4978fec8:::benchmark_data_smithi194_196924_object131058:head# 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:497ea2d4:::benchmark_data_smithi194_196924_object136424:head# 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:497ed9a4:::benchmark_data_smithi194_196924_object123090:head# 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498185e6:::benchmark_data_smithi194_196924_object96590:head# 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.312 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498270b3:::benchmark_data_smithi194_196924_object123742:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4983d037:::benchmark_data_smithi194_196924_object148456:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4984e19b:::benchmark_data_smithi194_196924_object104569:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4986e0a8:::benchmark_data_smithi194_196924_object97157:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4988b71e:::benchmark_data_smithi194_196924_object139113:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49891123:::benchmark_data_smithi194_196924_object125068:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4989133c:::benchmark_data_smithi194_196924_object109890:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49899b11:::benchmark_data_smithi194_196924_object152293:head# 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.313 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4989baed:::benchmark_data_smithi194_196924_object101268:head# 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498bff9b:::benchmark_data_smithi194_196924_object135232:head# 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498c94ad:::benchmark_data_smithi194_196924_object140353:head# 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498d2936:::benchmark_data_smithi194_196924_object103385:head# 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498d51c4:::benchmark_data_smithi194_196924_object100185:head# 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498dd66b:::benchmark_data_smithi194_196924_object100879:head# 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:498f189f:::benchmark_data_smithi194_196924_object138929:head# 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.314 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4990d614:::benchmark_data_smithi194_196924_object114178:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4990db49:::benchmark_data_smithi194_196924_object104115:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49923700:::benchmark_data_smithi194_196924_object110095:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499274f9:::benchmark_data_smithi194_196924_object131096:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49939454:::benchmark_data_smithi194_196924_object127122:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4993f09b:::benchmark_data_smithi194_196924_object146535:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4994264c:::benchmark_data_smithi194_196924_object115017:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49964fab:::benchmark_data_smithi194_196924_object156468:head# 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499674ba:::benchmark_data_smithi194_196924_object122052:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4998b9d1:::benchmark_data_smithi194_196924_object140951:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4998bd08:::benchmark_data_smithi194_196924_object122869:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499a8805:::benchmark_data_smithi194_196924_object101992:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499b9f13:::benchmark_data_smithi194_196924_object128335:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499d398d:::benchmark_data_smithi194_196924_object110947:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499d50f4:::benchmark_data_smithi194_196924_object125889:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499e7fa2:::benchmark_data_smithi194_196924_object148644:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:499fe4ce:::benchmark_data_smithi194_196924_object118240:head# 2024-04-29T15:00:11.316 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49a06bc5:::benchmark_data_smithi194_196924_object124909:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49a332c8:::benchmark_data_smithi194_196924_object121125:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49a44a95:::benchmark_data_smithi194_196924_object134736:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49a5e80e:::benchmark_data_smithi194_196924_object109663:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49a61da9:::benchmark_data_smithi194_196924_object108099:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49a89238:::benchmark_data_smithi194_196924_object98794:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49aa266e:::benchmark_data_smithi194_196924_object159436:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49aa8b32:::benchmark_data_smithi194_196924_object103850:head# 2024-04-29T15:00:11.317 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ab9aaa:::benchmark_data_smithi194_196924_object98306:head# 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ac66ba:::benchmark_data_smithi194_196924_object141089:head# 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ac8d27:::benchmark_data_smithi194_196924_object138514:head# 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ad632f:::benchmark_data_smithi194_196924_object122113:head# 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ada826:::benchmark_data_smithi194_196924_object151391:head# 2024-04-29T15:00:11.318 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.323 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49afe27b:::benchmark_data_smithi194_196924_object154595:head# 2024-04-29T15:00:11.323 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.323 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b1f3bb:::benchmark_data_smithi194_196924_object97819:head# 2024-04-29T15:00:11.323 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.323 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b3c380:::benchmark_data_smithi194_196924_object146196:head# 2024-04-29T15:00:11.323 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.323 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b442d8:::benchmark_data_smithi194_196924_object142862:head# 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b45e14:::benchmark_data_smithi194_196924_object156209:head# 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b4b956:::benchmark_data_smithi194_196924_object132985:head# 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b69a5f:::benchmark_data_smithi194_196924_object123776:head# 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b6e4e4:::benchmark_data_smithi194_196924_object145843:head# 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b74674:::benchmark_data_smithi194_196924_object113077:head# 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b76645:::benchmark_data_smithi194_196924_object121722:head# 2024-04-29T15:00:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b7bbac:::benchmark_data_smithi194_196924_object95645:head# 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b859e0:::benchmark_data_smithi194_196924_object145251:head# 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49b8f2a1:::benchmark_data_smithi194_196924_object96982:head# 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ba0372:::benchmark_data_smithi194_196924_object151325:head# 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bb4fa0:::benchmark_data_smithi194_196924_object146003:head# 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bbe0c0:::benchmark_data_smithi194_196924_object149027:head# 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bd042c:::benchmark_data_smithi194_196924_object127066:head# 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.325 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bd1619:::benchmark_data_smithi194_196924_object105823:head# 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bd1dcb:::benchmark_data_smithi194_196924_object128427:head# 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49be0c34:::benchmark_data_smithi194_196924_object135187:head# 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bf4595:::benchmark_data_smithi194_196924_object120493:head# 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bf8fed:::benchmark_data_smithi194_196924_object150072:head# 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49bfca29:::benchmark_data_smithi194_196924_object102931:head# 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49c29316:::benchmark_data_smithi194_196924_object106834:head# 2024-04-29T15:00:11.326 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49c36cfe:::benchmark_data_smithi194_196924_object154981:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49c38038:::benchmark_data_smithi194_196924_object157136:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49c44f13:::benchmark_data_smithi194_196924_object132034:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49c90ee9:::benchmark_data_smithi194_196924_object108520:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ca1203:::benchmark_data_smithi194_196924_object114683:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ca28ef:::benchmark_data_smithi194_196924_object114001:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ca67bb:::benchmark_data_smithi194_196924_object112635:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ca9deb:::benchmark_data_smithi194_196924_object126063:head# 2024-04-29T15:00:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49cb7d3f:::benchmark_data_smithi194_196924_object132121:head# 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ccf0fe:::benchmark_data_smithi194_196924_object119932:head# 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49cf5ecc:::benchmark_data_smithi194_196924_object104074:head# 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49cfc46c:::benchmark_data_smithi194_196924_object151742:head# 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49d06520:::benchmark_data_smithi194_196924_object103081:head# 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49d0668d:::benchmark_data_smithi194_196924_object135876:head# 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49d195da:::benchmark_data_smithi194_196924_object145769:head# 2024-04-29T15:00:11.328 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49d44ce4:::benchmark_data_smithi194_196924_object141139:head# 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49d97942:::benchmark_data_smithi194_196924_object151595:head# 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49da337b:::benchmark_data_smithi194_196924_object129407:head# 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49db0a09:::benchmark_data_smithi194_196924_object101851:head# 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49db2693:::benchmark_data_smithi194_196924_object110141:head# 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49dd3501:::benchmark_data_smithi194_196924_object138423:head# 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49df40cf:::benchmark_data_smithi194_196924_object140285:head# 2024-04-29T15:00:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e0695d:::benchmark_data_smithi194_196924_object145331:head# 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e15d1e:::benchmark_data_smithi194_196924_object157717:head# 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e1c1df:::benchmark_data_smithi194_196924_object122479:head# 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e31ecd:::benchmark_data_smithi194_196924_object141495:head# 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e3b489:::benchmark_data_smithi194_196924_object136072:head# 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.334 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e4b124:::benchmark_data_smithi194_196924_object156987:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e5d4f3:::benchmark_data_smithi194_196924_object150027:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e5dca3:::benchmark_data_smithi194_196924_object131684:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49e6369c:::benchmark_data_smithi194_196924_object112378:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ebeb1c:::benchmark_data_smithi194_196924_object104480:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ed5f04:::benchmark_data_smithi194_196924_object147842:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49eff11b:::benchmark_data_smithi194_196924_object126886:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f051a2:::benchmark_data_smithi194_196924_object122042:head# 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.335 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f0bda4:::benchmark_data_smithi194_196924_object156040:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f21b73:::benchmark_data_smithi194_196924_object146673:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f2ba7e:::benchmark_data_smithi194_196924_object154371:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f4dabf:::benchmark_data_smithi194_196924_object115712:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f769c1:::benchmark_data_smithi194_196924_object151478:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f8398a:::benchmark_data_smithi194_196924_object133439:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49f8e3b7:::benchmark_data_smithi194_196924_object157871:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49fa0cfd:::benchmark_data_smithi194_196924_object122862:head# 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.336 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49fb0e7b:::benchmark_data_smithi194_196924_object150858:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49feddb0:::benchmark_data_smithi194_196924_object104828:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:49ffe727:::benchmark_data_smithi194_196924_object151455:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0178aa:::benchmark_data_smithi194_196924_object96711:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a020566:::benchmark_data_smithi194_196924_object102596:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a024747:::benchmark_data_smithi194_196924_object104039:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0278a0:::benchmark_data_smithi194_196924_object116496:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a02882d:::benchmark_data_smithi194_196924_object152107:head# 2024-04-29T15:00:11.337 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a050a95:::benchmark_data_smithi194_196924_object140783:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a06c351:::benchmark_data_smithi194_196924_object106522:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a075a26:::benchmark_data_smithi194_196924_object136252:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a088fcb:::benchmark_data_smithi194_196924_object156448:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a090325:::benchmark_data_smithi194_196924_object133675:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a09aee6:::benchmark_data_smithi194_196924_object144590:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0a6407:::benchmark_data_smithi194_196924_object159773:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0b19c4:::benchmark_data_smithi194_196924_object122173:head# 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0b3c1a:::benchmark_data_smithi194_196924_object142744:head# 2024-04-29T15:00:11.339 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.339 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:11.344 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0b4c4e:::benchmark_data_smithi194_196924_object96121:head# 2024-04-29T15:00:11.344 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.344 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0b5cda:::benchmark_data_smithi194_196924_object135182:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0c3c0a:::benchmark_data_smithi194_196924_object155063:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0c5886:::benchmark_data_smithi194_196924_object111098:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0d13df:::benchmark_data_smithi194_196924_object119150:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0d39b0:::benchmark_data_smithi194_196924_object145683:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0eff1a:::benchmark_data_smithi194_196924_object121353:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0f1c11:::benchmark_data_smithi194_196924_object133596:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0f49ce:::benchmark_data_smithi194_196924_object130816:head# 2024-04-29T15:00:11.345 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0f5aaf:::benchmark_data_smithi194_196924_object157426:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a0f69d8:::benchmark_data_smithi194_196924_object146120:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a102a24:::benchmark_data_smithi194_196924_object105260:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a10be3d:::benchmark_data_smithi194_196924_object117054:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a125b06:::benchmark_data_smithi194_196924_object103593:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a130084:::benchmark_data_smithi194_196924_object138322:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a130415:::benchmark_data_smithi194_196924_object133673:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1424d9:::benchmark_data_smithi194_196924_object146971:head# 2024-04-29T15:00:11.346 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1457da:::benchmark_data_smithi194_196924_object141915:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a164067:::benchmark_data_smithi194_196924_object141490:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a16940b:::benchmark_data_smithi194_196924_object153443:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a19bf9d:::benchmark_data_smithi194_196924_object113356:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1a851b:::benchmark_data_smithi194_196924_object102600:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1aa372:::benchmark_data_smithi194_196924_object134723:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1acec9:::benchmark_data_smithi194_196924_object133888:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1b8a22:::benchmark_data_smithi194_196924_object117123:head# 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.347 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1bbab7:::benchmark_data_smithi194_196924_object115727:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1e57cd:::benchmark_data_smithi194_196924_object158929:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1e8704:::benchmark_data_smithi194_196924_object126362:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1e9f6b:::benchmark_data_smithi194_196924_object123299:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a1f31da:::benchmark_data_smithi194_196924_object141770:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a207c61:::benchmark_data_smithi194_196924_object152675:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a20eb15:::benchmark_data_smithi194_196924_object138186:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a2203f3:::benchmark_data_smithi194_196924_object123055:head# 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.348 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a220938:::benchmark_data_smithi194_196924_object115536:head# 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a2365ed:::benchmark_data_smithi194_196924_object99564:head# 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a23c807:::benchmark_data_smithi194_196924_object108830:head# 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a23f982:::benchmark_data_smithi194_196924_object124291:head# 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a25b8f3:::benchmark_data_smithi194_196924_object130178:head# 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a260e7d:::benchmark_data_smithi194_196924_object102058:head# 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a28450d:::benchmark_data_smithi194_196924_object145078:head# 2024-04-29T15:00:11.349 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a2bbe79:::benchmark_data_smithi194_196924_object104066:head# 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a2ce73c:::benchmark_data_smithi194_196924_object149449:head# 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a2f1a08:::benchmark_data_smithi194_196924_object128580:head# 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a30aeb3:::benchmark_data_smithi194_196924_object103116:head# 2024-04-29T15:00:11.352 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a310987:::benchmark_data_smithi194_196924_object97985:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3201a7:::benchmark_data_smithi194_196924_object118717:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a338abd:::benchmark_data_smithi194_196924_object137797:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3431f5:::benchmark_data_smithi194_196924_object133777:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a35eb38:::benchmark_data_smithi194_196924_object109951:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a376ecd:::benchmark_data_smithi194_196924_object98767:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a37e94e:::benchmark_data_smithi194_196924_object114945:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a38b619:::benchmark_data_smithi194_196924_object105113:head# 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.353 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a39ef51:::benchmark_data_smithi194_196924_object120358:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3a580a:::benchmark_data_smithi194_196924_object99739:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3b179e:::benchmark_data_smithi194_196924_object115103:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3d3b4f:::benchmark_data_smithi194_196924_object102635:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3d5a24:::benchmark_data_smithi194_196924_object146425:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3eb348:::benchmark_data_smithi194_196924_object124938:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a3f5dc7:::benchmark_data_smithi194_196924_object140389:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a41dc42:::benchmark_data_smithi194_196924_object134219:head# 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.354 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a42086e:::benchmark_data_smithi194_196924_object99463:head# 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a420cdf:::benchmark_data_smithi194_196924_object136418:head# 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a42d36d:::benchmark_data_smithi194_196924_object148149:head# 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a435ae6:::benchmark_data_smithi194_196924_object135484:head# 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a450570:::benchmark_data_smithi194_196924_object114234:head# 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a46688c:::benchmark_data_smithi194_196924_object138574:head# 2024-04-29T15:00:11.355 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4671f4:::benchmark_data_smithi194_196924_object110556:head# 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a46adf8:::benchmark_data_smithi194_196924_object110059:head# 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a48f3a1:::benchmark_data_smithi194_196924_object116691:head# 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4a17f9:::benchmark_data_smithi194_196924_object120914:head# 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4a624c:::benchmark_data_smithi194_196924_object125199:head# 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4b1643:::benchmark_data_smithi194_196924_object113465:head# 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4bf4e3:::benchmark_data_smithi194_196924_object146116:head# 2024-04-29T15:00:11.360 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4c4f5d:::benchmark_data_smithi194_196924_object152707:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4cb9e1:::benchmark_data_smithi194_196924_object148671:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4e7e34:::benchmark_data_smithi194_196924_object132978:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a4e84f6:::benchmark_data_smithi194_196924_object97121:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a506cc8:::benchmark_data_smithi194_196924_object151117:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a50908c:::benchmark_data_smithi194_196924_object110819:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a50a715:::benchmark_data_smithi194_196924_object111280:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a50edf5:::benchmark_data_smithi194_196924_object113869:head# 2024-04-29T15:00:11.361 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a519d02:::benchmark_data_smithi194_196924_object103489:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a51d541:::benchmark_data_smithi194_196924_object128957:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a530037:::benchmark_data_smithi194_196924_object127253:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a560ceb:::benchmark_data_smithi194_196924_object157600:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5683cb:::benchmark_data_smithi194_196924_object96667:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a59d8cc:::benchmark_data_smithi194_196924_object152173:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5ade6b:::benchmark_data_smithi194_196924_object111228:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5b4c37:::benchmark_data_smithi194_196924_object137848:head# 2024-04-29T15:00:11.362 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5bb606:::benchmark_data_smithi194_196924_object119569:head# 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5c0eff:::benchmark_data_smithi194_196924_object123794:head# 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5cc67a:::benchmark_data_smithi194_196924_object95946:head# 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5d2c8c:::benchmark_data_smithi194_196924_object123654:head# 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5d74c1:::benchmark_data_smithi194_196924_object151307:head# 2024-04-29T15:00:11.363 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a5df602:::benchmark_data_smithi194_196924_object116427:head# 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a62356a:::benchmark_data_smithi194_196924_object153110:head# 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a632246:::benchmark_data_smithi194_196924_object158081:head# 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a64cead:::benchmark_data_smithi194_196924_object97938:head# 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a64eea9:::benchmark_data_smithi194_196924_object157474:head# 2024-04-29T15:00:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a658749:::benchmark_data_smithi194_196924_object149880:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a676f35:::benchmark_data_smithi194_196924_object103296:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a677d1b:::benchmark_data_smithi194_196924_object120166:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a67d5fe:::benchmark_data_smithi194_196924_object112788:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a67f7d4:::benchmark_data_smithi194_196924_object107290:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a699e8e:::benchmark_data_smithi194_196924_object149613:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a6ba52e:::benchmark_data_smithi194_196924_object99434:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a6bec77:::benchmark_data_smithi194_196924_object135119:head# 2024-04-29T15:00:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a6ef6f5:::benchmark_data_smithi194_196924_object115763:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a6f37e8:::benchmark_data_smithi194_196924_object139930:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a6f45ad:::benchmark_data_smithi194_196924_object155380:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a6fd5ba:::benchmark_data_smithi194_196924_object120643:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a709d15:::benchmark_data_smithi194_196924_object116575:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a735129:::benchmark_data_smithi194_196924_object108735:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a754117:::benchmark_data_smithi194_196924_object136653:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a75987f:::benchmark_data_smithi194_196924_object98493:head# 2024-04-29T15:00:11.370 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a75a97a:::benchmark_data_smithi194_196924_object132956:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a762a6b:::benchmark_data_smithi194_196924_object154942:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a775adf:::benchmark_data_smithi194_196924_object101957:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a779a15:::benchmark_data_smithi194_196924_object117456:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a77eb67:::benchmark_data_smithi194_196924_object143714:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a785e25:::benchmark_data_smithi194_196924_object138120:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a7ae1e3:::benchmark_data_smithi194_196924_object139122:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a7b10c5:::benchmark_data_smithi194_196924_object112888:head# 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.371 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a7d41b1:::benchmark_data_smithi194_196924_object125766:head# 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a7dd0a0:::benchmark_data_smithi194_196924_object132418:head# 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a7ea6f5:::benchmark_data_smithi194_196924_object146661:head# 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a7f9e61:::benchmark_data_smithi194_196924_object151628:head# 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8277a9:::benchmark_data_smithi194_196924_object122815:head# 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a83232b:::benchmark_data_smithi194_196924_object120257:head# 2024-04-29T15:00:11.372 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a837b96:::benchmark_data_smithi194_196924_object96358:head# 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8386ff:::benchmark_data_smithi194_196924_object138975:head# 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a83a606:::benchmark_data_smithi194_196924_object144647:head# 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a84ee2e:::benchmark_data_smithi194_196924_object149549:head# 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a85d79b:::benchmark_data_smithi194_196924_object157928:head# 2024-04-29T15:00:11.376 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8831a5:::benchmark_data_smithi194_196924_object127260:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8834e5:::benchmark_data_smithi194_196924_object112062:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8a818b:::benchmark_data_smithi194_196924_object142756:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8d1509:::benchmark_data_smithi194_196924_object138804:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8d361d:::benchmark_data_smithi194_196924_object156948:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8e5c39:::benchmark_data_smithi194_196924_object140728:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a8e75e1:::benchmark_data_smithi194_196924_object98817:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9040aa:::benchmark_data_smithi194_196924_object113216:head# 2024-04-29T15:00:11.377 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9063c9:::benchmark_data_smithi194_196924_object108334:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a926b75:::benchmark_data_smithi194_196924_object115505:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a927349:::benchmark_data_smithi194_196924_object145684:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a92813d:::benchmark_data_smithi194_196924_object104460:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a94a66f:::benchmark_data_smithi194_196924_object139960:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a966109:::benchmark_data_smithi194_196924_object99879:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a97fd08:::benchmark_data_smithi194_196924_object122784:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a988006:::benchmark_data_smithi194_196924_object141204:head# 2024-04-29T15:00:11.378 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a989987:::benchmark_data_smithi194_196924_object121511:head# 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a990411:::benchmark_data_smithi194_196924_object114210:head# 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a99ab15:::benchmark_data_smithi194_196924_object113835:head# 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9a24a0:::benchmark_data_smithi194_196924_object132389:head# 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9b56e9:::benchmark_data_smithi194_196924_object116895:head# 2024-04-29T15:00:11.379 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9b59fd:::benchmark_data_smithi194_196924_object98716:head# 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9dc289:::benchmark_data_smithi194_196924_object155279:head# 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9e28b4:::benchmark_data_smithi194_196924_object127955:head# 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9ecfa3:::benchmark_data_smithi194_196924_object125438:head# 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4a9fab59:::benchmark_data_smithi194_196924_object112950:head# 2024-04-29T15:00:11.390 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa0c2c3:::benchmark_data_smithi194_196924_object106875:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa0daaf:::benchmark_data_smithi194_196924_object99701:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa13a07:::benchmark_data_smithi194_196924_object96477:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa1d90d:::benchmark_data_smithi194_196924_object102516:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa2390a:::benchmark_data_smithi194_196924_object108331:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa27b54:::benchmark_data_smithi194_196924_object147867:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa46a66:::benchmark_data_smithi194_196924_object145563:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa5a272:::benchmark_data_smithi194_196924_object99762:head# 2024-04-29T15:00:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa5ce06:::benchmark_data_smithi194_196924_object132669:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aa7d053:::benchmark_data_smithi194_196924_object154397:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aab8690:::benchmark_data_smithi194_196924_object141124:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aabf827:::benchmark_data_smithi194_196924_object152187:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aad4ace:::benchmark_data_smithi194_196924_object116191:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aaef6ba:::benchmark_data_smithi194_196924_object132668:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aaf148e:::benchmark_data_smithi194_196924_object119742:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aaf6a3a:::benchmark_data_smithi194_196924_object108056:head# 2024-04-29T15:00:11.392 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab0de37:::benchmark_data_smithi194_196924_object109157:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab16fca:::benchmark_data_smithi194_196924_object115490:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab2d506:::benchmark_data_smithi194_196924_object119286:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab41411:::benchmark_data_smithi194_196924_object136839:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab5a8f4:::benchmark_data_smithi194_196924_object124514:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab7b053:::benchmark_data_smithi194_196924_object121302:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab7db70:::benchmark_data_smithi194_196924_object154040:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab7de5e:::benchmark_data_smithi194_196924_object146331:head# 2024-04-29T15:00:11.393 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab7e7dd:::benchmark_data_smithi194_196924_object145266:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ab7faa4:::benchmark_data_smithi194_196924_object143122:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aba024e:::benchmark_data_smithi194_196924_object96933:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4abb16a0:::benchmark_data_smithi194_196924_object97175:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4abbd993:::benchmark_data_smithi194_196924_object108792:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4abe7eca:::benchmark_data_smithi194_196924_object154122:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ac019cc:::benchmark_data_smithi194_196924_object153727:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ac05b76:::benchmark_data_smithi194_196924_object152640:head# 2024-04-29T15:00:11.394 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ac06b7f:::benchmark_data_smithi194_196924_object104858:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ac826dd:::benchmark_data_smithi194_196924_object159321:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ac8fe52:::benchmark_data_smithi194_196924_object153225:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4accaa16:::benchmark_data_smithi194_196924_object116681:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4acdb53d:::benchmark_data_smithi194_196924_object150757:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aced5c8:::benchmark_data_smithi194_196924_object97238:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ad030f4:::benchmark_data_smithi194_196924_object110090:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ad4508e:::benchmark_data_smithi194_196924_object111222:head# 2024-04-29T15:00:11.395 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ad59305:::benchmark_data_smithi194_196924_object150407:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ad643af:::benchmark_data_smithi194_196924_object157173:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ad6d1da:::benchmark_data_smithi194_196924_object127466:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ad74bff:::benchmark_data_smithi194_196924_object133775:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ad98a76:::benchmark_data_smithi194_196924_object126652:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4adbedef:::benchmark_data_smithi194_196924_object137106:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4adc030a:::benchmark_data_smithi194_196924_object135888:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4add2a56:::benchmark_data_smithi194_196924_object154798:head# 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.396 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4add3504:::benchmark_data_smithi194_196924_object131101:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4addea79:::benchmark_data_smithi194_196924_object139541:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae13ea9:::benchmark_data_smithi194_196924_object152197:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae1dc75:::benchmark_data_smithi194_196924_object115603:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae270aa:::benchmark_data_smithi194_196924_object97874:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae41474:::benchmark_data_smithi194_196924_object116053:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae5b7e5:::benchmark_data_smithi194_196924_object124584:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae7a39b:::benchmark_data_smithi194_196924_object143827:head# 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.397 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae85547:::benchmark_data_smithi194_196924_object124610:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ae91748:::benchmark_data_smithi194_196924_object119099:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aeb257c:::benchmark_data_smithi194_196924_object100571:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aeb6387:::benchmark_data_smithi194_196924_object106028:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aec448f:::benchmark_data_smithi194_196924_object130426:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aecf7cb:::benchmark_data_smithi194_196924_object108222:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aedacdc:::benchmark_data_smithi194_196924_object97125:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aedb1de:::benchmark_data_smithi194_196924_object109247:head# 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.398 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aedf2df:::benchmark_data_smithi194_196924_object111530:head# 2024-04-29T15:00:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aee111c:::benchmark_data_smithi194_196924_object95399:head# 2024-04-29T15:00:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aef0362:::benchmark_data_smithi194_196924_object146973:head# 2024-04-29T15:00:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aef489c:::benchmark_data_smithi194_196924_object152328:head# 2024-04-29T15:00:11.399 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.413 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aef9f67:::benchmark_data_smithi194_196924_object148729:head# 2024-04-29T15:00:11.413 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.413 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af186ef:::benchmark_data_smithi194_196924_object144904:head# 2024-04-29T15:00:11.413 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af1e1d5:::benchmark_data_smithi194_196924_object152452:head# 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af2ca5f:::benchmark_data_smithi194_196924_object110444:head# 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af2f322:::benchmark_data_smithi194_196924_object132712:head# 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af436cf:::benchmark_data_smithi194_196924_object141914:head# 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af48ab6:::benchmark_data_smithi194_196924_object104573:head# 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af589c2:::benchmark_data_smithi194_196924_object117029:head# 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af5da7d:::benchmark_data_smithi194_196924_object122377:head# 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.414 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af82584:::benchmark_data_smithi194_196924_object97780:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af85f3a:::benchmark_data_smithi194_196924_object152340:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af911a1:::benchmark_data_smithi194_196924_object111416:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4af93810:::benchmark_data_smithi194_196924_object102733:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4afaef05:::benchmark_data_smithi194_196924_object119380:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4afb4416:::benchmark_data_smithi194_196924_object99761:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4afc5342:::benchmark_data_smithi194_196924_object138526:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4afd4d97:::benchmark_data_smithi194_196924_object122275:head# 2024-04-29T15:00:11.415 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4afd72c3:::benchmark_data_smithi194_196924_object105435:head# 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4afe4769:::benchmark_data_smithi194_196924_object103107:head# 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4afee656:::benchmark_data_smithi194_196924_object152667:head# 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4aff28e7:::benchmark_data_smithi194_196924_object144960:head# 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4affe136:::benchmark_data_smithi194_196924_object99130:head# 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b00ecf5:::benchmark_data_smithi194_196924_object132387:head# 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b01af93:::benchmark_data_smithi194_196924_object116578:head# 2024-04-29T15:00:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b045958:::benchmark_data_smithi194_196924_object112517:head# 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b049323:::benchmark_data_smithi194_196924_object146978:head# 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0525bd:::benchmark_data_smithi194_196924_object147509:head# 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b054252:::benchmark_data_smithi194_196924_object118798:head# 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b05f2e4:::benchmark_data_smithi194_196924_object124733:head# 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b07c915:::benchmark_data_smithi194_196924_object118509:head# 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b07d4b3:::benchmark_data_smithi194_196924_object140253:head# 2024-04-29T15:00:11.417 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0862e9:::benchmark_data_smithi194_196924_object156976:head# 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b09a733:::benchmark_data_smithi194_196924_object122999:head# 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b09cc94:::benchmark_data_smithi194_196924_object103749:head# 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0bbce3:::benchmark_data_smithi194_196924_object153324:head# 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0c7e29:::benchmark_data_smithi194_196924_object96685:head# 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0d475e:::benchmark_data_smithi194_196924_object147652:head# 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0d9c1b:::benchmark_data_smithi194_196924_object132654:head# 2024-04-29T15:00:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0df60d:::benchmark_data_smithi194_196924_object147372:head# 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b0f466e:::benchmark_data_smithi194_196924_object141375:head# 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b1026e2:::benchmark_data_smithi194_196924_object112343:head# 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b10c68b:::benchmark_data_smithi194_196924_object152764:head# 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b10e08c:::benchmark_data_smithi194_196924_object100679:head# 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b11c5e1:::benchmark_data_smithi194_196924_object113717:head# 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b138eb7:::benchmark_data_smithi194_196924_object103983:head# 2024-04-29T15:00:11.419 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b13a9e8:::benchmark_data_smithi194_196924_object126272:head# 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b150218:::benchmark_data_smithi194_196924_object138832:head# 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b155b64:::benchmark_data_smithi194_196924_object114878:head# 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b158397:::benchmark_data_smithi194_196924_object137259:head# 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b168135:::benchmark_data_smithi194_196924_object107024:head# 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b1892d8:::benchmark_data_smithi194_196924_object117878:head# 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b199920:::benchmark_data_smithi194_196924_object98200:head# 2024-04-29T15:00:11.420 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b199a6e:::benchmark_data_smithi194_196924_object135082:head# 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b1aef67:::benchmark_data_smithi194_196924_object122911:head# 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b1ba3e4:::benchmark_data_smithi194_196924_object153464:head# 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b1c9803:::benchmark_data_smithi194_196924_object151703:head# 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b1caace:::benchmark_data_smithi194_196924_object146875:head# 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b239778:::benchmark_data_smithi194_196924_object99251:head# 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b252fe2:::benchmark_data_smithi194_196924_object138224:head# 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.421 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b254feb:::benchmark_data_smithi194_196924_object147162:head# 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b2639d4:::benchmark_data_smithi194_196924_object148203:head# 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b275120:::benchmark_data_smithi194_196924_object148465:head# 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b27c976:::benchmark_data_smithi194_196924_object157727:head# 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b28c62b:::benchmark_data_smithi194_196924_object116981:head# 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b28e85e:::benchmark_data_smithi194_196924_object133112:head# 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b29f4d6:::benchmark_data_smithi194_196924_object113559:head# 2024-04-29T15:00:11.422 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b2b9efa:::benchmark_data_smithi194_196924_object102771:head# 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b2ec8d4:::benchmark_data_smithi194_196924_object119211:head# 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b2ed018:::benchmark_data_smithi194_196924_object143977:head# 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b2f6670:::benchmark_data_smithi194_196924_object125396:head# 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b2f96af:::benchmark_data_smithi194_196924_object95735:head# 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b320270:::benchmark_data_smithi194_196924_object105013:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b32f32e:::benchmark_data_smithi194_196924_object144981:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b341117:::benchmark_data_smithi194_196924_object124675:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b369425:::benchmark_data_smithi194_196924_object131126:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b37675c:::benchmark_data_smithi194_196924_object159079:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b37abee:::benchmark_data_smithi194_196924_object139301:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b392df3:::benchmark_data_smithi194_196924_object158300:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3a5715:::benchmark_data_smithi194_196924_object157965:head# 2024-04-29T15:00:11.424 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3af547:::benchmark_data_smithi194_196924_object134249:head# 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3b690f:::benchmark_data_smithi194_196924_object153481:head# 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3beda3:::benchmark_data_smithi194_196924_object156603:head# 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3d5143:::benchmark_data_smithi194_196924_object133125:head# 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3ea7dd:::benchmark_data_smithi194_196924_object147894:head# 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3eed62:::benchmark_data_smithi194_196924_object158426:head# 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3f0c60:::benchmark_data_smithi194_196924_object153003:head# 2024-04-29T15:00:11.425 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3f225d:::benchmark_data_smithi194_196924_object132676:head# 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b3f60e4:::benchmark_data_smithi194_196924_object138270:head# 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b405ab1:::benchmark_data_smithi194_196924_object99852:head# 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b40d972:::benchmark_data_smithi194_196924_object146080:head# 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b42c3d9:::benchmark_data_smithi194_196924_object129148:head# 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b42faf5:::benchmark_data_smithi194_196924_object137608:head# 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b4327f8:::benchmark_data_smithi194_196924_object149987:head# 2024-04-29T15:00:11.426 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.427 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b456d90:::benchmark_data_smithi194_196924_object139906:head# 2024-04-29T15:00:11.427 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.439 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b45d89b:::benchmark_data_smithi194_196924_object119108:head# 2024-04-29T15:00:11.439 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.439 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b47cae0:::benchmark_data_smithi194_196924_object130133:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b493e5e:::benchmark_data_smithi194_196924_object107557:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b4a913b:::benchmark_data_smithi194_196924_object143134:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b4cc43b:::benchmark_data_smithi194_196924_object102469:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b4d1388:::benchmark_data_smithi194_196924_object121016:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b4e54ad:::benchmark_data_smithi194_196924_object144120:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b4eda8c:::benchmark_data_smithi194_196924_object153210:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b4f12d3:::benchmark_data_smithi194_196924_object153883:head# 2024-04-29T15:00:11.440 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b50fdb7:::benchmark_data_smithi194_196924_object136783:head# 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b516455:::benchmark_data_smithi194_196924_object156320:head# 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b53fb7c:::benchmark_data_smithi194_196924_object125471:head# 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b53ff80:::benchmark_data_smithi194_196924_object134344:head# 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b54482e:::benchmark_data_smithi194_196924_object111719:head# 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b548fb9:::benchmark_data_smithi194_196924_object127885:head# 2024-04-29T15:00:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b561180:::benchmark_data_smithi194_196924_object117414:head# 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b57ec7d:::benchmark_data_smithi194_196924_object110724:head# 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b5864c4:::benchmark_data_smithi194_196924_object131849:head# 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b595a8c:::benchmark_data_smithi194_196924_object154137:head# 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b597ab4:::benchmark_data_smithi194_196924_object110332:head# 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b5aa1d1:::benchmark_data_smithi194_196924_object130640:head# 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.442 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b5ad66d:::benchmark_data_smithi194_196924_object155205:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b5c54d1:::benchmark_data_smithi194_196924_object135704:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b5cc9ed:::benchmark_data_smithi194_196924_object98562:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b5e1088:::benchmark_data_smithi194_196924_object150398:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b5eef88:::benchmark_data_smithi194_196924_object137134:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b601178:::benchmark_data_smithi194_196924_object125613:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b607b2c:::benchmark_data_smithi194_196924_object99326:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b610980:::benchmark_data_smithi194_196924_object152352:head# 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.443 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b623de5:::benchmark_data_smithi194_196924_object139622:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b625b01:::benchmark_data_smithi194_196924_object146666:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b63a2df:::benchmark_data_smithi194_196924_object136873:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b647aae:::benchmark_data_smithi194_196924_object158036:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6562a2:::benchmark_data_smithi194_196924_object99441:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b66b071:::benchmark_data_smithi194_196924_object107230:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b67263c:::benchmark_data_smithi194_196924_object142328:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b681376:::benchmark_data_smithi194_196924_object135236:head# 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.444 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b68362a:::benchmark_data_smithi194_196924_object101205:head# 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6842b9:::benchmark_data_smithi194_196924_object146103:head# 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b694116:::benchmark_data_smithi194_196924_object116423:head# 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b69f010:::benchmark_data_smithi194_196924_object140058:head# 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6a818a:::benchmark_data_smithi194_196924_object104649:head# 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6b7508:::benchmark_data_smithi194_196924_object156325:head# 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6c60cb:::benchmark_data_smithi194_196924_object121909:head# 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6cf62d:::benchmark_data_smithi194_196924_object122502:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6e1c7c:::benchmark_data_smithi194_196924_object128687:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b6f650e:::benchmark_data_smithi194_196924_object118434:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b742407:::benchmark_data_smithi194_196924_object114555:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7497dc:::benchmark_data_smithi194_196924_object152050:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b75108e:::benchmark_data_smithi194_196924_object138699:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b771aa6:::benchmark_data_smithi194_196924_object113199:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b783a67:::benchmark_data_smithi194_196924_object118086:head# 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.446 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7902be:::benchmark_data_smithi194_196924_object154143:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b790c3f:::benchmark_data_smithi194_196924_object107957:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7ac03c:::benchmark_data_smithi194_196924_object150128:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7b7dc9:::benchmark_data_smithi194_196924_object144728:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7c0e04:::benchmark_data_smithi194_196924_object103978:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7c6461:::benchmark_data_smithi194_196924_object136392:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7d905c:::benchmark_data_smithi194_196924_object139187:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7e1c72:::benchmark_data_smithi194_196924_object104991:head# 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.447 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b7ee92d:::benchmark_data_smithi194_196924_object152016:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b81aedf:::benchmark_data_smithi194_196924_object158219:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b82cbc9:::benchmark_data_smithi194_196924_object115357:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b847806:::benchmark_data_smithi194_196924_object125252:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b849f95:::benchmark_data_smithi194_196924_object132819:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b87b3b3:::benchmark_data_smithi194_196924_object159065:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b87c5e7:::benchmark_data_smithi194_196924_object111864:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b88d789:::benchmark_data_smithi194_196924_object150223:head# 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.448 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b89ef3c:::benchmark_data_smithi194_196924_object156089:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b8bb409:::benchmark_data_smithi194_196924_object136767:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b8d3ca2:::benchmark_data_smithi194_196924_object114298:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b8d9e6e:::benchmark_data_smithi194_196924_object120929:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b8ebb58:::benchmark_data_smithi194_196924_object159043:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b90a71f:::benchmark_data_smithi194_196924_object110191:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b923c33:::benchmark_data_smithi194_196924_object135741:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b964b3a:::benchmark_data_smithi194_196924_object127826:head# 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.449 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b967afc:::benchmark_data_smithi194_196924_object122674:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b97084e:::benchmark_data_smithi194_196924_object124929:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b971d84:::benchmark_data_smithi194_196924_object145491:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b97a190:::benchmark_data_smithi194_196924_object120454:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b9829e5:::benchmark_data_smithi194_196924_object106032:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b9aedb6:::benchmark_data_smithi194_196924_object145017:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b9bed3a:::benchmark_data_smithi194_196924_object108202:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b9d3c6f:::benchmark_data_smithi194_196924_object102742:head# 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.450 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b9d6322:::benchmark_data_smithi194_196924_object106444:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b9da915:::benchmark_data_smithi194_196924_object123406:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4b9fa3a2:::benchmark_data_smithi194_196924_object137942:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba1a82b:::benchmark_data_smithi194_196924_object140412:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba207b5:::benchmark_data_smithi194_196924_object118076:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba23648:::benchmark_data_smithi194_196924_object141175:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba35579:::benchmark_data_smithi194_196924_object127886:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba3901c:::benchmark_data_smithi194_196924_object140038:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba3afad:::benchmark_data_smithi194_196924_object119311:head# 2024-04-29T15:00:11.451 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba717e6:::benchmark_data_smithi194_196924_object158192:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba81c4f:::benchmark_data_smithi194_196924_object135887:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ba99534:::benchmark_data_smithi194_196924_object139545:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bac71b9:::benchmark_data_smithi194_196924_object141405:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bad68f3:::benchmark_data_smithi194_196924_object127188:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bada802:::benchmark_data_smithi194_196924_object131036:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb0e7dc:::benchmark_data_smithi194_196924_object101406:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb29700:::benchmark_data_smithi194_196924_object104654:head# 2024-04-29T15:00:11.452 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb2a208:::benchmark_data_smithi194_196924_object152100:head# 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb4a28f:::benchmark_data_smithi194_196924_object135618:head# 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb62a85:::benchmark_data_smithi194_196924_object154246:head# 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb837fe:::benchmark_data_smithi194_196924_object126103:head# 2024-04-29T15:00:11.453 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb891b6:::benchmark_data_smithi194_196924_object100186:head# 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bb9c41e:::benchmark_data_smithi194_196924_object117162:head# 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bba9189:::benchmark_data_smithi194_196924_object152258:head# 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bbabfdd:::benchmark_data_smithi194_196924_object151550:head# 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bbdcf21:::benchmark_data_smithi194_196924_object102873:head# 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.464 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bbf7d2b:::benchmark_data_smithi194_196924_object149590:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc0c41c:::benchmark_data_smithi194_196924_object145940:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc179b0:::benchmark_data_smithi194_196924_object110204:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc18683:::benchmark_data_smithi194_196924_object126773:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc305c6:::benchmark_data_smithi194_196924_object131583:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc443e3:::benchmark_data_smithi194_196924_object156590:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc48c94:::benchmark_data_smithi194_196924_object121381:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc4f6eb:::benchmark_data_smithi194_196924_object152424:head# 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.465 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc5117e:::benchmark_data_smithi194_196924_object110635:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc5158d:::benchmark_data_smithi194_196924_object157390:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc554bb:::benchmark_data_smithi194_196924_object148451:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc57456:::benchmark_data_smithi194_196924_object114088:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc5c7d8:::benchmark_data_smithi194_196924_object119385:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc5cdc1:::benchmark_data_smithi194_196924_object125009:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc5eaf9:::benchmark_data_smithi194_196924_object128626:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc65354:::benchmark_data_smithi194_196924_object156038:head# 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.466 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc73ae7:::benchmark_data_smithi194_196924_object103861:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc757de:::benchmark_data_smithi194_196924_object120259:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc809b6:::benchmark_data_smithi194_196924_object144963:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc87a6f:::benchmark_data_smithi194_196924_object151046:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bc8dc80:::benchmark_data_smithi194_196924_object111616:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bca0668:::benchmark_data_smithi194_196924_object129020:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bca5064:::benchmark_data_smithi194_196924_object131296:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bcb5280:::benchmark_data_smithi194_196924_object139879:head# 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bcbe56b:::benchmark_data_smithi194_196924_object99867:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bcc05b8:::benchmark_data_smithi194_196924_object103135:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bce90e3:::benchmark_data_smithi194_196924_object132147:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bcf5074:::benchmark_data_smithi194_196924_object126997:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bcf8683:::benchmark_data_smithi194_196924_object105790:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bd15a19:::benchmark_data_smithi194_196924_object105419:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bd727e2:::benchmark_data_smithi194_196924_object132417:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bd767ba:::benchmark_data_smithi194_196924_object126104:head# 2024-04-29T15:00:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bd7c61c:::benchmark_data_smithi194_196924_object120653:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bd7ec29:::benchmark_data_smithi194_196924_object141857:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bd96832:::benchmark_data_smithi194_196924_object127428:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bda7a02:::benchmark_data_smithi194_196924_object149650:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bda7c68:::benchmark_data_smithi194_196924_object121317:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bda7ec8:::benchmark_data_smithi194_196924_object154856:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bdadf23:::benchmark_data_smithi194_196924_object108303:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bdba08c:::benchmark_data_smithi194_196924_object143890:head# 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.469 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bdd521e:::benchmark_data_smithi194_196924_object102966:head# 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bdf0041:::benchmark_data_smithi194_196924_object114735:head# 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bdfda26:::benchmark_data_smithi194_196924_object121030:head# 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be0c925:::benchmark_data_smithi194_196924_object131522:head# 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be15de6:::benchmark_data_smithi194_196924_object131033:head# 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be1e897:::benchmark_data_smithi194_196924_object107019:head# 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be1fa8e:::benchmark_data_smithi194_196924_object106599:head# 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.470 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be386d5:::benchmark_data_smithi194_196924_object146766:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be3f2ec:::benchmark_data_smithi194_196924_object103601:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be4ed8f:::benchmark_data_smithi194_196924_object99916:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be684b4:::benchmark_data_smithi194_196924_object104254:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be735f4:::benchmark_data_smithi194_196924_object133345:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be90471:::benchmark_data_smithi194_196924_object147388:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4be96ff7:::benchmark_data_smithi194_196924_object129907:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bea0f21:::benchmark_data_smithi194_196924_object100792:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4beb7332:::benchmark_data_smithi194_196924_object151352:head# 2024-04-29T15:00:11.471 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4beb7ed6:::benchmark_data_smithi194_196924_object107345:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4beb89f2:::benchmark_data_smithi194_196924_object115977:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bedf680:::benchmark_data_smithi194_196924_object143926:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf0e532:::benchmark_data_smithi194_196924_object152252:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf11f2b:::benchmark_data_smithi194_196924_object101557:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf210e7:::benchmark_data_smithi194_196924_object122508:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf339d3:::benchmark_data_smithi194_196924_object107319:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf50151:::benchmark_data_smithi194_196924_object125176:head# 2024-04-29T15:00:11.472 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf502b9:::benchmark_data_smithi194_196924_object124311:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf56f52:::benchmark_data_smithi194_196924_object95740:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf582c2:::benchmark_data_smithi194_196924_object108156:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf5f25d:::benchmark_data_smithi194_196924_object148835:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf79e2e:::benchmark_data_smithi194_196924_object116138:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf7edfb:::benchmark_data_smithi194_196924_object143476:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf93f59:::benchmark_data_smithi194_196924_object133540:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bf99e0c:::benchmark_data_smithi194_196924_object128815:head# 2024-04-29T15:00:11.473 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bfb58a0:::benchmark_data_smithi194_196924_object152065:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bfc47f0:::benchmark_data_smithi194_196924_object138692:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bfd0e85:::benchmark_data_smithi194_196924_object159535:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bfe0f73:::benchmark_data_smithi194_196924_object106137:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bfea78d:::benchmark_data_smithi194_196924_object148233:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4bff26ee:::benchmark_data_smithi194_196924_object113251:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c00ade7:::benchmark_data_smithi194_196924_object153556:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c01ca76:::benchmark_data_smithi194_196924_object146436:head# 2024-04-29T15:00:11.474 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c01db8a:::benchmark_data_smithi194_196924_object107603:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c034bdd:::benchmark_data_smithi194_196924_object148189:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c04c15b:::benchmark_data_smithi194_196924_object108744:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c0709b1:::benchmark_data_smithi194_196924_object111721:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c0b938c:::benchmark_data_smithi194_196924_object104139:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c0cc73e:::benchmark_data_smithi194_196924_object152856:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c0e19ee:::benchmark_data_smithi194_196924_object126678:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c0e48bc:::benchmark_data_smithi194_196924_object119440:head# 2024-04-29T15:00:11.475 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c0eeeee:::benchmark_data_smithi194_196924_object117411:head# 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c0f3c6c:::benchmark_data_smithi194_196924_object124331:head# 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c100cb4:::benchmark_data_smithi194_196924_object131910:head# 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c106007:::benchmark_data_smithi194_196924_object107322:head# 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c106e6f:::benchmark_data_smithi194_196924_object127616:head# 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c11475e:::benchmark_data_smithi194_196924_object106804:head# 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1270f7:::benchmark_data_smithi194_196924_object118254:head# 2024-04-29T15:00:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c139ab2:::benchmark_data_smithi194_196924_object102221:head# 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c141d65:::benchmark_data_smithi194_196924_object140490:head# 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c154503:::benchmark_data_smithi194_196924_object102941:head# 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1731a4:::benchmark_data_smithi194_196924_object153004:head# 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c197258:::benchmark_data_smithi194_196924_object147422:head# 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.489 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1985fe:::benchmark_data_smithi194_196924_object134321:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1ab4a2:::benchmark_data_smithi194_196924_object122329:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1b3802:::benchmark_data_smithi194_196924_object122315:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1bd9ff:::benchmark_data_smithi194_196924_object150331:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1d0eef:::benchmark_data_smithi194_196924_object152104:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1f2a5b:::benchmark_data_smithi194_196924_object125954:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c1f52a3:::benchmark_data_smithi194_196924_object145606:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c290fc2:::benchmark_data_smithi194_196924_object140735:head# 2024-04-29T15:00:11.490 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2aa971:::benchmark_data_smithi194_196924_object106152:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2ae4ed:::benchmark_data_smithi194_196924_object101156:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2b4fb5:::benchmark_data_smithi194_196924_object109796:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2b5b49:::benchmark_data_smithi194_196924_object136080:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2bec8d:::benchmark_data_smithi194_196924_object98702:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2c2eaa:::benchmark_data_smithi194_196924_object141194:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2d262c:::benchmark_data_smithi194_196924_object140326:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2dc297:::benchmark_data_smithi194_196924_object148018:head# 2024-04-29T15:00:11.491 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2dc438:::benchmark_data_smithi194_196924_object125660:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2e120a:::benchmark_data_smithi194_196924_object132926:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c2f907f:::benchmark_data_smithi194_196924_object128383:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c339388:::benchmark_data_smithi194_196924_object125932:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c35f54f:::benchmark_data_smithi194_196924_object159349:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c37191a:::benchmark_data_smithi194_196924_object142523:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c380316:::benchmark_data_smithi194_196924_object98432:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c38c302:::benchmark_data_smithi194_196924_object153379:head# 2024-04-29T15:00:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c3df8a5:::benchmark_data_smithi194_196924_object134831:head# 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c408b3b:::benchmark_data_smithi194_196924_object154395:head# 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c41d538:::benchmark_data_smithi194_196924_object146086:head# 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c43bb45:::benchmark_data_smithi194_196924_object110301:head# 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c486d0f:::benchmark_data_smithi194_196924_object123464:head# 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c4a7cbc:::benchmark_data_smithi194_196924_object114976:head# 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c4c9c12:::benchmark_data_smithi194_196924_object101665:head# 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.493 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c4d284a:::benchmark_data_smithi194_196924_object118929:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c4e5e1e:::benchmark_data_smithi194_196924_object119761:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c51b90b:::benchmark_data_smithi194_196924_object127248:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c526c30:::benchmark_data_smithi194_196924_object128884:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c52eb33:::benchmark_data_smithi194_196924_object117835:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c54e30c:::benchmark_data_smithi194_196924_object110428:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c55a6dc:::benchmark_data_smithi194_196924_object139361:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c572c86:::benchmark_data_smithi194_196924_object114901:head# 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.494 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c577550:::benchmark_data_smithi194_196924_object155139:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c578c73:::benchmark_data_smithi194_196924_object141482:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c590e0c:::benchmark_data_smithi194_196924_object95549:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c596483:::benchmark_data_smithi194_196924_object109640:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c5c271b:::benchmark_data_smithi194_196924_object105197:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c5ca21a:::benchmark_data_smithi194_196924_object131116:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c5cd63b:::benchmark_data_smithi194_196924_object150923:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c5d3ce6:::benchmark_data_smithi194_196924_object117232:head# 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c5e79ca:::benchmark_data_smithi194_196924_object139952:head# 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c5f1b39:::benchmark_data_smithi194_196924_object105296:head# 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c5f6b8b:::benchmark_data_smithi194_196924_object115228:head# 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c60f4df:::benchmark_data_smithi194_196924_object107595:head# 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c6164da:::benchmark_data_smithi194_196924_object141385:head# 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c61b2f5:::benchmark_data_smithi194_196924_object101888:head# 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c627b3a:::benchmark_data_smithi194_196924_object146203:head# 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.496 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c63c76f:::benchmark_data_smithi194_196924_object115557:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c64642c:::benchmark_data_smithi194_196924_object126829:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c6605c6:::benchmark_data_smithi194_196924_object133025:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c679cc1:::benchmark_data_smithi194_196924_object149846:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c68f68b:::benchmark_data_smithi194_196924_object114471:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c697265:::benchmark_data_smithi194_196924_object140244:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c6e1275:::benchmark_data_smithi194_196924_object108782:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c6ede89:::benchmark_data_smithi194_196924_object119862:head# 2024-04-29T15:00:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c6f8e46:::benchmark_data_smithi194_196924_object136292:head# 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c6f9821:::benchmark_data_smithi194_196924_object147609:head# 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c70b1a1:::benchmark_data_smithi194_196924_object129587:head# 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c712de1:::benchmark_data_smithi194_196924_object112526:head# 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c7130fb:::benchmark_data_smithi194_196924_object132539:head# 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c71684e:::benchmark_data_smithi194_196924_object135147:head# 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c728081:::benchmark_data_smithi194_196924_object154091:head# 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.498 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c735bc3:::benchmark_data_smithi194_196924_object140894:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c73b84d:::benchmark_data_smithi194_196924_object127703:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c73f946:::benchmark_data_smithi194_196924_object158146:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c746029:::benchmark_data_smithi194_196924_object123160:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c76edec:::benchmark_data_smithi194_196924_object117871:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c77aed9:::benchmark_data_smithi194_196924_object147935:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c780c6c:::benchmark_data_smithi194_196924_object141038:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c792228:::benchmark_data_smithi194_196924_object145386:head# 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c7cb3ff:::benchmark_data_smithi194_196924_object135212:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c7d01ec:::benchmark_data_smithi194_196924_object113211:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c7d7e5a:::benchmark_data_smithi194_196924_object137149:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c8065bc:::benchmark_data_smithi194_196924_object108129:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c839edf:::benchmark_data_smithi194_196924_object103683:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c83b571:::benchmark_data_smithi194_196924_object154334:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c855e45:::benchmark_data_smithi194_196924_object106808:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c878845:::benchmark_data_smithi194_196924_object152740:head# 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.500 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c87ca80:::benchmark_data_smithi194_196924_object148780:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c87ecd1:::benchmark_data_smithi194_196924_object142880:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c883d0b:::benchmark_data_smithi194_196924_object117649:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c8868fa:::benchmark_data_smithi194_196924_object144661:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c8a73f7:::benchmark_data_smithi194_196924_object141811:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c8c5f52:::benchmark_data_smithi194_196924_object141459:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c8deb3d:::benchmark_data_smithi194_196924_object96053:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c8e7fbf:::benchmark_data_smithi194_196924_object124709:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c8ec80a:::benchmark_data_smithi194_196924_object150969:head# 2024-04-29T15:00:11.501 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.502 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c91123d:::benchmark_data_smithi194_196924_object117557:head# 2024-04-29T15:00:11.502 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.512 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c915876:::benchmark_data_smithi194_196924_object136600:head# 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c92115e:::benchmark_data_smithi194_196924_object120017:head# 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c92c028:::benchmark_data_smithi194_196924_object153922:head# 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c92cf26:::benchmark_data_smithi194_196924_object157485:head# 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c93e2f2:::benchmark_data_smithi194_196924_object125161:head# 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c940be3:::benchmark_data_smithi194_196924_object112347:head# 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c96ecc2:::benchmark_data_smithi194_196924_object111104:head# 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.513 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c97e99d:::benchmark_data_smithi194_196924_object159624:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c99d5c5:::benchmark_data_smithi194_196924_object121816:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c9c6284:::benchmark_data_smithi194_196924_object107301:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4c9d4b8f:::benchmark_data_smithi194_196924_object142761:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ca11530:::benchmark_data_smithi194_196924_object138099:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ca1e149:::benchmark_data_smithi194_196924_object107890:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ca24816:::benchmark_data_smithi194_196924_object157593:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ca59d29:::benchmark_data_smithi194_196924_object125496:head# 2024-04-29T15:00:11.514 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ca8ed09:::benchmark_data_smithi194_196924_object119008:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ca98a81:::benchmark_data_smithi194_196924_object133223:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ca9ecb6:::benchmark_data_smithi194_196924_object117240:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4caa9624:::benchmark_data_smithi194_196924_object115724:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4caac754:::benchmark_data_smithi194_196924_object151698:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cac4771:::benchmark_data_smithi194_196924_object150846:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb02298:::benchmark_data_smithi194_196924_object129098:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb30f13:::benchmark_data_smithi194_196924_object112021:head# 2024-04-29T15:00:11.515 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb31c4e:::benchmark_data_smithi194_196924_object106081:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb60272:::benchmark_data_smithi194_196924_object133542:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb69995:::benchmark_data_smithi194_196924_object148722:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb69d96:::benchmark_data_smithi194_196924_object119816:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb7a9ed:::benchmark_data_smithi194_196924_object146989:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb80e86:::benchmark_data_smithi194_196924_object122247:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb90775:::benchmark_data_smithi194_196924_object159370:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cb99efc:::benchmark_data_smithi194_196924_object154960:head# 2024-04-29T15:00:11.516 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cbcc5d2:::benchmark_data_smithi194_196924_object107108:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cbdf61f:::benchmark_data_smithi194_196924_object159001:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cbf22dc:::benchmark_data_smithi194_196924_object142010:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cbf26a2:::benchmark_data_smithi194_196924_object108853:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc0469a:::benchmark_data_smithi194_196924_object95566:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc0cc1b:::benchmark_data_smithi194_196924_object126079:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc25fd0:::benchmark_data_smithi194_196924_object149477:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc386bc:::benchmark_data_smithi194_196924_object114051:head# 2024-04-29T15:00:11.517 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc395a8:::benchmark_data_smithi194_196924_object103202:head# 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc5bdcd:::benchmark_data_smithi194_196924_object152334:head# 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc8c585:::benchmark_data_smithi194_196924_object150901:head# 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc8c59a:::benchmark_data_smithi194_196924_object131481:head# 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc910dd:::benchmark_data_smithi194_196924_object95831:head# 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cc9fc90:::benchmark_data_smithi194_196924_object142181:head# 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cca1064:::benchmark_data_smithi194_196924_object144916:head# 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.518 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cca7a3e:::benchmark_data_smithi194_196924_object106136:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cccf1e4:::benchmark_data_smithi194_196924_object96095:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ccd48f2:::benchmark_data_smithi194_196924_object95794:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ccd4e7e:::benchmark_data_smithi194_196924_object157645:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ccde8f0:::benchmark_data_smithi194_196924_object141009:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ccdf090:::benchmark_data_smithi194_196924_object106700:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ccfa033:::benchmark_data_smithi194_196924_object115986:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cd20097:::benchmark_data_smithi194_196924_object147650:head# 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.519 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cd2c233:::benchmark_data_smithi194_196924_object107831:head# 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cd576f1:::benchmark_data_smithi194_196924_object102863:head# 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cd6a461:::benchmark_data_smithi194_196924_object116947:head# 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cd71444:::benchmark_data_smithi194_196924_object135959:head# 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cda1fc8:::benchmark_data_smithi194_196924_object139233:head# 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cdd6962:::benchmark_data_smithi194_196924_object104531:head# 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cde8b91:::benchmark_data_smithi194_196924_object157367:head# 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.520 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce17798:::benchmark_data_smithi194_196924_object136874:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce1f6ae:::benchmark_data_smithi194_196924_object136794:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce20c1c:::benchmark_data_smithi194_196924_object154473:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce33e4c:::benchmark_data_smithi194_196924_object114725:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce41912:::benchmark_data_smithi194_196924_object111316:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce49992:::benchmark_data_smithi194_196924_object116687:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce4d5fc:::benchmark_data_smithi194_196924_object100691:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce54cb1:::benchmark_data_smithi194_196924_object152232:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce5a51e:::benchmark_data_smithi194_196924_object130059:head# 2024-04-29T15:00:11.521 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce7b5e9:::benchmark_data_smithi194_196924_object156407:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ce863d9:::benchmark_data_smithi194_196924_object148756:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ceb6e1d:::benchmark_data_smithi194_196924_object112976:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ceb975b:::benchmark_data_smithi194_196924_object127572:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cee22b2:::benchmark_data_smithi194_196924_object129793:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ceeb697:::benchmark_data_smithi194_196924_object138281:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cefd4a6:::benchmark_data_smithi194_196924_object107379:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ceff6fc:::benchmark_data_smithi194_196924_object118419:head# 2024-04-29T15:00:11.522 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf0cac6:::benchmark_data_smithi194_196924_object98223:head# 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf28376:::benchmark_data_smithi194_196924_object148487:head# 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf33a55:::benchmark_data_smithi194_196924_object141113:head# 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf38536:::benchmark_data_smithi194_196924_object140213:head# 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf5976f:::benchmark_data_smithi194_196924_object123051:head# 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf70d4e:::benchmark_data_smithi194_196924_object145114:head# 2024-04-29T15:00:11.523 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf7402d:::benchmark_data_smithi194_196924_object150408:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf83dae:::benchmark_data_smithi194_196924_object112775:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cf9b906:::benchmark_data_smithi194_196924_object135708:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cfa321c:::benchmark_data_smithi194_196924_object129569:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cfc653e:::benchmark_data_smithi194_196924_object103472:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4cfe45be:::benchmark_data_smithi194_196924_object111480:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d00a0fa:::benchmark_data_smithi194_196924_object126340:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d021983:::benchmark_data_smithi194_196924_object156303:head# 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.524 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d02b270:::benchmark_data_smithi194_196924_object97648:head# 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d050e7f:::benchmark_data_smithi194_196924_object97823:head# 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d05a6c7:::benchmark_data_smithi194_196924_object149585:head# 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d05b40e:::benchmark_data_smithi194_196924_object111226:head# 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d07c00a:::benchmark_data_smithi194_196924_object108582:head# 2024-04-29T15:00:11.525 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d086b5f:::benchmark_data_smithi194_196924_object114255:head# 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d0b593d:::benchmark_data_smithi194_196924_object108281:head# 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d0be5ca:::benchmark_data_smithi194_196924_object134386:head# 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d0c2f49:::benchmark_data_smithi194_196924_object154028:head# 2024-04-29T15:00:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d0d38cd:::benchmark_data_smithi194_196924_object122324:head# 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d0dd7df:::benchmark_data_smithi194_196924_object107231:head# 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d0f5ed0:::benchmark_data_smithi194_196924_object114377:head# 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d100fd9:::benchmark_data_smithi194_196924_object119010:head# 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d13122a:::benchmark_data_smithi194_196924_object102892:head# 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d179596:::benchmark_data_smithi194_196924_object107853:head# 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.536 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1851d7:::benchmark_data_smithi194_196924_object155517:head# 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1911cc:::benchmark_data_smithi194_196924_object117321:head# 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d19dd8b:::benchmark_data_smithi194_196924_object106603:head# 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1a4c3e:::benchmark_data_smithi194_196924_object99813:head# 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1a9751:::benchmark_data_smithi194_196924_object117627:head# 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1b1620:::benchmark_data_smithi194_196924_object105767:head# 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1d3bd2:::benchmark_data_smithi194_196924_object111906:head# 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.537 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1e7bb6:::benchmark_data_smithi194_196924_object146812:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d1e809d:::benchmark_data_smithi194_196924_object157623:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d205899:::benchmark_data_smithi194_196924_object124182:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d20f81c:::benchmark_data_smithi194_196924_object157960:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d211a9d:::benchmark_data_smithi194_196924_object157591:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d214717:::benchmark_data_smithi194_196924_object121413:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d224bb8:::benchmark_data_smithi194_196924_object117692:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d231885:::benchmark_data_smithi194_196924_object118336:head# 2024-04-29T15:00:11.538 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d2695c1:::benchmark_data_smithi194_196924_object131855:head# 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d26be2d:::benchmark_data_smithi194_196924_object120065:head# 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d26dd85:::benchmark_data_smithi194_196924_object139191:head# 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d272998:::benchmark_data_smithi194_196924_object141721:head# 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d280383:::benchmark_data_smithi194_196924_object158143:head# 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d294e5c:::benchmark_data_smithi194_196924_object124446:head# 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d2953f5:::benchmark_data_smithi194_196924_object118821:head# 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.539 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d295b5b:::benchmark_data_smithi194_196924_object151281:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d29913d:::benchmark_data_smithi194_196924_object141627:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d2d55b8:::benchmark_data_smithi194_196924_object152439:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3089c0:::benchmark_data_smithi194_196924_object128822:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d312240:::benchmark_data_smithi194_196924_object138722:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3290a9:::benchmark_data_smithi194_196924_object101433:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3312df:::benchmark_data_smithi194_196924_object98445:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d341b2e:::benchmark_data_smithi194_196924_object109560:head# 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.540 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d34b9f2:::benchmark_data_smithi194_196924_object141307:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d35ab3c:::benchmark_data_smithi194_196924_object152775:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d37122a:::benchmark_data_smithi194_196924_object159057:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d37a329:::benchmark_data_smithi194_196924_object127674:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3a6ede:::benchmark_data_smithi194_196924_object116750:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3b494a:::benchmark_data_smithi194_196924_object131145:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3d7182:::benchmark_data_smithi194_196924_object95963:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3db4c4:::benchmark_data_smithi194_196924_object101552:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3e0d08:::benchmark_data_smithi194_196924_object147780:head# 2024-04-29T15:00:11.541 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d3f2914:::benchmark_data_smithi194_196924_object152554:head# 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d401edd:::benchmark_data_smithi194_196924_object144992:head# 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d40f4e1:::benchmark_data_smithi194_196924_object118968:head# 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d42c6d7:::benchmark_data_smithi194_196924_object155974:head# 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d442aec:::benchmark_data_smithi194_196924_object113035:head# 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d454491:::benchmark_data_smithi194_196924_object146663:head# 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d455d4e:::benchmark_data_smithi194_196924_object122927:head# 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.542 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d459473:::benchmark_data_smithi194_196924_object100122:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d45be9e:::benchmark_data_smithi194_196924_object119039:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d47289a:::benchmark_data_smithi194_196924_object121020:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d486221:::benchmark_data_smithi194_196924_object129291:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d490462:::benchmark_data_smithi194_196924_object112494:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d49a7f9:::benchmark_data_smithi194_196924_object136310:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d49c2b4:::benchmark_data_smithi194_196924_object126165:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d4a2118:::benchmark_data_smithi194_196924_object121581:head# 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d4c59bd:::benchmark_data_smithi194_196924_object138507:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d4c7e77:::benchmark_data_smithi194_196924_object149984:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d4c88f4:::benchmark_data_smithi194_196924_object143857:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d4c996f:::benchmark_data_smithi194_196924_object157869:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d4d5b6a:::benchmark_data_smithi194_196924_object122676:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d4ebbf7:::benchmark_data_smithi194_196924_object114197:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d51620a:::benchmark_data_smithi194_196924_object124213:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d53c411:::benchmark_data_smithi194_196924_object153933:head# 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d5474f5:::benchmark_data_smithi194_196924_object112774:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d54d8e4:::benchmark_data_smithi194_196924_object105893:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d554126:::benchmark_data_smithi194_196924_object124141:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d55924a:::benchmark_data_smithi194_196924_object145031:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d57ebf9:::benchmark_data_smithi194_196924_object96833:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d58122c:::benchmark_data_smithi194_196924_object119691:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d591004:::benchmark_data_smithi194_196924_object101964:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d5c1317:::benchmark_data_smithi194_196924_object99226:head# 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.545 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d5c95cf:::benchmark_data_smithi194_196924_object126595:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d5df3f2:::benchmark_data_smithi194_196924_object136452:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d612f54:::benchmark_data_smithi194_196924_object106227:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d6152fe:::benchmark_data_smithi194_196924_object119609:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d61f72a:::benchmark_data_smithi194_196924_object133419:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d62fa61:::benchmark_data_smithi194_196924_object104887:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d64c4b3:::benchmark_data_smithi194_196924_object138798:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d66317b:::benchmark_data_smithi194_196924_object119472:head# 2024-04-29T15:00:11.546 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.564 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d668acd:::benchmark_data_smithi194_196924_object144669:head# 2024-04-29T15:00:11.564 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.564 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d67e822:::benchmark_data_smithi194_196924_object158647:head# 2024-04-29T15:00:11.564 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.564 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d68f449:::benchmark_data_smithi194_196924_object123661:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d6b917c:::benchmark_data_smithi194_196924_object107392:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d6e5d71:::benchmark_data_smithi194_196924_object107106:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d6ec20e:::benchmark_data_smithi194_196924_object101514:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d6f29d3:::benchmark_data_smithi194_196924_object154523:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d71431c:::benchmark_data_smithi194_196924_object116701:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d7176d2:::benchmark_data_smithi194_196924_object112908:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d71dfd4:::benchmark_data_smithi194_196924_object156805:head# 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d723531:::benchmark_data_smithi194_196924_object117520:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d72685c:::benchmark_data_smithi194_196924_object124468:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d72a417:::benchmark_data_smithi194_196924_object114348:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d72a471:::benchmark_data_smithi194_196924_object110707:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d72bcec:::benchmark_data_smithi194_196924_object101561:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d730f56:::benchmark_data_smithi194_196924_object134035:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d73a5a7:::benchmark_data_smithi194_196924_object106386:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d73c9d4:::benchmark_data_smithi194_196924_object159601:head# 2024-04-29T15:00:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d749aec:::benchmark_data_smithi194_196924_object109674:head# 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d764ea3:::benchmark_data_smithi194_196924_object95663:head# 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d78d526:::benchmark_data_smithi194_196924_object140126:head# 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d7a9248:::benchmark_data_smithi194_196924_object110307:head# 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d7aab1b:::benchmark_data_smithi194_196924_object109254:head# 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d7b614a:::benchmark_data_smithi194_196924_object154506:head# 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d7d84f0:::benchmark_data_smithi194_196924_object103715:head# 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.567 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d7e4fa3:::benchmark_data_smithi194_196924_object96849:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d82937c:::benchmark_data_smithi194_196924_object116465:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8447fe:::benchmark_data_smithi194_196924_object118584:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8555ed:::benchmark_data_smithi194_196924_object138752:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d858ba9:::benchmark_data_smithi194_196924_object142213:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d862d65:::benchmark_data_smithi194_196924_object133636:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d863958:::benchmark_data_smithi194_196924_object150298:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d86680f:::benchmark_data_smithi194_196924_object123773:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d886004:::benchmark_data_smithi194_196924_object116827:head# 2024-04-29T15:00:11.568 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d888e67:::benchmark_data_smithi194_196924_object155689:head# 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d889db7:::benchmark_data_smithi194_196924_object119625:head# 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8a00e3:::benchmark_data_smithi194_196924_object151616:head# 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8a4829:::benchmark_data_smithi194_196924_object153436:head# 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8b0582:::benchmark_data_smithi194_196924_object125713:head# 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8c5e27:::benchmark_data_smithi194_196924_object154794:head# 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8c76b3:::benchmark_data_smithi194_196924_object136450:head# 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.569 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8e760c:::benchmark_data_smithi194_196924_object150409:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8f5838:::benchmark_data_smithi194_196924_object141098:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d8fca79:::benchmark_data_smithi194_196924_object159754:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d94c0d9:::benchmark_data_smithi194_196924_object109527:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d94d971:::benchmark_data_smithi194_196924_object118349:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d950ba2:::benchmark_data_smithi194_196924_object125014:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d959f07:::benchmark_data_smithi194_196924_object95438:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d95b031:::benchmark_data_smithi194_196924_object97279:head# 2024-04-29T15:00:11.570 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9608ed:::benchmark_data_smithi194_196924_object139723:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d96b824:::benchmark_data_smithi194_196924_object154038:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d975ee5:::benchmark_data_smithi194_196924_object105484:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d97e487:::benchmark_data_smithi194_196924_object111623:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d98dbcf:::benchmark_data_smithi194_196924_object138580:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d99520b:::benchmark_data_smithi194_196924_object95873:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9a11d3:::benchmark_data_smithi194_196924_object130915:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9b6800:::benchmark_data_smithi194_196924_object159594:head# 2024-04-29T15:00:11.571 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9c469d:::benchmark_data_smithi194_196924_object118566:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9c5ebe:::benchmark_data_smithi194_196924_object97964:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9cf4e4:::benchmark_data_smithi194_196924_object124775:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9da0d8:::benchmark_data_smithi194_196924_object158911:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9de951:::benchmark_data_smithi194_196924_object156508:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9e6f78:::benchmark_data_smithi194_196924_object100907:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9ea2e7:::benchmark_data_smithi194_196924_object127171:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9eca67:::benchmark_data_smithi194_196924_object113734:head# 2024-04-29T15:00:11.572 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9ef074:::benchmark_data_smithi194_196924_object127570:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4d9ef479:::benchmark_data_smithi194_196924_object109582:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da0933d:::benchmark_data_smithi194_196924_object98987:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da2dc73:::benchmark_data_smithi194_196924_object101426:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da55b21:::benchmark_data_smithi194_196924_object127855:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da62715:::benchmark_data_smithi194_196924_object148500:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da6afe0:::benchmark_data_smithi194_196924_object146449:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da7857f:::benchmark_data_smithi194_196924_object143530:head# 2024-04-29T15:00:11.573 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da79230:::benchmark_data_smithi194_196924_object132385:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4da855ef:::benchmark_data_smithi194_196924_object152939:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dacb21f:::benchmark_data_smithi194_196924_object141231:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dae6ed0:::benchmark_data_smithi194_196924_object124652:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db20f51:::benchmark_data_smithi194_196924_object151252:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db36c07:::benchmark_data_smithi194_196924_object131074:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db3ff50:::benchmark_data_smithi194_196924_object106188:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db4e9dc:::benchmark_data_smithi194_196924_object122404:head# 2024-04-29T15:00:11.574 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db5c8ec:::benchmark_data_smithi194_196924_object102679:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db6c065:::benchmark_data_smithi194_196924_object110150:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db6e1d0:::benchmark_data_smithi194_196924_object108283:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4db7a1bf:::benchmark_data_smithi194_196924_object95721:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dbbd87a:::benchmark_data_smithi194_196924_object152435:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dbc9972:::benchmark_data_smithi194_196924_object102700:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dbcbbfd:::benchmark_data_smithi194_196924_object139751:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dbdbb44:::benchmark_data_smithi194_196924_object114324:head# 2024-04-29T15:00:11.575 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dc00f48:::benchmark_data_smithi194_196924_object139552:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dc18dbf:::benchmark_data_smithi194_196924_object139870:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dc4df5a:::benchmark_data_smithi194_196924_object146917:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dc4f43b:::benchmark_data_smithi194_196924_object148048:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dc60887:::benchmark_data_smithi194_196924_object115512:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dc66a4d:::benchmark_data_smithi194_196924_object122561:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dca2290:::benchmark_data_smithi194_196924_object149832:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dcb2ac8:::benchmark_data_smithi194_196924_object138169:head# 2024-04-29T15:00:11.576 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dcb89b1:::benchmark_data_smithi194_196924_object114168:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dccb574:::benchmark_data_smithi194_196924_object97567:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dccf0aa:::benchmark_data_smithi194_196924_object142390:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dce0606:::benchmark_data_smithi194_196924_object120822:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dce5b45:::benchmark_data_smithi194_196924_object131035:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dcf645a:::benchmark_data_smithi194_196924_object149632:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dcfc5bf:::benchmark_data_smithi194_196924_object154749:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd095eb:::benchmark_data_smithi194_196924_object141001:head# 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.577 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd0efc0:::benchmark_data_smithi194_196924_object110288:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd169c4:::benchmark_data_smithi194_196924_object145944:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd1963b:::benchmark_data_smithi194_196924_object133127:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd20fe4:::benchmark_data_smithi194_196924_object136419:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd218ec:::benchmark_data_smithi194_196924_object127233:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd3c3c3:::benchmark_data_smithi194_196924_object106920:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd45a72:::benchmark_data_smithi194_196924_object141885:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd59d6a:::benchmark_data_smithi194_196924_object116627:head# 2024-04-29T15:00:11.578 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd5c9f4:::benchmark_data_smithi194_196924_object112262:head# 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd5d288:::benchmark_data_smithi194_196924_object113185:head# 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd7b63c:::benchmark_data_smithi194_196924_object138074:head# 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd93a96:::benchmark_data_smithi194_196924_object134087:head# 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dd9c6e0:::benchmark_data_smithi194_196924_object107488:head# 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ddae920:::benchmark_data_smithi194_196924_object154554:head# 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ddd199a:::benchmark_data_smithi194_196924_object156700:head# 2024-04-29T15:00:11.579 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.580 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: pgmap v2919: 17 pgs: 17 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 979 op/s 2024-04-29T15:00:11.581 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:11 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ddda9ec:::benchmark_data_smithi194_196924_object151147:head# 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ddf9529:::benchmark_data_smithi194_196924_object149831:head# 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de0fd9f:::benchmark_data_smithi194_196924_object154904:head# 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de2371d:::benchmark_data_smithi194_196924_object158469:head# 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de302ec:::benchmark_data_smithi194_196924_object98164:head# 2024-04-29T15:00:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de36f82:::benchmark_data_smithi194_196924_object101097:head# 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de507e5:::benchmark_data_smithi194_196924_object141828:head# 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de56161:::benchmark_data_smithi194_196924_object148769:head# 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de581e2:::benchmark_data_smithi194_196924_object137536:head# 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4de6c12b:::benchmark_data_smithi194_196924_object122638:head# 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4deaf81b:::benchmark_data_smithi194_196924_object110826:head# 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dedfa6b:::benchmark_data_smithi194_196924_object98741:head# 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4def9531:::benchmark_data_smithi194_196924_object144434:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df119b0:::benchmark_data_smithi194_196924_object110287:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df33d4e:::benchmark_data_smithi194_196924_object139097:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df369e2:::benchmark_data_smithi194_196924_object132218:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df402e7:::benchmark_data_smithi194_196924_object123243:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df412fa:::benchmark_data_smithi194_196924_object149488:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df44aba:::benchmark_data_smithi194_196924_object104803:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df55c54:::benchmark_data_smithi194_196924_object151250:head# 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4df9d406:::benchmark_data_smithi194_196924_object154153:head# 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dfa4b14:::benchmark_data_smithi194_196924_object104426:head# 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dfac436:::benchmark_data_smithi194_196924_object125348:head# 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dfb0bb0:::benchmark_data_smithi194_196924_object147272:head# 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dfcc513:::benchmark_data_smithi194_196924_object131127:head# 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dfea619:::benchmark_data_smithi194_196924_object142069:head# 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dfeb0a1:::benchmark_data_smithi194_196924_object123789:head# 2024-04-29T15:00:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4dfeb295:::benchmark_data_smithi194_196924_object142797:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e011b73:::benchmark_data_smithi194_196924_object125555:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e01e3d8:::benchmark_data_smithi194_196924_object133134:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e0326d0:::benchmark_data_smithi194_196924_object150961:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e04d6b2:::benchmark_data_smithi194_196924_object103638:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e0783a4:::benchmark_data_smithi194_196924_object144553:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e08008f:::benchmark_data_smithi194_196924_object112666:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e0842ce:::benchmark_data_smithi194_196924_object142927:head# 2024-04-29T15:00:11.611 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e09583e:::benchmark_data_smithi194_196924_object132897:head# 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e09c16b:::benchmark_data_smithi194_196924_object105870:head# 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e0d6ce4:::benchmark_data_smithi194_196924_object134819:head# 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e0f8da1:::benchmark_data_smithi194_196924_object123208:head# 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e12f7f3:::benchmark_data_smithi194_196924_object137934:head# 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e146055:::benchmark_data_smithi194_196924_object106851:head# 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e14dcb1:::benchmark_data_smithi194_196924_object135629:head# 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.612 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1519f9:::benchmark_data_smithi194_196924_object150136:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e152a48:::benchmark_data_smithi194_196924_object134551:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e16b800:::benchmark_data_smithi194_196924_object131551:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e182b25:::benchmark_data_smithi194_196924_object135435:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e196aa8:::benchmark_data_smithi194_196924_object149991:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e198468:::benchmark_data_smithi194_196924_object118139:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1b099e:::benchmark_data_smithi194_196924_object157627:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1c7609:::benchmark_data_smithi194_196924_object113646:head# 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.613 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1c9a7a:::benchmark_data_smithi194_196924_object131582:head# 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1cb27f:::benchmark_data_smithi194_196924_object129544:head# 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1d6e2e:::benchmark_data_smithi194_196924_object134001:head# 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1de130:::benchmark_data_smithi194_196924_object113182:head# 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e1e1fdf:::benchmark_data_smithi194_196924_object137556:head# 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e211ffe:::benchmark_data_smithi194_196924_object120970:head# 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e21335a:::benchmark_data_smithi194_196924_object121253:head# 2024-04-29T15:00:11.614 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e226026:::benchmark_data_smithi194_196924_object100689:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e23b245:::benchmark_data_smithi194_196924_object139124:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e2466a5:::benchmark_data_smithi194_196924_object130725:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e24c0c1:::benchmark_data_smithi194_196924_object105963:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e271e67:::benchmark_data_smithi194_196924_object120261:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e279810:::benchmark_data_smithi194_196924_object130277:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e283ec9:::benchmark_data_smithi194_196924_object116268:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e297229:::benchmark_data_smithi194_196924_object155592:head# 2024-04-29T15:00:11.615 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e2a4a9f:::benchmark_data_smithi194_196924_object121133:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e2ab65a:::benchmark_data_smithi194_196924_object154769:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e2b5fc5:::benchmark_data_smithi194_196924_object111829:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e2d2bda:::benchmark_data_smithi194_196924_object96482:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e2fc46a:::benchmark_data_smithi194_196924_object145555:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e310c4b:::benchmark_data_smithi194_196924_object96906:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e3239d1:::benchmark_data_smithi194_196924_object110384:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e32fddc:::benchmark_data_smithi194_196924_object138869:head# 2024-04-29T15:00:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e333fbe:::benchmark_data_smithi194_196924_object107413:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e346b27:::benchmark_data_smithi194_196924_object159236:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e35e6d4:::benchmark_data_smithi194_196924_object139249:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e3624a9:::benchmark_data_smithi194_196924_object147049:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e37120f:::benchmark_data_smithi194_196924_object124473:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e374cf1:::benchmark_data_smithi194_196924_object131328:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e39f860:::benchmark_data_smithi194_196924_object123000:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e3a210d:::benchmark_data_smithi194_196924_object132253:head# 2024-04-29T15:00:11.617 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e3c8b72:::benchmark_data_smithi194_196924_object122342:head# 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e3da5f1:::benchmark_data_smithi194_196924_object118623:head# 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e3e7d84:::benchmark_data_smithi194_196924_object102687:head# 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e3fb17b:::benchmark_data_smithi194_196924_object156914:head# 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e41aeba:::benchmark_data_smithi194_196924_object101687:head# 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e41c863:::benchmark_data_smithi194_196924_object140979:head# 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.618 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e44cdd1:::benchmark_data_smithi194_196924_object108730:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e455c8d:::benchmark_data_smithi194_196924_object96356:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e470889:::benchmark_data_smithi194_196924_object157929:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e477f04:::benchmark_data_smithi194_196924_object99665:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e47ca92:::benchmark_data_smithi194_196924_object135556:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e482732:::benchmark_data_smithi194_196924_object115056:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e4930ae:::benchmark_data_smithi194_196924_object145893:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e4a1add:::benchmark_data_smithi194_196924_object140852:head# 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.619 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e4c251f:::benchmark_data_smithi194_196924_object124936:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e4c8d92:::benchmark_data_smithi194_196924_object115664:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e4f1f53:::benchmark_data_smithi194_196924_object122376:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e537cf4:::benchmark_data_smithi194_196924_object106176:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e546d7e:::benchmark_data_smithi194_196924_object153424:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e54aed7:::benchmark_data_smithi194_196924_object145801:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e554bd2:::benchmark_data_smithi194_196924_object157289:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e55922e:::benchmark_data_smithi194_196924_object135402:head# 2024-04-29T15:00:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e55f5da:::benchmark_data_smithi194_196924_object100412:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5769e1:::benchmark_data_smithi194_196924_object153767:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e59dd80:::benchmark_data_smithi194_196924_object148677:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5be809:::benchmark_data_smithi194_196924_object146831:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5c7fad:::benchmark_data_smithi194_196924_object148400:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5cb7c5:::benchmark_data_smithi194_196924_object102701:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5d07e0:::benchmark_data_smithi194_196924_object105403:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5d5226:::benchmark_data_smithi194_196924_object115044:head# 2024-04-29T15:00:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5d5692:::benchmark_data_smithi194_196924_object125665:head# 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5d5a19:::benchmark_data_smithi194_196924_object144091:head# 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5d9847:::benchmark_data_smithi194_196924_object145016:head# 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5e2435:::benchmark_data_smithi194_196924_object130035:head# 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5f56f5:::benchmark_data_smithi194_196924_object147764:head# 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5f7cc0:::benchmark_data_smithi194_196924_object124897:head# 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.622 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5fc5e4:::benchmark_data_smithi194_196924_object139090:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e5fc6ec:::benchmark_data_smithi194_196924_object129812:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e63286e:::benchmark_data_smithi194_196924_object134677:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e647e12:::benchmark_data_smithi194_196924_object102819:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e65bf8e:::benchmark_data_smithi194_196924_object106212:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e6b63f6:::benchmark_data_smithi194_196924_object116402:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e6bb1b8:::benchmark_data_smithi194_196924_object126086:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e6c3c74:::benchmark_data_smithi194_196924_object98759:head# 2024-04-29T15:00:11.623 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e6f6162:::benchmark_data_smithi194_196924_object97301:head# 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e6fc834:::benchmark_data_smithi194_196924_object138387:head# 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e708e1e:::benchmark_data_smithi194_196924_object116034:head# 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e70c45a:::benchmark_data_smithi194_196924_object135843:head# 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e746840:::benchmark_data_smithi194_196924_object119854:head# 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e7783cd:::benchmark_data_smithi194_196924_object152430:head# 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.624 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e77cf28:::benchmark_data_smithi194_196924_object97643:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e77ee9b:::benchmark_data_smithi194_196924_object109004:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e789787:::benchmark_data_smithi194_196924_object111438:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e793c46:::benchmark_data_smithi194_196924_object105901:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e798c81:::benchmark_data_smithi194_196924_object105117:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e79a92b:::benchmark_data_smithi194_196924_object152083:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e79ccfb:::benchmark_data_smithi194_196924_object140567:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e7b04ff:::benchmark_data_smithi194_196924_object156699:head# 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.625 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e7c0f8b:::benchmark_data_smithi194_196924_object134005:head# 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e7ca61a:::benchmark_data_smithi194_196924_object110906:head# 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e7ea8e7:::benchmark_data_smithi194_196924_object127840:head# 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e7f8fb5:::benchmark_data_smithi194_196924_object115873:head# 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8046ec:::benchmark_data_smithi194_196924_object126088:head# 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e80f931:::benchmark_data_smithi194_196924_object147280:head# 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e839daf:::benchmark_data_smithi194_196924_object111560:head# 2024-04-29T15:00:11.626 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e848c67:::benchmark_data_smithi194_196924_object153838:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8572a7:::benchmark_data_smithi194_196924_object135485:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e85b51c:::benchmark_data_smithi194_196924_object116634:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8628c4:::benchmark_data_smithi194_196924_object103014:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8806e8:::benchmark_data_smithi194_196924_object136664:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8818f2:::benchmark_data_smithi194_196924_object115246:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8bfc60:::benchmark_data_smithi194_196924_object154629:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8c4db7:::benchmark_data_smithi194_196924_object118664:head# 2024-04-29T15:00:11.627 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8dce47:::benchmark_data_smithi194_196924_object95773:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8e6855:::benchmark_data_smithi194_196924_object104958:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8e7335:::benchmark_data_smithi194_196924_object129644:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e8ebd10:::benchmark_data_smithi194_196924_object109975:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9148ab:::benchmark_data_smithi194_196924_object143543:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e91bb9d:::benchmark_data_smithi194_196924_object159122:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e94f9b8:::benchmark_data_smithi194_196924_object130377:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e96d49c:::benchmark_data_smithi194_196924_object99645:head# 2024-04-29T15:00:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e98a57c:::benchmark_data_smithi194_196924_object96325:head# 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9a5fc3:::benchmark_data_smithi194_196924_object131542:head# 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9ac15d:::benchmark_data_smithi194_196924_object130857:head# 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9ad557:::benchmark_data_smithi194_196924_object134684:head# 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9bf66b:::benchmark_data_smithi194_196924_object102141:head# 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9cf51d:::benchmark_data_smithi194_196924_object117587:head# 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9d05e2:::benchmark_data_smithi194_196924_object129696:head# 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.629 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9f13d4:::benchmark_data_smithi194_196924_object112173:head# 2024-04-29T15:00:11.630 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.630 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4e9f625b:::benchmark_data_smithi194_196924_object141359:head# 2024-04-29T15:00:11.630 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.636 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea27255:::benchmark_data_smithi194_196924_object117171:head# 2024-04-29T15:00:11.636 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.636 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea394a8:::benchmark_data_smithi194_196924_object110619:head# 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea51e70:::benchmark_data_smithi194_196924_object111646:head# 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea7440b:::benchmark_data_smithi194_196924_object140989:head# 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea79307:::benchmark_data_smithi194_196924_object157986:head# 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea7ea7b:::benchmark_data_smithi194_196924_object129570:head# 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea82b63:::benchmark_data_smithi194_196924_object100407:head# 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea8779d:::benchmark_data_smithi194_196924_object155163:head# 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ea8f6dd:::benchmark_data_smithi194_196924_object139427:head# 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eab98e1:::benchmark_data_smithi194_196924_object153552:head# 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eac9fb4:::benchmark_data_smithi194_196924_object116700:head# 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eace7cd:::benchmark_data_smithi194_196924_object111695:head# 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eaea503:::benchmark_data_smithi194_196924_object155704:head# 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eaf1159:::benchmark_data_smithi194_196924_object130002:head# 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb066dc:::benchmark_data_smithi194_196924_object102838:head# 2024-04-29T15:00:11.638 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb0a1a7:::benchmark_data_smithi194_196924_object151831:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb1eb07:::benchmark_data_smithi194_196924_object104354:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb31094:::benchmark_data_smithi194_196924_object108444:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb33cfa:::benchmark_data_smithi194_196924_object159456:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb40b3d:::benchmark_data_smithi194_196924_object155902:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb49efa:::benchmark_data_smithi194_196924_object125742:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb61f0c:::benchmark_data_smithi194_196924_object108862:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb6330a:::benchmark_data_smithi194_196924_object153369:head# 2024-04-29T15:00:11.639 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb88921:::benchmark_data_smithi194_196924_object147191:head# 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb8ab54:::benchmark_data_smithi194_196924_object98450:head# 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb90952:::benchmark_data_smithi194_196924_object144932:head# 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eb97d98:::benchmark_data_smithi194_196924_object101023:head# 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eba4a69:::benchmark_data_smithi194_196924_object103691:head# 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ebb098a:::benchmark_data_smithi194_196924_object109774:head# 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ebc3ae8:::benchmark_data_smithi194_196924_object102181:head# 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.640 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ebc97bc:::benchmark_data_smithi194_196924_object150649:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ebcebce:::benchmark_data_smithi194_196924_object152325:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ebdab59:::benchmark_data_smithi194_196924_object110875:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ebf4961:::benchmark_data_smithi194_196924_object145271:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec0a2dc:::benchmark_data_smithi194_196924_object159547:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec2e7ee:::benchmark_data_smithi194_196924_object116070:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec2e9d7:::benchmark_data_smithi194_196924_object120843:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec344c7:::benchmark_data_smithi194_196924_object141734:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec4eacd:::benchmark_data_smithi194_196924_object118582:head# 2024-04-29T15:00:11.641 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec56041:::benchmark_data_smithi194_196924_object158194:head# 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec5b3c5:::benchmark_data_smithi194_196924_object131088:head# 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec68de7:::benchmark_data_smithi194_196924_object142657:head# 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec7c203:::benchmark_data_smithi194_196924_object122487:head# 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec81d90:::benchmark_data_smithi194_196924_object159802:head# 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec89d6c:::benchmark_data_smithi194_196924_object139744:head# 2024-04-29T15:00:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec8b9f8:::benchmark_data_smithi194_196924_object129477:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec8fc5b:::benchmark_data_smithi194_196924_object123678:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ec982c2:::benchmark_data_smithi194_196924_object98623:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eca1a83:::benchmark_data_smithi194_196924_object144012:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ecaaa6c:::benchmark_data_smithi194_196924_object138279:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ecad55c:::benchmark_data_smithi194_196924_object136667:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ecbb392:::benchmark_data_smithi194_196924_object100092:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ecc3367:::benchmark_data_smithi194_196924_object111961:head# 2024-04-29T15:00:11.643 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ecd9ba8:::benchmark_data_smithi194_196924_object131113:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ece1b61:::benchmark_data_smithi194_196924_object153428:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eced644:::benchmark_data_smithi194_196924_object122045:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ed013a8:::benchmark_data_smithi194_196924_object120654:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ed15990:::benchmark_data_smithi194_196924_object139098:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ed17470:::benchmark_data_smithi194_196924_object131734:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ed50866:::benchmark_data_smithi194_196924_object132936:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ed79a12:::benchmark_data_smithi194_196924_object97953:head# 2024-04-29T15:00:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ed7c93d:::benchmark_data_smithi194_196924_object125828:head# 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eda0e28:::benchmark_data_smithi194_196924_object98158:head# 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4edc1114:::benchmark_data_smithi194_196924_object129751:head# 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4edd47c1:::benchmark_data_smithi194_196924_object104852:head# 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4edfb342:::benchmark_data_smithi194_196924_object100507:head# 2024-04-29T15:00:11.645 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: pgmap v2919: 17 pgs: 17 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 979 op/s 2024-04-29T15:00:11.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:11 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.646 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee01f3a:::benchmark_data_smithi194_196924_object126009:head# 2024-04-29T15:00:11.646 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.647 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee042d1:::benchmark_data_smithi194_196924_object117737:head# 2024-04-29T15:00:11.647 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: pgmap v2919: 17 pgs: 17 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 979 op/s 2024-04-29T15:00:11.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:11 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee047ad:::benchmark_data_smithi194_196924_object135502:head# 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee06bb7:::benchmark_data_smithi194_196924_object139678:head# 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee12864:::benchmark_data_smithi194_196924_object141380:head# 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee2f11a:::benchmark_data_smithi194_196924_object134902:head# 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee31881:::benchmark_data_smithi194_196924_object141601:head# 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee41aab:::benchmark_data_smithi194_196924_object128650:head# 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee4fc43:::benchmark_data_smithi194_196924_object143627:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee612d1:::benchmark_data_smithi194_196924_object100466:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee6d14b:::benchmark_data_smithi194_196924_object152508:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee72ce0:::benchmark_data_smithi194_196924_object118812:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ee92a54:::benchmark_data_smithi194_196924_object105139:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eea57e5:::benchmark_data_smithi194_196924_object127067:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eebba1a:::benchmark_data_smithi194_196924_object151327:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eec433b:::benchmark_data_smithi194_196924_object114033:head# 2024-04-29T15:00:11.674 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eed8dd4:::benchmark_data_smithi194_196924_object113748:head# 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eedf9e7:::benchmark_data_smithi194_196924_object102818:head# 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eef6cb2:::benchmark_data_smithi194_196924_object132114:head# 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eefa933:::benchmark_data_smithi194_196924_object158601:head# 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eeffe83:::benchmark_data_smithi194_196924_object137168:head# 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ef21f56:::benchmark_data_smithi194_196924_object122307:head# 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ef3a412:::benchmark_data_smithi194_196924_object99403:head# 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ef6931e:::benchmark_data_smithi194_196924_object142513:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ef7149d:::benchmark_data_smithi194_196924_object114231:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efa2f88:::benchmark_data_smithi194_196924_object158973:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efa3599:::benchmark_data_smithi194_196924_object153792:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efb0046:::benchmark_data_smithi194_196924_object116805:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efb152d:::benchmark_data_smithi194_196924_object153517:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efb3368:::benchmark_data_smithi194_196924_object151388:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efcdcfd:::benchmark_data_smithi194_196924_object105822:head# 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efd49c3:::benchmark_data_smithi194_196924_object142188:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efd9ebd:::benchmark_data_smithi194_196924_object109119:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efe81cc:::benchmark_data_smithi194_196924_object128018:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eff319e:::benchmark_data_smithi194_196924_object102930:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4eff6915:::benchmark_data_smithi194_196924_object119869:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4effd7b4:::benchmark_data_smithi194_196924_object137450:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4effec28:::benchmark_data_smithi194_196924_object155051:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4efff612:::benchmark_data_smithi194_196924_object142796:head# 2024-04-29T15:00:11.677 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0037c8:::benchmark_data_smithi194_196924_object152378:head# 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0130cd:::benchmark_data_smithi194_196924_object112999:head# 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0147e4:::benchmark_data_smithi194_196924_object116325:head# 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f014812:::benchmark_data_smithi194_196924_object121500:head# 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f02d595:::benchmark_data_smithi194_196924_object150756:head# 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f036539:::benchmark_data_smithi194_196924_object124335:head# 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f03704d:::benchmark_data_smithi194_196924_object109805:head# 2024-04-29T15:00:11.678 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f044a56:::benchmark_data_smithi194_196924_object150488:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f057564:::benchmark_data_smithi194_196924_object115493:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f067abc:::benchmark_data_smithi194_196924_object98079:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f07f0e8:::benchmark_data_smithi194_196924_object102667:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f080b03:::benchmark_data_smithi194_196924_object97242:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f09ecf6:::benchmark_data_smithi194_196924_object146319:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0a4846:::benchmark_data_smithi194_196924_object117751:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0b11ca:::benchmark_data_smithi194_196924_object129150:head# 2024-04-29T15:00:11.679 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0bdeb4:::benchmark_data_smithi194_196924_object117075:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0cc654:::benchmark_data_smithi194_196924_object153174:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0cf13e:::benchmark_data_smithi194_196924_object146700:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0d8812:::benchmark_data_smithi194_196924_object130826:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f0ff84d:::benchmark_data_smithi194_196924_object154531:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f10121e:::benchmark_data_smithi194_196924_object98585:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f12a8be:::benchmark_data_smithi194_196924_object107314:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f12c428:::benchmark_data_smithi194_196924_object135237:head# 2024-04-29T15:00:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f12c7b7:::benchmark_data_smithi194_196924_object146428:head# 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f1307c1:::benchmark_data_smithi194_196924_object144268:head# 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f1565ae:::benchmark_data_smithi194_196924_object144466:head# 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f16e7be:::benchmark_data_smithi194_196924_object101165:head# 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f1741b7:::benchmark_data_smithi194_196924_object101336:head# 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f1aedb7:::benchmark_data_smithi194_196924_object120538:head# 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f1f5508:::benchmark_data_smithi194_196924_object134359:head# 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.681 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f1fc1f6:::benchmark_data_smithi194_196924_object139160:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f208792:::benchmark_data_smithi194_196924_object122556:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f211602:::benchmark_data_smithi194_196924_object139798:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f214a35:::benchmark_data_smithi194_196924_object125588:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f223f7a:::benchmark_data_smithi194_196924_object108123:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f231da9:::benchmark_data_smithi194_196924_object103742:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f233fa1:::benchmark_data_smithi194_196924_object159846:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f247ed1:::benchmark_data_smithi194_196924_object127057:head# 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.682 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f26f934:::benchmark_data_smithi194_196924_object107182:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f2708db:::benchmark_data_smithi194_196924_object103722:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f293ffd:::benchmark_data_smithi194_196924_object131538:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f29c36c:::benchmark_data_smithi194_196924_object145141:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f2aa3f5:::benchmark_data_smithi194_196924_object138643:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f2c0f32:::benchmark_data_smithi194_196924_object156869:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f2ec060:::benchmark_data_smithi194_196924_object106897:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f2eceab:::benchmark_data_smithi194_196924_object124383:head# 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3042bc:::benchmark_data_smithi194_196924_object116798:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f305ed3:::benchmark_data_smithi194_196924_object125709:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3186f0:::benchmark_data_smithi194_196924_object128558:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f321b72:::benchmark_data_smithi194_196924_object124958:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f332945:::benchmark_data_smithi194_196924_object121415:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f334dba:::benchmark_data_smithi194_196924_object126802:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f335016:::benchmark_data_smithi194_196924_object118813:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f33b965:::benchmark_data_smithi194_196924_object151761:head# 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.684 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3472d6:::benchmark_data_smithi194_196924_object150916:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f35b019:::benchmark_data_smithi194_196924_object120789:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f35e2cb:::benchmark_data_smithi194_196924_object127451:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f37e63c:::benchmark_data_smithi194_196924_object111734:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f37f8f1:::benchmark_data_smithi194_196924_object128522:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f39a06d:::benchmark_data_smithi194_196924_object115928:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3a61f8:::benchmark_data_smithi194_196924_object114335:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3b01fc:::benchmark_data_smithi194_196924_object104971:head# 2024-04-29T15:00:11.685 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3c1cca:::benchmark_data_smithi194_196924_object108300:head# 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3d856e:::benchmark_data_smithi194_196924_object112270:head# 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f3ea510:::benchmark_data_smithi194_196924_object117352:head# 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f408271:::benchmark_data_smithi194_196924_object147864:head# 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f410dfa:::benchmark_data_smithi194_196924_object110584:head# 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f43f283:::benchmark_data_smithi194_196924_object140593:head# 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f44064e:::benchmark_data_smithi194_196924_object157109:head# 2024-04-29T15:00:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f445ccc:::benchmark_data_smithi194_196924_object143995:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4530ff:::benchmark_data_smithi194_196924_object103843:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f462c97:::benchmark_data_smithi194_196924_object135512:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f464ea3:::benchmark_data_smithi194_196924_object146169:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4703a2:::benchmark_data_smithi194_196924_object159715:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f471f23:::benchmark_data_smithi194_196924_object115548:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f478a80:::benchmark_data_smithi194_196924_object97947:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f48b6a0:::benchmark_data_smithi194_196924_object101113:head# 2024-04-29T15:00:11.687 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f49a135:::benchmark_data_smithi194_196924_object150763:head# 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4c5a59:::benchmark_data_smithi194_196924_object122399:head# 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4cef85:::benchmark_data_smithi194_196924_object97440:head# 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4e05b3:::benchmark_data_smithi194_196924_object157723:head# 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4ef104:::benchmark_data_smithi194_196924_object117570:head# 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4f6725:::benchmark_data_smithi194_196924_object119519:head# 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f4fdc1f:::benchmark_data_smithi194_196924_object139839:head# 2024-04-29T15:00:11.688 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f516930:::benchmark_data_smithi194_196924_object145704:head# 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f518907:::benchmark_data_smithi194_196924_object154341:head# 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f51cbbc:::benchmark_data_smithi194_196924_object155651:head# 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5340b8:::benchmark_data_smithi194_196924_object97027:head# 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f53ae62:::benchmark_data_smithi194_196924_object151575:head# 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f53d6dd:::benchmark_data_smithi194_196924_object105934:head# 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f54b53a:::benchmark_data_smithi194_196924_object136101:head# 2024-04-29T15:00:11.689 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5638e2:::benchmark_data_smithi194_196924_object117852:head# 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f575abf:::benchmark_data_smithi194_196924_object102358:head# 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f595890:::benchmark_data_smithi194_196924_object105938:head# 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f597286:::benchmark_data_smithi194_196924_object96751:head# 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5ae8cf:::benchmark_data_smithi194_196924_object97485:head# 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5b0350:::benchmark_data_smithi194_196924_object122402:head# 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5b14a7:::benchmark_data_smithi194_196924_object155401:head# 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.690 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5b6827:::benchmark_data_smithi194_196924_object144122:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5b7a1d:::benchmark_data_smithi194_196924_object124215:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5bfb56:::benchmark_data_smithi194_196924_object105862:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5c74e6:::benchmark_data_smithi194_196924_object130135:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5ced61:::benchmark_data_smithi194_196924_object156882:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5d0eda:::benchmark_data_smithi194_196924_object150225:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5d476a:::benchmark_data_smithi194_196924_object106097:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5d71f7:::benchmark_data_smithi194_196924_object142638:head# 2024-04-29T15:00:11.691 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5d91d3:::benchmark_data_smithi194_196924_object95723:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f5f0ced:::benchmark_data_smithi194_196924_object96650:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6066d0:::benchmark_data_smithi194_196924_object114520:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f608f74:::benchmark_data_smithi194_196924_object112757:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f60de22:::benchmark_data_smithi194_196924_object155245:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6233f6:::benchmark_data_smithi194_196924_object117780:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f638667:::benchmark_data_smithi194_196924_object123155:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f642472:::benchmark_data_smithi194_196924_object153231:head# 2024-04-29T15:00:11.692 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f64ab3d:::benchmark_data_smithi194_196924_object145668:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f66d9ef:::benchmark_data_smithi194_196924_object127611:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f67eae2:::benchmark_data_smithi194_196924_object123723:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f692640:::benchmark_data_smithi194_196924_object111588:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6ae4bb:::benchmark_data_smithi194_196924_object118215:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6c79f1:::benchmark_data_smithi194_196924_object142921:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6cb195:::benchmark_data_smithi194_196924_object107153:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6cce49:::benchmark_data_smithi194_196924_object116605:head# 2024-04-29T15:00:11.693 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6cfe87:::benchmark_data_smithi194_196924_object109213:head# 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6dc379:::benchmark_data_smithi194_196924_object151773:head# 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6e89f2:::benchmark_data_smithi194_196924_object99304:head# 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f6fcc4b:::benchmark_data_smithi194_196924_object137163:head# 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f701334:::benchmark_data_smithi194_196924_object126176:head# 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f72187c:::benchmark_data_smithi194_196924_object105109:head# 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f73e368:::benchmark_data_smithi194_196924_object147833:head# 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f73ec24:::benchmark_data_smithi194_196924_object142692:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f751dd2:::benchmark_data_smithi194_196924_object99145:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f763efa:::benchmark_data_smithi194_196924_object138690:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f76c2a1:::benchmark_data_smithi194_196924_object134192:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f76c36e:::benchmark_data_smithi194_196924_object98568:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f7b4094:::benchmark_data_smithi194_196924_object125079:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f7bc34c:::benchmark_data_smithi194_196924_object135873:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f7d6467:::benchmark_data_smithi194_196924_object97906:head# 2024-04-29T15:00:11.695 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f7e2af7:::benchmark_data_smithi194_196924_object98906:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f7f5b58:::benchmark_data_smithi194_196924_object134962:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f8423d6:::benchmark_data_smithi194_196924_object118609:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f844534:::benchmark_data_smithi194_196924_object104994:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f85b2df:::benchmark_data_smithi194_196924_object96449:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f85d9df:::benchmark_data_smithi194_196924_object129474:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f85ebbf:::benchmark_data_smithi194_196924_object126257:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f861a30:::benchmark_data_smithi194_196924_object152021:head# 2024-04-29T15:00:11.696 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f879de8:::benchmark_data_smithi194_196924_object109576:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f88d56f:::benchmark_data_smithi194_196924_object128960:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f8a0aad:::benchmark_data_smithi194_196924_object130723:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f8ba089:::benchmark_data_smithi194_196924_object150475:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f8cc660:::benchmark_data_smithi194_196924_object124450:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f8e6171:::benchmark_data_smithi194_196924_object143964:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f8ea5cf:::benchmark_data_smithi194_196924_object141421:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f8f7974:::benchmark_data_smithi194_196924_object157077:head# 2024-04-29T15:00:11.697 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f90beb7:::benchmark_data_smithi194_196924_object121236:head# 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9110af:::benchmark_data_smithi194_196924_object131316:head# 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f91bbe1:::benchmark_data_smithi194_196924_object156531:head# 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f926fde:::benchmark_data_smithi194_196924_object131701:head# 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f93f6db:::benchmark_data_smithi194_196924_object111482:head# 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9570d4:::benchmark_data_smithi194_196924_object128210:head# 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f959ad0:::benchmark_data_smithi194_196924_object155108:head# 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.698 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f981dac:::benchmark_data_smithi194_196924_object109300:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f98c279:::benchmark_data_smithi194_196924_object126137:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f990858:::benchmark_data_smithi194_196924_object122067:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f99776a:::benchmark_data_smithi194_196924_object116968:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f99a0f4:::benchmark_data_smithi194_196924_object137929:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9a5dba:::benchmark_data_smithi194_196924_object119538:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9a8614:::benchmark_data_smithi194_196924_object153962:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9b015d:::benchmark_data_smithi194_196924_object103453:head# 2024-04-29T15:00:11.699 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9cd40d:::benchmark_data_smithi194_196924_object127216:head# 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9db4ee:::benchmark_data_smithi194_196924_object101802:head# 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9e9838:::benchmark_data_smithi194_196924_object107524:head# 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9f5c7b:::benchmark_data_smithi194_196924_object156180:head# 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.712 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4f9fd4cd:::benchmark_data_smithi194_196924_object133537:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa08521:::benchmark_data_smithi194_196924_object155389:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa26b0c:::benchmark_data_smithi194_196924_object101887:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa29ab3:::benchmark_data_smithi194_196924_object138685:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa29cc3:::benchmark_data_smithi194_196924_object129623:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa410f7:::benchmark_data_smithi194_196924_object155071:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa5f2ed:::benchmark_data_smithi194_196924_object133879:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa622ed:::benchmark_data_smithi194_196924_object125813:head# 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.713 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa664dc:::benchmark_data_smithi194_196924_object102662:head# 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa781bc:::benchmark_data_smithi194_196924_object140836:head# 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa8418c:::benchmark_data_smithi194_196924_object142042:head# 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa8c952:::benchmark_data_smithi194_196924_object100339:head# 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fa9b52e:::benchmark_data_smithi194_196924_object105327:head# 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fac8233:::benchmark_data_smithi194_196924_object149995:head# 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fad7f54:::benchmark_data_smithi194_196924_object127100:head# 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.714 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4faee032:::benchmark_data_smithi194_196924_object117415:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fafb52f:::benchmark_data_smithi194_196924_object110061:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb00f65:::benchmark_data_smithi194_196924_object107148:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb10af0:::benchmark_data_smithi194_196924_object140795:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb1477b:::benchmark_data_smithi194_196924_object119927:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb18ebb:::benchmark_data_smithi194_196924_object157071:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb3016c:::benchmark_data_smithi194_196924_object138908:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb3e6f2:::benchmark_data_smithi194_196924_object114779:head# 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.715 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb4a69d:::benchmark_data_smithi194_196924_object152011:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb6f75d:::benchmark_data_smithi194_196924_object154138:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb70a5c:::benchmark_data_smithi194_196924_object159967:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fb746a0:::benchmark_data_smithi194_196924_object152186:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fba793f:::benchmark_data_smithi194_196924_object157253:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fbbb5c7:::benchmark_data_smithi194_196924_object107896:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fbc25d9:::benchmark_data_smithi194_196924_object139665:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fbc9b10:::benchmark_data_smithi194_196924_object157223:head# 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.716 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fbe7461:::benchmark_data_smithi194_196924_object95917:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fbf992f:::benchmark_data_smithi194_196924_object111955:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc026e4:::benchmark_data_smithi194_196924_object98151:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc2b9da:::benchmark_data_smithi194_196924_object159559:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc2db7e:::benchmark_data_smithi194_196924_object119704:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc4ad3e:::benchmark_data_smithi194_196924_object124753:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc59513:::benchmark_data_smithi194_196924_object151872:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc6047f:::benchmark_data_smithi194_196924_object121301:head# 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.717 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc617b3:::benchmark_data_smithi194_196924_object118148:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fc8c78c:::benchmark_data_smithi194_196924_object131525:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fca440e:::benchmark_data_smithi194_196924_object158027:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fcc0be2:::benchmark_data_smithi194_196924_object139489:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fccfd15:::benchmark_data_smithi194_196924_object95718:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fcd5ec2:::benchmark_data_smithi194_196924_object155047:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fce350d:::benchmark_data_smithi194_196924_object140726:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fcf18a6:::benchmark_data_smithi194_196924_object154705:head# 2024-04-29T15:00:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd1160a:::benchmark_data_smithi194_196924_object158832:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd129dc:::benchmark_data_smithi194_196924_object106093:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd2af88:::benchmark_data_smithi194_196924_object123099:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd35621:::benchmark_data_smithi194_196924_object139535:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd43f36:::benchmark_data_smithi194_196924_object129776:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd4c29e:::benchmark_data_smithi194_196924_object96713:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd659db:::benchmark_data_smithi194_196924_object123633:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd735e6:::benchmark_data_smithi194_196924_object148942:head# 2024-04-29T15:00:11.719 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd8bbae:::benchmark_data_smithi194_196924_object99327:head# 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fd9e26c:::benchmark_data_smithi194_196924_object95701:head# 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fdb4121:::benchmark_data_smithi194_196924_object97284:head# 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fdb452c:::benchmark_data_smithi194_196924_object132140:head# 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fdf4ee5:::benchmark_data_smithi194_196924_object130525:head# 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fdf8270:::benchmark_data_smithi194_196924_object149515:head# 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe035c4:::benchmark_data_smithi194_196924_object112444:head# 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe15bf1:::benchmark_data_smithi194_196924_object115654:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe18403:::benchmark_data_smithi194_196924_object99125:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe1b980:::benchmark_data_smithi194_196924_object135409:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe44ca6:::benchmark_data_smithi194_196924_object127893:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe4ee75:::benchmark_data_smithi194_196924_object106496:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe56887:::benchmark_data_smithi194_196924_object156456:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe616d2:::benchmark_data_smithi194_196924_object121615:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe6c0a4:::benchmark_data_smithi194_196924_object118994:head# 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.721 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe7e6ee:::benchmark_data_smithi194_196924_object153322:head# 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe894f7:::benchmark_data_smithi194_196924_object123368:head# 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe8e399:::benchmark_data_smithi194_196924_object158785:head# 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe9e038:::benchmark_data_smithi194_196924_object125513:head# 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fe9e96d:::benchmark_data_smithi194_196924_object139696:head# 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4febd3da:::benchmark_data_smithi194_196924_object152757:head# 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fec9de8:::benchmark_data_smithi194_196924_object148003:head# 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.722 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4feca833:::benchmark_data_smithi194_196924_object96146:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fed50bd:::benchmark_data_smithi194_196924_object121589:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fed51bc:::benchmark_data_smithi194_196924_object132125:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fee1be5:::benchmark_data_smithi194_196924_object130150:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4fee4a09:::benchmark_data_smithi194_196924_object142684:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff0165d:::benchmark_data_smithi194_196924_object97875:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff0f37f:::benchmark_data_smithi194_196924_object148521:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff1f61b:::benchmark_data_smithi194_196924_object155880:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff21035:::benchmark_data_smithi194_196924_object114971:head# 2024-04-29T15:00:11.723 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff240a1:::benchmark_data_smithi194_196924_object157961:head# 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff2d6d6:::benchmark_data_smithi194_196924_object157878:head# 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff5e42f:::benchmark_data_smithi194_196924_object150684:head# 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff64e5d:::benchmark_data_smithi194_196924_object121201:head# 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff6ef0a:::benchmark_data_smithi194_196924_object118758:head# 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff7d812:::benchmark_data_smithi194_196924_object136892:head# 2024-04-29T15:00:11.724 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff7f6c2:::benchmark_data_smithi194_196924_object98185:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ff94091:::benchmark_data_smithi194_196924_object158625:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ffaa82a:::benchmark_data_smithi194_196924_object117385:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ffc3fe3:::benchmark_data_smithi194_196924_object143170:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ffd4245:::benchmark_data_smithi194_196924_object117169:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ffdbc38:::benchmark_data_smithi194_196924_object130719:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ffea7d1:::benchmark_data_smithi194_196924_object147394:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:Write #9:4ffed6a1:::benchmark_data_smithi194_196924_object121710:head# 2024-04-29T15:00:11.725 INFO:teuthology.orchestra.run.smithi040.stdout:snapset 0=[]:{} 2024-04-29T15:00:11.726 INFO:teuthology.orchestra.run.smithi040.stdout:write_pg epoch 1121 info 9.2( v 1121'15904 (1083'5900,1121'15904] local-lis/les=1065/1066 n=3974 ec=1065/1065 lis/c=1065/1065 les/c/f=1066/1066/0 sis=1065) 2024-04-29T15:00:11.774 INFO:teuthology.orchestra.run.smithi040.stdout:Import successful 2024-04-29T15:00:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:11.807 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:12.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:12.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:12.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:12.576 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:12 smithi194 ceph-mon[44089]: osdmap e1138: 12 total, 11 up, 7 in 2024-04-29T15:00:12.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:12.620 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:12.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:12.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:12 smithi040 ceph-mon[39581]: osdmap e1138: 12 total, 11 up, 7 in 2024-04-29T15:00:12.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:12.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:12.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:12.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:12 smithi121 ceph-mon[44533]: osdmap e1138: 12 total, 11 up, 7 in 2024-04-29T15:00:12.644 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:12.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:12.752 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:12.780 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:12.804 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:12.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:12.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:12.978 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:13.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:13.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:13.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:13.183 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:13.281 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:13 smithi121 ceph-mon[44533]: pgmap v2921: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 780 op/s 2024-04-29T15:00:13.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:13 smithi194 ceph-mon[44089]: pgmap v2921: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 780 op/s 2024-04-29T15:00:13.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:13 smithi040 ceph-mon[39581]: pgmap v2921: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 780 op/s 2024-04-29T15:00:13.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-29T15:00:14.224 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:14.541 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:14.541 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:14.541 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:14 smithi121 ceph-mon[44533]: osdmap e1139: 12 total, 11 up, 7 in 2024-04-29T15:00:14.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:14.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:14.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:14 smithi194 ceph-mon[44089]: osdmap e1139: 12 total, 11 up, 7 in 2024-04-29T15:00:14.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:14.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:14.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:14 smithi040 ceph-mon[39581]: osdmap e1139: 12 total, 11 up, 7 in 2024-04-29T15:00:15.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:15 smithi040 ceph-mon[39581]: pgmap v2923: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T15:00:15.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:15 smithi194 ceph-mon[44089]: pgmap v2923: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T15:00:15.640 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:15 smithi121 ceph-mon[44533]: pgmap v2923: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-29T15:00:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:16.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:16.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:16.227 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:16.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:16.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:16.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/exp.9.2.1 2024-04-29T15:00:16.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:16.569 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-29T15:00:16.569 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-29T15:00:16.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T15:00:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:16.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:16.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:16.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:16.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:16.845 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:16.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:16.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:17.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:17.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:17.125 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:17.252 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:17.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:17.348 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:17.349 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:17.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:17.386 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:17.407 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:17.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:17.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:17.558 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:17.573 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:17.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:17.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:17.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:17 smithi121 ceph-mon[44533]: 9.8 deep-scrub starts 2024-04-29T15:00:17.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:17 smithi121 ceph-mon[44533]: pgmap v2924: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T15:00:17.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:17 smithi121 ceph-mon[44533]: 9.9 deep-scrub starts 2024-04-29T15:00:17.715 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:17.745 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:17.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:17 smithi194 ceph-mon[44089]: 9.8 deep-scrub starts 2024-04-29T15:00:17.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:17 smithi194 ceph-mon[44089]: pgmap v2924: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T15:00:17.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:17 smithi194 ceph-mon[44089]: 9.9 deep-scrub starts 2024-04-29T15:00:17.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:17 smithi040 ceph-mon[39581]: 9.8 deep-scrub starts 2024-04-29T15:00:17.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:17 smithi040 ceph-mon[39581]: pgmap v2924: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-29T15:00:17.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:17 smithi040 ceph-mon[39581]: 9.9 deep-scrub starts 2024-04-29T15:00:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:17.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:17.949 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:17.976 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:17.997 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:18.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:18.103 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:18.132 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:18.154 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:18.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:18.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-29T15:00:18.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:18.288 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:18.414 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:18.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:18.518 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:18.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1027086421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:18.545 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T15:00:18.545 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:18.569 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:18.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:18.673 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:18.701 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:18.724 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:18.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:18.778 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:18.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1027086421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:18.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:18.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1027086421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:18.860 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:19.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T15:00:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:19.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:19.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:19.771 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:19 smithi194 ceph-mon[44089]: pgmap v2925: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 801 op/s 2024-04-29T15:00:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:19 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1027086421' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T15:00:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:19 smithi194 ceph-mon[44089]: osdmap e1140: 12 total, 11 up, 7 in 2024-04-29T15:00:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:19 smithi040 ceph-mon[39581]: pgmap v2925: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 801 op/s 2024-04-29T15:00:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:19 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1027086421' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T15:00:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:19 smithi040 ceph-mon[39581]: osdmap e1140: 12 total, 11 up, 7 in 2024-04-29T15:00:19.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:19.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:19.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:19 smithi121 ceph-mon[44533]: pgmap v2925: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 801 op/s 2024-04-29T15:00:19.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:19 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:19.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1027086421' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T15:00:19.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:19 smithi121 ceph-mon[44533]: osdmap e1140: 12 total, 11 up, 7 in 2024-04-29T15:00:20.550 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T15:00:20.553 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T15:00:20.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:20.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:20.838 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:20.865 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:20.886 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:20.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:20.990 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:21.041 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:21.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:21.145 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:21.173 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:21.197 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:21.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:21.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:21.391 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:21.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:21.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:21.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:21.607 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:21.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:21 smithi121 ceph-mon[44533]: pgmap v2927: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-04-29T15:00:21.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:21 smithi121 ceph-mon[44533]: from='client.18969 v1:172.21.15.40:0/3922138041' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:00:21.665 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:21.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:21.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:21.772 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T15:00:21.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:21 smithi040 ceph-mon[39581]: pgmap v2927: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-04-29T15:00:21.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:21 smithi040 ceph-mon[39581]: from='client.18969 v1:172.21.15.40:0/3922138041' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:00:21.820 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:21 smithi194 ceph-mon[44089]: pgmap v2927: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.2 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-04-29T15:00:21.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:21 smithi194 ceph-mon[44089]: from='client.18969 v1:172.21.15.40:0/3922138041' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:00:21.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:21.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:21.979 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:21.997 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2927,"stamp":"2024-04-29T15:00:20.450576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2333147704,"num_objects":35595,"num_object_clones":0,"num_object_copies":71190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35595,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":284596,"num_write_kb":10243131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24337,"num_bytes_recovered":1597669112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10937664,"kb_used_data":4641152,"kb_used_omap":663,"kb_used_meta":6295400,"kb_avail":645278400,"statfs":{"total":671965249536,"available":660765081600,"internally_reserved":0,"allocated":4752539648,"data_stored":4749826630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":679259,"internal_metadata":6446490277},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-507641856,"num_objects":-7746,"num_object_clones":0,"num_object_copies":-15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7746,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001283"},"pg_stats":[{"pgid":"9.f","version":"1140'17784","reported_seq":26283,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:19.359150+0000","last_change":"2024-04-29T15:00:03.987384+0000","last_active":"2024-04-29T15:00:19.359150+0000","last_peered":"2024-04-29T15:00:19.359150+0000","last_clean":"2024-04-29T15:00:19.359150+0000","last_became_active":"2024-04-29T14:59:46.623746+0000","last_became_peered":"2024-04-29T14:59:46.623746+0000","last_unstale":"2024-04-29T15:00:19.359150+0000","last_undegraded":"2024-04-29T15:00:19.359150+0000","last_fullsized":"2024-04-29T15:00:19.359150+0000","mapping_epoch":1132,"log_start":"1092'7700","ondisk_log_start":"1092'7700","created":1065,"last_epoch_clean":1133,"parent":"0.0","parent_split_bits":0,"last_scrub":"1136'16946","last_scrub_stamp":"2024-04-29T15:00:03.987306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:03.987306+0000","objects_scrubbed":3051,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:09.616446+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17784,"num_write_kb":639360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4091,"num_bytes_recovered":268107776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"1139'17804","reported_seq":18111,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-04-29T15:00:16.237354+0000","last_change":"2024-04-29T14:59:08.047926+0000","last_active":"2024-04-29T15:00:16.237354+0000","last_peered":"2024-04-29T15:00:16.237354+0000","last_clean":"2024-04-29T15:00:16.237354+0000","last_became_active":"2024-04-29T14:55:59.715887+0000","last_became_peered":"2024-04-29T14:55:59.715887+0000","last_unstale":"2024-04-29T15:00:16.237354+0000","last_undegraded":"2024-04-29T15:00:16.237354+0000","last_fullsized":"2024-04-29T15:00:16.237354+0000","mapping_epoch":1065,"log_start":"1092'7800","ondisk_log_start":"1092'7800","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'16079","last_scrub_stamp":"2024-04-29T14:59:08.047680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:08.047680+0000","objects_scrubbed":4065,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:19.260481+0000","scrub_duration":926,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151650304,"num_objects":2314,"num_object_clones":0,"num_object_copies":4628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17804,"num_write_kb":643776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"1139'17601","reported_seq":26086,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-04-29T15:00:16.257939+0000","last_change":"2024-04-29T14:59:47.686999+0000","last_active":"2024-04-29T15:00:16.257939+0000","last_peered":"2024-04-29T15:00:16.257939+0000","last_clean":"2024-04-29T15:00:16.257939+0000","last_became_active":"2024-04-29T14:59:47.627975+0000","last_became_peered":"2024-04-29T14:59:47.627975+0000","last_unstale":"2024-04-29T15:00:16.257939+0000","last_undegraded":"2024-04-29T15:00:16.257939+0000","last_fullsized":"2024-04-29T15:00:16.257939+0000","mapping_epoch":1133,"log_start":"1091'7600","ondisk_log_start":"1091'7600","created":1065,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15798","last_scrub_stamp":"2024-04-29T14:59:07.283663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:07.283663+0000","objects_scrubbed":4217,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:37.216810+0000","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155844632,"num_objects":2379,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17601,"num_write_kb":639297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4080,"num_bytes_recovered":267321368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"1140'17354","reported_seq":17721,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:18.635114+0000","last_change":"2024-04-29T15:00:04.665679+0000","last_active":"2024-04-29T15:00:18.635114+0000","last_peered":"2024-04-29T15:00:18.635114+0000","last_clean":"2024-04-29T15:00:18.635114+0000","last_became_active":"2024-04-29T14:55:59.711280+0000","last_became_peered":"2024-04-29T14:55:59.711280+0000","last_unstale":"2024-04-29T15:00:18.635114+0000","last_undegraded":"2024-04-29T15:00:18.635114+0000","last_fullsized":"2024-04-29T15:00:18.635114+0000","mapping_epoch":1065,"log_start":"1091'7300","ondisk_log_start":"1091'7300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1136'16539","last_scrub_stamp":"2024-04-29T15:00:04.665571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:04.665571+0000","objects_scrubbed":3019,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:12.526650+0000","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143523840,"num_objects":2190,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17354,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"1139'17815","reported_seq":18124,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-04-29T15:00:15.858436+0000","last_change":"2024-04-29T14:59:05.142714+0000","last_active":"2024-04-29T15:00:15.858436+0000","last_peered":"2024-04-29T15:00:15.858436+0000","last_clean":"2024-04-29T15:00:15.858436+0000","last_became_active":"2024-04-29T14:55:59.713356+0000","last_became_peered":"2024-04-29T14:55:59.713356+0000","last_unstale":"2024-04-29T15:00:15.858436+0000","last_undegraded":"2024-04-29T15:00:15.858436+0000","last_fullsized":"2024-04-29T15:00:15.858436+0000","mapping_epoch":1065,"log_start":"1092'7800","ondisk_log_start":"1092'7800","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15800","last_scrub_stamp":"2024-04-29T14:59:05.142554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:05.142554+0000","objects_scrubbed":4395,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:24.885893+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153944064,"num_objects":2349,"num_object_clones":0,"num_object_copies":4698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17815,"num_write_kb":645248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"1140'17923","reported_seq":26333,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:19.350081+0000","last_change":"2024-04-29T15:00:02.210504+0000","last_active":"2024-04-29T15:00:19.350081+0000","last_peered":"2024-04-29T15:00:19.350081+0000","last_clean":"2024-04-29T15:00:19.350081+0000","last_became_active":"2024-04-29T14:59:47.627794+0000","last_became_peered":"2024-04-29T14:59:47.627794+0000","last_unstale":"2024-04-29T15:00:19.350081+0000","last_undegraded":"2024-04-29T15:00:19.350081+0000","last_fullsized":"2024-04-29T15:00:19.350081+0000","mapping_epoch":1133,"log_start":"1092'7900","ondisk_log_start":"1092'7900","created":1065,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'16896","last_scrub_stamp":"2024-04-29T15:00:02.210271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:02.210271+0000","objects_scrubbed":3195,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:14.974147+0000","scrub_duration":996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140050432,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17923,"num_write_kb":641920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4048,"num_bytes_recovered":265289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"1140'18479","reported_seq":18807,"reported_epoch":1140,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T15:00:19.360073+0000","last_change":"2024-04-29T15:00:16.954348+0000","last_active":"2024-04-29T15:00:19.360073+0000","last_peered":"2024-04-29T15:00:19.360073+0000","last_clean":"2024-04-29T15:00:19.360073+0000","last_became_active":"2024-04-29T14:55:59.713461+0000","last_became_peered":"2024-04-29T14:55:59.713461+0000","last_unstale":"2024-04-29T15:00:19.360073+0000","last_undegraded":"2024-04-29T15:00:19.360073+0000","last_fullsized":"2024-04-29T15:00:19.360073+0000","mapping_epoch":1065,"log_start":"1093'8400","ondisk_log_start":"1093'8400","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'16271","last_scrub_stamp":"2024-04-29T14:59:14.920032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:14.920032+0000","objects_scrubbed":323,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142344192,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18156,"num_write_kb":650496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"1140'17513","reported_seq":17821,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:19.374642+0000","last_change":"2024-04-29T14:59:10.891018+0000","last_active":"2024-04-29T15:00:19.374642+0000","last_peered":"2024-04-29T15:00:19.374642+0000","last_clean":"2024-04-29T15:00:19.374642+0000","last_became_active":"2024-04-29T14:55:59.716261+0000","last_became_peered":"2024-04-29T14:55:59.716261+0000","last_unstale":"2024-04-29T15:00:19.374642+0000","last_undegraded":"2024-04-29T15:00:19.374642+0000","last_fullsized":"2024-04-29T15:00:19.374642+0000","mapping_epoch":1065,"log_start":"1091'7500","ondisk_log_start":"1091'7500","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'15659","last_scrub_stamp":"2024-04-29T14:59:10.890881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:10.890881+0000","objects_scrubbed":4009,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:28.847712+0000","scrub_duration":339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":4310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17513,"num_write_kb":629376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"1140'17584","reported_seq":17893,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:19.361460+0000","last_change":"2024-04-29T14:59:08.874426+0000","last_active":"2024-04-29T15:00:19.361460+0000","last_peered":"2024-04-29T15:00:19.361460+0000","last_clean":"2024-04-29T15:00:19.361460+0000","last_became_active":"2024-04-29T14:55:59.713183+0000","last_became_peered":"2024-04-29T14:55:59.713183+0000","last_unstale":"2024-04-29T15:00:19.361460+0000","last_undegraded":"2024-04-29T15:00:19.361460+0000","last_fullsized":"2024-04-29T15:00:19.361460+0000","mapping_epoch":1065,"log_start":"1091'7500","ondisk_log_start":"1091'7500","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'15741","last_scrub_stamp":"2024-04-29T14:59:08.874243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:08.874243+0000","objects_scrubbed":3988,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:35.288440+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":4264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17584,"num_write_kb":630912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"1140'17918","reported_seq":18229,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:19.299545+0000","last_change":"2024-04-29T14:59:06.601409+0000","last_active":"2024-04-29T15:00:19.299545+0000","last_peered":"2024-04-29T15:00:19.299545+0000","last_clean":"2024-04-29T15:00:19.299545+0000","last_became_active":"2024-04-29T14:55:59.714571+0000","last_became_peered":"2024-04-29T14:55:59.714571+0000","last_unstale":"2024-04-29T15:00:19.299545+0000","last_undegraded":"2024-04-29T15:00:19.299545+0000","last_fullsized":"2024-04-29T15:00:19.299545+0000","mapping_epoch":1065,"log_start":"1092'7900","ondisk_log_start":"1092'7900","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15931","last_scrub_stamp":"2024-04-29T14:59:06.601200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:06.601200+0000","objects_scrubbed":4247,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:25.586236+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144310272,"num_objects":2202,"num_object_clones":0,"num_object_copies":4404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17918,"num_write_kb":643840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"1139'17733","reported_seq":26323,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-04-29T15:00:16.485869+0000","last_change":"2024-04-29T14:59:57.283298+0000","last_active":"2024-04-29T15:00:16.485869+0000","last_peered":"2024-04-29T15:00:16.485869+0000","last_clean":"2024-04-29T15:00:16.485869+0000","last_became_active":"2024-04-29T14:59:47.620513+0000","last_became_peered":"2024-04-29T14:59:47.620513+0000","last_unstale":"2024-04-29T15:00:16.485869+0000","last_undegraded":"2024-04-29T15:00:16.485869+0000","last_fullsized":"2024-04-29T15:00:16.485869+0000","mapping_epoch":1133,"log_start":"1091'7700","ondisk_log_start":"1091'7700","created":1065,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'16567","last_scrub_stamp":"2024-04-29T14:59:57.283047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:57.283047+0000","objects_scrubbed":3502,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:13.569604+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151977984,"num_objects":2319,"num_object_clones":0,"num_object_copies":4638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17733,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4100,"num_bytes_recovered":268697600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"1140'17803","reported_seq":18115,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:18.685932+0000","last_change":"2024-04-29T14:59:08.454735+0000","last_active":"2024-04-29T15:00:18.685932+0000","last_peered":"2024-04-29T15:00:18.685932+0000","last_clean":"2024-04-29T15:00:18.685932+0000","last_became_active":"2024-04-29T14:55:59.713602+0000","last_became_peered":"2024-04-29T14:55:59.713602+0000","last_unstale":"2024-04-29T15:00:18.685932+0000","last_undegraded":"2024-04-29T15:00:18.685932+0000","last_fullsized":"2024-04-29T15:00:18.685932+0000","mapping_epoch":1065,"log_start":"1092'7800","ondisk_log_start":"1092'7800","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15941","last_scrub_stamp":"2024-04-29T14:59:08.454584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:08.454584+0000","objects_scrubbed":4075,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:35.256381+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143196160,"num_objects":2185,"num_object_clones":0,"num_object_copies":4370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17803,"num_write_kb":639616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"1140'17717","reported_seq":25976,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:18.654889+0000","last_change":"2024-04-29T15:00:03.502213+0000","last_active":"2024-04-29T15:00:18.654889+0000","last_peered":"2024-04-29T15:00:18.654889+0000","last_clean":"2024-04-29T15:00:18.654889+0000","last_became_active":"2024-04-29T14:59:46.622595+0000","last_became_peered":"2024-04-29T14:59:46.622595+0000","last_unstale":"2024-04-29T15:00:18.654889+0000","last_undegraded":"2024-04-29T15:00:18.654889+0000","last_fullsized":"2024-04-29T15:00:18.654889+0000","mapping_epoch":1132,"log_start":"1092'7700","ondisk_log_start":"1092'7700","created":1065,"last_epoch_clean":1133,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'16823","last_scrub_stamp":"2024-04-29T15:00:03.502007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:03.502007+0000","objects_scrubbed":3071,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:17.897513+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141623296,"num_objects":2161,"num_object_clones":0,"num_object_copies":4322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17717,"num_write_kb":636096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3974,"num_bytes_recovered":260440064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"1140'17923","reported_seq":18238,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-04-29T15:00:18.634500+0000","last_change":"2024-04-29T14:59:05.661282+0000","last_active":"2024-04-29T15:00:18.634500+0000","last_peered":"2024-04-29T15:00:18.634500+0000","last_clean":"2024-04-29T15:00:18.634500+0000","last_became_active":"2024-04-29T14:55:59.761514+0000","last_became_peered":"2024-04-29T14:55:59.761514+0000","last_unstale":"2024-04-29T15:00:18.634500+0000","last_undegraded":"2024-04-29T15:00:18.634500+0000","last_fullsized":"2024-04-29T15:00:18.634500+0000","mapping_epoch":1065,"log_start":"1092'7900","ondisk_log_start":"1092'7900","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15934","last_scrub_stamp":"2024-04-29T14:59:05.661010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:05.661010+0000","objects_scrubbed":4193,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:33.830638+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142409728,"num_objects":2173,"num_object_clones":0,"num_object_copies":4346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17923,"num_write_kb":643072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"1139'17781","reported_seq":26213,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-04-29T15:00:15.655915+0000","last_change":"2024-04-29T15:00:03.085644+0000","last_active":"2024-04-29T15:00:15.655915+0000","last_peered":"2024-04-29T15:00:15.655915+0000","last_clean":"2024-04-29T15:00:15.655915+0000","last_became_active":"2024-04-29T14:59:46.624041+0000","last_became_peered":"2024-04-29T14:59:46.624041+0000","last_unstale":"2024-04-29T15:00:15.655915+0000","last_undegraded":"2024-04-29T15:00:15.655915+0000","last_fullsized":"2024-04-29T15:00:15.655915+0000","mapping_epoch":1132,"log_start":"1091'7700","ondisk_log_start":"1091'7700","created":1065,"last_epoch_clean":1133,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'17026","last_scrub_stamp":"2024-04-29T15:00:03.084761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:03.084761+0000","objects_scrubbed":3113,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:19.540590+0000","scrub_duration":656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153681920,"num_objects":2345,"num_object_clones":0,"num_object_copies":4690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17781,"num_write_kb":644032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4030,"num_bytes_recovered":264110080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1140'18355","reported_seq":18690,"reported_epoch":1140,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T15:00:19.360116+0000","last_change":"2024-04-29T15:00:16.348875+0000","last_active":"2024-04-29T15:00:19.360116+0000","last_peered":"2024-04-29T15:00:19.360116+0000","last_clean":"2024-04-29T15:00:19.360116+0000","last_became_active":"2024-04-29T14:55:59.714624+0000","last_became_peered":"2024-04-29T14:55:59.714624+0000","last_unstale":"2024-04-29T15:00:19.360116+0000","last_undegraded":"2024-04-29T15:00:19.360116+0000","last_fullsized":"2024-04-29T15:00:19.360116+0000","mapping_epoch":1065,"log_start":"1093'8300","ondisk_log_start":"1093'8300","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'16087","last_scrub_stamp":"2024-04-29T14:59:10.526052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:10.526052+0000","objects_scrubbed":349,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18006,"num_write_kb":646080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2328,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-04-29T15:00:13.786292+0000","last_change":"2024-04-29T14:59:05.320473+0000","last_active":"2024-04-29T15:00:13.786292+0000","last_peered":"2024-04-29T15:00:13.786292+0000","last_clean":"2024-04-29T15:00:13.786292+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T15:00:13.786292+0000","last_undegraded":"2024-04-29T15:00:13.786292+0000","last_fullsized":"2024-04-29T15:00:13.786292+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:59:05.320342+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:59:05.320342+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:18.845818+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2332557336,"num_objects":35593,"num_object_clones":0,"num_object_copies":71186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35593,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284415,"num_write_kb":10240193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24323,"num_bytes_recovered":1593966616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6325415936,"data_stored":6325403720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160587,"ondisk_log_size":160587,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384329,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2155804,"kb_used_data":842064,"kb_used_omap":1,"kb_used_meta":1313726,"kb_avail":91589348,"statfs":{"total":95995035648,"available":93787492352,"internally_reserved":0,"allocated":862273536,"data_stored":861882754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1345255872},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":4591320039476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":753996,"kb_used_data":456260,"kb_used_omap":4,"kb_used_meta":297659,"kb_avail":92991156,"statfs":{"total":95995035648,"available":95222943744,"internally_reserved":0,"allocated":467210240,"data_stored":466825742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":304803520},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234644,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2146460,"kb_used_data":1036432,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":91598692,"statfs":{"total":95995035648,"available":93797060608,"internally_reserved":0,"allocated":1061306368,"data_stored":1060914246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1136523722},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528985,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1365332,"kb_used_data":301196,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92379820,"statfs":{"total":95995035648,"available":94596935680,"internally_reserved":0,"allocated":308424704,"data_stored":308036654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1089599942},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359739064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497497,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183083,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1388620,"kb_used_data":567492,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92356532,"statfs":{"total":95995035648,"available":94573088768,"internally_reserved":0,"allocated":581111808,"data_stored":580727310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":840825288},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986849,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1424716,"kb_used_data":302340,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92320436,"statfs":{"total":95995035648,"available":94536126464,"internally_reserved":0,"allocated":309596160,"data_stored":309211662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1702736,"kb_used_data":1135368,"kb_used_omap":651,"kb_used_meta":566580,"kb_avail":92042416,"statfs":{"total":95995035648,"available":94251433984,"internally_reserved":0,"allocated":1162616832,"data_stored":1162228262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":580178696},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1633226752,"data_stored":1633222680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":570687488,"data_stored":570687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":302055424,"data_stored":302055424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1140592640,"data_stored":1140588568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":304545792,"data_stored":304545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1059000320,"data_stored":1058996248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":460718080,"data_stored":460718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":854589440,"data_stored":854589440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T15:00:21.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T15:00:22.013 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:22.265 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:22.265 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:22.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:22.510 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:22.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:23.410 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:23.411 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:23.411 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:23.411 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:23.411 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:23.441 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:23.466 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:23.486 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:23.509 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:23.530 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:23.551 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:23.552 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:23.552 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:23.552 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:23.552 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:23.552 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: pgmap v2928: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-04-29T15:00:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: osdmap e1141: 12 total, 11 up, 7 in 2024-04-29T15:00:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3805330325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:23.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:23 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:23.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:23.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:23.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: pgmap v2928: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-04-29T15:00:23.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: osdmap e1141: 12 total, 11 up, 7 in 2024-04-29T15:00:23.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:23.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:23.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3805330325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:23.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:23 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: pgmap v2928: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-04-29T15:00:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: osdmap e1141: 12 total, 11 up, 7 in 2024-04-29T15:00:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3805330325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:23 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:23.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:23.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:23.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-29T15:00:24.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:24.432 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:24.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:24.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:24.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:24 smithi040 ceph-mon[39581]: osdmap e1142: 12 total, 11 up, 7 in 2024-04-29T15:00:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:24 smithi194 ceph-mon[44089]: osdmap e1142: 12 total, 11 up, 7 in 2024-04-29T15:00:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:24.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:24.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:24 smithi121 ceph-mon[44533]: osdmap e1142: 12 total, 11 up, 7 in 2024-04-29T15:00:25.009 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:25.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:25.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:25.116 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:25.146 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:25.171 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:25.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:25.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:25.319 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:25.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:25.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:25.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:25.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:25.571 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:25.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:25.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:25.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:25 smithi121 ceph-mon[44533]: pgmap v2931: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-04-29T15:00:25.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:25.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:25.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:25.756 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:25.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:25.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:25.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:25 smithi194 ceph-mon[44089]: pgmap v2931: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-04-29T15:00:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:25.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:25.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:25 smithi040 ceph-mon[39581]: pgmap v2931: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 935 op/s 2024-04-29T15:00:25.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:25.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:25.904 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:25.932 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:25.955 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:25.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:26.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T15:00:26.589 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:26 smithi121 ceph-mon[44533]: osdmap e1143: 12 total, 11 up, 7 in 2024-04-29T15:00:26.589 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4289304962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:26.589 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:26 smithi121 ceph-mon[44533]: from='client.28684 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:26.680 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T15:00:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255371 9.0 9.94076500 (undecoded) ondisk+retry+write+known_if_redirected e1129)", 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:36.578260+0000", 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.092523313000001, 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27234614400000001, 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255371 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578260+0000", 2024-04-29T15:00:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578260+0000", 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578264+0000", 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578273+0000", 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1030000000000008e-06 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578298+0000", 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4762000000000001e-05 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578312+0000", 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4270999999999999e-05 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578469+0000", 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015742600000000001 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578487+0000", 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8099000000000001e-05 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850566+0000", 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.272078608 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850607+0000", 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0601e-05 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255374 9.0 9.99e421a0 (undecoded) ondisk+retry+write+known_if_redirected e1129)", 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:36.578367+0000", 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.092417251000001, 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27247176699999998, 2024-04-29T15:00:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255374 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578367+0000", 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578367+0000", 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578368+0000", 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5069999999999999e-06 2024-04-29T15:00:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578375+0000", 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.379e-06 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578387+0000", 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1260999999999999e-05 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578396+0000", 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0990000000000005e-06 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578551+0000", 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155224 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578556+0000", 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7679999999999999e-06 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850826+0000", 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27226981099999997 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850838+0000", 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2717999999999999e-05 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255386 9.0 9.9fcd8d10 (undecoded) ondisk+retry+write+known_if_redirected e1129)", 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:36.578412+0000", 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.092371563999997, 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27245972499999999, 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255386 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578412+0000", 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578412+0000", 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578413+0000", 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.16e-07 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578417+0000", 2024-04-29T15:00:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9829999999999998e-06 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578423+0000", 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9540000000000001e-06 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578429+0000", 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7610000000000004e-06 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578568+0000", 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001394 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578572+0000", 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4800000000000001e-06 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850866+0000", 2024-04-29T15:00:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27229388100000002 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850872+0000", 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3500000000000002e-06 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255388 9.0 9.506447c0 (undecoded) ondisk+retry+write+known_if_redirected e1129)", 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:36.578435+0000", 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.092348723999997, 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27246863300000002, 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255388 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578435+0000", 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578435+0000", 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578436+0000", 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.44e-07 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578439+0000", 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9610000000000001e-06 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578443+0000", 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8280000000000002e-06 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578449+0000", 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0939999999999998e-06 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578584+0000", 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013506400000000001 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.578587+0000", 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4130000000000002e-06 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850897+0000", 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27231037899999999 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.850904+0000", 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2500000000000003e-06 2024-04-29T15:00:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255371 9.0 9:00a6e029:::benchmark_data_smithi194_196924_object95365:head [delete] snapc 0=[] RETRY=7 ondisk+retry+write+known_if_redirected e1133)", 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:46.623111+0000", 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.047672423999998, 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.047228689999999997, 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255371 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623111+0000", 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623111+0000", 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623116+0000", 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9119999999999997e-06 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623121+0000", 2024-04-29T15:00:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2100000000000001e-06 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623132+0000", 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0975000000000001e-05 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623140+0000", 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.204e-06 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623183+0000", 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3838999999999998e-05 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623190+0000", 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.99e-06 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.668017+0000", 2024-04-29T15:00:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.044826205000000001 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.670340+0000", 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002323355 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255374 9.0 9:05842799:::benchmark_data_smithi194_196924_object95368:head [delete] snapc 0=[] RETRY=7 ondisk+retry+write+known_if_redirected e1133)", 2024-04-29T15:00:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:46.623152+0000", 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.047631666999997, 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.047575392000000001, 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255374 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623152+0000", 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623152+0000", 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623153+0000", 2024-04-29T15:00:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6400000000000003e-07 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623155+0000", 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1660000000000001e-06 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623158+0000", 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5319999999999998e-06 2024-04-29T15:00:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623162+0000", 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5300000000000001e-06 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623214+0000", 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2417000000000003e-05 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623218+0000", 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.89e-06 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.670649+0000", 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.047430607 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.670727+0000", 2024-04-29T15:00:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8786000000000001e-05 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255386 9.0 9:08b1b3f9:::benchmark_data_smithi194_196924_object95380:head [delete] snapc 0=[] RETRY=6 ondisk+retry+write+known_if_redirected e1133)", 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:46.623175+0000", 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.047608523000001, 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.056857969000000001, 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255386 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623175+0000", 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623175+0000", 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623176+0000", 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9799999999999999e-07 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623177+0000", 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5409999999999999e-06 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623181+0000", 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4970000000000001e-06 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623185+0000", 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1239999999999998e-06 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623241+0000", 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6305999999999999e-05 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623244+0000", 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3400000000000002e-06 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.670755+0000", 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.047511048 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.671466+0000", 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00071086200000000004 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.671574+0000", 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010801799999999999 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.679890+0000", 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083160270000000001 2024-04-29T15:00:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.679966+0000", 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5644999999999998e-05 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.680007+0000", 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1368000000000001e-05 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.680033+0000", 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5795e-05 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255388 9.0 9:03e2260a:::benchmark_data_smithi194_196924_object95382:head [delete] snapc 0=[] RETRY=6 ondisk+retry+write+known_if_redirected e1133)", 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:46.623189+0000", 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.047594945999997, 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.056893547000000003, 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255388 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623189+0000", 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623189+0000", 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623189+0000", 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5699999999999998e-07 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623191+0000", 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3850000000000001e-06 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623193+0000", 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.847e-06 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623197+0000", 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2490000000000001e-06 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623258+0000", 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1334000000000006e-05 2024-04-29T15:00:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.623261+0000", 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3749999999999999e-06 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.676254+0000", 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.052993044000000003 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.676333+0000", 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8799000000000002e-05 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.676418+0000", 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4758000000000004e-05 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.679928+0000", 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035104720000000002 2024-04-29T15:00:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.680040+0000", 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111581 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.680063+0000", 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3229000000000001e-05 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:46.680082+0000", 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9117e-05 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28381.0:269879 9.1 e1135/1065 9:86eb974e:::benchmark_data_smithi194_196924_object109873:head v 1135'17037, mlcod=1135'17032)", 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:02.514095+0000", 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.156688892999998, 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029897158, 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.514095+0000", 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.514095+0000", 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.514096+0000", 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.252e-06 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.514103+0000", 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0110000000000003e-06 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.514112+0000", 2024-04-29T15:00:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.568e-06 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.514118+0000", 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6120000000000004e-06 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.518267+0000", 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041486400000000003 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.518286+0000", 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8635999999999999e-05 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.543930+0000", 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025644059 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.543992+0000", 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2379999999999996e-05 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28381.0:270043 9.1 e1135/1065 9:8f73f56c:::benchmark_data_smithi194_196924_object110037:head v 1135'17047, mlcod=1135'17045)", 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:02.722866+0000", 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.947917766, 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029119636000000001, 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722866+0000", 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722866+0000", 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722868+0000", 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0339999999999999e-06 2024-04-29T15:00:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722879+0000", 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0641e-05 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722893+0000", 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.451e-05 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722903+0000", 2024-04-29T15:00:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4290000000000007e-06 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722953+0000", 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0816999999999998e-05 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.722968+0000", 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5011000000000001e-05 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.751929+0000", 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028960981 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.751986+0000", 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6212999999999999e-05 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28381.0:270044 9.1 e1135/1065 9:88a27d51:::benchmark_data_smithi194_196924_object110038:head v 1135'17048, mlcod=1135'17045)", 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:02.724997+0000", 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.945786829999999, 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032145137999999997, 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.724997+0000", 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.724997+0000", 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.724999+0000", 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.685e-06 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.725009+0000", 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037999999999999e-05 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.725021+0000", 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2197e-05 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.725029+0000", 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4209999999999995e-06 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.726948+0000", 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00191902 2024-04-29T15:00:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.726970+0000", 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1526999999999999e-05 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.757095+0000", 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030124989000000001 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:02.757142+0000", 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7261e-05 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:271389 9.0 9:02ecfb32:::benchmark_data_smithi194_196924_object111383:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1136)", 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:04.242550+0000", 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.428233974000001, 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034641835000000003, 2024-04-29T15:00:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 271389 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.242550+0000", 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.242550+0000", 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.242550+0000", 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.84e-07 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.242552+0000", 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-29T15:00:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.242555+0000", 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7369999999999998e-06 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.242559+0000", 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.194e-06 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.247052+0000", 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044936059999999998 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.247142+0000", 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9956000000000004e-05 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.247222+0000", 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9269000000000003e-05 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.265473+0000", 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018251793999999998 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.277091+0000", 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011617736 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.277148+0000", 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6762000000000003e-05 2024-04-29T15:00:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:04.277192+0000", 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3751999999999997e-05 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272156 9.0 9:09dfb8cf:::benchmark_data_smithi194_196924_object112150:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1137)", 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.029153+0000", 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.641630983999999, 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032743857000000001, 2024-04-29T15:00:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272156 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.029153+0000", 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.029153+0000", 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.029154+0000", 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4750000000000001e-06 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.029160+0000", 2024-04-29T15:00:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.356e-06 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.029170+0000", 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0042e-05 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.029177+0000", 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.481e-06 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.037261+0000", 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080837370000000006 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.037339+0000", 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7962999999999999e-05 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.037409+0000", 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0270000000000003e-05 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.041306+0000", 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038967899999999998 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.061798+0000", 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020492059999999999 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.061858+0000", 2024-04-29T15:00:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0163e-05 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.061897+0000", 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8519999999999997e-05 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272528 9.0 9:0c0c67b9:::benchmark_data_smithi194_196924_object112522:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1137)", 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.403181+0000", 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.267603199, 2024-04-29T15:00:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035905156000000001, 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272528 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.403181+0000", 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.403181+0000", 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.403181+0000", 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4900000000000005e-07 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.403185+0000", 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4740000000000001e-06 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.403191+0000", 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2179999999999996e-06 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.403197+0000", 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6590000000000004e-06 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.410617+0000", 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074203050000000003 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.410699+0000", 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2052000000000007e-05 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.410764+0000", 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4698999999999998e-05 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.422395+0000", 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011631776 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439032+0000", 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016636396000000001 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439058+0000", 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6242999999999999e-05 2024-04-29T15:00:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439086+0000", 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7585000000000001e-05 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272531 9.0 9:02616055:::benchmark_data_smithi194_196924_object112525:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1137)", 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.404042+0000", 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.266741329999999, 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035087372999999998, 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272531 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.404042+0000", 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.404042+0000", 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.404044+0000", 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2020000000000001e-06 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.404048+0000", 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5650000000000003e-06 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.404057+0000", 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.404064+0000", 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9199999999999998e-06 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.414497+0000", 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010432983999999999 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.414603+0000", 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010688000000000001 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.414688+0000", 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4272999999999998e-05 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.422438+0000", 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077498159999999996 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439091+0000", 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016653244000000001 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439112+0000", 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1217999999999999e-05 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439130+0000", 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7827999999999999e-05 2024-04-29T15:00:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272538 9.0 9:08993b20:::benchmark_data_smithi194_196924_object112532:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1137)", 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.405630+0000", 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.265154190000001, 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033541194000000003, 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272538 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.405630+0000", 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.405630+0000", 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.405631+0000", 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3239999999999999e-06 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.405637+0000", 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5579999999999997e-06 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.405646+0000", 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2379999999999997e-06 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.405654+0000", 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8709999999999997e-06 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.418540+0000", 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012886448999999999 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.418657+0000", 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011697300000000001 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.418744+0000", 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7318999999999995e-05 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.422479+0000", 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037343390000000001 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439134+0000", 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016655426000000001 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439154+0000", 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.961e-05 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439171+0000", 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7087000000000001e-05 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272539 9.0 9:0bb580ae:::benchmark_data_smithi194_196924_object112533:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1137)", 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.409825+0000", 2024-04-29T15:00:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.260958574, 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029446766999999999, 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272539 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.409825+0000", 2024-04-29T15:00:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.409825+0000", 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.409827+0000", 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0030000000000001e-06 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.409834+0000", 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8850000000000001e-06 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.409845+0000", 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0631e-05 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.409851+0000", 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.46e-06 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.422509+0000", 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012657999 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.422580+0000", 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0403999999999999e-05 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.422646+0000", 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6173000000000005e-05 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439175+0000", 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016529023 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439219+0000", 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3807000000000003e-05 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439247+0000", 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8214000000000001e-05 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439272+0000", 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5168e-05 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272566 9.0 9:01b0ec83:::benchmark_data_smithi194_196924_object112560:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e1137)", 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.439315+0000", 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.231468976999999, 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033374704999999998, 2024-04-29T15:00:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272566 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439315+0000", 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439315+0000", 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439315+0000", 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5699999999999999e-07 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439316+0000", 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.221e-06 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439319+0000", 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439322+0000", 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439912+0000", 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058997400000000005 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439977+0000", 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5710000000000006e-05 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.440040+0000", 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2519000000000002e-05 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472490+0000", 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032449734000000001 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472592+0000", 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102128 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472644+0000", 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2778999999999997e-05 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472689+0000", 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5062000000000003e-05 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272567 9.0 9:0dc01780:::benchmark_data_smithi194_196924_object112561:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e1137)", 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.439324+0000", 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.231459478000001, 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033457293999999999, 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272567 2024-04-29T15:00:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439324+0000", 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439324+0000", 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439325+0000", 2024-04-29T15:00:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2899999999999999e-07 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439326+0000", 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.184e-06 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439328+0000", 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4899999999999999e-06 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.439332+0000", 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.641e-06 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.462671+0000", 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023339084999999999 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.462798+0000", 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012733500000000001 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.462915+0000", 2024-04-29T15:00:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117099 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472538+0000", 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00962293 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472697+0000", 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000158419 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472738+0000", 2024-04-29T15:00:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0881999999999999e-05 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472782+0000", 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3900000000000003e-05 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:272571 9.0 9:0c0e14f2:::benchmark_data_smithi194_196924_object112565:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1137)", 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T15:00:05.440012+0000", 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.230771736000001, 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032853668000000003, 2024-04-29T15:00:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 272571 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.440012+0000", 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.440012+0000", 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.440013+0000", 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.440015+0000", 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5919999999999999e-06 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.440022+0000", 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3960000000000003e-06 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.440026+0000", 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4649999999999996e-06 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.467431+0000", 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027404437 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.467546+0000", 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011544699999999999 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 8", 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.467656+0000", 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110003 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472579+0000", 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049230430000000002 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472790+0000", 2024-04-29T15:00:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021086999999999999 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472830+0000", 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9835000000000001e-05 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T15:00:05.472866+0000", 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5930000000000003e-05 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:26.795 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T15:00:26.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:26 smithi040 ceph-mon[39581]: osdmap e1143: 12 total, 11 up, 7 in 2024-04-29T15:00:26.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4289304962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:26.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:26 smithi040 ceph-mon[39581]: from='client.28684 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:27.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T15:00:27.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:26 smithi194 ceph-mon[44089]: osdmap e1143: 12 total, 11 up, 7 in 2024-04-29T15:00:27.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4289304962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:27.093 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:26 smithi194 ceph-mon[44089]: from='client.28684 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:27 smithi040 ceph-mon[39581]: pgmap v2933: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 662 op/s 2024-04-29T15:00:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:27 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:27 smithi040 ceph-mon[39581]: from='client.28684 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:27.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:27 smithi040 ceph-mon[39581]: osdmap e1144: 12 total, 11 up, 7 in 2024-04-29T15:00:27.879 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T15:00:27.882 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T15:00:27.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:27 smithi121 ceph-mon[44533]: pgmap v2933: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 662 op/s 2024-04-29T15:00:27.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:27 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:27.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:27 smithi121 ceph-mon[44533]: from='client.28684 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:27.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:27 smithi121 ceph-mon[44533]: osdmap e1144: 12 total, 11 up, 7 in 2024-04-29T15:00:27.918 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:27.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:28.024 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:28.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:27 smithi194 ceph-mon[44089]: pgmap v2933: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 662 op/s 2024-04-29T15:00:28.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:27 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:28.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:27 smithi194 ceph-mon[44089]: from='client.28684 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:28.052 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:27 smithi194 ceph-mon[44089]: osdmap e1144: 12 total, 11 up, 7 in 2024-04-29T15:00:28.054 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:28.079 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:28.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:28.183 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:28.220 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:28.245 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:28.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:28.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-29T15:00:28.350 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:28.382 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:28.406 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:28.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:28.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:28.629 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:28.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:28.740 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:28.750 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":2934,"stamp":"2024-04-29T15:00:26.576578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2060321336,"num_objects":31432,"num_object_clones":0,"num_object_copies":62864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31432,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":288759,"num_write_kb":10243131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24337,"num_bytes_recovered":1597669112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160888,"ondisk_log_size":160888,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10369020,"kb_used_data":4045104,"kb_used_omap":663,"kb_used_meta":6323176,"kb_avail":645847044,"statfs":{"total":671965249536,"available":661347373056,"internally_reserved":0,"allocated":4142186496,"data_stored":4139348448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":679259,"internal_metadata":6474932901},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-205586432,"num_objects":-3137,"num_object_clones":0,"num_object_copies":-6274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.125520"},"pg_stats":[{"pgid":"9.f","version":"1142'17970","reported_seq":26472,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:24.388375+0000","last_change":"2024-04-29T15:00:03.987384+0000","last_active":"2024-04-29T15:00:24.388375+0000","last_peered":"2024-04-29T15:00:24.388375+0000","last_clean":"2024-04-29T15:00:24.388375+0000","last_became_active":"2024-04-29T14:59:46.623746+0000","last_became_peered":"2024-04-29T14:59:46.623746+0000","last_unstale":"2024-04-29T15:00:24.388375+0000","last_undegraded":"2024-04-29T15:00:24.388375+0000","last_fullsized":"2024-04-29T15:00:24.388375+0000","mapping_epoch":1132,"log_start":"1092'7900","ondisk_log_start":"1092'7900","created":1065,"last_epoch_clean":1133,"parent":"0.0","parent_split_bits":0,"last_scrub":"1136'16946","last_scrub_stamp":"2024-04-29T15:00:03.987306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:03.987306+0000","objects_scrubbed":3051,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:09.616446+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131727360,"num_objects":2010,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17970,"num_write_kb":639360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4091,"num_bytes_recovered":268107776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"1143'18216","reported_seq":18531,"reported_epoch":1143,"state":"active+clean","last_fresh":"2024-04-29T15:00:26.382825+0000","last_change":"2024-04-29T14:59:08.047926+0000","last_active":"2024-04-29T15:00:26.382825+0000","last_peered":"2024-04-29T15:00:26.382825+0000","last_clean":"2024-04-29T15:00:26.382825+0000","last_became_active":"2024-04-29T14:55:59.715887+0000","last_became_peered":"2024-04-29T14:55:59.715887+0000","last_unstale":"2024-04-29T15:00:26.382825+0000","last_undegraded":"2024-04-29T15:00:26.382825+0000","last_fullsized":"2024-04-29T15:00:26.382825+0000","mapping_epoch":1065,"log_start":"1093'8200","ondisk_log_start":"1093'8200","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'16079","last_scrub_stamp":"2024-04-29T14:59:08.047680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:08.047680+0000","objects_scrubbed":4065,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:30.220983+0000","scrub_duration":926,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124649472,"num_objects":1902,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18216,"num_write_kb":643776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"1143'18025","reported_seq":26518,"reported_epoch":1143,"state":"active+clean","last_fresh":"2024-04-29T15:00:26.331731+0000","last_change":"2024-04-29T14:59:47.686999+0000","last_active":"2024-04-29T15:00:26.331731+0000","last_peered":"2024-04-29T15:00:26.331731+0000","last_clean":"2024-04-29T15:00:26.331731+0000","last_became_active":"2024-04-29T14:59:47.627975+0000","last_became_peered":"2024-04-29T14:59:47.627975+0000","last_unstale":"2024-04-29T15:00:26.331731+0000","last_undegraded":"2024-04-29T15:00:26.331731+0000","last_fullsized":"2024-04-29T15:00:26.331731+0000","mapping_epoch":1133,"log_start":"1092'8000","ondisk_log_start":"1092'8000","created":1065,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15798","last_scrub_stamp":"2024-04-29T14:59:07.283663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:07.283663+0000","objects_scrubbed":4217,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:37.216810+0000","scrub_duration":1182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128057368,"num_objects":1955,"num_object_clones":0,"num_object_copies":3910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18025,"num_write_kb":639297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4080,"num_bytes_recovered":267321368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"1142'17562","reported_seq":17932,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:23.690468+0000","last_change":"2024-04-29T15:00:04.665679+0000","last_active":"2024-04-29T15:00:23.690468+0000","last_peered":"2024-04-29T15:00:23.690468+0000","last_clean":"2024-04-29T15:00:23.690468+0000","last_became_active":"2024-04-29T14:55:59.711280+0000","last_became_peered":"2024-04-29T14:55:59.711280+0000","last_unstale":"2024-04-29T15:00:23.690468+0000","last_undegraded":"2024-04-29T15:00:23.690468+0000","last_fullsized":"2024-04-29T15:00:23.690468+0000","mapping_epoch":1065,"log_start":"1092'7500","ondisk_log_start":"1092'7500","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1136'16539","last_scrub_stamp":"2024-04-29T15:00:04.665571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:04.665571+0000","objects_scrubbed":3019,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:12.526650+0000","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129892352,"num_objects":1982,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17562,"num_write_kb":625408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"1143'18192","reported_seq":18509,"reported_epoch":1143,"state":"active+clean","last_fresh":"2024-04-29T15:00:25.856261+0000","last_change":"2024-04-29T14:59:05.142714+0000","last_active":"2024-04-29T15:00:25.856261+0000","last_peered":"2024-04-29T15:00:25.856261+0000","last_clean":"2024-04-29T15:00:25.856261+0000","last_became_active":"2024-04-29T14:55:59.713356+0000","last_became_peered":"2024-04-29T14:55:59.713356+0000","last_unstale":"2024-04-29T15:00:25.856261+0000","last_undegraded":"2024-04-29T15:00:25.856261+0000","last_fullsized":"2024-04-29T15:00:25.856261+0000","mapping_epoch":1065,"log_start":"1092'8100","ondisk_log_start":"1092'8100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15800","last_scrub_stamp":"2024-04-29T14:59:05.142554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:05.142554+0000","objects_scrubbed":4395,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:30.381724+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129236992,"num_objects":1972,"num_object_clones":0,"num_object_copies":3944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18192,"num_write_kb":645248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"1142'18112","reported_seq":26526,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:24.338332+0000","last_change":"2024-04-29T15:00:02.210504+0000","last_active":"2024-04-29T15:00:24.338332+0000","last_peered":"2024-04-29T15:00:24.338332+0000","last_clean":"2024-04-29T15:00:24.338332+0000","last_became_active":"2024-04-29T14:59:47.627794+0000","last_became_peered":"2024-04-29T14:59:47.627794+0000","last_unstale":"2024-04-29T15:00:24.338332+0000","last_undegraded":"2024-04-29T15:00:24.338332+0000","last_fullsized":"2024-04-29T15:00:24.338332+0000","mapping_epoch":1133,"log_start":"1092'8100","ondisk_log_start":"1092'8100","created":1065,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'16896","last_scrub_stamp":"2024-04-29T15:00:02.210271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:02.210271+0000","objects_scrubbed":3195,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:14.974147+0000","scrub_duration":996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127664128,"num_objects":1948,"num_object_clones":0,"num_object_copies":3896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18112,"num_write_kb":641920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4048,"num_bytes_recovered":265289728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"1142'19386","reported_seq":19719,"reported_epoch":1142,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T15:00:24.446115+0000","last_change":"2024-04-29T15:00:16.954348+0000","last_active":"2024-04-29T15:00:24.446115+0000","last_peered":"2024-04-29T15:00:24.446115+0000","last_clean":"2024-04-29T15:00:24.446115+0000","last_became_active":"2024-04-29T14:55:59.713461+0000","last_became_peered":"2024-04-29T14:55:59.713461+0000","last_unstale":"2024-04-29T15:00:24.446115+0000","last_undegraded":"2024-04-29T15:00:24.446115+0000","last_fullsized":"2024-04-29T15:00:24.446115+0000","mapping_epoch":1065,"log_start":"1096'9301","ondisk_log_start":"1096'9301","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'16271","last_scrub_stamp":"2024-04-29T14:59:14.920032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:14.920032+0000","objects_scrubbed":1023,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 7s","scrub_duration":336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128778240,"num_objects":1965,"num_object_clones":0,"num_object_copies":3930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18363,"num_write_kb":650496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"1142'17697","reported_seq":18009,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:24.409151+0000","last_change":"2024-04-29T14:59:10.891018+0000","last_active":"2024-04-29T15:00:24.409151+0000","last_peered":"2024-04-29T15:00:24.409151+0000","last_clean":"2024-04-29T15:00:24.409151+0000","last_became_active":"2024-04-29T14:55:59.716261+0000","last_became_peered":"2024-04-29T14:55:59.716261+0000","last_unstale":"2024-04-29T15:00:24.409151+0000","last_undegraded":"2024-04-29T15:00:24.409151+0000","last_fullsized":"2024-04-29T15:00:24.409151+0000","mapping_epoch":1065,"log_start":"1092'7600","ondisk_log_start":"1092'7600","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'15659","last_scrub_stamp":"2024-04-29T14:59:10.890881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:10.890881+0000","objects_scrubbed":4009,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:28.847712+0000","scrub_duration":339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129171456,"num_objects":1971,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17697,"num_write_kb":629376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"1142'17759","reported_seq":18074,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:24.386834+0000","last_change":"2024-04-29T14:59:08.874426+0000","last_active":"2024-04-29T15:00:24.386834+0000","last_peered":"2024-04-29T15:00:24.386834+0000","last_clean":"2024-04-29T15:00:24.386834+0000","last_became_active":"2024-04-29T14:55:59.713183+0000","last_became_peered":"2024-04-29T14:55:59.713183+0000","last_unstale":"2024-04-29T15:00:24.386834+0000","last_undegraded":"2024-04-29T15:00:24.386834+0000","last_fullsized":"2024-04-29T15:00:24.386834+0000","mapping_epoch":1065,"log_start":"1092'7700","ondisk_log_start":"1092'7700","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'15741","last_scrub_stamp":"2024-04-29T14:59:08.874243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:08.874243+0000","objects_scrubbed":3988,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:35.288440+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128253952,"num_objects":1957,"num_object_clones":0,"num_object_copies":3914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17759,"num_write_kb":630912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"1142'18117","reported_seq":18433,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:24.376175+0000","last_change":"2024-04-29T14:59:06.601409+0000","last_active":"2024-04-29T15:00:24.376175+0000","last_peered":"2024-04-29T15:00:24.376175+0000","last_clean":"2024-04-29T15:00:24.376175+0000","last_became_active":"2024-04-29T14:55:59.714571+0000","last_became_peered":"2024-04-29T14:55:59.714571+0000","last_unstale":"2024-04-29T15:00:24.376175+0000","last_undegraded":"2024-04-29T15:00:24.376175+0000","last_fullsized":"2024-04-29T15:00:24.376175+0000","mapping_epoch":1065,"log_start":"1092'8100","ondisk_log_start":"1092'8100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15931","last_scrub_stamp":"2024-04-29T14:59:06.601200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:06.601200+0000","objects_scrubbed":4247,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:25.586236+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131268608,"num_objects":2003,"num_object_clones":0,"num_object_copies":4006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18117,"num_write_kb":643840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"1143'18117","reported_seq":26715,"reported_epoch":1143,"state":"active+clean","last_fresh":"2024-04-29T15:00:26.547561+0000","last_change":"2024-04-29T14:59:57.283298+0000","last_active":"2024-04-29T15:00:26.547561+0000","last_peered":"2024-04-29T15:00:26.547561+0000","last_clean":"2024-04-29T15:00:26.547561+0000","last_became_active":"2024-04-29T14:59:47.620513+0000","last_became_peered":"2024-04-29T14:59:47.620513+0000","last_unstale":"2024-04-29T15:00:26.547561+0000","last_undegraded":"2024-04-29T15:00:26.547561+0000","last_fullsized":"2024-04-29T15:00:26.547561+0000","mapping_epoch":1133,"log_start":"1093'8100","ondisk_log_start":"1093'8100","created":1065,"last_epoch_clean":1134,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'16567","last_scrub_stamp":"2024-04-29T14:59:57.283047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:57.283047+0000","objects_scrubbed":3502,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:13.569604+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126812160,"num_objects":1935,"num_object_clones":0,"num_object_copies":3870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18117,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4100,"num_bytes_recovered":268697600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"1142'18004","reported_seq":18321,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:23.694259+0000","last_change":"2024-04-29T14:59:08.454735+0000","last_active":"2024-04-29T15:00:23.694259+0000","last_peered":"2024-04-29T15:00:23.694259+0000","last_clean":"2024-04-29T15:00:23.694259+0000","last_became_active":"2024-04-29T14:55:59.713602+0000","last_became_peered":"2024-04-29T14:55:59.713602+0000","last_unstale":"2024-04-29T15:00:23.694259+0000","last_undegraded":"2024-04-29T15:00:23.694259+0000","last_fullsized":"2024-04-29T15:00:23.694259+0000","mapping_epoch":1065,"log_start":"1092'8000","ondisk_log_start":"1092'8000","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15941","last_scrub_stamp":"2024-04-29T14:59:08.454584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:08.454584+0000","objects_scrubbed":4075,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:35.256381+0000","scrub_duration":779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":1984,"num_object_clones":0,"num_object_copies":3968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18004,"num_write_kb":639616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"1142'17887","reported_seq":26150,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:23.673759+0000","last_change":"2024-04-29T15:00:03.502213+0000","last_active":"2024-04-29T15:00:23.673759+0000","last_peered":"2024-04-29T15:00:23.673759+0000","last_clean":"2024-04-29T15:00:23.673759+0000","last_became_active":"2024-04-29T14:59:46.622595+0000","last_became_peered":"2024-04-29T14:59:46.622595+0000","last_unstale":"2024-04-29T15:00:23.673759+0000","last_undegraded":"2024-04-29T15:00:23.673759+0000","last_fullsized":"2024-04-29T15:00:23.673759+0000","mapping_epoch":1132,"log_start":"1092'7800","ondisk_log_start":"1092'7800","created":1065,"last_epoch_clean":1133,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'16823","last_scrub_stamp":"2024-04-29T15:00:03.502007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:03.502007+0000","objects_scrubbed":3071,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:17.897513+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130482176,"num_objects":1991,"num_object_clones":0,"num_object_copies":3982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17887,"num_write_kb":636096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3974,"num_bytes_recovered":260440064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"1142'18130","reported_seq":18449,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-04-29T15:00:23.685017+0000","last_change":"2024-04-29T14:59:05.661282+0000","last_active":"2024-04-29T15:00:23.685017+0000","last_peered":"2024-04-29T15:00:23.685017+0000","last_clean":"2024-04-29T15:00:23.685017+0000","last_became_active":"2024-04-29T14:55:59.761514+0000","last_became_peered":"2024-04-29T14:55:59.761514+0000","last_unstale":"2024-04-29T15:00:23.685017+0000","last_undegraded":"2024-04-29T15:00:23.685017+0000","last_fullsized":"2024-04-29T15:00:23.685017+0000","mapping_epoch":1065,"log_start":"1092'8100","ondisk_log_start":"1092'8100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1121'15934","last_scrub_stamp":"2024-04-29T14:59:05.661010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:05.661010+0000","objects_scrubbed":4193,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:33.830638+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128843776,"num_objects":1966,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18130,"num_write_kb":643072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"1143'18203","reported_seq":26643,"reported_epoch":1143,"state":"active+clean","last_fresh":"2024-04-29T15:00:25.664954+0000","last_change":"2024-04-29T15:00:03.085644+0000","last_active":"2024-04-29T15:00:25.664954+0000","last_peered":"2024-04-29T15:00:25.664954+0000","last_clean":"2024-04-29T15:00:25.664954+0000","last_became_active":"2024-04-29T14:59:46.624041+0000","last_became_peered":"2024-04-29T14:59:46.624041+0000","last_unstale":"2024-04-29T15:00:25.664954+0000","last_undegraded":"2024-04-29T15:00:25.664954+0000","last_fullsized":"2024-04-29T15:00:25.664954+0000","mapping_epoch":1132,"log_start":"1093'8200","ondisk_log_start":"1093'8200","created":1065,"last_epoch_clean":1133,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'17026","last_scrub_stamp":"2024-04-29T15:00:03.084761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T15:00:03.084761+0000","objects_scrubbed":3113,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:19.540590+0000","scrub_duration":656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126025728,"num_objects":1923,"num_object_clones":0,"num_object_copies":3846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18203,"num_write_kb":644032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4030,"num_bytes_recovered":264110080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1142'19173","reported_seq":19536,"reported_epoch":1142,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-29T15:00:24.371709+0000","last_change":"2024-04-29T15:00:16.348875+0000","last_active":"2024-04-29T15:00:24.371709+0000","last_peered":"2024-04-29T15:00:24.371709+0000","last_clean":"2024-04-29T15:00:24.371709+0000","last_became_active":"2024-04-29T14:55:59.714624+0000","last_became_peered":"2024-04-29T14:55:59.714624+0000","last_unstale":"2024-04-29T15:00:24.371709+0000","last_undegraded":"2024-04-29T15:00:24.371709+0000","last_fullsized":"2024-04-29T15:00:24.371709+0000","mapping_epoch":1065,"log_start":"1096'9100","ondisk_log_start":"1096'9100","created":1065,"last_epoch_clean":1066,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'16087","last_scrub_stamp":"2024-04-29T14:59:10.526052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-29T14:55:58.697144+0000","last_clean_scrub_stamp":"2024-04-29T14:59:10.526052+0000","objects_scrubbed":949,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128843776,"num_objects":1966,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18224,"num_write_kb":646080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"1003'139","reported_seq":2336,"reported_epoch":1143,"state":"active+clean","last_fresh":"2024-04-29T15:00:25.570376+0000","last_change":"2024-04-29T14:59:05.320473+0000","last_active":"2024-04-29T15:00:25.570376+0000","last_peered":"2024-04-29T15:00:25.570376+0000","last_clean":"2024-04-29T15:00:25.570376+0000","last_became_active":"2024-04-29T14:44:49.524155+0000","last_became_peered":"2024-04-29T14:44:49.524155+0000","last_unstale":"2024-04-29T15:00:25.570376+0000","last_undegraded":"2024-04-29T15:00:25.570376+0000","last_fullsized":"2024-04-29T15:00:25.570376+0000","mapping_epoch":842,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":843,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T14:59:05.320342+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T14:51:31.273868+0000","last_clean_scrub_stamp":"2024-04-29T14:59:05.320342+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:30.220945+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2059730968,"num_objects":31430,"num_object_clones":0,"num_object_copies":62860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31430,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":288578,"num_write_kb":10240193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24323,"num_bytes_recovered":1593966616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5730611200,"data_stored":5730598984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":0,"seq":377957122691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384330,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091444,"kb_used_data":767976,"kb_used_omap":1,"kb_used_meta":1323454,"kb_avail":91653708,"statfs":{"total":95995035648,"available":93853396992,"internally_reserved":0,"allocated":786407424,"data_stored":786004164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1355217344},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":4591320039477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423271,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682608,"kb_used_data":379368,"kb_used_omap":4,"kb_used_meta":303227,"kb_avail":93062544,"statfs":{"total":95995035648,"available":95296045056,"internally_reserved":0,"allocated":388472832,"data_stored":388069592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":310505152},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234646,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1981256,"kb_used_data":858880,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":91763896,"statfs":{"total":95995035648,"available":93966229504,"internally_reserved":0,"allocated":879493120,"data_stored":879075992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1149303242},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528987,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1315324,"kb_used_data":247088,"kb_used_omap":1,"kb_used_meta":1068222,"kb_avail":92429828,"statfs":{"total":95995035648,"available":94648143872,"internally_reserved":0,"allocated":253018112,"data_stored":252611320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1093859782},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359739065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1116691497498,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183084,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337832,"kb_used_data":512540,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92407320,"statfs":{"total":95995035648,"available":94625095680,"internally_reserved":0,"allocated":524840960,"data_stored":524443934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":845085128},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367744,"kb_used_data":245364,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92377408,"statfs":{"total":95995035648,"available":94594465792,"internally_reserved":0,"allocated":251252736,"data_stored":250843232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1592812,"kb_used_data":1033888,"kb_used_omap":651,"kb_used_meta":558260,"kb_avail":92152340,"statfs":{"total":95995035648,"available":94363996160,"internally_reserved":0,"allocated":1058701312,"data_stored":1058300214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":571659016},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1633226752,"data_stored":1633222680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":519176192,"data_stored":519176192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":250281984,"data_stored":250281984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1038684160,"data_stored":1038680088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":249757696,"data_stored":249757696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":875040768,"data_stored":875036696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":383713280,"data_stored":383713280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":780730368,"data_stored":780730368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T15:00:28.751 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-29T15:00:28.751 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-29T15:00:28.758 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:28.767 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:28.789 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:28.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:28.804 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:28.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:28 smithi121 ceph-mon[44533]: from='client.18987 v1:172.21.15.40:0/3645206098' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:00:28.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:28 smithi121 ceph-mon[44533]: pgmap v2935: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 530 op/s 2024-04-29T15:00:28.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:28.927 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:29.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:28 smithi040 ceph-mon[39581]: from='client.18987 v1:172.21.15.40:0/3645206098' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:00:29.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:28 smithi040 ceph-mon[39581]: pgmap v2935: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 530 op/s 2024-04-29T15:00:29.029 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:29.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:29.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:28 smithi194 ceph-mon[44089]: from='client.18987 v1:172.21.15.40:0/3645206098' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:00:29.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:28 smithi194 ceph-mon[44089]: pgmap v2935: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 530 op/s 2024-04-29T15:00:29.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:29.183 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:29.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:29.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:29.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:29.356 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:29.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:29.709 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:29.738 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:29.761 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:29.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:29.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:29.932 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:29.952 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T15:00:29.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:29.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:30.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:30.114 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:30.238 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:30.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:30.315 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:30.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:30.425 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:31.084 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:31.104 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:31.131 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:31.168 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:31.189 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:31.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:31.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:31.225 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:31.257 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:31.278 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:31.299 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:31.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:31.320 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:31.345 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:31.345 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:31.345 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:31.345 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:31.345 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:31.345 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:31.345 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:31.348 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:31.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:31 smithi194 ceph-mon[44089]: pgmap v2936: 17 pgs: 17 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-04-29T15:00:31.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:31 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2895510761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:31.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:31 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:31 smithi040 ceph-mon[39581]: pgmap v2936: 17 pgs: 17 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-04-29T15:00:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:31 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2895510761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:31 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:31 smithi121 ceph-mon[44533]: pgmap v2936: 17 pgs: 17 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-04-29T15:00:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:31 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2895510761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:31.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:31 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.077 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:32.077 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:32.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:32.301 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:32.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:32.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:32.485 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:32.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:32.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:32.761 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.761 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.761 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.761 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.761 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.761 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:32 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:32 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:32 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:32.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:32.927 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:33.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:33.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:33.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-29T15:00:33.408 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:33.437 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:33.461 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:33.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:33.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:33 smithi121 ceph-mon[44533]: pgmap v2937: 17 pgs: 17 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 693 op/s 2024-04-29T15:00:33.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:33.596 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:33.619 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:33.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:33.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:33.809 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:33.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:33 smithi194 ceph-mon[44089]: pgmap v2937: 17 pgs: 17 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 693 op/s 2024-04-29T15:00:33.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:33.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:33.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:33 smithi040 ceph-mon[39581]: pgmap v2937: 17 pgs: 17 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 693 op/s 2024-04-29T15:00:33.971 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:34.005 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:34.029 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:34.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:34.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:34.215 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:34.250 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:34.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:34.279 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T15:00:34.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:34.406 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T15:00:34.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2173333612' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:00:34.571 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:34 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:00:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2173333612' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:00:34.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:34 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:00:34.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2173333612' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:00:34.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:34 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:00:35.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:35.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:35.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:35.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: pgmap v2938: 17 pgs: 17 active+clean; 1.5 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 883 op/s 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2173333612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: osdmap e1145: 12 total, 11 up, 7 in 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: 9.8 scrub starts 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: 9.8 scrub ok 2024-04-29T15:00:35.535 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:35 smithi121 ceph-mon[44533]: 9.7 scrub starts 2024-04-29T15:00:35.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:35.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:35.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: pgmap v2938: 17 pgs: 17 active+clean; 1.5 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 883 op/s 2024-04-29T15:00:35.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:35.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2173333612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:00:35.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: osdmap e1145: 12 total, 11 up, 7 in 2024-04-29T15:00:35.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: 9.8 scrub starts 2024-04-29T15:00:35.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: 9.8 scrub ok 2024-04-29T15:00:35.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:35 smithi040 ceph-mon[39581]: 9.7 scrub starts 2024-04-29T15:00:35.686 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:35.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:35.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:35.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: pgmap v2938: 17 pgs: 17 active+clean; 1.5 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 883 op/s 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2173333612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: osdmap e1145: 12 total, 11 up, 7 in 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: 9.8 scrub starts 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: 9.8 scrub ok 2024-04-29T15:00:35.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:35 smithi194 ceph-mon[44089]: 9.7 scrub starts 2024-04-29T15:00:35.863 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:36.123 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:36.123 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:36.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:36.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:36.237 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:36.270 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:36.292 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:36.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:36.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:36.469 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: 9.7 scrub ok 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: 9.5 scrub starts 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:36 smithi040 ceph-mon[39581]: 9.5 scrub ok 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: 9.7 scrub ok 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: 9.5 scrub starts 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:36 smithi194 ceph-mon[44089]: 9.5 scrub ok 2024-04-29T15:00:36.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: 9.7 scrub ok 2024-04-29T15:00:36.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: 9.5 scrub starts 2024-04-29T15:00:36.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:00:36.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:00:36.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:00:36.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:36 smithi121 ceph-mon[44533]: 9.5 scrub ok 2024-04-29T15:00:37.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:37.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:37.432 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:37.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T15:00:37.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:37.458 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:37.479 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:37.486 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:37.506 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:37.511 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:37.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:37.536 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:37.566 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:37.612 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:37.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:37.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:37 smithi121 ceph-mon[44533]: pgmap v2940: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-04-29T15:00:37.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1897936820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:37.652 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:37.681 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:37.705 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:37.729 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:37.729 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:37.729 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:37.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:37 smithi040 ceph-mon[39581]: pgmap v2940: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-04-29T15:00:37.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1897936820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:37 smithi194 ceph-mon[44089]: pgmap v2940: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 875 op/s 2024-04-29T15:00:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1897936820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:38.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:38.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:38.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-29T15:00:38.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:38.397 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:38.441 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:38.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:38.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:38.567 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.585 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:00:38.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:00:38.586 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:38 smithi121 ceph-mon[44533]: 9.1 scrub starts 2024-04-29T15:00:38.618 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:38.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:38.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:00:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:38.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:00:38.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:38 smithi040 ceph-mon[39581]: 9.1 scrub starts 2024-04-29T15:00:39.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:39.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:39.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:39.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:00:39.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:00:39.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:38 smithi194 ceph-mon[44089]: 9.1 scrub starts 2024-04-29T15:00:39.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:39.254 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:39.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:39.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:39.438 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:39.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:39 smithi040 ceph-mon[39581]: 9.1 scrub ok 2024-04-29T15:00:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:39 smithi040 ceph-mon[39581]: pgmap v2941: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-04-29T15:00:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:39 smithi040 ceph-mon[39581]: osdmap e1146: 12 total, 11 up, 7 in 2024-04-29T15:00:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:39 smithi040 ceph-mon[39581]: 9.9 scrub starts 2024-04-29T15:00:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:39 smithi040 ceph-mon[39581]: 9.9 scrub ok 2024-04-29T15:00:39.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:39 smithi040 ceph-mon[39581]: 9.b scrub starts 2024-04-29T15:00:39.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:39 smithi121 ceph-mon[44533]: 9.1 scrub ok 2024-04-29T15:00:39.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:39 smithi121 ceph-mon[44533]: pgmap v2941: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-04-29T15:00:39.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:39 smithi121 ceph-mon[44533]: osdmap e1146: 12 total, 11 up, 7 in 2024-04-29T15:00:39.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:39 smithi121 ceph-mon[44533]: 9.9 scrub starts 2024-04-29T15:00:39.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:39 smithi121 ceph-mon[44533]: 9.9 scrub ok 2024-04-29T15:00:39.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:39 smithi121 ceph-mon[44533]: 9.b scrub starts 2024-04-29T15:00:40.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:39 smithi194 ceph-mon[44089]: 9.1 scrub ok 2024-04-29T15:00:40.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:39 smithi194 ceph-mon[44089]: pgmap v2941: 17 pgs: 17 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-04-29T15:00:40.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:39 smithi194 ceph-mon[44089]: osdmap e1146: 12 total, 11 up, 7 in 2024-04-29T15:00:40.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:39 smithi194 ceph-mon[44089]: 9.9 scrub starts 2024-04-29T15:00:40.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:39 smithi194 ceph-mon[44089]: 9.9 scrub ok 2024-04-29T15:00:40.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:39 smithi194 ceph-mon[44089]: 9.b scrub starts 2024-04-29T15:00:40.321 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:40.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:40.430 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:40.457 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:40.482 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:40.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:40.568 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T15:00:40.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:40.600 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:40 smithi121 ceph-mon[44533]: 9.b scrub ok 2024-04-29T15:00:40.645 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:40.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:40.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:40.797 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:40.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:40 smithi194 ceph-mon[44089]: 9.b scrub ok 2024-04-29T15:00:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:40.857 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:40.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:40 smithi040 ceph-mon[39581]: 9.b scrub ok 2024-04-29T15:00:40.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:40.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:40.897 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:40.962 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:40.991 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:41.014 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:41.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:41.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:41.149 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:41.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:41.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55786 8.d 8:b7ad01c4:::benchmark_data_smithi194_186548_object55785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.479136+0000", 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.76551853000001, 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.206405844, 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55786 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479136+0000", 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479136+0000", 2024-04-29T15:00:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479144+0000", 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3610000000000001e-06 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479206+0000", 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1859000000000005e-05 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479223+0000", 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7119e-05 2024-04-29T15:00:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479232+0000", 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3710000000000006e-06 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479282+0000", 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0160000000000001e-05 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479387+0000", 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104498 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.479547+0000", 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001605 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.481617+0000", 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002069843 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.685418+0000", 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20380108299999999 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.685487+0000", 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8468999999999998e-05 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.685541+0000", 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4580999999999998e-05 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55798 8.d 8:b0e864d1:::benchmark_data_smithi194_186548_object55797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.483978+0000", 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.760675816, 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20377042300000001, 2024-04-29T15:00:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55798 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.483978+0000", 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.483978+0000", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.483980+0000", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.875e-06 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484041+0000", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0939e-05 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484052+0000", 2024-04-29T15:00:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1252e-05 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484061+0000", 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2239999999999995e-06 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484112+0000", 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1056e-05 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484221+0000", 2024-04-29T15:00:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010945799999999999 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484348+0000", 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012649100000000001 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488272+0000", 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039242139999999997 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687645+0000", 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19937343900000001 2024-04-29T15:00:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687704+0000", 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8479e-05 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687749+0000", 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4996000000000001e-05 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55800 8.d 8:b9d6e6f9:::benchmark_data_smithi194_186548_object55799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.484076+0000", 2024-04-29T15:00:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.76057806599999, 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383693, 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55800 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484076+0000", 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484076+0000", 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484077+0000", 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9699999999999995e-07 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484126+0000", 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9508000000000001e-05 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484134+0000", 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5079999999999999e-06 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484140+0000", 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5150000000000003e-06 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.486180+0000", 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020398460000000001 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.486327+0000", 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014636599999999999 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.486467+0000", 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014013799999999999 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.490065+0000", 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035985079999999998 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687807+0000", 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19774185 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687867+0000", 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0146000000000003e-05 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.687913+0000", 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5748000000000003e-05 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55801 8.d 8:bb82fe4c:::benchmark_data_smithi194_186548_object55800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.484152+0000", 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.76050234899998, 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20525239100000001, 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55801 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484152+0000", 2024-04-29T15:00:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484152+0000", 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484153+0000", 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6000000000000002e-07 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484239+0000", 2024-04-29T15:00:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6113999999999996e-05 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484252+0000", 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3665999999999999e-05 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.484258+0000", 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9880000000000001e-06 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488314+0000", 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040557700000000002 2024-04-29T15:00:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488466+0000", 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015224100000000001 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.488605+0000", 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013902700000000001 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.490212+0000", 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016065820000000001 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689307+0000", 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19909527899999999 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689362+0000", 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4366000000000001e-05 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689404+0000", 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2497999999999999e-05 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55825 8.d 8:b6e45766:::benchmark_data_smithi194_186548_object55824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.494640+0000", 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.75001385799999, 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.194866867, 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55825 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494640+0000", 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494640+0000", 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494643+0000", 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.931e-06 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494713+0000", 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9857999999999994e-05 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494729+0000", 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6413e-05 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494737+0000", 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8439999999999998e-06 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494799+0000", 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1389000000000005e-05 2024-04-29T15:00:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494904+0000", 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104845 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.494992+0000", 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8125000000000001e-05 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.497062+0000", 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020707059999999998 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689449+0000", 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19238635400000001 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689481+0000", 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2147000000000002e-05 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689507+0000", 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6254999999999998e-05 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55843 8.d 8:b8b16e17:::benchmark_data_smithi194_186548_object55842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.501883+0000", 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.74277091800002, 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.187756907, 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55843 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501883+0000", 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501883+0000", 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501886+0000", 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5799999999999999e-06 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501966+0000", 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0462000000000003e-05 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501979+0000", 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2486e-05 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.501987+0000", 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5439999999999999e-06 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.502046+0000", 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8774999999999999e-05 2024-04-29T15:00:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.502144+0000", 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8028000000000004e-05 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.502242+0000", 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7589000000000004e-05 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.503867+0000", 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001625004 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689551+0000", 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18568444200000001 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689604+0000", 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2485999999999999e-05 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.689640+0000", 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6511000000000003e-05 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55847 8.d 8:b2675a97:::benchmark_data_smithi194_186548_object55846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.504082+0000", 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.74057254100001, 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18729451899999999, 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55847 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504082+0000", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504082+0000", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504083+0000", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7129999999999999e-06 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504162+0000", 2024-04-29T15:00:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8981999999999997e-05 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504174+0000", 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1877e-05 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504186+0000", 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1912999999999999e-05 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504240+0000", 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3683000000000001e-05 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504375+0000", 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000135337 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.504523+0000", 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014781700000000001 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506003+0000", 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00148029 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691301+0000", 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18529801800000001 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691348+0000", 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6669000000000003e-05 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691376+0000", 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8220000000000001e-05 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55853 8.d 8:b6da80e1:::benchmark_data_smithi194_186548_object55852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.506457+0000", 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.73819680399998, 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.185007753, 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55853 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506457+0000", 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506457+0000", 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506459+0000", 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8369999999999999e-06 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506548+0000", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8329000000000001e-05 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506561+0000", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3273e-05 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506568+0000", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5279999999999998e-06 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506616+0000", 2024-04-29T15:00:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7604e-05 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506725+0000", 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108674 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.506843+0000", 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011833900000000001 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.508399+0000", 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015563759999999999 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691410+0000", 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18301077299999999 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691441+0000", 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0508999999999999e-05 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.691465+0000", 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4511e-05 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55865 8.d 8:b5fea464:::benchmark_data_smithi194_186548_object55864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.512697+0000", 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.73195698199999, 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17965973299999999, 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55865 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512697+0000", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512697+0000", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512699+0000", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3039999999999999e-06 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512778+0000", 2024-04-29T15:00:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8614000000000003e-05 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512791+0000", 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3050999999999999e-05 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512798+0000", 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3529999999999997e-06 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.512855+0000", 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6495e-05 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.513015+0000", 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016034400000000001 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.513179+0000", 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016350800000000001 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.516866+0000", 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036867520000000002 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692265+0000", 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.175399061 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692316+0000", 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1211999999999997e-05 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692357+0000", 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1038999999999998e-05 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55870 8.d 8:b73c258c:::benchmark_data_smithi194_186548_object55869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.514805+0000", 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.72984880899998, 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17769389799999999, 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55870 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514805+0000", 2024-04-29T15:00:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514805+0000", 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514808+0000", 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2330000000000001e-06 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514921+0000", 2024-04-29T15:00:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113451 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514932+0000", 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1071e-05 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514941+0000", 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7229999999999993e-06 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.514995+0000", 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3783999999999999e-05 2024-04-29T15:00:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515123+0000", 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128608 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515244+0000", 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120907 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.518630+0000", 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033863610000000001 2024-04-29T15:00:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692404+0000", 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17377337800000001 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692459+0000", 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4839000000000001e-05 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.692499+0000", 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0543000000000001e-05 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55874 8.d 8:b6a90c97:::benchmark_data_smithi194_186548_object55873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.515297+0000", 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.72935667899998, 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.178947574, 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55874 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515297+0000", 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515297+0000", 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515299+0000", 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3090000000000001e-06 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515408+0000", 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109038 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515419+0000", 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1345999999999999e-05 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.515427+0000", 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7549999999999994e-06 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.516892+0000", 2024-04-29T15:00:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146514 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.517033+0000", 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014117000000000001 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.517174+0000", 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014041300000000001 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.518748+0000", 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015739149999999999 2024-04-29T15:00:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.694148+0000", 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.175400532 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.694204+0000", 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5454999999999997e-05 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.694245+0000", 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501000000000001e-05 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55881 8.d 8:b86e0694:::benchmark_data_smithi194_186548_object55880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.520125+0000", 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.72452924499999, 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17558306000000001, 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55881 2024-04-29T15:00:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520125+0000", 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520125+0000", 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520127+0000", 2024-04-29T15:00:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3420000000000002e-06 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520197+0000", 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9814000000000006e-05 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520212+0000", 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4543e-05 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520221+0000", 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0699999999999996e-06 2024-04-29T15:00:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520275+0000", 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3844999999999997e-05 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.520279+0000", 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5310000000000003e-06 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.522866+0000", 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025868760000000001 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.522988+0000", 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012194 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.523105+0000", 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116737 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.524986+0000", 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018813829999999999 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695621+0000", 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17063498299999999 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695676+0000", 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5096000000000002e-05 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695708+0000", 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1900000000000003e-05 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55896 8.d 8:b67bbaf5:::benchmark_data_smithi194_186548_object55895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.534147+0000", 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.71050675200001, 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.161730709, 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55896 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534147+0000", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534147+0000", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534150+0000", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8100000000000002e-06 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534212+0000", 2024-04-29T15:00:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1506000000000004e-05 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534225+0000", 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3664e-05 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534233+0000", 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.79e-06 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534280+0000", 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6791e-05 2024-04-29T15:00:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534382+0000", 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102073 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534468+0000", 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5833000000000003e-05 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.538324+0000", 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038565439999999999 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695779+0000", 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.157454439 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695835+0000", 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5769000000000001e-05 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695878+0000", 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3489999999999999e-05 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55898 8.d 8:b5f4d5c8:::benchmark_data_smithi194_186548_object55897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.534722+0000", 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.70993248100001, 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16130372000000001, 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55898 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534722+0000", 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534722+0000", 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534723+0000", 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.53e-06 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534784+0000", 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1001000000000001e-05 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534794+0000", 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7550000000000006e-06 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.534800+0000", 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3659999999999997e-06 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536179+0000", 2024-04-29T15:00:41.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013789379999999999 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536294+0000", 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011436499999999999 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536387+0000", 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3579000000000006e-05 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.538329+0000", 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001941952 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695920+0000", 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.157591237 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.695988+0000", 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7735000000000001e-05 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696025+0000", 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7261999999999997e-05 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28144.0:55899 8.d 8:b55d3b13:::benchmark_data_smithi194_186548_object55898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:52:28.536001+0000", 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 492.70865331499999, 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16021872400000001, 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28144", 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 55899 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536001+0000", 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536001+0000", 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536002+0000", 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.119e-06 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536073+0000", 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0591000000000002e-05 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536080+0000", 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5519999999999999e-06 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.536085+0000", 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0849999999999996e-06 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.537244+0000", 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011585149999999999 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.537339+0000", 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5093999999999995e-05 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.537420+0000", 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1573e-05 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.538506+0000", 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001085981 2024-04-29T15:00:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696128+0000", 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15762129799999999 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696180+0000", 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2200999999999997e-05 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:52:28.696220+0000", 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9715000000000002e-05 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:233140 9.e 9:778e5c7f:::benchmark_data_smithi194_196924_object73134:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1116)", 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:58:47.996431+0000", 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 113.24822364000001, 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105830615, 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 233140 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.996431+0000", 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.996431+0000", 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.996432+0000", 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8700000000000001e-06 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.996438+0000", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6160000000000001e-06 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.996449+0000", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0750999999999999e-05 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.996457+0000", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6610000000000006e-06 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.000104+0000", 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036467169999999998 2024-04-29T15:00:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.000221+0000", 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011642900000000001 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.000334+0000", 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011359899999999999 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.097384+0000", 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097049707999999998 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.102166+0000", 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047824829999999997 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.102215+0000", 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8251999999999998e-05 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.102261+0000", 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6529000000000001e-05 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:233141 9.e 9:7f89d63d:::benchmark_data_smithi194_196924_object73135:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1116)", 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:58:47.998006+0000", 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 113.24664776, 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.114043374, 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 233141 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.998006+0000", 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.998006+0000", 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.998008+0000", 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7150000000000001e-06 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.998014+0000", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5199999999999997e-06 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.998023+0000", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8050000000000001e-06 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:47.998031+0000", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1389999999999998e-06 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.005369+0000", 2024-04-29T15:00:41.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073386509999999999 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.005509+0000", 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014005800000000001 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.005621+0000", 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112039 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.102276+0000", 2024-04-29T15:00:41.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.096655018999999995 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.111953+0000", 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096764920000000001 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.112003+0000", 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9685999999999998e-05 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.112050+0000", 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7250000000000003e-05 2024-04-29T15:00:41.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.28381.0:233153 9.7 e1116/1065 9:eda9f72d:::benchmark_data_smithi194_196924_object73147:head v 1116'14316, mlcod=1116'14313)", 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:58:48.007833+0000", 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 113.236821296, 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103029229, 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.007833+0000", 2024-04-29T15:00:41.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.007833+0000", 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.007833+0000", 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2800000000000002e-07 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.007839+0000", 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7089999999999999e-06 2024-04-29T15:00:41.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.007844+0000", 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0429999999999998e-06 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.007849+0000", 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6600000000000003e-06 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.097255+0000", 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089406168999999994 2024-04-29T15:00:41.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.097281+0000", 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6258000000000002e-05 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.110824+0000", 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013542398000000001 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:58:48.110862+0000", 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8563999999999999e-05 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255383 9.d 9.3a66498d (undecoded) ondisk+retry+write+known_if_redirected e1130)", 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:36.590811+0000", 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.653843383999998, 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14023327099999999, 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255383 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590811+0000", 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590811+0000", 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590814+0000", 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6009999999999999e-06 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590823+0000", 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5879999999999998e-06 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590835+0000", 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1724e-05 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590847+0000", 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2707999999999999e-05 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590947+0000", 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9384999999999998e-05 2024-04-29T15:00:41.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590958+0000", 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1473e-05 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.731004+0000", 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14004565799999999 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.731044+0000", 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0133999999999999e-05 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.347 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.28381.0:255389 9.d 9.bbc0c4fd (undecoded) ondisk+retry+write+known_if_redirected e1130)", 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-29T14:59:36.590875+0000", 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.653779322999995, 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14030000300000001, 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.28381", 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 255389 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-29T15:00:41.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590875+0000", 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590875+0000", 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590876+0000", 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5900000000000005e-07 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-29T15:00:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590880+0000", 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2880000000000003e-06 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590886+0000", 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.319e-06 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590893+0000", 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5729999999999996e-06 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590981+0000", 2024-04-29T15:00:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7688999999999993e-05 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.590986+0000", 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2449999999999998e-06 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.731165+0000", 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.140178683 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-29T14:59:36.731175+0000", 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0247e-05 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-29T15:00:41.352 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-29T15:00:41.352 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-29T15:00:41.352 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:41.379 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:41.402 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:41.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:41.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:41.556 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:41.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:41.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:41.614 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T15:00:41.705 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:41 smithi040 ceph-mon[39581]: pgmap v2943: 17 pgs: 17 active+clean; 1.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 991 op/s 2024-04-29T15:00:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:41 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1170449814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:41 smithi040 ceph-mon[39581]: from='client.38354 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:41.737 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:41 smithi194 ceph-mon[44089]: pgmap v2943: 17 pgs: 17 active+clean; 1.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 991 op/s 2024-04-29T15:00:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:41 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1170449814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:41 smithi194 ceph-mon[44089]: from='client.38354 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:41.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:41 smithi121 ceph-mon[44533]: pgmap v2943: 17 pgs: 17 active+clean; 1.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 991 op/s 2024-04-29T15:00:41.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:41 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1170449814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:41.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:41 smithi121 ceph-mon[44533]: from='client.38354 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:41.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:42.096 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:42.125 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:42.147 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:42.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:42.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:42.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:42.322 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:42.621 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:42 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:00:42.621 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:42 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T15:00:42.621 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:42 smithi121 ceph-mon[44533]: from='client.38354 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:42.621 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:42 smithi121 ceph-mon[44533]: osdmap e1147: 12 total, 11 up, 7 in 2024-04-29T15:00:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:42 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:00:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:42 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T15:00:42.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:42 smithi040 ceph-mon[39581]: from='client.38354 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:42.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:42 smithi040 ceph-mon[39581]: osdmap e1147: 12 total, 11 up, 7 in 2024-04-29T15:00:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:42 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:00:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:42 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T15:00:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:42 smithi194 ceph-mon[44089]: from='client.38354 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:42 smithi194 ceph-mon[44089]: osdmap e1147: 12 total, 11 up, 7 in 2024-04-29T15:00:43.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-29T15:00:43.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:43.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:43.313 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:43.343 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:43.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:43.495 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:00:43.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:43.626 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:43.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:43.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:43.774 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:43.797 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:43.804 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:43.820 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:43.834 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:43.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:43.848 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:43.872 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:43 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T15:00:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:43 smithi040 ceph-mon[39581]: pgmap v2945: 17 pgs: 17 active+clean; 1.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-04-29T15:00:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:43 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T15:00:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:43 smithi040 ceph-mon[39581]: 9.6 scrub starts 2024-04-29T15:00:43.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:43 smithi040 ceph-mon[39581]: 9.6 scrub ok 2024-04-29T15:00:43.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:43 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T15:00:43.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:43 smithi121 ceph-mon[44533]: pgmap v2945: 17 pgs: 17 active+clean; 1.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-04-29T15:00:43.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:43 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T15:00:43.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:43 smithi121 ceph-mon[44533]: 9.6 scrub starts 2024-04-29T15:00:43.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:43 smithi121 ceph-mon[44533]: 9.6 scrub ok 2024-04-29T15:00:43.902 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:43.927 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:43.939 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:43.950 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:43.966 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:43 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T15:00:43.966 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:43 smithi194 ceph-mon[44089]: pgmap v2945: 17 pgs: 17 active+clean; 1.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-04-29T15:00:43.966 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:43 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T15:00:43.966 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:43 smithi194 ceph-mon[44089]: 9.6 scrub starts 2024-04-29T15:00:43.966 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:43 smithi194 ceph-mon[44089]: 9.6 scrub ok 2024-04-29T15:00:43.969 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:43.974 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:43.998 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:44.018 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:44.018 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:44.096 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:44.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:44.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:44.243 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:44.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2994504883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:44 smithi040 ceph-mon[39581]: 9.3 scrub starts 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2994504883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:44.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:44 smithi121 ceph-mon[44533]: 9.3 scrub starts 2024-04-29T15:00:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:44.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:44.973 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:45.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2994504883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:45.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.036 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.037 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.037 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.037 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.037 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:44 smithi194 ceph-mon[44089]: 9.3 scrub starts 2024-04-29T15:00:45.097 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:45.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:45.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:45.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:45.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:45.447 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:45.467 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:45.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:45.572 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:45.607 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:45.710 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:45.718 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.718 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:45 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:45 smithi194 ceph-mon[44089]: 9.e scrub starts 2024-04-29T15:00:45.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:45 smithi194 ceph-mon[44089]: 9.3 scrub ok 2024-04-29T15:00:45.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:45 smithi194 ceph-mon[44089]: pgmap v2946: 17 pgs: 17 active+clean; 1.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-04-29T15:00:45.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:45 smithi194 ceph-mon[44089]: 9.e scrub ok 2024-04-29T15:00:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:45.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:45.741 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:45.741 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:45.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:45 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:45 smithi040 ceph-mon[39581]: 9.e scrub starts 2024-04-29T15:00:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:45 smithi040 ceph-mon[39581]: 9.3 scrub ok 2024-04-29T15:00:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:45 smithi040 ceph-mon[39581]: pgmap v2946: 17 pgs: 17 active+clean; 1.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-04-29T15:00:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:45 smithi040 ceph-mon[39581]: 9.e scrub ok 2024-04-29T15:00:45.886 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:45.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:45 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:45.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:45 smithi121 ceph-mon[44533]: 9.e scrub starts 2024-04-29T15:00:45.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:45 smithi121 ceph-mon[44533]: 9.3 scrub ok 2024-04-29T15:00:45.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:45 smithi121 ceph-mon[44533]: pgmap v2946: 17 pgs: 17 active+clean; 1.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-04-29T15:00:45.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:45 smithi121 ceph-mon[44533]: 9.e scrub ok 2024-04-29T15:00:45.920 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:45.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:46.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:46.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:46.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:46.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:46.529 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:46.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:46.561 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-29T15:00:46.636 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:46 smithi194 ceph-mon[44089]: 9.d scrub starts 2024-04-29T15:00:46.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:46 smithi194 ceph-mon[44089]: 9.d scrub ok 2024-04-29T15:00:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:46 smithi040 ceph-mon[39581]: 9.d scrub starts 2024-04-29T15:00:46.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:46 smithi040 ceph-mon[39581]: 9.d scrub ok 2024-04-29T15:00:46.935 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:00:46.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T15:00:47.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:46 smithi121 ceph-mon[44533]: 9.d scrub starts 2024-04-29T15:00:47.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:46 smithi121 ceph-mon[44533]: 9.d scrub ok 2024-04-29T15:00:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:47 smithi194 ceph-mon[44089]: pgmap v2947: 17 pgs: 17 active+clean; 835 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T15:00:48.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:47 smithi040 ceph-mon[39581]: pgmap v2947: 17 pgs: 17 active+clean; 835 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T15:00:48.138 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:47 smithi121 ceph-mon[44533]: pgmap v2947: 17 pgs: 17 active+clean; 835 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-29T15:00:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:48.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:48.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:48.477 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:48.601 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:48.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:48.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:48.767 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:48.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:48.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:48.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:48 smithi040 ceph-mon[39581]: pgmap v2948: 17 pgs: 17 active+clean; 817 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-04-29T15:00:48.923 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:48.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:48.959 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:48 smithi194 ceph-mon[44089]: pgmap v2948: 17 pgs: 17 active+clean; 817 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-04-29T15:00:48.962 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:48.970 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:48.971 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:48.984 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:48.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:49.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:49.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:48 smithi121 ceph-mon[44533]: pgmap v2948: 17 pgs: 17 active+clean; 817 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-04-29T15:00:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:49.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:49.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:49.348 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:49.382 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:49.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:49.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:49.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:49.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:49.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:49.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:49.739 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:49.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:49.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:49.782 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-29T15:00:49.822 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is set 2024-04-29T15:00:49.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:49.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:49.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3617818423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:49 smithi194 ceph-mon[44089]: from='client.38363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:50.075 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:50.113 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3617818423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:49 smithi040 ceph-mon[39581]: from='client.38363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:50.138 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:50.140 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:00:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:50.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:50.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3617818423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:50.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:49 smithi121 ceph-mon[44533]: from='client.38363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-29T15:00:50.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:50.292 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:50.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:50.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:50.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:50.491 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:50.533 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:50.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:50.643 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:50.679 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:50.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:50 smithi040 ceph-mon[39581]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:50.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:50 smithi040 ceph-mon[39581]: from='client.38363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T15:00:50.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:50 smithi040 ceph-mon[39581]: osdmap e1148: 12 total, 11 up, 7 in 2024-04-29T15:00:50.950 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:50 smithi040 ceph-mon[39581]: pgmap v2950: 17 pgs: 17 active+clean; 625 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-04-29T15:00:51.036 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:51.039 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:51.039 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:50 smithi194 ceph-mon[44089]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:51.040 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:50 smithi194 ceph-mon[44089]: from='client.38363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T15:00:51.040 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:50 smithi194 ceph-mon[44089]: osdmap e1148: 12 total, 11 up, 7 in 2024-04-29T15:00:51.040 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:50 smithi194 ceph-mon[44089]: pgmap v2950: 17 pgs: 17 active+clean; 625 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-04-29T15:00:51.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:51.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:50 smithi121 ceph-mon[44533]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:51.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:50 smithi121 ceph-mon[44533]: from='client.38363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-29T15:00:51.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:50 smithi121 ceph-mon[44533]: osdmap e1148: 12 total, 11 up, 7 in 2024-04-29T15:00:51.142 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:50 smithi121 ceph-mon[44533]: pgmap v2950: 17 pgs: 17 active+clean; 625 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-04-29T15:00:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:51.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:51.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:51.329 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:51.348 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:51.356 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:51.380 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:51.383 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:51.413 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:51.434 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:51.457 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:51.487 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:51.507 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:51.528 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:51.551 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:51.575 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:52.032 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:52.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:51 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2062318995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:51 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:51 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2062318995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:51 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:52.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:51 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2062318995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:52.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:51 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:52.234 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T15:00:52.234 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T15:00:52.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T15:00:52.234 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:56603 8.9 8:99841e75:::benchmark_data_smithi194_186548_object56602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:28.946636+0000", 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 503.28095745100001, 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21160589699999999, 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 56603 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946636+0000", 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946636+0000", 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946638+0000", 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5579999999999999e-06 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946664+0000", 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5816000000000002e-05 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946677+0000", 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2904999999999999e-05 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946687+0000", 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0054000000000001e-05 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946764+0000", 2024-04-29T15:00:52.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6526999999999994e-05 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.946898+0000", 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013447200000000001 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.947041+0000", 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000142509 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.948898+0000", 2024-04-29T15:00:52.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001857385 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.158138+0000", 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.209240011 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.158176+0000", 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7727e-05 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.158242+0000", 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5932999999999994e-05 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.239 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:56626 8.9 8:9947ac6e:::benchmark_data_smithi194_186548_object56625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:28.953953+0000", 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 503.27364013800002, 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.204454735, 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 56626 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.240 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.953953+0000", 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.953953+0000", 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.953955+0000", 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.618e-06 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.953976+0000", 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0729000000000001e-05 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.953985+0000", 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8020000000000003e-06 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.953993+0000", 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7270000000000007e-06 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.954049+0000", 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6035000000000001e-05 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.954180+0000", 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013140499999999999 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.954312+0000", 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013188699999999999 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.956306+0000", 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019932510000000001 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.158320+0000", 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.202014795 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.158358+0000", 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7554e-05 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.158408+0000", 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9932000000000002e-05 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:56655 8.9 8:90f28d5a:::benchmark_data_smithi194_186548_object56654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:28.967011+0000", 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 503.26058198499999, 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.192718848, 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.245 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 56655 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967011+0000", 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967011+0000", 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967013+0000", 2024-04-29T15:00:52.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9080000000000002e-06 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967034+0000", 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0976e-05 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967046+0000", 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1694e-05 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967054+0000", 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7440000000000007e-06 2024-04-29T15:00:52.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967108+0000", 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.427e-05 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967235+0000", 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012718699999999999 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.967379+0000", 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000143969 2024-04-29T15:00:52.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.975809+0000", 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084294420000000005 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.159652+0000", 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18384384600000001 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.159683+0000", 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0264999999999999e-05 2024-04-29T15:00:52.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.159730+0000", 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7546999999999997e-05 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:56662 8.9 8:98444a42:::benchmark_data_smithi194_186548_object56661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e1018)", 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:28.968668+0000", 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 503.25892500499998, 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19150777999999999, 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 56662 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968668+0000", 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968668+0000", 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968669+0000", 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1999999999999999e-07 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968692+0000", 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3048999999999999e-05 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968700+0000", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7689999999999996e-06 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968706+0000", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2129999999999996e-06 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.973008+0000", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043021939999999996 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.973103+0000", 2024-04-29T15:00:52.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5239000000000001e-05 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.973213+0000", 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109262 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.975825+0000", 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026127709999999998 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.160074+0000", 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18424895499999999 2024-04-29T15:00:52.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.160135+0000", 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0715999999999999e-05 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.160176+0000", 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0992000000000003e-05 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:56666 8.9 8:93dc4376:::benchmark_data_smithi194_186548_object56665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:28.968749+0000", 2024-04-29T15:00:52.255 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 503.258844684, 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19155491499999999, 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 56666 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968749+0000", 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968749+0000", 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968749+0000", 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0999999999999997e-07 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968772+0000", 2024-04-29T15:00:52.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2790999999999999e-05 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968781+0000", 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3100000000000001e-06 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.968787+0000", 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2339999999999999e-06 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.974349+0000", 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0055624280000000003 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.974449+0000", 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9748999999999997e-05 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.974547+0000", 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8103000000000006e-05 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:28.977652+0000", 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003105203 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.160228+0000", 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18257622800000001 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.160280+0000", 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1542000000000002e-05 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:29.160304+0000", 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3516999999999999e-05 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:71217 8.5 8:ac75f492:::benchmark_data_smithi194_186548_object71216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:52.261 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:35.393474+0000", 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.83411901199997, 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20761114999999999, 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 71217 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393474+0000", 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393474+0000", 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393476+0000", 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4839999999999999e-06 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393502+0000", 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6268999999999999e-05 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393514+0000", 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1585e-05 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393522+0000", 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1170000000000004e-06 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393574+0000", 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2638e-05 2024-04-29T15:00:52.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393711+0000", 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000136582 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.393853+0000", 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014205000000000001 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396009+0000", 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021556779999999998 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.601005+0000", 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20499651999999999 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.601042+0000", 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6563999999999998e-05 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.601085+0000", 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3662999999999999e-05 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:71225 8.5 8:a52aa8ef:::benchmark_data_smithi194_186548_object71224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1018)", 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:35.396470+0000", 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 496.83112341399999, 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20963954800000001, 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 71225 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396470+0000", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396470+0000", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396471+0000", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5799999999999999e-07 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396490+0000", 2024-04-29T15:00:52.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9922e-05 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396496+0000", 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9549999999999997e-06 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396502+0000", 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6250000000000004e-06 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396541+0000", 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.862e-05 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396677+0000", 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013643600000000001 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.396795+0000", 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117434 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.398721+0000", 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001926412 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.606027+0000", 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.207306411 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.606077+0000", 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9845000000000001e-05 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:35.606109+0000", 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2230000000000001e-05 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:79538 8.5 8:ae6abbf7:::benchmark_data_smithi194_186548_object79537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1019)", 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:39.344752+0000", 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 492.88284132299998, 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.210380029, 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.272 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 79538 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.344752+0000", 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.344752+0000", 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.344754+0000", 2024-04-29T15:00:52.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.852e-06 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.344780+0000", 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6131000000000001e-05 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.344790+0000", 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9320000000000008e-06 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.344798+0000", 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.977e-06 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.347831+0000", 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030333640000000002 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.347946+0000", 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115116 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.348074+0000", 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000127448 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.350401+0000", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002327459 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.555049+0000", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.204647783 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.555093+0000", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3822000000000002e-05 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.555132+0000", 2024-04-29T15:00:52.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9144999999999999e-05 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28144.0:79565 8.5 8:abd3dcde:::benchmark_data_smithi194_186548_object79564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1019)", 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:52:39.357114+0000", 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 492.87047916500001, 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.199910011, 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28144", 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/1303314979", 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 79565 2024-04-29T15:00:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357114+0000", 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357114+0000", 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357116+0000", 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6920000000000001e-06 2024-04-29T15:00:52.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357139+0000", 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2883000000000001e-05 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357148+0000", 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2080000000000006e-06 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357156+0000", 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7100000000000007e-06 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357211+0000", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5500000000000001e-05 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357359+0000", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014823399999999999 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 4", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.357476+0000", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117032 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.360121+0000", 2024-04-29T15:00:52.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026445150000000001 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.556913+0000", 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19679186500000001 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.556975+0000", 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2347999999999993e-05 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:52:39.557024+0000", 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9023999999999997e-05 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.281 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:54103 9.9 9:903b4e77:::benchmark_data_smithi194_196924_object54102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1076)", 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:56:27.171630+0000", 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 265.05596323600003, 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.204300698, 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.282 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 54103 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171630+0000", 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171630+0000", 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171632+0000", 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5409999999999999e-06 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171678+0000", 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6072999999999997e-05 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171689+0000", 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0934000000000001e-05 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171695+0000", 2024-04-29T15:00:52.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6159999999999998e-06 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171744+0000", 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8378000000000003e-05 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171845+0000", 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010107000000000001 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.171937+0000", 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2406999999999998e-05 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.173482+0000", 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015447130000000001 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.375826+0000", 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20234449600000001 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.375882+0000", 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5828999999999997e-05 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.375931+0000", 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8640999999999997e-05 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:54110 9.9 9:9c7dd38b:::benchmark_data_smithi194_196924_object54109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1076)", 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:56:27.174343+0000", 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 265.05325035700002, 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.205719288, 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.287 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 54110 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174343+0000", 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174343+0000", 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174344+0000", 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.248e-06 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174382+0000", 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7827000000000002e-05 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174390+0000", 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2519999999999999e-06 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174396+0000", 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6899999999999997e-06 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174432+0000", 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5556000000000003e-05 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174521+0000", 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9116999999999994e-05 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174616+0000", 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5111000000000005e-05 2024-04-29T15:00:52.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.177268+0000", 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026517530000000002 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380004+0000", 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.202736162 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380037+0000", 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3324000000000001e-05 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380062+0000", 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5248e-05 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:54111 9.9 9:915d223a:::benchmark_data_smithi194_196924_object54110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1076)", 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:56:27.174779+0000", 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 265.05281422000002, 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20543547000000001, 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.292 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 54111 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174779+0000", 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174779+0000", 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174780+0000", 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8299999999999996e-07 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174813+0000", 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3200000000000001e-05 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174819+0000", 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0480000000000004e-06 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.174824+0000", 2024-04-29T15:00:52.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.403e-06 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.175689+0000", 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00086563199999999997 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.175830+0000", 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014060400000000001 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.175952+0000", 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012176999999999999 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.177349+0000", 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0013970650000000001 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380116+0000", 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.20276770299999999 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380172+0000", 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5803000000000001e-05 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380215+0000", 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2459000000000001e-05 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:54129 9.9 9:971d37bb:::benchmark_data_smithi194_196924_object54128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1076)", 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:56:27.181299+0000", 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 265.04629428999999, 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.199020214, 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.297 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 54129 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181299+0000", 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181299+0000", 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181301+0000", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7e-06 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181350+0000", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9076000000000002e-05 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181361+0000", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1141999999999999e-05 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181367+0000", 2024-04-29T15:00:52.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8610000000000003e-06 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181410+0000", 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3594000000000003e-05 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181525+0000", 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011415400000000001 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.181612+0000", 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7329000000000004e-05 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.183063+0000", 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014513359999999999 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380257+0000", 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.197194077 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380297+0000", 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9734999999999998e-05 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:56:27.380319+0000", 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2209999999999999e-05 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:149071 9.9 9:93e0e2e1:::benchmark_data_smithi194_196924_object149070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1097)", 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:30.888872+0000", 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 201.33872137, 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.219134355, 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.302 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 149071 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.888872+0000", 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.888872+0000", 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.888874+0000", 2024-04-29T15:00:52.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0509999999999999e-06 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.888896+0000", 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2183000000000001e-05 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.888904+0000", 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5789999999999998e-06 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.888912+0000", 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0120000000000005e-06 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.888967+0000", 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5458000000000003e-05 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.889077+0000", 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109919 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.889173+0000", 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6316000000000004e-05 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.892872+0000", 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036986789999999999 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.107889+0000", 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21501642100000001 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.107958+0000", 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9185999999999998e-05 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.108006+0000", 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8551000000000003e-05 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:149077 9.9 9:9481412a:::benchmark_data_smithi194_196924_object149076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1097)", 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:30.894594+0000", 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 201.332998844, 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21728678200000001, 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.307 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 149077 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894594+0000", 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894594+0000", 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894596+0000", 2024-04-29T15:00:52.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5239999999999999e-06 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894617+0000", 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1192e-05 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894626+0000", 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8079999999999994e-06 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894637+0000", 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1229999999999999e-05 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894678+0000", 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0800000000000002e-05 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894783+0000", 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010516799999999999 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.894876+0000", 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3061999999999998e-05 2024-04-29T15:00:52.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.898462+0000", 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035854329999999998 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.111798+0000", 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.213335842 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.111841+0000", 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3773000000000003e-05 2024-04-29T15:00:52.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.111881+0000", 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9950000000000002e-05 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:149089 9.9 9:9050f0ea:::benchmark_data_smithi194_196924_object149088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1097)", 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:30.909546+0000", 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 201.31804686999999, 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.202452244, 2024-04-29T15:00:52.312 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 149089 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909546+0000", 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909546+0000", 2024-04-29T15:00:52.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909548+0000", 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.61e-06 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909570+0000", 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2206999999999999e-05 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909580+0000", 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8779999999999993e-06 2024-04-29T15:00:52.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909586+0000", 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0399999999999998e-06 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909631+0000", 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5289e-05 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909759+0000", 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000127739 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.909900+0000", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000140548 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.914661+0000", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047608490000000002 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.111924+0000", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.197263469 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.111974+0000", 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0368000000000003e-05 2024-04-29T15:00:52.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.111999+0000", 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4247e-05 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:149099 9.9 9:97adfa56:::benchmark_data_smithi194_196924_object149098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1097)", 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:30.919360+0000", 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 201.308233801, 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.200181941, 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.317 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 149099 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919360+0000", 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919360+0000", 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919361+0000", 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9199999999999998e-06 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919386+0000", 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5029e-05 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919398+0000", 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1304000000000001e-05 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919406+0000", 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8939999999999993e-06 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919453+0000", 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7250999999999999e-05 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919552+0000", 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9282000000000003e-05 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.919645+0000", 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2606000000000001e-05 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.923235+0000", 2024-04-29T15:00:52.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035899 2024-04-29T15:00:52.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.119476+0000", 2024-04-29T15:00:52.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.19624143299999999 2024-04-29T15:00:52.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.119511+0000", 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4754e-05 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.119541+0000", 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0568000000000003e-05 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.324 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:149113 9.9 9:97a33062:::benchmark_data_smithi194_196924_object149112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e1097)", 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:57:30.948440+0000", 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 201.27915323600001, 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.193466786, 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.325 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 149113 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948440+0000", 2024-04-29T15:00:52.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948440+0000", 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948442+0000", 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.581e-06 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948464+0000", 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1996000000000001e-05 2024-04-29T15:00:52.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948473+0000", 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8390000000000002e-06 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948480+0000", 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2829999999999998e-06 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948531+0000", 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1226e-05 2024-04-29T15:00:52.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948666+0000", 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013516900000000001 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.948770+0000", 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000104023 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:30.952440+0000", 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003669598 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.141780+0000", 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.18934020700000001 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.141861+0000", 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0817999999999998e-05 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:57:31.141907+0000", 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6045999999999999e-05 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.330 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:289279 9.9 9:9e94b9d6:::benchmark_data_smithi194_196924_object129273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1143)", 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T15:00:25.958918+0000", 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.268674928999999, 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.216846291, 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 289279 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.958918+0000", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.958918+0000", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.958920+0000", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9630000000000001e-06 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.958926+0000", 2024-04-29T15:00:52.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5690000000000004e-06 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.958937+0000", 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1148000000000001e-05 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.958945+0000", 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4510000000000003e-06 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.995407+0000", 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.036462320999999999 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.995500+0000", 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2799000000000004e-05 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.995576+0000", 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6725999999999997e-05 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.007053+0000", 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011476416999999999 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.175637+0000", 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16858441099999999 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.175697+0000", 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9744000000000003e-05 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.175765+0000", 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7742000000000003e-05 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.28381.0:289293 9.9 9:9bc78c1c:::benchmark_data_smithi194_196924_object129287:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1143)", 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T15:00:25.980658+0000", 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.246935474000001, 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.195225863, 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.28381", 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "v1:172.21.15.194:0/3798306957", 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 289293 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:00:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.980658+0000", 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.980658+0000", 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.980660+0000", 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3530000000000002e-06 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.980666+0000", 2024-04-29T15:00:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5910000000000003e-06 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.980673+0000", 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3270000000000003e-06 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.980679+0000", 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0680000000000002e-06 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.998868+0000", 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018188451000000001 2024-04-29T15:00:52.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.998986+0000", 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011832699999999999 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for subops from 5", 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:25.999110+0000", 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012354600000000001 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op_commit", 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.007204+0000", 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0080944840000000007 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_commit_rec", 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.175810+0000", 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16860583000000001 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.175844+0000", 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4063999999999998e-05 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T15:00:26.175884+0000", 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9821999999999999e-05 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:00:52.340 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T15:00:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:52.341 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:52.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:52.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:52.479 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:52.561 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:52.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:52.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:52.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:52.722 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:52.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:52.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:52.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:52 smithi040 ceph-mon[39581]: pgmap v2951: 17 pgs: 17 active+clean; 535 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-04-29T15:00:52.934 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:52.966 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:52.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:52 smithi121 ceph-mon[44533]: pgmap v2951: 17 pgs: 17 active+clean; 535 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-04-29T15:00:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:52.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:53.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-29T15:00:53.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:53.138 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:53.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:53.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:53.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:53.309 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:53.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:53.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:52 smithi194 ceph-mon[44089]: pgmap v2951: 17 pgs: 17 active+clean; 535 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-04-29T15:00:53.331 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:53.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:53.373 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:00:53.434 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:53.462 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:53.630 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:53.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:53.736 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:53.769 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:53.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-29T15:00:53.927 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:53.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:54.031 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:00:54.062 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:54.135 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:54.262 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:00:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:54.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:00:54.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:54.422 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:54.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:54.576 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:00:54.604 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:54.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T15:00:54.859 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:00:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:54.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:00:54.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:55.013 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:00:55.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:55.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:55.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:00:55.158 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:55.554 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T15:00:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-04-29T15:00:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-04-29T15:00:55.554 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T15:00:55.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:55 smithi194 ceph-mon[44089]: pgmap v2952: 17 pgs: 17 active+clean; 455 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-04-29T15:00:55.842 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:55 smithi040 ceph-mon[39581]: pgmap v2952: 17 pgs: 17 active+clean; 455 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-04-29T15:00:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:55 smithi121 ceph-mon[44533]: pgmap v2952: 17 pgs: 17 active+clean; 455 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-04-29T15:00:56.358 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-29T15:00:56.509 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:00:56.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:56.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:00:56.613 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:56.656 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:56.683 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:56.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:56.703 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:00:56.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:56.850 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:56.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:56.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:56.978 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:00:57.022 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:57.050 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:00:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:57.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:00:57.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:00:57.183 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:57.223 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:00:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:57.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:00:57.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:00:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:00:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:57.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:00:57.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:00:57.589 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:00:57.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:57 smithi121 ceph-mon[44533]: pgmap v2953: 17 pgs: 17 active+clean; 282 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 885 op/s 2024-04-29T15:00:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:57.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:00:57.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:00:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:57 smithi194 ceph-mon[44089]: pgmap v2953: 17 pgs: 17 active+clean; 282 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 885 op/s 2024-04-29T15:00:57.868 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:00:57.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:57 smithi040 ceph-mon[39581]: pgmap v2953: 17 pgs: 17 active+clean; 282 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 885 op/s 2024-04-29T15:00:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:00:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:57.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:00:57.892 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:00:57.917 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:00:57.941 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:00:57.971 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:00:57.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:57.998 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:00:58.041 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:58.044 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:00:58.064 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:00:58.087 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:00:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:58.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:00:58.109 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:00:58.132 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:00:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:00:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:00:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:00:58.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:00:58.234 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:58.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:00:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:58.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:00:58.376 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:00:58.408 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is set 2024-04-29T15:00:58.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Removed 160003 objects 2024-04-29T15:00:58.558 INFO:tasks.radosbench.radosbench.2.smithi194.stdout:Clean up completed and total clean up time :195.989 2024-04-29T15:00:58.790 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:00:58.790 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:00:58.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1671126752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:58.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2926396527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:58.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.810 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:58 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1671126752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2926396527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:58 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1671126752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:00:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2926396527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:00:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:58.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:58 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:00:59.020 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:00:59.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:59 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:59 smithi040 ceph-mon[39581]: pgmap v2954: 17 pgs: 17 active+clean; 223 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-04-29T15:00:59.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:59 smithi040 ceph-mon[39581]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:59.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1671126752' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:59.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:00:59 smithi040 ceph-mon[39581]: osdmap e1149: 12 total, 11 up, 7 in 2024-04-29T15:00:59.632 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-29T15:00:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:00:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:59.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:00:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:59 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:59 smithi194 ceph-mon[44089]: pgmap v2954: 17 pgs: 17 active+clean; 223 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-04-29T15:00:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:59 smithi194 ceph-mon[44089]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1671126752' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:59.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:00:59 smithi194 ceph-mon[44089]: osdmap e1149: 12 total, 11 up, 7 in 2024-04-29T15:00:59.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:00:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:59 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:00:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:59 smithi121 ceph-mon[44533]: pgmap v2954: 17 pgs: 17 active+clean; 223 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-04-29T15:00:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:59 smithi121 ceph-mon[44533]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:00:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1671126752' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-29T15:00:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:00:59 smithi121 ceph-mon[44533]: osdmap e1149: 12 total, 11 up, 7 in 2024-04-29T15:00:59.899 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:00:59.900 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:00:59.921 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:00:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:00:59.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:01:00.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:01:00.060 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:01:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:00.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:01:00.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:01:00.216 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:00.766 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:01:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:00.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:01:00.871 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:01:00.900 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:01.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-29T15:01:01.310 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:01.710 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T15:01:01.720 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:01 smithi194 ceph-mon[44089]: pgmap v2956: 17 pgs: 17 active+clean; 44 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-04-29T15:01:01.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:01 smithi040 ceph-mon[39581]: pgmap v2956: 17 pgs: 17 active+clean; 44 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-04-29T15:01:01.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:01 smithi121 ceph-mon[44533]: pgmap v2956: 17 pgs: 17 active+clean; 44 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-04-29T15:01:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:01:03.534 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:01:03.554 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:01:03.575 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:01:03.596 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:01:03.616 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:01:03.636 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:01:03.657 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:01:03.678 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:01:03.698 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:01:03.718 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:01:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:01:03.752 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:01:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:03.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:01:03.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:03 smithi194 ceph-mon[44089]: pgmap v2957: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-04-29T15:01:03.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:03 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3290115756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:01:03.857 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:01:03.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:03 smithi040 ceph-mon[39581]: pgmap v2957: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-04-29T15:01:03.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:03 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3290115756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:01:03.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:03 smithi121 ceph-mon[44533]: pgmap v2957: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-04-29T15:01:03.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:03 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3290115756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:01:03.886 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:04.036 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:01:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:04.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:01:04.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:01:04.176 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:04.197 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:01:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:04.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:01:04.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:01:04.319 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-04-29T15:01:04.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e997/942 cost 1049576)", 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:51:10.506429+0000", 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 593.812540797, 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001124194, 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T15:01:04.323 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.506429+0000", 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.506429+0000", 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.506431+0000", 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2029999999999999e-06 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.506456+0000", 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4465e-05 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.506469+0000", 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3657e-05 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.506477+0000", 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0430000000000008e-06 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:01:04.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.506528+0000", 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1153000000000003e-05 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:10.507553+0000", 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001024673 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPull(7.b e999/942 cost 1049576)", 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-04-29T14:51:14.867730+0000", 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 589.45123986299996, 2024-04-29T15:01:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001093475, 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.867730+0000", 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.867730+0000", 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-04-29T15:01:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.867732+0000", 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4499999999999998e-06 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.867749+0000", 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6245000000000001e-05 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.867765+0000", 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6526999999999999e-05 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-04-29T15:01:04.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.867775+0000", 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9010000000000006e-06 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.867883+0000", 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107773 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-04-29T14:51:14.868823+0000", 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00094057899999999998 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:01:04.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-29T15:01:04.330 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-29T15:01:04.330 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T15:01:04.333 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:04.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:01:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:04.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:01:04.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:01:04.485 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:01:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:04.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:01:04.610 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:01:04.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:04 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.637 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.638 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:04 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.639 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:04.738 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:01:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:04.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:01:04.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.876 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:04 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:04.877 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:04.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:01:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:04.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:01:05.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:01:05.033 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:05.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-29T15:01:05.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:01:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:05.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:01:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:01:05.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:01:05.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:01:05.223 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:05.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:01:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:05.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:01:05.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:01:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:05.588 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:01:05.590 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:05 smithi040 ceph-mon[39581]: pgmap v2958: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-04-29T15:01:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:05.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:01:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:05.639 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:05.692 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:01:05.723 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:05.745 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:01:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:05.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:01:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:05 smithi194 ceph-mon[44089]: pgmap v2958: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-04-29T15:01:05.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:01:05.879 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:05 smithi121 ceph-mon[44533]: pgmap v2958: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-04-29T15:01:06.427 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:01:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:06.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:01:06.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:01:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T15:01:06.564 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:01:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:06.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:01:06.690 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:01:06.723 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:06.745 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:01:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:06.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:01:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:06 smithi194 ceph-mon[44089]: from='client.28753 v1:172.21.15.40:0/805191145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:01:06.850 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:01:06.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:06 smithi040 ceph-mon[39581]: from='client.28753 v1:172.21.15.40:0/805191145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:01:06.880 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:06.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:06 smithi121 ceph-mon[44533]: from='client.28753 v1:172.21.15.40:0/805191145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:01:06.901 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:01:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:06.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:01:07.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:01:07.036 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:07.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-29T15:01:07.384 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:01:07.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:07.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:01:07.489 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:01:07.521 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:07.543 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:01:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:07.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:01:07.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:01:07.681 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:07.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:01:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:07.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:01:07.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:07 smithi040 ceph-mon[39581]: pgmap v2959: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 450 op/s 2024-04-29T15:01:07.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:07 smithi040 ceph-mon[39581]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:01:07.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:07 smithi040 ceph-mon[39581]: from='client.28753 v1:172.21.15.40:0/805191145' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:01:07.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:07 smithi040 ceph-mon[39581]: osdmap e1150: 12 total, 11 up, 7 in 2024-04-29T15:01:07.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:01:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:07 smithi194 ceph-mon[44089]: pgmap v2959: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 450 op/s 2024-04-29T15:01:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:07 smithi194 ceph-mon[44089]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:01:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:07 smithi194 ceph-mon[44089]: from='client.28753 v1:172.21.15.40:0/805191145' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:01:07.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:07 smithi194 ceph-mon[44089]: osdmap e1150: 12 total, 11 up, 7 in 2024-04-29T15:01:07.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:07 smithi121 ceph-mon[44533]: pgmap v2959: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 450 op/s 2024-04-29T15:01:07.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:07 smithi121 ceph-mon[44533]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-29T15:01:07.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:07 smithi121 ceph-mon[44533]: from='client.28753 v1:172.21.15.40:0/805191145' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:01:07.834 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:07 smithi121 ceph-mon[44533]: osdmap e1150: 12 total, 11 up, 7 in 2024-04-29T15:01:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:07.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:01:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:07.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:01:07.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:01:07.989 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:08.008 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:01:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:08.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:01:08.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:01:08.144 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:08.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:01:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:08.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:01:08.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5 2024-04-29T15:01:08.298 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:08.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-29T15:01:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:08.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.5' 2024-04-29T15:01:08.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:01:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:08.573 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:08.576 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:01:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:08.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:01:08.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:01:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:08.753 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-29T15:01:08.753 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-04-29T15:01:08.753 DEBUG:teuthology.orchestra.run.smithi040:osd.1> sudo journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.1.service 2024-04-29T15:01:08.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl start ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.1 2024-04-29T15:01:08.784 INFO:journalctl@ceph.osd.1.smithi040.stdout:-- Logs begin at Mon 2024-04-29 13:48:52 UTC. -- 2024-04-29T15:01:08.858 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:01:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:08.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:01:08.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:01:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:09.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:01:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:09.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:01:09.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:08 smithi194 ceph-mon[44089]: pgmap v2961: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 359 op/s 2024-04-29T15:01:09.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:01:09.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:08 smithi040 ceph-mon[39581]: pgmap v2961: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 359 op/s 2024-04-29T15:01:09.124 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:08 smithi040 systemd[1]: Starting Ceph osd.1 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:01:09.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:08 smithi121 ceph-mon[44533]: pgmap v2961: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 359 op/s 2024-04-29T15:01:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:09.172 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:01:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:09.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:01:09.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:01:09.305 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:01:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:09.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:01:09.428 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:01:09.460 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:09.482 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:01:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:09.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:01:09.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:01:09.614 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:09.634 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:01:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:09.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:01:09.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0 2024-04-29T15:01:09.772 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:10.122 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-29T15:01:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:10.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.0' 2024-04-29T15:01:10.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:01:10.259 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:10.282 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:01:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:10.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:01:10.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:01:10.419 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:10.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:01:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:10.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:01:10.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:01:10.573 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:10.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T15:01:10.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:01:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:10.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:01:10.698 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2 2024-04-29T15:01:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T15:01:10.725 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T15:01:10.726 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T15:01:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T15:01:10.728 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:10.931 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-29T15:01:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:10.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.2' 2024-04-29T15:01:11.035 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:01:11.068 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:11.091 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:01:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:11.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:01:11.195 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9 2024-04-29T15:01:11.226 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:11.247 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9" found: no such container 2024-04-29T15:01:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:11.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.9' 2024-04-29T15:01:11.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:01:11.386 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:11.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-29T15:01:11.491 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:01:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:11.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:01:11.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:01:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:11 smithi040 ceph-mon[39581]: pgmap v2962: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 69 op/s 2024-04-29T15:01:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:11 smithi040 ceph-mon[39581]: 9.f scrub starts 2024-04-29T15:01:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:11 smithi040 ceph-mon[39581]: 9.f scrub ok 2024-04-29T15:01:11.630 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:11.653 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:01:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:11.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:01:11.760 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8 2024-04-29T15:01:11.789 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:11 smithi194 ceph-mon[44089]: pgmap v2962: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 69 op/s 2024-04-29T15:01:11.789 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:11 smithi194 ceph-mon[44089]: 9.f scrub starts 2024-04-29T15:01:11.789 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:11 smithi194 ceph-mon[44089]: 9.f scrub ok 2024-04-29T15:01:11.791 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:11.813 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8" found: no such container 2024-04-29T15:01:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:11.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.8' 2024-04-29T15:01:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:11 smithi121 ceph-mon[44533]: pgmap v2962: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 69 op/s 2024-04-29T15:01:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:11 smithi121 ceph-mon[44533]: 9.f scrub starts 2024-04-29T15:01:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:11 smithi121 ceph-mon[44533]: 9.f scrub ok 2024-04-29T15:01:11.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4 2024-04-29T15:01:11.950 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:11.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-29T15:01:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:11.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.4' 2024-04-29T15:01:12.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:01:12.106 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:12.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:01:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:12.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:01:12.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6 2024-04-29T15:01:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:12.278 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-29T15:01:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:12.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.6' 2024-04-29T15:01:12.382 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11 2024-04-29T15:01:12.413 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:12.436 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11" found: no such container 2024-04-29T15:01:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:12.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.11' 2024-04-29T15:01:12.468 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0: {} 2024-04-29T15:01:12.468 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2: {} 2024-04-29T15:01:12.468 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11: {} 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-29T15:01:12.469 INFO:teuthology.orchestra.run.smithi040.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-29T15:01:12.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-29T15:01:12.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-29T15:01:12.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-29T15:01:12.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-29T15:01:12.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-29T15:01:12.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-29T15:01:12.470 INFO:teuthology.orchestra.run.smithi040.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-29T15:01:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_7' removed 2024-04-29T15:01:12.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10 2024-04-29T15:01:12.569 INFO:teuthology.orchestra.run.smithi194.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:12.591 INFO:teuthology.orchestra.run.smithi194.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10" found: no such container 2024-04-29T15:01:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:12.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi194 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.10' 2024-04-29T15:01:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1467146868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T15:01:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4154477411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:01:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:12 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7 2024-04-29T15:01:12.722 INFO:teuthology.orchestra.run.smithi121.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-29T15:01:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:12.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.7' 2024-04-29T15:01:12.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1467146868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T15:01:12.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4154477411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:01:12.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:12 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3 2024-04-29T15:01:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-29T15:01:12.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1467146868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-29T15:01:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4154477411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-29T15:01:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:12 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:13.097 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:13.524 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-29T15:01:13.525 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-29T15:01:13.525 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-29T15:01:13.525 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-29T15:01:13.525 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-29T15:01:13.525 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-29T15:01:13.525 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-29T15:01:13.525 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:13 smithi040 bash[397651]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-29T15:01:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: pgmap v2963: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:13.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1467146868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T15:01:13.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:13 smithi194 ceph-mon[44089]: osdmap e1151: 12 total, 11 up, 7 in 2024-04-29T15:01:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: pgmap v2963: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1467146868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T15:01:13.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:13 smithi040 ceph-mon[39581]: osdmap e1151: 12 total, 11 up, 7 in 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: pgmap v2963: 17 pgs: 17 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1467146868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-29T15:01:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:13 smithi121 ceph-mon[44533]: osdmap e1151: 12 total, 11 up, 7 in 2024-04-29T15:01:14.029 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-29T15:01:14.039 INFO:tasks.thrashosds:joining thrashosds 2024-04-29T15:01:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:Error: no container with name or ID "ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-29T15:01:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-29T15:01:14.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi040 with status 125: 'sudo docker kill -s 1 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd.3' 2024-04-29T15:01:15.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:14 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:15.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:14 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:15.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:14 smithi040 ceph-mon[39581]: osdmap e1152: 12 total, 11 up, 7 in 2024-04-29T15:01:15.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:14 smithi040 ceph-mon[39581]: pgmap v2966: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:15.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:14 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:15.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:14 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:15.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:14 smithi194 ceph-mon[44089]: osdmap e1152: 12 total, 11 up, 7 in 2024-04-29T15:01:15.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:14 smithi194 ceph-mon[44089]: pgmap v2966: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:15.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:14 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:15.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:14 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:15.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:14 smithi121 ceph-mon[44533]: osdmap e1152: 12 total, 11 up, 7 in 2024-04-29T15:01:15.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:14 smithi121 ceph-mon[44533]: pgmap v2966: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:16.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T15:01:16.662 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:16 smithi040 bash[398172]: 6117be7a7d9fd8f9b17b83b625b1f2c663509e83f792e9243bd9f62493d93462 2024-04-29T15:01:16.662 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:16 smithi040 systemd[1]: Started Ceph osd.1 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T15:01:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-29T15:01:17.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-29T15:01:17.373 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:17 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[398246]: 2024-04-29T15:01:17.089+0000 7f0ab8df77c0 -1 Falling back to public interface 2024-04-29T15:01:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:17 smithi194 ceph-mon[44089]: pgmap v2967: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:17 smithi040 ceph-mon[39581]: pgmap v2967: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:17.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:17 smithi121 ceph-mon[44533]: pgmap v2967: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:18.342 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T15:01:18.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3139476752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:01:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3139476752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:01:18.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3139476752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:01:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:19 smithi194 ceph-mon[44089]: pgmap v2968: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:19 smithi194 ceph-mon[44089]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:01:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:19 smithi194 ceph-mon[44089]: Cluster is now healthy 2024-04-29T15:01:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3139476752' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:01:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:19 smithi194 ceph-mon[44089]: osdmap e1153: 12 total, 11 up, 7 in 2024-04-29T15:01:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:19 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:19.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:19 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:19 smithi040 ceph-mon[39581]: pgmap v2968: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:19 smithi040 ceph-mon[39581]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:01:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:19 smithi040 ceph-mon[39581]: Cluster is now healthy 2024-04-29T15:01:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3139476752' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:01:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:19 smithi040 ceph-mon[39581]: osdmap e1153: 12 total, 11 up, 7 in 2024-04-29T15:01:19.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:19 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:19.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:19 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:19 smithi121 ceph-mon[44533]: pgmap v2968: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:19 smithi121 ceph-mon[44533]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-29T15:01:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:19 smithi121 ceph-mon[44533]: Cluster is now healthy 2024-04-29T15:01:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3139476752' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:01:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:19 smithi121 ceph-mon[44533]: osdmap e1153: 12 total, 11 up, 7 in 2024-04-29T15:01:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:19 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:19.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:19 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:20.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:20 smithi040 ceph-mon[39581]: osdmap e1154: 12 total, 11 up, 7 in 2024-04-29T15:01:20.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:20 smithi194 ceph-mon[44089]: osdmap e1154: 12 total, 11 up, 7 in 2024-04-29T15:01:20.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:20 smithi121 ceph-mon[44533]: osdmap e1154: 12 total, 11 up, 7 in 2024-04-29T15:01:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-04-29T15:01:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:0 2024-04-29T15:01:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:1 2024-04-29T15:01:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2 2024-04-29T15:01:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:abort 2024-04-29T15:01:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:assert 2024-04-29T15:01:21.371 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs debug_inject_read_zeros 2024-04-29T15:01:21.371 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs files list 2024-04-29T15:01:21.371 INFO:teuthology.orchestra.run.smithi040.stderr:bluefs stats 2024-04-29T15:01:21.371 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator dump block 2024-04-29T15:01:21.371 INFO:teuthology.orchestra.run.smithi040.stderr:bluestore allocator fragmentation block 2024-04-29T15:01:21.371 INFO:teuthology.orchestra.run.smithi040.stderr:admin_socket: invalid command 2024-04-29T15:01:21.479 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:21.480 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:21.623 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:21 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[398246]: 2024-04-29T15:01:21.371+0000 7f0ab8df77c0 -1 osd.1 1121 log_to_monitors true 2024-04-29T15:01:21.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:21 smithi040 ceph-mon[39581]: pgmap v2971: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:21.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:21 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:21.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:21 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:21.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:21 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-29T15:01:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:21 smithi194 ceph-mon[44089]: pgmap v2971: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:21 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:21 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:21.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:21 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-29T15:01:21.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:21 smithi121 ceph-mon[44533]: pgmap v2971: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:21.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:21 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:21.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:21 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:21.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:21 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-29T15:01:22.080 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:01:21 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[398246]: 2024-04-29T15:01:21.995+0000 7f0aa6247700 -1 osd.1 1121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-29T15:01:22.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-29T15:01:22.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:22 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-29T15:01:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:22 smithi194 ceph-mon[44089]: osdmap e1155: 12 total, 11 up, 7 in 2024-04-29T15:01:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:22 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T15:01:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:22 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:22.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:22 smithi194 ceph-mon[44089]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' 2024-04-29T15:01:22.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:22 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-29T15:01:22.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:22 smithi040 ceph-mon[39581]: osdmap e1155: 12 total, 11 up, 7 in 2024-04-29T15:01:22.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:22 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T15:01:22.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:22 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:22.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:22 smithi040 ceph-mon[39581]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' 2024-04-29T15:01:22.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:22 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-29T15:01:22.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:22 smithi121 ceph-mon[44533]: osdmap e1155: 12 total, 11 up, 7 in 2024-04-29T15:01:22.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:22 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-29T15:01:22.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:22 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:22.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:22 smithi121 ceph-mon[44533]: from='osd.1 v1:172.21.15.40:6805/3527439899' entity='osd.1' 2024-04-29T15:01:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-29T15:01:23.036 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-29T15:01:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:23.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: pgmap v2973: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:23.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: osd.1 v1:172.21.15.40:6805/3527439899 boot 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: osdmap e1156: 12 total, 12 up, 7 in 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:01:23.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:23 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:01:23.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: pgmap v2973: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:23.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: osd.1 v1:172.21.15.40:6805/3527439899 boot 2024-04-29T15:01:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: osdmap e1156: 12 total, 12 up, 7 in 2024-04-29T15:01:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:01:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:01:23.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:01:23.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:01:23.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:01:23.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:01:23.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:23 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:01:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: pgmap v2973: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: osd.1 v1:172.21.15.40:6805/3527439899 boot 2024-04-29T15:01:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: osdmap e1156: 12 total, 12 up, 7 in 2024-04-29T15:01:23.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:01:23.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:23 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:01:25.314 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-29T15:01:25.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:25.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:25 smithi194 ceph-mon[44089]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:25.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:25 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:25.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:25 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:25.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:25 smithi194 ceph-mon[44089]: osdmap e1157: 12 total, 12 up, 7 in 2024-04-29T15:01:25.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:25 smithi194 ceph-mon[44089]: pgmap v2976: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:25.330 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:25.331 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:25.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:25.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:25 smithi040 ceph-mon[39581]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:25.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:25 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:25.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:25 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:25.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:25 smithi040 ceph-mon[39581]: osdmap e1157: 12 total, 12 up, 7 in 2024-04-29T15:01:25.344 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:25 smithi040 ceph-mon[39581]: pgmap v2976: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:25.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:25.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:25 smithi121 ceph-mon[44533]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-29T15:01:25.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:25 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:25.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:25 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:25.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:25 smithi121 ceph-mon[44533]: osdmap e1157: 12 total, 12 up, 7 in 2024-04-29T15:01:25.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:25 smithi121 ceph-mon[44533]: pgmap v2976: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:26.167 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-29T15:01:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:26.444 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:01:27.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:27.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:27.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:27 smithi194 ceph-mon[44089]: osdmap e1158: 12 total, 12 up, 7 in 2024-04-29T15:01:27.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:27 smithi194 ceph-mon[44089]: pgmap v2978: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:27.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:27.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:27.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:27 smithi040 ceph-mon[39581]: osdmap e1158: 12 total, 12 up, 7 in 2024-04-29T15:01:27.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:27 smithi040 ceph-mon[39581]: pgmap v2978: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:27.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:27.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:27.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:27 smithi121 ceph-mon[44533]: osdmap e1158: 12 total, 12 up, 7 in 2024-04-29T15:01:27.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:27 smithi121 ceph-mon[44533]: pgmap v2978: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:28.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T15:01:28.335 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:28.473 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:28.473 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:29.069 INFO:teuthology.orchestra.run.smithi040.stderr:noscrub is unset 2024-04-29T15:01:29.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-29T15:01:29.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:29 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:29.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:29 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:29.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:29 smithi194 ceph-mon[44089]: osdmap e1159: 12 total, 12 up, 7 in 2024-04-29T15:01:29.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2451791060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:01:29.325 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:29 smithi194 ceph-mon[44089]: pgmap v2980: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:29.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:29 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:29.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:29 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:29.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:29 smithi040 ceph-mon[39581]: osdmap e1159: 12 total, 12 up, 7 in 2024-04-29T15:01:29.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2451791060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:01:29.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:29 smithi040 ceph-mon[39581]: pgmap v2980: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:29.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:29 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:29.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:29 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:29.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:29 smithi121 ceph-mon[44533]: osdmap e1159: 12 total, 12 up, 7 in 2024-04-29T15:01:29.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2451791060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-29T15:01:29.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:29 smithi121 ceph-mon[44533]: pgmap v2980: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:29.397 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:01:29.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-29T15:01:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2451791060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:01:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: osdmap e1160: 12 total, 12 up, 7 in 2024-04-29T15:01:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:01:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:01:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:01:30.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:01:30.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:01:30.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:01:30.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:30 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:01:30.224 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2451791060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:01:30.224 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: osdmap e1160: 12 total, 12 up, 7 in 2024-04-29T15:01:30.224 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:01:30.225 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:01:30.225 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:01:30.225 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:01:30.225 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:01:30.225 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:01:30.225 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:30 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2451791060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: osdmap e1160: 12 total, 12 up, 7 in 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:01:30.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:30 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:01:30.479 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:31.114 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:31.115 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:31.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:31 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:31.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:31 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:31.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:31 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:31.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:31 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:31.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:31 smithi194 ceph-mon[44089]: osdmap e1161: 12 total, 12 up, 7 in 2024-04-29T15:01:31.138 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:31 smithi194 ceph-mon[44089]: pgmap v2983: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:31.225 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T15:01:31.225 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T15:01:31.225 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T15:01:31.225 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T15:01:31.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:31 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:31.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:31 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:31.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:31 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:31.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:31 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:31.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:31 smithi040 ceph-mon[39581]: osdmap e1161: 12 total, 12 up, 7 in 2024-04-29T15:01:31.247 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:31 smithi040 ceph-mon[39581]: pgmap v2983: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:31 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:31 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:31 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:31 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:31 smithi121 ceph-mon[44533]: osdmap e1161: 12 total, 12 up, 7 in 2024-04-29T15:01:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:31 smithi121 ceph-mon[44533]: pgmap v2983: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:31.714 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-29T15:01:31.982 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:01:32.632 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-29T15:01:32.633 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-04-29T15:01:32.633 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-29T15:01:32.633 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-29T15:01:32.633 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-04-29T15:01:33.261 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:01:33.797 INFO:teuthology.orchestra.run.smithi040.stderr:nodeep-scrub is unset 2024-04-29T15:01:33.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:33 smithi194 ceph-mon[44089]: pgmap v2984: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:33.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/818642581' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:01:33.818 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:33 smithi040 ceph-mon[39581]: pgmap v2984: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:33.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/818642581' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:01:33.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:33 smithi121 ceph-mon[44533]: pgmap v2984: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:33.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/818642581' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-29T15:01:34.206 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:34.207 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/818642581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:01:34.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:34 smithi040 ceph-mon[39581]: osdmap e1162: 12 total, 12 up, 7 in 2024-04-29T15:01:34.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:34 smithi040 ceph-mon[39581]: pgmap v2986: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:35.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/818642581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:01:35.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:34 smithi194 ceph-mon[44089]: osdmap e1162: 12 total, 12 up, 7 in 2024-04-29T15:01:35.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:34 smithi194 ceph-mon[44089]: pgmap v2986: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:35.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/818642581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-29T15:01:35.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:34 smithi121 ceph-mon[44533]: osdmap e1162: 12 total, 12 up, 7 in 2024-04-29T15:01:35.137 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:34 smithi121 ceph-mon[44533]: pgmap v2986: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:36.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-29T15:01:36.257 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-29T15:01:36.257 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "" 2024-04-29T15:01:36.257 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-29T15:01:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:01:37.637 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:37 smithi121 ceph-mon[44533]: pgmap v2987: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:37.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:37 smithi194 ceph-mon[44089]: pgmap v2987: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:37 smithi040 ceph-mon[39581]: pgmap v2987: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:38.303 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:38.303 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:38.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-29T15:01:39.225 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:01:39.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:39 smithi194 ceph-mon[44089]: pgmap v2988: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:39.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:39 smithi040 ceph-mon[39581]: pgmap v2988: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:39 smithi121 ceph-mon[44533]: pgmap v2988: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:41.021 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:41.021 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:41.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:41 smithi121 ceph-mon[44533]: pgmap v2989: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:41.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-29T15:01:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:41 smithi194 ceph-mon[44089]: pgmap v2989: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:41.872 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-29T15:01:41.872 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-04-29T15:01:41.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 9 2024-04-29T15:01:41.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:41 smithi040 ceph-mon[39581]: pgmap v2989: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.b/config 2024-04-29T15:01:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:42.679 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:42.679 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:42.680 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:42 smithi040 ceph-mon[39581]: osdmap e1163: 12 total, 12 up, 7 in 2024-04-29T15:01:42.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:42.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:42.811 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:42 smithi121 ceph-mon[44533]: osdmap e1163: 12 total, 12 up, 7 in 2024-04-29T15:01:42.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:42.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:42.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:42 smithi194 ceph-mon[44089]: osdmap e1163: 12 total, 12 up, 7 in 2024-04-29T15:01:43.664 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:43.665 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:43.685 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:43 smithi121 ceph-mon[44533]: pgmap v2991: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:43 smithi194 ceph-mon[44089]: pgmap v2991: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:43.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:43 smithi040 ceph-mon[39581]: pgmap v2991: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:44.410 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-29T15:01:44.551 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.9. 2024-04-29T15:01:44.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:44 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4288568212' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-04-29T15:01:44.694 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:01:44.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:44 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4288568212' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-04-29T15:01:44.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:44 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T15:01:44.373+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1163 definitely_dead 0 2024-04-29T15:01:44.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:44 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4288568212' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-04-29T15:01:45.321 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-04-29T15:01:45.322 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-29T15:01:45.322 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-04-29T15:01:45.322 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 11 2024-04-29T15:01:45.605 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:45.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:45 smithi040 ceph-mon[39581]: pgmap v2992: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:45.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:45 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4288568212' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-04-29T15:01:45.802 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:45 smithi040 ceph-mon[39581]: osdmap e1164: 12 total, 12 up, 8 in 2024-04-29T15:01:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:45 smithi194 ceph-mon[44089]: pgmap v2992: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:45 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4288568212' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-04-29T15:01:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:45 smithi194 ceph-mon[44089]: osdmap e1164: 12 total, 12 up, 8 in 2024-04-29T15:01:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:45 smithi121 ceph-mon[44533]: pgmap v2992: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:45 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4288568212' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-04-29T15:01:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:45 smithi121 ceph-mon[44533]: osdmap e1164: 12 total, 12 up, 8 in 2024-04-29T15:01:46.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:46 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:46.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:46 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:46.590 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:46.590 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:46.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:46 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:46.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:46 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:46 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:46.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:46 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:47.300 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-29T15:01:47.566 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:01:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:47 smithi194 ceph-mon[44089]: pgmap v2994: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:47 smithi194 ceph-mon[44089]: osdmap e1165: 12 total, 12 up, 8 in 2024-04-29T15:01:47.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:47 smithi040 ceph-mon[39581]: pgmap v2994: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:47.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:47 smithi040 ceph-mon[39581]: osdmap e1165: 12 total, 12 up, 8 in 2024-04-29T15:01:47.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:47 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T15:01:47.637+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1165 definitely_dead 0 2024-04-29T15:01:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:47 smithi121 ceph-mon[44533]: pgmap v2994: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:47.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:47 smithi121 ceph-mon[44533]: osdmap e1165: 12 total, 12 up, 8 in 2024-04-29T15:01:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.11. 2024-04-29T15:01:48.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:48 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1606460870' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-29T15:01:48.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:48 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1606460870' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-29T15:01:48.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:48 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1606460870' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-29T15:01:49.370 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-04-29T15:01:49.371 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-29T15:01:49.371 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-04-29T15:01:49.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 1 2024-04-29T15:01:49.406 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:49.406 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:49.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:49 smithi040 ceph-mon[39581]: pgmap v2996: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:49.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1606460870' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-04-29T15:01:49.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:49 smithi040 ceph-mon[39581]: osdmap e1166: 12 total, 12 up, 9 in 2024-04-29T15:01:49.655 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:49.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:49 smithi194 ceph-mon[44089]: pgmap v2996: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:49.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1606460870' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-04-29T15:01:49.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:49 smithi194 ceph-mon[44089]: osdmap e1166: 12 total, 12 up, 9 in 2024-04-29T15:01:49.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:49 smithi121 ceph-mon[44533]: pgmap v2996: 1 pgs: 1 active+clean; 577 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail 2024-04-29T15:01:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1606460870' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-04-29T15:01:49.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:49 smithi121 ceph-mon[44533]: osdmap e1166: 12 total, 12 up, 9 in 2024-04-29T15:01:50.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-29T15:01:50.421 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.c/config 2024-04-29T15:01:51.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:51 smithi040 ceph-mon[39581]: pgmap v2998: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-04-29T15:01:51.725 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:51 smithi194 ceph-mon[44089]: pgmap v2998: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-04-29T15:01:51.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:51 smithi121 ceph-mon[44533]: pgmap v2998: 1 pgs: 1 active+clean; 577 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-04-29T15:01:52.328 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-04-29T15:01:52.329 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-29T15:01:52.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:52 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T15:01:52.194+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1166 definitely_dead 0 2024-04-29T15:01:52.573 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.1. 2024-04-29T15:01:52.725 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:52 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T15:01:52.725 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:52 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T15:01:52.726 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:52 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3897583299' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-29T15:01:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:52 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T15:01:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:52 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T15:01:52.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:52 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3897583299' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-29T15:01:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:52 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T15:01:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:52 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T15:01:52.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:52 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3897583299' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-29T15:01:53.067 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-29T15:01:53.388 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-04-29T15:01:53.388 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-29T15:01:53.388 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-29T15:01:53.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 2 2024-04-29T15:01:53.443 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:53 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:53 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:53 smithi040 ceph-mon[39581]: pgmap v2999: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3897583299' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-29T15:01:53.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:53 smithi040 ceph-mon[39581]: osdmap e1167: 12 total, 12 up, 10 in 2024-04-29T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:53 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:53 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:53 smithi194 ceph-mon[44089]: pgmap v2999: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3897583299' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-29T15:01:53.825 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:53 smithi194 ceph-mon[44089]: osdmap e1167: 12 total, 12 up, 10 in 2024-04-29T15:01:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:53 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:53 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:53 smithi121 ceph-mon[44533]: pgmap v2999: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3897583299' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-29T15:01:53.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:53 smithi121 ceph-mon[44533]: osdmap e1167: 12 total, 12 up, 10 in 2024-04-29T15:01:54.313 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:54.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:54 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:01:54.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:54 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:01:54.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:54 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:01:54.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:54 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:01:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:54 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:01:55.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:54 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:01:55.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:55 smithi040 ceph-mon[39581]: pgmap v3001: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:55.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:55 smithi040 ceph-mon[39581]: osdmap e1168: 12 total, 12 up, 10 in 2024-04-29T15:01:55.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:55 smithi121 ceph-mon[44533]: pgmap v3001: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:55.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:55 smithi121 ceph-mon[44533]: osdmap e1168: 12 total, 12 up, 10 in 2024-04-29T15:01:56.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:55 smithi194 ceph-mon[44089]: pgmap v3001: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:56.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:55 smithi194 ceph-mon[44089]: osdmap e1168: 12 total, 12 up, 10 in 2024-04-29T15:01:56.354 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T15:01:56.354 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T15:01:56.355 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-04-29T15:01:56.355 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T15:01:57.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-29T15:01:57.603 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:01:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:57 smithi040 ceph-mon[39581]: pgmap v3003: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:57.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:57 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T15:01:57.783+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1168 definitely_dead 0 2024-04-29T15:01:57.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:57 smithi121 ceph-mon[44533]: pgmap v3003: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:58.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:57 smithi194 ceph-mon[44089]: pgmap v3003: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 798 GiB / 805 GiB avail 2024-04-29T15:01:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.2. 2024-04-29T15:01:58.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3250479820' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T15:01:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3250479820' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T15:01:59.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3250479820' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-29T15:01:59.767 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T15:01:59.768 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-04-29T15:01:59.768 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-04-29T15:01:59.768 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-04-29T15:01:59.768 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T15:01:59.796 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:59 smithi040 ceph-mon[39581]: pgmap v3004: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T15:01:59.796 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:59 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3250479820' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-29T15:01:59.796 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:01:59 smithi040 ceph-mon[39581]: osdmap e1169: 12 total, 12 up, 11 in 2024-04-29T15:01:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:59 smithi121 ceph-mon[44533]: pgmap v3004: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T15:01:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:59 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3250479820' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-29T15:01:59.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:01:59 smithi121 ceph-mon[44533]: osdmap e1169: 12 total, 12 up, 11 in 2024-04-29T15:02:00.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:59 smithi194 ceph-mon[44089]: pgmap v3004: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T15:02:00.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:59 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3250479820' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-29T15:02:00.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:01:59 smithi194 ceph-mon[44089]: osdmap e1169: 12 total, 12 up, 11 in 2024-04-29T15:02:00.092 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-29T15:02:00.093 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-29T15:02:00.093 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-29T15:02:00.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd in 0 2024-04-29T15:02:00.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-29T15:02:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:01.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:01 smithi194 ceph-mon[44089]: pgmap v3006: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T15:02:01.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:01 smithi040 ceph-mon[39581]: pgmap v3006: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T15:02:01.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:01 smithi121 ceph-mon[44533]: pgmap v3006: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 887 GiB / 894 GiB avail 2024-04-29T15:02:03.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:03 smithi040 ceph-mon[39581]: pgmap v3007: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:03.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:03 smithi121 ceph-mon[44533]: pgmap v3007: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:04.052 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-29T15:02:04.052 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-04-29T15:02:04.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-04-29T15:02:04.052 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-04-29T15:02:04.052 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-29T15:02:04.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:03 smithi194 ceph-mon[44089]: pgmap v3007: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:04.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:04 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T15:02:04.045+0000 7f0ead4dc700 -1 mon.a@0(leader).osd e1169 definitely_dead 0 2024-04-29T15:02:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:marked in osd.0. 2024-04-29T15:02:04.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:04 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3331804161' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-29T15:02:04.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:04 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3331804161' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-29T15:02:05.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:04 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3331804161' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-29T15:02:05.417 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-29T15:02:05.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-29T15:02:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:05 smithi040 ceph-mon[39581]: pgmap v3008: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3331804161' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-29T15:02:05.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:05 smithi040 ceph-mon[39581]: osdmap e1170: 12 total, 12 up, 12 in 2024-04-29T15:02:05.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:05 smithi121 ceph-mon[44533]: pgmap v3008: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3331804161' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-29T15:02:05.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:05 smithi121 ceph-mon[44533]: osdmap e1170: 12 total, 12 up, 12 in 2024-04-29T15:02:06.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:05 smithi194 ceph-mon[44089]: pgmap v3008: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:06.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3331804161' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-29T15:02:06.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:05 smithi194 ceph-mon[44089]: osdmap e1170: 12 total, 12 up, 12 in 2024-04-29T15:02:07.555 INFO:teuthology.orchestra.run.smithi040.stdout:mon.a: {} 2024-04-29T15:02:07.555 INFO:teuthology.orchestra.run.smithi040.stdout:mon.c: {} 2024-04-29T15:02:07.566 INFO:teuthology.orchestra.run.smithi040.stdout:mon.b: {} 2024-04-29T15:02:07.567 INFO:teuthology.orchestra.run.smithi040.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-29T15:02:07.567 INFO:teuthology.orchestra.run.smithi040.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-29T15:02:07.567 INFO:teuthology.orchestra.run.smithi040.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-29T15:02:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: pgmap v3010: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2198321994,config=1059521453,health=3897446751,kv=3334916729}) 2024-04-29T15:02:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3747064071}) 2024-04-29T15:02:07.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3547149234}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=290548462}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3246695086}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3842805879}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077567007}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956294642}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755385328}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087538415}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3144596886}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2004184786}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1649359355}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1221126446}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440358698}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128664823}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479066975}) 2024-04-29T15:02:07.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125435115}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038130952}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359674947}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4105495496}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61669057}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1508622426}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=171346299}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1994807399}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3858298659}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911186289}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875665207}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1271927653}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744760872}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1040201443}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750016987}) 2024-04-29T15:02:07.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187962750}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825326725}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3336090253}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930984814}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473843967}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=964752963}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376565675}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1764052519}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042837568}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612239104}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351857910}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241879333}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4095791338}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913493251}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946111624}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578458259}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4001114777}) 2024-04-29T15:02:07.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=104532234}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445738868}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172473237}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1232866566}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299619938}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2953897392}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454813714}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2159061753}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600879889}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279378973}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=505938602}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2487763978}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=39824174}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3518352261}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2223038002}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906936473}) 2024-04-29T15:02:07.877 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795079275}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513459161}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811386265}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3105361595}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4276700532}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123605717}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791404208}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4291544292}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078269599}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1208947176}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451225726}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730486203}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=694555282}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284865038}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1131977357}) 2024-04-29T15:02:07.878 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=938820236}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2188914629}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3402960635}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1571901380}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3306026796}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085736112}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4025707299}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138628478}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=864896300}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411928361}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1724546129}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520508583}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2290785047}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1265083364}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663041104}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1866381882}) 2024-04-29T15:02:07.879 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342516266}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86670786}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889943078}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373620964}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639360901}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801528871}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3353027105}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=396912586}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494630504}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=896271674}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2804234603}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4262897353}) 2024-04-29T15:02:07.880 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1147808495}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322559741}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2739688109}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1739152276}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4116155324}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655115303}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162755617}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214549348}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426208652}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1043827443}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356156055}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921278956}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3240986771}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1400334239}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723918877}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742137012}) 2024-04-29T15:02:07.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406105773}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1190616869}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=14494242}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252205769}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2604080586}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346341168}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=37665514,mdsmap=2840952586,mgr=174855211}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mgr=16,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=63} crc {mgr=1862919834,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1905048608,mon_config_key=182276460}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=980967877}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4087376447}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3701855171,monmap=455069592,osd_metadata=4136139636,osd_pg_creating=3671540919,osdmap=2287584774}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=51} crc {osdmap=797226951}) 2024-04-29T15:02:07.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:07 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3051702932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T15:02:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: pgmap v3010: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2198321994,config=1059521453,health=3897446751,kv=3334916729}) 2024-04-29T15:02:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3747064071}) 2024-04-29T15:02:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3547149234}) 2024-04-29T15:02:07.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=290548462}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3246695086}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3842805879}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077567007}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956294642}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755385328}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087538415}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3144596886}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2004184786}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1649359355}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1221126446}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440358698}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128664823}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479066975}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125435115}) 2024-04-29T15:02:07.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038130952}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359674947}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4105495496}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61669057}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1508622426}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=171346299}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1994807399}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3858298659}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911186289}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875665207}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1271927653}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744760872}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1040201443}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750016987}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187962750}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825326725}) 2024-04-29T15:02:07.888 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3336090253}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930984814}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473843967}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=964752963}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376565675}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1764052519}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042837568}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612239104}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351857910}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241879333}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4095791338}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913493251}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946111624}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578458259}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4001114777}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=104532234}) 2024-04-29T15:02:07.889 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445738868}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172473237}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1232866566}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299619938}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2953897392}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454813714}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2159061753}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600879889}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279378973}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=505938602}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2487763978}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=39824174}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3518352261}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2223038002}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906936473}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795079275}) 2024-04-29T15:02:07.890 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513459161}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811386265}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3105361595}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4276700532}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123605717}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791404208}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4291544292}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078269599}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1208947176}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451225726}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730486203}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=694555282}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284865038}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1131977357}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=938820236}) 2024-04-29T15:02:07.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2188914629}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3402960635}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1571901380}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3306026796}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085736112}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4025707299}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138628478}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=864896300}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411928361}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1724546129}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520508583}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2290785047}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1265083364}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663041104}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1866381882}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342516266}) 2024-04-29T15:02:07.892 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86670786}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889943078}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373620964}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639360901}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801528871}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3353027105}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=396912586}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494630504}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=896271674}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2804234603}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4262897353}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1147808495}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322559741}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2739688109}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1739152276}) 2024-04-29T15:02:07.893 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4116155324}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655115303}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162755617}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214549348}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426208652}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1043827443}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356156055}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921278956}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3240986771}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1400334239}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723918877}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742137012}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406105773}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1190616869}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=14494242}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252205769}) 2024-04-29T15:02:07.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2604080586}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346341168}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=37665514,mdsmap=2840952586,mgr=174855211}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mgr=16,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=63} crc {mgr=1862919834,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1905048608,mon_config_key=182276460}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=980967877}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4087376447}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3701855171,monmap=455069592,osd_metadata=4136139636,osd_pg_creating=3671540919,osdmap=2287584774}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=51} crc {osdmap=797226951}) 2024-04-29T15:02:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:07 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3051702932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: pgmap v3010: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 976 GiB / 983 GiB avail 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2198321994,config=1059521453,health=3897446751,kv=3334916729}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3747064071}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3547149234}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=290548462}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3246695086}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3842805879}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077567007}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956294642}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755385328}) 2024-04-29T15:02:08.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087538415}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3144596886}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2004184786}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1649359355}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1221126446}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3440358698}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128664823}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479066975}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125435115}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038130952}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359674947}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4105495496}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61669057}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1508622426}) 2024-04-29T15:02:08.076 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=171346299}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1994807399}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3858298659}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911186289}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875665207}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1271927653}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744760872}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1040201443}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750016987}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187962750}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825326725}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3336090253}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930984814}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473843967}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=964752963}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376565675}) 2024-04-29T15:02:08.077 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1764052519}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042837568}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612239104}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351857910}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241879333}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4095791338}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913493251}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946111624}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578458259}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4001114777}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=104532234}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445738868}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172473237}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1232866566}) 2024-04-29T15:02:08.078 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299619938}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2953897392}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454813714}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2159061753}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600879889}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279378973}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=505938602}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2487763978}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=39824174}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3518352261}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2223038002}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906936473}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795079275}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513459161}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811386265}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3105361595}) 2024-04-29T15:02:08.079 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4276700532}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=123605717}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791404208}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4291544292}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078269599}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1208947176}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451225726}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730486203}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=694555282}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284865038}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1131977357}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=938820236}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2188914629}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3402960635}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1571901380}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3306026796}) 2024-04-29T15:02:08.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085736112}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4025707299}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138628478}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=864896300}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411928361}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1724546129}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520508583}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2290785047}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1265083364}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663041104}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1866381882}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=342516266}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86670786}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889943078}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373620964}) 2024-04-29T15:02:08.081 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639360901}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3801528871}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3353027105}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=396912586}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494630504}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=896271674}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2804234603}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4262897353}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1147808495}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322559741}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2739688109}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1739152276}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4116155324}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=655115303}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162755617}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214549348}) 2024-04-29T15:02:08.082 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426208652}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1043827443}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356156055}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921278956}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3240986771}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1400334239}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723918877}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742137012}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406105773}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1190616869}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=14494242}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252205769}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2604080586}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346341168}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=37665514,mdsmap=2840952586,mgr=174855211}) 2024-04-29T15:02:08.083 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mgr=16,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=63} crc {mgr=1862919834,mgr_command_descs=4152233241,mgr_metadata=1121496347,mgrstat=1905048608,mon_config_key=182276460}) 2024-04-29T15:02:08.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=980967877}) 2024-04-29T15:02:08.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4087376447}) 2024-04-29T15:02:08.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3701855171,monmap=455069592,osd_metadata=4136139636,osd_pg_creating=3671540919,osdmap=2287584774}) 2024-04-29T15:02:08.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=51} crc {osdmap=797226951}) 2024-04-29T15:02:08.084 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:07 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3051702932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-29T15:02:08.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-29T15:02:08.655 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:09.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:08 smithi194 ceph-mon[44089]: pgmap v3011: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:09.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:08 smithi040 ceph-mon[39581]: pgmap v3011: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:09.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:08 smithi121 ceph-mon[44533]: pgmap v3011: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:11.539 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-04-29T15:02:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:11 smithi194 ceph-mon[44089]: pgmap v3012: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:11 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/627664123' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T15:02:11.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:11 smithi040 ceph-mon[39581]: pgmap v3012: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:11.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:11 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/627664123' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T15:02:11.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:11 smithi121 ceph-mon[44533]: pgmap v3012: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:11.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:11 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/627664123' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T15:02:12.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 0 1 2024-04-29T15:02:12.553 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:12.800 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:12 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/627664123' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T15:02:12.801 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:12 smithi040 ceph-mon[39581]: osdmap e1171: 12 total, 12 up, 12 in 2024-04-29T15:02:12.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:12 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/627664123' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T15:02:12.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:12 smithi194 ceph-mon[44089]: osdmap e1171: 12 total, 12 up, 12 in 2024-04-29T15:02:12.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:12 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/627664123' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-29T15:02:12.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:12 smithi121 ceph-mon[44533]: osdmap e1171: 12 total, 12 up, 12 in 2024-04-29T15:02:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:13 smithi194 ceph-mon[44089]: pgmap v3014: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:13 smithi040 ceph-mon[39581]: pgmap v3014: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:13.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:13 smithi121 ceph-mon[44533]: pgmap v3014: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:15.546 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-29T15:02:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:15 smithi194 ceph-mon[44089]: pgmap v3015: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2177893345' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T15:02:15.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:15 smithi040 ceph-mon[39581]: pgmap v3015: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:15.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2177893345' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T15:02:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:15 smithi121 ceph-mon[44533]: pgmap v3015: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2177893345' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-29T15:02:16.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-29T15:02:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:16 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2177893345' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-29T15:02:16.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:16 smithi194 ceph-mon[44089]: osdmap e1172: 12 total, 12 up, 12 in 2024-04-29T15:02:16.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:16 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2177893345' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-29T15:02:16.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:16 smithi040 ceph-mon[39581]: osdmap e1172: 12 total, 12 up, 12 in 2024-04-29T15:02:16.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:16 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2177893345' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-29T15:02:16.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:16 smithi121 ceph-mon[44533]: osdmap e1172: 12 total, 12 up, 12 in 2024-04-29T15:02:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:17 smithi194 ceph-mon[44089]: pgmap v3017: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:17 smithi040 ceph-mon[39581]: pgmap v3017: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:17 smithi121 ceph-mon[44533]: pgmap v3017: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:18.813 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-04-29T15:02:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:18 smithi194 ceph-mon[44089]: pgmap v3018: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:19.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3566652273' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T15:02:19.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:18 smithi040 ceph-mon[39581]: pgmap v3018: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:19.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3566652273' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T15:02:19.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:18 smithi121 ceph-mon[44533]: pgmap v3018: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:19.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3566652273' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T15:02:19.559 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-29T15:02:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:19 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3566652273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T15:02:20.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:19 smithi194 ceph-mon[44089]: osdmap e1173: 12 total, 12 up, 12 in 2024-04-29T15:02:20.074 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:19 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3566652273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T15:02:20.074 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:19 smithi040 ceph-mon[39581]: osdmap e1173: 12 total, 12 up, 12 in 2024-04-29T15:02:20.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:19 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3566652273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-29T15:02:20.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:19 smithi121 ceph-mon[44533]: osdmap e1173: 12 total, 12 up, 12 in 2024-04-29T15:02:21.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:20 smithi194 ceph-mon[44089]: pgmap v3020: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:21.085 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:20 smithi040 ceph-mon[39581]: pgmap v3020: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:21.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:20 smithi121 ceph-mon[44533]: pgmap v3020: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:21.930 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-29T15:02:22.253 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/460502985' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T15:02:22.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/460502985' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T15:02:22.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/460502985' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-29T15:02:22.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-29T15:02:22.905 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:23.195 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/460502985' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-29T15:02:23.195 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:22 smithi040 ceph-mon[39581]: osdmap e1174: 12 total, 12 up, 12 in 2024-04-29T15:02:23.195 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:22 smithi040 ceph-mon[39581]: pgmap v3022: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:23.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/460502985' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-29T15:02:23.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:22 smithi194 ceph-mon[44089]: osdmap e1174: 12 total, 12 up, 12 in 2024-04-29T15:02:23.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:22 smithi194 ceph-mon[44089]: pgmap v3022: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:23.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/460502985' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-29T15:02:23.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:22 smithi121 ceph-mon[44533]: osdmap e1174: 12 total, 12 up, 12 in 2024-04-29T15:02:23.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:22 smithi121 ceph-mon[44533]: pgmap v3022: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:25.538 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.4 to 1 (10000) 2024-04-29T15:02:25.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:25 smithi194 ceph-mon[44089]: pgmap v3023: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:25.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/371006048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T15:02:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:25 smithi040 ceph-mon[39581]: pgmap v3023: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/371006048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T15:02:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:25 smithi121 ceph-mon[44533]: pgmap v3023: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/371006048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T15:02:26.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 4 1 2024-04-29T15:02:26.611 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:26.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:26 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/371006048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T15:02:26.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:26 smithi194 ceph-mon[44089]: osdmap e1175: 12 total, 12 up, 12 in 2024-04-29T15:02:26.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:26 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:26.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:26 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:26.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:26 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/371006048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T15:02:26.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:26 smithi040 ceph-mon[39581]: osdmap e1175: 12 total, 12 up, 12 in 2024-04-29T15:02:26.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:26 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:26.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:26 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:26 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/371006048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-29T15:02:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:26 smithi121 ceph-mon[44533]: osdmap e1175: 12 total, 12 up, 12 in 2024-04-29T15:02:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:26 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:26.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:26 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:27.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:27 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:27.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:27 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:27.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:27 smithi040 ceph-mon[39581]: pgmap v3025: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:27.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:27 smithi040 ceph-mon[39581]: osdmap e1176: 12 total, 12 up, 12 in 2024-04-29T15:02:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:27 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:27 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:27 smithi194 ceph-mon[44089]: pgmap v3025: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:27.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:27 smithi194 ceph-mon[44089]: osdmap e1176: 12 total, 12 up, 12 in 2024-04-29T15:02:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:27 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:27 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:27 smithi121 ceph-mon[44533]: pgmap v3025: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:27 smithi121 ceph-mon[44533]: osdmap e1176: 12 total, 12 up, 12 in 2024-04-29T15:02:28.819 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-29T15:02:29.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:28 smithi194 ceph-mon[44089]: pgmap v3027: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:29.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1746828050' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T15:02:29.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:28 smithi040 ceph-mon[39581]: pgmap v3027: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:29.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1746828050' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T15:02:29.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:28 smithi121 ceph-mon[44533]: pgmap v3027: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:29.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1746828050' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-29T15:02:29.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-29T15:02:29.826 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1746828050' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-29T15:02:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:29 smithi194 ceph-mon[44089]: osdmap e1177: 12 total, 12 up, 12 in 2024-04-29T15:02:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:29 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:02:30.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1746828050' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-29T15:02:30.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:29 smithi040 ceph-mon[39581]: osdmap e1177: 12 total, 12 up, 12 in 2024-04-29T15:02:30.108 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:29 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:02:30.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1746828050' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-29T15:02:30.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:29 smithi121 ceph-mon[44533]: osdmap e1177: 12 total, 12 up, 12 in 2024-04-29T15:02:30.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:29 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:02:31.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.374 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:31 smithi040 ceph-mon[39581]: pgmap v3029: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:31.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:02:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:02:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.386 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:31 smithi121 ceph-mon[44533]: pgmap v3029: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:02:31.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:02:31.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:02:31.575 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:31 smithi194 ceph-mon[44089]: pgmap v3029: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:32.252 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.6 to 1 (10000) 2024-04-29T15:02:32.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3101281421' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T15:02:32.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3101281421' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T15:02:32.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3101281421' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T15:02:32.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-29T15:02:33.254 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:33.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3101281421' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T15:02:33.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:33 smithi194 ceph-mon[44089]: osdmap e1178: 12 total, 12 up, 12 in 2024-04-29T15:02:33.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:33 smithi194 ceph-mon[44089]: pgmap v3031: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3101281421' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T15:02:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:33 smithi040 ceph-mon[39581]: osdmap e1178: 12 total, 12 up, 12 in 2024-04-29T15:02:33.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:33 smithi040 ceph-mon[39581]: pgmap v3031: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:33.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3101281421' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-29T15:02:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:33 smithi121 ceph-mon[44533]: osdmap e1178: 12 total, 12 up, 12 in 2024-04-29T15:02:33.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:33 smithi121 ceph-mon[44533]: pgmap v3031: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:34.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:34 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:34.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:34 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:34 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:34.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:34 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:34 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:34.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:34 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:35.269 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-29T15:02:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:35 smithi194 ceph-mon[44089]: osdmap e1179: 12 total, 12 up, 12 in 2024-04-29T15:02:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:35 smithi194 ceph-mon[44089]: pgmap v3033: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:35.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2600955246' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T15:02:35.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:35 smithi040 ceph-mon[39581]: osdmap e1179: 12 total, 12 up, 12 in 2024-04-29T15:02:35.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:35 smithi040 ceph-mon[39581]: pgmap v3033: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:35.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2600955246' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T15:02:35.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:35 smithi121 ceph-mon[44533]: osdmap e1179: 12 total, 12 up, 12 in 2024-04-29T15:02:35.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:35 smithi121 ceph-mon[44533]: pgmap v3033: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:35.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2600955246' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-29T15:02:35.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-29T15:02:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:36.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2600955246' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-29T15:02:36.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:36 smithi194 ceph-mon[44089]: osdmap e1180: 12 total, 12 up, 12 in 2024-04-29T15:02:36.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2600955246' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-29T15:02:36.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:36 smithi040 ceph-mon[39581]: osdmap e1180: 12 total, 12 up, 12 in 2024-04-29T15:02:36.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2600955246' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-29T15:02:36.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:36 smithi121 ceph-mon[44533]: osdmap e1180: 12 total, 12 up, 12 in 2024-04-29T15:02:37.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:37 smithi040 ceph-mon[39581]: pgmap v3035: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:37.574 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:37 smithi194 ceph-mon[44089]: pgmap v3035: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:37.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:37 smithi121 ceph-mon[44533]: pgmap v3035: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.7 to 1 (10000) 2024-04-29T15:02:38.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2232420474' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T15:02:38.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2232420474' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T15:02:38.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2232420474' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T15:02:39.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 7 1 2024-04-29T15:02:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:39 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2232420474' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T15:02:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:39 smithi040 ceph-mon[39581]: osdmap e1181: 12 total, 12 up, 12 in 2024-04-29T15:02:39.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:39 smithi040 ceph-mon[39581]: pgmap v3037: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:39.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:39 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2232420474' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T15:02:39.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:39 smithi121 ceph-mon[44533]: osdmap e1181: 12 total, 12 up, 12 in 2024-04-29T15:02:39.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:39 smithi121 ceph-mon[44533]: pgmap v3037: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:39.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:39 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2232420474' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-29T15:02:39.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:39 smithi194 ceph-mon[44089]: osdmap e1181: 12 total, 12 up, 12 in 2024-04-29T15:02:39.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:39 smithi194 ceph-mon[44089]: pgmap v3037: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:41.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:41 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:41.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:41 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:41.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:41 smithi121 ceph-mon[44533]: osdmap e1182: 12 total, 12 up, 12 in 2024-04-29T15:02:41.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:41 smithi121 ceph-mon[44533]: pgmap v3039: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:41 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:41 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:41 smithi194 ceph-mon[44089]: osdmap e1182: 12 total, 12 up, 12 in 2024-04-29T15:02:41.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:41 smithi194 ceph-mon[44089]: pgmap v3039: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:41.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:41 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:41.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:41 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:41.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:41 smithi040 ceph-mon[39581]: osdmap e1182: 12 total, 12 up, 12 in 2024-04-29T15:02:41.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:41 smithi040 ceph-mon[39581]: pgmap v3039: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-29T15:02:42.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1319420874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T15:02:42.747 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1319420874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T15:02:42.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1319420874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-29T15:02:43.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-29T15:02:43.420 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:43 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1319420874' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-29T15:02:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:43 smithi121 ceph-mon[44533]: osdmap e1183: 12 total, 12 up, 12 in 2024-04-29T15:02:43.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:43 smithi121 ceph-mon[44533]: pgmap v3041: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:43.659 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:43 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1319420874' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-29T15:02:43.660 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:43 smithi040 ceph-mon[39581]: osdmap e1183: 12 total, 12 up, 12 in 2024-04-29T15:02:43.660 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:43 smithi040 ceph-mon[39581]: pgmap v3041: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:43 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1319420874' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-29T15:02:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:43 smithi194 ceph-mon[44089]: osdmap e1183: 12 total, 12 up, 12 in 2024-04-29T15:02:43.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:43 smithi194 ceph-mon[44089]: pgmap v3041: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:44.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:44 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:44.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:44 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:44.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:44 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:44.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:44 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:44.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:44 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:44.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:44 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:45 smithi194 ceph-mon[44089]: osdmap e1184: 12 total, 12 up, 12 in 2024-04-29T15:02:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:45 smithi194 ceph-mon[44089]: pgmap v3043: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:45 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:45.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:45 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:45.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:45 smithi040 ceph-mon[39581]: osdmap e1184: 12 total, 12 up, 12 in 2024-04-29T15:02:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:45 smithi040 ceph-mon[39581]: pgmap v3043: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:45 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:45.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:45 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:45 smithi121 ceph-mon[44533]: osdmap e1184: 12 total, 12 up, 12 in 2024-04-29T15:02:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:45 smithi121 ceph-mon[44533]: pgmap v3043: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:45 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:45.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:45 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:46.403 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.10 to 1 (10000) 2024-04-29T15:02:46.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:46 smithi040 ceph-mon[39581]: osdmap e1185: 12 total, 12 up, 12 in 2024-04-29T15:02:46.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:46 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3941637770' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T15:02:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:46 smithi194 ceph-mon[44089]: osdmap e1185: 12 total, 12 up, 12 in 2024-04-29T15:02:46.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:46 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3941637770' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T15:02:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:46 smithi121 ceph-mon[44533]: osdmap e1185: 12 total, 12 up, 12 in 2024-04-29T15:02:46.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:46 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3941637770' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T15:02:47.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 10 1 2024-04-29T15:02:47.383 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:47.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:47 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3941637770' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T15:02:47.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:47 smithi040 ceph-mon[39581]: osdmap e1186: 12 total, 12 up, 12 in 2024-04-29T15:02:47.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:47 smithi040 ceph-mon[39581]: pgmap v3046: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:47 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3941637770' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T15:02:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:47 smithi194 ceph-mon[44089]: osdmap e1186: 12 total, 12 up, 12 in 2024-04-29T15:02:47.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:47 smithi194 ceph-mon[44089]: pgmap v3046: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:47 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3941637770' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-29T15:02:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:47 smithi121 ceph-mon[44533]: osdmap e1186: 12 total, 12 up, 12 in 2024-04-29T15:02:47.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:47 smithi121 ceph-mon[44533]: pgmap v3046: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:49.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:48 smithi040 ceph-mon[39581]: pgmap v3047: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:49.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:48 smithi194 ceph-mon[44089]: pgmap v3047: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:49.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:48 smithi121 ceph-mon[44533]: pgmap v3047: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:49.833 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-29T15:02:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2294539386' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T15:02:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2294539386' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T15:02:50.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2294539386' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-29T15:02:50.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-29T15:02:50.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:50 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2294539386' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-29T15:02:50.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:50 smithi040 ceph-mon[39581]: osdmap e1187: 12 total, 12 up, 12 in 2024-04-29T15:02:50.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:50 smithi040 ceph-mon[39581]: pgmap v3049: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:50.931 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:51.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:50 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2294539386' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-29T15:02:51.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:50 smithi121 ceph-mon[44533]: osdmap e1187: 12 total, 12 up, 12 in 2024-04-29T15:02:51.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:50 smithi121 ceph-mon[44533]: pgmap v3049: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:51.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:50 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2294539386' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-29T15:02:51.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:50 smithi194 ceph-mon[44089]: osdmap e1187: 12 total, 12 up, 12 in 2024-04-29T15:02:51.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:50 smithi194 ceph-mon[44089]: pgmap v3049: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:51 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:52.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:51 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:52.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:51 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:52.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:51 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:52.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:51 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:52.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:51 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:52 smithi040 ceph-mon[39581]: osdmap e1188: 12 total, 12 up, 12 in 2024-04-29T15:02:53.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:52 smithi040 ceph-mon[39581]: pgmap v3051: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:53.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:52 smithi121 ceph-mon[44533]: osdmap e1188: 12 total, 12 up, 12 in 2024-04-29T15:02:53.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:52 smithi121 ceph-mon[44533]: pgmap v3051: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:53.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:52 smithi194 ceph-mon[44089]: osdmap e1188: 12 total, 12 up, 12 in 2024-04-29T15:02:53.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:52 smithi194 ceph-mon[44089]: pgmap v3051: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:53.858 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.11 to 1 (10000) 2024-04-29T15:02:54.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:53 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3501240589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T15:02:54.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:53 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3501240589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T15:02:54.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:53 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3501240589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T15:02:54.639 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 11 1 2024-04-29T15:02:54.921 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:55.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:54 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3501240589' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T15:02:55.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:54 smithi040 ceph-mon[39581]: osdmap e1189: 12 total, 12 up, 12 in 2024-04-29T15:02:55.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:54 smithi040 ceph-mon[39581]: pgmap v3053: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:55.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:54 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3501240589' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T15:02:55.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:54 smithi121 ceph-mon[44533]: osdmap e1189: 12 total, 12 up, 12 in 2024-04-29T15:02:55.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:54 smithi121 ceph-mon[44533]: pgmap v3053: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:55.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:54 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3501240589' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-29T15:02:55.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:54 smithi194 ceph-mon[44089]: osdmap e1189: 12 total, 12 up, 12 in 2024-04-29T15:02:55.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:54 smithi194 ceph-mon[44089]: pgmap v3053: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:02:56.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:55 smithi040 ceph-mon[39581]: osdmap e1190: 12 total, 12 up, 12 in 2024-04-29T15:02:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:55 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:56.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:55 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:55 smithi121 ceph-mon[44533]: osdmap e1190: 12 total, 12 up, 12 in 2024-04-29T15:02:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:55 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:56.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:55 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:56.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:56.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:56.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:55 smithi194 ceph-mon[44089]: osdmap e1190: 12 total, 12 up, 12 in 2024-04-29T15:02:56.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:55 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:56.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:55 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:57.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:56 smithi040 ceph-mon[39581]: osdmap e1191: 12 total, 12 up, 12 in 2024-04-29T15:02:57.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:56 smithi040 ceph-mon[39581]: pgmap v3056: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-29T15:02:57.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:56 smithi121 ceph-mon[44533]: osdmap e1191: 12 total, 12 up, 12 in 2024-04-29T15:02:57.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:56 smithi121 ceph-mon[44533]: pgmap v3056: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-29T15:02:57.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:56 smithi194 ceph-mon[44089]: osdmap e1191: 12 total, 12 up, 12 in 2024-04-29T15:02:57.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:56 smithi194 ceph-mon[44089]: pgmap v3056: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-29T15:02:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-29T15:02:58.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:57 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2789967842' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T15:02:58.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:57 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2789967842' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T15:02:58.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:57 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2789967842' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-29T15:02:58.612 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-29T15:02:58.896 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:02:59.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:58 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2789967842' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-29T15:02:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:58 smithi121 ceph-mon[44533]: osdmap e1192: 12 total, 12 up, 12 in 2024-04-29T15:02:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:58 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:02:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:58 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:02:59.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:58 smithi121 ceph-mon[44533]: pgmap v3058: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-29T15:02:59.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:58 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2789967842' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-29T15:02:59.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:58 smithi040 ceph-mon[39581]: osdmap e1192: 12 total, 12 up, 12 in 2024-04-29T15:02:59.137 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:58 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:02:59.137 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:58 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:02:59.137 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:58 smithi040 ceph-mon[39581]: pgmap v3058: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-29T15:02:59.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:58 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2789967842' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-29T15:02:59.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:58 smithi194 ceph-mon[44089]: osdmap e1192: 12 total, 12 up, 12 in 2024-04-29T15:02:59.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:58 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:02:59.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:58 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:02:59.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:58 smithi194 ceph-mon[44089]: pgmap v3058: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-29T15:03:00.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:02:59 smithi040 ceph-mon[39581]: osdmap e1193: 12 total, 12 up, 12 in 2024-04-29T15:03:00.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:02:59 smithi121 ceph-mon[44533]: osdmap e1193: 12 total, 12 up, 12 in 2024-04-29T15:03:00.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:02:59 smithi194 ceph-mon[44089]: osdmap e1193: 12 total, 12 up, 12 in 2024-04-29T15:03:01.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:00 smithi194 ceph-mon[44089]: pgmap v3060: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-29T15:03:01.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:00 smithi040 ceph-mon[39581]: pgmap v3060: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-29T15:03:01.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:00 smithi121 ceph-mon[44533]: pgmap v3060: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-29T15:03:01.901 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-04-29T15:03:02.262 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:01 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2386954597' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-29T15:03:02.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:01 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2386954597' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-29T15:03:02.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:01 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2386954597' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-29T15:03:02.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 2 1 2024-04-29T15:03:02.962 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:03.199 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:02 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2386954597' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-29T15:03:03.199 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:02 smithi040 ceph-mon[39581]: osdmap e1194: 12 total, 12 up, 12 in 2024-04-29T15:03:03.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:02 smithi040 ceph-mon[39581]: pgmap v3062: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:03.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:02 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2386954597' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-29T15:03:03.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:02 smithi194 ceph-mon[44089]: osdmap e1194: 12 total, 12 up, 12 in 2024-04-29T15:03:03.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:02 smithi194 ceph-mon[44089]: pgmap v3062: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:02 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2386954597' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-29T15:03:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:02 smithi121 ceph-mon[44533]: osdmap e1194: 12 total, 12 up, 12 in 2024-04-29T15:03:03.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:02 smithi121 ceph-mon[44533]: pgmap v3062: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-29T15:03:05.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:05 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:03:05.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:05 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:03:05.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:05 smithi040 ceph-mon[39581]: osdmap e1195: 12 total, 12 up, 12 in 2024-04-29T15:03:05.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:05 smithi040 ceph-mon[39581]: pgmap v3064: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:05.624 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:05 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2420837382' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-29T15:03:05.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:05 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:03:05.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:05 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:03:05.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:05 smithi121 ceph-mon[44533]: osdmap e1195: 12 total, 12 up, 12 in 2024-04-29T15:03:05.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:05 smithi121 ceph-mon[44533]: pgmap v3064: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:05.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:05 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2420837382' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-29T15:03:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:05 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:03:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:05 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:03:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:05 smithi194 ceph-mon[44089]: osdmap e1195: 12 total, 12 up, 12 in 2024-04-29T15:03:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:05 smithi194 ceph-mon[44089]: pgmap v3064: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:05.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:05 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2420837382' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-29T15:03:06.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-29T15:03:06.390 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:06.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:06 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2420837382' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-29T15:03:06.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:06 smithi040 ceph-mon[39581]: osdmap e1196: 12 total, 12 up, 12 in 2024-04-29T15:03:06.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:06 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2420837382' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-29T15:03:06.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:06 smithi121 ceph-mon[44533]: osdmap e1196: 12 total, 12 up, 12 in 2024-04-29T15:03:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:06 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2420837382' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-29T15:03:06.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:06 smithi194 ceph-mon[44089]: osdmap e1196: 12 total, 12 up, 12 in 2024-04-29T15:03:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:07 smithi040 ceph-mon[39581]: pgmap v3066: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:07.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:07 smithi121 ceph-mon[44533]: pgmap v3066: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:07 smithi194 ceph-mon[44089]: pgmap v3066: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:08.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:08 smithi040 ceph-mon[39581]: 1.0 deep-scrub starts 2024-04-29T15:03:08.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:08 smithi040 ceph-mon[39581]: 1.0 deep-scrub ok 2024-04-29T15:03:08.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:08 smithi121 ceph-mon[44533]: 1.0 deep-scrub starts 2024-04-29T15:03:08.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:08 smithi121 ceph-mon[44533]: 1.0 deep-scrub ok 2024-04-29T15:03:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:08 smithi194 ceph-mon[44089]: 1.0 deep-scrub starts 2024-04-29T15:03:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:08 smithi194 ceph-mon[44089]: 1.0 deep-scrub ok 2024-04-29T15:03:09.362 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.8 to 1 (10000) 2024-04-29T15:03:09.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:09 smithi040 ceph-mon[39581]: pgmap v3067: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:09.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:09 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2993259588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T15:03:09.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:09 smithi121 ceph-mon[44533]: pgmap v3067: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:09.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:09 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2993259588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T15:03:09.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:09 smithi194 ceph-mon[44089]: pgmap v3067: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:09.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:09 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2993259588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T15:03:10.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 8 1 2024-04-29T15:03:10.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:10 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2993259588' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T15:03:10.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:10 smithi040 ceph-mon[39581]: osdmap e1197: 12 total, 12 up, 12 in 2024-04-29T15:03:10.435 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:10.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:10 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2993259588' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T15:03:10.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:10 smithi121 ceph-mon[44533]: osdmap e1197: 12 total, 12 up, 12 in 2024-04-29T15:03:10.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:10 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2993259588' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-29T15:03:10.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:10 smithi194 ceph-mon[44089]: osdmap e1197: 12 total, 12 up, 12 in 2024-04-29T15:03:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:11 smithi040 ceph-mon[39581]: pgmap v3069: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:11 smithi121 ceph-mon[44533]: pgmap v3069: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:11 smithi194 ceph-mon[44089]: pgmap v3069: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:13.554 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-29T15:03:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:13 smithi194 ceph-mon[44089]: pgmap v3070: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:13 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3221366910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T15:03:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:13 smithi040 ceph-mon[39581]: pgmap v3070: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:13.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:13 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3221366910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T15:03:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:13 smithi121 ceph-mon[44533]: pgmap v3070: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:13.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:13 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3221366910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-29T15:03:14.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-29T15:03:14.589 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:14.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:14 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3221366910' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-29T15:03:14.823 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:14 smithi040 ceph-mon[39581]: osdmap e1198: 12 total, 12 up, 12 in 2024-04-29T15:03:14.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:14 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3221366910' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-29T15:03:14.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:14 smithi194 ceph-mon[44089]: osdmap e1198: 12 total, 12 up, 12 in 2024-04-29T15:03:14.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:14 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3221366910' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-29T15:03:14.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:14 smithi121 ceph-mon[44533]: osdmap e1198: 12 total, 12 up, 12 in 2024-04-29T15:03:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:15 smithi194 ceph-mon[44089]: pgmap v3072: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:15.870 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:15 smithi040 ceph-mon[39581]: pgmap v3072: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:15 smithi121 ceph-mon[44533]: pgmap v3072: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:17.568 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.5 to 1 (10000) 2024-04-29T15:03:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:17 smithi194 ceph-mon[44089]: pgmap v3073: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:17.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:17 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/344178261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T15:03:17.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:17 smithi040 ceph-mon[39581]: pgmap v3073: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:17.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:17 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/344178261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T15:03:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:17 smithi121 ceph-mon[44533]: pgmap v3073: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:17 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/344178261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T15:03:18.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-29T15:03:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:18 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/344178261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T15:03:18.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:18 smithi194 ceph-mon[44089]: osdmap e1199: 12 total, 12 up, 12 in 2024-04-29T15:03:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:18 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/344178261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T15:03:18.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:18 smithi040 ceph-mon[39581]: osdmap e1199: 12 total, 12 up, 12 in 2024-04-29T15:03:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:18 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/344178261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-29T15:03:18.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:18 smithi121 ceph-mon[44533]: osdmap e1199: 12 total, 12 up, 12 in 2024-04-29T15:03:18.909 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:19.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:19 smithi040 ceph-mon[39581]: pgmap v3075: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:19.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:19 smithi194 ceph-mon[44089]: pgmap v3075: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:19.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:19 smithi121 ceph-mon[44533]: pgmap v3075: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-29T15:03:21.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:21 smithi040 ceph-mon[39581]: pgmap v3076: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:21.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2616924899' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T15:03:21.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:21 smithi121 ceph-mon[44533]: pgmap v3076: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:21.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2616924899' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T15:03:22.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:21 smithi194 ceph-mon[44089]: pgmap v3076: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:22.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2616924899' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-29T15:03:22.334 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-29T15:03:22.583 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2616924899' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-29T15:03:22.583 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:22 smithi040 ceph-mon[39581]: osdmap e1200: 12 total, 12 up, 12 in 2024-04-29T15:03:22.622 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:22.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2616924899' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-29T15:03:22.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:22 smithi121 ceph-mon[44533]: osdmap e1200: 12 total, 12 up, 12 in 2024-04-29T15:03:23.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2616924899' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-29T15:03:23.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:22 smithi194 ceph-mon[44089]: osdmap e1200: 12 total, 12 up, 12 in 2024-04-29T15:03:23.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:23 smithi040 ceph-mon[39581]: pgmap v3078: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:23.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:23 smithi121 ceph-mon[44533]: pgmap v3078: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:24.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:23 smithi194 ceph-mon[44089]: pgmap v3078: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.9 to 1 (10000) 2024-04-29T15:03:24.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3556266866' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T15:03:24.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3556266866' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T15:03:25.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3556266866' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T15:03:25.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 9 1 2024-04-29T15:03:25.655 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:25 smithi040 ceph-mon[39581]: pgmap v3079: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3556266866' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T15:03:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:25 smithi040 ceph-mon[39581]: osdmap e1201: 12 total, 12 up, 12 in 2024-04-29T15:03:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:25 smithi121 ceph-mon[44533]: pgmap v3079: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3556266866' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T15:03:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:25 smithi121 ceph-mon[44533]: osdmap e1201: 12 total, 12 up, 12 in 2024-04-29T15:03:26.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:25 smithi194 ceph-mon[44089]: pgmap v3079: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:26.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3556266866' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-29T15:03:26.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:25 smithi194 ceph-mon[44089]: osdmap e1201: 12 total, 12 up, 12 in 2024-04-29T15:03:27.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:27 smithi040 ceph-mon[39581]: pgmap v3081: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:27.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:27 smithi121 ceph-mon[44533]: pgmap v3081: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:28.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:27 smithi194 ceph-mon[44089]: pgmap v3081: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:28.608 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-29T15:03:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2076264987' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T15:03:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2076264987' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T15:03:29.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2076264987' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-29T15:03:29.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-29T15:03:29.640 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:29.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:29 smithi040 ceph-mon[39581]: pgmap v3082: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:29.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2076264987' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-29T15:03:29.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:29 smithi040 ceph-mon[39581]: osdmap e1202: 12 total, 12 up, 12 in 2024-04-29T15:03:29.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:29 smithi121 ceph-mon[44533]: pgmap v3082: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:29.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2076264987' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-29T15:03:29.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:29 smithi121 ceph-mon[44533]: osdmap e1202: 12 total, 12 up, 12 in 2024-04-29T15:03:30.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:29 smithi194 ceph-mon[44089]: pgmap v3082: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2076264987' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-29T15:03:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:29 smithi194 ceph-mon[44089]: osdmap e1202: 12 total, 12 up, 12 in 2024-04-29T15:03:30.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:30 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:03:30.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:30 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:03:31.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:30 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: pgmap v3084: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:03:31.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:03:31.874 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:31 smithi040 ceph-mon[39581]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: pgmap v3084: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:03:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:03:31.886 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:31 smithi121 ceph-mon[44533]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: pgmap v3084: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:32.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:03:32.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:03:32.075 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:31 smithi194 ceph-mon[44089]: from='mgr.14153 ' entity='mgr.y' 2024-04-29T15:03:32.621 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-04-29T15:03:32.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:32 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/139081564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T15:03:32.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:32 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/139081564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T15:03:33.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:32 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/139081564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T15:03:33.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-29T15:03:33.670 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:33.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:33 smithi040 ceph-mon[39581]: pgmap v3085: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:33.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:33 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/139081564' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T15:03:33.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:33 smithi040 ceph-mon[39581]: osdmap e1203: 12 total, 12 up, 12 in 2024-04-29T15:03:33.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:33 smithi121 ceph-mon[44533]: pgmap v3085: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:33.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:33 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/139081564' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T15:03:33.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:33 smithi121 ceph-mon[44533]: osdmap e1203: 12 total, 12 up, 12 in 2024-04-29T15:03:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:33 smithi194 ceph-mon[44089]: pgmap v3085: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:33 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/139081564' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-29T15:03:34.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:33 smithi194 ceph-mon[44089]: osdmap e1203: 12 total, 12 up, 12 in 2024-04-29T15:03:35.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:35 smithi040 ceph-mon[39581]: pgmap v3087: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:35 smithi121 ceph-mon[44533]: pgmap v3087: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:36.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:35 smithi194 ceph-mon[44089]: pgmap v3087: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:36.632 INFO:teuthology.orchestra.run.smithi040.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-29T15:03:36.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:36 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/696712218' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T15:03:36.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:36 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/696712218' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T15:03:37.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:36 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/696712218' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-29T15:03:37.428 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-29T15:03:37.428 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-29T15:03:37.428 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-29T15:03:37.428 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-29T15:03:37.428 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-29T15:03:37.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T15:03:37.714 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:37 smithi040 ceph-mon[39581]: pgmap v3088: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:37 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/696712218' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-29T15:03:37.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:37 smithi040 ceph-mon[39581]: osdmap e1204: 12 total, 12 up, 12 in 2024-04-29T15:03:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:37 smithi121 ceph-mon[44533]: pgmap v3088: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:37 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/696712218' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-29T15:03:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:37 smithi121 ceph-mon[44533]: osdmap e1204: 12 total, 12 up, 12 in 2024-04-29T15:03:38.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:37 smithi194 ceph-mon[44089]: pgmap v3088: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:38.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:37 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/696712218' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-29T15:03:38.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:37 smithi194 ceph-mon[44089]: osdmap e1204: 12 total, 12 up, 12 in 2024-04-29T15:03:39.684 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T15:03:39.684 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":1204,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T15:03:36.617891+0000","last_up_change":"2024-04-29T15:01:22.998366+0000","last_in_change":"2024-04-29T15:02:04.586580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":32,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"796","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":937,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":1121,"up_from":1156,"up_thru":1065,"down_at":1122,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":3527439899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":3527439899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":3527439899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":3527439899}]},"public_addr":"172.21.15.40:6805/3527439899","cluster_addr":"172.21.15.40:6806/3527439899","heartbeat_back_addr":"172.21.15.40:6808/3527439899","heartbeat_front_addr":"172.21.15.40:6807/3527439899","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":230,"up_from":260,"up_thru":874,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3416690575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3416690575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3416690575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3416690575}]},"public_addr":"172.21.15.40:6809/3416690575","cluster_addr":"172.21.15.40:6810/3416690575","heartbeat_back_addr":"172.21.15.40:6812/3416690575","heartbeat_front_addr":"172.21.15.40:6811/3416690575","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":1132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":388,"last_clean_end":457,"up_from":561,"up_thru":1133,"down_at":459,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2522198013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2522198013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2522198013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2522198013}]},"public_addr":"172.21.15.121:6804/2522198013","cluster_addr":"172.21.15.121:6805/2522198013","heartbeat_back_addr":"172.21.15.121:6807/2522198013","heartbeat_front_addr":"172.21.15.121:6806/2522198013","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":1132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":64,"last_clean_end":320,"up_from":354,"up_thru":1065,"down_at":321,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":165633203}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":165633203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":165633203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":165633203}]},"public_addr":"172.21.15.194:6800/165633203","cluster_addr":"172.21.15.194:6801/165633203","heartbeat_back_addr":"172.21.15.194:6803/165633203","heartbeat_front_addr":"172.21.15.194:6802/165633203","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":72,"last_clean_end":1019,"up_from":1069,"up_thru":1011,"down_at":1020,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":3807855610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":3807855610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":3807855610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":3807855610}]},"public_addr":"172.21.15.194:6804/3807855610","cluster_addr":"172.21.15.194:6805/3807855610","heartbeat_back_addr":"172.21.15.194:6807/3807855610","heartbeat_front_addr":"172.21.15.194:6806/3807855610","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":524,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:54.756725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:40.170874+0000","dead_epoch":1121},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:51.018060+0000","dead_epoch":230},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:57.922470+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:03:04.282075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:26.342299+0000","dead_epoch":457},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:33.344277+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:55.242868+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:25.887119+0000","dead_epoch":320},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:43.518977+0000","dead_epoch":1019},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:44.849674+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:43.536667+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T15:03:39.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:39 smithi121 ceph-mon[44533]: pgmap v3090: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:40.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:39 smithi194 ceph-mon[44089]: pgmap v3090: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:40.075 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:39 smithi040 ceph-mon[39581]: pgmap v3090: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:40.378 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-29T15:03:40.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-29T15:03:40.658 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:40.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:40 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/850425901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T15:03:41.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:40 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/850425901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T15:03:41.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:40 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/850425901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T15:03:41.846 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:41 smithi040 ceph-mon[39581]: pgmap v3091: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:42.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:41 smithi194 ceph-mon[44089]: pgmap v3091: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:42.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:41 smithi121 ceph-mon[44533]: pgmap v3091: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T15:03:42.597 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":1205,"fsid":"998f2e22-0630-11ef-bc93-c7b262605968","created":"2024-04-29T14:00:01.568926+0000","modified":"2024-04-29T15:03:41.632944+0000","last_up_change":"2024-04-29T15:01:22.998366+0000","last_in_change":"2024-04-29T15:02:04.586580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":32,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-29T14:03:05.389591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"796","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"967eec9c-202c-4f98-a22a-a67db85bd462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":937,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":4255137508}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":4255137508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":4255137508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":4255137508}]},"public_addr":"172.21.15.40:6801/4255137508","cluster_addr":"172.21.15.40:6802/4255137508","heartbeat_back_addr":"172.21.15.40:6804/4255137508","heartbeat_front_addr":"172.21.15.40:6803/4255137508","state":["exists","up"]},{"osd":1,"uuid":"7f52729c-855b-465c-b134-8326f99f1422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":1121,"up_from":1156,"up_thru":1065,"down_at":1122,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":3527439899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":3527439899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":3527439899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":3527439899}]},"public_addr":"172.21.15.40:6805/3527439899","cluster_addr":"172.21.15.40:6806/3527439899","heartbeat_back_addr":"172.21.15.40:6808/3527439899","heartbeat_front_addr":"172.21.15.40:6807/3527439899","state":["exists","up"]},{"osd":2,"uuid":"372baf31-26e1-4865-a272-898e7ff0c2f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":230,"up_from":260,"up_thru":874,"down_at":231,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":3416690575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":3416690575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3416690575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":3416690575}]},"public_addr":"172.21.15.40:6809/3416690575","cluster_addr":"172.21.15.40:6810/3416690575","heartbeat_back_addr":"172.21.15.40:6812/3416690575","heartbeat_front_addr":"172.21.15.40:6811/3416690575","state":["exists","up"]},{"osd":3,"uuid":"975c2fbc-115b-4950-88ef-b0b2ad778765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":1132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":4012292460}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":4012292460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6816","nonce":4012292460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":4012292460}]},"public_addr":"172.21.15.40:6813/4012292460","cluster_addr":"172.21.15.40:6814/4012292460","heartbeat_back_addr":"172.21.15.40:6816/4012292460","heartbeat_front_addr":"172.21.15.40:6815/4012292460","state":["exists","up"]},{"osd":4,"uuid":"6aa13ecc-0959-4915-a788-03c4739da269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":3987960551}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":3987960551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":3987960551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":3987960551}]},"public_addr":"172.21.15.121:6800/3987960551","cluster_addr":"172.21.15.121:6801/3987960551","heartbeat_back_addr":"172.21.15.121:6803/3987960551","heartbeat_front_addr":"172.21.15.121:6802/3987960551","state":["exists","up"]},{"osd":5,"uuid":"6573ccd7-67bf-4b46-a9a6-10d6ad9dbfc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":388,"last_clean_end":457,"up_from":561,"up_thru":1133,"down_at":459,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2522198013}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2522198013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2522198013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2522198013}]},"public_addr":"172.21.15.121:6804/2522198013","cluster_addr":"172.21.15.121:6805/2522198013","heartbeat_back_addr":"172.21.15.121:6807/2522198013","heartbeat_front_addr":"172.21.15.121:6806/2522198013","state":["exists","up"]},{"osd":6,"uuid":"40ecb14f-7b61-45d2-b48c-0a374912edcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":1132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":1755659098}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":1755659098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":1755659098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":1755659098}]},"public_addr":"172.21.15.121:6808/1755659098","cluster_addr":"172.21.15.121:6809/1755659098","heartbeat_back_addr":"172.21.15.121:6811/1755659098","heartbeat_front_addr":"172.21.15.121:6810/1755659098","state":["exists","up"]},{"osd":7,"uuid":"38e15653-4a3d-4543-8240-e17d95bdfe27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":82319362}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":82319362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":82319362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":82319362}]},"public_addr":"172.21.15.121:6812/82319362","cluster_addr":"172.21.15.121:6813/82319362","heartbeat_back_addr":"172.21.15.121:6815/82319362","heartbeat_front_addr":"172.21.15.121:6814/82319362","state":["exists","up"]},{"osd":8,"uuid":"d93f2b45-6c82-4ead-9a66-e86f9e83e0de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":64,"last_clean_end":320,"up_from":354,"up_thru":1065,"down_at":321,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":165633203}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":165633203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":165633203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":165633203}]},"public_addr":"172.21.15.194:6800/165633203","cluster_addr":"172.21.15.194:6801/165633203","heartbeat_back_addr":"172.21.15.194:6803/165633203","heartbeat_front_addr":"172.21.15.194:6802/165633203","state":["exists","up"]},{"osd":9,"uuid":"a720d016-7cf7-4755-a25b-4d081c1f13a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":72,"last_clean_end":1019,"up_from":1069,"up_thru":1011,"down_at":1020,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":3807855610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":3807855610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":3807855610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":3807855610}]},"public_addr":"172.21.15.194:6804/3807855610","cluster_addr":"172.21.15.194:6805/3807855610","heartbeat_back_addr":"172.21.15.194:6807/3807855610","heartbeat_front_addr":"172.21.15.194:6806/3807855610","state":["exists","up"]},{"osd":10,"uuid":"87c864f5-90a1-48e9-bca8-65cf404bc2a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":1065,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":1320459116}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":1320459116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":1320459116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":1320459116}]},"public_addr":"172.21.15.194:6808/1320459116","cluster_addr":"172.21.15.194:6809/1320459116","heartbeat_back_addr":"172.21.15.194:6811/1320459116","heartbeat_front_addr":"172.21.15.194:6810/1320459116","state":["exists","up"]},{"osd":11,"uuid":"5c85066b-5780-4393-be30-0033b9d2dd1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":88,"up_thru":524,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":20977082}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":20977082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":20977082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":20977082}]},"public_addr":"172.21.15.194:6812/20977082","cluster_addr":"172.21.15.194:6813/20977082","heartbeat_back_addr":"172.21.15.194:6815/20977082","heartbeat_front_addr":"172.21.15.194:6814/20977082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:54.756725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:40.170874+0000","dead_epoch":1121},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:51.018060+0000","dead_epoch":230},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:57.922470+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:03:04.282075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:26.342299+0000","dead_epoch":457},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:03:41.612309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:55.242868+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:25.887119+0000","dead_epoch":320},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:43.518977+0000","dead_epoch":1019},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:44.849674+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-29T15:02:43.536667+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/3316779772":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/2003707927":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/2722668908":"2024-04-30T14:00:29.409543+0000","172.21.15.40:6800/2004115894":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/3681930020":"2024-04-30T14:01:01.302559+0000","172.21.15.40:6800/3860863692":"2024-04-30T14:01:01.302559+0000","172.21.15.40:0/3976042573":"2024-04-30T14:00:29.409543+0000","172.21.15.40:0/1925168828":"2024-04-30T14:01:01.302559+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-29T15:03:43.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:42 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:03:43.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:42 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:03:43.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:42 smithi040 ceph-mon[39581]: osdmap e1205: 12 total, 12 up, 12 in 2024-04-29T15:03:43.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:42 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/4125777081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T15:03:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:42 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:03:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:42 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:03:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:42 smithi194 ceph-mon[44089]: osdmap e1205: 12 total, 12 up, 12 in 2024-04-29T15:03:43.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:42 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/4125777081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T15:03:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:42 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:03:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:42 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:03:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:42 smithi121 ceph-mon[44533]: osdmap e1205: 12 total, 12 up, 12 in 2024-04-29T15:03:43.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:42 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/4125777081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-29T15:03:43.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-29T15:03:43.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-29T15:03:43.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-29T15:03:43.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-29T15:03:43.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-29T15:03:43.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-29T15:03:43.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-29T15:03:43.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-29T15:03:43.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-29T15:03:43.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-29T15:03:43.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-29T15:03:43.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-29T15:03:44.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:43 smithi194 ceph-mon[44089]: pgmap v3093: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:44.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:43 smithi040 ceph-mon[39581]: pgmap v3093: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:44.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:43 smithi121 ceph-mon[44533]: pgmap v3093: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:44.262 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.262 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.270 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.270 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.283 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.290 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:44.345 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:45.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:45 smithi040 ceph-mon[39581]: pgmap v3094: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:46.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:45 smithi194 ceph-mon[44089]: pgmap v3094: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:46.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:45 smithi121 ceph-mon[44533]: pgmap v3094: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:47.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:47 smithi040 ceph-mon[39581]: pgmap v3095: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:48.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:47 smithi194 ceph-mon[44089]: pgmap v3095: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:48.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:47 smithi121 ceph-mon[44533]: pgmap v3095: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:03:49.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:48 smithi040 ceph-mon[39581]: pgmap v3096: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:49.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:48 smithi121 ceph-mon[44533]: pgmap v3096: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:49.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:48 smithi194 ceph-mon[44089]: pgmap v3096: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:49.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:03:49.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:03:49.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:49 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:03:49.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:49 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:03:49.946 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:49 smithi040 ceph-mon[39581]: osdmap e1206: 12 total, 12 up, 12 in 2024-04-29T15:03:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:03:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:03:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:49 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:03:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:49 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:03:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:49 smithi121 ceph-mon[44533]: osdmap e1206: 12 total, 12 up, 12 in 2024-04-29T15:03:50.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:03:50.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:03:50.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:49 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:03:50.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:49 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:03:50.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:49 smithi194 ceph-mon[44089]: osdmap e1206: 12 total, 12 up, 12 in 2024-04-29T15:03:51.096 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:50 smithi040 ceph-mon[39581]: pgmap v3098: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:51.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:50 smithi121 ceph-mon[44533]: pgmap v3098: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:51.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:50 smithi194 ceph-mon[44089]: pgmap v3098: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:53.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:53 smithi194 ceph-mon[44089]: pgmap v3099: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:53.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:53 smithi040 ceph-mon[39581]: pgmap v3099: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:53.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:53 smithi121 ceph-mon[44533]: pgmap v3099: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:54.065 INFO:teuthology.orchestra.run.smithi040.stdout:193273529029 2024-04-29T15:03:54.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-29T15:03:55.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:55 smithi040 ceph-mon[39581]: pgmap v3100: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:55.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:55 smithi194 ceph-mon[44089]: pgmap v3100: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:55.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:55 smithi121 ceph-mon[44533]: pgmap v3100: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:57 smithi194 ceph-mon[44089]: pgmap v3101: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:57 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:03:57.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:57 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:03:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:57 smithi040 ceph-mon[39581]: pgmap v3101: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:57 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:03:57.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:57 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:03:57.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:57 smithi121 ceph-mon[44533]: pgmap v3101: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:57.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:57 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:03:57.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:57 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:03:58.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:58 smithi194 ceph-mon[44089]: osdmap e1207: 12 total, 12 up, 12 in 2024-04-29T15:03:58.867 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:58 smithi040 ceph-mon[39581]: osdmap e1207: 12 total, 12 up, 12 in 2024-04-29T15:03:58.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:58 smithi121 ceph-mon[44533]: osdmap e1207: 12 total, 12 up, 12 in 2024-04-29T15:03:59.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:03:59 smithi040 ceph-mon[39581]: pgmap v3103: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:59.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:03:59 smithi194 ceph-mon[44089]: pgmap v3103: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:03:59.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:03:59 smithi121 ceph-mon[44533]: pgmap v3103: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:01.410 INFO:teuthology.orchestra.run.smithi040.stdout:133143986894 2024-04-29T15:04:01.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-29T15:04:01.639 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:01 smithi040 ceph-mon[39581]: pgmap v3104: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:01.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:01 smithi121 ceph-mon[44533]: pgmap v3104: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:02.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:01 smithi194 ceph-mon[44089]: pgmap v3104: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:03.175 INFO:teuthology.orchestra.run.smithi040.stdout:107374183127 2024-04-29T15:04:03.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-29T15:04:03.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:03 smithi040 ceph-mon[39581]: pgmap v3105: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:03.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:03 smithi121 ceph-mon[44533]: pgmap v3105: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:04.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:03 smithi194 ceph-mon[44089]: pgmap v3105: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:05.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:04 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:05.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:04 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:05.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:04 smithi040 ceph-mon[39581]: osdmap e1208: 12 total, 12 up, 12 in 2024-04-29T15:04:05.113 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:04 smithi040 ceph-mon[39581]: pgmap v3107: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:05.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:04 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:05.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:04 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:05.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:04 smithi121 ceph-mon[44533]: osdmap e1208: 12 total, 12 up, 12 in 2024-04-29T15:04:05.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:04 smithi121 ceph-mon[44533]: pgmap v3107: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:05.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:04 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:05.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:04 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:05.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:04 smithi194 ceph-mon[44089]: osdmap e1208: 12 total, 12 up, 12 in 2024-04-29T15:04:05.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:04 smithi194 ceph-mon[44089]: pgmap v3107: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:05.601 INFO:teuthology.orchestra.run.smithi040.stdout:4591320039521 2024-04-29T15:04:05.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-29T15:04:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:07 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:07.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:07 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:07 smithi040 ceph-mon[39581]: osdmap e1209: 12 total, 12 up, 12 in 2024-04-29T15:04:07.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:07 smithi040 ceph-mon[39581]: pgmap v3109: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:07.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:07 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:07.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:07 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:07.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:07 smithi121 ceph-mon[44533]: osdmap e1209: 12 total, 12 up, 12 in 2024-04-29T15:04:07.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:07 smithi121 ceph-mon[44533]: pgmap v3109: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:07.636 INFO:teuthology.orchestra.run.smithi040.stdout:1520418423315 2024-04-29T15:04:07.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-29T15:04:07.721 INFO:teuthology.orchestra.run.smithi040.stdout:377957122736 2024-04-29T15:04:07.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-29T15:04:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:07 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:07.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:07 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:07 smithi194 ceph-mon[44089]: osdmap e1209: 12 total, 12 up, 12 in 2024-04-29T15:04:07.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:07 smithi194 ceph-mon[44089]: pgmap v3109: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:08.089 INFO:teuthology.orchestra.run.smithi040.stdout:4964982194211 2024-04-29T15:04:08.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-29T15:04:08.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:08 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:08.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:08 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:08.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:08 smithi040 ceph-mon[39581]: osdmap e1210: 12 total, 12 up, 12 in 2024-04-29T15:04:08.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:08 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:08.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:08 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:08.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:08 smithi121 ceph-mon[44533]: osdmap e1210: 12 total, 12 up, 12 in 2024-04-29T15:04:08.734 INFO:teuthology.orchestra.run.smithi040.stdout:343597384374 2024-04-29T15:04:08.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-29T15:04:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:08 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:08.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:08 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:08.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:08 smithi194 ceph-mon[44089]: osdmap e1210: 12 total, 12 up, 12 in 2024-04-29T15:04:09.132 INFO:teuthology.orchestra.run.smithi040.stdout:231928234690 2024-04-29T15:04:09.133 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-29T15:04:09.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:09 smithi040 ceph-mon[39581]: pgmap v3111: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:09.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:09 smithi121 ceph-mon[44533]: pgmap v3111: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:09.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:09 smithi194 ceph-mon[44089]: pgmap v3111: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:34359739109 2024-04-29T15:04:09.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-29T15:04:10.370 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:10.425 INFO:teuthology.orchestra.run.smithi040.stdout:2409476653456 2024-04-29T15:04:10.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-29T15:04:10.767 INFO:teuthology.orchestra.run.smithi040.stdout:1116691497542 2024-04-29T15:04:10.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-29T15:04:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:11.023 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:11.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:11 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:11 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:11 smithi040 ceph-mon[39581]: osdmap e1211: 12 total, 12 up, 12 in 2024-04-29T15:04:11.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:11 smithi040 ceph-mon[39581]: pgmap v3113: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:11 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:11 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:11 smithi121 ceph-mon[44533]: osdmap e1211: 12 total, 12 up, 12 in 2024-04-29T15:04:11.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:11 smithi121 ceph-mon[44533]: pgmap v3113: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:11 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:11 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:11 smithi194 ceph-mon[44089]: osdmap e1211: 12 total, 12 up, 12 in 2024-04-29T15:04:11.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:11 smithi194 ceph-mon[44089]: pgmap v3113: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:12.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:12 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:12.623 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:12 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:12.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:12 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:12.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:12 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:12 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:12.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:12 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:13.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:13 smithi040 ceph-mon[39581]: osdmap e1212: 12 total, 12 up, 12 in 2024-04-29T15:04:13.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:13 smithi040 ceph-mon[39581]: pgmap v3115: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:13.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:13 smithi194 ceph-mon[44089]: osdmap e1212: 12 total, 12 up, 12 in 2024-04-29T15:04:13.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:13 smithi194 ceph-mon[44089]: pgmap v3115: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:13.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:13 smithi121 ceph-mon[44533]: osdmap e1212: 12 total, 12 up, 12 in 2024-04-29T15:04:13.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:13 smithi121 ceph-mon[44533]: pgmap v3115: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:15.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:15 smithi194 ceph-mon[44089]: pgmap v3116: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:15.824 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:15 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3025874229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-29T15:04:15.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:15 smithi040 ceph-mon[39581]: pgmap v3116: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:15.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:15 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3025874229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-29T15:04:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:15 smithi121 ceph-mon[44533]: pgmap v3116: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:15.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:15 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3025874229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-29T15:04:16.802 INFO:teuthology.orchestra.run.smithi040.stdout:193273529034 2024-04-29T15:04:17.820 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:17 smithi040 ceph-mon[39581]: pgmap v3117: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:17.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:17 smithi194 ceph-mon[44089]: pgmap v3117: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:17.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:17 smithi121 ceph-mon[44533]: pgmap v3117: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:19.046 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273529029 got 193273529034 for osd.6 2024-04-29T15:04:19.047 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:19.705 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:19 smithi040 ceph-mon[39581]: 1.0 scrub starts 2024-04-29T15:04:19.706 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:19 smithi040 ceph-mon[39581]: 1.0 scrub ok 2024-04-29T15:04:19.706 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:19 smithi040 ceph-mon[39581]: pgmap v3118: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:19 smithi194 ceph-mon[44089]: 1.0 scrub starts 2024-04-29T15:04:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:19 smithi194 ceph-mon[44089]: 1.0 scrub ok 2024-04-29T15:04:19.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:19 smithi194 ceph-mon[44089]: pgmap v3118: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:19.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:19 smithi121 ceph-mon[44533]: 1.0 scrub starts 2024-04-29T15:04:19.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:19 smithi121 ceph-mon[44533]: 1.0 scrub ok 2024-04-29T15:04:19.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:19 smithi121 ceph-mon[44533]: pgmap v3118: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:20.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:20 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:20.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:20 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:20.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:20 smithi040 ceph-mon[39581]: osdmap e1213: 12 total, 12 up, 12 in 2024-04-29T15:04:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:20 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:20 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:20.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:20 smithi194 ceph-mon[44089]: osdmap e1213: 12 total, 12 up, 12 in 2024-04-29T15:04:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:20 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:20.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:20 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:20.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:20 smithi121 ceph-mon[44533]: osdmap e1213: 12 total, 12 up, 12 in 2024-04-29T15:04:21.238 INFO:teuthology.orchestra.run.smithi040.stdout:4591320039526 2024-04-29T15:04:21.397 INFO:teuthology.orchestra.run.smithi040.stdout:107374183132 2024-04-29T15:04:21.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:21 smithi194 ceph-mon[44089]: pgmap v3120: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:21.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2661385670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-29T15:04:21.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:21 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2048453074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-29T15:04:21.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:21 smithi040 ceph-mon[39581]: pgmap v3120: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:21.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2661385670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-29T15:04:21.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:21 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2048453074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-29T15:04:21.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:21 smithi121 ceph-mon[44533]: pgmap v3120: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:21.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2661385670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-29T15:04:21.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:21 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2048453074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-29T15:04:22.031 INFO:teuthology.orchestra.run.smithi040.stdout:4964982194214 2024-04-29T15:04:22.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:22 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:22.837 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:22 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:22.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:22 smithi040 ceph-mon[39581]: osdmap e1214: 12 total, 12 up, 12 in 2024-04-29T15:04:22.838 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:22 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2731749743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-29T15:04:22.839 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374183127 got 107374183132 for osd.3 2024-04-29T15:04:22.839 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:22.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:22 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:22.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:22 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:22.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:22 smithi121 ceph-mon[44533]: osdmap e1214: 12 total, 12 up, 12 in 2024-04-29T15:04:22.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:22 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2731749743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-29T15:04:23.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:22 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:23.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:22 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:23.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:22 smithi194 ceph-mon[44089]: osdmap e1214: 12 total, 12 up, 12 in 2024-04-29T15:04:23.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:22 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2731749743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-29T15:04:23.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:23 smithi040 ceph-mon[39581]: pgmap v3122: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:23.868 INFO:teuthology.orchestra.run.smithi040.stdout:133143986899 2024-04-29T15:04:23.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:23 smithi121 ceph-mon[44533]: pgmap v3122: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:24.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:23 smithi194 ceph-mon[44089]: pgmap v3122: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:24.839 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:24 smithi040 ceph-mon[39581]: purged_snaps scrub starts 2024-04-29T15:04:24.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:24 smithi040 ceph-mon[39581]: purged_snaps scrub ok 2024-04-29T15:04:24.840 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:24 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1013039831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-29T15:04:24.841 INFO:teuthology.orchestra.run.smithi040.stdout:1520418423319 2024-04-29T15:04:24.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:24 smithi121 ceph-mon[44533]: purged_snaps scrub starts 2024-04-29T15:04:24.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:24 smithi121 ceph-mon[44533]: purged_snaps scrub ok 2024-04-29T15:04:24.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:24 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1013039831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-29T15:04:25.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:24 smithi194 ceph-mon[44089]: purged_snaps scrub starts 2024-04-29T15:04:25.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:24 smithi194 ceph-mon[44089]: purged_snaps scrub ok 2024-04-29T15:04:25.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:24 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1013039831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-29T15:04:25.395 INFO:teuthology.orchestra.run.smithi040.stdout:343597384378 2024-04-29T15:04:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:25 smithi040 ceph-mon[39581]: pgmap v3123: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:25 smithi040 ceph-mon[39581]: osdmap e1215: 12 total, 12 up, 12 in 2024-04-29T15:04:25.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/287784237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-29T15:04:25.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:25 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1980599693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-29T15:04:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:25 smithi121 ceph-mon[44533]: pgmap v3123: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:25 smithi121 ceph-mon[44533]: osdmap e1215: 12 total, 12 up, 12 in 2024-04-29T15:04:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/287784237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-29T15:04:25.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:25 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1980599693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-29T15:04:26.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:25 smithi194 ceph-mon[44089]: pgmap v3123: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:26.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:25 smithi194 ceph-mon[44089]: osdmap e1215: 12 total, 12 up, 12 in 2024-04-29T15:04:26.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/287784237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-29T15:04:26.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:25 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1980599693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-29T15:04:26.186 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4964982194211 got 4964982194214 for osd.1 2024-04-29T15:04:26.186 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:27.747 INFO:teuthology.orchestra.run.smithi040.stdout:2409476653460 2024-04-29T15:04:27.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:27 smithi040 ceph-mon[39581]: pgmap v3125: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:27.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:27 smithi121 ceph-mon[44533]: pgmap v3125: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:28.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:27 smithi194 ceph-mon[44089]: pgmap v3125: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:28.220 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597384374 got 343597384378 for osd.10 2024-04-29T15:04:28.221 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:28.737 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1520418423315 got 1520418423319 for osd.8 2024-04-29T15:04:28.737 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:28.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:28 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/763240531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-29T15:04:28.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:28 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/763240531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-29T15:04:29.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:28 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/763240531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-29T15:04:29.167 INFO:teuthology.orchestra.run.smithi040.stdout:377957122741 2024-04-29T15:04:29.327 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986894 got 133143986899 for osd.4 2024-04-29T15:04:29.327 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:29.816 INFO:tasks.cephadm.ceph_manager.ceph:need seq 377957122736 got 377957122741 for osd.11 2024-04-29T15:04:29.817 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:29.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:29 smithi040 ceph-mon[39581]: pgmap v3126: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:29.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:29 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2129065973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-29T15:04:29.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:29 smithi121 ceph-mon[44533]: pgmap v3126: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:29.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:29 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2129065973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-29T15:04:30.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:29 smithi194 ceph-mon[44089]: pgmap v3126: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:30.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:29 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2129065973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-29T15:04:30.524 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:30.566 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4591320039521 got 4591320039526 for osd.9 2024-04-29T15:04:30.566 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:30.802 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:31.342 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:31.381 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2409476653456 got 2409476653460 for osd.5 2024-04-29T15:04:31.381 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:31.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:31 smithi040 ceph-mon[39581]: pgmap v3127: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:31.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:31 smithi121 ceph-mon[44533]: pgmap v3127: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:32.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:31 smithi194 ceph-mon[44089]: pgmap v3127: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:32.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:32 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:32.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:32 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:33.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:32 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:33.675 INFO:teuthology.orchestra.run.smithi040.stdout:1116691497549 2024-04-29T15:04:33.701 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:33 smithi040 ceph-mon[39581]: pgmap v3128: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:33.702 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:04:33.702 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:04:33.702 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:33 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:04:33.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:33 smithi121 ceph-mon[44533]: pgmap v3128: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:33.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:04:33.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:04:33.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:33 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:04:34.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:33 smithi194 ceph-mon[44089]: pgmap v3128: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:34.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-29T15:04:34.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-29T15:04:34.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:33 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-29T15:04:34.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:34 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/2899994669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-29T15:04:34.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:34 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/2899994669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-29T15:04:35.008 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1116691497542 got 1116691497549 for osd.2 2024-04-29T15:04:35.008 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:35.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:34 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/2899994669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-29T15:04:35.162 INFO:teuthology.orchestra.run.smithi040.stdout:231928234696 2024-04-29T15:04:35.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:35 smithi040 ceph-mon[39581]: pgmap v3129: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:35.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:35 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3687304736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-29T15:04:35.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:35 smithi121 ceph-mon[44533]: pgmap v3129: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:35.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:35 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3687304736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-29T15:04:35.978 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234690 got 231928234696 for osd.7 2024-04-29T15:04:35.978 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:36.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:35 smithi194 ceph-mon[44089]: pgmap v3129: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:36.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:35 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3687304736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-29T15:04:37.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:37 smithi040 ceph-mon[39581]: pgmap v3130: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:37.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:37 smithi121 ceph-mon[44533]: pgmap v3130: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:38.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:37 smithi194 ceph-mon[44089]: pgmap v3130: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:38.582 INFO:teuthology.orchestra.run.smithi040.stdout:34359739116 2024-04-29T15:04:39.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:38 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/3372262319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-29T15:04:39.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:38 smithi040 ceph-mon[39581]: pgmap v3131: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:39.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:38 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/3372262319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-29T15:04:39.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:38 smithi121 ceph-mon[44533]: pgmap v3131: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:39.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359739109 got 34359739116 for osd.0 2024-04-29T15:04:39.191 DEBUG:teuthology.parallel:result is None 2024-04-29T15:04:39.191 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-29T15:04:39.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T15:04:39.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:38 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/3372262319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-29T15:04:39.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:38 smithi194 ceph-mon[44089]: pgmap v3131: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:39.800 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:41.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:41 smithi194 ceph-mon[44089]: pgmap v3132: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:41.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:41 smithi040 ceph-mon[39581]: pgmap v3132: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:41.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:41 smithi121 ceph-mon[44533]: pgmap v3132: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T15:04:42.639 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:42.991 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: pgmap v3133: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:42.992 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:42 smithi040 ceph-mon[39581]: from='client.19431 v1:172.21.15.40:0/2224987676' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:04:43.318 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":3133,"stamp":"2024-04-29T15:04:42.502359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":8574016,"kb_used_data":10160,"kb_used_omap":951,"kb_used_meta":8562760,"kb_avail":1116367808,"statfs":{"total":1151940427776,"available":1143160635392,"internally_reserved":0,"allocated":10403840,"data_stored":5166700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":974733,"internal_metadata":8768266355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002070"},"pg_stats":[{"pgid":"1.0","version":"1003'139","reported_seq":2522,"reported_epoch":1215,"state":"active+clean","last_fresh":"2024-04-29T15:04:24.584175+0000","last_change":"2024-04-29T15:04:17.982007+0000","last_active":"2024-04-29T15:04:24.584175+0000","last_peered":"2024-04-29T15:04:24.584175+0000","last_clean":"2024-04-29T15:04:24.584175+0000","last_became_active":"2024-04-29T15:02:54.997514+0000","last_became_peered":"2024-04-29T15:02:54.997514+0000","last_unstale":"2024-04-29T15:04:24.584175+0000","last_undegraded":"2024-04-29T15:04:24.584175+0000","last_fullsized":"2024-04-29T15:04:24.584175+0000","mapping_epoch":1189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":1190,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T15:04:17.981897+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T15:03:07.115164+0000","last_clean_scrub_stamp":"2024-04-29T15:04:17.981897+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:05:22.831510+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740492,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004660,"statfs":{"total":95995035648,"available":95236771840,"internally_reserved":0,"allocated":1310720,"data_stored":871315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":756939204},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1324228,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1323454,"kb_avail":92420924,"statfs":{"total":95995035648,"available":94639026176,"internally_reserved":0,"allocated":778240,"data_stored":342387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1355217344},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":1069,"seq":4591320039530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":90884,"kb_used_data":760,"kb_used_omap":159,"kb_used_meta":89952,"kb_avail":93654268,"statfs":{"total":95995035648,"available":95901970432,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":163395,"internal_metadata":92111293},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":330372,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":329595,"kb_avail":93414780,"statfs":{"total":95995035648,"available":95656734720,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":337505984},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234698,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151436,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":1150142,"kb_avail":92593716,"statfs":{"total":95995035648,"available":94815965184,"internally_reserved":0,"allocated":1310720,"data_stored":871335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1177745866},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273529040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1064836,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92680316,"statfs":{"total":95995035648,"available":94904643584,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1089599942},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":1156,"seq":4964982194218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":62276,"kb_used_data":760,"kb_used_omap":121,"kb_used_meta":61382,"kb_avail":93682876,"statfs":{"total":95995035648,"available":95931265024,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":124476,"internal_metadata":62855620},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739118,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":978756,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92766396,"statfs":{"total":95995035648,"available":94992789504,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497552,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":231620,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":230843,"kb_avail":93513532,"statfs":{"total":95995035648,"available":95757856768,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":236383940},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183137,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":826052,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92919100,"statfs":{"total":95995035648,"available":95149158400,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":845085128},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1141188,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":92603964,"statfs":{"total":95995035648,"available":94826459136,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1167784389},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":631876,"kb_used_data":760,"kb_used_omap":651,"kb_used_meta":630452,"kb_avail":93113276,"statfs":{"total":95995035648,"available":95347994624,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":645583624},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T15:04:43.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-29T15:04:43.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:43.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:43.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:43.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:43.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:43.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: pgmap v3133: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:43.324 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:42 smithi194 ceph-mon[44089]: from='client.19431 v1:172.21.15.40:0/2224987676' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: pgmap v3133: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:43.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:42 smithi121 ceph-mon[44533]: from='client.19431 v1:172.21.15.40:0/2224987676' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:04:43.599 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:45.761 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-29T15:04:45.762 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-29T15:04:45.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:45 smithi194 ceph-mon[44089]: pgmap v3134: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:45.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:45 smithi040 ceph-mon[39581]: pgmap v3134: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:45.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:45 smithi121 ceph-mon[44533]: pgmap v3134: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:46.553 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":3134,"stamp":"2024-04-29T15:04:44.502713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":8574016,"kb_used_data":10160,"kb_used_omap":951,"kb_used_meta":8562760,"kb_avail":1116367808,"statfs":{"total":1151940427776,"available":1143160635392,"internally_reserved":0,"allocated":10403840,"data_stored":5166700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":974733,"internal_metadata":8768266355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002053"},"pg_stats":[{"pgid":"1.0","version":"1003'139","reported_seq":2522,"reported_epoch":1215,"state":"active+clean","last_fresh":"2024-04-29T15:04:24.584175+0000","last_change":"2024-04-29T15:04:17.982007+0000","last_active":"2024-04-29T15:04:24.584175+0000","last_peered":"2024-04-29T15:04:24.584175+0000","last_clean":"2024-04-29T15:04:24.584175+0000","last_became_active":"2024-04-29T15:02:54.997514+0000","last_became_peered":"2024-04-29T15:02:54.997514+0000","last_unstale":"2024-04-29T15:04:24.584175+0000","last_undegraded":"2024-04-29T15:04:24.584175+0000","last_fullsized":"2024-04-29T15:04:24.584175+0000","mapping_epoch":1189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":1190,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'139","last_scrub_stamp":"2024-04-29T15:04:17.981897+0000","last_deep_scrub":"1003'139","last_deep_scrub_stamp":"2024-04-29T15:03:07.115164+0000","last_clean_scrub_stamp":"2024-04-29T15:04:17.981897+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:05:22.831510+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":88,"seq":377957122744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740492,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004660,"statfs":{"total":95995035648,"available":95236771840,"internally_reserved":0,"allocated":1310720,"data_stored":871315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":756939204},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":80,"seq":343597384382,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1324228,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1323454,"kb_avail":92420924,"statfs":{"total":95995035648,"available":94639026176,"internally_reserved":0,"allocated":778240,"data_stored":342387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1355217344},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":1069,"seq":4591320039531,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":90884,"kb_used_data":760,"kb_used_omap":159,"kb_used_meta":89952,"kb_avail":93654268,"statfs":{"total":95995035648,"available":95901970432,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":163395,"internal_metadata":92111293},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":354,"seq":1520418423323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":330372,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":329595,"kb_avail":93414780,"statfs":{"total":95995035648,"available":95656734720,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4416,"internal_metadata":337505984},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234698,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151436,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":1150142,"kb_avail":92593716,"statfs":{"total":95995035648,"available":94815965184,"internally_reserved":0,"allocated":1310720,"data_stored":871335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1177745866},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273529040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1064836,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92680316,"statfs":{"total":95995035648,"available":94904643584,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1089599942},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":1156,"seq":4964982194219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":62276,"kb_used_data":760,"kb_used_omap":121,"kb_used_meta":61382,"kb_avail":93682876,"statfs":{"total":95995035648,"available":95931265024,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":124476,"internal_metadata":62855620},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739118,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":978756,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92766396,"statfs":{"total":95995035648,"available":94992789504,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":260,"seq":1116691497552,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":231620,"kb_used_data":760,"kb_used_omap":4,"kb_used_meta":230843,"kb_avail":93513532,"statfs":{"total":95995035648,"available":95757856768,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4412,"internal_metadata":236383940},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374183138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":826052,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92919100,"statfs":{"total":95995035648,"available":95149158400,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":845085128},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":1141188,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":92603964,"statfs":{"total":95995035648,"available":94826459136,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1167784389},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":561,"seq":2409476653463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":631876,"kb_used_data":760,"kb_used_omap":651,"kb_used_meta":630452,"kb_avail":93113276,"statfs":{"total":95995035648,"available":95347994624,"internally_reserved":0,"allocated":778240,"data_stored":342407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":666872,"internal_metadata":645583624},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-29T15:04:46.554 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-29T15:04:46.554 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7fe685c922bae3b52755209e35feddd972749cff shell --fsid 998f2e22-0630-11ef-bc93-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-29T15:04:46.836 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/mon.a/config 2024-04-29T15:04:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:47 smithi194 ceph-mon[44089]: from='client.19437 v1:172.21.15.40:0/2782224434' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:04:47.823 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:47 smithi194 ceph-mon[44089]: pgmap v3135: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:47.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:47 smithi040 ceph-mon[39581]: from='client.19437 v1:172.21.15.40:0/2782224434' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:04:47.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:47 smithi040 ceph-mon[39581]: pgmap v3135: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:47 smithi121 ceph-mon[44533]: from='client.19437 v1:172.21.15.40:0/2782224434' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-29T15:04:47.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:47 smithi121 ceph-mon[44533]: pgmap v3135: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:49.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:48 smithi040 ceph-mon[39581]: pgmap v3136: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:49.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:48 smithi121 ceph-mon[44533]: pgmap v3136: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:49.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:48 smithi194 ceph-mon[44089]: pgmap v3136: 1 pgs: 1 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-29T15:04:49.609 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-29T15:04:49.619 INFO:tasks.cephadm:Teardown begin 2024-04-29T15:04:49.619 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-29T15:04:49.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-29T15:04:49.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-29T15:04:49.759 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-29T15:04:49.759 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-29T15:04:49.785 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-29T15:04:49.809 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-29T15:04:49.840 INFO:tasks.cephadm:Stopping all daemons... 2024-04-29T15:04:49.840 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-29T15:04:49.840 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a 2024-04-29T15:04:50.073 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1478218402' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-29T15:04:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:49 smithi194 ceph-mon[44089]: from='client.? v1:172.21.15.40:0/1478218402' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-29T15:04:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:50.074 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:49 smithi194 ceph-mon[44089]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1478218402' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-29T15:04:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:49 smithi040 ceph-mon[39581]: from='client.? v1:172.21.15.40:0/1478218402' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-29T15:04:50.122 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:49 smithi040 ceph-mon[39581]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:50.123 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:49 smithi040 systemd[1]: Stopping Ceph mon.a for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:04:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1478218402' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-29T15:04:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:49 smithi121 ceph-mon[44533]: from='client.? v1:172.21.15.40:0/1478218402' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-29T15:04:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-29T15:04:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-29T15:04:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-29T15:04:50.135 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:49 smithi121 ceph-mon[44533]: from='mgr.14153 v1:172.21.15.40:0/1111425110' entity='mgr.y' 2024-04-29T15:04:50.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:50 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T15:04:50.127+0000 7f0eb2ce7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:04:50.515 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:50 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a[39558]: 2024-04-29T15:04:50.127+0000 7f0eb2ce7700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-29T15:04:50.872 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 29 15:04:50 smithi040 bash[417759]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-a 2024-04-29T15:04:50.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.a.service' 2024-04-29T15:04:51.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:04:51.006 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-29T15:04:51.007 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-29T15:04:51.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.b 2024-04-29T15:04:51.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:51 smithi121 systemd[1]: Stopping Ceph mon.b for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:04:51.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:51 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-b[44510]: 2024-04-29T15:04:51.291+0000 7f71f6ed4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:04:51.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:51 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-b[44510]: 2024-04-29T15:04:51.291+0000 7f71f6ed4700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-29T15:04:52.134 INFO:journalctl@ceph.mon.b.smithi121.stdout:Apr 29 15:04:51 smithi121 bash[210038]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-b 2024-04-29T15:04:52.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.b.service' 2024-04-29T15:04:52.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:04:52.318 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-29T15:04:52.318 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-29T15:04:52.318 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.c 2024-04-29T15:04:52.653 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:52 smithi194 systemd[1]: Stopping Ceph mon.c for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:04:52.963 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-c[44066]: 2024-04-29T15:04:52.647+0000 7faa8f2e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:04:52.963 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-c[44066]: 2024-04-29T15:04:52.647+0000 7faa8f2e6700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-29T15:04:53.323 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 29 15:04:53 smithi194 bash[212114]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-mon-c 2024-04-29T15:04:53.511 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mon.c.service' 2024-04-29T15:04:53.553 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:04:53.553 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-29T15:04:53.553 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-29T15:04:53.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.y 2024-04-29T15:04:53.840 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 15:04:53 smithi040 systemd[1]: Stopping Ceph mgr.y for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:04:54.122 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 15:04:53 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y[39838]: 2024-04-29T15:04:53.835+0000 7f4cdf2f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:04:54.622 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 29 15:04:54 smithi040 bash[417904]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-y 2024-04-29T15:04:54.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-29T15:04:54.731 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:04:54.731 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-29T15:04:54.731 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-29T15:04:54.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.x 2024-04-29T15:04:55.135 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 15:04:54 smithi121 systemd[1]: Stopping Ceph mgr.x for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:04:55.635 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 15:04:55 smithi121 bash[210195]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-mgr-x 2024-04-29T15:04:55.747 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 15:04:55 smithi121 systemd[1]: ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-29T15:04:55.747 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Apr 29 15:04:55 smithi121 systemd[1]: ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-29T15:04:55.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-29T15:04:55.784 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:04:55.785 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-29T15:04:55.785 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-29T15:04:55.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.0 2024-04-29T15:04:56.073 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 15:04:55 smithi040 systemd[1]: Stopping Ceph osd.0 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:04:56.372 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 15:04:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-0[58500]: 2024-04-29T15:04:56.068+0000 7fcdbb753700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:04:56.372 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 15:04:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-0[58500]: 2024-04-29T15:04:56.068+0000 7fcdbb753700 -1 osd.0 1215 *** Got signal Terminated *** 2024-04-29T15:04:56.372 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 15:04:56 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-0[58500]: 2024-04-29T15:04:56.068+0000 7fcdbb753700 -1 osd.0 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:02.373 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 29 15:05:02 smithi040 bash[418049]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-0 2024-04-29T15:05:04.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.0.service' 2024-04-29T15:05:04.117 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:05:04.117 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-29T15:05:04.117 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-29T15:05:04.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.1 2024-04-29T15:05:04.372 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:05:04 smithi040 systemd[1]: Stopping Ceph osd.1 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:05:04.872 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:05:04 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[398246]: 2024-04-29T15:05:04.431+0000 7f0aaf45e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:05:04.872 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:05:04 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[398246]: 2024-04-29T15:05:04.431+0000 7f0aaf45e700 -1 osd.1 1215 *** Got signal Terminated *** 2024-04-29T15:05:04.872 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:05:04 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1[398246]: 2024-04-29T15:05:04.431+0000 7f0aaf45e700 -1 osd.1 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:10.622 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 29 15:05:10 smithi040 bash[418318]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-1 2024-04-29T15:05:12.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.1.service' 2024-04-29T15:05:12.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:05:12.090 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-29T15:05:12.090 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-29T15:05:12.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2 2024-04-29T15:05:12.372 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 15:05:12 smithi040 systemd[1]: Stopping Ceph osd.2 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:05:12.872 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 15:05:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[136028]: 2024-04-29T15:05:12.396+0000 7f9830c84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:05:12.872 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 15:05:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[136028]: 2024-04-29T15:05:12.396+0000 7f9830c84700 -1 osd.2 1215 *** Got signal Terminated *** 2024-04-29T15:05:12.872 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 15:05:12 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2[136028]: 2024-04-29T15:05:12.396+0000 7f9830c84700 -1 osd.2 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:18.622 INFO:journalctl@ceph.osd.2.smithi040.stdout:Apr 29 15:05:18 smithi040 bash[418586]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-2 2024-04-29T15:05:20.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.2.service' 2024-04-29T15:05:20.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:05:20.465 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-29T15:05:20.466 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-29T15:05:20.466 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.3 2024-04-29T15:05:20.741 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 29 15:05:20 smithi040 systemd[1]: Stopping Ceph osd.3 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:05:21.122 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 29 15:05:20 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-3[80949]: 2024-04-29T15:05:20.737+0000 7f4509f7b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:05:21.122 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 29 15:05:20 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-3[80949]: 2024-04-29T15:05:20.737+0000 7f4509f7b700 -1 osd.3 1215 *** Got signal Terminated *** 2024-04-29T15:05:21.122 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 29 15:05:20 smithi040 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-3[80949]: 2024-04-29T15:05:20.737+0000 7f4509f7b700 -1 osd.3 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:23.073 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:22.699+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:23.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:23.296+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:23.990 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:23.703+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:24.253 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:23.987+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:24.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:24.251+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:24.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:24 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:24.388+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:25.022 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:24.750+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:25.291 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:25.019+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:25.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:25.288+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:25.634 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:25 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:25.414+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:25.635 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:25 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:25.268+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:26.073 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:25.780+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:26.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:25.990+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:26.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:26.246+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:26.634 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:26 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:26.283+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:26.635 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:26 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:26.460+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:26.872 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 29 15:05:26 smithi040 bash[418843]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-3 2024-04-29T15:05:27.073 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:26.755+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:27.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:26.972+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:27.506 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:27 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:27.245+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:27.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:27.206+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:27.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:27 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:27.502+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:27.989 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:27.717+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:28.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:28.175+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:28.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:27.986+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:28.552 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:28 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:28.241+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:28.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.3.service' 2024-04-29T15:05:28.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:28 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:28.549+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:28.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:05:28.915 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-29T15:05:28.915 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-29T15:05:28.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.4 2024-04-29T15:05:28.993 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:28.710+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:29.238 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 15:05:28 smithi121 systemd[1]: Stopping Ceph osd.4 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:05:29.238 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 15:05:29 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4[49952]: 2024-04-29T15:05:29.178+0000 7f07a5fff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:05:29.238 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 15:05:29 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4[49952]: 2024-04-29T15:05:29.178+0000 7f07a5fff700 -1 osd.4 1215 *** Got signal Terminated *** 2024-04-29T15:05:29.238 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 15:05:29 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4[49952]: 2024-04-29T15:05:29.178+0000 7f07a5fff700 -1 osd.4 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:29.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:28.991+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:29.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:29.136+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:29.593 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:29 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:29.234+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:29.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:29 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:29.589+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:29.994 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:29.720+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:30.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:30.181+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:29.992+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:30.628 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:30 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:30.259+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:30.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:30 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:30.625+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:30.991 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:30.731+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:31.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:31 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:31.146+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:30.988+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:31.596 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:31 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:31.270+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:31.597 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:31 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:31.461+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:31.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:31 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:31.593+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:32.036 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:31 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:31.703+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:32.036 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:31 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:31.587+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:32.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:32 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:32.032+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:32.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:32 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:32.191+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:32.594 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 15:05:32 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4[49952]: 2024-04-29T15:05:32.284+0000 7f07a25f3700 -1 osd.4 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.089818+0000 front 2024-04-29T15:05:06.089949+0000 (oldest deadline 2024-04-29T15:05:31.389995+0000) 2024-04-29T15:05:32.595 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:32 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:32.312+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:32.595 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:32 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:32.481+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:32.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:32 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:32.590+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:33.035 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:32 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:32.570+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:33.035 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:32 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:32.657+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:33.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:33 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:33.192+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:33.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:33 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:33.031+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:33.611 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:33 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:33.491+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:33.612 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 15:05:33 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4[49952]: 2024-04-29T15:05:33.310+0000 7f07a25f3700 -1 osd.4 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.089818+0000 front 2024-04-29T15:05:06.089949+0000 (oldest deadline 2024-04-29T15:05:31.389995+0000) 2024-04-29T15:05:33.612 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:33 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:33.280+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:33.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:33 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:33.608+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:34.024 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:33 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:33.659+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:34.025 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:33 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:33.596+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:34.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:34 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:34.236+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:34.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:34 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:34.021+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:34.564 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:34 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:34.261+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:34.565 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:34 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:34.464+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:34.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:34 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:34.561+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:35.027 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:34 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:34.629+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:35.027 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:34 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:34.671+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:35.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:35 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:35.245+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:35.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:35 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:35.022+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:35.430 INFO:journalctl@ceph.osd.4.smithi121.stdout:Apr 29 15:05:35 smithi121 bash[210349]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-4 2024-04-29T15:05:35.430 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:35 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:35.275+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:35.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:35 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:35.548+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:35.884 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:35 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:35.425+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:35.992 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:35 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:35.651+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:35.992 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:35 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:35.639+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:36.284 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:35 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:35.988+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:36.384 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:36 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:36.311+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:36.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:36 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:36.281+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:36.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:36 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:36.530+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:36.885 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:36 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:36.418+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:36.967 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:36 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:36.689+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:36.967 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:36 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:36.618+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:37.257 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:36 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:36.963+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:37.547 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:37 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:37.272+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:37.547 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:37 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:37.381+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:37.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:37 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:37.254+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:37.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.4.service' 2024-04-29T15:05:37.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:05:37.692 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-29T15:05:37.692 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-29T15:05:37.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5 2024-04-29T15:05:37.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:37 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:37.544+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:37.884 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:37 smithi121 systemd[1]: Stopping Ceph osd.5 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:05:37.945 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:37 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:37.739+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:37.945 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:37 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:37.581+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:38.231 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:37 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:37.941+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:38.315 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:37 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:37.997+0000 7f95118ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:05:38.315 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:37 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:37.997+0000 7f95118ec700 -1 osd.5 1215 *** Got signal Terminated *** 2024-04-29T15:05:38.315 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:37 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:37.997+0000 7f95118ec700 -1 osd.5 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:38.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:38 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:38.228+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:38.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:38 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:38.228+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:38.634 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:38 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:38.312+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:38.634 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:38 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:38.552+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:38.635 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:38 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:38.395+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:38.969 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:38 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:38.600+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:38.969 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:38 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:38.703+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:38.969 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:38 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:38.703+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:39.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:38 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:38.965+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:39.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:39 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:39.203+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:39.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:39 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:39.203+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:39.589 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:39 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:39.283+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:39.589 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:39 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:39.418+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:39.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:39 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:39.586+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:39.954 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:39 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:39.624+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:39.954 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:39 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:39.624+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:39.955 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:39 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:39.716+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:39.955 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:39 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:39.716+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:40.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:39 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:39.951+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:40.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:40 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:40.172+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:40.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:40 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:40.172+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:40.608 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:40 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:40.261+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:40.608 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:40 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:40.261+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:40.608 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:40 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:40.384+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:40.608 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:40 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:40.384+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.759569+0000 front 2024-04-29T15:05:16.759489+0000 (oldest deadline 2024-04-29T15:05:40.259577+0000) 2024-04-29T15:05:40.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:40 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:40.604+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:40.920 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:40 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:40.643+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:40.920 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:40 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:40.643+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:40.920 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:40 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:40.755+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:40.920 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:40 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:40.755+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:41.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:40 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:40.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:41.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:41.136+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:41.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:41.136+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:41.581 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:41 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:41.302+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:41.581 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:41 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:41.302+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:41.581 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:41 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:41.382+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:41.581 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:41 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:41.382+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.759569+0000 front 2024-04-29T15:05:16.759489+0000 (oldest deadline 2024-04-29T15:05:40.259577+0000) 2024-04-29T15:05:41.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:41 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:41.577+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:42.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:41.905+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:42.073 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:41.666+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:42.074 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:41.666+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:42.074 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:41.797+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:42.074 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:41 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:41.797+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:42.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:42.150+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:42.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:42.150+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:42.585 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:42 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:42.259+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:42.585 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:42 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:42.259+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:42.585 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:42 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:42.348+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:06.857931+0000 front 2024-04-29T15:05:06.857810+0000 (oldest deadline 2024-04-29T15:05:30.957806+0000) 2024-04-29T15:05:42.585 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:42 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5[137220]: 2024-04-29T15:05:42.348+0000 7f950dee0700 -1 osd.5 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.759569+0000 front 2024-04-29T15:05:16.759489+0000 (oldest deadline 2024-04-29T15:05:40.259577+0000) 2024-04-29T15:05:42.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:42 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:42.581+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:42.947 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:42.664+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:42.948 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:42.664+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:42.948 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:42.769+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:42.948 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:42.769+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:43.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:42 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:42.944+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:43.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:43.162+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:43.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:43.162+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:43.601 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:43 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:43.264+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:43.601 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:43 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:43.264+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:43.878 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:43 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:43.598+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:43.878 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:43 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:43.598+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:43.970 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:43.665+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:43.970 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:43.665+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:43.970 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:43.760+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:43.970 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:43.760+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:44.134 INFO:journalctl@ceph.osd.5.smithi121.stdout:Apr 29 15:05:44 smithi121 bash[210619]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-5 2024-04-29T15:05:44.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:44.196+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:44.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:44.196+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:44.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:43.967+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:44.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:43 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:43.967+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:44.604 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:44 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:44.311+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:44.604 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:44 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:44.311+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:44.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:44 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:44.601+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:44.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:44 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:44.601+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:44.982 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:44.785+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:44.983 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:44.785+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:44.983 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:44.672+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:44.983 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:44.672+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:45.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:45.186+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:45.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:45.186+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:45.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:44.979+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:45.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:44 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:44.979+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:45.460 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:45 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:45.306+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:45.460 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:45 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:45.306+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:45.721 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:45 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:45.610+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:45.721 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:45 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:45.610+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:45.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.5.service' 2024-04-29T15:05:45.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:05:45.775 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-29T15:05:45.775 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-29T15:05:45.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.6 2024-04-29T15:05:45.963 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:45.693+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:45.964 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:45.693+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:45.964 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:45.817+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:45.964 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:45.817+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:46.045 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:45 smithi121 systemd[1]: Stopping Ceph osd.6 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:05:46.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:46.143+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:46.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:46.143+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:46.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:45.960+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:46.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:45 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:45.960+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:46.347 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:46 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:46.042+0000 7fcb4bf8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:05:46.347 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:46 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:46.042+0000 7fcb4bf8d700 -1 osd.6 1215 *** Got signal Terminated *** 2024-04-29T15:05:46.347 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:46 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:46.042+0000 7fcb4bf8d700 -1 osd.6 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:46.347 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:46 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:46.344+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:46.603 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:46 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:46.344+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:46.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:46 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:46.600+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:46.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:46 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:46.600+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:46.995 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:46.667+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:46.995 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:46.667+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:46.995 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:46.667+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:46.995 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:46.770+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:46.995 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:46.770+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:47.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:47 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:47.189+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:47.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:47 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:47.189+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:47.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:46.991+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:47.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:46 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:46.992+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:47.634 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:47 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:47.366+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:47.634 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:47 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:47.366+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:47.635 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:47 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:47.604+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:47.635 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:47 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:47.604+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:48.039 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:47 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:47.673+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:48.040 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:47 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:47.673+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:48.040 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:47 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:47.673+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:48.040 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:47 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:47.772+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:48.040 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:47 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:47.772+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:48.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:48.235+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:48.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:48.235+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:48.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:48.036+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:48.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:48.036+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:48.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:48.036+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:48.884 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:48 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:48.407+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:48.884 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:48 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:48.407+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:48.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:48 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:48.559+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:48.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:48 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:48.559+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:49.002 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:48.704+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:49.002 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:48.704+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:49.002 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:48.704+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:49.002 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:48.780+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:49.002 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:48.780+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:49.003 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:48 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:48.780+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:49.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:49.202+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:49.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:49.202+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:49.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:49.202+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:49.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:48.999+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:49.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:48.999+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:49.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:48.999+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:49.884 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:49 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:49.399+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:49.884 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:49 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:49.399+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:49.884 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:49 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:49.399+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.020220+0000 front 2024-04-29T15:05:25.020444+0000 (oldest deadline 2024-04-29T15:05:48.520308+0000) 2024-04-29T15:05:49.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:49 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:49.541+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:49.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:49 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:49.541+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:49.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:49 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:49.541+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:50.037 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:49.656+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:50.037 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:49.656+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:50.037 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:49.656+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:50.038 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:49.766+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:50.038 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:49.766+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:50.038 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:49 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:49.766+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:50.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:50.219+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:50.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:50.219+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:50.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:50.219+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:50.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:50.034+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:50.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:50.034+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:50.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:50.034+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:50.634 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:50 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:50.360+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.314037+0000 front 2024-04-29T15:05:00.314192+0000 (oldest deadline 2024-04-29T15:05:24.414208+0000) 2024-04-29T15:05:50.634 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:50 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:50.360+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:14.618157+0000 front 2024-04-29T15:05:14.618293+0000 (oldest deadline 2024-04-29T15:05:39.318242+0000) 2024-04-29T15:05:50.634 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:50 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6[64288]: 2024-04-29T15:05:50.360+0000 7fcb48581700 -1 osd.6 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.020220+0000 front 2024-04-29T15:05:25.020444+0000 (oldest deadline 2024-04-29T15:05:48.520308+0000) 2024-04-29T15:05:50.635 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:50 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:50.519+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:50.635 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:50 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:50.519+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:50.635 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:50 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:50.519+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:50.999 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:50.614+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:50.999 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:50.614+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:50.999 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:50.614+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:50.999 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:50.759+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:50.999 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:50.759+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:50.999 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:50.759+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:51.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:50.996+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:51.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:50.996+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:51.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:50 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:50.996+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:51.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:51.205+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:51.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:51.205+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:51.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:51.205+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:51.817 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:51 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:51.512+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:51.817 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:51 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:51.512+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:51.817 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:51 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:51.512+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:51.952 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:51.605+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:51.953 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:51.605+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:51.953 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:51.605+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:51.953 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:51.756+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:51.953 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:51.756+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:51.953 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:51.756+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:52.135 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:51 smithi121 bash[210889]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-6 2024-04-29T15:05:52.227 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:51.949+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:52.227 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:51.949+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:52.227 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:51 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:51.949+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:52.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:52.224+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:52.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:52.224+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:52.573 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:52.224+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:52.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:52 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:52.524+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:52.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:52 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:52.524+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:52.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:52 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:52.524+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:52.935 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:52.726+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:52.935 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:52.726+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:52.935 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:52.726+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:52.935 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:52.589+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:52.935 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:52.589+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:52.935 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:52.589+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:53.207 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:52.932+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:53.207 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:52.932+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:53.207 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:52 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:52.932+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:53.521 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:53 smithi121 systemd[1]: ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.6.service: Succeeded. 2024-04-29T15:05:53.521 INFO:journalctl@ceph.osd.6.smithi121.stdout:Apr 29 15:05:53 smithi121 systemd[1]: Stopped Ceph osd.6 for 998f2e22-0630-11ef-bc93-c7b262605968. 2024-04-29T15:05:53.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.6.service' 2024-04-29T15:05:53.542 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:53.204+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:53.542 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:53.204+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:53.542 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:53.204+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:53.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:05:53.563 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-29T15:05:53.563 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-29T15:05:53.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.7 2024-04-29T15:05:53.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:53.539+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:53.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:53.539+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:53.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:53.539+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:53.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:53.684+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:53.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:53.684+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:53.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:53.684+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:53.864 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:53 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:53.531+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:53.864 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:53 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:53.531+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:53.864 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:53 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:53.531+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:53.864 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:53 smithi121 systemd[1]: Stopping Ceph osd.7 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:05:53.864 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:53 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:53.860+0000 7f9e028a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:05:53.864 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:53 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:53.860+0000 7f9e028a1700 -1 osd.7 1215 *** Got signal Terminated *** 2024-04-29T15:05:53.864 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:53 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:53.860+0000 7f9e028a1700 -1 osd.7 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:05:54.162 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:53.904+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:54.162 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:53.904+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:54.162 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:53 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:53.904+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:54.554 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:54.159+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:54.555 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:54.159+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:54.555 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:54.159+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:54.825 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:54.552+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:54.825 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:54.552+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:54.825 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:54.552+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:54.825 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:54.636+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:54.825 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:54.636+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:54.826 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:54.636+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:54.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:54 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:54.574+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:54.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:54 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:54.574+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:54.885 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:54 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:54.574+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:55.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:54.932+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:55.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:54.932+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:55.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:54 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:54.932+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:55.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:55.112+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:55.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:55.112+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:55.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:55.112+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:55.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:55.616+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:55.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:55.616+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:55.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:55.616+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:55.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:55.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:55.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:55.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:55.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:55.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:55.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:55.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:05:55.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:55 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:55.587+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:55.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:55 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:55.587+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:55.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:55 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:55.587+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:56.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:55.924+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:56.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:55.924+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:56.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:55.924+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:56.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:55 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:55.924+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:05:56.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:56.079+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:56.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:56.079+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:56.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:56.079+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:56.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:56.568+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:56.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:56.568+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:56.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:56.568+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:56.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:56.584+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:56.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:56.584+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:56.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:56.584+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:56.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:56.584+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:05:56.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:56 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:56.618+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:56.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:56 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:56.618+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:56.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:56 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:56.618+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:56.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:56 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:56.618+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.433781+0000 front 2024-04-29T15:05:32.433887+0000 (oldest deadline 2024-04-29T15:05:56.434507+0000) 2024-04-29T15:05:57.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:56.969+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:57.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:56.969+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:57.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:56.969+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:57.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:56 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:56.969+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:05:57.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:57.060+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:57.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:57.060+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:57.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:57.060+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:57.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:57.060+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:05:57.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:57.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:57.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:57.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:57.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:57.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:57.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:57.546+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:05:57.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:57.527+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:57.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:57.527+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:57.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:57.527+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:57.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:57.527+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:05:57.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:57 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:57.581+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:57.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:57 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:57.581+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:57.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:57 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:57.581+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:57.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:57 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:57.581+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.433781+0000 front 2024-04-29T15:05:32.433887+0000 (oldest deadline 2024-04-29T15:05:56.434507+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:58.077+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:58.077+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:58.077+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:58.077+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:57.940+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:57.940+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:57.940+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:58.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:57 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:57.940+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:05:58.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:58.542+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:58.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:58.542+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:58.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:58.542+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:58.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:58.542+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:05:58.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:58.568+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:58.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:58.568+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:58.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:58.568+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:58.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:58.568+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:05:58.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:58 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:58.605+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.928436+0000 front 2024-04-29T15:04:57.928455+0000 (oldest deadline 2024-04-29T15:05:23.828296+0000) 2024-04-29T15:05:58.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:58 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:58.605+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:17.231606+0000 front 2024-04-29T15:05:17.231620+0000 (oldest deadline 2024-04-29T15:05:43.131651+0000) 2024-04-29T15:05:58.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:58 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:58.605+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.132302+0000 front 2024-04-29T15:05:23.132335+0000 (oldest deadline 2024-04-29T15:05:49.032343+0000) 2024-04-29T15:05:58.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:58 smithi121 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7[71461]: 2024-04-29T15:05:58.605+0000 7f9dfae8d700 -1 osd.7 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.433781+0000 front 2024-04-29T15:05:32.433887+0000 (oldest deadline 2024-04-29T15:05:56.434507+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:59.098+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:59.098+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:59.098+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:05:59.098+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:58.942+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:58.942+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:58.942+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:05:59.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:58 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:58.942+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:05:59.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:59.520+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:05:59.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:59.520+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:05:59.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:59.520+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:05:59.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:05:59.520+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:05:59.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:59.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:05:59.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:59.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:05:59.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:59.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:05:59.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:05:59.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:05:59.884 INFO:journalctl@ceph.osd.7.smithi121.stdout:Apr 29 15:05:59 smithi121 bash[211157]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-7 2024-04-29T15:06:00.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:59.922+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:00.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:59.922+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:00.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:59.922+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:00.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:05:59 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:05:59.922+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:00.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:00.083+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:00.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:00.083+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:00.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:00.083+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:00.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:00.083+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:00.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:00.478+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:06:00.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:00.478+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:06:00.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:00.478+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:06:00.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:00.478+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:06:00.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:00.552+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:00.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:00.552+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:00.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:00.552+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:00.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:00.552+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:01.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:01.061+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:01.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:01.061+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:01.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:01.061+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:01.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:01.061+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:01.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:00.959+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:01.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:00.959+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:01.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:00.959+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:01.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:00 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:00.959+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:01.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.7.service' 2024-04-29T15:06:01.442 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:06:01.442 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-29T15:06:01.442 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-29T15:06:01.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.8 2024-04-29T15:06:01.749 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:01.437+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:01.437+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:01.437+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:01.437+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 systemd[1]: Stopping Ceph osd.8 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:01.591+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:01.591+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:01.591+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:01.750 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:01.591+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:02.073 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:01.746+0000 7f2385be7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:06:02.073 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:01.746+0000 7f2385be7700 -1 osd.8 1215 *** Got signal Terminated *** 2024-04-29T15:06:02.073 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:01.746+0000 7f2385be7700 -1 osd.8 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:06:02.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:01.972+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:02.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:01.972+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:02.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:01.972+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:02.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:01 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:01.972+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:02.482 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:02.092+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:02.482 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:02.092+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:02.482 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:02.092+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:02.482 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:02.092+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:02.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:02.479+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:06:02.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:02.479+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:06:02.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:02.479+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:06:02.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:02.479+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:06:02.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:02.569+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:02.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:02.569+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:02.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:02.569+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:02.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:02.569+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:03.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:02.930+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:03.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:02.930+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:03.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:02.930+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:03.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:02 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:02.930+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:03.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:03.079+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:03.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:03.079+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:03.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:03.079+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:03.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:03.079+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:03.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:03.536+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:03.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:03.536+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:03.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:03.536+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:03.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:03.536+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:03.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:03.501+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:06:03.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:03.501+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:06:03.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:03.501+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:06:03.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:03.501+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:06:04.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:04.126+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:04.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:04.126+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:04.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:04.126+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:04.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:04.126+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:04.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:03.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:04.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:03.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:04.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:03.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:04.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:03 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:03.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:04.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:04.543+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:06:04.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:04.543+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:06:04.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:04.543+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:06:04.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:04.543+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:06:04.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:04.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:04.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:04.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:04.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:04.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:04.824 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:04.530+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:05.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:05.103+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:05.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:05.103+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:05.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:05.103+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:05.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:05.103+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:05.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:04.878+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:05.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:04.878+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:05.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:04.878+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:05.325 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:04.878+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:05.325 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:04 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:04.878+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:05.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:05.548+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:05.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:05.548+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:05.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:05.548+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:05.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:05.548+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:05.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:05.547+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:06:05.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:05.547+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:06:05.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:05.547+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:06:05.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:05.547+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:06:05.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:05.547+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.144488+0000 front 2024-04-29T15:05:41.144353+0000 (oldest deadline 2024-04-29T15:06:05.244324+0000) 2024-04-29T15:06:06.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:06.057+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:06.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:06.057+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:06.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:06.057+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:06.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:06.057+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:06.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:06.057+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:06.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:05.840+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:06.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:05.840+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:06.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:05.840+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:06.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:05.840+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:06.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:05 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:05.840+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:06.799 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:06.514+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:06.799 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:06.514+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:06.799 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:06.514+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:06.799 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:06.514+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:06.799 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:06.514+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:06.799 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:06.530+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:57.838051+0000 front 2024-04-29T15:04:57.838065+0000 (oldest deadline 2024-04-29T15:05:21.937855+0000) 2024-04-29T15:06:06.800 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:06.530+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:15.340767+0000 front 2024-04-29T15:05:15.340913+0000 (oldest deadline 2024-04-29T15:05:39.440664+0000) 2024-04-29T15:06:06.800 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:06.530+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:21.742053+0000 front 2024-04-29T15:05:21.741903+0000 (oldest deadline 2024-04-29T15:05:46.441833+0000) 2024-04-29T15:06:06.800 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:06.530+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:30.543064+0000 front 2024-04-29T15:05:30.542960+0000 (oldest deadline 2024-04-29T15:05:55.242978+0000) 2024-04-29T15:06:06.800 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8[106574]: 2024-04-29T15:06:06.530+0000 7f23821db700 -1 osd.8 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.144488+0000 front 2024-04-29T15:05:41.144353+0000 (oldest deadline 2024-04-29T15:06:05.244324+0000) 2024-04-29T15:06:07.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:07.013+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:07.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:07.013+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:07.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:07.013+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:07.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:07.013+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:07.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:07.013+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:07.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:06.795+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:07.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:06.795+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:07.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:06.795+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:07.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:06.795+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:07.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:06 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:06.795+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:07.823 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 29 15:06:07 smithi194 bash[212284]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-8 2024-04-29T15:06:07.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:07.486+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:07.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:07.486+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:07.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:07.486+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:07.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:07.486+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:07.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:07.486+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.000+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.000+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.000+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.000+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.000+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:07.827+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:07.827+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:08.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:07.827+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:08.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:07.827+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:08.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:07 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:07.827+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:08.813 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:08.458+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:08.813 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:08.458+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:08.813 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:08.458+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:08.813 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:08.458+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:08.813 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:08.458+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:09.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:08.810+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:09.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:08.810+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:09.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:08.810+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:09.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:08.810+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:09.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:08.810+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:09.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.976+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:09.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.976+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:09.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.976+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:09.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.976+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:09.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:08 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:08.976+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:09.679 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:09.421+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:09.680 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:09.421+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:09.680 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:09.421+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:09.680 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:09.421+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:09.680 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:09.421+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:09.687 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.8.service' 2024-04-29T15:06:09.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:06:09.730 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-29T15:06:09.730 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-29T15:06:09.730 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.9 2024-04-29T15:06:10.025 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:09 smithi194 systemd[1]: Stopping Ceph osd.9 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:06:10.025 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:09.777+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:10.025 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:09.777+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:10.025 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:09.777+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:10.026 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:09.777+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:10.026 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:09 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:09.777+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.055+0000 7fb54e1e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.055+0000 7fb54e1e0700 -1 osd.9 1215 *** Got signal Terminated *** 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.055+0000 7fb54e1e0700 -1 osd.9 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:10.022+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:10.022+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:10.022+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:10.022+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:10.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:10.022+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:10.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.396+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:10.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.396+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:10.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.396+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:10.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.396+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:10.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:10.396+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:11.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.003+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:11.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.003+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:11.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.003+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:11.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.003+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:11.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.003+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:11.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:10.820+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:11.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:10.820+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:11.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:10.820+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:11.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:10.820+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:11.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:10 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:10.820+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:11.796 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:11.351+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:11.796 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:11.351+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:11.796 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:11.351+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:11.796 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:11.351+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:11.797 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:11.351+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:12.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:11.794+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:12.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:11.794+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:12.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:11.794+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:12.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:11.794+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:12.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:11.794+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:12.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.983+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:12.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.983+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:12.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.983+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:12.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.983+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:12.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:11 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:11.983+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:12.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:12.398+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:12.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:12.398+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:12.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:12.398+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:12.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:12.398+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:12.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:12.398+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:12.829+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:12.829+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:12.829+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:12.829+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:12.829+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:12.958+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:12.958+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:12.958+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:12.958+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:13.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:12.958+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:13.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:12 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:12.958+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:13.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:13.412+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:13.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:13.413+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:13.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:13.413+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:13.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:13.413+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:13.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:13.413+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:14.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:13.865+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:14.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:13.865+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:14.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:13.865+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:14.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:13.865+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:14.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:13.865+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:14.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:13.937+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:14.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:13.937+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:14.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:13.937+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:14.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:13.937+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:14.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:13.937+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:14.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:13 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:13.937+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:14.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:14.371+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6807 osd.1 since back 2024-04-29T15:05:08.029471+0000 front 2024-04-29T15:05:08.029409+0000 (oldest deadline 2024-04-29T15:05:30.929260+0000) 2024-04-29T15:06:14.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:14.371+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.731167+0000 front 2024-04-29T15:05:16.731075+0000 (oldest deadline 2024-04-29T15:05:38.430961+0000) 2024-04-29T15:06:14.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:14.371+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:22.531997+0000 front 2024-04-29T15:05:22.532013+0000 (oldest deadline 2024-04-29T15:05:47.831882+0000) 2024-04-29T15:06:14.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:14.371+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.234402+0000 front 2024-04-29T15:05:32.234433+0000 (oldest deadline 2024-04-29T15:05:56.934157+0000) 2024-04-29T15:06:14.823 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9[195422]: 2024-04-29T15:06:14.371+0000 7fb54a7d4700 -1 osd.9 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:42.735729+0000 front 2024-04-29T15:05:42.735696+0000 (oldest deadline 2024-04-29T15:06:06.235587+0000) 2024-04-29T15:06:15.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:14.881+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:15.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:14.881+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:15.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:14.881+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:15.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:14.881+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:15.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:14.881+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:15.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:14.904+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:15.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:14.904+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:15.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:14.904+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:15.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:14.904+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:15.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:14.904+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:15.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:14 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:14.904+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:16.073 INFO:journalctl@ceph.osd.9.smithi194.stdout:Apr 29 15:06:15 smithi194 bash[212549]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-9 2024-04-29T15:06:16.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:15.909+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:16.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:15.909+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:16.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:15.909+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:15.909+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:15.909+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:15.909+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:15.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:15.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:15.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:15.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:16.074 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:15 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:15.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:17.270 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:16.891+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:16.891+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:16.891+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:16.891+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:16.891+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:16.891+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:16.860+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:16.860+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:16.860+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:16.860+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:16.860+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:17.271 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:16 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:16.860+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:17.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.9.service' 2024-04-29T15:06:17.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:06:17.590 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-29T15:06:17.590 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-29T15:06:17.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.10 2024-04-29T15:06:17.823 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 systemd[1]: Stopping Ceph osd.10 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:06:18.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:17.836+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:18.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:17.836+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:18.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:17.836+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:18.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:17.836+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:18.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:17.836+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:18.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:17.836+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.881+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.881+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.881+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.881+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.881+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.881+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.885+0000 7f5af1f99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.885+0000 7f5af1f99700 -1 osd.10 1215 *** Got signal Terminated *** 2024-04-29T15:06:18.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:17 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:17.885+0000 7f5af1f99700 -1 osd.10 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:06:19.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:18.850+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:19.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:18.850+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:19.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:18.850+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:19.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:18.850+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:19.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:18.850+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:19.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:18.850+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:19.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:18.861+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:19.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:18.861+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:19.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:18.861+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:19.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:18.861+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:19.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:18.861+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:19.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:18 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:18.861+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:19.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:19.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:19.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:19.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:19.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:19.867+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:19.874+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:19.874+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:19.874+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:19.874+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:20.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:19.874+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:20.325 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:19 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:19.874+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:21.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:20.842+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:21.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:20.842+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:21.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:20.842+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:21.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:20.842+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:21.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:20.842+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:21.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:20.842+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:21.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:20.907+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:21.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:20.907+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:21.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:20.907+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:21.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:20.907+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:21.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:20.907+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:21.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:20.907+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:21.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:20 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:20.907+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:21.890+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:21.890+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:21.890+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:21.890+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:21.890+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:21.890+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:21.890+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:22.323 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:21.885+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:05:00.627081+0000 front 2024-04-29T15:05:00.626890+0000 (oldest deadline 2024-04-29T15:05:22.326844+0000) 2024-04-29T15:06:22.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:21.885+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:12.228744+0000 front 2024-04-29T15:05:12.228768+0000 (oldest deadline 2024-04-29T15:05:38.128877+0000) 2024-04-29T15:06:22.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:21.885+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:25.630923+0000 front 2024-04-29T15:05:25.630990+0000 (oldest deadline 2024-04-29T15:05:48.530905+0000) 2024-04-29T15:06:22.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:21.885+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.532818+0000 front 2024-04-29T15:05:32.532732+0000 (oldest deadline 2024-04-29T15:05:56.632676+0000) 2024-04-29T15:06:22.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:21.885+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:41.933711+0000 front 2024-04-29T15:05:41.933594+0000 (oldest deadline 2024-04-29T15:06:05.433692+0000) 2024-04-29T15:06:22.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:21.885+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.035448+0000 front 2024-04-29T15:05:50.035486+0000 (oldest deadline 2024-04-29T15:06:12.335256+0000) 2024-04-29T15:06:22.324 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:21 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10[65079]: 2024-04-29T15:06:21.885+0000 7f5aee58d700 -1 osd.10 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:55.836477+0000 front 2024-04-29T15:05:55.836523+0000 (oldest deadline 2024-04-29T15:06:21.736461+0000) 2024-04-29T15:06:23.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:22.870+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:23.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:22.870+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:23.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:22.870+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:23.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:22.870+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:23.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:22.870+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:23.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:22.870+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:23.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:22 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:22.870+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.10.smithi194.stdout:Apr 29 15:06:23 smithi194 bash[212818]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-10 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:23.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:23.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:23.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:23.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:23.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:23.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:24.073 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:23 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:23.917+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:25.272 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:24.871+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:25.272 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:24.871+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:25.272 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:24.871+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:25.272 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:24.871+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:25.272 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:24.871+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:25.272 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:24.871+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:25.272 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:24 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:24.871+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:25.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.10.service' 2024-04-29T15:06:25.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:06:25.597 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-29T15:06:25.597 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-29T15:06:25.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.11 2024-04-29T15:06:25.823 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 systemd[1]: Stopping Ceph osd.11 for 998f2e22-0630-11ef-bc93-c7b262605968... 2024-04-29T15:06:26.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.895+0000 7fcdb10dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.895+0000 7fcdb10dd700 -1 osd.11 1215 *** Got signal Terminated *** 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.895+0000 7fcdb10dd700 -1 osd.11 1215 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.900+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.900+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.900+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.900+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.900+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.900+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:26.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:25 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:25.900+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:27.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:26.887+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:27.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:26.887+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:27.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:26.887+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:27.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:26.887+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:27.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:26.887+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:27.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:26.887+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:27.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:26 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:26.887+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:28.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:27.853+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:28.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:27.853+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:28.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:27.853+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:28.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:27.853+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:28.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:27.853+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:28.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:27.853+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:28.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:27 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:27.853+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:29.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:28.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:29.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:28.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:29.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:28.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:29.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:28.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:29.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:28.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:29.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:28.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:29.324 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:28 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:28.894+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:30.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:29 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:29.857+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.194:6802 osd.8 since back 2024-04-29T15:06:05.437847+0000 front 2024-04-29T15:06:05.437873+0000 (oldest deadline 2024-04-29T15:06:29.537950+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6803 osd.0 since back 2024-04-29T15:04:59.826052+0000 front 2024-04-29T15:04:59.826337+0000 (oldest deadline 2024-04-29T15:05:23.926040+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6811 osd.2 since back 2024-04-29T15:05:16.629373+0000 front 2024-04-29T15:05:16.629609+0000 (oldest deadline 2024-04-29T15:05:43.029899+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.40:6815 osd.3 since back 2024-04-29T15:05:23.531239+0000 front 2024-04-29T15:05:23.531163+0000 (oldest deadline 2024-04-29T15:05:47.630978+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6802 osd.4 since back 2024-04-29T15:05:32.832646+0000 front 2024-04-29T15:05:32.832676+0000 (oldest deadline 2024-04-29T15:05:55.732459+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6806 osd.5 since back 2024-04-29T15:05:38.033651+0000 front 2024-04-29T15:05:38.033713+0000 (oldest deadline 2024-04-29T15:06:03.933525+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6810 osd.6 since back 2024-04-29T15:05:50.736219+0000 front 2024-04-29T15:05:50.736285+0000 (oldest deadline 2024-04-29T15:06:16.036290+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.121:6814 osd.7 since back 2024-04-29T15:05:56.036868+0000 front 2024-04-29T15:05:56.036843+0000 (oldest deadline 2024-04-29T15:06:20.736852+0000) 2024-04-29T15:06:31.323 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:30 smithi194 ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11[72952]: 2024-04-29T15:06:30.886+0000 7fcda96c9700 -1 osd.11 1215 heartbeat_check: no reply from 172.21.15.194:6802 osd.8 since back 2024-04-29T15:06:05.437847+0000 front 2024-04-29T15:06:05.437873+0000 (oldest deadline 2024-04-29T15:06:29.537950+0000) 2024-04-29T15:06:31.823 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 29 15:06:31 smithi194 bash[213082]: ceph-998f2e22-0630-11ef-bc93-c7b262605968-osd-11 2024-04-29T15:06:33.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-998f2e22-0630-11ef-bc93-c7b262605968@osd.11.service' 2024-04-29T15:06:33.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-29T15:06:33.598 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-29T15:06:33.598 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 998f2e22-0630-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-29T15:06:33.839 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T15:06:35.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 998f2e22-0630-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-29T15:06:35.393 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T15:06:36.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 998f2e22-0630-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-29T15:06:36.742 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T15:06:37.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-29T15:06:37.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-29T15:06:37.795 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-29T15:06:37.825 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-29T15:06:37.827 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi040/crash 2024-04-29T15:06:37.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash -- . 2024-04-29T15:06:37.888 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-29T15:06:37.888 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-04-29T15:06:37.890 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi121/crash 2024-04-29T15:06:37.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash -- . 2024-04-29T15:06:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-29T15:06:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2024-04-29T15:06:37.959 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi194/crash 2024-04-29T15:06:37.960 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash -- . 2024-04-29T15:06:38.033 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/998f2e22-0630-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-29T15:06:38.034 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-04-29T15:06:38.035 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-29T15:06:38.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-29T15:06:38.137 INFO:teuthology.orchestra.run.smithi040.stdout:2024-04-29T14:14:35.207826+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-29T15:06:38.138 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-29T15:06:38.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[SEC\]' /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-29T15:06:38.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]' /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-29T15:06:38.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[WRN\]' /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-29T15:06:38.340 INFO:teuthology.orchestra.run.smithi040.stdout:2024-04-29T14:14:35.207826+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-29T15:06:38.340 INFO:tasks.cephadm:Compressing logs... 2024-04-29T15:06:38.341 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-29T15:06:38.383 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-29T15:06:38.385 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-29T15:06:38.426 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-29T15:06:38.426 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-29T15:06:38.429 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-29T15:06:38.429 INFO:teuthology.orchestra.run.smithi040.stderr:: No such file or directory 2024-04-29T15:06:38.434 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-29T15:06:38.434 INFO:teuthology.orchestra.run.smithi121.stderr:: No such file or directory 2024-04-29T15:06:38.435 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-29T15:06:38.436 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-29T15:06:38.436 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-29T15:06:38.437 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.11.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-29T15:06:38.437 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.6.log 2024-04-29T15:06:38.437 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-29T15:06:38.437 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-29T15:06:38.437 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log 2024-04-29T15:06:38.437 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-29T15:06:38.438 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-29T15:06:38.438 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log 2024-04-29T15:06:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-29T15:06:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-29T15:06:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-29T15:06:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-29T15:06:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-29T15:06:38.440 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-29T15:06:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log 2024-04-29T15:06:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-29T15:06:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log 2024-04-29T15:06:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-29T15:06:38.443 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log 2024-04-29T15:06:38.444 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-29T15:06:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.a.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.1.log: 82.8% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-29T15:06:38.491 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.4.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/cephadm.log: 91.4% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-29T15:06:38.515 INFO:teuthology.orchestra.run.smithi121.stderr: 82.5% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-29T15:06:38.562 INFO:teuthology.orchestra.run.smithi040.stderr: 94.3% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-29T15:06:38.577 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log: 82.4% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-29T15:06:38.632 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-29T15:06:38.808 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-29T15:06:38.852 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-29T15:06:38.866 INFO:teuthology.orchestra.run.smithi194.stderr: 91.2% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-29T15:06:38.947 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-29T15:06:39.011 INFO:teuthology.orchestra.run.smithi194.stderr: 89.6% 95.4% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-29T15:06:39.011 INFO:teuthology.orchestra.run.smithi194.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-29T15:06:39.115 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-29T15:06:39.135 INFO:teuthology.orchestra.run.smithi121.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-29T15:06:39.137 INFO:teuthology.orchestra.run.smithi194.stderr: 87.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-29T15:06:39.950 INFO:teuthology.orchestra.run.smithi194.stderr: 94.2% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.6.log.gz 2024-04-29T15:06:39.985 INFO:teuthology.orchestra.run.smithi040.stderr: 96.0% 87.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.log.gz -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-29T15:06:39.985 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-29T15:06:39.997 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-29T15:06:40.150 INFO:teuthology.orchestra.run.smithi194.stderr: 94.0% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-29T15:06:40.939 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-29T15:06:42.790 INFO:teuthology.orchestra.run.smithi194.stderr: 90.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-29T15:06:43.506 INFO:teuthology.orchestra.run.smithi121.stderr: 90.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-29T15:06:51.297 INFO:teuthology.orchestra.run.smithi040.stderr: 89.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-29T15:10:14.640 INFO:teuthology.orchestra.run.smithi040.stderr: 93.4% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-29T15:11:30.441 INFO:teuthology.orchestra.run.smithi194.stderr: 92.3% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-29T15:12:02.157 INFO:teuthology.orchestra.run.smithi040.stderr: 92.2% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-29T15:14:02.577 INFO:teuthology.orchestra.run.smithi194.stderr: 91.2% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-29T15:14:17.045 INFO:teuthology.orchestra.run.smithi040.stderr: 90.5% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-29T15:15:09.879 INFO:teuthology.orchestra.run.smithi121.stderr: 92.8% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-29T15:15:10.775 INFO:teuthology.orchestra.run.smithi194.stderr: 91.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-29T15:15:14.437 INFO:teuthology.orchestra.run.smithi121.stderr: 92.6% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-29T15:15:19.387 INFO:teuthology.orchestra.run.smithi121.stderr: 92.6% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-29T15:15:31.017 INFO:teuthology.orchestra.run.smithi194.stderr: 92.1% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-29T15:15:31.021 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-04-29T15:15:31.021 INFO:teuthology.orchestra.run.smithi194.stderr:real 8m52.617s 2024-04-29T15:15:31.021 INFO:teuthology.orchestra.run.smithi194.stderr:user 11m15.152s 2024-04-29T15:15:31.022 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m27.594s 2024-04-29T15:16:00.983 INFO:teuthology.orchestra.run.smithi040.stderr: 90.7% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-29T15:16:00.987 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-29T15:16:00.987 INFO:teuthology.orchestra.run.smithi040.stderr:real 9m22.587s 2024-04-29T15:16:00.987 INFO:teuthology.orchestra.run.smithi040.stderr:user 11m53.366s 2024-04-29T15:16:00.987 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m26.290s 2024-04-29T15:16:41.657 INFO:teuthology.orchestra.run.smithi121.stderr: 91.8% -- replaced with /var/log/ceph/998f2e22-0630-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-29T15:16:41.658 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-04-29T15:16:41.658 INFO:teuthology.orchestra.run.smithi121.stderr:real 10m3.252s 2024-04-29T15:16:41.658 INFO:teuthology.orchestra.run.smithi121.stderr:user 11m47.820s 2024-04-29T15:16:41.658 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m28.901s 2024-04-29T15:16:41.659 INFO:tasks.cephadm:Archiving logs... 2024-04-29T15:16:41.659 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi040/log 2024-04-29T15:16:41.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-29T15:18:40.831 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi121/log 2024-04-29T15:18:40.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-29T15:20:35.362 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi194/log 2024-04-29T15:20:35.363 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-29T15:22:29.731 INFO:tasks.cephadm:Removing cluster... 2024-04-29T15:22:29.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 998f2e22-0630-11ef-bc93-c7b262605968 --force 2024-04-29T15:22:30.086 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T15:22:31.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 998f2e22-0630-11ef-bc93-c7b262605968 --force 2024-04-29T15:22:31.689 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T15:22:32.703 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 998f2e22-0630-11ef-bc93-c7b262605968 --force 2024-04-29T15:22:33.156 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 998f2e22-0630-11ef-bc93-c7b262605968 2024-04-29T15:22:34.138 INFO:tasks.cephadm:Removing cephadm ... 2024-04-29T15:22:34.138 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-29T15:22:34.171 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-29T15:22:34.203 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-29T15:22:34.239 INFO:tasks.cephadm:Teardown complete 2024-04-29T15:22:34.240 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-29T15:22:34.253 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-29T15:22:34.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-29T15:22:34.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-29T15:22:34.258 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-29T15:22:34.404 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-29T15:22:34.413 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-29T15:22:34.413 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T15:22:34.416 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T15:22:34.419 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-29T15:22:34.434 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2024-04-29T15:22:34.436 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-04-29T15:22:34.437 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2024-04-29T15:22:34.456 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T15:22:34.456 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-04-29T15:22:34.456 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv01.front.sepia.ceph.com 3 8 377 227 +353us[ +442us] +/- 58ms 2024-04-29T15:22:34.456 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1755us[-1755us] +/- 72ms 2024-04-29T15:22:34.456 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +1535us[+1535us] +/- 67ms 2024-04-29T15:22:34.456 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T15:22:34.480 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T15:22:34.480 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-29T15:22:34.481 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +290us[ +334us] +/- 61ms 2024-04-29T15:22:34.481 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1820us[-1820us] +/- 72ms 2024-04-29T15:22:34.481 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +1450us[+1450us] +/- 67ms 2024-04-29T15:22:34.481 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T15:22:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-29T15:22:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-29T15:22:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv01.front.sepia.ceph.com 3 6 377 33 +275us[ +292us] +/- 61ms 2024-04-29T15:22:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1827us[-1810us] +/- 72ms 2024-04-29T15:22:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +1448us[+1448us] +/- 67ms 2024-04-29T15:22:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-29T15:22:34.517 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-29T15:22:34.526 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-29T15:22:34.527 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-29T15:22:34.550 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-29T15:22:34.793 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-29T15:22:34.882 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-29T15:22:34.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-29T15:22:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T15:22:35.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-29T15:22:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T15:22:35.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-29T15:22:35.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T15:22:35.078 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-29T15:22:35.088 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-29T15:22:35.142 INFO:teuthology.task.internal:Duration was 5571.739976 seconds 2024-04-29T15:22:35.143 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-29T15:22:35.176 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-29T15:22:35.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-29T15:22:35.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-29T15:22:35.181 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-29T15:22:35.264 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T15:22:35.291 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T15:22:35.305 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-29T15:22:35.669 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-29T15:22:35.669 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-29T15:22:35.669 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-29T15:22:35.711 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-04-29T15:22:35.711 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-29T15:22:35.743 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-04-29T15:22:35.744 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-29T15:22:35.780 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-29T15:22:35.780 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-29T15:22:35.783 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-29T15:22:35.786 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-29T15:22:35.962 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-29T15:22:35.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-29T15:22:35.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-29T15:22:35.968 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-29T15:22:36.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-29T15:22:36.300 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-29T15:22:36.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-29T15:22:36.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-29T15:22:36.421 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-29T15:22:36.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-29T15:22:36.507 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-29T15:22:36.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-29T15:22:36.512 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-29T15:22:36.541 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-04-29T15:22:36.550 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-04-29T15:22:36.553 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-29T15:22:36.568 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-29T15:22:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T15:22:36.586 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-29T15:22:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T15:22:36.622 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-29T15:22:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-29T15:22:36.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-29T15:22:36.650 INFO:teuthology.task.internal:Transferring archived files... 2024-04-29T15:22:36.651 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi040 2024-04-29T15:22:36.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-29T15:22:36.726 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi121 2024-04-29T15:22:36.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-29T15:22:36.778 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421/remote/smithi194 2024-04-29T15:22:36.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-29T15:22:36.829 INFO:teuthology.task.internal:Removing archive directory... 2024-04-29T15:22:36.830 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-29T15:22:36.832 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-29T15:22:36.835 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-29T15:22:36.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-29T15:22:36.891 INFO:teuthology.task.internal:Not uploading archives. 2024-04-29T15:22:36.891 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-29T15:22:36.918 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-29T15:22:36.918 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-29T15:22:36.921 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-29T15:22:36.923 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-29T15:22:36.936 INFO:teuthology.orchestra.run.smithi040.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 29 15:22 /home/ubuntu/cephtest 2024-04-29T15:22:36.939 INFO:teuthology.orchestra.run.smithi121.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 29 15:22 /home/ubuntu/cephtest 2024-04-29T15:22:36.941 INFO:teuthology.orchestra.run.smithi194.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 29 15:22 /home/ubuntu/cephtest 2024-04-29T15:22:36.942 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-29T15:22:36.955 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-29T15:22:37.016 INFO:teuthology.nuke:Checking targets against current locks 2024-04-29T15:22:37.039 DEBUG:teuthology.nuke:shortname: smithi040 2024-04-29T15:22:37.040 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-29T15:22:37.063 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 13:41:08.899798', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJmHQQkD1bREDUR//8JHRmFsHHxnXXjYGdMeG8lxVxeTmtyrZm8NJXJWTRvSgb+4/L98vO+TzhzZw6jRJBnsA2o='} 2024-04-29T15:22:37.067 DEBUG:teuthology.nuke:shortname: smithi121 2024-04-29T15:22:37.067 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-29T15:22:37.090 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 13:41:08.903325', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA1VeLUMJ6vUm3FCY9hiAxRSrogfkKei1PjjD4Cu3FvinZMnK9/482YsgEEZ4c2eFN8ftqJ6BPgE70NXO5CNFas='} 2024-04-29T15:22:37.093 DEBUG:teuthology.nuke:shortname: smithi194 2024-04-29T15:22:37.094 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-29T15:22:37.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-04-29_10:44:48-rados-wip-pereman2-buffer-space-distro-default-smithi/7678421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-29 13:41:08.901804', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjeWGqWUSXhzb7sXQf/Q0dcM/QY+48mbtd09GedxFcMBZ9JffGaI6hrOGJZYVNQxT2ja7ImS64Bxf+a74Q0g24='} 2024-04-29T15:22:37.127 INFO:teuthology.orchestra.console.smithi040:Power off 2024-04-29T15:22:37.127 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-29T15:22:37.152 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-04-29T15:22:37.153 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:37.167 INFO:teuthology.orchestra.console.smithi121:Power off 2024-04-29T15:22:37.167 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-29T15:22:37.178 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-29T15:22:37.188 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-04-29T15:22:37.188 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:37.205 INFO:teuthology.orchestra.console.smithi194:Power off 2024-04-29T15:22:37.205 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-29T15:22:37.214 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-29T15:22:37.228 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-04-29T15:22:37.228 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:37.249 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-29T15:22:41.179 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:41.216 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:41.250 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:41.288 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-29T15:22:41.325 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-29T15:22:41.360 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-29T15:22:45.294 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:45.326 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:45.362 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:45.406 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-04-29T15:22:45.437 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-29T15:22:45.471 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-29T15:22:45.507 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-04-29T15:22:49.438 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:49.472 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-29T15:22:49.551 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-04-29T15:22:49.584 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-04-29T15:22:49.651 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-04-29T15:22:49.685 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-04-29T15:22:49.791 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/radosbench} duration: 5571.739976406097 failure_reason: '"2024-04-29T14:14:35.207826+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' flavor: default owner: scheduled_pdiazbou@teuthology success: false 2024-04-29T15:22:49.792 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-29T15:22:49.893 INFO:teuthology.run:FAIL